New update has come from Google which say that Google to stop supporting no-index directive in robot.txt file. This rule will effective from September 1,2019. Google will stop supporting unsupported and unpublished rules in the robots exclusive protocol. Google will no longer support robots.txt files with the no-index directive listed within the file.
For those of you who relied on the no-index indexing directive in the robots.txt file, which controls crawling, there are a number of alternative option.
Google listed the following options, the ones you probably should have been using anyway:-
Noindex in robots meta tags: Supported both in the HTTP response headers and in HTML, the no-index directive is the most effective way to remove URLs from the index when crawling is allowed.
404 and 410 HTTP status codes: Both status codes mean that the page does not exist, which will drop such URLs from Google’s index once they’re crawled and processed.
Password protection: Unless markup is used to indicate subscription or paywalled content, hiding a page behind a login will generally remove it from Google’s index.
Disallow in robots.txt: Search engines can only index pages that they know about, so blocking the page from being crawled often means its content won’t be indexed. While the search engine may also index a URL based on links from other pages, without seeing the content itself, we aim to make such pages less visible in the future.
Search Console Remove URL tool: The tool is a quick and easy method to remove a URL temporarily from Google’s search results.
Google has been looking to change this for years and with standardizing the protocol, it can now move forward. Google said it “analyzed the usage of robots.txt rules.” Google focuses on looking at unsupported implementations of the internet draft, such as crawl-delay, no-follow, and no-index. “Since these rules were never documented by Google, naturally, their usage in relation to Google bot is very low,” Google said. “These mistakes hurt websites presence in Google’s search results in ways we don’t think webmasters intended.”
The most important thing is to make sure that you are not using the no-index directive in the robots.txt file. If you are, you will want to make the suggested changes above before September 1.
Source : Search Engine Land
Learn About SEO On Page Techique
For those of you who relied on the no-index indexing directive in the robots.txt file, which controls crawling, there are a number of alternative option.
Google listed the following options, the ones you probably should have been using anyway:-
Noindex in robots meta tags: Supported both in the HTTP response headers and in HTML, the no-index directive is the most effective way to remove URLs from the index when crawling is allowed.
404 and 410 HTTP status codes: Both status codes mean that the page does not exist, which will drop such URLs from Google’s index once they’re crawled and processed.
Password protection: Unless markup is used to indicate subscription or paywalled content, hiding a page behind a login will generally remove it from Google’s index.
Disallow in robots.txt: Search engines can only index pages that they know about, so blocking the page from being crawled often means its content won’t be indexed. While the search engine may also index a URL based on links from other pages, without seeing the content itself, we aim to make such pages less visible in the future.
Search Console Remove URL tool: The tool is a quick and easy method to remove a URL temporarily from Google’s search results.
Google has been looking to change this for years and with standardizing the protocol, it can now move forward. Google said it “analyzed the usage of robots.txt rules.” Google focuses on looking at unsupported implementations of the internet draft, such as crawl-delay, no-follow, and no-index. “Since these rules were never documented by Google, naturally, their usage in relation to Google bot is very low,” Google said. “These mistakes hurt websites presence in Google’s search results in ways we don’t think webmasters intended.”
The most important thing is to make sure that you are not using the no-index directive in the robots.txt file. If you are, you will want to make the suggested changes above before September 1.
Source : Search Engine Land
Learn About SEO On Page Techique
