John Mueller from Google explained why Search Console shows thousands of URLs as indexed even though they are blocked by robots.txt and provided advice on how to address this issue.
Content is indexed even if it is blocked by robots.txt.
A Reddit user sought help regarding a situation where Google Search Console showed over 51,000 pages labeled as “Indexed, though blocked by robots.txt.” These pages were mostly WooCommerce product URLs that included add-to-cart URL parameters such as “?add-to-cart=”.
The site owner wondered if the sudden issue was caused by the robots.txt rules and considered whether removing them would assist Google in processing canonical signals and removing the reported URLs from Search Console.
The individual inquired:
Since last month, our WooCommerce website has been experiencing the problem of “Indexed, but blocked by robots.txt.”
Total of 51,000 pages have been affected.
At the end of the URL, I often see ?page&post_type=product&product=slug&add-to-cart=98063.
After examining those URLs, I discovered that they are configured with an index tag and that the robots.txt file was present.
Prevent access to URLs containing “?add-to-cart=” and “?*add-to-cart=”.
I deleted the two rules from robots.txt and am optimistic that the problem with those pages will be resolved since they have the correct product set as canonical. Will that address the issue?
Should I implement noindex rules as well? Will this impact our crawl budget for our large WooCommerce site? I would appreciate hearing your thoughts and any advice on how to address this issue without harming our SEO or functionality.
Google indicates that there is no need to index Add-To-Cart URLs.
Mueller stated that add-to-cart URLs do not require indexing and can be blocked using robots.txt.
Even if Google shows those URLs as indexed, they are unlikely to show up in regular search results due to being blocked by robots.txt. Users typically do not search for those URLs directly, so they are not likely to be visible in search results.
John Mueller replied:
Blocking add-to-cart URLs in robots.txt is sufficient because even if they are indexed, they are unlikely to appear in search results unless specific queries are made for them.
Mueller’s statement about “robots.txt” making it “unlikely” for URLs to appear in Search is questionable because robots.txt only prevents Googlebot from crawling pages, not from showing them in Google Search.
Noindex may not be an effective solution.
One Redditor proposed adding a noindex robots tag to the parameterized URLs as a potential solution. However, this may not work effectively since the pages with and without the URL parameters are essentially identical. Unless WooCommerce can differentiate between them and apply the noindex to the parameterized URLs while excluding it from the regular page, this approach is not a viable solution.
Google reports URLs that are indexed but cannot be crawled.
Another Reddit user provided a potential reason for the plethora of URLs shown in Search Console, proposing that Google might have come across links with add-to-cart parameters on the website and consequently included those URLs in its databases.
I recommend that the individual who posed the initial question uses Screaming Frog to analyze the website, examines the internal linking to locate the pages linking to those pages, and then decides on a course of action, such as removing the links or adding a rel=”nofollow” attribute to them.
The most effective approach is to utilize the robots.txt block to stop crawling, keeping in mind that this is its only function. For added assurance, one can pinpoint the locations of these links and include the nofollow attribute as an additional precautionary measure, signaling to Google. Nofollow serves as a strong suggestion, though it is not a directive.
Search Console Warnings may not necessarily signal a search issue.
Search Console reports may highlight technical issues that appear concerning but may not impact search performance significantly. For instance, while 404 error reports offer valuable insights, a 404 server response may not necessarily require fixing as it may be the appropriate response in certain cases.
Takeout
Mueller’s reply emphasizes that not all Search Console alerts need immediate action, but in this instance, addressing internal links with shopping cart URL parameters might be necessary. If these links are essential, adding a rel=”nofollow” attribute can signal Google not to follow them. This showcases the intricacies of technical SEO.
Image provided by Shutterstock/Orange Line Media
