.Google.com's John Mueller responded to an inquiry concerning why Google.com marks pages that are refused coming from creeping through robots.txt and why the it is actually safe to dismiss the associated Browse Console reports concerning those creeps.Robot Traffic To Concern Specification URLs.The individual talking to the concern chronicled that robots were developing hyperlinks to non-existent concern criterion Links (? q= xyz) to webpages along with noindex meta tags that are actually also blocked in robots.txt. What cued the inquiry is that Google.com is creeping the hyperlinks to those pages, obtaining blocked out by robots.txt (without noticing a noindex robots meta tag) then acquiring reported in Google.com Search Console as "Indexed, though shut out by robots.txt.".The person asked the observing question:." But listed below is actually the major question: why will Google index web pages when they can't also view the web content? What is actually the perk in that?".Google.com's John Mueller confirmed that if they can't crawl the web page they can't see the noindex meta tag. He also produces a fascinating acknowledgment of the site: hunt driver, encouraging to ignore the end results due to the fact that the "ordinary" users won't observe those outcomes.He wrote:." Yes, you're proper: if our company can not crawl the webpage, we can't see the noindex. That stated, if we can not creep the pages, after that there's certainly not a great deal for us to mark. Thus while you could see a few of those web pages with a targeted website:- concern, the typical individual will not observe them, so I would not bother it. Noindex is also alright (without robots.txt disallow), it merely implies the Links will wind up being crawled (and end up in the Browse Console document for crawled/not catalogued-- neither of these statuses induce problems to the remainder of the website). The important part is that you don't make all of them crawlable + indexable.".Takeaways:.1. Mueller's response confirms the limits being used the Website: search progressed hunt operator for analysis reasons. One of those explanations is since it's certainly not linked to the normal hunt mark, it is actually a distinct thing completely.Google.com's John Mueller talked about the site search driver in 2021:." The quick response is actually that an internet site: question is actually not suggested to become comprehensive, neither made use of for diagnostics purposes.A web site inquiry is actually a details sort of search that confines the end results to a particular site. It is actually generally simply the word website, a colon, and after that the website's domain.This question limits the end results to a specific site. It's not implied to become a complete assortment of all the web pages from that web site.".2. Noindex tag without utilizing a robots.txt is actually great for these kinds of scenarios where a crawler is actually linking to non-existent web pages that are actually receiving found through Googlebot.3. URLs with the noindex tag will certainly generate a "crawled/not listed" entry in Search Console and that those will not possess a bad result on the rest of the internet site.Check out the inquiry as well as address on LinkedIn:.Why would certainly Google.com mark webpages when they can't also find the material?Included Image by Shutterstock/Krakenimages. com.