Do you know that Google Search checks about 4 billion host names every day for robots.txt functions? Gary Illyes mentioned within the December Search Off The Report podcast “now we have about 4 billion host names that we verify each single day for robots.txt.”
He mentioned this on the 20:31 mark within the video. He mentioned in the event that they verify 4 billion host names each day, then “the variety of websites might be over or very probably over 4 billion.”
I noticed this video Glenn Gabe:
Google’s Gary Illyes within the newest SOTR Podcast: Google has about 4 billion hostnames that it checks each single day for robots.txt https://t.co/Irc2outOM4 pic.twitter.com/lyb68pnR7d
— Glenn Gabe (@glenngabe) December 22, 2023
Right here is the transcript:
GARY ILLYES: Yeah, and I imply, that is one of many issues that we introduced up early on. If we implement one thing or if we come up or counsel one thing that might work, that ought to not put extra pressure on publishers as a result of if you concentrate on it, should you undergo our robots.txt cache, you may see that now we have about 4 billion host names that we verify each single day for robots.txt. Now, to illustrate that every one of these have subdirectories, for instance. So the variety of websites might be over or very probably over 4 billion.
JOHN MUELLER: What number of of these are in Search Console? I’m wondering.
GARY ILLYES: John, cease it.
JOHN MUELLER: I am sorry.
GARY ILLYES: Anyway, so you probably have 4 billion hostnames plus a bunch extra in subdirectories, then how do you implement one thing that won’t make them go bankrupt after they wish to implement some choose out mechanism?
JOHN MUELLER: It is sophisticated.
GARY ILLYES: It is sophisticated. And I do know that persons are annoyed that we do not have one thing already. However it’s not one thing to–
MARTIN SPLITT: Be taken flippantly, yeah.
GARY ILLYES: Yeah.
Right here is the video embed firstly time:
Discussion board dialogue at X.