Gary Illyes posted a brand new weblog submit on the Google Search Central web site asking all of you to cease utilizing 403 and 404 server standing codes to scale back the crawl fee of Googlebot. He mentioned they’ve seen an uptick within the variety of websites and CDNs doing this and they should minimize it out.
Gary wrote, “Over the previous few months we seen an uptick in web site homeowners and a few content material supply networks (CDNs) trying to make use of 404 and different 4xx shopper errors (however not 429) to aim to scale back Googlebot’s crawl fee.” “The quick model of this weblog submit is: please do not try this,” he added.
As an alternative, he mentioned Google has documentation about how one can cut back Googlebot’s crawl fee. “Learn that as a substitute and learn to successfully handle Googlebot’s crawl fee,” he added.
Gary additionally posted on LinkedIn saying, “Friday rumble… ramble? A type of. Anyway: the 403 and 404 standing codes won’t show you how to shortly cut back crawl fee. If something, they may have the other impact. Now we have documentation about how one can cut back crawl fee and unsurprisingly 403/404 just isn’t in them.”
There are extra particulars within the weblog submit.
Discussion board dialogue at LinkedIn.
