The SEO Bots That ~140 Million Websites Block the Most

Ever wonder which SEO bots are the most blocked? This can impact the quality of the data the tools provide.

Blocking these bots will mostly impact the link index of the tools. They won’t be able to crawl the pages, so they can’t check where those pages are linking. It doesn’t matter for traffic estimates, keyword rankings, top pages, etc. Those are built from different data sources.

For Ahrefs, it would also impact the internal links we show and the page history feature that shows changes to your pages over time, which you might need at some point. Ahrefsbot also powers the index for our search engine, Yep.com, so blocking Ahrefsbot means you wouldn’t show in Yep’s search results.

We looked at ~140 million websites to see how often SEO bots were blocked. I want to give a huge thanks to our data scientist Xibeijia Guan for pulling this data.