Does Blocking AI in Robots.txt Hurt Your AI Performance?

By now, there's zero question about whether or not AI is going to affect marketing and SEO. The real question is, how do you navigate this new and changing environment?

Some people reject it entirely. They refuse to use it, they refuse to engage with it, they refuse to hire anyone who works with it.

Others have embraced it, to the point of getting rid of anyone they can replace with AI systems instead. Coders, writers, artists, managers, assistants; it seems like no role is truly off-limits.

Most people just want to keep their business going, using the tools that work and avoiding the ones that don't. This is where AI presents a significant challenge.

The Problem with AI (No, Not That One)

There are a lot of problems with AI that you've definitely heard of. There's the copyright issues with training data. There's the environmental issues with data centers. There's the impact on current economics, and the future of many roles where the pipeline from apprentice to master is severed.

None of those are the problem I want to talk about today, even if they're all extensive and interesting discussions to have.

No, the problem I'm talking about today is one of technical capacity.

Looking Back to Look Forward

How many of you were around when the internet was fairly young? Say, the mid-90s and early 2000s? 

I was, and one of the big concerns back then was bandwidth. You would pay for a web host, and part of the web hosting package you got was the amount of bandwidth available to you. Lower-tier plans would have lower caps on throughput.

Every visitor to a site consumes some amount of bandwidth, as they download assets and data from your web host server to render the website on their local machine. Tools like caching and minification help cut that down, but there's always going to be some data transfer. 

Back in the day, you had to carefully balance your website growth. If you went viral, there was a pretty good chance one of two things would happen.

• You would see your site go down, since you hit the bandwidth cap and your web host pulled it, until you either upgrade your plan or the monthly limit resets.
• You would have a site that works fine, but be hit by a surprise bill in the thousands or tens of thousands of dollars for the bandwidth usage.

I personally know of more than one website that had to struggle with this, several of which never recovered.

Eventually, technology adapted. Better internet infrastructure expanded across the globe, server capacity outstripped consumption, web developers got better at keeping things small, and web hosts greatly raised or outright removed bandwidth caps.

Some of you see where I'm going with all of this.

The Denial of Service

I say that the problem was more or less resolved, but we did eventually reach a new equilibrium. After all, there are still limits, some of which are imposed by the laws of physics.

Today, servers can still be overwhelmed. It's just a lot harder, and takes either a lot of people or a lot of "people" to do. You see it when huge new product releases happen, when social media struggles to keep up or when platforms like storefronts go down. LEGO's website buckles under highly sought-after new releases, Steam falters during major sales with huge game releases, and so on.

And, of course, there's always the fact that malicious actors can use compromised machines all around the world, botnets numbering in the hundreds of thousands or even millions of devices, leveraging lightweight scripts to force connections to a server that can't possibly handle the capacity.

It's a Distributed Denial of Service Attack, and it's a common threat today. DDoS attacks can be spurred on by vengeful competitors, slighted customers, or just malicious attackers hoping to use the collapse of the server to break security, infiltrate, and steal data.

There are a lot of reasons why these happen. It's incredibly easy to spin up a lot of micro-devices to force millions of connection attempts. Web hosts try to operate on slim margins with lightweight hardware for all but their most dedicated enterprise customers. Distributed infrastructure across redundant data centers can't always funnel traffic equally and create chokepoints that get clogged up.

AIDoS: Malicious and Not

Enter AI.

AI has contributed to this problem for millions of website owners in two different ways.

One (and not the one I'm focusing on today) is that AI coding tools have made it much easier for malicious attackers to spin up attacks. 

• AI makes it easier for unsophisticated attackers to create more sophisticated attacks.
• AI makes it easier to automate attacks.
• AI makes it easier to identify exploitable vulnerabilities.

Malware has absolutely exploded with the use of AI, and AI can't really be an effective security tool because it presents many unique and novel vectors for exploitation as well. It's almost better suited as an attacker than a defender, at least in its current state.

Obviously, if AI is being used maliciously, that's a problem. But the real problem is AI not being used maliciously. Or, at least, not intentionally maliciously.

The internet is full of stories like these.

• AI Web Crawlers are Destroying Websites in Their Never-Ending Hunger for Any and All Content
• When AI Scrapers Attack
• "This Was Essentially a Two-Week Long DDoS Attack"
• Meta's AI Crawler Scraped My Site 7.9 Million Times in 30 Days

So, here's the thing. For a system, whether it's Google's search index or Anthropic's AI searching or ChatGPT's training data, the input needs to be there. None of these systems can output something that's not in them in the first place, which means they need to go out and get that data somehow.

All of the tech companies have decided that the easiest way to get that data is to go grab it themselves. They use bots and crawlers to harvest data to then feed back to humans.

Or to bots. Nearly 60% of internet traffic is now bots of various types, from scrapers to indexers to spammers to attackers and more.

Google has been doing this since 1996, so why is it a problem now?

It comes down to restraint.

Google has a lot of built-in controls and limiters on their scraping. They use the most lightweight scrapers they can. They harvest as minimal data as they can. They check sitemaps to look for updates, and don't repeatedly scrape sites they have no reason to believe have changed since the last time they looked.

More importantly, they obey directives. The robots.txt file, formally known as the Robots Exclusion Protocol, is a standardized method introduced in 1994 to control rules-obeying bots. If you use robots.txt to tell Google to ignore a page, they will ignore it.

Malicious bots, of course, ignore this. There's no built-in way for code to be controlled by other code if the first code isn't designed to. It's part of why things like DDoS attacks are so difficult to handle.

The problem right now is that the AI companies (yes, all of them) simply don't care. They want to hoover up as much data as they can, and they haven't put in any of the thought Google has. They'll crawl the same site thousands of times in the span of minutes.

Why? Well, that's the mystery, isn't it? I don't have an inside viewpoint, but I have to assume that the combination of the value of the data they get for their own ends, and the complete lack of any accountability or punishment, means there's no reason for them not to.

As a site owner today, you're continually at risk of these scrapers swooping in and destroying your site to extract the essential oil of your content.

Why Not Block AI and Be Done with It?

If you don't care about your content being archived and accessible in ChatGPT or Claude or Perplexity or whatever other system, why not just block the AI crawlers?

There are a few reasons.

One is that it's not so easy. Companies like Google publish a list of their bots and identifiers. Companies like OpenAI and Perplexity leverage hundreds or thousands of IP addresses simultaneously. You're playing whack-a-mole with a potentially endless list of numbers.

Another is that there is, potentially, value in the AIs having the data. Google's AI Overviews can still drive impressions and traffic. Some AI-focused traffic, such as from ChatGPT, has a high conversion rate. Blocking the AI scraping means no longer being visible in those results, and losing that value.

There's also one big detail I haven't yet mentioned: the difference between scraping for results and scraping for training.

Some of the AI companies make this distinction, while others don't. 

The idea is that scraping pages for training means your content is just being shoved into the mass library of training data used to refine LLM function. There's no attribution, and it doesn't mean your site shows up in their results at all; just that the grammar, syntax, word choice, and framing you've worked on are fed into the machine.

Scraping for results is more like traditional SEO, where the AI companies send their LLM out to find attributable data and provide results. When you ask Claude or ChatGPT with its live results for sites and sources, you get live results, because they either know or can find the information.

You would want to block the trainers, but not the indexers. If you can. Are they identified? Not always.

Conflicting Goals

There are tensions on the rise right now in this discussion.

On one hand, an increasing number of company owners, CEOs, and other executives are looking at AI as the wave of the future. They want to go all-in, get as much value by riding the wave as possible. AI sends people to your site or recommends your products, so do everything you can for AIO. 

On the other hand, we've seen firsthand the damage unchecked AI scrapers can do. Excessive billing, sites taken down under the strain, unique content stolen for LLM usage, the rise in zero-click searches because users have their questions answered by the AI and don't need a website for it.

It begs the question: if a user's question is answered, but the site that provided the answer dies off because it lost all traffic to the AI, is it still good? A user is satisfied, but a business has died.

There are a lot of different possible solutions to this issue, and a lot of them are taking aim at the AI companies. Should content scraping be opt-in? It wasn't for Google, but Google is much more restrained. Should AI scrapers pay to access? Should they be legally required to obey robots.txt, or face penalties?

Cloudflare recently made a move in this direction. They've announced that they're going to be blocking AI scrapers by default, and introducing a way to force AI companies to pay for access. If a business is going to lose customers to the AI, at least the AI can pay for it.

Will this work? That remains to be seen. Cloudflare's pay-per-crawl is still in private beta, and it's entirely possible the AI companies will simply not pay. Then again, they're pretty well used to spending exorbitant amounts of money as it is.

Practicalities for the Modern Webmaster

As website owners, we have a dilemma.

Do we block AI traffic, and possibly take the hit with lowered visibility, even if a lot of that visibility takes the form of zero-click searches?

Or do we allow the AI, and risk all our unique value being scraped away, our server bills potentially spiking, and our sites potentially being taken down under the baleful eye of the bots?

A lot of this discussion hinges on whether or not any value is actually lost by blocking AI.

Here's the thing: the existence of AI Overviews and similar AI search results has already cost you traffic and revenue. The rise in zero-click searches, the drop in overall traffic, the loss of direct revenue from display advertising or impulse sales- that's all very real. Some estimates place this at a 34-46% reduction, before you take any kind of action at all.

Does blocking the AI, when the AI is responsible primarily for zero-click searches, hurt your traffic any more than keeping it as-is?

Maybe. Not showing up in AI at all can be a hit against you from the small amount of referral traffic you'd actually get. Not being visible to AI scrapers can be a second-hand algorithmic loss of trust.

One study found websites that blocked AI crawlers saw a traffic drop by 23%. 14% of the decline was, at least ostensibly, human traffic. That said, not all human traffic is definitely human (especially when bots are trying to evade blocks), and not all human traffic is valuable.

That same study also found that mid-sized and even smaller publishers actually had an increase in traffic from blocking AI bots. 

What Should You Do?

Frankly, I don't know. I can't predict the future.

If Cloudflare's efforts to monetize AI scraping work out and hit the wider market, I would say go for it. If you're going to lose revenue from zero-click searches, you might as well get the bag somewhere else.

Otherwise, I say watch your traffic and make a decision based on what you experience.

If AI scrapers are taking up huge amounts of resources, and especially if you're getting higher bills because of it, block them.

If you haven't been hit in that kind of way, then you probably don't need to.

You can also simply block the bots and see what happens. If your traffic (and more importantly, your revenue) drops because of it, then undo the block. If not, or if your metrics improve, then blocking can be the right move.

If you, the person reading this right now, have already done this experiment, let me know! I'm really curious; how big is your site, what niche is it in, and what did blocking the AIs do for you, positive or negative? Leave me a comment so we can build a body of evidence.