Does blocking Internet Archive bots in robots.txt actually stop AI companies from scraping content?

No. AI crawlers from OpenAI, Google, and others operate their own bots independently of the Internet Archive. Blocking archive.org_bot and ia_archiver only prevents the nonprofit Wayback Machine from preserving your pages. AI companies that ignore robots.txt directives will continue scraping regardless.

Why are publishers blocking the Internet Archive instead of AI crawlers directly?

Publishers discovered that AI training datasets like Google's C4 corpus included content pulled from Wayback Machine snapshots. The Archive ranked 187th out of 15 million domains in that dataset. Publishers view the Archive as an indirect pipeline to AI training, even though the Archive itself isn't an AI company.

What happens to archived web pages when a site blocks the Internet Archive?

The Wayback Machine stops capturing new snapshots of that site. Existing snapshots may also become inaccessible depending on the blocking method. Nieman Lab reported an 87 percent drop in page captures among news publications between May and October 2025 after widespread blocking began.

Which robots.txt directives actually block AI training crawlers?

Use User-agent: GPTBot with Disallow: / to block OpenAI's training crawler. CCBot (Common Crawl) and Bytespider (ByteDance) are the main sources for AI training datasets. GPTBot, ClaudeBot, and Google-Extended are citation-capable bots for live search, blocking them removes your site from AI Overviews and ChatGPT search results, which is a different tradeoff than blocking training.

Is it worth blocking AI crawlers in robots.txt if they ignore it?

Blocking well-behaved crawlers like GPTBot is effective, OpenAI, Anthropic, and Google state they honor robots.txt for their known bots. The problem is unknown or bad-faith scrapers that don't identify themselves and ignore directives. For those, robots.txt is ineffective. Rate limiting and IP-based blocking at the server or CDN level are more reliable defenses against aggressive scrapers.

Blocking the Internet Archive Won't Stop AI Crawlers

Blocking the Internet Archive Won't Stop AI Bots

Publishers are blocking the Internet Archive to fight AI scraping, but robots.txt won't stop training bots. It will only destroy decades of web history.

March 21, 2026 · Marcus Bennett · 5 min read

Updated: March 30, 2026

Quick Take

Blocking the Internet Archive won't stop AI training bots, most already have your content or ignore robots.txt. You're more likely to destroy 30 years of web history for researchers and journalists than to protect your work from AI scraping.

Quick take: Over 200 news publishers now block Internet Archive crawlers, causing an 87% drop in page captures (Nieman Lab, 2025). This doesn't stop AI training bots at all. It only destroys the web's historical record. Block GPTBot and ClaudeBot directly instead.

I tried to pull up a cached version of a news article last week. Gone. The Wayback Machine returned nothing. That page had been archived since 2019, and now it's just a blank entry because the publisher started blocking Internet Archive crawlers in late 2025.

This is happening everywhere. And it's a disaster for anyone who builds on the open web.

The Blocking Wave

The New York Times, The Guardian, Reddit, and the entire Gannett network, over 200 news sites in total, now block Internet Archive bots through robots.txt. The Times stated openly that they're "hard blocking" the Archive's crawlers because the Wayback Machine "provides unfettered access to Times content, including by AI companies, without authorization."

Here's what a typical block looks like in robots.txt:

User-agent: archive.org_bot
Disallow: /

User-agent: ia_archiver
Disallow: /

Two lines. That's all it takes to erase your site from the historical record.

Nieman Lab found that page captures among news publishers dropped 87 percent between May and October 2025. Think about that number. We're talking about the primary sources future historians and researchers will need to understand this era. And we're deleting them in real time.

Why This Won't Actually Stop AI Training

Here's the part that frustrates me the most. Blocking the Internet Archive doesn't block AI crawlers. Not even close.

OpenAI runs GPTBot. Google runs Google-Extended. Anthropic runs ClaudeBot. Meta runs its own scrapers. These bots operate completely independently of the Internet Archive. If you want to block AI training, you need to block those specific user agents. Blocking archive.org_bot does absolutely nothing to stop them.

Yes, AI companies have used Wayback Machine data in training sets. Google's C4 dataset included Archive content, the domain ranked 187th out of 15 million sources. That's a real concern. But the fix isn't to burn down the library because someone photocopied a book there.

The Internet Archive itself doesn't build AI models. It's a nonprofit digital library. Treating it like an adversary is like banning public libraries because someone might use the photocopier wrong. Does that logic hold up under any scrutiny?

robots.txt Was Never Built for This

The robots.txt standard dates back to 1994. It's a gentleman's agreement. There's no enforcement mechanism, no authentication layer, no legal penalty for ignoring it. Well-behaved crawlers respect it. Bad actors don't.

The Internet Archive has always honored robots.txt. That's the irony here. The one crawler that actually follows the rules is the one getting blocked. Meanwhile, sketchy AI scraping operations spin up new user agent strings every week and crawl whatever they want.

I've watched this pattern play out in my own projects. I ran a side project that tracked crawl logs across several domains. The bots that identified themselves honestly, Googlebot, Bingbot, archive.org_bot, those you could manage. The anonymous scrapers with rotating IPs and fake user agents? They ignored robots.txt entirely. Every developer who's looked at server access logs knows this story.

The Real Cost to Developers

Link rot is already terrible. A 2021 Harvard study found that 25 percent of links in New York Times articles pointed to dead pages. The Wayback Machine was the safety net. When a URL died, you could find the archived version.

Now we're removing that safety net for some of the web's most important content.

If you've ever used the Wayback Machine to check what an API endpoint returned three years ago, or to find documentation for a deprecated library, or to verify what a page actually said during a dispute, you understand the value. I used it just last month to recover the original docs for a payment gateway SDK that the vendor had completely rewritten without versioning. That kind of recovery becomes impossible when publishers opt out of archiving.

And here's my controversial take: publishers blocking the Archive aren't really fighting AI companies. They're fighting for advantage in licensing deals. Reddit didn't block the Archive because it opposes AI training. Reddit signed multimillion-dollar deals with Google and OpenAI. Gannett signed a licensing agreement with Perplexity in July 2025. They want to get paid. The Archive blocking is a negotiating tactic, not a principled stand.

What Should Happen Instead

The EFF published a detailed analysis in March 2026 arguing that blocking the Archive is the wrong response. I agree. Here's what would actually work:

Block AI crawlers directly. Add GPTBot, Google-Extended, ClaudeBot, and other AI-specific user agents to robots.txt. That's targeted and doesn't sacrifice preservation.
Push for legal frameworks. The courts are already handling AI training copyright cases. Let that process play out instead of scorched-earth blocking.
Support the Archive's opt-out tools. The Internet Archive has exclusion mechanisms that don't require nuking your entire archival history.

The web is fragile. Pages disappear constantly. Servers go down. Companies fold. Domain registrations lapse. The Wayback Machine has captured over 866 billion pages since 1996. It's the closest thing we have to a permanent record of the internet.

Destroying that record to fight a battle it can't win isn't strategy. It's collateral damage. And the people who'll pay the price aren't AI companies, they're researchers, journalists, developers, and anyone who needs to know what the web actually said.

We can fight AI scraping without torching the library. We just have to aim at the right target. For more on how the AI industry is shifting for developers, see our coverage of Mistral's Forge platform for enterprise AI and what it means for the three pillars of JavaScript bloat when AI-generated code enters production codebases.

Strava aircraft carrier leak - another case where the gap between what data an app collects and what users expect reveals real security and privacy failures
Mistral Releases Forge for Enterprise AI Models - the enterprise AI model market is precisely the reason publishers want control over their training data
AI coding tools 2026 - understanding how AI tools are trained puts the Archive blocking debate in context

Share: Post on X Share on LinkedIn

Frequently Asked Questions

Does blocking Internet Archive bots in robots.txt actually stop AI companies from scraping content?: No. AI crawlers from OpenAI, Google, and others operate their own bots independently of the Internet Archive. Blocking archive.org_bot and ia_archiver only prevents the nonprofit Wayback Machine from preserving your pages. AI companies that ignore robots.txt directives will continue scraping regardless.
Why are publishers blocking the Internet Archive instead of AI crawlers directly?: Publishers discovered that AI training datasets like Google's C4 corpus included content pulled from Wayback Machine snapshots. The Archive ranked 187th out of 15 million domains in that dataset. Publishers view the Archive as an indirect pipeline to AI training, even though the Archive itself isn't an AI company.
What happens to archived web pages when a site blocks the Internet Archive?: The Wayback Machine stops capturing new snapshots of that site. Existing snapshots may also become inaccessible depending on the blocking method. Nieman Lab reported an 87 percent drop in page captures among news publications between May and October 2025 after widespread blocking began.
Which robots.txt directives actually block AI training crawlers?: Use User-agent: GPTBot with Disallow: / to block OpenAI's training crawler. CCBot (Common Crawl) and Bytespider (ByteDance) are the main sources for AI training datasets. GPTBot, ClaudeBot, and Google-Extended are citation-capable bots for live search, blocking them removes your site from AI Overviews and ChatGPT search results, which is a different tradeoff than blocking training.
Is it worth blocking AI crawlers in robots.txt if they ignore it?: Blocking well-behaved crawlers like GPTBot is effective, OpenAI, Anthropic, and Google state they honor robots.txt for their known bots. The problem is unknown or bad-faith scrapers that don't identify themselves and ignore directives. For those, robots.txt is ineffective. Rate limiting and IP-based blocking at the server or CDN level are more reliable defenses against aggressive scrapers.