Block the AI crawlers and you vanish from the answers they feed
Millions of sites spent 2025 slamming the door on AI bots, and the instinct is understandable — why let a model read your work to answer for you? But "block the AI crawlers" hides a distinction that decides whether you are protecting your content or quietly deleting yourself from the answers people now ask first.
the short answer
Blocking AI crawlers protects your content from training, but a broad block also removes you from the answers those engines cite. The bots do different jobs: GPTBot trains, while OAI-SearchBot and PerplexityBot fetch you to cite you. The considered move is selective — disallow the training crawlers, allow the retrieval ones — because a crawler that cannot read your page cannot cite it. By December 2025, about 5.6 million sites block GPTBot; the common mistake is blocking far more than they meant to.
key takeaways
- Blocking AI bots protects your content from training — but a broad block also erases you from the answers those engines generate.
- It is not one bot: OpenAI runs three (GPTBot for training, OAI-SearchBot for search, ChatGPT-User for retrieval); blocking one does not block the others.
- The considered move is selective: disallow GPTBot and other training crawlers, but allow the retrieval bots — OAI-SearchBot, PerplexityBot — that cite you.
- By December 2025, about 5.6 million sites block GPTBot, up roughly 70% since July; the common error is over-blocking by accident.
- A crawler that cannot read your page cannot cite it: for GEO, retrieval access is the entry condition, not an optional extra.
one company, several bots — block the wrong one and you lose the citation
In words, so the table does not carry it alone: GPTBot trains OpenAI's models, so blocking it keeps your work out of training — a defensible stance with little search cost on its own. OAI-SearchBot indexes pages for ChatGPT's search, so blocking it makes you invisible there. PerplexityBot fetches pages live to build cited answers, so blocking it removes you from Perplexity's responses. Same instinct — "block the AI bot" — but three different outcomes, and only the first is the one most brands actually want.
Why so many sites blocked, and what they were protecting
The blocking wave is real and it accelerated through 2025. By December, roughly 5.6 million sites disallowed GPTBot in robots.txt, up about 70% from July, making it the most-blocked crawler on the open web; the New York Times, CNN and more than thirty of the top hundred sites are among them. One monitoring firm reported a 336% rise in sites blocking AI crawlers over the prior year. The grievance underneath is coherent: producing good content costs time and money, and a model that reads it to answer in your place breaks the old bargain where being crawled at least bought you a visit.
Infrastructure moved with the mood. Cloudflare introduced a pay-per-crawl model in July 2025, letting publishers charge for bot access instead of blocking outright, and began defaulting new sites to block AI crawlers. So the impulse to wall off your content is neither fringe nor irrational — it is a direct response to a real shift in who captures the value of a page. The problem is not the instinct. The problem is that the bluntest expression of it — block everything that smells like AI — also closes the channel through which AI now sends people to you.
The grievance has numbers behind it. Through mid-2025, Google referrals to premium publishers fell about 10% year over year, and non-news brands saw declines closer to 14%, as AI answers resolved more queries before any click. For a publisher watching that line drop, walling off the bots reading their work can feel like the only lever left. The catch is that the lever cuts both ways: the same engines draining the referrals are the ones that, allowed in, can still send qualified readers — but only to sources they are permitted to reach.
The decision, in three parts
Getting this right is less about a stance and more about precision. Open each layer to see the distinction that matters, the trap that catches most teams, and why the broad version of the block costs you the thing you were trying to grow.
01 Training, retrieval, indexing — three different jobs
Not all AI crawlers want the same thing, and conflating them is where most mistakes start. A training bot collects content to teach a model; blocking it keeps your work out of the next training run. A retrieval bot fetches your page live, the moment a user asks, to ground or cite an answer; blocking it removes you from that answer. An indexing bot pre-builds the corpus an engine searches against; blocking it keeps you out of what that engine can find. Three jobs, three consequences. Decide what you actually object to — being trained on, or being uncited — because the answer points at different bots, and blocking the wrong one solves a problem you do not have while creating one you do.
02 The three-bot problem
A single AI company usually runs several bots, one per job. OpenAI runs three: GPTBot for training, OAI-SearchBot for search indexing, and ChatGPT-User for user-initiated retrieval. Anthropic and Perplexity follow the same pattern. The practical trap is that blocking one does not block the others, and a robots.txt rule written for "the OpenAI bot" often misses two of the three — or, when written too broadly, sweeps up the ones you wanted to keep. In December 2025 OpenAI revised its crawler documentation, including a change that ChatGPT-User no longer treats robots.txt as binding for user-initiated actions. The lesson is not the specific edit; it is that these policies move, and a set-and-forget block from a year ago may no longer do what you think.
03 Why a broad block backfires
The reciprocity that once tied crawling to discovery has frayed: bots read your page, then serve the answer themselves. That is the real grievance behind the wave of blocks — by December 2025 roughly 5.6 million sites disallow GPTBot, up about 70% since July, and the New York Times, CNN and more than thirty of the top hundred sites are among them. But blocking the retrieval bots to protest the training bots throws out the visibility with the grievance. A crawler that cannot reach your page cannot cite it, so a brand that blocks broadly removes itself from a discovery channel that is still growing. The defensible version of the block is narrow and deliberate; the version that hurts is the broad one applied in frustration.
How to decide, bot by bot
Replace the all-or-nothing question with a per-bot one. If your objection is to your content training someone else's model for free, disallow the training crawlers — GPTBot, Google-Extended, and their equivalents — which carries little cost to search or AI-citation visibility because those run on separate bots. If your goal is to be found and quoted, leave the retrieval and search bots allowed: OAI-SearchBot for ChatGPT, PerplexityBot for Perplexity, and the indexing bots that let an engine know your page exists. Most brands want exactly this combination — kept out of training, kept inside the answers — and it is a few precise lines in robots.txt, not a wall.
Then audit what you already have, because the dangerous blocks are usually old and accidental. A rule written a year ago against "the OpenAI bot" may now miss two of the three, or a broad disallow may be quietly keeping out the very crawlers that would cite you. Check which bots you actually allow, confirm the retrieval ones are among them, and watch your server logs to see who is really reaching your pages. The distinction between a principled, narrow block and a self-defeating broad one is the difference between protecting your work and disappearing from the surfaces where people increasingly start.
AI crawlers: quick answers
Should I block GPTBot?
It depends on what you are trying to protect, and GPTBot is the one bot where blocking is most defensible. GPTBot gathers training data for OpenAI’s models; blocking it keeps your content out of future training runs without, on its own, removing you from ChatGPT’s live answers — because those answers are served by a different bot, OAI-SearchBot. So if your concern is that your work is used to train a model for free, disallowing GPTBot is a coherent choice. The mistake is assuming GPTBot is the only OpenAI crawler and that blocking it is harmless to visibility. It is harmless to search visibility only if you leave the retrieval and search bots allowed; block all of them with a broad rule and you have quietly opted out of being cited.
Will blocking AI crawlers hurt my Google ranking?
Blocking the AI-specific bots — GPTBot, Google-Extended, PerplexityBot — does not directly change classic Google ranking, because Googlebot is a separate crawler and Google-Extended governs only training use, not Search indexing. You can block training bots and still rank normally in the blue links. The real risk is broader and sloppier: blocking whole crawl paths, headers, or scripts to keep bots out often hides the same signals the search and AI systems need to read your pages at all. The damage usually comes not from a precise AI-bot disallow but from a blunt one that also blocks the crawlers feeding the answers — and increasingly, AI surfaces are a discovery channel you do not want to disappear from even as classic ranking holds.
What is the difference between training bots and retrieval bots?
Training bots collect content to teach a model; retrieval bots fetch pages live to ground or cite an answer the moment a user asks. Blocking a training bot like GPTBot stops your content being used to build the model. Blocking a retrieval or search bot — OAI-SearchBot for ChatGPT, PerplexityBot for Perplexity — stops you appearing when that engine pulls live sources to answer a question with citations. The two have opposite implications for a brand that wants to be found: keeping training bots out can be a principled stance with little visibility cost, while keeping retrieval bots out directly removes you from the cited answers. The single most common configuration error is treating them as one thing and blocking both with a broad rule.
How do I let AI engines cite me without giving away training data?
Be selective per bot rather than all-or-nothing. In robots.txt you can disallow the training crawlers — GPTBot, Google-Extended, and the equivalents — while allowing the retrieval and search bots that put you into cited answers, such as OAI-SearchBot and PerplexityBot. That combination keeps your content out of model training while leaving you eligible to be quoted when someone asks a question your page answers. Two caveats keep this honest: not every bot obeys robots.txt — a measurable share of requests ignore it — and any crawler can spoof a user-agent, so directives are a strong signal, not a guarantee. But for the goal most brands actually have — be cited, do not be mined for free training — selective allow-listing is the move, not a blanket block.
A note on sources and certainty
The figures here — roughly 5.6 million sites blocking GPTBot by December 2025, up about 70% since July, and OpenAI's crawler-documentation revision that month — come from contemporaneous reporting and from the bot operators' own published policies. Two honest caveats apply. First, robots.txt is a request, not a fence: a measurable share of crawler requests ignore it, and any bot can spoof a user-agent, so directives are a strong signal rather than a guarantee. Second, these policies change often — the December documentation update is one of several over GPTBot's life — so any specific rule should be reviewed, not set once. What is stable is the underlying distinction between training and retrieval, and the consequence that follows from it. The AC Group has spent 27 years on the unglamorous side of discovery — who can reach a page and why — and this is that work applied to the door most sites are slamming without checking which bots are on the other side.