GPTBot vs OAI-SearchBot: Block Training, Keep Your Citations

The promise: opt out of training without going invisible

For a long time the choice felt binary. Either you let OpenAI's crawler in and accepted your content might train a model that answers your buyers without crediting you, or you blocked it and hoped you were not cutting off something you needed. Plenty of businesses chose the block, reasonably, and only later discovered what it cost them.

The good news is that the binary is gone. OpenAI now runs separate crawlers for separate jobs, and the one that trains its models is not the one that decides whether you show up in ChatGPT search. Once you understand which bot does what, the robots.txt writes itself, and you get to keep the upside while shedding the part you did not want.

That distinction is worth internalising, because your standing in ChatGPT now hinges on it more than on almost any single piece of content you publish. ChatGPT has become a first stop for buyers weighing up their options, and whether you surface there is decided by a crawler you control rather than by luck. Getting this one file right is among the highest-leverage technical moves available, precisely because so much now rides on a single bot being allowed through.

Meet OpenAI's crawlers

Start by learning the cast, because the whole strategy depends on telling them apart. As Search Engine Journal reported when OpenAI's crawler documentation expanded to list OAI-AdsBot, OpenAI now names four distinct bots, each with its own purpose.

GPTBot collects training data for OpenAI's foundation models. OAI-SearchBot surfaces your content in ChatGPT search, which is how you get cited in answers. ChatGPT-User fetches a page when a person asks ChatGPT to look at it directly, which makes it user-triggered rather than an automatic crawler. And OAI-AdsBot, the newest, checks pages submitted as ChatGPT ads, with its data explicitly not used for training.

For this decision, two of those bots matter most: GPTBot and OAI-SearchBot. They pull in opposite directions, and conflating them is exactly the error that costs businesses their visibility. Understanding how ChatGPT selects and cites its sources makes clear why the search crawler is the one you cannot afford to lose.

What GPTBot does, and why you might block it

GPTBot exists to feed the training pipeline. Large language models are built by learning patterns from enormous volumes of text, much of it scraped from the open web, a practice the Wikipedia entry on large language models describes as the web-as-corpus approach that now underpins these systems. GPTBot is one of the pipes carrying your content into that process.

The scale is not small, and it is growing fast. Cloudflare's analysis of who is crawling the web in 2025 found that GPTBot's requests surged by around 305% in a single year and its share of AI crawling jumped from roughly 5% to 30%, making it the dominant training crawler on the web. That is a lot of fetching for something that, unlike a search crawler, does not send visitors back to you.

The asymmetry is the whole argument. A search crawler that indexes you at least offers something in return, in the form of citations and the clicks they bring, but a training crawler is pure outflow: it reads your content, folds it into a model, and that model may then answer your buyers' questions without ever pointing them back to you. Seen in that light, blocking the training crawler is not hostile to AI, it is simply declining a transaction that only ever runs one way.

So the case for blocking GPTBot is straightforward. If you would rather your content not be used to train models, or you simply see no return in letting a training crawler take it, disallowing GPTBot is a reasonable, low cost decision, because it does not touch your presence in ChatGPT search. For businesses whose written work is a genuine asset, that protection is worth having, and it pairs naturally with the content thinking behind generative engine optimisation.

What OAI-SearchBot does, and why you must keep it

OAI-SearchBot is the opposite kind of bot. It is not taking your content to train on; it is indexing your pages so ChatGPT can find, cite and link to you when a user asks a relevant question. This is the crawler that earns you visibility, and blocking it is the expensive mistake to avoid.

The separation is deliberate on OpenAI's part. When Search Engine Land covered the launch of SearchGPT, it noted that the product gives answers with clear sources and links, and crucially that publishers can manage their appearance in search separately from AI training, so a site can still show up in results even if it has opted out of training. That single design choice is what makes the whole block-training-keep-search strategy possible.

For most businesses, being cited in ChatGPT is worth far more than the marginal protection of blocking its search crawler. If your goal is to be discovered and chosen, OAI-SearchBot is your friend, and the way to be found is to be genuinely useful and easy to cite, which is what a serious answer engine optimisation effort delivers. Locking out the search bot to stop training is like boarding up your shopfront to keep out a survey taker.

The exact robots.txt: block GPTBot, allow OAI-SearchBot

Here is the practical heart of it. Because the bots are named separately, you can address them separately in robots.txt, blocking the training crawler while explicitly welcoming the search crawler:

# Block OpenAI's training crawler

User-agent: GPTBot
‍
Disallow: /

# Allow OpenAI's search crawler so you stay visible in ChatGPT

User-agent: OAI-SearchBot

Allow: /

‍

That is the whole move. The first block signals your content should not be used for training; the second keeps you eligible to appear and be cited in ChatGPT search. You can extend the same logic to other providers by naming their training and search bots the same way, giving you a single file that opts out of training across the board while preserving AI search visibility everywhere it matters.

The same pattern applies bot for bot across the other assistants. If you want to stay visible in Perplexity, for instance, you allow its search crawler while remaining free to block any training crawler it runs, and the logic repeats for every provider that has separated the two. One well organised file can therefore protect your content from training everywhere while keeping every door to AI search wide open.

If you want help generating and validating rules for the full set of AI crawlers, Search Engine Land maintains a useful rundown of the tools and software for managing AI crawlers, from robots.txt generators to enterprise controls. Getting this file right is foundational technical work, and it sits alongside everything else in a solid AI SEO setup.

The caveats that matter

Three things stop this from being a magic switch, and honesty about them is what separates a real strategy from a copied snippet. First, robots.txt is advisory, not enforcement. Well behaved crawlers respect it, but it is a request, not a lock, and some bots ignore it entirely.

That is why Cloudflare built the ability to block AI bots at the network level with a single click, using behavioural detection that recognises a scraper even when it lies about its user agent. If your goal is genuine protection rather than a polite signal, you will need controls at that level, not just a line of text.

Second, blocking GPTBot today does nothing about content a model already trained on. The opt out is forward looking, so the sooner you set it the better, but it is not a delete button for the past. Third, ChatGPT-User is user-triggered, so it behaves differently from the automatic crawlers and does not fetch pages on its own schedule the way GPTBot does. Keeping the whole picture straight is exactly what a periodic audit of your generative AI visibility is for.

Who should block GPTBot, and who should not

The right call still depends on what you are optimising for. If your content is a genuine competitive asset, a publisher, a research firm, a business whose written expertise is the product, blocking GPTBot is a sensible protective move that costs you nothing in ChatGPT search. You give up letting a training crawler take your work, and you lose no visibility in return.

If you are a service business, local operator or brand whose main goal is simply to be found and chosen, the training question matters less, and some owners happily allow GPTBot too, reasoning that broader model familiarity with their brand is no bad thing. Either way, the non negotiable is keeping OAI-SearchBot allowed, because that is the crawler tied directly to whether you appear in ChatGPT. When your content is well structured, the search crawler can lift a clean answer from it, which is where structured data for answer engine optimisation does real work.

There is no universally correct answer on GPTBot, only the right answer for your situation. What is universal is the discipline of deciding on purpose rather than by accident, and never letting a blanket block take out your search visibility as collateral damage. Framing content to be lifted cleanly is the heart of optimising content for generative AI search.

How to verify it worked

Changes to robots.txt fail silently, so test rather than assume. Confirm your file is serving the rules you intended, check that GPTBot is disallowed and OAI-SearchBot is allowed exactly as written, and watch your server logs over the following weeks to see which OpenAI bots are actually hitting your site and being turned away.

Server logs are the ground truth here, because they show you what the crawlers did rather than what you hoped they would do. If GPTBot keeps appearing after you disallowed it, that tells you either your file is not being read the way you intended or the crawler is one of the ones that ignores the rule, and both are worth catching early rather than discovering months down the track.

Then check the outcome that matters: whether you still appear when you ask ChatGPT the questions your buyers would. If your citations hold steady after you block GPTBot, the strategy is working as intended. If they fade, something is blocking the search crawler and needs fixing fast.

If a past blanket block has already cost you ground, the recovery path is the one we map in winning back visibility lost to AI search, and our case study on rebuilding AI visibility shows how untangling exactly this kind of crawler confusion turned an invisible site into a cited one.

If you would rather have it checked for you, a structured AI visibility audit will confirm your crawler setup and where your citations stand. The principle behind our approach to AI search is simple: protect what you want to protect, but never at the price of being found.