The robots.txt Setup That Maximises AI Citations

What does the ideal robots.txt for AI citations look like?

The ideal file is short, deliberate and welcoming to the crawlers that generate citations. In practice it allows the AI search crawlers, keeps your important content and its resources crawlable, and hands crawlers a sitemap so they can find everything you publish.

Here is a clean starting point:

# Allow all crawlers to reach the site by default

User-agent: *

Allow: /

# Block only private or low-value areas

Disallow: /cart/

Disallow: /checkout/

Disallow: /wp-admin/


# Point every crawler to your sitemap

Sitemap: https://www.example.com/sitemap.xml

‍

That is the shape of it: open by default, restrictive only where it must be, and explicit about where your content lives. The robots.txt standard itself, formalised as the Robots Exclusion Protocol in RFC 9309, is deliberately simple, and simplicity is your friend here. Every extra rule is another chance to accidentally block something that earns you citations.

The mistake most sites make is inheriting a defensive file built for a different era. When your goal is AI visibility, the default posture flips from keeping crawlers out to inviting the right ones in, because a crawler that cannot reach you cannot cite you. This is the technical bedrock beneath any AI SEO programme.

Why can your robots.txt quietly kill your AI visibility?

Because the failure is silent. A robots.txt mistake does not throw an error or send you a warning; it just stops a crawler, and you discover the damage weeks later when your citations have faded and you cannot explain why. That silence is exactly what makes this file so dangerous when it is wrong.

The asymmetry with normal SEO problems is what catches people out. A broken page returns an error you can see, and a slow page shows up in your analytics. A robots.txt block shows you nothing at all, because from your side the site looks perfectly healthy. The only symptom is an absence, the citations that never appear, and absences are almost impossible to notice until you go looking for them.

The classic version is a blanket block added in a panic. A site owner reads that AI is scraping the web, adds a rule to disallow everything with AI in the name, and unknowingly locks out the very search crawlers that put them in ChatGPT and Perplexity answers. Understanding how ChatGPT selects and cites its sources makes the cost obvious: if the crawler never reaches the page, the page can never be the source.

There is a second, quieter version too. An old rule disallowing a folder, a stray wildcard, or a block on a resource directory can cut off content that would otherwise be cited, and because it was written long ago for reasons no one remembers, it never gets questioned. Auditing your generative AI search visibility regularly is how you surface these ghosts before they cost you.

Which AI crawlers should your robots.txt allow?

Allow the AI search crawlers, always, because they are the ones that index your pages to cite them. These are distinct from the training crawlers, and the difference decides your visibility. Search crawlers like OAI-SearchBot, PerplexityBot and the search-focused bots from other providers earn you citations; blocking them removes you from those answers entirely.

Training crawlers are the ones you can reasonably choose to block if content protection matters to you, and Cloudflare's work on controlling content use for AI training with managed robots.txt shows how many publishers now do exactly that, blocking training while keeping search open. The key is precision: block the training bot by name if you must, but never let that block spill onto the search crawler beside it.

This is where naming matters. Providers deliberately give their training and search crawlers different names precisely so you can treat them differently, and the whole strategy depends on you using those names exactly.

A rule aimed at a training bot that accidentally matches the search bot beside it hands back all the protection you gained and takes your citations with it, which is the opposite of what you set out to do.

When you write these rules, precision in the file matters as much as intent. Google's guide to how it interprets the robots.txt specification explains that the most specific rule wins and that a careless wildcard can catch far more than you meant.

Naming each crawler explicitly, and testing the result, is how you make sure your rules land where you aimed them.

Your visibility across ChatGPT and Perplexity hinges on those search crawlers being welcome.

The same holds for Gemini and every other assistant your buyers use, so the safe default is to allow all of them and block only the specific training bots you have a real reason to exclude.

How does the Sitemap directive boost AI discovery?

A single line pointing to your sitemap helps crawlers find every page you want cited, which is why it belongs in every AI-ready robots.txt. The directive is part of the open Sitemaps protocol, and it is refreshingly simple: add Sitemap: followed by the full URL, and it works regardless of where you place it in the file.

The payoff is discovery. Crawlers do not have infinite patience, and a sitemap hands them a clean map of your content instead of making them find it by following links. Google's guidance on building and submitting a sitemap adds the practical detail: keep it to canonical URLs you actually want surfaced, stay within the size limits, and submit it through Search Console as well as referencing it in robots.txt.

For AI citations specifically, this matters because a page that is never discovered is never a candidate to be cited. Pair the sitemap with content built to be lifted cleanly, the discipline behind optimising content for generative AI search, and you give crawlers both the map and a reason to quote what they find.

What should your robots.txt never block?

Never block the resources a crawler needs to read and render your pages. Disallowing your CSS or JavaScript directories used to feel tidy, but it can stop a crawler from seeing your content the way a browser does, and a page the crawler cannot render properly is a page it cannot confidently cite.

Think of it from the crawler's point of view. It is trying to understand your page the way a person would, so if the stylesheet that lays the page out or the script that loads the main content is off limits, it sees a broken, half-empty version and judges you on that.

Modern crawlers render pages much like a browser does, which means anything a browser needs to display the page properly is something the crawler needs too.

Equally, never block the content pages you actually want surfaced. It sounds obvious, yet audits routinely turn up a stray disallow on a blog folder, a category, or an entire section that someone restricted years ago and forgot.

Your money pages, your best articles and the structured content behind them all need to stay reachable, which is why structured data for answer engine optimisation only pays off when the pages carrying it are crawlable in the first place.

The rule of thumb is simple: block private, transactional or genuinely low-value paths, and leave everything you would be happy to be cited for wide open. When in doubt, allow.

The cost of an over-eager block is invisible and ongoing, while the cost of leaving a low-value page crawlable is close to nothing.

Do you need an llms.txt file too?

Maybe, eventually, but it is a complement rather than a fix. The llms.txt proposal suggests a markdown file at /llms.txt that gives models a curated, easy to parse guide to your most important content, solving the problem that full web pages are messy for a model to read. It is a genuinely interesting idea for AI discovery.

The honest caveat is adoption. The standard is still emerging rather than widely supported, so it is worth adding if you can do so cheaply, but it is not a substitute for getting your robots.txt and sitemap right.

Treat it as a low cost experiment layered on top of the fundamentals, not a shortcut around them. The work that reliably moves citations today is the generative engine optimisation groundwork, and llms.txt sits alongside that rather than replacing it.

How do you test your robots.txt is helping, not hurting?

Test it deliberately, because this file fails silently and assumptions are expensive. Start by confirming the file loads at your root, then check that each rule does what you intended, paying special attention to any AI crawler blocks and any wildcards that might be catching more than you meant.

A validator is the fastest first check. Paste your file in, point it at a specific URL and a specific crawler, and it tells you whether that crawler would be allowed or blocked, which turns a guessing game into a definite answer.

Do this for each AI search crawler you care about, and for a handful of your most important pages, before you trust the file in production.

Then watch the real world. Review your server logs to see which AI crawlers are actually reaching your site and which are being turned away, and cross-check that against whether you still appear when you ask the assistants your buyers' questions.

If your citations are healthy and the search crawlers are getting through, the file is doing its job. If not, the logs usually point straight at the offending rule.

Make this a habit, not a one-off, since the crawler landscape shifts and a file that was correct last year may be blocking the wrong thing today. The framing we use at SkyScale is that robots.txt is a living part of your technical setup, reviewed on a schedule alongside everything else that feeds your AI visibility.

Common robots.txt mistakes that cost you citations

The biggest mistake is the blanket AI block, disallowing every bot with AI in its name and erasing yourself from AI search to stop training you barely slowed. The second is blocking resource directories, cutting off the CSS and JavaScript a crawler needs to render your pages correctly.

The third is the forgotten legacy rule, an old disallow on a folder that now holds content you want cited. The fourth is skipping the sitemap directive, leaving crawlers to find your pages the hard way.

And the fifth is never testing, trusting a file you have not verified in years. Avoid those five and your robots.txt turns from a silent liability into a quiet advantage.

If all of this feels like a lot to get right, that is precisely why it pays to check rather than assume. A structured AI visibility audit will confirm whether your robots.txt is helping or quietly hurting, and our case study on rebuilding AI visibility shows how often a single corrected line was the difference between invisible and cited.

And if a past misconfiguration has already cost you ground, the recovery is very doable. The same crawlers that were shut out can be welcomed back, and the path from there is the one we map in winning back visibility lost to AI search. Get the file right once, keep it under review, and it compounds quietly in your favour.