How to Set Up robots.txt for AI SEO in 2026

What is robots.txt for AI?

robots.txt for AI means using your site's robots.txt file to set separate rules for AI crawlers. It is a plain text file at the root of your domain that names user agents and the paths they may not fetch, under the Robots Exclusion Protocol.

The file is not new, but the crawlers are. One company can now run several bots with different jobs: OpenAI has one for training, one for search and one for user requests. A rule that treats them as one bot will either block too much or too little.

Why does robots.txt matter for AI SEO now?

It matters because AI crawlers now take up a real share of traffic, and your rules decide whether AI answers can use your pages. Cloudflare's 2025 Year in Review put AI bots at 4.2% of HTML requests, close to Googlebot's 4.5%.

Blocking also has a measurable cost for some sites. A working paper by Hangcheng Zhao of Rutgers and Ron Berman of Wharton found about a 7% decline in weekly visits in the six weeks after news publishers blocked AI crawlers. The estimate shrank across revisions of the paper and covers news publishers only, so read it as a warning rather than a rule.

What are the four types of AI crawler?

AI crawlers fall into four groups based on when they fetch your page and what they do with it. Sort every bot into one of these before you write a rule.

Which bots should you keep for AI citations?

You should keep every bot that fetches pages to answer a person's question, because those are the ones that can cite and link to you. This is the core list:

OpenAI is direct about the stakes: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers".

Brave needs a special note. Brave says its crawler does not use a separate user agent, and that it follows what Googlebot is allowed. Lists that include "Bravebot" are unreliable, so allowing Googlebot is the part that counts.

Should you block GPTBot and ClaudeBot?

Most brands should keep GPTBot and ClaudeBot, because a model that has read about you may describe you better. Publishers who sell their content have good reason to block them. These bots train models and do not power live search, so blocking them will not remove you from ChatGPT or Claude search.

The decision comes down to what you sell. If you sell a product and want models to know your brand, the content is marketing and you benefit from reach. If the content is the product, as with paid research or journalism, training access gives away what you charge for.

Anthropic's documentation separates ClaudeBot from Claude-SearchBot and Claude-User, so you can treat each one differently.

Google and Apple work differently. Google-Extended and Applebot-Extended are tokens with no crawler of their own. They are switches that control how content already crawled gets used for AI.

Which bots should you block?

You should consider blocking crawlers that use your pages for training or datasets and return no traffic. Bot operators document most of these, so you can check each decision for yourself.

Two of these need more thought than the rest.

CCBot is not a pure AI scraper. Common Crawl is a non-profit open dataset, and its pages make no statement about AI training. But LLM builders use it: Mozilla found that at least 64% of 47 LLMs it studied used Common Crawl data. If you keep GPTBot so models learn your brand, blocking CCBot works against that goal.

Bytespider's compliance is disputed. TollBit data recorded disallowed fetches under that name, though spoofing is possible, so a robots.txt rule alone may not stop it.

Meta runs a separate crawler for AI search. Its crawler documentation says meta-webindexer improves Meta AI search results, so keep that one if Meta AI citations matter to you.

Should you block SEO crawlers?

You should usually leave SEO crawlers alone unless they cause load problems. Barkrowler and SERankingBacklinksBot collect link data for SEO tools, and they do not train AI models.

Blocking one saves some bandwidth, but it likely means your links stop appearing in that vendor's data, which affects anyone researching you. Both say they honor robots.txt, and Barkrowler supports crawl-delay, so slowing it down is a softer option than a block.

What should your robots.txt file look like?

Your file should allow by default, then block a short named list. Use this template and adjust the training section to match your decision.

# Search and AI answer bots: covered by the allow-all group at the end.
# Listed here so the policy is explicit.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: DuckAssistBot
User-agent: YouBot
User-agent: ExaSearchBot
User-agent: LinkupBot
User-agent: ShapBot
Disallow:

# Training bots: kept for brand reach. Change to "Disallow: /" to opt out.
User-agent: GPTBot
User-agent: ClaudeBot
Disallow:

# Link previews. Blocking these breaks shared link cards.
User-agent: facebookexternalhit
User-agent: Twitterbot
Disallow:

# Training and scraping with no traffic return.
User-agent: meta-externalagent
User-agent: meta-externalads
User-agent: Bytespider
User-agent: Reflectionbot
User-agent: ElevenlabsCrawler
User-agent: CCBot
User-agent: AwarioSmartBot
User-agent: AwarioRssBot
User-agent: SenderBot
User-agent: SpiderLing
Disallow: /

# Everyone else, including Googlebot, bingbot and Applebot.
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Several User-agent lines above one rule form a single group, which is valid under RFC 9309. An empty Disallow: means nothing is blocked.

A number of other tokens circulate on blocklists, including AionBot, TokenFoundryBot and TernCrawler. We could not find an operator's documentation for these, so we left them out of the template. Adding them is low risk if they appear in your logs.

How do you set up and test robots.txt for AI?

You set it up by checking who really visits, writing the rules, and confirming the result in your logs. Follow these steps:

1. Read your server logs. List the bots that actually hit your site, and how often.
2. Sort each bot by type. Use the four groups above.
3. Check your CDN settings. A firewall rule can block a bot your robots.txt allows.
4. Publish the file at your domain root. It must live at /robots.txt, on every subdomain you want covered.
5. Test it. Fetch the file and confirm it returns a 200 status as plain text.
6. Recheck your logs in a week. Confirm that blocked bots stopped.
7. Review it every quarter. New crawlers appear often, and old lists go stale.

If you also want to describe your site to AI tools, our free llms.txt generator builds that file. Keep expectations modest: Google says Google Search does not use llms.txt.

What mistakes should you avoid in robots.txt for AI?

The most damaging mistake is blocking by company name instead of by job. Writing one rule for OpenAI's training bot is fine. Copying a "block all AI" list usually removes you from the search products you wanted to appear in.

• Expecting a named group to inherit from *. A bot that matches a named group ignores the * group, so repeat any path rules it needs.
• Using Google-Extended to leave AI Overviews. It does not do that. Use Googlebot rules, or the Search Console setting that Google says will not affect your ranking.
• Blocking Googlebot to avoid AI. That removes you from Google Search too.
• Blocking facebookexternalhit. Your shared links will lose their previews. Check them with our Open Graph checker.
• Treating robots.txt as security. Private pages need authentication, since the file itself is public and lists what you are hiding.
• Forgetting user-triggered fetchers. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.

How do you get cited once crawlers can read your site?

You get cited by being mentioned on the pages AI engines trust, since access only makes you eligible. Ahrefs found in a study of 75,000 brands that branded web mentions had the strongest correlation with AI Overview visibility, at 0.664.

That turns citation into an outreach job. You need other relevant sites talking about you, which is what link building produces. Our guide on how many backlinks you need to rank shows how to set a target. Seowly runs that outreach from your own inbox, and you can check your current standing with our AI visibility map.

The bottom line on robots.txt for AI

Sort bots by job, not by company. Keep the crawlers that cite you and the ones that power link previews, make a deliberate choice on training, and block the rest by name. Then read your logs to confirm it worked, and review the file every quarter, because this list will change.