Which AI crawlers should you allow in robots.txt?
If you want to be found and recommended by AI assistants, allow their search and user crawlers: OAI-SearchBot and ChatGPT-User, Claude-SearchBot and Claude-User, PerplexityBot and Perplexity-User, plus Googlebot and Bingbot. Training crawlers are a separate choice.
The mistake we see most is a blanket block. Someone reads that AI companies are scraping the web, adds a rule to keep them all out, and the business quietly disappears from ChatGPT, Claude and Perplexity answers. The table further down lists each crawler, what it does and what happens if you block it.
What's the difference between search, training and user crawlers?
They do three different jobs, and blocking one doesn't block the others.
- Search crawlers fetch pages so an assistant can find them and cite them in answers. OAI-SearchBot, Claude-SearchBot and PerplexityBot are examples. Block these and you drop out of that assistant's search answers.
- Training crawlers collect pages to train future models. GPTBot, ClaudeBot and CCBot are examples, and Google-Extended and Applebot-Extended are tokens that control training use. Blocking them keeps your content out of future training data, but doesn't stop you being cited in search.
- User agents fetch a page when someone asks an assistant to read it, such as ChatGPT-User, Claude-User and Perplexity-User. OpenAI and Perplexity say robots.txt may not apply to these, because a person made the request.
How do robots.txt rules work for AI crawlers?
A crawler reads your robots.txt, finds the group that names it, and follows only that group. If no group names it, it follows the catch-all group under User-agent: *.
That detail trips people up. If you have a strict catch-all group and a separate group naming GPTBot, GPTBot follows its own group and ignores the catch-all. And if your catch-all says Disallow: / and nothing names OAI-SearchBot, you've blocked ChatGPT search without meaning to.
robots.txt is a request, not a lock. Reputable crawlers from OpenAI, Anthropic, Google, Microsoft and Perplexity say they follow it. It won't stop a bad actor, and it isn't the place for anything private.
Should you block GPTBot and other training crawlers?
It depends on what your content is worth to you on its own. For most service businesses, being known by future models helps more than it costs.
A publisher, a course provider or anyone who sells their writing has a real reason to keep it out of training data. A plumber, an accountant or an agency mostly wants AI tools to know they exist and what they do. We allow every reputable crawler on our own site, training ones included, because being in model knowledge helps people find us.
Whatever you decide, decide it per crawler, on purpose. Don't let an old rule or a plugin decide for you.
What else can block AI crawlers?
Plenty, even when robots.txt says yes. These are the usual suspects.
- Bot protection: some hosting and security services can block AI crawlers, and some switch that on by default. Check the settings.
- Firewalls and rate limits that treat unfamiliar crawlers as attacks.
- CAPTCHAs and JavaScript challenges on pages that should be public.
- Content that only appears after scripts run, a cookie wall or a login.
- A noindex tag or a robots meta tag left over from a staging site.
How do you check and fix your robots.txt, step by step?
Eight steps. Most sites can be checked in half an hour.
- Open yourdomain.co.uk/robots.txt in a browser and read it top to bottom.
- Look for Disallow: / under User-agent: *, and check whether any AI crawler is left relying on that group.
- Check every group that names an AI crawler, and what it allows.
- Paste the file into our AI crawler robots.txt checker to see which crawlers are let in.
- Check your hosting, CDN and security settings for AI crawler or bot blocking.
- Decide on training crawlers one by one, and write a rule for each.
- Publish the new file, keeping your sitemap line at the bottom.
- Check your server logs or analytics over the next few weeks for visits from the crawlers you allowed. Changes aren't instant.
Does llms.txt control AI crawlers?
No. llms.txt describes your site for AI tools. It can't allow or block anything. Access is still controlled by robots.txt and your server settings.