Free tool
Check robots.txt for GPTBot and other AI crawlers
Paste it in. See who's let in, in seconds.
Check robots.txt for GPTBot and 13 other AI and search crawlers in one go. Paste your robots.txt, pick a path to test, and the checker shows whether each crawler is allowed or blocked, and the exact line that decides it.
It covers OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple and Common Crawl. It runs in your browser, and nothing is sent to us.
Runs in your browser. Nothing is sent to us.
40x
more Google AI answers in 3 months
How to check robots.txt for GPTBot
Three steps. Results update as you paste or edit.
- 1
Open your robots.txt
Go to yoursite.co.uk/robots.txt in a new tab and copy everything.
- 2
Paste it in
Paste it into the box. Change the path to test a page or folder, such as /services/, or paste a full URL.
- 3
Read the results
Each crawler shows Allowed or Blocked, the line that decided it, and whether that came from its own group or the * group.
Check your robots.txt
Paste your file below. Nothing you paste is sent to us.
Open yoursite.co.uk/robots.txt in a new tab, copy everything and paste it here. An empty file allows every crawler.
A path like /services/ or a full URL. Testing: /
All 14 crawlers can fetch /. 0 groups found.
- Allowed
GPTBot OpenAI
Collects content that may be used to train OpenAI's models.
No group applies, so it's allowed by default.
- Allowed
OAI-SearchBot OpenAI
Finds and shows sites in ChatGPT search results.
No group applies, so it's allowed by default.
- Allowed
ChatGPT-User OpenAI
Visits a page when a ChatGPT user asks it to.
No group applies, so it's allowed by default.
- Allowed
ClaudeBot Anthropic
Collects content that may be used to train Anthropic's models.
No group applies, so it's allowed by default.
- Allowed
Claude-SearchBot Anthropic
Indexes pages to improve search results in Claude.
No group applies, so it's allowed by default.
- Allowed
Claude-User Anthropic
Visits a page when a Claude user asks it to.
No group applies, so it's allowed by default.
- Allowed
PerplexityBot Perplexity
Finds and links sites in Perplexity's search results.
No group applies, so it's allowed by default.
- Allowed
Perplexity-User Perplexity
Visits a page when a Perplexity user asks a question.
No group applies, so it's allowed by default.
- Allowed
Google-Extended Google
Not a crawler: a token that controls use in Gemini training and grounding. It doesn't affect Google Search.
No group applies, so it's allowed by default.
- Allowed
Googlebot Google
Google Search crawler. Search features such as AI Overviews use what it crawls.
No group applies, so it's allowed by default.
- Allowed
Bingbot Microsoft
Bing's crawler. Bing's index also feeds Microsoft Copilot.
No group applies, so it's allowed by default.
- Allowed
Applebot Apple
Crawls for Apple search features such as Siri and Spotlight.
No group applies, so it's allowed by default.
- Allowed
Applebot-Extended Apple
Not a crawler: a token that controls use in training Apple's AI models.
No group applies, so it's allowed by default.
- Allowed
CCBot Common Crawl
Builds the open Common Crawl dataset, widely used to train AI models.
No group applies, so it's allowed by default.
Follows RFC 9309: a crawler uses the groups that name it, or the * group if none do. The longest matching rule wins and Allow wins a tie. robots.txt is a request, not a lock: well-behaved crawlers follow it, and some user-triggered fetchers may treat it differently.
Why do I have to paste my robots.txt in?
Because a web page can't fetch another site's files for you. Browsers block a page on one site from reading files on another site unless that site allows it, and most robots.txt files don't. This tool is a static page with no server behind it, so pasting is the only way, and it means your file never leaves your browser.
Which AI crawlers does the checker test?
Fourteen user agents. Some fetch pages for search answers, some collect training data, and two are control tokens rather than crawlers.
| User agent | Owner | What it does |
|---|---|---|
| GPTBot | OpenAI | Collects content that may be used to train OpenAI's models. |
| OAI-SearchBot | OpenAI | Finds and shows sites in ChatGPT search results. |
| ChatGPT-User | OpenAI | Visits a page when a ChatGPT user asks it to. |
| ClaudeBot | Anthropic | Collects content that may be used to train Anthropic's models. |
| Claude-SearchBot | Anthropic | Indexes pages to improve search results in Claude. |
| Claude-User | Anthropic | Visits a page when a Claude user asks it to. |
| PerplexityBot | Perplexity | Finds and links sites in Perplexity's search results. |
| Perplexity-User | Perplexity | Visits a page when a Perplexity user asks a question. |
| Google-Extended | A token, not a crawler: controls use in Gemini training and grounding. Doesn't affect Google Search. | |
| Googlebot | Google Search crawler. AI Overviews use what it crawls. | |
| Bingbot | Microsoft | Bing's crawler. Bing's index also feeds Microsoft Copilot. |
| Applebot | Apple | Crawls for Apple search features such as Siri and Spotlight. |
| Applebot-Extended | Apple | A token, not a crawler: controls use in training Apple's AI models. |
| CCBot | Common Crawl | Builds the open Common Crawl dataset, widely used to train AI models. |
Should I block GPTBot?
That's your call, but know what it does. Blocking GPTBot stops OpenAI using your pages for training. It doesn't stop ChatGPT search, which uses OAI-SearchBot. If you want to be recommended, the search crawlers are the ones to keep open: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot.
How does the checker read robots.txt rules?
It follows RFC 9309, the robots.txt standard. A crawler uses the groups that name it, or the * group if none do. The longest matching rule wins, and Allow wins a tie. An empty file allows everything.
robots.txt is a request, not a lock. Well-behaved crawlers follow it, some user-triggered fetchers may treat it differently, and a firewall or CDN bot setting can still block a crawler that robots.txt allows.
Is my information private?
Yes. The checker runs in your browser. The robots.txt you paste and the paths you test are not sent to us or anyone else.
What to do if AI crawlers are blocked
Fix the file, then check nothing else is in the way.
- Find the line.The checker shows the line number that blocks each crawler. Often it's a blanket Disallow: / in the * group.
- Allow the search crawlers.Add a group for OAI-SearchBot, Claude-SearchBot and PerplexityBot with Allow: /, if you want to show up in their answers.
- Check your firewall and CDN.Bot protection can block crawlers that robots.txt allows. Check the settings with your host.
- Paste it again.Run the updated file through the checker before you upload it.
AI crawler robots.txt checker FAQs
Yes. No sign-up and no email needed.
Browsers stop a page on one site from reading files on another, and this tool has no server to fetch it for you. Pasting keeps it simple and private.
GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot finds sites to show in ChatGPT search. You can block one and allow the other.
No. Google-Extended controls use in Gemini training and grounding. AI Overviews are part of Google Search, which uses Googlebot.
Crawler access is only the first step. Assistants also need plain facts on your pages, consistent listings and reviews. Try our AI visibility checker.
Yes. We check robots.txt, firewall and CDN settings as part of our AI SEO work, then fix what's blocking you. We'll reply within the hour (working hours).
Related
Crawlers blocked? Get moving.
Tell us what you need. We'll reply within the hour (working hours), and most websites go live about 5 days after kick-off.
We'll reply within the hour (working hours).
Or email hello@foundryagency.co.uk