AI robots.txt checker
See which AI bots can crawl your site, which line of your robots.txt blocks each one, and generate the right rules.
FreeNo sign-up10 seconds
How it works
- Enter your domain or the URL of a specific page: robots.txt rules can change from one path to another.
- We read your robots.txt and test it for 14 bots using the same rules Google applies: the most specific group and the longest matching rule.
- You see each bot grouped by what it does (search, user-requested visits or training), whether it is allowed and the exact line that decides it.
- Generate a block for your robots.txt by choosing what to allow and what to block, then copy or download it.
Which AI bots does it check?
We check 14 bots from the main AI companies and search engines, grouped by what they do:
- Search and citations: OAI-SearchBot (ChatGPT), Claude-SearchBot (Claude), PerplexityBot, Googlebot and Bingbot. They index your site so it can be shown and cited in answers.
- User-requested visits: ChatGPT-User, Claude-User and Perplexity-User. They come in when someone asks an assistant to read a specific page.
- Training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent and CCBot (Common Crawl). They collect text to train models.
Bot names change over time: we update the list when companies publish new ones.
Does blocking training remove me from ChatGPT?
No. Blocking GPTBot or ClaudeBot stops your content being used to train future models, but it does not affect search: ChatGPT uses OAI-SearchBot to search and ChatGPT-User to read pages, and Claude does the same with Claude-SearchBot and Claude-User. Google-Extended controls whether your content is used to improve Gemini and Google's other AI; your presence in Google Search, including its AI answers, depends on Googlebot.
If you want to appear in AI answers, what matters is not blocking the search bots or the user-visit bots. Many sites block them by accident with a catch-all rule such as “User-agent: *” followed by “Disallow: /”, or with their CDN's option to block AI bots.
How are robots.txt rules read?
Each bot first looks for a group with its own name (“User-agent: GPTBot”). If there is none, it follows the general group (“User-agent: *”). Within the group, the longest rule that matches the path wins and, in a tie, Allow wins. The asterisk matches any text and the $ sign marks the end of the URL.
That is why the path matters: “Disallow: /blog/” blocks your articles but not your homepage. The checker shows you the exact line that decides each case and highlights it in your file, so you know what to change.
To check the rest of the signals, use the AI visibility checker. If you would rather we handle it, see our generative engine optimization service.
Frequently asked questions
Should I allow GPTBot?
It depends on whether you want your content used to train OpenAI's models. It has no bearing on whether ChatGPT cites you in its searches, which depends on OAI-SearchBot and ChatGPT-User. Many businesses allow the search bots and decide separately about the training ones.
Where is my robots.txt?
Always at the root of your domain: https://yourdomain.com/robots.txt. On WordPress, Shopify or Wix you edit it through a plugin or the SEO settings; otherwise, by uploading the file to your hosting.
Do AI bots respect robots.txt?
OpenAI, Anthropic, Google, Perplexity, Apple, Meta and Common Crawl state that their bots respect it. Even so, robots.txt is a request, not a technical block: to truly stop access you need to block it at the server or the CDN.
How often should I check it?
Whenever you change your website, SEO plugin or CDN, and a couple of times a year, because AI companies release new bots. Some CDNs add rules that block AI bots without you noticing.
Keep exploring
Answer engine optimization
Get ChatGPT, Perplexity, Gemini and Google AI Overviews to cite your business when people ask about what you sell.
AI visibility checker
Check whether ChatGPT, Claude and Perplexity can read and cite your site: a score, fixes ranked by impact and a comparison.
llms.txt validator and generator
Validate your llms.txt against the llmstxt.org format or generate a draft from your sitemap, ready to copy.
GEO (Generative Engine Optimization)
GEO (Generative Engine Optimization) is the set of techniques that help AI answer engines, such as ChatGPT, Perplexity, Gemini or Google's AI overviews, find a website's content, understand it and cite or recommend it in their answers.
llms.txt
llms.txt is a Markdown file placed at the root of a website (/llms.txt) that gives language models a summary of the site and a list of links to its most important content. Jeremy Howard proposed it in 2024.
Pick a time yourself
30 minutes by video call. Free, no commitment.
Prefer us to get in touch? Leave your details.