# AI robots.txt checker > Free AI robots.txt checker: see whether your robots.txt blocks GPTBot, ClaudeBot, PerplexityBot and other AI bots, and generate rules that allow search. URL: https://gradual.pro/en/tools/ai-robots-txt-checker Updated: 2026-09-30 See which AI bots can crawl your site, which line of your robots.txt blocks each one, and generate the right rules. ## Which AI bots does it check? We check 14 bots from the main AI companies and search engines, grouped by what they do: - **Search and citations**: OAI-SearchBot (ChatGPT), Claude-SearchBot (Claude), PerplexityBot, Googlebot and Bingbot. They index your site so it can be shown and cited in answers. - **User-requested visits**: ChatGPT-User, Claude-User and Perplexity-User. They come in when someone asks an assistant to read a specific page. - **Training**: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent and CCBot (Common Crawl). They collect text to train models. Bot names change over time: we update the list when companies publish new ones. ## Does blocking training remove me from ChatGPT? No. Blocking GPTBot or ClaudeBot stops your content being used to train future models, but it does not affect search: ChatGPT uses OAI-SearchBot to search and ChatGPT-User to read pages, and Claude does the same with Claude-SearchBot and Claude-User. Google-Extended controls whether your content is used to improve Gemini and Google's other AI; your presence in Google Search, including its AI answers, depends on Googlebot. If you want to appear in AI answers, what matters is not blocking the search bots or the user-visit bots. Many sites block them by accident with a catch-all rule such as “User-agent: *” followed by “Disallow: /”, or with their CDN's option to block AI bots. ## How are robots.txt rules read? Each bot first looks for a group with its own name (“User-agent: GPTBot”). If there is none, it follows the general group (“User-agent: *”). Within the group, the longest rule that matches the path wins and, in a tie, Allow wins. The asterisk matches any text and the $ sign marks the end of the URL. That is why the path matters: “Disallow: /blog/” blocks your articles but not your homepage. The checker shows you the exact line that decides each case and highlights it in your file, so you know what to change. To check the rest of the signals, use the [AI visibility checker](/en/tools/ai-visibility-checker). If you would rather we handle it, see our [generative engine optimization](/en/services/generative-engine-optimization) service. 1. Enter your domain or the URL of a specific page: robots.txt rules can change from one path to another. 2. We read your robots.txt and test it for 14 bots using the same rules Google applies: the most specific group and the longest matching rule. 3. You see each bot grouped by what it does (search, user-requested visits or training), whether it is allowed and the exact line that decides it. 4. Generate a block for your robots.txt by choosing what to allow and what to block, then copy or download it. ## FAQ ### Should I allow GPTBot? It depends on whether you want your content used to train OpenAI's models. It has no bearing on whether ChatGPT cites you in its searches, which depends on OAI-SearchBot and ChatGPT-User. Many businesses allow the search bots and decide separately about the training ones. ### Where is my robots.txt? Always at the root of your domain: https://yourdomain.com/robots.txt. On WordPress, Shopify or Wix you edit it through a plugin or the SEO settings; otherwise, by uploading the file to your hosting. ### Do AI bots respect robots.txt? OpenAI, Anthropic, Google, Perplexity, Apple, Meta and Common Crawl state that their bots respect it. Even so, robots.txt is a request, not a technical block: to truly stop access you need to block it at the server or the CDN. ### How often should I check it? Whenever you change your website, SEO plugin or CDN, and a couple of times a year, because AI companies release new bots. Some CDNs add rules that block AI bots without you noticing.