Skip to content
Numinam, home page
AI visibility

Robots.txt and AI: which crawlers should you let in?

Tell search crawlers apart from training crawlers, then check the live robots.txt file and your firewall rules.

Robots.txt and AI: which crawlers should you let in?
Published
Updated
Reading time
8 min
Author
SBSébastien BalieuFounder of Numinam

If you want AI assistants to find your company’s public pages, check that their search crawlers can reach them. Whether your content may be used to train models is a separate decision. Your robots.txt file records both choices. Confidential information needs access control on top.

Search and training: which crawlers should you tell apart?

Each vendor documents its crawlers on an official page. The table gives their exact names and the behaviour they declare, checked in October 2026.

VendorCrawlerDocumented userobots.txt rules
OpenAIOAI-SearchBotChatGPT search resultsFollowed
OpenAIChatGPT-UserActions a user starts in ChatGPTMay not apply
OpenAIGPTBotContent that may be used for trainingFollowed
AnthropicClaude-SearchBotQuality of Claude’s search answersFollowed
AnthropicClaude-UserPages fetched for a user’s questionFollowed
AnthropicClaudeBotContent that may be used for trainingFollowed, with Crawl-delay
PerplexityPerplexityBotPerplexity search results, no trainingFollowed
PerplexityPerplexity-UserFetches made at a user’s requestGenerally ignored
GoogleGooglebotSearch, AI Overviews and AI ModeFollowed
GoogleGoogle-ExtendedTraining and grounding in Gemini apps and Vertex AIrobots.txt token, no separate crawler
Common CrawlCCBotOpen web archive, reusable by anyoneBlocking via robots.txt documented

OpenAI documents separate settings for each of its crawlers. Perplexity also separates its search crawler from user-requested fetches, and those fetches generally ignore robots.txt. Blocking PerplexityBot therefore gives you no guarantee that your brand will never be cited.

OpenAI also states that ChatGPT-User plays no part in deciding whether a page appears in ChatGPT search. OAI-SearchBot does that job. A site that refuses OAI-SearchBot can still show up as a plain navigation link.

Anthropic says its crawlers follow standard robots.txt directives and that ClaudeBot accepts the Crawl-delay extension. It warns that blocking its IP addresses will not reliably keep you out over time, and recommends robots.txt instead. Blocking Claude-User reduces your visibility in searches that users run from Claude.

Common Crawl builds an open archive of the web that anyone can download and analyse. Blocking CCBot keeps your new pages out of that archive. Copies already published stay where they are.

Google states that access to its AI features in Search depends on Googlebot. Google-Extended is a different control. Google-Extended has no HTTP user agent of its own: it is a token that Google’s crawlers read in your file, and it has no effect on inclusion or ranking in Google Search.

How do you leave AI Overviews and stay in Google?

Blocking Googlebot would also remove your site from the standard results. Google offers narrower controls.

Search Console has a setting for generative AI features in Search. It covers AI Overviews, AI Mode and AI Overviews in Discover, and it has been available to every site since 31 August 2026 (Google’s announcement). Google warns that an excluded site gets no more visits or impressions from those features, and that the setting is not used as a ranking signal anywhere else in Search.

For page-level control, the robots meta tags nosnippet and max-snippet and the data-nosnippet attribute limit the snippets Google can reuse. They also affect snippets in the standard results. Test them on a few pages before rolling them out.

How do you choose your rules?

For your service pages and public proof, allowing search crawlers fits a visibility goal. For training, decide according to your content and your usage rights. None of these choices guarantees a recommendation.

The type of content you publish helps you decide. If your pages present an offer you sell elsewhere, a model reusing them costs you little. If the content is the product itself (paid courses, a database, a publisher’s articles), a model trained on those pages can compete with what you sell, and blocking the training crawlers is a reasonable call.

Example of separate rules, to adapt to your site:

User-agent: OAI-SearchBot
Allow: /
Disallow: /private-area/

User-agent: GPTBot
Disallow: /

User-agent: PerplexityBot
Allow: /
Disallow: /private-area/

A group for a specific crawler does not automatically inherit the rules in the * group. Have the excluded paths checked in every group concerned. This example covers only the groups shown, so add your site’s other rules to it before you use it.

To block training at the four vendors in the table and keep search open, add these groups:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

These rules apply to pages read after they take effect. Content already collected stays in the existing data.

For a private area, require authentication. Google points out that robots.txt is not a way to hide confidential information.

How does a crawler read your file?

Google describes how it reads the file in its robots.txt specification:

  • A crawler applies a single group: the one whose User-agent matches its name most specifically. The order of the groups in the file makes no difference.
  • Several groups declared for the same crawler are merged. The * group is not added to a specific group.
  • When rules conflict, Google applies the least restrictive one.
  • Google reads at most 500 KiB and caches the file for up to 24 hours.
  • A missing file (4xx error, except 429) means no restrictions. A 5xx server error pauses crawling at first.

The other vendors give similar delays: about 24 hours at OpenAI for its search results, up to 24 hours at Perplexity.

Why check Cloudflare or your firewall as well?

A crawler your file allows can still be blocked by a firewall or an anti-bot challenge. Cloudflare can also manage or add robots.txt directives.

With its managed robots.txt, Cloudflare puts its own directives before yours as soon as a policy blocks training. The example in its documentation disallows eight crawlers: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. Since July 2025, Cloudflare has asked every new domain whether it wants to allow AI crawlers. A setting picked at sign-up can therefore contradict the file your developer wrote.

Ask your provider to check the file actually served on the public domain, then the HTTP responses in the logs. A local copy of the file can differ from the live one. If the firewall has to let a crawler through, verify its identity with the methods and IP ranges its vendor publishes, as well as its declared name.

OpenAI publishes its ranges per crawler (for example openai.com/searchbot.json), Perplexity does the same for its two crawlers, Common Crawl publishes index.commoncrawl.org/ccbot.json, and Anthropic keeps a list too. Common Crawl warns that fake crawlers use the CCBot name.

What did I choose for my own site?

The file on the Numinam site allows the OpenAI, Anthropic and Perplexity search crawlers. It also allows GPTBot, ClaudeBot, Google-Extended and CCBot. Other crawlers, Bytespider among them, are blocked. That choice fits my site. Your business may call for different rules.

The site has no client area, so every group allows the whole site. The crawlers that fetch a page at a user’s request, ChatGPT-User and Claude-User, are declared explicitly. For your own site, list the sections that must stay public before you write the rules.

How do you check after a change?

Open the public robots.txt, test the rules on a few URLs and search the logs for successful fetches and refusals. Note the date of the change so you can compare the crawler’s visits before and after.

In Search Console, the robots.txt report shows which version of the file Google fetched and any parsing errors. For other crawlers, search the server logs for their names and check the IP address of each visit against the list the vendor publishes.

If a blocked crawler keeps reading your pages, write to its vendor from an address on your domain. Anthropic, for example, gives claudebot@anthropic.com and asks for that domain address to verify the report.

Then measure citations and visits separately. A successful fetch shows the crawler could read the page. Whether that reading led to a citation is a second measurement.

The GEO and SEO guide suggests a way to track the questions your prospects ask.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot is about training. Appearing in ChatGPT search depends on OAI-SearchBot, which OpenAI controls separately.

Does blocking Google-Extended take me out of AI Overviews?

No. AI Overviews and AI Mode depend on Googlebot. To opt out, use the Search Console setting for generative AI, or the snippet tags on the pages concerned.

Do I need a robots.txt file to be cited by AI assistants?

No. Without a file, Google treats the whole site as allowed. You need the file once you want to exclude a section or handle training and search differently.

Does robots.txt protect my content?

No. Following the file is voluntary, and some user-requested fetchers ignore it. A confidential area needs authentication.

How long before a change is taken into account?

Allow up to 24 hours at Google, OpenAI and Perplexity, according to their documentation. The actual delay depends on the crawler’s next visit to your site.