How do I block AI crawlers like GPTBot in robots.txt?

Updated October 13, 2026 · Check Before Publish

To block an AI crawler, add a robots.txt group that names its token with Disallow: /. Each operator uses separate tokens for training, search and user-requested fetches, so blocking one does not block the others. robots.txt is a request, not access control, and some user-triggered fetchers may ignore it.

The short version

• A crawler obeys the group that matches its own token, and uses the * group only if no group names it.
• Tokens differ by purpose. OpenAI: GPTBot for training, OAI-SearchBot for search, ChatGPT-User for fetches a user asks for. Anthropic: ClaudeBot, Claude-SearchBot, Claude-User.
• Blocking Google-Extended does not affect Search: Google says it “does not impact a site's inclusion in Google Search”. Google's AI features in Search are controlled through Googlebot and snippet controls.
• A change takes time: OpenAI and Perplexity say up to about 24 hours, and Google caches robots.txt for up to 24 hours.
• Blocking looks forward. None of the operator pages we read says it removes what was already collected; Anthropic says a restriction signals that a site's “future materials” should be excluded.

The tokens, in each operator's words

The rules to write

Block a training crawler and keep a search crawler (the independence OpenAI describes):

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Block several, each by its own token:

User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /

The file must be at the root of each host: https://www.example.com/robots.txt, and a subdomain has its own. The RFC that defines the protocol says plainly that these rules “are not a form of access authorization”, so do not rely on robots.txt to keep anything private.

Check what your file really says for one URL

Paste a page address. The checker reads that site's live robots.txt and reports, for Googlebot, Bingbot and six AI and dataset crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot, Applebot-Extended), whether each may crawl that URL, and which line decides.

What we tested

On 2026-10-13 we ran the live Robots.txt Checker on real, public sites. We chose them because their robots.txt files treat AI crawlers in different ways, and then compared the checker's verdicts with the file itself.

What this shows:

• The token must match. Cloudflare lists two Anthropic names but not ClaudeBot, which is the token Anthropic's current help page gives for training, so the checker reports ClaudeBot as allowed.
• The answer is per URL. The same GitHub file allows its home page and blocks a repository's commit history for the same crawlers. Check the URLs you care about, not only the home page.
• Not naming a bot means the * group. That is how the Guardian's file blocks some AI crawlers and leaves GPTBot allowed. The RFC says that when no group matches, a crawler obeys the * group.

One thing about our own tool. On the Cloudflare and Guardian files the checker listed a line it does not recognise (a Content-Signal line and a License line) under “Review”, with the advice “Fix the spelling, or remove the lines”. Both are deliberate extensions by those sites, so that advice does not fit them; treat the line as a note that the checker does not know the directive.

What this can't tell you

The checker reads the file; it does not test any crawler's behaviour. Whether an operator honours its own token is the operator's statement, quoted above, not something we measured. Operators' pages say user-initiated fetchers may ignore robots.txt, and Google and Common Crawl both warn that user agent strings can be faked. Several operators (OpenAI, Anthropic, Google, Perplexity, Apple and Common Crawl) publish IP ranges, but Anthropic cautions that blocking its IP addresses can stop it reading your robots.txt, so the opt-out may not work.

Related: Blocked by robots.txt (the Search Console status) and Blocked due to access forbidden (403).

Fixed this one? Before you send people to the page, run the full pre-publish check for everything else: indexing signals, link preview, broken links, alt text, structured data and HTTPS in one pass.