AI Crawlers: How to Control GPTBot, ClaudeBot, OAI-SearchBot and the Rest

In short: AI crawlers come in three kinds. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) decide whether ChatGPT, Claude and Perplexity can find and cite your pages. Training crawlers (GPTBot, ClaudeBot, CCBot) and control tokens (Google-Extended, Applebot-Extended) decide whether your content may train future models. User fetchers (ChatGPT-User, Claude-User, Perplexity-User) visit a page because one person asked about it. Allow the search crawlers if you want to appear in AI answers, decide on training separately, and check your firewall, because a CDN rule can block all of them while robots.txt says "allow".

A site owner adds Disallow: / for GPTBot to stay out of training data and believes the job is done. Meanwhile the CDN's bot protection answers every AI user agent with a 403, so ChatGPT search, Claude and Perplexity never see the site at all. Or the opposite: robots.txt blocks everything AI-related, including the search crawlers that would have cited the company in answers.

Both mistakes come from treating "AI crawlers" as one thing. They are not, and each provider documents its bots separately. This guide covers which crawler does what, the three layers that decide whether a crawler actually gets your content, how our free AI crawler checker tests them, and how to fix each problem it can report.

The three kinds of AI crawler

Every major AI provider now separates bots by purpose. The table below follows each provider's own documentation as of October 2026.

Sources: OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl.

Three details in that documentation change how you should write rules:

• Google's AI features run on Googlebot. AI Overviews and AI Mode use the normal Search index. Google says a page must be indexed and eligible to show with a snippet, with no additional technical requirements (AI features and your website). Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." Blocking it is a training and Gemini grounding decision, not a Search decision.
• Applebot falls back to Googlebot rules. Apple says that if your robots.txt does not mention Applebot but mentions Googlebot, Applebot follows the Googlebot instructions. A Googlebot-only block therefore hides you from Siri, Spotlight and Safari suggestions as well.
• Changes are not instant. OpenAI says it can take about 24 hours after a robots.txt update for OAI-SearchBot to adjust, and Perplexity gives up to 24 hours.

Why AI crawler access matters

An assistant can only cite a page it was able to read. If OAI-SearchBot cannot fetch your pricing page, ChatGPT search will answer "how much does X cost" from someone else's page about you, which may be a reseller, an old review or a competitor's comparison. The same holds for Claude and Perplexity.

Training is a separate question with no single right answer. Some publishers block training crawlers to protect paid content; many software companies allow them because they want future models to know their product. What matters is that the decision is deliberate and that it does not take the search crawlers down with it. Our guide to generative engine optimization covers what happens after access: how assistants pick and quote sources.

Three layers decide whether a crawler gets your content

1. robots.txt

robots.txt is the first thing a well-behaved crawler reads. Under RFC 9309, a crawler looks for the group whose User-agent matches its product token (case-insensitive), combines all groups that match, and falls back to the * group only if no group names it. The most specific rule, the one with the longest matching path, wins, and when an allow and a disallow rule are equally specific, allow should be used.

That fallback is where most surprises come from. A file with only this:

User-agent: *
Disallow: /

blocks every AI crawler, Googlebot and Bingbot together. A file with a GPTBot group and a * group gives GPTBot only its own group; it ignores the * rules completely, which matters if you put important disallows there.

2. The firewall, CDN or host

A request can be refused before robots.txt is consulted. Bot protection products, web application firewalls and some hosts match user agents and answer 403, show a challenge page, or rate-limit with 429. Google notes that 429 and 5xx responses make its crawlers slow down (HTTP status codes and Google Search), and AI crawlers face the same walls.

Cloudflare is the most common case. Its original "Block AI bots" setting, which Cloudflare's docs list as deprecating on September 15, 2026, blocked verified bots classified as crawling for AI training. It was replaced by AI bot traffic policies with three presets, Search, Agent and Training, each of which can be allowed, blocked on all pages, or blocked only on pages that show ads (Cloudflare changelog). For new domains from September 15, 2026, Cloudflare's defaults block Training and Agent bots on pages that display ads while Search stays allowed, and Cloudflare notes that multi-purpose crawlers that combine search and training are affected by the training block. Cloudflare's examples for the Agent preset include chat fetch bots, so check that setting if you want user-requested fetches such as ChatGPT-User to work.

3. Rendering

Even when a crawler gets a 200 response, it may not run your JavaScript. Vercel analyzed AI crawler traffic across its network in December 2024 and reported that "none of the major AI crawlers currently render JavaScript," naming GPTBot, ClaudeBot and PerplexityBot (The rise of the AI crawler). If your text is injected by a client-side app, those crawlers receive an empty shell.

How the Searvia AI crawler checker works

The AI crawler checker runs the same engine as our website SEO checker: you enter a domain or a page address, it reads that page and the site files around it, and runs 38 checks in about 20 seconds. The three checks this tool is about appear first in a highlighted box, the first one expanded. Everything below describes what the code actually does.

Which page. The checker audits the exact address you enter, or the home page if you enter only a domain; robots.txt and other site files are always read from the domain. It tries https:// first and falls back to http://, follows up to six redirects, and identifies itself as SearviaBot/1.0 for the normal visit.

AI crawler access (high priority). It downloads /robots.txt from the final host. A 404 or another 4xx means there is no file, so everything is allowed. A 5xx, a 429 or a connection error means the file is unreachable: every token then shows "Unknown" and the check fails, because RFC 9309 tells crawlers to treat the site as fully blocked. A 200 that returns an HTML page is not a valid robots.txt. For each of 10 crawler tokens and 2 training opt-out tokens, it picks the group that names the token, or the * group if none does, and asks one question: may this token fetch the path of the page you entered (/ for the home page)? It uses the longest-match rule with ties going to allow, understands * and $ wildcards, comments, several User-agent lines sharing one set of rules, and repeated groups for the same token.

Googlebot and Bingbot are in the search column because Google's AI features run on Googlebot and Microsoft Copilot grounds its answers in Bing. Google-Extended and Applebot-Extended never visit your site; a rule for them only controls whether content that Googlebot or Applebot already crawled may be used for AI training. Each row shows Allowed, Blocked or Unknown.

The firewall test. Next, the checker requests the same page once more with the user agent OpenAI documents for GPTBot (GPTBot/1.4), without following redirects. If the normal visit succeeded (status below 400) but this request gets 401, 403, 429 or 503, the report warns that requests identifying as GPTBot were refused. The request does not come from OpenAI's IP addresses, so the real crawler may be treated differently.

Content without JavaScript (medium priority). It counts the words of main text in the raw HTML, leaving out menus and footers. If there are fewer than 80 words and the page either has an empty app container (<div id="root">, app, __next, __nuxt or svelte) or loads three or more external scripts, it warns that crawlers which do not run JavaScript may see an empty page.

robots.txt (medium priority). The third focused check reports whether robots.txt exists, whether it answers with a server error or an HTML page, and whether it blocks Googlebot from the home page or from the page you entered. Our robots.txt guide covers that file in depth.

Status rules. AI crawler access fails if any of the seven search tokens is blocked or robots.txt cannot be reached. A refused firewall test gives a warning. Blocked training crawlers and opt-out tokens are listed but the check still passes, because that is a legitimate choice.

What the checker does not test

Knowing the limits keeps you from over-reading a green result:

• One path per run. Only the path of the page you enter is tested (/ for a bare domain). A rule such as Disallow: /pricing/ for OAI-SearchBot is caught only if you enter that page. Read your file for section-level blocks.
• The firewall test uses one user agent from our servers. It sends GPTBot's name, not OAI-SearchBot's, and it does not come from OpenAI's IP addresses. A firewall that verifies bots by IP can block our request as an impostor while letting the real GPTBot through, and a firewall that blocks OpenAI's IP ranges but not the user agent passes our test. Treat a firewall warning as "check your bot settings", then confirm in logs.
• Tokens not on the list, such as Perplexity-User or OAI-AdsBot, are not tested.

Reading the result

The table under the AI crawler row shows each token as Allowed, Blocked or Unknown, with its label. The overall A+ to F grade covers all 38 checks, weighted by priority: high-priority checks count 10 points, medium 5 and low 2, a warning earns half, and informational results are left out. "Fix these first" lists up to eight problems, high priority first. Each row also shows what share of audited sites pass the same check, which tells you whether a problem is rare or common.

How to fix every problem the checker reports

1. robots.txt blocks AI search crawlers

Open https://yourdomain.com/robots.txt and find the group that applies to each blocked token: a group that names it, otherwise *. Then decide your policy and write it explicitly. This file allows AI search and user fetches, blocks training, and keeps the usual rules for everyone else:

# AI search and user-requested fetches
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# Model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

Step by step:

1. Remove any User-agent: * plus Disallow: / left over from a staging site. That single pair blocks everything.
2. If the AI search group needs the same private-area rules as everyone else, copy them in. A named group does not inherit the * rules.
3. If you use a plugin or CDN feature that writes robots.txt for you, change it there; editing the file by hand is overwritten on the next deploy.
4. Save, reload the live file in a private window, and run the AI crawler checker again.
5. Allow a day before expecting the change to take effect on OpenAI's and Perplexity's side.

2. The firewall turned the crawler away

1. Find the rule. In your CDN or firewall's event log, filter by user agents containing GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot and note which rule acted: bot protection, a managed rule set, a rate limit or a country block.
2. On Cloudflare, open AI Crawl Control: it lists each AI crawler with Allow or Block, so you can allow AI search crawlers and still block training crawlers. Also review the AI bot traffic policies in your zone's security settings. Set Search to allow if you want to be cited. Decide Training and Agent on their own merits, and remember that "block on pages with ads" only applies to pages Cloudflare detects as showing ads.
3. On other WAFs, add an allow rule for the bots you want, matched on user agent and the provider's published IP ranges, as Perplexity recommends. OpenAI publishes searchbot.json, gptbot.json and chatgpt-user.json, Perplexity perplexitybot.json and perplexity-user.json, Anthropic claude.com/crawling/bots.json, and Common Crawl ccbot.json. Give the allow rule a higher priority than blocking rules and refresh the ranges regularly, since they change.
4. Rate limits. If the probe got 429, your rate limit is stricter than a crawler's pace. Raise it for verified bots instead of turning it off.
5. Do not block by IP to opt out of training. Anthropic warns that blocking its IP addresses may not opt you out reliably, because its bots then cannot read your robots.txt. Use robots.txt for policy and the firewall for abuse.

If you are sure your firewall verifies bots by IP and the warning comes from our unverified request, confirm with logs: real requests from OpenAI's ranges should be getting 200.

3. Training crawlers are blocked

This is reported but not counted as a failure. Check two things. First, that the block is what you want: blocking Google-Extended also removes your content from grounding in Gemini apps, and blocking Applebot-Extended only affects training, not Apple's search features. Second, that the block lives in a named group and not in *, where it would also catch search crawlers.

4. Only a few words are visible before JavaScript runs

1. View the page source (not the inspector) of your home page and key pages. If the main text is not there, crawlers that do not render JavaScript see nothing.
2. Render key templates on the server: server-side rendering, static generation or pre-rendering. In Next.js, for example, use Server Components or static generation instead of fetching the text in the browser.
3. Make sure the essentials are in the HTML: the product description, pricing, plans and FAQ answers. Interactive widgets can stay client-side.
4. Run the checker again; the word count in the message should rise.

5. robots.txt is missing or blocks Google

The checker tells the cases apart. "No robots.txt file was found" means a 404 or another 4xx, which is harmless: crawlers treat it as "no restrictions". "robots.txt answers with status N" or "could not be reached" means a 5xx, a 429 or a timeout, which is serious: RFC 9309 says a crawler must then assume complete disallow, and Google stops crawling for the first 12 hours (Google's robots.txt specification). Fix the server so /robots.txt answers 200 with plain text, or 404 if you need no rules, and confirm with curl -I https://yourdomain.com/robots.txt. A robots.txt that returns an HTML page is flagged too. If robots.txt blocks Googlebot from /, fix that first: it also blocks Google's AI features and, through Apple's fallback, Applebot.

Verify crawlers in your logs

Anyone can send a request that says "GPTBot". Before you allow a bot through a firewall or blame one for load, verify it:

• OpenAI, Anthropic, Perplexity, Common Crawl: compare the IP with the JSON lists above.
• Google: run a reverse DNS lookup on the IP, check that the name ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup and confirm it returns the same IP (Verifying Googlebot). Google also publishes JSON lists such as common-crawlers.json and user-triggered-fetchers.json.
• Common Crawl notes that some crawlers falsely identify as CCBot, which is why it publishes its ranges with reverse DNS.

Worked example (hypothetical)

The following is a hypothetical example, not real data. Larkspur Analytics, an invented B2B dashboard tool, finds that ChatGPT describes its pricing from a 2024 review site. The AI crawler checker reports:

The robots.txt was fine; the team had blocked training on purpose. The 403 came from a bot protection rule a contractor enabled during a scraping incident, which matched every user agent containing "bot". The fixes:

1. Replaced the broad rule with a rate limit, and added allow rules for OAI-SearchBot, Claude-SearchBot and PerplexityBot matched on user agent and published IP ranges.
2. Moved the pricing and features pages to static generation, so the HTML carries the full text.
3. Reran the checker: AI crawler access passed and the visible word count rose to 1,180.
4. Two weeks later, server logs showed verified OAI-SearchBot and PerplexityBot requests to the pricing page, and new ChatGPT answers cited it.

A 20-minute AI crawler checklist

1. Run the free AI crawler checker on your domain.
2. Write down your policy: search allowed, training allowed or blocked, user fetches allowed.
3. Express it in named robots.txt groups, never only in *.
4. Check the firewall or CDN bot settings, including Cloudflare's AI Crawl Control.
5. View the source of your key pages and confirm the main text is in the HTML.
6. Confirm in logs, a week later, that verified search crawlers are getting 200.

How Searvia helps with AI crawlers

The free AI crawler checker gives you the robots.txt picture for 10 AI and search crawlers and 2 training opt-out tokens, a firewall test and the JavaScript check in one run, with a shareable report. The broader AI search readiness checker adds llms.txt, identity markup, question-and-answer content and freshness. If your site runs on Cloudflare, connect Cloudflare with a read-only token: the AI crawlers tab of your website project shows which AI crawlers really visit, how often and which pages they read, with fake crawlers filtered out. Access is only the first step; AI visibility tracking then checks every week whether assistants such as ChatGPT, Gemini and Perplexity actually mention and cite you. Start with the free plan on the pricing page.