Robots.txt: How It Works, Examples and How to Test It

In short: robots.txt is a plain text file at /robots.txt that tells crawlers which URLs they may fetch. Rules are grouped by User-agent; a crawler uses the group that names it, or the * group if none does, and the longest matching Allow or Disallow path wins. It controls crawling, not indexing: to keep a page out of Google, use noindex. A 404 means "crawl everything", a 5xx means "stop". Most damage comes from one line, Disallow: /, left over from a staging site.

robots.txt is the shortest file on most websites and the one with the largest blast radius. A single Disallow: / under User-agent: * hides the whole site from Google, Bing, Apple and every AI search crawler. A missing slash, a rule in the wrong group or a server error on the file itself can do nearly as much damage, and nothing on the visible site looks broken.

This guide covers how crawlers read the file according to RFC 9309 and Google's specification, copy-ready examples, how our free robots.txt checker tests your file, and how to fix every problem it reports.

What robots.txt does, and what it doesn't

Google's introduction puts it plainly: a robots.txt file "tells search engine crawlers which URLs the crawler can access on your site," and its main job is managing crawler traffic. It is "not a mechanism for keeping a web page out of Google." A disallowed URL "might still" be indexed "if it is linked from other places on the web," typically shown without a description.

Three more limits from the same sources:

• It is a request, not a lock. RFC 9309 says the rules "are not a form of access authorization." Respectable crawlers obey; others might not. Protect private content with authentication.
• It applies per host, protocol and port. https://www.example.com/robots.txt does not cover https://shop.example.com or http://www.example.com. Each needs its own file, and Google does not look for robots.txt files in subdirectories.
• Different crawlers interpret syntax differently. Stick to the standard fields and simple patterns, and every major crawler will read them the same way.

The syntax

A robots.txt file is a list of groups. Each group starts with one or more User-agent lines followed by Allow and Disallow rules. Sitemap lines can appear anywhere.

# Comments start with a hash
User-agent: Googlebot
User-agent: Bingbot
Disallow: /cart/
Disallow: /search
Allow: /search/help

User-agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

Field names are case-insensitive; paths are case-sensitive, so Disallow: /Admin/ does not block /admin/. Two special characters are supported: * matches any sequence of characters, and $ marks the end of the URL. Disallow: /*.pdf$ blocks every URL ending in .pdf; Disallow: /*? blocks every URL with a query string.

Which group applies to a crawler

This is the rule people get wrong most often. Per Google, "only one group is valid for a particular crawler": the group with the most specific matching user agent. Groups that name the same agent are merged, but a named group is not combined with the * group. RFC 9309 says the same: crawlers match their product token case-insensitively, combine matching groups, and fall back to * only "if no matching group exists."

So in the file above, Googlebot follows only its own group. It may crawl /admin/, because the * group does not apply to it. If a rule should apply to everyone, repeat it in each named group.

Two crawler-specific details are worth knowing. Google's Googlebot token covers Search, Discover, Images, Video and News; more specific tokens like Googlebot-Image let you control one product (Google's common crawlers). And Apple says that if robots.txt does not mention Applebot but mentions Googlebot, Applebot follows the Googlebot instructions (About Applebot).

Which rule wins

Within a group, the most specific rule applies, meaning the one with the longest matching path. When an allow and a disallow rule match equally, the least restrictive one, allow, wins. Order in the file does not matter.

How status codes change everything

Crawlers request /robots.txt before anything else, and its HTTP status matters as much as its content. From Google's specification and RFC 9309:

The practical consequence: a 404 for robots.txt is harmless, but a firewall or server that answers 500 or 503 for the file can stop crawling of the whole site. Google also warns not to use 401 or 403 to limit crawl rate.

Two limits round it out: Google caches the file for up to 24 hours, so changes are not instant, and it ignores content after 500 KiB. The file must be UTF-8.

robots.txt vs noindex

Google's noindex documentation describes the trap: if robots.txt blocks a page, the crawler never sees its noindex rule, and the page can still appear in results. The right tool depends on the goal:

Copy-ready robots.txt examples

Allow everything and list the sitemap (the right file for most small sites):

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

A typical online shop:

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /*?sort=
Disallow: /*?filter=

Sitemap: https://www.example.com/sitemap.xml

A staging site (and only a staging site; protect it with a password too):

User-agent: *
Disallow: /

AI search allowed, AI training blocked:

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

Blocking Google-Extended does not affect Google Search; Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." AI Overviews and AI Mode run on Googlebot (AI features and your website). OpenAI says opting out of OAI-SearchBot keeps you out of ChatGPT search answers, so do not block it unless that is what you want (OpenAI crawlers). Our AI crawlers guide covers every AI token.

Do not block CSS, JavaScript or images that pages need to render. Google says you can block unimportant resources if pages "won't be significantly affected by the loss", but blocking files that carry your content or layout stops Google from understanding the page.

How the Searvia robots.txt checker works

The robots.txt checker runs the same engine as our website SEO checker: enter a domain or a page address and it reads that page (your home page if you enter only a domain) and the site files around it, runs 38 checks in about 20 seconds, and shows the four checks this tool is about first. Here is what each one does.

robots.txt (medium priority). The checker follows the page to its final address, then requests /robots.txt on that same host, following redirects. It reads the status code the way the table above does: a 404 or another 4xx means no file, a 5xx, a 429 or no answer means the file is unreachable, and a 200 that returns an HTML page is not a valid robots.txt. It parses the file the way RFC 9309 describes: User-agent groups (several lines may share one set of rules), merged duplicate groups, the * fallback, Allow and Disallow with * and $ wildcards, comments, longest match with ties going to allow, and Sitemap lines. It then asks whether Googlebot may fetch the home page path / and, if you entered an inner page, that page's path.

• Pass: the file exists and Googlebot may crawl / and the page you entered.
• Fail: the file blocks Googlebot from / (in practice the whole site) or from the page you entered, or robots.txt is unreachable.
• Warning: no file was found (a 4xx), or the file is an HTML page.

AI crawler access (high priority). The same parser runs for 10 crawler tokens: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Claude-User, Googlebot and Bingbot (labeled AI search), and GPTBot, ClaudeBot and CCBot (labeled AI training), plus Google-Extended and Applebot-Extended, which are labeled as AI training opt-out tokens because they are not crawlers. It also sends one request that identifies as GPTBot to see whether a firewall refuses it with 401, 403, 429 or 503; since the request does not come from OpenAI's addresses, a refusal is a warning, not proof. The check fails if any search token is blocked or robots.txt is unreachable; blocked training tokens are listed but treated as your choice.

XML sitemap (medium priority). It tries every Sitemap: line in your robots.txt, then /sitemap.xml and /sitemap_index.xml (up to four addresses), and passes on the first that answers 200 with a <urlset> or <sitemapindex>, showing the number of URLs.

Indexing allowed (high priority). It looks for noindex in a robots or googlebot meta tag on the page and in the X-Robots-Tag response header. Either one fails the check, because it keeps the page out of results whatever robots.txt says.

What it does not test

• One page per run. Googlebot is tested for / and the page you enter. A rule like Disallow: /blog/ is flagged only if you enter a page under /blog/. Read the file for section-level rules, or test specific URLs in Search Console's URL Inspection.
• Rules for other hosts. Only the robots.txt of the host your page ends on is read.

Reading the result

The report gives an A+ to F grade for all 38 checks, weighted by priority (high 10, medium 5, low 2; warnings earn half), a "Fix these first" list and a shareable link. Each check also shows what share of audited sites pass it.

How to fix every problem the checker reports

robots.txt blocks Google from the whole site

1. Open the live file and find the group that applies to Googlebot: a User-agent: Googlebot group if there is one, otherwise User-agent: *.
2. Look for Disallow: / or a pattern like Disallow: /* in that group. Remove it, or replace it with the specific paths you meant to block.
3. Find where the file comes from before editing: a static file, a CMS setting, an SEO plugin, a CDN rule or a deploy script. In Shopify, edit the robots.txt.liquid theme template.
4. Load https://yourdomain.com/robots.txt in a private window to confirm the change is live, then rerun the robots.txt checker.
5. In Search Console, open the robots.txt report and choose Request a recrawl for the file. Google caches it for up to 24 hours otherwise.
6. Check the Page indexing report over the following days for "Blocked by robots.txt" counts dropping.

robots.txt is missing, unreachable or an HTML page

The message tells you which case you have; confirm the status code with:

curl -I https://yourdomain.com/robots.txt

• 404: harmless, but publish a simple file anyway so you can list your sitemap. Use the "allow everything" example above.
• 200 with Content-Type: text/html: your app returns a web page for unknown URLs. Serve a real text file before the app's catch-all route.
• 403: a security rule blocks the file. Allow /robots.txt for everyone.
• 5xx or timeout: fix this first. Google stops crawling for 12 hours each time it sees it, and RFC 9309 tells crawlers to assume the whole site is disallowed.

robots.txt blocks AI search crawlers

Find the group each blocked token uses (its own, or *) and decide your policy per token. Allow OAI-SearchBot, Claude-SearchBot and PerplexityBot if you want to be cited in ChatGPT, Claude and Perplexity answers; block training tokens separately if you wish. Remember that a named group does not inherit the * rules. If the firewall test gave a warning instead, the fix is in your CDN or WAF settings, not in robots.txt; see the AI crawlers guide.

No XML sitemap was found

1. Generate a sitemap of your canonical, indexable URLs. Most CMSs do it: WordPress core serves /wp-sitemap.xml, and Yoast SEO and Rank Math serve /sitemap_index.xml; in Next.js, add app/sitemap.ts.
2. Add the absolute URL to robots.txt: Sitemap: https://www.example.com/sitemap.xml.
3. Submit it in Google Search Console and Bing Webmaster Tools.
4. Keep it under Google's limits of 50,000 URLs or 50 MB uncompressed per file (Build and submit a sitemap); split larger sites with a sitemap index.

A noindex tag or X-Robots-Tag header blocks indexing

1. View the page source and search for noindex. If it is in a meta tag, find the setting that adds it: an SEO plugin's per-page option, a theme setting or a staging flag. In WordPress, check Settings › Reading › Search engine visibility, which is often left on after a launch.
2. If the source is clean, check the headers with curl -I https://yourdomain.com/. An X-Robots-Tag: noindex header usually comes from the server, a CDN rule or a hosting "staging" mode.
3. Remove it, then request indexing for the home page in Search Console's URL Inspection.

Worked example (hypothetical)

The following is a hypothetical example, not real data. Oakridge Dental, an invented clinic, relaunches its site and loses most of its Google traffic within two weeks. The robots.txt checker reports:

The file contains only User-agent: * and Disallow: /, copied from the staging server by the deploy script. The fix took ten minutes: the team replaced the file with the "allow everything" example plus the sitemap line, excluded robots.txt from the staging sync, requested a recrawl in Search Console and reran the checker, which passed. Because the file also blocked Bingbot and the AI search crawlers, the same fix restored the clinic's eligibility for Bing, Copilot and AI answers.

A 15-minute robots.txt checklist

1. Run the free robots.txt checker and note every failing check.
2. Confirm /robots.txt answers 200 as plain text, not 5xx or HTML.
3. Read every group: which crawler it names, and whether it accidentally blocks / or key sections.
4. Repeat important rules in every named group; they do not inherit *.
5. List your sitemap with an absolute URL.
6. Make sure nothing you want out of search relies on robots.txt instead of noindex.
7. After changes, request a recrawl in Search Console and recheck.

How Searvia helps

The free robots.txt checker tests your file for Googlebot, 9 other AI and search crawlers and 2 AI training opt-out tokens, finds your sitemap and checks the noindex signals in one shareable report. For the rest of the site, the site audit in a Searvia account crawls your pages the way a search engine does, respects your robots.txt, and flags noindex pages in the sitemap, broken pages and redirects every week, so a bad deploy is caught within days instead of after a traffic drop. Connect Search Console to see "Blocked by robots.txt" and indexing trends next to the audit, and read our technical SEO audit guide for where robots.txt fits in the full checklist.