Robots.txt Tester
Find out who your robots.txt is really blocking
Robots.txt looks simple and behaves anything but: the longest matching rule wins, ties go to Allow, wildcards change everything, and each crawler picks one user-agent group and ignores the rest. This tool applies those rules exactly - for Googlebot, Bingbot, and the AI crawlers that decide whether your site appears in AI answers - and explains every verdict in plain English.
Click any verdict to see which rule decided it, and why.
How robots.txt precedence actually works
Three rules explain almost every surprising robots.txt result. First, a crawler follows the group - or combined groups - whose user-agent token most specifically matches its own, falling back to User-agent: *. Several groups naming the same token merge into one rule set, but a more specific match means the * group no longer applies at all - its rules are not inherited.
Second, within that group the most specific matching rule wins, measured by the length of the rule's path pattern. Allow: /admin/public/ beats Disallow: /admin/ for anything under /admin/public/ because it is the longer pattern - the order of lines in the file is irrelevant.
Third, when an Allow and a Disallow pattern are exactly as specific as each other, the tie goes to Allow. Add wildcards - * matches any run of characters and $ pins a pattern to the end of the URL - and you have a system that genuinely needs testing rather than squinting.
The AI crawler question
Robots.txt used to be a conversation with search engines. Now it also decides whether your content can appear in AI answers - and the two are different decisions. Blocking GPTBot keeps you out of OpenAI's training data, but it also removes a channel where potential customers increasingly ask for recommendations. There is no right answer, but it should be a decision, not an accident.
| Token | Operator | What it feeds |
|---|---|---|
| Googlebot | Google Search - the index itself. Blocking this stops Google crawling your pages; URLs it discovers elsewhere can still appear in Search as bare links (see "Robots.txt is not access control" below). | |
| Google-Extended | Gemini model training, plus grounding in Gemini Apps and Vertex AI's Grounding with Google Search. A control token, not a separate crawler - no effect on Search inclusion or ranking. | |
| Bingbot | Microsoft | Bing Search, which also feeds Copilot and other services that license Bing's index. |
| GPTBot | OpenAI | Model training for ChatGPT and successors. |
| OAI-SearchBot | OpenAI | ChatGPT's search feature - citations and links in answers, separate from training. |
| ClaudeBot | Anthropic | Model training for Claude. |
| Claude-SearchBot | Anthropic | Claude's web search index - whether you can appear when Claude searches the web for answers, separate from training. |
| PerplexityBot | Perplexity | Perplexity's answer-engine index, where sources are cited and linked. |
| CCBot | Common Crawl | The open Common Crawl dataset, used as training data by many AI labs. |
| Applebot-Extended | Apple | Apple Intelligence training. Like Google-Extended, a control token layered on the normal Applebot. |
| Meta-ExternalAgent | Meta | AI model training for Meta's Llama family. |
| Bytespider | ByteDance | Training data for ByteDance and TikTok AI features. Historically patchy at obeying robots.txt. |
| Amazonbot | Amazon | Alexa answers and Amazon AI services. |
The blunt trade-off: blocking training bots protects your content from becoming model weights; blocking answer-engine bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) removes you from cited, linked answers that send real visitors.
Mistakes we see constantly
Robots.txt is not access control
A disallowed URL can still appear in Google - as a bare link with no snippet - if other pages link to it. Worse, blocking a page also blocks crawlers from seeing its noindex tag. To keep a page out of the index, allow the crawl and use noindex; to keep it genuinely private, require authentication.
Groups do not inherit
Adding a friendly User-agent: Googlebot group silently detaches Googlebot from every rule in your * group. If the new group is empty apart from an Allow, you have just unblocked everything you thought was blocked. Each crawler follows exactly one rule set: the groups matching its most specific token, combined.
Blocked assets break rendering
Disallowing /assets/ or /js/ stops Google rendering your pages the way visitors see them, which can hurt rankings on its own. Crawlers need your CSS and JavaScript; there is almost never a good reason to block them.
Crawl-delay support is patchy
Google has never supported Crawl-delay - its crawl rate is governed by how fast your server responds, one more reason site speed matters. But Bing, Anthropic's bots and some other crawlers do honour it, each with their own interpretation, so it is not dead weight - just never rely on it for Google.
Check the rest of your search presence
Crawl access is step one; what searchers actually see is step two. Our SERP Snippet Preview shows how your titles and descriptions will render in Google before you publish, and the free RankRecon browser extension turns any live results page into a full competitor analysis - locally, with nothing leaving your machine.