AI crawler field guide
How to check whether AI crawlers can access your website
Crawler access is a technical permission question, not a visibility score. Check the live robots.txt file, evaluate the correct user-agent group against a specific URL, then verify that the server and any CDN or WAF permit the request.
Reviewed against primary sources ·
At a glance
An allowed result means the applicable robots.txt rules do not prohibit that fetch. It does not promise indexing, inclusion in an answer, a citation, a ranking, or model training.
What “can access” actually means
A useful audit separates three gates. robots.txt permission says whether the applicable crawler rules allow a URL. network access says whether DNS, the origin, CDN, and WAF accept the request.operator selection is the crawler operator’s later decision to fetch or use the page. A robots.txt checker can establish the first gate; it cannot manufacture the third.
| Result | What you know | What you do not know |
|---|---|---|
| Allowed | No matching robots.txt rule prohibits the tested URL for that agent. | Whether it will be fetched, indexed, cited, ranked, or used. |
| Blocked | A matching rule disallows the tested URL for that agent. | Whether old copies already exist or a user-triggered agent follows that rule. |
| Unknown | The policy could not be fetched or evaluated reliably. | Everything beyond the failed check; investigate rather than rounding up to “allowed.” |
A practical verification workflow
Choose one exact public URL
Test the URL you care about, not merely the homepage. Robots rules are path-sensitive. Fetch
/robots.txtfrom the same scheme, host, and port as that URL; a policy on one subdomain does not quietly govern all the others.Select the applicable user-agent group
Under Google’s documented REP interpretation, the most specific matching user-agent group wins. Repeated groups for that same specific agent are combined, while the specific group is not blended with
User-agent: *. This catches a common false reading: a global block does not override an explicit bot-specific allow when that bot selects the more specific group.Match the requested path
Compare every
AllowandDisallowpath in the selected group. Google applies the most specific matching path; if equally specific rules conflict, its implementation uses the least restrictive rule. Wildcards and the end anchor ($) count, so substring searching is not a parser.Verify the delivery layer
An allowed robots result can still meet a 403, authentication wall, rate limit, JavaScript challenge, or server error. Check origin and CDN logs for the documented user-agent, and where the operator publishes IP ranges, verify both the range and current operator guidance. A user-agent string alone is trivial to spoof.
Recheck after deployment
Fetch the public file again after caches clear and test representative paths. OpenAI and Perplexity both say robots changes may take about 24 hours to propagate through their systems, so an immediate product-side observation is not a sound policy test.
Robots.txt precedence example
User-agent: *
Disallow: /private/
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
Allow: /public-research/For OAI-SearchBot, the specific group allows the site; the wildcard group is not merged into it under Google’s documented group-selection model. For GPTBot, /public-research/report is allowed because the longer matching path beats Disallow: /, while /private/report remains blocked. Test the exact path: “the bot is allowed” is often too broad to be useful.
Which major agent should you test?
| Operator | Agent | Documented purpose | Policy question |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Automatic crawl for ChatGPT search results. | May search crawl this path? |
| OpenAI | GPTBot | Content that may be used to improve and train foundation models. | May training-oriented collection crawl this path? |
| OpenAI | ChatGPT-User | Certain user-triggered visits; not automatic web crawling. | Can a user-directed visit retrieve it? OpenAI says robots rules may not apply. |
| Anthropic | Claude-SearchBot | Search indexing and result quality. | May search-oriented crawling access this path? |
| Anthropic | ClaudeBot | Collection that could contribute to model training. | May training-oriented collection crawl this path? |
| Perplexity | PerplexityBot | Surface and link websites in Perplexity search; not foundation-model training. | May search crawl this path? |
Decision checklist
- You want search discovery but not training collection: evaluate search agents and training agents separately; do not use one blanket “AI bot” rule.
- You want to block a private area: use authentication. Robots.txt is a public crawl request, not access control.
- The checker says allowed but logs show no request: inspect WAF/CDN policy and operator IP guidance, then wait for policy propagation. Do not infer a ranking penalty.
- You added llms.txt: useful as a content guide, but it neither overrides robots.txt nor grants crawler access.
Primary sources
Checked on . Crawler policies change; the operator pages below remain the authority.
- OpenAI — Overview of OpenAI crawlers ↗Official roles, user agents, published IP lists, and robots.txt controls for OAI-SearchBot, GPTBot, and ChatGPT-User.
- Anthropic — Web crawler controls ↗Official descriptions and robots.txt guidance for ClaudeBot, Claude-SearchBot, and Claude-User.
- Perplexity — Perplexity crawlers ↗Official description of PerplexityBot and Perplexity-User, with user-agent and IP verification details.
- Google — How Google interprets robots.txt ↗A clear primary reference for group selection, path specificity, wildcards, anchors, and conflict handling.