urldiagnostics.com

OpenAI crawler guide

GPTBot vs OAI-SearchBot vs ChatGPT-User

These three names are not interchangeable. OpenAI documents GPTBot for potential model-training collection, OAI-SearchBot for ChatGPT search, and ChatGPT-User for certain user-triggered visits.

Reviewed against primary sources ·

At a glance

Allowing any OpenAI user agent removes one access restriction for its documented purpose. It does not promise that a page will be crawled, selected, cited, ranked, included in search, or used for training.

The short comparison

User agentDocumented jobAutomatic crawl?Use this control for
OAI-SearchBotSurface websites in ChatGPT search results.YesChatGPT search opt-in or opt-out policy.
GPTBotCrawl content that may be used to train and improve OpenAI foundation models.YesPotential training-use collection policy.
ChatGPT-UserVisit pages for certain actions initiated by ChatGPT or Custom GPT users.NoUser-triggered retrieval, not Search inclusion.

OpenAI describes the OAI-SearchBot and GPTBot settings as independent. You may allow search crawling while disallowing training-oriented crawling. OpenAI also notes that, when both are allowed, it may use one crawl’s results for both purposes to avoid duplicate crawling; that does not make the policy controls the same.

OAI-SearchBot: ChatGPT search

OpenAI says OAI-SearchBot is used to surface websites in ChatGPT search features. Its documentation says sites opted out of this agent will not be shown in ChatGPT search answers, although navigational links may still appear. Use this agent—not GPTBot or ChatGPT-User—when setting the robots.txt policy for automatic ChatGPT search crawling.

GPTBot: potential foundation-model training use

OpenAI says GPTBot crawls content that may be used to make its generative AI foundation models more useful and safe. Disallowing GPTBot indicates that site content should not be used in training those models. “May be used” is the important wording: an allowed crawl is permission, not evidence that a page was fetched or entered a dataset.

ChatGPT-User: a user-triggered visit

OpenAI says ChatGPT-User may visit a page when a person asks ChatGPT or a Custom GPT a question, and that it is not used for automatic web crawling or to decide Search inclusion. Because these visits are user-initiated, OpenAI explicitly cautions that robots.txt rules may not apply. If content must not be public, enforce authentication or authorization at the server; a robots file was never a lock, merely a politely typeset notice.

Three policy patterns

Allow ChatGPT search, block training-oriented crawling

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Block ChatGPT search, allow GPTBot

User-agent: OAI-SearchBot
Disallow: /

User-agent: GPTBot
Allow: /

Allow public documentation, block account paths for both

User-agent: OAI-SearchBot
Allow: /docs/
Disallow: /account/

User-agent: GPTBot
Allow: /docs/
Disallow: /account/

Keep explicit groups for the agents whose purposes matter to your policy. Then test representative URLs—not just the root—and remember that private account content belongs behind authorization regardless of robots.txt.

Which rule should you change?

  • Your question is “Can this page appear in ChatGPT search answers?” Inspect OAI-SearchBot. OpenAI says robots updates can take about 24 hours to reach its search systems.
  • Your question is “May OpenAI collect this for potential model training?” Inspect GPTBot.
  • Your question is “Can ChatGPT open this when a user requests it?” Inspect delivery controls and ChatGPT-User, while accounting for OpenAI’s warning that robots rules may not apply.
  • Your question is “Will this earn citations?” None of these rules answers it. Access is necessary for some retrieval paths, but never a performance promise.

Primary sources

Checked on . Crawler policies change; the operator pages below remain the authority.