GPTBot vs OAI-SearchBot vs ChatGPT-User: What Each OpenAI Crawler Does
By Seth Black · Published
OpenAI runs three crawlers that matter to a website, and each one does a different job. GPTBot collects pages that may be used to train OpenAI’s models, OAI-SearchBot builds the index that ChatGPT search cites from, and ChatGPT-User fetches a page because a person asked ChatGPT to read it. If you want to show up in ChatGPT answers, allow OAI-SearchBot and ChatGPT-User; whether you allow GPTBot is a separate, editorial decision about training.
What does each OpenAI crawler do?
OpenAI documents its user agents at developers.openai.com/api/docs/bots (read on 31 October 2026). The short version:
| User agent | Job | Obeys robots.txt? | IP list |
|---|---|---|---|
GPTBot |
Crawls content that may be used to train generative AI foundation models | Yes | openai.com/gptbot.json |
OAI-SearchBot |
Surfaces websites in ChatGPT’s search features | Yes | openai.com/searchbot.json |
ChatGPT-User |
Visits a page when a user, or a Custom GPT, asks ChatGPT to | “robots.txt rules may not apply” | openai.com/chatgpt-user.json |
The full strings start with a normal Mozilla/5.0 browser prefix and then carry the token, for example compatible; GPTBot/1.4; +https://openai.com/gptbot and compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot. ChatGPT-User identifies as compatible; ChatGPT-User/1.0; +https://openai.com/bot. The version numbers change, so match on the token, not the whole string.
OpenAI also lists a fourth agent, OAI-AdsBot, which only visits landing pages submitted as ChatGPT ads to check their safety. If you do not advertise on ChatGPT, you will not see it.
Are the three settings independent?
Yes. OpenAI’s page says “Each setting is independent of the others.” Blocking GPTBot does not remove you from ChatGPT search, and allowing OAI-SearchBot does not opt you into training. That independence is the whole reason there are three tokens.
Two details from the same page are worth knowing:
- A site that disallows OAI-SearchBot is excluded from ChatGPT search answers, but OpenAI says it may still appear as a navigational link.
- After a robots.txt change, OpenAI says it can take about 24 hours for its search systems to adjust.
Does ChatGPT-User follow robots.txt?
Not reliably, and OpenAI says so. Its documentation describes ChatGPT-User as handling user-initiated actions and states: “Because these actions are initiated by a user, robots.txt rules may not apply.” It also says “ChatGPT-User is not used to determine whether content may appear in Search.”
So a Disallow for ChatGPT-User is a request that may be ignored when a person pastes your URL into ChatGPT. If you truly need to stop those fetches, it has to happen at the firewall or CDN, matched against the published IP list. For most sites that is a bad idea: blocking a user-initiated fetch breaks the request of a person who was already interested in your page.
How often do they actually visit?
These are fetches on this site, blackseoanalyzer.com, from our nginx logs:
| User agent | May 2026 | Sep 2026 | Oct 2026 |
|---|---|---|---|
| GPTBot | 1 | 1,674 | 2,126 |
| OAI-SearchBot | 216 | 1,626 | 1,159 |
| ChatGPT-User | 1,516 | 1,153 | 1,012 |
GPTBot went from a single request in May to more than two thousand in October. ChatGPT-User was the steadiest of the three, which makes sense: it moves with how often people paste our links into ChatGPT, not with OpenAI’s crawl schedule. In the last 90 days, the pages fetched on behalf of a user (ChatGPT-User plus the Perplexity and Claude equivalents) were overwhelmingly the homepage, then the CLI documentation, then one old blog post.
Keep the scale honest, though. In twelve months ChatGPT sent this site about 60 human visitors. Google sent 354 unique humans in October alone. AI search sends fetches far more than it sends people. The full breakdown is in which AI crawlers hit a website in 2026.
Which OpenAI crawlers should you allow?
For most sites that want to be found:
# Appear in ChatGPT search and let shared links work
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
# Training: your call. This opts out.
User-agent: GPTBot
Disallow: /
If you are happy for your content to train models, change the last group to Allow: / or delete it. A site with no robots.txt rules for these tokens allows all three by default.
The common mistake goes the other way. Someone copies a “block AI” snippet that disallows GPTBot, OAI-SearchBot and ChatGPT-User together, meaning only to stop training. The result is that the site disappears from ChatGPT search and every link a user shares with ChatGPT fails. The AI crawler robots.txt guide covers the same split for every vendor.
How do you verify a request really came from OpenAI?
User agents are trivial to fake. OpenAI publishes the IP ranges for each crawler as JSON. Check the source IP of a suspicious request against the matching list:
# fetch the GPTBot ranges and list the prefixes
curl -s https://openai.com/gptbot.json | jq -r '.prefixes[] | .ipv4Prefix // .ipv6Prefix'
# then count GPTBot requests per IP in your logs
grep 'GPTBot' access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head
An IP outside the published ranges that claims to be GPTBot is somebody else’s scraper. Treat it as you would any other unverified bot.
What else does ChatGPT need besides access?
Access is the first gate, not the last. None of these crawlers runs JavaScript the way a browser does, so a page that ships an empty app shell is invisible to them even when robots.txt allows everything. Can AI crawlers see JavaScript content explains the test, and how to check whether ChatGPT can read your website walks through it step by step.
bseoa’s AI-readiness checks read your robots.txt for each of these tokens and report the three cases separately: a blocked retrieval crawler as Let AI Search Crawlers Read Your Site So It Can Be Cited, a blocked user-initiated fetcher as Allow User-Initiated Fetches So Shared Links Work, and a blocked training crawler as Understand What Blocking an AI Training Crawler Does. To test a single rule right now, paste your robots.txt and a user agent into the free robots.txt tester.
For how all of this fits next to classic search, see GEO vs SEO.
-Sethers