ClaudeBot, Claude-User and Claude-SearchBot: Anthropic's Three Crawlers Explained
By Seth Black · Published
Anthropic runs three crawlers, and each one has its own robots.txt token. ClaudeBot collects web content that may be used to train Anthropic’s models, Claude-SearchBot indexes pages so Claude can find them when it searches, and Claude-User fetches a page when a person asks Claude about it. To be cited by Claude while opting out of training, allow Claude-SearchBot and Claude-User and disallow ClaudeBot.
What does each Anthropic crawler do?
Anthropic documents its crawlers in a support article, Does Anthropic crawl data from the web, and how can site owners block the crawler? (read on 31 October 2026). Here is what it says about each one:
| Token | Job | What blocking it does, per Anthropic |
|---|---|---|
ClaudeBot |
Collects web content that may help train Anthropic’s models | Signals that your future material should be excluded from training datasets |
Claude-SearchBot |
Navigates the web “to improve search result quality for users” | Stops Anthropic indexing your content for search, which may reduce your visibility in answers |
Claude-User |
Fetches a page when a Claude user asks a question that needs it | “prevents our system from retrieving your content in response to a user query” |
The three are separate on purpose. A rule for ClaudeBot does not cover Claude-SearchBot or Claude-User. Each needs its own User-agent group, or none at all if you are happy to allow it.
Do Anthropic’s crawlers obey robots.txt?
Anthropic says its bots “respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt.” That statement covers the bots as a group. Unlike OpenAI and Perplexity, Anthropic does not carve out an exception for its user-initiated fetcher, so the documented behavior is that Claude-User obeys robots.txt too. If you disallow Claude-User, a person who pastes your link into Claude gets told the page could not be read.
Three more details from the same article:
- Crawl-delay is supported. Anthropic honors the non-standard
Crawl-delayextension, withUser-agent: ClaudeBotandCrawl-delay: 1as its example. Google ignores this directive; Anthropic does not. - Every subdomain needs its own file. robots.txt is per host, and Anthropic reminds you to repeat the rule on each subdomain you want to opt out.
- Do not block by IP. Anthropic warns that IP blocking “may not work correctly or persistently guarantee an opt-out,” because it can stop the crawler reading your robots.txt in the first place. It publishes its addresses at
claude.com/crawling/bots.jsonfor identifying traffic, not for firewall rules.
How often do they actually visit?
From the nginx logs of this site, blackseoanalyzer.com:
| User agent | May 2026 | Sep 2026 | Oct 2026 |
|---|---|---|---|
| ClaudeBot | 63 | 786 | 2,995 |
| Claude-User | not in May’s top nine | 1,465 | 829 |
| Claude-SearchBot | not in May’s top nine | 567 | 820 |
May’s figures only cover the nine busiest AI agents that month, and neither Claude-User nor Claude-SearchBot made the list. ClaudeBot is the striking line: 63 fetches in May, nearly 3,000 in October, which made it the third busiest AI crawler on the site that month behind Meta and Amazonbot. Claude-User dropped from September to October while Claude-SearchBot rose, which is what you would expect if more of Claude’s answers came from its own index instead of a live fetch. That is a guess from the shape of the numbers, not something Anthropic has said.
Human visits are a different story. Claude sent this site 11 human referrals in twelve months. Fetches are not visitors, and which AI crawlers hit a website in 2026 puts these numbers next to every other vendor’s.
What robots.txt rules should you use?
Allow citation, opt out of training:
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: ClaudeBot
Disallow: /
Allow everything but slow the training crawler down:
User-agent: ClaudeBot
Crawl-delay: 5
Allow: /
Opt out of Anthropic entirely:
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /
A group with several User-agent lines applies its rules to each of them. Before you ship the last one, be sure it is what you mean: it removes you from Claude’s search answers and breaks every link a user shares with Claude.
You may still find older advice that blocks anthropic-ai or Claude-Web. Anthropic’s current article does not list either token. The three above are the ones it documents today.
How do you verify a request is really Anthropic?
Check the source IP against the published list:
# list Anthropic's published prefixes
curl -s https://claude.com/crawling/bots.json | jq -r '.prefixes[].ipv4Prefix'
# requests per Anthropic token in a combined-format log
grep -Eo 'ClaudeBot|Claude-SearchBot|Claude-User' access.log | sort | uniq -c
A request that says ClaudeBot but comes from an address outside that list is not Anthropic.
What does Claude need once it can get in?
The same things every AI crawler needs. Claude-SearchBot and Claude-User read the HTML your server sends, so content built in the browser after JavaScript runs may never reach them; can AI crawlers see JavaScript content shows how to check. And Google’s position that SEO for AI is the same as SEO for search holds for Claude too: crawlable HTML, a clear canonical and good structure do most of the work.
bseoa’s AI-readiness checks read your robots.txt for all three Anthropic tokens and report a blocked Claude-SearchBot as Let AI Search Crawlers Read Your Site So It Can Be Cited, a blocked Claude-User as Allow User-Initiated Fetches So Shared Links Work, and a blocked ClaudeBot as a training choice. You can also run bseoa inside Claude itself through the connector, which lets Claude crawl a site and query the findings in the chat.
The bigger picture of how assistant search differs from classic search is on GEO vs SEO.
-Sethers