---
title: "ClaudeBot, Claude-User and Claude-SearchBot Explained"
description: "What Anthropic's three crawlers do, how often each hit a real site in 2026, and the robots.txt rules that let Claude cite you without training on your pages."
image: "https://www.blackseoanalyzer.com/static/images/black-seo-analyzer-og-image.png"
canonical: "https://www.blackseoanalyzer.com/en/blog/claudebot-claude-user-claude-searchbot"
language: "en"
---

# ClaudeBot, Claude-User and Claude-SearchBot: Anthropic's Three Crawlers Explained

By [Seth Black](https://www.blackseoanalyzer.com/en/about) · Published November 01, 2026

Anthropic runs three crawlers, and each one has its own robots.txt token. ClaudeBot collects web content that may be used to train Anthropic’s models, Claude-SearchBot indexes pages so Claude can find them when it searches, and Claude-User fetches a page when a person asks Claude about it. To be cited by Claude while opting out of training, allow Claude-SearchBot and Claude-User and disallow ClaudeBot.

## What does each Anthropic crawler do?

Anthropic documents its crawlers in a support article, [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (read on 31 October 2026). Here is what it says about each one:

| Token | Job | What blocking it does, per Anthropic |
| --- | --- | --- |
| `ClaudeBot` | Collects web content that may help train Anthropic’s models | Signals that your future material should be excluded from training datasets |
| `Claude-SearchBot` | Navigates the web “to improve search result quality for users” | Stops Anthropic indexing your content for search, which may reduce your visibility in answers |
| `Claude-User` | Fetches a page when a Claude user asks a question that needs it | “prevents our system from retrieving your content in response to a user query” |

The three are separate on purpose. A rule for ClaudeBot does not cover Claude-SearchBot or Claude-User. Each needs its own `User-agent` group, or none at all if you are happy to allow it.

## Do Anthropic’s crawlers obey robots.txt?

Anthropic says its bots “respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt.” That statement covers the bots as a group. Unlike OpenAI and Perplexity, Anthropic does not carve out an exception for its user-initiated fetcher, so the documented behavior is that Claude-User obeys robots.txt too. If you disallow Claude-User, a person who pastes your link into Claude gets told the page could not be read.

Three more details from the same article:

- **Crawl-delay is supported.** Anthropic honors the non-standard `Crawl-delay` extension, with `User-agent: ClaudeBot` and `Crawl-delay: 1` as its example. Google ignores this directive; Anthropic does not.
- **Every subdomain needs its own file.** robots.txt is per host, and Anthropic reminds you to repeat the rule on each subdomain you want to opt out.
- **Do not block by IP.** Anthropic warns that IP blocking “may not work correctly or persistently guarantee an opt-out,” because it can stop the crawler reading your robots.txt in the first place. It publishes its addresses at `claude.com/crawling/bots.json` for identifying traffic, not for firewall rules.

## How often do they actually visit?

From the nginx logs of this site, blackseoanalyzer.com:

| User agent | May 2026 | Sep 2026 | Oct 2026 |
| --- | --- | --- | --- |
| ClaudeBot | 63 | 786 | 2,995 |
| Claude-User | not in May’s top nine | 1,465 | 829 |
| Claude-SearchBot | not in May’s top nine | 567 | 820 |

May’s figures only cover the nine busiest AI agents that month, and neither Claude-User nor Claude-SearchBot made the list. ClaudeBot is the striking line: 63 fetches in May, nearly 3,000 in October, which made it the third busiest AI crawler on the site that month behind Meta and Amazonbot. Claude-User dropped from September to October while Claude-SearchBot rose, which is what you would expect if more of Claude’s answers came from its own index instead of a live fetch. That is a guess from the shape of the numbers, not something Anthropic has said.

Human visits are a different story. Claude sent this site 11 human referrals in twelve months. Fetches are not visitors, and [which AI crawlers hit a website in 2026](https://www.blackseoanalyzer.com/en/blog/which-ai-crawlers-hit-a-website-2026) puts these numbers next to every other vendor’s.

## What robots.txt rules should you use?

Allow citation, opt out of training:

```text
User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /
```

Allow everything but slow the training crawler down:

```text
User-agent: ClaudeBot
Crawl-delay: 5
Allow: /
```

Opt out of Anthropic entirely:

```text
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /
```

A group with several `User-agent` lines applies its rules to each of them. Before you ship the last one, be sure it is what you mean: it removes you from Claude’s search answers and breaks every link a user shares with Claude.

You may still find older advice that blocks `anthropic-ai` or `Claude-Web`. Anthropic’s current article does not list either token. The three above are the ones it documents today.

## How do you verify a request is really Anthropic?

Check the source IP against the published list:

```bash
# list Anthropic's published prefixes
curl -s https://claude.com/crawling/bots.json | jq -r '.prefixes[].ipv4Prefix'

# requests per Anthropic token in a combined-format log
grep -Eo 'ClaudeBot|Claude-SearchBot|Claude-User' access.log | sort | uniq -c
```

A request that says ClaudeBot but comes from an address outside that list is not Anthropic.

## What does Claude need once it can get in?

The same things every AI crawler needs. Claude-SearchBot and Claude-User read the HTML your server sends, so content built in the browser after JavaScript runs may never reach them; [can AI crawlers see JavaScript content](https://www.blackseoanalyzer.com/en/blog/can-ai-crawlers-see-javascript-content) shows how to check. And Google’s position that [SEO for AI is the same as SEO for search](https://www.blackseoanalyzer.com/en/blog/google-says-seo-for-ai-is-the-same-as-seo-for-search) holds for Claude too: crawlable HTML, a clear canonical and good structure do most of the work.

bseoa’s AI-readiness checks read your robots.txt for all three Anthropic tokens and report a blocked Claude-SearchBot as [Let AI Search Crawlers Read Your Site So It Can Be Cited](https://www.blackseoanalyzer.com/en/technical-seo/ai-readiness-analyzer/ai-search-crawler-blocked), a blocked Claude-User as [Allow User-Initiated Fetches So Shared Links Work](https://www.blackseoanalyzer.com/en/technical-seo/ai-readiness-analyzer/ai-user-action-crawler-blocked), and a blocked ClaudeBot as a training choice. You can also run bseoa inside Claude itself through the [connector](https://www.blackseoanalyzer.com/en/mcp), which lets Claude crawl a site and query the findings in the chat.

The bigger picture of how assistant search differs from classic search is on [GEO vs SEO](https://www.blackseoanalyzer.com/en/geo-vs-seo).

-Sethers

About the author

[Seth Black](https://www.blackseoanalyzer.com/en/about)

Seth is a software engineer and engineering leader who builds bseoa, a Rust-based technical SEO crawler with a GUI and CLI. He writes about the crawling, rendering and indexing problems he runs into on real sites.

[LinkedIn](https://www.linkedin.com/in/seth-black-tx/) · [GitHub](https://github.com/sethblack) · [YouTube](https://www.youtube.com/@SethBlack)

[Back to Blog](https://www.blackseoanalyzer.com/en/blog)

## Discover hundreds of SEO Issues in Seconds

Without Monthly Subscriptions

Comprehensive technical SEO analysis powered by ML and 16 specialized modules. Optional AI-powered insights from Claude, GPT-4, or Gemini. Get actionable insights in seconds, and never pay monthly fees again.

[Download Free Trial](https://www.blackseoanalyzer.com/en/free-trial)

I use AI to generate images for my posts and for general editing, updates, and ironically SEO purposes. I used to draw all of the images for my personal blog (taleas) myself, but as the volume of content I produce has increased, I've turned to AI tools to help create visuals that complement my writing. I go out of my way to generate images that look strange, and don't represent real people. If you ever want to chat about my use of AI, please reach out.

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://www.blackseoanalyzer.com/#organization","name":"Fiscus Technology, LLC","url":"https://www.blackseoanalyzer.com/","logo":{"@type":"ImageObject","url":"https://www.blackseoanalyzer.com/static/images/black-seo-analyzer.png"},"sameAs":["https://sethserver.com/","https://github.com/sethblack"],"founder":{"@id":"https://www.blackseoanalyzer.com/#seth"}},{"@type":"WebSite","@id":"https://www.blackseoanalyzer.com/#website","name":"bseoa","url":"https://www.blackseoanalyzer.com/","publisher":{"@id":"https://www.blackseoanalyzer.com/#organization"}},{"@type":"Person","@id":"https://www.blackseoanalyzer.com/#seth","name":"Seth Black","url":"https://www.blackseoanalyzer.com/about","jobTitle":"Creator of bseoa","sameAs":["https://www.linkedin.com/in/seth-black-tx/","https://github.com/sethblack","https://www.youtube.com/@SethBlack","https://sethserver.com/"]},{"@type":"BlogPosting","@id":"https://www.blackseoanalyzer.com/en/blog/claudebot-claude-user-claude-searchbot#article","headline":"ClaudeBot, Claude-User and Claude-SearchBot: Anthropic's Three Crawlers Explained","name":"ClaudeBot, Claude-User and Claude-SearchBot: Anthropic's Three Crawlers Explained","description":"What Anthropic's three crawlers do, how often each hit a real site in 2026, and the robots.txt rules that let Claude cite you without training on your pages.","url":"https://www.blackseoanalyzer.com/en/blog/claudebot-claude-user-claude-searchbot","datePublished":"2026-11-01T05:21:23","dateModified":"2026-11-01T05:21:23","author":{"@id":"https://www.blackseoanalyzer.com/#seth"},"publisher":{"@id":"https://www.blackseoanalyzer.com/#organization"},"image":{"@type":"ImageObject","url":"https://www.blackseoanalyzer.com/static/images/black-seo-analyzer-og-image.png","width":1200,"height":630},"mainEntityOfPage":{"@type":"WebPage","@id":"https://www.blackseoanalyzer.com/en/blog/claudebot-claude-user-claude-searchbot"},"isPartOf":{"@id":"https://www.blackseoanalyzer.com/#website"},"inLanguage":"en-US","isAccessibleForFree":true,"wordCount":977},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://www.blackseoanalyzer.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://www.blackseoanalyzer.com/en/blog"},{"@type":"ListItem","position":3,"name":"ClaudeBot, Claude-User and Claude-SearchBot: Anthropic's Three Crawlers Explained","item":"https://www.blackseoanalyzer.com/en/blog/claudebot-claude-user-claude-searchbot"}]}]}
```