Which AI Crawlers Actually Hit a Website in 2026: 12 Months of Server Logs
By Seth Black · Published
On this site, blackseoanalyzer.com, sixteen AI user agents made about 25,000 requests in October 2026, led by Meta (4,878), Amazonbot (4,846), ClaudeBot (2,995) and GPTBot (2,126). In May, most of them barely showed up. Over the same twelve months, AI products sent about 85 human visitors in total, while Google sent 354 unique humans in October alone: AI search fetches a lot and refers very little.
Which AI crawlers hit this site, and how often?
These are fetches from our nginx logs, counted by user agent token. May is included for contrast. In the spring, every agent except Meta, Amazonbot and ChatGPT-User stayed under a few hundred hits a month; May’s figures cover only the nine busiest agents that month, so a blank cell means the agent was not among them, not that it made zero requests.
| User agent | Operator | May 2026 | Sep 2026 | Oct 2026 |
|---|---|---|---|---|
| Meta agents | Meta | 2,348 | 3,136 | 4,878 |
Amazonbot |
Amazon | 2,229 | 3,158 | 4,846 |
ClaudeBot |
Anthropic | 63 | 786 | 2,995 |
GPTBot |
OpenAI | 1 | 1,674 | 2,126 |
Applebot |
Apple | 230 | 881 | 1,295 |
Bytespider |
ByteDance | 386 | 3,494 | 1,266 |
PerplexityBot |
Perplexity | 191 | 1,736 | 1,182 |
OAI-SearchBot |
OpenAI | 216 | 1,626 | 1,159 |
ChatGPT-User |
OpenAI | 1,516 | 1,153 | 1,012 |
Perplexity-User |
Perplexity | 994 | 866 | |
DuckAssistBot |
DuckDuckGo | 208 | 842 | |
Claude-User |
Anthropic | 1,465 | 829 | |
Claude-SearchBot |
Anthropic | 567 | 820 | |
CCBot |
Common Crawl | 89 | 567 | |
MistralAI |
Mistral | 164 | 330 | |
Google-Extended |
Google (see below) | 605 | 295 |
Our log classification groups Meta’s crawlers under one label, so the Meta row does not say which of Meta’s agents made the requests. The Mistral row matches the MistralAI prefix, which is how Mistral’s documented agent, MistralAI-User, identifies itself.
Three things stand out. GPTBot went from one request in May to over 2,000 in October. ClaudeBot grew almost fifty-fold over the same period. And Bytespider jumped ninefold into first place in September, then fell by almost two thirds in October.
What is each crawler for, according to its operator?
I read each operator’s documentation on 31 October 2026. Where the docs and our logs disagree, both are below.
| User agent | Documented purpose | Obeys robots.txt, per the docs |
|---|---|---|
GPTBot |
Training data for OpenAI models (OpenAI) | Yes |
OAI-SearchBot |
ChatGPT search index | Yes |
ChatGPT-User |
Fetches a page a user asked about | “robots.txt rules may not apply” |
ClaudeBot |
Training data for Anthropic models (Anthropic) | Yes, with Crawl-delay |
Claude-SearchBot |
Claude’s search index | Yes |
Claude-User |
Fetches a page a Claude user asked about | Yes, per Anthropic’s group statement |
PerplexityBot |
Perplexity search index, not training (Perplexity) | Yes |
Perplexity-User |
Fetches a page a user asked about | “generally ignores robots.txt rules” |
Amazonbot |
Improves Amazon products; “may be used to train Amazon AI models” (Amazon) | Yes, no Crawl-delay |
Applebot |
Spotlight, Siri and Safari search; may train Apple foundation models (Apple) | Yes, no Crawl-delay |
meta-externalagent |
Training foundation models and indexing (Meta) | Yes |
meta-externalfetcher |
Fetches links a user requested | “may bypass robots.txt rules” |
DuckAssistBot |
Real-time fetches for DuckDuckGo’s AI answers, not training (DuckDuckGo) | Yes, changes take 72 hours |
CCBot |
Common Crawl’s open web archive (Common Crawl) | Yes |
MistralAI-User |
Fetches pages for Vibe users, “not used for crawling the web in any automatic fashion” (Mistral) | Token governs which sites it may fetch |
Bytespider |
ByteDance crawler | No ByteDance documentation found |
Meta also documents meta-webindexer for Meta AI search and facebookexternalhit for link previews. Amazon documents Amzn-SearchBot and Amzn-User alongside Amazonbot; neither showed up as its own line in our counts.
Where do the logs and the docs disagree?
Google-Extended. Google says “Google-Extended doesn’t have a separate HTTP request user agent string” (Google). It is a robots.txt token that controls whether content Google already crawled may be used for Gemini training and grounding, and it does not affect Search. Yet 605 requests in September and 295 in October carried that string. By Google’s own documentation, those requests did not come from Google. Treat them as unverified.
Bytespider. I could not find a crawler page published by ByteDance. Third-party reports, including Fortune in October 2024, say Bytespider has ignored robots.txt; others say ByteDance states it honors the standard. I cannot verify either from a primary source, so the honest answer is that its behavior is undocumented. If it matters to you, block it at the firewall rather than trusting robots.txt.
Mistral. Mistral describes its agent as user-initiated only. Our logs show 330 requests in October, which, if Mistral’s description is right, means 330 user-triggered fetches, not a crawl.
Which pages did assistants fetch for users?
ChatGPT-User, Perplexity-User and Claude-User fetch a page because a person’s question needed it, which makes them the nearest thing to a citation log. In the last 90 days those three fetched the homepage 1,537 times, then /en/cli-documentation 28 times and an older Semrush comparison post 25 times. Everything else on the list was scanner noise: requests for secrets files and paths like /fetch and /proxy, sent through assistant fetchers. How to see which pages ChatGPT and Perplexity cite has the queries.
How many humans did AI send?
Human referrals from AI products over twelve months, excluding sign-ins to our connector:
| Source | Human visitors, 12 months |
|---|---|
| ChatGPT | about 60 |
| Perplexity | 12 |
| Claude | 11 |
| Gemini | 1 |
| Kagi | 1 |
| Google organic, October 2026 alone | 354 unique |
That is the honest frame for generative engine optimization in 2026. Being readable by AI crawlers is cheap and worth doing, because the fetch volume shows the assistants are reading. It is not yet a traffic channel on the scale of search. Google says SEO for AI is the same as SEO for search, and for most sites that is the right priority order.
How do you count AI crawlers on your own site?
On a combined-format access log:
grep -Eio 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Amazonbot|Applebot|Bytespider|meta-external[a-z]+|DuckAssistBot|CCBot|MistralAI-User|Google-Extended' access.log \
| sort | uniq -c | sort -rn
Then decide, per token, what you want. The AI crawler robots.txt guide has the rules, and the free robots.txt tester checks a rule against any user agent before you ship it. bseoa’s AI-readiness checks read your robots.txt for thirteen AI tokens, including all three OpenAI and Anthropic agents, both Perplexity agents, Google-Extended, Applebot-Extended, CCBot, Bytespider and meta-externalagent, and flag the costly cases: every AI crawler blocked and every retrieval crawler blocked.
For what changes and what stays the same when the searcher is an assistant, see GEO vs SEO.
-Sethers