---
title: "Which AI Crawlers Hit a Website in 2026: Server Log Data"
description: "Sixteen AI user agents, counted in a real site's nginx logs: Meta, Amazonbot, ClaudeBot, GPTBot and more, what each is for, and how few humans AI sent back."
image: "https://www.blackseoanalyzer.com/static/images/black-seo-analyzer-og-image.png"
canonical: "https://www.blackseoanalyzer.com/en/blog/which-ai-crawlers-hit-a-website-2026"
language: "en"
---

# Which AI Crawlers Actually Hit a Website in 2026: 12 Months of Server Logs

By [Seth Black](https://www.blackseoanalyzer.com/en/about) · Published November 01, 2026

On this site, blackseoanalyzer.com, sixteen AI user agents made about 25,000 requests in October 2026, led by Meta (4,878), Amazonbot (4,846), ClaudeBot (2,995) and GPTBot (2,126). In May, most of them barely showed up. Over the same twelve months, AI products sent about 85 human visitors in total, while Google sent 354 unique humans in October alone: AI search fetches a lot and refers very little.

## Which AI crawlers hit this site, and how often?

These are fetches from our nginx logs, counted by user agent token. May is included for contrast. In the spring, every agent except Meta, Amazonbot and ChatGPT-User stayed under a few hundred hits a month; May’s figures cover only the nine busiest agents that month, so a blank cell means the agent was not among them, not that it made zero requests.

| User agent | Operator | May 2026 | Sep 2026 | Oct 2026 |
| --- | --- | --- | --- | --- |
| Meta agents | Meta | 2,348 | 3,136 | 4,878 |
| `Amazonbot` | Amazon | 2,229 | 3,158 | 4,846 |
| `ClaudeBot` | Anthropic | 63 | 786 | 2,995 |
| `GPTBot` | OpenAI | 1 | 1,674 | 2,126 |
| `Applebot` | Apple | 230 | 881 | 1,295 |
| `Bytespider` | ByteDance | 386 | 3,494 | 1,266 |
| `PerplexityBot` | Perplexity | 191 | 1,736 | 1,182 |
| `OAI-SearchBot` | OpenAI | 216 | 1,626 | 1,159 |
| `ChatGPT-User` | OpenAI | 1,516 | 1,153 | 1,012 |
| `Perplexity-User` | Perplexity | | 994 | 866 |
| `DuckAssistBot` | DuckDuckGo | | 208 | 842 |
| `Claude-User` | Anthropic | | 1,465 | 829 |
| `Claude-SearchBot` | Anthropic | | 567 | 820 |
| `CCBot` | Common Crawl | | 89 | 567 |
| `MistralAI` | Mistral | | 164 | 330 |
| `Google-Extended` | Google (see below) | | 605 | 295 |

Our log classification groups Meta’s crawlers under one label, so the Meta row does not say which of Meta’s agents made the requests. The Mistral row matches the `MistralAI` prefix, which is how Mistral’s documented agent, `MistralAI-User`, identifies itself.

Three things stand out. GPTBot went from one request in May to over 2,000 in October. ClaudeBot grew almost fifty-fold over the same period. And Bytespider jumped ninefold into first place in September, then fell by almost two thirds in October.

## What is each crawler for, according to its operator?

I read each operator’s documentation on 31 October 2026. Where the docs and our logs disagree, both are below.

| User agent | Documented purpose | Obeys robots.txt, per the docs |
| --- | --- | --- |
| `GPTBot` | Training data for OpenAI models ([OpenAI](https://developers.openai.com/api/docs/bots)) | Yes |
| `OAI-SearchBot` | ChatGPT search index | Yes |
| `ChatGPT-User` | Fetches a page a user asked about | “robots.txt rules may not apply” |
| `ClaudeBot` | Training data for Anthropic models ([Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)) | Yes, with `Crawl-delay` |
| `Claude-SearchBot` | Claude’s search index | Yes |
| `Claude-User` | Fetches a page a Claude user asked about | Yes, per Anthropic’s group statement |
| `PerplexityBot` | Perplexity search index, not training ([Perplexity](https://docs.perplexity.ai/guides/bots)) | Yes |
| `Perplexity-User` | Fetches a page a user asked about | “generally ignores robots.txt rules” |
| `Amazonbot` | Improves Amazon products; “may be used to train Amazon AI models” ([Amazon](https://developer.amazon.com/amazonbot)) | Yes, no `Crawl-delay` |
| `Applebot` | Spotlight, Siri and Safari search; may train Apple foundation models ([Apple](https://support.apple.com/en-us/119829)) | Yes, no `Crawl-delay` |
| `meta-externalagent` | Training foundation models and indexing ([Meta](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)) | Yes |
| `meta-externalfetcher` | Fetches links a user requested | “may bypass robots.txt rules” |
| `DuckAssistBot` | Real-time fetches for DuckDuckGo’s AI answers, not training ([DuckDuckGo](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot)) | Yes, changes take 72 hours |
| `CCBot` | Common Crawl’s open web archive ([Common Crawl](https://commoncrawl.org/ccbot)) | Yes |
| `MistralAI-User` | Fetches pages for Vibe users, “not used for crawling the web in any automatic fashion” ([Mistral](https://docs.mistral.ai/robots)) | Token governs which sites it may fetch |
| `Bytespider` | ByteDance crawler | No ByteDance documentation found |

Meta also documents `meta-webindexer` for Meta AI search and `facebookexternalhit` for link previews. Amazon documents `Amzn-SearchBot` and `Amzn-User` alongside Amazonbot; neither showed up as its own line in our counts.

## Where do the logs and the docs disagree?

**Google-Extended.** Google says “Google-Extended doesn’t have a separate HTTP request user agent string” ([Google](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)). It is a robots.txt token that controls whether content Google already crawled may be used for Gemini training and grounding, and it does not affect Search. Yet 605 requests in September and 295 in October carried that string. By Google’s own documentation, those requests did not come from Google. Treat them as unverified.

**Bytespider.** I could not find a crawler page published by ByteDance. Third-party reports, including Fortune in October 2024, say Bytespider has ignored robots.txt; others say ByteDance states it honors the standard. I cannot verify either from a primary source, so the honest answer is that its behavior is undocumented. If it matters to you, block it at the firewall rather than trusting robots.txt.

**Mistral.** Mistral describes its agent as user-initiated only. Our logs show 330 requests in October, which, if Mistral’s description is right, means 330 user-triggered fetches, not a crawl.

## Which pages did assistants fetch for users?

ChatGPT-User, Perplexity-User and Claude-User fetch a page because a person’s question needed it, which makes them the nearest thing to a citation log. In the last 90 days those three fetched the homepage 1,537 times, then `/en/cli-documentation` 28 times and an older Semrush comparison post 25 times. Everything else on the list was scanner noise: requests for secrets files and paths like `/fetch` and `/proxy`, sent through assistant fetchers. [How to see which pages ChatGPT and Perplexity cite](https://www.blackseoanalyzer.com/en/blog/which-pages-does-chatgpt-cite-server-logs) has the queries.

## How many humans did AI send?

Human referrals from AI products over twelve months, excluding sign-ins to our connector:

| Source | Human visitors, 12 months |
| --- | --- |
| ChatGPT | about 60 |
| Perplexity | 12 |
| Claude | 11 |
| Gemini | 1 |
| Kagi | 1 |
| Google organic, October 2026 alone | 354 unique |

That is the honest frame for generative engine optimization in 2026. Being readable by AI crawlers is cheap and worth doing, because the fetch volume shows the assistants are reading. It is not yet a traffic channel on the scale of search. [Google says SEO for AI is the same as SEO for search](https://www.blackseoanalyzer.com/en/blog/google-says-seo-for-ai-is-the-same-as-seo-for-search), and for most sites that is the right priority order.

## How do you count AI crawlers on your own site?

On a combined-format access log:

```bash
grep -Eio 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Amazonbot|Applebot|Bytespider|meta-external[a-z]+|DuckAssistBot|CCBot|MistralAI-User|Google-Extended' access.log \
  | sort | uniq -c | sort -rn
```

Then decide, per token, what you want. [The AI crawler robots.txt guide](https://www.blackseoanalyzer.com/en/blog/ai-crawler-robots-txt-guide) has the rules, and the free [robots.txt tester](https://www.blackseoanalyzer.com/en/tools/robots-txt-tester) checks a rule against any user agent before you ship it. bseoa’s AI-readiness checks read your robots.txt for thirteen AI tokens, including all three OpenAI and Anthropic agents, both Perplexity agents, Google-Extended, Applebot-Extended, CCBot, Bytespider and meta-externalagent, and flag the costly cases: [every AI crawler blocked](https://www.blackseoanalyzer.com/en/technical-seo/ai-readiness-analyzer/all-ai-crawlers-blocked) and [every retrieval crawler blocked](https://www.blackseoanalyzer.com/en/technical-seo/ai-readiness-analyzer/no-ai-retrieval-access).

For what changes and what stays the same when the searcher is an assistant, see [GEO vs SEO](https://www.blackseoanalyzer.com/en/geo-vs-seo).

-Sethers

About the author

[Seth Black](https://www.blackseoanalyzer.com/en/about)

Seth is a software engineer and engineering leader who builds bseoa, a Rust-based technical SEO crawler with a GUI and CLI. He writes about the crawling, rendering and indexing problems he runs into on real sites.

[LinkedIn](https://www.linkedin.com/in/seth-black-tx/) · [GitHub](https://github.com/sethblack) · [YouTube](https://www.youtube.com/@SethBlack)

[Back to Blog](https://www.blackseoanalyzer.com/en/blog)

## Discover hundreds of SEO Issues in Seconds

Without Monthly Subscriptions

Comprehensive technical SEO analysis powered by ML and 16 specialized modules. Optional AI-powered insights from Claude, GPT-4, or Gemini. Get actionable insights in seconds, and never pay monthly fees again.

[Download Free Trial](https://www.blackseoanalyzer.com/en/free-trial)

I use AI to generate images for my posts and for general editing, updates, and ironically SEO purposes. I used to draw all of the images for my personal blog (taleas) myself, but as the volume of content I produce has increased, I've turned to AI tools to help create visuals that complement my writing. I go out of my way to generate images that look strange, and don't represent real people. If you ever want to chat about my use of AI, please reach out.

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://www.blackseoanalyzer.com/#organization","name":"Fiscus Technology, LLC","url":"https://www.blackseoanalyzer.com/","logo":{"@type":"ImageObject","url":"https://www.blackseoanalyzer.com/static/images/black-seo-analyzer.png"},"sameAs":["https://sethserver.com/","https://github.com/sethblack"],"founder":{"@id":"https://www.blackseoanalyzer.com/#seth"}},{"@type":"WebSite","@id":"https://www.blackseoanalyzer.com/#website","name":"bseoa","url":"https://www.blackseoanalyzer.com/","publisher":{"@id":"https://www.blackseoanalyzer.com/#organization"}},{"@type":"Person","@id":"https://www.blackseoanalyzer.com/#seth","name":"Seth Black","url":"https://www.blackseoanalyzer.com/about","jobTitle":"Creator of bseoa","sameAs":["https://www.linkedin.com/in/seth-black-tx/","https://github.com/sethblack","https://www.youtube.com/@SethBlack","https://sethserver.com/"]},{"@type":"BlogPosting","@id":"https://www.blackseoanalyzer.com/en/blog/which-ai-crawlers-hit-a-website-2026#article","headline":"Which AI Crawlers Actually Hit a Website in 2026: 12 Months of Server Logs","name":"Which AI Crawlers Actually Hit a Website in 2026: 12 Months of Server Logs","description":"Sixteen AI user agents, counted in a real site's nginx logs: Meta, Amazonbot, ClaudeBot, GPTBot and more, what each is for, and how few humans AI sent back.","url":"https://www.blackseoanalyzer.com/en/blog/which-ai-crawlers-hit-a-website-2026","datePublished":"2026-11-01T05:21:23","dateModified":"2026-11-01T05:21:23","author":{"@id":"https://www.blackseoanalyzer.com/#seth"},"publisher":{"@id":"https://www.blackseoanalyzer.com/#organization"},"image":{"@type":"ImageObject","url":"https://www.blackseoanalyzer.com/static/images/black-seo-analyzer-og-image.png","width":1200,"height":630},"mainEntityOfPage":{"@type":"WebPage","@id":"https://www.blackseoanalyzer.com/en/blog/which-ai-crawlers-hit-a-website-2026"},"isPartOf":{"@id":"https://www.blackseoanalyzer.com/#website"},"inLanguage":"en-US","isAccessibleForFree":true,"wordCount":1229},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://www.blackseoanalyzer.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://www.blackseoanalyzer.com/en/blog"},{"@type":"ListItem","position":3,"name":"Which AI Crawlers Actually Hit a Website in 2026: 12 Months of Server Logs","item":"https://www.blackseoanalyzer.com/en/blog/which-ai-crawlers-hit-a-website-2026"}]}]}
```