Who Blocks AI Crawlers? robots.txt, llms.txt and JavaScript Across the Top 2,500 Sites (November 2026)
By Seth Black · Published
Corrected 4 November 2026. The first version of this post, published 4 November, had errors. It said blocking ChatGPT-User makes a pasted link fail, which OpenAI’s documentation does not support. It called 113 thin homepages “bot walls and consent screens” without evidence. It reported llms.txt adoption only as a share of sites that responded. It counted one homepage as thin because of a read limit in my script. It said the CSV flagged a condition it did not flag. It also described a cross-check as independent and complete when neither was true. The list of changes is at the end.
On 4 November 2026 I fetched robots.txt, /llms.txt, /llms-full.txt and the homepage of the 2,500 highest-ranked domains in the Tranco list with a plain HTTP client. 1,254 of those domains served an HTML homepage to that client, and 1,071 of the 1,254 published a robots.txt. Those 1,071 sites are the sample for the robots.txt figures below, and the 1,254 are the sample for llms.txt. The full per-site table is a downloadable CSV.
These denominators matter, because half of the list did not answer usefully. Many of those domains are CDNs, API hosts and ad servers, and some refused my client. Other published studies use other denominators, which is much of why their numbers differ. The comparison is below.
The short version:
- 32.4% of the 1,071 sites name at least one AI crawler in robots.txt, and 23.7% block at least one from the whole site.
- Named blocks are aimed at training more than at answers. 21.6% block a model-training crawler (GPTBot, ClaudeBot, Google-Extended, CCBot and the like). 9.4% block a search-index crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot). 8.7% name a fetcher that acts for a person in the chat (ChatGPT-User, Claude-User, Perplexity-User and others).
- Those user-fetcher rules may not do anything. OpenAI says robots.txt rules “may not apply” to ChatGPT-User, and Perplexity says its fetcher “generally ignores robots.txt rules”. Anthropic says Claude-User honours them.
- llms.txt: 194 domains serve a file that starts with a Markdown heading. That is 7.8% of the 2,500 domains, or 15.5% of the 1,254 that served a homepage.
- 14.5% of measurable homepages gave my client almost no text. I can show that the text was missing, but mostly not why.
How the data was collected
Every domain got four requests over HTTPS with a Mozilla/5.0 (compatible; bseoa-study/1.0; +https://www.blackseoanalyzer.com/) user agent: /robots.txt, /llms.txt, /llms-full.txt and /. Redirects were followed, nothing was rendered, and no site was crawled beyond those four URLs. A domain counts as a site when its homepage answered 200 with more than 500 bytes of HTML. That excludes hosts with no homepage and also real sites that refused my client. nytimes.com, which answered 403, is one of them.
robots.txt was parsed by user-agent group. A crawler is blocked when a group names it explicitly and contains Disallow: / with no Allow: /. It is partial when the group disallows some paths but not the root. Rules inherited from User-agent: * are reported separately, so “does not block GPTBot” below only means no group names it with a full disallow. The 23 crawler tokens checked come from OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, ByteDance, Common Crawl, Cohere, Diffbot, DuckDuckGo, Mistral, AI2, You.com and Huawei.
An llms.txt counts when /llms.txt returns 200, is not HTML, and starts with a Markdown heading. Another 33 sites return a non-HTML 200 at that path that does not start with one. They are excluded from the figures here and marked in the CSV’s llms_txt_has_h1 column. I did not validate the rest of each file against the format.
A homepage is thin when the visible text left after removing scripts, styles and <noscript> blocks is under 300 characters, or under 1,500 characters with a <noscript> block mentioning JavaScript. My client stopped reading at 2 MB. That limit hit 20 homepages, and it made one of them, cnn.com, look thin when the text probably sat past the cutoff. cnn.com is left out of the thin figures, so that denominator is 1,253.
The collection and analysis scripts are two short Python files. The whole run took six minutes.
Which crawlers get blocked
| Crawler | What it does | Sites naming it | Blocked from whole site | Blocked, % of 1,071 sites |
|---|---|---|---|---|
| CCBot | Common Crawl (training corpora) | 209 | 168 | 15.7% |
| GPTBot | OpenAI training | 251 | 167 | 15.6% |
| Bytespider | ByteDance | 189 | 162 | 15.1% |
| ClaudeBot | Anthropic training | 234 | 156 | 14.6% |
| Google-Extended | Gemini training opt-out | 213 | 136 | 12.7% |
| Diffbot | Diffbot | 141 | 134 | 12.5% |
| cohere-ai | Cohere | 151 | 129 | 12.0% |
| meta-externalagent | Meta training | 153 | 128 | 12.0% |
| Amazonbot | Amazon | 156 | 127 | 11.9% |
| Applebot-Extended | Apple Intelligence training | 155 | 121 | 11.3% |
| anthropic-ai | Anthropic (not in its current documentation) | 157 | 118 | 11.0% |
| PetalBot | Huawei | 126 | 117 | 10.9% |
| PerplexityBot | Perplexity search index | 186 | 97 | 9.1% |
| YouBot | You.com | 91 | 75 | 7.0% |
| ChatGPT-User | OpenAI, fetches for a person | 158 | 73 | 6.8% |
| Perplexity-User | Perplexity, fetches for a person | 102 | 61 | 5.7% |
| Claude-User | Anthropic, fetches for a person | 100 | 61 | 5.7% |
| Claude-SearchBot | Anthropic search index | 104 | 57 | 5.3% |
| DuckAssistBot | DuckDuckGo AI | 76 | 56 | 5.2% |
| ai2bot | AI2 | 60 | 53 | 4.9% |
| meta-externalfetcher | Meta, fetches for a person | 61 | 52 | 4.9% |
| OAI-SearchBot | OpenAI search index (ChatGPT search) | 133 | 49 | 4.6% |
| MistralAI-User | Mistral, fetches for a person | 58 | 46 | 4.3% |
Training crawlers are blocked about three times as often as OpenAI’s search crawler. GPTBot is blocked on 15.6% of sites and OAI-SearchBot on 4.6%. Of the 133 sites that name OAI-SearchBot, 33 explicitly allow it, 51 restrict part of the site and 49 shut it out. 123 sites block a training crawler without naming a full block on any search crawler or user fetcher. 119 block GPTBot without also blocking OAI-SearchBot, and 98 block ClaudeBot without also blocking Claude-User. What each token does is covered in GPTBot vs OAI-SearchBot vs ChatGPT-User and ClaudeBot, Claude-User and Claude-SearchBot.
An old token is still common. anthropic-ai is named by 157 sites and blocked by 118. Anthropic’s crawler article, as I read it on 4 November 2026, does not list it.
Long lists look copied. 47 sites block exactly one crawler. The next most common count is twelve (35 sites), then seven (20 sites), and 28 sites block twenty or more. Repeated counts suggest shared block lists, though I did not compare the files to confirm it. Most of the 28 are news and commerce sites, including amazon.com and its country sites, cnn.com, msn.com, theverge.com, cnet.com, zdnet.com, nbcnews.com, mashable.com, vox.com, ign.com, corriere.it, elmundo.es, marca.com, expressen.se and dagbladet.no.
The user-fetcher rules
93 sites block at least one fetcher that runs when a person asks an assistant to read a page. They include amazon.com, tiktok.com, yahoo.com, msn.com, cnn.com, bbc.com, espn.com, cnbc.com, wired.com, techcrunch.com, theverge.com, cnet.com, nbcnews.com, huffpost.com, investopedia.com, nypost.com, lemonde.fr, spiegel.de, bild.de, elpais.com, repubblica.it, aljazeera.com, dw.com, abc.net.au, figma.com, genius.com and statista.com.
Those rules state what the site wants. Whether they change anything depends on the vendor:
| Fetcher | What the vendor’s documentation says |
|---|---|
| ChatGPT-User | “Because these actions are initiated by a user, robots.txt rules may not apply.” |
| Perplexity-User | “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.” |
| Claude-User | Anthropic’s bots “respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt”. Disabling Claude-User “prevents our system from retrieving your content in response to a user query”. |
I did not test what any assistant does when it meets these rules. A site that wants these fetchers stopped needs to enforce that at the server or CDN and check the result in its own logs.
45 sites block all three OpenAI tokens (GPTBot, OAI-SearchBot and ChatGPT-User).
llms.txt adoption
| Sites | Share | |
|---|---|---|
| /llms.txt starts with a Markdown heading | 194 | 7.8% of 2,500 domains, 15.5% of 1,254 sites |
| /llms-full.txt is a non-HTML file of at least 100 bytes | 36 | 1.4% of 2,500, 2.9% of 1,254 |
| robots.txt mentions llms.txt | 27 | 2.5% of 1,071 |
| Has an llms.txt and blocks a training crawler | 27 |
The named adopters are mostly software and infrastructure companies: cloudflare.com, github.com, fastly.net, adobe.com, wordpress.org, stripe.com, shopify.com, sentry.io, dropbox.com, ubuntu.com, slack.com, hubspot.com, zendesk.com, okta.com, datadoghq.com, atlassian.com, digitalocean.com, mailchimp.com, zoho.com, supabase.co, auth0.com, cursor.sh, tailscale.com, grafana.com and coinbase.com. Of 30 news and media domains I checked from a fixed list, four have one: foxnews.com, businessinsider.com, weather.com and prnewswire.com.
By rank, as a share of sites in each band: 16.3% of the top 100 (49 sites), 22.7% of ranks 101 to 500, 20.8% of ranks 501 to 1,000 and 12.4% of ranks 1,001 to 2,500.
Whether any assistant reads the file is a separate question, and the evidence is thin. I made that case in llms.txt: does it matter?, and the format is in how to write an llms.txt.
Homepages with almost no text
182 of 1,253 homepages (14.5%) are thin by the definition above. 175 of those have under 300 characters of text, and the other 7 qualify through the <noscript> rule. Here is what I can and cannot say about them:
| Group | Sites | Examples |
|---|---|---|
Framework root or <noscript> JavaScript notice present |
70 | tiktok.com, chatgpt.com, twitch.tv, bsky.app, msn.com, naver.com, pinterest.com |
| No marker, HTML over 50 KB | 30 | google.com, youtube.com, instagram.com, bing.com, threads.com, temu.com |
| No marker, HTML 3 KB to 50 KB | 52 | reddit.com, linkedin.com |
| No marker, HTML under 3 KB | 30 | amazon.com, dzen.ru, example.com, dns.google, mega.nz |
The first group is probably client-side rendering. The other 112 could be client-side rendering, a challenge page, or a page that simply has little text. Some are not content sites at all. Only two of the 182 have a title that names a challenge or denial: linkedin.com (“Checking your browser - reCAPTCHA”) and lowes.com (“Access Denied”). Those two are the only ones I can call bot walls from this data.
The result reflects what my user agent received. A site may serve a real AI crawler something different, in either direction. Vercel’s published test found that the major AI crawlers it examined do not render JavaScript, so client-rendered text is a reasonable concern, but I did not test those crawlers here.
The share falls with rank: 32.7% of the top 100 (49 sites), 21.6% of ranks 101 to 500, 15.7% of 501 to 1,000 and 11.7% of 1,001 to 2,500.
Next.js markers appear on 205 of the 1,254 sites, and 11 of those 205 are thin. Seven sites set a noai or noimageai directive in a meta robots tag or X-Robots-Tag header. 49 sites disallow everything for every crawler.
How this compares with other studies
| Study | Sample | Finding | Denominator |
|---|---|---|---|
| This post | Tranco top 2,500 | GPTBot blocked 15.6%, any AI crawler 23.7% | 1,071 sites with robots.txt and a usable homepage |
| StackFox, Feb 2026 | 7,636 sites | GPTBot blocked 17% | Sites with robots.txt |
| Thunderbit, May 2026 | Tranco top 10,000 | Any AI crawler 20.3% | 7,248 analysable sites |
| BuzzStream | Top UK and US news sites | 79% block a training bot, 40% block ChatGPT-User | News sites only |
| This post | Tranco top 2,500 | llms.txt 7.8% or 15.5% | All 2,500, or the 1,254 with a homepage |
| Rankability, Sep 2026 | Tranco top 1,000 and top 10,000 | llms.txt or llms-full.txt 9.3% and 8.3% | Every sampled domain, reachable or not |
| SEOmator, Sep 2026 | Cloudflare Radar top 100,000 | llms.txt 8.6% or 11.4% | All domains, or the 75,353 that responded |
The robots.txt figures are in the same range as the others. Counted over every domain, the llms.txt figure also sits close to Rankability’s, so my 15.5% is high mainly because of its denominator. These studies use different lists, dates and rules, so the gaps are not trends.
What bseoa says about the same sites
I ran 13 of these domains through check_page, the single-page audit in bseoa, the product this site sells. It is a second parser, not an independent source. The counts below are what bseoa reports.
| Site | Training crawlers blocked | Search crawlers blocked | User fetchers blocked | Other AI-readiness findings |
|---|---|---|---|---|
| nytimes.com | all 13 known AI crawlers (reported as one finding) | 403 to the fetcher, no llms.txt | ||
| cnn.com | 5 | 3 | 3 | no retrieval access, no llms.txt, no question headings |
| bbc.com | 7 | 3 | 2 | no retrieval access, no llms.txt |
| theverge.com | 6 | 2 | 3 | no llms.txt, no question headings |
| wired.com | 6 | 2 | 2 | no llms.txt, no sameAs |
| spiegel.de | 6 | 2 | 2 | no llms.txt |
| investopedia.com | 6 | 2 | 2 | 403 to the fetcher, no llms.txt |
| techcrunch.com | 6 | 1 | no llms.txt | |
| github.com | 1 | |||
| cloudflare.com | no extractable structure | |||
| stripe.com | none | |||
| shopify.com | none | |||
| atlassian.com | none |
Every crawler bseoa named on these sites as blocked is also blocked in my data. My data shows more blocks on some sites because it checks tokens bseoa does not, such as anthropic-ai, mistralai-user and meta-externalfetcher. Thirteen sites is a spot check of the parser, not a validation of the dataset.
What this means if you run a site
- Decide per role, not per vendor. OpenAI says disallowing GPTBot signals that content should not be used for training. It also says sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers, though can still appear as navigational links.”
- Do not rely on robots.txt to stop user-initiated fetches. See the vendor table above, and check your logs.
- Check whether you still need
anthropic-ai. Anthropic’s current article lists ClaudeBot, Claude-User and Claude-SearchBot. - Check what a non-browser client sees.
curl -A "Mozilla/5.0" https://yoursite/ | sed 's/<[^>]*>//g' | wc -cgives a rough count, and it includes script contents, so a large number proves little. The ten-minute check goes further. - llms.txt is cheap and unproven. If your documentation build generates one, publish it. Fix a robots.txt that blocks crawlers you want first.
The AI search readiness checklist and the robots.txt tester cover the checks for your own site.
Data
- Per-site table (1,254 rows): ai-crawler-study-2026-11.csv. Columns: rank, domain, robots.txt status, wildcard rule, crawlers named, crawlers fully blocked, crawlers partially blocked, blocks-training, blocks-search and blocks-user-fetch flags, llms.txt, llms-full.txt and H1 flags, HTML bytes, truncation flag, text characters, thin-homepage flag, framework markers, title.
- Source list: Tranco, downloaded 4 November 2026.
- Collected 4 November 2026 in a single six-minute run. robots.txt files change, and the CSV is what each site said at that moment.
Limitations
- One vantage point. One fetch per URL from one network with one user agent, and one retry on failure. Sites that treat unknown clients differently will look different here than they do to a real crawler.
- Certificate checks were off, so a site with a broken certificate still counts.
- robots.txt shows stated policy, not behaviour. It does not show enforcement at the server or whether a crawler complied.
- The parser is simple. It does not evaluate path patterns, so “partial” only means a non-root disallow exists for that crawler. It reads a crawler’s first matching group.
- Small samples at the top. The top-100 band has 49 sites with a homepage, so its percentages move a lot on a few sites.
- Tranco is not a list of publishers. It over-represents infrastructure, and rank bands are a rough proxy for size.
What changed on 4 November
- Removed the claim that blocking ChatGPT-User makes a pasted link fail, and added the vendors’ own wording. My site’s earlier posts already quoted it correctly.
- Replaced “113 bot walls and consent screens” with what the data supports. Several sites I had listed as examples belonged to the other group.
- Removed the claim that ChatGPT-User, Claude-User and Perplexity-User do not execute JavaScript, which I had not tested, and cited Vercel’s test of AI crawlers instead.
- Added llms.txt as a share of all 2,500 domains, and changed the rank bands to the heading-checked count. Added the 100-byte floor to llms-full.txt, which removes nine files, eight of them 8-byte placeholders. The overlap with training blocks went from 34 to 27.
- Excluded cnn.com from the thin figures, which moved 14.6% to 14.5% and the 101 to 500 band from 22.1% to 21.6%.
- Added the
llms_txt_has_h1andhtml_truncatedCSV columns. The first version said the CSV flagged non-heading files when it did not. - Described the bseoa check accurately: 13 sites, the product I sell, compared crawler by crawler only on 4 November.
- Removed a statement that the data was released under CC BY 4.0, which had not been decided.
- Removed guesses about what site owners had read or intended, added the comparison with other studies, and added the fetch settings to the limitations.