AI crawler policy › crawler

Which sites block Claude-Web?

Anthropic · declared purpose: on-demand fetch for a user's prompt · latest sealed date 2026-08-07.

On 2026-08-07, 2,855 of the 61,552 panel domains that served a parseable robots.txt name claude-web explicitly. 2,150 of them (3.5% of parsed) disallow it from the entire site, 313 restrict it to part of the site, and 392 name it and allow it. The remaining 58,697 parsed domains do not mention it at all — their wildcard group governs it.
What the denominator is. On 2026-08-07 we attempted /robots.txt on 100,000 panel domains. 24,130 could not be reached at all (DNS failure, TLS failure, timeout, refused connection) and 14,318 answered with something other than a 200. Both of those groups have no observed AI-crawler policy, and neither is counted as allowing anything — an unreachable site is UNKNOWN, not permissive. Every crawler figure below is out of the 61,552 domains that returned a robots.txt we could parse on that date.

Blocks outright — the file has a group for that exact product token containing Disallow: / with no overriding allow. Partly restricts — a group exists with at least one non-empty Disallow, but not the whole site. Names & allows — a group exists with no effective Disallow. Everything else is not mentioned: the wildcard * group governs it, and wildcard effects are counted separately rather than folded in, because a site that blocks everything has not made a decision about any AI crawler in particular. Matching is exact and case-insensitive per RFC 9309 — never substring.

Get told when Claude-Web change their AI crawler rules

Sites publish today's robots.txt and overwrite it — no archive, no history, no notification. We keep a dated daily copy. Leave your email and we'll send you what moved for Claude-Web: who started blocking, who stopped, and on which day.

Email me changes for Claude-Web →

Work email only. No newsletter, no obligation.

Prefer to just buy it? See the sealed-archive monitoring offers, priced, with a dated sample.

What moved: Claude-Web verdict changes

Counted domains whose robots.txt changed what Claude-Web may do, between one sealed date and the previous one. Aggregate only — we do not publish which site.

Across the whole archive: 690 Claude-Web verdict transitions, of which 226 moved TO a full block and 238 moved AWAY from one.

Sealed dateVerdict movedDomains
2026-08-07blocked outright → not mentioned6
2026-08-07not mentioned → blocked outright4
2026-08-07not mentioned → named and allowed4
2026-08-07named and allowed → partly restricted3
2026-08-07not mentioned → partly restricted1
2026-08-07partly restricted → not mentioned1
2026-08-06not mentioned → named and allowed5
2026-08-06not mentioned → partly restricted1
2026-08-06not mentioned → blocked outright1
2026-08-06blocked outright → not mentioned1
2026-08-05not mentioned → partly restricted2
2026-08-05not mentioned → named and allowed2
2026-08-05named and allowed → not mentioned2
2026-08-05blocked outright → not mentioned1
2026-08-05named and allowed → partly restricted1
2026-08-04not mentioned → named and allowed3
2026-08-04blocked outright → not mentioned3
2026-08-04not mentioned → partly restricted2
2026-08-04not mentioned → blocked outright2
2026-08-03named and allowed → not mentioned2
2026-08-03partly restricted → not mentioned1
2026-08-03blocked outright → not mentioned1
2026-08-02not mentioned → named and allowed1
2026-08-02blocked outright → not mentioned1
2026-08-01not mentioned → named and allowed2
2026-08-01not mentioned → blocked outright1
2026-08-01blocked outright → not mentioned1
2026-07-31blocked outright → not mentioned19
2026-07-31not mentioned → blocked outright2
2026-07-31named and allowed → not mentioned2
2026-07-30not mentioned → blocked outright2
2026-07-30not mentioned → named and allowed2
2026-07-30not mentioned → partly restricted1
2026-07-29not mentioned → blocked outright4
2026-07-29not mentioned → partly restricted3
2026-07-29not mentioned → named and allowed2
2026-07-29blocked outright → not mentioned2
2026-07-29named and allowed → not mentioned1
2026-07-28not mentioned → blocked outright4
2026-07-28not mentioned → named and allowed2
Read this before quoting a trend. We hold 55 sealed daily snapshots between 2026-06-09 and 2026-08-07, out of 60 calendar days. 5 calendar day(s) inside that span have no collection run at all: 2026-06-11, 2026-06-12, 2026-06-19, 2026-07-04, 2026-07-15. Consecutive rows in every table below are consecutive SEALED dates, not consecutive days — where a gap falls, that row's diff covers more than 24 hours, and the span is printed with it. We do not interpolate across a day we did not collect.

Every sealed date

The denominator column is the number of panel domains that served a parseable robots.txt that day. It moves, so a raw count moves with it — compare the ratio, not the count.

Sealed dateNames itBlocks outrightPartly restrictsNames & allowsParsed that day
2026-08-072,8552,15031339261,552
2026-08-062,8522,15131039161,421
2026-08-052,8472,15230938661,527
2026-08-042,8472,15630538661,620
2026-08-032,8462,15830538361,533
2026-08-022,8472,15830538461,559
2026-08-012,8452,16130338161,326
2026-07-312,8562,17230438061,595
2026-07-302,8672,18330338161,444
2026-07-292,8532,17330337761,492
2026-07-282,8512,17329838061,562
2026-07-272,8532,17929937561,440
2026-07-262,8492,17429837761,573
2026-07-252,8542,17729837961,584
2026-07-242,8492,17329737961,576
2026-07-232,8542,17530237761,758
2026-07-222,8552,17830037761,744
2026-07-212,8402,17030037061,565
2026-07-202,8362,17329237161,665
2026-07-192,8422,18029436861,451
2026-07-182,8492,18329836861,782
2026-07-172,8442,17929736861,659
2026-07-162,8322,17629336361,526
2026-07-142,8272,17629235961,542
2026-07-132,8232,17429035961,448
2026-07-122,8182,17228735961,518
2026-07-112,8222,17528736061,526
2026-07-102,8192,17528735761,425
2026-07-092,8182,17828235861,495
2026-07-082,8072,17428135261,364
2026-07-072,8092,17528335161,491
2026-07-062,8062,17428434861,455
2026-07-052,8072,17728035061,562
2026-07-032,8032,17228234961,535
2026-07-022,7992,16628434961,553
2026-07-012,7972,17028034761,599
2026-06-302,8022,17328134861,590
2026-06-292,7972,16728634461,814
2026-06-282,8042,17528334661,835
2026-06-272,8002,17327934861,848
2026-06-262,8062,17928134661,851
2026-06-252,7862,17127833761,867
2026-06-242,7182,10827533561,660
2026-06-232,7332,12427433561,877
2026-06-222,7292,12027533461,911
2026-06-212,7292,12027433561,905
2026-06-202,7222,11627333361,754
2026-06-182,7182,11527133261,805
2026-06-172,7912,18627033561,952
2026-06-162,7752,18326133161,645
2026-06-152,7902,19626233261,715
2026-06-142,7932,19426433561,697
2026-06-132,7902,19426233461,648
2026-06-102,7702,18025633461,489
2026-06-092,7602,17725333061,599
Why you will not find a robots.txt file here. These pages render mechanical verdicts and aggregate diffs, never a fetched body. Site operators leave contact addresses in robots.txt comments — on the 2026-08-06 snapshot 2,200 of the distinct bodies we fetched contain an email-shaped string — and republishing bodies would republish those. Nothing on this surface identifies an individual site, either: every figure is a count.

Compiled from each site's own published /robots.txt, /ai.txt, /llms.txt and /.well-known/tdmrep.json, fetched daily over a frozen 100,000-domain research panel (source list: Tranco; no Tranco rank is republished here). Verdicts are mechanical readings of what those files say under RFC 9309 grouping with exact product-token matching — they are not legal advice, not a statement about any site's intentions, and not a claim about what any crawler actually did. A robots.txt is a request, not an access control. Figures are provided for informational purposes only and carry no warranty of accuracy or completeness.

AI crawler policy home · What changed · How this is collected · US Tech Automations.