Which sites block ClaudeBot?

Anthropic · declared purpose: model training · latest sealed date 2026-08-07.

On 2026-08-07, 9,757 of the 61,552 panel domains that served a parseable robots.txt name claudebot explicitly. 7,581 of them (12.3% of parsed) disallow it from the entire site, 927 restrict it to part of the site, and 1,249 name it and allow it. The remaining 51,795 parsed domains do not mention it at all — their wildcard group governs it.

What the denominator is. On 2026-08-07 we attempted /robots.txt on 100,000 panel domains. 24,130 could not be reached at all (DNS failure, TLS failure, timeout, refused connection) and 14,318 answered with something other than a 200. Both of those groups have no observed AI-crawler policy, and neither is counted as allowing anything — an unreachable site is UNKNOWN, not permissive. Every crawler figure below is out of the 61,552 domains that returned a robots.txt we could parse on that date.

Blocks outright — the file has a group for that exact product token containing Disallow: / with no overriding allow. Partly restricts — a group exists with at least one non-empty Disallow, but not the whole site. Names & allows — a group exists with no effective Disallow. Everything else is not mentioned: the wildcard * group governs it, and wildcard effects are counted separately rather than folded in, because a site that blocks everything has not made a decision about any AI crawler in particular. Matching is exact and case-insensitive per RFC 9309 — never substring.

Get told when ClaudeBot change their AI crawler rules

Sites publish today's robots.txt and overwrite it — no archive, no history, no notification. We keep a dated daily copy. Leave your email and we'll send you what moved for ClaudeBot: who started blocking, who stopped, and on which day.

Email me changes for ClaudeBot →

Work email only. No newsletter, no obligation.

Prefer to just buy it? See the sealed-archive monitoring offers, priced, with a dated sample.

What moved: ClaudeBot verdict changes

Counted domains whose robots.txt changed what ClaudeBot may do, between one sealed date and the previous one. Aggregate only — we do not publish which site.

Across the whole archive: 2,626 ClaudeBot verdict transitions, of which 1,071 moved TO a full block and 661 moved AWAY from one.

Sealed date	Verdict moved	Domains
2026-08-07	not mentioned → blocked outright	31
2026-08-07	blocked outright → not mentioned	11
2026-08-07	not mentioned → named and allowed	9
2026-08-07	not mentioned → partly restricted	4
2026-08-07	named and allowed → not mentioned	3
2026-08-07	named and allowed → partly restricted	3
2026-08-07	partly restricted → not mentioned	2
2026-08-07	blocked outright → named and allowed	2
2026-08-07	partly restricted → named and allowed	1
2026-08-07	blocked outright → partly restricted	1
2026-08-06	not mentioned → blocked outright	16
2026-08-06	not mentioned → named and allowed	11
2026-08-06	blocked outright → not mentioned	6
2026-08-06	named and allowed → not mentioned	4
2026-08-06	not mentioned → partly restricted	2
2026-08-06	partly restricted → not mentioned	2
2026-08-06	named and allowed → blocked outright	2
2026-08-06	named and allowed → partly restricted	1
2026-08-05	not mentioned → blocked outright	14
2026-08-05	blocked outright → not mentioned	10
2026-08-05	named and allowed → partly restricted	5
2026-08-05	not mentioned → named and allowed	4
2026-08-05	not mentioned → partly restricted	3
2026-08-05	partly restricted → not mentioned	2
2026-08-05	named and allowed → not mentioned	2
2026-08-04	not mentioned → blocked outright	16
2026-08-04	not mentioned → partly restricted	15
2026-08-04	blocked outright → not mentioned	11
2026-08-04	not mentioned → named and allowed	9
2026-08-04	partly restricted → not mentioned	3
2026-08-04	named and allowed → partly restricted	3
2026-08-04	named and allowed → not mentioned	2
2026-08-04	partly restricted → named and allowed	1
2026-08-04	blocked outright → partly restricted	1
2026-08-03	not mentioned → blocked outright	10
2026-08-03	partly restricted → not mentioned	5
2026-08-03	named and allowed → not mentioned	5
2026-08-03	blocked outright → not mentioned	4
2026-08-03	not mentioned → named and allowed	3
2026-08-03	blocked outright → partly restricted	3

Read this before quoting a trend. We hold 55 sealed daily snapshots between 2026-06-09 and 2026-08-07, out of 60 calendar days. 5 calendar day(s) inside that span have no collection run at all: 2026-06-11, 2026-06-12, 2026-06-19, 2026-07-04, 2026-07-15. Consecutive rows in every table below are consecutive SEALED dates, not consecutive days — where a gap falls, that row's diff covers more than 24 hours, and the span is printed with it. We do not interpolate across a day we did not collect.

Every sealed date

The denominator column is the number of panel domains that served a parseable robots.txt that day. It moves, so a raw count moves with it — compare the ratio, not the count.

Sealed date	Names it	Blocks outright	Partly restricts	Names & allows	Parsed that day
2026-08-07	9,757	7,581	927	1,249	61,552
2026-08-06	9,703	7,538	920	1,245	61,421
2026-08-05	9,731	7,567	919	1,245	61,527
2026-08-04	9,721	7,564	907	1,250	61,620
2026-08-03	9,708	7,562	897	1,249	61,533
2026-08-02	9,693	7,546	901	1,246	61,559
2026-08-01	9,646	7,520	893	1,233	61,326
2026-07-31	9,693	7,562	890	1,241	61,595
2026-07-30	9,680	7,555	887	1,238	61,444
2026-07-29	9,667	7,558	875	1,234	61,492
2026-07-28	9,662	7,563	869	1,230	61,562
2026-07-27	9,649	7,554	862	1,233	61,440
2026-07-26	9,640	7,545	862	1,233	61,573
2026-07-25	9,652	7,555	863	1,234	61,584
2026-07-24	9,626	7,535	859	1,232	61,576
2026-07-23	9,623	7,534	860	1,229	61,758
2026-07-22	9,605	7,518	857	1,230	61,744
2026-07-21	9,537	7,473	848	1,216	61,565
2026-07-20	9,589	7,517	848	1,224	61,665
2026-07-19	9,572	7,513	841	1,218	61,451
2026-07-18	9,587	7,522	848	1,217	61,782
2026-07-17	9,561	7,509	844	1,208	61,659
2026-07-16	9,493	7,464	838	1,191	61,526
2026-07-14	9,445	7,437	821	1,187	61,542
2026-07-13	9,424	7,423	816	1,185	61,448
2026-07-12	9,443	7,442	814	1,187	61,518
2026-07-11	9,435	7,432	814	1,189	61,526
2026-07-10	9,424	7,427	812	1,185	61,425
2026-07-09	9,401	7,410	810	1,181	61,495
2026-07-08	9,365	7,389	807	1,169	61,364
2026-07-07	9,340	7,367	803	1,170	61,491
2026-07-06	9,339	7,373	800	1,166	61,455
2026-07-05	9,338	7,377	796	1,165	61,562
2026-07-03	9,330	7,368	794	1,168	61,535
2026-07-02	9,301	7,343	797	1,161	61,553
2026-07-01	9,256	7,312	789	1,155	61,599
2026-06-30	9,261	7,322	785	1,154	61,590
2026-06-29	9,277	7,347	784	1,146	61,814
2026-06-28	9,267	7,349	779	1,139	61,835
2026-06-27	9,254	7,330	775	1,149	61,848
2026-06-26	9,252	7,333	774	1,145	61,851
2026-06-25	9,214	7,308	774	1,132	61,867
2026-06-24	9,118	7,245	754	1,119	61,660
2026-06-23	9,111	7,241	753	1,117	61,877
2026-06-22	9,090	7,222	751	1,117	61,911
2026-06-21	9,066	7,200	748	1,118	61,905
2026-06-20	9,060	7,199	747	1,114	61,754
2026-06-18	9,050	7,208	738	1,104	61,805
2026-06-17	9,094	7,259	732	1,103	61,952
2026-06-16	9,018	7,204	722	1,092	61,645
2026-06-15	9,013	7,204	724	1,085	61,715
2026-06-14	9,001	7,187	728	1,086	61,697
2026-06-13	8,976	7,164	727	1,085	61,648
2026-06-10	8,892	7,101	717	1,074	61,489
2026-06-09	8,877	7,095	712	1,070	61,599

Why you will not find a robots.txt file here. These pages render mechanical verdicts and aggregate diffs, never a fetched body. Site operators leave contact addresses in robots.txt comments — on the 2026-08-06 snapshot 2,200 of the distinct bodies we fetched contain an email-shaped string — and republishing bodies would republish those. Nothing on this surface identifies an individual site, either: every figure is a count.

Compiled from each site's own published /robots.txt, /ai.txt, /llms.txt and /.well-known/tdmrep.json, fetched daily over a frozen 100,000-domain research panel (source list: Tranco; no Tranco rank is republished here). Verdicts are mechanical readings of what those files say under RFC 9309 grouping with exact product-token matching — they are not legal advice, not a statement about any site's intentions, and not a claim about what any crawler actually did. A robots.txt is a request, not an access control. Figures are provided for informational purposes only and carry no warranty of accuracy or completeness.

AI crawler policy home · What changed · How this is collected · US Tech Automations.