Which sites block Anthropic's crawlers?

5 declared product tokens · latest sealed date 2026-08-07.

On 2026-08-07, of the 61,552 panel domains that served a parseable robots.txt, the most widely named Anthropic token is addressed by 9,757 of them, and at most 7,581 disallow any single Anthropic token site-wide. Per-token figures below — they are not the same number, and adding them up would double-count the sites that name several.

What the denominator is. On 2026-08-07 we attempted /robots.txt on 100,000 panel domains. 24,130 could not be reached at all (DNS failure, TLS failure, timeout, refused connection) and 14,318 answered with something other than a 200. Both of those groups have no observed AI-crawler policy, and neither is counted as allowing anything — an unreachable site is UNKNOWN, not permissive. Every crawler figure below is out of the 61,552 domains that returned a robots.txt we could parse on that date.

Every declared token, counted separately

Token	Declared purpose	Names it	Blocks outright	Partly restricts	Names & allows	% of parsed
ClaudeBot	model training	9,757	7,581	927	1,249	12.3%
anthropic-ai	model training	3,728	2,705	496	527	4.4%
Claude-Web	on-demand fetch for a user's prompt	2,855	2,150	313	392	3.5%
Claude-SearchBot	search / answer indexing	1,583	799	383	401	1.3%
Claude-User	on-demand fetch for a user's prompt	1,477	739	344	394	1.2%

Those tokens are not interchangeable. On 2026-08-07, 9,757 parsed domains name claudebot and 1,477 name claude-user — the same operator, a 8,280-domain difference in how widely each is addressed. A site that blocks one has not necessarily said anything about the others.

Blocks outright — the file has a group for that exact product token containing Disallow: / with no overriding allow. Partly restricts — a group exists with at least one non-empty Disallow, but not the whole site. Names & allows — a group exists with no effective Disallow. Everything else is not mentioned: the wildcard * group governs it, and wildcard effects are counted separately rather than folded in, because a site that blocks everything has not made a decision about any AI crawler in particular. Matching is exact and case-insensitive per RFC 9309 — never substring.

What this page cannot tell you. The token list is frozen at methodology census-v1. If Anthropic has shipped a crawler token since that freeze, we do not observe it and it is absent here — that is a gap in our coverage, not evidence the token is unused. We also observe only what robots.txt says; we do not observe what any crawler actually fetched.

Get told when Anthropic change their AI crawler rules

Sites publish today's robots.txt and overwrite it — no archive, no history, no notification. We keep a dated daily copy. Leave your email and we'll send you what moved for Anthropic: who started blocking, who stopped, and on which day.

Email me changes for Anthropic →

Work email only. No newsletter, no obligation.

Prefer to just buy it? See the sealed-archive monitoring offers, priced, with a dated sample.

Dated readings by declared crawler token

The newest 14 sealed daily readings below keep each crawler token separate. Every count is an aggregate observation of published robots.txt files, not crawler behavior or operator intent. Domains without a parseable response are excluded from the denominator; they are never treated as permissive.

Sealed date	Token	Names it	Blocks outright	Partly restricts	Names & allows	Parsed that day
2026-08-07	`anthropic-ai`	3,728	2,705	496	527	61,552
2026-08-07	`claude-searchbot`	1,583	799	383	401	61,552
2026-08-07	`claude-user`	1,477	739	344	394	61,552
2026-08-07	`claude-web`	2,855	2,150	313	392	61,552
2026-08-07	`claudebot`	9,757	7,581	927	1,249	61,552
2026-08-06	`anthropic-ai`	3,716	2,702	488	526	61,421
2026-08-06	`claude-searchbot`	1,574	795	376	403	61,421
2026-08-06	`claude-user`	1,470	739	339	392	61,421
2026-08-06	`claude-web`	2,852	2,151	310	391	61,421
2026-08-06	`claudebot`	9,703	7,538	920	1,245	61,421
2026-08-05	`anthropic-ai`	3,730	2,718	489	523	61,527
2026-08-05	`claude-searchbot`	1,585	806	379	400	61,527
2026-08-05	`claude-user`	1,461	735	338	388	61,527
2026-08-05	`claude-web`	2,847	2,152	309	386	61,527
2026-08-05	`claudebot`	9,731	7,567	919	1,245	61,527
2026-08-04	`anthropic-ai`	3,729	2,724	486	519	61,620
2026-08-04	`claude-searchbot`	1,584	815	371	398	61,620
2026-08-04	`claude-user`	1,459	740	331	388	61,620
2026-08-04	`claude-web`	2,847	2,156	305	386	61,620
2026-08-04	`claudebot`	9,721	7,564	907	1,250	61,620
2026-08-03	`anthropic-ai`	3,730	2,724	484	522	61,533
2026-08-03	`claude-searchbot`	1,568	815	359	394	61,533
2026-08-03	`claude-user`	1,444	740	320	384	61,533
2026-08-03	`claude-web`	2,846	2,158	305	383	61,533
2026-08-03	`claudebot`	9,708	7,562	897	1,249	61,533
2026-08-02	`anthropic-ai`	3,729	2,720	487	522	61,559
2026-08-02	`claude-searchbot`	1,559	809	357	393	61,559
2026-08-02	`claude-user`	1,434	733	318	383	61,559
2026-08-02	`claude-web`	2,847	2,158	305	384	61,559
2026-08-02	`claudebot`	9,693	7,546	901	1,246	61,559
2026-08-01	`anthropic-ai`	3,728	2,725	483	520	61,326
2026-08-01	`claude-searchbot`	1,562	813	356	393	61,326
2026-08-01	`claude-user`	1,438	738	316	384	61,326
2026-08-01	`claude-web`	2,845	2,161	303	381	61,326
2026-08-01	`claudebot`	9,646	7,520	893	1,233	61,326
2026-07-31	`anthropic-ai`	3,742	2,739	484	519	61,595
2026-07-31	`claude-searchbot`	1,577	827	357	393	61,595
2026-07-31	`claude-user`	1,454	752	316	386	61,595
2026-07-31	`claude-web`	2,856	2,172	304	380	61,595
2026-07-31	`claudebot`	9,693	7,562	890	1,241	61,595
2026-07-30	`anthropic-ai`	3,752	2,751	485	516	61,444
2026-07-30	`claude-searchbot`	1,539	824	327	388	61,444
2026-07-30	`claude-user`	1,418	748	290	380	61,444
2026-07-30	`claude-web`	2,867	2,183	303	381	61,444
2026-07-30	`claudebot`	9,680	7,555	887	1,238	61,444
2026-07-29	`anthropic-ai`	3,745	2,747	480	518	61,492
2026-07-29	`claude-searchbot`	1,518	809	322	387	61,492
2026-07-29	`claude-user`	1,396	733	283	380	61,492
2026-07-29	`claude-web`	2,853	2,173	303	377	61,492
2026-07-29	`claudebot`	9,667	7,558	875	1,234	61,492
2026-07-28	`anthropic-ai`	3,736	2,743	475	518	61,562
2026-07-28	`claude-searchbot`	1,515	811	319	385	61,562
2026-07-28	`claude-user`	1,398	736	282	380	61,562
2026-07-28	`claude-web`	2,851	2,173	298	380	61,562
2026-07-28	`claudebot`	9,662	7,563	869	1,230	61,562
2026-07-27	`anthropic-ai`	3,734	2,741	476	517	61,440
2026-07-27	`claude-searchbot`	1,510	811	318	381	61,440
2026-07-27	`claude-user`	1,392	735	280	377	61,440
2026-07-27	`claude-web`	2,853	2,179	299	375	61,440
2026-07-27	`claudebot`	9,649	7,554	862	1,233	61,440
2026-07-26	`anthropic-ai`	3,725	2,734	474	517	61,573
2026-07-26	`claude-searchbot`	1,507	811	318	378	61,573
2026-07-26	`claude-user`	1,389	734	281	374	61,573
2026-07-26	`claude-web`	2,849	2,174	298	377	61,573
2026-07-26	`claudebot`	9,640	7,545	862	1,233	61,573
2026-07-25	`anthropic-ai`	3,729	2,739	474	516	61,584
2026-07-25	`claude-searchbot`	1,512	817	317	378	61,584
2026-07-25	`claude-user`	1,393	740	280	373	61,584
2026-07-25	`claude-web`	2,854	2,177	298	379	61,584
2026-07-25	`claudebot`	9,652	7,555	863	1,234	61,584

Read this before quoting a trend. We hold 55 sealed daily snapshots between 2026-06-09 and 2026-08-07, out of 60 calendar days. 5 calendar day(s) inside that span have no collection run at all: 2026-06-11, 2026-06-12, 2026-06-19, 2026-07-04, 2026-07-15. Consecutive rows in every table below are consecutive SEALED dates, not consecutive days — where a gap falls, that row's diff covers more than 24 hours, and the span is printed with it. We do not interpolate across a day we did not collect.

Compiled from each site's own published /robots.txt, /ai.txt, /llms.txt and /.well-known/tdmrep.json, fetched daily over a frozen 100,000-domain research panel (source list: Tranco; no Tranco rank is republished here). Verdicts are mechanical readings of what those files say under RFC 9309 grouping with exact product-token matching — they are not legal advice, not a statement about any site's intentions, and not a claim about what any crawler actually did. A robots.txt is a request, not an access control. Figures are provided for informational purposes only and carry no warranty of accuracy or completeness.

AI crawler policy home · What changed · How this is collected · US Tech Automations.