Which sites block AI2Bot?

Ai2 · declared purpose: model training · latest sealed date 2026-08-07.

On 2026-08-07, 1,347 of the 61,552 panel domains that served a parseable robots.txt name ai2bot explicitly. 1,129 of them (1.8% of parsed) disallow it from the entire site, 158 restrict it to part of the site, and 60 name it and allow it. The remaining 60,205 parsed domains do not mention it at all — their wildcard group governs it.

What the denominator is. On 2026-08-07 we attempted /robots.txt on 100,000 panel domains. 24,130 could not be reached at all (DNS failure, TLS failure, timeout, refused connection) and 14,318 answered with something other than a 200. Both of those groups have no observed AI-crawler policy, and neither is counted as allowing anything — an unreachable site is UNKNOWN, not permissive. Every crawler figure below is out of the 61,552 domains that returned a robots.txt we could parse on that date.

Blocks outright — the file has a group for that exact product token containing Disallow: / with no overriding allow. Partly restricts — a group exists with at least one non-empty Disallow, but not the whole site. Names & allows — a group exists with no effective Disallow. Everything else is not mentioned: the wildcard * group governs it, and wildcard effects are counted separately rather than folded in, because a site that blocks everything has not made a decision about any AI crawler in particular. Matching is exact and case-insensitive per RFC 9309 — never substring.

Get told when the sites you care about change their AI crawler rules

Sites publish today's robots.txt and overwrite it — no archive, no history, no notification. We keep a dated daily copy. Leave your email and we'll send you what moved for the sites you care about: who started blocking, who stopped, and on which day.

Email me AI-crawler policy changes →

Work email only. No newsletter, no obligation.

Prefer to just buy it? See the sealed-archive monitoring offers, priced, with a dated sample.

What moved: AI2Bot verdict changes

Counted domains whose robots.txt changed what AI2Bot may do, between one sealed date and the previous one. Aggregate only — we do not publish which site.

Across the whole archive: 382 AI2Bot verdict transitions, of which 231 moved TO a full block and 102 moved AWAY from one.

Sealed date	Verdict moved	Domains
2026-08-07	not mentioned → blocked outright	2
2026-08-07	blocked outright → not mentioned	2
2026-08-07	not mentioned → partly restricted	1
2026-08-07	partly restricted → not mentioned	1
2026-08-06	blocked outright → not mentioned	2
2026-08-06	not mentioned → blocked outright	1
2026-08-06	partly restricted → not mentioned	1
2026-08-05	not mentioned → blocked outright	2
2026-08-05	blocked outright → not mentioned	2
2026-08-05	not mentioned → partly restricted	1
2026-08-04	not mentioned → blocked outright	2
2026-08-01	not mentioned → blocked outright	3
2026-07-31	not mentioned → blocked outright	2
2026-07-31	blocked outright → not mentioned	1
2026-07-30	not mentioned → blocked outright	2
2026-07-30	blocked outright → not mentioned	2
2026-07-30	not mentioned → partly restricted	1
2026-07-29	not mentioned → blocked outright	15
2026-07-29	blocked outright → not mentioned	1
2026-07-28	not mentioned → blocked outright	3
2026-07-28	not mentioned → partly restricted	1
2026-07-28	blocked outright → not mentioned	1
2026-07-27	not mentioned → blocked outright	1
2026-07-27	blocked outright → not mentioned	1
2026-07-25	not mentioned → blocked outright	2
2026-07-25	not mentioned → partly restricted	1
2026-07-25	blocked outright → not mentioned	1
2026-07-24	not mentioned → blocked outright	7
2026-07-24	partly restricted → not mentioned	4
2026-07-24	not mentioned → named and allowed	1
2026-07-23	not mentioned → partly restricted	3
2026-07-23	blocked outright → not mentioned	2
2026-07-23	named and allowed → not mentioned	1
2026-07-22	blocked outright → not mentioned	15
2026-07-22	not mentioned → blocked outright	6
2026-07-22	partly restricted → not mentioned	1
2026-07-21	blocked outright → not mentioned	3
2026-07-21	not mentioned → partly restricted	1
2026-07-21	not mentioned → named and allowed	1
2026-07-21	named and allowed → not mentioned	1

Read this before quoting a trend. We hold 55 sealed daily snapshots between 2026-06-09 and 2026-08-07, out of 60 calendar days. 5 calendar day(s) inside that span have no collection run at all: 2026-06-11, 2026-06-12, 2026-06-19, 2026-07-04, 2026-07-15. Consecutive rows in every table below are consecutive SEALED dates, not consecutive days — where a gap falls, that row's diff covers more than 24 hours, and the span is printed with it. We do not interpolate across a day we did not collect.

Every sealed date

The denominator column is the number of panel domains that served a parseable robots.txt that day. It moves, so a raw count moves with it — compare the ratio, not the count.

Sealed date	Names it	Blocks outright	Partly restricts	Names & allows	Parsed that day
2026-08-07	1,347	1,129	158	60	61,552
2026-08-06	1,342	1,125	157	60	61,421
2026-08-05	1,342	1,125	157	60	61,527
2026-08-04	1,343	1,128	155	60	61,620
2026-08-03	1,344	1,128	156	60	61,533
2026-08-02	1,342	1,125	157	60	61,559
2026-08-01	1,341	1,125	156	60	61,326
2026-07-31	1,351	1,135	156	60	61,595
2026-07-30	1,346	1,130	156	60	61,444
2026-07-29	1,335	1,121	154	60	61,492
2026-07-28	1,327	1,112	155	60	61,562
2026-07-27	1,324	1,110	154	60	61,440
2026-07-26	1,325	1,112	153	60	61,573
2026-07-25	1,327	1,114	153	60	61,584
2026-07-24	1,321	1,109	152	60	61,576
2026-07-23	1,323	1,107	157	59	61,758
2026-07-22	1,326	1,112	154	60	61,744
2026-07-21	1,333	1,119	154	60	61,565
2026-07-20	1,338	1,123	155	60	61,665
2026-07-19	1,339	1,124	154	61	61,451
2026-07-18	1,339	1,124	154	61	61,782
2026-07-17	1,333	1,119	153	61	61,659
2026-07-16	1,328	1,115	152	61	61,526
2026-07-14	1,327	1,114	151	62	61,542
2026-07-13	1,329	1,116	151	62	61,448
2026-07-12	1,328	1,116	150	62	61,518
2026-07-11	1,327	1,113	152	62	61,526
2026-07-10	1,326	1,115	149	62	61,425
2026-07-09	1,323	1,113	147	63	61,495
2026-07-08	1,321	1,111	149	61	61,364
2026-07-07	1,316	1,106	149	61	61,491
2026-07-06	1,315	1,107	149	59	61,455
2026-07-05	1,314	1,106	148	60	61,562
2026-07-03	1,347	1,139	149	59	61,535
2026-07-02	1,348	1,141	147	60	61,553
2026-07-01	1,259	1,049	150	60	61,599
2026-06-30	1,259	1,050	150	59	61,590
2026-06-29	1,254	1,043	152	59	61,814
2026-06-28	1,248	1,037	152	59	61,835
2026-06-27	1,246	1,037	150	59	61,848
2026-06-26	1,247	1,038	150	59	61,851
2026-06-25	1,244	1,037	151	56	61,867
2026-06-24	1,241	1,037	148	56	61,660
2026-06-23	1,248	1,042	150	56	61,877
2026-06-22	1,241	1,037	149	55	61,911
2026-06-21	1,239	1,034	150	55	61,905
2026-06-20	1,237	1,032	150	55	61,754
2026-06-18	1,227	1,024	150	53	61,805
2026-06-17	1,226	1,026	148	52	61,952
2026-06-16	1,222	1,024	146	52	61,645
2026-06-15	1,226	1,026	147	53	61,715
2026-06-14	1,227	1,026	148	53	61,697
2026-06-13	1,223	1,022	148	53	61,648
2026-06-10	1,221	1,019	148	54	61,489
2026-06-09	1,216	1,014	148	54	61,599

Why you will not find a robots.txt file here. These pages render mechanical verdicts and aggregate diffs, never a fetched body. Site operators leave contact addresses in robots.txt comments — on the 2026-08-06 snapshot 2,200 of the distinct bodies we fetched contain an email-shaped string — and republishing bodies would republish those. Nothing on this surface identifies an individual site, either: every figure is a count.

Compiled from each site's own published /robots.txt, /ai.txt, /llms.txt and /.well-known/tdmrep.json, fetched daily over a frozen 100,000-domain research panel (source list: Tranco; no Tranco rank is republished here). Verdicts are mechanical readings of what those files say under RFC 9309 grouping with exact product-token matching — they are not legal advice, not a statement about any site's intentions, and not a claim about what any crawler actually did. A robots.txt is a request, not an access control. Figures are provided for informational purposes only and carry no warranty of accuracy or completeness.

AI crawler policy home · What changed · How this is collected · US Tech Automations.