AI crawler policy by domain suffix

Latest sealed date 2026-08-07.

126 suffixes have enough panel domains and parseable robots.txt observations to support a page. Each row keeps the observed denominator next to the two policy counts; an unreachable domain is excluded, never read as permissive.

Suffix means the final DNS label, not the effective top-level domain. A .co.uk site is counted under .uk. The detail pages add a dated series and aggregate verdict changes, so the latest cross-section can be checked against the archive rather than read as a timeless rule.

What the denominator is. On 2026-08-07 we attempted /robots.txt on 100,000 panel domains. 24,130 could not be reached at all (DNS failure, TLS failure, timeout, refused connection) and 14,318 answered with something other than a 200. Both of those groups have no observed AI-crawler policy, and neither is counted as allowing anything — an unreachable site is UNKNOWN, not permissive. Every crawler figure below is out of the 61,552 domains that returned a robots.txt we could parse on that date.

Suffix	Panel domains	Parsed	Names any tracked crawler	Blocks any tracked crawler
.com	45,707	29,366	6,811	5,187
.net	6,353	2,667	569	491
.org	4,432	3,243	739	637
.ru	3,621	2,186	257	199
.de	2,000	1,509	450	366
.jp	1,990	842	189	139
.io	1,961	1,101	151	83
.br	1,531	922	163	107
.in	1,352	713	138	101
.uk	1,311	889	269	213
.cn	1,185	234	15	10
.fr	1,009	723	212	175
.co	894	568	143	112
.edu	849	696	72	53
.it	846	584	170	145
.pl	771	615	186	164
.nl	609	451	130	109
.au	594	393	98	71
.es	548	342	109	85
.tv	497	358	62	46
.xyz	459	150	39	35
.ca	449	331	63	51
.me	447	282	53	44
.gov	436	292	25	22
.id	436	270	45	43
.ai	428	352	68	36
.app	409	247	46	34
.info	392	217	48	43
.cz	386	311	73	61
.ua	360	263	42	33
.eu	346	192	34	20
.cloud	340	96	9	6
.cc	337	175	50	49
.tr	326	219	43	31
.tw	322	102	26	21
.kr	321	208	54	45
.ar	315	178	28	20
.se	308	247	66	52
.vn	299	205	33	23
.gr	291	218	45	38
.mx	282	164	38	26
.online	268	136	34	32
.ro	265	200	49	34
.ch	260	191	51	41
.za	259	177	43	37
.hu	255	196	28	23
.live	238	128	35	30
.pro	237	131	24	21
.us	230	93	24	20
.at	225	159	37	30
.be	224	156	40	34
.cl	224	148	30	19
.site	220	101	20	17
.tech	217	57	4	2
.biz	216	70	12	12
.fi	216	159	49	47
.top	202	77	12	12
.to	201	137	39	38
.pt	184	128	47	39
.my	176	104	15	12
.dev	169	84	13	7
.pk	162	107	16	13
.ir	159	87	3	2
.ph	156	88	34	30
.dk	155	126	42	35
.pe	154	94	23	13
.sk	150	116	24	23
.il	149	104	17	9
.no	144	119	50	45
.kz	140	102	15	10
.vip	136	82	28	27
.bg	131	94	13	10
.link	127	52	7	6
.by	121	95	12	11
.club	120	73	12	10
.gg	120	94	22	18
.ie	119	81	28	24
.rs	117	93	12	10
.nz	112	79	30	23
.hr	103	75	14	12
.mobi	103	66	10	9
.th	98	54	7	6
.blog	97	78	15	14
.space	97	37	11	11
.sg	92	62	21	15
.lt	91	65	15	15
.one	90	53	14	10
.az	87	54	2	2
.uz	84	68	7	5
.shop	82	35	5	5
.hk	80	46	7	5
.su	80	39	4	4
.sa	78	44	5	4
.lv	76	56	9	7
.media	74	45	3	3
.xxx	73	66	5	5
.ec	72	36	6	5
.ee	67	51	7	4
.fm	67	49	10	9
.fun	66	34	9	9
.store	66	36	9	8
.bd	65	30	5	4
.games	65	27	6	6
.ke	61	36	8	6
.ma	58	31	7	6
.ws	58	29	8	7
.ae	56	35	6	4
.st	55	30	0	0
.ng	54	36	8	7
.si	54	37	6	3
.life	51	28	6	6
.is	50	37	1	1
.uy	49	25	10	5
.world	49	26	8	7
.im	46	34	2	1
.bet	44	28	7	7
.video	44	27	2	2
.am	41	32	6	5
.news	41	35	14	13
.cat	39	30	6	6
.chat	39	29	11	4
.ge	37	27	3	3
.ly	36	27	7	7
.md	34	29	7	4
.porn	32	28	7	6
.tube	27	25	2	2

Get told when a domain suffix change their AI crawler rules

Sites publish today's robots.txt and overwrite it — no archive, no history, no notification. We keep a dated daily copy. Leave your email and we'll send you what moved for a domain suffix: who started blocking, who stopped, and on which day.

Email me changes for a domain suffix →

Work email only. No newsletter, no obligation.

Prefer to just buy it? See the sealed-archive monitoring offers, priced, with a dated sample.

Read this before quoting a trend. We hold 55 sealed daily snapshots between 2026-06-09 and 2026-08-07, out of 60 calendar days. 5 calendar day(s) inside that span have no collection run at all: 2026-06-11, 2026-06-12, 2026-06-19, 2026-07-04, 2026-07-15. Consecutive rows in every table below are consecutive SEALED dates, not consecutive days — where a gap falls, that row's diff covers more than 24 hours, and the span is printed with it. We do not interpolate across a day we did not collect.

Why you will not find a robots.txt file here. These pages render mechanical verdicts and aggregate diffs, never a fetched body. Site operators leave contact addresses in robots.txt comments — on the 2026-08-06 snapshot 2,200 of the distinct bodies we fetched contain an email-shaped string — and republishing bodies would republish those. Nothing on this surface identifies an individual site, either: every figure is a count.

Compiled from each site's own published /robots.txt, /ai.txt, /llms.txt and /.well-known/tdmrep.json, fetched daily over a frozen 100,000-domain research panel (source list: Tranco; no Tranco rank is republished here). Verdicts are mechanical readings of what those files say under RFC 9309 grouping with exact product-token matching — they are not legal advice, not a statement about any site's intentions, and not a claim about what any crawler actually did. A robots.txt is a request, not an access control. Figures are provided for informational purposes only and carry no warranty of accuracy or completeness.

AI crawler policy home · What changed · How this is collected · US Tech Automations.