AI crawler policy › domain suffix
Latest sealed date 2026-08-07.
In the latest sealed reading for .dev on 2026-08-07, 22 tracked crawler tokens were named by at least one observed file: Bytespider: 10 named, 7 blocked site-wide; CCBot: 10 named, 5 blocked site-wide; ClaudeBot: 10 named, 5 blocked site-wide; Google-Extended: 10 named, 5 blocked site-wide; GPTBot: 10 named, 5 blocked site-wide; Applebot-Extended: 9 named, 5 blocked site-wide; Amazonbot: 8 named, 5 blocked site-wide; meta-externalagent: 8 named, 5 blocked site-wide; ChatGPT-User: 5 named, 0 blocked site-wide; PerplexityBot: 4 named, 0 blocked site-wide; anthropic-ai: 3 named, 0 blocked site-wide; cohere-ai: 3 named, 0 blocked site-wide; AI2Bot: 2 named, 0 blocked site-wide; Claude-Web: 2 named, 0 blocked site-wide; OAI-SearchBot: 2 named, 0 blocked site-wide; cohere-training-data-crawler: 1 named, 0 blocked site-wide; Diffbot: 1 named, 0 blocked site-wide; DuckAssistBot: 1 named, 0 blocked site-wide; FacebookBot: 1 named, 0 blocked site-wide; meta-externalfetcher: 1 named, 0 blocked site-wide; omgili: 1 named, 0 blocked site-wide; omgilibot: 1 named, 0 blocked site-wide. Each entry keeps the named count before the fully-blocked count. A tracked token absent from the list had zero observed mentions; the counts describe file declarations, not crawler behavior.
.co.uk site is counted under .uk. Computing a true effective TLD needs the Public Suffix List, which this surface does not ship, and approximating one would put a wrong denominator under every figure on this page.| Crawler | Operator | Names it | Blocks outright | % of parsed |
|---|---|---|---|---|
| Bytespider | ByteDance | 10 | 7 | 8.3% |
| CCBot | Common Crawl | 10 | 5 | 6.0% |
| ClaudeBot | Anthropic | 10 | 5 | 6.0% |
| Google-Extended | 10 | 5 | 6.0% | |
| GPTBot | OpenAI | 10 | 5 | 6.0% |
| Applebot-Extended | Apple | 9 | 5 | 6.0% |
| Amazonbot | Amazon | 8 | 5 | 6.0% |
| meta-externalagent | Meta | 8 | 5 | 6.0% |
| ChatGPT-User | OpenAI | 5 | 0 | 0.0% |
| PerplexityBot | Perplexity | 4 | 0 | 0.0% |
| anthropic-ai | Anthropic | 3 | 0 | 0.0% |
| cohere-ai | Cohere | 3 | 0 | 0.0% |
| AI2Bot | Ai2 | 2 | 0 | 0.0% |
| Claude-Web | Anthropic | 2 | 0 | 0.0% |
| OAI-SearchBot | OpenAI | 2 | 0 | 0.0% |
| cohere-training-data-crawler | Cohere | 1 | 0 | 0.0% |
| Diffbot | Diffbot | 1 | 0 | 0.0% |
| DuckAssistBot | DuckDuckGo | 1 | 0 | 0.0% |
| FacebookBot | Meta | 1 | 0 | 0.0% |
| meta-externalfetcher | Meta | 1 | 0 | 0.0% |
| omgili | Webz.io | 1 | 0 | 0.0% |
| omgilibot | Webz.io | 1 | 0 | 0.0% |
Get told when .dev change their AI crawler rules
Sites publish today's robots.txt and overwrite it — no archive, no history, no notification. We keep a dated daily copy. Leave your email and we'll send you what moved for .dev: who started blocking, who stopped, and on which day.
Email me changes for .dev →Work email only. No newsletter, no obligation.
Prefer to just buy it? See the sealed-archive monitoring offers, priced, with a dated sample.
For each date, the table keeps the panel count beside the parseable robots.txt denominator, the sites naming or blocking a tracked AI crawler, and wildcard blocks. Unreachable or unusable responses are excluded rather than treated as permissive. These are aggregate observations of published files, not crawler behavior or the intent behind a declaration.
Across the 14 sealed readings shown for .dev, the newest recorded 84 domains publishing a parseable robots.txt, against an observed low of 84 on 2026-08-07 and a high of 88 on 2026-07-31. In the newest reading, 13 named at least one tracked AI crawler (range 13 on 2026-08-02 to 14 on 2026-08-01), while 7 blocked at least one (range 7 on 2026-08-02 to 8 on 2026-08-01). These are aggregate observations of published files. A movement can reflect changed fetch coverage as well as a changed file; it does not describe motive or crawler behavior.
| Sealed date | Panel domains | Parsed | Names any | Blocks any | Wildcard block |
|---|---|---|---|---|---|
| 2026-08-07 | 169 | 84 | 13 | 7 | 3 |
| 2026-08-06 | 169 | 85 | 13 | 7 | 3 |
| 2026-08-05 | 169 | 85 | 13 | 7 | 3 |
| 2026-08-04 | 169 | 85 | 13 | 7 | 3 |
| 2026-08-03 | 169 | 85 | 13 | 7 | 3 |
| 2026-08-02 | 169 | 85 | 13 | 7 | 3 |
| 2026-08-01 | 169 | 85 | 14 | 8 | 3 |
| 2026-07-31 | 169 | 88 | 14 | 8 | 3 |
| 2026-07-30 | 169 | 86 | 14 | 8 | 3 |
| 2026-07-29 | 169 | 86 | 14 | 8 | 3 |
| 2026-07-28 | 169 | 87 | 14 | 8 | 3 |
| 2026-07-27 | 169 | 87 | 14 | 8 | 3 |
| 2026-07-26 | 169 | 87 | 14 | 8 | 3 |
| 2026-07-25 | 169 | 87 | 14 | 8 | 3 |
Across the sealed archive, 8 crawler-verdict transitions were attributed to .dev domains on 1 sealed change dates. The newest aggregate movements are listed below; no domain name or fetched file body is published.
| Sealed date | Crawler | Verdict moved | Domains |
|---|---|---|---|
| 2026-08-04 | Amazonbot | named and allowed → partly restricted | 1 |
| 2026-08-04 | Applebot-Extended | named and allowed → partly restricted | 1 |
| 2026-08-04 | Bytespider | named and allowed → partly restricted | 1 |
| 2026-08-04 | CCBot | named and allowed → partly restricted | 1 |
| 2026-08-04 | ClaudeBot | named and allowed → partly restricted | 1 |
| 2026-08-04 | Google-Extended | named and allowed → partly restricted | 1 |
| 2026-08-04 | GPTBot | named and allowed → partly restricted | 1 |
| 2026-08-04 | meta-externalagent | named and allowed → partly restricted | 1 |
Compiled from each site's own published /robots.txt, /ai.txt, /llms.txt and /.well-known/tdmrep.json, fetched daily over a frozen 100,000-domain research panel (source list: Tranco; no Tranco rank is republished here). Verdicts are mechanical readings of what those files say under RFC 9309 grouping with exact product-token matching — they are not legal advice, not a statement about any site's intentions, and not a claim about what any crawler actually did. A robots.txt is a request, not an access control. Figures are provided for informational purposes only and carry no warranty of accuracy or completeness.