AI crawler policy › domain suffix
Latest sealed date 2026-08-07.
In the latest sealed reading for .ee on 2026-08-07, 28 tracked crawler tokens were named by at least one observed file: GPTBot: 6 named, 4 blocked site-wide; Amazonbot: 5 named, 3 blocked site-wide; Applebot-Extended: 5 named, 3 blocked site-wide; CCBot: 5 named, 3 blocked site-wide; ChatGPT-User: 5 named, 2 blocked site-wide; ClaudeBot: 5 named, 3 blocked site-wide; Google-Extended: 5 named, 3 blocked site-wide; Bytespider: 4 named, 3 blocked site-wide; Claude-Web: 4 named, 2 blocked site-wide; DuckAssistBot: 4 named, 2 blocked site-wide; meta-externalagent: 4 named, 3 blocked site-wide; meta-externalfetcher: 4 named, 2 blocked site-wide; OAI-SearchBot: 4 named, 2 blocked site-wide; PerplexityBot: 4 named, 2 blocked site-wide; AI2Bot: 3 named, 2 blocked site-wide; anthropic-ai: 3 named, 2 blocked site-wide; cohere-ai: 3 named, 2 blocked site-wide; cohere-training-data-crawler: 3 named, 2 blocked site-wide; Diffbot: 3 named, 2 blocked site-wide; FacebookBot: 3 named, 2 blocked site-wide; omgili: 3 named, 2 blocked site-wide; PanguBot: 3 named, 2 blocked site-wide; Timpibot: 3 named, 2 blocked site-wide; Claude-SearchBot: 1 named, 0 blocked site-wide; Claude-User: 1 named, 0 blocked site-wide; Google-CloudVertexBot: 1 named, 0 blocked site-wide; MistralAI-User: 1 named, 0 blocked site-wide; Perplexity-User: 1 named, 0 blocked site-wide. Each entry keeps the named count before the fully-blocked count. A tracked token absent from the list had zero observed mentions; the counts describe file declarations, not crawler behavior.
.co.uk site is counted under .uk. Computing a true effective TLD needs the Public Suffix List, which this surface does not ship, and approximating one would put a wrong denominator under every figure on this page.| Crawler | Operator | Names it | Blocks outright | % of parsed |
|---|---|---|---|---|
| GPTBot | OpenAI | 6 | 4 | 7.8% |
| Amazonbot | Amazon | 5 | 3 | 5.9% |
| Applebot-Extended | Apple | 5 | 3 | 5.9% |
| CCBot | Common Crawl | 5 | 3 | 5.9% |
| ChatGPT-User | OpenAI | 5 | 2 | 3.9% |
| ClaudeBot | Anthropic | 5 | 3 | 5.9% |
| Google-Extended | 5 | 3 | 5.9% | |
| Bytespider | ByteDance | 4 | 3 | 5.9% |
| Claude-Web | Anthropic | 4 | 2 | 3.9% |
| DuckAssistBot | DuckDuckGo | 4 | 2 | 3.9% |
| meta-externalagent | Meta | 4 | 3 | 5.9% |
| meta-externalfetcher | Meta | 4 | 2 | 3.9% |
| OAI-SearchBot | OpenAI | 4 | 2 | 3.9% |
| PerplexityBot | Perplexity | 4 | 2 | 3.9% |
| AI2Bot | Ai2 | 3 | 2 | 3.9% |
| anthropic-ai | Anthropic | 3 | 2 | 3.9% |
| cohere-ai | Cohere | 3 | 2 | 3.9% |
| cohere-training-data-crawler | Cohere | 3 | 2 | 3.9% |
| Diffbot | Diffbot | 3 | 2 | 3.9% |
| FacebookBot | Meta | 3 | 2 | 3.9% |
| omgili | Webz.io | 3 | 2 | 3.9% |
| PanguBot | Huawei | 3 | 2 | 3.9% |
| Timpibot | Timpi | 3 | 2 | 3.9% |
| Claude-SearchBot | Anthropic | 1 | 0 | 0.0% |
| Claude-User | Anthropic | 1 | 0 | 0.0% |
| Google-CloudVertexBot | 1 | 0 | 0.0% | |
| MistralAI-User | Mistral | 1 | 0 | 0.0% |
| Perplexity-User | Perplexity | 1 | 0 | 0.0% |
Get told when .ee change their AI crawler rules
Sites publish today's robots.txt and overwrite it — no archive, no history, no notification. We keep a dated daily copy. Leave your email and we'll send you what moved for .ee: who started blocking, who stopped, and on which day.
Email me changes for .ee →Work email only. No newsletter, no obligation.
Prefer to just buy it? See the sealed-archive monitoring offers, priced, with a dated sample.
For each date, the table keeps the panel count beside the parseable robots.txt denominator, the sites naming or blocking a tracked AI crawler, and wildcard blocks. Unreachable or unusable responses are excluded rather than treated as permissive. These are aggregate observations of published files, not crawler behavior or the intent behind a declaration.
Across the 14 sealed readings shown for .ee, the newest recorded 51 domains publishing a parseable robots.txt, against an observed low of 50 on 2026-07-28 and a high of 51 on 2026-08-07. In the newest reading, 7 named at least one tracked AI crawler (range 7 on 2026-07-25 to 8 on 2026-07-30), while 4 blocked at least one (range 4 on 2026-07-25 to 5 on 2026-07-30). These are aggregate observations of published files. A movement can reflect changed fetch coverage as well as a changed file; it does not describe motive or crawler behavior.
| Sealed date | Panel domains | Parsed | Names any | Blocks any | Wildcard block |
|---|---|---|---|---|---|
| 2026-08-07 | 67 | 51 | 7 | 4 | 5 |
| 2026-08-06 | 67 | 51 | 7 | 4 | 5 |
| 2026-08-05 | 67 | 51 | 7 | 4 | 5 |
| 2026-08-04 | 67 | 51 | 7 | 4 | 5 |
| 2026-08-03 | 67 | 50 | 7 | 4 | 5 |
| 2026-08-02 | 67 | 50 | 7 | 4 | 5 |
| 2026-08-01 | 67 | 50 | 7 | 4 | 5 |
| 2026-07-31 | 67 | 50 | 7 | 4 | 5 |
| 2026-07-30 | 67 | 51 | 8 | 5 | 5 |
| 2026-07-29 | 67 | 51 | 8 | 5 | 5 |
| 2026-07-28 | 67 | 50 | 7 | 4 | 5 |
| 2026-07-27 | 67 | 51 | 7 | 4 | 5 |
| 2026-07-26 | 67 | 51 | 7 | 4 | 5 |
| 2026-07-25 | 67 | 51 | 7 | 4 | 5 |
Across the sealed archive, 7 crawler-verdict transitions were attributed to .ee domains on 2 sealed change dates. The newest aggregate movements are listed below; no domain name or fetched file body is published.
| Sealed date | Crawler | Verdict moved | Domains |
|---|---|---|---|
| 2026-06-18 | Applebot-Extended | not mentioned → named and allowed | 1 |
| 2026-06-18 | CCBot | not mentioned → partly restricted | 1 |
| 2026-06-18 | ClaudeBot | not mentioned → partly restricted | 1 |
| 2026-06-18 | Google-Extended | not mentioned → named and allowed | 1 |
| 2026-06-18 | GPTBot | blocked outright → partly restricted | 1 |
| 2026-06-18 | PerplexityBot | not mentioned → partly restricted | 1 |
| 2026-06-13 | ChatGPT-User | not mentioned → partly restricted | 1 |
Compiled from each site's own published /robots.txt, /ai.txt, /llms.txt and /.well-known/tdmrep.json, fetched daily over a frozen 100,000-domain research panel (source list: Tranco; no Tranco rank is republished here). Verdicts are mechanical readings of what those files say under RFC 9309 grouping with exact product-token matching — they are not legal advice, not a statement about any site's intentions, and not a claim about what any crawler actually did. A robots.txt is a request, not an access control. Figures are provided for informational purposes only and carry no warranty of accuracy or completeness.