AI crawler policy › domain suffix

AI crawler blocking on .org sites

Latest sealed date 2026-08-07.

In the latest sealed reading for .org on 2026-08-07, 30 tracked crawler tokens were named by at least one observed file: GPTBot: 627 named, 552 blocked site-wide; CCBot: 540 named, 494 blocked site-wide; ClaudeBot: 533 named, 462 blocked site-wide; Google-Extended: 520 named, 459 blocked site-wide; Bytespider: 509 named, 481 blocked site-wide; Amazonbot: 489 named, 452 blocked site-wide; meta-externalagent: 456 named, 422 blocked site-wide; Applebot-Extended: 435 named, 405 blocked site-wide; ChatGPT-User: 187 named, 127 blocked site-wide; PerplexityBot: 174 named, 104 blocked site-wide; anthropic-ai: 156 named, 119 blocked site-wide; cohere-ai: 110 named, 94 blocked site-wide; omgili: 107 named, 98 blocked site-wide; Claude-Web: 103 named, 78 blocked site-wide; FacebookBot: 100 named, 85 blocked site-wide; Diffbot: 98 named, 87 blocked site-wide; OAI-SearchBot: 93 named, 43 blocked site-wide; omgilibot: 93 named, 86 blocked site-wide; ImagesiftBot: 71 named, 64 blocked site-wide; Timpibot: 58 named, 52 blocked site-wide; AI2Bot: 51 named, 43 blocked site-wide; cohere-training-data-crawler: 48 named, 41 blocked site-wide; DuckAssistBot: 40 named, 27 blocked site-wide; Perplexity-User: 38 named, 20 blocked site-wide; meta-externalfetcher: 37 named, 31 blocked site-wide; Claude-SearchBot: 36 named, 15 blocked site-wide; Claude-User: 33 named, 15 blocked site-wide; PanguBot: 32 named, 29 blocked site-wide; MistralAI-User: 28 named, 23 blocked site-wide; Google-CloudVertexBot: 22 named, 16 blocked site-wide. Each entry keeps the named count before the fully-blocked count. A tracked token absent from the list had zero observed mentions; the counts describe file declarations, not crawler behavior.

Every tracked crawler on .org

Blocks outright — the file has a group for that exact product token containing Disallow: / with no overriding allow. Partly restricts — a group exists with at least one non-empty Disallow, but not the whole site. Names & allows — a group exists with no effective Disallow. Everything else is not mentioned: the wildcard * group governs it, and wildcard effects are counted separately rather than folded in, because a site that blocks everything has not made a decision about any AI crawler in particular. Matching is exact and case-insensitive per RFC 9309 — never substring.

14 newest sealed readings for .org

For each date, the table keeps the panel count beside the parseable robots.txt denominator, the sites naming or blocking a tracked AI crawler, and wildcard blocks. Unreachable or unusable responses are excluded rather than treated as permissive. These are aggregate observations of published files, not crawler behavior or the intent behind a declaration.

Across the 14 sealed readings shown for .org, the newest recorded 3,243 domains publishing a parseable robots.txt, against an observed low of 3,233 on 2026-08-06 and a high of 3,264 on 2026-08-04. In the newest reading, 739 named at least one tracked AI crawler (range 727 on 2026-07-26 to 739 on 2026-08-07), while 637 blocked at least one (range 627 on 2026-07-26 to 637 on 2026-08-07). These are aggregate observations of published files. A movement can reflect changed fetch coverage as well as a changed file; it does not describe motive or crawler behavior.

Observed AI-crawler verdict changes on .org

Across the sealed archive, 1,062 crawler-verdict transitions were attributed to .org domains on 52 sealed change dates. The newest aggregate movements are listed below; no domain name or fetched file body is published.