AI crawler policy › domain suffix

AI crawler blocking on .com sites

Latest sealed date 2026-08-07.

In the latest sealed reading for .com on 2026-08-07, 30 tracked crawler tokens were named by at least one observed file: GPTBot: 5,508 named, 4,032 blocked site-wide; ClaudeBot: 4,844 named, 3,626 blocked site-wide; CCBot: 4,646 named, 4,008 blocked site-wide; Google-Extended: 4,483 named, 3,399 blocked site-wide; Bytespider: 4,184 named, 3,801 blocked site-wide; Amazonbot: 3,855 named, 3,258 blocked site-wide; meta-externalagent: 3,742 named, 3,176 blocked site-wide; Applebot-Extended: 3,708 named, 3,198 blocked site-wide; ChatGPT-User: 2,399 named, 1,178 blocked site-wide; PerplexityBot: 2,376 named, 1,118 blocked site-wide; anthropic-ai: 1,905 named, 1,363 blocked site-wide; OAI-SearchBot: 1,673 named, 564 blocked site-wide; Claude-Web: 1,521 named, 1,132 blocked site-wide; cohere-ai: 1,428 named, 1,119 blocked site-wide; FacebookBot: 1,377 named, 1,033 blocked site-wide; omgili: 1,345 named, 1,234 blocked site-wide; omgilibot: 1,217 named, 1,101 blocked site-wide; Diffbot: 1,203 named, 1,063 blocked site-wide; ImagesiftBot: 974 named, 902 blocked site-wide; Claude-SearchBot: 808 named, 358 blocked site-wide; Perplexity-User: 802 named, 351 blocked site-wide; Timpibot: 783 named, 715 blocked site-wide; Claude-User: 774 named, 343 blocked site-wide; AI2Bot: 662 named, 540 blocked site-wide; DuckAssistBot: 615 named, 380 blocked site-wide; meta-externalfetcher: 561 named, 397 blocked site-wide; cohere-training-data-crawler: 496 named, 414 blocked site-wide; Google-CloudVertexBot: 395 named, 328 blocked site-wide; MistralAI-User: 391 named, 271 blocked site-wide; PanguBot: 304 named, 279 blocked site-wide. Each entry keeps the named count before the fully-blocked count. A tracked token absent from the list had zero observed mentions; the counts describe file declarations, not crawler behavior.

Every tracked crawler on .com

Blocks outright — the file has a group for that exact product token containing Disallow: / with no overriding allow. Partly restricts — a group exists with at least one non-empty Disallow, but not the whole site. Names & allows — a group exists with no effective Disallow. Everything else is not mentioned: the wildcard * group governs it, and wildcard effects are counted separately rather than folded in, because a site that blocks everything has not made a decision about any AI crawler in particular. Matching is exact and case-insensitive per RFC 9309 — never substring.

14 newest sealed readings for .com

For each date, the table keeps the panel count beside the parseable robots.txt denominator, the sites naming or blocking a tracked AI crawler, and wildcard blocks. Unreachable or unusable responses are excluded rather than treated as permissive. These are aggregate observations of published files, not crawler behavior or the intent behind a declaration.

Across the 14 sealed readings shown for .com, the newest recorded 29,366 domains publishing a parseable robots.txt, against an observed low of 29,214 on 2026-08-01 and a high of 29,380 on 2026-08-04. In the newest reading, 6,811 named at least one tracked AI crawler (range 6,733 on 2026-07-26 to 6,811 on 2026-08-07), while 5,187 blocked at least one (range 5,150 on 2026-07-26 to 5,188 on 2026-08-04). These are aggregate observations of published files. A movement can reflect changed fetch coverage as well as a changed file; it does not describe motive or crawler behavior.

Observed AI-crawler verdict changes on .com

Across the sealed archive, 17,272 crawler-verdict transitions were attributed to .com domains on 54 sealed change dates. The newest aggregate movements are listed below; no domain name or fetched file body is published.