AI crawler policy › domain suffix

AI crawler blocking on .de sites

Latest sealed date 2026-08-07.

In the latest sealed reading for .de on 2026-08-07, 30 tracked crawler tokens were named by at least one observed file: GPTBot: 329 named, 269 blocked site-wide; CCBot: 303 named, 274 blocked site-wide; ClaudeBot: 280 named, 233 blocked site-wide; Bytespider: 271 named, 254 blocked site-wide; Google-Extended: 251 named, 202 blocked site-wide; meta-externalagent: 229 named, 200 blocked site-wide; ChatGPT-User: 215 named, 129 blocked site-wide; PerplexityBot: 213 named, 139 blocked site-wide; Applebot-Extended: 208 named, 184 blocked site-wide; Amazonbot: 187 named, 156 blocked site-wide; Diffbot: 175 named, 161 blocked site-wide; anthropic-ai: 156 named, 131 blocked site-wide; omgili: 155 named, 148 blocked site-wide; cohere-ai: 151 named, 130 blocked site-wide; FacebookBot: 149 named, 128 blocked site-wide; omgilibot: 135 named, 131 blocked site-wide; Timpibot: 130 named, 122 blocked site-wide; Claude-Web: 125 named, 100 blocked site-wide; OAI-SearchBot: 124 named, 60 blocked site-wide; ImagesiftBot: 119 named, 109 blocked site-wide; Perplexity-User: 113 named, 55 blocked site-wide; cohere-training-data-crawler: 112 named, 110 blocked site-wide; PanguBot: 103 named, 102 blocked site-wide; Claude-SearchBot: 98 named, 47 blocked site-wide; AI2Bot: 97 named, 87 blocked site-wide; Claude-User: 93 named, 48 blocked site-wide; DuckAssistBot: 91 named, 60 blocked site-wide; Google-CloudVertexBot: 81 named, 55 blocked site-wide; MistralAI-User: 77 named, 50 blocked site-wide; meta-externalfetcher: 56 named, 35 blocked site-wide. Each entry keeps the named count before the fully-blocked count. A tracked token absent from the list had zero observed mentions; the counts describe file declarations, not crawler behavior.

Every tracked crawler on .de

Blocks outright — the file has a group for that exact product token containing Disallow: / with no overriding allow. Partly restricts — a group exists with at least one non-empty Disallow, but not the whole site. Names & allows — a group exists with no effective Disallow. Everything else is not mentioned: the wildcard * group governs it, and wildcard effects are counted separately rather than folded in, because a site that blocks everything has not made a decision about any AI crawler in particular. Matching is exact and case-insensitive per RFC 9309 — never substring.

14 newest sealed readings for .de

For each date, the table keeps the panel count beside the parseable robots.txt denominator, the sites naming or blocking a tracked AI crawler, and wildcard blocks. Unreachable or unusable responses are excluded rather than treated as permissive. These are aggregate observations of published files, not crawler behavior or the intent behind a declaration.

Across the 14 sealed readings shown for .de, the newest recorded 1,509 domains publishing a parseable robots.txt, against an observed low of 1,505 on 2026-07-30 and a high of 1,516 on 2026-07-27. In the newest reading, 450 named at least one tracked AI crawler (range 447 on 2026-07-25 to 453 on 2026-07-28), while 366 blocked at least one (range 364 on 2026-08-06 to 389 on 2026-07-28). These are aggregate observations of published files. A movement can reflect changed fetch coverage as well as a changed file; it does not describe motive or crawler behavior.

Observed AI-crawler verdict changes on .de

Across the sealed archive, 1,377 crawler-verdict transitions were attributed to .de domains on 49 sealed change dates. The newest aggregate movements are listed below; no domain name or fetched file body is published.