AI crawler policy › domain suffix
Latest sealed date 2026-08-07.
In the latest sealed reading for .shop on 2026-08-07, 8 tracked crawler tokens were named by at least one observed file: Amazonbot: 5 named, 5 blocked site-wide; Applebot-Extended: 5 named, 5 blocked site-wide; Bytespider: 5 named, 5 blocked site-wide; CCBot: 5 named, 5 blocked site-wide; ClaudeBot: 5 named, 5 blocked site-wide; Google-Extended: 5 named, 5 blocked site-wide; GPTBot: 5 named, 5 blocked site-wide; meta-externalagent: 5 named, 5 blocked site-wide. Each entry keeps the named count before the fully-blocked count. A tracked token absent from the list had zero observed mentions; the counts describe file declarations, not crawler behavior.
.co.uk site is counted under .uk. Computing a true effective TLD needs the Public Suffix List, which this surface does not ship, and approximating one would put a wrong denominator under every figure on this page.| Crawler | Operator | Names it | Blocks outright | % of parsed |
|---|---|---|---|---|
| Amazonbot | Amazon | 5 | 5 | 14.3% |
| Applebot-Extended | Apple | 5 | 5 | 14.3% |
| Bytespider | ByteDance | 5 | 5 | 14.3% |
| CCBot | Common Crawl | 5 | 5 | 14.3% |
| ClaudeBot | Anthropic | 5 | 5 | 14.3% |
| Google-Extended | 5 | 5 | 14.3% | |
| GPTBot | OpenAI | 5 | 5 | 14.3% |
| meta-externalagent | Meta | 5 | 5 | 14.3% |
Get told when .shop change their AI crawler rules
Sites publish today's robots.txt and overwrite it — no archive, no history, no notification. We keep a dated daily copy. Leave your email and we'll send you what moved for .shop: who started blocking, who stopped, and on which day.
Email me changes for .shop →Work email only. No newsletter, no obligation.
Prefer to just buy it? See the sealed-archive monitoring offers, priced, with a dated sample.
For each date, the table keeps the panel count beside the parseable robots.txt denominator, the sites naming or blocking a tracked AI crawler, and wildcard blocks. Unreachable or unusable responses are excluded rather than treated as permissive. These are aggregate observations of published files, not crawler behavior or the intent behind a declaration.
Across the 14 sealed readings shown for .shop, the newest recorded 35 domains publishing a parseable robots.txt, against an observed low of 34 on 2026-07-30 and a high of 35 on 2026-08-07. In the newest reading, 5 named at least one tracked AI crawler (range 5 on 2026-07-25 to 5 on 2026-08-07), while 5 blocked at least one (range 5 on 2026-07-25 to 5 on 2026-08-07). These are aggregate observations of published files. A movement can reflect changed fetch coverage as well as a changed file; it does not describe motive or crawler behavior.
| Sealed date | Panel domains | Parsed | Names any | Blocks any | Wildcard block |
|---|---|---|---|---|---|
| 2026-08-07 | 82 | 35 | 5 | 5 | 3 |
| 2026-08-06 | 82 | 35 | 5 | 5 | 3 |
| 2026-08-05 | 82 | 35 | 5 | 5 | 3 |
| 2026-08-04 | 82 | 35 | 5 | 5 | 3 |
| 2026-08-03 | 82 | 35 | 5 | 5 | 3 |
| 2026-08-02 | 82 | 35 | 5 | 5 | 3 |
| 2026-08-01 | 82 | 34 | 5 | 5 | 2 |
| 2026-07-31 | 82 | 35 | 5 | 5 | 3 |
| 2026-07-30 | 82 | 34 | 5 | 5 | 3 |
| 2026-07-29 | 82 | 35 | 5 | 5 | 3 |
| 2026-07-28 | 82 | 35 | 5 | 5 | 3 |
| 2026-07-27 | 82 | 35 | 5 | 5 | 3 |
| 2026-07-26 | 82 | 35 | 5 | 5 | 3 |
| 2026-07-25 | 82 | 35 | 5 | 5 | 3 |
Across the sealed archive, 0 crawler-verdict transitions were attributed to .shop domains on 0 sealed change dates. The newest aggregate movements are listed below; no domain name or fetched file body is published.
No crawler-verdict transition is observed for this suffix across the sealed dates. That is a counted result over the archive, not a claim that every site stayed reachable.
Compiled from each site's own published /robots.txt, /ai.txt, /llms.txt and /.well-known/tdmrep.json, fetched daily over a frozen 100,000-domain research panel (source list: Tranco; no Tranco rank is republished here). Verdicts are mechanical readings of what those files say under RFC 9309 grouping with exact product-token matching — they are not legal advice, not a statement about any site's intentions, and not a claim about what any crawler actually did. A robots.txt is a request, not an access control. Figures are provided for informational purposes only and carry no warranty of accuracy or completeness.