AI crawler policy › changes
Every sealed date from 2026-06-09 to 2026-08-07.
"Compared" is the number of domains that served a parseable robots.txt on BOTH dates — a domain that was unreachable on either date cannot be diffed and is excluded from both columns rather than counted as unchanged.
| Sealed date | Compared against | Days spanned | Domains compared | File changed | AI verdict changed |
|---|---|---|---|---|---|
| 2026-08-07 | 2026-08-06 | 1 | 61,057 | 3,206 | 106 |
| 2026-08-06 | 2026-08-05 | 1 | 60,947 | 3,161 | 68 |
| 2026-08-05 | 2026-08-04 | 1 | 61,154 | 3,193 | 61 |
| 2026-08-04 | 2026-08-03 | 1 | 61,167 | 3,207 | 83 |
| 2026-08-03 | 2026-08-02 | 1 | 61,162 | 2,841 | 49 |
| 2026-08-02 | 2026-08-01 | 1 | 61,033 | 2,619 | 41 |
| 2026-08-01 | 2026-07-31 | 1 | 61,027 | 3,027 | 52 |
| 2026-07-31 | 2026-07-30 | 1 | 61,133 | 3,288 | 86 |
| 2026-07-30 | 2026-07-29 | 1 | 60,979 | 3,344 | 81 |
| 2026-07-29 | 2026-07-28 | 1 | 61,096 | 3,303 | 94 |
| 2026-07-28 | 2026-07-27 | 1 | 61,021 | 3,244 | 73 |
| 2026-07-27 | 2026-07-26 | 1 | 61,064 | 2,832 | 54 |
| 2026-07-26 | 2026-07-25 | 1 | 61,260 | 2,546 | 32 |
| 2026-07-25 | 2026-07-24 | 1 | 61,129 | 2,900 | 43 |
| 2026-07-24 | 2026-07-23 | 1 | 61,270 | 3,463 | 95 |
| 2026-07-23 | 2026-07-22 | 1 | 61,443 | 3,481 | 63 |
| 2026-07-22 | 2026-07-21 | 1 | 61,230 | 3,391 | 101 |
| 2026-07-21 | 2026-07-20 | 1 | 61,151 | 3,467 | 92 |
| 2026-07-20 | 2026-07-19 | 1 | 61,097 | 2,817 | 48 |
| 2026-07-19 | 2026-07-18 | 1 | 61,205 | 2,550 | 28 |
| 2026-07-18 | 2026-07-17 | 1 | 61,371 | 2,989 | 67 |
| 2026-07-17 | 2026-07-16 | 1 | 61,152 | 3,590 | 115 |
| 2026-07-16 | 2026-07-14 | 2 | 61,159 | 3,683 | 157 |
| 2026-07-14 | 2026-07-13 | 1 | 61,170 | 3,241 | 93 |
| 2026-07-13 | 2026-07-12 | 1 | 61,189 | 2,871 | 41 |
| 2026-07-12 | 2026-07-11 | 1 | 61,267 | 2,585 | 33 |
| 2026-07-11 | 2026-07-10 | 1 | 61,126 | 2,784 | 58 |
| 2026-07-10 | 2026-07-09 | 1 | 61,068 | 3,339 | 85 |
| 2026-07-09 | 2026-07-08 | 1 | 60,986 | 3,247 | 85 |
| 2026-07-08 | 2026-07-07 | 1 | 60,931 | 3,343 | 90 |
| 2026-07-07 | 2026-07-06 | 1 | 61,100 | 3,253 | 96 |
| 2026-07-06 | 2026-07-05 | 1 | 61,178 | 2,723 | 46 |
| 2026-07-05 | 2026-07-03 | 2 | 61,163 | 2,890 | 93 |
| 2026-07-03 | 2026-07-02 | 1 | 61,122 | 3,392 | 169 |
| 2026-07-02 | 2026-07-01 | 1 | 61,176 | 7,238 | 179 |
| 2026-07-01 | 2026-06-30 | 1 | 61,240 | 3,385 | 82 |
| 2026-06-30 | 2026-06-29 | 1 | 61,255 | 3,579 | 87 |
| 2026-06-29 | 2026-06-28 | 1 | 61,494 | 2,909 | 61 |
| 2026-06-28 | 2026-06-27 | 1 | 61,549 | 2,677 | 34 |
| 2026-06-27 | 2026-06-26 | 1 | 61,490 | 2,989 | 59 |
| 2026-06-26 | 2026-06-25 | 1 | 61,498 | 3,317 | 71 |
| 2026-06-25 | 2026-06-24 | 1 | 61,306 | 3,456 | 157 |
| 2026-06-24 | 2026-06-23 | 1 | 61,317 | 3,389 | 72 |
| 2026-06-23 | 2026-06-22 | 1 | 61,528 | 3,263 | 74 |
| 2026-06-22 | 2026-06-21 | 1 | 61,592 | 2,937 | 50 |
| 2026-06-21 | 2026-06-20 | 1 | 61,496 | 2,492 | 22 |
| 2026-06-20 | 2026-06-18 | 2 | 61,241 | 3,580 | 152 |
| 2026-06-18 | 2026-06-17 | 1 | 61,414 | 3,478 | 153 |
| 2026-06-17 | 2026-06-16 | 1 | 61,325 | 3,432 | 86 |
| 2026-06-16 | 2026-06-15 | 1 | 61,355 | 3,460 | 81 |
| 2026-06-15 | 2026-06-14 | 1 | 61,392 | 2,905 | 56 |
| 2026-06-14 | 2026-06-13 | 1 | 61,336 | 2,595 | 39 |
| 2026-06-13 | 2026-06-10 | 3 | 60,965 | 4,250 | 202 |
| 2026-06-10 | 2026-06-09 | 1 | 61,082 | 3,147 | 51 |
Get told when the sites you care about change their AI crawler rules
Sites publish today's robots.txt and overwrite it — no archive, no history, no notification. We keep a dated daily copy. Leave your email and we'll send you what moved for the sites you care about: who started blocking, who stopped, and on which day.
Email me AI-crawler policy changes →Work email only. No newsletter, no obligation.
Prefer to just buy it? See the sealed-archive monitoring offers, priced, with a dated sample.
Aggregate counts of how many domains moved a given crawler from one verdict to another, newest sealed date first. We do not publish which domain: a mechanical verdict attached to a named site reads as a claim about that site's intent, and we observed a file, not an intent.
| Sealed date | Crawler | Verdict moved | Domains |
|---|---|---|---|
| 2026-08-07 | meta-externalagent | not mentioned → blocked outright | 34 |
| 2026-08-07 | CCBot | not mentioned → blocked outright | 34 |
| 2026-08-07 | Bytespider | not mentioned → blocked outright | 33 |
| 2026-08-07 | Google-Extended | not mentioned → blocked outright | 31 |
| 2026-08-07 | ClaudeBot | not mentioned → blocked outright | 31 |
| 2026-08-07 | Applebot-Extended | not mentioned → blocked outright | 31 |
| 2026-08-07 | GPTBot | not mentioned → blocked outright | 30 |
| 2026-08-07 | Amazonbot | not mentioned → blocked outright | 30 |
| 2026-08-07 | GPTBot | blocked outright → not mentioned | 16 |
| 2026-08-07 | Google-Extended | blocked outright → not mentioned | 13 |
| 2026-08-07 | CCBot | blocked outright → not mentioned | 13 |
| 2026-08-07 | PerplexityBot | not mentioned → named and allowed | 11 |
| 2026-08-07 | GPTBot | not mentioned → named and allowed | 11 |
| 2026-08-07 | ClaudeBot | blocked outright → not mentioned | 11 |
| 2026-08-07 | Amazonbot | blocked outright → not mentioned | 11 |
| 2026-08-07 | Google-Extended | not mentioned → named and allowed | 10 |
| 2026-08-07 | ChatGPT-User | not mentioned → named and allowed | 10 |
| 2026-08-07 | meta-externalagent | blocked outright → not mentioned | 9 |
| 2026-08-07 | ClaudeBot | not mentioned → named and allowed | 9 |
| 2026-08-07 | Bytespider | blocked outright → not mentioned | 9 |
| 2026-08-07 | Applebot-Extended | blocked outright → not mentioned | 9 |
| 2026-08-07 | ChatGPT-User | not mentioned → partly restricted | 8 |
| 2026-08-07 | PerplexityBot | not mentioned → partly restricted | 7 |
| 2026-08-07 | Perplexity-User | not mentioned → partly restricted | 7 |
| 2026-08-07 | OAI-SearchBot | not mentioned → partly restricted | 7 |
| 2026-08-07 | OAI-SearchBot | not mentioned → named and allowed | 7 |
| 2026-08-07 | cohere-ai | not mentioned → blocked outright | 6 |
| 2026-08-07 | Claude-Web | blocked outright → not mentioned | 6 |
| 2026-08-07 | PerplexityBot | blocked outright → not mentioned | 5 |
| 2026-08-07 | Perplexity-User | not mentioned → named and allowed | 5 |
| 2026-08-07 | Google-Extended | not mentioned → partly restricted | 5 |
| 2026-08-07 | Diffbot | not mentioned → blocked outright | 5 |
| 2026-08-07 | anthropic-ai | not mentioned → blocked outright | 5 |
| 2026-08-07 | Amazonbot | not mentioned → partly restricted | 5 |
| 2026-08-07 | omgilibot | not mentioned → blocked outright | 4 |
| 2026-08-07 | omgili | not mentioned → blocked outright | 4 |
| 2026-08-07 | OAI-SearchBot | partly restricted → not mentioned | 4 |
| 2026-08-07 | OAI-SearchBot | named and allowed → partly restricted | 4 |
| 2026-08-07 | meta-externalagent | not mentioned → named and allowed | 4 |
| 2026-08-07 | ImagesiftBot | not mentioned → blocked outright | 4 |
| 2026-08-07 | GPTBot | not mentioned → partly restricted | 4 |
| 2026-08-07 | GPTBot | named and allowed → partly restricted | 4 |
| 2026-08-07 | ClaudeBot | not mentioned → partly restricted | 4 |
| 2026-08-07 | Claude-Web | not mentioned → blocked outright | 4 |
| 2026-08-07 | Claude-Web | not mentioned → named and allowed | 4 |
| 2026-08-07 | Claude-User | not mentioned → partly restricted | 4 |
| 2026-08-07 | Claude-SearchBot | not mentioned → partly restricted | 4 |
| 2026-08-07 | ChatGPT-User | blocked outright → not mentioned | 4 |
| 2026-08-07 | ChatGPT-User | named and allowed → partly restricted | 4 |
| 2026-08-07 | CCBot | not mentioned → named and allowed | 4 |
| 2026-08-07 | Applebot-Extended | not mentioned → partly restricted | 4 |
| 2026-08-07 | Applebot-Extended | not mentioned → named and allowed | 4 |
| 2026-08-07 | anthropic-ai | not mentioned → named and allowed | 4 |
| 2026-08-07 | Amazonbot | not mentioned → named and allowed | 4 |
| 2026-08-07 | PerplexityBot | not mentioned → blocked outright | 3 |
| 2026-08-07 | PerplexityBot | partly restricted → not mentioned | 3 |
| 2026-08-07 | PerplexityBot | named and allowed → partly restricted | 3 |
| 2026-08-07 | meta-externalfetcher | not mentioned → partly restricted | 3 |
| 2026-08-07 | Google-Extended | named and allowed → not mentioned | 3 |
| 2026-08-07 | FacebookBot | not mentioned → blocked outright | 3 |
Compiled from each site's own published /robots.txt, /ai.txt, /llms.txt and /.well-known/tdmrep.json, fetched daily over a frozen 100,000-domain research panel (source list: Tranco; no Tranco rank is republished here). Verdicts are mechanical readings of what those files say under RFC 9309 grouping with exact product-token matching — they are not legal advice, not a statement about any site's intentions, and not a claim about what any crawler actually did. A robots.txt is a request, not an access control. Figures are provided for informational purposes only and carry no warranty of accuracy or completeness.