Newest stored copy in this sample: 2026-08-26.
What you getstored copies dated 2026-08-19 to 2026-08-26
A dated report for one named site. It quotes the stored public machine-rules file (robots.txt) we already hold. Same shape as the sample report: blocked assistants with quoted lines, scrapers allowed or unnamed, three rule changes, and the copy's date. This page does not promise a same-day clock.
- Quoted lines from your own fileThe report prints the copy's date.
- Which assistants you shut out by nameThe usual wanted-in list: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-Web, Perplexity-User, PerplexityBot, YouBot.
- Which known scrapers you allow or never mentionThe usual wanted-out list: CCBot, Bytespider, Diffbot, omgili, ImagesiftBot, img2dataset, AI2Bot, Webzio-Extended, TimpiBot.
- Three rule changes that would flip thatIf you want assistants in and scrapers out. If you meant to keep ChatGPT out, the report says so rather than pretending otherwise.
25 real sites that match the patterncounted 2026-09-01 off stored copies · not a live fetch
A site matches when the latest stored answer names at least one assistant and sets it to blocked, and at least one scraper is allowed or never named. Silence is not recorded as permission. The 25-row file is a demonstration. We do not sell this list. The paid product is one named site quoting its own file.
From the store we hold: 1,345,244 rows, 53,501 sites. 514 sites matched the pattern. 497 still matched after re-reading the stored file with today's parser. 17 dropped. 0 missing bodies. These 25 are the top of those 497.
| Site | Blocked assistants | Scrapers allowed or unnamed | Rule source | Fact date |
|---|---|---|---|---|
| fastcompany.com | Claude-Web | CCBot (allowed); Bytespider (not named); Diffbot (not named); omgili (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| france24.com | ChatGPT-User,OAI-SearchBot,Claude-User,Claude-Web,Perplexity-User,PerplexityBot,YouBot | Bytespider (not named); Diffbot (not named); ImagesiftBot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| snopes.com | ChatGPT-User,Claude-Web,PerplexityBot,YouBot | AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| adweek.com | Claude-Web | Diffbot (not named); ImagesiftBot (not named); img2dataset (not named) | robots.txt | 2026-08-26 |
| theintercept.com | ChatGPT-User,PerplexityBot | Bytespider (not named); Diffbot (not named); omgili (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| nationalgeographic.org | ChatGPT-User,Claude-Web,PerplexityBot | img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| ilsole24ore.com | ChatGPT-User,OAI-SearchBot,Perplexity-User,PerplexityBot | Bytespider (not named); Diffbot (not named); omgili (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| mainichi.jp | OAI-SearchBot,PerplexityBot | ImagesiftBot (not named); img2dataset (not named) | robots.txt | 2026-08-19 |
| towardsdatascience.com | ChatGPT-User,OAI-SearchBot,Claude-Web,PerplexityBot | Diffbot (not named); omgili (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| derstandard.at | ChatGPT-User,Claude-Web,Perplexity-User,PerplexityBot | Diffbot (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| tvtropes.org | ChatGPT-User,PerplexityBot,YouBot | omgili (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| cochranelibrary.com | PerplexityBot | CCBot (not named); Diffbot (not named); omgili (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| devpost.com | ChatGPT-User | Diffbot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| dailycaller.com | ChatGPT-User | Diffbot (not named); omgili (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| vnexpress.net | Claude-Web,YouBot | img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named) | robots.txt | 2026-08-19 |
| yourdictionary.com | ChatGPT-User | CCBot (not named); Bytespider (not named); Diffbot (not named); omgili (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| palletsprojects.com | Claude-Web | ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named) | robots.txt | 2026-08-19 |
| sportskeeda.com | Claude-User,Claude-Web,PerplexityBot,YouBot | img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named) | robots.txt | 2026-08-19 |
| tagesanzeiger.ch | ChatGPT-User,Claude-Web | Diffbot (not named); omgili (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| dukeupress.edu | ChatGPT-User | Diffbot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| spacenews.com | ChatGPT-User,OAI-SearchBot,Claude-User,PerplexityBot | Diffbot (not named); omgili (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| laist.com | ChatGPT-User | Bytespider (not named); Diffbot (not named); omgili (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| nst.com.my | ChatGPT-User,OAI-SearchBot,PerplexityBot | omgili (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| timeslive.co.za | ChatGPT-User,Claude-Web,PerplexityBot,YouBot | CCBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-20 |
| freemusicarchive.org | ChatGPT-User | Diffbot (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
Sort key for this list: Majestic Million top-100k line order. Ranks are not in the file. Licensed under a Creative Commons Attribution 3.0 Unported License.
What a report quotes, for one site
Sample report site: yourdictionary.com. Copy date: 2026-08-19. Source: that site's own robots.txt, already stored. No new fetch. Quoted lines from that stored body:
User-agent: ChatGPT-User Disallow: / User-agent: * Allow: /
ChatGPT's answering / shopping reader (ChatGPT-User) is named and shut out of the whole site. The catch-all says every unnamed machine may read the site, so the known scrapers that are not named inherit that allow. We do not know whether the owner meant that. The file is just backwards relative to the usual shop goal.
What this cannot tell you
Money lost by a block is UNKNOWN. We do not have sales, search, or ChatGPT-shopping figures for a host. We will not invent a dollar amount.
This is not a live re-fetch. We did not fetch any live robots.txt for this page. Sample quotes are stored copies dated 2026-08-19 (one CSV row 2026-08-20, one 2026-08-26).
The 25 biggest matches are mostly publishers, not checkout shops. The pattern is still real. Shop-like hostnames in the 514: 2 (autorevue.cz, hingemarketing.com) by name match only. They are not in this 25.
17 of the 514 store matches fail today's parser. A Disallow: / plus Allow paths is restricted, not allowed. ign.com is the example: the store said scrapers allowed; the stored body says restricted. It is not in the CSV.
Googlebot is not one of the 25 readers in the store. Google-Extended is a training opt-out, not shopping.
We do not sell the store, and we do not sell other sites' files. Quoting a site's own rules back to that site is the product. Bulk resale of everyone else's rules is not this SKU.
Silence is not consent. A catch-all that allows is recorded as inherited, never as that reader being allowed.
It is not for sale from a button on this page.
See the file
You do not have to take our word for the pattern. Here are 25 real rows, carrying the same columns as the demonstration CSV, cut out of stored copies. Nothing in it is made up and nothing in it is tidied up. This list is not the product. The product is one named site's own file, quoted back.
- Open the sample as a CSVA plain spreadsheet file. It saves to your machine rather than painting itself into a browser tab, and it opens in Excel, Numbers or Google Sheets.
- The same rows as JSONThe same rows again, laid out for reading with code.
Not on sale right now
Not on sale right now. We cannot yet deliver this automatically after payment, so we are not taking payment.
Email a questionNo payment is taken on this page.