Newest stored copy in this sample: 2026-08-26. This is a one-time report from the copy we already hold, not a live re-fetch and not a feed we re-read for you after you pay.
What you get$249 once per site · stored copies dated 2026-08-19 to 2026-08-26
A dated report for one named site. It quotes that site's own public machine-rules file (robots.txt). Same shape as the sample report: blocked assistants with quoted lines, scrapers allowed or unnamed, three rule changes, and the copy's date. After you pay, a person emails you the Markdown and PDF the same day.
- Quoted lines from your own fileThe report prints the copy's date. It is not a live re-fetch unless you buy the yearly re-check.
- Which assistants you shut out by nameThe usual wanted-in list: ChatGPT-User, OAI-SearchBot, Claude-User, Claude-Web, Perplexity-User, PerplexityBot, YouBot.
- Which known scrapers you allow or never mentionThe usual wanted-out list: CCBot, Bytespider, Diffbot, omgili, ImagesiftBot, img2dataset, AI2Bot, Webzio-Extended, TimpiBot.
- Three rule changes that would flip thatIf you want assistants in and scrapers out. If you meant to keep ChatGPT out, the report says so rather than pretending otherwise.
- A money-back lineIf the report cannot quote at least one line from your own file that names a blocked assistant, you get the $249 back.
25 real sites that match the patterncounted 2026-09-01 off stored copies · not a live fetch
A site matches when the latest stored answer names at least one assistant and sets it to blocked, and at least one scraper is allowed or never named. Silence is not recorded as permission. The 25-row file is a demonstration. We do not sell this list. The paid product is one named site quoting its own file.
From the store we hold: 1,345,244 rows, 53,501 sites. 514 sites matched the pattern. 497 still matched after re-reading the stored file with today's parser. 17 dropped. 0 missing bodies. These 25 are the top of those 497.
| Site | Blocked assistants | Scrapers allowed or unnamed | Rule source | Fact date |
|---|---|---|---|---|
| fastcompany.com | Claude-Web | CCBot (allowed); Bytespider (not named); Diffbot (not named); omgili (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| france24.com | ChatGPT-User,OAI-SearchBot,Claude-User,Claude-Web,Perplexity-User,PerplexityBot,YouBot | Bytespider (not named); Diffbot (not named); ImagesiftBot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| snopes.com | ChatGPT-User,Claude-Web,PerplexityBot,YouBot | AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| adweek.com | Claude-Web | Diffbot (not named); ImagesiftBot (not named); img2dataset (not named) | robots.txt | 2026-08-26 |
| theintercept.com | ChatGPT-User,PerplexityBot | Bytespider (not named); Diffbot (not named); omgili (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| nationalgeographic.org | ChatGPT-User,Claude-Web,PerplexityBot | img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| ilsole24ore.com | ChatGPT-User,OAI-SearchBot,Perplexity-User,PerplexityBot | Bytespider (not named); Diffbot (not named); omgili (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| mainichi.jp | OAI-SearchBot,PerplexityBot | ImagesiftBot (not named); img2dataset (not named) | robots.txt | 2026-08-19 |
| towardsdatascience.com | ChatGPT-User,OAI-SearchBot,Claude-Web,PerplexityBot | Diffbot (not named); omgili (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| derstandard.at | ChatGPT-User,Claude-Web,Perplexity-User,PerplexityBot | Diffbot (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| tvtropes.org | ChatGPT-User,PerplexityBot,YouBot | omgili (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| cochranelibrary.com | PerplexityBot | CCBot (not named); Diffbot (not named); omgili (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| devpost.com | ChatGPT-User | Diffbot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| dailycaller.com | ChatGPT-User | Diffbot (not named); omgili (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| vnexpress.net | Claude-Web,YouBot | img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named) | robots.txt | 2026-08-19 |
| yourdictionary.com | ChatGPT-User | CCBot (not named); Bytespider (not named); Diffbot (not named); omgili (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| palletsprojects.com | Claude-Web | ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named) | robots.txt | 2026-08-19 |
| sportskeeda.com | Claude-User,Claude-Web,PerplexityBot,YouBot | img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named) | robots.txt | 2026-08-19 |
| tagesanzeiger.ch | ChatGPT-User,Claude-Web | Diffbot (not named); omgili (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| dukeupress.edu | ChatGPT-User | Diffbot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| spacenews.com | ChatGPT-User,OAI-SearchBot,Claude-User,PerplexityBot | Diffbot (not named); omgili (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| laist.com | ChatGPT-User | Bytespider (not named); Diffbot (not named); omgili (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| nst.com.my | ChatGPT-User,OAI-SearchBot,PerplexityBot | omgili (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
| timeslive.co.za | ChatGPT-User,Claude-Web,PerplexityBot,YouBot | CCBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-20 |
| freemusicarchive.org | ChatGPT-User | Diffbot (not named); ImagesiftBot (not named); img2dataset (not named); AI2Bot (not named); Webzio-Extended (not named); TimpiBot (not named) | robots.txt | 2026-08-19 |
Sort key for this list: Majestic Million top-100k line order. Ranks are not in the file. Licensed under a Creative Commons Attribution 3.0 Unported License.
What a report quotes, for one site
Sample report site: yourdictionary.com. Copy date: 2026-08-19. Source: that site's own robots.txt, already stored. No new fetch. Quoted lines from that stored body:
User-agent: ChatGPT-User Disallow: / User-agent: * Allow: /
ChatGPT's answering / shopping reader (ChatGPT-User) is named and shut out of the whole site. The catch-all says every unnamed machine may read the site, so the known scrapers that are not named inherit that allow. We do not know whether the owner meant that. The file is just backwards relative to the usual shop goal.
What this cannot tell you
Money lost by a block is UNKNOWN. We do not have sales, search, or ChatGPT-shopping figures for a host. We will not invent a dollar amount.
This is not a live re-fetch. We did not fetch any live robots.txt for this page. Sample quotes are stored copies dated 2026-08-19 (one CSV row 2026-08-20, one 2026-08-26).
The 25 biggest matches are mostly publishers, not checkout shops. The pattern is still real. Shop-like hostnames in the 514: 2 (autorevue.cz, hingemarketing.com) by name match only. They are not in this 25.
17 of the 514 store matches fail today's parser. A Disallow: / plus Allow paths is restricted, not allowed. ign.com is the example: the store said scrapers allowed; the stored body says restricted. It is not in the CSV.
Googlebot is not one of the 25 readers in the store. Google-Extended is a training opt-out, not shopping.
We do not sell the store, and we do not sell other sites' files. Quoting a site's own rules back to that site is the product. Bulk resale of everyone else's rules is not this SKU.
Silence is not consent. A catch-all that allows is recorded as inherited, never as that reader being allowed.
A yearly re-check exists as a note, not a second button. $249 once is one site, one dated copy. If you want the same check again when the file changes, that is a separate $349 a year product. It is not for sale from a button on this page.
How it works
- Email us the site you want checked. We reply with whether we already hold a copy that names a blocked assistant, before you pay anything.
- When a pay button exists, pay with the card, or stay in the email thread and we send the same checkout link there.
- After you pay, a person emails you the Markdown and PDF report the same day, from the copy we already hold.
See the file before you pay
You do not have to take our word for the pattern. Here are 25 real rows, carrying the same columns as the demonstration CSV, cut out of stored copies. Nothing in it is made up and nothing in it is tidied up. This list is not the product. The product is one named site's own file, quoted back.
- Open the sample as a CSVA plain spreadsheet file. It saves to your machine rather than painting itself into a browser tab, and it opens in Excel, Numbers or Google Sheets.
- The same rows as JSONThe same rows again, laid out for reading with code.
What arrives after you pay: a person emails you the Markdown and PDF report for the site you named, the same day as your payment — from the copy we already hold, not a login and not a live re-fetch.
What you would be paying for: $249 once, for the site named in the thread. Nothing on this page takes a card until a payment link is minted. After you pay, a person emails you the report the same day.
Buy the report
$249 · once, not a feed
Buy the report — $249 once$249 once, for the site you name. A $349 a year re-check is a note on the page, not a second button. Nothing on this page takes a card. If the report cannot quote at least one line from your own file that names a blocked assistant, you get the $249 back. After you pay, a person emails you the Markdown and PDF report for that site the same day, from the copy we already hold.
Rather ask first? Email operations@ustechautomations.com. $249 once, for the site named on this page. A $349 a year re-check is a note, not a second button. After you pay, a person emails you the report the same day from the copy we already hold.