Newest sealed read: 2026-09-17. We read this source every day up to 24 Aug 2026, and collection has paused since then with no date set for it to start again. We hold 559 sealed runs going back to 2026-06-09.
What this page is1,573 rows held · newest sealed read 2026-09-17
- 24 sites shut Google-Extended out of the whole site in this window. 32 took that block off.
- 1,456 stopped naming Google-Extended in the file at all, and 16 started naming it. Neither of those is the same as a block going on or coming off.
- 1,573 changes across 1,554 sites, out of 14,176 crawler-level changes we caught across every crawler we watch.
- We read 800 sites a day. On 17 Sep 2026, 481 of them gave us a real robots.txt and 94 of those name at least one AI crawler.
Real rows out of our sealed copies
These are rows we read out of dated copies we keep ourselves. The live source shows today only, so once a row moves, what it said before is gone from the place you would go to look.
| Site | What the file said before | What it says now | Between |
|---|---|---|---|
| allaboutjazz.com | blocked from the whole site | named, with 1 allow line and no block | 15 Sep 2026 → 16 Sep 2026 |
| 4sysops.com | not named in the file | blocked from the whole site | 15 Sep 2026 → 16 Sep 2026 |
| 1001freefonts.com | blocked from the whole site | not named in the file | 15 Sep 2026 → 16 Sep 2026 |
| jora.com | not named in the file | blocked from 14 folders | 15 Sep 2026 → 16 Sep 2026 |
| alltrails.com | blocked from the whole site, with 8 exceptions | blocked from the whole site, with 30 exceptions | 15 Sep 2026 → 16 Sep 2026 |
| americanalpineclub.org | blocked from the whole site, with 1 exception | blocked from 27 folders | 15 Sep 2026 → 16 Sep 2026 |
| bulletproof.com | not named in the file | blocked from the whole site | 15 Sep 2026 → 16 Sep 2026 |
| 1citywomen.ru | blocked from the whole site | not named in the file | 15 Sep 2026 → 16 Sep 2026 |
| port.ac.uk | not named in the file | blocked from 31 folders | 15 Sep 2026 → 16 Sep 2026 |
| cosentino.com | named, with 1 allow line and no block | blocked from 2 folders | 15 Sep 2026 → 16 Sep 2026 |
| analoghq.ai | blocked from the whole site, with 1 exception | blocked from 7 folders | 15 Sep 2026 → 16 Sep 2026 |
| businesstimes.com.sg | named, with 1 allow line and no block | blocked from the whole site, with 1 exception | 15 Sep 2026 → 16 Sep 2026 |
What this page cannot tell you
- Nothing on this page is newer than 17 Sep 2026. We read the 800 sites on one public list of the most-visited sites, and a site that was never on that list cannot appear here. That list was withdrawn on 24 Aug 2026 and collection has paused while it is replaced, so what you are reading is a dated archive with an end, not a running feed.
- We can only show a change between two of our own reads. A site that changed its file and changed it back between two of our reads did something we never saw, and it is not on this page.
- A crawler that is not named in the file is not the same as one that is allowed. When a crawler is not named, whatever the file says under User-agent: * applies to it instead. Us saying a name was dropped means only that: the name is gone.
- A site with no robots.txt at all has not given permission and has not blocked anyone. It has said nothing. We count those separately and never read them as a yes or a no.
- On our newest read, 42 sites refused us and 140 never answered. We do not know what their file said that day, and we do not guess.
- Two kinds of change are left out rather than sold to you. 2,803 came from sites whose file swings between an old and a new version day to day, which is two of the site's servers disagreeing, not a change of mind. 10 touched a file too big for us to save whole, so we could not be sure we had read all of it.
- What robots.txt says and what a crawler does are two different things. We report the file. We do not watch anyone's traffic, and we do not judge whether a site is right to block anyone.
See the file we hold
You do not have to take our word for what is in the file. Here are 25 rows of the real thing, carrying all 7 of its columns, cut out of the dated copies we sealed ourselves. Nothing in it is made up and nothing in it is tidied up.
- Open the 25 rows as a CSVA plain spreadsheet file. It saves to your machine rather than painting itself into a browser tab, and it opens in Excel, Numbers or Google Sheets.
- The same 25 rows as JSONThe same rows again, laid out for reading with code.
Nothing on this page is for sale. The sample file is free to read, and so is every row on the page above it.
These 25 rows are a slice of the file, not the whole of it. What we cannot show you here is how far back it goes: the file goes back further than these rows do.
Start the thread
No pay button on this one yet. Email operations@ustechautomations.com. We reply with what we hold for this one, and with what we do not hold, before you spend anything.
Email us about the copies we holdSay that you want Sites that changed their answer to Google-Extended and we will tell you which weeks we hold for it and since when.
More from this feed
- Up one level: AI-crawler policy changesThe whole feed, its price, and how the file arrives.
- What is and is not in this feedEvery source we read for it, how often, and the ones we refuse to collect.