robots.txt changes for GPTBot, CCBot and Google-Extended

Collection has pausedLast sealed copy 15 Oct 2026

Collection has paused. The list of sites this feed reads was withdrawn on 24 Aug 2026 and is being replaced, so nothing new has been collected since that day. This is not a feed running late and it is not catching up. We have set no date for it starting again, and we are not going to guess at one.

What we already hold is untouched and still available: 40,908,200 dated rows covering 160,276 sites over 123 days, from 9 Jun 2026 to 15 Oct 2026. Every row and every number on this page and its child pages is read out of that archive, so nothing below has changed and nothing below is going to move.

Public sample12 named sites · 47 sites changed their answer in these 8 reads

A site rewrites robots.txt whenever it likes and keeps no history of it. The file you fetch today is the only one there is, so the rule that applied to your crawler last week is not recoverable from the site. We saved a copy of each file every day up to 24 Aug 2026.

Of the 12 sites shown, 3 shut one out of the whole site, 3 stopped naming one at all, 2 let a crawler back in, 2 named one for the first time and 2 rewrote the rules under one. A name disappearing is not the same as a block coming off, and that is the one people most often get wrong: when a crawler is not named, whatever the file says under User-agent: * applies to it instead.

The biggest single rewrite in this window26 crawlers moved in one edit

ausa.org is the largest move we caught between two reads: on 9 Oct 2026 its file moved 26 named AI crawlers at once. One edit, one day, and nothing on the site to say it happened.

Across the whole window the crawlers this happened to most were ClaudeBot (25), Google-Extended (24), GPTBot (22), ChatGPT-User (18), Bytespider (17). Those names are read out of the saved files, spelled the way the sites themselves spell them, not matched against a list we wrote.

The size of the panel, and the honest count

We ask 1,600 sites for their robots.txt, and we hold a dated copy of the panel from 123 of the 129 days since 9 Jun 2026. 6 days are missing — a 2-day stretch from 11 Jun to 12 Jun, 19 Jun, 4 Jul, 15 Jul and 8 Aug 2026. We hold no copy from those days and no record of a run on them either. We are not going to invent a reason for that, and nothing on this page counts those days as read. On our newest read, 15 Oct 2026, 1,009 of them gave us a file we could save, 921 of those were a real robots.txt and not one of the 55 web pages or 33 plain files with no User-agent line sent in their place, and 215 of the real ones give at least one AI crawler a User-agent line of its own. Across the 8 reads this page counts, 8 Oct 2026 to 15 Oct 2026, 47 sites changed what their file says about a named AI crawler, across 424 crawler-level changes.

Every number in that paragraph was counted out of the stored files themselves when this page was built, and each one is shown again, broken down, on what is and is not in this feed. Ask us and we will show the working.

Doing this yourself

You can fetch any site’s robots.txt yourself in a second. What you cannot do is fetch yesterday’s. Nothing about that file is versioned, announced or archived.

To watch a panel this size you would have to fetch 1,600 files every day, store every copy, and then work out which differences are a real policy change and which are two of the site’s servers disagreeing. That last part is most of the work: it is the 59 file changes above that we throw away.

What you get

• Every site in your list that changed its answerSite, crawler, what it said before, what it says now, and both dates.
• Sites that flap between two versions are held back, not sentYou get changes, not noise.
• The saved copy behind any row, on requestSo you can read the whole rule, not just the part quoted on the page.
• Cancel any month by emailNo account to close, no notice period.

How it works

1. You email us the sites you care about, or ask for the whole panel.
2. We tell you which of them we already hold and since when, then send a checkout link in that thread.
3. The buyer receives the changes file, and names anything we could not collect.