AI-crawler policy sentinel
A site's robots.txt is free to read and impossible to look back at: publishers overwrite it in place, so the day a publisher started blocking a crawler is simply gone. We have held a dated daily copy across a 100,000-domain panel since 2026-06-09. Name the domains you care about and we tell you who started blocking which AI crawler, who stopped, and on which day — with the sealed snapshot behind every row.
Not on sale
Withdrawn 2026-08-24. The 100,000-domain panel this feed was built on was withdrawn, so daily robots collection stopped on that date. The sealed archive we already collected stays readable.
We are not taking new orders for this service, so there is no price and no way to order it here. The evidence we already hold stays readable and is still described on this page.
What you get
- A monthly change feed for the domains you name, across the AI crawler product tokens we track
- Who moved to blocked, who moved back, and the exact snapshot date the change first appeared in
- The wildcard-block case counted separately — a site that blocks every crawler made no decision about AI in particular
- Every row carries the two snapshot dates it was derived from
- A domain we could not reach that day is reported unreachable, never as a policy we did not observe
What a sentinel does that a live check cannot
Anyone can read a site’s robots.txt right now. Nobody can read the one it served last month, because the file is overwritten in place and no archive of it is published anywhere. So when a publisher starts blocking your crawler, the evidence that it used not to is gone by the time you notice the traffic drop.
We fetch the same panel every day and keep what came back, which is the only way an AI-crawler policy question still has an answer a month later. You give us the domains you care about and the user-agent tokens you run; each month the sentinel reports the ones whose answer changed and the date it changed on. A domain we could not reach that day is reported as unreached, because a site we failed to fetch has not given us permission.
This is an AI-crawler policy record, not a rank tracker and not a legal opinion. The sentinel reports what a file says under the matching rule the standard defines, and it says nothing about what a publisher meant by writing it.
Specimen status UNKNOWN
The catalog points to a specimen, but its compiled artifact cannot be read. No current proof is asserted.
View the machine-readable offer and no-specimen status
US Tech Automations is a data and automation company. These are our own service terms, not a government fee, filing cost, or permit charge. Nothing on this page is legal, engineering, or financial advice.