AI Crawler Blocking, Read Fresh Every Day

Every row below is read from the reading sealed on August 23, 2026, the newest day in the data itself. The panel is the same 100,000 domains every day, so a number that moves is a rule that moved, not a different set of sites.

The comparison day is August 22, 2026, the reading immediately before this one.

What moved since the previous reading

A site "changed verdict" when the stored rule for that named crawler said one thing on August 22, 2026 and a different thing on August 23, 2026. It does not mean the site changed its mind about AI, and it does not mean the crawler obeyed either version.

The panel underneath the table

Of the 100,000 domains asked on August 23, 2026, 61,301 answered with a readable robots file, 14,279 answered that they have none, and 24,420 could not be reached at all. 13,820 of the answering sites name at least one AI crawler by product token, and 10,838 block at least one — 17.7% of the sites that answered.

The unreachable group is not an "allow". It is the size of the question this reading cannot answer, and it is printed next to every rate for that reason.

What a row does and does not say

A rule that blocks the whole site says the site's own robots file disallows that product token at /. It says nothing about whether the crawler honoured it, whether the page was used for training, or whether anyone had permission to use the content. Those are different questions with different evidence, and none of them are in this table.

"Names it and allows it" is a smaller and more deliberate group than it looks: the site had to write the crawler's token down and then allow it. A site that has simply never mentioned the crawler is not counted there.

Where these numbers come from

The clock stores one sealed robots reading per domain per day and never edits an old one, so any row here can be rebuilt from the seal. This page is rewritten from that store on a schedule; the date at the top is the newest day in the store, not the day the file was last written.

• Source: US Tech Automations closing-web clock, method closing-web-v1, panel tranco_L8QG4_top100000.txt:8778484edc1487aa.

• Rule reading follows RFC 9309: exact token match, longest matching rule wins, an equal-length Allow beats a Disallow.

• Days sealed so far: 70, first 2026-06-09, newest 2026-08-23.

• Days the clock did not seal: 2026-06-11, 2026-06-12, 2026-06-19, 2026-07-04, 2026-07-15, 2026-08-08. Those days are absent from the series rather than interpolated.

This page is rewritten from the source clock and keeps the same url every time. Data through 2026-08-23. Row hash 781d8cfb2d7c2a87328dbb5655befe10e8b7974d023332acc75471d39195d004. Machine-readable copies: CSV · JSON.

Something here look wrong? Reply to any page on this site with the row and the date and we will recheck it against the seal.