58-Date AI Crawler Robots Policy Scoreboard

US Tech Automations Research compared five named AI crawler product tokens across every row-complete date in the frozen Closing Web panel. The result is a cross-crawler benchmark, not a list of sites and not a claim about what any crawler actually did.

The window contains 58 complete dates across 64 calendar days. Six calendar dates are gaps and are not interpolated. Each included date has one stored observation for every frozen-panel site and each of four policy resources. This report uses only the robots.txt observations to answer one narrow question: what verdict did the applicable stored rule produce for a named token requesting /?

Three outcomes are kept separate. DISALLOWED means the winning applicable rule disallowed the tested request. NOT_DISALLOWED_BY_TESTED_RULE means the tested rule did not produce that verdict. UNKNOWN means the stored response could not support a trustworthy classification. The middle label is deliberately long because shortening it changes the meaning.

NOT_DISALLOWED_BY_TESTED_RULE is not permission.

UNKNOWN is evidence missing from the denominator, not a hidden zero.

The latest cross-crawler scoreboard

The latest complete date is August 11, 2026. The exact-token column is a preflight measure: it counts classified policy bodies that name that product token in a User-agent record. A token needs at least 100 such policies on every included date before this report will emit its rates.

The rule evaluator still follows normal group selection. An exact token group wins when present; otherwise the wildcard group can apply. That is why an exact-token presence count can be smaller than the DISALLOWED count. Presence decides whether the comparison axis is real. The applicable rule decides the verdict.

GPTBot led at 14.8% of classified root policies. CCBot followed at 14.6%, ClaudeBot at 13.5%, Google-Extended at 12.6%, and PerplexityBot at 6%. This ordering is descriptive. It is not a quality score, an endorsement, a compliance grade, or an instruction to copy another publisher.

30,758 panel observations were UNKNOWN for every token. The classified denominator was therefore 69,242 rather than 100,000. The panel-share column retains all 100,000 sites in its denominator, so it gives a second view that does not make the unknown population disappear.

The shared UNKNOWN count does not mean every bot received an identical policy. UNKNOWN is assigned before token evaluation when the stored response itself is unusable: for example, a transport failure, server error, truncated body, invalid text, or HTML-shaped response at the robots location.

What changed over 58 complete dates

The first complete date is June 9 and the latest is August 11. The table compares the known-policy rate at those endpoints and also shows the observed minimum and maximum across all 58 included dates. It does not draw a straight line through the six gaps.

ClaudeBot increased 0.7 percentage point between the endpoints. GPTBot, CCBot, and Google-Extended each increased 0.6 point. PerplexityBot decreased 0.1 point. These are changes in aggregate rates over a denominator that also moved. They do not identify a site transition or establish why any policy changed.

The narrow ranges matter. GPTBot stayed between 14.2% and 14.8%; PerplexityBot stayed between 6% and 6.2%. A chart can make small changes look dramatic if its axis is cropped. The downloadable file keeps the integer counts beside every rate so a reader can reconstruct the scale before interpreting the movement.

This page and the weekly Closing Web Index have different jobs. The weekly index is the current recurring stat and a two-date operational read. This page is the full-window product-token comparison. Keeping those intents separate avoids turning the same table into two competing pages.

The July crawler-policy trend used an earlier edition method. It remains historical context, but its headline should not be spliced into this series. The 58-date table uses one frozen root-path evaluator across the whole issue.

What the three verdicts mean

The evaluator uses the stored robots.txt and the Robots Exclusion Protocol group-and-rule model. Exact product-token matching is case-insensitive. Duplicate exact groups merge. The wildcard group is selected only when no exact group matches.

For the tested path, the longest matching rule wins and Allow wins an equal-length tie. An empty Disallow is not a block. Records such as Sitemap do not end a user-agent group. These details prevent a text search for Disallow: / from assigning one crawler another crawler's rule.

A sealed 4xx response means no tested robots rule was available in that response, so the result is NOT_DISALLOWED_BY_TESTED_RULE. Transport failures, 5xx responses, redirects and other statuses, missing bodies, truncated bodies, HTML-shaped responses, and invalid UTF-8 stay UNKNOWN.

The original June Closing Web Index explains the broader four-resource clock. This scoreboard is deliberately narrower. It does not combine robots.txt with llms.txt, ai.txt, or tdmrep into a synthetic openness score because those resources express different things.

The absent-token safeguard

Comparing a bot that never appears by name can produce a plausible but meaningless wildcard rate. To prevent that, the method first counts exact User-agent presence for each product token on every date. The fixed material floor is 100 classified policies per date, equal to one tenth of one percent of the frozen panel.

All five tokens clear the floor on all 58 dates. The smallest observed minimum is PerplexityBot at 4,215 policies, more than 42 times the threshold. The floor was fixed before ranking the results and applies equally to every token.

The build also evaluates a fabricated product token against the same corpus. That token has zero exact appearances across the window. Its wildcard-derived raw classifications are discarded, and its published result is UNKNOWN with null numeric fields. A parser that turns absence into 0% cannot pass this gate.

This distinction is important because zero can look like a strong competitive finding. Here, zero is publishable only when the token has material exact presence and the evaluated numerator is actually zero. No presence means no axis, not a favorable score.

Coverage, integrity, and downloads

A complete date has 100,000 rows for each of four frozen resources, or 400,000 stored rows. The selected issue contains 23.2 million such row-hash memberships and 5.8 million robots observations. The builder recomputes every robots row hash and validates every referenced compressed body against its stored length and content hash.

June 9 was collected in two successful partitions: a 50-site smoke run and the remaining panel. Both stored roots match their disjoint rows. June 13 is different. A collector process left 390,000 append-only rows before it wrote a run receipt; a later run wrote the remaining 10,000 rows and its receipt. June 13 has row-and-blob verification but no complete stored collector root.

That limitation is exposed as PARTIAL_COLLECTOR_ROOT in the JSON instead of being labeled as a full historical root. The date remains row-complete, and all 100,000 robots rows still pass their individual projection and body checks. Readers who require a stored collector root for every date can exclude June 13 from the downloadable series.

The six calendar gaps are June 11, June 12, June 19, July 4, July 15, and August 8. No value is copied forward, averaged across a gap, or estimated from neighboring dates. For this report, nothing is estimated, modeled, smoothed, interpolated, or extrapolated.

Download the full 58-date CSV for 290 dated crawler rows. Download the hash-bound JSON for the full method, coverage record, exact-token preflight, source bindings, integrity status, and aggregate series.

Neither download includes a stored policy body, response header, observed-site identity, or per-site verdict. The public artifact supports independent arithmetic without turning a benchmark into a named-company claims list.

Put the benchmark to work

An SEO director can use the five-token table as outside context for a policy review. The review should begin with the organization's own paths, content types, crawler purposes, and accountable owner. A higher peer rate is not a reason to copy a rule.

A content operations team can use the daily CSV to decide when a change is large enough to investigate. The right trigger is a predefined movement threshold paired with the UNKNOWN share, not a dramatic-looking chart. If the classified denominator shrinks, the team should inspect evidence quality before interpreting the rate.

A retrieval or platform team can use the verdict schema as an event contract: token, path, policy date, mechanical result, unknown reason, and evidence hash. US Tech Automations can automate that capture-and-review workflow, but the public benchmark does not decide policy or promise an outcome.

The comparison method and current results were frozen without a sponsor. The existing proposed $457 Sponsored Category Benchmark Brief is request-only, and sponsorship cannot change this panel, token set, ranking rule, method, or conclusion. Request the separate benchmark scope only if that publisher-controlled boundary fits; the linked page owns the current terms and no checkout starts here.

Frequently asked questions

Q: Does 14.8% mean GPTBot was blocked on 14.8% of all 100,000 sites?

A: No. It is the DISALLOWED share among 69,242 classified root-policy observations on August 11. The full-panel share was 10.2%, and 30,758 observations were UNKNOWN. Both denominators are shown so the smaller classified set cannot be mistaken for the full panel.

Q: Why can DISALLOWED exceed exact-token policy presence?

A: Exact presence is the evidence floor for admitting a product token to the benchmark. Verdict calculation then follows applicable group selection. When a policy has no exact group for that token, its wildcard group can apply. The report does not pretend those are the same count.

Q: Does NOT_DISALLOWED_BY_TESTED_RULE mean a bot may use the content?

A: No. It answers only whether the applicable stored robots rule disallowed the named token at /. It does not decide another path, authenticated access, contract terms, server controls, or legal effect.

Q: Why keep June 13 if its collector receipt is partial?

A: The date has all expected rows, and every robots row and referenced body passes direct verification. The missing historical root affects 390,000 rows across four resources, so the JSON labels the date PARTIAL_COLLECTOR_ROOT. A reader can retain the row-complete observation or exclude it under a stricter receipt requirement.

Q: Is this evidence that one crawler obeys publishers more than another?

A: No. The source contains published robots instructions, not crawler request logs. The scoreboard measures policy text under one evaluator. It does not measure requests, enforcement, citations, indexing, model use, or obedience.

Q: Can a sponsor improve its position in this ranking?

A: No. The method, dates, panel, product tokens, arithmetic, ordering, caveats, and conclusion are publisher-controlled and frozen before any sponsorship. A sponsor can request correction of a publicly sourced fact about itself in a separate commissioned brief; it cannot buy a score here.

Source: US Tech Automations Research — derived from the append-only Closing Web clock for 58 complete dates from June 9 through August 11, 2026; aggregate root-path robots policy verdicts only.