AI-Crawler Blocking Climbed to 19.1% on Top Sites

Between two sealed editions of the US Tech Automations Closing Web Index, the share of top sites blocking AI crawlers went up. This report measures the change in AI-crawler access policy across the Tranco top 100,000 between June 9, 2026 and July 13, 2026 — a window that contains 4 sealed editions. Every "from" and "to" percentage below is copied verbatim from its own sealed edition, and every delta is the exact subtraction of those two figures, in percentage points.

This is a derived trend seal, not a fresh crawl. It reads only numbers that were already frozen on the day they were collected. Nothing here is modeled, annualized, or projected forward.

One in five top sites now blocks at least one AI crawler — and over this window, the share kept climbing.

What Changed, in One Line

Across the Tranco top 100,000, the any-AI-block share rose from 18.5% to 19.1%, a move of +0.6 percentage points over the 34-day window. Adoption of llms.txt — the emerging file that tells AI systems how a site wants to be used — climbed further, from 22.6% to 23.6%.

The any-AI-block share rose from 18.5% to 19.1% in 34 days.

The breadth of the shift matters more than any single number. Of the 21 AI crawlers the index tracks, 18 are now blocked by more sites than they were in June, 0 are blocked by fewer, and 3 are unchanged. When a policy moves in one direction across nearly every named crawler at once, that is a signal, not noise.

18 of 21 tracked crawlers are blocked by more sites; 0 by fewer.

Reading This Trend Honestly

Each edition's denominator is the set of sites that returned a parseable robots.txt on that day. Those two denominators differ slightly: 54904 sites in the June edition and 54771 in the July edition. Because the denominators differ, every delta below is a change in share, not a change in raw site count. We say so plainly rather than dressing a share move up as a headcount.

The window is short. A 34-day span of daily crawls shows direction and magnitude — it does not establish a long-term trend, and it must not be annualized. The scope is public robots.txt for the Tranco ranking only, not the whole web.

Why insist on the share-versus-count distinction? Because the two editions did not crawl an identical list. 54904 sites returned a readable robots.txt in June and 54771 in July, so a raw count of blockers could rise or fall for reasons that have nothing to do with policy — a site going offline, a file briefly malformed. Expressing every figure as a share of that day's parseable set keeps the +0.6 move an apples-to-apples statement about behavior, not an artifact of which sites happened to answer.

A 34-day window shows direction, not destiny. Report the movement; do not overclaim it.

The Aggregate Shift

Three site-wide measures moved between the two editions. Any-AI-block and llms.txt adoption both rose; the blanket * disallow — a robots.txt rule that tells every user agent to stay out — edged down.

llms.txt adoption climbed from 22.6% to 23.6%, up +1 point.

The fall in the blanket * disallow, from 4.2% to 3.9%, is small but worth naming: it suggests sites are shifting from crude "block everyone" rules toward naming specific AI crawlers, which is exactly what the rising per-crawler numbers show.

Which Crawlers Lost the Most Ground

Every tracked crawler either held flat or was blocked by a larger share of sites. ClaudeBot, Bytespider, and Meta-ExternalAgent each gained the most, up +0.7 points; PerplexityBot was the sole crawler that did not move.

ClaudeBot blocking rose the most among AI crawlers, up +0.7 points.

GPTBot remains the single most-blocked crawler at 15.4%, and it is still climbing. The gap between the most-blocked tier — GPTBot, CCBot — and the least-blocked — PerplexityBot at 4.4% — held roughly steady, because the whole field shifted up together rather than the leaders pulling away.

The Same Story by Operator

Rolling the crawlers up to the companies that run them shows the same broad drift. Meta gained the most at the operator level, up +0.8 points, while Perplexity and Mistral held flat.

OpenAI stays the most-blocked operator at 15.6%, tracking GPTBot's lead among individual crawlers. The clustering near the top — OpenAI, Common Crawl, Anthropic, ByteDance all between 13.9% and 15.6% — means no single company is being singled out; publishers are tightening policy across the board.

The one place the picture diverges is the long tail. Cohere, Diffbot, and Mistral sit far below the leaders, and they barely moved — Cohere +0.1, Diffbot +0.2, Mistral flat at 1%. Smaller or newer crawlers simply are not on most publishers' radar yet, so the tightening is concentrated on the names people already recognize. That is worth remembering before assuming every AI operator faces the same headwind.

The Shape of the Move

The four editions in the window rose in order, without a reversal. That monotonic climb is why we describe the movement as a drift rather than a one-day blip, while still declining to call 34 days a durable trend.

Each step is small, and each is real — a sealed reading from its own day. The steadiness is the point: the share did not spike and settle back; it walked up.

For a publisher, the ordering is the useful part. If the share had jumped straight to 19.1% in a single edition and held, you might dismiss it as a data hiccup. Instead it passed through 18.7% and 18.9% on the way, three consecutive increases from 18.5%. That consistency is why we are comfortable naming the direction as real, even as we refuse to forecast where the 34-day line goes next.

What This Means for Your Crawler Policy

If you run a site in the Tranco top 100,000 — or advise anyone who does — the practical takeaway is that AI-access policy is now a moving target. A robots.txt posture that matched the field in June is already slightly behind by July, because roughly one in five peers now blocks at least one AI crawler and the number keeps ticking up.

The right response is not a one-time edit but a schedule. US Tech Automations builds workflows that audit a site's robots.txt and llms.txt posture, compare it against the tracked crawler field, and re-check on a cadence so drift gets caught the week it happens rather than the quarter it happens. The companion July Closing Web Index carries the full point-in-time counts these deltas are built from, and our healthcare crawler report shows how far one sector can sit from the market-wide 19.1%.

Access policy also varies sharply by industry. Our breakdowns of accounting sites and news versus retail show that a market-wide share hides very different behavior sector to sector — useful context before you benchmark your own site against the 19.1% headline.

Questions Publishers Are Asking

Q: Are more sites really blocking AI now, or is this measurement noise?

A: More sites are blocking. The any-AI-block share rose from 18.5% to 19.1%, and the direction is consistent across the field: 18 of 21 tracked crawlers are blocked by more sites, and none by fewer. A single number could be noise; a broad, one-directional move across 18 crawlers is a real shift in behavior.

Q: Which crawlers gained the most blocks?

A: ClaudeBot, Bytespider, and Meta-ExternalAgent each rose +0.7 points, the largest per-crawler moves. GPTBot remains the most-blocked crawler overall at 15.4%. You can see how this splits by sector in our legal-industry crawler report.

Q: Is a 34-day change a trend?

A: No — and we will not call it one. A 34-day window of daily crawls shows direction and magnitude, nothing more. It is not annualized, not projected, and not extrapolated. Treat it as a fresh reading of where policy is heading, to be confirmed by the next editions.

Q: How is the change measured?

A: Each endpoint percentage is copied verbatim from its own sealed edition. Each delta is the exact subtraction of the two figures, in percentage points. Because the two editions have slightly different parseable-robots.txt denominators — 54904 then 54771 — a delta is a change in share, not in raw site count.

Method and Provenance

This is a derived TREND seal. Endpoint percentages are copied verbatim from the two sealed base editions named in the source line; deltas are the exact subtraction of those two figures (percentage points), nothing modeled or extrapolated. Each edition's denominator is the sites that returned a parseable robots.txt that day (the two denominators differ slightly — both are stated), so a delta is a change in share, not a change in raw site count.

Scope is public robots.txt for the Tranco ranking only — not the whole web. A short window of daily crawls is NOT a long-term trend; report direction and magnitude honestly and do not annualize.

Every figure in this report is copied from a sealed edition — nothing is estimated, modeled, or extrapolated. The two source editions are recorded with their own content hashes so any reader can re-derive each delta by hand.

Source: US Tech Automations Research — Closing Web Index, trend edition 2026-07, computed from two sealed editions (closingweb-tranco-100k-2026-06-09 and closingweb-tranco-100k-2026-07).