AI-Crawler Blocking Climbed to 18.8% on Top Sites

In the 17 days from June 9, 2026 to June 26, 2026, the share of Tranco top 100,000 sites blocking at least one named AI crawler climbed from 18.5% to 18.8% — a gain of +0.3 percentage points. The movement was broad and consistent: 17 of 21 tracked crawlers ended the window blocked by a larger share of sites, and 0 ended blocked by fewer. The US Tech Automations Closing Web Index captured 3 editions across this window, providing before-and-after measurements on 21 named AI crawlers and twelve distinct operators.

Two foundational caveats belong in the introduction before any analysis. First, this is a short-window observation: 17 days of daily crawls across 3 editions documents direction and magnitude, not a long-term trend. We do not annualize or project. Second, the denominators differ between editions — 54904 sites returned a parseable robots.txt in the opening edition and 55147 in the closing edition — so every delta expressed below is a change in share, not a change in raw site count.

What Moved in 17 Days

Three aggregate signals define the window and set the context for the per-crawler detail that follows. The any-AI-block share moved from 18.5% to 18.8%. The /llms.txt adoption share moved from 22.6% to 23.2%, the largest single aggregate move recorded across the window. The blanket wildcard "*" disallow held flat at 4.2%, indicating that sites are adding targeted per-crawler blocks rather than sweeping blanket rules.

17 of 21 tracked crawlers are blocked by more sites at window close.

The /llms.txt share widened +0.6 percentage points, the largest aggregate move in the window.

The mid-window data point supports a reading of gradual drift rather than a single step change. On June 17, 2026, the any-AI-block share read 18.7% — positioned between the 18.5% open and the 18.8% close. That intermediate reading is consistent with slow, continuous adoption of AI-crawler blocking rules rather than a wave of new configurations on a single day.

The divergence between the any-AI-block signal (18.5% to 18.8%) and the blanket "*" block signal (4.2%, flat) is meaningful. Sites are adding named, targeted disallow rules for specific crawlers rather than generic wildcards. That pattern implies deliberate policy decisions — operators choosing which AI agents to exclude rather than opting out of all bot traffic.

Share Change Is Not Count Change

The opening edition captured robots.txt data from 54904 sites; the closing edition captured 55147. These denominators differ for several reasons: sites enter and exit the Tranco top 100,000 ranking between editions, transient server availability affects whether a robots.txt file is served on a given crawl day, and sites occasionally add or remove robots.txt files entirely.

A delta of +0.3 percentage points does not mean that a fixed count of sites switched from non-blocking to blocking. It means the proportion of a slightly larger and slightly different site pool ended the window at a higher blocking rate.

Because each edition's denominator is the count of sites that returned a parseable robots.txt on that edition's crawl day, comparing raw counts across editions would produce misleading results. A site that entered the Tranco ranking between editions and happened to have a blocking rule would raise the closing-edition count without representing a policy change by an existing site. Share-based deltas neutralize this denominator drift.

The sealed methodology treats each edition independently. A site is not required to appear in both editions to be counted in its own edition's denominator.

This design choice reflects the practical reality of monitoring the open web: the ranked universe is not static, and the set of reachable robots.txt files on any given day is not identical to the set from the prior week. The Closing Web Index measures share, not absolute counts, because share is the stable comparison unit across editions with different denominators.

Per-Crawler Movement

9 individual crawlers are tracked by name in the Closing Web Index. Every named crawler in the set moved upward or held flat during the window; none lost ground. Five crawlers rose by +0.4 percentage points, three rose by +0.3 percentage points, and one held flat.

ClaudeBot, Google-Extended, Bytespider, Meta-ExternalAgent, and Applebot-Extended each rose +0.4 percentage points.

GPTBot remains the most-blocked single crawler at 15.2% — just ahead of CCBot at 14.7% at window close. PerplexityBot held flat at 4.4%, consistent with its position as a lower-blocking-rate target relative to the open-web crawlers in the set. The clustering of five crawlers at exactly +0.4 percentage points suggests that the underlying sites updating their policies are not uniformly targeting a single crawler — a site adding a new AI policy rule tends to add multiple entries at once.

For sector-level context on how these rates vary across industries, the industry-level analysis comparing news and retail sites documents how much blocking rates differ between high-editorial and high-commerce verticals. The aggregate figure of 18.8% conceals wide variance by industry type.

Per-Operator Movement

Operator-level data aggregates crawler signals by company. Twelve operators are tracked, covering the major AI-search, AI-assistant, data-training, and web-index categories. Five operators moved +0.4 percentage points, three moved +0.3 percentage points, one moved +0.1 percentage points, and three held flat.

Three operators held flat: Perplexity (4.4%), Cohere (4.1%), and Mistral (1%). Diffbot moved the least of any mover at +0.1 percentage points. OpenAI remained the most-blocked operator at 15.3% at window close, with Common Crawl close behind at 14.7%.

The three flat operators — Perplexity, Cohere, and Mistral — share a pattern: all three sit well below the any-AI-block average of 18.8%. This may reflect that sites adding new AI-crawler policies focus first on the highest-traffic and highest-profile crawlers, with less-prominent operators added in later policy iterations. For a detailed look at how one high-blocking vertical handles per-crawler targeting, see how news sites approach AI-crawler blocking.

What a 17-Day Move Does and Does Not Tell You

A 17-day window is short. It documents direction and magnitude across 3 editions; it cannot support claims about annual rates, seasonal patterns, acceleration, deceleration, or whether the movement will continue or reverse.

What the sealed data does establish: the direction of movement across this specific window was upward for most crawlers and operators, with 17 of 21 moving up and 0 moving down. The breadth of that movement — 17 upward, 0 downward, 4 unchanged — suggests the shift was distributed across the site pool rather than concentrated in a single sector or driven by a single policy event.

What the sealed data does not establish: the cause of the movement, whether it will persist, or how it maps to robots.txt adoption across the broader web beyond the Tranco top 100,000. The Closing Web Index scope is limited to public robots.txt files for sites in the Tranco ranking. It does not cover sites outside that ranking or robots.txt configurations that are rate-limited or conditionally served.

Legal and professional-services sites have historically shown higher blocking rates than the aggregate. Those vertical-specific patterns contribute to the overall pool figure — see how legal sites approach AI-crawler access policy for data on one of the higher-blocking verticals. Readers using the aggregate 18.8% figure as an industry benchmark should account for how their own vertical's rate compares.

Annualizing a 17-day movement would be speculative arithmetic. We report what two sealed editions show: a +0.3 percentage point rise in any-AI-block share, a +0.6 percentage point rise in /llms.txt adoption share, and a 17 of 21 breadth reading. That is the extent of the data claim.

Frequently Asked Questions

Q: What does "any-AI-block" mean in the Closing Web Index?
A: A site is counted under "any-AI-block" if its robots.txt disallows at least one of the 21 named AI crawlers tracked in the index. The site does not need to block all crawlers or use a wildcard rule — disallowing even a single named agent qualifies it for this count. A site that blocks only GPTBot and permits all others would still be counted under any-AI-block.

Q: Why do the opening and closing editions have different denominators?
A: Each edition's denominator is the count of sites that returned a parseable robots.txt file on that edition's crawl day. The opening edition had 54904 parseable sites; the closing edition had 55147. Denominator differences arise because sites enter and leave the Tranco top 100,000 ranking, transient server availability affects whether robots.txt files are served, and robots.txt file existence itself can change. Share-based percentages are the correct comparison unit because they neutralize this denominator drift; comparing raw blocked-site counts across editions with different denominators would be misleading.

Q: Is a +0.3 percentage point rise in 17 days large or small?
A: The Closing Web Index does not apply statistical significance testing — it is a census of public robots.txt files, not a sample survey. What the sealed data shows is that across two editions, the share of sites blocking at least one AI crawler rose from 18.5% to 18.8%, with a mid-window reading of 18.7% on June 17, 2026. Whether that magnitude is economically large or small depends on the reader's operational context. We report direction and magnitude, not its importance.

Q: Which crawlers remained flat across the window and which moved?
A: Four crawlers or operators ended the window unchanged: PerplexityBot (4.4% to 4.4%), Perplexity operator (4.4% to 4.4%), Cohere (4.1% to 4.1%), and Mistral (1% to 1%). The remaining 17 tracked crawlers and operators moved upward. Among movers, the largest per-crawler gain was +0.4 percentage points, recorded by ClaudeBot, Google-Extended, Bytespider, Meta-ExternalAgent, and Applebot-Extended. The smallest mover was Diffbot, which rose +0.1 percentage points from 3.7% to 3.8%.

Put the Movement Data to Work

An SEO team, content-operations lead, or retrieval-pipeline engineering team monitoring AI-crawler access policy faces a practical operational problem. robots.txt configurations change at unpredictable intervals, and manually rechecking 21 named crawlers across dozens or hundreds of target domains is not scalable. The Closing Web Index produces a macro signal across the Tranco top 100,000; individual domain monitoring requires a different layer.

US Tech Automations automates the per-domain rechecking workflow. A scheduled agent pulls the robots.txt file for each domain on a list, parses disallow entries against a named-crawler registry, compares the result against the prior run, and surfaces any policy changes as structured alerts. Teams running AI-powered content discovery or retrieval-augmented pipelines can wire this signal directly into their notification stack, receiving alerts when a specific publisher changes its stance on a crawler they depend on.

If your team tracks whether partner sites, competitor sites, or key publisher domains have changed their AI-crawler policy, a quarterly manual review will miss intra-quarter moves. The 17-day window documented here shows that moves happen on short timescales: a site that permitted a crawler on June 9, 2026 may have updated its robots.txt by June 26, 2026. Continuous, automated monitoring is the operational response to a policy landscape that updates on that cadence.

Data, Scope, and Methodology

Source: USTA Closing Web Index — trend computed from two sealed editions (closingweb-tranco-100k-2026-06-09, closingweb-tranco-100k-2026-06).

Scope: Change in AI-crawler access policy across the Tranco top 100,000 between two sealed closing-web editions: June 9, 2026 and June 26, 2026 (3 editions in the window). Every endpoint is a verbatim count from its own sealed edition; every delta is the arithmetic difference of those two sealed editions, in percentage points.

This is a derived trend seal. Endpoint percentages are copied verbatim from the two sealed base editions; deltas are the exact subtraction of those figures in percentage points; nothing is estimated, modeled or extrapolated. Each edition's denominator is the sites that returned a parseable robots.txt that day (54904 then 55147), so a delta is a change in share, not a change in raw site count. Scope is public robots.txt for the Tranco ranking only — not the whole web. A short window of daily crawls is not a long-term trend; we report direction and magnitude and do not annualize.

Source: US Tech Automations Research — computed from two sealed Closing Web Index editions, June 9 and June 26, 2026.