19.1% of Top Sites Block AI Crawlers: USTA Index

The Closing Web Index is a point-in-time census of AI-crawler access policy across the Tranco top 100,000 publicly accessible domains. On July 13, 2026, the clock fetched every public robots.txt it could reach, sealed the results, and published them without alteration or interpretation. Every figure in this report is derived directly from the files that were returned on that date.

Of 100000 domains attempted, 54,771 returned a parseable robots.txt. All percentages in this report are computed over that 54,771 parseable baseline — not the full 100000, and not a projection about the broader web. Domains that returned no file, a malformed file, or a server error are excluded from all rate denominators in every table and every sentence that follows.

Key Findings

Of the 54,771 parseable domains, 10458 block at least one of the 21 AI crawlers tracked by the US Tech Automations Closing Web Index. That is 19.1% — roughly one in five sites with a readable robots.txt has explicitly named at least one AI crawler and instructed it to stay out.

10458 sites (19.1%) block at least one AI crawler as of July 13, 2026.

GPTBot is blocked by 8432 sites — the most of any tracked crawler.

OpenAI carries the highest operator-level block count at 8525 sites (15.6%).

A blanket wildcard block — Disallow: / applied to the * user-agent, which cuts off every bot simultaneously — appears on only 2120 sites (3.9% of parseable domains). Most of the sites that restrict AI crawlers do so selectively, targeting individual named bots rather than shutting off all automated access at once. That pattern suggests webmaster intent is specific, not categorical.

The /llms.txt signal is running ahead of hard blocks. 18223 sites (23.6%) publish an /llms.txt file — a count that exceeds the 10458 any-block figure. Both numbers use the same 54,771 parseable-robots.txt denominator, so the comparison is direct.

The 12 operators tracked range from OpenAI at 15.6% down to Mistral at 1%. That spread of more than fourteen percentage points indicates that crawler name recognition — not a uniform policy posture across the web — is the primary driver of who gets blocked.

How We Count a Block

A site is counted as blocking a crawler only when two conditions are simultaneously present in its robots.txt. First, a user-agent group whose label matches the crawler's known name exactly. Second, that group includes the directive Disallow: /.

Partial-path restrictions (for example, Disallow: /api/ or Disallow: /wp-admin/) are not counted as blocks. A crawler-specific Allow: / directive that overrides a wildcard Disallow is not counted as a block for that named bot. The census covers 21 distinct bot identifiers spanning 12 operators; when one operator runs multiple bots, each bot identifier is counted independently against the 54,771 parseable denominator.

Sites without a parseable robots.txt are excluded from all rate denominators. That exclusion is why every percentage in this report carries the qualifier "of 54,771 parseable sites" rather than "of 100000 domains." A missing robots.txt is not treated as either an implicit block or an implicit allow.

Industry-level data from the same methodology is available for accounting firms and healthcare providers, allowing direct comparison of blocking rates across sectors that handle sensitive client data.

The census also covers legal services, where attorney-client privilege and competitive considerations create a different policy environment from general commercial domains.

The Most-Blocked AI Crawlers

The table below ranks nine crawlers by blocking rate against the 54,771 parseable-robots.txt denominator. The census tracks 21 bots in total; these nine are the bots for which per-bot block counts are reported in this edition.

GPTBot leads at 15.4%. CCBot, which powers Common Crawl datasets used by academic and commercial AI training pipelines, ranks second at 14.9%. The top four crawlers — GPTBot, CCBot, Bytespider, and ClaudeBot — are each blocked by more than thirteen percent of parseable sites, forming a distinct cluster above the rest of the list.

PerplexityBot sits noticeably lower at 4.4%. The gap between PerplexityBot and the top four is most consistent with a name-recognition explanation. Webmasters who hand-write robots.txt rules tend to add entries for crawlers they have encountered or read about; Perplexity has a shorter public history as an AI-search crawler than GPTBot or CCBot, so fewer robots.txt files include its name by the snapshot date.

Bytespider, operated by ByteDance, appears at 13.9% despite being less discussed in English-language coverage than GPTBot. Its presence high on the list suggests that domain operators monitoring inbound bot traffic — via server logs or CDN analytics — are adding Bytespider entries as they identify its traffic signature.

The relatively compressed range from CCBot (14.9%) down to Applebot-Extended (11.6%) is notable. Seven of the nine crawlers listed fall within a four-percentage-point band. That compression — from 11.6% (Applebot-Extended) up to 14.9% (CCBot) — suggests a cohort of sophisticated webmasters who systematically block a broad list of AI crawlers rather than targeting only one or two. PerplexityBot's lower rate (4.4%) stands apart from this tight cluster and reinforces the name-recognition hypothesis: crawlers whose names appear in mainstream coverage get blocked at roughly double the rate of those that do not.

Blocking by Operator

Some operators run more than one crawler. Operator-level blocking aggregates all user-agents tied to a company; if a site blocks any user-agent associated with an operator, that site is counted once in the operator's total. The table below covers all 12 operators tracked in this edition.

The spread across operators is wide. OpenAI faces a block on 15.6% of parseable sites; Mistral faces one on only 1%. That gap is consistent with manual robots.txt authorship patterns: webmasters add entries for operators whose crawlers they can name, and Mistral's crawlers have lower brand recognition among the webmaster community than OpenAI's.

Common Crawl at 14.9% is notable given that it operates as a non-profit academic data archive rather than a commercial AI product. Its high blocking rate suggests some site operators are treating it as a proxy for AI training data sourcing regardless of its organizational status.

Meta's operator total (7044, 12.9%) sits above its named Meta-ExternalAgent bot total (6462, 11.8%). The delta reflects sites that block a different Meta-affiliated user-agent by name rather than Meta-ExternalAgent specifically, consistent with Meta operating more than one crawler identifier.

Cohere and Diffbot — at 4.2% and 3.9% respectively — fall into the same low-recognition tier as Perplexity and Mistral. These operators run crawlers whose names rarely appear in webmaster forums or general coverage about AI training data, which depresses the probability that a given site's robots.txt will address them explicitly.

For a side-by-side comparison of how AI-crawler blocking rates differ between news publishers and retail domains, see AI crawler blocking: news vs. retail.

What 23.6% llms.txt Adoption Signals

18223 sites (23.6%) publish an /llms.txt file — a count that exceeds the 10458 (19.1%) that impose a hard robots.txt block on at least one AI crawler. The two metrics are measuring different things, and reading them together reveals something the numbers alone do not.

An /llms.txt file is a machine-readable declaration of AI-use preference. It does not enforce anything in robots.txt. A site can publish /llms.txt and still impose no crawl blocks, or it can block crawlers without publishing /llms.txt at all. The two signals operate on different layers of the same access-policy stack.

The 23.6% /llms.txt adoption rate and the 19.1% any-block rate measure different instrument layers. Reading them together suggests a share of the web prefers structured soft signaling over hard Disallow directives — at least in this snapshot.

The gap between the two figures implies that a segment of domains has invested in publishing a structured AI-use preference without escalating to a robots.txt block. That could represent caution, experimentation, or a deliberate choice to use a softer signaling layer for AI consumers while preserving crawl access for traditional search bots.

Whether /llms.txt publication functions as a leading indicator of future robots.txt blocking, or as a stable alternative to it, is a question the Closing Web Index is positioned to answer as additional editions accumulate.

At a Glance

Frequently Asked Questions

Q: What does it mean for a site to "block" an AI crawler?
A: A block means the site's robots.txt contains a user-agent group whose label exactly matches the crawler's known name and that group includes Disallow: /. Partial-path restrictions (Disallow: /api/) and Allow: / overrides are not counted. Only a full Disallow: / for the crawler's exact user-agent string qualifies.

Q: Why are all percentages over 54,771 rather than 100,000?
A: Only 54,771 of the 100000 domains attempted returned a parseable robots.txt on July 13, 2026. Domains without a parseable file are excluded from all rate denominators. The percentages reflect what robots.txt files actually say, not any assumption about the domains whose files were missing or malformed.

Q: Why is GPTBot's block count (8432) lower than OpenAI's operator count (8525)?
A: OpenAI operates more than one crawler. The operator total (8525) counts every site that blocks any OpenAI-affiliated user-agent; the GPTBot figure (8432) counts only sites that name GPTBot specifically in a Disallow: / directive. Sites that block a different OpenAI bot by name but not GPTBot are counted at the operator level but not in the GPTBot row.

Q: Does this index cover the entire web?
A: No. The scope is the Tranco top 100,000 domain ranking, a publicly available list weighted by estimated web traffic volume. The Closing Web Index is a census of that ranking's public robots.txt files only — not a census of all hosted domains, not a random sample of the web, and not a projection beyond the Tranco top 100,000.

Put the Closing Web Index to Work

For teams managing retrieval pipelines, AI-augmented research workflows, or content operations that depend on crawlable source material, the Closing Web Index answers a direct operational question: which domains in a target corpus currently restrict the crawlers feeding your system?

Manually checking 100000 robots.txt files on a recurring basis is not a realistic operational posture for most teams. US Tech Automations automates that monitoring — surfacing crawler-policy changes as they occur, routing alerts to the relevant workflow stage, and keeping retrieval allow-lists current without manual spot-checks on every domain of interest.

If a content or SEO lead, a RevOps manager, or a retrieval-pipeline engineer needs an audit trail of robots.txt policy changes over time, or is building a compliance workflow that depends on knowing when a key domain's access policy shifts, a USTA automation can wire that signal into your existing systems. The same workflow can track /llms.txt publication events alongside hard blocks, giving a complete view of the access-policy layer for any domain set.

Data, Scope, and Methodology

Source: Closing Web crawl — Tranco top 100,000 domains (public robots.txt + /llms.txt), collected and sealed point-in-time.

Point-in-time AI-crawler access policy for the Tranco top 100,000 domains, parsed from public robots.txt collected on July 13, 2026. 54,771 returned a parseable robots.txt.

Every figure is a verbatim count from robots.txt files the clock actually fetched on the snapshot date; nothing is estimated, modeled or extrapolated. A block means an exact user-agent group containing Disallow: / in the site's own robots.txt. Percentages are over the 54771 sites returning a parseable robots.txt, not all 100000 attempted; domains without a parseable robots.txt are excluded from rate denominators.

Source: US Tech Automations Research — computed from the sealed Closing Web Index snapshot, July 13, 2026.