AI crawler policy › operator

Which sites block Meta's crawlers?

3 declared product tokens · latest sealed date 2026-08-07.

Every declared token, counted separately

Those tokens are not interchangeable. On 2026-08-07, 7,784 parsed domains name meta-externalagent and 1,288 name meta-externalfetcher — the same operator, a 6,496-domain difference in how widely each is addressed. A site that blocks one has not necessarily said anything about the others.

Blocks outright — the file has a group for that exact product token containing Disallow: / with no overriding allow. Partly restricts — a group exists with at least one non-empty Disallow, but not the whole site. Names & allows — a group exists with no effective Disallow. Everything else is not mentioned: the wildcard * group governs it, and wildcard effects are counted separately rather than folded in, because a site that blocks everything has not made a decision about any AI crawler in particular. Matching is exact and case-insensitive per RFC 9309 — never substring.

Dated readings by declared crawler token

The newest 14 sealed daily readings below keep each crawler token separate. Every count is an aggregate observation of published robots.txt files, not crawler behavior or operator intent. Domains without a parseable response are excluded from the denominator; they are never treated as permissive.