Which sites block Webz.io's crawlers?

2 declared product tokens · latest sealed date 2026-08-07.

On 2026-08-07, of the 61,552 panel domains that served a parseable robots.txt, the most widely named Webz.io token is addressed by 2,509 of them, and at most 2,303 disallow any single Webz.io token site-wide. Per-token figures below — they are not the same number, and adding them up would double-count the sites that name several.

What the denominator is. On 2026-08-07 we attempted /robots.txt on 100,000 panel domains. 24,130 could not be reached at all (DNS failure, TLS failure, timeout, refused connection) and 14,318 answered with something other than a 200. Both of those groups have no observed AI-crawler policy, and neither is counted as allowing anything — an unreachable site is UNKNOWN, not permissive. Every crawler figure below is out of the 61,552 domains that returned a robots.txt we could parse on that date.

Every declared token, counted separately

Token	Declared purpose	Names it	Blocks outright	Partly restricts	Names & allows	% of parsed
omgili	model training	2,509	2,303	157	49	3.7%
omgilibot	model training	2,287	2,088	150	49	3.4%

Those tokens are not interchangeable. On 2026-08-07, 2,509 parsed domains name omgili and 2,287 name omgilibot — the same operator, a 222-domain difference in how widely each is addressed. A site that blocks one has not necessarily said anything about the others.

Blocks outright — the file has a group for that exact product token containing Disallow: / with no overriding allow. Partly restricts — a group exists with at least one non-empty Disallow, but not the whole site. Names & allows — a group exists with no effective Disallow. Everything else is not mentioned: the wildcard * group governs it, and wildcard effects are counted separately rather than folded in, because a site that blocks everything has not made a decision about any AI crawler in particular. Matching is exact and case-insensitive per RFC 9309 — never substring.

What this page cannot tell you. The token list is frozen at methodology census-v1. If Webz.io has shipped a crawler token since that freeze, we do not observe it and it is absent here — that is a gap in our coverage, not evidence the token is unused. We also observe only what robots.txt says; we do not observe what any crawler actually fetched.

Get told when Webz.io change their AI crawler rules

Sites publish today's robots.txt and overwrite it — no archive, no history, no notification. We keep a dated daily copy. Leave your email and we'll send you what moved for Webz.io: who started blocking, who stopped, and on which day.

Email me changes for Webz.io →

Work email only. No newsletter, no obligation.

Prefer to just buy it? See the sealed-archive monitoring offers, priced, with a dated sample.

Dated readings by declared crawler token

The newest 14 sealed daily readings below keep each crawler token separate. Every count is an aggregate observation of published robots.txt files, not crawler behavior or operator intent. Domains without a parseable response are excluded from the denominator; they are never treated as permissive.

Sealed date	Token	Names it	Blocks outright	Partly restricts	Names & allows	Parsed that day
2026-08-07	`omgili`	2,509	2,303	157	49	61,552
2026-08-07	`omgilibot`	2,287	2,088	150	49	61,552
2026-08-06	`omgili`	2,504	2,298	157	49	61,421
2026-08-06	`omgilibot`	2,279	2,081	149	49	61,421
2026-08-05	`omgili`	2,502	2,295	159	48	61,527
2026-08-05	`omgilibot`	2,300	2,100	152	48	61,527
2026-08-04	`omgili`	2,502	2,300	156	46	61,620
2026-08-04	`omgilibot`	2,301	2,103	151	47	61,620
2026-08-03	`omgili`	2,497	2,292	158	47	61,533
2026-08-03	`omgilibot`	2,293	2,094	152	47	61,533
2026-08-02	`omgili`	2,501	2,297	158	46	61,559
2026-08-02	`omgilibot`	2,300	2,103	151	46	61,559
2026-08-01	`omgili`	2,507	2,302	158	47	61,326
2026-08-01	`omgilibot`	2,294	2,096	152	46	61,326
2026-07-31	`omgili`	2,518	2,314	157	47	61,595
2026-07-31	`omgilibot`	2,300	2,103	151	46	61,595
2026-07-30	`omgili`	2,526	2,323	156	47	61,444
2026-07-30	`omgilibot`	2,311	2,116	150	45	61,444
2026-07-29	`omgili`	2,520	2,317	156	47	61,492
2026-07-29	`omgilibot`	2,320	2,124	151	45	61,492
2026-07-28	`omgili`	2,519	2,316	156	47	61,562
2026-07-28	`omgilibot`	2,313	2,119	149	45	61,562
2026-07-27	`omgili`	2,519	2,317	156	46	61,440
2026-07-27	`omgilibot`	2,311	2,117	150	44	61,440
2026-07-26	`omgili`	2,514	2,311	156	47	61,573
2026-07-26	`omgilibot`	2,302	2,108	150	44	61,573
2026-07-25	`omgili`	2,520	2,317	156	47	61,584
2026-07-25	`omgilibot`	2,304	2,110	150	44	61,584

Read this before quoting a trend. We hold 55 sealed daily snapshots between 2026-06-09 and 2026-08-07, out of 60 calendar days. 5 calendar day(s) inside that span have no collection run at all: 2026-06-11, 2026-06-12, 2026-06-19, 2026-07-04, 2026-07-15. Consecutive rows in every table below are consecutive SEALED dates, not consecutive days — where a gap falls, that row's diff covers more than 24 hours, and the span is printed with it. We do not interpolate across a day we did not collect.

Compiled from each site's own published /robots.txt, /ai.txt, /llms.txt and /.well-known/tdmrep.json, fetched daily over a frozen 100,000-domain research panel (source list: Tranco; no Tranco rank is republished here). Verdicts are mechanical readings of what those files say under RFC 9309 grouping with exact product-token matching — they are not legal advice, not a statement about any site's intentions, and not a claim about what any crawler actually did. A robots.txt is a request, not an access control. Figures are provided for informational purposes only and carry no warranty of accuracy or completeness.

AI crawler policy home · What changed · How this is collected · US Tech Automations.