Do Antiques Sites Block AI Crawlers? 3 of 8 Do

Most Antiques sites keep their doors open to AI. Of the 10 Antiques sites we checked, 8 returned a parseable robots.txt, and 3 of those disallow at least one AI crawler — a 37.5% block rate. The majority of published policies allow every crawler, which puts the answer firmly on the permissive side of the corpus.

The distinctive read is that the biggest marketplaces are the ones staying open. The largest dealer and auction-aggregation platforms in the set allow every crawler, while gating is concentrated among reference and specialty-sale sites. For a vertical built on discoverability, that posture makes commercial sense. This is a sealed-snapshot reading, not an estimate.

3 of 8 Antiques sites block at least one AI crawler.

A sealed snapshot is a hashed, point-in-time copy of each site's public robots.txt, frozen so the figures cannot drift afterward. Every number below is read straight from that set under sha c60e706824d5d127. The hash matters because policies move: a marketplace could revise its file next week, but this report is anchored to exactly what the files said on June 14, 2026, and the seal lets a reader confirm the figures have not been changed since.

Who Gets Disallowed Here

Three sites disallow at least one tracked AI crawler: rubylane.com, kovels.com, and ha.com. Five sites publish a policy allowing every crawler: 1stdibs.com, liveauctioneers.com, worthpoint.com, antiquetrader.com, and chairish.com. Two sites, invaluable.com and theantiquesalmanac.com, returned no parseable robots.txt and make no machine-readable statement.

The largest marketplaces — 1stdibs.com, liveauctioneers.com, chairish.com — all allow every AI crawler.

The gating sites have a different logic. kovels.com is a pricing-and-identification reference, ha.com runs structured auction-result archives, and rubylane.com is a curated dealer marketplace. Each holds data with curation value worth protecting, which is exactly where AI-access friction tends to concentrate.

Read each blocker on its own terms. kovels.com built decades of price guides and maker marks into a searchable reference; that body of pricing knowledge is the business, and an AI assistant reciting it for free undercuts the subscription. ha.com publishes structured records of what items actually sold for, a dataset worth shielding from bulk extraction. rubylane.com curates a vetted roster of dealers and listings, a quality signal it has reason to keep proprietary.

By contrast, 1stdibs.com, liveauctioneers.com, worthpoint.com, antiquetrader.com, and chairish.com all run on volume discovery — the wider their listings travel, the more buyers they reach — so an open policy is simply good for sales. The 37.5% rate is the boundary between selling access to data and selling the goods themselves.

Why Antiques Lands Where It Does

A 37.5% block rate puts Antiques above the corpus norm but well under the heavy gaters. The reason is structural: a marketplace lives on being found. The more an AI assistant can read a listing's provenance and price, the more often it routes a collector to the seller. That incentive keeps the big platforms open.

Antiques sites post a 37.5% AI-crawler block rate.

Gating is the exception, reserved for sites whose product is the data itself — pricing guides, auction-result histories, curated dealer rolls. Read that way, Antiques is a marketplace category with a reference-site minority, and the 37.5% figure is the visible seam between those two models.

The pattern mirrors another collector vertical. The vinyl record sites gate nothing at all, sitting at the 0% floor because they are almost entirely marketplaces and catalogs with no reference-data layer to protect. Antiques carries that same marketplace logic but adds a handful of data-holding references, which is why it lands above zero rather than at it. The three blockers are the entire difference between the two categories, and they are exactly the sites where the data, not the listing, is the product.

Corpus-wide, 220 of 670 sites block at least one AI crawler, a 32.8% rate Antiques sits just above.

How Antiques Compares to Its Neighbors

The focused window below centers Antiques among the categories ranking closest to it on block rate. Every value is verbatim from the sealed cross-category set, named by category, with no rank column.

Antiques sits in a tight band with Comics, Whiskey, and Golf — all at 37.5% on the same published base of 8. These are collector-and-enthusiast categories, which is the most coherent grouping in the window: hobbies where most sites want reach and only a data-holding minority gates. That shared shape is the signal, not a quirk of Antiques alone.

Just below the band, Travel, Agriculture, and Wine all land at 33.3% on a slightly larger published base, and the gap between them and Antiques is one site, not a trend. That closeness is the honest read: Antiques is an ordinary collector category that behaves like its peers, gating a data-holding minority while the marketplaces stay open.

The distinctive detail is not the rate but the identity of the three blockers — a price guide, an auction-result archive, and a curated dealer roster — which together explain the entire figure and tell a working collector exactly which sources may resist AI access.

The Bot-Level Picture Across the Corpus

When Antiques sites gate, the corpus-wide pattern shows which crawlers operators disallow most. The focused bot cut below — across all 670 sites — leads with the busiest agents; we render the bot view here so this report differs from its operator-focused siblings.

CCBot, the Common Crawl agent, tops the list — the broad archival crawler a pricing reference like kovels.com would most want to keep out of its data, since wide crawls are what feed downstream training corpora. ClaudeBot and GPTBot follow as the named agents of the leading assistant operators, with Bytespider and Meta-ExternalAgent rounding out the top group.

This cut is corpus-wide across all 670 sites, not Antiques-specific, so it describes where the whole field aims its gating rather than what the three Antiques blockers each chose. For an auction or pricing platform weighing its own policy, the value is in the ordering: these are the crawlers the rest of the web decided to stop first.

How the Snapshot Was Sealed

We fetched each site's robots.txt, parsed its user-agent and disallow directives, and recorded which of our 9 tracked AI crawlers each blocks. The counts are verbatim from those files; nothing is estimated, modeled, or extrapolated. A site counts as a blocker only when its published file disallows a tracked crawler. Corpus-wide, 152 sites also publish an llms.txt — 22.7% of the 670 with robots.txt. The full sweep spanned 803 sites across 80 categories, sealed under sha c60e706824d5d127 on June 14, 2026.

For Antiques the coverage caveat is the same shape as the rest of the study and worth being explicit about. The 37.5% rate is computed over the 8 published policies, not the 10 sites we checked. invaluable.com and theantiquesalmanac.com returned no parseable file, so they are excluded from the rate rather than scored either way.

That matters because invaluable.com is a major auction-aggregation platform; its silence is not evidence that it allows or blocks anything, only that it made no machine-readable statement at snapshot time. The figure speaks to the published set — three blockers and five allowers — and to nothing beyond it.

Frequently Asked Questions

Q: Does ha.com blocking a crawler in robots.txt actually stop it?

A: Only by agreement. robots.txt is an honor-system standard — compliant crawlers obey a disallow line, but nothing in the file enforces it technically. ha.com's entry records its intent to exclude those crawlers, which the major operators generally honor.

Q: Which Antiques sites block AI crawlers and which allow them?

A: Three block at least one: rubylane.com, kovels.com, and ha.com. Five allow all: 1stdibs.com, liveauctioneers.com, worthpoint.com, antiquetrader.com, and chairish.com. Those 8 published policies yield the 37.5% rate.

Q: Why do the big antiques marketplaces leave crawlers open?

A: Platforms like 1stdibs.com, liveauctioneers.com, and chairish.com profit when an AI assistant surfaces their listings to collectors, so an open policy drives sales. The gating sites — kovels.com, ha.com — instead protect pricing and auction-result data that is their product.

Q: Why does Antiques gate a little above the corpus average?

A: At 37.5%, Antiques runs just over the 32.8% corpus rate because a minority of its sites are data-holding references, not marketplaces. Its 37.5% neighbors Comics, Whiskey, and Golf share that collector-category shape, where reach beats control for most sites.

Put AI-Access Data to Work

An antiques-auction-platform data lead can monitor whether the open marketplaces — 1stdibs.com, liveauctioneers.com, chairish.com — ever add a disallow line, re-crawling weekly and alerting on the first new crawler token, because a rival platform gating AI search reshapes how listings surface to buyers. The recurring schedule is what turns this from a one-off audit into a live signal: the three blockers are known today, but the moment a fourth marketplace flips is the event that matters.

A collectibles-pricing analyst can watch the reference blockers like kovels.com and ha.com for tightening that would close off comparable data, treating any broadening of their disallow rules as a sign that pricing references are pulling further out of AI reach. The edge is seeing the first marketplace cross the line the week it does, well before a competitor reading a stale snapshot notices the shift.

A data-pipeline engineer ingesting listing or provenance data can confirm each source still permits access before every scheduled pull, preventing silent breakage when an open marketplace like worthpoint.com or antiquetrader.com changes its policy between runs.

US Tech Automations automates that watch with scheduled robots.txt and llms.txt crawls, change alerts, and an AI-access policy dashboard that flags new disallow tokens the moment they land. See the build on the platform agentic-workflows page, and compare Antiques against the quilting sites at an even split and the tattoo sites that gate the least.

Corpus-wide, 220 of 670 sites block at least one AI crawler.

Key Takeaways

• Of 10 Antiques sites checked, 8 returned a parseable robots.txt and 3 block at least one AI crawler — a 37.5% block rate.

• Blockers rubylane.com, kovels.com, and ha.com hold pricing, auction-result, or curated dealer data.

• Allowers include the largest marketplaces: 1stdibs.com, liveauctioneers.com, worthpoint.com, antiquetrader.com, and chairish.com.

• Antiques sits just above the 32.8% corpus rate, in a tight 37.5% band with Comics, Whiskey, and Golf.

• CCBot is the most-blocked crawler across all 670 sites, the agent a pricing reference most wants to gate.

Source: US Tech Automations Research — Closing Web edition; figures are verbatim counts from public robots.txt files sealed June 14, 2026 (snapshot sha c60e706824d5d127).