30 Software Vendors, One AI-Crawler Question

When an agency, firm, or contractor asks an assistant how to connect HawkSoft, Drake Tax, Clio, Yardi, or ServiceTitan to the rest of the stack, the assistant can only work from pages it is allowed to read. This series asks 30 named vertical software products, five in each of 6 industries, one question: what does your own published robots.txt say about 11 crawler names from 6 AI companies, read on September 1, 2026.

A robots.txt file is a plain-text request a site publishes about crawling. It is not a licence, not an API policy, and not permission to use the pages. This page is the evergreen series home for that census. The dated sibling is the September 2026 reading.

30 named vendors were asked on September 1, 2026.

Of the 30 named products asked, 26 returned a readable robots.txt. Of those 26, 5 named any AI company at all. Of those 26, 1 blocked any AI company. Of those 26, 21 named no AI company — 80.8% of the 26 that answered, inside a cohort of 30.

Thirty named vertical business-software products, five in each of six industries (Insurance agency software; Accounting and tax software; Healthcare practice software; Legal practice software; Property and construction software; Field service software), asked one question each: what does the robots.txt you publish say about 11 named AI crawlers from 6 AI companies.

The cohort is US Tech Automations’ own named list, written out in the builder, and is not a ranking, a sample, or anything derived from another publisher’s data.

Thirty named companies are not an industry. Nothing here is representative of insurance software, accounting software, or the web, and no figure may be presented as a rate for anything wider than these thirty. A vendor that never names an AI company has NOT allowed it — unmentioned is its own answer and must never be reported as allowed.

A vendor whose address returned an HTML page instead of a plain-text file has not answered at all; it is unknown, never open. robots.txt is a request about crawling, not a licence and not permission to use content, and a vendor blocking a crawler is not evidence about its API, its integrations, or its product. This edition is one reading on one day; a change between editions is a change in what the vendor published, not proof of intent.

The Question This Census Asks

The question is not whether an assistant can log into the product, call an API, or quote a price. The question is only what each vendor published, at the address it already uses for crawlers, about 11 named tokens from 6 companies: OpenAI, Anthropic, Google, Perplexity, Common Crawl, and Apple.

Those tokens are GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, CCBot, and Applebot-Extended. Blocking GPTBot is not a statement about ChatGPT search. Google-Extended is not a statement about Google Search indexing or ranking. Each token is its own row.

Three Answers, Never Two

Every vendor–crawler pair gets one of three answers. Blocked: the vendor names that AI company in its robots.txt and that company’s group contains a bare Disallow: / — the line that shuts the whole site. Allowed: the vendor names that AI company and does not shut the whole site to it. Unmentioned: the vendor has never named that AI company. This is not permission and it is not a block.

Folding unmentioned into allowed is the false headline this series exists to prevent. A file that never says GPTBot has not permitted GPTBot. It has also not blocked GPTBot. The honest word is silence.

A fourth outcome sits outside those three. Unreadable: the address answered with something that is not a plain-text robots.txt — an HTML block page, a challenge, a marketing page, or nothing at all. Unknown, not open. Closed-to-everyone is recorded on its own when the wildcard group contains a bare Disallow: /, and it is never folded into a per-company figure.

Unmentioned is not allowed: 21 of 26 answering vendors, in a cohort of 30, never named an AI company.

1 of 26 answering vendors blocked any of the 11 crawler names.

What Each AI Company Is Told

Shares below are over the 26 vendors that answered, not all 30 asked. The cohort size is 30. Most tokens sit in the same shape: a handful of named allows, a large unmentioned column, and a block count of 0. The exceptions are Google-Extended, CCBot, and Applebot-Extended, each with a blocked count of 1.

Perplexity-User is allowed in 2 files and unmentioned in 24. OAI-SearchBot and PerplexityBot are allowed in 5 files each. Those counts are published requests, not traffic.

Six Shelves, Five Products Each

Each industry row is five named products, not a market. Carry the answered-of-asked denominator on the row. The accounting and tax software cut and the field service software cut walk those shelves product by product.

Field service software is the shelf where naming happens: 3 of 5 products name an AI company, a 60% share of those five, and 0 of those 5 block any. Legal practice software is the opposite silence: 5 of 5 answered, 0 named any AI company. Accounting and tax software is the only shelf whose five include a blocker: 2 of 5 name any AI company, 1 of 5 blocks any.

21 of 26 named no AI company, cohort 30.

Insurance agency software, healthcare practice software, and property and construction software each show a 0% share naming any AI company among the files that answered. Property and construction software also has 2 unreadable hosts of 5 asked, so three readable silences sit next to two unknowns. Do not read a 0% row as “the industry is open” or “the industry is closed.” It is five named products, some of which never produced a file.

Four Addresses That Did Not Answer

The 4 unreadable vendors are named here with what came back instead of a plain-text file. They are unknown. They are not counted in the 26-vendor shares.

NowCerts at www.nowcerts.com: no readable robots.txt (answered 200 as text/html). athenahealth at www.athenahealth.com: no readable robots.txt (answered 403 as text/html). Yardi at www.yardi.com: no readable robots.txt (answered 403 as text/html). Buildertrend at buildertrend.com: no readable robots.txt (answered 403 as text/html).

A 200 is not the file when the body is HTML. A 403 is not a block of GPTBot; it is a failure to publish a parseable robots.txt at the address we asked. llms.txt is not permission and is not used as a substitute for a missing file.

Named Products, One Row Each

Every product in the cohort is listed. Unreadable rows show unknown in the three answer columns because those hosts did not produce a file to score. Closed-to-everyone on every readable row in this edition is no.

The five products that named any AI company are the rows whose allowed or blocked count is not 0: Drake Tax (allowed 9, unmentioned 2), Karbon (blocked 3, allowed 8, unmentioned 0), Jobber (allowed 8, unmentioned 3), Housecall Pro (allowed 8, unmentioned 3), and Simpro (allowed 8, unmentioned 3). Jobber’s file was reached after a redirect to https://www.getjobber.com/robots.txt and still counts as published a readable robots.txt.

Karbon is the blocker row: 3 names blocked, 8 allowed, 0 unmentioned. Drake Tax is the opposite shape inside the same shelf: 0 blocked, 9 allowed, 2 unmentioned. That split is why five accounting and tax products cannot be summarized as open or closed.

Legacy Names Still on File

Two tokens sit outside the 11-name list because the AI company no longer documents them on the crawler page we read. A legacy Anthropic token recorded as anthropic-ai is blocked in 1 readable file. Claude-Web is blocked in 0 readable files.

A leftover token is a string still on a published file on September 1, 2026, not proof of intent.

How the Snapshot Was Sealed

Read from each vendor’s own published robots.txt on September 1, 2026. The fetch identified itself as USTA-DataResearch, not as a browser. The research page that names that work is on the research index. Figures in this edition are copied from the sealed snapshot; nothing is estimated, modeled, or extrapolated.

Thirty named vertical business-software products, five in each of six industries (Insurance agency software; Accounting and tax software; Healthcare practice software; Legal practice software; Property and construction software; Field service software), asked one question each: what does the robots.txt you publish say about 11 named AI crawlers from 6 AI companies. The cohort is US Tech Automations’ own named list, written out in the builder, and is not a ranking, a sample, or anything derived from another publisher’s data.

Thirty named companies are not an industry. Nothing here is representative of insurance software, accounting software, or the web, and no figure may be presented as a rate for anything wider than these thirty. A vendor that never names an AI company has NOT allowed it — unmentioned is its own answer and must never be reported as allowed.

A vendor whose address returned an HTML page instead of a plain-text file has not answered at all; it is unknown, never open. robots.txt is a request about crawling, not a licence and not permission to use content, and a vendor blocking a crawler is not evidence about its API, its integrations, or its product. This edition is one reading on one day; a change between editions is a change in what the vendor published, not proof of intent.

The same sealed-file discipline is used on the broader Accounting sites crawler audit, which is a different cohort and must not be blended into these 30 names.

1. Collect. Fetch each named host’s robots.txt on September 1, 2026 under the research user-agent, following a redirect when one is published, as with Jobber.

2. Parse. Read user-agent groups for the 11 crawler names from 6 AI companies. Score blocked, allowed, or unmentioned. Treat HTML, a challenge, or an empty body as unreadable.

3. Seal. Freeze the answers, the 26-of-30 denominator, and the industry rows as a content-hashed snapshot. Do not infer APIs, pricing, or intent.

5 of 26 answering vendors named any AI company.

Key Takeaways

• 30 named products, 6 industries, 11 crawler names, one day.

• 26 of 30 returned a readable robots.txt.

• 5 of 26 named any AI company; 1 of 26 blocked any.

• 21 of 26 named none — 80.8% of the 26, cohort 30.

• Field service software: 3 of 5 name an AI company.

• Karbon blocked 3 names; Drake Tax allowed 9.

Frequently Asked Questions

Q: How is a sealed snapshot different from opening robots.txt yourself?
A: We asked the same 30 named products, on September 1, 2026, under one user-agent, and froze every answer. Re-querying tomorrow is a new reading. The seal is the dated file, not a live dashboard.

Q: What counts as blocked, allowed, or unmentioned here?
A: Blocked is a named group with Disallow: /. Allowed is a named group without that site-wide shutoff. Unmentioned is a token the file never names. Unmentioned is not allowed. Unreadable hosts are unknown, never open.

Q: Why is 80.8% not a rate for business software?
A: 80.8% is 21 of 26 answering vendors inside a cohort of 30 named products. Thirty named companies are not an industry. The share is over the 26 that answered, not all 30 asked, and not the web.

Q: Does a vendor that blocks a crawler refuse integrations?
A: No. A block is a request about that crawler. It is not evidence about an API, a marketplace, a partner program, or a price. Karbon blocked 3 names and allowed 8; that still says nothing about its product surface.

Q: Who should use this series page?
A: An operations lead whose firm already runs one of these 30 products. Use the industry cut for the five names on your shelf, then watch those files for drift.

Put AI-Access Data to Work

A tax-firm operations manager who already runs Drake Tax, Karbon, or TaxDome should treat this table as a watch list, not a one-time article. Re-read the same five accounting and tax addresses on a weekly cadence and alert the moment a file that today leaves 11 names unmentioned adds a Disallow: / group, or the moment Karbon’s blocked count of 3 moves.

A field-service operations manager running ServiceTitan, Jobber, or Housecall Pro has a different trigger: 3 of those five already name an AI company, and 0 of 5 block any. The recurring job is to catch the first block on a shelf that currently publishes none.

A vendor marketing lead at one of these 30 companies should diff their own robots.txt against this sealed row whenever they ship a docs or help-center change, because the assistant their buyers ask is reading from whatever that file currently requests.

US Tech Automations automates that monitoring with scheduled robots.txt reads, change alerts, and an access-policy workflow on agentic workflows. The company homepage is the public door for that work.

Source: US Tech Automations Research — computed from the sealed vendor AI-access snapshot, September 1, 2026.