Do Productivity Sites Block AI Crawlers? 1 of 10 Do

The Productivity category delivers the clearest signal of any category in this analysis: every single one of the 10 sites checked returned a parseable robots.txt file, yet only notion.so — 1 of those 10 — blocks any AI crawler at all. That produces a 10% block rate, the lowest of any category in the June 2026 Closing Web edition that returned more than a handful of parseable robots.txt files. The Productivity sector, as a category, has effectively chosen to stay open.

The contrast with the corpus is sharp. Across all 354 sites with parseable robots.txt files in this 418-site, 40-category edition, 139 block at least one AI crawler — a 39.3% rate. Productivity sits far below that line, more than twenty-five percentage points lower, with near-universal robots.txt participation and near-universal AI-crawler access.

Methodology and Sealed Data

Before the analysis: this report is based on a single sealed snapshot of public robots.txt files taken June 14, 2026. In this pipeline, nothing is estimated, modeled, or extrapolated. Every numeral in this post is a verbatim count from that snapshot, stored with sha 27ca61d890a647db.

1. Crawl. Each site was fetched for its robots.txt on June 14, 2026.

2. Parse. Each file was parsed for user-agent directives targeting known AI crawlers.

3. Classify. Sites were categorized as: no parseable robots.txt, parseable with at least one AI block, or parseable with no AI blocks.

4. Seal. The dataset was content-hashed (sha 27ca61d890a647db) for auditability.

1 of 10 Productivity sites block at least one AI crawler.

Productivity sites post a 10% AI-crawler block rate.

Corpus-wide, 139 of 354 sites block at least one AI crawler.

Key Takeaways

The Productivity category stands out as one of the most open categories in the dataset, combining complete robots.txt adoption with near-complete AI-crawler access.

Only 1 of 10 Productivity sites blocks any AI crawler — a 10% block rate.

Notion.so is the sole Productivity site blocking at least one AI crawler in the June 14, 2026 snapshot.

The Productivity block rate of 10% is far below the corpus average of 39.3% across all 354 sites.

What makes this category distinctive is the combination of two facts in the same direction: every site published a robots.txt, and almost none of them used it to restrict AI crawlers. That coherence is unusual. In categories like Outdoor, low robots.txt coverage coexists with a high block rate among those that do publish. In Productivity, universal robots.txt publication coincides with almost universal AI access. These are SaaS businesses that have either decided AI training access is neutral to their model or have affirmatively concluded that openness serves their interests in AI-driven feature development.

"1 of 10 Productivity sites blocks at least one AI crawler — the lowest block rate of any category with complete robots.txt coverage in this edition."

"All 10 Productivity sites returned a parseable robots.txt in the June 14, 2026 snapshot, making it one of the most robots.txt-compliant categories in the 418-site corpus."

The One Blocker — and the Nine Allowers

Notion.so is the sole Productivity site blocking at least one AI crawler. Notion is a collaborative workspace that has built a substantial library of user-generated templates, public wikis, and community content. Its decision to restrict at least one AI crawler may reflect concern about the scale at which its public-facing content could be scraped for training purposes, particularly given the volume of structured, templated content that sits on public notion.so subdomains.

The nine Productivity sites that allow every AI crawler — according to their robots.txt directives at this snapshot date — are slack.com, asana.com, trello.com, monday.com, zapier.com, airtable.com, clickup.com, todoist.com, and evernote.com. These span project management, team communication, automation, and personal productivity. The range is notable: from enterprise-grade SaaS (Slack, Asana, Monday) to individual productivity tools (Todoist, Evernote) to workflow automation (Zapier). All allow every AI crawler examined in this dataset.

slack.com, asana.com, trello.com, monday.com, zapier.com, airtable.com, clickup.com, todoist.com, and evernote.com each allow every AI crawler in the June 2026 snapshot.

The per-site breakdown across all 10 Productivity sites is shown below.

The Productivity sector's open posture likely reflects several structural realities: most user data lives behind authentication (and thus behind robots.txt even without explicit bot directives); the public-facing content is largely marketing and documentation; and these companies are themselves AI-forward, building AI features natively into their products and potentially valuing broader AI ecosystem participation.

Where Productivity Sits Among All 40 Categories

Productivity at 10% lands near the bottom of the 40-category ranking, above only three categories with a 0% block rate (Nonprofit, Streaming, Dating). In a corpus where 39.3% of sites block at least one AI crawler, a 10% rate in a category with complete robots.txt participation is remarkable.

The contrast with content-heavy categories is stark: Gaming blocks 8 of 9 sites, News blocks 14 of 17. The structural difference is that those categories generate indexable content as their primary product. Productivity SaaS generates value through software features and user data, not through publicly crawlable editorial content.

For contrast on how an adjacent-but-different category handles AI access, see how Genealogy sites approach blocking — a category that mixes commercial archives with community-built trees, yielding a 40% rate.

Corpus-Wide Bot Blocking: Who Faces the Most Resistance

These bot-level figures cover all 354 sites with parseable robots.txt across the full 418-site corpus — not Productivity specifically.

Across all 354 sites, CCBot (Common Crawl) is blocked 109 times — 30.8% of the corpus. ClaudeBot (Anthropic) is blocked 96 times. GPTBot (OpenAI) and Bytespider (ByteDance) are each blocked 83 times. At the operator level, Common Crawl faces blocks from 109 sites, Anthropic from 104, Meta from 89, OpenAI from 87, ByteDance from 83. Given that the Productivity category has only 1 blocker, these corpus-wide numbers tell a story driven by other, more restrictive categories.

Frequently Asked Questions

Q: Why would Productivity SaaS companies allow AI crawlers when so many other categories block them?

A: Most of a SaaS company's valuable data — user workspaces, task lists, communications — lives behind authentication and is not accessible to robots.txt-governed crawlers regardless of the directive. The public-facing robots.txt applies only to public URLs. A Productivity company that allows AI crawlers through robots.txt is likely not exposing sensitive user data; it is allowing crawlers to index its marketing pages, documentation, and any public-facing product content.

Q: What makes notion.so different from the other nine Productivity sites?

A: Notion hosts a significant volume of public-facing, user-generated content on public subdomains — templates, wikis, shared pages — that is accessible without authentication and that constitutes a meaningful body of structured knowledge. That public content footprint distinguishes Notion from tools like Slack and Asana, where user-generated data is almost entirely behind login.

Q: Does the Productivity sector having only 1 blocker suggest these companies are unconcerned about AI?

A: Not necessarily. It may reflect that their robots.txt-addressable content is not their primary competitive asset. These companies are investing heavily in building AI features natively. A company embedding AI throughout its product has different incentives regarding AI crawler access than a publisher whose content library is the product itself.

Q: How should an enterprise buyer interpret this data when evaluating Productivity tools?

A: The robots.txt posture of a SaaS vendor tells you their policy on public-content crawling, not their data handling inside the application. For enterprise data concerns, the relevant documents are the vendor's data processing agreements and security certifications, not their robots.txt file.

Q: How current is this data?

A: The snapshot was taken on June 14, 2026. Robots.txt files can change at any time. Any of the nine allower sites could add AI-crawler restrictions tomorrow. Monitoring for drift from this sealed baseline requires re-crawling on a scheduled basis.

Put AI-Access Data to Work

The Productivity category data is relevant for three audiences with distinct monitoring needs.

Enterprise IT and procurement teams evaluating SaaS tools that may touch AI training pipelines need a live view of which vendors have changed their public-content access posture. A vendor that adds a ClaudeBot block may signal a shift in their AI partnership strategy — worth flagging in a vendor review. The workflow: re-crawl the 10 Productivity sites weekly, alert the procurement team on any robots.txt change, and cross-reference against the sealed June 14 baseline documented here.

AI product teams at companies building features on top of productivity tools need to know whether their data sources are starting to restrict crawlers. If a competitor of notion.so adds AI-crawler blocks, that signals a possible content-protection push worth monitoring for product and business development implications. The workflow: track robots.txt changes across all 10 sites in this category, flag any new disallow directive, and route the signal to the AI product and partnership teams on a weekly cadence.

Competitive intelligence leads at Productivity vendors themselves benefit from monitoring peer companies. If asana.com or clickup.com adds a blocking directive that notion.so already has, that represents a policy shift worth understanding. This category compares interestingly with how Wedding sites handle AI access, where the block rate is five times higher — primarily because editorial content drives different protection incentives. See also how Outdoor sites handle AI access for contrast with a mixed editorial/platform category.

US Tech Automations automates this monitoring through scheduled robots.txt crawls that diff against sealed baselines like this one, alerting your team the moment any site in a tracked category changes its AI-access posture. Set up category monitoring at /platform/agentic-workflows.

Source: US Tech Automations Research — Closing Web edition; figures are verbatim counts from public robots.txt files sealed June 14, 2026 (snapshot sha 27ca61d890a647db).