Why Reused AI Tests Overstate Reliability: Our 10-Process Sweep

A declared test can stop measuring ordinary performance after an automated workflow has seen it. The workflow may recognize the input, memorize an answer, inherit the example through a model update, or receive special handling from operators. The reported score can rise while ordinary reliability does not.

That problem is commercially interesting only if a fresh test has objective truth, looks like ordinary work, is used once, and reveals a failure whose avoided loss materially exceeds the unit price. We screened for that exact product. We did not find a class ready to sell.

What we screened

Our July 29, 2026 sweep recorded 16 public sources across 10 unrelated process classes:

1. private LLM-agent tool use;

2. invoice extraction;

3. EDI mapping and transformation;

4. data-pipeline quality;

5. web transactions;

6. cross-browser UI behavior;

7. mobile-app device behavior;

8. non-safety visual inspection;

9. conversational-bot routing; and

10. laboratory proficiency testing.

The laboratory class was killed because health and safety records are outside the product boundary. The other nine became evidence jobs. None became an offer.

The public implementation and tests are available in the Unseen Challenge Foundry repository.

What the evidence actually supports

The contamination mechanism is real. An EMNLP Findings paper reports that benchmark contamination can overstate model performance relative to uncontaminated comparisons. That supports the decay hypothesis; it does not prove that a buyer will pay for a stream of fresh challenge units.

The market already pays for evaluation infrastructure. LangSmith, Braintrust, Confident AI, Datadog, and Checkly publish evaluation, synthetic-testing, or monitoring prices. Those are adjacent receipts, not exact receipts. Their priced objects are platforms, traces, checks, or usage—not independently validated, never-reused challenges sold as recurring inventory.

That distinction is why our result is “return none.” Adjacent spending proves a budget category. It does not prove the proposed SKU.

The four gaps that prevented an offer

Generated test cases are insufficient. A plausible-looking challenge with an unverified answer can create false alarms, reward the wrong behavior, or misdiagnose a correct workflow.

What a valid challenge unit would contain

A qualifying unit needs six frozen elements:

• the ordinary input visible to the workflow;

• the hidden variable intended to trigger the failure;

• independently reproduced objective truth;

• an indistinguishability envelope defining how closely it matches normal work;

• an exposure ledger proving the unit has not been reused; and

• an automatic answer-key release after response or deadline.

The buyer must control the process and have authority to insert the unit. Workers, applicants, patients, outsiders, and unconsenting participants are excluded. The unit detects a local process failure; it does not grade, certify, train, rank, or select a person or vendor.

A practical buyer-side decay experiment

An owner of an automated invoice extractor, agent workflow, or transformation pipeline can run a matched test without buying a product:

1. Freeze the workflow version and evaluation metric.

2. Build two challenge sets with the same schema, difficulty distribution, and independently reproduced answers.

3. Expose one set during tuning or rehearsal; keep the other sealed.

4. Preregister the minimum recognition effect that would matter operationally.

5. Compare accuracy, exception rate, handling time, and failure severity.

6. Repeat with a distinct sealed set before concluding that the effect reproduces.

If the recognized and fresh sets perform the same within the preregistered floor, freshness is not the valuable feature for that process. If they diverge, the next question is economic: did the fresh unit reveal a failure worth at least ten times its price?

What buyers can do now

There is no commercially admitted Unseen Challenge class today, so there is no specialized checkout pretending otherwise. Owners with a matching automated process can submit the process and loss evidence. Do not include production, confidential, regulated, or personal data in the form.

Teams that need ordinary workflow orchestration rather than single-exposure testing can review the self-managed platform plans and live Stripe checkout. That is a different product and is labeled separately.

Bottom line

Freshness is not automatically a product. The first sweep found a credible failure mechanism and established evaluation budgets, but it did not find exact recurring-unit receipts, reproduced truth, documented buyer loss, or a matched recognition-decay result. The honest commercial state is zero admitted classes until those gates pass.