18,589 Resolved Predictions, Scored: USTA Calibration Scorecard — July 2026

In June we published a prediction ledger with a score of exactly nothing: 48,726 hash-sealed predictions and an honest admission that none had resolved yet. We wrote then that whatever the numbers turned out to be, they would be published next to the original seals. This is that publication — the first scored edition — and it includes a number that got worse as the sample grew, because that is what actually happened.

Two things are new. First, the scorecard: six dated calibration issues, June 23 through July 9, 2026, each computed by a deterministic scorer over the sealed stores, scoring predictions whose outcomes have since become observable. Second, the anchor: every seal commitment we have ever produced — 405 of them — is now committed in a single manifest whose hash was submitted to public OpenTimestamps calendar servers on July 9, 2026, on its way into the Bitcoin blockchain. "These predictions were sealed before their outcomes" no longer requires trusting our database, or us.

Key Findings

• AUC 0.657 on 18,589 resolved prediction-market questions in the July 9 issue — up from 3,909 resolved questions in the first issue, a nearly fivefold larger scored sample.

• The discrimination score declined as the sample grew: 0.816 → 0.657. We publish the full series rather than the flattering endpoint, and explain what we know about why below.

• Calibration error stayed moderate throughout: expected calibration error between 0.062 and 0.100 across all six issues, Brier score between 0.075 and 0.109.

• Permit-outcome loops remain honestly unscored: 3,334 resolved permit records carry zero observed positive outcomes so far, so no AUC is reported for them — a null, not a boast.

• All 405 seal commitments are now Bitcoin-anchored via OpenTimestamps: 387 prediction batch seals, 4 simulation batch seals, 6 calibration issues, and 3 Datacenter Buildout Index issues, committed in one downloadable manifest with its timestamp proof.

AUC 0.657 · Brier 0.109 · ECE 0.100 · 18,589 resolved questions · 3,219 positives · six sealed issues · every seal Bitcoin-anchored.

The Headline Series, Including the Part That Fell

The most-resolved loop in the ledger scores sealed terminal probabilities on public prediction-market questions against the outcomes those markets eventually settled to, at a 12-month horizon. Here is every issue we have published, in full:

The honest read: the first issue's 0.816 was computed on the earliest-resolving 3,909 questions, and early resolvers are a biased sample — questions that settle quickly tend to be the easy ones. As the resolved pool widened toward the full sealed set, discrimination reverted to roughly 0.66 and has been stable there across the last four issues. An AUC of 0.657 means the sealed probabilities order outcomes meaningfully better than chance (0.5), and the Brier and calibration-error rows say the stated probabilities are reasonably honest about their own uncertainty. It is a modest, real number.

We could have published only the June 23 issue and led with 0.816 indefinitely. The entire architecture of this ledger exists to make that move impossible to hide: every issue is sealed with a content hash at generation time, the hashes are now anchored externally, and a skipped or rewritten issue would be visible to anyone holding the manifest.

The Full July 9 Scorecard

The scorer runs over every prediction loop in the sealed stores, not just the flattering one. All six, verbatim:

Three notes, stated plainly. The permit loops still have zero observed positive outcomes — the events they predict (such as property transfers following a permit) take months to surface in county records, so those rows report a null instead of a simulated score, exactly as the June ledger promised. The datacenter simulation row is a genuinely bad result on a tiny sample: three resolved cases, all positive, against near-zero predicted probabilities — a Brier score of 1.000, the worst possible. It stays in the table because the alternative is the ledger becoming marketing.

Every Seal, Anchored to Bitcoin

Until this week, "sealed before the outcome" rested on our append-only store: you could verify a published record matched its hash, but the claim that the hash existed on a given date still required trusting us. That gap is now closed.

On July 9, 2026 we generated a priority manifest — one canonical JSON file listing the seal commitment of every artifact this research program has produced:

The manifest's own content hash is c634d1ebacfef3a4655c6009f56f402e3b990267a1bb3098e9b3dde0af76df0d, with a Merkle root of e2cd10ac9421bb03a5fcda75a78a02e1a3d69a6e4cb706dc07f993d66b434571 over the entry hashes. That single hash was stamped via OpenTimestamps against four independent public calendar servers, which aggregate submissions into Bitcoin transactions. Anchoring one manifest anchors every seal listed inside it.

Anyone can verify this without asking our permission:

1. Download the manifest and its .ots proof.

2. Install the open-source client: pip install opentimestamps-client.

3. Run ots verify priority-manifest-2026-07-09.json.ots -f priority-manifest-2026-07-09.json.

4. The proof attests, via the Bitcoin blockchain, that this exact file — and therefore every seal hash in it — existed on the anchor date.

One caveat we will not bury: as published today, the proof carries pending calendar attestations; Bitcoin aggregation typically completes within hours, after which the proof upgrades to a full block-level attestation (ots upgrade). We will replace the published proof file with the upgraded version once the block confirmation lands, and the upgraded proof will verify against this same manifest byte-for-byte.

What This Does and Does Not Claim

It does claim: the predictions were sealed before outcomes were observable, the scorer is deterministic, the series is complete, and the existence-at-time of every seal is now provable against a public blockchain rather than our word.

It does not claim: that 0.657 is impressive (it is modestly better than chance, honestly calibrated), that the permit loops work (unscored until positives exist), or that past discrimination guarantees anything forward. A track record's value is that it removes the need to take our word for things — in both directions.

Frequently Asked Questions

Q: Why publish a number that got worse?
A: Because the alternative destroys the only thing this ledger is worth. The first issue's AUC of 0.816 was computed on the earliest 3,909 resolutions, a sample biased toward easy questions; the stable figure on 18,589 resolutions is 0.657. A ledger that stopped publishing when the number fell would be indistinguishable from every vendor highlight reel, and the seals would prove our selectivity rather than our discipline.

Q: What does AUC 0.657 actually mean?
A: Given a randomly chosen question that resolved YES and one that resolved NO, the sealed probability ranked the YES question higher 65.7% of the time. Chance is 50%. It is a real signal, honestly calibrated (ECE 0.100), and we describe it as modest because it is.

Q: What does the OpenTimestamps anchor prove — and not prove?
A: It proves existence at a time: that the manifest, and every seal hash inside it, existed no later than the Bitcoin block that attests it. It does not prove the predictions are good, that the underlying data is accurate, or anything else about quality. It removes exactly one trust assumption — that we could have backdated our own database — and that is all it removes.

Q: Can I verify a single prediction, not just the manifest?
A: Yes, in two steps: verify the manifest against its Bitcoin proof (four commands, listed above), then verify the individual record against the batch seal listed in the manifest — each sealed batch commits its per-record content hashes. Both steps require only open-source tooling and arithmetic.

Q: Why are the permit loops still unscored?
A: Zero positive outcomes have been observed against them so far — the outcomes they predict take months to appear in public records. Reporting "n/a" on 3,334 resolved records with 0 positives is the honest treatment; anything else would be a simulated score. When positives accrue, they will be scored against the same sealed set, whatever they show.

Q: What happens in the next edition?
A: The same scorer, the same fixed sealed sets, more resolutions — published in this same format, appended to this same series, with new issues added to a re-anchored manifest. If the number falls further, that appears here too. That is the deal, and this edition is evidence the deal gets kept when the number is unflattering.

Cite This Report

Canonical URL: https://ustechautomations.com/resources/blog/prediction-calibration-scorecard-july-2026 — attribute to "US Tech Automations Research". Machine-readable twins: JSON · CSV. Sealed snapshot sha256: c3fceb102c1ef8f820d670ff98c8020bcf90bb18c467fdc688587f43a7a61ed2. Anchored manifest: priority-manifest-2026-07-09.json · Bitcoin timestamp proof.

The sealed snapshots behind this scorecard also drive our research hub, and the raw permit corpus is browsable at permits.ustechautomations.com. Teams that want sealed, verifiable prediction data wired into their own workflows can contact us — and we will tell you what the data cannot support, too.

Source: US Tech Automations Research — computed from sealed calibration issues, June 23 – July 9, 2026, and the Bitcoin-anchored priority manifest of July 9, 2026.