Public data index › AI Answer Repeatability Report
Turn repeated AI answers into one defensible internal QA table.
For a GEO or SEO agency lead who already collects assistant answers for a client but cannot give an outside monitoring vendor account access.
One supplied session can describe that session; it cannot describe what every user sees.
The fixed engagement
$249 one time
Before payment, you freeze one client domain, exactly five prompts, exactly three fresh-session manual runs per prompt, and a fixed set of one, two, or three named assistants. That creates a planned panel of 15, 30, or 45 records. The assistant set and denominator never change after results are visible.
- Private PDF and CSV within three business days after scope confirmation.
- Planned, delivered, excluded, unreadable, and usable record counts.
- Raw citation targets beside their string-normalized forms.
- Observed citation counts written as k of n, never a rate or percentage.
- Repeat-disagreement counts for each prompt and assistant.
- A frozen protocol for a later remeasurement.
The buyer constraint this is built for
You already run a small question panel by hand for a client. You cannot grant an outside vendor login access, and the client does not permit a full account export to leave the agency. You can, however, share the agreed prompt responses under written client authorization. The value here is not discovering a link you already saw. It is a frozen normalization rule, tested matcher, counted gap register, and reusable remeasurement specification that your team does not have to invent.
The pre-payment attestation names the agency, authorizing person, one client domain, the exact assistant set, and the allowed transcript bundle. It confirms that the agency collected the sessions manually, is authorized to share them, and removed account identifiers and third-party personal information. If the same confidentiality rule that blocks account access also blocks transcript transfer, the request is declined.
What you supply — and what you do not
- Only the 15, 30, or 45 agreed prompt-response records in the frozen manifest.
- Timestamp and timezone, locale, account tier, memory/custom-instruction state, session freshness, and model label when the interface displays one.
- The exact target-domain policy and optional exact brand strings to match.
- No credentials, account headers, cookies, full chat history, full account export, unrelated prompts, private client records, or third-party personal information.
Capture-context fields are printed as OPERATOR-ASSERTED, UNVERIFIED. We did not observe collection and cannot authenticate the provider, account state, or freshness claim.
Three axes, never one blended score
Reads only machine-readable citation targets. It never infers a citation from answer prose.
Checks the readable response for the exact target-domain string under the frozen case rule.
Checks only customer-supplied exact brand strings with word boundaries; no fuzzy match, stemming, or synonym expansion.
Each axis has its own UNKNOWN state. A readable answer can be known on the text axis while its citation metadata is unknown. Those answers stay separate rather than being forced into a clean-looking verdict.
The citation labels
| Label | Mechanical condition | What it does not mean |
|---|---|---|
| CITED | At least one machine-readable citation target matches the frozen target-host policy. | No provider authentication, endorsement, ranking, or source-truth claim. |
| NOT OBSERVED | The citation-target list is complete and readable and no target-domain match appears. | Not absent from the assistant generally; only not observed in that record. |
| MIXED | Some citation metadata is readable, but another target is missing, partial, shortened, or wrapped beyond string-only interpretation. | Never counted as a clean negative. |
| UNKNOWN | Citation targets are absent or unreadable, including plain-text pastes that discarded link destinations. | Never converted to zero or NOT OBSERVED. |
The matcher must prove it can say yes and no
Every batch runs a positive fixture that must match the frozen target policy and a negative fixture that must not. Either control failing blocks delivery. This prevents a broken matcher, customer typo, or dead code path from manufacturing a page of zeros.
Record selection is fixed before analysis: prompt order, then assistant order, then the asserted run timestamp. If input exceeds the agreed cap, the excess is excluded by that order rather than by its answer. Every exclusion and its reason remains in the gap register, and delivered versus planned counts print beside every observed count.
Frozen URL policy
Normalization is string-only. The report lowercases a parsed hostname, removes one leading www. and a terminal dot, and preserves the raw target beside the normalized target. Query strings and paths do not alter the host match. Before payment you choose exact-host or apex-plus-subdomains policy. We do not fetch a URL, follow a redirect, expand a shortener, or unwrap a provider grounding link. An unresolved wrapper without a target match produces MIXED or UNKNOWN, never NOT OBSERVED.
Input SHA-256 values prove only that the received bytes used for the report did not change after receipt. They do not prove who produced the transcript or whether its capture context is accurate.
Why metadata capability is part of the result
Official documentation describes source-bearing answers in different shapes. OpenAI documents web-search responses, Anthropic documents web search with cited sources, and Google documents grounding with Google Search and citations. Those differences are why the format check records whether a supplied transcript preserves machine-readable citation targets before NOT OBSERVED is even reachable.
- OpenAI web search tool guide
- Anthropic web search tool documentation
- Gemini grounding with Google Search documentation
These links establish only that source and citation formats exist and differ. We make no compatibility claim, are not affiliated with these providers, use no provider logos, and do not treat a consumer transcript as equivalent to an API response.
Privacy and retention
Processing is local and mechanical. Customer transcripts are not sent to an external model provider and are not reused for training, cross-client comparisons, marketing, or a public dataset. Reproduced transcript text is limited to short spans around an exact match. Inputs are deleted no later than seven days after receipt. Our working report and hash copies are deleted no later than 30 days after delivery; the customer copy remains the customer's responsibility.
What the report cannot establish
It says nothing about what other accounts, locations, sessions, or users see. It does not measure ranking, share of voice, assistant quality, index coverage, currentness, truth, sentiment, training inclusion, causation, or future visibility. It is not comparative advertising support, a client-facing proof file, or a recommendation to change content. Manual collection queries the named assistants; we do not access those accounts, query providers ourselves, scrape interfaces, or automate a consumer assistant.
Request the bounded format check
- Send a manifest only: client domain, proposed assistants, five prompt labels, planned record count, formats, and authorization status. Do not send transcripts yet.
- We confirm whether the format preserves the fields needed for each axis and return the frozen matching policy. Unreadable or unsafe scopes are declined before payment.
- After written scope confirmation, supply only the accepted prompt records. The private PDF and CSV are due within three business days.