Does your model quietly fail on your work?

What you get

Six files on a private page the thank-you page opens. Nothing is emailed and nothing of yours is sent to us, then or later.

silent_check.py

The runner. One file, Python 3.10 or newer, nothing to install. It reads your task file, calls each model you name with the key in your own shell, checks every reply against the expected output you wrote, and writes one report.

README.md

How to run it, what each number means, and what it does not measure.

tasks.example.jsonl

Example task rows: an input, the check a reply must pass, and an optional size. Replace them with 20 to 50 of your own.

prices.example.json

Example price table. Edit it to your contract prices so cost per pass is your cost.

sample_report.md

A report produced from recorded replies, so you can see the shape before you run anything.

fixtures/recorded_replies.json

The recorded replies used by the offline demo. Try the demo before using any paid model credit.

The report prints, per model: pass rate, silent-refusal rate (replies that came back as a success but were a refusal, a content filter, a deflection or a cut-off), cost per pass from your price table, and the longest task that passed.

Public sample

This is the report the runner wrote from a recorded set of replies. It shows the shape and every column. It is not a ranking of models and the numbers say nothing about your tasks. The source is the included recorded-replies fixture; the supplied report was generated on 2026-09-10, not from a new live model run.

Recorded run on 2026-09-10 over 15 example tasks shipped with the kit, replies from a recorded fixture, not live model calls. Your report uses your tasks and your keys.

Whole sample: sample.csv · sample.json

How it works

1. Write your tasks in one file, one row per task: the input you send, and the check a good reply must pass (a phrase it must contain, a phrase it must not contain, or a regular expression).
2. Put your own model keys in your shell. The README names the variable for each provider. The runner reads them from your shell and never writes them anywhere.
3. Run one command. The runner calls each model you name, one task at a time, and records the reply, the stop reason, the token counts and the time taken.
4. Read the report. A reply counts as a silent refusal when the provider marks it as a refusal or a content filter, when it was cut off before the answer, or when it is a refusal in plain words that fails your check.

python3 silent_check.py --tasks tasks.jsonl --prices prices.json --out report.md

What this is not

• Not a benchmark. It measures the tasks you give it, on the day you run it, with the checks you wrote.
• Not a service. We never run your tasks, never hold your keys and never see your results. If you want someone to run it for you, this is not that.
• Not a guarantee. A model that passes today can be changed by its vendor tomorrow. Run the kit again after every release.
• Not free to run. Each run spends your own model credit at your own prices. The sample task set is small on purpose.