I fucked up my own benchmarks
I ran a code review benchmark with thinking turned off, and two of the models thought anyway. The CLI setting I trusted did not reliably disable it, the responses came back full of thinking tokens, and I filed those runs under "no reasoning" without looking. Careful measurement, clean tables, completely wrong (it looked great, too). I benchmarked the thing I assumed I had configured, not the thing I had actually run.
That was not a one-off. Most of the numbers on bench.kleb.sh (opens in new tab) were wrong at least once before they were right, and not one of those mistakes belongs to a model, a vendor, or a library. They are mine. The whole benchmark is built around the assumption that I will screw it up again.
The list of shame, so far
All of it is in the repository, with the fix next to it:
- Thinking that was not off. Calls labelled
nonereturned thinking tokens on Haiku 5.5 and Opus 5. Every configuration I measured that way is excluded and rerun with an explicit thinking-off request, verified at the proxy this time, because my word was clearly not enough. - A median that was not a median. Two scoring versions took the upper of the two middle values for even-sized cohorts. I wrote a median function and got it wrong. Version three averages them; the old files keep their convention, labelled, so nobody can pretend it did not happen. Least of all me.
- The wrong model. One campaign requested a model I did not mean to request, and I did not notice until the results were in. It was replaced, not relabelled - relabelling would have been lying with extra steps.
- A cap that cut answers. I set a 65,536-token output budget, and it truncated four Haiku cases. I scored half-finished answers as answers. The rerun at 128,000 completed 64 of 64; the truncated cohort stays in its own bucket.
- A score that measured the wrong thing. Early review scoring matched finding titles against known bugs. A matching title says nothing about whether the finding is right, which I should have seen before building a scorer around it. It was replaced by model-judged verdicts, frozen once recorded.
- An OCR run that never stood a chance. A GGUF build of an OCR model on an integrated GPU hit the 512-token generation limit I gave it on all six pages and scored a character error rate of 515 %. The same model scores 99.5 % character accuracy on a real GPU. I published the 515 anyway (it is still on the site), with the truncation disclosed, because hiding it would be worse than having it.
And one I cannot explain yet: the same decision-model weights scored 77.6 on average on CPU and 68.0 on an integrated GPU. Most of that is one dataset, where accuracy fell from 85.3 to 50.3. I do not know why. That result stays published as measured until I do, instead of disappearing because it annoys me.
What a number has to carry
A benchmark number is a claim about a protocol, so every result carries that protocol with it: model name, revision, and precision; device, backend, and hardware; dataset name and revision digest; warmups, repeats, batch size, token limits, and the exact timing boundary. Evidence adds hashes of the raw output, the dataset, and the model files. Load time including download is recorded and kept out of throughput.
Two rules do most of the work. Unknown values are null, never zero and
never an estimate - a zero I typed in is a measurement I invented. And a
change to the protocol means a new cohort (a group of results measured under one identical protocol), so results measured
differently never share a table.
Guards that refuse
My mistakes were never exotic. They were boring, sloppy, and repeatable, so the guards are boring too. A run is refused when:
- a GPU run would fall back to CPU, or a llama.cpp run does not offload every layer to the device it claims;
- a run ID already exists - results are never overwritten;
- evidence disagrees with its result on run ID, dataset digest, protocol, or metrics;
- a download does not match its pinned checksum;
- memory use crosses the configured ceiling, at which point the run stops rather than limping on.
Evidence is written before the result, so a visible number never points at missing evidence. The site build fails on duplicate IDs, mixed accuracy measures within one dataset, and evidence paths that escape their folder. The repository's own instruction is blunt: never bypass a failed guard or resume interrupted state. I wrote that rule for the person most likely to break it.
Why it is a Git repository
Repository JSON is the source of truth. There is no application server, no
database, no runtime API, and no telemetry; the dashboard is prerendered
static files. A correction in a database is an overwrite. A correction in
Git is a diff with a message, and the wrong version stays one git log
away, where it can keep embarrassing me.
And I have not even kept my own promise. When I reorganized the suites, I cut the embedding results down to the newest run per configuration - 29 kept - in a commit that deleted 1,059,972 lines. The superseded runs now exist only in Git history (Git remembers so I do not have to). "Never overwrite a historical run", says the README I wrote. I would rather confess that here than have someone else find it.
What it still is
The published embedding numbers come from a synthetic 24-sample retrieval set, not the full public suites the runners already support. Twenty-four samples. They are good for comparing engines and hardware against each other on my machines, and not much else. If I ever pick a production model off that table, I deserve what happens next (write that down). The source is public (opens in new tab); every claim above can be checked against it.
The rule I benchmark by: a number that cannot point at its protocol, its hashes, and the guard that would have refused it is an anecdote.