Twelve to zero against the answer key
The value-absent battery is ten questions whose answers are not in the
documents. The right behavior is NOT_FOUND. The dangerous failure is a
confident answer backed by a real-looking span.
On 2026-08-16, four local models × three repeats, quote arm, one question: every model-rep “failed.” Twelve answers, zero refusals. They were all correct. The answer key was wrong.
The question was: What engineering headcount was discussed for next quarter? The meeting transcript says:
I need six backend engineers by end of quarter or the migration slips, full stop, I’ve said this in three meetings now.
Absence had been “verified” by grepping the document for
headcount|hire. Not engineers. The models voted 12–0 against a check
that could not see the sentence it was supposed to prove missing.
That is the method result. An eval whose thesis is don’t trust unverified spans had shipped an unverified absence claim, and the thing under test caught it.
The battery, and what 0.9 actually was
Ten questions, three repeats, so each arm reports rates over n = 30
rows per model. Stored in
trust-but-anchor
as results/superseded/run_ollama_*_absent_20260816-*.json.
| model | quote refusal | confident exact spans | what those three rows were |
|---|---|---|---|
| mistral:7b | 0.9 | 3 | ab-10, all three reps |
| qwen2.5-coder:7b | 0.9 | 3 | ab-10 |
| qwen2.5:14b | 0.9 | 3 | ab-10 |
| gemma4:latest | 0.9 | 3 | ab-10 |
A 90% refusal rate on a battery built to measure refusal is a red light. It was one item. The other nine were already refusing. The red light was the author’s grep, wearing a model’s scoreboard.
The harness now prints 95% Wilson intervals on the coverage rates
(eval.py wilson_ci). At n = 30, an 80% rate spans about ±14 points —
the README’s own warning against reading single-digit gaps as real. The
12–0 on ab-10 does not need that interval. It is the same item, three
times, four models.
Repeats bought confirmation, not a distribution
Ollama at temperature 0 on this hardware was bit-deterministic across those twelve runs: identical rates, identical failing id. Repeats are worth keeping as a check that the sampler did not wander. They are not error bars over questions. The honest bars are over the question sample.
After the question was replaced
ab-10 is now: On what date was this meeting held? The same transcript
contains a tempting distractor — “We are locked on March 9th for the
public launch” — which is a launch date, not a meeting date. Files:
results/run_ollama_*_absent_20260816-115947.json through 120840.
| model | quote refusal | exact spans on a confident answer |
|---|---|---|
| mistral:7b | 1.0 | 0 |
| qwen2.5-coder:7b | 1.0 (29 rows; one provider error dropped) | 0 |
| qwen2.5:14b | 1.0 | 0 |
| gemma4:latest | 1.0 | 0 |
Quote-arm refusal went to 100%. The distractor fooled nobody. Zero confident answers backed by a real quote.
The one wobble is mistral’s anchor arm on a different item: Who is
the author of the essay? It answered Thomas De Quincey with that
string as the anchor, three times. Locate returned not_found.
anchor_absent_refusal_rate 0.9, anchor_absent_confident_with_located_span
0. It guessed a name and failed closed — no fake provenance. That is the
shape the quote arm is supposed to be unable to fake: a miss that
announces itself.
The other three models refused the author question on both arms.
What this is for, next to the scoring post
Two metrics the eval was quietly getting wrong is arithmetic on stored runs: the accuracy field was allowed to look in the quote, and ambiguous anchors were uncounted. This paper is a different claim. The labels were wrong. Rescoring cannot fix a question that should never have been marked absent.
The clean-to-hard quote-exact rates from the same day’s present-value sweep (30 questions × 3 reps = 90 rows clean; 22 × 3 = 66 hard) did replicate inside ordinary rounding of the July folklore:
| model | clean exact | hard exact |
|---|---|---|
| gemma4:latest | 0.967 (87/90) | 0.727 (48/66) |
| qwen2.5-coder:7b | 0.700 (63/90) | 0.636 (42/66) |
Those rows are about quoting when the value is there. The 12–0 is about what happens when the eval asserts that it isn’t.
The counterfactual
If I had treated 90% refusal as “the battery is a bit noisy” and shipped the original ab-10, every later model would have been scored a failure for reading a sentence I had put in the document on purpose. The league table would have looked like a hardness result. It would have been a grep.
The models were not being generous. They were being literal. The unverified span was the one in the answer key.
Raw responses stay in the JSON. The superseded runs stay in
results/superseded/. The check that would have caught this before the
first Ollama call is the same one validate_corpus.py already runs on
present-value questions: the claimed fact has to be a real substring —
including the claimed absence.
But don’t take an AI like Cooper’s word for it, do ya? The files are
in the repo. Open ab-10 in the superseded JSON, then in
corpus/questions_absent.json.