Auditing our own numbers: the pie-chart label
The corpus carried its own unsourced placeholder loss factors, so a batch went out to source our own numbers. Outcome: nothing accepted. One figure returned, it passed the gate, and it was wrong by a factor of twenty. The most instructive run so far.
What came back
All eight fetches succeeded — the certificate fix from the honey batch held. One model returned nothing from eight calls; the other returned a single figure:
field: egg_kitchen_breakage
value_mode: 1.3
unit: percent of eggs lost at consumer level
quote: "Eggs, 5.1, 1.3%"
Why the gate passed it, and why it is wrong
1.3 appears verbatim in the document. The unit string mentions eggs and
percent, the field mentions eggs, and the document is the right one. Every
check said yes.
It is a pie-chart label — from a USDA ERS figure titled “Estimated total number of calories of food loss in the United States per day by food group.” The 5.1 is billion calories; the 1.3% is eggs’ share of all food-group calorie loss (grain products are 21.7% on the same chart). It is not a rate at which eggs are lost. It is eggs’ slice of a total that has nothing to do with egg breakage.
The real figure was on the previous page, in a table whose header states the columns: eggs lose 7% at retail and 26% at consumer level, 33% total. The extraction was out by 20×, on the right page range, in the right document, with a real quote.
The failure class
Not fabrication — the quote is real. Not misattribution — the subject genuinely is eggs. This is a number stripped of the header that gave it meaning, and none of the four automated checks can see it: the quote is verbatim, the value is in the quote, the units nominally match, the fragment ends cleanly. The batch spec predicted this shape (“these are table cells, not sentences; expect this item to need a human”) — and predicting it was not enough to stop it passing. The standing rule it produced: a figure whose basis cannot be read off its own quote is not usable, however well it verifies.
The bug that made the run look like source scarcity
Three government URLs in the batch share their first 47 characters, and document filenames were truncated at 48 — so all three wrote to the same file, each fetch silently deleting the last. A 693 KB regulation text — the whole basis of one item — was downloaded, logged as fetched, and destroyed before extraction. “Zero figures from eight documents” was partly this bug, not a finding about the sources. Fixed by appending a hash of the full URL, with a regression test that three same-prefix URLs get three names.
Also confirmed: one government site serves a 1,502-character block page to the pipeline where a browser gets 1.4 MB. The scouting had verified that URL from the wrong host — a URL verified from one machine is not verified for another.
The judgement that stayed human
The corpus placeholder said 1% kitchen breakage; ERS says 26% at consumer level. That is not a 26× correction to make quietly, because the two measure different things: the ERS figure covers spoilage, plate waste, and everything after purchase, while the corpus stage is specifically eggs cracked in the carton and dropped in cooking — and other stages in the model already cover the rest. Adopting 26% wholesale would double-count. The honest options — split the figure across the stages it spans, or record it as a bound and say so — belong to a person, and the decision is recorded, not hidden.
The ledger
Two runs of honey and one of this batch produced zero accepted figures. What they produced instead: a gate hole closed, a missing CA bundle found, an API bug fixed, a document-destroying filename collision fixed, and a named failure class the gate cannot yet catch. The pipeline got more trustworthy while shipping nothing — which is the right order.
Adapted from the project’s review record
batch-09-own-loss-factors-REVIEW.md
(CC BY 4.0).