Silk: false guilt and false confidence
Run twice; verification failed both times and nothing was accepted. Recorded anyway, because the run was worth more as a bug report on the pipeline than it would have been as three figures about neckties. Seven items asked, five figures returned, two items returned nothing at all.
HTML entities were never decoded — false guilt
The fetcher stripped tags but never unescaped entities, so survived
into the stored documents — and the stored documents are what quotes are
verified against. A model that read the prose correctly and wrote it back as
ordinary words produced a quote that could never match, rejected as “quote
does not appear in the document” — which reads exactly like a model inventing
a citation.
It cost the batch’s single best row, a 2/2-consensus figure for cocoons per
thread. The correlation was exact: the document carrying 47 entities
produced the failing quote; the document carrying zero verified fine. After
the fix, the row passes. This is the mirror image of a filename-collision bug
found the same week: that one manufactured false confidence; this one
manufactured false guilt.
The scout confirmed a figure the pipeline cannot see
cocoons_per_tie returned nothing from both models — and they were right to
decline. The batch spec cited a retailer’s page for “about 120 to 130
cocoons” per tie, confirmed reachable during scouting. But the pipeline’s
fetch of that same URL contains no 120 and no 130; it opens with “NEW: Summer
Collection 2026 — SHOP HERE.” The scout’s fetcher renders JavaScript; the
pipeline’s does not. “Confirmed, with the figure present” had been confirmed
with the wrong tool — the check must be “present in what the pipeline
retrieves,” not “present in what a browser shows me.”
The grounding gap: one bound can be junk
cocoons_per_shirt passed every check and is wrong. Its quote:
It takes 1700 to 2000 cocoons to make one silk dress (or about 1,000 cocoons for a silk shirt).
The row stored lo 1700 / mode 1800 / hi 2000 — for a shirt. Those are the dress numbers from the same sentence; the shirt figure is about 1,000. It grounds because the rule asks only that any of the three values appear in the quote (deliberately, so an interpolated mode is allowed). The first run showed the gap more starkly: a filament-length row stored an upper bound appearing in no silk document at all, riding beside two legitimate numbers — and flipped the unit from metres to feet on the way.
A refinement was proposed rather than applied, because it changes what the gate accepts and that is not a machine’s call: bounds must each appear in the quote; only the mode may be interpolated, and only inside them. Worth checking against the whole accepted corpus before adopting.
Two-model disagreement did its job here: one model answered 1,000 (correct) against the other’s 1,800, and the row was flagged. It was the flag, not the gate, that caught it.
What the gate caught cleanly
Run two produced a row whose value field read estimate — the model wrote its
own confidence grade where a number belongs. Rejected with an exact message.
What was not the problem
Chunking. The going theory after the maple run was that retrieval was being starved; it was not. Every item’s own document was in front of the model, and the figures that failed did so with the right text on screen.
Adapted from the project’s review record
batch-08-silk-REVIEW.md
(CC BY 4.0).