Israel in Hebrew: ask in the document's language
Two batches, one controlled experiment. Result: matching the question’s language to the document’s is what matters, and the effect survived a downgrade in retrieval machinery.
Batch one: English questions, zero figures
Ten Hebrew documents, three items, two models, six calls each: nothing returned from any of them. The documents were fine — every URL fetched, including a 40-page State Comptroller PDF that extracted to 103,610 characters, with the Hebrew intact (18,137 Hebrew characters in one page, 10,285 in another). Chunking produced the expected count. Not a fetch failure, not a chunking failure.
A human reading the same artifacts found real figures in two documents within minutes — the second time in this project the returned raw documents were worth more than the extraction, and the reason every batch returns them.
The suspected cause: the questions were English, the chunks Hebrew, and the embedder English-centric. An English query embedding scored against Hebrew chunk embeddings is close to noise, so the chunks handed to the model were probably not the ones holding the numbers.
Batch two: the same documents, questions in Hebrew
| English batch | Hebrew batch | |
|---|---|---|
| Documents | ten | six of the same ten |
| Retrieval | real embeddings | keyword overlap (embedder down) |
| Model A | 0/6 | 0/6 |
| Model B | 0/6 | 2/6 |
| Quote gate | — | 2/2 passed |
Both figures came back with quotes matching the source character-for-character, in Hebrew. And by accident the run was stricter than designed: the embedding model was unavailable, so retrieval silently fell back to crude keyword overlap. English questions with real embeddings scored zero; Hebrew questions with worse retrieval scored two. If embeddings were the whole story, the fallback should have made things worse. Language match beat semantic search across a language boundary — which also means a multilingual embedder is no longer the diagnosis, and should not be adopted on the strength of this run.
The extraction independently reproduced what the human read had already found — a deliberate choice of target, because reproducing a known answer is what turns an anecdote into a finding. A negative control (four documents that provably contain no percentage sign in either language) returned nothing from both models: no hallucinated rate.
What the Hebrew found
A growers’-organisation sector summary for 2021 carried two figures worth the whole exercise:
- 604 broiler growers — against “about 600 large chicken farms” reported by a different industry body in a different publication. Two sources, one number: corroboration, not repetition.
- 244 million chicks placed by the breeding branch in 2021 — about 6% under the 260 million a year reported for 2025 by the other body, which is roughly what four years of growth looks like. This is the strongest corroboration Israel’s head count has.
Loaded carefully: chicks placed is not birds slaughtered — mortality sits between them — so the figure went in as its own measure, never as a slaughter count, with tests pinning both the distinction and the corroboration (the chick and head figures must stay within 25% of each other; the grower count within 550–650). If a future edit breaks the agreement that justified the promotion, the test suite says so rather than the claim quietly becoming false.
Also confirmed: none of the four halachic certification pages fetched contains a kosher-inspection rejection rate — no criticism of those pages, which are written for consumers and explain the rules thoroughly. Operational statistics simply live in a different literature, so that figure remains described and unquantified in the model, and the next attempt should target responsa collections and slaughterhouse-facing training material — the readership those numbers are actually written for.
The bug worth exporting
The first Hebrew run completed its extraction and then crashed printing the results: a Windows console encoding (cp1252) meeting Hebrew item names, taking the findings file with it. The pipeline had a language bug in its reporting layer, not just its reading layer — silent until the day something non-Latin reached it.
Adapted from the project’s review records
batch-05-israel-hebrew-REVIEW.md
and
batch-05b-israel-hebrew-questions-REVIEW.md
(CC BY 4.0).