Structured output, 8 of 26, and source traceability, 5 of 30
Make it quote the text,
then check the quote.
One client asks how you would stop a model presenting unsupported assumptions as verified findings. Another asks how you retain sources so every conclusion can be checked independently. The answer that actually works is not a better prompt.
Require a quote next to every value, then look the quote up in the source. Paste anything below, or use the three samples. The second one is written to tempt a guess.
Four decisions
The schema is checked here, not just requested
The API is asked for a strict JSON schema and usually obliges. On a path that matters, "usually" is doing far too much work, so the response is parsed and validated again before anything reads it. A field that is missing, a field nobody asked for, and a value that is a bare string instead of an object are all refused with the reason named.
Every value carries a quote, and the quote is looked up
This is the whole idea and it is embarrassingly simple. A model inventing a value almost always invents a plausible citation to go with it, and a citation that does not appear in the source is caught by a string search. No second model, no judgement call, no cost.
A real quote carrying the wrong number is caught too
A model can cite a genuine sentence and then report a number that is not in it, which passes any check that only asks whether the quote exists. So when the value is numeric it has to appear inside the quote as well. Both sides of that comparison are normalised the same way, which is a sentence that exists because the first version did not and reported a correct citation as unsupported.
Null is a first class answer
The schema asks for a field most texts do not state, and the instruction says returning null is correct and expected. A model with no permission to say nothing will fill the field, and the field it fills is the one somebody later acts on. Press the third sample: it says nothing concrete, and everything comes back empty.
Where this stops working, with a live example
Press the second sample. It mentions “an average job value in the low hundreds”, and the model fills the revenue field with exactly that phrase. The quote is real, the check passes, and the answer is wrong: an average job value is not annual revenue.
That is the honest boundary. This catches a value with no support in the text. It does not catch a value that misreads support which genuinely exists, because the quote really is there and the conclusion drawn from it is still wrong. Fixing that needs a second model judging whether the span actually answers the question, which is a harder problem and a different page.
The third sample shows the opposite failure. It returns null for the company name even though “Acme Ltd” is right there, because the instruction pushes hard towards caution. A check that only rewards saying nothing gets a system that says nothing.
Tested without the model
30 tests on the checking, with no API call in any of them: an invented quote, a real quote carrying a different number, a value with no quote, a null that is the right answer, and a response carrying fields nobody asked for.
npx tsx lib/extract/verify.test.ts
The endpoint lists what it checks. The whole list is 41 requirements from 114 job posts, with the gaps at the same size as the wins.