Prompt and model config versioning, 5 of 26
The obvious fix
made it worse.
A prompt is configuration. It changes behaviour, and it usually lives as a string in a route, edited on a Friday under a commit message saying “tweaked the prompt”.
Three versions here, scored against the same 8 cases, 3 runs each, on gpt-4.1-mini. The scores are low because the cases are nasty and a case only passes when it passes every time.
The obvious instruction. Ask for the fields and ask for a quote.
the baseline everything else is measured against
Say that null is a correct answer, and that the quote has to contain the value. The first version had no permission to say nothing, and a model with no permission to say nothing fills the field.
refused: it is wrong 5 times against 4, and being wrong more often is not a trade worth making for any score
Keep the permission to say nothing and take away the fear. Version two stopped inventing citations and started answering null to company names printed in the first three words of the text, so the rule needs to say what "not stated" actually means.
promoted over v1: the same 25%, and wrong 3 times against 4
What changed between v1 and v2
Extract the fields from the text.- Return a quote for each field.+ Rules:+ - Copy the quote character for character out of the text. Do not paraphrase it, do not tidy it, and do not join two separate sentences.+ - If the text does not state a field, return null for it. Returning null is correct and expected. Guessing is not.+ - The quote must contain the value. If it does not, you have the wrong quote.
What changed between v2 and v3
Extract the fields from the text.Rules:- Copy the quote character for character out of the text. Do not paraphrase it, do not tidy it, and do not join two separate sentences.- If the text does not state a field, return null for it. Returning null is correct and expected. Guessing is not.+ - Stated means written down, not proven. A company name printed in the text is stated, even in passing. Do not return null for something you can point at.- The quote must contain the value. If it does not, you have the wrong quote.+ - A number near a field is not that field. Patients are not staff, and the value of one job is not a year of revenue.
Case by case, 3 runs each
everything stated plainly
v3: locations: said null, key says ["Leeds"]
revenue is simply absent
v3: 1 citation not in the text | company: said null, key says ["Kestrel Logistics"]; locations: said 2, key says ["Bristol and Cardiff","Bristol, Cardiff"]
nothing concrete at all
v3: matches the key
a number that is about something else
v3: company: said null, key says ["Harbourside Dental"]
a tempting near miss
v3: 1 citation not in the text | company: said null, key says ["Fenwick Roofing"]
two dates, one of them wrong to use
v3: 1 citation not in the text | company: said null, key says ["Bramble and Co"]; locations: said null, key says ["Otley"]
no company name anywhere
v3: matches the key
everything hedged
v3: staff: said "a handful of people", key says [null]
Four decisions
One run is not a measurement
The first version of this eval ran each case once at temperature zero and treated the answer as the score. Two consecutive runs of the same prompt then came out 1 and 2 out of 8, a bigger gap than the one between two different prompts. Every case now runs three times and a case counts as passed only when it passed all three.
Two failures, counted apart
Answering nothing and answering something untrue are different failures. Add them together and the grader will happily promote a prompt that answers more questions and lies more often. So a version that invents more citations cannot be promoted. Not for one point, not for ten.
The rule decides, not the wording
Version two reads better than version one and it is refused, because it invents more citations. That refusal is the whole reason the registry exists: without it, a prompt ships because somebody preferred the way it read.
The answer key was written before anything ran
Each case names what the text does and does not state, decided by reading the text. A key built from what the model returned can never disagree with the model, which is the failure mode of most prompt evaluation people show you.
What the eval found in the checker
All three versions failed the same way, which usually means the grader is wrong rather than the thing being graded. It was. A citation ending a sentence, like “founded in 2011.”, produced the token “2011.” with the full stop still attached, so the checker called a correct answer unsupported.
That bug was live on the extraction page. It is fixed, with three tests holding it there, and it was found by building this rather than by anybody noticing.
What this does not do
Eight cases catches a prompt that invents citations. It comes nowhere near telling two good prompts apart. The spread on each version says how far one run can sit from the average, and at this size it is wide.
None of these three is good enough to ship on this task. That is the eval doing its job: the version that would have shipped on how it read is the one it refused.
The extraction endpoint is the thing being tuned, and the evals page is the same idea applied to a scoring engine. The whole list is 41 requirements from 114 job posts.