Named by 10 of 26 clients, and the word they use is evaluation
The right file comes back
first 77.8% of the time.
Every retrieval demonstration shows a question going in and an answer coming out, which tells you nothing about how often the right passage was found. So this one shows the passages and the scores instead of an answer, and publishes how often it gets them right.
The corpus is this repository: 255 chunks from 23 files of its own source and comments. Anything it tells you can be checked by opening the file it names.
The last one is deliberately outside the corpus. It should come back with nothing rather than with the least unrelated paragraph.
The score, and what the answer key is
18 questions the corpus can answer, each naming the file that should come back, and 5 it cannot. The key was written from knowing what is in the corpus, before anything was run. Writing the question, running retrieval, and recording what came back produces a key that can never disagree with the system.
The refusal half is the one usually missing. A retriever returns its nearest neighbours whatever you ask it, so it always returns something. Without a floor, a question about Kubernetes gets answered with whatever text was least unlike it, in the same confident tone as a correct answer.
Two things the score found that a demonstration would not
A question came back with five confident, well ranked, completely wrong passages
The file that answered it had never been indexed. lib/rate-limit.ts sits at the top of lib/ rather than in a subdirectory, and the walker only descended into subdirectories. The retriever cannot find what is not in the corpus, and a score that only measures similarity sends you to blame the embedding model or the chunking. Corpus coverage is the first thing to check when retrieval fails and the thing nobody checks, because the system looks like it is working.
One run scored 44% where every other run scored 78%
It happened immediately after a bulk reload of the table, and the tell was in the output rather than the number: several queries came back with three rows when five were asked for. I could not reproduce it afterwards, so it is recorded as observed once and not explained. The eval now counts short result sets and says the index was serving an incomplete view, rather than reporting a low score and letting somebody go and change the chunker.
Chunking decides most of this, and it is not a length
Code is split on the block comments that explain why something is the way it is, and each chunk carries the code underneath. That is the unit a question is actually about: “why does the guard resolve the name first” is answered by one comment and the function below it.
Splitting the same files every 500 characters produces chunks that begin mid-function and mean nothing on their own, and no embedding model recovers from that. The whole corpus cost about a tenth of a cent to embed with text-embedding-3-small, so the expensive part was deciding where to cut.
Every question, and what came back
| rank 3 | Why does the address guard resolve the hostname before deciding? wanted packages/site-console/src/safefetch.ts · got services/pdf-reader/guard.py at 0.557 |
| first | What happens when a link check gets a 400 back? wanted packages/site-console/src/audit.ts · got packages/site-console/src/audit.ts at 0.518 |
| first | Why is a failed write never retried? wanted lib/agent/loop.ts · got lib/agent/loop.ts at 0.466 |
| first | How does the agent stop a model calling a tool that does not exist? wanted lib/agent/loop.ts · got lib/agent/loop.ts at 0.559 |
| first | Why is the timestamp included in the webhook signature? wanted lib/webhook/verify.ts · got lib/webhook/verify.ts at 0.506 |
| first | What happens when the same webhook delivery arrives twice? wanted lib/webhook/verify.ts · got lib/webhook/verify.ts at 0.359 |
| rank 2 | How is the MCP tool list filtered for a session? wanted lib/mcp/grants.ts · got packages/site-console/README.md at 0.497 |
| rank 3 | Where does the permission check happen for an agent write? wanted lib/mcp/store.ts · got lib/mcp/grants.ts at 0.451 |
| first | Why does the Link header parser not split on a comma? wanted lib/integrations/rest.ts · got lib/integrations/rest.ts at 0.643 |
| first | How does the client know to slow down before hitting a rate limit? wanted lib/integrations/rest.ts · got lib/integrations/rest.ts at 0.539 |
| first | How does the PDF reader tell that a page needs OCR? wanted services/pdf-reader/extract.py · got services/pdf-reader/extract.py at 0.489 |
| first | Why does the Python service refuse a file that does not start with %PDF? wanted services/pdf-reader/extract.py · got services/pdf-reader/extract.py at 0.5 |
| rank 2 | Which IPv6 spellings of a loopback address are blocked? wanted services/pdf-reader/guard.py · got packages/site-console/src/safefetch.ts at 0.585 |
| first | Why does the status probe pass only when an attack is refused? wanted lib/status/probe.ts · got lib/status/probe.ts at 0.448 |
| first | What is the difference between the fixture split and the holdout split? wanted packages/site-console/eval/cases.ts · got packages/site-console/eval/cases.ts at 0.389 |
| first | How is the audit score calculated and why is it sometimes withheld? wanted packages/site-console/src/audit.ts · got packages/site-console/src/audit.ts at 0.477 |
| first | How are image dimensions read without downloading the whole file? wanted packages/site-console/src/imagesize.ts · got packages/site-console/src/imagesize.ts at 0.629 |
| first | Why is the rate limiter keyed on something the caller cannot forge? wanted lib/rate-limit.ts · got lib/rate-limit.ts at 0.473 |
| refused | How do I configure a Kubernetes ingress controller? best 0.184, floor 0.35 |
| refused | What is the capital of Portugal? best 0.135, floor 0.35 |
| refused | How do I set up Stripe subscriptions with proration? best 0.312, floor 0.35 |
| refused | What is the best way to train a diffusion model? best 0.245, floor 0.35 |
| refused | How do I write a Terraform module for an RDS instance? best 0.223, floor 0.35 |
4 found the right file but not first. Every one of those is a question where two files genuinely both discuss the thing, which is a ranking problem rather than a retrieval failure, and the honest fix is a reranker rather than more embedding.
What this is not
18 questions is a small set, and I wrote all of them. A real evaluation set comes from what people actually asked, which needs people asking first.
There is no reranker and no hybrid keyword search. Both would help the four cases that found the right file second or third, and neither is here, so neither is claimed.
The floor of 0.35 was chosen by looking at where the out-of-corpus questions landed. That is tuning, not derivation, and it is printed on this page rather than buried so anybody can see how it was picked.
This returns passages and not answers. Generating from them is the easy half, and every claim a model makes on top of these passages would need its own check for whether the passage actually supports it.
Scored on 2026-09-07 with node scripts/rag-eval.mjs. The whole list is 41 requirements from 114 job posts, with the gaps at the same size as the wins.