A pragmatic guide to RAG evaluation

RAG projects fail in predictable ways — almost always because there is no evaluation. Here is the shortest path from no evaluation to trustworthy evaluation, in five steps.

1. Freeze a query set

Sample 200–500 real user queries from logs. Not synthetic. Not “representative examples someone typed up.” Real ones. Stratify by intent buckets if you have them.

2. Build a judgment list

For each query, have a domain expert mark:

• The ideal answer in one sentence.
• The must-cite documents (0–5 per query).
• A binary “was the question actually answerable” flag.

Don’t skip that last one. About 15–25% of user queries are unanswerable and your system should say so. If your metrics don’t reward abstention, your system won’t learn to abstain.

3. Pick offline metrics

• Retrieval: Recall@k and nDCG@10 against must-cite docs.
• Answer quality: faithfulness and answer-relevance graded by an LLM judge (with a spot-check human sample).
• Refusal correctness: did the model correctly abstain on unanswerable queries?

4. Wire in an LLM grader — carefully

Use a different model to grade than the one that generates. Provide the judgment answer, the system answer, and ask for a scored rubric with explanations. Then sample 10% of grades weekly for human audit. Grader drift is real.

5. Run it on every PR

Cheap subset (30 queries) as a pre-commit gate. Full run (all 500) nightly. Track deltas. Ship the dashboard where product can see it.

---

Do this in week one and every subsequent decision — chunking, embedding model, reranker, prompt — becomes a measurable decision instead of a taste test.