Lesson 16: Build a test set for RAG

After getting quantity right, a system may still fail closure, history, and deposit questions. Use questions that expose these differences rather than a few easy successes.

Set criteria before reading outputs

A test set contains reusable queries and judging criteria. A table with ID, question, expected passages, facts, and conditions is enough. Define it before running, rather than relax standards for an appealing output.

Renewal needs once, seven days, unreserved, and before due date. Seven days alone misses count; anytime changes eligibility. Wording can vary while required facts stay fixed.

Ten questions should test different things

The pack provides ten questions: four direct rules, two conversational phrasings for Saturday and room duration, two evidence-combination questions, and two unsupported fees. Hide the answer key and predict sources first.

Question seven compares the 18th with ordinary Sunday and needs A/D. Closure alone is correct but incomplete. Question eight needs B/E for two years; archive status does not make E irrelevant.

Fee questions pass by acknowledging insufficient evidence without invented amounts or zero-cost inferences. With no required-passage denominator, evaluate uncertainty separately from ordinary recall.

Keep two fresh phrasings aside

Repeated tuning on ten queries may improve only familiar wording. Reserve two equivalent paraphrases for final checks, such as how many ordinary books may be taken at once, without adding live inventory conditions.

This separates tuning from an additional check. Keep the exercise simple and avoid treating ten passes as universal reliability. Update test versions when source rules change.

For renewal, date comparison, and fines, specify required and prohibited claims. Ask another reader to apply your criteria; unclear pass decisions reveal criteria that need refinement.

Minimum checks for three questions

Retain renewal count, duration, and conditions. Cite both normal hours and closure for the comparison. Report unknown fines rather than zero. Judge these facts rather than exact reference wording.

Further reading: Anthropic: Define success criteria and build evaluations

← Previous · Learning plan · Fictional library practice sources · Next →