Lesson 20: Improve RAG from failure cases

For the last article, take one failed record and ask where it went wrong, what to change, and how to verify improvement. These questions put the preceding lessons to work.

Keep the original result for comparison

Fix corpus version, ten queries, candidate method, prompt, and criteria; save original outputs as a baseline. Do not remove hard questions or repair outputs before claiming improvement. Label manual simulations accurately.

Locate failures: add sources for missing facts, repair chunks for lost conditions, inspect retrieval for absent evidence, and inspect evidence use for wrong answers despite support. Change one diagnosed factor at a time.

Fix current rules without breaking history

Suppose question one uses old E. Preserve B/E versions and require applicability checks. Predict current answers citing B and historical answers retaining E, then retest one and eight.

Deleting archives can appear to fix current questions while breaking historical comparison. Retest former successes alongside failures; this is regression checking.

Missing D needs A/D coverage; invented fines need uncertainty review. Avoid changing model, prompt, and retrieval together so you can identify what helped.

Report unresolved failures alongside the score

Report before/after passes, denominators, failures, and regressions, then check held-out paraphrases. A higher aggregate with continued fee guesses leaves an important defect. Limit conclusions to what the small test measured.

Revert complexity without benefits. Progress means a diagnosed failure improves for an explainable reason, not simply more components.

Keep a result you can explain

Assemble sources, tests, prompt, original records, grading, and one improvement. Explain the observation, causal evidence, change, and retest, distinguishing hypotheses from findings. Can another person follow it to the evidence?

A sufficient final exercise

For missing D, save candidates, add D, review the comparison, and retest ten queries. Manual insertion demonstrates the workflow, not improved automated search. Show the complete evidence-based judgment rather than only a polished score sheet.

If you want to build next, bring these tests to a no-code knowledge base or learn Python for a minimal workflow. You now know which questions to use when evaluating a tool.

Further reading: Microsoft: RAG LLM evaluation phase

← Previous · Learning plan · Fictional library practice sources