Lesson 17: How to evaluate retrieval
Set generated answers aside and inspect search results. Question seven needs A/D but receives A/H. You can already identify the missing notice. Recall and precision make this judgment explicit.
How much required evidence was found?
One retrieved required passage out of two gives 50% recall. A/H/D contains both, giving 100%.
Define the unit: passage IDs here. Real collections can contain equivalent support, which criteria should recognize to avoid marking valid retrieval as failure.
How many returned passages help this query?
In A/H/D, A and D help; H concerns reference books and is noise for this query. Two useful passages out of three gives about 67% precision.
| Candidates | Both A/D? | Recall | Precision |
|---|---|---|---|
| A, H | No | 1/2 | 1/2 |
| A, H, D | Yes | 2/2 | 2/3 |
| A, D | Yes | 2/2 | 2/2 |
More results can complete evidence while carrying noise. A alone is clean and precise but incomplete. Check both metrics and whether all necessary evidence is present.
Order changes the evidence actually used
A/D/H and H/A/D have the same set, but only the first covers both in its first two results. Preserve order and k. Reranking can help useful evidence appear earlier.
A bare retrieval percentage omits query count, k, corpus version, and relevance standard. Mark ordinary recall not applicable for unsupported fees rather than assign perfect scores.
Calculate historical comparison with required B/E and E/H/B candidates at one, two, and three results. Record metrics and complete coverage. These manual lists are not tool-performance measurements.
Check your calculation
All three give 2/2 recall and 2/3 precision. One or two results both retrieve only E, giving 1/2 recall; two give 1/2 precision. E is relevant to historical comparison; a current-limit query changes the standard.
Further reading: Microsoft: RAG information retrieval
← Previous · Learning plan · Fictional library practice sources · Next →