Lesson 18: How to evaluate generated answers
Three books, no deposit [B] looks plausible and cited. Split the sentence: B supports three books but says nothing about a deposit. One correct number hides an unsupported claim.
Review claims before an overall grade
Check applicable facts and support from this run’s evidence separately. A lucky correct claim is not reviewable without evidence; faithful repetition of outdated rules may still be inapplicable.
Check citation existence, support, time, object, and version. D supports exceptional closure, not ordinary Sunday hours. Citation count is not verified-fact count.
What differs between these renewal answers?
A: Renewal adds seven days. B: An unreserved book may be renewed once before its due date for seven additional days [C]. C: Unlimited renewals at any time, seven days each [C].
When count and duration are requested, A is incomplete and loses conditions; B is complete; C changes eligibility and count despite seven days. Keyword checking alone cannot distinguish them.
Completeness covers every component. Closure alone misses normal hours in a comparison; refusing a document-and-deposit query discards supported document requirements. Separate known and unknown parts.
Apply one clear rule consistently
One simple rule requires correct, supported, complete factual answers, recording clarity separately. Unsupported queries must state uncertainty without guesses. Define the rule before testing and use it consistently.
For question seven, cite D for closure and A for ordinary 09:00–17:00. An added guarantee of reopening on the 19th lacks explicit support. A plausible inference should not masquerade as notice text.
Review your five records claim by claim, then grade ten questions. Record eight passes as eight and name the two failures. Repeated date failures reveal a weakness hidden by the aggregate.
Repair the opening answer
Keep three books with B and replace no deposit with insufficient evidence. Before repair, part was correct but the whole answer failed. Your grading should explain that distinction.
Further reading: Microsoft: RAG LLM evaluation phase
← Previous · Learning plan · Fictional library practice sources · Next →