Lesson 15: Complete an auditable RAG experiment

You can now connect the exercises: assemble evidence, select passages, write a prompt, and review the answer. No program is required for this experiment.

Make a record someone else can follow

It looked good does not preserve useful evidence. Save the query, corpus version, ordered passages, exact prompt, original answer, and pass or failure reasons. Another reader should see what the model actually received.

Label manual selection and handwriting as simulation; supplying selected cards to a model is manual retrieval plus generation. Neither measures automated vector search. Record visible tool names and time without inventing hidden settings.

Run the same date question twice

Ask about visiting on the 18th and comparison with ordinary Sunday. Expected evidence is A/D. First supply only A with the previous prompt: ordinary hours are supported, exceptional arrangements are not.

Keep the question and requirements fixed, adding only D. Expect closure [D] compared with ordinary hours [A]. One changed notice makes the missing-evidence effect observable.

Record unsupported confidence in the first run or ignored evidence in the second. Keep both originals rather than silently fixing failures before saving.

Add three runs to complete the exercise

Add renewal with C, required document with F, and fines with B, for five records total. The last deliberately lacks support to test uncertainty. Review against sources rather than fluency.

Keep every result when repeating. One pass does not establish reliability. A complete reviewable record helps more than repeatedly switching tools and saving only the best answer.

What each record checks

The first date run checks uncertainty; the second checks separate A/D support. Renewal checks count, duration, and conditions; documentation checks a valid student card; fines check avoidance of guesses. Copy the record template from the pack.

Finish with one finding and one unverified claim. Next week puts these judgments into fixed tests for comparison.

Further reading: Anthropic: Define success criteria and build evaluations

← Previous · Learning plan · Fictional library practice sources · Next →