Evaluation & Debugging
Testing prompts, metrics, and iterative improvement.
Module Overview
Systematic methods for evaluating prompts and pipelines: metric definitions, test harnesses, A/B testing, human annotation, and debug workflows to iteratively improve outputs. Emphasizes scientific approach to prompt iteration.
Learning Objectives
- Define and implement evaluation metrics for different tasks (accuracy, F1, BLEU-like, human-rated usefulness).
- Design A/B tests and interpret results with statistical thinking appropriate for small-sample educational pilots.
- Establish logging and reproducibility so bugs are traceable and fixes validated.
Lesson-by-Lesson Breakdown
Choosing metrics aligned with task goals.
Building test sets and gold-standard labels.
Running controlled experiments and interpreting results.
Human evaluation protocols and inter-rater reliability basics.
Debugging prompt pipelines: isolation, reproduction, and fixes.
Documentation & changelogs for iterative improvement.
Hands-on Activities & Deliverables
Activities
Run an end-to-end experiment comparing two prompt designs on a 50–100 item test set; present results with simple statistical summaries and recommendations.
📦 Deliverable
Experiment report, dataset, and recommended next-step plan.
Required Tools & Readings
Evaluation script templates, examples of human-eval forms, reading on small-sample experiment interpretation.
Assessment & Rubric
- Experimental design quality40%
- Clarity and rigor of analysis40%
- Recommended improvements20%
Prerequisites
Modules 1–5.
Parent-Friendly Value
Demonstrates that student work is measured and improved using data — tangible evidence of learning and reliability.