Standards used in this guide
The recipes use published measures and methods wherever one exists. Where the program's workstreams have an evaluation practice, the guide uses it.
Measures
| Measure or method | Use | In this guide |
|---|---|---|
| Recall@k, MRR, nDCG (Manning, Raghavan, and Schütze, Introduction to Information Retrieval, chapter 8) | Retrieval quality with known relevant items, and with graded relevance | Chapter 3 |
| Precision, recall, F1 | Extraction and classification quality; exact and lenient span matching as in the CoNLL and SemEval traditions | Chapter 5 |
| Cohen's kappa (Cohen, 1960) | Agreement between labellers, and between a grader and humans, corrected for chance | Chapters 2 and 6 |
| Bootstrap confidence intervals (Efron and Tibshirani, 1993) | Intervals on pass rates and on paired differences between runs | Chapters 7 and 9 |
| Paired comparison | The same questions through two versions, so that the difference is between versions and not between question sets | Chapter 7 |
| Stratified sampling | Review samples that cover every language and intent, with oversampling of failures reported separately | Chapter 8 |
Program methods and tools
| Resource | Use | In this guide |
|---|---|---|
| PI-FT toolkit and Data Discoverability | Retrieval evaluation with Recall@k, MRR, and nDCG, and graded evaluation with a model as judge | Chapters 3 and 6 |
| Proof-Carrying Numbers | Verification of stated numbers against the source, the basis of numeric accuracy | Chapter 4 |
| Monitoring of Data Use | Mention extraction with labelled samples and precision and recall | Chapter 5 |
| Metadata Reviewer | Human review of model suggestions with recorded decisions, the source of curation evaluation sets | Chapter 2 |
| The evaluation chapters of the other cookbooks (dissemination, agents, monitoring, curation) | The component-specific suites this guide generalizes | All chapters |
Frameworks and formats
| Standard | Use | In this guide |
|---|---|---|
| Inspect | An open-source evaluation framework for language model systems with tool use and graders | Chapters 4, 6, and 7 |
| ranx and BEIR | Retrieval evaluation libraries and benchmark formats | Chapter 3 |
| Model cards and datasheets for datasets | Documentation formats for the component and for the question set | Chapter 9 |
| ISO/IEC 42001 and NIST AI RMF | Management and risk frameworks that require measured performance and monitoring | Chapters 1 and 8 |
Corrections
Standards change versions and URLs. If a reference here is wrong or out of date, open an issue with the row and the corrected source.