Select the statement that best describes the team today, one per chapter.
The result shows a level for each chapter and the chapters to read first.
Answers are stored in this browser only.
1. Inventory Which components are evaluated?Not yet None in a systematic way; demonstrations are judged by eye. Foundational A list of AI components exists with the decision each makes and a required score for the most important. AI-ready Every component has a suite, ordered by the cost of its errors, with required scores set before testing. AI-native The inventory is reviewed each period from incidents and production samples. 2. Questions Where do the test questions come from?Not yet Staff write a few examples when needed. Foundational A versioned set with expected answers, including questions with no answer. AI-ready Real queries and failures feed the set; two labellers with measured agreement; every supported language covered. AI-native The set grows each month from production samples, with a labelling guide maintained from disagreements. 3. Retrieval How is search measured?Not yet By trying a few queries. Foundational Recall@k and MRR on the set, scored by a script. AI-ready Scores per language and per question type, with nDCG where graded relevance exists, and no-answer questions scored by refusal. AI-native Scores tracked per version with intervals, and slices below the threshold block a change. 4. Answers How are generated answers measured?Not yet By reading some. Foundational Numeric accuracy and citation validity scored automatically on the set. AI-ready Refusal accuracy and unsupported numbers added; a grader with a rubric for caveats and period; per language. AI-native Answer scores gate every change of model, prompt, index, or metadata. 5. Extraction How are extraction and classification measured?Not yet Not measured. Foundational Precision and recall on a labelled sample. AI-ready Exact and lenient matching, an error taxonomy, and scores per document type and language. AI-native The taxonomy drives fixes, and the labelled sample grows from production errors. 6. Graders How are automatic graders validated?Not yet They are trusted. Foundational A sample of grades is spot-checked by a person. AI-ready Agreement and kappa with human scores measured on a hundred items; length and position bias checked; rubric adjusted. AI-native Graders are re-validated on every rubric or model change, with the validation kept with the suite. 7. Gates How do changes ship?Not yet When they look better. Foundational The suite runs before a change and the overall score must not drop. AI-ready Paired comparison per slice with intervals and a tolerance; small slices go to a person. AI-native The gate runs in continuous integration, and the suite versions with the component. 8. Production How is a deployed component evaluated?Not yet It is not, after launch. Foundational Logged traffic is reviewed occasionally. AI-ready A stratified sample of traffic is reviewed each month, declined and down-rated queries included; failures enter the set. AI-native Drift in scores and in traffic mix is monitored with thresholds, and the suite is re-run on schedule. 9. Reporting How are results reported?Not yet In conversation. Foundational A score per release. AI-ready A report card per release with intervals, slices, set version, and model versions. AI-native Report cards are published, and sets follow a shared format so that organizations can compare. 0 of 9 answered Show reading path Reset
Interpretation of the result
Chapters 1 and 2 first: without an inventory and a real question set,
every measure in the later chapters measures the wrong thing well.
Chapter 7 is the one that changes behaviour: a suite that gates changes
is run; a suite that only reports is forgotten.
Chapter 6 before relying on any grader: an unvalidated grader is a
second system to be wrong.