6. Evaluation

Question: Can we tell whether our AI-ready steps improved anything?

Rationale

A deeper treatment exists

This chapter gives the first steps. The Evaluation Suites for Statistical AI cookbook covers question sets, labelling, retrieval and answer measures, graders, gates, and report cards in full, with scripts and a running example.

Each step in this guide is meant to improve a measurable outcome. After the metadata work, more questions should find the right series. After the provenance work, more answers should carry the right number and its source. Without a fixed set of test questions, none of that can be shown. A change to the metadata, the search, or the model cannot be compared with what came before, and problems appear first in front of users.

A question set is a small investment that pays on every later change. It is a file of questions whose right answer is known, and it is run before and after each change, so that the scores say whether the change helped. This chapter builds the set, scores the search with it, and then extends it to the answers an assistant gives.

Target state

• A small, versioned set of test questions with known correct answers.
• The same measures are run before and after each change.
• Results are reported by language, topic, and question type.
• Failures are reviewed and added to the test set.

Maturity levels

Level	Steps
Foundational	Write 30 to 50 known-item questions. Run them by hand against the search page each quarter and record the rank of the correct result.
AI-ready	Automate the run. Report Recall@k and MRR for retrieval. Keep the question set in version control. Test each supported language separately.
AI-native	Add answer-level tests: numeric accuracy, citation validity, and correct refusals. Use a language model as a first-pass grader with human spot checks. Run the suite before every model or metadata change.

Running example

Fifteen questions over the five-series catalog, in three languages, including one question with no answer. Keyword search finds 8 of the 10 English questions and none of the French or Spanish ones. That is the baseline the semantic search in chapter 1 has to beat.

Recipes

Recipe

6.1 Build a known-item question set

Level

Foundational

Skills

Dissemination staff; speakers of each supported language

Indicative time

One day for the first 30 questions

Result: the organization has a file of questions, each paired with the series that should come first in the results. Anyone in the organization can add to it, and every later measurement runs on it.

A known-item question is one whose right answer is known: for "how many people go hungry", the right series is the prevalence of undernourishment. The set is a CSV with one row per question:

Column	Meaning
question_id, language	The question's identifier and its language, coded with ISO 639 (en, fr, es)
question	The question as a user would type it
expected_id	The series that should come first, or NONE when the organization publishes no matching series
note	Whether the question is a paraphrase, uses the official title, is a near concept, or needs review

question_id,language,question,expected_id,note
q01,en,how many people go hungry,FS_UNDERNOURISH_PCT,paraphrase
q02,en,share of kids in primary school,ED_PRIMARY_NER_PCT,paraphrase
q03,en,child deaths before age 5,HE_U5_MORT_RATE,paraphrase
q04,en,jobless rate,LF_UNEMP_PCT,paraphrase; description field empty
q05,en,how many people are poor,PV_HEADCOUNT_NPL_PCT,paraphrase
q06,en,prevalence of undernourishment,FS_UNDERNOURISH_PCT,official title
q07,en,unemployment rate latest quarter,LF_UNEMP_PCT,official title with period
q08,en,net enrolment primary,ED_PRIMARY_NER_PCT,official title fragment
q09,en,infant mortality,HE_U5_MORT_RATE,near concept; under-five is the closest published series
q10,en,poverty line,PV_HEADCOUNT_NPL_PCT,keyword
q11,fr,taux de chômage,LF_UNEMP_PCT,needs native-speaker review
q12,fr,mortalité des enfants de moins de cinq ans,HE_U5_MORT_RATE,needs native-speaker review
q13,es,tasa de desempleo,LF_UNEMP_PCT,needs native-speaker review
q14,es,desnutrición,FS_UNDERNOURISH_PCT,needs native-speaker review
q15,en,inflation rate,NONE,not published; the system should return nothing confident

1. Take real questions from search logs and help-desk tickets first.
2. Write paraphrases of the official title ("jobless rate"), since users rarely use the exact words. Mark which questions are paraphrases and which use the official title; the scores for the two groups differ and both matter.
3. Include near concepts where the organization publishes something close but different ("infant mortality" when only under-five mortality exists).
4. Include questions with no answer and mark the expected identifier NONE.
5. Include each supported language, written by speakers of that language. Machine translation is a starting point and needs review; the example file marks its French and Spanish rows as unreviewed.
6. Keep the file in version control and record who added each question.

Worked example. The running example's file has fifteen rows. The first three are English paraphrases: "how many people go hungry" expects the undernourishment series, "share of kids in primary school" expects the net enrolment rate, and "child deaths before age 5" expects the under-five mortality rate. Other rows use the official titles, so that the two groups can be scored apart. One row asks about inflation and expects NONE. Two rows are in French and two in Spanish, and their notes say "needs review", because they were drafted by machine translation and no speaker has checked them yet.

What this does not do. A question set written by the organization reflects what the organization expects users to ask. Real queries from search logs (recipe 4.1) and failed answers (recipe 4.3) correct it over time. A set that is never extended measures last year's users.

Recipe

6.2 Score retrieval automatically

Level

AI-ready

Skills

Python

Indicative time

Two hours

Result: for each language, three numbers say how well the search finds the right series. The script prints them in seconds, and it works for any search system that returns a ranked list of identifiers.

Recall@5 and Recall@10 are the share of questions for which the right series is in the first five, or ten, results. MRR is the mean reciprocal rank: 1 when the right series comes first, one half when it comes second, and so on, averaged over the questions. The script includes a plain keyword search so that it runs with no dependencies and gives a baseline. The organization replaces the search() function with a call to its own search to score it:

"""Score a search system on a known-item question set.

Reads a question set (CSV with question_id, language, question, expected_id)
and a catalog, runs each question through a search function, and prints
Recall@5, Recall@10, and MRR overall and per language.

The built-in search is a plain keyword overlap over the catalog's `name` and
`definition_long` fields (World Bank indicator schema names). It exists so
that the script runs without any dependencies and gives a baseline. Replace
`search()` with a call to your own search API to score it.

Uses pandas and scikit-learn (TF-IDF for the baseline search).

Usage:
    python score_retrieval.py eval_questions.csv example_catalog.csv
"""

from __future__ import annotations

import argparse
import re
import sys

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

TOKEN = re.compile(r"\w+")  # Unicode-aware: words in any script
ID_FIELD = "idno"
TEXT_FIELDS = ("name", "definition_long")
K_VALUES = (5, 10)


def tokens(text: str) -> set[str]:
    return set(TOKEN.findall(text.lower()))


class KeywordSearch:
    """A TF-IDF keyword search over the catalog records: the baseline to replace with the real search."""

    def __init__(self, catalog: pd.DataFrame) -> None:
        self.ids = catalog[ID_FIELD].tolist()
        texts = catalog[list(TEXT_FIELDS)].fillna("").agg(" ".join, axis=1)
        self.vectorizer = TfidfVectorizer(token_pattern=TOKEN.pattern, lowercase=True)
        self.matrix = self.vectorizer.fit_transform(texts)

    def __call__(self, question: str, k: int = 10) -> list[str]:
        """Return up to k catalog ids ranked by similarity; records that share no word are left out."""
        scores = cosine_similarity(self.vectorizer.transform([question]), self.matrix)[
            0
        ]
        ranked = sorted(
            (i for i in range(len(self.ids)) if scores[i] > 0),
            key=lambda i: (-scores[i], i),
        )
        return [self.ids[i] for i in ranked[:k]]


def search(question: str, catalog: list[dict[str, str]], k: int = 10) -> list[str]:
    """Rank the catalog for one question; kept as a function so that a real search can replace it."""
    return KeywordSearch(pd.DataFrame(catalog))(question, k)


def rank_in(ranked: list[str], expected: str) -> float:
    return ranked.index(expected) + 1 if expected in ranked else float("nan")


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    parser.add_argument("questions")
    parser.add_argument("catalog")
    args = parser.parse_args(argv)
    catalog = pd.read_csv(args.catalog, dtype=str)
    questions = pd.read_csv(args.questions, dtype=str)
    questions = questions[questions["expected_id"] != "NONE"].copy()
    find = KeywordSearch(catalog)

    print(f"{'id':<5} {'lang':<5} {'rank':>4}  question")
    ranks = []
    for q in questions.itertuples():
        rank = rank_in(find(q.question, k=max(K_VALUES)), q.expected_id)
        ranks.append(rank)
        print(
            f"{q.question_id:<5} {q.language:<5} {'-' if pd.isna(rank) else int(rank):>4}  {q.question}"
        )
    questions["rank"] = ranks
    for k in K_VALUES:
        questions[f"R@{k}"] = (questions["rank"] <= k).astype(float)
    questions["MRR"] = (1 / questions["rank"]).fillna(0.0)

    groups = pd.concat(
        [questions.assign(group="all"), questions.assign(group=questions["language"])]
    )
    table = groups.groupby("group", sort=False)[
        [f"R@{k}" for k in K_VALUES] + ["MRR"]
    ].agg(["mean", "size"])
    print()
    print(
        f"{'group':<6} {'n':>3} "
        + " ".join(f"{'R@' + str(k):>6}" for k in K_VALUES)
        + f" {'MRR':>6}"
    )
    for group in ["all"] + sorted(g for g in table.index if g != "all"):
        r = table.loc[group]
        recalls = " ".join(f"{r[(f'R@{k}', 'mean')]:>6.2f}" for k in K_VALUES)
        print(
            f"{group:<6} {int(r[('MRR', 'size')]):>3} {recalls} {r[('MRR', 'mean')]:>6.2f}"
        )
    return 0


if __name__ == "__main__":
    sys.exit(main())

Output for the running example (keyword baseline):

group    n    R@5   R@10    MRR
all     14   0.57   0.57   0.57
en      10   0.80   0.80   0.80
es       2   0.00   0.00   0.00
fr       2   0.00   0.00   0.00

The first row scores all questions together; the other rows score each language. The n column counts the questions with an expected series, so the NONE question is left out.

1. Run the script with the baseline and keep the output with the date. Here the baseline misses the two English paraphrases that share no words with the records ("how many people go hungry", "how many people are poor") and every non-English question.
2. Plug in the semantic search of recipe 1.3 and run again. A multilingual semantic search should fix both kinds of miss, and the script shows by how much.
3. Report both, per language, with the model name and the catalog version.

Worked example. With the keyword baseline, eight of the ten English questions find their series in the top five, which gives a Recall@5 of 0.80. The two misses are the paraphrases that share no words with any record. The French and Spanish questions all miss, because the records are in English only. After the semantic search of chapter 1 is plugged in, the English Recall@5 rises to 1.00 and the French and Spanish questions find their series as well. The team records both runs with the date, the model name intfloat/multilingual-e5-small, and the catalog version, and the pair of outputs becomes the first entry in the evaluation log.

What this does not do. Fifteen questions over five records give a demonstration, and the scores move by 0.07 with a single question. A usable set has at least 30 to 50 questions per language and is drawn from real queries, and differences between systems are read with that sample size in mind. The built-in keyword search is a baseline for the exercise; the organization's own search will be better than it.

Recipe

6.3 Add answer-level tests

Level

AI-native

Skills

Python; subject-matter reviewers

Indicative time

One week to set up; part of each release

Result: the assistant has four scores that are reported with every change: numeric accuracy, citation validity, refusal accuracy, and hallucination rate, which is the share of answers that contain a claim with no support in the retrieved records.

1. For each question in the set, add the expected answer values and the expected behaviour (answer or decline).
2. Run every question through the assistant and keep the answer with the retrieved records.
3. Score automatically what can be scored automatically:
   • Numeric accuracy: run the number check of chapter 5 on each answer; the share of verified numbers.
   • Citation validity: each cited series identifier and URL exists and was among the retrieved records.
   • Refusal accuracy: the share of NONE questions where the assistant declined.
4. Use a language model as a first-pass grader for the rest, with a short rubric (right series, right period, caveats stated), and have a reviewer check 20% of the grades each run. Disagreements go into the rubric.
5. Run the whole suite before any change of model, index, prompt, or metadata, and keep the results with the change.

Worked example. The team extends the fifteen questions with expected values and behaviours. "How many people go hungry" now expects the value 7.6 and the behaviour answer; the inflation question expects decline. A run of the assistant produces fifteen answers. The number check verifies every number in thirteen of them, which gives a numeric accuracy of 0.87. Every cited series was among the retrieved records, so citation validity is 1.00. The inflation question was declined, so refusal accuracy is 1.00. The grader flags one answer that described a trend the records did not support, which gives a hallucination rate of 0.07, and the reviewer, checking three of the fifteen grades, agrees with all three.

The program's PI-FT pipeline includes an evaluation step that reports Recall@k, MRR, and nDCG@k and compares several embedding models on the same queries, and its method page describes graded evaluation with a language model as judge. The evaluation cookbook covers the full suite.

What this does not do. Automatic scores cover numbers, citations, and refusals. The grader covers the rest with an error rate known from the reviewer's checks. An answer that is right and unhelpful (the right series without the caveat the user needed) scores as right unless the rubric names the caveat.

Measures

Retrieval

• Recall@k: the share of questions for which the correct item appears in the top k results.
• MRR (mean reciprocal rank): the average of 1 divided by the rank of the first correct result.
• nDCG@k: a graded measure that credits partially relevant results.

Answers

• Numeric accuracy: the share of numbers in answers that match the official value, with a stated tolerance for rounding.
• Citation validity: the share of citations that resolve and support the statement.
• Refusal accuracy: the share of unanswerable questions that the system declines.
• Hallucination rate: the share of answers that contain a claim with no support in the retrieved records.

Implementation options

• Question sets: a versioned CSV with expected ids, languages, paraphrase flags, and NONE rows; real queries added each month.
• Retrieval scoring: the script in this chapter (Recall@k, MRR); ranx or the PI-FT pipeline's evaluation step for larger suites and more measures.
• Answer scoring: automatic checks for numbers, citations, and refusals; a language-model grader with a rubric and a reviewed share of grades; Inspect or a similar framework for suites with tool use.
• Gates: scores kept per change of model, index, prompt, or metadata; publication blocked below the previous run.

Resources

• Evaluation questions and score_retrieval.py: the running example's question set and the retrieval scorer.
• BEIR and MTEB: retrieval benchmarks and the measures (recall, MRR, nDCG) the scorer follows, for comparison with published models.
• ranx: a library for information retrieval evaluation with the standard measures, for suites larger than the script here.
• Inspect: an open-source framework for answer-level evaluation of language model systems with graders.
• Data Discoverability and the PI-FT toolkit: the program's evaluation of search by meaning over structured records.

Common mistakes

• Testing with the official titles only. Scores look good and users still fail. Paraphrases are the test.
• One language. See the baseline above: 0.80 in English and 0.00 elsewhere is one system.
• Changing the question set and the system at the same time. The scores cannot be compared. Version the set.
• Grading with a model and never checking. Model graders drift and share blind spots with the system under test. Spot-check every run.

Verification of the suite

Confirm that a deliberately worse configuration, such as search with empty descriptions, scores lower. If it does not, the suite is not measuring what matters.

Checklist

0 of 8 done

• Versioned question set with known answers
• Paraphrases, near concepts, and no-answer questions included
• Each supported language included and reviewed by a speaker
• Recall@k and MRR reported for retrieval, per language
• Baseline scores kept with date and configuration
• Numeric accuracy, citation validity, and refusal accuracy measured for answers
• Human review of a sample of graded results each run
• Suite run before every model, index, prompt, or metadata change

Ticks are saved in this browser only.