7. Evaluation of agent access

Question: How is the interface evaluated?

Rationale

An agent interface has more ways to fail than an API. The model can call the wrong tool, or the right tool with the wrong code. It can read the right value and misstate it. It can answer a question it should have declined, or answer correctly without a citation. None of these failures shows in the server's logs as an error, because from the server's point of view every call succeeded.

The only way to see them is to run questions with known answers through an assistant, log what it called and what it answered, and score the result. The question set of chapter 1 is the suite. This chapter runs it, scores it, and makes the scores the condition for any change to go live.

Target state

• The question set is run through at least one assistant on every change to the server, the tools, the guidance, or the catalog.
• The traces are scored for tool selection, series, value, citation, and behaviour, per language, and the scores are kept with the version they were measured on.
• A sample of answers is reviewed by a person for wording, caveats, and the efficiency of the tool calls.

Maturity levels

Level	Steps
Foundational	The first ten questions are tried by hand with an assistant connected to the API or the server, and what went wrong is written down.
AI-ready	The question set runs through a harness that logs traces. The traces are scored with the evaluation script, a sample is reviewed, and the results are kept.
AI-native	The suite runs on every change, per language and per assistant. Logged real questions are added to it, and publication is gated on the scores.

Running example

Eight questions run through an assistant connected to the example server. Six pass every check. The French question receives an answer that states 6.3 percent where the quarter's value is 6.1, which the value check flags. The regional question receives a regional figure the series does not have, which the behaviour check flags, because the assistant should have declined.

Recipes

Recipe

7.1 Run the question set and log the traces

Level

AI-ready

Skills

Developer

Indicative time

One day to set up the harness; minutes per run

Result: for every question in the set, the organization has a record of what the assistant did: the tools it called, the series it used, the answer it gave, whether it declined, and how long it took.

Such a record is called a trace. A harness is a small program that sends each question to an assistant, captures the tool calls and the answer, and writes the trace. The traces are kept as a file with one JSON object per line, with these fields:

Field	Meaning
question_id	The question from the set.
tools_called	The tools the assistant called, in order.
series	The series identifier it used, if any.
answer	The final answer text.
declined	Whether the assistant declined to answer.
latency_s	Seconds from question to answer.

The traces of the running example:

{"question_id": "a01", "tools_called": ["search_series", "get_observations"], "series": "FS_UNDERNOURISH_PCT", "answer": "The prevalence of undernourishment was 7.6% of the population in 2024 (release of 14 March 2025). Source: https://stats.example/series/FS_UNDERNOURISH_PCT", "declined": false, "latency_s": 2.1}
{"question_id": "a02", "tools_called": ["search_series", "get_observations"], "series": "FS_UNDERNOURISH_PCT", "answer": "Undernourishment fell from 8.4% in 2022 to 7.9% in 2023 and 7.6% in 2024 (three-year averages). Source: https://stats.example/series/FS_UNDERNOURISH_PCT", "declined": false, "latency_s": 2.4}
{"question_id": "a03", "tools_called": ["search_series", "get_observations"], "series": "LF_UNEMP_PCT", "answer": "The unemployment rate was 6.1% of the labour force in 2025-Q2; this value is provisional. Source: https://stats.example/series/LF_UNEMP_PCT", "declined": false, "latency_s": 1.9}
{"question_id": "a04", "tools_called": ["search_series", "get_series_metadata"], "series": "FS_UNDERNOURISH_PCT", "answer": "Prevalence of undernourishment is the share of the population whose habitual food consumption is insufficient to provide the dietary energy levels required to maintain a normal active and healthy life, as a three-year average. Source: https://stats.example/series/FS_UNDERNOURISH_PCT", "declined": false, "latency_s": 1.6}
{"question_id": "a05", "tools_called": ["search_series", "get_observations"], "series": "LF_UNEMP_PCT", "answer": "Le taux de chômage était de 6,3 % de la population active au deuxième trimestre 2025 (valeur provisoire). Source : https://stats.example/series/LF_UNEMP_PCT", "declined": false, "latency_s": 2.7}
{"question_id": "a06", "tools_called": ["search_series"], "series": null, "answer": "The organization does not publish an inflation rate series. The closest published series concern employment and food security.", "declined": true, "latency_s": 1.2}
{"question_id": "a07", "tools_called": ["search_series", "get_observations"], "series": "HE_U5_MORT_RATE", "answer": "The under-five mortality rate in the Northern Region was 41 per 1,000 live births in 2020. Source: https://stats.example/series/HE_U5_MORT_RATE", "declined": false, "latency_s": 2.3}
{"question_id": "a08", "tools_called": ["search_series", "get_observations"], "series": "ED_PRIMARY_NER_PCT", "answer": "Net primary enrolment rose from 91.2% in 2023 to 92.0% in 2024. Source: https://stats.example/series/ED_PRIMARY_NER_PCT", "declined": false, "latency_s": 2.0}

1. Connect an assistant to the server. This can be an assistant with tool use or a small harness built on a tool-use API. Load the guidance resource as its instructions, so that the run reflects what a well-configured client sees.
2. Send each question of the set and log the trace in the form above.
3. Run with more than one model where the organization can. Tool selection differs between models, and the server's descriptions have to work for the assistants that users bring.
4. Keep the traces with the date, the server version, the model, and the guidance version, so that a later run can be compared with this one.

Worked example. The trace of question a03, "What is the unemployment rate in the latest quarter?", records two tool calls, search_series and then get_observations, the series LF_UNEMP_PCT, and the answer "The unemployment rate was 6.1% of the labour force in 2025-Q2; this value is provisional. Source: https://stats.example/series/LF_UNEMP_PCT". The assistant did not decline, and the answer took 1.9 seconds. Every field the scoring script needs is in that one line.

What this does not do. The harness runs the organization's questions through an assistant the organization controls. Third-party assistants that users bring are observed only through the AI visibility check of the dissemination cookbook.

Recipe

7.2 Score the traces

Level

AI-ready

Skills

Python

Indicative time

Minutes

Result: five scores per run, for the right tools, the right series, the right value, a citation present, and the right behaviour, reported per language, with each failure listed so that its cause can be found.

The script compares each trace with the expected columns of the question set. A tool check passes when the trace's tool calls include the expected tools. A series check passes when the series matches. A value check passes when the expected value appears in the answer. A citation check passes when the answer contains a source URL. A behaviour check passes when the assistant answered or declined as expected.

"""Score an agent's logged runs against a question set.

Reads a question set (CSV: question_id, language, question, expected_tools
separated by ";", expected_series, expected_value, expected_behaviour
answer|decline) and the traces an agent harness logged (JSON lines:
question_id, tools_called, series, answer, declined, latency_s), and
reports per question and in total:

    tools     every expected tool was called
    series    the series the agent used is the expected one
    value     the expected value appears in the answer (within tolerance)
    cite      the answer contains a source URL
    behave    the agent answered when it should and declined when it should

Exit status is 0 when every check passes, 1 otherwise. Uses pandas.

What this does not do: it scores what the trace records. Whether the
answer's wording is correct beyond the value and the citation, whether
caveats were repeated, and whether the tool calls were efficient are
judged by a reviewer on a sample. Eight questions give a demonstration; a
suite has 30 to 50 per language.

Usage:
    python agent_eval.py agent_questions.csv agent_traces.jsonl
"""

from __future__ import annotations

import argparse
import math
import re
import sys

import pandas as pd

NUMBER = re.compile(r"-?\d[\d,]*(?:[.,]\d+)?")
URL = re.compile(r"https?://\S+")
CHECKS = ("tools", "series", "value", "cite", "behave")


def numbers_in(text: str) -> list[float]:
    """Numbers in a text; '6,3' (a French decimal) is 6.3 and '12,500' is 12500."""
    out = []
    for raw in NUMBER.findall(text):
        cleaned = raw.replace(" ", "")
        cleaned = (
            cleaned.replace(",", ".")
            if re.fullmatch(r"-?\d+,\d{1,2}", cleaned)
            else cleaned.replace(",", "")
        )
        try:
            out.append(float(cleaned))
        except ValueError:
            continue
    return out


def judge(q: pd.Series, t: dict, tolerance: float) -> dict[str, bool | None]:
    """Apply the five checks to one question and its trace; None means the check does not apply."""
    expected_tools = [x for x in q["expected_tools"].split(";") if x]
    should_decline = q["expected_behaviour"] == "decline"
    r: dict[str, bool | None] = {
        "tools": all(tool in t.get("tools_called", []) for tool in expected_tools),
        "behave": bool(t.get("declined")) == should_decline,
        "series": None,
        "value": None,
        "cite": None,
    }
    if not should_decline:
        r["series"] = (
            (t.get("series") == q["expected_series"]) if q["expected_series"] else None
        )
        if q["expected_value"]:
            target = float(q["expected_value"])
            r["value"] = any(
                math.isclose(n, target, abs_tol=tolerance)
                for n in numbers_in(t.get("answer", ""))
            )
        r["cite"] = bool(URL.search(t.get("answer", "")))
    return r


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    parser.add_argument("questions")
    parser.add_argument("traces")
    parser.add_argument("--tolerance", type=float, default=0.05)
    args = parser.parse_args(argv)
    questions = pd.read_csv(args.questions, dtype=str).fillna("")
    traces = {
        t["question_id"]: t
        for t in pd.read_json(args.traces, lines=True).to_dict("records")
    }

    results, failures, latencies = [], [], []
    cell = lambda v: "-" if v is None else ("ok" if v else "FAIL")
    print(
        f"{'id':<4} {'lang':<5} {'tools':<6} {'series':<7} {'value':<6} {'cite':<5} {'behave':<7} question"
    )
    for _, q in questions.iterrows():
        t = traces.get(q["question_id"])
        if t is None:
            failures.append(f"{q['question_id']}: no trace")
            continue
        latencies.append(float(t.get("latency_s", 0)))
        r = judge(q, t, args.tolerance)
        results.append(r)
        failures += [
            f"{q['question_id']}: {c} failed ({q['note'] or q['question']})"
            for c in CHECKS
            if r[c] is False
        ]
        print(
            f"{q['question_id']:<4} {q['language']:<5} {cell(r['tools']):<6} {cell(r['series']):<7} {cell(r['value']):<6} {cell(r['cite']):<5} {cell(r['behave']):<7} {q['question']}"
        )

    print()
    scores = pd.DataFrame(results)
    for c in CHECKS:
        applicable = scores[c].notna().sum()
        if applicable:
            print(
                f"{c:<7} {int(scores[c].sum())}/{applicable}  {scores[c].sum() / applicable:.2f}"
            )
    if latencies:
        s = pd.Series(latencies)
        print(f"latency median {s.median():.1f}s, max {s.max():.1f}s")
    if failures:
        print("\nfailures:")
        for f in failures:
            print(f"  {f}")
    print(
        "\nNot checked here: wording, caveats, and efficiency of the tool calls; a reviewer samples those."
    )
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main())

It runs with the question set and the traces as arguments:

python agent_eval.py agent_questions.csv agent_traces.jsonl

For the running example the output, abridged to the rows that matter, is:

id   lang  tools  series  value  cite  behave  question
a01  en    ok     ok      ok     ok    ok      What was the prevalence of undernourishment in 2024?
a05  fr    ok     ok      FAIL   ok    ok      Quel est le taux de chômage au deuxième trimestre 2025?
a06  en    ok     -       -      -     ok      What is the inflation rate?
a07  en    FAIL   -       -      -     FAIL    What is the under-five mortality rate in the Northern Region?
tools   7/8  0.88
series  6/6  1.00
value   4/5  0.80
cite    6/6  1.00
behave  7/8  0.88

The table has one row per question with ok, FAIL, or - for each check, where - means the check does not apply, for example a value check on a question that should be declined. The summary lines give each score as a fraction and a rate.

1. Read each failure to its cause. In the run above, the French failure is a wrong figure in the answer, which the number check of chapter 4 also catches. The regional failure is a missing refusal. The guidance resource, and a list_code_values call before get_observations, are the fixes for it.
2. Fix the cause, whether it is in the guidance text, a tool description, a code-list check, or a catalog field, and run the set again. The scores show whether the fix worked.
3. Keep the scores per language and per model with the version.

Worked example. Question a07 asks for the under-five mortality rate in the Northern Region. The expected behaviour is decline, because the series is national only. The trace shows that the assistant called search_series and get_observations without calling list_code_values, and answered that the rate in the Northern Region was 41 per 1,000 live births in 2020. That is the national figure, presented as the region's; no response contained a regional value. The tool check fails because the expected list_code_values call is missing, and the behaviour check fails because the assistant answered. The fix has two parts: the guidance text now says "call list_code_values before asking for a region", and the metadata of the series now states that it is published at the national level only. In the next run the assistant declines and explains why.

What this does not do. The script scores what the trace records. Wording, repeated caveats, and the efficiency of the tool calls are judged by a reviewer in recipe 7.3. Eight questions demonstrate the method; a suite has 30 to 50 per language.

Recipe

7.3 Review a sample and gate changes

Level

AI-ready

Skills

Subject-matter staff; developer

Indicative time

Half a day per run

Result: the organization has a measure of answer quality beyond what the script can check, and a rule that no change to the server goes live if the scores fall.

A script can confirm that the right number appears in an answer. It cannot tell whether the answer repeated the caveat in a way a reader would notice, whether it was concise, or whether the assistant called the tools five times where two would do. A reviewer reads a sample for those things, and the sample includes answers that passed every automated check, because those are the ones nobody would otherwise look at.

1. Draw 20 answers from each run, including some that passed every automated check. Score each on a short rubric: right series, right number, period and unit stated, caveats repeated, citation present, nothing invented, and concise.
2. Add every failure to the question set with the expected answer. The suite grows from its own failures.
3. Set the gate. A change to the server, the tools, the guidance, or the catalog ships only when the automated scores are at or above the previous run's scores, per language, and the reviewed sample shows no new kind of failure.
4. Run the suite again after model updates on the assistants the organization operates. A model change is a change like any other.

Worked example. In the review of the running example's run, the answer to question a01 passed every automated check. The reviewer still marked it down on the rubric, for two reasons. It gave the 2024 value without the 2023 value that the question's context invited, and it repeated the series definition in full where one phrase would do. The reviewer added a note to the guidance text asking for the previous value where a trend is implied. The answer to a03 scored full marks: it stated the value, the quarter, the unit, the provisional status, and the citation in two sentences.

What this does not do. A gate holds the line. It does not raise it. The scores rise when causes are fixed in the catalog, the API, and the guidance, which the dissemination cookbook's chapters cover.

Implementation options

• Harness. An assistant with tool use connected to the server, or a small program on a tool-use API that logs the calls. The official SDK's client can drive the server directly.
• Scores. Tool selection, series, value, citation, and behaviour, with latency, and the number check over the logged responses.
• Review. A rubric and a sample per run. A language model can grade first, with a person checking a share of its grades, as the dissemination cookbook's evaluation chapter describes.
• Gate. Scores are kept per version, and publication is blocked when they fall below the previous run.

World Bank examples

• The dissemination cookbook's evaluation chapter provides the question-set method and the retrieval and answer measures that this chapter extends with tool selection and behaviour.
• Proof-Carrying Numbers provides the number verification applied to logged answers.

Resources

• The MCP Python SDK client drives the server directly in a harness that logs every call.
• The evaluation chapter of the dissemination cookbook describes the measures, the question-set discipline, and the grader pattern this chapter extends.
• Proof-Carrying Numbers describes the verification of stated numbers against logged responses.
• The agent evaluation script and the traces are the scoring of the running example.
• Inspect is an open-source framework for evaluating language model agents with tool use, suited to larger suites.

Common mistakes

• Testing with one model. Tool selection differs between assistants. The descriptions have to work for several.
• Scores without causes. A failed value and a missing refusal need different fixes. The failure list names them.
• A suite that never grows. Real questions and real failures are the source of new items.
• Changes without a re-run. A one-word change to a tool description changes tool selection.

Verification

• Runs. Traces and scores exist for the current version, per language and per model.
• Causes. Every failure has a recorded cause and either a fix or a decision.
• Gate. The last change shipped with scores at or above the previous run.

Checklist

0 of 6 done

• Harness logs traces with tools, series, answer, declined, latency
• Question set run on every change, with more than one model where possible
• Scores for tools, series, value, cite, behave, per language
• Failures read to their cause and fixed at the source
• Sample reviewed on a rubric; failures added to the set
• Publication gated on scores and the reviewed sample

Ticks are saved in this browser only.