7. Gates on changes

Question: How does the suite gate changes?

Rationale

A suite that reports is read once; a suite that gates is run every time. The gate is a rule with five parts. The same questions run through the version before and the version after a change. The difference is computed per slice. An interval around the difference says what thirty or three hundred questions can decide. A tolerance states how much regression the organization accepts. And the rule says what happens when a slice is too small to decide. A change that passes ships; a change that regresses a slice is blocked; a change that may have regressed a small slice goes to a person, and the set grows there. The rule is written down and run by a machine, so that it applies to every change of model, prompt, index, or metadata, including the ones made in a hurry.

Target state

• Every change to a component runs the suite before and after, paired by question, with the difference per slice and a bootstrap interval.
• A tolerance for regression is set per component, and a slice whose interval shows a regression beyond it blocks the change.
• Slices too small to decide send the change to a person, and the question set grows in that slice.
• The gate runs in continuous integration, and the component and the set are versioned together.

Maturity levels

Level	Steps
Foundational	Run the suite before and after a change by hand; do not ship a lower overall score.
AI-ready	Pair the runs; compute differences per slice with intervals; set the tolerance; apply the review rule; record every decision.
AI-native	Run the gate in continuous integration on every change; version the component with the set; report the held-out part separately.

Running example

Two runs of thirty questions. Overall the change gains three points with an interval from minus seventeen to plus twenty-three; French goes from two in six to five in six; Spanish drops from four in six to two in six. Nothing is decidable at thirty questions; the Spanish drop and a drop on the no-answer questions send the change to a person.

Recipes

Recipe

7.1 Compare paired runs with intervals and a tolerance

Level

AI-ready

Skills

Python

Indicative time

Minutes per change

Result: every change to the system has a comparison of the test results before and after, slice by slice, with a statement of how sure the comparison is and a verdict. The verdict says that the change may ship, must not ship, or needs a person to decide.

The comparison is paired: the same questions are run through the version before the change and the version after, so that any difference comes from the change. With thirty questions, a difference of a few points can be luck; the script puts an interval around each difference that says how far the true difference could lie from the measured one. The organization also sets a tolerance, the drop in score it is willing to accept on any slice, from the cost of error in chapter 1.

The two example files record, for thirty questions, whether each passed in version 1 and in version 2, with each question's language and slice. The script compares them:

"""Compare two evaluation runs and decide whether a change may ship.

Reads two runs of the same question set (CSV: question_id, language,
slice, pass 0|1) and reports, overall and per language and slice, the
pass rate of each run, the difference, and a bootstrap confidence
interval on the difference over the paired questions. The gate rule:
the change ships when no slice's interval lies entirely below zero
minus the tolerated regression (exit 1 otherwise). A slice that dropped
beyond the tolerance on too few questions for the interval to decide
sends the change to a person (exit 2). Standard library only; the
bootstrap uses a fixed seed so that the result is reproducible.

Uses pandas and scipy (percentile bootstrap).

Usage:
    python compare_runs.py run_scores_v1.csv run_scores_v2.csv --tolerance 0.05

What this does not do: thirty questions give wide intervals, which is
the point: the script shows how little thirty questions can decide. A
slice that the set does not cover (a language with two questions) is
not protected by the gate, and the question chapter is where the set
grows.
"""

from __future__ import annotations

import argparse
import sys

import numpy as np
import pandas as pd
from scipy.stats import bootstrap


def interval(diffs: pd.Series, seed: int = 7) -> tuple[float, float]:
    """95% percentile bootstrap interval for the mean difference."""
    if (
        diffs.nunique() == 1
    ):  # bootstrap needs variation; a constant difference has none
        return float(diffs.iloc[0]), float(diffs.iloc[0])
    res = bootstrap(
        (diffs.to_numpy(),),
        np.mean,
        confidence_level=0.95,
        method="percentile",
        random_state=seed,
    )
    return float(res.confidence_interval.low), float(res.confidence_interval.high)


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    parser.add_argument("before")
    parser.add_argument("after")
    parser.add_argument(
        "--tolerance",
        type=float,
        default=0.05,
        help="regression tolerated on any slice",
    )
    args = parser.parse_args(argv)
    a = pd.read_csv(args.before, dtype={"question_id": str})
    b = pd.read_csv(args.after, dtype={"question_id": str})
    paired = a.merge(
        b[["question_id", "pass"]], on="question_id", suffixes=("_before", "_after")
    )
    paired["diff"] = paired["pass_after"] - paired["pass_before"]
    groups = pd.concat(
        [
            paired.assign(group="all"),
            paired.assign(group="language=" + paired["language"]),
            paired.assign(group="slice=" + paired["slice"]),
        ]
    )

    print(f"{len(paired)} paired questions; tolerance {args.tolerance:.2f}\n")
    print(
        f"{'group':<18} {'n':>3} {'before':>7} {'after':>6} {'diff':>6} {'95% interval':>16}  verdict"
    )
    blocking: list[str] = []
    review: list[str] = []
    for name, g in groups.groupby("group", sort=False):
        before, after = g["pass_before"].mean(), g["pass_after"].mean()
        lo, hi = interval(g["diff"])
        if hi < -args.tolerance:
            verdict = "REGRESSION"
            blocking.append(name)
        elif after - before < -args.tolerance:
            verdict = "possible regression, too few to decide"
            review.append(name)
        else:
            verdict = "ok (small)" if len(g) < 30 else "ok"
        print(
            f"{name:<18} {len(g):>3} {before:>7.2f} {after:>6.2f} {after - before:>+6.2f} {f'[{lo:+.2f}, {hi:+.2f}]':>16}  {verdict}"
        )
    print()
    if blocking:
        print(f"gate: BLOCKED by {', '.join(blocking)}")
        return 1
    if review:
        print(
            f"gate: REVIEW ({', '.join(review)} dropped beyond the tolerance on too few questions to decide; a person decides, and the set grows there)"
        )
        return 2
    print(
        "gate: PASS (no slice shows a regression beyond the tolerance; small slices show direction only)"
    )
    return 0


if __name__ == "__main__":
    sys.exit(main())

python compare_runs.py run_scores_v1.csv run_scores_v2.csv --tolerance 0.05

Output for the running example:

30 paired questions; tolerance 0.05

group                n  before  after   diff     95% interval  verdict
all                 30    0.70   0.73  +0.03   [-0.17, +0.23]  ok
language=en         18    0.83   0.83  +0.00   [-0.22, +0.22]  ok (small)
language=fr          6    0.33   0.83  +0.50   [+0.17, +0.83]  ok (small)
language=es          6    0.67   0.33  -0.33   [-0.67, +0.00]  possible regression, too few to decide
slice=official       8    0.75   0.75  +0.00   [-0.50, +0.50]  ok (small)
slice=paraphrase    17    0.65   0.76  +0.12   [-0.12, +0.35]  ok (small)
slice=none           5    0.80   0.60  -0.20   [-0.60, +0.00]  possible regression, too few to decide

gate: REVIEW (language=es, slice=none dropped beyond the tolerance on too few questions to decide; a person decides, and the set grows there)

The table has one row per slice, with the pass rate before and after, the difference, a 95 percent interval around the difference from a paired bootstrap, and a verdict for the slice. The last line is the verdict for the change as a whole, with the slices that caused it.

1. Score both versions on the same version of the question set. The scripts of chapters 3 to 5 write the per-question pass files. Run the comparison on the two.
2. Read the verdicts. BLOCKED means a slice dropped beyond the tolerance and the whole interval lies below it; the change must not ship. REVIEW means a slice dropped beyond the tolerance on too few questions for the interval to decide; a person decides. PASS means neither happened.
3. Set the tolerance per component from its cost of error. A public assistant tolerates less than a search.
4. Record the verdict, the person who decided a review case, and the reason, together with the change.

Worked example. The change compared above replaced the embedding model behind the search. Overall the pass rate rose by three points, and the interval from minus 17 to plus 23 says that thirty questions cannot tell whether the change helped at all. French rose from two of six to five of six, and even that interval excludes zero only just. Spanish fell from four of six to two of six, and the no-answer slice fell from four of five to three of five. Both drops exceed the tolerance of five points, and both slices are too small for the interval to decide, so the verdict is REVIEW. The dissemination lead reads the four Spanish questions that failed, finds that the new model handles accented characters differently, and holds the change until the Spanish slice has twenty questions and the model's tokenizer has been checked.

What this does not do. Thirty questions decide very little, and the script says so: the intervals are the honest statement of it. The remedy is more questions in the slices that matter, which is chapter 2.

Recipe

7.2 Put the gate in continuous integration

Level

AI-native

Skills

Developer

Indicative time

One day

Result: every change to the component's code, prompts, search index, or settings runs the comparison automatically, and a change that fails cannot reach production.

Continuous integration is the practice of running tests automatically whenever code changes; most software teams already have a system for it. The comparison of recipe 7.1 becomes one of those tests.

1. Keep the question set, the scoring scripts, the required scores, and the tolerance in the same repository as the component, so that they are versioned together.
2. On every change, run the suite on the new version and compare it with the stored scores of the previous version on the same set version. Act on the verdict: fail the pipeline on BLOCKED; pause it on REVIEW with the output attached for a person.
3. Report the held-out part of the set separately, so that a change tuned on the rest is also seen on questions it never touched.
4. Store the run files and the decision with the change, so that a problem found later can be traced to the change that let it through.

Worked example. The search's repository holds the question set, the scoring script, the comparison script, and a file with the required scores and the tolerance. A job in the repository's continuous integration runs on every proposed change. It scores the new version on set version 1.2, compares the result with the stored run of the current version, and prints the table. A BLOCKED verdict fails the job, and the change cannot be merged. A REVIEW verdict marks the job as waiting and posts the table where the dissemination lead sees it. The held-out ten questions are scored in a separate row of the output, and the run file is stored with the change's identifier.

What this does not do. The gate protects what the question set covers. A change that breaks a kind of question the set lacks passes the gate, which is why the production review of chapter 8 keeps adding questions.

Recipe

7.3 Decide the review cases and grow the set where they occur

Level

AI-ready

Skills

Head of the unit; dissemination staff

Indicative time

An hour per review case

Result: every change the comparison sent to a person has a recorded decision, and the slice that was too small to decide has more questions.

1. For a REVIEW verdict, read the questions in that slice that failed after the change, with the answers before and after. A change that fixes one kind of question and breaks another is a change to split in two.
2. Decide: ship with the drop accepted and recorded, ship after a fix, or hold. Record the decision and the reason.
3. Add questions to the slice that could not decide, from logs and from native speakers, so that the next change can be decided there.
4. Review the tolerance and the required scores each period in the light of the decisions taken.

Worked example. The REVIEW verdict of recipe 7.1 is recorded as "held: Spanish regression traced to tokenization of accented characters; fix the preprocessing and re-run; Spanish slice to grow to twenty questions first". The Spanish-speaking statistician writes fourteen questions from the Spanish search log over the following week, and the slice reaches twenty. The fix is applied, the comparison runs again, and Spanish shows 0.70 before and 0.85 after on twenty questions, with an interval that excludes zero. The change ships, and the decision log carries both entries. At the quarterly review of the tolerance, the lead keeps five points for the search and sets two for the assistant.

What this does not do. A review case is a judgement. The record of past judgements is what keeps the next one consistent, and the growth of the set is the only thing that makes review cases rarer.

Implementation options

• Comparison: the script in this chapter; paired bootstrap on per-question outcomes; a tolerance per component.
• Verdicts: BLOCKED (interval below the tolerance), REVIEW (drop beyond the tolerance, too few questions), PASS.
• Integration: the gate in the repository's continuous integration; run files and decisions stored with the change; the held-out part reported separately.
• Records: decisions with reasons; tolerance and required scores reviewed per period.

World Bank examples

• The program's continuous integration for cookbook scripts runs the scripts of every cookbook on their examples on each change, the same mechanism applied to a suite.
• The curation cookbook gates prompt, manifest, and model changes on an evaluation set built from curator decisions.
• The agents cookbook keeps scores per version and blocks publication below the previous run.

Resources

• Bootstrap methods (Efron and Tibshirani, 1993): the paired bootstrap the comparison uses.
• GitHub Actions or any continuous integration service: where the gate runs on every change.
• Inspect: evaluation logs and comparisons for suites with many measures.
• Runs and comparison script: the running example.

Common mistakes

• Comparing different question sets. The difference is then between sets; pair by question on one set version.
• A gate on the overall score. The slice that broke is invisible.
• No tolerance. Every change regresses something by one question; the tolerance says what matters.
• Review cases decided and not recorded. The next decision contradicts the last.

Verification

• Pairing. Every comparison is on the same set version, with per-question pairing.
• Verdicts. Verdicts are given per slice with intervals, and BLOCKED and REVIEW produce the stated actions.
• Integration. The gate runs on every change, and the decisions and run files are stored with the change.

Checklist

0 of 6 done

• Paired runs on the same set version
• Differences per slice with bootstrap intervals
• Tolerance per component set from the cost of error
• BLOCKED, REVIEW, and PASS rules applied and recorded
• Gate in continuous integration; held-out part reported separately
• Set grown in the slices that could not decide

Ticks are saved in this browser only.