5. Trust and provenance of answers

Question: Can answers be trusted and traced?

Rationale

Official statistics carry the authority of the organization that publishes them. When an AI system quotes a number, readers assume it comes from that organization, and they rarely check. A wrong or outdated value, shown with the organization's name, damages trust in the source, and the organization usually learns of it last.

Trust therefore has to be built into the chain that carries a number from the catalog to the reader. The page the number comes from says how to cite it. The API response that carries it says which release it came from. And the answer that quotes it is checked against the values that were retrieved before the reader sees it. Each recipe in this chapter adds one link to that chain, from the cheapest to the most demanding.

Target state

• Every number in an answer links to the series, release, and date it came from.
• Numbers shown to users are checked against the official record before display.
• Answers show the publication date and the latest release, so outdated values are visible.
• When the system is unsure, it says so.

Maturity levels

Level	Steps
Foundational	Show the source, release date, and citation text on every dataset page. Provide a suggested citation in a consistent format. Publish a revisions log.
AI-ready	Return source, identifier, and release date with every API response. Publish citation metadata in machine-readable form. Mark superseded series and point to their replacements.
AI-native	Check each number in a generated answer against the official record and flag those that do not match. Attach provenance to the answer. Log answers with the retrieved records for audit.

Running example

An assistant writes "Undernourishment fell from 8.4% in 2022 to 7.2% in 2024." The retrieved values say 7.6 for 2024. Recipe 5.3 flags the 7.2 before the user sees it.

Recipes

Recipe

5.1 Put source, release, and a citation on every page

Level

Foundational

Skills

Web editor

Indicative time

Half a day for the template

Result: every series page shows where its numbers come from and how to cite them, in the same place and in the same format, so that a reader or a crawler always knows where to look.

The citation block names the organization, the year, the series with its identifier, the release, the page, and the date of access:

National Statistics Office (example). 2025. Prevalence of undernourishment
(FS_UNDERNOURISH_PCT), release of 14 March 2025.
https://stats.example/series/FS_UNDERNOURISH_PCT. Accessed [date].

1. Add a fixed block to the series page template with the source, the release date, the next release date if known, and the citation in the format above.
2. Register a DOI (a persistent identifier) for each dataset through DataCite where the organization can, and put it in the citation. The World Bank Microdata Library assigns DataCite DOIs to its datasets (prefix 10.48529), and the World Bank schemas carry a datacite block for the registration metadata and a citation_requirement field for the citation text.
3. State the licence in the record and on the page. World Bank datasets use Creative Commons Attribution 4.0 by default, which requires attribution and permits reuse.
4. Offer the same citation as a file download (BibTeX or RIS) and in the schema.org record of chapter 1 (dateModified, publisher, identifier, license).
5. Link the revisions log (chapter 3) from the block.
6. When a series is discontinued or replaced, keep the page and add a notice that names the replacement.

Worked example. The block on the undernourishment page shows the source as "Food balance sheets and household budget survey", the release as 14 March 2025, the next release as March 2026, and the citation text shown above. The licence line reads "CC BY 4.0: reuse is permitted with attribution". A BibTeX file sits beside it, and the schema.org record of the page carries the same publisher, identifier, and licence. When the organization replaced its old poverty series in 2022, it kept the old page with the notice "This series was discontinued in 2022 and replaced by PV_HEADCOUNT_NPL_PCT", and citations of the old page still resolve.

What this does not do. A citation block tells people how to cite and cannot make them cite. The monitoring of chapter 7 shows whether the format is used, and the DOI is what makes citations countable through the identifier services.

Recipe

5.2 Return provenance with every API response

Level

AI-ready

Skills

Backend developer

Indicative time

One day

Result: any client, including an AI system, receives enough information with each response to cite it correctly without looking anything up.

The fields to include at the top level of every data response:

Field	Example
SERIES	FS_UNDERNOURISH_PCT
UNIT_MEASURE	% of population
RELEASE	2025-03-14
SOURCE_URL	https://stats.example/series/FS_UNDERNOURISH_PCT
retrieved_at	2026-10-02T09:14:00Z
license	https://creativecommons.org/licenses/by/4.0/

An SDMX web service already returns the structure reference and the attributes with the data; the fields above are the minimum for a custom API.

1. Include the fields above in every data response.
2. Return OBS_STATUS per observation from the SDMX code list (provisional, break, estimated).
3. For a superseded series, return the replacement identifier in the error or the record.
4. Test: fetch any value and write the citation from the response alone.

Worked example. A client requests the observations of the unemployment rate for 2025. The response carries SERIES as LF_UNEMP_PCT, UNIT_MEASURE as "% of labour force", RELEASE as 2025-08-20, SOURCE_URL as https://stats.example/series/LF_UNEMP_PCT, the retrieval time, and the licence URL. The second-quarter observation carries OBS_STATUS P. From that response alone the client writes: "Unemployment rate, 2025 Q2: 6.1% of the labour force, provisional (National Statistics Office, release of 20 August 2025, https://stats.example/series/LF_UNEMP_PCT)". Nothing in the citation had to be guessed.

What this does not do. The provenance fields identify the source of a response. They do not show that the values were quoted unchanged afterwards. Recipe 5.3 checks the numbers in an answer against the response, which is the second half.

Recipe

5.3 Verify the numbers in a generated answer

Level

AI-native

Skills

Python developer

Indicative time

Half a day for the basic check

Result: every number in a generated answer is marked verified or unverified against the values that were retrieved for it, before the user sees it.

The script reads an answer and the values that were retrieved for it, finds every number in the answer, and looks for each among the values, within a tolerance for rounding. Four-digit numbers between 1900 and 2099 are treated as years:

"""Check the numbers in a generated answer against retrieved values.

Takes an answer text and a CSV of the values that were retrieved for it,
with SDMX cross-domain concept names as columns (SERIES, REF_AREA,
TIME_PERIOD, OBS_VALUE, UNIT_MEASURE). Every number in the answer is matched
to a retrieved value within a tolerance and marked verified or unverified.
Uses pandas.

The tolerance is the larger of an absolute and a relative allowance, so that
7.6 reported as 7.60 passes and 12,500 reported as 12,480 passes at 0.2
percent, while 7.6 reported as 7.2 fails.

This is the simplest form of the check described in the Proof-Carrying
Numbers paper (https://arxiv.org/abs/2509.06902). Production systems also
check that the series and period named in the sentence match the value.

Usage:
    python verify_numbers.py answer.txt example_values.csv
    echo "Undernourishment fell from 8.4% in 2022 to 7.2% in 2024." \\
        | python verify_numbers.py - example_values.csv

Exit status is 0 when every number is verified, 1 otherwise.
"""

from __future__ import annotations

import argparse
import math
import re
import sys
from collections.abc import Iterator
from pathlib import Path

import pandas as pd

NUMBER = re.compile(r"-?\d[\d,]*(?:\.\d+)?")
YEAR = re.compile(r"^(19|20)\d\d$")


def find_numbers(text: str) -> Iterator[str]:
    """Yield number strings, skipping unit text such as "per 1,000"."""
    for match in NUMBER.finditer(text):
        if text[max(0, match.start() - 4) : match.start()].lower() == "per ":
            continue
        yield match.group().rstrip(",")


def parse(raw: str) -> float:
    return float(raw.replace(",", ""))


def matches(number: float, value: float, abs_tol: float, rel_tol: float) -> bool:
    """True when the number is within the larger of the absolute and the relative tolerance of the value."""
    return math.isclose(number, value, rel_tol=rel_tol, abs_tol=abs_tol)


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    parser.add_argument("answer", help="text file, or - for stdin")
    parser.add_argument("values", type=Path, help="CSV of retrieved values")
    parser.add_argument(
        "--abs-tolerance",
        type=float,
        default=0.05,
        help="absolute allowance (default 0.05)",
    )
    parser.add_argument(
        "--rel-tolerance",
        type=float,
        default=0.002,
        help="relative allowance (default 0.2%%)",
    )
    args = parser.parse_args(argv)

    text = (
        sys.stdin.read()
        if args.answer == "-"
        else Path(args.answer).read_text(encoding="utf-8")
    )
    values = pd.read_csv(args.values, dtype=str)
    values["OBS_VALUE"] = values["OBS_VALUE"].astype(float)

    unverified = 0
    found = list(find_numbers(text))
    print(f"{'number':>8}  {'status':<10} match")
    for raw in found:
        if YEAR.match(raw):
            print(f"{raw:>8}  {'period':<10}")
            continue
        number = parse(raw)
        hits = values[
            values["OBS_VALUE"].apply(
                lambda v, n=number: matches(
                    n, v, args.abs_tolerance, args.rel_tolerance
                )
            )
        ]
        if hits.empty:
            unverified += 1
            print(f"{raw:>8}  {'UNVERIFIED':<10} no retrieved value within tolerance")
        else:
            m = hits.iloc[0]
            print(
                f"{raw:>8}  {'verified':<10} {m.SERIES} {m.REF_AREA} {m.TIME_PERIOD} = {m.OBS_VALUE:g} {m.UNIT_MEASURE}"
            )

    print(f"\n{len(found)} numbers, {unverified} unverified")
    return 1 if unverified else 0


if __name__ == "__main__":
    sys.exit(main())

Output for the running example:

  number  status     match
     8.4  verified   FS_UNDERNOURISH_PCT NAT 2022 = 8.4 % of population
    2022  period
     7.2  UNVERIFIED no retrieved value within tolerance
    2024  period
4 numbers, 1 unverified

Each line names one number found in the answer. A year is marked as a period and left alone. A value that matches a retrieved observation is marked verified, with the observation it matched. A value that matches nothing is marked unverified, and the last line counts them.

1. Run the check on every answer, with the values retrieved for that answer.
2. Show flagged numbers as unverified, or regenerate the answer with a note that the number was not found. Never show an unverified number as a plain figure.
3. Log the result with the answer. The share of flagged numbers is one of the measures in chapter 6.
4. Extend the check to match the series and period named in the sentence, so that a correct number attached to the wrong year is also caught. The Proof-Carrying Numbers paper describes that fuller approach.

Worked example. The answer in the running example reads "Undernourishment fell from 8.4% in 2022 to 7.2% in 2024". The values retrieved for the question are 8.4, 7.9, and 7.6 for 2022 to 2024. The check finds four numbers. It marks 2022 and 2024 as periods, matches 8.4 to the 2022 observation, and finds no value within tolerance of 7.2, so it marks 7.2 unverified. The interface shows the answer with 7.2 highlighted as unverified, and the log records one unverified number out of two values. A reviewer later finds that the model had misread 7.6 and tightens the instruction to quote values exactly as retrieved.

What this does not do. The script matches each number to any retrieved value within a tolerance. It accepts a correct value quoted for the wrong series, geography, or year, and it cannot check derived numbers such as a change or an average unless the inputs appear in the text. The fuller method binds each number to the claim it supports and checks the claim, which is the part that needs the paper's approach.

Implementation options

• Provenance vocabulary: W3C PROV describes the origin and derivation of data, and DDI-CDI covers provenance across data types. For metadata records harvested between catalogs, the World Bank schemas carry an OAI provenance block. A simpler form is a fixed set of fields: source, identifier, release, retrieval date.
• Citation: DataCite DOIs and a standard citation string make it easy for systems and people to cite the same way.
• Licence: state it in the record, the page, and the API; CC BY 4.0 is the World Bank default.
• Grounding: instruct the model to answer only from retrieved material, and return that material with the answer (chapter 4).
• Numeric verification: extract numbers from the generated text, match them to the retrieved values, and mark each one verified or unverified.
• Freshness: include the date of the latest release in API responses, and have the answer interface mention when it is using an older release.

World Bank examples

• The Microdata Library assigns a DataCite DOI and a citation requirement to each dataset, and World Bank datasets carry the CC BY 4.0 licence by default.
• Proof-Carrying Numbers checks each number in a chatbot answer against the official record and marks it as verified or flagged. The home page shows an example.
• Anomaly Detection and Explanation classifies unusual values in a series and cites the evidence, which supports review before publication.

Resources

• DataCite and the DataCite metadata schema: DOI registration and the citation metadata for datasets.
• Creative Commons BY 4.0 and the World Bank terms of use for datasets: the licence text and a published example of its application.
• W3C PROV and DDI-CDI: provenance vocabularies for exchange as linked data.
• Proof-Carrying Numbers: the method behind recipe 5.3.
• verify_numbers.py: the number check of recipe 5.3; dataset.jsonld: the page markup with source and licence.

Common mistakes

• Citing the organization without the release. A correct number from an old release looks wrong next to the current one. The release date is part of the citation.
• Trusting the model to cite. Models produce plausible citations. The citation must come from the retrieved record, and the number must be checked against it.
• Deleting superseded pages. Citations break and the replacement is not found. Keep the page with a notice.
• Verifying only the final answer. Intermediate values a model computes (a change, an average) need the inputs shown so a reader can check them.

Verification

• Number match rate. On a test set, the share of numbers in answers that match the official values is reported (recipe 5.3).
• Citation check. In a sample of answers, each citation resolves and supports the statement it is attached to.
• Staleness test. A revision published in a test environment appears in the answers that follow it.
• Adversarial prompts. Asked for numbers the organization does not publish, the system declines and invents nothing.

Checklist

0 of 10 done

• Source, release, and citation text on every series page
• DOI or other persistent identifier registered for each dataset
• Licence stated in the record, on the page, and in the API
• Citation available in a machine-readable form
• Source, release date, and license in every API response
• Revisions and superseded series documented and linked
• Numbers in generated answers checked against retrieved values
• Flagged numbers never shown as plain figures
• Answers logged with the records they used
• Declining behavior tested

Ticks are saved in this browser only.