06 · Verification
Open in Colab View the notebook on GitHub
The outputs on this page are the ones committed with the notebook. Without an API key, the notebook replays recorded model answers, and the same recordings appear here.
Session 01 · Building AI applications for research and data · notebook 6 of 6
Objective
Notebooks 01 to 05 put the right evidence in front of the model: context, retrieved passages, exact records from a data tool, and a bounded agent loop. The answer still contains numbers in plain text, and nothing in the application checks them against the records. This notebook adds the check.
Verification follows the Proof-Carrying Numbers protocol (PCN), the program's approach to trustworthy numbers (paper, code). The model writes every figure from the data tool inside a claim tag that names the record the figure came from and the rule for comparing the two. A verifier reads each tag, fetches the record, compares, and marks the number verified or flagged. A number without a tag is unmarked.
Two terms:
- A claim tag is a label the model writes around a number. It names the record the number comes from and how to compare the two. It is the
claim_idof notebook 05 with a comparison rule added. - A verifier reads the tags and checks each number against its record. Here it is
pcn-core, the Python core of the protocol. Each number gets one mark: verified (it matches), flagged (it does not), or unmarked (no tag, so nothing to check against).
The session has used three kinds of check:
| Check | Question it answers | Where |
|---|---|---|
| Retrieval | Did the application find the right evidence? | Notebook 03, section 6 |
| Grounding | Is the answer written from that evidence? | Citations in the answer |
| Verification | Does each number match its source record? | Claim tags and pcn-core, this notebook |
The application after this notebook
Contents
- Section 1 shows a claim tag and the main question answered with every tagged number checked: 14 verified, none flagged, and 21 untagged numbers left unmarked.
- Section 2 replays an answer with one altered figure and one untagged figure, and shows the flag and the absence of a mark.
- Section 3 states what the check covers and what it leaves to the reader.
How to run it
Run the cells in order. Without an API key, the model's answers are recorded ones, labelled [recorded output: ...], as in notebook 01. With a key in WORKSHOP_API_KEY, the same cells call the model live, and the answers will differ in wording from the recorded ones. The checks and the tables are computed when a cell runs, in both modes. The exercises at the end run without a key.
The next cell installs the session's tools and loads its helper module. The first run takes about a minute. The cell is setup only.
# Setup. In Colab this clones the repository and installs the dependencies; locally it uses the folder as is.
import os, sys, subprocess, pathlib
REPO_URL = "https://github.com/worldbank/ai4data.git" # the program's repository
BRANCH = "main" # the branch the Colab setup clones
SESSION = "training/session-01-building-ai-applications/notebooks"
if not pathlib.Path("workshop.py").exists():
subprocess.run(["git", "clone", "-q", "--depth", "1", "--branch", BRANCH, REPO_URL, "/content/ai4data"], check=True)
os.chdir(f"/content/ai4data/{SESSION}")
subprocess.run([sys.executable, "-m", "pip", "install", "-q", "-r", "requirements.txt"], check=True)
sys.path.insert(0, os.getcwd())
# Optional: a key for an OpenAI-compatible API. Without one, the notebook replays recorded outputs and says so.
try:
from google.colab import userdata # type: ignore
for name in ("WORKSHOP_API_KEY", "WORKSHOP_BASE_URL", "WORKSHOP_MODEL"):
if userdata.get(name):
os.environ[name] = userdata.get(name)
except Exception:
pass
import workshop as w
model = w.make_model()
print("Model:", w.mode(model))
Model: recorded outputs (no WORKSHOP_API_KEY set)
The helper functions used in this notebook, in the order they appear. w.use and w.show are as in notebook 01.
| Call | What it does |
|---|---|
w.indicator_lookup(indicator, country, start=...) | The function behind the get_indicator tool of notebook 04; here its results are placed in the prompt directly. |
w.observations_from_results(results) | A dictionary from claim id to the value the tool returned. |
w.show_chat(question, answer, observations, results, raw) | The answer as a chatbot draws it, with a mark after each checked number. |
w.verify(answer, observations) | The verifier: one check per number in the answer, with its status. |
w.show_checks(checks) | The checks as a table. |
The next cell writes the instructions. The new rule is the claim tag. Its id is indicator:country:year, the key under which the tool returned the value. policy="rounded" with decimals="1" means that the record, rounded to one decimal, must equal the number shown; policy="exact" requires equality.
import json
from langchain_core.messages import HumanMessage, SystemMessage
INSTRUCTIONS_CLAIMS = """You are a research assistant for development data.
Use get_indicator for every statistic and search_documents for definitions and context. Do not quote figures from memory.
Write every number that comes from get_indicator inside a claim tag that names its claim_id, like
<claim id="IT.NET.USER.ZS:KEN:2024" policy="rounded" decimals="1">35.0</claim>.
Name the indicator and the source. If the tools do not provide what is needed, say so.
If the question is ambiguous, ask which reading is meant."""
1. Claim tags
A claim tag is a label around a number. The verifier takes the record named by the tag's id, applies the tag's rule, and compares the result with the number shown. The animation shows the three outcomes: a tagged figure that matches its record is verified, a tagged figure that differs is flagged, and a figure without a tag stays unmarked.
The next cell gives the model the four series the main question needs, placed in the prompt as notebook 04 did, and asks it to answer with tags. w.observations_from_results builds the dictionary from claim id to the value the tool returned, which is what the verifier compares against. w.show_chat draws the answer as a chatbot would: a mark after each checked number, a popover on hover or focus with the record and the rule, and, with raw=True, a fold that holds the tagged text as the model wrote it.
results = [w.indicator_lookup(i, c, start=2010) for i in ["IT.NET.USER.ZS", "NY.GDP.PCAP.CD"] for c in ["Kenya", "Rwanda"]]
observations = w.observations_from_results(results) # claim_id -> the value the tool returned
w.use("nb06_claims_main")
reply = model.invoke([SystemMessage(content=INSTRUCTIONS_CLAIMS), HumanMessage(content="DATA:\n" + json.dumps(results) + "\n\nQUESTION: " + w.MAIN_QUESTION)])
w.show_chat(w.MAIN_QUESTION, reply.content, observations, results, raw=True)
Both countries show strong gains in internet use since 2010, with Kenya's rising faster. Both also show GDP per capita roughly doubling in current US dollars, though with different timing. The figures below are the latest years in the supplied data (2024) and the 2010 starting point.
Internet use (% of population) Source: World Development Indicators, World Telecommunication/ICT Indicators Database
| Year | Kenya | Rwanda |
|---|---|---|
| 2010 | 7.2✓✓ Verified: the two are equalRecordIT.NET.USER.ZS:KEN:2010IndicatorIndividuals using the Internet (% of population), KEN, 2010SourceWorld Development Indicators (World Telecommunication/ICT Indicators Database)Returned by the tool7.2Display rulerounded to 1 decimal, which gives 7.2Shown in the answer7.2 | 8.0✓✓ Verified: the two are equalRecordIT.NET.USER.ZS:RWA:2010IndicatorIndividuals using the Internet (% of population), RWA, 2010SourceWorld Development Indicators (World Telecommunication/ICT Indicators Database)Returned by the tool8Display rulerounded to 1 decimal, which gives 8.0Shown in the answer8.0 |
| 2015 | 16.6✓✓ Verified: the two are equalRecordIT.NET.USER.ZS:KEN:2015IndicatorIndividuals using the Internet (% of population), KEN, 2015SourceWorld Development Indicators (World Telecommunication/ICT Indicators Database)Returned by the tool16.5879Display rulerounded to 1 decimal, which gives 16.6Shown in the answer16.6 | 12.5✓✓ Verified: the two are equalRecordIT.NET.USER.ZS:RWA:2015IndicatorIndividuals using the Internet (% of population), RWA, 2015SourceWorld Development Indicators (World Telecommunication/ICT Indicators Database)Returned by the tool12.456Display rulerounded to 1 decimal, which gives 12.5Shown in the answer12.5 |
| 2020 | 24.7✓✓ Verified: the two are equalRecordIT.NET.USER.ZS:KEN:2020IndicatorIndividuals using the Internet (% of population), KEN, 2020SourceWorld Development Indicators (World Telecommunication/ICT Indicators Database)Returned by the tool24.7228Display rulerounded to 1 decimal, which gives 24.7Shown in the answer24.7 | 23.8✓✓ Verified: the two are equalRecordIT.NET.USER.ZS:RWA:2020IndicatorIndividuals using the Internet (% of population), RWA, 2020SourceWorld Development Indicators (World Telecommunication/ICT Indicators Database)Returned by the tool23.8Display rulerounded to 1 decimal, which gives 23.8Shown in the answer23.8 |
| 2024 | 35.0✓✓ Verified: the two are equalRecordIT.NET.USER.ZS:KEN:2024IndicatorIndividuals using the Internet (% of population), KEN, 2024SourceWorld Development Indicators (World Telecommunication/ICT Indicators Database)Returned by the tool34.9762Display rulerounded to 1 decimal, which gives 35.0Shown in the answer35.0 | 31.7✓✓ Verified: the two are equalRecordIT.NET.USER.ZS:RWA:2024IndicatorIndividuals using the Internet (% of population), RWA, 2024SourceWorld Development Indicators (World Telecommunication/ICT Indicators Database)Returned by the tool31.7033Display rulerounded to 1 decimal, which gives 31.7Shown in the answer31.7 |
GDP per capita (current US$) Source: World Development Indicators, Country official statistics
| Year | Kenya | Rwanda |
|---|---|---|
| 2010 | 1092✓✓ Verified: the two are equalRecordNY.GDP.PCAP.CD:KEN:2010IndicatorGDP per capita (current US$), KEN, 2010SourceWorld Development Indicators (Country official statistics)Returned by the tool1091.52Display rulerounded to 0 decimals, which gives 1,092Shown in the answer1092 | 587✓✓ Verified: the two are equalRecordNY.GDP.PCAP.CD:RWA:2010IndicatorGDP per capita (current US$), RWA, 2010SourceWorld Development Indicators (Country official statistics)Returned by the tool586.6Display rulerounded to 0 decimals, which gives 587Shown in the answer587 |
| 2020 | 1928✓✓ Verified: the two are equalRecordNY.GDP.PCAP.CD:KEN:2020IndicatorGDP per capita (current US$), KEN, 2020SourceWorld Development Indicators (Country official statistics)Returned by the tool1927.66Display rulerounded to 0 decimals, which gives 1,928Shown in the answer1928 | 803✓✓ Verified: the two are equalRecordNY.GDP.PCAP.CD:RWA:2020IndicatorGDP per capita (current US$), RWA, 2020SourceWorld Development Indicators (Country official statistics)Returned by the tool802.639Display rulerounded to 0 decimals, which gives 803Shown in the answer803 |
| 2024 | 2133✓✓ Verified: the two are equalRecordNY.GDP.PCAP.CD:KEN:2024IndicatorGDP per capita (current US$), KEN, 2024SourceWorld Development Indicators (Country official statistics)Returned by the tool2133.46Display rulerounded to 0 decimals, which gives 2,133Shown in the answer2133 | 1060✓✓ Verified: the two are equalRecordNY.GDP.PCAP.CD:RWA:2024IndicatorGDP per capita (current US$), RWA, 2024SourceWorld Development Indicators (Country official statistics)Returned by the tool1059.94Display rulerounded to 0 decimals, which gives 1,060Shown in the answer1060 |
Main patterns
-
Internet use rose sharply in both countries. Kenya went from 7.2% to 35.0% of the population, a gain of about 27.8 percentage points. Rwanda went from 8.0% to 31.7%, a gain of about 23.7 points. Kenya's growth was steady, with its largest annual gains in 2012–2014 and 2018–2020. Rwanda's series is less smooth: it fell from 8.0% in 2010 to 7.0% in 2011 and was still around 12.5% in 2015. Its fastest growth came after 2015. Kenya was ahead from about 2013 onward, and the gap widened in recent years. The 2010–2011 Rwanda dip may reflect survey or revision effects, which I have not verified.
-
GDP per capita roughly doubled in both countries, but from very different levels. Kenya rose from about $1,092 to about $2,133 (roughly 1.95 times). Rwanda rose from about $587 to about $1,060 (roughly 1.8 times). Kenya's level is about double Rwanda's in 2024, similar to the gap in 2010.
-
Economic growth has been uneven. Kenya's GDP per capita dipped in 2020 and again in 2023 (to about $1,943), and then recovered in 2024. Rwanda's rose steadily until 2021, then jumped from about $843 in 2021 to about $1,007 in 2022, and continued to about $1,061 in 2023 before easing slightly in 2024.
-
Internet growth has not tracked income growth in a simple way. Both countries saw internet adoption climb through periods where GDP per capita was flat or falling. Rwanda's 2022 income jump came alongside a large internet increase (from about 24.4% in 2021 to 28.0% in 2022), but the data cannot show causation.
Caveats
- I did not run search_documents, so I have no definitions or methodology notes from the document store. The indicator names and sources come from the supplied data block.
- The GDP series is in current US dollars, so exchange-rate movements can affect year-to-year changes. I have not verified this against the documents.
- The percentage-point gains and multiples above are my own arithmetic on the supplied values, not figures reported directly by the source.
- The data covers 2010–2024 only, and the question did not specify a comparison period. If you meant a different window, or wanted a per-capita comparison with other countries, let me know.
What the model actually wrote (open to see the labels)
The model put its figures in two tables, one tag per cell, and wrote four patterns under them. Every tagged number carries a ✓. The numbers in the patterns carry no mark: they are the same figures repeated without tags, or values the model computed, such as a gain of 27.8 points or a ratio of 1.95. The next cell lists every check as a table.
w.show_checks(w.verify(reply.content, observations))
| number | claim_id | the record holds | mark |
|---|---|---|---|
| 7.2 | IT.NET.USER.ZS:KEN:2010 | 7.2 | verified |
| 8.0 | IT.NET.USER.ZS:RWA:2010 | 8 | verified |
| 16.6 | IT.NET.USER.ZS:KEN:2015 | 16.5879 | verified |
| 12.5 | IT.NET.USER.ZS:RWA:2015 | 12.456 | verified |
| 24.7 | IT.NET.USER.ZS:KEN:2020 | 24.7228 | verified |
| 23.8 | IT.NET.USER.ZS:RWA:2020 | 23.8 | verified |
| 35.0 | IT.NET.USER.ZS:KEN:2024 | 34.9762 | verified |
| 31.7 | IT.NET.USER.ZS:RWA:2024 | 31.7033 | verified |
| 1092 | NY.GDP.PCAP.CD:KEN:2010 | 1091.52 | verified |
| 587 | NY.GDP.PCAP.CD:RWA:2010 | 586.6 | verified |
| 1928 | NY.GDP.PCAP.CD:KEN:2020 | 1927.66 | verified |
| 803 | NY.GDP.PCAP.CD:RWA:2020 | 802.639 | verified |
| 2133 | NY.GDP.PCAP.CD:KEN:2024 | 2133.46 | verified |
| 1060 | NY.GDP.PCAP.CD:RWA:2024 | 1059.94 | verified |
| 7.2 | no tag | none | unmarked |
| 35.0 | no tag | none | unmarked |
| 27.8 | no tag | none | unmarked |
| 8.0 | no tag | none | unmarked |
| 31.7 | no tag | none | unmarked |
| 23.7 | no tag | none | unmarked |
| 8.0 | no tag | none | unmarked |
| 7.0 | no tag | none | unmarked |
| 12.5 | no tag | none | unmarked |
| 1,092 | no tag | none | unmarked |
| 2,133 | no tag | none | unmarked |
| 1.95 | no tag | none | unmarked |
| 587 | no tag | none | unmarked |
| 1,060 | no tag | none | unmarked |
| 1.8 | no tag | none | unmarked |
| 1,943 | no tag | none | unmarked |
| 843 | no tag | none | unmarked |
| 1,007 | no tag | none | unmarked |
| 1,061 | no tag | none | unmarked |
| 24.4 | no tag | none | unmarked |
| 28.0 | no tag | none | unmarked |
14 verified, 0 flagged, 21 unmarked
Fourteen numbers are verified, none is flagged, and 21 are unmarked. The record for Kenya's 2024 internet use holds 34.9762; rounded to one decimal it is 35.0, so the comparison holds. The 21 unmarked numbers are the repeats and the computed values in the patterns. A computed value has no single record to check against, so the protocol leaves it unmarked, and the reader sees that it was not checked. A check by value alone, without tags, could not tell a source from a coincidence: 28, a difference the model computed, equals Rwanda's 2022 internet share of 28.0. Years are not figures and are skipped.
The chat view imitates ClaimMark, the component of the program's @pcn-js/ui package that draws the same marks in a web chatbot (worldbank/pcn). The marks here come from the Python verifier.
2. A flagged number and an unmarked number
A flag means that the number does not match the record named in its tag. An unmarked number carries no tag, so there is nothing to compare, and no mark is drawn whether or not the number is right. The next cell replays a hand-made answer with one of each: Rwanda's internet use altered to 35.7 against the record's 31.7, and a figure of about 70 percent for mobile money that did not come from the tool.
w.use("nb06_claims_error")
flawed = model.invoke([SystemMessage(content=INSTRUCTIONS_CLAIMS), HumanMessage(content="DATA:\n" + json.dumps(results) + "\n\nQUESTION: " + w.MAIN_QUESTION)])
w.show_chat(w.MAIN_QUESTION, flawed.content, observations, results)
w.show_checks(w.verify(flawed.content, observations))
Internet use reached 35.0✓✓ Verified: the two are equalRecordIT.NET.USER.ZS:KEN:2024IndicatorIndividuals using the Internet (% of population), KEN, 2024SourceWorld Development Indicators (World Telecommunication/ICT Indicators Database)Returned by the tool34.9762Display rulerounded to 1 decimal, which gives 35.0Shown in the answer35.0 percent of the population in Kenya and 35.7⚠⚠ Not verified: the two differRecordIT.NET.USER.ZS:RWA:2024IndicatorIndividuals using the Internet (% of population), RWA, 2024SourceWorld Development Indicators (World Telecommunication/ICT Indicators Database)Returned by the tool31.7033Display rulerounded to 1 decimal, which gives 31.7Shown in the answer35.7 percent in Rwanda in 2024 (World Development Indicators, IT.NET.USER.ZS). GDP per capita in current US dollars was 2,133✓✓ Verified: the two are equalRecordNY.GDP.PCAP.CD:KEN:2024IndicatorGDP per capita (current US$), KEN, 2024SourceWorld Development Indicators (Country official statistics)Returned by the tool2133.46Display rulerounded to 0 decimals, which gives 2,133Shown in the answer2,133 in Kenya and 1,060✓✓ Verified: the two are equalRecordNY.GDP.PCAP.CD:RWA:2024IndicatorGDP per capita (current US$), RWA, 2024SourceWorld Development Indicators (Country official statistics)Returned by the tool1059.94Display rulerounded to 0 decimals, which gives 1,060Shown in the answer1,060 in Rwanda. About 70 percent of adults in both countries use mobile money, which runs on basic phones and does not require internet access.
| number | claim_id | the record holds | mark |
|---|---|---|---|
| 35.0 | IT.NET.USER.ZS:KEN:2024 | 34.9762 | verified |
| 35.7 | IT.NET.USER.ZS:RWA:2024 | 31.7033; shown 35.7 | flagged |
| 2,133 | NY.GDP.PCAP.CD:KEN:2024 | 2133.46 | verified |
| 1,060 | NY.GDP.PCAP.CD:RWA:2024 | 1059.94 | verified |
| 70 | no tag | none | unmarked |
3 verified, 1 flagged, 1 unmarked
The verifier flags 35.7 with a ⚠, and the popover shows the record, the value returned, and the failed comparison. The 70 carries no mark. The absence is deliberate. The protocol states the principle as: trust is earned only by proof, and the absence of a verification mark communicates uncertainty.
3. What the check covers
The verifier answers one question about each number: does it match the record it names? It says nothing about whether that record is the right one for the question, whether the method behind the record is sound, or whether the sentence around the number interprets it correctly. Those judgements remain with the reader, and the marks tell the reader which numbers need none of that scrutiny on the first question. A test of the application across many questions, with a fixed set of expected answers, is the subject of the program's evaluation cookbook; notebook 03, section 6, showed the retrieval half of such a test.
Key points
- A claim tag names the record a number comes from and the rule for comparing them. The verifier compares the two and marks the number verified or flagged. A number without a tag is unmarked.
- An unmarked number is neither right nor wrong; it is unchecked, and the reader is told so.
- Verification checks a number against its record. Whether the record answers the question, and whether the sentence reads it correctly, stay with the reader.
Exercise
All of these run without a key, on the answer of section 1 (reply).
- Change one number inside a tag:
tampered = reply.content.replace(">35.0<", ">35.7<"), thenw.show_checks(w.verify(tampered, observations)). Which mark changes, and what does the record hold? - Remove all the tags:
plain = w.strip_claims(reply.content), thenw.show_checks(w.verify(plain, observations)). What happens to every number, and why is the result not a "verified" one? - Change the policy of one tag from rounded to exact:
strict = reply.content.replace('policy="rounded" decimals="1"', 'policy="exact"', 1), then verify it. Why does 35.0 stop matching a record of 34.9762, and which policy suits a published percentage?
Where this connects to the program
- Practical Guide to Serving Official Statistics to AI Agents: the design of tools over a statistical catalog, in full.
- Practical Guide to Evaluation Suites for Statistical AI: how to build and run the tests.
- Proof-Carrying Numbers: the claim-tag protocol the verifier in this session follows. See also the paper and the
pcnrepository. - Data360 MCP server: the tools of notebook 04, exposed to any MCP client over the World Bank's data platform.
- End-to-end reference implementation: documentation, validation, publication, agent access, and verification in one run.