04 · Tools and structured outputs

Open in Colab View the notebook on GitHub

The outputs on this page are the ones committed with the notebook. Without an API key, the notebook replays recorded model answers, and the same recordings appear here.

Session 01 · Building AI applications for research and data · notebook 4 of 6

Objective

Notebook 03 ended on a limit of search. Even when the right documents come back, the passages may not hold the figures an answer needs: for the main question, the four passages held two of the four numbers. For numbers, the application can skip the search and ask a tool instead, which looks the figure up in the data itself. This notebook gives the model such a tool, watches it use the tool, makes its answer a structured output that other software can read, and then offers the same tool to other programs.

The application so far keeps documents in a store, searches the store, and hands the model the passages it finds (notebook 03). That stays, for definitions and context. This notebook adds a second source of evidence, a tool for exact numbers, and a second kind of answer.

Four terms, in plain words:

A tool is a function that the application can run for the model. The model cannot run it itself: it asks, the application runs it, and the result goes back to the model.
A structured output is an answer returned in a fixed shape, with named fields, instead of free text, so that another program can read it without guessing.
A schema is the description of that shape: the named fields and their types ("year: a whole number") that the answer must fill in.
MCP, the Model Context Protocol, is a standard plug for tools. A tool offered through it can be used by any program that speaks the standard, such as a chatbot or a desktop assistant.

The application after this notebook

The light boxes are what notebooks 01 to 03 built. The dark boxes are what this notebook adds: the tool call and the data it returns, the structured output, and the same tool served over MCP. Passages are for definitions and context. Numbers come from a tool over the data.

Section 1 compares the search of notebook 03 with the tool on the four figures the main question needs.
Section 2 shows the tool as the model sees it: a name, a description, and a form to fill in.
Section 3 runs one exchange: the model asks, the application runs the tool, the result goes back.
Section 4 asks for a year that has no data. The tool returns nothing and the model says so.
Section 5 returns the answer as a structured output, with named fields that a program reads, and shows what happens when a field is damaged.
Section 6 serves the same tool over MCP and calls it from a client and an agent.

How to run it

Run the cells in order. Without an API key, the model's answers are recorded ones, labelled [recorded output: ...], as in notebook 01. With a key in WORKSHOP_API_KEY, the same cells call the model live, the answer streams into its bubble as the model writes, and the wording differs from the recorded answers. The tool results, tables and checks are computed when the cell runs. The last cell reaches a public server and needs the internet.

# Setup. In Colab this clones the repository and installs the dependencies; locally it uses the folder as is.
import os, sys, subprocess, pathlib

REPO_URL = "https://github.com/worldbank/ai4data.git"   # the program's repository
BRANCH = "main"   # the branch the Colab setup clones
SESSION = "training/session-01-building-ai-applications/notebooks"

if not pathlib.Path("workshop.py").exists():
    subprocess.run(["git", "clone", "-q", "--depth", "1", "--branch", BRANCH, REPO_URL, "/content/ai4data"], check=True)
    os.chdir(f"/content/ai4data/{SESSION}")
    subprocess.run([sys.executable, "-m", "pip", "install", "-q", "-r", "requirements.txt"], check=True)
sys.path.insert(0, os.getcwd())

# Optional: a key for an OpenAI-compatible API. Without one, the notebook replays recorded outputs and says so.
try:
    from google.colab import userdata  # type: ignore
    for name in ("WORKSHOP_API_KEY", "WORKSHOP_BASE_URL", "WORKSHOP_MODEL"):
        if userdata.get(name):
            os.environ[name] = userdata.get(name)
except Exception:
    pass

import workshop as w
model = w.make_model()
print("Model:", w.mode(model))

Model: recorded outputs (no WORKSHOP_API_KEY set)

1. Search and tool, for the same four figures

Section 6 of notebook 03 found that the main question needs four numbers: internet use and GDP per capita, for Kenya and Rwanda, in 2024. The cell rebuilds the store of notebook 03 in three lines, runs the same search for the four closest passages, and checks for each number whether it is in them. Beside it, the tool's answer for the same figure.

docs = w.load_documents()
store = w.build_store(w.split_sections(docs))   # the store of notebooks 02 and 03

needed = next(item for item in w.load_questions() if item["id"] == "q1")["expected_numbers"]
passages = store.similarity_search(w.MAIN_QUESTION, k=4)
text = " ".join(p.page_content for p in passages)

countries = {"KEN": "Kenya", "RWA": "Rwanda"}
figures = {"IT.NET.USER.ZS": ("internet use, % of population", "{:.1f}"), "NY.GDP.PCAP.CD": ("GDP per capita, US dollars", "{:,.0f}")}

rows = []
for n in needed:
    name, fmt = figures[n["indicator"]]
    result = w.indicator_lookup(n["indicator"], countries[n["country"]], start=n["year"], end=n["year"])
    rows.append((countries[n["country"]], name, n["year"],
                 "hit" if w.numbers_present(text, [n])[0] == 1 else "miss",
                 fmt.format(w.latest(result)["value"])))
w.show_table(["country", "figure", "year", "in the 4 passages", "from the tool"], rows, source="The search is the one of notebook 03, section 1.")

country	figure	year	in the 4 passages	from the tool
Kenya	internet use, % of population	2024	hit	35.0
Rwanda	internet use, % of population	2024	hit	31.7
Kenya	GDP per capita, US dollars	2024	miss	2,133
Rwanda	GDP per capita, US dollars	2024	miss	1,060

The search is the one of notebook 03, section 1.

The search found two of the four. The tool returned all four, each with its indicator, year and source. A tool does not guess which passage might hold a figure: the application asks for the figure by name.

The model's own memory is a third way to get a number, and the weakest. The cell asks the model for Rwanda's GDP per capita in 2023, then asks the tool.

w.use("nb04_from_memory")
reply = w.answer(model, "What was Rwanda's GDP per capita in 2023?")
print()
r = w.indicator_lookup("NY.GDP.PCAP.CD", "Rwanda", start=2023, end=2023)
w.show_table(["year", "value"], [(o["year"], f"{o['value']:,.1f}") for o in r["observations"]], source=f"{r['name']} · {r['source']}")

assistant · recorded output: nb04_from_memory

Rwanda's GDP per capita in 2023 was roughly US$950–1,000 in current terms. The World Bank's estimate is about $966. Figures vary slightly by source (World Bank, IMF) and get revised, so check the latest data if you need an exact number.

year	value
2023	1,061.1

GDP per capita (current US$) · World Development Indicators (Country official statistics)

The answer from memory gives a range and a figure of about 966 dollars, hedged, and nothing tells a reader how far that is from the record. The tool's answer is exact, 1,061, and carries its source. The application can now get an exact number, with its source, instead of hoping that a passage holds it.

2. The tool, as the model sees it

get_indicator is a Python function decorated with LangChain's @tool. Its docstring becomes the description the model reads, and its type hints become the form the model fills in: one field for each parameter. The cell shows both, and the five indicators the data holds.

print("name:       ", w.get_indicator.name)
print("description:", w.get_indicator.description)
print()

meaning = {"indicator": "which series, as an id such as IT.NET.USER.ZS", "country": "a name or a three-letter code",
           "start": "the first year wanted", "end": "the last year wanted"}
kinds = {"string": "text", "integer": "whole number"}
fields = []
for field, spec in w.get_indicator.args.items():
    kind = spec.get("type") or " or ".join(a["type"] for a in spec["anyOf"] if a["type"] != "null")
    fields.append((field, kinds.get(kind, kind), "optional" if "default" in spec else "required", meaning[field]))
w.show_table(["field", "type", "", "meaning"], fields, source="The form the model fills in, from the tool's type hints.")
print()
w.show_table(["indicator id", "name"], [(i["id"], i["name"]) for i in w.list_indicators()], source="The series the tool can look up.")

name:        get_indicator
description: Observations of a World Development Indicators series for one country. Use it for
    any statistic; do not quote figures from memory. `indicator` is an id such as
    IT.NET.USER.ZS or NY.GDP.PCAP.CD; `country` is a name or ISO3 code; `start` and `end`
    bound the years.

field	type		meaning
indicator	text	required	which series, as an id such as IT.NET.USER.ZS
country	text	required	a name or a three-letter code
start	whole number	optional	the first year wanted
end	whole number	optional	the last year wanted

The form the model fills in, from the tool's type hints.

indicator id	name
NY.GDP.PCAP.CD	GDP per capita (current US$)
IT.NET.USER.ZS	Individuals using the Internet (% of population)
NY.GDP.MKTP.KD.ZG	GDP growth (annual %)
SP.POP.TOTL	Population, total
IT.CEL.SETS.P2	Mobile cellular subscriptions (per 100 people)

The series the tool can look up.

3. The model asks, the application runs the tool

A model does not run tools. It replies with a request that names the tool and fills in the form, the application runs the tool, and the result goes back to the model, which then answers. The animation shows the exchange.

Animation: the application sends the model the question and the list of tools; the model replies with a request for get_indicator with GDP per capita, Rwanda and 2023; the application runs the tool, which returns 1,061 with its source; the application hands that back and the model answers that it was 1,061 US dollars in 2023.

bind_tools tells the model which tools exist. The cell does the exchange once, written out, and labels the three steps. Notebook 05 lets LangGraph run this loop.

import json
from langchain_core.messages import HumanMessage, SystemMessage, ToolMessage

INSTRUCTIONS = """You are a research assistant for development data.
Use get_indicator for every statistic; do not quote figures from memory. Name the indicator and the source with each figure.
If the tool returns no observation for the year asked, say so and give the latest available year."""

model_with_tools = model.bind_tools([w.get_indicator])
messages = [SystemMessage(content=INSTRUCTIONS), HumanMessage(content="What was Rwanda's GDP per capita in 2023?")]

w.use("nb04_tool_call")
print("1. The model asks for the tool:")
reply = w.answer(model_with_tools, messages)
for call in reply.tool_calls:
    result = w.get_indicator.invoke(call["args"])
    print()
    print("2. The application runs it, and the tool returns:")
    w.show_table(["year", "value"], [(o["year"], f"{o['value']:,.1f}") for o in result["observations"]], source=f"{result['name']} · {result['source']}")
    messages += [reply, ToolMessage(content=json.dumps(result), tool_call_id=call["id"])]
print()
print("3. The model writes its answer from the result:")
final = w.answer(model_with_tools, messages)

1. The model asks for the tool:

assistant · recorded output: nb04_tool_call

→ tool call get_indicator({"indicator": "NY.GDP.PCAP.CD", "country": "Rwanda", "start": 2023, "end": 2023})


2. The application runs it, and the tool returns:

year	value
2023	1,061.1

GDP per capita (current US$) · World Development Indicators (Country official statistics)


3. The model writes its answer from the result:

assistant · recorded output: nb04_tool_call

Rwanda's GDP per capita in 2023 was approximately US$1,061.06 (current US$).

Indicator: GDP per capita (current US$), NY.GDP.PCAP.CD
Source: World Development Indicators (World Bank, country official statistics)
Year: 2023

The application can now use a tool: the model asks, the application runs it, and the result goes back to the model.

4. Data that does not exist

The planted problem: a year with no observation. The tool returns an empty list, and the instructions say what to do with it. The model may ask again, for example for a wider range of years, so the cell allows up to three rounds, prints each tool result, and then shows the answer.

messages = [SystemMessage(content=INSTRUCTIONS), HumanMessage(content="What share of Kenya's population used the internet in 2025?")]
w.use("nb04_missing_year")
for round in range(3):   # the model may ask again, for example for a wider range of years
    reply = model_with_tools.invoke(messages)
    if not reply.tool_calls:
        break
    messages.append(reply)
    for call in reply.tool_calls:
        result = w.get_indicator.invoke(call["args"])
        print(f"tool result for {call['args']}: {len(result['observations'])} observations")
        messages.append(ToolMessage(content=json.dumps(result), tool_call_id=call["id"]))
print()
w.show(reply)

tool result for {'indicator': 'IT.NET.USER.ZS', 'country': 'Kenya', 'start': 2025, 'end': 2025}: 0 observations
tool result for {'indicator': 'IT.NET.USER.ZS', 'country': 'Kenya', 'start': 2015, 'end': 2025}: 10 observations

assistant · recorded output: nb04_missing_year

The World Development Indicators database has no value for Kenya's internet use in 2025, so I can't give that figure. The latest available year is 2024: 34.98% of the population were internet users.

Indicator: Individuals using the Internet (% of population) (IT.NET.USER.ZS)
Source: World Development Indicators (World Telecommunication/ICT Indicators Database)

For context, the series rose from 16.59% in 2015 to 34.98% in 2024.

5. Structured output: an answer in a fixed shape

So far the model's answers are sentences, which suits a person. Suppose instead that a dashboard has a box labelled "GDP per capita, Rwanda" and must fill it from the model's answer. The cell does what a simple program might do: it takes the first number in the sentence from section 3.

import re

first = re.findall(r"\d[\d,]*\.?\d*", final.content)[0]
print("The answer:")
w.show(final.content, role="assistant")
print("\nThe dashboard's box: ", first)

The answer:

assistant

Rwanda's GDP per capita in 2023 was approximately US$1,061.06 (current US$).

Indicator: GDP per capita (current US$), NY.GDP.PCAP.CD
Source: World Development Indicators (World Bank, country official statistics)
Year: 2023


The dashboard's box:  2023

The box gets 2023, which is the year and not a figure. The sentence holds a year and a figure, and the words around them change from one answer to the next, so a program reading a sentence has to guess which number is wanted.

The fix is to stop asking for a sentence. We hand the model a blank form with a named box for each thing we need: a country, a year and a value for internet use, a year and a value for GDP per capita, a comparison, the sources. The model's job is to fill in the boxes, and the program reads each box by name. The blank form is called a schema, and an answer that follows one is a structured output. In code the form is a class written with a library called Pydantic, and with_structured_output sends it to the model with the request. The next cell gives the model the tool's data, asks it to fill in the form, and draws the blank form beside the filled one.

data = [w.indicator_lookup(i, c, start=2024) for i in ["IT.NET.USER.ZS", "NY.GDP.PCAP.CD"] for c in ["Kenya", "Rwanda"]]
structured = model.with_structured_output(w.ComparisonAnswer)

w.use("nb04_structured")
parsed = structured.invoke([
    SystemMessage(content="Fill the schema from the data you are given. Use the values exactly as given."),
    HumanMessage(content="DATA:\n" + json.dumps(data) + "\n\nQUESTION: " + w.MAIN_QUESTION),
])
w.show_form(w.ComparisonAnswer, parsed)

The blank form (the schema)

question [ text ]

countries (one block per country)

country [ KEN or RWA ]

internet_latest [ year: whole number value: number ]

gdp_per_capita_latest [ year: whole number value: number ]

trend [ text ]

comparison [ text ]

sources [ list of text ]

caveats [ list of text ]

The same form, filled in by the model

question How have internet access and economic development evolved...

countries (one block per country)

country KEN

internet_latest year: 2024 value: 35.0

gdp_per_capita_latest year: 2024 value: 2,133

trend The provided data contains only the 2024 observation for ...

country RWA

internet_latest year: 2024 value: 31.7

gdp_per_capita_latest year: 2024 value: 1,060

trend The provided data contains only the 2024 observation for ...

comparison In 2024, Kenya had slightly higher internet use, with 34....

sources World Development Indicators (World Telecommunication/ICT...

caveats Only 2024 observations were provided, so trends since 201...

Now the dashboard does not guess. It reads the box it needs by its name, and gets the figure and its year.

box = next(cs for cs in parsed.countries if cs.country == "RWA").gdp_per_capita_latest
print("GDP per capita, Rwanda:", f"{box.value:,.0f} US dollars, in {box.year}")

GDP per capita, Rwanda: 1,060 US dollars, in 2024

A form also has rules, and the program checks the answer against them before using it. A box that should hold a year cannot hold words. The cell damages the answer by putting words in a year box, and shows the message the program would receive, which names the exact box. Without the check, a wrong value would pass through with no error.

from pydantic import ValidationError

damaged = parsed.model_dump_json().replace('"year":2024', '"year":"twenty twenty-four"', 1)
try:
    w.ComparisonAnswer.model_validate_json(damaged)
except ValidationError as e:
    box = " > ".join(str(part) for part in e.errors()[0]["loc"])
    print("box with the problem:", box)
    print("what is wrong:       ", e.errors()[0]["msg"])

box with the problem: countries > 0 > internet_latest > year
what is wrong:        Input should be a valid integer, unable to parse string as an integer

The application can now return an answer that other software reads box by box, and reject one that breaks the form.

6. The same tool over MCP

MCP is the standard plug: a tool written once as a server can be used by any program that speaks the standard, whether a chatbot, a desktop assistant or an agent. indicator_server.py wraps get_indicator in a FastMCP server in a dozen lines. The FastMCP client below starts it, lists its tools and calls one.

from fastmcp import Client

async with Client(w.mcp_server_transport()) as client:
    for t in await client.list_tools():
        print(f"{t.name:16s} {t.description[:90]}")
    result = await client.call_tool("get_indicator", {"indicator": "IT.NET.USER.ZS", "country": "Rwanda", "start": 2022})
    data = result.data
    print()
    w.show_table(["year", "value"], [(o["year"], f"{o['value']:,.1f}") for o in data["observations"]], source=f"{data['name']} · {data['source']}")

list_indicators  The indicators this server holds, with their ids and names.
get_indicator    Observations of a World Development Indicators series for one country. `indicator`
is an i

year	value
2022	28.0
2023	28.8
2024	31.7

Individuals using the Internet (% of population) · World Development Indicators (World Telecommunication/ICT Indicators Database)

A LangChain tool can wrap the MCP call, so an agent uses the server like any other tool. The cell defines one and runs an agent over it. An agent is a model that chooses its own tool calls; notebook 05 is about agents.

from langchain_core.tools import tool
from langchain.agents import create_agent


@tool("get_indicator")
async def get_indicator_over_mcp(indicator: str, country: str, start: int | None = None, end: int | None = None) -> dict:
    """Observations of a World Development Indicators series for one country, from the indicator MCP server."""
    async with Client(w.mcp_server_transport()) as client:
        args = {k: v for k, v in dict(indicator=indicator, country=country, start=start, end=end).items() if v is not None}
        return (await client.call_tool("get_indicator", args)).data


agent = create_agent(model, [get_indicator_over_mcp], system_prompt=INSTRUCTIONS)
w.use("nb04_mcp")
result = await agent.ainvoke({"messages": [HumanMessage(content="How has internet use in Rwanda changed since 2022?")]})
w.print_trace(result)
print()
w.show(w.final_text(result), role="assistant")

step 1: get_indicator({"indicator": "Individuals using the Internet (% of population)", "country": "Rwanda", "start": 2022})
   -> {"error": "unknown indicator 'Individuals using the Internet (% of population)'", "availab…
step 2: get_indicator({"indicator": "IT.NET.USER.ZS", "country": "Rwanda", "start": 2022})
   -> {"indicator": "IT.NET.USER.ZS", "country": "RWA", "name": "Individuals using the Internet …

assistant

Internet use in Rwanda rose steadily from 2022 to 2024, with the largest gain in the latest year.

2022: 28.02% of the population used the internet
2023: 28.81%, up about 0.8 percentage points
2024: 31.70%, up about 2.9 percentage points

Overall, the share rose about 3.7 percentage points, or roughly 13% in relative terms, over two years. Growth was slow in 2023 and faster in 2024.

Source: World Development Indicators (World Telecommunication/ICT Indicators Database), indicator IT.NET.USER.ZS, "Individuals using the Internet (% of population)." 2024 is the latest year available, so I can't report figures for 2025.

The application can now offer its tool to any program that speaks MCP.

The program's Data360 MCP server offers search, metadata and data tools over the World Bank's data platform in the same way, at a public address. The cell lists its tools and runs one search. It reaches a live server, so the list can differ from run to run, and if the call fails the cell says so and the notebook carries on.

try:
    async with Client(w.DATA360_MCP_URL) as client:
        tools = await client.list_tools()
        print(len(tools), "tools, for example:")
        for t in tools[:5]:
            print("  ", t.name)
        found = await client.call_tool("data360_search_indicators", {"query": "individuals using the internet", "limit": 3})
        for ind in found.data["indicators"]:
            print(f"  {ind['idno']:28s} {ind['name']}")
except Exception as e:
    print("Data360 MCP call skipped:", type(e).__name__)

16 tools, for example:
   data360_search_indicators
   data360_search_datasets
   data360_get_metadata
   data360_get_data
   data360_get_disaggregation

  WEF_GCIHH_NETUSERPCT         Individuals using Internet, %
  WB_WDI_IT_NET_USER_ZS        Individuals using the Internet (% of population)
  WEF_TTDI_NETUSERPCT          Individuals using Internet, % of pop

Key points

Numbers come from a tool over authoritative data, and the model's memory is for language. In this notebook the search found two of the four figures the main question needs, and the tool returned all four.
A tool is a function with a description the model can read and a form the model fills in. The model does not run it: the application does, and hands the result back.
Absence of data is a tool result that the application can test for, instead of a gap the model fills with a guess.
A structured output is an answer in a fixed shape: the model fills in named fields instead of writing free text, which a program would have to guess at. The schema is the contract: checked on the way in, readable by other software on the way out.
MCP is the standard plug: the same tool, written once as a server, is available to any program that speaks it.

Exercise

In section 1, change k=4 to k=8 and run the cell. How many of the four figures does the search find now, and what does the extra search cost in passages? Does the tool's column change?
The data holds population too, series SP.POP.TOTL. Run w.indicator_lookup("SP.POP.TOTL", "Rwanda", start=2024, end=2024) and w.indicator_lookup("IT.NET.USER.ZS", "Rwanda", start=2024, end=2024), and work out how many people in Rwanda used the internet in 2024. An answer like that needs two tool results and a calculation: should the model do the arithmetic, or should the application?
Run w.indicator_lookup("IT.NET.USER.ZS", "Kenya", start=2024, end=2025). The tool returns the year that exists and says nothing about the one that does not. Why does the instruction "if the tool returns no observation for the year asked, say so" matter for a result like this?
In section 5, damage a different field of the answer, for example replace a value with the word "high", and read the error. Which field does it name?

Where this connects to the program

Practical Guide to Serving Official Statistics to AI Agents: the design of tools over a statistical catalog, in full.
Practical Guide to Evaluation Suites for Statistical AI: how to build and run the tests.
Proof-Carrying Numbers: the claim-tag protocol the verifier in this session follows. See also the paper and the pcn repository.
Data360 MCP server: the tools of notebook 04, exposed to any MCP client over the World Bank's data platform.
End-to-end reference implementation: documentation, validation, publication, agent access, and verification in one run.

Next: 05 · Workflows and agents

Objective​

The application after this notebook​

Contents​

How to run it​

1. Search and tool, for the same four figures​

2. The tool, as the model sees it​

3. The model asks, the application runs the tool​

4. Data that does not exist​

5. Structured output: an answer in a fixed shape​

6. The same tool over MCP​

Key points​

Exercise​

Where this connects to the program​