Skip to main content

05 · Workflows and agents

Open in Colab View the notebook on GitHub

The outputs on this page are the ones committed with the notebook. Without an API key, the notebook replays recorded model answers, and the same recordings appear here.

Session 01 · Building AI applications for research and data · notebook 5 of 6

Objective​

Notebook 04 gave the model one tool and one question at a time. The main question of this session needs more than that: figures for two countries, which come from the tool, and context from the documents, which comes from the search of notebook 03. Someone has to decide which calls to make, and in what order. This notebook shows two ways of deciding, runs both on the main question, and then looks at what an answer built from several calls still lacks.

The application so far can search a store of documents (notebook 03) and call a tool for exact numbers (notebook 04). Here the search becomes a second tool, next to the first, and the application gains the means to coordinate them.

Four terms, in plain words:

  • A workflow is a fixed order of steps written in code: search, then look up, then write. Every run makes the same calls.
  • An agent is a model that chooses its own next step. It asks for a tool, sees the result, and decides what to do next, until it can answer. The agent here is built with LangGraph, a library for loops of this kind.
  • A trace is the list of every call the agent made and what came back.
  • A step limit (recursion_limit in LangGraph) is the number of steps after which a run is stopped, so that a loop that never ends cannot run for ever. A step is one turn of the loop: the model's turn, or the tools' turn.

The application after this notebook​

Diagram: the question goes to a model that chooses the next step. It can call search_documents, the search of notebook 03 now offered as a tool, which returns passages, or get_indicator, the tool of notebook 04, which returns data. The results go back to the model, which writes the answer. New in this notebook: the model chooses the steps, a step limit and a trace bound and record the loop, and the answer ends on the question of whether its numbers can be trusted, which notebook 06 takes up.

Contents​

  1. Section 1 answers the main question with a workflow: the steps are fixed in code.
  2. Section 2 answers it with an agent that chooses its own calls, and shows the trace.
  3. Section 3 asks a question the tools cannot answer, and shows what the agent does.
  4. Section 4 makes a call fail, and shows the agent recover from the error.
  5. Section 5 runs a loop that never ends, and stops it with the step limit.
  6. Section 6 compares the two designs and when each fits.
  7. Section 7 asks whether the numbers in the answer can be trusted, the question for notebook 06.

How to run it​

Run the cells in order. Without an API key, the model's answers are recorded ones, labelled [recorded output: ...], as in notebook 01. With a key in WORKSHOP_API_KEY, the same cells call the model live, the answer streams into its bubble as the model writes, and the wording differs from the recorded answers. The searches, the tool results, the tables and the checks are computed when the cell runs. The exercises at the end run without a key.

# Setup. In Colab this clones the repository and installs the dependencies; locally it uses the folder as is.
import os, sys, subprocess, pathlib

REPO_URL = "https://github.com/worldbank/ai4data.git"   # the program's repository
BRANCH = "main"   # the branch the Colab setup clones
SESSION = "training/session-01-building-ai-applications/notebooks"

if not pathlib.Path("workshop.py").exists():
    subprocess.run(["git", "clone", "-q", "--depth", "1", "--branch", BRANCH, REPO_URL, "/content/ai4data"], check=True)
    os.chdir(f"/content/ai4data/{SESSION}")
    subprocess.run([sys.executable, "-m", "pip", "install", "-q", "-r", "requirements.txt"], check=True)
sys.path.insert(0, os.getcwd())

# Optional: a key for an OpenAI-compatible API. Without one, the notebook replays recorded outputs and says so.
try:
    from google.colab import userdata  # type: ignore
    for name in ("WORKSHOP_API_KEY", "WORKSHOP_BASE_URL", "WORKSHOP_MODEL"):
        if userdata.get(name):
            os.environ[name] = userdata.get(name)
except Exception:
    pass

import workshop as w
model = w.make_model()
print("Model:", w.mode(model))
Model: recorded outputs (no WORKSHOP_API_KEY set)
import json
from langchain.agents import create_agent
from langchain_core.messages import HumanMessage
from langchain_core.prompts import ChatPromptTemplate

docs = w.load_documents()
store = w.build_store(w.split_sections(docs))   # the store of notebooks 02 and 03
search_documents = w.make_search_tool(store, k=3, since="2020")   # the search of notebook 03, offered as a tool
tools = [search_documents, w.get_indicator]   # the tool of notebook 04 is the second one

INSTRUCTIONS = """You are a research assistant for development data.
Use get_indicator for every statistic and search_documents for definitions, context, and explanations.
Name the indicator and the source with each figure; cite passages in the square-bracket form given.
If the tools do not provide what is needed, say so. Do not quote figures from memory."""

1. A workflow: the order is written in code​

The picture contrasts the two designs. In a workflow, the steps are written in code in advance: search the documents, look up the four series, then ask the model to write. The model is called once, at the end, and every run makes the same calls. The cell runs the workflow on the main question and prints each step as it happens.

Diagram: in a workflow the order of the calls is written in code: search the documents, look up the four series, the model writes the answer, the same on every run. In an agent the model decides the next step: it asks for search_documents or get_indicator, gets the result back, and answers in text when it has enough; a step limit stops a loop that never ends and a trace lists every call.

compose = ChatPromptTemplate.from_messages([("system", INSTRUCTIONS), ("human", "PASSAGES:\n{passages}\n\nDATA:\n{data}\n\nQUESTION: {question}")]) | model


def fixed_workflow(question):
    passages = search_documents.invoke({"query": question})
    print(f"step 1: searched the documents, found {len(passages)} passages")
    series = [w.indicator_lookup(i, c, start=2010) for i in ["IT.NET.USER.ZS", "NY.GDP.PCAP.CD"] for c in ["Kenya", "Rwanda"]]
    print(f"step 2: looked up {len(series)} series: internet use and GDP per capita, for Kenya and Rwanda")
    answer = w.answer(compose, {
        "passages": "\n\n".join(f"{p['citation']}\n{p['text']}" for p in passages),
        "data": json.dumps(series),
        "question": question,
    })
    print("step 3: the model wrote the answer\n")
    return answer, series


w.use("nb05_compose")
workflow_answer, workflow_series = fixed_workflow(w.MAIN_QUESTION)
step 1: searched the documents, found 3 passages
step 2: looked up 4 series: internet use and GDP per capita, for Kenya and Rwanda
assistant · recorded output: nb05_compose

Internet access and GDP per capita in Rwanda and Kenya, 2010 to 2024

The figures below come only from the data and passages provided. I did not have live access to the get_indicator or search_documents tools, so I have not added any figures from memory.

Internet access

Indicator: Individuals using the Internet (% of population), IT.NET.USER.ZS Source: World Development Indicators (World Telecommunication/ICT Indicators Database)

YearRwanda (%)Kenya (%)
20108.07.2
201512.5 (12.456)16.6 (16.5879)
202023.824.7 (24.7228)
202228.0 (28.0168)29.4 (29.4059)
202328.8 (28.8058)32.1 (32.0704)
202431.7 (31.7033)35.0 (34.9762)

Sources for the table: [rwanda-digital · Internet use · 2026] and [kenya-digital · Internet use · 2026], with values matching the IT.NET.USER.ZS observations (claim_ids such as IT.NET.USER.ZS:RWA:2024 and IT.NET.USER.ZS:KEN:2024).

  • Rwanda: Internet use rose by about 23.7 percentage points between 2010 and 2024. Growth was fastest from 2015 to 2020, when the share nearly doubled. It slowed between 2020 and 2023 and picked up again in 2024 [rwanda-digital · Internet use · 2026].
  • Kenya: Internet use rose by about 27.8 percentage points over the same period. The fastest absolute gain came between 2010 and 2015, when the share more than doubled. Growth continued after 2020 [kenya-digital · Internet use · 2026].
  • Comparison: Rwanda started slightly higher in 2010 (8.0% vs 7.2%), but Kenya moved ahead by 2011 (8.8% vs 7.0%) and has stayed ahead since. The gap in 2024 is about 3.3 percentage points (35.0% vs 31.7%). Between 2020 and 2024, Kenya gained about 10.3 points and Rwanda about 7.9 points.

Economic development (GDP per capita, current US$)

Indicator: GDP per capita (current US$), NY.GDP.PCAP.CD Source: World Development Indicators (Country official statistics)

YearRwanda (US$)Kenya (US$)
2010586.61,091.5
2020802.61,927.7
20221,007.12,109.6
20231,061.11,942.6
20241,059.92,133.5
  • Rwanda: GDP per capita rose about 81% from 2010 to 2024. The largest single-year rise was in 2022 (about 19.5%, from 842.6 to 1,007.1). It was flat in 2023 and 2024.
  • Kenya: GDP per capita rose about 95% from 2010 to 2024. It fell in 2020 and again in 2023 (about 7.9%, from 2,109.6 to 1,942.6) before recovering in 2024.
  • Comparison: Kenya's GDP per capita was about double Rwanda's in 2024 (2,133.5 vs 1,059.9), and roughly 1.9 times in 2010.

Main patterns

  1. Both countries more than tripled their internet-use shares over 2010 to 2024 (Rwanda from 8.0% to 31.7%, Kenya from 7.2% to 35.0%), with Kenya ahead since 2011.
  2. Rwanda's internet growth was strongest in 2015 to 2020, and Kenya's in 2010 to 2015.
  3. GDP per capita grew in both countries over the period, but Kenya's level is about double Rwanda's. Rwanda's series is flat after 2022, and Kenya's dipped in 2020 and 2023.
  4. The two series move in the same general direction, but the data do not show that internet access caused the changes in GDP per capita, or the reverse.

Limits of the evidence

  • The provided materials do not explain the drivers of the GDP per capita swings, such as the 2022 jump in Rwanda or the 2023 dip in Kenya. The GDP figures are in current US dollars, so exchange-rate movements may contribute, but the provided materials do not say so.
  • The mobile-money note contains no figures. It explains that mobile money can be used without internet access, so mobile money adoption should not be read from the internet-use series.
  • The provided materials include no other development measures, such as poverty, education, or sectoral data, so "economic development" here is limited to GDP per capita.
step 3: the model wrote the answer

The application can now answer the main question with a fixed set of steps. It is predictable: the same question makes the same calls, so it is easy to test.

2. An agent: the model chooses the steps​

In an agent, the model is offered the tools and their descriptions, and chooses. create_agent builds the loop with LangGraph: the model replies with the calls it wants, the loop runs them, the results go back to the model, and the loop stops when the model answers in text. The cell asks the same question, then shows the trace as a table, one row per call, grouped by round.

agent = create_agent(model, tools, system_prompt=INSTRUCTIONS)

w.use("nb05_agent_main")
result = agent.invoke({"messages": [HumanMessage(content=w.MAIN_QUESTION)]}, config={"recursion_limit": 10})
w.show_trace(result)
print()
w.show(w.final_text(result), role="assistant")
roundcalltoolwhat was askedwhat came back
11get_indicatorindicator=IT.NET.USER.ZS, country=Rwanda15 observations, 2010 to 2024, latest 31.7
12get_indicatorindicator=IT.NET.USER.ZS, country=Kenya15 observations, 2010 to 2024, latest 35.0
13get_indicatorindicator=NY.GDP.PCAP.CD, country=Rwanda15 observations, 2010 to 2024, latest 1,059.9
14get_indicatorindicator=NY.GDP.PCAP.CD, country=Kenya15 observations, 2010 to 2024, latest 2,133.5
15search_documentsquery=internet access and economic development in Rwanda and Kenya3 passages, first [rwanda-digital · Internet use · 2026]
assistant

Internet access and GDP per capita: Rwanda and Kenya

Internet use (% of population) Source: World Development Indicators, World Telecommunication/ICT Indicators Database (IT.NET.USER.ZS)

YearRwandaKenya
20108.07.2
201512.516.6
202023.824.7
202431.735.0

GDP per capita (current US$) Source: World Development Indicators, Country official statistics (NY.GDP.PCAP.CD)

YearRwandaKenya
2010586.61,091.5
2020802.61,927.7
20241,059.92,133.5

The series for both countries are complete for 2010–2024. The 2024 values are the latest available.

Main patterns

  1. Internet use rose in both countries, and Kenya stayed ahead for most of the period. Rwanda's share rose about 24 percentage points from 2010 to 2024, and Kenya's about 28 points. Rwanda was slightly ahead in 2010 (8.0 vs 7.2), but Kenya has been higher in every year since 2011. The two countries were nearly level in 2020 (23.8 vs 24.7).

  2. The pace of growth differed by period. Kenya's fastest gains came early, between 2010 and 2015, when its share more than doubled. Rwanda's fastest period was 2015 to 2020, when its share nearly doubled. The Rwanda digital note describes growth slowing between 2020 and 2023 and picking up again in 2024 [rwanda-digital · Internet use · 2026]. Kenya's growth continued after 2020, with the share reaching 29.4% in 2022, 32.1% in 2023, and 35.0% in 2024 [kenya-digital · Internet use · 2026].

  3. Kenya has a much higher income level, and both countries grew. GDP per capita roughly doubled in Kenya (about 1.96 times its 2010 value) and rose about 1.8 times in Rwanda. Kenya's income level was about 1.9 times Rwanda's in 2010 and about 2.0 times in 2024, so the gap in levels has slightly widened. Kenya's series dipped in 2020 and again in 2023 (1,942.6, down from 2,109.6 in 2022). Rwanda's series had a large step up in 2022 (from 842.6 in 2021 to 1,007.1).

Limits of this comparison

  • The sources give the figures but do not explain the 2022 jump in Rwanda's GDP per capita or the 2023 dip in Kenya's. These series are in current US dollars, so exchange-rate movements may account for some of the changes, but I have not verified this from the documents.
  • The data show that internet use and income rose together in both countries. They do not show that one caused the other.
  • Mobile money is a related but separate indicator. The mobile money note says Kenya's service launched in 2007 and became a reference case, while Rwanda's grew later and faster, supported by interoperability and use for public fees and transport. It also notes that mobile money can run on basic phones without internet access, so mobile money use and internet use are different measures [mobile-money · intro · 2026]. That note contains no figures, so I have not quoted any mobile money numbers.

The model chose the same five calls that the workflow made. On a different question it would choose differently. That is the point of an agent, and also its risk: the calls are not known in advance.

The first column of the table is the round: one message from the model, which can ask for several tools at once. Here it asked for all five in its first message, so they share round 1; in section 4, a call that fails takes a second round to correct. The application can now let the model choose between its tools.

3. A question the tools cannot answer​

A good agent searches, finds nothing usable, and says so. The question is about coffee exports. The collection holds one note on coffee that carries no figures, and the data tool has no series on exports.

w.use("nb05_agent_offscope")
result = agent.invoke({"messages": [HumanMessage(content="How have Kenya's coffee export earnings changed in recent years?")]}, config={"recursion_limit": 10})
w.show_trace(result)
print()
w.show(w.final_text(result), role="assistant")
roundcalltoolwhat was askedwhat came back
11search_documentsquery=Kenya coffee export earnings3 passages, first [coffee · intro · 2026]
12get_indicatorindicator=TX.VAL.FOOD.ZS.UN, country=Kenyaerror: unknown indicator 'TX.VAL.FOOD.ZS.UN'
13get_indicatorindicator=TX.VAL.AGRI.ZS.UN, country=Kenyaerror: unknown indicator 'TX.VAL.AGRI.ZS.UN'
assistant

I couldn't find figures on Kenya's coffee export earnings in the tools available to me, so I can't say how they have changed.

  • Indicator data: The World Development Indicators tool has no series for coffee exports or for agricultural export shares. My two attempts used guessed IDs, which the tool rejected. The indicators it does offer are GDP per capita, GDP growth, population, internet users, and mobile subscriptions. None measures coffee earnings.
  • Documents: The collection describes the sector but gives no numbers. It notes that "Export earnings in both countries depend on world prices, on the volume of the harvest, and on the share of the crop that reaches specialty grades. Weather, plant disease, and the age of the trees affect volumes from year to year." [coffee · intro · 2026]. It also says most Kenyan coffee is sold through a central auction and that growers are mainly smallholder cooperatives and estates [coffee · intro · 2026].

For a trend, you would need a source that reports coffee export values, such as Kenya's national statistics office, the Central Bank of Kenya, or the International Coffee Organization. If you can add one of those series to the dataset, I can pull and compare it.

The collection does have one related economic series: Kenya's GDP per capita in current US dollars, which was 1,092 in 2010, 2,110 in 2022, and 2,133 in 2024. It is not a coffee measure, and the document notes that exchange-rate movements affect it [kenya-economy · GDP per capita · 2026].

The agent made one search and tried two indicator ids that the tool rejected, then declined to invent figures and said what it had looked for. The application can now say that it cannot answer, and what would be needed.

4. A call that fails​

The model asks for an indicator id that does not exist. The tool does not crash: it returns an error as data, with the list of valid ids. The model reads it and tries again. Returning errors as data is what lets the loop recover.

w.use("nb05_agent_error")
result = agent.invoke({"messages": [HumanMessage(content="What share of Rwanda's population used the internet in 2023?")]}, config={"recursion_limit": 10})
w.show_trace(result)
print()
w.show(w.final_text(result), role="assistant")
roundcalltoolwhat was askedwhat came back
11get_indicatorindicator=IT.NET.USERS.ZS, country=Rwanda, start=2023, end=2023error: unknown indicator 'IT.NET.USERS.ZS'
22get_indicatorindicator=IT.NET.USER.ZS, country=Rwanda, start=2023, end=20231 observations, 2023 to 2023, latest 28.8
assistant

In 2023, 28.8 percent of Rwanda's population used the internet (World Development Indicators, IT.NET.USER.ZS).

The first row is the failed call, in round 1; the second is the retry that worked, in round 2. The application can now recover from a wrong call, as long as its tools return errors that the model can read.

5. A loop that never ends​

A model can also keep calling tools without ever answering. The step limit stops it: after the given number of steps, the run ends with an error (GraphRecursionError), and the application decides what to do next. It can answer with what it has, ask the user, or give up. The recorded scenario below is a model that never stops: the cell sets the limit to 6, and the model keeps asking for tools until the loop is cut off.

from langgraph.errors import GraphRecursionError

w.use("nb05_runaway")
try:
    agent.invoke({"messages": [HumanMessage(content=w.MAIN_QUESTION)]}, config={"recursion_limit": 6})
except GraphRecursionError as e:
    print("stopped:", str(e)[:80])
    print("The application now chooses: here, it tells the user that it could not finish.")
stopped: Recursion limit of 6 reached without hitting a stop condition. You can increase 
The application now chooses: here, it tells the user that it could not finish.

The application can now stop a loop that does not end, instead of waiting for it.

6. Choosing between the two​

WorkflowAgent
Calls per runKnown in advanceVaries with the question
Behaviour on a new kind of questionSame calls, possibly irrelevantAdapts, possibly wrongly
CostPredictableBounded by the step limit
TestingTest the final step and the dataTest the trace and the answer
Use whenThe questions have one shapeThe questions vary and the tools are few and safe

The worked example has one shape, so a workflow is the right design for it. The agent earns its place when users ask questions the designer did not anticipate, and only with a step limit, a trace, and tools that fail as data.

7. Can we trust the numbers?​

Both designs produce an answer that holds several numbers. Some come straight from a tool, some come from a passage, and some are worked out by the model itself: "rose by about 23.7 percentage points" is a subtraction, which no tool returned. The answer is plain text, and a reader, or a program, has no way to tell which number is which.

The tools have already laid some groundwork. Every figure the data tool returns carries a label, called a claim_id, made of the indicator, the country and the year. The cell shows the latest figure of each series that the workflow fetched.

latest = [(s["indicator"], s["country"], o["claim_id"], f"{o['value']:,.1f}" if o["value"] < 1000 else f"{o['value']:,.0f}") for s in workflow_series for o in s["observations"][-1:]]
w.show_table(["indicator", "country", "claim_id", "value"], latest, source="Every figure the tool returns has a claim_id: indicator, country, year.")
indicatorcountryclaim_idvalue
IT.NET.USER.ZSKENIT.NET.USER.ZS:KEN:202435.0
IT.NET.USER.ZSRWAIT.NET.USER.ZS:RWA:202431.7
NY.GDP.PCAP.CDKENNY.GDP.PCAP.CD:KEN:20242,133
NY.GDP.PCAP.CDRWANY.GDP.PCAP.CD:RWA:20241,060

Every figure the tool returns has a claim_id: indicator, country, year.

The labels sit on the tool's side, and nothing in the answer points back to them. What if every number in an answer carried its label, and a program checked each number against the record it names? The program has a protocol for exactly that, Proof-Carrying Numbers (PCN), its approach to trustworthy data, and notebook 06 builds it into the application.

Key points​

  • A workflow is a fixed order of steps written in code: predictable and easy to test. Prefer it when the questions have one shape.
  • An agent is a model that chooses its next step. It adapts to the question, so it needs a step limit, a trace, and tools that return their errors as data.
  • Both designs need the same from their tools: clear descriptions and results the model can read.
  • An answer built from several calls mixes numbers that come from tools, from passages and from the model's own arithmetic, and plain text does not say which is which. The labels the tool already attaches are the groundwork for checking that, in notebook 06.

Exercise​

  1. In section 2, change recursion_limit from 10 to 3 and run the cell, then try 4. What is the smallest limit that lets this question finish? What does the application see when the limit is too low?
  2. After section 2, run sum(len(m.tool_calls) for m in result["messages"] if getattr(m, "tool_calls", None)). How many calls did the agent make, and how many did the workflow make in section 1? Why are the two numbers equal here, and when would they differ?
  3. Run w.indicator_lookup("IT.NET.USER.ZS", "Uganda", start=2010)["observations"]. The tool returns an empty list. What would the workflow of section 1 do with this result, and what would the agent do?

Where this connects to the program​

Next: 06 · Verification