Skip to main content

03 · Retrieval-augmented generation

Open in Colab View the notebook on GitHub

The outputs on this page are the ones committed with the notebook. Without an API key, the notebook replays recorded model answers, and the same recordings appear here.

Session 01 · Building AI applications for research and data · notebook 3 of 6

Objective​

A language model answers from the text in its prompt. Notebook 01 placed four briefs in the prompt by hand. A real collection holds thousands of documents and a prompt holds a few pages, so an application has to select the pages for each question. Notebook 02 prepared the collection for that selection: it split the eight documents into 21 section chunks, kept each document's date as metadata, and stored a vector for each chunk. This notebook performs the selection and the answer. The pattern is retrieval-augmented generation (RAG): retrieve the chunks closest to the question, add them to the prompt, and generate the answer from them. A retrieved chunk is called a passage.

The notebook then shows three ways retrieval fails: a question with two parts, an outdated passage that outranks current ones, and a question the collection cannot answer. Each failure has a fix on the retrieval side, and the last section tests retrieval without the model.

The application after this notebook​

Diagram: the question goes to retrieval, a search of the vector store that returns the closest chunks as passages; the passages and the question go into the prompt, then to generation, where the model writes an answer with citations.

Retrieval selects the passages. The model's step is now called generation, because the model writes from what retrieval supplied.

Contents​

  1. Section 1 answers the main question from four retrieved passages. The answer covers internet use only.
  2. Section 2 shows the ranking behind that answer. The economy chunks fall below the cut-off, and one search per half of the question brings them back.
  3. Section 3 asks for the latest figures. The 2016 summary outranks the 2026 briefs, and the answer reports a 2015 value.
  4. Section 4 removes the summary with a filter on the date. Rewording the question does not remove it.
  5. Section 5 asks a question the collection cannot answer. The search returns passages anyway.
  6. Section 6 tests retrieval alone on the eight evaluation questions.

How to run it​

Run the cells in order. Without an API key, the model's answers are recorded ones, labelled [recorded output: ...], as in notebook 01. With a key in WORKSHOP_API_KEY, the same cells call the model live, the answer streams into its bubble as the model writes, and the wording differs from the recorded answers. The searches, the rankings, and the tables are computed when a cell runs, in both modes.

The next cell installs the session's tools and loads its helper module. The first run takes about a minute. The cell is setup only.

# Setup. In Colab this clones the repository and installs the dependencies; locally it uses the folder as is.
import os, sys, subprocess, pathlib

REPO_URL = "https://github.com/worldbank/ai4data.git" # the program's repository
BRANCH = "main" # the branch the Colab setup clones
SESSION = "training/session-01-building-ai-applications/notebooks"

if not pathlib.Path("workshop.py").exists():
subprocess.run(["git", "clone", "-q", "--depth", "1", "--branch", BRANCH, REPO_URL, "/content/ai4data"], check=True)
os.chdir(f"/content/ai4data/{SESSION}")
subprocess.run([sys.executable, "-m", "pip", "install", "-q", "-r", "requirements.txt"], check=True)
sys.path.insert(0, os.getcwd())

# Optional: a key for an OpenAI-compatible API. Without one, the notebook replays recorded outputs and says so.
try:
from google.colab import userdata # type: ignore
for name in ("WORKSHOP_API_KEY", "WORKSHOP_BASE_URL", "WORKSHOP_MODEL"):
if userdata.get(name):
os.environ[name] = userdata.get(name)
except Exception:
pass

import workshop as w
model = w.make_model()
print("Model:", w.mode(model))
Model: recorded outputs (no WORKSHOP_API_KEY set)

The helper functions used in this notebook, in the order they appear. w.use, w.show, and w.answer are as in notebook 01.

CallWhat it does
w.load_documents(), w.split_sections(docs), w.build_store(chunks)The three steps of notebook 02: load the documents, split them into section chunks, embed the chunks into an in-memory vector store.
store.similarity_search(q, k=...)LangChain's search: the k chunks whose vectors are closest to the question's. similarity_search_with_score also returns each score.
w.format_context(chunks)Joins chunks into the context text, each prefixed with its citation.
w.citation(chunk)The citation of a chunk: document id, section, and year, from its metadata.
w.print_hits(results), w.show_ranking(...), w.show_search_comparison(...)Print or draw search results.
w.load_questions()The eight evaluation questions of the session, each with its expected documents and numbers.
w.numbers_present(text, expected), w.retrieval_hit(chunks, expected_docs)The two retrieval checks: how many expected numbers a text contains, and whether an expected document is among the chunks.

Each Colab notebook starts with an empty session, so the next cell rebuilds the store of notebook 02 with the same three calls.

docs = w.load_documents() # the eight session documents
chunks = w.split_sections(docs) # one chunk per Markdown section, metadata carried over
store = w.build_store(chunks) # a vector for every chunk, in memory
print(len(chunks), "chunks in the store, the same collection as in notebook 02")
21 chunks in the store, the same collection as in notebook 02

The application is the one of notebook 01 with retrieval in front of it. The instructions now speak of passages and of a citation form, which w.format_context writes before each passage. make_rag builds the chain: a search step that takes the question and returns the k closest chunks, formatted as context; a pass-through that forwards the question unchanged; the prompt; the model. hits runs the search alone and keeps the scores. The optional where is a filter on metadata, used in section 4.

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnableLambda, RunnablePassthrough

INSTRUCTIONS = """You are a research assistant for development data.
Answer only from the supplied passages. After each figure, cite the passage in the square-bracket form given.
If the passages do not contain what is needed, say so."""

prompt = ChatPromptTemplate.from_messages([("system", INSTRUCTIONS), ("human", "PASSAGES:\n{context}\n\nQUESTION: {question}")])


def make_rag(k=4, where=None):
"""RAG (retrieval-augmented generation): search the store, put the passages in the prompt, ask the model."""
retrieve = RunnableLambda(lambda q: store.similarity_search(q, k=k, filter=where)) # question -> k closest chunks
return {"context": retrieve | w.format_context, "question": RunnablePassthrough()} | prompt | model


def hits(question, k=4, where=None):
"""The search alone: (chunk, score) pairs, best first."""
return store.similarity_search_with_score(question, k=k, filter=where)

1. The main question answered from the store​

Retrieval scores every chunk by the distance between its vector and the question's vector, and returns the k closest. A vector is a list of numbers that summarises what a text is about, computed by the embedding model of notebook 02. The score printed with each passage is the cosine similarity: 1.0 means identical meaning and 0 means unrelated. The next cell shows the question, prints the four passages that hits returns for it with their scores, and then calls the chain through w.answer, which repeats the same search inside and asks the model.

w.show(w.MAIN_QUESTION, role="user")
print()
w.print_hits(hits(w.MAIN_QUESTION, k=4)) # what the model is about to read
print()
w.use("nb03_rag_main")
reply = w.answer(make_rag(k=4), w.MAIN_QUESTION)
user

How have internet access and economic development evolved in Rwanda and Kenya? Compare the recent available data, summarize the main patterns, and cite the sources.


0.763 [rwanda-digital · Internet use · 2026]
0.748 [regional-2016 · intro · 2016]
0.676 [kenya-digital · Internet use · 2026]
0.591 [mobile-money · intro · 2026]
assistant · recorded output: nb03_rag_main

Internet access and economic development in Rwanda and Kenya

Economic development: not covered. The passages contain no GDP, income, poverty, or other economic indicators for either country, so I can't assess how economic development has changed. The discussion below covers internet access and mobile subscriptions only.

Internet use, latest available data

  • Rwanda: 31.7% of the population in 2024, up from 8.0% in 2010 [rwanda-digital · Internet use · 2026].
  • Kenya: 35.0% of the population in 2024, up from 7.2% in 2010 [kenya-digital · Internet use · 2026].

Main patterns

  • Both countries grew substantially. Rwanda gained about 24 percentage points and Kenya about 28 over 2010–2024 [rwanda-digital · Internet use · 2026; kenya-digital · Internet use · 2026].
  • Rwanda's fastest growth was 2015–2020, when its share nearly doubled, from 12.5% to 23.8%. Growth then slowed to 2023 and picked up in 2024 [rwanda-digital · Internet use · 2026].
  • Kenya's fastest growth was 2010–2015, when its share more than doubled, from 7.2% to 16.6%. Growth continued after 2020, reaching 24.7% in 2020, 29.4% in 2022, 32.1% in 2023, and 35.0% in 2024 [kenya-digital · Internet use · 2026].
  • Kenya led in every year where both countries have figures. The gap was 4.1 points in 2015 (16.6% vs 12.5%) and narrowed to 0.9 points in 2020 (24.7% vs 23.8%). It widened again to 3.3 points in 2024 (35.0% vs 31.7%) [regional-2016 · intro · 2016; rwanda-digital · Internet use · 2026; kenya-digital · Internet use · 2026].
  • Mobile subscriptions were high in both countries in 2015: 80.1 per 100 people in Kenya and 75.3 per 100 in Rwanda. No later figures are given [regional-2016 · intro · 2016].

Caveats

  • The 2016 regional summary projected that internet use would reach about 25% by 2020 and about half the population by 2025 [regional-2016 · intro · 2016]. The 2020 figures came close (23.8% for Rwanda, 24.7% for Kenya). The 2025 projection cannot be checked against the supplied data, and the 2024 values sit well below half. That summary also says later values would be added as released, so its figures are superseded by the 2026 country notes.
  • The mobile money note gives no figures. It says mobile money use is not the same as internet use, since the service works through USSD on basic phones [mobile-money · intro · 2026]. It also describes Rwanda's services as having grown later and faster than Kenya's, but supplies no indicator values to support that comparison [mobile-money · intro · 2026].

The answer opens by saying that the passages contain no economic indicators, and it covers internet use with a citation after each figure. The instructions require both behaviours. The GDP figures are in the store, in the two economy briefs, and retrieval did not return them. Section 2 shows why.

2. The ranking behind the answer​

A similarity search compares meanings. The question's vector lies closer to the internet chunks than to the economy chunks, because most of the question's words concern connectivity. The search ranks by this closeness alone. It cannot reserve places for the second half of a question, because it never identifies the halves. The next cell asks for the ten closest chunks, labels each with its topic from a small dictionary keyed by document id, and draws the ranking with the cut-off after the fourth.

topic = {"kenya-digital": "internet", "rwanda-digital": "internet", "regional-2016": "internet", "internet-indicator": "internet",
"kenya-economy": "economy", "rwanda-economy": "economy", "coffee": "other", "mobile-money": "other"}
legend = {"internet": ("internet use", "#2f5f9e"), "economy": ("economy", "#d9822b"), "other": ("other topics", "#8a94a3")}

ranked = hits(w.MAIN_QUESTION, k=10)
w.show_ranking([(w.citation(d), score, topic[d.metadata["id"]]) for d, score in ranked], k=4, legend=legend, question=w.MAIN_QUESTION)
Question: How have internet access and economic development evolved in Rwanda and Kenya? Compare the recent available data, summarize the main patterns, and cite the sources.
internet useeconomyother topics
1
[rwanda-digital · Internet use · 2026]
0.763
2
[regional-2016 · intro · 2016]
0.748
3
[kenya-digital · Internet use · 2026]
0.676
4
[mobile-money · intro · 2026]
0.591
only the top 4 go to the model
5
[rwanda-economy · Reading the two indicators together · 2026]
0.570
6
[rwanda-economy · GDP per capita · 2026]
0.516
7
[coffee · intro · 2026]
0.487
8
[kenya-economy · Reading the two indicators together · 2026]
0.476
9
[kenya-economy · GDP per capita · 2026]
0.442
10
[rwanda-digital · Population · 2026]
0.414

The four chunks above the line are three internet chunks, including the 2016 summary, and the mobile-money note. The first economy chunk is fifth at 0.570 against 0.591 for fourth place, and Kenya's GDP per capita is ninth. Raising k is a poor remedy: the ninth place costs nine passages, the coffee note enters at seventh, and every passage adds tokens.

The fix is one search per half of the question, with the results merged. An application can split the question with a model; here the two halves are written by hand. The next cell runs a search for each half, merges the results without duplicates, and compares the merged list with the single search of section 1. The comparison marks which of the four figures a correct answer needs appear in each list; needed comes from the evaluation file of the session, and w.numbers_present looks for each figure in the text of the passages.

sub_queries = ["internet use in Kenya and Rwanda, recent figures", "GDP per capita and economic growth in Kenya and Rwanda, recent figures"]
merged = []
for q in sub_queries:
for d in store.similarity_search(q, k=4): # four passages per half
if d not in merged:
merged.append(d)

one_search = [d for d, _ in hits(w.MAIN_QUESTION, k=4)]
needed = next(item for item in w.load_questions() if item["id"] == "q1")["expected_numbers"]
countries = {"KEN": "Kenya", "RWA": "Rwanda"}
names = {"IT.NET.USER.ZS": "internet use", "NY.GDP.PCAP.CD": "GDP per capita"}
checks = [f"{countries[n['country']]} {names[n['indicator']]}, {n['year']}: {n['value']:,}" for n in needed]


def column(title, tone, found):
text = " ".join(d.page_content for d in found)
return {"title": title, "tone": tone, "hits": [(w.citation(d), d.page_content) for d in found],
"found": [w.numbers_present(text, [n])[0] == 1 for n in needed]}


w.show_search_comparison(w.MAIN_QUESTION, [column("One search for the whole question: 4 passages", "bad", one_search),
column(f"The question broken into two, one search each, merged: {len(merged)} passages", "good", merged)], checks)
Question: How have internet access and economic development evolved in Rwanda and Kenya? Compare the recent available data, summarize the main patterns, and cite the sources.
One search for the whole question: 4 passages
[rwanda-digital · Internet use · 2026]
# Rwanda: internet use and mobile subscriptions, 2010 to 2024 ## Internet use The share of … slowed between 2020 and 2023 and picked up again in 2024.
[regional-2016 · intro · 2016]
# Regional digital summary, 2016 Internet use in East Africa continues to expand. In Kenya, 16.6 … Values for later years will be added as they are released.
[kenya-digital · Internet use · 2026]
# Kenya: internet use and mobile subscriptions, 2010 to 2024 ## Internet use The share of … between 2010 and 2015, when the share more than doubled.
[mobile-money · intro · 2026]
# Mobile money in Kenya and Rwanda Mobile money services let a subscriber hold a balance on a … database and in national financial-inclusion surveys.
✓ found Kenya internet use, 2024: 35.0
✓ found Rwanda internet use, 2024: 31.7
✗ missing Kenya GDP per capita, 2024: 2,133
✗ missing Rwanda GDP per capita, 2024: 1,060
The question broken into two, one search each, merged: 8 passages
[regional-2016 · intro · 2016]
# Regional digital summary, 2016 Internet use in East Africa continues to expand. In Kenya, 16.6 … Values for later years will be added as they are released.
[rwanda-digital · Internet use · 2026]
# Rwanda: internet use and mobile subscriptions, 2010 to 2024 ## Internet use The share of … slowed between 2020 and 2023 and picked up again in 2024.
[kenya-digital · Internet use · 2026]
# Kenya: internet use and mobile subscriptions, 2010 to 2024 ## Internet use The share of … between 2010 and 2015, when the share more than doubled.
[mobile-money · intro · 2026]
# Mobile money in Kenya and Rwanda Mobile money services let a subscriber hold a balance on a … database and in national financial-inclusion surveys.
[rwanda-economy · Reading the two indicators together · 2026]
## Reading the two indicators together Rwanda recorded growth above 7 percent in 10 of the 15 years … 2010 and 2024 from a lower starting level than Kenya's.
[kenya-economy · GDP per capita · 2026]
# Kenya: GDP per capita and growth, 2010 to 2024 ## GDP per capita GDP per capita in current US … 2023 decline coincides with a depreciation of the shilling.
[kenya-economy · Reading the two indicators together · 2026]
## Reading the two indicators together Kenya's economy grew in every year of the period except … rate accounting for part of the year-to-year variation.
[rwanda-economy · GDP per capita · 2026]
# Rwanda: GDP per capita and growth, 2010 to 2024 ## GDP per capita GDP per capita in current US … exchange rates and prices affect it as well as real output.
✓ found Kenya internet use, 2024: 35.0
✓ found Rwanda internet use, 2024: 31.7
✓ found Kenya GDP per capita, 2024: 2,133
✓ found Rwanda GDP per capita, 2024: 1,060

The merged search returns eight passages where the single search returned four, and all four needed figures are among them. The 2016 summary and the mobile-money note are among them too. The next cell gives the merged passages to the model. It calls prompt | model directly, because the chain of make_rag would run its own search.

w.use("nb03_rag_two_queries")
reply = w.answer(prompt | model, {"context": w.format_context(merged), "question": w.MAIN_QUESTION})
assistant · recorded output: nb03_rag_two_queries

Internet use and GDP per capita in Rwanda and Kenya, 2010 to 2024

Internet use (share of population)

  • Rwanda rose from 8.0% in 2010 to 12.5% in 2015, 23.8% in 2020, 28.0% in 2022, 28.8% in 2023, and 31.7% in 2024 [rwanda-digital · Internet use · 2026]. Its fastest growth was 2015 to 2020, when the share nearly doubled. Growth slowed between 2020 and 2023 and picked up again in 2024 [rwanda-digital · Internet use · 2026].
  • Kenya rose from 7.2% in 2010 to 16.6% in 2015, 24.7% in 2020, 29.4% in 2022, 32.1% in 2023, and 35.0% in 2024 [kenya-digital · Internet use · 2026]. Its fastest absolute gain came between 2010 and 2015, when the share more than doubled [kenya-digital · Internet use · 2026].
  • Kenya had a slightly lower starting point in 2010 and a higher share in 2024. Over the full period the share rose by about 28 percentage points in Kenya [kenya-digital · Internet use · 2026], compared with roughly 24 points in Rwanda (calculated from 8.0% and 31.7%) [rwanda-digital · Internet use · 2026].
  • The 2016 regional summary projected that internet use in both countries would reach about 25% by 2020 and about half the population by 2025 [regional-2016 · intro · 2016]. The 2020 actuals were close to that projection: 24.7% in Kenya and 23.8% in Rwanda [kenya-digital · Internet use · 2026; rwanda-digital · Internet use · 2026]. The passages do not include a 2025 figure, so the half-population projection cannot be checked.

GDP per capita (current US dollars)

  • Rwanda rose from 587 in 2010 to 726 in 2015, 803 in 2020, 1,007 in 2022, 1,061 in 2023, and 1,060 in 2024 [rwanda-economy · GDP per capita · 2026]. It passed 1,000 in 2022. The series was roughly 80% higher in 2024 than in 2010 [rwanda-economy · Reading the two indicators together · 2026]. Rwanda's growth exceeded 7% in 10 of the 15 years, and 2020 was the only year of contraction [rwanda-economy · Reading the two indicators together · 2026].
  • Kenya rose from 1,092 in 2010 to 1,489 in 2015, 1,928 in 2020, 2,110 in 2022, 1,943 in 2023, and 2,133 in 2024 [kenya-economy · GDP per capita · 2026]. GDP per capita roughly doubled over the period. Kenya's economy grew in every year except 2020 [kenya-economy · Reading the two indicators together · 2026].
  • Kenya's GDP per capita was higher than Rwanda's throughout the period, with the gap widening from about 500 dollars in 2010 to about 1,070 dollars in 2024 (calculated from the figures above).

Main patterns

  1. Internet use grew in both countries. Kenya had the larger gain and the higher 2024 share, and both countries grew fastest in the early part of the period. Rwanda's fastest growth came between 2015 and 2020 [rwanda-digital · Internet use · 2026; kenya-digital · Internet use · 2026].
  2. Both economies grew over the period, with a shared dip in 2020. Rwanda's per-capita figure stalled after 2023, and Kenya's fell in 2023 [rwanda-economy · GDP per capita · 2026; kenya-economy · GDP per capita · 2026].
  3. The GDP series are in current dollars, so exchange rates and prices affect them as well as real output. The 2023 decline in Kenya coincides with a depreciation of the shilling [kenya-economy · GDP per capita · 2026]. The same caveat applies to Rwanda's series [rwanda-economy · GDP per capita · 2026].

Limitations

  • The passages show that internet use and GDP per capita rose together over the period, but they do not establish a causal link between them.
  • The passages contain no mobile money figures. The mobile money note states that account indicators are published in the Global Findex database and in national financial-inclusion surveys [mobile-money · intro · 2026].

The answer covers both halves with a citation after each figure. The two extra passages did no harm: the model used the 2016 summary for a projection and said the projection could not be checked, and it reported that the mobile-money note holds no figures. Section 3 shows what happens when the 2016 summary is the passage an answer depends on.

3. Outdated evidence in the ranking​

A vector encodes what a text is about and nothing about when it was written. A 2016 summary that uses the phrase "latest available figures" scores high for a question that uses the same phrase, whatever its date. The next cell asks for the latest figures on internet use in Kenya and Rwanda with k = 2, prints the two passages with their scores, and shows the answer.

q = "What are the latest available figures for internet use in Kenya and Rwanda?"
w.print_hits(hits(q, k=2))
print()
w.use("nb03_rag_outdated")
reply = w.answer(make_rag(k=2), q)
0.816 [regional-2016 · intro · 2016]
0.765 [rwanda-digital · Internet use · 2026]
assistant · recorded output: nb03_rag_outdated

Rwanda: 31.7 percent of the population used the internet in 2024, the latest year available. [rwanda-digital · Internet use · 2026]

Kenya: 16.6 percent of the population used the internet in 2015. [regional-2016 · intro · 2016] The passages contain no later Kenya figure, so this is the most recent value I can report. The regional summary is from 2016, and it says that values for later years would be added as they were released.

The 2016 summary ranks first at 0.816 and Rwanda's 2026 brief second at 0.765. Kenya's brief is absent, so the model's Kenya figure is the summary's: 16.6 percent in 2015. The model states that the summary dates from 2016 and that 16.6 is the most recent value in its passages. The answer is correct for its passages and wrong for the user, who asked for the latest figures while Kenya's 2024 value sits in the store. Generation cannot repair a selection made before the model saw anything. The fix belongs to retrieval.

4. Three fixes for outdated evidence​

  1. A filter on the date. The store checks metadata before it scores similarity, so a document dated before 2020 is never ranked. Notebook 02 kept the date for this.
  2. A reworded question. Rewriting the question to resemble the expected passage changes the question's vector. The ranking can move either way.
  3. Sorting by date after the search, for a store without a filter. It does the same job as the filter and is not shown.

The next cell defines the filter as a function of a chunk, since_2020, which the store calls on each chunk's metadata before scoring. It then runs the same search with and without the filter and marks which of the two needed figures each returns.

since_2020 = lambda d: str(d.metadata.get("date", "")) >= "2020" # the filter: keep chunks dated 2020 or later
expected = next(item for item in w.load_questions() if item["id"] == "q8")["expected_numbers"]
countries = {"KEN": "Kenya", "RWA": "Rwanda"}
checks = [f"{countries[e['country']]}, {e['year']}: {e['value']}" for e in expected]


def search(title, tone, where=None):
found = store.similarity_search(q, k=2, filter=where)
text = " ".join(h.page_content for h in found)
return {"title": title, "tone": tone, "hits": [(w.citation(h), h.page_content) for h in found],
"found": [w.numbers_present(text, [e])[0] == 1 for e in expected]}


w.show_search_comparison(q, [search("No filter", "bad"), search("Date filter: 2020 or later", "good", since_2020)], checks)
Question: What are the latest available figures for internet use in Kenya and Rwanda?
No filter
[regional-2016 · intro · 2016]
# Regional digital summary, 2016 Internet use in East Africa continues to expand. In Kenya, 16.6 … Values for later years will be added as they are released.
[rwanda-digital · Internet use · 2026]
# Rwanda: internet use and mobile subscriptions, 2010 to 2024 ## Internet use The share of … slowed between 2020 and 2023 and picked up again in 2024.
✗ missing Kenya, 2024: 35.0
✓ found Rwanda, 2024: 31.7
Date filter: 2020 or later
[rwanda-digital · Internet use · 2026]
# Rwanda: internet use and mobile subscriptions, 2010 to 2024 ## Internet use The share of … slowed between 2020 and 2023 and picked up again in 2024.
[kenya-digital · Internet use · 2026]
# Kenya: internet use and mobile subscriptions, 2010 to 2024 ## Internet use The share of … between 2010 and 2015, when the share more than doubled.
✓ found Kenya, 2024: 35.0
✓ found Rwanda, 2024: 31.7

With the filter, Kenya's brief replaces the summary and both 2024 figures are present. The next cell passes the filter to make_rag and shows the answer from the filtered passages.

w.use("nb03_rag_filtered")
reply = w.answer(make_rag(k=2, where=since_2020), q)
assistant · recorded output: nb03_rag_filtered

The latest available figures are for 2024:

  • Kenya: 35.0 percent of the population used the internet [kenya-digital · Internet use · 2026].
  • Rwanda: 31.7 percent of the population used the internet [rwanda-digital · Internet use · 2026].

The second fix, rewording, is tried next without the filter. The question is rewritten in the vocabulary of the 2026 briefs.

w.print_hits(hits("Kenya and Rwanda internet use 2024, percent of population", k=2))
0.871 [regional-2016 · intro · 2016]
0.833 [rwanda-digital · Internet use · 2026]

The reworded question puts the summary first at 0.871, above Rwanda's brief at 0.833. The summary shares more vocabulary with the new wording than the brief does, so the rewording strengthened the match. The date is the only property that separates the summary from the briefs, and only metadata carries it.

5. A question outside the collection​

A similarity search ranks every chunk and returns the top k. It has no notion of a chunk being unrelated: the top three for an unanswerable question are the three least unrelated chunks. The next cell asks for the population of Rwanda, which the collection holds, and for inflation in Uganda in 2023, which it does not, and prints three passages for each.

for q2 in ["What is the population of Rwanda?", "What was inflation in Uganda in 2023?"]:
print(q2)
w.print_hits(hits(q2, k=3))
print()
What is the population of Rwanda?
0.739 [rwanda-digital · Population · 2026]
0.598 [rwanda-economy · Reading the two indicators together · 2026]
0.595 [rwanda-economy · GDP per capita · 2026]

What was inflation in Uganda in 2023?
0.534 [rwanda-economy · GDP per capita · 2026]
0.523 [kenya-economy · GDP per capita · 2026]
0.506 [rwanda-economy · Reading the two indicators together · 2026]

The population question returns the right chunk first at 0.739. The inflation question returns three GDP chunks about Kenya and Rwanda, the best at 0.534. The scores separate the two cases, and a cut-off between them would stop the second search from reaching the model. The cut-off depends on the embedding model and has to be set from examples. The second safeguard is the instruction from notebook 01 that tells the model to say when the passages do not answer, which notebook 05 shows in use. An application uses both.

6. A test of the search without the model​

Retrieval can be tested alone, because its output is a list of passages that can be checked without reading an answer. The next cell runs the eight evaluation questions through the search and makes two checks per question. w.retrieval_hit reports whether one of the expected documents is among the passages. w.numbers_present counts how many of the numbers a correct answer needs appear in the passages' text. K sets the number of passages, and the exercise changes it.

K = 4 # passages returned by the search: try 2, then 8

rows = []
for item in w.load_questions():
passages = store.similarity_search(item["question"], k=K)
text = " ".join(p.page_content for p in passages)
needed = item.get("expected_numbers", [])
found = sum(w.numbers_present(text, [n])[0] for n in needed)
rows.append((item["id"], item["category"],
"hit" if w.retrieval_hit(passages, item["expected_documents"]) else "miss",
f"{found}/{len(needed)}" if needed else "none needed"))
w.show_table(["id", "category", "expected document", "needed numbers"], rows, source=f"The search returns {K} passages per question.")
idcategoryexpected documentneeded numbers
q1mainhit2/4
q2factualhit1/1
q3ambiguoushitnone needed
q4missing-datahitnone needed
q5out-of-scopehitnone needed
q6factualhit2/2
q7definitionhitnone needed
q8outdated-evidencehit2/2

The search returns 4 passages per question.

At K = 4 every question has an expected document among its passages, and q1, the main question, has two of its four numbers: the problem of section 2. A missing number here cannot be fixed by any instruction to the model. When both checks pass and the answer is still wrong, the fault lies in generation: the instructions or the model. Testing the two halves separately shows which one to change. Notebook 06 adds a check of a different kind, of each number in the answer against its source record.

Key points​

  • Retrieval scores every chunk by the distance between its vector and the question's, and returns the k closest as passages for the prompt.
  • A vector encodes meaning. A question with two parts matches the chunks of one part and misses the other; one search per part fixes it.
  • A vector encodes nothing about dates. A filter on metadata removes outdated passages; rewording the question does not.
  • A search returns the top k chunks even when none is related. A cut-off score stops unrelated passages from reaching the model, and an instruction tells the model to say when the passages do not answer.
  • Retrieval can be tested on its own by checking that the expected documents and numbers are among the passages. When a number is absent from the passages, the model never saw it, so no instruction to the model can put it in the answer; the fix is in retrieval.

Exercise​

  1. Look at the ranking in section 2. Which chunk takes fourth place, and is it a good use of one of the four slots? Say what in the chunk explains its score.
  2. In the cell of section 6, set K to 2, run it, then set it to 8 and run it again. Which questions lose numbers at 2, and which gain at 8? At 8, does every question find all of its needed numbers?
  3. Set K to 2 and look at q8, the question from section 3. Explain why it passes at 4 and fails at 2, using the two searches in section 4.

Where this connects to the program​

Next: 04 · Tools and structured outputs