02 · Preparing knowledge for an application
Open in Colab View the notebook on GitHub
The outputs on this page are the ones committed with the notebook. Without an API key, the notebook replays recorded model answers, and the same recordings appear here.
Session 01 · Building AI applications for research and data · notebook 2 of 6
Objective
Notebook 01 ended with a problem: an application cannot hand the model every document it has, so it has to select what to supply. Selecting needs a collection that has been prepared: documents together with information about them, cut into pieces, called chunks, that keep their meaning, and placed where a search can find them by meaning. This notebook builds that collection, and shows how the choices made here decide what retrieval can find later.
The application after this notebook
It shows the pipeline this notebook builds, and where notebook 03 picks it up.
Contents
- Section 1 loads the eight documents. Each has its text and a header of information about it.
- Section 2 shows what metadata records that the text cannot say about itself, such as the date of the oldest document.
- Section 3 cuts one brief two ways, by length and by heading, and compares the chunks.
- Section 4 turns chunks and a question into numbers and ranks the chunks by similarity.
- Section 5 searches the same collection prepared badly and prepared well, and counts which of the needed figures each search finds.
- Section 6 opens one record of the vector store: its text, its metadata, and its numbers.
How to run it
Run the cells in order. This notebook only prepares material for the model that writes the answers, so it does not use that model, and no API key is needed. The model comes back in notebook 03. Everything below is computed when you run the cell. It does use a small embedding model, which turns text into numbers; it downloads about 90 MB the first time and runs inside the notebook.
The next cell installs the session's tools and loads its helper module. The first run takes about a minute. The cell is setup only.
# Setup. In Colab this clones the repository and installs the dependencies; locally it uses the folder as is.
import os, sys, subprocess, pathlib
REPO_URL = "https://github.com/worldbank/ai4data.git" # the program's repository
BRANCH = "main" # the branch the Colab setup clones
SESSION = "training/session-01-building-ai-applications/notebooks"
if not pathlib.Path("workshop.py").exists():
subprocess.run(["git", "clone", "-q", "--depth", "1", "--branch", BRANCH, REPO_URL, "/content/ai4data"], check=True)
os.chdir(f"/content/ai4data/{SESSION}")
subprocess.run([sys.executable, "-m", "pip", "install", "-q", "-r", "requirements.txt"], check=True)
sys.path.insert(0, os.getcwd())
import workshop as w
print("Ready.")
Ready.
The tools in this notebook are LangChain's: Document, MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter, HuggingFaceEmbeddings, and InMemoryVectorStore.
1. The collection
The collection is eight short documents. Each file has two parts: a header with information about the document (its id, title, type, country, date, and source), and the text itself. LangChain loads each file as a Document: the text is the page_content, and the header becomes the metadata.
Four are the country briefs the worked example needs, one defines the internet indicator, one is a regional summary from 2016, and two, on coffee and on mobile money, are about other topics and are there to distract. See assets/README.md for what each is for.
docs = w.load_documents()
print(f"{'id':20s} {'type':16s} {'date':12s} title")
for d in docs:
m = d.metadata
print(f"{m['id']:20s} {m['type']:16s} {str(m['date']):12s} {m['title']}")
id type date title
coffee sector note 2026-10-13 Coffee exports in East Africa
internet-indicator indicator note 2026-10-13 How the internet-users indicator is defined and measured
kenya-digital country brief 2026-10-13 Kenya: internet use and mobile subscriptions, 2010 to 2024
kenya-economy country brief 2026-10-13 Kenya: GDP per capita and growth, 2010 to 2024
mobile-money topic note 2026-10-13 Mobile money in Kenya and Rwanda
regional-2016 summary 2016-06-01 Regional digital summary, 2016
rwanda-digital country brief 2026-10-13 Rwanda: internet use and mobile subscriptions, 2010 to 2024
rwanda-economy country brief 2026-10-13 Rwanda: GDP per capita and growth, 2010 to 2024
2. Metadata: what the text cannot say about itself
Metadata is information about a document that is not in its text: when it was written, what kind of document it is, which country it covers, where its numbers come from.
The cell below prints this document: some of its metadata (the date, the type, and a note), then its full text.
The 2016 summary shows why that matters. Read on its own, its text sounds current: it ends with "The latest available figures are for 2015." Its figures were correct when it was written. A reader might notice the year in the title, but an application does not read for that: it can only filter or rank on the date, which is kept as metadata, and which says that the summary was written in 2016, ten years before the briefs.
The date does nothing by itself: an application has to use it. Section 5 shows what is lost when the metadata is dropped, and notebook 03 uses the date to keep the 2016 summary out of an answer about the latest figures.
d = next(d for d in docs if d.metadata["id"] == "regional-2016")
for k in ("date", "type", "note"):
print(f"{k:6s}", d.metadata[k])
print()
w.show(d.page_content)
date 2016-06-01
type summary
note Written for the training workshop as an example of an outdated document. Not an official publication.
Regional digital summary, 2016
Internet use in East Africa continues to expand. In Kenya, 16.6 percent of the population used the internet in 2015, up from 7.2 percent in 2010. In Rwanda the share was 12.5 percent in 2015, up from 8.0 percent in 2010.
Mobile cellular subscriptions reached 80.1 per 100 people in Kenya and 75.3 per 100 people in Rwanda in 2015.
On current trends, internet use in both countries is expected to reach about 25 percent of the population by 2020 and about half of the population by 2025. Expansion of mobile broadband coverage and falling device prices are expected to drive the increase.
The latest available figures are for 2015. Values for later years will be added as they are released.
3. Chunking: fixed windows against sections
A chunk is the unit that is stored and searched: a search returns whole chunks. Two LangChain splitters on the same brief:
RecursiveCharacterTextSplitter: windows of 400 characters with an overlap. Simple, and blind to the document's structure.MarkdownHeaderTextSplitter: one chunk per heading, with the heading kept in the text and the section name added to the metadata.
The animation loops. It is drawn from the real brief and the two splitters. The cell below prints how many chunks each splitter made (4 windows and 4 sections) and shows two of them: window 2 of the fixed split, and the Mobile subscriptions section.
kenya = [d for d in docs if d.metadata["id"] == "kenya-digital"]
fixed = w.split_fixed(kenya, chunk_size=400, chunk_overlap=50)
sections = w.split_sections(kenya)
print(f"fixed windows: {len(fixed)} chunks · sections: {len(sections)} chunks\n")
print("A fixed window (note where it starts and ends):\n")
w.show(repr(fixed[1].page_content))
print("\nA section chunk:\n")
w.show(sections[1].page_content)
print("\nmetadata:", sections[1].metadata["section"], "|", sections[1].metadata["date"], "|", sections[1].metadata["country"])
fixed windows: 4 chunks · sections: 4 chunks
A fixed window (note where it starts and ends):
'Over the full period the share increased by about 28 percentage points. The fastest growth in absolute terms came between 2010 and 2015, when the share more than doubled.\n\n## Mobile subscriptions'
A section chunk:
Mobile subscriptions
Mobile cellular subscriptions per 100 people stood at 60.0 in 2010, 80.1 in 2015, and 117.6 in 2020. The figure was 121.2 in 2022, 120.6 in 2023, and 126.5 in 2024. Values above 100 reflect multiple subscriptions per person and inactive SIM cards, so the indicator measures subscriptions, and no inference about the share of people with a phone follows from it.
metadata: Mobile subscriptions | 2026-10-13 | Kenya
Window 2 cuts by length, not by meaning. It starts in the middle of the Internet use section, carries no heading of its own, and ends with the heading of the next section, separated from the paragraph it introduces. A passage about internet use is then stored next to a mobile-subscriptions heading, and nothing in the chunk says which section it belongs to. The Mobile subscriptions section carries its title and heading, which both the retriever and the model can use.
4. Embeddings
An embedding turns a text into a list of numbers that places it on a map of meaning: texts with similar meaning land close together. The session uses a small embedding model through LangChain's HuggingFaceEmbeddings; it downloads about 90 MB on first use. The cell embeds a few sections and a question and ranks the sections by similarity.
The animation loops. The six chunks and their similarity scores are the ones the cell below prints, for the question "share of people using the internet"; the dot positions and the example numbers are illustrative.
import numpy as np
chunks = w.split_sections(docs)
embeddings = w.make_embeddings()
sample = [c for c in chunks if c.metadata["section"] in ("Internet use", "Mobile subscriptions", "Growth")][:6]
vectors = np.array(embeddings.embed_documents([c.page_content for c in sample]))
query = np.array(embeddings.embed_query("share of people using the internet"))
print(f"{len(chunks)} chunks · vectors of {vectors.shape[1]} dimensions\n")
for c, v in sorted(zip(sample, vectors), key=lambda cv: -float(cv[1] @ query)):
print(f"{float(v @ query):6.3f} {c.metadata['id']:16s} {c.metadata['section']}")
21 chunks · vectors of 384 dimensions
0.407 kenya-digital Internet use
0.398 rwanda-digital Internet use
0.290 kenya-digital Mobile subscriptions
0.235 rwanda-digital Mobile subscriptions
0.045 rwanda-economy Growth
0.042 kenya-economy Growth
Sections about internet use score highest for a question about internet use, the mobile-subscription sections next, and growth last. The ranking depends on what the chunk text contains, which is why the splitter choice in section 3 matters here.
5. What poor preparation does
Two versions of the same collection, searched with the same question.
- Prepared badly: the metadata dropped, every document joined into one long text, and cut into fixed windows.
- Prepared well: what sections 1 to 4 built, with metadata and one chunk per section.
Each version is kept in a vector store: a place that holds the chunks together with their numbers, and can find the ones closest to a question. The question needs two figures, Kenya's latest and Rwanda's latest. The cell shows the top three chunks each search returns, and marks which of the two figures turn up in them.
from langchain_core.documents import Document
blob = Document(page_content="\n\n".join(d.page_content for d in docs), metadata={"id": "blob", "date": "", "section": "(fixed window)"})
poor = w.build_store(w.split_fixed([blob], chunk_size=400, chunk_overlap=50), embeddings)
good = w.build_store(chunks, embeddings)
q = "What are the latest available figures for internet use in Kenya and Rwanda?"
expected = next(item for item in w.load_questions() if item["id"] == "q8")["expected_numbers"]
countries = {"KEN": "Kenya", "RWA": "Rwanda"}
checks = [f"{countries[e['country']]}, {e['year']}: {e['value']}" for e in expected]
def search(store, title, tone):
hits = store.similarity_search(q, k=3)
text = " ".join(h.page_content for h in hits)
return {"title": title, "tone": tone, "hits": [(w.citation(h), h.page_content) for h in hits],
"found": [w.numbers_present(text, [e])[0] == 1 for e in expected]}
w.show_search_comparison(q, [search(poor, "Prepared badly", "bad"), search(good, "Prepared well", "good")], checks)
def numbers_found(store):
found = total = 0
for item in w.load_questions():
text = " ".join(h.page_content for h in store.similarity_search(item["question"], k=3))
for e in item.get("expected_numbers", []):
found += w.numbers_present(text, [e])[0]
total += 1
return found, total
for name, store in (("Prepared badly", poor), ("Prepared well", good)):
found, total = numbers_found(store)
print(f"{name}: {found} of {total} needed numbers appear in the top 3 chunks, across the eight evaluation questions")
Prepared badly: 5 of 9 needed numbers appear in the top 3 chunks, across the eight evaluation questions
Prepared well: 7 of 9 needed numbers appear in the top 3 chunks, across the eight evaluation questions
Prepared badly, the search does not find Kenya's latest figure. Its top three chunks are the 2016 summary, Rwanda's internet paragraph, and a window that starts in the last section of Rwanda's brief and runs on into the title of the next document. Every chunk is labelled blob, so a reader cannot tell where any of it came from or how old it is. Prepared well, the same question finds both figures, and each chunk names its document, section, and year. The 2016 summary comes first in both, because a search matches meaning, not age; its date is what the next notebook uses to deal with it.
The second cell counts the same thing across all eight evaluation questions. The gap is smaller there, because for several questions both versions find what they need.
6. What a vector store holds
For each chunk, a vector store holds three things: its text, its metadata, and its numbers. The cell opens one record of the prepared collection, the Mobile subscriptions section of the Kenya brief.
store = good # the prepared collection of section 5
print(len(store.store), "chunks in the vector store. One of them:\n")
record = next(r for r in store.store.values() if r["metadata"]["id"] == "kenya-digital" and r["metadata"]["section"] == "Mobile subscriptions")
print("text: ", record["text"][:90].replace("\n", " ") + " …")
print("metadata:", {k: record["metadata"][k] for k in ("id", "section", "country", "date")})
print("numbers: ", [round(x, 3) for x in record["vector"][:6]], "… (384 in all)")
21 chunks in the vector store. One of them:
text: ## Mobile subscriptions Mobile cellular subscriptions per 100 people stood at 60.0 in 20 …
metadata: {'id': 'kenya-digital', 'section': 'Mobile subscriptions', 'country': 'Kenya', 'date': '2026-10-13'}
numbers: [-0.036, 0.013, 0.046, -0.004, -0.012, 0.036] … (384 in all)
This store lives in memory and disappears when the notebook closes, which is enough here. A real application keeps its store in a database, so that it lasts and can be shared with others. Notebook 03 builds the same store and searches it to answer questions.
Key points
- Metadata is information about a document that its text does not say: date, type, country, source. The 2016 summary sounds current; its date, kept as metadata, is how an application can tell that it is old. An application can use the date only if it kept it.
- Chunk boundaries decide what retrieval can return. Fixed windows cut by length and can separate a heading from its paragraph; section chunks keep the heading with its meaning.
- Embeddings place text by meaning, so a question lands near the chunks that answer it. They know nothing about dates.
- A collection prepared badly returns chunks that cannot be cited or dated, and can miss the figures a question needs. Prepared well, each hit names its document, section, and year, and the figures turn up.
- Preparation decides what retrieval can find. The next notebook uses the collection to answer questions, and shows where it still goes wrong.
Exercise
- Look at the table in section 1. If someone asked for the latest figures on internet use, which document would you be wary of, and which two are about something else? Say what in the table told you.
- In the cell of section 3, change
chunk_size=400to200and run it again. How many fixed windows are there now, and where do they cut? Try800. The section chunks do not change, because they follow the headings.
Where this connects to the program
- Practical Guide to Serving Official Statistics to AI Agents: the design of tools over a statistical catalog, in full.
- Practical Guide to Evaluation Suites for Statistical AI: how to build and run the tests.
- Proof-Carrying Numbers: the claim-tag protocol the verifier in this session follows. See also the paper and the
pcnrepository. - Data360 MCP server: the tools of notebook 04, exposed to any MCP client over the World Bank's data platform.
- End-to-end reference implementation: documentation, validation, publication, agent access, and verification in one run.