02 · Tagging reports with topics
Open in Colab View the notebook on GitHub
The outputs on this page are the ones committed with the notebook. Without an API key, the notebook replays recorded model answers, and the same recordings appear here.
Session 02 · Building specialized small AI using synthetic data · notebook 2 of 2
Objective
A catalogue of reports assigns topics to each report, so that a reader can find every report on water or on labor markets. A specialist who assigns the topics by hand reads each report, and a large collection holds hundreds of thousands of reports. This notebook builds a small model that reads the abstract of a report and assigns up to five of 86 topics.
The model is trained without a single labelled report. A large language model wrote 20,296 sentences of the kind found in development reports and listed the topics of each sentence. The small model learns from those sentences. Real abstracts with the topics of the catalogue enter at two later points: 488 of them set one number, the threshold, and 300 others test the result.
Contents
- The task and the topics loads the 86 topics.
- Sentences written by a large model shows one request and the sentences that came back.
- Checks on the sentences counts repeated sentences, topics outside the list, and the sentences of each topic.
- Training the small model fine-tunes a pre-trained model on the sentences.
- From sentences to a report scores each sentence of an abstract and takes the mean.
- The threshold is chosen on 488 real abstracts.
- The test on real abstracts measures the model on 300 abstracts and compares it with large models.
- Size and speed measures the model and the time for one abstract.
- Rounds of improvement and limits describes the steps that follow a first model.
How to run it
Run the cells in order. The training cell needs a GPU. In Colab, the menu Runtime → Change runtime type offers a T4 GPU at no cost. Without a GPU the notebook trains on a sample of the examples, and the scores are then lower than the recorded ones.
No API key is needed. The examples that the large model wrote are stored in the repository, and the notebook reads them from there. One optional cell sends a new request to a large model when a key is set.
The next cell prepares the environment and reports the device. The first run in Colab installs the packages and takes longer than a later run. The cell is setup only.
# Setup. In Colab this clones the repository and installs the packages; locally it uses the folder as is.
import os, sys, subprocess, pathlib, warnings
REPO_URL = "https://github.com/worldbank/ai4data.git"
BRANCH = "main" # the branch that the Colab setup clones
SESSION = "training/session-02-small-ai-synthetic-data/notebooks"
if not pathlib.Path("visuals.py").exists():
subprocess.run(["git", "clone", "-q", "--depth", "1", "--branch", BRANCH, REPO_URL, "/content/ai4data"], check=True)
os.chdir(f"/content/ai4data/{SESSION}")
subprocess.run([sys.executable, "-m", "pip", "install", "-q", "-r", "requirements.txt"], check=True)
sys.path.insert(0, os.getcwd())
# Optional: a key for an OpenAI-compatible API, used by one cell that writes new examples. The notebook runs without it.
try:
from google.colab import userdata # type: ignore
for name in ("WORKSHOP_API_KEY", "WORKSHOP_BASE_URL", "WORKSHOP_MODEL"):
if userdata.get(name):
os.environ[name] = userdata.get(name)
except Exception:
pass
os.environ["HF_HUB_DISABLE_PROGRESS_BARS"] = "1" # keeps the download messages out of the outputs
warnings.filterwarnings("ignore")
import datasets, torch, transformers
import visuals as v
datasets.disable_progress_bars()
transformers.logging.set_verbosity_error()
ASSETS = pathlib.Path("../assets")
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
print("Device:", torch.cuda.get_device_name(0) if DEVICE == "cuda" else "CPU. No GPU was found, so the notebook trains on a sample.")
print("transformers", transformers.__version__, "| torch", torch.__version__.split("+")[0])
Device: NVIDIA H100 NVL
transformers 4.57.3 | torch 2.9.1
The notebook uses four widely used libraries and one small helper file.
| Need | Library |
|---|---|
| Tables | pandas |
| The pre-trained model, its tokenizer, and the training loop | Transformers |
| Examples in the form that the training loop reads | Datasets |
| Scores | scikit-learn |
visuals.py draws the pictures from the results of the cells. It holds no part of the pipeline.
1. The task and the topics
The task has a fixed list of answers: the 86 topics of the catalogue. A report can have several topics, and a topic can have a parent topic. The next cell loads the list and shows eight rows of it.
import pandas as pd
topics = pd.read_csv(ASSETS / "topics/topics.csv")
TOPICS = topics["topic"].tolist()
print(len(TOPICS), "topics.", int(topics["parent"].notna().sum()), "of them have a parent topic.")
topics.sample(8, random_state=1).fillna("")
86 topics. 60 of them have a parent topic.
| topic | parent | |
|---|---|---|
| 38 | Indigenous Peoples | Social Sustainability and Inclusion |
| 44 | Jobs and Development | Social Protection |
| 27 | Forests and Landscapes | Environment |
| 40 | Infectious diseases and Vaccines | Health |
| 36 | Higher Education | Education |
| 81 | Urban Development | |
| 58 | Pandemic Preparedness and COVID-19 | Health |
| 69 | Small and Medium Enterprises Finance | Financial Inclusion |
A task of this kind is called multi-label classification: for each topic, the model answers whether the topic applies to the text. The model therefore gives 86 probabilities for a text, one for each topic.
2. Sentences written by a large model
Training needs texts whose topics are known. The organization in this example has no such texts, so a large language model writes them. One request names eight topics drawn at random and asks for 20 sentences, each with the list of its topics. The picture shows one request and four of the sentences that came back.
The next cell prints the text of that request. Its rules ask for varied lengths and styles, because a model trained on uniform sentences handles real text poorly. They also ask that a third of the sentences cover two or three topics, because real reports do.
import json
overview = json.loads((ASSETS / "overview.json").read_text(encoding="utf-8"))
PROMPT = overview["synthetic"]["prompt"]["allow"]
print(PROMPT)
Write 20 different sentences of the kind found in reports, working papers, policy notes, and news about economic and social development.
Each sentence must be about one or more of these topics, and its "labels" must list every topic from this list that the sentence is about:
- Biodiversity
- Community-Driven Development
- Disability Inclusion
- Financial Sector
- Labor Markets
- Migration
- Trade Facilitation and Logistics
- Urban Development
Rules:
- Use the words a report on the subject would use, including the topic names or their key words where an author would use them.
- Vary the length from 10 to 45 words, and vary the style: findings, policy statements, project descriptions, statistics, questions a study asks.
- Mention places, institutions, and numbers only as an author would; invent nothing that reads as a quotation of a real document.
- About a third of the sentences should be about two or three of the topics at once.
A thousand requests of this form were sent to an open large model (Qwen3.8-27B) on a server, and the replies were stored in the repository. The next cell loads them and prints six sentences of the request shown above.
sentences = pd.read_json(ASSETS / "topics/sentences.jsonl.gz", lines=True)
print(f"{len(sentences):,} sentences from {sentences['request'].nunique():,} requests\n")
for row in sentences[sentences["request"] == overview["synthetic"]["request"]].head(6).itertuples():
print("-", row.text)
print(" topics:", ", ".join(row.topics))
20,296 sentences from 1,000 requests
- The study finds that enhanced trade facilitation measures in East Asia significantly reduced logistics costs for small and medium enterprises.
topics: Trade Facilitation and Logistics
- Urban development strategies in Nairobi must integrate biodiversity corridors to mitigate the adverse effects of rapid population growth.
topics: Biodiversity, Urban Development
- Financial sector reforms aimed at expanding access to microfinance have shown promising results in alleviating rural poverty.
topics: Financial Sector
- How does community-driven development impact local governance structures in post-conflict regions of the Middle East?
topics: Community-Driven Development
- Inclusive labor market policies are essential for ensuring that persons with disabilities have equal opportunities for employment and advancement.
topics: Disability Inclusion, Labor Markets
- The new migration compact emphasizes the need for secure pathways for legal migration while enhancing border security.
topics: Migration
The next cell is optional. With a key for an OpenAI-compatible API in WORKSHOP_API_KEY and a model name in WORKSHOP_MODEL, it sends the same request to that model and prints five of the new sentences. Without a key the cell prints a message and the notebook continues with the stored sentences.
if os.environ.get("WORKSHOP_API_KEY"):
from openai import OpenAI
client = OpenAI(api_key=os.environ["WORKSHOP_API_KEY"], base_url=os.environ.get("WORKSHOP_BASE_URL"))
form = '\n\nAnswer with JSON of this form: {"examples": [{"text": "...", "labels": ["..."]}]}'
reply = client.chat.completions.create(
model=os.environ["WORKSHOP_MODEL"],
messages=[{"role": "user", "content": PROMPT + form}],
response_format={"type": "json_object"},
temperature=1.0,
)
for item in json.loads(reply.choices[0].message.content)["examples"][:5]:
print("-", item["text"], "\n topics:", ", ".join(item["labels"]))
else:
print("No key is set. The notebook continues with the stored sentences.")
No key is set. The notebook continues with the stored sentences.
3. Checks on the sentences
A large model follows a request imperfectly. It can repeat a sentence, invent a topic outside the list, or return a sentence without a topic. A short script finds these cases before training. The next cell runs the three checks and then counts the sentences of each topic.
by_topic = sentences.explode("topics")
print("Repeated sentences: ", int(sentences["text"].duplicated().sum()))
print("Sentences without a topic: ", int((sentences["topics"].map(len) == 0).sum()))
print("Topics outside the list of 86:", int((~by_topic["topics"].isin(TOPICS)).sum()))
per_topic = by_topic["topics"].value_counts()
print(f"\nSentences per topic: {per_topic.min()} to {per_topic.max()}, median {int(per_topic.median())}")
print("Topics per sentence:", sentences["topics"].map(len).value_counts().sort_index().to_dict())
Repeated sentences: 0
Sentences without a topic: 0
Topics outside the list of 86: 0
Sentences per topic: 197 to 651, median 409
Topics per sentence: {1: 6508, 2: 12027, 3: 1735, 4: 25, 5: 1}
The three checks find nothing to remove in this set. Every topic has at least 197 sentences, because the requests drew the topics at random with equal chances. About a third of the sentences have one topic, and about six in ten have two.
The checks concern form only. A script cannot tell whether a sentence is really about its topics. For a task with one answer per text, a second large model can label each text again, and texts with two different answers are removed. Notebook 01 uses that check.
4. Training the small model
The small model starts from DeBERTa-v3-small, a model with 142 million parameters that was pre-trained on a large amount of English text. Pre-training taught it how words relate in a sentence. It knows nothing about the 86 topics. Fine-tuning continues the training on the sentences of section 3, with a new output layer of 86 values, one for each topic.
A model reads numbers, so a tokenizer first cuts each text into pieces called tokens and replaces each token with its number in the model's vocabulary. The next cell loads the tokenizer and shows the tokens of one short sentence.
from transformers import AutoTokenizer
BASE = "microsoft/deberta-v3-small"
tokenizer = AutoTokenizer.from_pretrained(BASE)
example = "Mobile money accounts raised household savings in rural districts."
encoded = tokenizer(example)
print(tokenizer.convert_ids_to_tokens(encoded["input_ids"]))
print(encoded["input_ids"])
['[CLS]', '▁Mobile', '▁money', '▁accounts', '▁raised', '▁household', '▁savings', '▁in', '▁rural', '▁districts', '.', '[SEP]']
[1, 3991, 580, 2681, 2107, 3913, 3632, 267, 4127, 7226, 260, 2]
The answer for a sentence also becomes numbers: a row of 86 values with 1 at the topics of the sentence and 0 elsewhere. The next cell defines the two conversions and sets one request in ten aside. The model never trains on the sentences set aside, so the loss on them shows whether it learns patterns that carry over to new sentences. The split is by request: all sentences of a request go to the same side, because sentences from one reply resemble each other.
import numpy as np
from datasets import Dataset
position = {topic: i for i, topic in enumerate(TOPICS)}
def targets(topic_lists):
"""One row per text and one column per topic: 1 where the topic applies, 0 elsewhere."""
table = np.zeros((len(topic_lists), len(TOPICS)), dtype=np.float32)
for row, names in enumerate(topic_lists):
table[row, [position[name] for name in names]] = 1
return table
def encode(frame):
data = Dataset.from_dict({"text": frame["text"].tolist(), "labels": targets(frame["topics"]).tolist()})
return data.map(lambda batch: tokenizer(batch["text"], truncation=True, max_length=128), batched=True)
requests = sentences["request"].drop_duplicates().sample(frac=1, random_state=0)
set_aside = set(requests[: len(requests) // 10])
train_part = sentences[~sentences["request"].isin(set_aside)]
check_part = sentences[sentences["request"].isin(set_aside)]
if DEVICE == "cpu": # a sample, to shorten training without a GPU
train_part, check_part = train_part.sample(2000, random_state=0), check_part.sample(300, random_state=0)
print(f"{len(train_part):,} sentences for training, {len(check_part):,} set aside")
18,296 sentences for training, 2,000 set aside
The next cell trains the model with the Trainer of the Transformers library. The settings are common defaults for fine-tuning a model of this size: four passes over the sentences (epochs), 32 sentences per step, and a learning rate of 0.00005 that rises during the first steps and then falls to zero. The recorded run below was made on one NVIDIA H100 GPU, and the last line of the output gives its duration. The cell takes longer on a Colab T4 GPU.
The table that the cell displays has one row per epoch. The training loss measures the distance between the model's answers and the right answers on the training sentences. The validation loss measures the same distance on the sentences set aside.
from transformers import AutoModelForSequenceClassification, DataCollatorWithPadding, Trainer, TrainingArguments, set_seed
set_seed(42)
model = AutoModelForSequenceClassification.from_pretrained(
BASE, num_labels=len(TOPICS), problem_type="multi_label_classification" # 86 independent yes-or-no answers
)
settings = TrainingArguments(
output_dir="topic-model",
num_train_epochs=4 if DEVICE == "cuda" else 1,
per_device_train_batch_size=32,
per_device_eval_batch_size=128,
learning_rate=5e-5,
weight_decay=0.01,
warmup_ratio=0.06,
eval_strategy="epoch",
logging_strategy="epoch",
save_strategy="no",
report_to="none",
disable_tqdm=False, # shows the progress bar and the table of losses
seed=42,
)
trainer = Trainer(
model=model,
args=settings,
train_dataset=encode(train_part),
eval_dataset=encode(check_part),
data_collator=DataCollatorWithPadding(tokenizer), # pads the sentences of a step to the same length
)
run = trainer.train()
print(f"Training took {run.metrics['train_runtime']:.0f} seconds.")
assert np.isfinite(run.training_loss), "The loss is not a number. See the note on library versions below."
| Epoch | Training Loss | Validation Loss |
|---|---|---|
| 1 | 0.169400 | 0.083050 |
| 2 | 0.067000 | 0.051847 |
| 3 | 0.046700 | 0.040381 |
| 4 | 0.038700 | 0.037225 |
Training took 57 seconds.
Both losses fall in every epoch, and the validation loss stays close to the training loss. The model therefore learned patterns that hold for sentences it had not seen. A validation loss that rises while the training loss falls would mean that the model memorizes its training sentences, and training should then stop earlier.
A note on library versions: requirements.txt asks for a version of Transformers below 5. In a test for this session, DeBERTa-v3-small trained normally with version 4.57 and returned a loss that was not a number with versions 5.20 and 5.22. The last line of the cell stops the notebook if that happens.
5. From sentences to a report
The model learned from single sentences, and an abstract has several. The next cell therefore splits an abstract into sentences, asks the model for the 86 probabilities of each sentence, and takes the mean over the sentences. A topic that several sentences support gets a high mean. A topic that one sentence mentions in passing gets a low one.
import re
import torch
BOUNDARY = re.compile(r"(?<=[.!?])\s+(?=[A-Z0-9\"'(])") # a sentence ends at . ! or ? followed by a capital letter or a digit
def split_sentences(text):
parts = [part.strip() for part in BOUNDARY.split(" ".join(text.split()))]
return [part for part in parts if len(part.split()) >= 4] or [text.strip()]
def sentence_scores(texts, batch=64):
"""The 86 probabilities of each text."""
model.eval()
out = []
with torch.inference_mode():
for start in range(0, len(texts), batch):
inputs = tokenizer(texts[start:start + batch], truncation=True, max_length=128, padding=True, return_tensors="pt").to(model.device)
out.append(torch.sigmoid(model(**inputs).logits).float().cpu().numpy())
return np.concatenate(out)
def report_scores(texts):
"""The 86 probabilities of each abstract: the mean over its sentences."""
return np.stack([sentence_scores(split_sentences(text)).mean(axis=0) for text in texts])
The reports for the threshold and for the test are abstracts of Policy Research Working Papers published from 2016 to 2020, taken from the World Bank's Documents and Reports database. The topics of each report come from the subjects that the database lists for it. The next cell loads both files and shows the path of one test abstract through the model. The abstract reviews studies of the way a change in a central bank's interest rate reaches the lending rates of banks. The picture is animated: the sentences appear with their most probable topic, and the bars then show the mean over the sentences for the six best topics.
threshold_reports = pd.read_json(ASSETS / "topics/reports-threshold.jsonl.gz", lines=True, dtype={"id": str})
test_reports = pd.read_json(ASSETS / "topics/reports-test.jsonl.gz", lines=True, dtype={"id": str})
print(len(threshold_reports), "abstracts for the threshold,", len(test_reports), "abstracts for the test")
report = test_reports.iloc[176]
parts = split_sentences(report["text"])
scores = sentence_scores(parts)
mean = scores.mean(axis=0)
best = np.argsort(-mean)[:6]
v.report_view(
report["title"],
parts,
tops=[(TOPICS[row.argmax()], float(row.max())) for row in scores],
means=[(TOPICS[i], float(mean[i])) for i in best],
threshold=0.05,
catalogue=report["topics"],
)
488 abstracts for the threshold, 300 abstracts for the test
The means are much lower than the probabilities of single sentences, because each sentence is about one or two topics and adds almost nothing to the other 84. A report therefore gets a topic when its mean passes a low threshold. The picture uses 0.05, and the next section explains where that number comes from.
In the recorded run the model assigns three topics to this abstract. Financial Sector and Macroeconomics are also in the catalogue record. Financial Stability is absent from the record, so the test counts it as an error, although a reader could defend it for a paper on bank lending rates. The record also lists Trade, which no sentence of the abstract mentions. A model that reads the abstract alone cannot find that topic.
6. The threshold
The threshold is the one setting that needs real reports with known topics. It is chosen on 488 abstracts that play no part in the test. For each threshold in a short list, the next cell assigns the topics whose mean is at or above the threshold, five at most, and compares them with the topics of the catalogue.
The comparison uses three scores. Precision is the share of assigned topics that the catalogue also lists. Recall is the share of the catalogue's topics that the model assigned. F1 combines the two, and it is high only when both are high. The scores are computed over all reports and topics together, which is called the micro average.
from sklearn.metrics import precision_recall_fscore_support
def assign(scores, threshold, at_most=5):
"""1 for each topic that a report gets: mean at or above the threshold, and among the five best."""
chosen = np.zeros_like(scores, dtype=int)
for row, order in enumerate(np.argsort(-scores, axis=1)[:, :at_most]):
chosen[row, order[scores[row, order] >= threshold]] = 1
return chosen
def evaluate(scores, reports, threshold):
truth = targets(reports["topics"]).astype(int)
measured = truth.sum(axis=0) > 0 # topics that occur in these reports
precision, recall, f1, _ = precision_recall_fscore_support(
truth[:, measured], assign(scores, threshold)[:, measured], average="micro", zero_division=0
)
return {"precision": precision, "recall": recall, "f1": f1, "topics measured": int(measured.sum())}
threshold_scores = report_scores(threshold_reports["text"])
GRID = (0.02, 0.03, 0.05, 0.075, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5)
f1_at = [evaluate(threshold_scores, threshold_reports, g)["f1"] for g in GRID]
THRESHOLD = GRID[int(np.argmax(f1_at))]
v.threshold_chart(GRID, f1_at, THRESHOLD)
The score is highest at a low threshold and falls quickly above 0.1. At the common default of 0.5 the model assigns almost no topic, because a mean over many sentences rarely reaches one half. A model trained on sentences and used on whole abstracts needs this calibration step, and a few hundred real cases are enough for it.
7. The test on real abstracts
The 300 test abstracts took no part in training or in the choice of the threshold. The next cell scores them once, with the threshold of section 6.
test_scores = report_scores(test_reports["text"])
result = evaluate(test_scores, test_reports, THRESHOLD)
print(f"Threshold {THRESHOLD:g}, at most 5 topics per report, {result['topics measured']} topics measured")
print(f"Precision {result['precision']:.3f} Recall {result['recall']:.3f} F1 {result['f1']:.3f}")
Threshold 0.05, at most 5 topics per report, 44 topics measured
Precision 0.534 Recall 0.402 F1 0.459
In the recorded run, about half of the assigned topics are in the catalogue (precision), and the model finds about four in ten of the catalogue's topics (recall). The catalogue can omit a topic that an abstract covers, and such a case counts as an error of the model. The 300 abstracts cover 44 of the 86 topics, so the test says nothing about the other 42.
The next cell places the result beside recorded results on the same 300 abstracts. Three large models received each abstract with the list of topics and were asked for the topics directly. The fourth recorded result is the small model after five rounds of improvement, which section 9 describes.
recorded = overview["workflow"]
rows = [("Small model of this notebook\ntrained on the first set of sentences", result["f1"], "small")]
after_rounds = sum(r["arms"][2]["f1"] for r in recorded["rounds"]) / len(recorded["rounds"])
rows += [("Small model after five further rounds\nrecorded, mean of two runs", after_rounds, "small")]
rows += [(f"{m['model']}\nlarge model asked directly, recorded", m["f1"], "large") for m in recorded["large"]]
v.result_bars("Micro F1 on the 300 test abstracts", rows, top_value=0.6)
In the recorded run, the small model of this notebook is about 0.07 below the best large model. After the recorded rounds of improvement the small model is 0.01 below it. A new run of this notebook can give a slightly different score, because the result of training depends on the GPU and on the versions of the libraries.
The overall score hides differences between topics. The next cell computes F1 for each topic with at least eight test abstracts and prints the five best and the five weakest.
truth = targets(test_reports["topics"]).astype(int)
assigned = assign(test_scores, THRESHOLD)
_, _, f1_topic, support = precision_recall_fscore_support(truth, assigned, average=None, zero_division=0)
table = pd.DataFrame({"topic": TOPICS, "test abstracts": support, "assigned by the model": assigned.sum(axis=0), "F1": f1_topic.round(2)})
table = table[table["test abstracts"] >= 8].sort_values("F1", ascending=False)
pd.concat([table.head(5), table.tail(5)]).set_index("topic")
| test abstracts | assigned by the model | F1 | |
|---|---|---|---|
| topic | |||
| Teachers | 11 | 10 | 0.86 |
| Climate Change | 10 | 7 | 0.71 |
| Agriculture and Food | 35 | 25 | 0.70 |
| Disaster Risk Management | 14 | 12 | 0.69 |
| Financial Sector | 30 | 38 | 0.65 |
| Urban Development | 13 | 5 | 0.33 |
| Inequality and Shared Prosperity | 72 | 24 | 0.29 |
| Governance | 39 | 25 | 0.28 |
| Fragility, Conflict, and Violence | 17 | 57 | 0.27 |
| Water Resources Management | 19 | 1 | 0.10 |
The weakest topics of the recorded run show two kinds of problem. The model assigns Fragility, Conflict, and Violence to 57 abstracts, and the catalogue lists it for 17. Synthetic sentences that describe a topic too broadly produce this pattern. The model assigns Water Resources Management to one abstract, and the catalogue lists it for 19. This second pattern appears when real abstracts use wording that is absent from the synthetic sentences. Both findings tell the team which topics need more sentences or better ones.
8. Size and speed
The reasons for building a small model are practical: it fits on an ordinary computer, and it answers quickly. The next cell counts the parameters of the trained model, computes the memory that they need, and times the model on the 300 test abstracts.
import time
parameters = sum(p.numel() for p in model.parameters())
print(f"{parameters / 1e6:.0f} million parameters, {parameters * 4 / 1e9:.2f} GB at 4 bytes per parameter")
start = time.perf_counter()
report_scores(test_reports["text"])
seconds = (time.perf_counter() - start) / len(test_reports)
print(f"{seconds * 1000:.1f} milliseconds per abstract on this device ({DEVICE})")
142 million parameters, 0.57 GB at 4 bytes per parameter
7.0 milliseconds per abstract on this device (cuda)
A large model of 27 billion parameters needs about 31 GB of memory in an 8-bit version and a GPU of that size. The model of this notebook needs about half a gigabyte and also runs on a laptop without a GPU. In a recorded measurement on four CPU threads, it took about 0.2 seconds per abstract.
9. Rounds of improvement and limits
The model of this notebook is a first version. The process of the session continues with rounds of improvement.
In two recorded runs, five rounds raised the F1 score on the test abstracts from 0.47 to 0.52 on average. Each round tested the model on real abstracts kept for that purpose, selected the topics with the lowest scores, and had the large model write 3,000 further sentences for them. A control added the same number of sentences for topics drawn at random and reached 0.48. The gain therefore came from the choice of topics. The larger number of sentences alone added about 0.01.
These results have four limits.
- The right answers are the topics of a catalogue record. A record can omit a topic that the abstract covers.
- The test covers 44 of the 86 topics.
- All abstracts are working papers in English from one series. Other kinds of report and other languages need their own test.
- The model assigns topics from the abstract only. It has no information on the rest of the report.
Before such a model enters use, people who know the catalogue should compare its topics with their own on a sample of reports, and the test should be repeated when new kinds of reports arrive.
Notebook 01 builds a model for a second task, coding the answers to a survey question on occupation, and adds a check by a second large model.