Skip to main content

01 · Coding survey answers on occupation

Open in Colab View the notebook on GitHub

The outputs on this page are the ones committed with the notebook. Without an API key, the notebook replays recorded model answers, and the same recordings appear here.

Session 02 · Building specialized small AI using synthetic data · notebook 1 of 2

Objective​

A labour force survey asks each respondent about the main job, and the interviewer writes the answer down in the respondent's words. A coder then assigns each answer a code of the International Standard Classification of Occupations (ISCO-08), which has 436 detailed groups. This notebook builds a small model that reads an answer and suggests five groups, so that the coder chooses from a short list.

The organization in this example has no survey answers with codes. It has 10,134 job titles from job advertisements, each with a code. An advertisement words a job differently from a respondent. A large language model therefore rewrites each title as three answers of the kind a respondent gives, and each answer keeps the code of its title. A second large model checks the answers.

The advertisements cover 309 of the 436 groups. For the full list of groups, a large model also wrote job titles from the definitions of the classification. The small model learns from the three sets of texts.

Stages of the occupation-coding pipeline: labelled job titles, answers written by a large model, a check by a second model, titles written from the definitions for the groups without a job title, training of the small model, tests.

Contents​

  1. The task and the classification loads the groups of ISCO-08.
  2. Labelled job titles from another source loads the job titles and counts them by major group.
  3. Survey-style answers written by a large model shows the request and the answers that came back.
  4. The check by a second model removes the answers that a second model codes differently.
  5. Groups without a job title adds titles that a large model wrote from the definitions.
  6. Training the small model fine-tunes a pre-trained model on the three sets.
  7. Three tests measure the model and compare it with recorded results for other training sets.
  8. Suggestions for one answer shows the five groups that the model suggests for a typed answer.
  9. Limits and rounds of improvement lists the questions that the tests leave open.

How to run it​

Run the cells in order. The training cell needs a GPU. In Colab, the menu Runtime → Change runtime type offers a T4 GPU at no cost. Without a GPU the notebook trains on a sample of the examples, and the scores are then lower than the recorded ones.

No API key is needed. The examples that the large model wrote are stored in the repository, and the notebook reads them from there. One optional cell sends a new request to a large model when a key is set.

The next cell prepares the environment and reports the device. The first run in Colab installs the packages and takes longer than a later run. The cell is setup only.

# Setup. In Colab this clones the repository and installs the packages; locally it uses the folder as is.
import os, sys, subprocess, pathlib, warnings

REPO_URL = "https://github.com/worldbank/ai4data.git"
BRANCH = "main" # the branch that the Colab setup clones
SESSION = "training/session-02-small-ai-synthetic-data/notebooks"

if not pathlib.Path("visuals.py").exists():
subprocess.run(["git", "clone", "-q", "--depth", "1", "--branch", BRANCH, REPO_URL, "/content/ai4data"], check=True)
os.chdir(f"/content/ai4data/{SESSION}")
subprocess.run([sys.executable, "-m", "pip", "install", "-q", "-r", "requirements.txt"], check=True)
sys.path.insert(0, os.getcwd())

# Optional: a key for an OpenAI-compatible API, used by one cell that writes new examples. The notebook runs without it.
try:
from google.colab import userdata # type: ignore
for name in ("WORKSHOP_API_KEY", "WORKSHOP_BASE_URL", "WORKSHOP_MODEL"):
if userdata.get(name):
os.environ[name] = userdata.get(name)
except Exception:
pass

os.environ["HF_HUB_DISABLE_PROGRESS_BARS"] = "1" # keeps the download messages out of the outputs
warnings.filterwarnings("ignore")

import datasets, torch, transformers
import visuals as v

datasets.disable_progress_bars()
transformers.logging.set_verbosity_error()

ASSETS = pathlib.Path("../assets")
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
print("Device:", torch.cuda.get_device_name(0) if DEVICE == "cuda" else "CPU. No GPU was found, so the notebook trains on a sample.")
print("transformers", transformers.__version__, "| torch", torch.__version__.split("+")[0])
Device: NVIDIA H100 NVL
transformers 4.57.3 | torch 2.9.1

The notebook uses four widely used libraries and one small helper file.

NeedLibrary
Tablespandas
The pre-trained model, its tokenizer, and the training loopTransformers
Examples in the form that the training loop readsDatasets
The download of the classificationRequests

visuals.py draws the pictures from the results of the cells. It holds no part of the pipeline.

1. The task and the classification​

ISCO-08 arranges occupations on four levels. Ten major groups divide into 43 sub-major groups, 130 minor groups, and 436 unit groups. A code has one digit for each level, so the unit group 5212 lies in major group 5, sub-major group 52, and minor group 521. A statistical office codes each survey answer to a unit group.

The unit group 5212 Street food salespersons under minor group 521, sub-major group 52, and major group 5. The four levels of the classification have 10, 43, 130, and 436 groups.

The next cell downloads the classification from the website of the International Labour Organization and keeps the name of each group. The notebook uses the names to display results. If the download fails, the notebook continues and shows codes without names.

import io

import pandas as pd
import requests

ISCO_URL = "https://webapps.ilo.org/ilostat-files/ISCO/newdocs-08-2021/ISCO-08/ISCO-08%20EN%20Structure%20and%20definitions.xlsx"
LEVELS = {"1": "major groups", "2": "sub-major groups", "3": "minor groups", "4": "unit groups"}

try:
reply = requests.get(ISCO_URL, headers={"User-Agent": "Mozilla/5.0"}, timeout=60)
reply.raise_for_status()
isco = pd.read_excel(io.BytesIO(reply.content), dtype=str)
isco = isco.rename(columns={"Level": "level", "ISCO 08 Code": "code", "Title EN": "name"})[["level", "code", "name"]]
print(", ".join(f"{count} {LEVELS[level]}" for level, count in isco["level"].value_counts().sort_index().items()))
except Exception as error:
print("The classification could not be downloaded. Results show codes without names.", error)
isco = pd.DataFrame(columns=["level", "code", "name"])

NAMES = dict(zip(isco["code"], isco["name"]))


def group_name(code):
return NAMES.get(code, "")


isco[isco["code"].str.startswith("52")].set_index("code")
10 major groups, 43 sub-major groups, 130 minor groups, 436 unit groups
levelname
code
522Sales Workers
5213Street and Market Salespersons
52114Stall and Market Salespersons
52124Street Food Salespersons
5223Shop Aalespersons
52214Shopkeepers
52224Shop Supervisors
52234Shop Sales Assistants
5233Cashiers and Ticket Clerks
52304Cashiers and Ticket Clerks
5243Other Sales Workers
52414Fashion and Other Models
52424Sales Demonstrators
52434Door-to-door Salespersons
52444Contact Centre Salespersons
52454Service Station Attendants
52464Food Service Counter Attendants
52494Sales Workers Not Elsewhere Classified

The table shows the part of the classification under sub-major group 52, Sales Workers. Neighbouring unit groups differ in details that a short answer can leave out. The answer "sells food" fits a street food salesperson (5212), a stall salesperson at a market (5211), and a shop assistant (5223). A list of five suggestions suits this situation: the coder sees the candidates side by side and chooses with the rest of the questionnaire at hand.

2. Labelled job titles from another source​

Training needs texts whose code is known, and the organization in this example has no coded survey answers. A public dataset supplies a related kind of text: job titles from the job board hahu.jobs in Ethiopia, published by the Tabiya project under the CC BY 4.0 licence. The job board assigned each title an occupation of the European classification ESCO, and the first four digits of an ESCO code are the ISCO-08 unit group. No trained coder reviewed these codes.

The next cell loads the titles kept for training and shows ten of them with their groups.

titles = pd.read_json(ASSETS / "occupation/titles.jsonl.gz", lines=True, dtype=str)
print(f"{len(titles):,} job titles in {titles['code'].nunique()} of the 436 unit groups")

sample = titles.sample(10, random_state=3)
sample.assign(group=sample["code"].map(group_name))[["title", "code", "group"]].set_index("title")
10,134 job titles in 309 of the 436 unit groups
codegroup
title
Assistance Teachers5312Teachers' Aides
Senior Education Technical Advisor2359Teaching Professionals Not Elsewhere Classified
Career and Technical Education2320Vocational Education Teachers
HIV Service Officer & Care and Support Service Officer3253Community Health Workers
Software front end engineer2512Software Developers
District System Administrator2522Systems Aministrators
Regional Sales Manager I1221Sales and Marketing Managers
Senior Project Officer I – Cooperative Based Seed Production (CBSP)3343Administrative and Executive Secretaries
Call Center Officer/Representative5244Contact Centre Salespersons
Marketing promotion and Branding Officer IV2431Advertising and Marketing Professionals

A title in an advertisement names a position, often with a grade, a department, or a note from the employer. A respondent describes the work: "teacher at the primary school", "sell vegetables at the market". A model trained on titles alone has seen few texts of the second kind.

The titles also cover the labour market unevenly. The next cell counts them by major group.

by_major = titles["code"].str[0].value_counts().sort_index()
v.result_bars(
"Job titles by major group",
[(f"{digit} · {group_name(digit) or 'major group ' + digit}", count, "muted") for digit, count in by_major.items()],
fmt="{:,.0f}", label_chars=54,
)
Job titles by major group1 · Managers2,1212 · Professionals4,4033 · Technicians and Associate Professionals1,9364 · Clerical Support Workers6885 · Service and Sales Workers3816 · Skilled Agricultural, Forestry and Fishery Workers177 · Craft and Related Trades Workers2888 · Plant and Machine Operators, and Assemblers2209 · Elementary Occupations80

Major groups 1, 2, and 3 (managers, professionals, and technicians) hold 83% of the titles. Major group 6 (skilled agricultural, forestry, and fishery workers) has 17 titles, and major group 9 (elementary occupations) has 80. A job board advertises salaried positions. A household survey also records farmers, street vendors, and domestic workers, and the job board carries few advertisements for such work. Sections 3 and 4 change the wording of the titles and keep their groups. Section 5 adds the groups that are absent.

3. Survey-style answers written by a large model​

The large model receives one job title with the name and the definition of its group. It writes three answers that a respondent with this job could give: a short answer of one to four words, an answer that adds the place of work, and an answer that describes the tasks without the job title. Each answer takes the code of its title, so no person codes the new texts.

One job title with its code and three answers written from it by the large model. Each answer keeps the code of the title.

The next cell prints the request. Its rules ask for the plain words of a respondent and exclude a copy of the title.

import json

results = json.loads((ASSETS / "occupation/results.json").read_text(encoding="utf-8"))
PROMPT = results["prompt"]
print(PROMPT)
A household survey asks two questions about work: "What is your main job?" and "What are your main tasks or duties in this job?". The interviewer writes the answer down in the respondent's own words.

A respondent holds this job:
Job title in a job advertisement: {title}
Occupation group: {label}

Write three different answers that this respondent could give.
- Answer 1: a short answer of one to four words, the way many respondents answer.
- Answer 2: the job together with the place of work or what the employer does, in four to twelve words.
- Answer 3: the main tasks in everyday words, in six to twenty words, without the job title.

Rules:
- Use the plain words of a respondent, and avoid the wording of a job advertisement. Do not copy the job title as it is written.
- Write the way an interviewer records an answer: mostly lower case, short, sometimes an incomplete sentence, in the first person or with no subject.
- Leave out the seniority grade, the company name, and anything a respondent would not say.
- Each answer must describe this job, and must fit this occupation group better than any related group.

The request was sent once for each title to an open large model (Qwen3.8-27B) on a server, and the replies are stored in the repository. The next cell loads them and prints the answers for three titles. Each line ends with the result of the check that section 4 describes.

answers = pd.read_json(ASSETS / "occupation/answers.jsonl.gz", lines=True, dtype={"id": str, "code": str, "second_code": str})
print(f"{len(answers):,} answers for {answers['id'].nunique():,} job titles\n")


def show_answers(title):
group = answers[answers["title"] == title]
code = group["code"].iloc[0]
print(f"{title} [{code} {group_name(code)}]")
for row in group.itertuples():
print(f" {row.style}. {row.answer} ({'kept' if row.kept else 'removed'} in section 4)")
print()


for title in ["Pediatric Nurse & NICU", "Security Officers", "Construction Surveyor"]:
show_answers(title)
30,299 answers for 10,105 job titles

Pediatric Nurse & NICU [2221 Nursing Professionals]
1. nurse (kept in section 4)
2. nurse at the children's hospital (kept in section 4)
3. checking on sick babies and giving them medicine (kept in section 4)

Security Officers [5414 Security Guards]
1. security guard (kept in section 4)
2. guard at the shopping mall parking lot (kept in section 4)
3. checking badges and walking rounds to stop people from stealing (kept in section 4)

Construction Surveyor [2165 Cartographers and Surveyors]
1. measuring the land (kept in section 4)
2. working out the exact positions of boundaries on site (kept in section 4)
3. using instruments to take measurements and map out the boundaries of the land and buildings (kept in section 4)

The next cell is optional. With a key for an OpenAI-compatible API in WORKSHOP_API_KEY and a model name in WORKSHOP_MODEL, it sends the request for one job title to that model and prints the three answers. Without a key the cell prints a message and the notebook continues with the stored answers.

if os.environ.get("WORKSHOP_API_KEY"):
from openai import OpenAI

client = OpenAI(api_key=os.environ["WORKSHOP_API_KEY"], base_url=os.environ.get("WORKSHOP_BASE_URL"))
one = titles.iloc[0]
# The recorded run also sent the definition of the group. The name alone is enough for a trial.
request = PROMPT.format(title=one["title"], label=f"{one['code']} {group_name(one['code'])}")
reply = client.chat.completions.create(
model=os.environ["WORKSHOP_MODEL"],
messages=[{"role": "user", "content": request + '\n\nAnswer with JSON of this form: {"answers": ["...", "...", "..."]}'}],
response_format={"type": "json_object"},
temperature=1.0,
)
print(one["title"])
for answer in json.loads(reply.choices[0].message.content)["answers"]:
print(" -", answer)
else:
print("No key is set. The notebook continues with the stored answers.")
No key is set. The notebook continues with the stored answers.

4. The check by a second model​

An answer can move away from the code of its title. A short answer such as "driver" fits several groups, and the writer can describe a neighbouring occupation. A second large model (Gemma 4 31B) therefore reads each answer without the title and without the code. It chooses among the group of the title, the groups next to it in the classification, groups drawn at random, and the reply that none of the listed groups fits. An answer is kept when the second model chooses the group of the title.

Four answers with the code of their job title and the code chosen by a second model. Answers with two different codes are removed.

The next cell counts the answers kept and removed for each of the three kinds of answer.

KINDS = {1: "Short answer", 2: "Job and place of work", 3: "Tasks in everyday words"}

counts = answers.groupby("style")["kept"].agg(["sum", "size"])
v.kept_removed([(KINDS[kind], int(row["sum"]), int(row["size"] - row["sum"])) for kind, row in counts.iterrows()])
print(f"{int(answers['kept'].sum()):,} of {len(answers):,} answers are kept.")
Answers kept and removed by the second model's checkShort answer6,304 kept, 3,785 removed (62% kept)Job and place of work7,878 kept, 2,227 removed (78% kept)Tasks in everyday words8,086 kept, 2,019 removed (80% kept)keptremoved
22,268 of 30,299 answers are kept.

The check keeps 62% of the short answers, 78% of the answers with a place of work, and 80% of the answers that describe tasks. A short answer carries the least information, so the second model more often replies that none of the listed groups fits. The next cell prints six removed answers with the group that the second model chose.

for row in answers[~answers["kept"]].sample(6, random_state=2).itertuples():
second = f"{row.second_code} {group_name(row.second_code)}" if row.second_code else "none of the listed groups"
print(f"“{row.answer}” from the job title “{row.title}”")
print(f" group of the title: {row.code} {group_name(row.code)}")
print(f" second model: {second}\n")
“secretary” from the job title “Administrative Assistant & Secretary”
group of the title: 3343 Administrative and Executive Secretaries
second model: none of the listed groups

“content development” from the job title “Graduate Trainee/Intern Content Development, Graphics Design, & Multimedia Support (ECCE”
group of the title: 2432 Public Relations Professionals
second model: none of the listed groups

“i back up the data and make sure the system runs smooth” from the job title “Database Administrator-I”
group of the title: 2521 Database Designers and Administrators
second model: 2522 Systems Aministrators

“managing the health program for the district hospital” from the job title “Field Health Program Manager”
group of the title: 1213 Policy and Planning Managers
second model: none of the listed groups

“meal officer” from the job title “Senior Meal Officer”
group of the title: 2422 Policy Administration Professionals
second model: none of the listed groups

“checking how projects are working and writing reports on what we learn” from the job title “Senior Monitoring, Evaluation, Accountability, and learning (MEAL) Officer”
group of the title: 2422 Policy Administration Professionals
second model: 2421 Management and Organization Analysts

The removed answers are of three kinds. Some describe a neighbouring occupation. Some are correct and too short to identify one group among 436. For others the code of the job board is doubtful, and the second model disagrees with the code more than with the answer. The check cannot separate these cases. It removes all three, and the set that remains is smaller and more consistent.

The check compares an answer with the code and never reads the job title. The next cell prints the answers for a title whose code on the job board is doubtful.

show_answers("Quality Auditor")
Quality Auditor [3154 Air Traffic Controllers]
1. i control the planes (kept in section 4)
2. air traffic control at the airport (kept in section 4)
3. talk to pilots on the radio and watch the radar screen (kept in section 4)

The job board placed the title "Quality Auditor" in group 3154, Air Traffic Controllers. The large model wrote its answers from the name and the definition of the group, so they describe an air traffic controller, and the second model confirmed them. The three answers are correct texts for group 3154. The title keeps its doubtful code in the training set, because no step of the pipeline checks the codes of the job board.

5. Groups without a job title​

The job titles, and therefore the answers written from them, cover 309 of the 436 unit groups. A model can suggest only the groups of its training texts. The groups without a job title include 6310 Subsistence Crop Farmers and 5212 Street Food Salespersons, which a household survey records.

A large model can write texts for a group without any labelled example, because ISCO-08 publishes a definition of each group with its tasks and with examples of occupations. In a separate recorded run, a large model read the definition of each group and wrote job titles for it, and a second model checked them as in section 4. The next cell loads these titles and shows ten of them for groups that the job board lacks.

written = pd.read_json(ASSETS / "occupation/definition-titles.jsonl.gz", lines=True, dtype=str)
print(f"{len(written):,} titles written from the definitions, in {written['code'].nunique()} unit groups")

new_groups = written[~written["code"].isin(set(titles["code"]))]
sample = new_groups.sample(10, random_state=1)
sample.assign(group=sample["code"].map(group_name))[["title", "code", "group"]].set_index("title")
15,865 titles written from the definitions, in 436 unit groups
codegroup
title
Professional Palm Reader5161Astrologers, Fortune-tellers and Related Workers
Hockey Center3421Athletes and Sports Players
Astrophysics Researcher2111Physicists and Astronomers
Agricultural Weed Controller7544Fumigators and Other Pest and Weed Controllers
Field Scribe4414Scribes and Related Workers
Minister of Finance1111Legislators
Surgical Technologist2240Paramedical Practitioners
Compositor7321Pre-press Technicians
Iron Smith7221Blacksmiths, Hammersmiths and Forging Press Wo...
Factory Hand9329Manufacturing Labourers Not Elsewhere Classified
v.result_bars(
"Unit groups with at least one training text",
[
("Job titles and survey-style answers", titles["code"].nunique(), "muted"),
("With titles written from the definitions", pd.concat([titles["code"], written["code"]]).nunique(), "small"),
],
top_value=436, fmt="{:.0f}",
)
Unit groups with at least one training textJob titles and survey-style answers309With titles written from the definitions436

The job titles carry codes that an employer or a job board assigned to real advertisements. The survey-style answers add the wording of a respondent for the same groups. The titles written from the definitions add the remaining groups, in the wording of a job title. No set holds survey-style answers for the groups that the job board lacks, and section 9 returns to this gap.

6. Training the small model​

The training set joins the three sets. Each text has one right group. A task of this kind is called single-label classification: the model gives one probability for each group, and the probabilities sum to 1. The next cell builds the set and counts its texts by source.

kept = answers[answers["kept"]]
examples = pd.concat([
titles[["id", "title", "code"]].rename(columns={"title": "text"}).assign(source="job title"),
kept[["id", "answer", "code"]].rename(columns={"answer": "text"}).assign(source="survey-style answer"),
written[["id", "title", "code"]].rename(columns={"title": "text"}).assign(source="title written from a definition"),
]).drop_duplicates("text").reset_index(drop=True)

CODES = sorted(examples["code"].unique())
position = {code: i for i, code in enumerate(CODES)}
print(f"{len(examples):,} texts in {len(CODES)} unit groups")
print(examples["source"].value_counts().to_string())
44,058 texts in 436 unit groups
source
survey-style answer 18342
title written from a definition 15582
job title 10134

The small model starts from DeBERTa-v3-small, a model with 142 million parameters that was pre-trained on a large amount of English text. A tokenizer cuts each text into pieces called tokens and replaces each token with its number in the model's vocabulary. The next cell loads the tokenizer and shows the tokens of one answer. The mark ▁ at the start of a token stands for a space, and the tokens [CLS] and [SEP] mark the start and the end of the text.

from transformers import AutoTokenizer

BASE = "microsoft/deberta-v3-small"
tokenizer = AutoTokenizer.from_pretrained(BASE)

example = "sells roasted maize by the roadside"
print(tokenizer.convert_ids_to_tokens(tokenizer(example)["input_ids"]))
['[CLS]', '▁sells', '▁roasted', '▁maize', '▁by', '▁the', '▁roadside', '[SEP]']

The next cell sets one job title in ten aside, together with the answers written from it. The model never trains on these texts, so its results on them show whether it learns patterns that carry over to new texts. The split is by title, because an answer resembles the title that it was written from.

import numpy as np
from datasets import Dataset


def encode(frame):
data = Dataset.from_dict({"text": frame["text"].tolist(), "label": [position[code] for code in frame["code"]]})
return data.map(lambda batch: tokenizer(batch["text"], truncation=True, max_length=64), batched=True)


ids = examples["id"].drop_duplicates().sample(frac=1, random_state=0)
set_aside = set(ids[: len(ids) // 10])
train_part = examples[~examples["id"].isin(set_aside)]
check_part = examples[examples["id"].isin(set_aside)]
if DEVICE == "cpu": # a sample, to shorten training without a GPU
train_part, check_part = train_part.sample(3000, random_state=0), check_part.sample(300, random_state=0)
print(f"{len(train_part):,} texts for training, {len(check_part):,} set aside")
39,674 texts for training, 4,384 set aside

The next cell trains the model with the Trainer of the Transformers library: eight passes over the texts (epochs), 32 texts per step, and a learning rate of 0.00005 that rises during the first steps and then falls to zero. The function accuracy computes two measures on the texts set aside after each epoch. Top 1 is the share of texts whose right group is the model's first suggestion. Top 5 is the share whose right group is among the first five.

The recorded run below was made on one NVIDIA H100 GPU, and the last line of the output gives its duration. The cell takes longer on a Colab T4 GPU.

from transformers import AutoModelForSequenceClassification, DataCollatorWithPadding, Trainer, TrainingArguments, set_seed


def accuracy(prediction):
best_five = np.argsort(-prediction.predictions, axis=1)[:, :5]
right = best_five == prediction.label_ids[:, None]
return {"top 1": right[:, 0].mean(), "top 5": right.any(axis=1).mean()}


set_seed(42)
model = AutoModelForSequenceClassification.from_pretrained(BASE, num_labels=len(CODES))
settings = TrainingArguments(
output_dir="occupation-model",
num_train_epochs=8 if DEVICE == "cuda" else 1,
per_device_train_batch_size=32,
per_device_eval_batch_size=128,
learning_rate=5e-5,
weight_decay=0.01,
warmup_ratio=0.06,
eval_strategy="epoch",
logging_strategy="epoch",
save_strategy="no",
report_to="none",
disable_tqdm=False, # shows the progress bar and the table of results
seed=42,
)
trainer = Trainer(
model=model,
args=settings,
train_dataset=encode(train_part),
eval_dataset=encode(check_part),
data_collator=DataCollatorWithPadding(tokenizer), # pads the texts of a step to the same length
compute_metrics=accuracy,
)
run = trainer.train()
print(f"Training took {run.metrics['train_runtime']:.0f} seconds.")
assert np.isfinite(run.training_loss), "The loss is not a number. Check that the installed version of transformers is below 5."
[9920/9920 03:53, Epoch 8/8]
EpochTraining LossValidation LossTop 1Top 5
13.9664002.2530320.5098080.733349
21.9277001.6369050.6058390.838732
31.3197001.3299620.6626370.890967
40.9627001.1890100.7050640.914462
50.7410001.1218520.7351730.916743
60.5728001.1053800.7452100.924042
70.4560001.1094780.7493160.926095
80.3837001.1067030.7500000.927920

Training took 234 seconds.

The recorded run took about four minutes. On the texts set aside, the share with the right group as the first suggestion rose from 0.51 after the first epoch to 0.75 after the eighth. The share with the right group among five suggestions rose from 0.73 to 0.93. The validation loss stopped falling after the sixth epoch, so the run has enough epochs.

The texts set aside come from the same three sources as the training texts and resemble them. Section 7 tests the model on texts from other advertisements.

7. Three tests​

Three sets test the model. The first two come from 1,000 advertisements of the job board, and none of these advertisements took part in training or in the rewriting step. A job board repeats titles across advertisements: 90 of the 1,000 test titles also occur word for word among the training texts, 26 of them as the title of another advertisement and the others among the texts written by the large models. None of the 1,000 test answers occurs among the training texts. Of the 29 illustrative answers, 3 occur word for word among the training texts.

  • Job titles. 1,000 titles of the same job board, kept apart from the start.
  • Survey-style answers. The same 1,000 titles, each rewritten as one survey answer by a third large model (Gemini 3.8 Flash) with a different request. No check removed any of these answers.
  • Illustrative answers. 29 short answers composed for the slides of this session in the style of a household survey, each with an expected code. No trained coder reviewed them.

The next cell defines the measure and applies it to the three sets.

import torch


def probabilities(texts, batch=128):
"""One row per text and one column per group of CODES."""
model.eval()
out = []
with torch.inference_mode():
for start in range(0, len(texts), batch):
inputs = tokenizer(texts[start:start + batch], truncation=True, max_length=64, padding=True, return_tensors="pt").to(model.device)
out.append(torch.softmax(model(**inputs).logits.float(), dim=-1).cpu().numpy())
return np.concatenate(out)


def top(texts, codes, sizes=(1, 3, 5)):
"""The share of texts whose right code is among the first 1, 3, and 5 suggestions."""
order = np.argsort(-probabilities(list(texts)), axis=1)[:, : max(sizes)]
right = np.array(CODES)[order] == np.array(list(codes))[:, None]
return {f"top {size}": round(float(right[:, :size].any(axis=1).mean()), 3) for size in sizes}


test_titles = pd.read_json(ASSETS / "occupation/test-titles.jsonl.gz", lines=True, dtype=str)
test_answers = pd.read_json(ASSETS / "occupation/test-answers.jsonl.gz", lines=True, dtype=str)
illustrative = pd.read_json(ASSETS / "occupation/illustrative-answers.json", dtype=str)

scores = {
"job titles": top(test_titles["title"], test_titles["code"]),
"survey-style answers": top(test_answers["answer"], test_answers["code"]),
"illustrative answers": top(illustrative["answer"], illustrative["code"]),
}
pd.DataFrame(scores).T.assign(texts=[len(test_titles), len(test_answers), len(illustrative)])
top 1top 3top 5texts
job titles0.6170.7880.8461000
survey-style answers0.7090.9180.9511000
illustrative answers0.4140.6210.79329

The right group is among the five suggestions for 0.846 of the job titles, 0.951 of the survey-style answers, and 0.793 of the illustrative answers (23 of 29). The right group is the first suggestion for 0.617, 0.709, and 0.414 of the texts (12 of the 29 illustrative answers). A list of five suggestions holds the right group for 0.23 to 0.38 more of the texts than the first suggestion alone. This difference is the reason for giving a coder a list to choose from.

The next cell compares training sets. Each recorded model was trained on one set or on a combination, three times with different random seeds, and the table gives the mean share of texts whose right group is among five suggestions.

TESTS = {"report_titles": "job titles", "survey_style": "survey-style answers", "handwritten": "illustrative answers"}

recorded = pd.DataFrame(
{name: [arm["tests"][test]["top5"] for arm in results["arms"]] for test, name in TESTS.items()},
index=[arm["label"] for arm in results["arms"]],
).assign(texts=[arm["examples"] for arm in results["arms"]])
recorded.loc["The model of this notebook"] = [scores[name]["top 5"] for name in TESTS.values()] + [len(examples)]
recorded.astype({"texts": int})
job titlessurvey-style answersillustrative answerstexts
Titles written from the definitions0.5650.6950.75915865
Job titles from the job board0.8120.7740.20710134
Survey-style answers0.7130.9130.34518346
Job titles and survey-style answers0.8300.9240.28728476
All three sets0.8350.9380.74744058
The model of this notebook0.8460.9510.79344058
SETS = {arm["arm"]: arm for arm in results["arms"]}
rows = [(f"{arm['label']}\nrecorded, mean of {arm['runs']} runs", arm["tests"]["survey_style"]["top5"], "muted") for arm in results["arms"]]
rows.append(("The model of this notebook\nthe three sets", scores["survey-style answers"]["top 5"], "small"))
v.result_bars("Share of survey-style answers with the right group among five suggestions", rows, top_value=1.0)
Share of survey-style answers with the right group among five suggestionsTitles written from the definitionsrecorded, mean of 3 runs0.695Job titles from the job boardrecorded, mean of 3 runs0.774Survey-style answersrecorded, mean of 3 runs0.913Job titles and survey-style answersrecorded, mean of 3 runs0.924All three setsrecorded, mean of 3 runs0.938The model of this notebookthe three sets0.951

Rewriting the job titles as survey-style answers raises the score on the survey-style test from 0.774 to 0.924. Both models take their codes from the same 10,134 labelled titles. The score on the job titles is 0.812 with the titles alone and 0.830 with both sets, so the added answers do not lower it.

The three models trained without the titles written from the definitions score between 0.207 and 0.345 on the illustrative answers. 14 of the 29 illustrative answers have a code that no job title carries, and a model cannot suggest a group absent from its training texts. With all three sets, the score is 0.747.

The illustrative answers also give a result against combining the sets. A model trained only on the titles written from the definitions scores 0.759 on them, about the same as the model trained on all three sets. Its first suggestion is right for 0.529 of the illustrative answers, and the first suggestion of the model trained on all three sets is right for 0.368 (the file results.json holds these scores under top1). The difference is about five of the 29 answers. A test on coded answers from a real survey is needed to settle whether the job-board texts lower the quality of the first suggestion on survey answers.

The model of this notebook is a single run. Its three scores are within 0.05 of the recorded means for all three sets.

rows = [("The model of this notebook\nfirst suggestion", scores["job titles"]["top 1"], "small")]
rows += [(f"{m['model']}\nlarge model asked directly, recorded", m["top1"], "large") for m in results["large_on_titles"]]
v.result_bars("Share of job titles with the right group as the first suggestion", rows, top_value=1.0)
Share of job titles with the right group as the first suggestionThe model of this notebookfirst suggestion0.617Gemini 3.8 Flashlarge model asked directly, recorded0.497Claude Sonnet 5.5large model asked directly, recorded0.433Qwen3.8-27Blarge model asked directly, recorded0.406

The first suggestion of the small model is right for 0.617 of the 1,000 job titles. The three large models score 0.497, 0.433, and 0.406. Each large model received the titles with the codes and names of the 436 unit groups, without definitions and without examples.

The comparison favours the small model in one respect. Its training set holds 10,134 titles of the same job board, so the model has learned the coding habits of that board, and the test codes follow the same habits. A recorded model trained only on the titles written from the definitions scores 0.351 on this test, below the three large models. No large model was tested on the survey-style answers.

8. Suggestions for one answer​

In use, the model receives one answer and returns its five most probable groups. The next cell follows one answer through the steps. The picture is animated: the answer becomes tokens, the model gives one probability for each group, and the five highest become the suggestions.

def suggest(answer, size=5):
"""The most probable groups for one answer: (code, name, probability)."""
row = probabilities([answer])[0]
return [(CODES[i], group_name(CODES[i]), float(row[i])) for i in np.argsort(-row)[:size]]


ANSWER = "sells roasted maize by the roadside"
v.answer_path(ANSWER, tokenizer.tokenize(ANSWER), suggest(ANSWER), probabilities([ANSWER])[0])
From an answer to five suggested groupsAnswer in the survey“sells roasted maize by theroadside”Tokens▁sells▁roasted▁maize▁by▁the▁roadsideSmall modelDeBERTa-v3-smallone probability for eachof the 436 groupsThe five most probable groups5212Street Food Salespersons24%7511Butchers, Fishmongers and…23%9121Hand Launderers and Presse…15%9211Crop Farm Labourers6%7513Dairy Products Makers6%A coder chooses among the suggestions or enters another code.

The next cell shows the suggestions for three of the illustrative answers, with the expected code of each. A green bar marks the expected code when it is among the suggestions. The variable MINE holds a further answer, and any text can replace it.

for row in illustrative.iloc[[2, 3, 9]].itertuples():
v.suggestions(row.answer, suggest(row.answer), truth=row.code)

MINE = "i repair phones in a small shop"
v.suggestions(MINE, suggest(MINE), note="No expected code is recorded for this answer.")
“drives a motorcycle taxi”Groups suggested by the small model and the probability of each8322Car, Taxi and Van Drivers98%8321Motorcycle Drivers1%8332Heavy Truck and Lorry Drivers1%8331Bus and Tram Drivers0%4412Mail Carriers and Sorting Clerks0%Code in the test file: 8321. It is among the suggestions (green).
“grows maize and beans to feed the family”Groups suggested by the small model and the probability of each6130Mixed Crop and Animal Producers68%6111Field Crop and Vegetable Growers11%6114Mixed Crop Growers8%9213Mixed Crop and Livestock Farm Labourers3%6310Subsistence Crop Farmers3%Code in the test file: 6310. It is among the suggestions (green).
“has a small shop selling soap and sugar”Groups suggested by the small model and the probability of each5223Shop Sales Assistants60%5222Shop Supervisors17%1420Retail and Wholesale Trade Managers12%5221Shopkeepers8%3322Commercial Sales Representatives1%Code in the test file: 5221. It is among the suggestions (green).
“i repair phones in a small shop”Groups suggested by the small model and the probability of each7422Information and Communications TechnologyInstallers and Servicers43%7421Electronics Mechanics and Servicers38%3512Information and Communications TechnologyUser Support Technicians5%3114Electronics Engineering Technicians2%3511Information and Communications TechnologyOperations Technicians2%No expected code is recorded for this answer.

In the three illustrative cases, the expected group is in the list, and the first suggestion is a neighbouring group. For “drives a motorcycle taxi”, the model gives 98% to Car, Taxi and Van Drivers and 1% to the expected group, Motorcycle Drivers. A high probability therefore does not confirm the first suggestion, and a coder reviews the list in every case. The answer in MINE divides between two groups of repair workers, 7422 at 43% and 7421 at 38%. A coder decides between them with the definitions of the two groups.

9. Limits and rounds of improvement​

The results of section 7 have five limits.

  • No survey answer coded by a trained coder took part in this notebook. The survey-style test and the illustrative answers stand in for such answers. A test on answers from a real survey is required before the model enters use.
  • The large model that wrote the survey-style test saw the name of the right group, and the model that wrote the training answers saw its name and definition. The answers can contain words of the name. The score on the survey-style test can therefore be higher than the score on real answers.
  • The codes of the job titles come from the job board. Errors in them enter the training set and both tests on those titles.
  • For 127 of the 436 unit groups, the training set holds titles written from the definitions and no survey-style answer. Farming, street trade, and domestic work are among them.
  • The illustrative set has 29 answers, so one answer changes its score by about 0.03.

The process of the session continues with rounds of improvement.

A cycle of four steps: test the model, find the groups with the most errors, have the large model write more answers for them, train again.

A round tests the model on real cases kept for that purpose, finds the groups with the most errors, and has the large model write further texts for those groups. No round was run for this notebook. The results of section 7 indicate the content of a first round: survey-style answers for the 127 unit groups without a job title, written from the definitions and checked as in section 4. That run has not been made, and its effect on the scores is unknown.

Notebook 02 builds a model for a second task, assigning topics to reports, where a text can have several right answers and no labelled text is used for training.