3. Interpretation of variables

Question: Can AI interpret the variables?

Rationale

A data file is a grid of codes. The meaning of each column lives in the data dictionary: the label, the type, the value codes and their labels, the codes that mean "missing", the universe the variable applies to, and the question that produced it. A person reading the file with the questionnaire open can reconstruct some of this. Software cannot. An AI system given a file in which relationship has values 1 to 8 and no labels will either refuse to use it or invent the labels.

The dictionary is also the input to everything in the later chapters. Concept links, thematic groups, variable search, and quality statements all attach to variables, and each of them is only as good as the dictionary row it attaches to. This chapter builds the dictionary, checks it, and converts it into the schema the catalog stores.

Target state

• Every variable has a label that adds information to the name, a type, and a universe.
• Every categorical variable has value codes with labels; every numeric variable states its missing-value codes.
• Every collected variable carries its question text; every derived variable carries its derivation.
• The dictionary is stored in the World Bank microdata schema (variables and data_files) and checked before each release.

Maturity levels

Level	Steps
Foundational	Produce a data dictionary with one row per variable: file, name, label, type, value codes, missing codes. Publish it with the data.
AI-ready	Add universe and question text. Run the dictionary check before release and fix every error. Convert the dictionary into the schema's variables section and validate the study record.
AI-native	Add concepts (chapter 4), derivations, and summary statistics per variable. Run the check in the release pipeline, and use AI-assisted review for label and description quality.

Running example

The dictionary of EX-LFS-2025-Q2 has 18 variables and five planted defects. Recipe 3.1 finds them, recipe 3.2 converts the repaired dictionary into the schema and validates the study record at the AI-ready level.

Recipes

Recipe

3.1 Check the data dictionary before release

Level

Foundational

Skills

Data curator; Python

Indicative time

One hour the first time; minutes on each release

Result: the curator has a list of the variables that software or a user would misread, each with the reason, and the share of variables that have each field filled.

The dictionary is a CSV with one row per variable:

Column	Meaning
file_id, name	The data file and the variable name
label	What the variable measures, in words
type	numeric, categorical, or string
universe	Whom the variable applies to ("Employed persons aged 15 and over")
question	The question text as asked
values	The codes and their labels, as code=label;code=label
missing	The codes that mean a missing or inapplicable answer, as code;code, or none
concept	The concept the variable measures (chapter 4)

file_id,name,label,type,universe,question,values,missing,concept
F1,hhid,Household identifier,string,All households,,,,
F1,pid,Person identifier within household,string,All persons,,,,
F1,region,Region of residence,categorical,All persons,,R01=Northern Region;R02=Central Region,,
F1,urban,Urban or rural area,categorical,All persons,,1=Urban;2=Rural,,
F1,sex,Sex of the person,categorical,All persons,Q3. Sex of [NAME],1=Male;2=Female,,Sex
F1,age,Age in completed years,numeric,All persons,Q4. How old was [NAME] at last birthday?,,-9,
F1,relationship,Relationship to household head,categorical,All persons,Q5. What is [NAME]'s relationship to the head of household?,,,
F1,educ,Highest level of education completed,categorical,Persons aged 5 and over,Q7. What is the highest level of education [NAME] has completed?,0=None;1=Primary;2=Lower secondary;3=Upper secondary;4=Post-secondary non-tertiary;5=Tertiary,-9,ISCED 2011 level
F1,lfs_status,Labour force status,categorical,Persons aged 15 and over,Derived from Q10 to Q16,1=Employed;2=Unemployed;3=Outside the labour force,,Labour force status (ICLS 19)
F1,occupation,Occupation in main job,categorical,Employed persons aged 15 and over,Q12. What kind of work does [NAME] do in the main job?,11=Chief executives and senior officials;21=Science and engineering professionals;52=Sales workers;61=Market-oriented skilled agricultural workers;92=Agricultural labourers,-8;-9,ISCO-08 sub-major group
F1,industry,Industry of main job,categorical,Employed persons aged 15 and over,Q13. What does the business where [NAME] works mainly do?,01=Crop and animal production;10=Manufacture of food products;47=Retail trade;85=Education,-8;-9,ISIC Rev.4 division
F1,hours,Hours worked last week in all jobs,numeric,Employed persons aged 15 and over,Q14. How many hours did [NAME] work last week in all jobs?,,-9,
F1,income,Monthly income from main job,numeric,Employed persons aged 15 and over,Q15. What was [NAME]'s income from the main job last month?,,,
F1,informal,informal,categorical,Employed persons aged 15 and over,Derived from Q12 and Q13,1=Yes;2=No,,
F1,jobsearch,Looked for work in the last four weeks,categorical,Persons aged 15 and over not employed,Q16. Did [NAME] look for work in the last four weeks?,1=Yes;2=No,-9,
F1,q17b,,categorical,Persons who looked for work,Q17b. Which methods did [NAME] use to look for work?,1=Employment office;2=Friends and relatives;3=Direct application;4=Online platform;5=Other,,
F1,hours,Hours worked last week in main job,numeric,Employed persons aged 15 and over,Q14a. How many hours did [NAME] work last week in the main job?,,-9,
F1,weight,Calibrated person weight,numeric,All persons,Design weight adjusted for non-response and calibrated to population projections,,,

The check reads the dictionary and reports what is missing or inconsistent:

"""Check a data dictionary before it is published.

Reads a data dictionary with one row per variable (CSV) and reports what an
AI system or a user would find missing or misleading:

errors (exit status 1)
  - a variable without a label
  - a label that repeats the variable name
  - duplicate variable names within a file
  - a categorical variable without value labels

warnings
  - a numeric variable without a statement of its missing-value codes
  - a variable without a universe
  - a collected variable without its question text
  - a variable without a concept, where a concept map could link it to a
    classification

Columns expected: file_id, name, label, type (numeric | categorical |
string), universe, question, values ("code=label;code=label"), missing
("code;code"), concept. Other columns are ignored. Uses pandas.

What this does not check: whether the labels are correct, whether the
value labels match the data, or whether the universe statements agree with
the skip patterns. Those need the data file and the questionnaire.

Usage:
    python check_dictionary.py lfs_2025q2_dictionary.csv
"""

from __future__ import annotations

import argparse
import sys

import pandas as pd

FIELDS = ("label", "universe", "question", "values", "missing", "concept")


def is_identifier(name: str) -> bool:
    return name.lower().endswith("id")


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    parser.add_argument("dictionary", help="CSV with one row per variable")
    args = parser.parse_args(argv)
    df = pd.read_csv(args.dictionary, dtype=str).fillna("")
    if df.empty:
        sys.exit("no variables found")
    for col in FIELDS:
        if col not in df:
            df[col] = ""
    df = df.apply(lambda s: s.str.strip())
    df["type"] = df["type"].str.lower()
    df["ref"] = df["file_id"] + "/" + df["name"]
    df["is_id"] = df["name"].map(is_identifier)

    errors: list[str] = []
    warnings: list[str] = []
    dup = df.groupby("ref").size()
    errors += [f"{ref}: name appears {n} times" for ref, n in dup[dup > 1].items()]
    for r in df.drop_duplicates("ref").itertuples():
        if not r.label:
            errors.append(f"{r.ref}: no label")
        elif r.label.lower() == r.name.lower():
            errors.append(f"{r.ref}: label repeats the name ({r.label!r})")
        if r.type == "categorical" and not r.values:
            errors.append(f"{r.ref}: categorical variable without value labels")
        if r.type == "numeric" and not r.missing and not r.is_id:
            warnings.append(
                f"{r.ref}: numeric variable without a missing-value statement"
            )
        if not r.universe:
            warnings.append(f"{r.ref}: no universe")
        if not r.question and not r.is_id:
            warnings.append(f"{r.ref}: no question or derivation text")
        if r.type == "categorical" and not r.concept and not r.is_id:
            warnings.append(f"{r.ref}: no concept or classification named")

    print(f"{len(df)} variables in {df['file_id'].nunique()} file(s)")
    filled = (df[list(FIELDS)] != "").sum()
    for col, n in filled.items():
        print(f"  {col:<10} {n:>4} of {len(df)}  {n / len(df):>4.0%}")
    print()
    for w in warnings:
        print(f"warning  {w}")
    for e in errors:
        print(f"error    {e}")
    print(f"\n{len(errors)} error(s), {len(warnings)} warning(s)")
    print(
        "Not checked here: whether labels and value labels are correct, or whether universes match the skip patterns."
    )
    return 1 if errors else 0


if __name__ == "__main__":
    sys.exit(main())

Output for the running example (abridged):

18 variables in 1 file(s)
  label        17 of 18   94%
  values       10 of 18   56%
  missing       7 of 18   39%
  concept       5 of 18   28%
warning  F1/income: numeric variable without a missing-value statement
error    F1/hours: name appears 2 times
error    F1/relationship: categorical variable without value labels
error    F1/informal: label repeats the name ('informal')
error    F1/q17b: no label
4 error(s), 10 warning(s)

The first block reports, for each field, how many variables have it filled. The lines that follow are the findings. An error is something that will cause a misreading, such as a categorical variable without value labels. A warning is something a curator should decide, such as a numeric variable with no statement about missing codes.

1. Export the dictionary in the form above.
2. Run the check.
3. Fix every error before release. Here: label q17b, give informal a label that says what it measures, add the value codes of relationship, and rename the second hours (the main-job variable). Decide on the warnings: income has no missing code because none is used, and that is worth stating as none.
4. Keep the check in the release procedure and keep its output with the release.

Worked example. The run above is the running example. The curator reads the four errors in order. The variable q17b gets the label "Looked for work in the last four weeks", taken from the question text recovered in chapter 2. The variable informal gets the label "Informal employment in main job". The variable relationship gets its eight value labels from the questionnaire, from 1=Head to 8=Other relative. The second hours variable, which measures hours in the main job only, is renamed hours_main. For the warning on income, the curator confirms with the processing team that no missing code is used and writes none. The check runs again and reports zero errors and nine warnings, each of which has a recorded decision.

What this does not do. The check reads the dictionary alone. It does not verify that the value labels match the codes in the data, that the universe matches the skip pattern in the questionnaire, or that labels are correct. A data-side check (frequency tables against the value codes) and a questionnaire review add those.

Recipe

3.2 Convert the dictionary into the DDI-based schema

Level

AI-ready

Skills

Python

Indicative time

One hour

Result: the study record has its data_files and variables sections filled from the dictionary, so that it is complete at the AI-ready level and can be imported into NADA or the Metadata Editor.

The converter maps each dictionary column to the corresponding field of the schema (the label to var_label, the question to var_qstn_qstnlit, the codes to var_catgry) and merges the result into the study record:

"""Convert a data dictionary CSV into the World Bank microdata schema.

Produces the `data_files` and `variables` sections of a microdata record
(worldbank/metadata-schemas, microdata-schema.json, which follows DDI
Codebook), and merges them into an existing study record when one is given.
The result can be imported into NADA or the Metadata Editor, or validated
with the metadata checker from the dissemination cookbook.

Mapping from the CSV columns to DDI fields:

    name      -> name            values    -> var_catgry [{value, label}]
    label     -> labl            missing   -> var_catgry entries flagged in var_notes
    type      -> var_format.type question  -> var_qstn_qstnlit
    universe  -> var_universe    concept   -> var_concept [{title}]

Uses pandas.

What this does not do: it does not compute summary statistics (var_sumstat)
or category frequencies, which need the data file, and it does not validate
the result against the schema; run check_metadata.py from the dissemination
cookbook for that.

Usage:
    python dictionary_to_ddi.py lfs_2025q2_dictionary.csv --study example_microdata.json -o study_with_variables.json
    python dictionary_to_ddi.py lfs_2025q2_dictionary.csv            # prints the sections only
"""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

import pandas as pd

FORMAT = {"numeric": "numeric", "categorical": "numeric", "string": "character"}


def parse_values(text: str) -> list[dict[str, str]]:
    """'1=Yes;2=No' -> [{value: '1', label: 'Yes'}, ...]."""
    pairs = (item.split("=", 1) for item in text.split(";") if "=" in item)
    return [{"value": v.strip(), "label": label.strip()} for v, label in pairs]


def variable(i: int, r: pd.Series) -> dict:
    """One DDI variable from one dictionary row."""
    var: dict = {
        "vid": f"V{i}",
        "file_id": r["file_id"],
        "name": r["name"],
        "labl": r["label"],
        "var_format": {"type": FORMAT.get(r["type"].lower(), "character")},
    }
    if r["universe"]:
        var["var_universe"] = r["universe"]
    if r["question"]:
        var["var_qstn_qstnlit"] = r["question"]
    missing = [m.strip() for m in r["missing"].split(";") if m.strip()]
    categories = parse_values(r["values"]) + [
        {"value": code, "label": "Missing"} for code in missing
    ]
    if categories:
        var["var_catgry"] = categories
    if missing:
        var["var_notes"] = "Missing-value codes: " + ", ".join(missing)
    if r["concept"]:
        var["var_concept"] = [{"title": r["concept"]}]
    return var


def convert(rows: list[dict[str, str]]) -> tuple[list[dict], list[dict]]:
    df = pd.DataFrame(rows).fillna("")
    for col in (
        "label",
        "type",
        "universe",
        "question",
        "values",
        "missing",
        "concept",
    ):
        if col not in df:
            df[col] = ""
    df = df.apply(lambda s: s.astype(str).str.strip())
    counts = df["file_id"].value_counts(sort=False)
    files = [
        {"file_id": f, "file_name": f"{f}.csv", "var_count": int(n)}
        for f, n in counts.items()
    ]
    variables = [variable(i, r) for i, (_, r) in enumerate(df.iterrows(), 1)]
    return files, variables


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    parser.add_argument("dictionary", type=Path)
    parser.add_argument(
        "--study", type=Path, help="study record (JSON) to merge the sections into"
    )
    parser.add_argument(
        "-o", "--output", type=Path, help="where to write the result (default: stdout)"
    )
    args = parser.parse_args(argv)

    rows = pd.read_csv(args.dictionary, dtype=str).fillna("").to_dict("records")
    files, variables = convert(rows)
    record = json.loads(args.study.read_text(encoding="utf-8")) if args.study else {}
    record.update({"data_files": files, "variables": variables})

    text = json.dumps(record, indent=2, ensure_ascii=False)
    if args.output:
        args.output.write_text(text + "\n", encoding="utf-8")
        print(
            f"wrote {args.output}: {len(files)} file(s), {len(variables)} variable(s)"
        )
    else:
        print(text)
    return 0


if __name__ == "__main__":
    sys.exit(main())

python dictionary_to_ddi.py lfs_2025q2_dictionary.csv --study example_microdata.json -o study_with_variables.json

1. Run the converter on the repaired dictionary.

2. Validate the merged record against the schema and the profile:

   python check_metadata.py study_with_variables.json profile_microdata.json --level ai-ready

   For the running example the merged record is valid against the schema and complete at the AI-ready level. The two fields the study record lacked in chapter 1 were data_files and variables.

3. Import the record into the catalog. The variables appear on the study page with their labels, value codes, universes, and question text, and the catalog's variable search indexes them.

Worked example. The converter reads the repaired dictionary of 18 rows and writes one variables entry per row. The row for relationship becomes an entry with var_label set to "Relationship to household head", eight var_catgry entries from the value codes, var_universe set to "All household members", and var_qstn_qstnlit set to the question text. The data_files section gains one entry for the file, with its identifier, name, and case count of 41. The validation at the AI-ready level reports the record valid and complete, where the same record without these two sections had failed on both in chapter 1.

What this does not do. The converter maps the dictionary's columns to DDI fields and nothing more. Summary statistics and category frequencies need the data file; the Metadata Editor computes them on import. Interval types, formats, and locations in fixed-width files are left for the curator to add where they apply.

Recipe

3.3 Document missing codes, universes, and derivations

Level

AI-ready

Skills

Subject-matter statistician

Indicative time

Two to four hours per survey

Result: for every variable, a user or a system can tell which codes are not values, whom the variable applies to, and how a derived variable was computed.

1. Missing codes. List every code that marks a missing, inapplicable, refused, or unknown response, and give each its own label (-8 = Not applicable, -9 = Missing). In the schema they are categories with a note; in the CSV they are the missing column. A numeric variable that uses no missing code states none.
2. Universes. Write the universe as the population the question was asked of, in the terms of the questionnaire ("Employed persons aged 15 and over"). A universe explains why a variable has fewer cases than the file. Without it, a system reads the gap as missing data.
3. Derivations. For each derived variable (lfs_status, informal), state the inputs and the rule in var_derivation, and name the standard the rule follows (the ICLS resolution for labour force status).
4. Record the questionnaire version and the skip pattern that each universe depends on, so that a change in the questionnaire shows up as a change in the dictionary.

Worked example. The derived variable lfs_status takes the values 1 = Employed, 2 = Unemployed, and 3 = Outside the labour force. Its var_derivation entry reads: "Employed if hours in the reference week are greater than zero or the person has a job to return to (questions 14 and 15). Unemployed if not employed, looked for work in the last four weeks (question 17b), and available to start within two weeks (question 18). Otherwise outside the labour force. Follows the 19th ICLS resolution." The universe of hours is "Employed persons aged 15 and over", which explains why 27 of the 41 records have a value. The questionnaire version is recorded as v2, and the skip pattern from question 14 is noted beside the universe.

What this does not do. This recipe documents the rules. It does not verify them against the data. A consistency check (for example, that lfs_status equals 1 for every case with positive hours) belongs to the data quality procedures of chapter 7.

Implementation options

• DDI Codebook holds variable-level documentation: label, text, question, universe, categories, derivation, concepts, and summary statistics. The World Bank schema is its JSON form.
• The Metadata Editor builds the dictionary from the data file (names, types, frequencies) and lets the curator add the rest; it exports the schema and publishes to NADA.
• Statistical software exports. SPSS, Stata, and R files carry labels and value labels, which the Metadata Editor and DDI tools read; a dictionary CSV as in this chapter is the portable form.
• DDI Lifecycle adds reusable questions and concepts for organizations that manage questionnaires centrally (chapter 4).

World Bank examples

• The microdata schema defines variables (with var_catgry, var_universe, var_qstn_qstnlit, var_derivation, var_concept, var_sumstat) and data_files.
• Metadata Augmentation reads dictionaries in CSV, JSON, or from a NADA catalog, and its adapters map source columns to a canonical form, which is the same step as recipe 3.2.
• The Metadata Reviewer reviews label and description text with a curator in the loop.

Resources

• DDI Codebook 2.5 variable elements: var, labl, catgry, universe, qstn, derivation, concept, sumStat.
• Metadata Editor user and practice guide: the variable-level screens, frequencies, and validation.
• World Bank metadata schemas: the microdata schema (variables, data_files) and the Python library for building records.
• pyreadstat: reads labelled Stata, SPSS, and SAS files for checks outside the editor.
• Metadata Augmentation: the program's dictionary adapters and variable grouping, which read the dictionary form used here.

Common mistakes

• Labels that repeat the name. informal labelled "informal" tells a system nothing; "Informal employment in main job" does.
• Missing codes left as values. A -9 without a label becomes a negative age in any computation.
• Universes in footnotes. A system reads fields; a footnote is invisible to it. The universe belongs in the variable record.
• Dictionaries that drift from the file. A dictionary edited by hand after the file changed is wrong in ways no check detects; regenerate it from the file and add the curated fields.

Verification

• Dictionary check. The check reports zero errors, and every warning has a recorded decision.
• Schema check. The merged study record is valid and complete at the AI-ready level.
• Data check. Every value in a categorical variable appears in its value codes, and every negative value in a numeric variable is a documented missing code.
• Interpretation questions. For ten variables, an assistant asked for the meaning of a code and the universe answers in agreement with the dictionary.

Checklist

0 of 8 done

• One dictionary row per variable, regenerated from the file
• Every variable labelled with more than its name
• Value codes with labels for every categorical variable
• Missing-value codes listed and labelled; "none" stated where none apply
• Universe for every variable
• Question text for collected variables; derivation for derived ones
• Dictionary check run with zero errors before release
• Study record complete at the AI-ready level after conversion

Ticks are saved in this browser only.