6. Alignment with vocabularies

Question: Can free text be aligned with controlled vocabularies?

Rationale

Keywords written freely by many curators over many years say the same thing in many ways. "Hunger", "food insecurity", and "undernourishment" describe one concept, and a user who filters on one of them misses the records tagged with the other two. Search, filters, and AI systems all treat the three as different, because nothing tells them otherwise.

A controlled vocabulary gives each concept one preferred label, a list of the other forms people write, and a web address that identifies it. Mapping the catalog's free text to the vocabulary makes the catalog consistent for every system that reads it. Most of the mapping is plain string matching against the alternates, and it runs in seconds. The terms that do not match are where a language model can propose a concept for a curator to confirm, and where the vocabulary grows.

Target state

• Keywords and topics come from a vocabulary with preferred labels, alternates, and URIs, published as a SKOS concept scheme.
• Free text in the catalog is mapped to the vocabulary, with unmapped terms reviewed each period.
• Labels exist in each supported language, produced and reviewed as chapter 7 describes.

Maturity levels

Level	Steps
Foundational	Build a list of preferred terms from the most frequent keywords; curators consult it.
AI-ready	Publish the vocabulary as SKOS with alternates and URIs; map the catalog's free text to it; review unmapped terms; store mapped terms in the records' keywords and topics with the vocabulary name and URI.
AI-native	Let a model propose mappings for unmapped terms with a confidence, confirmed in the board; expose the vocabulary through the catalog; add labels per language through the translation workflow of chapter 7.

Running example

Fourteen free-text keywords across the five records against a vocabulary of ten concepts. Ten map exactly to a preferred label or an alternate, two map by similarity, and two ("hunger" and "jobless") are unmapped until an alternate is added or a model proposes the concept.

Recipes

Recipe

6.1 Build the vocabulary

Level

Foundational

Skills

Curator; subject-matter staff

Indicative time

Two days for the first version

Result: the catalog has a list of the concepts it uses for keywords and topics, each with one preferred label, a web address that identifies it, and the other forms curators have written for it.

A controlled vocabulary turns "poverty rate", "poverty headcount", and "poverty incidence" into one filter. Its minimum form is a CSV with one row per concept:

Column	Meaning
preferred_label	The one form used in filters and on pages
uri	A stable web address that identifies the concept, under the organization's domain
alternates	The other forms curators wrote, separated by ;

preferred_label,uri,alternates
Poverty,https://stats.example.org/vocab/poverty,poverty rate;poverty headcount;poverty incidence
Unemployment,https://stats.example.org/vocab/unemployment,unemployment rate;unemployed persons
Labour force,https://stats.example.org/vocab/labour-force,labor force;labour market;economically active population
Under-five mortality,https://stats.example.org/vocab/under-five-mortality,child mortality;u5 mortality;under-5 mortality rate
Health,https://stats.example.org/vocab/health,health outcomes;health status
Undernourishment,https://stats.example.org/vocab/undernourishment,prevalence of undernourishment;chronic hunger
Food insecurity,https://stats.example.org/vocab/food-insecurity,severe food insecurity;moderate or severe food insecurity
Malnutrition,https://stats.example.org/vocab/malnutrition,undernutrition;stunting;wasting
Food security,https://stats.example.org/vocab/food-security,food supply;dietary energy
Education,https://stats.example.org/vocab/education,schooling;enrolment;enrollment

1. Start from the keywords already in the catalog, ranked by frequency, and from the topic list the catalog uses. Merge synonyms into one concept each.
2. Give each concept a preferred label, its alternates, and a URI under the organization's domain. Where an international vocabulary covers the concept, record its URI as well.
3. Publish the scheme as SKOS, the web standard for such vocabularies; the CSV is its minimum form. Statistical classifications use XKOS, an extension of SKOS.
4. Record the vocabulary's name and URI in the records' keywords and topics fields, which carry a vocabulary and a URI in the World Bank schema.

Worked example. The vocabulary of the running example has ten concepts. The row for undernourishment has the preferred label "Undernourishment", the URI https://stats.example.org/vocab/undernourishment, and the alternates "prevalence of undernourishment; chronic hunger". Food insecurity and malnutrition have rows of their own, because an alternate is a synonym of the preferred label and these are related but distinct measures. Undernourishment is insufficient dietary energy (SDG indicator 2.1.1). Food insecurity is limited access to food (SDG indicator 2.1.2). Malnutrition covers outcomes such as stunting and wasting (SDG target 2.2). The row for unemployment has the alternates "unemployment rate; unemployment rates; joblessness". The curator building it started from the 140 distinct keywords in the catalog, ranked by frequency, and merged them into ten concepts in a morning, recording the SDG indicator URI beside the undernourishment concept because SDG 2.1.1 covers it. The CSV is converted to a SKOS file and published at the vocabulary's address.

What this does not do. A vocabulary built from existing keywords reflects what curators wrote, which may omit what users search for. The search logs and the question sets of the dissemination cookbook show the user side.

Recipe

6.2 Map free text to the vocabulary

Level

AI-ready

Skills

Python; curator for the review

Indicative time

Minutes to run; an hour per period to review

Result: every free-text keyword in the catalog is either mapped to a preferred label of the vocabulary or listed as unmapped for a curator to decide.

The input is a file of the keywords as curators wrote them, one row per record and keyword (record_id, keyword). The script matches each against the vocabulary's preferred labels and alternates, first exactly and then by string similarity, and lists what it could not match:

"""Map free-text keywords to a controlled vocabulary.

Reads keywords as curators or models wrote them (CSV: record_id, keyword)
and a vocabulary (CSV: preferred_label, uri, alternates separated by ";")
and assigns each keyword a preferred label by exact match on the preferred
label or an alternate, or by similarity above a threshold (difflib ratio),
or marks it unmapped. Prints the mapping and the unmapped keywords, which
are the candidates for new alternates or new concepts. Uses pandas and rapidfuzz (fuzzy matching).

The vocabulary format is the minimum of a SKOS concept scheme: a preferred
label, a URI, and alternate labels. A full scheme adds definitions,
broader and narrower relations, and labels per language.

Usage:
    python map_vocabulary.py keywords_freetext.csv vocabulary.csv

What this does not do: string matching maps known forms. A keyword that
names a concept in different words ("hunger" for undernourishment) is
unmapped until an alternate is added or a semantic match (embeddings) is
used, and a language model can propose the mapping for a curator to
confirm.
"""

from __future__ import annotations

import argparse
import re
import sys

import pandas as pd
from rapidfuzz import fuzz, process

WORD = re.compile(r"[a-z0-9]+")


def norm(text: str) -> str:
    return " ".join(WORD.findall(text.lower()))


def build_index(vocabulary: pd.DataFrame) -> dict[str, str]:
    """Every normalized preferred label and alternate, mapped to its preferred label."""
    index = {}
    for c in vocabulary.itertuples():
        index[norm(c.preferred_label)] = c.preferred_label
        for alt in c.alternates.split(";"):
            if alt.strip():
                index[norm(alt)] = c.preferred_label
    return index


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    parser.add_argument("keywords")
    parser.add_argument("vocabulary")
    parser.add_argument("--threshold", type=float, default=0.85)
    args = parser.parse_args(argv)
    index = build_index(pd.read_csv(args.vocabulary, dtype=str).fillna(""))
    keywords = pd.read_csv(args.keywords, dtype=str).fillna("")

    counts = dict.fromkeys(("exact", "fuzzy", "none"), 0)
    unmapped: list[str] = []
    print(
        f"{'match':<6} {'score':>5}  {'record':<22} {'keyword':<34} -> preferred label"
    )
    for k in keywords.itertuples():
        text = norm(k.keyword)
        if text in index:
            kind, score, label = "exact", 1.0, index[text]
        elif hit := process.extractOne(
            text, list(index), scorer=fuzz.ratio, score_cutoff=args.threshold * 100
        ):
            kind, score, label = "fuzzy", hit[1] / 100, index[hit[0]]
        else:
            kind, score, label = "none", 0.0, "-"
            unmapped.append(f"{k.record_id}: {k.keyword!r}")
        counts[kind] += 1
        print(
            f"{kind:<6} {score:>5.2f}  {k.record_id:<22} {k.keyword!r:<34} -> {label}"
        )
    print(
        f"\n{len(keywords)} keywords: "
        + ", ".join(f"{k} {v}" for k, v in counts.items())
    )
    if unmapped:
        print(
            "unmapped (add an alternate, add a concept, or let a model propose a mapping for review):"
        )
        for u in unmapped:
            print(f"  {u}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

python map_vocabulary.py keywords_freetext.csv vocabulary.csv

Output for the running example (abridged):

exact   1.00  FS_UNDERNOURISH_PCT     'food insecurity'                  -> Food insecurity
fuzzy   0.92  LF_UNEMP_PCT            'unemployment rates'               -> Unemployment
none    0.00  FS_UNDERNOURISH_PCT     'hunger'                           -> -
14 keywords: exact 10, fuzzy 2, none 2
unmapped (add an alternate, add a concept, or let a model propose a mapping for review):
  FS_UNDERNOURISH_PCT: 'hunger'
  LF_UNEMP_PCT: 'jobless'

Each line shows the match type, the similarity score, the record, the keyword as the curator wrote it, and the preferred label it was mapped to. An exact match scores 1.00, a fuzzy match scores by string similarity, and an unmapped keyword scores zero and appears again in the list at the end.

1. Accept exact matches. Check fuzzy matches on a sample; a similarity threshold of 0.85 catches plurals and spelling, and some false friends.
2. For each unmapped term, decide: add it as an alternate of an existing concept ("hunger" under Undernourishment), add a new concept, or drop it. A language model given the term, its record, and the vocabulary can propose the concept with a confidence, for the curator to confirm in the board.
3. Write the mapped terms into the records with the vocabulary and URI, through the editor.

Worked example. In the run above, "food insecurity" maps exactly to Food insecurity, a concept of its own beside Undernourishment, and "unemployment rates" maps to Unemployment by similarity with a score of 0.92, because the alternate is the singular. The curator checks the two fuzzy matches and confirms both. For the two unmapped terms, the model is given each term, its record, and the vocabulary, and it proposes Undernourishment for "hunger" with a confidence of 0.9 and Unemployment for "jobless" with 0.95. The curator confirms both in the board, and "hunger" and "jobless" are added as alternates. The next run maps all fourteen keywords exactly, and the records' keywords fields now carry the preferred labels with the vocabulary name and URI.

What this does not do. String matching maps known forms. Matching by meaning (with embeddings) maps more, and a model proposes the rest. Both end in a curator's confirmation, because a wrong mapping is a wrong filter for every user.

Implementation options

• Vocabulary format: SKOS for concepts and labels, XKOS for classifications; a CSV with preferred label, URI, and alternates as the working form.
• Mapping: exact and fuzzy string matching (this chapter); embeddings for semantic matching; a model for proposals with a confidence.
• Storage: keywords and topics with vocabulary and URI in the World Bank schemas; the Metadata Editor's controlled lists.
• International vocabularies: the SDG indicator framework, the classifications of the concepts chapter of the microdata cookbook, and thematic thesauri, as sources of URIs.

World Bank examples

• Generative AI for Metadata Quality describes controlled vocabulary standardization as a model task: mapping free-text topic keywords to a vocabulary with a confidence.
• The World Bank schemas carry keywords and topics with vocabulary and uri per entry.

Resources

• SKOS and the SKOS primer: concept schemes, preferred and alternate labels, and how to publish one.
• XKOS: statistical classifications and correspondences.
• Skosmos: an open-source browser and API for SKOS vocabularies, for publishing the scheme.
• UN classifications registry and the SDG indicator framework: sources of URIs for concepts the organization reuses.
• Vocabulary, keywords, and mapping script: the running example.

Common mistakes

• A vocabulary nobody uses. The mapping has to write back into the records, or the list is decoration.
• Fuzzy matches accepted blind. "Employment" and "unemployment" are close strings and opposite concepts.
• Dropping unmapped terms. They are where the vocabulary learns what curators and users say.

Verification

• Vocabulary. Every concept has a preferred label, a URI, and alternates, and the scheme is published as SKOS.
• Mapping. The share of keywords mapped exactly, by similarity, and unmapped is reported each period, and the unmapped terms have been decided.
• Records. The keywords and topics carry the vocabulary and the URI.
• Languages. Labels exist per language with their review status, from chapter 7.

Checklist

0 of 6 done

• Vocabulary with preferred labels, alternates, and URIs
• Published as a SKOS concept scheme; classifications as XKOS
• Free text mapped; fuzzy matches sampled; unmapped terms decided
• Model proposals for unmapped terms confirmed by a curator
• Mapped terms written to records with vocabulary and URI
• Labels per language through the translation workflow

Ticks are saved in this browser only.