1. Findability of statistics

Question: Can people and AI find our statistics?

Rationale

AI systems increasingly stand between users and data. When someone asks a search assistant about hunger in their country, the assistant reads page titles, descriptions, and catalog records to decide which series to show. A series with a vague title, an empty description, or no machine-readable record may never appear in the answer, even when it is exactly the right series. The organization has no way of knowing that it was passed over.

Users also describe what they need in their own words. "How many people go hungry" and "prevalence of undernourishment" refer to the same topic. A keyword index treats them as unrelated, because they share no words. A search that matches by meaning finds the series anyway, but only if the record contains enough text for the meaning to be read from it.

This chapter therefore starts with the records, because every later step reads them. It then makes the records readable by crawlers, and finally adds search by meaning.

Target state

Every dataset and indicator has a stable identifier and a complete record: title, description, unit, time coverage, geography, frequency, source, and contact.
The catalog is published in a machine-readable form that crawlers and harvesters can read.
Search finds results by meaning as well as by exact words, in the languages users use.

Maturity levels

Level	Steps
Foundational	Fill in the description, unit, and coverage fields for every published series. Give each dataset its own page with a permanent URL. Publish an XML sitemap and a clear `robots.txt`.
AI-ready	Publish catalog records as DCAT or schema.org `Dataset` markup (JSON-LD) on each dataset page. Expose a catalog API or a harvestable feed. Add translated titles and descriptions for the main national and user languages.
AI-native	Add semantic search over the catalog using embeddings, combined with keyword search. Use AI to find and repair missing or inconsistent metadata, with human review. Offer the catalog to AI assistants through an agent interface (see chapter 2).

Running example

The example organization has five series. One of them, LF_UNEMP_PCT, has no description. The recipes below find that gap, publish each series as a schema.org record, and add a search that understands "jobless rate".

Recipes

Recipe

1.1 Check the records against the schema and a completeness profile

Level: Foundational
Skills: Anyone who can export the catalog and run a Python script; a metadata lead to set the profile
Indicative time: Half a day the first time; minutes after that

Result: for every series record in the catalog, the organization has three answers. Does the record have the right structure? Does it have every field the organization requires at its target level? Are the values well-formed? The count of records that pass is the first measure to track, because every later step in this guide depends on these fields.

A catalog record is a description of a dataset or a series: its name, definition, unit, source, and so on. A schema fixes the names and types of those fields, and a profile is the organization's own list of which fields it requires. The check has three layers, and they answer different questions:

Layer	Question	Who decides the rule
Structure	Is the record a valid instance of the World Bank metadata schema for its type?	The schema. The indicator schema requires only `idno` and `name`; the other 73 fields of `series_description` are optional. The other types are similar.
Completeness	Does the record have every field the organization requires at this maturity level?	The organization, in a profile file, one per data type. The schema does not define completeness.
Validity	Do dates parse, do periods run in order, are URLs URLs, do values come from the organization's vocabulary, is the definition more than the name repeated?	Rules named in the profile and applied by the script.

The same check runs for indicators, survey microdata, geospatial data, documents, tables, scripts, images, and videos: the profile names the type, and the script fetches that type's schema. The Application by data type page lists the profiles and example records for each.

The script that runs the check needs the jsonschema package (4.18 or later) for the structure layer and downloads the schema files the first time:

"""Check catalog metadata in three layers, for any World Bank metadata type.

1. Structure: each record is validated against the World Bank metadata
   schema for its type (worldbank/metadata-schemas): indicator, indicators
   database, microdata (DDI Codebook), geospatial (ISO 19115/19139),
   document (Dublin Core), table, script, image, or video. The schemas
   require very little (an identifier and a title), so passing this layer
   means the record is well-formed, and nothing more.
2. Completeness: each record is checked against a profile, a JSON file in
   which the office lists the fields it requires at each maturity level,
   as dotted paths into the record, with a reason per field. The schema
   does not decide what is complete; the office does.
3. Validity: rules named in the profile: dates parse, periods are ordered,
   URLs look like URLs, values come from a vocabulary, a text field has a
   minimum length, two fields are not identical, placeholders such as
   "n/a" are absent.

What this does not check: whether the text is correct, whether a title and
an abstract describe the same thing, whether a unit matches the values, or
whether the text is specific enough to be useful. Those are semantic
checks; see the program's Generative AI for Metadata Quality work.

Input: a JSON file with one record or a list of records in the nested form
of the schema (what NADA and the Metadata Editor export), or, for flat
types such as indicators, a CSV with one row per record. CSV columns are
dotted paths (or bare names placed under the profile's "csv_root"); list
fields take ";" between values and each value fills the item's main
property, which is read from the schema.

Usage:
    python check_metadata.py example_catalog.csv profile_indicator.json
    python check_metadata.py example_microdata.json profile_microdata.json
    python check_metadata.py records.json profile_geospatial.json --level ai-ready
    python check_metadata.py records.json profile.json --schema-dir ./wb-schemas

Exit status is 0 when every record passes all three layers, 1 otherwise,
so the script can run in a scheduled job or a CI step.

Needs the `jsonschema` package (4.18 or later, which brings `referencing`)
for layer 1. Without it, layer 1 is skipped and the report says so. Schema
files are downloaded on first use into a folder next to this script.
"""

from __future__ import annotations

import argparse
import csv
import json
import re
import sys
import urllib.request
from collections import Counter
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any

RAW = "https://raw.githubusercontent.com/worldbank/metadata-schemas/main/schemas/"
TYPES = {
    "indicator": "timeseries-schema.json",
    "indicators-db": "timeseries-db-schema.json",
    "microdata": "microdata-schema.json",
    "geospatial": "geospatial-schema.json",
    "document": "document-schema.json",
    "table": "table-schema.json",
    "script": "script-schema.json",
    "image": "image-schema.json",
    "video": "video-schema.json",
}

DATE = re.compile(r"^\d{4}-\d{2}-\d{2}$")
PERIOD = re.compile(r"^\d{4}(-\d{2}(-\d{2})?|-Q[1-4]|-W\d{2}|-S[12])?$")
URL = re.compile(r"^https?://\S+$")
IDNO = re.compile(r"^[A-Za-z0-9_.\-]+$")
PLACEHOLDERS = {"n/a", "na", "none", "-", "tbd", "unknown", "?"}

Record = dict[str, Any]
Schema = dict[str, Any]


def warn(message: str) -> None:
    print(message, file=sys.stderr)


# --- schemas ---------------------------------------------------------------


class SchemaStore:
    """Loads World Bank schema files by name, downloading them on first use.

    The type schemas reference the others by relative file name
    ("ddi-schema.json"), so every reference is resolved by basename.
    """

    def __init__(self, schema_dir: Path) -> None:
        self.dir = schema_dir
        self._cache: dict[str, Schema | None] = {}

    def get(self, name: str) -> Schema | None:
        if name not in self._cache:
            self._cache[name] = self._load(name)
        return self._cache[name]

    def _load(self, name: str) -> Schema | None:
        path = self.dir / name
        if not path.exists():
            self.dir.mkdir(parents=True, exist_ok=True)
            try:
                with urllib.request.urlopen(RAW + name, timeout=20) as resp:
                    path.write_bytes(resp.read())
            except OSError as exc:
                warn(f"could not download {name}: {exc}")
                return None
        with path.open(encoding="utf-8") as fh:
            return json.load(fh)

    # Navigation through $ref (across files) and allOf.

    def deref(self, name: str, node: Schema) -> tuple[str, Schema]:
        """Follow $ref chains; returns the file name and the resolved node."""
        while isinstance(node, dict) and "$ref" in node:
            file, _, fragment = node["$ref"].partition("#")
            if file:
                name = file.rsplit("/", 1)[-1]
            target = self.get(name)
            if target is None:
                return name, {}
            node = target
            for part in fragment.split("/"):
                if part:
                    node = node[part]
        return name, node

    def child(self, name: str, node: Schema, key: str) -> tuple[str, Schema] | None:
        """The schema of property `key`, looking through allOf."""
        name, node = self.deref(name, node)
        if key in node.get("properties", {}):
            return self.deref(name, node["properties"][key])
        for sub in node.get("allOf", []):
            found = self.child(name, sub, key)
            if found:
                return found
        return None

    def node_at(self, root: str, path: str) -> tuple[str, Schema] | None:
        """The schema node for a dotted path, stepping into array items."""
        schema = self.get(root)
        if schema is None:
            return None
        name, node = self.deref(root, schema)
        for key in path.split("."):
            if node.get("type") == "array":
                name, node = self.deref(name, node.get("items", {}))
            found = self.child(name, node, key)
            if not found:
                return None
            name, node = found
        return name, node

    def is_array(self, root: str, path: str) -> bool:
        found = self.node_at(root, path)
        return bool(found) and found[1].get("type") == "array"

    def item_key(self, root: str, path: str) -> str | None:
        """For an array field: the property a bare value fills, "" for plain items, None if not an array."""
        found = self.node_at(root, path)
        if not found or found[1].get("type") != "array":
            return None
        _, item = self.deref(found[0], found[1].get("items", {}))
        if item.get("type") == "object" or "properties" in item:
            required = item.get("required") or []
            props = list(item.get("properties", {}))
            return (required or props or [""])[0]
        return ""


# --- records ---------------------------------------------------------------


@dataclass
class Column:
    """How one CSV column maps into the nested record. Computed once per file."""

    path: str
    item_key: str | None  # None: scalar; "": list of strings; "x": list of {x: value}
    array_prefixes: set[str] = field(
        default_factory=set
    )  # path prefixes that are arrays


def plan_columns(
    header: list[str], profile: dict, store: SchemaStore, root: str
) -> list[Column]:
    """Resolve each CSV column against the profile and the schema, once."""
    csv_root = profile.get("csv_root", "")
    renames = profile.get("csv_columns", {})
    # The schema's first (required) property is the default item key for a
    # list column; the profile can name another, for example "uri" for
    # reference lists whose first property is "source".
    item_keys = profile.get("csv_item_keys", {})
    plan = []
    for col in header:
        path = renames.get(col, col)
        if csv_root and not path.startswith(csv_root + "."):
            path = f"{csv_root}.{path}"
        parts = path.split(".")
        prefixes = {".".join(parts[: i + 1]) for i in range(len(parts) - 1)}
        plan.append(
            Column(
                path=path,
                item_key=item_keys.get(path, store.item_key(root, path)),
                array_prefixes={p for p in prefixes if store.is_array(root, p)},
            )
        )
    return plan


def set_path(record: Record, column: Column, value: Any) -> None:
    """Set a value at the column's path, creating the first item of any array on the way."""
    parts = column.path.split(".")
    node = record
    for i, key in enumerate(parts[:-1]):
        if ".".join(parts[: i + 1]) in column.array_prefixes:
            node = node.setdefault(key, [{}])[0]
        else:
            node = node.setdefault(key, {})
    node[parts[-1]] = value


def csv_value(column: Column, raw: str) -> Any:
    if column.item_key is None:
        return raw
    items = [v.strip() for v in raw.split(";") if v.strip()]
    if column.item_key == "":
        return items
    return [{column.item_key: v} for v in items]


def load_records(
    path: Path, profile: dict, store: SchemaStore, root: str
) -> list[Record]:
    if path.suffix.lower() == ".json":
        with path.open(encoding="utf-8") as fh:
            data = json.load(fh)
        return data if isinstance(data, list) else [data]
    with path.open(newline="", encoding="utf-8") as fh:
        reader = csv.DictReader(fh)
        plan = plan_columns(reader.fieldnames or [], profile, store, root)
        records = []
        for row in reader:
            record: Record = {}
            for column, col in zip(plan, reader.fieldnames or []):
                raw = (row.get(col) or "").strip()
                if raw:
                    set_path(record, column, csv_value(column, raw))
            records.append(record)
    return records


def values_at(obj: Any, parts: list[str]) -> list[Any]:
    """All values at a dotted path, flattening through lists."""
    if not parts:
        return [obj]
    if isinstance(obj, list):
        return [v for item in obj for v in values_at(item, parts)]
    if isinstance(obj, dict) and parts[0] in obj:
        return values_at(obj[parts[0]], parts[1:])
    return []


def nonempty(value: Any) -> bool:
    if value is None:
        return False
    if isinstance(value, str):
        return bool(value.strip())
    if isinstance(value, (list, dict)):
        return len(value) > 0
    return True


def present(record: Record, path: str) -> bool:
    return any(nonempty(v) for v in values_at(record, path.split(".")))


def first(record: Record, path: str) -> Any:
    return next((v for v in values_at(record, path.split(".")) if nonempty(v)), None)


def strings_at(record: Record, path: str) -> list[str]:
    return [
        v
        for v in values_at(record, path.split("."))
        if isinstance(v, str) and v.strip()
    ]


def record_id(record: Record, id_path: str, index: int) -> str:
    return str(first(record, id_path) or f"record {index + 1}")


# --- layers ----------------------------------------------------------------


@dataclass
class StructureResult:
    valid: int | None  # None when the layer could not run
    problems: list[str]


@dataclass
class CompletenessResult:
    wanted: list[str]
    missing_by_record: dict[str, list[str]]
    missing_count: Counter


def layer_structure(
    records: list[Record], store: SchemaStore, root: str, id_path: str
) -> StructureResult:
    try:
        import jsonschema
        from referencing import Registry, Resource
        from referencing.exceptions import NoSuchResource, Unresolvable
        from referencing.jsonschema import DRAFT202012
    except ImportError:
        return StructureResult(
            None, ["jsonschema 4.18+ not installed; layer 1 skipped"]
        )
    root_schema = store.get(root)
    if root_schema is None:
        return StructureResult(None, ["schema not available; layer 1 skipped"])

    def strip(schema: Schema) -> Schema:
        # The files name a metaschema that jsonschema does not know; validate as 2020-12.
        return {k: v for k, v in schema.items() if k != "$schema"}

    def retrieve(uri: str) -> Resource:
        contents = store.get(uri.rsplit("/", 1)[-1])
        if contents is None:
            raise NoSuchResource(ref=uri)
        return Resource.from_contents(
            strip(contents), default_specification=DRAFT202012
        )

    validator = jsonschema.Draft202012Validator(
        strip(root_schema), registry=Registry(retrieve=retrieve)
    )
    problems: list[str] = []
    invalid: set[str] = set()
    for i, record in enumerate(records):
        rid = record_id(record, id_path, i)
        try:
            for err in validator.iter_errors(record):
                where = "/".join(str(p) for p in err.absolute_path)
                problems.append(f"{rid}: {where}: {err.message[:90]}")
                invalid.add(rid)
        except (Unresolvable, jsonschema.exceptions.SchemaError) as exc:
            problems.append(f"{rid}: validation failed: {exc}")
            invalid.add(rid)
    return StructureResult(len(records) - len(invalid), problems)


def layer_completeness(
    records: list[Record], profile: dict, level: str, id_path: str
) -> CompletenessResult:
    levels = profile["levels"]
    order = list(levels)
    wanted = [path for lvl in order[: order.index(level) + 1] for path in levels[lvl]]
    missing_by_record: dict[str, list[str]] = {}
    missing_count: Counter = Counter()
    for i, record in enumerate(records):
        missing = [path for path in wanted if not present(record, path)]
        if missing:
            missing_by_record[record_id(record, id_path, i)] = missing
            missing_count.update(missing)
    return CompletenessResult(wanted, missing_by_record, missing_count)


def layer_validity(records: list[Record], profile: dict, id_path: str) -> list[str]:
    rules = profile.get("rules", {})
    issues: list[str] = []
    seen: Counter = Counter()
    for i, record in enumerate(records):
        rid = record_id(record, id_path, i)
        seen[rid] += 1
        if first(record, id_path) and not IDNO.match(rid):
            issues.append(f"{rid}: identifier has characters outside A-Z, 0-9, _ . -")
        for path in rules.get("dates", []):
            issues += [
                f"{rid}: {path} {v!r} is not YYYY-MM-DD"
                for v in strings_at(record, path)
                if not DATE.match(v)
            ]
        for path in rules.get("periods", []):
            issues += [
                f"{rid}: {path} {v!r} is not an ISO 8601 or SDMX period"
                for v in strings_at(record, path)
                if not PERIOD.match(v)
            ]
        for start_path, end_path in rules.get("period_order", []):
            start, end = first(record, start_path), first(record, end_path)
            ordered_pair = (
                isinstance(start, str)
                and isinstance(end, str)
                and PERIOD.match(start)
                and PERIOD.match(end)
            )
            if ordered_pair and end < start:
                issues.append(
                    f"{rid}: {end_path} ({end}) is before {start_path} ({start})"
                )
        for path in rules.get("urls", []):
            issues += [
                f"{rid}: {path} {v!r} is not a URL"
                for v in strings_at(record, path)
                if not URL.match(v)
            ]
        for path, allowed in rules.get("vocabularies", {}).items():
            issues += [
                f"{rid}: {path} {v!r} is not in the profile vocabulary"
                for v in strings_at(record, path)
                if v not in allowed
            ]
        for path, n in rules.get("min_words", {}).items():
            issues += [
                f"{rid}: {path} has fewer than {n} words"
                for v in strings_at(record, path)
                if len(v.split()) < n
            ]
        for a, b in rules.get("not_equal", []):
            va, vb = first(record, a), first(record, b)
            if (
                isinstance(va, str)
                and isinstance(vb, str)
                and va.strip().lower() == vb.strip().lower()
            ):
                issues.append(f"{rid}: {b} repeats {a}")
        for path in rules.get("placeholders", []):
            issues += [
                f"{rid}: {path} is a placeholder ({v!r})"
                for v in strings_at(record, path)
                if v.strip().lower() in PLACEHOLDERS
            ]
    issues += [
        f"{rid}: identifier appears {n} times" for rid, n in seen.items() if n > 1
    ]
    return issues


# --- report ----------------------------------------------------------------


def report(
    *,
    dtype: str,
    root: str,
    n: int,
    level: str,
    profile: dict,
    structure: StructureResult,
    completeness: CompletenessResult,
    issues: list[str],
) -> None:
    print(f"Layer 1  Structure (World Bank {root})")
    if structure.valid is None:
        print(f"         {structure.problems[0]}")
    else:
        print(
            f"         {structure.valid} of {n} records are valid instances of the schema"
        )
        for problem in structure.problems[:20]:
            print(f"         {problem}")
    print(
        "         Note: the schemas require little more than an identifier and a title. Validity is not completeness.\n"
    )

    complete = n - len(completeness.missing_by_record)
    print(
        f"Layer 2  Completeness against profile {profile.get('name', dtype)!r}, "
        f"level {level!r} ({len(completeness.wanted)} fields)"
    )
    print(f"         {complete} of {n} records complete")
    reasons = profile.get("fields", {})
    for path, count in completeness.missing_count.most_common():
        print(f"         missing {path:<48} in {count} record(s)")
        if path in reasons:
            print(f"                 why: {reasons[path]}")
    for rid, missing in completeness.missing_by_record.items():
        print(f"         {rid}: {', '.join(missing)}")
    print()

    print(f"Layer 3  Validity: {len(issues)} issue(s)")
    for issue in issues:
        print(f"         {issue}")
    print()
    print(
        "Not checked here: whether the text is correct, whether title and abstract agree, whether a unit"
    )
    print(
        "matches the values, or whether the text is specific. Those need review or a semantic check"
    )
    print("(see Generative AI for Metadata Quality).")


# --- main ------------------------------------------------------------------


def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
    parser = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    parser.add_argument(
        "records", type=Path, help="JSON records (nested) or CSV (flat)"
    )
    parser.add_argument(
        "profile", type=Path, help="completeness profile JSON for the data type"
    )
    parser.add_argument(
        "--type",
        choices=sorted(TYPES),
        help='data type; default: the profile\'s "type"',
    )
    parser.add_argument(
        "--level", help="profile level to check up to; default: the first level"
    )
    parser.add_argument(
        "--schema-dir",
        type=Path,
        default=Path(__file__).with_name("wb-schemas"),
        help="folder holding the World Bank schema files (downloaded on first use)",
    )
    return parser.parse_args(argv)


def main(argv: list[str] | None = None) -> int:
    args = parse_args(argv)
    with args.profile.open(encoding="utf-8") as fh:
        profile = json.load(fh)
    dtype = args.type or profile.get("type")
    if dtype not in TYPES:
        warn(f"unknown data type {dtype!r}; choose from {sorted(TYPES)}")
        return 2
    level = args.level or next(iter(profile["levels"]))
    if level not in profile["levels"]:
        warn(f"unknown level {level!r}; profile has {list(profile['levels'])}")
        return 2
    root = TYPES[dtype]
    id_path = profile.get("rules", {}).get("id", "idno")

    store = SchemaStore(args.schema_dir)
    records = load_records(args.records, profile, store, root)
    print(f"{len(records)} {dtype} record(s) from {args.records}\n")

    structure = layer_structure(records, store, root, id_path)
    completeness = layer_completeness(records, profile, level, id_path)
    issues = layer_validity(records, profile, id_path)
    report(
        dtype=dtype,
        root=root,
        n=len(records),
        level=level,
        profile=profile,
        structure=structure,
        completeness=completeness,
        issues=issues,
    )

    failed = (
        bool(completeness.missing_by_record)
        or bool(issues)
        or (structure.valid is not None and structure.valid < len(records))
    )
    return 1 if failed else 0


if __name__ == "__main__":
    sys.exit(main())

Output for the example catalog at the foundational level:

Layer 1  Structure (World Bank timeseries-schema.json)
         5 of 5 records are valid instances of the schema
         Note: the schemas require little more than an identifier and a title. Validity is not completeness.
Layer 2  Completeness against profile 'recommended indicator profile', level 'foundational' (9 fields)
         4 of 5 records complete
         missing series_description.definition_long               in 1 record(s)
                 why: Search by meaning and answers both read this text; without it a system matches on the title alone (chapter 1).
         LF_UNEMP_PCT: series_description.definition_long
Layer 3  Validity: 0 issue(s)

At --level ai-ready the same catalog has 0 of 5 records complete, with methodology, universe, series_break, limitation, license, keywords, aliases, and contacts missing from every record. That is the normal result for a catalog that was documented for people and is now being prepared for software, and it is the work list for the AI-ready level.

Export the records. A catalog that uses the schemas, through NADA or the Metadata Editor, exports nested JSON, which works for every type. For indicators, a CSV with one row per series also works: the metadata template shows the layout, and list fields such as sources and geographic_units take ; between values.
Set the profile. Start from the profile for the type (indicator, microdata, geospatial, document). Each is a recommended default. It names the fields required at the foundational, AI-ready, and AI-native levels, gives the reason each field matters, and sets the validity rules (which fields are dates, which are URLs, which take a vocabulary, which need a minimum length). An organization can remove a field from a level, and the reason states what that costs. Keep the edited profile in version control; it is the organization's definition of "complete". Offices that document with the Metadata Editor can mirror the profile in its templates, which set the required fields at entry time and validate records before publication to NADA.

Run the check. For the other types, the same command takes the type's profile and a JSON export:

python check_metadata.py example_microdata.json  profile_microdata.json  --level ai-ready
python check_metadata.py example_geospatial.json profile_geospatial.json
python check_metadata.py example_document.json   profile_document.json

Assign the incomplete records to a curator and set a date. Re-run the check monthly at the target level and keep the counts. The trend is the evidence that the catalog is improving.

Worked example. The example organization exports its five series as example_catalog.csv and runs the check at the foundational level with the recommended indicator profile. All five records are valid instances of the schema. Four are complete; the unemployment rate, LF_UNEMP_PCT, has no long definition, and the output says why that matters: search by meaning and answers both read that text. The metadata lead assigns the record to the labour statistics unit with a two-week deadline. The team then runs the check at the AI-ready level and finds that no record is complete, because fields such as methodology, series_break, and license are empty everywhere. That list of eight fields times five records becomes the work plan for the quarter.

What this does not do. The three layers check form. A record can pass all of them with a definition that is wrong, a name that does not match the definition, a unit that does not match the values, text too vague to be useful, or keywords nobody searches for. Those are semantic checks: the program's Generative AI for Metadata Quality work scores records on completeness, semantic alignment, specificity, and consistency with a language model, the Metadata Reviewer puts its suggestions in front of a curator, and the curation cookbook describes the workflow. Run the three layers first, because a semantic check on records with empty fields has nothing to work with.

Recipe

1.2 Publish a schema.org Dataset record on every series page

Level: AI-ready
Skills: Web developer with access to the page template
Indicative time: Half a day for the template, then automatic

Result: every series page carries a block of structured data that dataset search engines and crawlers read. The block is generated from the same catalog record that the page shows, so the two can never disagree.

Schema.org is a vocabulary that search engines agree on for describing things on web pages. A Dataset record in that vocabulary, embedded in the page as JSON-LD (a block of JSON inside a script element), is how a dataset search engine learns what the page is about. The block for the running example's undernourishment series:

{
  "@context": "https://schema.org/",
  "@type": "Dataset",
  "@id": "https://stats.example/series/FS_UNDERNOURISH_PCT",
  "identifier": "FS_UNDERNOURISH_PCT",
  "name": "Prevalence of undernourishment",
  "description": "Share of the population whose habitual food consumption is insufficient to provide the dietary energy levels required to maintain a normal active and healthy life. Three-year average.",
  "url": "https://stats.example/series/FS_UNDERNOURISH_PCT",
  "keywords": ["food security", "undernourishment", "hunger", "SDG 2.1.1"],
  "inLanguage": "en",
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "isAccessibleForFree": true,
  "temporalCoverage": "2010/2024",
  "spatialCoverage": {
    "@type": "Place",
    "name": "National and regional"
  },
  "variableMeasured": {
    "@type": "PropertyValue",
    "name": "Prevalence of undernourishment",
    "unitText": "% of population"
  },
  "dateModified": "2025-03-14",
  "publisher": {
    "@type": "Organization",
    "name": "National Statistics Office (example)",
    "url": "https://stats.example/"
  },
  "distribution": [
    {
      "@type": "DataDownload",
      "encodingFormat": "text/csv",
      "contentUrl": "https://stats.example/api/v1/series/FS_UNDERNOURISH_PCT/data.csv"
    },
    {
      "@type": "DataDownload",
      "encodingFormat": "application/json",
      "contentUrl": "https://stats.example/api/v1/series/FS_UNDERNOURISH_PCT/data.json"
    }
  ]
}

Offices that run NADA already have this: NADA writes the block on every catalog page from the stored record (the World Bank Microdata Library pages show it), and the Metadata Editor can export schema.org, Croissant, and DCAT from the same records. The steps below are for an organization with its own website.

Map the catalog fields to the schema.org properties as in the example. The required properties are name, description, and url; the others make the record useful.
Insert the block in the page template inside a script element with type="application/ld+json", filled from the catalog at render time.
Validate one page with the Schema Markup Validator.
Add the series pages to the XML sitemap and confirm that robots.txt does not block them.
Check after a few weeks whether the pages appear in a dataset search engine. Indexing takes time.

Worked example. The block for the undernourishment series names the dataset "Prevalence of undernourishment", gives its description from the catalog record, its URL at https://stats.example/series/FS_UNDERNOURISH_PCT, its licence as CC BY 4.0, its temporal coverage as 2010 to 2024, and the organization as the creator. The web developer adds the block to the series page template and fills it from the catalog at render time. The Schema Markup Validator parses it without errors. Three weeks later the series page appears in a dataset search engine under its own name, which it had never done before.

For catalogs that already run a data portal, check whether the portal software can emit DCAT for the whole catalog. DCAT and schema.org markup can coexist. For datasets that are published for model training, Croissant extends the same schema.org record with the fields machine-learning tools need to load the data.

What this does not do. Markup makes a page readable by crawlers; it does not get the page indexed or ranked. Indexing depends on the sitemap, crawlability, and the search engine's own policies, and takes weeks, so the check in step 5 is the only evidence that it worked.

Recipe

1.3 Add semantic search next to keyword search

Level: AI-native
Skills: Developer with Python; a server with a few GB of memory
Indicative time: One to two weeks including evaluation

Result: the catalog search returns the unemployment rate for the query "jobless rate" and the undernourishment series for "how many people go hungry", because it matches by meaning. The keyword search stays in place for exact codes and names.

Semantic search matches by meaning. A language model turns each record and each query into a list of numbers (an embedding) such that texts with similar meaning come out close together, even when they share no words. The search then ranks records by that closeness. The snippet below does this for a small catalog with an open multilingual model that runs on a CPU:

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("intfloat/multilingual-e5-small")
records = [...]  # list of dicts with id, title, description
texts = ["passage: " + r["title"] + ". " + r["description"] for r in records]
index = model.encode(texts, normalize_embeddings=True)

def semantic_search(query, k=10):
    q = model.encode("query: " + query, normalize_embeddings=True)
    scores = util.cos_sim(q, index)[0]
    top = scores.argsort(descending=True)[:k]
    return [records[i]["id"] for i in top]

NADA AI provides semantic search over a NADA catalog's records and is being integrated into NADA, so an organization that runs NADA gets this layer with the catalog. The steps below are for an organization with its own search, and the evaluation in step 4 applies to both.

Build the question set and baseline scores first (chapter 6). Without them there is no way to tell whether the new search is better.
Embed each catalog record (title plus description) as in the snippet.
Combine the semantic ranking with the keyword ranking, for example by taking the best rank of each record across both lists. Keep the keyword path so that an exact identifier always wins.
Re-run the scoring script from chapter 6 and compare Recall@5 and MRR per language with the baseline. Keep the result with the date and model name.
Re-embed the catalog whenever records change, on a schedule.

Worked example. The example organization embeds its five records with the multilingual model in the snippet. For the query "how many people go hungry", the keyword search returns nothing, because no record contains those words. The semantic search ranks FS_UNDERNOURISH_PCT first, with a similarity score of about 0.8, and the youth unemployment series last. For the query "LF_UNEMP_PCT" the keyword search returns the exact record and the combined ranking keeps it at the top. On the question set of chapter 6, Recall@5 for the English paraphrases rises from 0.2 with keywords alone to 0.9 with the combination.

What this does not do. The snippet embeds each record as one string and ranks by similarity. A production search also indexes the keyword field and aliases, handles thousands of records with a vector index, filters by topic and geography, re-ranks the top results, and logs queries for the monthly review (chapter 4). General embedding models also underperform on structured records with short labels, which is the problem the program's Data Discoverability work and the PI-FT pipeline address by fine-tuning an embedding model on structured records.

Implementation options

Catalog records

The World Bank metadata schemas define records for indicators, indicator databases, microdata (based on DDI Codebook), documents (Dublin Core), geospatial data (ISO 19115/19139), tables, images (IPTC), and scripts, with a DataCite block for DOIs. They come with a Python library and Excel templates.
NADA, the open-source catalog that runs the World Bank Microdata Library, stores records in these schemas and supports DDI Codebook, Dublin Core, ISO 19115/19139, and IPTC.
Each record gets a persistent identifier, with a DOI through DataCite where the organization can register one (chapter 5).

Machine-readable catalog metadata for the web

DCAT (W3C) describes catalogs, datasets, and distributions. It is the base for many open data portals.
schema.org Dataset markup in JSON-LD on dataset pages is read by general web crawlers and dataset search engines. Croissant extends it for machine-learning datasets.
DDI describes surveys and microdata. SDMX describes aggregate statistical data and its structure.

Search

Keyword search (BM25) remains valuable for exact codes, acronyms, and names.
Dense retrieval with embedding models matches by meaning. Combining both usually gives the best results.
Multilingual embedding models support queries in languages other than the one the metadata is written in. Translated metadata still improves results.

Metadata quality

Start with a completeness report, then move to consistency: does the title describe the same thing as the definition?
Use an LLM to draft missing descriptions or flag inconsistencies, and keep a person responsible for approval.

World Bank examples

The metadata schemas of the Development Data Group define the record fields used in this chapter, and the Microdata Library runs on NADA with DDI records and DataCite DOIs.
Generative AI for Metadata Quality scores indicator metadata on completeness, semantic alignment, specificity, and consistency.
Metadata Augmentation groups survey variables into DDI-style themes.
Data Discoverability describes semantic search over indicator catalogs, and the PI-FT toolkit fine-tunes embedding models for structured metadata.

Resources

World Bank metadata schemas: the record schemas for every data type, the JSON Schema files the checker validates against, and the Python library.
Metadata Editor user and practice guide: documenting records in these schemas with templates and validation, and publishing to NADA.
schema.org Dataset, Google's dataset structured-data guide, and the Rich Results Test: the markup and the tool to verify it on a page.
DCAT 3 and Croissant: catalog and machine-learning dataset descriptions, exported by the Metadata Editor.
Sentence Transformers and rank_bm25: the embedding and keyword components of the search in recipe 1.3; the PI-FT toolkit for fine-tuning.
Metadata checker and the profiles: the three-layer check of recipe 1.1.

Common mistakes

Launching semantic search over incomplete metadata. The model can only match what the record says. Recipe 1.1 comes first.
Redesigning the website and changing every URL. Crawlers, citations, and AI systems lose the pages. Keep identifiers and set redirects for anything that must move.
Translating metadata by machine without review. A wrong translation of a definition is worse than none. Translate titles first and have them checked.
Measuring once. Scores drift as the catalog and models change. Keep the question set and re-run it.

Verification

Completeness. The share of records that pass the schema and the organization's profile at the target level is reported, with the count of missing values per field (recipe 1.1).
Known-item queries. For 30 to 50 queries whose correct series is known, written the way users write, the correct series appears in the top 5 and the top 10 results (chapter 6).
Crawler view. A dataset page opened in a structured-data validator shows the record parsed without errors.
Language check. The known-item queries are repeated in each supported language with comparable results.

Rationale​

Target state​

Maturity levels​

Recipes​

Implementation options​

World Bank examples​

Resources​

Common mistakes​

Verification​

Checklist​

Rationale

Target state

Maturity levels

Recipes

Implementation options

World Bank examples

Resources

Common mistakes

Verification

Checklist