6. Detection of mentions

Question: Can mentions of the data be found in documents?

Rationale

Keyword search finds the canonical name and nothing else. Documents refer to a survey as "the labour survey", to an indicator by its definition, and to a report by its author; they name the dataset once and use a pronoun after. Named-entity extraction finds these mentions from the text around them, with a confidence, and records whether the document uses the data or cites it. The program's extractor does this with a zero-shot model and a fine-tuned adapter for dataset mentions.

An extractor's output is only as good as its measured precision and recall, so the chapter pairs the model with a labelled sample.

Target state

• Mentions are extracted from every collected document with the dataset name, acronym, producer, usage flag, context, and confidence.
• Precision and recall are measured on a labelled sample across document types and languages, and reported.
• Thresholds and deduplication are tuned on the sample, and the sample grows with each review.

Maturity levels

Level	Steps
Foundational	Search the collected text for the canonical names and acronyms. Read the hits and record the mentions with their context.
AI-ready	Run the extractor on every document. Label a sample of sentences and measure precision and recall. Set the confidence threshold and deduplicate.
AI-native	Run extraction in the pipeline on each collection run, keep the labelled sample growing from reviews, and re-measure when the model or adapter changes.

Running example

The extractor found twelve mentions in the six documents. Against the labelled sample of fourteen sentences, it found eleven of the twelve labelled mentions, missed "Population and Housing Census 2014", and extracted one generic phrase ("food security report" in the regional outlook) that a person did not count as a mention.

Recipes

Recipe

6.1 Run the extractor on the collected documents

Level

AI-ready

Skills

Developer with Python; a machine that runs a small language model

Indicative time

One day to set up; seconds per document

Result: for every document in the collection, the organization has one record per mention of a dataset, with the fields the later chapters need.

The program's extractor reads a document's text and finds the places where a dataset is named. It uses a zero-shot named-entity model (a model that finds names of a given kind in text without having been trained on the organization's documents) with the program's adapter for dataset mentions. For each mention it returns the dataset name, the acronym and the producer where present, whether the data are used, the usage context, and a confidence. The code to run it on one document:

from ai4data.data_use import extract_from_document

result = extract_from_document("D01.pdf", include_confidence=True)
for mention in result["dataset_mention"]:
    print(mention["dataset_name"]["text"], mention["is_used"], mention["usage_context"])

The mentions are stored one per line as JSON objects:

Field	Meaning
document_id	The document the mention was found in
text	The mention as written
context	The sentence it was found in
dataset_tag	named or unnamed
is_used, usage_context	Whether the data are used, and as primary, secondary, or background (chapter 1)
confidence	The extractor's confidence, 0 to 1

{"document_id": "D01", "text": "Household Budget Survey 2021", "context": "We use the Household Budget Survey 2021 to estimate consumption per adult equivalent.", "dataset_tag": "named", "is_used": true, "usage_context": "primary", "confidence": 0.93}
{"document_id": "D01", "text": "HBS", "context": "The HBS sample covers 6,000 households in both urban and rural areas.", "dataset_tag": "named", "is_used": true, "usage_context": "primary", "confidence": 0.81}
{"document_id": "D01", "text": "prevalence of undernourishment series", "context": "Our estimates are consistent with the official prevalence of undernourishment series.", "dataset_tag": "unnamed", "is_used": false, "usage_context": "background", "confidence": 0.64}
{"document_id": "D02", "text": "National Statistics Office food security report 2025", "context": "As the National Statistics Office food security report 2025 shows, undernourishment fell to 7.6 percent.", "dataset_tag": "named", "is_used": false, "usage_context": "background", "confidence": 0.77}
{"document_id": "D02", "text": "Prevalence of undernourishment (FS_UNDERNOURISH_PCT)", "context": "Targets are set against the Prevalence of undernourishment (FS_UNDERNOURISH_PCT) indicator.", "dataset_tag": "named", "is_used": true, "usage_context": "primary", "confidence": 0.90}
{"document_id": "D03", "text": "Labour Force Survey", "context": "According to the Labour Force Survey, unemployment fell to 6.1 percent in the second quarter.", "dataset_tag": "named", "is_used": true, "usage_context": "primary", "confidence": 0.95}
{"document_id": "D03", "text": "the labour survey", "context": "The labour survey also shows a rise in informal employment.", "dataset_tag": "unnamed", "is_used": true, "usage_context": "primary", "confidence": 0.58}
{"document_id": "D04", "text": "Household Budget Survey (HBS) 2021", "context": "We combine the Household Budget Survey (HBS) 2021 with anthropometric data.", "dataset_tag": "named", "is_used": true, "usage_context": "primary", "confidence": 0.92}
{"document_id": "D04", "text": "Demographic and Health Survey 2016", "context": "Anthropometric measures come from the Demographic and Health Survey 2016.", "dataset_tag": "named", "is_used": true, "usage_context": "secondary", "confidence": 0.90}
{"document_id": "D05", "text": "unemployment rate from the LFS", "context": "The unemployment rate from the LFS stood at 6.3 percent at the start of the year.", "dataset_tag": "named", "is_used": true, "usage_context": "primary", "confidence": 0.71}
{"document_id": "D06", "text": "World Development Indicators", "context": "Regional comparisons draw on the World Development Indicators.", "dataset_tag": "named", "is_used": true, "usage_context": "secondary", "confidence": 0.90}
{"document_id": "D06", "text": "food security report", "context": "Each country publishes a food security report with its own methodology.", "dataset_tag": "unnamed", "is_used": false, "usage_context": "background", "confidence": 0.52}

1. Install the program's package with the data-use dependencies (pip install "ai4data[datause]") and run the extractor on the text of each document.
2. Store the mentions in the form above.
3. Record the model, adapter, and threshold used with each run. A change in any of them changes the counts.

Worked example. Run on the 2023 journal article, the extractor returns two mentions. The first is "Household Budget Survey 2021" in the sentence "We estimate poverty from the Household Budget Survey 2021", tagged as named, with is_used true, the usage context "primary", and a confidence of 0.97. The second is "HBS" in the sentence "Weights from the HBS are applied throughout", also used, with a confidence of 0.91. Run on the regional food security outlook, it returns "food security report" in the sentence "as several food security reports in the region have noted", with a confidence of 0.52, which the labelled sample of recipe 6.2 shows to be a false positive. The run is recorded with the model, the adapter version, and no threshold applied yet.

What this does not do. The extractor finds mentions in the text it is given. A document without a text layer yields nothing, and a mention in a figure or in a reference list stored as an image is invisible. The model was trained on English development literature; other languages need the measurement of recipe 6.2 before the counts are trusted.

Recipe

6.2 Measure precision and recall on a labelled sample

Level

AI-ready

Skills

Dissemination officer for labelling; Python

Indicative time

One day for the first hundred sentences

Result: the organization has two numbers that say how good the extractor is on its own documents, together with the lists of what it missed and what it wrongly extracted.

Precision is the share of extracted mentions that are real; recall is the share of real mentions that were extracted. Both need a labelled sample: sentences a person has read and marked. The sample is a CSV with one row per sentence:

Column	Meaning
document_id	The document the sentence comes from
sentence	The sentence
mention_text	The dataset mention it contains, or empty when it contains none
note	The labeller's remark

document_id,sentence,mention_text,note
D01,We use the Household Budget Survey 2021 to estimate consumption per adult equivalent.,Household Budget Survey 2021,
D01,The HBS sample covers 6000 households in both urban and rural areas.,HBS,acronym
D01,Our estimates are consistent with the official prevalence of undernourishment series.,prevalence of undernourishment series,unnamed mention of a series
D02,As the National Statistics Office food security report 2025 shows undernourishment fell to 7.6 percent.,National Statistics Office food security report 2025,a document cited as a source
D02,Targets are set against the Prevalence of undernourishment (FS_UNDERNOURISH_PCT) indicator.,Prevalence of undernourishment (FS_UNDERNOURISH_PCT),
D03,According to the Labour Force Survey unemployment fell to 6.1 percent in the second quarter.,Labour Force Survey,
D03,The labour survey also shows a rise in informal employment.,the labour survey,
D04,We combine the Household Budget Survey (HBS) 2021 with anthropometric data.,Household Budget Survey (HBS) 2021,
D04,Anthropometric measures come from the Demographic and Health Survey 2016.,Demographic and Health Survey 2016,external dataset
D04,Sampling weights follow the Population and Housing Census 2014 frame.,Population and Housing Census 2014,missed by the extractor
D05,The unemployment rate from the LFS stood at 6.3 percent at the start of the year.,unemployment rate from the LFS,
D06,Regional comparisons draw on the World Development Indicators.,World Development Indicators,external dataset
D06,Each country publishes a food security report with its own methodology.,,generic phrase; not a mention of a specific dataset
D06,The outlook covers twelve countries in the region.,,no mention

The script compares the extractor's output with the sample:

"""Score mention extraction against a labelled sample.

Reads the extractor's mentions (JSON lines) and a labelled sample (CSV:
document_id, sentence, mention_text, with mention_text empty when the
sentence contains no dataset mention) and reports precision, recall, and
F1 at the level of (document, mention text), with the false positives and
false negatives listed. Mentions are compared after normalization (lower
case, punctuation removed). Uses pandas.

Usage:
    python score_mentions.py mentions_example.jsonl labelled_sample.csv

What this does not do: fourteen labelled sentences give a demonstration.
A measure needs a sample of a few hundred sentences drawn across document
types, with two annotators and an agreement check, and the comparison
should also score the usage flag and the context, which this script leaves
out.
"""

from __future__ import annotations

import argparse
import re
import sys

import pandas as pd

WORD = re.compile(r"[a-z0-9]+")


def norm(text: str) -> str:
    return " ".join(WORD.findall(text.lower()))


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    parser.add_argument("mentions")
    parser.add_argument("labelled")
    args = parser.parse_args(argv)
    mentions = pd.read_json(args.mentions, lines=True)
    labelled = pd.read_csv(args.labelled, dtype=str).fillna("")

    found = set(zip(mentions["document_id"], mentions["text"].map(norm)))
    expected = {
        (d, norm(t))
        for d, t in zip(labelled["document_id"], labelled["mention_text"])
        if t.strip()
    }
    tp, fp, fn = found & expected, found - expected, expected - found
    precision = len(tp) / len(found) if found else 0.0
    recall = len(tp) / len(expected) if expected else 0.0
    f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0

    print(
        f"labelled mentions {len(expected)}, extracted {len(found)}, correct {len(tp)}"
    )
    print(f"precision {precision:.2f}  recall {recall:.2f}  F1 {f1:.2f}")
    for title, pairs in (
        ("false positives (extracted, not labelled as a mention):", fp),
        ("false negatives (labelled, not extracted):", fn),
    ):
        if pairs:
            print(title)
            for d, t in sorted(pairs):
                print(f"  {d}: {t!r}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Output for the running example:

labelled mentions 12, extracted 12, correct 11
precision 0.92  recall 0.92  F1 0.92
false positives (extracted, not labelled as a mention):
  D06: 'food security report'
false negatives (labelled, not extracted):
  D04: 'population and housing census 2014'

The first two lines give the counts and the three measures. Precision is the share of extracted mentions that the labellers also marked, recall is the share of labelled mentions that the extractor found, and F1 is their harmonic mean. The two lists that follow name the disagreements in each direction, which is where the review starts.

1. Draw sentences from the collected documents across types and languages, and label each. Two labellers and an agreement check make the sample trustworthy.
2. Run the script.
3. Read both lists. Here the false positive is a generic phrase with low confidence (0.52), which a threshold removes. The false negative is a census named as a sampling frame, which the adapter may treat as a method; a review adds it to the sample as a positive example.
4. Keep the sample and the scores with the model version, and re-run on every change.

Worked example. The false negative in the run above is the sentence "The sampling frame was the Population and Housing Census 2014" in the working paper. The extractor did not mark it, and the two labellers disagreed about it at first: one held that a census used as a frame is not "used" in the typology's sense, the other that it is a mention whatever the use. They settled on marking it as a mention with the context "background", which is what the sample now records, and the sentence is added as a positive example. The false positive, "food security report" in a sentence about reports in general, is a phrase the extractor took for a document name. Its confidence of 0.52 is the lowest in the run, which is what the threshold of recipe 6.3 uses.

What this does not do. Fourteen sentences demonstrate the measure. A few hundred sentences per document type and language, with two labellers, give a number to report. The script scores the mention text only; the usage flag and the context are scored by review.

Recipe

6.3 Set thresholds and deduplicate

Level

AI-ready

Skills

Developer

Indicative time

Half a day

Result: the confidence threshold is chosen on the sample for the use the counts serve, and each document has one record per distinct mention.

1. Compute precision and recall on the sample at several thresholds of the extractor's confidence, and choose the threshold for the use the counts serve: a higher one for a published report, a lower one for a review queue.
2. Deduplicate within a document with the program's deduplicate_extraction, which merges repeated mentions of the same dataset, clusters acronyms with their long forms ("HBS" with "Household Budget Survey 2021"), and keeps the context windows.
3. Keep the mention count after deduplication as the secondary figure next to the document count (chapter 1, recipe 1.3).

Worked example. On the labelled sample, a threshold of 0.5 gives a precision of 0.92 and a recall of 0.92. Raising it to 0.6 removes the "food security report" false positive and gives a precision of 1.00 with the same recall, because no true mention in the sample scored below 0.6. The developer sets 0.6 for the published report and 0.4 for the review queue, where a missed mention costs more than a wrong one. Deduplication then merges the article's "Household Budget Survey 2021" and "HBS" into one mention with both forms and both context windows, so that the article counts as one use of the survey with two mentions.

What this does not do. A threshold trades precision for recall; it does not improve the model. Recurring misses of one kind (datasets named as frames, mentions in the national language) call for additions to the labelled sample and, at the AI-native level, for adapting the model.

Implementation options

• Extractor: the program's ai4data.data_use package (GLiNER with the ai4data/datause-extraction adapter), run locally; keyword search over canonical names as the foundational method.
• Labelling: a spreadsheet with document, sentence, and mention; two labellers; agreement measured.
• Measures: precision, recall, and F1 at the mention level; the same per document type and language.
• Deduplication: the program's deduplicate_extraction (overlap filtering, acronym clustering, fuzzy clusters).

World Bank examples

• Monitoring of Data Use documents the extractor, its output fields, and the deduplication step, with the implementation files in the repository.
• The program publishes its extraction adapter on Hugging Face (ai4data/datause-extraction) for reuse.

Resources

• Monitoring of Data Use: the program's extraction package, with extract_from_text, extract_from_document, and deduplicate_extraction.
• GLiNER and the ai4data/datause-extraction adapter: the zero-shot named-entity model and the adapter trained for dataset mentions.
• Label Studio: open-source text annotation for the labelled sample, with inter-annotator agreement.
• scikit-learn classification metrics: precision, recall, and F1 as computed in the scoring script, with the definitions.
• Coleridge Initiative "Show US the Data" competition: a public benchmark of dataset-mention detection in papers, for comparison of methods.

Common mistakes

• Trusting the model's scores. A reported recall from another corpus is not the organization's recall; measure on its documents.
• Threshold chosen by eye. The sample shows what each threshold costs.
• Counting duplicates. The same survey mentioned ten times in one paper is one use.
• English only. A labelled sample in the national language is the only evidence that the extractor works there.

Verification

• Extraction. Every collected document has a mentions file, with the model, the adapter, and the threshold recorded.
• Measurement. Precision and recall per document type and language have been measured on the labelled sample, with the date.
• Deduplication. A sample of documents shows one record per distinct mention.

Checklist

0 of 6 done

• Extractor run on every collected document
• Mentions stored as JSON lines with document, text, context, and fields
• Model, adapter, and threshold recorded per run
• Labelled sample across document types and languages, two labellers
• Precision and recall measured and kept with the model version
• Threshold chosen on the sample; mentions deduplicated per document

Ticks are saved in this browser only.