AI for Data - Data for AI

The Challenge

AI is becoming how people reach development data

People increasingly put questions about poverty, employment, climate, and food security to AI systems. Whether the answers draw on official statistics depends on whether AI can find, interpret, and verify the data. When it cannot, AI relies on secondary sources that may be outdated or wrong. Official statistics remain the most trusted evidence, and they lose influence when AI cannot reach them.

Find

Metadata is incomplete or inconsistent

AI cannot reliably find relevant data when descriptions are missing, vague, or contradictory.

Interpret

Context is limited

Semantic context is thin and data models differ across organizations, so AI can misread what a number means.

Access

Data is hard for AI to retrieve

Many statistics are published as PDFs and spreadsheets. Where an API exists, each organization has its own, so an AI system needs a separate integration for every source.

Verify

Answers cannot be traced

AI answers rarely link back to the authoritative figure, so outdated or wrong values go unnoticed.

Examples

See the tools at work

Five short examples show what the program's tools do and how they could fit your work. Scroll to step through them, and select anything in a panel to try it. Parts that are illustrations are labeled.

Example 1 of 5Data Snapshots

A PDF page enters the pipeline. Select a step or press Run.

PDF page

Page 12 of a World Bank working paper with a bar chart

Snapshot

No region yet

Structured data

Not extracted yet

Page, bounding box, and snapshot are from the ai4data/data-snapshot dataset. The page is from a working paper by Beck, Klapper, and Mendoza. The structured output is an illustration.

GDP growth (annual %), Nigeria

Classificationexternal_driver

Confidence0.92

Evidence strengthstrong_direct

Explanation

Nigeria experienced a sharp GDP contraction in 2016, , driven by the beginning in 2014–2015 and .

CNBC31 Aug 2016Justina Crabtree, special to CNBC.com

Nigeria’s economy drops into recession ↗

Nigeria’s statistics office said Wednesday that the country has dropped into recession as its all-important oil industry has suffered under weak global prices. […] An economic adviser to President Muhammadu Buhari, Adeyemi Dipeolu, told the Associated Press that the bleak data was largely attributed to “a sharp contraction in the oil sector due to huge losses of crude oil production,” resulting from vandalism and “sabotage.”

Supported. The excerpt says the oil industry “has suffered under weak global prices.”

The CNBC excerpt was added to illustrate checking a cited event.

How fast did the Philippines’ economy grow in 2024?

Show the tagged model output

The Philippines' economy grew by <claim id="0328">5.7</claim>% in 2024.

Show the claim payload

{
  "claim_id": "0328",
  "indicator": "GDP growth (annual %)",
  "country": "Philippines",
  "date": "2024",
  "value": 5.69201612823412,
  "unit": "Percentage",
  "source": "World Development Indicators"
}

Simplified for illustration. The application passes this record to the model, which cites it by claim ID.

Data Snapshots

From a PDF figure to structured data

A layout detection model locates a figure or table on a PDF page. The region is saved as a data snapshot, and the snapshot is converted to structured data that other tools can search and compute on.

A PDF page enters the pipeline. Select a step or press Run.

PDF page

Snapshot

No region yet

Structured data

Not extracted yet

Page, bounding box, and snapshot are from the ai4data/data-snapshot dataset. The page is from a working paper by Beck, Klapper, and Mendoza. The structured output is an illustration.

Read the benchmark paper →

Metadata Reviewer

Five-agent metadata review pipeline

Detection, filtering, classification, and scoring run as specialized, sequential agents. Each step has a narrow task, which keeps the output consistent and auditable across thousands of records.

A metadata record enters the pipeline. Select a stage or press Run.

Metadata record

name: GDP per capita (curent US$)
measurement_unit: Constant 2015 US$
time_coverage: 1960-2023
idno: WB_WDI_NY.GDP.PCAP.CD

Illustrative run on the record from the quick-start guide.

Issues

Waiting for the first agent.

See the full pipeline reference →

Anomaly Detection and Explanation

Evidence-backed anomaly explanations

A statistical detector flags a data point as unusual. The elicitation pipeline turns that flag into a structured, evidence-backed explanation with a classification, a confidence score, and a cited source. A reviewer can check it in seconds.

GDP growth (annual %), Nigeria

Classificationexternal_driver

Confidence0.92

Evidence strengthstrong_direct

Explanation

Nigeria experienced a sharp GDP contraction in 2016, , driven by the beginning in 2014–2015 and .

CNBC31 Aug 2016Justina Crabtree, special to CNBC.com

Nigeria’s economy drops into recession ↗

Nigeria’s statistics office said Wednesday that the country has dropped into recession as its all-important oil industry has suffered under weak global prices. […] An economic adviser to President Muhammadu Buhari, Adeyemi Dipeolu, told the Associated Press that the bleak data was largely attributed to “a sharp contraction in the oil sector due to huge losses of crude oil production,” resulting from vandalism and “sabotage.”

Supported. The excerpt says the oil industry “has suffered under weak global prices.”

The CNBC excerpt was added to illustrate checking a cited event.

See the full input/output schema →

Data Discoverability & MCP

Search and MCP access to the catalog

The catalog can be searched in plain language, and the results are ranked by meaning. A result can match a query without sharing any of its words. The same query can also be sent by an AI agent as a Model Context Protocol (MCP) tool call.

Model avsolatorio/GIST-all-MiniLM-L6-v2Index names of 1,498 WDI indicatorsValues WDI, latest year, Philippines

See the MCP integration guide →

Proof-Carrying Numbers

Number verification in chatbot answers

In a chatbot answer built on official statistics, each number carries a tag naming the record it came from. The application compares the number with that record and shows a check mark when they match. The application draws the check mark, so the model cannot produce one. Numbers that differ from the record are flagged, and numbers without a tag receive no mark.

How fast did the Philippines’ economy grow in 2024?

Show the tagged model output

The Philippines' economy grew by <claim id="0328">5.7</claim>% in 2024.

Show the claim payload

{
  "claim_id": "0328",
  "indicator": "GDP growth (annual %)",
  "country": "Philippines",
  "date": "2024",
  "value": 5.69201612823412,
  "unit": "Percentage",
  "source": "World Development Indicators"
}

Simplified for illustration. The application passes this record to the model, which cites it by claim ID.

Read the PCN paper →

The Program

AI for Data and Data for AI

AI for Data applies AI to the production, curation, and dissemination of data. Data for AI prepares data for use by AI systems. Both aim at development data that AI can find, interpret, and verify. All methods, software, and guidance are developed as open resources.

The program builds on the FAIR principles and extends them for AI systems as consumers of data. Browse the workstreams by pillar or by the AI-ready dimension they improve. The program also develops an AI-readiness assessment framework for national statistical organizations.

Show GSBPM tags

Apply AI to produce, curate, and disseminate development data with less manual effort and higher quality.

Data production

Statistical classification and codingAI-assisted coding of survey responses and records against standard classifications, with human review of the results.

Data SnapshotsLayout detection models that locate figures and tables in PDF documents, so that the data they contain can be extracted. Includes a benchmark of open-source models.

Small and Agentic AISmall language models and agentic workflows for statistical processes and specialized tasks, and AI assistants built on them.

Synthetic dataGenerates synthetic tabular and relational data with REaLTabFormer, an open-source transformer model, for data sharing and methodological research.

Data quality and metadata

Generative AI for Metadata QualityScores metadata on four dimensions: completeness, semantic alignment, specificity, and consistency. The LLM output is structured and auditable.Metadata AugmentationOrganizes hundreds of survey variables into DDI-style thematic groups using semantic clustering.Anomaly Detection and ExplanationClassifies flagged anomalies as an external driver, data error, measurement update, or insufficient data, and cites the evidence.

Responsible AI guidanceGuidance and evaluation methods for using AI in official statistics, aligned with the UN Fundamental Principles of Official Statistics.

Discovery and trustworthy dissemination

Data DiscoverabilityConceptual search returns food insecurity indicators for the query “how many people go hungry”, although none of their titles contains those words.Proof-Carrying NumbersChecks each number in a chatbot answer against the official record and marks it as verified or flagged.Monitoring of Data UseExtracts dataset mentions from reports and papers with zero-shot NER, including name variants such as “WDI” and “World Development Indicators.”

Methods and open tools

Inclusive AI ApplicationsMultilingual embedding models covering 50+ languages, and batch inference at roughly half the cost of synchronous calls.

Open benchmarks and evaluationReusable benchmarks and evaluation methods for semantic search, metadata quality, and trustworthy dissemination.

GSBPM tags show where a workstream applies in the Generic Statistical Business Process Model. See the full mapping →

Get Involved

Open-source tooling and documentation

Methods, software, and guidance are developed as open resources, with documentation for each released workstream.

Explore the Repository

Making development data AI-ready.