Practical Guide to Extracting Data from Documents
A cookbook for dissemination, library, and research data teams. Version 0.1 (working draft), October 2026.
This guide is a working draft from the AI for Data – Data for AI program. It collects practical approaches and open standards. It does not represent official World Bank Group guidance or policy and is not legal advice; an organization applies it under its own statistical law, data protection rules, and institutional policies. Comments and corrections are welcome as GitHub issues, and the contributing page explains how to propose a recipe.
A large share of the statistics a national statistical organization has ever published exists only inside documents: reports, bulletins, yearbooks, census volumes. The tables and charts in them are read by people and copied by hand. Software cannot query them, an AI system that reads the PDF will transcribe them with errors, and nothing links a number quoted from a report back to the page it came from.
This guide is organized around nine questions that a team can ask about the data in its documents. The questions follow the order of the work. The first two are about the documents themselves: which of them hold data, and whether scanned documents can be read reliably. The next three are about getting the data out: whether the tables and figures can be located, whether the data can be extracted into a structured form, and whether the extracted values can be verified against the document. The last four are about what happens afterwards: whether tables from many documents can be combined into one series, whether the data can be published next to the document with their provenance, whether they can be found and cited back to the page, and whether the pipeline can run over the whole backlog. Each chapter answers one question with recipes, maturity levels, worked examples, and a checklist.
The recipes build on the program's Data Snapshots work, which benchmarks open-source layout detection models on institutional documents and publishes the annotated dataset, and on the World Bank document and table schemas, which hold the records for a document and for a table extracted from it. The standards page lists each standard and where it applies.
An extracted value without a check is a transcription. The recipes in chapter 5 use what the document itself offers (printed value labels, row totals, derived rates) to verify values, and the tidy files carry the result per value. Values that cannot be verified are published as estimates and say so.
Prerequisites
The recipes assume the following are in place or can be arranged:
- The publications the organization has produced, as PDF files, and a list of them (a publication register or a folder).
- A person who can run Python scripts and install open-source tools (OCR engines, layout models), and a dissemination officer who knows which tables users ask for.
- For the AI-ready level, a machine with a GPU or patience for layout detection, and access to a vision-language model for charts; for the AI-native level, a developer to run the pipeline.
First steps
For a team with limited time, this sequence gives the most useful result first:
- Build the inventory of chapter 1. It takes a day, and it says where the data are, which files are scans, and how large the work is.
- Pick the five most requested tables, extract them from the text layer as chapter 4 describes, and run the verification of chapter 5. This takes two days. Then publish them next to the documents with their provenance, following chapter 7.
- For scanned documents, run OCR on one priority document and score it, following chapter 2, before investing in the rest.
Layout detection (chapter 3), series across documents (chapter 6), and the pipeline (chapter 9) come once the first tables are published and used.
Self-assessment
The self-assessment takes about five minutes and places a document collection at a level in each chapter.
Chapters and questions
| Chapter | Question | Practical topics |
|---|---|---|
| 1. Documents that hold data | Which documents hold data, and which data? | Inventory, document records, text layers and OCR, links to series and surveys |
| 2. Text from scanned documents | Can scanned documents be read reliably? | Scans in the inventory, OCR engines per language, quality score per page, searchable PDF/A |
| 3. Location of tables and figures | Can the tables and figures be located on the page? | Layout detection, data snapshots, bounding boxes, review against annotations |
| 4. Extraction of structured data | Can the data be extracted into a structured form? | Table extraction, chart reading, units and notes, tidy files with provenance |
| 5. Verification of extracted values | Can the extracted values be verified against the document? | Printed labels, totals, derived rates, sample review, status per value |
| 6. Series across documents | Can tables from many documents be combined into one series? | Table-to-series mapping, revisions and confirmations, breaks, series record |
| 7. Publication next to the document | Can the extracted data be published with their provenance? | Tidy files, table records, licences, links between document and data |
| 8. Findability and citation | Can the data be found and cited back to the page? | Catalog records, schema.org, citation to the page, monitoring and corrections |
| 9. Operation at scale | Can the pipeline run over the whole backlog? | Resumable manifest, cost per page, model choice, monitoring, corrections |
Maturity levels
Each chapter describes three levels. A document collection can sit at different levels in different chapters, and the first level is a complete, useful state on its own.
| Level | Meaning |
|---|---|
| Foundational | The documents are inventoried and catalogued, with a text layer, and their data are extracted by hand where needed. |
| AI-ready | Tables and figures are located and extracted with tools, verified against the document, and published as tidy files with provenance next to the document. |
| AI-native | Extraction runs as a pipeline on new documents, extracted data are catalogued and searchable, and every value cites its page and bounding box. |
Running example
The recipes use two documents. The first is real: page 12 of World Bank
Policy Research Working Paper 4771, from the data-snapshot dataset, with
a bar chart whose values are printed on the bars; the same page is the
example on the program's home page. The second is fictional: the Food
security report 2025 of the example organization (EX-REP-2025-03), with a
district table in which one value is planted as an extraction error. A
small document inventory of eight fictional documents, one OCR page of
the fictional 2014 census report with planted recognition errors, two
extracted tables from successive food security reports, and a processing
log complete the example. All values of the fictional documents are
invented.
Scope of the examples
The examples are small on purpose: one chart, one table, eight documents, so that a reader can run them in an afternoon and see the shape of each check. They are starting points. Each recipe ends with a What this does not do note that says where the example stops and what the full version adds. The scripts handle the structured outputs; the extraction itself (layout detection, table parsing, chart reading) uses the tools named in each chapter.
Files
| File | Used in |
|---|---|
| document_inventory.csv | Chapter 1: the example inventory |
| inventory_report.py | Chapter 1: report on the inventory and the backlog |
| ocr_sample_page.txt and ocr_quality.py | Chapter 2: an OCR page with planted errors, and the quality score |
| example_snapshot.json | Chapters 2 to 4: a chart snapshot with its bounding box and extracted series |
| example_table.json | Chapters 3 to 5: an extracted table with a total row and a derived column |
| verify_extraction.py | Chapter 5: verification against printed labels, totals, and derived rates |
| snapshot_to_tidy.py | Chapters 4 and 7: tidy CSV with provenance and status per value |
| tidy_fs2024.csv, tidy_fs2025.csv, and series_mapping.csv | Chapter 6: tidy extractions from two reports and the table-to-series mapping |
| stack_series.py and series_fs.csv | Chapter 6: the combined series with provenance and revision status |
| table_record.json | Chapter 7: the extracted table as a World Bank table record |
| profile_table.json | Chapter 7: completeness profile for table records |
| processing_status.csv, build_manifest.py, and manifest.csv | Chapter 9: the processing log, the manifest builder, and its output |
| example_document.json | Chapter 1: the document record, from the dissemination cookbook |
| check_metadata.py | Chapters 1 and 7: validation of document and table records, from the dissemination cookbook |
Related resources
| Resource | Purpose |
|---|---|
| This guide | What a team can do with the data in its documents, and why, in a sequence of practical steps. |
| Data Snapshots | The benchmark of layout detection models on institutional documents, and the dataset. |
| Practical Guide to AI-Ready Data Dissemination | The dissemination system the extracted data join: catalog, APIs, trust, evaluation. |
| Monitoring of Data Use | The program's document parsing and mention extraction, which read the same documents. |
| Code | Open-source software to deploy or adapt. |
Citation
World Bank. 2026. Practical Guide to Extracting Data from Documents: A cookbook for dissemination, library, and research data teams. Version 0.1. AI for Data – Data for AI. https://worldbank.github.io/ai4data/cookbook/data-from-documents/
Version history
- 0.1 (October 2026). First draft: six chapters, recipes, scripts, self-assessment.