4. Extraction of structured data

Question: Can the data be extracted into a structured form?

Rationale

A located table is still a picture of numbers. Extraction turns it into rows and columns with labels, units, and notes, and turns a chart into categories, series, and values. Two kinds of source need two kinds of method: a table in a PDF with a text layer can be parsed from the text and its positions; a chart, or a table in a scanned document, has to be read from the image.

Whatever the method, the output has to carry its provenance from the first step: the document, the page, the bounding box, and the method. A value that loses that link cannot be verified in chapter 5 or cited in chapter 8.

Target state

• Tables are extracted into a structured form with column labels, row labels, units from the header or the note, and the footnote.
• Charts are extracted into chart type, categories, series, and values, with a record of whether the values were printed on the chart or read from the axes.
• Every extraction is a tidy file with one row per value and the provenance columns, ready for verification.

Maturity levels

Level	Steps
Foundational	Extract tables from the text layer with a PDF table parser. Review headers and footnotes by hand. Read charts by hand where their data are needed.
AI-ready	Extract charts from snapshots with a vision-language model, recording whether value labels were printed. Convert every extraction to a tidy file with provenance and a status per value.
AI-native	Run extraction as a pipeline after detection, with the model choice per class and document type, and with the verification of chapter 5 in the same run.

Running example

The chart snapshot from the working paper becomes two series over two categories, with the four values that are printed on the bars. The district table from the fictional report becomes five rows by three numeric columns, with units from the header and a footnote, and one value that the extraction got wrong.

Recipes

Recipe

4.1 Extract tables from the text layer

Level

Foundational

Skills

Python

Indicative time

One hour to set up; review time per table

Result: each table is a JSON record of its columns, rows, units, and footnote, and the record keeps the page and the box the table came from.

The record for the running example's district table shows the fields. Beyond the rows themselves, it names the total row, the column that is computed from two others (derived), the unit of each column, and the method by which it was produced:

{
  "document_id": "EX-REP-2025-03",
  "document_title": "Food security report 2025",
  "document_url": "https://stats.example/documents/EX-REP-2025-03.pdf",
  "page": 24,
  "pdf_page": 26,
  "bbox": [0.08, 0.21, 0.92, 0.52],
  "class": "Table",
  "method": "text-layer extraction (PyMuPDF), header and footnote reviewed by a person",
  "title": "Table 3.1 Undernourishment by district, 2024",
  "columns": ["District", "Population (thousands)", "Undernourished (thousands)", "Prevalence (%)"],
  "rows": [
    ["Northern Region", 1240, 118, 9.5],
    ["Central Region", 2110, 141, 6.7],
    ["Eastern Region", 860, 79, 8.2],
    ["Western Region", 1430, 96, 6.7],
    ["National", 5640, 434, 7.7]
  ],
  "total_row": "National",
  "derived": {
    "Prevalence (%)": {"numerator": "Undernourished (thousands)", "denominator": "Population (thousands)", "scale": 100}
  },
  "units": {"Population (thousands)": "thousands of persons", "Undernourished (thousands)": "thousands of persons", "Prevalence (%)": "% of population"},
  "footnote": "Three-year average centred on 2024. Source: FS_UNDERNOURISH_PCT and household budget survey.",
  "note": "Fictional table. One value is planted as an extraction error so that the verification recipe has something to find."
}

1. Parse the table region from the text layer with a PDF table tool. pdfplumber and PyMuPDF both extract tables from text positions, and the box from chapter 3 restricts the region.
2. Review the header. Merged headers ("Population (thousands)") become one column label with the unit; multi-row headers are joined. Put the units in the units map and the footnote in footnote.
3. Mark the total row and any derived column (a rate computed from two others), because chapter 5 verifies against them.
4. Record the method in method, including that a person reviewed the header and footnote.

Worked example. The parser reads the region of page 24 inside the adjusted box and returns a grid of six rows by four columns. The header row reads "District", "Population (thousands)", "Undernourished (thousands)", and "Prevalence (%)". The reviewer splits each header into a column label and a unit, so that the units map reads "thousands" for the two counts and "%" for the rate. The last row, "National", is marked as the total row. The prevalence column is marked as derived from the two counts, because the footnote says so. The footnote itself goes into the footnote field, and method records "pdfplumber 0.11; header and footnote reviewed by hand". The result is the record shown above, with one value, the Eastern Region's prevalence, that the parser read wrongly and that chapter 5 finds.

What this does not do. Text-layer parsing fails on tables whose cells are images, on rotated tables, and on scanned pages. Those go through recipe 4.2 as images. A parser that returns a grid cannot tell a total row from a data row; that is a reading task, done once per table layout.

Recipe

4.2 Extract chart data from snapshots

Level

AI-ready

Skills

Developer with access to a vision-language model

Indicative time

One day to set up; seconds per chart, plus review

Result: each chart is a JSON record of its type, its categories, its series and their values, and a statement of whether the values were printed on the chart or estimated from the axes.

A vision-language model is a model that takes an image and a question and answers in text. Given a chart image and a fixed instruction, it returns the chart's content in a fixed form. The example of recipe 3.1 is such a record: a bar chart with two categories and two series, whose values were printed on the bars.

1. Send the snapshot image to the model with a fixed instruction: return the chart type, the title, the categories, the series with their values, the unit if stated, and whether each value was printed on the chart or read from the axis. Ask for JSON in the form of the example.
2. Keep value_labels_printed and printed_values honest. When the chart prints its values, the extraction is a transcription that chapter 5 can verify. When it does not, the values are estimates from the axes, and the tidy file says so.
3. Record the model name and version in method. Model changes change results.
4. Review a sample of charts against the snapshot, in particular stacked bars, dual axes, and charts with more than four series, where models fail most.

Worked example. The snapshot of the bar chart from page 12 of the working paper is sent to the model with the fixed instruction. The model returns a record with the chart type "bar", the title "Type of guarantee systems", the categories MUTUAL and PUBLIC, and two series, HIGH and LOW_MIDDLE, with the values 15 and 7 for HIGH and 7 and 17 for LOW_MIDDLE. It reports value_labels_printed as true, because each bar carries its value, and it lists the four printed values. The method field records the model name and version. The reviewer compares the record with the snapshot and confirms all four values, which is the case the verification of chapter 5 then confirms automatically.

What this does not do. Values read from axes are estimates with an error that depends on the chart's resolution. They are published as estimates (chapter 7) and never as the official figure. When the underlying series exists in the catalog, the chart's values are a pointer to the series, which is the number to use.

Recipe

4.3 Convert extractions to tidy files with provenance

Level

AI-ready

Skills

Python

Indicative time

Minutes

Result: every table or chart has a CSV file with one row per value, and every value carries the columns that trace it back to the page.

A tidy file has one value per row. The columns are:

Column	Meaning
document_id, page, bbox	Where the value came from: the document, the printed page, and the box on it
class, title	Table or Figure, and its printed title
row, column	The row label and the column label the value sits under
value, unit	The value and its unit
status	extracted until verification sets verified, flagged, or estimated

The converter reads a table or chart record and writes the tidy file:

"""Convert an extracted chart or table into a tidy CSV with provenance.

One row per value, with the columns that let any later user trace the
value back to the page: document_id, page, bbox, class, title, row,
column, value, unit, status. The status column records the verification
result when a verification report from verify_extraction.py is given;
otherwise every value is "extracted".

Uses pandas.

Usage:
    python snapshot_to_tidy.py example_table.json -o T3.1.csv
    python verify_extraction.py example_table.json > report.txt
    python snapshot_to_tidy.py example_table.json --verification report.txt -o T3.1.csv

What this does not do: it does not change values. A flagged value stays in
the file with status "flagged" so that the correction is a documented
edit, with the page open.
"""

from __future__ import annotations

import argparse
import json
import re
import sys
from pathlib import Path

import pandas as pd

REPORT_LINE = re.compile(r"^\s+(verified|flagged|estimated)\s+(\S.*?)\s{2,}")
FIELDS = [
    "document_id",
    "page",
    "bbox",
    "class",
    "title",
    "row",
    "column",
    "value",
    "unit",
    "status",
]


def read_statuses(report: Path | None) -> dict[str, str]:
    """Map 'row/column' to its status from the output of verify_extraction.py."""
    if report is None:
        return {}
    matches = (
        REPORT_LINE.match(line)
        for line in report.read_text(encoding="utf-8").splitlines()
    )
    return {m.group(2).strip(): m.group(1) for m in matches if m}


def rows_for(x: dict, statuses: dict[str, str]) -> list[dict]:
    """One row per value of a chart or a table record, with its provenance."""
    base = {
        "document_id": x["document_id"],
        "page": x["page"],
        "bbox": " ".join(map(str, x["bbox"])),
        "class": x["class"],
        "title": x.get("title", ""),
    }
    if x["class"].lower() == "figure":
        cells = [
            (cat, s["name"], v, x.get("unit", ""))
            for s in x["series"]
            for cat, v in zip(x["categories"], s["values"])
        ]
    else:
        units = x.get("units", {})
        cells = [
            (r[0], col, r[j], units.get(col, ""))
            for r in x["rows"]
            for j, col in enumerate(x["columns"][1:], start=1)
        ]
    return [
        {
            **base,
            "row": row,
            "column": col,
            "value": v,
            "unit": unit,
            "status": statuses.get(
                f"{row}/{col}" if x["class"].lower() != "figure" else f"{col}/{row}",
                "extracted",
            ),
        }
        for row, col, v, unit in cells
    ]


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    parser.add_argument("extraction", type=Path)
    parser.add_argument(
        "--verification", type=Path, help="output of verify_extraction.py"
    )
    parser.add_argument("-o", "--output", type=Path)
    args = parser.parse_args(argv)

    x = json.loads(args.extraction.read_text(encoding="utf-8"))
    tidy = pd.DataFrame(rows_for(x, read_statuses(args.verification)), columns=FIELDS)
    tidy.to_csv(args.output or sys.stdout, index=False)
    if args.output:
        print(f"wrote {args.output}: {len(tidy)} values")
    return 0


if __name__ == "__main__":
    sys.exit(main())

python snapshot_to_tidy.py example_table.json -o T3.1.csv

1. Run the converter on each record.
2. After the verification of chapter 5, run it again with the verification report, so that it writes the status per value.
3. Name the file from the document and table identifiers and keep it with the document, the snapshot, and the sidecar record.

Worked example. The converter turns the district table into a file of fifteen rows, one per value. The row for the Eastern Region's population reads: document EX-REP-2025-03, page 24, the adjusted box, class Table, the printed title, row "Eastern Region", column "Population (thousands)", value 860, unit "thousands", status extracted. After the verification of chapter 5, the converter runs again with the report and sets the status of fourteen rows to verified and of the Eastern Region's prevalence row to flagged. The file is named EX-REP-2025-03-T3.1.csv and stored beside the snapshot.

What this does not do. The tidy file carries the values as extracted. It does not correct them. A flagged value stays flagged until a person corrects it with the page open, and the correction is a documented edit.

Implementation options

• Table parsing: pdfplumber, PyMuPDF (page.find_tables), Camelot, or Tabula for text-layer tables; a vision-language model for image tables.
• Chart reading: a vision-language model with a fixed instruction and JSON output; open-weight models run locally where documents cannot leave the organization (Efficient and Inclusive AI).
• Tidy form: one row per value with provenance, as in this chapter; the same layout as the values files of the dissemination cookbook, so that extracted data can join the catalog.
• Batching: extraction over a backlog of hundreds of documents runs as a batch job with the review sample drawn from each batch.

World Bank examples

• The program's home page shows the chart of the running example extracted into categories, series, and values, with the note that the output is an illustration.
• Monitoring of Data Use parses documents with PyMuPDF and keeps the page structure, which is the same text layer the table parser reads.
• Efficient and Inclusive AI covers model choice and batch processing for extraction at scale.

Resources

• pdfplumber and Camelot: table extraction from the text layer, by lines and by text alignment.
• Docling: table structure recognition that returns cells with row and column spans, for tables with merged headers.
• PaddleOCR: OCR with table and layout recognition (PP-Structure) for image tables in many languages.
• Efficient and Inclusive AI: the program's guidance on running open-weight vision-language models locally for chart reading.
• Tidy data (Wickham, 2014): the one-row-per-value layout the tidy files follow.

Common mistakes

• Units lost in the header. "Population (thousands)" extracted as "Population" multiplies every value by a thousand downstream.
• Estimates presented as readings. A value read from an axis is an estimate; the file has to say so.
• A total row treated as data. It doubles every sum and hides the check that chapter 5 relies on.
• Provenance added later. Page and box recorded at extraction time are reliable; added afterwards they are guesses.

Verification

• Structure. Every table record has columns, rows, units, and a footnote, and every chart record has categories, series, and value_labels_printed.
• Tidy files. Every value has a document, a page, a bounding box, and a status.
• Sample. A sample of extractions has been opened next to the snapshot by a person, and the error rate is recorded (chapter 5).

Checklist

0 of 6 done

• Tables parsed from the text layer with headers and footnotes reviewed
• Units recorded per column; total rows and derived columns marked
• Charts extracted with chart type, categories, series, and values
• Printed values distinguished from values read from axes
• Model or tool and version recorded per extraction
• Tidy file per extraction with provenance columns and a status per value

Ticks are saved in this browser only.