Skip to main content

5. Verification of extracted values

Question: Can the extracted values be verified against the document?

Rationale​

Extraction tools make errors: a 9 read as an 8, a column shifted by one row, a value label attached to the wrong bar. An extracted value published without a check is a transcription with an unknown error rate, and a user or an AI system that quotes it attributes the error to the organization.

Documents offer their own checks. Charts print value labels; tables have total rows and columns computed from other columns; the text repeats key figures. Each is a constraint that an extracted value either satisfies or fails. This chapter uses them to give every value a status, and a reviewed sample to measure what the checks cannot see.

Target state​

  • Every extracted value has a status: verified against the document, flagged for correction, or estimated where no check exists.
  • Flagged values are corrected by a person with the page open, and the correction is recorded.
  • A sample of extractions is reviewed by a person each batch, and the error rate is reported.

Maturity levels​

LevelSteps
FoundationalReview a sample of extracted tables against the page by hand and record the error rate.
AI-readyRun the automated checks (printed labels, totals, derived rates) on every extraction. Give each value a status and correct flagged values before publication. Review a sample each batch.
AI-nativeRun the checks in the extraction pipeline, block publication on flagged values, cross-check extracted figures against the series in the catalog where they exist, and report error rates per document type and model.
Running example

The chart's four values all match a printed label and are verified. The district table's population and undernourished columns sum to the national row; the prevalence column is derived from them, and one extracted value, 8.2 for the Eastern Region, does not match the computed 9.19 and is flagged.

Recipes​

Recipe

5.1 Check values against printed labels, totals, and derived rates

Level
AI-ready
Skills
Python
Indicative time
Minutes per batch

Result: every extracted value has a status, set by the checks that the document itself makes possible, and the script returns an exit status that a pipeline can act on.

A document constrains its own numbers in three ways, and each is a check. A chart that prints its values can be checked against the printed labels. A table with a total row can be checked by adding up the rows. A column computed from two others (a rate from a count and a population) can be recomputed. The script applies whichever checks the record allows:

"""Verify data extracted from a document against the document itself.

Reads an extraction JSON (a chart snapshot or a table, see the example
files) and runs the checks that the document makes possible:

  chart   each extracted value is matched to a value label printed on the
          chart (the `printed_values` list, read from the snapshot image);
          charts without printed labels are reported as estimated, which
          this check cannot verify
  table   each numeric column sums to the total row within a tolerance;
          each derived column (a rate computed from two other columns)
          matches the computed value within a tolerance

Every value gets a status: verified, flagged, or estimated. Exit status is
0 when nothing is flagged, 1 otherwise. Uses pandas.

What this does not check: whether the extraction picked the right table or
chart, whether the row and column labels are correct, or whether a value
without a printed label, a total, or a derivation is right. Those need a
person with the page open; recipe 4.2 samples them.

Usage:
    python verify_extraction.py example_snapshot.json
    python verify_extraction.py example_table.json --abs-tolerance 0.5 --rel-tolerance 0.02
"""

from __future__ import annotations

import argparse
import json
import math
import sys
from pathlib import Path

import pandas as pd

Result = tuple[str, str, str]  # (location, status, detail)


def close(a: float, b: float, abs_tol: float, rel_tol: float) -> bool:
    return math.isclose(a, b, rel_tol=rel_tol, abs_tol=abs_tol)


def verify_chart(x: dict) -> list[Result]:
    """Check each chart value against the labels printed on the chart, where there are any."""
    cells = [
        (f"{s['name']}/{cat}", v)
        for s in x["series"]
        for cat, v in zip(x["categories"], s["values"])
    ]
    if not x.get("value_labels_printed"):
        return [
            (where, "estimated", f"{v}: no printed label; read from the axis")
            for where, v in cells
        ]
    remaining = list(x.get("printed_values", []))
    results: list[Result] = []
    for where, v in cells:
        if v in remaining:
            remaining.remove(v)
            results.append((where, "verified", f"{v} matches a printed label"))
        else:
            results.append((where, "flagged", f"{v} is not among the printed labels"))
    results += [
        ("printed label", "flagged", f"{left} printed on the chart but not extracted")
        for left in remaining
    ]
    return results


def verify_table(x: dict, abs_tol: float, rel_tol: float) -> list[Result]:
    """Check derived columns by recomputing them and other columns against the total row."""
    table = pd.DataFrame(x["rows"], columns=x["columns"]).set_index(x["columns"][0])
    total_label = x.get("total_row")
    body = table.drop(index=total_label) if total_label in table.index else table
    results: list[Result] = []
    for col in table.columns:
        if not pd.api.types.is_numeric_dtype(table[col]):
            continue
        if col in x.get("derived", {}):
            spec = x["derived"][col]
            computed = (
                table[spec["numerator"]]
                / table[spec["denominator"]]
                * spec.get("scale", 1)
            )
            for label, v, c in zip(table.index, table[col], computed):
                results.append(
                    (
                        f"{label}/{col}",
                        "verified" if close(v, c, abs_tol, rel_tol) else "flagged",
                        f"{v} vs computed {c:.2f}",
                    )
                )
        elif total_label in table.index:
            total, s = table.at[total_label, col], body[col].sum()
            ok = close(total, s, abs_tol, rel_tol)
            results.append(
                (
                    f"{total_label}/{col}",
                    "verified" if ok else "flagged",
                    f"total {total} vs sum of rows {s:g}",
                )
            )
            results += [
                (
                    f"{label}/{col}",
                    "verified" if ok else "estimated",
                    f"{v} (covered by the total check)",
                )
                for label, v in body[col].items()
            ]
        else:
            results += [
                (
                    f"{label}/{col}",
                    "estimated",
                    f"{v}: no total or derivation to check against",
                )
                for label, v in table[col].items()
            ]
    return results


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    parser.add_argument("extraction", type=Path, help="chart snapshot or table JSON")
    parser.add_argument("--abs-tolerance", type=float, default=0.5)
    parser.add_argument("--rel-tolerance", type=float, default=0.02)
    args = parser.parse_args(argv)

    x = json.loads(args.extraction.read_text(encoding="utf-8"))
    kind = x.get("class", "").lower()
    if kind == "figure":
        results = verify_chart(x)
    elif kind == "table":
        results = verify_table(x, args.abs_tolerance, args.rel_tolerance)
    else:
        sys.exit("the extraction must have class Figure or Table")

    print(f"{x['document_id']} page {x['page']}: {x.get('title', '')} ({kind})")
    for where, status, detail in results:
        print(f"  {status:<9} {where:<32} {detail}")
    counts = pd.Series([s for _, s, _ in results]).value_counts()
    print(
        f"\n{counts.get('verified', 0)} verified, {counts.get('flagged', 0)} flagged, {counts.get('estimated', 0)} estimated"
    )
    print(
        "Not checked here: whether the right table or chart was extracted, or whether labels are correct."
    )
    return 1 if counts.get("flagged", 0) else 0


if __name__ == "__main__":
    sys.exit(main())

Run on the chart of chapter 3:

python verify_extraction.py example_snapshot.json
WB-PRWP-4771 page 12: Type of guarantee systems (figure)
  verified  HIGH/MUTUAL                      15 matches a printed label
  verified  HIGH/PUBLIC                      7 matches a printed label
  verified  LOW_MIDDLE/MUTUAL                7 matches a printed label
  verified  LOW_MIDDLE/PUBLIC                17 matches a printed label

4 verified, 0 flagged, 0 estimated
Not checked here: whether the right table or chart was extracted, or whether labels are correct.

Run on the table of chapter 4:

python verify_extraction.py example_table.json
EX-REP-2025-03 page 24: Table 3.1 Undernourishment by district, 2024 (table)
  verified  National/Population (thousands)  total 5640 vs sum of rows 5640
  verified  Northern Region/Population (thousands) 1240 (covered by the total check)
  verified  Central Region/Population (thousands) 2110 (covered by the total check)
  verified  Eastern Region/Population (thousands) 860 (covered by the total check)
  verified  Western Region/Population (thousands) 1430 (covered by the total check)
  verified  National/Undernourished (thousands) total 434 vs sum of rows 434
  verified  Northern Region/Undernourished (thousands) 118 (covered by the total check)
  verified  Central Region/Undernourished (thousands) 141 (covered by the total check)
  verified  Eastern Region/Undernourished (thousands) 79 (covered by the total check)
  verified  Western Region/Undernourished (thousands) 96 (covered by the total check)
  verified  Northern Region/Prevalence (%)   9.5 vs computed 9.52
  verified  Central Region/Prevalence (%)    6.7 vs computed 6.68
  flagged   Eastern Region/Prevalence (%)    8.2 vs computed 9.19
  verified  Western Region/Prevalence (%)    6.7 vs computed 6.71
  verified  National/Prevalence (%)          7.7 vs computed 7.70

14 verified, 1 flagged, 0 estimated
Not checked here: whether the right table or chart was extracted, or whether labels are correct.

Each line names a value, its status, and the check that set the status. A value in a column with a total row is verified by the total check, and the rows that make up a verified total are marked as covered by it. A value in a derived column is compared with the value recomputed from its inputs, and a difference beyond the rounding tolerance is flagged.

  1. Run the check on every record.
  2. Correct flagged values with the page open. Here the page prints 9.2, and the extraction misread it. Record the correction (value, reason, who, date) and re-run the check.
  3. Write the statuses into the tidy file (recipe 4.3 takes the report).

Worked example. In the run on the district table, the Eastern Region's prevalence is flagged: the extraction read 8.2, and 79 thousand undernourished out of 860 thousand people is 9.19 percent. The dissemination officer opens page 24 and sees that the printed value is 9.2 and that the parser took the 9 for an 8 because of a faint line through the cell. The correction is recorded as "8.2 corrected to 9.2; misread digit; reviewed against page 24 on 2026-10-02", and the check runs again with fifteen values verified. The chart's four values pass on the first run, because each matches a printed label.

What this does not do. The checks verify what the document constrains. A value in a column with no total and no derivation is marked estimated, which means unverified by this method. A wrong value whose row total was extracted with the same error passes the total check. Recipe 5.2 covers what the checks cannot.

Recipe

5.2 Review a sample with the page open

Level
Foundational
Skills
Dissemination officer
Indicative time
Half a day per batch

Result: the organization has a measured error rate for the extraction, per document type and per method, from a sample that a person has compared with the page.

  1. Draw a sample from each batch: 20 tables or charts, across document types and extraction methods, including some that the automated checks marked verified.
  2. For each, compare every value with the page and record the errors by kind: wrong digit, shifted row or column, wrong unit, missing row, wrong total treated as data.
  3. Report the error rate (errors per hundred values) per document type and method, and keep the sample and its results.
  4. Feed the findings back. A method with a high error rate on one document type is replaced for that type; a kind of error that recurs becomes an automated check where possible.

Worked example. From the first batch of 180 tables and 40 charts, the officer draws 20: 14 tables and 6 charts across the four document types, including 12 that the automated checks had passed. The tables hold 410 values and the charts 48. The review finds 7 errors in the tables, 5 of them wrong digits in the yearbook's OCR pages and 2 of them a shifted row in a bulletin, and 2 errors in the charts, both values read from an axis without printed labels. The error rates are 1.7 per hundred values for tables and 4.2 for charts. The two chart errors lead to a rule that charts without printed labels are published as estimates only, and the yearbook pages are sent back for re-OCR at a higher resolution.

What this does not do. A sample measures; it does not fix the batch. At a high error rate the batch is re-extracted with another method, and the sample is drawn again.

Recipe

5.3 Cross-check extracted figures against the catalog

Level
AI-native
Skills
Developer
Indicative time
One day to set up

Result: every extracted figure that the catalog already holds as a series value is checked against the catalog, and the number that is published is the catalog's.

  1. Match extracted tables to series through the document record's data_sources and the table's column mapping. The example table's prevalence column maps to FS_UNDERNOURISH_PCT.
  2. For each matched value, compare the extracted figure with the catalog value for the same geography and period, with the tolerance of the number check in the dissemination cookbook.
  3. Where they differ, the catalog is authoritative for the current figure, and the document's figure is historical. Keep both, and record in the tidy file that the value is as printed in the document on its date.
  4. Where the catalog has no such series, the extracted table is the only source. That is the case for most historical documents, and the reason extraction is worth the effort.

Worked example. The national row of the district table gives a prevalence of 7.7 percent for 2024. The document record's data_sources field names FS_UNDERNOURISH_PCT, and the table's prevalence column is mapped to it. The catalog holds 7.6 for 2024, from the release of 14 March 2025. The two differ by 0.1, which is outside the rounding tolerance, and the footnote of the table says that the figure is provisional. The tidy file keeps 7.7 with the status "as printed, superseded by the catalog release of 2025-03-14", and the published series value stays 7.6. The four regional rows have no catalog series, so the extracted values are the only source and are published as verified.

What this does not do. A catalog match verifies the figure. It does not verify the document's context (a different definition, a revised series). The document's own footnote and the series' revisions log are what resolve that.

Implementation options​

  • Automated checks: printed labels, totals, derived rates, and text repetition (a figure quoted in the running text that matches a table cell).
  • Review: a sample per batch with a fixed rubric and an error rate; a review board for flagged values, as the Metadata Reviewer provides for metadata.
  • Status codes: verified, flagged, estimated, corrected; the same role as SDMX OBS_STATUS in the values files of the dissemination cookbook.
  • Pipeline gate: publication blocked while any value is flagged.

World Bank examples​

Resources​

  • Proof-Carrying Numbers: the program's method for binding a stated number to its source so that a reader can check it.
  • Metadata Reviewer review board: the review pattern (current and proposed side by side, decision recorded) applied to flagged values.
  • verify_numbers.py from the dissemination cookbook: checking numbers in text against the catalog, the same check as recipe 5.3 from the other side.
  • SDMX OBS_STATUS code list: the published status codes the tidy file's status column maps to when values join the catalog.

Common mistakes​

  • Publishing before checking. The error rate is unknown and the organization's name is on the number.
  • Treating "no check possible" as verified. A value with no printed label, total, or derivation is an estimate and is published as one.
  • Correcting in place without a record. The next extraction run reintroduces the error; the correction has to be recorded against the document and page.
  • Reviewing only flagged values. The sample has to include values the checks passed, or the checks' blind spots stay unmeasured.

Verification​

  • Status. Every value in every tidy file has a status, and no published file contains a flagged value.
  • Error rate. The error rate is reported per batch, document type, and method, from the reviewed sample.
  • Corrections. Each correction is recorded with the value, the reason, the person, and the date.

Checklist​

0 of 6 done

Ticks are saved in this browser only.