1. Documents that hold data

Question: Which documents hold data, and which data?

Rationale

Extraction starts with a list. An organization that does not know which of its documents hold tables and figures, how many, and whether the files have a text layer cannot plan the work or measure it. The list also serves a second purpose: a catalog record for each document, with the series and surveys it draws on, is what lets a user or an AI system move from a report to the data behind it and back.

Scanned documents are the usual obstacle. A PDF without a text layer is an image; nothing can be extracted from it until optical character recognition has produced the text, and the inventory is where that need is recorded.

Target state

Every published document is in an inventory with its type, year, page count, text-layer status, and counts of tables and figures.
Every document has a record in the World Bank document schema, with the series and surveys it draws on in data_sources.
Every document has a text layer, produced by OCR where the original was scanned, so that it can be searched and parsed.

Maturity levels

Level	Steps
Foundational	Build the inventory from the publication list. Record whether each file has a text layer. Estimate table and figure counts from the table of contents.
AI-ready	Give each document a record in the document schema with `data_sources`. Run OCR on scanned documents. Extract the text layer for search and for the parsing in chapter 4.
AI-native	Keep the inventory current from the publication workflow, with table and figure counts from the layout detection in chapter 3, and expose the records through the catalog.

Running example

The example organization has eight documents in its inventory, from a 12-page bulletin to a 310-page yearbook. One, the 2014 census report, has no text layer. The report in recipe 1.1 shows where the tables are and what the backlog looks like.

Recipes

Recipe

1.1 Build the document inventory

Level: Foundational
Skills: Librarian or dissemination officer; Python for the report
Indicative time: One day for a few hundred documents

Result: the organization has one row per published document with the facts that decide how much extraction work it needs, and a report that says where the tables are and which documents to start with.

The inventory is a CSV with one row per document:

Column	Meaning
`document_id`, `title`, `year`, `type`	What the document is; `type` is report, yearbook, bulletin, census report, and so on
`pages`	The page count
`text_layer`	`yes` when the PDF contains selectable text, `no` when its pages are images (a scan), `ocr` when an earlier tool added text to a scan
`tables`, `figures`	Estimated counts, from the table of contents until the detection of chapter 3 counts them
`extracted`	`none`, `partial`, or `done`
`data_sources`	The series and surveys the document draws on, by catalog identifier, separated by `;`

document_id,title,year,type,pages,text_layer,tables,figures,extracted,data_sources
EX-REP-2025-03,Food security report 2025,2025,report,64,yes,14,9,partial,FS_UNDERNOURISH_PCT;EX-HBS-2021
EX-REP-2024-03,Food security report 2024,2024,report,58,yes,12,8,no,FS_UNDERNOURISH_PCT;EX-HBS-2021
EX-REP-2025-01,Labour market report 2025 Q1,2025,report,32,yes,18,6,no,EX-LFS-2025-Q1
EX-REP-2025-02,Labour market report 2025 Q2,2025,report,33,yes,18,6,no,EX-LFS-2025-Q2
EX-BUL-2025-06,Statistical bulletin June 2025,2025,bulletin,12,yes,9,4,yes,LF_UNEMP_PCT;FS_UNDERNOURISH_PCT
EX-CEN-2014-01,Population and housing census 2014 main report,2014,census report,220,no,96,31,no,EX-CEN-2014
EX-YB-2023,Statistical yearbook 2023,2023,yearbook,310,yes,142,22,no,various
EX-MET-2021-02,Household budget survey 2021 methodology,2021,methodological note,40,yes,6,2,yes,EX-HBS-2021

The report script reads the inventory and summarizes it by document type, then lists the backlog from largest to smallest:

"""Report on a document inventory: where the data are, and what is left to do.

Reads an inventory CSV with one row per document (document_id, title,
year, type, pages, text_layer, tables, figures, extracted, data_sources)
and prints counts by type and year, the number of tables and figures, the
share of documents with a text layer (documents without one need OCR
before any text-based extraction), and the extraction backlog ordered by
the number of tables and figures. Uses pandas.

Usage:
    python inventory_report.py document_inventory.csv

What this does not do: it reports what the inventory says. Counting tables
and figures per document is the work of the layout detection in chapter 2;
until then the counts are estimates entered by hand.
"""

from __future__ import annotations

import argparse
import sys

import pandas as pd


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    parser.add_argument("inventory")
    args = parser.parse_args(argv)
    docs = pd.read_csv(args.inventory, dtype=str).fillna("")
    if docs.empty:
        sys.exit("no documents in the inventory")
    for col in ("tables", "figures"):
        docs[col] = pd.to_numeric(docs[col], errors="coerce").fillna(0).astype(int)
    docs["has_text"] = docs["text_layer"].str.strip().str.lower() == "yes"
    docs["extracted"] = docs["extracted"].str.strip().str.lower()

    n = len(docs)
    print(
        f"{n} documents, {docs['tables'].sum()} tables, {docs['figures'].sum()} figures; {docs['has_text'].sum()} of {n} with a text layer"
    )

    by_type = (
        docs.groupby("type")
        .agg(
            documents=("document_id", "size"),
            tables=("tables", "sum"),
            figures=("figures", "sum"),
        )
        .sort_values("tables", ascending=False)
    )
    print(f"\n{'type':<22} {'docs':>5} {'tables':>7} {'figures':>8}")
    for t, c in by_type.iterrows():
        print(f"{t:<22} {c.documents:>5} {c.tables:>7} {c.figures:>8}")

    status = docs["extracted"].value_counts().sort_index()
    print("\nextraction status: " + ", ".join(f"{k} {v}" for k, v in status.items()))

    backlog = (
        docs[docs["extracted"] != "yes"]
        .assign(size=lambda d: d["tables"] + d["figures"])
        .sort_values("size", ascending=False, kind="stable")
    )
    print("\nbacklog (largest first):")
    for d in backlog.itertuples():
        print(
            f"  {d.document_id:<18} {d.year}  {d.tables:>4} tables {d.figures:>3} figures  {d.extracted}{'' if d.has_text else '  needs OCR'}"
        )
    return 0


if __name__ == "__main__":
    sys.exit(main())

Output for the example inventory (abridged):

8 documents, 315 tables, 88 figures; 7 of 8 with a text layer
type                    docs  tables  figures
yearbook                   1     142       22
census report              1      96       31
report                     4      62       29
...
backlog (largest first):
  EX-YB-2023         2023   142 tables  22 figures  no
  EX-CEN-2014-01     2014    96 tables  31 figures  no  needs OCR

The first line gives the totals. The table that follows groups the documents by type, and the backlog lists them from the largest to the smallest, with a note on the ones that need OCR before anything else can be done.

List the documents from the publication register or the website.
For each document, record whether the PDF has a text layer (open it and try to select text, or run a PDF tool that reports it), the page count, and estimated counts of tables and figures.
Record which series and surveys the document draws on, using the identifiers from the catalog. This is what later links the extracted tables to the indicators they report.
Run the report and decide the order of work from the backlog. Here the yearbook and the census report hold three quarters of the tables, and the census report needs OCR first.

Worked example. The librarian lists the eight documents from the publication register. For each, she opens the PDF, tries to select text on three pages, and records the result: seven have a text layer, and the 2014 census report does not. She counts the tables and figures from each document's list of tables, which gives 142 for the yearbook and 96 for the census report. For the 2025 food security report she records FS_UNDERNOURISH_PCT and EX-HBS-2021 as the data sources, because the report draws on that series and that survey. The report script confirms that the yearbook and the census report hold 238 of the 315 tables, and the unit decides to start with the yearbook while the census report is sent for OCR.

What this does not do. The inventory does not say which tables matter. That is a decision about which data users ask for, informed by the monitoring of data use.

Recipe

1.2 Give each document a catalog record with its data sources

Level: AI-ready
Skills: Data curator
Indicative time: Half an hour per document; less with the Metadata Editor

Result: every document has a catalog record in the World Bank document schema that lists the series and surveys it draws on, and the record is validated against the schema and the completeness profile.

The running example is the fictional food security report's record, from the dissemination cookbook:

{
  "document_description": {
    "title_statement": {
      "idno": "EX-REP-2025-03",
      "title": "Food security report 2025"
    },
    "authors": [{"full_name": "National Statistics Office (example), Food security unit"}],
    "date_published": "2025-06-30",
    "type": "report",
    "abstract": "Annual report on food security, combining the prevalence of undernourishment series with household budget survey results on food expenditure and dietary diversity. The 2025 edition reports a decline in undernourishment from 8.4 percent in 2022 to 7.6 percent in 2024 and presents district-level estimates for the first time, with their margins of error.",
    "languages": [{"name": "English", "code": "en"}],
    "publisher": "National Statistics Office (example)",
    "url": "https://stats.example/documents/EX-REP-2025-03.pdf",
    "identifiers": [{"type": "doi", "identifier": "10.00000/example.fsr.2025"}],
    "keywords": [{"name": "food security"}, {"name": "undernourishment"}],
    "topics": [{"name": "Food security"}],
    "ref_country": [{"name": "Example country", "code": "EX"}],
    "temporal_coverage": "2022-2024",
    "license": [{"name": "CC BY 4.0", "uri": "https://creativecommons.org/licenses/by/4.0/"}],
    "data_sources": [
      {"name": "Prevalence of undernourishment (FS_UNDERNOURISH_PCT)", "uri": "https://stats.example/series/FS_UNDERNOURISH_PCT"},
      {"name": "Household Budget Survey 2021 (EX-HBS-2021)", "uri": "https://stats.example/microdata/EX-HBS-2021"}
    ]
  }
}

Document the publication in the Metadata Editor, which has a template for documents (Dublin Core based), or write the JSON directly.
Fill data_sources with the series and surveys the document reports, by identifier and URL. This field is what makes a document findable from a series and a series findable from a document.
Validate the record with the metadata checker and the document profile from the dissemination cookbook:
```
python check_metadata.py example_document.json profile_document.json --level ai-ready
```
Publish the record in the catalog (NADA stores documents alongside microdata and tables), so that the document has a stable page and, where possible, a DOI.

Worked example. The record of the 2025 food security report has the identifier EX-REP-2025-03, the title, the publication date of March 2025, the organization as the author, an abstract, and 48 pages. Its data_sources field lists FS_UNDERNOURISH_PCT with the series URL and EX-HBS-2021 with the survey's catalog URL. The checker at the AI-ready level reports the record valid and complete, with one validity note that the abstract is shorter than the profile recommends. The curator expands the abstract, and the record is published in the catalog as document 88, with a DOI.

What this does not do. The record describes the document. The tables extracted from it get their own records in chapter 7, linked to this one.

Recipe

1.3 Produce the text layer for search and parsing

Level: AI-ready
Skills: Developer
Indicative time: Minutes per document; OCR time for scanned ones

Result: every document has machine-readable text, page by page. A search index can use the text, and the table extraction of chapter 4 can parse it.

For a PDF that already contains text, the text is read out page by page. The program's document parser does this with PyMuPDF, a Python library for reading PDFs, and keeps the page structure:

import fitz  # PyMuPDF

doc = fitz.open("EX-REP-2025-03.pdf")
pages = [page.get_text() for page in doc]

For documents with a text layer, extract the text per page as above.
For scanned documents, produce the text layer with OCR (optical character recognition, the reading of text from an image) as chapter 2 describes. Keep the OCR quality score with each page; pages marked for re-OCR are not extracted.
Index the text per page in the site's search, so that a user searching for a table title finds the document and the page.
Keep the extracted text with the document, versioned with it. It is also the input to the mention extraction of the monitoring of data use.

Worked example. The developer runs the extraction over the seven documents with a text layer and writes one text file per page, 573 files in all. Page 24 of the food security report yields the title "Table 3.1 Undernourishment by district, 2024", the header row, the five data rows, and the footnote as plain text with the reading order kept. The 573 files are indexed in the site search, and a search for "undernourishment by district" returns the report with the page number. The census report is left for chapter 2, and its inventory row says so.

What this does not do. A text layer makes text searchable. It does not give the structure of a table, which depends on positions on the page; chapter 4 handles that. OCR output on complex layouts (multi-column yearbooks, rotated tables) needs review.

Implementation options

Inventory: a CSV or a catalog export; the publication register where one exists.
Document records: the World Bank document schema (Dublin Core based), through the Metadata Editor or JSON; NADA catalogs documents next to data.
Text extraction: PyMuPDF (used by the program), pdfplumber, or poppler's pdftotext; Tesseract for OCR.
Links to data: data_sources in the document record; the reverse link from a series record to the documents that use it comes from the monitoring of data use.

World Bank examples

The document schema carries data_sources, toc_structured, identifiers, and bibliographic_citation for a document record.
Monitoring of Data Use parses documents with PyMuPDF and extracts the datasets they mention, which fills data_sources for documents the organization did not write.
The Documents and Reports repository of the World Bank, from which the data-snapshot dataset's policy research working papers come.

Resources

World Bank document schema: the record for a publication, with data_sources linking it to the series and surveys it uses.
Metadata Editor user and practice guide: documenting publications from the document template and publishing to NADA.
PyMuPDF: page text, images, and table detection from PDFs; the parser the program's document tools use.
pdfplumber: character-level access to the text layer, useful to tell a real text layer from an OCR layer.
Monitoring of Data Use: the program's document parsing and mention extraction, which read the same inventory.

Common mistakes

Starting extraction without an inventory. Effort goes to whatever document is at hand, and the yearbook with 142 tables waits.
Treating a scanned PDF as text. Nothing can be parsed from an image; OCR comes first and its errors have to be reviewed.
Document records without data sources. The document stays unlinked to the series it reports.
Counting tables once. New documents arrive every quarter; the inventory is part of the publication workflow.

Verification

Inventory. Every document on the publication list is in the inventory, and a sample of ten has been checked against the files for text layer and page count.
Records. The checker reports the document records valid and complete at the target level.
Text layer. A search for a table title returns the document and the page.

Checklist

0 of 6 done

Inventory with type, year, pages, text layer, tables, figures
Backlog ordered by the number of tables and figures
Document record in the document schema for every document
data_sources filled with series and survey identifiers
OCR run on scanned documents, with confidence kept
Text per page extracted, indexed, and versioned with the document

Ticks are saved in this browser only.

Rationale​

Target state​

Maturity levels​

Recipes​

Implementation options​

World Bank examples​

Resources​

Common mistakes​

Verification​

Checklist​

Rationale

Target state

Maturity levels

Recipes

Implementation options

World Bank examples

Resources

Common mistakes

Verification

Checklist