Standards used in this guide
The recipes use published standards wherever one exists. Where the World Bank's Development Data Group has adopted a standard for its own catalogs, the guide uses that one.
Documents and tables
| Standard | Use | World Bank use | In this guide |
|---|---|---|---|
| World Bank document schema | Record for a document (Dublin Core based) with data_sources, toc_structured, identifiers, and citation | The Development Data Group's catalogs; the Metadata Editor; NADA | Chapter 1 |
| World Bank table schema | Record for a statistical table: columns, rows, data sources, definitions, licence, citation, relations | The same catalogs and tools | Chapter 7 |
| Dublin Core | Core metadata elements for documents | The basis of the document schema | Chapter 1 |
| Metadata Editor | Documentation of documents and tables with templates and validation; publishing to NADA | Used by the Development Data Group's curators | Chapters 1 and 7 |
| NADA | Catalog for documents, tables, microdata, and indicators, with schema.org markup, search, and a REST API; semantic search and an MCP server through NADA AI, which is being integrated into NADA | Runs the Microdata Library | Chapters 1, 7, and 8 |
Layout detection and extraction
| Standard or resource | Use | In this guide |
|---|---|---|
| Data Snapshots (Dy and Solatorio) | Benchmark of open-source layout detection models on institutional documents | Chapter 3 |
| data-snapshot dataset | 476 annotated regions (Figure, Table) with normalized XYXY bounding boxes, from UNHCR reports, World Bank policy research working papers, and refugee documents; MIT licence | Chapter 3: annotation format and test set |
| COCO format | Interchange format for object detection annotations and tools | Chapter 3 |
| PyMuPDF | PDF text, page rendering, and table extraction; used by the program's document parser | Chapters 1 and 4 |
| PDF/A (ISO 19005) | Archival PDF with a text layer, the output of the OCR stage | |
| OCRmyPDF | OCR to searchable PDF/A with Tesseract | |
| Tesseract | Open-source OCR for scanned documents | Chapters 1 and 2 |
Files, provenance, and citation
| Standard | Use | World Bank use | In this guide |
|---|---|---|---|
| CSV on the Web | Column types and annotations for tidy files | Chapters 4 and 7 | |
| W3C PROV | Provenance of derived data (wasDerivedFrom) | Chapter 7 | |
| schema.org Dataset | Markup on table pages, with isBasedOn and distribution | NADA writes schema.org on catalog pages | Chapter 8 |
| DataCite | DOIs for documents and extracted tables | The Microdata Library assigns DataCite DOIs to datasets, and the Open Knowledge Repository assigns DOIs to publications (prefix 10.1596) | Chapter 8 |
| Creative Commons Attribution 4.0 | Licence inherited by extracted data | The default for World Bank datasets; World Bank publications carry CC BY 3.0 IGO | Chapter 7 |
SDMX OBS_STATUS | The model for a status per value (verified, flagged, estimated) | Chapter 5 |
Corrections
Standards change versions and URLs. If a reference here is wrong or out of date, open an issue with the row and the corrected source.