Practical Guide to the Data Snapshot End-to-End Workflow
A cookbook for developers and data curators working with institutional PDFs. Version 0.1 (working draft), October 2026.
This guide is working material of the AI for Data – Data for AI program. It does not represent official World Bank Group guidance or policy. The contributing page explains how to report corrections and propose reviewed examples.
Institutional reports contain charts, tables, maps, and monitoring summaries whose analytical meaning depends on their visual structure. A data snapshot preserves a bounded analytical region as an image. The production ai4data.data_snapshot workflow detects candidate regions and can describe their visible content with structured semantic metadata.
This guide follows the practical questions involved in preparing a PDF, extracting data snapshots, generating metadata, and reviewing the files. It then explains combined execution, recovery, model selection, and proposed improvements. The recorded workflow processes the sample PDF used in the tutorial with DocLayout-YOLO and includes separate metadata requests for one figure and one table.
The result is a set of PNG images and Schema v1.4 JSON files. Numerical transcription, publication, search, and other downstream processing are possible next steps, covered separately by the Extracting Data from Documents cookbook. This guide is self-contained for the Data Snapshot workflow and does not require completing that cookbook first.
Prerequisites
- Python 3.11 or later, an isolated environment, and a checkout of the repository.
- A developer can install optional dependencies and run commands; a curator can inspect institutional charts and tables.
- The sample PDF used in the tutorial is downloaded from its publisher following chapter 1.
- Metadata execution requires an authorized provider key and credits. Snapshot-only execution does not require that key.
First steps
The shortest path uses the recorded example's default configuration and the two selected data snapshots. Alternative models can be considered after these outputs have been reviewed.
- Prepare the sample PDF and Python environment in chapter 1. The examples use DocLayout-YOLO on CPU and the unchanged OpenRouter preset. Allow about forty minutes for preparation, plus dependency installation time.
- Run snapshot-only extraction and inspect the chart and table in chapter 2. Allow about twenty-five minutes for command preparation and data snapshot inspection, plus detector execution time.
- Generate metadata for only those two data snapshots and inspect the JSON in chapter 3. The extraction script validates each response before saving; repeating validation is optional. Allow about thirty minutes for selection, submission, and local inspection, plus provider waiting time. The downloadable responses can also be inspected without making new requests.
- Review the source pages, data snapshots, and raw responses in chapter 4. Allow about forty minutes to identify missing context and questionable metadata and consider the suggested next steps.
Chapter 5 then explains the supported combined command and its expected output tree. Chapter 6 covers recovery and request control, chapter 7 explains model selection and alternatives, and chapter 8 describes proposed improvements. All times are indicative estimates of human work. They are not measured model runtimes or provider latency.
Chapters and questions
| Chapter | Question | Practical topics |
|---|---|---|
| 1. Environment and source documents | What must be prepared before extraction? | Source publication, environment, dependencies, and credentials |
| 2. Snapshot extraction | How are figures and tables extracted as images? | Candidate PNG outputs, selected data snapshots, and missing context |
| 3. Semantic metadata extraction | How is a data snapshot described in structured metadata? | Two selected requests, raw JSON, and schema validation |
| 4. Data snapshot and metadata review | Are the extracted outputs fit for use? | Source comparison, candidate completeness, and metadata accuracy |
| 5. Combined execution and output structure | How are both stages invoked together? | One-step execution for a PDF or directory and expected output pairs |
| 6. Recovery and controlled execution | How can the pipeline be operated reliably? | Partial failures, skip rules, safe reruns, and request scope |
| 7. Models and settings | Which models and settings are available, and how are they selected? | Detector parameters, benchmark results, and metadata provider selection |
| 8. Extraction improvements and future models | How can we improve snapshot extraction performance? | New detector benchmarks, relevance filtering, and context association |
Maturity levels
| Level | Meaning |
|---|---|
| Foundational | A person can inspect and document the source and results. |
| AI-ready | Documented interfaces and validated representations support repeatable processing. |
| AI-native | Monitored automation retains evidence and human review; proposed extensions are explicitly identified. |
Running example
The sample PDF used in the tutorial is the institutional publication From Stateless to Citizens: The Journey of Shona in Kenya. The ReliefWeb report links to the PDF attachment. PDF page 4 contains both selected artifacts. Their numerical observations remain source content. No numerical transcription was performed for this guide.
The recorded detector output contains seven figure-labelled candidate images and one table-labelled candidate image, including an irrelevant photograph. The selected chart is figure_001; the existing gold dataset uses figure_000 for a different extraction history. Those filenames are not interchangeable identifiers for reviewed metadata.
Scope of the examples
The figure and table data snapshots are actual production outputs; their metadata records come from separate selected-image requests. The combined section shows the expected output layout for both stages. Other detectors are documented but not benchmarked here. The second-stage classifier and improvement roadmap are proposed extensions. The example was run on 30 October 2026 using DocLayout-YOLO 0.0.4, CPU, 300 DPI, and the unchanged packaged OpenRouter preset. The two JSON files are unedited model responses, with interpretation findings in chapter 3 and review observations in chapter 4.
Files
| File | Used in |
|---|---|
| inspect_metadata.py | Chapter 3 uses this helper, which leaves its input unchanged, to validate one metadata record and display its populated fields. |
| policy_brief_on_shona_community_from_stateless_to_citizens_figure_001.json | Chapters 3 and 4 inspect this unedited chart response, including the axis-label title finding. |
| policy_brief_on_shona_community_from_stateless_to_citizens_figure_001.png | Chapters 2 to 4 use this selected chart data snapshot as a metadata input and compare its boundaries with the page. |
| policy_brief_on_shona_community_from_stateless_to_citizens_table_000.json | Chapters 3 and 4 inspect this unedited table response and its relationship to the visible rows and year columns. |
| policy_brief_on_shona_community_from_stateless_to_citizens_table_000.png | Chapters 2 to 4 use this selected demographic-table data snapshot as the second metadata input. |
| shona-page-4.webp | Chapters 2 and 4 use this source-page image to locate both artifacts and identify captions and source notes outside the data snapshots. |
Related resources
| Resource | Purpose |
|---|---|
| Data Snapshots overview | This page introduces the workstream and implementation boundaries. |
| Schema reference | This reference defines semantic fields and validation rules. |
| Production README | This reference documents supported interfaces and configuration. |
| Benchmark paper | This paper defines analytical snapshots and evaluates detection and spatial quality. |
| Data Snapshot dataset | This dataset provides source documents and human-reviewed annotations. |
Citation
World Bank. 2026. Practical Guide to the Data Snapshot End-to-End Workflow. Version 0.1. AI for Data – Data for AI. https://worldbank.github.io/ai4data/cookbook/data-snapshot-end-to-end/
Version history
- 0.1 (October 2026). This first draft documents the tutorial example and production interfaces, with proposed improvements explicitly identified.