Practical Guide to Refining Data Snapshot Metadata Extraction Policies with Model Councils
A cookbook for data scientists, ML engineers, and AI application developers. Version 0.1 (working draft), November 2026.
This guide documents a proposed experiment. The snapshots and published human-reviewed annotations are real. All round extractions, reviews, deliberations, and policy revisions are illustrative.
Metadata extracted from a data snapshot can conform to the schema even when the model assigns an axis label to the title field or infers a value that the snapshot does not support. These errors can help identify where the extraction policy, the instructions that guide the model, needs to be more precise.
This guide proposes an experiment that uses a council of models to refine an initial extraction policy. The council uses findings from each review round to propose and critique changes. The experiment aims to develop a metadata extraction pipeline by pairing a fixed reference extractor with the final policy. The guide starts with a prepared example so the team can see how an error becomes a candidate rule before setting up a round.
The registration script in Chapter 2 runs offline on local files. Chapters 3–6 use the prepared round records to explain the proposed Pi workflow, human decisions, council deliberation, and next-round choices. The Pi runner and its manifest-generation tool still need implementation.
Prerequisites
The walkthrough uses the examples supplied with the guide. For a round using the team's own data, the following are needed.
- Familiarity with multimodal model inference, prompts, structured outputs, and basic evaluation.
- Existing data snapshots, their source references, and the Data Snapshot Metadata Schema v1.4 reference.
- A human reviewer who understands the metadata schema and can assess extraction issues against the source evidence.
First steps
- Start with the prepared D1 example in the workflow chapter. Follow one title error from the data snapshot to a proposed policy revision. Allow about an hour.
- Use the preparation chapter to set up a round. Verify the data snapshots and their references, then record the starting policy and settings. Allow about two hours.
- Use the startup chapter to walk through how the proposed Pi setup would turn those inputs into extraction records and judge findings. The walkthrough stops at the point where a finding needs a human decision.
The self-assessment helps teams decide where to start.
Chapters and questions
| Chapter | Question | What the reader does |
|---|---|---|
| 1. One round at a glance | What does a round start with, and what does it produce? | Follow D1 from the starting policy to a proposed rule revision. |
| 2. Round preparation | What must be prepared before starting the experiment? | Select the data snapshots, verify the annotations, and record the batch, starting policy, and settings. |
| 3. Round configuration and startup | How would the prepared batch become extraction records and judge findings? | Trace the proposed Pi setup through extraction and diagnosis and identify where it would pause. |
| 4. Human review of findings | What needs a human decision before the council begins? | Resolve escalated findings and spot-check selected fields before deliberation. |
| 5. Council deliberation and the next policy | How do the reviewed findings become a policy for the next round? | Follow the proposals and critiques, then check what the orchestrator changed and carried forward. |
| 6. Decisions after a round | Should the team run another batch, investigate an open issue, or move to independent evaluation? | Use the round findings to decide what to do next. |
Maturity levels
| Level | Meaning |
|---|---|
| Foundational | People can inspect the workflow and review a round from its records. |
| AI-ready | Structured artifacts, stable identifiers, and documented checks support repeatable handoffs. |
| AI-native | The team designs model coordination with provenance, scoped access, and verification built into each handoff. |
Running example
The Data Snapshot dataset contains more than 3,900 data snapshots. The gold metadata release contains 102 human-reviewed records. The walkthrough uses the following four data snapshots, two with human-reviewed annotations.
| Sample | Batch | Snapshot and focus |
|---|---|---|
| D1 | Discovery | The Shona labor-force chart distinguishes a caption from an axis label. |
| D2 | Discovery | The WPS5070 rural Indonesia regression table distinguishes variables, statistics, and lag groupings. |
| C1 | Challenge | The Shona demographics table shows row-specific units but omits the caption. |
| C2 | Challenge | The WPS4771 guarantee-systems chart shows categories and legend entries but does not identify the counted entity. |
The sample register pins the dataset revisions, source pages, and image fingerprints. The preparation chapter explains the pairings and reuse information.
Scope of the examples
The example illustrates one round, from the starting policy and state (P001 and S001) to the candidate policy and updated state (P002 and S002). The walkthrough uses selected fields to make each decision inspectable. Observations across rounds would be needed to assess stabilization, and independent evaluation would measure extraction quality. Distinguishability judging, fine-tuning, and operational deployment are outside this design.
Files
The downloadable round files, registration script, and role instructions are in website/static/cookbook-files/data-snapshot-metadata-council/. In the supplied teaching register, image and annotation paths are relative to website/static/. A register generated for a local experiment uses paths relative to its own directory, as described in Recipe 2.1.
| File | Used in |
|---|---|
| Attribution and reuse notes | Source attribution and artifact overview |
| Sample register | Exact data snapshots, source documents, annotation availability, and checksums |
| Registration script and source CSV | Build a register from a local batch directory and its source attribution |
| Role instructions | Templates for extraction, diagnosis, review, deliberation, and synthesis |
| D1 image and D1 gold | Shona labor-force chart and its published annotation |
| D2 image and D2 gold | Rural Indonesia regression table and its published annotation |
| C1 image | Shona demographics table snapshot |
| C2 image | WPS4771 guarantee-systems chart |
| Initial policy P001 and candidate P002 | Starting instructions and proposed revision |
| Initial state S001 and state S002 | Initial conditions and handoff to the next round |
| Extraction records | Schema v1.4 payloads under P001 |
| Judge reviews | Field evidence, reference status, and routing |
| Human review | Escalation and spot-check decisions |
| Council deliberation | Proposals, cross-critiques, revisions, and synthesis |
| Round findings | Dispositions, rule tally, and open issues |
| Round manifest | Batch, version relationships, and artifact inventory |
Related resources
| Resource | Purpose |
|---|---|
| Extracting Data from Documents | Learn how to extract and verify numerical observations. |
| Publishing ML-Ready Datasets | Document and version datasets and plan splits. |
| Evaluation Suites for Statistical AI | Design independent quality assessment. |
Citation
World Bank. 2026. Practical Guide to Refining Data Snapshot Metadata Extraction Policies with Model Councils. Version 0.1. AI for Data – Data for AI. https://worldbank.github.io/ai4data/cookbook/data-snapshot-metadata-council/
Version history
- 0.1 (November 2026). Working draft of the experiment design and illustrative round, revised for editorial review under the current title.