Skip to main content

Practical Guide to Refining Data Snapshot Metadata Extraction Policies with Model Councils

A cookbook for data scientists, ML engineers, and AI application developers. Version 0.1 (working draft), November 2026.

Working draft

This guide documents a proposed experiment. The snapshots and published human-reviewed annotations are real. All round extractions, reviews, deliberations, and policy revisions are illustrative.

Metadata extracted from a data snapshot can conform to the schema even when the model assigns an axis label to the title field or infers a value that the snapshot does not support. These errors can help identify where the extraction policy, the instructions that guide the model, needs to be more precise.

This guide proposes an experiment that uses a council of models to refine an initial extraction policy. The council uses findings from each review round to propose and critique changes. The experiment aims to develop a metadata extraction pipeline by pairing a fixed reference extractor with the final policy. The guide starts with a prepared example so the team can see how an error becomes a candidate rule before setting up a round.

The registration script in Chapter 2 runs offline on local files. Chapters 3–6 use the prepared round records to explain the proposed Pi workflow, human decisions, council deliberation, and next-round choices. The Pi runner and its manifest-generation tool still need implementation.

Prerequisites​

The walkthrough uses the examples supplied with the guide. For a round using the team's own data, the following are needed.

  • Familiarity with multimodal model inference, prompts, structured outputs, and basic evaluation.
  • Existing data snapshots, their source references, and the Data Snapshot Metadata Schema v1.4 reference.
  • A human reviewer who understands the metadata schema and can assess extraction issues against the source evidence.

First steps​

  1. Start with the prepared D1 example in the workflow chapter. Follow one title error from the data snapshot to a proposed policy revision. Allow about an hour.
  2. Use the preparation chapter to set up a round. Verify the data snapshots and their references, then record the starting policy and settings. Allow about two hours.
  3. Use the startup chapter to walk through how the proposed Pi setup would turn those inputs into extraction records and judge findings. The walkthrough stops at the point where a finding needs a human decision.

The self-assessment helps teams decide where to start.

Chapters and questions​

ChapterQuestionWhat the reader does
1. One round at a glanceWhat does a round start with, and what does it produce?Follow D1 from the starting policy to a proposed rule revision.
2. Round preparationWhat must be prepared before starting the experiment?Select the data snapshots, verify the annotations, and record the batch, starting policy, and settings.
3. Round configuration and startupHow would the prepared batch become extraction records and judge findings?Trace the proposed Pi setup through extraction and diagnosis and identify where it would pause.
4. Human review of findingsWhat needs a human decision before the council begins?Resolve escalated findings and spot-check selected fields before deliberation.
5. Council deliberation and the next policyHow do the reviewed findings become a policy for the next round?Follow the proposals and critiques, then check what the orchestrator changed and carried forward.
6. Decisions after a roundShould the team run another batch, investigate an open issue, or move to independent evaluation?Use the round findings to decide what to do next.

Maturity levels​

LevelMeaning
FoundationalPeople can inspect the workflow and review a round from its records.
AI-readyStructured artifacts, stable identifiers, and documented checks support repeatable handoffs.
AI-nativeThe team designs model coordination with provenance, scoped access, and verification built into each handoff.

Running example​

The Data Snapshot dataset contains more than 3,900 data snapshots. The gold metadata release contains 102 human-reviewed records. The walkthrough uses the following four data snapshots, two with human-reviewed annotations.

SampleBatchSnapshot and focus
D1DiscoveryThe Shona labor-force chart distinguishes a caption from an axis label.
D2DiscoveryThe WPS5070 rural Indonesia regression table distinguishes variables, statistics, and lag groupings.
C1ChallengeThe Shona demographics table shows row-specific units but omits the caption.
C2ChallengeThe WPS4771 guarantee-systems chart shows categories and legend entries but does not identify the counted entity.

The sample register pins the dataset revisions, source pages, and image fingerprints. The preparation chapter explains the pairings and reuse information.

Scope of the examples​

The example illustrates one round, from the starting policy and state (P001 and S001) to the candidate policy and updated state (P002 and S002). The walkthrough uses selected fields to make each decision inspectable. Observations across rounds would be needed to assess stabilization, and independent evaluation would measure extraction quality. Distinguishability judging, fine-tuning, and operational deployment are outside this design.

Files​

The downloadable round files, registration script, and role instructions are in website/static/cookbook-files/data-snapshot-metadata-council/. In the supplied teaching register, image and annotation paths are relative to website/static/. A register generated for a local experiment uses paths relative to its own directory, as described in Recipe 2.1.

FileUsed in
Attribution and reuse notesSource attribution and artifact overview
Sample registerExact data snapshots, source documents, annotation availability, and checksums
Registration script and source CSVBuild a register from a local batch directory and its source attribution
Role instructionsTemplates for extraction, diagnosis, review, deliberation, and synthesis
D1 image and D1 goldShona labor-force chart and its published annotation
D2 image and D2 goldRural Indonesia regression table and its published annotation
C1 imageShona demographics table snapshot
C2 imageWPS4771 guarantee-systems chart
Initial policy P001 and candidate P002Starting instructions and proposed revision
Initial state S001 and state S002Initial conditions and handoff to the next round
Extraction recordsSchema v1.4 payloads under P001
Judge reviewsField evidence, reference status, and routing
Human reviewEscalation and spot-check decisions
Council deliberationProposals, cross-critiques, revisions, and synthesis
Round findingsDispositions, rule tally, and open issues
Round manifestBatch, version relationships, and artifact inventory
ResourcePurpose
Extracting Data from DocumentsLearn how to extract and verify numerical observations.
Publishing ML-Ready DatasetsDocument and version datasets and plan splits.
Evaluation Suites for Statistical AIDesign independent quality assessment.

Citation​

World Bank. 2026. Practical Guide to Refining Data Snapshot Metadata Extraction Policies with Model Councils. Version 0.1. AI for Data – Data for AI. https://worldbank.github.io/ai4data/cookbook/data-snapshot-metadata-council/

Version history​

  • 0.1 (November 2026). Working draft of the experiment design and illustrative round, revised for editorial review under the current title.