Data Governance Principles

Raw data (SharePoint) | Code (git+GitHub) | Results (disposable|local or shared)

Flow diagram in four blocks stacked vertically. At the top, a read-only SharePoint block holds the raw country-data archive. Below it sits the local git repository, split into git-tracked folders on the left — code, analysis-specific inputs, documentation — and git-ignored folders on the right, where a runs/ folder holds three products in sequence: harmonised microdata, simulated microdata, and analysis outputs. Arrows run one way only: from the archive and from code into runs/, and from each stage to the next, never back into the archive. At the bottom, arrows from the whole repository point to a second SharePoint block, the completed-projects archive.
Figure 1: Where each kind of file lives, and which way files are allowed to move.

If you remember only one sentence: raw data is an input you never touch, results are an output you can always throw away and reproduce, and everything in between is code under version control.

Principle 1: raw data lives outside the code

  • Raw data sits on SharePoint, read-only, in _data-repo/.

  • Code sits on GitHub, in github.com/worldbank/your_repo: GitHub is a source of truth!

  • The two are joined at run time by a path global: never by copying data into the repository.

TipWhat goes in Git
  • Any code
  • Analysis-specific inputs that are assumptions chosen by the analyst, not observations.
  • Template excel dashboard for the client
    • copied to the results at the analysis stage and populated with the results data.
  • Any number that could be followed by “why is this chosen?”, e.g., poverty lines, conversion rates, equivalence scales
  • Scenario definitions
  • Concordance and mapping tables (sector crosswalks, region recodes, etc.)
  • Files must be small (<5–10 MB), text-based, human-readable diff (.csv, .do, .md, .yaml, etc.), free of personal data

Principle 2: results are disposable

Every file the analysis produces must be reproducible from the raw data by running the code — no matter how long the run takes and how many stages it has.

  • Delete the entire runs/ folder, run 00_master.do, and get your results back.: Nothing produced is precious.

  • Raw and inputs are never written to.

    • Have an excel dashboard? Copy it to runs/ and populate it there. Do not edit the template in inputs/.

  • Manual edits are not allowed.

    • Need to fix the excel dashboard? Edit it in inputs/.
NoteDisposable during the project, frozen at delivery

Disposability is about the working life of a project, not about the record. When a project is delivered, a dated, self-contained copy — raw data, code, inputs, and the outputs actually shown to the client — is deposited in _projects-archive/ and stops changing. The two rules do not conflict: you archive a snapshot precisely because the live folder is meant to be thrown away and rebuilt.

Principle 3: Any data pipeline has three stages

Every MML analysis runs in the same three stages, in this order.

Stage File Reads Produces
Pre-simulation XX_presim.do the raw archive (the only stage that may) harmonised, analysis-ready micro data
Simulation YY_sim.do the pre-sim product (never from the raw archive) simulated household-level data
Post-simulation ZZ_postsim.do the sim product (never from the raw archive or presim) indicators, tables, figures, dashboards

Two rules make the stages worth having.

  • Each stage reads only its predecessor’s product.
  • Each stage hands off through a folder on disk, not through memory.

Why this particular cut:

  • Pre-sim where all source-specific decisions live — variable names, value labels, missing codes, deflators, survey quirks. Isolating them means swapping a country or a survey round touches one file.

  • Sim where the method lives. It sees a clean, standard dataset, knows nothing about where it came from, but knows everything about how it should be transformed.

  • Post-sim where the presentation lives: tables, figures, dashboard (.xlsx/shiny). It never recomputes the simulation, so a formatting change can never move a number.

Where these three principles come from?

None of this is invented here. Founding principals are coming from:

ETL/ELT

Two-row schematic comparing two pipelines. In the top row, labelled ETL, data from three sources passes through a funnel labelled Transform and lands in a block labelled DWH, where the coloured circles are already sorted into neat rows by colour and feed a fixed set of report documents. In the bottom row, labelled ELT, the same source data passes through a funnel labelled Replicate straight into a block labelled Data Lake, where the circles stay mixed and unsorted, and a separate Transform step sits between the lake and a more open-ended set of charts and formulas.
Figure 2: The same data, two orders of operations: ETL sorts before storing, ELT stores first and sorts later.

Every data pipeline does the same three things in a different order:

  • extract data from its sources,
  • transform it into an analysis-ready shape, and
  • load it into wherever the analysis happens.

ETL transforms the data before it is saved in operational format; ELT loads the raw data first and transforms it afterwards on the fly.

Our three principles are ELT, done in Stata.

  • extract: depositing raw data from SharePoint;
  • load: read-only archive - never written to.
  • transform: pre-sim, sim and post-sim everything downstream of the archive is derived by code and can be rebuilt from scratch.

ETL — extract, transform, load

Data is pulled from the sources, cleansed, standardised, aggregated and enriched in a staging area, and only the finished product is written into the target data warehouse.

Pros

  • Consistency, quality and security are enforced before anything lands.
  • Sensitive fields can be masked or encrypted on the way in — good for compliance.
  • Keeps the transformation load off the target system.
  • Fits smaller sources with complex transformation logic.
  • Granular control over every step of the process.

Cons

  • Slower ingestion overall.
  • Extra processing overhead in the intermediary system.
  • Hard to scale to very large data without buying infrastructure.
  • The transformation is fixed up front: a new question means re-running the pipeline.

ELT — extract, load, transform

Data is pulled from the sources and written into the target — a warehouse or a data lake — untouched, and the transformations are run inside the target afterwards.

Pros

  • Flexible: the transformation logic can be decided, and changed, after loading.
  • Faster ingestion, because the data moves once.
  • Uses the massively parallel compute of the target system.
  • More efficient for very large datasets.
  • Adapts quickly when the question changes.

Cons

  • Raw, unmasked data sits in the target until the transformations are done — a compliance risk.
  • Depends on what the target system can do, rather than on your own tooling.
  • Less granular control during the transformation phase.
  • A poor fit where data-handling rules are strict.

Medallion Architecture

Left-to-right pipeline diagram. Batch and streaming sources feed three successive tables: Bronze, described as raw integration and a landing zone with no schema needed; Silver, described as filtered, cleaned and augmented, where structure is defined and schema enforced; and Gold, business-level aggregates delivered continuously to downstream users and apps. An arrow across the top reads Improve Data Quality, and the Gold layer feeds BI and machine-learning consumers.
Figure 3: Bronze → silver → gold: quality improves at each hop, and every layer can be rebuilt from the one before it.

Medallion architecture organises data in three layers that progressively improve its structure and quality:

  • bronze holds every source’s data exactly as it arrived, plus load metadata, and acts as the historical archive from which anything downstream can be rebuilt without going back to the source;

  • silver is the cleaned, filtered and conformed view of that data, standardised across sources and fit for general use;

  • gold is read-optimised, project-specific output — the aggregates, indicators and tables actually consumed by reports and models.

Our pre-sim, sim and post-sim are the same three tiers under different names, and they are cut this way for the same reason: because each layer is derived from the one before it, you can always throw the later ones away and recreate them.

NoteExample: Fiscal Incidence Analysis

The post-simulation stage of the Fiscal Incidence Analysis developed by Pechi and colleagues follows the bronze-silver-gold layer pattern.

Cookiecutter Data Science

Cookiecutter Data Science is “a logical, flexible, and reasonably standardized project structure for doing and sharing data science work” — a project template that fixes where code, data and outputs live, so every project looks the same to the next person who opens it. It is where our “raw data is immutable” rule comes from: data/raw is the original, immutable data dump, and everything else under data/ is derived from it.

ImportantKey difference from our workflow

Cookiecutter builds the pipeline on functions, not scripts. Reproducibility is guaranteed because Python functions encapsulate every step and are called in sequence. Our equivalent is Stata scripts (00_master.do calling numbered .do files), which serves the same purpose in a different language.

├── LICENSE            <- Open-source license if one is chosen
├── Makefile           <- Makefile with convenience commands like `make data` or `make train`
├── README.md          <- The top-level README for developers using this project.
├── data
│   ├── external       <- Data from third party sources.
│   ├── interim        <- Intermediate data that has been transformed.
│   ├── processed      <- The final, canonical data sets for modeling.
│   └── raw            <- The original, immutable data dump.
├── docs               <- A default mkdocs project; see www.mkdocs.org for details
├── models             <- Trained and serialized models, model predictions, or model summaries
├── notebooks          <- Jupyter notebooks. Naming convention is a number (for ordering),
│                         the creator's initials, and a short `-` delimited description, e.g.
│                         `1.0-jqp-initial-data-exploration`.
├── pyproject.toml     <- Project configuration file with package metadata for
│                         {{ cookiecutter.module_name }} and configuration for tools like black
├── references         <- Data dictionaries, manuals, and all other explanatory materials.
├── reports            <- Generated analysis as HTML, PDF, LaTeX, etc.
│   └── figures        <- Generated graphics and figures to be used in reporting
├── requirements.txt   <- The requirements file for reproducing the analysis environment, e.g.
│                         generated with `pip freeze > requirements.txt`
├── setup.cfg          <- Configuration file for flake8
└── {{ cookiecutter.module_name }}   <- Source code for use in this project.
    ├── __init__.py             <- Makes {{ cookiecutter.module_name }} a Python module
    ├── config.py               <- Store useful variables and configuration
    ├── dataset.py              <- Scripts to download or generate data
    ├── features.py             <- Code to create features for modeling
    ├── modeling
    │   ├── __init__.py
    │   ├── predict.py          <- Code to run model inference with trained models
    │   └── train.py            <- Code to train models
    └── plots.py                <- Code to create visualizations

Other literature

The three principles are the standard reproducibility practice of empirical economics and of data engineering generally, stated in MML’s vocabulary.

Source What it says that matches
Wilson et al. (2017), Good Enough Practices in Scientific Computing Literally our Principles 1 and 2: “save the raw data”, “treat data as read-only”, “treat generated output as disposable”
Gentzkow & Shapiro (2014), Code and Data for the Social Sciences Automate everything; one script runs the whole project end to end; separate directories by function and separate inputs from outputs
Bjärkefur et al. (2021), Development Research in Practice (DIME Analytics, World Bank) The same pipeline, cut into four: acquisition → processing → construction → analysis. Our pre-sim covers processing and construction; sim and post-sim split their analysis step
Cookiecutter Data Science “Data is immutable”; data/raw is never edited; the analysis is a directed graph from raw to results
Medallion / lakehouse architecture (bronze → silver → gold) The same three tiers in engineering language: immutable landing zone, conformed data, presentation-layer aggregates
Wilkinson et al. (2016), FAIR principles Findable, Accessible, Interoperable, Reusable — the metadata and documentation side of the archive readme.md requirement
World Bank Reproducible Research Repository The delivery-time test: a third party re-runs the package from scratch and gets the same numbers