Code Reproducibility Practices

The standard every MML data project must follow — setup, data provenance, analysis parameters, staged outputs, cleanup, and package dependency

Start from the template, not an empty folder

Clone mmlTemplate — it already implements the standard described on this page.

The goal is simple to state and demanding to achieve: any colleague should be able to take your project, run it on their own machine, and reproduce your results exactly. The template is not an illustration of the standard — it is the standard, in code you can run today. Everything below explains what the template does and why.

All code here is Stata, because the template is Stata. Stata 16 is the floor: the parameter table lives in a frame. R and Python equivalents will follow later.

Two neighbouring documents carry the rest of the picture. Where the data comes from and where finished work is deposited is Data management and archiving. How the code itself is versioned and shared is Version control: Git + GitHub.

The Git exercise is not an example of this standard

Git and GitHub: your solo workflow uses a small hh-survey project stripped to one do-file and one dataset so that nothing distracts from the commands. It shows how a project is versioned, not what a project must contain. The contract below is the part you apply when you build a real project.

1 Start here: clone the template

Open https://github.com/worldbank/mmlTemplate and press Use this template to create your own repository. Clone it, open code/00_master.do, and run it. It works before you have any data.

Out of the box you get:

code/00_master.do — the single entry point that initialises, cleans, and runs the three stages in order.
code/00_init.do — paths, folder creation, package loading, and the fail-loud checks.
code/00_ado.do — external and project-owned Stata packages.
code/01_presim.do, 02_sim.do, 03_postsim.do — the three stages, wired end to end on synthetic data.
code/ado/ — five small mml_* helper commands, each with a .sthlp help file.
inputs/parameters.csv and inputs/parameters.do — the analysis assumptions, committed to git.
A README.md new-project checklist and a .gitignore that already excludes produced data.

Fill in the README checklist first, before writing any analysis code. It asks the four questions a stranger will ask: is the data documented and obtainable, does the code say where to start, are the environment and dependencies documented, and are the results reproducible. Answering them at the end is how projects end up undocumented.

2 Anatomy of a project

Three folders live in git, and one archive sits outside the working space entirely.

mmlTemplate/
├── code/            pipeline do-files, flat and stage-prefixed
│   └── ado/         Stata commands owned and versioned by this project
├── inputs/          analysis assumptions — committed, never data
├── runs/            everything the pipeline produces — gitignored
│   └── <username>/  one identically shaped folder package per person
└── README.md        what this is, where to start, how to reproduce it

archive-microdata/   read-only raw data, OUTSIDE the repo, never in git
└── <COUNTRY>/<SURVEY>/

The split is the whole design. Code and assumptions are versioned, produced data is disposable, and raw data is untouchable.

3 Setup: two paths per person, plus one optional

A user sets two globals in a block keyed on their username. Everything else is derived from them.

global suser = lower(c(username))

if ("${suser}"=="wb532966") {
    global gdProject = "C:/Users/${suser}/eb-local/mmlTemplate"
    global gdArchive = "C:/Users/${suser}/OneDrive - WBG/MML-sharepoint/_data-repo"
    *global gdRuns   = "C:/Users/${suser}/"  // OPTIONAL: runs outside the repo
}

$gdProject is the working copy of the repository, synced through GitHub. $gdArchive is the read-only raw-data archive, synced through OneDrive. $gdRuns is the optional third: set it to put the run folders on a shared or larger drive, leave it commented and runs land in ${gdProject}/runs. A new collaborator adds one if block and touches nothing else.

$gdProject is checked immediately, before anything derives from it. It is the one path that cannot be checked by a helper command, because the helper commands live inside it — so init lists the folder directly, and an unset global fails the same test as a typo:

cap local ls : dir "${gdProject}" dirs "*"
if _rc {
    di as error "gdProject is not set, or does not exist: [${gdProject}]"
    di as text "Configure 'code/00_init.do' before running the code."
    exit 601
}

cd "${gdProject}"

The cd puts every relative path in the session on the project root, so a stage never depends on where the user happened to be.

Init then defines $gdCode and $gdAdo, loads 00_ado.do — which puts code/ado/ on the adopath — and only afterwards uses the mml_* commands for everything else. The archive is checked with a warning, not an error, because it is an external OneDrive folder and a run on synthetic data does not need it:

mml_check_dir gdArchive, globals ///
    msg("Set it in 'code/00_init.do' if this run reads raw data.")

globals means the argument is the name of a global, so an unset one is reported as gdArchive rather than as an empty path nobody can act on. A stage that actually needs the archive stops on its own.

Every derived path follows, with no trailing slashes, since every consumer writes "${gdPresim}/file.dta".

global gdInputs = "${gdProject}/inputs"
if ("${gdRuns}" == "") global gdRuns = "${gdProject}/runs"
global gdRun = "${gdRuns}/${suser}"

Every script in code/ opens with the same self-initialising block, so each stage runs standalone and init is a no-op once it has run in the session. The candidate paths are tested with confirm, not tried under cap do: running the wrong candidate that way printed a spurious “file not found” on every run from code/.

if ("${init_success}" != "1") {
    foreach f in "code/00_init.do" "00_init.do" "../code/00_init.do" {
        cap confirm file "`f'"
        if (!_rc & "${init_success}" != "1") do "`f'"
    }
}

if ("${init_success}" != "1") {
    di as error "00_init.do did not run — cd to the project root or code/ first."
    exit 601
}

The second check is not redundant. Without init every ${gd*} global expands to nothing, and the stage would happily write example_presim.dta to the filesystem root.

4 Data provenance

Raw microdata lives in a read-only archive outside the working space and is never committed. Init derives its location and never creates it:

global gdRaw = "${gdArchive}/{COUNTRY}/{SURVEYSUBFOLDER}"

A fresh clone still carries the {COUNTRY} placeholder, and that is deliberate: the template runs end to end on synthetic data before it is pointed at an archive, so the wiring is testable before the data arrives.

Only 01_presim.do may touch raw. Every later stage reads a product folder, so the archive is opened exactly once in the project and the harmonisation step is the single place where source decisions are made.

Every file opens with a header block naming the project, the description, the authors, and the date it was last updated. Every stage header additionally declares what it reads and what it writes:

*# 02_sim.do: EXAMPLE simulation stage ----------------------------------------
*#
*# READS  : ${gdPresim} ONLY. Not the raw archive, not the temp folders.
*# WRITES : ${gdSimTemp}/example_sim_step1.dta   (intermediate, disposable)
*#          ${gdSim}/example_sim.dta             (the stage's product)

What does “provenance” mean? The documented origin and chain of custody of a dataset: where it came from, who produced it, when, under what version or survey round, and what was done to it along the way. A dataset with good provenance can be traced back to its source and regenerated; a dataset without it is a mystery file whose results cannot be trusted.

5 Analysis parameters: `inputs/`, in two forms

Not everything a run depends on is data. Growth rates, elasticities, equivalence scales, poverty lines, scenario definitions — these are assumptions about the analysis, and they change the numbers in the results. They live in inputs/, they are committed to git, and git log inputs/ is the record of which assumptions produced which results.

Two forms are provided, and both are loaded once, by 00_init.do.

Form	Use it when	Example
`inputs/parameters.csv`	the parameter is a plain scalar	`growth_rate,0.05`
`inputs/parameters.do`	the parameter is derived, is a list, or is a table	`p_growth_pct`, `p_scenarios`, scenario × multiplier

Rule of thumb from the template: if you would have to write the same number twice, or compute it inside a stage, it belongs in parameters.do.

Both files are optional. Init loads whichever is present:

cap confirm file "${gdInputs}/parameters.csv"
if (!_rc) mml_csv_to_globals using "${gdInputs}/parameters.csv", prefix(p_)

cap confirm file "${gdInputs}/parameters.do"
if (!_rc) qui do "${gdInputs}/parameters.do"

Optional does not mean silent. The stage that needs a parameter is the place that stops, and it stops by name (see below), which is more useful than init refusing to start over a file the run may not need.

5.1 The flat form

name,value,description
growth_rate,0.05,Income growth applied to every household in 02_sim
adult_equivalence,2,Divisor turning household income into per-adult-equivalent income

Anyone can edit it without touching Stata code, and a diff reads as one line per assumption. mml_csv_to_globals turns each row into a global, and prefix(p_) makes growth_rate into ${p_growth_rate}.

The prefix is not decoration. Un-prefixed, a row named suser would silently overwrite the run identity and the mistake would only surface as wrong output paths — which is why the command refuses to define suser whatever the csv says. It also reads the file in a temporary frame, so the data you had in memory survives, and imports every column as a string so that 0.05 is kept exactly as written instead of round-tripping through a float. Stata expands the global into the expression, so arithmetic on ${p_growth_rate} still works.

5.2 The code form

inputs/parameters.do runs immediately after the csv, so it can build on what the csv set.

global p_base_year = 2024
global p_scenarios = "low base high"   // the menu of scenarios in the table
global p_scenario  = "high"            // the ACTIVE scenario for this run

mml_check_global p_growth_rate, fail ///
    msg("It is a row of inputs/parameters.csv. Restore the row, or set it here.")
global p_growth_pct = 100 * ${p_growth_rate}

clear
input str8 scenario double multiplier str24 label
"low"  0.5 "Low growth"
"base" 1.0 "Baseline growth"
"high" 2.0 "High growth"
end

cap frame drop params
frame put scenario multiplier label, into(params)
clear

frame params {
    qui count if scenario == "${p_scenario}"
    if (r(N) != 1) {
        di as error "p_scenario = [${p_scenario}] is not a row of the parameter table."
        di as text  "Known scenarios: ${p_scenarios}. See inputs/parameters.do."
        error 459
    }
}

Three things are happening here.

The derived value is computed once. p_growth_pct is printed in the figure subtitle by 03_postsim; a second copy of 100 * somewhere else is how a plot and its table start disagreeing. The mml_check_global guard above it is there because the csv is optional: without it, 100 * ${p_growth_rate} with an empty macro is invalid syntax, which names nothing.

A scenario × multiplier table is the shape a flat name,value file cannot hold at all. It is kept in a Stata frame rather than a .dta, so no generated binary lands in git and no stage has to reload it from disk. Frames survive the use and clear that stages run — but not clear all, which is exactly when init re-runs anyway.

The active scenario is validated against the table. A typo in ${p_scenario} would otherwise reach 02_sim as an empty multiplier and silently simulate nothing.

5.3 Data does not go in `inputs/`

inputs/ is for the numbers a reader of the paper would want to see. Data is $gdRaw: read-only, outside the repository, never committed.

5.4 How stages consume them

A stage names what it needs and stops if it is missing, listing every missing parameter at once rather than one per re-run:

mml_check_global p_growth_rate p_adult_equivalence p_scenario, fail ///
    msg("They come from inputs/parameters.csv and inputs/parameters.do.")

Then it reads the globals instead of hard-coding the values.

qui gen double income_pc = income / ${p_adult_equivalence}

That line is in 01_presim.do, and 02_sim.do divides by the same global. It cannot drift between stages the way two hard-coded 2s would.

03_postsim.do goes one step further and writes the parameters into both the exported table and the figure, so a stray .csv or .png on someone’s desktop still says which assumptions produced it.

frame params {
    qui levelsof label if scenario == "${p_scenario}", local(slab) clean
    qui su multiplier if scenario == "${p_scenario}", meanonly
    local mult = r(mean)
}
local gr_eff = ${p_growth_pct} * `mult'

qui gen double growth_effective = ${p_growth_rate} * `mult'
qui export delimited using "${gdOut}/example_indicators.csv", replace

qui graph bar baseline simulated, ///
    subtitle("`slab' — effective growth `gr_eff'% — base year ${p_base_year}") ///
    note("EXAMPLE figure produced by 03_postsim.do" ///
         " — parameters from inputs/parameters.csv and inputs/parameters.do")

Note that it reports effective growth — the rate times the scenario multiplier — not the raw rate. Labelling the plot with 5% while the bars move by 10% is how a figure ends up contradicting itself.

6 Staged outputs: pre-simulation, simulation, post-simulation

A project runs in three stages, and each writes into a folder package that is separated per user but identical in shape, so results can be compared across people line for line.

foreach dir in "${gdRuns}" "${gdRun}" "${gdInputs}" ///
               "${gdPresim}" "${gdPresimTemp}" ///
               "${gdSim}" "${gdSimTemp}" ///
               "${gdPostsim}" "${gdPostsimTemp}" ///
               "${gdOut}" {
    mml_mkdir "`dir'"
}

That is runs/<username>/ containing 01-presim, 01-presim-temp, 02-sim, 02-sim-temp, 03-postsim, 03-postsim-temp, and outputs. Seven folders, one per stage product, one per stage scratch space, one for the finished table and figure. Add sub-folders under them when a real project needs validation or diagnostic output; do not add a parallel tree beside them. runs/ is gitignored: it holds produced data, and init recreates it on every run.

The tree is built by mml_mkdir, because Stata’s own mkdir creates only one level and errors when the parent is missing.

Each stage obeys one contract:

Reads only the previous stage’s product folder — never raw, never another stage’s temp.
Writes intermediates to its matching *-temp folder, and its product to the stage folder.

Because the hand-off is a folder on disk, any stage can be re-run on its own from what is already there. A missing input is loud, never a silent skip:

cap confirm file "${gdPresim}/example_presim.dta"
if _rc {
    di as error "02_sim: ${gdPresim}/example_presim.dta not found — run 01_presim first."
    exit 601
}
qui use "${gdPresim}/example_presim.dta", clear

The stage folder was emptied seconds earlier by the master. A silent skip would leave an empty folder and no explanation.

7 Cleanup of temporary files

Temporary files left behind quietly pollute the next run. 00_master.do therefore empties each stage folder and its temp folder immediately before running that stage.

foreach dir in "${gdSim}" "${gdSimTemp}" {
    mml_clean "`dir'"
}
do "${gdCode}/02_sim.do"

Two properties of mml_clean are worth stating explicitly.

It leaves the directory skeleton standing. Stages save straight into sub-folders of the folders being cleaned, and removing those sub-folders here would make every later save fail.

It refuses any path that is not at or below its root — $gdRun unless root() says otherwise — and re-checks on every recursive call, not just the outer one.

if (`"`cpath'"' != `"`croot'"' & strpos(`"`cpath'"', `"`croot'/"') != 1) {
    di as error `"mml_clean: [`path'] is outside [`root'] — refusing."'
    exit 198
}

The guard exists because this program deletes recursively and without asking. An unset global expands to nothing, so an unguarded mml_clean "${gdSim}" after a failed init would walk the working directory instead of a run folder.

Three details in that one comparison are each a hole that had to be closed. The trailing / on the root matters: without it, a root of …/out would also authorise the unrelated sibling …/output. Paths are lower-cased first on Windows, because C:/Runs and c:/runs are one folder and refusing the second would be wrong. And .. is rejected outright anywhere in either path, because "${gdRun}/.." passes a prefix test and then resolves to the parent.

8 Helper commands (`code/ado/`)

00_ado.do puts code/ado/ on the adopath, so these five are available in every stage. Each ships with a help file — help mml_mkdir and friends.

Command	What it does
`mml_mkdir`	Creates a folder and every missing parent above it.
`mml_clean`	Erases every file below a folder but keeps the folders; refuses any path outside `$gdRun`.
`mml_check_dir`	Reports every missing folder at once, or with `globals` the name of the global holding it; warns by default, `fail` to stop.
`mml_check_global`	Reports every global in a list that is not set; warns by default, `fail` to stop.
`mml_csv_to_globals`	Turns a `name,value` csv into globals, optionally with `prefix()`.

They share one design rule: report everything at once, then decide whether to stop. A check that dies on the first missing folder makes the user re-run the pipeline once per problem. warn is the default and fail is the opt-in, so init can be lenient about the archive while a stage is strict about the parameters it actually uses.

9 Package dependency

00_ado.do installs external commands once, then puts the project’s own commands on the ado path. 00_init.do runs it, so no stage has to.

qui cap which ineqdeco
if _rc {
    cap ssc install ineqdeco, replace
    if _rc di as error "  could not install ineqdeco (rc=" _rc ") - offline?"
}

qui cap which un_pop
if _rc {
    cap net install un_pop, ///
        from("https://raw.githubusercontent.com/wbEPL/unPopAdo/main/") replace
    if _rc di as error "  could not install un_pop (rc=" _rc ") - offline?"
}

if ("${gdAdo}" != "" & !strpos(`"`c(adopath)'"', "${gdAdo}")) adopath ++ "${gdAdo}"

Every install is capped and only warns. No internet, a corporate proxy, or a dead GitHub URL must not stop a run that never uses the package. The adopath line checks before adding, because re-adding the same folder on every init would grow c(adopath) all session.

Anything the project itself owns goes in code/ado/ and is version-controlled with the project. The five mml_* commands and sp_groupfunction.ado are the template’s examples: vendored into the repository, so a stranger cloning the repo gets exactly the copies that produced the results.

Be honest about the limit. Pinning versions of user-written Stata packages remains unsolved — ssc install fetches whatever is current today. Vendoring the commands that matter into code/ado/ under git is the current partial answer, not a complete one.

10 The reproducibility contract in one breath

A project built on the template can be handed to any colleague and reproduced exactly, because it:

Sets up from two per-user globals in a c(username) block — three with the optional runs root — derives every other path, and stops on gdProject before anything else can go wrong.
Keeps its data provenance in a read-only archive outside the repo, touched by one stage only, with every file declaring what it reads and writes.
Keeps its analysis parameters in inputs/, committed to git, as a flat csv for scalars and a do-file for anything derived, listed, or tabular — and stops by name in the stage that needs one.
Runs in three staged outputs into a per-user, identically shaped folder tree, where each stage reads only its predecessor’s product.
Cleans each stage folder and its temp folder before that stage runs, under a guard that refuses to delete outside the run folder.
Declares its package dependencies in 00_ado.do and vendors its own commands into code/ado/.

The template already does all six. Starting from it means the contract is something you maintain, not something you build.

1 Start here: clone the template

2 Anatomy of a project

3 Setup: two paths per person, plus one optional

4 Data provenance

5 Analysis parameters: inputs/, in two forms

5.1 The flat form

5.2 The code form

5.3 Data does not go in inputs/

5.4 How stages consume them

6 Staged outputs: pre-simulation, simulation, post-simulation

7 Cleanup of temporary files

8 Helper commands (code/ado/)

9 Package dependency

10 The reproducibility contract in one breath

5 Analysis parameters: `inputs/`, in two forms

5.3 Data does not go in `inputs/`

8 Helper commands (`code/ado/`)