Back to Home

PRWP Reproducibility Protocol

This protocol outlines the step-by-step process for verifying the reproducibility of research packages submitted to the World Bank’s PRWP verification team. It ensures that findings can be independently reproduced using the submitted code, data, and instructions. The protocol covers the full workflow—from submission and completeness checks (including data access and documentation), to running the package in a clean environment, tracking changes via version control, verifying consistency with the manuscript, and ultimately publishing the final reproducibility package.

In brief, the process looks like:

image

Find the detailed workflow below:

1. Receive Submission

RR_NGA_2024_213/          <- named after the package ID
├── files_submitted/      <- files submitted by the authors
│   ├── v1/
│   └── v2/               <- added if a new version is received
├── results/              <- verification report and outputs
│   ├── v1/
│   └── v2/
└── package RRR/          <- final package as published on RRR
    ├── v1/
    └── v2/

General notes:

2. Download and Set Up the Package

## This .gitignore is for reproducibility packages to track changes in outputs only
 
## Binary output files - Git cannot show meaningful diffs for these
*.pdf
*.xlsx
*.xls
*.dta
*.zip
*.docx
*.doc
*.stswp
*.rar
*.7z
 
## R and system files
.Rhistory
.DS_Store
 
## Ignore the raw data folder
## Note: adapt the folder name to match the package structure - common names include
## raw_data, data, Data, raw, Raw Data, etc.
/raw_data

Important: This is a local repository only. Do not publish it to GitHub.com at any point.

This setup allows you to track any changes made to the code and see how outputs change across runs, making it easy to identify discrepancies between the author’s original package and your working version.

3. Review Package Contents

3.a Verify Completeness

Before proceeding, verify that the package has enough to work with. If anything critical is missing, return the package to the author before running the code.

When to return the package to the author:

3.b Verify Data Access

3.c Verify Public Data

4. Run the Code

4.a Set Up a Clean Environment

Set up a clean programming environment before running the package. This ensures that the results are not affected by packages or dependencies already installed on your machine, and that the environment can be replicated by others.

4.b Run the Package

Document these critical aspects as you run the package:

Dependencies

System Information

4.c Verify Stability

4.d Send Confirmation to Authors

Do not proceed to the next step if any of the following apply - return the package to the authors first:

5. Verify Consistency with the Manuscript

6. Draft Reproducibility Report

For detailed instructions on how to complete the report, refer to the Reproducibility Report Template Instructions.

7. Prepare Metadata and Publication

For full instructions on preparing metadata and publishing the package, refer to the Metadata Editor Protocol.

8. Publish Package to reproducibility.worldbank.org

9. Send Reproducibility Report and Package to Authors

10. Update the Status of the Review in the GitHub Dashboard

Special Cases

New Versions of a Published Entry

New versions are updates of the code or data that follow a modification of a manuscript exhibit or a new exhibit that was not included in the original reproducibility package. A new version should be included as a new entry in RRR. Updates only to the documentation or metadata are not considered new versions and can be updated in the same entry. Cases that do not fall in these examples should be discussed in the weekly team meeting.

Follow these steps to update a new version of a reproducibility package:

Back to Home