3. Gathering and Preparing the Documentation

Documentation should make the data and the processes used to produce them verifiable. This includes recording methods, decisions, transformations, and known quality issues clearly enough for others to review.

All information related to the study throughout the data production lifecycle may be useful and should be archived (even if not all will be disseminated to the public). This includes not only technical documents such as the questionnaires or list of codes (obviously needed by data users), but also administrative reports (potentially useful for implementation of future microdata collection projects), and other documents such as a compilation of the comments provided by stakeholders at the time the questionnaire was designed, etc. All archived materials should follow a standardized folder structure and file naming convention to facilitate discovery, preservation, and future reuse. Resources to be included if available include:

Administrative Documents and Governance Records

• Project charters, budgets, and planning documents
• Key correspondence related to data collection, processing, or management
• Meeting minutes and decision records
• Steering committee or advisory group decisions
• Approval, ethical clearance, and authorization documents
• Data access, sharing, and licensing agreements
• Data governance policies and procedures

Methodology and Data Production Documentation

• Data collection or data acquisition methodology
• Data source descriptions and data lineage documentation (i.e. records of where data came from and the main steps through which they were collected, combined, transformed, or revised.)
• Sampling design documentation (where applicable)
• Weighting methodologies and calculation procedures (where applicable)
• Data integration, linkage, or matching procedures
• Data processing workflows and business rules
• Non-response adjustment procedures (where applicable)
• Fieldwork, operational, or system implementation reports
• Pilot, pre-test, or system testing reports
• Project timelines and production schedules
• Version history and change logs for data and metadata
• Records linking significant edits or transformations to the responsible script, decision, or approval.

Geospatial Resources

• GIS shapefiles and boundary files
• Geographic reference maps
• Spatial sampling or coverage maps
• Geographic code lists and crosswalks
• Geospatial metadata and documentation

Data Collection and Source Materials

• Questionnaires, forms, or data collection instruments
• Administrative forms and reporting templates
• Computer-assisted data collection instruments (CAPI/CATI/CAWI)
• Data submission templates and specifications
• Code lists, classifications, and controlled vocabularies
• Consent forms and participant information sheets (where applicable)
• Reference materials used during data collection
• User guides and operational instructions

Data Processing and Quality Assurance Documentation

• Data validation specifications and business rules
• Data entry and capture procedures
• Data transformation, integration, and processing workflows
• Data cleaning and editing logs
• Quality assurance and quality control reports
• Data consistency and validation checks
• Imputation methodologies
• Derived variable specifications and algorithms
• Version histories and change logs
• System testing and verification reports

Access, Confidentiality, and Dissemination Documentation

• Data access policies
• Terms of use
• Privacy and confidentiality procedures
• Disclosure risk assessments
• Anonymization and de-identification reports
• Data licensing information
• Access restrictions and distribution conditions
• Data sharing agreements
• Metadata publication and dissemination records

Outputs, Analysis, and Communication Materials

• Statistical reports and analytical publications
• Research papers, policy briefs, and working papers
• Dashboards and data visualizations
• Presentations and training materials
• Communication and outreach materials
• Technical notes and methodological papers
• Tables, indicators, and summary statistics
• Photos, videos, and other supporting media
• Documentation of data use cases and applications

Computer programs: (scripts used for data entry, editing, anonymization, tabulation and analysis)

Preserving Documents

Documents available in electronic format (MS-Word, Excel, and others) must be preserved in their original format and in PDF format.

All documents available only on hard copy/paper must be scanned. Use low resolution graphics, and black & white option (unless it is crucial to preserve colours e.g. where color conveys meaning or interpretation) to avoid large file sizes. A minimum scanning resolution of 300 dpi is recommended. Save the scanned documents in searchable PDF format where possible. Maintain checksums or other file integrity verification mechanisms for digital preservation and periodically verify that files remain accessible and uncorrupted. Scan all resources with an updated virus detection application. Document the procedures used to maintain file integrity and accessibility over time.

For preservation copies, prefer formats with open, published specifications, wide adoption, good metadata support, and a reasonable history of backward compatibility. Retain the original file as well. Maintain more than one copy of critical materials, keep at least one copy separate from the main storage location, and periodically test that files and restoration procedures still work.

Organizing data and resources is a critical part of the documentation and archiving process. However, simply organizing files and documentation does not make them interoperable, machine-readable, or ready for ingestion into a data catalog. To ensure consistency, discoverability, and long-term usability, these resources must also be described using structured metadata that complies with established standards.

The next section introduces the metadata standards commonly used for documenting microdata and related resources, and explains how standards-compliant, machine-readable metadata can be created to support cataloging, discovery, preservation, and dissemination.