5. Review of suggestions

Question: How do curators review and decide on suggestions?

Rationale

Drafts and flags are only as good as the review they get. A suggestion that a curator accepts without reading is a published guess, and a suggestion nobody looks at is wasted. A review workflow therefore has three parts. A pipeline produces suggestions in a fixed form that says what is wrong, how severe it is, and what to change. A board shows a curator the current and the proposed value side by side, and that is where the curator decides. A record keeps the decisions with their reasons, so that the system can learn from them and an auditor can see why each record changed.

The program's Metadata Reviewer provides the first two parts. Its pipeline is a chain of model steps: several detect issues, a critic checks the detections, a categorizer assigns each a category, and a severity scorer rates it from 1 to 5. Its board shows each issue as a word-level diff. The third part, the decisions file, is a CSV that this chapter defines.

Target state

• Suggestions arrive as issues with a description, a category, a severity from 1 to 5, the current value, and a proposed value.
• Curators review in a board with a diff view, and apply accepted changes through the editor as a separate, deliberate step.
• Every decision is recorded with the reason, the reviewer, and the date, and the record feeds chapter 8.

Maturity levels

Level	Steps
Foundational	Review model suggestions in a spreadsheet with current and proposed values; record accept, edit, or reject with a reason.
AI-ready	Run the Metadata Reviewer on the records; review in the board; record decisions in a fixed form; apply through the Metadata Editor.
AI-native	Route issues by severity and category to the right curator, track time per decision, and feed decisions to the evaluation set.

Running example

The five records run through the reviewer produce issues such as a name that does not match its definition (PV_HEADCOUNT_NPL_PCT, severity 4), a unit flagged against the values (HE_U5_MORT_RATE, severity 3, which the curator rejects as a wrong flag), and free-text keywords outside the vocabulary. The curator decides each in the board and records the reason.

Recipes

Recipe

5.1 Run the review pipeline

Level

AI-ready

Skills

Developer; curator

Indicative time

One day to set up; minutes per batch

Result: every record has a list of detected issues with a category, a severity, and a proposed fix, in a form the review board can show.

The program's Metadata Reviewer is a pipeline of several model calls (agents): two detectors look for problems, a critic checks the detections, a categorizer labels each, and a scorer assigns a severity from 1 (trivial) to 5 (critical). The set of agents and their prompts is a manifest file the organization can adapt. The client code below submits one record and waits for its issues:

from ai4data.metadata.reviewer import MetadataReviewerClient

client = MetadataReviewerClient.from_openai(model="gpt-4o-mini", api_key="...")
# or from_ollama(...) for a local model; from_azure and from_anthropic also exist
job = client.submit(record)  # a dict in the World Bank schema
issues = job.wait_sync(timeout=300)

Each issue carries five fields: detected_issue (what is wrong), issue_category, issue_severity, current_metadata (the text as it is), and suggested_metadata (the proposed fix).

1. Install the program's package and submit records with the client.
2. Choose the agents manifest: the default, or one of the lighter ones shipped with the package (single-pass, typo-only), or an adapted one.
3. Choose the model through the client's factory methods. A local model satisfies the confidentiality rule for unpublished records.
4. Write the issues to the board's input workbook with the record identifier, so that decisions can be traced to the record.

Worked example. The five records of the running example are submitted with the default manifest and a hosted model, because all five are public. The pipeline returns eleven issues. For the poverty headcount it returns one issue with the description "The name refers to the national poverty line but the definition describes the international line of 2.15 dollars a day". The issue has the category "Inconsistency / Conflict" and severity 4, and it carries the current definition and a proposed definition that names the national line. For the under-five mortality rate it returns "Measurement unit may not match the values", severity 3, with a proposed unit of "per 100 live births", which the curator later rejects. For three records it returns keywords outside the vocabulary at severity 2. The issues are written to the workbook with their record identifiers.

What this does not do. The pipeline detects and proposes; it does not apply. A proposed fix is as fallible as a draft. The critic agent reduces false flags without removing them, which the rejected unit flag in the example of recipe 3.3 shows.

Recipe

5.2 Review in the board

Level

AI-ready

Skills

Curator

Indicative time

One to three minutes per issue

Result: a curator has seen every issue with the current and the proposed text side by side, and has taken a decision on each.

The review board is a single HTML file that opens in a browser with no server. It shows one row per record with issue counts by severity and category. Selecting a record lists its issues from most to least severe, with the current and suggested values shown as a word-level diff (changed words highlighted).

1. Open the board with the workbook from recipe 5.1.
2. Review from the most severe down. For each issue decide: accept the proposed value, edit it, or reject the flag. The board is a review aid and does not write back, so the decision is recorded in the decisions file (recipe 5.3) and applied in the Metadata Editor.
3. For a category that recurs with the same decision (keywords outside the vocabulary, all accepted), apply the decision as a rule and let the pipeline handle the category with lighter review.

Worked example. The curator opens the board and sees the five records with their issue counts. The poverty headcount is first, with one severity-4 issue. Its diff highlights "international poverty line of 2.15 dollars a day" in the current definition and "national poverty line as set by the household budget survey" in the proposed one. The curator knows the series, confirms that the definition was copied from the wrong record years ago, and accepts the proposal. The under-five mortality rate's unit issue shows "per 1,000 live births" against "per 100 live births"; the curator checks the values, which run from 41 to 48, and rejects the flag. The three keyword issues are all accepted, and after the second batch shows the same pattern the unit makes "keyword outside the vocabulary, replacement in the vocabulary" a category that a junior curator reviews.

What this does not do. The diff shows what changed. Whether the change is right is a separate question, and the curator's knowledge of the series answers it. That is why severity-4 and 5 issues on method or definition fields go to the subject-matter curator.

Recipe

5.3 Record the decisions

Level

Foundational

Skills

Curator

Indicative time

Part of each review

Result: a file links every suggestion to a decision, a final text, a reason, a reviewer, and a date. It is the audit trail of why each record changed, and it is the input to every measurement in this guide.

The decisions file has one row per suggestion:

Column	Meaning
record_id, field	Which record and which field the suggestion concerned
decision	accept, edit, or reject
final_text	The text as published, where the curator edited it
reason	Why, in one sentence; the reasons are later coded into the error taxonomy
reviewer, date	Who decided and when

record_id,field,decision,final_text,reason,reviewer,date
LF_UNEMP_PCT,definition_long,edit,"Share of the labour force aged 15 and over that is without work, available for work, and seeking work during the reference week, as a percentage of the labour force. Source: quarterly labour force survey.",Added the age threshold from the survey documentation; the draft could not know it,curator-a,2026-09-14
LF_UNEMP_PCT,keywords,accept,,,curator-a,2026-09-14
FS_UNDERNOURISH_PCT,keywords,edit,undernourishment; food insecurity,Dropped hunger: outside the vocabulary,curator-a,2026-09-14
PV_HEADCOUNT_NPL_PCT,keywords,accept,,,curator-a,2026-09-14
PV_HEADCOUNT_NPL_PCT,name,accept,,,curator-b,2026-09-15
HE_U5_MORT_RATE,measurement_unit,reject,,"The unit is correct; the values are per 1,000 as stated. Flag was wrong.",curator-b,2026-09-15
PV_HEADCOUNT_NPL_PCT,definition_long,accept,,,curator-b,2026-09-15
FS_UNDERNOURISH_PCT,topics,accept,,,curator-a,2026-09-15
ED_PRIMARY_NER_PCT,topics,edit,Education,Children is not a topic in the list,curator-a,2026-09-15
ED_PRIMARY_NER_PCT,periodicity,accept,,,curator-a,2026-09-15

The suggestions file holds the model's side, one line per suggestion, with the current and suggested text, the category, the severity, and the model's confidence:

{"record_id": "LF_UNEMP_PCT", "field": "definition_long", "source": "draft", "current": "", "suggested": "Share of the labour force that is without work, available for work, and seeking work during the reference week. Expressed as a percentage of the labour force, from the quarterly labour force survey.", "issue_category": "Missing / Redundant Information", "issue_severity": 4, "model_confidence": 0.78}
{"record_id": "LF_UNEMP_PCT", "field": "keywords", "source": "draft", "current": "", "suggested": "unemployment; labour force", "issue_category": "Missing / Redundant Information", "issue_severity": 2, "model_confidence": 0.9}
{"record_id": "FS_UNDERNOURISH_PCT", "field": "keywords", "source": "draft", "current": "food insecurity; hunger", "suggested": "undernourishment; food insecurity; hunger", "issue_category": "Missing / Redundant Information", "issue_severity": 1, "model_confidence": 0.85}
{"record_id": "PV_HEADCOUNT_NPL_PCT", "field": "keywords", "source": "draft", "current": "poverty rate", "suggested": "poverty; poverty rate; poverty headcount", "issue_category": "Missing / Redundant Information", "issue_severity": 1, "model_confidence": 0.88}
{"record_id": "PV_HEADCOUNT_NPL_PCT", "field": "name", "source": "reviewer", "current": "Poverty gap at national poverty line", "suggested": "Poverty headcount ratio at national poverty line", "issue_category": "Inconsistency / Conflict", "issue_severity": 4, "model_confidence": 0.93}
{"record_id": "HE_U5_MORT_RATE", "field": "measurement_unit", "source": "reviewer", "current": "per 1,000 live births", "suggested": "per 100,000 live births", "issue_category": "Incorrect / Invalid Content", "issue_severity": 3, "model_confidence": 0.55}
{"record_id": "PV_HEADCOUNT_NPL_PCT", "field": "definition_long", "source": "reviewer", "current": "Share of the population living below the poverty line.", "suggested": "Share of the population whose consumption per capita is below the national poverty line.", "issue_category": "Ambiguity / Unclear", "issue_severity": 2, "model_confidence": 0.8}
{"record_id": "FS_UNDERNOURISH_PCT", "field": "topics", "source": "reviewer", "current": "Food Security, Health", "suggested": "Food security; Health", "issue_category": "Formatting / Structure", "issue_severity": 1, "model_confidence": 0.95}
{"record_id": "ED_PRIMARY_NER_PCT", "field": "topics", "source": "draft", "current": "", "suggested": "Education; Children", "issue_category": "Missing / Redundant Information", "issue_severity": 2, "model_confidence": 0.7}
{"record_id": "ED_PRIMARY_NER_PCT", "field": "periodicity", "source": "reviewer", "current": "Anual", "suggested": "Annual", "issue_category": "Typo / Language", "issue_severity": 1, "model_confidence": 0.99}

1. Record one row per suggestion as decisions are taken.
2. Code the reasons as they recur (wrong flag, invented fact, style, wrong vocabulary term). The codes are the error taxonomy of chapter 8.
3. Apply accepted and edited values in the Metadata Editor, which validates the record and publishes to NADA.

Worked example. The decisions file after the first review has ten rows. The row for the poverty headcount's definition reads: decision accept, final_text empty because the proposal was taken as written, reason "definition described the wrong poverty line; proposal is correct", the senior curator's initials, and the date. The row for the under-five mortality rate's unit reads: decision reject, reason "the unit is correct; the values are per 1,000 as stated; flag was wrong". When the reasons are coded a month later, the first becomes "wrong content in record" and the second "wrong flag", and both codes appear in the taxonomy of chapter 8. The accepted definition is applied in the Metadata Editor, which validates the record and publishes it.

What this does not do. The file records decisions; it does not make them consistent between curators. A sample reviewed by a second curator measures that, and disagreements refine the typology and the rubric.

Implementation options

• Pipeline: the Metadata Reviewer with its default or an adapted agents manifest; a single-pass prompt for small teams.
• Board: the reviewer's HTML board over an Excel workbook; a spreadsheet at the foundational level.
• Decisions: a CSV under version control, or the editor's own change log where it records reasons.
• Application: the Metadata Editor, which validates and publishes to NADA.

World Bank examples

• The Metadata Reviewer: the agentic approach, the output schema with categories and severity, the client, and the review board.
• The Metadata Editor applies the accepted changes with validation.

Resources

• Metadata Reviewer: the agentic pipeline, the output schema with categories and severity, and the client API.
• Review board: the HTML board over the issues workbook with the word-level diff.
• Agents manifest reference: the manifest format, with the default, single-pass, and typo-only manifests shipped with the package.
• Metadata Editor user and practice guide: applying accepted changes with validation and publishing to NADA.
• Decisions file: the running example's decision record, a template for the organization's own.

Common mistakes

• Applying suggestions in bulk. Every unreviewed fix is a published guess.
• A board without a decisions file. The changes happen; the reasons are lost, and nothing improves.
• Reviewing from the top of the list. Severity orders the work; a critical issue waits while trivial ones are decided.
• One curator for every category. Method and definition issues need the subject-matter curator.

Verification

• Pipeline. Every issue has a category, a severity, a current value, and a proposed value.
• Board. A review is recorded for every issue of the batch, and none has been applied without a decision.
• Decisions file. The file is complete, its reasons are coded, and each decision can be traced to the editor's changes.

Checklist

0 of 6 done

• Review pipeline run with a recorded model and manifest
• Issues in the output schema with category and severity
• Board used with the diff view; severity-ordered review
• Decision per suggestion recorded with reason, reviewer, date
• Reasons coded into an error taxonomy
• Accepted changes applied through the editor with validation

Ticks are saved in this browser only.