Skip to main content

1. Scope of model assistance

Question: Which curation tasks can a model assist, and which stay with people?

Rationale​

Curation tasks differ in what a wrong output costs. A model that drafts a keyword list costs a curator a minute to correct. A model that rewrites a definition can change the meaning of a series for every future user, and nobody may notice for years. The first decision is therefore which tasks a model may take, in which role, and which decisions stay with a person. The roles are few: a model can draft text, flag a possible problem, align text with a vocabulary, translate, or summarize, and in every case someone decides what happens next.

The second decision is what may be sent to a model at all. A model run by a provider receives whatever text is sent to it. A record of a public series is public, and sending it is no concern. An unpublished record or a confidential microdata dictionary is a different matter, and the rule has to be written before anyone is tempted. This chapter writes both decisions down and chooses the records on which to try the workflow.

Target state​

  • A typology lists the curation tasks, the role a model takes in each, and the decision that remains with a curator.
  • A rule states that nothing reaches the published catalog without a named curator's approval, and that confidential or unpublished records are never sent to external services.
  • The records to start with are chosen from use and from completeness, and the team knows what success looks like for them.

Maturity levels​

LevelSteps
FoundationalWrite the typology and the two rules. Choose twenty records to start with.
AI-readyAdd decision rights per field (which fields a model may draft, which it may only flag). Record the model, provider, and data sensitivity in the AI component register.
AI-nativeReview the scope each period from acceptance rates and incidents, and extend or restrict the tasks accordingly.
Running example

The example organization allows a model to draft definitions and keywords, to flag inconsistencies in any field, and to propose vocabulary mappings, for its five public indicator records. Units and methodology text may be flagged but never drafted. Every suggestion goes to one curator, who records a decision.

Recipes​

Recipe

1.1 Write the task typology with decision rights

Level
Foundational
Skills
Head of the curation unit
Indicative time
Half a day

Result: the curation unit has a one-page table that says, for each kind of curation task, what a language model is allowed to do and what a curator decides.

Curation work falls into a few families of task: filling a missing field, proposing keywords, flagging a field that disagrees with another, translating, summarizing. For each family, the organization decides the model's role. A model that drafts writes text for a curator to accept or change. A model that flags points at a possible problem and proposes a fix. In both cases the curator decides. The table below is a starting typology:

TaskModel roleCurator decides
Missing description or definitionDraft from the record and the schemaAccept, edit, or reject the draft
Keywords and topicsDraft; map free text to the vocabularyConfirm the mapping
Name against definition, unit against valuesFlag with a severity and a proposed fixWhether the flag is right and what the fix is
Methodology and limitations textFlag onlyAny change
Translations of labelsDraft, marked machine-translatedNative-speaker review before publication
Summaries for catalog pagesDraftAccept or edit
  1. Start from the table and adapt it to the organization's fields and risks. The program's metadata-quality work names the same task families: field generation, semantic alignment, vocabulary standardization, translation, summarization, and diagnostics.
  2. Mark which fields a model may never draft (units, methods, legal text) and which it may only flag.
  3. Name the curator role that decides, and where decisions are recorded (chapter 5, recipe 5.3).

Worked example. The example organization's typology follows the table above with two changes. Its curators decided that a model may draft keywords and definitions for the five indicator records, may flag any field, and may propose vocabulary mappings. The unit and the methodology text are listed as "flag only", because the unit of the under-five mortality rate was once changed by a well-meaning editor and took a year to notice. Translations are allowed as drafts into French, with the native-speaking statistician named as the reviewer. The senior metadata curator is named as the person who decides on every suggestion, and the decisions file of chapter 5 is named as the record.

What this does not do. The typology sets roles. It does not make drafts good; chapters 3 and 4 measure them.

Recipe

1.2 Set the confidentiality and approval rules

Level
Foundational
Skills
Head of the curation unit; legal
Indicative time
One day

Result: two written rules exist that every staff member and every pipeline follows. The first says which records may be sent to which model. The second says what has to happen before a model's output reaches the published catalog.

A model run by a provider receives whatever text is sent to it. For a public catalog record that is no concern; for an unpublished record or a microdata dictionary it may be. A model the organization runs on its own server receives nothing that leaves the building.

  1. Confidentiality: only public records, or records cleared for publication, are sent to a provider's model. Unpublished records and microdata dictionaries go only to models the organization runs itself. Record the rule in the AI-use policy of the dissemination cookbook.
  2. Approval: a model's output reaches the published catalog only after a named curator records a decision. Until then it is marked as model output (a draft status, or a note field).
  3. Register the pipeline (model, provider, data used, owner, review date) in the AI component register. The program's tools accept any provider, including a local model, so the rule can be met either way.

Worked example. The example organization's confidentiality rule reads: "Records of published series and published surveys may be sent to the hosted model named in the register. Records that are not yet published, and every microdata dictionary, are processed only by the open-weight model on the organization's own server." Its approval rule reads: "A model's suggestion is stored with the status draft and the model's name. It becomes part of the published record only when the senior metadata curator records accept or edit in the decisions file." The register entry for the pipeline names the hosted model and the local model. It names the public catalog records as the data and the head of the curation unit as the owner, and it sets a review date six months out.

What this does not do. Rules state the practice. The review board and the decision log of chapter 5 are where the practice is visible and auditable.

Recipe

1.3 Choose the records to start with

Level
Foundational
Skills
Curator; dissemination officer
Indicative time
Two hours

Result: the unit has a first set of about twenty records on which model assistance is worth the review effort, and a written definition of what success on them looks like.

  1. Take the records with the highest use (the monitoring cookbook counts it) and the lowest completeness (the completeness check at the AI-ready level lists the missing fields).
  2. Start with twenty: enough to measure acceptance, few enough to review every suggestion.
  3. Define success in three numbers: the share of drafts accepted or edited, the share of flags the curator confirms, and the completeness gain after the decisions are applied.
  4. Record the baseline (completeness, curator time per record) before the first run, so that the gain can be measured.

Worked example. The example organization has only five indicator records in the running example, so its start set is those five plus fifteen survey records chosen from the monitoring report's list of most used datasets. The completeness check at the AI-ready level shows that the five indicators lack between two and eight fields each and that the unemployment rate has no definition at all. The baseline reads: completeness 0 of 20 records at the AI-ready level, and 25 minutes of curator time per record when a definition is written from scratch. The definition of success reads: at least 70 percent of drafts accepted or edited, at least 80 percent of flags confirmed, and at least 15 of the 20 records complete at the AI-ready level after the decisions are applied.

What this does not do. Twenty records calibrate the workflow. The catalog-scale run comes after the acceptance rates of chapter 8 are known.

Implementation options​

  • Typology: the task families of the program's metadata-quality work; roles draft, flag, align, translate, summarize.
  • Rules: the AI-use policy template and the AI component register of the dissemination cookbook.
  • Models: hosted or local through a provider-agnostic client (litellm, as the program's tools use); local models for anything unpublished.
  • Record selection: use counts from monitoring and completeness from the profile check.

World Bank examples​

  • Generative AI for Metadata Quality lists the task families and the risks (plausible but incorrect text, inconsistency, bias) that the rules answer.
  • The Metadata Reviewer is designed to be LLM-agnostic, so that the model choice follows the confidentiality rule.

Resources​

Common mistakes​

  • Scope by tool. The question is which tasks, in which role; the tool follows.
  • Pasting unpublished records into a public assistant. The first rule exists for this.
  • Drafting the method text. A model that rewrites the methodology changes the series' meaning; flag only.
  • Starting with the whole catalog. Review capacity decides the scale; twenty records first.

Verification​

  • Typology. Every curation task in the team's procedures has a row with a role and a decision.
  • Rules. Staff can state both rules, and the register lists the pipeline with an owner.
  • Start set. Twenty records have been chosen with their use and completeness recorded, and a baseline exists.

Checklist​

0 of 6 done

Ticks are saved in this browser only.