Skip to main content

4. Expert Concept Refinement and Assessment

Question: Which concepts should remain, and how does the development evidence support them?

Rationale​

AI-generated concepts can overlap, fall outside the agreed scope, or preserve distinctions that people cannot apply consistently. Reviewing their definitions against actual examples turns a plausible list into meanings useful for the application. A merge, rename, or exclusion needs a reason tied to the materials and intended use.

Checking how the development observations fit the reviewed concepts helps reveal what the list covers and what needs another look. Rare concepts can still matter to a specialized use, and an unmatched profile can reflect either a deliberate exclusion or a missed distinction. Reviewing that evidence before field design avoids treating frequency or model confidence as a decision rule.

Target state​

  • A reviewed concept list records the reasons for keeping, merging, renaming, or excluding candidates.
  • A saved mapping links profiles to reviewed concepts or explicitly marks them as unmatched.
  • Review notes explain rare concepts, uncertain assignments, and any evidence needing further investigation.

Prerequisites​

  • Candidate concepts, profiles, and supporting observations from Chapter 3.
  • A reviewer who understands the materials and their intended uses.
  • A spreadsheet or another way to record decisions and inspect support counts.
  • New model-assisted assignments need the hosted or local setup described in the overview. The prepared reviews require no calls.

Maturity levels​

LevelSteps
FoundationalA reviewer records concept decisions and their reasons.
AI-readySaved versions of the concept list and assignments make support counts traceable.
AI-nativeAI suggests assignments to the fixed list; reviewers inspect uncertain matches and decide changes.
Running example

Recorded decisions. The title concepts continue into human review. Financing and sample-size examples show why rarity and an unmatched assignment need interpretation before field design.

Recipes​

Recipe

4.1 Record concept merges, renames, and exclusions

Level
Foundational
Skills
Domain expert; metadata reviewer
Indicative time
45–60 minutes for the selected decisions

Result: A review worksheet records concept decisions, supporting evidence, and reasons.

Review definitions and examples before deciding to keep, merge, rename, or exclude a concept. A useful decision explains which meaning is preserved and why it serves the intended task. A spreadsheet and the supplied worksheet are sufficient for this review.

Input example. The human-concept-decisions example in evidence.json retains these definitions for three concepts considered for merging. Compare what each meaning would preserve before reading the recorded dispositions.

Concept under reviewRecorded definition
measure_nameName of the quantitative measure or value column reported in the snapshot.
axis_variableVariable, indicator, dimension, or measure represented along a chart axis.
outcome_variableDependent variable, outcome, result, or main effect being estimated, measured, or explained.
  1. Compare related definitions with their observed values. Identify what a merge would preserve or lose.
  2. Decide whether each distinction helps the intended use and can be recognized consistently. Keep uncertain cases open for further review.
  3. Save a working copy of the concept-review template. Record the evidence and proposed action separately from the final decision.
  4. Record the reviewer, reason, and retained target for a merge or rename. In the prepared exercise, label new explanations as teaching interpretations.

Worked example. The following representative decisions are transcribed from the paper's Table 11.

CandidatesRecorded dispositionRecorded case-specific rationale
measure_name, axis_variable, outcome_variableMerge into indicator_name.The concepts represent equivalent analytical variables expressed through different visualization or disciplinary terminology.
presentation_typeRename to visualization_type.The name improves conceptual clarity and applicability across visualization types.
data_values, legend_semanticsExclude from the reviewed inventory.In this review, they described extracted contents outside the intended reusable metadata concepts.

The indicator merge retains the distinction between what is measured and the numeric observations themselves. A cost, rate, or estimated outcome can share the retained measured-concept meaning without making its individual values metadata. Excluding legend_semantics also does not discard every legend label: a legend can supply a value for a retained category or indicator concept.

Completed teaching record. This worksheet transcription shows the recorded measure_name → indicator_name merge. Explanations are labelled as teaching restatements; an unrecorded reviewer or date has not been invented.

Worksheet columnTeaching transcription
concept_namemeasure_name
evidence_referencehuman-concept-decisions; paper Table 11
proposed_actionmerge
target_conceptindicator_name
semantic_rationaleTeaching restatement: the recorded definitions describe analytical variables or quantities under different labels.
scope_rationaleTeaching restatement: retain the measured-concept description within this study's metadata scope.
reviewerNot supplied in the retained decision.
decisionRecorded merge into indicator_name.
decision_reasonTable 11 reports equivalent analytical variables expressed through different visualization or disciplinary terminology.

Verification. A second reviewer can trace each demonstrated action to the prepared evidence and explain its meaning or scope rationale. Merge targets are retained concepts. The worksheet separates a current reviewer's proposed action from the reported historical outcome. Do not infer the unrecorded dispositions needed to reconcile every candidate from the row count alone.

What this does not do. The study retained 41 concepts and eight merge rows. Those 49 rows are not a complete ledger of all 70 candidate decisions, and the reviewed concepts do not yet determine the operational fields.

Recipe

4.2 Review profile assignments against the concepts

Level
AI-native
Skills
LLM practitioner; metadata reviewer
Indicative time
30–45 minutes for the retained assignment review

Result: Review notes check profile assignments against the concept definitions and flag uncertain fits. A new mapping uses the complete fixed concept list.

Map profiles to a fixed concept list to see how well the reviewed meanings cover the development evidence. Allow an unmatched category so a poor fit is not forced into a concept. The study called that category not_in_ontology; this label does not make the list a formal ontology.

Input example. Compare the title profiles from Recipe 3.2 with the reviewed title concept: “The displayed or inferred title, heading, or caption that identifies the snapshot.” The study retained that definition. Its original sample_size and panel_titles profiles provide contrasting unmatched cases. The prepared exercise reviews existing assignments; creating a new mapping requires the complete reviewed concept list.

The downloadable assessment request is an independently written, unexecuted teaching adaptation of the historical classification task. It describes logical inputs and checks, without supplying a live API runner.

Concept assessment request specification
# Concept assessment request: teaching adaptation

This independently written, unexecuted specification illustrates classification
after human concept refinement. It is not the complete historical prompt or a
ready-to-send provider payload. No live model call is needed for the exercises.

## Reference and input

Supply the fixed reviewed concept inventory. Each entry contains its canonical
name, definition, and illustrative discovered field names. Include the examples
from concepts merged into a retained concept without making the merged names
additional classification targets.

Supply one exact-name field profile: metadata_field, top_description_values,
and top_observed_values. Keep its identity and the inventory version beside the
request. The historical classifier received these semantic profile attributes;
frequency analysis was a separate use of the resulting assignments.

## Classification instruction

Choose exactly one allowed canonical name. Compare the profile's intended
meaning with the definitions and examples. A similar name is insufficient.
Examples illustrate meanings and do not exhaust the permitted vocabulary of
discovered field names. Do not create concepts, propose merges, or modify the
reference inventory during classification.

Use the historical residual label not_in_ontology when no retained concept
adequately describes the profile. Intentionally excluded analytical values and
extraction-oriented contents may belong here. This label records a residual
classification; it does not assert that the reference is a formal ontology.

Return canonical_name and confidence. Constrain canonical_name to the reviewed
names plus not_in_ontology. Confidence is high, medium, or low. High means a clear
fit; medium means several concepts appear plausible; low means no fit is clear,
including a residual assignment. Confidence remains a model judgment.

## Inspection

Check the returned name against the allowed set, then inspect the semantic fit.
Retain the field identity, response, model and prompt settings, and execution
status. Flag ambiguous or surprising assignments for human review. Record a
human correction separately from the original model result.

## Historical boundary

The study used GPT-5.4 mini with medium reasoning through the Responses API.
The classification target was 41 retained concepts plus not_in_ontology.
A Pydantic enumeration constrained the output names; confidence also had a
fixed three-value vocabulary. Maximum output tokens and temperature used the
default API settings. These are historical settings, not a current model choice.

The original assessment mapping and later operational revisions are different
artifacts. A fresh replication must preserve that distinction and save new
outputs separately. Different models and prompts may return different labels.

Source: Dy & Solatorio (2026). Artifact-Driven Metadata Schema Development:
A Human-in-the-Loop Methodology with Multimodal LLMs. Working paper. Appendix D.
  1. Identify the concept-list version used for the assignments. In the prepared review, use the title definition above and the original 41-concept stage. A new mapping requires the complete fixed list and can use the assessment request.
  2. Compare each profile's name, descriptions, and observed values with its assigned meaning. Merged names should point to their retained concepts.
  3. Check that labels come from the fixed list or the unmatched category. Keep model confidence separate from a review decision; the study used not_in_ontology and confidence high, medium, or low.
  4. Check that the profiles under review have results. Investigate uncertain or surprising fits and save any human correction alongside the original assignment.

Worked example. The original assignments are retained under original_profile_assignments in canonical-title and rare-and-residual, and original_profile_assignment in operational-panel-title. Confidence high records the model's judgment of fit.

Exact-name profileOriginal concept assignmentConfidence
figure_titletitlehigh
table_titletitlehigh
chart_titletitlehigh
sample_sizenot_in_ontologyhigh
panel_titlesnot_in_ontologyhigh

The title rows continue chapter 3's lexical comparison. The chart_title profile includes internal headings, so even a valid high-confidence title assignment deserves inspection of its scope. Conversely, the residual label for sample_size records an intentional boundary; it is not a request to add a field automatically.

Study mapping versions and execution limits

Recorded configuration and chronology. The classifier used GPT-5.4 mini with medium reasoning, a Pydantic enumeration of the 41 names plus the residual, and default API output-token and temperature settings. A later mapping changed five assignments, including panel_titles to panel_title. The other changed profiles were unit_of_analysis, event_context, operational_context, and contextual_event. Those later edits are excluded from this chapter's original coverage statistics. They do not establish that panel_title was among the 41.

Execution-record limit. The original mapping has 833 labelled rows. The retained execution log contains 832 completed results and one preprocessing failure for missing_value_marker. The mapping assigns that profile to interpretive_note, but the inspected log supplies no successful parsed result for it. Its resolution is undocumented. The mapping is therefore an analysis artifact whose every row cannot be claimed as a retained successful model call.

Verification. Reconcile the five selected rows above against the original assignment entries in the prepared evidence. An allowed label confirms vocabulary compliance only. Inspect the profile's meaning, confidence, execution status, and any review record independently. Keep mapping versions separate; the later panel_title assignment belongs to chapter 5.

What this does not do. No new classification is executed. Optional live replication requires a fixed vocabulary, provider integration, a budget, and a separate output location. Model confidence is not calibrated semantic accuracy, and assignments do not grant authority to modify the inventory or schema.

Recipe

4.3 Review rare concepts and unmatched profiles

Level
AI-ready
Skills
Python or spreadsheet practitioner; domain reviewer
Indicative time
30–45 minutes for the support review

Result: Review notes explain rare concepts and unmatched profiles before recommending any change.

Review uncommon concepts and unmatched profiles in context. A rare financing concept may matter to project tables; an unmatched observation may reflect an intentional exclusion or a missed distinction. Neither result determines a schema change on its own.

Input example. The prepared rare-and-residual example reports seven supporting snapshots for financing_instrument and nine for data_collection_method. The following expanded counts were checked by joining the full observations to the original assessment mapping.

Assigned conceptObservationsExact-name profilesDistinct snapshotsSupport out of 210
financing_instrument7773.3%
data_collection_method11694.3%
  1. Link the observations to the saved profile assignments using exact field names. For the prepared review, use the supplied summary and rare-and-residual example.
  2. Count distinct supporting items separately from observations and profiles. Divide by the full development sample when reporting collection support; do not double-count items shared by several profiles.
  3. Inspect values and source contexts. Ask whether rarity reflects a specialized need, redundant meaning, or an unsuitable boundary, and examine unmatched profiles in the same way.
  4. Record the interpretation and proposed action separately. Change the reviewed concept list only after a human records the supporting reason.

These full-study summaries cannot be recomputed from the selected excerpts. For data_collection_method, treating 11 observations as 11 items would overstate support: only nine distinct snapshots contributed them.

Worked example. The Pakistan financing table's recorded value “IDA Credit; IDA Grant; Guarantee Amount” explains why financing modality is meaningful. All seven supporting snapshots occur in the Refugee Project Appraisal Document corpus. That concentration supports a specialized interpretation; it does not establish universal usefulness or a minimum inclusion threshold.

The Lebanon health-access figure separately proposed sample_size with value “n=44.” Its exact-name profile has 15 observations from 15 snapshots, and the original classification is residual. The figure's recorded title also includes “(n=44).” Preserving an explicit title or explanatory statement in a later operational record is a different question from accepting a dedicated sample-size concept. Chapter 5 demonstrates that distinction without rewriting the coverage assignment.

Verification. Calculate 7 / 210 and 9 / 210 and round to one decimal percentage point. Explain why six method profiles can contribute 11 observations but only nine distinct snapshots. A defensible interpretation identifies the source context and scope rationale before recommending retention or revision.

What this does not do. The summaries characterize the selected development evidence. They do not measure retrieval performance, establish a missing-field rate, prove generalizability, or replace held-out validation. The study found no compelling basis for further conceptual revision before operational design.

Implementation options​

Manual comparison is suitable for a small candidate list. The existing review worksheet makes decisions and their reasons easier to revisit without requiring model calls. For larger profile collections, model-assisted classification can supply draft assignments to the fixed concept list; it still needs review of uncertain and unmatched cases. Choose the review depth according to the importance of the distinctions. Confidence and frequency alone do not establish an acceptance rule.

Resources​

  • The prepared evidence contains human-concept-decisions, canonical-title, rare-and-residual, and the original panel assignment.
  • The existing concept-review template provides separate proposal, rationale, reviewer, and disposition columns.
  • The assessment request explains the fixed-vocabulary task and historical configuration.
  • Dy & Solatorio (2026). Artifact-Driven Metadata Schema Development: A Human-in-the-Loop Methodology with Multimodal LLMs. Working paper. Sections 6.3–6.4, Tables 11–13, and Appendix D supply the recorded decisions and assessment method.

Common mistakes​

  • A partial decision record can be mistaken for a complete review history.
  • A valid concept label or high confidence can be mistaken for a final human judgment.
  • Later assignment revisions can silently replace the evidence used in an earlier assessment.
  • Observation totals or summed profile counts can be mistaken for distinct concept-supporting snapshots.
  • Low frequency or residual membership can be used as an automatic field-design rule.

Verification​

  • Concept decisions name their evidence, reason, reviewer, and retained target where applicable.
  • Assignments use the saved concept list or an unmatched label, with corrections recorded separately.
  • Support counts use distinct items and a stated denominator; rare and unmatched cases have been reviewed in context.
  • Prepared study reviews keep the original mapping separate from later edits and disclose its execution-record limit.

Checklist​

0 of 4 done

Ticks are saved in this browser only.

Use the reviewed concepts and decision record in Chapter 5 to design usable fields.