4. Expert Concept Refinement and Assessment
Question: Which concepts should remain, and how does the development evidence support them?
Rationale
AI-generated concepts can overlap, fall outside the agreed scope, or preserve distinctions that people cannot apply consistently. Reviewing their definitions against actual examples turns a plausible list into meanings useful for the application. A merge, rename, or exclusion needs a reason tied to the materials and intended use.
Checking how the development observations fit the reviewed concepts helps reveal what the list covers and what needs another look. Rare concepts can still matter to a specialized use, and an unmatched profile can reflect either a deliberate exclusion or a missed distinction. Reviewing that evidence before field design avoids treating frequency or model confidence as a decision rule.
Target state
- A reviewed concept list records the reasons for keeping, merging, renaming, or excluding candidates.
- A saved mapping links profiles to reviewed concepts or explicitly marks them as unmatched.
- Review notes explain rare concepts, uncertain assignments, and any evidence needing further investigation.
Prerequisites
- Candidate concepts, profiles, and supporting observations from Chapter 3.
- A reviewer who understands the materials and their intended uses.
- A spreadsheet or another way to record decisions and inspect support counts.
- New model-assisted assignments need the hosted or local setup described in the overview. The prepared reviews require no calls.
Maturity levels
| Level | Steps |
|---|---|
| Foundational | A reviewer records concept decisions and their reasons. |
| AI-ready | Saved versions of the concept list and assignments make support counts traceable. |
| AI-native | AI suggests assignments to the fixed list; reviewers inspect uncertain matches and decide changes. |
Recorded decisions. The title concepts continue into human review. Financing and sample-size examples show why rarity and an unmatched assignment need interpretation before field design.
Recipes
4.1 Record concept merges, renames, and exclusions
- Level
- Foundational
- Skills
- Domain expert; metadata reviewer
- Indicative time
- 45–60 minutes for the selected decisions
Result: A review worksheet records concept decisions, supporting evidence, and reasons.
Review definitions and examples before deciding to keep, merge, rename, or exclude a concept. A useful decision explains which meaning is preserved and why it serves the intended task. A spreadsheet and the supplied worksheet are sufficient for this review.
Input example. The human-concept-decisions example in evidence.json retains these definitions for three concepts considered for merging. Compare what each meaning would preserve before reading the recorded dispositions.
| Concept under review | Recorded definition |
|---|---|
| measure_name | Name of the quantitative measure or value column reported in the snapshot. |
| axis_variable | Variable, indicator, dimension, or measure represented along a chart axis. |
| outcome_variable | Dependent variable, outcome, result, or main effect being estimated, measured, or explained. |
- Compare related definitions with their observed values. Identify what a merge would preserve or lose.
- Decide whether each distinction helps the intended use and can be recognized consistently. Keep uncertain cases open for further review.
- Save a working copy of the concept-review template. Record the evidence and proposed action separately from the final decision.
- Record the reviewer, reason, and retained target for a merge or rename. In the prepared exercise, label new explanations as teaching interpretations.
Worked example. The following representative decisions are transcribed from the paper's Table 11.
| Candidates | Recorded disposition | Recorded case-specific rationale |
|---|---|---|
| measure_name, axis_variable, outcome_variable | Merge into indicator_name. | The concepts represent equivalent analytical variables expressed through different visualization or disciplinary terminology. |
| presentation_type | Rename to visualization_type. | The name improves conceptual clarity and applicability across visualization types. |
| data_values, legend_semantics | Exclude from the reviewed inventory. | In this review, they described extracted contents outside the intended reusable metadata concepts. |
The indicator merge retains the distinction between what is measured and the numeric observations themselves. A cost, rate, or estimated outcome can share the retained measured-concept meaning without making its individual values metadata. Excluding legend_semantics also does not discard every legend label: a legend can supply a value for a retained category or indicator concept.
Completed teaching record. This worksheet transcription shows the recorded measure_name → indicator_name merge. Explanations are labelled as teaching restatements; an unrecorded reviewer or date has not been invented.
| Worksheet column | Teaching transcription |
|---|---|
| concept_name | measure_name |
| evidence_reference | human-concept-decisions; paper Table 11 |
| proposed_action | merge |
| target_concept | indicator_name |
| semantic_rationale | Teaching restatement: the recorded definitions describe analytical variables or quantities under different labels. |
| scope_rationale | Teaching restatement: retain the measured-concept description within this study's metadata scope. |
| reviewer | Not supplied in the retained decision. |
| decision | Recorded merge into indicator_name. |
| decision_reason | Table 11 reports equivalent analytical variables expressed through different visualization or disciplinary terminology. |
Verification. A second reviewer can trace each demonstrated action to the prepared evidence and explain its meaning or scope rationale. Merge targets are retained concepts. The worksheet separates a current reviewer's proposed action from the reported historical outcome. Do not infer the unrecorded dispositions needed to reconcile every candidate from the row count alone.
What this does not do. The study retained 41 concepts and eight merge rows. Those 49 rows are not a complete ledger of all 70 candidate decisions, and the reviewed concepts do not yet determine the operational fields.
4.2 Review profile assignments against the concepts
- Level
- AI-native
- Skills
- LLM practitioner; metadata reviewer
- Indicative time
- 30–45 minutes for the retained assignment review
Result: Review notes check profile assignments against the concept definitions and flag uncertain fits. A new mapping uses the complete fixed concept list.
Map profiles to a fixed concept list to see how well the reviewed meanings cover the development evidence. Allow an unmatched category so a poor fit is not forced into a concept. The study called that category not_in_ontology; this label does not make the list a formal ontology.
Input example. Compare the title profiles from Recipe 3.2 with the reviewed title concept: “The displayed or inferred title, heading, or caption that identifies the snapshot.” The study retained that definition. Its original sample_size and panel_titles profiles provide contrasting unmatched cases. The prepared exercise reviews existing assignments; creating a new mapping requires the complete reviewed concept list.
The downloadable assessment request
is an independently written, unexecuted teaching adaptation of the historical
classification task. It describes logical inputs and checks, without supplying
a live API runner.Concept assessment request specification
- Identify the concept-list version used for the assignments. In the prepared review, use the title definition above and the original 41-concept stage. A new mapping requires the complete fixed list and can use the assessment request.
- Compare each profile's name, descriptions, and observed values with its assigned meaning. Merged names should point to their retained concepts.
- Check that labels come from the fixed list or the unmatched category. Keep model confidence separate from a review decision; the study used not_in_ontology and confidence high, medium, or low.
- Check that the profiles under review have results. Investigate uncertain or surprising fits and save any human correction alongside the original assignment.
Worked example. The original assignments are retained under original_profile_assignments in canonical-title and rare-and-residual, and original_profile_assignment in operational-panel-title. Confidence high records the model's judgment of fit.
| Exact-name profile | Original concept assignment | Confidence |
|---|---|---|
| figure_title | title | high |
| table_title | title | high |
| chart_title | title | high |
| sample_size | not_in_ontology | high |
| panel_titles | not_in_ontology | high |
The title rows continue chapter 3's lexical comparison. The chart_title
profile includes internal headings, so even a valid high-confidence title
assignment deserves inspection of its scope. Conversely, the residual label
for sample_size records an intentional boundary; it is not a request to add a
field automatically. Recorded configuration and chronology. The classifier used GPT-5.4 mini
with medium reasoning, a Pydantic enumeration of the 41 names plus the residual,
and default API output-token and temperature settings. A later mapping changed
five assignments, including panel_titles to panel_title. The other changed
profiles were unit_of_analysis, event_context, operational_context, and
contextual_event. Those later edits are excluded from this chapter's original
coverage statistics. They do not establish that panel_title was among the 41. Execution-record limit. The original mapping has 833 labelled rows. The
retained execution log contains 832 completed results and one preprocessing
failure for missing_value_marker. The mapping assigns that profile to
interpretive_note, but the inspected log supplies no successful parsed result
for it. Its resolution is undocumented. The mapping is therefore an analysis
artifact whose every row cannot be claimed as a retained successful model call.Study mapping versions and execution limits
Verification. Reconcile the five selected rows above against the original assignment entries in the prepared evidence. An allowed label confirms vocabulary compliance only. Inspect the profile's meaning, confidence, execution status, and any review record independently. Keep mapping versions separate; the later panel_title assignment belongs to chapter 5.
What this does not do. No new classification is executed. Optional live replication requires a fixed vocabulary, provider integration, a budget, and a separate output location. Model confidence is not calibrated semantic accuracy, and assignments do not grant authority to modify the inventory or schema.
4.3 Review rare concepts and unmatched profiles
- Level
- AI-ready
- Skills
- Python or spreadsheet practitioner; domain reviewer
- Indicative time
- 30–45 minutes for the support review
Result: Review notes explain rare concepts and unmatched profiles before recommending any change.
Review uncommon concepts and unmatched profiles in context. A rare financing concept may matter to project tables; an unmatched observation may reflect an intentional exclusion or a missed distinction. Neither result determines a schema change on its own.
Input example. The prepared rare-and-residual example reports seven supporting snapshots for financing_instrument and nine for data_collection_method. The following expanded counts were checked by joining the full observations to the original assessment mapping.
| Assigned concept | Observations | Exact-name profiles | Distinct snapshots | Support out of 210 |
|---|---|---|---|---|
| financing_instrument | 7 | 7 | 7 | 3.3% |
| data_collection_method | 11 | 6 | 9 | 4.3% |
- Link the observations to the saved profile assignments using exact field names. For the prepared review, use the supplied summary and rare-and-residual example.
- Count distinct supporting items separately from observations and profiles. Divide by the full development sample when reporting collection support; do not double-count items shared by several profiles.
- Inspect values and source contexts. Ask whether rarity reflects a specialized need, redundant meaning, or an unsuitable boundary, and examine unmatched profiles in the same way.
- Record the interpretation and proposed action separately. Change the reviewed concept list only after a human records the supporting reason.
These full-study summaries cannot be recomputed from the selected excerpts. For data_collection_method, treating 11 observations as 11 items would overstate support: only nine distinct snapshots contributed them.
Worked example. The Pakistan financing table's recorded value “IDA Credit; IDA Grant; Guarantee Amount” explains why financing modality is meaningful. All seven supporting snapshots occur in the Refugee Project Appraisal Document corpus. That concentration supports a specialized interpretation; it does not establish universal usefulness or a minimum inclusion threshold.
The Lebanon health-access figure separately proposed sample_size with value “n=44.” Its exact-name profile has 15 observations from 15 snapshots, and the original classification is residual. The figure's recorded title also includes “(n=44).” Preserving an explicit title or explanatory statement in a later operational record is a different question from accepting a dedicated sample-size concept. Chapter 5 demonstrates that distinction without rewriting the coverage assignment.
Verification. Calculate 7 / 210 and 9 / 210 and round to one decimal percentage point. Explain why six method profiles can contribute 11 observations but only nine distinct snapshots. A defensible interpretation identifies the source context and scope rationale before recommending retention or revision.
What this does not do. The summaries characterize the selected development evidence. They do not measure retrieval performance, establish a missing-field rate, prove generalizability, or replace held-out validation. The study found no compelling basis for further conceptual revision before operational design.
Implementation options
Manual comparison is suitable for a small candidate list. The existing review worksheet makes decisions and their reasons easier to revisit without requiring model calls. For larger profile collections, model-assisted classification can supply draft assignments to the fixed concept list; it still needs review of uncertain and unmatched cases. Choose the review depth according to the importance of the distinctions. Confidence and frequency alone do not establish an acceptance rule.
Resources
- The prepared evidence contains human-concept-decisions, canonical-title, rare-and-residual, and the original panel assignment.
- The existing concept-review template provides separate proposal, rationale, reviewer, and disposition columns.
- The assessment request explains the fixed-vocabulary task and historical configuration.
- Dy & Solatorio (2026). Artifact-Driven Metadata Schema Development: A Human-in-the-Loop Methodology with Multimodal LLMs. Working paper. Sections 6.3–6.4, Tables 11–13, and Appendix D supply the recorded decisions and assessment method.
Common mistakes
- A partial decision record can be mistaken for a complete review history.
- A valid concept label or high confidence can be mistaken for a final human judgment.
- Later assignment revisions can silently replace the evidence used in an earlier assessment.
- Observation totals or summed profile counts can be mistaken for distinct concept-supporting snapshots.
- Low frequency or residual membership can be used as an automatic field-design rule.
Verification
- Concept decisions name their evidence, reason, reviewer, and retained target where applicable.
- Assignments use the saved concept list or an unmatched label, with corrections recorded separately.
- Support counts use distinct items and a stated denominator; rare and unmatched cases have been reviewed in context.
- Prepared study reviews keep the original mapping separate from later edits and disclose its execution-record limit.
Checklist
Ticks are saved in this browser only.
Use the reviewed concepts and decision record in Chapter 5 to design usable fields.