Skip to main content

2. Metadata Discovery

Question: How can descriptive requirements emerge independently from artifacts?

Rationale​

Starting with a familiar list of fields can hide information that matters to the collection. AI-assisted discovery helps explore possible fields by examining examples within the scope defined in Chapter 1. Each metadata observation records a suggested property, a value, and the evidence behind it, so reviewers can judge whether the suggestion is useful.

Inspecting each item independently gives different examples room to reveal different needs. Feeding earlier suggestions into later requests can steer the model toward an emerging field list too soon. Keep the instructions consistent, identify any supporting context, and save the responses for comparison. Checking the response format catches processing errors; reviewing the source evidence catches misreadings and unsupported suggestions.

Target state​

  • A discovery request states the scope, inputs, allowed context, and information to retain for each suggestion.
  • Recorded observations link each suggested property and value to an identified development item and its evidence.
  • A processing record identifies successful responses, failures, and retries before observations are compared across the sample.

Prerequisites​

  • The scope statement and development sample from Chapter 1.
  • Familiarity with API requests and JSON records for preparing a new run.
  • Python 3.10 or newer for the response-inspection example; no external packages are needed.
  • New LLM runs need the hosted or local setup described in the overview. The prepared inspection needs no LLM credentials.

Maturity levels​

LevelSteps
FoundationalA reviewer reads suggested properties alongside their source evidence.
AI-readyStructured records retain item identifiers, values, evidence sources, and processing status.
AI-nativeIndependent AI requests suggest properties, with recorded settings and checks before human review.
Running example

Recorded excerpts. The Jordan chart, Afghanistan table, and Russian chart show how different items produce different title suggestions. The prepared responses come from the study; the response-inspection example checks those excerpts.

Recipes​

Start with one discovery request, then read and check its result. To follow the worked example without a model call, read the request in Recipe 2.1 and continue with the recorded responses in Recipe 2.2.

Recipe

2.1 Run independent metadata discovery

Level
AI-native
Skills
LLM practitioner; domain reviewer
Indicative time
45–60 minutes for setup; execution time varies

Result: A recorded model response proposes descriptive properties and values for one development item.

Choose an input the model can inspect without losing relevant evidence: text, images, structured records, audio, or a combination. Record permitted context separately. The study used images and parent-document metadata; other input types need their own integration and checks.

Fix the response format while leaving the proposed field names open. Record the property, value or uncertainty, description, evidence source, and reason it could be useful. The teaching request uses six string attributes; those attributes organize observations and do not prescribe the final schema fields.

Input example. A Jordan request pairs the chart image with identified parent-document context. The source register provides the title “Jordan - Emergency Food Security Project” and its metadata link. This teaching request is not a reconstruction of the complete historical messages.

The discovery request is a teaching adaptation that can be used as a starting point for a new run. It has not been executed to produce the recorded examples. The complete text is available below. Its user section identifies logical input contents; it is not a ready-to-send provider-specific API payload.

Discovery request specification
# Independent discovery request: teaching adaptation

This independently written prompt illustrates the methodology. It is not the
complete historical prompt and has not been executed to produce new results.
The study supplied one snapshot image and available associated document
metadata per request. The examples below describe logical request contents;
they are not a provider-specific API payload.

For another artifact class, first select a representation and model that can
inspect its relevant evidence, and define allowable context and evidence-origin
labels. The image inputs and snapshot/document enums below remain specific to
this teaching adaptation; changing them requires a separately documented request.

## System instruction

Act as a metadata designer. From one analytical artifact and its supplied
document context, propose reusable descriptive properties that support
identification, interpretation, filtering, cataloging, and discovery.
Do not populate a predefined operational metadata schema. Distinguish reusable
field names from observed values and from numerical data contained in the
artifact. Prefer generic meanings. Keep potentially useful distinctions for
later review; no field proposed here is automatically accepted.

Ground every observed value in the image or supplied context. Identify the
source level as snapshot, document, or both. Preserve uncertainty; when no
value is identifiable, state that explicitly. Label inferred values as inferred.
Do not infer a specific value from vague contextual evidence. Do not add
administrative extraction fields as descriptive candidates.

Return an object with a fields array. Each item has nonempty string values for
metadata_field, observed_value, description, source_level, discovery_value,
and reasoning. Use snake_case field names. source_level is snapshot, document,
or both; discovery_value is high, medium, or low. The reasoning explains the
descriptive value and is evidence for later human review.

## User content

Supply the selected snapshot image and its exact snapshot identifier.
Supply a separately identified document-context object and its source record.
If context is unavailable, record that fact. Analyze this artifact independently
without a vocabulary or outputs accumulated from earlier requests.

## Historical boundary

The study asked for the 10–15 most useful fields; this was a prompt preference,
not a response-count guarantee. Discovery used GPT-5.5, medium reasoning, and
a 5,000-token output ceiling. These are historical settings, not current model
recommendations. The downloadable excerpts preserve selected results, not the
complete input messages. A document title shown in the source register must
not be represented as the complete historical document-context payload.

Source: Dy & Solatorio (2026). Artifact-Driven Metadata Schema Development:
A Human-in-the-Loop Methodology with Multimodal LLMs. Working paper. Appendix B.
  1. Set up a hosted or local model interface using the overview prerequisites. Select one development item and supply available contextual metadata separately. State what context is unavailable.
  2. Explain the artifact class, descriptive purpose, and content boundary from Chapter 1. Ask for reusable descriptive properties and evidence-supported values. Do not turn discovery into extracting the artifact's complete contents.
  3. Specify how to label the evidence source and uncertainty. State whether inference is allowed and how it must be identified.
  4. Use the same discovery instructions independently for each artifact. Do not insert earlier observations or an inventory updated after every request.
  5. Adapt the request to the chosen interface. Supply the actual image or other selected representation, its identifier, and the separately labelled context. Send the request with the chosen model settings. A filename or link alone does not supply image content unless the interface explicitly retrieves it.
  6. Save the original response and parsed observations in a new output location, together with the item identifier, model, instructions, settings, and completion status. Retain any error or incomplete response for inspection. Recipe 2.2 explains how to read the observations; Recipe 2.3 checks stored responses and covers collection processing.

Worked example. For the Jordan chart, the request setup contains:

Request elementPrepared example
ArtifactThe chart image identified as 027_Jordan-Emergency-Food-Security-Project_figure_001.png.
ContextThe separately labelled parent-document title, Jordan - Emergency Food Security Project, from the source register.
TaskPropose reusable descriptive properties and evidence-supported values for this item independently.
Response formatA JSON object with a fields array; each observation has the six string attributes specified in the request.

Recipe 2.2 shows recorded study responses of this kind and explains how to read them. A new run of the teaching request may suggest different names or values.

Study configuration

Recorded configuration. The study used GPT-5.5 with medium reasoning and a 5,000-token output ceiling for discovery. Available associated document metadata accompanied the image. The prompt requested the 10–15 most useful fields, but the study observed a range of 10–16. That preference was not a strict response-array constraint. These settings document the historical execution. A fresh replication needs its own model choice and configuration review.

Verification. Confirm that the model received the intended artifact and context, and that the saved response belongs to that item. Check for completion or an explicit failure before reviewing the observations. A completed request does not establish that its suggestions are correct.

What this does not do. The guide supplies a request specification rather than a provider-specific runner. Execution requires the configured hosted or local interface. An artifact-only request uses less context and changes the input protocol; record that variation. New outputs need their own review.

Recipe

2.2 Read discovery results against the source

Level
Foundational
Skills
Domain reviewer; technical practitioner
Indicative time
30–45 minutes for three artifacts

Result: Review notes connect suggested properties and values to their source evidence.

After discovery, read each suggested name together with its value and explanation. For example, geographic scope is a property; Afghanistan is a value for one item. A plausible suggestion still needs review before it becomes a schema field.

Input example. Use the response saved in Recipe 2.1, or follow the recorded study excerpts below to see what discovery results can look like. The exact identities and full selected observations are retained in the research evidence. In its examples array, find the entry with id independent-naming and read outputs.recorded_observations. Later recipes name the corresponding entries in the same way.

ArtifactProposed fieldRecorded observed value
Jordan food-expenditure chartfigure_titleFigure 2: Share of food expenditure in total expenditure p.c. (%)
Afghanistan income tabletable_titleMonthly labor income in Afghanistan and remittances from abroad
Russian fiscal chartchart_titleStrength of Total Revenue and Total Expenditure Anti-Cyclicality
The same Russian chartfigure_titleRussia: Strength of Anti-Cyclicality of Fiscal Revenues and Expenditures
Observation attributeInterpretation
metadata_fieldThis is the proposed reusable property name.
observed_valueThis is the source-grounded value or explicitly recorded uncertainty.
descriptionThis explains the intended property meaning in this observation.
source_levelThis identifies snapshot, document, or combined support.
discovery_valueThis records the model's high, medium, or low utility assessment.
reasoningThis is the model's brief stated justification for descriptive usefulness.
snapshot_id, source, and modelThese identify the item, its collection, and the model used.

Jordan's geographic observation states “Jordan, inferred from parent document title and project name; not visible in snapshot.” Keep that evidence distinction when comparing observations across items.

  1. Open the artifact alongside its discovery response. For the prepared example, use the linked images, the table above, and independent-naming in evidence.json.
  2. Read the proposed name, value, and description together. Check whether they describe the item within the scope set in Chapter 1.
  3. Check where the evidence comes from: the item itself, supplied context, or both. Keep uncertainty and labelled inference visible.
  4. Record questionable readings or out-of-scope suggestions for later review. Preserve the original observation alongside any correction.

Worked example. Reviewing the Russian records preserves an internal heading and a broader caption. The model rated their usefulness medium and high respectively; these are judgments, not measured retrieval scores. Similar purposes do not make the two strings interchangeable.

Next, check the stored response records in Recipe 2.3 before comparing observations across the sample.

Verification. Match each selected row to its identity and source register. For the Russian chart, explain why two headings were observed without assuming two final schema fields. For Jordan, identify the evidence source separately from the usefulness of the candidate property.

What this does not do. This inspection does not certify the model's reading, accept field names, or extract numerical series. The four selected observations are not the complete output for the three artifacts.

Recipe

2.3 Check responses and plan collection processing

Level
AI-native
Skills
Python and API practitioner; metadata reviewer
Indicative time
30 minutes for the worked example; live processing varies

Result: Response records have been checked for item identity, required fields, and completion status. A processing plan explains how a new run will preserve failures and retries.

The source review in Recipe 2.2 checks what the suggestions mean. This step checks that the saved records are in order: each identifies the intended item, contains the required fields, and records whether processing completed. For a new run, plan how to save attempts, identify failures, and retry them without losing the earlier records. One request per item can be processed sequentially; a provider Batch API is optional.

Prepared input. The response inspector loops over the three selected records in discovery_excerpt.jsonl. It verifies that their identities belong to development in source_images.csv, checks the recorded corpus, requires a non-empty model identifier, and validates the selected fields of successful records. It writes nothing and has no network or model client. Download these two files and source_images.csv into one working directory.

The response excerpt has one record per selected item. These fields explain what the checker reads:

Record fieldMeaning
snapshot_id, source, modelThese identify the item, collection, and recorded model.
status, error, incomplete_detailsThese record whether processing completed successfully.
parsed_output.fieldsThis contains the selected metadata observations.
original_field_countThis records how many fields the complete response contained.

Procedure. Inspect the prepared records against these requirements. For a new collection run, implement these requirements with the chosen model interface before repeating Recipe 2.1 across the development sample.

  1. Read frozen development identities from the register. Resolve each artifact's chosen input representation and allowed context from its source version. Keep reserved validation evidence outside discovery.
  2. Keep the discovery prompt fixed across the collection and submit one artifact at a time. Record the model, prompt version or checksum, settings, artifact identity, and context provenance with each attempt.
  3. Preserve the original response and parsed observations in a new run directory. Validate the result before marking the attempt successful. Log status, incomplete output, and exceptions even when no fields can be parsed.
  4. Specify resumption before running. For a proposed replication, skip identities with a validated successful result under the same configuration. Keep failed attempts inspectable and make retries explicit. Use a new run for a changed configuration.
  5. Check which items have responses, whether processing succeeded, and whether the required fields are present before combining observations from different items. Missing records and invalid responses must remain distinguishable from artifacts with no useful observations.

Worked example. Run the inspector from the directory containing all three downloads. Use Python 3.10 or newer; if the interpreter is named python3, substitute that name in the command. No packages, images, or LLM credentials are needed for this check.

python inspect_discovery.py discovery_excerpt.jsonl source_images.csv
3 selected response records: 3 valid, 0 failed
4 retained fields; 43 fields in the original valid responses
These excerpts omit original fields and raw responses.

The excerpts retain four title fields from 43 original fields. The printed counts distinguish the selected teaching records from the complete responses.

How the study handled repeated attempts

Steps 2–4 describe an auditable replication design. The historical runner wrote a row for each attempt, including failures, and resumed by skipping previously logged identities. Its skip rule included failed rows. It did not automatically implement the success-only retry policy proposed above. The available excerpts also omit raw responses and full input messages; exact byte-for-byte request reconstruction is not claimed.

Verification. The supplied excerpt exits with status 0. The inspector writes nothing, so it can be rerun with the same inputs. A logged execution failure produces status 1. Unknown or duplicate identities, inconsistent provenance, malformed selected observations, and input-path errors produce status 2. The script does not create a retry or send a request; its source shows the checks a collection runner must apply before considering a record usable. The processing plan also identifies its success rule, retry policy, and separate run directory before any optional execution.

What this does not do. The inspector cannot measure full-study completeness, replay omitted responses, or establish observational accuracy. Live collection processing remains optional and requires provider-specific integration, an explicit budget, and independent verification of new outputs.

Implementation options​

Choose inputs and a model interface that preserve the evidence needed for the artifact class. Hosted execution needs provider credentials and access; local execution needs an available model, compatible interface, and sufficient computing resources. Adapt the teaching request and response checks to that interface. Their formats are not guaranteed to work unchanged across providers.

Sequential requests are straightforward for small collections. Batching can manage larger runs, provided each artifact is still examined independently and failures remain identifiable. Context may be omitted when the artifact is sufficient; when supplied, identify it separately from direct evidence.

Resources​

  • The teaching discovery request separates the logical request specification from the historical configuration.
  • The response excerpt and source register support identity and response checks.
  • Dy & Solatorio (2026). Artifact-Driven Metadata Schema Development: A Human-in-the-Loop Methodology with Multimodal LLMs. Working paper. Sections 4.1 and 6.1 and Appendix B explain the discovery objective and execution.

Common mistakes​

  • A response contract can be confused with a fixed descriptive field inventory.
  • Observed values or extracted numerical contents can be accepted as reusable field names.
  • Inputting earlier discoveries into later requests can change independent discovery into an evolving vocabulary workflow.
  • A successful response status can be mistaken for a correct interpretation.
  • Blindly skipping logged identities can leave failed attempts permanently unprocessed.

Verification​

  • Each observation can be traced to a development item and its evidence source.
  • The response format and processing status are checked, and questionable readings are flagged for review.
  • A new-run plan identifies inputs, settings, output location, and failure handling.

Checklist​

0 of 4 done

Ticks are saved in this browser only.

Use the checked observations in Chapter 3 to compare repeated names and propose shared meanings.