Skip to main content

3. Round configuration and startup

Question: How would the prepared batch become extraction records and judge findings?

Rationale​

Chapter 2 prepares the batch and starting settings and explains how to check the manifest. This chapter shows how the proposed runner would turn that preparation into extraction records and judge findings. The runner is the experiment controller. It would use Pi to manage model sessions and deterministic tools, whose checks follow fixed rules, to verify sample IDs, validate metadata, and save records.

A record can pass schema validation even when it assigns the wrong role to visible text. The judge uses a fixed set of review questions to check the record against the data snapshot and the policy. Saving the extractions and findings gives the human reviewer a record of what was assessed. A finding that needs a human decision pauses the workflow before the council begins.

Target state​

  • The proposed startup sequence identifies what the extractor and judge receive, what they return, and what the tools check.
  • The extraction and review records can be traced to the prepared batch and policy. A finding that needs human judgment has a clear stopping point.

Maturity levels​

LevelSteps
FoundationalTrace the prepared records from the manifest through extraction and diagnosis.
AI-readySpecify the checks that each record must pass and when the round pauses.
AI-nativeDesign scoped Pi sessions and tools that can resume from validated records.
Running example

The round 001 files illustrate what a startup would read and save. The walkthrough uses the supplied records. The Pi workflow is proposed.

Recipes​

Recipe

3.1 Connect the manifest to extraction and diagnosis

Level
AI-native
Skills
Application developer and experiment lead
Indicative time
60 minutes

Result: A startup plan identifies the inputs and expected output of each model session.

Open the round 001 manifest. It lists D1 and D2 as Discovery samples and C1 and C2 as Challenge samples. Its artifacts.policy.used entry points to P001. Follow D2 through the two sessions.

The startup plan specifies what the proposed runner would send to each role and what it would save. Use the supplied records to check those handoffs.

SessionD2 inputsExpected record
Reference extractorD2 image, P001, and Schema v1.4A metadata record that the runner tags with D2 and P001 in the extraction records.
Diagnostic judgeThe same image, policy, and schema, plus the D2 extraction, the full human-reviewed annotation, and the review questionsA D2 review in the judge reviews.
  1. Include manifest generation and the checks from Recipe 2.3 as the first startup step. Require the selected model settings and provider credentials before model calls. The illustrative manifest’s unassigned models identify configuration needed for a future run.
  2. Plan a separate Pi session for the extractor and the judge. Pass the registered image and current policy to the extractor. Use a script or constrained tool to validate the metadata against the schema and save it with the round, sample, and policy IDs.
  3. Pass the validated extraction and its evidence to the judge with the supplied review questions and role instructions. Include the full annotation for Discovery samples. For Challenge samples, mark reference comparison as unavailable.
  4. Save one review per sample. Check that every expected sample has exactly one validated extraction and review selected for the round before handing the batch to human review. Keep any retry attempts separate so they do not appear as extra samples.

Worked example. The startup plan for D2 sends the registered table image to the extractor under P001. Once the metadata record passes validation, the judge receives it with the same image and the full D2 annotation. The two supplied records carry the same sample and policy IDs, so the review can be traced to the extraction it assesses.

What this does not do. The supplied records show the expected files for a startup plan. The project-specific Pi workflow still needs implementation and testing.

Recipe

3.2 Inspect the judge records and identify the pause

Level
AI-ready
Skills
Application developer and metadata reviewer
Indicative time
40 minutes

Result: A startup check identifies whether the round can proceed or needs a human decision.

Find F03 in D2’s illustrative judge review. These fields identify the issue and signal a pause.

{
"finding_id": "F03",
"field": "interpretive_notes",
"human_attention": true
}

The supplied review questions cover what the image supports, which relevant fields are missing, whether the output follows the policy, and which uncertainties need human judgment. In D2, the judge points to the stars and the footer that refers to the notes to table 1. The image leaves their thresholds open.

  1. Match the review records to the extraction records by round, sample, and policy IDs. Check that Discovery reviews report a reference comparison and Challenge reviews mark it as unavailable.
  2. Check that each finding identifies the affected field, visible evidence, applicable rules, and any uncertainty. Read attention_reason to understand why the judge did or did not flag it for human review.
  3. If a sample has no valid extraction or review, keep the stage incomplete. Send the findings and their evidence to the human reviewer. A finding with human_attention set to true blocks deliberation until the reviewer records a decision.

Worked example. The illustrative D2 review gives the team a clear next step. F03 asks whether the visible cross-reference is enough for the metadata, so the workflow would pause at human review. Chapter 4 takes up that decision and shows how to spot-check a case without an escalation.

What this does not do. A complete set of review records establishes which samples and fields were checked. Judge accuracy needs its own assessment.

Implementation options​

  • A controller can start separate Pi processes for the two roles or use Pi’s remote procedure call (RPC) interface when it needs to control an ongoing process. Use ordinary scripts or constrained tools for validation and file writes.
  • Save validated records as each stage finishes. If C2’s review is missing after an interruption, retain reviews whose inputs and settings still match and request C2’s review with the same image, extraction, policy, schema, model settings, and review questions.

Resources​

Common mistakes​

  • Allowing a model session to select samples or rewrite the manifest can change the recorded conditions.
  • Passing the human-reviewed annotation to the extractor gives it information outside the registered image.
  • Treating an incomplete review stage as ready for deliberation hides missing evidence.

Verification​

  • Trace D2 from the manifest to the extraction and review records and check that their sample and policy IDs agree.
  • Check that the proposed sessions receive only the files each role needs.
  • Use F03 and a missing C2 review to check that the workflow design pauses for the right reason in each case.

Checklist​

0 of 3 done

Ticks are saved in this browser only.