Evaluator-optimizer

The evaluator-optimizer loop produces one artifact and improves it: a worker drafts a complete answer, a judge grades that draft against a rubric, and a below-threshold score sends the draft plus the critique back for another pass that refines it. It repeats until the draft clears the rubric or the loop hits its iteration cap. Use it when there is a single thing to get right — a document, a plan, a piece of code — and quality is worth spending extra passes on.

This page explains the loop. For how a loop is chosen, see Coordination loops.

The loop

Rendering diagram…

A draft is graded against a rubric and refined until it passes the threshold

Under the hood the loop rides the VMAO skeleton rather than a separate topology, so it inherits VMAO’s checkpointing, budget carve, and bounded iteration for free:

• Generate maps to PLAN + EXECUTE. The planner authors a single generation brief; EXECUTE dispatches one worker — the generator — to produce a complete draft.
• Evaluate maps to VERIFY. A validator-tier LLM judge scores the draft against the rubric. A pass proceeds to synthesis; a fail carries the draft and the critique back; an evaluation-infrastructure error surfaces as a typed failure.
• Refine maps to REPLAN. The judge’s critique becomes revision feedback: the planner embeds the prior draft verbatim in a new brief so the next generator refines the last draft rather than starting over.

The rubric drives termination

The loop’s stopping condition is the rubric, not a turn count. A rubric is a checklist of weighted criteria plus a minimum score; each criterion is graded pass/fail and contributes its weight on a pass. The draft passes when the weighted sum of passing criteria meets the threshold. A rubric has no meaningful default — a rubric-less judge would pass every draft — so a session must supply one, and an absent or malformed rubric is rejected up front rather than run on defaults. The judge defaults to a stronger (Smart-class) model, since grading is a reasoning-heavy role. If the draft never clears the threshold, VMAO’s coverage cap bounds the loop and routes to the stall path instead of looping forever.

Research lineage

This is the evaluator-optimizer pattern from Anthropic’s Building Effective Agents — the same generator-critic shape as Self-Refine (Madaan et al., 2023), Reflexion (Shinn et al., 2023), and CRITIC (Gou et al., 2023): a generator produces a candidate, a critic scores it, and the critique feeds the next generation. apomesh carries the feedback through VMAO’s replan edge — draft-plus-critique refinement mediated by the planner — rather than a hidden inner loop.

When to choose it

Choose evaluator-optimizer when there is one artifact to perfect against explicit criteria: a report that must hit a quality bar, generated code that must satisfy a checklist, a plan that must cover named requirements. If instead you want diverse independent attempts reconciled to one answer, reach for parallel-sample; if you want adversarial scrutiny of a claim, reach for debate.

Where this shows up

• Reference: the substrate vmao module’s evaluator-optimizer strategy under Reference.