Blueprints BP-07

Eval harness and release gate

Nobody can say whether the new prompt or model is better.

EvalsLogisticsFinanceEnterprise opsLast reviewed · November 2026

01 / Fit

Use it when

  • An AI system is in production, or about to be, and it changes often.
  • Changes are currently judged by trying a few examples by hand.
  • A regression would reach customers or a system of record.
  • There are reviewed cases, or people who can produce them.

Avoid it when

  • The system is still a prototype and the task definition changes weekly. A small hand-checked set is enough for now.
  • Nobody owns the golden set. An unmaintained set gives false confidence.
  • There is no agreed definition of a good output. Settle that before automating the scoring.

02 / The diagram

Run it. Then run the failure path.

Select a node to see what it does, how it fails and what we measure.

Idle
last scoresregressionCHANGEPrompt, modelGOLDEN SETChecked casesREPLAYCandidate runsSCORERules, judgeGATEPass or blockRELEASEStaged rolloutBASELINELast releaseBLOCKEDKeep baselineOWNER REVIEWFix or waive
  1. Change (Prompt, model). A proposed change to a prompt, a model, a tool or retrieval, submitted like a code change. Fails when: Changes reach production without passing through the harness. Safeguard: The release pipeline accepts only builds that carry a gate result.
  2. Golden set (Checked cases). Holds real cases with known-good outcomes, grouped by scenario and difficulty. Fails when: It goes stale, or prompts are tuned until they fit it. Safeguard: Refreshed from reviewed production cases. A held-out slice is never used for tuning.
  3. Replay (Candidate runs). Runs the candidate over every case under production settings and records full traces. Fails when: Runs are not repeatable, so two replays disagree. Safeguard: Pinned versions, fixed settings and repeated runs for the cases that vary.
  4. Score (Rules, judge). Scores candidate outputs with rule checks first and a model judge for what rules cannot express. Fails when: The judge is lenient, or shifts when its own model changes. Safeguard: The judge is calibrated against human labels and pinned to a version.
  5. Gate (Pass or block). Compares candidate scores with the baseline, scenario by scenario, and passes or blocks the release. Fails when: An overall average hides a regression in one important scenario. Safeguard: Thresholds are set per scenario. Critical cases must all pass.
  6. Release (Staged rollout). Ships the candidate to a share of traffic first and records it as the new baseline once it holds. Fails when: Production behaves differently from the golden set. Safeguard: Staged rollout with live monitoring and a one-step rollback.
  7. Baseline (Last release). Stores the scores and outputs of the current release for the same cases. Fails when: The baseline was scored with an older judge or an older golden set. Safeguard: The baseline is rescored whenever the judge or the set changes.
  8. Blocked (Keep baseline). Stops the release and keeps the current version live. Produces a report of the failing cases. Fails when: The report says a score dropped and not which cases caused it. Safeguard: The report lists each failing case with baseline and candidate outputs side by side.
  9. Owner review (Fix or waive). The change owner reads the failing cases and either fixes the change or signs a waiver with a reason. Fails when: Waivers become routine. Safeguard: Each waiver is logged with a name and a reason. Waivers are reviewed together on a schedule.
Tuned for · Regressions caught before releaseReference design
DataModelGuardHumanActive pathFallback path
Read the diagram as text
  1. Change (Prompt, model). A proposed change to a prompt, a model, a tool or retrieval, submitted like a code change. Fails when: Changes reach production without passing through the harness. Safeguard: The release pipeline accepts only builds that carry a gate result.
  2. Golden set (Checked cases). Holds real cases with known-good outcomes, grouped by scenario and difficulty. Fails when: It goes stale, or prompts are tuned until they fit it. Safeguard: Refreshed from reviewed production cases. A held-out slice is never used for tuning.
  3. Replay (Candidate runs). Runs the candidate over every case under production settings and records full traces. Fails when: Runs are not repeatable, so two replays disagree. Safeguard: Pinned versions, fixed settings and repeated runs for the cases that vary.
  4. Score (Rules, judge). Scores candidate outputs with rule checks first and a model judge for what rules cannot express. Fails when: The judge is lenient, or shifts when its own model changes. Safeguard: The judge is calibrated against human labels and pinned to a version.
  5. Gate (Pass or block). Compares candidate scores with the baseline, scenario by scenario, and passes or blocks the release. Fails when: An overall average hides a regression in one important scenario. Safeguard: Thresholds are set per scenario. Critical cases must all pass.
  6. Release (Staged rollout). Ships the candidate to a share of traffic first and records it as the new baseline once it holds. Fails when: Production behaves differently from the golden set. Safeguard: Staged rollout with live monitoring and a one-step rollback.
  7. Baseline (Last release). Stores the scores and outputs of the current release for the same cases. Fails when: The baseline was scored with an older judge or an older golden set. Safeguard: The baseline is rescored whenever the judge or the set changes.
  8. Blocked (Keep baseline). Stops the release and keeps the current version live. Produces a report of the failing cases. Fails when: The report says a score dropped and not which cases caused it. Safeguard: The report lists each failing case with baseline and candidate outputs side by side.
  9. Owner review (Fix or waive). The change owner reads the failing cases and either fixes the change or signs a waiver with a reason. Fails when: Waivers become routine. Safeguard: Each waiver is logged with a name and a reason. Waivers are reviewed together on a schedule.

03 / Components

Every component, and how it fails.

ComponentJobTypical choicesFails whenSafeguard
ChangeA proposed change to a prompt, a model, a tool or retrieval, submitted like a code change.Version control and a release pipeline that treats prompts and model settings as codeChanges reach production without passing through the harness.The release pipeline accepts only builds that carry a gate result.
Golden setHolds real cases with known-good outcomes, grouped by scenario and difficulty.Versioned dataset store with scenario tags and a held-out sliceIt goes stale, or prompts are tuned until they fit it.Refreshed from reviewed production cases. A held-out slice is never used for tuning.
ReplayRuns the candidate over every case under production settings and records full traces.Batch runner with pinned model versions, fixed settings and full tracingRuns are not repeatable, so two replays disagree.Pinned versions, fixed settings and repeated runs for the cases that vary.
ScoreScores candidate outputs with rule checks first and a model judge for what rules cannot express.Rule-based checks, plus a model judge calibrated against human labelsThe judge is lenient, or shifts when its own model changes.The judge is calibrated against human labels and pinned to a version.
GateCompares candidate scores with the baseline, scenario by scenario, and passes or blocks the release.Threshold comparison per scenario, run as a required pipeline checkAn overall average hides a regression in one important scenario.Thresholds are set per scenario. Critical cases must all pass.
ReleaseShips the candidate to a share of traffic first and records it as the new baseline once it holds.Staged rollout with live monitoring and rollbackProduction behaves differently from the golden set.Staged rollout with live monitoring and a one-step rollback.
BaselineStores the scores and outputs of the current release for the same cases.Stored scores and outputs for the current releaseThe baseline was scored with an older judge or an older golden set.The baseline is rescored whenever the judge or the set changes.
BlockedStops the release and keeps the current version live. Produces a report of the failing cases.Failed pipeline check with a case-level comparison reportThe report says a score dropped and not which cases caused it.The report lists each failing case with baseline and candidate outputs side by side.
Owner reviewThe change owner reads the failing cases and either fixes the change or signs a waiver with a reason.Review and sign-off step with a logged waiverWaivers become routine.Each waiver is logged with a name and a reason. Waivers are reviewed together on a schedule.

04 / Budgets and evals

What the design is tuned for.

Regressions caught before release. These are design targets, not results.

StageDesign target
ReplayFast enough to run on every change. A smaller smoke set runs first.
ScoreRule checks wherever a rule can decide. The model judge is kept for what is left.
GateDecisions per scenario, so one weak area cannot hide in an average.
Owner reviewFailing cases shown side by side, so the decision takes minutes of reading, not a rerun.

Eval plan 01

Golden set

Real cases with known-good outcomes, tagged by scenario and difficulty. It holds past failures as permanent cases and a held-out slice that is never used for tuning.

Eval plan 02

Release gates

Score per scenario against the baseline, a full pass on the critical cases, and judge agreement with human labels above the agreed bar.

Eval plan 03

Catching regressions

Every change replays the set and is compared with the baseline case by case. Live quality is compared with eval scores after release, and gaps become new cases.

05 / Design notes

Where the whiteboard and production differ.

  1. Teams usually start with too many metrics. A few that people trust and act on are worth more.
  2. A model judge needs its own evaluation. Without human-labelled calibration it drifts and nobody notices.
  3. Golden sets lose value quietly, as production traffic moves away from the cases they hold.
  4. Model outputs vary between runs, so a single replay can pass or fail by chance. Repeated runs on the unstable cases make the gate fair.
  5. A gate that blocks often and wrongly gets bypassed. The false-block rate needs as much attention as the regressions caught.

06 / Deploy

Deploy this with us.

A pod can take BP-07 from this page to your stack. Tell us the workflow and we will tell you what changes for your systems.

You will talk to an engineer

Book a scoping call

Start with one workflow.

Tell us where the work piles up. You will talk to an engineer, not a salesperson.

What do you need?

After the call you get a one-page scoping note: what we would build first, how we would measure it, and what it takes.