Blueprints BP-07
Eval harness and release gate
Nobody can say whether the new prompt or model is better.
01 / Fit
Use it when
- An AI system is in production, or about to be, and it changes often.
- Changes are currently judged by trying a few examples by hand.
- A regression would reach customers or a system of record.
- There are reviewed cases, or people who can produce them.
Avoid it when
- The system is still a prototype and the task definition changes weekly. A small hand-checked set is enough for now.
- Nobody owns the golden set. An unmaintained set gives false confidence.
- There is no agreed definition of a good output. Settle that before automating the scoring.
02 / The diagram
Run it. Then run the failure path.
Select a node to see what it does, how it fails and what we measure.
- Change (Prompt, model). A proposed change to a prompt, a model, a tool or retrieval, submitted like a code change. Fails when: Changes reach production without passing through the harness. Safeguard: The release pipeline accepts only builds that carry a gate result.
- Golden set (Checked cases). Holds real cases with known-good outcomes, grouped by scenario and difficulty. Fails when: It goes stale, or prompts are tuned until they fit it. Safeguard: Refreshed from reviewed production cases. A held-out slice is never used for tuning.
- Replay (Candidate runs). Runs the candidate over every case under production settings and records full traces. Fails when: Runs are not repeatable, so two replays disagree. Safeguard: Pinned versions, fixed settings and repeated runs for the cases that vary.
- Score (Rules, judge). Scores candidate outputs with rule checks first and a model judge for what rules cannot express. Fails when: The judge is lenient, or shifts when its own model changes. Safeguard: The judge is calibrated against human labels and pinned to a version.
- Gate (Pass or block). Compares candidate scores with the baseline, scenario by scenario, and passes or blocks the release. Fails when: An overall average hides a regression in one important scenario. Safeguard: Thresholds are set per scenario. Critical cases must all pass.
- Release (Staged rollout). Ships the candidate to a share of traffic first and records it as the new baseline once it holds. Fails when: Production behaves differently from the golden set. Safeguard: Staged rollout with live monitoring and a one-step rollback.
- Baseline (Last release). Stores the scores and outputs of the current release for the same cases. Fails when: The baseline was scored with an older judge or an older golden set. Safeguard: The baseline is rescored whenever the judge or the set changes.
- Blocked (Keep baseline). Stops the release and keeps the current version live. Produces a report of the failing cases. Fails when: The report says a score dropped and not which cases caused it. Safeguard: The report lists each failing case with baseline and candidate outputs side by side.
- Owner review (Fix or waive). The change owner reads the failing cases and either fixes the change or signs a waiver with a reason. Fails when: Waivers become routine. Safeguard: Each waiver is logged with a name and a reason. Waivers are reviewed together on a schedule.
Read the diagram as text
- Change (Prompt, model). A proposed change to a prompt, a model, a tool or retrieval, submitted like a code change. Fails when: Changes reach production without passing through the harness. Safeguard: The release pipeline accepts only builds that carry a gate result.
- Golden set (Checked cases). Holds real cases with known-good outcomes, grouped by scenario and difficulty. Fails when: It goes stale, or prompts are tuned until they fit it. Safeguard: Refreshed from reviewed production cases. A held-out slice is never used for tuning.
- Replay (Candidate runs). Runs the candidate over every case under production settings and records full traces. Fails when: Runs are not repeatable, so two replays disagree. Safeguard: Pinned versions, fixed settings and repeated runs for the cases that vary.
- Score (Rules, judge). Scores candidate outputs with rule checks first and a model judge for what rules cannot express. Fails when: The judge is lenient, or shifts when its own model changes. Safeguard: The judge is calibrated against human labels and pinned to a version.
- Gate (Pass or block). Compares candidate scores with the baseline, scenario by scenario, and passes or blocks the release. Fails when: An overall average hides a regression in one important scenario. Safeguard: Thresholds are set per scenario. Critical cases must all pass.
- Release (Staged rollout). Ships the candidate to a share of traffic first and records it as the new baseline once it holds. Fails when: Production behaves differently from the golden set. Safeguard: Staged rollout with live monitoring and a one-step rollback.
- Baseline (Last release). Stores the scores and outputs of the current release for the same cases. Fails when: The baseline was scored with an older judge or an older golden set. Safeguard: The baseline is rescored whenever the judge or the set changes.
- Blocked (Keep baseline). Stops the release and keeps the current version live. Produces a report of the failing cases. Fails when: The report says a score dropped and not which cases caused it. Safeguard: The report lists each failing case with baseline and candidate outputs side by side.
- Owner review (Fix or waive). The change owner reads the failing cases and either fixes the change or signs a waiver with a reason. Fails when: Waivers become routine. Safeguard: Each waiver is logged with a name and a reason. Waivers are reviewed together on a schedule.
03 / Components
Every component, and how it fails.
| Component | Job | Typical choices | Fails when | Safeguard |
|---|---|---|---|---|
| Change | A proposed change to a prompt, a model, a tool or retrieval, submitted like a code change. | Version control and a release pipeline that treats prompts and model settings as code | Changes reach production without passing through the harness. | The release pipeline accepts only builds that carry a gate result. |
| Golden set | Holds real cases with known-good outcomes, grouped by scenario and difficulty. | Versioned dataset store with scenario tags and a held-out slice | It goes stale, or prompts are tuned until they fit it. | Refreshed from reviewed production cases. A held-out slice is never used for tuning. |
| Replay | Runs the candidate over every case under production settings and records full traces. | Batch runner with pinned model versions, fixed settings and full tracing | Runs are not repeatable, so two replays disagree. | Pinned versions, fixed settings and repeated runs for the cases that vary. |
| Score | Scores candidate outputs with rule checks first and a model judge for what rules cannot express. | Rule-based checks, plus a model judge calibrated against human labels | The judge is lenient, or shifts when its own model changes. | The judge is calibrated against human labels and pinned to a version. |
| Gate | Compares candidate scores with the baseline, scenario by scenario, and passes or blocks the release. | Threshold comparison per scenario, run as a required pipeline check | An overall average hides a regression in one important scenario. | Thresholds are set per scenario. Critical cases must all pass. |
| Release | Ships the candidate to a share of traffic first and records it as the new baseline once it holds. | Staged rollout with live monitoring and rollback | Production behaves differently from the golden set. | Staged rollout with live monitoring and a one-step rollback. |
| Baseline | Stores the scores and outputs of the current release for the same cases. | Stored scores and outputs for the current release | The baseline was scored with an older judge or an older golden set. | The baseline is rescored whenever the judge or the set changes. |
| Blocked | Stops the release and keeps the current version live. Produces a report of the failing cases. | Failed pipeline check with a case-level comparison report | The report says a score dropped and not which cases caused it. | The report lists each failing case with baseline and candidate outputs side by side. |
| Owner review | The change owner reads the failing cases and either fixes the change or signs a waiver with a reason. | Review and sign-off step with a logged waiver | Waivers become routine. | Each waiver is logged with a name and a reason. Waivers are reviewed together on a schedule. |
04 / Budgets and evals
What the design is tuned for.
Regressions caught before release. These are design targets, not results.
| Stage | Design target |
|---|---|
| Replay | Fast enough to run on every change. A smaller smoke set runs first. |
| Score | Rule checks wherever a rule can decide. The model judge is kept for what is left. |
| Gate | Decisions per scenario, so one weak area cannot hide in an average. |
| Owner review | Failing cases shown side by side, so the decision takes minutes of reading, not a rerun. |
Eval plan 01
Golden set
Real cases with known-good outcomes, tagged by scenario and difficulty. It holds past failures as permanent cases and a held-out slice that is never used for tuning.
Eval plan 02
Release gates
Score per scenario against the baseline, a full pass on the critical cases, and judge agreement with human labels above the agreed bar.
Eval plan 03
Catching regressions
Every change replays the set and is compared with the baseline case by case. Live quality is compared with eval scores after release, and gaps become new cases.
05 / Design notes
Where the whiteboard and production differ.
- Teams usually start with too many metrics. A few that people trust and act on are worth more.
- A model judge needs its own evaluation. Without human-labelled calibration it drifts and nobody notices.
- Golden sets lose value quietly, as production traffic moves away from the cases they hold.
- Model outputs vary between runs, so a single replay can pass or fail by chance. Repeated runs on the unstable cases make the gate fair.
- A gate that blocks often and wrongly gets bypassed. The false-block rate needs as much attention as the regressions caught.
06 / Deploy
Deploy this with us.
A pod can take BP-07 from this page to your stack. Tell us the workflow and we will tell you what changes for your systems.
You will talk to an engineer