Blueprints BP-08

Latency and cost model router

One model for every request is too slow or too expensive.

EvalsLogisticsFinanceEnterprise opsLast reviewed · November 2026

01 / Fit

Use it when

  • Request volume is high and most requests are routine.
  • The quality bar can be measured per task type.
  • One large model for everything misses the latency or cost target.
  • An eval set exists that can show where a smaller model is good enough.

Avoid it when

  • Volume is low. The saving will not cover the cost of building and maintaining the router.
  • Quality cannot be measured. Routing without a quality bar only moves errors to the cheaper tier.
  • Every request is hard. The router adds a step and escalates almost everything.

02 / The diagram

Run it. Then run the failure path.

Select a node to see what it does, how it fails and what we measure.

Idle
cache hitlow scoreescalatedthresholdsREQUESTTask, budgetCACHESeen beforeROUTERPick a tierSMALL MODELFast, low costQUALITY CHECKAccept or notRESPONSEWith route logROUTE EVALSSets the barLARGE MODELHarder casesHANDOFFPerson answers
  1. Request (Task, budget). Arrives with a task type, a latency budget and a quality bar set by the product. Fails when: Every request is treated as equally hard and equally urgent. Safeguard: Task type and budgets are set by the caller, not guessed downstream.
  2. Cache (Seen before). Returns a stored answer when the same request, with the same context, has been answered before. Fails when: A cached answer is served after the underlying data has changed. Safeguard: Cache keys include context versions. Entries expire with their sources.
  3. Router (Pick a tier). Chooses the cheapest model tier expected to meet the quality bar for this request. Fails when: It sends hard requests to the small model, or everything to the large one. Safeguard: Routing rules are tuned on an eval set and start conservative.
  4. Small model (Fast, low cost). Answers the routine majority of requests quickly and cheaply. Fails when: It returns a fluent answer to a request beyond its ability. Safeguard: Its output is never returned unchecked.
  5. Quality check (Accept or not). Decides whether the answer meets the bar, using rule checks and task-specific signals. Fails when: The check costs as much as the larger model would have. Safeguard: Cheap checks first: format, grounding, required fields. A model check only where rules cannot decide.
  6. Response (With route log). Returns the answer and logs which route produced it, at what cost and latency. Fails when: Cost and quality cannot be traced back to a route. Safeguard: Every response is logged with its route, model version and check result.
  7. Route evals (Sets the bar). Replays a labelled set through every tier to find where the small model is good enough. Fails when: A model update shifts quality and the thresholds stay where they were. Safeguard: Thresholds are recomputed on every model or prompt change.
  8. Large model (Harder cases). Takes the requests the small model could not answer to the bar. Fails when: It becomes the default route and the saving disappears. Safeguard: An alert on escalation rate. A sustained rise triggers a routing review.
  9. Handoff (Person answers). A person takes the request when the larger model also fails the check or the budget is spent. Fails when: The handoff arrives after the latency budget has gone. Safeguard: One time limit across all tiers. The user is told the request has been passed on.
Tuned for · Cost per task at a fixed quality bar, p95 latencyReference design
DataModelGuardHumanActive pathFallback path
Read the diagram as text
  1. Request (Task, budget). Arrives with a task type, a latency budget and a quality bar set by the product. Fails when: Every request is treated as equally hard and equally urgent. Safeguard: Task type and budgets are set by the caller, not guessed downstream.
  2. Cache (Seen before). Returns a stored answer when the same request, with the same context, has been answered before. Fails when: A cached answer is served after the underlying data has changed. Safeguard: Cache keys include context versions. Entries expire with their sources.
  3. Router (Pick a tier). Chooses the cheapest model tier expected to meet the quality bar for this request. Fails when: It sends hard requests to the small model, or everything to the large one. Safeguard: Routing rules are tuned on an eval set and start conservative.
  4. Small model (Fast, low cost). Answers the routine majority of requests quickly and cheaply. Fails when: It returns a fluent answer to a request beyond its ability. Safeguard: Its output is never returned unchecked.
  5. Quality check (Accept or not). Decides whether the answer meets the bar, using rule checks and task-specific signals. Fails when: The check costs as much as the larger model would have. Safeguard: Cheap checks first: format, grounding, required fields. A model check only where rules cannot decide.
  6. Response (With route log). Returns the answer and logs which route produced it, at what cost and latency. Fails when: Cost and quality cannot be traced back to a route. Safeguard: Every response is logged with its route, model version and check result.
  7. Route evals (Sets the bar). Replays a labelled set through every tier to find where the small model is good enough. Fails when: A model update shifts quality and the thresholds stay where they were. Safeguard: Thresholds are recomputed on every model or prompt change.
  8. Large model (Harder cases). Takes the requests the small model could not answer to the bar. Fails when: It becomes the default route and the saving disappears. Safeguard: An alert on escalation rate. A sustained rise triggers a routing review.
  9. Handoff (Person answers). A person takes the request when the larger model also fails the check or the budget is spent. Fails when: The handoff arrives after the latency budget has gone. Safeguard: One time limit across all tiers. The user is told the request has been passed on.

03 / Components

Every component, and how it fails.

ComponentJobTypical choicesFails whenSafeguard
RequestArrives with a task type, a latency budget and a quality bar set by the product.API gateway with task type, latency budget and quality bar per requestEvery request is treated as equally hard and equally urgent.Task type and budgets are set by the caller, not guessed downstream.
CacheReturns a stored answer when the same request, with the same context, has been answered before.Response cache keyed on the request and context versions, with expiryA cached answer is served after the underlying data has changed.Cache keys include context versions. Entries expire with their sources.
RouterChooses the cheapest model tier expected to meet the quality bar for this request.Rules or a small classifier over task type and request featuresIt sends hard requests to the small model, or everything to the large one.Routing rules are tuned on an eval set and start conservative.
Small modelAnswers the routine majority of requests quickly and cheaply.Small, fast language model tierIt returns a fluent answer to a request beyond its ability.Its output is never returned unchecked.
Quality checkDecides whether the answer meets the bar, using rule checks and task-specific signals.Rule-based output checks, with a lightweight model check where neededThe check costs as much as the larger model would have.Cheap checks first: format, grounding, required fields. A model check only where rules cannot decide.
ResponseReturns the answer and logs which route produced it, at what cost and latency.Response handler with per-route cost, latency and version loggingCost and quality cannot be traced back to a route.Every response is logged with its route, model version and check result.
Route evalsReplays a labelled set through every tier to find where the small model is good enough.Labelled eval set replayed across all model tiersA model update shifts quality and the thresholds stay where they were.Thresholds are recomputed on every model or prompt change.
Large modelTakes the requests the small model could not answer to the bar.Larger language model tier, used on escalationIt becomes the default route and the saving disappears.An alert on escalation rate. A sustained rise triggers a routing review.
HandoffA person takes the request when the larger model also fails the check or the budget is spent.Human support queue, with a notice to the userThe handoff arrives after the latency budget has gone.One time limit across all tiers. The user is told the request has been passed on.

04 / Budgets and evals

What the design is tuned for.

Cost per task at a fixed quality bar, p95 latency. These are design targets, not results.

StageDesign target
Cache and routerClose to no added latency. Rules or a small classifier, not a large model call.
Small modelCarries the routine majority, which is where the cost saving comes from.
Quality checkMuch cheaper than the escalation it prevents. Rules first.
EscalationOne time limit across all tiers, so a slow escalation cannot break the latency budget.

Eval plan 01

Golden set

Labelled requests per task type, each with outputs from every tier scored against the quality bar. It includes hard cases the small model is known to get wrong.

Eval plan 02

Release gates

Quality at the bar for every route, cost per task, tail latency and the false-accept rate of the quality check.

Eval plan 03

Catching regressions

Thresholds are recomputed when any model, prompt or check changes. Accepted small-model answers are sampled in production and scored against the larger model.

05 / Design notes

Where the whiteboard and production differ.

  1. Escalation rate drifts upward as the request mix changes, and the saving shrinks without any alert firing.
  2. An escalated request pays for both models and both latencies. The saving depends on how rarely that happens.
  3. A model update changes which tier is good enough for which task. Routing thresholds are tied to model versions.
  4. The quality check is the hardest part to build. A weak check turns the router into a way of returning cheaper wrong answers.
  5. Caching often saves more than routing for repetitive workloads, and it is simpler to get right.

06 / Deploy

Deploy this with us.

A pod can take BP-08 from this page to your stack. Tell us the workflow and we will tell you what changes for your systems.

You will talk to an engineer

Book a scoping call

Start with one workflow.

Tell us where the work piles up. You will talk to an engineer, not a salesperson.

What do you need?

After the call you get a one-page scoping note: what we would build first, how we would measure it, and what it takes.