Blueprints BP-08
Latency and cost model router
One model for every request is too slow or too expensive.
01 / Fit
Use it when
- Request volume is high and most requests are routine.
- The quality bar can be measured per task type.
- One large model for everything misses the latency or cost target.
- An eval set exists that can show where a smaller model is good enough.
Avoid it when
- Volume is low. The saving will not cover the cost of building and maintaining the router.
- Quality cannot be measured. Routing without a quality bar only moves errors to the cheaper tier.
- Every request is hard. The router adds a step and escalates almost everything.
02 / The diagram
Run it. Then run the failure path.
Select a node to see what it does, how it fails and what we measure.
- Request (Task, budget). Arrives with a task type, a latency budget and a quality bar set by the product. Fails when: Every request is treated as equally hard and equally urgent. Safeguard: Task type and budgets are set by the caller, not guessed downstream.
- Cache (Seen before). Returns a stored answer when the same request, with the same context, has been answered before. Fails when: A cached answer is served after the underlying data has changed. Safeguard: Cache keys include context versions. Entries expire with their sources.
- Router (Pick a tier). Chooses the cheapest model tier expected to meet the quality bar for this request. Fails when: It sends hard requests to the small model, or everything to the large one. Safeguard: Routing rules are tuned on an eval set and start conservative.
- Small model (Fast, low cost). Answers the routine majority of requests quickly and cheaply. Fails when: It returns a fluent answer to a request beyond its ability. Safeguard: Its output is never returned unchecked.
- Quality check (Accept or not). Decides whether the answer meets the bar, using rule checks and task-specific signals. Fails when: The check costs as much as the larger model would have. Safeguard: Cheap checks first: format, grounding, required fields. A model check only where rules cannot decide.
- Response (With route log). Returns the answer and logs which route produced it, at what cost and latency. Fails when: Cost and quality cannot be traced back to a route. Safeguard: Every response is logged with its route, model version and check result.
- Route evals (Sets the bar). Replays a labelled set through every tier to find where the small model is good enough. Fails when: A model update shifts quality and the thresholds stay where they were. Safeguard: Thresholds are recomputed on every model or prompt change.
- Large model (Harder cases). Takes the requests the small model could not answer to the bar. Fails when: It becomes the default route and the saving disappears. Safeguard: An alert on escalation rate. A sustained rise triggers a routing review.
- Handoff (Person answers). A person takes the request when the larger model also fails the check or the budget is spent. Fails when: The handoff arrives after the latency budget has gone. Safeguard: One time limit across all tiers. The user is told the request has been passed on.
Read the diagram as text
- Request (Task, budget). Arrives with a task type, a latency budget and a quality bar set by the product. Fails when: Every request is treated as equally hard and equally urgent. Safeguard: Task type and budgets are set by the caller, not guessed downstream.
- Cache (Seen before). Returns a stored answer when the same request, with the same context, has been answered before. Fails when: A cached answer is served after the underlying data has changed. Safeguard: Cache keys include context versions. Entries expire with their sources.
- Router (Pick a tier). Chooses the cheapest model tier expected to meet the quality bar for this request. Fails when: It sends hard requests to the small model, or everything to the large one. Safeguard: Routing rules are tuned on an eval set and start conservative.
- Small model (Fast, low cost). Answers the routine majority of requests quickly and cheaply. Fails when: It returns a fluent answer to a request beyond its ability. Safeguard: Its output is never returned unchecked.
- Quality check (Accept or not). Decides whether the answer meets the bar, using rule checks and task-specific signals. Fails when: The check costs as much as the larger model would have. Safeguard: Cheap checks first: format, grounding, required fields. A model check only where rules cannot decide.
- Response (With route log). Returns the answer and logs which route produced it, at what cost and latency. Fails when: Cost and quality cannot be traced back to a route. Safeguard: Every response is logged with its route, model version and check result.
- Route evals (Sets the bar). Replays a labelled set through every tier to find where the small model is good enough. Fails when: A model update shifts quality and the thresholds stay where they were. Safeguard: Thresholds are recomputed on every model or prompt change.
- Large model (Harder cases). Takes the requests the small model could not answer to the bar. Fails when: It becomes the default route and the saving disappears. Safeguard: An alert on escalation rate. A sustained rise triggers a routing review.
- Handoff (Person answers). A person takes the request when the larger model also fails the check or the budget is spent. Fails when: The handoff arrives after the latency budget has gone. Safeguard: One time limit across all tiers. The user is told the request has been passed on.
03 / Components
Every component, and how it fails.
| Component | Job | Typical choices | Fails when | Safeguard |
|---|---|---|---|---|
| Request | Arrives with a task type, a latency budget and a quality bar set by the product. | API gateway with task type, latency budget and quality bar per request | Every request is treated as equally hard and equally urgent. | Task type and budgets are set by the caller, not guessed downstream. |
| Cache | Returns a stored answer when the same request, with the same context, has been answered before. | Response cache keyed on the request and context versions, with expiry | A cached answer is served after the underlying data has changed. | Cache keys include context versions. Entries expire with their sources. |
| Router | Chooses the cheapest model tier expected to meet the quality bar for this request. | Rules or a small classifier over task type and request features | It sends hard requests to the small model, or everything to the large one. | Routing rules are tuned on an eval set and start conservative. |
| Small model | Answers the routine majority of requests quickly and cheaply. | Small, fast language model tier | It returns a fluent answer to a request beyond its ability. | Its output is never returned unchecked. |
| Quality check | Decides whether the answer meets the bar, using rule checks and task-specific signals. | Rule-based output checks, with a lightweight model check where needed | The check costs as much as the larger model would have. | Cheap checks first: format, grounding, required fields. A model check only where rules cannot decide. |
| Response | Returns the answer and logs which route produced it, at what cost and latency. | Response handler with per-route cost, latency and version logging | Cost and quality cannot be traced back to a route. | Every response is logged with its route, model version and check result. |
| Route evals | Replays a labelled set through every tier to find where the small model is good enough. | Labelled eval set replayed across all model tiers | A model update shifts quality and the thresholds stay where they were. | Thresholds are recomputed on every model or prompt change. |
| Large model | Takes the requests the small model could not answer to the bar. | Larger language model tier, used on escalation | It becomes the default route and the saving disappears. | An alert on escalation rate. A sustained rise triggers a routing review. |
| Handoff | A person takes the request when the larger model also fails the check or the budget is spent. | Human support queue, with a notice to the user | The handoff arrives after the latency budget has gone. | One time limit across all tiers. The user is told the request has been passed on. |
04 / Budgets and evals
What the design is tuned for.
Cost per task at a fixed quality bar, p95 latency. These are design targets, not results.
| Stage | Design target |
|---|---|
| Cache and router | Close to no added latency. Rules or a small classifier, not a large model call. |
| Small model | Carries the routine majority, which is where the cost saving comes from. |
| Quality check | Much cheaper than the escalation it prevents. Rules first. |
| Escalation | One time limit across all tiers, so a slow escalation cannot break the latency budget. |
Eval plan 01
Golden set
Labelled requests per task type, each with outputs from every tier scored against the quality bar. It includes hard cases the small model is known to get wrong.
Eval plan 02
Release gates
Quality at the bar for every route, cost per task, tail latency and the false-accept rate of the quality check.
Eval plan 03
Catching regressions
Thresholds are recomputed when any model, prompt or check changes. Accepted small-model answers are sampled in production and scored against the larger model.
05 / Design notes
Where the whiteboard and production differ.
- Escalation rate drifts upward as the request mix changes, and the saving shrinks without any alert firing.
- An escalated request pays for both models and both latencies. The saving depends on how rarely that happens.
- A model update changes which tier is good enough for which task. Routing thresholds are tied to model versions.
- The quality check is the hardest part to build. A weak check turns the router into a way of returning cheaper wrong answers.
- Caching often saves more than routing for repetitive workloads, and it is simpler to get right.
06 / Deploy
Deploy this with us.
A pod can take BP-08 from this page to your stack. Tell us the workflow and we will tell you what changes for your systems.
You will talk to an engineer