Cost-Aware Multi-Agent LLM Router

A router that sits in front of multiple LLMs of different price and quality tiers and decides, per prompt, which tier should answer it. Most traffic goes to a cheap model; only genuinely hard prompts pay for an expensive one. The pattern is well established (LMSYS RouteLLM[1], RouterBench[2], cascade-routing approaches like FrugalGPT[3], and the "adaptive model" approach used in products like Devin/Windsurf). The value here is engineering rigor and honest evaluation of a known pattern, not a new idea, including publishing an earlier version that didn't save money and explaining why.

0.9%cheaper than always calling the top-tier model

0.0%escalation rate at the production error budget (ε=0.15)

0.0%accuracy, no measurable quality loss vs. always-capable

$21,163per year saved, illustrative at 1M requests/month

How it works

A logistic regression, fit on real labeled outcomes (did the draft's answer actually match ground truth?), turns those four numbers into a calibrated probability:

where $\sigma$ is the logistic (sigmoid) function, $\sigma(z) = \frac{1}{1 + e^{-z}}$, squashing the linear score into a probability between 0 and 1.

The router accepts the cheap tier's answer only if its predicted error stays under a configured error budget $\epsilon$, using a direct hypothesis-testing framing rather than a threshold pulled from a prompt:

If escalating past mid also exceeds budget (when a mid-tier score is available), it escalates once more to the capable tier, treating the top of the ladder as a trusted fallback rather than separately calibrated.

Calibration, not just accuracy

A router that is simply "accurate" isn't enough. A model that says "90% confident" should be right about 90% of the time it says that. The calibrator's confidence is checked for honesty, not just raw accuracy, using Expected Calibration Error[5] over $M$ confidence bins:

This is the actual differentiator versus a naive difficulty classifier: routing on a number that's been checked for honesty, not just a plausible-sounding LLM output. Training on 80 rows and validating on 20 held-out rows (from the same distribution, fit at training time) produced:

That validation split is a sanity check, not the real generalization test. It's drawn from the same distribution the model fit on, just held out at fit time. The real test is a fully disjoint 50-prompt holdout set the calibrator never touched, covered in Results below.

The bug I caught

After the first full held-out evaluation, the calibrated router cost more than always calling the top-tier model directly across every error budget from 0.05 to 0.50. Sweeping the threshold didn't fix it, because the threshold wasn't the bug.

Checking config/pricing.json directly, rather than assuming the tier ladder was correctly ordered, showed why: the "mid" tier was Claude Haiku 4.5 at $1.00/$5.00 per 1M tokens, and the "capable" tier was Gemini 3.1 Flash-Lite at $0.25/$1.50. Mid was priced above capable. Every escalation to mid was strictly cost-dominated by just going straight to capable. No error-budget value could route around a ladder that was upside down. Swapping the assignments (Gemini to mid, Claude Sonnet 5 to capable, genuinely priced above Gemini) flipped the result completely.

Results

At the production default (ε = 0.15): 93.9% cheaper, no measurable quality loss. Held-out evaluation on 50 GSM8K[6]/MMLU[7] prompts from data/eval_holdout.jsonl, never seen during calibrator training, against always_capable (every prompt sent to Claude Sonnet 5 directly, $0.09387 total / $0.0018774 per prompt):

• Cost: $0.00569 vs. $0.09387, a 93.9% reduction.
• Escalation rate: 50.0% of prompts escalate past the cheap tier at ε=0.15. The other half are answered by the cheap tier alone. This is the number that determines how often the router actually pays for a pricier model.
• Accuracy: 96.0% (86.5-98.9% CI) vs. 92.0% (83.8-97.9% CI). The confidence intervals overlap, so this is not a statistically distinguishable difference at n = 50. That's parity, not a claimed improvement. A router that's simultaneously cheaper and more accurate than the model it routes to would be a free-lunch claim a 50-example sample can't support.
• At scale (illustrative, extrapolated from the real measured per-prompt rate, not a claim about any specific deployment's traffic): 1M requests/month would cost $113.80 with the router vs. $1,877.40 always-capable, saving $21,163/year.

Full 4-policy comparison on the same held-out set:

Sweeping the error budget from 0.05 to 0.50 shows accuracy peaking at ε = 0.10-0.20 (96.0%, 50-52% escalation rate) before falling off as the budget loosens past 0.30 (86.0%, 8-10% escalation rate). This was recomputed entirely from logged (calibrated P(correct), actual outcome) pairs, with zero additional API calls. The threshold itself is treated as a hyperparameter tuned on held-out data, following the cascade-routing deferral literature[8][9], rather than hand-picked.

Honest caveat: n = 50, and the confidence intervals are real and non-trivial. This is a genuine, reproducible result, not a fully settled one. A larger held-out set would tighten the claim.

The obvious approach first

Cost went up, not down. The cheap model answered roughly 2.5x more verbosely than the capable one, and cost is tokens × price, not just the per-token rate. That single finding is why the router above exists: a routing system that only looks at per-token pricing can miss the actual cost driver completely, and no amount of tuning the same prompted-classifier approach would have fixed it. There was also a deeper problem with the difficulty score itself. A "3 out of 5" from an LLM is a plausible-sounding number with no calibration, no confidence interval, and no way to know if it is actually well-founded. The calibrated router's ECE/Brier checks (above) are built to catch exactly this.

Production hardening

Two bugs only showed up once this actually ran on Cloud Run, not in local testing: a 1Gi memory allocation that crashed under sentence-transformers' real footprint (fixed by bumping to 2Gi), and an embedding model that re-downloaded from HuggingFace on every cold start under Cloud Run's scale-to-zero behavior, hitting HuggingFace's rate limit and crashing the container before it finished starting (fixed by baking the model into the Docker image at build time).

A concurrency test (Locust, 8 concurrent users, 45 seconds, against the live /route endpoint) produced 86 requests with zero client-visible failures, and unplanned, triggered a real, organic circuit-breaker trip. Gemini's breaker was closed before the burst, tripped open immediately after (real transient provider failures under concurrency hit the 5-consecutive-failure threshold), and every one of the 86 requests still returned 200 OK via fallback to the capable tier. The breaker recovered to closed after the 30-second reset window, confirmed live against the running /metrics endpoint.

Try it

First request after a cold start takes roughly 40 seconds (container start, model load, real provider calls); warm requests run 2 to 3 seconds.

Stack

References

1. LMSYS RouteLLM. github.com/lm-sys/RouteLLM.
2. RouterBench. arXiv:2403.12031.
3. Chen, L., Zaharia, M., & Zou, J. (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. Stanford University. arXiv:2305.05176.
4. Wang, X., et al. (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv:2203.11171.
5. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. arXiv:1706.04599.
6. Cobbe, K., et al. (2021). Training Verifiers to Solve Math Word Problems (GSM8K). arXiv:2110.14168.
7. Hendrycks, D., et al. (2021). Measuring Massive Multitask Language Understanding (MMLU). arXiv:2009.03300.
8. Kang, et al. C3PO-style deferral rules for cascade routing. arXiv:2604.14251.
9. Decision-theoretic cascade literature. arXiv:2605.06350.