LLM model routing: balance cost, latency and quality inside policy

The strongest model was available. It wasn't allowed.

A benefits administrator uses an AI application to review employee medical appeals. It can route requests to a private deployment in an EU region, a specialized language model, and several externally operated model endpoints.

One appeal contains health information and French clinical evidence. The contract says processing must stay in an approved EU region, with no provider retention of the content. The usual private regional route is down for maintenance.

The router scores candidates on predicted quality, latency and price. Residency and retention are in the score too, as small negative weights. The top scorer is a globally operated model with strong benchmarks and low latency right now. It wins.

The answer is fluent and useful. It matches the JSON schema. Nothing errors. Quality is green, latency is green, and the call was cheaper than the regional route.

Health data crossed a prohibited boundary. The provider wasn't covered by the application's evaluation or retention terms. Nobody can defend why that deployment received the case.

What is LLM model routing?

It's choosing, for each request, which model, provider, region and deployment does the work. In an enterprise, that's two decisions: which routes are allowed to see this request, then which allowed route is best right now.

The common description is simpler: predict which model will answer well, estimate cost and latency, pick the best trade-off. Research on cascades and learned routers shows that this can cut cost substantially while keeping quality in evaluated settings. It's a useful technique. It's not a complete design.

A route is more than a model name. It's a provider, a model revision, a deployment mode, a region, a serving configuration, a context limit, an output contract, a retention posture, tool behavior and an availability class. And the right route depends on the request: the task, the data class, the purpose, whose authority it runs under, the region rules, the deadline and what happens if the answer is wrong.

So there's often no single best model. One route has the highest quality but is prohibited for the data class. Another is cheap but lacks the language or structured-output support. A third fits the task but will miss the deadline under today's queue. A fourth runs locally but has never been evaluated for this population.

Is model routing a cost decision or a policy decision?

Policy first. Data class, region, retention, provider approval and evaluation coverage decide which routes may compete. Only then do cost, latency and quality choose among the survivors.

Reverse the order and you build a system where enough savings or quality can appear to justify a route that should never have been in the running.

The policy stage asks concrete questions:

• Data class: Does this request contain personal, health, financial, privileged or export-controlled data? Which routes are approved for that class?
• Region and residency: Where may this tenant's data be processed? Does the route keep processing, logs and any caches inside that boundary?
• Provider approval: Is there a contract, a security review and acceptable retention and training-use terms for this provider and this purpose?
• Identity and purpose: Is the person or agent behind the request permitted to use this route for this purpose?
• Evaluation coverage: Has this route been evaluated for this task, language and population, at this level of consequence?
• Capability: Does it support the modality, context size, structured output and tool use the step needs?

Any "no" removes the route. Not a penalty. Removal.

Hard constraints and objectives aren't the same thing

Weighted scores are mathematically convenient and architecturally dangerous. A score implies that a big enough gain on one axis makes up for a breach on another. For residency, retention and data class, nothing makes up for it.

Cost and latency can go either way. Usually they're objectives. But if the application declares a hard spend ceiling or a regulatory deadline, they become constraints. The metric doesn't decide which it is; the application's declared requirements do.

Settle deterministic steps before any model

The cheapest, fastest and most reliable model call is the one you don't make. Before routing anything to a model, settle what known facts and rules can answer: exact lookups in the system of record, calculations, policy thresholds, format checks, identity matching. Then route only the genuinely ambiguous part, and send it a smaller, cleaner input.

This changes routing economics more than any model swap. It also shrinks the policy surface: data that never goes to a model can't cross a prohibited boundary. (See deterministic-first routing.)

Describe every route as a contract

A registry that holds only a model alias, a price and a context-window size can't support governed routing. Each route needs a machine-readable description:

Record the model that actually answered, not just the one requested. Hosted aliases can resolve to new revisions without notice.

Why isn't a benchmark score enough to choose a model?

Because benchmarks measure general ability, not fitness for your task, data, language, tools and consequences. A route is only covered for the conditions you've actually evaluated it on.

A route tested on short English prompts isn't covered for long multilingual contracts. A model tested without tools may degrade when it has to select and fill in enterprise tool calls. A hosted endpoint and a private quantized deployment of the same model family can differ in context handling, latency and output quality.

That doesn't mean every request needs a study. It means the router shouldn't claim coverage nobody has measured. When a request falls outside coverage, the options are to exclude the route, use it for advisory output only, add deterministic verification, escalate to another route or a person, or decline.

Fallbacks must never widen permission

Outages, rate limits and capacity crunches are where routing policy gets tested. The tempting fix is to fail over to "whatever's up." That's exactly what happened in the opening scenario.

Write fallback chains in advance, and validate them against policy when you publish the workflow, not during the incident. For each step, declare what happens when no allowed route is available within the deadline: queue with a timer, hand off to a reviewer, return a typed "not available" result, or decline the case. Each of those is a legitimate, explainable outcome. A silent cross-region call is not.

The same rule applies to retries. A retry after a model timeout goes to the same allowed set, and it shouldn't resend more context than the first attempt.

Optimize what's left on outcomes, not unit prices

Inside the allowed set, real trade-offs remain. Compare routes on cost per resolved outcome, not price per token: a cheaper model that triggers more fallbacks, retries and human review often costs more per case (see AI cost attribution). Use latency distributions, not averages, against the step's deadline. Weigh quality as evaluated margin for this task, including how often the route abstains appropriately. Consider provider concentration as a resilience objective.

Then measure what happened. Log the allowed set, the chosen route, the reason it won, the fallback path if any, and the outcome. That record lets you tune routing on evidence and answer the auditor's question: why was this model allowed to see this case?

Common anti-patterns

Ten rules for LLM model routing

1. Settle deterministic steps before calling any model.
2. Decide the allowed route set before scoring anything.
3. Treat data class, region, retention and provider approval as filters, not weights.
4. Require evaluation coverage for the task, language and consequence level.
5. Describe every route as a contract, including where it runs and what it keeps.
6. Score only allowed routes on cost, latency and quality.
7. Compare cost per outcome, not price per token.
8. Draw fallbacks only from the allowed set; never widen permission.
9. Define what happens when no allowed route is available.
10. Record the allowed set, the chosen route and why it was chosen.

The result: routing you can defend

Model routing done well cuts cost and latency meaningfully. Done as a single score, it quietly turns policy into a preference that a good enough number can override. The fix isn't a smarter scorecard. It's a strict order: settle what you can without a model, decide which routes are allowed, then optimize among them, and keep every fallback inside the same boundary.

That order gives you the savings and an answer to the question that matters most after an incident: why was this model allowed to see this case?