“Model routing” comes up in almost every LLM project we’ve seen at clients over the past few months, and every team means something different by it. One uses a classifier to send simple requests to a small model and save money. The next has a proxy that falls back to a second provider when it gets an HTTP 429. A third runs a gateway that never lets prompts containing customer data reach an external provider in the first place. All three say “we route”, and in a joint workshop they talk past each other.
We wanted to sort this out for ourselves. Clients ask us which approach fits them, and we’d like a better answer than “it depends”. We also route ourselves: in the Trade Fair Data Platform, which keeps more than 650 trade fairs up to date with two AI agents, and in the financial sector, where we work with Gravitee. While sorting it out, we noticed that in our own project we had made one of these decisions on the wrong criterion for a long time.
Three questions, one term
The routers we’ve seen in client projects each answer one of three questions:
- Is the request allowed to go there? This is governance: what data is in the prompt, which models are approved for it, and has the team already used up its budget?
- Which model? This trades quality against cost: is a small model good enough, or does the task need the big one?
- Which provider? This is about availability. Many models are offered by several providers or in several regions, and not all of them respond reliably and fast enough at all times.
A request goes through the decisions in this order. Anything that fails the first question doesn’t need a model choice, and the smartest model choice is worthless if the provider is currently returning 503s.
At conferences and in client conversations, almost everything revolves around the second question. In our experience, the first and third more often decide whether a system actually works in production.
If you host models yourself, there’s a fourth question: which replica on which GPU gets the request? In Kubernetes, projects like llm-d and the Gateway API Inference Extension handle that. It’s a topic of its own and out of scope here.
The approaches at a glance
For orientation, here are the approaches we’ve come across in projects. The following sections go through them in this order.
| Approach | Answers | Where it lives | Makes sense for |
|---|---|---|---|
| Policy routing | Is the request allowed there? | API gateway, e.g. Gravitee | regulated environments, multiple teams |
| Static per task | Which model? | Application config | pipelines with known steps |
| Cascade | Which model? | Application | tasks with verifiable results |
| Classifier / learned router | Which model? | Router service, e.g. vLLM Semantic Router, RouteLLM, Bedrock Intelligent Prompt Routing | open-ended input at high volume |
| Provider routing | Which provider? | Gateway, aggregator, or client library like LiteLLM | every system in production |
Is the request allowed there: routing as governance
In the financial sector we work with Gravitee, and there routing has little to do with cost optimization. The questions are: which application may use which model? What happens to personal data in the prompt? Who consumed how many tokens, and can that be charged to a cost center? Up front: we’re a Gravitee partner, so this section isn’t neutral.
Gravitee’s LLM Proxy puts a single front door in front of multiple providers. Applications call an alias, and the gateway decides which backend sits behind it. Token-based rate limiting, PII filters and guardrails apply before a request reaches the model, and every request records which model answered with how many input and output tokens.
Which model a request gets is decided by the client, through the model name, an alias, or a policy that evaluates something like a header. With version 4.13, Gravitee says it is adding budget-based routing: as long as a consumer has budget left, their requests go to the best models, and as the budget shrinks, to cheaper ones. The documentation doesn’t describe choosing a model based on the content or difficulty of a request. If you need that, you put your own router behind the gateway.
Technically, this is API management applied to LLM traffic, the same thing we described for tool calls in the MCP Maturity Model. The gain is a clean split of responsibilities: the product team decides on model choice inside its application, while the platform team and compliance define at the gateway what’s allowed, without touching every application. When a model gets blocked or a provider replaced, no application needs to be redeployed.
In regulated environments, the place of processing is a hard exclusion criterion: a request that has to stay in the EU must not reach an endpoint outside it, regardless of price or latency. What often gets overlooked is that the gateway and the model each have their own region. An EU gateway in front of a model in us-east-1 doesn’t meet the requirement. Fallbacks, too, may only run within the allowed pool. If the EU endpoint goes down, an error is the right response, and a silent switch to another region is the wrong one. And data residency isn’t sovereignty: a US provider with a data center in Frankfurt is still subject to the CLOUD Act.
If you start with model choice in a regulated environment and push governance to later, you end up building it into every application separately, and that gets more painful with each new one.
Which model: static, cascade, or learned
This is the question that gets the most attention. We’ve seen three approaches, from simplest to most involved.
Static is better than its reputation
The least exciting option assigns a model per task: summarization gets the small one, contract review the big one. It looks unambitious, but for backend pipelines it’s usually the right call, because you already know the steps and how hard they are at design time. A router that re-evaluates this for every request at runtime will at best make the same decision.
What matters is where the mapping lives. If the model name is hardcoded in twelve files, every model change becomes a small project. An alias per task (extraction, summary, review), resolved in exactly one place, takes an hour to set up and turns the next model change into a config change.
Cascades: escalate when the result isn’t good enough
A cascade tries the cheap option first and escalates when the result falls short. In the trade fair platform we do this one level up, with agents instead of individual models: the primary agent processes the source material in a single pass under a fixed budget per run. If required fields are still missing afterwards, a fallback agent takes over and searches iteratively and in a targeted way. This now also covers the roughly 20% of trade fairs that previously couldn’t be processed at all.
It works because the escalation condition is hard: either all required fields in the schema are filled or they aren’t. Many cascades fail right here. If you ask the model whether it’s confident in its answer, you get an answer, but not a reliable signal. Without a verifiable condition such as schema validation, tests or required fields, we wouldn’t build a cascade.
The chain also needs an end. Ours deliberately has two links and a cap on iterations. A cascade without a stop condition is an infinite loop that sends you an invoice.
Learned routers: impressive numbers, hard to transfer
Learned routers estimate per request whether a small model is good enough. The simpler variant is a small classifier that detects domain or intent and picks the model through a policy, the way the vLLM Semantic Router does. The more ambitious one predicts per request how well each candidate would answer. The best-known open approach is RouteLLM from LMSYS: the team reports over 85% cost savings at 95% of GPT-4 quality on MT Bench, 45% on MMLU and 35% on GSM8K. AWS offers Bedrock Intelligent Prompt Routing, a managed router that picks within a model family, and cites up to 30% savings.
These numbers transfer poorly to your own project, for two reasons. First, they’re measured against “always use the biggest model”. If you already map models statically per task, you start from a different baseline, and the savings shrink accordingly. Second, the values vary a lot by dataset: with RouteLLM, the gap between 85% and 35% is just a different benchmark. What a router does for your requests only shows up when you measure it on your requests, and few teams have the evaluation set for that.
Independent comparisons are more sober than vendor claims. In early 2026, LLMRouterBench compared ten routing methods on over 400,000 instances with 33 models. Several of them, including a commercial router, didn’t reliably beat the best single model and didn’t cut costs without losing quality.
Then there’s a downside no benchmark captures: the same question gets answered by the small model today and the big one tomorrow. In a chat, nobody minds. In a pipeline whose results someone has to trace, it’s one more variable in every debugging session.
We don’t run learned routers in production so far. We see them making sense where input is open-ended and volume is high enough to justify the evaluation work: chat assistants, support, internal search.
The cache makes model switches expensive
Cascades and learned routers have an opponent that’s missing from many router comparisons: the prompt cache. It only applies to the model that wrote it. At Anthropic, cached input costs a tenth of the normal price, and writing the cache costs 1.25x. If a router switches models in the middle of a long agent session, the entire context so far gets reprocessed at full price, and that can exceed what the cheaper model saves. The longer the session has been running, the more expensive the switch.
The picture isn’t entirely one-sided, though. LiteLLM measured on its own gateway traffic that the previous model’s cache was still warm in 97.4% of 4,684 switches back, and router plus caching came out cheaper than caching on a fixed model. The same measurement shows what happens without caching: a router with caching turned off cost about four times as much. These are vendor numbers, but they show that routing and caching have to be planned together.
Our takeaway: decide once per task or session, then stay with the model. A switch fits where the context starts fresh anyway, for example when a sub-agent starts or after the history so far has been summarized.
In agent systems, the decision is therefore increasingly moving into the harness, the orchestrator that drives the steps and sub-agents. It’s the only place that knows how long the context already is, whether the cache is warm, and which step comes next: a review, a web search, a refactoring. A gateway only sees the individual request. Factory makes exactly this argument and reports 58% lower costs compared to “always the frontier model”, again a vendor number. What we find more interesting: model choice per step in the harness is usually a static mapping by task type again, just in the place that knows about the cache.
Often you don’t need to switch models at all. Since GPT-5, ChatGPT routes between a fast answer and longer thinking, and with many current models you set the reasoning effort through an API parameter. What used to take two models and a router can now often be done with one model and two settings. How a changed setting affects the cache differs by provider and is worth checking beforehand.
Which provider: what we changed
Many models are available from several providers: directly from the vendor, through the big clouds, through inference services and aggregators. The weights are the same; price and capacity differ.
In the trade fair platform, we initially routed by price. That seemed reasonable, since the pipeline runs in the background and nobody is waiting for a response. In practice, we regularly ended up with a provider that was running a promotion. Promotions attract load, and precisely during those phases the provider was overloaded. So the cheapest provider was least reliable exactly when routing favored it. Requests hit timeouts and rate limits, runs had to be repeated, and trade fairs stayed incomplete longer than we liked.
Today we route by reliability and throughput. The price difference per run is smaller than the cost of a failed run: the retry, the delay until the data is current, and the noise in monitoring that buries real errors. In hindsight, we should have configured it this way from the start.
The switch isn’t hard. Gateways, aggregators and client libraries come with the necessary settings: sorting by price, throughput or latency, fallback lists, cooldowns for endpoints that are currently failing. The problem is the default. It’s often set to price, or gets set that way during setup because it seems obvious, and then nobody looks at it again until the first day with load spikes.
Fallbacks between providers come with one caveat: they’re only harmless if it’s really the same model answering. If the fallback switches to a “similar” model from another vendor, your application’s behavior changes. That’s a model choice and needs its own evaluation. Even with the same model, providers sometimes differ in details like maximum context length or quantization, which you should check once before a provider goes on the fallback list. In regulated environments, the rule from above applies too: only providers and regions from the allowed pool belong on the list.
Where we’d start
When someone asks us today where to start with model routing, we recommend this order. It’s based on what pays off most in production, which is why it differs from the numbering above.
- Measure first. Collect traces with model, cost, cache hit rate and task success. Without that baseline, you can’t judge later whether a router helps.
- Governance early, if you work in a regulated environment or multiple teams use LLMs. A gateway is easy to put in place at the start; later you have to touch every application individually.
- Provider routing next, with fallbacks and a default based on reliability. In production, this does more than any classifier.
- Static model choice per task, with aliases configured in one place.
- Cascades only where the result can be verified, and always with a fixed cap.
- Learned routers only once you have an evaluation set and the volume. Then measure them against your static mapping with a warm cache, not against “always the biggest model”. A fixed set of prompts with expected routing belongs in CI, because managed routers change their model pool without you deploying anything.
For our own projects, we’ve taken two things from this: we choose providers by reliability rather than price by default, and model names live in configuration, not in code. For everything else, a simple exercise helps: write down which of the three questions your router is supposed to answer before you pick one. If the answer is “all of them”, check which ones your tool really answers and which only on the side. A gateway brings governance and fallbacks, a cloud provider a learned router for its own catalog. You don’t necessarily need three products for that, but you do need three decisions that are configured and tested separately.