What is an LLM router?
An LLM router is a system that inspects each incoming query and sends it to the most suitable large language model, deciding on task complexity, cost per token, latency budget, or required capability. It sits between the application and the model providers: product code asks one question, and the policy decides which model answers.
Routing became a practical concern once providers began shipping families of models at different price and speed points, a small fast model, a mid-tier model, and a frontier model, all exposed through roughly the same API shape. The router turns that menu into a decision made per request.
How an LLM router works
A router runs four stages in order: extract, score, dispatch, verify.
Extraction reads the request before any large language model sees it: message length, detected intent, language, whether attachments or tool calls are involved, and how many AI tokens the assembled prompt and its retrieved context will consume. Those features are the entire basis for the decision, so a router that only sees the raw user message will misjudge long conversations.
Scoring applies the policy: a rule table, a trained classifier, or a small model that predicts whether the cheap option will succeed. Most policies emit a predicted quality alongside a predicted cost, which is where a confidence score enters, since the router needs a number it can threshold.
Dispatch sends the request to the chosen provider with that provider's prompt template, because a prompt tuned for one model transfers imperfectly to another.
Verification closes the loop. The router checks the response for refusals, empty tool calls, or low confidence, and retries on a stronger model when the check fails. Without that stage, a mis-route stays invisible until a customer complains.
Types of LLM routers
Rule-based routing: Static conditions map query attributes to models, using language, channel, customer tier, or the presence of a tool call, which is easy to audit but drifts as intents change.
Classifier routing: A trained model predicts, before generation, which tier is likely to answer correctly, learned from labeled traffic and outcomes on past requests.
Cascade routing: The cheapest model answers first and the request escalates to a stronger one when a quality check fails, buying lower average cost with extra latency on hard queries.
Capability routing: Requests are matched to models by what those models can do, such as vision, long context, function calling, or a specific language, with price treated as a secondary input.
LLM router vs model gateway vs mixture of experts vs ticket routing
Four things get called routing, and buyers conflate them because each one picks a destination for a request. A model gateway brokers credentials, quotas, and failover for a request whose model has already been chosen. Mixture of experts picks specialist sub-networks inside a single model, invisibly, on every token generated. Ticket routing picks the human queue for a conversation that automation could not close. An LLM router picks which model answers, before generation, on grounds of predicted quality, price, and latency.
What it decides | Where it sits | What drives the decision | Who owns it | Choose it when | |
|---|---|---|---|---|---|
LLM router | Which model answers a request | Between the app and model providers | Query features, cost, latency, confidence | Platform or AI engineering | Traffic is mixed and one model fits most of it badly |
Model gateway | Which key, region, and fallback are used | In front of provider APIs | Quotas, availability, credentials | Infrastructure | You need one auth and failover surface |
Mixture of experts | Which sub-networks fire per token | Inside the model weights | Learned gating | The model provider | Never; it ships inside the model you buy |
Ticket routing | Which human queue owns the case | After automation hands off | Skills, language, priority, SLA | Support operations | A conversation needs a person |
If your cost curve is driven by sending easy questions to an expensive reasoner, you need the router. If it is driven by conversations landing on the wrong human team, you need ticket routing, and most support organisations have both problems running at once.
Why an LLM router matters for customer experience
Without a router, a deployment picks one compromise model and lives with it. Pick the frontier model and password resets get answered by the most expensive reasoner available, while the bill caps how much volume you can automate at all. Pick the small model and the hard cases, a billing dispute spanning three orders or an ambiguous refund policy, come back fluent, confident, and wrong, which customers experience as an accuracy failure.
Selective matching of query to model is one of the core techniques teams use to hold accuracy steady while automating more volume, and it is part of why lean CX teams can run automation at a unit cost that would otherwise be out of reach.
The tradeoff is genuine. Routing adds a decision before every answer, so it adds latency, a new failure surface, and a policy someone has to own. On low, uniform traffic that overhead buys nothing.
How is an LLM router measured?
Routers are judged on the gap between what they save and what they break. Four measures carry most of the weight: routed accuracy, meaning the share of requests answered correctly end to end; cost per resolved request; the escalation rate to a stronger model; and added routing latency, read at the tail rather than the mean.
The moving target underneath all four is model price. The Stanford HAI AI Index (2025) reported inference price declines ranging from roughly 9x to 900x per year depending on the task, so a cost gap that justified a routing policy one quarter can narrow before the policy is finished being tuned.
Scoring routed accuracy at volume usually means an LLM as a judge pass over sampled transcripts, calibrated against human review. Read that number beside the cost number, always. A router that pushes more traffic to the small model is only working if routed accuracy holds; when it slips, customers are paying for the savings.
How AI agents change LLM routing
An agent turns one customer question into many model calls: a plan, several tool calls, a retrieval step, a summarisation, a final reply. Each has a different difficulty profile, so routing moves from per-conversation to per-step. Choosing a tool argument is cheap classification work; drafting a refund explanation that cites the right policy clause is not.
Two consequences follow. The router gains agent state to route on, including which step of the loop is executing, how many retries have already fired, and how much context has accumulated. And a bad route compounds, because a weak plan at step one sends every later step down the wrong path, and the customer receives a confident answer built on a broken premise.
That compounding is why routing turns into an operational concern for teams running AI agents on tier 1 support: the cheap model carries the bulk of the loop, and the expensive one is held for the steps where being wrong is costly.
Choosing an LLM router
Start with coverage: which providers and model families the router can reach, whether self-hosted weights count as a destination, and whether adding a model is a config change or a code change.
Integration surface shapes the build. Streaming, function calling, and structured output behave differently across providers, so a router that normalises them saves you writing a translation layer per model.
Governance is where routers quietly fail. The policy is production logic, so it needs versioning, a review path, and an offline evaluation run before any change is promoted. A policy anyone can edit live will drift within weeks.
On compliance, two questions do most of the work. GDPR-driven buyers ask which provider processed a given request and in which region, so per-request model attribution has to be logged. ISO 42001 surfaces when procurement wants an AI management system documented.
The constraint teams underestimate is prompt maintenance: every destination model needs its own tuned prompt and its own regression set, so each model added carries a permanent upkeep cost.
LLM routers and support routing
Ticket routing assigns a case to the queue that can resolve it, and call routing does the same job for an inbound phone call. Both run once the system has accepted that a human is involved.
An LLM router runs earlier, inside the automated turn, and its settings change how often those two are needed at all. Better model selection on hard cases means fewer conversations reach a queue, while an over-aggressive cost policy pushes them there faster. Tuning the router without watching queue volume hides that link.
What does an LLM router mean in plain terms?
Think of an LLM router as the triage desk at a clinic. Everyone who arrives is seen quickly by someone cheap, most people leave with exactly what they needed, and the few who need the specialist are sent through with their notes already taken. LLM stands for large language model, so the full form here is large language model router.
Take the desk away and one of two things happens. The specialist sees everybody, the queue grows, and the bill becomes absurd. Or the junior sees everybody, and the serious cases leave reassured and unhelped.
The tradeoff is the triage step itself. It costs a moment of everyone's time and it can be wrong, and when it is wrong, the person who needed the specialist walks out with a confident answer from the wrong desk. That risk is what you accept in exchange for paying specialist rates only when they are warranted.
Common LLM router mistakes
Routing on message length as a stand-in for difficulty. Length correlates with typing habits, not with reasoning load: "can I get a refund on this?" is four seconds of typing and a policy question with three conditions attached.
Shipping without a verification stage. A cheap model that answers wrongly produces no error signal at all, since the response looks well formed and the request closes normally. The savings look real on the dashboard because the failures never got counted.
Tuning the policy against price alone. Cost per request is easy to instrument and moves immediately, so it becomes the target, while resolution quality falls somewhere nobody has joined back to the routing decision.
Freezing the policy after launch. Providers reprice, deprecate tiers, and ship new ones, so a policy tuned once becomes an increasingly inaccurate map of a market that kept moving.
What is an LLM router in simple terms?
An LLM router is a dispatcher for AI requests. It looks at each incoming query, judges how hard it is and what capabilities it needs, then sends it to the model best matched to that job. Simple questions go to a small, fast, inexpensive model, and genuinely difficult ones go to a stronger one.
What is the difference between an LLM router and a model gateway?
An LLM router chooses which model should answer a request based on the content and difficulty of that request. A model gateway handles the plumbing around a model that has already been selected: API keys, regional endpoints, rate limits, retries, and failover. Many production stacks run both, with the router deciding and the gateway delivering.
LLM routing vs mixture of experts: what is the difference?
LLM routing operates outside the models, choosing between separate deployed systems from different providers or size tiers on a per-request basis. Mixture of experts operates inside one model's weights, with a learned gate activating a few specialist sub-networks per token. You configure a router yourself; mixture of experts arrives as part of the model you licence.
Does an LLM router actually save money?
An LLM router saves money when traffic is genuinely mixed, meaning a large share of easy requests currently paid for at frontier-model rates. Savings come from moving that share down a tier while holding accuracy steady. On uniform or low-volume traffic, the engineering and maintenance overhead usually exceeds whatever the policy recovers.
How does an LLM router decide which model to use?
The router extracts features from the request, including intent, language, context length, tool requirements, and conversation history. A policy then scores those features, using rules, a trained classifier, or a small predictor model, and picks a destination. Well-built routers also verify the response afterwards and retry on a stronger model when the check fails.
Can an LLM router make answer quality worse?
An LLM router degrades quality whenever its difficulty prediction is wrong and no verification step catches the miss. The cheap model returns something fluent, the request closes, and the error reaches the customer unflagged. This is why routed accuracy has to be sampled and scored continuously alongside the cost figures the router was built to improve.