Key takeaways
- Model routing can lower inference costs and latency by reserving frontier models for complex tasks while directing routine or specialised work to smaller, cheaper models.
- The router becomes a sensitive AI control plane, creating new risks around privacy, latency, quality drift, security and dependence on an external provider.
- Independent routers have an opportunity to offer neutrality across competing models, but they will need transparent economics and something resembling “best execution” to prove that their recommendations serve the customer.
For the first phase of generative AI, the default strategy was simple: use the most powerful model available. Makes sense, no?
That made sense when companies were experimenting and usage was relatively limited. It makes less sense as AI moves into production, agents make hundreds of calls to complete a single workflow and inference becomes a meaningful operating expense. Tokens became a concern, and suddenly which model you choose for a task has a big implication on cost and capacity usage.
More importantly, not every problem requires a model trained on half the internet. A customer asking where to find an invoice does not need the same intelligence as an engineer debugging an unfamiliar codebase. Extracting a date from a contract does not require frontier-level reasoning. In some specialised domains, a smaller model trained or fine-tuned for the task may be faster, cheaper and more accurate than a much larger general-purpose model.
This is the theory behind model routing: instead of selecting one model for an application, choose the best model for each request.
The biggest model is not always the best model
Model performance is not a single linear scale. Models have different strengths in coding, mathematics, retrieval, writing, tool use, languages, multimodal reasoning and structured output. They also vary dramatically in price, latency, context windows and privacy guarantees.
Larger models are generally more capable across a broad range of tasks, but breadth can be a disadvantage when an application requires narrow, predictable behaviour. A general model may draw on irrelevant associations, provide unnecessarily elaborate answers or introduce information that was not contained in the company’s approved data.
This does not mean larger models inherently hallucinate more. In many benchmarks they hallucinate less. The problem is that a frontier model’s additional knowledge does not guarantee greater accuracy inside a specific workflow. If a compliance assistant is expected to answer exclusively from an internal policy manual, its knowledge of the wider internet may be irrelevant and potentially distracting.
Accuracy in production is therefore not simply about choosing the model with the highest benchmark score. It is about matching the model, context, tools and guardrails to the task.
A well-designed AI system might use:
- A small model to classify a request or extract structured data.
- A domain-specific model for legal, medical or financial terminology.
- A fast general model for summarisation and routine communication.
- A reasoning model for complex analysis and planning.
- A private or locally hosted model when sensitive information is involved.
- Multiple models when the output needs to be checked or compared.
Model routing turns that selection process into infrastructure.
What a model router actually does
A model router sits between an application and a pool of models. It analyses each request and selects a model according to variables such as task type, difficulty, expected quality, latency, cost, privacy, context length and tool compatibility.
The simplest routers are based on rules. A company might send code to a coding model, personally identifiable information to a private deployment and all requests longer than a defined threshold to a model with a larger context window.
More sophisticated routers use classifiers or embeddings to identify the type and complexity of a request. Some estimate the probability that each available model will answer correctly. Others use a cascade: begin with a cheaper model, measure its confidence or validate its output, and escalate to a more powerful model only when necessary.
The latest generation is moving towards agentic routing. Rather than making a single choice before inference, these systems can call one model, examine the result and decide whether to escalate, ask another model or combine several answers. A 2026 survey of dynamic model routing distinguishes between routing based on difficulty, preferences, clustering, uncertainty and reinforcement learning. It argues that carefully designed routers can outperform a single powerful model by combining specialised capabilities.
This is different from a mixture-of-experts model. In a mixture-of-experts architecture, routing happens internally between parts of one model. Model routing chooses between independently trained models, potentially operated by entirely different providers.
Routing is becoming part of the AI stack
Model routing started as an optimisation technique, but it is rapidly becoming a product category.
AWS Bedrock Intelligent Prompt Routing routes between models within the same family. AWS says it can reduce costs by up to 30 percent without compromising accuracy, although the service has limitations, including its reliance on the router’s original training data and restrictions on which models can be combined.
Microsoft Foundry’s Model Router analyses the full request, including conversation history, system instructions and tools. Customers can optimise for cost, quality or a balance between the two, while limiting the models eligible to receive a request. Microsoft says the routing decision adds only a small fraction of overall inference time and that prompts are not stored by the router.
OpenRouter provides access to models from multiple vendors behind a common API. Its automatic router classifies prompts by task and can adjust the trade-off between cost and quality. It also supports session stickiness, keeping a conversation on the same model to preserve behaviour and improve prompt caching.
Snowflake is now adding dynamic model routing to Cortex AI Gateway. The system will select models based on quality, speed, customer preferences and cost, while keeping governance and usage controls within Snowflake. In one internal evaluation, Snowflake said an agent building a dbt pipeline achieved up to three times greater token efficiency than a frontier-model-only approach without reducing quality. In another test, engineering teams completed the same number of pull requests with 25 percent greater token efficiency. These are vendor-reported results, but they illustrate the potential economics.
NVIDIA’s NeMo Switchyard takes a different approach. It is an open-source library intended to help enterprises build and operate their own routing systems. The strategic logic is clear: if companies use more small, specialised and open models, they still need infrastructure to decide which model should handle each task.
There is also a growing independent routing layer. Martian, Not Diamond and OpenRouter were among the earlier companies in the category, while TrustedRouter, Sapiom and Concentrate AI are attacking different parts of the market. Some optimise model selection, some aggregate access to inference providers, and others focus on cost management, privacy, failover or operating open models. TrustedRouter, for example, is positioning privacy as a core differentiator, while Sapiom combines routing with infrastructure for running agents.
The distinction matters. A gateway can provide one API and switch providers when one is unavailable without intelligently choosing the best model for the task. A true router must predict which model will produce the best outcome under a set of constraints.
Can a model company be a neutral router?
The major AI labs are also likely to develop more routing internally. OpenAI, Anthropic and Google already operate families of models with different combinations of speed, cost and reasoning capability. Over time, users may interact with a single interface while the provider automatically decides whether a request needs a small model, a frontier model or a specialised reasoning system.
This will make model selection largely invisible to the user, but it is unlikely to create genuinely open routing. Anthropic may route between its own Haiku, Sonnet and Opus models, but it has little incentive to conclude that a particular request would be better handled by OpenAI, Gemini, AI21 or an open-source model. The same applies to every major model provider.
Cloud platforms occupy a slightly more neutral position because they already offer models from multiple vendors. Even then, AWS, Microsoft, Google and Snowflake have commercial relationships, infrastructure economics and strategic products that may influence which models they support, how they rank them and what information they expose to customers.
This creates an opening for independent routers. Their promise is not simply better model selection, but credible neutrality across competing providers. Yet neutrality is difficult to prove. A router may receive preferential pricing, rebates or commercial incentives from model providers. It may optimise for its own margin rather than the customer’s outcome, or favour models that generate more routing revenue.
The model-routing market may therefore need something resembling the “best execution” principle in financial markets. Customers should be able to see which models were considered, why one was selected, what alternatives would have cost and whether the router receives any economic benefit from its choice. Without that transparency, a supposedly intelligent router risks becoming an opaque distribution channel.
This also sharpens the investment thesis. The defensible layer is not just the routing algorithm. It is the combination of neutrality, proprietary performance data, transparent economics, enterprise trust and the ability to evaluate models against each customer’s real workloads.
The economic case for routing
The clearest benefit is cost. If 70 percent of an application’s requests can be handled by a model costing one-tenth as much, routing can transform the unit economics.
The savings could become even more important with agents. A chatbot may make one model call per user message. An agent can make dozens or hundreds of calls as it plans, searches, retrieves data, invokes tools, checks results and corrects mistakes. Even a modest saving per call compounds quickly.
Routing can also improve latency. Small models often respond faster, and routine requests do not need to wait behind computationally expensive reasoning. AWS reported latency benefits alongside cost reductions in its internal routing evaluations, although the results varied substantially by model family and workload.
Reliability is another advantage. A router can move traffic when a provider is unavailable, rate-limited or performing poorly. It can also support data residency and compliance policies, sending some requests to approved regional or private models while using public frontier models for less sensitive work.
Finally, routing reduces model lock-in. Applications can retain a stable interface while the router changes the underlying models as prices, capabilities and availability evolve.
The router creates a new point of failure
The benefits are real, but model routing is not free.
The first issue is privacy. An external router may see every prompt, system instruction, uploaded document and model response. That potentially gives a third party access to some of the organisation’s most sensitive operational data.
Even when a routing provider promises not to store prompts, the company must still understand where the request is processed, which metadata is logged and which underlying model provider eventually receives the data. Routing multiplies the number of possible data paths, which can make compliance more complicated rather than less.
The second issue is latency. A routing decision takes time. A lightweight classifier may add only milliseconds, but a router that calls another LLM to understand the request could add meaningful delay and additional token cost. Cascades are even more complicated because the system may generate an initial answer, evaluate it and then repeat the task with a stronger model.
There is also a quality measurement problem. The router must predict which model will answer correctly before it sees the answer. Benchmarks can help, but every company’s data and workflows are different. A router trained on generic questions may be poor at distinguishing between easy and difficult requests in insurance underwriting, cybersecurity or enterprise software support.
Routing errors can also be difficult to diagnose. When an application uses one model, changes in behaviour are relatively easy to trace. With dynamic routing, two apparently similar requests may be handled by different models. A provider may silently add a new model, change routing weights or remove an older one, producing quality drift without any changes to the application itself.
The router can also become a target. Researchers from the Hebrew University of Jerusalem and Cornell demonstrated that specially constructed strings could manipulate several routers into sending requests to more powerful and expensive models. Their research describes this as an attack on the integrity of the AI “control plane.”
For a high-volume consumer service, forcing the router to select the most expensive model could become a form of economic denial-of-service attack.
Build, buy or stay with one model?
There is no universal answer. The right architecture depends on volume, sensitivity and how much quality varies across tasks.
External routers make the most sense for startups that want immediate access to multiple models, need provider failover and do not yet have enough traffic or evaluation data to justify building their own system. They simplify integration but require trust in another company’s infrastructure and routing decisions.
Cloud-native routers from AWS, Microsoft and Snowflake provide better integration with existing security and governance systems. However, they can limit model choice and create a new form of platform lock-in.
Self-hosted routers provide the most control. Companies can define their own policies, keep sensitive prompts inside their environment and optimise against proprietary evaluations. The trade-off is that model evaluation, monitoring and routing logic become a permanent engineering responsibility.
For many companies, the best starting point will be less glamorous: route by workflow rather than by individual prompt. Customer support, document extraction, coding and strategic analysis can each be assigned a default model based on testing. Dynamic routing can then be added only where the variance in complexity and cost justifies it.
The Israeli angle
Israel already has an interesting position in this emerging layer.
AI21 Labs is developing Maestro as an optimisation framework for production AI agents. Maestro combines model routing with prompt compression, task decomposition and other inference-time techniques. AI21’s pitch is not simply to choose a cheaper model, but to optimise the entire cost, latency and accuracy frontier for an agentic workflow. Its documentation allows users to select a model directly or let Maestro choose among AI21 and third-party models.
The Hebrew University research on attacks against routers is equally relevant. As routers become responsible for cost, privacy and model selection, protecting the routing layer could become a distinct cybersecurity market. Israeli companies have historically been strong when new infrastructure creates a new control plane and therefore a new attack surface.
There is room for more. A defensible Israeli routing startup is unlikely to win by offering another generic API to the same collection of models. The larger opportunity may be in enterprise-grade routing that can operate inside a customer’s environment, learn from proprietary evaluations, enforce security policies and optimise multi-step agent workflows rather than isolated prompts.
Routing becomes part of the product
Model routing pushes AI models further towards becoming interchangeable infrastructure. If an application can switch between OpenAI, Anthropic, Google, AI21 and open models without changing its interface, the value moves away from the model endpoint and towards the layer that understands the task, owns the evaluation data and decides how intelligence should be delivered.
That is why Snowflake, Microsoft, AWS, NVIDIA and a growing number of startups are all moving into routing. They are not only trying to reduce inference costs. They are competing to become the control plane for enterprise AI.
The major model providers will build increasingly sophisticated routers, but most will remain confined to their own product families. Cloud platforms will offer broader choice, although not necessarily complete neutrality. Independent routers therefore have an opportunity, but only if customers trust both their technical decisions and their economic incentives.
The winners may not be the companies with access to the largest number of models. They will be the ones that can reliably answer a much harder question: which model, using which context and tools, is most likely to produce the right outcome for this specific task?
That is the real promise of model routing. Not cheaper intelligence at any cost, but a more deliberate way to decide how much intelligence each problem actually needs.
He is a former General Partner at Google Ventures (GV) in Europe, former head of Google for Entrepreneurs in Europe, and founding head of Campus London, Google's first startup hub. Eze writes on Israeli tech, venture capital, artificial intelligence, and founder strategy.
He is also the founder of Techbikers, a nonprofit that brings together the startup ecosystem on cycling challenges in support of Room to Read.
- The Right Model for the Job - September 1, 2026
- Weekly Firgun Newsletter – August 28 2026 - August 28, 2026
- The Israeli Startup’s US Playbook - August 26, 2026
