How AI Agents Choose Which Model to Use

Why Single-Model Agents Are Inefficient

A typical agent workflow includes many turns of varying complexity. Some turns involve complex multi-step reasoning, strategic planning, or nuanced content generation. These turns genuinely benefit from the full capability of a frontier model. Other turns involve simple classifications, data extraction, format conversions, or routine tool parameter generation. Using a frontier model for these simple turns wastes money and adds unnecessary latency.

Consider a research agent that processes 40 turns to complete a task. Perhaps 5 of those turns involve complex analysis and synthesis that requires a frontier model. The other 35 turns are routine operations: generating search queries, extracting key facts from search results, classifying relevance, and formatting output. If the frontier model costs ten times more per token than a mid-tier model, running all 40 turns on the frontier model costs roughly 5x more than routing the 35 routine turns to the cheaper model.

The quality impact of routing is usually negligible for routine turns. A smaller model can generate a web search query just as well as a frontier model. It can extract a date from a text passage, classify a document by topic, or format data into JSON with comparable accuracy. The frontier model only adds meaningful quality for turns that require sophisticated reasoning, handling of ambiguity, or creative problem-solving.

Routing Strategies

Rule-based routing assigns models based on predetermined rules. Tool calls always go to the mid-tier model. Planning steps always go to the frontier model. Classification tasks always go to the small model. Rule-based routing is simple to implement, easy to understand, and predictable. The downside is that rules cannot adapt to the actual difficulty of each specific turn, so some complex tool calls get routed to an insufficiently capable model while some simple planning steps get routed to an unnecessarily expensive one.

Classifier-based routing uses a lightweight model to assess the complexity of each turn before routing it. The classifier examines the current context and the expected action, estimates the difficulty, and routes accordingly. This approach adapts to the actual content of each turn rather than relying on static rules. The overhead is one additional small-model call per turn, which is typically inexpensive (fractions of a cent) and fast (milliseconds). The accuracy of routing depends on the quality of the classifier, which improves with training data from historical agent interactions.

Adaptive routing starts with a smaller model and escalates to a larger one when the smaller model signals uncertainty. If the smaller model generates a response with low confidence, requests clarification, or produces output that fails validation, the system automatically retries with a more capable model. This approach minimizes cost by defaulting to the cheapest model that might work, only paying for the expensive model when the cheap one is demonstrably insufficient.

Speculative routing runs both a small and large model simultaneously and uses the small model result if it meets quality thresholds. The large model result is only used when the small model falls short. This approach optimizes latency (the result is available as soon as the faster small model completes) at the cost of running both models in parallel. It is economical when the small model is sufficient most of the time, making the wasted large model calls infrequent.

Model Selection Criteria

Task complexity is the primary criterion. Complex reasoning tasks, multi-step planning, ambiguity resolution, and creative generation require larger models. Simple extraction, classification, formatting, and routine tool calls work well with smaller models. The boundary between "complex" and "simple" is fuzzy and varies by domain, which is why classifier-based and adaptive routing outperform rigid rules.

Output type influences model selection. Code generation benefits from models specifically trained on code (or code-focused configurations). Long-form writing benefits from models with strong coherence over extended outputs. Structured data extraction (JSON, tables, lists) benefits from models with reliable formatting. Multilingual tasks benefit from models with strong cross-lingual capabilities. Matching the model to the output type improves quality without necessarily increasing cost.

Latency requirements constrain model selection. Interactive agents serving live users need fast responses, favoring smaller models that respond in under a second over larger models that take several seconds. Background agents processing tasks asynchronously can afford the additional latency of larger models because no user is waiting. The same agent system might use different models for the same task type depending on whether it is running interactively or in batch mode.

Cost budgets set hard limits on model selection. A task with a budget of $0.05 cannot afford 40 turns of a frontier model at $0.01 per turn. The routing logic must stay within budget while maximizing quality. This optimization problem, maximizing quality subject to a cost constraint, is the fundamental challenge of model routing. Different routing strategies solve it with different tradeoffs between simplicity, optimality, and implementation effort.

Multi-Provider Strategies

Production agent systems increasingly work with models from multiple providers. Anthropic Claude models might handle reasoning and analysis, OpenAI models might handle code generation, and Google Gemini models might handle multimodal tasks involving images or video. Multi-provider strategies increase resilience (if one provider has an outage, others can handle traffic) and let the system leverage each provider strengths.

The complexity of multi-provider setups lies in normalization. Different providers use different message formats, tool calling conventions, and response structures. The agent runtime needs an abstraction layer that translates between the universal internal format and each provider specific format. This abstraction also handles provider-specific features like prompt caching, extended thinking, and streaming, exposing them through a consistent interface.

Failover routing automatically switches to an alternative provider when the primary provider fails or responds too slowly. If Claude is the primary model but experiences elevated latency, the routing logic can temporarily redirect traffic to an alternative model. This failover should be transparent to the agent logic, which should not need to know or care which specific model is handling each turn.

Measuring Routing Effectiveness

Routing effectiveness is measured by comparing quality and cost against a single-model baseline. If routing achieves 95 percent of the quality at 30 percent of the cost, the routing is highly effective. If routing achieves only 80 percent of the quality at 50 percent of the cost, the quality degradation may not justify the savings. The acceptable quality-cost tradeoff depends on the use case: customer-facing applications tolerate less quality degradation than internal analysis tools.

A/B testing provides the most reliable measurement. Run a portion of traffic through the routed configuration and a portion through the single-model baseline. Compare quality metrics (task completion rate, accuracy, user satisfaction) and cost metrics (total spend, cost per successful task) between the two groups. This direct comparison reveals whether routing improves the overall system or introduces quality problems that offset the cost savings.

Model Evaluation and Benchmarking

Before deploying a routing strategy, each candidate model must be benchmarked on the specific task types the agent handles. General-purpose benchmarks provide a starting point, but they rarely correlate perfectly with performance on domain-specific tasks. A model that scores highest on academic reasoning benchmarks might underperform on practical customer support tasks, while a model with lower benchmark scores might excel at the structured extraction tasks your agent needs most.

Task-specific evaluation suites include representative examples from each task category the agent handles, with known correct answers or quality criteria. Running each candidate model through the evaluation suite produces accuracy scores, latency measurements, and cost data that enable informed routing decisions. The evaluation should be repeated whenever a new model version is released, since provider updates can change the performance characteristics that the routing logic depends on.

Shadow testing runs the new routing configuration alongside the existing one without serving the new results to users. Both configurations process the same tasks, and the results are compared offline. Shadow testing reveals quality differences, edge cases, and failure modes before they affect production users. It is the safest way to validate routing changes, though it costs double the compute during the testing period because both configurations run simultaneously.