Best Self-Hosted Models for AI Agents

What Makes a Model Good for Agents

Agent workloads place specific demands on language models that differ from simple chat or summarization. The critical capabilities are:

Tool calling reliability: Agents use function calling to interact with external systems (APIs, databases, file systems, web browsers). The model must generate syntactically correct function call JSON consistently, handle multi-tool scenarios, and know when to call a tool versus respond directly. Models that produce malformed JSON or hallucinate tool names cause agent failures.

Instruction following precision: Agents operate from detailed system prompts that define their behavior, constraints, and available tools. The model must follow these instructions precisely, even when the user query might tempt it to deviate. Weaker models tend to ignore parts of long system prompts or blend agent instructions with user queries.

Reasoning capability: Agents frequently need to plan multi-step actions: deciding which tools to call, in what order, how to interpret results, and when to ask for clarification. This requires the model to reason about task decomposition, maintain awareness of what has been accomplished, and adjust plans based on intermediate results.

Context window utilization: Agent conversations accumulate tool call results, system messages, and multi-turn history rapidly. A model with a small effective context window loses track of earlier actions and repeats work or contradicts previous decisions.

Top Models for Agent Workloads in 2026

Llama 4 Scout (109B MoE, 17B active)

Scout is arguably the best self-hosted model for agents in 2026. Its 10 million token context window means it never loses track of long agent sessions. The MoE architecture keeps inference costs low despite the large total parameter count. Tool calling support was a focus of the Llama 4 training, and Scout handles multi-tool scenarios reliably. It runs on a single H100 or a high-memory Mac, making it accessible for serious agent deployments.

Mistral Small 4 (119B MoE, 24B active)

Mistral Small 4 combines strong reasoning with native tool calling support and image understanding. The multimodal capability is valuable for agents that need to process screenshots, documents, or visual information. Its coding ability makes it effective for code-writing agents. The MoE architecture provides good quality at moderate inference cost.

Llama 3.3 70B

The dense 70B model remains a workhorse for agent deployments that prioritize reliability over cutting-edge features. Its 128K context window is sufficient for most agent sessions. Tool calling works well with proper system prompt engineering. The large ecosystem of fine-tuned variants means you can find specialized versions optimized for specific agent patterns.

Qwen 2.5 72B

Qwen 2.5 deserves mention for its particularly strong tool calling implementation. The model was trained with extensive function calling data and handles complex multi-tool workflows reliably. It also supports structured output generation well, making it effective for agents that need to produce specific JSON schemas.

Small Models for Agent Routing

Not every part of an agent system needs a large model. Small models (3-8B parameters) excel at the routing layer, classifying user intents, extracting parameters, and deciding which tool to invoke. Llama 3.2 3B and Phi-3 Mini handle these lightweight agent tasks at extremely low latency, freeing the large model for complex reasoning steps.

Agent-Specific Considerations

Structured output: Use models that support JSON mode or guided decoding (available in vLLM) to guarantee valid tool call output. Freeform generation occasionally produces malformed JSON that crashes agent loops.

System prompt length: Agent system prompts are often 2,000-5,000 tokens long, containing tool definitions, behavior rules, and examples. Test your model with the full system prompt to ensure it does not degrade instruction following at that prompt length.

Thinking models: Chain-of-thought (thinking) models, which reason step-by-step before responding, generally produce better agent decisions but at higher latency. If your agent can tolerate 2-5 second response times, enabling thinking mode improves tool selection accuracy and plan quality.