Fine-Tuning Models for AI Agent Tasks

Why Agents Benefit Most from Fine-Tuning

Standard chatbot applications are forgiving. If the model produces a slightly different format or takes an unexpected approach, the user reads the response and moves on. Agent applications are not forgiving. When a model selects the wrong tool, the downstream system executes the wrong action. When it produces malformed JSON in a function call, the tool-calling framework throws a parsing error. When it makes an incorrect judgment about whether to continue or stop, the entire workflow fails.

These failures compound across multi-step tasks. An agent that makes 15 sequential tool calls with 90% accuracy per call has only a 20.6% chance of completing the full sequence without an error (0.9^15). At 95% accuracy, the completion rate jumps to 46.3%. At 99%, it reaches 86.0%. Fine-tuning is the most direct path to that last 5-10% of per-step accuracy that determines whether an agent is usable in production.

The three agent-specific behaviors that benefit most from fine-tuning are: tool selection (choosing which function to call given a situation), argument formatting (producing syntactically valid, semantically correct parameters), and trajectory planning (deciding when to call another tool, when to ask for clarification, and when to return a final answer).

Training Data for Agent Fine-Tuning

Agent training data looks different from standard chatbot training data. Instead of single-turn question/answer pairs, you need complete multi-turn trajectories that show the model navigating through an entire task. Each trajectory includes the user request, one or more tool calls with their results, any intermediate reasoning, and the final response.

A single agent training example might contain 8-12 messages: user request, assistant tool call, tool result, assistant reasoning, assistant second tool call, tool result, assistant synthesis, final response to user. This is significantly longer than a standard chatbot example, which means you need fewer total examples (each one carries more learning signal) but each example costs more tokens.

The distillation approach is the most practical way to collect agent training data. Run your agent in production using a strong model like GPT-4o or Claude as the backbone. Log every successful interaction, including the complete message history with tool calls and results. Filter for interactions where the task completed successfully (confirmed by user feedback or automated checks). Use these successful trajectories as training data for a smaller, cheaper model.

This approach has a significant advantage: the training data captures real-world input distributions, including the messy, ambiguous requests your agent actually receives. Synthetic agent data generated from hypothetical scenarios misses the specific edge cases, typos, and unusual requests that cause failures in production.

Collecting tool selection data requires covering your full tool set comprehensively. If your agent has 12 tools, your training data should include examples of each tool being used correctly, examples where the model must choose between two or more plausible tools, and examples where no tool is appropriate and the agent should ask for clarification or respond directly. Imbalanced tool coverage, where 80% of examples use 3 of your 12 tools, teaches the model to default to those 3 tools and ignore the others.

Include failure recovery examples. Agents encounter tool errors, timeouts, and unexpected results regularly. Your training data should show the model handling these gracefully: retrying with different parameters, trying an alternative tool, or explaining to the user what went wrong and what to do next. Models that only see successful trajectories during training do not know how to recover from failures, and they hallucinate tool results instead of acknowledging the error.

Fine-Tuning for Structured Output

Agent tool calls require valid, parseable structured output. A function call with an extra comma in the JSON, a missing required parameter, or a parameter of the wrong type crashes the agent loop. Base models produce valid JSON roughly 90-95% of the time when prompted carefully. Fine-tuning pushes this above 99%.

The key to structured output fine-tuning is consistency in your training data. Every tool call in every training example must use exactly the same parameter names, types, and formatting. If your search tool accepts a "query" parameter, never use "search_query" or "q" in any example. If dates should be ISO 8601 format, every date in every example must follow that format. The model learns the patterns it sees, and any variation in training data becomes variation in production output.

Include examples with various argument complexities: simple calls with one parameter, complex calls with nested objects, calls with optional parameters both included and omitted, and calls with array parameters. This teaches the model the full grammar of your tool API rather than just the most common patterns.

For models that support native function calling (like Mistral and GPT-4.1), fine-tune using the function-calling format rather than asking the model to produce JSON in plain text. Native function calling uses the model's built-in structured generation capabilities, which are more reliable than free-form text generation. Your training examples should use the "tool_calls" message format rather than embedding JSON in the assistant's content field.

Multi-Step Reasoning and Planning

Complex agent tasks require the model to plan a sequence of actions, execute them in order, adapt the plan based on intermediate results, and decide when the task is complete. This is where the difference between a prompted and fine-tuned model is most visible.

A prompted model might successfully handle a 3-step task but lose coherence on a 10-step task because it cannot hold the full plan and current state in its effective context window. A fine-tuned model has internalized common task patterns and does not need to reason through the plan from scratch each time. It recognizes "I just got search results, now I should filter them" as a pattern it has seen in training, rather than reasoning about it from first principles.

Training data for multi-step reasoning should include tasks of varying complexity. Include some 2-step tasks, some 5-step tasks, and some 10+ step tasks. If you only train on simple tasks, the model will try to complete complex tasks in too few steps, skipping necessary intermediate actions. If you only train on complex tasks, the model will over-complicate simple requests.

Include examples where the plan changes mid-execution. A real-world agent often needs to adapt: a search returns no results, so the agent broadens the query. A tool returns an error, so the agent tries an alternative approach. The first result reveals that the original question was based on a false assumption, so the agent corrects the user. These adaptive behaviors are difficult to teach through prompting because they require the model to deviate from its initial plan, which conflicts with the instruction-following behavior that prompting reinforces.

Distillation: Building Small, Fast Agent Models

The production pattern for agent fine-tuning is distillation: use a large, expensive model to generate high-quality trajectories, then fine-tune a smaller model to reproduce those trajectories. The smaller model serves as the production backbone with lower latency and cost.

A typical distillation pipeline: deploy your agent with GPT-4o or Claude Sonnet as the LLM backend. Collect 2,000-5,000 successful task completions over a few weeks. Clean and filter: remove trajectories with errors, excessive tool calls, or poor final answers. Fine-tune Llama 3.1 8B or Mistral 7B with QLoRA on the cleaned trajectories. Evaluate the fine-tuned model on a held-out set of tasks. Deploy the fine-tuned model as the production backend.

The cost savings are substantial. GPT-4o costs $2.50/$10 per million input/output tokens. A self-hosted fine-tuned 8B model costs roughly $0.05-0.10 per million tokens at scale. If your agent processes 100,000 tasks per month with an average of 5,000 tokens per task, switching from GPT-4o to a fine-tuned 8B model saves $3,000-6,000 per month in inference costs. The fine-tuning itself costs under $30.

The quality trade-off depends on task complexity. For routine agent tasks (CRM updates, data retrieval, form filling), a fine-tuned 8B model matches GPT-4o quality because the task patterns are well-represented in the training data. For complex reasoning tasks with many possible approaches, the fine-tuned model will be noticeably weaker. The common solution is a routing approach: use the fine-tuned model for routine tasks and fall back to GPT-4o for tasks the fine-tuned model flags as uncertain.

Evaluation Metrics for Agent Models

Task completion rate: The percentage of end-to-end tasks the model completes successfully. This is the metric that matters most for production agents. Measure it against a diverse test set of 200+ tasks covering your full use case range.

Tool selection accuracy: What percentage of tool calls use the correct tool? Measure this separately because a model might complete tasks by making extra calls (trying multiple tools until one works), which masks the underlying selection quality.

Parse success rate: What percentage of tool calls produce syntactically valid, semantically correct arguments? Track both syntax errors (invalid JSON) and semantic errors (valid JSON but wrong parameter values).

Average steps per task: How many tool calls does the model make to complete a task? Compare against the optimal number of steps (determined by expert review). A model that consistently takes more steps than necessary is wasting tokens and time. A model that takes fewer steps might be skipping necessary verification.