How to Test AI Agent Tool Calling and Function Use
Tool calling evaluation differs from general output evaluation because it has concrete, verifiable correctness criteria. Unlike open-ended text generation where quality is subjective, a tool call either matches the expected schema or it does not, either passes valid arguments or it does not, either selects the right tool or it does not. This makes tool calling one of the most testable aspects of agent behavior, and also one of the most impactful to get right because a malformed tool call produces zero useful output regardless of how good the reasoning was.
Test Tool Selection Accuracy
Tool selection testing verifies that the agent chooses the appropriate tool for a given user intent. This is essentially a classification problem: given an input, which tool (or combination of tools) should the agent invoke?
Build a test set of user requests paired with expected tool selections. Cover every tool in the agent's toolkit with at least five to ten test cases each, including cases where the intent is ambiguous or could reasonably map to multiple tools. Include negative cases where no tool should be called (the agent should answer from its training data alone) and cases where multiple tools should be called in sequence.
Measure tool selection accuracy as the percentage of test cases where the agent selected the correct tool. Break this down by tool to identify which tools are most frequently confused with others. Common failure patterns include: selecting a search tool when a database query tool would be more appropriate, selecting a write tool when the user only asked to read, and calling unnecessary tools for simple questions that require no external data.
For agents with many tools (ten or more), pay special attention to tools with overlapping capabilities. If your agent has both a "search documents" tool and a "search web" tool, test explicitly with queries that should route to each one, plus ambiguous queries where either could be reasonable. The distinction between these tools should be clear from their descriptions and the test cases should validate that the model interprets those descriptions correctly.
Include test cases where the user's request does not match any available tool. The correct behavior is for the agent to either ask for clarification or explain that it cannot perform the requested action, not to force-fit the request into an inappropriate tool. An agent that always calls a tool, even when none is relevant, has poor tool selection precision.
Validate Argument Formatting
Even when the agent selects the correct tool, it must format arguments that the tool can actually process. Argument validation testing checks that every parameter matches its expected type, falls within valid ranges, and adheres to any format requirements like date strings, email addresses, or enum values.
For each tool, define its argument schema with types, required fields, optional fields, and validation constraints. Then create test cases that should produce valid arguments and verify them against the schema programmatically. JSON Schema validation is the standard approach: define the tool's interface as a JSON Schema document, capture the agent's actual tool call arguments, and validate them against the schema. Any schema violation is a test failure.
Test edge cases specifically: very long string arguments (does the agent respect max length), special characters in string arguments (quotes, newlines, unicode), numerical arguments at boundary values (zero, negative numbers, very large numbers), and required fields that the user's request does not explicitly provide (does the agent infer reasonable defaults or ask for clarification).
Test argument extraction from natural language. When a user says "look up the weather in San Francisco next Tuesday," the agent must extract location="San Francisco" and date=[next Tuesday's date in the expected format]. Create test cases with varying natural language phrasings that all map to the same underlying arguments, verifying that the extraction is robust across different ways users express the same intent.
Monitor argument formatting in production by logging every tool call with its arguments and running schema validation on the logs. Even a 1% format failure rate across thousands of daily calls means dozens of users per day hit broken functionality. Set alerts when format validation failures exceed your baseline.
Test Error Handling and Recovery
Tools fail. APIs time out, return 500 errors, return unexpected response formats, or return valid responses that indicate the requested operation could not be completed. Your agent must handle every failure mode gracefully rather than crashing, entering an infinite retry loop, or hallucinating a response.
Create a mock tool layer that simulates failure conditions on demand. For each tool, define failure scenarios: timeout after 30 seconds, HTTP 500 internal server error, HTTP 429 rate limit exceeded, HTTP 404 not found, malformed JSON response, empty response, and valid response indicating "no results found." Run your agent against each failure scenario and verify that it either retries appropriately (for transient errors like 429 and 500), informs the user that the operation failed (for permanent errors like 404), or tries an alternative approach (for "no results" scenarios).
Test cascading failures where multiple tools fail in sequence. An agent that handles a single tool failure gracefully may behave poorly when its backup plan also fails. Verify that the agent has a final fallback behavior (typically informing the user that it cannot complete the request and explaining why) rather than looping indefinitely between failing tools.
Test partial success scenarios where a tool returns some but not all requested data. For example, a database query that returns three results when five were expected, or an API that returns results for some items but errors on others. The agent should work with the partial data rather than treating it as a complete failure, but it should also communicate the limitation to the user.
Verify retry behavior explicitly. Count how many retries the agent attempts before giving up. Verify that it respects exponential backoff or rate limit headers rather than hammering a failing API repeatedly. Check that retry logic does not reset when the agent moves to a different step and then returns, which can create unbounded retry cycles that are invisible to simple per-step retry limits.
Verify Result Interpretation
After a tool returns a response, the agent must correctly extract and use the relevant information. This is where subtle failures often occur: the agent reads a response correctly but misinterprets a field, uses an outdated value when a newer one is available, or ignores relevant data because it did not match what the agent expected to find.
Build test cases with known tool responses and verify that the agent's subsequent reasoning uses the correct values. For a database query that returns three customer records, verify that the agent reports the correct names, amounts, and dates rather than transposing or confusing them. For a search API that returns ten results, verify that the agent uses the most relevant results rather than arbitrarily picking the first one.
Test with tool responses that contain ambiguous or conflicting information. If a search returns two results with different prices for the same product, does the agent note the discrepancy or silently pick one? If a database query returns a null value for a field the user asked about, does the agent report "no data available" or hallucinate a value?
Test result interpretation across different response formats. If your tools can return data in various structures (nested objects, arrays, paginated results), verify that the agent handles each format correctly. A common failure is agents that work fine with simple flat responses but break when the tool returns deeply nested JSON or paginated results that require following a "next page" token.
Use trace-based evaluation to verify interpretation. Capture the tool response and the agent's next action, then use an LLM judge to assess whether the action correctly reflects the information in the response. This catches subtle misinterpretations that schema validation cannot detect because the output format is correct but the content is wrong.
Run Integration Tests with Real Tools
Unit tests with mocked tools catch formatting and logic errors, but integration tests with real (or realistic) tools catch issues that only appear in actual usage: rate limiting, authentication failures, response format changes, and timing-dependent behaviors that mocks do not reproduce.
Set up a staging environment with real tool access but against test data. For database tools, use a test database with representative data. For API tools, use sandbox or staging environments that the API provider offers (Stripe test mode, Twilio test credentials, etc.). For tools without test environments, use recorded responses (response replay) that capture real API behavior including headers, timing, and edge cases.
Run integration tests less frequently than unit tests because they are slower, more expensive, and depend on external service availability. A typical cadence is: unit tests on every commit, integration tests daily or on release candidates, and full end-to-end tests weekly. This balances thoroughness with speed and cost.
Monitor tool API changes that could break your agent. Subscribe to changelog notifications from APIs your agent uses, and add regression tests whenever an API announces a format change or deprecation. Many agent failures in production are caused by upstream API changes that no test caught because the test suite used frozen mocks that did not reflect the new API behavior.
Include performance testing in your integration suite. Measure how long real tool calls take under load and verify that your agent's timeout settings accommodate real-world latency. An agent configured with a 5-second timeout against a tool that occasionally takes 8 seconds will experience intermittent failures that are invisible in unit tests with instant mock responses.
Tool calling is the most testable part of agent behavior because it has concrete correctness criteria. Invest in comprehensive tool tests early because tool failures account for the majority of production agent issues, and they are far easier to detect and fix than subtle reasoning errors.