AI Agent Framework Performance Benchmarks

What Performance Means for Agents

Agent performance benchmarks are misleading when they measure the wrong thing. Many published benchmarks measure framework overhead in isolation by timing how fast the framework can process a request without actually calling an LLM. This produces sub-millisecond numbers that are technically accurate but practically irrelevant, because real agent execution is dominated by LLM API calls that take 500 to 5,000 milliseconds each.

The performance metrics that matter for production agents are end-to-end latency (how long a user waits from request to complete response), throughput (how many concurrent agent executions the system can handle), memory footprint (how much RAM each agent execution consumes), cold start time (how long it takes to initialize a new agent instance, especially relevant for serverless), and streaming responsiveness (how quickly the first token appears in the response stream). These metrics vary significantly based on the model provider, the complexity of the task, the number of tool calls, and the deployment infrastructure, not just the framework.

End-to-End Latency Breakdown

A typical single-agent task with two tool calls breaks down into these latency components: framework initialization (5-50ms), first LLM call for reasoning (800-3,000ms depending on model and prompt size), tool execution (50-500ms depending on the tool), second LLM call to process tool results (600-2,000ms), final response formatting (1-5ms). Total end-to-end latency: 1,500-5,500ms. The framework's contribution to this total is typically under 100ms, or less than 5% of the total execution time.

Lightweight frameworks like the OpenAI Agents SDK and Phidata add the least overhead because they have fewer abstraction layers between your code and the API call. LangGraph adds moderate overhead for graph compilation, state management, and checkpointing. CrewAI adds overhead for agent coordination, context assembly, and role-based routing. AutoGen adds the most overhead because conversational message routing involves additional processing at each turn.

These overhead differences matter primarily at high volume. If framework overhead is 10ms versus 50ms, the difference is negligible for a single request. Over 100,000 daily requests, the difference is 40ms times 100,000, which equals about 67 minutes of additional compute time per day. This translates to real infrastructure cost but does not affect user-perceived latency for individual requests.

Throughput and Concurrency

Agent throughput measures how many agent executions the system handles simultaneously. This metric is constrained by three factors: the framework's concurrency model, the available memory for concurrent agent instances, and the model provider's rate limits.

Python frameworks using asyncio (LangGraph, LlamaIndex, Phidata) handle concurrent agent executions efficiently for I/O-bound workloads. Each agent execution awaits API calls without blocking other agents, allowing a single Python process to run dozens of concurrent agents. The practical limit is usually memory rather than CPU, because each concurrent agent consumes memory for its state, conversation history, and tool results. A typical agent execution consumes 50-200MB of memory depending on context size and tool output volume.

Node.js frameworks (Vercel AI SDK, Mastra, LangChain.js) benefit from Node's event loop for concurrent I/O operations. A single Node.js process can handle more concurrent agent executions than a single Python process for I/O-bound workloads because Node's event loop is more efficient than Python's asyncio for high-concurrency scenarios. The practical limit is again memory, with each concurrent agent consuming 30-150MB depending on the framework and context size.

Model provider rate limits are often the actual throughput bottleneck. OpenAI's rate limits vary by tier but typically allow hundreds to thousands of requests per minute for production accounts. Anthropic and Google have similar tiered rate limiting. When your agent system reaches the provider's rate limit, additional requests queue or fail regardless of how much throughput your framework and infrastructure can handle. Multi-provider routing (sending overflow traffic to a backup provider) is the most effective strategy for scaling beyond a single provider's rate limits.

Memory Usage Profiles

Memory consumption varies significantly across frameworks and directly determines how many concurrent agents you can run on a given server.

The OpenAI Agents SDK has the smallest memory footprint because it maintains minimal state beyond the conversation context. A single agent execution typically consumes 30-50MB. On a server with 4GB of available memory, you can run approximately 50-80 concurrent agent executions.

Phidata and the Vercel AI SDK have moderate memory footprints of 50-100MB per agent execution. The additional memory supports features like knowledge base connections, memory persistence, and response streaming infrastructure.

LangGraph consumes 80-200MB per agent execution depending on graph complexity and checkpoint size. The graph compilation, state management, and checkpointing infrastructure require more memory than simpler frameworks. This is the cost of durable execution and explicit state control.

CrewAI consumes 100-250MB per multi-agent workflow because it maintains state for multiple agents simultaneously, plus the shared memory system for inter-agent context sharing. A three-agent crew consumes roughly three times the memory of a single agent plus coordination overhead.

AutoGen's memory consumption is the highest at 150-400MB per multi-agent conversation because it maintains the full conversation history for all participants, plus any sub-conversations and code execution environments. Long conversations with many rounds consume more memory as the history grows.

Cold Start Performance

Cold start time matters for serverless deployments where the framework initializes fresh for each invocation (or at least after idle periods). Fast cold starts mean lower latency for the first request after a period of inactivity and better cost efficiency on serverless platforms that charge by execution time.

The OpenAI Agents SDK and Vercel AI SDK have the fastest cold starts at 100-300ms because they have minimal initialization requirements. These frameworks are well-suited for serverless deployment on AWS Lambda, Vercel Functions, or Cloudflare Workers.

Phidata cold starts at 300-600ms, adding time for knowledge base connections and tool initialization. This is still acceptable for serverless deployment but adds noticeable latency to the first request.

LangGraph cold starts at 500-1,500ms depending on graph complexity and whether the checkpoint store requires a database connection. Complex graphs with many nodes take longer to compile, and database connection establishment adds latency. LangGraph works on serverless platforms but benefits from provisioned concurrency or warm-keeping strategies to avoid cold start latency.

CrewAI and AutoGen cold start at 1,000-3,000ms because they initialize multiple agents, load tool configurations, and establish any necessary service connections. These frameworks are better suited for always-on container deployments rather than serverless functions.

Streaming Performance

Streaming responsiveness measures how quickly the first token of the response appears in the output stream. For interactive applications where users watch the agent's response appear in real time, time-to-first-token determines perceived responsiveness. Users perceive a 200ms time-to-first-token as instantaneous and a 2,000ms time-to-first-token as slow, even if the total response time is identical.

The Vercel AI SDK has the best streaming performance because streaming is a core design principle. The SDK minimizes the overhead between the model's streaming response and the client's rendering, achieving time-to-first-token within 50-100ms of the model's first token emission. The React hooks provide progressive rendering that updates the UI with each chunk.

The OpenAI Agents SDK provides efficient streaming with time-to-first-token within 100-200ms of the model's first token. The SDK passes through the model's streaming response with minimal buffering.

LangGraph's streaming adds overhead because responses may need to pass through multiple graph nodes before reaching the output. Time-to-first-token depends on the graph structure, with simple graphs adding 100-300ms and complex graphs with pre-processing nodes adding more.