How to Design an AI Agent System

Define the Problem Precisely

Before writing any code or choosing any framework, define exactly what the agent will do. "Build a customer support agent" is not a definition. "Build an agent that handles tier-one customer support tickets by answering questions from the knowledge base, escalating to a human when confidence is below 80%, and tagging tickets by category" is a definition.

The definition should specify the inputs (what triggers the agent and what information it receives), the outputs (what the agent produces and in what format), the success criteria (how you will measure whether the agent is doing its job well), and the boundaries (what the agent explicitly should not do). Write these down. Share them with stakeholders. Get agreement before proceeding.

Pay special attention to the boundaries. Agents that are not explicitly told what they cannot do will try to help with everything, often poorly. If the agent should not handle billing disputes, refund requests, or account cancellations, state that explicitly. Clear boundaries prevent scope creep during development and prevent the agent from attempting tasks it is not equipped to handle.

Quantify the success criteria wherever possible. "Good customer support" is subjective. "Resolves 70% of tier-one tickets without human intervention, with a customer satisfaction score above 4.0/5.0 and an average response time under 30 seconds" is measurable. Quantified criteria let you evaluate candidate designs objectively and tell you when the deployed system is meeting its goals.

Map the Task Structure

Take the defined problem and decompose it into the concrete steps the agent must perform. For the customer support example: receive the ticket, classify its type, search the knowledge base for relevant articles, synthesize a response from the relevant articles, assess confidence in the response, either send the response or escalate based on confidence, and tag the ticket by category.

For each step, determine whether it requires LLM reasoning or can be handled programmatically. Ticket classification might use an LLM or a simpler classifier model. Knowledge base search is a retrieval operation, not a reasoning task. Response synthesis requires LLM reasoning. Confidence assessment might use a combination of model logprobs and heuristic rules. Sending the response and tagging the ticket are programmatic operations.

Identify dependencies between steps. Classification must happen before knowledge base search (you search for different things depending on the ticket type). Knowledge base search must happen before response synthesis (you need the articles to synthesize from). Dependencies determine the ordering constraints and identify which steps can run in parallel.

Estimate the cost and latency of each step. An LLM call with a large context costs more and takes longer than an LLM call with a small context. A database query is essentially free and instant compared to an LLM call. These estimates inform architecture decisions: if a single agent can handle all steps within an acceptable latency budget, simple architecture suffices. If the total latency exceeds requirements, parallelism or optimization is needed.

Choose Your Architecture Pattern

Start with the simplest pattern that fits your task structure and only add complexity when you have a concrete reason. For most projects, this means starting with a single-agent architecture.

Consider multi-agent architecture only when you have clear evidence that a single agent cannot handle the workload. Evidence includes: the full prompt exceeds the model's effective context window, different steps require fundamentally different model configurations, or independent steps benefit significantly from parallel execution.

If you choose multi-agent, decide on the coordination pattern. An orchestrator is simplest and most predictable. Peer-to-peer is more flexible but harder to debug. Choose based on how complex the coordination logic is and how predictable you need the system's behavior to be.

Select runtime patterns based on how work arrives. If work arrives as external events, use event-driven execution. If work arrives at variable rates and needs reliable processing, use queue-based execution. If the agent needs to proactively check for work, use tick-based execution. If the agent maintains complex state across interactions, use the GenServer pattern.

Design the Tool Set

List every external capability the agent needs. For each capability, design a tool with a clear name that describes what it does (not how it works), a precise description that tells the model when to use this tool and what results to expect, an explicit input schema with required and optional parameters, an explicit output schema that the agent can reliably parse, and documented error cases with guidance on how the agent should handle each error.

Keep tools focused. A tool should do one thing. "search_knowledge_base" is better than "interact_with_knowledge_system" that handles searches, updates, and deletions. Focused tools give the agent precise control and produce predictable results.

Include mock implementations of every tool from the start. Mock tools let you test the agent's reasoning and tool selection logic without connecting to real external systems. When the agent's behavior is correct with mocks, switch to real implementations. This approach catches reasoning issues early, before they are confounded by integration issues.

Limit the initial tool set to the minimum required for the core workflow. You can always add tools later. Starting with too many tools degrades the model's tool selection accuracy and makes debugging harder because there are more possible tool call sequences to reason about.

Build the Prompt Architecture

Design the prompt using composition from the start, even if the initial prompt is small. Separate the identity (who the agent is), the instructions (how it should work), the tool descriptions (what it can do), the context injection points (what it knows about this specific task), and the output constraints (what its response should look like).

Write the instructions in order of importance. The most critical instructions should appear first and last in the prompt, where the model pays the most attention. Put behavioral guardrails early. Put output format requirements near the end, right before the task begins.

Include examples in the prompt for any behavior that is not obvious from the instructions alone. An example of a well-handled edge case is worth more than a paragraph of instructions describing how to handle it. Examples should cover the common case (what the agent does most of the time), one or two edge cases (situations that require judgment), and one failure case (how the agent should respond when it cannot help).

Test the prompt with real examples from your problem domain before connecting it to tools or infrastructure. Send the prompt and a sample task to the model API directly and evaluate the response. This rapid feedback loop lets you iterate on the prompt in minutes rather than the hours required to test through a full agent system.

Define Failure Modes and Recovery

For each component of the system, list the ways it can fail and define the recovery strategy. LLM API calls can timeout, return rate limit errors, return malformed responses, or produce responses that do not follow instructions. Tool calls can fail with network errors, authentication errors, data errors, or unexpected results. External services can be unavailable, slow, or return stale data.

For each failure mode, define the recovery action: retry (with what backoff), fallback (to what alternative), escalate (to whom, with what context), or degrade (what reduced functionality is acceptable). Document the maximum number of retries, the timeout for each operation, and the escalation path when automated recovery fails.

Design circuit breakers for external dependencies. If the LLM API is consistently slow or failing, stop sending requests and either queue work for later or switch to a fallback model. If a tool's backing service is down, disable the tool and inform the agent that the capability is temporarily unavailable. Circuit breakers prevent a single failing dependency from consuming the entire system's resources on futile retry attempts.

Plan for the case where the agent itself is the problem. What if the agent enters a reasoning loop? What if it consistently produces wrong answers? What if it calls the wrong tools? Step limits, cost budgets, output validation, and human-in-the-loop checkpoints catch these failure modes before they cause damage.

Plan for Production

Production planning covers four areas: observability, cost control, security, and deployment.

Observability: define what to log (every LLM call, tool call, decision point, and error), what to measure (latency, cost, success rate, error rate per task type), and what to trace (end-to-end task execution through every component). Build dashboards that show the agent's health at a glance and alerts that fire when key metrics deviate from expected ranges.

Cost control: set token budgets per task, per hour, and per day. Implement cost tracking that attributes spending to specific task types and agents. Set alerts for spending anomalies. Consider implementing a cost-aware agent that factors token consumption into its decisions, choosing cheaper tool calls when the budget is tight.

Security: apply the principle of least privilege to every agent. Each agent gets only the tools and data access it needs for its specific tasks. Validate all inputs to tools. Sanitize all outputs before they reach users. Implement rate limiting to prevent abuse. Audit every action the agent takes for compliance and investigation purposes.

Deployment: start with a staging environment that mirrors production. Deploy to a canary group first. Monitor the canary for a defined period before expanding. Maintain the ability to roll back instantly. Never deploy on a Friday or before a holiday. These practices are not unique to agent systems, but they are especially important because agent behavior is less predictable than traditional software.