AI Agent Architecture: How the Pieces Fit

The Language Model Core

The language model sits at the center of every agent architecture. It receives structured context on each turn, including the system prompt, conversation history, tool descriptions, and results from previous actions. From this context, it generates the next action: a tool call, a response to the user, or a planning step that sets up future actions. The model itself is stateless. It does not remember anything between turns. All continuity comes from the context that the orchestration layer assembles and passes to the model on each invocation.

Model selection affects every aspect of the architecture. Larger models like Claude Opus or GPT-4o handle complex reasoning chains, ambiguous instructions, and multi-step planning more reliably than smaller models. But they cost more per token and respond more slowly. Many production architectures use multiple models: a frontier model for complex reasoning and planning, a mid-tier model for routine tool calls, and a small model for classification and extraction tasks. The routing logic that decides which model handles each turn becomes a critical architectural component.

The interface between the model and the rest of the system is the message format. Every major provider (Anthropic, OpenAI, Google) uses a similar structure: an array of messages with roles (system, user, assistant, tool) and content. The model generates structured tool calls as part of its response, which the runtime intercepts and executes. This standardized interface means that agent architectures can swap models without changing the rest of the system, though differences in model capabilities may require adjustments to prompts and tool descriptions.

The Tool Layer

Tools give the agent the ability to interact with the world beyond text generation. The tool layer has three responsibilities: describing available tools to the model, validating and executing tool calls, and formatting results for the model to process.

Tool descriptions are JSON schemas that tell the model what each tool does, what parameters it accepts, and what format the results will be in. The quality of these descriptions directly affects how well the model uses the tools. Vague descriptions lead to incorrect tool calls. Overly detailed descriptions waste context window space. The best tool descriptions are concise but unambiguous, with clear parameter names, type constraints, and examples of valid inputs.

Execution is where the tool layer connects to the real world. A web search tool sends HTTP requests to a search API. A database tool runs SQL queries. A file tool reads from and writes to disk. A code execution tool runs scripts in a sandboxed environment. Each tool implementation handles authentication, error handling, rate limiting, and result formatting independently. The tool layer provides a consistent interface to the model regardless of how different the underlying implementations are.

Tool registries manage the available tools dynamically. Instead of hardcoding all tools into the system prompt, a registry lets the agent discover and load tools at runtime. This is especially important when the agent has access to dozens or hundreds of tools, since loading all tool descriptions into the context window would consume too many tokens and degrade the model ability to select the right tool. The Model Context Protocol (MCP) standardizes how tools are discovered and invoked across different providers and platforms.

The Memory System

Memory in an agent architecture operates at multiple timescales. Working memory holds the information the agent needs for the current task: the conversation so far, intermediate results, and the current plan. Session memory persists across turns within a single interaction but resets when the session ends. Long-term memory persists across sessions, allowing the agent to recall past interactions, learned preferences, and accumulated knowledge.

Working memory is typically the conversation context itself. Every message, tool call, and result is appended to the conversation array, and the full array is passed to the model on each turn. This gives the model complete visibility into what has happened so far. The limitation is size: as the conversation grows, it eventually exceeds the context window, requiring summarization or truncation strategies.

Long-term memory requires external storage. Vector databases (Pinecone, Weaviate, ChromaDB, pgvector) store information as numerical embeddings that can be searched by semantic similarity. When the agent needs to recall something from a past session, it generates an embedding of its current question and searches the vector database for the most similar stored entries. This retrieval-augmented generation (RAG) pattern lets agents access vast knowledge bases without loading everything into the context window.

Episodic memory stores complete records of past interactions, not just facts but the full sequence of actions, decisions, and outcomes. When the agent encounters a situation similar to one it has handled before, it can retrieve the relevant episode and use it as a guide. This is particularly valuable for error recovery: if the agent failed at a similar task in the past, the episodic record shows what went wrong and what alternative approach succeeded.

The Orchestration Runtime

The orchestration runtime is the control plane that manages everything outside the model itself. It assembles the context for each model call, dispatches tool executions, manages state transitions, enforces budgets and timeouts, and handles errors. Most agent frameworks (LangChain, CrewAI, Anthropic Agent SDK) are essentially orchestration runtimes with different design philosophies.

The core loop of the runtime is straightforward: assemble context, call the model, parse the response, execute any tool calls, append results to context, repeat. The complexity lies in the details. How does the runtime handle a model response that contains multiple tool calls? Does it execute them in parallel or sequentially? What happens if a tool call fails? How does the runtime decide when to stop the loop? These implementation decisions define the behavior of the agent.

Resource management is a critical runtime responsibility. Every model call costs money and takes time. The runtime enforces token budgets (maximum tokens per task), turn limits (maximum reasoning turns per task), and wall-clock timeouts (maximum elapsed time per task). Without these limits, a malfunctioning agent can enter an infinite loop, consuming thousands of dollars in API calls while making no progress on the task.

Error handling in the runtime operates at a different level than error handling in the agent. The agent handles semantic errors (wrong data, unexpected results, failed strategies). The runtime handles infrastructure errors (API timeouts, rate limits, malformed responses, crashed tools). The runtime catches these infrastructure errors before they reach the agent, retries when appropriate, and only surfaces the error to the agent when automated recovery fails.

The Perception Layer

The perception layer transforms raw inputs into structured formats that the reasoning engine can process. For a text-based agent, perception is relatively simple: the input is already text. But modern agents process images, audio, video, PDFs, spreadsheets, code files, and structured data from APIs. Each input type requires parsing, validation, and sometimes transformation before the model can work with it effectively.

Multimodal perception is increasingly important as agents handle more diverse inputs. An agent that processes customer support tickets might receive text, screenshots, log files, and database records in a single task. The perception layer normalizes these inputs into a format the model can process, adding metadata about the source, type, and relevance of each input. Good perception design reduces the cognitive load on the model by presenting information in the clearest possible format.