Queue-Based Agent Architecture

The Producer-Consumer Model

Queue-based architecture separates the system into three components: producers that create tasks, the queue that stores them, and consumers (agents) that process them. Producers do not know which agent will handle their task. Consumers do not know which producer submitted the task. The queue is the only point of contact between them.

This decoupling provides several immediate benefits. Producers can submit tasks even when no agents are available. The tasks wait in the queue until an agent is ready. Agents can process tasks even when no new tasks are being submitted. They drain the queue at their own pace. Producers and consumers can scale independently. Adding more producers increases task volume without requiring more consumers, and adding more consumers increases throughput without requiring changes to producers.

The queue itself serves as a buffer that absorbs demand variability. When tasks arrive faster than agents can process them, the queue grows. When agents are faster than the incoming task rate, the queue shrinks. This buffering eliminates the need for producers and consumers to be synchronized, which is especially valuable for AI agent systems where processing time per task is highly variable. One task might require three LLM calls and take 10 seconds. Another might require 30 calls and take two minutes. The queue absorbs this variability naturally.

In practice, the queue is implemented using a message broker like RabbitMQ, Apache Kafka, Amazon SQS, Redis Streams, or a database-backed queue. The choice of queue technology depends on the system's requirements for durability, ordering, throughput, and operational complexity. For most AI agent workloads, a managed service like SQS or a hosted Redis instance provides sufficient capability with minimal operational overhead.

Fault Tolerance Through Acknowledgment

The most valuable property of queue-based architecture is automatic fault tolerance through the acknowledgment protocol. When an agent pulls a task from the queue, the task is not immediately removed. Instead, it becomes invisible to other consumers for a configured period called the visibility timeout. The agent processes the task and, upon successful completion, sends an acknowledgment that permanently removes the task from the queue.

If the agent fails before acknowledging, because of a crash, a timeout, a network partition, or any other failure, the visibility timeout expires and the task becomes visible again. Another agent picks it up and processes it. From the system's perspective, the failure never happened. The task was simply delayed. This at-least-once delivery guarantee means that no task is lost due to agent failure, which is a critical property for production systems where every task represents real work that must be completed.

The visibility timeout must be set carefully. Too short, and the task becomes visible before the agent finishes processing it, resulting in duplicate processing. Too long, and a failed task waits unnecessarily before being retried. A common approach is to set the visibility timeout to a generous multiple of the expected processing time (3x to 5x) and extend it periodically during processing if the agent detects it needs more time. Most queue services support explicit timeout extension, allowing agents to signal that they are still working and need more time.

Idempotency becomes important because at-least-once delivery means tasks can be processed more than once. If an agent completes a task but fails before sending the acknowledgment, the task will be processed again by another agent. The system must be designed so that processing the same task twice produces the same result as processing it once. For tasks that produce output (generating a report, analyzing data), this means checking whether the output already exists before producing it again. For tasks that have side effects (sending an email, updating a record), this means using idempotency keys or deduplication mechanisms to prevent duplicate actions.

Priority and Ordering

Not all tasks are equally important, and queue-based architecture provides mechanisms for differentiating them.

Priority queues assign a priority level to each task. Agents process high-priority tasks before low-priority ones. This is implemented either through a single queue with priority-based ordering or through multiple queues (one per priority level) with agents checking higher-priority queues first. Priority queues are essential when the system handles a mix of urgent and routine work. A customer-facing request that needs a response in seconds should not wait behind a batch of analytical tasks that can tolerate minutes of latency.

FIFO ordering guarantees that tasks are processed in the order they were submitted. This is important when task order carries semantic meaning, like processing a sequence of user edits to a document or applying a series of database migrations. Standard queues do not guarantee ordering, so FIFO behavior requires either a FIFO queue implementation or application-level ordering logic that sequences tasks by a timestamp or sequence number.

Task grouping ensures that related tasks are processed by the same agent or processed in sequence. If a customer submits multiple messages in rapid succession, they should be handled in order by the same conversation context. Task grouping is typically implemented through group IDs or partition keys that the queue uses to route related tasks to the same consumer.

In AI agent systems, the tension between throughput and ordering is particularly acute. Strict ordering limits parallelism because ordered tasks must be processed sequentially. For many workloads, the right approach is to enforce ordering only where it matters (within a single conversation or workflow) and allow full parallelism across independent tasks.

Dead Letter Queues

Some tasks fail repeatedly regardless of how many times they are retried. The data they reference might have been deleted. The external API they depend on might be permanently down. The task itself might be malformed in a way that no agent can process. Without intervention, these tasks cycle indefinitely between the main queue and failed processing attempts, consuming resources and agent capacity on work that will never succeed.

Dead letter queues (DLQs) catch these persistent failures. After a task has been attempted and failed a configured number of times (typically three to five), it is automatically moved to the dead letter queue instead of being returned to the main queue. The DLQ stores failed tasks with their failure metadata: how many times they were attempted, what errors occurred, which agents attempted them, and when each attempt happened.

DLQ management is an operational concern that deserves explicit attention. Someone or something needs to monitor the DLQ, investigate why tasks ended up there, and decide what to do about them. Options include fixing the underlying issue and replaying the tasks back to the main queue, manually processing them, or discarding them if they are no longer relevant. Unmonitored DLQs accumulate failed tasks that represent unfinished work, leading to a growing backlog of issues that becomes increasingly difficult to address.

Smart DLQ processing can also feed back into system improvement. Patterns in DLQ contents reveal systemic issues: if tasks involving a specific API consistently end up in the DLQ, that API integration needs attention. If tasks of a certain type fail disproportionately, the agent handling that type needs better prompting, tools, or error handling. The DLQ is an early warning system for system-wide issues that might not be apparent from individual task monitoring.

Scaling with Queues

Queue-based architecture provides the cleanest scaling model of any agent pattern. Throughput is directly proportional to the number of consumers. If one agent processes 10 tasks per minute and you need to process 100 tasks per minute, you run 10 agents. If demand doubles, you add 10 more agents. If demand drops, you remove agents. The queue absorbs the difference between current capacity and current demand.

Auto-scaling adjusts the number of agent instances based on queue depth. When the queue grows beyond a threshold, new agents are spawned. When the queue is consistently empty, excess agents are terminated. The scaling policy defines the relationship between queue depth and agent count, including minimum and maximum instance counts, scale-up and scale-down thresholds, and cooldown periods that prevent rapid oscillation between scaling states.

Cost optimization in queue-based systems focuses on matching agent capacity to actual demand. Over-provisioning wastes money on idle agents. Under-provisioning lets the queue grow, increasing task latency. The optimal approach depends on the latency requirements and the cost model. For latency-sensitive workloads, maintain enough agents to keep the queue near zero. For latency-tolerant batch workloads, allow the queue to accumulate tasks and process them in efficient batches during off-peak hours when compute costs are lower.

Multi-queue architectures use separate queues for different task types, each with its own pool of specialized agents. Customer support tasks go to the support queue, processed by agents with support-specific prompts and tools. Code review tasks go to the review queue, processed by agents with code-specific capabilities. This separation prevents one task type from starving another during demand spikes and allows each queue to scale independently based on its specific demand pattern.

Queue-Based Pipelines

Queues can connect pipeline stages, combining the benefits of both patterns. Each pipeline stage has its own input queue and its own pool of consumer agents. When a stage completes processing, it places the result on the next stage's input queue. This architecture provides pipeline-style sequential processing with queue-style fault tolerance and scalability at each stage.

Each stage can scale independently based on its specific throughput requirements. If the data extraction stage is the bottleneck, you add more extraction agents without changing the number of agents at other stages. If the review stage is the fastest, you run fewer review agents. This per-stage scaling optimizes resource usage across the entire pipeline.

Stage-level queues also provide natural isolation between pipeline stages. If the analysis stage is temporarily slow (perhaps due to API rate limits), tasks accumulate in its input queue without affecting the extraction stage, which continues processing and filling the queue. When the analysis stage recovers, it drains its backlog without requiring the extraction stage to slow down or speed up.