Pipeline Pattern: Sequential Agent Processing

How Pipelines Work

A pipeline consists of ordered stages, each implemented by an agent with a specific responsibility. The input to the pipeline enters the first stage. That agent processes the input and produces an output. The output becomes the input to the second stage. This continues through every stage until the final agent produces the pipeline's result.

The power of the pattern comes from decomposition. A complex task like "analyze this market and produce a strategy document" becomes a sequence of simpler tasks: gather data from specified sources, extract relevant metrics and trends, compare metrics against industry benchmarks, identify strategic opportunities and risks, draft a recommendations document, review the document for accuracy and completeness. Each of these tasks is substantially easier than the original, and each can be handled by an agent specifically designed for that type of work.

Pipelines enforce a discipline that improves quality. Because each stage must produce a well-defined output that the next stage can consume, there is a natural validation point between every pair of stages. If the data extraction stage produces malformed data, the analysis stage will fail immediately rather than producing a subtly incorrect analysis that goes undetected. These natural validation points make pipelines self-checking in a way that monolithic approaches are not.

The sequential nature of pipelines also simplifies debugging. When the final output is wrong, you can inspect the intermediate results at each stage boundary to identify exactly where the problem originated. This is dramatically easier than debugging a single agent that performs all stages internally, where the reasoning that led to a bad decision is buried in a long chain of thought that mixes multiple concerns.

Designing Pipeline Stages

The most critical design decision in a pipeline is where to draw the boundaries between stages. Stages that are too fine-grained create excessive overhead from inter-stage communication and context transfer. Stages that are too coarse-grained lose the benefits of decomposition and specialization.

A well-designed stage has a single clear responsibility, a well-defined input format, and a well-defined output format. The stage should be comprehensible in isolation: someone reading the stage's prompt and tool configuration should understand what it does without needing to know about other stages. This independence is what makes stages reusable across different pipelines and testable in isolation during development.

The input and output contracts between stages deserve explicit design attention. Implicit contracts, where the output format of one stage happens to match what the next stage expects, are fragile. A small change to one stage's output can silently break the downstream stage. Explicit contracts, where the expected format is documented and validated, catch mismatches early. The validation can be as simple as checking that the output is valid JSON with required fields, or as sophisticated as running a schema validator against a formal specification.

Each stage should also define its failure behavior explicitly. When a stage encounters an error, should it retry internally, pass the error downstream as a special output, report the error and halt the pipeline, or attempt a degraded output with whatever partial results it managed to produce? The right answer depends on the specific stage and the overall pipeline's tolerance for partial results.

Stage granularity follows a practical rule: each stage should correspond to a distinct cognitive mode. If you would assign two different people with different skills to do the work, it belongs in two stages. If you would assign the same person, it belongs in one stage. A research stage and an analysis stage require different skills (information gathering versus pattern recognition), so they belong in separate stages. Two different types of data cleaning (deduplication and format normalization) require the same skill applied to different aspects of the data, so they can be a single cleaning stage with two substeps.

Model selection per stage is a practical optimization that pipelines enable. Early stages that perform straightforward tasks like data extraction or classification can use smaller, faster, cheaper models. Later stages that require sophisticated reasoning like analysis or strategic recommendations can use larger, more capable models. A pipeline that uses a small model for its first three stages and a large model for its last two stages can cost 60-70% less than a pipeline that uses the large model throughout, with negligible quality difference in the stages where the small model is sufficient.

Common Pipeline Architectures

ETL pipelines (extract, transform, load) are the most common pattern in data processing. An extraction agent reads data from source systems. A transformation agent cleans, normalizes, and enriches the data. A loading agent writes the processed data to the destination system. This three-stage structure maps directly to the ETL paradigm that data engineers have used for decades, making it immediately familiar and well-understood.

Content creation pipelines mirror the editorial workflow of a publishing organization. A research agent gathers source material. An outline agent organizes the material into a logical structure. A drafting agent writes the full content. A review agent checks for accuracy, consistency, and quality. An optimization agent applies formatting, SEO adjustments, and metadata. Each stage corresponds to a distinct skill that benefits from specialized prompting and tool access.

Code generation pipelines decompose software development into sequential phases. A requirements agent interprets the specification and identifies the components needed. A design agent produces the architecture and interface definitions. An implementation agent writes the code. A testing agent creates and runs tests. A review agent checks for bugs, security issues, and style violations. A documentation agent produces inline comments and external docs. This pipeline produces higher quality code than a single agent because each stage focuses entirely on its concern.

Analysis pipelines process information through increasingly sophisticated levels of interpretation. A data collection agent gathers raw information. A cleaning agent removes noise and standardizes formats. A statistical agent computes metrics and identifies patterns. An interpretation agent translates statistical findings into business language. A recommendation agent produces actionable suggestions based on the interpretation. Each stage adds a layer of value that the previous stages cannot provide because they lack the necessary perspective.

Handling Inter-Stage Data

The data that flows between pipeline stages is the connective tissue of the entire system. How this data is structured, sized, and transmitted determines the pipeline's reliability and performance.

Structured intermediate formats use JSON, XML, or another schema-based format for inter-stage data. This approach enables automated validation, makes debugging straightforward (you can read and understand the intermediate data), and ensures compatibility between stages even when they are developed independently. The downside is that forcing all information into a structured format can lose nuance. A research agent's insights about source credibility or contextual caveats may not fit neatly into a predefined schema.

Natural language intermediates pass free-form text between stages. This preserves all nuance and context but makes validation harder and introduces the risk of misinterpretation. A subsequent stage might misread an ambiguous phrase in the previous stage's output and proceed with an incorrect assumption. Natural language intermediates work best when the pipeline stages are tightly designed together and the expected communication patterns are well-understood.

Hybrid intermediates combine structured data with natural language annotations. The structured portion contains the core data: extracted facts, computed metrics, generated code. The natural language portion contains context, caveats, confidence assessments, and reasoning explanations that inform how downstream stages should interpret the structured data. This approach captures the benefits of both formats while mitigating their individual weaknesses.

Data volume is also a concern. A research stage might produce thousands of words of source material that the next stage needs to process. If this exceeds the downstream agent's context window, the pipeline needs a compression strategy: summarize the output, select the most relevant portions, or store the full output in an external location and pass a reference with a summary. The compression approach should be explicit in the pipeline design rather than something that individual stages handle ad hoc.

Pipeline Variations

Fan-out/fan-in pipelines add parallelism to the sequential model. After a planning stage decomposes a task into independent subtasks, the pipeline fans out: multiple instances of the next stage run in parallel, one for each subtask. After all parallel stages complete, a fan-in stage aggregates the results into a single output that continues through the rest of the pipeline. This pattern is useful when one stage of the pipeline involves processing multiple independent items, like analyzing multiple documents or generating multiple code modules.

Conditional pipelines include branch points where the output of one stage determines which subsequent stages execute. A classification stage might route a customer request to a technical support pipeline, a billing pipeline, or a sales pipeline based on the request type. Conditional routing adds flexibility but also complexity, since the pipeline is no longer a simple linear sequence and the set of possible execution paths grows with each branch point.

Iterative pipelines include feedback loops where a later stage can request re-execution of an earlier stage. A review stage that finds quality issues sends the content back to the drafting stage with specific revision instructions. This loop continues until the review stage approves the output or a maximum iteration count is reached. Iterative pipelines produce higher quality results but take longer and cost more due to the repeated execution of earlier stages.

Pipeline Observability

Pipelines provide naturally superior observability compared to other multi-agent patterns because the data flow is linear and the stage boundaries are explicit. This is not just a convenience during development. It is a production advantage that pays dividends every time you need to diagnose a quality issue, optimize cost, or validate that a change improved the system.

Each stage boundary is a natural instrumentation point. Capture the input size, output size, processing time, token consumption, and model used at every boundary. These measurements reveal exactly where your pipeline spends its time and money. If 80% of the cost comes from one stage, that is where optimization effort belongs. If one stage takes 10x longer than the others, it is the bottleneck that limits pipeline throughput.

Intermediate outputs should be stored, at least temporarily, for every pipeline execution. When the final output has a quality issue, you can trace it to the specific stage where the problem originated by examining the intermediate outputs. A research stage that retrieved irrelevant sources causes different downstream symptoms than an analysis stage that drew incorrect conclusions from good sources. Without stored intermediates, diagnosing the root cause requires guesswork.

Quality metrics per stage, rather than just for the final output, enable targeted improvement. If the research stage consistently retrieves high-quality sources but the synthesis stage loses key findings, the fix is a better synthesis prompt, not a redesign of the entire pipeline. Per-stage quality tracking also detects regression immediately: if a prompt change to the analysis stage improves analysis quality but degrades the downstream recommendation quality, the per-stage metrics reveal the interaction within one pipeline run rather than requiring a lengthy investigation.

Pipeline Limitations

The biggest limitation of pure pipelines is their inability to handle tasks that require backtracking or exploratory reasoning. If the analysis stage discovers that the data collection stage missed critical information, the pipeline cannot easily go back and collect more data. The options are to re-run the pipeline from the beginning (wasteful), add a feedback loop (adds complexity), or accept partial results (may compromise quality).

Pipelines also struggle with tasks where the optimal decomposition is not known in advance. If the number and type of stages depend on the specific input, a fixed pipeline structure is too rigid. These tasks are better served by orchestrator-based multi-agent architectures that can dynamically plan and adjust the workflow.

Latency accumulates linearly with the number of stages. A ten-stage pipeline takes at least ten times as long as a single stage, assuming no parallelism. For time-sensitive applications, this cumulative latency may be unacceptable, and a single-agent approach that sacrifices quality for speed may be more appropriate.