Structured Output from LLMs: Getting Reliable JSON, Tables, and Schema-Compliant Data

Why Structured Output Matters for Agents

An AI agent rarely works in isolation. It calls tools, receives results, makes decisions, and passes data to other components. Every handoff between the agent and another system requires a shared format. When a customer support agent looks up order status, the tool returns structured data. When the agent constructs a response, it might need to populate a template with specific fields. When the agent decides to escalate, it needs to produce a structured handoff object with the ticket ID, customer context, and escalation reason, formatted exactly as the downstream system expects.

Free-form text output breaks these integrations. If the agent returns "The order #12345 was placed on March 3rd and is currently being shipped" instead of {"order_id": "12345", "date": "2026-03-03", "status": "shipping"}, every downstream system that needs to use that data must parse natural language, which is unreliable and fragile. Structured output eliminates the parsing problem by ensuring the model produces data in a format that code can consume directly.

The challenge is that language models are designed to produce natural language. Asking them to produce perfectly formatted JSON is asking them to operate outside their primary mode, and the result is that without proper techniques, structured output is inconsistent. A model might wrap JSON in markdown code blocks, add explanatory text before or after the JSON, use single quotes instead of double quotes, omit required fields, or generate JSON that is syntactically invalid. Each of these failures breaks downstream systems, and each has a specific solution.

Prompt-Level Techniques for Structured Output

Before using API features, there are prompt engineering techniques that significantly improve structured output reliability.

Explicit format instructions with examples. The most reliable prompt-level technique is combining a format specification with one or two examples of correctly formatted output. The specification defines the schema in words: "return a JSON object with keys: customer_name (string), issue_type (one of: billing, technical, general), priority (one of: low, medium, high, urgent), and summary (string, max 100 words)." The examples show exactly what this looks like in practice. As discussed in the few-shot prompting guide, examples anchor format compliance far more reliably than instructions alone.

The "respond with only" constraint. Adding "respond with only the JSON object, no other text" eliminates the most common structured output failure: the model wrapping its JSON in explanatory prose. Without this constraint, models frequently add preambles like "Here is the extracted data:" or postambles like "Let me know if you need anything else." These additions make the output unparseable by standard JSON parsers. The "respond with only" constraint suppresses this behavior in the vast majority of cases.

Schema in the prompt. For complex schemas, including the schema definition directly in the prompt provides an explicit specification the model can reference. This works particularly well with JSON Schema notation, which models have seen extensively in training data. Including {"type": "object", "properties": {"name": {"type": "string"}, "score": {"type": "number", "minimum": 0, "maximum": 100}}} in the prompt gives the model a machine-readable specification alongside the natural language description, and the combination produces higher compliance than either alone.

Field-by-field extraction. When the model struggles to produce a complex structure in one shot, breaking the extraction into sequential steps improves reliability. First extract the customer name. Then extract the issue type. Then extract the priority. Then combine them into the final JSON object. Each extraction is simpler and more reliable than producing the entire structure at once, and the sequential approach lets you validate each field independently.

API-Level Structured Output Features

Modern LLM APIs provide features specifically designed to enforce structured output at the inference level, which is more reliable than prompt-level techniques alone.

JSON mode. OpenAI, Anthropic, and Google all offer JSON mode or equivalent settings that constrain the model to produce valid JSON. When enabled, the model's token selection is restricted to tokens that result in syntactically valid JSON, making it impossible for the output to be anything other than parseable JSON. This eliminates syntax errors, markdown wrapping, and explanatory text. However, JSON mode only guarantees valid JSON syntax; it does not guarantee that the JSON matches your specific schema. You still need prompt-level instructions to control which fields appear and what values they contain.

Function calling and tool use. Function calling, which Anthropic calls tool use, takes structured output a step further. You define a function with a typed parameter schema, and the model produces output that conforms to that schema. This is the most reliable approach for structured output because the schema is enforced at the API level: the model must produce a value for each required parameter, each value must match the declared type, and enum parameters are restricted to the declared values. For agents, function calling serves the dual purpose of structuring output and enabling tool invocation.

Structured outputs with strict mode. OpenAI's structured outputs feature allows you to provide a JSON Schema and guarantee that the model's output conforms to it exactly, including nested objects, arrays, and enum types. This is the strongest available guarantee for schema compliance, eliminating the need for any output validation or retry logic. The tradeoff is that the schema must be pre-declared in the API call and the first call with a new schema incurs a processing delay while the API compiles the schema into a constrained generation grammar.

Constrained generation with open-source models. For self-hosted models, libraries like Outlines, Guidance, and LMQL provide constrained generation by modifying the token sampling process to follow a grammar or schema. These tools work at inference time, masking out tokens that would produce invalid output, so the model can only generate text that conforms to your specification. The result is 100 percent schema compliance without relying on the model's instruction-following ability. The tradeoff is increased inference complexity and potential quality degradation if the constraints are too tight, forcing the model into outputs it would not naturally produce.

Handling Structured Output Failures

Even with the best techniques, structured output can fail, and production systems need strategies for handling those failures gracefully.

Validation and retry. The simplest approach is to validate the model's output against the expected schema and retry if it fails. Parse the JSON, check that all required fields are present and correctly typed, and if any check fails, send the request again, optionally with a modified prompt that includes the error message. Most structured output failures are intermittent, so a single retry resolves the majority of cases. Set a maximum retry count (typically two or three) to prevent infinite loops when the model consistently fails on a particular input.

Fallback parsing. When the model wraps valid JSON in extra text, a regex or string search that extracts the JSON substring can recover the structured data without a retry. Looking for the first { and last } in the output, or parsing out content between markdown code fences, handles the most common wrapping patterns. This is a pragmatic solution that saves the cost of a retry call at the expense of slightly more complex parsing code.

Default values. For fields that are useful but not critical, providing defaults when the model omits them is more efficient than retrying. If the model returns a valid object but omits an optional "confidence" field, inserting a default value of "medium" may be acceptable for your use case. This approach requires careful judgment about which fields are truly optional versus which are critical to downstream processing.

Streaming validation. For agents that stream structured output, validating the structure incrementally as tokens arrive can detect failures early. If the first tokens suggest the model is generating prose instead of JSON, you can cancel the generation and retry immediately rather than waiting for the full output. Streaming validation reduces latency on failure cases by catching them sooner.

Structured Output Patterns for Production Agents

Several patterns have proven reliable in production agent systems that depend on structured output.

The envelope pattern. The agent always returns output in a standard envelope structure that separates the agent's action from its reasoning. For example: {"action": "respond", "reasoning": "Customer asked about order status, found order in database", "data": {"message": "Your order #12345 shipped on March 3rd"}}. The envelope gives the orchestrator structured access to both the decision and the content, enabling logging, auditing, and conditional routing based on the action type.

The typed action pattern. Each possible agent action has a predefined schema, and the agent selects and populates the appropriate one. A customer support agent might have schemas for "respond_to_customer," "escalate_to_human," "call_tool," and "request_clarification," each with different required fields. This pattern maps naturally to function calling, where each action type is a separate function the model can invoke.

The progressive extraction pattern. For complex documents where a single extraction pass is unreliable, the agent runs multiple passes, each extracting a subset of the fields. The first pass extracts entities. The second pass extracts relationships between entities. The third pass extracts quantities and dates. Each pass uses a focused, simple schema, and the results are merged afterward. This produces higher extraction accuracy than a single comprehensive pass, because each individual extraction is within the model's reliable capability range.

The self-validation pattern. After producing structured output, the agent runs a second prompt that validates the output against the original input. "Does this JSON accurately represent the information in the source text? Check each field and respond with a list of any errors." This catches hallucinated fields, incorrect extractions, and schema violations that the initial prompt missed. The validation pass costs additional tokens but catches errors before they reach downstream systems.