Structured Data Extraction with AI

Why Schema-Based Extraction Matters

Free-form extraction, where you simply ask an LLM to "pull out the important information from this page," produces inconsistent output. One run might return prices as formatted strings like "$29.99," the next as plain numbers like "29.99," and a third with the currency spelled out. Field names vary between runs, optional fields appear unpredictably, and the overall structure changes in ways that break automated processing.

Schema-based extraction solves this by giving the model a contract to follow. A JSON schema specifies exactly which fields to return, what type each field should be, which fields are required versus optional, and what each field means. The model fills in this template with data from the page, producing output that is structurally identical across runs even when the input pages vary dramatically in layout and formatting.

This consistency is essential for production systems. Databases expect specific columns with specific types. APIs require predictable response shapes. Analytics dashboards break when field names change between records. Schema-based extraction provides these guarantees at the extraction layer, eliminating the need for extensive post-processing to normalize free-form LLM output.

Designing Effective Schemas

The quality of your extraction depends heavily on how well you design the schema. A schema that is too broad, requesting "all product information" without specifying fields, produces unreliable results. A schema that is too narrow may miss important data. The best schemas are focused, specific, and thoroughly documented with field descriptions.

Start with the minimum set of fields your downstream system actually needs. If you only use product name, price, and availability, do not add rating, review count, and image URL to the schema just because they are available on the page. Every additional field increases the chance of extraction errors and adds complexity to validation.

Field descriptions are the single most impactful feature for improving extraction accuracy. Instead of a bare field name like "price," add a description: "The current selling price shown to the customer after any active discounts, as a decimal number without currency symbols or thousand separators." This level of specificity eliminates ambiguity when pages display multiple price values, such as list price, sale price, member price, and bulk pricing.

Use appropriate data types for each field. Prices should be numbers, not strings. Dates should follow ISO 8601 format. Boolean fields like availability should be true or false, not "In Stock" or "Available." Specifying types in the schema enables the validation layer to catch extraction errors early.

Handling Complex Data Structures

Real-world extraction tasks often involve nested and repeated data. A product page might list multiple variants with different sizes, colors, and prices. A search results page contains a list of items, each with its own set of fields. A company profile page might include multiple office locations, each with an address, phone number, and set of services.

Array types in the schema handle repeated structures. Define the parent field as an array and specify the structure of each item in the array's item schema. The model identifies the repeating pattern on the page and extracts each instance into a separate array element. For a search results page, this might produce an array of 10 to 20 result objects, each with title, URL, snippet, and rating fields.

Nested objects handle hierarchical data within individual records. A product with variant-specific pricing can use a nested structure where the product object contains an array of variant objects, each with its own color, size, price, and stock status fields. The model navigates this hierarchy naturally because it understands the relationships between the data points on the page.

Optional fields handle variation between pages. Not every product has a rating, not every listing includes a phone number, and not every article has a publication date. Marking these fields as optional in the schema tells the model to include them when present and omit them when not, rather than fabricating values or returning null for every optional field.

Extraction Accuracy Optimization

Several techniques improve extraction accuracy beyond basic schema design. Providing examples of expected output in the extraction prompt helps the model understand the desired format, particularly for ambiguous fields. Showing the model one or two completed extraction examples alongside the target page significantly reduces formatting inconsistencies.

Content preprocessing affects accuracy by controlling what the model sees. Aggressive cleaning that removes navigation, footers, and sidebar content focuses the model on the main content area, reducing the chance of extracting data from irrelevant page sections. Conversely, some pages embed important data in sidebars or header elements, requiring more selective cleaning.

Model temperature settings affect the determinism of extraction. Lower temperatures (0 to 0.2) produce more consistent output across runs, which is generally preferable for structured data extraction. Higher temperatures introduce more variation, which can be useful for creative tasks but is counterproductive for data extraction where consistency matters.

Retry logic with prompt refinement handles initial extraction failures. If the first attempt returns missing required fields or fails validation, a retry with more specific instructions or a larger model often succeeds. Production systems typically attempt extraction with a smaller model first, retrying with a larger model only when the initial attempt fails quality checks.

Validation and Post-Processing

Even well-designed schemas produce output that needs validation. Type checking ensures prices are actually numbers, dates are valid, and enumerated fields contain expected values. Range checking catches obviously wrong extractions, like negative prices or ratings above the maximum scale. Format normalization standardizes variations in currency formatting, date representation, and text encoding.

Cross-field validation catches logical inconsistencies. A sale price higher than the original price, an availability of "in stock" combined with zero quantity, or a rating of 4.5 out of 5 with zero reviews all indicate potential extraction errors. These rules encode domain knowledge about what constitutes valid data for a given use case.

Confidence scoring, when available from the extraction tool, provides another quality signal. Low-confidence extractions can be routed to a larger model for re-extraction, flagged for human review, or excluded from the dataset entirely. This graduated response to quality issues balances throughput against accuracy in production systems.