Screenshot Analysis: How Agents See Web Pages

Two Ways to See a Page

An agent can perceive a web page in two fundamentally different ways. The first is reading the page's underlying structure, the document object model, which lists every element with its text and attributes. This gives precise, machine-readable access to what is on the page. The second is visual: capturing a screenshot of the rendered page and interpreting the image, the way a person looks at a screen and understands it.

These approaches have different strengths, which is why both exist. Reading the structure is precise and gives exact targets for actions, but the structure can be messy, obfuscated, or misleading, and it does not always reflect what the visitor actually sees. Visual perception matches the human view directly and handles cases where the structure is unhelpful, but it can be less precise about exact element boundaries. Understanding both is key to understanding how modern agents perceive the web, a topic introduced in how AI browser automation works.

How Screenshot Analysis Works

In the visual approach, the agent captures a screenshot of the current rendered page. This image represents exactly what a visitor would see, with the content laid out and styled as the site intended. The agent then passes this image to a vision-capable model, one that can interpret images and understand their content, identifying the elements, reading the text, and understanding the layout.

The vision model locates interactive elements like buttons and fields by their appearance and position, much as a person would recognize a search box by its look and placement. The agent uses this understanding to decide what to do and where to act. Because the screenshot captures the fully rendered page, this approach depends on the page being completely loaded, which ties back to handling JavaScript and dynamic content so that the captured image reflects the real, complete page rather than a half-loaded one.

When Visual Perception Wins

Screenshot analysis is especially valuable on pages where the underlying structure is unhelpful. Some sites have deeply nested, obfuscated, or non-semantic structure that is hard to interpret, while their visual appearance is perfectly clear. On such pages, a vision model that looks at the rendered result can understand the page when structural parsing struggles.

Visual perception also captures meaning that lives in the layout rather than the markup. The spatial relationship between elements, the visual grouping of related items, and the prominence of certain content are all things a person reads from the visual design, and a vision model can read them too. For tasks where understanding the page as a human would is important, the visual approach provides that perspective directly, which structural parsing alone cannot fully replicate.

A concrete technique that makes visual perception more precise is labeling the interactive elements directly on the screenshot. Before the image goes to the model, the agent overlays a numbered marker on each clickable or fillable element, so the model can name the number of the element it wants to act on rather than estimating screen coordinates. This approach, sometimes described as marking the set of interactive elements, pairs the holistic understanding of vision with a reliable way to specify exact targets, and it has become a common pattern for grounding a vision model's decisions in actions the agent can execute precisely.

The Limits of the Visual Approach

Visual perception has its own weaknesses. Pinpointing the exact boundaries of an element for a precise click can be less reliable from an image than from the structure, which gives exact coordinates and element identities. Vision models can also misread cluttered or ambiguous visuals, and processing images is more computationally expensive than processing text, which affects cost and speed at scale.

There is also the matter of resolution and scope. A screenshot captures what is visible, so content below the fold or hidden in collapsed sections may not appear in a single image, requiring scrolling and multiple captures. Managing this adds complexity. These limits do not undermine the value of visual perception, but they explain why it is usually one part of an agent's perception rather than the whole of it.

Combining Both Approaches

The most capable agents do not choose between structure and vision. They combine them, using each for what it does best. The structure provides precise element identities and exact targets for actions, while the visual view provides holistic understanding and handles cases where the structure is unhelpful. Together they give the agent both accuracy and comprehension.

A common pattern is to use the visual view to understand the page and decide what to do, then use the structure to execute the action precisely on the identified element. This pairing draws on the strengths of both and is more reliable than either alone. It often involves more than one model working together, a vision model for seeing and a reasoning model for planning, which connects to the broader practice of using multiple models together. The result is perception that is both grounded in what the visitor sees and precise enough to act on dependably.