Anti-Detection Techniques for AI Scraping

How Bot Detection Works

Modern bot detection systems analyze multiple signals simultaneously to classify incoming traffic. No single signal definitively identifies a bot, so detection relies on combining dozens of indicators into a confidence score. Understanding what these systems look for is the foundation for building scrapers that avoid detection.

Browser fingerprinting examines the characteristics of the browser itself. Detection scripts check the user agent string, screen resolution, installed plugins, WebGL renderer, timezone, language settings, and dozens of other browser properties. Headless browsers often have telltale fingerprints, such as missing plugins, unusual screen dimensions, or navigator properties that reveal automation frameworks.

Behavioral analysis looks at how the client interacts with the page. Bots tend to access pages in rapid succession, follow predictable patterns, skip mouse movements and scroll events, and request resources in different orders than real browsers. Detection systems build behavioral profiles and flag sessions that deviate significantly from typical human patterns.

Network-level signals include IP reputation, request frequency, geographic consistency, and TLS fingerprinting. Known datacenter IP ranges carry higher bot risk scores than residential IPs. Rapid-fire requests from a single IP are more suspicious than distributed, time-varied requests. And the TLS handshake characteristics of different HTTP clients have distinct signatures that detection systems catalog.

Browser Fingerprint Management

The first layer of anti-detection is making the headless browser look like a regular browser. Default Playwright and Puppeteer configurations expose several signals that bot detection scripts check for. The most commonly detected are the navigator.webdriver property (set to true in automated browsers), missing or inconsistent plugin arrays, and Chrome DevTools protocol traces.

Stealth plugins address these signals systematically. Libraries like playwright-extra with the stealth plugin modify the browser instance to remove automation indicators. These plugins override navigator.webdriver, add realistic plugin and mime type arrays, fix inconsistencies in the WebGL renderer and canvas fingerprint, and ensure that the browser's feature detection results match those of a real Chrome installation.

User agent rotation changes the browser's identity string across requests. A single user agent making thousands of requests raises suspicion. Rotating through a realistic set of current user agent strings, weighted toward common browser versions and operating systems, distributes requests across apparent browser populations. The user agent must match the actual browser behavior, so a Chrome user agent paired with Firefox-specific JavaScript behavior will be flagged.

Viewport and screen resolution randomization prevents fingerprint-based session linking. Setting random but realistic viewport sizes (common resolutions like 1920x1080, 1440x900, 1366x768) for each session prevents detection systems from correlating requests that share identical, unusual dimensions.

Proxy Strategy and IP Management

IP-based detection is the simplest and most widely deployed anti-bot measure. Websites track request volume per IP and block or rate-limit addresses that exceed thresholds. Proxy rotation distributes requests across many IP addresses to stay below per-IP limits.

Datacenter proxies are the cheapest option, with bulk pricing as low as a few dollars per thousand IPs. However, many bot detection services maintain databases of known datacenter IP ranges and automatically flag traffic from these addresses. Datacenter proxies work well for sites with minimal bot detection but are quickly blocked on well-protected targets.

Residential proxies route traffic through real consumer internet connections, making requests appear to originate from household devices. These are much harder to detect because the IP addresses belong to legitimate ISPs and pass IP reputation checks. The cost is significantly higher, typically ten to fifty times more per gigabyte than datacenter proxies, but the success rate on protected sites justifies the expense.

ISP proxies combine the speed of datacenter infrastructure with the reputation of residential IPs. These are hosted in datacenters but assigned IP addresses from ISP ranges, giving them residential-like reputation at lower latency than true residential proxies. They represent a middle ground in both cost and detection avoidance.

Session persistence is important for sites that track user behavior across multiple pages. Using the same proxy IP for an entire browsing session, including login, navigation, and extraction, appears more natural than switching IPs between every request. Sticky sessions that maintain the same IP for a configurable duration balance detection avoidance with IP rotation frequency.

Behavioral Mimicry

Making a headless browser look real is necessary but not sufficient. Detection systems also analyze behavioral patterns to distinguish bots from humans. Incorporating realistic behavioral signals significantly reduces detection rates.

Mouse movement simulation generates natural-looking cursor activity on the page. Humans move their mice in curved paths with acceleration and deceleration, not in straight lines. Libraries like ghost-cursor generate realistic mouse movement trajectories that include the micro-adjustments and overshoots characteristic of human motor control.

Scroll behavior should vary in speed and timing. Humans scroll at irregular intervals, pause to read content, sometimes scroll back up, and vary their scroll speed. Automated scrolling at a constant rate or in exactly equal increments is easily detected. Random delays, variable scroll distances, and occasional pauses make scroll behavior appear natural.

Request timing introduces random delays between page loads. Accessing pages at precise intervals (exactly every 5 seconds, for example) is a strong bot signal. Adding random jitter to request intervals, with delays drawn from a distribution that mimics human browsing patterns (typically 3 to 15 seconds between page navigations), makes timing analysis less effective.

Click patterns on interactive elements should not follow the same path every time. Clicking the exact center of every button is a detectable pattern. Introducing small random offsets to click coordinates, within the bounds of the target element, produces more natural interaction patterns.

Rate Limiting and Politeness

Respecting rate limits is both an ethical practice and a practical necessity. Overloading a target site with requests degrades its performance for legitimate users and virtually guarantees detection and blocking. Polite scraping operates well below the site's capacity, distributing load over time and across sessions.

Adaptive rate limiting adjusts request frequency based on server response signals. Increasing response times suggest the server is under load, and the scraper should slow down. HTTP 429 (Too Many Requests) responses indicate explicit rate limits that must be respected. Server errors (5xx) may indicate overload that the scraper is contributing to.

Domain-specific rate limits customize the request frequency for each target site based on its observed tolerance. A large e-commerce platform can handle higher request volumes than a small business website. Setting per-domain limits prevents overwhelming smaller sites while maintaining throughput on larger ones.

Respecting robots.txt directives, while not legally required, signals good faith and is checked by some bot detection systems. Crawling pages that robots.txt explicitly disallows is both ethically questionable and a detection signal that some systems specifically monitor for.