sodcalculator

10 Tech How-To

Detect & Handle Anti-Bot Measures in Automation

Guide 10-57167 · Updated · Step by step

Browser automation scripts often hit walls in the form of anti-bot systems that detect and block non-human traffic. This guide walks you through identifying the common signals these systems look for, adjusting your automation setup to avoid them, and handling situations where you still get blocked. You’ll need a browser automation tool like Playwright, Puppeteer, or Selenium, and a basic understanding of your scripting language of choice.

What You’ll Need

  • A browser automation library (Playwright, Puppeteer, Selenium)
  • A programming environment (Node.js, Python, etc.)
  • A test website you have permission to automate
  • Basic knowledge of HTTP requests and browser dev tools

Step-by-Step

  1. Start with a clean profile. Launch your browser with a fresh, non-default user data directory. Most automation tools use a temporary profile by default, but if you’re reusing one, clear it. This removes cookies, local storage, and other artifacts that mark the session as automated.
  2. Check for the obvious markers. Open the browser’s developer tools (F12) and look at the console and network tabs. Search for terms like “webdriver,” “automation,” or “headless” in the console messages. Many sites log these to the console when they detect automation. Also check the navigator.webdriver property in the console — it should return false for a real user, but automation tools often leave it as true.
  3. Hide the WebDriver flag. If navigator.webdriver is true, you need to override it. In Playwright, use addInitScript to redefine the property. In Puppeteer, use page.evaluateOnNewDocument. A common snippet is: Object.defineProperty(navigator, 'webdriver', {get: () => undefined}). Run this before any page scripts execute.
  4. Spoof other navigator properties. Real browsers have consistent values for navigator.plugins, navigator.languages, navigator.platform, and navigator.hardwareConcurrency. Automation tools often return empty or default values. Override these to match a typical desktop browser. For example, set navigator.languages to ['en-US', 'en'] and navigator.plugins to an array with a few common plugin objects.
  5. Use a real browser, not a headless one. Headless browsers are the most heavily detected. If possible, run in headed mode (with a visible window). If you must use headless, use the “new” headless mode (e.g., chromium.launch({headless: 'new'}) in Puppeteer) which is less distinguishable from headed mode. Playwright’s default headless is also improved but still detectable.
  6. Mimic human interaction patterns. Add random delays between actions. Instead of clicking instantly, move the mouse with a slight curve over 200-500 milliseconds. Type text character by character with variable speed, rather than pasting it all at once. Scroll the page in small increments rather than jumping to the bottom. These micro-behaviors are hard for scripts to replicate perfectly but easy for detection systems to spot when missing.
  7. Rotate user agents and viewport sizes. If you’re making multiple requests, vary the user agent string and the viewport dimensions. A real user on a desktop might have a 1920×1080 screen, but another might have 1366×768. Use a pool of realistic user agents from current browser versions and rotate them between sessions.
  8. Handle CAPTCHAs gracefully. When you encounter a CAPTCHA, don’t try to solve it automatically — that’s a separate, complex problem. Instead, detect it (look for elements like #captcha, .g-recaptcha, or iframe with recaptcha) and pause your script. You can notify a human to solve it manually, or wait a random amount of time and retry. Some sites only show CAPTCHAs after a few requests, so slowing down may prevent them entirely.
  9. Respect rate limits. Many anti-bot systems are simply rate limiters. If you’re making requests too quickly, you’ll get blocked. Add a delay between page loads — 2 to 5 seconds is often safe. Use exponential backoff if you get a 429 (Too Many Requests) response. This means waiting 1 second, then 2, then 4, and so on, up to a maximum.
  10. Inspect response headers and status codes. When you get blocked, look at the HTTP response. A 403 Forbidden, 429 Too Many Requests, or a 301 redirect to a challenge page are common. Check the Server header — names like “cloudflare” or “akamai” indicate specific anti-bot services. Read the response body for clues like “Access Denied” or “Please verify you are a human.”
  11. Use a session that persists. Save cookies and local storage between runs. A real user doesn’t re-identify themselves every time. Use the storage state feature in Playwright (context.storageState()) or Puppeteer’s userDataDir to persist the session. This builds a history that looks more legitimate.
  12. Test incrementally. Don’t write a 500-line script and run it blind. Start with a single page load, check if it succeeds. Then add one interaction, test again. If you get blocked, you’ll know exactly which step triggered it, making the fix much easier.

Common Mistakes

  • Overriding too many properties. Changing everything can make your browser look less like a real one. Real browsers have a specific set of values; changing them all to something random is a red flag. Only override what’s actually detected.
  • Ignoring the network layer. Anti-bot systems can check TLS fingerprints, HTTP/2 settings, and header order. If your automation tool uses a different HTTP stack than a real browser, you’ll be detected even with perfect DOM spoofing. Using a real browser via automation (not raw HTTP requests) avoids most of this.
  • Running without delays. Scripts that complete actions in milliseconds are obvious. Even if you don’t get blocked immediately, you’ll accumulate a reputation that triggers blocks later.
  • Assuming “headless” is undetectable. Headless browsers have many detectable differences: missing fonts, different window.chrome object, different rendering behavior. If you must use headless, test thoroughly.
  • Not handling redirects. Some sites redirect bots to a challenge page that looks like the real site. If your script doesn’t check the final URL, you might be scraping a decoy page.

Tips

  • Use a residential proxy. If you’re making many requests, a proxy with residential IPs (not data center IPs) is far less likely to be flagged. Data center IPs are often on blocklists.
  • Keep your browser version current. Anti-bot systems target known vulnerabilities and outdated fingerprints. Using an old browser version makes you easier to detect.
  • Monitor your own behavior. Record the time between your actions and compare it to how a human would behave. If your average time between clicks is 50ms, you’re too fast.
  • Use stealth plugins. Libraries like puppeteer-extra-plugin-stealth handle many of the common detection vectors automatically. They’re not perfect, but they’re a good starting point.
  • Read the site’s robots.txt. If the site explicitly disallows scraping, respect that. You’ll avoid legal issues and wasted effort on a site that’s actively hostile to bots.

Troubleshooting

You’re still getting blocked after applying all the steps. First, verify you’re actually using the overrides. Open the browser console and check navigator.webdriver again — sometimes the override runs after the page script, so it doesn’t take effect. Move your init script to the very beginning of the page load. Second, check if the block is IP-based. Try a different network (like your phone’s hotspot) — if that works, you need a proxy. Third, consider that the site may be using advanced fingerprinting that requires a more sophisticated tool like a real browser profile exported from a normal browsing session.

You’re getting CAPTCHAs on every request. This usually means your IP or browser fingerprint is already flagged. Stop immediately, wait at least an hour, and try again with a fresh browser profile and a different user agent. If it persists, switch to a residential proxy.

The site loads but returns empty data. The anti-bot system might be serving you a blank page or a JavaScript challenge that doesn’t execute properly. Check the page source — if it’s mostly empty or contains challenge scripts, your automation isn’t executing the JavaScript correctly. Ensure your script waits for the network to be idle and for specific elements to appear before scraping.

Frequently Asked Questions

Q: Is it legal to bypass anti-bot measures?
A: It depends on your jurisdiction and the site’s terms of service. Many sites prohibit scraping in their ToS, and bypassing technical measures could violate computer fraud laws. Always check the site’s terms and consider whether your use case is legitimate (e.g., public data for research).

Q: Can I use Selenium for this, or do I need Playwright?
A: Both work. Selenium is older and has more detection issues, but it’s still viable with the right setup. Playwright and Puppeteer have better APIs for stealth techniques like overriding properties before page load.

Q: How do I know if my script is being detected?
A: Look for signs: a CAPTCHA appears, you get a 403 or 429 status code, the page content is different from what a normal browser shows, or your script hangs waiting for elements that never appear. You can also check the console for warnings.

Q: Is there a 100% undetectable method?
A: No. Anti-bot systems evolve constantly, and any method can be detected. The goal is to be “good enough” — to look like a real user so you don’t trigger the thresholds. For high-value targets, you may need to invest in commercial stealth solutions.

Q: Should I use a real browser instead of automation?
A: If you’re doing a one-time scrape, manually copying data may be simpler. For ongoing automation, you need a script. The best approach is to use automation that drives a real browser instance (not a headless one) with realistic settings.

Q: What’s the difference between a bot and a crawler?
A: A crawler is typically a search engine bot that follows links and indexes content. A bot in this context is any automated script. Anti-bot systems often treat search engine crawlers differently — they whitelist known crawler IPs. If you’re building a crawler, you can identify yourself in the user agent, which may avoid blocks.

Related

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →