Browser Automation Scripts: Step-by-Step Guide

This guide walks you through writing scripts that automate repetitive browser tasks like form filling, data scraping, or testing. You’ll learn the core concepts, practical steps, and common pitfalls—from setting up your environment to debugging flaky selectors.

What You’ll Need

Node.js (LTS version) installed on your computer.
A code editor (VS Code, Sublime Text, or any editor you prefer).
Basic familiarity with JavaScript syntax (variables, functions, async/await).
A test website to practice on—your own local HTML file or a public demo site.

Step-by-Step

Choose your tool. For most browser automation, Puppeteer or Playwright are the go-to libraries. Playwright supports multiple browsers (Chrome, Firefox, WebKit) and has a simpler API for waiting and assertions. Puppeteer is Chrome-only but lighter. This guide uses Playwright, but the concepts apply to both.
Set up a new project. Create a folder, open a terminal in it, and run npm init -y. Then install Playwright with npm install playwright. After that, run npx playwright install to download the browser binaries.

Write your first script. Create a file named script.js. Start with the minimal boilerplate:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com');
  console.log(await page.title());
  await browser.close();
})();

Run it with node script.js. If you see the page title printed, your setup works.

Identify elements to interact with. Open the page in your regular browser, right-click the element you want to click or type into, and select “Inspect”. Look for stable attributes: id, name, or data-testid. Avoid classes that look generated like css-1a2b3c.
Add interactions. Use page.fill(selector, text) for text inputs, page.click(selector) for buttons, and page.selectOption(selector, value) for dropdowns. For example:
```
await page.fill('#username', 'myuser');
await page.fill('#password', 'mypassword');
await page.click('button[type="submit"]');
```
Wait for results. After clicking, the page may take time to load. Use await page.waitForSelector('.success-message') or await page.waitForURL('**/dashboard'). Avoid hardcoded setTimeout—they’re brittle and slow down your script.
Extract data. To scrape information, use page.textContent(selector) for a single element or page.$$eval(selector, elements => elements.map(e => e.textContent)) for multiple. For example, to get all product names:
```
const names = await page.$$eval('.product-name', els => els.map(e => e.textContent.trim()));
console.log(names);
```
Handle navigation and multiple pages. If your task spans several pages, use page.goto() for direct URLs, or click links and wait for the next page to load. For multi-tab scenarios, listen for the popup event:
```
const [popup] = await Promise.all([
  page.waitForEvent('popup'),
  page.click('a[target="_blank"]')
]);
await popup.waitForLoadState();
```
Make your script robust. Use try/catch blocks to handle errors gracefully. Set timeouts on actions: page.click(selector, { timeout: 5000 }). Log each step to the console so you know where the script failed.
Run headless vs. headed. For debugging, launch with headless: false to see the browser in action. For production, keep headless (default) for speed. You can also slow down with slowMo: 100 in the launch options.
Store credentials securely. Never hardcode passwords. Use environment variables (process.env.USERNAME) or a separate config file that is not committed to version control.
Test and iterate. Run your script multiple times. If it fails on a selector, revisit step 4. If it’s slow, look for unnecessary waits. Once stable, consider adding a scheduling tool like cron (macOS/Linux) or Task Scheduler (Windows) to run it automatically.

Common Mistakes

Using CSS classes that change. Generated classes (like _3xK9f) break your script after a site update. Prefer IDs, names, or text-based selectors.
Not waiting for elements. Clicking a button before it’s enabled, or reading text before it appears, causes flaky failures. Always use waitForSelector or waitForLoadState.
Hardcoding sleeps. setTimeout(5000) is a guess. If the page loads faster, you waste time; if slower, you get a failure. Use explicit waits instead.
Ignoring iframes. Elements inside an <iframe> are not directly accessible. You need page.frameLocator('iframe#id').locator(...) to interact with them.
Forgetting to close the browser. If your script crashes before browser.close(), you’ll have orphan processes. Use try/finally or the process.on('exit') handler.
Assuming the same selector works on all pages. Single-page apps often reuse classes. Verify your selector is unique on each page you navigate to.

Tips

Use Playwright’s code generator. Run npx playwright codegen to open a browser where you can click around—it writes the script for you. Then refine it.
Keep selectors simple and semantic. Use getByRole('button', { name: 'Submit' }) instead of long CSS paths. It’s more readable and resilient.
Break your script into functions. One function for login, one for scraping, one for logout. This makes debugging easier and allows you to reuse parts.
Take screenshots on failure. In a catch block, call page.screenshot({ path: 'error.png' }) to see exactly what the browser saw.
Use environment variables for anything site-specific. URLs, credentials, and test data should be configurable, not hardcoded.
Version-control your scripts. Even small scripts benefit from Git. You can roll back when a site update breaks your automation.

Troubleshooting

Script fails with “Element not found” — The selector is wrong or the element loads late. First, copy the selector from DevTools and test it in the browser console with document.querySelector. Then add a longer wait or use waitForSelector with a timeout. If the element is inside a shadow DOM or iframe, adjust your approach.

Script works once, then fails — The site may have changed (new class names) or your script left state behind. Clear cookies between runs: context.clearCookies(). Also check if the site rotates content—use a more generic selector.

Browser opens but nothing happens — Check your console for errors. The most common cause is a typo in the URL or the page not loading due to network issues. Add await page.goto(url, { waitUntil: 'networkidle' }) to ensure the page is fully loaded.

Script is too slow — Remove unnecessary waits. Replace waitForTimeout with waitForSelector. If you’re scraping many pages, consider using parallel contexts (but be careful with rate limits).

Captcha or bot detection appears — Some sites block automated browsers. You can try using a real user-agent string, disable the headless mode, or add a human-like delay between actions. If the site requires solving a captcha, you may need to use a captcha-solving service, but that’s beyond this guide’s scope.

Script runs but doesn’t produce expected output — Add console.log statements at each step to trace the flow. Print the page’s HTML with page.content() to see what the browser actually sees. Often the data is there but your selector is off by one level.

Frequently Asked Questions

Do I need to know JavaScript to write automation scripts? Yes, most browser automation libraries are JavaScript-based. You’ll need to understand async/await, functions, and basic DOM concepts. If you prefer Python, check out Selenium or Playwright’s Python version—the concepts are similar.

Can I automate a site that requires login? Yes, you can fill in the login form just like a user would. For repeated runs, store the session cookies after the first login and reuse them to avoid logging in every time.

Is browser automation legal? It depends on the site’s terms of service. Automating public data for personal use is usually fine, but scraping behind a login or at high volume may violate terms. Always check the site’s robots.txt and terms before automating.

What’s the difference between Puppeteer and Playwright? Puppeteer is older, Chrome-only, and slightly simpler. Playwright supports all major browsers and has better built-in waiting and test features. For new projects, Playwright is generally recommended.

How do I run my script on a schedule? On Windows, use Task Scheduler to run your Node.js script at specific times. On macOS/Linux, use cron jobs. Alternatively, you can use a CI service like GitHub Actions to run it on a schedule.

Can I automate a mobile browser? Yes, Playwright can emulate mobile devices by setting the viewport and user agent. For real mobile testing, you’d need a device farm, but for most automation tasks, emulation is sufficient.

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →