How to Build a Web Crawler with Headless Chrome

This guide walks you through building a web crawler that uses headless Chrome to fetch and parse pages, handle JavaScript-rendered content, and extract links or data. You’ll end with a working script you can adapt to your own projects. Before starting, you need Node.js installed (version 14 or later) and a basic understanding of JavaScript and the command line.

What You’ll Need

Node.js (v14 or newer) and npm (comes with Node)
A code editor (VS Code, Atom, or any text editor)
Basic familiarity with JavaScript promises and async/await
Internet connection to install packages

Step-by-Step

Set up your project directory. Open a terminal and run mkdir crawler then cd crawler. Initialize a new Node project with npm init -y. This creates a package.json file.
Install Puppeteer. Puppeteer is the official Node library for controlling headless Chrome. Run npm install puppeteer. This will download a compatible version of Chrome automatically (unless you set PUPPETEER_SKIP_DOWNLOAD).
Create the main script. Create a file named crawler.js in your project folder. Open it in your editor.

Import Puppeteer and set up the basic async function. At the top of crawler.js, write:

const puppeteer = require('puppeteer');

async function crawl(url) {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto(url, { waitUntil: 'networkidle2' });
  // ... extraction code goes here
  await browser.close();
}

crawl('https://example.com').catch(console.error);

Extract links and page text. Inside the crawl function, after page.goto, add code to grab all anchor tags and the page’s text content:

const data = await page.evaluate(() => {
  const links = Array.from(document.querySelectorAll('a')).map(a => a.href);
  const text = document.body.innerText;
  return { links, text };
});
console.log(data);

Filter and deduplicate links. To avoid crawling the same page multiple times, keep a Set of visited URLs. Modify the script to only follow links that match the same domain and are HTTP/HTTPS:

const visited = new Set();
const base = new URL(url).origin;

// inside the loop, before visiting a link:
if (link.startsWith(base) && !visited.has(link)) {
  visited.add(link);
  await crawl(link); // recursive call
}

Add a delay between requests. Being polite to servers is important. Use await page.waitForTimeout(1000); after each page load to avoid overwhelming the site.
Handle JavaScript-rendered content. Headless Chrome executes JavaScript, so content that appears after page load will be present. If you need to wait for a specific element, use await page.waitForSelector('.content', { timeout: 5000 }); before extracting.
Run the crawler. In your terminal, execute node crawler.js. You should see the extracted links and text printed to the console.
Save results to a file. Instead of just logging, write the data to a JSON file using Node’s fs module:
```
const fs = require('fs');
fs.writeFileSync('output.json', JSON.stringify(data, null, 2));
```

Common Mistakes

Not waiting for network to be idle – using waitUntil: 'load' may miss dynamically loaded content. Use 'networkidle2' or wait for specific elements.
Forgetting to close the browser – always call browser.close() in a finally block to avoid memory leaks.
Infinite recursion – without a visited set or a depth limit, your crawler may loop forever. Always track visited URLs and set a maximum depth.
Blocked by robots.txt – many sites disallow crawling. Respect robots.txt and check the site’s terms of service.
Using page.evaluate incorrectly – the function runs in the browser context, so you can’t reference Node variables directly unless you pass them as arguments.

Tips

Use concurrency carefully – opening too many tabs at once can crash Chrome. Limit concurrency to 5 or use a library like p-limit.
Set a user agent – some sites block default headless user agents. Use page.setUserAgent('Mozilla/5.0 ...') to mimic a real browser.
Take screenshots for debugging – call await page.screenshot({ path: 'debug.png' }) to see what the page looks like when something goes wrong.
Use page.setViewport – some sites render differently on mobile. Set a desktop viewport for consistent results.
Handle errors gracefully – wrap each page load in a try/catch and log failures instead of crashing the whole crawl.

Troubleshooting

Chrome won’t launch – ensure you have the required system libraries. On Linux, run sudo apt-get install gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils. Or set puppeteer.launch({ headless: false }) to see errors.
Page loads but no data extracted – the content might be inside an iframe or shadow DOM. Try page.frames() to access iframes, or use page.content() to inspect raw HTML.
Script times out – increase the timeout in page.goto (e.g., { timeout: 60000 }) or use waitUntil: 'domcontentloaded' if the page never becomes idle.
Getting 403 or 429 errors – the site is blocking you. Add a delay between requests, rotate user agents, or use a proxy. Respect the site’s rate limits.
Memory usage grows too high – close pages after processing them with await page.close(), and consider using browser.newPage() for each new URL instead of reusing the same page.

Frequently Asked Questions

Q: Can I use headless Chrome without Puppeteer?
A: Yes, you can use Chrome’s remote debugging protocol directly, but Puppeteer simplifies the process significantly. It’s the recommended approach for most users.

Q: How do I crawl a site that requires login?
A: Use page.type() to fill in login forms, or use page.setCookie() to set session cookies after logging in manually once. Store cookies for future runs.

Q: Is it legal to crawl any website?
A: No. Always check the site’s robots.txt and terms of service. Some sites prohibit scraping. Crawling public data for personal use is generally acceptable, but commercial use may require permission.

Q: How do I avoid being blocked?
A: Use a respectful crawl rate (e.g., 1 request per second), rotate user agents, and use proxies if needed. Some sites use CAPTCHAs, which headless Chrome cannot solve automatically.

Q: Can I crawl a single-page application (SPA)?
A: Yes, headless Chrome executes JavaScript, so SPAs work fine. Just ensure you wait for the content to render, often using waitForSelector or networkidle2.

Q: How do I extract data from tables or lists?
A: Use page.evaluate with DOM queries like document.querySelectorAll('table tr') and parse each row into an object. You can also use libraries like Cheerio for parsing after fetching HTML, but that won’t execute JavaScript.

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →