Scrape Dynamic Websites with JavaScript: A How-To

This guide walks you through scraping websites that load content dynamically via JavaScript, such as infinite-scroll feeds, single-page apps, or pages with content revealed after user interaction. You’ll learn how to use headless browsers and automation tools to extract data that traditional HTTP requests can’t reach. By the end, you’ll have a working script that can handle modern, JavaScript-heavy sites.

What You’ll Need

Node.js (version 14 or later) installed on your computer
A code editor (like VS Code)
Basic familiarity with JavaScript and the command line
Access to the website you want to scrape (and permission to do so, per its terms of service)

Step-by-Step

Set up your project. Create a new folder for your scraper and open it in your terminal. Run npm init -y to create a package.json file.
Install Puppeteer. Puppeteer is a Node.js library that controls a headless Chrome browser. Install it with npm install puppeteer. This will download a compatible version of Chrome for you.

Write a basic script. Create a file named scraper.js. Start with the following boilerplate:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({ headless: true });
  const page = await browser.newPage();
  await page.goto('https://example.com', { waitUntil: 'networkidle2' });
  // Your scraping logic goes here
  await browser.close();
})();

Replace the URL with the site you’re targeting.

Wait for dynamic content. If the content loads after the initial page load, use await page.waitForSelector('your-selector') to wait for a specific element to appear. For example, await page.waitForSelector('.product-item').
Extract data. Use page.evaluate() to run JavaScript in the page context and pull out the data you need. For example:
```
const data = await page.evaluate(() => {
  const items = document.querySelectorAll('.product-item');
  return Array.from(items).map(item => item.textContent.trim());
});
```
This returns an array of text content from each product item.
Handle infinite scroll. If the page loads more content as you scroll, you’ll need to automate scrolling. Use a loop that scrolls to the bottom, waits for new content, and repeats until a condition is met (e.g., a maximum number of scrolls or no new elements):
```
for (let i = 0; i < 5; i++) {
  await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
  await page.waitForTimeout(2000); // wait for new content to load
}
```
Adjust the number of iterations and wait time based on the site’s behavior.
Click buttons to load more. Some sites use a “Load More” button instead of infinite scroll. Click it with await page.click('button.load-more'), then wait for the new content to appear.

Interact with forms or dropdowns. If the content depends on user input, use Puppeteer’s type and click methods. For example:

await page.type('input#search', 'laptops');
await page.click('button[type="submit"]');
await page.waitForSelector('.results');

Save your data. Once you’ve extracted the data, write it to a file. You can use fs.writeFileSync('data.json', JSON.stringify(data, null, 2)) to save it as JSON.
Run your script. Execute node scraper.js in your terminal. If everything works, you’ll see the data in your JSON file.

Common Mistakes

Not waiting long enough. Dynamic content can be slow. If you don’t wait for the right selector, you’ll get empty results.
Using the wrong selector. Selectors that work in your browser’s DevTools may not match the same elements in headless mode, especially if the page uses iframes or shadow DOM.
Scraping too fast. Sending too many requests in a short time can get your IP blocked. Add delays between requests.
Ignoring the site’s terms of service. Scraping may be prohibited by the website. Always check robots.txt and the terms of service.
Forgetting to close the browser. If your script errors out, the browser process may stay open. Use try...finally to ensure browser.close() runs.

Tips

Use headless: false during development. Set headless: false in puppeteer.launch() to see what the browser is doing. This helps debug selectors and timing issues.
Set a realistic user agent. Some sites block default headless user agents. Set one with page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36').
Take screenshots. Use await page.screenshot({ path: 'screenshot.png' }) to see what the page looks like at any point.
Use page.waitForTimeout() sparingly. It’s better to wait for a specific selector than to rely on fixed delays, which can be unreliable.
Handle pagination. If the site uses pagination, loop through pages by changing the URL or clicking next buttons.

Troubleshooting

Content doesn’t appear. If your selectors return nothing, try increasing the wait time, or use page.waitForFunction() to wait for a specific condition. Also check if the content is inside an iframe—you’ll need to switch to the iframe with page.frames().find(frame => frame.url().includes('...')).

Page loads but script times out. This often happens with waitUntil: 'networkidle2' on sites with continuous network requests. Try waitUntil: 'domcontentloaded' instead, and then wait for your selector.

Browser crashes. If Chrome crashes, try puppeteer.launch({ args: ['--no-sandbox', '--disable-setuid-sandbox'] }) on Linux, or update Puppeteer to the latest version.

Data is incomplete. Some sites lazy-load images or content only when they come into view. Scroll to the element before extracting, or use page.hover() to trigger mouse-over events.

Your IP gets blocked. If you receive 403 errors, slow down your requests, use a proxy, or rotate user agents. Respect the site’s rate limits.

Frequently Asked Questions

Q: Is it legal to scrape dynamic websites?
A: It depends on the website’s terms of service and your jurisdiction. Always check the site’s robots.txt and terms. Scraping public data for personal use is often tolerated, but commercial use may require permission.

Q: Can I use Puppeteer with other languages?
A: Puppeteer is JavaScript-only, but there are similar tools for other languages, like Playwright (Python, Java, .NET) or Selenium (many languages). The concepts are the same.

Q: How do I scrape a site that requires login?
A: You can use page.type() to enter credentials and page.click() to submit the form. Make sure you have permission to access the data behind the login.

Q: What’s the difference between Puppeteer and a simple HTTP request?
A: A simple HTTP request (like using fetch in Node.js) only gets the raw HTML, which doesn’t include content rendered by JavaScript. Puppeteer runs the actual browser, so it executes JavaScript and gives you the fully rendered page.

Q: Can I scrape a site that uses infinite scroll without scrolling?
A: No, you need to trigger the scroll or click to load more content. The data isn’t in the initial HTML; it’s fetched via AJAX as you scroll.

Q: How do I avoid getting blocked?
A: Use delays between requests, rotate user agents, use proxies if needed, and limit your request rate. Also, consider using the site’s official API if one exists.

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →