sodcalculator

10 Tech How-To

Scrape Glassdoor Data with Browser Automation Permission Alternatives

Guide 10-33480 · Updated · Step by step

This guide shows you how to collect Glassdoor data using browser automation while respecting their anti-bot measures and staying within legal and ethical bounds. You’ll learn permission alternatives (like using official APIs or manual exports) and safe Playwright practices to avoid being blocked. We cover setup, step-by-step scraping with Playwright, handling login and CAPTCHAs, and how to troubleshoot common issues.

What You’ll Need

  • Node.js (v16 or later) installed on your machine
  • Basic knowledge of JavaScript and command line
  • A text editor (VS Code recommended)
  • Playwright library installed (we’ll show how)
  • A Glassdoor account (for some data, login is required)
  • Understanding of Glassdoor’s Terms of Service – scraping may violate them, so use responsibly

Step-by-Step

  1. Check for official alternatives first. Before scraping, see if Glassdoor offers an official API or data export. As of now, Glassdoor does not have a public API for job or salary data, but they do allow users to export their own saved data. For company reviews, consider using their public pages with proper attribution.
  2. Set up your project. Create a new directory and initialize npm: mkdir glassdoor-scraper && cd glassdoor-scraper && npm init -y. Then install Playwright: npm install playwright. Also install the browser binaries: npx playwright install chromium.
  3. Write a basic Playwright script. Create a file named scrape.js. Start with a simple script that launches a headless browser, goes to Glassdoor, and takes a screenshot to verify it works. Use chromium.launch({ headless: true }) and page.goto('https://www.glassdoor.com').
  4. Handle login (if needed). For pages that require authentication, you’ll need to log in. Use Playwright’s page.fill() and page.click() to enter credentials. To avoid repeated logins, save the storage state: after logging in, use context.storageState({ path: 'state.json' }) and later load it with context = browser.newContext({ storageState: 'state.json' }).
  5. Navigate to the target data. For example, to scrape job listings, go to https://www.glassdoor.com/Job/jobs.htm and enter a search query. Use Playwright to fill the search box and submit. Wait for the results to load using page.waitForSelector('.jobListing') or a similar selector.
  6. Extract the data. Use page.$$eval() to grab elements. For example, to get job titles: const titles = await page.$$eval('.job-title', els => els.map(el => el.textContent.trim())). Collect multiple fields like company, location, salary, and rating.
  7. Pagination. Loop through pages by clicking the next button. Use page.click('.next') or find the pagination selector. Be careful not to click too fast – add a random delay between requests to mimic human behavior.
  8. Respect rate limits. Use page.waitForTimeout() with random delays (e.g., 5-10 seconds) between page loads. This reduces the chance of being blocked.
  9. Handle CAPTCHAs. If you encounter a CAPTCHA, you can try to solve it manually by pausing the script and letting a human solve it. In Playwright, you can use page.pause() to open the inspector and manually interact. Alternatively, use a CAPTCHA solving service, but that may be against Glassdoor’s terms.
  10. Save the data. Write the extracted data to a JSON or CSV file using Node’s fs module. Example: fs.writeFileSync('data.json', JSON.stringify(results)).
  11. Use stealth techniques (optional). To avoid detection, you can add a user agent, viewport size, and locale. Playwright allows you to set these in the browser context. For example: browser.newContext({ userAgent: 'Mozilla/5.0...', viewport: { width: 1920, height: 1080 } }).
  12. Test with a small sample first. Always run your script on a few pages before scaling up. This helps you catch errors and adjust selectors.

Common Mistakes

  • Ignoring Terms of Service. Scraping Glassdoor may violate their ToS. Always check and consider legal alternatives first.
  • Using headless mode too aggressively. Headless browsers are easier to detect. Use headful mode (headless: false) for small tasks, or use stealth plugins.
  • No delays. Sending too many requests too quickly will get your IP blocked. Always add delays.
  • Hardcoding selectors. Glassdoor’s HTML changes frequently. Use robust selectors and test them regularly.
  • Not handling login sessions. If you don’t save storage state, you’ll be logged out and may hit login walls.
  • Scraping without a backup plan. If you get blocked, you need a fallback like using proxies or rotating IPs (but be aware of legal issues).

Tips

  • Use Playwright’s codegen. Run npx playwright codegen to record your actions and generate selectors automatically.
  • Set a realistic user agent. Use a common browser user agent string to appear more human.
  • Randomize your behavior. Vary the time between actions, scroll randomly, and move the mouse occasionally.
  • Use residential proxies (if allowed). If you must scrape at scale, consider using a proxy service to rotate IPs, but ensure you comply with laws and ToS.
  • Cache data. Save each page’s HTML to disk so you can re-parse without re-scraping.
  • Monitor for changes. Regularly check if your selectors still work by running a small test.

Troubleshooting

  • Blocked or 403 errors. You’re being detected. Try using headful mode, adding delays, and using a different user agent. If it persists, wait a few hours and try again.
  • CAPTCHA appears. Pause the script and solve it manually. For automation, you might need to integrate a CAPTCHA solving service, but be aware of ToS.
  • Selectors not finding elements. Glassdoor’s page structure may have changed. Use Playwright’s inspector to re-locate the elements, or use more generic selectors like div[class*='job'].
  • Login fails. Ensure your credentials are correct. If you’re using storage state, make sure the state file is valid and not expired.
  • Page loads slowly. Increase the timeout in page.goto() and use waitUntil: 'networkidle' to ensure the page is fully loaded.
  • Data is empty. Check if the page requires JavaScript to render data. If so, wait for specific elements to appear before extracting.

Frequently Asked Questions

Is scraping Glassdoor legal?

It depends on your jurisdiction and Glassdoor’s Terms of Service. Scraping public data for personal use may be tolerated, but commercial use or bypassing login walls is likely prohibited. Always consult a lawyer.

Can I use Glassdoor’s official API?

Glassdoor does not currently offer a public API for job or salary data. They have a partner program, but it’s not open to the public.

How do I avoid getting blocked?

Use delays, rotate user agents, and limit request frequency. Also, use headful mode and consider using proxies if you need to scale.

What data can I scrape from Glassdoor?

You can scrape job titles, company names, locations, salaries (if shown), ratings, and review text. However, some data may be behind login or require premium access.

How do I handle CAPTCHAs in Playwright?

You can pause the script and solve it manually, or integrate a CAPTCHA solving service. Some services offer APIs that can be called from your script.

Can I scrape Glassdoor without login?

Yes, for some public pages like job listings, login may not be required. But for detailed salary or review data, you’ll likely need to log in.

Related

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →