How to Build a Web Scraper in Python

This guide walks you through building a web scraper in Python from scratch. You’ll learn how to fetch a webpage, parse its HTML, extract the data you want, and save it to a file â€“ all with free, open-source libraries. By the end, you’ll have a working script you can adapt to your own projects.

What You’ll Need

Python 3.7 or newer installed on your computer.
A code editor (VS Code, PyCharm, or even Notepad works).
Basic familiarity with Python syntax (variables, loops, functions).
An internet connection to install packages and fetch pages.

Step-by-Step

Set up a project folder. Create a new directory for your scraper, e.g., my_scraper. Inside it, open a terminal (Command Prompt on Windows, Terminal on macOS/Linux).
Create a virtual environment (optional but recommended). Run python -m venv venv to isolate your project’s dependencies. Activate it: on Windows use venv\Scripts\activate, on macOS/Linux use source venv/bin/activate.
Install the required libraries. Run pip install requests beautifulsoup4 lxml. These are the core tools: requests fetches web pages, BeautifulSoup parses HTML, and lxml speeds up parsing.
Write a basic fetch script. Create a file named scraper.py and start with:
```
import requests
from bs4 import BeautifulSoup

url = "https://example.com"
response = requests.get(url)
print(response.status_code)
```
Run it with python scraper.py. A status code of 200 means the page loaded successfully.
Parse the HTML. Modify your script to create a BeautifulSoup object:
```
soup = BeautifulSoup(response.text, "lxml")
print(soup.title.text)
```
This prints the page’s title, confirming you can access the content.
Inspect the target website. Open the site you want to scrape in your browser, right-click the data you need, and select “Inspect” (or “Inspect Element”). Look for the HTML tags, classes, or IDs that wrap the data. For example, product names might be in <h2 class="product-name">.
Extract specific elements. Use BeautifulSoup’s find() or find_all() to locate elements. For instance:
```
items = soup.find_all("h2", class_="product-name")
for item in items:
    print(item.text.strip())
```
Adjust the tag and class to match your target site.
Handle pagination or multiple pages. If the data spans several pages, loop through them by changing the URL (e.g., ?page=2) or by following “Next” links. Use a for loop and add a small delay (time.sleep(1)) between requests to be polite.

Save the data to a CSV file. Use Python’s built-in csv module:

import csv

with open("output.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.writer(f)
    writer.writerow(["Name", "Price"])
    for item in items:
        writer.writerow([item.text.strip(), price])

Add error handling. Wrap your requests in a try-except block to catch network issues or missing elements:

try:
    response = requests.get(url, timeout=10)
    response.raise_for_status()
except requests.exceptions.RequestException as e:
    print(f"Error: {e}")

Run and test. Execute your script and check the output. If you get no data, print the HTML around the expected elements to debug.

Common Mistakes

Not checking the robots.txt file. Some sites disallow scraping. Always review yourwebsite.com/robots.txt to see which paths are off-limits.
Scraping too fast. Sending hundreds of requests per second can overload a server and get your IP blocked. Always add delays.
Using the wrong parser. If you don’t have lxml installed, BeautifulSoup falls back to Python’s built-in html.parser, which is slower but works. Make sure you installed lxml.
Assuming the HTML structure never changes. Websites update frequently. Your selectors may break; be prepared to adjust them.
Not handling JavaScript-rendered content. Many modern sites load data via JavaScript. requests won’t see that content. You’ll need a tool like Selenium or Playwright, which is beyond this guide’s scope.

Tips

Use CSS selectors for cleaner code. Instead of chaining find() calls, use soup.select("div.product h2") to target nested elements.
Store data incrementally. Write to your CSV inside the loop rather than collecting everything in memory first â€“ this protects your work if the script crashes.
Set a custom User-Agent. Some sites block default Python requests. Add a header like headers = {"User-Agent": "Mozilla/5.0"} to appear as a regular browser.
Use a session for multiple pages. requests.Session() keeps cookies and connection settings, which can help with sites that require login.
Respect the site’s terms. Scrape only public data, and consider contacting the site owner if you need large volumes.

Troubleshooting

Getting a 403 Forbidden error. The site is blocking your request. Add a User-Agent header, or try using a different IP (e.g., via a VPN) if you’re being rate-limited.
No data extracted, but status code is 200. The content might be loaded via JavaScript. Check the raw HTML (print(response.text)) to see if your target data exists. If not, you’ll need a browser automation tool.
AttributeError: ‘NoneType’ object has no attribute ‘text’. Your selector didn’t match anything. Double-check the class name or tag â€“ it might have changed or be case-sensitive.
Encoding issues (garbled text). Set the response encoding explicitly: response.encoding = response.apparent_encoding before parsing.
Timeout errors. Increase the timeout in requests.get() or add retry logic with a loop.

Frequently Asked Questions

Is web scraping legal?

It depends on the website’s terms of service and your jurisdiction. Scraping public data for personal use is generally tolerated, but always check robots.txt and the site’s terms. Avoid scraping personal data or copyrighted content without permission.

Can I scrape sites that require login?

Yes, but you need to handle sessions and cookies. Use requests.Session() to log in first (by posting your credentials to the login form), then make subsequent requests with that session.

Why do I get different results than in my browser?

Your browser executes JavaScript, which requests does not. The HTML you get from requests is the raw source before any JS runs. Use browser dev tools to check the network tab for API endpoints that return JSON data â€“ those are often easier to scrape.

How do I scrape data from multiple pages?

Identify the URL pattern for pagination (e.g., ?page=1, ?page=2) and loop through them. Or find the “Next” button’s link and follow it until it no longer exists.

What if the site blocks my IP?

Slow down your requests, add random delays, and rotate User-Agent strings. For large-scale scraping, consider using proxies, but be aware of legal and ethical boundaries.

Can I scrape data from PDFs or images?

Not directly with BeautifulSoup. You’d need to download the file and use a PDF parser (like PyPDF2) or an OCR library (like Tesseract) to extract text.

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →