This tutorial takes you from zero to a working web scraper. By the end, you’ll have a Python script that fetches web pages, extracts structured data from HTML, handles errors, and saves results to a file. We’ll use a public practice website designed specifically for learning scraping, so you can experiment without worrying about rate limits or terms of service.

What You’ll Build

A scraper that extracts book data (title, price, rating, availability) from Books to Scrape, a sandbox site built for scraping practice. The final script will:

Fetch HTML pages over HTTP
Parse the HTML and extract specific data points
Follow pagination links to scrape all 1,000 books
Save the results to a CSV file
Handle errors without crashing

Prerequisites

Python 3.8 or newer installed on your machine
A terminal (Terminal on macOS/Linux, PowerShell or Command Prompt on Windows)
A text editor (VS Code, Sublime Text, or anything you’re comfortable with)
Basic Python knowledge — variables, loops, functions, and import statements

No prior scraping experience needed. We’ll explain every concept as we go.

Step 1: Set Up Your Environment

Create a Project Directory

mkdir first-scraper
cd first-scraper

Create and Activate a Virtual Environment

Virtual environments keep your project’s dependencies isolated from other Python projects on your machine.

# Create the virtual environment
python -m venv venv

# Activate it
# macOS/Linux:
source venv/bin/activate

# Windows (PowerShell):
venv\Scripts\Activate.ps1

# Windows (Command Prompt):
venv\Scripts\activate.bat

You’ll know it’s active when your terminal prompt shows (venv) at the beginning.

Install Dependencies

We need two packages:

requests: Makes HTTP requests (fetches web pages)
beautifulsoup4: Parses HTML and lets you search through it

pip install requests beautifulsoup4

Create Your Script File

touch scraper.py

Open scraper.py in your editor. This is where all our code will go.

Step 2: Fetch a Web Page

Let’s start with the most basic operation: downloading an HTML page.

# scraper.py — Step 2
import requests

url = "https://books.toscrape.com"
response = requests.get(url, timeout=30)

print(f"Status code: {response.status_code}")
print(f"Content length: {len(response.text)} characters")
print(f"First 500 characters:\n{response.text[:500]}")

Run it:

python scraper.py

You should see output like:

Status code: 200
Content length: 51258 characters
First 500 characters:
<!DOCTYPE html>
<!--[if lt IE 7]>      <html lang="en-us" class="no-js lt-ie9 ...

What’s happening:

requests.get(url) sends an HTTP GET request to the URL — the same thing your browser does when you visit a page
The server responds with an HTTP status code (200 means success) and the HTML content
response.text contains the full HTML as a string

Understanding HTTP Status Codes

Code	Meaning	What to do
200	Success	Parse the HTML
301/302	Redirect	`requests` follows these automatically
403	Forbidden	The site blocked you — check your headers
404	Not found	The URL doesn’t exist
429	Too many requests	You’re scraping too fast — add delays
500	Server error	The site is having problems — retry later

Step 3: Parse HTML with BeautifulSoup

Raw HTML is just a big string. BeautifulSoup turns it into a navigable tree structure.

# scraper.py — Step 3
import requests
from bs4 import BeautifulSoup

url = "https://books.toscrape.com"
response = requests.get(url, timeout=30)

# Parse the HTML
soup = BeautifulSoup(response.text, "html.parser")

# Find elements by CSS selector
page_title = soup.select_one("title").text
print(f"Page title: {page_title}")

# Find all book titles on the page
books = soup.select("article.product_pod")
print(f"\nFound {len(books)} books on this page:\n")

for book in books[:5]:  # Show first 5
    title = book.select_one("h3 a")["title"]
    price = book.select_one("p.price_color").text
    print(f"  {title} — {price}")

Run it:

Page title: All products | Books to Scrape - Sandbox

Found 20 books on this page:

  A Light in the Attic — £51.77
  Tipping the Velvet — £53.74
  Soumission — £50.10
  Sharp Objects — £47.82
  Sapiens: A Brief History of Humankind — £54.23

How CSS Selectors Work

CSS selectors are patterns that match HTML elements. If you’ve written CSS for a website, you already know these. If not, here are the essentials:

Selector	Matches	Example
`tag`	Element by tag name	`h1`, `p`, `div`
`.class`	Element by class	`.price_color`, `.product_pod`
`#id`	Element by ID	`#search-bar`
`tag.class`	Tag with specific class	`article.product_pod`
`parent child`	Child inside parent	`article h3`
`tag[attr]`	Tag with attribute	`a[href]`
`tag::text`	Text content (Scrapy only)	N/A in BS4

Two key methods:

soup.select("selector") — returns a list of all matching elements
soup.select_one("selector") — returns the first matching element (or None)

Finding the Right Selectors

How do you figure out which selectors to use? Browser DevTools. Here’s the process:

Open the target page in your browser
Right-click the data you want to extract
Click “Inspect” (or “Inspect Element”)
Look at the HTML structure in the Elements panel
Identify the tag names, class names, and nesting

For example, on Books to Scrape, each book is wrapped in:

<article class="product_pod">
  <h3><a href="..." title="A Light in the Attic">A Light in the ...</a></h3>
  <div class="product_price">
    <p class="price_color">£51.77</p>
    <p class="instock availability">
      <i class="icon-ok"></i> In stock
    </p>
  </div>
</article>

From this HTML, we can build our selectors:

All books: article.product_pod
Book title: h3 a (get the title attribute for the full name)
Price: p.price_color (get the text content)
Availability: p.availability (get the text content)

Step 4: Extract Structured Data

Now let’s build a proper extraction function that pulls out all the data we want from each book.

# scraper.py — Step 4
import requests
from bs4 import BeautifulSoup

def extract_books(soup):
    """Extract book data from a parsed page."""
    books = []

    for article in soup.select("article.product_pod"):
        # Title: stored in the "title" attribute of the link
        title = article.select_one("h3 a")["title"]

        # Price: text content of the price element, strip the pound sign
        price_text = article.select_one("p.price_color").text.strip()
        price = float(price_text.replace("£", ""))

        # Rating: stored as a CSS class like "star-rating Three"
        rating_class = article.select_one("p.star-rating")["class"]
        rating = rating_class[1]  # ["star-rating", "Three"] → "Three"

        # Availability: check if "In stock" appears in the text
        avail_text = article.select_one("p.availability").text.strip()
        in_stock = "In stock" in avail_text

        books.append({
            "title": title,
            "price": price,
            "rating": rating,
            "in_stock": in_stock,
        })

    return books


# Fetch and parse
url = "https://books.toscrape.com"
response = requests.get(url, timeout=30)
soup = BeautifulSoup(response.text, "html.parser")

# Extract
books = extract_books(soup)

print(f"Extracted {len(books)} books:\n")
for book in books[:5]:
    stock = "In stock" if book["in_stock"] else "Out of stock"
    print(f"  {book['title']}")
    print(f"    Price: £{book['price']:.2f} | Rating: {book['rating']} | {stock}")
    print()

Run it:

Extracted 20 books:

  A Light in the Attic
    Price: £51.77 | Rating: Three | In stock

  Tipping the Velvet
    Price: £53.74 | Rating: One | In stock

  Soumission
    Price: £50.10 | Rating: One | In stock

Key techniques used:

Attribute access: element["attribute"] gets an HTML attribute (like href or title)
Text content: element.text gets the visible text inside an element
String cleaning: .strip() removes whitespace, .replace() removes unwanted characters
Type conversion: float() converts the price string to a number

Step 5: Follow Pagination

The site has 50 pages of books (1,000 books total). Let’s scrape them all by following the “next” link on each page.

# scraper.py — Step 5
import requests
from bs4 import BeautifulSoup
import time

BASE_URL = "https://books.toscrape.com/catalogue"

def extract_books(soup):
    """Extract book data from a parsed page."""
    books = []

    for article in soup.select("article.product_pod"):
        title = article.select_one("h3 a")["title"]

        price_text = article.select_one("p.price_color").text.strip()
        price = float(price_text.replace("£", ""))

        rating_class = article.select_one("p.star-rating")["class"]
        rating = rating_class[1]

        avail_text = article.select_one("p.availability").text.strip()
        in_stock = "In stock" in avail_text

        books.append({
            "title": title,
            "price": price,
            "rating": rating,
            "in_stock": in_stock,
        })

    return books


def get_next_page_url(soup):
    """Find the URL of the next page, or None if we're on the last page."""
    next_btn = soup.select_one("li.next a")
    if next_btn:
        return f"{BASE_URL}/{next_btn['href']}"
    return None


def scrape_all_books():
    """Scrape all books from all pages."""
    url = f"{BASE_URL}/page-1.html"
    all_books = []
    page_num = 1

    while url:
        print(f"Page {page_num}: {url}")

        response = requests.get(url, timeout=30)
        response.raise_for_status()

        soup = BeautifulSoup(response.text, "html.parser")
        books = extract_books(soup)
        all_books.extend(books)

        print(f"  → Extracted {len(books)} books (total: {len(all_books)})")

        url = get_next_page_url(soup)
        page_num += 1

        # Be polite: wait 1 second between requests
        if url:
            time.sleep(1)

    return all_books


books = scrape_all_books()
print(f"\nDone! Scraped {len(books)} books total.")

Important: time.sleep(1)

This 1-second delay between requests matters. Without it, your script would fire 50 requests as fast as your connection allows — potentially overwhelming the server. Even for a practice site, it’s good to build this habit. For real websites, check robots.txt for a Crawl-delay directive and respect it.

Run it:

Page 1: https://books.toscrape.com/catalogue/page-1.html
  → Extracted 20 books (total: 20)
Page 2: https://books.toscrape.com/catalogue/page-2.html
  → Extracted 20 books (total: 40)
...
Page 50: https://books.toscrape.com/catalogue/page-50.html
  → Extracted 20 books (total: 1000)

Done! Scraped 1000 books total.

Step 6: Save Results to CSV

Data in memory is useless after the script exits. Let’s save it to a CSV file.

# Add this import at the top
import csv

def save_to_csv(books, filename="books.csv"):
    """Save book data to a CSV file."""
    if not books:
        print("No books to save.")
        return

    fieldnames = ["title", "price", "rating", "in_stock"]

    with open(filename, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=fieldnames)
        writer.writeheader()
        writer.writerows(books)

    print(f"Saved {len(books)} books to {filename}")

Add this call at the end of the script:

books = scrape_all_books()
save_to_csv(books)

The output file will look like:

title,price,rating,in_stock
A Light in the Attic,51.77,Three,True
Tipping the Velvet,53.74,One,True
Soumission,50.1,One,True

You can open this in Excel, Google Sheets, pandas, or any tool that reads CSV.

Step 7: Add Error Handling

Real-world scraping involves failures: network timeouts, missing elements, server errors. Let’s make the scraper resilient.

# scraper.py — Final version
import requests
from bs4 import BeautifulSoup
import csv
import time
import sys

BASE_URL = "https://books.toscrape.com/catalogue"
DELAY_SECONDS = 1
MAX_RETRIES = 3


def fetch_page(url):
    """Fetch a page with retry logic."""
    for attempt in range(1, MAX_RETRIES + 1):
        try:
            response = requests.get(url, timeout=30)
            response.raise_for_status()
            return response
        except requests.exceptions.RequestException as e:
            print(f"  Attempt {attempt}/{MAX_RETRIES} failed: {e}")
            if attempt < MAX_RETRIES:
                wait = attempt * 2  # exponential-ish backoff: 2s, 4s
                print(f"  Retrying in {wait} seconds...")
                time.sleep(wait)

    print(f"  Failed to fetch {url} after {MAX_RETRIES} attempts. Skipping.")
    return None


def extract_books(soup):
    """Extract book data from a parsed page."""
    books = []

    for article in soup.select("article.product_pod"):
        try:
            title_el = article.select_one("h3 a")
            title = title_el["title"] if title_el else "Unknown"

            price_el = article.select_one("p.price_color")
            if price_el:
                price = float(price_el.text.strip().replace("£", ""))
            else:
                price = 0.0

            rating_el = article.select_one("p.star-rating")
            rating = rating_el["class"][1] if rating_el else "Unknown"

            avail_el = article.select_one("p.availability")
            in_stock = "In stock" in avail_el.text if avail_el else False

            books.append({
                "title": title,
                "price": price,
                "rating": rating,
                "in_stock": in_stock,
            })
        except (KeyError, IndexError, ValueError) as e:
            print(f"  Warning: Could not extract a book: {e}")
            continue

    return books


def get_next_page_url(soup):
    """Find the URL of the next page, or None if we're on the last page."""
    next_btn = soup.select_one("li.next a")
    if next_btn:
        return f"{BASE_URL}/{next_btn['href']}"
    return None


def save_to_csv(books, filename="books.csv"):
    """Save book data to a CSV file."""
    if not books:
        print("No books to save.")
        return

    fieldnames = ["title", "price", "rating", "in_stock"]

    with open(filename, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=fieldnames)
        writer.writeheader()
        writer.writerows(books)

    print(f"Saved {len(books)} books to {filename}")


def scrape_all_books():
    """Scrape all books from all pages."""
    url = f"{BASE_URL}/page-1.html"
    all_books = []
    page_num = 1

    while url:
        print(f"Page {page_num}: {url}")

        response = fetch_page(url)
        if response is None:
            # Skip this page but continue with next if possible
            break

        soup = BeautifulSoup(response.text, "html.parser")
        books = extract_books(soup)
        all_books.extend(books)

        print(f"  Extracted {len(books)} books (total: {len(all_books)})")

        url = get_next_page_url(soup)
        page_num += 1

        if url:
            time.sleep(DELAY_SECONDS)

    return all_books


def main():
    print("Starting scraper...")
    print(f"Target: {BASE_URL}")
    print(f"Delay: {DELAY_SECONDS}s between requests")
    print(f"Max retries: {MAX_RETRIES}")
    print()

    start_time = time.time()
    books = scrape_all_books()
    elapsed = time.time() - start_time

    print(f"\nScraping complete in {elapsed:.1f} seconds")
    print(f"Total books: {len(books)}")

    if books:
        save_to_csv(books)

        # Quick stats
        prices = [b["price"] for b in books]
        print(f"\nQuick stats:")
        print(f"  Average price: £{sum(prices) / len(prices):.2f}")
        print(f"  Most expensive: £{max(prices):.2f}")
        print(f"  Cheapest: £{min(prices):.2f}")
        print(f"  In stock: {sum(1 for b in books if b['in_stock'])}/{len(books)}")


if __name__ == "__main__":
    main()

What the Error Handling Does

fetch_page() with retries:

If a request fails (timeout, connection error, HTTP error), it waits and tries again
Each retry waits longer than the last (2 seconds, then 4 seconds)
After 3 failed attempts, it gives up and returns None

extract_books() with try/except:

If an individual book’s HTML is malformed or missing expected elements, it logs a warning and moves on to the next book
One bad book doesn’t crash the entire scrape

Graceful None handling:

Every select_one() call is checked for None before accessing attributes
Missing data gets a default value instead of crashing

Step 8: Run the Complete Scraper

python scraper.py

Expected output:

Starting scraper...
Target: https://books.toscrape.com/catalogue
Delay: 1s between requests
Max retries: 3

Page 1: https://books.toscrape.com/catalogue/page-1.html
  Extracted 20 books (total: 20)
Page 2: https://books.toscrape.com/catalogue/page-2.html
  Extracted 20 books (total: 40)
...
Page 50: https://books.toscrape.com/catalogue/page-50.html
  Extracted 20 books (total: 1000)

Scraping complete in 52.3 seconds
Total books: 1000
Saved 1000 books to books.csv

Quick stats:
  Average price: £35.07
  Most expensive: £59.99
  Cheapest: £10.00
  In stock: 1000/1000

Troubleshooting

”ModuleNotFoundError: No module named ‘bs4’”

Your virtual environment isn’t active, or you installed the packages globally. Run:

# Make sure venv is active (you should see (venv) in your prompt)
source venv/bin/activate  # macOS/Linux
pip install beautifulsoup4 requests

“ConnectionError: Max retries exceeded”

The website might be down, or your internet connection is unstable. Check:

Can you visit the URL in your browser?
Is your firewall blocking Python?
Are you behind a corporate proxy? You may need to configure requests to use it.

”SSLError: certificate verify failed”

Your Python installation might have outdated CA certificates. Fix:

# macOS
pip install certifi
# or
/Applications/Python\ 3.x/Install\ Certificates.command

The scraper gets different HTML than what you see in the browser

Some sites serve different HTML to different user-agents. Add a realistic User-Agent header:

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
}
response = requests.get(url, headers=headers, timeout=30)

Note: Books to Scrape doesn’t do this, but many real sites do.

The scraper runs but extracts 0 books

Your CSS selectors don’t match the current HTML. Websites change their HTML structure. Open the page in your browser, inspect the elements, and update the selectors.

The script crashes midway through

If your scraper crashes after scraping 500 of 1000 books, you lose all data. To avoid this, save incrementally:

# Save after each page instead of at the end
def scrape_all_books():
    url = f"{BASE_URL}/page-1.html"
    all_books = []
    page_num = 1

    while url:
        response = fetch_page(url)
        if response is None:
            break

        soup = BeautifulSoup(response.text, "html.parser")
        books = extract_books(soup)
        all_books.extend(books)

        # Save progress after every page
        save_to_csv(all_books)

        url = get_next_page_url(soup)
        page_num += 1

        if url:
            time.sleep(DELAY_SECONDS)

    return all_books

What You’ve Learned

After completing this tutorial, you can:

Fetch web pages using requests and understand HTTP status codes
Parse HTML with BeautifulSoup and navigate the document tree
Write CSS selectors to target specific elements on a page
Extract data from HTML attributes and text content
Follow pagination to scrape across multiple pages
Save structured data to CSV files
Handle errors with retries, fallbacks, and defensive parsing

Next Steps

This scraper works, but it’s sequential and basic. Here’s where to go from here:

Respect robots.txt — Learn how to check robots.txt before scraping (critical for real websites)
Try Scrapy — When your scraping needs grow beyond a single script, Scrapy provides a complete framework
Handle JavaScript — Some sites render content with JavaScript. Tools like Playwright or Selenium can handle this
Store data in databases — For larger datasets, CSV files become unwieldy. SQLite or PostgreSQL are better options
Add proxy support — For large-scale scraping, proxy rotation prevents IP bans

On This Page

Building Your First Web Scraper with Python

What You’ll Build

Prerequisites

Step 1: Set Up Your Environment

Create a Project Directory

Create and Activate a Virtual Environment

Install Dependencies

Create Your Script File

Step 2: Fetch a Web Page

Understanding HTTP Status Codes

Step 3: Parse HTML with BeautifulSoup

How CSS Selectors Work

Finding the Right Selectors

Step 4: Extract Structured Data

Step 6: Save Results to CSV

Step 7: Add Error Handling

What the Error Handling Does

Step 8: Run the Complete Scraper

Troubleshooting

”ModuleNotFoundError: No module named ‘bs4’”

“ConnectionError: Max retries exceeded”

”SSLError: certificate verify failed”

The scraper gets different HTML than what you see in the browser

The scraper runs but extracts 0 books

The script crashes midway through

What You’ve Learned

Next Steps

Comments