Data Extraction with lxml cssselect | Snippets

lxml is a C-based XML/HTML parser that’s significantly faster than BeautifulSoup for large documents. Its cssselect module lets you use familiar CSS selectors to locate elements, making the transition from browser DevTools to Python code nearly seamless.

This snippet shows how to parse an HTML page, extract a list of items with nested fields, and handle missing elements gracefully.

from lxml import html
import httpx

def extract_products(page_html: str) -> list[dict]:
    """Extract product data from HTML using CSS selectors."""
    tree = html.fromstring(page_html)
    products = []

    # Select every product card on the page
    for card in tree.cssselect("article.product_pod"):
        # Extract title from the <a> inside <h3>
        title_el = card.cssselect("h3 a")
        title = title_el[0].get("title", "").strip() if title_el else "Unknown"

        # Extract price — grab text content, strip currency symbol
        price_el = card.cssselect("p.price_color")
        price_text = price_el[0].text_content().strip() if price_el else "0"
        price = float(price_text.replace("£", "").replace("$", ""))

        # Extract star rating from class name (e.g., "star-rating Three")
        rating_el = card.cssselect("p.star-rating")
        rating_class = rating_el[0].get("class", "") if rating_el else ""
        rating_map = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}
        rating = next((v for k, v in rating_map.items() if k in rating_class), 0)

        # Check stock availability
        stock_el = card.cssselect("p.instock")
        in_stock = bool(stock_el)

        products.append({
            "title": title,
            "price": price,
            "rating": rating,
            "in_stock": in_stock,
        })

    return products

# Usage
if __name__ == "__main__":
    resp = httpx.get("https://books.toscrape.com/")
    products = extract_products(resp.text)
    for p in products[:5]:
        print(f"{p['title'][:40]:40s} £{p['price']:5.2f}  {'★' * p['rating']}")

When to use this

Choose lxml + cssselect over BeautifulSoup when you need raw parsing speed — it’s 5-10x faster on large documents. It’s ideal for high-volume pipelines where you’re parsing thousands of pages. Use BeautifulSoup instead when you need its more forgiving broken-HTML handling or its simpler API for one-off scripts.