Async Scraping with httpx | Snippets

httpx is a modern Python HTTP client that supports async out of the box. Unlike requests, it lets you fire off many requests concurrently without threads, making it ideal for scraping jobs that hit dozens or hundreds of pages.

This snippet creates an async scraper that fetches a list of URLs concurrently with a configurable concurrency limit using asyncio.Semaphore, so you don’t overwhelm the target server.

import asyncio
import httpx

async def fetch_page(client: httpx.AsyncClient, url: str, semaphore: asyncio.Semaphore) -> dict:
    """Fetch a single URL with concurrency control."""
    async with semaphore:
        response = await client.get(url, follow_redirects=True)
        response.raise_for_status()
        return {"url": url, "status": response.status_code, "html": response.text}

async def scrape_all(urls: list[str], max_concurrent: int = 5) -> list[dict]:
    """Scrape multiple URLs concurrently with a connection limit."""
    semaphore = asyncio.Semaphore(max_concurrent)
    # Timeout: 30s connect, 60s read
    timeout = httpx.Timeout(connect=30.0, read=60.0, write=30.0, pool=30.0)
    async with httpx.AsyncClient(timeout=timeout, headers={
        "User-Agent": "Mozilla/5.0 (compatible; MyScraper/1.0)"
    }) as client:
        tasks = [fetch_page(client, url, semaphore) for url in urls]
        # gather with return_exceptions so one failure doesn't cancel all
        results = await asyncio.gather(*tasks, return_exceptions=True)
    # Separate successes from errors
    pages = []
    for result in results:
        if isinstance(result, Exception):
            print(f"Error: {result}")
        else:
            pages.append(result)
    return pages

# Usage
if __name__ == "__main__":
    urls = [f"https://books.toscrape.com/catalogue/page-{i}.html" for i in range(1, 11)]
    pages = asyncio.run(scrape_all(urls, max_concurrent=3))
    print(f"Fetched {len(pages)} pages successfully")

When to use this

Use this pattern when you need to scrape many pages from the same site and want significantly faster execution than sequential requests. The semaphore prevents you from opening too many connections at once, which avoids getting rate-limited or banned. Start with max_concurrent=3-5 and increase carefully.