This tutorial walks you through installing ScraperGuard, defining a schema for your scraped data, running your first health check, and reading the report. By the end, you’ll have a working validation layer you can add to any scraper.

Step 1: Install ScraperGuard

Clone the repository and install in development mode:

git clone https://github.com/mostafam-dev/scraperguard.git
cd scraperguard
pip install -e .

Verify the installation:

scraperguard --help

You should see five commands: run, validate, diff, report, and serve.

If you plan to use the Scrapy or Playwright integrations, install the relevant extras:

pip install -e ".[scrapy]"       # For Scrapy middleware
pip install -e ".[playwright]"   # For Playwright observer
pip install -e ".[dev]"          # For testing (pytest, ruff, mypy)

Step 2: Define Your Schema

A schema tells ScraperGuard what correct data looks like. Create a file called product_schema.py:

from scraperguard.schema import BaseSchema, validators

class ProductSchema(BaseSchema):
    title: str
    price: float = validators.range(min=0.01)
    availability: bool
    rating: float = validators.range(min=0, max=5)

This schema says:

title must be a non-null string
price must be a positive float (at least 0.01)
availability must be a boolean
rating must be a float between 0 and 5

ScraperGuard builds on Pydantic v2, so all Pydantic features work. The validators module provides convenience wrappers:

# Numeric range constraints
price: float = validators.range(min=0.01, max=99999.99)

# Regex pattern matching
sku: str = validators.pattern(r"^[A-Z]{2}-\d{6}$")

# List length constraints
images: list[str] = validators.cardinality(min=1, max=20)

# Explicit non-null
description: str = validators.not_null()

Real-World Schema Examples

Here are schemas for common scraping scenarios to give you a sense of what production schemas look like:

from scraperguard.schema import BaseSchema, validators
from typing import Optional

# E-commerce product schema
class ProductSchema(BaseSchema):
    title: str
    price: float = validators.range(min=0.01, max=99999.99)
    availability: bool
    rating: float = validators.range(min=0, max=5)
    sku: str = validators.pattern(r"^[A-Z]{2}-\d{6}$")
    images: list[str] = validators.cardinality(min=1, max=20)
    description: str = validators.not_null()

# Job listing schema
class JobSchema(BaseSchema):
    title: str
    company: str
    location: str
    salary_min: Optional[float] = validators.range(min=0)
    salary_max: Optional[float] = validators.range(min=0)
    posted_date: str
    url: str

# News article schema
class ArticleSchema(BaseSchema):
    headline: str = validators.not_null()
    author: str
    published: str
    body: str
    category: str
    word_count: int = validators.range(min=1)

Each schema captures the invariants of its domain. The more specific your validators, the earlier ScraperGuard can catch silent breakage — a price that suddenly becomes 0.00, a SKU that no longer matches the expected format, or a job listing missing salary data.

Step 3: Prepare Your Data

ScraperGuard can analyze data from two sources:

A live URL — ScraperGuard fetches the page, captures the HTML, and you provide extracted data via schema:

scraperguard run https://shop.example.com/products --schema product_schema.py --selectors ".price-tag,.product-title"

A JSON file — If your scraper already exports JSON, point ScraperGuard at the file:

scraperguard run data.json --schema product_schema.py --selectors ".price,.title"

For this tutorial, let’s create a sample data file. Create sample_data.json:

[
  {"title": "Wireless Headphones", "price": 79.99, "availability": true, "rating": 4.5},
  {"title": "USB-C Cable", "price": 12.99, "availability": true, "rating": 4.2},
  {"title": "Bluetooth Speaker", "price": null, "availability": true, "rating": 3.8},
  {"title": "Phone Case", "price": 24.99, "availability": false, "rating": 4.0},
  {"title": "", "price": 9.99, "availability": true, "rating": 4.7},
  {"title": "Laptop Stand", "price": -5.00, "availability": true, "rating": 4.1},
  {"title": "Webcam HD", "price": 49.99, "availability": true, "rating": 6.2},
  {"title": "Mouse Pad", "price": 15.99, "availability": true, "rating": 3.5},
  {"title": "Keyboard", "price": 89.99, "availability": true, "rating": null},
  {"title": "Monitor Light", "price": 34.99, "availability": true, "rating": 4.3}
]

This data has intentional problems:

Item 3: price is null
Item 5: title is empty string (passes type check but empty)
Item 6: price is negative (violates min=0.01)
Item 7: rating is 6.2 (violates max=5)
Item 9: rating is null (but rating isn’t required if you use Optional)

Step 4: Run Your First Health Check

scraperguard run sample_data.json --schema product_schema.py

ScraperGuard runs the full pipeline: validate every item against the schema, compute null ratios, check for drift against historical baselines (none yet for the first run), and produce a health report.

The output looks something like:

╔══════════════════════════════════════════════════════════════════╗
║  Health Score: 72/100 (degraded)                                  ║
╠══════════════════════════════════════════════════════════════════╣
║                                                                    ║
║  Schema Compliance: 70.0%                                         ║
║    7/10 items passed schema validation (70.0%)                    ║
║                                                                    ║
║  Extraction Completeness: 86.0%                                   ║
║    Field completeness: 86.0%. Weakest: price at 90.0%             ║
║                                                                    ║
║  Failures:                                                        ║
║    price: 1 missing, 1 range violation                            ║
║    rating: 1 range violation                                      ║
║                                                                    ║
╚══════════════════════════════════════════════════════════════════╝

The report tells you immediately: 70% of items pass, price and rating have issues, and the overall health is “degraded.”

Understanding the Python API

You can also use ScraperGuard as a Python library. The simplest entry point is BaseSchema.validate_batch(), which validates a list of items and returns structured results:

from scraperguard.schema import BaseSchema, validators

class ProductSchema(BaseSchema):
    title: str
    price: float = validators.range(min=0.01)
    availability: bool
    rating: float = validators.range(min=0, max=5)

items = [
    {"title": "Widget Pro", "price": 29.99, "availability": True, "rating": 4.5},
    {"title": "Gadget X", "price": None, "availability": False, "rating": 3.2},
]

result = ProductSchema.validate_batch(items, run_id="run-001", url="https://example.com")
print(f"Passed: {result.passed_count}/{result.total_items}")
print(f"Null ratios: {result.null_ratios}")
# Passed: 1/2
# Null ratios: {'title': 0.0, 'price': 0.5, 'availability': 0.0, 'rating': 0.0}

You can also introspect a schema to see which fields are required vs. optional:

print(ProductSchema.get_required_fields())   # ['title', 'price', 'availability', 'rating']
print(ProductSchema.get_optional_fields())   # []

For the full Python pipeline (validation + drift detection + DOM diffing + health scoring), see the schema validation tutorial and production architecture guide.

Step 5: Run Again to See Drift Detection

Now let’s simulate what happens when data quality degrades between runs. Create sample_data_degraded.json with more null prices:

[
  {"title": "Wireless Headphones", "price": null, "availability": true, "rating": 4.5},
  {"title": "USB-C Cable", "price": null, "availability": true, "rating": 4.2},
  {"title": "Bluetooth Speaker", "price": null, "availability": true, "rating": 3.8},
  {"title": "Phone Case", "price": null, "availability": false, "rating": 4.0},
  {"title": "Screen Protector", "price": null, "availability": true, "rating": 4.7},
  {"title": "Laptop Stand", "price": null, "availability": true, "rating": 4.1},
  {"title": "Webcam HD", "price": 49.99, "availability": true, "rating": 4.3},
  {"title": "Mouse Pad", "price": 15.99, "availability": true, "rating": 3.5},
  {"title": "Keyboard", "price": null, "availability": true, "rating": 4.6},
  {"title": "Monitor Light", "price": null, "availability": true, "rating": 4.3}
]

Run it:

scraperguard run sample_data_degraded.json --schema product_schema.py

This time, because the first run’s results are stored as a baseline, the drift detector will compare:

╔══════════════════════════════════════════════════════════════════╗
║  Health Score: 28/100 (critical)                                  ║
╠══════════════════════════════════════════════════════════════════╣
║                                                                    ║
║  Schema Compliance: 20.0%                                         ║
║    2/10 items passed schema validation (20.0%)                    ║
║                                                                    ║
║  Extraction Completeness: 40.0%                                   ║
║    Field completeness: 40.0%. Weakest: price at 20.0%             ║
║                                                                    ║
║  Drift Alerts:                                                    ║
║    [critical] price: null ratio from 10.0% to 80.0% (+70.0%)     ║
║                                                                    ║
╚══════════════════════════════════════════════════════════════════╝

The drift detector caught it: price went from 10% null to 80% null. That’s a critical severity event — the kind that should wake someone up.

Step 6: Export Reports

Get the report in different formats for integration with other tools:

# JSON format (for programmatic processing)
scraperguard report --url sample_data_degraded.json --format json

# CSV format (for spreadsheets)
scraperguard report --url sample_data_degraded.json --format csv

# Human-readable text (default)
scraperguard report --url sample_data_degraded.json --format text

Comparing Runs Over Time

After several runs, you can inspect how health evolved over time. The JSON report includes component-level breakdowns that make it easy to script trend analysis:

# After several runs, you can see how health evolved:
scraperguard report --url https://example.com --format json | python -c "
import sys, json
data = json.load(sys.stdin)
print(f'Score: {data[\"overall_score\"]}/100')
print(f'Status: {data[\"status\"]}')
for comp in data['components']:
    print(f'  {comp[\"name\"]}: {comp[\"score\"]:.0%} (weight: {comp[\"weight\"]})')
"

This is particularly useful in CI/CD pipelines where you want to gate deployments on data quality, or in scheduled jobs where you need to track whether a target site has changed its structure.

Step 7: Configure Alerting

Create a scraperguard.yaml configuration file:

schema:
  strict: true
  null_drift_threshold: 0.15

storage:
  backend: sqlite
  connection: scraperguard.db

alerts:
  slack:
    enabled: true
    webhook: ${SLACK_WEBHOOK_URL}
  email_enabled: false
  webhook_url: ""
  thresholds:
    health_score: 60
    schema_failure: critical
    selector_break: warning

snapshots:
  store_raw_html: false
  retention_days: 30

Set your Slack webhook URL as an environment variable:

export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"

All string values in the config support ${ENV_VAR} interpolation, keeping secrets out of your config files.

Complete Configuration Reference

Here is the full scraperguard.yaml with every available option documented:

# scraperguard.yaml — Complete Reference

schema:
  strict: true                    # Strict validation mode (fail on unknown fields)
  null_drift_threshold: 0.15      # Minimum delta to trigger drift event (0.0-1.0)

storage:
  backend: sqlite                 # Options: sqlite (postgres, jsonl planned for v1.1)
  connection: scraperguard.db     # SQLite file path or connection string

alerts:
  slack:
    enabled: false                # Enable Slack notifications
    webhook: ${SLACK_WEBHOOK_URL} # Slack webhook URL (env var interpolation)
  email_enabled: false            # Enable email notifications (SMTP)
  webhook_url: ""                 # Generic webhook URL for custom integrations
  thresholds:
    health_score: 60              # Alert when health score drops below this
    schema_failure: critical      # Minimum severity for schema failure alerts
    selector_break: warning       # Minimum severity for selector break alerts

snapshots:
  store_raw_html: false           # Store original HTML (increases storage)
  retention_days: 30              # Days to keep snapshots before cleanup

All string values in the configuration support ${ENV_VAR} interpolation. ScraperGuard uses os.environ.get() to resolve variables at config load time. This keeps secrets out of your config files — ideal for CI/CD pipelines and Docker deployments.

Step 8: Start the API

For dashboard-style access to health data:

scraperguard serve --host 0.0.0.0 --port 8000

Then browse to http://localhost:8000/docs for the auto-generated API documentation. Key endpoints:

GET /api/health         → System health overview
GET /api/runs           → List all runs with health scores
GET /api/validation/URL → Validation history for a specific URL
GET /api/drift/URL      → Drift events for a URL
GET /api/selectors/URL  → Selector tracking history

Full REST API Reference

Here is the complete list of API endpoints available when the server is running:

GET  /api/health                 → System health overview
GET  /api/runs                   → List all runs (newest first, limit 20)
GET  /api/runs/{run_id}          → Get run details including health score
GET  /api/runs/{run_id}/snapshots → List snapshots for a specific run
GET  /api/snapshots/{snapshot_id} → Get snapshot details with normalized HTML
GET  /api/validation/{url}       → Validation history for a URL
GET  /api/drift/{url}            → Drift events history for a URL
GET  /api/report/{run_id}        → Full health report for a specific run
GET  /api/selectors/{url}        → Selector tracking history for a URL

# Example API usage with curl:
curl http://localhost:8000/api/runs | python -m json.tool
curl http://localhost:8000/api/drift/https%3A%2F%2Fshop.example.com

Note that URL parameters containing special characters (like :// and /) must be URL-encoded when passed in the path. The /docs endpoint provides an interactive Swagger UI where you can test all endpoints directly in the browser.

What’s Next

You now have a working ScraperGuard setup with schema validation, drift detection, and health scoring. From here:

Add it to your Scrapy project → See the Scrapy integration tutorial
Use it with Playwright → See the Playwright monitoring tutorial
Understand the architecture → Read From Scripts to Systems: Production Scraping Architecture
Learn what each failure type means → Read From Scripts to Systems: Production Scraping Architecture

On This Page

Getting Started with ScraperGuard: Your First Scraper Health Check in 5 Minutes