If you run Scrapy spiders in production, you’ve experienced the nightmare: a spider runs for days producing bad data because a site changed. No errors, no crashes. Just garbage in the database.

This tutorial shows you how to add ScraperGuard’s full observability pipeline to an existing Scrapy project. The integration uses Scrapy’s native middleware and pipeline system — no changes to your spider code.

What You Get

After integration, every spider run will automatically:

Capture raw HTML for every response (via the downloader middleware)
Validate extracted items against your schema (via the item pipeline)
Compute null ratios per field and detect drift against historical baselines
Compare DOM snapshots between runs and diff structural changes
Track selector health — know immediately if a selector stopped matching
Classify failures — SELECTOR_BREAK, CAPTCHA_INJECTION, DOM_RESTRUCTURE, etc.
Produce a health score (0-100) for the run
Send alerts via Slack or webhook if thresholds are breached
Fire custom Scrapy signals you can hook into

All without changing a single line of spider code.

Step 1: Install

pip install -e ".[scrapy]"

This installs ScraperGuard with the Scrapy optional dependency.

Step 2: Define Your Schema

Create a schema file in your Scrapy project. For example, myproject/schemas.py:

from scraperguard.schema import BaseSchema, validators

class ProductSchema(BaseSchema):
    title: str
    price: float = validators.range(min=0.01)
    availability: bool
    rating: float = validators.range(min=0, max=5)
    url: str

The schema should match the fields your spider yields.

Step 3: Add Two Lines to settings.py

# Add the observer middleware (captures HTML from every response)
DOWNLOADER_MIDDLEWARES = {
    "scraperguard.integrations.scrapy.ObserverMiddleware": 543,
}

# Add the validation pipeline (validates items, runs analysis at spider close)
ITEM_PIPELINES = {
    "scraperguard.integrations.scrapy.ValidationPipeline": 300,
}

# Point to your schema class (dotted Python path)
SCRAPERGUARD_SCHEMA = "myproject.schemas.ProductSchema"

That’s it. Your spider code stays exactly the same.

Step 4: (Optional) Add CSS Selectors to Track

If you want ScraperGuard to monitor specific CSS selectors between runs, add them to the spider or settings:

# In settings.py
SCRAPERGUARD_SELECTORS = [".price-tag", ".product-title", ".stock-status", ".rating"]

Or in the spider class:

class ProductSpider(scrapy.Spider):
    name = "products"
    scraperguard_selectors = [".price-tag", ".product-title"]

    def parse(self, response):
        yield {
            "title": response.css(".product-title::text").get(),
            "price": float(response.css(".price-tag::text").get("0")),
            "availability": bool(response.css(".stock-status")),
            "rating": float(response.css(".rating::attr(data-value)").get("0")),
            "url": response.url,
        }

How It Works Internally

The Observer Middleware

ScraperGuardObserverMiddleware is a downloader middleware that intercepts every response:

class ScraperGuardObserverMiddleware:
    def process_response(self, request, response, spider):
        # Capture raw HTML and metadata — never modify the response
        self._capture(response.url, response.text, {
            "status": response.status,
            "headers": dict(response.headers),
            "latency": request.meta.get("download_latency"),
        })
        return response  # Always return unmodified

Key design: the middleware never modifies the response. It captures a copy for analysis. If ScraperGuard has a bug, the worst outcome is a wrong health report — your spider continues functioning exactly as before.

The Validation Pipeline

ScraperGuardValidationPipeline is an item pipeline that buffers every yielded item:

class ScraperGuardValidationPipeline:
    def process_item(self, item, spider):
        # Buffer item for batch analysis at spider close — never modify or filter
        self._buffer(dict(item))
        return item  # Always return unmodified

    def close_spider(self, spider):
        # Run the full analysis pipeline:
        # snapshot → validate → drift detect → diff → classify → health → alert
        self._analyze(spider)

All analysis happens in close_spider, after the spider has finished its work. This means:

Items flow through the pipeline at normal speed (no per-item validation overhead)
The full batch is available for statistical analysis (null ratios, drift detection)
DOM snapshots can be compared in bulk

The Full Analysis Pipeline

When close_spider fires, the ValidationPipeline runs the same 7-step analysis pipeline described in Production Scraping Architecture: capture snapshot → validate → drift detect → DOM diff → selector track → classify → health score. All steps are wrapped in try/except so a failure in one step doesn’t block the others.

The Scrapy-specific part is what happens after the pipeline: the results are broadcast as Scrapy signals and routed through the alert dispatcher.

Selector-to-Field Inference

When a selector breaks, ScraperGuard automatically infers which data fields are affected using _infer_affected_fields(). This powers the “affected_fields” in the classification output:

# From scraperguard/core/classify/classifier.py
def _infer_affected_fields(selector: str) -> list[str]:
    """Infer field names from a selector using simple substring matching."""
    parts = re.findall(r"[\w-]+", selector)
    common_fields = [
        "price", "title", "name", "description", "image",
        "url", "rating", "review", "stock", "availability",
        "category", "brand", "sku",
    ]
    fields = []
    for part in parts:
        part_lower = part.lower()
        for field_name in common_fields:
            if field_name in part_lower and field_name not in fields:
                fields.append(field_name)
    return fields

# ".price-tag" → ["price"]
# ".product-title" → ["title"]
# ".stock-status" → ["stock"]

When a broken selector correlates with high null ratios in the validation result (same field name, null ratio > 50%), the classifier’s confidence jumps from 0.70 to 0.90. This is how ScraperGuard connects “selector .price-tag broke” to “that’s why the price field is 80% null.”

Custom Scrapy Signals

The pipeline fires three custom signals you can connect to:

from scraperguard.integrations.scrapy.pipeline import (
    scraperguard_health_computed,
    scraperguard_drift_detected,
    scraperguard_selector_broken,
)

class MyExtension:
    @classmethod
    def from_crawler(cls, crawler):
        ext = cls()
        crawler.signals.connect(ext.on_health, signal=scraperguard_health_computed)
        crawler.signals.connect(ext.on_drift, signal=scraperguard_drift_detected)
        crawler.signals.connect(ext.on_broken, signal=scraperguard_selector_broken)
        return ext

    def on_health(self, health_report, **kwargs):
        if health_report.overall_score < 50:
            logger.critical(f"Spider health critical: {health_report.overall_score}/100")

    def on_drift(self, drift_events, **kwargs):
        for event in drift_events:
            logger.warning(f"Drift: {event.message}")

    def on_broken(self, broken_selectors, **kwargs):
        for sel in broken_selectors:
            logger.error(f"Selector broken: {sel.selector}")

These signals integrate with Scrapy’s existing signal system. You can connect them to custom extensions for logging, database writes, or external notifications beyond what ScraperGuard’s built-in alerting provides.

Signal Data Format

Each signal passes specific data to your handlers:

# scraperguard_health_computed
def on_health(self, health_report, url, **kwargs):
    """
    health_report: HealthReport object with:
      .overall_score  → int (0-100)
      .status         → "healthy" | "degraded" | "critical"
      .components     → list[HealthComponent]
      .classifications → list[Classification]
      .drift_events   → list[DriftEvent]
      .run_id, .url, .timestamp
    """
    if health_report.overall_score < 50:
        # Stop the spider when health is critical
        self.crawler.engine.close_spider(spider, reason="health_critical")

# scraperguard_drift_detected
def on_drift(self, drift_events, url, **kwargs):
    """
    drift_events: list[DriftEvent] with:
      .field_name      → str
      .current_ratio   → float (0.0-1.0)
      .baseline_ratio  → float
      .delta           → float
      .severity        → "critical" | "warning" | "info"
      .message         → str
    """
    for event in drift_events:
        db.record_drift(url, event.field_name, event.delta)

# scraperguard_selector_broken
def on_broken(self, selector_statuses, url, **kwargs):
    """
    selector_statuses: list[SelectorStatus] (only broken ones) with:
      .selector         → str (CSS selector)
      .current_matches  → int (always 0 for broken)
      .previous_matches → int (was > 0)
      .status           → "broken"
      .message          → str
    """
    for sel in selector_statuses:
        pagerduty.trigger(f"Selector {sel.selector} broke on {url}")

Step 5: Add Configuration

Create scraperguard.yaml in your project root:

schema:
  strict: true
  null_drift_threshold: 0.15

storage:
  backend: sqlite
  connection: scraperguard.db

alerts:
  slack:
    enabled: true
    webhook: ${SLACK_WEBHOOK_URL}
  thresholds:
    health_score: 60
    schema_failure: critical
    selector_break: warning

snapshots:
  store_raw_html: false
  retention_days: 30

ScraperGuard looks for this file in the current working directory, or you can specify a path via the SCRAPERGUARD_CONFIG environment variable.

Step 6: Run Your Spider

scrapy crawl products

At the end of the spider run (in close_spider), ScraperGuard prints the health report:

╔══════════════════════════════════════════════════════════════════╗
║  Health Score: 92/100 (healthy)                                   ║
╠══════════════════════════════════════════════════════════════════╣
║  Schema Compliance: 96.0%                                        ║
║    48/50 items passed schema validation                          ║
║  Extraction Completeness: 94.0%                                  ║
║    Field completeness: 94.0%. Weakest: rating at 88.0%           ║
║  Selector Stability: 100.0%                                      ║
║    4/4 selectors stable                                          ║
║  Structural Stability: 100.0%                                    ║
║    0 structural changes detected                                 ║
╚══════════════════════════════════════════════════════════════════╝

Next run (after a site change):

╔══════════════════════════════════════════════════════════════════╗
║  Health Score: 35/100 (critical)                                  ║
╠══════════════════════════════════════════════════════════════════╣
║  Schema Compliance: 20.0%                                        ║
║    10/50 items passed schema validation                          ║
║  Selector Stability: 50.0%                                       ║
║    2/4 selectors stable. Broken: .price-tag, .stock-status       ║
║  Failures:                                                       ║
║    [critical] selector_break (confidence: 0.90)                  ║
║  Drift Alerts:                                                   ║
║    [critical] price: null ratio from 4.0% to 80.0% (+76.0%)     ║
╚══════════════════════════════════════════════════════════════════╝

If Slack alerting is enabled, you’ll get a notification within seconds.

Debugging Past Runs

After the spider finishes, use the CLI to investigate:

# View the latest health report
scraperguard report --url https://shop.example.com/products

# Compare the last 2 DOM snapshots
scraperguard diff --url https://shop.example.com/products --last 2

# Check validation history
scraperguard validate --url https://shop.example.com/products --schema myproject/schemas.py

# Start the API for dashboard access
scraperguard serve --port 8000

All data is stored in scraperguard.db (SQLite). You can also query it directly:

sqlite3 scraperguard.db "SELECT url, health_score, status FROM runs ORDER BY timestamp DESC LIMIT 10"

Multiple Spiders, One ScraperGuard

If your project has multiple spiders with different schemas, you can set the schema per spider:

class ProductSpider(scrapy.Spider):
    name = "products"
    custom_settings = {
        "SCRAPERGUARD_SCHEMA": "myproject.schemas.ProductSchema",
    }

class ArticleSpider(scrapy.Spider):
    name = "articles"
    custom_settings = {
        "SCRAPERGUARD_SCHEMA": "myproject.schemas.ArticleSchema",
    }

Each spider’s results are tracked separately in storage. The health report, drift detection, and structural comparison all operate per-URL, so different spiders monitoring different sites each have their own baselines.

Accessing Captured Data in Spider Callbacks

The observer middleware stores captured HTML and metadata in request.meta, which you can access in your spider:

class ProductSpider(scrapy.Spider):
    name = "products"

    def parse(self, response):
        # ScraperGuard's middleware adds these to request.meta:
        sg_html = response.meta.get("scraperguard_html")        # Raw HTML captured
        sg_meta = response.meta.get("scraperguard_metadata")    # SnapshotMetadata object

        if sg_meta:
            self.logger.info(f"Response latency: {sg_meta.latency_ms:.0f}ms")
            self.logger.info(f"Response size: {sg_meta.response_size_bytes} bytes")

        # Your normal parsing continues unchanged
        yield {
            "title": response.css(".product-title::text").get(),
            "price": float(response.css(".price-tag::text").get("0")),
        }

Troubleshooting

“ImportError: Scrapy integration requires scrapy” — Install with the Scrapy extra: pip install -e ".[scrapy]"

No health report printed — The pipeline runs in close_spider. If the spider is killed (Ctrl+C, OOM), close_spider may not execute. Use Scrapy’s CLOSESPIDER_* settings to ensure graceful shutdown.

Selectors not tracked — Make sure you’ve set SCRAPERGUARD_SELECTORS in settings or scraperguard_selectors on the spider class. Without selectors, the selector tracking component is skipped.

All items show as failed — Check that your item fields match the schema field names exactly. Scrapy items are case-sensitive.

Advanced: Per-Spider Configuration

You can override ScraperGuard settings per spider using Scrapy’s custom_settings:

class HighValueSpider(scrapy.Spider):
    name = "high_value_products"
    custom_settings = {
        "SCRAPERGUARD_SCHEMA": "myproject.schemas.HighValueProductSchema",
        "SCRAPERGUARD_SELECTORS": [".premium-price", ".luxury-brand", ".availability"],
        "SCRAPERGUARD_STORE_RAW_HTML": True,   # Keep raw HTML for debugging
        "SCRAPERGUARD_CONFIG_PATH": "scraperguard-strict.yaml",  # Stricter thresholds
    }

class BulkSpider(scrapy.Spider):
    name = "bulk_catalog"
    custom_settings = {
        "SCRAPERGUARD_SCHEMA": "myproject.schemas.BasicProductSchema",
        "SCRAPERGUARD_SELECTORS": [".product", ".price"],
        "SCRAPERGUARD_STORE_RAW_HTML": False,  # Save storage space
    }

Each spider’s data is tracked independently in storage. Health scores, drift baselines, and structural comparisons all operate per-URL, so different spiders monitoring different sites maintain separate histories.

Integrating with Scrapy Cloud / Scrapyd

For deployment on Scrapy Cloud or Scrapyd, set configuration via environment variables:

# .env or deployment config
SCRAPERGUARD_CONFIG=/path/to/scraperguard.yaml
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...

# The config file uses ${ENV_VAR} interpolation:
# alerts.slack.webhook: ${SLACK_WEBHOOK_URL}

On This Page

Integrating ScraperGuard with Scrapy: Add Scraper Monitoring in 5 Minutes