Building ScraperGuard: Architecture Decisions and Tradeoffs

I recently released ScraperGuard, an open-source observability layer for web scraping pipelines. This case study covers the major architecture decisions, why I made them, what I’d do differently, and what I learned about building developer tools.

The Starting Point

Every scraping project I’ve worked on had the same failure pattern: scraper runs fine, data is broken, nobody knows for days. I kept writing one-off validation scripts. After the fourth time, I decided to build it properly.

That decision forced me to answer design questions that one-off scripts never ask.

Decision 1: Library, Not Service

The first question was where ScraperGuard sits in the stack.

Option A: Standalone monitoring service. Periodically fetches pages, compares against baselines, reports changes. Like an uptime monitor for page structure.

Option B: Library that integrates into existing scrapers. Hooks into Scrapy pipelines and Playwright scripts, observing the scraper’s own output.

Option C: Proxy layer. Sits between the scraper and the web, intercepting responses.

I went with Option B for three reasons:

Accuracy. A standalone service re-fetches pages — but might see a different version than the scraper (different IP, no cookies, no session state). False positives everywhere.
Context. A library sees exactly what the scraper sees: the raw response AND the extracted data. It can correlate structural changes with data quality failures. A proxy sees responses but not extracted data. A standalone service sees neither.
Adoption. Adding a library is pip install + config. Standing up a service is infrastructure.

The tradeoff: ScraperGuard is coupled to the scraper’s runtime. If the scraper crashes before reaching validation, ScraperGuard never runs. Acceptable, because the problem being solved is silent failure (scraper succeeds, data is wrong) — not crashes (your scheduler already catches those).

Decision 2: Observer Pattern, Not Interceptor

Should ScraperGuard modify the pipeline (reject bad items, halt on failures) or just observe and report?

I chose observer because of a hard lesson: any tool that can halt a production pipeline will halt it at the worst possible time.

If ScraperGuard has a bug in observer mode → wrong health score. If it has a bug in interceptor mode → rejected valid data, crashed pipeline, production incident.

Implementation:

# Scrapy middleware: never modifies the response
def process_response(self, request, response, spider):
    self._capture(response)
    return response  # Always return unmodified

# Scrapy pipeline: never filters items
def process_item(self, item, spider):
    self._buffer(dict(item))
    return item  # Always return unmodified

All analysis runs after the scraper finishes (close_spider for Scrapy, __aexit__ for Playwright). The scraper’s execution is never affected.

Decision 3: Rule-Based Classification, Not ML

When you hear “classify failure types automatically,” the instinct is ML. Train a classifier on labeled examples of CAPTCHA pages, rate-limited responses, structural changes.

I went with pure rule-based classification. This is the most important technical decision in the project.

Determinism. When ScraperGuard says “SELECTOR_BREAK, confidence 0.90,” I can explain exactly why: selector .price-tag went from 3 matches to 0, page returned HTTP 200 with full body. An ML model’s explanation is “the feature vector was close to the SELECTOR_BREAK cluster.” Useless for debugging.

Well-defined categories. A CAPTCHA page contains known strings. A rate-limited response has status 429. These are if-then rules, not statistical patterns.

Zero training data. To train an ML classifier, I’d need thousands of labeled examples of each failure type across different sites. That data doesn’t exist.

Debuggable edge cases. When a rule produces a wrong classification, I read the rule, understand why it fired, fix it. When an ML model misclassifies, I’m retraining with more data I don’t have.

Each classifier function checks specific signals:

def _check_captcha(self, context):
    if not context.raw_html:
        return None

    html_lower = context.raw_html.lower()
    captcha_signatures = [
        "g-recaptcha", "www.google.com/recaptcha",
        "hcaptcha.com", "h-captcha",
        "challenges.cloudflare.com", "cf-challenge",
    ]

    found = [sig for sig in captcha_signatures if sig in html_lower]
    if not found:
        return None

    confidence = min(0.95, 0.70 + len(found) * 0.10)
    return Classification(
        failure_type=FailureType.CAPTCHA_INJECTION,
        confidence=confidence,
        severity="critical",
        evidence=[f"CAPTCHA signature detected: {sig}" for sig in found],
        recommendation="Rotate IP/proxy, add CAPTCHA solving, or reduce request rate"
    )

Multiple classifications can coexist. A run might show both DOM_RESTRUCTURE and SELECTOR_BREAK simultaneously (the restructure caused the break). The system reports all, sorted by confidence.

Could ML improve this later? Maybe for ambiguous edge cases. But as a foundation, rules are more honest, more debuggable, and more testable.

The Complete Classifier Rule Chain

The full classification pipeline runs 8 checks in a fixed order. Each check function receives the same ClassificationInput and returns a Classification or None. All matches accumulate — multiple classifications can coexist:

def classify_failure(inp: ClassificationInput) -> list[Classification]:
    checks = [
        _check_empty_response,    # Response body < 100 bytes?
        _check_captcha,           # reCAPTCHA/hCaptcha/Cloudflare signatures?
        _check_js_challenge,      # Minimal visible text + script tags?
        _check_rate_limit,        # HTTP 429/403 or rate-limit text?
        _check_selector_break,    # Any tracked selector → 0 matches?
        _check_dom_restructure,   # ≥5 high-severity DOM changes?
        _check_ab_variant,        # Partial selector failure + moderate changes?
        _check_partial_extraction,# Some items valid, some fail?
    ]

    results: list[Classification] = []
    for check in checks:
        result = check(inp)
        if result is not None:
            results.append(result)

    # Fallback: if failures exist but no rule matched
    if not results:
        has_failures = False
        if inp.validation_result and inp.validation_result.failed_count > 0:
            has_failures = True
        if any(s.status == "broken" for s in inp.selector_statuses):
            has_failures = True
        if has_failures:
            results.append(Classification(
                failure_type=FailureType.UNKNOWN,
                confidence=0.3,
                evidence=["No specific failure pattern matched"],
                recommended_action="Manual investigation recommended.",
                severity="warning",
            ))

    results.sort(key=lambda c: c.confidence, reverse=True)
    return results

The ordering matters for clarity, not for priority. Every check runs regardless of previous results. The UNKNOWN fallback ensures no failure goes unreported — if something is wrong but no rule fires, the system says “I don’t know what this is, investigate manually.” Sorting by confidence puts the most reliable classification first for display.

Each check function has its own confidence calibration. Simple signals get high confidence: HTTP 429 → RATE_LIMIT at 0.95, because there’s no ambiguity. Compound signals get lower confidence: partial selector failure + moderate DOM changes → AB_VARIANT at 0.55, because it’s a guess. This matters when multiple classifications coexist — a run showing both DOM_RESTRUCTURE (0.85) and SELECTOR_BREAK (0.90) tells you the structural change caused the selector break, with the selector break being the more confident diagnosis.

The _check_selector_break function is the most interesting because it correlates signals across subsystems:

def _check_selector_break(inp: ClassificationInput) -> Classification | None:
    broken = [s for s in inp.selector_statuses if s.status == "broken"]
    if not broken:
        return None

    evidence = [
        f"Selector '{s.selector}' returned 0 matches (was {s.previous_matches})"
        for s in broken
    ]

    # Infer which data fields are affected by the broken selectors
    affected_fields: list[str] = []
    for s in broken:
        for f in _infer_affected_fields(s.selector):
            if f not in affected_fields:
                affected_fields.append(f)

    # Higher confidence if broken selectors correlate with high null ratios
    has_validation_correlation = False
    if inp.validation_result and inp.validation_result.null_ratios:
        for f in affected_fields:
            if inp.validation_result.null_ratios.get(f, 0.0) > 0.5:
                has_validation_correlation = True
                break

    confidence = 0.90 if has_validation_correlation else 0.70
    # ...

If .price-tag broke AND the price field has 80% nulls, confidence is 0.90 — the data confirms the structural diagnosis. If .price-tag broke but price null ratios are normal, confidence drops to 0.70 — maybe the selector name is misleading. This cross-referencing between selector tracking and schema validation is what makes the classifications useful — you know exactly what broke and can verify it.

Decision 4: Pydantic v2, Not JSON Schema

Native Python. Schemas look like dataclasses, not JSON blobs. Users define schemas in .py files that their IDE understands, with autocomplete and type checking.

Performance. Pydantic v2’s Rust core validates items 10-50x faster than jsonschema. For batch validation of thousands of items per run, this matters.

Structured errors. Pydantic returns error objects with field path, error type, and input value. This feeds directly into per-field failure statistics and null ratio computation.

Composable constraints. I wrapped Pydantic’s Field() in convenience functions:

validators.range(min=0.01)          # → Field(ge=0.01)
validators.pattern(r"^SKU-\d+$")    # → Field(pattern=...)
validators.cardinality(min=1)       # → List length check

The tradeoff: schemas are Python code, not portable JSON/YAML. JSON Schema would be better for dynamic schemas defined by non-Python tools. It’s on the v1.1 roadmap.

Decision 5: Two Fingerprints, Not One

Initially I computed one SHA-256 hash of the normalized HTML. If it changed, the differ ran.

Problem: hashes changed constantly. A price updating from $79 to $80 triggered a diff that found only a text content change. Expensive and noisy.

Fix: dual fingerprinting.

Content fingerprint: Hash of full normalized HTML. For “did anything change?”
Structural fingerprint: Hash of tag skeleton only (no text, no attribute values). For “should I run the diff?”

The structural signature for <div class="product"><h1 class="title">Headphones</h1><span class="price">$79.99</span></div> is div.product(h1.title,span.price). Price text changes don’t change the signature. Replacing <span class="price"> with <div class="price-wrapper"> does.

This eliminated 90%+ of unnecessary diff operations.

The Actual Fingerprint Implementation

The structural fingerprint works by recursively building a tag-nesting string, then hashing it:

def _build_structure_string(element: HtmlElement) -> str:
    """Produces strings like html(body(div(h1,span,span),div(span)))
    where only tag names and nesting are captured."""
    if not isinstance(element.tag, str):
        return ""

    children_parts: list[str] = []
    for child in element:
        part = _build_structure_string(child)
        if part:
            children_parts.append(part)

    tag = element.tag.lower()
    if children_parts:
        return f"{tag}({','.join(children_parts)})"
    return tag


def fingerprint_structure(normalized_html: str) -> str:
    """SHA-256 of the tag skeleton. Two pages with identical DOM
    structure but different text produce the same fingerprint."""
    if not normalized_html:
        return hashlib.sha256(b"").hexdigest()

    tree = lxml.html.fromstring(normalized_html)
    structure = _build_structure_string(tree)
    return hashlib.sha256(structure.encode("utf-8")).hexdigest()

The content fingerprint is simpler — just hashlib.sha256(normalized_html.encode("utf-8")).hexdigest(). A should_diff gating function compares the structural fingerprints of two snapshots. If they match, the diff engine is skipped entirely.

Performance: Fingerprinting is sub-millisecond for typical product pages. The _build_structure_string traversal is O(n) in DOM nodes, and SHA-256 hashing of the resulting string is negligible. For a page with 500 nodes, the structure string might be ~2KB — hashing that takes microseconds. The real savings are downstream: skipping the tree diff (5-15ms for typical pages, 50-100ms for large pages) when only text content changed.

Decision 6: SQLite Default

Zero setup. No database server. storage.backend: sqlite creates the file automatically.

Sufficient performance. ScraperGuard writes per-run, not per-item. Even 200 runs/day is trivial for SQLite.

Portable. The database file is part of the project. Copy it, back it up, or rm it to reset.

WAL mode. Enabled by default for concurrent read performance when the API queries while a spider writes.

PostgreSQL is on the v1.1 roadmap for teams needing multi-worker coordination and centralized dashboards.

The storage layer uses an abstract base class so backends are interchangeable:

class StorageBackend(ABC):
    @abstractmethod
    def save_snapshot(self, snapshot: Snapshot) -> None: ...
    @abstractmethod
    def save_validation_result(self, result: ValidationResult) -> None: ...
    # etc.

One detail: methods use domain objects (Snapshot, ValidationResult), not raw dictionaries. This forces each backend to handle its own serialization, preventing storage details from leaking into the engine.

The SQLite Schema

Three tables, deliberately minimal:

CREATE TABLE IF NOT EXISTS runs (
    id TEXT PRIMARY KEY,
    scraper_name TEXT NOT NULL,
    timestamp TEXT NOT NULL,
    config TEXT NOT NULL,
    status TEXT NOT NULL DEFAULT 'running'
);

CREATE TABLE IF NOT EXISTS snapshots (
    id TEXT PRIMARY KEY,
    run_id TEXT NOT NULL REFERENCES runs(id),
    url TEXT NOT NULL,
    fingerprint TEXT NOT NULL,
    normalized_html TEXT NOT NULL,
    raw_html TEXT,
    extracted_items TEXT NOT NULL,
    metadata TEXT NOT NULL,
    timestamp TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_snapshots_url_timestamp
    ON snapshots(url, timestamp DESC);

CREATE TABLE IF NOT EXISTS validation_results (
    id TEXT PRIMARY KEY,
    run_id TEXT NOT NULL REFERENCES runs(id),
    url TEXT NOT NULL,
    schema_name TEXT NOT NULL,
    total_items INTEGER NOT NULL,
    passed_count INTEGER NOT NULL,
    failed_count INTEGER NOT NULL,
    field_failures TEXT NOT NULL,
    null_ratios TEXT NOT NULL,
    timestamp TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_vr_url_schema_timestamp
    ON validation_results(url, schema_name, timestamp DESC);

Design choices worth noting:

Text IDs, not autoincrement. UUIDs generated in Python, not by SQLite. This avoids coupling to SQLite’s rowid semantics and makes the schema portable to PostgreSQL.
JSON in TEXT columns. field_failures, null_ratios, extracted_items, metadata, and config are serialized JSON stored as TEXT. This keeps the schema flat while storing structured data. Querying inside JSON is a rare operation — you look up by run_id or url, then deserialize in Python.
Descending timestamp indexes. Most queries want the latest data: “last 5 validation results for this URL.” The DESC index supports this without sorting at query time.
raw_html is nullable. Storing raw HTML is optional and off by default (store_raw_html: false in config). Normalized HTML is always stored for diffing. This saves significant storage — raw HTML can be 10-50x larger than normalized HTML.

Storage footprint: ~1KB per snapshot (normalized HTML for a typical product page after stripping scripts/styles/comments), ~500B per validation result. A scraper running 200 times/day against 10 URLs produces ~60MB/month. Negligible.

Decision 7: CLI-First

Developer tools are adopted from the terminal. The first thing someone does after pip install is run a command to see if it works. No CLI means they have to write a script just to test.

scraperguard run https://example.com --schema product_schema.py --selectors ".price,.title"

One command runs the full pipeline and prints a health report. That’s the first experience. The API (scraperguard serve) comes second.

The CLI commands map to pipeline stages:

Command	What It Does
`run`	Full pipeline: fetch → snapshot → validate → drift → diff → classify → score → alert
`validate`	Schema validation only (on stored data)
`diff`	Structural comparison between stored snapshots
`report`	Generate health report (text/json/csv)
`serve`	Start FastAPI server

Each subsequent command runs a subset for targeted debugging. This is deliberate — after run shows a problem, you use diff or validate to investigate specific aspects.

A Complete CLI Debugging Session

Here’s a realistic debugging workflow using the CLI:

# 1. Run health check on a URL
$ scraperguard run https://shop.example.com/products \
    --schema schemas.py --selectors ".price,.title,.stock"

╔══════════════════════════════════════════════════════════════════╗
║  Health Score: 35/100 (critical)                                  ║
╠══════════════════════════════════════════════════════════════════╣
║  Schema Compliance: 20.0%                                        ║
║  Selector Stability: 33.3%                                       ║
║    Broken: .price, .stock                                        ║
║  Failures:                                                       ║
║    [critical] selector_break (confidence: 0.87)                  ║
║  Drift Alerts:                                                   ║
║    [critical] price: null ratio from 2.0% to 80.0% (+78.0%)     ║
╚══════════════════════════════════════════════════════════════════╝

# 2. Investigate: what changed in the DOM?
$ scraperguard diff --url https://shop.example.com/products --last 2

[high] node_removed: span.price removed at body > div.container > div.product > span.price
[high] node_removed: span.stock removed at body > div.container > div.product > span.stock
[medium] node_added: div.price-wrapper added at body > div.container > div.product
[medium] node_added: div.availability added at body > div.container > div.product

# 3. Check validation history
$ scraperguard validate --url https://shop.example.com/products --schema schemas.py

Validation Results (last 5 runs):
  Run 5: 48/50 passed (96.0%) — 2025-02-27
  Run 4: 49/50 passed (98.0%) — 2025-02-26
  Run 3: 50/50 passed (100.0%) — 2025-02-25
  Run 2: 47/50 passed (94.0%) — 2025-02-24
  Run 1: 10/50 passed (20.0%) — 2025-02-28  ← today's broken run

# 4. Export for further analysis
$ scraperguard report --url https://shop.example.com/products --format json > report.json

Decision 8: Fault Tolerance by Design

The observer pattern is a philosophy. Fault tolerance is the engineering that enforces it. ScraperGuard wraps every pipeline stage in try/except — if any stage fails, the scraper continues unaffected.

Middleware initialization: fail to no-op

If ScraperGuard itself can’t initialize (bad config, missing storage file, import error), it creates a no-op instance that passes everything through:

@classmethod
def from_crawler(cls, crawler):
    try:
        config = load_config(config_path)
        storage = get_storage_backend(config)
        run_meta = storage.create_run(scraper_name="scrapy")
        # ... normal initialization ...
        return instance
    except Exception:
        logger.exception("ScraperGuard: Failed to initialize, creating no-op instance")
        instance = cls.__new__(cls)
        instance.storage = None
        instance._captured = {}
        return instance

The scraper never knows ScraperGuard failed to start. It just runs without monitoring.

Response capture: fail silently

Every process_response call is wrapped. If metadata extraction, header parsing, or HTML capture throws, the response passes through unchanged:

def process_response(self, request, response, spider):
    try:
        raw_html = response.text
        metadata = SnapshotMetadata(
            http_status=response.status,
            latency_ms=latency_ms,
            timestamp=datetime.now(timezone.utc),
            headers=headers_dict,
            response_size_bytes=len(response.body),
        )
        self._captured[response.url] = (raw_html, metadata)
    except Exception:
        logger.exception("ScraperGuard: Error capturing response for %s", response.url)

    return response  # Always return unchanged

Pipeline stages: independent failure domains

In the Playwright observer, each pipeline stage (validation, drift, diffing, classification, scoring) is wrapped independently. If DOM diffing crashes, classification still runs on whatever data is available:

async def _run_pipeline(self) -> None:
    snapshot = capture_snapshot(url=url, raw_html=self._raw_html, ...)

    # Schema validation — independent failure domain
    validation_result = None
    drift_events = []
    if self._schema is not None and self._items:
        try:
            validation_result = self._schema.validate_batch(self._items, ...)
            try:
                drift_events = run_drift_analysis(validation_result, self._storage)
            except Exception:
                logger.exception("ScraperGuard: Drift analysis failed")
            self._storage.save_validation_result(validation_result)
        except Exception:
            logger.exception("ScraperGuard: Schema validation failed")

    # DOM diff — independent failure domain
    dom_changes = []
    try:
        prev = self._storage.list_snapshots(url, limit=2)
        if prev and should_diff(snapshot.fingerprint, prev.fingerprint):
            dom_changes = diff_trees(parse_to_tree(prev.normalized_html),
                                     parse_to_tree(snapshot.normalized_html))
    except Exception:
        logger.exception("ScraperGuard: DOM diff failed")

    # Selector tracking — independent failure domain
    selector_statuses = []
    try:
        selector_statuses = track_selectors(current_tree, prev_tree, self._selectors)
    except Exception:
        logger.exception("ScraperGuard: Selector tracking failed")

    # Classification — uses whatever data survived
    classifications = []
    try:
        classifications = classify_failure(ClassificationInput(
            validation_result=validation_result,
            dom_changes=dom_changes,
            selector_statuses=selector_statuses, ...
        ))
    except Exception:
        logger.exception("ScraperGuard: Classification failed")

    # Health score — uses whatever survived
    try:
        self._health_report = compute_health_score(
            validation_result=validation_result,
            selector_statuses=selector_statuses,
            dom_changes=dom_changes,
            classifications=classifications, ...
        )
    except Exception:
        logger.exception("ScraperGuard: Health score failed")

The context manager guarantee

The Playwright observer’s __aexit__ ensures the scraper’s own exceptions are never suppressed:

async def __aexit__(self, exc_type, exc_val, exc_tb) -> bool:
    try:
        await self._run_pipeline()
    except Exception:
        logger.exception("ScraperGuard: Playwright observer pipeline failed")
    return False  # Never suppress user exceptions

return False is the critical line. If the scraper raises an exception, ScraperGuard lets it propagate normally. If ScraperGuard itself crashes during _run_pipeline, the scraper’s exception (if any) still propagates. The monitoring tool never masks the thing it’s monitoring.

What I’d Do Differently

Start with the README as specification. I wrote the README as marketing, then built code to match. Several times, the README promised things that were hard to implement as described. A spec-first README would have caught design problems earlier.

Define the storage interface before any implementation. I built SQLite first, then extracted the abstract interface. The interface is shaped by SQLite’s characteristics, and some methods don’t translate cleanly to other backends. Interface-first would be cleaner.

Ship smaller. I tried to ship a complete v1.0 with all features. This delayed the release while I fixed edge cases in rarely-used features (CSV export, email alerting). I should have shipped core functionality first and added features in point releases.

Testing: 212 Tests in 2.3 Seconds

Testing philosophy for developer tools: the test suite is part of the product.

Unit tests for every algorithm. 17 normalizer tests, 20 classifier tests, 12 differ tests.
Integration tests for the full pipeline. Define schema → create HTML fixtures → run pipeline → verify health score, classifications, drift events.
Mock externals. Tests use mock Scrapy Request/Response objects and mock Playwright pages. No Scrapy or Playwright installation required to run tests.
Test contracts, not implementations. Storage tests verify save/retrieve/list behavior, not SQLite internals. They work for any backend.
Realistic fixtures. tests/fixtures/ contains three HTML files: normal product page, changed product page, CAPTCHA page.

Test Architecture Patterns

Three patterns that proved valuable:

1. In-memory storage for tests:

# Tests use SQLiteBackend(":memory:") — no file cleanup needed
@pytest.fixture
def storage():
    return SQLiteBackend(":memory:")

def test_snapshot_persistence(storage):
    run = storage.create_run("test_spider")
    snapshot = capture_snapshot(url, html, items, metadata, storage, run_id=run.id)
    retrieved = storage.get_snapshot_by_id(snapshot.id)
    assert retrieved.fingerprint == snapshot.fingerprint

2. HTML fixtures for realistic testing:

# tests/fixtures/ contains three carefully crafted HTML files:
# - normal_product_page.html (baseline)
# - changed_product_page.html (with structural changes)
# - captcha_page.html (Cloudflare challenge page)

@pytest.fixture
def normal_html():
    return Path("tests/fixtures/normal_product_page.html").read_text()

def test_captcha_classification(captcha_html):
    classifications = classify_failure(ClassificationInput(raw_html=captcha_html, ...))
    assert classifications[0].failure_type == FailureType.CAPTCHA_INJECTION

3. Mock framework objects for integration tests:

# Tests don't require Scrapy or Playwright installed
class MockResponse:
    def __init__(self, url, body, status=200):
        self.url = url
        self.text = body
        self.status = status
        self.headers = {}

class MockPage:
    def __init__(self, html):
        self._html = html
        self.url = "https://example.com"

    async def content(self):
        return self._html

4. Factory helpers for test data construction:

# Rather than constructing full dataclass instances in every test,
# factory functions with sensible defaults keep tests focused on
# the specific signal being tested.

def _make_validation_result(
    total_items: int = 10,
    passed_count: int = 10,
    failed_count: int = 0,
    field_failures: list[FieldFailure] | None = None,
    null_ratios: dict[str, float] | None = None,
) -> ValidationResult:
    return ValidationResult(
        run_id="test-run", url="https://example.com",
        schema_name="TestSchema", total_items=total_items,
        passed_count=passed_count, failed_count=failed_count,
        field_failures=field_failures or [], null_ratios=null_ratios or {},
    )

def _make_selector_status(
    selector: str = ".product-title",
    current_matches: int = 0,
    previous_matches: int = 10,
    status: str = "broken",
) -> SelectorStatus:
    return SelectorStatus(
        selector=selector, current_matches=current_matches,
        previous_matches=previous_matches, status=status,
        message=f"Selector '{selector}' matches {current_matches} nodes",
    )

# Now a test for selector_break classification reads clearly:
def test_selector_break_with_null_correlation():
    inp = ClassificationInput(
        validation_result=_make_validation_result(
            failed_count=8, null_ratios={"price": 0.8}),
        selector_statuses=[_make_selector_status(".price", 0, 10)],
        dom_changes=[], raw_html="<html>...</html>",
        http_status=200, response_size_bytes=5000,
    )
    results = classify_failure(inp)
    assert results[0].failure_type == FailureType.SELECTOR_BREAK
    assert results[0].confidence == 0.90  # Correlation boosts confidence

This pattern scales. The classifier has 20+ test cases covering every rule, edge case, and combination. Each test overrides only the relevant defaults, making the signal-under-test immediately visible.

By the Numbers

Metric	Value
Test suite	212 tests, 0 failures, ~2.3s
Core modules	10
Lines of production code	~4,000
Lines of test code	~3,500
Dependencies (core)	6
Failure types classified	8
Health score components	4
CLI commands	5
REST endpoints	9

What Stuck

Observer > interceptor for any tool that wraps production pipelines. Never risk crashing the thing you’re monitoring.
Rules > ML when categories are well-defined and debuggability matters more than generalization.
Dual fingerprinting eliminates expensive operations that trigger on irrelevant changes.
CLI-first adoption. If someone can’t test your tool in one command, they won’t test it at all.
Spec-first README. Write the interface documentation before the implementation.
Test the contract. Tests that verify behavior (not implementation) survive refactors.
Default to zero-config. SQLite, sensible thresholds, no infrastructure. Make the first run effortless.

ScraperGuard is MIT licensed: github.com/mostafam-dev/scraperguard. Contributions welcome.

On This Page