DOM Diffing Deep Dive: How to Detect Structural Changes in Web Pages

When a website changes its layout, your web scraper breaks. But it doesn’t break loudly. It starts returning incomplete or incorrect data, and you don’t find out until the damage is done.

The challenge is detecting these structural changes before they cause data quality problems. Text-based comparison (like diff on two HTML files) doesn’t work — it generates thousands of irrelevant differences because of dynamic content, session tokens, and tracking scripts.

What you need is structural comparison — understanding whether the shape of the page changed, not whether the text content changed.

This article explains the complete DOM diffing pipeline: HTML normalization, structural fingerprinting, recursive tree comparison, and CSS selector tracking. This is how ScraperGuard’s core/dom_diff/ module works internally.

Why Text Diffs Don’t Work for HTML

Take this product page from Monday:

<div class="product" data-analytics-id="a8f3k2">
  <h1 class="title">Wireless Headphones</h1>
  <span class="price" data-timestamp="1706000000">$79.99</span>
  <script>analytics.track("view", {id: "a8f3k2", t: 1706000000})</script>
</div>

And Tuesday (identical structure, different dynamic values):

<div class="product" data-analytics-id="b9g4l3">
  <h1 class="title">Wireless Headphones</h1>
  <span class="price" data-timestamp="1706086400">$79.99</span>
  <script>analytics.track("view", {id: "b9g4l3", t: 1706086400})</script>
</div>

A text diff reports three changes. Structurally, nothing changed. Your scraper works identically on both.

Now Wednesday (actual restructuring):

<div class="product" data-analytics-id="c0h5m4">
  <h1 class="title">Wireless Headphones</h1>
  <div class="price-wrapper" data-timestamp="1706172800">
    <span class="amount">79</span>.<span class="cents">99</span>
  </div>
  <script>analytics.track("view", {id: "c0h5m4", t: 1706172800})</script>
</div>

A text diff shows changes on almost every line. The important change — <span class="price"> replaced with <div class="price-wrapper"> — is buried among noise. Structural diffing surfaces exactly this signal.

Step 1: HTML Normalization

Before comparing two pages structurally, remove everything that isn’t structural.

What to Strip

Noise Element	Why It’s Noise
`<script>`, `<style>`, `<noscript>`	Behavior/presentation, not structure
HTML comments	Invisible to scraper
`data-analytics-`, `data-gtm-`, `data-tracking-*`	Change per request
`onclick`, `onload`, `on*` attributes	JavaScript behavior
`style` attribute	Visual presentation
Whitespace noise	Inconsistent across responses

What to Keep

Structural Element	Why It Matters
All HTML tags and nesting	Defines the page skeleton
`class` and `id` attributes	Used by CSS selectors
Semantic `data-*` attrs (`data-price`, `data-sku`)	Carry extractable values
Tag order among siblings	Affects selector targeting

ScraperGuard’s Full Noise Removal List

The normalizer (core/snapshot/normalizer.py) uses carefully curated lists to distinguish noise from signal:

# Tags removed entirely (including children)
NOISE_TAGS = {"script", "style", "noscript"}

# Attributes removed from all elements
REMOVE_ATTRIBUTE_NAMES = {"style", "jsaction", "jscontroller", "jsmodel", "jsname"}

# data-* attributes removed by suffix pattern
REMOVE_DATA_SUFFIXES = {
    "analytics", "tracking", "gtm", "reactid", "testid",
    "timestamp", "session", "token", "nonce", "csrf"
}
# Any data-* attribute whose name ends with one of these suffixes is stripped.
# For example: data-analytics-id, data-gtm-container, data-tracking-source

# data-* attributes explicitly KEPT (useful for scraping)
KEEP_DATA_ATTRIBUTES = {
    "data-price", "data-product", "data-id", "data-sku",
    "data-category", "data-available", "data-rating",
    "data-name", "data-value", "data-type", "data-status"
}

# Event handler attributes (removed — all on* attributes)
# onclick, onload, onmouseover, etc. — anything starting with "on"

# Empty elements that are kept even without content
KEEP_EMPTY_ELEMENTS = {"td", "th", "li"}

# Self-closing elements (not removed even when empty)
VOID_ELEMENTS = {"img", "br", "hr", "input", "meta", "link"}

The normalizer also performs multi-pass empty element removal. A <div> containing only a <span> that itself is empty gets cleaned up in two passes — the inner <span> first, then the now-empty <div>.

Implementation (lxml)

This is the approach ScraperGuard’s normalizer (core/snapshot/normalizer.py) uses:

import lxml.html
from lxml import etree
import re

REMOVE_TAGS = {"script", "style", "noscript"}
REMOVE_ATTR_PREFIXES = (
    "data-analytics", "data-tracking", "data-gtm", "data-reactid",
    "data-testid", "data-timestamp", "data-session", "data-nonce",
)
REMOVE_ATTR_NAMES = {"style", "jsaction", "jscontroller", "jsmodel", "jsname"}
KEEP_DATA_ATTRS = {
    "data-price", "data-product", "data-id", "data-sku",
    "data-category", "data-available", "data-rating", "data-value",
}

def normalize_html(raw_html: str) -> str:
    if not raw_html or not raw_html.strip():
        return ""

    tree = lxml.html.fromstring(raw_html)

    # Remove noise elements
    for tag in REMOVE_TAGS:
        for el in tree.xpath(f"//{tag}"):
            el.getparent().remove(el)

    # Remove comments
    for comment in tree.xpath("//comment()"):
        parent = comment.getparent()
        if parent is not None:
            parent.remove(comment)

    # Clean attributes
    for el in tree.iter():
        if not isinstance(el.tag, str):
            continue
        to_remove = []
        for attr in el.attrib:
            if attr.startswith("on"):
                to_remove.append(attr)
            elif attr in REMOVE_ATTR_NAMES:
                to_remove.append(attr)
            elif attr.startswith("data-") and attr not in KEEP_DATA_ATTRS:
                if any(attr.startswith(p) for p in REMOVE_ATTR_PREFIXES):
                    to_remove.append(attr)
        for attr in to_remove:
            del el.attrib[attr]

    # Collapse whitespace in text nodes
    for el in tree.iter():
        if el.text:
            el.text = re.sub(r'\s+', ' ', el.text).strip() or None
        if el.tail:
            el.tail = re.sub(r'\s+', ' ', el.tail).strip() or None

    return lxml.html.tostring(tree, encoding="unicode", method="html").strip()

Critical property: determinism. The same input must always produce the exact same output. Without this, fingerprinting and diffing produce false positives.

The normalizer also provides a text extraction function used by the failure classifier:

def extract_text_content(html: str) -> str:
    """Extract all visible text from normalized HTML.

    Used by the failure classifier to detect JS challenge pages
    (pages where visible text < 50 characters but script tags are present).
    """
    normalized = normalize_html(html)
    if not normalized:
        return ""
    tree = lxml.html.fromstring(normalized)
    return " ".join(tree.xpath("//text()")).strip()

Step 2: Structural Fingerprinting

Once normalized, you need a fast way to check if two pages have the same structure without doing a full tree comparison.

Two Fingerprints, Not One

ScraperGuard computes two fingerprints per snapshot (core/snapshot/fingerprint.py):

Content fingerprint: SHA-256 of the full normalized HTML. Changes when ANY content changes — even a price updating from $79 to $80.

Structural fingerprint: SHA-256 of the tag-and-class skeleton only. Ignores text content and attribute values. Changes only when the DOM structure changes.

The structural fingerprint gates whether the diff engine runs. This is a critical optimization — without it, the differ runs on every page load because prices and stock counts change constantly.

# From scraperguard/core/snapshot/fingerprint.py
import hashlib

def fingerprint_html(normalized_html: str) -> str:
    """Content fingerprint — changes when ANY content changes."""
    return hashlib.sha256(normalized_html.encode("utf-8")).hexdigest()

def fingerprint_structure(normalized_html: str) -> str:
    """Structural fingerprint — changes only when DOM structure changes."""
    if not normalized_html:
        return hashlib.sha256(b"").hexdigest()
    tree = lxml.html.fromstring(normalized_html)
    signature = _build_signature(tree)
    return hashlib.sha256(signature.encode("utf-8")).hexdigest()

def are_structurally_identical(fp_a: str, fp_b: str) -> bool:
    """Convenience helper for comparing structural fingerprints."""
    return fp_a == fp_b

This is used in the snapshot capture pipeline:

# From scraperguard/core/snapshot/capture.py
def should_diff(current_fingerprint: str, previous_fingerprint: str | None) -> bool:
    """Gate the diff engine. Only diff when structure actually changed."""
    if previous_fingerprint is None:
        return True  # First run, always diff
    return not are_structurally_identical(current_fingerprint, previous_fingerprint)

Building the Structural Signature

import hashlib

def fingerprint_structure(normalized_html: str) -> str:
    if not normalized_html:
        return hashlib.sha256(b"").hexdigest()
    tree = lxml.html.fromstring(normalized_html)
    signature = _build_signature(tree)
    return hashlib.sha256(signature.encode("utf-8")).hexdigest()

def _build_signature(element) -> str:
    if not isinstance(element.tag, str):
        return ""

    tag = element.tag.lower()
    classes = sorted(element.get("class", "").split())
    tag_sig = f"{tag}.{'.'.join(classes)}" if classes else tag

    child_sigs = [_build_signature(c) for c in element if _build_signature(c)]

    return f"{tag_sig}({','.join(child_sigs)})" if child_sigs else tag_sig

For this HTML:

<div class="product">
  <h1 class="title">Wireless Headphones</h1>
  <span class="price">$79.99</span>
</div>

The signature is: div.product(h1.title,span.price)

If the price text changes to “$89.99”, the signature stays identical. If <span class="price"> becomes <div class="price-wrapper">, the signature changes. Exactly the gating behavior we want.

Step 3: Recursive Tree Comparison

When fingerprints differ, the diff engine identifies what specifically changed.

The DOMNode Tree Representation

ScraperGuard converts normalized HTML into a lightweight tree of DOMNode objects for comparison:

# From scraperguard/core/dom_diff/parser.py
@dataclass
class DOMNode:
    tag: str                          # Element tag name (lowercase)
    attributes: dict[str, str]        # Element attributes
    children: list["DOMNode"]         # Child nodes
    text: str = ""                    # Direct text content
    tail: str = ""                    # Text after closing tag
    depth: int = 0                    # Nesting depth (0 = root)
    path: str = ""                    # CSS-like path: "body > div.product > span.price"

The path field is built during parsing by concatenating tag names and class names:

def _build_path(element, parent_path: str) -> str:
    tag = element.tag.lower()
    classes = element.get("class", "").split()
    node_id = f"{tag}.{'.'.join(sorted(classes))}" if classes else tag
    return f"{parent_path} > {node_id}" if parent_path else node_id

The Challenge: Index Matching Fails

A naive approach compares children by position: first child vs. first child, second vs. second. This breaks immediately if the site inserts a banner at the top — every element shifts by one index, and the differ reports every element as “changed.”

The Solution: Greedy Similarity Matching

ScraperGuard’s differ (core/dom_diff/differ.py) uses greedy matching:

For each child in the “before” tree, find the most similar child in the “after” tree
Similarity is based on tag name and class overlap
Matched pairs are compared recursively
Unmatched “before” children are removals
Unmatched “after” children are additions

Similarity scoring:

def _node_similarity(a, b) -> float:
    if a.tag != b.tag:
        return 0.0                      # Different tags: no match

    a_classes = set(a.get("class", "").split())
    b_classes = set(b.get("class", "").split())

    if a_classes == b_classes:
        return 1.0                      # Same tag + same classes: strong
    elif a_classes & b_classes:
        return 0.7                      # Same tag + some class overlap
    else:
        return 0.5                      # Same tag + no class overlap

A match requires similarity > 0.3. This threshold was tuned empirically — it’s low enough to match elements whose classes changed slightly, but high enough to avoid matching unrelated elements.

Change Types Detected

ScraperGuard classifies structural changes into 7 types:

Change Type	Severity	Signal
`NODE_REMOVED`	High	Element present in before, absent in after
`NODE_ADDED`	Medium	Element present in after, absent in before
`TAG_CHANGED`	High	Same position, different tag (e.g., `span` → `div`)
`ATTRIBUTES_CHANGED`	Medium (high if class changed)	Attribute values differ
`CHILDREN_REORDERED`	Medium	Same children, different order
`NODE_MOVED`	Medium	Element exists in both but at different position
`TEXT_CHANGED`	Low	Text content differs (usually irrelevant noise)

Full Differ Implementation

from dataclasses import dataclass, field
from enum import Enum

class ChangeType(str, Enum):
    NODE_REMOVED = "node_removed"
    NODE_ADDED = "node_added"
    TAG_CHANGED = "tag_changed"
    ATTRIBUTES_CHANGED = "attributes_changed"
    CHILDREN_REORDERED = "children_reordered"
    TEXT_CHANGED = "text_changed"

@dataclass
class DOMChange:
    change_type: ChangeType
    path: str           # CSS-like path: "body > div.product > span.price"
    severity: str       # "high", "medium", "low"
    message: str
    details: dict = field(default_factory=dict)

def diff_trees(before_html: str, after_html: str) -> list[DOMChange]:
    before = lxml.html.fromstring(before_html)
    after = lxml.html.fromstring(after_html)
    changes = []
    _diff_nodes(before, after, "", changes)

    severity_order = {"high": 0, "medium": 1, "low": 2}
    changes.sort(key=lambda c: severity_order.get(c.severity, 3))
    return changes

def _diff_nodes(before, after, parent_path, changes):
    path = _node_path(before, parent_path)

    # Compare tags
    if isinstance(before.tag, str) and isinstance(after.tag, str) and before.tag != after.tag:
        changes.append(DOMChange(
            change_type=ChangeType.TAG_CHANGED,
            path=path, severity="high",
            message=f"Tag changed: '{before.tag}' → '{after.tag}' at {path}",
            details={"old_tag": before.tag, "new_tag": after.tag}
        ))

    # Compare attributes (especially class)
    if dict(before.attrib) != dict(after.attrib):
        _diff_attributes(before, after, path, changes)

    # Compare text
    before_text = (before.text or "").strip()
    after_text = (after.text or "").strip()
    if before_text != after_text:
        changes.append(DOMChange(
            change_type=ChangeType.TEXT_CHANGED,
            path=path, severity="low",
            message=f"Text changed at {path}",
            details={"old": before_text[:100], "new": after_text[:100]}
        ))

    # Compare children via greedy matching
    _diff_children(list(before), list(after), path, changes)

def _diff_children(before_list, after_list, parent_path, changes):
    used_after = set()
    matches = []

    for i, bc in enumerate(before_list):
        best_j, best_score = -1, 0.3
        for j, ac in enumerate(after_list):
            if j in used_after:
                continue
            score = _node_similarity(bc, ac)
            if score > best_score:
                best_j, best_score = j, score
        if best_j >= 0:
            matches.append((i, best_j))
            used_after.add(best_j)

    matched_before = {m[0] for m in matches}
    matched_after = {m[1] for m in matches}

    # Removals
    for i, child in enumerate(before_list):
        if i not in matched_before and isinstance(child.tag, str):
            changes.append(DOMChange(
                change_type=ChangeType.NODE_REMOVED,
                path=_node_path(child, parent_path), severity="high",
                message=f"Element '{child.tag}' removed",
                details={"tag": child.tag, "classes": child.get("class", "")}
            ))

    # Additions
    for j, child in enumerate(after_list):
        if j not in matched_after and isinstance(child.tag, str):
            changes.append(DOMChange(
                change_type=ChangeType.NODE_ADDED,
                path=_node_path(child, parent_path), severity="medium",
                message=f"New element '{child.tag}' added",
                details={"tag": child.tag, "classes": child.get("class", "")}
            ))

    # Recurse into matched pairs
    for bi, ai in matches:
        _diff_nodes(before_list[bi], after_list[ai], parent_path, changes)

Testing the Diff

before = normalize_html("""
<div class="product">
    <h1 class="title">Headphones</h1>
    <span class="price">$79.99</span>
    <span class="stock">In Stock</span>
</div>
""")

after = normalize_html("""
<div class="product">
    <h1 class="title">Headphones</h1>
    <div data-price="79.99" class="new-price">$79.99</div>
    <div class="stock-status">Available</div>
</div>
""")

changes = diff_trees(before, after)
for c in changes:
    print(f"[{c.severity}] {c.change_type.value}: {c.message}")

Output:

[high] node_removed: Element 'span' removed at div.product > span.price
[high] node_removed: Element 'span' removed at div.product > span.stock
[medium] node_added: New element 'div' added at div.product > div.new-price
[medium] node_added: New element 'div' added at div.product > div.stock-status

The diff correctly identifies the structural change: old <span> elements replaced with new <div> elements. That’s the signal you actually care about.

Step 4: CSS Selector Tracking

The final piece connects structural changes to scraper impact. If the DOM changed, which selectors are affected?

ScraperGuard’s selector tracker (core/dom_diff/selector_tracker.py) counts element matches per selector in both snapshots:

@dataclass
class SelectorStatus:
    selector: str
    current_matches: int
    previous_matches: int | None
    status: str          # "stable", "broken", "degraded", "improved", "new"

def track_selectors(current_html, previous_html, selectors):
    current_tree = lxml.html.fromstring(current_html)
    previous_tree = lxml.html.fromstring(previous_html) if previous_html else None

    results = []
    for sel in selectors:
        current_count = len(current_tree.cssselect(sel))
        prev_count = len(previous_tree.cssselect(sel)) if previous_tree else None

        if prev_count is None:
            status = "new"
        elif current_count == 0 and prev_count > 0:
            status = "broken"
        elif current_count < prev_count:
            status = "degraded"
        elif current_count > prev_count:
            status = "improved"
        else:
            status = "stable"

        results.append(SelectorStatus(sel, current_count, prev_count, status))
    return results

Output:

✗ BROKEN: '.price' matches 0 elements (was 1)
✓ Stable: '.title' matches 1 elements
✗ BROKEN: '.stock' matches 0 elements (was 1)

Combined with the DOM diff, you have a complete picture: what changed in the page AND which selectors broke.

Selector Matching Implementation

ScraperGuard’s selector matching (core/dom_diff/parser.py) supports a subset of CSS selectors — enough for tracking common scraper selectors without pulling in a full CSS engine:

def find_nodes_by_selector(tree: DOMNode, selector: str) -> list[DOMNode]:
    """Find nodes matching a simplified CSS selector.

    Supported selectors:
      - Tag: "div", "span"
      - Class: ".price-tag", ".product.featured"
      - Combined: "div.price-tag"
      - Descendant: "div.container span.price"
      - Direct child: "div.container > span.price"

    NOT supported (use lxml.cssselect for full CSS):
      - Pseudo-classes (:first-child, :nth-child)
      - Attribute selectors ([data-price])
      - ID selectors (#main)
    """

def count_selector_matches(tree: DOMNode, selector: str) -> int:
    """Convenience wrapper returning match count."""
    return len(find_nodes_by_selector(tree, selector))

For the selector tracker, this is combined with lxml’s cssselect for richer matching when the original HTML is available:

# The tracker uses both approaches:
# 1. DOMNode tree matching (for structural analysis)
# 2. lxml cssselect (for accurate match counting)
current_count = len(current_tree.cssselect(selector))
previous_count = len(previous_tree.cssselect(selector)) if previous_tree else None

The Complete Pipeline

Raw HTML → Normalize (strip noise) → Fingerprint structure
                                          │
                    ┌─────────────────────┤
                    │                     │
            fingerprint same?       fingerprint changed?
                    │                     │
              skip diffing          diff the trees
                                          │
                                    ┌─────┴─────┐
                                    │           │
                              DOM changes  selector tracking
                                    │           │
                                    └─────┬─────┘
                                          │
                                   feed to classifier

This pipeline is exactly what ScraperGuard executes on every run. The structural fingerprint gates diffing (eliminating 90%+ of unnecessary comparisons), the tree diff identifies what changed, and the selector tracker tells you the scraper impact.

Performance Characteristics

Operation	Typical Page (200-500 nodes)	Large Page (2000+ nodes)
Normalization	2-5ms	10-20ms
Fingerprinting	<1ms	1-2ms
Tree diffing	5-15ms	50-100ms
Selector tracking	<1ms per selector	2-5ms per selector

All lxml-powered operations benefit from the C-extension performance. The entire pipeline for a typical product page completes in under 25ms.

Why Performance Matters: The Gating Pattern

The dual fingerprinting strategy eliminates the most expensive operation (tree diffing) for the vast majority of page loads. In a typical monitoring setup:

~95% of page loads only change text content (prices, stock counts, timestamps). The structural fingerprint stays the same → no diff runs.
~4% of page loads have minor structural changes (ad banners, A/B test elements). Quick diff → a few changes detected.
~1% of page loads have significant restructuring. Full diff → many changes, immediate alert.

This means the per-page overhead in normal operation is just normalization (~3ms) + fingerprinting (~1ms) = ~4ms total. The expensive diff only runs when it matters.

# The capture pipeline gates diffing automatically:
snapshot = capture_snapshot(url, html, items, metadata, storage, run_id=run.id)
prev = storage.get_latest_snapshot(url)

if prev and should_diff(snapshot.fingerprint, prev.fingerprint):
    # Structure changed — run the diff
    changes = diff_trees(parse_to_tree(snapshot.normalized_html),
                         parse_to_tree(prev.normalized_html))
else:
    # Structure unchanged — skip expensive diffing
    changes = []

On This Page