Understanding robots.txt: What Every Web Scraper Needs to Know

Before you write a single line of scraping code, there’s something you need to understand: robots.txt. It’s the first file your scraper should check and the foundation of ethical data collection. This tutorial covers what robots.txt is, how to read its directives, and how to build a reusable Python class that enforces compliance automatically.

What Is robots.txt?

Every website can place a file at /robots.txt (e.g., https://example.com/robots.txt) that tells automated programs — search engine crawlers, scrapers, bots — which parts of the site they’re allowed to access. It’s part of the Robots Exclusion Protocol, a de facto standard since 1994.

Here’s a real-world example of what you might find:

# Example robots.txt
User-agent: *
Disallow: /api/
Disallow: /admin/
Disallow: /user/*/settings
Allow: /api/public/
Crawl-delay: 2

User-agent: Googlebot
Allow: /

Sitemap: https://example.com/sitemap.xml

If you’re building scrapers, this isn’t optional reading. Ignoring robots.txt can get your IP banned, your project shut down, or — in some jurisdictions — land you in legal trouble.

Anatomy of a robots.txt File

Let’s break down each directive you’ll encounter.

User-agent

Specifies which bot the following rules apply to. The wildcard * means “all bots.”

User-agent: *          # Rules for everyone
User-agent: MyBot/1.0  # Rules specifically for MyBot

If a site has rules for your specific user-agent string, those take priority over the wildcard rules.

Disallow

Tells bots not to access matching paths. An empty Disallow: means nothing is blocked.

Disallow: /private/     # Block everything under /private/
Disallow: /search       # Block /search and /search?q=anything
Disallow: /             # Block the entire site
Disallow:               # Block nothing (allow everything)

Allow

Overrides a Disallow for specific paths. Useful for exposing a subdirectory inside a blocked parent.

Disallow: /api/
Allow: /api/public/     # /api/ is blocked, but /api/public/ is allowed

Crawl-delay

Requests a minimum delay (in seconds) between consecutive requests from a bot. Not all crawlers respect this, but yours should.

Crawl-delay: 2          # Wait at least 2 seconds between requests

Sitemap

Points to the site’s XML sitemap. Useful if you need to discover URLs to scrape.

Sitemap: https://example.com/sitemap.xml

Reading robots.txt in Python

Python ships with urllib.robotparser in the standard library — no pip installs required.

Basic Usage

from urllib.robotparser import RobotFileParser

rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()

# Check if a specific URL is allowed
url = "https://example.com/products/shoes"
user_agent = "MyScraperBot/1.0"

if rp.can_fetch(user_agent, url):
    print(f"Allowed: {url}")
else:
    print(f"Blocked: {url}")

That’s it for the basics. Three lines to load, one line to check. But production scrapers need more — caching, error handling, and rate limiting. Let’s build that.

Building a Reusable RobotsTxtChecker Class

Here’s a class you can drop into any scraping project:

import time
from urllib.robotparser import RobotFileParser
from urllib.parse import urlparse, urljoin
from typing import Optional

class RobotsTxtChecker:
    """
    Checks robots.txt rules with caching and crawl-delay support.

    Usage:
        checker = RobotsTxtChecker(user_agent="MyBot/1.0")
        if checker.is_allowed("https://example.com/page"):
            # safe to scrape
            checker.wait()  # respect crawl-delay
            response = requests.get(...)
    """

    def __init__(self, user_agent: str = "PythonScraper/1.0"):
        self.user_agent = user_agent
        self._cache: dict[str, Optional[RobotFileParser]] = {}
        self._last_request: dict[str, float] = {}

    def _get_base_url(self, url: str) -> str:
        """Extract scheme + netloc from a URL."""
        parsed = urlparse(url)
        return f"{parsed.scheme}://{parsed.netloc}"

    def _load(self, base_url: str) -> Optional[RobotFileParser]:
        """Load and cache a robots.txt parser for a domain."""
        if base_url in self._cache:
            return self._cache[base_url]

        rp = RobotFileParser()
        rp.set_url(urljoin(base_url, "/robots.txt"))

        try:
            rp.read()
            self._cache[base_url] = rp
        except Exception:
            # If robots.txt is unreachable, treat as "allow all"
            self._cache[base_url] = None

        return self._cache[base_url]

    def is_allowed(self, url: str) -> bool:
        """Check whether the URL is allowed by robots.txt."""
        base_url = self._get_base_url(url)
        rp = self._load(base_url)

        if rp is None:
            return True  # No robots.txt found — assume allowed

        return rp.can_fetch(self.user_agent, url)

    def get_crawl_delay(self, url: str) -> float:
        """Return the crawl-delay for this domain, defaulting to 1 second."""
        base_url = self._get_base_url(url)
        rp = self._load(base_url)

        if rp is not None:
            delay = rp.crawl_delay(self.user_agent)
            if delay is not None:
                return float(delay)

        return 1.0  # sensible default

    def wait(self, url: str) -> None:
        """Sleep if needed to respect crawl-delay for this domain."""
        base_url = self._get_base_url(url)
        delay = self.get_crawl_delay(url)

        if base_url in self._last_request:
            elapsed = time.time() - self._last_request[base_url]
            if elapsed < delay:
                time.sleep(delay - elapsed)

        self._last_request[base_url] = time.time()

    def clear_cache(self) -> None:
        """Clear cached robots.txt data (useful for long-running scrapers)."""
        self._cache.clear()
        self._last_request.clear()

Using the Checker

import requests

checker = RobotsTxtChecker(user_agent="MyBot/1.0 (+https://mysite.com/bot)")

urls = [
    "https://httpbin.org/html",
    "https://httpbin.org/headers",
]

for url in urls:
    if not checker.is_allowed(url):
        print(f"Skipping (blocked by robots.txt): {url}")
        continue

    checker.wait(url)  # respect crawl-delay
    response = requests.get(url, headers={"User-Agent": checker.user_agent})
    print(f"Fetched {url} — {len(response.text)} bytes")

Why This Design Matters

1. Caching: robots.txt is fetched once per domain, not once per URL. Without caching, a scraper hitting 10,000 pages on one domain would make 10,000 extra HTTP requests just for robots.txt.

2. Crawl-delay enforcement: The wait() method tracks the last request time per domain and sleeps when needed. This prevents you from hammering a server even if your code runs in a tight loop.

3. Graceful failure: If robots.txt can’t be fetched (404, timeout, DNS error), the checker assumes scraping is allowed. This follows the standard — a missing robots.txt means no restrictions.

Real-World robots.txt Examples

Let’s look at what major sites actually put in their robots.txt files.

GitHub

# Relevant section from github.com/robots.txt
User-agent: *
Disallow: /*/archive/
Disallow: /*/refs/
Disallow: /*/blame/
Disallow: /search

GitHub blocks archive downloads, Git refs, blame views, and the search endpoint. Product pages and repositories are allowed.

Amazon

# Relevant section from amazon.com/robots.txt
User-agent: *
Disallow: /exec/obidos/account-access-login
Disallow: /gp/cart
Disallow: /gp/customer-reviews/write-a-review
Disallow: /ap/signin

Amazon blocks login pages, cart, review forms, and authentication endpoints. Product listing pages are generally allowed.

Wikipedia

# Relevant section from en.wikipedia.org/robots.txt
User-agent: *
Disallow: /w/
Disallow: /wiki/Special:
Allow: /w/api.php

Wikipedia blocks the MediaWiki internals but explicitly allows access to its API.

The Pattern

Notice something? These sites all follow the same logic: block private/interactive/authenticated endpoints, allow public content. When you’re building a scraper, if a path shows up in Disallow, there’s usually a good reason. It might be a login form, a rate-limited API, or content that’s only meaningful with authentication.

Testing Your Checker

Here’s a quick script to test your checker against known robots.txt rules:

from robots_checker import RobotsTxtChecker

checker = RobotsTxtChecker(user_agent="TestBot/1.0")

# Test cases — these use a hypothetical robots.txt:
# User-agent: *
# Disallow: /admin/
# Disallow: /api/
# Allow: /api/public/

test_cases = [
    ("https://httpbin.org/html", True),       # No specific block
    ("https://httpbin.org/robots.txt", True),  # robots.txt itself is always fetchable
]

for url, expected in test_cases:
    result = checker.is_allowed(url)
    status = "PASS" if result == expected else "FAIL"
    print(f"[{status}] {url} — allowed={result} (expected={expected})")

Common Mistakes to Avoid

1. Checking robots.txt once at startup and never again

Websites update their robots.txt. For long-running scrapers, clear the cache periodically:

import time

checker = RobotsTxtChecker(user_agent="MyBot/1.0")
last_cache_clear = time.time()

for url in url_list:
    # Refresh cache every hour
    if time.time() - last_cache_clear > 3600:
        checker.clear_cache()
        last_cache_clear = time.time()

    if checker.is_allowed(url):
        checker.wait(url)
        # scrape...

2. Using a generic or misleading user-agent

Don’t pretend to be Googlebot. Use a descriptive user-agent with a URL or email where site owners can reach you:

# Good
user_agent = "DataResearchBot/1.0 (+https://yoursite.com/bot; [email protected])"

# Bad
user_agent = "Googlebot"
user_agent = "Mozilla/5.0"  # pretending to be a browser

3. Treating “Disallow” as just a suggestion

The robots.txt specification is technically advisory — there’s no technical enforcement. But courts have used robots.txt violations as evidence of unauthorized access. Treat it as a rule, not a suggestion.

4. Not handling crawl-delay

Many scrapers check Disallow but ignore Crawl-delay. If a site says Crawl-delay: 10, they’re telling you their server can’t handle fast requests from bots. Respect it.

Integrating with Your Scraper

Here’s how the RobotsTxtChecker fits into a typical scraping workflow:

import requests
from robots_checker import RobotsTxtChecker

class MyScraper:
    def __init__(self):
        self.checker = RobotsTxtChecker(
            user_agent="MyScraper/1.0 (+https://scrapem.com/bot)"
        )
        self.session = requests.Session()
        self.session.headers["User-Agent"] = self.checker.user_agent

    def scrape(self, url: str) -> str | None:
        # Step 1: Check robots.txt
        if not self.checker.is_allowed(url):
            print(f"Blocked by robots.txt: {url}")
            return None

        # Step 2: Respect crawl-delay
        self.checker.wait(url)

        # Step 3: Make the request
        response = self.session.get(url, timeout=30)
        response.raise_for_status()
        return response.text

scraper = MyScraper()
html = scraper.scrape("https://httpbin.org/html")
if html:
    print(f"Got {len(html)} characters")

The key insight: robots.txt checking happens before the request, and rate limiting happens between requests. Build this into your scraper’s architecture from the start, not as an afterthought.

Next Steps

Now that you understand robots.txt, you’re ready to build scrapers that are both effective and responsible. Here’s where to go next:

• Build your first scraper: Put these concepts into practice with a hands-on project
• Learn about rate limiting patterns: Go beyond crawl-delay with exponential backoff and adaptive throttling
• Understand the legal side: See how courts have interpreted robots.txt in scraping cases