Introduction

One of the fundamental principles of ethical web scraping is respecting the robots.txt file. This tutorial will show you how to build Python scrapers that automatically check and follow robots.txt rules before making any requests.

What you’ll learn:

Understanding robots.txt structure and syntax
Implementing robots.txt checking in Python
Building a reusable scraper class with built-in compliance
Handling edge cases and errors gracefully

Prerequisites:

Basic Python knowledge (functions, classes)
Familiarity with HTTP requests
Python 3.8+ installed

Time to complete: ~20 minutes

Why robots.txt Matters

The robots.txt file is a website’s way of communicating which parts of their site should or shouldn’t be accessed by automated tools. Ignoring it is a bad idea for several reasons:

Courts in multiple jurisdictions have treated robots.txt violations as evidence of unauthorized access
Website owners set these rules for a reason — respecting them is the baseline of ethical scraping
Getting banned or sued costs more than slowing down your scraper ever will

Understanding robots.txt

A robots.txt file typically looks like this:

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/
Crawl-delay: 1

Key directives:

User-agent: Specifies which bots the rules apply to
Disallow: Paths that should not be accessed
Allow: Explicitly allowed paths (overrides Disallow)
Crawl-delay: Minimum time between requests (in seconds)

Step 1: Environment Setup

Install Dependencies

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\\Scripts\\activate

# Install required packages
pip install requests

Project Structure

ethical-scraper/
├── scraper.py          # Main scraper class
├── example.py          # Usage examples
├── requirements.txt    # Dependencies
└── README.md          # Documentation

Step 2: Basic robots.txt Checking

Python’s standard library includes urllib.robotparser for parsing robots.txt files.

from urllib.robotparser import RobotFileParser
from urllib.parse import urlparse

def can_fetch(url, user_agent='*'):
    """
    Check if a URL can be fetched according to robots.txt

    Args:
        url: The URL to check
        user_agent: Your bot's user agent string

    Returns:
        bool: True if allowed, False if disallowed
    """
    # Parse the URL to get the base domain
    parsed = urlparse(url)
    base_url = f"{parsed.scheme}://{parsed.netloc}"
    robots_url = f"{base_url}/robots.txt"

    # Create and configure the parser
    rp = RobotFileParser()
    rp.set_url(robots_url)

    try:
        rp.read()
        return rp.can_fetch(user_agent, url)
    except Exception as e:
        # If robots.txt doesn't exist or can't be read,
        # we assume scraping is allowed (defensive approach)
        print(f"Warning: Could not read robots.txt: {e}")
        return True

# Example usage
url = "https://example.com/products"
if can_fetch(url):
    print(f"Allowed to scrape: {url}")
else:
    print(f"Disallowed by robots.txt: {url}")

How it works:

Parse the target URL to extract the base domain
Construct the robots.txt URL (always at /robots.txt)
Download and parse the robots.txt file
Check if our URL is allowed for our user agent

Step 3: Building a Reusable Scraper Class

Let’s create a scraper class with built-in robots.txt compliance:

import requests
import time
from urllib.robotparser import RobotFileParser
from urllib.parse import urlparse, urljoin

class EthicalScraper:
    """
    A web scraper that respects robots.txt and implements rate limiting
    """

    def __init__(self, user_agent=None, respect_robots_txt=True):
        self.user_agent = user_agent or "EthicalScraper/1.0 (+https://scrapem.com)"
        self.respect_robots_txt = respect_robots_txt
        self.robots_cache = {}  # Cache robots.txt parsers by domain
        self.last_request_time = {}  # Track last request time per domain

    def _get_robots_parser(self, base_url):
        """Get or create a RobotFileParser for a domain"""
        if base_url not in self.robots_cache:
            rp = RobotFileParser()
            rp.set_url(urljoin(base_url, '/robots.txt'))
            try:
                rp.read()
                self.robots_cache[base_url] = rp
            except Exception as e:
                print(f"Warning: Could not read robots.txt for {base_url}: {e}")
                self.robots_cache[base_url] = None

        return self.robots_cache[base_url]

    def _get_crawl_delay(self, base_url):
        """Get the crawl delay for a domain from robots.txt"""
        rp = self._get_robots_parser(base_url)
        if rp:
            delay = rp.crawl_delay(self.user_agent)
            return float(delay) if delay else 1.0
        return 1.0

    def _apply_rate_limit(self, base_url):
        """Apply rate limiting based on robots.txt crawl-delay"""
        crawl_delay = self._get_crawl_delay(base_url)

        if base_url in self.last_request_time:
            elapsed = time.time() - self.last_request_time[base_url]
            if elapsed < crawl_delay:
                sleep_time = crawl_delay - elapsed
                print(f"Rate limiting: sleeping for {sleep_time:.2f} seconds")
                time.sleep(sleep_time)

        self.last_request_time[base_url] = time.time()

    def can_fetch(self, url):
        """Check if a URL can be fetched according to robots.txt"""
        if not self.respect_robots_txt:
            return True

        parsed = urlparse(url)
        base_url = f"{parsed.scheme}://{parsed.netloc}"

        rp = self._get_robots_parser(base_url)
        if rp is None:
            return True

        return rp.can_fetch(self.user_agent, url)

    def fetch(self, url):
        """
        Fetch a URL with robots.txt compliance and rate limiting

        Raises:
            ValueError: If the URL is disallowed by robots.txt
        """
        if not self.can_fetch(url):
            raise ValueError(f"URL disallowed by robots.txt: {url}")

        parsed = urlparse(url)
        base_url = f"{parsed.scheme}://{parsed.netloc}"

        self._apply_rate_limit(base_url)

        headers = {'User-Agent': self.user_agent}
        response = requests.get(url, headers=headers, timeout=30)
        response.raise_for_status()

        return response

if __name__ == "__main__":
    scraper = EthicalScraper()

    test_url = "https://httpbin.org/html"

    if scraper.can_fetch(test_url):
        print(f"Fetching: {test_url}")
        response = scraper.fetch(test_url)
        print(f"Status: {response.status_code}")
        print(f"Content length: {len(response.text)} bytes")
    else:
        print(f"Blocked by robots.txt: {test_url}")

Key features:

robots.txt caching: Parser objects are cached per domain for efficiency
Automatic rate limiting: Respects Crawl-delay directive
Clear error handling: Raises exceptions for disallowed URLs
Transparent user agent: Identifies itself clearly

Step 4: Handling Edge Cases

Let’s add handling for common edge cases:

from ethical_scraper import EthicalScraper
import requests
from urllib.parse import urlparse

class AdvancedEthicalScraper(EthicalScraper):
    """
    Enhanced scraper with better error handling and edge case support
    """

    def fetch(self, url, fallback_on_robots_error=True):
        try:
            parsed = urlparse(url)
            if not parsed.scheme or not parsed.netloc:
                raise ValueError(f"Invalid URL: {url}")

            if self.respect_robots_txt:
                try:
                    if not self.can_fetch(url):
                        print(f"Blocked by robots.txt: {url}")
                        return None
                except Exception as e:
                    if not fallback_on_robots_error:
                        raise
                    print(f"robots.txt check failed, proceeding: {e}")

            base_url = f"{parsed.scheme}://{parsed.netloc}"
            self._apply_rate_limit(base_url)

            headers = {'User-Agent': self.user_agent}
            response = requests.get(
                url,
                headers=headers,
                timeout=30,
                allow_redirects=True
            )
            response.raise_for_status()

            return response

        except requests.exceptions.Timeout:
            print(f"Timeout fetching {url}")
            return None
        except requests.exceptions.HTTPError as e:
            print(f"HTTP error {e.response.status_code}: {url}")
            return None
        except Exception as e:
            print(f"Error fetching {url}: {e}")
            return None

scraper = AdvancedEthicalScraper(
    user_agent="MyBot/1.0 (+https://scrapem.com; [email protected])"
)

urls = [
    "https://httpbin.org/html",
    "https://httpbin.org/delay/2",
    "https://example.com/robots.txt",
]

for url in urls:
    print(f"\nFetching: {url}")
    response = scraper.fetch(url)
    if response:
        print(f"Success: {response.status_code}")
    else:
        print(f"Failed")

Complete Working Example

Here’s a complete scraper that demonstrates all concepts:

import requests
import time
from urllib.robotparser import RobotFileParser
from urllib.parse import urlparse, urljoin
from typing import Optional, Dict
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class ProductionScraper:
    """
    Production-ready ethical scraper with full robots.txt compliance
    """

    def __init__(
        self,
        user_agent: str = "EthicalBot/1.0",
        respect_robots_txt: bool = True,
        default_delay: float = 1.0
    ):
        self.user_agent = user_agent
        self.respect_robots_txt = respect_robots_txt
        self.default_delay = default_delay
        self.robots_cache: Dict[str, Optional[RobotFileParser]] = {}
        self.last_request_time: Dict[str, float] = {}

    def get_base_url(self, url: str) -> str:
        parsed = urlparse(url)
        return f"{parsed.scheme}://{parsed.netloc}"

    def load_robots_txt(self, base_url: str) -> Optional[RobotFileParser]:
        if base_url in self.robots_cache:
            return self.robots_cache[base_url]

        rp = RobotFileParser()
        robots_url = urljoin(base_url, '/robots.txt')
        rp.set_url(robots_url)

        try:
            rp.read()
            self.robots_cache[base_url] = rp
            logger.info(f"Loaded robots.txt from {robots_url}")
            return rp
        except Exception as e:
            logger.warning(f"Could not load robots.txt from {robots_url}: {e}")
            self.robots_cache[base_url] = None
            return None

    def is_allowed(self, url: str) -> bool:
        if not self.respect_robots_txt:
            return True

        base_url = self.get_base_url(url)
        rp = self.load_robots_txt(base_url)

        if rp is None:
            return True

        return rp.can_fetch(self.user_agent, url)

    def get_crawl_delay(self, base_url: str) -> float:
        rp = self.load_robots_txt(base_url)
        if rp:
            delay = rp.crawl_delay(self.user_agent)
            return float(delay) if delay else self.default_delay
        return self.default_delay

    def wait_if_needed(self, base_url: str) -> None:
        delay = self.get_crawl_delay(base_url)
        if base_url in self.last_request_time:
            elapsed = time.time() - self.last_request_time[base_url]
            if elapsed < delay:
                wait_time = delay - elapsed
                logger.info(f"Rate limiting: waiting {wait_time:.2f}s")
                time.sleep(wait_time)
        self.last_request_time[base_url] = time.time()

    def fetch(self, url: str, **kwargs) -> Optional[requests.Response]:
        try:
            if not self.is_allowed(url):
                logger.warning(f"Blocked by robots.txt: {url}")
                return None

            base_url = self.get_base_url(url)
            self.wait_if_needed(base_url)

            headers = kwargs.pop('headers', {})
            headers['User-Agent'] = self.user_agent

            logger.info(f"Fetching: {url}")
            response = requests.get(
                url,
                headers=headers,
                timeout=kwargs.pop('timeout', 30),
                **kwargs
            )
            response.raise_for_status()
            return response

        except Exception as e:
            logger.error(f"Error fetching {url}: {e}")
            return None

if __name__ == "__main__":
    scraper = ProductionScraper(
        user_agent="MyBot/1.0 (+https://scrapem.com/about; [email protected])",
        default_delay=2.0
    )

    test_urls = [
        "https://httpbin.org/html",
        "https://httpbin.org/user-agent",
        "https://example.com",
    ]

    for url in test_urls:
        response = scraper.fetch(url)
        if response:
            print(f"Fetched {len(response.content)} bytes from {url}")
        else:
            print(f"Could not fetch {url}")

Common Issues and Troubleshooting

Issue 1: “Connection timeout” when reading robots.txt

Error message:

urllib.error.URLError: <urlopen error [Errno 60] Operation timed out>

Solution: The robots.txt file might not exist or the server is slow. Add timeout handling:

import socket

# Set a global timeout for URL operations
socket.setdefaulttimeout(10)

# Or handle in your code
try:
    rp.read()
except socket.timeout:
    logger.warning("robots.txt request timed out, assuming allowed")
    return True

Issue 2: robots.txt returns 404

Solution: This is normal! If robots.txt doesn’t exist, assume scraping is allowed:

from urllib.error import HTTPError

try:
    rp.read()
except HTTPError as e:
    if e.code == 404:
        logger.info("No robots.txt found, proceeding")
        return True
    raise

Issue 3: Crawl-delay is too aggressive (e.g., 10 seconds)

Solution: Respect the delay or don’t scrape that site. Consider:

Using the site’s official API if available
Contacting the site owner for permission
Finding alternative data sources

Best Practices Summary

Do:

Always check robots.txt before scraping
Use descriptive user agents with contact information
Respect crawl-delay directives
Cache robots.txt parsers to avoid repeated requests
Handle errors gracefully when robots.txt is unavailable
Log your actions for debugging and compliance tracking
Be conservative — if in doubt, don’t scrape

Don’t:

Bypass robots.txt checks
Use misleading user agents (like “Googlebot”)
Ignore rate limits even if technically possible
Scrape aggressively — respect server resources

Ethical and Legal Considerations

This project/tutorial follows strict ethical guidelines for responsible data collection:

🤖 robots.txt Compliance

All code examples in this tutorial demonstrate proper robots.txt checking using Python's built-in urllib.robotparser module. The scrapers check robots.txt before making any requests and cache the results for efficiency.

⏱️ Rate Limiting

The production scraper implements rate limiting based on the Crawl-delay directive from robots.txt:

Default delay: 1-2 seconds between requests
Respects site-specific crawl-delay directives
Per-domain rate limiting to avoid overwhelming servers

👤 User Agent Identification

All examples use descriptive user agents that identify the bot and provide contact information:
MyBot/1.0 (+https://scrapem.com/about; [email protected])

This allows website owners to contact you if there are issues.

🔒 Data Privacy

This tutorial focuses on accessing publicly available web pages only. No authentication bypassing, personal data collection, or login-protected content scraping is demonstrated.

📜 Terms of Service

The code respects robots.txt which is the primary mechanism for websites to communicate their scraping policies. Always review a website's Terms of Service before scraping.

Note: All scraping and data collection activities should comply with applicable laws, respect website owner rights, and follow industry best practices. This includes but is not limited to GDPR, CCPA, and local data protection regulations.

Next Steps

Now that you understand robots.txt compliance, explore these related topics:

Advanced Rate Limiting: Implement exponential backoff and request queues
Proxy Rotation: Scale scraping while maintaining compliance
Error Handling: Add retry logic and failure recovery to your scrapers
Data Validation: Verify your scraped data is correct

On This Page

Respecting robots.txt in Python Web Scrapers: A Complete Guide

Introduction

Why robots.txt Matters

Understanding robots.txt

Step 1: Environment Setup

Install Dependencies

Project Structure

Step 2: Basic robots.txt Checking

Step 3: Building a Reusable Scraper Class

Step 4: Handling Edge Cases

Complete Working Example

Common Issues and Troubleshooting

Issue 1: “Connection timeout” when reading robots.txt

Issue 2: robots.txt returns 404

Issue 3: Crawl-delay is too aggressive (e.g., 10 seconds)

Best Practices Summary

Ethical and Legal Considerations

🤖 robots.txt Compliance

⏱️ Rate Limiting

👤 User Agent Identification

🔒 Data Privacy

📜 Terms of Service

Next Steps

Comments

Introduction

Why robots.txt Matters

Understanding robots.txt

Step 1: Environment Setup

Install Dependencies

Project Structure

Step 2: Basic robots.txt Checking

Step 3: Building a Reusable Scraper Class

Step 4: Handling Edge Cases

Complete Working Example

Common Issues and Troubleshooting

Issue 1: “Connection timeout” when reading robots.txt

Issue 2: robots.txt returns 404

Issue 3: Crawl-delay is too aggressive (e.g., 10 seconds)

Best Practices Summary

Ethical and Legal Considerations

🤖 robots.txt Compliance

⏱️ Rate Limiting

👤 User Agent Identification

🔒 Data Privacy

📜 Terms of Service

Next Steps

Related Resources

Comments