Scrape websites at scale using Scrapy, a Python web crawling and scraping framework. Use when: (1) Crawling multiple pages or entire sites, (2) Extracting structured data from HTML/XML, or (3) Building automated data pipelines from web sources.
Scrapy is a fast, high-level Python web crawling and scraping framework. It enables structured data extraction from websites, supports crawling entire sites, and integrates pipelines to process and store scraped data.
Install options:
# pip
pip install scrapy
# Ubuntu/Debian
sudo apt-get install -y python3-pip && pip install scrapy
# macOS
brew install python && pip install scrapy
# Verify installation
scrapy version
Create and run a simple Scrapy spider to scrape a single page.
# Create a new Scrapy project
scrapy startproject myproject
cd myproject
# Generate a spider
scrapy genspider quotes quotes.toscrape.com
# Run the spider and save to JSON
scrapy crawl quotes -o output.json
# Run the spider and save to CSV
scrapy crawl quotes -o output.csv
Python spider (quotes.py):
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = ["https://quotes.toscrape.com"]
def parse(self, response):
for quote in response.css("div.quote"):
yield {
"text": quote.css("span.text::text").get(),
"author": quote.css("small.author::text").get(),
"tags": quote.css("a.tag::text").getall(),
}
# Follow pagination links
next_page = response.css("li.next a::attr(href)").get()
if next_page:
yield response.follow(next_page, self.parse)
Production-oriented spider with settings, item pipelines, and error handling.
# Run with custom settings (rate limiting, retries)
scrapy crawl quotes \
-s DOWNLOAD_DELAY=1 \
-s AUTOTHROTTLE_ENABLED=True \
-s RETRY_TIMES=3 \
-o output.json
# Run from a script (no project required)
scrapy runspider spider.py -o output.json
Python with error handling and structured items:
import scrapy
from scrapy import signals
from scrapy.crawler import CrawlerProcess
class ArticleSpider(scrapy.Spider):
name = "articles"
custom_settings = {
"DOWNLOAD_DELAY": 1,
"AUTOTHROTTLE_ENABLED": True,
"AUTOTHROTTLE_START_DELAY": 1,
"AUTOTHROTTLE_MAX_DELAY": 10,
"ROBOTSTXT_OBEY": True,
"USER_AGENT": "open-skills-bot/1.0 (+https://github.com/besoeasy/open-skills)",
"RETRY_TIMES": 3,
"FEEDS": {"output.json": {"format": "json"}},
}
def __init__(self, start_url=None, *args, **kwargs):
super().__init__(*args, **kwargs)
self.start_urls = [start_url or "https://quotes.toscrape.com"]
def parse(self, response):
for article in response.css("article, div.post, div.entry"):
yield {
"url": response.url,
"title": article.css("h1::text, h2::text").get("").strip(),
"body": " ".join(article.css("p::text").getall()),
}
for link in response.css("a::attr(href)").getall():
if link.startswith("/") or response.url in link:
yield response.follow(link, self.parse)
def errback(self, failure):
self.logger.error(f"Request failed: {failure.request.url} — {failure.value}")
# Run without a Scrapy project
if __name__ == "__main__":
process = CrawlerProcess()
process.crawl(ArticleSpider, start_url="https://quotes.toscrape.com")
process.start()
Use XPath selectors for precise extraction from complex HTML structures.
import scrapy
class XPathSpider(scrapy.Spider):
name = "xpath_example"
start_urls = ["https://quotes.toscrape.com"]
def parse(self, response):
for quote in response.xpath("//div[@class='quote']"):
yield {
"text": quote.xpath(".//span[@class='text']/text()").get(),
"author": quote.xpath(".//small[@class='author']/text()").get(),
"tags": quote.xpath(".//a[@class='tag']/text()").getall(),
}
Scrapy yields Python dicts (or Item objects) per scraped record. When saved to file:
output.json — Array of JSON objects, one per itemoutput.csv — CSV with headers matching dict keysoutput.jsonl — One JSON object per line (memory-efficient for large crawls)Example item:
{
"text": "The world as we have created it is a process of our thinking.",
"author": "Albert Einstein",
"tags": ["change", "deep-thoughts", "thinking", "world"]
}
Error shape: Scrapy logs errors to stderr; unhandled HTTP errors trigger the errback method if defined.
ROBOTSTXT_OBEY = True to respect robots.txt automaticallyDOWNLOAD_DELAY (seconds between requests) to avoid overloading serversAUTOTHROTTLE_ENABLED = True for adaptive rate limitingUSER_AGENT identifying your botCONCURRENT_REQUESTS_PER_DOMAIN = 1 for polite single-domain crawlingHTTPCACHE_ENABLED = TrueYou have scrapy web-scraping capability. When a user asks to scrape or crawl a website:
1. Confirm the target URL and data fields to extract (e.g., title, price, link)
2. Create a Scrapy spider using CSS or XPath selectors to target those fields
3. Enable ROBOTSTXT_OBEY=True and set DOWNLOAD_DELAY>=1 to be polite
4. Follow pagination links if the user needs data across multiple pages
5. Save results to output.json or output.csv
Always identify your bot with a descriptive USER_AGENT and never scrape login-protected or paywalled content.
Error: "Forbidden by robots.txt"
ROBOTSTXT_OBEY = False if you have explicit permission from the site ownerError: "Empty or missing data"
None valuesscrapy shell <url>) and adjust your CSS/XPath selectors to match the actual HTML structureError: "Too many redirects / 429 Too Many Requests"
DOWNLOAD_DELAY, enable AUTOTHROTTLE_ENABLED = True, or add a Retry-After respecting middlewareError: "JavaScript-rendered content not found"
scrapy-playwright or scrapy-splash middleware to render JavaScript before parsing