Web scraping framework · async crawling at scale · verified against Scrapy 2.18 (2026)

Scrapy cheat sheet

Scrapy is the framework for large-scale web scraping: you write a Spider that issues Requests and parses the Responses, yielding scraped items and follow-up requests. It handles concurrency, throttling, retries, cookies, and export — an async engine (native async/await since 2.14). Great for structured data collection to feed ML/analytics. This sheet targets Scrapy 2.18 (Python 3.10+).

project & spider selectors (extract) requests & following items & pipelines settings & tools gotcha ★ most common

Verified 2026-08-24 against the official docs at docs.scrapy.org (Scrapy 2.18) and scrapy.org. Maintained by Zyte. Scrape responsibly — respect robots.txt, rate limits & terms of service.

Outline

A spider yields two things from parse: items (data) and requests (more pages to crawl). Everything else is extracting fields and configuring behavior.

Build a Spider

Scaffold a project, write the crawl entry point.

1Project & spider2.18

pip install scrapy scrapy startproject myproj★
Scaffolds settings, pipelines, and a spiders/ folder. scrapy genspider quotes quotes.toscrape.com stubs a spider.
import scrapy class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = ["https://quotes.toscrape.com/"]★
name is unique & how you run it. start_urls seed the crawl (default callback is parse).
async def start(self): yield scrapy.Request("https://...", self.parse)
2.14+ async entry point — use instead of start_urls/start_requests when you need custom headers/auth per seed.

2parse & runthe callback

def parse(self, response): for q in response.css("div.quote"): yield {"text": q.css("span.text::text").get()}★
parse receives the Response; yield dicts/items (data) and yield Requests (more pages).
scrapy crawl quotes -O quotes.jsonl★
Run it. -O overwrites, -o appends; format from the extension (.json/.jsonl/.csv/.xml).
scrapy crawl quotes -a category=books -s DOWNLOAD_DELAY=1
-a passes spider args (arrive as self.category); -s overrides a setting for the run.

Extract

Pull fields out of the response.

3CSS & XPath selectorsquery the DOM

response.css("h1.title::text")★
CSS with Scrapy's ::text (node text) and ::attr(href) pseudo-elements. Returns a SelectorList.
response.xpath("//h1[@class='title']/text()")★
XPath when you need axes/predicates CSS can't express. Mix freely: sel.css(...).xpath("./@href").
for row in response.css("table tr"): row.css("td::text").getall()★
Selectors are chainable — iterate container elements, then extract relative fields from each (use . in relative XPath).

4get / getall / attribto Python values

sel.get() # first match as str, or None sel.getall() # list of all matches★
get() returns None if nothing matched (safe); get(default="") to set a fallback.
sel.attrib["href"] · sel.css("a::attr(href)").get()★
Two ways to read an attribute.
sel.re(r"\d+") · sel.re_first(r"\$(\d+)")
Regex over the matched text when structure isn't enough.
response.json() # parse a JSON API response
Many "scrapes" are really hidden JSON APIs — check the network tab first; response.json() beats HTML parsing.

Crawl

Follow links and craft requests.

5Following linkspagination & detail

yield response.follow("a.next::attr(href)", callback=self.parse)★
follow resolves relative URLs for you — the clean way to paginate.
yield from response.follow_all(css="div.item a", callback=self.parse_item)★
Follow many links at once (a listing page of detail links).
yield scrapy.Request(url, callback=self.parse_detail)★
The explicit form when you have a full URL or need request options.

6Requests, meta, formscustomize

scrapy.Request(url, cb_kwargs={"category": cat}) def parse_detail(self, response, category): ...★
cb_kwargs is the modern way to pass data to the callback (cleaner than meta).
scrapy.Request(url, meta={"item": item}, headers={...}, dont_filter=True)
meta also carries per-request config (proxy, download_timeout). dont_filter bypasses dedup for intentional re-requests.
yield scrapy.FormRequest(url, formdata={"user":..., "pass":...}, callback=self.after_login)★
POST a form (login/search). FormRequest.from_response(response, formdata=...) auto-fills hidden fields. JsonRequest for JSON APIs.

Structure & Store

Typed items and post-processing.

7Items & loadersstructured output

class Product(scrapy.Item): name = scrapy.Field() price = scrapy.Field()★
A schema for scraped data (vs a raw dict) — typos on field names raise. Dataclasses/attrs items also supported.
from scrapy.loader import ItemLoader l = ItemLoader(Product(), response) l.add_css("name", "h1::text"); l.add_xpath(...); return l.load_item()★
ItemLoader collects fields and applies input/output processors (strip, join, take-first) — cleaner than manual dicts.
MapCompose(str.strip) · TakeFirst() · Join()
Processors normalize values as they're loaded (whitespace, single-vs-list).

8Pipelines & exportprocess items

class PricePipeline: def process_item(self, item, spider): item["price"] = float(item["price"]); return item★
Pipelines post-process every yielded item — clean, validate, dedupe, or write to a DB. Raise DropItem to discard.
ITEM_PIPELINES = {"myproj.pipelines.PricePipeline": 300}★
Enable in settings; the number is the order (low runs first).
FEEDS = {"out.jsonl": {"format": "jsonlines"}}
Configure exports in settings (to S3/GCS too) instead of -o; supports multiple feeds & per-feed fields.

Operate

Tune behavior and debug interactively.

9Settings & the shellbe a good citizen

ROBOTSTXT_OBEY = True DOWNLOAD_DELAY = 1.0 · CONCURRENT_REQUESTS_PER_DOMAIN = 8★
Politeness knobs. Respect robots.txt, throttle requests — the difference between scraping and getting banned.
AUTOTHROTTLE_ENABLED = True★
Adaptively adjusts delay to the site's latency — the recommended default for real crawls.
USER_AGENT = "mybot (+https://example.com)" · DEFAULT_REQUEST_HEADERS = {...}
Identify your bot; set headers/cookies. HTTPCACHE_ENABLED caches during development.
scrapy shell "https://site.com/page"★
The dev superpower: an interactive REPL with response loaded — perfect your selectors before writing the spider.

!Common gotchasread before shipping

JavaScript-rendered pages
Scrapy fetches raw HTML — it doesn't run JS. If content is missing, find the underlying API (response.json()) or use scrapy-playwright.
get() returns None, not ""
A missing match yields None — float(None) crashes. Use get(default="") or guard before converting.
duplicate filter drops re-requests
Scrapy dedupes URLs by default. Intentionally re-request the same URL with dont_filter=True.
relative XPath needs a leading dot
Inside a loop, row.xpath("//td") re-queries the whole page; use row.xpath(".//td") for row-relative.
scrape responsibly
Honor robots.txt/ToS, rate-limit, and don't collect personal data you're not allowed to. Legal & ethical, not just technical.

Worth memorizing

yield items AND requestsparse returns data (dicts/Items) and follow-up Requests

::text and ::attr(href)Scrapy's CSS pseudo-elements; .get()/.getall()

get() is None-safeget(default="") before converting types

response.follow for paginationresolves relative URLs; follow_all for many

cb_kwargs to pass datacleaner than meta for callback arguments

scrapy shell "url"perfect selectors interactively before coding

check for a hidden JSON APIresponse.json() beats parsing JS-rendered HTML

AutoThrottle + robots + delaypoliteness = not getting banned

pipelines process every itemclean/validate/store; DropItem to discard

Scrapy cheat sheet (scrapy_cheatsheet v0).
Verified 2026-08-24 against docs.scrapy.org (Scrapy 2.18, Python 3.10+) and scrapy.org (maintained by Zyte). Async crawling framework: Spider → Request → Response → parse, with selectors, items/loaders, pipelines and feed exports. For JS-heavy sites pair with scrapy-playwright. Always scrape within robots.txt, rate limits and terms of service — and applicable law.
Design system = shared cheat-sheet tokens (Inter + JetBrains Mono, light theme). Content only.