Scrapy gets filed under "can't handle modern websites" because it doesn't run JavaScript. That reputation is backwards. Not rendering the page is the entire idea, and once you see the idea in action it stops looking like a missing feature.
I proved it to myself in a single run. I built a JavaScript-rendered catalog fixture, pointed Scrapy at the page a browser would show, and pulled back 0 product cards. Then I aimed the same spider at the JSON endpoint that page was quietly calling in the background and got 8/8 items, clean. Same tool, same session, opposite outcomes — and the gap between those two numbers is what this whole review is about.
What Scrapy actually is (and isn't)
Scrapy is a Python framework for crawling sites and pulling out structured data. That's the maintainers' own framing in the overview docs, and after using it, it's accurate — no marketing inflation to correct. It's old enough and established enough to be the reflex answer when a Python developer asks what serious people scrape with, and the repo backs that up: roughly 62,981 GitHub stars as of 2026-07-07 (scrapy/scrapy), with 11,773 forks and 590 open issues on the same day. BSD-3-Clause license, Python 3.10 or newer, and the version I put through its paces was 2.17.0, which happened to ship the morning I ran these tests — so no trailing-version asterisk on this one.
Here's the line that separates it from the newer AI-crawler crowd: Scrapy is HTTP-only by default. No browser. No rendering engine. It pulls HTML down over the wire, hands it to a parser, and lets you lift fields out with CSS selectors or XPath. Calling that a limitation is half right and misses the design entirely. Scrapy's premise is that spinning up a headless Chrome for a routine scrape is usually the wrong move — the smarter path is to find the data request the page already makes and hit it straight on.
That premise isn't me projecting onto the tool. The official dynamic content docs state it outright: locate and reproduce the underlying data request first, and reach for a headless browser only as a fallback when reproducing that request isn't practical. Most scrapers open the browser first and never think about the API. Scrapy flips the default.
Key features, and the design call behind each
Under the hood, Scrapy is a stack of parts that each assume one thing about you: that you're a developer who wants control, not a one-click wizard.
Spiders. You write a class, hand it start URLs, and define a parse callback that yields items or follows more links. That's more typing than a no-code extractor — the extraction rules are yours to author — and in exchange you get exact say over what gets captured and where the crawl heads next.
Selectors. Parsing sits on parsel, which is lxml underneath. CSS and XPath are both first-class citizens, not bolt-ons. The lxml backing is why selection stays fast and why the extraction code reads like intent instead of a tangle of string-slicing.
Feed exports. Aim a spider at a file and Scrapy serializes your items to JSON, JSON Lines, CSV, or XML with no extra plumbing. In my run, one static-catalog spider dropped both JSON and CSV without a single line of export code from me — the feed export story is real, and it does what it says.
AutoThrottle and crawl controls. Requests are scheduled asynchronously on Twisted, and you get concurrency caps, download delays, depth restrictions, AutoThrottle for adaptive rate-limiting, and robots.txt obedience. These are the controls that keep a broad crawl from turning into a server-hammering incident.
HTTP-only, restated as a feature. No browser means low memory, high throughput, and no rendering engine to nurse — provided the data you want is reachable over plain HTTP. Which, more often than the browser-first crowd assumes, it is.
Setup: the dependency stack nobody screenshots
Installation was a non-event, which for a framework this size is worth stating plainly. pip install Scrapy==2.17.0 finished cleanly in a fresh virtual environment on macOS arm64, pulling binary wheels, with nothing compiling itself into a wall. Nothing dramatic to report — and that's the point.
Look at what came down the pipe, though. scrapy version -v reported Scrapy 2.17.0 riding on lxml 6.1.1, Twisted 26.4.0, pyOpenSSL 26.3.0, and cryptography 49.0.0, with parsel, cssselect, and tldextract rounding out the crowd. That's a genuine footprint — a whole crawling framework's worth of dependencies, not a one-file HTML parser. On this machine, wheels existed for all of it and the install stayed painless. On other setups, the official docs still warn about platform-specific dependency friction, and historically it's the cryptography and Twisted end of the stack where that bites, so budget for it if you're on something unusual. Setup was smooth here; the size of what gets installed is still a fair thing to know before you commit, because you're pulling in a framework and it weighs what a framework weighs.
Hands-on: what held up
Once it was installed, the static path was clean. Full recall, nothing dropped.
| Test | Result | Runtime |
|---|---|---|
| Local static catalog + pagination | 12/12 products | 0.557s |
| Static catalog CSV export | 12 rows written | (same run) |
| Article extraction | title + 3/3 body paragraphs | 0.416s |
| Crawl graph, DEPTH_LIMIT=2 | 11 pages across depths 0/1/2 | 0.904s |
| Local 500 page | status 500 captured, no crash | 0.424s |
| Books to Scrape (public) | 20 products | 2.053s |
| Quotes to Scrape spider (public) | 12 quote items | 3.465s |
The static-catalog spider walked pagination from page one to page two and captured 12/12 expected records, then wrote them out as JSON and CSV in the same pass. The article fixture is the one worth sitting with. Scrapy made no attempt to auto-clean the page into tidy Markdown — instead it let me target article fields with explicit selectors and parked nav and footer text in separate fields, so I got 3/3 body paragraphs with the boilerplate quarantined rather than smeared through the output. That's the trade: you write the selectors, you get exactly what you asked for and nothing you didn't.
Crawl control checked out at small scale. With DEPTH_LIMIT=2, a short download delay, per-domain concurrency, and robots.txt on, the crawl graph saw 11 pages across depths 0, 1, and 2, and the depth counting behaved. Failure handling was just as undramatic. The intentional 500 page came back as a structured item with status 500 surfaced through handle_httpstatus_list — no exception, no dead run. Scrapy treats an error status as something you handle inside spider logic instead of a surprise that takes the crawl down with it.
Hands-on: the JavaScript wall and the door beside it
Now the result this review is built around.
I pointed Scrapy's HTTP fetcher at a JavaScript-rendered catalog fixture. It downloaded the source HTML, found 0 .product-card nodes, and moved on — because it never ran the script that would have drawn those cards. The public Quotes to Scrape JS page told the same story: 0 rendered quote nodes. Stop the test there and you'd write Scrapy off as unfit for anything built this decade.
Don't stop there. That JS catalog was populated by a JSON API in the background, the way most of them are. I pointed the same Scrapy spider at that endpoint and got 8/8 products in 0.416s — no browser, no rendering, just a request to the URL the page was already calling and a parse of the JSON that came back.
That side-by-side is the reproduce-the-request philosophy in miniature. The rendered page is a decoy; the data was sitting behind an API the whole time, and Scrapy's design steers you toward hitting it directly instead of paying for a headless browser to sit and watch a page assemble itself. It's faster, it's lighter, and it breaks less — an API contract is a steadier thing to depend on than a pile of client-side DOM. The catch is that it's manual. You have to open the network tab, find the request, and reproduce its headers and params yourself. Scrapy won't discover the API for you; it just makes hitting it trivial once you have.
Two boundaries, stated plainly. When there genuinely is no underlying request to reproduce — data baked in by client-side rendering with no API behind it — Scrapy needs a headless-browser integration you wire in yourself, and I didn't exercise that path in this pass. And everything above ran on small fixtures and public demo pages. I did not run a 100-to-1,000-page crawl, so I'm making no claims about memory, throughput, or retry behavior at scale — the async core and crawl controls are strong signals, but a signal isn't a measurement.
Pros and cons
Pros:
- HTTP-only design is fast and light — 12/12 static recall in about half a second, 8/8 from a JSON API in 0.416s, zero browser overhead.
- The reproduce-the-request approach genuinely delivers: a JS page that returned 0 gave up all 8 items through its backing API.
lxml-backed CSS and XPath selectors keep extraction code readable and quick.- Feed exports to JSON/CSV/XML with no export plumbing to write.
- Explicit error handling — a 500 comes back as a status you catch, not a crash.
- Mature crawl controls: concurrency, delays, depth limits, AutoThrottle, robots.txt.
- Permissive BSD-3-Clause license; a clean install on a current machine.
Cons:
- Does not render JavaScript by design — 0 nodes on a client-rendered page until you locate the API yourself.
- Finding the underlying request is manual; Scrapy won't point you at the endpoint.
- Sizable dependency stack (Twisted, lxml, cryptography, pyOpenSSL, parsel, tldextract) — smooth here, but historically a friction point on unusual platforms.
- More code than no-code or auto-extraction tools; the spiders are yours to write and maintain.
- My testing covered small fixtures and demo sites, not large crawls — scale reliability is unproven in this pass.
Who it's for, and who should walk past
Scrapy is for developers who want code-level control and think in requests, not pages. If your gut reaction to a sluggish JavaScript site is "there's an API under here somewhere," the tool is built for exactly that instinct. It rewards people who are comfortable writing selectors, reading a network tab, and owning their extraction logic end to end. For static sites, paginated catalogs, and anything backed by a discoverable JSON endpoint, it's quick and it's precise.
Walk past it — or at least pair it with something else — if writing and maintaining spider code isn't how you want to spend your time, or if your targets render their data purely client-side with no reproducible request and you'd rather not bolt on a headless browser yourself. And if the dream was pointing a tool at a URL and getting clean structured output without authoring extraction rules, that was never Scrapy's job, and it has never pretended otherwise.
Alternatives, and where Thunderbit fits
Try Thunderbit for Web Data Extraction
Start from what you're signing up for: a free, open-source framework you run and maintain yourself. You own the spiders, the dependency stack, and the work of finding each site's data request. In exchange you pay nothing per request, keep everything in-house, and get exact control. For plenty of teams that's the correct call, and this review isn't here to talk anyone out of it.
The trade-off lives in the rendering-and-drift problem, and Scrapy's answer is that you solve it: you find the API, you reproduce the request, and you handle the no-API case by wiring in a browser yourself. A managed AI scraping API lifts that layer off your plate instead. That's the slot Thunderbit's developer stack sits in for technical readers — an AI scraping API plus MCP server plus CLI, not the browser extension the sales-and-ops crowd uses. POST /distill turns a page into clean, LLM-ready Markdown; POST /extract returns structured JSON against a schema you define; and both handle JavaScript rendering, anti-bot, and dynamic content server-side — including the client-rendered case where Scrapy asks you to reach for a browser. There's an MCP server for AI agents and coding assistants (with a free thunderbit_suggest_fields to scope a page before you spend anything), and a CLI through npx @thunderbit/thunderbit-cli for terminal, CI, or cron work.
The difference isn't quality, it's ownership. Scrapy is an explicit engineering framework: you maintain the spider, the pipeline, and the JS strategy, and you get total control at zero per-call cost. Thunderbit's stack hands off the render-and-extract layer as a managed service, so you skip the network-tab spelunking and pay per call instead. Small, code-first, and you like owning every step? Scrapy's control is the better fit. Scaling across a hundred sites and you'd rather not hand-reproduce a request per site? The managed route removes that entire category of work.
For the wider field, these benchmark write-ups cover the neighbors: the full open-source scraper comparison, Colly's no-browser Go crawler review, and Scrapling's adaptive-selector review.
Verdict
Should you use Scrapy? Yes — if you're a developer who wants control and you buy the worldview: don't render the page, find the request behind it. In testing, that philosophy paid off exactly as advertised. A JavaScript catalog handed the HTTP fetcher 0 cards; the JSON API feeding it gave up all 8 items to the same spider. Static extraction hit 12/12, article selectors kept 3/3 paragraphs clear of boilerplate, the crawl graph respected its depth limit across 11 pages, and a 500 came back as a handled status instead of a crash.
Size the claims correctly, though. Scrapy does not render JavaScript, and it won't find the API for you — that reflex is yours to build. The dependency stack is a framework's worth and can bite on odd platforms even though it was clean here. And I tested fixtures and demo pages, not a thousand-page crawl, so treat the scale story as promising but unproven. Inside those lines, Scrapy is the tool that best commits to a quietly radical idea: the fastest way through a web page is usually not through the web page at all.
Try Thunderbit for Web Data Extraction Get Started Free
FAQs
Can Scrapy scrape JavaScript-rendered pages? Not with its default HTTP fetcher — it returned 0 nodes on both a JS fixture and the public Quotes JS page, because it downloads HTML without running a browser. The intended path is to find the underlying data request the page makes and hit that directly; in my test, the JSON API behind a JS catalog gave up all 8 items. For pages with no reproducible request, you wire in a headless browser yourself.
What does "reproduce the request" actually mean? Most dynamic pages load their data from a JSON API in the background, then render it client-side. Rather than running a browser to watch that happen, you open your network tab, find that API call, and point Scrapy at it directly. It's faster and steadier than rendering — an API contract breaks less often than a DOM — but it's manual work, and Scrapy won't locate the endpoint for you.
Is Scrapy hard to install?
It was clean for me — pip install Scrapy==2.17.0 finished with no compilation errors in a fresh venv on macOS using binary wheels. But it pulls in a sizable stack (Twisted, lxml, cryptography, pyOpenSSL, parsel, tldextract), and the official docs still warn about platform-specific dependency friction on some systems, so budget for that if you're on something unusual.
What output formats does Scrapy support? Feed exports cover JSON, JSON Lines, CSV, and XML out of the box — point a spider at a file and it serializes your items with no extra code. In my run, one spider produced both JSON and CSV from a single pass. Note that it exports the fields you selected; it doesn't auto-clean a page into Markdown.
Is Scrapy free for commercial use? It's BSD-3-Clause, which is permissive and commercially friendly. As always, confirm the current license on the repo before you build on it, and keep your user-agent, proxy, and rate-limit choices responsible — capability isn't permission.
