Apache Nutch is an Apache Software Foundation crawler whose development began in 2004. It is a JVM system built on Hadoop, organized as a loop rather than a single streaming command: seed URLs are injected into a persistent database, followed by rounds of generate → fetch → parse → updatedb, with plugin slots for protocol, parser, URL filter, and scoring. The usual output feeds a search index such as Solr or Elasticsearch rather than a CSV.
I ran Nutch 1.22 against a controlled local test site — one that logs every request server-side, so results are judged by what the server actually saw rather than what the crawler claimed. Four boundaries shaped the run: the JDK version, http.agent.name, crawl scope, and whether parse-js appears in plugin.includes. The full cycle ran repeatedly with the tested configuration; change the JDK or leave the agent identity unset and it stops before fetching useful pages.
The JDK boundary appears before any crawl begins. Nutch 1.22 did not start here on JDK 26.0.1: the first Hadoop job died inside Subject.getSubject() after Java removed the SecurityManager path. Nutch bundles Hadoop 3.4.2, while the fix landed in Hadoop 3.4.3 seven days after Nutch 1.22 shipped. Separately, parse-js changed recovery of two JavaScript-file literals from 0/2 to 2/2 without executing a browser.
What Nutch is for, and what it isn't
Nutch is not a scraper. Structured field extraction is not its job: it discovers and fetches URLs at volume, maintains a persistent database of those URLs and their states (the crawldb), and hands you segments that something else turns into an index. Point it at a catalog expecting a table of names and prices and you'll get a crawldb instead.
That architecture explains most of what follows. Nutch predates the single-binary crawler era by roughly two decades, and it's built for the problem Hadoop was built for: crawling more pages than fit on one machine. Running it on a laptop against a 12-page fixture is like renting a freight train to move a bookshelf — informative about the train, unfair to expect a bicycle's ergonomics.
The current release is 1.22, announced 17 February 2026. It's Apache-2.0, the repo sat at 3,272 stars with 8 open issues when I checked on July 27, 2026, and master had been pushed to four days before that. This is a maintained project, not an abandoned one — which is the right lens for the JDK problem: a packaging window that closed a week too early, not neglect.
The version matrix: JDK 24+, Hadoop 3.4.2, and a two-line fix
The blocker is a three-way version interaction, and the only part of it under your control is which JDK Nutch runs on. The very first Hadoop job, on the host's default JDK, died at initialization:
java.lang.UnsupportedOperationException: getSubject is not supported
at java.base/javax.security.auth.Subject.getSubject(Subject.java:277)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:588)
at org.apache.nutch.crawl.Injector.inject(Injector.java:473)
Return code 255. Zero pages fetched. bin/nutch inject doesn't even reach the network — it initializes a Hadoop LocalJobRunner, which asks who the current user is, which calls Subject.getSubject(), which JEP 486 turned into an unconditional exception when JDK 24 permanently removed the SecurityManager. My host JDK was OpenJDK 26.0.1, well past that line.
The traditional escape hatch doesn't work either. Adding -Djava.security.manager=allow, the flag that used to re-enable the old behavior, gets rejected by the VM before Nutch's code loads at all:
Error occurred during initialization of VM
java.lang.Error: A command line option has attempted to allow or enable the Security
Manager. Enabling a Security Manager is not supported.
That's return code 1, and it's a dead end by design — the flag was removed along with the feature.
The root cause is in the Hadoop version bundled with Nutch 1.22. The getSubject problem is tracked as HADOOP-19212 and fixed in Hadoop 3.4.3 and 3.5.0; Nutch 1.22 bundles hadoop-common-3.4.2. Nutch 1.22 shipped on 17 February 2026, and Hadoop 3.4.3 followed about a week later.
Nor is it a Solr or Hadoop-cluster dependency problem. There's a common assumption that Nutch needs a Hadoop cluster and a running Solr to do anything. It doesn't. Local mode runs Hadoop's in-process LocalJobRunner — no HDFS daemon, no YARN, no cluster. The entire inject → generate → fetch → parse → updatedb cycle runs on a single machine with nothing else installed. The JDK wall is purely a bundled-library version issue, and it stops you before any of that infrastructure question comes up.
The practical version matrix, all three rows measured:
| JDK used | Command | Result |
|---|---|---|
| OpenJDK 26.0.1 | bin/nutch inject | Fails, rc=255 — UnsupportedOperationException: getSubject is not supported |
| OpenJDK 26.0.1 | bin/nutch inject + -Djava.security.manager=allow | Fails, rc=1 — VM refuses to start |
| OpenJDK 17.0.20 (LTS) | bin/nutch inject | Works, rc=0 — Total new urls injected: 1 |
The fix is two commands. Install an LTS JDK and point Nutch at it:
brew install openjdk@17
export NUTCH_JAVA_HOME=/opt/homebrew/opt/openjdk@17
Keg-only, so it doesn't touch the system default. Nutch's own CI targets Java 17, and the project has publicly announced that 1.22 is the last release to run on Java 11 and that 1.23 will require Java 17. So an LTS JDK isn't a workaround — it's the supported configuration. The mismatch is between what Nutch supports and what brew install openjdk hands you in 2026, and those are two different questions that happen to collide at the very first command.
Everything from here on ran on OpenJDK 17.0.20, where the whole cycle is clean.
Setup, measured: 396 MB and one property that blocks everything
"Heavyweight" is the adjective everyone reaches for, and it's useless without a scale. Here is what the unpacked Nutch 1.22 binary distribution actually holds:
| Item | Nutch 1.22 binary distribution |
|---|---|
| Unpacked size | ≈396 MB |
Jars in lib/ | 188 (≈113 MB) |
| — of which the bundled Hadoop stack | 13 |
| Plugin directories | 78 |
| Jars inside those plugin directories | 533 |
| Config files | 35 |
Scripts in bin/ | 2 — crawl and nutch |
For scale, a modern Go crawler like katana ships as one ~50 MB binary with no JVM and no external jars.
Then there's the gate nobody warns you about. The shipped nutch-site.xml is empty, and http.agent.name defaults to an empty string. With it unset, my first crawl fetched zero paths and logged:
ERROR Fetcher: No agents listed in 'http.agent.name' property.
Setting that one property — nothing else — flipped it to a working fetch. With the property empty, the command completed without fetching pages and the log reported the agent-name error above; it was not silent.
Minimum viable config turned out to be three artifacts: conf/nutch-site.xml (agent name, plugin set, scope), conf/regex-urlfilter.txt (host scoping), and a seed URL file. That's not a bad number. It's just three more files than crawler run <url>.
What it found: the plugin toggle that matters
The test site had three deliberately different classes of endpoint, and Nutch's behavior split cleanly along them:
- Class A — ordinary HTML links (4 pages, plus a 3-link-deep chain)
- Class B — endpoints that exist only as string literals inside a linked JavaScript file: one as a call argument,
fetch('/api/js-endpoint-7'), one as an assignment,const other = "/api/js-endpoint-8" - Class C — an endpoint that exists only after JavaScript runs and injects it into the DOM
Results, from server-side hit logs, repeated three times:
| Plugin configuration | Class A (HTML links) | Class B (JS-file literals) | Class C (runtime DOM) |
|---|---|---|---|
Shipped default — parse-(html|tika) | 4/4 (recall 1.0) | 0/2 (recall 0.0) | not reached |
With parse-js — parse-(html|tika|js) | 4/4 (recall 1.0) | 2/2 (recall 1.0) | not reached |
Identical across all three repeats. Deterministic.
The class-B jump is the part that gets underestimated. Nutch found both JavaScript-embedded endpoints without running a browser, using the parse-js plugin's regex-based scan of JavaScript content. The app.js file itself was fetched in both configurations — Nutch treats <script src> as an outlink regardless — so the entire difference is whether anything reads the file's contents looking for URL-shaped strings. Turn the plugin on, and it catches both literal forms.
On this fixture, Nutch's default and katana's standard mode reached the same class-A set, while Nutch with parse-js and katana with -jc reached classes A and B without a browser. The Katana version and full command are not captured in this article, so that result is context rather than a strict product benchmark.
Class C is the honest ceiling. No static plugin configuration reached it, which is expected: recovering an endpoint that only exists after script execution requires actually executing the script. I did try swapping protocol-http for protocol-htmlunit, Nutch's pure-Java JS-executing protocol. It loaded and ran without crashing, but in the same four-round harness it completed only one round, fetched just the seed page and app.js, reached none of A/B/C, and round two reported 0 records selected for fetching. That is an under-configured probe, not a verdict on HtmlUnit's capability. What it establishes is narrower: swapping in a JS-executing protocol is not a drop-in change, and class C stayed unreached in every configuration I tested.
Crawl control and failure behavior
Depth is not a flag. There is no --depth 3 in Nutch; depth is however many generate → fetch → parse → updatedb rounds you run, because round R fetches the frontier discovered in round R-1. My depth chain confirmed it precisely:
| Rounds run | Deepest path reached |
|---|---|
| 2 | /depth/1 |
| 3 | /depth/2 |
| 4 | /depth/3 |
Clean and mechanical, but it means depth is a loop count in your script, not a parameter.
Now the trap. Nutch's shipped default is db.ignore.external.links=false, paired with a permissive +. URL filter — which means a default Nutch crawl will follow links off your seed host. I seeded a page linking one in-scope path and one link to a different hostname, and the crawl fetched the foreign host. Two independent signals agreed: Nutch's own crawldb marked it db_fetched, and the other host's server counter registered the hit.
Staying in scope is opt-in, and both remedies verifiably work:
| Configuration | External host in crawldb | External host's server hit | Contained? |
|---|---|---|---|
db.ignore.external.links=false (shipped default) | db_fetched | +1 | No |
db.ignore.external.links=true | absent | 0 | Yes |
Host rule in regex-urlfilter.txt (+^http://127.0.0.1: then -.) | absent | 0 | Yes |
If you're crawling one site, set one of those before your first real run. One methodology caveat: this particular test is sensitive to load on the local server, so those three rows come from a run with nothing else touching the fixture. The behavior itself is mechanically clear and backed by two independent signals; the specific row values are one clean run, not an average of many.
Sitemaps are a separate step. Politeness is on — a normal crawl fetched /robots.txt — but the sitemap itself needs its own command:
| Approach | /sitemap.xml requested? | Endpoints that existed only in the sitemap |
|---|---|---|
| A normal crawl | never requested | 0/2 |
bin/nutch sitemap, run explicitly against the crawldb | fetched | 2/2 entries injected, full recall |
katana's inline -kf known-files mode, same fixture, on an IP host | not recorded | 0/2 |
Different model from crawlers that pull known files inline, and it costs you an extra command, but it does the job completely.
Two smaller behaviors held up well.
Error handling: a crawl over a page linking a 500 and a 404 completed all rounds cleanly, still fetched all four class-A pages, and recorded each failure distinctly:
| Failing response linked from the page | crawldb state recorded |
|---|---|
| 500 | db_unfetched (eligible for retry) |
| 404 | db_gone |
Nothing derailed.
Politeness: with one thread per queue, the gap between same-host fetches followed the setting:
fetcher.server.delay | Median gap between same-host fetches |
|---|---|
| 1.0 second | 1.009 s (minimum 1.006 s) |
| 0.0 | 0.002 s |
The knob does exactly what it says. The shipped default is 5.0 seconds, which is conservative and, again, probably correct for a tool designed to crawl strangers' servers.
The batch tax, in seconds
Every Nutch command is a fresh JVM. That single fact dominates the timing profile more than anything about fetching.
| Phase (per round) | Median seconds |
|---|---|
inject (once) | 1.81 |
generate | 3.93 |
fetch | 2.82 |
parse | 1.78 |
updatedb | 1.81 |
| One full round | 12.14 |
The effective per-job floor — JVM startup plus Hadoop initialization, measured as the cheapest trivial-work phase — is about 1.77 seconds. Multiply that by four commands per round, add the initial inject, and the whole-crawl picture looks like this:
| Tool | Depth-4 crawl of the 12-page fixture | Processes |
|---|---|---|
| Nutch | roughly 45 seconds (I measured 45.8 s and 45.0 s across two configurations) | around 17 JVM launches, essentially none of which are doing network work |
katana standard mode, same fixture | about 13 seconds | one process |
That gap isn't about fetch throughput; both tools request the same handful of pages. It's architectural. Nutch pays a fixed process cost per phase because those phases are designed as MapReduce jobs. On a tiny local crawl, setup dominates. The fixed cost should become a smaller share of a longer job, but this test did not measure the scale at which Nutch and katana cross over, or whether their ratio reverses.
Pros and cons
Pros
- Deterministic static discovery: 4/4 HTML class, 3/3 depth chain, identical across three repeated runs.
parse-jsrecovers JavaScript-file-literal endpoints (2/2) with no browser, catching both call-argument and assignment forms.- Two verified scope controls that fully contain a crawl (
db.ignore.external.linksand hostregex-urlfilter). - Sitemap ingestion via
bin/nutch sitemapachieved full 2/2 recall on endpoints a normal crawl missed entirely. - Robust under failure: 500 and 404 both handled with distinct crawldb states, crawl continues.
- In this local run, the observed same-host interval was consistent with the configured 1.0-second delay; the shipped default is 5.0 seconds.
- Apache-2.0, actively maintained, 78 plugins, and a persistent crawldb that tracks per-URL state across rounds.
- Runs in local mode with no cluster, no HDFS, and no Solr required.
Cons
- Will not run on JDK 24 or newer, where the SecurityManager removal bites (I measured the failure on 26.0.1) — the bundled Hadoop 3.4.2 predates the upstream fix and the escape flag is gone, so an LTS JDK pin is a hard prerequisite rather than a preference.
- ≈396 MB unpacked, 188 library jars, 78 plugin directories, 35 config files.
- Fresh JVM per command means ~1.77 s of fixed overhead per phase; ~45 s for a depth-4 crawl of 12 pages versus ~13 s for a single-binary crawler on identical ground.
- The shipped default follows links to external hosts; staying on one site is opt-in.
http.agent.nameships empty and the fetcher refuses to run until you set it.- No depth flag — depth is a loop count you manage yourself.
- Runtime-DOM endpoints were unreachable in every configuration tested, and swapping in the JS-executing protocol was not a drop-in change.
- I tested local mode on a single host with a small fixture. Distributed/HDFS mode, Solr indexing, hostdb, resume, and incremental re-crawl scheduling were outside this pass — treat them as untested here, not as endorsed.
Who should use it, and who should walk away
Nutch earns its keep when the crawl itself is the hard part. If you're building a search index, running a broad multi-domain crawl, need a persistent URL database with per-URL state and retry semantics, or expect to eventually distribute the work across machines, this is infrastructure that has been doing that specific job since before most alternatives existed. The plugin system means you can change protocol, parser, filter, and scoring behavior without forking anything. The politeness defaults are conservative in a way that suggests the maintainers have thought hard about being a good citizen.
Walk away if you want structured data out of a handful of pages. Nutch will fetch and parse them, then hand you a crawldb and segments and expect you to bring an indexer. Walk away if your targets are client-rendered single-page apps — class C stayed unreached in everything I ran. Walk away if your team doesn't run a JVM, because you'd be adding a Java toolchain, an LTS JDK pin, and 396 MB of jars to a stack that currently has none of that. And if the workload is "crawl one site, four levels deep, once a week," you'll spend more time on the round loop and config files than the crawl deserves.
For most people shopping for a scraper, that last case is the actual case. Which is not a criticism of Nutch — it's a mismatch between the tool and the errand. If you want a sense of the wider field, our rundown of open-source scrapers and the best web scraping GitHub projects cover the lighter end of the spectrum in more detail.
Alternatives, including where our own stack fits
The fair framing first: Nutch is free, Apache-licensed, self-hosted, and yours to run forever with no per-request cost. That's a genuine advantage, and nothing below erases it.
Related review: Browsertrix Crawler review.
Within the open-source world, the comparison depends on what you're optimizing. If you want a Python framework with crawl control and a request-first philosophy, Scrapy is a closer analogue for many projects; this article did not measure its install footprint on the same basis. If you want a compact Go crawler with no browser, Colly is another shape to evaluate. If your problem is turning pages into LLM-ready content rather than discovering URLs, Crawl4AI is aimed at a different layer.
A managed service such as Thunderbit moves fetching, rendering, and extraction behind an API, while Nutch keeps crawl state and infrastructure under your control. Thunderbit was not run on this fixture, so this is an ownership-model comparison rather than a claim about matched recall or dynamic-page performance.
The trade is ownership versus overhead, and it's not subtle. Nutch gives you total control, a persistent crawldb, cluster scalability by design, and zero marginal cost — in exchange for a JVM, an LTS JDK pin, 396 MB of jars, a round loop, and your own indexing layer. A managed API gives you structured output on the first call and no infrastructure — in exchange for per-call pricing and less control over the crawl frontier. If your job is "index 50 million pages," Nutch's model is correct and an API would be absurd. If your job is "get structured records off 200 product pages by Thursday," the reverse is true.
Try Thunderbit for Web Data Extraction
Verdict
Apache Nutch is worth evaluating if you're running an ongoing, multi-domain crawl and already operate JVM infrastructure. On this fixture its static discovery was deterministic across repeats, parse-js found both literal JavaScript endpoints, failures remained represented in the crawldb, and observed request spacing matched the configured delay.
Size the entry cost honestly. Nutch 1.22 failed here on JDK 26.0.1; OpenJDK 17.0.20 is the LTS configuration actually verified in this review, while Java 21 was not tested. Then set http.agent.name, decide your scope explicitly, and account for the observed ~1.77-second fixed floor per phase in this small local run. Whether that trade makes sense depends on the crawl's duration, breadth, and need for persistent state.
Try Thunderbit for Web Data Extraction Get Started Free
FAQs
Why does Apache Nutch fail with "getSubject is not supported"?
On JDK 24 or newer, JEP 486 made Subject.getSubject() throw unconditionally, while the bundled Hadoop 3.4.2 still called it. The first Hadoop job therefore dies before any page is fetched, and the old -Djava.security.manager=allow escape hatch no longer starts the VM. Use the verified Java 17 configuration and set NUTCH_JAVA_HOME; Java 21 may be supported, but this review did not run the full cycle on it.
Which Java version should I run Nutch 1.22 on?
Java 17 is the safest answer — Nutch's own CI targets it, and it worked cleanly in my testing on OpenJDK 17.0.20. Java 11 also remains supported for 1.22, though the project has announced that 1.23 will require Java 17. Anything from JDK 24 up will not run. A keg-only Homebrew install (brew install openjdk@17) plus NUTCH_JAVA_HOME keeps your system default JDK untouched.
Can Nutch crawl JavaScript-heavy sites?
Partially, and the distinction matters. With the parse-js plugin enabled, Nutch found both endpoints that existed only as string literals inside a linked JavaScript file — 2/2, with no browser involved. It found neither with the default plugin set. But an endpoint that only appears after JavaScript executes and modifies the DOM stayed unreachable in every static configuration I tested, and swapping in the HtmlUnit protocol was not a drop-in change in my run. For client-rendered apps, plan on a JS-executing protocol plus real configuration work, or a different tool.
Does Nutch need Hadoop and Solr installed?
No. Local mode runs Hadoop's in-process LocalJobRunner — no cluster, no HDFS daemon, no YARN — and the whole inject → generate → fetch → parse → updatedb cycle works on one machine with nothing else installed. Solr is the usual indexing destination, but the crawl itself doesn't require it. That said, Hadoop jars are bundled (13 of them, version 3.4.2), which is exactly why the JDK compatibility issue exists at all.
How do I stop Nutch from crawling other websites?
Set it explicitly, because the shipped default doesn't. Nutch 1.22 ships db.ignore.external.links=false with a permissive URL filter, and in my test the default crawl followed a link to a different host and fetched it. Either set db.ignore.external.links=true in nutch-site.xml, or add a host rule to conf/regex-urlfilter.txt (for example +^https://example\.com/ followed by -.). Both fully contained the crawl in testing, verified from Nutch's own crawldb and from the other server's request log.
