Heritrix Review: WARC Fidelity and the Operational Cost of Archival Crawling

Heritrix is the Internet Archive's open-source archival crawler — the software lineage behind the Wayback Machine, in production for two decades. Its job is to capture what a site served, faithfully, into WARC files that can be replayed years later. Under the hood it's a Java engine in which every crawl is a Spring bean graph written as XML, and the whole job lifecycle is driven over a REST API. It is not a scraper: no selector syntax, no field mapping, no rows at the end.

I tested 3.16.0 against a controlled local fixture and counted what it actually wrote, record by record. Completeness is what stands out: twenty fetched URIs produced sixty-one WARC records, with a payload digest and a capture IP on every response and every request linked back to the response it belongs to, none of which required touching a config knob. Operating it is the opposite experience — 41 MB, 114 jars (lib/ holds 142 files; the other 28 are bundled LICENSE and NOTICE texts), and a ~750-line job config before the first fetch — though every crawl here ran headlessly over curl, with no web UI clicks anywhere.

The storage result exposed a less obvious default. Two fixture URLs returned byte-for-byte identical content, and the stock profile fetched and archived both in full: two response records and zero revisit records, despite assigning both responses the same payload digest. Content-digest deduplication works after its history processors are added; it is not enabled by the stock profile, and it saves storage rather than bandwidth.

What Heritrix actually is

Archiving is not scraping, and that's the category confusion worth clearing up. Heritrix doesn't hand you rows. There is no CSV at the end. Its output is the HTTP conversation itself — headers and bodies and capture metadata — stored in a format designed for preservation, and it's the reference implementation for that whole category of work. Asking it for a product price list is like asking a court stenographer for a summary.

Current numbers as of July 27, 2026: the repo sits at 3,285 stars with 36 open issues, and 3.16.0 is the latest release, published 2026-07-03. That's exactly the build I tested, so nothing here is a stale-version complaint. Licensing has one wrinkle: the LICENSE file is plain Apache-2.0, but GitHub's own detector reports "Other" because some bundled third-party files carry their own terms. If Heritrix is going inside a commercial product, that's worth five minutes of someone's legal attention rather than a glance at the sidebar badge.

One structural fact shapes everything else: Heritrix does not drive a browser. It fetches over HTTP and extracts links from the bytes that came back. Its sibling in modern archiving, Browsertrix Crawler, does the opposite — real Chromium, records what the browser actually did. Both write WARC, but this review measured only Heritrix's non-browser path; it did not benchmark either system's scale or throughput.

Chains, beans, and SURT: how a crawl is actually assembled

Under the hood, a Heritrix job is a Spring application context. Not "configured with Spring" — it is a Spring bean graph, written as XML, and every part of the crawl is a bean you can swap.

The frontier holds the URI queue, partitioned by host. That partitioning is why politeness works the way it does, and it matters later.

Processor chains do the work in three stages: a candidate chain (should this discovered URI be scheduled?), a fetch chain (DNS, robots, HTTP fetch, link extraction), and a disposition chain (write to WARC, update state). Adding a capability to Heritrix usually means inserting a processor bean at the right point in the right chain, which is exactly how I turned dedup on.

Scope is a stack of DecideRules operating on SURT — Sort-friendly URI Reordering Transform, which rewrites http://www.example.com/a into http://(com,example,www,)/a so that hostname prefixes sort into hierarchies. Default scope is generated from your seeds' SURT prefixes. Rules accept and reject in sequence, last match wins.

The WARC writer sits in the disposition chain, and politeness lives in the frontier as three numbers: delayFactor, minDelayMs, maxDelayMs. Robots obedience is a policy string on the fetcher.

And all of it is drivable over a REST API, which turned out to matter more than I expected.

Setup is the heaviest part of the entire experience

What you deploy, before the first fetch:

Most of that job config you'll never touch. But you cannot skip it, and two fields are mandatory before the crawler will fetch anything: your seed, and metadata.operatorContactUrl. The stock value is a placeholder, and it will not crawl until you replace it with a real URL identifying whoever is running the crawl.

That requirement creates an accountability hook: an operator must supply a contact URL before the crawler runs. It does not prove that the identity is accurate, that a crawl is authorized, or that it complies with applicable rules, but it makes contact information part of the job rather than an optional convention.

Two setup findings genuinely surprised me.

It ran on a newer JDK than the minimum in the docs. The Getting Started docs ask for Java 17 or later. Heritrix 3.16.0 booted, served its REST API, and completed every crawl in this fixture on OpenJDK 26.0.1, without --add-opens, --enable-preview, or Security Manager workarounds. That is one macOS arm64 result, not a compatibility matrix, but it confirms that this tested build was not limited to JDK 17 on this host.

You never have to touch the web UI. The entire job lifecycle is REST, and I automated the whole thing with curl: create the job, PUT the beans file, build, launch, unpause, poll until the controller state reads FINISHED, terminate, teardown. That's the real answer to "is Heritrix operable in a pipeline" — yes, headlessly, no browser clicks anywhere. Most write-ups screenshot the Jetty UI and imply it's the interface. It's a convenience, not a requirement.

One host-specific deployment note: this Mac runs a system-wide HTTP proxy through Surge. Heritrix's Java client inherited that proxy and sent even 127.0.0.1 fixture traffic through it, producing 503 responses despite the OS exception list and NO_PROXY. Starting the JVM with -Djava.net.useSystemProxies=false changed the run from 2×503 to 18×200. This was an environment interaction, not a Heritrix defect; the flag is relevant only when inheriting system proxy settings is undesirable.

What actually lands in the archive

I ran the stock default profile against a controlled fixture — a local server with a known endpoint set, including HTML pages, a three-level depth chain, a robots.txt and sitemap, and deliberate 404 and 500 routes — and then parsed the resulting WARC record by record instead of trusting a summary line.

Twenty fetched URIs produced sixty-one records:

A clean 1:1:1 response/request/metadata ratio per URI, out of the box, with no configuration on my part. And the per-record completeness held up under inspection:

And HTTP status was preserved verbatim, including the ugly ones: 200 OK, 404 Not Found, and 500 Internal Server Error all appear as real status lines in stored responses rather than being dropped as failures.

That last point separates archiving from scraping more sharply than anything else. A scraper treats a 500 as an error to retry or skip. An archiver treats it as what the server said at that moment, which is a fact worth preserving. The Heritrix Output wiki describes this record structure; what I hadn't seen anywhere was the measured multiplicity and the 20/20 linkage on a known endpoint set. It holds.

The dedup result, measured both ways

My fixture served /dup/one and /dup/two with byte-identical bodies. Different URLs, same content — the exact case content-digest dedup exists to collapse. I ran it twice: once with the stock profile, once after inserting the digest-history chain (BdbContentDigestHistory, plus a ContentDigestHistoryLoader in the fetch chain and a ContentDigestHistoryStorer after the WARC writer).

Out of the box, both responses received the same digest, but no history processors acted on it and both payloads were written in full. Adding the chain converted the second capture into a WARC revisit record pointing at the identical payload digest, the behavior the WARC 1.1 specification defines revisits for.

None of this is a secret. The Duplication Reduction Processors wiki page states that skipIdenticalDigests defaults to false and that URL-agnostic dedup needs those loader and storer beans. This isn't hidden behavior uncovered; like nearly everything measured here, it's documented Heritrix behavior with a first-hand number attached. The gap is between the documentation and what people believe, and in my experience the belief is usually "Heritrix dedups," full stop, with no asterisk about configuration.

Two corollaries worth internalizing:

Dedup is write-time, not bandwidth. This one is mechanism rather than something I metered separately, but it follows directly from how content digests work: you can only compare a digest after the bytes have arrived, so the second URL gets fetched from the origin either way. Enabling the chain reduces what you store, not what you transfer or what the target server has to serve. Anyone budgeting dedup as a politeness or bandwidth win has it backwards.

Storage estimates built on "it'll dedup" can be badly wrong. If you're archiving a site with heavy template duplication — mirrored PDFs, boilerplate landing pages, print-view variants of the same articles — and size disks assuming identical bodies collapse, the stock profile may consume substantially more storage than that estimate. The two-URL fixture establishes the default behavior, not its effect at million-URI scale; that effect depends on duplicate rate, payload size, and recrawl design.

Scope and robots did exactly what they promise

A crawler's own word for what it didn't fetch is worth very little, so both of these were measured with a server-side hit counter — the target server counting requests itself, independent of anything Heritrix logged.

Scope. Meanwhile the in-scope host was crawled normally, so scope was disciplined rather than the crawl being broken. Worth a caveat: I ran only the default-scope arm here, not a widened-scope positive control, so read this as confirmation of documented design rather than a two-sided proof.

Robots. The link was always reachable; only the robots policy suppressed it. The obedience is real and the escape hatch is real, which is the correct arrangement — some archiving mandates legitimately override robots, and that should require deliberately typing ignore into a config file.

Politeness: 57.7 seconds to crawl twenty local pages

One number decides whether Heritrix fits your project.

Same fixture, same three-run treatment, on a single local host with sub-millisecond latency:

The realized delay sits right on the minDelayMs floor: on a sub-millisecond origin, delayFactor × fetch-time is negligible and the minimum dominates by construction. The ratio between the two rows is not useful because the zero-politeness denominator is only tens of milliseconds and jitters run to run. The stable result is the absolute floor: default Heritrix waited about three seconds between requests to the same host in this fixture, turning a twenty-page crawl into roughly a minute of wall time.

A concrete way to feel it. Say a university library needs to archive a 50,000-page government site before it's decommissioned, and it's all on one host. At a 3-second per-host floor, that's 150,000 seconds of enforced waiting — roughly 42 hours, or about a day and three-quarters, before you count fetch time. That's arithmetic from my measured floor of 3,036 ms — 50,000 x 3.036 s = 151,800 s = 42.2 h; even at the configured 3,000 ms minimum it is 41.7 h, which rounds to 42, not 41. Not a measured crawl, but it's the arithmetic your project plan needs.

In fairness, Heritrix's politeness is per host, because the frontier queues by host. A broad crawl across thousands of domains parallelizes across those queues and doesn't inherit this ceiling globally. My fixture was a single host, which is the worst case for this particular number. If your archiving target is one big site, that worst case is your case.

And it's a feature. The delay is what makes an archival crawler something a site owner tolerates instead of blocks. Tuning it down is a decision about someone else's server, and the tool makes that decision explicit rather than defaulting to aggressive.

What I didn't test

These measurements cover crawl discipline on a controlled fixture, and nothing wider. Outside their boundary:

• Cross-crawl dedup and recrawl. I measured intra-crawl content-digest dedup only. Persisting a URI history database across separate crawls (FetchHistoryProcessor + PersistLog) is a different mechanism and I didn't exercise it.
• Scale and long-run stability. No million-URI frontier, no checkpoint-and-restore, no multi-day run. My fixture measures discipline, not endurance.
• JavaScript-rendered capture. Heritrix's default capture is non-browser and that's what I measured. The optional browser-based behaviors are untested here.
• Politeness on a high-latency origin. Local latency is sub-millisecond, so minDelayMs dominated by construction. How delayFactor scales against a slow real-world server is not isolated in my data.
• Sitemap recall. robots.txt was requested and the sitemap directive was followed, but I didn't separately assert recall of every <loc> entry.

What to make explicit before the first production job

The stock profile is long, but the decisions that change the meaning of an archive are fairly compact. Start with scope. Seeds generate default SURT prefixes, and DecideRules can widen or narrow them in sequence. Review the final rule order with representative in-scope and out-of-scope URLs, then verify it against server-side traffic or another independent request log. A crawl report alone cannot prove that an excluded host was never contacted.

Next, decide what robots policy and operator identity mean for the collection. The tested default obeyed the fixture's disallow rule, while changing robotsPolicyName to ignore caused the blocked path to be fetched. That switch is mechanically simple and institutionally significant. Record who approved it and why, alongside a working operatorContactUrl; the required URL gives a site owner a route back to the operator but does not supply the authorization rationale.

Storage planning needs its own explicit choice. If identical bodies should become revisit records, add and review the content-digest history processors before sizing the archive. The tested chain affected representation after fetching, so origin traffic should still be budgeted for both URLs. Cross-crawl deduplication is a separate mechanism and should not be inferred from this two-URL, single-crawl result. A small validation crawl with known duplicate bodies is a cheap way to confirm that the deployed bean graph produces the intended record types.

Finally, treat politeness as a scheduling input rather than a last-minute tuning knob. On the local single-host fixture, minDelayMs dominated total time. A real project should calculate the configured per-host floor against the number of target hosts and collection deadline, then test on representative latency. Broad and single-site crawls stress the host-partitioned frontier differently; this review measured only the latter. Keep the REST lifecycle in the runbook as well: build, launch, unpause, poll, terminate, and teardown are separate states worth observing in automation.

Pros and cons

Pros

• Archive completeness is excellent by default: 20/20 responses with payload digest, capture IP, and full request↔response linkage, no configuration required.
• Preserves error responses as facts — 200, 404, and 500 status lines all stored verbatim.
• Scope discipline verified against a server-side counter: zero out-of-scope fetches while the in-scope host crawled normally.
• Robots obedience genuinely suppresses the fetch, with a deliberate ignore escape hatch for mandated archiving.
• Fully headless via REST — create, build, launch, poll, teardown, all over curl, no UI clicks.
• Runs clean on OpenJDK 26.0.1 with no JVM flags, which is better modern-Java hygiene than most twenty-year-old codebases manage.
• Mandatory operator contact URL means the crawler can't run anonymously.
• Every part of the crawl is a swappable bean, which is why enabling dedup was three bean insertions rather than a fork.

Cons

• Content-digest dedup is off by default and writes identical payloads in full — a real storage-planning trap.
• Deployment is heavy: 41 MB distribution, 114 jars, a Java engine plus Jetty, and a ~750-line Spring job config.
• Default politeness imposes about a 3-second per-host floor; a 20-URI single-host crawl took 57.7 seconds.
• No JavaScript rendering in the default path, so client-side-only content won't be captured.
• The configuration surface rewards expertise and punishes casual use; there's no five-minute path to a first crawl.
• Nothing in the output is structured data. Getting fields out of a WARC is a separate project.

Who should run it, and who should walk away

Heritrix is for institutions and teams whose deliverable is the archive itself. Libraries, national archives, legal and compliance preservation, research groups capturing the web as a primary source, anyone who needs to prove in five years what a URL served on a particular day. If the words "WARC," "replay," and "provenance" are already in your vocabulary, this is the tool the rest of your ecosystem was built around, and its weight is the price of that interoperability.

Its host-partitioned frontier and long use in web archiving make it a plausible candidate for broad crawls across many domains. That is an architectural and project-history consideration, not a scale result from this fixture; long-run throughput, checkpoint recovery, and million-URI behavior remain untested here.

Skip it if you want data rather than an archive. If your goal is a spreadsheet of products, listings, or contacts, Heritrix will do an immaculate job of capturing pages you then have to write a separate pipeline to parse — and you'll have paid a Java engine, a Spring config, and a 3-second politeness floor for the privilege. Skip it too if your targets are client-rendered single-page apps, where a non-browser fetcher captures the shell and not the content; that's where a browser-based archiver is the correct instrument. And skip it if you need a first result today, because the setup curve is real.

Alternatives, and where a managed API fits

Within archiving, the direct modern counterpart is Browsertrix Crawler — a browser-based archiver that drives Chromium and records what the browser did. It can capture JavaScript-produced content absent from Heritrix's default HTTP responses, while adding browser deployment and runtime overhead. This review did not run a head-to-head benchmark, so the decision starts with capture requirements: browser-generated state points toward a browser archiver; conventional HTTP resources remain Heritrix's native path.

Related review: Browsertrix Crawler review.

For a different shape of problem entirely — you don't want an archive, you want structured data out of pages — compare Heritrix with extraction systems by output rather than treating them as equivalent crawlers. Heritrix is free and self-hosted: you run the JVM, own the beans file, size the disks, and tune politeness. That model fits when preservation is the deliverable.

Disclosure: Thunderbit is the publisher's product and was not tested in this Heritrix fixture. It belongs to the managed extraction category: its output is page content or structured records rather than preservation-grade WARC files. Choose an archiver when replayable capture and provenance are required; consider an extraction service when the deliverable is rows or document text and managed operation is acceptable.

Archiving carries its own permissions question, and it's not the same as scraping's. Heritrix obeys robots.txt by default and requires you to identify yourself before it fetches a byte, which is a good baseline — but a robots-compliant crawl is not automatically an authorized one. Copyright, terms of service, personal data, and your own institution's mandate all sit on top of it, and the ignore policy exists for organizations with a legal basis to use it, not as a convenience toggle. If you're standing up an archiving program, sort the authorization scope out before the disks fill, and read up on the legal side of web scraping and archiving if it's new territory.

Try Thunderbit for Web Data Extraction

Verdict

Should you use Heritrix? Yes — if your output is an archive and you have someone willing to learn Spring beans.

The fixture supports a clear decision: the stock profile preserved response, request, and capture metadata consistently; scope and robots controls affected server-side fetches as configured; and the full job lifecycle ran over REST. Those are useful properties for an archival pipeline, within the small single-host boundary tested here.

The operational costs are equally clear: a Java distribution and large Spring configuration, a configured per-host delay that dominated this local crawl, and content-digest deduplication that requires additional history processors. Teams that need WARC fidelity may accept those costs; teams that need extracted fields should start in another category.

Before a production crawl, verify the dedup chain, politeness settings, scope, robots policy, operator identity, and storage assumptions against the actual job configuration. The stock profile is a starting point, not an implicit statement of those operational choices.

Try Thunderbit for Web Data Extraction Get Started Free

FAQs

Does operatorContactUrl make a crawl authorized? No. It forces the job to include a contact URL, which gives site operators an accountability path, but it does not establish permission, identity accuracy, copyright status, or regulatory compliance. Those remain deployment decisions outside the crawler.

Does content-digest deduplication reduce requests to the origin? Not in the configuration tested here. Both URLs were fetched before their payload digests could be compared. The history chain changed how the second payload was represented in WARC storage; it did not turn the second URL into a skipped network request.

Can Heritrix run without its web UI? Yes. The tested lifecycle — create, upload configuration, build, launch, unpause, poll, terminate, and teardown — ran through the REST API with curl. The embedded Jetty UI was not required.

Does the stock Heritrix profile deduplicate identical payloads? Not in the fixture as configured. The stock profile stored both identical responses as full response records. Adding the content-digest history chain changed the second capture to a revisit record, so deduplication is a pipeline choice you must configure and verify rather than an automatic default.

How should I choose a politeness delay? Treat the local timings here as a mechanism check, not a production recommendation. Set the delay from the target site's rules, operator agreement, server capacity, crawl purpose, and your own retry/concurrency policy, then confirm the actual same-host request gaps in logs.