The Extractor That Leaked the Most Boilerplate Missed No Content in This Fixture Set

Six libraries, one annotated fixture set, one scorer. In these 22 synthetic fixtures, the library with the highest boilerplate leak — Mozilla's Readability, at 23.5% — was also the only one that recovered every labelled article unit.

That is the trade-off in one sentence, and most write-ups on this subject never surface it, because most of them score precision and stop.

What was actually measured

Every fixture in this set carries per-unit ground truth. Each block of the page — the article paragraphs, the nav, the ad, the sidebar, the comment thread, the promo — is labelled article or boilerplate and tagged with a unique sentinel token. So "did the extractor recover this unit" is exact substring membership, not a similarity score. A sentinel either survives in the output or it doesn't.

Twenty-two fixtures, 91 units. Six extractors: Mozilla Readability 0.6.0 (via jsdom 30.0.1), trafilatura 2.2.0, resiliparse 1.0.9, newspaper4k 0.9.6, goose3 3.1.22, and jusText 3.0.2. Python 3.14.2 and Node 22 ran on the same machine. The article does not preserve the OS/CPU, exact invocations, repetition count, or warm-up policy, so the timing column is a local observation rather than a portable benchmark.

Two rules I enforced on myself before anything ran. Each Python library installed into its own empty virtualenv, so its footprint is its own and not inherited from whatever a sibling pulled in. And no runner computes a metric — every one dumps raw extracted text, and a single scorer produces every number, so six tools are compared on identical arithmetic rather than six similar-looking definitions of "precision".

The headline table

Recall is aggregated over all 22 fixtures. Leak rate, aggregate content-token precision and contamination use the 11 fixtures containing both article and boilerplate units; “answered” shows how many of those produced output. Full per-fixture numbers are in sixway-scores.json.

One row in that table is not a default. resiliparse's extract_plain_text takes main_content=False as its default, and I called it with main_content=True. The difference is not small: at defaults it leaks 17 of 17 boilerplate units across the set — every nav, ad, sidebar, comment thread and promo — against 1 of 17 with the flag on. Every other library above was called at its defaults. So resiliparse's 0.0588 leak rate is what it does when you ask for main content, and extract_plain_text(html) on its own is a different product (default-vs-main-content.json).

Read the first column and the second column together, because reading either alone gets you the wrong tool.

Readability never misses. Perfect recall on all 22 fixtures, and it is alone in that. It pays for it: 4 of 17 boilerplate units leaked, 35 contaminating tokens, four times trafilatura's leak rate. Three of its four leaks are the same shape — a neutral-classed promo block sitting as a sibling of the article, which its sibling-append heuristic swallows. If you feed its output to a model, you are paying for those tokens and the model is reading them as article.

newspaper4k is the balanced one. Zero leak, zero contaminating tokens, 0.9865 recall, and output on all 22 fixtures. If I had to pick one without knowing the workload, this is the one, and it is not the one most people reach for.

goose3 has perfect precision and the worst recall in the test. Every content word it returned was article content. It also recovered nothing at all on two fixtures and produced no output on those same two. Perfect precision is cheap if you are allowed to decline.

The precision number that flattered two libraries

That last point is worth making concrete, because it is a trap I nearly published.

Precision and F1 here are conditional on producing output. A library that returns an empty string on a fixture contributes nothing to the numerator and nothing to the denominator — so declining is free, and a conservative extractor's precision looks better than a thorough one's for no reason other than silence.

goose3's aggregate precision was 1.0000 across the 10 scored fixtures where it returned output. jusText's was 0.8760 across 10 of 11. Readability, trafilatura, resiliparse and newspaper4k answered 11 of 11. The table now shows that denominator beside precision so abstention cannot disappear behind a flattering ratio.

There was a worse version of this. My first scorer averaged article recall over the same 11-fixture content-fidelity set — the set that excludes fixtures with no boilerplate, which is correct for measuring leak. It reported resiliparse at 1.0000 recall. Over all 22 fixtures resiliparse is at 0.9054, because on the fixture whose article lives entirely inside <li> elements with no <p> anywhere, it returns output and recovers 0 of 6 article units. That fixture has no boilerplate, so it fell outside the average, and a real failure hid behind a perfect score.

Where each one actually breaks

Every one of these is a specific, reproducible behaviour rather than a general "worse at extraction":

• resiliparse and goose3 both assume paragraphs. Point either at a page whose body is a list — a changelog, a spec, a FAQ, a recipe — and resiliparse hands back text with none of the list content in it, while goose3 hands back nothing. resiliparse is the more dangerous of the two here, because returning something looks like success.
• jusText has a length cliff, and it is sharp. More on that below.
• The near-empty document is the one case where returning nothing is arguably correct, so I would not hold that against either library.

jusText: a cliff, not a slope

jusText produced output on 19 of 22 fixtures and leaked 47% of boilerplate — the highest in the test, which is the opposite of its reputation. But the interesting number is the one that made me re-run everything.

jusText classifies each block by stopword density against a language stoplist, then runs a context-sensitive pass that promotes a neargood block to good only when it sits next to an existing good block. A block reaches good on its own only above length_high, which defaults to 200 characters. On a document where nothing crosses that line, nothing seeds the promotion and the whole page decays to boilerplate.

I swept it on a document whose longest paragraph is 151 characters:

Zero to 832 characters when exactly one paragraph crosses the threshold, and then nothing changes no matter how much further you loosen it. One paragraph over the line unlocks the entire document.

Before concluding that, I swept length_low across four values and max_link_density across two — eight combinations, all returning zero. This project's rule is that a negative capability claim needs at least three parameter shapes probed or the vendor's own error naming the field, and one unproductive parameter is not a finding about the library. The numbers are in justext-length-threshold.json.

None of this says jusText extracts badly. On a real natural-language page at defaults it returned 1,190 characters of clean article text. It says jusText has a documented knob that behaves like a switch, and that the default position of that switch is wrong for short-paragraph documents.

What you install, and what it costs to import

Same fixtures, same machine, each library in its own empty virtualenv.

In this run, resiliparse had the lowest cold-import and median extraction values: 15 ms and 0.06 ms. Exact cross-runtime ratios would overstate what the incomplete protocol can support, especially because its slowest single extraction was 1,098 ms. Separate startup, first-call, and steady-state distributions are needed before using these numbers for serverless sizing.

trafilatura and resiliparse are a dead heat on quality — 0.9697 against 0.9681 content-token F1, identical 0.0588 leak rate — and I am not calling a winner on a difference that small. On footprint they are not close: 21.0 MiB against 69.9 MiB, 5 packages against 17. The trade you are actually making is resiliparse's list-blindness against trafilatura's three extra prerequisites.

Two bugs in my own testbed, found before publishing

The comparison above nearly did not happen, and the reason is worth more than any single row in it.

The fixture set could not see two of the six libraries. The original fixtures write every unit as a run of unique nonsense tokens — zzart01vf64 zzart01v56i — which is exactly what makes recall exact. It also means the fixtures contain no English function words at all. Readability, trafilatura and resiliparse decide structurally, from the DOM, so they were unaffected. goose3 and jusText decide lexically, by counting stopwords, and there were none to count: both returned an empty string on all 22 fixtures.

A table with two libraries scored at zero would have looked authoritative and meant nothing. I checked before writing it, on a real page: goose3 returned 1,017 characters and jusText 1,190. The libraries were fine. The testbed could not represent them.

So the fixtures were rebuilt with English prose carrying the sentinels — same structure, same classes, same DOM positions, same unit boundaries, same sentinels, 1,568 tokens swapped one for one. goose3 went from 0 to 20 of 22.

Then the rebuild broke two things of its own, and both were mine. An English word is about six characters; zzart01vf64 is about twelve. Replacing one for one halved every unit — 21,646 characters of unit text became 10,986, and the longest unit fell from 1,513 to 622. That silently rewrote the fixtures whose entire purpose is length. jusText, whose behaviour is a length cliff, went from 19 of 22 to 6 of 22 on that alone. Had I published the halved version, jusText's number would have been wrong by a factor of three, in the direction that makes it look like a worse library.

The second one: drawing every unit from one shared corpus restored the stopword density but destroyed the property the token-level scoring depends on. Article and boilerplate vocabularies must be disjoint, or "extracted tokens that are boilerplate tokens" counts the word the. Ten of 22 fixtures ended up with overlapping vocabularies, against zero in the originals. The fix was to suffix content words per unit and leave function words bare — real stopwords for the lexical libraries to count, disjoint content vocabulary for the scorer.

That is also why the token-level columns here are named content_token_* and not reused from the published Readability-versus-trafilatura numbers. They are a different quantity, measured over content words only, and quoting one as the other would be wrong.

While rebuilding, one more thing turned up that was not mine: three link-density fixtures place </a> inside a word — <a href="/x">zzsibp015qlhf zzsi</a>bp015qbht — because the anchor was positioned by character offset to hit an exact ratio. The rendered text is unchanged, so the original scoring never noticed, but any extractor working per element rather than per text run sees two fragments where the others see one word. Repaired, with the linked-character delta recorded rather than quietly absorbed.

Who should use what

Feeding a model and paying per token? newspaper4k or goose3. Both leaked zero boilerplate units and zero contaminating tokens. newspaper4k if you want an answer on every page; goose3 if you would rather have silence than a guess, and your pages have paragraphs.

Optimizing a latency-sensitive Python path? Include resiliparse in the bake-off. It had the lowest observed import and median extraction here and tied trafilatura closely on quality — but only with main_content=True, which is not the default. Check list-heavy layouts first, and do not turn these local timings into an exact cross-runtime speed ratio.

Archiving, or anything where missing content is worse than extra content? Readability. It is the only one that recovered every article unit on every fixture, and 35 stray tokens is a cheap price if the alternative is losing a paragraph.

Multilingual work? jusText is a candidate to include because it ships language stoplists. This study did not test multilingual extraction, so that feature is a reason to evaluate it, not evidence that it wins. Test length_high against representative paragraph lengths.

Anything that isn't an article? None of these. They are all built on the assumption that a page has one main body of prose, and a product listing, a search results page or a dashboard breaks that assumption in ways no parameter fixes.

Where a managed API fits

Everything above is a library you run: you provide HTML and receive text. Their observed failure modes vary by page shape, so validate the chosen defaults against your corpus. Structured field extraction and fetching/rendering are outside this comparison.

Author note: Thunderbit is our managed service for URL-in workflows and structured output. It was not run through these fixtures, so no quality comparison is implied. The relevant decision boundary is whether you already hold the HTML and want a local text extractor, or want fetching/rendering and operations handled by a service.

The honest framing: if you already have the HTML and you want text, one of these six is free and good, and this table tells you which. If you are fetching pages at scale, or you want rows rather than prose, that is a different purchase.

If you are choosing between hosted fetchers instead, our web scraping API roundup covers that field and our cost comparison of SEO and data APIs covers what they charge. For the self-hosted side, the open-source scraper pillar is the wider view, and if what you actually need is Markdown rather than plain text, converting HTML to Markdown in Python is where most of the loss happens.

Try Thunderbit for Web Data Extraction

Verdict

There is no winner, and a table that named one would be lying about a real trade-off.

Build a small acceptance corpus before choosing: include list-only articles, short paragraphs, promo siblings, a near-empty page, and examples where returning nothing is preferable to contamination. Score article recovery, boilerplate leakage, and abstention separately. On these fixtures, Readability favored recall, newspaper4k produced the most balanced row, and resiliparse was a latency candidate with a list-content blind spot; those labels should not travel beyond the tested shapes without validation.

What I would actually tell you is narrower than any of that: run the fixtures against your own page shapes before you pick. Two of the six could not see the testbed I started with, and one of them scored a perfect recall that hid a total failure. A comparison table is a starting point for that, not a substitute.

Try Thunderbit for Web Data Extraction Get Started Free

FAQs

Are these numbers comparable with the published benchmarks for these libraries? No, and I would not quote them that way. These are controlled fixtures with synthetic-but-labelled units, so all six saw identical bytes and the comparison between them is fair. Published figures like the scrapinghub article-extraction benchmark use real-world corpora, which measure a different and harder thing. Use this table to compare these six against each other, not against a number from a paper.

Why is Readability's leak rate so much higher than trafilatura's if they are both DOM-based? Because of where each one draws the boundary. Three of Readability's four leaks are neutral-classed promo blocks sitting as siblings of the article, which its sibling-append heuristic pulls in on the theory that adjacent long low-link content is probably part of the story. Often it is. On these fixtures it is a promo. trafilatura is stricter about what it appends and leaked one of the same units.

Should I trust the precision numbers for goose3 and jusText? Only alongside the sample count. Both were scored on 10 of the 11 fixtures that have both article and boilerplate units, because they returned nothing on one of them, and a fixture with no output contributes to neither side of the ratio. goose3's 1.0000 precision is real for the pages where it answered; its 0.8243 recall across all 22 fixtures is the other half of the same fact.

Does jusText's length threshold matter on real pages? It depends entirely on your paragraph lengths. A news article with 300-character paragraphs will cross length_high on the first one and behave normally — that is why jusText returned 1,190 clean characters on a real page at defaults. A page of short paragraphs, list items, or product blurbs may never cross it, and then jusText returns an empty string rather than a partial answer. Set it explicitly instead of finding out in production.

What was not tested here? Real-world pages, at all. Multilingual extraction, despite jusText's stoplists being its main selling point. Memory under load. Any page that is not an article — no product listings, no search results, no dashboards. Encoding edge cases. And the two Node and Python ecosystems were compared on library behaviour, not on runtime performance, so the millisecond figures across that boundary should be read as orders of magnitude rather than precise ratios.