jusText's Threshold Cliff on a Short-Paragraph Fixture

On a document whose longest paragraph is 151 characters, jusText at its defaults returns zero characters. Not a partial extraction — an empty string. Lower one threshold so that exactly one paragraph crosses it, and the same document yields 832 characters. Lower it further and you still get 832.

On this fixture, the behavior is a cliff rather than a slope. Whether the default is on the wrong side depends on the paragraph lengths and boilerplate distribution in the target corpus.

What jusText is, and why lexical density matters

Compared with the other extractors in this fixture set, jusText depends unusually heavily on language-specific stopword density. A block with a high proportion of function words — the, and, of, was — is more likely to be prose. That lexical signal is not the whole classifier: paragraph length, link density, HTML-derived block boundaries, heading distance, neighboring classes, and a context-sensitive pass also affect the result.

Official reference: jusText's official repository.

Because the classifier takes a language stoplist, jusText ships 100 of them. That explicit language-specific lexical surface is its clearest differentiator in this comparison, although multilingual quality was not tested.

Version tested: 3.0.2, BSD 2-Clause, 822 GitHub stars. Python 3.14.2.

The cliff, measured

jusText's classifier runs in two passes. A context-free pass assigns each paragraph good, bad, short or neargood. Then a context-sensitive pass promotes neargood to good — but only when it sits next to a block that is already good. And a paragraph earns good on its own only when it exceeds length_high, which defaults to 200 characters.

On a document where nothing crosses 200, nothing seeds the promotion, and every neargood block decays to boilerplate. The whole page comes back empty.

I swept the threshold on a fixture whose longest paragraph is 151 characters:

justext-length-threshold.json.

On this threshold fixture, one paragraph crossing the line changed all eight target paragraphs to output, and loosening further added nothing. The trace is consistent with the context pass promoting eligible neargood neighbors around an existing good block; it does not imply unconditional promotion of every neighbor on arbitrary pages.

Before attributing the switch to length_high, I swept length_low across four values crossed with max_link_density across two: eight combinations, all returning zero. Changing either of those settings did not recover output on this fixture.

Across the whole 22-fixture set the pattern holds: 2 of 22 fixtures produced output at length_high=200, 9 of 22 at 150, 15 of 22 at 120.

Two things this does not say. It does not say jusText extracts badly — on a real natural-language page at defaults it returned 1,190 characters of clean article text, because a real news paragraph clears 200 characters on the first try. And it does not say the default is wrong; it says the default assumes long paragraphs, and you should decide whether your corpus has them before you run anything.

The other side of this fixture set: higher leakage at defaults

On this labeled fixture set at defaults, jusText has the highest measured boilerplate leak.

sixway-scores.json. One fixture set, one scorer, every unit tagged with a unique sentinel so recovery is exact substring membership.

47% boilerplate leak and 74 contaminating tokens — double Readability's leak rate and more than twice its contamination on this labeled synthetic set. This is a diagnostic result at defaults, not a stable product-wide ranking.

The observed leak is consistent with the same context pass involved in the cliff. Eligible neargood blocks can be promoted when the surrounding block classes and distance rules permit it; a prose-like promo or comment beside article text can therefore cross the boundary. The fixture result demonstrates which labeled blocks leaked, while the simplified mechanism remains conditional on jusText's exact context rules.

Its recall is 0.8378, third from the bottom, and all of the loss is the threshold: it recovered nothing on a single 129-character article, nothing on a page of ten short paragraphs, and nothing on the near-empty document.

The language stoplist inventory

Ninety-nine other stoplists.

justext.get_stoplists() returns 100 languages. The classifier is language-parameterised by design, not by translation of an English heuristic, and swapping languages is one argument:

import justext
paragraphs = justext.justext(html, justext.get_stoplist("Czech"))
text = "\n".join(p.text for p in paragraphs if not p.is_boilerplate)

trafilatura and goose3 also expose language-related behavior, but this review did not score any extractor on non-English ground truth. jusText's 100 bundled stoplists make it a clear candidate for multilingual evaluation; the inventory alone does not establish extraction quality across those languages or prove that competitors cover them less effectively.

The API is two functions and nine tunable constants — length_low 70, length_high 200, stopwords_low 0.30, stopwords_high 0.32, max_link_density 0.20, max_heading_distance 200, plus encoding handling. Small, legible, and all of it documented in the signature.

Install and speed

pip install justext pulls 3 packages — the fewest in the comparison — and 22.4 MiB, in under two seconds.

Official reference: jusText on PyPI.

install-and-import.json. Each library in its own empty virtualenv.

Three packages and 22.4 MiB is a small dependency in this comparison, and 0.56 ms median extraction is close to trafilatura's 0.51 in this harness. The environment measurement does not break package size down by file type, so it cannot say how much of the disk footprint comes from stoplists.

The maintenance question, answered carefully

The dated maintenance measures used here are the last repository push and the last PyPI release, both 2025-02-25. That was seventeen months before testing. Eight releases total, 91 forks, 9 open issues, not archived.

The PyPI classifiers list Python support up to 3.9. I ran it on 3.14.2 and it installed, imported in 0.777 s and extracted on 19 of 22 fixtures without a single exception.

So the metadata is five Python versions behind the reality, and the reality is that it works. That is the useful distinction: a quiet repository is a signal about support, not automatically about function. For a library whose entire algorithm is a published 2011 method plus a set of word lists, "finished" is a plausible state — there is not much left to change, and the stoplists do not rot the way an anti-bot workaround does.

What a quiet repo does mean: if you hit a bug, you are fixing it yourself or forking. Weigh that against nine open issues, which is not the profile of a library drowning in unaddressed problems.

Memory, and what broken HTML does to it

Two operational questions are measured separately here.

The broader stress-test context is in the ten-library memory and malformed-HTML comparison.

Peak resident memory, via /usr/bin/time -l, one fresh process per cell — the import floor is what the library costs loaded and idle, the peaks include the document.

memory-results.json. Python and Node baselines are not comparable to each other; the interpreter is inside both.

jusText's floor is 30.3 MiB and its peak on this 10 MB fixture is 431.2 MiB, about 14.2 times the import floor and 43.1 times the input size in absolute RSS. One fixture and one process do not establish a general scaling curve; Python and Node baselines also remain non-comparable.

Broken HTML. Twelve documents each breaking exactly one thing — unclosed tags, mis-nested inline elements, unquoted attributes with spaces, stray closers, no <html> at all, duplicate attributes, a document truncated mid-tag, bad entities, an unclosed <script>, a lying charset declaration, a comment containing markup, and 600 levels of nesting — plus two well-formed controls at matched sizes, because "it returned nothing" only says something about malformedness if the library is not also silent on a clean document of the same size.

justext raised on 0 of 14 and returned nothing on 13, recovering 0/33 scored sentinels across the broken fixtures (malformed-results.json). The well-formed short control also returned 0 characters; the well-formed long control was the sole non-empty result at 1,333 characters. The twelve malformed fixtures all stayed empty, and the unclosed-<script> case is excluded from sentinel survival scoring. This batch therefore shows parser tolerance — no exception — but cannot attribute silence to malformedness rather than the already-measured size threshold.

Pros and cons

In its favour. 100 language stoplists, and a classifier actually built around them rather than translated into them. The smallest dependency count in the comparison at 3 packages. 0.56 ms median extraction. Nine documented, legible tuning constants. BSD 2-Clause. Runs cleanly on Python 3.14 despite classifiers that stop at 3.9.

Against it. The highest boilerplate leak in this synthetic test at defaults: 47%, with 74 contaminating tokens. The short-paragraph fixture returned an empty string when no paragraph crossed length_high. Recall was 0.8378 on this set. The last recorded repository push and release were seventeen months before testing, so maintenance ownership needs consideration.

Who should use it, and who should not

Evaluate jusText for multilingual corpora because the language-specific stoplist surface is explicit and broad. This test inventoried 100 stoplists but did not measure multilingual quality. It is also a candidate when a three-package dependency and explicit threshold controls suit the deployment.

Skip it if you are feeding a model per token, where 74 contaminating tokens against goose3's and newspaper4k's zero is a direct cost. Skip it for short-paragraph pages — product blurbs, listings, changelogs, FAQ entries — unless you have set length_high deliberately. And skip it if you need an actively maintained dependency for a compliance or procurement reason, which is a real constraint even when the code works.

If you do use it, tune length_high against a labeled validation sample rather than copying the default or a percentile rule. Sweep plausible thresholds and measure both article recall and boilerplate precision; lowering the gate can recover short prose while also promoting unwanted neighboring blocks.

Where a managed API fits

jusText takes HTML you already have, like everything in this comparison. None of these libraries fetches a page, renders JavaScript, or handles an anti-bot layer.

For the same fixtures across all six extractors, see the six-library extraction comparison.

A hosted fetch/render/extraction service, including our own Thunderbit, covers a different responsibility boundary. Thunderbit was not benchmarked here. The distinction is supplied-HTML prose classification versus a service that acquires and processes a URL; this article provides no same-metric quality comparison.

The honest framing: jusText's multilingual stoplists are a real capability and free. If your problem is fetching pages in many languages rather than classifying prose in them, that is a different purchase.

For the wider field, our web scraping API roundup covers hosted options and the open-source scraper pillar covers self-hosted ones. If the output is bound for a model, converting HTML to Markdown in Python is where most fidelity gets lost.

Try Thunderbit for Web Data Extraction

Should you use jusText?

It is a candidate if the corpus is multilingual or its paragraph-length distribution benefits from explicit threshold tuning. Validate both properties before deployment.

The 100 bundled stoplists are a real design feature, but multilingual accuracy was not tested. The threshold cliff is tunable; whether a lower setting is acceptable depends on the recall and boilerplate precision measured on a labeled sample.

On this English synthetic set at defaults, newspaper4k leaked zero labeled boilerplate units and recovered 0.9865 of article units. That makes it a comparison candidate for this workload, not a universal replacement recommendation.

Try Thunderbit for Web Data Extraction Get Started Free

FAQs

Why does jusText return an empty string instead of a partial extraction? Its classifier has two passes. A paragraph earns the good class on its own only above length_high (default 200 characters); the second pass then promotes neighbouring neargood blocks. With no paragraph over the threshold there is no seed, so every candidate decays to boilerplate and the result is empty. It is all-or-nothing by construction, not a failure to find a partial answer.

Is jusText abandoned? The last repository push and the last PyPI release were both 2025-02-25 — seventeen months before testing — and the PyPI classifiers stop at Python 3.9. But it installed and ran on Python 3.14.2 without an exception, and 9 open issues is not a large backlog. Read the dates as maintenance risk rather than proof of broken behavior; the practical risk is that you may need to fix your own bugs.

Why does it leak more boilerplate than Readability? On this fixture set, eligible prose-like neighboring blocks crossed the output boundary under the default context rules. Promotion is conditional on block class, distance, and context rather than automatic for every neighbor. The measured result was a 47% boilerplate-unit leak and 74 contaminating tokens at defaults.

How do I use it for another language? justext.justext(html, justext.get_stoplist("German")). get_stoplists() returns all 100 available. The stoplist is the language-specific lexical input to a classifier that also uses paragraph length, link density, HTML-derived segmentation, heading distance, and neighboring-block context. This changes the language input; it does not by itself validate German extraction quality.

What was not tested here? Real-world pages, at all — these are controlled fixtures with labelled units. The multilingual capability, which is the library's main selling point, was inventoried at 100 stoplists but not scored on non-English text. Encoding edge cases, despite jusText exposing encoding, default_encoding and enc_errors parameters. And max_heading_distance and the two stopword-ratio thresholds were left at defaults throughout.