trafilatura Rejected More Labeled Boilerplate Than Readability on the Same Fixtures

trafilatura is a no-browser Python content extractor. On the labeled fixtures in this article, its only same-HTML extraction-quality opponent is Mozilla Readability: trafilatura leaked one of 17 annotated boilerplate units while Readability leaked five. Browser-backed tools appear only in deployment-footprint context and were not quality-tested on these pages.

Its core job is main-content extraction: HTML in, article-oriented text and metadata out, with page furniture filtered by heuristics. It can serialize output as JSON and other formats, but it does not produce a user-defined row schema or repeated typed catalog records. Version 2.1.0 was tested. It does not execute JavaScript, and the Readability comparison shows a precision-versus-retention trade-off rather than a category-wide win.

Where trafilatura fits, and what it refuses to do

trafilatura describes itself as a Python and command-line tool to gather text and metadata on the web — crawling, scraping, extraction — with output as CSV, JSON, HTML, Markdown, TXT, or XML. That's a broad pitch. In practice the part that carries the tool is narrower and sharper: it takes an HTML document and returns the main content, stripped of the surrounding page furniture.

Here's the mental model, because it explains both the strength and the boundary. Most scrapers you point at a page are selector-driven — you tell them "grab the element with class product-price," and they return whatever sits at that address. trafilatura works the other way. It reads the whole document and decides which blocks are the actual article versus which are boilerplate, using content heuristics rather than a selector you wrote. That's why it needs no per-site rules to clean a page. And it's exactly why it can't hand you a structured catalog: there's no schema, no typed rows, just "here is the meaningful text." It's an extractor, not a parser you aim.

It also requires no separately downloaded browser runtime. The dependency tree includes platform wheels such as lxml, so “no browser download” should not be confused with a source-only or native-code-free install.

Setup: small dependency footprint, no browser install

pip install trafilatura in a fresh virtualenv on Python 3.14 installed 17 packages; the largest recorded wheel was lxml at 8.6 MB, per pip-install-trafilatura.log. No separate browser installation or post-install browser command was required.

For scale, the Scrapling pack in this same research base needed four separate pip invocations to get its fetchers working and ended at 26 packages, two of which are 42.2 MB Playwright driver wheels (tools/scrapling/artifacts/logs/pip-install-scrapling.log) — and that is before any browser binary is downloaded. The Crawlee pack measured that separate Chromium download at ~81.7 MiB (tools/crawlee/research-materials.md). Those tools do a bigger job, so this isn't a fair fight on capability; it's just what your disk and your CI cache see. And the version pip gave me (2.1.0) equals the current release, so there's no "you tested something stale" asterisk on any of the numbers below.

Hands-on: what the extractor actually returned

I ran it against fixtures built to hit both its sweet spot and its wall, with the raw outputs saved in the benchmark repo. The article test is the one that sold me.

I took a local article fixture and wrapped the real content in noise: a login prompt, a "Subscribe" nag, navigation links, a copyright footer — the standard boilerplate that a naive scraper drags in along with the body. trafilatura returned the title plus all 3 of 3 body paragraphs, and every one of those boilerplate markers — Login, Subscribe, Copyright — was gone from the output. On top of the text, it pulled the author and the date correctly from the page metadata. The full result, across .txt, .md, and .json, is in results/local_article.json.

The same saved HTML was exported as plain text, Markdown, and JSON. The article does not establish that all three formats came from one simultaneous call; they are alternative serializations of the extraction path. JSON here packages extracted text and metadata, not user-defined repeated records.

Here is the whole first run in one place, so you can check the claims above rather than take them:

Every row is read out of artifacts/raw/trafilatura-test-summary.json in the trafilatura pack — trafilatura 2.1.0, Python 3.14, macOS arm64, one run on one machine. Treat the times as observations, not a benchmark.

On the 500 fixture, fetch_url returned None after about 30 seconds while nearby local endpoints were fast. The run establishes the delay, not whether retry/backoff, a fixed timeout, or another internal path caused it. This article did not verify a supported per-call timeout parameter for fetch_url; the evidenced control is to fetch with a caller-controlled client and pass the resulting HTML to trafilatura.

The same tests also expose three boundaries.

First and biggest: trafilatura is a content extractor, not a structured scraper. I ran it on a product catalog fixture. It returned all 12 product names — as text — and exactly 0 structured rows (478 characters of flat text, per results/local_catalog_extraction.txt). The prices came through too, $18.00 down to $51.00, sitting on their own lines under each name. Everything you wanted is in the output; none of it is a field. If you need [{name, price, rating}, …], this is the wrong tool and no amount of configuration changes that — it's a design choice, not a bug.

Second: it does not render JavaScript. It consumes static HTML. Point it at a client-rendered page and you get whatever the server sent before the JS ran, which is often nothing useful. Pair it with a renderer if your targets are JS-heavy; trafilatura won't do that half for you.

Third, a caveat on my own evidence: that first public extraction test used a product-description block, not a genuine news article, because the public sandbox has no news pages. The article-cleaning result above is from a controlled local fixture. I trust it — the boilerplate removal is unambiguous — but I wasn't willing to dress up a product page as proof of newsroom-grade extraction, so the pack carried it as an open gap. A follow-up run on 2026-07-14 closed it, and the results are the next section.

Real-page boilerplate sanity check

Two real pages were fetched once, saved as offline fixtures with SHA-256 hashes, and re-run without network access. No timing or labeled ground truth was recorded. This is a real-page boilerplate sanity check, not a fidelity score.

Both rows come from artifacts/results/trafilatura-fidelity-summary.json. The markers checked were "Jump to content", "Privacy policy", "Powered by MediaWiki", "This page was last edited", plus "free news source" on the Wikinews page — all dropped, none leaked, on both pages.

The body/raw ratio measures reduction, not accuracy; omitted article content was not scored. author and sitename were null on these two MediaWiki templates, while the controlled local fixture supplied an author. This is a MediaWiki-specific miss observed in the test, not evidence of general byline reliability. The extracted dates are reported as returned and were not verified against ground truth.

Two extractors, the same HTML bytes, one scoring script

There is exactly one same-testbed comparison of trafilatura in this research base, and trafilatura's own pack didn't build it. The mozilla-readability pack did, as a control arm: 22 hand-labeled fixtures where every text block is tagged ARTICLE or BOILERPLATE and every word carries a unique sentinel token, so an extracted word maps back to exactly one labeled block. Both tools got identical HTML bytes. Readability ran under jsdom 29.1.1 on Node v22.22.3; trafilatura ran its own parser.

Source: tools/mozilla-readability/artifacts/raw/comparison.json in this research base — 11 fixtures in the content-fidelity set, 40 article units and 17 boilerplate units. These are synthetic fixtures deliberately weighted toward the adversarial cases, so read them as mechanism, not as a real-corpus ranking.

Neither tool sweeps it, and the split is the interesting part. trafilatura keeps chrome out — one leaked block against Readability's five, three contaminating tokens against 41. That is the precision half of "clean article text," and it's the half that decides whether your LLM context window fills with sidebar links. But Readability recovers content trafilatura throws away: on the non-prose fixture, trafilatura dropped a <figcaption> that Readability kept, and on a very short article it scored 0.571 against Readability's 0.800. Aggressive cleaning is a trade, not a free win. If your corpus is full of stubs, captions, and table-heavy pages, that trade goes against you.

How to turn the comparison into a selection test

Start by declaring which error is more expensive for your pipeline. If page furniture consumes downstream tokens or pollutes search results, boilerplate-unit leakage and contaminating-token counts deserve more weight. If dropping a caption, short post, or non-prose block is unacceptable, content recall by page shape deserves more weight. The shared fixture set makes that trade visible, but it does not choose the weights for a newsroom archive, documentation corpus, or retrieval pipeline.

The broader stress-test context is in the ten-library memory and malformed-HTML comparison.

Build the bake-off from saved HTML rather than live URLs so both extractors receive identical bytes. Include ordinary long-form articles, short notices, caption-heavy pieces, table- or code-heavy documents, and templates from each publisher you actually ingest. Label a small number of required content units and known furniture blocks before running either tool. Then score empty output, required-unit recovery, unwanted-unit leakage, and metadata fields separately. A single aggregate “quality” score can hide the exact failure mode that matters operationally.

Also validate the output contract downstream. trafilatura's JSON can carry extracted content and metadata, but a JSON serialization is not the same thing as a typed row schema. For article text, check minimum body length and required markers rather than accepting any non-empty output. For metadata, distinguish absence from incorrect values and keep the source URL plus extraction version so misses can be replayed.

Fetching should be tested as its own layer. The 30-second local failure observation belongs to fetch_url, not to extraction from already-held HTML, and its internal mechanism was not established. If deadlines, retries, authentication, or proxy policy matter, use a client whose behavior you control, record the final response bytes, and pass those bytes into the extractor. This isolates acquisition failures from content-selection failures and makes comparisons repeatable.

Finally, measure deployment properties on the intended platform. The 17-package Python 3.14 installation needed no separate browser, but that does not determine throughput, memory growth, wheel availability on every architecture, or behavior under concurrent workers. The article establishes a useful precision/retention mechanism against Readability; production suitability still depends on corpus and runtime tests.

For an external cross-check that isn't this base's own work, trafilatura's own evaluation page and the public ScrapingHub article-extraction-benchmark both put it ahead of readability-lxml on word-F1 across ~181 real pages — same direction as the table above. One caveat that most write-ups citing that benchmark skip: its widely quoted trafilatura F1 (~0.945) is attached to an older 0.5.1 line, not the 2.1.0 tested here.

Pros and cons

Pros:

• Lower labeled-boilerplate leakage than Readability on the shared fixture set: 1 block out of 17 versus 5.
• Pulled author and date correctly on the local fixture, and title/date/hostname on both real article pages.
• The same HTML can be serialized as text, Markdown, or JSON.
• Seventeen-package install with no separate browser runtime.
• Failure is quiet: fetch_url returns None on HTTP 500 instead of throwing.
• Tested version equals latest (2.1.0) — no version drift.
• Apache-2.0 licensed — permissive and commercially friendly.

Cons:

• Not a structured scraper: the catalog test returned 12 names and 12 prices as text and 0 typed rows. No schema, no fields.
• No JavaScript rendering — static HTML only; needs a separate renderer for client-side pages.
• author and sitename came back null on both real MediaWiki pages, so byline extraction is not a given.
• Aggressive cleaning has a cost: it dropped a <figcaption> Readability kept, and scored 0.571 to Readability's 0.800 on a very short article.
• That quiet 500 took 30.012 seconds to return None; use a caller-controlled fetcher when deadline control matters.
• The built-in crawl/sitemap spider and CSV/XML output formats weren't exercised in this pass, so I make no claims about them.

Who it's for — and who should skip it

trafilatura is aimed at article-oriented main-content extraction. It is a candidate for building text corpora, readable archives, or NLP inputs from static HTML when heuristic cleaning is acceptable. The shared fixture comparison supports lower boilerplate leakage than Readability, while Readability retained more of the short and non-prose cases.

Skip it if what you actually need is structured extraction — priced product rows, typed records, key: value fields — because it hands you text, not tables (the catalog test recovered all 12 names and all 12 prices, and 0 rows). Skip it too if your targets render their content in the browser and you're not willing to bolt on a separate renderer, since trafilatura reads static HTML and stops there. And think twice if your corpus is mostly short stubs, image captions, and table-heavy pages: on the labeled fixtures that is exactly where its cleaning cuts into real content (0.571 F1 on the very-short article, a dropped <figcaption>). Match the tool to the job: full article text, yes; structured JSON, JS pages, or 100-word stubs, look elsewhere.

Alternatives, including where Thunderbit fits

trafilatura is an Apache-2.0 self-hosted library with no vendor per-call fee; compute, bandwidth, fetching, monitoring, and maintenance remain operator costs. It handles article-oriented extraction from supplied HTML but not browser execution or user-defined repeated-row schemas.

For the same fixtures across all six extractors, see the six-library extraction comparison.

A managed service can place acquisition, rendering, and schema shaping behind a vendor boundary. We build Thunderbit, but did not run it against these fixtures, so this article supports no quality, rendering, latency, anti-bot, or cost comparison. The decision is self-hosted HTML-to-content control versus a hosted acquisition and structured-extraction boundary.

Related benchmark reviews: the full open-source scraper comparison, Crawl4AI's browser-backed Markdown review, and Firecrawl's self-hosted Markdown review.

Try Thunderbit for Web Data Extraction

Verdict

Consider trafilatura when the job is article-oriented extraction from static HTML and your priority is rejecting labeled boilerplate. On the shared fixtures it leaked one annotated boilerplate unit against Readability's five. That advantage came with lower retention on the very-short and non-prose fixtures, so corpus shape should decide the choice.

It does not execute JavaScript or emit user-defined repeated rows. On the two MediaWiki pages, author and sitename were null, a template-scoped observation. Readability retained the caption trafilatura dropped and scored higher on the fixture labeled f3_short_120; the visible evidence supports “lower F1 on the very-short fixture,” not a character count or a claim that the output was mangled.

Try Thunderbit for Web Data Extraction Get Started Free

FAQs

What kind of structure does trafilatura return? It can serialize extracted content and metadata as JSON, Markdown, text, HTML, XML, or CSV. That is structured serialization, but not a user-defined repeated-row schema such as product name, price, and rating. The catalog fixture returned values as flat content rather than typed rows.

Can trafilatura scrape a product catalog into structured rows? No. It's a content extractor, not a structured scraper. On a catalog fixture it returned all 12 product names as flat text and 0 structured rows — the names are in the output but they're not fields. If you need typed records like name/price/rating, use a selector-based parser or a schema-driven extraction API instead.

Does trafilatura render JavaScript? No. It consumes static HTML only. Point it at a client-rendered page and you get whatever the server sent before the JavaScript ran, which is often not the content you want. Pair it with a separate renderer if your targets are JS-heavy.

Is trafilatura hard to install? In the tested Python 3.14 virtualenv, installation pulled 17 packages and required no separate browser runtime. The largest recorded wheel was lxml at 8.6 MB. Separately, one local 500 call took 30.012 seconds to return None; use a caller-controlled HTTP client if you need a verified deadline.

Is trafilatura free for commercial use? It's Apache-2.0 licensed, which is permissive and commercially friendly. As always, confirm the current license on the repo before you build on it.