newspaper4k Returned Output on 22/22 Controlled Article Fixtures

Across one annotated, article-oriented fixture set, newspaper4k produced output on all 22 fixtures, recovered 98.65% of scored article units, and included none of the labeled boilerplate tokens. Those three dimensions make it a strong candidate under this test's priorities; they do not define a universal winner.

Nothing else in this set combined those three observed results. Mozilla Readability recovered every scored content unit but included more labeled boilerplate; goose3 included no labeled boilerplate but returned empty strings twice. Other decision rules can prefer another library.

One fetching default deserves explicit attention before deployment.

What newspaper4k is

newspaper4k is a maintained fork of newspaper3k, which was itself the Python-3 continuation of the original newspaper. That lineage matters when you are searching for help, because most of what you will find online refers to the ancestor and some of its API has moved.

Official reference: newspaper4k's official repository.

The most common way to get this library wrong, for instance, is to reach for set_html(). That method does not exist. HTML goes in through download():

from newspaper import Article
a = Article(url="https://example.com/story")
a.download(input_html=html)     # not set_html()
a.parse()
text = a.text

I got that wrong on my first run and scored the library at 0 of 22 fixtures before checking whether the fault was mine. It was.

What you get back is not just text. The Article object exposes fields such as text, title, authors, publish_date, top_image, images, movies, meta_description, meta_lang, tags, and article_html. keywords and summary require the optional NLP installation and corpus setup described below. The field surface was inventoried but metadata accuracy was not scored.

Version tested: 0.9.6, MIT, 1,135 GitHub stars, with a repository push dated 2026-07-31. That dated activity is a snapshot, not a full assessment of maintenance health. Python 3.14.2.

The result

sixway-scores.json. Every unit in every fixture carries a unique sentinel token, so "recovered" and "leaked" are exact substring membership rather than a similarity score. Recall is over all 22 fixtures; leak and precision over the 11 containing both article and boilerplate.

Three columns are worth separating.

It answered every page. goose3 and jusText did not — 20 and 19 of 22. That matters more than it looks, because precision and F1 in a table like this are conditional on producing output: a library that returns an empty string contributes to neither side of the ratio, so declining is free. goose3's 1.0000 precision was scored on 10 of 11 fixtures; newspaper4k's 0.9452 on 11 of 11. They are not quite the same measurement.

It leaked nothing. The fixtures include deliberately adversarial boilerplate — neutral-classed promo blocks sitting as siblings of the article, comment threads on innocuous class names, ad blocks that do not say "ad". Readability's sibling-append heuristic swallowed several; newspaper4k took none of them.

It missed one unit out of 74, and the one it missed is shared.

The miss is on the non-prose fixture — a page built from tables, a code block, short items and an image caption rather than paragraphs — and the unit it dropped is the caption. It is not alone there:

Three libraries drop the same caption and no other unit, which reads less like three separate bugs and more like a shared inherited assumption about what a caption is worth. If your content is documentation, recipes or anything where the caption carries information the paragraph does not, that is worth testing before you commit — and Readability and jusText both kept it.

The same fixture is where goose3 falls apart entirely, losing three-quarters of the page, so "non-prose content" is an axis on which these six differ far more than the headline table suggests.

On speed, newspaper4k's median extraction across the 22 fixtures was 2.69 ms, the slowest of the six, with a worst case of 199.81 ms. Against resiliparse's 0.06 ms median that is a 45× gap in this controlled run. The median may be small in a single-page workflow, but throughput and tail latency under load were not tested. Cold import is measured separately below.

The default I would change on line one

Official reference: newspaper4k documentation.

Reading the shipped Configuration object turns up twenty-two settings. One of them is this:

_honor_robotstxt = False

newspaper4k does not respect robots.txt unless you tell it to. If you let it fetch — Article(url).download() with no input_html — it will fetch whatever you point it at, regardless of what the site's robots file says.

That is a defensible engineering default for a library whose primary use is parsing HTML you already hold, and it is an indefensible one to discover in production after you have pointed it at a thousand URLs. Set honor_robotstxt=True on your Configuration, or pass input_html and do your own fetching, which is what I did throughout this test.

Two more worth knowing:

number_threads = 10. The default parallelism for its multi-article helpers is ten. Concurrency is not requests per second, but it can create a burst of simultaneous requests unless you set explicit per-host limits and scheduling.

fetch_images = True. Image fetching is on by default, which raises an offline-use question. With socket.connect blocked, the tested newspaper4k 0.9.6 path—download(input_html=…) followed by parse() on one held-HTML input—completed, returned 1,292 characters, and attempted zero network connections. This supports that exact path, not every configuration, plugin, content type, or future release.

The rest are sensible: min_word_count 300, min_sent_count 7, max_text 100,000, http_success_only True, memorize_articles True, follow_meta_refresh False, allow_binary_content False.

Setup reality

pip install newspaper4k pulls 22 packages and 47.5 MiB in about six seconds. Cold import in a fresh subprocess: 2.812 s — the slowest in the comparison.

install-and-import.json. Each library in its own empty virtualenv, so nothing inherits a sibling's dependencies.

2.812 seconds to import is 187 times resiliparse's 15 milliseconds. In a long-running worker you pay it once and it is irrelevant. In a serverless function you pay it on every cold start, and that is the case where newspaper4k is the wrong choice regardless of how good its extraction is.

One rough edge on install. Importing prints a warning:

UserWarning: nltk is not installed. Some NLP features will be unavailable. Install it with: pip install 'newspaper4k[nlp]'

Nothing in this test needed those features and extraction worked fine without them, but the base install is not the complete install, and newspaper4k[nlp] pulls a substantially heavier tree plus corpus downloads. Budget for that only if you want keywords and summaries.

Memory, and what broken HTML does to it

Two things every review in this batch listed as untested, now measured.

The broader stress-test context is in the ten-library memory and malformed-HTML comparison.

Peak resident memory, via /usr/bin/time -l, one fresh process per cell — the import floor is what the library costs loaded and idle, the peaks include the document.

memory-results.json. Python and Node baselines are not comparable to each other; the interpreter is inside both.

newspaper4k's floor is 52.6 MiB and its 10 MB peak is 668.5 MiB — second-heaviest of the Python libraries. Neither number is a problem for ordinary pages; both matter if you are batching large documents in a memory-capped worker.

Broken HTML. Twelve documents each breaking exactly one thing — unclosed tags, mis-nested inline elements, unquoted attributes with spaces, stray closers, no <html> at all, duplicate attributes, a document truncated mid-tag, bad entities, an unclosed <script>, a lying charset declaration, a comment containing markup, and 600 levels of nesting — plus two well-formed controls at matched sizes, because "it returned nothing" only says something about malformedness if the library is not also silent on a clean document of the same size.

The controls and malformed cases separate cleanly in the raw results:

The two controls produced 70 and 1,351 characters, while ten of twelve malformed inputs produced an empty string. That makes the silence attributable to malformedness in this fixture design rather than simply to short input. The scorer checks heading, paragraph, and link sentinels across the eleven eligible malformed documents. The unclosed-<script> fixture is excluded because, under HTML5 parsing, following markup remains script content. See malformed-results.json. This is a significant recovery limitation to carry into selection, not merely a “did not raise” success.

Pros and cons

In its favour. No labeled boilerplate tokens in this comparison, output on all 22 controlled article fixtures, and 0.9865 scored article recall. It exposes article text plus metadata fields, although metadata accuracy was not tested. The tested held-HTML path made no network attempts with socket.connect blocked. The repository had a recent dated push when checked; broader maintenance health was not evaluated.

Against it. The heaviest cold import in the set at 2.812 s and 22 packages / 47.5 MiB of measured site-packages. honor_robotstxt defaults to False, and multi-article helpers default to ten threads. The base install warns about missing NLTK, so keywords and summaries need a heavier optional install. Most importantly, ten of twelve malformed fixtures returned empty output even though both matched well-formed controls produced text.

Who should use it, and who should not

Consider newspaper4k when your priority order is: produce non-empty text on the controlled article-oriented corpus, minimize labeled boilerplate in that corpus, and accept a slower cold import. Under that explicit rule it led this fixture comparison. A different rule can choose Readability for maximum scored content retention or resiliparse for startup and median extraction speed.

Skip or bake-test it carefully when cold start dominates, when 47.5 MiB of measured site-packages is material, or when malformed recovery matters. Resiliparse imported 187× faster here, but it did not tie trafilatura on all quality columns: trafilatura had higher article recall, while their leakage and precision profiles also differed. newspaper4k is article-oriented; product listings and dashboards were not tested, so no behavior is claimed for them.

Whatever you do, set honor_robotstxt if you let it fetch. That is not a performance note.

Where a managed API fits

newspaper4k can parse HTML supplied by the caller and also has fetching paths. A managed extraction service places acquisition, rendering, and schema work behind a vendor boundary instead. We build Thunderbit, but it was not run against this fixture set, so this review supports no quality, rendering, anti-bot, latency, or cost comparison. For held article HTML, the evidence here is about newspaper4k alone; non-article targets need their own evaluation.

For the same fixtures across all six extractors, see the six-library extraction comparison.

For the hosted field, our web scraping API roundup is the wider view; for self-hosted alternatives, the open-source scraper pillar. If the text is headed into a model, converting HTML to Markdown in Python covers where fidelity is lost.

Try Thunderbit for Web Data Extraction

Should you use newspaper4k?

Treat newspaper4k as a strong candidate for article-text extraction when HTML is already available, then run a real-site bake-off on your corpus before adopting it. The controlled set supports high scored article recall, no labeled boilerplate, and non-empty output on all 22 article-oriented fixtures. It does not cover real pages or metadata accuracy, and ten of twelve malformed fixtures returned empty output.

If you use its fetching path, review honor_robotstxt=False, ten-thread parallelism, and per-host rate controls explicitly. If cold start or dependency footprint matters, measure the 2.812-second local import and 47.5 MiB site-packages observation in your deployment rather than treating them as universal container costs.

Try Thunderbit for Web Data Extraction Get Started Free

FAQs

Why doesn't set_html() work? Because it does not exist in newspaper4k. HTML goes in through download(input_html=html), then parse(). Search results can surface newspaper3k examples, making this an easy mistake; I made it on the first run and corrected the harness before scoring.

Does newspaper4k respect robots.txt? Not by default. honor_robotstxt ships as False. Set it to True on your Configuration if you let the library fetch, or pass input_html and fetch it yourself. Multi-article work also defaults to ten threads. That is unchosen parallelism, not a fixed request rate; set explicit per-host limits for fetching.

Does parse() make network requests? On newspaper4k 0.9.6, the tested download(input_html=…) plus parse() path made zero connection attempts for one held-HTML input while socket.connect was blocked. That does not establish every parser configuration, plugin, content type, or future release as network-free.

What is the NLTK warning on import? The base install does not include NLTK, so keyword extraction and summarisation are unavailable and the library says so at import. Extraction itself is unaffected — everything measured here ran on the base install. pip install 'newspaper4k[nlp]' adds them along with a heavier dependency tree and corpus downloads.

What did this review not test? Real-world pages—these are controlled fixtures with labeled units. Metadata fields were inventoried but not scored for title, author, date, or image accuracy. Multilingual extraction, NLP extras, multi-threaded source crawling, and throughput under load were also not tested. Process peak memory was measured on one 226 KB and one 10 MB HTML input, not under concurrency or sustained load.