BeautifulSoup is the library almost everyone reaches for the first time they scrape a web page in Python, and it is genuinely the slowest of the serious HTML parsers. Both of those things are true, and neither is a criticism. The interesting part is that "slowest" turns out to be a precise, buyable number rather than a vibe.
I put bs4 (that's beautifulsoup4, version 4.15.0, shipped June 2026, MIT-licensed) through a mix of fresh capability tests and reused timing data from the same benchmark rig, and the picture is consistent: you trade a roughly order-of-magnitude speed penalty for the friendliest API and the strongest error tolerance in the field. Whether that trade is smart depends entirely on your workload, so this review keeps both halves of it on the table.
What BeautifulSoup Actually Is (and Isn't)
Most tutorials skip the part that matters most: BeautifulSoup does not parse HTML. It's a wrapper. Under the hood it hands your document to one of three real parsers — Python's built-in html.parser, lxml, or html5lib — and then wraps whatever tree they produce in a single, very friendly navigation and search API. bs4's job isn't to parse. It's to make the result pleasant to walk around in.
Its own author calls it a "screen-scraping library," and the pitch has always been the same: point it at HTML so malformed a browser would wince, and it'll still fish out the data you asked for. That reputation is earned, with one asterisk we'll get to.
A few facts worth pinning down before anything else:
| Field | Value |
|---|---|
| Package | beautifulsoup4 (import as bs4) |
| Tested version | 4.15.0 (uploaded 2026-06-07) |
| Python requirement | >=3.7.0 |
| License | MIT |
| Canonical home | crummy.com/software/BeautifulSoup |
| Source + bug tracker | Launchpad — not GitHub |
| Maintenance | Active (4.15.0 in June 2026, six releases in the past year) |
That "not GitHub" line matters more than it looks. bs4 is a 20-year-old library that lives on crummy.com and Launchpad, so the usual GitHub star-count vibe check doesn't apply. Judge its health by release cadence instead, and by that measure it's alive and well.
One subtlety on licensing, for anyone who has to answer to a compliance team: the wrapper is MIT, but what "using bs4" actually pulls into your dependency tree depends on which backend you install. html.parser is the Python standard library (PSF license, zero extra dependencies). lxml is BSD, but it sits on libxml2/libxslt — an external C dependency you either compile or pull as a prebuilt wheel. html5lib is pure Python and MIT. If you want the cleanest dependency footprint, the built-in html.parser gives it to you — which, as it happens, is also the backend with the biggest catch. More on that shortly.
The Speed Tax, Quantified
Let's put the number down first, because it's the headline and hiding it would be dishonest. On a realistic parse-then-extract task — parse the string, pull every <h3 class="title"> and every <a href> — BeautifulSoup is the slowest parser in this comparison, and it isn't close.
These timings are reused from the selectolax benchmark rig (same machine, same 3-run methodology, as-of 2026-07-13); this review doesn't re-run any timing benchmark of its own, to avoid CPU contention and duplicated work. Median p50 latency, in milliseconds:
| Page size | bs4 (html.parser) | bs4 (lxml) | selectolax-Lexbor | lxml | bs4-hp slower | bs4-lxml slower |
|---|---|---|---|---|---|---|
| 1 KB | 0.323 | 0.281 | 0.027 | 0.036 | 12.0x | 10.5x |
| 10 KB | 2.049 | 1.705 | 0.160 | 0.166 | 12.8x | 10.7x |
| 100 KB | 20.599 | 16.540 | 1.464 | 1.423 | 14.1x | 11.3x |
| 1 MB | 232.558 | 181.855 | 14.901 | 14.177 | 15.6x | 12.2x |
| 10 MB | 2788.746 | 2261.557 | 159.935 | 172.933 | 17.4x | 14.1x |
So bs4(html.parser) runs about 12–17x slower than a C parser like selectolax-Lexbor, and switching to the lxml backend only pulls it back to 10.5–14x — still a full order of magnitude behind. The reason is structural, not a bug: no matter which backend does the parsing, bs4 builds a complete Python object (a Tag or a NavigableString) for every single node. That object-materialization layer is a tax the C parsers simply don't pay.
Notice the multiplier climbs as pages grow — 12.0x at 1 KB, 17.4x at 10 MB. That tells you this isn't fixed startup overhead you can amortize away. It's a per-node tax that scales linearly with how many nodes you build.
Now for the reframe, because "10x slower" sounds scarier than it usually is. On a 1 MB page, that's 232 ms versus 15 ms. If your job is "scrape a few hundred to a few thousand pages, a few hundred KB each," that absolute difference is invisible — you will not feel it, and optimizing it away buys you nothing. If your job is a million-page pipeline, the same ratio is the difference between a job that finishes and one that doesn't. Same number, opposite verdict. Weigh it against your actual volume, not against the benchmark.
No, Switching Backends Doesn't Fix It
There's a persistent myth that you can hand bs4 the lxml backend and get lxml's speed. You can't, and it's worth understanding why. On a 100,000-node batch CSS query (select every <a> and read its href, tree pre-built), the throughput split is stark:
| Parser | Query p50 | Nodes/sec |
|---|---|---|
| lxml | 33.30 ms | 3,002,646 |
| selectolax-Modest | 34.19 ms | 2,924,550 |
| selectolax-Lexbor | 39.46 ms | 2,534,027 |
| bs4 (lxml) | 250.56 ms | 399,111 |
bs4(lxml) manages about 399,000 nodes/second — roughly 6.3–7.5x slower than the three C engines, even though its own backend is lxml. The backend accelerates tree building. Querying and traversal still route through soupsieve into bs4 Tag objects, and every matched node still gets boxed in Python. So the mental model "give bs4 lxml and it's lxml-fast" is wrong: the backend speeds up one phase, and the slowest phase isn't that phase.
Memory and cold start round out the cost. On a 10 MB document, bs4 uses about 1.5–1.75x the resident memory of selectolax or lxml (218–226 MB versus 129–145 MB) — same root cause, one Python object per node. And importing bs4 takes about 33.4 ms versus 14.1 ms for lxml.html, so it's 2.36x slower to import. That last one is a rounding error for a long-running process, but for a CLI tool or a serverless function that cold-starts constantly, it's a small real cost worth knowing about.
Why More Threads Won't Save You
If your instinct on a slow CPU-bound task is "throw threads at it," bs4 will punish that instinct. On a 1 MB page parsed 48 times, single-threaded versus four threads:
| Parser | 1 thread | 4 threads | Speedup |
|---|---|---|---|
| selectolax-Lexbor | 0.563 s | 0.159 s | 3.54x |
| lxml | 0.459 s | 0.378 s | 1.21x |
| bs4 (lxml) | 6.945 s | 26.842 s | 0.26x |
Read that bottom row twice. Four threads made bs4 about 3.9x slower, not faster. The empirical signal is "likely holds the GIL": bs4's tree construction is pure Python, so it serializes under the Global Interpreter Lock, and piling on threads just adds scheduling overhead to a job that can't actually run in parallel. selectolax gets its ~3.5x speedup because its C core releases the lock; bs4 has no such room.
For the free-threading era this is the practical takeaway: if you need to parallelize BeautifulSoup, reach for multiprocessing (ProcessPoolExecutor), not threads. selectolax and lxml can scale on threads; bs4 can't. One caveat on rigor — this is a single observation at one thread count (4) on one page size (1 MB), and the "holds the GIL" mechanism is a hypothesis inferred from wall-clock behavior, not something I confirmed by instrumenting which code path holds the lock. The direction is clear; the exact mechanism is provisional.
The Default Backend Is the Trap. Read This First.
If you take one thing from this review, take this. A plain BeautifulSoup(html) with no second argument uses html.parser, and html.parser does not implement HTML5's optional-end-tag rules. That sounds academic until it silently corrupts your data.
I ran 15 deliberately malformed HTML samples through all three backends, with a backend-agnostic structural assertion pre-registered for each one before running (so nobody gets to pick the winner after the fact). The scores:
| Backend | Meets expectation / 15 |
|---|---|
| lxml | 15 |
| html5lib | 15 |
| html.parser | 12 |
The three failures all share one root cause. Take an unclosed table: <table><tr><td>a<td>b<tr><td>c<td>d</table>. Under html.parser, the extracted cell text comes out as ['abcd','bcd','cd','d'] — each <td> swallows everything after it, because the parser nests the cells instead of closing them. lxml and html5lib both correctly return ['a','b','c','d']. Bare list items behave the same way: <li>a<li>b<li>c gives you the nested ['abc','bc','c'] under html.parser, and the clean ['a','b','c'] under the other two. Duplicate attributes flip too — <div id="first" id="second"> keeps "second" under html.parser but "first" under lxml/html5lib, and the HTML5 spec says keep the first.
Here's why that's dangerous rather than merely annoying: it happens without raising an error. A scraper that casually does BeautifulSoup(html) and hits an unclosed table or list — which is depressingly common on old sites, hand-written HTML, and templates that forgot a closing tag — will leak adjacent cell text together into one field, hand you dirty data, and never once complain. The fix is one argument: BeautifulSoup(html, "lxml") or BeautifulSoup(html, "html5lib").
To be fair to html.parser, the other 12 of 15 malformed samples came out identical across all three backends — mis-nested tags like <b><i></b></i>, missing html/body skeletons, unquoted attributes, orphaned closing tags, unclosed comments, nested forms, mixed case, and more. bs4's tolerance really is strong across the board; the divergence is concentrated almost entirely on the optional-end-tag family. And none of this is a discovery — bs4's own "Differences between parsers" documentation already says html.parser is "less lenient" in plain language. What the malformed matrix adds is the specific, reproducible cases where "less lenient" turns into wrong output.
What You Don't Give Up: The API and the CSS Are the Best Part
So bs4 is slow, single-threaded, and has a default-backend trap. People still reach for it anyway, because the "friendly" half of the trade-off is completely real — and it holds up under testing.
I ran 29 API probes covering search, CSS, tree navigation, text extraction, and DOM modification. All 29 passed, with each probe's result computed by comparing the actual return against an expected value rather than eyeballed. Two of those capabilities are ergonomics that the C parsers just don't offer:
- Function predicates in
find/find_all. You can writesoup.find(lambda t: t.name == "a" and "btn" in t.get("class", []))and express a complicated condition in a single line of Python — no "select everything, then filter" two-step required. - Named, bidirectional tree navigation.
.parent,.next_sibling,.find_parent,.stripped_strings,.descendants— the traversals read like English and go both directions. selectolax needs multiple steps for some of these, or doesn't offer them at all.
That's the "buys you developer time" half made concrete. It's not marketing; it's 29 green checks.
Two traps to note, since a fair review names both sides. First, boolean attributes: <input disabled> returns an empty string "" for disabled in bs4 (selectolax returns None). Both are falsy, so if node.get("disabled") silently misses a boolean attribute that's actually present under either library — the safe test is "disabled" in tag.attrs. Second, get_text(strip=True) concatenates node text with no separator after stripping, so "...with " + "link1" becomes "withlink1". Pass separator=" " when you need word boundaries. Neither trap is bs4-specific; both are cross-library gotchas.
And now the part that surprises people: choosing bs4 does not cost you CSS coverage. Its CSS engine, soupsieve, is the most complete implementation in this entire comparison. On the 41-case base matrix (reused from the selectolax rig) soupsieve scored 41/41 — the only perfect score in the field, ahead of selectolax-Lexbor's 39/41 and cssselect's (lxml/parsel) 37/41. I then ran 20 additional extended cases soupsieve's docs advertise, and it went 20/20, including selectors Lexbor outright rejects: :lang(en), the soupsieve-only :-soup-contains('featured'), :is(), :where(), and :has(> a). The only real gaps are XPath (soupsieve is CSS-only) and parsel's ::text / ::attr() pseudo-elements, which are Scrapy extensions. If you live in XPath, that migration will hurt.
The verdict for this section is clean: what you sacrifice by choosing BeautifulSoup is speed. It is not API ergonomics, and it is definitely not CSS coverage.
Two Production Gotchas Worth Budgeting For
Beyond the default backend, two behaviors will bite you specifically in long-running or non-UTF-8 workloads.
Reference Cycles: Call decompose() in Long Loops
Every bs4 Tag holds a reference to its parent and to its children, which forms a reference cycle. CPython's reference counting can't reclaim a cycle on its own — that's the generational garbage collector's job. To see how much that matters, I built and deleted a tree 300 times with GC turned off, then counted the surviving Tag objects still sitting in memory:
| Scenario | Tags retained after del |
|---|---|
| GC off | 120,900 (300 cycles, nothing reclaimed) |
| GC on | 26,598 (generational GC fired mid-loop) |
After forced gc.collect() | 0 (all reclaimed) |
| No-cycle control (list of strings, GC off) | delta 0 |
With GC off, del soup reclaimed nothing — all 120,900 objects stayed resident, because the reference cycle defeats reference counting. A single gc.collect() cleared every one of them. The no-cycle control group (a plain list of strings, known to have no cycle) held a delta of zero, which proves the buildup came from bs4's cycle and not from measurement noise. bs4's own docs say the objects are "densely interconnected ... exactly the sort a garbage collector would have trouble with," so this is documented behavior; what the test adds is the retained-object count and the proof that collect() zeroes it.
The practical rule: in a pipeline that parses many large pages in a tight loop, if your code (or some high-throughput setting) disables GC or doesn't trigger it often enough, bs4 trees will linger and memory will climb. Call soup.decompose() after each page — bs4 provides it precisely to break the cycle and reclaim early. The C trees from selectolax and lxml don't have this problem at all.
Encoding: UnicodeDammit Is bs4's Quiet Advantage
bs4 ships a component the fast parsers don't: UnicodeDammit, which sniffs a document's encoding and converts it to Unicode automatically. I gave it an 8-case "declared vs. actual charset" matrix:
| Case | True encoding | UnicodeDammit guessed | Recovered? |
|---|---|---|---|
| utf8_no_decl | utf-8 | utf-8 | Yes |
| utf16_bom | utf-16 | utf-16le | Yes |
| gbk_chinese | gbk | gb18030 | Yes (superset) |
| shiftjis | shift_jis | cp932 | Yes (superset) |
| latin1_declared_utf8 | latin-1 (declared utf-8) | iso-8859-1 | Yes (ignored the lie) |
| latin1_no_decl | latin-1 | cp720 | No |
| cp1252_no_decl | cp1252 | cp862 | No |
| utf8_declared_latin1 | utf-8 (declared latin-1) | iso-8859-1 | No (followed the lie) |
Five of eight recovered. UTF-8, UTF-16 with a BOM, GBK, Shift-JIS, and even mislabeled latin-1 all came back correctly, and the superset guesses (GBK→gb18030, Shift-JIS→cp932) still decode fine. The two failure modes are worth knowing: short latin-1/cp1252 byte samples get misjudged as DOS code pages, because the statistical detector isn't reliable on short inputs and DOS box-drawing characters overlap Latin-1's code points; and when a <meta charset> declaration is simply wrong, UnicodeDammit trusts the declaration. bs4's docs flag both — a sample can be "so short that Unicode, Dammit can't get a lock on it," and more data means a better guess.
Against selectolax, which silently corrupts non-UTF-8 bytes and expects you to decode them yourself, this is a genuine advantage: bs4 at least attempts to sniff and often succeeds. But it's not a guarantee. For a known encoding, skip the guessing and be explicit: BeautifulSoup(bytes, from_encoding="...").
Do the Backends Ever Actually Disagree on Real Pages?
The malformed matrix shows the backends diverging on deliberately broken input. The obvious next question is whether that matters in the wild, so I ran all three backends over 11 real fetched pages — BBC, Wikipedia, Craigslist, MDN, old.reddit, Python docs, Hacker News, Books to Scrape, webscraper.io, whitehouse.gov, and a JS-rendered quotes page — comparing link, heading, and image counts.
All three agreed on all 11 pages. Zero divergence. Which means the backend disagreement from the trap section shows up only on deliberately malformed HTML; when a modern production site is structured well enough — even a "messy" one — the backend choice doesn't change what you extract. The practical read: for mainstream, well-formed sites, html.parser is perfectly fine and saves you the dependency. Only when you're scraping visibly non-standard, hand-written, or ancient HTML does the backend choice start moving your results, and that's when you switch to lxml or html5lib.
One aside from that run, because it's a real edge case. The MDN page contains a <template> element, and all bs4 backends returned 508 links — meaning bs4 flattens <template> contents into the main tree. That puts bs4 on the same side as lxml, and opposite selectolax-Lexbor, which strictly follows the HTML5 spec (a <template> is an inert DocumentFragment) and returns 497, silently dropping the 11 links inside the template. So bs4 will capture data inside a <template> — useful, but also a way to pick up "phantom" content a browser would never render. Neither behavior is wrong; they're different spec interpretations, and you should know which one you're getting.
Where BeautifulSoup Fits — and Where It Doesn't
Rather than crush all of this into a single 0–100 score (which would hide exactly the trade-offs that matter), here's the dimension-level scorecard, with a caveat on each row:
| Dimension | What the tests found | Reader caveat |
|---|---|---|
| Install / first run | Pure wrapper, no browser/setup; html.parser zero-dep; all prebuilt wheels | lxml backend needs a C dependency |
| Speed vs C parsers | 12–17x slower (html.parser) / 10.5–14x (lxml backend), all sizes | Single rig; reused selectolax data |
| CSS query throughput | ~6–7.5x slower on 100k nodes; lxml backend doesn't rescue it | Reused; pays the Python Tag tax |
| Memory | 1.5–1.75x selectolax/lxml; heaviest | Reused; measured by RSS |
| Import cold start | 2.36x slower (33.4 vs 14.1 ms) | Reused; small item |
| Thread scaling | bs4-lxml ~3.9x slower at 4 threads (holds GIL) | Single observation; use multiprocessing |
| API ergonomics | 29/29 probes; function-predicate find + bidirectional nav | Empty-string bool-attr and strip word-boundary traps |
| CSS coverage | soupsieve strongest: 41/41 base + 20/20 extended; supports :lang | No XPath, no ::text |
| 3-backend tolerance | lxml/html5lib 15/15; html.parser 12/15 | Divergence only on malformed HTML |
| Real-page consistency | 3 backends agree 11/11; all flatten <template> (508) | Well-formed sites: backend doesn't matter |
| Reference-cycle GC | Tree is a cycle; 300 loops retained 120,900 objects, collect zeroed it | Long loops need decompose() |
| Encoding | UnicodeDammit recovers 5/8; misjudges short samples, follows bad declarations | Single observation |
| Maintenance | Active (4.15.0, June 2026); MIT | Home on crummy/Launchpad, not GitHub |
So who is BeautifulSoup for? Anyone who values a readable API and forgiving parsing over raw throughput, working at moderate volume — prototypes, one-off scrapes, internal tools, teams where developer time costs more than runtime. Who should look elsewhere? Million-page pipelines where the speed tax compounds into real money, workloads that need thread-level parallelism, and anyone married to XPath.
A note on where this fits in a real scraping stack, and where our own tool comes in. BeautifulSoup assumes you already have the HTML. It doesn't fetch pages, it doesn't render JavaScript, and it does nothing about anti-bot defenses or CAPTCHAs — that's a separate job entirely, and a genuinely hard one on the modern web. This is where an AI scraping API sits at a different layer: Thunderbit's developer stack — a REST API, an MCP server, and a CLI — handles the fetch, the JS rendering, and the anti-bot problem, then returns either clean Markdown (POST /distill) or schema-matched structured JSON (POST /extract) without you writing selectors at all. The two aren't competitors; they're complementary. bs4 parses HTML you already hold; Thunderbit's API, MCP, and CLI get you the HTML you can't easily reach in the first place. If your bottleneck is parsing, bs4 is a fine answer. If your bottleneck is acquisition, that's the other layer.
Try Thunderbit for Web Data Extraction
The Bottom Line
BeautifulSoup gives you the friendliest API, the strongest malformed-HTML tolerance, and the most complete CSS engine in this comparison — bought with a roughly order-of-magnitude speed tax and the heaviest memory footprint. That's the whole trade, stated plainly. The default html.parser backend is the one real trap: it silently mangles unclosed tables and lists, so pass "lxml" or "html5lib" whenever your input might be ugly. Threads won't speed it up — multiprocessing will. And in long-running loops, decompose() each page to keep the reference cycles from piling up.
Two limitations to close on. Everything here was measured on a single platform (macOS arm64, Python 3.14, prebuilt wheels), and the timing multipliers are reused from the selectolax rig (same bench, as-of 2026-07-13) rather than re-run — so they inherit that single-platform limitation, and a Linux x86_64 or source-compiled setup could shift the exact figures. And nothing in these results is a novel discovery: bs4 is a 20-year-old library, so every behavior tested is either documented or publicly recorded. The value isn't a scoop. It's putting a real number on trade-offs the docs only describe qualitatively.
Frequently Asked Questions
Is BeautifulSoup slow?
Yes, measurably. On a parse-plus-extract task it runs about 12–17x slower than a C parser like selectolax-Lexbor with the default html.parser backend, and 10.5–14x slower with the lxml backend, because it builds a Python object for every node. Whether that matters depends on scale: on a 1 MB page it's 232 ms versus 15 ms, invisible for a few thousand pages but decisive for a million-page pipeline.
Which BeautifulSoup parser should I use — html.parser, lxml, or html5lib?
For well-formed, mainstream sites, the default html.parser is fine and adds no dependencies. But it doesn't implement HTML5 optional-end-tags, so on unclosed tables or lists it leaks adjacent text together without erroring. When your input might be malformed, hand-written, or old, pass "lxml" or "html5lib" explicitly — both scored a clean 15/15 on a malformed-HTML matrix where html.parser scored 12/15.
Can BeautifulSoup parse in parallel with threads?
No. bs4's tree building is pure Python and holds the GIL, so adding threads makes it slower, not faster — in testing, four threads ran a 1 MB parse about 3.9x slower than one thread. To parallelize bs4, use multiprocessing (ProcessPoolExecutor). Libraries with C cores, like selectolax and lxml, are the ones with thread-level parallelism to gain.
Does BeautifulSoup handle broken HTML well?
Broadly yes — across a range of malformed samples (mis-nested tags, missing skeletons, unquoted attributes, and more), all three backends recovered cleanly. The one weak spot is the default html.parser and optional-end-tags: unclosed <td>/<li> get nested instead of closed, corrupting extracted text. Switch to the lxml or html5lib backend and that class of problem goes away.
BeautifulSoup vs lxml — which is better?
They're different tools. lxml is far faster at both tree building and querying, and supports XPath. BeautifulSoup wraps lxml (among others) in a much friendlier API and actually has broader CSS coverage through soupsieve. Just don't expect the lxml backend to make bs4 lxml-fast — the backend only accelerates parsing, while queries and traversal still pay bs4's per-node Python object cost, leaving it roughly 6–7.5x slower on large batch selections.
Try Thunderbit for Web Data Extraction Get Started Free
