Every few months a faster HTML parser shows up, the benchmarks make the rounds, and someone declares the old guard obsolete. Then you go to select every paragraph that contains a certain word, or grab the parent of a matched node, and remember why lxml is still open in your other tab.
lxml is a 20-year-old libxml2 binding. It is not exciting. It is not new. And for one specific job — anything that needs real XPath — nothing else in mainstream Python actually competes. This is a hands-on review of what it does, where it quietly wins, and the couple of places its defaults will bite you if you don't know they're there.
lxml in One Paragraph: What It Actually Is
lxml is a Python binding to the C libraries libxml2 and libxslt. It is a parser and serializer, not a scraper and not a browser — it turns markup into a tree you can query and edit, and turns the tree back into bytes. It gives you an ElementTree-compatible API, a complete XPath 1.0 engine, XSLT 1.0, and schema validation, maintained by Stefan Behnel under the tagline "the most feature-rich and easy-to-use library for processing XML and HTML in the Python language".
Here's where it stands, as of a GitHub and PyPI snapshot pulled on 2026-07-14:
| Field | Value |
|---|---|
| Repo | lxml/lxml |
| Stars | 3,043 |
| Forks | 620 |
| Open issues | 16 |
| License | BSD-3-Clause |
| Created | 2011-02-11 |
| Last push | 2026-07-02 |
| PyPI stable | 6.1.1 (2026-05-18) |
| Bundled engine | libxml2 2.14.6 + libxslt 1.1.43 |
One thing to get out of the way before anyone accuses me of hype: there are no secrets in this review. lxml is old enough that every behavior here is documented somewhere in the lxml docs, a libxml2 changelog, or a launchpad thread. I found no exclusive, undocumented trick, and I'm not going to invent one. The value in what follows is that it's systematized, quantified, and organized around lxml as the subject — not that it's news.
The Test Setup (and Why the Timing Numbers Are Borrowed)
Two categories of data go into this review, and they come from two different places, so I'll be upfront about which is which.
The capability tests — XPath behavior, the two parser APIs, namespaces, encoding, node lifecycle — I ran fresh on one machine: macOS arm64, Python 3.14.2, lxml 6.1.1, libxml2 2.14.6. Every number in those artifacts/raw/*.json files is computed by a script run, not typed in by hand. Capability tests are deterministic booleans and enums, so a single run is stable — machine load doesn't change whether //a/@href returns an attribute string.
The timing and memory-footprint numbers are not from this pack. They're reused verbatim from the earlier selectolax benchmark pack — same machine, same virtual environment, same lxml and libxml2 build, benchmarks as-of 2026-07-13 — and I did not re-run them here. That's deliberate. Re-running timing benchmarks alongside a batch of capability scripts invites CPU contention that would pollute the reused figures, and it would be duplicate work: lxml was already a fully measured control library in that pack. Reusing the same bench keeps everything apples-to-apples instead of introducing a second, subtly different measurement. So when you see a millisecond figure below, read it as "same test rig, as-of 2026-07-13," not "I re-timed this today."
Findings carry a confidence tag: single-observation for the deterministic capability tests, triple-run for the reused timing distributions, hypothesis where I'm proposing a mechanism I didn't isolate.
XPath: The One Thing selectolax and BeautifulSoup Just Don't Have
This is the headline, so I'll start here.
I put lxml's xpath() through a 37-item matrix that was pre-registered — the expected result for every case was written into the source before the test ran, so I couldn't accidentally grade on a curve. Ten axes, nine predicate styles, ten built-in functions, three scalar return types, and five deliberate trap cases using XPath 2.0-only syntax that lxml's 1.0 engine should reject.
| Category | Coverage | Result |
|---|---|---|
| Axes | child / descendant / parent / ancestor / following-sibling / preceding-sibling / self / attribute / following / preceding | 10/10 pass |
| Predicates | [1] / last() / position()<n / attribute equality / attribute existence / and / or / nested [.//a] / not() | 9/9 pass |
| Functions | text() / contains() / starts-with() / count() / string-length() / normalize-space() / concat() / substring() / name() / string() | 10/10 pass |
| Return types | boolean / number scalars | 3/3 pass |
| Trap cases | matches() / sequences / if-then-else / except / syntax error | 5/5 correctly rejected |
The score is 37/37, and the trap column is the part that matters. matches(), sequence expressions, if/then/else, and except are all XPath 2.0 syntax, and libxml2's 1.0 engine doesn't half-support them — it raises XPathEvalError and refuses, rather than silently returning a wrong node set. So this is a perfect score after trying to break it, not a perfect score assembled from softballs. Every behavior here is exactly what the lxml XPath docs describe, which is the point.
I'll admit one thing the harness got wrong, because it's the version of "37/37" you can actually trust. My first expected set for //div[.//a[@href]] predicted two hits; the run returned one. I assumed lxml was wrong for about thirty seconds, then checked the fixture and found the second element was a <footer>, not a <div> — my expectation was wrong, not the engine. I fixed the expected set and left the mistake in a source comment. That's the correct order of blame: suspect your own test before the 20-year-old C library.
XPath vs CSS: What You Literally Cannot Express in CSS
The abstract "XPath is more powerful" claim deserves a concrete number, so I quantified the gap. lxml gives you both .xpath() and .cssselect() (the latter translates CSS to XPath under the hood). I took ten selection targets and checked which ones CSS can actually express.
| Target | XPath | CSS (cssselect) |
|---|---|---|
Filter by text content (contains(text(),"bargain")) | Yes | No text predicate |
Select parent from child (//b/parent::p) | Yes | No parent selector |
Return an attribute value (//a/@href) | Yes | Elements only |
Return a text node (//p/text()) | Yes | No text nodes |
Ancestor axis (//td/ancestor::div) | Yes | No upward navigation |
Filter parent by child count (//ul[count(li)=4]) | Yes | No count predicate |
Filter by text length (string-length(text())>5) | Yes | No length predicate |
nth-child / last-child / adjacent sibling | Yes | Yes (3 baseline) |
Seven of ten targets have no CSS equivalent at all. Text-content filtering, upward navigation to parents and ancestors, pulling an attribute value or a bare text node as the result, counting-based predicates — CSS can't say any of it. Only three (nth-child, last-child, adjacent sibling) work in both. That's the quantified answer to "what do I actually gain by reaching for lxml." selectolax is CSS-only and has no xpath() method at all, so those seven query types either become multi-step Python loops there or don't happen. If your scraping logic leans on any of them, that's your decision made.
(And yes, the harness caught me a second time here: I predicted an empty set for string-length(text())>5, but two six-character strings matched. Fixed the expectation, not the tool.)
Three Gears of Strictness: etree vs recover vs lxml.html
XPath is the reason to pick lxml. The three-speed strictness control is the reason to keep it.
Most parsers give you one behavior for broken input. lxml gives you three, and they're predictable enough that I fed six classes of malformed markup through each and pre-registered how every path should behave.
| Malformed input | lxml.etree (strict) | etree + recover=True | lxml.html (lenient) |
|---|---|---|---|
Unclosed tag <root><a>x</root> | raises | recovers | accepts |
Mis-nested <b><i></b></i> | raises | recovers | accepts |
Undefined entity | raises | recovers | accepts |
Bare & (Tom & Jerry) | raises | recovers | accepts |
Multiple roots <a>1</a><b>2</b> | raises | recovers | accepts |
| Well-formed XML | accepts | accepts (0 errors) | accepts |
Boolean attribute <input disabled> | raises | recovers | accepts |
Seven out of seven matched the pre-registered expectation. lxml.etree raises XMLSyntaxError on all six malformed classes. Add recover=True to the same parser and it swallows the errors and reconstructs a usable tree — and this is the underrated part — parser.error_log then enumerates every error it swallowed. lxml.html accepts everything without complaint.
The classifier that decides "raised vs recovered vs accepted" is itself driven by the runtime error_log length, not hard-coded, which is why a well-formed document run under recover=True gets correctly labeled "accepts" (empty log) rather than "recovers." My first version of that classifier tagged any recover=True result as "recovers" and mislabeled the clean input; reading the actual error_log fixed it.
What this buys you in practice: strict validation where a broken feed should fail loudly, use lxml.etree. Dirty real-world HTML you just need to get through, use lxml.html. And the middle case that most tools can't do — "be lenient, but tell me exactly what was broken so I can log it" — use recover=True and read the error log. selectolax has the lenient gear and nothing else, no strict mode and no error log.
iterparse: The Streaming Gear selectolax Doesn't Have At All
Here's a capability line, not a speed knob. selectolax only ingests a whole string — there's no incremental interface. lxml's iterparse yields elements as they close, and paired with the classic fast_iter pattern (call elem.clear() and delete preceding siblings as you go) it holds memory flat no matter how big the document gets.
I measured the memory characteristic directly — peak RSS via ru_maxrss, each subject in its own fresh process, on 300,000 <record> elements totaling about 15 MB.
| Mode | Peak RSS delta | Notes |
|---|---|---|
iterparse + clear (fast_iter) | ~1-2 MB | released as it goes; flat regardless of count |
iterparse without clear | ~386 MB | keeps references; as heavy as full load |
etree.parse (full load, anchor) | ~386 MB | known-heavy; proves the meter reads magnitude |
The bounded mode holds peak RSS delta to roughly 1-2 MB against a full load's ~386 MB — a 0.3-0.4% magnitude difference — and the first record event fires before the file is even finished reading, so it's genuinely incremental, not fake streaming. The instructive line is the middle one. Run the same iterparse loop but skip clear(), and memory climbs right back to ~386 MB, because you're holding references to everything. The win lives in clear(), not in iterparse by itself. The full-load anchor reading far higher than the bounded mode also confirms the RSS meter can actually see the magnitude gap rather than reading blind. (This memory test is one I ran in this pack — it's a footprint measurement, distinct from the borrowed timing numbers.)
The real-world version of this: a multi-gigabyte XML export that won't fit in RAM has no selectolax path whatsoever. It's lxml's streaming parser or a different language.
Namespaces: RSS, SVG, and the Default-Namespace Trap
Twelve namespace cases, covering RSS across three namespaces, SVG with a default namespace plus xlink, and default-namespace XML. All twelve passed.
lxml pulls //dc:creator/text() out of an RSS feed as exactly ["Alice", "Bob"], resolves //atom:link/@href and //content:encoded across three separate namespaces in the same document, handles //s:rect and //s:use/@xlink:href in SVG's second namespace, splits Clark-notation {uri}local names with QName, and introspects via nsmap. This is the maintained-and-documented behavior, and it's a whole dimension selectolax doesn't touch, because selectolax is HTML5-only and doesn't process arbitrary XML namespaces.
There is one documented trap worth committing to memory. XPath has no concept of a default namespace. Point //book at a document declaring xmlns="urn:..." and you get zero hits — the empty prefix is undefined for XPath, as the lxml docs spell out. You have to bind an artificial prefix (//c:book with namespaces={"c": "urn:..."}, which found all three) or fall back to //*[local-name()='book'] (also three). Not a bug — that's the XPath spec, faithfully implemented. It just surprises everyone exactly once.
Real Dirty Pages: Fidelity on 11 Actual Scrapes
Synthetic tests are clean; the web is not. I reused eleven real captured pages from the selectolax pack's fixture set (as-of 2026-07-10, read-only) and put lxml.html through them with lxml as the subject.
| Fixture | Size | Links | libxml2 recovered errors | Strict XML |
|---|---|---|---|---|
| news_bbc.html | 398 KB | 255 | 4 | ok |
| docs_mdn_array.html | 243 KB | 508 | 0 | raised |
| wiki_scraping.html | 227 KB | 460 | 0 | raised |
| gov_whitehouse.html | 289 KB | 154 | 0 | raised |
| oldstyle_craigslist.html | 561 KB | 351 | 0 | raised |
| forum_reddit.html | 129 KB | 318 | 0 | raised |
| docs_python.html | 80 KB | 341 | 2 | raised |
| ecommerce_books.html | 51 KB | 94 | 0 | raised |
| news_hackernews.html | 35 KB | 229 | 0 | raised |
| ecommerce_webscraper_allinone.html | 16 KB | 35 | 0 | raised |
| spa_quotes_js.html | 6 KB | 5 | 0 | raised |
All eleven parsed with lxml.html, and the link, heading, and image counts matched the reused lxml counts from the selectolax pack on all eleven — cross-check true. That agreement is what tells me the reuse is legitimately apples-to-apples and not two different measurements wearing the same label.
The side finding: the strict XML parser raised on ten of the eleven pages. Real web pages are overwhelmingly not well-formed XML, which is exactly why libxml2's HTML recovery mode exists to eat them. The lone exception was BBC News, rendered by Next.js and well-formed enough to survive strict XML parsing. Not everything labeled "HTML" needs the recovery path.
One counting note that's easy to trip on. On docs_python.html, //a[@href] (attribute-existence) counted 343, while the selectolax pack's if n.get("href") (truthy value) counted 341. The two extra are empty href="" links. That's a counting-convention difference — attribute exists versus attribute is non-empty — not an lxml behavior difference, and the counts reconcile once you align the predicate. Worth knowing when you scrape: whether empty hrefs count is your filter's choice, not the parser's.
The Depth Limit That Looks Like a Bug (But Isn't)
The selectolax pack had recorded lxml dropping the deepest content on 1,000- and 5,000-level nested <div> markup, and framed it as "lxml silently loses the deepest content." I wanted the mechanism, so I ran the default parser against huge_tree=True.
| Requested depth | Default parser reaches | huge_tree=True reaches |
|---|---|---|
| 300 | 253 (drops rest) | 299 (recovered) |
| 1000 | 253 (drops rest) | 999 (recovered) |
| 5000 | 253 (drops rest) | 2045 (still drops) |
The default parser truncates at about 253 levels and silently drops anything deeper. That's not a bug — it's libxml2's DoS defense, a roughly 256-level nesting cap that stops a hostile document from blowing the stack, and it's documented in the lxml launchpad thread on XML_PARSE_HUGE. Set huge_tree=True and depths 300 and 1,000 come back completely. Depth 5,000, though, only reaches 2,045 even with huge_tree on — there's a second, harder libxml2 recursion ceiling above the configurable one, and huge_tree doesn't clear it.
So the action item is concrete: when you parse deep markup from a source you trust, use lxml.html.HTMLParser(huge_tree=True). What this pack adds on top of the reused observation is the mechanism (a safety cap, not data corruption), the fix (huge_tree), and the fact that there's a second ceiling the fix doesn't reach.
Read/Write DOM, Serialization, Encoding
lxml is a full read/write tree, not a read-only extractor, and I verified the editing surface case by case. All eight DOM operations passed: SubElement, insert, remove, replace, strip_tags (drop tags, keep their text), strip_elements (drop tags and their text), drop_tree (an lxml.html exclusive), and the text/tail dual-slot model that trips up newcomers — in <p>head<b>bold</b>tail</p>, p.text is "head", b.text is "bold", and b.tail is "tail".
Serialization passed five for five: tostring in XML and HTML modes (HTML correctly leaves void elements un-self-closed), pretty_print, C14N canonicalization (method="c14n", another lxml exclusive), and a clean round-trip.
Encoding is where lxml quietly separates itself. Feed it non-UTF-8 bytes — "<p>café éè</p>".encode("latin-1") through lxml.html.fromstring — and it recovers café éè intact, no U+FFFD replacement characters, no dropped bytes. That directly reproduces its role as the "clean reference" in the selectolax pack, where the same input silently corrupted under the other two engines (Lexbor produced replacement characters, Modest dropped bytes outright). lxml's libxml2-backed charset detection is simply steadier here.
The flip side is strictness about how you declare an encoding. encoding="latin-1" in an XML declaration raises XMLSyntaxError: Unsupported encoding: latin-1, while the IANA-canonical encoding="ISO-8859-1" parses fine and returns café. libxml2 only accepts canonical encoding names, not aliases — a detail documented back in launchpad #613302. Annoying if you don't know it, trivial once you do.
Last, node lifecycle. I ran three stale-handle scenarios in isolated subprocesses (a hard crash would show as a non-zero exit): holding a node after its tree is garbage-collected, reading a handle after drop_tree(), and using a node after remove(). No segfaults in any of them — lxml keeps a node's reference to its tree alive to prevent use-after-free. Same clean bill of health selectolax got on this test.
Speed and Memory (Borrowed, and Honest About It)
Everything in this section is reused from the selectolax pack, as-of 2026-07-13. This pack produced zero timing numbers of its own, and I'd rather say that twice than have you think I re-timed anything.
| Dimension | lxml value | Reading |
|---|---|---|
| Pure parse p50 (10 MB) | 77.9 ms | ~33-34% faster than selectolax-Lexbor |
| Full parse + extract p50 (1 MB / 10 MB) | 14.18 ms / 172.9 ms | roughly even with Lexbor at small sizes |
| 100k-node CSS throughput | 3,002,646 nodes/s | fastest tier of the three C engines |
| 10 MB RSS delta | 128.9 MB | leanest of six parsers, ~1.7x leaner than BeautifulSoup |
| Import cold start | 14.1 ms | ~2.3x faster than parsel-style imports |
The pure-parse and throughput numbers are strong, and lxml is the most memory-frugal of the six parsers measured. The threading picture needs a caveat, though. Reused data shows a 4-thread wall-clock speedup of just 1.21x, marked inconclusive — but that's the shared-default-parser path. The lxml FAQ is explicit that the GIL is released during parsing only when each thread uses its own parser (or a copied default); a shared parser serializes access. I structurally verified the API surface for doing it right (XMLParser.copy() exists, get/set_default_parser exist, XPathEvaluator carries an internal lock), but I did not measure the per-thread-parser speedup — that would be a new timing measurement, and this pack doesn't produce those. So read "1.21x" as "under the naive shared path," not as lxml's threading ceiling.
And one asterisk on all of it: these are single-platform numbers, macOS arm64. The claim that lxml's pure parse beats Lexbor runs against the usual consensus that the Lexbor-backed parser is fastest, so it genuinely wants a Linux x86_64 recheck before anyone treats it as settled.
Licensing: The Boring Win
lxml ships under BSD-3-Clause, and the C libraries it bundles — libxml2 and libxslt — are both MIT. That's a fully permissive chain with no copyleft anywhere in it, which matters the moment you redistribute. For contrast, the selectolax wheel bundles LGPL-2.1 Modest and Apache-2.0 Lexbor, so lxml is the cleaner story for shipping in a closed product.
There's a practical install upside too: lxml publishes prebuilt wheels that statically link libxml2 and libxslt, so pip install lxml generally needs no system libxml2 and no compiler on your box — a different experience from building it from source.
Where lxml Fits — and Where an AI Extraction Layer Takes Over
Time to be clear about the boundary, because it's an easy category error. lxml is a parsing library. It hands you a tree and a superb query engine, and everything around that tree is still your job: fetching the page, rendering JavaScript, getting past anti-bot defenses, writing and maintaining the XPath, and structuring the result. That's a different layer from a hosted extraction service, and the two aren't rivals so much as neighbors.
For a developer who'd rather not own the fetch-render-select-maintain stack, that upper layer is where something like Thunderbit lives — and for this audience it's the API, MCP server, and CLI, not the browser extension. The Thunderbit Open API exposes POST /distill to turn a page into clean Markdown and POST /extract to pull structured data against a JSON Schema, with a renderMode switch and batch jobs for volume. The same engine is available as an MCP server (thunderbit_suggest_fields, thunderbit_distill, thunderbit_extract) for agents and coding assistants, and as a CLI you can run straight from a terminal via npx @thunderbit/thunderbit-cli. It handles JS rendering, anti-bot, and CAPTCHAs out of the box and returns schema-matched JSON — which is the layer above parsing, not a replacement for it.
Try Thunderbit for Web Data Extraction
The framing is a simple fork. Reach for lxml when you own the pipeline and want surgical XPath control over a tree you understand. Reach for an AI extraction API when you'd rather not maintain selectors and rendering at all. Plenty of real systems use both — lxml for the structured feeds they control, an extraction service for the messy long-tail pages they don't.
What This Review Did Not Test
This is a provisional review, not a final scorecard, so here's what it doesn't cover.
All timing and memory figures are reused, single-platform (macOS arm64, Python 3.14), and inherit that pack's caveats — the "lxml is faster at pure parsing" result is counter-consensus and needs a Linux x86_64 recheck. The per-thread-parser threading speedup is untested (it'd require new timing). I measured iterparse memory on 300k records but not GB-scale real XML, not iterparse on HTML versus XML, and not a multi-hour soak. lxml's XSLT 1.0, its RelaxNG / XMLSchema / DTD validation, and EXSLT extensions are entirely untested here — a large capability surface, but beyond the parsing-and-selection core. I observed the second depth ceiling at 2,045 but didn't pin down libxml2's exact recursion constant. Only stable 6.1.1 was tested, not the 7.0.0 alpha. Windows, source builds, and the free-threaded 3.14t build are all untested. And within XPath itself, I covered built-in functions but not XPath variables, custom Python extension functions, or precompiled etree.XPath object reuse.
The Verdict
lxml is not the fast new thing, and that's exactly the recommendation. It's a two-decade-old libxml2 binding with a full XPath 1.0 engine no mainstream Python alternative matches, three predictable gears of parsing strictness with an error log in the middle, a real streaming parser for documents that don't fit in memory, correct multi-namespace and encoding handling, and a fully permissive license. The couple of sharp edges — the ~253-level depth cap and the shared-parser threading number — are documented, configurable, and now explained.
If you own your scraping pipeline and lean on XPath, lxml is still the parser you reach for. If you'd rather not maintain selectors and rendering, that's what an AI extraction layer like the Thunderbit API, MCP, and CLI is for — a clean division of labor, not a competition. Either way, treat these numbers as provisional and re-check the timing on your own platform before you quote it in a design doc.
Try Thunderbit for Web Data Extraction Get Started Free
FAQs
Is lxml a web scraper?
No. lxml is a parser and serializer — a Python binding to libxml2/libxslt that turns markup into an editable, queryable tree. It doesn't fetch pages, render JavaScript, or handle anti-bot defenses; you supply the request layer (via requests, httpx, a headless browser, or a scraping service) and hand the bytes to lxml.
When should I use lxml instead of BeautifulSoup or selectolax? Reach for lxml when you need XPath. BeautifulSoup can actually use lxml as its backend parser but exposes no native XPath, and selectolax is CSS-only and faster in its narrow niche. If your selection logic needs text-content filtering, parent or ancestor navigation, attribute/text-node extraction, or count predicates, lxml's XPath engine is the only mainstream Python option that expresses them directly.
Why does lxml silently drop deeply nested content?
Its default parser caps nesting at about 253 levels — libxml2's DoS defense against hostile documents, not a bug. Set huge_tree=True (for example lxml.html.HTMLParser(huge_tree=True)) and it recovers depths of 300 and 1,000 fully. Note a second, harder recursion ceiling around 2,045 levels that huge_tree doesn't clear.
Does lxml release the GIL for multithreaded parsing? Only under the right conditions. The lxml FAQ states the GIL is released during parsing when each thread uses its own parser or a copied default parser; a shared parser serializes access instead. The reused 4-thread speedup of 1.21x reflects the naive shared-parser path, not the per-thread-parser ceiling, which wasn't measured here.
Is lxml still maintained in 2026? Yes. Stable release 6.1.1 shipped on 2026-05-18, the repository was last pushed on 2026-07-02, and there's a 7.0.0 alpha in progress. With roughly 3,000 GitHub stars and an actively maintained libxml2 underneath, it remains a current, well-supported library rather than a legacy one.
