A Mid-Body Stall Escapes a Common requests Timeout Handler

Write the guard everyone writes:

try:
    r = requests.get(url, timeout=0.5)
except requests.exceptions.Timeout:
    retry()

Point it at a server that sends its status line and headers immediately and then stalls before the body. The read times out. The guard does not fire. The exception that comes out is ConnectionError, and ConnectionError is not a subclass of Timeout.

The same stall against httpx raises ReadTimeout, which is a TimeoutException, which the equivalent guard catches.

I went looking for how httpx differs from requests in the ways that break scrapers, expecting to write about async. Async turned out to be the least interesting thing on the list.

What I tested and how

Eight probes against a local fixture server, because a client reporting what it did is not evidence about what it did. The server counts TCP connections — incremented once per accepted socket, before any request line is parsed — and paths actually fetched. Connection reuse and redirect-following are both claims about the wire, and the wire is where they get checked.

httpx 0.28.1 with the http2 extra, requests 2.34.2, Python 3.14.2, macOS arm64. Both in one fresh virtualenv so neither inherits a footprint from the other. Raw output: httpx-probes.json.

Six predictions went into the harness before the first run and stayed there afterwards. Three landed, two were wrong, one was right about the case I thought of and missed the case that mattered. prediction-scorecard.json has the accounting.

The defaults that change under you

Redirects, socket reuse, exception outcomes, decoding, and protocol negotiation were observed in probes. Timeout API shape and requests' lack of an HTTP/2 flag are API-capability observations. httpx-probes.json.

Three of those rows will change what your code does on the day you switch, silently.

Redirects: off by default, and the server proves it

A four-hop redirect chain ending at /ok:

The five is four hops plus the destination. My prediction said four, which was arithmetic I did not check; the direction was the claim and the count is corrected here rather than quietly in the prose.

This is documented httpx behaviour and it is a defensible design — a redirect is a thing the caller might want to know about. It is also the single most likely way a migration breaks without raising anything. Your code gets a 302, response.text is empty, your parser finds no rows, and your logs say 200 OK… except they say 302, and nothing was watching the status code because with requests there was never anything to watch.

The timeout finding, which is the one I got backwards

I predicted httpx would name the phase that failed and requests would blur both into one class. It is the other way round.

Official reference: Requests timeout documentation.

Official reference: HTTPX timeout documentation.

httpx gives the same, accurate name to both. requests splits them — and splits them across the boundary that retry code is written against.

The consequence is not an inference from the class hierarchy. I ran the guard:

timeout-retry-guard.json. requests.exceptions.ConnectionError is not a subclass of requests.exceptions.Timeout; httpx.ReadTimeout is a subclass of httpx.TimeoutException.

The requests exception message says Read timed out. inside a ConnectionError. The library knows what happened. It just does not tell the type system, and the type system is what your except clause consults.

The measured case is specific: headers arrive, then body progress stops long enough to exceed the read timeout. A response that continues delivering chunks within the timeout window, including an intentional stream, can behave differently and was not tested here.

Connection pooling: the client API is the whole difference

Ten GETs, four ways, sockets counted server-side:

Identical, and worth stating because it is the most common piece of folk knowledge about this pair — that httpx pools and requests does not. Neither pools at module level. Both pool through their client object. If you are calling requests.get() in a loop today, moving to httpx.get() in a loop changes nothing about your socket churn.

HTTP/2 is explicit and requires the extra

Against one public HTTP/2 endpoint recorded in the artifact:

Official reference: RFC 9113: HTTP/2.

You need the httpx[http2] extra. I assumed plain pip install httpx would leave you with a client that quietly negotiates 1.1, and went to check before writing it down:

ImportError: Using http2=True, but the 'h2' package is not installed.
Make sure to install httpx using `pip install httpx[http2]`.

It raises at Client construction, before a single request, and the message names the fix. That is the good version of this failure, and I had it backwards (http2-extra-missing.json).

This probe establishes successful protocol negotiation on that endpoint. It does not establish a scraping-speed benefit; no matched HTTP/1.1 workload was tested.

Sequential versus concurrent fixture throughput

Twenty requests against an endpoint that sleeps 0.3 s:

The concurrent run completed in 0.357 seconds versus 6.138 seconds for the sequential run. It also opened twenty connections while the synchronous client reused one, so the experiment changes execution model and effective concurrency rather than isolating library speed.

That is the honest framing of the number. It is a measurement of concurrency against a deliberately slow endpoint, not a measurement of httpx. Any client with a working async story lands in the same neighbourhood, and against a fast endpoint the gap collapses.

The charset case I did not think to predict

I predicted that a response whose header lies — charset=iso-8859-1 on utf-8 bytes — would produce identical mojibake in both. It does. Both return Café Ubersetzung â naïve résumé where the source says Café Ubersetzung — naïve résumé.

The case I did not predict is the one that matters:

requests falls back to ISO-8859-1 when the header says nothing, while httpx defaults to utf-8. In the missing-charset fixture, the clients therefore produced different decoded text through .text; consumers using response.content retain the same original bytes.

Memory, since it is cheap to measure

Peak RSS, /usr/bin/time -l, one fresh process per cell:

These are one-process snapshots, and requests' one-GET value being slightly below its import-only value exposes the run noise. They support no directional memory conclusion; repeated samples and ranges would be required.

What this means when you pick

Finding bugs in existing requests code? Audit redirect assumptions, handlers that catch only requests.exceptions.Timeout but expect to cover a mid-body stall, and .text consumers receiving responses without a charset.

Migrating mechanically to httpx? Change exception namespaces to httpx.TimeoutException or the narrower phase classes, decide whether to enable follow_redirects, and retest decoding assumptions. The existing requests handler already misses the demonstrated mid-body case; migration does not create that particular bug.

Writing something new that fetches many URLs? httpx is a candidate when you need AsyncClient and separate connect, read, write, and pool timeouts. Those phases tell you where waiting occurred—connection establishment, response-body progress, request upload, or local pool acquisition—not why a remote host behaved that way.

Writing something small and synchronous? requests is fine and it is everywhere. The reason to move is not speed.

Whatever you pick, use the client object rather than the module-level function. That is the one change on this list that is a straight win in both libraries.

Where a managed API fits

Everything above is the fetch layer, and the fetch layer is the easy part. None of it renders JavaScript, none of it handles an anti-bot challenge, and none of it turns HTML into the rows you wanted.

Author note: Thunderbit is our managed option for URL-in rendering and extraction. It was not tested in this HTTP-client harness. Consider that category only when page acquisition or structured extraction—not HTTP client semantics—is the problem you are trying to remove.

If you are fetching ordinary pages and parsing them yourself, either client remains in scope. A managed service is a separate build-versus-buy decision, not evidence for choosing between these libraries.

For the wider field, our web scraping API roundup covers the hosted options and the open-source scraper pillar the self-hosted ones.

Try Thunderbit for Web Data Extraction

Verdict

For a new Python fetch layer that needs async concurrency, phase-specific timeouts, and a UTF-8 fallback, httpx is my default under the constraints tested here. Requests remains viable for mature synchronous code where migration risk outweighs those benefits. Proxies, retry policies, TLS fingerprinting, streaming, uploads, and realistic network variation were not tested, so this is not a universal scraping-client ranking.

The reason to be careful is the redirect default, and it is a genuine hazard precisely because it is a good design decision. Explicit is better than implicit right up until the implicit thing was load-bearing in code you already shipped.

The preregistered scorecard ended with three correct predictions, two incorrect ones, and one incomplete prediction. The useful correction was the mid-body exception class; the rest of the decision should come from the observed behavior rather than the scorecard narrative.

Try Thunderbit for Web Data Extraction Get Started Free

FAQs

Does httpx really not follow redirects? Not by default, no. The server counted one request for a four-hop chain, and the response came back as a 302. Pass follow_redirects=True per call, or set it once on the Client. This is documented and deliberate; it is still the most likely thing to break quietly in a migration, because the failure is an empty parse rather than an exception.

Is except requests.exceptions.Timeout really not enough? Not for a server that stalls after sending headers. That case raises ConnectionError, which is not a subclass of Timeout, so the guard misses it — demonstrated directly rather than inferred. Catch requests.exceptions.RequestException if you want both, and accept that you are also catching things that are not timeouts.

Is httpx faster than requests? Not meaningfully, for one request at a time — that is not what it is for. The 17.2× in this test is a measurement of twenty concurrent requests against a 0.3 s endpoint, which is a measurement of concurrency. If your workload is sequential, expect no speedup and choose on the defaults instead.

Do I need the http2 extra? Only if you want HTTP/2 — and if you set http2=True without it, httpx raises ImportError when you construct the Client, with a message telling you to install httpx[http2]. No silent downgrade to worry about. I expected one and checked instead of writing it down.

What was not tested here? Proxy behaviour, which matters a great deal for scraping and needs its own harness. Retries — httpx ships no retry logic and requests gets its from urllib3, so a fair comparison is really a comparison of two retry libraries. TLS fingerprinting, which is the axis anti-bot systems actually look at and which neither library addresses. Streaming and file uploads. And everything here is one machine, one Python version, and localhost for six of the eight probes — latency numbers from a fixture server are a measurement of the design, not of your network.