Colly Review: Static HTML, Direct JSON, and the Browser Boundary

I built a fixture site to test the part of Colly that a page-count or speed reputation does not answer: whether its callbacks extract the expected records, route an HTTP error, follow a bounded graph, and expose content delivered outside rendered HTML. This was a correctness-and-boundary review, not a throughput benchmark.

On the controlled fixtures it extracted every expected static record, routed one 500 response to OnError, and visited 17 URLs in the configured depth-limited graph. It returned zero target elements on two pages whose elements appeared only after JavaScript execution. A directly accessible JSON endpoint remained usable without a browser, an important distinction from rendering the client UI.

What Colly actually is

Colly calls itself an "elegant scraper and crawler framework for Golang," and that description is doing more work than it looks. It's a Go library — around 25,300 GitHub stars and 1,850 forks — licensed Apache-2.0. It is not a CLI you download and point at a URL. You write Go, import Colly, register a few callbacks, and compile the whole thing into one executable.

The mental model is event-driven. You attach handlers to a Collector: OnHTML runs extraction logic for matching CSS selectors, OnResponse supplies the raw response body, and OnError handles request failures. Link handlers call Visit() on discovered URLs, while MaxDepth bounds traversal. The target host needs no separately installed Go runtime or browser for these HTTP-only paths; whether the executable is fully static depends on build flags and CGO use, which this test did not record.

Key features, and how they run under the hood

The callback model is the thing to understand, because it's why Colly feels different from a request-and-parse script. Three callbacks carried every test I ran.

OnHTML(selector, handler) is the workhorse. Register it against .product or article p and Colly invokes your handler once per matched element as it parses the DOM. This is where structured extraction lives, and it reads cleanly — you're describing what to grab, not writing a parse loop.

OnResponse(handler) sits one level lower and gives you the raw bytes. When a target hands back JSON instead of HTML, you skip the DOM entirely and unmarshal the body yourself. That single callback is why Colly handled a JSON API cleanly in my testing without any HTML parsing at all.

OnError(handler) handles request failures and can expose a response status to caller code. In this test one fixture response with status 500 reached the registered callback. Retries, timeouts, DNS failures, connection resets, callback panics, persistence, and alerting were not tested.

On top of the callbacks sit two operational features. MaxDepth bounds link traversal according to Colly's depth semantics. A compiled Go executable also avoids a separately installed language runtime on the target host. This run did not record build flags or CGO status, so it does not claim that every resulting binary is fully static.

Setup: the required Go toolchain

The dependency story is short but real, so here it is before you install anything. The machine I tested on had no Go installed, and Colly is a Go library — so step zero was putting a Go toolchain on the box (I installed Go 1.26.5 via Homebrew). If your team doesn't already live in Go, that's the friction: not Colly itself, but the language environment it needs before a single line compiles.

Once Go was there, go get github.com/gocolly/colly/v2 resolved to v2.3.0. The tested paths required no browser or headless Chrome.

One version surface can confuse readers. The Go module resolved to v2.3.0 (published December 2025), while the newest entry visible in GitHub's Releases UI was v2.2.0 (March 2025) when checked. The distinction is module/repository version versus GitHub Release entry, not module versus Git tag. I tested v2.3.0.

Hands-on: extraction and operating boundaries

I ran Colly against a self-contained fixture server (Go's httptest) plus two public demo sites. The current benchmark directory and results/colly-test-summary.json expose artifacts, but both links follow a moving branch. The article does not provide a tested commit, exact command, build flags, or fixture seed, so this is not yet an immutable reproduction recipe.

On the controlled static fixtures, the configured selectors produced 12 of 12 expected product records and all three expected article paragraphs. The direct JSON response never touched an HTML parser: OnResponse supplied the body and the harness decoded all eight expected items. The single 500 fixture reached OnError with its status exposed and did not crash that run; it does not establish unattended reliability. On the public Books to Scrape page, the selector returned 20 products as a public-site smoke test.

For traversal, the collector was configured with MaxDepth(2) using the harness's seed-depth convention and visited 17 URLs in the fixture graph. The result is crawl coverage, not speed. The observed trace—not a broader claim about arbitrary graphs—is in results/local_crawl_graph.json.

Colly does not run JavaScript. The JavaScript-rendered fixture produced 0 target cards, and the public Quotes to Scrape JS page produced 0 target quotes. If elements exist only after browser execution and no accessible backing endpoint supplies them, the HTTP-only path cannot see those elements as rendered DOM. Pair it with a renderer, or call the backing endpoint directly when one is available, as the JSON fixture demonstrates.

I did not stress the async collector, rate-limiting or politeness configuration, proxy rotation, retries, or queue and storage backends. No elapsed time, throughput, concurrency, CPU, memory, target latency, or comparison baseline was measured. This article therefore makes no speed or unattended-reliability claim.

How to interpret the fixture results

The three successful content paths exercise different contracts. The catalog and article cases test CSS selection over server-returned HTML. Their denominators are fixture expectations written before extraction: twelve product records and three article paragraphs. Reporting those as “expected records extracted” is deliberate. The run does not define fuzzy matching, duplicate handling, partial-field tolerance, or a corpus-wide recall metric, so the result should not be promoted into general extraction accuracy.

The JSON case bypasses DOM selection. Colly receives the response bytes through OnResponse, and the harness performs the JSON decoding. This is why “Colly does not render JavaScript” does not mean that every client-backed site is unavailable. If the data source used by the client is a directly callable endpoint and the request is reproducible outside the browser, the HTTP crawler may still be sufficient. Authentication, generated signatures, browser-only state, and anti-bot controls can change that answer; none was exercised here.

The 500 route tests dispatch, not recovery. It shows that the registered OnError callback received that fixture response and its status. A production crawler still needs an explicit policy for retryable codes, backoff, terminal failures, persistence, and alerting. The test supplies no evidence for those choices, and “the callback fired” should not be read as “the job can be trusted unattended.”

The 17-URL graph is similarly narrow. It confirms the visited set produced by this fixture, this seed convention, and MaxDepth(2). It does not establish pages per second, fairness across hosts, memory growth, or behavior on cycles and duplicate URL forms. Those require separate workload and queue tests.

A selection checklist grounded in this run

Start by looking at the response Colly actually receives. If the required fields are present in server-returned HTML, use OnHTML and validate field counts or required keys before accepting a record. If the response is JSON, handle the body through OnResponse and validate its schema. If the HTML is only an application shell, inspect whether an accessible backing request contains the data before adding a browser.

When browser execution is necessary, treat it as another component rather than expecting a Colly flag to enable rendering. The browser must establish readiness, expose the rendered content or backing responses, and pass data into the rest of the pipeline. This review did not test such an integration.

For deployment, record the Go version, module version, build flags, CGO status, exact command, fixture seed, and repository commit. Those details are missing from the current publication links and are the difference between inspectable artifacts and a durable reproduction. For operations, add a failure matrix and measure the workload you actually care about before calling the system fast or reliable.

Pros and cons

Pros:

• Extracted 12/12 expected catalog products and 3/3 expected article paragraphs via OnHTML.
• Clean JSON handling through OnResponse, no DOM parsing needed — 8/8 API items.
• The tested 500 response reached OnError with the status exposed.
• Depth-limited crawl reached 17 pages from a single collector.
• Compiles into a Go executable; the target needs no separately installed Go runtime for the tested paths.
• Permissive Apache-2.0 license.

Cons:

• No JavaScript execution — client-rendered content returns 0, full stop.
• Requires a Go toolchain; non-Go teams pay that setup cost before writing any scraper.
• The tested module (v2.3.0) is ahead of the newest GitHub Release entry observed (v2.2.0).
• Output is your own code — Colly hands you callbacks, not a built-in dataset/feed exporter like Scrapy's.
• Async, rate-limiting, proxy, and queue backends exist but went untested here; throughput and scale remain unmeasured.

Who it's for — and who should skip it

Colly fits if you already write Go and target server-rendered HTML or directly accessible JSON. The callback model separates structured matches, raw payloads, and request failures. A compiled executable also avoids a separately installed language environment on the target machine, although fully static linking was not verified here.

Add a renderer when required target elements appear only after browser execution and no usable backing endpoint exists. A direct JSON endpoint can still be requested without rendering. Colly is also a weaker fit for teams that do not want a Go toolchain or that want an extraction service to own schema shaping and selector maintenance.

Alternatives, including where Thunderbit fits

Colly is open-source software you run yourself. It has no vendor usage fee, but compute, bandwidth, proxies, storage, observability, and engineering remain your costs. You own request behavior, parsing callbacks, crawl logic, and browser integration if a target needs rendering.

A managed extraction service moves some of those responsibilities to a vendor. We build Thunderbit, but did not test it against these fixtures, so this article makes no rendering, anti-bot, quality, latency, or cost comparison. The supported distinction is ownership: Colly exposes HTTP responses and callbacks in your Go process; a managed service can own acquisition and schema shaping for a per-call fee.

Related benchmark reviews: the full open-source scraper comparison, Scrapy's Python crawler review, and Scrapling's adaptive selector review.

Try Thunderbit for Web Data Extraction

Verdict

Colly is a strong candidate for Go teams targeting server-rendered HTML or direct JSON and willing to own their extraction code. The fixtures support expected-record extraction, one bounded crawl trace, and one observed 500 callback—not speed, scale, or unattended reliability. Browser-rendered DOM requires another path unless the underlying data endpoint can be called directly.

Try Thunderbit for Web Data Extraction Get Started Free

FAQs

Did this review measure Colly's speed? No. It measured expected-record extraction, direct JSON handling, one error callback, and coverage of a fixture crawl graph. It did not measure elapsed time, throughput, concurrency, CPU, memory, or a comparison baseline.

Can Colly scrape JavaScript-rendered pages? Colly does not execute the page's JavaScript. The tested HTTP path therefore found no target elements that appeared only in the rendered DOM. It can still request an accessible backing JSON endpoint directly, as the JSON fixture shows. Use a renderer when execution is required and no reproducible backing request supplies the data.

Do I need to know Go to use Colly? Yes. Colly is a Go library, not a standalone CLI — you import it, register callbacks (OnHTML, OnResponse, OnError), and compile. The machine I tested on had no Go, so setup started with installing a Go toolchain (1.26.5). If your team isn't already in Go, that environment is the real setup cost.

Why does the version I install not match Colly's latest GitHub release? The Go module resolved to v2.3.0 (December 2025), while the newest GitHub Release entry observed was v2.2.0 (March 2025). I tested v2.3.0; this is a distinction between version surfaces, not evidence of a broken install.

Is Colly free for commercial use? It's Apache-2.0, which is permissive and commercially friendly. As always, confirm the current license on the repo before you build on it.

Before production adoption, add tests that match the operating risk rather than extending the fixture result by analogy. Time repeated crawls on representative targets, record CPU and peak memory, exercise retryable and terminal failures, and verify politeness under concurrency. If persistence matters, stop and resume a crawl while inspecting duplicate handling and queue state. If deployment simplicity matters, record the exact compiler and linker configuration and inspect the produced executable's runtime dependencies. None of those checks changes what the current fixture established; they determine whether the same library configuration fits a particular production job.