# Thunderbit Open API

> Production-grade web extraction infrastructure for AI applications. Distill any URL into clean Markdown, extract structured fields with JSON Schema, and batch up to 100 URLs with webhooks.

## Get Started

- [Introduction](http://localhost:3000/docs/introduction): Production-grade web extraction infrastructure for LLM and AI apps. Markdown distillation, JSON extraction, and async batch jobs from one Open API.
- [Quickstart](http://localhost:3000/docs/quickstart): Make your first Thunderbit API call in 60 seconds. Authenticate with a Bearer token, send a URL, get clean Markdown back. cURL, Python, and Node examples.

## Guides

- [Anti-bot & JS Rendering](http://localhost:3000/docs/guides/anti-bot-handling): Anti-bot defenses and JS rendering the Thunderbit web scraping API handles automatically — and the cases (CAPTCHAs, login walls) you must plan for.
- [Batch Job Lifecycle](http://localhost:3000/docs/guides/batch-lifecycle): Sync versus async patterns, batch job states (queued, running, completed, failed, cancelled), polling cadence, partial results, and webhook notifications.
- [Best Practices](http://localhost:3000/docs/guides/best-practices): Thunderbit web scraping API production tips: concurrency, retries with backoff, idempotency, error handling, schema design, and batch sizing.
- [Distill vs Extract](http://localhost:3000/docs/guides/distill-vs-extract): When to use Markdown distillation vs. JSON extraction with the Thunderbit web scraping API. Decision matrix for RAG, agents, KBs, and CMS imports.
- [FAQ](http://localhost:3000/docs/guides/faq): FAQ for the Thunderbit web scraping API: pricing, rate limits, schemas, robots.txt, regional access, and handling JS-heavy or login-walled sites.
- [Output Formats](http://localhost:3000/docs/guides/output-formats): Nine output formats from the Thunderbit web scraping API: Markdown, typed JSON, metadata, HTML, links, images, AI summary, Q&A, and highlights.
- [Rate Limits](http://localhost:3000/docs/guides/rate-limits): Thunderbit Open API rate limits: per-minute and per-day quotas, response headers, and recommended 429 backoff strategy for production web scraping.
- [Render Modes](http://localhost:3000/docs/guides/render-modes): Three render modes for the Thunderbit web scraping API: none, basic, and full JS rendering. Pick the right one for static, dynamic, and SPA pages.
- [Schema Design](http://localhost:3000/docs/guides/schema-design): Design JSON Schemas Thunderbit extraction API can fill reliably. Field naming, types, nesting, optional fields, and patterns that boost accuracy.
- [Units](http://localhost:3000/docs/guides/units): How Thunderbit units work: per-endpoint cost, when units are deducted, refunds for cancelled or failed pages, and how to monitor usage in production.
- [Webhooks](http://localhost:3000/docs/guides/webhooks): Configure webhooks for the Thunderbit web scraping API to receive async batch job completion alerts. Payload, retries, and signature verification.

## Recipes

- [Agent Read-URL Tool](http://localhost:3000/docs/recipes/agent-read-url): Give an LLM agent a 'read this URL' tool with the Thunderbit web scraping API. Drop-in function def for OpenAI tools, Claude, and agent loops.
- [Docs to llm.txt](http://localhost:3000/docs/recipes/docs-to-llm): Convert any documentation site into one LLM-ready Markdown file using the Thunderbit web scraping API. Crawl, distill, and concatenate in batch.
- [Job Listing Tracker](http://localhost:3000/docs/recipes/job-listing-tracker): Track new job listings with the Thunderbit web scraping API: schedule batch extracts, diff results, and notify when new postings appear.
- [Lead Enrichment](http://localhost:3000/docs/recipes/lead-enrichment): Enrich CRM leads from company websites with the Thunderbit web scraping API. Batch-extract typed JSON for contacts, tech stack, and firmographics.
- [News Aggregator](http://localhost:3000/docs/recipes/news-aggregator): Aggregate headlines from multiple news sources with the Thunderbit web scraping API. Two batch passes: discover article URLs, then extract content.
- [Price Monitoring](http://localhost:3000/docs/recipes/price-monitoring): Track competitor pricing on a schedule with the Thunderbit web scraping API. Batch-extract JSON, diff against history, and trigger price alerts.
- [RAG Knowledge Base](http://localhost:3000/docs/recipes/rag-knowledge-base): Build a RAG vector store from any documentation site using the Thunderbit web scraping API. Batch-distill to Markdown, chunk, embed, and index.
- [Search and Distill](http://localhost:3000/docs/recipes/search-and-distill): Run a Google search and Distill the top results in one call with the Thunderbit web scraping API. Markdown plus AI summary, Q&A, and highlights.

## SDKs

- [Bash / cURL](http://localhost:3000/docs/sdks/bash): Bash and cURL recipes for the Thunderbit web scraping API: shell one-liners, jq pipelines for distill and extract, and ad-hoc batch jobs.
- [Dart / Flutter](http://localhost:3000/docs/sdks/dart): Dart web scraping API client patterns for the Thunderbit Open API. Idiomatic async/await examples for Flutter apps and Dart server projects.
- [.NET / C#](http://localhost:3000/docs/sdks/dotnet): .NET / C# web scraping API client patterns for the Thunderbit Open API. Idiomatic HttpClient examples for distill, extract, and batch jobs.
- [Elixir](http://localhost:3000/docs/sdks/elixir): Elixir web scraping API client patterns for the Thunderbit Open API. Idiomatic Phoenix and OTP examples for distill, extract, and batch jobs.
- [Go](http://localhost:3000/docs/sdks/go): Go web scraping API client patterns for the Thunderbit Open API. Idiomatic net/http examples for distill, extract, and async batch jobs.
- [Java](http://localhost:3000/docs/sdks/java): Java web scraping API client patterns for the Thunderbit Open API. Idiomatic HttpClient and Jackson examples for distill, extract, and batch.
- [Kotlin](http://localhost:3000/docs/sdks/kotlin): Kotlin web scraping API client patterns for the Thunderbit Open API. Idiomatic coroutine examples for Android, JVM, and Ktor projects.
- [Node.js / TypeScript](http://localhost:3000/docs/sdks/node): Node.js / TypeScript web scraping API client patterns for the Thunderbit Open API. Idiomatic fetch and undici examples for distill and extract.
- [SDKs Overview](http://localhost:3000/docs/sdks/overview): Thunderbit web scraping API client recipes in 13 languages: Python, Node, Go, Ruby, Java, PHP, .NET, Rust, Swift, Kotlin, Elixir, Dart, Bash.
- [PHP](http://localhost:3000/docs/sdks/php): PHP web scraping API client patterns for the Thunderbit Open API. Idiomatic Guzzle and cURL examples for distill, extract, and batch jobs.
- [Python](http://localhost:3000/docs/sdks/python): Python web scraping API client patterns for the Thunderbit Open API. Idiomatic requests / httpx examples for distill, extract, and batch jobs.
- [Ruby](http://localhost:3000/docs/sdks/ruby): Ruby web scraping API client patterns for the Thunderbit Open API. Idiomatic Net::HTTP and Faraday examples for distill, extract, and batch.
- [Rust](http://localhost:3000/docs/sdks/rust): Rust web scraping API client patterns for the Thunderbit Open API. Idiomatic reqwest and serde examples for distill, extract, and batch jobs.
- [Swift](http://localhost:3000/docs/sdks/swift): Swift web scraping API client patterns for the Thunderbit Open API. Idiomatic URLSession examples for iOS, macOS, and server-side Swift apps.

## Integrations

- [AutoGen](http://localhost:3000/docs/integrations/autogen): Connect Microsoft AutoGen agents to the Thunderbit web scraping API. Wrap distill and extract as tools for autonomous web research workflows.
- [CrewAI](http://localhost:3000/docs/integrations/crewai): Add the Thunderbit web scraping API to CrewAI agents as a custom tool. Distill and extract any URL inside crew tasks for research and enrichment.
- [Dify](http://localhost:3000/docs/integrations/dify): Add the Thunderbit web scraping API to Dify as a custom tool. Import the OpenAPI schema once, then call distill and extract from any workflow.
- [Flowise](http://localhost:3000/docs/integrations/flowise): Wrap the Thunderbit web scraping API as a Flowise Custom Tool. Add web distill and structured extraction to chains and agents — no code.
- [Haystack](http://localhost:3000/docs/integrations/haystack): Add the Thunderbit web scraping API to Haystack as a custom component. Distill and extract web pages inside RAG ingestion pipelines.
- [LangChain](http://localhost:3000/docs/integrations/langchain): Use the Thunderbit web scraping API in LangChain as a document loader or agent tool. Distill URLs to Markdown or extract JSON inside any chain.
- [LlamaIndex](http://localhost:3000/docs/integrations/llamaindex): Plug the Thunderbit web scraping API into LlamaIndex as a reader or tool. Distill URLs to Markdown for indexing or extract typed JSON inline.
- [Make.com](http://localhost:3000/docs/integrations/make): Build no-code Thunderbit web scraping API scenarios in Make.com using HTTP and Webhooks modules. Distill or extract any URL and route the output.
- [Mastra](http://localhost:3000/docs/integrations/mastra): Wrap the Thunderbit web scraping API as a Mastra tool. Use distill and extract inside TypeScript Mastra agents and durable workflows.
- [Model Context Protocol (MCP)](http://localhost:3000/docs/integrations/mcp): Expose the Thunderbit web scraping API as an MCP server so Claude, Cursor, Windsurf, and Claude Code can read any URL with one tool call.
- [n8n](http://localhost:3000/docs/integrations/n8n): Build no-code Thunderbit web scraping API workflows in n8n using HTTP and Webhook nodes. Distill, extract, and route results in any pipeline.
- [Integrations Overview](http://localhost:3000/docs/integrations/overview): Connect the Thunderbit web scraping API to LangChain, LlamaIndex, n8n, Zapier, Make, Pipedream, MCP, AutoGen, CrewAI, Mastra, and more in minutes.
- [Pipedream](http://localhost:3000/docs/integrations/pipedream): Call the Thunderbit web scraping API from a Pipedream Code step in Node.js or Python. Distill or extract URLs and pipe results downstream.
- [Vercel AI SDK](http://localhost:3000/docs/integrations/vercel-ai-sdk): Wrap the Thunderbit web scraping API as a Vercel AI SDK tool. Type-safe distill and extract for streamText and generateText agent applications.
- [Zapier](http://localhost:3000/docs/integrations/zapier): Trigger the Thunderbit web scraping API from any of Zapier's 7000+ apps. Send a URL via Webhooks by Zapier and route distill or extract output.

## API Reference

- [API Overview](http://localhost:3000/docs/api-reference/overview): Thunderbit web scraping API reference: base URL, Bearer auth, JSON envelope, error format, rate limits, HTTP status codes, and full error code list.
- [Batch Distill Status](http://localhost:3000/docs/api-reference/endpoints/batch-distill-status): Poll a distillation batch by job ID to get state, progress, partial results, per-URL failures, and the final Markdown output once the job completes.
- [Batch Distill](http://localhost:3000/docs/api-reference/endpoints/batch-distill): Submit up to 100 URLs per job for asynchronous Markdown extraction. Poll status, cancel running jobs, and retrieve results when each batch completes.
- [Batch Extract Status](http://localhost:3000/docs/api-reference/endpoints/batch-extract-status): Poll an extraction batch by job ID for state, progress, per-URL results, partial JSON output, and failure reasons once the asynchronous job completes.
- [Batch Extract](http://localhost:3000/docs/api-reference/endpoints/batch-extract): Submit up to 50 URLs per job for asynchronous structured JSON extraction. Define your schema once, apply it across the batch, poll for typed results.
- [Cancel Batch Distill](http://localhost:3000/docs/api-reference/endpoints/cancel-batch-distill): Cancel a running or queued Markdown distillation batch by job ID. Stop further URL processing immediately and free up units that were not yet consumed.
- [Cancel Batch Extract](http://localhost:3000/docs/api-reference/endpoints/cancel-batch-extract): Cancel a running or queued JSON extraction batch by job ID. Stop further URL processing immediately and free up units that were not yet consumed.
- [Distill Single Page](http://localhost:3000/docs/api-reference/endpoints/distill): POST a URL to convert any web page into clean, LLM-optimized Markdown. Automatic JS rendering, ad and nav stripping, no scraper or browser to maintain.
- [Extract Structured Data](http://localhost:3000/docs/api-reference/endpoints/extract): Extract typed JSON from any webpage using your own JSON Schema. No CSS selectors or XPath — the AI handles field mapping and renders JS automatically.
- [Search and Distill](http://localhost:3000/docs/api-reference/endpoints/search): POST a Google search query and get the top results Distilled in one call. Each result returns the search snippet plus optional Markdown, summary, question/answer, highlights, links, and images.

## Models

- [Error](http://localhost:3000/docs/models/error): Thunderbit Open API error response model: envelope shape, error.code reference, HTTP status mapping, and retryable vs non-retryable categories.