Convert a web page into clean, LLM-ready Markdown format.

• Prepare web content for RAG (Retrieval-Augmented Generation)
• Extract article content for AI processing
• Convert documentation pages to markdown
• Process dynamic web applications

• Clean markdown content with preserved structure
• Automatic removal of ads, navigation, and boilerplate
• Metadata extraction (title, description, language)
• JavaScript rendering for dynamic content
• Automatic handling of anti-bot measures

Returns markdown optimized for LLM consumption with minimal noise and maximum signal.

Request Body

• url (string) *required: The URL of the web page to distill
• timeout (number): Request timeout in milliseconds (default: 30000, max: 60000)
• waitFor (number): Time to wait (in milliseconds) after page load for dynamic content to render before extracting content
• includeTags (string[]): Only include content from these HTML tags (e.g., ['article', 'main', 'div.content'])
• excludeTags (string[]): Exclude content from these HTML tags (e.g., ['nav', 'footer', 'aside'])
• headers (object): Custom HTTP headers to send with the request

Response (200): Success response

• success (boolean):
• data (object):
  • url (string): The URL that was distilled
  • markdown (string): Clean markdown content extracted from the page
  • html (string): Raw HTML content (optional, only if requested)
  • metadata (object):
    • title (string): Page title extracted from <title> tag or Open Graph
    • description (string): Meta description or excerpt
    • language (string): Detected language code (ISO 639-1)
    • author (string): Article author if available
    • publishedDate (string): Publication date if available
    • image (string): Featured image URL from Open Graph or Twitter Card
    • sourceURL (string): Original URL (may differ from requested URL due to redirects)
    • statusCode (integer): HTTP status code of the response
    • contentLength (integer): Length of the markdown content in characters
  • links (object[]): Links found in the content

Example Request

curl 'https://open.thunderbit.com/v1/distill' \
  --header 'Authorization: Bearer YOUR_SECRET_TOKEN' \
  --header 'Content-Type: application/json' \
  --data '{"url":"https://example.com/article","timeout":30000,"waitFor":2000,"includeTags":["article","main"],"excludeTags":["nav","footer","aside"],"headers":{"User-Agent":"MyBot/1.0"}}'

Example Response

{
  "success": true,
  "data": {
    "url": "https://example.com/article",
    "markdown": "# Article Title\n\nContent...",
    "html": "<article>...</article>",
    "metadata": {
      "title": "string",
      "description": "string",
      "language": "string",
      "author": "string",
      "publishedDate": "2025-01-01T00:00:00Z",
      "image": "string",
      "sourceURL": "string",
      "statusCode": 1,
      "contentLength": 1
    },
    "links": [
      {
        "text": "string",
        "href": "string"
      }
    ]
  }
}

Distill multiple web pages simultaneously with asynchronous processing.

• Process entire website sections or categories
• Batch import content into your knowledge base
• Large-scale content migration
• Periodic content updates from multiple sources

• Asynchronous processing for high throughput
• Automatic retry on failures
• Webhook notifications when complete
• Detailed per-URL status and error reporting

Request Body

• urls (string[]) *required: List of URLs to distill, maximum 100
• timeout (number): Timeout per request in milliseconds, default 30000
• webhook (object): Webhook callback configuration, notifies when task completes
  • url (string): Webhook callback URL, must be HTTPS
  • headers (object): Custom callback headers, can be used for authentication

Response (200): Success response

• success (boolean):
• data (object):
  • id (string): Batch task ID
  • status (string):
  • total (integer):
  • completed (integer):
  • creditsUsed (integer):

Example Request

curl 'https://open.thunderbit.com/v1/batch/distill' \
  --header 'Authorization: Bearer YOUR_SECRET_TOKEN' \
  --header 'Content-Type: application/json' \
  --data '{"urls":["https://example.com/page1","https://example.com/page2"],"timeout":1,"webhook":{"url":"string","headers":{}}}'

Example Response

{
  "success": true,
  "data": {
    "id": "batch_abc123",
    "status": "processing",
    "total": 3,
    "completed": 0,
    "creditsUsed": 0
  }
}

Check the status and retrieve results of a batch distill job.

• processing: Job is currently running
• completed: All URLs have been processed
• failed: Job encountered a fatal error

• Poll every 5-10 seconds for jobs with <10 URLs
• Poll every 30-60 seconds for larger jobs
• Use webhooks for better efficiency

You can retrieve completed results while the job is still processing. The results array will contain all URLs processed so far.

Parameters

• id (string) *required: Batch task ID

Response (200): Success response

• success (boolean):
• data (object):
  • id (string):
  • status (string):
  • total (integer):
  • completed (integer):
  • creditsUsed (integer):
  • results (object[]):

Example Request

curl 'https://open.thunderbit.com/v1/batch/distill/{id}' \
  --header 'Authorization: Bearer YOUR_SECRET_TOKEN'

Example Response

{
  "success": true,
  "data": {
    "id": "batch_abc123",
    "status": "string",
    "total": 1,
    "completed": 1,
    "creditsUsed": 1,
    "results": [
      {
        "url": "string",
        "success": true,
        "markdown": "string",
        "error": {
          "code": "string",
          "message": "string"
        }
      }
    ]
  }
}

Extract structured data from web pages using AI. Define your desired output structure with JSON Schema, and our AI will intelligently extract the information.

• Extract product information from e-commerce pages
• Parse job listings into structured format
• Extract contact information and business details
• Convert news articles into structured data
• Scrape pricing tables and specifications

Use JSON Schema to define your desired output structure:

• Field types: string, number, boolean, array, object
• Field descriptions: Help the AI understand what to extract
• Required fields: Mark critical fields as required
• Nested structures: Support for complex, nested data

{
  "type": "object",
  "properties": {
    "title": {
      "type": "string",
      "description": "Product name or title"
    },
    "price": {
      "type": "number",
      "description": "Current price in USD"
    },
    "availability": {
      "type": "boolean",
      "description": "Whether the product is in stock"
    },
    "features": {
      "type": "array",
      "items": {"type": "string"},
      "description": "List of key product features"
    }
  },
  "required": ["title", "price"]
}

Request Body

• url (string) *required: The URL of the web page to extract data from
• schema (object) *required: Data structure definition in JSON Schema format
• timeout (number): Request timeout in milliseconds, default 30000

Response (200): Success response

• success (boolean):
• data (object):
  • url (string):
  • extract (object): Extracted structured data matching your schema
  • metadata (object):
    • sourceURL (string):
    • statusCode (integer):
    • extractedAt (string):
    • confidence (number): AI confidence score (0-1) for the extraction quality
    • processingTime (integer): Time taken to process in milliseconds

Example Request

curl 'https://open.thunderbit.com/v1/extract' \
  --header 'Authorization: Bearer YOUR_SECRET_TOKEN' \
  --header 'Content-Type: application/json' \
  --data '{"url":"https://example.com/product","schema":{"type":"object","properties":{"name":{"type":"string"},"price":{"type":"number"}},"required":["name","price"]},"timeout":1}'

Example Response

{
  "success": true,
  "data": {
    "url": "https://example.com/product",
    "extract": {
      "name": "iPhone 15 Pro",
      "price": 999,
      "currency": "USD"
    },
    "metadata": {
      "sourceURL": "string",
      "statusCode": 1,
      "extractedAt": "2025-01-01T00:00:00Z",
      "confidence": 1,
      "processingTime": 1
    }
  }
}

Extract structured data from multiple URLs simultaneously using AI.

• Scrape product catalogs from multiple pages
• Extract data from search result pages
• Batch process listings or directory pages
• Collect competitive intelligence at scale

• Same schema applied to all URLs
• Parallel processing for speed
• Individual error handling per URL
• Webhook notifications available

Request Body

• urls (string[]) *required: List of URLs to extract data from, maximum 50
• schema (object) *required: Data structure definition in JSON Schema format
• timeout (number): Timeout per request in milliseconds, default 30000
• webhook (object): Webhook callback configuration, notifies when task completes
  • url (string): Webhook callback URL, must be HTTPS
  • headers (object):

Response (200): Success response

• success (boolean):
• data (object):
  • id (string):
  • status (string):
  • total (integer):
  • completed (integer):

Example Request

curl 'https://open.thunderbit.com/v1/extract/batch' \
  --header 'Authorization: Bearer YOUR_SECRET_TOKEN' \
  --header 'Content-Type: application/json' \
  --data '{"urls":["string"],"schema":{},"timeout":1,"webhook":{"url":"string","headers":{}}}'

Example Response

{
  "success": true,
  "data": {
    "id": "batch_ext_xyz789",
    "status": "string",
    "total": 1,
    "completed": 1
  }
}

Check status and retrieve extracted data from a batch extraction job.

• processing: Extraction in progress
• completed: All extractions finished
• failed: Job failed (check error details)

Each URL in the results array contains:

• Extracted data matching your schema
• Success/failure status
• Individual error messages if applicable
• Confidence scores for extraction quality

Parameters

• id (string) *required: Batch task ID

Response (200): Success response

• success (boolean):
• data (object):
  • id (string):
  • status (string):
  • total (integer):
  • completed (integer):
  • creditsUsed (integer):
  • results (object[]):

Example Request

curl 'https://open.thunderbit.com/v1/extract/batch/{id}' \
  --header 'Authorization: Bearer YOUR_SECRET_TOKEN'

Example Response

{
  "success": true,
  "data": {
    "id": "string",
    "status": "string",
    "total": 1,
    "completed": 1,
    "creditsUsed": 1,
    "results": [
      {
        "url": "string",
        "success": true,
        "extract": {},
        "error": {
          "code": "string",
          "message": "string"
        }
      }
    ]
  }
}