Thunderbit Open API home page
SupportAPI Console
Recipes

Agent Read-URL Tool

Drop-in "read this URL" tool for an LLM agent loop

Give your agent a single tool that turns any URL into clean Markdown. The agent calls it whenever it needs to research a page — Thunderbit handles JS, anti-bot, and content cleaning, so the agent's context stays signal-dense.

Tool definition (Python)

import httpx

API = "https://openapi.thunderbit.com/openapi/v1"
H = {"Authorization": "Bearer YOUR_API_KEY"}

def read_url(url: str) -> str:
    """Fetch a URL and return clean Markdown.

    Use for any web research task: docs, articles, search results, product pages.
    Returns the page as Markdown with metadata stripped.
    """
    resp = httpx.post(f"{API}/distill",
                      headers=H,
                      json={"url": url, "renderMode": "basic"},
                      timeout=60.0)
    resp.raise_for_status()
    return resp.json()["data"]["markdown"]

OpenAI function-calling example

tools = [{
    "type": "function",
    "function": {
        "name": "read_url",
        "description": "Fetch a URL and return clean Markdown for the agent to read.",
        "parameters": {
            "type": "object",
            "properties": {
                "url": {"type": "string", "description": "The URL to fetch"}
            },
            "required": ["url"],
        },
    },
}]

When the model calls read_url, dispatch to the function above and feed the result back as a tool message.

Tips

  • Keep renderMode at basic for the agent — it's a good cost / coverage default
  • Cap the returned Markdown length (e.g. 8k tokens) before feeding to the model
  • For bulk research, prefer /batch/distill — see RAG Knowledge Base

This recipe is being expanded with LangChain / LlamaIndex / CrewAI variants — check back soon.

On this page

Tool definition (Python)OpenAI function-calling exampleTipsRelated