Integrations

CrewAI

Give CrewAI agents a Thunderbit-powered web research tool

CrewAI agents need fresh, clean web content as input. Wrap /distill as a CrewAI tool so any agent in the crew can read URLs on demand.

Install

pip install crewai httpx

Custom tool

from crewai.tools import BaseTool
import httpx

API = "https://openapi.thunderbit.com/openapi/v1"
H = {"Authorization": "Bearer YOUR_API_KEY"}

class ReadUrlTool(BaseTool):
    name: str = "read_url"
    description: str = (
        "Fetch a URL and return clean Markdown. Use for any web research task: "
        "docs, articles, product pages, search results."
    )

    def _run(self, url: str) -> str:
        resp = httpx.post(f"{API}/distill",
                          headers=H,
                          json={"url": url, "renderMode": "basic"},
                          timeout=60.0)
        resp.raise_for_status()
        return resp.json()["data"]["markdown"]

Wire into a Crew

from crewai import Agent, Task, Crew

researcher = Agent(
    role="Web Researcher",
    goal="Gather authoritative information from public web pages",
    backstory="Skilled at distilling long pages into key facts.",
    tools=[ReadUrlTool()],
)

task = Task(
    description="Research how vector databases compare in 2026.",
    expected_output="A concise comparison table.",
    agent=researcher,
)

Crew(agents=[researcher], tasks=[task]).kickoff()

Tips

  • For multi-source research, expose /batch/distill as a second tool (read_urls) so the agent can fan out
  • Cap returned Markdown to ~8k tokens before handing it to the agent — avoid context bloat

This integration is being expanded with multi-agent crew templates — check back soon.