Gemini Web Scraping That Actually Works (Code + No-Code)

Most "gemini web scraping" tutorials read like they were written for the same person: a Python developer who already has a virtual environment, a Pydantic schema, and a strong opinion about async libraries. If that's you, great — we'll get to the code. But if you're in sales, marketing, or ecommerce ops and just want structured data from a bunch of web pages without learning what markdownify does, you're not alone.

Gemini is Google's multimodal AI family, and it's quickly becoming one of the go-to engines for web data extraction. The 2025 Stack Overflow Developer Survey found that are using or planning to use AI tools — and LLM-powered scraping is a big part of that wave. But there's a real gap between a "cool demo on one URL" and a pipeline that handles pagination, subpages, anti-bot walls, and messy HTML at scale. This guide covers both the Python (code) and no-code routes, walks through model selection with actual token math, tackles multi-page scraping (the step every other tutorial skips), and is honest about where Gemini scraping falls apart. By the end, you'll know which path fits your workflow — and how to avoid the pitfalls I've seen trip up both developers and business users.

What Is Gemini Web Scraping?

Gemini web scraping means feeding a web page's content — HTML, Markdown, or even a screenshot — to one of Google's Gemini AI models, which then interprets the page and returns structured data. No CSS selectors. No XPath. No brittle rules that break the moment a site tweaks its layout.

The core workflow looks like this:

1. Fetch the page (with requests, a headless browser, or a Chrome extension)
2. Clean and convert the content (usually HTML → Markdown, to cut token costs)
3. Send to Gemini with a schema describing the fields you want
4. Receive structured JSON back — ready for your spreadsheet, CRM, or database

Compare that to traditional scraping with BeautifulSoup or Selenium, where you hard-code selectors like div.product-title > span.price and pray the site doesn't redesign next Tuesday. Gemini reads the page the way a human would — it understands context, adapts to layout changes, and handles messy formatting without custom rules.

One more thing worth noting: Gemini is natively multimodal. It processes text, images, video, audio, PDFs, and code in a single request. That opens up scraping approaches — like sending a screenshot instead of HTML — that most other LLMs simply can't match. We'll get to that later.

Why Gemini Web Scraping Matters for Business Teams

If you're wondering why a marketing manager or ecommerce analyst should care about LLMs and web scraping, here's the short version: it saves a staggering amount of time, and it doesn't break every time a website updates.

The from about $1 billion in 2025 to over $2 billion by 2030 — and AI-driven extraction is the fastest-growing segment. That's not hype; it reflects a real shift in how teams collect data.

Here's where Gemini scraping fits into everyday business workflows:

The practical upside is twofold. First, you skip the cycle of writing, testing, and debugging parsing scripts — the model reads the page fresh each time. Second, you don't need to hire a developer every time a site moves a <div>. The Gemini free tier means experimentation is nearly zero-cost for small-scale jobs: , no credit card required.

Which Gemini Model Should You Pick? (Flash Lite vs. Flash vs. Pro)

Not all Gemini models are equal for scraping. This is the practical comparison I wish every tutorial included, because picking the wrong tier either wastes money or produces garbage data.

All three current Gemini 2.5 models share a 1,048,576-token context window and are multimodal. The differences are in cost, speed, and how well they handle complex extraction.

(Pricing from . Batch API is 50% off these rates.)

Gemini 2.5 Flash Lite: Fast and Cheap, but Watch for Gaps

Flash Lite is the budget option. It's ideal for simple, flat data — product names, prices, single-level listings — at high volume. But it has documented issues with optional fields, timestamps, and nested data. One developer on Google's forum that Flash Lite "goes nuts" when schemas include non-required properties, emitting repetitive text until the token limit. If your schema has more than two levels of nesting, or fields that might be absent on some pages, Flash Lite will burn your tokens and your patience.

Gemini 2.5 Flash: The Sweet Spot for Most Scraping Jobs

Flash is where I'd start for almost any real scraping task. It handles structured extraction well, manages pagination logic, and costs about 3× more than Flash Lite on input — but the accuracy jump is worth it. On , Flash is within a few points of Pro, which means it handles the inferring, normalizing, and flattening that scraping actually requires.

Gemini 2.5 Pro: Maximum Accuracy for Complex Data

Pro is the precision tool. Use it when you're extracting deeply nested schemas (think: product specs with multiple variant groups, each with sizes, colors, and prices), or when fabricated fields are unacceptable (legal, financial, medical data). It's roughly 12× the input cost of Flash Lite, so reserve it for jobs where accuracy matters more than price.

Worked Cost Example: 10,000 Product Pages

Assuming you preprocess HTML to Markdown (which you should — more on that below), a typical product page drops from ~20,000 tokens of raw HTML to ~4,000 tokens of Markdown. Output JSON is about 500 tokens per page.

Without Markdown preprocessing (raw HTML at ~200M tokens input), those numbers jump 4-5×. Preprocessing is the single highest-leverage optimization in the pipeline.

Code vs. No-Code: Two Paths to Gemini Web Scraping

Here's the fork in the road. If you're a developer building a custom pipeline, the Python + Gemini API route gives you maximum control. If you're a business user who needs data now and doesn't want to touch a terminal, a no-code AI scraper gets you there faster.

is the no-code option we built at Thunderbit — a Chrome extension that uses AI (including Gemini, ChatGPT, Claude, and others under the hood) to suggest fields, scrape in two clicks, and export to your tool of choice. I'll walk through both paths below.

For spreadsheet-first users, Quadratic is another option worth knowing — it's an AI spreadsheet that can run Gemini-powered web scraping inside the sheet itself. But for workflows that start from a known web page (product listings, directories, lead databases), Thunderbit matches the user's mental model more closely.

Step-by-Step: Gemini Web Scraping with Python

This section is for developers. If you want the no-code path, skip ahead.

Before you start:

• Difficulty: Intermediate (Python familiarity required)
• Time Required: ~20–30 minutes for first scrape
• What You'll Need: Python 3.10+, a Google AI Studio account (free), a target URL

Step 1: Set Up Your Python Environment and Gemini API Key

Create a project folder and virtual environment, then install the required libraries:

1mkdir gemini-scraper && cd gemini-scraper
2python -m venv venv && source venv/bin/activate
3pip install -U google-genai requests beautifulsoup4 markdownify pydantic

Important: The only correct SDK in 2026 is google-genai. The older google-generativeai package hit end-of-life on 2025-11-30 and is now deprecated. If you see import google.generativeai as genai in a tutorial, that code is outdated.

Next, get your API key from . Click "Get API Key," create a new key, and store it as an environment variable:

1export GEMINI_API_KEY="your-key-here"

You should now have a working Python environment with all dependencies installed and your API key ready.

Step 2: Fetch the Target Page HTML

Use requests to grab the page. For this example, let's scrape a product page:

1import requests
2url = "https://example.com/product/widget-pro"
3response = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=30)
4html = response.text

If the site uses heavy JavaScript rendering or anti-bot protection, requests.get() may return an empty shell or a 403. We'll cover mitigations in the limitations section — but for many public sites, this works fine.

Step 3: Clean HTML and Convert to Markdown

This is the step most tutorials mention but don't quantify. Raw HTML for a typical product page runs about 20,000 tokens. After BeautifulSoup pruning and Markdown conversion, you're looking at roughly 765–4,000 tokens — a that saves real money and reduces hallucination.

1from bs4 import BeautifulSoup
2from markdownify import markdownify
3soup = BeautifulSoup(html, "html.parser")
4main = soup.select_one("main") or soup  # grab only the content area
5markdown_content = markdownify(str(main))

The select_one("main") call strips out headers, footers, nav bars, and scripts — all noise that wastes tokens and confuses the model. If the site doesn't use a <main> tag, try .product-detail, #content, or whatever wraps the actual data.

After this step, you should have a clean Markdown string with just the page's meaningful content.

Step 4: Define Your Data Schema and Send to Gemini

Use Pydantic to define what you want back. The google-genai SDK accepts a Pydantic BaseModel directly as response_schema:

1from google import genai
2from google.genai import types
3from pydantic import BaseModel
4class Product(BaseModel):
5    name: str
6    price: str
7    sku: str | None = None
8    description: str
9    sizes: list[str] = []
10    colors: list[str] = []
11client = genai.Client()  # reads GEMINI_API_KEY from env
12response = client.models.generate_content(
13    model="gemini-2.5-flash",
14    contents=f"Extract product details from this page:\n\n{markdown_content}",
15    config=types.GenerateContentConfig(
16        response_mime_type="application/json",
17        response_schema=Product,
18    ),
19)
20product = response.parsed
21print(product)

A few gotchas from the :

• Do not use Field(default=...) in schemas sent to Gemini — the API raises a ValueError. Use sku: str | None = None at the type level instead.
• Keep nesting shallow (3 levels max). Deeply nested schemas cause Flash and Flash Lite to produce recursive output or unclosed brackets.
• Mark fields as required when using Flash Lite, and use sentinel empty strings instead of omission — Flash Lite's handling of optional fields is .

You should now have a parsed Product object with structured data from the page.

Step 5: Export and Store Your Scraped Data

Save the result as JSON or CSV:

1import json
2with open("products.json", "w") as f:
3    json.dump(product.model_dump(), f, indent=2)

For piping into Google Sheets, you can use the gspread library. For databases, serialize to your ORM of choice. The structured output from Gemini is clean enough to go straight into most downstream tools.

Step-by-Step: Gemini Web Scraping Without Code (Using Thunderbit)

This is the path for business users — or developers who'd rather not write throwaway scraping scripts.

Before you start:

• Difficulty: Beginner
• Time Required: ~5 minutes for first scrape
• What You'll Need: Chrome browser, (free tier works)

Step 1: Install the Thunderbit Chrome Extension

Head to the and click "Add to Chrome." Sign up with your email — the whole process takes under a minute. Compare that to the 15–30 minute Python setup above.

Step 2: Open Your Target Page and Click "AI Suggest Fields"

Navigate to the website you want to scrape — a product listing, real estate directory, lead database, whatever. Click the Thunderbit icon in your browser toolbar, then hit "AI Suggest Fields."

Thunderbit's AI reads the page and recommends column names and data types automatically — things like "Product Name," "Price," "Rating," "Image URL." You can adjust column names, remove fields you don't need, or add custom AI prompts per column (for example, "categorize as High/Medium/Low" or "translate to English").

You should see a table preview with your configured columns before scraping a single row.

Step 3: Click "Scrape" and Review Results

One click. Thunderbit handles pagination — both click-based "Next" buttons and infinite scroll — and extracts data into a structured table. You can choose between:

• Cloud Scraping: Faster, processes up to 50 pages simultaneously. Works on public sites.
• Browser Scraping: Runs in your logged-in browser tab. Use this for sites that require authentication (CRMs, gated directories, internal tools).

The results appear in a table right in the extension sidebar. Scan for obvious errors before exporting.

Step 4: Export to Excel, Google Sheets, Airtable, or Notion

Click the export button and pick your format. Thunderbit exports to Excel, Google Sheets, Airtable, and Notion — free, no paywall. Image fields are uploaded directly into Notion and Airtable image libraries, which is a nice touch if you're scraping product photos or headshots.

No JSON parsing. No scripting. Data is ready to use immediately.

Multi-Page and Subpage Scraping with Gemini

Most tutorials quietly end after one URL. Real scraping jobs don't.

Scraping 500 product pages with pagination and detail subpages is a job — and the gap between a single-URL demo and that reality is enormous.

Handling Pagination with the Gemini API (Code Approach)

For page-number URLs (the most common pattern), loop through pages until you get an empty result:

1import time
2all_products = []
3for page in range(1, 101):  # up to 100 pages
4    url = f"https://example.com/products?page={page}"
5    md = fetch_clean(url)  # your HTML→Markdown function from earlier
6    response = client.models.generate_content(
7        model="gemini-2.5-flash-lite",  # cheap for listing pages
8        contents=f"Extract product names and URLs:\n\n{md}",
9        config=types.GenerateContentConfig(
10            response_mime_type="application/json",
11            response_schema=list[ListingItem],
12        ),
13    )
14    items = response.parsed
15    if not items:
16        break
17    all_products.extend(items)
18    time.sleep(4)  # respect free-tier rate limits

For cursor-based or infinite-scroll sites, you'll need to intercept the XHR endpoint the front-end calls (check your browser's Network tab) and loop that endpoint directly. Cheaper than re-rendering, and you only send items through Gemini if fields need LLM cleaning.

Watch your token costs here — each page multiplies the bill. Use Flash Lite for simple listing pages and upgrade to Flash only for detail extraction.

Scraping Subpages for Richer Data (Code Approach)

The classic two-stage pattern: Stage 1 scrapes a listing page for URLs, Stage 2 visits each detail page for richer data.

1# Stage 1: harvest URLs with cheap Flash Lite
2class Listing(BaseModel):
3    product_urls: list[str]
4listing = client.models.generate_content(
5    model="gemini-2.5-flash-lite",
6    contents=f"Extract product URLs:\n{listing_md}",
7    config=types.GenerateContentConfig(
8        response_mime_type="application/json",
9        response_schema=Listing,
10    ),
11).parsed
12# Stage 2: extract details with Flash
13class ProductDetail(BaseModel):
14    name: str
15    price: str
16    specs: dict[str, str]
17    reviews: list[str]
18for url in listing.product_urls:
19    md = fetch_clean(url)
20    detail = client.models.generate_content(
21        model="gemini-2.5-flash",
22        contents=f"Extract product detail:\n{md}",
23        config=types.GenerateContentConfig(
24            response_mime_type="application/json",
25            response_schema=ProductDetail,
26        ),
27    ).parsed
28    # save detail...
29    time.sleep(0.5)

This works, but it's a lot of plumbing: URL deduplication, error handling, rate limiting, retry logic, caching raw HTML so schema tweaks don't re-run fetches. For 50 pages it's manageable. For 5,000, you're building infrastructure.

The No-Code Alternative: Thunderbit's Built-In Pagination and Subpage Scraping

Thunderbit handles both click-based and infinite-scroll pagination automatically — no loop scripting needed. For subpage enrichment, the "Scrape Subpages" feature visits each detail page linked from your listing and enriches the original table with deeper fields. One click, not one script.

Cloud scraping mode processes up to 50 pages simultaneously, which makes a real difference when you're scraping a product catalog or real estate directory at scale. For anyone who doesn't want to manage Python loops and retry logic, this is the practical choice. (For more on , we have a separate walkthrough.)

Screenshot Scraping: Gemini's Multimodal Shortcut

Here's an approach most tutorials skip entirely: sending a screenshot of a web page to Gemini's vision API instead of raw HTML. One developer that a single screenshot costs only ~258 tokens — compared to thousands for even cleaned Markdown. That's a dramatic cost difference for simple extractions.

How to Use Gemini's Vision API for Web Scraping

Capture a screenshot with Playwright, encode it, and send it to Gemini:

1from playwright.sync_api import sync_playwright
2from google import genai
3from google.genai import types
4from pydantic import BaseModel
5class Product(BaseModel):
6    title: str
7    price: str
8with sync_playwright() as p:
9    page = p.chromium.launch().new_page()
10    page.goto("https://example.com/product/widget-pro")
11    page.wait_for_load_state("networkidle")
12    png_bytes = page.screenshot(full_page=False)  # above-the-fold only
13client = genai.Client()
14resp = client.models.generate_content(
15    model="gemini-2.5-flash",
16    contents=[
17        {"inline_data": {"mime_type": "image/png", "data": png_bytes}},
18        "Extract the product title and price as JSON.",
19    ],
20    config=types.GenerateContentConfig(
21        response_mime_type="application/json",
22        response_schema=Product,
23    ),
24)
25print(resp.parsed)

According to Google's , an image where both dimensions are ≤ 384 pixels costs 258 tokens. Larger images are tiled into 768×768 chunks at 258 tokens each. A short above-the-fold screenshot (258–1,600 tokens) beats raw HTML handily — but a very tall full-page screenshot (~5,000 tokens) can actually lose to clean Markdown (~765–1,200 tokens).

Limitations of Screenshot Scraping

• Lower precision for dense tables: Multi-column layouts, small fonts, and overlapping elements cause partial reads — not hallucinations, but missed labels and mis-aligned headers.
• Can't follow links: Vision returns text, not clickable anchors. No pagination, no subpage enrichment.
• Resolution ceiling: Text smaller than ~10 px is frequently misread. Google downsamples to ~1,568 px on the longest edge.
• Capture overhead: Playwright launch + networkidle wait is 2–5 seconds per page, which adds up at scale.

Screenshot scraping shines for JS-heavy pages, bot-blocked sites (where requests.get() returns a 403 but a browser renders fine), and pages with data embedded in charts or images. For long, text-heavy pages, Markdown is still the better bet.

Thunderbit's image and PDF scraping uses a similar AI-vision approach — drop an image or PDF in and it returns a structured table, no screenshot scripting or base64 plumbing required. (See also: .)

When Gemini Web Scraping Fails (and What to Do Instead)

Gemini is an extraction engine, not a fetching engine. If you can't get the page content to Gemini, it can't help you. Period.

There are several common scenarios where the whole approach breaks down, and most tutorials treat them as an afterthought. I'd rather be direct.

A few of these deserve extra context.

Anti-bot is now actively LLM-aware. Cloudflare as of July 2025, with 416 billion AI bot requests blocked in the first five months. Datadome added LLM-specific detection in 2025 and saw LLM bot traffic quadruple. A plain requests.get() + Gemini is essentially dead against Datadome-protected sites. The fingerprint is the problem, not the IP — rotating IPs alone does nothing if the TLS fingerprint screams "Python requests."

Hallucination is subtle. LLMs trained to be helpful will fill optional fields with plausible fabrications rather than return null. I've seen models guess a product's brand from the URL slug, infer currency from the TLD, and write plausible-but-fake review counts from skeleton loaders. The mitigation stack: strict Pydantic schemas, retry loops with validation feedback, grounding validators that check extracted values actually appear in the source HTML, and (Flash extracts, Pro validates a sample).

The 1M context window is not usable at 1M. and others shows that reasoning quality degrades well before the token limit. Treat ~200K–300K tokens as the practical ceiling for structured extraction.

Decision Tree: Which Tool Should You Use?

• Low volume + simple pages + developer → Gemini API free tier + Python
• Medium volume + complex schemas + developer → Gemini 2.5 Flash paid + Python + structured outputs + preprocessing
• Any volume + non-developer + login-walled or pagination-heavy →
• Very high volume + heavy anti-bot + mission-critical → Managed scraping infrastructure (proxy services) + Gemini as the extraction layer

Gemini Web Scraping: Tips to Save Time and Money

Whether you're writing Python or clicking buttons, these will save you headaches.

1. Always preprocess HTML to Markdown before sending to Gemini. is typical; 95% is achievable when you also pre-trim with BeautifulSoup.
2. Use google-genai only. Do not use the deprecated google-generativeai package — it's EOL.
3. Start on Flash Lite only for flat schemas. Upgrade to Flash the moment nesting or optional fields appear.
4. Avoid Field(default=...) in Pydantic schemas you pass to Gemini. Use sku: str | None = None at the type level.
5. Pydantic + response_schema is load-bearing — it's both a contract and a hallucination guardrail.
6. Use the for jobs over 1,000 pages — 50% off and doesn't count against real-time RPM.
7. Validate a random 10–50 row sample by hand before scaling a new extractor. Accuracy drift is invisible until you look.
8. Cache raw HTML to disk — schema tweaks shouldn't re-run fetches.
9. Record the source URL on every row so you can re-crawl individual pages without re-running the whole job.
10. For no-code users: use custom AI prompts per column in Thunderbit to push prompt engineering into the spreadsheet layer — translate, categorize, summarize at the column level.

And one more: don't ship the free tier to production. Limits were cut 50–80% in December 2025 and could be cut again without notice.

Wrapping Up

The distance between a single-URL Gemini demo and a production pipeline is larger than most tutorials let on.

The Python + Gemini API route gives developers full control over model selection, preprocessing, pagination, and schema design. The no-code route — tools like — gives business users the same structured data extraction without touching a terminal.

Here's what I'd take away:

• Model selection matters. Flash Lite for volume, Flash for balance, Pro for complexity. Don't default to the cheapest option and wonder why your data is wrong.
• Multi-page and subpage scraping is where tutorials fall short — and where real work happens. Both paths covered here address that gap.
• Honest limitations save you time. If a site blocks API requests, no amount of prompt engineering will help. Pick the right tool for the job, not the fanciest one.
• Preprocessing HTML to Markdown is the single highest-leverage optimization — it cuts costs by 75%+ and reduces hallucination.

If you want to try the no-code path, lets you scrape a handful of pages and see the results for yourself. If you prefer coding, Gemini's free API tier is enough to prototype a pipeline in an afternoon. Either way, you'll have structured data faster than copy-pasting ever allowed. For more on or , we've covered those topics in depth on our blog.

FAQs

How much does it cost to use Gemini for web scraping?

The Gemini API has a free tier with roughly 100 requests/day on Pro, 500/day on Flash, and 1,000/day on Flash Lite (as of early 2026 — these limits were reduced in December 2025). On the paid tier, scraping 10,000 product pages costs approximately $1.50 with Flash Lite, $6 with Flash, or $25 with Pro — assuming you preprocess HTML to Markdown first. Without preprocessing, costs jump 4–5×. The Batch API offers a 50% discount for non-real-time jobs.

Can Gemini scrape websites that require login?

Gemini's API alone cannot log into websites — it only processes the content you send it. You need to fetch the HTML yourself using your own authenticated session (e.g., with a headless browser and stored cookies). Thunderbit's Browser Scraping mode handles this natively: it runs in your logged-in Chrome tab, so any site you can see in your browser, Thunderbit can scrape.

Is Gemini web scraping legal?

Legality depends on the website's terms of service, the data type, and your jurisdiction. In the US, post-hiQ v. LinkedIn and Meta v. Bright Data, scraping publicly accessible data without logging in is generally considered permissible — but every case is fact-specific. Scraping behind a login carries higher legal risk. EU residents' personal data is subject to GDPR regardless of whether the site is public. Always respect robots.txt and terms of service, and avoid scraping personal data without a lawful basis.

Can I use Gemini to scrape dynamic JavaScript-heavy sites?

Yes, but you need to render the JavaScript first — either with a headless browser (Playwright, Puppeteer) or by intercepting the site's API endpoints directly. Once you have the rendered HTML, clean it and send it to Gemini as usual. Alternatively, screenshot scraping with Gemini's vision API bypasses JS rendering entirely — if it renders in a browser, Gemini can see it. Thunderbit handles JS-rendered pages automatically in both Cloud and Browser scraping modes.

What's the difference between using Gemini for scraping vs. a dedicated scraping tool like Thunderbit?

Gemini is an extraction engine — it interprets content and returns structured data. It doesn't visit websites, handle pagination, manage authentication, or export to spreadsheets. You still need something to get the page content to Gemini and something to do useful things with the output. Dedicated tools like combine fetching, rendering, AI extraction, pagination, subpage enrichment, and export in one package — no plumbing required.

Learn More