ChatGPT Web Scraping: What Works, What Breaks, What's Better

Last week, a colleague on our sales team asked me to help him pull contact info from about 200 business directory pages. His plan? Copy-paste each one into a spreadsheet. I suggested he try ChatGPT to generate a Python scraper instead. Twenty minutes later, he had a script. Thirty minutes after that, he was in my DMs: "It worked on the first five pages and then just… stopped."

That experience is weirdly common. ChatGPT is genuinely great at writing scraping code—until it isn’t. And most tutorials online stop at the “look, it works on this toy site” stage, leaving you stranded the moment you hit a real-world page with JavaScript, anti-bot walls, or pagination. In this guide, I’m going to walk you through what ChatGPT web scraping actually looks like in practice: the full workflow, five reusable prompt templates (not just one example), an honest breakdown of where things break, and what to do when they do—including no-code alternatives like that skip the code entirely.

What Is ChatGPT Web Scraping?

“ChatGPT web scraping” means using ChatGPT to help you extract data from websites. But there’s a crucial distinction most people miss: ChatGPT doesn’t scrape websites itself. It can’t visit a URL, fetch HTML, or click through pages. What it can do is generate the code (usually Python) that does those things, or parse raw HTML you paste into the chat and return structured data.

There are two main approaches:

1. ChatGPT as code generator: You describe the page and the data you want, and ChatGPT writes a Python script (typically using BeautifulSoup, Selenium, or Playwright) that you run on your own machine.
2. ChatGPT as data parser: You copy-paste raw HTML into the chat (or upload it via Code Interpreter), and ChatGPT extracts the fields you need into JSON or CSV format.

In both cases, you’re the one doing the fetching and running. ChatGPT is the brain, not the hands. Even with the newer ChatGPT Atlas browser (launched October 2025), which can browse the web conversationally, it returns answers—not structured CSV tables of 500 product rows. It’s a browsing assistant, not a data extraction pipeline.

Why Use ChatGPT for Web Scraping (And Who It’s For)

ChatGPT lowers the barrier to entry for web scraping dramatically. According to the , of developers now use or plan to use AI tools in their workflow, and ChatGPT leads the pack at 82% share. But the audience for “ChatGPT web scraping” isn’t just developers. It’s SDRs building prospect lists, ecommerce managers tracking competitor prices, real estate analysts pulling listing data, and marketing teams aggregating content.

Here’s a quick look at common use cases and who benefits:

Manually copying data from 100 pages might take 3–5 hours. A ChatGPT-generated script can do the same in minutes—if it works. And that “if” is the whole point of this article.

Gartner forecasts that by 2026, developers outside formal IT departments will account for of low-code tool users. The people searching for “ChatGPT web scraping” are increasingly non-developers who want data without hiring an engineer. For them, ChatGPT is the first stop—and tools like Thunderbit are what they reach for when the script refuses to run.

How ChatGPT Web Scraping Works: Step-by-Step Guide

Here’s the full workflow, end to end, using a business directory listing page—not a toy site.

• Difficulty: Intermediate (you need basic comfort running Python)
• Time Required: ~15–30 minutes for a first scrape
• What You’ll Need: Chrome browser, a Python environment (Python 3.10+), ChatGPT (free tier works), and a target URL

Step 1: Inspect the Website and Identify the Data You Need

Open the page you want to scrape in Chrome. Right-click on a piece of data you want (say, a business name) and choose Inspect. This opens Chrome DevTools and highlights the HTML element.

Look for the CSS selectors—things like h2.business-name, span.phone, or a.website-link. The more specific your selectors, the better ChatGPT’s output will be. Copy a representative snippet of the HTML (one “card” or “row” of data) to paste into your prompt.

You should now have a short list of field names (e.g., business_name, phone, website_url) and their corresponding CSS selectors.

Step 2: Write a Detailed ChatGPT Prompt

This is where most tutorials fail—they give you a vague prompt and hope for the best. A good scraping prompt has six parts:

1. Language and library: “Write a Python 3.11 script using BeautifulSoup 4.”
2. Target URL: The exact page to scrape.
3. CSS selectors: For each field, the selector you found in Step 1.
4. Output format: CSV, JSON, or both.
5. Special instructions: Encoding, error handling, delays.
6. HTML snippet: Paste 20–40 lines of the actual page HTML so ChatGPT can see the structure.

Here’s a sample prompt (annotated):

1You are a senior Python engineer. Write a web scraper using Python 3.11 and BeautifulSoup 4.
2Target URL: https://example.com/businesses
3Goal: Extract every business card on the page and return one row per business.
4Fields needed (CSS selectors in parentheses):
5- business_name (h2.biz-name)
6- phone (span.phone-number)
7- website_url (a.biz-link, href)
8- rating (div.stars[data-rating])
9Output: save to businesses.csv with UTF-8 encoding and a header row.
10Requirements:
11- Use requests with a realistic User-Agent header
12- Handle missing fields gracefully (None, not crash)
13- Print the number of businesses extracted at the end
14- Add a 1-second delay between requests if you loop
15Here is a representative HTML snippet from the page (one business card):
16<PASTE 20-40 LINES OF THE ACTUAL HTML HERE>

Tip: Including the HTML snippet is the single biggest accuracy booster. ChatGPT can’t visit the URL, so the snippet is its only ground truth.

Step 3: Review and Test the Generated Code

Don’t just run ChatGPT’s code blindly. Read through it first. Look for:

• Hallucinated selectors: ChatGPT sometimes invents CSS classes that don’t exist on the page.
• Missing libraries: Make sure pip install requests beautifulsoup4 (or playwright, etc.) is covered.
• Hardcoded values: Check that the URL, field names, and file paths are correct.

Set up a Python virtual environment, install the dependencies, and run the script on a small sample (one or two pages). Check the output CSV—are the columns populated? Are there blanks where you expected data?

Step 4: Refine with Follow-Up Prompts

ChatGPT shines in iteration. If the first script only gets page 1, ask:

“The script only scrapes the first page. Can you add pagination to scrape all pages? The site uses ?page=1, ?page=2, etc. Stop when a page returns zero results or after 50 pages.”

If fields are missing, ask ChatGPT to add regex fallbacks for emails or phone numbers. If the site is JS-heavy, ask for a Playwright version. Each follow-up prompt builds on the previous code—think of it as pair programming with a very fast (but sometimes overconfident) partner.

5 Copy-Paste ChatGPT Prompt Templates for Web Scraping

I haven’t found another guide that offers this. I’ve drafted, tested, and refined five prompt templates organized by scenario. Copy them, swap in your URL and HTML snippet, and ChatGPT will return working code on the first try—or very close to it.

Template 1: Listing Page Scraper (Product Catalogs, Directories)

When to use: You’re on a page with many items (products, businesses, job listings) and want one row per item.

1You are a senior Python engineer. Write a web scraper using Python 3.11 and BeautifulSoup 4.
2Target URL: [YOUR URL]
3Goal: Extract every item card on the page and return one row per item.
4Fields needed (CSS selectors in parentheses — derived from Inspect):
5- [field_1] ([selector_1])
6- [field_2] ([selector_2])
7- [field_3] ([selector_3])
8- [field_4] ([selector_4, attribute if needed])
9Output: save to items.csv with UTF-8 encoding and a header row.
10Requirements:
11- Use requests with a realistic User-Agent header
12- Handle missing fields gracefully (None, not crash)
13- Print the number of items extracted at the end
14- Add a 1-second delay between requests if you loop
15Here is a representative HTML snippet from the page (one item card):
16[PASTE 20-40 LINES OF THE ACTUAL HTML HERE]

Expected output: A CSV file with one row per item, columns matching your field names.

Template 2: Detail/Subpage Scraper (Individual Product or Profile Pages)

When to use: You have a single page with rich detail (a product page, a person’s profile, a property listing) and want to extract everything into one structured record.

1Write a Python function `scrape_detail(url)` that takes a detail page URL and returns a dict with these keys:
2- [field_1]
3- [field_2]
4- [field_3]
5- [field_4]
6- [field_5]
7Use BeautifulSoup. Gracefully handle any missing field (return None for it).
8Include regex fallbacks for email and phone — not every page wraps them in consistent tags.
9Return the dict, and also append it as one row to details.csv (create the file with header on first call).
10Reference HTML snippet from a real detail page:
11[PASTE 40-60 LINES OF ONE DETAIL PAGE HTML]

Expected output: A dict per page and a growing CSV file with one row per detail page.

Template 3: Dynamic/JS-Rendered Page Scraper (Playwright)

When to use: The page loads content via JavaScript (React, Angular, etc.)—you see an empty <div id="root"> in the HTML source.

1Write a Python web scraper using Playwright (sync API) for a JavaScript-rendered page.
2Target URL: [YOUR URL]
3Goal: extract all result cards that appear after the page finishes loading dynamically.
4Requirements:
5- Use `page.wait_for_selector('[YOUR CARD SELECTOR]', timeout=15000)` to wait for content
6- Scroll to the bottom of the page twice with a 1-second pause between scrolls to trigger lazy-loaded results
7- For each card extract: [field_1], [field_2], [field_3], [field_4]
8- Save to results.json as a list of dicts, UTF-8
9- Run headless=False first (so I can watch it) and add a 2-second pause at the end before closing
10Do not use requests or BeautifulSoup — Playwright only.

Expected output: A JSON file with one object per result card, all fields populated.

Template 4: Pagination Handler (Multi-Page Scraping)

When to use: You already have a working single-page scraper and need to loop through all pages.

1Take the existing BeautifulSoup scraper below and wrap it in a pagination loop that collects ALL pages, not just page 1.
2The site uses URL-param pagination: ?page=1, ?page=2, etc.
3Stop condition: when the current page yields zero items, OR when the response status is not 200, OR when you hit page 100 (safety cap).
4Add:
5- A 1.5-second polite delay between page requests
6- A try/except around each request that logs the error and continues
7- A progress print every 5 pages: "Page 15 → 300 items so far"
8- Final save to items_all.csv
9Existing scraper:
10[PASTE YOUR CURRENT SINGLE-PAGE SCRAPER HERE]

Expected output: A single CSV with all items from all pages, plus console output showing progress.

Template 5: Data Cleaning and Structuring (the “Paste HTML” Approach)

When to use: You already have raw HTML (from curl, from your browser, from a file) and just want ChatGPT to parse it into clean structured data—no code needed.

1I will paste raw HTML from a product detail page. You do not need to write code — just return the extracted data as a JSON object matching this schema:
2{
3  "name": string,
4  "brand": string,
5  "price": number,
6  "currency": string (ISO 4217),
7  "availability": "in_stock" | "out_of_stock" | "preorder" | "unknown",
8  "rating": number (0-5) or null,
9  "review_count": integer or null,
10  "description": string (max 500 chars),
11  "key_specs": [{"name": string, "value": string}]
12}
13Use null for anything you genuinely cannot find — do NOT hallucinate.
14Return ONLY the JSON object, no prose, no markdown fence.
15HTML:
16[PASTE THE FULL PAGE HTML HERE]

Expected output: A single JSON object, ready to drop into a spreadsheet or database.

Where ChatGPT Web Scraping Breaks (Honest Limitations)

Most tutorials gloss over this part entirely. I’ve spent enough time debugging ChatGPT-generated scrapers to know exactly where they fall apart—and the confirms that only of developers “highly trust” AI output. Here’s why.

JavaScript-Heavy and Dynamic Websites

Over of websites use JavaScript for client-side functionality. React alone now runs on 7.2% of all websites—a in a single year. When you ask ChatGPT to “scrape this page,” its default output is a requests + BeautifulSoup script. That script fetches the raw HTML—and on a React or Angular site, the raw HTML is just an empty <div id="root">. The actual data loads after JavaScript executes, which requests never does.

ChatGPT can generate Selenium or Playwright code if you ask, but those scripts are slower (Playwright averages vs. sub-second for static requests) and often need debugging for wait conditions, scroll triggers, and element selectors that ChatGPT guesses wrong.

Anti-Bot Protections and CAPTCHAs

Cloudflare protects roughly , and services like DataDome claim . A bare requests.get() with a Python user-agent is, to put it bluntly, a textbook bot fingerprint. ChatGPT-generated scripts include no proxy rotation, no TLS fingerprint spoofing, no cookie handling, and no CAPTCHA solving. On any commercial site with even basic protection, the script gets blocked on the first request.

Pagination and Large-Scale Scraping

ChatGPT’s default pagination loop iterates ?page=N or clicks a .next button. Real-world sites use cursor-based pagination, infinite scroll with IntersectionObserver, or GraphQL calls. ChatGPT can’t generate correct code for those unless you show it the exact network call—and even then, the loops are fragile. and both flag pagination as the number-one place their example scrapers need a second or third prompt.

Ongoing and Scheduled Scraping

ChatGPT gives you a one-shot script. There’s no scheduler, no change detection, no alerting. If you want “check competitor prices every morning,” you need to learn cron, Airflow, or Lambda—none of which ChatGPT covers in its initial response. For business users who need recurring data, this is a dead end.

The Speed and Cost Problem

For JS-heavy sites, real-world per-page times with Selenium or Playwright land at 3–10 seconds per page under ideal conditions, and 40–60 seconds per page with retries and anti-bot waits—a frustration in forums and tutorials.

If you use the ChatGPT API to parse HTML (the “paste HTML” approach at scale), token costs add up fast. At current GPT-4o pricing (~$2.50/M input tokens, $10/M output), parsing 1,000 product pages costs roughly $95–$105 in tokens alone. With GPT-4o mini, it’s about $6.50 for the same volume. Add proxy costs ($3–10/GB), local crawler maintenance, and developer time, and the “just use ChatGPT” approach starts looking expensive.

Estimates assume ~50K input tokens and ~2K output tokens per page. Actual costs vary by page size and output complexity.

ChatGPT Web Scraping vs. No-Code AI Scrapers vs. Custom Code: Decision Framework

Not every scraping job needs the same tool. This is the decision framework I’ve been using at Thunderbit after testing all three approaches on real projects.

In short: use ChatGPT for quick one-off scripts and learning. Use a no-code tool like Thunderbit for production-quality, recurring, or non-developer scraping. Use custom code + proxies for enterprise-scale engineering projects where you need full control.

The No-Code Alternative: How Thunderbit Handles Web Scraping Tasks Without Code

For readers who don’t code—or who’ve burned enough evenings debugging ChatGPT scripts—there’s a different path entirely. ChatGPT generates the code. skips it.

I work on the Thunderbit team, so I’ll be upfront about that. But I also genuinely believe this is the fastest path for most business users. Here’s what the workflow looks like.

AI Suggest Fields: Auto-Detect Data Structure on Any Page

Open any webpage, click the , and hit “AI Suggest Fields.” Thunderbit’s AI reads the rendered page—including JS-loaded content—and proposes column names and data types. No Inspect, no CSS selectors, no prompt engineering. Then click “Scrape.”

Compare that to the ChatGPT approach: open DevTools, find selectors, write a prompt, review the code, install dependencies, run the script, check the output, iterate. Thunderbit collapses all of that into two clicks.

Subpage Scraping to Enrich Listings Automatically

After scraping a listing page, click “Scrape Subpages.” Thunderbit visits each row’s detail page and appends additional fields—like email, phone, or bio—to your existing table. With ChatGPT, you’d need a separate script, a loop, error handling for each subpage, and a way to merge the data. Thunderbit handles it in one step.

Export Anywhere: Google Sheets, Airtable, Notion, Excel

Thunderbit offers free, one-click export to Google Sheets, Airtable, Notion, and Excel—not just CSV. A ChatGPT-generated script typically writes to a local CSV or JSON file. Pushing data to Sheets or Airtable requires extra libraries and authentication code.

Cloud Scraping vs. Browser Scraping

Thunderbit gives you two modes. Cloud scraping runs on Thunderbit’s servers, handles ~50 pages per batch, and is fast for public sites. Browser scraping uses your logged-in session for gated or login-protected pages. With ChatGPT, you’d need to configure proxies, cookies, and session handling in code—each of which is a separate debugging adventure.

Under the hood, Thunderbit routes through multiple AI models (including ChatGPT, Gemini, Claude, and others) to visually read pages and detect what to extract. So in a sense, Thunderbit already uses ChatGPT—plus three other frontier models—and handles fetching, rendering, anti-bot, pagination, and export for you.

Real-World Use Cases: Sales, Ecommerce, and Real Estate

Most ChatGPT scraping tutorials use “Books to Scrape” or some other toy site. Here’s what real business scraping looks like—with both the ChatGPT approach and the Thunderbit shortcut.

Sales Lead Extraction from Business Directories

Scenario: You need names, emails, and phone numbers from a business directory for outbound sales.

ChatGPT approach: Use Template 1 (listing page) to scrape the directory, then Template 2 (detail page) to visit each profile for contact info. You’ll need regex fallbacks for emails and phones, a polite delay, and a dedupe pass. Expect 30–60 minutes of setup and debugging.

Thunderbit approach: Open the directory, click “AI Suggest Fields,” scrape the listing, then click “Scrape Subpages” to pull contact details from each profile. Export to your CRM-ready spreadsheet. Total time: about 3 minutes. Thunderbit’s built-in handle the parsing automatically.

Ecommerce Competitor Price Monitoring

Scenario: You want to track competitor product prices, availability, and SKUs on a weekly basis.

ChatGPT approach: Generate a scraper with Template 1, add pagination with Template 4, and run it manually each week. If the competitor changes their page layout, the selectors break and you start over.

Thunderbit approach: Set up a scraper once, use Thunderbit’s scheduled cloud scraping to run it daily or weekly, and export to Google Sheets. The AI re-reads the page structure each run, so layout changes don’t break anything. For more on this workflow, see our .

Real Estate Listing Data Collection

Scenario: You need property prices, addresses, beds/baths, and agent info from a listings site.

ChatGPT approach: Most real estate sites (Zillow-style) are React SPAs with aggressive anti-bot protections. A requests + BeautifulSoup script returns an empty page. A Playwright version gets rate-limited within minutes.

Thunderbit approach: Cloud scraping with AI field detection handles the JS rendering and adapts to layout changes. Real estate portals redesign frequently—Thunderbit’s AI reads the page fresh each time, so you don’t need to update selectors. See our for a walkthrough.

Beyond One-Off Scrapes: ChatGPT API Pipelines vs. Thunderbit Extract API

If you’re building scraping into a product or pipeline, the question shifts: ChatGPT API to parse HTML, or a purpose-built scraping API?

Using the ChatGPT API to Parse HTML

The approach: use a local crawler (requests, Playwright) to fetch HTML, then send it to OpenAI’s API to extract structured JSON. This is the “paste HTML” loophole at scale.

It works. The costs and maintenance, though, are real. At GPT-4o pricing, 1,000 pages costs ~$95 in tokens. You manage the crawler, the proxies, the prompt engineering, and the output schema. When the site changes, your prompt breaks and you retune.

Thunderbit Extract API: Purpose-Built for Structured Web Data

Thunderbit’s offers a different model. You define a JSON Schema, POST a URL, and get structured data back. JS rendering and anti-bot handling are built in. Batch processing supports up to 100 URLs per request.

For teams that want structured web data as a service without maintaining a scraping pipeline, Thunderbit’s API is the shorter path to production. Check for credit costs per extraction.

Tips to Get Better Results from ChatGPT Web Scraping

A few things I’ve learned the hard way.

Be specific in your prompts. Always include: programming language, library, target URL, CSS selectors, output format, and edge case instructions. Vague prompts produce vague code.

Paste HTML snippets, not just URLs. ChatGPT can’t visit URLs. The HTML snippet is its only source of truth for the page structure. Pasting even 20–40 lines of a single data card dramatically improves accuracy.

Ask ChatGPT to lint and optimize. After generating a script, ask: “Review this code for errors, add error handling, and optimize for performance.” It catches its own mistakes surprisingly well on a second pass.

Always test on a small sample first. Run the script on 1–2 pages before scaling up. Catching a broken selector on page 1 saves you from discovering it after 500 failed requests.

Iterate, don’t start over. If the first script is 80% right, paste the output back and ask ChatGPT to fix the remaining 20%. The iterative conversation is where ChatGPT is strongest.

Ethical and Legal Considerations for ChatGPT Web Scraping

The legal side matters, so here’s the short version.

Under current US precedent, scraping publicly available data is not a federal computer crime. The ruling established that, and the (January 2024) reinforced it—a judge found that scraping public, logged-out data from Facebook and Instagram did not violate Meta’s terms of service, because a visitor without an account isn’t a “user” bound by those terms.

That said, scraping gated or authenticated data, or violating a site’s Terms of Service after agreeing to them, can still create legal risk. And when you’re scraping personal data (emails, phone numbers), EU and California data-protection laws (GDPR, CCPA) apply regardless of where the data came from.

Always check robots.txt and Terms of Service before scraping. Respect rate limits. Handle personal data responsibly. And use tools with built-in compliance features—Thunderbit, for example, respects robots.txt and offers responsible data practices by default. For a deeper dive, see our .

When to Use ChatGPT for Web Scraping—and When to Reach for Something Better

ChatGPT is a genuinely powerful tool for web scraping—it generates quick prototypes and helps you learn how scraping works under the hood. For quick one-off scripts on simple static pages, it’s hard to beat.

But for production-quality, ongoing, or large-scale scraping—especially if you’re not a developer—a purpose-built tool like Thunderbit is faster, more reliable, and requires zero maintenance. And for enterprise-scale engineering projects, custom code with proxy infrastructure gives you full control.

My decision cheat sheet:

• Quick one-off, learning, or prototyping: ChatGPT + Python
• Business users, no code, recurring scrapes:
• Developer pipelines, structured API access:
• Enterprise-scale, full control: Custom code + proxies + orchestration

If you want to try the no-code path, Thunderbit offers a free tier so you can experiment on a small scale and see the results for yourself. And if you want to see the tool in action, our has walkthroughs for different use cases.

FAQs

Can ChatGPT actually scrape websites on its own?

No. ChatGPT generates scraping code or parses HTML you provide, but it doesn’t visit URLs, fetch pages, or execute scripts. Even ChatGPT Atlas (the built-in browser launched in October 2025) is a conversational browsing assistant—it can summarize a page, but it won’t hand you a structured CSV of 500 rows.

Is ChatGPT web scraping free?

The free tier of ChatGPT can generate scraping code at no cost. But running the code requires Python and libraries (free), and if you use the OpenAI API to parse HTML at scale, you’ll pay token costs—roughly $6.50 per 1,000 pages with GPT-4o mini, or ~$95 with GPT-4o. Proxies and infrastructure are additional.

What’s the best Python library for ChatGPT-generated web scrapers?

For static HTML pages, BeautifulSoup with the requests library is the simplest and fastest. For JavaScript-rendered pages, Playwright is the modern choice—it’s faster than Selenium (averaging about 2.9 seconds per page load vs. 4.8 seconds) and has a cleaner API. Selenium is mainly useful for legacy projects.

Can I use ChatGPT to scrape data without any coding?

Not directly. ChatGPT generates code you still need to run. If you want a truly no-code option, tools like let you scrape in two clicks—no Python, no terminal, no debugging. You get AI-suggested fields, one-click export to Google Sheets or Airtable, and built-in handling for JS rendering and anti-bot protections.

Is it legal to scrape websites using ChatGPT-generated code?

Scraping publicly available, logged-out data is generally legal under current US precedent (hiQ v. LinkedIn, Meta v. Bright Data). However, scraping gated content, violating a site’s Terms of Service, or mishandling personal data (emails, phone numbers) can create legal risk under contract law or privacy regulations like GDPR and CCPA. Always check robots.txt and the site’s ToS before scraping.

Learn More