How to Write a Web Scraper with Python: From Start to Finish

I’ll be honest: the first time I wrote a web scraper, I felt like I’d just discovered a secret superpower. Suddenly, all those hours I’d spent copying and pasting data from websites for sales leads or price checks felt like a distant memory. But here’s the thing—writing your own web scraper with Python is a rite of passage for anyone who wants to automate the boring stuff on the internet. And if you’re a business user, it can be the difference between spending your Friday night wrestling with spreadsheets or actually making it to happy hour.

In this guide, I’ll walk you through how to write a web scraper with Python, step by step, with real code you can use. Then, I’ll show you why, for most business teams, there’s a much easier way—using an that gets the job done in two clicks, no code required. Whether you’re a Python enthusiast or just want the data without the headache, you’ll find the right approach for your needs.

What is Web Scraping with Python? A Simple Introduction

Let’s start with the basics. Web scraping is just a fancy term for automatically collecting information from websites. Think of it as sending a robot intern to do all your copy-pasting—except the robot never gets bored or asks for a raise.

A web scraper is a script or program that:

• Accesses a webpage (just like your browser does)
• Extracts specific data (like product names, prices, or contact info)
• Saves it in a structured format (think spreadsheets or JSON files)

Python is the go-to language for this because it’s readable, has a ton of great libraries, and is basically the Swiss Army knife of programming. In fact, .

Here’s the core workflow:

1. Fetch the webpage (grab the HTML)
2. Parse the HTML to find the data you want
3. Save the results somewhere useful

It’s like baking a cake: get the ingredients (HTML), pick out the good stuff (data), and serve it up (export).

Why Web Scraping Matters for Business Teams

Web scraping isn’t just for techies or data nerds. It’s become a must-have for sales, marketing, ecommerce, real estate, and anyone who needs fresh, accurate web data to make decisions. The , and it’s growing at 28% a year. That’s a lot of data—and a lot of opportunity.

Let’s look at some real-world business use cases:

The bottom line: web scraping saves time, reduces manual work, and gives you a competitive edge. And if you’re still copying and pasting, your competitors are probably already a step ahead.

Getting Ready: Tools and Skills Needed for Writing a Web Scraper

Before you dive into code, let’s talk about what you need in your toolbox.

The Basics

• Python Installed: Download the latest version () and make sure you can run python in your terminal.
• Code Editor: VS Code, PyCharm, or even Notepad++ will do. I’m a fan of VS Code for its Python support.
• Virtual Environment: Not strictly required, but highly recommended to keep your project dependencies tidy. Set one up with python -m venv venv.

Key Python Libraries

• Requests: For fetching web pages ().
• BeautifulSoup: For parsing HTML and finding elements ().
• Selenium: For scraping sites that load content with JavaScript ().

Install them with:

1pip install requests beautifulsoup4 lxml selenium

Understanding HTML

You don’t need to be a web developer, but you should know how to inspect a page’s HTML. Right-click, choose “Inspect,” and you’ll see the DOM tree. This is where you’ll find the tags and classes your scraper needs to target ().

Step-by-Step: How to Write a Web Scraper with Python

Let’s roll up our sleeves and build a simple web scraper from scratch. I’ll use a real-world example—scraping product listings or news headlines. You can adapt this to your own use case.

Setting Up Your Python Environment

First, create a project folder and set up a virtual environment:

1mkdir my-scraper
2cd my-scraper
3python -m venv venv
4# Activate the venv:
5# On Windows:
6venv\Scripts\activate
7# On macOS/Linux:
8source venv/bin/activate

Install the libraries:

1pip install requests beautifulsoup4 lxml

Create a file called scraper.py and open it in your editor.

Fetching and Parsing Web Pages

Let’s fetch the HTML from a target site. For this example, I’ll use (a classic for scraping demos).

1import requests
2from bs4 import BeautifulSoup
3url = "https://news.ycombinator.com/"
4response = requests.get(url)
5if response.status_code == 200:
6    html_content = response.content
7else:
8    print(f"Request failed with status {response.status_code}")
9    exit()

Now, parse the HTML with BeautifulSoup:

1soup = BeautifulSoup(html_content, "html.parser")
2print(soup.title.string)  # Should print "Hacker News"

Extracting the Data You Need

Let’s say you want to grab all the story titles and their links. By inspecting the page, you’ll see each title is in an <a class="storylink"> tag.

1stories = soup.find_all('a', class_='storylink')
2data = []
3for story in stories:
4    title = story.get_text()
5    link = story['href']
6    data.append({"title": title, "url": link})
7    print(title, "->", link)

If you’re scraping products, you’d look for something like <div class="product-item"> and extract fields inside it. Here’s a generic pattern:

1products = soup.find_all('div', class_='product-item')
2for prod in products:
3    name = prod.find('h2').get_text()
4    price = prod.find('span', class_='price').get_text()
5    url = prod.find('a')['href']
6    data.append({"name": name, "price": price, "url": url})

Saving Scraped Data to CSV or JSON

Now, let’s save the data so you can actually use it.

To CSV:

1import csv
2with open("output.csv", mode="w", newline="", encoding="utf-8") as f:
3    writer = csv.writer(f)
4    writer.writerow(["Title", "URL"])
5    for item in data:
6        writer.writerow([item["title"], item["url"]])

To JSON:

1import json
2with open("output.json", mode="w", encoding="utf-8") as f:
3    json.dump(data, f, indent=2)

Open up your CSV in Excel or your JSON in any text editor—and voilà, you’ve just automated hours of manual work.

Level Up: Handling Pagination and Dynamic Content

Most real-world sites don’t fit on one page. Here’s how to handle more advanced scenarios.

Pagination

If the site uses URL-based pagination (e.g., ?page=2), you can loop through page numbers:

1base_url = "https://example.com/products?page="
2for page_num in range(1, 6):
3    url = base_url + str(page_num)
4    resp = requests.get(url)
5    if resp.status_code != 200:
6        break
7    soup = BeautifulSoup(resp.content, "html.parser")
8    # Extract data as before

()

If the site uses a “Next” button, find the link and follow it:

1url = "https://example.com/products"
2while url:
3    resp = requests.get(url)
4    soup = BeautifulSoup(resp.content, "html.parser")
5    # Extract data
6    next_link = soup.find('a', class_='next-page')
7    if next_link and 'href' in next_link.attrs:
8        url = "https://example.com" + next_link['href']
9    else:
10        url = None

Dynamic Content (JavaScript-Rendered)

If the data isn’t in the HTML (e.g., loaded by JavaScript), you’ll need Selenium:

1from selenium import webdriver
2driver = webdriver.Chrome()
3driver.get("https://example.com/complex-page")
4driver.implicitly_wait(5)
5page_html = driver.page_source
6soup = BeautifulSoup(page_html, "html.parser")
7# Now extract data as before

()

Selenium can also click “Load More” buttons or scroll the page. Just be aware: it’s slower and more resource-intensive than plain Requests.

Common Pitfalls and Challenges When Writing Your Own Web Scraper

Here’s where things get real. Writing a web scraper is fun—until the website changes and your script breaks at 2am before a big deadline. Here are the most common headaches:

• Website Structure Changes: If the site redesigns or changes class names, your scraper might stop working. Maintenance is a constant battle ().
• Anti-Bot Measures: CAPTCHAs, rate limits, and IP blocks are everywhere. .
• Legal and Ethical Issues: Always check the site’s robots.txt and terms of service. Public data is usually fair game, but don’t scrape private or copyrighted content ().
• Data Quality: Scraped data can be messy. You might need to clean up HTML tags, whitespace, or broken text.
• Performance: Scraping lots of pages is slow unless you use threading or async techniques.
• Maintenance Burden: Every new site or change means more scripts to fix. It’s a never-ending game of whack-a-mole.

If you’re a developer who loves a challenge, this is part of the fun. If you just want the data, though, it can get old fast.

The Smarter Alternative: AI Web Scraper Tools Like Thunderbit

Here’s where I get to put on my Thunderbit hat (which, in my mind, is a lightning-bolt-shaped baseball cap). Most business users don’t want to write or maintain code—they just want the data, now.

That’s why we built , an that lets you scrape any website, PDF, or image in two clicks. No code, no setup, no HTML knowledge required.

What Makes Thunderbit Different?

• 2-Click Setup: Open a webpage, click “AI Suggest Fields,” then “Scrape.” Done.
• AI Field Suggestions: Thunderbit’s AI reads the page and recommends the best columns (like product name, price, rating, etc.).
• Subpage and Pagination Scraping: Automatically follows “Next” links or dives into detail pages to enrich your data.
• Instant Data Export: Export to Excel, Google Sheets, Airtable, Notion, CSV, or JSON—free, with no hoops to jump through.
• Rich Data Types: Extract emails, phone numbers, images, even text from PDFs or images (thanks to built-in OCR).
• Cloud or Browser Scraping: Scrape up to 50 pages at once in the cloud, or use your browser for sites that need your login.
• No Maintenance Headaches: The AI adapts to layout changes automatically, so you don’t have to fix broken scripts.

Side-by-Side Comparison: Python vs. Thunderbit

For most business users, Thunderbit is the shortcut to getting structured data without the pain.

When Should You Write Your Own Web Scraper vs. Use an AI Web Scraper?

So, which approach is right for you? Here’s my honest take:

Write Your Own Scraper When:

• You need very custom logic (e.g., logging in with 2FA, multi-step workflows, or deep integration with your own backend).
• You have strong coding skills and enjoy tinkering.
• The site is stable and you’re okay maintaining scripts.
• You need to integrate scraping into a larger software system.
• You’re scraping data that’s behind a login or isn’t supported by AI tools.

Use an AI Web Scraper (Thunderbit) When:

• You don’t want to code or maintain scripts.
• You need data fast (for a one-off or recurring task).
• The site changes often or has anti-bot measures (Thunderbit handles this for you).
• You want built-in features like OCR, email/phone extraction, or direct export to your favorite tools.
• You value your time and want to focus on analysis, not debugging.

Here’s a quick decision checklist:

• Is the data public and not behind a tricky login? → Thunderbit is probably your best bet.
• Is this a one-off or ad-hoc need? → Thunderbit.
• Do you need deep customization or integration? → Python script.
• Do you have a developer on hand and love to code? → Python script.
• Do you want to avoid maintenance headaches? → Thunderbit.

And remember, you can always start with Thunderbit for quick wins, then invest in custom scripts if your needs get more complex.

For more on how AI web scrapers work and when to use them, check out our .

Key Takeaways: Making Web Scraping Work for Your Business

Let’s wrap it up:

• Web scraping with Python is powerful and flexible, but comes with a learning curve and ongoing maintenance.
• AI web scrapers like Thunderbit make data extraction accessible to everyone—no code, no setup, just results.
• For most business users, the fastest path to value is using an AI tool, unless you have highly specialized needs.
• The web is a goldmine of data, and the right approach can save you hours (or even days) of manual work.

FAQs

1. What is web scraping and why is Python commonly used for it?

Web scraping is the automated process of collecting data from websites. Python is popular for web scraping because of its readability, wide library support (like requests, BeautifulSoup, and Selenium), and ease of use for handling HTML content.

2. What are common business use cases for web scraping?

Businesses use web scraping for sales lead generation, price monitoring, market intelligence, and real estate data aggregation. It helps automate repetitive data collection tasks and provides up-to-date insights for decision-making.

3. What are the main challenges of writing your own web scraper?

Common challenges include dealing with changing website structures, anti-bot protections like CAPTCHAs, legal and ethical concerns, data quality issues, and the ongoing maintenance required to keep scrapers functioning.

4. How does Thunderbit’s AI web scraper differ from traditional Python-based scrapers?

Thunderbit offers a no-code solution with AI-powered field suggestions, automatic pagination, and export options. It requires minimal setup, adapts to site changes automatically, and is accessible to non-developers, unlike Python scripts which demand coding skills and manual maintenance.

5. When should you use an AI scraper like Thunderbit instead of coding your own?

Use Thunderbit if you need fast, reliable scraping without coding, especially for public data or ad-hoc tasks. Opt for a custom Python scraper if you need full control, deep integration, or are scraping complex, login-restricted content.

Further Reading:

If you want to dive deeper into web scraping, check out some of our other guides:

And if you’re ready to try the easiest way to scrape the web, and see for yourself. Your Friday nights (and your data) will thank you.