Scrape TripAdvisor with Python (Without Getting Blocked)

Last week, I tried to pull hotel ratings and review counts for about 200 properties across three European cities from TripAdvisor. My first script — a basic requests.get() with default headers — returned a beautiful 403 Forbidden on every single request. Not a single byte of useful data.

TripAdvisor is one of the richest public data sources in the travel industry: over 1 billion reviews, 8+ million business listings, and roughly 460 million unique monthly visitors. It influences more than $60 billion in annual travel spending. But getting that data programmatically? That's where things get tricky. TripAdvisor uses DataDome bot detection, Cloudflare WAF, TLS fingerprinting, and JavaScript challenges — a layered defense stack that blocks most naive scraping attempts before they even start. This guide is the single resource I wish I'd had: a head-to-head comparison of three Python scraping approaches (plus a no-code option), complete code for each, a structured anti-bot troubleshooting section, and reusable patterns that work across hotels, restaurants, and attractions. Whether you're a Python beginner or an experienced developer, this should save you a lot of wasted 403s.

Don't Want to Write Code? Scrape TripAdvisor the Easy Way

I want to be upfront about something. A lot of people searching "scrape TripAdvisor with Python" aren't actually married to the idea of writing code. They just want the data — hotel names, ratings, review counts, prices — in a spreadsheet, fast. If that sounds like you, there's a much shorter path.

Thunderbit is an AI-powered Chrome extension we built that can read any TripAdvisor page and automatically suggest the right columns to extract. The workflow is genuinely two clicks:

1. Open a TripAdvisor listing page (e.g., "Hotels in Paris" search results).
2. Click "AI Suggest Fields" in the Thunderbit sidebar. The AI scans the page and proposes columns like Hotel Name, Rating, Review Count, Price, and Location.
3. Click "Scrape." Thunderbit extracts data from every listing on the page — and handles pagination automatically if you need more results.
4. Export to Excel, Google Sheets, Airtable, or Notion. Exports are free on every plan.

Thunderbit works across hotels, restaurants, and attractions without any configuration changes — the AI adapts to whatever's on the page. For paginated results, it auto-detects "Next" buttons and infinite scroll. And because it runs inside your real Chrome browser, it inherits your session cookies and browser fingerprint, which gives it a natural advantage against bot detection.

You can try it with the Thunderbit Chrome Extension — the free tier gives you 6 pages/month, enough to test the workflow.

If you need programmatic control, custom parsing logic, or plan to scrape 10,000+ pages, Python is the way to go. Keep reading.

Why Scrape TripAdvisor with Python?

TripAdvisor data has direct, measurable business impact. A Cornell University study found that a 1-point increase in a hotel's 100-point Global Review Index leads to a 0.89% increase in average daily rate and a 1.42% increase in Revenue Per Available Room. A separate ScienceDirect study showed that an exogenous 1-star increase in TripAdvisor rating translates to $55,000–$75,000 in additional yearly revenue for an average hotel. Reviews aren't just vanity metrics — they're revenue drivers.

Here's how different teams use TripAdvisor data:

Use Case	Who Benefits	Data Needed
Hotel competitor analysis	Hotel chains, revenue managers	Ratings, prices, review volume, amenities
Restaurant market research	Restaurant groups, food brands	Cuisine types, price ranges, review sentiment
Attraction trend tracking	Tour operators, tourism boards	Popularity rankings, seasonal patterns
Sentiment analysis	Researchers, data analysts	Full review text, star ratings, dates
Lead generation	Sales teams, travel agencies	Business names, contact info, locations

Why Python specifically? Three reasons. First, the ecosystem: BeautifulSoup, Selenium, Playwright, Scrapy, httpx, pandas — Python has more mature scraping and data analysis libraries than any other language. Second, 71.7% of web scraping developers use Python, which means more community support, more StackOverflow answers, and more up-to-date guides. Third, the pipeline advantage: you can scrape with BeautifulSoup, clean with pandas, run sentiment analysis with Hugging Face Transformers, and build dashboards — all in one language. No context switching.

Three Ways to Scrape TripAdvisor with Python (Compared)

Every competing guide picks one approach and runs with it. That's not helpful when you're trying to decide before writing code. Here's the comparison table I wish someone had given me:

Approach	Speed	JS Support	Anti-Bot Resistance	Complexity	Best For
requests + BeautifulSoup	⚡ Fast (~120–200 pages/min raw)	❌ None	⚠️ Low	Easy	Static listing pages, small-scale projects
Selenium / Headless Browser	🐢 Slow (~8–20 pages/min)	✅ Full	⚠️ Medium	Medium	Dynamic content, "Read more" clicks, cookie banners
Hidden JSON / GraphQL API	⚡⚡ Fastest (~200–600 pages/min raw)	N/A	✅ Higher	Hard	Large-scale review/hotel extraction
No-code (Thunderbit)	⚡ Fast	✅ Built-in	✅ Built-in	Easiest	Non-devs, quick one-off exports

A few important caveats. Those raw speeds are theoretical — TripAdvisor's rate limits (~10–15 requests per minute per IP) constrain actual throughput to roughly 10 pages/minute per IP regardless of approach. The hidden JSON method gets you the most data per request, which means fewer total requests and less exposure to rate limiting. Selenium is 5x slower than request-based approaches in real-world benchmarks, but it's the only option when you need to click buttons or render JavaScript.

The rest of this guide walks through all three Python methods with complete code. Pick the one that fits your situation, or combine them (I often use requests+BS4 for listing pages and hidden JSON for detail pages).

Setting Up Your Python Environment

Before diving in, let's get the environment ready. You'll need Python 3.10+ (I recommend 3.12 or 3.13 — all major packages support them with no known issues).

Install everything at once:

pip install requests beautifulsoup4 selenium httpx parsel pandas curl-cffi

Package notes:

• requests (2.33.1) — HTTP requests, requires Python 3.10+
• beautifulsoup4 (4.14.3) — HTML parsing
• selenium (4.43.0) — Browser automation, requires Python 3.10+
• httpx (0.28.1) — Async HTTP client
• parsel (1.11.0) — CSS/XPath selectors (lighter than BS4)
• pandas (3.0.2) — Data export, requires Python 3.11+
• curl_cffi (0.15.0) — TLS fingerprint impersonation (critical for bypassing Cloudflare)

ChromeDriver: If you're using Selenium, good news — since Selenium 4.6+, Selenium Manager automatically downloads and caches the correct ChromeDriver binary. No manual installation needed. It resolves version matching dynamically, so you don't have to worry about Chrome version mismatches.

Virtual environment (recommended):

python -m venv tripadvisor-scraper
source tripadvisor-scraper/bin/activate  # macOS/Linux
tripadvisor-scraper\Scripts\activate     # Windows

Approach 1: Scrape TripAdvisor with Requests and BeautifulSoup

This is the simplest approach. It works well for scraping listing pages (hotel search results, restaurant lists) where the data you need is present in the static HTML. No browser, no JavaScript rendering, minimal resource usage.

Understanding TripAdvisor URL Patterns

TripAdvisor URLs follow predictable patterns by category:

• Hotels: https://www.tripadvisor.com/Hotels-g{locationId}-{Location_Name}-Hotels.html
• Restaurants: https://www.tripadvisor.com/Restaurants-g{locationId}-{Location_Name}.html
• Attractions: https://www.tripadvisor.com/Attractions-g{locationId}-Activities-{Location_Name}.html

Pagination uses the oa (offset anchors) parameter, inserted into the URL. Each page shows 30 results:

• Page 1: base URL (no oa parameter)
• Page 2: Hotels-g187768-oa30-Italy-Hotels.html
• Page 3: Hotels-g187768-oa60-Italy-Hotels.html

For review pages, the offset parameter is or with increments of 10:

• Page 1: Reviews-or0-Hotel_Name.html
• Page 2: Reviews-or10-Hotel_Name.html

To get reviews in all languages, append ?filterLang=ALL to the URL.

Sending Requests with Realistic Headers

TripAdvisor checks headers aggressively. A request with default Python headers gets blocked instantly. You need to mimic a real Chrome browser:

import requests
import time
import random

session = requests.Session()

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Referer": "https://www.tripadvisor.com/",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-CH-UA": '"Google Chrome";v="135", "Not-A.Brand";v="8", "Chromium";v="135"',
    "Sec-CH-UA-Mobile": "?0",
    "Sec-CH-UA-Platform": '"Windows"',
}

session.headers.update(headers)

url = "https://www.tripadvisor.com/Hotels-g187147-Paris_Ile_de_France-Hotels.html"
response = session.get(url)

print(f"Status: {response.status_code}")
print(f"Content length: {len(response.text)} characters")

Key detail: TripAdvisor validates that your User-Agent and Sec-CH-UA Client Hints headers are consistent. If you claim to be Chrome 135 in the User-Agent but your Sec-CH-UA says Chrome 120, you'll get flagged. Always rotate entire header sets together, not individual headers.

Parsing Listings with BeautifulSoup

Once you have a successful response, extract the data using BeautifulSoup. TripAdvisor uses data-automation and data-test-attribute attributes that are more stable than CSS class names (which change frequently):

from bs4 import BeautifulSoup

soup = BeautifulSoup(response.text, "html.parser")

# Find all hotel listing cards
cards = soup.select('div[data-test-attribute="location-results-card"]')

hotels = []
for card in cards:
    # Hotel name
    title_el = card.select_one('div[data-automation="hotel-card-title"]')
    name = title_el.get_text(strip=True) if title_el else None

    # Link to detail page
    link_el = card.select_one('div[data-automation="hotel-card-title"] a')
    link = "https://www.tripadvisor.com" + link_el["href"] if link_el else None

    # Rating
    rating_el = card.select_one('[data-automation="bubbleRatingValue"]')
    rating = rating_el.get_text(strip=True) if rating_el else None

    # Review count
    review_el = card.select_one('[data-automation="bubbleReviewCount"]')
    review_count = review_el.get_text(strip=True).replace(",", "").split()[0] if review_el else None

    hotels.append({
        "name": name,
        "rating": rating,
        "review_count": review_count,
        "url": link,
    })

print(f"Found {len(hotels)} hotels on this page")
for h in hotels[:3]:
    print(h)

A note on selectors: TripAdvisor uses obfuscated CSS class names (like FGwzt, yyzcQ) that change with every site update. The data-automation and data-test-target attributes are far more stable. Always prefer data attributes over class names.

Handling Pagination

To scrape multiple pages, loop through the offset parameter with a polite delay between requests:

import pandas as pd

all_hotels = []
base_url = "https://www.tripadvisor.com/Hotels-g187147-oa{offset}-Paris_Ile_de_France-Hotels.html"

for page in range(5):  # First 5 pages
    offset = page * 30
    url = base_url.format(offset=offset) if page > 0 else "https://www.tripadvisor.com/Hotels-g187147-Paris_Ile_de_France-Hotels.html"

    response = session.get(url)
    if response.status_code != 200:
        print(f"Page {page + 1}: Got status {response.status_code}, stopping.")
        break

    soup = BeautifulSoup(response.text, "html.parser")
    cards = soup.select('div[data-test-attribute="location-results-card"]')

    for card in cards:
        title_el = card.select_one('div[data-automation="hotel-card-title"]')
        name = title_el.get_text(strip=True) if title_el else None
        rating_el = card.select_one('[data-automation="bubbleRatingValue"]')
        rating = rating_el.get_text(strip=True) if rating_el else None
        review_el = card.select_one('[data-automation="bubbleReviewCount"]')
        review_count = review_el.get_text(strip=True).replace(",", "").split()[0] if review_el else None

        all_hotels.append({"name": name, "rating": rating, "review_count": review_count})

    print(f"Page {page + 1}: {len(cards)} hotels found")
    time.sleep(random.uniform(3, 7))  # Random delay to avoid rate limiting

df = pd.DataFrame(all_hotels)
print(f"\nTotal hotels scraped: {len(df)}")

The time.sleep(random.uniform(3, 7)) is important. TripAdvisor's rate limit threshold is roughly 10–15 requests per minute per IP. Going faster than that triggers CAPTCHAs or 429 errors.

Limitations of This Approach

Where does this fall apart? The requests+BS4 approach fails when:

• TripAdvisor serves JavaScript-rendered content (some search result pages require JS)
• Review text is truncated behind "Read more" buttons
• Anti-bot measures escalate to JavaScript challenges or CAPTCHAs
• You need data that only appears after client-side rendering (prices, availability)

For these scenarios, you need either Selenium (Approach 2) or the hidden JSON method (Approach 3).

Approach 2: Scrape TripAdvisor with Selenium (Headless Browser)

Selenium launches a real browser, which means it can render JavaScript, click buttons, handle cookie consent banners, and interact with dynamic content. The cost: it's roughly 5x slower and uses 300–500MB of RAM per browser instance.

Configuring Selenium with Anti-Detection Settings

Out of the box, Selenium is trivially detectable. TripAdvisor's fingerprinting catches it immediately. You need to disable automation flags:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("--headless=new")  # Use new headless mode (Chrome 112+)
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_argument("--window-size=1920,1080")
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option("useAutomationExtension", False)

driver = webdriver.Chrome(options=options)

# Remove webdriver property from navigator
driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
    "source": "Object.defineProperty(navigator, 'webdriver', {get: () => undefined})"
})

Is this enough for TripAdvisor? For small-scale scraping (under 50 pages), this setup with residential proxies usually works. For larger volumes, you may need undetected-chromedriver or nodriver — TripAdvisor's DataDome protection analyzes over 1,000 signals per request, including TLS fingerprints that vanilla Selenium can't spoof.

Scraping Hotel Search Results with Selenium

import time
import random

url = "https://www.tripadvisor.com/Hotels-g187147-Paris_Ile_de_France-Hotels.html"
driver.get(url)

# Wait for hotel cards to load
wait = WebDriverWait(driver, 15)
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'div[data-test-attribute="location-results-card"]')))

# Handle cookie consent popup (if it appears)
try:
    cookie_btn = driver.find_element(By.ID, "onetrust-accept-btn-handler")
    cookie_btn.click()
    time.sleep(1)
except:
    pass  # No cookie popup

# Extract hotel data
cards = driver.find_elements(By.CSS_SELECTOR, 'div[data-test-attribute="location-results-card"]')

hotels = []
for card in cards:
    try:
        name = card.find_element(By.CSS_SELECTOR, 'div[data-automation="hotel-card-title"]').text
    except:
        name = None
    try:
        rating = card.find_element(By.CSS_SELECTOR, '[data-automation="bubbleRatingValue"]').text
    except:
        rating = None
    try:
        reviews = card.find_element(By.CSS_SELECTOR, '[data-automation="bubbleReviewCount"]').text
    except:
        reviews = None

    hotels.append({"name": name, "rating": rating, "review_count": reviews})

print(f"Scraped {len(hotels)} hotels")
for h in hotels[:3]:
    print(h)

This took about 8 seconds for a single page on my machine — compared to under 1 second with requests+BS4. That 8x difference adds up fast when you're scraping hundreds of pages.

Expanding "Read More" and Scraping Full Reviews

Review pages truncate long reviews behind a "Read more" button. Selenium can click it:

review_url = "https://www.tripadvisor.com/Hotel_Review-g187147-d188726-Reviews-Le_Marais_Hotel-Paris_Ile_de_France.html"
driver.get(review_url)

wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'div[data-reviewid]')))
time.sleep(2)

# Click all "Read more" buttons
read_more_buttons = driver.find_elements(By.XPATH, '//button//*[contains(text(), "Read more")]/..')
for btn in read_more_buttons:
    try:
        driver.execute_script("arguments[0].click();", btn)
        time.sleep(0.3)
    except:
        pass

# Extract reviews
review_elements = driver.find_elements(By.CSS_SELECTOR, 'div[data-reviewid]')

reviews = []
for rev in review_elements:
    try:
        title = rev.find_element(By.CSS_SELECTOR, 'div[data-test-target="review-title"]').text
    except:
        title = None
    try:
        body = rev.find_element(By.CSS_SELECTOR, 'q.IRsGHoPm span').text
    except:
        try:
            body = rev.find_element(By.CSS_SELECTOR, 'p.partial_entry').text
        except:
            body = None
    try:
        rating_class = rev.find_element(By.CSS_SELECTOR, 'div[data-test-target="review-rating"] span').get_attribute("class")
        # Rating encoded in class like "ui_bubble_rating bubble_50" = 5.0
        rating_num = [c for c in rating_class.split() if "bubble_" in c][0].replace("bubble_", "")
        rating = int(rating_num) / 10
    except:
        rating = None

    reviews.append({"title": title, "body": body, "rating": rating})

print(f"Scraped {len(reviews)} reviews")

Adding Proxy Rotation to Selenium

For sustained scraping, you'll need proxy rotation. Since selenium-wire has been deprecated since January 2024, use Chrome's built-in proxy support:

# With authentication-free proxy
proxy = "http://your-proxy-address:port"
options.add_argument(f"--proxy-server={proxy}")

# For proxies with authentication, use a Chrome extension or Selenium 4's BiDi protocol

For rotating proxies programmatically, create a new driver instance with a different proxy for each batch of requests. It's not elegant, but it's reliable.

Approach 3: The Hidden JSON Method (Skip HTML Parsing Entirely)

Most guides skip this approach entirely, which is a shame — it's the fastest and cleanest of the three. TripAdvisor embeds structured data as JSON directly in its HTML pages — inside <script> tags as JavaScript variables like pageManifest and urqlCache. Extracting this JSON gives you cleaner data (ratings as numbers, dates in ISO format) with fewer requests and no need for JavaScript rendering.

Finding the Embedded JSON in Page Source

The key insight: you can use a simple requests.get() to fetch the page, then extract the JSON from the raw HTML without ever rendering JavaScript.

import requests
import re
import json

headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Referer": "https://www.tripadvisor.com/",
    "Sec-CH-UA": '"Google Chrome";v="135", "Not-A.Brand";v="8", "Chromium";v="135"',
    "Sec-CH-UA-Mobile": "?0",
    "Sec-CH-UA-Platform": '"macOS"',
}

url = "https://www.tripadvisor.com/Hotel_Review-g188590-d194317-Reviews-NH_City_Centre_Amsterdam.html"
response = requests.get(url, headers=headers)

# Extract the pageManifest JSON blob
match = re.search(r"pageManifest:({.+?})};", response.text)
if match:
    page_data = json.loads(match.group(1))
    print("Found pageManifest data")
    print(f"Keys: {list(page_data.keys())[:10]}")

How to find the variable name yourself: Open any TripAdvisor hotel page in Chrome, right-click → View Page Source, then Ctrl+F for pageManifest or urqlCache or aggregateRating. The data is there, waiting to be parsed.

Parsing the JSON and Extracting Structured Data

TripAdvisor also embeds application/ld+json schema.org data that's even easier to extract:

from parsel import Selector

sel = Selector(text=response.text)

# Extract JSON-LD structured data
json_ld_scripts = sel.xpath("//script[@type='application/ld+json']/text()").getall()

for script in json_ld_scripts:
    data = json.loads(script)
    if isinstance(data, dict) and data.get("@type") in ["Hotel", "Restaurant", "TouristAttraction"]:
        print(f"Name: {data.get('name')}")
        print(f"Rating: {data.get('aggregateRating', {}).get('ratingValue')}")
        print(f"Review Count: {data.get('aggregateRating', {}).get('reviewCount')}")
        print(f"Price Range: {data.get('priceRange')}")
        print(f"Address: {data.get('address', {}).get('streetAddress')}")
        print(f"Coordinates: {data.get('geo', {}).get('latitude')}, {data.get('geo', {}).get('longitude')}")
        break

The JSON-LD data is embedded in static HTML and does NOT require JavaScript rendering. It gives you the property name, aggregate rating, review count, address, coordinates, price range, and photo URLs — all without parsing a single HTML tag.

For richer data (individual reviews, rating breakdowns, amenity lists), you need the urqlCache object:

# Extract urqlCache for detailed review data
cache_match = re.search(r'"urqlCache"\s*:\s*({.+?})\s*,\s*"redux"', response.text)
if cache_match:
    cache_data = json.loads(cache_match.group(1))
    # Navigate the cache to find review data
    for key, value in cache_data.items():
        if "reviews" in str(value).lower()[:100]:
            reviews_data = json.loads(value.get("data", "{}")) if isinstance(value, dict) else None
            if reviews_data:
                print(f"Found review cache entry: {key[:50]}...")
                break

The exact JSON paths change occasionally when TripAdvisor updates its frontend, but the general structure — JSON-LD for summary data, urqlCache for detailed data — has been stable for years.

Reverse-Engineering TripAdvisor's GraphQL API (Advanced)

For large-scale extraction, TripAdvisor's GraphQL endpoints return structured data directly. This is the fastest method but requires the most maintenance.

import httpx
import random
import string

def generate_request_id():
    """Generate the X-Requested-By header value"""
    random_chars = ''.join(random.choices(string.ascii_letters + string.digits, k=180))
    return f"TNI1625!{random_chars}"

# Search for hotels in Paris
search_payload = [{
    "variables": {
        "request": {
            "query": "hotels in Paris",
            "limit": 10,
            "scope": "WORLDWIDE",
            "locale": "en-US",
            "scopeGeoId": 1,
            "searchCenter": None,
            "types": ["LOCATION", "QUERY_SUGGESTION", "RESCUE_RESULT"],
            "locationTypes": ["GEO", "AIRPORT", "ACCOMMODATION", "ATTRACTION", "EATERY", "NEIGHBORHOOD"]
        }
    },
    "extensions": {
        "preRegisteredQueryId": "84b17ed122fbdbd4"
    }
}]

graphql_headers = {
    "Content-Type": "application/json",
    "Accept": "*/*",
    "Accept-Language": "en-US,en;q=0.9",
    "Origin": "https://www.tripadvisor.com",
    "Referer": "https://www.tripadvisor.com/Hotels",
    "X-Requested-By": generate_request_id(),
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36",
}

with httpx.Client() as client:
    response = client.post(
        "https://www.tripadvisor.com/data/graphql/ids",
        json=search_payload,
        headers=graphql_headers
    )
    if response.status_code == 200:
        results = response.json()
        print(json.dumps(results, indent=2)[:1000])
    else:
        print(f"GraphQL request failed: {response.status_code}")

For fetching reviews via GraphQL:

review_payload = [{
    "variables": {
        "locationId": 194317,  # NH City Centre Amsterdam
        "offset": 0,
        "limit": 20,
        "filters": {},
        "sortType": None,
        "sortBy": "date",
        "language": "en",
        "doMachineTranslation": False,
        "photosPerReviewLimit": 3
    },
    "extensions": {
        "preRegisteredQueryId": "ef1a9f94012220d3"
    }
}]

with httpx.Client() as client:
    response = client.post(
        "https://www.tripadvisor.com/data/graphql/ids",
        json=review_payload,
        headers=graphql_headers
    )
    if response.status_code == 200:
        data = response.json()
        reviews = data[0]["data"]["locations"][0]["reviewListPage"]["reviews"]
        total = data[0]["data"]["locations"][0]["reviewListPage"]["totalCount"]
        print(f"Total reviews: {total}")
        for r in reviews[:3]:
            print(f"  [{r['rating']}/5] {r['title']} - {r['createdDate']}")

Important caveat: The preRegisteredQueryId values (like 84b17ed122fbdbd4 for search and ef1a9f94012220d3 for reviews) can break when TripAdvisor redeploys. When they do, your requests will fail silently. You'll need to re-discover the query IDs by monitoring network requests in browser DevTools.

Why This Method Reduces the Need for Proxies

The math is simple. With requests+BS4, scraping 100 hotel detail pages requires 100 requests. With the hidden JSON method, each request returns all the data you need from a single page load — no additional requests for expanding reviews or loading dynamic content. With GraphQL, a single API call can return 20 reviews at once. Fewer requests = less exposure to rate limiting = less need for proxy rotation. For small-to-medium projects (under 1,000 pages), you may not need proxies at all if you add sensible delays.

Scrape Hotels, Restaurants, and Attractions with One Reusable Script

Four out of five competing guides only cover hotels. But TripAdvisor has three core content categories, and the URL patterns and data fields differ between them. Here's how to build one function that handles all three.

Data Fields Available per Category

Field	Hotels	Restaurants	Attractions
Name	✅	✅	✅
Rating	✅	✅	✅
Review count	✅	✅	✅
Price/Price range	✅	✅	Sometimes
Address	✅	✅	✅
Cuisine type	❌	✅	❌
Duration/Tour type	❌	❌	✅
Amenities	✅	❌	❌
Coordinates	✅	✅	✅

Building a Reusable scrape_tripadvisor() Function

import requests
from bs4 import BeautifulSoup
import pandas as pd
import time
import random
import re
import json

def scrape_tripadvisor(category, location_id, location_name, num_pages=3):
    """
    Scrape TripAdvisor listings across hotels, restaurants, or attractions.

    Args:
        category: "hotels", "restaurants", or "attractions"
        location_id: TripAdvisor geo ID (e.g., "187147" for Paris)
        location_name: URL-friendly name (e.g., "Paris_Ile_de_France")
        num_pages: Number of pages to scrape
    """
    url_patterns = {
        "hotels": "https://www.tripadvisor.com/Hotels-g{geo}-oa{offset}-{name}-Hotels.html",
        "restaurants": "https://www.tripadvisor.com/Restaurants-g{geo}-oa{offset}-{name}.html",
        "attractions": "https://www.tripadvisor.com/Attractions-g{geo}-oa{offset}-Activities-{name}.html",
    }

    first_page_patterns = {
        "hotels": "https://www.tripadvisor.com/Hotels-g{geo}-{name}-Hotels.html",
        "restaurants": "https://www.tripadvisor.com/Restaurants-g{geo}-{name}.html",
        "attractions": "https://www.tripadvisor.com/Attractions-g{geo}-Activities-{name}.html",
    }

    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Referer": "https://www.tripadvisor.com/",
        "Sec-CH-UA": '"Google Chrome";v="135", "Not-A.Brand";v="8", "Chromium";v="135"',
        "Sec-CH-UA-Mobile": "?0",
        "Sec-CH-UA-Platform": '"Windows"',
    }

    session = requests.Session()
    session.headers.update(headers)

    all_items = []

    for page in range(num_pages):
        offset = page * 30
        if page == 0:
            url = first_page_patterns[category].format(geo=location_id, name=location_name)
        else:
            url = url_patterns[category].format(geo=location_id, offset=offset, name=location_name)

        response = session.get(url)
        if response.status_code != 200:
            print(f"  Page {page + 1}: Status {response.status_code}, stopping.")
            break

        soup = BeautifulSoup(response.text, "html.parser")
        cards = soup.select('div[data-test-attribute="location-results-card"]')

        for card in cards:
            item = {"category": category}
            title_el = card.select_one('div[data-automation="hotel-card-title"]') or card.select_one('a[data-automation]')
            item["name"] = title_el.get_text(strip=True) if title_el else None
            rating_el = card.select_one('[data-automation="bubbleRatingValue"]')
            item["rating"] = rating_el.get_text(strip=True) if rating_el else None
            review_el = card.select_one('[data-automation="bubbleReviewCount"]')
            item["review_count"] = review_el.get_text(strip=True) if review_el else None
            all_items.append(item)

        print(f"  Page {page + 1}: {len(cards)} items found")
        time.sleep(random.uniform(3, 7))

    return pd.DataFrame(all_items)

# Usage examples
print("=== Hotels in Paris ===")
hotels_df = scrape_tripadvisor("hotels", "187147", "Paris_Ile_de_France", num_pages=2)
print(hotels_df.head())

print("\n=== Restaurants in Rome ===")
restaurants_df = scrape_tripadvisor("restaurants", "187791", "Rome_Lazio", num_pages=2)
print(restaurants_df.head())

print("\n=== Attractions in Barcelona ===")
attractions_df = scrape_tripadvisor("attractions", "187497", "Barcelona_Catalonia", num_pages=2)
print(attractions_df.head())

One function, three categories, zero code duplication. If TripAdvisor changes a selector, you fix it in one place.

What to Do When TripAdvisor Blocks You (Anti-Bot Troubleshooting)

This is the section I needed most when I started scraping TripAdvisor, and it's the section no competing guide provides in a structured way. TripAdvisor uses DataDome (analyzing 5+ trillion data points per day) and Cloudflare WAF together. Here's a diagnostic table for the most common failure modes:

Symptom	Likely Cause	Fix
HTTP 403 response	Missing or suspicious headers; Cloudflare JS challenge	Set realistic User-Agent, Accept-Language, Referer, and Sec-CH-UA headers. Ensure header consistency.
CAPTCHA page instead of data	Rate limiting or browser fingerprinting	Rotate residential proxies, add random delays (2–7 seconds between requests)
Empty HTML or blank page body	JavaScript not rendered by requests	Switch to Selenium or extract from hidden JSON in page source
Partial reviews / "Read more" not expanding	Content loaded on click event	Use Selenium .click() or extract from embedded JSON blob
Reviews only in one language	Missing language parameter	Append ?filterLang=ALL to the review URL
Data stops loading after N pages	Session-based rate limit	Rotate sessions, clear cookies between batches
HTTP 1020 Access Denied	IP/ASN banned by Cloudflare	Switch from datacenter to residential proxies
Challenge loop (infinite CAPTCHA)	Broken cookie persistence	Warm up sessions by visiting homepage first; maintain cookie jar

Retry Logic with Exponential Backoff

No competing article actually shows this code. Here's a reusable retry function:

import time
import random
import requests

def fetch_with_retry(session, url, max_retries=4, base_delay=2, max_delay=60):
    """
    Fetch a URL with exponential backoff and jitter.
    Rotates User-Agent on each retry.
    """
    user_agents = [
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36",
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36",
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36",
    ]

    for attempt in range(max_retries):
        # Rotate User-Agent on retry
        if attempt > 0:
            session.headers["User-Agent"] = random.choice(user_agents)

        try:
            response = session.get(url, timeout=30)

            if response.status_code == 200:
                return response

            if response.status_code == 429:
                # Respect Retry-After header if present
                retry_after = int(response.headers.get("Retry-After", base_delay * (2 ** attempt)))
                print(f"  Rate limited (429). Waiting {retry_after}s...")
                time.sleep(retry_after)
                continue

            if response.status_code in (403, 503):
                wait = min(base_delay * (2 ** attempt) + random.uniform(0, 1), max_delay)
                print(f"  Got {response.status_code}. Retry {attempt + 1}/{max_retries} in {wait:.1f}s...")
                time.sleep(wait)
                continue

            # Other error codes — don't retry
            print(f"  Unexpected status {response.status_code} for {url}")
            return response

        except requests.exceptions.Timeout:
            wait = min(base_delay * (2 ** attempt) + random.uniform(0, 1), max_delay)
            print(f"  Timeout. Retry {attempt + 1}/{max_retries} in {wait:.1f}s...")
            time.sleep(wait)

    print(f"  All {max_retries} retries exhausted for {url}")
    return None

Rotating Headers, Proxies, and Sessions

For sustained scraping, maintain a pool of header sets and rotate them together:

import random

HEADER_SETS = [
    {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36",
        "Sec-CH-UA": '"Google Chrome";v="135", "Not-A.Brand";v="8", "Chromium";v="135"',
        "Sec-CH-UA-Platform": '"Windows"',
    },
    {
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36",
        "Sec-CH-UA": '"Google Chrome";v="135", "Not-A.Brand";v="8", "Chromium";v="135"',
        "Sec-CH-UA-Platform": '"macOS"',
    },
    {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36",
        "Sec-CH-UA": '"Google Chrome";v="134", "Not-A.Brand";v="8", "Chromium";v="134"',
        "Sec-CH-UA-Platform": '"Windows"',
    },
]

PROXY_LIST = [
    "http://user:pass@residential-proxy-1:port",
    "http://user:pass@residential-proxy-2:port",
    # Add more residential proxies
]

def get_rotated_session():
    """Create a new session with rotated headers and proxy."""
    session = requests.Session()

    # Pick a random header set
    header_set = random.choice(HEADER_SETS)
    base_headers = {
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept-Encoding": "gzip, deflate, br",
        "Referer": "https://www.tripadvisor.com/",
        "Sec-Fetch-Dest": "document",
        "Sec-Fetch-Mode": "navigate",
        "Sec-CH-UA-Mobile": "?0",
    }
    base_headers.update(header_set)
    session.headers.update(base_headers)

    # Pick a random proxy
    if PROXY_LIST:
        proxy = random.choice(PROXY_LIST)
        session.proxies = {"http": proxy, "https": proxy}

    return session

Proxy type matters. Datacenter proxies get blocked almost immediately by TripAdvisor (HTTP 1020 Access Denied). Residential proxies are mandatory for sustained scraping — they route through consumer ISPs and are indistinguishable from real users. Expect to pay $2.50–$8.40/GB depending on the provider.

Exporting and Storing Your Scraped TripAdvisor Data

Once you have the data, getting it into a usable format is straightforward.

CSV Export (Most Common)

import pandas as pd

df = pd.DataFrame(all_hotels)
df.to_csv("tripadvisor_hotels_paris.csv", index=False, encoding="utf-8-sig")
print(f"Exported {len(df)} rows to CSV")

The encoding='utf-8-sig' is important — it ensures Excel correctly displays non-Latin characters (French accents, Chinese characters, etc.) when opening the CSV.

JSON Export (For Nested Data)

When you have reviews nested under hotels, JSON preserves the hierarchy:

# Hierarchical structure
hotel_data = {
    "property_id": "d194317",
    "name": "NH City Centre Amsterdam",
    "rating": 4.0,
    "reviews": [
        {"title": "Great location", "rating": 5, "date": "2025-03-15", "text": "..."},
        {"title": "Average stay", "rating": 3, "date": "2025-03-10", "text": "..."},
    ]
}

# For flat analysis, use json_normalize
flat_reviews = pd.json_normalize(
    hotel_data,
    record_path="reviews",
    meta=["property_id", "name"]
)
flat_reviews.to_csv("reviews_flat.csv", index=False)

Two-File Approach for Relational Data

For large datasets, I use two CSV files:

• hotels.csv — One row per property (flat)
• reviews.csv — One row per review, with property_id as a foreign key

This makes it easy to join in pandas, load into a database, or import into BI tools.

If you don't want to deal with any of this export logic, Thunderbit lets you export scraped data directly to Excel, Google Sheets, Airtable, or Notion — all free, all without code. Useful when you need to share results with non-technical teammates.

Tips for Responsible and Efficient TripAdvisor Scraping

Responsible scraping in six bullets:

• Check robots.txt: TripAdvisor's robots.txt blocks AI training bots (GPTBot, ClaudeBot, etc.) entirely. Standard crawlers face selective path restrictions. Review it at tripadvisor.com/robots.txt.
• Add delays: 3–7 seconds between requests is a safe range. Going faster than 10–15 requests per minute per IP triggers rate limiting.
• Scrape only public data. Don't log in to access restricted content.
• Store data securely and comply with GDPR/CCPA if handling personal information (reviewer names, etc.).
• Consider TripAdvisor's official API if you need commercial-scale data. The Developer Portal offers access to business details plus up to 5 reviews and 5 photos per location — limited, but legal and stable.
• Be aware of legal context: The EU Court Ryanair ruling (December 2025) strengthened ToS-based scraping prohibitions across the EU. TripAdvisor's Terms of Service explicitly prohibit scraping. Scrape responsibly and at your own risk.

Wrapping Up

That's the full picture.

• Requests + BeautifulSoup is the simplest path. It works for static listing pages, requires minimal setup, and is fast. Start here if you're scraping fewer than 100 pages and don't need JavaScript-rendered content.
• Selenium handles everything requests can't: dynamic content, "Read more" buttons, cookie banners. It's 5x slower and resource-heavy, but it's the only option when you need to interact with the page.
• Hidden JSON / GraphQL is the cleanest and fastest approach. It gives you structured data without parsing HTML, reduces the number of requests (and therefore the need for proxies), and returns data in analysis-ready formats. It requires more reverse-engineering upfront and occasional maintenance when TripAdvisor changes its data structure.

The reusable scrape_tripadvisor() function covers hotels, restaurants, and attractions. You shouldn't need a second tutorial.

And if you decide mid-tutorial that coding isn't for you — or you just need 50 hotels in a spreadsheet by end of day — Thunderbit's Chrome extension can do it in two clicks with AI-powered field detection, automatic pagination, and free export to Excel or Google Sheets. No Python required.

If you want to go deeper, we have more scraping walkthroughs on the Thunderbit blog and our YouTube channel.

FAQs

1. Is it legal to scrape TripAdvisor?

TripAdvisor's Terms of Service explicitly prohibit scraping. However, courts have generally held that scraping publicly available data (not behind a login) does not violate the Computer Fraud and Abuse Act in the US. That said, the 2025 EU Court Ryanair ruling strengthened ToS-based restrictions in Europe. Scrape only public data, respect robots.txt, don't republish copyrighted content, and consult legal counsel if you're using the data commercially.

2. Can I scrape TripAdvisor without Python?

Yes. No-code tools like Thunderbit can scrape TripAdvisor directly from your browser with AI-powered field detection and automatic pagination. You can also use browser extensions, Google Sheets add-ons, or commercial scraping APIs. Python gives you the most control and flexibility, but it's not the only option.

3. How do I avoid getting blocked when scraping TripAdvisor?

The key tactics: use realistic and consistent headers (especially User-Agent and Sec-CH-UA), rotate residential proxies (datacenter IPs get blocked immediately), add random delays of 3–7 seconds between requests, use the hidden JSON method to minimize total requests, implement retry logic with exponential backoff, and warm up sessions by visiting the homepage before scraping deep pages.

4. What data can I scrape from TripAdvisor?

Hotels, restaurants, and attractions — including names, ratings, review counts, price ranges, addresses, coordinates, amenities (hotels), cuisine types (restaurants), tour durations (attractions), and full review text with individual ratings and dates. The hidden JSON and GraphQL approaches return the richest data per request.

5. How many pages can I scrape from TripAdvisor per day?

With a single IP and sensible delays: roughly 600–1,000 pages per day. With 20 rotating residential proxies: approximately 200,000–300,000 pages per day using request-based approaches. Selenium is slower — expect 8,000–12,000 pages per day per proxy. The hidden JSON/GraphQL approach gets you the most data per request, so you may need far fewer total pages to get the same amount of information.

Learn More

• How To Web Scrape With Python
• Best Automated Web Scraping Tools
• Web Scraping For Price Comparison
• Best Mobile Proxy Providers
• How To Create A Web Scraper