Get 5000 credits for free when you subscribe to any plan.
Join the beta now 
Back to Blog

How to Scrape a React Single-Page Application (SPA): A Step-by-Step Guide

How to Scrape a React Single-Page Application (SPA): A Step-by-Step Guide

React and other modern JavaScript frameworks have revolutionized web development, creating fast, dynamic user experiences. However, these Single-Page Applications (SPAs) present unique challenges for web scraping. Traditional scraping methods often fail because they don't execute JavaScript or wait for dynamic content to load.

This guide provides a comprehensive approach to extracting data from React SPAs, covering both DIY methods and how Supacrawler's API simplifies the process.

Why Scraping React SPAs Is Challenging

React SPAs differ from traditional websites in several key ways that affect scraping:

  1. Client-side rendering - Content is generated in the browser via JavaScript rather than delivered as HTML from the server
  2. Dynamic content loading - Data often loads asynchronously after the initial page render
  3. State-based UI - Page elements appear, disappear, or change based on application state
  4. Virtual DOM - React's virtual DOM can make element selection tricky as the actual DOM structure changes
  5. API-driven data - Content is often fetched from APIs rather than embedded in the initial HTML

Let's tackle each of these challenges with practical solutions.

Method 1: DIY Approach with Headless Browsers

If you need to build your own scraper for React SPAs, you'll need a headless browser that can execute JavaScript and interact with the page.

Step 1: Set Up a Headless Browser Environment

We'll use Playwright, a modern headless browser automation library, though similar approaches work with Puppeteer or Selenium.

# Install with: pip install playwright
# Initialize with: playwright install
from playwright.sync_api import sync_playwright


def scrape_react_spa(url):
    with sync_playwright() as p:
        # Launch a headless browser
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        
        # Navigate to the SPA
        page.goto(url, wait_until="networkidle")
        
        # Now we can start extracting data...
        
        browser.close()

The wait_until="networkidle" parameter is crucial - it tells Playwright to wait until network connections are idle, which helps ensure dynamic content has loaded.

Step 2: Wait for Content to Load

React SPAs often load content asynchronously. You need to wait for specific elements to appear before extracting data:

# Wait for a specific selector to appear
page.wait_for_selector('.product-list')


# Or wait for a specific network request to complete
page.wait_for_response("**/api/products")


# You can also set a custom timeout (in milliseconds)
page.wait_for_selector('.product-list', timeout=10000)

Step 3: Handle Pagination and Infinite Scrolling

Many React SPAs use infinite scrolling or dynamic pagination. Here's how to handle it:

# For infinite scrolling, scroll down until no more content loads
last_height = page.evaluate("document.body.scrollHeight")
while True:
    # Scroll to bottom
    page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
    
    # Wait for potential new content to load
    page.wait_for_timeout(2000)
    
    # Calculate new scroll height
    new_height = page.evaluate("document.body.scrollHeight")
    
    # Break the loop if no new content loaded
    if new_height == last_height:
        break
    
    last_height = new_height

For pagination with "Load More" buttons:

# Keep clicking "Load More" until it disappears
while True:
    if page.query_selector('.load-more-button'):
        page.click('.load-more-button')
        page.wait_for_timeout(2000)  # Wait for content to load
    else:
        break

Step 4: Extract Data from the Rendered DOM

Once content is loaded, you can extract data using selectors:

# Extract product data from a list
products = []
product_elements = page.query_selector_all('.product-card')


for element in product_elements:
    product = {
        'name': element.query_selector('.product-name').inner_text(),
        'price': element.query_selector('.product-price').inner_text(),
        'image': element.query_selector('img').get_attribute('src'),
        'rating': element.query_selector('.rating')?.inner_text() or 'N/A'
    }
    products.append(product)


return products

Step 5: Handle State Changes and User Interactions

Sometimes you need to interact with the SPA to reveal data:

# Click a tab to reveal different content
page.click('.tab-reviews')
page.wait_for_selector('.review-list')


# Extract reviews after tab switch
reviews = []
review_elements = page.query_selector_all('.review-item')
for element in review_elements:
    review = {
        'author': element.query_selector('.author').inner_text(),
        'text': element.query_selector('.review-text').inner_text(),
        'rating': element.query_selector('.stars').get_attribute('data-rating')
    }
    reviews.append(review)

Step 6: Handle Authentication (If Needed)

Many SPAs require authentication. Here's how to handle login:

# Navigate to login page
page.goto('https://example.com/login')


# Fill in login form
page.fill('#email', '[email protected]')
page.fill('#password', 'your-password')
page.click('#login-button')


# Wait for redirect or dashboard to load
page.wait_for_selector('.dashboard')


# Now you can scrape authenticated content

Complete DIY Example: Scraping a React Product Catalog

Here's a complete example that ties everything together:

from playwright.sync_api import sync_playwright
import json


def scrape_react_product_catalog(url):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        
        # Navigate and wait for initial load
        page.goto(url, wait_until="networkidle")
        page.wait_for_selector('.product-grid')
        
        # Handle filtering (if needed)
        page.click('.filter-dropdown')
        page.click('.filter-option-bestsellers')
        page.wait_for_selector('.product-grid')  # Wait for filtered results
        
        # Handle infinite scroll
        last_height = page.evaluate("document.body.scrollHeight")
        product_count = 0
        
        # Keep scrolling until we have enough products or no more load
        while True:
            current_count = page.evaluate("document.querySelectorAll('.product-card').length")
            
            if current_count > product_count:
                product_count = current_count
                print(f"Found {product_count} products so far...")
                
                if product_count >= 100:  # Stop after collecting 100 products
                    break
                    
                # Scroll down
                page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
                page.wait_for_timeout(2000)
                
                # Check if page height increased
                new_height = page.evaluate("document.body.scrollHeight")
                if new_height == last_height:
                    break  # No more content loaded
                last_height = new_height
            else:
                break  # No new products loaded
        
        # Extract product data
        products = []
        product_elements = page.query_selector_all('.product-card')
        
        for element in product_elements:
            try:
                # Extract basic info
                name = element.query_selector('.product-name').inner_text()
                price = element.query_selector('.product-price').inner_text()
                
                # Click to reveal more details (modal popup)
                element.click()
                page.wait_for_selector('.product-modal')
                
                # Get additional details from modal
                description = page.query_selector('.product-description').inner_text()
                features = [li.inner_text() for li in page.query_selector_all('.feature-list li')]
                
                # Close modal
                page.click('.modal-close')
                page.wait_for_selector('.product-modal', state='hidden')
                
                products.append({
                    'name': name,
                    'price': price,
                    'description': description,
                    'features': features
                })
            except Exception as e:
                print(f"Error extracting product: {e}")
                continue
        
        browser.close()
        return products


# Run the scraper
if __name__ == "__main__":
    results = scrape_react_product_catalog('https://example.com/products')
    
    # Save results to JSON file
    with open('products.json', 'w') as f:
        json.dump(results, f, indent=2)
    
    print(f"Scraped {len(results)} products successfully!")

Method 2: Using Supacrawler API (The Simpler Approach)

While the DIY approach gives you complete control, it requires significant development and maintenance effort. Supacrawler's API provides a much simpler solution by handling all the complex parts for you.

Step 1: Set Up Supacrawler

First, install the Supacrawler SDK:

pip install supacrawler

Then initialize the client:

from supacrawler import SupacrawlerClient
import os


# Initialize client with API key
client = SupacrawlerClient(api_key=os.environ.get('SUPACRAWLER_API_KEY'))

Step 2: Scrape a React SPA with a Single API Call

With Supacrawler, you can extract content from a React SPA with a single API call:

# Scrape a React SPA with JavaScript rendering enabled
response = client.scrape(
    url="https://example.com/react-app",
    render_js=True,  # Enable JavaScript rendering
    wait_for=".product-grid"  # Wait for specific element to appear
)


# Access the extracted content
content = response.content  # Markdown content
html = response.html  # HTML content (if requested)
title = response.title  # Page title

Step 3: Extract Specific Data with CSS Selectors

You can use CSS selectors to extract specific data:

# Extract product information using selectors
response = client.scrape(
    url="https://example.com/react-app",
    render_js=True,
    selectors={
        "products": {
            "selector": ".product-card",
            "multiple": True,
            "fields": {
                "name": ".product-name",
                "price": ".product-price",
                "image": "img.product-image@src",
                "rating": ".rating@data-score"
            }
        }
    }
)


# Access structured data
products = response.data.get("products", [])
for product in products:
    print(f"Product: {product['name']} - {product['price']}")

Step 4: Handle Pagination and Infinite Scrolling

Supacrawler can automatically handle pagination and infinite scrolling:

# Handle infinite scrolling
response = client.scrape(
    url="https://example.com/react-app",
    render_js=True,
    scroll_to_bottom=True,  # Automatically scroll to load more content
    max_scroll_attempts=10  # Maximum number of scroll attempts
)


# Handle "Load More" button pagination
response = client.scrape(
    url="https://example.com/react-app",
    render_js=True,
    pagination={
        "button_selector": ".load-more-button",
        "max_clicks": 5,
        "wait_for": ".product-card"
    }
)

Step 5: Handle Authentication

Supacrawler can also handle authentication for protected SPAs:

# Login and then scrape authenticated content
response = client.scrape(
    url="https://example.com/dashboard",
    render_js=True,
    authentication={
        "type": "form",
        "login_url": "https://example.com/login",
        "username_selector": "#email",
        "password_selector": "#password",
        "username": "[email protected]",
        "password": "your-password",
        "submit_selector": "#login-button"
    }
)

Complete Supacrawler Example: Scraping a React Product Catalog

Here's the same product catalog scraping example, but using Supacrawler:

from supacrawler import SupacrawlerClient
import os
import json


# Initialize client
client = SupacrawlerClient(api_key=os.environ.get('SUPACRAWLER_API_KEY'))


# Scrape product catalog
response = client.scrape(
    url="https://example.com/products",
    render_js=True,
    wait_for=".product-grid",
    scroll_to_bottom=True,
    max_scroll_attempts=10,
    selectors={
        "products": {
            "selector": ".product-card",
            "multiple": True,
            "fields": {
                "name": ".product-name",
                "price": ".product-price",
                "image_url": "img.product-image@src",
                "rating": ".rating@data-score"
            }
        }
    }
)


# Save results to JSON file
products = response.data.get("products", [])
with open('products.json', 'w') as f:
    json.dump(products, f, indent=2)


print(f"Scraped {len(products)} products successfully!")

Notice how much simpler this is compared to the DIY approach - just 20 lines of code versus over 100!

Advanced Techniques for Complex React SPAs

Intercepting Network Requests

Sometimes it's easier to intercept the API calls that the React app makes rather than scraping the rendered HTML:

DIY Approach:

# Intercept network requests with Playwright
page.route("**/api/products", lambda route: 
    print(f"API Request intercepted: {route.request.url}")
    route.continue_()  # Let the request continue
)


# Or capture the response data
responses = []
page.on("response", lambda response: 
    if "api/products" in response.url:
        try:
            responses.append(response.json())
        except:
            pass
)

Supacrawler Approach:

# Intercept API requests with Supacrawler
response = client.scrape(
    url="https://example.com/products",
    render_js=True,
    intercept_requests={
        "patterns": ["**/api/products*"],
        "include_responses": True
    }
)


# Access intercepted API responses
api_data = response.intercepted_requests

Handling React Router Navigation

Many React SPAs use React Router for navigation, which doesn't trigger full page loads:

DIY Approach:

# Navigate using React Router links
page.click('a[href="/products/category/electronics"]')
page.wait_for_selector('.category-title:has-text("Electronics")')


# Extract data from the new "page"
category_products = page.query_selector_all('.product-card')

Supacrawler Approach:

# Navigate within SPA and extract data
response = client.scrape(
    url="https://example.com/products",
    render_js=True,
    spa_navigation=[
        {
            "click": 'a[href="/products/category/electronics"]',
            "wait_for": '.category-title:has-text("Electronics")',
            "extract_as": "electronics_products",
            "selectors": {
                "products": {
                    "selector": ".product-card",
                    "multiple": True,
                    "fields": {
                        "name": ".product-name",
                        "price": ".product-price"
                    }
                }
            }
        }
    ]
)


# Access data from different "pages"
electronics = response.data.get("electronics_products", {}).get("products", [])

Common Challenges and Solutions

Challenge 1: Detecting When Content Has Fully Loaded

React SPAs often don't trigger standard page load events when content changes.

Solution: Wait for specific elements that indicate the content has loaded, or monitor network activity:

# DIY: Wait for a loading indicator to disappear
page.wait_for_selector('.loading-spinner', state='hidden')


# Supacrawler: Use the wait_for parameter
response = client.scrape(
    url="https://example.com/app",
    render_js=True,
    wait_for=".content-loaded",
    wait_for_timeout=10000  # 10 seconds max
)

Challenge 2: Dealing with AJAX Filters and Facets

Many React SPAs use AJAX to filter content without page reloads.

Solution: Interact with filters and wait for content updates:

# DIY approach
page.click('.filter-price-range')
page.click('.price-option-medium')
page.wait_for_response("**/api/products?filter=price-medium")


# Supacrawler approach
response = client.scrape(
    url="https://example.com/products",
    render_js=True,
    interactions=[
        {"click": ".filter-price-range"},
        {"click": ".price-option-medium"},
        {"wait_for_response": "**/api/products?filter=price-medium"}
    ]
)

Challenge 3: Handling Lazy-Loaded Images

React SPAs often use lazy loading for images.

Solution: Scroll through the page to trigger image loading:

# DIY approach
page.evaluate("window.scrollBy(0, 1000)")  # Scroll down 1000px
page.wait_for_timeout(1000)  # Wait for images to load


# Supacrawler approach
response = client.scrape(
    url="https://example.com/gallery",
    render_js=True,
    scroll_behavior="smooth",  # Gradually scroll to trigger lazy loading
    wait_after_scroll=1000  # Wait 1 second after each scroll
)

Best Practices for Scraping React SPAs

  1. Respect robots.txt - Check if scraping is allowed on the target website
  2. Add delays between requests - Avoid overloading the server with too many requests
  3. Use a realistic user agent - Some sites block requests with non-standard user agents
  4. Handle errors gracefully - React SPAs can be unpredictable, so implement robust error handling
  5. Cache results when possible - Avoid unnecessary repeat scraping of the same content
  6. Monitor for site changes - React SPAs can change frequently, breaking your scraper

Conclusion

Scraping React SPAs presents unique challenges due to their dynamic, JavaScript-driven nature. While you can build your own solution using headless browsers like Playwright, Supacrawler's API provides a much simpler approach by handling the complex parts for you.

Whether you choose the DIY route or Supacrawler's API, the key is to understand how React SPAs work and adapt your scraping strategy accordingly. With the techniques covered in this guide, you'll be able to extract data from even the most complex React applications.

Ready to try it yourself? Sign up for a Supacrawler account and start scraping React SPAs with just a few lines of code.

Additional Resources

By Supacrawler Team
Published on August 26, 2025