Skip to content
Home

/

Blog

/

Web scraping

/

How to Scrape and Crawl Dynamic Websites (Why JavaScript Pages Come Back Empty)

Published · Last updated

12 min read

How to Scrape and Crawl Dynamic Websites (Why JavaScript Pages Come Back Empty)

Why requests, BeautifulSoup and Google Sheets' IMPORTXML return empty results on JavaScript pages, and the four ways to get the data: embedded JSON, the site's own XHR/JSON endpoints, a rendered browser with proper waits, and handling infinite scroll, load-more buttons and JS-built links when crawling.

Autonoly Team

Autonoly Team

AI Automation Experts

scrape dynamic website
crawl dynamic website
scrape dynamic web page google sheets
scrape javascript website
importxml javascript
playwright wait for selector
scrape infinite scroll
xhr json scraping

Why Dynamic Pages Come Back Empty

When you request a page with requests, cURL, a no-code HTTP scraper or Google Sheets' IMPORTXML, you get the HTML the server sends before any JavaScript runs. On a dynamic site (React, Vue, Angular, Next.js in client mode, most dashboards, many shops and job boards) that HTML is a shell: a <div id="root"></div> and a few script tags. The browser then runs the scripts, which call an API, receive JSON and build the product cards, prices or table rows you see.

So the data you see in the browser's Inspect panel is not in the document your scraper downloaded. Your selector is fine; it is pointing at something that doesn't exist yet.

The 30-second test

  1. Open the page, right-click and choose View page source (not Inspect). Press Ctrl+F and search for a value you can see on screen, such as a price or a product name.
  2. Found it? The data is in the raw HTML. A plain HTTP scraper or IMPORTXML can get it; your problem is the selector or XPath, or the site blocking you.
  3. Not found? The data is loaded by JavaScript. Use one of the routes below.

A second quick check: in Chrome DevTools press Ctrl+Shift+P, run "Disable JavaScript" and reload. What's left is roughly what a non-browser scraper sees.

Four Ways to Get the Data, Cheapest First

RouteWhen it worksCost
1. JSON embedded in the HTMLThe values are in View Source inside a <script> tag (Next.js __NEXT_DATA__, application/ld+json, window.__INITIAL_STATE__)Cheapest: one HTTP request
2. The site's own JSON endpointDevTools Network tab shows a Fetch/XHR request returning the dataCheap and fast, clean data
3. A rendered browserEverything else, including logins, clicks, scroll-to-loadSlower and heavier, but works on anything a person can see
4. A tool that does 3 for youYou don't want to write or host browser codePaid, no maintenance

Route 1: Check for JSON Already in the Page

Many "dynamic" sites are server-rendered and ship their data as JSON inside the page for the JavaScript to pick up. Search View Source for:

  • <script id="__NEXT_DATA__" type="application/json"> on Next.js sites
  • <script type="application/ld+json">, which product, recipe, job and event pages often use for structured data (name, price, availability, rating)
  • window.__INITIAL_STATE__ =, window.__APOLLO_STATE__ or similar variables
import json, requests
from bs4 import BeautifulSoup

html = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=20).text
tag = BeautifulSoup(html, "html.parser").find("script", id="__NEXT_DATA__")
data = json.loads(tag.string)
products = data["props"]["pageProps"]["products"]   # path differs per site

If this works, you don't need a browser at all.

Route 2: Find the XHR/JSON Endpoint the Page Calls

The page's JavaScript got the data from somewhere. Often you can call that same endpoint directly:

  1. Open DevTools, go to the Network tab, filter by Fetch/XHR, and reload the page.
  2. Click through the requests and open the Response or Preview tab. Look for the one containing your products, listings or rows. Scrolling or clicking "Load more" usually triggers the next page's request too.
  3. Right-click it and choose Copy as cURL to get the URL, query parameters and headers.
  4. Look for paging parameters (page=2, offset=40, cursor=...) and change them to walk through the results.
import requests
rows, page = [], 1
while True:
    r = requests.get("https://example.com/api/search",
                     params={"q": "laptops", "page": page},
                     headers={"Accept": "application/json", "User-Agent": "Mozilla/5.0"},
                     timeout=20)
    items = r.json().get("results", [])
    if not items:
        break
    rows += items
    page += 1

Catches: some endpoints need cookies, a CSRF token or a signed header that the page generates in JavaScript; GraphQL APIs need the right query body (copy it from the request payload); and undocumented endpoints can change without notice. Check the site's terms and keep request rates low. If the endpoint won't answer outside the browser, move to route 3.

Route 3: Render the Page in a Real Browser, and Wait Properly

A browser automation library (Playwright, Puppeteer, Selenium) loads the page in Chromium, runs the JavaScript, and lets you read the finished DOM. The single most common bug is reading too early. Wait for the element you need, not for a fixed number of seconds:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com/search?q=laptops")
    page.wait_for_selector(".product-card")           # wait for the data, not a timer
    cards = page.locator(".product-card")
    rows = [{
        "name":  c.locator("h2").inner_text(),
        "price": c.locator(".price").inner_text(),
        "url":   c.locator("a").first.get_attribute("href"),
    } for c in cards.all()]
    browser.close()

Useful wait strategies:

  • wait_for_selector(".product-card"): the default choice.
  • page.expect_response(lambda r: "/api/search" in r.url): wait for the API call and read its JSON straight from the browser, which combines routes 2 and 3 and gets around tokens the page generates.
  • wait_until="networkidle" on goto: convenient, but pages with analytics beacons or live updates may never go idle, so use it with a timeout.
  • Wait for a spinner or "Loading..." text to disappear when results replace a placeholder.

Headless browsers are easier for sites to detect than a normal browser; see anti-bot detection for what to expect. For a library comparison, see Playwright vs Selenium vs Puppeteer.

Infinite Scroll, Load More Buttons and Lazy Loading

Infinite scroll: scroll, wait for the item count to grow, repeat until it stops growing or you hit a cap.

prev = 0
for _ in range(50):                                  # hard cap
    page.mouse.wheel(0, 4000)
    page.wait_for_timeout(1500)
    count = page.locator(".product-card").count()
    if count == prev:
        break
    prev = count

Load more buttons: click while the button is visible, waiting for new items after each click.

btn = page.get_by_role("button", name="Load more")
while btn.is_visible():
    before = page.locator(".product-card").count()
    btn.click()
    page.wait_for_function(f"document.querySelectorAll('.product-card').length > {before}")

Lazy-loaded images and prices often only fill in once the element enters the viewport, so scroll each card into view (scroll_into_view_if_needed()) before reading it. Some feeds also "virtualize" the list, removing cards above the viewport as you scroll. In that case, extract as you go and de-duplicate by URL or ID rather than reading everything at the end.

Before writing any of this, check the Network tab while scrolling: infinite scroll is almost always a paginated JSON endpoint underneath (route 2), and calling it directly is far faster.

Crawling a Dynamic Website (Many Pages, Not One)

Crawling means discovering URLs as well as extracting data. On a JavaScript site, a plain crawler fails twice: the content is missing and so are many of the links, because menus, pagination and product grids are built by scripts. Practical order of attack:

  1. Start from the sitemap. Check /robots.txt for a Sitemap: line, or try /sitemap.xml. A sitemap gives you every product or article URL without rendering anything.
  2. Use the listing API for discovery. If category pages load via a JSON endpoint, page through it to collect item URLs or IDs.
  3. Render only what you must. Use a browser-based crawler for pages where the data truly needs JavaScript. Crawlee's PlaywrightCrawler and Scrapy with the scrapy-playwright plugin both queue URLs, render each page and let you enqueue links found in the rendered DOM.
  4. Budget for it. A rendered page costs far more CPU and memory than an HTTP request, so crawl politely: limit concurrency, honour robots.txt, and cache pages you've already fetched.

Scraping a Dynamic Web Page Into Google Sheets

IMPORTXML, IMPORTHTML and IMPORTDATA fetch the raw HTML from Google's servers and never run JavaScript. On a dynamic page they return "Imported content is empty" or #N/A, even though your XPath matches what you see in Inspect. Apps Script's UrlFetchApp has the same limit: it downloads the HTML, it doesn't render it. (If you're seeing "Could not fetch url" instead, that's a different problem; see IMPORTXML could not fetch url: causes and fixes.)

What does work, in order of effort:

  • Embedded JSON (route 1): if the value is in View Source inside a script tag, =REGEXEXTRACT(IMPORTXML(A1,"//script[@id='__NEXT_DATA__']"),"price.:([0-9.]+)") can sometimes pull it out. Fragile, but free.
  • JSON endpoint (route 2) via Apps Script: UrlFetchApp.fetch(apiUrl), JSON.parse the response and write rows with sheet.getRange(...).setValues(rows). Works when the endpoint doesn't need browser-generated tokens.
  • A rendered-browser scraper that writes to Sheets (route 3/4): run Playwright yourself and push rows with the Sheets API, or use a tool that renders the page and has a Google Sheets output.

The general guide to scraping a website into Google Sheets covers the formula basics.

Doing It Without Code in Autonoly

Autonoly runs a real Chromium browser in the cloud, so extraction sees the page after JavaScript has rendered it. Two ways to use it:

Ask the agent in chat. For example: "Open example.com/jobs, scroll until no more listings load, extract title, company, location and link for each, and write them to my 'Jobs' Google Sheet." The agent waits for content, scrolls infinite feeds, clicks "Load more" and "Next", and can log in when you give it credentials. It can also fetch a JSON endpoint directly when you point it at one.

Save it as a scheduled workflow. Convert the session, or build it in the visual builder:

  1. Launch Browser on the page.
  2. Wait For the item selector (or text) to appear.
  3. Scroll in progressive mode for infinite feeds; it keeps going until no new content loads.
  4. Extract Child Collection for repeated cards, Extract Table for tables, or Extract Data for single values. List and table extraction can follow a Next button for pagination.
  5. Google Sheets (write) to append the rows, then run it on a schedule.

If you found a clean JSON endpoint, the HTTP Request node can call it without a browser at all. See data extraction for how each step works.

Limits, honestly: "Load more" buttons and page-number URLs need one step per page in a saved workflow (there's no loop node yet) or the agent; saved workflows can't sign in with stored credentials, so pages behind a login are an agent task; there's no proxy rotation; and reCAPTCHA v2 and hCaptcha can be solved when enabled, while other challenges are handed to you via live takeover.

Frequently Asked Questions

Because the data is added by JavaScript after the page loads. HTTP scrapers and IMPORTXML only receive the initial HTML, which on a dynamic site is mostly an empty shell. Check View page source for a visible value: if it isn't there, use the site's JSON endpoint or a rendered browser.

Related terms, automations and guides

The concepts and workflows referenced in this article.

GuideIMPORTXML "Could Not Fetch URL" and "Imported Content Is Empty": Causes and FixesWhy Google Sheets' IMPORTXML and IMPORTHTML return "Could not fetch url", "Imported content is empty" or endless Loading, how to tell which cause you have in a minute, and what to use when the page is JavaScript-rendered, blocked or behind a login.GuideHow to Scrape Any Website Directly Into Google Sheets Without CodeLearn how to scrape data from any website and send it directly to Google Sheets without writing code. This comprehensive guide covers the end-to-end flow from website to structured spreadsheet data, including handling dynamic sites, pagination, data cleaning, and scheduled runs.GuidePlaywright vs Selenium vs Puppeteer: Which Browser Automation Framework Wins?An in-depth comparison of Playwright, Selenium, and Puppeteer for browser automation and web scraping. Covers architecture, performance, reliability, language support, and when to choose each framework.GuideWeb Scraping with Python: A Practical Guide for BeginnersLearn web scraping with Python from scratch. Covers Requests, BeautifulSoup, Scrapy, and Playwright for extracting data from static and dynamic websites with practical code examples and best practices.GuideAnti-Bot Detection Explained: Why Your Scraper Gets Blocked (and What Actually Helps)How Cloudflare, DataDome, HUMAN and other bot-management systems decide your scraper is a bot, what 403, 429 and challenge pages mean, and the legitimate ways to get blocked less: APIs, rate limits, real browsers and asking for access.GuideThe 7 Best Web Scraping Tools Compared: Features, Pricing, and Use CasesA detailed comparison of the best web scraping tools available, including Autonoly, Apify, Octoparse, Browse AI, ScrapingBee, Bright Data, and Playwright. Covers features, pricing, ease of use, anti-detection capabilities, and which tool fits each use case.DefinitionStructured DataStructured data is information organized in a predefined, predictable format — typically rows and columns in databases, spreadsheets, or tabular files like CSV and JSON. Each field has a defined type and position, making it easy to query, filter, and analyze programmatically.DefinitionPlaywrightPlaywright is Microsoft's open-source browser automation framework that provides a single API to control Chromium, Firefox, and WebKit browsers. It supports headless and headed modes, auto-waiting, and network interception, making it a leading choice for testing and web scraping.TemplateLinkedIn Company Page Scraper AI AgentAutomatically extract comprehensive company information from LinkedIn company pages with this intelligent AI Agent workflow. This template…GuideHow to Scrape Complete Y Combinator Startup Data in 3 Minutes Without Writing a Single Line of CodeLearn how to automatically extract Y Combinator's entire startup database using Autonoly's no-code automation platform. This step-by-step…GuideThe Automation ROI Report: Which Business Functions Pay Back in 30 DaysDiscover which business functions deliver automation ROI in 30 days or less. Data-driven analysis of quick-win automation opportunities…

Put this into practice

Build this workflow in 2 minutes — no code required

Describe what you need in plain English. The AI agent handles the rest.

Free forever up to 100 tasks/month