Why Dynamic Pages Come Back Empty
When you request a page with requests, cURL, a no-code HTTP scraper or Google Sheets' IMPORTXML, you get the HTML the server sends before any JavaScript runs. On a dynamic site (React, Vue, Angular, Next.js in client mode, most dashboards, many shops and job boards) that HTML is a shell: a <div id="root"></div> and a few script tags. The browser then runs the scripts, which call an API, receive JSON and build the product cards, prices or table rows you see.
So the data you see in the browser's Inspect panel is not in the document your scraper downloaded. Your selector is fine; it is pointing at something that doesn't exist yet.
The 30-second test
- Open the page, right-click and choose View page source (not Inspect). Press Ctrl+F and search for a value you can see on screen, such as a price or a product name.
- Found it? The data is in the raw HTML. A plain HTTP scraper or IMPORTXML can get it; your problem is the selector or XPath, or the site blocking you.
- Not found? The data is loaded by JavaScript. Use one of the routes below.
A second quick check: in Chrome DevTools press Ctrl+Shift+P, run "Disable JavaScript" and reload. What's left is roughly what a non-browser scraper sees.
Four Ways to Get the Data, Cheapest First
| Route | When it works | Cost |
|---|---|---|
| 1. JSON embedded in the HTML | The values are in View Source inside a <script> tag (Next.js __NEXT_DATA__, application/ld+json, window.__INITIAL_STATE__) | Cheapest: one HTTP request |
| 2. The site's own JSON endpoint | DevTools Network tab shows a Fetch/XHR request returning the data | Cheap and fast, clean data |
| 3. A rendered browser | Everything else, including logins, clicks, scroll-to-load | Slower and heavier, but works on anything a person can see |
| 4. A tool that does 3 for you | You don't want to write or host browser code | Paid, no maintenance |
Route 1: Check for JSON Already in the Page
Many "dynamic" sites are server-rendered and ship their data as JSON inside the page for the JavaScript to pick up. Search View Source for:
<script id="__NEXT_DATA__" type="application/json">on Next.js sites<script type="application/ld+json">, which product, recipe, job and event pages often use for structured data (name, price, availability, rating)window.__INITIAL_STATE__ =,window.__APOLLO_STATE__or similar variables
import json, requests
from bs4 import BeautifulSoup
html = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=20).text
tag = BeautifulSoup(html, "html.parser").find("script", id="__NEXT_DATA__")
data = json.loads(tag.string)
products = data["props"]["pageProps"]["products"] # path differs per site
If this works, you don't need a browser at all.
Route 2: Find the XHR/JSON Endpoint the Page Calls
The page's JavaScript got the data from somewhere. Often you can call that same endpoint directly:
- Open DevTools, go to the Network tab, filter by Fetch/XHR, and reload the page.
- Click through the requests and open the Response or Preview tab. Look for the one containing your products, listings or rows. Scrolling or clicking "Load more" usually triggers the next page's request too.
- Right-click it and choose Copy as cURL to get the URL, query parameters and headers.
- Look for paging parameters (
page=2,offset=40,cursor=...) and change them to walk through the results.
import requests
rows, page = [], 1
while True:
r = requests.get("https://example.com/api/search",
params={"q": "laptops", "page": page},
headers={"Accept": "application/json", "User-Agent": "Mozilla/5.0"},
timeout=20)
items = r.json().get("results", [])
if not items:
break
rows += items
page += 1
Catches: some endpoints need cookies, a CSRF token or a signed header that the page generates in JavaScript; GraphQL APIs need the right query body (copy it from the request payload); and undocumented endpoints can change without notice. Check the site's terms and keep request rates low. If the endpoint won't answer outside the browser, move to route 3.
Route 3: Render the Page in a Real Browser, and Wait Properly
A browser automation library (Playwright, Puppeteer, Selenium) loads the page in Chromium, runs the JavaScript, and lets you read the finished DOM. The single most common bug is reading too early. Wait for the element you need, not for a fixed number of seconds:
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
page.goto("https://example.com/search?q=laptops")
page.wait_for_selector(".product-card") # wait for the data, not a timer
cards = page.locator(".product-card")
rows = [{
"name": c.locator("h2").inner_text(),
"price": c.locator(".price").inner_text(),
"url": c.locator("a").first.get_attribute("href"),
} for c in cards.all()]
browser.close()
Useful wait strategies:
wait_for_selector(".product-card"): the default choice.page.expect_response(lambda r: "/api/search" in r.url): wait for the API call and read its JSON straight from the browser, which combines routes 2 and 3 and gets around tokens the page generates.wait_until="networkidle"ongoto: convenient, but pages with analytics beacons or live updates may never go idle, so use it with a timeout.- Wait for a spinner or "Loading..." text to disappear when results replace a placeholder.
Headless browsers are easier for sites to detect than a normal browser; see anti-bot detection for what to expect. For a library comparison, see Playwright vs Selenium vs Puppeteer.
Infinite Scroll, Load More Buttons and Lazy Loading
Infinite scroll: scroll, wait for the item count to grow, repeat until it stops growing or you hit a cap.
prev = 0
for _ in range(50): # hard cap
page.mouse.wheel(0, 4000)
page.wait_for_timeout(1500)
count = page.locator(".product-card").count()
if count == prev:
break
prev = count
Load more buttons: click while the button is visible, waiting for new items after each click.
btn = page.get_by_role("button", name="Load more")
while btn.is_visible():
before = page.locator(".product-card").count()
btn.click()
page.wait_for_function(f"document.querySelectorAll('.product-card').length > {before}")
Lazy-loaded images and prices often only fill in once the element enters the viewport, so scroll each card into view (scroll_into_view_if_needed()) before reading it. Some feeds also "virtualize" the list, removing cards above the viewport as you scroll. In that case, extract as you go and de-duplicate by URL or ID rather than reading everything at the end.
Before writing any of this, check the Network tab while scrolling: infinite scroll is almost always a paginated JSON endpoint underneath (route 2), and calling it directly is far faster.
Crawling a Dynamic Website (Many Pages, Not One)
Crawling means discovering URLs as well as extracting data. On a JavaScript site, a plain crawler fails twice: the content is missing and so are many of the links, because menus, pagination and product grids are built by scripts. Practical order of attack:
- Start from the sitemap. Check
/robots.txtfor aSitemap:line, or try/sitemap.xml. A sitemap gives you every product or article URL without rendering anything. - Use the listing API for discovery. If category pages load via a JSON endpoint, page through it to collect item URLs or IDs.
- Render only what you must. Use a browser-based crawler for pages where the data truly needs JavaScript. Crawlee's
PlaywrightCrawlerand Scrapy with thescrapy-playwrightplugin both queue URLs, render each page and let you enqueue links found in the rendered DOM. - Budget for it. A rendered page costs far more CPU and memory than an HTTP request, so crawl politely: limit concurrency, honour robots.txt, and cache pages you've already fetched.
Scraping a Dynamic Web Page Into Google Sheets
IMPORTXML, IMPORTHTML and IMPORTDATA fetch the raw HTML from Google's servers and never run JavaScript. On a dynamic page they return "Imported content is empty" or #N/A, even though your XPath matches what you see in Inspect. Apps Script's UrlFetchApp has the same limit: it downloads the HTML, it doesn't render it. (If you're seeing "Could not fetch url" instead, that's a different problem; see IMPORTXML could not fetch url: causes and fixes.)
What does work, in order of effort:
- Embedded JSON (route 1): if the value is in View Source inside a script tag,
=REGEXEXTRACT(IMPORTXML(A1,"//script[@id='__NEXT_DATA__']"),"price.:([0-9.]+)")can sometimes pull it out. Fragile, but free. - JSON endpoint (route 2) via Apps Script:
UrlFetchApp.fetch(apiUrl),JSON.parsethe response and write rows withsheet.getRange(...).setValues(rows). Works when the endpoint doesn't need browser-generated tokens. - A rendered-browser scraper that writes to Sheets (route 3/4): run Playwright yourself and push rows with the Sheets API, or use a tool that renders the page and has a Google Sheets output.
The general guide to scraping a website into Google Sheets covers the formula basics.
Doing It Without Code in Autonoly
Autonoly runs a real Chromium browser in the cloud, so extraction sees the page after JavaScript has rendered it. Two ways to use it:
Ask the agent in chat. For example: "Open example.com/jobs, scroll until no more listings load, extract title, company, location and link for each, and write them to my 'Jobs' Google Sheet." The agent waits for content, scrolls infinite feeds, clicks "Load more" and "Next", and can log in when you give it credentials. It can also fetch a JSON endpoint directly when you point it at one.
Save it as a scheduled workflow. Convert the session, or build it in the visual builder:
- Launch Browser on the page.
- Wait For the item selector (or text) to appear.
- Scroll in progressive mode for infinite feeds; it keeps going until no new content loads.
- Extract Child Collection for repeated cards, Extract Table for tables, or Extract Data for single values. List and table extraction can follow a Next button for pagination.
- Google Sheets (write) to append the rows, then run it on a schedule.
If you found a clean JSON endpoint, the HTTP Request node can call it without a browser at all. See data extraction for how each step works.
Limits, honestly: "Load more" buttons and page-number URLs need one step per page in a saved workflow (there's no loop node yet) or the agent; saved workflows can't sign in with stored credentials, so pages behind a login are an agent task; there's no proxy rotation; and reCAPTCHA v2 and hCaptcha can be solved when enabled, while other challenges are handed to you via live takeover.