Why Your Scraper Returns an Empty List (and How to Fix It)

You opened devtools, right-clicked the element, copied the selector, pasted it into your scraper, and got back []. The selector is correct. The page is right there in your browser. Nothing is broken.

In the overwhelming majority of cases the cause is one thing:

The page you inspected is not the page your scraper downloaded.

Devtools shows you the DOM — the live document after the browser has fetched the HTML, run every script, and applied whatever those scripts did to the page. requests.get() returns the raw HTML the server sent, before any of that. If the content you want is inserted by JavaScript, it does not exist in what your scraper received, and no selector can find it.

Everything below is a way of narrowing down which version of this you have.

Before changing a single line of code, find out whether the element is in the raw HTML at all.

Fastest: paste the URL into the CSS selector tester and press Fetch HTML. It loads exactly what the server returns — the same bytes requests would get — and runs your selector against it. If the selector matches there, your selector is fine and the problem is elsewhere. If it matches in devtools but not there, the content is JavaScript-rendered.

Or from a terminal:

curl -s https://example.com/page | grep -i "the text you want"

Or in the browser: type view-source:https://example.com/page in the address bar and search it. view-source: shows the original HTML; the Elements panel does not.

If your text is not in the raw HTML, stop debugging the selector. It is not the selector.

1. The content is rendered by JavaScript

The most common cause by a wide margin. Single-page apps built with React, Vue, Angular or Svelte typically ship an almost empty <div id="root"></div> and build the page in the browser.

You have three options, in increasing order of cost:

Find the API instead. Open the Network tab, filter to Fetch/XHR, reload. The data is usually arriving as JSON from an endpoint you can call directly — faster, more stable and easier to parse than any HTML. This is worth ten minutes before you reach for a browser.

Look for embedded state. Many frameworks serialise the initial data into the HTML: __NEXT_DATA__ for Next.js, window.__NUXT__ for Nuxt, or a <script type="application/ld+json"> block. Search the raw HTML for those before assuming it is empty.

Render the page. Playwright or Selenium will run the JavaScript and give you the finished DOM. It is slower and heavier, and it is the correct answer when the first two fail.

2. You are being served a different page because you look like a bot

If the raw HTML is short, or contains the words "Just a moment", "Enable JavaScript and cookies", or a consent wall, you did not get the page — you got a challenge.

The usual causes are a missing or default User-Agent (python-requests/2.x is an obvious tell), no Accept-Language, no cookies from a prior visit, and requests arriving faster than a person could click. Print len(response.text) and the first 500 characters — that tells you immediately whether you have a page or a challenge.

3. The class names are generated

CSS-in-JS builds emit classes like css-1x2y3z or sc-bdVaJa. They change on every deploy, so a selector copied today breaks next week.

Anchor on something the build cannot rename: an id, a data-testid, an ARIA role, or the structure itself (article > h2:first-child). If you must use a generated class, match a stable fragment with contains() in XPath rather than the whole string.

4. The element is inside an iframe

An <iframe> is a separate document with its own URL. A selector on the parent page will never reach inside it, and there is no error to tell you so — just an empty result.

Find the iframe's src and fetch that URL directly. It is usually simpler than the parent page.

5. Copied selectors from devtools are too brittle

"Copy selector" produces things like #root > div:nth-child(2) > div > ul > li:nth-child(4) > a. That describes one element's exact position in one render of one page, and breaks when anything above it shifts.

Write the selector by hand, describing what the element is rather than where it sits: .product-card a.title.

6. Your parser is not the one you think

In BeautifulSoup, html.parser is stricter about malformed markup than lxml, and real-world HTML is frequently malformed. A broken tag partway up the document can cause the parser to silently drop the subtree containing your target.

Try BeautifulSoup(html, "lxml") against BeautifulSoup(html, "html.parser") and compare. If one finds the element and the other does not, that was it.

BeautifulSoup returns an empty list

find_all() returns [] and find() returns None rather than raising, so the failure surfaces later as a NoneType error somewhere else. Check the input first:

r = requests.get(url, headers={"User-Agent": "Mozilla/5.0 (X11; Linux x86_64)"})
print(r.status_code, len(r.text))
print(r.text[:500])          # a page, or a challenge?
print("target text" in r.text)

Scrapy returns an empty list

Same cause, plus two of its own. Scrapy obeys ROBOTSTXT_OBEY = True by default and will silently skip disallowed URLs, so check the log for a robotstxt filter line. And scrapy shell caches the response — use fetch(url) again after changing anything.

scrapy shell "https://example.com/page"
>>> view(response)     # opens what Scrapy actually got, in your browser
>>> response.css(".product::text").getall()

view(response) is the fastest way to see the difference between Scrapy's view of the page and yours.

Selenium or Playwright returns nothing

Here the HTML is usually fine and the timing is not: you queried before the content arrived. Wait for the element rather than sleeping a fixed number of seconds.

page.wait_for_selector(".product", timeout=10000)
items = page.locator(".product").all()

If it still fails, the element may be inside an iframe — use page.frame_locator() — or the site may have detected automation.

lxml or parsel returns nothing

Check for namespaces. On an XHTML or XML document, //div matches nothing until the default namespace is registered, because the parser sees {namespace}div. Either register a prefix or use local-name(): //*[local-name()='div'].

Two habits remove most of these before they cost you an afternoon.

Assert on the input, not the output. A scraper that finds nothing should fail loudly at the point of failure, not return an empty list that something downstream misinterprets as "no results today":

items = soup.select(".product")
if not items:
    raise ValueError(f"No .product on {url} — {len(html)} bytes received")

Without that, a site redesign turns into a database quietly filling with zeroes.

Check the raw HTML first, every time. The habit of loading the URL into a selector tester before writing the parsing code turns a category of afternoon-long debugging sessions into a thirty-second check.

When the page needs a browser, every time

If the answer keeps coming back "the content is JavaScript-rendered", you have crossed the line where a fetch-and-parse scraper is the wrong tool. Rendering the page is only the first cost — you then own browser lifecycle, memory, timeouts, retries, sessions, and the sites that log you out overnight.

This is what Autonoly handles: an AI agent that drives a real browser, so the DOM it sees is the one you see in devtools, and it can log in, wait for content, page through results and write them somewhere useful. When the target is behind a login or renders entirely client-side, that is the difference between a scraper you maintain and one that runs.

Related reading: how anti-bot detection works, handling pagination, and the XPath tester for the cases CSS cannot express.