HTTP: 404 Not Found and soft 404s

404 means the server cannot find anything at that URL. It does not say whether the absence is temporary or permanent; RFC 9110 reserves 410 Gone for resources that are deliberately removed.

Scrapers meet two variants. A real 404 is easy: the status code tells you. A soft 404 is harder: the server returns 200 OK with a page that says "not found" or shows an empty search result, so a script that only checks status codes stores junk. Google's Search documentation uses the same term for these pages. The opposite also happens: some sites return 404 to clients they choose not to serve, so a URL that works in the browser can 404 in a script.

What is not wrong: a 404 is not a network or server failure, and retrying will not help unless the URL itself changes.

r = requests.get(url, timeout=30)
if r.status_code == 404 or "Page not found" in r.text:
    mark_missing(url)   # covers real and soft 404s

Generate URLs from sitemaps, feeds or an API rather than by guessing patterns, and validate scraped records (for example, a product page must have a title and price). Track 404 rates per run; a sudden spike usually means the site changed its URL structure.