Why Your Scraper Gets Blocked: The Short Answer
Websites block scrapers for three reasons: volume (too many requests too fast), identity (the request does not look like a normal browser, or comes from an IP range known for automation) and policy (the site owner has decided not to allow automated access to that content). Bot-management services such as Cloudflare, DataDome, HUMAN (formerly PerimeterX), Akamai and Kasada combine dozens of signals into a score and then allow, challenge or block the request.
This guide explains how that detection works and what legitimately reduces blocks. It deliberately does not give recipes for defeating protections a site has put in place to keep you out. Those techniques (residential proxy rotation to dodge IP bans, fingerprint spoofing, CAPTCHA farms) raise legal risk, break constantly and are usually unnecessary: most blocks on ordinary scraping jobs come from going too fast or using a client that cannot run the page's JavaScript.
What "blocked" looks like
| Symptom | Usual cause | First thing to try |
|---|---|---|
429 Too Many Requests | Rate limit | Slow down, honour Retry-After, back off |
403 Forbidden on the first request | Default library User-Agent, datacenter IP, missing headers, geo restriction | Check the response body; try in a normal browser from the same machine |
403/503 with "Just a moment..." or "Checking your browser" | JavaScript challenge (Cloudflare and others) | The page needs a real browser that runs JavaScript; an HTTP client cannot pass it |
| A CAPTCHA or Turnstile widget | Risk score is borderline | Slow down; reduce volume; check for an API |
200 OK but empty, partial or different data | Content rendered by JavaScript, or a soft block serving decoy content | Compare with the page in your browser; render with a browser or call the site's JSON endpoint |
| Redirect to a login page | Content requires an account, or your session expired | Check the terms; logged-in scraping carries contract risk |
| Worked for a week, now always blocked | Your IP or pattern was flagged; the site tightened its rules | Stop, lower the rate, and consider asking for access |
Hammering a block with retries makes it worse: it confirms to the scoring model that the traffic is automated.
How Bot Detection Works
Bot management runs at several layers. Knowing them explains most blocks.
1. Request volume and pattern
The oldest and still the most common check. Too many requests per IP per minute, perfectly regular intervals, hitting thousands of product URLs without ever loading a category page, or requesting pages in the exact same order every run all stand out. Many sites also detect bursts: ten fast requests and a pause still look automated.
2. IP reputation
Every IP has a reputation. Addresses from cloud providers and hosting companies are treated with suspicion on many sites because real visitors rarely browse from a data center. IPs that recently sent abusive traffic are scored lower. This is why the same script can work from your laptop and fail from a server.
3. Client fingerprint
Before any HTML is sent, the TLS handshake and HTTP/2 settings reveal what kind of client is connecting (fingerprints known as JA3/JA4). Python's requests, curl and Node's HTTP clients look different from Chrome. A request claiming to be Chrome in its User-Agent but with a non-browser TLS fingerprint is an easy catch. The default User-Agents of HTTP libraries (python-requests/2.x) are blocked outright by many sites.
4. Browser environment
On pages protected by a JavaScript challenge, a script inspects the browser: whether navigator.webdriver is set, whether the reported platform, screen, fonts, timezone and language are consistent with each other and with the IP, how canvas and WebGL render, and whether the browser behaves like a headless build. The challenge result is usually stored in a cookie (for example cf_clearance, _px or datadome) that later requests must carry.
5. Behaviour over the session
Some systems score the whole session: time between page load and first click, scrolling, mouse movement, navigation paths, and whether the session suddenly switches from browsing to systematic extraction.
6. Challenges
When the score is uncertain, the site asks for proof: a JavaScript proof-of-work, Cloudflare Turnstile (often invisible), reCAPTCHA or hCaptcha. When it is clearly bad, the site blocks without asking. See CAPTCHA for how the main types differ.
No single setting gets you past this, because the score combines everything. That is also why the reliable fixes are about behaving like legitimate traffic rather than imitating it.
What Legitimately Reduces Blocks
In rough order of how often they solve the problem:
1. Use the official route if there is one
Check for a public API, a data export, an RSS or Atom feed, a sitemap, or a bulk dataset before scraping. Many sites that block scrapers aggressively offer exactly the data you want through an API with clear limits. It is faster, more stable and comes with permission.
2. Slow down and back off
Most blocks on small and medium jobs are rate limits. One request at a time, a randomised delay of a few seconds between pages, and proper handling of 429 responses fix more scrapers than any other change:
import random, time, requests
def polite_get(session, url, max_tries=5):
for attempt in range(max_tries):
r = session.get(url, timeout=30)
if r.status_code not in (429, 503):
return r
# Honour Retry-After if present, otherwise back off exponentially
wait = r.headers.get("Retry-After")
delay = int(wait) if wait and wait.isdigit() else 2 ** attempt * 5
time.sleep(delay + random.uniform(0, 2))
r.raise_for_status()
s = requests.Session()
s.headers["User-Agent"] = "acme-price-check/1.0 (+https://acme.example/bot; [email protected])"
for url in urls:
page = polite_get(s, url)
time.sleep(random.uniform(2, 4)) # one page every few seconds, one at a timeAlso fetch less: cache pages you already have, use If-Modified-Since/ETag where the site supports them, and on scheduled runs only fetch what changed (new listings, the first few pages), not the whole site every time.
3. Use a real browser when the site needs one
If the content is rendered by JavaScript or sits behind a JavaScript check that all visitors get, an HTTP client will never see it, however you configure it. A real browser automated with Playwright, Puppeteer or Selenium runs the page as any visitor's browser does. This is not about hiding: it is about being able to load the page at all. See how to scrape dynamic websites. A browser does not entitle you to get past a challenge that is meant to stop automation; if a site keeps challenging a slow, normal browser session, read that as a no.
4. Identify yourself
For crawlers and monitoring jobs, a descriptive User-Agent with a URL or contact address (as in the code above) lets site operators reach you instead of blocking you. Search engines and many legitimate crawlers do exactly this. Some bot-management providers, Cloudflare among them, run verified-bot programmes for well-behaved automated traffic.
5. Respect robots.txt and the terms
Paths disallowed in robots.txt are often the ones most tightly protected. Staying off them, and off pages that require you to log in, avoids a lot of blocks and most of the legal risk. See is web scraping legal? for why logged-in scraping and ignoring a cease-and-desist are where the trouble starts.
6. Ask for access
If the data matters to your business, email the site. Partners, researchers and customers regularly get API keys, allowlisted IPs or data feeds simply by asking, especially if they explain the volume and purpose. It is the only fix that is permanent.
What We Don't Recommend, and Why
Search results for this topic are full of guides on rotating residential proxies, stealth plugins, fingerprint spoofing, synthetic mouse movement and CAPTCHA-solving farms. Many are written by companies that sell those services. We do not recommend them for getting past a site that is actively blocking you:
- Legal exposure. Public-data scraping is on reasonably solid ground in the US, but continuing after a site has blocked you or sent a cease-and-desist is exactly where contract, trespass and unauthorized-access claims become stronger. In the EU, personal data stays regulated regardless of how you obtained it.
- It is an arms race you pay for. Detection vendors update continuously; evasion setups that work this month stop working next month, and maintaining them often costs more than the data is worth.
- Residential proxies route your traffic through other people's connections, and how those people consented is not always clear.
- It hides the real problem. If you need evasion to get the data, the site has said no. The durable options are an API, a licence, a data vendor that has an agreement with the source, or a different source.
There is a difference between making your scraper behave well enough not to be mistaken for an attack (slow, consistent, identifiable, using a real browser where one is needed) and disguising it to defeat a deliberate block. This guide is about the first.
If You Want to Block Web Scraping on Your Own Site
If you landed here from the other side, the same layers work in reverse:
- Rate-limit per IP and per session at your CDN or reverse proxy, and return
429withRetry-Afterso polite clients slow down. - Turn on bot management in your CDN or WAF (Cloudflare, Akamai, Fastly and others offer it), and challenge rather than block uncertain traffic so real users are not locked out.
- Put valuable data behind an account with clickwrap terms that prohibit scraping; that gives you both a technical and a contractual control.
- Use robots.txt to tell well-behaved crawlers, including AI crawlers, what they may fetch. It will not stop bad actors, but it sets your position clearly.
- Offer an API or export with keys and limits. Many scrapers would rather use it, and you get to set the terms.
What Autonoly Does and Doesn't Do
Autonoly is a browser automation tool, not an anti-detection service. Concretely:
- Each agent session and workflow run uses a real Chromium browser in its own container, automated with Playwright through patchright (a build that removes a common automation fingerprint) and running headed on a virtual display. Each session keeps one consistent user agent, timezone and locale. That lets it load JavaScript-heavy pages the way a normal browser does. Details are on the browser automation page.
- It runs from Autonoly's cloud infrastructure. There is no proxy rotation and no residential IPs, so sites with aggressive bot detection may still block it.
- When CAPTCHA solving is enabled, reCAPTCHA v2 and hCaptcha are solved automatically. Other challenges, including Cloudflare Turnstile, are not: the agent pauses and offers Take Control in the live view so a person can deal with it, or stop.
- You control the pace and volume: what runs, how often a workflow is scheduled, and which pages it visits.
That makes it a good fit for sites that simply need a real browser and a sensible pace, such as dashboards, portals and JavaScript-rendered listings, and a poor fit for getting data out of sites that are determined to keep automation out. For those, use their API or ask for access.