HTTP errors in web scraping and automation, explained
Last verified
When a script or browser automation fetches a web page, every failure eventually reaches you in one of two forms: an HTTP status code that the server chose to send, or a client-side exception raised because no usable response arrived at all. The first family (403, 429, 503 and friends) is the server telling you something on purpose. The second family (certificate errors, connection resets, redirect loops, "Max retries exceeded") comes from your HTTP library and usually wraps a lower-level network cause.
The most common confusion is that a page works in your browser but not in your script. That difference is real and usually informative. Browsers carry cookies, run JavaScript, use the operating system's certificate store and send a few requests a minute from a residential connection. Scripts often skip all of that, run from cloud IP ranges, and send many requests quickly. Sites notice, and their responses (rate limits, blocks, challenges) are a statement of policy.
Each entry below explains what the code or exception means, what is not wrong, the specific causes, and fixes you can test. The fixes are deliberately the legitimate ones: read robots.txt and the terms of service, slow down and honor Retry-After, use an official API or export when one exists, and ask the site owner for access when you have a real need. If a site has decided not to serve automated clients, the right move is to respect that decision, not to disguise your traffic. Examples use Python requests, but the status codes mean the same thing in any client. Cloudflare-specific block pages are covered in the Cloudflare errors reference.
Official reference: MDN Web Docs - HTTP response status codes
How to read a HTTP error
Read the status code first, then the headers, then the body. With Python requests, raise_for_status() produces messages in this format:
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://example.com/api
# | | | |
# status 4xx=client reason phrase the exact URL that failed
# 5xx=serverFor connection failures, the useful part is after Caused by:
HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: /
(Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] ...')))Also check the Server header (for example cloudflare), Retry-After, and any request or ray ID in the body; site owners need that ID to look up why you were blocked.
All 15 messages
Jump to: Access and blocking (401, 403, 407, 451) (4) · Rate limiting (429) (1) · Wrong URL or method (404, 405) (2) · Server-side failures (500, 502, 503, 504) (4) · Client and connection exceptions (4)
Access and blocking (401, 403, 407, 451)
| Message |
|---|
403 Forbiddenrequests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://example.com/page |
401 Unauthorizedrequests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://api.example.com/v1/data |
407 Proxy Authentication RequiredProxyError('Unable to connect to proxy', OSError('Tunnel connection failed: 407 Proxy Authentication Required')) |
| 451 Unavailable For Legal Reasons |
Rate limiting (429)
| Message |
|---|
429 Too Many Requestsrequests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://example.com/api/items |
Wrong URL or method (404, 405)
| Message |
|---|
404 Not Found and soft 404srequests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://example.com/product/123 |
| 405 Method Not Allowed |
Server-side failures (500, 502, 503, 504)
| Message |
|---|
500 Internal Server Errorrequests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://example.com/search?q=test |
| 502 Bad Gateway |
| 503 Service Unavailable |
| 504 Gateway Timeout |
Client and connection exceptions
| Message |
|---|
SSL: CERTIFICATE_VERIFY_FAILEDssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate… |
ConnectionResetError: Connection reset by peerrequests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) |
TooManyRedirects: Exceeded 30 redirectsrequests.exceptions.TooManyRedirects: Exceeded 30 redirects. |
Max retries exceeded with urlrequests.exceptions.ConnectionError: HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: /page (Caused by… |
Definitions
- CAPTCHACAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) is a challenge-response m
- Cloudflare Bot ManagementCloudflare Bot Management is the set of Cloudflare features (Bot Fight Mode, Super Bot Fight Mode and the Ente
- DataDomeDataDome is a commercial bot-protection service that scores every request to a website, app or API in real tim
- HUMAN Security (PerimeterX)PerimeterX, now HUMAN Security's Bot Defender after a 2022 merger, is a bot-protection service that scores eve
- User AgentA user agent is a string that browsers send with every HTTP request identifying the browser type, version, and
- AI Web ScraperAn AI web scraper is an AI-powered data extraction tool that understands page structure, identifies relevant d
- Data ScrapingData scraping is the broad practice of programmatically extracting data from any digital source — websites, ap
- Web CrawlerA web crawler (also called a spider or bot) is a program that systematically browses the internet by following
Other error directories
- Cloudflare error pages errors16 messages
- Google Apps Script errors16 messages
- Google Sheets IMPORT functions errors15 messages
- Make (formerly Integromat) errors9 messages
- n8n errors11 messages
- Playwright errors16 messages
- Puppeteer errors15 messages
- Selenium WebDriver errors14 messages
- Zapier errors8 messages
Frequently asked questions
Why does a page load in my browser but return 403 to my script?
Sites see different signals from each. Your browser has cookies from earlier visits, runs JavaScript, and makes a few requests from a home or office IP. A script often has no cookies, runs from a cloud IP, and requests many pages quickly. Firewalls and bot-management tools block on those differences. Check robots.txt and the terms, slow down, and use an official API or ask the site for access if you need the data.
Is it legal to scrape a website that returns 403?
A 403 is the site stating that it does not want to serve that request. Legality depends on your jurisdiction, the data, and the site's terms, and this page is not legal advice. Practically, a deliberate block is a strong signal to stop, look for an official API or licensed dataset, or contact the owner. Working around access controls can create legal and contractual risk in many places.
How long should I wait after a 429 Too Many Requests?
If the response includes Retry-After, wait exactly that long: it is either a number of seconds or an HTTP date. If not, back off exponentially, for example 2, 4, 8, then 16 seconds, with a little random jitter, and cap the attempts. Then lower your steady request rate so you stop hitting the limit instead of just recovering from it.
What does robots.txt actually control?
robots.txt, standardized as RFC 9309, tells automated clients which paths the site owner asks them not to crawl. It is a request, not an access control, but reputable crawlers honor it, and ignoring it is often cited when sites block clients or raise legal claims. Read it before building a scraper and honor Crawl-delay where present, even though RFC 9309 does not define that field.
Should I retry on every error?
No. Retry with backoff on 429, 502, 503, 504 and on transient connection errors such as resets and timeouts. Do not retry 400, 401, 403, 404, 405 or 451 automatically: the same request will get the same answer. Fix the request, the credentials or the approach instead, and cap retries so a broken job cannot generate unlimited traffic.
Is it safe to set verify=False to fix SSL errors?
Only as a short local diagnostic, never in production. verify=False turns off certificate checking, so anyone between you and the server can read or alter traffic. Fix the underlying cause instead: update certifi, install the correct root certificate for a corporate proxy, or ask the site to serve a complete certificate chain.
What is a soft 404?
A soft 404 is a page that says the content does not exist but returns 200 OK instead of 404. Google's Search documentation uses the term for these pages. Scrapers that only check status codes save them as real records, so validate content too: look for the site's not-found message or for the fields a valid page must have.
Why do I get 503 or a challenge page instead of the content?
Some sites and CDNs respond to automated-looking traffic with a 503 or an HTML challenge page that only an interactive browser completes. That is the site restricting automation. Check whether it offers an API or data feed, reduce your request rate, and ask the site owner if you need access. Cloudflare's codes are explained in the Cloudflare errors reference.
Retries and timeouts without writing the loop
Autonoly's HTTP Request node and browser steps have per-node retries, timeouts and continue-on-fail, so a transient 429 or 503 can back off instead of failing the run. It has no proxies or IP rotation: if a site blocks automated traffic, that block applies to Autonoly too.