HTTP: 500 Internal Server Error
Last verified
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://example.com/search?q=testWhat it means
500 is the server's generic "something broke" response: the application hit an unexpected condition and could not complete the request. It is the server's fault by definition, but in automation the trigger is often something unusual about the request.
Scripts tend to send requests that real pages never send: a missing parameter the front end always includes, a malformed JSON body, an out-of-range page number, or a header combination the backend does not expect. Those can crash poorly validated code, producing 500 for your script while the same page works in the browser. Truly transient 500s also happen during deploys or database hiccups.
What is not wrong: you are not blocked. A block uses 403, 429 or a CDN error page.
What it looks like
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://example.com/search?q=testWhy it happens
- A required parameter or header is missing or malformed, and the server fails instead of returning 400.
- Pagination or search parameters beyond what the backend handles.
- A temporary outage, deploy or database problem on the site.
- Very heavy requests (huge page sizes) that time out inside the application.
How to fix it
- Retry once or twice with a delay; if it succeeds, treat it as transient.
- If it fails consistently, compare your request field by field with one the site's own page sends successfully.
- Reduce page size or query complexity.
- Check the site's status page if it has one.
- Report reproducible errors to the API provider with the URL, time and any request ID header.
How to stop it recurring
Use a retry policy with backoff only for 5xx and network errors, cap attempts, and alert on sustained 500 rates. Validate your own requests against the API documentation so malformed input never reaches the server.
Official documentation: developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/500. See all HTTP error messages.
Related errors
Definitions
- CAPTCHACAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) is a challenge-response m
- Cloudflare Bot ManagementCloudflare Bot Management is the set of Cloudflare features (Bot Fight Mode, Super Bot Fight Mode and the Ente
- DataDomeDataDome is a commercial bot-protection service that scores every request to a website, app or API in real tim
- HUMAN Security (PerimeterX)PerimeterX, now HUMAN Security's Bot Defender after a 2022 merger, is a bot-protection service that scores eve
- User AgentA user agent is a string that browsers send with every HTTP request identifying the browser type, version, and
- AI Web ScraperAn AI web scraper is an AI-powered data extraction tool that understands page structure, identifies relevant d
- Data ScrapingData scraping is the broad practice of programmatically extracting data from any digital source — websites, ap
- Web CrawlerA web crawler (also called a spider or bot) is a program that systematically browses the internet by following
Quick answers
What does "500 Internal Server Error" mean in HTTP?
500 is the server's generic "something broke" response: the application hit an unexpected condition and could not complete the request. It is the server's fault by definition, but in automation the trigger is often something unusual about the request.
How do I fix "500 Internal Server Error"?
1. Retry once or twice with a delay; if it succeeds, treat it as transient. 2. If it fails consistently, compare your request field by field with one the site's own page sends successfully. 3. Reduce page size or query complexity. 4. Check the site's status page if it has one. 5. Report reproducible errors to the API provider with the URL, time and any request ID header.
Retries and timeouts without writing the loop
Autonoly's HTTP Request node and browser steps have per-node retries, timeouts and continue-on-fail, so a transient 429 or 503 can back off instead of failing the run. It has no proxies or IP rotation: if a site blocks automated traffic, that block applies to Autonoly too.
Also see Error handling in workflows.