HTTP errors in web scraping and automation · error message

HTTP: 401 Unauthorized

ConfigurationHTTP 401Access and blocking (401, 403, 407, 451)

Last verified

The exact message
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://api.example.com/v1/data

What it means

Despite the name, 401 means unauthenticated: the request did not carry valid credentials for this resource. RFC 9110 requires the server to send a WWW-Authenticate header describing the scheme it expects, such as Basic or Bearer. That header is the most useful clue on the response.

In scraping and automation, 401 typically appears when a script calls an API endpoint that the website's own JavaScript calls with a token, or when a session cookie the browser had is missing in the script. Your browser shows the page because it is logged in; the script is not.

What is not wrong: you are not blocked, and the endpoint exists. Compare with 403, where the server knows who you are and still refuses.

What it looks like

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="api", error="invalid_token"

# fix: send the documented header
requests.get(url, headers={"Authorization": f"Bearer {token}"}, timeout=30)

Why it happens

  • No Authorization header, or a header with the wrong scheme (Token instead of Bearer).
  • An API key or OAuth access token that has expired or was revoked.
  • A session cookie that was valid in the browser but was not sent, or has timed out.
  • Credentials for a test environment used against production, or the reverse.

How to fix it

  1. Inspect the WWW-Authenticate response header to see which scheme the server expects.
  2. Check the provider's API documentation for how to obtain credentials, and use an official API key or OAuth flow.
  3. Print the outgoing headers (masking secrets) to confirm the Authorization header is present and correctly formatted.
  4. Refresh or regenerate expired tokens, and handle token refresh in code before retrying.
  5. Only access accounts and data you are authorized to use; do not reuse someone else's session.

How to stop it recurring

Store credentials in a secrets manager or environment variables, validate them at startup with a cheap authenticated call, and implement token refresh before expiry. Prefer documented APIs over replaying browser sessions, which expire unpredictably and may break the site's terms.

Official documentation: developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/401. See all HTTP error messages.

Related errors

Definitions

Quick answers

What does "401 Unauthorized" mean in HTTP?

Despite the name, 401 means unauthenticated: the request did not carry valid credentials for this resource. RFC 9110 requires the server to send a WWW-Authenticate header describing the scheme it expects, such as Basic or Bearer . That header is the most useful clue on the response.

How do I fix "401 Unauthorized"?

1. Inspect the WWW-Authenticate response header to see which scheme the server expects. 2. Check the provider's API documentation for how to obtain credentials, and use an official API key or OAuth flow. 3. Print the outgoing headers (masking secrets) to confirm the Authorization header is present and correctly formatted. 4. Refresh or regenerate expired tokens, and handle token refresh in code before retrying. 5. Only access accounts and data you are authorized to use; do not reuse someone else's session.

Retries and timeouts without writing the loop

Autonoly's HTTP Request node and browser steps have per-node retries, timeouts and continue-on-fail, so a transient 429 or 503 can back off instead of failing the run. It has no proxies or IP rotation: if a site blocks automated traffic, that block applies to Autonoly too.

Also see Error handling in workflows.