HTTP: 429 Too Many Requests

429 is defined in RFC 6585: the client has sent too many requests in a given amount of time. It is the clearest message a site can send to an automated client, because it says exactly what to change. The server may include a Retry-After header, either as a number of seconds or as an HTTP date, telling you when you may try again.

A browser rarely sees 429 because a person clicks a few pages a minute; a script that loops over hundreds of URLs can cross the threshold in seconds. Limits can be counted per IP, per API key, per account or per endpoint, and many APIs return headers such as X-RateLimit-Remaining so you can see your budget.

What is not wrong: your request is valid and you are not banned. Keep sending at the same pace, though, and many sites escalate to a 403 or a longer block.

import time, requests

def get(url, attempts=5):
    for i in range(attempts):
        r = requests.get(url, timeout=30)
        if r.status_code != 429:
            return r
        wait = r.headers.get("Retry-After")
        time.sleep(int(wait) if wait and wait.isdigit() else 2 ** (i + 1))
    r.raise_for_status()

Design for the limit up front: a per-host rate limiter, a small worker pool, caching, and conditional requests (If-Modified-Since, ETag) so unchanged pages are not downloaded again. Log rate-limit headers so you see the budget shrinking before it runs out. Behind Cloudflare, the equivalent block page is error 1015.