How to Retry with Backoff in a Web Scraper

· Updated:

Retry with backoff means that after a temporary failure your scraper waits, then tries again, and waits longer after each further failure, with some randomness (jitter) so many workers do not retry at the same moment. Retry only errors that can be temporary, honor the server's Retry-After header on 429 and 503, cap the number of attempts, and stop sending to a site that keeps failing. The examples below use Python requests.

Stable static IPs for well-behaved scrapers

What retry with backoff does in a scraper

Networks drop connections, servers restart and busy sites shed load. A scraper that gives up on the first error loses data; one that retries immediately in a tight loop adds load to a site that is already struggling and can turn a short incident into a long one. Backoff sits between the two: each retry waits longer than the last, so a brief glitch costs a second or two while a real outage costs you time rather than the site.

Backoff is part of rate limiting, not a replacement for it. Your normal pace for each site should already respect its documented limits and robots.txt. Retries handle the exceptions, and a 429 is the site telling you that your normal pace was too fast.

Exponential backoff

The wait doubles after each failed attempt: with a base of two seconds you wait about 2, 4, 8 and 16 seconds. Always set a cap on the single wait and a maximum number of attempts, otherwise a long outage leaves workers sleeping for hours or retrying forever.

Why jitter matters

If twenty workers fail at the same second and all compute the same wait, they all retry at the same second too, and the site sees the same spike again. Jitter spreads them out. With "full jitter" each worker sleeps a random time between zero and the exponential value for that attempt, which spreads retries evenly across the window.

Which errors to retry: 429, 503 and the rest

Retry only failures that can go away on their own. Errors that describe the request itself (a missing page, a forbidden path, bad credentials) will fail again, so log them and move on. A 403 in particular usually means the site does not want this client on that resource; treat it as a stop signal, not as something to retry from a different IP or with different headers.

ResponseUsual meaningWhat the scraper should do
429 Too Many RequestsYou exceeded the site's rate limitWait at least Retry-After, then continue at a slower pace for that site
503 Service UnavailableOverload or maintenanceWait for Retry-After if present, otherwise back off; pause the host if it repeats
500, 502, 504Server or gateway problemRetry a few times with backoff, then log and requeue
Connection error or timeoutNetwork problem on either sideRetry with backoff; check your proxy if it affects every site
400, 401, 403, 404, 410Problem with the request or accessDo not retry; log it and review

Reading Retry-After

Retry-After holds either a number of seconds or an HTTP date. When it is present, never retry sooner than it says, even if your own backoff formula gives a shorter wait. If it asks for a long pause, do not block a worker on it: put the URL back in the queue with a not-before time and let the worker move to another site.

Treat a 429 as information about the whole site, not just one URL. Slow down every request to that host for the rest of the run, not only the retried one. Switching to a different proxy to keep the old pace defeats the purpose of the limit, and it is not something this guide recommends.

Exponential backoff in Python with requests and urllib3

For most requests scripts the simplest option is the retry support built into urllib3. Mount an HTTPAdapter with a Retry object on a session. The proxy setup itself is covered in the Python requests proxy guide.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

PROXY = "http://USER:[email protected]:8080"

retry = Retry(
    total=4,                      # at most 4 retries per request
    backoff_factor=1.5,           # waits grow 0, 3, 6, 12 s (urllib3 2.x)
    backoff_jitter=1.0,           # add up to 1 s of random jitter
    backoff_max=120,              # never wait longer than this between tries
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET", "HEAD"],
    respect_retry_after_header=True,
    raise_on_status=False,        # return the last response instead of raising
)

session = requests.Session()
adapter = HTTPAdapter(max_retries=retry)
session.mount("http://", adapter)
session.mount("https://", adapter)
session.proxies = {"http": PROXY, "https": PROXY}
session.headers["User-Agent"] = "ExampleCoBot/1.0 (+https://example.com/bot; [email protected])"

r = session.get("https://proxyv4.com/api/my-ip", timeout=(5, 20))
print(r.status_code, r.text.strip())

In urllib3 2.x the first retry happens without a pause and later waits are backoff_factor * 2 ** (n - 1) seconds for the n-th consecutive error, capped by backoff_max; backoff_jitter (added in 2.0) adds a random amount on top. With respect_retry_after_header=True, urllib3 sleeps for the server's Retry-After on 413, 429 and 503 responses. Retries apply only to the methods in allowed_methods, so a POST is not retried unless you add it, which is the safe default for requests that change data. The full parameter list is in the urllib3 Retry documentation.

One caveat: urllib3 will sleep inside the call for whatever Retry-After says, which can block a worker for a long time. If you need control over that, or full jitter from zero, write the loop yourself.

A custom loop with full jitter and Retry-After

This function retries connection errors and the retryable status codes, waits a random time up to the exponential value, never retries sooner than Retry-After, and gives up (returning the last response) when the server asks for a pause longer than you want a worker to sleep. Pass it the session from the previous example.

import random
import time
from email.utils import parsedate_to_datetime

import requests

RETRYABLE = {429, 500, 502, 503, 504}
MAX_WAIT = 300  # give up on a single wait longer than 5 minutes; reschedule the URL instead


def retry_after_seconds(value):
    if not value:
        return None
    if value.strip().isdigit():
        return float(value)
    try:
        return max(0.0, parsedate_to_datetime(value).timestamp() - time.time())
    except (TypeError, ValueError):
        return None


def get_with_backoff(session, url, attempts=5, base=2.0, cap=120.0):
    for attempt in range(attempts):
        try:
            r = session.get(url, timeout=(5, 20))
        except (requests.ConnectionError, requests.Timeout) as exc:
            r, error = None, exc
        else:
            if r.status_code not in RETRYABLE:
                return r                      # success, or an error that retrying will not fix
            error = None
        if attempt == attempts - 1:
            break
        wait = random.uniform(0, min(cap, base * 2 ** attempt))   # full jitter
        if r is not None:
            server_wait = retry_after_seconds(r.headers.get("Retry-After"))
            if server_wait is not None:
                if server_wait > MAX_WAIT:
                    break                     # the site wants a long pause: stop and requeue later
                wait = max(wait, server_wait)  # never retry sooner than the server asked
        time.sleep(wait)
    if r is not None:
        return r
    raise error

A circuit breaker per host

Per-request retries cannot fix a site that is down or a site that keeps telling you to slow down. Without a breaker, every queued URL for that host burns through all its retries, which wastes your time and keeps load on the site. A circuit breaker tracks failures per host and, past a threshold, stops sending to that host for a cooldown period.

import time


class HostCircuitBreaker:
    """Stops all requests to one host after repeated failures, then probes once."""

    def __init__(self, failure_threshold=5, cooldown=300):
        self.failure_threshold = failure_threshold
        self.cooldown = cooldown
        self.failures = 0
        self.opened_at = None
        self.probe_in_flight = False

    def allow(self):
        if self.opened_at is None:
            return True                                   # closed
        if time.monotonic() - self.opened_at < self.cooldown:
            return False                                  # open: skip this host for now
        if self.probe_in_flight:
            return False                                  # half-open: one probe at a time
        self.probe_in_flight = True
        return True

    def record(self, ok):
        self.probe_in_flight = False
        if ok:
            self.failures, self.opened_at = 0, None
            return
        self.failures += 1
        if self.opened_at is not None or self.failures >= self.failure_threshold:
            self.opened_at = time.monotonic()             # (re)open and restart the cooldown

Keep one breaker per host. Before each request call allow(); if it returns False, requeue the URL for later. After the request call record(ok) with True for a normal response and False for a retryable failure. In the closed state everything flows. After failure_threshold failures the breaker opens and the host is skipped. When the cooldown passes, one probe request is allowed (half-open); success closes the breaker and failure opens it again for another cooldown. Lengthen the cooldown or lower the threshold for sites that throttle you often.

Where static IPs help a retry strategy, and where they do not

A good retry layer makes the proxy the least interesting part of your stack, which is the point. What static IPs do give you is predictability: the same exit address on every run, so a site operator can recognize your crawler, contact you through the User-Agent and, if they choose, allow your traffic. That fits the scheduled, polite collection jobs described in the web scraping use case.

ProxyV4 sells private datacenter IPv4 in seven countries and static ISP proxies in the United States and Vietnam, one tenant per IP for the whole term. Every proxy supports HTTP(S) and SOCKS5 with username and password, traffic is unmetered, and renewing before expiry keeps the same IP. Rotating proxies are waitlist only and IPv6 proxies are a separate product that only works on IPv6-ready sites.

It is not a fit if you are looking for IPs to keep scraping a site that is returning 429s; the fix there is a slower pace or the site's API. It is also not a fit for jobs that need mobile carrier IPs. If static IPs match your job, check live prices, buy one IP, and test your retry logic against it before you scale.

Frequently asked questions

How many times should a scraper retry a failed request?

Three to five attempts with exponential backoff is a common choice for temporary errors. After that, log the URL and requeue it for a later run instead of retrying in a loop.

What is the difference between linear and exponential backoff?

Linear backoff adds the same amount of wait each time, while exponential backoff multiplies it, usually doubling. Exponential backoff reduces load faster during a longer outage, which is why it is the usual choice.

Should I switch proxies when I get a 429?

No. A 429 means the site wants less traffic from you. Wait for Retry-After, slow down for that site, and if the limit is too low for your needs, ask the operator or use their API.

Does urllib3 add jitter to retries?

Yes, from urllib3 2.0 the Retry class accepts backoff_jitter, which adds a random number of seconds up to that value. Older 1.x versions have no jitter option.

Will my ProxyV4 proxy keep the same IP when I renew?

Yes. Renewing before the proxy expires keeps the same IP address, and you can renew all proxies or a selection.

Stable static IPs for well-behaved scrapers

Private IPv4 and static ISP proxies with HTTP(S) and SOCKS5, one tenant per IP and unmetered traffic. Buy one IP and test your retry and backoff logic before you scale.

See live prices

Related articles

Use cases
Designing a scraper around a pool of dedicated IPs
A planning guide for data teams: size a pool of private static IPs, rotate them in your own code, pick IPv4, ISP or IPv6, and collect data responsibly.
Proxy setup
Python requests through an authenticated proxy, step by step
Send Python requests through an authenticated HTTP(S) proxy: build the proxies dict, attach it to a Session, encode odd passwords, rotate a list, check the IP.
Proxies by platform
Proxies for web scraping: a fixed pool of IPs with unmetered traffic
Web scraping proxies for Python, Scrapy, Playwright and Selenium: a pool of private IPv4 and static ISP addresses, unlimited bandwidth, flat per-IP pricing.
Free tools
Check proxies in bulk: status, exit IP and latency
Free online proxy checker: paste up to 100 ip:port:user:pass lines and see which proxies work, their exit IP and latency, plus what each failure result means.
Knowledge base
Rotating User Agents and Proxies: What to Do Instead
Should you rotate user agent and proxy together? Why random rotation backfires, and how to bind one honest header profile to each static IP in Python.
Knowledge base
Is Web Scraping with Proxies Legal? What Actually Decides It
Is web scraping with proxies legal? A proxy does not change the answer. Learn which rules apply: terms of service, computer-misuse, copyright and privacy law.
Knowledge base
Routing Python httpx traffic through an authenticated proxy
Configure an httpx proxy with username and password, SOCKS5 via httpx[socks], AsyncClient, per-host mounts and env vars, plus the errors 0.28 throws.
Products
Dedicated datacenter proxies: one private IPv4 per customer
Private IPv4 proxies in the US, Vietnam, Japan, South Korea, Singapore, the UK and Hong Kong. One tenant per IP, HTTP(S) and SOCKS5, no traffic metering.
Products
Static ISP proxies: residential IPs on consumer networks
Static ISP proxies in the United States and Vietnam: fixed IPs registered to consumer internet providers, rented to one customer, with unmetered traffic.
Pricing
Proxy prices per IP, by location and volume
Current per-IP rates for private IPv4 and static ISP proxies in the US, Vietnam, Japan, South Korea, Singapore, the UK and Hong Kong, with volume discounts.