How to Retry with Backoff in a Web Scraper
By the ProxyV4 Engineering Team · Updated:
Retry with backoff means that after a temporary failure your scraper waits, then tries again, and waits longer after each further failure, with some randomness (jitter) so many workers do not retry at the same moment. Retry only errors that can be temporary, honor the server's Retry-After header on 429 and 503, cap the number of attempts, and stop sending to a site that keeps failing. The examples below use Python requests.
What retry with backoff does in a scraper
Networks drop connections, servers restart and busy sites shed load. A scraper that gives up on the first error loses data; one that retries immediately in a tight loop adds load to a site that is already struggling and can turn a short incident into a long one. Backoff sits between the two: each retry waits longer than the last, so a brief glitch costs a second or two while a real outage costs you time rather than the site.
Backoff is part of rate limiting, not a replacement for it. Your normal pace for each site should already respect its documented limits and robots.txt. Retries handle the exceptions, and a 429 is the site telling you that your normal pace was too fast.
Exponential backoff
The wait doubles after each failed attempt: with a base of two seconds you wait about 2, 4, 8 and 16 seconds. Always set a cap on the single wait and a maximum number of attempts, otherwise a long outage leaves workers sleeping for hours or retrying forever.
Why jitter matters
If twenty workers fail at the same second and all compute the same wait, they all retry at the same second too, and the site sees the same spike again. Jitter spreads them out. With "full jitter" each worker sleeps a random time between zero and the exponential value for that attempt, which spreads retries evenly across the window.
Which errors to retry: 429, 503 and the rest
Retry only failures that can go away on their own. Errors that describe the request itself (a missing page, a forbidden path, bad credentials) will fail again, so log them and move on. A 403 in particular usually means the site does not want this client on that resource; treat it as a stop signal, not as something to retry from a different IP or with different headers.
| Response | Usual meaning | What the scraper should do |
|---|---|---|
| 429 Too Many Requests | You exceeded the site's rate limit | Wait at least Retry-After, then continue at a slower pace for that site |
| 503 Service Unavailable | Overload or maintenance | Wait for Retry-After if present, otherwise back off; pause the host if it repeats |
| 500, 502, 504 | Server or gateway problem | Retry a few times with backoff, then log and requeue |
| Connection error or timeout | Network problem on either side | Retry with backoff; check your proxy if it affects every site |
| 400, 401, 403, 404, 410 | Problem with the request or access | Do not retry; log it and review |
Reading Retry-After
Retry-After holds either a number of seconds or an HTTP date. When it is present, never retry sooner than it says, even if your own backoff formula gives a shorter wait. If it asks for a long pause, do not block a worker on it: put the URL back in the queue with a not-before time and let the worker move to another site.
Treat a 429 as information about the whole site, not just one URL. Slow down every request to that host for the rest of the run, not only the retried one. Switching to a different proxy to keep the old pace defeats the purpose of the limit, and it is not something this guide recommends.
Exponential backoff in Python with requests and urllib3
For most requests scripts the simplest option is the retry support built into urllib3. Mount an HTTPAdapter with a Retry object on a session. The proxy setup itself is covered in the Python requests proxy guide.
import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry
PROXY = "http://USER:[email protected]:8080"
retry = Retry(
total=4, # at most 4 retries per request
backoff_factor=1.5, # waits grow 0, 3, 6, 12 s (urllib3 2.x)
backoff_jitter=1.0, # add up to 1 s of random jitter
backoff_max=120, # never wait longer than this between tries
status_forcelist=[429, 500, 502, 503, 504],
allowed_methods=["GET", "HEAD"],
respect_retry_after_header=True,
raise_on_status=False, # return the last response instead of raising
)
session = requests.Session()
adapter = HTTPAdapter(max_retries=retry)
session.mount("http://", adapter)
session.mount("https://", adapter)
session.proxies = {"http": PROXY, "https": PROXY}
session.headers["User-Agent"] = "ExampleCoBot/1.0 (+https://example.com/bot; [email protected])"
r = session.get("https://proxyv4.com/api/my-ip", timeout=(5, 20))
print(r.status_code, r.text.strip())In urllib3 2.x the first retry happens without a pause and later waits are backoff_factor * 2 ** (n - 1) seconds for the n-th consecutive error, capped by backoff_max; backoff_jitter (added in 2.0) adds a random amount on top. With respect_retry_after_header=True, urllib3 sleeps for the server's Retry-After on 413, 429 and 503 responses. Retries apply only to the methods in allowed_methods, so a POST is not retried unless you add it, which is the safe default for requests that change data. The full parameter list is in the urllib3 Retry documentation.
One caveat: urllib3 will sleep inside the call for whatever Retry-After says, which can block a worker for a long time. If you need control over that, or full jitter from zero, write the loop yourself.
A custom loop with full jitter and Retry-After
This function retries connection errors and the retryable status codes, waits a random time up to the exponential value, never retries sooner than Retry-After, and gives up (returning the last response) when the server asks for a pause longer than you want a worker to sleep. Pass it the session from the previous example.
import random
import time
from email.utils import parsedate_to_datetime
import requests
RETRYABLE = {429, 500, 502, 503, 504}
MAX_WAIT = 300 # give up on a single wait longer than 5 minutes; reschedule the URL instead
def retry_after_seconds(value):
if not value:
return None
if value.strip().isdigit():
return float(value)
try:
return max(0.0, parsedate_to_datetime(value).timestamp() - time.time())
except (TypeError, ValueError):
return None
def get_with_backoff(session, url, attempts=5, base=2.0, cap=120.0):
for attempt in range(attempts):
try:
r = session.get(url, timeout=(5, 20))
except (requests.ConnectionError, requests.Timeout) as exc:
r, error = None, exc
else:
if r.status_code not in RETRYABLE:
return r # success, or an error that retrying will not fix
error = None
if attempt == attempts - 1:
break
wait = random.uniform(0, min(cap, base * 2 ** attempt)) # full jitter
if r is not None:
server_wait = retry_after_seconds(r.headers.get("Retry-After"))
if server_wait is not None:
if server_wait > MAX_WAIT:
break # the site wants a long pause: stop and requeue later
wait = max(wait, server_wait) # never retry sooner than the server asked
time.sleep(wait)
if r is not None:
return r
raise errorA circuit breaker per host
Per-request retries cannot fix a site that is down or a site that keeps telling you to slow down. Without a breaker, every queued URL for that host burns through all its retries, which wastes your time and keeps load on the site. A circuit breaker tracks failures per host and, past a threshold, stops sending to that host for a cooldown period.
import time
class HostCircuitBreaker:
"""Stops all requests to one host after repeated failures, then probes once."""
def __init__(self, failure_threshold=5, cooldown=300):
self.failure_threshold = failure_threshold
self.cooldown = cooldown
self.failures = 0
self.opened_at = None
self.probe_in_flight = False
def allow(self):
if self.opened_at is None:
return True # closed
if time.monotonic() - self.opened_at < self.cooldown:
return False # open: skip this host for now
if self.probe_in_flight:
return False # half-open: one probe at a time
self.probe_in_flight = True
return True
def record(self, ok):
self.probe_in_flight = False
if ok:
self.failures, self.opened_at = 0, None
return
self.failures += 1
if self.opened_at is not None or self.failures >= self.failure_threshold:
self.opened_at = time.monotonic() # (re)open and restart the cooldownKeep one breaker per host. Before each request call allow(); if it returns False, requeue the URL for later. After the request call record(ok) with True for a normal response and False for a retryable failure. In the closed state everything flows. After failure_threshold failures the breaker opens and the host is skipped. When the cooldown passes, one probe request is allowed (half-open); success closes the breaker and failure opens it again for another cooldown. Lengthen the cooldown or lower the threshold for sites that throttle you often.
Where static IPs help a retry strategy, and where they do not
A good retry layer makes the proxy the least interesting part of your stack, which is the point. What static IPs do give you is predictability: the same exit address on every run, so a site operator can recognize your crawler, contact you through the User-Agent and, if they choose, allow your traffic. That fits the scheduled, polite collection jobs described in the web scraping use case.
ProxyV4 sells private datacenter IPv4 in seven countries and static ISP proxies in the United States and Vietnam, one tenant per IP for the whole term. Every proxy supports HTTP(S) and SOCKS5 with username and password, traffic is unmetered, and renewing before expiry keeps the same IP. Rotating proxies are waitlist only and IPv6 proxies are a separate product that only works on IPv6-ready sites.
It is not a fit if you are looking for IPs to keep scraping a site that is returning 429s; the fix there is a slower pace or the site's API. It is also not a fit for jobs that need mobile carrier IPs. If static IPs match your job, check live prices, buy one IP, and test your retry logic against it before you scale.
Frequently asked questions
How many times should a scraper retry a failed request?
Three to five attempts with exponential backoff is a common choice for temporary errors. After that, log the URL and requeue it for a later run instead of retrying in a loop.
What is the difference between linear and exponential backoff?
Linear backoff adds the same amount of wait each time, while exponential backoff multiplies it, usually doubling. Exponential backoff reduces load faster during a longer outage, which is why it is the usual choice.
Should I switch proxies when I get a 429?
No. A 429 means the site wants less traffic from you. Wait for Retry-After, slow down for that site, and if the limit is too low for your needs, ask the operator or use their API.
Does urllib3 add jitter to retries?
Yes, from urllib3 2.0 the Retry class accepts backoff_jitter, which adds a random number of seconds up to that value. Older 1.x versions have no jitter option.
Will my ProxyV4 proxy keep the same IP when I renew?
Yes. Renewing before the proxy expires keeps the same IP address, and you can renew all proxies or a selection.
Stable static IPs for well-behaved scrapers
Private IPv4 and static ISP proxies with HTTP(S) and SOCKS5, one tenant per IP and unmetered traffic. Buy one IP and test your retry and backoff logic before you scale.
See live prices