Building a Python Proxy Rotation Script with requests
By the ProxyV4 Engineering Team · Updated:
A Python proxy rotation script keeps a list of proxies, hands the next one to each request (round robin) or pins one proxy to a whole task (sticky session), and drops proxies that keep failing. The examples below do exactly that with requests and the standard library: a thread-safe pool, sticky sessions, failover that retires dead proxies, and a 429 handler that waits for Retry-After instead of switching IPs. Every request has a timeout.
What a Python proxy rotation script needs
Rotation in code spreads load across a pool you control and keeps one failing proxy from stopping a job. It needs four parts: a parser that turns your proxy lines into URLs requests understands, a selector (round robin or sticky), failure tracking that retires dead proxies, and timeouts so a stuck connection cannot hang a worker.
It is not a way around a site's limits. Spreading requests over several IPs does not change what a site allows; respect robots.txt, rate limits and each site's terms, and slow down when a site answers 429.
From ip:port:user:pass to a proxy URL
Proxies often arrive in the colon-separated format ip:port:user:pass. requests needs a URL such as http://USER:PASS@IP:PORT. Split with split(':', 3) so a password containing a colon stays in one piece, and URL-encode the username and password so characters such as @ or # do not break the URL.
The proxies dictionary has http and https keys that refer to the scheme of the target URL. Both usually point at the same http:// proxy URL; HTTPS targets are tunneled through it with CONNECT.
SOCKS5 in requests
For SOCKS5, install the extra with pip install "requests[socks]". Without it, requests raises InvalidSchema: Missing dependencies for SOCKS support. Use socks5h:// rather than socks5:// when you want DNS lookups done by the proxy instead of on your machine. The pool below takes the scheme as a parameter.
A thread-safe round robin proxy list in Python
itertools.cycle gives round-robin order, but it is not documented as thread-safe, so the pool wraps next() in a threading.Lock. get_next_proxy() returns both the original line and the proxies dictionary, so the caller can report failures against the right proxy. A proxy that fails twice in a row is marked dead and skipped; a success resets its count.
import itertools
import threading
from urllib.parse import quote
import requests
class ProxyPool:
"""Round-robin pool of ip:port:user:pass lines that threads can share."""
def __init__(self, lines, scheme="http", max_failures=2):
self.lines = [line.strip() for line in lines if line.strip()]
self.scheme = scheme # "http", or "socks5h" with requests[socks]
self.max_failures = max_failures
self.failures = {}
self.dead = set()
self._cycle = itertools.cycle(self.lines)
self._lock = threading.Lock() # itertools.cycle is not documented as thread-safe
def _to_proxies(self, line):
ip, port, user, password = line.split(":", 3)
url = f"{self.scheme}://{quote(user, safe='')}:{quote(password, safe='')}@{ip}:{port}"
return {"http": url, "https": url}
def get_next_proxy(self):
"""Return (line, proxies_dict) for the next proxy that is not dead."""
with self._lock:
for _ in range(len(self.lines)):
line = next(self._cycle)
if line not in self.dead:
return line, self._to_proxies(line)
raise RuntimeError("No live proxies left in the pool")
def mark_dead(self, line):
with self._lock:
self.dead.add(line)
def record_failure(self, line):
with self._lock:
self.failures[line] = self.failures.get(line, 0) + 1
count = self.failures[line]
if count >= self.max_failures: # second failure in a row drops the proxy
self.mark_dead(line)
def record_success(self, line):
with self._lock:
self.failures[line] = 0
PROXY_ERRORS = (requests.exceptions.ProxyError, requests.exceptions.ConnectTimeout)
with open("proxies.txt") as f: # one ip:port:user:pass line per row
pool = ProxyPool(f)
session = requests.Session()
urls = ["https://example.com/page/1", "https://example.com/page/2", "https://example.com/page/3"]
for url in urls:
line, proxies = pool.get_next_proxy()
try:
r = session.get(url, proxies=proxies, timeout=(5, 15))
except PROXY_ERRORS as err:
pool.record_failure(line)
print("proxy", line.split(":")[0], "failed:", type(err).__name__)
continue
pool.record_success(line)
print(url, r.status_code, "via", line.split(":")[0])The loop uses one Session, so connections to each proxy are reused instead of opened for every request. Only ProxyError and ConnectTimeout count as proxy failures: they mean the proxy itself could not be reached or refused the connection. A slow target page (ReadTimeout) or an HTTP error from the site is not the proxy's fault and should not retire it. The log prints only the IP, never the password.
Sticky session per proxy
Some work has to stay on one IP: a login followed by several pages, a cart, or any flow where the site ties a session cookie to the address it saw. A sticky session keeps one requests.Session, with its cookies and connections, on one proxy for the whole task.
# Uses ProxyPool from the first example (same file, same imports).
class StickySessions:
"""One requests.Session per task key, each pinned to one proxy until closed."""
def __init__(self, pool):
self.pool = pool
self.sessions = {}
self._lock = threading.Lock()
def get(self, key):
with self._lock:
if key not in self.sessions:
line, proxies = self.pool.get_next_proxy()
s = requests.Session()
s.trust_env = False # stop HTTP(S)_PROXY env vars overriding s.proxies
s.proxies.update(proxies)
self.sessions[key] = (line, s)
return self.sessions[key]
def close(self, key):
with self._lock:
entry = self.sessions.pop(key, None)
if entry:
entry[1].close()
sticky = StickySessions(pool)
line, s = sticky.get("client-a-report")
first = s.get("https://proxyv4.com/api/my-ip", timeout=10).json()["ip"]
second = s.get("https://proxyv4.com/api/my-ip", timeout=10).json()["ip"]
print(first, second, first == second) # same exit IP for the whole task
sticky.close("client-a-report")requests.Session has no default timeout, so each call passes timeout=10. Setting trust_env = False matters: when HTTP_PROXY or HTTPS_PROXY is set in the environment, it takes precedence over session.proxies, and the session would quietly skip your proxy. Close the session when the task ends so its sockets are released and the key can be reused.
Failover, dead proxies and HTTP 429
Two different things can go wrong, and they need different responses. A proxy fault (the proxy is unreachable or refuses the connection) is fixed by trying another proxy. An HTTP 429 from the target is the site saying you are sending too much; moving to another IP to keep the same pace ignores that answer.
Failover that honors Retry-After
# Uses ProxyPool and PROXY_ERRORS from the first example.
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
def retry_after_seconds(response, default=30.0):
"""Seconds to wait from a Retry-After header (delta-seconds or HTTP date)."""
value = response.headers.get("Retry-After", "").strip()
if value.isdigit():
return float(value)
try:
when = parsedate_to_datetime(value)
return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())
except (TypeError, ValueError):
return default
def fetch(url, pool, session, max_attempts=4, max_wait=120):
line, proxies = pool.get_next_proxy()
for _ in range(max_attempts):
try:
r = session.get(url, proxies=proxies, timeout=(5, 15))
except PROXY_ERRORS:
pool.record_failure(line) # a proxy fault: try the next proxy
line, proxies = pool.get_next_proxy()
continue
pool.record_success(line)
if r.status_code == 429: # the site asked us to slow down
wait = retry_after_seconds(r)
if wait > max_wait:
raise RuntimeError(f"{url}: asked to wait {wait:.0f}s, stopping")
time.sleep(wait) # same proxy, after the requested wait
continue
r.raise_for_status()
return r
raise RuntimeError(f"Gave up on {url} after {max_attempts} attempts")On a proxy error the function records the failure and moves to the next proxy; the pool retires a proxy on its second consecutive failure. On 429 it reads Retry-After (either seconds or an HTTP date), waits on the same proxy, and stops the job if the requested wait is longer than you are willing to sleep. If you see 429s regularly, lower your overall request rate rather than raising max_wait.
For simple retries without failover, urllib3's Retry (mounted with HTTPAdapter(max_retries=...)) can retry 502, 503 and 504 responses and respects Retry-After by default when 429 is in its status_forcelist.
Reading the errors
A ProxyError mentioning 407 means the username or password was rejected; check the line, and see the 407 Proxy Authentication Required guide. A ConnectTimeout means the proxy did not answer in time: a wrong port, a firewall, or a protocol mismatch such as an http:// URL sent to a SOCKS5-only port. The proxy timeout guide covers the checks in order.
Rotating your own static pool
Rotating in code works with any list of fixed proxies. You decide the order, which tasks stick to which IP, and when a proxy is retired, and you can see in your own logs which IP made which request. That visibility helps when you debug a crawl or explain traffic to a site owner.
This model suits the web scraping use case: catalog and price monitoring, public data collection, and QA checks from several countries, at a pace each site accepts. For basic proxy configuration in requests, the Python requests setup guide covers the single-proxy case.
When a ProxyV4 pool fits your rotation code
ProxyV4 sells private datacenter IPv4 in seven countries (United States, Vietnam, Japan, South Korea, Singapore, United Kingdom, Hong Kong) and static ISP IPs registered to consumer internet providers in the United States and Vietnam. Each IP has one tenant for the whole term and is delivered as an ip:port:user:pass line, which the pool above reads as-is.
Every proxy supports HTTP(S) and SOCKS5 with username and password. On the order's page you choose HTTP only, SOCKS5 only, or both on the same port; switching can move a proxy to a new port, so re-export the list afterwards. Traffic is unmetered, and renewing before expiry keeps the same IPs, so your pool file stays valid.
It does not fit if you want a provider-side gateway that hands out a new IP per request: rotating proxies are waitlist-only. It also does not fit jobs that need mobile carrier IPs or IPv6, which is sold separately as IPv6 proxies. Prices are per IP per month on the pricing page; a single IP can be ordered to test your script before you build a larger pool.
Check that the pool really rotates
A 200 response does not prove the request went through the proxy you expected. Run one request per proxy against an IP echo endpoint and count the distinct exit IPs.
# Uses pool and session from the first example.
seen = set()
for _ in range(len(pool.lines)):
line, proxies = pool.get_next_proxy()
ip = session.get("https://proxyv4.com/api/my-ip", proxies=proxies, timeout=10).json()["ip"]
seen.add(ip)
print(len(seen), "distinct exit IPs for", len(pool.lines), "proxies")The count should equal the number of live proxies. If it is lower, two lines point at the same IP, or a proxy is being skipped because it was marked dead. For a one-off check of a single line, the free proxy checker is quicker.
Frequently asked questions
Does rotating proxies in Python avoid rate limits?
No. Rotation spreads load across IPs, but you should still respect robots.txt, rate limits and each site's terms, and back off when a site returns 429.
Can I rotate SOCKS5 proxies with requests?
Yes. Install requests[socks] and build URLs with socks5:// or socks5h://; the second resolves DNS through the proxy.
What is the difference between round robin and a sticky session?
Round robin gives each request the next proxy in the list. A sticky session keeps one proxy, and its cookies, for every request in a task.
Will my proxy list change when I renew?
No. Renewing ProxyV4 proxies before they expire keeps the same IPs, and you can renew all proxies or a selection.
Is bandwidth billed when a script runs all day?
Not on ProxyV4 private IPv4 or static ISP proxies: traffic is unmetered and you pay per IP per month. A proxy that fails for a technical reason is covered by the refund policy as an account balance credit.
Your own static pool, rotated in code
Private IPv4 and static ISP proxies delivered as ip:port:user:pass, with HTTP(S) and SOCKS5, unmetered traffic and the same IPs on renewal. Start with one IP.
Build your IP pool