Building a Python Proxy Rotation Script with requests

· Updated:

A Python proxy rotation script keeps a list of proxies, hands the next one to each request (round robin) or pins one proxy to a whole task (sticky session), and drops proxies that keep failing. The examples below do exactly that with requests and the standard library: a thread-safe pool, sticky sessions, failover that retires dead proxies, and a 429 handler that waits for Retry-After instead of switching IPs. Every request has a timeout.

Your own static pool, rotated in code

What a Python proxy rotation script needs

Rotation in code spreads load across a pool you control and keeps one failing proxy from stopping a job. It needs four parts: a parser that turns your proxy lines into URLs requests understands, a selector (round robin or sticky), failure tracking that retires dead proxies, and timeouts so a stuck connection cannot hang a worker.

It is not a way around a site's limits. Spreading requests over several IPs does not change what a site allows; respect robots.txt, rate limits and each site's terms, and slow down when a site answers 429.

From ip:port:user:pass to a proxy URL

Proxies often arrive in the colon-separated format ip:port:user:pass. requests needs a URL such as http://USER:PASS@IP:PORT. Split with split(':', 3) so a password containing a colon stays in one piece, and URL-encode the username and password so characters such as @ or # do not break the URL.

The proxies dictionary has http and https keys that refer to the scheme of the target URL. Both usually point at the same http:// proxy URL; HTTPS targets are tunneled through it with CONNECT.

SOCKS5 in requests

For SOCKS5, install the extra with pip install "requests[socks]". Without it, requests raises InvalidSchema: Missing dependencies for SOCKS support. Use socks5h:// rather than socks5:// when you want DNS lookups done by the proxy instead of on your machine. The pool below takes the scheme as a parameter.

A thread-safe round robin proxy list in Python

itertools.cycle gives round-robin order, but it is not documented as thread-safe, so the pool wraps next() in a threading.Lock. get_next_proxy() returns both the original line and the proxies dictionary, so the caller can report failures against the right proxy. A proxy that fails twice in a row is marked dead and skipped; a success resets its count.

import itertools
import threading
from urllib.parse import quote

import requests


class ProxyPool:
    """Round-robin pool of ip:port:user:pass lines that threads can share."""

    def __init__(self, lines, scheme="http", max_failures=2):
        self.lines = [line.strip() for line in lines if line.strip()]
        self.scheme = scheme              # "http", or "socks5h" with requests[socks]
        self.max_failures = max_failures
        self.failures = {}
        self.dead = set()
        self._cycle = itertools.cycle(self.lines)
        self._lock = threading.Lock()     # itertools.cycle is not documented as thread-safe

    def _to_proxies(self, line):
        ip, port, user, password = line.split(":", 3)
        url = f"{self.scheme}://{quote(user, safe='')}:{quote(password, safe='')}@{ip}:{port}"
        return {"http": url, "https": url}

    def get_next_proxy(self):
        """Return (line, proxies_dict) for the next proxy that is not dead."""
        with self._lock:
            for _ in range(len(self.lines)):
                line = next(self._cycle)
                if line not in self.dead:
                    return line, self._to_proxies(line)
        raise RuntimeError("No live proxies left in the pool")

    def mark_dead(self, line):
        with self._lock:
            self.dead.add(line)

    def record_failure(self, line):
        with self._lock:
            self.failures[line] = self.failures.get(line, 0) + 1
            count = self.failures[line]
        if count >= self.max_failures:    # second failure in a row drops the proxy
            self.mark_dead(line)

    def record_success(self, line):
        with self._lock:
            self.failures[line] = 0


PROXY_ERRORS = (requests.exceptions.ProxyError, requests.exceptions.ConnectTimeout)

with open("proxies.txt") as f:          # one ip:port:user:pass line per row
    pool = ProxyPool(f)

session = requests.Session()
urls = ["https://example.com/page/1", "https://example.com/page/2", "https://example.com/page/3"]

for url in urls:
    line, proxies = pool.get_next_proxy()
    try:
        r = session.get(url, proxies=proxies, timeout=(5, 15))
    except PROXY_ERRORS as err:
        pool.record_failure(line)
        print("proxy", line.split(":")[0], "failed:", type(err).__name__)
        continue
    pool.record_success(line)
    print(url, r.status_code, "via", line.split(":")[0])

The loop uses one Session, so connections to each proxy are reused instead of opened for every request. Only ProxyError and ConnectTimeout count as proxy failures: they mean the proxy itself could not be reached or refused the connection. A slow target page (ReadTimeout) or an HTTP error from the site is not the proxy's fault and should not retire it. The log prints only the IP, never the password.

Sticky session per proxy

Some work has to stay on one IP: a login followed by several pages, a cart, or any flow where the site ties a session cookie to the address it saw. A sticky session keeps one requests.Session, with its cookies and connections, on one proxy for the whole task.

# Uses ProxyPool from the first example (same file, same imports).

class StickySessions:
    """One requests.Session per task key, each pinned to one proxy until closed."""

    def __init__(self, pool):
        self.pool = pool
        self.sessions = {}
        self._lock = threading.Lock()

    def get(self, key):
        with self._lock:
            if key not in self.sessions:
                line, proxies = self.pool.get_next_proxy()
                s = requests.Session()
                s.trust_env = False       # stop HTTP(S)_PROXY env vars overriding s.proxies
                s.proxies.update(proxies)
                self.sessions[key] = (line, s)
            return self.sessions[key]

    def close(self, key):
        with self._lock:
            entry = self.sessions.pop(key, None)
        if entry:
            entry[1].close()


sticky = StickySessions(pool)
line, s = sticky.get("client-a-report")
first = s.get("https://proxyv4.com/api/my-ip", timeout=10).json()["ip"]
second = s.get("https://proxyv4.com/api/my-ip", timeout=10).json()["ip"]
print(first, second, first == second)   # same exit IP for the whole task
sticky.close("client-a-report")

requests.Session has no default timeout, so each call passes timeout=10. Setting trust_env = False matters: when HTTP_PROXY or HTTPS_PROXY is set in the environment, it takes precedence over session.proxies, and the session would quietly skip your proxy. Close the session when the task ends so its sockets are released and the key can be reused.

Failover, dead proxies and HTTP 429

Two different things can go wrong, and they need different responses. A proxy fault (the proxy is unreachable or refuses the connection) is fixed by trying another proxy. An HTTP 429 from the target is the site saying you are sending too much; moving to another IP to keep the same pace ignores that answer.

Failover that honors Retry-After

# Uses ProxyPool and PROXY_ERRORS from the first example.
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime


def retry_after_seconds(response, default=30.0):
    """Seconds to wait from a Retry-After header (delta-seconds or HTTP date)."""
    value = response.headers.get("Retry-After", "").strip()
    if value.isdigit():
        return float(value)
    try:
        when = parsedate_to_datetime(value)
        return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())
    except (TypeError, ValueError):
        return default


def fetch(url, pool, session, max_attempts=4, max_wait=120):
    line, proxies = pool.get_next_proxy()
    for _ in range(max_attempts):
        try:
            r = session.get(url, proxies=proxies, timeout=(5, 15))
        except PROXY_ERRORS:
            pool.record_failure(line)             # a proxy fault: try the next proxy
            line, proxies = pool.get_next_proxy()
            continue
        pool.record_success(line)
        if r.status_code == 429:                  # the site asked us to slow down
            wait = retry_after_seconds(r)
            if wait > max_wait:
                raise RuntimeError(f"{url}: asked to wait {wait:.0f}s, stopping")
            time.sleep(wait)                      # same proxy, after the requested wait
            continue
        r.raise_for_status()
        return r
    raise RuntimeError(f"Gave up on {url} after {max_attempts} attempts")

On a proxy error the function records the failure and moves to the next proxy; the pool retires a proxy on its second consecutive failure. On 429 it reads Retry-After (either seconds or an HTTP date), waits on the same proxy, and stops the job if the requested wait is longer than you are willing to sleep. If you see 429s regularly, lower your overall request rate rather than raising max_wait.

For simple retries without failover, urllib3's Retry (mounted with HTTPAdapter(max_retries=...)) can retry 502, 503 and 504 responses and respects Retry-After by default when 429 is in its status_forcelist.

Reading the errors

A ProxyError mentioning 407 means the username or password was rejected; check the line, and see the 407 Proxy Authentication Required guide. A ConnectTimeout means the proxy did not answer in time: a wrong port, a firewall, or a protocol mismatch such as an http:// URL sent to a SOCKS5-only port. The proxy timeout guide covers the checks in order.

Rotating your own static pool

Rotating in code works with any list of fixed proxies. You decide the order, which tasks stick to which IP, and when a proxy is retired, and you can see in your own logs which IP made which request. That visibility helps when you debug a crawl or explain traffic to a site owner.

This model suits the web scraping use case: catalog and price monitoring, public data collection, and QA checks from several countries, at a pace each site accepts. For basic proxy configuration in requests, the Python requests setup guide covers the single-proxy case.

When a ProxyV4 pool fits your rotation code

ProxyV4 sells private datacenter IPv4 in seven countries (United States, Vietnam, Japan, South Korea, Singapore, United Kingdom, Hong Kong) and static ISP IPs registered to consumer internet providers in the United States and Vietnam. Each IP has one tenant for the whole term and is delivered as an ip:port:user:pass line, which the pool above reads as-is.

Every proxy supports HTTP(S) and SOCKS5 with username and password. On the order's page you choose HTTP only, SOCKS5 only, or both on the same port; switching can move a proxy to a new port, so re-export the list afterwards. Traffic is unmetered, and renewing before expiry keeps the same IPs, so your pool file stays valid.

It does not fit if you want a provider-side gateway that hands out a new IP per request: rotating proxies are waitlist-only. It also does not fit jobs that need mobile carrier IPs or IPv6, which is sold separately as IPv6 proxies. Prices are per IP per month on the pricing page; a single IP can be ordered to test your script before you build a larger pool.

Check that the pool really rotates

A 200 response does not prove the request went through the proxy you expected. Run one request per proxy against an IP echo endpoint and count the distinct exit IPs.

# Uses pool and session from the first example.
seen = set()
for _ in range(len(pool.lines)):
    line, proxies = pool.get_next_proxy()
    ip = session.get("https://proxyv4.com/api/my-ip", proxies=proxies, timeout=10).json()["ip"]
    seen.add(ip)
print(len(seen), "distinct exit IPs for", len(pool.lines), "proxies")

The count should equal the number of live proxies. If it is lower, two lines point at the same IP, or a proxy is being skipped because it was marked dead. For a one-off check of a single line, the free proxy checker is quicker.

Frequently asked questions

Does rotating proxies in Python avoid rate limits?

No. Rotation spreads load across IPs, but you should still respect robots.txt, rate limits and each site's terms, and back off when a site returns 429.

Can I rotate SOCKS5 proxies with requests?

Yes. Install requests[socks] and build URLs with socks5:// or socks5h://; the second resolves DNS through the proxy.

What is the difference between round robin and a sticky session?

Round robin gives each request the next proxy in the list. A sticky session keeps one proxy, and its cookies, for every request in a task.

Will my proxy list change when I renew?

No. Renewing ProxyV4 proxies before they expire keeps the same IPs, and you can renew all proxies or a selection.

Is bandwidth billed when a script runs all day?

Not on ProxyV4 private IPv4 or static ISP proxies: traffic is unmetered and you pay per IP per month. A proxy that fails for a technical reason is covered by the refund policy as an account balance credit.

Your own static pool, rotated in code

Private IPv4 and static ISP proxies delivered as ip:port:user:pass, with HTTP(S) and SOCKS5, unmetered traffic and the same IPs on renewal. Start with one IP.

Build your IP pool

Related articles

Proxy setup
Python requests through an authenticated proxy, step by step
Send Python requests through an authenticated HTTP(S) proxy: build the proxies dict, attach it to a Session, encode odd passwords, rotate a list, check the IP.
Use cases
Designing a scraper around a pool of dedicated IPs
A planning guide for data teams: size a pool of private static IPs, rotate them in your own code, pick IPv4, ISP or IPv6, and collect data responsibly.
Knowledge base
Static or Rotating Proxy? Start From the Task, Not the Product
Static vs rotating proxies, decided job by job: account farming, ad accounts, marketplace shops, scraping and SEO checks, plus sticky sessions and DIY rotation.
Free tools
Check proxies in bulk: status, exit IP and latency
Free online proxy checker: paste up to 100 ip:port:user:pass lines and see which proxies work, their exit IP and latency, plus what each failure result means.
Proxies by platform
Proxies for web scraping: a fixed pool of IPs with unmetered traffic
Web scraping proxies for Python, Scrapy, Playwright and Selenium: a pool of private IPv4 and static ISP addresses, unlimited bandwidth, flat per-IP pricing.
Knowledge base
How to Test a Proxy List in Bulk with Python
Test proxies in bulk with Python: parse ip:port:user:pass, check many at once with timeouts and bounded concurrency, and confirm each exit IP. Two scripts.
Knowledge base
How to Run a Python Telegram Bot Through a Proxy
Set up a Telegram Bot API proxy in Python with python-telegram-bot v21+ and aiogram 3: HTTP or SOCKS5, authentication, polling and a connectivity test.
Knowledge base
Sending Go net/http requests through an authenticated proxy
Set a golang http client proxy with http.ProxyURL and a login, use SOCKS5 natively or with x/net/proxy, pick a proxy per request and read proxyconnect errors.
Products
Dedicated datacenter proxies: one private IPv4 per customer
Private IPv4 proxies in the US, Vietnam, Japan, South Korea, Singapore, the UK and Hong Kong. One tenant per IP, HTTP(S) and SOCKS5, no traffic metering.
Products
Static ISP proxies: residential IPs on consumer networks
Static ISP proxies in the United States and Vietnam: fixed IPs registered to consumer internet providers, rented to one customer, with unmetered traffic.