Python requests through an authenticated proxy, step by step
Updated:
To use a proxy in Python requests, pass a proxies dict whose 'http' and 'https' keys both point to http://user:pass@ip:port. Put that dict on a requests.Session() and the same connection to the proxy is reused for every call, which is faster than opening a new one per request. Below: the dict, the session, the errors you are likely to hit, and a one-line check of the ProxyV4 exit IP.
proxies = {'http': 'http://user:pass@ip:port', 'https': 'http://user:pass@ip:port'}The proxies dict, explained
| Setting | Value | Reason |
|---|---|---|
'http' key | 'http://user:pass@ip:port' | Used for plain http:// URLs |
'https' key | 'http://user:pass@ip:port' | Still an http:// proxy URL: urllib3 asks the proxy for a CONNECT tunnel and runs TLS inside it |
timeout | timeout=(5, 15) | 5 s to connect, 15 s to read, so a dead target cannot freeze a thread |
| Rotation | itertools.cycle(proxy_list) | Hands out your static IPs in turn |
Why a Session beats bare requests.get()
Each top-level requests.get() call builds a fresh connection pool, so the TCP connection to the proxy and the TLS handshake with the site are repeated every time. A Session keeps those connections alive between calls. For a script that fetches hundreds of pages through one IP, that difference is visible in the total run time.
Set s.proxies.update(proxies) once and every s.get() and s.post() goes through the proxy. One thing to know: if HTTP_PROXY or HTTPS_PROXY are set in the environment, requests may combine them with the session's settings; pass proxies= on the call itself when you need to be certain which one is used.
Errors and how to read them
ProxyErrorwith407 Proxy Authentication Required: the proxy is reachable but the login in your URL does not match the order. See the 407 error page- Passwords containing
@,:or/break the URL; encode them first withurllib.parse.quote(password, safe='') - Switching the proxy URL to
https://usually breaks the connection; keephttp://for both keys unless your proxy port is specifically a TLS port ConnectTimeoutpoints at the IP or port, not the credentials; test reachability with the port checker
Same pattern in httpx and aiohttp
The URL format carries over. In httpx, create httpx.Client(proxy='http://user:pass@ip:port'); in aiohttp, pass proxy= to session.get(url, proxy=...). Larger crawls with scheduling and retries are usually easier in Scrapy.
Step by step
- 1
Build the proxy URL
Rewrite each ProxyV4
ip:port:user:passline ashttp://user:pass@ip:port, or convert the whole list with the proxy format converter. - 2
Create the dict
Write
proxies = {'http': 'http://user:pass@ip:port', 'https': 'http://user:pass@ip:port'}. - 3
Attach it to a Session
Create
s = requests.Session()and calls.proxies.update(proxies)so connections are kept alive between requests. - 4
Print the exit IP
Run
print(s.get('https://proxyv4.com/api/my-ip', timeout=10).json()); the IP in the output should be the proxy's.
Frequently asked questions
Why is the 'https' entry an http:// URL?
The scheme describes how Python talks to the proxy, not to the website. urllib3 connects to the proxy over plain HTTP, sends CONNECT, and then does TLS with the https:// site through that tunnel, so the page traffic is still encrypted.
How do I spread requests over 25 static proxies?
Put the 25 URLs in a list, create pool = itertools.cycle(PROXIES), and for each request take p = next(pool) and pass proxies={'http': p, 'https': p}. Keep a sensible delay per IP so no single address is hammered.
Is traffic from Python scripts metered?
No. ProxyV4 private IPv4 and static ISP plans come with unlimited bandwidth, so long-running scripts do not run into a data cap.
A static IP pool for Python scripts
Private IPv4 in 7 countries plus static ISP in the US and Vietnam. Buy a single IP or a batch; the price per IP falls with quantity.
Check live US prices