Scrapy proxies: meta['proxy'], middleware order and rotation
Updated:
In Scrapy, a proxy is set per request through request.meta['proxy'] = 'http://user:pass@ip:port', and the built-in HttpProxyMiddleware (order 750) turns the user:pass part into a Proxy-Authorization header. Add a small downloader middleware that runs before it, at 740, and every request your spider yields can pick an IP from your ProxyV4 pool.
request.meta['proxy'] = 'http://user:pass@ip:port'Why the middleware number matters
Downloader middlewares run in ascending order on the way out. HttpProxyMiddleware at 750 reads meta['proxy'], strips the credentials from the URL and writes the auth header. If your own middleware sets the proxy afterwards, at 800 for example, the built-in one has already run, the credentials are never converted, and the proxy answers 407. Registering yours at 740 puts it first.
| Setting | Suggested value | Effect |
|---|---|---|
request.meta['proxy'] | 'http://user:pass@ip:port' | Sends that request through the given ProxyV4 proxy |
| Your middleware order | 740 | Runs before HttpProxyMiddleware (750), so the login is encoded |
CONCURRENT_REQUESTS_PER_IP | 2 to 4 | Limits parallel requests per destination IP, a polite cap on how hard each server is hit |
AUTOTHROTTLE_ENABLED | True | Lets Scrapy stretch or shorten delays based on how fast the target responds |
A rotation middleware in outline
- Read the proxy list from settings in
from_crawler()and keep it on the middleware instance - In
process_request(), assignrequest.meta['proxy'] = random.choice(self.proxies)unless the request already carries one - When a response comes back
429or503, retry it with a different proxy from the pool instead of the one that was just refused - Combine
DOWNLOAD_DELAY = 1.0withRANDOMIZE_DOWNLOAD_DELAY = Trueso the gap between requests varies rather than ticking like a metronome
Sizing the pool
A Scrapy crawl can push a lot of requests per minute, so the number of IPs sets how gently each one is used. With ProxyV4 there is no minimum order and the price per IP drops as quantity rises, and private IPv4 traffic is not metered, so a large crawl costs the same as a small one on bandwidth. Each IP is used by one customer only, so its reputation reflects your own crawl and nobody else's. For planning crawl volume and politeness, see the web scraping guide.
Step by step
- 1
Prepare the proxy URLs
Convert your ProxyV4
ip:port:user:passlines tohttp://user:pass@ip:portwith the proxy format converter and store them in settings or a file. - 2
Write ProxyPoolMiddleware
In
middlewares.py, implementprocess_request(self, request, spider)so it setsrequest.meta['proxy'] = random.choice(self.proxies). - 3
Register it before the built-in one
In
settings.py, add'myproject.middlewares.ProxyPoolMiddleware': 740toDOWNLOADER_MIDDLEWARESand setCONCURRENT_REQUESTS_PER_IP = 2. - 4
Confirm rotation
Fetch
https://proxyv4.com/api/my-ipa few times withscrapy shellor a short test spider and check that the reported IP changes across your pool.
Frequently asked questions
My proxy middleware at priority 800 gets 407 errors. Why?
At 800 it runs after HttpProxyMiddleware (750), so the user:pass in your proxy URL is never turned into a Proxy-Authorization header. Move your middleware below 750, e.g. 740.
Can I set the proxy for just one request?
Yes. Pass meta={'proxy': 'http://user:pass@ip:port'} when you create the Request; the built-in middleware handles the login for that request only.
Will a crawl of millions of pages hit a bandwidth limit?
Not on ProxyV4 private IPv4 or static ISP proxies; both have unlimited bandwidth. The practical limit is how fast the target lets you crawl.
Unmetered static IPs for Scrapy crawls
Private IPv4 in 7 countries plus static ISP in the US and Vietnam, from a single IP to a large pool, with live volume pricing.
Check live US prices