Designing a scraper around a pool of dedicated IPs
Updated:
Web scraping proxies spread your requests over many addresses so no single IP carries the whole crawl. For most targets, a pool of private IPv4 addresses is the fastest and cheapest option; static ISP addresses are the fallback for sites that block datacenter ranges. You rotate the pool in your own code, which means you always know which IP fetched which page.
Why one IP stops working
Sites cap how many requests an address may send. A crawler running from your office line soon hits that cap, and the block can land on everyone else in the building. A proxy pool divides the load, keeps your own network out of the logs and lets you fetch the version of a page that visitors in a given country receive.
Typical users: e-commerce teams tracking competitor prices and stock, analysts building public datasets, travel, property and job-listing aggregators, and developers testing their own sites from other countries.
Static pool versus rotating gateway
With a static pool, the list of addresses is yours for the rental period. Your scheduler chooses an IP per request or per session, logs it, and retires any address that starts getting throttled. Each IP is single-tenant, so its reputation reflects your traffic only, and the fixed list is easy to allowlist in internal tools.
Bandwidth on private IPv4 and static ISP is not metered. Large pages, images and long crawls are paid for per IP, not per gigabyte.
Matching the product to the target
| Target | Product | Reason |
|---|---|---|
| Public catalogs, APIs, most sites | Private IPv4 | Fast, cost-effective, seven countries |
| Sites that refuse datacenter ranges | Static ISP, US or Vietnam | Addresses registered to consumer ISPs |
| Very large crawls of IPv6-ready sites | IPv6 | Many addresses at a low price per IP; confirm the site accepts IPv6 |
| Localized pages in Asia | Private IPv4 in Japan, South Korea, Singapore, Hong Kong or Vietnam | Fetch what local visitors see |
Pool sizing and request discipline
- Size the pool from your target request rate: more addresses at a gentle per-IP rate outlast a few addresses pushed hard.
- Pin one address for the length of any logged-in or cart session, then switch.
- Send realistic headers, add jitter between requests and back off on HTTP 429 and 403.
- Validate responses so captcha pages and block pages never land in the dataset.
- Screen addresses with the proxy checker (up to 100 at a time, passwords are not stored) and drop slow or blocked ones.
Plugging the pool into your stack
Every address is an HTTP(S) proxy with username and password, delivered as ip:port:user:pass under Orders. Common libraries accept that directly; see the setup pages for Python requests, Scrapy, Playwright and Puppeteer. Desktop tools without proxy settings can be routed through Proxifier. The web scraping proxy page adds target-specific notes.
Collect responsibly
A proxy changes where a request comes from, not what you are entitled to collect. Respect robots.txt, the site's terms and the data-protection law that applies to you, and do not gather personal data you have no right to process. Flooding a site with parallel requests, scraping behind logins with accounts you do not own, and heavy abuse all break our terms as well.
Frequently asked questions
Should I start with datacenter or ISP proxies for scraping?
Datacenter. Private IPv4 is faster and available in more countries. Move a target to static ISP only if it blocks datacenter ranges. The proxy type comparison explains the trade-off.
Is a rotating proxy service required?
No. A static pool rotated by your own code gives you full control over which address handles which request.
Is traffic metered?
No. Private IPv4 and static ISP have unlimited bandwidth; you pay per IP for the rental term.
Can I use SOCKS5 with Scrapy or Playwright?
Paid proxies are HTTP(S) with username and password, which those libraries support natively. See HTTP vs SOCKS5.
Is web scraping legal?
It depends on what you collect, where you operate and the site's terms. Gathering public, non-personal data while respecting robots.txt is common practice; check the rules for your own project.
Start a scraping pool
Order as many private IPv4 addresses as your crawl needs; the price per IP falls with quantity and bandwidth is not metered.
Check US datacenter prices