How Many Requests per Second per Proxy Can You Send?
By the ProxyV4 Engineering Team · Updated:
There is no single safe number of requests per second per proxy: the limit belongs to each website, not to the IP, and it is whatever the site's terms, API documentation, robots.txt and live responses tell you. Start slow, keep one request in flight per site, and slow down further whenever the site answers with HTTP 429 or 503. This guide shows how to find and respect that budget, and how it should shape your proxy pool.
How many requests per second per proxy is safe?
The honest answer is "as many as the site allows, and no more." Tables that promise a safe rate for "strict" or "permissive" sites are guesses: the same request can be cheap on one site and expensive on another, and limits change without notice. A proxy does not raise that ceiling. Rate limits are usually tracked per client, per account, per API key or per session, and many sites also look at the total load your project puts on them.
So think of the request budget per IP as the sum of the budgets of the sites that IP talks to. One proxy can talk to many different sites at the same time, each at its own polite pace, without coming close to any one site's limit. What it should not do is help you send more to a single site than that site accepts. Splitting traffic for one site across many IPs to stay under a per-IP threshold is a way around the operator's limit, and the operator can still see the combined load.
The practical work is therefore measuring each site's limit and building a client that respects it. The rest of this guide covers where the limits are written down, how to read them from responses, and how to enforce them in code.
Where to find a site's real rate limit
Check the written sources before you send a single automated request. They often answer the question outright, and they also tell you whether automated access is welcome at all.
API documentation and terms of service
If the site offers an API, use it. API docs usually state the quota (per second, per minute or per day) and the headers that report it, and an API is a far more stable data source than HTML. Read the terms of service too: some sites forbid automated collection, some allow it under conditions, and some ask you to contact them for bulk access. If the terms rule out what you plan, ask the owner for permission or use a licensed data source instead.
robots.txt and Crawl-delay
robots.txt tells you which paths the operator does not want crawled. Some files also carry a Crawl-delay line, the number of seconds the operator wants between requests. Crawl-delay is not part of the robots.txt standard (RFC 9309) and some major crawlers ignore it, but when a site publishes one it is the clearest statement of the pace it wants, so treat it as a floor. Python's urllib.robotparser reads it with crawl_delay(user_agent).
Rate-limit headers and Retry-After
Many APIs and some websites report your remaining quota in response headers. Common names include X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset, and some servers use the newer RateLimit fields being standardized at the IETF; names and meanings vary by site, so check its docs. When you hit the limit, a well-behaved server returns 429 Too Many Requests or 503 Service Unavailable, often with Retry-After given as seconds or an HTTP date. That header is an instruction: wait at least that long before the next request to that site.
Measure, then set concurrency per IP
When nothing is written down, measure carefully instead of guessing high. Begin with one request in flight per site and a generous pause between requests. Log the status code, the response time and any rate-limit headers for every request. If response times stay flat and you see no 429, 503 or challenge pages over a long run, you can shorten the pause in small steps. If latency climbs or any throttling signal appears, go back to the previous setting and stay there.
Rising latency is the signal to watch most closely, because it often shows strain before errors do. A site that answers slower as you speed up is telling you that you are now a noticeable part of its load.
Concurrency per IP and request rate are linked by a simple relationship: requests in flight ≈ request rate × average response time. If a site's documented limit is two requests per second and its pages take half a second, one connection is enough; opening more only adds load and gains nothing. Size concurrency from the rate the site allows, not the other way round.
- Per site, not per pool. Keep a separate budget for every host you crawl, even when several hosts share one proxy.
- Cache and skip. Conditional requests (
If-Modified-Since,ETag) and skipping unchanged pages cut the number of requests you need at all. - Off-peak scheduling. Large jobs are easier on the site at its local night time.
- Stop on repeated throttling. If a site keeps returning 429 after you slow down, pause the job and review it rather than pushing on.
A per-host rate limiter in Python
The script below enforces one request in flight per host and a minimum gap between requests, raises that gap to the site's Crawl-delay when robots.txt sets one, and on 429 or 503 pauses the host for the Retry-After time (or a default) and halves its pace for the rest of the run. It uses httpx 0.26 or newer (pip install httpx), sends a descriptive User-Agent with contact details, and routes everything through one proxy. It deliberately does not switch IPs when throttled.
import asyncio
import time
from email.utils import parsedate_to_datetime
from urllib.parse import urlsplit
from urllib.robotparser import RobotFileParser
import httpx
PROXY = "http://USER:[email protected]:8080"
UA = "ExampleCoBot/1.0 (+https://example.com/bot; [email protected])"
START_INTERVAL = 5.0 # seconds between requests to one host; start slow, adjust from the site's rules
def retry_after_seconds(value):
"""Retry-After is either a number of seconds or an HTTP date."""
if not value:
return None
if value.strip().isdigit():
return float(value)
try:
return max(0.0, parsedate_to_datetime(value).timestamp() - time.time())
except (TypeError, ValueError):
return None
class HostBudget:
"""One request in flight per host and a minimum gap between request starts."""
def __init__(self, interval):
self.interval = interval
self.gate = asyncio.Semaphore(1)
self.next_start = 0.0
self.robots_checked = False
async def wait_turn(self):
delay = self.next_start - time.monotonic()
if delay > 0:
await asyncio.sleep(delay)
self.next_start = time.monotonic() + self.interval
def pause(self, seconds):
self.next_start = max(self.next_start, time.monotonic() + seconds)
async def apply_crawl_delay(client, host, budget):
"""Raise the gap to robots.txt Crawl-delay when the site sets one."""
budget.robots_checked = True
try:
r = await client.get(f"https://{host}/robots.txt")
except httpx.HTTPError:
return
if r.status_code == 200:
rp = RobotFileParser()
rp.parse(r.text.splitlines())
delay = rp.crawl_delay(UA)
if delay:
budget.interval = max(budget.interval, float(delay))
async def fetch(client, budgets, url):
host = urlsplit(url).hostname
budget = budgets.setdefault(host, HostBudget(START_INTERVAL))
async with budget.gate:
if not budget.robots_checked:
await apply_crawl_delay(client, host, budget)
await budget.wait_turn()
r = await client.get(url)
if r.status_code in (429, 503):
wait = retry_after_seconds(r.headers.get("Retry-After")) or 60.0
budget.pause(wait)
budget.interval *= 2 # the site asked us to slow down: stay slower for the rest of the run
return url, r.status_code, round(r.elapsed.total_seconds(), 2)
async def main(urls):
budgets = {}
async with httpx.AsyncClient(proxy=PROXY, timeout=15.0, headers={"User-Agent": UA}) as client:
for result in await asyncio.gather(*(fetch(client, budgets, u) for u in urls)):
print(result)
if __name__ == "__main__":
asyncio.run(main(["https://example.com/a", "https://example.com/b", "https://example.org/"]))In Scrapy the same ideas map onto settings: ROBOTSTXT_OBEY = True, CONCURRENT_REQUESTS_PER_DOMAIN = 1, a DOWNLOAD_DELAY you choose per site, and AUTOTHROTTLE_ENABLED = True, which adjusts delays from observed latency. The Scrapy proxy setup guide shows where the proxy goes. For retry and backoff patterns that sit on top of this limiter, see the web scraping use case.
How the per-site budget shapes your proxy pool
Because the limit is set per site, adding IPs does not let you collect faster from one site. Size the pool from the jobs it serves instead. Useful questions are: how many countries do you need exit IPs in, how many clients or projects must stay on separate IPs, and how many sites run in parallel on each IP before your own monitoring gets hard to read.
A simple planning model: for each site, the allowed rate is fixed by that site. Total throughput is the sum of those rates across all sites you crawl in parallel. Each proxy carries the sites assigned to it, each at its own pace. You add IPs when you add locations, separate projects or more sites, not to push one site harder.
Where static private IPs fit, and where they do not
Static IPs suit polite, scheduled collection well. The operator sees the same address every time, which makes your traffic easy to recognize, contact and, if they choose, allow. ProxyV4 sells private datacenter IPv4 in seven countries (United States, Vietnam, Japan, South Korea, Singapore, United Kingdom and Hong Kong) and static ISP proxies in the United States and Vietnam, one tenant per IP for the whole term. Traffic on both is unmetered, so cost follows the number of IPs, not data volume, and renewing before expiry keeps the same address. See the web scraping proxy page for how this fits common collection jobs.
They do not fit every job. If a site only works from mobile carrier IPs, or your plan depends on a fresh IP per request, static IPs are the wrong tool; ProxyV4 sells no mobile proxies, and rotating proxies are waitlist only. If your goal is to exceed a site's limit by spreading requests across addresses, no proxy product is the right answer: use the site's API or ask for higher access. Every proxy supports HTTP(S) and SOCKS5 with username and password. Check live prices, start with one IP, and confirm it works with the free proxy checker before you scale.
Frequently asked questions
Is one request per second per proxy safe?
Not as a rule. Some sites accept more, some want one request every several seconds, and some do not allow automated access at all. Use the site's documented limit or Crawl-delay, and slow down on 429, 503 or rising latency.
Does adding more proxies let me scrape a site faster?
It should not. Rate limits protect the site's capacity, and many are tracked per client or account rather than per IP. More IPs help when you crawl more sites or need more locations, not to exceed one site's limit.
What should my scraper do when it gets HTTP 429?
Stop sending to that site, wait at least the Retry-After time (or back off exponentially if there is none), and continue at a slower pace. Switching to another IP to keep going ignores the site's request.
Is bandwidth metered on ProxyV4 proxies?
No. Traffic on private IPv4 and static ISP proxies is not metered or billed; you pay per IP per month.
What happens if a proxy I buy does not work?
A proxy that fails for a technical reason, such as not connecting or showing the wrong location, is covered by the refund policy and credited to your account balance.
Steady static IPs for polite, per-site crawling
Private IPv4 in seven countries and static ISP in the US and Vietnam, one tenant per IP and unmetered traffic. Start with one IP and scale once your crawler respects each site's limits.
Get a static IP