Rotating User Agents and Proxies: What to Do Instead
By the ProxyV4 Engineering Team · Updated:
Rotating the user agent and proxy on every request is meant to make one scraper look like many different visitors, and that is the problem: it is a way around a site's limits, and sites that care usually see through it. A better pattern is one honest User-Agent that names your crawler, one fixed proxy per job or market, and a session per proxy so headers, cookies and IP stay consistent. This guide explains why and shows the Python code.
Should you rotate user agent and proxy together?
Most tutorials on this topic pair a random IP with a random browser User-Agent on every request. The aim is to stop the site from recognizing that the traffic comes from one program. If the site has rate limits or blocks automated clients, that is evasion, and it can breach the site's terms. We do not recommend it, and this guide does not show how to make it work better.
It also tends to fail on its own terms. A site that bothers to block scrapers looks at far more than one header and one IP, and a stream of "different browsers" that all behave like the same script is easy to spot. You end up maintaining lists of browser strings and burning IPs instead of collecting data.
What does make sense is the opposite: keep the identity stable and honest, and choose proxies for real reasons, such as needing an exit IP in a specific country or keeping different clients' jobs on different addresses.
How sites read the User-Agent and IP together
The User-Agent header exists so a server knows what software is making the request. HTTP's specification (RFC 9110) describes it as identifying the client, and well-behaved crawlers use it to say who they are. Servers use it for compatibility, analytics and, increasingly, for deciding how to treat automated traffic.
Why a header fingerprint gives copied browser strings away
A real browser sends a whole set of headers in a characteristic way, and it also runs JavaScript, keeps cookies and loads images and scripts. An HTTP library that borrows a browser's User-Agent string still sends its own default headers and behaves like a library. Sites compare these signals, which is why a copied string rarely changes how a site treats the traffic.
Why random rotation backfires
Changing the User-Agent between requests on the same IP produces a client that claims to be a different browser every few seconds. Rotating IPs per request breaks sessions and cookies, so sites that rely on them return errors or start over. And because both changes are designed to hide the source, a site that notices them has a reason to block more broadly.
Use one honest identity and a fixed proxy per job
Give your crawler a name and a way to contact you. A common format is a product token, a version and a link or email in parentheses, for example ExampleCoPriceMonitor/2.1 (+https://example.com/bot; [email protected]). Put a short page at that link that explains what you collect, how often, and how an operator can ask you to stop or slow down. The From header can carry the same email.
Then decide what the proxies are for. Legitimate reasons include viewing a site as it appears in another country, keeping each client's or project's traffic on its own address, and giving partners a fixed IP they can allow in their firewall. In each case one proxy stays bound to one job for a long time, rather than rotating per request.
- Same User-Agent everywhere. Your crawler's identity does not change with the IP.
- One session per proxy. Cookies, keep-alive connections and headers stay together.
- Language from the market, not at random.
Accept-Languageshould match the content you want to see. - Pace per site. Respect robots.txt,
Retry-Afterand each site's limits regardless of how many proxies you hold.
Bind a header profile to each static IP in Python
The script below keeps one requests.Session per market. Each session has its own fixed proxy and an Accept-Language that matches the market, while the User-Agent and contact header are the same everywhere. A fetch(market, url) call always uses the same IP for that market, so cookies and connections persist and the site sees a consistent client. Proxy configuration details are in the Python requests proxy guide.
import requests
USER_AGENT = "ExampleCoPriceMonitor/2.1 (+https://example.com/bot; [email protected])"
# One fixed proxy per market, each with the language that market's pages are served in
PROFILES = {
"us": {"proxy": "http://USER:[email protected]:8080", "lang": "en-US,en;q=0.8"},
"jp": {"proxy": "http://USER:[email protected]:8080", "lang": "ja-JP,ja;q=0.8,en;q=0.5"},
}
def build_session(profile):
s = requests.Session()
s.proxies = {"http": profile["proxy"], "https": profile["proxy"]}
s.headers.update({
"User-Agent": USER_AGENT, # the same honest identity on every proxy
"From": "[email protected]",
"Accept-Language": profile["lang"],
})
return s
SESSIONS = {market: build_session(p) for market, p in PROFILES.items()}
def fetch(market, url):
"""Fetch a page as seen from one market, always through that market's proxy."""
r = SESSIONS[market].get(url, timeout=(5, 20))
r.raise_for_status()
return r
if __name__ == "__main__":
for market in SESSIONS:
print(market, fetch(market, "https://proxyv4.com/api/my-ip").json())Run it and each line should show the proxy's IP and country from the echo endpoint. You can also open the free my IP tool through each proxy to check the exit address and location by hand.
The same idea in Scrapy
In Scrapy, set USER_AGENT once in settings.py with your crawler name and contact, keep ROBOTSTXT_OBEY = True, and pass the proxy for each request through request.meta['proxy'] based on the market or client the request belongs to. The Scrapy proxy setup guide shows the middleware and credential options.
Accept-Language for regional pages
Many sites choose a language or region from both the IP and Accept-Language. If the two disagree, you may get a language picker, a redirect or content for the wrong market. Matching them to the market you are checking gives you the page a local visitor would see, which is the point of a regional proxy.
When a site still blocks your crawler
If a site blocks a crawler that identifies itself and keeps to a slow pace, take that as the operator's answer. Check whether there is an API, a data feed or a partner program. Read the terms and robots.txt for what they allow. If you need the data for a legitimate reason, contact the operator with your crawler page, your IPs and your request rate, and ask for access. A rotating set of browser identities is not a substitute for that conversation. The web scraping use case covers the wider workflow for permitted collection.
Where static proxies fit this approach
Binding identities to fixed IPs works best when the IPs are yours alone and stay the same. ProxyV4 sells private datacenter IPv4 in seven countries (United States, Vietnam, Japan, South Korea, Singapore, United Kingdom and Hong Kong) and static ISP proxies in the United States and Vietnam, one tenant per IP for the whole term. The IP stays the same while the proxy is active and when you renew before expiry, so allow-lists and regional profiles keep working. Every proxy supports HTTP(S) and SOCKS5 with username and password, and traffic is unmetered.
It does not fit a plan built on a new IP per request: rotating proxies are waitlist only, and there are no mobile proxies. IPv6 proxies are a separate product that only works on IPv6-ready sites; private IPv4 works on IPv4 and dual-stack sites. Check live prices, buy one IP per market you need, and confirm the regional pages look right before you add more.
Frequently asked questions
Should I rotate the user agent on every request?
No. It makes one program pretend to be many browsers, which can breach a site's terms and rarely works for long. Use one descriptive User-Agent for your crawler instead.
What User-Agent should a scraper send?
A name for your crawler, a version, and a link or email where operators can reach you, for example ExampleBot/1.0 (+https://example.com/bot). It lets an operator contact you to ask for a slower pace instead of only being able to block you.
Can I use different proxies for different countries?
Yes. Using a proxy in each market to see local pages is a common, legitimate use. Keep one proxy per market and match Accept-Language to it.
Can I use HTTP and SOCKS5 on the same ProxyV4 proxy?
Yes. For each order you choose HTTP only, SOCKS5 only, or both on the same port in the dashboard. Switching can move the proxy to a new port, so copy the line again afterwards.
Is traffic metered on private IPv4?
No. Traffic on private IPv4 and static ISP proxies is not metered or billed; you pay per IP per month.
Bind a header profile to each static IP
Private IPv4 in seven countries and static ISP in the US and Vietnam, one tenant per IP, same IP on renewal. Buy one IP per market and test your crawler's profiles first.
Build your IP pool