Configuring Proxies in Crawlee: proxyUrls, Tiers, Custom Functions and Sessions
By the ProxyV4 Engineering Team · Updated:
Crawlee proxy configuration happens in one object: create a ProxyConfiguration with your proxyUrls (or tieredProxyUrls, or a newUrlFunction) and pass it to the crawler as proxyConfiguration. Crawlee then hands each request a proxy, and when the session pool is on it keeps every session on the same IP. Below is how each of the three modes behaves in the JavaScript and Python versions, and how to size the session pool for a small set of static IPs.
How Crawlee chooses a proxy for a request
Before each request, the crawler asks its ProxyConfiguration for a URL, passing the current session ID when the session pool is enabled. With a plain list, requests without a session rotate round-robin. A session ID seen for the first time gets the next proxy in the rotation, and every later call with that ID returns the same proxy. That binding is what keeps cookies, logins and the IP consistent for one simulated visitor.
The proxy used is exposed in the request handler as proxyInfo in JavaScript and context.proxy_info in Python, with fields such as url, hostname, port, sessionId and, when tiers are used, the tier number. Log it while you develop; it is the quickest way to tell a proxy failure from a parsing bug.
proxyUrls: a fixed list of your own proxies
This is the right mode when you own a known set of static IPs. Put the credentials in each URL with the http:// scheme. If a password contains @, : or /, encode it first with encodeURIComponent in Node or urllib.parse.quote in Python. A null (JavaScript) or None (Python) entry means a direct connection, which is handy for mixing your own IP into the rotation during tests.
JavaScript and TypeScript
import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';
const proxyConfiguration = new ProxyConfiguration({
proxyUrls: [
'http://USER:[email protected]:8080',
'http://USER:[email protected]:8080',
'http://USER:[email protected]:8080',
],
});
const crawler = new PlaywrightCrawler({
proxyConfiguration,
useSessionPool: true,
persistCookiesPerSession: true,
sessionPoolOptions: { maxPoolSize: 3 },
maxConcurrency: 3,
async requestHandler({ request, page, proxyInfo, session, log, enqueueLinks }) {
log.info(`${request.url} via ${proxyInfo?.hostname} (session ${session?.id})`);
await enqueueLinks();
},
});
await crawler.run(['https://example.com/']);The same options work on CheerioCrawler, HttpCrawler, JSDOMCrawler and PuppeteerCrawler. For browser crawlers, Crawlee passes the username and password to the browser for you, which saves the extension workarounds that raw Chrome automation needs.
Python
In Crawlee for Python the constructor takes snake_case keyword arguments, and you must pass exactly one of proxy_urls, tiered_proxy_urls or new_url_function. The session pool is on by default.
import asyncio
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
from crawlee.proxy_configuration import ProxyConfiguration
from crawlee.sessions import SessionPool
proxy_configuration = ProxyConfiguration(proxy_urls=[
'http://USER:[email protected]:8080',
'http://USER:[email protected]:8080',
'http://USER:[email protected]:8080',
])
async def main() -> None:
crawler = PlaywrightCrawler(
proxy_configuration=proxy_configuration,
session_pool=SessionPool(max_pool_size=3),
max_requests_per_crawl=100,
)
@crawler.router.default_handler
async def handler(context: PlaywrightCrawlingContext) -> None:
info = context.proxy_info
context.log.info(f'{context.request.url} via {info.hostname if info else "direct"}')
await context.enqueue_links()
await crawler.run(['https://example.com/'])
asyncio.run(main())Binding the Crawlee session pool to a small IP set
With rented static IPs, the session pool settings matter more than the proxy list. If the pool holds many more sessions than you have IPs, several sessions share each address and the site sees one IP juggling multiple cookie jars. A practical starting point is one session per IP: maxPoolSize (Python: max_pool_size) equal to the number of proxies, and concurrency no higher than that.
When a session is retired, for example after a 403 or a captcha page, its replacement is bound to the next proxy in the rotation. On a small pool that soon cycles back to the address that was just blocked, so slow down instead of retrying harder. Lower maxRequestsPerMinute, keep persistCookiesPerSession on so returning sessions look like returning visitors, and treat repeated blocks on one domain as a signal about your crawl rate rather than about the IPs.
tieredProxyUrls: escalate only when you are blocked
Tiers replace rotation with escalation. Crawlee starts each domain at the lowest tier, moves up when it detects blocking, and periodically tries lower tiers again. This only works while the configuration is used by a crawler, not through manual newUrl() calls.
const proxyConfiguration = new ProxyConfiguration({
tieredProxyUrls: [
[null], // tier 0: direct
['http://USER:[email protected]:8080',
'http://USER:[email protected]:8080'], // tier 1: datacenter IPv4
['http://USER:[email protected]:8080'], // tier 2: static ISP
],
});In Python the argument is tiered_proxy_urls with the same list-of-lists shape and None for a direct tier. A sensible ladder for static IPs is direct for friendly sites, datacenter IPv4 next, and static ISP addresses held back for domains that are strict about datacenter ranges.
newUrlFunction: your own routing rules
Use a function when the choice depends on the request, such as sending .jp domains through Japanese IPs and everything else through US IPs. The function receives the session ID and, when available, the request, and returns a proxy URL or null. Crawlee no longer manages stickiness for you here, so derive the choice from the session ID if you want sessions to stay on one IP.
const POOLS = {
jp: ['http://USER:[email protected]:8080', 'http://USER:[email protected]:8080'],
us: ['http://USER:[email protected]:8080', 'http://USER:[email protected]:8080'],
};
const hash = (s) => [...s].reduce((h, c) => (h * 31 + c.charCodeAt(0)) >>> 0, 0);
const proxyConfiguration = new ProxyConfiguration({
newUrlFunction: (sessionId, { request } = {}) => {
const host = request ? new URL(request.url).hostname : '';
const pool = host.endsWith('.jp') ? POOLS.jp : POOLS.us;
return pool[hash(sessionId ?? '') % pool.length];
},
});import zlib
from urllib.parse import urlparse
from crawlee.proxy_configuration import ProxyConfiguration
POOLS = {
'jp': ['http://USER:[email protected]:8080', 'http://USER:[email protected]:8080'],
'us': ['http://USER:[email protected]:8080', 'http://USER:[email protected]:8080'],
}
def pick_proxy(session_id=None, request=None):
host = urlparse(request.url).hostname if request else ''
pool = POOLS['jp'] if host.endswith('.jp') else POOLS['us']
return pool[zlib.crc32((session_id or '').encode()) % len(pool)]
proxy_configuration = ProxyConfiguration(new_url_function=pick_proxy)In Python the function may be a plain or an async function; Crawlee awaits the result when it is awaitable. Keep it fast and free of network calls, since it runs before every request.
Sizing a static IP pool for a Crawlee project
Crawlee does not need a giant rotating network to be useful. For catalog monitoring, price tracking or research crawls on a known set of sites, a handful of dedicated addresses with sane rate limits often does the job, and every IP in the pool behaves the same way on every run. A ProxyV4 private IPv4 has one tenant for the whole term, keeps its address while active and on renewal, and has unmetered traffic, which matters once Playwright starts downloading full pages. The web scraping use case goes through planning a crawl around static IPs.
Each proxy arrives as ip:port:user:pass and supports HTTP(S) and SOCKS5 with a login; use the HTTP form for Crawlee, since Chromium cannot authenticate to SOCKS5 proxies. Datacenter IPv4 is available in seven countries, which pairs naturally with the newUrlFunction routing above, and static ISP addresses in the US and Vietnam can fill a higher tier.
Be honest about the limits. If a job needs thousands of fresh addresses or a new IP for every request, a static pool is the wrong shape, and ProxyV4 does not sell rotating proxies. Crawl only what you are allowed to collect, and respect each site's terms and rate limits. Start with one or two IPs, then grow the list from the live prices.
Debugging proxy failures in a crawl
Every request fails at once
Usually a credential or format problem. A 407 or ERR_TUNNEL_CONNECTION_FAILED from Playwright points at the URL string, often an unencoded password character. The tunnel connection guide lists the other causes, and the proxy checker tests each line outside Crawlee.
One proxy fails, others work
Log proxyInfo.url with each failure and group by it. A single bad entry in round-robin mode shows up as every third or fourth request failing. Remove it, or let a tiered setup skip past it.
The site sees your real IP
Check that proxyConfiguration is passed to the crawler, not just created, and that no null or None entry sits in the list. For browser crawlers, also confirm that launch options do not override the proxy; Playwright's own proxy options take a different shape from Crawlee's.
Frequently asked questions
Does Crawlee rotate proxies automatically?
With proxyUrls, yes: requests without a session rotate round-robin, and each new session is bound to the next proxy. With newUrlFunction, rotation is whatever your function returns.
Can I use proxyUrls and tieredProxyUrls together?
No. Choose one mode per ProxyConfiguration; the Python version raises an error if you pass more than one of the three options.
How do I see which proxy a request used?
Read proxyInfo in the JavaScript request handler or context.proxy_info in Python. Both include the URL, host, port and session ID.
Can Crawlee use SOCKS5 proxies?
HTTP crawlers can, but browser crawlers on Chromium cannot authenticate to SOCKS5. ProxyV4 proxies accept HTTP and SOCKS5 on the same order, so use the HTTP form in Crawlee.
Is bandwidth billed if Playwright loads heavy pages?
Not on ProxyV4 private IPv4 or static ISP proxies, where traffic is unmetered. You pay per IP per month, so heavy pages do not change the cost.
Build a Crawlee pool from your own static IPs
Order private IPv4 in seven countries or static ISP in the US and Vietnam, each with one tenant, a fixed address and unmetered traffic.
Order one IP