Configuring Proxies in Crawlee: proxyUrls, Tiers, Custom Functions and Sessions

· Updated:

Crawlee proxy configuration happens in one object: create a ProxyConfiguration with your proxyUrls (or tieredProxyUrls, or a newUrlFunction) and pass it to the crawler as proxyConfiguration. Crawlee then hands each request a proxy, and when the session pool is on it keeps every session on the same IP. Below is how each of the three modes behaves in the JavaScript and Python versions, and how to size the session pool for a small set of static IPs.

Build a Crawlee pool from your own static IPs

How Crawlee chooses a proxy for a request

Before each request, the crawler asks its ProxyConfiguration for a URL, passing the current session ID when the session pool is enabled. With a plain list, requests without a session rotate round-robin. A session ID seen for the first time gets the next proxy in the rotation, and every later call with that ID returns the same proxy. That binding is what keeps cookies, logins and the IP consistent for one simulated visitor.

The proxy used is exposed in the request handler as proxyInfo in JavaScript and context.proxy_info in Python, with fields such as url, hostname, port, sessionId and, when tiers are used, the tier number. Log it while you develop; it is the quickest way to tell a proxy failure from a parsing bug.

proxyUrls: a fixed list of your own proxies

This is the right mode when you own a known set of static IPs. Put the credentials in each URL with the http:// scheme. If a password contains @, : or /, encode it first with encodeURIComponent in Node or urllib.parse.quote in Python. A null (JavaScript) or None (Python) entry means a direct connection, which is handy for mixing your own IP into the rotation during tests.

JavaScript and TypeScript

import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';

const proxyConfiguration = new ProxyConfiguration({
    proxyUrls: [
        'http://USER:[email protected]:8080',
        'http://USER:[email protected]:8080',
        'http://USER:[email protected]:8080',
    ],
});

const crawler = new PlaywrightCrawler({
    proxyConfiguration,
    useSessionPool: true,
    persistCookiesPerSession: true,
    sessionPoolOptions: { maxPoolSize: 3 },
    maxConcurrency: 3,
    async requestHandler({ request, page, proxyInfo, session, log, enqueueLinks }) {
        log.info(`${request.url} via ${proxyInfo?.hostname} (session ${session?.id})`);
        await enqueueLinks();
    },
});

await crawler.run(['https://example.com/']);

The same options work on CheerioCrawler, HttpCrawler, JSDOMCrawler and PuppeteerCrawler. For browser crawlers, Crawlee passes the username and password to the browser for you, which saves the extension workarounds that raw Chrome automation needs.

Python

In Crawlee for Python the constructor takes snake_case keyword arguments, and you must pass exactly one of proxy_urls, tiered_proxy_urls or new_url_function. The session pool is on by default.

import asyncio
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
from crawlee.proxy_configuration import ProxyConfiguration
from crawlee.sessions import SessionPool

proxy_configuration = ProxyConfiguration(proxy_urls=[
    'http://USER:[email protected]:8080',
    'http://USER:[email protected]:8080',
    'http://USER:[email protected]:8080',
])

async def main() -> None:
    crawler = PlaywrightCrawler(
        proxy_configuration=proxy_configuration,
        session_pool=SessionPool(max_pool_size=3),
        max_requests_per_crawl=100,
    )

    @crawler.router.default_handler
    async def handler(context: PlaywrightCrawlingContext) -> None:
        info = context.proxy_info
        context.log.info(f'{context.request.url} via {info.hostname if info else "direct"}')
        await context.enqueue_links()

    await crawler.run(['https://example.com/'])

asyncio.run(main())

Binding the Crawlee session pool to a small IP set

With rented static IPs, the session pool settings matter more than the proxy list. If the pool holds many more sessions than you have IPs, several sessions share each address and the site sees one IP juggling multiple cookie jars. A practical starting point is one session per IP: maxPoolSize (Python: max_pool_size) equal to the number of proxies, and concurrency no higher than that.

When a session is retired, for example after a 403 or a captcha page, its replacement is bound to the next proxy in the rotation. On a small pool that soon cycles back to the address that was just blocked, so slow down instead of retrying harder. Lower maxRequestsPerMinute, keep persistCookiesPerSession on so returning sessions look like returning visitors, and treat repeated blocks on one domain as a signal about your crawl rate rather than about the IPs.

tieredProxyUrls: escalate only when you are blocked

Tiers replace rotation with escalation. Crawlee starts each domain at the lowest tier, moves up when it detects blocking, and periodically tries lower tiers again. This only works while the configuration is used by a crawler, not through manual newUrl() calls.

const proxyConfiguration = new ProxyConfiguration({
    tieredProxyUrls: [
        [null],                                            // tier 0: direct
        ['http://USER:[email protected]:8080',
         'http://USER:[email protected]:8080'],            // tier 1: datacenter IPv4
        ['http://USER:[email protected]:8080'],           // tier 2: static ISP
    ],
});

In Python the argument is tiered_proxy_urls with the same list-of-lists shape and None for a direct tier. A sensible ladder for static IPs is direct for friendly sites, datacenter IPv4 next, and static ISP addresses held back for domains that are strict about datacenter ranges.

newUrlFunction: your own routing rules

Use a function when the choice depends on the request, such as sending .jp domains through Japanese IPs and everything else through US IPs. The function receives the session ID and, when available, the request, and returns a proxy URL or null. Crawlee no longer manages stickiness for you here, so derive the choice from the session ID if you want sessions to stay on one IP.

const POOLS = {
    jp: ['http://USER:[email protected]:8080', 'http://USER:[email protected]:8080'],
    us: ['http://USER:[email protected]:8080', 'http://USER:[email protected]:8080'],
};

const hash = (s) => [...s].reduce((h, c) => (h * 31 + c.charCodeAt(0)) >>> 0, 0);

const proxyConfiguration = new ProxyConfiguration({
    newUrlFunction: (sessionId, { request } = {}) => {
        const host = request ? new URL(request.url).hostname : '';
        const pool = host.endsWith('.jp') ? POOLS.jp : POOLS.us;
        return pool[hash(sessionId ?? '') % pool.length];
    },
});
import zlib
from urllib.parse import urlparse
from crawlee.proxy_configuration import ProxyConfiguration

POOLS = {
    'jp': ['http://USER:[email protected]:8080', 'http://USER:[email protected]:8080'],
    'us': ['http://USER:[email protected]:8080', 'http://USER:[email protected]:8080'],
}

def pick_proxy(session_id=None, request=None):
    host = urlparse(request.url).hostname if request else ''
    pool = POOLS['jp'] if host.endswith('.jp') else POOLS['us']
    return pool[zlib.crc32((session_id or '').encode()) % len(pool)]

proxy_configuration = ProxyConfiguration(new_url_function=pick_proxy)

In Python the function may be a plain or an async function; Crawlee awaits the result when it is awaitable. Keep it fast and free of network calls, since it runs before every request.

Sizing a static IP pool for a Crawlee project

Crawlee does not need a giant rotating network to be useful. For catalog monitoring, price tracking or research crawls on a known set of sites, a handful of dedicated addresses with sane rate limits often does the job, and every IP in the pool behaves the same way on every run. A ProxyV4 private IPv4 has one tenant for the whole term, keeps its address while active and on renewal, and has unmetered traffic, which matters once Playwright starts downloading full pages. The web scraping use case goes through planning a crawl around static IPs.

Each proxy arrives as ip:port:user:pass and supports HTTP(S) and SOCKS5 with a login; use the HTTP form for Crawlee, since Chromium cannot authenticate to SOCKS5 proxies. Datacenter IPv4 is available in seven countries, which pairs naturally with the newUrlFunction routing above, and static ISP addresses in the US and Vietnam can fill a higher tier.

Be honest about the limits. If a job needs thousands of fresh addresses or a new IP for every request, a static pool is the wrong shape, and ProxyV4 does not sell rotating proxies. Crawl only what you are allowed to collect, and respect each site's terms and rate limits. Start with one or two IPs, then grow the list from the live prices.

Debugging proxy failures in a crawl

Every request fails at once

Usually a credential or format problem. A 407 or ERR_TUNNEL_CONNECTION_FAILED from Playwright points at the URL string, often an unencoded password character. The tunnel connection guide lists the other causes, and the proxy checker tests each line outside Crawlee.

One proxy fails, others work

Log proxyInfo.url with each failure and group by it. A single bad entry in round-robin mode shows up as every third or fourth request failing. Remove it, or let a tiered setup skip past it.

The site sees your real IP

Check that proxyConfiguration is passed to the crawler, not just created, and that no null or None entry sits in the list. For browser crawlers, also confirm that launch options do not override the proxy; Playwright's own proxy options take a different shape from Crawlee's.

Frequently asked questions

Does Crawlee rotate proxies automatically?

With proxyUrls, yes: requests without a session rotate round-robin, and each new session is bound to the next proxy. With newUrlFunction, rotation is whatever your function returns.

Can I use proxyUrls and tieredProxyUrls together?

No. Choose one mode per ProxyConfiguration; the Python version raises an error if you pass more than one of the three options.

How do I see which proxy a request used?

Read proxyInfo in the JavaScript request handler or context.proxy_info in Python. Both include the URL, host, port and session ID.

Can Crawlee use SOCKS5 proxies?

HTTP crawlers can, but browser crawlers on Chromium cannot authenticate to SOCKS5. ProxyV4 proxies accept HTTP and SOCKS5 on the same order, so use the HTTP form in Crawlee.

Is bandwidth billed if Playwright loads heavy pages?

Not on ProxyV4 private IPv4 or static ISP proxies, where traffic is unmetered. You pay per IP per month, so heavy pages do not change the cost.

Build a Crawlee pool from your own static IPs

Order private IPv4 in seven countries or static ISP in the US and Vietnam, each with one tenant, a fixed address and unmetered traffic.

Order one IP

Related articles

Use cases
Designing a scraper around a pool of dedicated IPs
A planning guide for data teams: size a pool of private static IPs, rotate them in your own code, pick IPv4, ISP or IPv6, and collect data responsibly.
Proxy setup
Playwright proxy authentication, one static IP per context
Give each Playwright browser context its own authenticated HTTP(S) proxy in Node.js or Python, match timezone and locale to the IP and confirm the exit address.
Proxy setup
Puppeteer with an authenticated proxy: launch flag plus login
Puppeteer proxy authentication, working example first: --proxy-server for the address, page.authenticate() for the login, then SOCKS5, contexts and fixes.
Knowledge base
Static or Rotating Proxy? Start From the Task, Not the Product
Static vs rotating proxies, decided job by job: account farming, ad accounts, marketplace shops, scraping and SEO checks, plus sticky sessions and DIY rotation.
Proxies by platform
Proxies for web scraping: a fixed pool of IPs with unmetered traffic
Web scraping proxies for Python, Scrapy, Playwright and Selenium: a pool of private IPv4 and static ISP addresses, unlimited bandwidth, flat per-IP pricing.
Knowledge base
Sending Postman Requests Through a Proxy, With Login and Exceptions
Postman proxy settings explained: system vs custom proxy, proxy auth, the bypass list, env-variable overrides, agent caveats, and how to prove the exit IP.
Knowledge base
Chaining Burp Suite to an Upstream or SOCKS Proxy for Authorized Testing
Set a Burp Suite upstream proxy or SOCKS proxy with authentication, route only in-scope hosts, and confirm the egress IP before an authorized test starts.
Knowledge base
Routing WordPress Outbound Requests Through a Proxy with wp-config.php
Set WordPress proxy settings in wp-config with WP_PROXY_HOST and four related constants, test them with wp_remote_get, and learn which plugin calls ignore them.
Free tools
Check proxies in bulk: status, exit IP and latency
Free online proxy checker: paste up to 100 ip:port:user:pass lines and see which proxies work, their exit IP and latency, plus what each failure result means.
Products
Dedicated datacenter proxies: one private IPv4 per customer
Private IPv4 proxies in the US, Vietnam, Japan, South Korea, Singapore, the UK and Hong Kong. One tenant per IP, HTTP(S) and SOCKS5, no traffic metering.