Is Web Scraping with Proxies Legal? What Actually Decides It
By the ProxyV4 Engineering Team · Updated:
Using a proxy does not make web scraping legal or illegal; the answer depends on what you collect, how you access it, what you do with it and which country's laws apply. The rules that usually matter are the site's terms of service, computer-misuse laws, copyright and database rights, and personal-data law such as the GDPR. This page is general information, not legal advice: for a real project, talk to a lawyer who knows your jurisdiction.
Is web scraping with proxies legal? The short answer
A proxy relays your requests through another IP address. That is a routine networking tool, used for testing, monitoring and keeping company traffic separate, and owning or using one is lawful in most places. It does not change the legal character of the scraping itself: lawful collection stays lawful through a proxy, and unlawful collection does not become lawful because the IP is different.
Where a proxy can matter is in how it is used. Rotating through addresses to keep going after a site has blocked you, told you to stop or revoked your access is very different from routing permitted traffic through a fixed IP. If an operator has clearly withdrawn permission, getting around that with new IPs can strengthen the case against you rather than protect you.
The kinds of rules that apply to scraping
Several separate bodies of law and contract can apply to one scraping project. Passing one test does not mean you pass the others.
Terms of service and contracts
Many sites forbid or limit automated collection in their terms. How enforceable those terms are depends on the jurisdiction and on how you agreed to them; terms you accepted when creating an account are generally easier to enforce than a link in a footer. A breach of contract claim does not depend on whether the data was public.
Computer-misuse laws
Most countries have laws against accessing computers without authorization, such as the Computer Fraud and Abuse Act in the United States and the Computer Misuse Act 1990 in the United Kingdom. Bypassing logins, using someone else's credentials, or getting around technical barriers raises the most risk under these laws. How they apply to public pages differs by country and, in the US, by court.
Copyright and database rights
Plain facts are generally not protected by copyright, but the way they are written, selected or arranged can be, and articles, photos and reviews usually are. The European Union also protects databases that required substantial investment with a separate database right, which can cover extracting large parts of a database even when individual entries are factual. Republishing scraped content is riskier than analyzing it internally.
Personal data and privacy law
Under the GDPR, personal data that someone posted publicly is still personal data. Collecting names, profiles or contact details needs a lawful basis, a clear purpose, data minimization and respect for people's rights, and several European data protection authorities have taken action over scraped personal data. Other regions have their own laws, such as the California Consumer Privacy Act. If your project touches personal data, involve whoever handles privacy compliance before you start.
What US courts have said about public data
Two US decisions come up in almost every discussion of scraping, and both are often overstated. In Van Buren v. United States (2021) the Supreme Court read the CFAA's "exceeds authorized access" clause narrowly: it covers obtaining information from parts of a computer that are off limits to you, not misusing information you were entitled to access. The case was not about scraping, but it shaped later scraping disputes. The Van Buren opinion is on the Supreme Court's site.
In hiQ Labs v. LinkedIn, the Ninth Circuit in April 2022, on remand after Van Buren, again affirmed a preliminary injunction, finding that hiQ had raised serious questions about whether LinkedIn could use the CFAA against the scraping of publicly available profiles (Ninth Circuit opinion). That was a ruling at the preliminary injunction stage, not a final judgment that scraping public data is legal. Later in 2022 the district court found that hiQ had breached LinkedIn's user agreement, and the parties settled with a consent judgment against hiQ.
The practical lesson is narrower than most headlines: in that circuit, the CFAA may be a weaker weapon against scraping of truly public pages, but contract claims, copyright and privacy law still apply, and courts elsewhere may see it differently.
robots.txt and scraping terms of service
robots.txt is a voluntary standard, now written down as RFC 9309, that tells crawlers which paths the operator does not want visited. It is not a law by itself, but it is a clear, machine-readable statement of the owner's wishes. Ignoring it makes good faith harder to argue, and some terms of service refer to it directly.
Treat both as the operator's instructions. If robots.txt disallows a path, skip it. If the terms prohibit automated access, look for an API or a data license, or ask for written permission. If the operator sends you a cease-and-desist letter or blocks you, stop and get legal advice before doing anything else.
| Signal from the site | What it is | Sensible response |
|---|---|---|
| robots.txt Disallow | The owner's crawling preferences | Do not fetch those paths |
| Terms of service clause on automation | Contract terms; enforceability varies | Use an API or license, or ask for permission |
| Login or paywall | Access control | Do not automate behind it without permission |
| HTTP 429 or Retry-After | A rate limit | Slow down and wait as instructed |
| Block, cease-and-desist or revoked access | Permission withdrawn | Stop and consult a lawyer |
Habits that keep scraping on the right side
None of these makes a project legal on its own, but together they reduce both legal and practical risk, and they make it easier to show you acted in good faith.
- Prefer official channels. Use an API, a data feed or a license where one exists.
- Stay out of protected areas. Do not automate behind logins or paywalls without the owner's permission.
- Collect only what you need. Avoid personal data unless you have a lawful basis, and delete what you do not use.
- Do not republish wholesale. Analysis and facts carry less risk than copying articles, photos or reviews.
- Be gentle and identifiable. Respect robots.txt and rate limits, and send a User-Agent with a contact address.
- Stop when asked. Treat blocks and complaints as a reason to pause and review, not to switch IPs.
A polite crawler in Python
This sketch checks robots.txt before each URL, uses the site's Crawl-delay when it sets one, identifies itself with a descriptive User-Agent, and stops the run when the site signals a rate limit. Run it with requests installed; it goes through one fixed proxy.
import time
from urllib.robotparser import RobotFileParser
import requests
PROXY = "http://USER:[email protected]:8080"
UA = "ExampleCoResearch/1.0 (+https://example.com/crawler; [email protected])"
session = requests.Session()
session.proxies = {"http": PROXY, "https": PROXY}
session.headers["User-Agent"] = UA
robots = RobotFileParser()
resp = session.get("https://example.com/robots.txt", timeout=10)
robots.parse(resp.text.splitlines() if resp.status_code == 200 else [])
delay = robots.crawl_delay(UA) or 5 # seconds between requests; use the site's value when it sets one
for url in ["https://example.com/catalog/1", "https://example.com/catalog/2"]:
if not robots.can_fetch(UA, url):
print("skipped by robots.txt:", url)
continue
r = session.get(url, timeout=10)
if r.status_code in (429, 503):
wait = r.headers.get("Retry-After", "60")
print("site asked us to slow down; stopping this run. Retry-After:", wait)
break
print(r.status_code, url)
time.sleep(delay)For a full collection workflow, including scheduling and storage, see the web scraping use case.
Where accountable static IPs fit
For scraping you are allowed to do, a fixed IP is the more accountable choice: the operator sees the same address every time, can match it to your User-Agent and contact details, and can allow or rate-limit it on purpose. Some data partners will only grant access to traffic from IPs you register with them. ProxyV4 sells private datacenter IPv4 in seven countries and static ISP proxies in the United States and Vietnam, one tenant per IP for the whole term, with HTTP(S) and SOCKS5, username and password login, unmetered traffic and the same IP when you renew before expiry. The web scraping proxy page covers typical jobs.
It is the wrong tool for getting around blocks, rate limits or terms, and no proxy changes your legal position. It also does not fit work that needs a new IP per request (rotating proxies are waitlist only) or mobile carrier IPs. If static IPs suit your project, check live prices and test one IP first.
Frequently asked questions
Is scraping public data legal in the United States?
Often, but not automatically. Public availability can lower the risk under the CFAA after cases like hiQ v. LinkedIn, but terms of service, copyright and privacy laws can still apply. Ask a lawyer about your specific case.
Is ignoring robots.txt illegal?
robots.txt is a voluntary standard rather than a law in itself. Ignoring it can still count against you in a dispute and may breach terms that reference it.
Does using a proxy make scraping anonymous or safer legally?
No. A proxy changes the IP a site sees, not your legal obligations, and using new IPs to continue after being blocked can make your position worse.
Can I scrape personal data from public profiles under GDPR?
Public personal data is still personal data under the GDPR, so you need a lawful basis and must meet the regulation's other duties. Get privacy advice before collecting it.
Does ProxyV4 sell rotating proxies for scraping?
Not at the moment; rotating proxies are waitlist only. ProxyV4 sells private datacenter IPv4 and static ISP proxies with unmetered traffic, one tenant per IP.
Sources
- Computer Fraud and Abuse Act, 18 U.S.C. § 1030 (Cornell LII)
- Van Buren v. United States (2021), Supreme Court opinion, No. 19-783
- hiQ Labs v. LinkedIn, Ninth Circuit opinion, No. 17-16783 (April 18, 2022)
- European Commission: Legal framework of EU data protection (GDPR)
- RFC 9309: Robots Exclusion Protocol (RFC Editor)
Responsible scraping from accountable, private IPs
Fixed private IPv4 and static ISP proxies, one tenant per IP, so site operators always see the same address. Start with a single IP and scale once your project is cleared.
See live prices