Awọn proxy Scrapy: meta['proxy'], eto middleware ati yiyi pada
Láti ọwọ́ Ẹgbẹ́ Onímọ̀-ẹ̀rọ ProxyV4 · Updated:
Ninu Scrapy, a n ṣeto proxy fun ibeere kọọkan nipasẹ request.meta['proxy'] = 'http://user:pass@ip:port', ati pe HttpProxyMiddleware ti a ṣe sinu rẹ (eto 750) n sọ abala user:pass di akọsori Proxy-Authorization. Fi downloader middleware kekere kan ti n ṣiṣẹ ṣaaju rẹ kun, ni 740, ki gbogbo ibeere ti spider rẹ ba gbe jade le mu IP kan lati inu adagun ProxyV4 rẹ.
request.meta['proxy'] = 'http://user:pass@ip:port'- IP 1$1.85
- 3+ IP$1.70
- 10+ IP$1.60
- 25+ IP$1.50
- 50+ IP$1.40
- 100+ IP$1.30
- 200+ IP$1.20
- 500+ IP$1.10
- IP 1$1.85
- 3+ IP$1.70
- 10+ IP$1.60
- 25+ IP$1.50
- 50+ IP$1.40
- 100+ IP$1.30
- 200+ IP$1.20
- 500+ IP$1.10
Idi ti nọmba middleware fi ṣe pataki
Awọn downloader middleware maa n ṣiṣẹ ni ilana ti n pọ si siwaju nigbati wọn ba n jade. HttpProxyMiddleware ni 750 n ka meta['proxy'], o n yọ awọn alaye iwọle kuro ninu URL, o si n kọ akọsori auth. Ti middleware ti tirẹ ba ṣeto proxy lẹhin rẹ, fun apẹẹrẹ ni 800, eyi ti a ṣe sinu rẹ ti ṣiṣẹ tẹlẹ, a ko ni yi awọn alaye iwọle pada rara, proxy naa yoo si da 407 pada. Fiforukọsilẹ ti tirẹ ni 740 n fi si ipo akọkọ.
| Eto | Iye ti a daba | Ipa |
|---|---|---|
request.meta['proxy'] | 'http://user:pass@ip:port' | N fi ibeere yẹn ranṣẹ nipasẹ proxy ProxyV4 ti a pese |
| Eto middleware rẹ | 740 | N ṣiṣẹ ṣaaju HttpProxyMiddleware (750), nitorinaa a n se koodu alaye iwọle naa |
CONCURRENT_REQUESTS_PER_IP | 2 si 4 | N fi opin si awọn ibeere igbakanna fun IP ibi-afẹde kọọkan, iye iwọntunwọnsi lori bi a ṣe n kan olupin kọọkan le to |
AUTOTHROTTLE_ENABLED | True | N gba Scrapy laaye lati fa akoko idaduro gùn tabi kuru da lori bi ibi-afẹde naa ṣe n dahun ni kiakia to |
Agbekalẹ middleware yiyi pada ni kukuru
- Ka akojọ proxy lati inu awọn eto ni
from_crawler()ki o si tọju rẹ sori middleware instance - Ninu
process_request(), firequest.meta['proxy'] = random.choice(self.proxies)si i ayafi ti ibeere naa ba ti ni ọkan tẹlẹ - Nigbati idahun ba pada pẹlu
429tabi503, tun gbiyanju pẹlu proxy miiran lati inu adagun dipo eyi ti a kọ silẹ lẹsẹkẹsẹ - Da
DOWNLOAD_DELAY = 1.0pọ mọRANDOMIZE_DOWNLOAD_DELAY = Trueki aaye laarin awọn ibeere le yatọ dipo ki o ma lu ni akoko kanna bi aago metronome
Ṣiṣe iṣiro iwọn adagun naa
Crawl Scrapy le fi ọpọlọpọ awọn ibeere ranṣẹ ni iṣẹju kan, nitorinaa iye awọn IP n pinnu bi a ṣe n lo ọkọọkan ni pẹlẹmọṣẹ to. Pẹlu ProxyV4 ko si aṣẹ ti o kere julọ ti a gbọdọ ra ati pe iye owo fun IP kọọkan n dinku bi iye ti o n ra ṣe n pọ si, ati pe a ko n wọn iye data IPv4 aladani, nitorinaa crawl nla n na iye kanna lori bandwidth bi crawl kekere. Onibara kan ṣoṣo lo n lo IP kọọkan, nitorinaa orukọ rere rẹ n fi iṣẹ crawl ti tirẹ nikan han kii ṣe ti elomiran. Fun ṣiṣe eto iwọn crawl ati iwa rere si olupin, wo itọsọna web scraping.
Ìgbésẹ̀ ní ìgbésẹ̀
- 1
Mura awọn URL proxy silẹ
Yi awọn ila
ip:port:user:passti ProxyV4 rẹ pada sihttp://user:pass@ip:portpẹlu oluyipada ọna kika proxy ki o si tọju wọn sinu settings tabi faili kan. - 2
Kọ ProxyPoolMiddleware
Ninu
middlewares.py, kọprocess_request(self, request, spider)ki o le ṣetorequest.meta['proxy'] = random.choice(self.proxies). - 3
Forukọsilẹ rẹ ṣaaju eyi ti a ṣe sinu rẹ
Ninu
settings.py, fi'myproject.middlewares.ProxyPoolMiddleware': 740kunDOWNLOADER_MIDDLEWARESki o si ṣetoCONCURRENT_REQUESTS_PER_IP = 2. - 4
Mudaniloju yiyi pada
Gba
https://proxyv4.com/api/my-ipni igba diẹ pẹluscrapy shelltabi spider idanwo kukuru kan ki o si rii daju pe IP ti o n gba n yipada kọja adagun rẹ.
Àwọn ìbéèrè tí a ń béèrè lemọ́lemọ́
Proxy middleware mi ni ipo pataki 800 n gba awọn aṣiṣe 407. Kí nìdí?
Ni 800 o n ṣiṣẹ lẹhin HttpProxyMiddleware (750), nitorinaa user:pass ti o wa ninu URL proxy rẹ ko ni yipada di akọsori Proxy-Authorization rara. Gbe middleware rẹ si isalẹ 750, fun apẹẹrẹ 740.
Ṣe MO le ṣeto proxy fun ibeere kan ṣoṣo?
Bẹẹni. Fi meta={'proxy': 'http://user:pass@ip:port'} ranṣẹ nigbati o ba n ṣẹda Request naa; middleware ti a ṣe sinu rẹ yoo ṣakoso iwọle fun ibeere yẹn nikan.
Ṣe crawl ti awọn miliọnu oju-iwe yoo de opin bandwidth bi?
Kii ṣe lori awọn proxy IPv4 aladani tabi static ISP ti ProxyV4; awọn mejeeji ni bandwidth ailopin. Opin gidi ni bi ibi-afẹde naa ṣe gba ọ laaye lati crawl ni kiakia to.
Àwọn orísun
Awọn IP adaduro ti a ko wọn fun awọn crawl Scrapy
IPv4 aladani ni awọn orilẹ-ede 7 pẹlu static ISP ni US ati Vietnam, lati IP kan ṣoṣo si adagun nla kan, pẹlu idiyele gidi ti o dinku bi iye ti n pọ si.
Wo awọn idiyele US lọwọlọwọ