Ama-proxy e-Scrapy: meta['proxy'], ukulandelana kwe-middleware nokushintshanisa
Ibhalwe yiThimba Lonjiniyela lakwa-ProxyV4 · Updated:
Ku-Scrapy, i-proxy isethwa esicelweni ngasinye kusetshenziswa request.meta['proxy'] = 'http://user:pass@ip:port', kanti i-HttpProxyMiddleware eyakhelwe ngaphakathi (ukulandelana 750) iguqula ingxenye ye-user:pass ibe yisihloko se-Proxy-Authorization. Engeza i-downloader middleware encane esebenza ngaphambi kwayo, ku-740, ukuze sonke isicelo i-spider yakho esikhiphayo sikwazi ukukhetha i-IP eqoqweni lakho le-ProxyV4.
request.meta['proxy'] = 'http://user:pass@ip:port'- 1 IP$1.85
- 3+ IPs$1.70
- 10+ IPs$1.60
- 25+ IPs$1.50
- 50+ IPs$1.40
- 100+ IPs$1.30
- 200+ IPs$1.20
- 500+ IPs$1.10
- 1 IP$1.85
- 3+ IPs$1.70
- 10+ IPs$1.60
- 25+ IPs$1.50
- 50+ IPs$1.40
- 100+ IPs$1.30
- 200+ IPs$1.20
- 500+ IPs$1.10
Kungani inombolo ye-middleware ibalulekile
Ama-downloader middleware asebenza ngokulandelana okwenyukayo lapho kuphunywa. I-HttpProxyMiddleware ku-750 ifunda i-meta['proxy'], isuse iziqinisekiso kwi-URL bese ibhala isihloko sokufakazela ubuqiniso. Uma i-middleware yakho isetha i-proxy ngemva kwalokho, ngokwesibonelo ku-800, eyakhelwe ngaphakathi isuke isisebenzile kakade, iziqinisekiso aziguqulwa neze, bese i-proxy iphendula ngo-407. Ukubhalisa eyakho ku-740 kuyibeka phambili.
| Isilungiselelo | Inani eliphakanyisiwe | Umphumela |
|---|---|---|
request.meta['proxy'] | 'http://user:pass@ip:port' | Ithumela leso sicelo nge-proxy enikeziwe ye-ProxyV4 |
| Ukulandelana kwe-middleware yakho | 740 | Isebenza ngaphambi kwe-HttpProxyMiddleware (750), ukuze imininingwane yokungena ibhalwe ngekhodi |
CONCURRENT_REQUESTS_PER_IP | 2 kuya ku-4 | Inquma izicelo ezihambisana ngesikhathi esisodwa nge-IP ngayinye okuyiwa kuyo, okuwumkhawulo omuhle wendlela iseva ngayinye ecelwa ngayo |
AUTOTHROTTLE_ENABLED | True | Ivumela i-Scrapy ukuba yelule noma ifinyeze ukulibala kuye ngokuthi isizinda siphendula ngokushesha okungakanani |
Uhlaka lwe-middleware yokushintshanisa
- Funda uhlu lwama-proxy kuzilungiselelo ku-
from_crawler()bese uligcina kusibonelo se-middleware - Ku-
process_request(), nikarequest.meta['proxy'] = random.choice(self.proxies)ngaphandle uma isicelo sesivele sinayo - Uma impendulo ibuyela ingasho u-
429noma u-503, yiphinde usebenzise i-proxy ehlukile eqoqweni esikhundleni sale esanda kwaliwa - Hlanganisa i-
DOWNLOAD_DELAY = 1.0ne-RANDOMIZE_DOWNLOAD_DELAY = Trueukuze isikhathi phakathi kwezicelo sihluke kunokuba sishaye njalo njengewashi
Ukukala ubukhulu beqoqo
Ukucwaninga kwe-Scrapy kungathumela izicelo eziningi ngomzuzu, ngakho inani lama-IP linquma indlela ngayinye esetshenziswa ngayo ngobumnene. Nge-ProxyV4 akukho ukuthenga okuncane okuyimpoqo futhi intengo nge-IP ngayinye yehla njengoba inani lenyuka, kanti ithrafikhi ye-IPv4 yangasese ayikalwa, ngakho ukucwaninga okukhulu kubiza ngokufanayo nokuncane kwi-bandwidth. I-IP ngayinye isetshenziswa ikhasimende elilodwa kuphela, ngakho isithunzi sayo sibonisa ukucwaninga kwakho kuphela hhayi okomunye umuntu. Ukuhlela umthamo wokucwaninga nenhlonipho, bheka i-web scraping guide.
Isinyathelo ngesinyathelo
- 1
Lungisa ama-URL e-proxy
Guqula imigqa yakho ye-ProxyV4 ethi
ip:port:user:passibe yi-http://user:pass@ip:portnge-proxy format converter bese uyigcina kuzilungiselelo noma efayeleni. - 2
Bhala i-ProxyPoolMiddleware
Ku-
middlewares.py, faka ikhodi ye-process_request(self, request, spider)ukuze isetherequest.meta['proxy'] = random.choice(self.proxies). - 3
Yibhalise ngaphambi kweyakhelwe ngaphakathi
Ku-
settings.py, engeza'myproject.middlewares.ProxyPoolMiddleware': 740ku-DOWNLOADER_MIDDLEWARESbese usethaCONCURRENT_REQUESTS_PER_IP = 2. - 4
Qinisekisa ukushintshanisa
Landa i-
https://proxyv4.com/api/my-ipizikhathi ezimbalwa nge-scrapy shellnoma nge-spider emfushane yokuhlola bese uhlola ukuthi i-IP eboniswayo iyashintsha yini eqoqweni lakho.
Imibuzo evame ukubuzwa
I-proxy middleware yami enokubaluleka okungu-800 ithola amaphutha angu-407. Kungani?
Ku-800 isebenza ngemva kwe-HttpProxyMiddleware (750), ngakho i-user:pass kwi-URL yakho ye-proxy ayiguqulwa neze ibe yisihloko se-Proxy-Authorization. Hambisa i-middleware yakho ngaphansi kuka-750, isib. 740.
Ngingakwazi ukusetha i-proxy esicelweni esisodwa nje kuphela?
Yebo. Dlulisela i-meta={'proxy': 'http://user:pass@ip:port'} lapho wenza i-Request; i-middleware eyakhelwe ngaphakathi iphatha ukungena kwaleso sicelo kuphela.
Ingabe ukucwaninga izigidi zamakhasi kuzofinyelela emkhawulweni we-bandwidth?
Hhayi kuma-proxy e-IPv4 angasese noma ama-static ISP e-ProxyV4; womabili ane-bandwidth engenamkhawulo. Umkhawulo wangempela yindlela isizinda esikuvumela ngayo ukucwaninga ngokushesha.
Imithombo
Ama-IP amile angenamkhawulo wokucwaninga kwe-Scrapy
I-IPv4 yangasese emazweni angu-7 kanye ne-ISP emile e-US nase-Vietnam, kusukela ku-IP eyodwa kuya eqoqweni elikhulu, enamanani abukhoma ngokwenani.
Bheka izintengo ezibukhoma zase-US