Proxy-yada Scrapy: meta['proxy'], kala-horreynta middleware iyo wareejinta
Waxaa qoray Kooxda Injineernimada ee ProxyV4 · Updated:
Gudaha Scrapy, proxy waxaa loo dejiyaa codsi kasta iyadoo la adeegsanayo request.meta['proxy'] = 'http://user:pass@ip:port', waxaana HttpProxyMiddleware-ka dhex-dhisan (kala-horreyntiisu tahay 750) uu qeybta user:pass u beddelaa madax-madaxeedka Proxy-Authorization. Ku dar downloader middleware yar oo ka hor shaqeeya, oo ah 740, codsi kasta oo uu spider-kaagu soo saarana wuxuu IP ka dooran karaa kaydkaaga ProxyV4.
request.meta['proxy'] = 'http://user:pass@ip:port'- 1 IPUS$1.85
- 3+ IPUS$1.70
- 10+ IPUS$1.60
- 25+ IPUS$1.50
- 50+ IPUS$1.40
- 100+ IPUS$1.30
- 200+ IPUS$1.20
- 500+ IPUS$1.10
- 1 IPUS$1.85
- 3+ IPUS$1.70
- 10+ IPUS$1.60
- 25+ IPUS$1.50
- 50+ IPUS$1.40
- 100+ IPUS$1.30
- 200+ IPUS$1.20
- 500+ IPUS$1.10
Sababta lambarka middleware-ku muhiim u yahay
Downloader middleware-yadu waxay u socdaan hab korodhsiimo ah marka ay baxayaan. HttpProxyMiddleware-ka ku jira 750 wuxuu akhriyaa meta['proxy'], wuxuu URL-ka ka saaraa xogta aqoonsiga wuxuuna qoraa madax-madaxeedka xaqiijinta. Haddii middleware-kaaga gaarka ahi uu proxy-ga dejiyo dabadeed, tusaale ahaan 800, kii dhex-dhisaa mar hore ayuu shaqeeyay, xogta aqoonsigana weligeed lama beddelo, proxy-guna wuxuu ku jawaabayaa 407. Diiwaangelinta kaaga lambarka 740 waxay ka dhigaysaa midka ugu horreeya.
| Dejinta | Qiimaha la soo jeediyay | Saamaynta |
|---|---|---|
request.meta['proxy'] | 'http://user:pass@ip:port' | Codsigaas wuxuu ku dirayaa proxy-ga la bixiyay ee ProxyV4 |
| Kala-horreynta middleware-kaaga | 740 | Wuxuu shaqeeyaa ka hor HttpProxyMiddleware (750), si gelitaanka loo koodheeyo |
CONCURRENT_REQUESTS_PER_IP | 2 to 4 | Wuxuu xaddidayaa codsiyada isku mar ah halkii IP ee loo socdo, xad edeb leh oo ku saabsan sida loo culeysiyo server kasta |
AUTOTHROTTLE_ENABLED | True | Wuxuu u oggolaanayaa Scrapy inuu kordhiyo ama soo gaabiyo dib-u-dhacyada iyadoo lagu saleynayo sida degdegga ah ee bartilmaameedku uga jawaabo |
Dulmar guud oo ku saabsan middleware-ka wareejinta
- Ka akhri liiska proxy-yada goobaha dejinta gudaha
from_crawler()kuna hay tusaalaha middleware-ka - Gudaha
process_request(), u qoondeerequest.meta['proxy'] = random.choice(self.proxies)haddii uusan codsigu mar hore mid sidanin - Markii jawaabtu ku soo noqoto
429ama503, dib ugu day proxy kale oo kaydka ka tirsan halkii aad ka isticmaali lahayd kii hadda la diiday - Isku dar
DOWNLOAD_DELAY = 1.0iyoRANDOMIZE_DOWNLOAD_DELAY = Truesi farqiga u dhexeeya codsiyadu u kala duwanaado halkii uu u garaaci lahaa sida saacad
Qiyaasidda xaddiga kaydka
Xog-ururinta Scrapy waxay daqiiqaddii diri kartaa codsiyo badan, sidaas darteed tirada IP-yada ayaa go'aamisa sida taxaddarka leh ee mid kasta loo isticmaalo. ProxyV4 ma lahan dalab ugu yar, qiimaha halkii IP-na wuu hoos u dhacaa markii tiradu korodho, taraafikada gaarka ah ee IPv4-na lama qiyaaso, markaa xog-ururin weyn iyo mid yar waxay ku kacayaan isku qiimo dhanka xaddiga xogta. IP kasta waxaa isticmaala hal macaamiil oo keliya, markaa sumcaddiisu waxay ka tarjumaysaa xog-ururintaada gaarka ah mana jirto cid kale. Si aad u qorshayso mugga xog-ururinta iyo hab-dhaqanka edebta leh, eeg hagaha web scraping-ka.
Tallaabo tallaabo
- 1
Diyaari URL-yada proxy-ga
U beddel khadadkaaga ProxyV4 ee
ip:port:user:passuna beddelhttp://user:pass@ip:portadigoo isticmaalaya beddelaha qaabka proxy-ga oo ku keydi dejimaha ama fayl. - 2
Qor ProxyPoolMiddleware
Gudaha
middlewares.py, ka hirgeliprocess_request(self, request, spider)si uu u dejiyorequest.meta['proxy'] = random.choice(self.proxies). - 3
Diiwaangeli ka hor kan dhex-dhisan
Gudaha
settings.py, ku dar'myproject.middlewares.ProxyPoolMiddleware': 740qaybtaDOWNLOADER_MIDDLEWARESoo dejiCONCURRENT_REQUESTS_PER_IP = 2. - 4
Xaqiiji wareejinta
Dhawr jeer soo doono
https://proxyv4.com/api/my-ipadigoo adeegsanayascrapy shellama spider tijaabo ah oo gaaban, oo hubi in IP-ga la soo sheegay uu isbeddelayo guud ahaan kaydkaaga.
Su'aalaha badanaa la isweydiiyo
Proxy middleware-kayga mudnaantiisu tahay 800 wuxuu helayaa khaladaadka 407. Maxaa sababay?
Markay tahay 800 wuxuu shaqeeyaa ka dib HttpProxyMiddleware (750), markaa user:pass ee ku jira URL-kaaga proxy weligood looma beddelo madax-madaxeedka Proxy-Authorization. U rar middleware-kaaga meel ka hoosaysa 750, tusaale 740.
Ma u dejin karaa proxy-ga hal codsi oo keliya?
Haa. Gudbi meta={'proxy': 'http://user:pass@ip:port'} marka aad abuureyso Request-ka; middleware-ka dhex-dhisan ayaa maareynaya gelitaanka codsigaas oo keliya.
Xog-ururinta malaayiin bog ma ku dhici doontaa xaddidaadda xogta?
Kuma dhacayso ProxyV4 IPv4 gaar ah ama proxy-yada ISP ee ma guurtada ah; labaduba waxay leeyihiin bandwidth xadla'aan ah. Xadka dhabta ahi waa sida degdegga ah ee bartilmaameedku kuu oggolaanayo inaad xogta u ururisid.
Ilaha
IP-yo ma guurto ah oo aan la xaddidin oo loogu talagalay Scrapy
IPv4 gaar ah oo ku yaal 7 waddan iyo weliba ISP ma guurto ah gudaha US iyo Vietnam, laga bilaabo hal IP ilaa kayd weyn, oo leh qiimaha tooska ah ee xaddiga.
Hubi qiimaha tooska ah ee US