Scrapy proxies: meta['proxy'], tsarin jeri na middleware da juyawa
Daga Tawagar Injiniyoyi ta ProxyV4 · Updated:
A Scrapy, ana saita proxy ga kowace buƙata ta hanyar request.meta['proxy'] = 'http://user:pass@ip:port', kuma ginanniyar HttpProxyMiddleware (lambar tsari 750) tana mayar da sashin user:pass zuwa kanun Proxy-Authorization. Ƙara ƙaramar downloader middleware da ke aiki gabaninta, a 740, kuma kowace buƙata da spider ɗinka ta bayar za ta iya zaɓar IP daga rukunin ProxyV4 ɗinka.
request.meta['proxy'] = 'http://user:pass@ip:port'- 1 IP$ 1.85
- 3+ IPs$ 1.70
- 10+ IPs$ 1.60
- 25+ IPs$ 1.50
- 50+ IPs$ 1.40
- 100+ IPs$ 1.30
- 200+ IPs$ 1.20
- 500+ IPs$ 1.10
- 1 IP$ 1.85
- 3+ IPs$ 1.70
- 10+ IPs$ 1.60
- 25+ IPs$ 1.50
- 50+ IPs$ 1.40
- 100+ IPs$ 1.30
- 200+ IPs$ 1.20
- 500+ IPs$ 1.10
Dalilin da ya sa lambar middleware ke da muhimmanci
Downloader middlewares suna aiki ne bisa tsari mai hauhawa a hanyar fita. HttpProxyMiddleware a 750 tana karanta meta['proxy'], tana cire bayanan shiga daga URL sannan ta rubuta kanun auth. Idan middleware ɗinka ta saita proxy a bayanta, a 800 misali, ginanniyar ta riga ta yi aiki, ba za a taɓa canza bayanan shiga ba, kuma proxy za ta ba da amsar 407. Yin rajistar taka a 740 yana sanya ta zama ta farko.
| Saitawa | Ƙimar da aka ba da shawara | Sakamako |
|---|---|---|
request.meta['proxy'] | 'http://user:pass@ip:port' | Yana aika wannan buƙatar ta hanyar proxy na ProxyV4 da aka bayar |
| Tsarin jeri na middleware ɗinka | 740 | Yana aiki kafin HttpProxyMiddleware (750), don haka ana canza bayanan shiga zuwa lamba |
CONCURRENT_REQUESTS_PER_IP | 2 zuwa 4 | Yana iyakance buƙatun lokaci ɗaya ga kowane IP na makoma, iyakar ladabi kan yadda ake tura buƙatu ga kowace uwar garke |
AUTOTHROTTLE_ENABLED | True | Yana ba Scrapy damar tsawaita ko taƙaita jinkiri dangane da yadda manufar ke ba da amsa cikin sauri |
Taƙaitaccen bayanin middleware na juyawa
- Karanta jerin proxy daga saituna a cikin
from_crawler()kuma a ajiye shi a sashin middleware - A cikin
process_request(), sanyarequest.meta['proxy'] = random.choice(self.proxies)sai dai idan buƙatar tana ɗauke da ɗaya tuni - Lokacin da amsa ta dawo
429ko503, sake gwadawa da wata proxy daban daga rukunin maimakon wadda aka ƙi yanzu - Haɗa
DOWNLOAD_DELAY = 1.0tare daRANDOMIZE_DOWNLOAD_DELAY = Truedomin tazarar tsakanin buƙatu ta bambanta maimakon bugawa kamar agogon metronome
Ƙayyade girman rukunin
Crawl na Scrapy na iya tura buƙatu masu yawa a minti ɗaya, don haka adadin IP ɗin yana ƙayyade yadda ake amfani da kowannensu a hankali. Tare da ProxyV4 babu mafi ƙarancin oda kuma farashin kowane IP yana raguwa yayin da yawa ke ƙaruwa, sannan ba a auna zirga-zirgar keɓaɓɓen IPv4 ba, don haka babban crawl yana da farashi ɗaya da ƙarami a kan bandwidth. Abokin ciniki ɗaya ne kawai ke amfani da kowane IP, don haka kimarsa tana nuna crawl ɗinka ne kawai ba na wani daban ba. Don tsara yawan crawl da ladabi, duba web scraping guide.
Mataki bayan mataki
- 1
Shirya adiresoshin URL na proxy
Canza layukan ProxyV4 ɗinka na
ip:port:user:passzuwahttp://user:pass@ip:porttare da proxy format converter kuma adana su a saituna ko fayil. - 2
Rubuta ProxyPoolMiddleware
A cikin
middlewares.py, aiwatar daprocess_request(self, request, spider)domin ya saitarequest.meta['proxy'] = random.choice(self.proxies). - 3
Yi rajistarta kafin ginanniyar
A cikin
settings.py, ƙara'myproject.middlewares.ProxyPoolMiddleware': 740zuwaDOWNLOADER_MIDDLEWARESkuma saitaCONCURRENT_REQUESTS_PER_IP = 2. - 4
Tabbatar da juyawa
Ɗauko
https://proxyv4.com/api/my-ipsau kaɗan tare dascrapy shellko ɗan gajeren spider na gwaji kuma duba cewa IP ɗin da aka bayar yana canzawa a cikin rukuninka.
Tambayoyi da aka fi yi
Proxy middleware ɗina a matsayin fifiko na 800 yana samun kuskuren 407. Me ya sa?
A 800 yana aiki ne bayan HttpProxyMiddleware (750), don haka ba a taɓa mayar da user:pass da ke cikin URL ɗin proxy ɗinka zuwa kanun Proxy-Authorization ba. Mayar da middleware ɗinka ƙasa da 750, misali 740.
Zan iya saita proxy don buƙata ɗaya kawai?
Ee. Bayar da meta={'proxy': 'http://user:pass@ip:port'} lokacin da kake ƙirƙirar Request; ginanniyar middleware tana kula da shigar don wannan buƙatar kawai.
Shin crawl na miliyoyin shafuka zai kai ga iyakar bandwidth?
A'a a kan keɓaɓɓen IPv4 ko tsayayyun proxies na ISP na ProxyV4; dukansu suna da bandwidth mara iyaka. Iyaka ta zahiri ita ce saurin da manufar ke ba ka damar yin crawl.
Tushe
Tsayayyun IP marasa iyaka don ayyukan crawl na Scrapy
Keɓaɓɓen IPv4 a cikin ƙasashe 7 haɗe da tsayayyen ISP a Amurka da Vietnam, daga IP guda ɗaya zuwa babban rukuni, tare da farashin rangwame na kai tsaye bisa yawa.
Duba farashin Amurka na kai tsaye