ProxyV4

Scrapy proxies: meta['proxy'], tsarin jeri na middleware da juyawa

· Updated:

A Scrapy, ana saita proxy ga kowace buƙata ta hanyar request.meta['proxy'] = 'http://user:pass@ip:port', kuma ginanniyar HttpProxyMiddleware (lambar tsari 750) tana mayar da sashin user:pass zuwa kanun Proxy-Authorization. Ƙara ƙaramar downloader middleware da ke aiki gabaninta, a 740, kuma kowace buƙata da spider ɗinka ta bayar za ta iya zaɓar IP daga rukunin ProxyV4 ɗinka.

Scrapy · HTTP
request.meta['proxy'] = 'http://user:pass@ip:port'
Private IPv4 (datacenter)
🇺🇸 United States
Akwai a ajiye
Daga
$ 1.10 / IP / wata
Rangwamen yawan oda
  • 1 IP$ 1.85
  • 3+ IPs$ 1.70
  • 10+ IPs$ 1.60
  • 25+ IPs$ 1.50
  • 50+ IPs$ 1.40
  • 100+ IPs$ 1.30
  • 200+ IPs$ 1.20
  • 500+ IPs$ 1.10
Sayi United States
Private IPv4 (datacenter)
🇬🇧 United Kingdom
Akwai a ajiye
Daga
$ 1.10 / IP / wata
Rangwamen yawan oda
  • 1 IP$ 1.85
  • 3+ IPs$ 1.70
  • 10+ IPs$ 1.60
  • 25+ IPs$ 1.50
  • 50+ IPs$ 1.40
  • 100+ IPs$ 1.30
  • 200+ IPs$ 1.20
  • 500+ IPs$ 1.10
Sayi United Kingdom

Dalilin da ya sa lambar middleware ke da muhimmanci

Downloader middlewares suna aiki ne bisa tsari mai hauhawa a hanyar fita. HttpProxyMiddleware a 750 tana karanta meta['proxy'], tana cire bayanan shiga daga URL sannan ta rubuta kanun auth. Idan middleware ɗinka ta saita proxy a bayanta, a 800 misali, ginanniyar ta riga ta yi aiki, ba za a taɓa canza bayanan shiga ba, kuma proxy za ta ba da amsar 407. Yin rajistar taka a 740 yana sanya ta zama ta farko.

SaitawaƘimar da aka ba da shawaraSakamako
request.meta['proxy']'http://user:pass@ip:port'Yana aika wannan buƙatar ta hanyar proxy na ProxyV4 da aka bayar
Tsarin jeri na middleware ɗinka740Yana aiki kafin HttpProxyMiddleware (750), don haka ana canza bayanan shiga zuwa lamba
CONCURRENT_REQUESTS_PER_IP2 zuwa 4Yana iyakance buƙatun lokaci ɗaya ga kowane IP na makoma, iyakar ladabi kan yadda ake tura buƙatu ga kowace uwar garke
AUTOTHROTTLE_ENABLEDTrueYana ba Scrapy damar tsawaita ko taƙaita jinkiri dangane da yadda manufar ke ba da amsa cikin sauri

Taƙaitaccen bayanin middleware na juyawa

  • Karanta jerin proxy daga saituna a cikin from_crawler() kuma a ajiye shi a sashin middleware
  • A cikin process_request(), sanya request.meta['proxy'] = random.choice(self.proxies) sai dai idan buƙatar tana ɗauke da ɗaya tuni
  • Lokacin da amsa ta dawo 429 ko 503, sake gwadawa da wata proxy daban daga rukunin maimakon wadda aka ƙi yanzu
  • Haɗa DOWNLOAD_DELAY = 1.0 tare da RANDOMIZE_DOWNLOAD_DELAY = True domin tazarar tsakanin buƙatu ta bambanta maimakon bugawa kamar agogon metronome

Ƙayyade girman rukunin

Crawl na Scrapy na iya tura buƙatu masu yawa a minti ɗaya, don haka adadin IP ɗin yana ƙayyade yadda ake amfani da kowannensu a hankali. Tare da ProxyV4 babu mafi ƙarancin oda kuma farashin kowane IP yana raguwa yayin da yawa ke ƙaruwa, sannan ba a auna zirga-zirgar keɓaɓɓen IPv4 ba, don haka babban crawl yana da farashi ɗaya da ƙarami a kan bandwidth. Abokin ciniki ɗaya ne kawai ke amfani da kowane IP, don haka kimarsa tana nuna crawl ɗinka ne kawai ba na wani daban ba. Don tsara yawan crawl da ladabi, duba web scraping guide.

Mataki bayan mataki

  1. 1

    Shirya adiresoshin URL na proxy

    Canza layukan ProxyV4 ɗinka na ip:port:user:pass zuwa http://user:pass@ip:port tare da proxy format converter kuma adana su a saituna ko fayil.

  2. 2

    Rubuta ProxyPoolMiddleware

    A cikin middlewares.py, aiwatar da process_request(self, request, spider) domin ya saita request.meta['proxy'] = random.choice(self.proxies).

  3. 3

    Yi rajistarta kafin ginanniyar

    A cikin settings.py, ƙara 'myproject.middlewares.ProxyPoolMiddleware': 740 zuwa DOWNLOADER_MIDDLEWARES kuma saita CONCURRENT_REQUESTS_PER_IP = 2.

  4. 4

    Tabbatar da juyawa

    Ɗauko https://proxyv4.com/api/my-ip sau kaɗan tare da scrapy shell ko ɗan gajeren spider na gwaji kuma duba cewa IP ɗin da aka bayar yana canzawa a cikin rukuninka.

Tambayoyi da aka fi yi

Proxy middleware ɗina a matsayin fifiko na 800 yana samun kuskuren 407. Me ya sa?

A 800 yana aiki ne bayan HttpProxyMiddleware (750), don haka ba a taɓa mayar da user:pass da ke cikin URL ɗin proxy ɗinka zuwa kanun Proxy-Authorization ba. Mayar da middleware ɗinka ƙasa da 750, misali 740.

Zan iya saita proxy don buƙata ɗaya kawai?

Ee. Bayar da meta={'proxy': 'http://user:pass@ip:port'} lokacin da kake ƙirƙirar Request; ginanniyar middleware tana kula da shigar don wannan buƙatar kawai.

Shin crawl na miliyoyin shafuka zai kai ga iyakar bandwidth?

A'a a kan keɓaɓɓen IPv4 ko tsayayyun proxies na ISP na ProxyV4; dukansu suna da bandwidth mara iyaka. Iyaka ta zahiri ita ce saurin da manufar ke ba ka damar yin crawl.

Tushe

Tsayayyun IP marasa iyaka don ayyukan crawl na Scrapy

Keɓaɓɓen IPv4 a cikin ƙasashe 7 haɗe da tsayayyen ISP a Amurka da Vietnam, daga IP guda ɗaya zuwa babban rukuni, tare da farashin rangwame na kai tsaye bisa yawa.

Duba farashin Amurka na kai tsaye

Related articles

Furoksi ta manhaja
Proxies don web scraping: kafaffen rukunin IP tare da zirga-zirgar bayanai marar iyaka
Proxies na web scraping don Python, Scrapy, Playwright da Selenium: rukunin keɓaɓɓen IPv4 da static ISP, bandwidth marar iyaka, kafaffen farashin kowace IP.
Saitin furoksi
Bukatar Python ta hanyar ingantaccen proxy, mataki-mataki
Aika bukatar Python ta hanyar ingantaccen proxy na HTTP(S): gina proxies dict, haɗa shi da Session, sauya kalmomin sirri, juya lissafi, sannan duba IP.
Saitin furoksi
Tabbatar da Playwright proxy, tsayayyen IP guda daya ga kowane context
Ba kowane Playwright browser context nasa proxy na HTTP(S) mai tabbatarwa a Node.js ko Python, daidaita lokaci da yare da IP, sannan ka duba adireshin fita.
Kayan aiki na kyauta
Sake rubuta jerin proxy zuwa tsarin da kayan aikinku ke buƙata
Sauya jerin proxy a burawzarku tsakanin ip:port:user:pass, user:pass@ip:port, http:// URLs, SOCKS5 URIs, arrays na JSON da fayilolin FoxyProxy a dunkule.
Furoksi ta manhaja
Wakilai don bin diddigin matsayin SEO: ga shafin sakamako kamar yadda mai bincike na asali yake gani
Wakilan SEO masu zaman kansu na IPv4 a Amurka, Birtaniya da sauran kasuwanni, domin matsayi da tallace-tallace su zama daidai da abin da 'yan gari ke gani.
Furoksi ta manhaja
Proxy na dillalan Amazon masu ba kowane asusu adireshin dindindin
Proxy na Amazon don Seller Central, KDP da Merch: kafaffen US ISP ko keɓaɓɓen UK/Japan IPv4 ga kowane asusu, na ku kaɗai kuma yana nan yayin sabuntawa.
IPV4 · united states
Rukunin Keɓaɓɓun IPv4 Proxy na Amurka domin Scraping, Bibiyar Matsayi da Duba Talla
Sayi US private IPv4 proxy: tsayayyun IP na datacenter na US, mutum ɗaya kowace IP, bandwidth marar iyaka. Don scraping, bin SERP da duba tallan Amurka.
IPV4 · united kingdom
Keɓaɓɓen IPv4 Proxy na Birtaniya domin Kasuwannin Yanar Gizo, Bincike da Kamfen
UK private IPv4 proxy mai tsayayyen IP na datacenter na Birtaniya ga mutum daya, don binciken eBay UK da Amazon.co.uk, bibiyar google.co.uk da tallan UK.