ProxyV4

Ama-proxy e-Scrapy: meta['proxy'], ukulandelana kwe-middleware nokushintshanisa

· Updated:

Ku-Scrapy, i-proxy isethwa esicelweni ngasinye kusetshenziswa request.meta['proxy'] = 'http://user:pass@ip:port', kanti i-HttpProxyMiddleware eyakhelwe ngaphakathi (ukulandelana 750) iguqula ingxenye ye-user:pass ibe yisihloko se-Proxy-Authorization. Engeza i-downloader middleware encane esebenza ngaphambi kwayo, ku-740, ukuze sonke isicelo i-spider yakho esikhiphayo sikwazi ukukhetha i-IP eqoqweni lakho le-ProxyV4.

Scrapy · HTTP
request.meta['proxy'] = 'http://user:pass@ip:port'
I-IPv4 eyimfihlo (datacenter)
🇺🇸 United States
Kukhona esitokweni
Kusukela
$1.10 / IP / ngenyanga
Izaphulelo zenani
  • 1 IP$1.85
  • 3+ IPs$1.70
  • 10+ IPs$1.60
  • 25+ IPs$1.50
  • 50+ IPs$1.40
  • 100+ IPs$1.30
  • 200+ IPs$1.20
  • 500+ IPs$1.10
Thenga e-United States
I-IPv4 eyimfihlo (datacenter)
🇬🇧 United Kingdom
Kukhona esitokweni
Kusukela
$1.10 / IP / ngenyanga
Izaphulelo zenani
  • 1 IP$1.85
  • 3+ IPs$1.70
  • 10+ IPs$1.60
  • 25+ IPs$1.50
  • 50+ IPs$1.40
  • 100+ IPs$1.30
  • 200+ IPs$1.20
  • 500+ IPs$1.10
Thenga e-United Kingdom

Kungani inombolo ye-middleware ibalulekile

Ama-downloader middleware asebenza ngokulandelana okwenyukayo lapho kuphunywa. I-HttpProxyMiddleware ku-750 ifunda i-meta['proxy'], isuse iziqinisekiso kwi-URL bese ibhala isihloko sokufakazela ubuqiniso. Uma i-middleware yakho isetha i-proxy ngemva kwalokho, ngokwesibonelo ku-800, eyakhelwe ngaphakathi isuke isisebenzile kakade, iziqinisekiso aziguqulwa neze, bese i-proxy iphendula ngo-407. Ukubhalisa eyakho ku-740 kuyibeka phambili.

IsilungiseleloInani eliphakanyisiweUmphumela
request.meta['proxy']'http://user:pass@ip:port'Ithumela leso sicelo nge-proxy enikeziwe ye-ProxyV4
Ukulandelana kwe-middleware yakho740Isebenza ngaphambi kwe-HttpProxyMiddleware (750), ukuze imininingwane yokungena ibhalwe ngekhodi
CONCURRENT_REQUESTS_PER_IP2 kuya ku-4Inquma izicelo ezihambisana ngesikhathi esisodwa nge-IP ngayinye okuyiwa kuyo, okuwumkhawulo omuhle wendlela iseva ngayinye ecelwa ngayo
AUTOTHROTTLE_ENABLEDTrueIvumela i-Scrapy ukuba yelule noma ifinyeze ukulibala kuye ngokuthi isizinda siphendula ngokushesha okungakanani

Uhlaka lwe-middleware yokushintshanisa

  • Funda uhlu lwama-proxy kuzilungiselelo ku-from_crawler() bese uligcina kusibonelo se-middleware
  • Ku-process_request(), nika request.meta['proxy'] = random.choice(self.proxies) ngaphandle uma isicelo sesivele sinayo
  • Uma impendulo ibuyela ingasho u-429 noma u-503, yiphinde usebenzise i-proxy ehlukile eqoqweni esikhundleni sale esanda kwaliwa
  • Hlanganisa i-DOWNLOAD_DELAY = 1.0 ne-RANDOMIZE_DOWNLOAD_DELAY = True ukuze isikhathi phakathi kwezicelo sihluke kunokuba sishaye njalo njengewashi

Ukukala ubukhulu beqoqo

Ukucwaninga kwe-Scrapy kungathumela izicelo eziningi ngomzuzu, ngakho inani lama-IP linquma indlela ngayinye esetshenziswa ngayo ngobumnene. Nge-ProxyV4 akukho ukuthenga okuncane okuyimpoqo futhi intengo nge-IP ngayinye yehla njengoba inani lenyuka, kanti ithrafikhi ye-IPv4 yangasese ayikalwa, ngakho ukucwaninga okukhulu kubiza ngokufanayo nokuncane kwi-bandwidth. I-IP ngayinye isetshenziswa ikhasimende elilodwa kuphela, ngakho isithunzi sayo sibonisa ukucwaninga kwakho kuphela hhayi okomunye umuntu. Ukuhlela umthamo wokucwaninga nenhlonipho, bheka i-web scraping guide.

Isinyathelo ngesinyathelo

  1. 1

    Lungisa ama-URL e-proxy

    Guqula imigqa yakho ye-ProxyV4 ethi ip:port:user:pass ibe yi-http://user:pass@ip:port nge-proxy format converter bese uyigcina kuzilungiselelo noma efayeleni.

  2. 2

    Bhala i-ProxyPoolMiddleware

    Ku-middlewares.py, faka ikhodi ye-process_request(self, request, spider) ukuze isethe request.meta['proxy'] = random.choice(self.proxies).

  3. 3

    Yibhalise ngaphambi kweyakhelwe ngaphakathi

    Ku-settings.py, engeza 'myproject.middlewares.ProxyPoolMiddleware': 740 ku-DOWNLOADER_MIDDLEWARES bese usetha CONCURRENT_REQUESTS_PER_IP = 2.

  4. 4

    Qinisekisa ukushintshanisa

    Landa i-https://proxyv4.com/api/my-ip izikhathi ezimbalwa nge-scrapy shell noma nge-spider emfushane yokuhlola bese uhlola ukuthi i-IP eboniswayo iyashintsha yini eqoqweni lakho.

Imibuzo evame ukubuzwa

I-proxy middleware yami enokubaluleka okungu-800 ithola amaphutha angu-407. Kungani?

Ku-800 isebenza ngemva kwe-HttpProxyMiddleware (750), ngakho i-user:pass kwi-URL yakho ye-proxy ayiguqulwa neze ibe yisihloko se-Proxy-Authorization. Hambisa i-middleware yakho ngaphansi kuka-750, isib. 740.

Ngingakwazi ukusetha i-proxy esicelweni esisodwa nje kuphela?

Yebo. Dlulisela i-meta={'proxy': 'http://user:pass@ip:port'} lapho wenza i-Request; i-middleware eyakhelwe ngaphakathi iphatha ukungena kwaleso sicelo kuphela.

Ingabe ukucwaninga izigidi zamakhasi kuzofinyelela emkhawulweni we-bandwidth?

Hhayi kuma-proxy e-IPv4 angasese noma ama-static ISP e-ProxyV4; womabili ane-bandwidth engenamkhawulo. Umkhawulo wangempela yindlela isizinda esikuvumela ngayo ukucwaninga ngokushesha.

Imithombo

Ama-IP amile angenamkhawulo wokucwaninga kwe-Scrapy

I-IPv4 yangasese emazweni angu-7 kanye ne-ISP emile e-US nase-Vietnam, kusukela ku-IP eyodwa kuya eqoqweni elikhulu, enamanani abukhoma ngokwenani.

Bheka izintengo ezibukhoma zase-US

Related articles

Ama-proxy ngezinkundla
Ama-proxy okukrwela iwebhu: iqoqo elimile lama-IP anethrafikhi engalinganiselwa
Ama-proxy okukrwela iwebhu e-Python, Scrapy, Playwright ne-Selenium: iqoqo lama-IPv4 ayimfihlo nama-ISP amile, i-bandwidth engapheli, namanani anqunyiwe nge-IP.
Ukusetha i-proxy
Izicelo ze-Python nge-proxy eqinisekisiwe, isinyathelo ngesinyathelo
Thumela izicelo ze-Python nge-proxy ye-HTTP(S) enokungena: yakha i-dict, yinamathisele ku-Session, shintsha amaphasiwedi, jikelezisa uhlu, hlola i-IP.
Ukusetha i-proxy
Ukuqinisekiswa kwe-proxy ye-Playwright, i-IP eyodwa emile kumongo ngamunye
Nikeza umongo we-Playwright ngamunye i-proxy ye-HTTP(S) eqinisekisiwe ku-Node.js noma Python, fanisa isikhathi nolimi ne-IP bese uqinisekisa ikheli lokuphuma.
Amathuluzi amahhala
Bhala kabusha uhlu lwe-proxy lube yifomethi elindelwe ithuluzi lakho
Guqula izinhlu ze-proxy kusiphequluli phakathi kwe-ip:port:user:pass, user:pass@ip:port, ama-URL e-http://, ama-URI e-SOCKS5, i-JSON ne-FoxyProxy, ngobuningi.
Ama-proxy ngezinkundla
Ama-proxy wokulandelela izinga le-SEO: bona ikhasi lemiphumela elibonwa umuseshi wendawo
Ama-proxy e-SEO ane-IPv4 eyimfihlo e-US, UK, Vietnam, Japan nezinye izimakethe ezi-4 ukuze izinga, ama-local pack nezikhangiso kufane nalokho okubonwa abendawo.
Ama-proxy ngezinkundla
Ama-proxy oMthengisi we-Amazon anikeza i-akhawunti ngayinye ikheli elihlala njalo
Ama-proxy e-Amazon e-Seller Central, KDP ne-Merch: i-static ISP yase-US noma i-IPv4 yase-UK/Japan nge-akhawunti, eqashelwa wena wedwa negcinwa uma uvuselela.
IPV4 · united states
Iqoqo Lama-Proxy E-IPv4 Yangasese Yase-US Lokukhipha Idatha, Ukulandelela Amazinga Nokuhlola Izikhangiso
Thenga ama-proxy e-IPv4 yangasese ase-US: ama-IP e-datacenter, ikhasimende lilinye nge-IP, i-bandwidth engapheli. Yenzelwe uku-scrape, i-SERP nezikhangiso.
IPV4 · united kingdom
I-Proxy Ye-IPv4 Yangasese Yase-UK Yezindawo Zokuhweba, Ukusesha Nemikhankaso
I-proxy ye-IPv4 yase-UK ene-IP yase-Britain ezinzile yomsebenzisi oyedwa, yocwaningo lwe-eBay UK ne-Amazon.co.uk, i-google.co.uk nokuhlola izikhangiso zase-UK.