ProxyV4

Proxy-yada Scrapy: meta['proxy'], kala-horreynta middleware iyo wareejinta

· Updated:

Gudaha Scrapy, proxy waxaa loo dejiyaa codsi kasta iyadoo la adeegsanayo request.meta['proxy'] = 'http://user:pass@ip:port', waxaana HttpProxyMiddleware-ka dhex-dhisan (kala-horreyntiisu tahay 750) uu qeybta user:pass u beddelaa madax-madaxeedka Proxy-Authorization. Ku dar downloader middleware yar oo ka hor shaqeeya, oo ah 740, codsi kasta oo uu spider-kaagu soo saarana wuxuu IP ka dooran karaa kaydkaaga ProxyV4.

Scrapy · HTTP
request.meta['proxy'] = 'http://user:pass@ip:port'
IPv4 gaar ah (datacenter)
🇺🇸 Mareykanka
Waa la heli karaa
Laga bilaabo
US$1.10 / IP / bishii
Qiimo dhimista tirada
  • 1 IPUS$1.85
  • 3+ IPUS$1.70
  • 10+ IPUS$1.60
  • 25+ IPUS$1.50
  • 50+ IPUS$1.40
  • 100+ IPUS$1.30
  • 200+ IPUS$1.20
  • 500+ IPUS$1.10
Iibso Mareykanka
IPv4 gaar ah (datacenter)
🇬🇧 Boqortooyada Midowday
Waa la heli karaa
Laga bilaabo
US$1.10 / IP / bishii
Qiimo dhimista tirada
  • 1 IPUS$1.85
  • 3+ IPUS$1.70
  • 10+ IPUS$1.60
  • 25+ IPUS$1.50
  • 50+ IPUS$1.40
  • 100+ IPUS$1.30
  • 200+ IPUS$1.20
  • 500+ IPUS$1.10
Iibso Boqortooyada Midowday

Sababta lambarka middleware-ku muhiim u yahay

Downloader middleware-yadu waxay u socdaan hab korodhsiimo ah marka ay baxayaan. HttpProxyMiddleware-ka ku jira 750 wuxuu akhriyaa meta['proxy'], wuxuu URL-ka ka saaraa xogta aqoonsiga wuxuuna qoraa madax-madaxeedka xaqiijinta. Haddii middleware-kaaga gaarka ahi uu proxy-ga dejiyo dabadeed, tusaale ahaan 800, kii dhex-dhisaa mar hore ayuu shaqeeyay, xogta aqoonsigana weligeed lama beddelo, proxy-guna wuxuu ku jawaabayaa 407. Diiwaangelinta kaaga lambarka 740 waxay ka dhigaysaa midka ugu horreeya.

DejintaQiimaha la soo jeediyaySaamaynta
request.meta['proxy']'http://user:pass@ip:port'Codsigaas wuxuu ku dirayaa proxy-ga la bixiyay ee ProxyV4
Kala-horreynta middleware-kaaga740Wuxuu shaqeeyaa ka hor HttpProxyMiddleware (750), si gelitaanka loo koodheeyo
CONCURRENT_REQUESTS_PER_IP2 to 4Wuxuu xaddidayaa codsiyada isku mar ah halkii IP ee loo socdo, xad edeb leh oo ku saabsan sida loo culeysiyo server kasta
AUTOTHROTTLE_ENABLEDTrueWuxuu u oggolaanayaa Scrapy inuu kordhiyo ama soo gaabiyo dib-u-dhacyada iyadoo lagu saleynayo sida degdegga ah ee bartilmaameedku uga jawaabo

Dulmar guud oo ku saabsan middleware-ka wareejinta

  • Ka akhri liiska proxy-yada goobaha dejinta gudaha from_crawler() kuna hay tusaalaha middleware-ka
  • Gudaha process_request(), u qoondee request.meta['proxy'] = random.choice(self.proxies) haddii uusan codsigu mar hore mid sidanin
  • Markii jawaabtu ku soo noqoto 429 ama 503, dib ugu day proxy kale oo kaydka ka tirsan halkii aad ka isticmaali lahayd kii hadda la diiday
  • Isku dar DOWNLOAD_DELAY = 1.0 iyo RANDOMIZE_DOWNLOAD_DELAY = True si farqiga u dhexeeya codsiyadu u kala duwanaado halkii uu u garaaci lahaa sida saacad

Qiyaasidda xaddiga kaydka

Xog-ururinta Scrapy waxay daqiiqaddii diri kartaa codsiyo badan, sidaas darteed tirada IP-yada ayaa go'aamisa sida taxaddarka leh ee mid kasta loo isticmaalo. ProxyV4 ma lahan dalab ugu yar, qiimaha halkii IP-na wuu hoos u dhacaa markii tiradu korodho, taraafikada gaarka ah ee IPv4-na lama qiyaaso, markaa xog-ururin weyn iyo mid yar waxay ku kacayaan isku qiimo dhanka xaddiga xogta. IP kasta waxaa isticmaala hal macaamiil oo keliya, markaa sumcaddiisu waxay ka tarjumaysaa xog-ururintaada gaarka ah mana jirto cid kale. Si aad u qorshayso mugga xog-ururinta iyo hab-dhaqanka edebta leh, eeg hagaha web scraping-ka.

Tallaabo tallaabo

  1. 1

    Diyaari URL-yada proxy-ga

    U beddel khadadkaaga ProxyV4 ee ip:port:user:pass una beddel http://user:pass@ip:port adigoo isticmaalaya beddelaha qaabka proxy-ga oo ku keydi dejimaha ama fayl.

  2. 2

    Qor ProxyPoolMiddleware

    Gudaha middlewares.py, ka hirgeli process_request(self, request, spider) si uu u dejiyo request.meta['proxy'] = random.choice(self.proxies).

  3. 3

    Diiwaangeli ka hor kan dhex-dhisan

    Gudaha settings.py, ku dar 'myproject.middlewares.ProxyPoolMiddleware': 740 qaybta DOWNLOADER_MIDDLEWARES oo deji CONCURRENT_REQUESTS_PER_IP = 2.

  4. 4

    Xaqiiji wareejinta

    Dhawr jeer soo doono https://proxyv4.com/api/my-ip adigoo adeegsanaya scrapy shell ama spider tijaabo ah oo gaaban, oo hubi in IP-ga la soo sheegay uu isbeddelayo guud ahaan kaydkaaga.

Su'aalaha badanaa la isweydiiyo

Proxy middleware-kayga mudnaantiisu tahay 800 wuxuu helayaa khaladaadka 407. Maxaa sababay?

Markay tahay 800 wuxuu shaqeeyaa ka dib HttpProxyMiddleware (750), markaa user:pass ee ku jira URL-kaaga proxy weligood looma beddelo madax-madaxeedka Proxy-Authorization. U rar middleware-kaaga meel ka hoosaysa 750, tusaale 740.

Ma u dejin karaa proxy-ga hal codsi oo keliya?

Haa. Gudbi meta={'proxy': 'http://user:pass@ip:port'} marka aad abuureyso Request-ka; middleware-ka dhex-dhisan ayaa maareynaya gelitaanka codsigaas oo keliya.

Xog-ururinta malaayiin bog ma ku dhici doontaa xaddidaadda xogta?

Kuma dhacayso ProxyV4 IPv4 gaar ah ama proxy-yada ISP ee ma guurtada ah; labaduba waxay leeyihiin bandwidth xadla'aan ah. Xadka dhabta ahi waa sida degdegga ah ee bartilmaameedku kuu oggolaanayo inaad xogta u ururisid.

Ilaha

IP-yo ma guurto ah oo aan la xaddidin oo loogu talagalay Scrapy

IPv4 gaar ah oo ku yaal 7 waddan iyo weliba ISP ma guurto ah gudaha US iyo Vietnam, laga bilaabo hal IP ilaa kayd weyn, oo leh qiimaha tooska ah ee xaddiga.

Hubi qiimaha tooska ah ee US

Related articles

Proxy-yada madallada
Wakiillada web scraping: koox go'an oo IP-yo ah oo leh taraafik aan xadidnayn
Wakiillada web scraping ee Python, Scrapy, Playwright iyo Selenium: koox IPv4 gaar ah iyo static ISP, xog xadla'aan ah, qiimo go'an oo halkii IP ah.
Habaynta proxy-ga
Codsiyada Python ee soo mara wakiil la xaqiijiyay, tallaabo-tallaabo
Ku dir codsiyada Python wakiil HTTP(S) la xaqiijiyay: dhis dict-ka proxies, ku xidh Session, furayaasha qariibka ah codee, liiska wareeji, hubina IP-ga.
Habaynta proxy-ga
Xaqiijinta proxy-ga Playwright, hal IP ma guurto ah context kasta
Sii Playwright context kasta proxy HTTP(S) oo la xaqiijiyay gudaha Node.js ama Python, waafaji saacadda iyo luuqadda IP-ga, xaqiijina cinwaanka bixitaanka.
Aalado bilaash ah
U dib-u-qor liiska proxy qaabka uu qalabkaagu doonayo
Ku beddel liisaska proxy biraawsarkaaga: ip:port:user:pass, user:pass@ip:port, http:// URL, SOCKS5 URI, JSON iyo faylasha soo dejinta FoxyProxy si wadar ah.
Proxy-yada madallada
Wakiillada raadraaca darajada SEO: arag bogga natiijooyinka uu arko raadiyaha maxalliga ahi
Wakiillada raadraaca darajada SEO ee leh IPv4 gaar ah gudaha US, UK, Vietnam, Japan iyo afar suuq oo kale, si natiijooyinka u arkaan dadka deegaanka.
Proxy-yada madallada
Proxy-yada Amazon seller ee koonto kasta siiya cinwaan ma guurto ah
Proxy-yada Amazon ee Seller Central, KDP iyo Merch: ISP US ma guurto ah ama IPv4 UK/Japan koonto kasta, adiga kaliya laguu kireeyay lana hayo cusboonaysiinta.
IPV4 · united states
Barkadda Proxy-yada Gaarka ah ee IPv4 ee US ee Ururinta Xogta, La Socodka Heerarka iyo Hubinta Xayeysiisyada
Iibso proxy IPv4 gaar ah ee US: IP-yo datacenter Mareykan, hal qof per IP, bandwidth aan xad lahayn. Loogu talagalay scraping, SERP iyo hubinta xayaysiinta US.
IPV4 · united kingdom
Proxy-ga Gaarka ah ee IPv4 ee UK ee Suuqyada Ganacsiga, Raadinta iyo Ololayaasha
UK private IPv4 proxy leh IP datacenter Ingiriis joogto ah oo hal qof ah, cilmi-baarista eBay UK, Amazon.co.uk, raadraaca google.co.uk iyo xayeysiinta UK.