Proxies don web scraping: kafaffen rukunin IP tare da zirga-zirgar bayanai marar iyaka
Updated:
Proxies na web scraping da ake caji a kowace gigabyte suna sa crawler ƙara tsada a duk lokacin da shafi ya yi nauyi, shi ya sa crawlers masu aiki a kai a kai kuma a kayyade suka fi arha a kan kafaffen rukunin adiresoshi masu zirga-zirgar bayanai marar iyaka. ProxyV4 yana sayar da keɓaɓɓun IPv4 a ƙasashe bakwai da kuma static ISP a Amurka da Vietnam, kowanne da farashin kowace IP ba tare da auna zirga-zirgar bayanai ba. Kana daidaita girman rukunin gwargwadon yawan buƙatun da shafukan da kake bi za su iya ɗauka a sakan ɗaya, kuma lissafin kuɗin yana nan daram ko da nauyin shafukan ya ƙaru.
- 1 IP$ 1.85
- 3+ IPs$ 1.70
- 10+ IPs$ 1.60
- 25+ IPs$ 1.50
- 50+ IPs$ 1.40
- 100+ IPs$ 1.30
- 200+ IPs$ 1.20
- 500+ IPs$ 1.10
- 1 IP$ 1.85
- 3+ IPs$ 1.70
- 10+ IPs$ 1.60
- 25+ IPs$ 1.50
- 50+ IPs$ 1.40
- 100+ IPs$ 1.30
- 200+ IPs$ 1.20
- 500+ IPs$ 1.10
Farashin kowace IP idan aka kwatanta da farashin kowace GB ga crawler
| Tambaya | Kafaffen rukuni (kowace IP) | Zirga-zirgar bayanai da ake aunawa (kowace GB) |
|---|---|---|
| Abin da ke kawo tsada | Yawan IPs da tsawon lokacin haya | Bytes da aka saukar, har da HTML, JS da sauran fayiloli |
| Abin da ke faruwa idan shafuka suka yi nauyi | Babu komai; bandwidth marar iyaka ne | Kuɗin yana ƙaruwa tare da su |
| Abin da ke iyakance saurin aiki | Ƙa'idar iyakar buƙata a kowace IP ta kowane shafi | Kasafin kuɗi |
| Yadda za a faɗaɗa | Ƙara IPs; farashin kowace IP yana sauka yayin da yawa ya ƙaru | Sayi ƙarin zirga-zirgar bayanai |
Dalilin da ya sa ake toshe scrapers, da kuma abin da rukuni ke sauyawa
A wurin tattara bayanan kasida, jeri da kuma public API, yawancin toshewa ba don wanene kai ba ne, sai dai yadda kake aika buƙatu da sauri: adireshi ɗaya da ya wuce ƙa'idar yawan buƙatun shafi yana samun HTTP 429. Rukunin static IPs 10 zuwa 100, kowanne a ƙarƙashin adadin aiki mai sauƙi, yana kiyaye kowane adireshi a ƙarƙashin iyakar yayin da crawler gaba ɗaya ke gudana a saurin da kake buƙata.
Keɓancewa ga mutum ɗaya na taimakawa a nan ma. Kowane IP na ProxyV4 na abokin ciniki ɗaya ne a tsawon lokacin hayan, saboda haka tarihin buƙatun da shafi ke gani a kan wannan adireshin naka ne kawai.
Daidaita rukunin da shafin da ake nema
- Kundayen adireshi na jama'a da kuma APIs masu yawan aiki: keɓaɓɓen IPv4 a Amurka ko Singapore, mafi ƙarancin farashi a kowace IP
- Kasuwancin yanar gizo da manyan shafukan Vietnam: Vietnam IPv4 domin samun shafukan cikin gida da farashin gida
- Shafukan da ke tace hosting ASNs a bayan WAF: static ISP a Amurka ko Vietnam, masu rajista a ISPs na masu amfani na yau da kullum, wanda yawanci ke nufin ƙarancin ƙalubalen Cloudflare
- Shafukan da ke buƙatar sabon IP a kowace buƙata: ba a fara sayar da rotating proxies na Vietnam ba tukuna; shafin rotating na Vietnam yana ɗauke da jerin jiran samun dama
Saitunan crawler da ke sa kafaffen rukuni ya daɗe yana aiki
Kowane layin haɗi yana amfani da bayanan shiga na yau da kullum, saboda haka a Python proxy ɗin yana kasancewa kawai proxies = {"http": "http://user:pass@ip:port", "https": "http://user:pass@ip:port"}.
- Rarraba buƙatu ta hanyar round-robin a cikin jerin sannan ka ƙara ɗan jinkiri na bazata, kimanin sakan 2 zuwa 5 a kowace IP ga kowane domain
- Sake amfani da hanyoyin haɗin yanar gizo tare da
requests.Session()ko client mai amfani da rukuni domin kada ka riƙa biyan kuɗin musayar TLS a kowace buƙata - A cikin Playwright da Puppeteer, toshe hotuna, rubutu da sauran fayilolin da ba ka buƙatar tattara bayansu
- Lokacin da IP ya samu HTTP 429 ko 503, cire shi daga rarrabawa domin ya huta sannan ka ɗan dakata maimakon sake gwadawa nan take
Mataki bayan mataki
- 1
- 2
Mayar da jerin zuwa URLs
Manna layukan
ip:port:user:passa cikin na'urar sauya fasalin proxy domin samun rubutunhttp://user:pass@ip:portko JSON. - 3
Haɗa rukunin da tsarin aikinka
Yi amfani da shafukan saitawa na Python Requests, Scrapy, Selenium ko Node.js Axios.
- 4
Auna sauri kafin cikakken aiki na farko
Duba lokacin amsawa tare da gwajin saurin proxy sannan ka cire duk layin da ya yi jinkiri ga shafin da kake son tattarawa.
Tambayoyi da aka fi yi
Shin ana auna zirga-zirgar bayanan scraping a kan proxies na ProxyV4?
A'a. Keɓaɓɓun IPv4 da static ISP ana sanya musu farashi a kowace IP na tsawon lokacin haya tare da bandwidth marar iyaka.
Shafuka nawa a rana static IP guda ɗaya zai iya ɗauka?
Ya danganta da ƙa'idar iyakar buƙatu ta shafin. A buƙatu 1 zuwa 3 a minti ɗaya ga kowane domain, IP ɗaya yana ɗauko kimanin shafuka 1,500 zuwa 4,000 a rana ba tare da ya jawo hankali ba.
Kuna sayar da rotating proxies don scraping?
Ba a yanzu ba. Rotating proxies na Vietnam a jerin jiran samun dama kawai suke. A yanzu haka, kafaffen rukuni mai rarraba buƙatu ta round-robin yana yin aikin ga yawancin ayyukan tattara bayanai da aka tsara.
Ta yaya zan canza proxies 50 zuwa fasalin Scrapy da sauri?
Manna su a cikin na'urar sauya fasalin proxy. Yana mayar da layukan ip:port:user:pass zuwa rubutun URL ko jerin JSON a cikin browser.
Gina rukunin scraping tare da kafaffen kuɗin wata-wata
Keɓaɓɓen IPv4 a ƙasashe bakwai da static ISP a Amurka da Vietnam, a kan farashin kowace IP tare da zirga-zirgar bayanai marar iyaka.
Duba farashin IPv4 na Amurka na yanzu