How to use the ip proxy pool in python crawler

This article is about how the ip proxy pool is used in python crawlers. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

Description

1. Crawl all proxy IP using for loop. Looking at the source code of the web page, it is found that all the IP and ports are under the second and third td, combined with BeautifulSoup can easily grab information.

2. Check the availability of IP. Whether IP is available depends on whether the page can be opened within 2 seconds after connecting to the agent.

If you can, think that IP is available and add it to the list as a backup.

Example

Import urllib2from bs4 import BeautifulSoupimport csv def IPspider (numpage): csvfile = file ('ips.csv',' wb') writer = csv.writer (csvfile) url=' http://www.xicidaili.com/nn/' user_agent='IP' headers= {'User-agent':user_agent} for num in xrange (1): ipurl=url+str (num) print' Now downloading the'+ str (num*100) + 'ips' request=urllib2.Request (ipurl) Headers=headers) content=urllib2.urlopen (request). Read () bs=BeautifulSoup (content) 'html.parser') res=bs.find_all (' tr') for item in res: try: temp= [] tds=item.find_all ('td') temp.append (tds [1] .text.encode (' utf-8')) temp.append (tds [2] .text.encode ('utf-8') ) writer.writerow (temp) except IndexError: pass # suppose crawling all the IP and port IPspider of the first ten pages (10) Thank you for reading! This is the end of this article on "how to use the ip proxy pool in python crawler". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it out for more people to see!