Shulou(Shulou.com)06/03 Report--

Recently, I was studying crawlers and needed to deploy the IP proxy pool in front of me, so I found proxy pool in open source China. It can automatically crawl the IP of several free IP proxy websites in China, and verify the availability of IP in real time, and the database is SSDB.

IP proxy Pool website:

Http://www.data5u.com/

Http://www.data5u.com/free/

Http://www.data5u.com/free/gngn/index.shtml

Http://www.data5u.com/free/gnpt/index.shtml

Http://www.66ip.cn/

Http://www.ip181.com/

Http://www.xicidaili.com/nn

Http://www.xicidaili.com/nt

Http://www.goubanjia.com/free/gngn/index.shtml

Http://www.xdaili.cn/ipagent/freeip/getFreeIps?page=1&rows=10

2 yum-y install git

3 yum-y install wget

4 yum install curl-devel expat-devel gettext-devel openssl-devel zlib-devel

5 yum install gcc perl-ExtUtils-MakeMaker epel-release gcc-c++

8 cd / usr/src/

16 wget https://www.kernel.org/pub/software/scm/git/git-2.9.5.tar.gz # install the cloning tool git

17 tar-xzf git-2.9.5.tar.gz

18 cd git-2.9.5

19 make prefix=/usr/local/git all

20 make prefix=/usr/local/git install

21 echo "export PATH=$PATH:/usr/local/git/bin" > > / etc/bashrc

22 source / etc/bashrc

23 cd..

24 git clone https://github.com/jhao104/proxy_pool.git # Clone proxy_pool

25 cd proxy_pool/

26 python-V # View python version 2.7.5

27 yum-y install python34 # install python 3.4

28 wget-no-check-certificate https://bootstrap.pypa.io/get-pip.py

29 python3 get-pip.py # install pip

30 pip install-r requirements.txt # proxy_pool installation dependency package

32 cd / usr/local/

33 git clone https://github.com/ideawu/ssdb.git # Clone SSDB

34 cd ssdb

35 yum-y install autoconf

37 cd deps/snappy-1.1.0/ # compile Snappy

38. / configure

39 make

40 cd / usr/local/ssdb

41 make # install SSDB

42 make install

43 ln-sf / usr/local/ssdb/ssdb-server / usr/local/bin/ssdb-server

44 ln-sf / usr/local/ssdb/tools/ssdb-cli / usr/local/bin/ssdb-cli

45 ln-sf / usr/local/ssdb/tools/ssdb-dump / usr/local/bin/ssdb-dump

46 ln-sf / usr/local/ssdb/tools/ssdb-repair / usr/local/bin/ssdb-repair

47 ln-sf / usr/local/ssdb/tools/ssdb.sh / etc/rc.d/init.d/ssdb

48 chkconfig-add ssdb

49 chkconfig ssdb on

50 systemctl stop firewalld.service # turn off the firewall

51 systemctl disable firewalld.service

52 firewall-cmd-state

53 pip install-upgrade pyssdb

Start proxy_pool sequence

1 cd / usr/local/ssdb

2. / ssdb-server-d. / ssdb.conf-s start

Note: when the ssdb service starts, it will generate a ssdb.pid file in the var directory. When ssdb shuts down unexpectedly, this file will not be deleted, so when the ssdb service is restarted, there will be an error, so you can restart manually by running the following two lines of command.

. / ssdb-server ssdb.conf-s stop

. / ssdb-server-d. / ssdb.conf-s restart

3 cd / usr/src/proxy_pool/Run

4 python3.4 main.py # start with python3.4

Client access

Http://ip:5010/

Http://ip:5010/get (randomly call an IP and port)

Http://ip:5010/get_all (access all available IP and port)

For specific use, please refer to https://github.com/jhao104/proxy_pool

This article provides details of the deployment under cenots7. I would like to thank contributor and j_hao104 for their selfless dedication!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.