Shulou(Shulou.com)06/02 Report--

Today, I will talk to you about how to achieve a crawler function in Python. Many people may not know much about it. In order to make you understand better, the editor has summarized the following content for you. I hope you can get something according to this article.

Running result

Python 2.6.6 (r266GCC Python 84292, Jun 20 2019, 14:14:55) [GCC 4.4.7 20120313 (Red Hat 4.4.7-23)] on linux2Type "help", "copyright", "credits" or "license" for more information. > > import requestsTraceback (most recent call last): File ", line 1, in File" / usr/lib/python2.6/site-packages/requests/__init__.py ", line 43 In import urllib3 File "/ usr/lib/python2.6/site-packages/urllib3/__init__.py", line 7, in from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool, connection_from_url File "/ usr/lib/python2.6/site-packages/urllib3/connectionpool.py", line 100 _ blocking_errnos = {errno.EAGAIN, errno.EWOULDBLOCK} ^ SyntaxError: invalid syntax

Since the Python version 2.6.6 on the Linux server has already been reported as an error, this library is not available. I tried several ways to make this mistake, and all of them failed. The previous article was completed in the windows environment Python2.7.

You might say upgrade the Python version? I have learned a painful lesson from upgrading the glibc to kill the server, so I dare not upgrade it any more, and the machine I tested is a production server running on other systems 24 hours a day. Once there is a problem with the upgrade, it will lead to a production accident. So I can only develop crawlers under 2. 6 Python. Using the urllib2 library instead of the requests library, the implementation process is basically the same simple and clear. The code is as follows

# coding=utf-8import urllib2exact_url=' https://news.qq.com/zt2020/page/feiyan.htm'try: r=urllib2.urlopen (exact_url) # grab the data of setting url, and change it to any address you want: except urllib2.URLError,e: print e.code exit () r.encoding='utf8'html=r.read () print html # print the crawl result

So if your server Python version is 2.6 or lower, try using the urllib2 library!

Lower versions of Python will also fail to install mongodb's driver package pymongo, as follows

In this case, you can choose MySQL's Python driver package instead. Pip may not be successful in installing MySQL's Python driver package. I finally installed it successfully with yum.

What is even more bizarre is that django cannot be installed successfully at 2.6pm.

After reading the above, do you have any further understanding of how to implement a crawler function in Python? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.