User Agent of python crawler

In the process of learning crawlers, sometimes do not use the head in the use of python crawler script just crawled twice, just a test can not open this page, at first has been confused, until later know that python does crawlers when the default user agent is the large version of python, python2.7. User-Agent: Python-urllib/2.7;python3.5. User-Agent: Python-urllib/3.5

Let's do an experiment:

The python code is as follows:

Python2

Import urllib2url = "http://www.baidu.com/"request = urllib2.Request (url) response = urllib2.urlopen (request) print (response.read ())

Python3

From urllib import requesturl = "http://www.baidu.com/"req = request.Request (url) response = request.urlopen (req) print (response.read () .decode ()

We open fiddler, run the completion code, and then look at our data on fiddler.

It's obviously the version of python.

Therefore, no matter what we climb when we learn about crawlers, it is best to add this header message to the code.

Let's add a header to the code.

From urllib import requestheaders = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64"} url = "http://www.baidu.com/"req = request.Request (url,headers=headers) response = request.urlopen (req) print (response.read (). Decode ())

The result of grabbing the package is as follows: