Shulou(Shulou.com)06/01 Report--

This article mainly shows you "how python is disguised as a browser". The content is simple and clear. I hope it can help you solve your doubts. Let me lead you to study and learn this article "how to disguise python as a browser".

Disguise as a browser

Some websites are disgusted with the visit of the crawler, so they all refuse the request. So HTTP Error 403: Forbidden often occurs when you visit a website directly with urllib2.

Pay special attention to some header. The Server side will check for these header:

Some Server or Proxy of User-Agent check this value to determine whether the Request is initiated by the browser.

When Content-Type uses the REST interface, Server checks this value to determine how to parse the contents of the HTTP Body. This can be achieved by modifying the header in the http package. The code snippet is as follows:

Import urllib2headers = {'User-Agent':'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'} request = urllib2.Request (url =' http://my.oschina.net/jhao104/blog?catalog=3463517', headers = headers) print urllib2.urlopen (request). Read () is all the content of the article "how does python pretend to be a browser". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.