Shulou(Shulou.com)06/01 Report--

This article mainly introduces python how to achieve gzip/deflate support, the article is very detailed, has a certain reference value, interested friends must read it!

Gzip/deflate support

Today's web pages generally support gzip compression, which can often solve a large number of transmission time. Take the VeryCD home page as an example, uncompressed version 247K, compressed after 45K, for the original 1max 5. This means that the grab speed will be five times faster.

However, the urllib/urllib2 of python does not support compression by default. To return the compression format, you must write 'accept-encoding',' in the header of request and then read response. It is tedious and trivial to check the header to see if there is a 'content-encoding' item to determine whether it needs to be decoded. How to make urllib2 automatically support gzip and defalte?

In fact, you can inherit the BaseHanlder class and then handle it in a build_opener way:

Import urllib2from gzip import GzipFilefrom StringIO import StringIOclass ContentEncodingProcessor (urllib2.BaseHandler): "" A handler to add gzip capabilities to urllib2 requests "" # add headers to requests def http_request (self, req): req.add_header ("Accept-Encoding", "gzip, deflate") return req # decode def http_response (self, req) Resp): old_resp = resp # gzip if resp.headers.get ("content-encoding") = = "gzip": gz = GzipFile (fileobj=StringIO (resp.read ()), mode= "r") resp = urllib2.addinfourl (gz, old_resp.headers, old_resp.url Old_resp.code) resp.msg = old_resp.msg # deflate if resp.headers.get ("content-encoding") = = "deflate": gz = StringIO (deflate (resp.read () resp = urllib2.addinfourl (gz, old_resp.headers, old_resp.url Old_resp.code) # 'class to add info () and resp.msg = old_resp.msg return resp# deflate supportimport zlibdef deflate (data): # zlib only provides the zlib compress format, not the deflate format Try: # so on top of all there's this workaround: return zlib.decompress (data,-zlib.MAX_WBITS) except zlib.error: return zlib.decompress (data) then it's simple, encoding_support = ContentEncodingProcessoropener = urllib2.build_opener (encoding_support, urllib2.HTTPHandler) # Open the web page directly with opener If the server supports gzip/defalte, it automatically decompresses content = opener.open (url). Read () is all the contents of the article "how python implements gzip/deflate support". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.