Downloading compressed content over HTTP using Python

compression, gzip, http, httplib2, python

Solution

httplib2 supports 'deflate' and 'gzip' compression.

Example

import httplib2
h = httplib2.Http(".cache")
resp, content = h.request("http://example.org/", "GET")

The content is decompressed as necessary.

Problem

How do I take advantage of HTTP 1.1's compression when downloading web pages using Python? I am currently using the built-in urllib module for downloading web content. Reading through the documentation I couldn't find any information that is indeed using compression. Is it already built-in into urllib or is there another library that I can use?

Original source