urllib2 not retrieving entire HTTP response

http, python, urllib2

Solution

Best way to get all of the data:

fp = urllib2.urlopen("http://www.example.com/index.cfm")

response = ""
while 1:
    data = fp.read()
    if not data:         # This might need to be    if data == "":   -- can't remember
        break
    response += data

print response

The reason is that `.read()` isn't guaranteed to return the entire response, given the nature of sockets. I thought this was discussed in the documentation (maybe `urllib`) but I cannot find it.

Problem

I'm perplexed as to why I'm not able to download the entire contents of some JSON responses from FriendFeed using urllib2. ``` >>> import urllib2 >>> stream = urllib2.urlopen('http://friendfeed.com/api/room/the-life-scientists/profile?format=json') >>> stream.headers['content-length'] '168928' >>> data = stream.read() >>> len(data) 61058 >>> # We can see here that I did not retrieve the full JSON ... # given that the stream doesn't end with a closing } ... >>> data[-40:] 'ce2-003048343a40","name":"Vincent Racani' ``` How can I retrieve the full response with urllib2?

Original source