requests - how to stream upload - partial file
python, python-requests
Solution
Based off Greg's answers to my questions I think the following will work best:
First you'll need something to wrap your open file so that it limits how much data can be read:
class FileLimiter(object):
def __init__(self, file_obj, read_limit):
self.read_limit = read_limit
self.amount_seen = 0
self.file_obj = file_obj
# So that requests doesn't try to chunk the upload but will instead stream it:
self.len = read_limit
def read(self, amount=-1):
if self.amount_seen >= self.read_limit:
return b''
remaining_amount = self.read_limit - self.amount_seen
data = self.file_obj.read(min(amount, remaining_amount))
self.amount_seen += len(data)
return data
This should roughly work as a good wrapper object. Then you would use it like so:
with open('my_large_file', 'rb') as file_obj:
file_obj.seek(my_offset)
upload = FileLimiter(file_obj, my_chunk_limit)
r = requests.post(url, data=upload, headers={'Content-Type': 'application/octet-stream'})
The headers are obviously optional, but when streaming data to a server, it's a good idea to be a considerate user and tell the server what the type of the content is that you're sending.
Problem
My goal is to do a PUT of part of a file using requests and stream the file (i.e., not load it into memory and then do the PUT). This page explains how you would do that for an entire file: Requests supports streaming uploads, which allow you to send large streams or files without reading them into memory. To stream and upload, simply provide a file-like object for your body: ``` with open('massive-body', 'rb') as f: requests.post('http://some.url/streamed', data=f) ``` However in my case I want to only send one chunk of the file. Is there a way to accomplish this? In concept, something like: ``` with open('massive-body', 'rb') as f: requests.post('http://some.url/streamed', data=f.read(chunksize)) ```