Adapt an iterator to behave like a file-like object in Python
python
Solution
Here's a solution that should read from your iterator in chunks.
class some_magic_adaptor:
def __init__( self, it ):
self.it = it
self.next_chunk = ""
def growChunk( self ):
self.next_chunk = self.next_chunk + self.it.next()
def read( self, n ):
if self.next_chunk == None:
return None
try:
while len(self.next_chunk)<n:
self.growChunk()
rv = self.next_chunk[:n]
self.next_chunk = self.next_chunk[n:]
return rv
except StopIteration:
rv = self.next_chunk
self.next_chunk = None
return rv
def str_fn():
for c in 'a', 'b', 'c':
yield c * 3
ff = some_magic_adaptor( str_fn() )
while True:
data = ff.read(4)
if not data:
break
print data
Problem
I have a generator producing a list of strings. Is there a utility/adapter in Python that could make it look like a file? For example, ``` >>> def str_fn(): ... for c in 'a', 'b', 'c': ... yield c * 3 ... >>> for s in str_fn(): ... print s ... aaa bbb ccc >>> stream = some_magic_adaptor(str_fn()) >>> while True: ... data = stream.read(4) ... if not data: ... break ... print data aaab bbcc c ``` Because data may be big and needs to be streamable (each fragment is a few kilobytes, the entire stream is tens of megabytes), I do not want to eagerly evaluate the whole generator before passing it to stream adaptor.