Write data to disk in Python as a background process

file, multiprocessing, python

Solution

You could try using multiple processes like this:

import multiprocessing as mp

def compute(j):
    # compute a bunch of data
    return data

def write(data):
    # write data to disk

if __name__ == '__main__':
    pool = mp.Pool()
    for j in xrange(200):
        pool.apply_async(compute, args=(j, ), callback=write)
    pool.close()
    pool.join()

`pool = mp.Pool()` will create a pool of worker processes. By default, the number of workers equals the number of CPU cores your machine has.

Each pool.apply_async call queues a task to be run by a worker in the pool of worker processes. When a worker is available, it runs `compute(j)`. When the worker returns a value, `data`, a thread in the main process runs the callback function `write(data)`, with `data` being the data returned by the worker.

Some caveats:

- The data has to be picklable, since it is being communicated from the worker process back to the main process via a Queue.

- There is no guarantee that the order in which the workers complete tasks is the same as the order in which the tasks were sent to the pool. So the order in which the data is written to disk may not correspond to `j` ranging from 0 to 199. One way around this problem would be to write the data to a sqlite (or other kind of) database with `j` as one of the fields of data. Then, when you wish to read the data in order, you could `SELECT * FROM table ORDER BY j`.

Using multiple processes will increase the amount of memory required as data is generated by the worker processes and data waiting to be written to disk accumulates in the Queue. You might be able to reduce the amount of memory required by using NumPy arrays. If that is not possible, then you might have to reduce the number of processes:

pool = mp.Pool(processes=1)

That will create one worker process (to run `compute`), leaving the main process to run `write`. Since `compute` takes longer than `write`, the Queue won't get backed up with more than one chunk of data to be written to disk. However, you would still need enough memory to compute on one chunk of data while writing a different chunk of data to disk.

If you do not have enough memory to do both simultaneously, then you have no choice -- your original code, which runs `compute` and `write` sequentially, is the only way.

Problem

I have a program in Python that basically does the following: ``` for j in xrange(200): # 1) Compute a bunch of data # 2) Write data to disk ``` 1) takes about 2-5 minutes 2) takes about ~1 minute Note that there is too much data to keep in memory. Ideally what I would like to do is write the data to disk in a way that avoids idling the CPU. Is this possible in Python? Thanks!

Original source