Python multiprocessing, ValueError: I/O operation on closed file
file, multiprocessing, python
Solution
I had similar issues in the past. Not sure whether it is done within the multiprocessing module or whether `open` sets the close-on-exec flag by default but I know for sure that file handles opened in the main process are closed in the multiprocessing children.
The obvious work around is to pass the filename as a parameter to the child process' init function and open it once within each child (if using a pool), or to pass it as a parameter to the target function and open/close on each invocation. The former requires the use of a global to store the file handle (not a good thing) - unless someone can show me how to avoid that :) - and the latter can incur a performance hit (but can be used with multiprocessing.Process directly).
Example of the former:
filehandle = None
def child_init(filename):
global filehandle
filehandle = open(filename,...)
../..
def child_target(args):
../..
if __name__ == '__main__':
# some code which defines filename
proc = multiprocessing.Pool(processes=1,initializer=child_init,initargs=[filename])
proc.apply(child_target,args)
Problem
I'm having a problem with the Python multiprocessing package. Below is a simple example code that illustrates my problem. ``` import multiprocessing as mp import time def test_file(f): f.write("Testing...\n") print f.name return None if __name__ == "__main__": f = open("test.txt", 'w') proc = mp.Process(target=test_file, args=[f]) proc.start() proc.join() ``` When I run this, I get the following error. ``` Process Process-1: Traceback (most recent call last): File "C:\Python27\lib\multiprocessing\process.py", line 258, in _bootstrap self.run() File "C:\Python27\lib\multiprocessing\process.py", line 114, in run self.target(*self._args, **self._kwargs) File "C:\Users\Ray\Google Drive\Programming\Python\tests\follow_test.py", line 24, in test_file f.write("Testing...\n") ValueError: I/O operation on closed file Press any key to continue . . . ``` It seems that somehow the file handle is 'lost' during the creation of the new process. Could someone please explain what's going on?