Multiprocessing : NULL result without error in PyObject_Call
multidimensional-array, multiprocessing, numpy, python, queue
Solution
The bug report your reference states that multiprocessing module is unable to push huge arguments to subprocess.
The reason is that it needs to pickle these arguments and store the pickled blob somewhere in memory.
You, however, don't need to pass arrays as arguments.
Possible causes:
- passing a closure `loop` as a target
- passing `mp.Queue()` as argument
Please see http://stevenengelhardt.com/2013/01/16/python-multiprocessing-module-and-closures/ about converting your closure to a class.
Set up full state before you give control to multiprocessing.
Problem
Here is a sample program where I use multiprocessing. The calculations are done with `multiprocessing.Process` and the results are collected using `multiprocessing.Queue`. ``` #THIS PROGRAM RUNS WITH ~40Gb RAM. (you can reduce a,b,c for less RAM #but then it works for smaller values) #PROBLEM OCCURS ONLY FOR HUGE DATA. from numpy import * import multiprocessing as mp a = arange(0, 3500, 5) b = arange(0, 3500, 5) c = arange(0, 3500, 5) a0 = 540. #random values b0 = 26. c0 = 826. def rand_function(a, b, c, a0, b0, c0): Nloop = 100. def loop(Nloop, out): res_total = zeros((700, 700, 700), dtype = 'float') n = 1 while n <= Nloop: rad = sqrt((a-a0)**2 + (b-b0)**2 + (c-c0)**2) res_total += rad n +=1 out.put(res_total) out = mp.Queue() jobs = [] Nprocs = mp.cpu_count() print "No. of processors : ", Nprocs for i in range(Nprocs): p = mp.Process(target = loop, args=(Nloop/Nprocs, out)) jobs.append(p) p.start() final_result = zeros((700, 700, 700), dtype = 'float') for i in range(Nprocs): final_result = final_result + out.get() p.join() test = rand_function(a,b,c,a0, b0, c0) ``` Here is the error message : ``` Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/queues.py", line 266, in _feed send(obj) SystemError: NULL result without error in PyObject_Call ``` I read here that it is a bug. But I am unable to understand. Can anyone please tell me any way out to calculate huge data using multiprocessing? Thank you very much