Multiprocessing incompatible with NumPy
multiprocessing, numpy, python
Solution
(First Post sorry if it is not well formulated or alligned)
You can stop Numpy to use multithreading by seting the MKL_NUM_THREADS to 1
Under debian I used:
export MKL_NUM_THREADS=1
Source from related stackoverflow post: Python: How do you stop numpy from multithreading?
Result:
user@pc:~/tmp$ python multi.py
CostlyFunc : elapsed time 3.847009 s
CostlyFunc : elapsed time 3.253226 s
CostlyFunc : elapsed time 3.415734 s
Elapsed time (serial) : 10.5163660049
CostlyFunc : elapsed time 4.218424 s
CostlyFunc : elapsed time 5.252429 s
CostlyFunc : elapsed time 4.862513 s
Elapsed time (multiporcessing) : 9.11713695526
user@pc:~/tmp$ export MKL_NUM_THREADS=1
user@pc:~/tmp$ python multi.py
CostlyFunc : elapsed time 3.014677 s
CostlyFunc : elapsed time 3.102548 s
CostlyFunc : elapsed time 3.060915 s
Elapsed time (serial) : 9.17840886116
CostlyFunc : elapsed time 3.720322 s
CostlyFunc : elapsed time 3.950583 s
CostlyFunc : elapsed time 3.656165 s
Elapsed time (multiporcessing) : 7.399310112
I am not sure if that helps because I guess eventually you want numpy to run in parallel maybe try to adjust the number of threads for numpy to your machine.
Problem
I am trying to run a simple test using multiprocessing. The test works well until I import numpy (even though it is not used in the program). Here is the code: ``` from multiprocessing import Pool import time import numpy as np #this is the problematic line def CostlyFunc(N): """""" tstart = time.time() x = 0 for i in xrange(N): for j in xrange(N): if i % 2: x += 2 else: x -= 2 print "CostlyFunc : elapsed time %f s" % (time.time() - tstart) return x #serial application ResultList0 = [] StartTime = time.time() for i in xrange(3): ResultList0.append(CostlyFunc(5000)) print "Elapsed time (serial) : ", time.time() - StartTime #multiprocessing application StartTime = time.time() pool = Pool() asyncResult = pool.map_async(CostlyFunc, [5000, 5000, 5000]) ResultList1 = asyncResult.get() print "Elapsed time (multiporcessing) : ", time.time() - StartTime ``` If I don't import numpy the result is: ``` CostlyFunc : elapsed time 2.866265 s CostlyFunc : elapsed time 2.793213 s CostlyFunc : elapsed time 2.794936 s Elapsed time (serial) : 8.45455098152 CostlyFunc : elapsed time 2.889815 s CostlyFunc : elapsed time 2.891556 s CostlyFunc : elapsed time 2.898898 s Elapsed time (multiporcessing) : 2.91595196724 ``` The total elapsed time is similar to the time required for 1 process, meaning that the computation has been parallelized. If I do import numpy the result becomes : ``` CostlyFunc : elapsed time 2.877116 s CostlyFunc : elapsed time 2.866778 s CostlyFunc : elapsed time 2.860894 s Elapsed time (serial) : 8.60492110252 CostlyFunc : elapsed time 8.450145 s CostlyFunc : elapsed time 8.473006 s CostlyFunc : elapsed time 8.506402 s Elapsed time (multiporcessing) : 8.55398178101 ``` The total time elapsed is the same for both serial and multiprocessing methods because only one core is used. It is clear that the problem comes from numpy. Is it possible that I have an incompatibility between my versions of multiprocessing and NumPy? I am currently using Python2.7, NumPy 1.6.2 and multiprocessing 0.70a1 on linux