Python: is the iteration of the multidimensional array super slow?
numpy, performance, python
Solution
Python is a much more dynamic language than C or C#. The main reason why the loop is so slow is that on every pass, the CPython interpreter is doing some extra work that wastes time: specifically, it is binding the name `x` with the next object from the iterator, then when it evaluates the assignment it has to look up the name `x` again.
As @Sven Marnach noted, you can call a method function `numpy.fill()` and it is fast. That function is compiled C or maybe Fortran, and it will simply loop over the addresses of the `numpy.array` data structure and fill in the values. Much less dynamic than Python, which is good for this simple case.
But now consider PyPy. Once you run your program under PyPy, a JIT analyzes what your code is actually doing. In this example, it notes that the name `x` isn't used for anything but the assignment, and it can optimize away binding the name. This example should be one that PyPy speeds up tremendously; likely PyPy will be ten times faster than plain Python (so only one-tenth as fast as C, rather than 1/100 as fast).
http://pypy.org
As I understand it, PyPy won't be working with Numpy for a while yet, so you can't just run your existing Numpy code under PyPy yet. But the day is coming.
I'm excited about PyPy. It offers the hope that we can write in a very high-level language (Python) and yet get nearly the performance of writing things in "portable assembly language" (C). For examples like this one, the Numpy might even beat the performance of naive C code, by using SIMD instructions from the CPU (SSE2, NEON, or whatever). For this example, with SIMD, you could set four integers to 123 with each loop, and that would be faster than a plain C loop. (Unless the C compiler used a SIMD optimization also! Which, come to think of it, is likely for this case. So we are back to "nearly the speed of C" rather than faster for this example. But we can imagine trickier cases that the C compiler isn't smart enough to optimize, where a future PyPy might.)
But never mind PyPy for now. If you will be working with Numpy, it is a good idea to learn all the functions like `numpy.fill()` that are there to speed up your code.
Problem
I have to iterate all items in two-dimensional array of integers and change the value (according to some rule, not important). I'm surprised how significant difference in performance is there between python runtime and C# or java runtime. Did I wrote totally wrong python code (v2.7.2)? ``` import numpy a = numpy.ndarray((5000,5000), dtype = numpy.int32) for x in numpy.nditer(a.T): x = 123 ``` ``` >python -m timeit -n 2 -r 2 -s "import numpy; a = numpy.ndarray((5000,5000), dtype=numpy.int32)" "for x in numpy.nditer(a.T):" " x = 123" 2 loops, best of 2: 4.34 sec per loop ``` For example the C# code performs only 50ms, i.e. python is almost 100 times slower! (suppose the `matrix` variable is already initialized) ``` for (y = 0; y < 5000; y++) for (x = 0; x < 5000; x++) matrix[y][x] = 123; ```