Cythonizing for loops that iterate over generators
coroutine, cython, for-loop, generator, python
Solution
Some recommendations:
- Cython supports generators out of the box, so you should try passing your Python code with generators to `cython` and see what kind of speedup you get.
- Next step is to add as much static typing information to your loops to speed up the work the generators are doing.
- Python generators are cool, but if performance is important, they aren't the fastest way to do things. You're much better off converting your bottlenecks to working with contiguous arrays.
- Check out Cython's typed memoryviews.
- You can also use Cython with C++ std::vectors and other high-performance container objects.
We'll need more information about your goals and constraints to provide more help here. A stripped down example would be helpful.
Problem
I have Python code that has lots of loops that consume data from Python generators. Some also re yield the processed data. This is a bottleneck and I want to speed this part up and was thinking of using Cython. What is the recommended way to deal with generators and yield. I would like to - Convert Python generators into Cython without data copies - Make Cython `for` loops consume data produced by Python generators - Yield data like a generator I would guess this is a common enough use case, what is the recommended ways to do this.