Cython Pickling in Package "not found as" Error
cython, pickle, python
Solution
The problem was in the build script. The `Pickle` module uses the `__module__` attribute of a function/class for pickling. That `__module__` attribute comes from the first argument to the `Extension()` constructor in the `setup.py` script. Since I defined my constructor to be `Extension('reduce', ['reduce.pyx'])`, the `__module__` attribute is `reduce`. It should be `bar/reduce` though since it's now in a package.
Making `setup.py` look like:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [Extension('bar/reduce', ['reduce.pyx'])]
)
solves the problem.
Problem
I'm having trouble pickling a Cython class, but only when it's defined inside a package. This problem was noted previously online, but they didn't state how it was resolved. There are two components here: the Cython pickling using a `__reduce__` method and a package error. Cython Pickling Success I'll first show how it works without the package part. This example works correctly. Cython File My Cython file is `reudce.pyx`: ``` cdef class Foo(object): cdef int n def __init__(self, n): self.n = n def __reduce__(self): return Foo, (self.n,) ``` Setup File This may be compiled with a `setup.py`: ``` from distutils.core import setup from distutils.extension import Extension from Cython.Distutils import build_ext setup( cmdclass = {'build_ext': build_ext}, ext_modules = [Extension("reduce", ["reduce.pyx"])] ) ``` by executing `python setup.py build && cp build/lib*/reduce.so .` Test Script The test script is called `test_reduce.py` and is: ``` import reduce import pickle f = reduce.Foo(4) print pickle.dumps(f) ``` Executing `python test_reduce.py` works fine. Cython Pickling in Package Failure However, once the `reduce.pyx` is put into a package, there is an error. Package Creation To reproduce this, first create a package called `bar`. ``` mkdir bar mv reduce.so bar echo "from reduce import Foo" > bar/__init__.py ``` Test Script Change the `test_reduce.py` file to be: ``` import bar import pickle f = bar.Foo(4) print pickle.dumps(f) ``` Error Message Running `python test_reduce.py` gives the following error: ``` File "/usr/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/usr/lib/python2.7/pickle.py", line 748, in save_global (obj, module, name)) pickle.PicklingError: Can't pickle <type 'reduce.Foo'>: it's not found as reduce.Foo ``` There is a catch of errors which are all turned into a PicklingError in `pickle.py` After looking at that code, the specific error which is occuring is: ``` ImportError: No module named reduce ``` Sanity Test To check that there is not some kind of scope or other issue, if I run the steps which the pickle module should execute, everything works: ``` f = bar.Foo(4) call, args = f.__reduce__() print call(*args) ``` So what's going on here?!