Pickle doesn't take my registered custom type reduce function into account on python 2.6

pickle, python, python-2.6

Solution

A new feature was added to Python 2.7.3 that allows for dynamically created classes to be pickled. See issue 7689:

Currently, it's impossible to use the usual pickle mechanisms to pickle a dynamically created class, even if the user requests a different pickling mechanism via copy_reg. The attached patch makes this customization possible by simply transposing two blocks of code.

and

The copy_reg module exists to handle custom pickling of objects, which is exactly what's needed here. However, the code from #494904 that checks for instances of a metaclass does this *just before* it looks at the copy_reg dispatch table. The patches just reorder these tests in pickle.py and cPickle.c, so that one can register a custom pickler for instances of a metaclass.

And from the 2.7.3 changelog:

Issue #7689: Allow pickling of dynamically created classes when their metaclass is registered with copy_reg. Patch by Nicolas M. Thiéry and Craig Citro.

The only way around this for Python 2.6 is to port the `pickle.py` module over to that version. You could try to just bundle the 2.7.3 `pickle.py` module with 2.6, as a pure Python implementation it should Just Work.

Alternatively, monkey-patch the `pickle.Pickler.save` method:

from copy_reg import dispatch_table
from types import TypeType, StringType, TupleType
from pickle import Pickler, PicklingError


def pickler_save(self, obj):
    # Check for persistent id (defined by a subclass)
    pid = self.persistent_id(obj)
    if pid:
        self.save_pers(pid)
        return

    # Check the memo
    x = self.memo.get(id(obj))
    if x:
        self.write(self.get(x[0]))
        return

    # Check the type dispatch table
    t = type(obj)
    f = self.dispatch.get(t)
    if f:
        f(self, obj) # Call unbound method with explicit self
        return

    # Check copy_reg.dispatch_table
    reduce = dispatch_table.get(t)
    if reduce:
        rv = reduce(obj)
    else:
        # Check for a class with a custom metaclass; treat as regular class
        try:
            issc = issubclass(t, TypeType)
        except TypeError: # t is not a class (old Boost; see SF #502085)
            issc = 0
        if issc:
            self.save_global(obj)
            return

        # Check for a __reduce_ex__ method, fall back to __reduce__
        reduce = getattr(obj, "__reduce_ex__", None)
        if reduce:
            rv = reduce(self.proto)
        else:
            reduce = getattr(obj, "__reduce__", None)
            if reduce:
                rv = reduce()
            else:
                raise PicklingError("Can't pickle %r object: %r" %
                                    (t.__name__, obj))

    # Check for string returned by reduce(), meaning "save as global"
    if type(rv) is StringType:
        self.save_global(obj, rv)
        return

    # Assert that reduce() returned a tuple
    if type(rv) is not TupleType:
        raise PicklingError("%s must return string or tuple" % reduce)

    # Assert that it returned an appropriately sized tuple
    l = len(rv)
    if not (2 <= l <= 5):
        raise PicklingError("Tuple returned by %s must have "
                            "two to five elements" % reduce)

    # Save the reduce() output and finally memoize the object
    self.save_reduce(obj=obj, *rv)


Pickler.save = pickler_save

With the above monkey patch, your example works on Python 2.6.8:

>>> pickle.dumps(B)
Hey there
'c__main__\nB\np0\n.'

Problem

Original source