How to convert `lambda` object to `function` object for pickling in Python?

function, functools, lambda, object, python

Solution

`pickle` does not save the code object, just its name. However, giving a function a name is not enough to make it pickleable, because `unpickle` deserializes it by importing the name. Naturally, this means it must be importable by the unpickling runtime. Functions `def`ed with names can still fail to pickle for this reason. So even if you could convert your lambdas to named functions, that still wouldn't work:

>>> import pickle
>>> def foo():
...   def bar():
...     pass
...   return bar
...
>>> pickle.dumps(foo())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: Can't pickle local object 'foo.<locals>.bar'

You say your reason for not using `dill` instead is

I've seen a use case where dill can be imported as pickle which is pretty cool but gets confusing in the code since I used pickle and dill for different types of serialization.

But the `pickle` serialization format is extensible to any internal format you please —`json`, `gzip`, even `dill`. You can have your cake and eat it too!

Let's try it with `dill`, because that makes it easy:

import pickle
import dill
​
class DillPickle:
    def __init__(self, e):
        self.e = e
    def __getstate__(self):
        return dill.dumps(self.e)
    def __setstate__(self, state):
        self.e = dill.loads(state)

Pickle normally serializes the state of custom class instances by recursively pickling the object's `__dict__`. On unpickling, it can then make an uninitialized instance and update its `__dict__` with the saved values. If all of its contents are pickleable all the way down, this works. But if not, (as in the case of the lambdas) pickling would fail. But the `pickle` module provides an interface to override this behavior: `__getstate__()` for serializing the object's state, and `__setstate__()` for restoring it from that value. If the return value of `__getstate__()` is picklable (and `__setstate__()` is implemented), this works. In this case, we return the picklable bytestring returned by `dill.dumps()`.

Demonstration:

>>> pickle.dumps(DillPickle(lambda x, y: x+y))
b'\x80\x03c__main__\nDillPickle\nq\x00)\x81q\x01C\xe7\x80\x03cdill._dill\n_create_function\nq\x00(cdill._dill\n_load_type\nq\x01X\x08\x00\x00\x00CodeTypeq\x02\x85q\x03Rq\x04(K\x02K\x00K\x02K\x02KCC\x08|\x00|\x01\x17\x00S\x00q\x05N\x85q\x06)X\x01\x00\x00\x00xq\x07X\x01\x00\x00\x00yq\x08\x86q\tX\x1f\x00\x00\x00<ipython-input-18-48b9de2c6f55>q\nX\x08\x00\x00\x00<lambda>q\x0bK\x01C\x00q\x0c))tq\rRq\x0ec__builtin__\n__main__\nh\x0bNN}q\x0fNtq\x10Rq\x11.q\x02b.'
>>> pickle.loads(_).e('foo', 'bar')
'foobar'

Notice that we loaded it with `pickle`, not `dill`! And the lambda does not have a name. Of course, `dill` must be available to the runtime in order to unpickle, but so does the rest of the `DillPickle` class, just like for any custom type serialized with `pickle`. `dill` is now a mere implementation detail used internally by `DillPickle`. The interface is `pickle`.

You could even make it a callable, so you don't need to add the `.e` yourself.

class DillPickleCallable(DillPickle):
    def __call__(self, *args, **kwargs):
        return self.e(*args, **kwargs)

Problem

Original source

Related problems

• Can Python pickle lambda functions?