cPickle - different results pickling the same object

pickle, python, serialization

Solution

It is looking at the reference counts, from the cPickle source:

if (Py_REFCNT(args) > 1) {
    if (!( py_ob_id = PyLong_FromVoidPtr(args)))
        goto finally;

    if (PyDict_GetItem(self->memo, py_ob_id)) {
        if (get(self, py_ob_id) < 0)
            goto finally;

        res = 0;
        goto finally;
    }
}

The pickle protocol has to deal with pickling multiple references to the same object. In order to prevent duplicating the object when depickled it uses a memo. The memo basically maps indexes to the various objects. The PUT (p) opcode in the pickle stores the current object in this memo dictionary.

However, if there is only a single reference to an object, there is no reason to store it it the memo because it is impossible to need to reference it again because it only has one reference. Thus the cPickle code checks the reference count for a little optimization at this point.

So yes, its the reference counts. But not that's not a problem. The objects unpickled will have the correct reference counts, it just produces a slightly shorter pickle when the reference counts are at 1.

Now, I don't know what you are you doing that you care about this. But you really shouldn't assume that pickling the same object will always give you the same result. If nothing else, I'd expect dictionaries to give you problems because the order of the keys is undefined. Unless you have python documentation that guarantees the pickle is the same each time I highly recommend you don't depend on it.

Problem

Is anyone able to explain the comment under `testLookups()` in this code snippet? I've run the code and indeed what the comment sais is true. However I'd like to understand why it's true, i.e. why is cPickle outputting different values for the same object depending on how it is referenced. Does it have anything to do with reference count? If so, isn't that some kind of a bug - i.e. the pickled and deserialized object would have an abnormally high reference count and in effect would never get garbage collected?

Original source