Can python abstract base classes inherit from C extensions?

abc, gevent, greenlets, inheritance, python

Solution

The cause of your problem is that it's the `object.__new__` method that does the check for instantiation of an abstract class, and in this case `object.__new__` isn't being invoked: `gevent.Greenlet` inherits from `greenlet.greenlet`, and `greenlet.greenlet` is a C extension type whose `__new__` implementation doesn't call `object.__new__` at any point (see the `green_new` function in the `greenlet` C source).

You can see the same effect by subclassing some of the other builtin types that implement their own `__new__` method and don't refer back to `object.__new__` (the `float` type, for example). The issue is not particular to C extension types, though: you can also replicate it with pure Python types. Consider the code below:

import abc

class A(object):
    def __new__(cls):
        # self = object.__new__(cls)
        return 42

class B(A):
    __metaclass__ = abc.ABCMeta

    @abc.abstractmethod
    def foo(self):
        pass

b = B()  # No exception.

The class `B` is correctly registered as an abstract class (internally, its `Py_TPFLAGS_IS_ABSTRACT` bit is set in the `tp_flags` field), but `object.__new__` is never called, so there's no error when `B` is instantiated. However, if you uncomment the `self = object.__new__(cls)` method call in `A`, you'll see the expected error on instantiation.

As far as the 'right way' to implement this, unfortunately I think the right way is to fix the `greenlet` type so that its `__new__` method calls `object.__new__`. I guess you could add a `__new__` method to `ActorBase` that explicitly calls both the base class `__new__` and `object.__new__` (and throws away the result of the latter), but I'd consider that an ugly workaround rather than the 'right way'. (EDIT: And on top of that, it doesn't work. I get `TypeError: object.__new__(ActorBase) is not safe, use greenlet.greenlet.__new__()` from the `object.__new__` call.) I've opened an issue on the greenlet tracker.

EDIT: This problem seemed somewhat familiar, and I just did some digging into the Enthought Traits source, which defines a `CHasTraits` class implemented in C which does play nicely with ABCs. And its `__new__` method starts like this (comments are from the original source, not mine):

PyObject *
has_traits_new ( PyTypeObject * type, PyObject * args, PyObject * kwds ) {

    // Call PyBaseObject_Type.tp_new to do the actual construction.
    // This allows things like ABCMeta machinery to work correctly
    // which is implemented at the C level.
    has_traits_object * obj = (has_traits_object *) PyBaseObject_Type.tp_new(type, empty_tuple, empty_dict);

So perhaps the long-term solution is to persuade the `greenlet` folks to do something similar.

Problem

Original source