Using threading lock in ctypes callback function

callback, ctypes, multithreading, python

Solution

The first thing that ctypes `_CallPythonObject` does is call `PyGILState_Ensure()`, which will call `PyThreadState_New` to create a new thread state if necessary. Beyond that it's vanilla Python threaded code, so your lock should work fine. It worked for me in the example below (Linux, Python 2.7.3):

from ctypes import *
import threading

lock = threading.Lock()
lib = CDLL('./tmp.so')
callback_t = CFUNCTYPE(c_int)

def f():
  with lock:
    print threading.current_thread()
  return 21

callback = callback_t(f)
lib.registerCallback(callback)

>>> lock.acquire()
True
>>> t = threading.Thread(target=lib.event)
>>> t.start()
>>> lock.locked()
True
>>> lock.release()
>>> <_DummyThread(Dummy-2, started daemon -1230402704)>
res: 21

The `DummyThread` output is from printing the current thread in the callback. Threads created outside of Python get a 'dummy' name.

tmp.c:

#include <stdio.h>
#include <pthread.h>

typedef int (*callback_t)(void);
callback_t callback = NULL;

void *doCallback(void *arg)
{
    int res = callback();
    printf("res: %d\n", res);
    pthread_exit(0);
}

int event(void)
{
    pthread_t callback_thread;
    pthread_create(&callback_thread, NULL, doCallback, NULL);
    pthread_join(callback_thread, NULL);
    return 0;
}

int registerCallback(callback_t foo)
{
    callback = foo;
    return 0;
}

Problem

I want to use a ctypes dll from a twisted application. Minimal example concocted here: ``` from ctypes import * from threading import Lock lock = Lock() dll = windll.LoadLibrary('mydll.dll') l = [1,2,3] def callback(): lock.acquire() l.pop() lock.release() return 0 C_CALLBACK = CFUNCTYPE(c_int) c_callback = C_CALLBACK(callback) # this is a non blocking function call starts a hardware task that fires a callback upon completion dll.registerCallback(c_callback) while(True): # in reality this block gets called from a twisted server application lock.acquire() l.append(l.pop() + 1) lock.release() ``` The dll has a function (`dll.registerCallback`) that takes in a ctypes callback function, starts a hardware event, and fires a callback when the hardware indicates the hardware task is complete. From the API documentation: The callback function is called in a DAQmx thread. Somewhere on the web they attempt to explain what a "DAQmx thread" is: ...your callback will be called and run in a DAQmx driver thread and will be run asynchronously (not in the same thread) in relation to your program. The full documentation cound be found here. I changed the function signature in my example for simplicity. So I suppose we can safely assume that the dll is spawning a thread. Will the locks I have in place ensure that the callback function does not attempt to access the list `l` when it's in the middle of the `pop` operation in the mainloop, and vice versa? Or does this scheme only work when you use threads created using the `threading` library? What is the recommended practice here?

Original source