Linux kernel - how to stop a kthread waiting for a semaphore?

c, kernel-module, linux-kernel, multithreading, semaphore

Solution

The missing code means that this answer can only use educated guesses.

Here are my assumptions about your missing code:

If `take_from_list` returns a valid entry, `consumer_kthread` does something with the entry and calls `up(&sem_list_consumer)` to match the call to `down_interruptible(&sem_list_consumer)` in `take_from_list`.

If `take_from_list` returns `NULL`, `consumer_kthread` does some handling of the `NULL` pointer, and assumes the `sem_list_consumer` semaphore is in its original state.

Given those assumptions, there is a bug in `take_from_list` because it sometimes returns `NULL` without calling `up(&sem_list_consumer)` first. That means that any subsequent calls to `take_from_list` will block on the call to `down_interruptible(&sem_list_consumer)` until they are interrupted by a signal. To fix that bug, change `take_from_list` to always leave the semaphore in the state it left it whenever it returns `NULL`:

struct some_struct * take_from_list() {
    int rv;
    some_struct *entry;

    rv = down_interruptible(&sem_list_consumer);
    if(rv != 0) {
        return NULL;
    }

    rv = mutex_lock_interruptible(&list_lock);
    if(rv != 0) {
        up(&sem_list_consumer);
        return NULL;
    }

    if (list_empty(&list)) {
        mutex_unlock(&list_lock);
        up(&sem_list_consumer);  /* <-- this line was missing */
        return NULL;
    } else {
        entry = list_last_entry(&list, struct some_struct, list);
        if (entry) {
            list_del(&entry->list);
        }
    }

    mutex_unlock(&list_lock);
    return entry;
}

AMENDED

If there is some place in the missing code of `consumer_kthread` that adds itself to a wait queue and goes to sleep, a call to `kthread_should_stop()` should be included in the wake-up conditions. The wake-up condition should be satisfied by the other conditions OR (`||`) `kthread_should_stop()`.

A call to `kthread_stop(consumer_task)` from your `exitModule` function will wake up the consumer thread. If it is waiting on an event, the first thing it will do is check the wake up conditions and go back to sleep if they are not satisfied. By including `kthread_should_stop()` as one of the possible wake-up conditions, you ensure that the consumer thread does not immediately go back to sleep.

Problem

While writing a Linux kernel module, I faced a problem with a kthread that I am unable to wake up while waiting for a semaphore to unlock. This causes the thread to be unstoppable and `rmmod` to freeze when trying to unload the module. Please note: This module runs on a 3.10 kernel and I have no way to update this to a newer version (customer demands running on stock CentOS 7, which features a 3.10 kernel). Below are the interesting parts from the module source code. It represents a simple producer consumer problem, the list is not limited in size (thus no producer semaphore is required) and guarded by a mutex. The function to take something from the list is guarded by a semaphore that is upped by the producer and downed by the consumer. The producer function is called from an external event (in fact a char device) not shown in this code snippets to keep is as small as possible. The process works perfectly, except for the module unloading. The parts that cause the freezing are marked with comments in the code snippets. The only way I know of to stop a kthread is to call `kthread_stop` on it, which fails in this case since it apparently cannot wake up the sleeping thread. Because it waits for the thread to exit, the call will never return and the module will not unload. How can I wake up and stop the kthread waiting for the semaphore to unload the module successfully? List implementation: ``` #include <linux/mutex.h> #include <linux/list.h> #include <linux/semaphore.h> static LIST_HEAD(list); DEFINE_MUTEX(list_lock); DEFINE_SEMAPHORE(sem_list_consumer); void add_to_list(struct *some_struct) { int rv = mutex_lock_interruptible(&list_lock); if(rv != 0) { return; } list_add(&some_struct->list, &list); mutex_unlock(&list_lock); up(&sem_list_consumer); } struct some_struct * take_from_list() { int rv; some_struct *entry; /* this is where the kthread will freeze when module is unloaded */ rv = down_interruptible(&sem_list_consumer); if(rv != 0) { return NULL; } rv = mutex_lock_interruptible(&list_lock); if(rv != 0) { up(&sem_list_consumer); return NULL; } if (list_empty(&list)) { mutex_unlock(&list_lock); return NULL; } else { entry = list_last_entry(&list, struct some_struct, list); if (entry) { list_del(&entry->list); } } mutex_unlock(&list_lock); return entry; } ``` Consumer kthread implementation: ``` #include <linux/kthread.h> #include <linux/sched.h> int consumer_kthread(void *data) { struct some_struct *entry; set_current_state(TASK_INTERRUPTIBLE); while (!kthread_should_stop()) { /* Here the function including the semaphore is called */ entry = take_from_list(); if(entry != NULL) { /* Do something with 'entry' here */ } else { /* Some handling of returned NULL pointers */ } set_current_state(TASK_INTERRUPTIBLE); } set_current_state(TASK_RUNNING); return 0; } ``` Module implementation: ``` #include <linux/init.h> #include <linux/kthread.h> #include <linux/module.h> #include <linux/sched.h> static struct task_struct *consumer_task; static int __init initModule(void) { consumer_task = kthread_run(consumer_kthread, NULL, "list-consumer"); return 0; } static void __exit exitModule(void) { /* this call will cause rmmod to freeze forever */ kthread_stop(consumer_task); } module_init(initModule); module_exit(exitModule); MODULE_LICENSE("GPL v2"); MODULE_DESCRIPTION("My Module"); ```

Original source