Skip to contentSkip to conversation

Stopping Kernel Threads at module unload time

DISCUSSION Aritz Bastida 3 participants 10 messages Nov 9, 2005

lore
Share & tools
lore
#0 lore raw

Hello

I've got some questions about kernel threads.I am writing a module which spawns some kernel threads, which would be removed when the module unloads. For that purpose i call kthread_stop() at module unload time. When issuing rmmod on the module, it deadlocks at that point (in the call to kthread_stop), and never returns.

In the thread main function the code was something like this (it's of course simplified).

thread_main()

{
     while( ! kthread_should_stop())
     {
           .............
           wait_event_interruptible(stop_wq, kthread_should_stop() );
     }

     return 0;
}

So if kthread_stop() first sets the thread "closing flag", and then calls wake_up_process(), the thread should wake up, see he should stop, and
end the loop. That doesnt actually never happen.

I have also tried what it is done in kernel/sched.c to finish:

	/* wait for kthread_stop */
	set_current_state(TASK_INTERRUPTIBLE);
	while (!kthread_should_stop()) {	
		schedule();
		set_current_state(TASK_INTERRUPTIBLE);
	}
	__set_current_state(TASK_RUNNING);
                return 0;

I have ensured it actually arrives to that point by using printks, but when the thread goes to sleep, it does never wake up again, so the call to kthread_stop() lasts forever.

I dont know why this happens. Is the module cleanup code in the context of a user process just like system calls? Can that code sleep? If it can't sleep then the answer would be quite easy: kthread_stop() wakes up the processes and then waits for the threads to finish (on call wait_for_completion), but doesnt actually let them execute, because it cannot sleep, so it deadlocks.

So I would be grateful if anyone can help me in this matter. Regards

Aritz

#1 lore raw
Quoted 49 lines · Aritz Bastida

Hello

I've got some questions about kernel threads.I am writing a module which spawns some kernel threads, which would be removed when the module unloads. For that purpose i call kthread_stop() at module unload time. When issuing rmmod on the module, it deadlocks at that point (in the call to kthread_stop), and never returns.

In the thread main function the code was something like this (it's of course simplified).

thread_main()

{
    while( ! kthread_should_stop())
    {
          .............
          wait_event_interruptible(stop_wq, kthread_should_stop() );
    }

    return 0;
}

So if kthread_stop() first sets the thread "closing flag", and then calls wake_up_process(), the thread should wake up, see he should stop, and
end the loop. That doesnt actually never happen.

I have also tried what it is done in kernel/sched.c to finish:

	/* wait for kthread_stop */
	set_current_state(TASK_INTERRUPTIBLE);
	while (!kthread_should_stop()) {
		schedule();
		set_current_state(TASK_INTERRUPTIBLE);
	}
	__set_current_state(TASK_RUNNING);
               return 0;

I have ensured it actually arrives to that point by using printks, but when the thread goes to sleep, it does never wake up again, so the call to kthread_stop() lasts forever.

I dont know why this happens. Is the module cleanup code in the context of a user process just like system calls? Can that code sleep? If it can't sleep then the answer would be quite easy: kthread_stop() wakes up the processes and then waits for the threads to finish (on call wait_for_completion), but doesnt actually let them execute, because it cannot sleep, so it deadlocks.

Quoted 4 lines

So I would be grateful if anyone can help me in this matter. Regards

Aritz

Here is code that actually works. There have been some kernel API changes that convert some of this stuff to macros. Nevertheless. This stuff works.

void make_daemon(const char *dev, int32_t prior)
{

#ifdef OLD_DAEMON              // kernel 2.4.x
     exit_files(current);
     daemonize();
     spin_lock_irq(&current->sigmask_lock);
     sigemptyset(&current->blocked);
     sigdelset(&current->blocked, SIGTERM);
     recalc_sigpending(current);
     spin_unlock_irq(&current->sigmask_lock);
     task_lock(current);
     current->nice = prior;
     strcpy(current->comm, dev);
     task_unlock(current);
#else				// kernel 2.6.x
     daemonize("%s", dev);
     allow_signal(SIGTERM);
     set_user_nice(current, prior);

#endif
//
// Destroy process privs just in case somebody is able to hack this // into a shell.
//

     task_lock(current);
     current->uid  = current->gid  = current->sgid = current->fsgid =
     current->euid = current->egid = current->suid = current->fsuid = 666;
     task_unlock(current);
}

Note above that the task is already running inside the kernel and has access to all your data. You don't need it to be 'root' so that possible back-door gets closed.

//-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= //

//    This is the kernel thread. It handles things that should not
//    be handled in an interrupt. It is awakened from the interrupt
//    service routine.
//
static int32_t local_thread(void *unused)
{
     make_daemon(task_name, PRIOR);
     set_current_state(TASK_INTERRUPTIBLE);	// For the following sleep
     schedule_timeout(1);                        // Let insmod complete
     set_current_state(TASK_RUNNING);		// Remember to restore
     __asm__ __volatile__("thread:\n.global thread\n");
     for(;;)				// Do forever.......
     {
         set_current_state(TASK_INTERRUPTIBLE);
         if(signal_pending(current))
             complete_and_exit(&info->quit, 0);	// This is how you get out
         interruptible_sleep_on(&info->twait);	// Sleep until something to do
         set_current_state(TASK_RUNNING);

         do_stuff(.....);    // Whatever the thread needs to do....

     }
     return 0;
}

This is how the kernel thread is started.

     info->pid = kernel_thread(local_thread, NULL, CLONE_FS|CLONE_FILES);

This is how the kernel thread is stopped.

     if(info->pid)
     {
         (void)kill_proc(info->pid, SIGTERM, 1);
         wait_for_completion(&info->quit);
     }

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13.4 on an i686 machine (5589.55 BogoMips). Warning : 98.36% of all statistics are fiction.
.

****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

#2 lore raw

Hello,

Quoted 8 lines

This is how the kernel thread is stopped.

     if(info->pid)
     {
         (void)kill_proc(info->pid, SIGTERM, 1);
         wait_for_completion(&info->quit);
     }

I actually would prefer to do it with the new kernel thread API.
So, to create the thread: kthread_create

bind it to a cpu:                  kthread_bind
stop it:                                kthread_stop

Now, if I call kthread_stop() in module unload time, does that code run in user process context just like system calls do? That is important, because if it cannot sleep, it would deadlock.

#3 lore raw
Quoted 11 lines · Aritz Bastida

Hello,

Quoted 8 lines

This is how the kernel thread is stopped.

     if(info->pid)
     {
         (void)kill_proc(info->pid, SIGTERM, 1);
         wait_for_completion(&info->quit);
     }
Quoted 5 lines

I actually would prefer to do it with the new kernel thread API.
So, to create the thread: kthread_create

bind it to a cpu:                  kthread_bind
stop it:                                kthread_stop

Sure your original question was about how to properly stop a kernel thread prior to module removal. This, I showed you. Now you say you "prefer" to do it some other way. Good luck. Unless your code runs exactly like the prototype that the new macros were written for, you may have fix the macros.

Quoted 4 lines

Now, if I call kthread_stop() in module unload time, does that code run in user process context just like system calls do? That is important, because if it cannot sleep, it would deadlock.

Not relevent. You have apparently made up your mind that you need to do it "your" way. Fine.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13.4 on an i686 machine (5589.55 BogoMips). Warning : 98.36% of all statistics are fiction.
.

****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

#4 lore raw
Quoted 8 lines
Quoted 5 lines

Now, if I call kthread_stop() in module unload time, does that code run in user process context just like system calls do? That is important, because if it cannot sleep, it would deadlock.

Not relevent. You have apparently made up your mind that you need to do it "your" way. Fine.

No, no, that is not MY way. That is the 2.6 way.
(see http://lwn.net/Articles/64444/).

With the old API (kernel_thread) there was no way (at least, direct way) to bind a thread to a cpu, which i actually need. And this is not MY own approach , it is used throught the kernel:

* migration thread at sched.c:

           http://lxr.linux.no/source/kernel/sched.c#L4196

* ksoftirqd thread at softirq.c:

           http://lxr.linux.no/source/kernel/softirq.c#L350

* worker thread used in workqueues:

           http://lxr.linux.no/source/kernel/workqueue.c#L182

...and so on

All these threads are standard and well debugged, and all use the API which I was talking about. Of course, as they are included in the official kernel sources, none of them is stopped at module unload time.

Thus, all I want to know is why kthread_stop() cant be called at module unload time, if the cleanup code can sleep and if there is a workaround for that.

I hope my questions are clearer now.
Thanks for your help

Aritz

#5 lore raw
Quoted 32 lines · Aritz Bastida
Quoted 8 lines
Quoted 5 lines

Now, if I call kthread_stop() in module unload time, does that code run in user process context just like system calls do? That is important, because if it cannot sleep, it would deadlock.

Not relevent. You have apparently made up your mind that you need to do it "your" way. Fine.

No, no, that is not MY way. That is the 2.6 way.
(see http://lwn.net/Articles/64444/).

With the old API (kernel_thread) there was no way (at least, direct way) to bind a thread to a cpu, which i actually need. And this is not MY own approach , it is used throught the kernel:

* migration thread at sched.c:

          http://lxr.linux.no/source/kernel/sched.c#L4196

* ksoftirqd thread at softirq.c:

          http://lxr.linux.no/source/kernel/softirq.c#L350

* worker thread used in workqueues:

          http://lxr.linux.no/source/kernel/workqueue.c#L182

...and so on

All these threads are standard and well debugged, and all use the API which I was talking about. Of course, as they are included in the official kernel sources, none of them is stopped at module unload time.

Quoted 6 lines

Thus, all I want to know is why kthread_stop() cant be called at module unload time, if the cleanup code can sleep and if there is a workaround for that.

I hope my questions are clearer now.
Thanks for your help

Then if you are going to use the macros to bind to a certain CPU when you start up a kernel thread, then you need to use the complimenting macros to run them down.

You need to use the for_each_cpu() stuff to pick up each thread. Then you may actually get to talk to the right process.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13.4 on an i686 machine (5589.55 BogoMips). Warning : 98.36% of all statistics are fiction.
.

****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

#6 lore raw
In reply to Aritz Bastida
Quoted 3 lines · Aritz Bastida

Now, if I call kthread_stop() in module unload time, does that code run in user process context just like system calls do? That is important, because if it cannot sleep, it would deadlock.

Yes, it runs in the context of the delete_module system call. I think it's more likely that you're not returning from your thread loop.

Please post a URL for your module source code so we can see what goes wrong there.

	Arnd <><
#7 lore raw

2005/11/9, Arnd Bergmann <[email protected]>:

Quoted 5 lines · Aritz Bastida
Quoted 3 lines

Now, if I call kthread_stop() in module unload time, does that code run in user process context just like system calls do? That is important, because if it cannot sleep, it would deadlock.

Quoted 8 lines

Yes, it runs in the context of the delete_module system call. I think it's more likely that you're not returning from your thread loop.

Please post a URL for your module source code so we can see what goes wrong there.

        Arnd <><

Than you very much Arnd!
You solved my problem. hehe

I began to write a test module for showing you this and have just realized about the problem. As I create as many threads as CPUs, I have to delete them all when finishing.

I killed them like this:

        /* We don't need the distraction of CPUs appearing and vanishing. */
        lock_cpu_hotplug();
        for_each_online_cpu(cpu) {
                p = per_cpu(ksensord_info, cpu);
                kthread_stop(p);
        }
        unlock_cpu_hotplug();

I locked the cpu hotplug lock to protect the for_each_online_cpu() code in case a cpu appears/vanishes, so I am actually calling kthread_stop() in an atomic context, so it wakes up the process, but dont let it run!

This is quite a subtle error, but of course it's my complete fault :P May be a BUG_ON(in_atomic()) within kthread_stop() would let this kind of errors be acknowledged more easily.

Thank you for your help
Regards

Aritz Bastida

#8 lore raw

Thank you very much Arnd!
You solved my problem. hehe

I began to write a test module for showing you this and have just realized about the problem. As I create as many threads as CPUs, I have to delete them all when finishing.

I killed them like this:

        /* We don't need the distraction of CPUs appearing and vanishing. */
        lock_cpu_hotplug();
        for_each_online_cpu(cpu) {
                p = per_cpu(ksensord_info, cpu);
                kthread_stop(p);
        }
        unlock_cpu_hotplug();

I locked the cpu hotplug lock to protect the for_each_online_cpu() code in case a cpu appears/vanishes, so I am actually calling kthread_stop() in an atomic context, so it wakes up the process, but dont let it run!

This is quite a subtle error, but of course it's my complete fault :P May be a BUG_ON(in_atomic()) within kthread_stop() would let this kind of errors be acknowledged more easily.

Thank you for your help
Regards

Aritz Bastida

PS: Sorry Arnd, I forgot adding the linux-kernel address to the message

#9 lore raw
In reply to Aritz Bastida
Quoted 37 lines · Aritz Bastida

2005/11/9, Arnd Bergmann <[email protected]>:

Quoted 5 lines · Aritz Bastida
Quoted 3 lines

Now, if I call kthread_stop() in module unload time, does that code run in user process context just like system calls do? That is important, because if it cannot sleep, it would deadlock.

Quoted 8 lines

Yes, it runs in the context of the delete_module system call. I think it's more likely that you're not returning from your thread loop.

Please post a URL for your module source code so we can see what goes wrong there.

        Arnd <><

Than you very much Arnd!
You solved my problem. hehe

I began to write a test module for showing you this and have just realized about the problem. As I create as many threads as CPUs, I have to delete them all when finishing.

I killed them like this:

       /* We don't need the distraction of CPUs appearing and vanishing. */
       lock_cpu_hotplug();
       for_each_online_cpu(cpu) {
               p = per_cpu(ksensord_info, cpu);
               kthread_stop(p);
       }
       unlock_cpu_hotplug();

I locked the cpu hotplug lock to protect the for_each_online_cpu() code in case a cpu appears/vanishes, so I am actually calling kthread_stop() in an atomic context, so it wakes up the process, but dont let it run!

Quoted 8 lines

This is quite a subtle error, but of course it's my complete fault :P May be a BUG_ON(in_atomic()) within kthread_stop() would let this kind of errors be acknowledged more easily.

Thank you for your help
Regards

Aritz Bastida

Ah, yes.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13.4 on an i686 machine (5589.55 BogoMips). Warning : 98.36% of all statistics are fiction.
.

****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.