Stopping Kernel Threads at module unload time
Hello
I've got some questions about kernel threads.I am writing a module which spawns some kernel threads, which would be removed when the module unloads. For that purpose i call kthread_stop() at module unload time. When issuing rmmod on the module, it deadlocks at that point (in the call to kthread_stop), and never returns.
In the thread main function the code was something like this (it's of course simplified).
thread_main()
{
while( ! kthread_should_stop())
{
.............
wait_event_interruptible(stop_wq, kthread_should_stop() );
}
return 0;
}So if kthread_stop() first sets the thread "closing flag", and then calls wake_up_process(), the thread should wake up, see he should stop, and
end the loop. That doesnt actually never happen.
I have also tried what it is done in kernel/sched.c to finish:
/* wait for kthread_stop */
set_current_state(TASK_INTERRUPTIBLE);
while (!kthread_should_stop()) {
schedule();
set_current_state(TASK_INTERRUPTIBLE);
}
__set_current_state(TASK_RUNNING);
return 0;I have ensured it actually arrives to that point by using printks, but when the thread goes to sleep, it does never wake up again, so the call to kthread_stop() lasts forever.
I dont know why this happens. Is the module cleanup code in the context of a user process just like system calls? Can that code sleep? If it can't sleep then the answer would be quite easy: kthread_stop() wakes up the processes and then waits for the threads to finish (on call wait_for_completion), but doesnt actually let them execute, because it cannot sleep, so it deadlocks.
So I would be grateful if anyone can help me in this matter. Regards
Aritz
Quoted 49 lines · Aritz Bastida
Hello
I've got some questions about kernel threads.I am writing a module which spawns some kernel threads, which would be removed when the module unloads. For that purpose i call
kthread_stop()at module unload time. When issuing rmmod on the module, it deadlocks at that point (in the call tokthread_stop), and never returns.In the thread main function the code was something like this (it's of course simplified).
thread_main(){ while( ! kthread_should_stop()) { ............. wait_event_interruptible(stop_wq, kthread_should_stop() ); } return 0; }So if
kthread_stop()first sets the thread "closing flag", and then callswake_up_process(), the thread should wake up, see he should stop, and
end the loop. That doesnt actually never happen.I have also tried what it is done in
kernel/sched.cto finish:/* wait for kthread_stop */ set_current_state(TASK_INTERRUPTIBLE); while (!kthread_should_stop()) { schedule(); set_current_state(TASK_INTERRUPTIBLE); } __set_current_state(TASK_RUNNING); return 0;I have ensured it actually arrives to that point by using printks, but when the thread goes to sleep, it does never wake up again, so the call to
kthread_stop()lasts forever.I dont know why this happens. Is the module cleanup code in the context of a user process just like system calls? Can that code sleep? If it can't sleep then the answer would be quite easy:
kthread_stop()wakes up the processes and then waits for the threads to finish (on callwait_for_completion), but doesnt actually let them execute, because it cannot sleep, so it deadlocks.
Quoted 4 lines
So I would be grateful if anyone can help me in this matter. Regards
Aritz
Here is code that actually works. There have been some kernel API changes that convert some of this stuff to macros. Nevertheless. This stuff works.
void make_daemon(const char *dev, int32_t prior)
{
#ifdef OLD_DAEMON // kernel 2.4.x
exit_files(current);
daemonize();
spin_lock_irq(¤t->sigmask_lock);
sigemptyset(¤t->blocked);
sigdelset(¤t->blocked, SIGTERM);
recalc_sigpending(current);
spin_unlock_irq(¤t->sigmask_lock);
task_lock(current);
current->nice = prior;
strcpy(current->comm, dev);
task_unlock(current);
#else // kernel 2.6.x
daemonize("%s", dev);
allow_signal(SIGTERM);
set_user_nice(current, prior);#endif
//
// Destroy process privs just in case somebody is able to hack this // into a shell.
//
task_lock(current);
current->uid = current->gid = current->sgid = current->fsgid =
current->euid = current->egid = current->suid = current->fsuid = 666;
task_unlock(current);
}Note above that the task is already running inside the kernel and has access to all your data. You don't need it to be 'root' so that possible back-door gets closed.
//-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= //
// This is the kernel thread. It handles things that should not
// be handled in an interrupt. It is awakened from the interrupt
// service routine.
//
static int32_t local_thread(void *unused)
{
make_daemon(task_name, PRIOR);
set_current_state(TASK_INTERRUPTIBLE); // For the following sleep
schedule_timeout(1); // Let insmod complete
set_current_state(TASK_RUNNING); // Remember to restore
__asm__ __volatile__("thread:\n.global thread\n");
for(;;) // Do forever.......
{
set_current_state(TASK_INTERRUPTIBLE);
if(signal_pending(current))
complete_and_exit(&info->quit, 0); // This is how you get out
interruptible_sleep_on(&info->twait); // Sleep until something to do
set_current_state(TASK_RUNNING);
do_stuff(.....); // Whatever the thread needs to do....
}
return 0;
}This is how the kernel thread is started.
info->pid = kernel_thread(local_thread, NULL, CLONE_FS|CLONE_FILES);
This is how the kernel thread is stopped.
if(info->pid)
{
(void)kill_proc(info->pid, SIGTERM, 1);
wait_for_completion(&info->quit);
}Cheers,
Dick Johnson
Penguin : Linux version 2.6.13.4 on an i686 machine (5589.55 BogoMips). Warning : 98.36% of all statistics are fiction.
.
****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.
Thank you.
Hello,
Quoted 8 lines
This is how the kernel thread is stopped.
if(info->pid) { (void)kill_proc(info->pid, SIGTERM, 1); wait_for_completion(&info->quit); }
I actually would prefer to do it with the new kernel thread API.
So, to create the thread: kthread_create
bind it to a cpu: kthread_bind stop it: kthread_stop
Now, if I call kthread_stop() in module unload time, does that code run in user process context just like system calls do? That is important, because if it cannot sleep, it would deadlock.
Quoted 11 lines · Aritz Bastida
Hello,
Quoted 8 lines
This is how the kernel thread is stopped.
if(info->pid) { (void)kill_proc(info->pid, SIGTERM, 1); wait_for_completion(&info->quit); }
Quoted 5 lines
I actually would prefer to do it with the new kernel thread API.
So, to create the thread:kthread_createbind it to a cpu: kthread_bind stop it: kthread_stop
Sure your original question was about how to properly stop a kernel thread prior to module removal. This, I showed you. Now you say you "prefer" to do it some other way. Good luck. Unless your code runs exactly like the prototype that the new macros were written for, you may have fix the macros.
Quoted 4 lines
Now, if I call
kthread_stop()in module unload time, does that code run in user process context just like system calls do? That is important, because if it cannot sleep, it would deadlock.
Not relevent. You have apparently made up your mind that you need to do it "your" way. Fine.
Cheers,
Dick Johnson
Penguin : Linux version 2.6.13.4 on an i686 machine (5589.55 BogoMips). Warning : 98.36% of all statistics are fiction.
.
****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.
Thank you.
Quoted 8 lines
Quoted 5 lines
Now, if I call
kthread_stop()in module unload time, does that code run in user process context just like system calls do? That is important, because if it cannot sleep, it would deadlock.Not relevent. You have apparently made up your mind that you need to do it "your" way. Fine.
No, no, that is not MY way. That is the 2.6 way.
(see http://lwn.net/Articles/64444/).
With the old API (kernel_thread) there was no way (at least, direct way) to bind a thread to a cpu, which i actually need. And this is not MY own approach , it is used throught the kernel:
* migration thread at sched.c:
http://lxr.linux.no/source/kernel/sched.c#L4196
* ksoftirqd thread at softirq.c:
http://lxr.linux.no/source/kernel/softirq.c#L350
* worker thread used in workqueues:
http://lxr.linux.no/source/kernel/workqueue.c#L182
...and so on
All these threads are standard and well debugged, and all use the API which I was talking about. Of course, as they are included in the official kernel sources, none of them is stopped at module unload time.
Thus, all I want to know is why kthread_stop() cant be called at module unload time, if the cleanup code can sleep and if there is a workaround for that.
I hope my questions are clearer now.
Thanks for your help
Aritz
Quoted 32 lines · Aritz Bastida
Quoted 8 lines
Quoted 5 lines
Now, if I call
kthread_stop()in module unload time, does that code run in user process context just like system calls do? That is important, because if it cannot sleep, it would deadlock.Not relevent. You have apparently made up your mind that you need to do it "your" way. Fine.
No, no, that is not MY way. That is the 2.6 way.
(see http://lwn.net/Articles/64444/).With the old API (
kernel_thread) there was no way (at least, direct way) to bind a thread to a cpu, which i actually need. And this is not MY own approach , it is used throught the kernel:* migration thread at
sched.c:http://lxr.linux.no/source/kernel/sched.c#L4196* ksoftirqd thread at
softirq.c:http://lxr.linux.no/source/kernel/softirq.c#L350* worker thread used in workqueues:
http://lxr.linux.no/source/kernel/workqueue.c#L182...and so on
All these threads are standard and well debugged, and all use the API which I was talking about. Of course, as they are included in the official kernel sources, none of them is stopped at module unload time.
Quoted 6 lines
Thus, all I want to know is why
kthread_stop()cant be called at module unload time, if the cleanup code can sleep and if there is a workaround for that.I hope my questions are clearer now.
Thanks for your help
Then if you are going to use the macros to bind to a certain CPU when you start up a kernel thread, then you need to use the complimenting macros to run them down.
You need to use the for_each_cpu() stuff to pick up each thread. Then you may actually get to talk to the right process.
Cheers,
Dick Johnson
Penguin : Linux version 2.6.13.4 on an i686 machine (5589.55 BogoMips). Warning : 98.36% of all statistics are fiction.
.
****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.
Thank you.
Quoted 3 lines · Aritz Bastida
Now, if I call
kthread_stop()in module unload time, does that code run in user process context just like system calls do? That is important, because if it cannot sleep, it would deadlock.
Yes, it runs in the context of the delete_module system call. I think it's more likely that you're not returning from your thread loop.
Please post a URL for your module source code so we can see what goes wrong there.
Arnd <><
2005/11/9, Arnd Bergmann <[email protected]>:
Quoted 5 lines · Aritz Bastida
Quoted 3 lines
Now, if I call
kthread_stop()in module unload time, does that code run in user process context just like system calls do? That is important, because if it cannot sleep, it would deadlock.
Quoted 8 lines
Yes, it runs in the context of the
delete_modulesystem call. I think it's more likely that you're not returning from your thread loop.Please post a URL for your module source code so we can see what goes wrong there.
Arnd <><
Than you very much Arnd!
You solved my problem. hehe
I began to write a test module for showing you this and have just realized about the problem. As I create as many threads as CPUs, I have to delete them all when finishing.
I killed them like this:
/* We don't need the distraction of CPUs appearing and vanishing. */
lock_cpu_hotplug();
for_each_online_cpu(cpu) {
p = per_cpu(ksensord_info, cpu);
kthread_stop(p);
}
unlock_cpu_hotplug();I locked the cpu hotplug lock to protect the for_each_online_cpu() code in case a cpu appears/vanishes, so I am actually calling kthread_stop() in an atomic context, so it wakes up the process, but dont let it run!
This is quite a subtle error, but of course it's my complete fault :P May be a BUG_ON(in_atomic()) within kthread_stop() would let this kind of errors be acknowledged more easily.
Thank you for your help
Regards
Aritz Bastida
Thank you very much Arnd!
You solved my problem. hehe
I began to write a test module for showing you this and have just realized about the problem. As I create as many threads as CPUs, I have to delete them all when finishing.
I killed them like this:
/* We don't need the distraction of CPUs appearing and vanishing. */
lock_cpu_hotplug();
for_each_online_cpu(cpu) {
p = per_cpu(ksensord_info, cpu);
kthread_stop(p);
}
unlock_cpu_hotplug();I locked the cpu hotplug lock to protect the for_each_online_cpu() code in case a cpu appears/vanishes, so I am actually calling kthread_stop() in an atomic context, so it wakes up the process, but dont let it run!
This is quite a subtle error, but of course it's my complete fault :P May be a BUG_ON(in_atomic()) within kthread_stop() would let this kind of errors be acknowledged more easily.
Thank you for your help
Regards
Aritz Bastida
PS: Sorry Arnd, I forgot adding the linux-kernel address to the message
Quoted 37 lines · Aritz Bastida
2005/11/9, Arnd Bergmann <[email protected]>:
Quoted 5 lines · Aritz Bastida
Quoted 3 lines
Now, if I call
kthread_stop()in module unload time, does that code run in user process context just like system calls do? That is important, because if it cannot sleep, it would deadlock.Quoted 8 lines
Yes, it runs in the context of the
delete_modulesystem call. I think it's more likely that you're not returning from your thread loop.Please post a URL for your module source code so we can see what goes wrong there.
Arnd <><Than you very much Arnd!
You solved my problem. heheI began to write a test module for showing you this and have just realized about the problem. As I create as many threads as CPUs, I have to delete them all when finishing.
I killed them like this:
/* We don't need the distraction of CPUs appearing and vanishing. */ lock_cpu_hotplug(); for_each_online_cpu(cpu) { p = per_cpu(ksensord_info, cpu); kthread_stop(p); } unlock_cpu_hotplug();I locked the cpu hotplug lock to protect the
for_each_online_cpu()code in case a cpu appears/vanishes, so I am actually callingkthread_stop()in an atomic context, so it wakes up the process, but dont let it run!
Quoted 8 lines
This is quite a subtle error, but of course it's my complete fault :P May be a BUG_ON(in_atomic()) within
kthread_stop()would let this kind of errors be acknowledged more easily.Thank you for your help
RegardsAritz Bastida
Ah, yes.
Cheers,
Dick Johnson
Penguin : Linux version 2.6.13.4 on an i686 machine (5589.55 BogoMips). Warning : 98.36% of all statistics are fiction.
.
****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.
Thank you.