Skip to contentSkip to conversation

Gracefully killing kswapd, or any kernel thread

DISCUSSION Kristis Makris 3 participants 8 messages Sep 7, 2005 – Sep 9, 2005

lore
Share & tools
lore
#0 lore raw

Hello,

I'm trying to kill a kernel thread gracefully, in particular kswapd, without any success.

The goal is to start another kernel thread that contains updated kswapd functionality, through a loadable module; no kernel recompilation.

I noticed that kernel threads block SIGKILL. Hence, on module load I'm running:

task = find_task_by_name("kswapd");

if (task != NULL) {
    spin_lock_irq(&task->sigmask_lock);
    sigdelset(&task->blocked, SIGKILL);
    recalc_sigpending(task);
    spin_unlock_irq(&task->sigmask_lock);
    // Also tried issuing here a: kill_proc(task->pid, SIGKILL, 1);
}

Then from userspace I issue:

# ps aux |grep -i swap
root         4  0.0  0.0     0    0 ?        SW   18:36   0:00 [kswapd]
$ kill -9 4

After the kill is issued, kswapd is taking up 99.9% of CPU time and remains at a runnable state:

# ps aux |grep -i swap
root         4  0.2  0.0     0    0 ?        RW   18:36   0:02 [kswapd]

Can anyone explain why this is happening ? I've tried this with linux kernels 2.2.19 and 2.4.27 (with patch kdb-4.3). What is the proper way of gracefully killing a kernel thread launched from the original kernel image (not a module) in kernels < 2.6 (ie. without the new kernel thread API that contains the stop_kthread call documented in
http://www.scs.ch/~frey/linux/kernelthreads.html)

I've also tried the same with kflushd, kupdate, and keventd in 2.2.19. When I do issue a "kill -9" for them I see:

# ps aux
USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root         2  0.0  0.0     0    0 ?        SW   12:18   0:00 [kflushd]
root         3  1.5  0.0     0    0 ?        RW   12:18   0:16 [kupdate]
root         5  0.0  0.0     0    0 ?        SW   12:18   0:00 [keventd]

All 3 kernel threads remain in the process list. kupdate also appears to be in a running state consuming 99.9% of the CPU when killed. What's so special about kupdate and kswapd that makes them stay at a running state, and why do all kernel threads seem unkillable?

Thanks,
Kristis

#1 lore raw
Quoted 48 lines · Kristis Makris

Hello,

I'm trying to kill a kernel thread gracefully, in particular kswapd, without any success.

The goal is to start another kernel thread that contains updated kswapd functionality, through a loadable module; no kernel recompilation.

I noticed that kernel threads block SIGKILL. Hence, on module load I'm running:

task = find_task_by_name("kswapd");

if (task != NULL) {
   spin_lock_irq(&task->sigmask_lock);
   sigdelset(&task->blocked, SIGKILL);
   recalc_sigpending(task);
   spin_unlock_irq(&task->sigmask_lock);
   // Also tried issuing here a: kill_proc(task->pid, SIGKILL, 1);
}

Then from userspace I issue:

# ps aux |grep -i swap
root         4  0.0  0.0     0    0 ?        SW   18:36   0:00 [kswapd]
$ kill -9 4

After the kill is issued, kswapd is taking up 99.9% of CPU time and remains at a runnable state:

# ps aux |grep -i swap
root         4  0.2  0.0     0    0 ?        RW   18:36   0:02 [kswapd]

Can anyone explain why this is happening ? I've tried this with linux kernels 2.2.19 and 2.4.27 (with patch kdb-4.3). What is the proper way of gracefully killing a kernel thread launched from the original kernel image (not a module) in kernels < 2.6 (ie. without the new kernel thread API that contains the stop_kthread call documented in
http://www.scs.ch/~frey/linux/kernelthreads.html)

I've also tried the same with kflushd, kupdate, and keventd in 2.2.19. When I do issue a "kill -9" for them I see:

# ps aux
USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root         2  0.0  0.0     0    0 ?        SW   12:18   0:00 [kflushd]
root         3  1.5  0.0     0    0 ?        RW   12:18   0:16 [kupdate]
root         5  0.0  0.0     0    0 ?        SW   12:18   0:00 [keventd]
Quoted 7 lines

All 3 kernel threads remain in the process list. kupdate also appears to be in a running state consuming 99.9% of the CPU when killed. What's so special about kupdate and kswapd that makes them stay at a running state, and why do all kernel threads seem unkillable?

Thanks,
Kristis

To kill a kernel thread, you need to make __it__ call exit(). It must be CODED to do that! You can't do it externally although you can send it a signal, after which it will spin forever....

// Main loop of the thread

     for(;;)
     {
         set_current_state(TASK_INTERRUPTIBLE);
         if(signal_pending(current))
             complete_and_exit(&info->quit, 0);  // <--- HERE!!!
         interruptible_sleep_on(&info->twait);
         do_work();
     }

// Start a thread

     info->pid = kernel_thread(local_thread, NULL, CLONE_FS|CLONE_FILES);

// Stop a thread

     if(info->pid)
     {
         (void)kill_proc(info->pid, SIGTERM, 1);
         wait_for_completion(&info->quit);   // MUST wait!
     }

With newer kernels, we use the allow_signal(SIGNAME) macro as one of the first things executed to allow the kernel thread to respond to a signal.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.51 BogoMips). Warning : 98.36% of all statistics are fiction.
.
I apologize for the following. I tried to kill it with the above dot :

****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

#2 lore raw
Quoted 1 line

To kill a kernel thread, you need to make __it__ call exit(). It must be

There must be another way to do it. Perhaps one could have another process effectively issue the contents of do_exit for the kswapd task_struct ?

Quoted 1 line

CODED to do that! You can't do it externally although you can send

I'm clearly asking for the case where the thread wasn't coded to do that.

Quoted 1 line

it a signal, after which it will spin forever....

kflushd and keventd don't seem to spin forever. I still haven't determined what makes kswapd spin forever after it receives the signal.

#3 lore raw
Quoted 5 lines · Kristis Makris
Quoted 1 line

To kill a kernel thread, you need to make __it__ call exit(). It must be

There must be another way to do it. Perhaps one could have another process effectively issue the contents of do_exit for the kswapd task_struct ?

Umm, so then the kernel does what, exactly? You have a process in some indeterminate state, possibly holding semaphores, definitely pinning memory/resources/etc, and you just stop it, turn it off, and expect things to continue working? This is similar in nature to that thread a while ago about kernel error recovery and killing uninterruptible user processes. To extend this to kernel threads, unless the kernel thread has been _specifically_ coded to be interruptible, it isn't, and furthermore, *can't* be.

Quoted 4 lines
Quoted 1 line

CODED to do that! You can't do it externally although you can send

I'm clearly asking for the case where the thread wasn't coded to do that.

You can't. This is flatly impossible. Go see the thread a while back about a hot-patch system call for several reasons why that is a bad idea. In particular, look at the post that discusses phone switches, the one with the quote "'So why don't you just reboot the affected switches?' [...] 'That assumes the switches had ever been booted in the first place'".

Quoted 4 lines
Quoted 1 line

it a signal, after which it will spin forever....

kflushd and keventd don't seem to spin forever. I still haven't determined what makes kswapd spin forever after it receives the signal.

Probably a while(1) loop that isn't intended to stop until the machine physically powers off. If you want to patch one specific kernel thread, you might be able to do that, but you can't just expect to hot-patch random parts of the kernel at runtime and have things work.

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r !y?(-) ------END GEEK CODE BLOCK------

#4 lore raw
In reply to Kristis Makris
Quoted 11 lines · Kristis Makris
Quoted 1 line

To kill a kernel thread, you need to make __it__ call exit(). It must be

There must be another way to do it. Perhaps one could have another process effectively issue the contents of do_exit for the kswapd task_struct ?

Quoted 1 line

CODED to do that! You can't do it externally although you can send

I'm clearly asking for the case where the thread wasn't coded to do that.

It was not clear. You just sent it a signal and expected it to go away. No tasks, even user-mode tasks, just go away! They need to execute the exit code in the context of the task that will expected to eventually disappear. It's contents of "current" that tells the exit code what to clean up.

This means that you could probably install a driver that patches some code that the kernel thread executes. Nothing like modifying code at run-time....

Quoted 5 lines
Quoted 1 line

it a signal, after which it will spin forever....

kflushd and keventd don't seem to spin forever. I still haven't determined what makes kswapd spin forever after it receives the signal.

It depends how the signal was masked and how the thread was written. Notice in the example posted, the code was reset to TASK_INTERRUPTIBLE each time the loop was executed. If this is not done, the task won't get interrupted by a signal again.

Quoted 2 lines

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.51 BogoMips). Warning : 98.36% of all statistics are fiction.
.
I apologize for the following. I tried to kill it with the above dot :

****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

#5 lore raw
Quoted 3 lines · linux-os (Dick Johnson)
Quoted 1 line · Kristis Makris
Quoted 1 line

To kill a kernel thread, you need to make __it__ call exit(). It must be

I was able to make it call do_exit(). However, even if I recompile a kernel to have kswapd issued a do_exit(), I still see from ps:

root 4 0.0 0.0 0 0 ? Z 06:20 0:00 [kswapd
<defunct>]

Why isn't the task_struct for it gone ?

#6 lore raw
Quoted 5 lines · Kristis Makris
Quoted 3 lines · linux-os (Dick Johnson)
Quoted 1 line · Kristis Makris
Quoted 1 line

To kill a kernel thread, you need to make __it__ call exit(). It must be

Quoted 8 lines

I was able to make it call do_exit(). However, even if I recompile a kernel to have kswapd issued a do_exit(), I still see from ps:

root 4 0.0 0.0 0 0 ? Z 06:20 0:00 [kswapd
<defunct>]

Why isn't the task_struct for it gone ?

Because it's now defunct <Z>, a zombie, waiting for somebody to reap its return status. You are almost there, you need to issue the kernel equivalent of waitpid() (sys_waitpid) to grab that status and throw it away. That's what the code I showed you did when it would shut down and remove a module that had a kernel thread.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.53 BogoMips). Warning : 98.36% of all statistics are fiction.
.
I apologize for the following. I tried to kill it with the above dot :

****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

#7 lore raw
Quoted 6 lines · linux-os (Dick Johnson)

Because it's now defunct <Z>, a zombie, waiting for somebody to reap its return status. You are almost there, you need to issue the kernel equivalent of waitpid() (sys_waitpid) to grab that status and throw it away. That's what the code I showed you did when it would shut down and remove a module that had a kernel thread.

It turns out I had to port reparent_to_init() from 2.4 to 2.2 to get it going. issuing a sys_wait4 wasn't enough. Apparently there was no way for the built-in kernel threads to exit at all in the first place. I hope 2.6 at least issues a reparent_to_init in kernel_thread().