Softlockup (out of cpu) killer
Subsystems (4)
PATCH 1/1 Softlockup (out of cpu) killer
In kernel, there is out of memory (OOM) killer, why not make an out of cpu (OOC) killer? I tested following patch by running an user-space cpu hogging process and the softlockukp detector killed the process successfully.
Softlockup could be caused by user-space process hogging cpu, add softlockup_kill kernel config to allow kernel to kill the user space cpu hogging process. this feature is useful for high availability systems that have uptime gurantees and where a softlockup must be resolved ASAP
echo 1 > /proc/sys/kernel/softlockukp_kill to enable cpu hog process killer echo 0 > /proc/sys/kernel/softlockup_kill to disable cpu hog process killer
--- Documentation/kernel-parameters.txt | 4 ++++ include/linux/sched.h | 1 + kernel/sysctl.c | 9 +++++++++ kernel/watchdog.c | 18 ++++++++++++++++++ lib/Kconfig.debug | 21 +++++++++++++++++++++ 5 files changed, 53 insertions(+), 0 deletions(-)
Documentation/kernel-parameters.txt +4 -0
@@ -2418,6 +2418,10 @@ bytes respectively. Such letter suffixes can also be entirely omitted. [KNL] Should the soft-lockup detector generate panics. Format: <integer> + softlockup_panic= + [KNL] Should the soft-lockup detector kill cpu hog process. + Format: <integer> + sonypi.*= [HW] Sony Programmable I/O Control Device driver See Documentation/laptops/sonypi.txt
include/linux/sched.h +1 -0
@@ -315,6 +315,7 @@ extern int proc_dowatchdog_thresh(struct ctl_table *table, int write,void __user *buffer, size_t *lenp, loff_t *ppos);extern unsigned int softlockup_panic; +extern unsigned int softlockup_kill;void lockup_detector_init (void );#else static inline void touch_softlockup_watchdog (void )
kernel/sysctl.c +9 -0
@@ -770,6 +770,15 @@ static struct ctl_table kern_table[] = { .extra2 = &one, }, { + .procname = "softlockup_kill", + .data = &softlockup_kill, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + .extra2 = &one, + }, + { .procname = "nmi_watchdog", .data = &watchdog_enabled, .maxlen = sizeof (int),
kernel/watchdog.c +18 -0
@@ -75,6 +75,17 @@ static int __init softlockup_panic_setup(char *str) }__setup ("softlockup_panic=" , softlockup_panic_setup); +unsigned int __read_mostly softlockup_kill = +CONFIG_BOOTPARAM_SOFTLOCKUP_KILL_VALUE ; + +static int __init softlockup_kill_setup (char *str) +{ + softlockup_kill =simple_strtoul (str,NULL ,0 ); + +return 1 ; +} +__setup ("softlockup_kill=" , softlockup_kill_setup); +static int __init nowatchdog_setup (char *str) { watchdog_enabled =0 ;
@@ -306,6 +317,13 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)else dump_stack (); +if (softlockup_kill) { +printk (KERN_ERR "Kill softlockup process [%s:%d] on CPU#%d\n" , + current->comm,task_pid_nr (current), +smp_processor_id ()); +force_sig (SIGKILL , current); + } +if (softlockup_panic)panic ("softlockup: hung tasks" );__this_cpu_write (soft_watchdog_warn, true);
lib/Kconfig.debug +21 -0
@@ -224,6 +224,27 @@ config BOOTPARAM_SOFTLOCKUP_PANIC_VALUE default 0 if !BOOTPARAM_SOFTLOCKUP_PANIC default 1 if BOOTPARAM_SOFTLOCKUP_PANIC +config BOOTPARAM_SOFTLOCKUP_KILL + bool "Kill (cpu hog process) On Soft Lockups" + depends on LOCKUP_DETECTOR + help + Say Y here to enable the kernel to kill cpu hog process on + "soft lockups", which are bugs that cause the kernel to + loop in kernel mode for more than 60 seconds, without giving + other tasks a chance to run. + + This feature is useful for high-availability systems that + have uptime guarantees and where a lockup must be resolved ASAP. + + Say N if unsure. + +config BOOTPARAM_SOFTLOCKUP_KILL_VALUE + int + depends on LOCKUP_DETECTOR + range 0 1 + default 0 if !BOOTPARAM_SOFTLOCKUP_KILL + default 1 if BOOTPARAM_SOFTLOCKUP_KILL + config DETECT_HUNG_TASK bool "Detect Hung Tasks" depends on DEBUG_KERNEL
Signature · 1 line
1.7.0.4
Quoted 11 lines · Vincent Li
In kernel, there is out of memory (OOM) killer, why not make an out of cpu (OOC) killer? I tested following patch by running an user-space cpu hogging process and the softlockukp detector killed the process successfully.
Softlockup could be caused by user-space process hogging cpu, add
softlockup_killkernel config to allow kernel to kill the user space cpu hogging process. this feature is useful for high availability systems that have uptime gurantees and where a softlockup must be resolved ASAPecho 1 >
/proc/sys/kernel/softlockukp_killto enable cpu hog process killer echo 0 >/proc/sys/kernel/softlockup_killto disable cpu hog process killer
That assumes a signal would be enough to pull a process out of its softlockup. I believe this is seldom the case. A process in a softlockup is stuck in some place that has preemption disabled. Unless it luckily polls there for pending signals, that won't work.
But may be that happens more often than I think. May be other people have more insight.
Quoted 9 lines · Vincent Li
In kernel, there is out of memory (OOM) killer, why not make an out of cpu (OOC) killer? I tested following patch by running an user-space cpu hogging process and the softlockukp detector killed the process successfully.
Softlockup could be caused by user-space process hogging cpu, add
softlockup_killkernel config to allow kernel to kill the user space cpu hogging process. this feature is useful for high availability systems that have uptime gurantees and where a softlockup must be resolved ASAP
Quoted 4 lines
echo 1 >
/proc/sys/kernel/softlockukp_killto enable cpu hog process killer echo 0 >/proc/sys/kernel/softlockup_killto disable cpu hog process killerSigned-off-by Vincent Li <[email protected]>
Your whole premise is broken. Being a cpu hog and the softlockup mechanism aren't related at all.
Furthermore, since the normal scheduling policy is a proportional one, a cpu hog can't in fact starve anybody (although a fork bomb could). And FIFO/RR are privileged ops.
Furthermore the distinction between memory and cpu-time is that memory isn't a renewable resource, whereas time is. There's always more time, but there's not always more memory.
So no, I don't think either you patch nor your concept make any sense. Consider it nacked.
Quoted 4 lines
Your whole premise is broken. Being a cpu hog and the softlockup mechanism aren't related at all.
I fully understand that I may misunderstand the the cpu hog and softlockup mechanism :)
Quoted 4 lines
Furthermore, since the normal scheduling policy is a proportional one, a cpu hog can't in fact starve anybody (although a fork bomb could). And FIFO/RR are privileged ops.
I have a test program with FIFO privileges
http://www.vcn.bc.ca/~vli/schedrtcpu.c.txt that reliably eat 100% cpu in top and the patch can kill it reliably, we have an user-space traffic processing program that runs on FIFO similar like the test program, under some condition, that user-space program could stuck on the cpu and we want to kill it for high availability reason. with this patch, we were able to do that.
I do notice that in the schedrtcpu.c test program, if I fork two process like below:
pid_t spawn() {
pid_t pid = fork();
if (pid == 0)
busyloop();
return pid;
}pid1 = spawn(); pid2 = spawn(); waitpid(pid1, &status, 0); waitpid(pid2, &status, 0);
and run it on two cpu box, I got "sched: RT throttling activated" on console and the test program wouldn't stuck on cpu, and can only reach to 95% percent, it is strange that if I don't fork process, and only runs the busyloop, it would not activate RT throttling and consistently eat 100% single cpu.
in our corner case, it appears that patch does help solve our problem.
Quoted 4 lines
Furthermore the distinction between memory and cpu-time is that memory isn't a renewable resource, whereas time is. There's always more time, but there's not always more memory.
understood, thanks
Quoted 2 lines
So no, I don't think either you patch nor your concept make any sense. Consider it nacked.