soft lockup: kill realtime threads before panic
PATCH 1/1 soft lockup: kill realtime threads before panic
From: Joern Engel <[email protected]>
We have observed cases where the soft lockup detector triggered, but no kernel bug existed. Instead we had a buggy realtime thread that monopolized a cpu. So let's kill the responsible party and not panic the entire system.
--- kernel/watchdog.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/watchdog.c +4 -1
@@ -428,7 +428,10 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) }add_taint (TAINT_SOFTLOCKUP ,LOCKDEP_STILL_OK ); -if (softlockup_panic) +if (rt_prio (current->prio)) { +pr_emerg ("killing realtime thread\n" ); +send_sig (SIGILL , current,0 ); + }else if (softlockup_panic)panic ("softlockup: hung tasks" );__this_cpu_write (soft_watchdog_warn, true); }else
Signature · 1 line
2.4.3
Quoted 4 lines · Spencer Baugh
We have observed cases where the soft lockup detector triggered, but no kernel bug existed. Instead we had a buggy realtime thread that monopolized a cpu. So let's kill the responsible party and not panic the entire system.
If you don't tell the kernel to panic, it won't, and if you don't remove its leash (the throttle), your not so tame rt beast won't maul you.
-Mike
Quoted 9 lines · Mike Galbraith
Quoted 4 lines · Spencer Baugh
We have observed cases where the soft lockup detector triggered, but no kernel bug existed. Instead we had a buggy realtime thread that monopolized a cpu. So let's kill the responsible party and not panic the entire system.
If you don't tell the kernel to panic, it won't, and if you don't remove its leash (the throttle), your not so tame rt beast won't maul you.
Not sure if this patch is something for mainline, but those two alternatives have problems of their own. Not panicking on lockups can leave a system disabled until some human come around. In many cases that human will do no better than power-cycle. A panic reduces the downtime.
And the realtime throttling gives non-realtime threads some minimum runtime, but does nothing to help low-priority realtime threads. It also introduces latencies, often when workloads are high and you would like any available cpu to get through that rough spot.
I don't think we have a good answer to this problem in the mainline kernel yet.
Jörn
--
To announce that there must be no criticism of the President, or that we are to stand by the President, right or wrong, is not only unpatriotic and servile, but is morally treasonable to the American public. -- Theodore Roosevelt, Kansas City Star, 1918
Quoted 8 lines · Jörn Engel
Quoted 7 lines · Mike Galbraith
Quoted 4 lines · Spencer Baugh
We have observed cases where the soft lockup detector triggered, but no kernel bug existed. Instead we had a buggy realtime thread that monopolized a cpu. So let's kill the responsible party and not panic the entire system.
Quoted 8 lines
Quoted 2 lines
If you don't tell the kernel to panic, it won't, and if you don't remove its leash (the throttle), your not so tame rt beast won't maul you.
Not sure if this patch is something for mainline, but those two alternatives have problems of their own. Not panicking on lockups can leave a system disabled until some human come around. In many cases that human will do no better than power-cycle. A panic reduces the downtime.
If a realtime task goes bonkers, the realtime game is over, you're down.
Quoted 4 lines
And the realtime throttling gives non-realtime threads some minimum runtime, but does nothing to help low-priority realtime threads. It also introduces latencies, often when workloads are high and you would like any available cpu to get through that rough spot.
You can use group scheduling as a debug crutch until the little beasts are housebroken.
Quoted 2 lines
I don't think we have a good answer to this problem in the mainline kernel yet.
IMHO, there's no point in trying to make rt warm/fuzzy/cuddly. Just don't stuff a Hells Angel into a super-suit, that gets real ugly ;-)
-Mike
Quoted 9 lines · Mike Galbraith
Quoted 6 lines · Jörn Engel
Not sure if this patch is something for mainline, but those two alternatives have problems of their own. Not panicking on lockups can leave a system disabled until some human come around. In many cases that human will do no better than power-cycle. A panic reduces the downtime.
If a realtime task goes bonkers, the realtime game is over, you're down.
Agreed. But a reboot will often solve the issue. So the automatic panic will repair the system within minutes, while no panic will leave the system broken for days, depending on human response time. Automatic panic is a great way to minimize downtime - or vulnerable time if you have HA.
One could argue that killing the realtime thread is even better than panic, as things can restart with a blank slate even faster. But the real benefit is that we get better debug data for the failing component. If we had a kernel bug, the backtrace would usually be sufficient to point fingers. With a bonkers realtime thread, not so much.
Anyway, this patch has been useful to us once. If someone deems it merge-worthy, great. If not, I won't lose any sleep either.
Jörn
--
The key to performance is elegance, not battalions of special cases. -- Jon Bentley and Doug McIlroy
Quoted 5 lines · Jörn Engel
One could argue that killing the realtime thread is even better than panic, as things can restart with a blank slate even faster. But the real benefit is that we get better debug data for the failing component. If we had a kernel bug, the backtrace would usually be sufficient to point fingers. With a bonkers realtime thread, not so much.
If userspace wants a watchdog, it should train a userspace dog, not turn the kernel watchdog into a userspace attack dog.
-Mike
Quoted 10 lines · Mike Galbraith
Quoted 5 lines · Jörn Engel
One could argue that killing the realtime thread is even better than panic, as things can restart with a blank slate even faster. But the real benefit is that we get better debug data for the failing component. If we had a kernel bug, the backtrace would usually be sufficient to point fingers. With a bonkers realtime thread, not so much.
If userspace wants a watchdog, it should train a userspace dog, not turn the kernel watchdog into a userspace attack dog.
I agree. The spirit of the watchdog was detection and panic (if configured that way). I don't think adding policy like this works well in the long run.
Cheers,
Don
Quoted 10 lines · Mike Galbraith
Quoted 5 lines · Jörn Engel
One could argue that killing the realtime thread is even better than panic, as things can restart with a blank slate even faster. But the real benefit is that we get better debug data for the failing component. If we had a kernel bug, the backtrace would usually be sufficient to point fingers. With a bonkers realtime thread, not so much.
If userspace wants a watchdog, it should train a userspace dog, not turn the kernel watchdog into a userspace attack dog.
Fair point. Let's drop this patch then.
Jörn
--
Money can buy bandwidth, but latency is forever.
-- John R. Mashey
Quoted 23 lines · Spencer Baugh
From: Joern Engel <[email protected]>
We have observed cases where the soft lockup detector triggered, but no kernel bug existed. Instead we had a buggy realtime thread that monopolized a cpu. So let's kill the responsible party and not panic the entire system.
Signed-off-by Joern Engel <[email protected]>Signed-off-by Spencer Baugh <[email protected]>---
kernel/watchdog.c| 5 ++++-1 file changed, 4 insertions(+), 1 deletion(-)kernel/watchdog.c +0
-0@@ -428,7 +428,10 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) }
kernel/watchdog.c static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) · line 437
Quoted 10 lines
kernel/watchdog.c +4
-1add_taint (TAINT_SOFTLOCKUP ,LOCKDEP_STILL_OK ); -if (softlockup_panic) +if (rt_prio (current->prio)) { +pr_emerg ("killing realtime thread\n" ); +send_sig (SIGILL , current,0 ); + }else if (softlockup_panic)panic ("softlockup: hung tasks" );__this_cpu_write (soft_watchdog_warn, true); }else
just my advice about this patch,
i think should add PF_KTHREAD condition like this:
if (rt_prio(current->prio) && !(current->flags & PF_KTHREAD)) {
+ pr_emerg("killing realtime thread\n");
+ send_sig(SIGILL, current, 0);
+ } else if (softlockup_panic)
if soft lockup is caused by kthread, should still panic .
Thanks
Quoted 12 lines · On Tue, 21 Jul 2015 15:07:57 -0700 Spencer Baugh <[email protected]> wrote:
From: Joern Engel <[email protected]>
We have observed cases where the soft lockup detector triggered, but no kernel bug existed. Instead we had a buggy realtime thread that monopolized a cpu. So let's kill the responsible party and not panic the entire system.
...
kernel/watchdog.c static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) · line 433
Quoted 7 lines
kernel/watchdog.c +3
-1@@ -428,7 +428,10 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) }add_taint (TAINT_SOFTLOCKUP ,LOCKDEP_STILL_OK ); -if (softlockup_panic) +if (rt_prio (current->prio)) { +pr_emerg ("killing realtime thread\n" ); +send_sig (SIGILL , current,0 );
Why choose SIGILL?
kernel/watchdog.c static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) · line 436
Quoted 3 lines
kernel/watchdog.c +1
-0+ }else if (softlockup_panic)panic ("softlockup: hung tasks" );__this_cpu_write (soft_watchdog_warn, true);
But what about a non-buggy realtime thread which happens to occasionally spend 15 seconds doing stuff?
Old behaviour: kernel blurts a softlockup message, everything keeps running.
New behaviour: thread gets killed, plane crashes.
Possibly a better approach would be to only kill the thread if softlockup_panic was set, because the system is going down anyway.
Also, perhaps some users would prefer that the kernel simply suppress the softlockup warning in this situation, rather than killing stuff!
Really, what you're trying to implement here is a watchdog for runaway realtime threads. And that sounds a worthy project but it's a rather separate thing from the softlockup detector. A realtime thread watchdog feature might have things as
- timeout duration separately configurable from softlockup
- enabled independently from sotflockup: people might want one and not the other.
- configurable signal, perhaps?
Now, the *implementation* of the realtime thread watchdog may well share code with the softlockup detector. But from a conceptual/configuration/documentation point of view, it's a separate thing, no?
Quoted 16 lines · Andrew Morton
Quoted 14 lines · On Tue, 21 Jul 2015 15:07:57 -0700 Spencer Baugh <[email protected]> wrote:
From: Joern Engel <[email protected]>
We have observed cases where the soft lockup detector triggered, but no kernel bug existed. Instead we had a buggy realtime thread that monopolized a cpu. So let's kill the responsible party and not panic the entire system.
...
kernel/watchdog.c +0
-0@@ -428,7 +428,10 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) }
Quoted 7 lines
kernel/watchdog.c static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) · line 433
Quoted 5 lines
kernel/watchdog.c +3
-1add_taint (TAINT_SOFTLOCKUP ,LOCKDEP_STILL_OK ); -if (softlockup_panic) +if (rt_prio (current->prio)) { +pr_emerg ("killing realtime thread\n" ); +send_sig (SIGILL , current,0 );Why choose SIGILL?
It is a random signal that happens to generate a stacktrace in userspace.
Quoted 31 lines
kernel/watchdog.c static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) · line 436
Quoted 3 lines
kernel/watchdog.c +1
-0+ }else if (softlockup_panic)panic ("softlockup: hung tasks" );__this_cpu_write (soft_watchdog_warn, true);But what about a non-buggy realtime thread which happens to occasionally spend 15 seconds doing stuff?
Old behaviour: kernel blurts a softlockup message, everything keeps running.
New behaviour: thread gets killed, plane crashes.
Possibly a better approach would be to only kill the thread if
softlockup_panicwas set, because the system is going down anyway.Also, perhaps some users would prefer that the kernel simply suppress the softlockup warning in this situation, rather than killing stuff!
Really, what you're trying to implement here is a watchdog for runaway realtime threads. And that sounds a worthy project but it's a rather separate thing from the softlockup detector. A realtime thread watchdog feature might have things as
- timeout duration separately configurable from softlockup
- enabled independently from sotflockup: people might want one and not the other.
Quoted 6 lines
- configurable signal, perhaps?
Now, the *implementation* of the realtime thread watchdog may well share code with the softlockup detector. But from a conceptual/configuration/documentation point of view, it's a separate thing, no?
Agreed. We needed this patch exactly once and it is a rather quick hack that yielded the necessary results. Realtime threads were well-behaved since and the patch has seen zero polish as a result.
I think it is better to drop the patch for now. If someone else keeps running into the same issue, it might be a starting point for a better implementation. They will find it in list archives.
Jörn
--
I can say that I spend most of my time fixing bugs even if I have lots of new features to implement in mind, but I give bugs more priority. -- Andrea Arcangeli, 2000