panic: Ensure preemption is disabled during panic()
PATCH 1/1 panic: Ensure preemption is disabled during panic()
Calling 'panic()' on a kernel with CONFIG_PREEMPT=y can leave the calling CPU in an infinite loop, but with interrupts and preemption enabled. From this state, userspace can continue to be scheduled, despite the system being "dead" as far as the kernel is concerned. This is easily reproducible on arm64 when booting with "nosmp" on the command line; a couple of shell scripts print out a periodic "Ping" message whilst another triggers a crash by writing to /proc/sysrq-trigger:
| sysrq: Trigger a crash | Kernel panic - not syncing: sysrq triggered crash | CPU: 0 PID: 1 Comm: init Not tainted 5.2.15 #1 | Hardware name: linux,dummy-virt (DT) | Call trace: | dump_backtrace+0x0/0x148 | show_stack+0x14/0x20 | dump_stack+0xa0/0xc4 | panic+0x140/0x32c | sysrq_handle_reboot+0x0/0x20 | __handle_sysrq+0x124/0x190 | write_sysrq_trigger+0x64/0x88 | proc_reg_write+0x60/0xa8 | __vfs_write+0x18/0x40 | vfs_write+0xa4/0x1b8 | ksys_write+0x64/0xf0 | __arm64_sys_write+0x14/0x20 | el0_svc_common.constprop.0+0xb0/0x168 | el0_svc_handler+0x28/0x78 | el0_svc+0x8/0xc | Kernel Offset: disabled | CPU features: 0x0002,24002004 | Memory Limit: none | ---[ end Kernel panic - not syncing: sysrq triggered crash ]--- | Ping 2! | Ping 1! | Ping 1! | Ping 2!
The issue can also be triggered on x86 kernels if CONFIG_SMP=n, otherwise local interrupts are disabled in 'smp_send_stop()'.
Disable preemption in 'panic()' before re-enabling interrupts.
--- kernel/panic.c | 1 + 1 file changed, 1 insertion(+)
kernel/panic.c +1 -0
@@ -180,6 +180,7 @@ void panic(const char *fmt, ...)* after setting panic_cpu) from invoking panic() again. */ local_irq_disable (); +preempt_disable_notrace ();/* * It's possible to come here directly from a panic-assertion and
Signature · 1 line
2.23.0.444.g18eeb5a265-goog
Quoted 37 lines · Will Deacon
Calling '
panic()' on a kernel withCONFIG_PREEMPT=ycan leave the calling CPU in an infinite loop, but with interrupts and preemption enabled. From this state, userspace can continue to be scheduled, despite the system being "dead" as far as the kernel is concerned. This is easily reproducible on arm64 when booting with "nosmp" on the command line; a couple of shell scripts print out a periodic "Ping" message whilst another triggers a crash by writing to/proc/sysrq-trigger:| sysrq: Trigger a crash | Kernel panic - not syncing: sysrq triggered crash | CPU: 0 PID: 1 Comm: init Not tainted 5.2.15 #1 | Hardware name: linux,dummy-virt (DT) | Call trace: | dump_backtrace+0x0/0x148 | show_stack+0x14/0x20 | dump_stack+0xa0/0xc4 | panic+0x140/0x32c | sysrq_handle_reboot+0x0/0x20 | __handle_sysrq+0x124/0x190 | write_sysrq_trigger+0x64/0x88 | proc_reg_write+0x60/0xa8 | __vfs_write+0x18/0x40 | vfs_write+0xa4/0x1b8 | ksys_write+0x64/0xf0 | __arm64_sys_write+0x14/0x20 | el0_svc_common.constprop.0+0xb0/0x168 | el0_svc_handler+0x28/0x78 | el0_svc+0x8/0xc | Kernel Offset: disabled | CPU features: 0x0002,24002004 | Memory Limit: none | ---[ end Kernel panic - not syncing: sysrq triggered crash ]--- | Ping 2! | Ping 1! | Ping 1! | Ping 2!
Quoted 4 lines
The issue can also be triggered on x86 kernels if
CONFIG_SMP=n, otherwise local interrupts are disabled in 'smp_send_stop()'.Disable preemption in '
panic()' before re-enabling interrupts.
Is this perhaps the correct solution for what commit c39ea0b9dd24 ("panic:
avoid the extra noise dmesg") was trying to fix?
Either way:
-Kees
Quoted 29 lines
Cc Russell King <[email protected]>Cc Greg Kroah-Hartman <[email protected]>Cc Ingo Molnar <[email protected]>Cc Kees Cook <[email protected]>Cc Andrew Morton <[email protected]>Cc <[email protected]>Reported-by Xogium <[email protected]>Signed-off-by Will Deacon <[email protected]>--- kernel/panic.c | 1 + 1 file changed, 1 insertion(+)kernel/panic.c +1
-0@@ -180,6 +180,7 @@ void panic(const char *fmt, ...)* after setting panic_cpu) from invoking panic() again. */ local_irq_disable (); +preempt_disable_notrace ();/* * It's possible to come here directly from a panic-assertion and Signature · 1 line
2.23.0.444.g18eeb5a265-goog
Signature · 1 line
Kees Cook
Hi Kees,
Quoted 38 lines · Kees Cook
Quoted 37 lines · Will Deacon
Calling '
panic()' on a kernel withCONFIG_PREEMPT=ycan leave the calling CPU in an infinite loop, but with interrupts and preemption enabled. From this state, userspace can continue to be scheduled, despite the system being "dead" as far as the kernel is concerned. This is easily reproducible on arm64 when booting with "nosmp" on the command line; a couple of shell scripts print out a periodic "Ping" message whilst another triggers a crash by writing to/proc/sysrq-trigger:| sysrq: Trigger a crash | Kernel panic - not syncing: sysrq triggered crash | CPU: 0 PID: 1 Comm: init Not tainted 5.2.15 #1 | Hardware name: linux,dummy-virt (DT) | Call trace: | dump_backtrace+0x0/0x148 | show_stack+0x14/0x20 | dump_stack+0xa0/0xc4 | panic+0x140/0x32c | sysrq_handle_reboot+0x0/0x20 | __handle_sysrq+0x124/0x190 | write_sysrq_trigger+0x64/0x88 | proc_reg_write+0x60/0xa8 | __vfs_write+0x18/0x40 | vfs_write+0xa4/0x1b8 | ksys_write+0x64/0xf0 | __arm64_sys_write+0x14/0x20 | el0_svc_common.constprop.0+0xb0/0x168 | el0_svc_handler+0x28/0x78 | el0_svc+0x8/0xc | Kernel Offset: disabled | CPU features: 0x0002,24002004 | Memory Limit: none | ---[ end Kernel panic - not syncing: sysrq triggered crash ]--- | Ping 2! | Ping 1! | Ping 1! | Ping 2!
Quoted 7 lines
Quoted 4 lines
The issue can also be triggered on x86 kernels if
CONFIG_SMP=n, otherwise local interrupts are disabled in 'smp_send_stop()'.Disable preemption in '
panic()' before re-enabling interrupts.Is this perhaps the correct solution for what commit
c39ea0b9dd24("panic:
avoid the extra noise dmesg") was trying to fix?
Hmm, maybe, although that looks like it's focussed more on irq handling than preemption. I've deliberately left the irq part alone, since I think having magic sysrq work via the keyboard interrupt is desirable from the panic loop.
Quoted 1 line
Reviewed-by Kees Cook <[email protected]>
Thanks!
Will
Quoted 44 lines · Will Deacon
Hi Kees,
Quoted 41 lines · Kees Cook
Quoted 40 lines · Will Deacon
Calling '
panic()' on a kernel withCONFIG_PREEMPT=ycan leave the calling CPU in an infinite loop, but with interrupts and preemption enabled. From this state, userspace can continue to be scheduled, despite the system being "dead" as far as the kernel is concerned. This is easily reproducible on arm64 when booting with "nosmp" on the command line; a couple of shell scripts print out a periodic "Ping" message whilst another triggers a crash by writing to/proc/sysrq-trigger:| sysrq: Trigger a crash | Kernel panic - not syncing: sysrq triggered crash | CPU: 0 PID: 1 Comm: init Not tainted 5.2.15 #1 | Hardware name: linux,dummy-virt (DT) | Call trace: | dump_backtrace+0x0/0x148 | show_stack+0x14/0x20 | dump_stack+0xa0/0xc4 | panic+0x140/0x32c | sysrq_handle_reboot+0x0/0x20 | __handle_sysrq+0x124/0x190 | write_sysrq_trigger+0x64/0x88 | proc_reg_write+0x60/0xa8 | __vfs_write+0x18/0x40 | vfs_write+0xa4/0x1b8 | ksys_write+0x64/0xf0 | __arm64_sys_write+0x14/0x20 | el0_svc_common.constprop.0+0xb0/0x168 | el0_svc_handler+0x28/0x78 | el0_svc+0x8/0xc | Kernel Offset: disabled | CPU features: 0x0002,24002004 | Memory Limit: none | ---[ end Kernel panic - not syncing: sysrq triggered crash ]--- | Ping 2! | Ping 1! | Ping 1! | Ping 2!The issue can also be triggered on x86 kernels if
CONFIG_SMP=n, otherwise local interrupts are disabled in 'smp_send_stop()'.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Disable preemption in '
panic()' before re-enabling interrupts.Is this perhaps the correct solution for what commit
c39ea0b9dd24("panic:
avoid the extra noise dmesg") was trying to fix?Hmm, maybe, although that looks like it's focussed more on irq handling than preemption.
Exactly, the backtrace mentioned in commit c39ea0b9dd24 ("panic: avoid the extra noise dmesg") is printed by wake_up() called from wake_up_klogd_work_func(). It is irq_work. Therefore disabling preemption would not prevent this.
Quoted 3 lines
I've deliberately left the irq part alone, since I think having magic sysrq work via the keyboard interrupt is desirable from the panic loop.
I agree that we should keep sysrq working.
One pity thing is that led_panic_blink() in leds/drivers/trigger/ledtrig-panic.c uses workqueues:
+ led_panic_blink()
+ led_trigger_event()
+ led_set_brightness()
+ schedule_work()
It means that it depends on the scheduler. I guess that it does not work in many panic situations. But this patch will always block it.
I agree that it is strange that userspace still works at this stage. But does it cause any real problems?
Best Regards,
Petr
Quoted 73 lines · Petr Mladek
Quoted 44 lines · Will Deacon
Hi Kees,
Quoted 41 lines · Kees Cook
Quoted 40 lines · Will Deacon
Calling '
panic()' on a kernel withCONFIG_PREEMPT=ycan leave the calling CPU in an infinite loop, but with interrupts and preemption enabled. From this state, userspace can continue to be scheduled, despite the system being "dead" as far as the kernel is concerned. This is easily reproducible on arm64 when booting with "nosmp" on the command line; a couple of shell scripts print out a periodic "Ping" message whilst another triggers a crash by writing to/proc/sysrq-trigger:| sysrq: Trigger a crash | Kernel panic - not syncing: sysrq triggered crash | CPU: 0 PID: 1 Comm: init Not tainted 5.2.15 #1 | Hardware name: linux,dummy-virt (DT) | Call trace: | dump_backtrace+0x0/0x148 | show_stack+0x14/0x20 | dump_stack+0xa0/0xc4 | panic+0x140/0x32c | sysrq_handle_reboot+0x0/0x20 | __handle_sysrq+0x124/0x190 | write_sysrq_trigger+0x64/0x88 | proc_reg_write+0x60/0xa8 | __vfs_write+0x18/0x40 | vfs_write+0xa4/0x1b8 | ksys_write+0x64/0xf0 | __arm64_sys_write+0x14/0x20 | el0_svc_common.constprop.0+0xb0/0x168 | el0_svc_handler+0x28/0x78 | el0_svc+0x8/0xc | Kernel Offset: disabled | CPU features: 0x0002,24002004 | Memory Limit: none | ---[ end Kernel panic - not syncing: sysrq triggered crash ]--- | Ping 2! | Ping 1! | Ping 1! | Ping 2!The issue can also be triggered on x86 kernels if
CONFIG_SMP=n, otherwise local interrupts are disabled in 'smp_send_stop()'.Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Disable preemption in '
panic()' before re-enabling interrupts.Is this perhaps the correct solution for what commit
c39ea0b9dd24("panic:
avoid the extra noise dmesg") was trying to fix?Hmm, maybe, although that looks like it's focussed more on irq handling than preemption.
Exactly, the backtrace mentioned in commit
c39ea0b9dd24("panic: avoid the extra noise dmesg") is printed bywake_up()called fromwake_up_klogd_work_func(). It isirq_work. Therefore disabling preemption would not prevent this.Quoted 3 lines
I've deliberately left the irq part alone, since I think having magic sysrq work via the keyboard interrupt is desirable from the panic loop.
I agree that we should keep sysrq working.
One pity thing is that
led_panic_blink()inleds/drivers/trigger/ledtrig-panic.cuses workqueues:+
led_panic_blink()
+led_trigger_event()
+led_set_brightness()
+schedule_work()
Quoted 6 lines
It means that it depends on the scheduler. I guess that it does not work in many panic situations. But this patch will always block it.
I agree that it is strange that userspace still works at this stage. But does it cause any real problems?
Yes, there are watchdog drivers that continue to pat their watchdog after the kernel has panic'd. It makes watchdogs useless (which is exactly how this problem was discovered.)
Signature · 8 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up According to speedtest.net: 11.9Mbps down 500kbps up _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 20 lines · Russell King - ARM Linux admin
Quoted 19 lines · Petr Mladek
Quoted 3 lines · Will Deacon
I've deliberately left the irq part alone, since I think having magic sysrq work via the keyboard interrupt is desirable from the panic loop.
I agree that we should keep sysrq working.
One pity thing is that
led_panic_blink()inleds/drivers/trigger/ledtrig-panic.cuses workqueues:+
led_panic_blink()
+led_trigger_event()
+led_set_brightness()
+schedule_work()It means that it depends on the scheduler. I guess that it does not work in many panic situations. But this patch will always block it.
Quoted 6 lines
Quoted 2 lines
I agree that it is strange that userspace still works at this stage. But does it cause any real problems?
Yes, there are watchdog drivers that continue to pat their watchdog after the kernel has panic'd. It makes watchdogs useless (which is exactly how this problem was discovered.)
Indeed, and I think the LED blinking is already unreliable if the brightness operation needs to sleep. For example, if the kernel isn't preemptible or the work gets queued up on a different CPU which is sitting in panic_smp_self_stop().
Will
Quoted 24 lines · Will Deacon
Quoted 23 lines · Russell King - ARM Linux admin
Quoted 21 lines · Petr Mladek
Quoted 3 lines · Will Deacon
I've deliberately left the irq part alone, since I think having magic sysrq work via the keyboard interrupt is desirable from the panic loop.
I agree that we should keep sysrq working.
One pity thing is that
led_panic_blink()inleds/drivers/trigger/ledtrig-panic.cuses workqueues:+
led_panic_blink()
+led_trigger_event()
+led_set_brightness()
+schedule_work()It means that it depends on the scheduler. I guess that it does not work in many panic situations. But this patch will always block it.
I agree that it is strange that userspace still works at this stage. But does it cause any real problems?
Quoted 8 lines
Quoted 3 lines
Yes, there are watchdog drivers that continue to pat their watchdog after the kernel has panic'd. It makes watchdogs useless (which is exactly how this problem was discovered.)
Indeed, and I think the LED blinking is already unreliable if the brightness operation needs to sleep. For example, if the kernel isn't preemptible or the work gets queued up on a different CPU which is sitting in
panic_smp_self_stop().
To make it clear. I do not want to block this patch. I just wanted to point out the problem. I am not sure how the blinking is important these days. Well, I could imagine that it might be useful on some embedded devices.
Another question is how many people want to end up with dead system these days. The watchdogs are likely used in data centers. I guess that automatic reboot in panic() is a better choice there.
Anyway, it might make sense to remove the panic blinking code when it will not have a chance to work.
Best Regards,
Petr
Quoted 34 lines · Petr Mladek
Quoted 24 lines · Will Deacon
Quoted 23 lines · Russell King - ARM Linux admin
Quoted 21 lines · Petr Mladek
Quoted 3 lines · Will Deacon
I've deliberately left the irq part alone, since I think having magic sysrq work via the keyboard interrupt is desirable from the panic loop.
I agree that we should keep sysrq working.
One pity thing is that
led_panic_blink()inleds/drivers/trigger/ledtrig-panic.cuses workqueues:+
led_panic_blink()
+led_trigger_event()
+led_set_brightness()
+schedule_work()It means that it depends on the scheduler. I guess that it does not work in many panic situations. But this patch will always block it.
I agree that it is strange that userspace still works at this stage. But does it cause any real problems?
Quoted 8 lines
Quoted 3 lines
Yes, there are watchdog drivers that continue to pat their watchdog after the kernel has panic'd. It makes watchdogs useless (which is exactly how this problem was discovered.)
Indeed, and I think the LED blinking is already unreliable if the brightness operation needs to sleep. For example, if the kernel isn't preemptible or the work gets queued up on a different CPU which is sitting in
panic_smp_self_stop().
Quoted 4 lines
To make it clear. I do not want to block this patch. I just wanted to point out the problem. I am not sure how the blinking is important these days. Well, I could imagine that it might be useful on some embedded devices.
When reviewing the c39ea0b9dd24 ("panic: avoid the extra noise dmesg"), there was similar discussion about what's the expectation for kernel when panic happens, as the earlier version patch is simply keeping the the local irq disabled, which may break the sysrq and the panic blink code, so we turned to handling it together with printk.
Quoted 7 lines
Another question is how many people want to end up with dead system these days. The watchdogs are likely used in data centers. I guess that automatic reboot in
panic()is a better choice there.Anyway, it might make sense to remove the panic blinking code when it will not have a chance to work.
I was also wondering if the panic blinking code still really works on any platforms.
Thanks,
Feng
Quoted 3 lines
Best Regards,
Petr
Quoted 2 lines · Will Deacon
Indeed, and I think the LED blinking is already unreliable if the brightness operation needs to sleep.
One thing is that led_set_brightness() can probably be forced to avoid the workqueue scheduling, by setting LED_BLINK_SW on the device (e.g. by issuing led_set_software_blink() during panic).
But I am afraid this still won't solve the issue completely, as USB keyboards need workqueues for blinking the LEDs in for URB management.
Signature · 2 lines
Jiri Kosina SUSE Labs
Quoted 47 lines · On Wed, 2 Oct 2019 13:35:38 +0100 Will Deacon <[email protected]> wrote:
Calling '
panic()' on a kernel withCONFIG_PREEMPT=ycan leave the calling CPU in an infinite loop, but with interrupts and preemption enabled. From this state, userspace can continue to be scheduled, despite the system being "dead" as far as the kernel is concerned. This is easily reproducible on arm64 when booting with "nosmp" on the command line; a couple of shell scripts print out a periodic "Ping" message whilst another triggers a crash by writing to/proc/sysrq-trigger:| sysrq: Trigger a crash | Kernel panic - not syncing: sysrq triggered crash | CPU: 0 PID: 1 Comm: init Not tainted 5.2.15 #1 | Hardware name: linux,dummy-virt (DT) | Call trace: | dump_backtrace+0x0/0x148 | show_stack+0x14/0x20 | dump_stack+0xa0/0xc4 | panic+0x140/0x32c | sysrq_handle_reboot+0x0/0x20 | __handle_sysrq+0x124/0x190 | write_sysrq_trigger+0x64/0x88 | proc_reg_write+0x60/0xa8 | __vfs_write+0x18/0x40 | vfs_write+0xa4/0x1b8 | ksys_write+0x64/0xf0 | __arm64_sys_write+0x14/0x20 | el0_svc_common.constprop.0+0xb0/0x168 | el0_svc_handler+0x28/0x78 | el0_svc+0x8/0xc | Kernel Offset: disabled | CPU features: 0x0002,24002004 | Memory Limit: none | ---[ end Kernel panic - not syncing: sysrq triggered crash ]--- | Ping 2! | Ping 1! | Ping 1! | Ping 2!The issue can also be triggered on x86 kernels if
CONFIG_SMP=n, otherwise local interrupts are disabled in 'smp_send_stop()'.Disable preemption in '
panic()' before re-enabling interrupts....
kernel/panic.c void panic(const char *fmt, ...) · line 186
Quoted 7 lines
kernel/panic.c +1
-0@@ -180,6 +180,7 @@ void panic(const char *fmt, ...)* after setting panic_cpu) from invoking panic() again. */ local_irq_disable (); +preempt_disable_notrace ();/* * It's possible to come here directly from a panic-assertion and
We still do a lot of stuff (kexec, kgdb, etc) after this preempt_disable() and I worry that something in there will now trigger a might_sleep() warning as a result?
Hi Andrew,
Thanks for having a look.
Quoted 13 lines · Andrew Morton
Quoted 12 lines · On Wed, 2 Oct 2019 13:35:38 +0100 Will Deacon <[email protected]> wrote:
Disable preemption in '
panic()' before re-enabling interrupts....
kernel/panic.c +1
-0@@ -180,6 +180,7 @@ void panic(const char *fmt, ...)* after setting panic_cpu) from invoking panic() again. */ local_irq_disable (); +preempt_disable_notrace ();
Quoted 6 lines
kernel/panic.c void panic(const char *fmt, ...) · line 186
Quoted 2 lines
kernel/panic.c +0
-0/* * It's possible to come here directly from a panic-assertion and We still do a lot of stuff (kexec, kgdb, etc) after this
preempt_disable()and I worry that something in there will now trigger amight_sleep()warning as a result?
Given that interrupts are already disabled at this point, I don't think we'll get any additional warnings here by disabling preemption as well.
Will