panic: Avoid the extra noise dmesg
Subsystems (3)
PATCH v4 1/1 panic: Avoid the extra noise dmesg
When kernel panic happens, it will first print the panic call stack, then the ending msg like:
[ 35.743249] ---[ end Kernel panic - not syncing: Fatal exception [ 35.749975] ------------[ cut here ]------------
The above message are very useful for debugging.
But if system is configured to not reboot on panic, say the "panic_timeout" parameter equals 0, it will likely print out many noisy message like WARN() call stack for each and every CPU except the panic one, messages like below:
WARNING: CPU: 1 PID: 280 at kernel/sched/core.c:1198 set_task_cpu+0x183/0x190 Call Trace: <IRQ> try_to_wake_up default_wake_function autoremove_wake_function __wake_up_common __wake_up_common_lock __wake_up wake_up_klogd_work_func irq_work_run_list irq_work_tick update_process_times tick_sched_timer __hrtimer_run_queues hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt
For people working in console mode, the screen will first show the panic call stack, but immediately overridded by these noisy extra messages, which makes debugging much more difficult, as the original context gets lost on screen.
Also these noisy messages will confuse some users, as I have seen many bug reporters posted the noisy message into bugzilla, instead of the real panic call stack and context.
Keeping the interrupt disabled will avoid the noisy message.
When code runs to this point, it means user has chosed to not reboot, or do any special handling by using the panic notifier method, the only reason to enable the interrupt may be sysrq migic key and panic_blink function (though may not work even with irq enabled).
So make the irq disabled by default and add a cmdline parameter "panic_keep_irq_on" to turn it on when needed.
---
Changelog:
v4:
- make the local_irq_enable conditional and default off to cover possible use of interrupt/scheduling, as mentioned by Sergey and Petr
v3:
- Make the change log clearer as suggested by Andrew Morton
v2:
- Move the solution from hacking arch/scheduler code back to panic.c
Documentation/admin-guide/kernel-parameters.txt | 6 ++++++ kernel/panic.c | 9 ++++++++- 2 files changed, 14 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +6 -0
@@ -3093,6 +3093,12 @@ panic_on_warn panic() instead of WARN(). Useful to cause kdump on a WARN(). + panic_keep_irq_on + To keep the panic call stack info clean and suppress + noisy kernel message, the interrupt on panic cpu is + disabled by default. Add this to cmdline to enable + interrupt for panic cpu. + crash_kexec_post_notifiers Run kdump after running panic-notifiers and dumping kmsg. This only for the users who doubt kdump always
kernel/panic.c +8 -1
@@ -40,6 +40,7 @@ static int pause_on_oops;static int pause_on_oops_flag;static DEFINE_SPINLOCK (pause_on_oops_lock);bool crash_kexec_post_notifiers; +bool panic_keep_irq_on;int panic_on_warn__read_mostly ;int panic_timeout =CONFIG_PANIC_TIMEOUT ;
@@ -322,7 +323,12 @@ void panic(const char *fmt, ...) }#endif pr_emerg ("---[ end Kernel panic - not syncing: %s ]---\n" , buf); -local_irq_enable (); + +if (panic_keep_irq_on) +local_irq_enable (); +else +pr_emerg ("Please add panic_keep_irq_on to cmdline if you want blink/sysrq to work." ); +for (i =0 ; ; i +=PANIC_TIMER_STEP ) {touch_softlockup_watchdog ();if (i >= i_next) {
@@ -684,6 +690,7 @@ core_param(panic, panic_timeout, int, 0644);core_param (panic_print, panic_print, ulong,0644 );core_param (pause_on_oops, pause_on_oops, int,0644 );core_param (panic_on_warn, panic_on_warn, int,0644 ); +core_param (panic_keep_irq_on, panic_keep_irq_on, bool,0644 );core_param (crash_kexec_post_notifiers, crash_kexec_post_notifiers, bool,0644 );static int __init oops_setup (char *s)
Signature · 1 line
2.7.4
Quoted 21 lines · On Fri, 7 Dec 2018 17:51:19 +0800 Feng Tang <[email protected]> wrote:
When kernel panic happens, it will first print the panic call stack, then the ending msg like:
[ 35.743249] ---[ end Kernel panic - not syncing: Fatal exception [ 35.749975] ------------[ cut here ]------------The above message are very useful for debugging.
But if system is configured to not reboot on panic, say the "
panic_timeout" parameter equals 0, it will likely print out many noisy message likeWARN()call stack for each and every CPU except the panic one, messages like below:Keeping the interrupt disabled will avoid the noisy message.
When code runs to this point, it means user has chosed to not reboot, or do any special handling by using the panic notifier method, the only reason to enable the interrupt may be sysrq migic key and
panic_blinkfunction (though may not work even with irq enabled).
Quoted 5 lines
So make the irq disabled by default and add a cmdline parameter "
panic_keep_irq_on" to turn it on when needed.Signed-off-by Feng Tang <[email protected]>
-- Steve
Quoted 25 lines · Steven Rostedt
Quoted 21 lines · On Fri, 7 Dec 2018 17:51:19 +0800 Feng Tang <[email protected]> wrote:
When kernel panic happens, it will first print the panic call stack, then the ending msg like:
[ 35.743249] ---[ end Kernel panic - not syncing: Fatal exception [ 35.749975] ------------[ cut here ]------------The above message are very useful for debugging.
But if system is configured to not reboot on panic, say the "
panic_timeout" parameter equals 0, it will likely print out many noisy message likeWARN()call stack for each and every CPU except the panic one, messages like below:Keeping the interrupt disabled will avoid the noisy message.
When code runs to this point, it means user has chosed to not reboot, or do any special handling by using the panic notifier method, the only reason to enable the interrupt may be sysrq migic key and
panic_blinkfunction (though may not work even with irq enabled).
Quoted 7 lines
Quoted 5 lines
So make the irq disabled by default and add a cmdline parameter "
panic_keep_irq_on" to turn it on when needed.Signed-off-by Feng Tang <[email protected]>
I'm fine with the new boot param, but I think we need to leave it how it was by default: systems that want to see the blinking aren't going to have a screen to read about what boot param they need to add. Currently, we'll blink and spew extra warnings. With this patch we'll not blink and not spew: a headless machine will have no indication that a panic happened.
-Kees
--
Kees Cook
Petr Mladek
maintainerQuoted 32 lines · Kees Cook
Quoted 31 lines · Steven Rostedt
Quoted 26 lines · On Fri, 7 Dec 2018 17:51:19 +0800 Feng Tang <[email protected]> wrote:
When kernel panic happens, it will first print the panic call stack, then the ending msg like:
[ 35.743249] ---[ end Kernel panic - not syncing: Fatal exception [ 35.749975] ------------[ cut here ]------------The above message are very useful for debugging.
But if system is configured to not reboot on panic, say the "
panic_timeout" parameter equals 0, it will likely print out many noisy message likeWARN()call stack for each and every CPU except the panic one, messages like below:Keeping the interrupt disabled will avoid the noisy message.
When code runs to this point, it means user has chosed to not reboot, or do any special handling by using the panic notifier method, the only reason to enable the interrupt may be sysrq migic key and
panic_blinkfunction (though may not work even with irq enabled).So make the irq disabled by default and add a cmdline parameter "
panic_keep_irq_on" to turn it on when needed.Signed-off-by Feng Tang <[email protected]>
Quoted 8 lines
Quoted 1 line
I'm fine with the new boot param, but I think we need to leave it how it was by default: systems that want to see the blinking aren't going to have a screen to read about what boot param they need to add. Currently, we'll blink and spew extra warnings. With this patch we'll not blink and not spew: a headless machine will have no indication that a panic happened.
Just to be sure, in case, you did not follow the other long discussion.
I had an alternative approach to switch printk() into nop at this panic() stage. It can be restored when sysrq is triggered. This approach allows to keep blinking and sysrq working. People debugging the blinking would need to block this change but it should be rare.
I do not resist on my solution. It looks a bit hacky. But it is simple and rather straightforward.
Best Regards,
Petr
Hi all,
Quoted 45 lines · Petr Mladek
Quoted 32 lines · Kees Cook
Quoted 31 lines · Steven Rostedt
Quoted 26 lines · On Fri, 7 Dec 2018 17:51:19 +0800 Feng Tang <[email protected]> wrote:
When kernel panic happens, it will first print the panic call stack, then the ending msg like:
[ 35.743249] ---[ end Kernel panic - not syncing: Fatal exception [ 35.749975] ------------[ cut here ]------------The above message are very useful for debugging.
But if system is configured to not reboot on panic, say the "
panic_timeout" parameter equals 0, it will likely print out many noisy message likeWARN()call stack for each and every CPU except the panic one, messages like below:Keeping the interrupt disabled will avoid the noisy message.
When code runs to this point, it means user has chosed to not reboot, or do any special handling by using the panic notifier method, the only reason to enable the interrupt may be sysrq migic key and
panic_blinkfunction (though may not work even with irq enabled).So make the irq disabled by default and add a cmdline parameter "
panic_keep_irq_on" to turn it on when needed.Signed-off-by Feng Tang <[email protected]>Quoted 8 lines
Quoted 1 line
I'm fine with the new boot param, but I think we need to leave it how it was by default: systems that want to see the blinking aren't going to have a screen to read about what boot param they need to add. Currently, we'll blink and spew extra warnings. With this patch we'll not blink and not spew: a headless machine will have no indication that a panic happened.
Just to be sure, in case, you did not follow the other long discussion.
Quoted 8 lines
I had an alternative approach to switch
printk()into nop at thispanic()stage. It can be restored when sysrq is triggered. This approach allows to keep blinking and sysrq working. People debugging the blinking would need to block this change but it should be rare.I do not resist on my solution. It looks a bit hacky. But it is simple and rather straightforward.
Sorry for the late response.
So currently, there are 2 proposals:
1. this v4 patch of "panic_keep_irq_on" flag (default off to be same as the current kernel behavior)
2. Petr's suggestion of adding a flag to suppress printk after enterring late panic phase (blinking time), while keeping the sysrq printk working.
Following is the draft patch based on Petr's suggestion:
Please review, thanks. I'm fine with both solutions.
- Feng
drivers/tty/sysrq.c +7 -0
@@ -528,6 +528,11 @@ void __handle_sysrq(int key, bool check_mask)struct sysrq_key_op *op_p;int orig_log_level;int i; +int old_val; + +/* save the old panic printk flag */ + old_val = panic_suppress_printk; + panic_suppress_printk =1 ;rcu_sysrq_start ();rcu_read_lock ();
@@ -574,6 +579,8 @@ void __handle_sysrq(int key, bool check_mask) }rcu_read_unlock ();rcu_sysrq_end (); + + panic_suppress_printk = old_val; }void handle_sysrq (int key)
include/linux/kernel.h +1 -0
@@ -534,6 +534,7 @@ extern int panic_on_io_nmi;extern int panic_on_warn;extern int sysctl_panic_on_rcu_stall;extern int sysctl_panic_on_stackoverflow; +extern int panic_suppress_printk;extern bool crash_kexec_post_notifiers;
kernel/panic.c +3 -0
@@ -46,6 +46,8 @@ int panic_on_warn __read_mostly;int panic_timeout =CONFIG_PANIC_TIMEOUT ;EXPORT_SYMBOL_GPL (panic_timeout); +int panic_suppress_printk; +#define PANIC_PRINT_TASK_INFO 0x00000001#define PANIC_PRINT_MEM_INFO 0x00000002#define PANIC_PRINT_TIMER_INFO 0x00000004
@@ -326,6 +328,7 @@ void panic(const char *fmt, ...) }#endif pr_emerg ("---[ end Kernel panic - not syncing: %s ]---\n" , buf); + panic_suppress_printk =1 ;local_irq_enable ();for (i =0 ; ; i +=PANIC_TIMER_STEP ) {touch_softlockup_watchdog ();
kernel/printk/printk.c +3 -0
@@ -1987,6 +1987,9 @@ asmlinkage __visible int printk(const char *fmt, ...)va_list args;int r; +if (unlikely (panic_suppress_printk)) +return 0 ; +va_start (args, fmt); r =vprintk_func (fmt, args);va_end (args);
Petr Mladek
maintainerQuoted 40 lines · Feng Tang
Hi all,
Quoted 12 lines · Petr Mladek
Quoted 11 lines · Kees Cook
Quoted 10 lines · Steven Rostedt
Quoted 6 lines · On Fri, 7 Dec 2018 17:51:19 +0800 Feng Tang <[email protected]> wrote:
When kernel panic happens, it will first print the panic call stack, then the ending msg like:
[ 35.743249] ---[ end Kernel panic - not syncing: Fatal exception [ 35.749975] ------------[ cut here ]------------Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
The above message are very useful for debugging.
But if system is configured to not reboot on panic, say the "
panic_timeout" parameter equals 0, it will likely print out many noisy message likeWARN()call stack for each and every CPU except the panic one, messages like below:So currently, there are 2 proposals:
1. this v4 patch of "panic_keep_irq_on" flag (default off to be same as the current kernel behavior)
2. Petr's suggestion of adding a flag to suppress printk after enterring late panic phase (blinking time), while keeping the sysrq printk working.Following is the draft patch based on Petr's suggestion:
Please review, thanks. I'm fine with both solutions.
- Feng
drivers/tty/sysrq.c +0
-0
drivers/tty/sysrq.c void __handle_sysrq(int key, bool check_mask) · line 533
Quoted 6 lines
drivers/tty/sysrq.c +3
-0@@ -528,6 +528,11 @@ void __handle_sysrq(int key, bool check_mask)struct sysrq_key_op *op_p;int orig_log_level;int i; +int old_val; + +/* save the old panic printk flag */
The comment is not needed. It is obvious.
drivers/tty/sysrq.c void __handle_sysrq(int key, bool check_mask) · line 534
Quoted 1 line
drivers/tty/sysrq.c +1
-0+ old_val = panic_suppress_printk;
s/old_val/orig_panic_suppress_printk/ to follow
the naming of orig_log_level.
drivers/tty/sysrq.c void __handle_sysrq(int key, bool check_mask) · line 535
Quoted 1 line
drivers/tty/sysrq.c +1
-0+ panic_suppress_printk =1 ;
We want to enable the messages in sysrq. This should be:
panic_suppress_printk = 0;
Quoted 16 lines
drivers/tty/sysrq.c +2
-0rcu_sysrq_start ();rcu_read_lock ();@@ -574,6 +579,8 @@ void __handle_sysrq(int key, bool check_mask) }rcu_read_unlock ();rcu_sysrq_end (); + + panic_suppress_printk = old_val; }void handle_sysrq (int key)kernel/panic.c +0
-0
kernel/panic.c void panic(const char *fmt, ...) · line 330
Quoted 3 lines
kernel/panic.c +0
-0@@ -326,6 +328,7 @@ void panic(const char *fmt, ...) }#endif pr_emerg ("---[ end Kernel panic - not syncing: %s ]---\n" , buf);
/* Do not scroll important messages with errors from blinking code. */
kernel/printk/printk.c asmlinkage __visible int printk(const char *fmt, ...) · line 1988
Quoted 11 lines
kernel/panic.c +1
-0+ panic_suppress_printk =1 ;local_irq_enable ();for (i =0 ; ; i +=PANIC_TIMER_STEP ) {touch_softlockup_watchdog ();kernel/printk/printk.c +0
-0@@ -1987,6 +1987,9 @@ asmlinkage __visible int printk(const char *fmt, ...)va_list args;int r;
/* Suppress messages from panic blinking code. */
kernel/printk/printk.c asmlinkage __visible int printk(const char *fmt, ...) · line 1991
Quoted 2 lines
kernel/printk/printk.c +2
-0+if (unlikely (panic_suppress_printk)) +return 0 ;
This should go to vprintk_emit() so that it works for all printk() interfaces.
kernel/printk/printk.c asmlinkage __visible int printk(const char *fmt, ...) · line 1995
Quoted 4 lines
kernel/printk/printk.c +1
-0+va_start (args, fmt); r =vprintk_func (fmt, args);va_end (args);
Best Regards,
Petr
Hi Petr,
Thanks for your review.
Quoted 36 lines · Petr Mladek
Quoted 35 lines · Feng Tang
Hi all,
Quoted 12 lines · Petr Mladek
Quoted 11 lines · Kees Cook
Quoted 10 lines · Steven Rostedt
Quoted 6 lines · On Fri, 7 Dec 2018 17:51:19 +0800 Feng Tang <[email protected]> wrote:
When kernel panic happens, it will first print the panic call stack, then the ending msg like:
[ 35.743249] ---[ end Kernel panic - not syncing: Fatal exception [ 35.749975] ------------[ cut here ]------------Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
The above message are very useful for debugging.
But if system is configured to not reboot on panic, say the "
panic_timeout" parameter equals 0, it will likely print out many noisy message likeWARN()call stack for each and every CPU except the panic one, messages like below:So currently, there are 2 proposals:
1. this v4 patch of "panic_keep_irq_on" flag (default off to be same as the current kernel behavior)
2. Petr's suggestion of adding a flag to suppress printk after enterring late panic phase (blinking time), while keeping the sysrq printk working.Following is the draft patch based on Petr's suggestion:
Please review, thanks. I'm fine with both solutions.
- Feng
Quoted 13 lines
drivers/tty/sysrq.c void __handle_sysrq(int key, bool check_mask) · line 533
Quoted 11 lines
drivers/tty/sysrq.c +3
-0@@ -528,6 +528,11 @@ void __handle_sysrq(int key, bool check_mask)struct sysrq_key_op *op_p;int orig_log_level;int i; +int old_val; + +/* save the old panic printk flag */ The comment is not needed. It is obvious.
ok, will remove.
Quoted 5 lines
drivers/tty/sysrq.c void __handle_sysrq(int key, bool check_mask) · line 534
Quoted 2 lines
drivers/tty/sysrq.c +1
-0+ old_val = panic_suppress_printk;s/old_val/orig_panic_suppress_printk/ to follow
the naming oforig_log_level.
Ok.
Quoted 6 lines
drivers/tty/sysrq.c void __handle_sysrq(int key, bool check_mask) · line 535
Quoted 2 lines
drivers/tty/sysrq.c +1
-0+ panic_suppress_printk =1 ;We want to enable the messages in sysrq. This should be:
panic_suppress_printk = 0;
Yes, will do.
Quoted 11 lines
Quoted 11 lines
drivers/tty/sysrq.c +2
-0rcu_sysrq_start ();rcu_read_lock ();@@ -574,6 +579,8 @@ void __handle_sysrq(int key, bool check_mask) }rcu_read_unlock ();rcu_sysrq_end (); + + panic_suppress_printk = old_val; }
Quoted 11 lines
kernel/panic.c void panic(const char *fmt, ...) · line 330
Quoted 9 lines
drivers/tty/sysrq.c +0
-0void handle_sysrq (int key)kernel/panic.c +0
-0@@ -326,6 +328,7 @@ void panic(const char *fmt, ...) }#endif pr_emerg ("---[ end Kernel panic - not syncing: %s ]---\n" , buf);/* Do not scroll important messages with errors from blinking code. */
Will add the comment, with one minor chanage that the noisy messages may also come from other places.
Quoted 14 lines
kernel/printk/printk.c asmlinkage __visible int printk(const char *fmt, ...) · line 1988
Quoted 12 lines
kernel/panic.c +1
-0+ panic_suppress_printk =1 ;local_irq_enable ();for (i =0 ; ; i +=PANIC_TIMER_STEP ) {touch_softlockup_watchdog ();kernel/printk/printk.c +0
-0@@ -1987,6 +1987,9 @@ asmlinkage __visible int printk(const char *fmt, ...)va_list args;int r;/* Suppress messages from panic blinking code. */
Ditto.
Quoted 5 lines
kernel/printk/printk.c asmlinkage __visible int printk(const char *fmt, ...) · line 1991
Quoted 2 lines
kernel/printk/printk.c +2
-0+if (unlikely (panic_suppress_printk)) +return 0 ;This should go to
vprintk_emit()so that it works for allprintk()interfaces.
Ok, will do.
Thanks,
Feng
Quoted 8 lines
kernel/printk/printk.c asmlinkage __visible int printk(const char *fmt, ...) · line 1995
Quoted 5 lines
kernel/printk/printk.c +1
-0+va_start (args, fmt); r =vprintk_func (fmt, args);va_end (args);Best Regards,
Petr