sys_info: prevent duplicate backtraces
Subsystems (4)
v3 cover sys_info: prevent duplicate backtraces
Some callers handle SYS_INFO_ALL_BT themselves before calling sys_info(). When they strip that bit, an all_bt-only mask becomes zero and sys_info(0) falls back to kernel_si_mask, potentially duplicating output.
This series adds sys_info_with_filter() to filter specific bits without triggering the kernel_si_mask fallback.
Changes since v2:
- Use sys_info_with_filter() instead of sys_info_without_all_bt() per Petr's suggestion
- Filter applied at __sys_info() level to handle kernel_si_mask correctly
- Added panic.c conversion
Bradley Morgan (4):
sys_info: add helper for callers that print some sys_info on their own
watchdog: use sys_info_with_filter() to avoid duplicate backtraces powerpc/watchdog: use sys_info_with_filter() to avoid duplicate
backtraces
panic: use sys_info_with_filter() to avoid duplicate backtraces
arch/powerpc/kernel/watchdog.c | 12 ++++++++---- include/linux/sys_info.h | 1 + kernel/panic.c | 2 +- kernel/watchdog.c | 12 ++++++++---- lib/sys_info.c | 20 ++++++++++++++++++-- 5 files changed, 36 insertions(+), 11 deletions(-)
Signature · 1 line
2.53.0
PATCH v3 1/4 sys_info: add helper for callers that print some sys_info on their own
Some callers print some sys_info on their own before calling sys_info().
Add a helper which would allow to prevent a duplicated output.
It is a bit tricky because kernel_si_mask should be used only when the call-specific si_mask is empty. But the duplicated output must be prevented there as well.
--- include/linux/sys_info.h | 1 + lib/sys_info.c | 20 ++++++++++++++++++-- 2 files changed, 19 insertions(+), 2 deletions(-)
include/linux/sys_info.h +1 -0
@@ -18,6 +18,7 @@#define SYS_INFO_BLOCKED_TASKS 0x00000080void sys_info (unsigned long si_mask); +void sys_info_with_filter (unsigned long si_mask,unsigned long si_ignore_mask);unsigned long sys_info_parse_param (char *str);#ifdef CONFIG_SYSCTL
lib/sys_info.c +18 -2
@@ -136,8 +136,10 @@ static int __init sys_info_sysctl_init(void)subsys_initcall (sys_info_sysctl_init);#endif -static void __sys_info (unsigned long si_mask) +static void __sys_info (unsigned long si_mask,unsigned long si_ignore_mask) { + si_mask &= ~si_ignore_mask; +if (si_mask &SYS_INFO_TASKS )show_state ();
@@ -160,7 +162,21 @@ static void __sys_info(unsigned long si_mask)show_state_filter (TASK_UNINTERRUPTIBLE ); } +void sys_info_with_filter (unsigned long si_mask,unsigned long si_ignore_mask) +{ +unsigned long dump_mask = si_mask & ~si_ignore_mask; + +/* +* Do not fall back to kernel_si_mask when the caller context +* required only the ignored information. +*/ +if (si_mask && !dump_mask) +return ; + +__sys_info (dump_mask ? : kernel_si_mask, si_ignore_mask); +} +void sys_info (unsigned long si_mask) { -__sys_info (si_mask ? : kernel_si_mask); +sys_info_with_filter (si_mask,0 ); }
Signature · 1 line
2.53.0
PATCH v3 2/4 watchdog: use sys_info_with_filter() to avoid duplicate backtraces
In reply to Bradley MorganThe watchdog prints all CPU backtraces itself. When the watchdog mask contains only SYS_INFO_ALL_BT, stripping that bit leaves zero and sys_info(0) falls back to kernel_sys_info.
Use sys_info_with_filter() so an explicit all_bt mask does not request the global default.
--- kernel/watchdog.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
kernel/watchdog.c +8 -4
@@ -208,6 +208,7 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs) {int hardlockup_all_cpu_backtrace;unsigned int this_cpu; +unsigned long si_mask;unsigned long flags;if (per_cpu (watchdog_hardlockup_touched, cpu)) {
@@ -216,7 +217,8 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs)return ; } - hardlockup_all_cpu_backtrace = (hardlockup_si_mask &SYS_INFO_ALL_BT ) ? + si_mask =READ_ONCE (hardlockup_si_mask); + hardlockup_all_cpu_backtrace = (si_mask &SYS_INFO_ALL_BT ) ?1 : sysctl_hardlockup_all_cpu_backtrace;/* * Check for a hardlockup by making sure the CPU's timer
@@ -286,7 +288,7 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs)clear_bit_unlock (0 , &hard_lockup_nmi_warn); } -sys_info (hardlockup_si_mask & ~SYS_INFO_ALL_BT ); +sys_info_with_filter (si_mask,SYS_INFO_ALL_BT );if (hardlockup_panic)nmi_panic (regs,"Hard LOCKUP" );
@@ -798,6 +800,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)struct pt_regs *regs =get_irq_regs ();int softlockup_all_cpu_backtrace;int duration, thresh_count; +unsigned long si_mask;unsigned long flags;if (!watchdog_enabled)
@@ -809,7 +812,8 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)if (panic_in_progress ())return HRTIMER_NORESTART ; - softlockup_all_cpu_backtrace = (softlockup_si_mask &SYS_INFO_ALL_BT ) ? + si_mask =READ_ONCE (softlockup_si_mask); + softlockup_all_cpu_backtrace = (si_mask &SYS_INFO_ALL_BT ) ?1 : sysctl_softlockup_all_cpu_backtrace;watchdog_hardlockup_kick ();
@@ -900,7 +904,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) }add_taint (TAINT_SOFTLOCKUP ,LOCKDEP_STILL_OK ); -sys_info (softlockup_si_mask & ~SYS_INFO_ALL_BT ); +sys_info_with_filter (si_mask,SYS_INFO_ALL_BT ); thresh_count = duration /get_softlockup_thresh ();if (softlockup_panic && thresh_count >= softlockup_panic)
Signature · 1 line
2.53.0
PATCH v3 3/4 powerpc/watchdog: use sys_info_with_filter() to avoid duplicate backtraces
In reply to Bradley MorganThe powerpc watchdog prints all CPU backtraces itself. When the watchdog mask contains only SYS_INFO_ALL_BT, stripping that bit leaves zero and sys_info(0) falls back to kernel_sys_info.
Use sys_info_with_filter() so an explicit all_bt mask does not request the global default.
--- arch/powerpc/kernel/watchdog.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
arch/powerpc/kernel/watchdog.c +8 -4
@@ -201,6 +201,7 @@ static bool set_cpu_stuck(int cpu)static void watchdog_smp_panic (int cpu) {static cpumask_t wd_smp_cpus_ipi;// protected by reporting +unsigned long si_mask;unsigned long flags;u64 tb, last_reset;int c;
@@ -236,8 +237,9 @@ static void watchdog_smp_panic(int cpu)pr_emerg ("CPU %d TB:%lld, last SMP heartbeat TB:%lld (%lldms ago)\n" , cpu, tb, last_reset,tb_to_ns (tb - last_reset) /1000000 ); + si_mask =READ_ONCE (hardlockup_si_mask);if (sysctl_hardlockup_all_cpu_backtrace || - (hardlockup_si_mask &SYS_INFO_ALL_BT )) { + (si_mask &SYS_INFO_ALL_BT )) {trigger_allbutcpu_cpu_backtrace (cpu);cpumask_clear (&wd_smp_cpus_ipi); }else {
@@ -251,7 +253,7 @@ static void watchdog_smp_panic(int cpu) } } -sys_info (hardlockup_si_mask & ~SYS_INFO_ALL_BT ); +sys_info_with_filter (si_mask,SYS_INFO_ALL_BT );if (hardlockup_panic)nmi_panic (NULL ,"Hard LOCKUP" );
@@ -371,6 +373,7 @@ static void watchdog_timer_interrupt(int cpu)DEFINE_INTERRUPT_HANDLER_NMI (soft_nmi_interrupt) { +unsigned long si_mask;unsigned long flags;int cpu =raw_smp_processor_id ();u64 tb;
@@ -418,11 +421,12 @@ DEFINE_INTERRUPT_HANDLER_NMI(soft_nmi_interrupt)xchg (&__wd_nmi_output,1 );// see wd_lockup_ipi + si_mask =READ_ONCE (hardlockup_si_mask);if (sysctl_hardlockup_all_cpu_backtrace || - (hardlockup_si_mask &SYS_INFO_ALL_BT )) + (si_mask &SYS_INFO_ALL_BT ))trigger_allbutcpu_cpu_backtrace (cpu); -sys_info (hardlockup_si_mask & ~SYS_INFO_ALL_BT ); +sys_info_with_filter (si_mask,SYS_INFO_ALL_BT );if (hardlockup_panic)nmi_panic (regs,"Hard LOCKUP" );
Signature · 1 line
2.53.0
Petr Mladek
reviewerRe: [PATCH v3 3/4] powerpc/watchdog: use sys_info_with_filter() to avoid duplicate backtraces
Quoted 13 lines · Bradley Morgan
The powerpc watchdog prints all CPU backtraces itself. When the watchdog mask contains only
SYS_INFO_ALL_BT, stripping that bit leaves zero andsys_info(0)falls back tokernel_sys_info.Use
sys_info_with_filter()so an explicitall_btmask does not request the global default.arch/powerpc/kernel/watchdog.c +0
-0@@ -418,11 +421,12 @@ DEFINE_INTERRUPT_HANDLER_NMI(soft_nmi_interrupt)xchg (&__wd_nmi_output,1 );// see wd_lockup_ipi
arch/powerpc/kernel/watchdog.c DEFINE_INTERRUPT_HANDLER_NMI(soft_nmi_interrupt) · line 431
Quoted 10 lines
arch/powerpc/kernel/watchdog.c +3
-2+ si_mask =READ_ONCE (hardlockup_si_mask);if (sysctl_hardlockup_all_cpu_backtrace || - (hardlockup_si_mask &SYS_INFO_ALL_BT )) + (si_mask &SYS_INFO_ALL_BT ))trigger_allbutcpu_cpu_backtrace (cpu); -sys_info (hardlockup_si_mask & ~SYS_INFO_ALL_BT ); +sys_info_with_filter (si_mask,SYS_INFO_ALL_BT );if (hardlockup_panic)nmi_panic (regs,"Hard LOCKUP" );
I thought more about it and it is even more complicated.
Even if we prevent the duplicated output with sys_info_with_filter() here. Then nmi_panic() might still trigger it once again.
We could say that this patch is a step in the right direction and fix the other problem later. But I am not sure. We might need a completely different approach and this is just a step aside.
And there is another problem in the panic() code. I am going to comment in it in the 4th patch.
Best Regards,
Petr
PATCH v3 4/4 panic: use sys_info_with_filter() to avoid duplicate backtraces
In reply to Bradley Morganpanic_other_cpus_shutdown() handles SYS_INFO_ALL_BT before stopping the other CPUs. Do not ask sys_info() to handle that bit again later in the panic path.
Use sys_info_with_filter() so panic_print=all_bt does not request more output after the CPUs are stopped.
--- kernel/panic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/panic.c +1 -1
@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)*/ atomic_notifier_call_chain (&panic_notifier_list,0 , buf); -sys_info (panic_print); +sys_info_with_filter (panic_print,SYS_INFO_ALL_BT );kmsg_dump_desc (KMSG_DUMP_PANIC , buf);
Signature · 1 line
2.53.0
Petr Mladek
reviewerRe: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces
Quoted 19 lines · Bradley Morgan
panic_other_cpus_shutdown()handlesSYS_INFO_ALL_BTbefore stopping the other CPUs. Do not asksys_info()to handle that bit again later in the panic path.Use
sys_info_with_filter()so panic_print=all_bt does not request more output after the CPUs are stopped.Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys info on system lockup")Signed-off-by Bradley Morgan <[email protected]>--- kernel/panic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/panic.c +0
-0
kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683
Quoted 5 lines
kernel/panic.c +1
-1@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)*/ atomic_notifier_call_chain (&panic_notifier_list,0 , buf); -sys_info (panic_print); +sys_info_with_filter (panic_print,SYS_INFO_ALL_BT );
Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?
They might be printed by:
static void panic_other_cpus_shutdown(bool crash_kexec)
{
if (panic_print & SYS_INFO_ALL_BT)
panic_trigger_all_cpu_backtrace();[...]
}
But it checks only "panic_print" variable. It won't do anything when (panic_print == 0).
In this case, we might still want to print the backraces when SYS_INFO_ALL_BT is set in kernel_si_info.
kernel/panic.c void vpanic(const char *fmt, va_list args) · line 685
Quoted 1 line
kernel/panic.c +0
-0kmsg_dump_desc (KMSG_DUMP_PANIC , buf);
Of course, we might fix panic_other_cpus_shutdown() to check also kernel_si_info.
But it all becomes very hairy. We have several levels:
+ watchdog-all_bt-specific option, e.g. sysctl_hardlockup_all_cpu_backtrace
+ watchdog-specific si_info preferences, e.g. hardlockup_si_mask
+ panic-specific si_info: panic_print
+ universal fallback for any layer: kernel_si_info
Now, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.
I think about another approach. The word "waterfall" comes to my mind. Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.
All the si_info actions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.
I am going to think more about it.
Please, do not send v4 until the discussion settles!
Best Regards,
Petr
Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces
Quoted 66 lines · Petr Mladek
Quoted 8 lines · Bradley Morgan
panic_other_cpus_shutdown()handlesSYS_INFO_ALL_BTbefore stopping the other CPUs. Do not asksys_info()to handle that bit again later in the panic path.Use
sys_info_with_filter()so panic_print=all_bt does not request more output after the CPUs are stopped.Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys info onsystem lockup")
Quoted 11 lines
Signed-off-by Bradley Morgan <[email protected]>--- kernel/panic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/panic.c +0
-0kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683
Quoted 5 lines
kernel/panic.c +1
-1@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)*/ atomic_notifier_call_chain (&panic_notifier_list,0 , buf); -sys_info (panic_print); +sys_info_with_filter (panic_print,SYS_INFO_ALL_BT );Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?
They might be printed by:
static void
panic_other_cpus_shutdown(bool crash_kexec){ if (panic_print & SYS_INFO_ALL_BT) panic_trigger_all_cpu_backtrace();[...]
}But it checks only "
panic_print" variable. It won't do anything when (panic_print== 0).In this case, we might still want to print the backraces when
SYS_INFO_ALL_BTis set inkernel_si_info.kernel/panic.c void vpanic(const char *fmt, va_list args) · line 685
Quoted 1 line
kernel/panic.c +0
-0kmsg_dump_desc (KMSG_DUMP_PANIC , buf);Of course, we might fix
panic_other_cpus_shutdown()to check alsokernel_si_info.But it all becomes very hairy. We have several levels:
+ watchdog-all_bt-specific option, e.g.
sysctl_hardlockup_all_cpu_backtrace+ watchdog-specific
si_infopreferences, e.g.hardlockup_si_mask+ panic-specific
si_info:panic_print+ universal fallback for any layer:
kernel_si_infoNow, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.
Quoted 8 lines
I think about another approach. The word "waterfall" comes to my mind. Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.
All the
si_infoactions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.
Not a bad idea.
Quoted 3 lines
I am going to think more about it.
Please, do not send v4 until the discussion settles!
I'll hold on V4.
When you've finished discussing, could I have your suggested patch? if I think there is issues. I'll fix it.
Quoted 3 lines
Best Regards,
Petr
Thanks!
Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces
In reply to Petr MladekQuoted 36 lines · Petr Mladek
Quoted 19 lines · Bradley Morgan
panic_other_cpus_shutdown()handlesSYS_INFO_ALL_BTbefore stopping the other CPUs. Do not asksys_info()to handle that bit again later in the panic path.Use
sys_info_with_filter()so panic_print=all_bt does not request more output after the CPUs are stopped.Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys info on system lockup")Signed-off-by Bradley Morgan <[email protected]>--- kernel/panic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/panic.c +0
-0kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683
Quoted 5 lines
kernel/panic.c +1
-1@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)*/ atomic_notifier_call_chain (&panic_notifier_list,0 , buf); -sys_info (panic_print); +sys_info_with_filter (panic_print,SYS_INFO_ALL_BT );Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?
They might be printed by:
static void
panic_other_cpus_shutdown(bool crash_kexec){ if (panic_print & SYS_INFO_ALL_BT) panic_trigger_all_cpu_backtrace();
Quoted 8 lines
[...]
}But it checks only "
panic_print" variable. It won't do anything when (panic_print== 0).In this case, we might still want to print the backraces when
SYS_INFO_ALL_BTis set inkernel_si_info.
Yep.
Quoted 12 lines
kernel/panic.c void vpanic(const char *fmt, va_list args) · line 685
Quoted 2 lines
kernel/panic.c +0
-0kmsg_dump_desc (KMSG_DUMP_PANIC , buf);Of course, we might fix
panic_other_cpus_shutdown()to check alsokernel_si_info.But it all becomes very hairy. We have several levels:
+ watchdog-all_bt-specific option, e.g.
sysctl_hardlockup_all_cpu_backtrace+ watchdog-specific
si_infopreferences, e.g.hardlockup_si_mask
Quoted 8 lines
+ panic-specific
si_info:panic_print+ universal fallback for any layer:
kernel_si_infoNow, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.
Agree :)
Quoted 4 lines
I think about another approach. The word "waterfall" comes to my mind. Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.
When initially reviewing V2's 4th patch, I thought about the 'panic_this_cpu_backtrace_printed', but it's a local variable which records the state.
Quoted 3 lines
All the
si_infoactions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.
IIUC, panic case is kind of special, as it has to separate the 'sys_info()' op in different stage. Can we do a merge in the start of vpanic() by:
panic_print = panic_print ?: kernel_si_mask;
as a addon patch ?
Thanks,
Feng
Quoted 6 lines
I am going to think more about it.
Please, do not send v4 until the discussion settles!
Best Regards,
Petr
Petr Mladek
reviewerRe: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces
In reply to Petr MladekQuoted 70 lines · Petr Mladek
Quoted 19 lines · Bradley Morgan
panic_other_cpus_shutdown()handlesSYS_INFO_ALL_BTbefore stopping the other CPUs. Do not asksys_info()to handle that bit again later in the panic path.Use
sys_info_with_filter()so panic_print=all_bt does not request more output after the CPUs are stopped.Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys info on system lockup")Signed-off-by Bradley Morgan <[email protected]>--- kernel/panic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/panic.c +0
-0kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683
Quoted 5 lines
kernel/panic.c +1
-1@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)*/ atomic_notifier_call_chain (&panic_notifier_list,0 , buf); -sys_info (panic_print); +sys_info_with_filter (panic_print,SYS_INFO_ALL_BT );Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?
They might be printed by:
static void
panic_other_cpus_shutdown(bool crash_kexec){ if (panic_print & SYS_INFO_ALL_BT) panic_trigger_all_cpu_backtrace();[...]
}But it checks only "
panic_print" variable. It won't do anything when (panic_print== 0).In this case, we might still want to print the backraces when
SYS_INFO_ALL_BTis set inkernel_si_info.kernel/panic.c void vpanic(const char *fmt, va_list args) · line 684
Quoted 1 line
kernel/panic.c +0
-0kmsg_dump_desc (KMSG_DUMP_PANIC , buf);Of course, we might fix
panic_other_cpus_shutdown()to check alsokernel_si_info.But it all becomes very hairy. We have several levels:
+ watchdog-all_bt-specific option, e.g.
sysctl_hardlockup_all_cpu_backtrace+ watchdog-specific
si_infopreferences, e.g.hardlockup_si_mask+ panic-specific
si_info:panic_print+ universal fallback for any layer:
kernel_si_infoNow, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.
I think about another approach. The word "waterfall" comes to my mind. Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.
Quoted 5 lines
All the
si_infoactions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.I am going to think more about it.
I have created a POC using Gemini. I haven't tested it. But it looks acceptable. And the logic seems to be more straightforward.
One drawback is that it requires adding the _reset() call for all sys_info() callers. It is fine in principle but it might complicate back-porting because all changes have to be done in one patch.
But honestly, this is a nice to have fix. Most people could live happily without it.
From 3c66436d9978030845a96bfaedd6b914536e2ac4 Mon Sep 17 00:00:00 2001
From: Petr Mladek <[email protected]>
Date: Fri, 26 Jun 2026 13:55:41 +0200
Subject: [POC] sys_info: Introduce state-tracking APIs to prevent duplicate
backtraces
In watchdog, panic, and hung task detection scenarios, sys_info() can be called multiple times or alongside direct backtrace triggers like trigger_allbutcpu_cpu_backtrace(). This results in identical backtraces being dumped repeatedly from all CPUs, cluttering the kernel log and delaying or obscuring critical debug details.
Introduce a state tracking bitmask and associated helpers:
- sys_info_done(mask): Marks specific sys_info bits as already printed.
- sys_info_reset(): Resets the tracking state.
- sys_info_is_done(mask): Checks if all bits in the mask have been printed.
Update sys_info() to automatically filter out already printed bits using this state. Integrate these APIs with the generic hardlockup and softlockup watchdogs, the PowerPC watchdog, the hung task detector, and the panic core. This ensures that each piece of system information and backtrace output is printed at most once per lockup/panic event, and the state is reset cleanly when a lockup does not trigger a panic.
Races between sys_info() callers are ignored. It should be acceptable because the output from various watchdogs has never been synchronized. And panic() never returns.
--- arch/powerpc/kernel/watchdog.c | 13 ++++++++++--- include/linux/sys_info.h | 3 +++ kernel/hung_task.c | 2 ++ kernel/panic.c | 4 +++- kernel/watchdog.c | 10 ++++++++-- lib/sys_info.c | 30 +++++++++++++++++++++++++++++- 6 files changed, 55 insertions(+), 7 deletions(-)
arch/powerpc/kernel/watchdog.c +10 -3
@@ -239,6 +239,7 @@ static void watchdog_smp_panic(int cpu)if (sysctl_hardlockup_all_cpu_backtrace || (hardlockup_si_mask &SYS_INFO_ALL_BT )) {trigger_allbutcpu_cpu_backtrace (cpu); +sys_info_done (SYS_INFO_ALL_BT );cpumask_clear (&wd_smp_cpus_ipi); }else {/*
@@ -251,10 +252,12 @@ static void watchdog_smp_panic(int cpu) } } -sys_info (hardlockup_si_mask & ~SYS_INFO_ALL_BT ); +sys_info (hardlockup_si_mask);if (hardlockup_panic)nmi_panic (NULL ,"Hard LOCKUP" ); +sys_info_reset (); +wd_end_reporting ();return ;
@@ -419,13 +422,17 @@ DEFINE_INTERRUPT_HANDLER_NMI(soft_nmi_interrupt)xchg (&__wd_nmi_output,1 );// see wd_lockup_ipi if (sysctl_hardlockup_all_cpu_backtrace || - (hardlockup_si_mask &SYS_INFO_ALL_BT )) + (hardlockup_si_mask &SYS_INFO_ALL_BT )) {trigger_allbutcpu_cpu_backtrace (cpu); +sys_info_done (SYS_INFO_ALL_BT ); + } -sys_info (hardlockup_si_mask & ~SYS_INFO_ALL_BT ); +sys_info (hardlockup_si_mask);if (hardlockup_panic)nmi_panic (regs,"Hard LOCKUP" ); +sys_info_reset (); +wd_end_reporting (); }/*
include/linux/sys_info.h +3 -0
@@ -18,6 +18,9 @@#define SYS_INFO_BLOCKED_TASKS 0x00000080void sys_info (unsigned long si_mask); +void sys_info_done (unsigned long si_mask); +void sys_info_reset (void ); +bool sys_info_is_done (unsigned long si_mask);unsigned long sys_info_parse_param (char *str);#ifdef CONFIG_SYSCTL
kernel/hung_task.c +2 -0
@@ -354,6 +354,8 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)if (hung_task_call_panic)panic ("hung_task: blocked tasks" ); + +sys_info_reset (); } static long hung_timeout_jiffies(unsigned long last_checked,
kernel/panic.c +3 -1
@@ -550,8 +550,10 @@ static void panic_trigger_all_cpu_backtrace(void)*/ static void panic_other_cpus_shutdown (bool crash_kexec) { -if (panic_print &SYS_INFO_ALL_BT ) +if ((panic_print &SYS_INFO_ALL_BT ) && !sys_info_is_done (SYS_INFO_ALL_BT )) {panic_trigger_all_cpu_backtrace (); +sys_info_done (SYS_INFO_ALL_BT ); + }/* * Note that smp_send_stop() is the usual SMP shutdown function,
kernel/watchdog.c +8 -2
@@ -282,14 +282,17 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs)if (hardlockup_all_cpu_backtrace) {trigger_allbutcpu_cpu_backtrace (cpu); +sys_info_done (SYS_INFO_ALL_BT );if (!hardlockup_panic)clear_bit_unlock (0 , &hard_lockup_nmi_warn); } -sys_info (hardlockup_si_mask & ~SYS_INFO_ALL_BT ); +sys_info (hardlockup_si_mask);if (hardlockup_panic)nmi_panic (regs,"Hard LOCKUP" ); +sys_info_reset (); +per_cpu (watchdog_hardlockup_warned, cpu) = true; }
@@ -895,16 +898,19 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)if (softlockup_all_cpu_backtrace) {trigger_allbutcpu_cpu_backtrace (smp_processor_id ()); +sys_info_done (SYS_INFO_ALL_BT );if (!softlockup_panic)clear_bit_unlock (0 , &soft_lockup_nmi_warn); }add_taint (TAINT_SOFTLOCKUP ,LOCKDEP_STILL_OK ); -sys_info (softlockup_si_mask & ~SYS_INFO_ALL_BT ); +sys_info (softlockup_si_mask); thresh_count = duration /get_softlockup_thresh ();if (softlockup_panic && thresh_count >= softlockup_panic)panic ("softlockup: hung tasks" ); + +sys_info_reset (); }return HRTIMER_RESTART ;
lib/sys_info.c +29 -1
@@ -160,7 +160,35 @@ static void __sys_info(unsigned long si_mask)show_state_filter (TASK_UNINTERRUPTIBLE ); } +static unsigned long sys_info_done_mask; + +void sys_info_done (unsigned long si_mask) +{ + sys_info_done_mask |= si_mask; +} + +void sys_info_reset (void ) +{ + sys_info_done_mask =0 ; +} + +bool sys_info_is_done (unsigned long si_mask) +{ +return (sys_info_done_mask & si_mask) == si_mask; +} +void sys_info (unsigned long si_mask) { -__sys_info (si_mask ? : kernel_si_mask); +unsigned long mask; + +if (si_mask) + mask = si_mask & ~sys_info_done_mask; +else + mask = kernel_si_mask & ~sys_info_done_mask; + +if (!mask) +return ; + +__sys_info (mask); +sys_info_done (mask); }
Signature · 1 line
2.54.0
Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces
Quoted 117 lines · Petr Mladek
Quoted 2 lines · Petr Mladek
Quoted 1 line · Bradley Morgan
panic_other_cpus_shutdown()handlesSYS_INFO_ALL_BTbefore stoppingthe
Quoted 1 line
Quoted 1 line
other CPUs. Do not ask
sys_info()to handle that bit again later inthe
Quoted 6 lines
Quoted 6 lines
panic path.
Use
sys_info_with_filter()so panic_print=all_bt does not request more output after the CPUs are stopped.Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys infoon system lockup")
Quoted 36 lines
Quoted 11 lines
Signed-off-by Bradley Morgan <[email protected]>--- kernel/panic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/panic.c +0
-0kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683
Quoted 5 lines
kernel/panic.c +1
-1@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)*/ atomic_notifier_call_chain (&panic_notifier_list,0 , buf); -sys_info (panic_print); +sys_info_with_filter (panic_print,SYS_INFO_ALL_BT );Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?
They might be printed by:
static void
panic_other_cpus_shutdown(bool crash_kexec){ if (panic_print & SYS_INFO_ALL_BT) panic_trigger_all_cpu_backtrace();[...]
}But it checks only "
panic_print" variable. It won't do anything when (panic_print== 0).In this case, we might still want to print the backraces when
SYS_INFO_ALL_BTis set inkernel_si_info.Quoted 8 lines
kernel/panic.c void vpanic(const char *fmt, va_list args) · line 684
Quoted 1 line
kernel/panic.c +0
-0kmsg_dump_desc (KMSG_DUMP_PANIC , buf);Of course, we might fix
panic_other_cpus_shutdown()to check alsokernel_si_info.But it all becomes very hairy. We have several levels:
+ watchdog-all_bt-specific option, e.g.
sysctl_hardlockup_all_cpu_backtraceQuoted 17 lines
+ watchdog-specific
si_infopreferences, e.g.hardlockup_si_mask+ panic-specific
si_info:panic_print+ universal fallback for any layer:
kernel_si_infoNow, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.
I think about another approach. The word "waterfall" comes to my mind. Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.
Quoted 5 lines
All the
si_infoactions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.I am going to think more about it.
I have created a POC using Gemini. I haven't tested it. But it looks acceptable. And the logic seems to be more straightforward.
One drawback is that it requires adding the
_reset()call for allsys_info()callers. It is fine in principle but it might complicate back-porting because all changes have to be done in one patch.But honestly, this is a nice to have fix. Most people could live happily without it.
From
3c66436d9978030845a96bfaedd6b914536e2ac4Mon Sep 17 00:00:00 2001
From: Petr Mladek <[email protected]>
Date: Fri, 26 Jun 2026 13:55:41 +0200
Subject: [POC]sys_info: Introduce state-tracking APIs to prevent duplicate backtracesIn watchdog, panic, and hung task detection scenarios,
sys_info()can be called multiple times or alongside direct backtrace triggers liketrigger_allbutcpu_cpu_backtrace(). This results in identical backtraces being dumped repeatedly from all CPUs, cluttering the kernel log and delaying or obscuring critical debug details.Introduce a state tracking bitmask and associated helpers:
-sys_info_done(mask): Marks specificsys_infobits as already printed.
-sys_info_reset(): Resets the tracking state.
-sys_info_is_done(mask): Checks if all bits in the mask have been printed.Update
sys_info()to automatically filter out already printed bits using this state. Integrate these APIs with the generic hardlockup and softlockup watchdogs, the PowerPC watchdog, the hung task detector, and the panic core. This ensures that each piece of system information and backtrace output is printed at most once per lockup/panic event, and the state is reset cleanly when a lockup does not trigger a panic.
Quoted 5 lines
Races between
sys_info()callers are ignored. It should be acceptable because the output from various watchdogs has never been synchronized. Andpanic()never returns.Assisted-by gemini-1.5-flash ?
Why not use gemini 3.5 flash?
I can try if you want.
Could I have the prompt you used? :)
Quoted 175 lines
Signed-off-by Petr Mladek <[email protected]>---
arch/powerpc/kernel/watchdog.c | 13 ++++++++++--- include/linux/sys_info.h | 3 +++ kernel/hung_task.c | 2 ++ kernel/panic.c | 4 +++- kernel/watchdog.c | 10 ++++++++-- lib/sys_info.c | 30 +++++++++++++++++++++++++++++- 6 files changed, 55 insertions(+), 7 deletions(-)arch/powerpc/kernel/watchdog.c +10
-3@@ -239,6 +239,7 @@ static void watchdog_smp_panic(int cpu)if (sysctl_hardlockup_all_cpu_backtrace || (hardlockup_si_mask &SYS_INFO_ALL_BT )) {trigger_allbutcpu_cpu_backtrace (cpu); +sys_info_done (SYS_INFO_ALL_BT );cpumask_clear (&wd_smp_cpus_ipi); }else {/* @@ -251,10 +252,12 @@ static void watchdog_smp_panic(int cpu) } } -sys_info (hardlockup_si_mask & ~SYS_INFO_ALL_BT ); +sys_info (hardlockup_si_mask);if (hardlockup_panic)nmi_panic (NULL ,"Hard LOCKUP" ); +sys_info_reset (); +wd_end_reporting ();return ;@@ -419,13 +422,17 @@ DEFINE_INTERRUPT_HANDLER_NMI(soft_nmi_interrupt)xchg (&__wd_nmi_output,1 );// see wd_lockup_ipi if (sysctl_hardlockup_all_cpu_backtrace || - (hardlockup_si_mask &SYS_INFO_ALL_BT )) + (hardlockup_si_mask &SYS_INFO_ALL_BT )) {trigger_allbutcpu_cpu_backtrace (cpu); +sys_info_done (SYS_INFO_ALL_BT ); + } -sys_info (hardlockup_si_mask & ~SYS_INFO_ALL_BT ); +sys_info (hardlockup_si_mask);if (hardlockup_panic)nmi_panic (regs,"Hard LOCKUP" ); +sys_info_reset (); +wd_end_reporting (); }/* include/linux/sys_info.h +3
-0@@ -18,6 +18,9 @@#define SYS_INFO_BLOCKED_TASKS 0x00000080void sys_info (unsigned long si_mask); +void sys_info_done (unsigned long si_mask); +void sys_info_reset (void ); +bool sys_info_is_done (unsigned long si_mask);unsigned long sys_info_parse_param (char *str);#ifdef CONFIG_SYSCTL kernel/hung_task.c +2
-0@@ -354,6 +354,8 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)if (hung_task_call_panic)panic ("hung_task: blocked tasks" ); + +sys_info_reset (); } static long hung_timeout_jiffies(unsigned long last_checked,kernel/panic.c +3
-1@@ -550,8 +550,10 @@ static void panic_trigger_all_cpu_backtrace(void)*/ static void panic_other_cpus_shutdown (bool crash_kexec) { -if (panic_print &SYS_INFO_ALL_BT ) +if ((panic_print &SYS_INFO_ALL_BT ) && !sys_info_is_done (SYS_INFO_ALL_BT )) {panic_trigger_all_cpu_backtrace (); +sys_info_done (SYS_INFO_ALL_BT ); + }/* * Note that smp_send_stop() is the usual SMP shutdown function, kernel/watchdog.c +8
-2@@ -282,14 +282,17 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs)if (hardlockup_all_cpu_backtrace) {trigger_allbutcpu_cpu_backtrace (cpu); +sys_info_done (SYS_INFO_ALL_BT );if (!hardlockup_panic)clear_bit_unlock (0 , &hard_lockup_nmi_warn); } -sys_info (hardlockup_si_mask & ~SYS_INFO_ALL_BT ); +sys_info (hardlockup_si_mask);if (hardlockup_panic)nmi_panic (regs,"Hard LOCKUP" ); +sys_info_reset (); +per_cpu (watchdog_hardlockup_warned, cpu) = true; }@@ -895,16 +898,19 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)if (softlockup_all_cpu_backtrace) {trigger_allbutcpu_cpu_backtrace (smp_processor_id ()); +sys_info_done (SYS_INFO_ALL_BT );if (!softlockup_panic)clear_bit_unlock (0 , &soft_lockup_nmi_warn); }add_taint (TAINT_SOFTLOCKUP ,LOCKDEP_STILL_OK ); -sys_info (softlockup_si_mask & ~SYS_INFO_ALL_BT ); +sys_info (softlockup_si_mask); thresh_count = duration /get_softlockup_thresh ();if (softlockup_panic && thresh_count >= softlockup_panic)panic ("softlockup: hung tasks" ); + +sys_info_reset (); }return HRTIMER_RESTART ;lib/sys_info.c +20
-1@@ -160,7 +160,35 @@ static void __sys_info(unsigned long si_mask)show_state_filter (TASK_UNINTERRUPTIBLE ); } +static unsigned long sys_info_done_mask; + +void sys_info_done (unsigned long si_mask) +{ + sys_info_done_mask |= si_mask; +} + +void sys_info_reset (void ) +{ + sys_info_done_mask =0 ; +} + +bool sys_info_is_done (unsigned long si_mask) +{ +return (sys_info_done_mask & si_mask) == si_mask; +} +void sys_info (unsigned long si_mask) { -__sys_info (si_mask ? : kernel_si_mask); +unsigned long mask; + +if (si_mask)
lib/sys_info.c static void __sys_info(unsigned long si_mask) · line 194
Quoted 10 lines
lib/sys_info.c +9
-0+ mask = si_mask & ~sys_info_done_mask; + else + mask = kernel_si_mask & ~sys_info_done_mask; + +if (!mask) +return ; + +__sys_info (mask); +sys_info_done (mask); }
Thanks!
Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces
Quoted 317 lines · Bradley Morgan
Quoted 9 lines · Petr Mladek
Quoted 2 lines · Petr Mladek
Quoted 1 line · Bradley Morgan
panic_other_cpus_shutdown()handlesSYS_INFO_ALL_BTbefore stoppingthe
Quoted 1 line
Quoted 1 line
other CPUs. Do not ask
sys_info()to handle that bit again later inthe
Quoted 3 lines
Quoted 3 lines
panic path.
Use
sys_info_with_filter()so panic_print=all_bt does not requestmore
Quoted 81 lines
Quoted 3 lines
Quoted 3 lines
output after the CPUs are stopped.
Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys infoon system lockup")
Quoted 36 lines
Quoted 11 lines
Signed-off-by Bradley Morgan <[email protected]>--- kernel/panic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/panic.c +0
-0kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683
Quoted 5 lines
kernel/panic.c +1
-1@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)*/ atomic_notifier_call_chain (&panic_notifier_list,0 , buf); -sys_info (panic_print); +sys_info_with_filter (panic_print,SYS_INFO_ALL_BT );Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?
They might be printed by:
static void
panic_other_cpus_shutdown(bool crash_kexec){ if (panic_print & SYS_INFO_ALL_BT) panic_trigger_all_cpu_backtrace();[...]
}But it checks only "
panic_print" variable. It won't do anything when (panic_print== 0).In this case, we might still want to print the backraces when
SYS_INFO_ALL_BTis set inkernel_si_info.Quoted 8 lines
kernel/panic.c void vpanic(const char *fmt, va_list args) · line 684
Quoted 1 line
kernel/panic.c +0
-0kmsg_dump_desc (KMSG_DUMP_PANIC , buf);Of course, we might fix
panic_other_cpus_shutdown()to check alsokernel_si_info.But it all becomes very hairy. We have several levels:
+ watchdog-all_bt-specific option, e.g.
sysctl_hardlockup_all_cpu_backtraceQuoted 17 lines
+ watchdog-specific
si_infopreferences, e.g.hardlockup_si_mask+ panic-specific
si_info:panic_print+ universal fallback for any layer:
kernel_si_infoNow, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.
I think about another approach. The word "waterfall" comes to my mind. Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.
Quoted 5 lines
All the
si_infoactions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.I am going to think more about it.
I have created a POC using Gemini. I haven't tested it. But it looks acceptable. And the logic seems to be more straightforward.
One drawback is that it requires adding the
_reset()call for allsys_info()callers. It is fine in principle but it might complicate back-porting because all changes have to be done in one patch.Quoted 7 lines
But honestly, this is a nice to have fix. Most people could live happily without it.
From
3c66436d9978030845a96bfaedd6b914536e2ac4Mon Sep 17 00:00:00 2001
From: Petr Mladek <[email protected]>
Date: Fri, 26 Jun 2026 13:55:41 +0200
Subject: [POC]sys_info: Introduce state-tracking APIs to preventduplicate
Quoted 8 lines
backtraces
In watchdog, panic, and hung task detection scenarios,
sys_info()can be called multiple times or alongside direct backtrace triggers liketrigger_allbutcpu_cpu_backtrace(). This results in identical backtraces being dumped repeatedly from all CPUs, cluttering the kernel log and delaying or obscuring critical debug details.Quoted 4 lines
Introduce a state tracking bitmask and associated helpers:
-sys_info_done(mask): Marks specificsys_infobits as already printed.
-sys_info_reset(): Resets the tracking state.
-sys_info_is_done(mask): Checks if all bits in the mask have beenprinted.
Quoted 8 lines
Update
sys_info()to automatically filter out already printed bits using this state. Integrate these APIs with the generic hardlockup and softlockup watchdogs, the PowerPC watchdog, the hung task detector, and the panic core. This ensures that each piece of system information and backtrace output is printed at most once per lockup/panic event, and the state is reset cleanly when a lockup does not trigger a panic.Quoted 5 lines
Races between
sys_info()callers are ignored. It should be acceptable because the output from various watchdogs has never been synchronized. Andpanic()never returns.Assisted-by gemini-1.5-flash ?Why not use gemini 3.5 flash?
I can try if you want.
Could I have the prompt you used? :)
Review
Quoted 11 lines
Signed-off-by Petr Mladek <[email protected]>---
arch/powerpc/kernel/watchdog.c | 13 ++++++++++--- include/linux/sys_info.h | 3 +++ kernel/hung_task.c | 2 ++ kernel/panic.c | 4 +++- kernel/watchdog.c | 10 ++++++++-- lib/sys_info.c | 30 +++++++++++++++++++++++++++++- 6 files changed, 55 insertions(+), 7 deletions(-)(quoted diff) +0
-0
b/arch/powerpc/kernel/watchdog.ckernel/hung_task.c static void check_hung_uninterruptible_tasks(unsigned
Quoted 64 lines
index
c40c69368476..0eab7894b9dc100644arch/powerpc/kernel/watchdog.c +10
-3@@ -239,6 +239,7 @@ static void watchdog_smp_panic(int cpu)if (sysctl_hardlockup_all_cpu_backtrace || (hardlockup_si_mask &SYS_INFO_ALL_BT )) {trigger_allbutcpu_cpu_backtrace (cpu); +sys_info_done (SYS_INFO_ALL_BT );cpumask_clear (&wd_smp_cpus_ipi); }else {/* @@ -251,10 +252,12 @@ static void watchdog_smp_panic(int cpu) } } -sys_info (hardlockup_si_mask & ~SYS_INFO_ALL_BT ); +sys_info (hardlockup_si_mask);if (hardlockup_panic)nmi_panic (NULL ,"Hard LOCKUP" ); +sys_info_reset (); +wd_end_reporting ();return ;@@ -419,13 +422,17 @@ DEFINE_INTERRUPT_HANDLER_NMI(soft_nmi_interrupt)xchg (&__wd_nmi_output,1 );// see wd_lockup_ipi if (sysctl_hardlockup_all_cpu_backtrace || - (hardlockup_si_mask &SYS_INFO_ALL_BT )) + (hardlockup_si_mask &SYS_INFO_ALL_BT )) {trigger_allbutcpu_cpu_backtrace (cpu); +sys_info_done (SYS_INFO_ALL_BT ); + } -sys_info (hardlockup_si_mask & ~SYS_INFO_ALL_BT ); +sys_info (hardlockup_si_mask);if (hardlockup_panic)nmi_panic (regs,"Hard LOCKUP" ); +sys_info_reset (); +wd_end_reporting (); }/* include/linux/sys_info.h +3
-0@@ -18,6 +18,9 @@#define SYS_INFO_BLOCKED_TASKS 0x00000080void sys_info (unsigned long si_mask); +void sys_info_done (unsigned long si_mask); +void sys_info_reset (void ); +bool sys_info_is_done (unsigned long si_mask);unsigned long sys_info_parse_param (char *str);#ifdef CONFIG_SYSCTL kernel/hung_task.c +0
-0@@ -354,6 +354,8 @@ static void check_hung_uninterruptible_tasks(unsignedlong timeout)
kernel/watchdog.c void watchdog_hardlockup_check(unsigned int cpu,
Quoted 29 lines
kernel/hung_task.c +2
-0if (hung_task_call_panic)panic ("hung_task: blocked tasks" ); + +sys_info_reset (); } static long hung_timeout_jiffies(unsigned long last_checked,kernel/panic.c +3
-1@@ -550,8 +550,10 @@ static void panic_trigger_all_cpu_backtrace(void)*/ static void panic_other_cpus_shutdown (bool crash_kexec) { -if (panic_print &SYS_INFO_ALL_BT ) +if ((panic_print &SYS_INFO_ALL_BT ) && !sys_info_is_done (SYS_INFO_ALL_BT )) {panic_trigger_all_cpu_backtrace (); +sys_info_done (SYS_INFO_ALL_BT ); + }/* * Note that smp_send_stop() is the usual SMP shutdown function, kernel/watchdog.c +0
-0@@ -282,14 +282,17 @@ void watchdog_hardlockup_check(unsigned int cpu,struct
pt_regs*regs)kernel/watchdog.c static enum hrtimer_restart
Quoted 19 lines
kernel/watchdog.c +4
-1if (hardlockup_all_cpu_backtrace) {trigger_allbutcpu_cpu_backtrace (cpu); +sys_info_done (SYS_INFO_ALL_BT );if (!hardlockup_panic)clear_bit_unlock (0 , &hard_lockup_nmi_warn); } -sys_info (hardlockup_si_mask & ~SYS_INFO_ALL_BT ); +sys_info (hardlockup_si_mask);if (hardlockup_panic)nmi_panic (regs,"Hard LOCKUP" ); +sys_info_reset (); +per_cpu (watchdog_hardlockup_warned, cpu) = true; }@@ -895,16 +898,19 @@ static enum hrtimer_restart
watchdog_timer_fn(struct hrtimer *hrtimer)Quoted 56 lines
kernel/watchdog.c +4
-1if (softlockup_all_cpu_backtrace) {trigger_allbutcpu_cpu_backtrace (smp_processor_id ()); +sys_info_done (SYS_INFO_ALL_BT );if (!softlockup_panic)clear_bit_unlock (0 , &soft_lockup_nmi_warn); }add_taint (TAINT_SOFTLOCKUP ,LOCKDEP_STILL_OK ); -sys_info (softlockup_si_mask & ~SYS_INFO_ALL_BT ); +sys_info (softlockup_si_mask); thresh_count = duration /get_softlockup_thresh ();if (softlockup_panic && thresh_count >= softlockup_panic)panic ("softlockup: hung tasks" ); + +sys_info_reset (); }return HRTIMER_RESTART ;lib/sys_info.c +25
-1@@ -160,7 +160,35 @@ static void __sys_info(unsigned long si_mask)show_state_filter (TASK_UNINTERRUPTIBLE ); } +static unsigned long sys_info_done_mask; + +void sys_info_done (unsigned long si_mask) +{ + sys_info_done_mask |= si_mask; +} + +void sys_info_reset (void ) +{ + sys_info_done_mask =0 ; +} + +bool sys_info_is_done (unsigned long si_mask) +{ +return (sys_info_done_mask & si_mask) == si_mask; +} +void sys_info (unsigned long si_mask) { -__sys_info (si_mask ? : kernel_si_mask); +unsigned long mask; + +if (si_mask) + mask = si_mask & ~sys_info_done_mask; +else + mask = kernel_si_mask & ~sys_info_done_mask; + +if (!mask)
Quoted 8 lines
lib/sys_info.c static void __sys_info(unsigned long si_mask) · line 194
Quoted 6 lines
lib/sys_info.c +4
-0+return ; + +__sys_info (mask); +sys_info_done (mask); }Thanks!
Hmm.. new idea
kernel/dump_filter.c ?
What this file could do is to handle a generic lockup state machine so any subsystem can log what it already dumped?
I know it may bloat, but it's better then cramming fixes in.
What do you guys think? Maybe we could start a RFC for this?
Thanks!
Petr Mladek
reviewerRe: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces
Quoted 31 lines · Bradley Morgan
Quoted 25 lines · Bradley Morgan
Quoted 23 lines · Petr Mladek
Quoted 4 lines · Petr Mladek
On Thu 2026-06-25 15:25:58, Bradley Morgan wrote:
But it all becomes very hairy. We have several levels:+ watchdog-all_bt-specific option, e.g.
sysctl_hardlockup_all_cpu_backtraceQuoted 17 lines
+ watchdog-specific
si_infopreferences, e.g.hardlockup_si_mask+ panic-specific
si_info:panic_print+ universal fallback for any layer:
kernel_si_infoNow, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.
I think about another approach. The word "waterfall" comes to my mind. Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
All the
si_infoactions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.
Quoted 8 lines
Hmm.. new idea
kernel/dump_filter.c?What this file could do is to handle a generic lockup state machine so any subsystem can log what it already dumped?
I know it may bloat, but it's better then cramming fixes in.
I am not sure what exactly you would like to achieve but it sounds a bit scary ;-)
Anyway, we should not synchronize the watchdog reports against each other, definitely. They are running in non-compatible contexts (task vs interrupt vs NMI). Also we should not add any locking because they usually print something when the system has enough troubles.
Also I think that it is not worth preventing duplicated backtraces or reports from a single CPU. IMHO, it is not a big problem in practice.
So, we are down to large reports, like backtraces from all CPUs, timers, locks, ... which are handled by sys_info(). So, I think that it should be enough to handle this inside the sys_info() API.
I do not want to say that my proposal was the best solution. I am sure that there are better ones. But we need to consider the gain vs. complexity.
Honestly, I am already a bit scared by the complexity which we the sys_info() API added. And it is hard to imagine that adding another API would make it easier. But I might be wrong.
Instead, it might make sense to integrate the conflicting subsystem-specific calls under the sys_info() API. I mean that, for example watchdog_hardlockup_check() won't call trigger_allbutcpu_cpu_backtrace() directly but it would call it via sys_info() API so that sys_info() could keep track of it. Something like:
void sys_info_allbutcpu_bt(int cpu)
{
trigger_allbutcpu_cpu_backtrace(cpu);
/*
* The caller likely printed backtrace of the given @cpu
* on its own. Prevent duplicate backtraces from all
* CPUs with potential next sys_info() call.
*/
sys_info_done(SYS_INFO_ALL_BT);
}But I am not sure if it is really easier to follow than calling sys_info_done() from the watchdog code.
Some watchdogs try to optimize the output and print backtraces only from CPUs which are relevant for the given lockup. We should keep the logic for selecting the set of CPUs in the watchdog code. We just need to solve how to elegantly make sys_info() aware of it or at least about the more massive reports.
Anyway, I would prefer to keep it simple until we see some problems in practice.
Best Regards,
Petr
Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces
Quoted 96 lines · Petr Mladek
Quoted 1 line · Bradley Morgan
On June 26, 2026 1:17:13 PM GMT+01:00, Bradley Morgan
Quoted 14 lines
Quoted 13 lines · wrote:
Quoted 11 lines · Petr Mladek
Quoted 4 lines · Petr Mladek
On Thu 2026-06-25 15:25:58, Bradley Morgan wrote:
But it all becomes very hairy. We have several levels:+ watchdog-all_bt-specific option, e.g.
sysctl_hardlockup_all_cpu_backtraceQuoted 5 lines
+ watchdog-specific
si_infopreferences, e.g.hardlockup_si_mask+ panic-specific
si_info:panic_printQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
+ universal fallback for any layer:
kernel_si_infoNow, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.
I think about another approach. The word "waterfall" comes to my
mind.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.
All the
si_infoactions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.Quoted 8 lines
Hmm.. new idea
kernel/dump_filter.c?What this file could do is to handle a generic lockup state machine so any subsystem can log what it already dumped?
I know it may bloat, but it's better then cramming fixes in.
I am not sure what exactly you would like to achieve but it sounds a bit scary ;-)
Anyway, we should not synchronize the watchdog reports against each other, definitely. They are running in non-compatible contexts (task vs interrupt vs NMI). Also we should not add any locking because they usually print something when the system has enough troubles.
Also I think that it is not worth preventing duplicated backtraces or reports from a single CPU. IMHO, it is not a big problem in practice.
So, we are down to large reports, like backtraces from all CPUs, timers, locks, ... which are handled by
sys_info(). So, I think that it should be enough to handle this inside thesys_info()API.I do not want to say that my proposal was the best solution. I am sure that there are better ones. But we need to consider the gain vs. complexity.
Honestly, I am already a bit scared by the complexity which we the
sys_info()API added. And it is hard to imagine that adding another API would make it easier. But I might be wrong.Instead, it might make sense to integrate the conflicting subsystem-specific calls under the
sys_info()API. I mean that, for examplewatchdog_hardlockup_check()won't calltrigger_allbutcpu_cpu_backtrace()directly but it would call it viasys_info()API so thatsys_info()could keep track of it. Something like:void
sys_info_allbutcpu_bt(int cpu){ trigger_allbutcpu_cpu_backtrace(cpu); /* * The caller likely printed backtrace of the given @cpu * on its own. Prevent duplicate backtraces from all * CPUs with potential next sys_info() call. */ sys_info_done(SYS_INFO_ALL_BT); }But I am not sure if it is really easier to follow than calling
sys_info_done()from the watchdog code.Some watchdogs try to optimize the output and print backtraces only from CPUs which are relevant for the given lockup. We should keep the logic for selecting the set of CPUs in the watchdog code. We just need to solve how to elegantly make
sys_info()aware of it or at least about the more massive reports.
Quoted 6 lines
Anyway, I would prefer to keep it simple until we see some problems in practice.
Best Regards,
Petr
I understand it's scary. To make a new file in the first place.
But I was a bit vague of what I wanted, and I'm sorry.
So, the reason why I'd suggest a new file, is because if any subsystem Theoretically bypasses sys_info to log a lockup, this completely misses the filter and duplicates the dump
My file would act as a generic lockless state machine that any subsystem can update regardless of how they dump logs.
If you have any questions, feel absolutely free to ask! :)
Discussion is a way to make everyone happy!
Thanks!
Petr Mladek
reviewerRe: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces
Quoted 114 lines · Bradley Morgan
Quoted 96 lines · Petr Mladek
Quoted 1 line · Bradley Morgan
On June 26, 2026 1:17:13 PM GMT+01:00, Bradley Morgan
Quoted 14 lines
Quoted 13 lines · wrote:
Quoted 11 lines · Petr Mladek
Quoted 4 lines · Petr Mladek
On Thu 2026-06-25 15:25:58, Bradley Morgan wrote:
But it all becomes very hairy. We have several levels:+ watchdog-all_bt-specific option, e.g.
sysctl_hardlockup_all_cpu_backtraceQuoted 5 lines
+ watchdog-specific
si_infopreferences, e.g.hardlockup_si_mask+ panic-specific
si_info:panic_printQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
+ universal fallback for any layer:
kernel_si_infoNow, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.
I think about another approach. The word "waterfall" comes to my
mind.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.
All the
si_infoactions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.Quoted 8 lines
Hmm.. new idea
kernel/dump_filter.c?What this file could do is to handle a generic lockup state machine so any subsystem can log what it already dumped?
I know it may bloat, but it's better then cramming fixes in.
I am not sure what exactly you would like to achieve but it sounds a bit scary ;-)
Anyway, we should not synchronize the watchdog reports against each other, definitely. They are running in non-compatible contexts (task vs interrupt vs NMI). Also we should not add any locking because they usually print something when the system has enough troubles.
Also I think that it is not worth preventing duplicated backtraces or reports from a single CPU. IMHO, it is not a big problem in practice.
So, we are down to large reports, like backtraces from all CPUs, timers, locks, ... which are handled by
sys_info(). So, I think that it should be enough to handle this inside thesys_info()API.I do not want to say that my proposal was the best solution. I am sure that there are better ones. But we need to consider the gain vs. complexity.
Honestly, I am already a bit scared by the complexity which we the
sys_info()API added. And it is hard to imagine that adding another API would make it easier. But I might be wrong.Instead, it might make sense to integrate the conflicting subsystem-specific calls under the
sys_info()API. I mean that, for examplewatchdog_hardlockup_check()won't calltrigger_allbutcpu_cpu_backtrace()directly but it would call it viasys_info()API so thatsys_info()could keep track of it. Something like:void
sys_info_allbutcpu_bt(int cpu){ trigger_allbutcpu_cpu_backtrace(cpu); /* * The caller likely printed backtrace of the given @cpu * on its own. Prevent duplicate backtraces from all * CPUs with potential next sys_info() call. */ sys_info_done(SYS_INFO_ALL_BT); }But I am not sure if it is really easier to follow than calling
sys_info_done()from the watchdog code.Some watchdogs try to optimize the output and print backtraces only from CPUs which are relevant for the given lockup. We should keep the logic for selecting the set of CPUs in the watchdog code. We just need to solve how to elegantly make
sys_info()aware of it or at least about the more massive reports.Quoted 6 lines
Anyway, I would prefer to keep it simple until we see some problems in practice.
Best Regards,
PetrI understand it's scary. To make a new file in the first place.
But I was a bit vague of what I wanted, and I'm sorry.
So, the reason why I'd suggest a new file, is because if any subsystem Theoretically bypasses
sys_infoto log a lockup, this completely misses the filter and duplicates the dump
Quoted 6 lines
My file would act as a generic lockless state machine that any subsystem can update regardless of how they dump logs.
If you have any questions, feel absolutely free to ask! :)
Discussion is a way to make everyone happy!
Honestly, I am more and more wondering whether your are a real person or AI bot.
Best Regards,
Petr
Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces
Quoted 122 lines · Petr Mladek
Quoted 7 lines · Bradley Morgan
Quoted 5 lines · Petr Mladek
Quoted 1 line · Bradley Morgan
On June 26, 2026 1:17:13 PM GMT+01:00, Bradley Morgan
Quoted 2 lines
Quoted 1 line · wrote:
On June 26, 2026 1:14:14 PM GMT+01:00, Petr Mladek
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines · wrote:
Quoted 4 lines · Petr Mladek
On Thu 2026-06-25 15:25:58, Bradley Morgan wrote:
But it all becomes very hairy. We have several levels:+ watchdog-all_bt-specific option, e.g.
sysctl_hardlockup_all_cpu_backtraceQuoted 2 lines
+ watchdog-specific
si_infopreferences, e.g.
hardlockup_si_maskQuoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
+ panic-specific
si_info:panic_print+ universal fallback for any layer:
kernel_si_infoQuoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Now, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.
I think about another approach. The word "waterfall" comes to my
mind.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Instead of checking all the settings back and forth, let's
process
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
each setting one by one and just remember what has been done and skip this in the next level.
All the
si_infoactions seems to dump a global system state. So, it would make sense to remember the state in a globalvariable
Quoted 83 lines
Quoted 64 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
even when it might be modified by more CPUs in parallel.
Hmm.. new idea
kernel/dump_filter.c?What this file could do is to handle a generic lockup state machine so any subsystem can log what it already dumped?
I know it may bloat, but it's better then cramming fixes in.
I am not sure what exactly you would like to achieve but it sounds a bit scary ;-)
Anyway, we should not synchronize the watchdog reports against each other, definitely. They are running in non-compatible contexts (task vs interrupt vs NMI). Also we should not add any locking because they usually print something when the system has enough troubles.
Also I think that it is not worth preventing duplicated backtraces or reports from a single CPU. IMHO, it is not a big problem in practice.
So, we are down to large reports, like backtraces from all CPUs, timers, locks, ... which are handled by
sys_info(). So, I think that it should be enough to handle this inside thesys_info()API.I do not want to say that my proposal was the best solution. I am sure that there are better ones. But we need to consider the gain vs. complexity.
Honestly, I am already a bit scared by the complexity which we the
sys_info()API added. And it is hard to imagine that adding another API would make it easier. But I might be wrong.Instead, it might make sense to integrate the conflicting subsystem-specific calls under the
sys_info()API. I mean that, for examplewatchdog_hardlockup_check()won't calltrigger_allbutcpu_cpu_backtrace()directly but it would call it viasys_info()API so thatsys_info()could keep track of it. Something like:void
sys_info_allbutcpu_bt(int cpu){ trigger_allbutcpu_cpu_backtrace(cpu); /* * The caller likely printed backtrace of the given @cpu * on its own. Prevent duplicate backtraces from all * CPUs with potential next sys_info() call. */ sys_info_done(SYS_INFO_ALL_BT); }But I am not sure if it is really easier to follow than calling
sys_info_done()from the watchdog code.Some watchdogs try to optimize the output and print backtraces only from CPUs which are relevant for the given lockup. We should keep the logic for selecting the set of CPUs in the watchdog code. We just need to solve how to elegantly make
sys_info()aware of it or at least about the more massive reports.Quoted 6 lines
Anyway, I would prefer to keep it simple until we see some problems in practice.
Best Regards,
PetrI understand it's scary. To make a new file in the first place.
But I was a bit vague of what I wanted, and I'm sorry.
So, the reason why I'd suggest a new file, is because if any subsystem Theoretically bypasses
sys_infoto log a lockup, this completely misses the filter and duplicates the dumpMy file would act as a generic lockless state machine that any subsystem can update regardless of how they dump logs.
Quoted 6 lines
Quoted 3 lines
If you have any questions, feel absolutely free to ask! :)
Discussion is a way to make everyone happy!
Honestly, I am more and more wondering whether your are a real person or AI bot.
Sigh..
I can verify myself through video call if you don't believe I am human :)
why I suggested a new file is because AI said it would be a good idea.
I told it what I should do, and it told me to do a new file.
I knew it was over engineering slightly, but I was a bit stressed, and I wanted some sort of just new API which is less buggy imho
I should've told you that I used AI to figure the whole new file idea,
Really sorry petr..
Quoted 3 lines
Best Regards,
Petr
Thanks!
Feng Tang
Reviewed-byRe: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces
In reply to Petr MladekQuoted 113 lines · Petr Mladek
Quoted 70 lines · Petr Mladek
Quoted 19 lines · Bradley Morgan
panic_other_cpus_shutdown()handlesSYS_INFO_ALL_BTbefore stopping the other CPUs. Do not asksys_info()to handle that bit again later in the panic path.Use
sys_info_with_filter()so panic_print=all_bt does not request more output after the CPUs are stopped.Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys info on system lockup")Signed-off-by Bradley Morgan <[email protected]>--- kernel/panic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/panic.c +0
-0kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683
Quoted 5 lines
kernel/panic.c +1
-1@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)*/ atomic_notifier_call_chain (&panic_notifier_list,0 , buf); -sys_info (panic_print); +sys_info_with_filter (panic_print,SYS_INFO_ALL_BT );Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?
They might be printed by:
static void
panic_other_cpus_shutdown(bool crash_kexec){ if (panic_print & SYS_INFO_ALL_BT) panic_trigger_all_cpu_backtrace();[...]
}But it checks only "
panic_print" variable. It won't do anything when (panic_print== 0).In this case, we might still want to print the backraces when
SYS_INFO_ALL_BTis set inkernel_si_info.kernel/panic.c void vpanic(const char *fmt, va_list args) · line 684
Quoted 1 line
kernel/panic.c +0
-0kmsg_dump_desc (KMSG_DUMP_PANIC , buf);Of course, we might fix
panic_other_cpus_shutdown()to check alsokernel_si_info.But it all becomes very hairy. We have several levels:
+ watchdog-all_bt-specific option, e.g.
sysctl_hardlockup_all_cpu_backtrace+ watchdog-specific
si_infopreferences, e.g.hardlockup_si_mask+ panic-specific
si_info:panic_print+ universal fallback for any layer:
kernel_si_infoNow, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.
I think about another approach. The word "waterfall" comes to my mind. Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.
Quoted 5 lines
All the
si_infoactions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.I am going to think more about it.
I have created a POC using Gemini. I haven't tested it. But it looks acceptable. And the logic seems to be more straightforward.
One drawback is that it requires adding the
_reset()call for allsys_info()callers. It is fine in principle but it might complicate back-porting because all changes have to be done in one patch.But honestly, this is a nice to have fix. Most people could live happily without it.
From
3c66436d9978030845a96bfaedd6b914536e2ac4Mon Sep 17 00:00:00 2001
From: Petr Mladek <[email protected]>
Date: Fri, 26 Jun 2026 13:55:41 +0200
Subject: [POC]sys_info: Introduce state-tracking APIs to prevent duplicate
backtracesIn watchdog, panic, and hung task detection scenarios,
sys_info()can be called multiple times or alongside direct backtrace triggers liketrigger_allbutcpu_cpu_backtrace(). This results in identical backtraces being dumped repeatedly from all CPUs, cluttering the kernel log and delaying or obscuring critical debug details.Introduce a state tracking bitmask and associated helpers:
-sys_info_done(mask): Marks specificsys_infobits as already printed.
-sys_info_reset(): Resets the tracking state.
-sys_info_is_done(mask): Checks if all bits in the mask have been printed.Update
sys_info()to automatically filter out already printed bits using this state. Integrate these APIs with the generic hardlockup and softlockup watchdogs, the PowerPC watchdog, the hung task detector, and the panic core. This ensures that each piece of system information and backtrace output is printed at most once per lockup/panic event, and the state is reset cleanly when a lockup does not trigger a panic.
Quoted 6 lines
Races between
sys_info()callers are ignored. It should be acceptable because the output from various watchdogs has never been synchronized. Andpanic()never returns.Assisted-by gemini-1.5-flashSigned-off-by Petr Mladek <[email protected]>
Yep. There are cases that people want panic on task-hung or sw/hw lockup, and this could remove much duplication of sys info dump, thanks!
Quoted 8 lines
--- arch/powerpc/kernel/watchdog.c | 13 ++++++++++--- include/linux/sys_info.h | 3 +++ kernel/hung_task.c | 2 ++ kernel/panic.c | 4 +++- kernel/watchdog.c | 10 ++++++++-- lib/sys_info.c | 30 +++++++++++++++++++++++++++++- 6 files changed, 55 insertions(+), 7 deletions(-)
Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces
Quoted 125 lines · Feng Tang
Quoted 3 lines · Petr Mladek
Quoted 2 lines · Petr Mladek
Quoted 1 line · Bradley Morgan
panic_other_cpus_shutdown()handlesSYS_INFO_ALL_BTbefore stoppingthe
Quoted 1 line
Quoted 1 line
Quoted 1 line
other CPUs. Do not ask
sys_info()to handle that bit again later inthe
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
panic path.
Use
sys_info_with_filter()so panic_print=all_bt does not requestmore
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
output after the CPUs are stopped.
Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sysinfo on system lockup")
Quoted 36 lines
Quoted 36 lines
Quoted 11 lines
Signed-off-by Bradley Morgan <[email protected]>--- kernel/panic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/panic.c +0
-0kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683
Quoted 5 lines
kernel/panic.c +1
-1@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)*/ atomic_notifier_call_chain (&panic_notifier_list,0 , buf); -sys_info (panic_print); +sys_info_with_filter (panic_print,SYS_INFO_ALL_BT );Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?
They might be printed by:
static void
panic_other_cpus_shutdown(bool crash_kexec){ if (panic_print & SYS_INFO_ALL_BT) panic_trigger_all_cpu_backtrace();[...]
}But it checks only "
panic_print" variable. It won't do anything when (panic_print== 0).In this case, we might still want to print the backraces when
SYS_INFO_ALL_BTis set inkernel_si_info.Quoted 8 lines
Quoted 8 lines
kernel/panic.c void vpanic(const char *fmt, va_list args) · line 684
Quoted 1 line
kernel/panic.c +0
-0kmsg_dump_desc (KMSG_DUMP_PANIC , buf);Of course, we might fix
panic_other_cpus_shutdown()to check alsokernel_si_info.But it all becomes very hairy. We have several levels:
+ watchdog-all_bt-specific option, e.g.
sysctl_hardlockup_all_cpu_backtraceQuoted 32 lines
Quoted 17 lines
+ watchdog-specific
si_infopreferences, e.g.hardlockup_si_mask+ panic-specific
si_info:panic_print+ universal fallback for any layer:
kernel_si_infoNow, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.
I think about another approach. The word "waterfall" comes to my mind. Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.
Quoted 5 lines
All the
si_infoactions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.I am going to think more about it.
I have created a POC using Gemini. I haven't tested it. But it looks acceptable. And the logic seems to be more straightforward.
One drawback is that it requires adding the
_reset()call for allsys_info()callers. It is fine in principle but it might complicate back-porting because all changes have to be done in one patch.Quoted 7 lines
But honestly, this is a nice to have fix. Most people could live happily without it.
From
3c66436d9978030845a96bfaedd6b914536e2ac4Mon Sep 17 00:00:00 2001
From: Petr Mladek <[email protected]>
Date: Fri, 26 Jun 2026 13:55:41 +0200
Subject: [POC]sys_info: Introduce state-tracking APIs to preventduplicate
Quoted 8 lines
backtraces
In watchdog, panic, and hung task detection scenarios,
sys_info()can be called multiple times or alongside direct backtrace triggers liketrigger_allbutcpu_cpu_backtrace(). This results in identical backtraces being dumped repeatedly from all CPUs, cluttering the kernel log and delaying or obscuring critical debug details.Quoted 4 lines
Introduce a state tracking bitmask and associated helpers:
-sys_info_done(mask): Marks specificsys_infobits as already printed.
-sys_info_reset(): Resets the tracking state.
-sys_info_is_done(mask): Checks if all bits in the mask have beenprinted.
Quoted 12 lines
Update
sys_info()to automatically filter out already printed bits using this state. Integrate these APIs with the generic hardlockup and softlockup watchdogs, the PowerPC watchdog, the hung task detector, and the panic core. This ensures that each piece of system information and backtrace output is printed at most once per lockup/panic event, and the state is reset cleanly when a lockup does not trigger a panic.Races between
sys_info()callers are ignored. It should be acceptable because the output from various watchdogs has never been synchronized. Andpanic()never returns.
Quoted 7 lines
Quoted 2 lines
Assisted-by gemini-1.5-flashSigned-off-by Petr Mladek <[email protected]>Yep. There are cases that people want panic on task-hung or sw/hw lockup, and this could remove much duplication of sys info dump, thanks!
Reviewed-by Feng Tang <[email protected]>
Thanks,
im feeling a new file to do all the force panic jazz, but putting tape on sys_info.c isn't bd either.
Quoted 9 lines
Quoted 8 lines
--- arch/powerpc/kernel/watchdog.c | 13 ++++++++++--- include/linux/sys_info.h | 3 +++ kernel/hung_task.c | 2 ++ kernel/panic.c | 4 +++- kernel/watchdog.c | 10 ++++++++-- lib/sys_info.c | 30 +++++++++++++++++++++++++++++- 6 files changed, 55 insertions(+), 7 deletions(-)
Thanks!
Petr Mladek
reviewerRe: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces
Quoted 133 lines · Bradley Morgan
Quoted 131 lines · Feng Tang
Quoted 3 lines · Petr Mladek
Quoted 2 lines · Petr Mladek
Quoted 1 line · Bradley Morgan
panic_other_cpus_shutdown()handlesSYS_INFO_ALL_BTbefore stoppingthe
Quoted 1 line
Quoted 1 line
Quoted 1 line
other CPUs. Do not ask
sys_info()to handle that bit again later inthe
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
panic path.
Use
sys_info_with_filter()so panic_print=all_bt does not requestmore
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
output after the CPUs are stopped.
Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sysinfo on system lockup")
Quoted 36 lines
Quoted 36 lines
Quoted 11 lines
Signed-off-by Bradley Morgan <[email protected]>--- kernel/panic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/panic.c +0
-0kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683
Quoted 5 lines
kernel/panic.c +1
-1@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)*/ atomic_notifier_call_chain (&panic_notifier_list,0 , buf); -sys_info (panic_print); +sys_info_with_filter (panic_print,SYS_INFO_ALL_BT );Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?
They might be printed by:
static void
panic_other_cpus_shutdown(bool crash_kexec){ if (panic_print & SYS_INFO_ALL_BT) panic_trigger_all_cpu_backtrace();[...]
}But it checks only "
panic_print" variable. It won't do anything when (panic_print== 0).In this case, we might still want to print the backraces when
SYS_INFO_ALL_BTis set inkernel_si_info.Quoted 8 lines
Quoted 8 lines
kernel/panic.c void vpanic(const char *fmt, va_list args) · line 684
Quoted 1 line
kernel/panic.c +0
-0kmsg_dump_desc (KMSG_DUMP_PANIC , buf);Of course, we might fix
panic_other_cpus_shutdown()to check alsokernel_si_info.But it all becomes very hairy. We have several levels:
+ watchdog-all_bt-specific option, e.g.
sysctl_hardlockup_all_cpu_backtraceQuoted 32 lines
Quoted 17 lines
+ watchdog-specific
si_infopreferences, e.g.hardlockup_si_mask+ panic-specific
si_info:panic_print+ universal fallback for any layer:
kernel_si_infoNow, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.
I think about another approach. The word "waterfall" comes to my mind. Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.
Quoted 5 lines
All the
si_infoactions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.I am going to think more about it.
I have created a POC using Gemini. I haven't tested it. But it looks acceptable. And the logic seems to be more straightforward.
One drawback is that it requires adding the
_reset()call for allsys_info()callers. It is fine in principle but it might complicate back-porting because all changes have to be done in one patch.Quoted 7 lines
But honestly, this is a nice to have fix. Most people could live happily without it.
From
3c66436d9978030845a96bfaedd6b914536e2ac4Mon Sep 17 00:00:00 2001
From: Petr Mladek <[email protected]>
Date: Fri, 26 Jun 2026 13:55:41 +0200
Subject: [POC]sys_info: Introduce state-tracking APIs to preventduplicate
Quoted 8 lines
backtraces
In watchdog, panic, and hung task detection scenarios,
sys_info()can be called multiple times or alongside direct backtrace triggers liketrigger_allbutcpu_cpu_backtrace(). This results in identical backtraces being dumped repeatedly from all CPUs, cluttering the kernel log and delaying or obscuring critical debug details.Quoted 4 lines
Introduce a state tracking bitmask and associated helpers:
-sys_info_done(mask): Marks specificsys_infobits as already printed.
-sys_info_reset(): Resets the tracking state.
-sys_info_is_done(mask): Checks if all bits in the mask have beenprinted.
Quoted 8 lines
Update
sys_info()to automatically filter out already printed bits using this state. Integrate these APIs with the generic hardlockup and softlockup watchdogs, the PowerPC watchdog, the hung task detector, and the panic core. This ensures that each piece of system information and backtrace output is printed at most once per lockup/panic event, and the state is reset cleanly when a lockup does not trigger a panic.Quoted 6 lines
Races between
sys_info()callers are ignored. It should be acceptable because the output from various watchdogs has never been synchronized. Andpanic()never returns.Assisted-by gemini-1.5-flashSigned-off-by Petr Mladek <[email protected]>Yep. There are cases that people want panic on task-hung or sw/hw lockup, and this could remove much duplication of sys info dump, thanks!
Quoted 6 lines
Quoted 1 line
Reviewed-by Feng Tang <[email protected]>Thanks,
im feeling a new file to do all the force panic jazz, but putting tape on
sys_info.cisn't bd either.
I wonder how to move forward with this.
Honestly, I am not sure what exactly you mean by creating another API for tracking the reports so I could not judge it. Feel free to sent some POC.
Otherwise, I would go with my proposal to remember the printed states by the sys_info API. I am not sure whether I should send a proper patch or you would like to somehow improve it.
Best Regards,
Petr
Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces
Quoted 158 lines · Petr Mladek
Quoted 6 lines · Bradley Morgan
Quoted 4 lines · Feng Tang
Quoted 3 lines · Petr Mladek
Quoted 2 lines · Petr Mladek
Quoted 1 line · Bradley Morgan
panic_other_cpus_shutdown()handlesSYS_INFO_ALL_BTbeforestopping
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
other CPUs. Do not ask
sys_info()to handle that bit again laterin
Quoted 59 lines
Quoted 59 lines
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
panic path.
Use
sys_info_with_filter()so panic_print=all_bt does not requestmore
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
output after the CPUs are stopped.
Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sysinfo on system lockup")
Quoted 36 lines
Quoted 36 lines
Quoted 11 lines
Signed-off-by Bradley Morgan <[email protected]>--- kernel/panic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/panic.c +0
-0kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683
Quoted 5 lines
kernel/panic.c +1
-1@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)*/ atomic_notifier_call_chain (&panic_notifier_list,0 , buf); -sys_info (panic_print); +sys_info_with_filter (panic_print,SYS_INFO_ALL_BT );Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?
They might be printed by:
static void
panic_other_cpus_shutdown(bool crash_kexec){ if (panic_print & SYS_INFO_ALL_BT) panic_trigger_all_cpu_backtrace();[...]
}But it checks only "
panic_print" variable. It won't do anything when (panic_print== 0).In this case, we might still want to print the backraces when
SYS_INFO_ALL_BTis set inkernel_si_info.Quoted 8 lines
Quoted 8 lines
kernel/panic.c void vpanic(const char *fmt, va_list args) · line 684
Quoted 1 line
kernel/panic.c +0
-0kmsg_dump_desc (KMSG_DUMP_PANIC , buf);Of course, we might fix
panic_other_cpus_shutdown()to check alsokernel_si_info.But it all becomes very hairy. We have several levels:
+ watchdog-all_bt-specific option, e.g.
sysctl_hardlockup_all_cpu_backtraceQuoted 5 lines
Quoted 5 lines
+ watchdog-specific
si_infopreferences, e.g.hardlockup_si_mask+ panic-specific
si_info:panic_printQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
+ universal fallback for any layer:
kernel_si_infoNow, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.
I think about another approach. The word "waterfall" comes to my
mind.
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 9 lines
Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.
All the
si_infoactions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.I am going to think more about it.
I have created a POC using Gemini. I haven't tested it. But it looks acceptable. And the logic seems to be more straightforward.
One drawback is that it requires adding the
_reset()call for allsys_info()callers. It is fine in principle but it might complicate back-porting because all changes have to be done in one patch.Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
But honestly, this is a nice to have fix. Most people could live happily without it.
From
3c66436d9978030845a96bfaedd6b914536e2ac4Mon Sep 17 00:00:002001
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
From: Petr Mladek <[email protected]>
Date: Fri, 26 Jun 2026 13:55:41 +0200
Subject: [POC]sys_info: Introduce state-tracking APIs to preventduplicate
Quoted 5 lines
backtraces
In watchdog, panic, and hung task detection scenarios,
sys_info()can be called multiple times or alongside direct backtrace triggers liketrigger_allbutcpu_cpu_backtrace(). This results in identicalbacktraces
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
being dumped repeatedly from all CPUs, cluttering the kernel log and delaying or obscuring critical debug details.
Introduce a state tracking bitmask and associated helpers:
-sys_info_done(mask): Marks specificsys_infobits as alreadyprinted.
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
-
sys_info_reset(): Resets the tracking state.
-sys_info_is_done(mask): Checks if all bits in the mask have beenprinted.
Quoted 4 lines
Update
sys_info()to automatically filter out already printed bits using this state. Integrate these APIs with the generic hardlockup and softlockup watchdogs, the PowerPC watchdog, the hung taskdetector,
Quoted 1 line
Quoted 1 line
Quoted 1 line
and the panic core. This ensures that each piece of system
information
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
and backtrace output is printed at most once per lockup/panic event, and the state is reset cleanly when a lockup does not trigger a
panic.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Races between
sys_info()callers are ignored. It should be acceptable because the output from various watchdogs has never beensynchronized.
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
And
panic()never returns.Assisted-by gemini-1.5-flashSigned-off-by Petr Mladek <[email protected]>Yep. There are cases that people want panic on task-hung or sw/hw
lockup,
Quoted 8 lines
Quoted 3 lines
and this could remove much duplication of sys info dump, thanks!
Reviewed-by Feng Tang <[email protected]>Thanks,
im feeling a new file to do all the force panic jazz, but putting tape on
sys_info.cisn't bd either.I wonder how to move forward with this.
Honestly, I am not sure what exactly you mean by creating another API for tracking the reports so I could not judge it. Feel free to sent some POC.
Quoted 7 lines
Otherwise, I would go with my proposal to remember the printed states by the
sys_infoAPI. I am not sure whether I should send a proper patch or you would like to somehow improve it.Best Regards,
Petr
sup petr, here's my poc
This should make my entire thing make sense
From eb587ed749ff5993c517f29799b369185c5ee7d8 Mon Sep 17 00:00:00 2001
From: Bradley Morgan <[email protected]>
Date: Thu, 2 Jul 2026 18:09:23 +0000
Subject: [POC] sys_info: Introduce incident state-tracking to prevent
duplicate diagnostics
In watchdog, panic, and hung task detection scenarios, sys_info() can be called multiple times or alongside direct debug output functions (like trigger_allbutcpu_cpu_backtrace(), print_modules(), print_irqtrace_events(), and dump_stack()). This leads to identical diagnostics and stack traces being dumped repeatedly, cluttering the kernel log and delaying critical panics.
Introduce a state tracking bitmask and helpers in a new file, lib/sys_info_filter.c:
- sys_info_filter_and_set(mask): Atomically tests which bits in a mask have not yet been printed during the current incident, marks them as printed, and returns that subset.
- sys_info_reset(): Clears the printed mask state.
Add SYS_INFO_MODULES, SYS_INFO_IRQTRACE, and SYS_INFO_STACK flags to include/linux/sys_info.h, and handle them inside sys_info's diagnostic dispatch.
Update the watchdogs, hung task detector, and panic core to call sys_info_filter_and_set() to deduplicate their diagnostic printouts, and sys_info_reset() when a warning incident concludes (e.g., when a stuck CPU recovers, or a new hung task check round begins).
This ensures each piece of system diagnostic is printed at most once per lockup/panic event, preventing console log spam.
--- arch/powerpc/kernel/watchdog.c | 21 +++++- include/linux/sys_info.h | 6 ++ kernel/hung_task.c | 4 +- kernel/panic.c | 9 ++- kernel/watchdog.c | 24 ++++++- lib/Makefile | 2 +- lib/sys_info.c | 38 ++--------- lib/sys_info_filter.c | 120 +++++++++++++++++++++++++++++++++ 8 files changed, 184 insertions(+), 40 deletions(-) create mode 100644 lib/sys_info_filter.c
arch/powerpc/kernel/watchdog.c +18 -3
@@ -160,6 +160,10 @@ static void wd_lockup_ipi(struct pt_regs *regs)else dump_stack (); +/* Mark what we already printed so panic() won't repeat it. */ +sys_info_filter_and_set (SYS_INFO_MODULES |SYS_INFO_IRQTRACE | +SYS_INFO_STACK ); +/* * __wd_nmi_output must be set after we printk from NMI context. *
@@ -238,7 +242,8 @@ static void watchdog_smp_panic(int cpu)if (sysctl_hardlockup_all_cpu_backtrace || (hardlockup_si_mask &SYS_INFO_ALL_BT )) { -trigger_allbutcpu_cpu_backtrace (cpu); +if (sys_info_filter_and_set (SYS_INFO_ALL_BT )) +trigger_allbutcpu_cpu_backtrace (cpu);cpumask_clear (&wd_smp_cpus_ipi); }else {/*
@@ -254,6 +259,8 @@ static void watchdog_smp_panic(int cpu)sys_info (hardlockup_si_mask & ~SYS_INFO_ALL_BT );if (hardlockup_panic)nmi_panic (NULL ,"Hard LOCKUP" ); +else +sys_info_reset ();wd_end_reporting ();
@@ -416,15 +423,23 @@ DEFINE_INTERRUPT_HANDLER_NMI(soft_nmi_interrupt)print_irqtrace_events (current);show_regs (regs); +/* Mark what we already printed so panic() won't repeat it. */ +sys_info_filter_and_set (SYS_INFO_MODULES |SYS_INFO_IRQTRACE | +SYS_INFO_STACK ); +xchg (&__wd_nmi_output,1 );// see wd_lockup_ipi if (sysctl_hardlockup_all_cpu_backtrace || - (hardlockup_si_mask &SYS_INFO_ALL_BT )) -trigger_allbutcpu_cpu_backtrace (cpu); + (hardlockup_si_mask &SYS_INFO_ALL_BT )) { +if (sys_info_filter_and_set (SYS_INFO_ALL_BT )) +trigger_allbutcpu_cpu_backtrace (cpu); + }sys_info (hardlockup_si_mask & ~SYS_INFO_ALL_BT );if (hardlockup_panic)nmi_panic (regs,"Hard LOCKUP" ); +else +sys_info_reset ();wd_end_reporting (); }
include/linux/sys_info.h +6 -0
@@ -16,8 +16,14 @@#define SYS_INFO_PANIC_CONSOLE_REPLAY 0x00000020#define SYS_INFO_ALL_BT 0x00000040#define SYS_INFO_BLOCKED_TASKS 0x00000080 +#define SYS_INFO_MODULES 0x00000100 +#define SYS_INFO_IRQTRACE 0x00000200 +#define SYS_INFO_STACK 0x00000400void sys_info (unsigned long si_mask); +unsigned long sys_info_effective_mask (unsigned long mask); +unsigned long sys_info_filter_and_set (unsigned long si_mask); +void sys_info_reset (void );unsigned long sys_info_parse_param (char *str);#ifdef CONFIG_SYSCTL
kernel/hung_task.c +3 -1
@@ -557,8 +557,10 @@ static int watchdog(void *dummy) t =hung_timeout_jiffies (hung_last_checked, interval);if (t <=0 ) {if (!atomic_xchg (&reset_hung_task,0 ) && - !hung_detector_suspended) + !hung_detector_suspended) { +sys_info_reset ();check_hung_uninterruptible_tasks (timeout); + } hung_last_checked = jiffies;continue ; }
kernel/panic.c +7 -2
@@ -550,8 +550,12 @@ static void panic_trigger_all_cpu_backtrace(void)*/ static void panic_other_cpus_shutdown (bool crash_kexec) { -if (panic_print &SYS_INFO_ALL_BT ) -panic_trigger_all_cpu_backtrace (); +unsigned long mask =sys_info_effective_mask (panic_print); + +if (mask &SYS_INFO_ALL_BT ) { +if (sys_info_filter_and_set (SYS_INFO_ALL_BT )) +panic_trigger_all_cpu_backtrace (); + }/* * Note that smp_send_stop() is the usual SMP shutdown function,
@@ -649,6 +653,7 @@ void vpanic(const char *fmt, va_list args) panic_this_cpu_backtrace_printed = true; }else if (IS_ENABLED (CONFIG_DEBUG_BUGVERBOSE )) {dump_stack (); +sys_info_filter_and_set (SYS_INFO_STACK ); panic_this_cpu_backtrace_printed = true; }
kernel/watchdog.c +22 -2
@@ -280,8 +280,13 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs)trigger_single_cpu_backtrace (cpu); } +/* Mark what we already printed so panic() won't repeat it. */ +sys_info_filter_and_set (SYS_INFO_MODULES |SYS_INFO_IRQTRACE | +SYS_INFO_STACK ); +if (hardlockup_all_cpu_backtrace) { -trigger_allbutcpu_cpu_backtrace (cpu); +if (sys_info_filter_and_set (SYS_INFO_ALL_BT )) +trigger_allbutcpu_cpu_backtrace (cpu);if (!hardlockup_panic)clear_bit_unlock (0 , &hard_lockup_nmi_warn); }
@@ -289,6 +294,8 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs)sys_info (hardlockup_si_mask & ~SYS_INFO_ALL_BT );if (hardlockup_panic)nmi_panic (regs,"Hard LOCKUP" ); +else +sys_info_reset ();per_cpu (watchdog_hardlockup_warned, cpu) = true; }
@@ -792,6 +799,8 @@ static int softlockup_fn(void *data) }/* watchdog kicker functions */ +static DEFINE_PER_CPU(bool, watchdog_stuck_previously); +static enum hrtimer_restart watchdog_timer_fn (struct hrtimer *hrtimer) {unsigned long touch_ts, period_ts, now;
@@ -864,6 +873,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) touch_ts =__this_cpu_read (watchdog_touch_ts); duration =is_softlockup (touch_ts, period_ts, now);if (unlikely (duration)) { +__this_cpu_write (watchdog_stuck_previously, true);#ifdef CONFIG_SYSFS ++softlockup_count;#endif
@@ -893,8 +903,13 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)dump_stack ();printk_cpu_sync_put_irqrestore (flags); +/* Mark what we already printed so panic() won't repeat it. */ +sys_info_filter_and_set (SYS_INFO_MODULES |SYS_INFO_IRQTRACE | +SYS_INFO_STACK ); +if (softlockup_all_cpu_backtrace) { -trigger_allbutcpu_cpu_backtrace (smp_processor_id ()); +if (sys_info_filter_and_set (SYS_INFO_ALL_BT )) +trigger_allbutcpu_cpu_backtrace (smp_processor_id ());if (!softlockup_panic)clear_bit_unlock (0 , &soft_lockup_nmi_warn); }
@@ -905,6 +920,11 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)if (softlockup_panic && thresh_count >= softlockup_panic)panic ("softlockup: hung tasks" ); + }else { +if (__this_cpu_read (watchdog_stuck_previously)) { +__this_cpu_write (watchdog_stuck_previously, false); +sys_info_reset (); + } }return HRTIMER_RESTART ;
lib/Makefile +1 -1
@@ -40,7 +40,7 @@ lib-y := ctype.o string.o vsprintf.o cmdline.o \ is_single_threaded.o plist.o decompress.o kobject_uevent.o \ earlycpio.o seq_buf.o siphash.o dec_and_lock.o \ nmi_backtrace.o win_minmax.o memcat_p.o \ - buildid.o objpool.o iomem_copy.o sys_info.o + buildid.o objpool.o iomem_copy.o sys_info.o sys_info_filter.o lib-$(CONFIG_UNION_FIND) += union_find.o lib-$(CONFIG_PRINTK) += dump_stack.o
lib/sys_info.c +7 -31
@@ -2,12 +2,9 @@#include <linux/array_size.h> #include <linux/bitops.h> #include <linux/cleanup.h> -#include <linux/console.h> #include <linux/log2.h> #include <linux/kernel.h> -#include <linux/ftrace.h> -#include <linux/nmi.h> -#include <linux/sched/debug.h> +#include <linux/slab.h> #include <linux/string.h> #include <linux/sysctl.h>
@@ -22,6 +19,9 @@ static const char * const si_names[] = { [ilog2(SYS_INFO_PANIC_CONSOLE_REPLAY)] = "", [ilog2(SYS_INFO_ALL_BT)] = "all_bt", [ilog2(SYS_INFO_BLOCKED_TASKS)] = "blocked_tasks", + [ilog2(SYS_INFO_MODULES)] = "modules", + [ilog2(SYS_INFO_IRQTRACE)] = "irqtrace", + [ilog2(SYS_INFO_STACK)] = "stack", };/*
@@ -29,7 +29,7 @@ static const char * const si_names[] = {* If a kernel module calls sys_info() with "parameter == 0", then * this mask will be used. */ -static unsigned long kernel_si_mask; +unsigned long kernel_si_mask;/* Expecting string like "xxx_sys_info=tasks,mem,timers,locks,ftrace,..." */ unsigned long sys_info_parse_param (char *str)
@@ -136,31 +136,7 @@ static int __init sys_info_sysctl_init(void)subsys_initcall (sys_info_sysctl_init);#endif -static void __sys_info (unsigned long si_mask) +unsigned long sys_info_effective_mask (unsigned long mask) { -if (si_mask &SYS_INFO_TASKS ) -show_state (); - -if (si_mask &SYS_INFO_MEM ) -show_mem (); - -if (si_mask &SYS_INFO_TIMERS ) -sysrq_timer_list_show (); - -if (si_mask &SYS_INFO_LOCKS ) -debug_show_all_locks (); - -if (si_mask &SYS_INFO_FTRACE ) -ftrace_dump (DUMP_ALL ); - -if (si_mask &SYS_INFO_ALL_BT ) -trigger_all_cpu_backtrace (); - -if (si_mask &SYS_INFO_BLOCKED_TASKS ) -show_state_filter (TASK_UNINTERRUPTIBLE ); -} - -void sys_info (unsigned long si_mask) -{ -__sys_info (si_mask ? : kernel_si_mask); +return mask ? :READ_ONCE (kernel_si_mask); }
lib/sys_info_filter.c new file +120 -0
120 lines not on this page · raw message
Signature · 11 lines
2.53.0 NOTE!!: This is AI generated!! This **MAY** not be the finished product, this is ONLY the model! Thanks!
Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces
Quoted 173 lines · Bradley Morgan
Quoted 15 lines · Petr Mladek
Quoted 6 lines · Bradley Morgan
Quoted 4 lines · Feng Tang
Quoted 3 lines · Petr Mladek
Quoted 2 lines · Petr Mladek
Quoted 1 line · Bradley Morgan
panic_other_cpus_shutdown()handlesSYS_INFO_ALL_BTbeforestopping
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
other CPUs. Do not ask
sys_info()to handle that bit again laterin
Quoted 4 lines
Quoted 4 lines
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
panic path.
Use
sys_info_with_filter()so panic_print=all_bt does notrequest
Quoted 46 lines
Quoted 46 lines
Quoted 46 lines
more
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
output after the CPUs are stopped.
Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sysinfo on system lockup")
Quoted 41 lines
Quoted 41 lines
Quoted 11 lines
Signed-off-by Bradley Morgan <[email protected]>--- kernel/panic.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/panic.c +0
-0kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683
Quoted 5 lines
kernel/panic.c +1
-1@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)*/ atomic_notifier_call_chain (&panic_notifier_list,0 , buf); -sys_info (panic_print); +sys_info_with_filter (panic_print,SYS_INFO_ALL_BT );Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?
They might be printed by:
static void
panic_other_cpus_shutdown(bool crash_kexec){ if (panic_print & SYS_INFO_ALL_BT) panic_trigger_all_cpu_backtrace();[...]
}But it checks only "
panic_print" variable. It won't do anything when (panic_print== 0).In this case, we might still want to print the backraces when
SYS_INFO_ALL_BTis set inkernel_si_info.kernel/panic.c void vpanic(const char *fmt, va_list args) · line 684
Quoted 1 line
kernel/panic.c +0
-0kmsg_dump_desc (KMSG_DUMP_PANIC , buf);Of course, we might fix
panic_other_cpus_shutdown()to check alsokernel_si_info.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
But it all becomes very hairy. We have several levels:
+ watchdog-all_bt-specific option, e.g.
sysctl_hardlockup_all_cpu_backtraceQuoted 2 lines
Quoted 2 lines
+ watchdog-specific
si_infopreferences, e.g.
hardlockup_si_maskQuoted 31 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
+ panic-specific
si_info:panic_print+ universal fallback for any layer:
kernel_si_infoNow, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.
I think about another approach. The word "waterfall" comes to my
mind.
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 9 lines
Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.
All the
si_infoactions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.I am going to think more about it.
I have created a POC using Gemini. I haven't tested it. But it looks acceptable. And the logic seems to be more straightforward.
One drawback is that it requires adding the
_reset()call for allsys_info()callers. It is fine in principle but it might complicate back-porting because all changes have to be done in one patch.Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
But honestly, this is a nice to have fix. Most people could live happily without it.
From
3c66436d9978030845a96bfaedd6b914536e2ac4Mon Sep 17 00:00:002001
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
From: Petr Mladek <[email protected]>
Date: Fri, 26 Jun 2026 13:55:41 +0200
Subject: [POC]sys_info: Introduce state-tracking APIs to preventduplicate
Quoted 3 lines
backtraces
In watchdog, panic, and hung task detection scenarios,
sys_info()can
Quoted 16 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
be called multiple times or alongside direct backtrace triggers like
trigger_allbutcpu_cpu_backtrace(). This results in identicalbacktraces
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
being dumped repeatedly from all CPUs, cluttering the kernel log and delaying or obscuring critical debug details.
Introduce a state tracking bitmask and associated helpers:
-sys_info_done(mask): Marks specificsys_infobits as alreadyprinted.
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
-
sys_info_reset(): Resets the tracking state.
-sys_info_is_done(mask): Checks if all bits in the mask have beenprinted.
Quoted 4 lines
Update
sys_info()to automatically filter out already printed bits using this state. Integrate these APIs with the generic hardlockup and softlockup watchdogs, the PowerPC watchdog, the hung taskQuoted 8 lines
detector,
Quoted 1 line
Quoted 1 line
Quoted 1 line
and the panic core. This ensures that each piece of system
information
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
and backtrace output is printed at most once per lockup/panic event, and the state is reset cleanly when a lockup does not trigger a
panic.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Races between
sys_info()callers are ignored. It should beacceptable
Quoted 24 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
because the output from various watchdogs has never been
synchronized.
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
And
panic()never returns.Assisted-by gemini-1.5-flashSigned-off-by Petr Mladek <[email protected]>Yep. There are cases that people want panic on task-hung or sw/hw
lockup,
Quoted 8 lines
Quoted 3 lines
and this could remove much duplication of sys info dump, thanks!
Reviewed-by Feng Tang <[email protected]>Thanks,
im feeling a new file to do all the force panic jazz, but putting tape on
sys_info.cisn't bd either.I wonder how to move forward with this.
Honestly, I am not sure what exactly you mean by creating another API for tracking the reports so I could not judge it. Feel free to sent some POC.
Quoted 7 lines
Otherwise, I would go with my proposal to remember the printed states by the
sys_infoAPI. I am not sure whether I should send a proper patch or you would like to somehow improve it.Best Regards,
Petr
Quoted 4 lines
sup petr, here's my poc
This should make my entire thing make sense
Actually, looks like churn to me.
we shouldn't do that.
It is on that gist though.
As in, I'm thinking about doing a new API to perhaps, make sys_info better?
But it's very complicated, and may need discussion, feel free to ignore that POC...
crappy old me, need a coffee heh.
Thanks!
Petr Mladek
reviewerRe: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces
In reply to Bradley MorganQuoted 41 lines · Bradley Morgan
Quoted 13 lines · Petr Mladek
Quoted 8 lines · Bradley Morgan
Quoted 6 lines · Feng Tang
Quoted 5 lines · Petr Mladek
Quoted 1 line · Petr Mladek
On Thu 2026-06-25 15:25:58, Bradley Morgan wrote:
In watchdog, panic, and hung task detection scenarios,
sys_info()can be called multiple times or alongside direct backtrace triggers liketrigger_allbutcpu_cpu_backtrace(). This results in identicalbacktraces
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
being dumped repeatedly from all CPUs, cluttering the kernel log and delaying or obscuring critical debug details.
Quoted 8 lines
Quoted 2 lines
im feeling a new file to do all the force panic jazz, but putting tape on
sys_info.cisn't bd either.I wonder how to move forward with this.
Honestly, I am not sure what exactly you mean by creating another API for tracking the reports so I could not judge it. Feel free to sent some POC.
sup petr, here's my poc
This should make my entire thing make sense
Quoted 1 line
From
eb587ed749ff5993c517f29799b369185c5ee7d8Mon Sep 17 00:00:00 2001From: Bradley Morgan <[email protected]>
Date: Thu, 2 Jul 2026 18:09:23 +0000
Subject: [POC]sys_info: Introduce incident state-tracking to prevent
duplicate diagnosticsIn watchdog, panic, and hung task detection scenarios,
sys_info()can be called multiple times or alongside direct debug output functions (liketrigger_allbutcpu_cpu_backtrace(),print_modules(),print_irqtrace_events(), anddump_stack()). This leads to identical diagnostics and stack traces being dumped repeatedly, cluttering the kernel log and delaying critical panics.
Quoted 2 lines
Introduce a state tracking bitmask and helpers in a new file,
lib/sys_info_filter.c:
New file suggests that it would implement an API using sys_info_filter() prefix.
Quoted 3 lines
-
sys_info_filter_and_set(mask): Atomically tests which bits in a mask have not yet been printed during the current incident, marks them as printed, and returns that subset.
The name of the funtion is a kind of puzzle. I think that we could do a better job.
Quoted 1 line
-
sys_info_reset(): Clears the printed mask state.
This function has sys_info* prefix. It would expect it in sys_info.c
Quoted 3 lines
Add
SYS_INFO_MODULES,SYS_INFO_IRQTRACE, andSYS_INFO_STACKflags toinclude/linux/sys_info.h, and handle them inside sys_info's diagnostic dispatch.
I though about adding an information that we printed backtrace for this CPU as well. But it not trivial. Different API shows different extra info, like modules, IRQ backtrace, registers, code. I would leave this complexity aside for now.
Quoted 19 lines
Update the watchdogs, hung task detector, and panic core to call
sys_info_filter_and_set()to deduplicate their diagnostic printouts, andsys_info_reset()when a warning incident concludes (e.g., when a stuck CPU recovers, or a new hung task check round begins).This ensures each piece of system diagnostic is printed at most once per lockup/panic event, preventing console log spam.
Assisted-by Gemini:gemini-3.5-flashSigned-off-by Bradley Morgan <[email protected]>lib/sys_info_filter.c new file +5
-0@@ -0,0 +1,120 @@ +static unsigned long sys_info_printed; + +unsigned long sys_info_filter_and_set (unsigned long si_mask) +{ +unsigned long old, new;
lib/sys_info_filter.c line 15
Quoted 10 lines
lib/sys_info_filter.c new file +10
-0+ +if (!si_mask) +return 0 ; + +do { + old =READ_ONCE (sys_info_printed); +if (!(si_mask & ~old)) +return 0 ; + new = old | si_mask; + }while (cmpxchg (&sys_info_printed, old, new) != old);
It is a good question whether to update the info using atomic operations. One problem is that the mask is "unsigned long". I am not sure if it natively atomic on all architectures. 32-bit architecures use extra locking when implementing atomic operations with 64-bit values. And we should rather avoid any locking in this code.
Well, long seems to be 32-bit on 32-bit x86 so it might be safe after all.
lib/sys_info_filter.c line 113
Quoted 3 lines
lib/sys_info_filter.c new file +3
-0+void sys_info_reset(void) +static void __sys_info(unsigned long si_mask) +void sys_info(unsigned long si_mask)
I wonder why this sys_info*() API implementation has been moved from sys_info.c to sys_info_filter.c.
I am sorry but I do not see any advantage in adding the new file sys_info_filter.c
Quoted 2 lines
NOTE!!: This is AI generated!! This **MAY** not be the finished product, this is ONLY the model!
IMHO, Gemini did pretty bad job in this case. Please, try to review the AI generated before you send it. And send it only when you think that it is reasonable enough. :-)
It is even fine to send "crap" but you should start the mail with a warning that you send it just give us an idea what you had it mind. And you should explain why you actually do not like.
Best Regards,
Petr
Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces
Quoted 129 lines · Petr Mladek
Quoted 9 lines · Bradley Morgan
Quoted 7 lines · Petr Mladek
Quoted 6 lines · Bradley Morgan
Quoted 4 lines · Feng Tang
Quoted 3 lines · Petr Mladek
Quoted 1 line · Petr Mladek
On Thu 2026-06-25 15:25:58, Bradley Morgan wrote:
In watchdog, panic, and hung task detection scenarios,
sys_info()can
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
be called multiple times or alongside direct backtrace triggers
like
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
trigger_allbutcpu_cpu_backtrace(). This results in identicalbacktraces
Quoted 1 line
Quoted 1 line
Quoted 1 line
being dumped repeatedly from all CPUs, cluttering the kernel log
and
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
delaying or obscuring critical debug details.
im feeling a new file to do all the force panic jazz, but putting
tape
Quoted 25 lines
Quoted 7 lines
Quoted 1 line
on
sys_info.cisn't bd either.I wonder how to move forward with this.
Honestly, I am not sure what exactly you mean by creating another API for tracking the reports so I could not judge it. Feel free to sent some POC.
sup petr, here's my poc
This should make my entire thing make sense
Quoted 1 line
From
eb587ed749ff5993c517f29799b369185c5ee7d8Mon Sep 17 00:00:00 2001From: Bradley Morgan <[email protected]>
Date: Thu, 2 Jul 2026 18:09:23 +0000
Subject: [POC]sys_info: Introduce incident state-tracking to prevent
duplicate diagnosticsIn watchdog, panic, and hung task detection scenarios,
sys_info()can be called multiple times or alongside direct debug output functions (liketrigger_allbutcpu_cpu_backtrace(),print_modules(),print_irqtrace_events(), anddump_stack()). This leads to identical diagnostics and stack traces being dumped repeatedly, cluttering the kernel log and delaying critical panics.Quoted 2 lines
Introduce a state tracking bitmask and helpers in a new file,
lib/sys_info_filter.c:New file suggests that it would implement an API using
sys_info_filter()prefix.Quoted 3 lines
-
sys_info_filter_and_set(mask): Atomically tests which bits in a mask have not yet been printed during the current incident, marks them as printed, and returns that subset.The name of the funtion is a kind of puzzle. I think that we could do a better job.
Quoted 1 line
-
sys_info_reset(): Clears the printed mask state.This function has sys_info* prefix. It would expect it in
sys_info.cQuoted 3 lines
Add
SYS_INFO_MODULES,SYS_INFO_IRQTRACE, andSYS_INFO_STACKflags toinclude/linux/sys_info.h, and handle them inside sys_info's diagnostic dispatch.I though about adding an information that we printed backtrace for this CPU as well. But it not trivial. Different API shows different extra info, like modules, IRQ backtrace, registers, code. I would leave this complexity aside for now.
Quoted 19 lines
Update the watchdogs, hung task detector, and panic core to call
sys_info_filter_and_set()to deduplicate their diagnostic printouts, andsys_info_reset()when a warning incident concludes (e.g., when a stuck CPU recovers, or a new hung task check round begins).This ensures each piece of system diagnostic is printed at most once per lockup/panic event, preventing console log spam.
Assisted-by Gemini:gemini-3.5-flashSigned-off-by Bradley Morgan <[email protected]>lib/sys_info_filter.c new file +5
-0@@ -0,0 +1,120 @@ +static unsigned long sys_info_printed; + +unsigned long sys_info_filter_and_set (unsigned long si_mask) +{ +unsigned long old, new;lib/sys_info_filter.c line 15
Quoted 10 lines
lib/sys_info_filter.c new file +10
-0+ +if (!si_mask) +return 0 ; + +do { + old =READ_ONCE (sys_info_printed); +if (!(si_mask & ~old)) +return 0 ; + new = old | si_mask; + }while (cmpxchg (&sys_info_printed, old, new) != old);It is a good question whether to update the info using atomic operations. One problem is that the mask is "unsigned long". I am not sure if it natively atomic on all architectures. 32-bit architecures use extra locking when implementing atomic operations with 64-bit values. And we should rather avoid any locking in this code.
Well, long seems to be 32-bit on 32-bit x86 so it might be safe after all.
lib/sys_info_filter.c line 113
Quoted 3 lines
lib/sys_info_filter.c new file +3
-0+void sys_info_reset(void) +static void __sys_info(unsigned long si_mask) +void sys_info(unsigned long si_mask)I wonder why this sys_info*() API implementation has been moved from
sys_info.ctosys_info_filter.c.I am sorry but I do not see any advantage in adding the new file
sys_info_filter.cQuoted 2 lines
NOTE!!: This is AI generated!! This **MAY** not be the finished product, this is ONLY the model!
IMHO, Gemini did pretty bad job in this case. Please, try to review the AI generated before you send it. And send it only when you think that it is reasonable enough. :-)
Quoted 7 lines
It is even fine to send "crap" but you should start the mail with a warning that you send it just give us an idea what you had it mind. And you should explain why you actually do not like.
Best Regards,
Petr
for now, I'll go with your approach, I'll split up and submit your patch(es) in the coming days.
Because the whole new file idea is super complicated and requires a load of discussion before a model could be completed.
One of my ideas is to just kill sys_info. and/or make it better.
Other ideas I need to think.
Thanks a lot for reviewing my model though.
Thanks!