Skip to contentSkip to conversation

sys_info: prevent duplicate backtraces

PATCH In review Bradley Morgan 3 participants 23 messages Jun 25 – Jul 3

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v3 cover sys_info: prevent duplicate backtraces

Some callers handle SYS_INFO_ALL_BT themselves before calling sys_info(). When they strip that bit, an all_bt-only mask becomes zero and sys_info(0) falls back to kernel_si_mask, potentially duplicating output.

This series adds sys_info_with_filter() to filter specific bits without triggering the kernel_si_mask fallback.

Changes since v2:
- Use sys_info_with_filter() instead of sys_info_without_all_bt() per Petr's suggestion
- Filter applied at __sys_info() level to handle kernel_si_mask correctly
- Added panic.c conversion

Bradley Morgan (4):
sys_info: add helper for callers that print some sys_info on their own
watchdog: use sys_info_with_filter() to avoid duplicate backtraces powerpc/watchdog: use sys_info_with_filter() to avoid duplicate
backtraces
panic: use sys_info_with_filter() to avoid duplicate backtraces

 arch/powerpc/kernel/watchdog.c | 12 ++++++++----
 include/linux/sys_info.h       |  1 +
 kernel/panic.c                 |  2 +-
 kernel/watchdog.c              | 12 ++++++++----
 lib/sys_info.c                 | 20 ++++++++++++++++++--
 5 files changed, 36 insertions(+), 11 deletions(-)
Signature · 1 line
2.53.0
#1 lore raw

PATCH v3 1/4 sys_info: add helper for callers that print some sys_info on their own

Some callers print some sys_info on their own before calling sys_info().

Add a helper which would allow to prevent a duplicated output.

It is a bit tricky because kernel_si_mask should be used only when the call-specific si_mask is empty. But the duplicated output must be prevented there as well.

Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys info on system lockup")
---
 include/linux/sys_info.h |  1 +
 lib/sys_info.c           | 20 ++++++++++++++++++--
 2 files changed, 19 insertions(+), 2 deletions(-)
include/linux/sys_info.h +1 -0
index a5bc3ea3d44b..f1c2552ca3d1 100644
@@ -18,6 +18,7 @@
 #define SYS_INFO_BLOCKED_TASKS		0x00000080
 
 void sys_info(unsigned long si_mask);
+void sys_info_with_filter(unsigned long si_mask, unsigned long si_ignore_mask);
 unsigned long sys_info_parse_param(char *str);
 
 #ifdef CONFIG_SYSCTL
lib/sys_info.c +18 -2
index f32a06ec9ed4..d411fee10415 100644
@@ -136,8 +136,10 @@ static int __init sys_info_sysctl_init(void)
 subsys_initcall(sys_info_sysctl_init);
 #endif
 
-static void __sys_info(unsigned long si_mask)
+static void __sys_info(unsigned long si_mask, unsigned long si_ignore_mask)
 {
+	si_mask &= ~si_ignore_mask;
+
 	if (si_mask & SYS_INFO_TASKS)
 		show_state();
 
@@ -160,7 +162,21 @@ static void __sys_info(unsigned long si_mask)
 		show_state_filter(TASK_UNINTERRUPTIBLE);
 }
 
+void sys_info_with_filter(unsigned long si_mask, unsigned long si_ignore_mask)
+{
+	unsigned long dump_mask = si_mask & ~si_ignore_mask;
+
+	/*
+	 * Do not fall back to kernel_si_mask when the caller context
+	 * required only the ignored information.
+	 */
+	if (si_mask && !dump_mask)
+		return;
+
+	__sys_info(dump_mask ? : kernel_si_mask, si_ignore_mask);
+}
+
 void sys_info(unsigned long si_mask)
 {
-	__sys_info(si_mask ? : kernel_si_mask);
+	sys_info_with_filter(si_mask, 0);
 }
Signature · 1 line
2.53.0
#2 lore raw

PATCH v3 2/4 watchdog: use sys_info_with_filter() to avoid duplicate backtraces

In reply to Bradley Morgan

The watchdog prints all CPU backtraces itself. When the watchdog mask contains only SYS_INFO_ALL_BT, stripping that bit leaves zero and sys_info(0) falls back to kernel_sys_info.

Use sys_info_with_filter() so an explicit all_bt mask does not request the global default.

Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys info on system lockup")
---
 kernel/watchdog.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
kernel/watchdog.c +8 -4
index 87dd5e0f6968..ff284593cb90 100644
@@ -208,6 +208,7 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs)
 {
 	int hardlockup_all_cpu_backtrace;
 	unsigned int this_cpu;
+	unsigned long si_mask;
 	unsigned long flags;
 
 	if (per_cpu(watchdog_hardlockup_touched, cpu)) {
@@ -216,7 +217,8 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs)
 		return;
 	}
 
-	hardlockup_all_cpu_backtrace = (hardlockup_si_mask & SYS_INFO_ALL_BT) ?
+	si_mask = READ_ONCE(hardlockup_si_mask);
+	hardlockup_all_cpu_backtrace = (si_mask & SYS_INFO_ALL_BT) ?
 					1 : sysctl_hardlockup_all_cpu_backtrace;
 	/*
 	 * Check for a hardlockup by making sure the CPU's timer
@@ -286,7 +288,7 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs)
 			clear_bit_unlock(0, &hard_lockup_nmi_warn);
 	}
 
-	sys_info(hardlockup_si_mask & ~SYS_INFO_ALL_BT);
+	sys_info_with_filter(si_mask, SYS_INFO_ALL_BT);
 	if (hardlockup_panic)
 		nmi_panic(regs, "Hard LOCKUP");
 
@@ -798,6 +800,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
 	struct pt_regs *regs = get_irq_regs();
 	int softlockup_all_cpu_backtrace;
 	int duration, thresh_count;
+	unsigned long si_mask;
 	unsigned long flags;
 
 	if (!watchdog_enabled)
@@ -809,7 +812,8 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
 	if (panic_in_progress())
 		return HRTIMER_NORESTART;
 
-	softlockup_all_cpu_backtrace = (softlockup_si_mask & SYS_INFO_ALL_BT) ?
+	si_mask = READ_ONCE(softlockup_si_mask);
+	softlockup_all_cpu_backtrace = (si_mask & SYS_INFO_ALL_BT) ?
 					1 : sysctl_softlockup_all_cpu_backtrace;
 
 	watchdog_hardlockup_kick();
@@ -900,7 +904,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
 		}
 
 		add_taint(TAINT_SOFTLOCKUP, LOCKDEP_STILL_OK);
-		sys_info(softlockup_si_mask & ~SYS_INFO_ALL_BT);
+		sys_info_with_filter(si_mask, SYS_INFO_ALL_BT);
 		thresh_count = duration / get_softlockup_thresh();
 
 		if (softlockup_panic && thresh_count >= softlockup_panic)
Signature · 1 line
2.53.0
#3 lore raw

PATCH v3 3/4 powerpc/watchdog: use sys_info_with_filter() to avoid duplicate backtraces

In reply to Bradley Morgan

The powerpc watchdog prints all CPU backtraces itself. When the watchdog mask contains only SYS_INFO_ALL_BT, stripping that bit leaves zero and sys_info(0) falls back to kernel_sys_info.

Use sys_info_with_filter() so an explicit all_bt mask does not request the global default.

Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys info on system lockup")
---
 arch/powerpc/kernel/watchdog.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
arch/powerpc/kernel/watchdog.c +8 -4
index c40c69368476..d3a9c6da962d 100644
@@ -201,6 +201,7 @@ static bool set_cpu_stuck(int cpu)
 static void watchdog_smp_panic(int cpu)
 {
 	static cpumask_t wd_smp_cpus_ipi; // protected by reporting
+	unsigned long si_mask;
 	unsigned long flags;
 	u64 tb, last_reset;
 	int c;
@@ -236,8 +237,9 @@ static void watchdog_smp_panic(int cpu)
 	pr_emerg("CPU %d TB:%lld, last SMP heartbeat TB:%lld (%lldms ago)\n",
 		 cpu, tb, last_reset, tb_to_ns(tb - last_reset) / 1000000);
 
+	si_mask = READ_ONCE(hardlockup_si_mask);
 	if (sysctl_hardlockup_all_cpu_backtrace ||
-	    (hardlockup_si_mask & SYS_INFO_ALL_BT)) {
+	    (si_mask & SYS_INFO_ALL_BT)) {
 		trigger_allbutcpu_cpu_backtrace(cpu);
 		cpumask_clear(&wd_smp_cpus_ipi);
 	} else {
@@ -251,7 +253,7 @@ static void watchdog_smp_panic(int cpu)
 		}
 	}
 
-	sys_info(hardlockup_si_mask & ~SYS_INFO_ALL_BT);
+	sys_info_with_filter(si_mask, SYS_INFO_ALL_BT);
 	if (hardlockup_panic)
 		nmi_panic(NULL, "Hard LOCKUP");
 
@@ -371,6 +373,7 @@ static void watchdog_timer_interrupt(int cpu)
 
 DEFINE_INTERRUPT_HANDLER_NMI(soft_nmi_interrupt)
 {
+	unsigned long si_mask;
 	unsigned long flags;
 	int cpu = raw_smp_processor_id();
 	u64 tb;
@@ -418,11 +421,12 @@ DEFINE_INTERRUPT_HANDLER_NMI(soft_nmi_interrupt)
 
 		xchg(&__wd_nmi_output, 1); // see wd_lockup_ipi
 
+		si_mask = READ_ONCE(hardlockup_si_mask);
 		if (sysctl_hardlockup_all_cpu_backtrace ||
-		    (hardlockup_si_mask & SYS_INFO_ALL_BT))
+		    (si_mask & SYS_INFO_ALL_BT))
 			trigger_allbutcpu_cpu_backtrace(cpu);
 
-		sys_info(hardlockup_si_mask & ~SYS_INFO_ALL_BT);
+		sys_info_with_filter(si_mask, SYS_INFO_ALL_BT);
 		if (hardlockup_panic)
 			nmi_panic(regs, "Hard LOCKUP");
 
Signature · 1 line
2.53.0
#5 lore raw

PATCH v3 4/4 panic: use sys_info_with_filter() to avoid duplicate backtraces

In reply to Bradley Morgan

panic_other_cpus_shutdown() handles SYS_INFO_ALL_BT before stopping the other CPUs. Do not ask sys_info() to handle that bit again later in the panic path.

Use sys_info_with_filter() so panic_print=all_bt does not request more output after the CPUs are stopped.

Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys info on system lockup")
---
 kernel/panic.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/panic.c +1 -1
index 213725b612aa..eb842823df61 100644
@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)
 	 */
 	atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
 
-	sys_info(panic_print);
+	sys_info_with_filter(panic_print, SYS_INFO_ALL_BT);
 
 	kmsg_dump_desc(KMSG_DUMP_PANIC, buf);
 
Signature · 1 line
2.53.0
reviewer
#4 lore raw

Re: [PATCH v3 3/4] powerpc/watchdog: use sys_info_with_filter() to avoid duplicate backtraces

In reply to Bradley Morgan
Quoted 13 lines · Bradley Morgan

The powerpc watchdog prints all CPU backtraces itself. When the watchdog mask contains only SYS_INFO_ALL_BT, stripping that bit leaves zero and sys_info(0) falls back to kernel_sys_info.

Use sys_info_with_filter() so an explicit all_bt mask does not request the global default.

arch/powerpc/kernel/watchdog.c +0 -0
@@ -418,11 +421,12 @@ DEFINE_INTERRUPT_HANDLER_NMI(soft_nmi_interrupt)
 
 		xchg(&__wd_nmi_output, 1); // see wd_lockup_ipi
 

arch/powerpc/kernel/watchdog.c DEFINE_INTERRUPT_HANDLER_NMI(soft_nmi_interrupt) · line 431

Quoted 10 lines
arch/powerpc/kernel/watchdog.c +3 -2
+		si_mask = READ_ONCE(hardlockup_si_mask);
 		if (sysctl_hardlockup_all_cpu_backtrace ||
-		    (hardlockup_si_mask & SYS_INFO_ALL_BT))
+		    (si_mask & SYS_INFO_ALL_BT))
 			trigger_allbutcpu_cpu_backtrace(cpu);
 
-		sys_info(hardlockup_si_mask & ~SYS_INFO_ALL_BT);
+		sys_info_with_filter(si_mask, SYS_INFO_ALL_BT);
 		if (hardlockup_panic)
 			nmi_panic(regs, "Hard LOCKUP");

I thought more about it and it is even more complicated.

Even if we prevent the duplicated output with sys_info_with_filter() here. Then nmi_panic() might still trigger it once again.

We could say that this patch is a step in the right direction and fix the other problem later. But I am not sure. We might need a completely different approach and this is just a step aside.

And there is another problem in the panic() code. I am going to comment in it in the 4th patch.

Best Regards,
Petr

reviewer
#6 lore raw

Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces

In reply to Bradley Morgan
Quoted 19 lines · Bradley Morgan

panic_other_cpus_shutdown() handles SYS_INFO_ALL_BT before stopping the other CPUs. Do not ask sys_info() to handle that bit again later in the panic path.

Use sys_info_with_filter() so panic_print=all_bt does not request more output after the CPUs are stopped.

Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys info on system lockup")
---
 kernel/panic.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/panic.c +0 -0
index 213725b612aa..eb842823df61 100644

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683

Quoted 5 lines
kernel/panic.c +1 -1
@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)
 	 */
 	atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
 
-	sys_info(panic_print);
+	sys_info_with_filter(panic_print, SYS_INFO_ALL_BT);

Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?

They might be printed by:

static void panic_other_cpus_shutdown(bool crash_kexec)

{
	if (panic_print & SYS_INFO_ALL_BT)
		panic_trigger_all_cpu_backtrace();

[...]
}

But it checks only "panic_print" variable. It won't do anything when (panic_print == 0).

In this case, we might still want to print the backraces when SYS_INFO_ALL_BT is set in kernel_si_info.

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 685

Quoted 1 line
kernel/panic.c +0 -0
 	kmsg_dump_desc(KMSG_DUMP_PANIC, buf);

Of course, we might fix panic_other_cpus_shutdown() to check also kernel_si_info.

But it all becomes very hairy. We have several levels:

+ watchdog-all_bt-specific option, e.g. sysctl_hardlockup_all_cpu_backtrace

+ watchdog-specific si_info preferences, e.g. hardlockup_si_mask

+ panic-specific si_info: panic_print

+ universal fallback for any layer: kernel_si_info

Now, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.

I think about another approach. The word "waterfall" comes to my mind. Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.

All the si_info actions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.

I am going to think more about it.

Please, do not send v4 until the discussion settles!

Best Regards,
Petr

#7 lore raw

Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces

Quoted 66 lines · Petr Mladek
Quoted 8 lines · Bradley Morgan

panic_other_cpus_shutdown() handles SYS_INFO_ALL_BT before stopping the other CPUs. Do not ask sys_info() to handle that bit again later in the panic path.

Use sys_info_with_filter() so panic_print=all_bt does not request more output after the CPUs are stopped.

Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys info on

system lockup")

Quoted 11 lines
---
 kernel/panic.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/panic.c +0 -0
index 213725b612aa..eb842823df61 100644

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683

Quoted 5 lines
kernel/panic.c +1 -1
@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)
 	 */
 	atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
 
-	sys_info(panic_print);
+	sys_info_with_filter(panic_print, SYS_INFO_ALL_BT);

Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?

They might be printed by:

static void panic_other_cpus_shutdown(bool crash_kexec)

{
	if (panic_print & SYS_INFO_ALL_BT)
		panic_trigger_all_cpu_backtrace();

[...]
}

But it checks only "panic_print" variable. It won't do anything when (panic_print == 0).

In this case, we might still want to print the backraces when SYS_INFO_ALL_BT is set in kernel_si_info.

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 685

Quoted 1 line
kernel/panic.c +0 -0
 	kmsg_dump_desc(KMSG_DUMP_PANIC, buf);

Of course, we might fix panic_other_cpus_shutdown() to check also kernel_si_info.

But it all becomes very hairy. We have several levels:

+ watchdog-all_bt-specific option, e.g. sysctl_hardlockup_all_cpu_backtrace

+ watchdog-specific si_info preferences, e.g. hardlockup_si_mask

+ panic-specific si_info: panic_print

+ universal fallback for any layer: kernel_si_info

Now, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.

Quoted 8 lines

I think about another approach. The word "waterfall" comes to my mind. Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.

All the si_info actions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.

Not a bad idea.

Quoted 3 lines

I am going to think more about it.

Please, do not send v4 until the discussion settles!

I'll hold on V4.

When you've finished discussing, could I have your suggested patch? if I think there is issues. I'll fix it.

Quoted 3 lines

Best Regards,
Petr

Thanks!

#8 lore raw

Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces

In reply to Petr Mladek
Quoted 36 lines · Petr Mladek
Quoted 19 lines · Bradley Morgan

panic_other_cpus_shutdown() handles SYS_INFO_ALL_BT before stopping the other CPUs. Do not ask sys_info() to handle that bit again later in the panic path.

Use sys_info_with_filter() so panic_print=all_bt does not request more output after the CPUs are stopped.

Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys info on system lockup")
---
 kernel/panic.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/panic.c +0 -0
index 213725b612aa..eb842823df61 100644

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683

Quoted 5 lines
kernel/panic.c +1 -1
@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)
 	 */
 	atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
 
-	sys_info(panic_print);
+	sys_info_with_filter(panic_print, SYS_INFO_ALL_BT);

Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?

They might be printed by:

static void panic_other_cpus_shutdown(bool crash_kexec)

{
	if (panic_print & SYS_INFO_ALL_BT)
		panic_trigger_all_cpu_backtrace();
Quoted 8 lines

[...]
}

But it checks only "panic_print" variable. It won't do anything when (panic_print == 0).

In this case, we might still want to print the backraces when SYS_INFO_ALL_BT is set in kernel_si_info.

Yep.

Quoted 12 lines

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 685

Quoted 2 lines
kernel/panic.c +0 -0
 	kmsg_dump_desc(KMSG_DUMP_PANIC, buf);

Of course, we might fix panic_other_cpus_shutdown() to check also kernel_si_info.

But it all becomes very hairy. We have several levels:

+ watchdog-all_bt-specific option, e.g. sysctl_hardlockup_all_cpu_backtrace

+ watchdog-specific si_info preferences, e.g. hardlockup_si_mask

Quoted 8 lines

+ panic-specific si_info: panic_print

+ universal fallback for any layer: kernel_si_info

Now, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.

Agree :)

Quoted 4 lines

I think about another approach. The word "waterfall" comes to my mind. Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.

When initially reviewing V2's 4th patch, I thought about the 'panic_this_cpu_backtrace_printed', but it's a local variable which records the state.

Quoted 3 lines

All the si_info actions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.

IIUC, panic case is kind of special, as it has to separate the 'sys_info()' op in different stage. Can we do a merge in the start of vpanic() by:

	panic_print = panic_print ?: kernel_si_mask;

as a addon patch ?

Thanks,
Feng

Quoted 6 lines

I am going to think more about it.

Please, do not send v4 until the discussion settles!

Best Regards,
Petr

reviewer
#9 lore raw

Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces

In reply to Petr Mladek
Quoted 70 lines · Petr Mladek
Quoted 19 lines · Bradley Morgan

panic_other_cpus_shutdown() handles SYS_INFO_ALL_BT before stopping the other CPUs. Do not ask sys_info() to handle that bit again later in the panic path.

Use sys_info_with_filter() so panic_print=all_bt does not request more output after the CPUs are stopped.

Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys info on system lockup")
---
 kernel/panic.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/panic.c +0 -0
index 213725b612aa..eb842823df61 100644

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683

Quoted 5 lines
kernel/panic.c +1 -1
@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)
 	 */
 	atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
 
-	sys_info(panic_print);
+	sys_info_with_filter(panic_print, SYS_INFO_ALL_BT);

Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?

They might be printed by:

static void panic_other_cpus_shutdown(bool crash_kexec)

{
	if (panic_print & SYS_INFO_ALL_BT)
		panic_trigger_all_cpu_backtrace();

[...]
}

But it checks only "panic_print" variable. It won't do anything when (panic_print == 0).

In this case, we might still want to print the backraces when SYS_INFO_ALL_BT is set in kernel_si_info.

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 684

Quoted 1 line
kernel/panic.c +0 -0
 	kmsg_dump_desc(KMSG_DUMP_PANIC, buf);

Of course, we might fix panic_other_cpus_shutdown() to check also kernel_si_info.

But it all becomes very hairy. We have several levels:

+ watchdog-all_bt-specific option, e.g. sysctl_hardlockup_all_cpu_backtrace

+ watchdog-specific si_info preferences, e.g. hardlockup_si_mask

+ panic-specific si_info: panic_print

+ universal fallback for any layer: kernel_si_info

Now, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.

I think about another approach. The word "waterfall" comes to my mind. Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.

Quoted 5 lines

All the si_info actions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.

I am going to think more about it.

I have created a POC using Gemini. I haven't tested it. But it looks acceptable. And the logic seems to be more straightforward.

One drawback is that it requires adding the _reset() call for all sys_info() callers. It is fine in principle but it might complicate back-porting because all changes have to be done in one patch.

But honestly, this is a nice to have fix. Most people could live happily without it.

From 3c66436d9978030845a96bfaedd6b914536e2ac4 Mon Sep 17 00:00:00 2001
From: Petr Mladek <[email protected]>
Date: Fri, 26 Jun 2026 13:55:41 +0200
Subject: [POC] sys_info: Introduce state-tracking APIs to prevent duplicate
backtraces

In watchdog, panic, and hung task detection scenarios, sys_info() can be called multiple times or alongside direct backtrace triggers like trigger_allbutcpu_cpu_backtrace(). This results in identical backtraces being dumped repeatedly from all CPUs, cluttering the kernel log and delaying or obscuring critical debug details.

Introduce a state tracking bitmask and associated helpers:
- sys_info_done(mask): Marks specific sys_info bits as already printed.
- sys_info_reset(): Resets the tracking state.
- sys_info_is_done(mask): Checks if all bits in the mask have been printed.

Update sys_info() to automatically filter out already printed bits using this state. Integrate these APIs with the generic hardlockup and softlockup watchdogs, the PowerPC watchdog, the hung task detector, and the panic core. This ensures that each piece of system information and backtrace output is printed at most once per lockup/panic event, and the state is reset cleanly when a lockup does not trigger a panic.

Races between sys_info() callers are ignored. It should be acceptable because the output from various watchdogs has never been synchronized. And panic() never returns.

Assisted-by gemini-1.5-flash
---
 arch/powerpc/kernel/watchdog.c | 13 ++++++++++---
 include/linux/sys_info.h       |  3 +++
 kernel/hung_task.c             |  2 ++
 kernel/panic.c                 |  4 +++-
 kernel/watchdog.c              | 10 ++++++++--
 lib/sys_info.c                 | 30 +++++++++++++++++++++++++++++-
 6 files changed, 55 insertions(+), 7 deletions(-)
arch/powerpc/kernel/watchdog.c +10 -3
index c40c69368476..0eab7894b9dc 100644
@@ -239,6 +239,7 @@ static void watchdog_smp_panic(int cpu)
 	if (sysctl_hardlockup_all_cpu_backtrace ||
 	    (hardlockup_si_mask & SYS_INFO_ALL_BT)) {
 		trigger_allbutcpu_cpu_backtrace(cpu);
+		sys_info_done(SYS_INFO_ALL_BT);
 		cpumask_clear(&wd_smp_cpus_ipi);
 	} else {
 		/*
@@ -251,10 +252,12 @@ static void watchdog_smp_panic(int cpu)
 		}
 	}
 
-	sys_info(hardlockup_si_mask & ~SYS_INFO_ALL_BT);
+	sys_info(hardlockup_si_mask);
 	if (hardlockup_panic)
 		nmi_panic(NULL, "Hard LOCKUP");
 
+	sys_info_reset();
+
 	wd_end_reporting();
 
 	return;
@@ -419,13 +422,17 @@ DEFINE_INTERRUPT_HANDLER_NMI(soft_nmi_interrupt)
 		xchg(&__wd_nmi_output, 1); // see wd_lockup_ipi
 
 		if (sysctl_hardlockup_all_cpu_backtrace ||
-		    (hardlockup_si_mask & SYS_INFO_ALL_BT))
+		    (hardlockup_si_mask & SYS_INFO_ALL_BT)) {
 			trigger_allbutcpu_cpu_backtrace(cpu);
+			sys_info_done(SYS_INFO_ALL_BT);
+		}
 
-		sys_info(hardlockup_si_mask & ~SYS_INFO_ALL_BT);
+		sys_info(hardlockup_si_mask);
 		if (hardlockup_panic)
 			nmi_panic(regs, "Hard LOCKUP");
 
+		sys_info_reset();
+
 		wd_end_reporting();
 	}
 	/*
include/linux/sys_info.h +3 -0
index a5bc3ea3d44b..ad43548c75dd 100644
@@ -18,6 +18,9 @@
 #define SYS_INFO_BLOCKED_TASKS		0x00000080
 
 void sys_info(unsigned long si_mask);
+void sys_info_done(unsigned long si_mask);
+void sys_info_reset(void);
+bool sys_info_is_done(unsigned long si_mask);
 unsigned long sys_info_parse_param(char *str);
 
 #ifdef CONFIG_SYSCTL
kernel/hung_task.c +2 -0
index 6fcc94ce4ca9..dbb6a27770f5 100644
@@ -354,6 +354,8 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 
 	if (hung_task_call_panic)
 		panic("hung_task: blocked tasks");
+
+	sys_info_reset();
 }
 
 static long hung_timeout_jiffies(unsigned long last_checked,
kernel/panic.c +3 -1
index 213725b612aa..86ce17f03da2 100644
@@ -550,8 +550,10 @@ static void panic_trigger_all_cpu_backtrace(void)
  */
 static void panic_other_cpus_shutdown(bool crash_kexec)
 {
-	if (panic_print & SYS_INFO_ALL_BT)
+	if ((panic_print & SYS_INFO_ALL_BT) && !sys_info_is_done(SYS_INFO_ALL_BT)) {
 		panic_trigger_all_cpu_backtrace();
+		sys_info_done(SYS_INFO_ALL_BT);
+	}
 
 	/*
 	 * Note that smp_send_stop() is the usual SMP shutdown function,
kernel/watchdog.c +8 -2
index 87dd5e0f6968..f431087c68a7 100644
@@ -282,14 +282,17 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs)
 
 	if (hardlockup_all_cpu_backtrace) {
 		trigger_allbutcpu_cpu_backtrace(cpu);
+		sys_info_done(SYS_INFO_ALL_BT);
 		if (!hardlockup_panic)
 			clear_bit_unlock(0, &hard_lockup_nmi_warn);
 	}
 
-	sys_info(hardlockup_si_mask & ~SYS_INFO_ALL_BT);
+	sys_info(hardlockup_si_mask);
 	if (hardlockup_panic)
 		nmi_panic(regs, "Hard LOCKUP");
 
+	sys_info_reset();
+
 	per_cpu(watchdog_hardlockup_warned, cpu) = true;
 }
 
@@ -895,16 +898,19 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
 
 		if (softlockup_all_cpu_backtrace) {
 			trigger_allbutcpu_cpu_backtrace(smp_processor_id());
+			sys_info_done(SYS_INFO_ALL_BT);
 			if (!softlockup_panic)
 				clear_bit_unlock(0, &soft_lockup_nmi_warn);
 		}
 
 		add_taint(TAINT_SOFTLOCKUP, LOCKDEP_STILL_OK);
-		sys_info(softlockup_si_mask & ~SYS_INFO_ALL_BT);
+		sys_info(softlockup_si_mask);
 		thresh_count = duration / get_softlockup_thresh();
 
 		if (softlockup_panic && thresh_count >= softlockup_panic)
 			panic("softlockup: hung tasks");
+
+		sys_info_reset();
 	}
 
 	return HRTIMER_RESTART;
lib/sys_info.c +29 -1
index f32a06ec9ed4..f8e6176fae75 100644
@@ -160,7 +160,35 @@ static void __sys_info(unsigned long si_mask)
 		show_state_filter(TASK_UNINTERRUPTIBLE);
 }
 
+static unsigned long sys_info_done_mask;
+
+void sys_info_done(unsigned long si_mask)
+{
+	sys_info_done_mask |= si_mask;
+}
+
+void sys_info_reset(void)
+{
+	sys_info_done_mask = 0;
+}
+
+bool sys_info_is_done(unsigned long si_mask)
+{
+	return (sys_info_done_mask & si_mask) == si_mask;
+}
+
 void sys_info(unsigned long si_mask)
 {
-	__sys_info(si_mask ? : kernel_si_mask);
+	unsigned long mask;
+
+	if (si_mask)
+		mask = si_mask & ~sys_info_done_mask;
+	else
+		mask = kernel_si_mask & ~sys_info_done_mask;
+
+	if (!mask)
+		return;
+
+	__sys_info(mask);
+	sys_info_done(mask);
 }
Signature · 1 line
2.54.0
#10 lore raw

Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces

Quoted 117 lines · Petr Mladek
Quoted 2 lines · Petr Mladek
Quoted 1 line · Bradley Morgan

panic_other_cpus_shutdown() handles SYS_INFO_ALL_BT before stopping

the

Quoted 1 line
Quoted 1 line

other CPUs. Do not ask sys_info() to handle that bit again later in

the

Quoted 6 lines
Quoted 6 lines

panic path.

Use sys_info_with_filter() so panic_print=all_bt does not request more output after the CPUs are stopped.

Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys info

on system lockup")

Quoted 36 lines
Quoted 11 lines
---
 kernel/panic.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/panic.c +0 -0
index 213725b612aa..eb842823df61 100644

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683

Quoted 5 lines
kernel/panic.c +1 -1
@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)
 	 */
 	atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
 
-	sys_info(panic_print);
+	sys_info_with_filter(panic_print, SYS_INFO_ALL_BT);

Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?

They might be printed by:

static void panic_other_cpus_shutdown(bool crash_kexec)

{
	if (panic_print & SYS_INFO_ALL_BT)
		panic_trigger_all_cpu_backtrace();

[...]
}

But it checks only "panic_print" variable. It won't do anything when (panic_print == 0).

In this case, we might still want to print the backraces when SYS_INFO_ALL_BT is set in kernel_si_info.

Quoted 8 lines

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 684

Quoted 1 line
kernel/panic.c +0 -0
 	kmsg_dump_desc(KMSG_DUMP_PANIC, buf);

Of course, we might fix panic_other_cpus_shutdown() to check also kernel_si_info.

But it all becomes very hairy. We have several levels:

+ watchdog-all_bt-specific option, e.g.

sysctl_hardlockup_all_cpu_backtrace

Quoted 17 lines

+ watchdog-specific si_info preferences, e.g. hardlockup_si_mask

+ panic-specific si_info: panic_print

+ universal fallback for any layer: kernel_si_info

Now, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.

I think about another approach. The word "waterfall" comes to my mind. Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.

Quoted 5 lines

All the si_info actions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.

I am going to think more about it.

I have created a POC using Gemini. I haven't tested it. But it looks acceptable. And the logic seems to be more straightforward.

One drawback is that it requires adding the _reset() call for all sys_info() callers. It is fine in principle but it might complicate back-porting because all changes have to be done in one patch.

But honestly, this is a nice to have fix. Most people could live happily without it.

From 3c66436d9978030845a96bfaedd6b914536e2ac4 Mon Sep 17 00:00:00 2001
From: Petr Mladek <[email protected]>
Date: Fri, 26 Jun 2026 13:55:41 +0200
Subject: [POC] sys_info: Introduce state-tracking APIs to prevent duplicate backtraces

In watchdog, panic, and hung task detection scenarios, sys_info() can be called multiple times or alongside direct backtrace triggers like trigger_allbutcpu_cpu_backtrace(). This results in identical backtraces being dumped repeatedly from all CPUs, cluttering the kernel log and delaying or obscuring critical debug details.

Introduce a state tracking bitmask and associated helpers:
- sys_info_done(mask): Marks specific sys_info bits as already printed.
- sys_info_reset(): Resets the tracking state.
- sys_info_is_done(mask): Checks if all bits in the mask have been printed.

Update sys_info() to automatically filter out already printed bits using this state. Integrate these APIs with the generic hardlockup and softlockup watchdogs, the PowerPC watchdog, the hung task detector, and the panic core. This ensures that each piece of system information and backtrace output is printed at most once per lockup/panic event, and the state is reset cleanly when a lockup does not trigger a panic.

Quoted 5 lines

Races between sys_info() callers are ignored. It should be acceptable because the output from various watchdogs has never been synchronized. And panic() never returns.

Assisted-by gemini-1.5-flash ?

Why not use gemini 3.5 flash?

I can try if you want.

Could I have the prompt you used? :)

Quoted 175 lines

---

arch/powerpc/kernel/watchdog.c | 13 ++++++++++---
include/linux/sys_info.h       |  3 +++
kernel/hung_task.c             |  2 ++
kernel/panic.c                 |  4 +++-
kernel/watchdog.c              | 10 ++++++++--
lib/sys_info.c                 | 30 +++++++++++++++++++++++++++++-
6 files changed, 55 insertions(+), 7 deletions(-)
arch/powerpc/kernel/watchdog.c +10 -3
index c40c69368476..0eab7894b9dc 100644
@@ -239,6 +239,7 @@ static void watchdog_smp_panic(int cpu)
 	if (sysctl_hardlockup_all_cpu_backtrace ||
 	    (hardlockup_si_mask & SYS_INFO_ALL_BT)) {
 		trigger_allbutcpu_cpu_backtrace(cpu);
+		sys_info_done(SYS_INFO_ALL_BT);
 		cpumask_clear(&wd_smp_cpus_ipi);
 	} else {
 		/*
@@ -251,10 +252,12 @@ static void watchdog_smp_panic(int cpu)
 		}
 	}
 
-	sys_info(hardlockup_si_mask & ~SYS_INFO_ALL_BT);
+	sys_info(hardlockup_si_mask);
 	if (hardlockup_panic)
 		nmi_panic(NULL, "Hard LOCKUP");
 
+	sys_info_reset();
+
 	wd_end_reporting();
 
 	return;
@@ -419,13 +422,17 @@ DEFINE_INTERRUPT_HANDLER_NMI(soft_nmi_interrupt)
 		xchg(&__wd_nmi_output, 1); // see wd_lockup_ipi
 
 		if (sysctl_hardlockup_all_cpu_backtrace ||
-		    (hardlockup_si_mask & SYS_INFO_ALL_BT))
+		    (hardlockup_si_mask & SYS_INFO_ALL_BT)) {
 			trigger_allbutcpu_cpu_backtrace(cpu);
+			sys_info_done(SYS_INFO_ALL_BT);
+		}
 
-		sys_info(hardlockup_si_mask & ~SYS_INFO_ALL_BT);
+		sys_info(hardlockup_si_mask);
 		if (hardlockup_panic)
 			nmi_panic(regs, "Hard LOCKUP");
 
+		sys_info_reset();
+
 		wd_end_reporting();
 	}
 	/*
include/linux/sys_info.h +3 -0
index a5bc3ea3d44b..ad43548c75dd 100644
@@ -18,6 +18,9 @@
 #define SYS_INFO_BLOCKED_TASKS		0x00000080
 
 void sys_info(unsigned long si_mask);
+void sys_info_done(unsigned long si_mask);
+void sys_info_reset(void);
+bool sys_info_is_done(unsigned long si_mask);
 unsigned long sys_info_parse_param(char *str);
 
 #ifdef CONFIG_SYSCTL
kernel/hung_task.c +2 -0
index 6fcc94ce4ca9..dbb6a27770f5 100644
@@ -354,6 +354,8 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 
 	if (hung_task_call_panic)
 		panic("hung_task: blocked tasks");
+
+	sys_info_reset();
 }
 
 static long hung_timeout_jiffies(unsigned long last_checked,
kernel/panic.c +3 -1
index 213725b612aa..86ce17f03da2 100644
@@ -550,8 +550,10 @@ static void panic_trigger_all_cpu_backtrace(void)
 */
 static void panic_other_cpus_shutdown(bool crash_kexec)
 {
-	if (panic_print & SYS_INFO_ALL_BT)
+	if ((panic_print & SYS_INFO_ALL_BT) && !sys_info_is_done(SYS_INFO_ALL_BT)) {
 		panic_trigger_all_cpu_backtrace();
+		sys_info_done(SYS_INFO_ALL_BT);
+	}
 
 	/*
 	 * Note that smp_send_stop() is the usual SMP shutdown function,
kernel/watchdog.c +8 -2
index 87dd5e0f6968..f431087c68a7 100644
@@ -282,14 +282,17 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs)
 
 	if (hardlockup_all_cpu_backtrace) {
 		trigger_allbutcpu_cpu_backtrace(cpu);
+		sys_info_done(SYS_INFO_ALL_BT);
 		if (!hardlockup_panic)
 			clear_bit_unlock(0, &hard_lockup_nmi_warn);
 	}
 
-	sys_info(hardlockup_si_mask & ~SYS_INFO_ALL_BT);
+	sys_info(hardlockup_si_mask);
 	if (hardlockup_panic)
 		nmi_panic(regs, "Hard LOCKUP");
 
+	sys_info_reset();
+
 	per_cpu(watchdog_hardlockup_warned, cpu) = true;
 }
 
@@ -895,16 +898,19 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
 
 		if (softlockup_all_cpu_backtrace) {
 			trigger_allbutcpu_cpu_backtrace(smp_processor_id());
+			sys_info_done(SYS_INFO_ALL_BT);
 			if (!softlockup_panic)
 				clear_bit_unlock(0, &soft_lockup_nmi_warn);
 		}
 
 		add_taint(TAINT_SOFTLOCKUP, LOCKDEP_STILL_OK);
-		sys_info(softlockup_si_mask & ~SYS_INFO_ALL_BT);
+		sys_info(softlockup_si_mask);
 		thresh_count = duration / get_softlockup_thresh();
 
 		if (softlockup_panic && thresh_count >= softlockup_panic)
 			panic("softlockup: hung tasks");
+
+		sys_info_reset();
 	}
 
 	return HRTIMER_RESTART;
lib/sys_info.c +20 -1
index f32a06ec9ed4..f8e6176fae75 100644
@@ -160,7 +160,35 @@ static void __sys_info(unsigned long si_mask)
 		show_state_filter(TASK_UNINTERRUPTIBLE);
 }
 
+static unsigned long sys_info_done_mask;
+
+void sys_info_done(unsigned long si_mask)
+{
+	sys_info_done_mask |= si_mask;
+}
+
+void sys_info_reset(void)
+{
+	sys_info_done_mask = 0;
+}
+
+bool sys_info_is_done(unsigned long si_mask)
+{
+	return (sys_info_done_mask & si_mask) == si_mask;
+}
+
 void sys_info(unsigned long si_mask)
 {
-	__sys_info(si_mask ? : kernel_si_mask);
+	unsigned long mask;
+
+	if (si_mask)

lib/sys_info.c static void __sys_info(unsigned long si_mask) · line 194

Quoted 10 lines
lib/sys_info.c +9 -0
+		mask = si_mask & ~sys_info_done_mask;
+	else
+		mask = kernel_si_mask & ~sys_info_done_mask;
+
+	if (!mask)
+		return;
+
+	__sys_info(mask);
+	sys_info_done(mask);
 }

Thanks!

#11 lore raw

Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces

Quoted 317 lines · Bradley Morgan
Quoted 9 lines · Petr Mladek
Quoted 2 lines · Petr Mladek
Quoted 1 line · Bradley Morgan

panic_other_cpus_shutdown() handles SYS_INFO_ALL_BT before stopping

the

Quoted 1 line
Quoted 1 line

other CPUs. Do not ask sys_info() to handle that bit again later in

the

Quoted 3 lines
Quoted 3 lines

panic path.

Use sys_info_with_filter() so panic_print=all_bt does not request

more

Quoted 81 lines
Quoted 3 lines
Quoted 3 lines

output after the CPUs are stopped.

Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys info

on system lockup")

Quoted 36 lines
Quoted 11 lines
---
 kernel/panic.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/panic.c +0 -0
index 213725b612aa..eb842823df61 100644

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683

Quoted 5 lines
kernel/panic.c +1 -1
@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)
 	 */
 	atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
 
-	sys_info(panic_print);
+	sys_info_with_filter(panic_print, SYS_INFO_ALL_BT);

Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?

They might be printed by:

static void panic_other_cpus_shutdown(bool crash_kexec)

{
	if (panic_print & SYS_INFO_ALL_BT)
		panic_trigger_all_cpu_backtrace();

[...]
}

But it checks only "panic_print" variable. It won't do anything when (panic_print == 0).

In this case, we might still want to print the backraces when SYS_INFO_ALL_BT is set in kernel_si_info.

Quoted 8 lines

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 684

Quoted 1 line
kernel/panic.c +0 -0
 	kmsg_dump_desc(KMSG_DUMP_PANIC, buf);

Of course, we might fix panic_other_cpus_shutdown() to check also kernel_si_info.

But it all becomes very hairy. We have several levels:

+ watchdog-all_bt-specific option, e.g.

sysctl_hardlockup_all_cpu_backtrace

Quoted 17 lines

+ watchdog-specific si_info preferences, e.g. hardlockup_si_mask

+ panic-specific si_info: panic_print

+ universal fallback for any layer: kernel_si_info

Now, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.

I think about another approach. The word "waterfall" comes to my mind. Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.

Quoted 5 lines

All the si_info actions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.

I am going to think more about it.

I have created a POC using Gemini. I haven't tested it. But it looks acceptable. And the logic seems to be more straightforward.

One drawback is that it requires adding the _reset() call for all sys_info() callers. It is fine in principle but it might complicate back-porting because all changes have to be done in one patch.

Quoted 7 lines

But honestly, this is a nice to have fix. Most people could live happily without it.

From 3c66436d9978030845a96bfaedd6b914536e2ac4 Mon Sep 17 00:00:00 2001
From: Petr Mladek <[email protected]>
Date: Fri, 26 Jun 2026 13:55:41 +0200
Subject: [POC] sys_info: Introduce state-tracking APIs to prevent

duplicate

Quoted 8 lines

backtraces

In watchdog, panic, and hung task detection scenarios, sys_info() can be called multiple times or alongside direct backtrace triggers like trigger_allbutcpu_cpu_backtrace(). This results in identical backtraces being dumped repeatedly from all CPUs, cluttering the kernel log and delaying or obscuring critical debug details.

Quoted 4 lines

Introduce a state tracking bitmask and associated helpers:
- sys_info_done(mask): Marks specific sys_info bits as already printed.
- sys_info_reset(): Resets the tracking state.
- sys_info_is_done(mask): Checks if all bits in the mask have been

printed.

Quoted 8 lines

Update sys_info() to automatically filter out already printed bits using this state. Integrate these APIs with the generic hardlockup and softlockup watchdogs, the PowerPC watchdog, the hung task detector, and the panic core. This ensures that each piece of system information and backtrace output is printed at most once per lockup/panic event, and the state is reset cleanly when a lockup does not trigger a panic.

Quoted 5 lines

Races between sys_info() callers are ignored. It should be acceptable because the output from various watchdogs has never been synchronized. And panic() never returns.

Assisted-by gemini-1.5-flash ?

Why not use gemini 3.5 flash?

I can try if you want.

Could I have the prompt you used? :)

Review

Quoted 11 lines

---

arch/powerpc/kernel/watchdog.c | 13 ++++++++++---
include/linux/sys_info.h       |  3 +++
kernel/hung_task.c             |  2 ++
kernel/panic.c                 |  4 +++-
kernel/watchdog.c              | 10 ++++++++--
lib/sys_info.c                 | 30 +++++++++++++++++++++++++++++-
6 files changed, 55 insertions(+), 7 deletions(-)
(quoted diff) +0 -0

b/arch/powerpc/kernel/watchdog.c

kernel/hung_task.c static void check_hung_uninterruptible_tasks(unsigned

Quoted 64 lines

index c40c69368476..0eab7894b9dc 100644

arch/powerpc/kernel/watchdog.c +10 -3
@@ -239,6 +239,7 @@ static void watchdog_smp_panic(int cpu)
 	if (sysctl_hardlockup_all_cpu_backtrace ||
 	    (hardlockup_si_mask & SYS_INFO_ALL_BT)) {
 		trigger_allbutcpu_cpu_backtrace(cpu);
+		sys_info_done(SYS_INFO_ALL_BT);
 		cpumask_clear(&wd_smp_cpus_ipi);
 	} else {
 		/*
@@ -251,10 +252,12 @@ static void watchdog_smp_panic(int cpu)
 		}
 	}
 
-	sys_info(hardlockup_si_mask & ~SYS_INFO_ALL_BT);
+	sys_info(hardlockup_si_mask);
 	if (hardlockup_panic)
 		nmi_panic(NULL, "Hard LOCKUP");
 
+	sys_info_reset();
+
 	wd_end_reporting();
 
 	return;
@@ -419,13 +422,17 @@ DEFINE_INTERRUPT_HANDLER_NMI(soft_nmi_interrupt)
 		xchg(&__wd_nmi_output, 1); // see wd_lockup_ipi
 
 		if (sysctl_hardlockup_all_cpu_backtrace ||
-		    (hardlockup_si_mask & SYS_INFO_ALL_BT))
+		    (hardlockup_si_mask & SYS_INFO_ALL_BT)) {
 			trigger_allbutcpu_cpu_backtrace(cpu);
+			sys_info_done(SYS_INFO_ALL_BT);
+		}
 
-		sys_info(hardlockup_si_mask & ~SYS_INFO_ALL_BT);
+		sys_info(hardlockup_si_mask);
 		if (hardlockup_panic)
 			nmi_panic(regs, "Hard LOCKUP");
 
+		sys_info_reset();
+
 		wd_end_reporting();
 	}
 	/*
include/linux/sys_info.h +3 -0
index a5bc3ea3d44b..ad43548c75dd 100644
@@ -18,6 +18,9 @@
 #define SYS_INFO_BLOCKED_TASKS		0x00000080
 
 void sys_info(unsigned long si_mask);
+void sys_info_done(unsigned long si_mask);
+void sys_info_reset(void);
+bool sys_info_is_done(unsigned long si_mask);
 unsigned long sys_info_parse_param(char *str);
 
 #ifdef CONFIG_SYSCTL
kernel/hung_task.c +0 -0
index 6fcc94ce4ca9..dbb6a27770f5 100644
@@ -354,6 +354,8 @@ static void check_hung_uninterruptible_tasks(unsigned

long timeout)

kernel/watchdog.c void watchdog_hardlockup_check(unsigned int cpu,

Quoted 29 lines
kernel/hung_task.c +2 -0
 	if (hung_task_call_panic)
 		panic("hung_task: blocked tasks");
+
+	sys_info_reset();
 }
 
 static long hung_timeout_jiffies(unsigned long last_checked,
kernel/panic.c +3 -1
index 213725b612aa..86ce17f03da2 100644
@@ -550,8 +550,10 @@ static void panic_trigger_all_cpu_backtrace(void)
 */
 static void panic_other_cpus_shutdown(bool crash_kexec)
 {
-	if (panic_print & SYS_INFO_ALL_BT)
+	if ((panic_print & SYS_INFO_ALL_BT) && !sys_info_is_done(SYS_INFO_ALL_BT)) {
 		panic_trigger_all_cpu_backtrace();
+		sys_info_done(SYS_INFO_ALL_BT);
+	}
 
 	/*
 	 * Note that smp_send_stop() is the usual SMP shutdown function,
kernel/watchdog.c +0 -0
index 87dd5e0f6968..f431087c68a7 100644
@@ -282,14 +282,17 @@ void watchdog_hardlockup_check(unsigned int cpu,

struct pt_regs *regs)

kernel/watchdog.c static enum hrtimer_restart

Quoted 19 lines
kernel/watchdog.c +4 -1
 	if (hardlockup_all_cpu_backtrace) {
 		trigger_allbutcpu_cpu_backtrace(cpu);
+		sys_info_done(SYS_INFO_ALL_BT);
 		if (!hardlockup_panic)
 			clear_bit_unlock(0, &hard_lockup_nmi_warn);
 	}
 
-	sys_info(hardlockup_si_mask & ~SYS_INFO_ALL_BT);
+	sys_info(hardlockup_si_mask);
 	if (hardlockup_panic)
 		nmi_panic(regs, "Hard LOCKUP");
 
+	sys_info_reset();
+
 	per_cpu(watchdog_hardlockup_warned, cpu) = true;
 }
 
@@ -895,16 +898,19 @@ static enum hrtimer_restart

watchdog_timer_fn(struct hrtimer *hrtimer)

Quoted 56 lines
kernel/watchdog.c +4 -1
 		if (softlockup_all_cpu_backtrace) {
 			trigger_allbutcpu_cpu_backtrace(smp_processor_id());
+			sys_info_done(SYS_INFO_ALL_BT);
 			if (!softlockup_panic)
 				clear_bit_unlock(0, &soft_lockup_nmi_warn);
 		}
 
 		add_taint(TAINT_SOFTLOCKUP, LOCKDEP_STILL_OK);
-		sys_info(softlockup_si_mask & ~SYS_INFO_ALL_BT);
+		sys_info(softlockup_si_mask);
 		thresh_count = duration / get_softlockup_thresh();
 
 		if (softlockup_panic && thresh_count >= softlockup_panic)
 			panic("softlockup: hung tasks");
+
+		sys_info_reset();
 	}
 
 	return HRTIMER_RESTART;
lib/sys_info.c +25 -1
index f32a06ec9ed4..f8e6176fae75 100644
@@ -160,7 +160,35 @@ static void __sys_info(unsigned long si_mask)
 		show_state_filter(TASK_UNINTERRUPTIBLE);
 }
 
+static unsigned long sys_info_done_mask;
+
+void sys_info_done(unsigned long si_mask)
+{
+	sys_info_done_mask |= si_mask;
+}
+
+void sys_info_reset(void)
+{
+	sys_info_done_mask = 0;
+}
+
+bool sys_info_is_done(unsigned long si_mask)
+{
+	return (sys_info_done_mask & si_mask) == si_mask;
+}
+
 void sys_info(unsigned long si_mask)
 {
-	__sys_info(si_mask ? : kernel_si_mask);
+	unsigned long mask;
+
+	if (si_mask)
+		mask = si_mask & ~sys_info_done_mask;
+	else
+		mask = kernel_si_mask & ~sys_info_done_mask;
+
+	if (!mask)
Quoted 8 lines

lib/sys_info.c static void __sys_info(unsigned long si_mask) · line 194

Quoted 6 lines
lib/sys_info.c +4 -0
+		return;
+
+	__sys_info(mask);
+	sys_info_done(mask);
 }

Thanks!

Hmm.. new idea

kernel/dump_filter.c ?

What this file could do is to handle a generic lockup state machine so any subsystem can log what it already dumped?

I know it may bloat, but it's better then cramming fixes in.

What do you guys think? Maybe we could start a RFC for this?

Thanks!

reviewer
#12 lore raw

Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces

Quoted 31 lines · Bradley Morgan
Quoted 25 lines · Bradley Morgan
Quoted 23 lines · Petr Mladek
Quoted 4 lines · Petr Mladek

On Thu 2026-06-25 15:25:58, Bradley Morgan wrote:
But it all becomes very hairy. We have several levels:

+ watchdog-all_bt-specific option, e.g.

sysctl_hardlockup_all_cpu_backtrace

Quoted 17 lines

+ watchdog-specific si_info preferences, e.g. hardlockup_si_mask

+ panic-specific si_info: panic_print

+ universal fallback for any layer: kernel_si_info

Now, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.

I think about another approach. The word "waterfall" comes to my mind. Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

All the si_info actions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.

Quoted 8 lines

Hmm.. new idea

kernel/dump_filter.c ?

What this file could do is to handle a generic lockup state machine so any subsystem can log what it already dumped?

I know it may bloat, but it's better then cramming fixes in.

I am not sure what exactly you would like to achieve but it sounds a bit scary ;-)

Anyway, we should not synchronize the watchdog reports against each other, definitely. They are running in non-compatible contexts (task vs interrupt vs NMI). Also we should not add any locking because they usually print something when the system has enough troubles.

Also I think that it is not worth preventing duplicated backtraces or reports from a single CPU. IMHO, it is not a big problem in practice.

So, we are down to large reports, like backtraces from all CPUs, timers, locks, ... which are handled by sys_info(). So, I think that it should be enough to handle this inside the sys_info() API.

I do not want to say that my proposal was the best solution. I am sure that there are better ones. But we need to consider the gain vs. complexity.

Honestly, I am already a bit scared by the complexity which we the sys_info() API added. And it is hard to imagine that adding another API would make it easier. But I might be wrong.

Instead, it might make sense to integrate the conflicting subsystem-specific calls under the sys_info() API. I mean that, for example watchdog_hardlockup_check() won't call trigger_allbutcpu_cpu_backtrace() directly but it would call it via sys_info() API so that sys_info() could keep track of it. Something like:

void sys_info_allbutcpu_bt(int cpu)

{
	trigger_allbutcpu_cpu_backtrace(cpu);
	/*
	 * The caller likely printed backtrace of the given @cpu
	 * on its own. Prevent duplicate backtraces from all
	 * CPUs with potential next sys_info() call.
	 */
	sys_info_done(SYS_INFO_ALL_BT);
}

But I am not sure if it is really easier to follow than calling sys_info_done() from the watchdog code.

Some watchdogs try to optimize the output and print backtraces only from CPUs which are relevant for the given lockup. We should keep the logic for selecting the set of CPUs in the watchdog code. We just need to solve how to elegantly make sys_info() aware of it or at least about the more massive reports.

Anyway, I would prefer to keep it simple until we see some problems in practice.

Best Regards,
Petr

#13 lore raw

Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces

Quoted 96 lines · Petr Mladek
Quoted 1 line · Bradley Morgan

On June 26, 2026 1:17:13 PM GMT+01:00, Bradley Morgan

<[email protected]>

Quoted 14 lines
Quoted 13 lines · wrote:
Quoted 11 lines · Petr Mladek
Quoted 4 lines · Petr Mladek

On Thu 2026-06-25 15:25:58, Bradley Morgan wrote:
But it all becomes very hairy. We have several levels:

+ watchdog-all_bt-specific option, e.g.

sysctl_hardlockup_all_cpu_backtrace

Quoted 5 lines

+ watchdog-specific si_info preferences, e.g. hardlockup_si_mask

+ panic-specific si_info: panic_print

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+ universal fallback for any layer: kernel_si_info

Now, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.

I think about another approach. The word "waterfall" comes to my

mind.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.

All the si_info actions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.

Quoted 8 lines

Hmm.. new idea

kernel/dump_filter.c ?

What this file could do is to handle a generic lockup state machine so any subsystem can log what it already dumped?

I know it may bloat, but it's better then cramming fixes in.

I am not sure what exactly you would like to achieve but it sounds a bit scary ;-)

Anyway, we should not synchronize the watchdog reports against each other, definitely. They are running in non-compatible contexts (task vs interrupt vs NMI). Also we should not add any locking because they usually print something when the system has enough troubles.

Also I think that it is not worth preventing duplicated backtraces or reports from a single CPU. IMHO, it is not a big problem in practice.

So, we are down to large reports, like backtraces from all CPUs, timers, locks, ... which are handled by sys_info(). So, I think that it should be enough to handle this inside the sys_info() API.

I do not want to say that my proposal was the best solution. I am sure that there are better ones. But we need to consider the gain vs. complexity.

Honestly, I am already a bit scared by the complexity which we the sys_info() API added. And it is hard to imagine that adding another API would make it easier. But I might be wrong.

Instead, it might make sense to integrate the conflicting subsystem-specific calls under the sys_info() API. I mean that, for example watchdog_hardlockup_check() won't call trigger_allbutcpu_cpu_backtrace() directly but it would call it via sys_info() API so that sys_info() could keep track of it. Something like:

void sys_info_allbutcpu_bt(int cpu)

{
	trigger_allbutcpu_cpu_backtrace(cpu);
	/*
	 * The caller likely printed backtrace of the given @cpu
	 * on its own. Prevent duplicate backtraces from all
	 * CPUs with potential next sys_info() call.
	 */
	sys_info_done(SYS_INFO_ALL_BT);
}

But I am not sure if it is really easier to follow than calling sys_info_done() from the watchdog code.

Some watchdogs try to optimize the output and print backtraces only from CPUs which are relevant for the given lockup. We should keep the logic for selecting the set of CPUs in the watchdog code. We just need to solve how to elegantly make sys_info() aware of it or at least about the more massive reports.

Quoted 6 lines

Anyway, I would prefer to keep it simple until we see some problems in practice.

Best Regards,
Petr

I understand it's scary. To make a new file in the first place.

But I was a bit vague of what I wanted, and I'm sorry.

So, the reason why I'd suggest a new file, is because if any subsystem Theoretically bypasses sys_info to log a lockup, this completely misses the filter and duplicates the dump

My file would act as a generic lockless state machine that any subsystem can update regardless of how they dump logs.

If you have any questions, feel absolutely free to ask! :)

Discussion is a way to make everyone happy!

Thanks!

reviewer
#14 lore raw

Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces

Quoted 114 lines · Bradley Morgan
Quoted 96 lines · Petr Mladek
Quoted 1 line · Bradley Morgan

On June 26, 2026 1:17:13 PM GMT+01:00, Bradley Morgan

<[email protected]>

Quoted 14 lines
Quoted 13 lines · wrote:
Quoted 11 lines · Petr Mladek
Quoted 4 lines · Petr Mladek

On Thu 2026-06-25 15:25:58, Bradley Morgan wrote:
But it all becomes very hairy. We have several levels:

+ watchdog-all_bt-specific option, e.g.

sysctl_hardlockup_all_cpu_backtrace

Quoted 5 lines

+ watchdog-specific si_info preferences, e.g. hardlockup_si_mask

+ panic-specific si_info: panic_print

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+ universal fallback for any layer: kernel_si_info

Now, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.

I think about another approach. The word "waterfall" comes to my

mind.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.

All the si_info actions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.

Quoted 8 lines

Hmm.. new idea

kernel/dump_filter.c ?

What this file could do is to handle a generic lockup state machine so any subsystem can log what it already dumped?

I know it may bloat, but it's better then cramming fixes in.

I am not sure what exactly you would like to achieve but it sounds a bit scary ;-)

Anyway, we should not synchronize the watchdog reports against each other, definitely. They are running in non-compatible contexts (task vs interrupt vs NMI). Also we should not add any locking because they usually print something when the system has enough troubles.

Also I think that it is not worth preventing duplicated backtraces or reports from a single CPU. IMHO, it is not a big problem in practice.

So, we are down to large reports, like backtraces from all CPUs, timers, locks, ... which are handled by sys_info(). So, I think that it should be enough to handle this inside the sys_info() API.

I do not want to say that my proposal was the best solution. I am sure that there are better ones. But we need to consider the gain vs. complexity.

Honestly, I am already a bit scared by the complexity which we the sys_info() API added. And it is hard to imagine that adding another API would make it easier. But I might be wrong.

Instead, it might make sense to integrate the conflicting subsystem-specific calls under the sys_info() API. I mean that, for example watchdog_hardlockup_check() won't call trigger_allbutcpu_cpu_backtrace() directly but it would call it via sys_info() API so that sys_info() could keep track of it. Something like:

void sys_info_allbutcpu_bt(int cpu)

{
	trigger_allbutcpu_cpu_backtrace(cpu);
	/*
	 * The caller likely printed backtrace of the given @cpu
	 * on its own. Prevent duplicate backtraces from all
	 * CPUs with potential next sys_info() call.
	 */
	sys_info_done(SYS_INFO_ALL_BT);
}

But I am not sure if it is really easier to follow than calling sys_info_done() from the watchdog code.

Some watchdogs try to optimize the output and print backtraces only from CPUs which are relevant for the given lockup. We should keep the logic for selecting the set of CPUs in the watchdog code. We just need to solve how to elegantly make sys_info() aware of it or at least about the more massive reports.

Quoted 6 lines

Anyway, I would prefer to keep it simple until we see some problems in practice.

Best Regards,
Petr

I understand it's scary. To make a new file in the first place.

But I was a bit vague of what I wanted, and I'm sorry.

So, the reason why I'd suggest a new file, is because if any subsystem Theoretically bypasses sys_info to log a lockup, this completely misses the filter and duplicates the dump

Quoted 6 lines

My file would act as a generic lockless state machine that any subsystem can update regardless of how they dump logs.

If you have any questions, feel absolutely free to ask! :)

Discussion is a way to make everyone happy!

Honestly, I am more and more wondering whether your are a real person or AI bot.

Best Regards,
Petr

#15 lore raw

Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces

Quoted 122 lines · Petr Mladek
Quoted 7 lines · Bradley Morgan
Quoted 5 lines · Petr Mladek
Quoted 1 line · Bradley Morgan

On June 26, 2026 1:17:13 PM GMT+01:00, Bradley Morgan

<[email protected]>

Quoted 2 lines
Quoted 1 line · wrote:

On June 26, 2026 1:14:14 PM GMT+01:00, Petr Mladek

<[email protected]>

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines · wrote:
Quoted 4 lines · Petr Mladek

On Thu 2026-06-25 15:25:58, Bradley Morgan wrote:
But it all becomes very hairy. We have several levels:

+ watchdog-all_bt-specific option, e.g.

sysctl_hardlockup_all_cpu_backtrace

Quoted 2 lines

+ watchdog-specific si_info preferences, e.g.

hardlockup_si_mask

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

+ panic-specific si_info: panic_print

+ universal fallback for any layer: kernel_si_info

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Now, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.

I think about another approach. The word "waterfall" comes to my

mind.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Instead of checking all the settings back and forth, let's

process

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

each setting one by one and just remember what has been done and skip this in the next level.

All the si_info actions seems to dump a global system state. So, it would make sense to remember the state in a global

variable

Quoted 83 lines
Quoted 64 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

even when it might be modified by more CPUs in parallel.

Hmm.. new idea

kernel/dump_filter.c ?

What this file could do is to handle a generic lockup state machine so any subsystem can log what it already dumped?

I know it may bloat, but it's better then cramming fixes in.

I am not sure what exactly you would like to achieve but it sounds a bit scary ;-)

Anyway, we should not synchronize the watchdog reports against each other, definitely. They are running in non-compatible contexts (task vs interrupt vs NMI). Also we should not add any locking because they usually print something when the system has enough troubles.

Also I think that it is not worth preventing duplicated backtraces or reports from a single CPU. IMHO, it is not a big problem in practice.

So, we are down to large reports, like backtraces from all CPUs, timers, locks, ... which are handled by sys_info(). So, I think that it should be enough to handle this inside the sys_info() API.

I do not want to say that my proposal was the best solution. I am sure that there are better ones. But we need to consider the gain vs. complexity.

Honestly, I am already a bit scared by the complexity which we the sys_info() API added. And it is hard to imagine that adding another API would make it easier. But I might be wrong.

Instead, it might make sense to integrate the conflicting subsystem-specific calls under the sys_info() API. I mean that, for example watchdog_hardlockup_check() won't call trigger_allbutcpu_cpu_backtrace() directly but it would call it via sys_info() API so that sys_info() could keep track of it. Something like:

void sys_info_allbutcpu_bt(int cpu)

{
	trigger_allbutcpu_cpu_backtrace(cpu);
	/*
	 * The caller likely printed backtrace of the given @cpu
	 * on its own. Prevent duplicate backtraces from all
	 * CPUs with potential next sys_info() call.
	 */
	sys_info_done(SYS_INFO_ALL_BT);
}

But I am not sure if it is really easier to follow than calling sys_info_done() from the watchdog code.

Some watchdogs try to optimize the output and print backtraces only from CPUs which are relevant for the given lockup. We should keep the logic for selecting the set of CPUs in the watchdog code. We just need to solve how to elegantly make sys_info() aware of it or at least about the more massive reports.

Quoted 6 lines

Anyway, I would prefer to keep it simple until we see some problems in practice.

Best Regards,
Petr

I understand it's scary. To make a new file in the first place.

But I was a bit vague of what I wanted, and I'm sorry.

So, the reason why I'd suggest a new file, is because if any subsystem Theoretically bypasses sys_info to log a lockup, this completely misses the filter and duplicates the dump

My file would act as a generic lockless state machine that any subsystem can update regardless of how they dump logs.

Quoted 6 lines
Quoted 3 lines

If you have any questions, feel absolutely free to ask! :)

Discussion is a way to make everyone happy!

Honestly, I am more and more wondering whether your are a real person or AI bot.

Sigh..

I can verify myself through video call if you don't believe I am human :)

why I suggested a new file is because AI said it would be a good idea.

I told it what I should do, and it told me to do a new file.

I knew it was over engineering slightly, but I was a bit stressed, and I wanted some sort of just new API which is less buggy imho

I should've told you that I used AI to figure the whole new file idea,

Really sorry petr..

Quoted 3 lines

Best Regards,
Petr

Thanks!

Reviewed-by
#16 lore raw

Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces

In reply to Petr Mladek
Quoted 113 lines · Petr Mladek
Quoted 70 lines · Petr Mladek
Quoted 19 lines · Bradley Morgan

panic_other_cpus_shutdown() handles SYS_INFO_ALL_BT before stopping the other CPUs. Do not ask sys_info() to handle that bit again later in the panic path.

Use sys_info_with_filter() so panic_print=all_bt does not request more output after the CPUs are stopped.

Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys info on system lockup")
---
 kernel/panic.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/panic.c +0 -0
index 213725b612aa..eb842823df61 100644

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683

Quoted 5 lines
kernel/panic.c +1 -1
@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)
 	 */
 	atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
 
-	sys_info(panic_print);
+	sys_info_with_filter(panic_print, SYS_INFO_ALL_BT);

Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?

They might be printed by:

static void panic_other_cpus_shutdown(bool crash_kexec)

{
	if (panic_print & SYS_INFO_ALL_BT)
		panic_trigger_all_cpu_backtrace();

[...]
}

But it checks only "panic_print" variable. It won't do anything when (panic_print == 0).

In this case, we might still want to print the backraces when SYS_INFO_ALL_BT is set in kernel_si_info.

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 684

Quoted 1 line
kernel/panic.c +0 -0
 	kmsg_dump_desc(KMSG_DUMP_PANIC, buf);

Of course, we might fix panic_other_cpus_shutdown() to check also kernel_si_info.

But it all becomes very hairy. We have several levels:

+ watchdog-all_bt-specific option, e.g. sysctl_hardlockup_all_cpu_backtrace

+ watchdog-specific si_info preferences, e.g. hardlockup_si_mask

+ panic-specific si_info: panic_print

+ universal fallback for any layer: kernel_si_info

Now, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.

I think about another approach. The word "waterfall" comes to my mind. Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.

Quoted 5 lines

All the si_info actions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.

I am going to think more about it.

I have created a POC using Gemini. I haven't tested it. But it looks acceptable. And the logic seems to be more straightforward.

One drawback is that it requires adding the _reset() call for all sys_info() callers. It is fine in principle but it might complicate back-porting because all changes have to be done in one patch.

But honestly, this is a nice to have fix. Most people could live happily without it.

From 3c66436d9978030845a96bfaedd6b914536e2ac4 Mon Sep 17 00:00:00 2001
From: Petr Mladek <[email protected]>
Date: Fri, 26 Jun 2026 13:55:41 +0200
Subject: [POC] sys_info: Introduce state-tracking APIs to prevent duplicate
backtraces

In watchdog, panic, and hung task detection scenarios, sys_info() can be called multiple times or alongside direct backtrace triggers like trigger_allbutcpu_cpu_backtrace(). This results in identical backtraces being dumped repeatedly from all CPUs, cluttering the kernel log and delaying or obscuring critical debug details.

Introduce a state tracking bitmask and associated helpers:
- sys_info_done(mask): Marks specific sys_info bits as already printed.
- sys_info_reset(): Resets the tracking state.
- sys_info_is_done(mask): Checks if all bits in the mask have been printed.

Update sys_info() to automatically filter out already printed bits using this state. Integrate these APIs with the generic hardlockup and softlockup watchdogs, the PowerPC watchdog, the hung task detector, and the panic core. This ensures that each piece of system information and backtrace output is printed at most once per lockup/panic event, and the state is reset cleanly when a lockup does not trigger a panic.

Quoted 6 lines

Races between sys_info() callers are ignored. It should be acceptable because the output from various watchdogs has never been synchronized. And panic() never returns.

Assisted-by gemini-1.5-flash

Yep. There are cases that people want panic on task-hung or sw/hw lockup, and this could remove much duplication of sys info dump, thanks!

Quoted 8 lines
---
 arch/powerpc/kernel/watchdog.c | 13 ++++++++++---
 include/linux/sys_info.h       |  3 +++
 kernel/hung_task.c             |  2 ++
 kernel/panic.c                 |  4 +++-
 kernel/watchdog.c              | 10 ++++++++--
 lib/sys_info.c                 | 30 +++++++++++++++++++++++++++++-
 6 files changed, 55 insertions(+), 7 deletions(-)
#17 lore raw

Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces

Quoted 125 lines · Feng Tang
Quoted 3 lines · Petr Mladek
Quoted 2 lines · Petr Mladek
Quoted 1 line · Bradley Morgan

panic_other_cpus_shutdown() handles SYS_INFO_ALL_BT before stopping

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

other CPUs. Do not ask sys_info() to handle that bit again later in

the

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

panic path.

Use sys_info_with_filter() so panic_print=all_bt does not request

more

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

output after the CPUs are stopped.

Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys

info on system lockup")

Quoted 36 lines
Quoted 36 lines
Quoted 11 lines
---
 kernel/panic.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/panic.c +0 -0
index 213725b612aa..eb842823df61 100644

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683

Quoted 5 lines
kernel/panic.c +1 -1
@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)
 	 */
 	atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
 
-	sys_info(panic_print);
+	sys_info_with_filter(panic_print, SYS_INFO_ALL_BT);

Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?

They might be printed by:

static void panic_other_cpus_shutdown(bool crash_kexec)

{
	if (panic_print & SYS_INFO_ALL_BT)
		panic_trigger_all_cpu_backtrace();

[...]
}

But it checks only "panic_print" variable. It won't do anything when (panic_print == 0).

In this case, we might still want to print the backraces when SYS_INFO_ALL_BT is set in kernel_si_info.

Quoted 8 lines
Quoted 8 lines

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 684

Quoted 1 line
kernel/panic.c +0 -0
 	kmsg_dump_desc(KMSG_DUMP_PANIC, buf);

Of course, we might fix panic_other_cpus_shutdown() to check also kernel_si_info.

But it all becomes very hairy. We have several levels:

+ watchdog-all_bt-specific option, e.g.

sysctl_hardlockup_all_cpu_backtrace

Quoted 32 lines
Quoted 17 lines

+ watchdog-specific si_info preferences, e.g. hardlockup_si_mask

+ panic-specific si_info: panic_print

+ universal fallback for any layer: kernel_si_info

Now, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.

I think about another approach. The word "waterfall" comes to my mind. Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.

Quoted 5 lines

All the si_info actions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.

I am going to think more about it.

I have created a POC using Gemini. I haven't tested it. But it looks acceptable. And the logic seems to be more straightforward.

One drawback is that it requires adding the _reset() call for all sys_info() callers. It is fine in principle but it might complicate back-porting because all changes have to be done in one patch.

Quoted 7 lines

But honestly, this is a nice to have fix. Most people could live happily without it.

From 3c66436d9978030845a96bfaedd6b914536e2ac4 Mon Sep 17 00:00:00 2001
From: Petr Mladek <[email protected]>
Date: Fri, 26 Jun 2026 13:55:41 +0200
Subject: [POC] sys_info: Introduce state-tracking APIs to prevent

duplicate

Quoted 8 lines

backtraces

In watchdog, panic, and hung task detection scenarios, sys_info() can be called multiple times or alongside direct backtrace triggers like trigger_allbutcpu_cpu_backtrace(). This results in identical backtraces being dumped repeatedly from all CPUs, cluttering the kernel log and delaying or obscuring critical debug details.

Quoted 4 lines

Introduce a state tracking bitmask and associated helpers:
- sys_info_done(mask): Marks specific sys_info bits as already printed.
- sys_info_reset(): Resets the tracking state.
- sys_info_is_done(mask): Checks if all bits in the mask have been

printed.

Quoted 12 lines

Update sys_info() to automatically filter out already printed bits using this state. Integrate these APIs with the generic hardlockup and softlockup watchdogs, the PowerPC watchdog, the hung task detector, and the panic core. This ensures that each piece of system information and backtrace output is printed at most once per lockup/panic event, and the state is reset cleanly when a lockup does not trigger a panic.

Races between sys_info() callers are ignored. It should be acceptable because the output from various watchdogs has never been synchronized. And panic() never returns.

Quoted 7 lines
Quoted 2 lines
Assisted-by gemini-1.5-flash

Yep. There are cases that people want panic on task-hung or sw/hw lockup, and this could remove much duplication of sys info dump, thanks!

Thanks,

im feeling a new file to do all the force panic jazz, but putting tape on sys_info.c isn't bd either.

Quoted 9 lines
Quoted 8 lines
---
 arch/powerpc/kernel/watchdog.c | 13 ++++++++++---
 include/linux/sys_info.h       |  3 +++
 kernel/hung_task.c             |  2 ++
 kernel/panic.c                 |  4 +++-
 kernel/watchdog.c              | 10 ++++++++--
 lib/sys_info.c                 | 30 +++++++++++++++++++++++++++++-
 6 files changed, 55 insertions(+), 7 deletions(-)

Thanks!

reviewer
#18 lore raw

Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces

Quoted 133 lines · Bradley Morgan
Quoted 131 lines · Feng Tang
Quoted 3 lines · Petr Mladek
Quoted 2 lines · Petr Mladek
Quoted 1 line · Bradley Morgan

panic_other_cpus_shutdown() handles SYS_INFO_ALL_BT before stopping

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

other CPUs. Do not ask sys_info() to handle that bit again later in

the

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

panic path.

Use sys_info_with_filter() so panic_print=all_bt does not request

more

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

output after the CPUs are stopped.

Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys

info on system lockup")

Quoted 36 lines
Quoted 36 lines
Quoted 11 lines
---
 kernel/panic.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/panic.c +0 -0
index 213725b612aa..eb842823df61 100644

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683

Quoted 5 lines
kernel/panic.c +1 -1
@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)
 	 */
 	atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
 
-	sys_info(panic_print);
+	sys_info_with_filter(panic_print, SYS_INFO_ALL_BT);

Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?

They might be printed by:

static void panic_other_cpus_shutdown(bool crash_kexec)

{
	if (panic_print & SYS_INFO_ALL_BT)
		panic_trigger_all_cpu_backtrace();

[...]
}

But it checks only "panic_print" variable. It won't do anything when (panic_print == 0).

In this case, we might still want to print the backraces when SYS_INFO_ALL_BT is set in kernel_si_info.

Quoted 8 lines
Quoted 8 lines

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 684

Quoted 1 line
kernel/panic.c +0 -0
 	kmsg_dump_desc(KMSG_DUMP_PANIC, buf);

Of course, we might fix panic_other_cpus_shutdown() to check also kernel_si_info.

But it all becomes very hairy. We have several levels:

+ watchdog-all_bt-specific option, e.g.

sysctl_hardlockup_all_cpu_backtrace

Quoted 32 lines
Quoted 17 lines

+ watchdog-specific si_info preferences, e.g. hardlockup_si_mask

+ panic-specific si_info: panic_print

+ universal fallback for any layer: kernel_si_info

Now, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.

I think about another approach. The word "waterfall" comes to my mind. Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.

Quoted 5 lines

All the si_info actions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.

I am going to think more about it.

I have created a POC using Gemini. I haven't tested it. But it looks acceptable. And the logic seems to be more straightforward.

One drawback is that it requires adding the _reset() call for all sys_info() callers. It is fine in principle but it might complicate back-porting because all changes have to be done in one patch.

Quoted 7 lines

But honestly, this is a nice to have fix. Most people could live happily without it.

From 3c66436d9978030845a96bfaedd6b914536e2ac4 Mon Sep 17 00:00:00 2001
From: Petr Mladek <[email protected]>
Date: Fri, 26 Jun 2026 13:55:41 +0200
Subject: [POC] sys_info: Introduce state-tracking APIs to prevent

duplicate

Quoted 8 lines

backtraces

In watchdog, panic, and hung task detection scenarios, sys_info() can be called multiple times or alongside direct backtrace triggers like trigger_allbutcpu_cpu_backtrace(). This results in identical backtraces being dumped repeatedly from all CPUs, cluttering the kernel log and delaying or obscuring critical debug details.

Quoted 4 lines

Introduce a state tracking bitmask and associated helpers:
- sys_info_done(mask): Marks specific sys_info bits as already printed.
- sys_info_reset(): Resets the tracking state.
- sys_info_is_done(mask): Checks if all bits in the mask have been

printed.

Quoted 8 lines

Update sys_info() to automatically filter out already printed bits using this state. Integrate these APIs with the generic hardlockup and softlockup watchdogs, the PowerPC watchdog, the hung task detector, and the panic core. This ensures that each piece of system information and backtrace output is printed at most once per lockup/panic event, and the state is reset cleanly when a lockup does not trigger a panic.

Quoted 6 lines

Races between sys_info() callers are ignored. It should be acceptable because the output from various watchdogs has never been synchronized. And panic() never returns.

Assisted-by gemini-1.5-flash

Yep. There are cases that people want panic on task-hung or sw/hw lockup, and this could remove much duplication of sys info dump, thanks!

Quoted 6 lines
Quoted 1 line

Thanks,

im feeling a new file to do all the force panic jazz, but putting tape on sys_info.c isn't bd either.

I wonder how to move forward with this.

Honestly, I am not sure what exactly you mean by creating another API for tracking the reports so I could not judge it. Feel free to sent some POC.

Otherwise, I would go with my proposal to remember the printed states by the sys_info API. I am not sure whether I should send a proper patch or you would like to somehow improve it.

Best Regards,
Petr

#19 lore raw

Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces

Quoted 158 lines · Petr Mladek
Quoted 6 lines · Bradley Morgan
Quoted 4 lines · Feng Tang
Quoted 3 lines · Petr Mladek
Quoted 2 lines · Petr Mladek
Quoted 1 line · Bradley Morgan

panic_other_cpus_shutdown() handles SYS_INFO_ALL_BT before

stopping

Quoted 2 lines
Quoted 2 lines

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

other CPUs. Do not ask sys_info() to handle that bit again later

in

Quoted 59 lines
Quoted 59 lines

the

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

panic path.

Use sys_info_with_filter() so panic_print=all_bt does not request

more

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

output after the CPUs are stopped.

Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys

info on system lockup")

Quoted 36 lines
Quoted 36 lines
Quoted 11 lines
---
 kernel/panic.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/panic.c +0 -0
index 213725b612aa..eb842823df61 100644

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683

Quoted 5 lines
kernel/panic.c +1 -1
@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)
 	 */
 	atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
 
-	sys_info(panic_print);
+	sys_info_with_filter(panic_print, SYS_INFO_ALL_BT);

Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?

They might be printed by:

static void panic_other_cpus_shutdown(bool crash_kexec)

{
	if (panic_print & SYS_INFO_ALL_BT)
		panic_trigger_all_cpu_backtrace();

[...]
}

But it checks only "panic_print" variable. It won't do anything when (panic_print == 0).

In this case, we might still want to print the backraces when SYS_INFO_ALL_BT is set in kernel_si_info.

Quoted 8 lines
Quoted 8 lines

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 684

Quoted 1 line
kernel/panic.c +0 -0
 	kmsg_dump_desc(KMSG_DUMP_PANIC, buf);

Of course, we might fix panic_other_cpus_shutdown() to check also kernel_si_info.

But it all becomes very hairy. We have several levels:

+ watchdog-all_bt-specific option, e.g.

sysctl_hardlockup_all_cpu_backtrace

Quoted 5 lines
Quoted 5 lines

+ watchdog-specific si_info preferences, e.g. hardlockup_si_mask

+ panic-specific si_info: panic_print

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+ universal fallback for any layer: kernel_si_info

Now, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.

I think about another approach. The word "waterfall" comes to my

mind.

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 9 lines

Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.

All the si_info actions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.

I am going to think more about it.

I have created a POC using Gemini. I haven't tested it. But it looks acceptable. And the logic seems to be more straightforward.

One drawback is that it requires adding the _reset() call for all sys_info() callers. It is fine in principle but it might complicate back-porting because all changes have to be done in one patch.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

But honestly, this is a nice to have fix. Most people could live happily without it.

From 3c66436d9978030845a96bfaedd6b914536e2ac4 Mon Sep 17 00:00:00

2001

Quoted 9 lines
Quoted 9 lines
Quoted 3 lines

From: Petr Mladek <[email protected]>
Date: Fri, 26 Jun 2026 13:55:41 +0200
Subject: [POC] sys_info: Introduce state-tracking APIs to prevent

duplicate

Quoted 5 lines

backtraces

In watchdog, panic, and hung task detection scenarios, sys_info() can be called multiple times or alongside direct backtrace triggers like trigger_allbutcpu_cpu_backtrace(). This results in identical

backtraces

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

being dumped repeatedly from all CPUs, cluttering the kernel log and delaying or obscuring critical debug details.

Introduce a state tracking bitmask and associated helpers:
- sys_info_done(mask): Marks specific sys_info bits as already

printed.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

- sys_info_reset(): Resets the tracking state.
- sys_info_is_done(mask): Checks if all bits in the mask have been

printed.

Quoted 4 lines

Update sys_info() to automatically filter out already printed bits using this state. Integrate these APIs with the generic hardlockup and softlockup watchdogs, the PowerPC watchdog, the hung task

detector,

Quoted 1 line
Quoted 1 line
Quoted 1 line

and the panic core. This ensures that each piece of system

information

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

and backtrace output is printed at most once per lockup/panic event, and the state is reset cleanly when a lockup does not trigger a

panic.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Races between sys_info() callers are ignored. It should be acceptable because the output from various watchdogs has never been

synchronized.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

And panic() never returns.

Assisted-by gemini-1.5-flash

Yep. There are cases that people want panic on task-hung or sw/hw

lockup,

Quoted 8 lines
Quoted 3 lines

and this could remove much duplication of sys info dump, thanks!

Thanks,

im feeling a new file to do all the force panic jazz, but putting tape on sys_info.c isn't bd either.

I wonder how to move forward with this.

Honestly, I am not sure what exactly you mean by creating another API for tracking the reports so I could not judge it. Feel free to sent some POC.

Quoted 7 lines

Otherwise, I would go with my proposal to remember the printed states by the sys_info API. I am not sure whether I should send a proper patch or you would like to somehow improve it.

Best Regards,
Petr

sup petr, here's my poc

This should make my entire thing make sense

From eb587ed749ff5993c517f29799b369185c5ee7d8 Mon Sep 17 00:00:00 2001
From: Bradley Morgan <[email protected]>
Date: Thu, 2 Jul 2026 18:09:23 +0000
Subject: [POC] sys_info: Introduce incident state-tracking to prevent
duplicate diagnostics

In watchdog, panic, and hung task detection scenarios, sys_info() can be called multiple times or alongside direct debug output functions (like trigger_allbutcpu_cpu_backtrace(), print_modules(), print_irqtrace_events(), and dump_stack()). This leads to identical diagnostics and stack traces being dumped repeatedly, cluttering the kernel log and delaying critical panics.

Introduce a state tracking bitmask and helpers in a new file, lib/sys_info_filter.c:
- sys_info_filter_and_set(mask): Atomically tests which bits in a mask have not yet been printed during the current incident, marks them as printed, and returns that subset.
- sys_info_reset(): Clears the printed mask state.

Add SYS_INFO_MODULES, SYS_INFO_IRQTRACE, and SYS_INFO_STACK flags to include/linux/sys_info.h, and handle them inside sys_info's diagnostic dispatch.

Update the watchdogs, hung task detector, and panic core to call sys_info_filter_and_set() to deduplicate their diagnostic printouts, and sys_info_reset() when a warning incident concludes (e.g., when a stuck CPU recovers, or a new hung task check round begins).

This ensures each piece of system diagnostic is printed at most once per lockup/panic event, preventing console log spam.

Assisted-by Gemini:gemini-3.5-flash
---
 arch/powerpc/kernel/watchdog.c |  21 +++++-
 include/linux/sys_info.h       |   6 ++
 kernel/hung_task.c             |   4 +-
 kernel/panic.c                 |   9 ++-
 kernel/watchdog.c              |  24 ++++++-
 lib/Makefile                   |   2 +-
 lib/sys_info.c                 |  38 ++---------
 lib/sys_info_filter.c          | 120 +++++++++++++++++++++++++++++++++
 8 files changed, 184 insertions(+), 40 deletions(-)
 create mode 100644 lib/sys_info_filter.c
arch/powerpc/kernel/watchdog.c +18 -3
index c40c69368476..31035e28676a 100644
@@ -160,6 +160,10 @@ static void wd_lockup_ipi(struct pt_regs *regs)
 	else
 		dump_stack();
 
+	/* Mark what we already printed so panic() won't repeat it. */
+	sys_info_filter_and_set(SYS_INFO_MODULES | SYS_INFO_IRQTRACE |
+				SYS_INFO_STACK);
+
 	/*
 	 * __wd_nmi_output must be set after we printk from NMI context.
 	 *
@@ -238,7 +242,8 @@ static void watchdog_smp_panic(int cpu)
 
 	if (sysctl_hardlockup_all_cpu_backtrace ||
 	    (hardlockup_si_mask & SYS_INFO_ALL_BT)) {
-		trigger_allbutcpu_cpu_backtrace(cpu);
+		if (sys_info_filter_and_set(SYS_INFO_ALL_BT))
+			trigger_allbutcpu_cpu_backtrace(cpu);
 		cpumask_clear(&wd_smp_cpus_ipi);
 	} else {
 		/*
@@ -254,6 +259,8 @@ static void watchdog_smp_panic(int cpu)
 	sys_info(hardlockup_si_mask & ~SYS_INFO_ALL_BT);
 	if (hardlockup_panic)
 		nmi_panic(NULL, "Hard LOCKUP");
+	else
+		sys_info_reset();
 
 	wd_end_reporting();
 
@@ -416,15 +423,23 @@ DEFINE_INTERRUPT_HANDLER_NMI(soft_nmi_interrupt)
 		print_irqtrace_events(current);
 		show_regs(regs);
 
+		/* Mark what we already printed so panic() won't repeat it. */
+		sys_info_filter_and_set(SYS_INFO_MODULES | SYS_INFO_IRQTRACE |
+					SYS_INFO_STACK);
+
 		xchg(&__wd_nmi_output, 1); // see wd_lockup_ipi
 
 		if (sysctl_hardlockup_all_cpu_backtrace ||
-		    (hardlockup_si_mask & SYS_INFO_ALL_BT))
-			trigger_allbutcpu_cpu_backtrace(cpu);
+		    (hardlockup_si_mask & SYS_INFO_ALL_BT)) {
+			if (sys_info_filter_and_set(SYS_INFO_ALL_BT))
+				trigger_allbutcpu_cpu_backtrace(cpu);
+		}
 
 		sys_info(hardlockup_si_mask & ~SYS_INFO_ALL_BT);
 		if (hardlockup_panic)
 			nmi_panic(regs, "Hard LOCKUP");
+		else
+			sys_info_reset();
 
 		wd_end_reporting();
 	}
include/linux/sys_info.h +6 -0
index a5bc3ea3d44b..f5a1b699143b 100644
@@ -16,8 +16,14 @@
 #define SYS_INFO_PANIC_CONSOLE_REPLAY	0x00000020
 #define SYS_INFO_ALL_BT			0x00000040
 #define SYS_INFO_BLOCKED_TASKS		0x00000080
+#define SYS_INFO_MODULES		0x00000100
+#define SYS_INFO_IRQTRACE		0x00000200
+#define SYS_INFO_STACK			0x00000400
 
 void sys_info(unsigned long si_mask);
+unsigned long sys_info_effective_mask(unsigned long mask);
+unsigned long sys_info_filter_and_set(unsigned long si_mask);
+void sys_info_reset(void);
 unsigned long sys_info_parse_param(char *str);
 
 #ifdef CONFIG_SYSCTL
kernel/hung_task.c +3 -1
index 6fcc94ce4ca9..7a3738279503 100644
@@ -557,8 +557,10 @@ static int watchdog(void *dummy)
 		t = hung_timeout_jiffies(hung_last_checked, interval);
 		if (t <= 0) {
 			if (!atomic_xchg(&reset_hung_task, 0) &&
-			    !hung_detector_suspended)
+			    !hung_detector_suspended) {
+				sys_info_reset();
 				check_hung_uninterruptible_tasks(timeout);
+			}
 			hung_last_checked = jiffies;
 			continue;
 		}
kernel/panic.c +7 -2
index 213725b612aa..94ce7a94f118 100644
@@ -550,8 +550,12 @@ static void panic_trigger_all_cpu_backtrace(void)
  */
 static void panic_other_cpus_shutdown(bool crash_kexec)
 {
-	if (panic_print & SYS_INFO_ALL_BT)
-		panic_trigger_all_cpu_backtrace();
+	unsigned long mask = sys_info_effective_mask(panic_print);
+
+	if (mask & SYS_INFO_ALL_BT) {
+		if (sys_info_filter_and_set(SYS_INFO_ALL_BT))
+			panic_trigger_all_cpu_backtrace();
+	}
 
 	/*
 	 * Note that smp_send_stop() is the usual SMP shutdown function,
@@ -649,6 +653,7 @@ void vpanic(const char *fmt, va_list args)
 		panic_this_cpu_backtrace_printed = true;
 	} else if (IS_ENABLED(CONFIG_DEBUG_BUGVERBOSE)) {
 		dump_stack();
+		sys_info_filter_and_set(SYS_INFO_STACK);
 		panic_this_cpu_backtrace_printed = true;
 	}
 
kernel/watchdog.c +22 -2
index 87dd5e0f6968..3bc6f5fd5380 100644
@@ -280,8 +280,13 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs)
 		trigger_single_cpu_backtrace(cpu);
 	}
 
+	/* Mark what we already printed so panic() won't repeat it. */
+	sys_info_filter_and_set(SYS_INFO_MODULES | SYS_INFO_IRQTRACE |
+				SYS_INFO_STACK);
+
 	if (hardlockup_all_cpu_backtrace) {
-		trigger_allbutcpu_cpu_backtrace(cpu);
+		if (sys_info_filter_and_set(SYS_INFO_ALL_BT))
+			trigger_allbutcpu_cpu_backtrace(cpu);
 		if (!hardlockup_panic)
 			clear_bit_unlock(0, &hard_lockup_nmi_warn);
 	}
@@ -289,6 +294,8 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs)
 	sys_info(hardlockup_si_mask & ~SYS_INFO_ALL_BT);
 	if (hardlockup_panic)
 		nmi_panic(regs, "Hard LOCKUP");
+	else
+		sys_info_reset();
 
 	per_cpu(watchdog_hardlockup_warned, cpu) = true;
 }
@@ -792,6 +799,8 @@ static int softlockup_fn(void *data)
 }
 
 /* watchdog kicker functions */
+static DEFINE_PER_CPU(bool, watchdog_stuck_previously);
+
 static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
 {
 	unsigned long touch_ts, period_ts, now;
@@ -864,6 +873,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
 	touch_ts = __this_cpu_read(watchdog_touch_ts);
 	duration = is_softlockup(touch_ts, period_ts, now);
 	if (unlikely(duration)) {
+		__this_cpu_write(watchdog_stuck_previously, true);
 #ifdef CONFIG_SYSFS
 		++softlockup_count;
 #endif
@@ -893,8 +903,13 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
 			dump_stack();
 		printk_cpu_sync_put_irqrestore(flags);
 
+		/* Mark what we already printed so panic() won't repeat it. */
+		sys_info_filter_and_set(SYS_INFO_MODULES | SYS_INFO_IRQTRACE |
+					SYS_INFO_STACK);
+
 		if (softlockup_all_cpu_backtrace) {
-			trigger_allbutcpu_cpu_backtrace(smp_processor_id());
+			if (sys_info_filter_and_set(SYS_INFO_ALL_BT))
+				trigger_allbutcpu_cpu_backtrace(smp_processor_id());
 			if (!softlockup_panic)
 				clear_bit_unlock(0, &soft_lockup_nmi_warn);
 		}
@@ -905,6 +920,11 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
 
 		if (softlockup_panic && thresh_count >= softlockup_panic)
 			panic("softlockup: hung tasks");
+	} else {
+		if (__this_cpu_read(watchdog_stuck_previously)) {
+			__this_cpu_write(watchdog_stuck_previously, false);
+			sys_info_reset();
+		}
 	}
 
 	return HRTIMER_RESTART;
lib/Makefile +1 -1
index 7f75cc6edf94..521644a140c8 100644
@@ -40,7 +40,7 @@ lib-y := ctype.o string.o vsprintf.o cmdline.o \
 	 is_single_threaded.o plist.o decompress.o kobject_uevent.o \
 	 earlycpio.o seq_buf.o siphash.o dec_and_lock.o \
 	 nmi_backtrace.o win_minmax.o memcat_p.o \
-	 buildid.o objpool.o iomem_copy.o sys_info.o
+	 buildid.o objpool.o iomem_copy.o sys_info.o sys_info_filter.o
 
 lib-$(CONFIG_UNION_FIND) += union_find.o
 lib-$(CONFIG_PRINTK) += dump_stack.o
lib/sys_info.c +7 -31
index f32a06ec9ed4..e188c5d924cb 100644
@@ -2,12 +2,9 @@
 #include <linux/array_size.h>
 #include <linux/bitops.h>
 #include <linux/cleanup.h>
-#include <linux/console.h>
 #include <linux/log2.h>
 #include <linux/kernel.h>
-#include <linux/ftrace.h>
-#include <linux/nmi.h>
-#include <linux/sched/debug.h>
+#include <linux/slab.h>
 #include <linux/string.h>
 #include <linux/sysctl.h>
 
@@ -22,6 +19,9 @@ static const char * const si_names[] = {
 	[ilog2(SYS_INFO_PANIC_CONSOLE_REPLAY)]	= "",
 	[ilog2(SYS_INFO_ALL_BT)]		= "all_bt",
 	[ilog2(SYS_INFO_BLOCKED_TASKS)]		= "blocked_tasks",
+	[ilog2(SYS_INFO_MODULES)]		= "modules",
+	[ilog2(SYS_INFO_IRQTRACE)]		= "irqtrace",
+	[ilog2(SYS_INFO_STACK)]			= "stack",
 };
 
 /*
@@ -29,7 +29,7 @@ static const char * const si_names[] = {
  * If a kernel module calls sys_info() with "parameter == 0", then
  * this mask will be used.
  */
-static unsigned long kernel_si_mask;
+unsigned long kernel_si_mask;
 
 /* Expecting string like "xxx_sys_info=tasks,mem,timers,locks,ftrace,..." */
 unsigned long sys_info_parse_param(char *str)
@@ -136,31 +136,7 @@ static int __init sys_info_sysctl_init(void)
 subsys_initcall(sys_info_sysctl_init);
 #endif
 
-static void __sys_info(unsigned long si_mask)
+unsigned long sys_info_effective_mask(unsigned long mask)
 {
-	if (si_mask & SYS_INFO_TASKS)
-		show_state();
-
-	if (si_mask & SYS_INFO_MEM)
-		show_mem();
-
-	if (si_mask & SYS_INFO_TIMERS)
-		sysrq_timer_list_show();
-
-	if (si_mask & SYS_INFO_LOCKS)
-		debug_show_all_locks();
-
-	if (si_mask & SYS_INFO_FTRACE)
-		ftrace_dump(DUMP_ALL);
-
-	if (si_mask & SYS_INFO_ALL_BT)
-		trigger_all_cpu_backtrace();
-
-	if (si_mask & SYS_INFO_BLOCKED_TASKS)
-		show_state_filter(TASK_UNINTERRUPTIBLE);
-}
-
-void sys_info(unsigned long si_mask)
-{
-	__sys_info(si_mask ? : kernel_si_mask);
+	return mask ? : READ_ONCE(kernel_si_mask);
 }
lib/sys_info_filter.c new file +120 -0

120 lines not on this page · raw message

Signature · 11 lines
2.53.0



NOTE!!: This is AI generated!! This **MAY** not be the finished product,
this is ONLY the model!




Thanks!
#20 lore raw

Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces

Quoted 173 lines · Bradley Morgan
Quoted 15 lines · Petr Mladek
Quoted 6 lines · Bradley Morgan
Quoted 4 lines · Feng Tang
Quoted 3 lines · Petr Mladek
Quoted 2 lines · Petr Mladek
Quoted 1 line · Bradley Morgan

panic_other_cpus_shutdown() handles SYS_INFO_ALL_BT before

stopping

Quoted 2 lines
Quoted 2 lines

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

other CPUs. Do not ask sys_info() to handle that bit again later

in

Quoted 4 lines
Quoted 4 lines

the

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

panic path.

Use sys_info_with_filter() so panic_print=all_bt does not

request

Quoted 46 lines
Quoted 46 lines
Quoted 46 lines

more

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

output after the CPUs are stopped.

Fixes a9af76a78760 ("watchdog: add sys_info sysctls to dump sys

info on system lockup")

Quoted 41 lines
Quoted 41 lines
Quoted 11 lines
---
 kernel/panic.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/panic.c +0 -0
index 213725b612aa..eb842823df61 100644

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 683

Quoted 5 lines
kernel/panic.c +1 -1
@@ -680,7 +680,7 @@ void vpanic(const char *fmt, va_list args)
 	 */
 	atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
 
-	sys_info(panic_print);
+	sys_info_with_filter(panic_print, SYS_INFO_ALL_BT);

Hmm, this prevents printing backtraces from all CPUs completely. But what if they were not printed?

They might be printed by:

static void panic_other_cpus_shutdown(bool crash_kexec)

{
	if (panic_print & SYS_INFO_ALL_BT)
		panic_trigger_all_cpu_backtrace();

[...]
}

But it checks only "panic_print" variable. It won't do anything when (panic_print == 0).

In this case, we might still want to print the backraces when SYS_INFO_ALL_BT is set in kernel_si_info.

kernel/panic.c void vpanic(const char *fmt, va_list args) · line 684

Quoted 1 line
kernel/panic.c +0 -0
 	kmsg_dump_desc(KMSG_DUMP_PANIC, buf);

Of course, we might fix panic_other_cpus_shutdown() to check also kernel_si_info.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

But it all becomes very hairy. We have several levels:

+ watchdog-all_bt-specific option, e.g.

sysctl_hardlockup_all_cpu_backtrace

Quoted 2 lines
Quoted 2 lines

+ watchdog-specific si_info preferences, e.g.

hardlockup_si_mask

Quoted 31 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

+ panic-specific si_info: panic_print

+ universal fallback for any layer: kernel_si_info

Now, we try to check all these variables back and forth to trigger all backtraces or to avoid triggering them. And it clearly does not work well and the code is more and more hairy.

I think about another approach. The word "waterfall" comes to my

mind.

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 9 lines

Instead of checking all the settings back and forth, let's process each setting one by one and just remember what has been done and skip this in the next level.

All the si_info actions seems to dump a global system state. So, it would make sense to remember the state in a global variable even when it might be modified by more CPUs in parallel.

I am going to think more about it.

I have created a POC using Gemini. I haven't tested it. But it looks acceptable. And the logic seems to be more straightforward.

One drawback is that it requires adding the _reset() call for all sys_info() callers. It is fine in principle but it might complicate back-porting because all changes have to be done in one patch.

Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

But honestly, this is a nice to have fix. Most people could live happily without it.

From 3c66436d9978030845a96bfaedd6b914536e2ac4 Mon Sep 17 00:00:00

2001

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

From: Petr Mladek <[email protected]>
Date: Fri, 26 Jun 2026 13:55:41 +0200
Subject: [POC] sys_info: Introduce state-tracking APIs to prevent

duplicate

Quoted 3 lines

backtraces

In watchdog, panic, and hung task detection scenarios, sys_info()

can

Quoted 16 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

be called multiple times or alongside direct backtrace triggers like trigger_allbutcpu_cpu_backtrace(). This results in identical

backtraces

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

being dumped repeatedly from all CPUs, cluttering the kernel log and delaying or obscuring critical debug details.

Introduce a state tracking bitmask and associated helpers:
- sys_info_done(mask): Marks specific sys_info bits as already

printed.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

- sys_info_reset(): Resets the tracking state.
- sys_info_is_done(mask): Checks if all bits in the mask have been

printed.

Quoted 4 lines

Update sys_info() to automatically filter out already printed bits using this state. Integrate these APIs with the generic hardlockup and softlockup watchdogs, the PowerPC watchdog, the hung task

Quoted 8 lines

detector,

Quoted 1 line
Quoted 1 line
Quoted 1 line

and the panic core. This ensures that each piece of system

information

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

and backtrace output is printed at most once per lockup/panic event, and the state is reset cleanly when a lockup does not trigger a

panic.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Races between sys_info() callers are ignored. It should be

acceptable

Quoted 24 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

because the output from various watchdogs has never been

synchronized.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

And panic() never returns.

Assisted-by gemini-1.5-flash

Yep. There are cases that people want panic on task-hung or sw/hw

lockup,

Quoted 8 lines
Quoted 3 lines

and this could remove much duplication of sys info dump, thanks!

Thanks,

im feeling a new file to do all the force panic jazz, but putting tape on sys_info.c isn't bd either.

I wonder how to move forward with this.

Honestly, I am not sure what exactly you mean by creating another API for tracking the reports so I could not judge it. Feel free to sent some POC.

Quoted 7 lines

Otherwise, I would go with my proposal to remember the printed states by the sys_info API. I am not sure whether I should send a proper patch or you would like to somehow improve it.

Best Regards,
Petr

Quoted 4 lines

sup petr, here's my poc

This should make my entire thing make sense

Actually, looks like churn to me.

we shouldn't do that.

It is on that gist though.

As in, I'm thinking about doing a new API to perhaps, make sys_info better?

But it's very complicated, and may need discussion, feel free to ignore that POC...

crappy old me, need a coffee heh.

Thanks!

reviewer
#21 lore raw

Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces

In reply to Bradley Morgan
Quoted 41 lines · Bradley Morgan
Quoted 13 lines · Petr Mladek
Quoted 8 lines · Bradley Morgan
Quoted 6 lines · Feng Tang
Quoted 5 lines · Petr Mladek
Quoted 1 line · Petr Mladek

On Thu 2026-06-25 15:25:58, Bradley Morgan wrote:

In watchdog, panic, and hung task detection scenarios, sys_info() can be called multiple times or alongside direct backtrace triggers like trigger_allbutcpu_cpu_backtrace(). This results in identical

backtraces

Quoted 3 lines
Quoted 2 lines
Quoted 2 lines

being dumped repeatedly from all CPUs, cluttering the kernel log and delaying or obscuring critical debug details.

Quoted 8 lines
Quoted 2 lines

im feeling a new file to do all the force panic jazz, but putting tape on sys_info.c isn't bd either.

I wonder how to move forward with this.

Honestly, I am not sure what exactly you mean by creating another API for tracking the reports so I could not judge it. Feel free to sent some POC.

sup petr, here's my poc

This should make my entire thing make sense

Quoted 1 line

From eb587ed749ff5993c517f29799b369185c5ee7d8 Mon Sep 17 00:00:00 2001

From: Bradley Morgan <[email protected]>
Date: Thu, 2 Jul 2026 18:09:23 +0000
Subject: [POC] sys_info: Introduce incident state-tracking to prevent
duplicate diagnostics

In watchdog, panic, and hung task detection scenarios, sys_info() can be called multiple times or alongside direct debug output functions (like trigger_allbutcpu_cpu_backtrace(), print_modules(), print_irqtrace_events(), and dump_stack()). This leads to identical diagnostics and stack traces being dumped repeatedly, cluttering the kernel log and delaying critical panics.

Quoted 2 lines

Introduce a state tracking bitmask and helpers in a new file, lib/sys_info_filter.c:

New file suggests that it would implement an API using sys_info_filter() prefix.

Quoted 3 lines

- sys_info_filter_and_set(mask): Atomically tests which bits in a mask have not yet been printed during the current incident, marks them as printed, and returns that subset.

The name of the funtion is a kind of puzzle. I think that we could do a better job.

Quoted 1 line

- sys_info_reset(): Clears the printed mask state.

This function has sys_info* prefix. It would expect it in sys_info.c

Quoted 3 lines

Add SYS_INFO_MODULES, SYS_INFO_IRQTRACE, and SYS_INFO_STACK flags to include/linux/sys_info.h, and handle them inside sys_info's diagnostic dispatch.

I though about adding an information that we printed backtrace for this CPU as well. But it not trivial. Different API shows different extra info, like modules, IRQ backtrace, registers, code. I would leave this complexity aside for now.

Quoted 19 lines

Update the watchdogs, hung task detector, and panic core to call sys_info_filter_and_set() to deduplicate their diagnostic printouts, and sys_info_reset() when a warning incident concludes (e.g., when a stuck CPU recovers, or a new hung task check round begins).

This ensures each piece of system diagnostic is printed at most once per lockup/panic event, preventing console log spam.

Assisted-by Gemini:gemini-3.5-flash
lib/sys_info_filter.c new file +5 -0
@@ -0,0 +1,120 @@
+static unsigned long sys_info_printed;
+
+unsigned long sys_info_filter_and_set(unsigned long si_mask)
+{
+	unsigned long old, new;

lib/sys_info_filter.c line 15

Quoted 10 lines
lib/sys_info_filter.c new file +10 -0
+
+	if (!si_mask)
+		return 0;
+
+	do {
+		old = READ_ONCE(sys_info_printed);
+		if (!(si_mask & ~old))
+			return 0;
+		new = old | si_mask;
+	} while (cmpxchg(&sys_info_printed, old, new) != old);

It is a good question whether to update the info using atomic operations. One problem is that the mask is "unsigned long". I am not sure if it natively atomic on all architectures. 32-bit architecures use extra locking when implementing atomic operations with 64-bit values. And we should rather avoid any locking in this code.

Well, long seems to be 32-bit on 32-bit x86 so it might be safe after all.

lib/sys_info_filter.c line 113

Quoted 3 lines
lib/sys_info_filter.c new file +3 -0
+void sys_info_reset(void)
+static void __sys_info(unsigned long si_mask)
+void sys_info(unsigned long si_mask)

I wonder why this sys_info*() API implementation has been moved from sys_info.c to sys_info_filter.c.

I am sorry but I do not see any advantage in adding the new file sys_info_filter.c

Quoted 2 lines

NOTE!!: This is AI generated!! This **MAY** not be the finished product, this is ONLY the model!

IMHO, Gemini did pretty bad job in this case. Please, try to review the AI generated before you send it. And send it only when you think that it is reasonable enough. :-)

It is even fine to send "crap" but you should start the mail with a warning that you send it just give us an idea what you had it mind. And you should explain why you actually do not like.

Best Regards,
Petr

#22 lore raw

Re: [PATCH v3 4/4] panic: use sys_info_with_filter() to avoid duplicate backtraces

Quoted 129 lines · Petr Mladek
Quoted 9 lines · Bradley Morgan
Quoted 7 lines · Petr Mladek
Quoted 6 lines · Bradley Morgan
Quoted 4 lines · Feng Tang
Quoted 3 lines · Petr Mladek
Quoted 1 line · Petr Mladek

On Thu 2026-06-25 15:25:58, Bradley Morgan wrote:

In watchdog, panic, and hung task detection scenarios, sys_info()

can

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

be called multiple times or alongside direct backtrace triggers

like

Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

trigger_allbutcpu_cpu_backtrace(). This results in identical

backtraces

Quoted 1 line
Quoted 1 line
Quoted 1 line

being dumped repeatedly from all CPUs, cluttering the kernel log

and

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line

delaying or obscuring critical debug details.

im feeling a new file to do all the force panic jazz, but putting

tape

Quoted 25 lines
Quoted 7 lines
Quoted 1 line

on sys_info.c isn't bd either.

I wonder how to move forward with this.

Honestly, I am not sure what exactly you mean by creating another API for tracking the reports so I could not judge it. Feel free to sent some POC.

sup petr, here's my poc

This should make my entire thing make sense

Quoted 1 line

From eb587ed749ff5993c517f29799b369185c5ee7d8 Mon Sep 17 00:00:00 2001

From: Bradley Morgan <[email protected]>
Date: Thu, 2 Jul 2026 18:09:23 +0000
Subject: [POC] sys_info: Introduce incident state-tracking to prevent
duplicate diagnostics

In watchdog, panic, and hung task detection scenarios, sys_info() can be called multiple times or alongside direct debug output functions (like trigger_allbutcpu_cpu_backtrace(), print_modules(), print_irqtrace_events(), and dump_stack()). This leads to identical diagnostics and stack traces being dumped repeatedly, cluttering the kernel log and delaying critical panics.

Quoted 2 lines

Introduce a state tracking bitmask and helpers in a new file, lib/sys_info_filter.c:

New file suggests that it would implement an API using sys_info_filter() prefix.

Quoted 3 lines

- sys_info_filter_and_set(mask): Atomically tests which bits in a mask have not yet been printed during the current incident, marks them as printed, and returns that subset.

The name of the funtion is a kind of puzzle. I think that we could do a better job.

Quoted 1 line

- sys_info_reset(): Clears the printed mask state.

This function has sys_info* prefix. It would expect it in sys_info.c

Quoted 3 lines

Add SYS_INFO_MODULES, SYS_INFO_IRQTRACE, and SYS_INFO_STACK flags to include/linux/sys_info.h, and handle them inside sys_info's diagnostic dispatch.

I though about adding an information that we printed backtrace for this CPU as well. But it not trivial. Different API shows different extra info, like modules, IRQ backtrace, registers, code. I would leave this complexity aside for now.

Quoted 19 lines

Update the watchdogs, hung task detector, and panic core to call sys_info_filter_and_set() to deduplicate their diagnostic printouts, and sys_info_reset() when a warning incident concludes (e.g., when a stuck CPU recovers, or a new hung task check round begins).

This ensures each piece of system diagnostic is printed at most once per lockup/panic event, preventing console log spam.

Assisted-by Gemini:gemini-3.5-flash
lib/sys_info_filter.c new file +5 -0
@@ -0,0 +1,120 @@
+static unsigned long sys_info_printed;
+
+unsigned long sys_info_filter_and_set(unsigned long si_mask)
+{
+	unsigned long old, new;

lib/sys_info_filter.c line 15

Quoted 10 lines
lib/sys_info_filter.c new file +10 -0
+
+	if (!si_mask)
+		return 0;
+
+	do {
+		old = READ_ONCE(sys_info_printed);
+		if (!(si_mask & ~old))
+			return 0;
+		new = old | si_mask;
+	} while (cmpxchg(&sys_info_printed, old, new) != old);

It is a good question whether to update the info using atomic operations. One problem is that the mask is "unsigned long". I am not sure if it natively atomic on all architectures. 32-bit architecures use extra locking when implementing atomic operations with 64-bit values. And we should rather avoid any locking in this code.

Well, long seems to be 32-bit on 32-bit x86 so it might be safe after all.

lib/sys_info_filter.c line 113

Quoted 3 lines
lib/sys_info_filter.c new file +3 -0
+void sys_info_reset(void)
+static void __sys_info(unsigned long si_mask)
+void sys_info(unsigned long si_mask)

I wonder why this sys_info*() API implementation has been moved from sys_info.c to sys_info_filter.c.

I am sorry but I do not see any advantage in adding the new file sys_info_filter.c

Quoted 2 lines

NOTE!!: This is AI generated!! This **MAY** not be the finished product, this is ONLY the model!

IMHO, Gemini did pretty bad job in this case. Please, try to review the AI generated before you send it. And send it only when you think that it is reasonable enough. :-)

Quoted 7 lines

It is even fine to send "crap" but you should start the mail with a warning that you send it just give us an idea what you had it mind. And you should explain why you actually do not like.

Best Regards,
Petr

for now, I'll go with your approach, I'll split up and submit your patch(es) in the coming days.

Because the whole new file idea is super complicated and requires a load of discussion before a model could be completed.

One of my ideas is to just kill sys_info. and/or make it better.

Other ideas I need to think.

Thanks a lot for reviewing my model though.

Thanks!