Skip to contentSkip to conversation

sched/debug: Reset watchdog on all CPUs while processing sysrq-t

PATCH Applied Wei Li 4 participants 5 messages Dec 26, 2019 – Jan 17, 2020 Scheduler

lore
Share & tools
lore

Wei Li

author
#0 lore raw

PATCH 1/1 sched/debug: Reset watchdog on all CPUs while processing sysrq-t

Lengthy output of sysrq-t may take a lot of time on slow serial console with lots of processes and CPUs.

So we need to reset NMI-watchdog to avoid spurious lockup messages, and we also reset softlockup watchdogs on all other CPUs since another CPU might be blocked waiting for us to process an IPI or stop_machine.

Add to sysrq_sched_debug_show() as what we did in show_state_filter().

---
 kernel/sched/debug.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/sched/debug.c +9 -2
index f7e4579e746c..879d3ccf3806 100644
@@ -751,9 +751,16 @@ void sysrq_sched_debug_show(void)
 	int cpu;
 
 	sched_debug_header(NULL);
-	for_each_online_cpu(cpu)
+	for_each_online_cpu(cpu) {
+		/*
+		 * Need to reset softlockup watchdogs on all CPUs, because
+		 * another CPU might be blocked waiting for us to process
+		 * an IPI or stop_machine.
+		 */
+		touch_nmi_watchdog();
+		touch_all_softlockup_watchdogs();
 		print_cpu(NULL, cpu);
-
+	}
 }
 
 /*
Signature · 1 line
2.17.1
reviewer Reviewed-by on v1 1/1
#1 lore raw
Quoted 6 lines · On Thu, 26 Dec 2019 16:52:24 +0800 Wei Li <[email protected]> wrote:

Lengthy output of sysrq-t may take a lot of time on slow serial console with lots of processes and CPUs.

So we need to reset NMI-watchdog to avoid spurious lockup messages, and we also reset softlockup watchdogs on all other CPUs since another CPU might be blocked waiting for us to process an IPI or stop_machine.

Have you had this triggered?

Quoted 15 lines

Add to sysrq_sched_debug_show() as what we did in show_state_filter().

---
 kernel/sched/debug.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/sched/debug.c +0 -0
index f7e4579e746c..879d3ccf3806 100644
@@ -751,9 +751,16 @@ void sysrq_sched_debug_show(void)
 	int cpu;
 

kernel/sched/debug.c void sysrq_sched_debug_show(void) · line 761

Quoted 10 lines
kernel/sched/debug.c +8 -1
 	sched_debug_header(NULL);
-	for_each_online_cpu(cpu)
+	for_each_online_cpu(cpu) {
+		/*
+		 * Need to reset softlockup watchdogs on all CPUs, because
+		 * another CPU might be blocked waiting for us to process
+		 * an IPI or stop_machine.
+		 */
+		touch_nmi_watchdog();
+		touch_all_softlockup_watchdogs();

This doesn't seem to hurt to add, thus.

-- Steve

Quoted 6 lines
kernel/sched/debug.c +1 -1
 		print_cpu(NULL, cpu);
-
+	}
 }
 
 /*
author
#2 lore raw

Hi Steven,
Yes, it can be triggered on the Hi1620 system (128 cores) as follows:
stress-ng -c 50 &
stress-ng -m 50 &
stress-ng -i 20 &
echo 7 > /proc/sys/kernel/printk
echo t > /proc/sysrq-trigger

Then a soft lockup will be reported at migration thread

[39636.303531] watchdog: BUG: soft lockup - CPU#67 stuck for 23s! [migration/67:348]

which is waiting for the CPU handling sysrq-t to process stop_two_cpus.

Thanks,
Wei

Quoted 54 lines · Steven Rostedt
Quoted 6 lines · On Thu, 26 Dec 2019 16:52:24 +0800 Wei Li <[email protected]> wrote:

Lengthy output of sysrq-t may take a lot of time on slow serial console with lots of processes and CPUs.

So we need to reset NMI-watchdog to avoid spurious lockup messages, and we also reset softlockup watchdogs on all other CPUs since another CPU might be blocked waiting for us to process an IPI or stop_machine.

Have you had this triggered?

Quoted 15 lines

Add to sysrq_sched_debug_show() as what we did in show_state_filter().

---
 kernel/sched/debug.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/sched/debug.c +0 -0
index f7e4579e746c..879d3ccf3806 100644
@@ -751,9 +751,16 @@ void sysrq_sched_debug_show(void)
 	int cpu;
 

kernel/sched/debug.c void sysrq_sched_debug_show(void) · line 761

Quoted 10 lines
kernel/sched/debug.c +8 -1
 	sched_debug_header(NULL);
-	for_each_online_cpu(cpu)
+	for_each_online_cpu(cpu) {
+		/*
+		 * Need to reset softlockup watchdogs on all CPUs, because
+		 * another CPU might be blocked waiting for us to process
+		 * an IPI or stop_machine.
+		 */
+		touch_nmi_watchdog();
+		touch_all_softlockup_watchdogs();

This doesn't seem to hurt to add, thus.

-- Steve

kernel/sched/debug.c void sysrq_sched_debug_show(void) · line 766

Quoted 6 lines
kernel/sched/debug.c +1 -1
 		print_cpu(NULL, cpu);
-
+	}
 }
 
 /*

.

maintainer
#3 lore raw
In reply to Steven Rostedt
Quoted 40 lines · Steven Rostedt
Quoted 6 lines · On Thu, 26 Dec 2019 16:52:24 +0800 Wei Li <[email protected]> wrote:

Lengthy output of sysrq-t may take a lot of time on slow serial console with lots of processes and CPUs.

So we need to reset NMI-watchdog to avoid spurious lockup messages, and we also reset softlockup watchdogs on all other CPUs since another CPU might be blocked waiting for us to process an IPI or stop_machine.

Have you had this triggered?

Quoted 15 lines

Add to sysrq_sched_debug_show() as what we did in show_state_filter().

---
 kernel/sched/debug.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/sched/debug.c +0 -0
index f7e4579e746c..879d3ccf3806 100644
@@ -751,9 +751,16 @@ void sysrq_sched_debug_show(void)
 	int cpu;
 

kernel/sched/debug.c void sysrq_sched_debug_show(void) · line 761

Quoted 10 lines
kernel/sched/debug.c +8 -1
 	sched_debug_header(NULL);
-	for_each_online_cpu(cpu)
+	for_each_online_cpu(cpu) {
+		/*
+		 * Need to reset softlockup watchdogs on all CPUs, because
+		 * another CPU might be blocked waiting for us to process
+		 * an IPI or stop_machine.
+		 */
+		touch_nmi_watchdog();
+		touch_all_softlockup_watchdogs();

This doesn't seem to hurt to add, thus.

Quoted 1 line

Thanks!

merge bot Applied
#4 lore raw
In reply to Wei Li
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     02d4ac5885a18d326b500b94808f0956dcce2832
Gitweb:        https://git.kernel.org/tip/02d4ac5885a18d326b500b94808f0956dcce2832
Author:        Wei Li <[email protected]>
AuthorDate:    Thu, 26 Dec 2019 16:52:24 +08:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Fri, 17 Jan 2020 10:19:20 +01:00

sched/debug: Reset watchdog on all CPUs while processing sysrq-t

Lengthy output of sysrq-t may take a lot of time on slow serial console with lots of processes and CPUs.

So we need to reset NMI-watchdog to avoid spurious lockup messages, and we also reset softlockup watchdogs on all other CPUs since another CPU might be blocked waiting for us to process an IPI or stop_machine.

Add to sysrq_sched_debug_show() as what we did in show_state_filter().

---
 kernel/sched/debug.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/sched/debug.c +9 -2
index f7e4579..879d3cc 100644
@@ -751,9 +751,16 @@ void sysrq_sched_debug_show(void)
 	int cpu;
 
 	sched_debug_header(NULL);
-	for_each_online_cpu(cpu)
+	for_each_online_cpu(cpu) {
+		/*
+		 * Need to reset softlockup watchdogs on all CPUs, because
+		 * another CPU might be blocked waiting for us to process
+		 * an IPI or stop_machine.
+		 */
+		touch_nmi_watchdog();
+		touch_all_softlockup_watchdogs();
 		print_cpu(NULL, cpu);
-
+	}
 }
 
 /*