sched/debug: Reset watchdog on all CPUs while processing sysrq-t
PATCH 1/1 sched/debug: Reset watchdog on all CPUs while processing sysrq-t
Lengthy output of sysrq-t may take a lot of time on slow serial console with lots of processes and CPUs.
So we need to reset NMI-watchdog to avoid spurious lockup messages, and we also reset softlockup watchdogs on all other CPUs since another CPU might be blocked waiting for us to process an IPI or stop_machine.
Add to sysrq_sched_debug_show() as what we did in show_state_filter().
--- kernel/sched/debug.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/sched/debug.c +9 -2
@@ -751,9 +751,16 @@ void sysrq_sched_debug_show(void)int cpu;sched_debug_header (NULL ); -for_each_online_cpu (cpu) +for_each_online_cpu (cpu) { +/* +* Need to reset softlockup watchdogs on all CPUs, because +* another CPU might be blocked waiting for us to process +* an IPI or stop_machine. +*/ +touch_nmi_watchdog (); +touch_all_softlockup_watchdogs ();print_cpu (NULL , cpu); - + } }/*
Signature · 1 line
2.17.1
Quoted 6 lines · On Thu, 26 Dec 2019 16:52:24 +0800 Wei Li <[email protected]> wrote:
Lengthy output of sysrq-t may take a lot of time on slow serial console with lots of processes and CPUs.
So we need to reset NMI-watchdog to avoid spurious lockup messages, and we also reset softlockup watchdogs on all other CPUs since another CPU might be blocked waiting for us to process an IPI or
stop_machine.
Have you had this triggered?
Quoted 15 lines
Add to
sysrq_sched_debug_show()as what we did inshow_state_filter().Signed-off-by Wei Li <[email protected]>--- kernel/sched/debug.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)kernel/sched/debug.c +0
-0@@ -751,9 +751,16 @@ void sysrq_sched_debug_show(void)int cpu;
kernel/sched/debug.c void sysrq_sched_debug_show(void) · line 761
Quoted 10 lines
kernel/sched/debug.c +8
-1sched_debug_header (NULL ); -for_each_online_cpu (cpu) +for_each_online_cpu (cpu) { +/* +* Need to reset softlockup watchdogs on all CPUs, because +* another CPU might be blocked waiting for us to process +* an IPI or stop_machine. +*/ +touch_nmi_watchdog (); +touch_all_softlockup_watchdogs ();
This doesn't seem to hurt to add, thus.
-- Steve
Quoted 6 lines
kernel/sched/debug.c +1
-1print_cpu (NULL , cpu); - + } }/*
Hi Steven,
Yes, it can be triggered on the Hi1620 system (128 cores) as follows:
stress-ng -c 50 &
stress-ng -m 50 &
stress-ng -i 20 &
echo 7 > /proc/sys/kernel/printk
echo t > /proc/sysrq-trigger
Then a soft lockup will be reported at migration thread
[39636.303531] watchdog: BUG: soft lockup - CPU#67 stuck for 23s! [migration/67:348]
which is waiting for the CPU handling sysrq-t to process stop_two_cpus.
Thanks,
Wei
Quoted 54 lines · Steven Rostedt
Quoted 6 lines · On Thu, 26 Dec 2019 16:52:24 +0800 Wei Li <[email protected]> wrote:
Lengthy output of sysrq-t may take a lot of time on slow serial console with lots of processes and CPUs.
So we need to reset NMI-watchdog to avoid spurious lockup messages, and we also reset softlockup watchdogs on all other CPUs since another CPU might be blocked waiting for us to process an IPI or
stop_machine.Have you had this triggered?
Quoted 15 lines
Add to
sysrq_sched_debug_show()as what we did inshow_state_filter().Signed-off-by Wei Li <[email protected]>--- kernel/sched/debug.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)kernel/sched/debug.c +0
-0@@ -751,9 +751,16 @@ void sysrq_sched_debug_show(void)int cpu;kernel/sched/debug.c void sysrq_sched_debug_show(void) · line 761
Quoted 10 lines
kernel/sched/debug.c +8
-1sched_debug_header (NULL ); -for_each_online_cpu (cpu) +for_each_online_cpu (cpu) { +/* +* Need to reset softlockup watchdogs on all CPUs, because +* another CPU might be blocked waiting for us to process +* an IPI or stop_machine. +*/ +touch_nmi_watchdog (); +touch_all_softlockup_watchdogs ();This doesn't seem to hurt to add, thus.
Reviewed-by Steven Rostedt (VMware) <[email protected]>-- Steve
kernel/sched/debug.c void sysrq_sched_debug_show(void) · line 766
Quoted 6 lines
kernel/sched/debug.c +1
-1print_cpu (NULL , cpu); - + } }/* .
Peter Zijlstra
maintainerQuoted 40 lines · Steven Rostedt
Quoted 6 lines · On Thu, 26 Dec 2019 16:52:24 +0800 Wei Li <[email protected]> wrote:
Lengthy output of sysrq-t may take a lot of time on slow serial console with lots of processes and CPUs.
So we need to reset NMI-watchdog to avoid spurious lockup messages, and we also reset softlockup watchdogs on all other CPUs since another CPU might be blocked waiting for us to process an IPI or
stop_machine.Have you had this triggered?
Quoted 15 lines
Add to
sysrq_sched_debug_show()as what we did inshow_state_filter().Signed-off-by Wei Li <[email protected]>--- kernel/sched/debug.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)kernel/sched/debug.c +0
-0@@ -751,9 +751,16 @@ void sysrq_sched_debug_show(void)int cpu;kernel/sched/debug.c void sysrq_sched_debug_show(void) · line 761
Quoted 10 lines
kernel/sched/debug.c +8
-1sched_debug_header (NULL ); -for_each_online_cpu (cpu) +for_each_online_cpu (cpu) { +/* +* Need to reset softlockup watchdogs on all CPUs, because +* another CPU might be blocked waiting for us to process +* an IPI or stop_machine. +*/ +touch_nmi_watchdog (); +touch_all_softlockup_watchdogs ();This doesn't seem to hurt to add, thus.
Quoted 1 line
Reviewed-by Steven Rostedt (VMware) <[email protected]>
Thanks!
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 02d4ac5885a18d326b500b94808f0956dcce2832 Gitweb: https://git.kernel.org/tip/02d4ac5885a18d326b500b94808f0956dcce2832 Author: Wei Li <[email protected]> AuthorDate: Thu, 26 Dec 2019 16:52:24 +08:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 17 Jan 2020 10:19:20 +01:00
sched/debug: Reset watchdog on all CPUs while processing sysrq-t
Lengthy output of sysrq-t may take a lot of time on slow serial console with lots of processes and CPUs.
So we need to reset NMI-watchdog to avoid spurious lockup messages, and we also reset softlockup watchdogs on all other CPUs since another CPU might be blocked waiting for us to process an IPI or stop_machine.
Add to sysrq_sched_debug_show() as what we did in show_state_filter().
--- kernel/sched/debug.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/sched/debug.c +9 -2
@@ -751,9 +751,16 @@ void sysrq_sched_debug_show(void)int cpu;sched_debug_header (NULL ); -for_each_online_cpu (cpu) +for_each_online_cpu (cpu) { +/* +* Need to reset softlockup watchdogs on all CPUs, because +* another CPU might be blocked waiting for us to process +* an IPI or stop_machine. +*/ +touch_nmi_watchdog (); +touch_all_softlockup_watchdogs ();print_cpu (NULL , cpu); - + } }/*