softlockup: fix problem with long kernel pauses from kgdb
PATCH 1/1 softlockup: fix problem with long kernel pauses from kgdb
Ingo,
Given that you are the maintainer of kernel/softlockup.c, I am seeking advice as to how to properly fix this problem.
The short version of the problem is:
* CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
* Attach to kgdb after boot
* Wait for 90 seconds
* Execute a continue in gdb
* You receive a warning about softlockup
The patch that follows is a lengthy analysis on the issue, but the question here is what is the right way to fix this?
It seems that a provision is required in order to get the clock synced up prior to touching the watch dog. It was not clear that it was a good idea to unconditionally call the sched_clock_tick() from the softlockup touch code.
Your input on this issue is greatly appreciated.
Thanks,
Jason.
----------
From: "Dongdong Deng" <[email protected]>
Subject: [PATCH] softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume
When CONFIG_HAVE_UNSTABLE_SCHED_CLOCK is set and the sched_clock() was gets the time from hardware, such as from TSC, kgdb often causes softlock warning messages on resuming or detaching from a debug session.
Sequence of events in the problem case:
1) "cpu sched clock" and "hardware time" are at 100 seconds prior to a call to kgdb_handle_exception()
2) Debugger waits in kgdb_handle_exception() for 80 seconds and on exit the following is called
...
touch_softlockup_watchdog()
--> __raw_get_cpu_var(touch_timestamp) = 0;3) "cpu sched clock" = 100s (it didn't be updated, because the interrupt was disabled in kgdb) but the "hardware time" = 180s
4) The first timer interrupt after resuming from kgdb_handle_exception updates the watchdog from the "cpu sched clock"
update_process_times()
{
...
run_local_timers()
--> softlockup_tick()
--> check (touch_timestamp == 0)
(it is "YES" here, we have set "touch_timestamp = 0"
at kgdb)
--> __touch_softlockup_watchdog()
***(A)--> reset "touch_timestamp" to "get_timestamp()"
(Here, the "touch_timestamp" will still be set to
100s.)
...
scheduler_tick()
***(B)--> sched_clock_tick()
(update "cpu sched clock" to "hardware time" = 180s)
...
}5) The Second timer interrupt handler appears to have a large jump and trips the softlockup warning.
update_process_times()
{
...
run_local_timers()
--> softlockup_tick()
--> "cpu sched clock" - "touch_timestamp" = 180s-100s > 60s
--> printk "soft lockup error messages"
...
}note:
***(A) reset "touch_timestamp" to "get_timestamp(this_cpu)"
Why "touch_timestamp" will be 100s, not 180s ?
We enable "CONFIG_HAVE_UNSTABLE_SCHED_CLOCK",
so the call trace of get_timestamp() is:
get_timestamp(this_cpu)
-->cpu_clock(this_cpu)
-->sched_clock_cpu(this_cpu)
-->__update_sched_clock(sched_clock_data, now)22 static u64 __update_sched_clock(struct sched_clock_data *scd, u64 = now)
23 {
24 s64 delta = now - scd->tick_raw;
25 u64 clock, min_clock, max_clock;
26
27 WARN_ON_ONCE(!irqs_disabled());
28
29 if (unlikely(delta < 0))
30 delta = 0;
31
32 clock = scd->tick_gtod + delta;
33
34 min_clock = wrap_max(scd->tick_gtod, scd->clock);
35 max_clock = wrap_max(scd->clock, scd->tick_gtod + TICK_NSEC);
36
37 clock = wrap_max(clock, min_clock);
38 clock = wrap_min(clock, max_clock);
39
40 scd->clock = clock;
41
42 return scd->clock;
43 }
Please pay attention to __update_sched_clock() function, it uses the GTOD tick value to create a window to filter crazy "now" values. So if "now" values is too big for sched_clock_data, it will be ignored.
as the data of "step 3)",
"now" = sched_clock() = "hardware time" = 180s and sched_clock_data = 100s.
180s is too big for 100s, it will be ignored.
That's why the touch_timestamp will be set to 100s, not 180s.
The fix is to simply invoke sched_clock_tick() to update "cpu sched clock" on exit from kgdb_handle_exception.
--- kernel/softlockup.c | 3 +++ 1 file changed, 3 insertions(+)
kernel/softlockup.c +3 -0
@@ -118,6 +118,9 @@ void softlockup_tick(void) }if (touch_timestamp ==0 ) { +/* If the time stamp was touched externally make sure the +* scheduler tick is up to date as well */ +sched_clock_tick ();__touch_softlockup_watchdog ();return ; }
Peter Zijlstra
maintainerQuoted 3 lines · Jason Wessel
The fix is to simply invoke
sched_clock_tick()to update "cpu sched clock" on exit fromkgdb_handle_exception.
Is that a regular IRQ context, or is that NMI context?
Quoted 12 lines
Signed-off-by Dongdong Deng <[email protected]>Signed-off-by Jason Wessel <[email protected]>Cc Ingo Molnar <[email protected]>--- kernel/softlockup.c | 3 +++ 1 file changed, 3 insertions(+)
kernel/softlockup.c void softlockup_tick(void) · line 126
Quoted 9 lines
kernel/softlockup.c +3
-0@@ -118,6 +118,9 @@ void softlockup_tick(void) }if (touch_timestamp ==0 ) { +/* If the time stamp was touched externally make sure the +* scheduler tick is up to date as well */ +sched_clock_tick ();__touch_softlockup_watchdog ();return ; }
Aside from the funny comment style (please fix) the fix does look sensible.
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Jason Wessel
The fix is to simply invoke
sched_clock_tick()to update "cpu sched clock" on exit fromkgdb_handle_exception.Is that a regular IRQ context, or is that NMI context?
The patch text there is incorrect (which I can fix), because there were 3 other versions of this particular patch. The version of this patch that tried to use sched_clock_tick() from the exception context hung the system because there was lock contention for CPUs that were not in the running state.
In this version, the sched_clock_tick() added in the softlockup_tick(), which did not result in a hard hang. I am pretty sure that softlockup_tick() runs as a part of the timer interrupt processing from run_local_timers(), so there was no further need to turn off the interrupts.
Jason.
Quoted 13 lines
Quoted 13 lines
Signed-off-by Dongdong Deng <[email protected]>Signed-off-by Jason Wessel <[email protected]>Cc Ingo Molnar <[email protected]>--- kernel/softlockup.c | 3 +++ 1 file changed, 3 insertions(+)kernel/softlockup.c +0
-0@@ -118,6 +118,9 @@ void softlockup_tick(void) }
Quoted 13 lines
kernel/softlockup.c void softlockup_tick(void) · line 126
Quoted 9 lines
kernel/softlockup.c +3
-0if (touch_timestamp ==0 ) { +/* If the time stamp was touched externally make sure the +* scheduler tick is up to date as well */ +sched_clock_tick ();__touch_softlockup_watchdog ();return ; }Aside from the funny comment style (please fix) the fix does look sensible.
No problem.
Thanks,
Jason.
Quoted 19 lines · Peter Zijlstra
Quoted 2 lines · Jason Wessel
The fix is to simply invoke
sched_clock_tick()to update "cpu sched clock" on exit fromkgdb_handle_exception.Is that a regular IRQ context, or is that NMI context?
Quoted 13 lines
Signed-off-by Dongdong Deng <[email protected]>Signed-off-by Jason Wessel <[email protected]>Cc Ingo Molnar <[email protected]>--- kernel/softlockup.c | 3 +++ 1 file changed, 3 insertions(+)kernel/softlockup.c +0
-0@@ -118,6 +118,9 @@ void softlockup_tick(void) }
Quoted 11 lines
kernel/softlockup.c void softlockup_tick(void) · line 126
Quoted 8 lines
kernel/softlockup.c +3
-0if (touch_timestamp ==0 ) { +/* If the time stamp was touched externally make sure the +* scheduler tick is up to date as well */ +sched_clock_tick ();__touch_softlockup_watchdog ();return ; }Aside from the funny comment style (please fix) the fix does look sensible.
It turns out that further testing of this patch shows a regression in the ability to detect certain lockups. It is a direct result of the way the scheduling code makes use of the touch_softlockup_watchdog(). With the above proposed patch the tick was getting updated after a resume, but was also getting updated with the run_timers(), and if that happened before the softlockup tick, no softlockup would get reported (note that I was using some test code to induce softlockups).
The patch below is a bit more invasive but solves the problem by allowing kgdb to request that the sched cpu clock is updated only when returning from a state where we know we need to force the update.
Would this change be an acceptable solution to allow kgdb to peacefully exist with the softlockup code?
Thanks,
Jason.
-----
From: Jason Wessel <[email protected]>
Subject: [PATCH] softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume
When CONFIG_HAVE_UNSTABLE_SCHED_CLOCK is set sched_clock() gets the time from hardware, such as from TSC. In this configuration kgdb will report a softlock warning messages on resuming or detaching from a debug session.
Sequence of events in the problem case:
1) "cpu sched clock" and "hardware time" are at 100 sec prior to a call to kgdb_handle_exception()
2) Debugger waits in kgdb_handle_exception() for 80 sec and on exit the following is called ... touch_softlockup_watchdog() --> __raw_get_cpu_var(touch_timestamp) = 0;
3) "cpu sched clock" = 100s (it was not updated, because the interrupt was disabled in kgdb) but the "hardware time" = 180 sec
4) The first timer interrupt after resuming from kgdb_handle_exception updates the watchdog from the "cpu sched clock"
update_process_times() { ... run_local_timers() --> softlockup_tick() --> check (touch_timestamp == 0) (it is "YES" here, we have set "touch_timestamp = 0" at kgdb) --> __touch_softlockup_watchdog() ***(A)--> reset "touch_timestamp" to "get_timestamp()" (Here, the "touch_timestamp" will still be set to 100s.) ...
scheduler_tick() ***(B)--> sched_clock_tick() (update "cpu sched
clock" to "hardware time" = 180s) ... }5) The Second timer interrupt handler appears to have a large jump and trips the softlockup warning.
update_process_times() { ... run_local_timers() --> softlockup_tick() --> "cpu sched clock" - "touch_timestamp" = 180s-100s > 60s --> printk "soft lockup error messages" ... }
note: ***(A) reset "touch_timestamp" to "get_timestamp(this_cpu)"
Why "touch_timestamp" is 100 sec, instead of 180 sec?
With the CONFIG_HAVE_UNSTABLE_SCHED_CLOCK" set the call trace of get_timestamp() is:
get_timestamp(this_cpu) -->cpu_clock(this_cpu) -->sched_clock_cpu(this_cpu) -->__update_sched_clock(sched_clock_data, now)
The __update_sched_clock() function uses the GTOD tick value to create a window to normalize the "now" values. So if "now" values is too big for sched_clock_data, it will be ignored.
The fix is to invoke sched_clock_tick() to update "cpu sched clock" in order to recover from this state. This is done by introducing the function touch_softlockup_watchdog_sync(), which allows kgdb to request that the sched clock is updated when the watchdog thread runs the first time after a resume from kgdb.
--- include/linux/sched.h | 4 ++++ kernel/kgdb.c | 6 +++--- kernel/softlockup.c | 16 ++++++++++++++++ 3 files changed, 23 insertions(+), 3 deletions(-)
kernel/softlockup.c +16 -0
@@ -79,6 +79,14 @@ void touch_softlockup_watchdog(void) }EXPORT_SYMBOL (touch_softlockup_watchdog); +static int softlock_touch_sync[NR_CPUS ]; + +void touch_softlockup_watchdog_sync (void ) +{ + softlock_touch_sync[raw_smp_processor_id ()] =1 ; +__raw_get_cpu_var (touch_timestamp) =0 ; +} +void touch_all_softlockup_watchdogs (void ) {int cpu;
@@ -118,6 +126,14 @@ void softlockup_tick(void) }if (touch_timestamp ==0 ) { +if (unlikely (softlock_touch_sync[this_cpu])) { +/* +* If the time stamp was touched atomically +* make sure the scheduler tick is up to date. +*/ + softlock_touch_sync[this_cpu] =0 ; +sched_clock_tick (); + }__touch_softlockup_watchdog ();return ; }
include/linux/sched.h +4 -0
@@ -300,6 +300,7 @@ extern void sched_show_task(struct task_#ifdef CONFIG_DETECT_SOFTLOCKUP extern void softlockup_tick (void );extern void touch_softlockup_watchdog (void ); +extern void touch_softlockup_watchdog_sync (void );extern void touch_all_softlockup_watchdogs (void ); extern int proc_dosoftlockup_thresh(struct ctl_table *table, int write, struct file *filp, void __user *buffer,
@@ -313,6 +314,9 @@ static inline void softlockup_tick(void)static inline void touch_softlockup_watchdog (void ) { } +static inline void touch_softlockup_watchdog_sync (void ) +{ +}static inline void touch_all_softlockup_watchdogs (void ) { }
kernel/kgdb.c +3 -3
@@ -590,7 +590,7 @@ static void kgdb_wait(struct pt_regs *re/* Signal the primary CPU that we are done: */ atomic_set (&cpu_in_kgdb[cpu],0 ); -touch_softlockup_watchdog (); +touch_softlockup_watchdog_sync ();clocksource_touch_watchdog ();local_irq_restore (flags); }
@@ -1433,7 +1433,7 @@ acquirelock: atomic_read(&kgdb_cpu_doing_single_step) != cpu) {atomic_set (&kgdb_active,-1 ); -touch_softlockup_watchdog (); +touch_softlockup_watchdog_sync ();clocksource_touch_watchdog ();local_irq_restore (flags);
@@ -1526,7 +1526,7 @@ acquirelock: kgdb_restore:/* Free kgdb_active */ atomic_set (&kgdb_active,-1 ); -touch_softlockup_watchdog (); +touch_softlockup_watchdog_sync ();clocksource_touch_watchdog ();local_irq_restore (flags);
Quoted 142 lines · Jason Wessel
Quoted 19 lines · Peter Zijlstra
Quoted 2 lines · Jason Wessel
The fix is to simply invoke
sched_clock_tick()to update "cpu sched clock" on exit fromkgdb_handle_exception.Is that a regular IRQ context, or is that NMI context?
Quoted 13 lines
Signed-off-by Dongdong Deng <[email protected]>Signed-off-by Jason Wessel <[email protected]>Cc Ingo Molnar <[email protected]>--- kernel/softlockup.c | 3 +++ 1 file changed, 3 insertions(+)kernel/softlockup.c +0
-0@@ -118,6 +118,9 @@ void softlockup_tick(void) }Quoted 11 lines
kernel/softlockup.c void softlockup_tick(void) · line 139
Quoted 8 lines
kernel/softlockup.c +3
-0if (touch_timestamp ==0 ) { +/* If the time stamp was touched externally make sure the +* scheduler tick is up to date as well */ +sched_clock_tick ();__touch_softlockup_watchdog ();return ; }Aside from the funny comment style (please fix) the fix does look sensible.
It turns out that further testing of this patch shows a regression in the ability to detect certain lockups. It is a direct result of the way the scheduling code makes use of the
touch_softlockup_watchdog(). With the above proposed patch the tick was getting updated after a resume, but was also getting updated with therun_timers(), and if that happened before the softlockup tick, no softlockup would get reported (note that I was using some test code to induce softlockups).The patch below is a bit more invasive but solves the problem by allowing kgdb to request that the sched cpu clock is updated only when returning from a state where we know we need to force the update.
Would this change be an acceptable solution to allow kgdb to peacefully exist with the softlockup code?
Thanks,
Jason.-----
From: Jason Wessel <[email protected]>
Subject: [PATCH] softlockup: addsched_clock_tick()to avoid kernel warning on kgdb resumeWhen
CONFIG_HAVE_UNSTABLE_SCHED_CLOCKis setsched_clock()gets the time from hardware, such as from TSC. In this configuration kgdb will report a softlock warning messages on resuming or detaching from a debug session.Sequence of events in the problem case:
1) "cpu sched clock" and "hardware time" are at 100 sec prior to a call to
kgdb_handle_exception()2) Debugger waits in
kgdb_handle_exception()for 80 sec and on exit the following is called ...touch_softlockup_watchdog()-->__raw_get_cpu_var(touch_timestamp)= 0;3) "cpu sched clock" = 100s (it was not updated, because the interrupt was disabled in kgdb) but the "hardware time" = 180 sec
4) The first timer interrupt after resuming from
kgdb_handle_exceptionupdates the watchdog from the "cpu sched clock"
update_process_times(){ ...run_local_timers()-->softlockup_tick()--> check (touch_timestamp== 0) (it is "YES" here, we have set "touch_timestamp= 0" at kgdb) -->__touch_softlockup_watchdog()***(A)--> reset "touch_timestamp" to "get_timestamp()" (Here, the "touch_timestamp" will still be set to 100s.) ...scheduler_tick() ***(B)--> sched_clock_tick() (update "cpu sched clock" to "hardware time" = 180s) ... }5) The Second timer interrupt handler appears to have a large jump and trips the softlockup warning.
update_process_times(){ ...run_local_timers()-->softlockup_tick()--> "cpu sched clock" - "touch_timestamp" = 180s-100s > 60s --> printk "soft lockup error messages" ... }note: ***(A) reset "
touch_timestamp" to "get_timestamp(this_cpu)"Why "
touch_timestamp" is 100 sec, instead of 180 sec?With the
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK" set the call trace ofget_timestamp()is:
get_timestamp(this_cpu)-->cpu_clock(this_cpu) -->sched_clock_cpu(this_cpu) -->__update_sched_clock(sched_clock_data, now)The
__update_sched_clock()function uses the GTOD tick value to create a window to normalize the "now" values. So if "now" values is too big forsched_clock_data, it will be ignored.The fix is to invoke
sched_clock_tick()to update "cpu sched clock" in order to recover from this state. This is done by introducing the functiontouch_softlockup_watchdog_sync(), which allows kgdb to request that the sched clock is updated when the watchdog thread runs the first time after a resume from kgdb.Signed-off-by Jason Wessel <[email protected]>Signed-off-by Dongdong Deng <[email protected]>Cc Ingo Molnar <[email protected]>--- include/linux/sched.h | 4 ++++ kernel/kgdb.c | 6 +++--- kernel/softlockup.c | 16 ++++++++++++++++ 3 files changed, 23 insertions(+), 3 deletions(-)kernel/softlockup.c +8
-0@@ -79,6 +79,14 @@ void touch_softlockup_watchdog(void) }EXPORT_SYMBOL (touch_softlockup_watchdog); +static int softlock_touch_sync[NR_CPUS ]; + +void touch_softlockup_watchdog_sync (void ) +{ + softlock_touch_sync[raw_smp_processor_id ()] =1 ; +__raw_get_cpu_var (touch_timestamp) =0 ; +} +void touch_all_softlockup_watchdogs (void ) {int cpu;@@ -118,6 +126,14 @@ void softlockup_tick(void) }
kernel/softlockup.c void softlockup_tick(void) · line 136
Quoted 10 lines
kernel/softlockup.c +8
-0if (touch_timestamp ==0 ) { +if (unlikely (softlock_touch_sync[this_cpu])) { +/* +* If the time stamp was touched atomically +* make sure the scheduler tick is up to date. +*/ + softlock_touch_sync[this_cpu] =0 ; +sched_clock_tick (); + }
Hm, this looks quite ugly. Peter, Thomas, can you think of a cleaner solution?
Ingo
Re: [PATCH] softlockup: fix problem with long kernel pauses fromkgdb
Quoted 6 lines · Ingo Molnar
Quoted 4 lines · Jason Wessel
-----
From: Jason Wessel <[email protected]>
Subject: [PATCH] softlockup: addsched_clock_tick()to avoid kernel warning on kgdb resume
Quoted 9 lines
Quoted 5 lines
When
CONFIG_HAVE_UNSTABLE_SCHED_CLOCKis setsched_clock()gets the time from hardware, such as from TSC. In this configuration kgdb will report a softlock warning messages on resuming or detaching from a debug session.Hm, this looks quite ugly. Peter, Thomas, can you think of a cleaner solution?
Below was a more specific test case I received from Dongdong Deng which did not require kgdb. The test case is not something to merge, it is just a simple module to build to demonstrate the problem outside of kgdb.
The patch I submitted might have been on the "ugly" side, but there was not an obvious way to solve the problem without making changes in kernel/sched*. I opted for something that was entirely self contained to the softlockup code.
Jason.
============= test case ================
/* Information about this test case.
* -------------------------------------
* 1: CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
*
* 2: export clocksource_touch_watchdog for this test case to build. *
*
* diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c
* index 7466cb8..c81f9a3 100644
* --- a/kernel/time/clocksource.c
* +++ b/kernel/time/clocksource.c
* @@ -319,6 +319,7 @@ void clocksource_touch_watchdog(void)
* {
* clocksource_resume_watchdog();
* }
* +EXPORT_SYMBOL(clocksource_touch_watchdog);
*
* 3: build following module, insmod and observe the softlockup
* warning or watch your system reboot if you have reboot
* on softlockup set.
*
*/
#include <linux/module.h>
#include <linux/init.h>
#include <linux/irqflags.h>
#include <linux/delay.h>
#include <linux/sched.h>
#include <linux/clocksource.h>
static int example_init(void)
{
unsigned long flags;
unsigned i;
printk(KERN_NOTICE"softlockup test init\n");
local_irq_save(flags);
for (i = 0; i < 70; i++)
mdelay(1000);
touch_softlockup_watchdog();
clocksource_touch_watchdog();
local_irq_restore(flags);
return 0;
}static void example_exit(void)
{
printk(KERN_NOTICE"softlockup test exit\n");
}module_init(example_init);module_exit(example_exit);MODULE_LICENSE("GPL");
Re: [PATCH] softlockup: fix problem with long kernel pauses fromkgdb
Quoted 22 lines · Jason Wessel
Quoted 6 lines · Ingo Molnar
Quoted 4 lines · Jason Wessel
-----
From: Jason Wessel <[email protected]>
Subject: [PATCH] softlockup: addsched_clock_tick()to avoid kernel warning on kgdb resumeQuoted 9 lines
Quoted 5 lines
When
CONFIG_HAVE_UNSTABLE_SCHED_CLOCKis setsched_clock()gets the time from hardware, such as from TSC. In this configuration kgdb will report a softlock warning messages on resuming or detaching from a debug session.Hm, this looks quite ugly. Peter, Thomas, can you think of a cleaner solution?
Below was a more specific test case I received from Dongdong Deng which did not require kgdb. The test case is not something to merge, it is just a simple module to build to demonstrate the problem outside of kgdb.
Quoted 4 lines
The patch I submitted might have been on the "ugly" side, but there was not an obvious way to solve the problem without making changes in
kernel/sched*. I opted for something that was entirely self contained to the softlockup code.
it's not really the fault of your patch really - all these integrations along dynticks, scheduler time and softlockup feel a bit awkward. Wondering whether there's some nicer way.
Ingo
Hi Jason & Ingo,
If we apply Peter's patch which was send out just a moment ago, I believe this issue will gone. :-)
Thanks,
Dongdong
[PATCH -rt] timer: delay waking softirqs from the jiffy tick
...
...
--- kernel/timer.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/timer.c +1 -1
@@ -1221,10 +1221,10 @@ void update_process_times(int user_tick)/* Note: this timer irq context must be accounted for as well. */ account_process_tick (p, user_tick); +scheduler_tick ();run_local_timers ();if (rcu_pending (cpu))rcu_check_callbacks (cpu, user_tick); -scheduler_tick ();run_posix_cpu_timers (p); }
Quoted 34 lines · Ingo Molnar
Quoted 22 lines · Jason Wessel
Quoted 6 lines · Ingo Molnar
Quoted 4 lines · Jason Wessel
-----
From: Jason Wessel <[email protected]>
Subject: [PATCH] softlockup: addsched_clock_tick()to avoid kernel warning on kgdb resumeQuoted 9 lines
Quoted 5 lines
When
CONFIG_HAVE_UNSTABLE_SCHED_CLOCKis setsched_clock()gets the time from hardware, such as from TSC. In this configuration kgdb will report a softlock warning messages on resuming or detaching from a debug session.Hm, this looks quite ugly. Peter, Thomas, can you think of a cleaner solution?
Below was a more specific test case I received from Dongdong Deng which did not require kgdb. The test case is not something to merge, it is just a simple module to build to demonstrate the problem outside of kgdb.
Quoted 4 lines
The patch I submitted might have been on the "ugly" side, but there was not an obvious way to solve the problem without making changes in
kernel/sched*. I opted for something that was entirely self contained to the softlockup code.it's not really the fault of your patch really - all these integrations along dynticks, scheduler time and softlockup feel a bit awkward. Wondering whether there's some nicer way.
Ingo
Quoted 146 lines · Ingo Molnar
Quoted 143 lines · Jason Wessel
Quoted 19 lines · Peter Zijlstra
Quoted 2 lines · Jason Wessel
The fix is to simply invoke
sched_clock_tick()to update "cpu sched clock" on exit fromkgdb_handle_exception.Is that a regular IRQ context, or is that NMI context?
Quoted 13 lines
Signed-off-by Dongdong Deng <[email protected]>Signed-off-by Jason Wessel <[email protected]>Cc Ingo Molnar <[email protected]>---
kernel/softlockup.c | 3 +++1 file changed, 3 insertions(+)kernel/softlockup.c +0
-0@@ -118,6 +118,9 @@ void softlockup_tick(void) }Quoted 11 lines
kernel/softlockup.c void softlockup_tick(void) · line 126
Quoted 8 lines
kernel/softlockup.c +3
-0if (touch_timestamp == 0) { + /* If the time stamp was touched externally make sure the + * scheduler tick is up to date as well */ + sched_clock_tick(); __touch_softlockup_watchdog(); return; }Aside from the funny comment style (please fix) the fix does look sensible.
It turns out that further testing of this patch shows a regression in the ability to detect certain lockups. It is a direct result of the way the scheduling code makes use of the
touch_softlockup_watchdog(). With the above proposed patch the tick was getting updated after a resume, but was also getting updated with therun_timers(), and if that happened before the softlockup tick, no softlockup would get reported (note that I was using some test code to induce softlockups).The patch below is a bit more invasive but solves the problem by allowing kgdb to request that the sched cpu clock is updated only when returning from a state where we know we need to force the update.
Would this change be an acceptable solution to allow kgdb to peacefully exist with the softlockup code?
Thanks,
Jason.-----
From: Jason Wessel <[email protected]>
Subject: [PATCH] softlockup: addsched_clock_tick()to avoid kernel warning on kgdb resumeWhen
CONFIG_HAVE_UNSTABLE_SCHED_CLOCKis setsched_clock()gets the time from hardware, such as from TSC. In this configuration kgdb will report a softlock warning messages on resuming or detaching from a debug session.Sequence of events in the problem case:
1) "cpu sched clock" and "hardware time" are at 100 sec prior to a call to
kgdb_handle_exception()2) Debugger waits in
kgdb_handle_exception()for 80 sec and on exit the following is called ... touch_softlockup_watchdog() -->__raw_get_cpu_var(touch_timestamp)= 0;3) "cpu sched clock" = 100s (it was not updated, because the interrupt was disabled in kgdb) but the "hardware time" = 180 sec
4) The first timer interrupt after resuming from
kgdb_handle_exceptionupdates the watchdog from the "cpu sched clock"
update_process_times(){ ... run_local_timers() -->softlockup_tick()--> check (touch_timestamp== 0) (it is "YES" here, we have set "touch_timestamp= 0" at kgdb) -->__touch_softlockup_watchdog()***(A)--> reset "touch_timestamp" to "get_timestamp()" (Here, the "touch_timestamp" will still be set to 100s.) ...
scheduler_tick()***(B)-->sched_clock_tick()(update "cpu sched clock" to "hardware time" = 180s) ... }5) The Second timer interrupt handler appears to have a large jump and trips the softlockup warning.
update_process_times(){ ... run_local_timers() -->softlockup_tick()--> "cpu sched clock" - "touch_timestamp" = 180s-100s > 60s --> printk "soft lockup error messages" ... }note: ***(A) reset "
touch_timestamp" to "get_timestamp(this_cpu)"Why "
touch_timestamp" is 100 sec, instead of 180 sec?With the
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK" set the call trace ofget_timestamp()is:
get_timestamp(this_cpu)-->cpu_clock(this_cpu) -->sched_clock_cpu(this_cpu) -->__update_sched_clock(sched_clock_data, now)The
__update_sched_clock()function uses the GTOD tick value to create a window to normalize the "now" values. So if "now" values is too big forsched_clock_data, it will be ignored.The fix is to invoke
sched_clock_tick()to update "cpu sched clock" in order to recover from this state. This is done by introducing the functiontouch_softlockup_watchdog_sync(), which allows kgdb to request that the sched clock is updated when the watchdog thread runs the first time after a resume from kgdb.Signed-off-by Jason Wessel <[email protected]>Signed-off-by Dongdong Deng <[email protected]>Cc Ingo Molnar <[email protected]>---
include/linux/sched.h | 4 ++++
kernel/kgdb.c | 6 +++---
kernel/softlockup.c | 16 ++++++++++++++++3 files changed, 23 insertions(+), 3 deletions(-)kernel/softlockup.c +8
-0@@ -79,6 +79,14 @@ void touch_softlockup_watchdog(void) } EXPORT_SYMBOL(touch_softlockup_watchdog); +static int softlock_touch_sync[NR_CPUS ]; + +void touch_softlockup_watchdog_sync (void ) +{ + softlock_touch_sync[raw_smp_processor_id()] = 1; + __raw_get_cpu_var(touch_timestamp) = 0; +} + void touch_all_softlockup_watchdogs(void) { int cpu;@@ -118,6 +126,14 @@ void softlockup_tick(void) }
Quoted 13 lines
kernel/softlockup.c void softlockup_tick(void) · line 136
Quoted 9 lines
kernel/softlockup.c +8
-0if (touch_timestamp == 0) { + if (unlikely(softlock_touch_sync[this_cpu])) { + /* + * If the time stamp was touched atomically + * make sure the scheduler tick is up to date. + */ + softlock_touch_sync[this_cpu] = 0; + sched_clock_tick(); + }Hm, this looks quite ugly. Peter, Thomas, can you think of a cleaner solution?
We need not to sync up sched_clock in softlockup_tick, it can just return and sync up the timestamp in the next tick. This will not touch what should be done by dynticks, scheduler time and so on and keep softlockup_tick clean.
I have rewrote the patch based on above, please check the attachment.
Thanks,
Yong
Quoted 7 lines
Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From 1821356596bdd28dfc69f708a6929394da351001 Mon Sep 17 00:00:00 2001
From: Yong Zhang <[email protected]>
Date: Sat, 26 Sep 2009 10:37:14 +0800
Subject: [PATCH] softlockup: introduce touch_softlockup_watchdog_sync()
In case of system halted for a long time, system will do time sync up in the first incoming tick. But if we also touch softlockup_tick timestamp at the same time, softlockup_tick could keep a old timestamp and this will lead annoying softlockup warning. So move softlockup syncing it's timestamp to the next tick.
LKML-Reference: <http://lkml.org/lkml/2009/7/27/317>
--- include/linux/sched.h | 4 ++++ kernel/softlockup.c | 18 ++++++++++++++++++ 2 files changed, 22 insertions(+), 0 deletions(-)
include/linux/sched.h +4 -0
@@ -307,6 +307,7 @@ extern void sched_show_task(struct task_struct *p);#ifdef CONFIG_DETECT_SOFTLOCKUP extern void softlockup_tick (void );extern void touch_softlockup_watchdog (void ); +extern void touch_softlockup_watchdog_sync (void );extern void touch_all_softlockup_watchdogs (void ); extern int proc_dosoftlockup_thresh(struct ctl_table *table, int write, void __user *buffer,
@@ -320,6 +321,9 @@ static inline void softlockup_tick(void)static inline void touch_softlockup_watchdog (void ) { } +static inline void touch_softlockup_watchdog_sync (void ) +{ +}static inline void touch_all_softlockup_watchdogs (void ) { }
kernel/softlockup.c +18 -0
@@ -25,6 +25,7 @@ static DEFINE_SPINLOCK(print_lock); static DEFINE_PER_CPU(unsigned long, touch_timestamp); static DEFINE_PER_CPU(unsigned long, print_timestamp); static DEFINE_PER_CPU(struct task_struct *, watchdog_task); +static DEFINE_PER_CPU(bool, softlock_touch_sync);static int __read_mostly did_panic;int __read_mostly softlockup_thresh =60 ;
@@ -79,6 +80,13 @@ void touch_softlockup_watchdog(void) }EXPORT_SYMBOL (touch_softlockup_watchdog); +void touch_softlockup_watchdog_sync (void ) +{ +__raw_get_cpu_var (touch_timestamp) =0 ; +__raw_get_cpu_var (softlock_touch_sync) = true; +} +EXPORT_SYMBOL (touch_softlockup_watchdog_sync); +void touch_all_softlockup_watchdogs (void ) {int cpu;
@@ -117,6 +125,15 @@ void softlockup_tick(void)return ; } +/* +* If the scheduler tick is not up to date. Skip this check +* and wait for the timestamp to to synced in the next tick +*/ +if (unlikely (per_cpu (softlock_touch_sync, this_cpu))) { +per_cpu (softlock_touch_sync, this_cpu) = false; +return ; + } +if (touch_timestamp ==0 ) {__touch_softlockup_watchdog ();return ;
@@ -216,6 +233,7 @@ cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu)return NOTIFY_BAD ; }per_cpu (touch_timestamp, hotcpu) =0 ; +per_cpu (softlock_touch_sync, hotcpu) = false;per_cpu (watchdog_task, hotcpu) = p;kthread_bind (p, hotcpu);break ;
Signature · 1 line
1.6.0.4