Skip to contentSkip to conversation

softlockup: fix problem with long kernel pauses from kgdb

PATCH In review Jason Wessel 5 participants 9 messages Jul 27, 2009 – Sep 26, 2009 Scheduler

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 softlockup: fix problem with long kernel pauses from kgdb

Ingo,

Given that you are the maintainer of kernel/softlockup.c, I am seeking advice as to how to properly fix this problem.

The short version of the problem is:

* CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
* Attach to kgdb after boot
* Wait for 90 seconds
* Execute a continue in gdb
* You receive a warning about softlockup

The patch that follows is a lengthy analysis on the issue, but the question here is what is the right way to fix this?

It seems that a provision is required in order to get the clock synced up prior to touching the watch dog. It was not clear that it was a good idea to unconditionally call the sched_clock_tick() from the softlockup touch code.

Your input on this issue is greatly appreciated.

Thanks,
Jason.

----------
From: "Dongdong Deng" <[email protected]>
Subject: [PATCH] softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume

When CONFIG_HAVE_UNSTABLE_SCHED_CLOCK is set and the sched_clock() was gets the time from hardware, such as from TSC, kgdb often causes softlock warning messages on resuming or detaching from a debug session.

Sequence of events in the problem case:

1) "cpu sched clock" and "hardware time" are at 100 seconds prior to a call to kgdb_handle_exception()

2) Debugger waits in kgdb_handle_exception() for 80 seconds and on exit the following is called

    ...
    touch_softlockup_watchdog()
       --> __raw_get_cpu_var(touch_timestamp) = 0;

3) "cpu sched clock" = 100s (it didn't be updated, because the interrupt was disabled in kgdb) but the "hardware time" = 180s

4) The first timer interrupt after resuming from kgdb_handle_exception updates the watchdog from the "cpu sched clock"

update_process_times()

{
    ...
    run_local_timers()
          --> softlockup_tick()
              --> check (touch_timestamp == 0)
                (it is "YES" here, we have set "touch_timestamp = 0"
                at kgdb)
                --> __touch_softlockup_watchdog()
              ***(A)--> reset "touch_timestamp" to "get_timestamp()"
                     (Here, the "touch_timestamp" will still be set to
                      100s.)
    ...

    scheduler_tick()
      ***(B)--> sched_clock_tick()
               (update "cpu sched clock" to "hardware time" = 180s)
    ...
}

5) The Second timer interrupt handler appears to have a large jump and trips the softlockup warning.

update_process_times()

{
    ...
    run_local_timers()
        --> softlockup_tick()
           --> "cpu sched clock" - "touch_timestamp" = 180s-100s > 60s
           --> printk "soft lockup error messages"
    ...
}

note:
***(A) reset "touch_timestamp" to "get_timestamp(this_cpu)"

Why "touch_timestamp" will be 100s, not 180s ?

We enable "CONFIG_HAVE_UNSTABLE_SCHED_CLOCK",
so the call trace of get_timestamp() is:

get_timestamp(this_cpu)

      -->cpu_clock(this_cpu)
         -->sched_clock_cpu(this_cpu)
            -->__update_sched_clock(sched_clock_data, now)

22 static u64 __update_sched_clock(struct sched_clock_data *scd, u64 = now)
23 {
24 s64 delta = now - scd->tick_raw;
25 u64 clock, min_clock, max_clock;
26
27 WARN_ON_ONCE(!irqs_disabled());
28
29 if (unlikely(delta < 0))
30 delta = 0;
31
32 clock = scd->tick_gtod + delta;
33
34 min_clock = wrap_max(scd->tick_gtod, scd->clock);
35 max_clock = wrap_max(scd->clock, scd->tick_gtod + TICK_NSEC);
36
37 clock = wrap_max(clock, min_clock);
38 clock = wrap_min(clock, max_clock);
39
40 scd->clock = clock;
41
42 return scd->clock;
43 }

Please pay attention to __update_sched_clock() function, it uses the GTOD tick value to create a window to filter crazy "now" values. So if "now" values is too big for sched_clock_data, it will be ignored.

as the data of "step 3)",
"now" = sched_clock() = "hardware time" = 180s and sched_clock_data = 100s.
180s is too big for 100s, it will be ignored.

That's why the touch_timestamp will be set to 100s, not 180s.

The fix is to simply invoke sched_clock_tick() to update "cpu sched clock" on exit from kgdb_handle_exception.

---
 kernel/softlockup.c |    3 +++
 1 file changed, 3 insertions(+)
kernel/softlockup.c +3 -0
@@ -118,6 +118,9 @@ void softlockup_tick(void)
 	}
 
 	if (touch_timestamp == 0) {
+		/* If the time stamp was touched externally make sure the
+		 * scheduler tick is up to date as well */
+		sched_clock_tick();
 		__touch_softlockup_watchdog();
 		return;
 	}
maintainer
#1 lore raw
Quoted 3 lines · Jason Wessel

The fix is to simply invoke sched_clock_tick() to update "cpu sched clock" on exit from kgdb_handle_exception.

Is that a regular IRQ context, or is that NMI context?

Quoted 12 lines
---
 kernel/softlockup.c |    3 +++
 1 file changed, 3 insertions(+)

kernel/softlockup.c void softlockup_tick(void) · line 126

Quoted 9 lines
kernel/softlockup.c +3 -0
@@ -118,6 +118,9 @@ void softlockup_tick(void)
        }
 
        if (touch_timestamp == 0) {
+               /* If the time stamp was touched externally make sure the
+                * scheduler tick is up to date as well */
+               sched_clock_tick();
                __touch_softlockup_watchdog();
                return;
        }

Aside from the funny comment style (please fix) the fix does look sensible.

#2 lore raw
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Jason Wessel

The fix is to simply invoke sched_clock_tick() to update "cpu sched clock" on exit from kgdb_handle_exception.

Is that a regular IRQ context, or is that NMI context?

The patch text there is incorrect (which I can fix), because there were 3 other versions of this particular patch. The version of this patch that tried to use sched_clock_tick() from the exception context hung the system because there was lock contention for CPUs that were not in the running state.

In this version, the sched_clock_tick() added in the softlockup_tick(), which did not result in a hard hang. I am pretty sure that softlockup_tick() runs as a part of the timer interrupt processing from run_local_timers(), so there was no further need to turn off the interrupts.

Jason.

Quoted 13 lines
Quoted 13 lines
---
 kernel/softlockup.c |    3 +++
 1 file changed, 3 insertions(+)
kernel/softlockup.c +0 -0
@@ -118,6 +118,9 @@ void softlockup_tick(void)
        }
 
Quoted 13 lines

kernel/softlockup.c void softlockup_tick(void) · line 126

Quoted 9 lines
kernel/softlockup.c +3 -0
        if (touch_timestamp == 0) {
+               /* If the time stamp was touched externally make sure the
+                * scheduler tick is up to date as well */
+               sched_clock_tick();
                __touch_softlockup_watchdog();
                return;
        }

Aside from the funny comment style (please fix) the fix does look sensible.

No problem.

Thanks,
Jason.

#3 lore raw
In reply to Peter Zijlstra
Quoted 19 lines · Peter Zijlstra
Quoted 2 lines · Jason Wessel

The fix is to simply invoke sched_clock_tick() to update "cpu sched clock" on exit from kgdb_handle_exception.

Is that a regular IRQ context, or is that NMI context?

Quoted 13 lines
---
 kernel/softlockup.c |    3 +++
 1 file changed, 3 insertions(+)
kernel/softlockup.c +0 -0
@@ -118,6 +118,9 @@ void softlockup_tick(void)
        }
 
Quoted 11 lines

kernel/softlockup.c void softlockup_tick(void) · line 126

Quoted 8 lines
kernel/softlockup.c +3 -0
        if (touch_timestamp == 0) {
+               /* If the time stamp was touched externally make sure the
+                * scheduler tick is up to date as well */
+               sched_clock_tick();
                __touch_softlockup_watchdog();
                return;
        }

Aside from the funny comment style (please fix) the fix does look sensible.

It turns out that further testing of this patch shows a regression in the ability to detect certain lockups. It is a direct result of the way the scheduling code makes use of the touch_softlockup_watchdog(). With the above proposed patch the tick was getting updated after a resume, but was also getting updated with the run_timers(), and if that happened before the softlockup tick, no softlockup would get reported (note that I was using some test code to induce softlockups).

The patch below is a bit more invasive but solves the problem by allowing kgdb to request that the sched cpu clock is updated only when returning from a state where we know we need to force the update.

Would this change be an acceptable solution to allow kgdb to peacefully exist with the softlockup code?

Thanks,
Jason.

-----
From: Jason Wessel <[email protected]>
Subject: [PATCH] softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume

When CONFIG_HAVE_UNSTABLE_SCHED_CLOCK is set sched_clock() gets the time from hardware, such as from TSC. In this configuration kgdb will report a softlock warning messages on resuming or detaching from a debug session.

Sequence of events in the problem case:

1) "cpu sched clock" and "hardware time" are at 100 sec prior to a call to kgdb_handle_exception()

2) Debugger waits in kgdb_handle_exception() for 80 sec and on exit the following is called ... touch_softlockup_watchdog() --> __raw_get_cpu_var(touch_timestamp) = 0;

3) "cpu sched clock" = 100s (it was not updated, because the interrupt was disabled in kgdb) but the "hardware time" = 180 sec

4) The first timer interrupt after resuming from kgdb_handle_exception updates the watchdog from the "cpu sched clock"

update_process_times() { ... run_local_timers() --> softlockup_tick() --> check (touch_timestamp == 0) (it is "YES" here, we have set "touch_timestamp = 0" at kgdb) --> __touch_softlockup_watchdog() ***(A)--> reset "touch_timestamp" to "get_timestamp()" (Here, the "touch_timestamp" will still be set to 100s.) ...

    scheduler_tick() ***(B)--> sched_clock_tick() (update "cpu sched
    clock" to "hardware time" = 180s) ...  }

5) The Second timer interrupt handler appears to have a large jump and trips the softlockup warning.

update_process_times() { ... run_local_timers() --> softlockup_tick() --> "cpu sched clock" - "touch_timestamp" = 180s-100s > 60s --> printk "soft lockup error messages" ... }

note: ***(A) reset "touch_timestamp" to "get_timestamp(this_cpu)"

Why "touch_timestamp" is 100 sec, instead of 180 sec?

With the CONFIG_HAVE_UNSTABLE_SCHED_CLOCK" set the call trace of get_timestamp() is:

get_timestamp(this_cpu) -->cpu_clock(this_cpu) -->sched_clock_cpu(this_cpu) -->__update_sched_clock(sched_clock_data, now)

The __update_sched_clock() function uses the GTOD tick value to create a window to normalize the "now" values. So if "now" values is too big for sched_clock_data, it will be ignored.

The fix is to invoke sched_clock_tick() to update "cpu sched clock" in order to recover from this state. This is done by introducing the function touch_softlockup_watchdog_sync(), which allows kgdb to request that the sched clock is updated when the watchdog thread runs the first time after a resume from kgdb.

---
 include/linux/sched.h |    4 ++++
 kernel/kgdb.c         |    6 +++---
 kernel/softlockup.c   |   16 ++++++++++++++++
 3 files changed, 23 insertions(+), 3 deletions(-)
kernel/softlockup.c +16 -0
@@ -79,6 +79,14 @@ void touch_softlockup_watchdog(void)
 }
 EXPORT_SYMBOL(touch_softlockup_watchdog);
 
+static int softlock_touch_sync[NR_CPUS];
+
+void touch_softlockup_watchdog_sync(void)
+{
+	softlock_touch_sync[raw_smp_processor_id()] = 1;
+	__raw_get_cpu_var(touch_timestamp) = 0;
+}
+
 void touch_all_softlockup_watchdogs(void)
 {
 	int cpu;
@@ -118,6 +126,14 @@ void softlockup_tick(void)
 	}
 
 	if (touch_timestamp == 0) {
+		if (unlikely(softlock_touch_sync[this_cpu])) {
+			/*
+			 * If the time stamp was touched atomically
+			 * make sure the scheduler tick is up to date.
+			 */
+			softlock_touch_sync[this_cpu] = 0;
+			sched_clock_tick();
+		}
 		__touch_softlockup_watchdog();
 		return;
 	}
include/linux/sched.h +4 -0
@@ -300,6 +300,7 @@ extern void sched_show_task(struct task_
 #ifdef CONFIG_DETECT_SOFTLOCKUP
 extern void softlockup_tick(void);
 extern void touch_softlockup_watchdog(void);
+extern void touch_softlockup_watchdog_sync(void);
 extern void touch_all_softlockup_watchdogs(void);
 extern int proc_dosoftlockup_thresh(struct ctl_table *table, int write,
 				    struct file *filp, void __user *buffer,
@@ -313,6 +314,9 @@ static inline void softlockup_tick(void)
 static inline void touch_softlockup_watchdog(void)
 {
 }
+static inline void touch_softlockup_watchdog_sync(void)
+{
+}
 static inline void touch_all_softlockup_watchdogs(void)
 {
 }
kernel/kgdb.c +3 -3
@@ -590,7 +590,7 @@ static void kgdb_wait(struct pt_regs *re
 
 	/* Signal the primary CPU that we are done: */
 	atomic_set(&cpu_in_kgdb[cpu], 0);
-	touch_softlockup_watchdog();
+	touch_softlockup_watchdog_sync();
 	clocksource_touch_watchdog();
 	local_irq_restore(flags);
 }
@@ -1433,7 +1433,7 @@ acquirelock:
 	    atomic_read(&kgdb_cpu_doing_single_step) != cpu) {
 
 		atomic_set(&kgdb_active, -1);
-		touch_softlockup_watchdog();
+		touch_softlockup_watchdog_sync();
 		clocksource_touch_watchdog();
 		local_irq_restore(flags);
 
@@ -1526,7 +1526,7 @@ acquirelock:
 kgdb_restore:
 	/* Free kgdb_active */
 	atomic_set(&kgdb_active, -1);
-	touch_softlockup_watchdog();
+	touch_softlockup_watchdog_sync();
 	clocksource_touch_watchdog();
 	local_irq_restore(flags);
 
#4 lore raw
Quoted 142 lines · Jason Wessel
Quoted 19 lines · Peter Zijlstra
Quoted 2 lines · Jason Wessel

The fix is to simply invoke sched_clock_tick() to update "cpu sched clock" on exit from kgdb_handle_exception.

Is that a regular IRQ context, or is that NMI context?

Quoted 13 lines
---
 kernel/softlockup.c |    3 +++
 1 file changed, 3 insertions(+)
kernel/softlockup.c +0 -0
@@ -118,6 +118,9 @@ void softlockup_tick(void)
        }
 
Quoted 11 lines

kernel/softlockup.c void softlockup_tick(void) · line 139

Quoted 8 lines
kernel/softlockup.c +3 -0
        if (touch_timestamp == 0) {
+               /* If the time stamp was touched externally make sure the
+                * scheduler tick is up to date as well */
+               sched_clock_tick();
                __touch_softlockup_watchdog();
                return;
        }

Aside from the funny comment style (please fix) the fix does look sensible.

It turns out that further testing of this patch shows a regression in the ability to detect certain lockups. It is a direct result of the way the scheduling code makes use of the touch_softlockup_watchdog(). With the above proposed patch the tick was getting updated after a resume, but was also getting updated with the run_timers(), and if that happened before the softlockup tick, no softlockup would get reported (note that I was using some test code to induce softlockups).

The patch below is a bit more invasive but solves the problem by allowing kgdb to request that the sched cpu clock is updated only when returning from a state where we know we need to force the update.

Would this change be an acceptable solution to allow kgdb to peacefully exist with the softlockup code?

Thanks,
Jason.

-----
From: Jason Wessel <[email protected]>
Subject: [PATCH] softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume

When CONFIG_HAVE_UNSTABLE_SCHED_CLOCK is set sched_clock() gets the time from hardware, such as from TSC. In this configuration kgdb will report a softlock warning messages on resuming or detaching from a debug session.

Sequence of events in the problem case:

1) "cpu sched clock" and "hardware time" are at 100 sec prior to a call to kgdb_handle_exception()

2) Debugger waits in kgdb_handle_exception() for 80 sec and on exit the following is called ... touch_softlockup_watchdog() --> __raw_get_cpu_var(touch_timestamp) = 0;

3) "cpu sched clock" = 100s (it was not updated, because the interrupt was disabled in kgdb) but the "hardware time" = 180 sec

4) The first timer interrupt after resuming from kgdb_handle_exception updates the watchdog from the "cpu sched clock"

update_process_times() { ... run_local_timers() --> softlockup_tick() --> check (touch_timestamp == 0) (it is "YES" here, we have set "touch_timestamp = 0" at kgdb) --> __touch_softlockup_watchdog() ***(A)--> reset "touch_timestamp" to "get_timestamp()" (Here, the "touch_timestamp" will still be set to 100s.) ...

    scheduler_tick() ***(B)--> sched_clock_tick() (update "cpu sched
    clock" to "hardware time" = 180s) ...  }

5) The Second timer interrupt handler appears to have a large jump and trips the softlockup warning.

update_process_times() { ... run_local_timers() --> softlockup_tick() --> "cpu sched clock" - "touch_timestamp" = 180s-100s > 60s --> printk "soft lockup error messages" ... }

note: ***(A) reset "touch_timestamp" to "get_timestamp(this_cpu)"

Why "touch_timestamp" is 100 sec, instead of 180 sec?

With the CONFIG_HAVE_UNSTABLE_SCHED_CLOCK" set the call trace of get_timestamp() is:

get_timestamp(this_cpu) -->cpu_clock(this_cpu) -->sched_clock_cpu(this_cpu) -->__update_sched_clock(sched_clock_data, now)

The __update_sched_clock() function uses the GTOD tick value to create a window to normalize the "now" values. So if "now" values is too big for sched_clock_data, it will be ignored.

The fix is to invoke sched_clock_tick() to update "cpu sched clock" in order to recover from this state. This is done by introducing the function touch_softlockup_watchdog_sync(), which allows kgdb to request that the sched clock is updated when the watchdog thread runs the first time after a resume from kgdb.

---
 include/linux/sched.h |    4 ++++
 kernel/kgdb.c         |    6 +++---
 kernel/softlockup.c   |   16 ++++++++++++++++
 3 files changed, 23 insertions(+), 3 deletions(-)
kernel/softlockup.c +8 -0
@@ -79,6 +79,14 @@ void touch_softlockup_watchdog(void)
 }
 EXPORT_SYMBOL(touch_softlockup_watchdog);
 
+static int softlock_touch_sync[NR_CPUS];
+
+void touch_softlockup_watchdog_sync(void)
+{
+	softlock_touch_sync[raw_smp_processor_id()] = 1;
+	__raw_get_cpu_var(touch_timestamp) = 0;
+}
+
 void touch_all_softlockup_watchdogs(void)
 {
 	int cpu;
@@ -118,6 +126,14 @@ void softlockup_tick(void)
 	}

kernel/softlockup.c void softlockup_tick(void) · line 136

Quoted 10 lines
kernel/softlockup.c +8 -0
 
 	if (touch_timestamp == 0) {
+		if (unlikely(softlock_touch_sync[this_cpu])) {
+			/*
+			 * If the time stamp was touched atomically
+			 * make sure the scheduler tick is up to date.
+			 */
+			softlock_touch_sync[this_cpu] = 0;
+			sched_clock_tick();
+		}

Hm, this looks quite ugly. Peter, Thomas, can you think of a cleaner solution?

Ingo

#5 lore raw

Re: [PATCH] softlockup: fix problem with long kernel pauses fromkgdb

Quoted 6 lines · Ingo Molnar
Quoted 4 lines · Jason Wessel

-----
From: Jason Wessel <[email protected]>
Subject: [PATCH] softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume

Quoted 9 lines
Quoted 5 lines

When CONFIG_HAVE_UNSTABLE_SCHED_CLOCK is set sched_clock() gets the time from hardware, such as from TSC. In this configuration kgdb will report a softlock warning messages on resuming or detaching from a debug session.

Hm, this looks quite ugly. Peter, Thomas, can you think of a cleaner solution?

Below was a more specific test case I received from Dongdong Deng which did not require kgdb. The test case is not something to merge, it is just a simple module to build to demonstrate the problem outside of kgdb.

The patch I submitted might have been on the "ugly" side, but there was not an obvious way to solve the problem without making changes in kernel/sched*. I opted for something that was entirely self contained to the softlockup code.

Jason.

============= test case ================
/* Information about this test case.
* -------------------------------------
* 1: CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
*
* 2: export clocksource_touch_watchdog for this test case to build. *
*
* diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c
* index 7466cb8..c81f9a3 100644
* --- a/kernel/time/clocksource.c
* +++ b/kernel/time/clocksource.c
* @@ -319,6 +319,7 @@ void clocksource_touch_watchdog(void)
* {
* clocksource_resume_watchdog();
* }
* +EXPORT_SYMBOL(clocksource_touch_watchdog);
*
* 3: build following module, insmod and observe the softlockup
* warning or watch your system reboot if you have reboot
* on softlockup set.
*
*/

#include <linux/module.h>
#include <linux/init.h>
#include <linux/irqflags.h>
#include <linux/delay.h>
#include <linux/sched.h>
#include <linux/clocksource.h>

static int example_init(void)

{
	unsigned long flags;
	unsigned i;

	printk(KERN_NOTICE"softlockup test init\n");

	local_irq_save(flags);

	for (i = 0; i < 70; i++)
		mdelay(1000);

	touch_softlockup_watchdog();
	clocksource_touch_watchdog();

	local_irq_restore(flags);

	return 0;
}

static void example_exit(void)

{
	printk(KERN_NOTICE"softlockup test exit\n");
}

module_init(example_init);
module_exit(example_exit);
MODULE_LICENSE("GPL");

#6 lore raw

Re: [PATCH] softlockup: fix problem with long kernel pauses fromkgdb

Quoted 22 lines · Jason Wessel
Quoted 6 lines · Ingo Molnar
Quoted 4 lines · Jason Wessel

-----
From: Jason Wessel <[email protected]>
Subject: [PATCH] softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume

Quoted 9 lines
Quoted 5 lines

When CONFIG_HAVE_UNSTABLE_SCHED_CLOCK is set sched_clock() gets the time from hardware, such as from TSC. In this configuration kgdb will report a softlock warning messages on resuming or detaching from a debug session.

Hm, this looks quite ugly. Peter, Thomas, can you think of a cleaner solution?

Below was a more specific test case I received from Dongdong Deng which did not require kgdb. The test case is not something to merge, it is just a simple module to build to demonstrate the problem outside of kgdb.

Quoted 4 lines

The patch I submitted might have been on the "ugly" side, but there was not an obvious way to solve the problem without making changes in kernel/sched*. I opted for something that was entirely self contained to the softlockup code.

it's not really the fault of your patch really - all these integrations along dynticks, scheduler time and softlockup feel a bit awkward. Wondering whether there's some nicer way.

Ingo

#7 lore raw

Hi Jason & Ingo,

If we apply Peter's patch which was send out just a moment ago, I believe this issue will gone. :-)

Thanks,
Dongdong

[PATCH -rt] timer: delay waking softirqs from the jiffy tick
...
...

---
 kernel/timer.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/timer.c +1 -1
index 8137cce..96ac1b4 100644
@@ -1221,10 +1221,10 @@ void update_process_times(int user_tick)
 
       /* Note: this timer irq context must be accounted for as well. */
       account_process_tick(p, user_tick);
+       scheduler_tick();
       run_local_timers();
       if (rcu_pending(cpu))
               rcu_check_callbacks(cpu, user_tick);
-       scheduler_tick();
       run_posix_cpu_timers(p);
 }
 
Quoted 34 lines · Ingo Molnar
Quoted 22 lines · Jason Wessel
Quoted 6 lines · Ingo Molnar
Quoted 4 lines · Jason Wessel

-----
From: Jason Wessel <[email protected]>
Subject: [PATCH] softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume

Quoted 9 lines
Quoted 5 lines

When CONFIG_HAVE_UNSTABLE_SCHED_CLOCK is set sched_clock() gets the time from hardware, such as from TSC. In this configuration kgdb will report a softlock warning messages on resuming or detaching from a debug session.

Hm, this looks quite ugly. Peter, Thomas, can you think of a cleaner solution?

Below was a more specific test case I received from Dongdong Deng which did not require kgdb. The test case is not something to merge, it is just a simple module to build to demonstrate the problem outside of kgdb.

Quoted 4 lines

The patch I submitted might have been on the "ugly" side, but there was not an obvious way to solve the problem without making changes in kernel/sched*. I opted for something that was entirely self contained to the softlockup code.

it's not really the fault of your patch really - all these integrations along dynticks, scheduler time and softlockup feel a bit awkward. Wondering whether there's some nicer way.

Ingo

#8 lore raw
In reply to Ingo Molnar
Quoted 146 lines · Ingo Molnar
Quoted 143 lines · Jason Wessel
Quoted 19 lines · Peter Zijlstra
Quoted 2 lines · Jason Wessel

The fix is to simply invoke sched_clock_tick() to update "cpu sched clock" on exit from kgdb_handle_exception.

Is that a regular IRQ context, or is that NMI context?

Quoted 13 lines

---
 kernel/softlockup.c |    3 +++

 1 file changed, 3 insertions(+)
kernel/softlockup.c +0 -0
@@ -118,6 +118,9 @@ void softlockup_tick(void)
         }
 
Quoted 11 lines

kernel/softlockup.c void softlockup_tick(void) · line 126

Quoted 8 lines
kernel/softlockup.c +3 -0
         if (touch_timestamp == 0) {
+               /* If the time stamp was touched externally make sure the
+                * scheduler tick is up to date as well */
+               sched_clock_tick();
                 __touch_softlockup_watchdog();
                 return;
         }

Aside from the funny comment style (please fix) the fix does look sensible.

It turns out that further testing of this patch shows a regression in the ability to detect certain lockups.  It is a direct result of the way the scheduling code makes use of the touch_softlockup_watchdog(). With the above proposed patch the tick was getting updated after a resume, but was also getting updated with the run_timers(), and if that happened before the softlockup tick, no softlockup would get reported (note that I was using some test code to induce softlockups).

The patch below is a bit more invasive but solves the problem by allowing kgdb to request that the sched cpu clock is updated only when returning from a state where we know we need to force the update.

Would this change be an acceptable solution to allow kgdb to peacefully exist with the softlockup code?

Thanks,
Jason.

-----
From: Jason Wessel <[email protected]>
Subject: [PATCH] softlockup: add sched_clock_tick() to avoid kernel warning on kgdb resume

When CONFIG_HAVE_UNSTABLE_SCHED_CLOCK is set sched_clock() gets the time from hardware, such as from TSC.  In this configuration kgdb will report a softlock warning messages on resuming or detaching from a debug session.

Sequence of events in the problem case:

1) "cpu sched clock" and "hardware time" are at 100 sec prior to a call to kgdb_handle_exception()

2) Debugger waits in kgdb_handle_exception() for 80 sec and on exit the following is called ...  touch_softlockup_watchdog() --> __raw_get_cpu_var(touch_timestamp) = 0;

3) "cpu sched clock" = 100s (it was not updated, because the interrupt was disabled in kgdb) but the "hardware time" = 180 sec

4) The first timer interrupt after resuming from kgdb_handle_exception updates the watchdog from the "cpu sched clock"

update_process_times() { ...  run_local_timers() --> softlockup_tick() --> check (touch_timestamp == 0) (it is "YES" here, we have set "touch_timestamp = 0" at kgdb) --> __touch_softlockup_watchdog() ***(A)--> reset "touch_timestamp" to "get_timestamp()" (Here, the "touch_timestamp" will still be set to 100s.)  ...

    scheduler_tick() ***(B)--> sched_clock_tick() (update "cpu sched clock" to "hardware time" = 180s) ...  }

5) The Second timer interrupt handler appears to have a large jump and trips the softlockup warning.

update_process_times() { ...  run_local_timers() --> softlockup_tick() --> "cpu sched clock" - "touch_timestamp" = 180s-100s > 60s --> printk "soft lockup error messages" ...  }

note: ***(A) reset "touch_timestamp" to "get_timestamp(this_cpu)"

Why "touch_timestamp" is 100 sec, instead of 180 sec?

With the CONFIG_HAVE_UNSTABLE_SCHED_CLOCK" set the call trace of get_timestamp() is:

get_timestamp(this_cpu) -->cpu_clock(this_cpu) -->sched_clock_cpu(this_cpu) -->__update_sched_clock(sched_clock_data, now)

The __update_sched_clock() function uses the GTOD tick value to create a window to normalize the "now" values.  So if "now" values is too big for sched_clock_data, it will be ignored.

The fix is to invoke sched_clock_tick() to update "cpu sched clock" in order to recover from this state.  This is done by introducing the function touch_softlockup_watchdog_sync(), which allows kgdb to request that the sched clock is updated when the watchdog thread runs the first time after a resume from kgdb.

---
 include/linux/sched.h |    4 ++++
 kernel/kgdb.c         |    6 +++---
 kernel/softlockup.c   |   16 ++++++++++++++++

 3 files changed, 23 insertions(+), 3 deletions(-)
kernel/softlockup.c +8 -0
@@ -79,6 +79,14 @@ void touch_softlockup_watchdog(void)
  }
  EXPORT_SYMBOL(touch_softlockup_watchdog);
 
+static int softlock_touch_sync[NR_CPUS];
+
+void touch_softlockup_watchdog_sync(void)
+{
+     softlock_touch_sync[raw_smp_processor_id()] = 1;
+     __raw_get_cpu_var(touch_timestamp) = 0;
+}
+
  void touch_all_softlockup_watchdogs(void)
  {
       int cpu;
@@ -118,6 +126,14 @@ void softlockup_tick(void)
       }
 
Quoted 13 lines

kernel/softlockup.c void softlockup_tick(void) · line 136

Quoted 9 lines
kernel/softlockup.c +8 -0
       if (touch_timestamp == 0) {
+             if (unlikely(softlock_touch_sync[this_cpu])) {
+                     /*
+                      * If the time stamp was touched atomically
+                      * make sure the scheduler tick is up to date.
+                      */
+                     softlock_touch_sync[this_cpu] = 0;
+                     sched_clock_tick();
+             }

Hm, this looks quite ugly. Peter, Thomas, can you think of a cleaner solution?

We need not to sync up sched_clock in softlockup_tick, it can just return and sync up the timestamp in the next tick. This will not touch what should be done by dynticks, scheduler time and so on and keep softlockup_tick clean.

I have rewrote the patch based on above, please check the attachment.

Thanks,
Yong

Quoted 7 lines

       Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

From 1821356596bdd28dfc69f708a6929394da351001 Mon Sep 17 00:00:00 2001
From: Yong Zhang <[email protected]>
Date: Sat, 26 Sep 2009 10:37:14 +0800
Subject: [PATCH] softlockup: introduce touch_softlockup_watchdog_sync()

In case of system halted for a long time, system will do time sync up in the first incoming tick. But if we also touch softlockup_tick timestamp at the same time, softlockup_tick could keep a old timestamp and this will lead annoying softlockup warning. So move softlockup syncing it's timestamp to the next tick.

LKML-Reference: <http://lkml.org/lkml/2009/7/27/317>

---
 include/linux/sched.h |    4 ++++
 kernel/softlockup.c   |   18 ++++++++++++++++++
 2 files changed, 22 insertions(+), 0 deletions(-)
include/linux/sched.h +4 -0
index 75e6e60..1aa4574 100644
@@ -307,6 +307,7 @@ extern void sched_show_task(struct task_struct *p);
 #ifdef CONFIG_DETECT_SOFTLOCKUP
 extern void softlockup_tick(void);
 extern void touch_softlockup_watchdog(void);
+extern void touch_softlockup_watchdog_sync(void);
 extern void touch_all_softlockup_watchdogs(void);
 extern int proc_dosoftlockup_thresh(struct ctl_table *table, int write,
 				    void __user *buffer,
@@ -320,6 +321,9 @@ static inline void softlockup_tick(void)
 static inline void touch_softlockup_watchdog(void)
 {
 }
+static inline void touch_softlockup_watchdog_sync(void)
+{
+}
 static inline void touch_all_softlockup_watchdogs(void)
 {
 }
kernel/softlockup.c +18 -0
index 81324d1..628d531 100644
@@ -25,6 +25,7 @@ static DEFINE_SPINLOCK(print_lock);
 static DEFINE_PER_CPU(unsigned long, touch_timestamp);
 static DEFINE_PER_CPU(unsigned long, print_timestamp);
 static DEFINE_PER_CPU(struct task_struct *, watchdog_task);
+static DEFINE_PER_CPU(bool, softlock_touch_sync);
 
 static int __read_mostly did_panic;
 int __read_mostly softlockup_thresh = 60;
@@ -79,6 +80,13 @@ void touch_softlockup_watchdog(void)
 }
 EXPORT_SYMBOL(touch_softlockup_watchdog);
 
+void touch_softlockup_watchdog_sync(void)
+{
+	__raw_get_cpu_var(touch_timestamp) = 0;
+	__raw_get_cpu_var(softlock_touch_sync) = true;
+}
+EXPORT_SYMBOL(touch_softlockup_watchdog_sync);
+
 void touch_all_softlockup_watchdogs(void)
 {
 	int cpu;
@@ -117,6 +125,15 @@ void softlockup_tick(void)
 		return;
 	}
 
+	/*
+	 * If the scheduler tick is not up to date. Skip this check
+	 * and wait for the timestamp to to synced in the next tick
+	 */
+	if (unlikely(per_cpu(softlock_touch_sync, this_cpu))) {
+		per_cpu(softlock_touch_sync, this_cpu) = false;
+		return;
+	}
+
 	if (touch_timestamp == 0) {
 		__touch_softlockup_watchdog();
 		return;
@@ -216,6 +233,7 @@ cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu)
 			return NOTIFY_BAD;
 		}
 		per_cpu(touch_timestamp, hotcpu) = 0;
+		per_cpu(softlock_touch_sync, hotcpu) = false;
 		per_cpu(watchdog_task, hotcpu) = p;
 		kthread_bind(p, hotcpu);
 		break;
Signature · 1 line
1.6.0.4