Skip to contentSkip to conversation

softlockup: do the wakeup from a hrtimer

PATCH In review Ingo Molnar 2 participants 4 messages Nov 20, 2007 – Nov 28, 2007 Nohz, dynticks support

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 softlockup: do the wakeup from a hrtimer

Subject: softlockup: do the wakeup from a hrtimer
From: Ingo Molnar <[email protected]>

David Miller reported soft lockup false-positives that trigger on NOHZ due to CPUs idling for more than 10 seconds.

The solution is to drive the wakeup of the watchdog threads not from the timer tick (which has no guaranteed frequency), but from the watchdog tasks themselves.

http://bugzilla.kernel.org/show_bug.cgi?id=9409

---
 kernel/softlockup.c |    6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)
kernel/softlockup.c +1 -5
Index: linux/kernel/softlockup.c
===================================================================
@@ -100,10 +100,6 @@ void softlockup_tick(void)
 
 	now = get_timestamp(this_cpu);
 
-	/* Wake up the high-prio watchdog task every second: */
-	if (now > (touch_timestamp + 1))
-		wake_up_process(per_cpu(watchdog_task, this_cpu));
-
 	/* Warn about unreasonable 10+ seconds delays: */
 	if (now <= (touch_timestamp + softlockup_thresh))
 		return;
@@ -141,7 +137,7 @@ static int watchdog(void *__bind_cpu)
 	while (!kthread_should_stop()) {
 		set_current_state(TASK_INTERRUPTIBLE);
 		touch_softlockup_watchdog();
-		schedule();
+		msleep(1000);
 	}
 
 	return 0;
#1 lore raw
Quoted 34 lines · On Tue, 20 Nov 2007 09:46:11 +0100 Ingo Molnar <[email protected]> wrote:

Subject: softlockup: do the wakeup from a hrtimer
From: Ingo Molnar <[email protected]>

David Miller reported soft lockup false-positives that trigger on NOHZ due to CPUs idling for more than 10 seconds.

The solution is to drive the wakeup of the watchdog threads not from the timer tick (which has no guaranteed frequency), but from the watchdog tasks themselves.

http://bugzilla.kernel.org/show_bug.cgi?id=9409

---
 kernel/softlockup.c |    6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

Index: linux/kernel/softlockup.c
===================================================================

kernel/softlockup.c +0 -4
@@ -100,10 +100,6 @@ void softlockup_tick(void)
 
 	now = get_timestamp(this_cpu);
 
-	/* Wake up the high-prio watchdog task every second: */
-	if (now > (touch_timestamp + 1))
-		wake_up_process(per_cpu(watchdog_task, this_cpu));
-
 	/* Warn about unreasonable 10+ seconds delays: */
 	if (now <= (touch_timestamp + softlockup_thresh))
 		return;

kernel/softlockup.c static int watchdog(void *__bind_cpu) · line 143

Quoted 8 lines
kernel/softlockup.c +1 -1
@@ -141,7 +137,7 @@ static int watchdog(void *__bind_cpu)
 	while (!kthread_should_stop()) {
 		set_current_state(TASK_INTERRUPTIBLE);
 		touch_softlockup_watchdog();
-		schedule();
+		msleep(1000);
 	}
 
 	return 0;

I think you wanted msleep_interruptible() there to avoid contributing to load average?

The set_current_state() can go away.

This will introduce an up-to-one-second delay in responding to kthread_should_stop(). Is that bad?

#2 lore raw
Quoted 44 lines · Andrew Morton
Quoted 41 lines · On Tue, 20 Nov 2007 09:46:11 +0100 Ingo Molnar <[email protected]> wrote:

Subject: softlockup: do the wakeup from a hrtimer
From: Ingo Molnar <[email protected]>

David Miller reported soft lockup false-positives that trigger on NOHZ due to CPUs idling for more than 10 seconds.

The solution is to drive the wakeup of the watchdog threads not from the timer tick (which has no guaranteed frequency), but from the watchdog tasks themselves.

http://bugzilla.kernel.org/show_bug.cgi?id=9409

---
 kernel/softlockup.c |    6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

Index: linux/kernel/softlockup.c
===================================================================

kernel/softlockup.c +1 -5
@@ -100,10 +100,6 @@ void softlockup_tick(void)
 
 	now = get_timestamp(this_cpu);
 
-	/* Wake up the high-prio watchdog task every second: */
-	if (now > (touch_timestamp + 1))
-		wake_up_process(per_cpu(watchdog_task, this_cpu));
-
 	/* Warn about unreasonable 10+ seconds delays: */
 	if (now <= (touch_timestamp + softlockup_thresh))
 		return;
@@ -141,7 +137,7 @@ static int watchdog(void *__bind_cpu)
 	while (!kthread_should_stop()) {
 		set_current_state(TASK_INTERRUPTIBLE);
 		touch_softlockup_watchdog();
-		schedule();
+		msleep(1000);
 	}
 
Quoted 6 lines

kernel/softlockup.c static int watchdog(void *__bind_cpu) · line 143

Quoted 1 line
kernel/softlockup.c +0 -0
 	return 0;

I think you wanted msleep_interruptible() there to avoid contributing to load average?

The set_current_state() can go away.

these can be fixed, but:

Quoted 2 lines

This will introduce an up-to-one-second delay in responding to kthread_should_stop(). Is that bad?

grumble, it's bad. I guess David is right that this should be fixed the right way ;-) So the above patch cannot go in.

Ingo

#3 lore raw

PATCH 1/1 softlockup: fix false positives on CONFIG_NOHZ

Quoted 7 lines · Ingo Molnar

these can be fixed, but:

Quoted 2 lines

This will introduce an up-to-one-second delay in responding to kthread_should_stop(). Is that bad?

grumble, it's bad. I guess David is right that this should be fixed the right way ;-) So the above patch cannot go in.

Thomas found the right fix. David, could you try the fix below, does it fix those false positives on your nohz Niagara cores?

Ingo

---------------->
Subject: softlockup: fix false positives on CONFIG_NOHZ
From: Thomas Gleixner <[email protected]>

David Miller reported soft lockup false-positives that trigger on NOHZ due to CPUs idling for more than 10 seconds.

The solution is touch the softlockup watchdog when we return from idle. (by definition we are not 'locked up' when we were idle)

http://bugzilla.kernel.org/show_bug.cgi?id=9409

---
 kernel/time/tick-sched.c |    2 ++
 1 file changed, 2 insertions(+)
kernel/time/tick-sched.c +2 -0
Index: linux/kernel/time/tick-sched.c
===================================================================
@@ -133,6 +133,8 @@ void tick_nohz_update_jiffies(void)
 	if (!ts->tick_stopped)
 		return;
 
+	touch_softlockup_watchdog();
+
 	cpu_clear(cpu, nohz_cpu_mask);
 	now = ktime_get();