Skip to contentSkip to conversation

2.6.24-git3: Major annoyance during suspend/hibernation on x86-64 (bisected)

Regression Rafael J. Wysocki 5 participants 27 messages Jan 27, 2008 – Feb 2, 2008

lore
Share & tools
lore
#0 lore raw

Hi,

2.6.24-git3 adds a 5 - 10 sec delay to the suspend and hibernation code paths (probably related to the disabling of nonboot CPUs), which is !@#$%^&*() annoying.

It's 100% reproducible on my HP nx6325 and bisection idendified the following commit as the first bad one:

commit 764a9d6fe4b52995c8aba277e3634385699354f4

Author: Steven Rostedt <[email protected]>
Date:   Fri Jan 25 21:08:04 2008 +0100

sched: track highest prio task queued

Greetings,
Rafael

#1 lore raw
Quoted 6 lines · Rafael J. Wysocki

Hi,

2.6.24-git3 adds a 5 - 10 sec delay to the suspend and hibernation code paths (probably related to the disabling of nonboot CPUs), which is !@#$%^&*() annoying.

Quoted 8 lines

It's 100% reproducible on my HP nx6325 and bisection idendified the following commit as the first bad one:

commit 764a9d6fe4b52995c8aba277e3634385699354f4

Author: Steven Rostedt <[email protected]>
Date:   Fri Jan 25 21:08:04 2008 +0100

sched: track highest prio task queued

hm, this patch is a NOP, so it's weird that it has an effect.

Do you have serial logging enabled perhaps? If the following WARN_ON() triggers:

+ WARN_ON(p->prio < rq->rt.highest_prio);

then perhaps that can cause a 5-10 seconds delay. (that's how much time it takes to printk a warning on the slowest serial settings)

but if you use suspend, then any such printks would be preserved in the dmesg, right? If the WARN_ON() triggers, and if you remove it, do things get faster?

this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch? It's not the next patch or the preceding patch by any chance? (which would also be scheduler patches)

Ingo

#2 lore raw
Quoted 23 lines · Ingo Molnar
Quoted 6 lines · Rafael J. Wysocki

Hi,

2.6.24-git3 adds a 5 - 10 sec delay to the suspend and hibernation code paths (probably related to the disabling of nonboot CPUs), which is !@#$%^&*() annoying.

Quoted 8 lines

It's 100% reproducible on my HP nx6325 and bisection idendified the following commit as the first bad one:

commit 764a9d6fe4b52995c8aba277e3634385699354f4

Author: Steven Rostedt <[email protected]>
Date:   Fri Jan 25 21:08:04 2008 +0100

sched: track highest prio task queued

hm, this patch is a NOP, so it's weird that it has an effect.

Do you have serial logging enabled perhaps? If the following WARN_ON() triggers:

Quoted 8 lines

+ WARN_ON(p->prio < rq->rt.highest_prio);

then perhaps that can cause a 5-10 seconds delay. (that's how much time it takes to printk a warning on the slowest serial settings)

but if you use suspend, then any such printks would be preserved in the dmesg, right? If the WARN_ON() triggers, and if you remove it, do things get faster?

No, this isn't the WARN_ON().

Quoted 2 lines

this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?

Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:

commit 82a1fcb90287052aabfa235e7ffc693ea003fe69

Author: Ingo Molnar <[email protected]>
Date:   Fri Jan 25 21:08:02 2008 +0100

softlockup: automatically detect hung TASK_UNINTERRUPTIBLE tasks

Reverting this commit (it reverts with some minor modifications) fixes the problem for me.

Thanks,
Rafael

#3 lore raw
Quoted 4 lines · Rafael J. Wysocki

No, this isn't the WARN_ON().

Quoted 1 line

this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?

Quoted 8 lines

Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:

commit 82a1fcb90287052aabfa235e7ffc693ea003fe69

Author: Ingo Molnar <[email protected]>
Date:   Fri Jan 25 21:08:02 2008 +0100

softlockup: automatically detect hung TASK_UNINTERRUPTIBLE tasks

Are you getting a bunch of prints from the softlockup detector in dmesg?
I wonder if the detector can detect a long timeout caused by suspend and resume and if not is triggering false positives?

-- Steve

Quoted 4 lines

Reverting this commit (it reverts with some minor modifications) fixes the problem for me.

#8 lore raw
In reply to Rafael J. Wysocki
Quoted 41 lines · Rafael J. Wysocki
Quoted 23 lines · Ingo Molnar
Quoted 6 lines · Rafael J. Wysocki

Hi,

2.6.24-git3 adds a 5 - 10 sec delay to the suspend and hibernation code paths (probably related to the disabling of nonboot CPUs), which is !@#$%^&*() annoying.

Quoted 8 lines

It's 100% reproducible on my HP nx6325 and bisection idendified the following commit as the first bad one:

commit 764a9d6fe4b52995c8aba277e3634385699354f4

Author: Steven Rostedt <[email protected]>
Date:   Fri Jan 25 21:08:04 2008 +0100

sched: track highest prio task queued

hm, this patch is a NOP, so it's weird that it has an effect.

Do you have serial logging enabled perhaps? If the following WARN_ON() triggers:

Quoted 8 lines

+ WARN_ON(p->prio < rq->rt.highest_prio);

then perhaps that can cause a 5-10 seconds delay. (that's how much time it takes to printk a warning on the slowest serial settings)

but if you use suspend, then any such printks would be preserved in the dmesg, right? If the WARN_ON() triggers, and if you remove it, do things get faster?

No, this isn't the WARN_ON().

Quoted 2 lines

this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?

Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:

Quoted 8 lines

commit 82a1fcb90287052aabfa235e7ffc693ea003fe69

Author: Ingo Molnar <[email protected]>
Date:   Fri Jan 25 21:08:02 2008 +0100

softlockup: automatically detect hung TASK_UNINTERRUPTIBLE tasks

Reverting this commit (it reverts with some minor modifications) fixes the problem for me.

What if you use the same kernel that triggers a problem and just disable this new 'softlockup' functionality:

echo 0 > /proc/sys/kernel/hung_task_timeout_secs

does the problem disapear?

TIA,

Quoted 4 lines

Thanks,
Rafael

Signature · 2 lines
Best regards,
Dmitry Adamushko
#4 lore raw
In reply to Steven Rostedt
Quoted 10 lines · Steven Rostedt
Quoted 8 lines · Rafael J. Wysocki

No, this isn't the WARN_ON().

Quoted 2 lines

this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?

Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:

Quoted 7 lines
Quoted 5 lines

commit 82a1fcb90287052aabfa235e7ffc693ea003fe69

Author: Ingo Molnar <[email protected]>
Date:   Fri Jan 25 21:08:02 2008 +0100

softlockup: automatically detect hung TASK_UNINTERRUPTIBLE tasks

Are you getting a bunch of prints from the softlockup detector in dmesg?

No, I don't. In fact, I don't get _any_ messages from it whatsoever.

Quoted 2 lines

I wonder if the detector can detect a long timeout caused by suspend and resume and if not is triggering false positives?

I'm not sure, but the code is supposed to be suspend-aware, IIRC. However, I'm seeing a similar symptom on poweroff on an SMP x86-64 box, so it may be more directly related to the CPU hotplug. I'll try to verify that.

Thanks,
Rafael

#9 lore raw
In reply to Dmitry Adamushko
Quoted 51 lines · Dmitry Adamushko
Quoted 41 lines · Rafael J. Wysocki
Quoted 23 lines · Ingo Molnar
Quoted 6 lines · Rafael J. Wysocki

Hi,

2.6.24-git3 adds a 5 - 10 sec delay to the suspend and hibernation code paths (probably related to the disabling of nonboot CPUs), which is !@#$%^&*() annoying.

Quoted 8 lines

It's 100% reproducible on my HP nx6325 and bisection idendified the following commit as the first bad one:

commit 764a9d6fe4b52995c8aba277e3634385699354f4

Author: Steven Rostedt <[email protected]>
Date:   Fri Jan 25 21:08:04 2008 +0100

sched: track highest prio task queued

hm, this patch is a NOP, so it's weird that it has an effect.

Do you have serial logging enabled perhaps? If the following WARN_ON() triggers:

Quoted 8 lines

+ WARN_ON(p->prio < rq->rt.highest_prio);

then perhaps that can cause a 5-10 seconds delay. (that's how much time it takes to printk a warning on the slowest serial settings)

but if you use suspend, then any such printks would be preserved in the dmesg, right? If the WARN_ON() triggers, and if you remove it, do things get faster?

No, this isn't the WARN_ON().

Quoted 2 lines

this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?

Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:

Quoted 8 lines

commit 82a1fcb90287052aabfa235e7ffc693ea003fe69

Author: Ingo Molnar <[email protected]>
Date:   Fri Jan 25 21:08:02 2008 +0100

softlockup: automatically detect hung TASK_UNINTERRUPTIBLE tasks

Reverting this commit (it reverts with some minor modifications) fixes the problem for me.

Quoted 6 lines

What if you use the same kernel that triggers a problem and just disable this new 'softlockup' functionality:

echo 0 > /proc/sys/kernel/hung_task_timeout_secs

does the problem disapear?

I haven't checked that yet, will do it later today.

Thanks,
Rafael

#10 lore raw
In reply to Dmitry Adamushko
Quoted 51 lines · Dmitry Adamushko
Quoted 41 lines · Rafael J. Wysocki
Quoted 23 lines · Ingo Molnar
Quoted 6 lines · Rafael J. Wysocki

Hi,

2.6.24-git3 adds a 5 - 10 sec delay to the suspend and hibernation code paths (probably related to the disabling of nonboot CPUs), which is !@#$%^&*() annoying.

Quoted 8 lines

It's 100% reproducible on my HP nx6325 and bisection idendified the following commit as the first bad one:

commit 764a9d6fe4b52995c8aba277e3634385699354f4

Author: Steven Rostedt <[email protected]>
Date:   Fri Jan 25 21:08:04 2008 +0100

sched: track highest prio task queued

hm, this patch is a NOP, so it's weird that it has an effect.

Do you have serial logging enabled perhaps? If the following WARN_ON() triggers:

Quoted 8 lines

+ WARN_ON(p->prio < rq->rt.highest_prio);

then perhaps that can cause a 5-10 seconds delay. (that's how much time it takes to printk a warning on the slowest serial settings)

but if you use suspend, then any such printks would be preserved in the dmesg, right? If the WARN_ON() triggers, and if you remove it, do things get faster?

No, this isn't the WARN_ON().

Quoted 2 lines

this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?

Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:

Quoted 8 lines

commit 82a1fcb90287052aabfa235e7ffc693ea003fe69

Author: Ingo Molnar <[email protected]>
Date:   Fri Jan 25 21:08:02 2008 +0100

softlockup: automatically detect hung TASK_UNINTERRUPTIBLE tasks

Reverting this commit (it reverts with some minor modifications) fixes the problem for me.

Quoted 6 lines

What if you use the same kernel that triggers a problem and just disable this new 'softlockup' functionality:

echo 0 > /proc/sys/kernel/hung_task_timeout_secs

does the problem disapear?

No, it doesn't. The setting doesn't seem to have any effect on it.

Thanks,
Rafael

#5 lore raw
In reply to Rafael J. Wysocki
Quoted 19 lines · Rafael J. Wysocki
Quoted 10 lines · Steven Rostedt
Quoted 8 lines · Rafael J. Wysocki

No, this isn't the WARN_ON().

Quoted 2 lines

this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?

Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:

Quoted 7 lines
Quoted 5 lines

commit 82a1fcb90287052aabfa235e7ffc693ea003fe69

Author: Ingo Molnar <[email protected]>
Date:   Fri Jan 25 21:08:02 2008 +0100

softlockup: automatically detect hung TASK_UNINTERRUPTIBLE tasks

Are you getting a bunch of prints from the softlockup detector in dmesg?

Quoted 8 lines

No, I don't. In fact, I don't get _any_ messages from it whatsoever.

Quoted 2 lines

I wonder if the detector can detect a long timeout caused by suspend and resume and if not is triggering false positives?

I'm not sure, but the code is supposed to be suspend-aware, IIRC. However, I'm seeing a similar symptom on poweroff on an SMP x86-64 box, so it may be more directly related to the CPU hotplug. I'll try to verify that.

As I expected, the delay is also observable when I do:

echo 0 > /sys/devices/system/cpu/cpu1/online

(it's variable, between 3 and 30 seconds). Again, no messages appear in dmesg when this happens.

I suspect I'll be able to reproduce it on another x86-64 SMP machine (I'm going to try that later today).

Thanks,
Rafael

#6 lore raw
Quoted 25 lines · Rafael J. Wysocki
Quoted 13 lines · Rafael J. Wysocki
Quoted 12 lines · Steven Rostedt
Quoted 10 lines · Rafael J. Wysocki

No, this isn't the WARN_ON().

Quoted 1 line

this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?

Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:

commit 82a1fcb90287052aabfa235e7ffc693ea003fe69

Author: Ingo Molnar <[email protected]>
Date:   Fri Jan 25 21:08:02 2008 +0100
Quoted 8 lines
Quoted 2 lines
Quoted 1 line

softlockup: automatically detect hung TASK_UNINTERRUPTIBLE tasks

Are you getting a bunch of prints from the softlockup detector in dmesg?

No, I don't. In fact, I don't get _any_ messages from it whatsoever.

Quoted 1 line

I wonder if the detector can detect a long timeout caused by suspend and resume and if not is triggering false positives?

I'm not sure, but the code is supposed to be suspend-aware, IIRC. However, I'm seeing a similar symptom on poweroff on an SMP x86-64 box, so it may be more directly related to the CPU hotplug. I'll try to verify that.

As I expected, the delay is also observable when I do:

Quoted 7 lines

echo 0 > /sys/devices/system/cpu/cpu1/online

(it's variable, between 3 and 30 seconds). Again, no messages appear in dmesg when this happens.

I suspect I'll be able to reproduce it on another x86-64 SMP machine (I'm going to try that later today).

Could you download my tracing patches from here:

http://people.redhat.com/srostedt/tracing/mcount-tracing-patches-v6.tar.bz2

Apply the above patches and select all but the histogram tracers. Then run this program

http://people.redhat.com/srostedt/tracing/trace-cmd.c

./trace-cmd -f echo 0 > /sys/devices/system/cpu/cpu1/online

then do this (mount debugfs):

bzip2 -c /debugfs/tracing/latency_trace > /tmp/trace.bz2

and send the result to myself and Ingo.

Thanks,

-- Steve

Quoted 3 lines

Thanks,
Rafael

#7 lore raw
In reply to Rafael J. Wysocki
Quoted 31 lines · Rafael J. Wysocki
Quoted 19 lines · Rafael J. Wysocki
Quoted 10 lines · Steven Rostedt
Quoted 8 lines · Rafael J. Wysocki

No, this isn't the WARN_ON().

Quoted 2 lines

this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?

Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:

Quoted 7 lines
Quoted 5 lines

commit 82a1fcb90287052aabfa235e7ffc693ea003fe69

Author: Ingo Molnar <[email protected]>
Date:   Fri Jan 25 21:08:02 2008 +0100

softlockup: automatically detect hung TASK_UNINTERRUPTIBLE tasks

Are you getting a bunch of prints from the softlockup detector in dmesg?

Quoted 8 lines

No, I don't. In fact, I don't get _any_ messages from it whatsoever.

Quoted 2 lines

I wonder if the detector can detect a long timeout caused by suspend and resume and if not is triggering false positives?

I'm not sure, but the code is supposed to be suspend-aware, IIRC. However, I'm seeing a similar symptom on poweroff on an SMP x86-64 box, so it may be more directly related to the CPU hotplug. I'll try to verify that.

As I expected, the delay is also observable when I do:

Quoted 7 lines

echo 0 > /sys/devices/system/cpu/cpu1/online

(it's variable, between 3 and 30 seconds). Again, no messages appear in dmesg when this happens.

I suspect I'll be able to reproduce it on another x86-64 SMP machine (I'm going to try that later today).

FWIW, the problem is reproducible on the second machine, with the same symptoms. Both machines are AMD-based.

Thanks,
Rafael

#11 lore raw
In reply to Rafael J. Wysocki
Quoted 41 lines · Rafael J. Wysocki
Quoted 23 lines · Ingo Molnar
Quoted 6 lines · Rafael J. Wysocki

Hi,

2.6.24-git3 adds a 5 - 10 sec delay to the suspend and hibernation code paths (probably related to the disabling of nonboot CPUs), which is !@#$%^&*() annoying.

Quoted 8 lines

It's 100% reproducible on my HP nx6325 and bisection idendified the following commit as the first bad one:

commit 764a9d6fe4b52995c8aba277e3634385699354f4

Author: Steven Rostedt <[email protected]>
Date:   Fri Jan 25 21:08:04 2008 +0100

sched: track highest prio task queued

hm, this patch is a NOP, so it's weird that it has an effect.

Do you have serial logging enabled perhaps? If the following WARN_ON() triggers:

Quoted 8 lines

+ WARN_ON(p->prio < rq->rt.highest_prio);

then perhaps that can cause a 5-10 seconds delay. (that's how much time it takes to printk a warning on the slowest serial settings)

but if you use suspend, then any such printks would be preserved in the dmesg, right? If the WARN_ON() triggers, and if you remove it, do things get faster?

No, this isn't the WARN_ON().

Quoted 2 lines

this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?

Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:

Quoted 8 lines

commit 82a1fcb90287052aabfa235e7ffc693ea003fe69

Author: Ingo Molnar <[email protected]>
Date:   Fri Jan 25 21:08:02 2008 +0100

softlockup: automatically detect hung TASK_UNINTERRUPTIBLE tasks

Reverting this commit (it reverts with some minor modifications) fixes the problem for me.

I can seem to reproduce this:

[root@opteron cpu1]# time echo 0 > online

real    0m6.230s
user    0m0.000s
sys     0m0.010s

[root@opteron cpu1]# echo 1 > online
[root@opteron cpu1]# time echo 0 > online

real    0m7.966s
user    0m0.000s
sys     0m0.011s

I'll have a look at it.

#12 lore raw
Quoted 62 lines · Peter Zijlstra
Quoted 41 lines · Rafael J. Wysocki
Quoted 23 lines · Ingo Molnar
Quoted 6 lines · Rafael J. Wysocki

Hi,

2.6.24-git3 adds a 5 - 10 sec delay to the suspend and hibernation code paths (probably related to the disabling of nonboot CPUs), which is !@#$%^&*() annoying.

Quoted 8 lines

It's 100% reproducible on my HP nx6325 and bisection idendified the following commit as the first bad one:

commit 764a9d6fe4b52995c8aba277e3634385699354f4

Author: Steven Rostedt <[email protected]>
Date:   Fri Jan 25 21:08:04 2008 +0100

sched: track highest prio task queued

hm, this patch is a NOP, so it's weird that it has an effect.

Do you have serial logging enabled perhaps? If the following WARN_ON() triggers:

Quoted 8 lines

+ WARN_ON(p->prio < rq->rt.highest_prio);

then perhaps that can cause a 5-10 seconds delay. (that's how much time it takes to printk a warning on the slowest serial settings)

but if you use suspend, then any such printks would be preserved in the dmesg, right? If the WARN_ON() triggers, and if you remove it, do things get faster?

No, this isn't the WARN_ON().

Quoted 2 lines

this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?

Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:

Quoted 8 lines

commit 82a1fcb90287052aabfa235e7ffc693ea003fe69

Author: Ingo Molnar <[email protected]>
Date:   Fri Jan 25 21:08:02 2008 +0100

softlockup: automatically detect hung TASK_UNINTERRUPTIBLE tasks

Reverting this commit (it reverts with some minor modifications) fixes the problem for me.

I can seem to reproduce this:

[root@opteron cpu1]# time echo 0 > online

real    0m6.230s
user    0m0.000s
sys     0m0.010s

[root@opteron cpu1]# echo 1 > online
[root@opteron cpu1]# time echo 0 > online

Quoted 6 lines
real    0m7.966s
user    0m0.000s
sys     0m0.011s

I'll have a look at it.

Much appreciated, thanks!

#13 lore raw
Quoted 12 lines · Rafael J. Wysocki
Quoted 10 lines · Peter Zijlstra

I can seem to reproduce this:

[root@opteron cpu1]# time echo 0 > online

real    0m6.230s
user    0m0.000s
sys     0m0.010s

[root@opteron cpu1]# echo 1 > online
[root@opteron cpu1]# time echo 0 > online

Quoted 8 lines
Quoted 6 lines
real    0m7.966s
user    0m0.000s
sys     0m0.011s

I'll have a look at it.

Much appreciated, thanks!

The below fixes it for me..

---
- restore the old wakeup mechanism
- fix break usage in do_each_thread() { } while_eac_thread().
- fix the hotplug switch stmt, a fall-through case was broken.

---

kernel/softlockup.c +20 -10
index c1d7655..7c2da88 100644
@@ -101,6 +101,10 @@ void softlockup_tick(void)
 
 	now = get_timestamp(this_cpu);
 
+	/* Wake up the high-prio watchdog task every second: */
+	if (now > (touch_timestamp + 1))
+		wake_up_process(per_cpu(watchdog_task, this_cpu));
+
 	/* Warn about unreasonable delays: */
 	if (now <= (touch_timestamp + softlockup_thresh))
 		return;
@@ -191,11 +195,11 @@ static void check_hung_uninterruptible_tasks(int this_cpu)
 	read_lock(&tasklist_lock);
 	do_each_thread(g, t) {
 		if (!--max_count)
-			break;
+			goto unlock;
 		if (t->state & TASK_UNINTERRUPTIBLE)
 			check_hung_task(t, now);
 	} while_each_thread(g, t);
-
+ unlock:
 	read_unlock(&tasklist_lock);
 }
 
@@ -218,14 +222,19 @@ static int watchdog(void *__bind_cpu)
 	 * debug-printout triggers in softlockup_tick().
 	 */
 	while (!kthread_should_stop()) {
+		set_current_state(TASK_INTERRUPTIBLE);
 		touch_softlockup_watchdog();
-		msleep_interruptible(10000);
+		schedule();
+
+		if (kthread_should_stop())
+			break;
 
 		if (this_cpu != check_cpu)
 			continue;
 
 		if (sysctl_hung_task_timeout_secs)
 			check_hung_uninterruptible_tasks(this_cpu);
+
 	}
 
 	return 0;
@@ -259,13 +268,6 @@ cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu)
 		wake_up_process(per_cpu(watchdog_task, hotcpu));
 		break;
 #ifdef CONFIG_HOTPLUG_CPU
-	case CPU_UP_CANCELED:
-	case CPU_UP_CANCELED_FROZEN:
-		if (!per_cpu(watchdog_task, hotcpu))
-			break;
-		/* Unbind so it can run.  Fall thru. */
-		kthread_bind(per_cpu(watchdog_task, hotcpu),
-			     any_online_cpu(cpu_online_map));
 	case CPU_DOWN_PREPARE:
 	case CPU_DOWN_PREPARE_FROZEN:
 		if (hotcpu == check_cpu) {
@@ -275,6 +277,14 @@ cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu)
 			check_cpu = any_online_cpu(temp_cpu_online_map);
 		}
 		break;
+
+	case CPU_UP_CANCELED:
+	case CPU_UP_CANCELED_FROZEN:
+		if (!per_cpu(watchdog_task, hotcpu))
+			break;
+		/* Unbind so it can run.  Fall thru. */
+		kthread_bind(per_cpu(watchdog_task, hotcpu),
+			     any_online_cpu(cpu_online_map));
 	case CPU_DEAD:
 	case CPU_DEAD_FROZEN:
 		p = per_cpu(watchdog_task, hotcpu);
#14 lore raw
Quoted 6 lines · Peter Zijlstra

The below fixes it for me..

---
- restore the old wakeup mechanism
- fix break usage in do_each_thread() { } while_eac_thread().
- fix the hotplug switch stmt, a fall-through case was broken.

thanks Peter, i've picked your fix up. I hope this solves the problem for Rafael too.

Ingo

#15 lore raw
Quoted 5 lines · Ingo Molnar
Quoted 2 lines · Peter Zijlstra

The below fixes it for me..

Quoted 7 lines
Quoted 4 lines

---
- restore the old wakeup mechanism
- fix break usage in do_each_thread() { } while_eac_thread().
- fix the hotplug switch stmt, a fall-through case was broken.

thanks Peter, i've picked your fix up. I hope this solves the problem for Rafael too.

No, it doesn't, unfortunately. Actually, it breaks things even more, as suspend now hangs (probably deadlocks) while disabling the nonboot CPU.

Thanks,
Rafael

#18 lore raw
In reply to Peter Zijlstra
Quoted 2 lines · Peter Zijlstra

---
- restore the old wakeup mechanism

and how does it change behavior, logically-wise?

do we somehow miss a 'wake-up' from kthread_stop() so that its caller gets blocked until watchdog's msleep_interruptible(10000) timeouts? On average, it would take +-5 sec. and might explain the first observation of Ravael -- "...adds a 5 - 10 sec delay..." (although, lately he reported up to +30 sec. delays).

(/me goint to also try reproducing it later today)

Quoted 1 line

[ ... ]

Signature · 2 lines
Best regards,
Dmitry Adamushko
#19 lore raw
Quoted 7 lines · Dmitry Adamushko
Quoted 2 lines · Peter Zijlstra

---
- restore the old wakeup mechanism

and how does it change behavior, logically-wise?

Quoted 7 lines

do we somehow miss a 'wake-up' from kthread_stop() so that its caller gets blocked until watchdog's msleep_interruptible(10000) timeouts? On average, it would take +-5 sec. and might explain the first observation of Ravael -- "...adds a 5 - 10 sec delay..." (although, lately he reported up to +30 sec. delays).

(/me goint to also try reproducing it later today)

thanks - i cannot reproduce it on my usual suspend/resume testbox because e1000 broke on it, and this is a pretty annoying regression. We'll have to undo the hung-tasks detection feature if it's not fixed quickly. (there's no point in debugging features that _add_ bugs)

Ingo

#20 lore raw
Quoted 5 lines · Ingo Molnar

thanks - i cannot reproduce it on my usual suspend/resume testbox because e1000 broke on it, and this is a pretty annoying regression. We'll have to undo the hung-tasks detection feature if it's not fixed quickly. (there's no point in debugging features that _add_ bugs)

I can reproduce it and Peter's patch does fix it for me.

e.g.

root@earth:/sys/devices/system/cpu/cpu1# time echo 0 > online
real    0m6.743s
root@earth:/sys/devices/system/cpu/cpu1# time echo 0 > online
real    0m7.770s

I've observed delays from ~3 s. up to ~8 s. (out of ~20 tests) so the 10s. delay of msleep_interruptible() might be related but I'm still looking for the reason why this fix helps (and what goes wrong with the current code).

Quoted 3 lines

Ingo

Signature · 2 lines
Best regards,
Dmitry Adamushko
#21 lore raw
Quoted 15 lines · Dmitry Adamushko
Quoted 5 lines · Ingo Molnar

thanks - i cannot reproduce it on my usual suspend/resume testbox because e1000 broke on it, and this is a pretty annoying regression. We'll have to undo the hung-tasks detection feature if it's not fixed quickly. (there's no point in debugging features that _add_ bugs)

I can reproduce it and Peter's patch does fix it for me.

e.g.

root@earth:/sys/devices/system/cpu/cpu1# time echo 0 > online
real    0m6.743s
root@earth:/sys/devices/system/cpu/cpu1# time echo 0 > online
real    0m7.770s
Quoted 5 lines

I've observed delays from ~3 s. up to ~8 s. (out of ~20 tests) so the 10s. delay of msleep_interruptible() might be related but I'm still looking for the reason why this fix helps (and what goes wrong with the current code).

heh... it's pretty obvious indeed. What's msleep_interruptible() is all about? :-)

"sleep waiting for signals"

so the 'watchdog' thread gets woken up

[ as a result of cpu_callback(action = CPU_DEAD) --> kthread_stop() ]

just to be immediately scheduled out again for as long as the remaining timeout > 0.

So it should work if we substitute msleep_interruptible() with schedule_timeout_interruptible().

Signature · 2 lines
Best regards,
Dmitry Adamushko
#22 lore raw
Quoted 12 lines · Dmitry Adamushko
Quoted 4 lines

I've observed delays from ~3 s. up to ~8 s. (out of ~20 tests) so the 10s. delay of msleep_interruptible() might be related but I'm still looking for the reason why this fix helps (and what goes wrong with the current code).

heh... it's pretty obvious indeed. What's msleep_interruptible() is all about? :-)

"sleep waiting for signals"

so the 'watchdog' thread gets woken up

Quoted 7 lines

[ as a result of cpu_callback(action = CPU_DEAD) --> kthread_stop() ]

just to be immediately scheduled out again for as long as the remaining timeout > 0.

So it should work if we substitute msleep_interruptible() with schedule_timeout_interruptible().

Doh. Could someone with ths problem please test the patch below, does it do the trick?

Ingo

---
 kernel/softlockup.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/softlockup.c +1 -1
Index: linux/kernel/softlockup.c
===================================================================
@@ -228,7 +228,7 @@ static int watchdog(void *__bind_cpu)
 	 */
 	while (!kthread_should_stop()) {
 		touch_softlockup_watchdog();
-		msleep_interruptible(10000);
+		schedule_timeout_interruptible(10*HZ);
 
 		if (this_cpu != check_cpu)
 			continue;
author Tested-by
#16 lore raw
In reply to Rafael J. Wysocki
Quoted 11 lines · Rafael J. Wysocki
Quoted 10 lines · Ingo Molnar
Quoted 6 lines · Peter Zijlstra

The below fixes it for me..

---
- restore the old wakeup mechanism
- fix break usage in do_each_thread() { } while_eac_thread().
- fix the hotplug switch stmt, a fall-through case was broken.

Quoted 5 lines
Quoted 2 lines

thanks Peter, i've picked your fix up. I hope this solves the problem for Rafael too.

No, it doesn't, unfortunately. Actually, it breaks things even more, as suspend now hangs (probably deadlocks) while disabling the nonboot CPU.

For some obscure reason, I'm no longer able to reproduce the problem with the Peter's patch and I think it's correct. You can add

to it.

Thanks,
Rafael

#23 lore raw
In reply to Ingo Molnar
Quoted 17 lines · Ingo Molnar
Quoted 14 lines · Dmitry Adamushko
Quoted 4 lines

I've observed delays from ~3 s. up to ~8 s. (out of ~20 tests) so the 10s. delay of msleep_interruptible() might be related but I'm still looking for the reason why this fix helps (and what goes wrong with the current code).

heh... it's pretty obvious indeed. What's msleep_interruptible() is all about? :-)

"sleep waiting for signals"

so the 'watchdog' thread gets woken up

[ as a result of cpu_callback(action = CPU_DEAD) --> kthread_stop() ]

Quoted 8 lines
Quoted 5 lines

just to be immediately scheduled out again for as long as the remaining timeout > 0.

So it should work if we substitute msleep_interruptible() with schedule_timeout_interruptible().

Doh. Could someone with ths problem please test the patch below, does it do the trick?

yeah, I was already on a half-way to check it out.

It does fix a problem for me.

Don't forget to take along these 2 fixes from Peter's patch:

- fix break usage in do_each_thread() { } while_each_thread().
- fix the hotplug switch stmt, a fall-through case was broken.

Rafael, does your system still hangs? I'd expect, yes -- as effectively this fix is not that different from Peter's one when it comes to suspend-case.

Can you then send your config? Anything special about this machine (say, some /proc tunables are altered)?

Quoted 3 lines

Ingo

Signature · 2 lines
Best regards,
Dmitry Adamushko
#17 lore raw
In reply to Rafael J. Wysocki
Quoted 18 lines · Rafael J. Wysocki
Quoted 11 lines · Rafael J. Wysocki
Quoted 10 lines · Ingo Molnar
Quoted 6 lines · Peter Zijlstra

The below fixes it for me..

---
- restore the old wakeup mechanism
- fix break usage in do_each_thread() { } while_eac_thread().
- fix the hotplug switch stmt, a fall-through case was broken.

Quoted 5 lines
Quoted 2 lines

thanks Peter, i've picked your fix up. I hope this solves the problem for Rafael too.

No, it doesn't, unfortunately. Actually, it breaks things even more, as suspend now hangs (probably deadlocks) while disabling the nonboot CPU.

Quoted 6 lines

For some obscure reason, I'm no longer able to reproduce the problem with the Peter's patch and I think it's correct. You can add

to it.

Great! I sent the fix to Linus with your Tested-by line. Thanks Rafael for all the testing!

Ingo

Tested-by Rafael J. Wysocki
#24 lore raw
In reply to Dmitry Adamushko
Quoted 8 lines · Dmitry Adamushko

yeah, I was already on a half-way to check it out.

It does fix a problem for me.

Don't forget to take along these 2 fixes from Peter's patch:

- fix break usage in do_each_thread() { } while_each_thread().
- fix the hotplug switch stmt, a fall-through case was broken.

Dmitry, i sent Peter's fix(es) below to Linus. Do you concur that it fixes all the practical and theoretical problems you could see with the code too?

Ingo

--------------->
Subject: debug: softlockup looping fix
From: Peter Zijlstra <[email protected]>

Rafael J. Wysocki reported weird, multi-seconds delays during suspend/resume and bisected it back to:

commit 82a1fcb90287052aabfa235e7ffc693ea003fe69
Author: Ingo Molnar <[email protected]>
Date: Fri Jan 25 21:08:02 2008 +0100

softlockup: automatically detect hung TASK_UNINTERRUPTIBLE tasks

fix it:

- restore the old wakeup mechanism
- fix break usage in do_each_thread() { } while_each_thread().
- fix the hotplug switch stmt, a fall-through case was broken.

---
 kernel/softlockup.c |   30 ++++++++++++++++++++----------
 1 file changed, 20 insertions(+), 10 deletions(-)
kernel/softlockup.c +20 -10
Index: linux/kernel/softlockup.c
===================================================================
@@ -101,6 +101,10 @@ void softlockup_tick(void)
 
 	now = get_timestamp(this_cpu);
 
+	/* Wake up the high-prio watchdog task every second: */
+	if (now > (touch_timestamp + 1))
+		wake_up_process(per_cpu(watchdog_task, this_cpu));
+
 	/* Warn about unreasonable delays: */
 	if (now <= (touch_timestamp + softlockup_thresh))
 		return;
@@ -191,11 +195,11 @@ static void check_hung_uninterruptible_t
 	read_lock(&tasklist_lock);
 	do_each_thread(g, t) {
 		if (!--max_count)
-			break;
+			goto unlock;
 		if (t->state & TASK_UNINTERRUPTIBLE)
 			check_hung_task(t, now);
 	} while_each_thread(g, t);
-
+ unlock:
 	read_unlock(&tasklist_lock);
 }
 
@@ -218,14 +222,19 @@ static int watchdog(void *__bind_cpu)
 	 * debug-printout triggers in softlockup_tick().
 	 */
 	while (!kthread_should_stop()) {
+		set_current_state(TASK_INTERRUPTIBLE);
 		touch_softlockup_watchdog();
-		msleep_interruptible(10000);
+		schedule();
+
+		if (kthread_should_stop())
+			break;
 
 		if (this_cpu != check_cpu)
 			continue;
 
 		if (sysctl_hung_task_timeout_secs)
 			check_hung_uninterruptible_tasks(this_cpu);
+
 	}
 
 	return 0;
@@ -259,13 +268,6 @@ cpu_callback(struct notifier_block *nfb,
 		wake_up_process(per_cpu(watchdog_task, hotcpu));
 		break;
 #ifdef CONFIG_HOTPLUG_CPU
-	case CPU_UP_CANCELED:
-	case CPU_UP_CANCELED_FROZEN:
-		if (!per_cpu(watchdog_task, hotcpu))
-			break;
-		/* Unbind so it can run.  Fall thru. */
-		kthread_bind(per_cpu(watchdog_task, hotcpu),
-			     any_online_cpu(cpu_online_map));
 	case CPU_DOWN_PREPARE:
 	case CPU_DOWN_PREPARE_FROZEN:
 		if (hotcpu == check_cpu) {
@@ -275,6 +277,14 @@ cpu_callback(struct notifier_block *nfb,
 			check_cpu = any_online_cpu(temp_cpu_online_map);
 		}
 		break;
+
+	case CPU_UP_CANCELED:
+	case CPU_UP_CANCELED_FROZEN:
+		if (!per_cpu(watchdog_task, hotcpu))
+			break;
+		/* Unbind so it can run.  Fall thru. */
+		kthread_bind(per_cpu(watchdog_task, hotcpu),
+			     any_online_cpu(cpu_online_map));
 	case CPU_DEAD:
 	case CPU_DEAD_FROZEN:
 		p = per_cpu(watchdog_task, hotcpu);
#26 lore raw
In reply to Dmitry Adamushko
Quoted 33 lines · Dmitry Adamushko
Quoted 17 lines · Ingo Molnar
Quoted 14 lines · Dmitry Adamushko
Quoted 4 lines

I've observed delays from ~3 s. up to ~8 s. (out of ~20 tests) so the 10s. delay of msleep_interruptible() might be related but I'm still looking for the reason why this fix helps (and what goes wrong with the current code).

heh... it's pretty obvious indeed. What's msleep_interruptible() is all about? :-)

"sleep waiting for signals"

so the 'watchdog' thread gets woken up

[ as a result of cpu_callback(action = CPU_DEAD) --> kthread_stop() ]

Quoted 8 lines
Quoted 5 lines

just to be immediately scheduled out again for as long as the remaining timeout > 0.

So it should work if we substitute msleep_interruptible() with schedule_timeout_interruptible().

Doh. Could someone with ths problem please test the patch below, does it do the trick?

yeah, I was already on a half-way to check it out.

It does fix a problem for me.

Don't forget to take along these 2 fixes from Peter's patch:

Quoted 7 lines

- fix break usage in do_each_thread() { } while_each_thread().
- fix the hotplug switch stmt, a fall-through case was broken.

Rafael, does your system still hangs? I'd expect, yes -- as effectively this fix is not that different from Peter's one when it comes to suspend-case.

I haven't tested it yet, but I don't expect it to hang, as the Peter's patch started to work for me. I'm going to test it in a while.

Thanks,
Rafael

#25 lore raw
In reply to Ingo Molnar
Quoted 7 lines · Ingo Molnar
Quoted 4 lines · Dmitry Adamushko

yeah, I was already on a half-way to check it out.

It does fix a problem for me.

Quoted 9 lines
Quoted 4 lines

Don't forget to take along these 2 fixes from Peter's patch:

- fix break usage in do_each_thread() { } while_each_thread().
- fix the hotplug switch stmt, a fall-through case was broken.

Dmitry, i sent Peter's fix(es) below to Linus. Do you concur that it fixes all the practical and theoretical problems you could see with the code too?

One comment: any argument for waking up the 'watchdog' thread from softlockup_tick() wrt just having schedule_timeout_interruptible(HZ) in watchdog() ?
(although, it's not that important).

IMHO, it looks good.

(I'll look again tomorrow when I'm less sleepy :-/)

Signature · 2 lines
Best regards,
Dmitry Adamushko