2.6.24-git3: Major annoyance during suspend/hibernation on x86-64 (bisected)
Hi,
2.6.24-git3 adds a 5 - 10 sec delay to the suspend and hibernation code paths (probably related to the disabling of nonboot CPUs), which is !@#$%^&*() annoying.
It's 100% reproducible on my HP nx6325 and bisection idendified the following commit as the first bad one:
commit 764a9d6fe4b52995c8aba277e3634385699354f4
Author: Steven Rostedt <[email protected]> Date: Fri Jan 25 21:08:04 2008 +0100
sched: track highest prio task queued
Greetings,
Rafael
Quoted 6 lines · Rafael J. Wysocki
Hi,
2.6.24-git3 adds a 5 - 10 sec delay to the suspend and hibernation code paths (probably related to the disabling of nonboot CPUs), which is !@#$%^&*() annoying.
Quoted 8 lines
It's 100% reproducible on my HP nx6325 and bisection idendified the following commit as the first bad one:
commit
764a9d6fe4b52995c8aba277e3634385699354f4Author: Steven Rostedt <[email protected]> Date: Fri Jan 25 21:08:04 2008 +0100sched: track highest prio task queued
hm, this patch is a NOP, so it's weird that it has an effect.
Do you have serial logging enabled perhaps? If the following WARN_ON() triggers:
+ WARN_ON(p->prio < rq->rt.highest_prio);
then perhaps that can cause a 5-10 seconds delay. (that's how much time it takes to printk a warning on the slowest serial settings)
but if you use suspend, then any such printks would be preserved in the dmesg, right? If the WARN_ON() triggers, and if you remove it, do things get faster?
this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch? It's not the next patch or the preceding patch by any chance? (which would also be scheduler patches)
Ingo
Quoted 23 lines · Ingo Molnar
Quoted 6 lines · Rafael J. Wysocki
Hi,
2.6.24-git3 adds a 5 - 10 sec delay to the suspend and hibernation code paths (probably related to the disabling of nonboot CPUs), which is !@#$%^&*() annoying.
Quoted 8 lines
It's 100% reproducible on my HP nx6325 and bisection idendified the following commit as the first bad one:
commit
764a9d6fe4b52995c8aba277e3634385699354f4Author: Steven Rostedt <[email protected]> Date: Fri Jan 25 21:08:04 2008 +0100sched: track highest prio task queued
hm, this patch is a NOP, so it's weird that it has an effect.
Do you have serial logging enabled perhaps? If the following
WARN_ON()triggers:
Quoted 8 lines
+
WARN_ON(p->prio < rq->rt.highest_prio);then perhaps that can cause a 5-10 seconds delay. (that's how much time it takes to printk a warning on the slowest serial settings)
but if you use suspend, then any such printks would be preserved in the dmesg, right? If the
WARN_ON()triggers, and if you remove it, do things get faster?
No, this isn't the WARN_ON().
Quoted 2 lines
this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?
Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:
commit 82a1fcb90287052aabfa235e7ffc693ea003fe69
Author: Ingo Molnar <[email protected]> Date: Fri Jan 25 21:08:02 2008 +0100
softlockup: automatically detect hung TASK_UNINTERRUPTIBLE tasks
Reverting this commit (it reverts with some minor modifications) fixes the problem for me.
Thanks,
Rafael
Quoted 4 lines · Rafael J. Wysocki
No, this isn't the
WARN_ON().Quoted 1 line
this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?
Quoted 8 lines
Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:
commit
82a1fcb90287052aabfa235e7ffc693ea003fe69Author: Ingo Molnar <[email protected]> Date: Fri Jan 25 21:08:02 2008 +0100softlockup: automatically detect hung
TASK_UNINTERRUPTIBLEtasks
Are you getting a bunch of prints from the softlockup detector in dmesg?
I wonder if the detector can detect a long timeout caused by suspend and resume and if not is triggering false positives?
-- Steve
Quoted 4 lines
Reverting this commit (it reverts with some minor modifications) fixes the problem for me.
Quoted 10 lines · Steven Rostedt
Quoted 8 lines · Rafael J. Wysocki
No, this isn't the
WARN_ON().Quoted 2 lines
this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?
Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:
Quoted 7 lines
Quoted 5 lines
commit
82a1fcb90287052aabfa235e7ffc693ea003fe69Author: Ingo Molnar <[email protected]> Date: Fri Jan 25 21:08:02 2008 +0100softlockup: automatically detect hung
TASK_UNINTERRUPTIBLEtasksAre you getting a bunch of prints from the softlockup detector in dmesg?
No, I don't. In fact, I don't get _any_ messages from it whatsoever.
Quoted 2 lines
I wonder if the detector can detect a long timeout caused by suspend and resume and if not is triggering false positives?
I'm not sure, but the code is supposed to be suspend-aware, IIRC. However, I'm seeing a similar symptom on poweroff on an SMP x86-64 box, so it may be more directly related to the CPU hotplug. I'll try to verify that.
Thanks,
Rafael
Quoted 19 lines · Rafael J. Wysocki
Quoted 10 lines · Steven Rostedt
Quoted 8 lines · Rafael J. Wysocki
No, this isn't the
WARN_ON().Quoted 2 lines
this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?
Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:
Quoted 7 lines
Quoted 5 lines
commit
82a1fcb90287052aabfa235e7ffc693ea003fe69Author: Ingo Molnar <[email protected]> Date: Fri Jan 25 21:08:02 2008 +0100softlockup: automatically detect hung
TASK_UNINTERRUPTIBLEtasksAre you getting a bunch of prints from the softlockup detector in dmesg?
Quoted 8 lines
No, I don't. In fact, I don't get _any_ messages from it whatsoever.
Quoted 2 lines
I wonder if the detector can detect a long timeout caused by suspend and resume and if not is triggering false positives?
I'm not sure, but the code is supposed to be suspend-aware, IIRC. However, I'm seeing a similar symptom on poweroff on an SMP x86-64 box, so it may be more directly related to the CPU hotplug. I'll try to verify that.
As I expected, the delay is also observable when I do:
echo 0 > /sys/devices/system/cpu/cpu1/online
(it's variable, between 3 and 30 seconds). Again, no messages appear in dmesg when this happens.
I suspect I'll be able to reproduce it on another x86-64 SMP machine (I'm going to try that later today).
Thanks,
Rafael
Quoted 25 lines · Rafael J. Wysocki
Quoted 13 lines · Rafael J. Wysocki
Quoted 12 lines · Steven Rostedt
Quoted 10 lines · Rafael J. Wysocki
No, this isn't the
WARN_ON().Quoted 1 line
this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?
Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:
commit
82a1fcb90287052aabfa235e7ffc693ea003fe69Author: Ingo Molnar <[email protected]> Date: Fri Jan 25 21:08:02 2008 +0100Quoted 8 lines
Quoted 2 lines
Quoted 1 line
softlockup: automatically detect hung
TASK_UNINTERRUPTIBLEtasksAre you getting a bunch of prints from the softlockup detector in dmesg?
No, I don't. In fact, I don't get _any_ messages from it whatsoever.
Quoted 1 line
I wonder if the detector can detect a long timeout caused by suspend and resume and if not is triggering false positives?
I'm not sure, but the code is supposed to be suspend-aware, IIRC. However, I'm seeing a similar symptom on poweroff on an SMP x86-64 box, so it may be more directly related to the CPU hotplug. I'll try to verify that.
As I expected, the delay is also observable when I do:
Quoted 7 lines
echo 0 >
/sys/devices/system/cpu/cpu1/online(it's variable, between 3 and 30 seconds). Again, no messages appear in dmesg when this happens.
I suspect I'll be able to reproduce it on another x86-64 SMP machine (I'm going to try that later today).
Could you download my tracing patches from here:
http://people.redhat.com/srostedt/tracing/mcount-tracing-patches-v6.tar.bz2
Apply the above patches and select all but the histogram tracers. Then run this program
http://people.redhat.com/srostedt/tracing/trace-cmd.c
./trace-cmd -f echo 0 > /sys/devices/system/cpu/cpu1/online
then do this (mount debugfs):
bzip2 -c /debugfs/tracing/latency_trace > /tmp/trace.bz2
and send the result to myself and Ingo.
Thanks,
-- Steve
Quoted 3 lines
Thanks,
Rafael
Quoted 31 lines · Rafael J. Wysocki
Quoted 19 lines · Rafael J. Wysocki
Quoted 10 lines · Steven Rostedt
Quoted 8 lines · Rafael J. Wysocki
No, this isn't the
WARN_ON().Quoted 2 lines
this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?
Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:
Quoted 7 lines
Quoted 5 lines
commit
82a1fcb90287052aabfa235e7ffc693ea003fe69Author: Ingo Molnar <[email protected]> Date: Fri Jan 25 21:08:02 2008 +0100softlockup: automatically detect hung
TASK_UNINTERRUPTIBLEtasksAre you getting a bunch of prints from the softlockup detector in dmesg?
Quoted 8 lines
No, I don't. In fact, I don't get _any_ messages from it whatsoever.
Quoted 2 lines
I wonder if the detector can detect a long timeout caused by suspend and resume and if not is triggering false positives?
I'm not sure, but the code is supposed to be suspend-aware, IIRC. However, I'm seeing a similar symptom on poweroff on an SMP x86-64 box, so it may be more directly related to the CPU hotplug. I'll try to verify that.
As I expected, the delay is also observable when I do:
Quoted 7 lines
echo 0 >
/sys/devices/system/cpu/cpu1/online(it's variable, between 3 and 30 seconds). Again, no messages appear in dmesg when this happens.
I suspect I'll be able to reproduce it on another x86-64 SMP machine (I'm going to try that later today).
FWIW, the problem is reproducible on the second machine, with the same symptoms. Both machines are AMD-based.
Thanks,
Rafael
Quoted 41 lines · Rafael J. Wysocki
Quoted 23 lines · Ingo Molnar
Quoted 6 lines · Rafael J. Wysocki
Hi,
2.6.24-git3 adds a 5 - 10 sec delay to the suspend and hibernation code paths (probably related to the disabling of nonboot CPUs), which is !@#$%^&*() annoying.
Quoted 8 lines
It's 100% reproducible on my HP nx6325 and bisection idendified the following commit as the first bad one:
commit
764a9d6fe4b52995c8aba277e3634385699354f4Author: Steven Rostedt <[email protected]> Date: Fri Jan 25 21:08:04 2008 +0100sched: track highest prio task queued
hm, this patch is a NOP, so it's weird that it has an effect.
Do you have serial logging enabled perhaps? If the following
WARN_ON()triggers:Quoted 8 lines
+
WARN_ON(p->prio < rq->rt.highest_prio);then perhaps that can cause a 5-10 seconds delay. (that's how much time it takes to printk a warning on the slowest serial settings)
but if you use suspend, then any such printks would be preserved in the dmesg, right? If the
WARN_ON()triggers, and if you remove it, do things get faster?No, this isn't the
WARN_ON().Quoted 2 lines
this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?
Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:
Quoted 8 lines
commit
82a1fcb90287052aabfa235e7ffc693ea003fe69Author: Ingo Molnar <[email protected]> Date: Fri Jan 25 21:08:02 2008 +0100softlockup: automatically detect hung
TASK_UNINTERRUPTIBLEtasksReverting this commit (it reverts with some minor modifications) fixes the problem for me.
What if you use the same kernel that triggers a problem and just disable this new 'softlockup' functionality:
echo 0 > /proc/sys/kernel/hung_task_timeout_secs
does the problem disapear?
TIA,
Quoted 4 lines
Thanks,
Rafael
Signature · 2 lines
Best regards, Dmitry Adamushko
Quoted 51 lines · Dmitry Adamushko
Quoted 41 lines · Rafael J. Wysocki
Quoted 23 lines · Ingo Molnar
Quoted 6 lines · Rafael J. Wysocki
Hi,
2.6.24-git3 adds a 5 - 10 sec delay to the suspend and hibernation code paths (probably related to the disabling of nonboot CPUs), which is !@#$%^&*() annoying.
Quoted 8 lines
It's 100% reproducible on my HP nx6325 and bisection idendified the following commit as the first bad one:
commit
764a9d6fe4b52995c8aba277e3634385699354f4Author: Steven Rostedt <[email protected]> Date: Fri Jan 25 21:08:04 2008 +0100sched: track highest prio task queued
hm, this patch is a NOP, so it's weird that it has an effect.
Do you have serial logging enabled perhaps? If the following
WARN_ON()triggers:Quoted 8 lines
+
WARN_ON(p->prio < rq->rt.highest_prio);then perhaps that can cause a 5-10 seconds delay. (that's how much time it takes to printk a warning on the slowest serial settings)
but if you use suspend, then any such printks would be preserved in the dmesg, right? If the
WARN_ON()triggers, and if you remove it, do things get faster?No, this isn't the
WARN_ON().Quoted 2 lines
this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?
Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:
Quoted 8 lines
commit
82a1fcb90287052aabfa235e7ffc693ea003fe69Author: Ingo Molnar <[email protected]> Date: Fri Jan 25 21:08:02 2008 +0100softlockup: automatically detect hung
TASK_UNINTERRUPTIBLEtasksReverting this commit (it reverts with some minor modifications) fixes the problem for me.
Quoted 6 lines
What if you use the same kernel that triggers a problem and just disable this new 'softlockup' functionality:
echo 0 >
/proc/sys/kernel/hung_task_timeout_secsdoes the problem disapear?
I haven't checked that yet, will do it later today.
Thanks,
Rafael
Quoted 51 lines · Dmitry Adamushko
Quoted 41 lines · Rafael J. Wysocki
Quoted 23 lines · Ingo Molnar
Quoted 6 lines · Rafael J. Wysocki
Hi,
2.6.24-git3 adds a 5 - 10 sec delay to the suspend and hibernation code paths (probably related to the disabling of nonboot CPUs), which is !@#$%^&*() annoying.
Quoted 8 lines
It's 100% reproducible on my HP nx6325 and bisection idendified the following commit as the first bad one:
commit
764a9d6fe4b52995c8aba277e3634385699354f4Author: Steven Rostedt <[email protected]> Date: Fri Jan 25 21:08:04 2008 +0100sched: track highest prio task queued
hm, this patch is a NOP, so it's weird that it has an effect.
Do you have serial logging enabled perhaps? If the following
WARN_ON()triggers:Quoted 8 lines
+
WARN_ON(p->prio < rq->rt.highest_prio);then perhaps that can cause a 5-10 seconds delay. (that's how much time it takes to printk a warning on the slowest serial settings)
but if you use suspend, then any such printks would be preserved in the dmesg, right? If the
WARN_ON()triggers, and if you remove it, do things get faster?No, this isn't the
WARN_ON().Quoted 2 lines
this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?
Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:
Quoted 8 lines
commit
82a1fcb90287052aabfa235e7ffc693ea003fe69Author: Ingo Molnar <[email protected]> Date: Fri Jan 25 21:08:02 2008 +0100softlockup: automatically detect hung
TASK_UNINTERRUPTIBLEtasksReverting this commit (it reverts with some minor modifications) fixes the problem for me.
Quoted 6 lines
What if you use the same kernel that triggers a problem and just disable this new 'softlockup' functionality:
echo 0 >
/proc/sys/kernel/hung_task_timeout_secsdoes the problem disapear?
No, it doesn't. The setting doesn't seem to have any effect on it.
Thanks,
Rafael
Quoted 41 lines · Rafael J. Wysocki
Quoted 23 lines · Ingo Molnar
Quoted 6 lines · Rafael J. Wysocki
Hi,
2.6.24-git3 adds a 5 - 10 sec delay to the suspend and hibernation code paths (probably related to the disabling of nonboot CPUs), which is !@#$%^&*() annoying.
Quoted 8 lines
It's 100% reproducible on my HP nx6325 and bisection idendified the following commit as the first bad one:
commit
764a9d6fe4b52995c8aba277e3634385699354f4Author: Steven Rostedt <[email protected]> Date: Fri Jan 25 21:08:04 2008 +0100sched: track highest prio task queued
hm, this patch is a NOP, so it's weird that it has an effect.
Do you have serial logging enabled perhaps? If the following
WARN_ON()triggers:Quoted 8 lines
+
WARN_ON(p->prio < rq->rt.highest_prio);then perhaps that can cause a 5-10 seconds delay. (that's how much time it takes to printk a warning on the slowest serial settings)
but if you use suspend, then any such printks would be preserved in the dmesg, right? If the
WARN_ON()triggers, and if you remove it, do things get faster?No, this isn't the
WARN_ON().Quoted 2 lines
this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?
Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:
Quoted 8 lines
commit
82a1fcb90287052aabfa235e7ffc693ea003fe69Author: Ingo Molnar <[email protected]> Date: Fri Jan 25 21:08:02 2008 +0100softlockup: automatically detect hung
TASK_UNINTERRUPTIBLEtasksReverting this commit (it reverts with some minor modifications) fixes the problem for me.
I can seem to reproduce this:
[root@opteron cpu1]# time echo 0 > online
real 0m6.230s user 0m0.000s sys 0m0.010s
[root@opteron cpu1]# echo 1 > online
[root@opteron cpu1]# time echo 0 > online
real 0m7.966s user 0m0.000s sys 0m0.011s
I'll have a look at it.
Quoted 62 lines · Peter Zijlstra
Quoted 41 lines · Rafael J. Wysocki
Quoted 23 lines · Ingo Molnar
Quoted 6 lines · Rafael J. Wysocki
Hi,
2.6.24-git3 adds a 5 - 10 sec delay to the suspend and hibernation code paths (probably related to the disabling of nonboot CPUs), which is !@#$%^&*() annoying.
Quoted 8 lines
It's 100% reproducible on my HP nx6325 and bisection idendified the following commit as the first bad one:
commit
764a9d6fe4b52995c8aba277e3634385699354f4Author: Steven Rostedt <[email protected]> Date: Fri Jan 25 21:08:04 2008 +0100sched: track highest prio task queued
hm, this patch is a NOP, so it's weird that it has an effect.
Do you have serial logging enabled perhaps? If the following
WARN_ON()triggers:Quoted 8 lines
+
WARN_ON(p->prio < rq->rt.highest_prio);then perhaps that can cause a 5-10 seconds delay. (that's how much time it takes to printk a warning on the slowest serial settings)
but if you use suspend, then any such printks would be preserved in the dmesg, right? If the
WARN_ON()triggers, and if you remove it, do things get faster?No, this isn't the
WARN_ON().Quoted 2 lines
this does have the feel of being scheduling related, but are you absolutely sure about the precise identity of the patch?
Actually, not quite. That's why I have verified it and found that another patch is really responsible for the issue, namely:
Quoted 8 lines
commit
82a1fcb90287052aabfa235e7ffc693ea003fe69Author: Ingo Molnar <[email protected]> Date: Fri Jan 25 21:08:02 2008 +0100softlockup: automatically detect hung
TASK_UNINTERRUPTIBLEtasksReverting this commit (it reverts with some minor modifications) fixes the problem for me.
I can seem to reproduce this:
[root@opteron cpu1]# time echo 0 > online
real 0m6.230s user 0m0.000s sys 0m0.010s[root@opteron cpu1]# echo 1 > online
[root@opteron cpu1]# time echo 0 > online
Quoted 6 lines
real 0m7.966s user 0m0.000s sys 0m0.011sI'll have a look at it.
Much appreciated, thanks!
Quoted 12 lines · Rafael J. Wysocki
Quoted 10 lines · Peter Zijlstra
I can seem to reproduce this:
[root@opteron cpu1]# time echo 0 > online
real 0m6.230s user 0m0.000s sys 0m0.010s[root@opteron cpu1]# echo 1 > online
[root@opteron cpu1]# time echo 0 > online
Quoted 8 lines
Quoted 6 lines
real 0m7.966s user 0m0.000s sys 0m0.011sI'll have a look at it.
Much appreciated, thanks!
The below fixes it for me..
---
- restore the old wakeup mechanism
- fix break usage in do_each_thread() { } while_eac_thread().
- fix the hotplug switch stmt, a fall-through case was broken.
---
kernel/softlockup.c +20 -10
@@ -101,6 +101,10 @@ void softlockup_tick(void) now =get_timestamp (this_cpu); +/* Wake up the high-prio watchdog task every second: */ +if (now > (touch_timestamp +1 )) +wake_up_process (per_cpu (watchdog_task, this_cpu)); +/* Warn about unreasonable delays: */ if (now <= (touch_timestamp + softlockup_thresh))return ;
@@ -191,11 +195,11 @@ static void check_hung_uninterruptible_tasks(int this_cpu)read_lock (&tasklist_lock); do_each_thread(g, t) {if (!--max_count) -break ; + goto unlock;if (t->state &TASK_UNINTERRUPTIBLE )check_hung_task (t, now); }while_each_thread (g, t); - + unlock:read_unlock (&tasklist_lock); }
@@ -218,14 +222,19 @@ static int watchdog(void *__bind_cpu)* debug-printout triggers in softlockup_tick(). */ while (!kthread_should_stop ()) { +set_current_state (TASK_INTERRUPTIBLE );touch_softlockup_watchdog (); -msleep_interruptible (10000 ); +schedule (); + +if (kthread_should_stop ()) +break ;if (this_cpu != check_cpu)continue ;if (sysctl_hung_task_timeout_secs)check_hung_uninterruptible_tasks (this_cpu); + }return 0 ;
@@ -259,13 +268,6 @@ cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu)wake_up_process (per_cpu (watchdog_task, hotcpu));break ;#ifdef CONFIG_HOTPLUG_CPU -case CPU_UP_CANCELED : -case CPU_UP_CANCELED_FROZEN : -if (!per_cpu (watchdog_task, hotcpu)) -break ; -/* Unbind so it can run. Fall thru. */ -kthread_bind (per_cpu (watchdog_task, hotcpu), -any_online_cpu (cpu_online_map));case CPU_DOWN_PREPARE :case CPU_DOWN_PREPARE_FROZEN :if (hotcpu == check_cpu) {
@@ -275,6 +277,14 @@ cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu) check_cpu =any_online_cpu (temp_cpu_online_map); }break ; + +case CPU_UP_CANCELED : +case CPU_UP_CANCELED_FROZEN : +if (!per_cpu (watchdog_task, hotcpu)) +break ; +/* Unbind so it can run. Fall thru. */ +kthread_bind (per_cpu (watchdog_task, hotcpu), +any_online_cpu (cpu_online_map));case CPU_DEAD :case CPU_DEAD_FROZEN : p =per_cpu (watchdog_task, hotcpu);
Quoted 6 lines · Peter Zijlstra
The below fixes it for me..
---
- restore the old wakeup mechanism
- fix break usage indo_each_thread(){ }while_eac_thread().
- fix the hotplug switch stmt, a fall-through case was broken.
thanks Peter, i've picked your fix up. I hope this solves the problem for Rafael too.
Ingo
Quoted 5 lines · Ingo Molnar
Quoted 2 lines · Peter Zijlstra
The below fixes it for me..
Quoted 7 lines
Quoted 4 lines
---
- restore the old wakeup mechanism
- fix break usage indo_each_thread(){ }while_eac_thread().
- fix the hotplug switch stmt, a fall-through case was broken.thanks Peter, i've picked your fix up. I hope this solves the problem for Rafael too.
No, it doesn't, unfortunately. Actually, it breaks things even more, as suspend now hangs (probably deadlocks) while disabling the nonboot CPU.
Thanks,
Rafael
Rafael J. Wysocki
Tested-byQuoted 11 lines · Rafael J. Wysocki
Quoted 10 lines · Ingo Molnar
Quoted 6 lines · Peter Zijlstra
The below fixes it for me..
---
- restore the old wakeup mechanism
- fix break usage indo_each_thread(){ }while_eac_thread().
- fix the hotplug switch stmt, a fall-through case was broken.
Quoted 5 lines
Quoted 2 lines
thanks Peter, i've picked your fix up. I hope this solves the problem for Rafael too.
No, it doesn't, unfortunately. Actually, it breaks things even more, as suspend now hangs (probably deadlocks) while disabling the nonboot CPU.
For some obscure reason, I'm no longer able to reproduce the problem with the Peter's patch and I think it's correct. You can add
to it.
Thanks,
Rafael
Quoted 18 lines · Rafael J. Wysocki
Quoted 11 lines · Rafael J. Wysocki
Quoted 10 lines · Ingo Molnar
Quoted 6 lines · Peter Zijlstra
The below fixes it for me..
---
- restore the old wakeup mechanism
- fix break usage indo_each_thread(){ }while_eac_thread().
- fix the hotplug switch stmt, a fall-through case was broken.Quoted 5 lines
Quoted 2 lines
thanks Peter, i've picked your fix up. I hope this solves the problem for Rafael too.
No, it doesn't, unfortunately. Actually, it breaks things even more, as suspend now hangs (probably deadlocks) while disabling the nonboot CPU.
Quoted 6 lines
For some obscure reason, I'm no longer able to reproduce the problem with the Peter's patch and I think it's correct. You can add
Tested-by Rafael J. Wysocki <[email protected]>to it.
Great! I sent the fix to Linus with your Tested-by line. Thanks Rafael for all the testing!
Ingo
Quoted 2 lines · Peter Zijlstra
---
- restore the old wakeup mechanism
and how does it change behavior, logically-wise?
do we somehow miss a 'wake-up' from kthread_stop() so that its caller gets blocked until watchdog's msleep_interruptible(10000) timeouts? On average, it would take +-5 sec. and might explain the first observation of Ravael -- "...adds a 5 - 10 sec delay..." (although, lately he reported up to +30 sec. delays).
(/me goint to also try reproducing it later today)
Quoted 1 line
[ ... ]
Signature · 2 lines
Best regards, Dmitry Adamushko
Quoted 7 lines · Dmitry Adamushko
Quoted 2 lines · Peter Zijlstra
---
- restore the old wakeup mechanismand how does it change behavior, logically-wise?
Quoted 7 lines
do we somehow miss a 'wake-up' from
kthread_stop()so that its caller gets blocked until watchdog'smsleep_interruptible(10000)timeouts? On average, it would take +-5 sec. and might explain the first observation of Ravael -- "...adds a 5 - 10 sec delay..." (although, lately he reported up to +30 sec. delays).(/me goint to also try reproducing it later today)
thanks - i cannot reproduce it on my usual suspend/resume testbox because e1000 broke on it, and this is a pretty annoying regression. We'll have to undo the hung-tasks detection feature if it's not fixed quickly. (there's no point in debugging features that _add_ bugs)
Ingo
Quoted 5 lines · Ingo Molnar
thanks - i cannot reproduce it on my usual suspend/resume testbox because e1000 broke on it, and this is a pretty annoying regression. We'll have to undo the hung-tasks detection feature if it's not fixed quickly. (there's no point in debugging features that _add_ bugs)
I can reproduce it and Peter's patch does fix it for me.
e.g.
root@earth:/sys/devices/system/cpu/cpu1# time echo 0 > online real 0m6.743s root@earth:/sys/devices/system/cpu/cpu1# time echo 0 > online real 0m7.770s
I've observed delays from ~3 s. up to ~8 s. (out of ~20 tests) so the 10s. delay of msleep_interruptible() might be related but I'm still looking for the reason why this fix helps (and what goes wrong with the current code).
Quoted 3 lines
Ingo
Signature · 2 lines
Best regards, Dmitry Adamushko
Quoted 15 lines · Dmitry Adamushko
Quoted 5 lines · Ingo Molnar
thanks - i cannot reproduce it on my usual suspend/resume testbox because e1000 broke on it, and this is a pretty annoying regression. We'll have to undo the hung-tasks detection feature if it's not fixed quickly. (there's no point in debugging features that _add_ bugs)
I can reproduce it and Peter's patch does fix it for me.
e.g.
root@earth:/sys/devices/system/cpu/cpu1# time echo 0 > online real 0m6.743s root@earth:/sys/devices/system/cpu/cpu1# time echo 0 > online real 0m7.770s
Quoted 5 lines
I've observed delays from ~3 s. up to ~8 s. (out of ~20 tests) so the 10s. delay of
msleep_interruptible()might be related but I'm still looking for the reason why this fix helps (and what goes wrong with the current code).
heh... it's pretty obvious indeed. What's msleep_interruptible() is all about? :-)
"sleep waiting for signals"
so the 'watchdog' thread gets woken up
[ as a result of cpu_callback(action = CPU_DEAD) --> kthread_stop() ]
just to be immediately scheduled out again for as long as the remaining timeout > 0.
So it should work if we substitute msleep_interruptible() with schedule_timeout_interruptible().
Signature · 2 lines
Best regards, Dmitry Adamushko
Quoted 12 lines · Dmitry Adamushko
Quoted 4 lines
I've observed delays from ~3 s. up to ~8 s. (out of ~20 tests) so the 10s. delay of
msleep_interruptible()might be related but I'm still looking for the reason why this fix helps (and what goes wrong with the current code).heh... it's pretty obvious indeed. What's
msleep_interruptible()is all about? :-)"sleep waiting for signals"
so the 'watchdog' thread gets woken up
Quoted 7 lines
[ as a result of
cpu_callback(action = CPU_DEAD)-->kthread_stop()]just to be immediately scheduled out again for as long as the remaining timeout > 0.
So it should work if we substitute
msleep_interruptible()withschedule_timeout_interruptible().
Doh. Could someone with ths problem please test the patch below, does it do the trick?
Ingo
--- kernel/softlockup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/softlockup.c +1 -1
@@ -228,7 +228,7 @@ static int watchdog(void *__bind_cpu)*/ while (!kthread_should_stop ()) {touch_softlockup_watchdog (); -msleep_interruptible (10000 ); +schedule_timeout_interruptible (10 *HZ );if (this_cpu != check_cpu)continue ;
Quoted 17 lines · Ingo Molnar
Quoted 14 lines · Dmitry Adamushko
Quoted 4 lines
I've observed delays from ~3 s. up to ~8 s. (out of ~20 tests) so the 10s. delay of
msleep_interruptible()might be related but I'm still looking for the reason why this fix helps (and what goes wrong with the current code).heh... it's pretty obvious indeed. What's
msleep_interruptible()is all about? :-)"sleep waiting for signals"
so the 'watchdog' thread gets woken up
[ as a result of
cpu_callback(action = CPU_DEAD)-->kthread_stop()]
Quoted 8 lines
Quoted 5 lines
just to be immediately scheduled out again for as long as the remaining timeout > 0.
So it should work if we substitute
msleep_interruptible()withschedule_timeout_interruptible().Doh. Could someone with ths problem please test the patch below, does it do the trick?
yeah, I was already on a half-way to check it out.
It does fix a problem for me.
Don't forget to take along these 2 fixes from Peter's patch:
- fix break usage in do_each_thread() { } while_each_thread().
- fix the hotplug switch stmt, a fall-through case was broken.
Rafael, does your system still hangs? I'd expect, yes -- as effectively this fix is not that different from Peter's one when it comes to suspend-case.
Can you then send your config? Anything special about this machine (say, some /proc tunables are altered)?
Quoted 3 lines
Ingo
Signature · 2 lines
Best regards, Dmitry Adamushko
Ingo Molnar
Tested-by Rafael J. WysockiQuoted 8 lines · Dmitry Adamushko
yeah, I was already on a half-way to check it out.
It does fix a problem for me.
Don't forget to take along these 2 fixes from Peter's patch:
- fix break usage in
do_each_thread(){ }while_each_thread().
- fix the hotplug switch stmt, a fall-through case was broken.
Dmitry, i sent Peter's fix(es) below to Linus. Do you concur that it fixes all the practical and theoretical problems you could see with the code too?
Ingo
--------------->
Subject: debug: softlockup looping fix
From: Peter Zijlstra <[email protected]>
Rafael J. Wysocki reported weird, multi-seconds delays during suspend/resume and bisected it back to:
commit 82a1fcb90287052aabfa235e7ffc693ea003fe69
Author: Ingo Molnar <[email protected]>
Date: Fri Jan 25 21:08:02 2008 +0100
softlockup: automatically detect hung TASK_UNINTERRUPTIBLE tasks
fix it:
- restore the old wakeup mechanism
- fix break usage in do_each_thread() { } while_each_thread().
- fix the hotplug switch stmt, a fall-through case was broken.
--- kernel/softlockup.c | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-)
kernel/softlockup.c +20 -10
@@ -101,6 +101,10 @@ void softlockup_tick(void) now =get_timestamp (this_cpu); +/* Wake up the high-prio watchdog task every second: */ +if (now > (touch_timestamp +1 )) +wake_up_process (per_cpu (watchdog_task, this_cpu)); +/* Warn about unreasonable delays: */ if (now <= (touch_timestamp + softlockup_thresh))return ;
@@ -191,11 +195,11 @@ static void check_hung_uninterruptible_tread_lock (&tasklist_lock); do_each_thread(g, t) {if (!--max_count) -break ; + goto unlock;if (t->state &TASK_UNINTERRUPTIBLE )check_hung_task (t, now); }while_each_thread (g, t); - + unlock:read_unlock (&tasklist_lock); }
@@ -218,14 +222,19 @@ static int watchdog(void *__bind_cpu)* debug-printout triggers in softlockup_tick(). */ while (!kthread_should_stop ()) { +set_current_state (TASK_INTERRUPTIBLE );touch_softlockup_watchdog (); -msleep_interruptible (10000 ); +schedule (); + +if (kthread_should_stop ()) +break ;if (this_cpu != check_cpu)continue ;if (sysctl_hung_task_timeout_secs)check_hung_uninterruptible_tasks (this_cpu); + }return 0 ;
@@ -259,13 +268,6 @@ cpu_callback(struct notifier_block *nfb,wake_up_process (per_cpu (watchdog_task, hotcpu));break ;#ifdef CONFIG_HOTPLUG_CPU -case CPU_UP_CANCELED : -case CPU_UP_CANCELED_FROZEN : -if (!per_cpu (watchdog_task, hotcpu)) -break ; -/* Unbind so it can run. Fall thru. */ -kthread_bind (per_cpu (watchdog_task, hotcpu), -any_online_cpu (cpu_online_map));case CPU_DOWN_PREPARE :case CPU_DOWN_PREPARE_FROZEN :if (hotcpu == check_cpu) {
@@ -275,6 +277,14 @@ cpu_callback(struct notifier_block *nfb, check_cpu =any_online_cpu (temp_cpu_online_map); }break ; + +case CPU_UP_CANCELED : +case CPU_UP_CANCELED_FROZEN : +if (!per_cpu (watchdog_task, hotcpu)) +break ; +/* Unbind so it can run. Fall thru. */ +kthread_bind (per_cpu (watchdog_task, hotcpu), +any_online_cpu (cpu_online_map));case CPU_DEAD :case CPU_DEAD_FROZEN : p =per_cpu (watchdog_task, hotcpu);
Quoted 7 lines · Ingo Molnar
Quoted 4 lines · Dmitry Adamushko
yeah, I was already on a half-way to check it out.
It does fix a problem for me.
Quoted 9 lines
Quoted 4 lines
Don't forget to take along these 2 fixes from Peter's patch:
- fix break usage in
do_each_thread(){ }while_each_thread().
- fix the hotplug switch stmt, a fall-through case was broken.Dmitry, i sent Peter's fix(es) below to Linus. Do you concur that it fixes all the practical and theoretical problems you could see with the code too?
One comment: any argument for waking up the 'watchdog' thread from softlockup_tick() wrt just having schedule_timeout_interruptible(HZ) in watchdog() ?
(although, it's not that important).
IMHO, it looks good.
(I'll look again tomorrow when I'm less sleepy :-/)
Signature · 2 lines
Best regards, Dmitry Adamushko
Quoted 33 lines · Dmitry Adamushko
Quoted 17 lines · Ingo Molnar
Quoted 14 lines · Dmitry Adamushko
Quoted 4 lines
I've observed delays from ~3 s. up to ~8 s. (out of ~20 tests) so the 10s. delay of
msleep_interruptible()might be related but I'm still looking for the reason why this fix helps (and what goes wrong with the current code).heh... it's pretty obvious indeed. What's
msleep_interruptible()is all about? :-)"sleep waiting for signals"
so the 'watchdog' thread gets woken up
[ as a result of
cpu_callback(action = CPU_DEAD)-->kthread_stop()]Quoted 8 lines
Quoted 5 lines
just to be immediately scheduled out again for as long as the remaining timeout > 0.
So it should work if we substitute
msleep_interruptible()withschedule_timeout_interruptible().Doh. Could someone with ths problem please test the patch below, does it do the trick?
yeah, I was already on a half-way to check it out.
It does fix a problem for me.
Don't forget to take along these 2 fixes from Peter's patch:
Quoted 7 lines
- fix break usage in
do_each_thread(){ }while_each_thread().
- fix the hotplug switch stmt, a fall-through case was broken.Rafael, does your system still hangs? I'd expect, yes -- as effectively this fix is not that different from Peter's one when it comes to suspend-case.
I haven't tested it yet, but I don't expect it to hang, as the Peter's patch started to work for me. I'm going to test it in a while.
Thanks,
Rafael