Skip to contentSkip to conversation

SwSusp to disk doesn't work - Try 2

DISCUSSION Thomas Meyer 4 participants 19 messages Mar 11, 2007

lore
Share & tools
lore
#0 lore raw

Suspend to disk doesn't work on my laptop.

The suspend seems to hang while enabling the non-boot cpus again.

with platform = "test" and state = "disk" i get this:
"
[cut]
acpi device:02: freeze
video video:00: freeze
acpi device:01: freeze
acpi PNP0C02:00: freeze
pci_root PNP0A08:00: freeze
button PNP0C0E:00: freeze
button PNP0C0C:00: freeze
acpi APP0002:00: freeze
button PNP0C0D:00: freeze
ac ACPI0003:00: freeze
acpi device:00: freeze
processor ACPI0007:01: freeze
processor ACPI0007:00: freeze
button button_power:00: freeze
acpi acpi_system:00: freeze
Disabling non-boot CPUs ...
CPU 1 is now offline
SMP alternatives: switching to UP code
PM: Removing info for No Bus:cpu1
PM: Removing info for No Bus:msr1
CPU1 is down
swsusp debug: Waiting for 5 seconds.
Enabling non-boot CPUs ...

----> Here the process hangs. But a fortunate coincidence showed me that an acpi event continues the process (pressing the power off button a few times... (2x - 4x) ).

SMP alternatives: switching to SMP code
Booting processor 1/1 eip 3000
CPU 1 irqstacks, hard=c0389000 soft=c0387000
Initializing CPU#1
Calibrating delay using timer specific routine.. 3663.73 BogoMIPS (lpj=6103576)
CPU: After generic identify, caps: bfe9fbff 00100000 00000000 00000000 0000c1a9 00000000 00000000 monitor/mwait feature present.
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 2048K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 1
CPU: After all inits, caps: bfe9fbff 00100000 00000000 00002940 0000c1a9 00000000 00000000 Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#1.
CPU1: Intel Genuine Intel(R) CPU T2400 @ 1.83GHz stepping 08
PM: Adding info for No Bus:cpu1
PM: Adding info for No Bus:msr1
CPU1 is up
acpi acpi_system:00: resuming
button button_power:00: resuming
processor ACPI0007:00: resuming
processor ACPI0007:01: resuming
acpi device:00: resuming
ac ACPI0003:00: resuming
button PNP0C0D:00: resuming
acpi APP0002:00: resuming
button PNP0C0C:00: resuming
button PNP0C0E:00: resuming
pci_root PNP0A08:00: resuming
"

Any ideas?

The same is true for disk = "platform".

With kind regards
thomas

#1 lore raw
Quoted 31 lines · Thomas Meyer

Suspend to disk doesn't work on my laptop.

The suspend seems to hang while enabling the non-boot cpus again.

with platform = "test" and state = "disk" i get this:
"
[cut]
acpi device:02: freeze
video video:00: freeze
acpi device:01: freeze
acpi PNP0C02:00: freeze
pci_root PNP0A08:00: freeze
button PNP0C0E:00: freeze
button PNP0C0C:00: freeze
acpi APP0002:00: freeze
button PNP0C0D:00: freeze
ac ACPI0003:00: freeze
acpi device:00: freeze
processor ACPI0007:01: freeze
processor ACPI0007:00: freeze
button button_power:00: freeze
acpi acpi_system:00: freeze
Disabling non-boot CPUs ...
CPU 1 is now offline
SMP alternatives: switching to UP code
PM: Removing info for No Bus:cpu1
PM: Removing info for No Bus:msr1
CPU1 is down
swsusp debug: Waiting for 5 seconds.
Enabling non-boot CPUs ...

Quoted 3 lines

----> Here the process hangs. But a fortunate coincidence showed me that an acpi event continues the process (pressing the power off button a few times... (2x - 4x) ).

Hm, interesting.

Quoted 34 lines

SMP alternatives: switching to SMP code
Booting processor 1/1 eip 3000
CPU 1 irqstacks, hard=c0389000 soft=c0387000
Initializing CPU#1
Calibrating delay using timer specific routine.. 3663.73 BogoMIPS (lpj=6103576)
CPU: After generic identify, caps: bfe9fbff 00100000 00000000 00000000 0000c1a9 00000000 00000000
monitor/mwait feature present.
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 2048K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 1
CPU: After all inits, caps: bfe9fbff 00100000 00000000 00002940 0000c1a9 00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#1.
CPU1: Intel Genuine Intel(R) CPU T2400 @ 1.83GHz stepping 08
PM: Adding info for No Bus:cpu1
PM: Adding info for No Bus:msr1
CPU1 is up
acpi acpi_system:00: resuming
button button_power:00: resuming
processor ACPI0007:00: resuming
processor ACPI0007:01: resuming
acpi device:00: resuming
ac ACPI0003:00: resuming
button PNP0C0D:00: resuming
acpi APP0002:00: resuming
button PNP0C0C:00: resuming
button PNP0C0E:00: resuming
pci_root PNP0A08:00: resuming
"

Quoted 1 line

Any ideas?

Could you please put some printk()s in kernel/cpu.c:_cpu_up() to see where it gets stuck? I bet one of the notifiers goes to sleep (cpufreq, maybe).

Greetings,
Rafael

Signature · 3 lines
If you don't have the time to read,
you don't have the time or the tools to write.
		- Stephen King
#2 lore raw
Quoted 4 lines · Rafael J. Wysocki

Could you please put some printk()s in kernel/cpu.c:_cpu_up() to see where it gets stuck? I bet one of the notifiers goes to sleep (cpufreq, maybe).

Here we go (ok. i forgot __FUNCTION__ ...):

Mar 11 19:31:33 [kernel] ac ACPI0003:00: freeze
Mar 11 19:31:33 [kernel] acpi device:00: freeze
Mar 11 19:31:33 [kernel] processor ACPI0007:01: freeze
Mar 11 19:31:33 [kernel] processor ACPI0007:00: freeze
Mar 11 19:31:33 [kernel] button button_power:00: freeze
Mar 11 19:31:33 [kernel] acpi acpi_system:00: freeze
Mar 11 19:31:33 [kernel] Disabling non-boot CPUs ...
Mar 11 19:31:33 [kernel] kvm: disabling virtualization on CPU1
Mar 11 19:31:33 [kernel] CPU 1 is now offline
Mar 11 19:31:33 [kernel] SMP alternatives: switching to UP code
Mar 11 19:31:33 [kernel] PM: Removing info for No Bus:cpu1
Mar 11 19:31:33 [kernel] PM: Removing info for No Bus:msr1
Mar 11 19:31:33 [kernel] CPU1 is down
Mar 11 19:31:33 [kernel] swsusp debug: Waiting for 5 seconds.
Mar 11 19:31:33 [kernel] Enabling non-boot CPUs ...
Mar 11 19:31:33 [kernel] <NULL>: before notifier CPU_UP_PREPARE.

Hung here.

Mar 11 19:31:33 [kernel] <NULL>: after notifier CPU_UP_PREPARE.
Mar 11 19:31:33 [kernel] SMP alternatives: switching to SMP code
Mar 11 19:31:33 [kernel] Booting processor 1/1 eip 3000
Mar 11 19:31:33 [kernel] CPU 1 irqstacks, hard=c0388000 soft=c0386000
Mar 11 19:31:33 [kernel] Initializing CPU#1
Mar 11 19:31:33 [kernel] Calibrating delay using timer specific routine.. 3663.72 BogoMIPS (lpj=6103555)
Mar 11 19:31:33 [kernel] CPU: After generic identify, caps: bfe9fbff 00100000 00000000 00000000 0000c1a9 00000000 00000000
Mar 11 19:31:33 [kernel] monitor/mwait feature present.
Mar 11 19:31:33 [kernel] CPU: L1 I cache: 32K, L1 D cache: 32K
Mar 11 19:31:33 [kernel] CPU: L2 cache: 2048K
Mar 11 19:31:33 [kernel] CPU: Physical Processor ID: 0
Mar 11 19:31:33 [kernel] CPU: Processor Core ID: 1
Mar 11 19:31:33 [kernel] CPU: After all inits, caps: bfe9fbff 00100000 00000000 00002940 0000c1a9 00000000 00000000
Mar 11 19:31:33 [kernel] CPU1: Intel Genuine Intel(R) CPU           T2400  @ 1.83GHz stepping 08
Mar 11 19:31:33 [kernel] <NULL>: after __cpu_up
Mar 11 19:31:33 [kernel] <NULL>: before notifier CPU_ONLINE.
Mar 11 19:31:33 [kernel] kvm: enabling virtualization on CPU1
Mar 11 19:31:33 [kernel] Switched to high resolution mode on CPU 1
Mar 11 19:31:33 [kernel] PM: Adding info for No Bus:cpu1
Mar 11 19:31:33 [kernel] PM: Adding info for No Bus:msr1
Mar 11 19:31:33 [kernel] <NULL>: after notifier CPU_ONLINE.
Mar 11 19:31:33 [kernel] CPU1 is up
Mar 11 19:31:33 [kernel] acpi acpi_system:00: resuming
Mar 11 19:31:33 [kernel] button button_power:00: resuming
Mar 11 19:31:33 [kernel] processor ACPI0007:00: resuming
Mar 11 19:31:33 [kernel] processor ACPI0007:01: resuming
#3 lore raw
Quoted 24 lines · Thomas Meyer
Quoted 4 lines · Rafael J. Wysocki

Could you please put some printk()s in kernel/cpu.c:_cpu_up() to see where it gets stuck? I bet one of the notifiers goes to sleep (cpufreq, maybe).

Here we go (ok. i forgot __FUNCTION__ ...):

Mar 11 19:31:33 [kernel] ac ACPI0003:00: freeze
Mar 11 19:31:33 [kernel] acpi device:00: freeze
Mar 11 19:31:33 [kernel] processor ACPI0007:01: freeze
Mar 11 19:31:33 [kernel] processor ACPI0007:00: freeze
Mar 11 19:31:33 [kernel] button button_power:00: freeze
Mar 11 19:31:33 [kernel] acpi acpi_system:00: freeze
Mar 11 19:31:33 [kernel] Disabling non-boot CPUs ...
Mar 11 19:31:33 [kernel] kvm: disabling virtualization on CPU1
Mar 11 19:31:33 [kernel] CPU 1 is now offline
Mar 11 19:31:33 [kernel] SMP alternatives: switching to UP code
Mar 11 19:31:33 [kernel] PM: Removing info for No Bus:cpu1
Mar 11 19:31:33 [kernel] PM: Removing info for No Bus:msr1
Mar 11 19:31:33 [kernel] CPU1 is down
Mar 11 19:31:33 [kernel] swsusp debug: Waiting for 5 seconds.
Mar 11 19:31:33 [kernel] Enabling non-boot CPUs ...
Mar 11 19:31:33 [kernel] <NULL>: before notifier CPU_UP_PREPARE.
Quoted 1 line

Hung here.

This means that one of the notifiers had not returned before you pressed the button. Now the question is which one (there are many).

I don't know if there's any nicer way to find out that, but I usually hack kernel/sys.c:notifier_call_chain() to print nb->notifier_call (as a pointer) before the call is made. Then I write down the address of the last one called before the hang/oops and use gdb to check which function it points to.

Greetings,
Rafael

#4 lore raw
In reply to Rafael J. Wysocki

Rafael J. Wysocki napsal(a):

Quoted 8 lines · Thomas Meyer
Quoted 7 lines

Suspend to disk doesn't work on my laptop.

The suspend seems to hang while enabling the non-boot cpus again.

with platform = "test" and state = "disk" i get this:

Enabling non-boot CPUs ...

...

Quoted 3 lines

Could you please put some printk()s in kernel/cpu.c:_cpu_up() to see where it gets stuck? I bet one of the notifiers goes to sleep (cpufreq, maybe).

Hi,
I see the same problem - 2.6.21-rc3 with NO_HZ set (tickless), short printk trace

enable_nonboot_cpus
_cpu_up
raw_notifier_callchain (CPU_UP_PREPARE)

    ...
    update_sched_domains
     detach_destroy_domains
       [waits here] --> synchronize_sched (==synchronize_rcu)

Milan

#5 lore raw
Quoted 15 lines · Milan Broz

Rafael J. Wysocki napsal(a):

Quoted 8 lines · Thomas Meyer
Quoted 7 lines

Suspend to disk doesn't work on my laptop.

The suspend seems to hang while enabling the non-boot cpus again.

with platform = "test" and state = "disk" i get this:

Hi,
I see the same problem - 2.6.21-rc3 with NO_HZ set (tickless), short printk trace

Quoted 9 lines

enable_nonboot_cpus
_cpu_up
raw_notifier_callchain (CPU_UP_PREPARE)

    ...
    update_sched_domains
     detach_destroy_domains
       [waits here] --> synchronize_sched (==synchronize_rcu)

Maybe this helps:

Mar 11 20:03:56 [kernel] PM: Removing info for No Bus:msr1
Mar 11 20:03:56 [kernel] CPU1 is down
Mar 11 20:03:56 [kernel] swsusp debug: Waiting for 5 seconds.
Mar 11 20:03:56 [kernel] Enabling non-boot CPUs ...
Mar 11 20:03:56 [kernel] _cpu_up: before notifier CPU_UP_PREPARE.
Mar 11 20:03:56 [kernel] migration_call: Hi!
Mar 11 20:03:56 [kernel] rcu_cpu_notify: Hi!
Mar 11 20:03:56 [kernel] timer_cpu_notify: Hi!
Mar 11 20:03:56 [kernel] hrtimer_cpu_notify: Hi!
Mar 11 20:03:56 [kernel] cpu_callback: Hi!
Mar 11 20:03:56 [kernel] workqueue_cpu_callback: Hi!
Mar 11 20:03:56 [kernel] topology_cpu_callback: Hi!

Guess what? Hang!

Mar 11 20:03:56 [kernel] _cpu_up: after notifier CPU_UP_PREPARE.
Mar 11 20:03:56 [kernel] SMP alternatives: switching to SMP code
Mar 11 20:03:56 [kernel] Booting processor 1/1 eip 3000
Mar 11 20:03:56 [kernel] CPU 1 irqstacks, hard=c0388000 soft=c0386000
Mar 11 20:03:56 [kernel] Initializing CPU#1

I use NO_HZ, too. Will try without it.

With kind regards
thomas

#6 lore raw
Quoted 42 lines · Thomas Meyer
Quoted 16 lines · Milan Broz

Rafael J. Wysocki napsal(a):

Quoted 8 lines · Thomas Meyer
Quoted 7 lines

Suspend to disk doesn't work on my laptop.

The suspend seems to hang while enabling the non-boot cpus again.

with platform = "test" and state = "disk" i get this:

Hi,
I see the same problem - 2.6.21-rc3 with NO_HZ set (tickless), short printk trace

Quoted 9 lines

enable_nonboot_cpus
_cpu_up
raw_notifier_callchain (CPU_UP_PREPARE)

    ...
    update_sched_domains
     detach_destroy_domains
       [waits here] --> synchronize_sched (==synchronize_rcu)

Maybe this helps:

Mar 11 20:03:56 [kernel] PM: Removing info for No Bus:msr1
Mar 11 20:03:56 [kernel] CPU1 is down
Mar 11 20:03:56 [kernel] swsusp debug: Waiting for 5 seconds.
Mar 11 20:03:56 [kernel] Enabling non-boot CPUs ...
Mar 11 20:03:56 [kernel] _cpu_up: before notifier CPU_UP_PREPARE.
Mar 11 20:03:56 [kernel] migration_call: Hi!
Mar 11 20:03:56 [kernel] rcu_cpu_notify: Hi!
Mar 11 20:03:56 [kernel] timer_cpu_notify: Hi!
Mar 11 20:03:56 [kernel] hrtimer_cpu_notify: Hi!
Mar 11 20:03:56 [kernel] cpu_callback: Hi!
Mar 11 20:03:56 [kernel] workqueue_cpu_callback: Hi!
Mar 11 20:03:56 [kernel] topology_cpu_callback: Hi!
Quoted 1 line

Guess what? Hang!

Hm, you may be hitting the problem described in this thread:
http://lkml.org/lkml/2007/3/6/364 .

Can you please apply the patch at http://lkml.org/lkml/2007/3/7/255
and see if that helps?

Rafael

#7 lore raw

Sorry the systems hangs here:

Mar 11 20:55:46 [kernel] CPU 1 is now offline
Mar 11 20:55:46 [kernel] SMP alternatives: switching to UP code
Mar 11 20:55:46 [kernel] PM: Removing info for No Bus:cpu1
Mar 11 20:55:46 [kernel] PM: Removing info for No Bus:msr1
Mar 11 20:55:46 [kernel] CPU1 is down
Mar 11 20:55:46 [kernel] swsusp debug: Waiting for 5 seconds.
Mar 11 20:55:46 [kernel] Enabling non-boot CPUs ...
Mar 11 20:55:46 [kernel] _cpu_up: before notifier CPU_UP_PREPARE.
Mar 11 20:55:46 [kernel] migration_call: Hi!
Mar 11 20:55:46 [kernel] rcu_cpu_notify: Hi!
Mar 11 20:55:46 [kernel] timer_cpu_notify: Hi!
Mar 11 20:55:46 [kernel] hrtimer_cpu_notify: Hi!
Mar 11 20:55:46 [kernel] cpu_callback: Hi!
Mar 11 20:55:46 [kernel] synchronize_rcu: Befor wait

---> System hangs here. After the first "Before wait" message.

- Last output repeated twice -

Mar 11 20:55:46 [kernel] synchronize_rcu: After wait
Mar 11 20:55:46 [kernel] workqueue_cpu_callback: Hi!
Mar 11 20:55:46 [kernel] topology_cpu_callback: Hi!
Mar 11 20:55:46 [kernel] _cpu_up: after notifier CPU_UP_PREPARE.
Mar 11 20:55:46 [kernel] SMP alternatives: switching to SMP code
Mar 11 20:55:46 [kernel] Booting processor 1/1 eip 3000
Mar 11 20:55:46 [kernel] CPU 1 irqstacks, hard=c0388000 soft=c0386000
Mar 11 20:55:46 [kernel] Initializing CPU#1
#8 lore raw
In reply to Milan Broz
Quoted 8 lines · Milan Broz

Rafael J. Wysocki napsal(a):

Quoted 6 lines · Thomas Meyer
Quoted 5 lines

Suspend to disk doesn't work on my laptop.

The suspend seems to hang while enabling the non-boot cpus again.

with platform = "test" and state = "disk" i get this:

Quoted 8 lines
Quoted 1 line
Quoted 1 line

Enabling non-boot CPUs ...

...

Quoted 3 lines

Could you please put some printk()s in kernel/cpu.c:_cpu_up() to see where it gets stuck? I bet one of the notifiers goes to sleep (cpufreq, maybe).

Hi,
I see the same problem - 2.6.21-rc3 with NO_HZ set (tickless),

Ah, NO_HZ. Thomas Gleixner's address added to the Cc list.

Quoted 9 lines

short printk trace

enable_nonboot_cpus
_cpu_up
raw_notifier_callchain (CPU_UP_PREPARE)

    ...
    update_sched_domains
     detach_destroy_domains
       [waits here] --> synchronize_sched (==synchronize_rcu)

Well, I think the call to wait_for_completion() does not return, probably because the task supposed to complete the completion is frozen at this point. Can you please try to confirm that it gets stuck on wait_for_completion() in synchronize_rcu()?

Rafael

#9 lore raw

Rafael J. Wysocki:

Quoted 12 lines

Ah, NO_HZ. Thomas Gleixner's address added to the Cc list.

Quoted 9 lines

short printk trace

enable_nonboot_cpus
_cpu_up
raw_notifier_callchain (CPU_UP_PREPARE)

    ...
    update_sched_domains
     detach_destroy_domains
       [waits here] --> synchronize_sched (==synchronize_rcu)
Quoted 4 lines

Well, I think the call to wait_for_completion() does not return, probably because the task supposed to complete the completion is frozen at this point. Can you please try to confirm that it gets stuck on wait_for_completion() in synchronize_rcu()?

Yes, it's in wait_for_completion() in synchronize_rcu(). As noted in some previous mail, it will wake up after event - key press etc.

Patch in http://lkml.org/lkml/2007/3/7/255 solves different problem. I added it to my quilt and applied anyway -> no change.

Milan

#10 lore raw
Quoted 18 lines · Milan Broz

Rafael J. Wysocki:

Quoted 12 lines

Ah, NO_HZ. Thomas Gleixner's address added to the Cc list.

Quoted 9 lines

short printk trace

enable_nonboot_cpus
_cpu_up
raw_notifier_callchain (CPU_UP_PREPARE)

    ...
    update_sched_domains
     detach_destroy_domains
       [waits here] --> synchronize_sched (==synchronize_rcu)
Quoted 4 lines

Well, I think the call to wait_for_completion() does not return, probably because the task supposed to complete the completion is frozen at this point. Can you please try to confirm that it gets stuck on wait_for_completion() in synchronize_rcu()?

Quoted 6 lines

Yes, it's in wait_for_completion() in synchronize_rcu(). As noted in some previous mail, it will wake up after event - key press etc.

Patch in http://lkml.org/lkml/2007/3/7/255 solves different problem. I added it to my quilt and applied anyway -> no change.

Does the problem go away if NO_HZ is unset?

Rafael

#11 lore raw
Quoted 26 lines · Rafael J. Wysocki
Quoted 24 lines · Milan Broz

Rafael J. Wysocki:

Quoted 13 lines

Ah, NO_HZ. Thomas Gleixner's address added to the Cc list.

Quoted 10 lines

short printk trace

enable_nonboot_cpus
_cpu_up
raw_notifier_callchain (CPU_UP_PREPARE)

    ...
    update_sched_domains
     detach_destroy_domains
       [waits here] --> synchronize_sched (==synchronize_rcu)
Quoted 5 lines

Well, I think the call to wait_for_completion() does not return, probably because the task supposed to complete the completion is frozen at this point. Can you please try to confirm that it gets stuck on wait_for_completion() in synchronize_rcu()?

Yes, it's in wait_for_completion() in synchronize_rcu(). As noted in some previous mail, it will wake up after event - key press etc.

Quoted 6 lines
Quoted 3 lines

Patch in http://lkml.org/lkml/2007/3/7/255 solves different problem. I added it to my quilt and applied anyway -> no change.

Does the problem go away if NO_HZ is unset?

i tried to boot with nohz=off, but the problem did persist.

#12 lore raw
Quoted 27 lines · Thomas Meyer
Quoted 26 lines · Rafael J. Wysocki
Quoted 24 lines · Milan Broz

Rafael J. Wysocki:

Quoted 13 lines

Ah, NO_HZ. Thomas Gleixner's address added to the Cc list.

Quoted 10 lines

short printk trace

enable_nonboot_cpus
_cpu_up
raw_notifier_callchain (CPU_UP_PREPARE)

    ...
    update_sched_domains
     detach_destroy_domains
       [waits here] --> synchronize_sched (==synchronize_rcu)
Quoted 5 lines

Well, I think the call to wait_for_completion() does not return, probably because the task supposed to complete the completion is frozen at this point. Can you please try to confirm that it gets stuck on wait_for_completion() in synchronize_rcu()?

Yes, it's in wait_for_completion() in synchronize_rcu(). As noted in some previous mail, it will wake up after event - key press etc.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Patch in http://lkml.org/lkml/2007/3/7/255 solves different problem. I added it to my quilt and applied anyway -> no change.

Does the problem go away if NO_HZ is unset?

i tried to boot with nohz=off, but the problem did persist.

Okay, but could you please compile the kernel without NO_HZ and retest?

Rafael

#13 lore raw
Quoted 4 lines · Rafael J. Wysocki

Okay, but could you please compile the kernel without NO_HZ and retest?

Sure.
But i get the same behaviour:

Mar 11 21:42:07 [kernel] processor ACPI0007:00: freeze
Mar 11 21:42:07 [kernel] button button_power:00: freeze
Mar 11 21:42:07 [kernel] acpi acpi_system:00: freeze
Mar 11 21:42:07 [kernel] Disabling non-boot CPUs ...
Mar 11 21:42:07 [kernel] kvm: disabling virtualization on CPU1
Mar 11 21:42:07 [kernel] synchronize_rcu: Befor wait
Mar 11 21:42:07 [kernel] synchronize_rcu: After wait
Mar 11 21:42:07 [kernel] CPU 1 is now offline
Mar 11 21:42:07 [kernel] SMP alternatives: switching to UP code
Mar 11 21:42:07 [kernel] PM: Removing info for No Bus:cpu1
Mar 11 21:42:07 [kernel] PM: Removing info for No Bus:msr1
Mar 11 21:42:07 [kernel] CPU1 is down
Mar 11 21:42:07 [kernel] swsusp debug: Waiting for 5 seconds.
Mar 11 21:42:07 [kernel] Enabling non-boot CPUs ...
Mar 11 21:42:07 [kernel] _cpu_up: before notifier CPU_UP_PREPARE.
Mar 11 21:42:07 [kernel] migration_call: Hi!
Mar 11 21:42:07 [kernel] rcu_cpu_notify: Hi!
Mar 11 21:42:07 [kernel] timer_cpu_notify: Hi!
Mar 11 21:42:07 [kernel] hrtimer_cpu_notify: Hi!
Mar 11 21:42:07 [kernel] cpu_callback: Hi!
Mar 11 21:42:07 [kernel] synchronize_rcu: Befor wait

----> Hang
Why does this message appear two times?

- Last output repeated twice -

Mar 11 21:42:07 [kernel] synchronize_rcu: After wait
Mar 11 21:42:07 [kernel] workqueue_cpu_callback: Hi!
Mar 11 21:42:07 [kernel] topology_cpu_callback: Hi!
Mar 11 21:42:07 [kernel] _cpu_up: after notifier CPU_UP_PREPARE.
Mar 11 21:42:07 [kernel] SMP alternatives: switching to SMP code
Mar 11 21:42:07 [kernel] Booting processor 1/1 eip 3000
Mar 11 21:42:07 [kernel] CPU 1 irqstacks, hard=c0387000 soft=c0385000
Mar 11 21:42:07 [kernel] Initializing CPU#1
Mar 11 21:42:07 [kernel] Calibrating delay using timer specific routine.. 3662.97 BogoMIPS (lpj=6102352)
Mar 11 21:42:07 [kernel] CPU: After generic identify, caps: bfe9fbff 00100000 00000000 00000000 0000c1a9 00000000 00000000
Mar 11 21:42:07 [kernel] monitor/mwait feature present.
Mar 11 21:42:07 [kernel] CPU: L1 I cache: 32K, L1 D cache: 32K
Mar 11 21:42:07 [kernel] CPU: L2 cache: 2048K
Mar 11 21:42:07 [kernel] CPU: Physical Processor ID: 0
Mar 11 21:42:07 [kernel] CPU: Processor Core ID: 1
Mar 11 21:42:07 [kernel] CPU: After all inits, caps: bfe9fbff 00100000 00000000 00002940 0000c1a9 00000000 00000000
Mar 11 21:42:07 [kernel] CPU1: Intel Genuine Intel(R) CPU           T2400  @ 1.83GHz stepping 08
Mar 11 21:42:07 [kernel] synchronize_rcu: After wait
Mar 11 21:42:07 [kernel] _cpu_up: after __cpu_up
Mar 11 21:42:07 [kernel] _cpu_up: before notifier CPU_ONLINE.
#14 lore raw
In reply to Thomas Meyer

Thomas Meyer napsal(a):

Quoted 27 lines
Quoted 26 lines · Rafael J. Wysocki
Quoted 24 lines · Milan Broz

Rafael J. Wysocki:

Quoted 13 lines

Ah, NO_HZ. Thomas Gleixner's address added to the Cc list.

Quoted 10 lines

short printk trace

enable_nonboot_cpus
_cpu_up
raw_notifier_callchain (CPU_UP_PREPARE)

    ...
    update_sched_domains
     detach_destroy_domains
       [waits here] --> synchronize_sched (==synchronize_rcu)
Quoted 5 lines

Well, I think the call to wait_for_completion() does not return, probably because the task supposed to complete the completion is frozen at this point. Can you please try to confirm that it gets stuck on wait_for_completion() in synchronize_rcu()?

Yes, it's in wait_for_completion() in synchronize_rcu(). As noted in some previous mail, it will wake up after event - key press etc.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Patch in http://lkml.org/lkml/2007/3/7/255 solves different problem. I added it to my quilt and applied anyway -> no change.

Does the problem go away if NO_HZ is unset?

i tried to boot with nohz=off, but the problem did persist.

Hmmmm, both variants (nohz=off or recompiled kernel without NO_HZ) works for me.

Milan

#15 lore raw
Quoted 42 lines · Milan Broz

Thomas Meyer napsal(a):

Quoted 40 lines
Quoted 38 lines · Rafael J. Wysocki
Quoted 24 lines · Milan Broz

Rafael J. Wysocki:

Quoted 15 lines

Ah, NO_HZ. Thomas Gleixner's address added to the Cc list.

Quoted 11 lines

short printk trace

enable_nonboot_cpus
_cpu_up
raw_notifier_callchain (CPU_UP_PREPARE)

    ...
    update_sched_domains
     detach_destroy_domains
       [waits here] --> synchronize_sched (==synchronize_rcu)
Quoted 6 lines

Well, I think the call to wait_for_completion() does not return, probably because the task supposed to complete the completion is frozen at this point. Can you please try to confirm that it gets stuck on wait_for_completion() in synchronize_rcu()?

Quoted 8 lines

Yes, it's in wait_for_completion() in synchronize_rcu(). As noted in some previous mail, it will wake up after event - key press etc.

Patch in http://lkml.org/lkml/2007/3/7/255 solves different problem. I added it to my quilt and applied anyway -> no change.

Does the problem go away if NO_HZ is unset?

Quoted 7 lines
Quoted 2 lines

i tried to boot with nohz=off, but the problem did persist.

Hmmmm, both variants (nohz=off or recompiled kernel without NO_HZ) works for me.

Milan

I got a working config:

Without hrtimers and without nohz it is working!

With hrtimers and without nohz it is not working
With hrtimers and with nohz it is not working

Now i want to test: without hrtimers and with nohz.

#16 lore raw
In reply to Milan Broz
Quoted 31 lines · Milan Broz

Thomas Meyer napsal(a):

Quoted 30 lines
Quoted 29 lines · Rafael J. Wysocki
Quoted 27 lines · Milan Broz

Rafael J. Wysocki:

Quoted 13 lines

Ah, NO_HZ. Thomas Gleixner's address added to the Cc list.

Quoted 10 lines

short printk trace

enable_nonboot_cpus
_cpu_up
raw_notifier_callchain (CPU_UP_PREPARE)

    ...
    update_sched_domains
     detach_destroy_domains
       [waits here] --> synchronize_sched (==synchronize_rcu)
Quoted 5 lines

Well, I think the call to wait_for_completion() does not return, probably because the task supposed to complete the completion is frozen at this point. Can you please try to confirm that it gets stuck on wait_for_completion() in synchronize_rcu()?

Yes, it's in wait_for_completion() in synchronize_rcu(). As noted in some previous mail, it will wake up after event - key press etc.

Patch in http://lkml.org/lkml/2007/3/7/255 solves different problem. I added it to my quilt and applied anyway -> no change.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Does the problem go away if NO_HZ is unset?

i tried to boot with nohz=off, but the problem did persist.

Hmmmm, both variants (nohz=off or recompiled kernel without NO_HZ) works for me.

Definitely something strange is going on here.

I think we need an advice from someone who knows the RCU internals.

Andrew, could you please tell me whom I should ask?

Rafael

#17 lore raw
Quoted 19 lines · Rafael J. Wysocki
Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
Quoted 4 lines
    update_sched_domains
     detach_destroy_domains
       [waits here] --> synchronize_sched (==synchronize_rcu)

Well, I think the call to wait_for_completion() does not return, probably because the task supposed to complete the completion is frozen at this point. Can you please try to confirm that it gets stuck on wait_for_completion() in synchronize_rcu()?

Quoted 7 lines

Yes, it's in wait_for_completion() in synchronize_rcu(). As noted in some previous mail, it will wake up after event - key press etc.

Patch in http://lkml.org/lkml/2007/3/7/255 solves different problem. I added it to my quilt and applied anyway -> no change.

Does the problem go away if NO_HZ is unset?

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

i tried to boot with nohz=off, but the problem did persist.

Hmmmm, both variants (nohz=off or recompiled kernel without NO_HZ) works for me.

Definitely something strange is going on here.

I think we need an advice from someone who knows the RCU internals.

RCU synchronization depends on the timer interrupt. Which kernel version are you guys talking about ?

tglx

#18 lore raw
Quoted 30 lines · Thomas Gleixner
Quoted 28 lines · Rafael J. Wysocki
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 11 lines
Quoted 5 lines
    update_sched_domains
     detach_destroy_domains
       [waits here] --> synchronize_sched (==synchronize_rcu)

Well, I think the call to wait_for_completion() does not return, probably because the task supposed to complete the completion is frozen at this point. Can you please try to confirm that it gets stuck on wait_for_completion() in synchronize_rcu()?

Yes, it's in wait_for_completion() in synchronize_rcu(). As noted in some previous mail, it will wake up after event - key press etc.

Patch in http://lkml.org/lkml/2007/3/7/255 solves different problem. I added it to my quilt and applied anyway -> no change.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Does the problem go away if NO_HZ is unset?

i tried to boot with nohz=off, but the problem did persist.

Hmmmm, both variants (nohz=off or recompiled kernel without NO_HZ) works for me.

Definitely something strange is going on here.

Quoted 8 lines
Quoted 2 lines

I think we need an advice from someone who knows the RCU internals.

RCU synchronization depends on the timer interrupt. Which kernel version are you guys talking about ?

tglx

I talk about be521466feb3bb1cd89de82a2b1d080e9ebd3cb6 (2.6.21-rc3+).

The worst config is with nohz and without hrtimers: the kernel even doesn't come back after pressing the power key.

But i stay with without nohz and without hrtimers for now, because here the suspend to disk works.