Skip to contentSkip to conversation

Power: fix suspend vt regression

PATCH In review Jiri Slaby 10 participants 22 messages Aug 11, 2009 – Sep 11, 2009

Subsystems (4)
Char and misc drivers Suspend to RAM Hibernation CPU hotplug
lore
Share & tools
lore

Jiri Slaby

author
#0 lore raw

PATCH 1/1 Power: fix suspend vt regression

vt_waitactive no longer accepts console parameter as console-1 since commit "vt: add an event interface". It expects console number directly (as viewed by userspace -- counting from 1).

Fix a deadlock suspend regression by redefining SUSPEND_CONSOLE to be MAX_NR_CONSOLES, not MAX_NR_CONSOLES-1.

Signed-off-by Jiri Slaby <[email protected]>
Cc Alan Cox <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc "Rafael J. Wysocki" <[email protected]>
---
 kernel/power/console.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/power/console.c +1 -1
index 5187136..6592a57 100644
@@ -11,7 +11,7 @@
 #include "power.h"
 
 #if defined(CONFIG_VT) && defined(CONFIG_VT_CONSOLE)
-#define SUSPEND_CONSOLE	(MAX_NR_CONSOLES-1)
+#define SUSPEND_CONSOLE	MAX_NR_CONSOLES
 
 static int orig_fgconsole, orig_kmsg;
Signature · 1 line
1.6.3.3
#1 lore raw
Quoted 3 lines · Jiri Slaby

vt_waitactive no longer accepts console parameter as console-1 since commit "vt: add an event interface". It expects console number directly (as viewed by userspace -- counting from 1).

As the event interface code is only in -next and not in mainline, this doesn't pertain to Linus's current tree, right?

thanks,

greg k-h

author
#2 lore raw
Quoted 7 lines · Greg KH
Quoted 3 lines · Jiri Slaby

vt_waitactive no longer accepts console parameter as console-1 since commit "vt: add an event interface". It expects console number directly (as viewed by userspace -- counting from 1).

As the event interface code is only in -next and not in mainline, this doesn't pertain to Linus's current tree, right?

Correct. But please ignore this one. The comment above is correct, but the change should have been done one level deeper. A new patch for this will follow.

However there is still a race or something. Sometimes the suspend goes through, sometimes it doesn't. I will investigate this further.

author
#3 lore raw

PATCH 1/1 Power: fix suspend vt regression

vt_waitactive no longer accepts console parameter as console-1 since commit "vt: add an event interface". It expects console number directly (as viewed by userspace -- counting from 1).

Fix a deadlock suspend regression by redefining adding one to vt in vt_move_to_console.

Signed-off-by Jiri Slaby <[email protected]>
Cc Alan Cox <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc "Rafael J. Wysocki" <[email protected]>
---
 drivers/char/vt_ioctl.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/char/vt_ioctl.c +1 -1
index 0fceb8f..e3d4d13 100644
@@ -1554,7 +1554,7 @@ int vt_move_to_console(unsigned int vt, int alloc)
 		return -EIO;
 	}
 	release_console_sem();
-	if (vt_waitactive(vt)) {
+	if (vt_waitactive(vt + 1)) {
 		pr_debug("Suspend: Can't switch VCs.");
 		return -EINTR;
 	}
Signature · 1 line
1.6.3.3
author
#4 lore raw

suspend race -next regression [Was: Power: fix suspend vt regression]

In reply to Jiri Slaby
Quoted 2 lines · Jiri Slaby

However there is still a race or something. Sometimes the suspend goes through, sometimes it doesn't. I will investigate this further.

Hmm, this took a loong time to track down a bit. Code instrumentation by outb(XX, 0x80) usually caused the issue to disappear.

However I found out that it's caused by might_sleep() calls in flush_workqueue() and flush_cpu_workqueue(). I.e. it looks like there is a task which deadlocks/spins forever. If we won't reschedule to it, suspend proceeds.

I replaced the latter might_sleep() by show_state() and removed refrigerated tasks afterwards. The thing is that I don't know if the prank task is there. I need a scheduler to store "next" task pid or whatever to see what it picked as "next" and so what will run due to might_sched(). I can then show it on port 80 display and read it when the hangup occurs.

Depending on which might_sleep(), either flush_workqueue() never (well, at least in next 5 minutes) proceeds to for_each_cpu() or wait_for_completion() in flush_cpu_workqueue() never returns.

It's a regression against some -rc1 based -next tree. Bisection impossible, suspend needs to be run even 7 times before it occurs. Maybe a s/might_sleep/yield/ could make it happen earlier (going to try)?

Attaching the filtered show_task() output:
PM: Syncing filesystems ... done.
Freezing user space processes ... (elapsed 0.00 seconds) done. Freezing remaining freezable tasks ... (elapsed 0.00 seconds) done. Suspending console(s) (use no_console_suspend to debug) sd 1:0:0:0: [sdb] Synchronizing SCSI cache
sd 1:0:0:0: [sdb] Stopping disk
sd 0:0:0:0: [sda] Synchronizing SCSI cache
sd 0:0:0:0: [sda] Stopping disk
parport_pc 00:08: disabled
serial 00:06: disabled
ath5k 0000:04:00.0: PCI INT A disabled
ACPI handle has no context!
ehci_hcd 0000:00:1d.7: PCI INT A disabled
uhci_hcd 0000:00:1d.2: PCI INT D disabled
uhci_hcd 0000:00:1d.1: PCI INT B disabled
uhci_hcd 0000:00:1d.0: PCI INT A disabled
HDA Intel 0000:00:1b.0: PCI INT A disabled
ehci_hcd 0000:00:1a.7: PCI INT D disabled
uhci_hcd 0000:00:1a.2: PCI INT C disabled
uhci_hcd 0000:00:1a.1: PCI INT B disabled
uhci_hcd 0000:00:1a.0: PCI INT A disabled
e1000e 0000:00:19.0: PCI INT A disabled
e1000e 0000:00:19.0: PME# enabled
e1000e 0000:00:19.0: wake-up capability enabled by ACPI ACPI handle has no context!
ehci_hcd 0000:00:1d.7: PME# disabled
ehci_hcd 0000:00:1a.7: PME# disabled
ACPI: Preparing to enter system sleep state S3
Disabling non-boot CPUs ...
task PC stack pid father
init D ffff8801cb8ffb60 0 1 0 0x00000000
ffff8801cb86fd88 0000000000000086 00000001cb86fd88 ffff8801cb854000 ffff8801cb854000 0000000000000ba2 ffff8801cb86fd28 00000000ffff36c7 ffff8801cb8542a8 000000000000df68 00000000000118c0 ffff8801cb8542a8

Call Trace:
 [<ffffffff8105d5fd>] refrigerator+0xad/0x100
 [<ffffffff8104eb64>] get_signal_to_deliver+0x84/0x380
 [<ffffffff8100b36c>] do_notify_resume+0xbc/0x7c0
 [<ffffffff8105ebe9>] ? ktime_get_ts+0xa9/0xe0
 [<ffffffff810d2c18>] ? poll_select_copy_remaining+0xf8/0x150
 [<ffffffff810d44d4>] ? sys_select+0x54/0x110
 [<ffffffff8100bfc5>] sysret_signal+0x6d/0xb7

kthreadd S 0000000000000001 0 2 0 0x00000000
ffff8801cb871f00 0000000000000046 ffff8801cb871f40 ffffffff8100ce32 0000000000000001 ffff8801cb8546b0 ffff8801cb8546b0 ffff8801c5dddcc0 ffff8801cb854958 000000000000df68 00000000000118c0 ffff8801cb854958

Call Trace:
 [<ffffffff8100ce32>] ? kernel_thread+0x82/0xe0
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff810562a5>] kthreadd+0x115/0x120
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff81056190>] ? kthreadd+0x0/0x120
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

migration/0 S 0000000000000000 0 3 2 0x00000000
ffff8801cb875e70 0000000000000046 ffff8800282918c0 0000000000000001 ffff8801cb875df0 ffffffff81036bdd ffff8801c5fef730 00000000ffff367a ffff8801cb8592e8 000000000000df68 00000000000118c0 ffff8801cb8592e8

Call Trace:
 [<ffffffff81036bdd>] ? enqueue_task_fair+0x3d/0x80
 [<ffffffff8103bafe>] migration_thread+0x1ae/0x2c0
 [<ffffffff8103b950>] ? migration_thread+0x0/0x2c0
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

ksoftirqd/0 S ffff8801cb8596f0 0 4 2 0x00000000
ffff8801cb877ea0 0000000000000046 0000000000000000 00000000000118c0 0000000000000000 ffffffff81595920 0000000000000000 00000000ffff35ed ffff8801cb859998 000000000000df68 00000000000118c0 ffff8801cb859998

Call Trace:
 [<ffffffff8103760d>] ? default_wake_function+0xd/0x10
 [<ffffffff810450b5>] ksoftirqd+0xc5/0x100
 [<ffffffff81044ff0>] ? ksoftirqd+0x0/0x100
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

watchdog/0 S 0000000000000001 0 5 2 0x00000000
ffff8801cb879eb0 0000000000000046 ffff8801cb879e20 ffffffff810687c3 ffff8801cb85a080 ffff8800282118c0 ffff8801cb879ea0 ffffffff81037f5f ffff8801cb85a328 000000000000df68 00000000000118c0 ffff8801cb85a328

Call Trace:
 [<ffffffff810687c3>] ? rt_mutex_adjust_pi+0x73/0x80
 [<ffffffff81037f5f>] ? __sched_setscheduler+0x19f/0x420
 [<ffffffff81080a20>] ? watchdog+0x0/0x90
 [<ffffffff81080a20>] ? watchdog+0x0/0x90
 [<ffffffff81080a72>] watchdog+0x52/0x90
 [<ffffffff81080a20>] ? watchdog+0x0/0x90
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

migration/1 S 0000000000000001 0 6 2 0x00000000
ffff8801cb87de70 0000000000000046 ffff8800282118c0 0000000000000000 ffff8801cb87ddf0 ffffffff81036bdd ffff8801c9080200 00000000ffff367a ffff8801cb85a9d8 000000000000df68 00000000000118c0 ffff8801cb85a9d8

Call Trace:
 [<ffffffff81036bdd>] ? enqueue_task_fair+0x3d/0x80
 [<ffffffff8103bafe>] migration_thread+0x1ae/0x2c0
 [<ffffffff8103b950>] ? migration_thread+0x0/0x2c0
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

ksoftirqd/1 S ffff8801cb8600c0 0 7 2 0x00000000
ffff8801cb87fea0 0000000000000046 ffff8801cb860368 000000000000df68 00000000000118c0 ffff8801cb860370 ffff8801cb9197f0 00000000ffff35bf ffff8801cb860368 000000000000df68 00000000000118c0 ffff8801cb860368

Call Trace:
 [<ffffffff810450b5>] ksoftirqd+0xc5/0x100
 [<ffffffff81044ff0>] ? ksoftirqd+0x0/0x100
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

watchdog/1 S 0000000000000001 0 8 2 0x00000000
ffff8801cb885eb0 0000000000000046 ffff8801cb885e20 ffffffff810687c3 ffff8801cb860770 ffff8800282918c0 ffff8801cb885ea0 00000000fffedc26 ffff8801cb860a18 000000000000df68 00000000000118c0 ffff8801cb860a18

Call Trace:
 [<ffffffff810687c3>] ? rt_mutex_adjust_pi+0x73/0x80
 [<ffffffff81080a20>] ? watchdog+0x0/0x90
 [<ffffffff81080a20>] ? watchdog+0x0/0x90
 [<ffffffff81080a72>] watchdog+0x52/0x90
 [<ffffffff81080a20>] ? watchdog+0x0/0x90
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

events/0 S ffff8801cb8617b0 0 9 2 0x00000000
ffff8801cb88be60 0000000000000046 ffff8801cb88be10 ffffffff8128fcf5 ffffffff81595ef8 ffff8800282142c8 ffff8801cb88bde0 00000000ffff36e5 ffff8801cb861a58 000000000000df68 00000000000118c0 ffff8801cb861a58

Call Trace:
 [<ffffffff8128fcf5>] ? e1000_watchdog_task+0x185/0x6e0
 [<ffffffff81051c3f>] ? run_workqueue+0xcf/0x140
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

events/1 S ffff8801cb864140 0 10 2 0x00000000
ffff8801cb88de60 0000000000000046 ffff8801cb88ddb0 ffff8801cb88ddb0 ffff8801cb88de00 ffff8800282942c0 ffffffff8136fd80 00000000ffff37ee ffff8801cb8643e8 000000000000df68 00000000000118c0 ffff8801cb8643e8

Call Trace:
 [<ffffffff8136fd80>] ? linkwatch_event+0x0/0x30
 [<ffffffff81051c3f>] ? run_workqueue+0xcf/0x140
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

cpuset S ffff8801cb8647f0 0 11 2 0x00000000
ffff8801cb891e60 0000000000000046 ffff8801cb891de0 ffffffff81036826 0000000000000000 ffff8801cb8647f0 ffff8800282918c0 00000000fffedb2d ffff8801cb864a98 000000000000df68 00000000000118c0 ffff8801cb864a98

Call Trace:
 [<ffffffff81036826>] ? dequeue_task_fair+0x86/0x90
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

khelper S ffff8801cb865180 0 12 2 0x00000000
ffff8801cb893e60 0000000000000046 0000000000000611 ffff8801c9ae4000 ffffffff810518c0 0000000000000000 ffffffff8100ce90 00000000ffff35bf ffff8801cb865428 000000000000df68 00000000000118c0 ffff8801cb865428

Call Trace:
 [<ffffffff810518c0>] ? wait_for_helper+0x0/0x80
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20
 [<ffffffff81051c3f>] ? run_workqueue+0xcf/0x140
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

netns S ffff8801cb865830 0 15 2 0x00000000
ffff8801cb8b9e60 0000000000000046 ffff8801cb8b9de0 ffffffff81036826 0000000000000000 ffff8801cb865830 ffff8800282918c0 00000000fffedb2d ffff8801cb865ad8 000000000000df68 00000000000118c0 ffff8801cb865ad8

Call Trace:
 [<ffffffff81036826>] ? dequeue_task_fair+0x86/0x90
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

async/mgr S ffff8801cb8e3e80 0 19 2 0x00000000
ffff8801cb8e3e70 0000000000000046 ffff8801cb8e3de0 ffffffff81032332 0000000000000001 ffff8800282918c0 ffff8801cb8e3e00 ffffffff810323a8 ffff8801cb8c2468 000000000000df68 00000000000118c0 ffff8801cb8c2468

Call Trace:
 [<ffffffff81032332>] ? enqueue_task+0x32/0x80
 [<ffffffff810323a8>] ? activate_task+0x28/0x40
 [<ffffffff8105c6e5>] async_manager_thread+0x65/0x100
 [<ffffffff81037600>] ? default_wake_function+0x0/0x10
 [<ffffffff8105c680>] ? async_manager_thread+0x0/0x100
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

pm D ffff8801cb8c2870 0 20 2 0x00000000
ffff8801cb8e5e30 0000000000000046 0000000000000000 ffff8801c9044080 0000000000000000 0000000000000000 ffff8801cb8e5e60 ffffffff8142dc98 ffff8801cb8c2b18 000000000000df68 00000000000118c0 ffff8801cb8c2b18

Call Trace:
 [<ffffffff8142dc98>] ? thread_return+0x3e/0x726
 [<ffffffff8105d5fd>] refrigerator+0xad/0x100
 [<ffffffff81051da5>] worker_thread+0xf5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

kblockd/0 S ffff8801cb8df200 0 159 2 0x00000000
ffff8801cba77e60 0000000000000046 ffff8801ca5114c8 ffff880028214688 ffff8801cba77fd8 ffff8801cb8df200 ffff8801cba77df0 00000000ffff362d ffff8801cb8df4a8 000000000000df68 00000000000118c0 ffff8801cb8df4a8

Call Trace:
 [<ffffffff81051c3f>] ? run_workqueue+0xcf/0x140
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

kblockd/1 S ffff8801cb8df8b0 0 160 2 0x00000000
ffff8801cba7be60 0000000000000046 ffff8801ca4544c8 ffff880028294688 ffff8801cba7bfd8 ffff8801cb8df8b0 ffff8801cba7bdf0 00000000ffff362a ffff8801cb8dfb58 000000000000df68 00000000000118c0 ffff8801cb8dfb58

Call Trace:
 [<ffffffff81051c3f>] ? run_workqueue+0xcf/0x140
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

kacpid S ffff8801cba7e240 0 162 2 0x00000000
ffff8801cba9de60 0000000000000046 ffff8801cba9ddb0 ffff8801cba9ddb0 0000000000000000 0000000000000286 ffff8801cba9ddf0 0000000000000202 ffff8801cba7e4e8 000000000000df68 00000000000118c0 ffff8801cba7e4e8

Call Trace:
 [<ffffffff81051c3f>] ? run_workqueue+0xcf/0x140
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

kacpi_notify S ffff8801cba7e8f0 0 163 2 0x00000000
ffff8801cba9fe60 0000000000000046 ffff8801cba9fde0 ffffffff81036826 ffff8801cb854048 0000000000000286 ffff8801cba9fdf0 0000000000000202 ffff8801cba7eb98 000000000000df68 00000000000118c0 ffff8801cba7eb98

Call Trace:
 [<ffffffff81036826>] ? dequeue_task_fair+0x86/0x90
 [<ffffffff81051c3f>] ? run_workqueue+0xcf/0x140
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

kacpi_hotplug S ffff8801cba7f280 0 164 2 0x00000000
ffff8801cbaa3e60 0000000000000046 ffff8801cbaa3db0 ffff8801cbaa3db0 0000000000000000 0000000000000286 ffff8801cbaa3df0 0000000000000202 ffff8801cba7f528 000000000000df68 00000000000118c0 ffff8801cba7f528

Call Trace:
 [<ffffffff81051c3f>] ? run_workqueue+0xcf/0x140
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

ata/0 S ffff8801cba7f930 0 246 2 0x00000000
ffff8801cb8e7e60 0000000000000046 ffff8801cb8e7de0 ffffffff81036826 0000000000000000 ffff8801cba7f930 ffff8800282118c0 ffffffff81595920 ffff8801cba7fbd8 000000000000df68 00000000000118c0 ffff8801cba7fbd8

Call Trace:
 [<ffffffff81036826>] ? dequeue_task_fair+0x86/0x90
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

ata/1 S ffff8801cb970000 0 247 2 0x00000000
ffff8801cb8c1e60 0000000000000046 ffff8801cb8c1de0 ffffffff81036826 ffff8801cb8c1df0 ffff8801cb970000 ffff8800282918c0 00000000fffedb2d ffff8801cb9702a8 000000000000df68 00000000000118c0 ffff8801cb9702a8

Call Trace:
 [<ffffffff81036826>] ? dequeue_task_fair+0x86/0x90
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

ata_aux S ffff8801cb9706b0 0 248 2 0x00000000
ffff8801cb907e60 0000000000000046 ffff8801cb907de0 ffffffff81036826 0000000000000000 ffff8801cb9706b0 ffff8800282918c0 00000000fffedb2d ffff8801cb970958 000000000000df68 00000000000118c0 ffff8801cb970958

Call Trace:
 [<ffffffff81036826>] ? dequeue_task_fair+0x86/0x90
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

<removed refrigerated tasks>
aio/0 S ffff8801cba521c0 0 333 2 0x00000000
ffff8801cb9e1e60 0000000000000046 ffff8801cb9e1de0 ffffffff81036826 0000000000000000 ffff8801cba521c0 ffff8800282118c0 ffffffff81595920 ffff8801cba52468 000000000000df68 00000000000118c0 ffff8801cba52468

Call Trace:
 [<ffffffff81036826>] ? dequeue_task_fair+0x86/0x90
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

aio/1 S ffff8801cb9f4180 0 334 2 0x00000000
ffff8801cb911e60 0000000000000046 ffff8801cb911de0 ffffffff81036826 ffff8801cb911df0 ffff8801cb9f4180 ffff8800282918c0 00000000fffedb2d ffff8801cb9f4428 000000000000df68 00000000000118c0 ffff8801cb9f4428

Call Trace:
 [<ffffffff81036826>] ? dequeue_task_fair+0x86/0x90
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

crypto/0 S ffff8801cb9f4830 0 336 2 0x00000000
ffff8801cb937e60 0000000000000046 ffff8801cb937de0 ffffffff81036826 0000000000000000 ffff8801cb9f4830 ffff8800282118c0 ffffffff81595920 ffff8801cb9f4ad8 000000000000df68 00000000000118c0 ffff8801cb9f4ad8

Call Trace:
 [<ffffffff81036826>] ? dequeue_task_fair+0x86/0x90
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

crypto/1 S ffff8801cb9f51c0 0 337 2 0x00000000
ffff8801cbb47e60 0000000000000046 ffff8801cbb47de0 ffffffff81036826 ffff8801cbb47df0 ffff8801cb9f51c0 ffff8800282918c0 00000000fffedb2d ffff8801cb9f5468 000000000000df68 00000000000118c0 ffff8801cb9f5468

Call Trace:
 [<ffffffff81036826>] ? dequeue_task_fair+0x86/0x90
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

i915/0 S ffff8801cb9f5870 0 437 2 0x00000000
ffff8801cab79e60 0000000000000046 ffff8801cab79de0 ffff8801cb9bc820 ffff8801cbb77e68 ffff8801cb9bc800 ffff8801cbb77000 00000000ffff310a ffff8801cb9f5b18 000000000000df68 00000000000118c0 ffff8801cb9f5b18

Call Trace:
 [<ffffffff81051c3f>] ? run_workqueue+0xcf/0x140
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

i915/1 S ffff8801cb974730 0 438 2 0x00000000
ffff8801cab7be60 0000000000000046 ffff8801cab7bde0 ffff8801cb9bc820 ffff8801cbb77e68 ffff8801cb9bc800 ffff8801cbb77000 00000000ffff2f47 ffff8801cb9749d8 000000000000df68 00000000000118c0 ffff8801cb9749d8

Call Trace:
 [<ffffffff81051c3f>] ? run_workqueue+0xcf/0x140
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

scsi_eh_0 S ffff8801cb301ea0 0 482 2 0x00000000
ffff8801cb301e50 0000000000000046 ffff8801cb301dc0 ffffffff8116b067 ffff8801cab6c800 0000000000000000 ffff8801cb301dd0 ffffffff8123af02 ffff8801cb9204a8 000000000000df68 00000000000118c0 ffff8801cb9204a8

Call Trace:
 [<ffffffff8116b067>] ? kobject_put+0x27/0x60
 [<ffffffff8123af02>] ? put_device+0x12/0x20
 [<ffffffff8125584e>] scsi_error_handler+0x7e/0x430
 [<ffffffff812557d0>] ? scsi_error_handler+0x0/0x430
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

scsi_eh_1 S ffff8801cbbc7ea0 0 485 2 0x00000000
ffff8801cbbc7e50 0000000000000046 ffff8801cbbc7dc0 ffffffff8116b067 ffff8801cbbd1000 0000000000000000 ffff8801cbbc7dd0 ffffffff8123af02 ffff8801cb91e468 000000000000df68 00000000000118c0 ffff8801cb91e468

Call Trace:
 [<ffffffff8116b067>] ? kobject_put+0x27/0x60
 [<ffffffff8123af02>] ? put_device+0x12/0x20
 [<ffffffff8125584e>] scsi_error_handler+0x7e/0x430
 [<ffffffff812557d0>] ? scsi_error_handler+0x0/0x430
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

scsi_eh_2 S ffff8801cb313ea0 0 488 2 0x00000000
ffff8801cb313e50 0000000000000046 ffff8801cb313de0 ffffffff81032cfe ffff8801cb313de0 0000000000000282 ffff8801cb31c000 00000000ffff36e5 ffff8801cb91c428 000000000000df68 00000000000118c0 ffff8801cb91c428

Call Trace:
 [<ffffffff81032cfe>] ? __wake_up+0x4e/0x70
 [<ffffffff8125584e>] scsi_error_handler+0x7e/0x430
 [<ffffffff812557d0>] ? scsi_error_handler+0x0/0x430
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

scsi_eh_3 S ffff8801cbbc3ea0 0 491 2 0x00000000
ffff8801cbbc3e50 0000000000000046 ffff8801cbbc3dc0 ffffffff8116b067 ffff8801cbbd4000 0000000000000000 ffff8801cbbc3dd0 00000000ffff36e5 ffff8801cb9193e8 000000000000df68 00000000000118c0 ffff8801cb9193e8

Call Trace:
 [<ffffffff8116b067>] ? kobject_put+0x27/0x60
 [<ffffffff8125584e>] scsi_error_handler+0x7e/0x430
 [<ffffffff812557d0>] ? scsi_error_handler+0x0/0x430
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

scsi_eh_4 S ffff8801cb309ea0 0 494 2 0x00000000
ffff8801cb309e50 0000000000000046 ffff8801cb309de0 ffffffff81032cfe ffff8801cb309de0 0000000000000282 ffff8801cbb84000 0000000000000000 ffff8801cb8e13a8 000000000000df68 00000000000118c0 ffff8801cb8e13a8

Call Trace:
 [<ffffffff81032cfe>] ? __wake_up+0x4e/0x70
 [<ffffffff8125584e>] scsi_error_handler+0x7e/0x430
 [<ffffffff812557d0>] ? scsi_error_handler+0x0/0x430
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

scsi_eh_5 S ffff8801cb30fea0 0 497 2 0x00000000
ffff8801cb30fe50 0000000000000046 ffff8801cb30fde0 ffffffff81032cfe ffff8801cb30fde0 0000000000000282 ffff8801cbb88000 0000000000000000 ffff8801cb8b1368 000000000000df68 00000000000118c0 ffff8801cb8b1368

Call Trace:
 [<ffffffff81032cfe>] ? __wake_up+0x4e/0x70
 [<ffffffff8125584e>] scsi_error_handler+0x7e/0x430
 [<ffffffff812557d0>] ? scsi_error_handler+0x0/0x430
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

kstriped S ffff8801cb92d140 0 562 2 0x00000000
ffff8801ca4a5e60 0000000000000046 ffff8801ca4a5de0 ffffffff81036826 ffff8801cb854048 ffff880028291800 ffff8800282918c0 00000000fffedb2d ffff8801cb92d3e8 000000000000df68 00000000000118c0 ffff8801cb92d3e8

Call Trace:
 [<ffffffff81036826>] ? dequeue_task_fair+0x86/0x90
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

usbhid_resume D ffff8801ca45a140 0 591 2 0x00000000
ffff8801ca4fbe30 0000000000000046 0000000000000000 ffff8801ca74e400 ffff8801ca59a280 0000000000000000 ffff8801ca4fbe60 ffffffff8142dc98 ffff8801ca45a3e8 000000000000df68 00000000000118c0 ffff8801ca45a3e8

Call Trace:
 [<ffffffff8142dc98>] ? thread_return+0x3e/0x726
 [<ffffffff8105d5fd>] refrigerator+0xad/0x100
 [<ffffffff81051da5>] worker_thread+0xf5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

hd-audio0 S ffff8801ca4c5240 0 601 2 0x00000000
ffff8801ca509e60 0000000000000046 ffff8801ca509de0 ffffffff81036826 ffff8801cb854048 ffff880028291800 ffff8800282918c0 00000000fffedb2d ffff8801ca4c54e8 000000000000df68 00000000000118c0 ffff8801ca4c54e8

Call Trace:
 [<ffffffff81036826>] ? dequeue_task_fair+0x86/0x90
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

md3_raid1 S ffff8801ca59fe80 0 656 2 0x00000000
ffff8801ca59fdb0 0000000000000046 ffff8801ca59fd20 ffffffff812e2743 ffff8801ca4c7aa0 0000000000000246 ffff8801ca59fe50 00000000ffff0f39 ffff8801cb989b18 000000000000df68 00000000000118c0 ffff8801cb989b18

Call Trace:
 [<ffffffff812e2743>] ? flush_pending_writes+0x13/0x90
 [<ffffffff8142e8d5>] schedule_timeout+0x1c5/0x230
 [<ffffffff812e286a>] ? raid1d+0xa/0x10d0
 [<ffffffff812eaaea>] md_thread+0xea/0x130
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff812eaa00>] ? md_thread+0x0/0x130
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

kjournald S ffff8801ca582898 0 668 2 0x00000000
ffff8801ca57de60 0000000000000046 ffff8801ca588000 ffff8801ca582968 ffff880100000000 00000001ca582888 ffff880100000fdc 00000000ffff367a ffff8801cb981328 000000000000df68 00000000000118c0 ffff8801cb981328

Call Trace:
 [<ffffffff8113b24d>] kjournald+0x20d/0x230
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8113b040>] ? kjournald+0x0/0x230
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

<removed refrigerated tasks>
phy0 S ffff8801c93860c0 0 1010 2 0x00000000
ffff8801ca643e60 0000000000000046 ffff8801ca643de0 ffffffff81036826 0000000000000001 ffff8801c93860c0 ffff8800282118c0 00000000fffee0bd ffff8801c9386368 000000000000df68 00000000000118c0 ffff8801c9386368

Call Trace:
 [<ffffffff81036826>] ? dequeue_task_fair+0x86/0x90
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

kdmflush S ffff8801c88660c0 0 1251 2 0x00000000
ffff8801c8825e60 0000000000000046 0000000000000000 ffff8801ca6a4cc8 ffff8801c8825fd8 ffff8801c88660c0 ffff8801c8825de0 00000000fffee367 ffff8801c8866368 000000000000df68 00000000000118c0 ffff8801c8866368

Call Trace:
 [<ffffffff81051c3f>] ? run_workqueue+0xcf/0x140
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

kdmflush S ffff8801ca5d36b0 0 1256 2 0x00000000
ffff8801c8b49e60 0000000000000046 0000000000000000 ffff8801ca6eb0c8 ffff8801c8b49fd8 ffff8801ca5d36b0 ffff8801c8b49de0 00000000fffee367 ffff8801ca5d3958 000000000000df68 00000000000118c0 ffff8801ca5d3958

Call Trace:
 [<ffffffff81051c3f>] ? run_workqueue+0xcf/0x140
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

kdmflush S ffff8801c8ba0140 0 1262 2 0x00000000
ffff8801c92d3e60 0000000000000046 0000000000000000 ffff8801cb329cc8 ffff8801c92d3fd8 ffff8801c8ba0140 ffff8801c92d3de0 ffffffff8105a599 ffff8801c8ba03e8 000000000000df68 00000000000118c0 ffff8801c8ba03e8

Call Trace:
 [<ffffffff8105a599>] ? up_write+0x9/0x10
 [<ffffffff81051c3f>] ? run_workqueue+0xcf/0x140
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

kjournald S ffff8801cb329498 0 1310 2 0x00000000
ffff8801c9347e60 0000000000000046 ffff8801c9347dd0 ffffffff810566d1 ffff8801c893f738 ffff8801cb329488 ffff8801c9347e20 ffff8801cb329400 ffff8801c9386a18 000000000000df68 00000000000118c0 ffff8801c9386a18

Call Trace:
 [<ffffffff810566d1>] ? autoremove_wake_function+0x11/0x40
 [<ffffffff8113b24d>] kjournald+0x20d/0x230
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8113b040>] ? kjournald+0x0/0x230
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

kjournald S ffff8801ca5c7c98 0 1311 2 0x00000000
ffff8801c88bfe60 0000000000000046 ffff8801c88bfdd0 ffffffff810566d1 0000000000000000 ffff8801ca5c7c88 ffff8801c88bfe20 ffff8801ca5c7c00 ffff8801ca72f3a8 000000000000df68 00000000000118c0 ffff8801ca72f3a8

Call Trace:
 [<ffffffff810566d1>] ? autoremove_wake_function+0x11/0x40
 [<ffffffff8113b24d>] kjournald+0x20d/0x230
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8113b040>] ? kjournald+0x0/0x230
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

kjournald S ffff8801ca6f7498 0 1312 2 0x00000000
ffff8801c887fe60 0000000000000046 ffff8801c887fdd0 ffffffff810566d1 ffff8801c893f738 ffff8801ca6f7488 ffff8801c887fe20 ffff8801ca6f7400 ffff8801ca6d54e8 000000000000df68 00000000000118c0 ffff8801ca6d54e8

Call Trace:
 [<ffffffff810566d1>] ? autoremove_wake_function+0x11/0x40
 [<ffffffff8113b24d>] kjournald+0x20d/0x230
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8113b040>] ? kjournald+0x0/0x230
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

kjournald S ffff8801ca78c898 0 1313 2 0x00000000
ffff8801c88c3e60 0000000000000046 ffff8801c88c3dd0 ffffffff810566d1 0000000000000000 ffff8801ca78c888 ffff8801c88c3e20 00000000fffee41f ffff8801c893f2e8 000000000000df68 00000000000118c0 ffff8801c893f2e8

Call Trace:
 [<ffffffff810566d1>] ? autoremove_wake_function+0x11/0x40
 [<ffffffff8113b24d>] kjournald+0x20d/0x230
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8113b040>] ? kjournald+0x0/0x230
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

kjournald S ffff8801ca78d898 0 1314 2 0x00000000
ffff8801c8843e60 0000000000000046 ffff8801c8843dd0 ffffffff810566d1 0000000000000000 ffff8801ca78d888 ffff8801c8843e20 00000000fffee424 ffff8801cb989468 000000000000df68 00000000000118c0 ffff8801cb989468

Call Trace:
 [<ffffffff810566d1>] ? autoremove_wake_function+0x11/0x40
 [<ffffffff8113b24d>] kjournald+0x20d/0x230
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8113b040>] ? kjournald+0x0/0x230
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

<removed refrigerated tasks>
rpciod/0 S ffff8801ca60f7f0 0 2535 2 0x00000000
ffff8801ca719e60 0000000000000046 ffff8801ca719de0 ffffffff81036826 ffff8801ca719df0 ffff8801ca60f7f0 ffff8800282118c0 00000000fffeeca9 ffff8801ca60fa98 000000000000df68 00000000000118c0 ffff8801ca60fa98

Call Trace:
 [<ffffffff81036826>] ? dequeue_task_fair+0x86/0x90
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

rpciod/1 S ffff8801c9362080 0 2536 2 0x00000000
ffff8801ca577e60 0000000000000046 ffff8801ca577de0 ffffffff81036826 ffff8801c9bb5188 ffff880028291800 ffff8800282918c0 00000000fffeecba ffff8801c9362328 000000000000df68 00000000000118c0 ffff8801c9362328

Call Trace:
 [<ffffffff81036826>] ? dequeue_task_fair+0x86/0x90
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

nfsiod S ffff8801c9b998f0 0 2571 2 0x00000000
ffff8801c9163e60 0000000000000046 ffff8801c9163de0 ffffffff81036826 ffff8801c9163df0 ffff8801c9b998f0 ffff8800282118c0 00000000fffeece9 ffff8801c9b99b98 000000000000df68 00000000000118c0 ffff8801c9b99b98

Call Trace:
 [<ffffffff81036826>] ? dequeue_task_fair+0x86/0x90
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

<removed refrigerated tasks>
pm-suspend D ffff8801cb880000 0 3423 3387 0x00000000
ffff8801c5dddc08 0000000000000086 0000000000000000 ffff8801c5dddc30 ffff8801c5dddb78 ffffffff8103321c ffff8801c5dddba8 00000000ffff37ee ffff8801ca59a528 000000000000df68 00000000000118c0 ffff8801ca59a528

Call Trace:
 [<ffffffff8103321c>] ? __enqueue_entity+0x7c/0x80
 [<ffffffff8142e864>] schedule_timeout+0x154/0x230
 [<ffffffff8104a5c0>] ? process_timeout+0x0/0x10
 [<ffffffff8142e587>] wait_for_common+0xc7/0x170
 [<ffffffff81037600>] ? default_wake_function+0x0/0x10
 [<ffffffff8142e6ae>] wait_for_completion_timeout+0xe/0x10
 [<ffffffff8141cbf6>] _cpu_down+0xe6/0x110
 [<ffffffff8104115b>] disable_nonboot_cpus+0xab/0x130
 [<ffffffff8106e84d>] suspend_devices_and_enter+0xad/0x1a0
 [<ffffffff8106ea1b>] enter_state+0xdb/0xf0
 [<ffffffff8106e151>] state_store+0x91/0x100
 [<ffffffff8116af27>] kobj_attr_store+0x17/0x20
 [<ffffffff8111b590>] sysfs_write_file+0xe0/0x160
 [<ffffffff810c3698>] vfs_write+0xb8/0x1b0
 [<ffffffff81432ab5>] ? do_page_fault+0x185/0x350
 [<ffffffff810c3cfc>] sys_write+0x4c/0x80
 [<ffffffff8100beeb>] system_call_fastpath+0x16/0x1b

kstop/0 R running task 0 3836 2 0x00000000
ffff8801c980fe60 0000000000000046 ffff8801c980fde0 ffffffff81036826 0000000000000001 ffff8801c6e241c0 ffff8800282118c0 00000000ffff380f ffff8801c6e24468 000000000000df68 00000000000118c0 ffff8801c6e24468

Call Trace:
 [<ffffffff81036826>] ? dequeue_task_fair+0x86/0x90
 [<ffffffff81051d95>] worker_thread+0xe5/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

kstop/1 R running task 0 3837 2 0x00000000
ffff8801c5f08000 0000000000000001 ffffe8ffffc89888 ffffe8ffffc89888 ffff8801c5f09e80 ffffe8ffffc87450 ffffffffffffff10 ffffffff81080556 0000000000000010 0000000000000293 ffff8801c5f09e00 0000000000000018

Call Trace:
 [<ffffffff81080556>] ? stop_cpu+0x76/0xe0
 [<ffffffff810804e0>] ? stop_cpu+0x0/0xe0
 [<ffffffff81051c28>] ? run_workqueue+0xb8/0x140
 [<ffffffff81051d5b>] ? worker_thread+0xab/0x120
 [<ffffffff810566c0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81051cb0>] ? worker_thread+0x0/0x120
 [<ffffffff8105633e>] ? kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] ? child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

kcpu_down R running task 0 3838 2 0x00000008
ffff8801c9867e50 ffffffff8105279a ffffffff815df680 0000000000000004 ffffffff815df680 ffff8801c5dddd58 ffff8801c9867e80 ffffffff810804ac 0000000000000001 ffff8801c9867ef8 ffff8801c5dddd58 0000000000000010

Call Trace:
 [<ffffffff8105279a>] ? flush_workqueue+0x5a/0x90
 [<ffffffff810804ac>] ? __stop_machine+0x10c/0x140
 [<ffffffff8141c882>] ? _cpu_down_thread+0xa2/0x2f0
 [<ffffffff8141c7e0>] ? _cpu_down_thread+0x0/0x2f0
 [<ffffffff8105633e>] ? kthread+0x8e/0xa0
 [<ffffffff8100ce9a>] ? child_rip+0xa/0x20
 [<ffffffff810562b0>] ? kthread+0x0/0xa0
 [<ffffffff8100ce90>] ? child_rip+0x0/0x20

CPU 1 is now offline
SMP alternatives: switching to UP code
CPU1 is down
Extended CMOS year: 2000
x86 PAT enabled: cpu 0, old 0x7010600070106, new 0x7010600070106 Back to C!
<removed devices prints>
Restarting tasks ... done.

#5 lore raw

Re: suspend race -next regression [Was: Power: fix suspend vt regression]

Quoted 19 lines · Jiri Slaby
Quoted 2 lines · Jiri Slaby

However there is still a race or something. Sometimes the suspend goes through, sometimes it doesn't. I will investigate this further.

Hmm, this took a loong time to track down a bit. Code instrumentation by outb(XX, 0x80) usually caused the issue to disappear.

However I found out that it's caused by might_sleep() calls in flush_workqueue() and flush_cpu_workqueue(). I.e. it looks like there is a task which deadlocks/spins forever. If we won't reschedule to it, suspend proceeds.

I replaced the latter might_sleep() by show_state() and removed refrigerated tasks afterwards. The thing is that I don't know if the prank task is there. I need a scheduler to store "next" task pid or whatever to see what it picked as "next" and so what will run due to might_sched(). I can then show it on port 80 display and read it when the hangup occurs.

Quoted 7 lines

Depending on which might_sleep(), either flush_workqueue() never (well, at least in next 5 minutes) proceeds to for_each_cpu() or wait_for_completion() in flush_cpu_workqueue() never returns.

It's a regression against some -rc1 based -next tree. Bisection impossible, suspend needs to be run even 7 times before it occurs. Maybe a s/might_sleep/yield/ could make it happen earlier (going to try)?

If /sys/class/rtc/rtc0/wakealarm works on this box, you can use it to trigger resume in a loop.

Basically, you can do

# echo 0 > /sys/class/rtc/rtc0/wakealarm
# date +%s -d "+60 seconds" > /sys/class/rtc/rtc0/wakealarm

then go to suspend and it will resume the box in ~1 minute.

Thanks,
Rafael

author
#6 lore raw

Re: suspend race -mm regression [Was: Power: fix suspend vt regression]

Quoted 31 lines · Rafael J. Wysocki
Quoted 19 lines · Jiri Slaby
Quoted 2 lines · Jiri Slaby

However there is still a race or something. Sometimes the suspend goes through, sometimes it doesn't. I will investigate this further.

Hmm, this took a loong time to track down a bit. Code instrumentation by outb(XX, 0x80) usually caused the issue to disappear.

However I found out that it's caused by might_sleep() calls in flush_workqueue() and flush_cpu_workqueue(). I.e. it looks like there is a task which deadlocks/spins forever. If we won't reschedule to it, suspend proceeds.

I replaced the latter might_sleep() by show_state() and removed refrigerated tasks afterwards. The thing is that I don't know if the prank task is there. I need a scheduler to store "next" task pid or whatever to see what it picked as "next" and so what will run due to might_sched(). I can then show it on port 80 display and read it when the hangup occurs.

Quoted 7 lines

Depending on which might_sleep(), either flush_workqueue() never (well, at least in next 5 minutes) proceeds to for_each_cpu() or wait_for_completion() in flush_cpu_workqueue() never returns.

It's a regression against some -rc1 based -next tree. Bisection impossible, suspend needs to be run even 7 times before it occurs. Maybe a s/might_sleep/yield/ could make it happen earlier (going to try)?

If /sys/class/rtc/rtc0/wakealarm works on this box, you can use it to trigger resume in a loop.

Quoted 6 lines

Basically, you can do

# echo 0 > /sys/class/rtc/rtc0/wakealarm
# date +%s -d "+60 seconds" > /sys/class/rtc/rtc0/wakealarm

then go to suspend and it will resume the box in ~1 minute.

Thanks, in the end I found it manually. Goddammit! It's an -mm thing:
cpu_hotplug-dont-affect-current-tasks-affinity.patch

Well, I don't know why, but when the kthread overthere runs under suspend conditions and gets rescheduled (e.g. by the might_sleep() inside) it never returns. pick_next_task always returns the idle task from the idle queue. State of the thread is TASK_RUNNING.

Why is it not enqueued into some queue? I tried also sched_setscheduler(current, FIFO, 99) in the thread itself. Unless I did it wrong, it seems like a global scheduler problem?

Ingo, any ideas?

Thanks.

author
#7 lore raw

Re: suspend race -mm regression [Was: Power: fix suspend vt regression]

CCs reduced.

Quoted 34 lines · Jiri Slaby
Quoted 33 lines · Rafael J. Wysocki
Quoted 19 lines · Jiri Slaby
Quoted 2 lines · Jiri Slaby

However there is still a race or something. Sometimes the suspend goes through, sometimes it doesn't. I will investigate this further.

Hmm, this took a loong time to track down a bit. Code instrumentation by outb(XX, 0x80) usually caused the issue to disappear.

However I found out that it's caused by might_sleep() calls in flush_workqueue() and flush_cpu_workqueue(). I.e. it looks like there is a task which deadlocks/spins forever. If we won't reschedule to it, suspend proceeds.

I replaced the latter might_sleep() by show_state() and removed refrigerated tasks afterwards. The thing is that I don't know if the prank task is there. I need a scheduler to store "next" task pid or whatever to see what it picked as "next" and so what will run due to might_sched(). I can then show it on port 80 display and read it when the hangup occurs.

Quoted 7 lines

Depending on which might_sleep(), either flush_workqueue() never (well, at least in next 5 minutes) proceeds to for_each_cpu() or wait_for_completion() in flush_cpu_workqueue() never returns.

It's a regression against some -rc1 based -next tree. Bisection impossible, suspend needs to be run even 7 times before it occurs. Maybe a s/might_sleep/yield/ could make it happen earlier (going to try)?

If /sys/class/rtc/rtc0/wakealarm works on this box, you can use it to trigger resume in a loop.

Basically, you can do

Quoted 7 lines
Quoted 4 lines
# echo 0 > /sys/class/rtc/rtc0/wakealarm
# date +%s -d "+60 seconds" > /sys/class/rtc/rtc0/wakealarm

then go to suspend and it will resume the box in ~1 minute.

Thanks, in the end I found it manually. Goddammit! It's an -mm thing:
cpu_hotplug-dont-affect-current-tasks-affinity.patch

BTW. when I reverted it, during suspend I got a warning:
SMP alternatives: switching to UP code

------------[ cut here ]------------
WARNING: at kernel/smp.c:124
__generic_smp_call_function_interrupt+0xfd/0x110()
Hardware name: To Be Filled By O.E.M.
Modules linked in: nfs lockd auth_rpcgss sunrpc ath5k ath
Pid: 3423, comm: pm-suspend Not tainted 2.6.31-rc8-mm1_64 #762
Call Trace:
 [<ffffffff8103fc48>] warn_slowpath_common+0x78/0xb0
 [<ffffffff8103fc8f>] warn_slowpath_null+0xf/0x20
 [<ffffffff8106950d>] __generic_smp_call_function_interrupt+0xfd/0x110
 [<ffffffff8106956a>] hotplug_cfd+0x4a/0xa0
 [<ffffffff81434e47>] notifier_call_chain+0x47/0x90
 [<ffffffff8105b311>] raw_notifier_call_chain+0x11/0x20
 [<ffffffff8141ece0>] _cpu_down+0x150/0x2d0
 [<ffffffff8104169b>] disable_nonboot_cpus+0xab/0x130
 [<ffffffff8106ee3d>] suspend_devices_and_enter+0xad/0x1a0
 [<ffffffff8106f00b>] enter_state+0xdb/0xf0
 [<ffffffff8106e741>] state_store+0x91/0x100
 [<ffffffff8116c157>] kobj_attr_store+0x17/0x20
 [<ffffffff8111c6a0>] sysfs_write_file+0xe0/0x160
 [<ffffffff810c3ce8>] vfs_write+0xb8/0x1b0
 [<ffffffff81434c35>] ? do_page_fault+0x185/0x350
 [<ffffffff810c434c>] sys_write+0x4c/0x80
 [<ffffffff8100be2b>] system_call_fastpath+0x16/0x1b
---[ end trace 73264e95657dec65 ]---

CPU1 is down

Quoted 10 lines

Well, I don't know why, but when the kthread overthere runs under suspend conditions and gets rescheduled (e.g. by the might_sleep() inside) it never returns. pick_next_task always returns the idle task from the idle queue. State of the thread is TASK_RUNNING.

Why is it not enqueued into some queue? I tried also sched_setscheduler(current, FIFO, 99) in the thread itself. Unless I did it wrong, it seems like a global scheduler problem?

Ingo, any ideas?

author
#8 lore raw

Re: suspend race -mm regression [Was: Power: fix suspend vt regression]

Quoted 5 lines · Jiri Slaby
WARNING: at kernel/smp.c:124
__generic_smp_call_function_interrupt+0xfd/0x110()
Hardware name: To Be Filled By O.E.M.
Modules linked in: nfs lockd auth_rpcgss sunrpc ath5k ath

Pid: 3423, comm: pm-suspend Not tainted 2.6.31-rc8-mm1_64 #762

Quoted 8 lines
Call Trace:
 [<ffffffff8103fc48>] warn_slowpath_common+0x78/0xb0
 [<ffffffff8103fc8f>] warn_slowpath_null+0xf/0x20
 [<ffffffff8106950d>] __generic_smp_call_function_interrupt+0xfd/0x110
 [<ffffffff8106956a>] hotplug_cfd+0x4a/0xa0
 [<ffffffff81434e47>] notifier_call_chain+0x47/0x90
 [<ffffffff8105b311>] raw_notifier_call_chain+0x11/0x20
 [<ffffffff8141ece0>] _cpu_down+0x150/0x2d0

It's the CPU_DEAD notifier:

ffffffff8141ecd0:       48 83 ce 07             or     $0x7,%rsi
ffffffff8141ecd4:       48 c7 c7 08 ff 5d 81    mov

$0xffffffff815dff08,%rdi
ffffffff8141ecdb: e8 20 c6 c3 ff callq ffffffff8105b300
<raw_notifier_call_chain>
ffffffff8141ece0: 3d 02 80 00 00 cmp $0x8002,%eax

author
#9 lore raw

[-mm] warning during suspend [was: suspend race -mm regression]

Quoted 15 lines · Jiri Slaby
Quoted 5 lines · Jiri Slaby
WARNING: at kernel/smp.c:124
__generic_smp_call_function_interrupt+0xfd/0x110()
Hardware name: To Be Filled By O.E.M.
Modules linked in: nfs lockd auth_rpcgss sunrpc ath5k ath

Pid: 3423, comm: pm-suspend Not tainted 2.6.31-rc8-mm1_64 #762

Quoted 8 lines
Call Trace:
 [<ffffffff8103fc48>] warn_slowpath_common+0x78/0xb0
 [<ffffffff8103fc8f>] warn_slowpath_null+0xf/0x20
 [<ffffffff8106950d>] __generic_smp_call_function_interrupt+0xfd/0x110
 [<ffffffff8106956a>] hotplug_cfd+0x4a/0xa0
 [<ffffffff81434e47>] notifier_call_chain+0x47/0x90
 [<ffffffff8105b311>] raw_notifier_call_chain+0x11/0x20
 [<ffffffff8141ece0>] _cpu_down+0x150/0x2d0
Quoted 7 lines

It's the CPU_DEAD notifier:

ffffffff8141ecd0:       48 83 ce 07             or     $0x7,%rsi
ffffffff8141ecd4:       48 c7 c7 08 ff 5d 81    mov

$0xffffffff815dff08,%rdi
ffffffff8141ecdb: e8 20 c6 c3 ff callq ffffffff8105b300
<raw_notifier_call_chain>
ffffffff8141ece0: 3d 02 80 00 00 cmp $0x8002,%eax

And it's due to:
generic-ipi-fix-the-race-between-generic_smp_call_function_-and-hotplug_cfd.patch

Should the WARN_ONs now warn only when run_callbacks is true?

#10 lore raw

Re: [-mm] warning during suspend [was: suspend race -mm regression]

Quoted 23 lines · Jiri Slaby
Quoted 14 lines · Jiri Slaby
Quoted 5 lines · Jiri Slaby
WARNING: at kernel/smp.c:124
__generic_smp_call_function_interrupt+0xfd/0x110()
Hardware name: To Be Filled By O.E.M.
Modules linked in: nfs lockd auth_rpcgss sunrpc ath5k ath

Pid: 3423, comm: pm-suspend Not tainted 2.6.31-rc8-mm1_64 #762

Quoted 8 lines
Call Trace:
 [<ffffffff8103fc48>] warn_slowpath_common+0x78/0xb0
 [<ffffffff8103fc8f>] warn_slowpath_null+0xf/0x20
 [<ffffffff8106950d>] __generic_smp_call_function_interrupt+0xfd/0x110
 [<ffffffff8106956a>] hotplug_cfd+0x4a/0xa0
 [<ffffffff81434e47>] notifier_call_chain+0x47/0x90
 [<ffffffff8105b311>] raw_notifier_call_chain+0x11/0x20
 [<ffffffff8141ece0>] _cpu_down+0x150/0x2d0
Quoted 7 lines

It's the CPU_DEAD notifier:

ffffffff8141ecd0:       48 83 ce 07             or     $0x7,%rsi
ffffffff8141ecd4:       48 c7 c7 08 ff 5d 81    mov

$0xffffffff815dff08,%rdi
ffffffff8141ecdb: e8 20 c6 c3 ff callq ffffffff8105b300
<raw_notifier_call_chain>
ffffffff8141ece0: 3d 02 80 00 00 cmp $0x8002,%eax

Quoted 3 lines

And it's due to:
generic-ipi-fix-the-race-between-generic_smp_call_function_-and-hotplug_cfd.patch

I think it has collision between my patch and below patch:

Commit-ID:  269c861baa2fe7c114c3bc7831292758d29eb336
Gitweb:     http://git.kernel.org/tip/269c861baa2fe7c114c3bc7831292758d29eb336
Author:     Suresh Siddha <[email protected]>
AuthorDate: Wed, 19 Aug 2009 18:05:35 -0700
Committer:  H. Peter Anvin <[email protected]>
CommitDate: Fri, 21 Aug 2009 16:25:43 -0700

generic-ipi: Allow cpus not yet online to call smp_call_function with irqs disabled

My patch is merged at -mm tree, but this patch is base on -tip tree later, so it has this problem

Suresh, what your opinion?

Thanks,
Xiao

Quoted 7 lines

Should the WARN_ONs now warn only when run_callbacks is true? --
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#11 lore raw

Re: [-mm] warning during suspend [was: suspend race -mm regression]

Quoted 37 lines · On Sat, 05 Sep 2009 22:41:37 +0800 Xiao Guangrong <[email protected]> wrote:

Jiri Slaby ______:

Quoted 23 lines · Jiri Slaby
Quoted 14 lines · Jiri Slaby
Quoted 5 lines
WARNING: at kernel/smp.c:124
__generic_smp_call_function_interrupt+0xfd/0x110()
Hardware name: To Be Filled By O.E.M.
Modules linked in: nfs lockd auth_rpcgss sunrpc ath5k ath

Pid: 3423, comm: pm-suspend Not tainted 2.6.31-rc8-mm1_64 #762

Quoted 8 lines
Call Trace:
 [<ffffffff8103fc48>] warn_slowpath_common+0x78/0xb0
 [<ffffffff8103fc8f>] warn_slowpath_null+0xf/0x20
 [<ffffffff8106950d>] __generic_smp_call_function_interrupt+0xfd/0x110
 [<ffffffff8106956a>] hotplug_cfd+0x4a/0xa0
 [<ffffffff81434e47>] notifier_call_chain+0x47/0x90
 [<ffffffff8105b311>] raw_notifier_call_chain+0x11/0x20
 [<ffffffff8141ece0>] _cpu_down+0x150/0x2d0
Quoted 7 lines

It's the CPU_DEAD notifier:

ffffffff8141ecd0:       48 83 ce 07             or     $0x7,%rsi
ffffffff8141ecd4:       48 c7 c7 08 ff 5d 81    mov

$0xffffffff815dff08,%rdi
ffffffff8141ecdb: e8 20 c6 c3 ff callq ffffffff8105b300
<raw_notifier_call_chain>
ffffffff8141ece0: 3d 02 80 00 00 cmp $0x8002,%eax

Quoted 3 lines

And it's due to:
generic-ipi-fix-the-race-between-generic_smp_call_function_-and-hotplug_cfd.patch

I think it has collision between my patch and below patch:

Commit-ID:  269c861baa2fe7c114c3bc7831292758d29eb336
Gitweb:     http://git.kernel.org/tip/269c861baa2fe7c114c3bc7831292758d29eb336
Author:     Suresh Siddha <[email protected]>
AuthorDate: Wed, 19 Aug 2009 18:05:35 -0700
Committer:  H. Peter Anvin <[email protected]>
CommitDate: Fri, 21 Aug 2009 16:25:43 -0700
Quoted 7 lines

generic-ipi: Allow cpus not yet online to call smp_call_function with irqs disabled

My patch is merged at -mm tree, but this patch is base on -tip tree later, so it has this problem

Suresh, what your opinion?

Suresh appears to be hiding.

Could you please propose a fix for this issue?

#12 lore raw

Re: [-mm] warning during suspend [was: suspend race -mm regression]

Quoted 27 lines · Andrew Morton
Quoted 24 lines · On Sat, 05 Sep 2009 22:41:37 +0800 Xiao Guangrong <[email protected]> wrote:

Jiri Slaby ______:

Quoted 23 lines · Jiri Slaby
Quoted 21 lines · Jiri Slaby
Quoted 5 lines
WARNING: at kernel/smp.c:124
__generic_smp_call_function_interrupt+0xfd/0x110()
Hardware name: To Be Filled By O.E.M.
Modules linked in: nfs lockd auth_rpcgss sunrpc ath5k ath

Pid: 3423, comm: pm-suspend Not tainted 2.6.31-rc8-mm1_64 #762

Quoted 8 lines
Call Trace:
 [<ffffffff8103fc48>] warn_slowpath_common+0x78/0xb0
 [<ffffffff8103fc8f>] warn_slowpath_null+0xf/0x20
 [<ffffffff8106950d>] __generic_smp_call_function_interrupt+0xfd/0x110
 [<ffffffff8106956a>] hotplug_cfd+0x4a/0xa0
 [<ffffffff81434e47>] notifier_call_chain+0x47/0x90
 [<ffffffff8105b311>] raw_notifier_call_chain+0x11/0x20
 [<ffffffff8141ece0>] _cpu_down+0x150/0x2d0

It's the CPU_DEAD notifier:

ffffffff8141ecd0:       48 83 ce 07             or     $0x7,%rsi
ffffffff8141ecd4:       48 c7 c7 08 ff 5d 81    mov

$0xffffffff815dff08,%rdi
ffffffff8141ecdb: e8 20 c6 c3 ff callq ffffffff8105b300
<raw_notifier_call_chain>
ffffffff8141ece0: 3d 02 80 00 00 cmp $0x8002,%eax

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

And it's due to:
generic-ipi-fix-the-race-between-generic_smp_call_function_-and-hotplug_cfd.patch

I think it has collision between my patch and below patch:

Xiao, I am not sure if the race that you are trying to fix here indeed exists. Doesn't the stop machine that we do as part of cpu down address and avoid the race that you mention? Have you seen any real crashes and hangs or is it theory?

And if even the race exists (which I don't think) calling the interrupt handler from the cpu down path looks like a hack.

Can you please elaborate why we need this patch? Then we can think of a cleaner solution if needed.

Quoted 10 lines
Quoted 10 lines
Commit-ID:  269c861baa2fe7c114c3bc7831292758d29eb336
Gitweb:     http://git.kernel.org/tip/269c861baa2fe7c114c3bc7831292758d29eb336
Author:     Suresh Siddha <[email protected]>
AuthorDate: Wed, 19 Aug 2009 18:05:35 -0700
Committer:  H. Peter Anvin <[email protected]>
CommitDate: Fri, 21 Aug 2009 16:25:43 -0700

generic-ipi: Allow cpus not yet online to call smp_call_function with irqs disabled

Quoted 7 lines
Quoted 5 lines

My patch is merged at -mm tree, but this patch is base on -tip tree later, so it has this problem

Suresh, what your opinion?

Suresh appears to be hiding.

Not any more. I am back from vacation :(

thanks,
suresh

#13 lore raw

Re: [-mm] warning during suspend [was: suspend race -mm regression]

Quoted 30 lines · Suresh Siddha
Quoted 29 lines · Andrew Morton
Quoted 26 lines · On Sat, 05 Sep 2009 22:41:37 +0800 Xiao Guangrong <[email protected]> wrote:

Jiri Slaby ______:

Quoted 25 lines · Jiri Slaby
Quoted 14 lines · Jiri Slaby
Quoted 5 lines
WARNING: at kernel/smp.c:124
__generic_smp_call_function_interrupt+0xfd/0x110()
Hardware name: To Be Filled By O.E.M.
Modules linked in: nfs lockd auth_rpcgss sunrpc ath5k ath

Pid: 3423, comm: pm-suspend Not tainted 2.6.31-rc8-mm1_64 #762

Quoted 8 lines
Call Trace:
 [<ffffffff8103fc48>] warn_slowpath_common+0x78/0xb0
 [<ffffffff8103fc8f>] warn_slowpath_null+0xf/0x20
 [<ffffffff8106950d>] __generic_smp_call_function_interrupt+0xfd/0x110
 [<ffffffff8106956a>] hotplug_cfd+0x4a/0xa0
 [<ffffffff81434e47>] notifier_call_chain+0x47/0x90
 [<ffffffff8105b311>] raw_notifier_call_chain+0x11/0x20
 [<ffffffff8141ece0>] _cpu_down+0x150/0x2d0
Quoted 7 lines

It's the CPU_DEAD notifier:

ffffffff8141ecd0:       48 83 ce 07             or     $0x7,%rsi
ffffffff8141ecd4:       48 c7 c7 08 ff 5d 81    mov

$0xffffffff815dff08,%rdi
ffffffff8141ecdb: e8 20 c6 c3 ff callq ffffffff8105b300
<raw_notifier_call_chain>
ffffffff8141ece0: 3d 02 80 00 00 cmp $0x8002,%eax

And it's due to:
generic-ipi-fix-the-race-between-generic_smp_call_function_-and-hotplug_cfd.patch

Quoted 7 lines
Quoted 1 line
Quoted 1 line

I think it has collision between my patch and below patch:

Xiao, I am not sure if the race that you are trying to fix here indeed exists. Doesn't the stop machine that we do as part of cpu down address and avoid the race that you mention? Have you seen any real crashes and hangs or is it theory?

Suresh, please see my explanation in another mail, see below URL:
http://marc.info/?l=linux-kernel&m=125265516529139&w=2

Thanks,
Xiao

author
#14 lore raw

PATCH 1/1 sched: fix cpu_down deadlock

In reply to Jiri Slaby
Quoted 11 lines · Jiri Slaby

Thanks, in the end I found it manually. Goddammit! It's an -mm thing:
cpu_hotplug-dont-affect-current-tasks-affinity.patch

Well, I don't know why, but when the kthread overthere runs under suspend conditions and gets rescheduled (e.g. by the might_sleep() inside) it never returns. pick_next_task always returns the idle task from the idle queue. State of the thread is TASK_RUNNING.

Why is it not enqueued into some queue? I tried also sched_setscheduler(current, FIFO, 99) in the thread itself. Unless I did it wrong, it seems like a global scheduler problem?

Actually not, it definitely seems like a cpu_down problem.

Quoted 1 line

Ingo, any ideas?

Apparently not, but nevermind :). What about the patch below?

--

After a cpu is taken down in __stop_machine, the kcpu_thread still may be rescheduled to that cpu, but in fact the cpu is not running at that moment.

This causes kcpu_thread to never run again, because its enqueued on another runqueue, hence pick_next_task never selects it on the set of newly running cpus.

We do set_cpus_allowed_ptr in _cpu_down_thread, but cpu_active_mask is updated to not contain the cpu which goes down even after the thread finishes (and _cpu_down returns).

For me this triggers mostly while suspending a SMP machine with FAIR_GROUP_SCHED enabled and cpu_hotplug-dont-affect-current-tasks-affinity patch applied. The patch adds kthread to the cpu_down pipeline.

Fix this issue by eliminating the to-be-killed-cpu from active_cpu locally.

Signed-off-by Jiri Slaby <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 kernel/cpu.c |   12 ++++++++++--
 1 files changed, 10 insertions(+), 2 deletions(-)
kernel/cpu.c +10 -2
index be9c5ad..17a3635 100644
@@ -196,6 +196,14 @@ static int __ref _cpu_down_thread(void *_param)
 	unsigned long mod = param->mod;
 	unsigned int cpu = param->cpu;
 	void *hcpu = (void *)(long)cpu;
+	cpumask_var_t active_mask;
+
+	if (!alloc_cpumask_var(&active_mask, GFP_KERNEL))
+		return -ENOMEM;
+
+	/* make sure we are not running on the cpu which goes down,
+	   cpu_active_mask is altered even after we return! */
+	cpumask_andnot(active_mask, cpu_active_mask, cpumask_of(cpu));
 
 	cpu_hotplug_begin();
 	err = __raw_notifier_call_chain(&cpu_chain, CPU_DOWN_PREPARE | mod,

@@ -211,7 +219,7 @@ static int __ref _cpu_down_thread(void *_param)
 	}
 
 	/* Ensure that we are not runnable on dying cpu */
-	set_cpus_allowed_ptr(current, cpu_active_mask);
+	set_cpus_allowed_ptr(current, active_mask);
 
 	err = __stop_machine(take_cpu_down, param, cpumask_of(cpu));
 	if (err) {

@@ -237,9 +245,9 @@ static int __ref _cpu_down_thread(void *_param)
 		BUG();
 
 	check_for_tasks(cpu);
-
 out_release:
 	cpu_hotplug_done();
+	free_cpumask_var(active_mask);
 	if (!err) {
 		if (raw_notifier_call_chain(&cpu_chain, CPU_POST_DEAD | mod,
 					    hcpu) == NOTIFY_BAD)
Signature · 1 line
1.6.3.3
maintainer
#15 lore raw

Re: [PATCH 1/1] sched: fix cpu_down deadlock

Quoted 3 lines · Jiri Slaby
Quoted 2 lines · Jiri Slaby

Thanks, in the end I found it manually. Goddammit! It's an -mm thing:
cpu_hotplug-dont-affect-current-tasks-affinity.patch

Is there a git tree with -mm in some place? I can't seem to find that patch in my inbox.

All I can find is some comments from Oleg that the patch looks funny.

author
#16 lore raw

Re: [PATCH 1/1] sched: fix cpu_down deadlock

Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Jiri Slaby
Quoted 2 lines · Jiri Slaby

Thanks, in the end I found it manually. Goddammit! It's an -mm thing:
cpu_hotplug-dont-affect-current-tasks-affinity.patch

Is there a git tree with -mm in some place? I can't seem to find that patch in my inbox.

All I can find is some comments from Oleg that the patch looks funny.

Yes, here:
git://git.zen-sources.org/zen/mmotm.git

Actually I found Oleg came up with better solution to add move_task_off_dead_cpu to take_cpu_down.

A discussion regarding this is at:
http://lkml.indiana.edu/hypermail/linux/kernel/0907.3/02278.html

So what's the status of the patches, please?

maintainer
#17 lore raw

Re: [PATCH 1/1] sched: fix cpu_down deadlock

Quoted 6 lines · Jiri Slaby
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines · Jiri Slaby
Quoted 2 lines · Jiri Slaby

Thanks, in the end I found it manually. Goddammit! It's an -mm thing:
cpu_hotplug-dont-affect-current-tasks-affinity.patch

Quoted 7 lines
Quoted 4 lines

Is there a git tree with -mm in some place? I can't seem to find that patch in my inbox.

All I can find is some comments from Oleg that the patch looks funny.

Yes, here:
git://git.zen-sources.org/zen/mmotm.git

Ah thanks, no wonder I didn't find it.

Quoted 7 lines

Actually I found Oleg came up with better solution to add move_task_off_dead_cpu to take_cpu_down.

A discussion regarding this is at:
http://lkml.indiana.edu/hypermail/linux/kernel/0907.3/02278.html

So what's the status of the patches, please?

Oleg's patch looks good to me.

#18 lore raw

Re: [PATCH 1/1] sched: fix cpu_down deadlock

In reply to Jiri Slaby
Quoted 15 lines · Jiri Slaby
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Jiri Slaby
Quoted 2 lines · Jiri Slaby

Thanks, in the end I found it manually. Goddammit! It's an -mm thing:
cpu_hotplug-dont-affect-current-tasks-affinity.patch

Is there a git tree with -mm in some place? I can't seem to find that patch in my inbox.

All I can find is some comments from Oleg that the patch looks funny.

Yes, here:
git://git.zen-sources.org/zen/mmotm.git

Quoted 7 lines

Actually I found Oleg came up with better solution to add move_task_off_dead_cpu to take_cpu_down.

A discussion regarding this is at:
http://lkml.indiana.edu/hypermail/linux/kernel/0907.3/02278.html

So what's the status of the patches, please?

This patch depends on another one, please see

"[PATCH] cpusets: rework guarantee_online_cpus() to fix deadlock with cpu_down()"
http://marc.info/?t=124910242400002

(as the changelog says, the patch is not complete: we need
->cpumask_lock every time we update cs->allowed, but this should be trivial)

In short: cpuset_lock() is buggy. But more importantly it is afaics unneeded, and imho should die. I seem to answer all Lai's questions, but the patch was ignored by maintainers.

I noticed another race in update_cpumask() which I was going to fix, but since maintainers ignore me I lost the motivtion ;) Besides, currently I dont have the time anyway.

So I think the original patch which creates the kthread is the best option.

Oleg.

#19 lore raw

Re: [PATCH 1/1] sched: fix cpu_down deadlock

In reply to Jiri Slaby
Quoted 77 lines · Jiri Slaby
Quoted 11 lines · Jiri Slaby

Thanks, in the end I found it manually. Goddammit! It's an -mm thing:
cpu_hotplug-dont-affect-current-tasks-affinity.patch

Well, I don't know why, but when the kthread overthere runs under suspend conditions and gets rescheduled (e.g. by the might_sleep() inside) it never returns. pick_next_task always returns the idle task from the idle queue. State of the thread is TASK_RUNNING.

Why is it not enqueued into some queue? I tried also sched_setscheduler(current, FIFO, 99) in the thread itself. Unless I did it wrong, it seems like a global scheduler problem?

Actually not, it definitely seems like a cpu_down problem.

Quoted 1 line

Ingo, any ideas?

Apparently not, but nevermind :). What about the patch below?

--

After a cpu is taken down in __stop_machine, the kcpu_thread still may be rescheduled to that cpu, but in fact the cpu is not running at that moment.

This causes kcpu_thread to never run again, because its enqueued on another runqueue, hence pick_next_task never selects it on the set of newly running cpus.

We do set_cpus_allowed_ptr in _cpu_down_thread, but cpu_active_mask is updated to not contain the cpu which goes down even after the thread finishes (and _cpu_down returns).

For me this triggers mostly while suspending a SMP machine with FAIR_GROUP_SCHED enabled and cpu_hotplug-dont-affect-current-tasks-affinity patch applied. The patch adds kthread to the cpu_down pipeline.

Fix this issue by eliminating the to-be-killed-cpu from active_cpu locally.

Signed-off-by Jiri Slaby <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 kernel/cpu.c |   12 ++++++++++--
 1 files changed, 10 insertions(+), 2 deletions(-)
kernel/cpu.c +9 -1
index be9c5ad..17a3635 100644
@@ -196,6 +196,14 @@ static int __ref _cpu_down_thread(void *_param)
 	unsigned long mod = param->mod;
 	unsigned int cpu = param->cpu;
 	void *hcpu = (void *)(long)cpu;
+	cpumask_var_t active_mask;
+
+	if (!alloc_cpumask_var(&active_mask, GFP_KERNEL))
+		return -ENOMEM;
+
+	/* make sure we are not running on the cpu which goes down,
+	   cpu_active_mask is altered even after we return! */
+	cpumask_andnot(active_mask, cpu_active_mask, cpumask_of(cpu));
 
 	cpu_hotplug_begin();
 	err = __raw_notifier_call_chain(&cpu_chain, CPU_DOWN_PREPARE | mod,

@@ -211,7 +219,7 @@ static int __ref _cpu_down_thread(void *_param)
 	}
 
 	/* Ensure that we are not runnable on dying cpu */
-	set_cpus_allowed_ptr(current, cpu_active_mask);
+	set_cpus_allowed_ptr(current, active_mask);
 
 	err = __stop_machine(take_cpu_down, param, cpumask_of(cpu));
 	if (err) {

kernel/cpu.c static int __ref _cpu_down_thread(void *_param) · line 253

Quoted 10 lines
kernel/cpu.c +1 -1
@@ -237,9 +245,9 @@ static int __ref _cpu_down_thread(void *_param)
 		BUG();
 
 	check_for_tasks(cpu);
-
 out_release:
 	cpu_hotplug_done();
+	free_cpumask_var(active_mask);
 	if (!err) {
 		if (raw_notifier_call_chain(&cpu_chain, CPU_POST_DEAD | mod,
 					    hcpu) == NOTIFY_BAD)

Hi, Jiri Slaby

Does this bug occur when a cpu is being offlined or when the system is being suspended?
Or Both?

Lai

author
#20 lore raw

Re: [PATCH 1/1] sched: fix cpu_down deadlock

Quoted 3 lines · Lai Jiangshan

Does this bug occur when a cpu is being offlined or when the system is being suspended?
Or Both?

Hi, I tried echo 0/1 > /sys/devices/system/cpu/cpu1/online in a loop, but it didn't trigger the bug. It happened only on suspend/resume cycle (in the end I found even swsusp in qemu suffers from this).

#21 lore raw

Re: [PATCH 1/1] sched: fix cpu_down deadlock

Quoted 5 lines · Jiri Slaby
Quoted 3 lines · Lai Jiangshan

Does this bug occur when a cpu is being offlined or when the system is being suspended?
Or Both?

Quoted 10 lines

Hi, I tried echo 0/1 > /sys/devices/system/cpu/cpu1/online in a loop, but it didn't trigger the bug. It happened only on suspend/resume cycle (in the end I found even swsusp in qemu suffers from this).
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

OK, I knew where this bug is.

I thought the corresponding bit in cpu_active_mask is cleared before _cpu_down(), but I missed the system-suspend path:disable_nonboot_cpus().

There is a bug in disable_nonboot_cpus() even if my patch is removed.

cpu_active_map is wrong during suspending.(scheduler system who uses cpu_active_map are still working while suspending)

You need:

int disable_nonboot_cpus(void)

{
	....
	....

	/*
	 * You need adding 'set_cpu_active(cpu, false);' here
	 * to fix this bug and make my patch works well.
	 */
	error = _cpu_down(cpu, 1);

	....
	....
}

Lai.