PM: Do not destroy/create devices while suspended
Subsystems (3)
cover PM: Do not destroy/create devices while suspended
Hi,
Some device drivers register CPU hotplug notifiers and use them to destroy device objects when removing the corresponding CPUs and to create these objects when adding the CPUs back.
Unfortunately, this is not the right thing to do during suspend/hibernation, since in that cases the CPU hotplug notifiers are called after suspending devices and before resuming them, so the operations in question are carried out on the objects representing suspended devices which shouldn't be unregistered behing the PM core's back. Although right now it usually doesn't lead to any practical complications, it will predictably deadlock if gregkh-driver-pm-acquire-device-locks-prior-to-suspending.patch is applied.
The solution is to prevent drivers from removing/adding devices from within CPU hotplug notifiers during suspend/hibernation using the FROZEN bit in the notifier's action argument. The following three patches modify the MSR, x86-64 MCE and cpuid drivers along these lines.
Thanks,
Rafael
PATCH 1/3 PM: Do not destroy/create devices while suspended in msr.c
From: Rafael J. Wysocki <[email protected]>
The MSR driver should not attempt to destroy/create a suspended device.
--- arch/x86/kernel/msr.c | 3 --- 1 file changed, 3 deletions(-)
arch/x86/kernel/msr.c +0 -3
@@ -155,13 +155,10 @@ static int __cpuinit msr_class_cpu_callbswitch (action) {case CPU_UP_PREPARE : -case CPU_UP_PREPARE_FROZEN : err =msr_device_create (cpu);break ;case CPU_UP_CANCELED : -case CPU_UP_CANCELED_FROZEN :case CPU_DEAD : -case CPU_DEAD_FROZEN :msr_device_destroy (cpu);break ; }
Re: [PATCH 1/3] PM: Do not destroy/create devices while suspended in msr.c
Quoted 6 lines · Rafael J. Wysocki
From: Rafael J. Wysocki <[email protected]>
The MSR driver should not attempt to destroy/create a suspended device.
Signed-off-by Rafael J. Wysocki <[email protected]>
ACK.
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Re: [PATCH 1/3] PM: Do not destroy/create devices while suspended in msr.c
In reply to Rafael J. WysockiQuoted 6 lines · Rafael J. Wysocki
From: Rafael J. Wysocki <[email protected]>
The MSR driver should not attempt to destroy/create a suspended device.
Signed-off-by Rafael J. Wysocki <[email protected]>
ACK.
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
PATCH 1/3 PM: Do not destroy/create devices while suspended in msr.c
In reply to Rafael J. WysockiFrom: Rafael J. Wysocki <[email protected]>
The MSR driver should not attempt to destroy/create a suspended device.
--- arch/x86/kernel/msr.c | 3 --- 1 file changed, 3 deletions(-)
arch/x86/kernel/msr.c +0 -3
@@ -155,13 +155,10 @@ static int __cpuinit msr_class_cpu_callbswitch (action) {case CPU_UP_PREPARE : -case CPU_UP_PREPARE_FROZEN : err =msr_device_create (cpu);break ;case CPU_UP_CANCELED : -case CPU_UP_CANCELED_FROZEN :case CPU_DEAD : -case CPU_DEAD_FROZEN :msr_device_destroy (cpu);break ; }
PATCH 2/3 PM: Do not destroy/create devices while suspended in mce_64.c
In reply to Rafael J. WysockiFrom: Rafael J. Wysocki <[email protected]>
The x86-64 MCE driver should not attempt to destroy/create a suspended device.
--- arch/x86/kernel/cpu/mcheck/mce_64.c | 2 -- 1 file changed, 2 deletions(-)
arch/x86/kernel/cpu/mcheck/mce_64.c +0 -2
@@ -862,11 +862,9 @@ mce_cpu_callback(struct notifier_block *switch (action) {case CPU_ONLINE : -case CPU_ONLINE_FROZEN :mce_create_device (cpu);break ;case CPU_DEAD : -case CPU_DEAD_FROZEN :mce_remove_device (cpu);break ; }
Re: [PATCH 2/3] PM: Do not destroy/create devices while suspended in mce_64.c
Quoted 6 lines · Rafael J. Wysocki
From: Rafael J. Wysocki <[email protected]>
The x86-64 MCE driver should not attempt to destroy/create a suspended device.
Signed-off-by Rafael J. Wysocki <[email protected]>
ACK.
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Re: [PATCH 2/3] PM: Do not destroy/create devices while suspended in mce_64.c
In reply to Rafael J. WysockiQuoted 6 lines · Rafael J. Wysocki
From: Rafael J. Wysocki <[email protected]>
The x86-64 MCE driver should not attempt to destroy/create a suspended device.
Signed-off-by Rafael J. Wysocki <[email protected]>
ACK.
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
PATCH 2/3 PM: Do not destroy/create devices while suspended in mce_64.c
In reply to Rafael J. WysockiFrom: Rafael J. Wysocki <[email protected]>
The x86-64 MCE driver should not attempt to destroy/create a suspended device.
--- arch/x86/kernel/cpu/mcheck/mce_64.c | 2 -- 1 file changed, 2 deletions(-)
arch/x86/kernel/cpu/mcheck/mce_64.c +0 -2
@@ -862,11 +862,9 @@ mce_cpu_callback(struct notifier_block *switch (action) {case CPU_ONLINE : -case CPU_ONLINE_FROZEN :mce_create_device (cpu);break ;case CPU_DEAD : -case CPU_DEAD_FROZEN :mce_remove_device (cpu);break ; }
PATCH 3/3 PM: Do not destroy/create devices while suspended in cpuid.c
In reply to Rafael J. WysockiFrom: Rafael J. Wysocki <[email protected]>
The cpuid driver should not attempt to destroy/create a suspended device.
--- arch/x86/kernel/cpuid.c | 3 --- 1 file changed, 3 deletions(-)
arch/x86/kernel/cpuid.c +0 -3
@@ -157,13 +157,10 @@ static int __cpuinit cpuid_class_cpu_calswitch (action) {case CPU_UP_PREPARE : -case CPU_UP_PREPARE_FROZEN : err =cpuid_device_create (cpu);break ;case CPU_UP_CANCELED : -case CPU_UP_CANCELED_FROZEN :case CPU_DEAD : -case CPU_DEAD_FROZEN :cpuid_device_destroy (cpu);break ; }
Re: [PATCH 3/3] PM: Do not destroy/create devices while suspended in cpuid.c
Quoted 6 lines · Rafael J. Wysocki
From: Rafael J. Wysocki <[email protected]>
The cpuid driver should not attempt to destroy/create a suspended device.
Signed-off-by Rafael J. Wysocki <[email protected]>
ACK.
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Re: [PATCH 3/3] PM: Do not destroy/create devices while suspended in cpuid.c
In reply to Rafael J. WysockiQuoted 6 lines · Rafael J. Wysocki
From: Rafael J. Wysocki <[email protected]>
The cpuid driver should not attempt to destroy/create a suspended device.
Signed-off-by Rafael J. Wysocki <[email protected]>
ACK.
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
PATCH 3/3 PM: Do not destroy/create devices while suspended in cpuid.c
In reply to Rafael J. WysockiFrom: Rafael J. Wysocki <[email protected]>
The cpuid driver should not attempt to destroy/create a suspended device.
--- arch/x86/kernel/cpuid.c | 3 --- 1 file changed, 3 deletions(-)
arch/x86/kernel/cpuid.c +0 -3
@@ -157,13 +157,10 @@ static int __cpuinit cpuid_class_cpu_calswitch (action) {case CPU_UP_PREPARE : -case CPU_UP_PREPARE_FROZEN : err =cpuid_device_create (cpu);break ;case CPU_UP_CANCELED : -case CPU_UP_CANCELED_FROZEN :case CPU_DEAD : -case CPU_DEAD_FROZEN :cpuid_device_destroy (cpu);break ; }
Quoted 14 lines · Rafael J. Wysocki
Hi,
Some device drivers register CPU hotplug notifiers and use them to destroy device objects when removing the corresponding CPUs and to create these objects when adding the CPUs back.
Unfortunately, this is not the right thing to do during suspend/hibernation, since in that cases the CPU hotplug notifiers are called after suspending devices and before resuming them, so the operations in question are carried out on the objects representing suspended devices which shouldn't be unregistered behing the PM core's back. Although right now it usually doesn't lead to any practical complications, it will predictably deadlock if gregkh-driver-pm-acquire-device-locks-prior-to-suspending.patch is applied.
Quoted 4 lines
The solution is to prevent drivers from removing/adding devices from within CPU hotplug notifiers during suspend/hibernation using the FROZEN bit in the notifier's action argument. The following three patches modify the MSR, x86-64 MCE and cpuid drivers along these lines.
Do we need to worry about the possibility that when the system wakes up from hibernation, the set of usable CPUs might be smaller than it was beforehand? Is any special handling needed for this, or is it already accounted for?
Alan Stern
Quoted 21 lines · Alan Stern
Quoted 14 lines · Rafael J. Wysocki
Hi,
Some device drivers register CPU hotplug notifiers and use them to destroy device objects when removing the corresponding CPUs and to create these objects when adding the CPUs back.
Unfortunately, this is not the right thing to do during suspend/hibernation, since in that cases the CPU hotplug notifiers are called after suspending devices and before resuming them, so the operations in question are carried out on the objects representing suspended devices which shouldn't be unregistered behing the PM core's back. Although right now it usually doesn't lead to any practical complications, it will predictably deadlock if gregkh-driver-pm-acquire-device-locks-prior-to-suspending.patch is applied.
Quoted 4 lines
The solution is to prevent drivers from removing/adding devices from within CPU hotplug notifiers during suspend/hibernation using the FROZEN bit in the notifier's action argument. The following three patches modify the MSR, x86-64 MCE and cpuid drivers along these lines.
Quoted 4 lines
Do we need to worry about the possibility that when the system wakes up from hibernation, the set of usable CPUs might be smaller than it was beforehand? Is any special handling needed for this, or is it already accounted for?
That should not happen... but it does in some error cases.... so handling it would be a bonus.
Waking up with one cpu out of 8 is bad, but still way better than not waking up at all ;-).
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 21 lines · Alan Stern
Quoted 14 lines · Rafael J. Wysocki
Hi,
Some device drivers register CPU hotplug notifiers and use them to destroy device objects when removing the corresponding CPUs and to create these objects when adding the CPUs back.
Unfortunately, this is not the right thing to do during suspend/hibernation, since in that cases the CPU hotplug notifiers are called after suspending devices and before resuming them, so the operations in question are carried out on the objects representing suspended devices which shouldn't be unregistered behing the PM core's back. Although right now it usually doesn't lead to any practical complications, it will predictably deadlock if gregkh-driver-pm-acquire-device-locks-prior-to-suspending.patch is applied.
Quoted 4 lines
The solution is to prevent drivers from removing/adding devices from within CPU hotplug notifiers during suspend/hibernation using the FROZEN bit in the notifier's action argument. The following three patches modify the MSR, x86-64 MCE and cpuid drivers along these lines.
Quoted 4 lines
Do we need to worry about the possibility that when the system wakes up from hibernation, the set of usable CPUs might be smaller than it was beforehand? Is any special handling needed for this, or is it already accounted for?
That should not happen... but it does in some error cases.... so handling it would be a bonus.
Waking up with one cpu out of 8 is bad, but still way better than not waking up at all ;-).
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 16 lines · Alan Stern
Quoted 14 lines · Rafael J. Wysocki
Hi,
Some device drivers register CPU hotplug notifiers and use them to destroy device objects when removing the corresponding CPUs and to create these objects when adding the CPUs back.
Unfortunately, this is not the right thing to do during suspend/hibernation, since in that cases the CPU hotplug notifiers are called after suspending devices and before resuming them, so the operations in question are carried out on the objects representing suspended devices which shouldn't be unregistered behing the PM core's back. Although right now it usually doesn't lead to any practical complications, it will predictably deadlock if gregkh-driver-pm-acquire-device-locks-prior-to-suspending.patch is applied.
Quoted 8 lines
Quoted 4 lines
The solution is to prevent drivers from removing/adding devices from within CPU hotplug notifiers during suspend/hibernation using the FROZEN bit in the notifier's action argument. The following three patches modify the MSR, x86-64 MCE and cpuid drivers along these lines.
Do we need to worry about the possibility that when the system wakes up from hibernation, the set of usable CPUs might be smaller than it was beforehand?
This is possible in error conditions.
Quoted 1 line
Is any special handling needed for this, or is it already accounted for?
Hm, well. The cleanest thing would be to allow the drivers to remove the device objects on CPU_UP_CANCELED_FROZEN, which means that we weren't able to bring the CPU up during a resume, but still that will deadlock with gregkh-driver-pm-acquire-device-locks-prior-to-suspending.patch.
Greetings,
Rafael
Quoted 26 lines · Rafael J. Wysocki
Quoted 16 lines · Alan Stern
Quoted 14 lines · Rafael J. Wysocki
Hi,
Some device drivers register CPU hotplug notifiers and use them to destroy device objects when removing the corresponding CPUs and to create these objects when adding the CPUs back.
Unfortunately, this is not the right thing to do during suspend/hibernation, since in that cases the CPU hotplug notifiers are called after suspending devices and before resuming them, so the operations in question are carried out on the objects representing suspended devices which shouldn't be unregistered behing the PM core's back. Although right now it usually doesn't lead to any practical complications, it will predictably deadlock if gregkh-driver-pm-acquire-device-locks-prior-to-suspending.patch is applied.
Quoted 8 lines
Quoted 4 lines
The solution is to prevent drivers from removing/adding devices from within CPU hotplug notifiers during suspend/hibernation using the FROZEN bit in the notifier's action argument. The following three patches modify the MSR, x86-64 MCE and cpuid drivers along these lines.
Do we need to worry about the possibility that when the system wakes up from hibernation, the set of usable CPUs might be smaller than it was beforehand?
Quoted 8 lines
This is possible in error conditions.
Quoted 1 line
Is any special handling needed for this, or is it already accounted for?
Hm, well. The cleanest thing would be to allow the drivers to remove the device objects on
CPU_UP_CANCELED_FROZEN, which means that we weren't able to bring the CPU up during a resume, but still that will deadlock with gregkh-driver-pm-acquire-device-locks-prior-to-suspending.patch.
Hmm. In principle, device objects may be destroyed on CPU_UP_CANCELED_FROZEN without acquiring the device locks, since in fact we know these objects won't be accessed concurrently at that time (the locks are already held by the PM core, but the PM core is not going to actually access the devices before the subsequent resume).
Comments?
Thanks,
Rafael
Quoted 13 lines · Rafael J. Wysocki
Quoted 4 lines
Quoted 3 lines
Do we need to worry about the possibility that when the system wakes up from hibernation, the set of usable CPUs might be smaller than it was beforehand?
Quoted 8 lines
This is possible in error conditions.
Quoted 1 line
Is any special handling needed for this, or is it already accounted for?
Hm, well. The cleanest thing would be to allow the drivers to remove the device objects on
CPU_UP_CANCELED_FROZEN, which means that we weren't able to bring the CPU up during a resume, but still that will deadlock with gregkh-driver-pm-acquire-device-locks-prior-to-suspending.patch.
Quoted 5 lines
Hmm. In principle, device objects may be destroyed on
CPU_UP_CANCELED_FROZENwithout acquiring the device locks, since in fact we know these objects won't be accessed concurrently at that time (the locks are already held by the PM core, but the PM core is not going to actually access the devices before the subsequent resume).
How about delaying the CPU_UP_CANCELED_FROZEN announcements until it's really safe to send them out? That is, after all devices have been resumed and the PM core no longer holds any of their locks. (Should this be before or after tasks leave the freezer? -- I'm not sure.)
So the idea is send appropriate announcements at the usual time for CPUs that do come back up normally, and don't send anything right away for CPUs that fail to come up. Just keep track of which ones failed, and then later take care of them.
Alan Stern
Quoted 13 lines · Rafael J. Wysocki
Quoted 4 lines
Quoted 3 lines
Do we need to worry about the possibility that when the system wakes up from hibernation, the set of usable CPUs might be smaller than it was beforehand?
Quoted 8 lines
This is possible in error conditions.
Quoted 1 line
Is any special handling needed for this, or is it already accounted for?
Hm, well. The cleanest thing would be to allow the drivers to remove the device objects on
CPU_UP_CANCELED_FROZEN, which means that we weren't able to bring the CPU up during a resume, but still that will deadlock with gregkh-driver-pm-acquire-device-locks-prior-to-suspending.patch.
Quoted 5 lines
Hmm. In principle, device objects may be destroyed on
CPU_UP_CANCELED_FROZENwithout acquiring the device locks, since in fact we know these objects won't be accessed concurrently at that time (the locks are already held by the PM core, but the PM core is not going to actually access the devices before the subsequent resume).
How about delaying the CPU_UP_CANCELED_FROZEN announcements until it's really safe to send them out? That is, after all devices have been resumed and the PM core no longer holds any of their locks. (Should this be before or after tasks leave the freezer? -- I'm not sure.)
So the idea is send appropriate announcements at the usual time for CPUs that do come back up normally, and don't send anything right away for CPUs that fail to come up. Just keep track of which ones failed, and then later take care of them.
Alan Stern
Quoted 26 lines · Alan Stern
Quoted 13 lines · Rafael J. Wysocki
Quoted 4 lines
Quoted 3 lines
Do we need to worry about the possibility that when the system wakes up from hibernation, the set of usable CPUs might be smaller than it was beforehand?
Quoted 8 lines
This is possible in error conditions.
Quoted 1 line
Is any special handling needed for this, or is it already accounted for?
Hm, well. The cleanest thing would be to allow the drivers to remove the device objects on
CPU_UP_CANCELED_FROZEN, which means that we weren't able to bring the CPU up during a resume, but still that will deadlock with gregkh-driver-pm-acquire-device-locks-prior-to-suspending.patch.Quoted 5 lines
Hmm. In principle, device objects may be destroyed on
CPU_UP_CANCELED_FROZENwithout acquiring the device locks, since in fact we know these objects won't be accessed concurrently at that time (the locks are already held by the PM core, but the PM core is not going to actually access the devices before the subsequent resume).How about delaying the
CPU_UP_CANCELED_FROZENannouncements until it's really safe to send them out? That is, after all devices have been resumed and the PM core no longer holds any of their locks. (Should this be before or after tasks leave the freezer? -- I'm not sure.)
Quoted 4 lines
So the idea is send appropriate announcements at the usual time for CPUs that do come back up normally, and don't send anything right away for CPUs that fail to come up. Just keep track of which ones failed, and then later take care of them.
However, we don't want to execute .resume() for device objects that correspond to the "dead" CPUs, so to a minimum we should remove them from the dpm_off list on CPU_UP_CANCELED_FROZEN. For this purpose, we can define a callback that will remove the device from dpm_off immediately and schedule its destruction after all devices have been resumed.
Rafael
Quoted 26 lines · Alan Stern
Quoted 13 lines · Rafael J. Wysocki
Quoted 4 lines
Quoted 3 lines
Do we need to worry about the possibility that when the system wakes up from hibernation, the set of usable CPUs might be smaller than it was beforehand?
Quoted 8 lines
This is possible in error conditions.
Quoted 1 line
Is any special handling needed for this, or is it already accounted for?
Hm, well. The cleanest thing would be to allow the drivers to remove the device objects on
CPU_UP_CANCELED_FROZEN, which means that we weren't able to bring the CPU up during a resume, but still that will deadlock with gregkh-driver-pm-acquire-device-locks-prior-to-suspending.patch.Quoted 5 lines
Hmm. In principle, device objects may be destroyed on
CPU_UP_CANCELED_FROZENwithout acquiring the device locks, since in fact we know these objects won't be accessed concurrently at that time (the locks are already held by the PM core, but the PM core is not going to actually access the devices before the subsequent resume).How about delaying the
CPU_UP_CANCELED_FROZENannouncements until it's really safe to send them out? That is, after all devices have been resumed and the PM core no longer holds any of their locks. (Should this be before or after tasks leave the freezer? -- I'm not sure.)
Quoted 4 lines
So the idea is send appropriate announcements at the usual time for CPUs that do come back up normally, and don't send anything right away for CPUs that fail to come up. Just keep track of which ones failed, and then later take care of them.
However, we don't want to execute .resume() for device objects that correspond to the "dead" CPUs, so to a minimum we should remove them from the dpm_off list on CPU_UP_CANCELED_FROZEN. For this purpose, we can define a callback that will remove the device from dpm_off immediately and schedule its destruction after all devices have been resumed.
Rafael
Quoted 16 lines · Alan Stern
Quoted 14 lines · Rafael J. Wysocki
Hi,
Some device drivers register CPU hotplug notifiers and use them to destroy device objects when removing the corresponding CPUs and to create these objects when adding the CPUs back.
Unfortunately, this is not the right thing to do during suspend/hibernation, since in that cases the CPU hotplug notifiers are called after suspending devices and before resuming them, so the operations in question are carried out on the objects representing suspended devices which shouldn't be unregistered behing the PM core's back. Although right now it usually doesn't lead to any practical complications, it will predictably deadlock if gregkh-driver-pm-acquire-device-locks-prior-to-suspending.patch is applied.
Quoted 8 lines
Quoted 4 lines
The solution is to prevent drivers from removing/adding devices from within CPU hotplug notifiers during suspend/hibernation using the FROZEN bit in the notifier's action argument. The following three patches modify the MSR, x86-64 MCE and cpuid drivers along these lines.
Do we need to worry about the possibility that when the system wakes up from hibernation, the set of usable CPUs might be smaller than it was beforehand?
This is possible in error conditions.
Quoted 1 line
Is any special handling needed for this, or is it already accounted for?
Hm, well. The cleanest thing would be to allow the drivers to remove the device objects on CPU_UP_CANCELED_FROZEN, which means that we weren't able to bring the CPU up during a resume, but still that will deadlock with gregkh-driver-pm-acquire-device-locks-prior-to-suspending.patch.
Greetings,
Rafael
Quoted 14 lines · Rafael J. Wysocki
Hi,
Some device drivers register CPU hotplug notifiers and use them to destroy device objects when removing the corresponding CPUs and to create these objects when adding the CPUs back.
Unfortunately, this is not the right thing to do during suspend/hibernation, since in that cases the CPU hotplug notifiers are called after suspending devices and before resuming them, so the operations in question are carried out on the objects representing suspended devices which shouldn't be unregistered behing the PM core's back. Although right now it usually doesn't lead to any practical complications, it will predictably deadlock if gregkh-driver-pm-acquire-device-locks-prior-to-suspending.patch is applied.
Quoted 4 lines
The solution is to prevent drivers from removing/adding devices from within CPU hotplug notifiers during suspend/hibernation using the FROZEN bit in the notifier's action argument. The following three patches modify the MSR, x86-64 MCE and cpuid drivers along these lines.
Do we need to worry about the possibility that when the system wakes up from hibernation, the set of usable CPUs might be smaller than it was beforehand? Is any special handling needed for this, or is it already accounted for?
Alan Stern