parallel suspend/resume
Subsystems (4)
Hi,
I spent some time working with the parallel suspend/resume idea, even wrote a simple patch (attached), but that obviously does not scale to large number of devices.
What would be a better approach?
drivers/base/power/main.c +2 -0
@@ -62,6 +62,8 @@ int device_pm_add(struct device * dev)down (&dpm_list_sem);list_add_tail (&dev->power.entry, &dpm_active);device_pm_set_parent (dev, dev->parent); +init_waitqueue_head (&dev->power.powered_on); +INIT_LIST_HEAD (&dev->power.entry_sus);if ((error =dpm_sysfs_add (dev)))list_del (&dev->power.entry);up (&dpm_list_sem);
drivers/base/power/resume.c +48 -2
99 lines not on this page · raw message
drivers/base/power/suspend.c +3 -1
@@ -169,8 +169,10 @@ int device_suspend(pm_message_t state)/* Check if the device got removed */ if (!list_empty (&dev->power.entry)) {/* Move it to the dpm_off list */ -if (!error) +if (!error) { + dev->power.power_state =PMSG_SUSPEND ;list_move (&dev->power.entry, &dpm_off); + } }if (error) printk(KERN_ERR "Could not suspend device %s: "
include/linux/pm.h +4 -0
@@ -24,6 +24,7 @@ #define _LINUX_PM_H#ifdef __KERNEL__#include <linux/list.h> +#include <linux/wait.h> #include <asm/atomic.h> /*
@@ -271,6 +272,9 @@ #ifdef CONFIG_PMvoid * saved_state;struct device * pm_parent;struct list_head entry; +struct task_struct * resume_thread; +wait_queue_head_t powered_on; +struct list_head entry_sus;#endif };
Quoted 7 lines · Marcelo Tosatti
Hi,
I spent some time working with the parallel suspend/resume idea, even wrote a simple patch (attached), but that obviously does not scale to large number of devices.
What would be a better approach?
In general, parallel suspend/resume isn't easy to do correctly. There are lots of constraints on the order in which devices can be suspended, many of them not explicitly stated anywhere. I don't think it is feasible to encapsulate the constraint information in the PM core; if anything, it should be left up to the individual subsystems to suspend or resume their devices in parallel.
Alan Stern
Quoted 7 lines · Marcelo Tosatti
Hi,
I spent some time working with the parallel suspend/resume idea, even wrote a simple patch (attached), but that obviously does not scale to large number of devices.
What would be a better approach?
No idea, but thread per device does not seem _that_ bad. Debuggability worries me more.
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 11 lines · Pavel Machek
Quoted 7 lines · Marcelo Tosatti
Hi,
I spent some time working with the parallel suspend/resume idea, even wrote a simple patch (attached), but that obviously does not scale to large number of devices.
What would be a better approach?
No idea, but thread per device does not seem _that_ bad. Debuggability worries me more.
What about ordering constraints? A lot of them aren't explicit. We just depend on the fact that devices get suspended in reverse order of registration. You can't do that in parallel.
Alan Stern
Quoted 3 lines
What about ordering constraints? A lot of them aren't explicit. We just depend on the fact that devices get suspended in reverse order of registration. You can't do that in parallel.
Those constraints should *become* explicit ... not stay implicit!
FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.
I'd expect this to eventually turn up some cases where Linux is right now doing a bad job of exposing devices with multiple such ordering constraint ... power management via I2C seems likely to turn up a few of them. We'd need some automatic way to detect and report such problems so that they could be fixed.
- Dave
============= SNIP!
Change how the PM list is constructed, so that devices are added right after their parents (when they have one) rather than at the end of the list. This preserves sequencing guarantees, but enables sequencing of suspend/resume operations by more important characteristics than "when device happened to enumerate" ... e.g. clocksources and clockevents at a clearly defined point during suspend and resume, power management chips nearer their I2C parent.
This patch has a potential downside for devices that have multiple power dependencies and which "just happened to work" before. Or looking at it the other way around: a potential upside for devices that didn't work before.
--- drivers/base/power/main.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
drivers/base/power/main.c +9 -3
@@ -40,11 +40,17 @@ int (*platform_enable_wakeup)(struct devvoid device_pm_add (struct device *dev) { -pr_debug ("PM: Adding info for %s:%s\n" , +struct device *parent = dev->parent; + +pr_debug ("PM: Adding info for %s:%s after %s\n" , dev->bus ? dev->bus->name :"No Bus" , -kobject_name (&dev->kobj)); +kobject_name (&dev->kobj), + parent ? parent->bus_id :"(no parent)" );mutex_lock (&dpm_list_mtx); -list_add_tail (&dev->power.entry, &dpm_active); +if (parent) +list_add (&dev->power.entry, &parent->power.entry); +else +list_add_tail (&dev->power.entry, &dpm_active);mutex_unlock (&dpm_list_mtx); }
Quoted 5 lines · David Brownell
Quoted 3 lines
What about ordering constraints? A lot of them aren't explicit. We just depend on the fact that devices get suspended in reverse order of registration. You can't do that in parallel.
Those constraints should *become* explicit ... not stay implicit!
Why do I get the feeling we've gone through this before? :-)
Quoted 1 line
FWIW the appended patch removes that rude "order of registration"
"Rude"? What's rude about it? It seems most logical to me: Since the devices obviously worked in the intermediate stages of registration, when the early-to-register devices were available and the late-to-register devices weren't, duplicating those same intermediate stages (with the same sets of devices available) should also work during suspend.
Quoted 1 line
policy, so that the suspend/resume list matches the device tree.
In what respect doesn't it match the device tree now?
As far as I know, the only way in which the list wouldn't match the tree is when the tree gets reorganized by device_move(), which doesn't change the list at all. (This is quite possibly a bug.) Your patch doesn't affect that.
Quoted 2 lines
It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.
I predict it will cause failure of the USB-Persist mechanism when a non-high-speed device is plugged into a dual high/full-speed port. This is because when an EHCI root hub is resumed after a loss of power, the driver looks for ports that had been owned by the companion controllers and tries to hand them over again -- but if the companion controllers haven't been resumed yet the handover won't work.
However to be fair, about the best one can say for the current way this situation is handled is that it works more or less by coincidence (EHCI controllers are required by the spec to have higher function numbers than their companion controllers on a PCI board, and PCI devices are enumerated in increasing order by function number). I have to agree that this ordering dependency really should be made explicit.
Quoted 5 lines
I'd expect this to eventually turn up some cases where Linux is right now doing a bad job of exposing devices with multiple such ordering constraint ... power management via I2C seems likely to turn up a few of them. We'd need some automatic way to detect and report such problems so that they could be fixed.
If you really want to sniff out dependencies, you should add each new device to the list at a _random_ position, subject only to the constraint that it must not precede its parent!
Alan Stern
Am Freitag, 7. Dezember 2007 19:01:12 schrieb David Brownell:
Quoted 4 lines
FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.
As it is a tree, why not store it as such?
Regards
Oliver
Quoted 7 lines · Oliver Neukum
Am Freitag, 7. Dezember 2007 19:01:12 schrieb David Brownell:
Quoted 4 lines
FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.
As it is a tree, why not store it as such?
IIRC because we do not want recursive tree walkers in the kernel -- stack limits.
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 9 lines · Pavel Machek
Quoted 7 lines · Oliver Neukum
Am Freitag, 7. Dezember 2007 19:01:12 schrieb David Brownell:
Quoted 5 lines
FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.
Quoted 7 lines
Quoted 2 lines
As it is a tree, why not store it as such?
IIRC because we do not want recursive tree walkers in the kernel -- stack limits.
Surely it is possible to code a tree walker that is not a recursive function?
I believe you can use a simple loop and the heap to store state that would otherwise be stored on the stack in a recursive function.
Am Samstag, 8. Dezember 2007 11:21:43 schrieb Pavel Machek:
Quoted 11 lines · Oliver Neukum
Quoted 7 lines
Am Freitag, 7. Dezember 2007 19:01:12 schrieb David Brownell:
Quoted 4 lines
FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.
As it is a tree, why not store it as such?
IIRC because we do not want recursive tree walkers in the kernel -- stack limits.
Then use a non-recursive algorithm.
Regards
Oliver
Am Samstag, 8. Dezember 2007 12:08:41 schrieb James Courtier-Dutton:
Quoted 18 lines
Quoted 9 lines · Pavel Machek
Quoted 7 lines · Oliver Neukum
Am Freitag, 7. Dezember 2007 19:01:12 schrieb David Brownell:
Quoted 5 lines
FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.
Quoted 7 lines
Quoted 2 lines
As it is a tree, why not store it as such?
IIRC because we do not want recursive tree walkers in the kernel -- stack limits.
Quoted 4 lines
Surely it is possible to code a tree walker that is not a recursive function?
I believe you can use a simple loop and the heap to store state that would otherwise be stored on the stack in a recursive function.
If you store a pointer to the parent in each node, you can do with static memory of only the current node and the previous node.
Regards
Oliver
Quoted 7 lines · Oliver Neukum
Am Freitag, 7. Dezember 2007 19:01:12 schrieb David Brownell:
Quoted 4 lines
FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.
As it is a tree, why not store it as such?
There's no need to "store" the tree ordering specially, since all the pointers already exist. The question is: In what order should the tree be traversed? About the only explicit constraint we have now is that children must be suspended before their parents, but there undoubtedly are plenty of undocumented implicit constraints (maybe some of them aren't known to anybody at all).
Given the vast number of possible orders, and given that the only order we _know_ works correctly is reverse order of registration, I don't see any big reason to change. Speeding things up by parallel suspension would be a valid reason, but it needs to be done with a great deal of care.
Alan Stern
Quoted 4 lines · David Brownell
FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.
Actually I'm surprised that it works with USB devices.
As they are registered, you patch adds new devices to the list immediately after their parents. This means that the children of a particular device will be listed in reverse order of registration, right?
So if you have a USB hub, the hub's children will be added following the hub, and the hub's interface will come after all the children since it gets registered before them. Hence during suspend, the interface's suspend method will be called before any of the children are suspended. But hub_suspend() will fail if there are unsuspended children.
This is an example of another implicit order dependency: A hub's interface must be suspended after all the other children of that hub. It's an artifact of the way USB devices are represented in the device hierarchy; strictly speaking they should be children of the hub's interface, not of the hub itself.
Alan Stern
Quoted 8 lines · Alan Stern
Quoted 6 lines · Oliver Neukum
Am Freitag, 7. Dezember 2007 19:01:12 schrieb David Brownell:
Quoted 4 lines
FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.
Quoted 8 lines
Quoted 1 line
As it is a tree, why not store it as such?
There's no need to "store" the tree ordering specially, since all the pointers already exist. The question is: In what order should the tree be traversed? About the only explicit constraint we have now is that children must be suspended before their parents, but there undoubtedly are plenty of undocumented implicit constraints (maybe some of them aren't known to anybody at all).
Yes, that makes me nervous every time someone suggests to change the ordering of suspending devices.
Quoted 5 lines
Given the vast number of possible orders, and given that the only order we _know_ works correctly is reverse order of registration, I don't see any big reason to change. Speeding things up by parallel suspension would be a valid reason, but it needs to be done with a great deal of care.
Agreed.
Greetings,
Rafael
Quoted 19 lines · Alan Stern
Quoted 5 lines · David Brownell
Quoted 3 lines
What about ordering constraints? A lot of them aren't explicit. We just depend on the fact that devices get suspended in reverse order of registration. You can't do that in parallel.
Those constraints should *become* explicit ... not stay implicit!
Why do I get the feeling we've gone through this before? :-)
Quoted 1 line
FWIW the appended patch removes that rude "order of registration"
"Rude"? What's rude about it? It seems most logical to me: Since the devices obviously worked in the intermediate stages of registration, when the early-to-register devices were available and the late-to-register devices weren't, duplicating those same intermediate stages (with the same sets of devices available) should also work during suspend.
Quoted 7 lines
Quoted 1 line
policy, so that the suspend/resume list matches the device tree.
In what respect doesn't it match the device tree now?
As far as I know, the only way in which the list wouldn't match the tree is when the tree gets reorganized by
device_move(), which doesn't change the list at all. (This is quite possibly a bug.)
IMO, this _is_ a bug. I think device_move() should move the device in question to the end of the list along with its children.
That might cause some breakage in the short run, but we'll have to do it at one point.
Greetings,
Rafael
Quoted 14 lines · Alan Stern
Quoted 4 lines · David Brownell
FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.
Actually I'm surprised that it works with USB devices.
As they are registered, you patch adds new devices to the list immediately after their parents. This means that the children of a particular device will be listed in reverse order of registration, right?
Quoted 5 lines
So if you have a USB hub, the hub's children will be added following the hub, and the hub's interface will come after all the children since it gets registered before them. Hence during suspend, the interface's suspend method will be called before any of the children are suspended. But
hub_suspend()will fail if there are unsuspended children.
I take it back. Interfaces get suspended along with their device; hence it doesn't matter in what order the interface is listed with respect to the children.
Alan Stern
Am Samstag, 8. Dezember 2007 16:43:25 schrieb Alan Stern:
Quoted 8 lines · Oliver Neukum
Quoted 7 lines
Am Freitag, 7. Dezember 2007 19:01:12 schrieb David Brownell:
Quoted 4 lines
FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.
Quoted 4 lines
Quoted 1 line
As it is a tree, why not store it as such?
There's no need to "store" the tree ordering specially, since all the pointers already exist. The question is: In what order should the tree
Nevertheless, we currently have a list. Why? To reverse the temporal order only?
Quoted 4 lines
be traversed? About the only explicit constraint we have now is that children must be suspended before their parents, but there undoubtedly are plenty of undocumented implicit constraints (maybe some of them aren't known to anybody at all).
We will need to know them for runtime pm.
Quoted 2 lines
Given the vast number of possible orders, and given that the only order we _know_ works correctly is reverse order of registration, I don't see
modus advocati diaboli:
Suppose I have a system with a FibreChannel disk. Now I hot plug another FibreChannel controller and connect it to the disk. Then I disconnect the disk from the original controller. What will happen if I suspend the system?
Quoted 3 lines
any big reason to change. Speeding things up by parallel suspension would be a valid reason, but it needs to be done with a great deal of care.
catch-22.
Regards
Oliver
Quoted 9 lines · Oliver Neukum
Am Samstag, 8. Dezember 2007 16:43:25 schrieb Alan Stern:
Quoted 8 lines · Oliver Neukum
Quoted 7 lines
Am Freitag, 7. Dezember 2007 19:01:12 schrieb David Brownell:
Quoted 4 lines
FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
As it is a tree, why not store it as such?
There's no need to "store" the tree ordering specially, since all the pointers already exist. The question is: In what order should the tree
Nevertheless, we currently have a list. Why? To reverse the temporal order only?
As far as I know, yes. Maybe also to simplify the traversal; it takes less code to follow a list than to go through a tree.
Quoted 6 lines
Quoted 4 lines
be traversed? About the only explicit constraint we have now is that children must be suspended before their parents, but there undoubtedly are plenty of undocumented implicit constraints (maybe some of them aren't known to anybody at all).
We will need to know them for runtime pm.
Maybe. But that knowledge will be localized in drivers and not consolidated in the PM core, since that's where runtime PM is handled. What we don't have is any way for drivers to export the knowledge.
Quoted 8 lines
Quoted 2 lines
Given the vast number of possible orders, and given that the only order we _know_ works correctly is reverse order of registration, I don't see
modus advocati diaboli:
Suppose I have a system with a FibreChannel disk. Now I hot plug another FibreChannel controller and connect it to the disk. Then I disconnect the disk from the original controller. What will happen if I suspend the system?
That seems a little strange. What happens during the intermediate stage when you have a single disk connected to the computer by two I/O pathways? Will the OS think it is really two disks? Will it get confused by seeing on-disk data structures in unstable incomplete states and subject to apparently random updates?
A better question is what happens when a Bluetooth communications driver creates a new tty device and then uses device_move() to put the Bluetooth device under the newer tty. Will the tty get suspended before its Bluetooth child?
Alan Stern
Quoted 9 lines · Alan Stern
Quoted 5 lines · David Brownell
Quoted 3 lines
What about ordering constraints? A lot of them aren't explicit. We just depend on the fact that devices get suspended in reverse order of registration. You can't do that in parallel.
Those constraints should *become* explicit ... not stay implicit!
Why do I get the feeling we've gone through this before? :-)
Why indeed.
Quoted 5 lines
Quoted 1 line
FWIW the appended patch removes that rude "order of registration"
"Rude"? What's rude about it? It seems most logical to me: Since the devices obviously worked in the intermediate stages of registration,
...
Rude because it keeps things implicit which should be explicit.
Quoted 3 lines
Quoted 1 line
policy, so that the suspend/resume list matches the device tree.
In what respect doesn't it match the device tree now?
It's neither a depth-first nor a breadth-first traversal; it's more of a random node visitation based on what devices were initialized when.
Such a random visitation seems like a clear problem when a goal is to introduce parallelism ...
Quoted 4 lines
As far as I know, the only way in which the list wouldn't match the tree is when the tree gets reorganized by
device_move(), which doesn't change the list at all. (This is quite possibly a bug.) Your patch doesn't affect that.
Right, device_move() should update the list. Different issue.
Quoted 5 lines
Quoted 2 lines
It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.
... I have to agree
that this ordering dependency really should be made explicit.
Good. :)
Quoted 9 lines
Quoted 5 lines
I'd expect this to eventually turn up some cases where Linux is right now doing a bad job of exposing devices with multiple such ordering constraint ... power management via I2C seems likely to turn up a few of them. We'd need some automatic way to detect and report such problems so that they could be fixed.
If you really want to sniff out dependencies, you should add each new device to the list at a _random_ position, subject only to the constraint that it must not precede its parent!
Nope; what we need is the ability to finger "this specific device has dependencies that aren't explicit". Being able to introduce random failures isn't really helpful. It's the difference between being able to use lockdep vs. having to wait until a random locking failure crops up in a way that can usefully be diagnosed.
- Dave
Quoted 7 lines · David Brownell
Quoted 5 lines
Quoted 1 line
FWIW the appended patch removes that rude "order of registration"
"Rude"? What's rude about it? It seems most logical to me: Since the devices obviously worked in the intermediate stages of registration,
...Rude because it keeps things implicit which should be explicit.
Your patch isn't really much better (IMHO). While its order may be less "random" in the sense that it is determined by the device tree, the tree itself is somewhat random and not fully determined by the physical hardware layout. It is affected by the order in which drivers are loaded, or the order in which PCI cards are probed, and so on.
And in any case, having a fixed order doesn't move us any closer to the goal of making all the dependencies explicit.
Quoted 2 lines
Such a random visitation seems like a clear problem when a goal is to introduce parallelism ...
So does a strict depth-first visitation. Or indeed, any fixed ordering, since parallelism will involve indeterminacy.
Quoted 5 lines
Nope; what we need is the ability to finger "this specific device has dependencies that aren't explicit". Being able to introduce random failures isn't really helpful. It's the difference between being able to use lockdep vs. having to wait until a random locking failure crops up in a way that can usefully be diagnosed.
I agree. Even better would be to say "this specific device has this specific dependency". But it's not easy to come up with strategies for finding this information. There's no clear-cut analog to lockdep.
Alan Stern