Skip to contentSkip to conversation

parallel suspend/resume

PATCH Marcelo Tosatti 7 participants 20 messages Oct 8, 2007 – Dec 10, 2007

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

Hi,

I spent some time working with the parallel suspend/resume idea, even wrote a simple patch (attached), but that obviously does not scale to large number of devices.

What would be a better approach?

drivers/base/power/main.c +2 -0
index 05dc876..45f5851 100644
@@ -62,6 +62,8 @@ int device_pm_add(struct device * dev)
 	down(&dpm_list_sem);
 	list_add_tail(&dev->power.entry, &dpm_active);
 	device_pm_set_parent(dev, dev->parent);
+	init_waitqueue_head(&dev->power.powered_on);
+	INIT_LIST_HEAD(&dev->power.entry_sus);
 	if ((error = dpm_sysfs_add(dev)))
 		list_del(&dev->power.entry);
 	up(&dpm_list_sem);
drivers/base/power/resume.c +48 -2

99 lines not on this page · raw message

drivers/base/power/suspend.c +3 -1
index 42d2b86..3f6e9b0 100644
@@ -169,8 +169,10 @@ int device_suspend(pm_message_t state)
 		/* Check if the device got removed */
 		if (!list_empty(&dev->power.entry)) {
 			/* Move it to the dpm_off list */
-			if (!error)
+			if (!error) {
+				dev->power.power_state = PMSG_SUSPEND;
 				list_move(&dev->power.entry, &dpm_off);
+			}
 		}
 		if (error)
 			printk(KERN_ERR "Could not suspend device %s: "
include/linux/pm.h +4 -0
index b2c4fde..4bfa8da 100644
@@ -24,6 +24,7 @@ #define _LINUX_PM_H
 #ifdef __KERNEL__
 
 #include <linux/list.h>
+#include <linux/wait.h>
 #include <asm/atomic.h>
 
 /*
@@ -271,6 +272,9 @@ #ifdef	CONFIG_PM
 	void			* saved_state;
 	struct device		* pm_parent;
 	struct list_head	entry;
+	struct task_struct 	* resume_thread;
+	wait_queue_head_t 	powered_on;
+	struct list_head entry_sus;
 #endif
 };
 
#1 lore raw
Quoted 7 lines · Marcelo Tosatti

Hi,

I spent some time working with the parallel suspend/resume idea, even wrote a simple patch (attached), but that obviously does not scale to large number of devices.

What would be a better approach?

In general, parallel suspend/resume isn't easy to do correctly. There are lots of constraints on the order in which devices can be suspended, many of them not explicitly stated anywhere. I don't think it is feasible to encapsulate the constraint information in the PM core; if anything, it should be left up to the individual subsystems to suspend or resume their devices in parallel.

Alan Stern

#2 lore raw
In reply to Marcelo Tosatti
Quoted 7 lines · Marcelo Tosatti

Hi,

I spent some time working with the parallel suspend/resume idea, even wrote a simple patch (attached), but that obviously does not scale to large number of devices.

What would be a better approach?

No idea, but thread per device does not seem _that_ bad. Debuggability worries me more.

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#3 lore raw
Quoted 11 lines · Pavel Machek
Quoted 7 lines · Marcelo Tosatti

Hi,

I spent some time working with the parallel suspend/resume idea, even wrote a simple patch (attached), but that obviously does not scale to large number of devices.

What would be a better approach?

No idea, but thread per device does not seem _that_ bad. Debuggability worries me more.

What about ordering constraints? A lot of them aren't explicit. We just depend on the fact that devices get suspended in reverse order of registration. You can't do that in parallel.

Alan Stern

#4 lore raw
Quoted 3 lines

What about ordering constraints? A lot of them aren't explicit. We just depend on the fact that devices get suspended in reverse order of registration. You can't do that in parallel.

Those constraints should *become* explicit ... not stay implicit!

FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.

I'd expect this to eventually turn up some cases where Linux is right now doing a bad job of exposing devices with multiple such ordering constraint ... power management via I2C seems likely to turn up a few of them. We'd need some automatic way to detect and report such problems so that they could be fixed.

- Dave

============= SNIP!
Change how the PM list is constructed, so that devices are added right after their parents (when they have one) rather than at the end of the list. This preserves sequencing guarantees, but enables sequencing of suspend/resume operations by more important characteristics than "when device happened to enumerate" ... e.g. clocksources and clockevents at a clearly defined point during suspend and resume, power management chips nearer their I2C parent.

This patch has a potential downside for devices that have multiple power dependencies and which "just happened to work" before. Or looking at it the other way around: a potential upside for devices that didn't work before.

---
 drivers/base/power/main.c |   12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
drivers/base/power/main.c +9 -3
@@ -40,11 +40,17 @@ int (*platform_enable_wakeup)(struct dev
 
 void device_pm_add(struct device *dev)
 {
-	pr_debug("PM: Adding info for %s:%s\n",
+	struct device *parent = dev->parent;
+
+	pr_debug("PM: Adding info for %s:%s after %s\n",
 		 dev->bus ? dev->bus->name : "No Bus",
-		 kobject_name(&dev->kobj));
+		 kobject_name(&dev->kobj),
+		 parent ? parent->bus_id : "(no parent)");
 	mutex_lock(&dpm_list_mtx);
-	list_add_tail(&dev->power.entry, &dpm_active);
+	if (parent)
+		list_add(&dev->power.entry, &parent->power.entry);
+	else
+		list_add_tail(&dev->power.entry, &dpm_active);
 	mutex_unlock(&dpm_list_mtx);
 }
 
#5 lore raw
Quoted 5 lines · David Brownell
Quoted 3 lines

What about ordering constraints? A lot of them aren't explicit. We just depend on the fact that devices get suspended in reverse order of registration. You can't do that in parallel.

Those constraints should *become* explicit ... not stay implicit!

Why do I get the feeling we've gone through this before? :-)

Quoted 1 line

FWIW the appended patch removes that rude "order of registration"

"Rude"? What's rude about it? It seems most logical to me: Since the devices obviously worked in the intermediate stages of registration, when the early-to-register devices were available and the late-to-register devices weren't, duplicating those same intermediate stages (with the same sets of devices available) should also work during suspend.

Quoted 1 line

policy, so that the suspend/resume list matches the device tree.

In what respect doesn't it match the device tree now?

As far as I know, the only way in which the list wouldn't match the tree is when the tree gets reorganized by device_move(), which doesn't change the list at all. (This is quite possibly a bug.) Your patch doesn't affect that.

Quoted 2 lines

It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.

I predict it will cause failure of the USB-Persist mechanism when a non-high-speed device is plugged into a dual high/full-speed port. This is because when an EHCI root hub is resumed after a loss of power, the driver looks for ports that had been owned by the companion controllers and tries to hand them over again -- but if the companion controllers haven't been resumed yet the handover won't work.

However to be fair, about the best one can say for the current way this situation is handled is that it works more or less by coincidence (EHCI controllers are required by the spec to have higher function numbers than their companion controllers on a PCI board, and PCI devices are enumerated in increasing order by function number). I have to agree that this ordering dependency really should be made explicit.

Quoted 5 lines

I'd expect this to eventually turn up some cases where Linux is right now doing a bad job of exposing devices with multiple such ordering constraint ... power management via I2C seems likely to turn up a few of them. We'd need some automatic way to detect and report such problems so that they could be fixed.

If you really want to sniff out dependencies, you should add each new device to the list at a _random_ position, subject only to the constraint that it must not precede its parent!

Alan Stern

#6 lore raw
Quoted 19 lines · Alan Stern
Quoted 5 lines · David Brownell
Quoted 3 lines

What about ordering constraints? A lot of them aren't explicit. We just depend on the fact that devices get suspended in reverse order of registration. You can't do that in parallel.

Those constraints should *become* explicit ... not stay implicit!

Why do I get the feeling we've gone through this before? :-)

Quoted 1 line

FWIW the appended patch removes that rude "order of registration"

"Rude"? What's rude about it? It seems most logical to me: Since the devices obviously worked in the intermediate stages of registration, when the early-to-register devices were available and the late-to-register devices weren't, duplicating those same intermediate stages (with the same sets of devices available) should also work during suspend.

Quoted 7 lines
Quoted 1 line

policy, so that the suspend/resume list matches the device tree.

In what respect doesn't it match the device tree now?

As far as I know, the only way in which the list wouldn't match the tree is when the tree gets reorganized by device_move(), which doesn't change the list at all. (This is quite possibly a bug.)

IMO, this _is_ a bug. I think device_move() should move the device in question to the end of the list along with its children.

That might cause some breakage in the short run, but we'll have to do it at one point.

Greetings,
Rafael

#7 lore raw
In reply to Alan Stern
Quoted 9 lines · Alan Stern
Quoted 5 lines · David Brownell
Quoted 3 lines

What about ordering constraints? A lot of them aren't explicit. We just depend on the fact that devices get suspended in reverse order of registration. You can't do that in parallel.

Those constraints should *become* explicit ... not stay implicit!

Why do I get the feeling we've gone through this before? :-)

Why indeed.

Quoted 5 lines
Quoted 1 line

FWIW the appended patch removes that rude "order of registration"

"Rude"? What's rude about it? It seems most logical to me: Since the devices obviously worked in the intermediate stages of registration,
...

Rude because it keeps things implicit which should be explicit.

Quoted 3 lines
Quoted 1 line

policy, so that the suspend/resume list matches the device tree.

In what respect doesn't it match the device tree now?

It's neither a depth-first nor a breadth-first traversal; it's more of a random node visitation based on what devices were initialized when.

Such a random visitation seems like a clear problem when a goal is to introduce parallelism ...

Quoted 4 lines

As far as I know, the only way in which the list wouldn't match the tree is when the tree gets reorganized by device_move(), which doesn't change the list at all. (This is quite possibly a bug.) Your patch doesn't affect that.

Right, device_move() should update the list. Different issue.

Quoted 5 lines
Quoted 2 lines

It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.

... I have to agree
that this ordering dependency really should be made explicit.

Good. :)

Quoted 9 lines
Quoted 5 lines

I'd expect this to eventually turn up some cases where Linux is right now doing a bad job of exposing devices with multiple such ordering constraint ... power management via I2C seems likely to turn up a few of them. We'd need some automatic way to detect and report such problems so that they could be fixed.

If you really want to sniff out dependencies, you should add each new device to the list at a _random_ position, subject only to the constraint that it must not precede its parent!

Nope; what we need is the ability to finger "this specific device has dependencies that aren't explicit". Being able to introduce random failures isn't really helpful. It's the difference between being able to use lockdep vs. having to wait until a random locking failure crops up in a way that can usefully be diagnosed.

- Dave

#8 lore raw
Quoted 7 lines · David Brownell
Quoted 5 lines
Quoted 1 line

FWIW the appended patch removes that rude "order of registration"

"Rude"? What's rude about it? It seems most logical to me: Since the devices obviously worked in the intermediate stages of registration,
...

Rude because it keeps things implicit which should be explicit.

Your patch isn't really much better (IMHO). While its order may be less "random" in the sense that it is determined by the device tree, the tree itself is somewhat random and not fully determined by the physical hardware layout. It is affected by the order in which drivers are loaded, or the order in which PCI cards are probed, and so on.

And in any case, having a fixed order doesn't move us any closer to the goal of making all the dependencies explicit.

Quoted 2 lines

Such a random visitation seems like a clear problem when a goal is to introduce parallelism ...

So does a strict depth-first visitation. Or indeed, any fixed ordering, since parallelism will involve indeterminacy.

Quoted 5 lines

Nope; what we need is the ability to finger "this specific device has dependencies that aren't explicit". Being able to introduce random failures isn't really helpful. It's the difference between being able to use lockdep vs. having to wait until a random locking failure crops up in a way that can usefully be diagnosed.

I agree. Even better would be to say "this specific device has this specific dependency". But it's not easy to come up with strategies for finding this information. There's no clear-cut analog to lockdep.

Alan Stern

#9 lore raw
In reply to David Brownell

Am Freitag, 7. Dezember 2007 19:01:12 schrieb David Brownell:

Quoted 4 lines

FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.

As it is a tree, why not store it as such?

Regards
Oliver

#10 lore raw
Quoted 7 lines · Oliver Neukum

Am Freitag, 7. Dezember 2007 19:01:12 schrieb David Brownell:

Quoted 4 lines

FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.

As it is a tree, why not store it as such?

IIRC because we do not want recursive tree walkers in the kernel -- stack limits.

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#11 lore raw
Quoted 9 lines · Pavel Machek
Quoted 7 lines · Oliver Neukum

Am Freitag, 7. Dezember 2007 19:01:12 schrieb David Brownell:

Quoted 5 lines

FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.

Quoted 7 lines
Quoted 2 lines

As it is a tree, why not store it as such?

IIRC because we do not want recursive tree walkers in the kernel -- stack limits.

Surely it is possible to code a tree walker that is not a recursive function?
I believe you can use a simple loop and the heap to store state that would otherwise be stored on the stack in a recursive function.

#12 lore raw

Am Samstag, 8. Dezember 2007 12:08:41 schrieb James Courtier-Dutton:

Quoted 18 lines
Quoted 9 lines · Pavel Machek
Quoted 7 lines · Oliver Neukum

Am Freitag, 7. Dezember 2007 19:01:12 schrieb David Brownell:

Quoted 5 lines

FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.

Quoted 7 lines
Quoted 2 lines

As it is a tree, why not store it as such?

IIRC because we do not want recursive tree walkers in the kernel -- stack limits.

Quoted 4 lines

Surely it is possible to code a tree walker that is not a recursive function?
I believe you can use a simple loop and the heap to store state that would otherwise be stored on the stack in a recursive function.

If you store a pointer to the parent in each node, you can do with static memory of only the current node and the previous node.

Regards
Oliver

#13 lore raw
In reply to Pavel Machek

Am Samstag, 8. Dezember 2007 11:21:43 schrieb Pavel Machek:

Quoted 11 lines · Oliver Neukum
Quoted 7 lines

Am Freitag, 7. Dezember 2007 19:01:12 schrieb David Brownell:

Quoted 4 lines

FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.

As it is a tree, why not store it as such?

IIRC because we do not want recursive tree walkers in the kernel -- stack limits.

Then use a non-recursive algorithm.

Regards
Oliver

#14 lore raw
In reply to Oliver Neukum
Quoted 7 lines · Oliver Neukum

Am Freitag, 7. Dezember 2007 19:01:12 schrieb David Brownell:

Quoted 4 lines

FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.

As it is a tree, why not store it as such?

There's no need to "store" the tree ordering specially, since all the pointers already exist. The question is: In what order should the tree be traversed? About the only explicit constraint we have now is that children must be suspended before their parents, but there undoubtedly are plenty of undocumented implicit constraints (maybe some of them aren't known to anybody at all).

Given the vast number of possible orders, and given that the only order we _know_ works correctly is reverse order of registration, I don't see any big reason to change. Speeding things up by parallel suspension would be a valid reason, but it needs to be done with a great deal of care.

Alan Stern

#15 lore raw
Quoted 8 lines · Alan Stern
Quoted 6 lines · Oliver Neukum

Am Freitag, 7. Dezember 2007 19:01:12 schrieb David Brownell:

Quoted 4 lines

FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.

Quoted 8 lines
Quoted 1 line

As it is a tree, why not store it as such?

There's no need to "store" the tree ordering specially, since all the pointers already exist. The question is: In what order should the tree be traversed? About the only explicit constraint we have now is that children must be suspended before their parents, but there undoubtedly are plenty of undocumented implicit constraints (maybe some of them aren't known to anybody at all).

Yes, that makes me nervous every time someone suggests to change the ordering of suspending devices.

Quoted 5 lines

Given the vast number of possible orders, and given that the only order we _know_ works correctly is reverse order of registration, I don't see any big reason to change. Speeding things up by parallel suspension would be a valid reason, but it needs to be done with a great deal of care.

Agreed.

Greetings,
Rafael

#16 lore raw
In reply to Alan Stern

Am Samstag, 8. Dezember 2007 16:43:25 schrieb Alan Stern:

Quoted 8 lines · Oliver Neukum
Quoted 7 lines

Am Freitag, 7. Dezember 2007 19:01:12 schrieb David Brownell:

Quoted 4 lines

FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.

Quoted 4 lines
Quoted 1 line

As it is a tree, why not store it as such?

There's no need to "store" the tree ordering specially, since all the pointers already exist. The question is: In what order should the tree

Nevertheless, we currently have a list. Why? To reverse the temporal order only?

Quoted 4 lines

be traversed? About the only explicit constraint we have now is that children must be suspended before their parents, but there undoubtedly are plenty of undocumented implicit constraints (maybe some of them aren't known to anybody at all).

We will need to know them for runtime pm.

Quoted 2 lines

Given the vast number of possible orders, and given that the only order we _know_ works correctly is reverse order of registration, I don't see

modus advocati diaboli:

Suppose I have a system with a FibreChannel disk. Now I hot plug another FibreChannel controller and connect it to the disk. Then I disconnect the disk from the original controller. What will happen if I suspend the system?

Quoted 3 lines

any big reason to change. Speeding things up by parallel suspension would be a valid reason, but it needs to be done with a great deal of care.

catch-22.

Regards
Oliver

#17 lore raw
Quoted 9 lines · Oliver Neukum

Am Samstag, 8. Dezember 2007 16:43:25 schrieb Alan Stern:

Quoted 8 lines · Oliver Neukum
Quoted 7 lines

Am Freitag, 7. Dezember 2007 19:01:12 schrieb David Brownell:

Quoted 4 lines

FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

As it is a tree, why not store it as such?

There's no need to "store" the tree ordering specially, since all the pointers already exist. The question is: In what order should the tree

Nevertheless, we currently have a list. Why? To reverse the temporal order only?

As far as I know, yes. Maybe also to simplify the traversal; it takes less code to follow a list than to go through a tree.

Quoted 6 lines
Quoted 4 lines

be traversed? About the only explicit constraint we have now is that children must be suspended before their parents, but there undoubtedly are plenty of undocumented implicit constraints (maybe some of them aren't known to anybody at all).

We will need to know them for runtime pm.

Maybe. But that knowledge will be localized in drivers and not consolidated in the PM core, since that's where runtime PM is handled. What we don't have is any way for drivers to export the knowledge.

Quoted 8 lines
Quoted 2 lines

Given the vast number of possible orders, and given that the only order we _know_ works correctly is reverse order of registration, I don't see

modus advocati diaboli:

Suppose I have a system with a FibreChannel disk. Now I hot plug another FibreChannel controller and connect it to the disk. Then I disconnect the disk from the original controller. What will happen if I suspend the system?

That seems a little strange. What happens during the intermediate stage when you have a single disk connected to the computer by two I/O pathways? Will the OS think it is really two disks? Will it get confused by seeing on-disk data structures in unstable incomplete states and subject to apparently random updates?

A better question is what happens when a Bluetooth communications driver creates a new tty device and then uses device_move() to put the Bluetooth device under the newer tty. Will the tty get suspended before its Bluetooth child?

Alan Stern

#18 lore raw
In reply to David Brownell
Quoted 4 lines · David Brownell

FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.

Actually I'm surprised that it works with USB devices.

As they are registered, you patch adds new devices to the list immediately after their parents. This means that the children of a particular device will be listed in reverse order of registration, right?

So if you have a USB hub, the hub's children will be added following the hub, and the hub's interface will come after all the children since it gets registered before them. Hence during suspend, the interface's suspend method will be called before any of the children are suspended. But hub_suspend() will fail if there are unsuspended children.

This is an example of another implicit order dependency: A hub's interface must be suspended after all the other children of that hub. It's an artifact of the way USB devices are represented in the device hierarchy; strictly speaking they should be children of the hub's interface, not of the hub itself.

Alan Stern

#19 lore raw
Quoted 14 lines · Alan Stern
Quoted 4 lines · David Brownell

FWIW the appended patch removes that rude "order of registration" policy, so that the suspend/resume list matches the device tree. It's behaved OK on PCs and, in light duty, a few development boards; I've carried it around most of this year.

Actually I'm surprised that it works with USB devices.

As they are registered, you patch adds new devices to the list immediately after their parents. This means that the children of a particular device will be listed in reverse order of registration, right?

Quoted 5 lines

So if you have a USB hub, the hub's children will be added following the hub, and the hub's interface will come after all the children since it gets registered before them. Hence during suspend, the interface's suspend method will be called before any of the children are suspended. But hub_suspend() will fail if there are unsuspended children.

I take it back. Interfaces get suspended along with their device; hence it doesn't matter in what order the interface is listed with respect to the children.

Alan Stern