Skip to contentSkip to conversation

PM updates for 2.6.33

GIT PULL In review Rafael J. Wysocki 14 participants 458 messages Dec 5, 2009 – Dec 25, 2009 Page 1 of 3

Subsystems (7)
USB ACPI PCI Driver core +3 more subsystems
lore
Share & tools
lore
#0 lore raw

Hi Linus,

Please pull power management updates for 2.6.33 from:

git://git.kernel.org/pub/scm/linux/kernel/git/rafael/suspend-2.6.git for-linus

They include:

* Asynchronous suspend and resume infrastructure. For now, PCI, ACPI and serio devices are enabled to suspend and resume asynchronously.

* Fixes for the runtime PM framework.

* Hibernate cleanups from Nigel and Jiri Slaby.

* Freezer optimisation from Tejun.

 Documentation/power/runtime_pm.txt |   12 +-
 drivers/acpi/glue.c                |    3 +
 drivers/acpi/scan.c                |    1 +
 drivers/base/core.c                |    4 +
 drivers/base/power/Makefile        |    2 +-
 drivers/base/power/common.c        |  283 +++++++++++++++
 drivers/base/power/main.c          |  677 +++++++++++++++++++++++++++++++++---
 drivers/base/power/power.h         |   42 ++-
 drivers/base/power/runtime.c       |   27 +-
 drivers/base/power/sysfs.c         |   47 +++
 drivers/input/serio/serio.c        |    1 +
 drivers/pci/pci.c                  |    1 +
 drivers/pci/pcie/portdrv_core.c    |    1 +
 include/linux/device.h             |   11 +
 include/linux/pm.h                 |   21 +-
 include/linux/pm_link.h            |   30 ++
 include/linux/pm_runtime.h         |   12 +
 include/linux/resume-trace.h       |    7 +
 kernel/power/Kconfig               |   14 +
 kernel/power/Makefile              |    2 +-
 kernel/power/hibernate.c           |   26 ++
 kernel/power/main.c                |   32 ++-
 kernel/power/process.c             |   14 +-
 kernel/power/swap.c                |  107 ++++++-
 kernel/power/swsusp.c              |  188 ----------
 25 files changed, 1281 insertions(+), 284 deletions(-)

---------------

Alan Stern (2):
PM / Runtime: Export the PM runtime workqueue
PM / Runtime: Use deferred_resume flag in pm_request_resume

Jaswinder Singh Rajput (1):
PM: Fix kernel-doc notation

Jiri Slaby (1):
PM / Hibernate: Swap, use KERN_CONT

Nigel Cunningham (2):
PM / Hibernate: Move swap functions to kernel/power/swap.c. PM / Hibernate: Shift remaining code from swsusp.c to hibernate.c

Rafael J. Wysocki (15):
PM: Introduce PM links framework
PM: Asynchronous resume of devices
PM: Asynchronous suspend of devices
PM: Allow PCI devices to suspend/resume asynchronously
PM: Allow ACPI devices to suspend/resume asynchronously
PM: Add a switch for disabling/enabling asynchronous suspend/resume
PM: Measure device suspend and resume times
PM: Add facility for advanced testing of async suspend/resume
PM: Measure suspend and resume times for individual devices
PM: Allow serio input devices to suspend/resume asynchronously PM / Runtime: Fix lockdep warning in __pm_runtime_set_status() PM / Runtime: Ensure timer_expires is nonzero in pm_schedule_suspend() PM / Runtime: Make documentation of runtime_idle() agree with the code PM / Runtime: Remove unnecessary braces in __pm_runtime_set_status()
PM: Add flag for devices capable of generating run-time wake-up events

Stephen Rothwell (1):
PM / Suspend: Using TASK_ macros requires sched.h

Tejun Heo (1):
PM / freezer: Don't get over-anxious while waiting

#1 lore raw
Quoted 3 lines · Rafael J. Wysocki

* Asynchronous suspend and resume infrastructure. For now, PCI, ACPI and serio devices are enabled to suspend and resume asynchronously.

I really think this is totally and utterly broken. Both from an implementation standpoint _and_ from a pure conceptual one.

Why isn't the suspend/resume async stuff just done like the init async stuff?

We don't need that crazy per-device flag for initialization, neither do we need drivers "enabling" any async code at all. They just do some things asynchronously, and then at the end of init time we wait for all those async events.

So why does suspend/resume need to do crazy sh*t instead?

It all looks terminally broken: you force async suspend for all PCI drivers, even when it makes no sense. Rather than let the drivers that already know how to do things like disk spinup asynchronously just do it that way.

The "timing" routines are also just crazy. What is the excuse for dpm_show_time() taking both start and stop times, since there is never any valid situation when it shouldn't have that do_gettimgofday(&stop) just before it? IOW - the whole end-time thing should be _inside_ dpm_show_time, rather than being done by the caller. No?

In other words - I'm not pulling this crazy thing. You'd better explain why it was done that way, when we already have done the same things better before in different ways.

Linus

#2 lore raw
Quoted 4 lines · Linus Torvalds

In other words - I'm not pulling this crazy thing. You'd better explain why it was done that way, when we already have done the same things better before in different ways.

I get the feeling that all the crazy infrastructure was due to worrying about the suspend/resume topology.

But the reason we don't worry about that during init is that it doesn't really tend to matter. Most slow operations are the things that aren't topology-aware, ie things like spinning up/down disks etc, that really could be done as a separate phase instead.

For example, is there really any reason why resume doesn't look exactly like the init sequence? Drivers that do slow things can start async work to do them, and then at the end of the resume sequence we just do a "wait for all the async work", exactly like we do for the current init sequences.

And yes, for the suspend sequence we obviously need to do any async work (and wait for it) before we actually shut down the controllers, but that would be _way_ more natural to do by just introducing a "pre-suspend" hook that walks the device tree and does any async stuff. And then just wait for the async stuff to finish before doing the suspend, and perhaps again before doing late_suspend (maybe somebody wants to do async stuff at the second stage too).

Then, because we need a way to undo things if things go wrong in the middle (and because it's also nice to be symmetric), we'd probably want to introduce that kind of "post_resume()" callback that allows you have a separate async wakeup thing for resume time too.

What are actually the expensive/slow things during suspend/resume? Am I wrong when I say it's things like disk spinup/spindown (and USB discovery, which needs USB-level support anyway, since it can involve devices that we didn't even know about before discovery started).

Linus

#3 lore raw
In reply to Linus Torvalds
Quoted 4 lines · Linus Torvalds

In other words - I'm not pulling this crazy thing. You'd better explain why it was done that way, when we already have done the same things better before in different ways.

I get the feeling that all the crazy infrastructure was due to worrying about the suspend/resume topology.

But the reason we don't worry about that during init is that it doesn't really tend to matter. Most slow operations are the things that aren't topology-aware, ie things like spinning up/down disks etc, that really could be done as a separate phase instead.

For example, is there really any reason why resume doesn't look exactly like the init sequence? Drivers that do slow things can start async work to do them, and then at the end of the resume sequence we just do a "wait for all the async work", exactly like we do for the current init sequences.

And yes, for the suspend sequence we obviously need to do any async work (and wait for it) before we actually shut down the controllers, but that would be _way_ more natural to do by just introducing a "pre-suspend" hook that walks the device tree and does any async stuff. And then just wait for the async stuff to finish before doing the suspend, and perhaps again before doing late_suspend (maybe somebody wants to do async stuff at the second stage too).

Then, because we need a way to undo things if things go wrong in the middle (and because it's also nice to be symmetric), we'd probably want to introduce that kind of "post_resume()" callback that allows you have a separate async wakeup thing for resume time too.

What are actually the expensive/slow things during suspend/resume? Am I wrong when I say it's things like disk spinup/spindown (and USB discovery, which needs USB-level support anyway, since it can involve devices that we didn't even know about before discovery started).

Linus

#4 lore raw
Quoted 6 lines · Linus Torvalds
Quoted 4 lines · Linus Torvalds

In other words - I'm not pulling this crazy thing. You'd better explain why it was done that way, when we already have done the same things better before in different ways.

OK, I'll send another pull request without these patches if the rest of the changes if fine with you (they are more important than the async stuff to me).

Quoted 2 lines

I get the feeling that all the crazy infrastructure was due to worrying about the suspend/resume topology.

Yes, that's the main reason.

Quoted 4 lines

But the reason we don't worry about that during init is that it doesn't really tend to matter. Most slow operations are the things that aren't topology-aware, ie things like spinning up/down disks etc, that really could be done as a separate phase instead.

It was based on the observation that in many cases the current drivers' suspend and resume callbacks can be run in parallel with the other drivers' callbacks without any changes to the drivers (and without introducing another phase of suspend for that matter), because there are no dependencies between them.

The approach you're suggesting would require modifying individual drivers which I just wanted to avoid. If you don't like that, we'll have to take the longer route, although I'm afraid that will take lots of time and we won't be able to exploit the entire possible parallelism this way.

Quoted 5 lines

For example, is there really any reason why resume doesn't look exactly like the init sequence? Drivers that do slow things can start async work to do them, and then at the end of the resume sequence we just do a "wait for all the async work", exactly like we do for the current init sequences.

During suspend we actually know what the dependences between the devicces are and we can use that information to do more things in parallel. For instance, in the majority of cases (I'm yet to find a counter example), the entire suspend callbacks of "leaf" PCI devices may be run in parallel with each other.

So, the point is not to look for "async stuff" in a driver's suspend/resume callbacks, but to execute the whole suspend/resume callbacks in parallel, if possible.

Quoted 8 lines

And yes, for the suspend sequence we obviously need to do any async work (and wait for it) before we actually shut down the controllers, but that would be _way_ more natural to do by just introducing a "pre-suspend" hook that walks the device tree and does any async stuff. And then just wait for the async stuff to finish before doing the suspend, and perhaps again before doing late_suspend (maybe somebody wants to do async stuff at the second stage too).

Quoted 4 lines

Then, because we need a way to undo things if things go wrong in the middle (and because it's also nice to be symmetric), we'd probably want to introduce that kind of "post_resume()" callback that allows you have a separate async wakeup thing for resume time too.

Yes, we can do that, but I'm afraid that the majority of drivers won't use the new hooks (people generally seem to be to reluctant to modify their suspend/resume callbacks not to break things).

Also, for an individual driver it really is difficult to separate the "async stuff" from the stuff which is not async, because everything that can be done in parallel with the other drivers' suspend callbacks is potentially async, as long as there are no dependences between the devices in question (like parent-child dependences, or PCI-shadow ACPI dependences). And it's generally worth doing that if a driver's suspend or resume callback calls msleep() for whatever the reason.

Quoted 4 lines

What are actually the expensive/slow things during suspend/resume? Am I wrong when I say it's things like disk spinup/spindown (and USB discovery, which needs USB-level support anyway, since it can involve devices that we didn't even know about before discovery started).

Disk spinup/spindown takes time, but also some ACPI devices resume slowly, serio devices do that too and there are surprisingly many drivers that wait (using msleep() during suspend and resume). Apart from this, every PCI device going from D0 to D3 during suspend and from D3 to D0 during resume requires us to sleep for 10 ms (the sleeping is done by the PCI core, so the drivers don't even realize its there).

Thanks,
Rafael

#5 lore raw
Quoted 8 lines · On Sun, 6 Dec 2009 00:55:36 +0100 "Rafael J. Wysocki" <[email protected]> wrote:

Disk spinup/spindown takes time, but also some ACPI devices resume slowly, serio devices do that too and there are surprisingly many drivers that wait (using msleep() during suspend and resume). Apart from this, every PCI device going from D0 to D3 during suspend and from D3 to D0 during resume requires us to sleep for 10 ms (the sleeping is done by the PCI core, so the drivers don't even realize its there).

maybe a good step is to make a scripts/bootgraph.pl equivalent for suspend/resume (or make a debug mode that outputs in a compatible format so that the script can be used as is.. I don't mind either way, and consider this my offer to help with such a script as long as there's sufficient logging in dmesg ;-)

that way we can SEE which ones are an issue.... and by how much.

Signature · 3 lines
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#6 lore raw
Quoted 12 lines · Arjan van de Ven
Quoted 8 lines · On Sun, 6 Dec 2009 00:55:36 +0100 "Rafael J. Wysocki" <[email protected]> wrote:

Disk spinup/spindown takes time, but also some ACPI devices resume slowly, serio devices do that too and there are surprisingly many drivers that wait (using msleep() during suspend and resume). Apart from this, every PCI device going from D0 to D3 during suspend and from D3 to D0 during resume requires us to sleep for 10 ms (the sleeping is done by the PCI core, so the drivers don't even realize its there).

Quoted 5 lines

maybe a good step is to make a scripts/bootgraph.pl equivalent for suspend/resume (or make a debug mode that outputs in a compatible format so that the script can be used as is.. I don't mind either way, and consider this my offer to help with such a script as long as there's sufficient logging in dmesg ;-)

OK, so what kind of logging is needed?

Quoted 1 line

that way we can SEE which ones are an issue.... and by how much.

Well, why not.

Thanks,
Rafael

#7 lore raw
Quoted 13 lines · On Sun, 6 Dec 2009 02:26:06 +0100 "Rafael J. Wysocki" <[email protected]> wrote:
Quoted 12 lines · Arjan van de Ven
Quoted 8 lines · On Sun, 6 Dec 2009 00:55:36 +0100 "Rafael J. Wysocki" <[email protected]> wrote:

Disk spinup/spindown takes time, but also some ACPI devices resume slowly, serio devices do that too and there are surprisingly many drivers that wait (using msleep() during suspend and resume). Apart from this, every PCI device going from D0 to D3 during suspend and from D3 to D0 during resume requires us to sleep for 10 ms (the sleeping is done by the PCI core, so the drivers don't even realize its there).

Quoted 7 lines
Quoted 5 lines

maybe a good step is to make a scripts/bootgraph.pl equivalent for suspend/resume (or make a debug mode that outputs in a compatible format so that the script can be used as is.. I don't mind either way, and consider this my offer to help with such a script as long as there's sufficient logging in dmesg ;-)

OK, so what kind of logging is needed?

basically the equivalent of the two initcall_debug paths in

init/main.c:do_one_initcall()

which prints a start time and an end time (and a pid) for each init function; if we have the same for suspend calls (and resume)... we can make the tool graph it.
Would be nice to get markers for start and end of the whole suspend sequence as well as for the resume sequence; those make it easier to know when the end is (so that the axis can be drawn etc)

shouldn't be too hard to implement...

Signature · 3 lines
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#8 lore raw
Quoted 32 lines · Arjan van de Ven
Quoted 13 lines · On Sun, 6 Dec 2009 02:26:06 +0100 "Rafael J. Wysocki" <[email protected]> wrote:
Quoted 12 lines · Arjan van de Ven
Quoted 8 lines · On Sun, 6 Dec 2009 00:55:36 +0100 "Rafael J. Wysocki" <[email protected]> wrote:

Disk spinup/spindown takes time, but also some ACPI devices resume slowly, serio devices do that too and there are surprisingly many drivers that wait (using msleep() during suspend and resume). Apart from this, every PCI device going from D0 to D3 during suspend and from D3 to D0 during resume requires us to sleep for 10 ms (the sleeping is done by the PCI core, so the drivers don't even realize its there).

Quoted 7 lines
Quoted 5 lines

maybe a good step is to make a scripts/bootgraph.pl equivalent for suspend/resume (or make a debug mode that outputs in a compatible format so that the script can be used as is.. I don't mind either way, and consider this my offer to help with such a script as long as there's sufficient logging in dmesg ;-)

OK, so what kind of logging is needed?

basically the equivalent of the two initcall_debug paths in

init/main.c:do_one_initcall()

which prints a start time and an end time (and a pid) for each init function; if we have the same for suspend calls (and resume)... we can make the tool graph it.

Quoted 5 lines

Would be nice to get markers for start and end of the whole suspend sequence as well as for the resume sequence; those make it easier to know when the end is (so that the axis can be drawn etc)

shouldn't be too hard to implement...

I think an even better option would be to extend 'perf timechart' to be suspend/resume aware: add a few tracepoint events and teach 'perf timechart' to draw them. (We should be able to do perf timechart record across suspend/resume cycles just fine.)

( Doing that would also improve the tracing facilities within
suspend/resume quite significantly. It wouldnt just be a single-purpose thing for graphing, but perf trace and perf stat would work equally well. )

Thanks,

Ingo

#9 lore raw
In reply to Arjan van de Ven
Quoted 32 lines · Arjan van de Ven
Quoted 13 lines · On Sun, 6 Dec 2009 02:26:06 +0100 "Rafael J. Wysocki" <[email protected]> wrote:
Quoted 12 lines · Arjan van de Ven
Quoted 8 lines · On Sun, 6 Dec 2009 00:55:36 +0100 "Rafael J. Wysocki" <[email protected]> wrote:

Disk spinup/spindown takes time, but also some ACPI devices resume slowly, serio devices do that too and there are surprisingly many drivers that wait (using msleep() during suspend and resume). Apart from this, every PCI device going from D0 to D3 during suspend and from D3 to D0 during resume requires us to sleep for 10 ms (the sleeping is done by the PCI core, so the drivers don't even realize its there).

Quoted 7 lines
Quoted 5 lines

maybe a good step is to make a scripts/bootgraph.pl equivalent for suspend/resume (or make a debug mode that outputs in a compatible format so that the script can be used as is.. I don't mind either way, and consider this my offer to help with such a script as long as there's sufficient logging in dmesg ;-)

OK, so what kind of logging is needed?

basically the equivalent of the two initcall_debug paths in

init/main.c:do_one_initcall()

which prints a start time and an end time (and a pid) for each init function; if we have the same for suspend calls (and resume)... we can make the tool graph it.

Quoted 5 lines

Would be nice to get markers for start and end of the whole suspend sequence as well as for the resume sequence; those make it easier to know when the end is (so that the axis can be drawn etc)

shouldn't be too hard to implement...

I think an even better option would be to extend 'perf timechart' to be suspend/resume aware: add a few tracepoint events and teach 'perf timechart' to draw them. (We should be able to do perf timechart record across suspend/resume cycles just fine.)

( Doing that would also improve the tracing facilities within
suspend/resume quite significantly. It wouldnt just be a single-purpose thing for graphing, but perf trace and perf stat would work equally well. )

Thanks,

Ingo

#10 lore raw
In reply to Rafael J. Wysocki
Quoted 13 lines · On Sun, 6 Dec 2009 02:26:06 +0100 "Rafael J. Wysocki" <[email protected]> wrote:
Quoted 12 lines · Arjan van de Ven
Quoted 8 lines · On Sun, 6 Dec 2009 00:55:36 +0100 "Rafael J. Wysocki" <[email protected]> wrote:

Disk spinup/spindown takes time, but also some ACPI devices resume slowly, serio devices do that too and there are surprisingly many drivers that wait (using msleep() during suspend and resume). Apart from this, every PCI device going from D0 to D3 during suspend and from D3 to D0 during resume requires us to sleep for 10 ms (the sleeping is done by the PCI core, so the drivers don't even realize its there).

Quoted 7 lines
Quoted 5 lines

maybe a good step is to make a scripts/bootgraph.pl equivalent for suspend/resume (or make a debug mode that outputs in a compatible format so that the script can be used as is.. I don't mind either way, and consider this my offer to help with such a script as long as there's sufficient logging in dmesg ;-)

OK, so what kind of logging is needed?

basically the equivalent of the two initcall_debug paths in

init/main.c:do_one_initcall()

which prints a start time and an end time (and a pid) for each init function; if we have the same for suspend calls (and resume)... we can make the tool graph it.
Would be nice to get markers for start and end of the whole suspend sequence as well as for the resume sequence; those make it easier to know when the end is (so that the axis can be drawn etc)

shouldn't be too hard to implement...

Signature · 3 lines
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#11 lore raw
In reply to Arjan van de Ven
Quoted 12 lines · Arjan van de Ven
Quoted 8 lines · On Sun, 6 Dec 2009 00:55:36 +0100 "Rafael J. Wysocki" <[email protected]> wrote:

Disk spinup/spindown takes time, but also some ACPI devices resume slowly, serio devices do that too and there are surprisingly many drivers that wait (using msleep() during suspend and resume). Apart from this, every PCI device going from D0 to D3 during suspend and from D3 to D0 during resume requires us to sleep for 10 ms (the sleeping is done by the PCI core, so the drivers don't even realize its there).

Quoted 5 lines

maybe a good step is to make a scripts/bootgraph.pl equivalent for suspend/resume (or make a debug mode that outputs in a compatible format so that the script can be used as is.. I don't mind either way, and consider this my offer to help with such a script as long as there's sufficient logging in dmesg ;-)

OK, so what kind of logging is needed?

Quoted 1 line

that way we can SEE which ones are an issue.... and by how much.

Well, why not.

Thanks,
Rafael

#12 lore raw
In reply to Rafael J. Wysocki
Quoted 8 lines · On Sun, 6 Dec 2009 00:55:36 +0100 "Rafael J. Wysocki" <[email protected]> wrote:

Disk spinup/spindown takes time, but also some ACPI devices resume slowly, serio devices do that too and there are surprisingly many drivers that wait (using msleep() during suspend and resume). Apart from this, every PCI device going from D0 to D3 during suspend and from D3 to D0 during resume requires us to sleep for 10 ms (the sleeping is done by the PCI core, so the drivers don't even realize its there).

maybe a good step is to make a scripts/bootgraph.pl equivalent for suspend/resume (or make a debug mode that outputs in a compatible format so that the script can be used as is.. I don't mind either way, and consider this my offer to help with such a script as long as there's sufficient logging in dmesg ;-)

that way we can SEE which ones are an issue.... and by how much.

Signature · 3 lines
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#13 lore raw
In reply to Rafael J. Wysocki
Quoted 3 lines · Rafael J. Wysocki

The approach you're suggesting would require modifying individual drivers which I just wanted to avoid.

In the init path, we had the reverse worry - not wanting to make everything (where "everything" can be some subsystem like just the set of PCI drivers, of course - not really "everything" in an absolute sense) async, and then having to try to work out with the random driver that couldn't handle it.

And there were _lots_ of drivers that couldn't handle it, because they knew they got woken up serially. The ATA layer needed to know about asynchronous things, because sometimes those independent devices aren't so independent at all. Which is why I don't think your approach is safe.

Just to take an example of the whole "independent devices are not necessarily independent" thing - things like multi-port PCMCIA controllers generally show up as multiple PCI devices. But they are _not_ independent, and they actually share some registers. Resuming them asynchronously might well be ok, but maybe it's not. Who knows?

In contrast, a device driver can generally know that certain _parts_ of the initialization is safe. As an example of that, I think the libata layer does all the port enumeration synchronously, but then once the ports have been identified, it does the rest async.

That's the kind of decision we can sanely make when we do the async part as a "drivers may choose to do certain parts asynchronously". Doing it at a higher level sounds like a problem to me.

Quoted 3 lines

If you don't like that, we'll have to take the longer route, although I'm afraid that will take lots of time and we won't be able to exploit the entire possible parallelism this way.

Sure. But I'd rather do the safe thing. Especially since there are likely just a few cases that really take a long time.

Quoted 5 lines

During suspend we actually know what the dependences between the devicces are and we can use that information to do more things in parallel. For instance, in the majority of cases (I'm yet to find a counter example), the entire suspend callbacks of "leaf" PCI devices may be run in parallel with each other.

See above. That's simply not at all guaranteed to be true.

And when it isn't true (ie different PCI leaf devices end up having subtle dependencies), now you need to start doing hacky things.

I'd much rather have the individual drivers say "I can do this part in parallel", and not force it on them. Because it is definitely _not_ guaranteed that PCI devices can do parallel resume and suspend.

Quoted 3 lines

Yes, we can do that, but I'm afraid that the majority of drivers won't use the new hooks (people generally seem to be to reluctant to modify their suspend/resume callbacks not to break things).

See above - I don't think this is a "majority" issue. I think it's a "let's figure out the problem spots, and fix _those_". IOW, get 2% of the coverage, and get 95% of the advantage.

Quoted 1 line

Disk spinup/spindown takes time, but also some ACPI devices resume slowly,

We actually saw that when we did async init. And it was horrible. There's nothing that says that the ACPI stuff necessarily even _can_ run in parallel.

I think we currently only do the ACPI battery ops asynchronously.

Linus

#14 lore raw
In reply to Rafael J. Wysocki
Quoted 3 lines · Rafael J. Wysocki

The approach you're suggesting would require modifying individual drivers which I just wanted to avoid.

In the init path, we had the reverse worry - not wanting to make everything (where "everything" can be some subsystem like just the set of PCI drivers, of course - not really "everything" in an absolute sense) async, and then having to try to work out with the random driver that couldn't handle it.

And there were _lots_ of drivers that couldn't handle it, because they knew they got woken up serially. The ATA layer needed to know about asynchronous things, because sometimes those independent devices aren't so independent at all. Which is why I don't think your approach is safe.

Just to take an example of the whole "independent devices are not necessarily independent" thing - things like multi-port PCMCIA controllers generally show up as multiple PCI devices. But they are _not_ independent, and they actually share some registers. Resuming them asynchronously might well be ok, but maybe it's not. Who knows?

In contrast, a device driver can generally know that certain _parts_ of the initialization is safe. As an example of that, I think the libata layer does all the port enumeration synchronously, but then once the ports have been identified, it does the rest async.

That's the kind of decision we can sanely make when we do the async part as a "drivers may choose to do certain parts asynchronously". Doing it at a higher level sounds like a problem to me.

Quoted 3 lines

If you don't like that, we'll have to take the longer route, although I'm afraid that will take lots of time and we won't be able to exploit the entire possible parallelism this way.

Sure. But I'd rather do the safe thing. Especially since there are likely just a few cases that really take a long time.

Quoted 5 lines

During suspend we actually know what the dependences between the devicces are and we can use that information to do more things in parallel. For instance, in the majority of cases (I'm yet to find a counter example), the entire suspend callbacks of "leaf" PCI devices may be run in parallel with each other.

See above. That's simply not at all guaranteed to be true.

And when it isn't true (ie different PCI leaf devices end up having subtle dependencies), now you need to start doing hacky things.

I'd much rather have the individual drivers say "I can do this part in parallel", and not force it on them. Because it is definitely _not_ guaranteed that PCI devices can do parallel resume and suspend.

Quoted 3 lines

Yes, we can do that, but I'm afraid that the majority of drivers won't use the new hooks (people generally seem to be to reluctant to modify their suspend/resume callbacks not to break things).

See above - I don't think this is a "majority" issue. I think it's a "let's figure out the problem spots, and fix _those_". IOW, get 2% of the coverage, and get 95% of the advantage.

Quoted 1 line

Disk spinup/spindown takes time, but also some ACPI devices resume slowly,

We actually saw that when we did async init. And it was horrible. There's nothing that says that the ACPI stuff necessarily even _can_ run in parallel.

I think we currently only do the ACPI battery ops asynchronously.

Linus

#15 lore raw
Quoted 12 lines · Linus Torvalds
Quoted 3 lines · Rafael J. Wysocki

The approach you're suggesting would require modifying individual drivers which I just wanted to avoid.

In the init path, we had the reverse worry - not wanting to make everything (where "everything" can be some subsystem like just the set of PCI drivers, of course - not really "everything" in an absolute sense) async, and then having to try to work out with the random driver that couldn't handle it.

Quoted 4 lines

And there were _lots_ of drivers that couldn't handle it, because they knew they got woken up serially. The ATA layer needed to know about asynchronous things, because sometimes those independent devices aren't so independent at all. Which is why I don't think your approach is safe.

While the current settings are probably unsafe (like enabling PCI devices to be suspended asynchronously by default if there are not any direct dependences between them), there are provisions to make eveything safe, if we have enough information (which also is needed to put the required logic into the drivers). The device tree represents a good deal of the dependences between devices and the other dependences may be represented as PM links enforcing specific ordering of the PM callbacks.

Quoted 5 lines

Just to take an example of the whole "independent devices are not necessarily independent" thing - things like multi-port PCMCIA controllers generally show up as multiple PCI devices. But they are _not_ independent, and they actually share some registers. Resuming them asynchronously might well be ok, but maybe it's not. Who knows?

I'd say if there's a worry that the same register may be accessed concurrently from two different code paths, there should be some locking in place.

Quoted 8 lines

In contrast, a device driver can generally know that certain _parts_ of the initialization is safe. As an example of that, I think the libata layer does all the port enumeration synchronously, but then once the ports have been identified, it does the rest async.

That's the kind of decision we can sanely make when we do the async part as a "drivers may choose to do certain parts asynchronously". Doing it at a higher level sounds like a problem to me.

The difference between suspend and initialization is that during suspend we have already enumerated all devices and we should know how they depend on each other (and we really should know that if we are to actually understand how things work), so we can represent that information somehow and use it to do things at the higher level.

How to represent it is a different matter, but in principle it should be possible.

Quoted 6 lines
Quoted 3 lines

If you don't like that, we'll have to take the longer route, although I'm afraid that will take lots of time and we won't be able to exploit the entire possible parallelism this way.

Sure. But I'd rather do the safe thing. Especially since there are likely just a few cases that really take a long time.

And there are lots of small sleeps here and there that accumulate and are entirely avoidable.

Quoted 6 lines
Quoted 5 lines

During suspend we actually know what the dependences between the devicces are and we can use that information to do more things in parallel. For instance, in the majority of cases (I'm yet to find a counter example), the entire suspend callbacks of "leaf" PCI devices may be run in parallel with each other.

Quoted 8 lines

See above. That's simply not at all guaranteed to be true.

And when it isn't true (ie different PCI leaf devices end up having subtle dependencies), now you need to start doing hacky things.

I'd much rather have the individual drivers say "I can do this part in parallel", and not force it on them. Because it is definitely _not_ guaranteed that PCI devices can do parallel resume and suspend.

OK, it's not guaranteed, but why not to do this on systems where it's known to work?

Quoted 7 lines
Quoted 3 lines

Yes, we can do that, but I'm afraid that the majority of drivers won't use the new hooks (people generally seem to be to reluctant to modify their suspend/resume callbacks not to break things).

See above - I don't think this is a "majority" issue. I think it's a "let's figure out the problem spots, and fix _those_". IOW, get 2% of the coverage, and get 95% of the advantage.

I wouldn't really like to add even more suspend/resume callbacks for this purpose, because we already have so many of them. And even if we do that, I don't really expect drivers to start using them any time soon.

Quoted 7 lines
Quoted 1 line

Disk spinup/spindown takes time, but also some ACPI devices resume slowly,

We actually saw that when we did async init. And it was horrible. There's nothing that says that the ACPI stuff necessarily even _can_ run in parallel.

I think we currently only do the ACPI battery ops asynchronously.

There are only a few ACPI devices that have real suspend/resume callbacks and I haven't see problems with these in practice.

Thanks,
Rafael

#16 lore raw
Quoted 57 lines · Rafael J. Wysocki
Quoted 12 lines · Linus Torvalds
Quoted 3 lines · Rafael J. Wysocki

The approach you're suggesting would require modifying individual drivers which I just wanted to avoid.

In the init path, we had the reverse worry - not wanting to make everything (where "everything" can be some subsystem like just the set of PCI drivers, of course - not really "everything" in an absolute sense) async, and then having to try to work out with the random driver that couldn't handle it.

Quoted 4 lines

And there were _lots_ of drivers that couldn't handle it, because they knew they got woken up serially. The ATA layer needed to know about asynchronous things, because sometimes those independent devices aren't so independent at all. Which is why I don't think your approach is safe.

While the current settings are probably unsafe (like enabling PCI devices to be suspended asynchronously by default if there are not any direct dependences between them), there are provisions to make eveything safe, if we have enough information (which also is needed to put the required logic into the drivers). The device tree represents a good deal of the dependences between devices and the other dependences may be represented as PM links enforcing specific ordering of the PM callbacks.

Quoted 5 lines

Just to take an example of the whole "independent devices are not necessarily independent" thing - things like multi-port PCMCIA controllers generally show up as multiple PCI devices. But they are _not_ independent, and they actually share some registers. Resuming them asynchronously might well be ok, but maybe it's not. Who knows?

I'd say if there's a worry that the same register may be accessed concurrently from two different code paths, there should be some locking in place.

Quoted 8 lines

In contrast, a device driver can generally know that certain _parts_ of the initialization is safe. As an example of that, I think the libata layer does all the port enumeration synchronously, but then once the ports have been identified, it does the rest async.

That's the kind of decision we can sanely make when we do the async part as a "drivers may choose to do certain parts asynchronously". Doing it at a higher level sounds like a problem to me.

The difference between suspend and initialization is that during suspend we have already enumerated all devices and we should know how they depend on each other (and we really should know that if we are to actually understand how things work), so we can represent that information somehow and use it to do things at the higher level.

How to represent it is a different matter, but in principle it should be possible.

Quoted 4 lines
Quoted 3 lines

If you don't like that, we'll have to take the longer route, although I'm afraid that will take lots of time and we won't be able to exploit the entire possible parallelism this way.

Quoted 5 lines
Quoted 2 lines

Sure. But I'd rather do the safe thing. Especially since there are likely just a few cases that really take a long time.

And there are lots of small sleeps here and there that accumulate and are entirely avoidable.

I mean, it is avoidable to do all these sleeps sequentially.

Thanks,
Rafael

#17 lore raw
In reply to Rafael J. Wysocki
Quoted 57 lines · Rafael J. Wysocki
Quoted 12 lines · Linus Torvalds
Quoted 3 lines · Rafael J. Wysocki

The approach you're suggesting would require modifying individual drivers which I just wanted to avoid.

In the init path, we had the reverse worry - not wanting to make everything (where "everything" can be some subsystem like just the set of PCI drivers, of course - not really "everything" in an absolute sense) async, and then having to try to work out with the random driver that couldn't handle it.

Quoted 4 lines

And there were _lots_ of drivers that couldn't handle it, because they knew they got woken up serially. The ATA layer needed to know about asynchronous things, because sometimes those independent devices aren't so independent at all. Which is why I don't think your approach is safe.

While the current settings are probably unsafe (like enabling PCI devices to be suspended asynchronously by default if there are not any direct dependences between them), there are provisions to make eveything safe, if we have enough information (which also is needed to put the required logic into the drivers). The device tree represents a good deal of the dependences between devices and the other dependences may be represented as PM links enforcing specific ordering of the PM callbacks.

Quoted 5 lines

Just to take an example of the whole "independent devices are not necessarily independent" thing - things like multi-port PCMCIA controllers generally show up as multiple PCI devices. But they are _not_ independent, and they actually share some registers. Resuming them asynchronously might well be ok, but maybe it's not. Who knows?

I'd say if there's a worry that the same register may be accessed concurrently from two different code paths, there should be some locking in place.

Quoted 8 lines

In contrast, a device driver can generally know that certain _parts_ of the initialization is safe. As an example of that, I think the libata layer does all the port enumeration synchronously, but then once the ports have been identified, it does the rest async.

That's the kind of decision we can sanely make when we do the async part as a "drivers may choose to do certain parts asynchronously". Doing it at a higher level sounds like a problem to me.

The difference between suspend and initialization is that during suspend we have already enumerated all devices and we should know how they depend on each other (and we really should know that if we are to actually understand how things work), so we can represent that information somehow and use it to do things at the higher level.

How to represent it is a different matter, but in principle it should be possible.

Quoted 4 lines
Quoted 3 lines

If you don't like that, we'll have to take the longer route, although I'm afraid that will take lots of time and we won't be able to exploit the entire possible parallelism this way.

Quoted 5 lines
Quoted 2 lines

Sure. But I'd rather do the safe thing. Especially since there are likely just a few cases that really take a long time.

And there are lots of small sleeps here and there that accumulate and are entirely avoidable.

I mean, it is avoidable to do all these sleeps sequentially.

Thanks,
Rafael

#18 lore raw
In reply to Rafael J. Wysocki
Quoted 6 lines · Rafael J. Wysocki

While the current settings are probably unsafe (like enabling PCI devices to be suspended asynchronously by default if there are not any direct dependences between them), there are provisions to make eveything safe, if we have enough information (which also is needed to put the required logic into the drivers).

I disagree.

Think of a situation that we already handle pretty poorly: USB mass storage devices over a suspend/resume.

Quoted 3 lines

The device tree represents a good deal of the dependences between devices and the other dependences may be represented as PM links enforcing specific ordering of the PM callbacks.

The device tree means nothing at all, because it may need to be entirely rebuilt at resume time.

Optimally, what we _should_ be doing (and aren't) for suspend/resume of USB is to just tear down the whole topology and rebuild it and re-connect the things like mass storage devices. IOW, there would be no device tree to describe the topology, because we're finding it anew. And it's one of the things we _would_ want to do asynchronously with other things.

We don't want to build up some irrelevant PM links and callbacks. We don't want to have some completely made-up new infrastructure for something that we _already_ want to handle totally differently for init time.

IOW, I argue very strongly against making up something PM-specific, when there really doesn't seem to be much of an advantage. We're much better off trying to share the init code than making up something new.

Quoted 2 lines

I'd say if there's a worry that the same register may be accessed concurrently from two different code paths, there should be some locking in place.

Yeah. And I wish ACPI didn't exist at all. We don't know.

And we want to _limit_ our exposure to these things.

Linus

#19 lore raw
Quoted 14 lines · Linus Torvalds
Quoted 6 lines · Rafael J. Wysocki

While the current settings are probably unsafe (like enabling PCI devices to be suspended asynchronously by default if there are not any direct dependences between them), there are provisions to make eveything safe, if we have enough information (which also is needed to put the required logic into the drivers).

I disagree.

Think of a situation that we already handle pretty poorly: USB mass storage devices over a suspend/resume.

Quoted 6 lines
Quoted 3 lines

The device tree represents a good deal of the dependences between devices and the other dependences may be represented as PM links enforcing specific ordering of the PM callbacks.

The device tree means nothing at all, because it may need to be entirely rebuilt at resume time.

With that assumption we have no choice but to leave the async stuff to the drivers, which generally I'm fine with, although I really don't expect to see it done.

Quoted 5 lines

Optimally, what we _should_ be doing (and aren't) for suspend/resume of USB is to just tear down the whole topology and rebuild it and re-connect the things like mass storage devices. IOW, there would be no device tree to describe the topology, because we're finding it anew. And it's one of the things we _would_ want to do asynchronously with other things.

I think you should tell that to the USB people, because they don't seem to think this way.

[Side note, I do think that at least some information in the device tree will remain valid over suspend/resume, but this is a different matter.]

Quoted 8 lines

We don't want to build up some irrelevant PM links and callbacks. We don't want to have some completely made-up new infrastructure for something that we _already_ want to handle totally differently for init time.

IOW, I argue very strongly against making up something PM-specific, when there really doesn't seem to be much of an advantage. We're much better off trying to share the init code than making up something new.

Quoted 6 lines
Quoted 2 lines

I'd say if there's a worry that the same register may be accessed concurrently from two different code paths, there should be some locking in place.

Yeah. And I wish ACPI didn't exist at all. We don't know.

And we want to _limit_ our exposure to these things.

Don't worry, I'm not going to touch async suspend/resume again, unless somebody makes me do it.

BTW, you seem to have some quite strong opinions about power management that you only share with people when somebody sends you patches you don't like. I guess it will be much more productive if we know your thoughts about it in advance, so I hope you won't mind being sent CCs of core PM patches posted to linux-pm for discussions.

Thanks,
Rafael

#20 lore raw
In reply to Linus Torvalds
Quoted 14 lines · Linus Torvalds
Quoted 6 lines · Rafael J. Wysocki

While the current settings are probably unsafe (like enabling PCI devices to be suspended asynchronously by default if there are not any direct dependences between them), there are provisions to make eveything safe, if we have enough information (which also is needed to put the required logic into the drivers).

I disagree.

Think of a situation that we already handle pretty poorly: USB mass storage devices over a suspend/resume.

Quoted 6 lines
Quoted 3 lines

The device tree represents a good deal of the dependences between devices and the other dependences may be represented as PM links enforcing specific ordering of the PM callbacks.

The device tree means nothing at all, because it may need to be entirely rebuilt at resume time.

With that assumption we have no choice but to leave the async stuff to the drivers, which generally I'm fine with, although I really don't expect to see it done.

Quoted 5 lines

Optimally, what we _should_ be doing (and aren't) for suspend/resume of USB is to just tear down the whole topology and rebuild it and re-connect the things like mass storage devices. IOW, there would be no device tree to describe the topology, because we're finding it anew. And it's one of the things we _would_ want to do asynchronously with other things.

I think you should tell that to the USB people, because they don't seem to think this way.

[Side note, I do think that at least some information in the device tree will remain valid over suspend/resume, but this is a different matter.]

Quoted 8 lines

We don't want to build up some irrelevant PM links and callbacks. We don't want to have some completely made-up new infrastructure for something that we _already_ want to handle totally differently for init time.

IOW, I argue very strongly against making up something PM-specific, when there really doesn't seem to be much of an advantage. We're much better off trying to share the init code than making up something new.

Quoted 6 lines
Quoted 2 lines

I'd say if there's a worry that the same register may be accessed concurrently from two different code paths, there should be some locking in place.

Yeah. And I wish ACPI didn't exist at all. We don't know.

And we want to _limit_ our exposure to these things.

Don't worry, I'm not going to touch async suspend/resume again, unless somebody makes me do it.

BTW, you seem to have some quite strong opinions about power management that you only share with people when somebody sends you patches you don't like. I guess it will be much more productive if we know your thoughts about it in advance, so I hope you won't mind being sent CCs of core PM patches posted to linux-pm for discussions.

Thanks,
Rafael

#21 lore raw
In reply to Linus Torvalds
Quoted 9 lines · Linus Torvalds

Think of a situation that we already handle pretty poorly: USB mass storage devices over a suspend/resume.

Quoted 3 lines

The device tree represents a good deal of the dependences between devices and the other dependences may be represented as PM links enforcing specific ordering of the PM callbacks.

The device tree means nothing at all, because it may need to be entirely rebuilt at resume time.

Nonsense.

Quoted 5 lines

Optimally, what we _should_ be doing (and aren't) for suspend/resume of USB is to just tear down the whole topology and rebuild it and re-connect the things like mass storage devices. IOW, there would be no device tree to describe the topology, because we're finding it anew. And it's one of the things we _would_ want to do asynchronously with other things.

That's ridiculous. Having gone to all the trouble of building a device tree, one which is presumably still almost entirely correct, why go to all the trouble of tearing it down only to rebuild it again? (Note:
I'm talking about resume-from-RAM here, not resume-from-hibernation.)

Instead what we do is verify that the devices we remember from before the suspend are still there, and then asynchronously handle new devices which have been plugged in during the meantime. Doing this involves relatively little extra or new code; most of the routines are shared with the runtime PM and device reset paths.

As for asynchronicity... At init time, USB device discovery truly is asynchronous. It can happen long after you log in (especially if you don't plug in the device until after you log in!). But at resume time we are more highly constrained. User processes cannot be unfrozen until all the devices have been resumed; otherwise they would encounter errors when trying to do I/O to a suspended device. (With the runtime PM framework this is much less of a problem, but plenty of drivers don't support runtime PM yet.)

Quoted 7 lines

We don't want to build up some irrelevant PM links and callbacks. We don't want to have some completely made-up new infrastructure for something that we _already_ want to handle totally differently for init time.

IOW, I argue very strongly against making up something PM-specific, when there really doesn't seem to be much of an advantage. We're much better off trying to share the init code than making up something new.

If I understand correctly, what you're suggesting is impractical. You would have each driver responsible for resuming the devices it registers. If it registered some children synchronously (during the parent's probe) then it would resume them synchronously (during the parent's resume); if it registered them asynchronously then it would resume them asynchronously. In essence, every single device_add() or device_register() call would have to be paired with a resume call.

To make such significant changes in every driver would be prohibitively difficult. What we need is a compromise which gives drivers control over the resume process without making them responsible for actually carrying it out.

So consider this suggestion: Let's define PM groups. Each device belongs to a group, and each group (except group 0, the initial group) has an owner device. By default a device is added to its parent's group during registration, but the driver can request that it be assigned to a different group, which must be owned by that parent.

During resume, each PM group would correspond to an async task. The devices in each group would be resumed sequentially, in order of registration, but asynchronously with respect to other groups. The async thread to resume a group would be launched after the group's owner device was resumed.

So for example, the sibling functions on a PCI card could all be assigned to the same group, but different cards could belong to different groups. Likewise for ATA and PCMCIA controllers. Extra cross-group constraints could be added if needed, but there should be relatively few of them.

This way drivers can decide which of their devices will be resumed in sequence or concurrently, but they won't have to do any of the necessary work.

Alan Stern

#22 lore raw
Quoted 5 lines · Alan Stern

On Sat, 5 Dec 2009, Linus Torvalds wrote:
That's ridiculous. Having gone to all the trouble of building a device tree, one which is presumably still almost entirely correct, why go to all the trouble of tearing it down only to rebuild it again? (Note:
I'm talking about resume-from-RAM here, not resume-from-hibernation.)

There should be nothing special or privileged at all about the device tree that gets built at boot time.

Consider the scenario of the laptop user with a docking station. Adding, removing, and rewriting vast swaths of the device tree across suspend/resume and hibernate/thaw is very easy to do when you are plugging a laptop into one or more docking stations.

Quoted 5 lines

Instead what we do is verify that the devices we remember from before the suspend are still there, and then asynchronously handle new devices which have been plugged in during the meantime. Doing this involves relatively little extra or new code; most of the routines are shared with the runtime PM and device reset paths.

Devices can vanish across suspend to RAM just as easily as they can be added.

Quoted 5 lines

_______________________________________________
linux-pm mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/linux-pm

#23 lore raw
In reply to Alan Stern
Quoted 5 lines · Alan Stern

On Sat, 5 Dec 2009, Linus Torvalds wrote:
That's ridiculous. Having gone to all the trouble of building a device tree, one which is presumably still almost entirely correct, why go to all the trouble of tearing it down only to rebuild it again? (Note:
I'm talking about resume-from-RAM here, not resume-from-hibernation.)

There should be nothing special or privileged at all about the device tree that gets built at boot time.

Consider the scenario of the laptop user with a docking station. Adding, removing, and rewriting vast swaths of the device tree across suspend/resume and hibernate/thaw is very easy to do when you are plugging a laptop into one or more docking stations.

Quoted 5 lines

Instead what we do is verify that the devices we remember from before the suspend are still there, and then asynchronously handle new devices which have been plugged in during the meantime. Doing this involves relatively little extra or new code; most of the routines are shared with the runtime PM and device reset paths.

Devices can vanish across suspend to RAM just as easily as they can be added.

Quoted 5 lines

_______________________________________________
linux-pm mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/linux-pm

#24 lore raw
In reply to Alan Stern
Quoted 68 lines · Alan Stern
Quoted 9 lines · Linus Torvalds

Think of a situation that we already handle pretty poorly: USB mass storage devices over a suspend/resume.

Quoted 3 lines

The device tree represents a good deal of the dependences between devices and the other dependences may be represented as PM links enforcing specific ordering of the PM callbacks.

The device tree means nothing at all, because it may need to be entirely rebuilt at resume time.

Nonsense.

Quoted 5 lines

Optimally, what we _should_ be doing (and aren't) for suspend/resume of USB is to just tear down the whole topology and rebuild it and re-connect the things like mass storage devices. IOW, there would be no device tree to describe the topology, because we're finding it anew. And it's one of the things we _would_ want to do asynchronously with other things.

That's ridiculous. Having gone to all the trouble of building a device tree, one which is presumably still almost entirely correct, why go to all the trouble of tearing it down only to rebuild it again? (Note:
I'm talking about resume-from-RAM here, not resume-from-hibernation.)

Instead what we do is verify that the devices we remember from before the suspend are still there, and then asynchronously handle new devices which have been plugged in during the meantime. Doing this involves relatively little extra or new code; most of the routines are shared with the runtime PM and device reset paths.

As for asynchronicity... At init time, USB device discovery truly is asynchronous. It can happen long after you log in (especially if you don't plug in the device until after you log in!). But at resume time we are more highly constrained. User processes cannot be unfrozen until all the devices have been resumed; otherwise they would encounter errors when trying to do I/O to a suspended device. (With the runtime PM framework this is much less of a problem, but plenty of drivers don't support runtime PM yet.)

Quoted 7 lines

We don't want to build up some irrelevant PM links and callbacks. We don't want to have some completely made-up new infrastructure for something that we _already_ want to handle totally differently for init time.

IOW, I argue very strongly against making up something PM-specific, when there really doesn't seem to be much of an advantage. We're much better off trying to share the init code than making up something new.

If I understand correctly, what you're suggesting is impractical. You would have each driver responsible for resuming the devices it registers. If it registered some children synchronously (during the parent's probe) then it would resume them synchronously (during the parent's resume); if it registered them asynchronously then it would resume them asynchronously. In essence, every single device_add() or device_register() call would have to be paired with a resume call.

To make such significant changes in every driver would be prohibitively difficult. What we need is a compromise which gives drivers control over the resume process without making them responsible for actually carrying it out.

So consider this suggestion: Let's define PM groups. Each device belongs to a group, and each group (except group 0, the initial group) has an owner device. By default a device is added to its parent's group during registration, but the driver can request that it be assigned to a different group, which must be owned by that parent.

Quoted 6 lines

During resume, each PM group would correspond to an async task. The devices in each group would be resumed sequentially, in order of registration, but asynchronously with respect to other groups. The async thread to resume a group would be launched after the group's owner device was resumed.

yes, we've talked about something similar to this before. :)

Hi, Linus,
can you please look at this patch set and see if the idea is right?
http://marc.info/?l=linux-kernel&m=124840449826386&w=2
http://marc.info/?l=linux-acpi&m=124840456826456&w=2
http://marc.info/?l=linux-acpi&m=124840456926459&w=2
http://marc.info/?l=linux-acpi&m=124840457026468&w=2
http://marc.info/?l=linux-acpi&m=124840457126471&w=2

If yes, I'll pick them up again and rework a patch set, including some good thoughts from Rafael.

thanks,
rui

Quoted 18 lines

So for example, the sibling functions on a PCI card could all be assigned to the same group, but different cards could belong to different groups. Likewise for ATA and PCMCIA controllers. Extra cross-group constraints could be added if needed, but there should be relatively few of them.

This way drivers can decide which of their devices will be resumed in sequence or concurrently, but they won't have to do any of the necessary work.

Alan Stern

--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#25 lore raw
Quoted 8 lines · Zhang Rui

Hi, Linus,
can you please look at this patch set and see if the idea is right?
http://marc.info/?l=linux-kernel&m=124840449826386&w=2
http://marc.info/?l=linux-acpi&m=124840456826456&w=2
http://marc.info/?l=linux-acpi&m=124840456926459&w=2
http://marc.info/?l=linux-acpi&m=124840457026468&w=2
http://marc.info/?l=linux-acpi&m=124840457126471&w=2

So I'm not entirely sure about that patch-set, but the thing I like about it is how drivers really sign up to it one by one, rather than having all PCI devices automatically signed up for async behavior.

That said, the thing I don't like about it is some of the same thing I don't necessarily like about the series in Rafael's tree either: it looks rather over-designed with the whole infrastructure for async device logic (your patch in http://marc.info/?l=linux-acpi&m=124840456926459&w=2). How would you explain that whole async_dev_register() logic in simple terms to somebody else?

(I think yours is simpler that the one in the PM tree, but I dunno. I've not really compared the two).

So let me explain my dislike by trying to outline some conceptually simple thing that doesn't have any call-backs, doesn't have any "classes", doesn't require registration etc. It just allows drivers at any level to decide to do some things (not necessarily everything) asynchronously.

Here's the outline:

- first off: drivers that don't know that they nest clearly don't do anything asynchronous. No "PCI devices can be done in parallel" crap, because they really can't - not in the general case. So just forget about that kind of logic entirely: it's just wrong.

- the 'suspend' thing is a depth-first tree walk. As we suspend a node, we first suspend the child nodes, and then we suspend the node itself. Everybody agrees about that, right?

- Trivial "async rule": the tree is walked synchronously, but as we walk it, any point in the tree may decide to do some or all of its suspend asynchronously. For example, when we hit a disk node, the disk driver may just decide that (a) it knows that the disk is an independent thing and (b) it's hierarchical wrt it's parent so (c) it can do the disk suspend asynchronously.

- To protect against a parent node being suspended before any async child work has completed, the child suspend - before it kicks off the actual async work - just needs to take a read-lock on the parent (read-lock, because you may have multiple children sharing a parent, and they don't lock each other out). Then the only thing the asynchronous code needs to do is to release the read lock when it is done.

- Now, the rule just becomes that the parent has to take a write lock on itself when it suspends itself. That will automatically block until all children are done.

Doesn't the above sound _simple_? Doesn't that sound like it should just obviously do the right thing? It sounds like something you can explain as a locking rule without having any complex semantic registration or anything at all.

Now, the problem remains that when you walk the device tree starting off all these potentially asynchronous events, you don't want to do that serialization part (the "parent suspend") as you walk the tree - because then you would only ever do one single level asynchronously. Which is why I suggested splitting the suspend into a "pre-suspend" phase (and a "post-resume" one). Because then the tree walk goes from

	# single depth-first thing
	suspend(root)
	{
		for_each_child(root) {
			// This may take the parent lock for
			// reading if it does something async
			suspend(child);
		}
		// This serializes with any async children
		write_lock(root->lock);
		suspend_one_node(root);
		write_unlock(root->lock);
	}

to

	# Phase one: walk the tree synchronously, starting any
	# async work on the leaves
	suspend_prepare(root)
	{
		for_each_child(root) {
			// This may take the parent lock for
			// reading if it does something async
			suspend_prepare(child);
		}
		suspend_prepare_one_node(root);
	}

	# Phase two: walk the tree synchronously, waiting for
	# and finishing the suspend
	suspend(root)
	{
		for_each_child(root) {
			suspend(child);
		}
		// This serializes with any async children started in phase 1
		write_lock(root->lock);
		suspend_one_node(root);
		write_unlock(root->lock);
	}

and I really think this should work.

The advantage: untouched drivers don't change ANY SEMANTICS AT ALL. If they don't have a 'suspend_prepare()' function, then they still see that exact same sequence of 'suspend()' calls. In fact, even if they have children that _do_ have drivers that have that async phase, they'll never know, because that simple write-semaphore trivially guarantees that whether there was async work or not, it will be completed by the time we call 'suspend()'.

And drivers that want to do things asynchronously don't need to register or worry: all they do is literally

- move their 'suspend()' function to 'suspend_prepare()' instead

- add a

	down_read(dev->parent->lock);
	async_run(mysuspend, dev);

to the point that they want to be asynchronous (which may be _all_ of it or just some slow part). The 'mysuspend' part would be the async part.

- add a

	up_read(dev->parent->lock);

to the end of their asynchronous 'mysuspend()' function, so that when the child has finished suspending, the parent down_write() will finally succeed.

Doesn't that all sound pretty simple? And it has a very clear architecture that is pretty easy to explain to anybody who knows about traversing trees depth-first.

No complex concepts. No change to existing tested drivers. No callbacks, no flags, no nothing. And a pretty simple way for a driver to decide: I'll do my suspends asynchronously (without parent drivers really ever even having to know about it).

I dunno. Maybe I'm overlooking something, but the above is much closer to what I think would be worth doing.

Linus

#26 lore raw
Quoted 3 lines · Linus Torvalds

And drivers that want to do things asynchronously don't need to register or worry: all they do is literally [...]

Side note: for specific bus implementations, you obviously don't have to even expose the choice. Things like the whole "suspend_late" and "resume_early" phases don't make sense for USB devices, and the USB core layer don't even expose those to the various USB drivers.

The same is true of the prepare_suspend/suspend split I'm proposing: I suspect that for something like USB, it would make most sense to just do normal node suspend in prepare_suspend, which would do everything asynchronously. Only USB hub devices would get involved at the later 'suspend()' phase.

So I'm not suggesting that "all drivers" would necessarily even need changing in order to take advantage of asynchronous behavior.

You could change just the _core_ USB layer would do everything automatically for USB devices, and now USB devices would automatically suspend asynchronously not because the generic device layer knows about it, but because the USB bus layer chose to do that "async_run()" on the leaf node suspend functions (or rather: a helper function that calls the leaf-node suspend, and then does the 'up_read()' call on the parent
lock: the actual usb driverrs would never know about any of this).

Linus

#27 lore raw
In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

And drivers that want to do things asynchronously don't need to register or worry: all they do is literally [...]

Side note: for specific bus implementations, you obviously don't have to even expose the choice. Things like the whole "suspend_late" and "resume_early" phases don't make sense for USB devices, and the USB core layer don't even expose those to the various USB drivers.

The same is true of the prepare_suspend/suspend split I'm proposing: I suspect that for something like USB, it would make most sense to just do normal node suspend in prepare_suspend, which would do everything asynchronously. Only USB hub devices would get involved at the later 'suspend()' phase.

So I'm not suggesting that "all drivers" would necessarily even need changing in order to take advantage of asynchronous behavior.

You could change just the _core_ USB layer would do everything automatically for USB devices, and now USB devices would automatically suspend asynchronously not because the generic device layer knows about it, but because the USB bus layer chose to do that "async_run()" on the leaf node suspend functions (or rather: a helper function that calls the leaf-node suspend, and then does the 'up_read()' call on the parent
lock: the actual usb driverrs would never know about any of this).

Linus

#28 lore raw
Quoted 8 lines · Linus Torvalds

The same is true of the prepare_suspend/suspend split I'm proposing:
I
suspect that for something like USB, it would make most sense to just do
normal node suspend in prepare_suspend, which would do everything asynchronously. Only USB hub devices would get involved at the later 'suspend()' phase.

Wasn't part of the goal with prepare_suspend() vs. suspend() to handle the problem of backing store vs the VM ?

IE. Once any device potentially in the VM path is suspended, things like kmalloc() or gfp() can potentially stall until resume or did we address that recently ?

Iirc, part of the idea behind prepare_* is that it's safe vs. the above. Now if you start suspending USB devices at prepare() then you break that assumption since those could be mass storage with dirty mmap'ed pages on them.

Now, I'm all for fixing it at the VM/allocator level (if we didn't already) turning pretty much everything into NO_IO once we start suspending devices but that's a whole different matter :-)

Cheers,
Ben.

#29 lore raw
In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds

The same is true of the prepare_suspend/suspend split I'm proposing:
I
suspect that for something like USB, it would make most sense to just do
normal node suspend in prepare_suspend, which would do everything asynchronously. Only USB hub devices would get involved at the later 'suspend()' phase.

Wasn't part of the goal with prepare_suspend() vs. suspend() to handle the problem of backing store vs the VM ?

IE. Once any device potentially in the VM path is suspended, things like kmalloc() or gfp() can potentially stall until resume or did we address that recently ?

Iirc, part of the idea behind prepare_* is that it's safe vs. the above. Now if you start suspending USB devices at prepare() then you break that assumption since those could be mass storage with dirty mmap'ed pages on them.

Now, I'm all for fixing it at the VM/allocator level (if we didn't already) turning pretty much everything into NO_IO once we start suspending devices but that's a whole different matter :-)

Cheers,
Ben.

#30 lore raw
In reply to Linus Torvalds
Quoted 10 lines · On Sun, 6 Dec 2009 21:57:55 -0800 (PST) Linus Torvalds…

Now, the problem remains that when you walk the device tree starting off all these potentially asynchronous events, you don't want to do that serialization part (the "parent suspend") as you walk the tree - because then you would only ever do one single level asynchronously. Which is why I suggested splitting the suspend into a "pre-suspend" phase (and a "post-resume" one). Because then the tree walk goes from

I dunno. Maybe I'm overlooking something, but the above is much closer to what I think would be worth doing.

with what you're describing I suspect the current async function calls could be used;
in the first tree walk, the drivers do an async_schedule() of the things they want done asynchronous;
all the core then needs to do is a full synchronization step between the two tree walks... and we get pretty much all the benefits without needing the read-then-write-lock primitive for synchronization.

alternative would be to do the synchronization in the part where we know there's a dependency (like your lock is doing);
but instead of a lock we could store the async cookie there; and just wait on that in the 2nd phase.... this would be more finegrained, and an optimization from the "global synchronize"... but I'm not sure it'll be worth it in practice; it will if there's significant cost in various parts of the tree AND in the 2nd run; if the 2nd run is cheap in general, you're not going to get real extra parallelism at the price of more complexity.

Signature · 3 lines
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#31 lore raw
In reply to Linus Torvalds
Quoted 10 lines · On Sun, 6 Dec 2009 21:57:55 -0800 (PST) Linus Torvalds…

Now, the problem remains that when you walk the device tree starting off all these potentially asynchronous events, you don't want to do that serialization part (the "parent suspend") as you walk the tree - because then you would only ever do one single level asynchronously. Which is why I suggested splitting the suspend into a "pre-suspend" phase (and a "post-resume" one). Because then the tree walk goes from

I dunno. Maybe I'm overlooking something, but the above is much closer to what I think would be worth doing.

with what you're describing I suspect the current async function calls could be used;
in the first tree walk, the drivers do an async_schedule() of the things they want done asynchronous;
all the core then needs to do is a full synchronization step between the two tree walks... and we get pretty much all the benefits without needing the read-then-write-lock primitive for synchronization.

alternative would be to do the synchronization in the part where we know there's a dependency (like your lock is doing);
but instead of a lock we could store the async cookie there; and just wait on that in the 2nd phase.... this would be more finegrained, and an optimization from the "global synchronize"... but I'm not sure it'll be worth it in practice; it will if there's significant cost in various parts of the tree AND in the 2nd run; if the 2nd run is cheap in general, you're not going to get real extra parallelism at the price of more complexity.

Signature · 3 lines
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#32 lore raw
In reply to Linus Torvalds
Quoted 27 lines · Linus Torvalds
	# Phase one: walk the tree synchronously, starting any
	# async work on the leaves
	suspend_prepare(root)
	{
		for_each_child(root) {
			// This may take the parent lock for
			// reading if it does something async
			suspend_prepare(child);
		}
		suspend_prepare_one_node(root);
	}

	# Phase two: walk the tree synchronously, waiting for
	# and finishing the suspend
	suspend(root)
	{
		for_each_child(root) {
			suspend(child);
		}
		// This serializes with any async children started in phase 1
		write_lock(root->lock);
		suspend_one_node(root);
		write_unlock(root->lock);
	}

and I really think this should work.

Quoted 7 lines

No complex concepts. No change to existing tested drivers. No callbacks, no flags, no nothing. And a pretty simple way for a driver to decide: I'll do my suspends asynchronously (without parent drivers really ever even having to know about it).

I dunno. Maybe I'm overlooking something, but the above is much closer to what I think would be worth doing.

You're overlooking resume. It's more difficult than suspend. The issue is that a child can't start its async part until the parent's synchronous part is finished.

So for example, suppose the device listing contains P, C, Q, where C is a child of P, Q is unrelated, and P has a long-lasting asynchronous requirement. The resume process will stall upon reaching C, waiting for P to finish. Thus even though P and Q might be able to resume in parallel, they won't get the chance.

An approach that handles resume well can probably be adapted to handle suspend too. The reverse isn't true, as this example shows.

Alan Stern

#33 lore raw
Quoted 7 lines · Alan Stern
Quoted 3 lines

I dunno. Maybe I'm overlooking something, but the above is much closer to what I think would be worth doing.

You're overlooking resume. It's more difficult than suspend. The issue is that a child can't start its async part until the parent's synchronous part is finished.

No, I haven't overlooked resume at all. I just assumed that it was obvious. It's the exact same thing, except in reverse (the locking ends up being slightly different, but the changes are actually fairly straightforward).

And by reverse, I mean that you walk the tree in the reverse order too, exactly like we already do - on suspend we walk it children-first, on resume we walk it parents-first (small detail: we actually just walk a simple linked list, but the list is topologically ordered, so walking it forwards/backwards is topologically the same thing as doing that depth-first search).

Quoted 5 lines

So for example, suppose the device listing contains P, C, Q, where C is a child of P, Q is unrelated, and P has a long-lasting asynchronous requirement. The resume process will stall upon reaching C, waiting for P to finish. Thus even though P and Q might be able to resume in parallel, they won't get the chance.

No. The resume process does EXCTLY THE SAME THING as I outlined for suspend, but just all in reverse. So now the resume process becomes the same two-phase thing:

	# Phase one
	resume(root)
	{
		// This can do things asynchronously if it wants,
		// but needs to take the write lock on itself until
		// it is done if it does
		resume_one_node(root);
		for_each_child(root)
			resume(child);
	}

	# Phase two
	post_resume(root)
	{
		post_resume_one_node(root);
		for_each_child(root)
			post_resume(child);
	}

Notice? It's _exactly_ the same thing as suspend - except all turned around. We do the nodes before the children ("walk the list backwards"), and we also do the locking the other way around (ie on suspend we'd lock the _parent_ if we wanted to do async stuff - to keep it around - but on resume we lock _ourselves_, so that the children can have something to wait on. Also note how we take a _write_ lock rather than a read lock).

(And again, I've only written it out in email, I've not tested it or thought about it all that deeply, so you'll excuse any stupid thinkos.)

Now, for something like PCI, I'd suggest (once more) leaving all drivers totally unchanged, and you end up with the exact same behavior as we had before (no real change to the whole resume ordering, and everything is synchronous so there is no relevant locking).

But how would the USB layer do this?

Simple: all the normal leaf devices would have their resume callback be called at "post_resume()" time (exactly the reverse of the suspend phase:
we suspend early, and we resume late - it's all a mirror image). And I'd suggest that the USB layer do it all totally asynchronously, except again turned around the other way.

Remember how on suspend, the suspend of a leaf device ended up being an issue of asynchronously calling a function that did the suspend, and then released the read-lock of the parent. Resume is the same, except now we'd actually want to take the parent read-lock asynchronously too, so you'd do

	down_write(leaf->lock);
	async_schedule(usb_node_resume, leaf);

where that function simply does

	usb_node_resume(node)
	{
		/* Wait for the parent to have resumed completely */
		down_read(node->parent->lock);
		node->resume(node)
		up_read(node->parent->lock);
		up_write(node->lock);
	}

and you're all done. Once more the ordering and the locking takes care of any need to serialize - there is no data structures to keep track of.

And what about USB hubs? They get resumed in the first phase (again, exactly the mirror image of the suspend), and the only thing they need to do is that _exact_ same thing above:

	down_write(hub->lock);
	async_schedule(usb_node_resume, hub);

- Ta-daa! All done.

Notice? It's really pretty straightforward, and there are _zero_ new concepts. And again, no callbacks, no nothing. Just the obvious mirror image of what happened when suspending. We do everything with simple async calls. And none of the tree walking actually blocks (yes, we do a "down_write()" on the nodes as we schedule the resume code, but it won't be a blocking one, since that is the first time we encounter that node:
the blocking will come later when the async threads actually need to wait for things).

Again, I do not guarantee that I've dotted every i, and crossed every t. It's just that I'm pretty sure that we really don't need any fancy "infrastructure" for something this simple. And I really much prefer "conceptually simple high-level model" over a model of "keep track of all the relationships and have some complex model of devices".

So let's just look at your example:

Quoted 3 lines

So for example, suppose the device listing contains P, C, Q, where C is a child of P, Q is unrelated, and P has a long-lasting asynchronous requirement.

The tree is:

	... -> P -> C
	    -> Q

and with what I suggest, during phase one, P will asynchronously start the resume. As part of its async resume it will have to wait for it's parents, of course, but all of that happens in a separate context, and the tree traversal goes on.

And during phase #1, C and Q won't do anything at all. We _could_ do them during this phase, and it would actually all work out fine, but we wouldn't want to do that for a simple reason: we _want_ the pre_suspend and post_resume phases to be total mirror images, because if we end up doing error handling for the pre-suspend case, then the post-resume phase would be the "fixup" for it, so we actually want leaf things to happen during phase #2 - not because it would screw up locking or ordering, but because of other issues.

When we hit phase #2, we then do C and Q, and do the same thing - we have an async call that does the read-lock on the parent to make sure it's all resumed, and then we resume C and Q. And they'll automatically resume in parallel (unless C is waiting for P, of course, in which case P and Q end up resuming in parallel, and C ends up waiting).

Now, the above just takes care of the inter-device ordering. There are unrelated semantics we want to give, like "all devices will have resumed before we start waking up user space". Those are unrelated to the topological requirements, of course, and are not a requirement imposed by the device tree, but by our _other_ semantics (IOW, in this respect it's kind of like how we wanted pre-suspend and post-resume to be mirror images for other outside reasons).

So we'd actually have a "phase #3", but that phase wouldn't be visible to the devices themselves, it would be a

	# Phase tree: make sure everything is resumed
	for_each_device() {
		read_lock(dev->lock);
		read_unlock(dev->lock);
	}

but as you can see, there's no actual device callbacks involved. It would be just the code device layer saying "ok, now I'm going to wait for all the devices to have finished their resume".

Linus

#34 lore raw
In reply to Alan Stern
Quoted 7 lines · Alan Stern
Quoted 3 lines

I dunno. Maybe I'm overlooking something, but the above is much closer to what I think would be worth doing.

You're overlooking resume. It's more difficult than suspend. The issue is that a child can't start its async part until the parent's synchronous part is finished.

No, I haven't overlooked resume at all. I just assumed that it was obvious. It's the exact same thing, except in reverse (the locking ends up being slightly different, but the changes are actually fairly straightforward).

And by reverse, I mean that you walk the tree in the reverse order too, exactly like we already do - on suspend we walk it children-first, on resume we walk it parents-first (small detail: we actually just walk a simple linked list, but the list is topologically ordered, so walking it forwards/backwards is topologically the same thing as doing that depth-first search).

Quoted 5 lines

So for example, suppose the device listing contains P, C, Q, where C is a child of P, Q is unrelated, and P has a long-lasting asynchronous requirement. The resume process will stall upon reaching C, waiting for P to finish. Thus even though P and Q might be able to resume in parallel, they won't get the chance.

No. The resume process does EXCTLY THE SAME THING as I outlined for suspend, but just all in reverse. So now the resume process becomes the same two-phase thing:

	# Phase one
	resume(root)
	{
		// This can do things asynchronously if it wants,
		// but needs to take the write lock on itself until
		// it is done if it does
		resume_one_node(root);
		for_each_child(root)
			resume(child);
	}

	# Phase two
	post_resume(root)
	{
		post_resume_one_node(root);
		for_each_child(root)
			post_resume(child);
	}

Notice? It's _exactly_ the same thing as suspend - except all turned around. We do the nodes before the children ("walk the list backwards"), and we also do the locking the other way around (ie on suspend we'd lock the _parent_ if we wanted to do async stuff - to keep it around - but on resume we lock _ourselves_, so that the children can have something to wait on. Also note how we take a _write_ lock rather than a read lock).

(And again, I've only written it out in email, I've not tested it or thought about it all that deeply, so you'll excuse any stupid thinkos.)

Now, for something like PCI, I'd suggest (once more) leaving all drivers totally unchanged, and you end up with the exact same behavior as we had before (no real change to the whole resume ordering, and everything is synchronous so there is no relevant locking).

But how would the USB layer do this?

Simple: all the normal leaf devices would have their resume callback be called at "post_resume()" time (exactly the reverse of the suspend phase:
we suspend early, and we resume late - it's all a mirror image). And I'd suggest that the USB layer do it all totally asynchronously, except again turned around the other way.

Remember how on suspend, the suspend of a leaf device ended up being an issue of asynchronously calling a function that did the suspend, and then released the read-lock of the parent. Resume is the same, except now we'd actually want to take the parent read-lock asynchronously too, so you'd do

	down_write(leaf->lock);
	async_schedule(usb_node_resume, leaf);

where that function simply does

	usb_node_resume(node)
	{
		/* Wait for the parent to have resumed completely */
		down_read(node->parent->lock);
		node->resume(node)
		up_read(node->parent->lock);
		up_write(node->lock);
	}

and you're all done. Once more the ordering and the locking takes care of any need to serialize - there is no data structures to keep track of.

And what about USB hubs? They get resumed in the first phase (again, exactly the mirror image of the suspend), and the only thing they need to do is that _exact_ same thing above:

	down_write(hub->lock);
	async_schedule(usb_node_resume, hub);

- Ta-daa! All done.

Notice? It's really pretty straightforward, and there are _zero_ new concepts. And again, no callbacks, no nothing. Just the obvious mirror image of what happened when suspending. We do everything with simple async calls. And none of the tree walking actually blocks (yes, we do a "down_write()" on the nodes as we schedule the resume code, but it won't be a blocking one, since that is the first time we encounter that node:
the blocking will come later when the async threads actually need to wait for things).

Again, I do not guarantee that I've dotted every i, and crossed every t. It's just that I'm pretty sure that we really don't need any fancy "infrastructure" for something this simple. And I really much prefer "conceptually simple high-level model" over a model of "keep track of all the relationships and have some complex model of devices".

So let's just look at your example:

Quoted 3 lines

So for example, suppose the device listing contains P, C, Q, where C is a child of P, Q is unrelated, and P has a long-lasting asynchronous requirement.

The tree is:

	... -> P -> C
	    -> Q

and with what I suggest, during phase one, P will asynchronously start the resume. As part of its async resume it will have to wait for it's parents, of course, but all of that happens in a separate context, and the tree traversal goes on.

And during phase #1, C and Q won't do anything at all. We _could_ do them during this phase, and it would actually all work out fine, but we wouldn't want to do that for a simple reason: we _want_ the pre_suspend and post_resume phases to be total mirror images, because if we end up doing error handling for the pre-suspend case, then the post-resume phase would be the "fixup" for it, so we actually want leaf things to happen during phase #2 - not because it would screw up locking or ordering, but because of other issues.

When we hit phase #2, we then do C and Q, and do the same thing - we have an async call that does the read-lock on the parent to make sure it's all resumed, and then we resume C and Q. And they'll automatically resume in parallel (unless C is waiting for P, of course, in which case P and Q end up resuming in parallel, and C ends up waiting).

Now, the above just takes care of the inter-device ordering. There are unrelated semantics we want to give, like "all devices will have resumed before we start waking up user space". Those are unrelated to the topological requirements, of course, and are not a requirement imposed by the device tree, but by our _other_ semantics (IOW, in this respect it's kind of like how we wanted pre-suspend and post-resume to be mirror images for other outside reasons).

So we'd actually have a "phase #3", but that phase wouldn't be visible to the devices themselves, it would be a

	# Phase tree: make sure everything is resumed
	for_each_device() {
		read_lock(dev->lock);
		read_unlock(dev->lock);
	}

but as you can see, there's no actual device callbacks involved. It would be just the code device layer saying "ok, now I'm going to wait for all the devices to have finished their resume".

Linus

#35 lore raw
Quoted 9 lines · Linus Torvalds

And during phase #1, C and Q won't do anything at all. We _could_ do them during this phase, and it would actually all work out fine, but we wouldn't want to do that for a simple reason: we _want_ the pre_suspend and post_resume phases to be total mirror images, because if we end up doing error handling for the pre-suspend case, then the post-resume phase would be the "fixup" for it, so we actually want leaf things to happen during phase #2 - not because it would screw up locking or ordering, but because of other issues.

Ho humm.

This part made me think. Since I started mulling over the fact that we could do the resume thing in a single phase (and really only wanted the second phase in order to be a mirror image to the suspend), I started thinking that we could perhaps do even the suspend with a single phase, and avoid introducing that pre-suspend/post-resume phase at all.

And now that I think about it, we can do that by simply changing the locking just a tiny bit.

I originally envisioned that two-pase suspend because I was thinking that the first phase would start off the suspend, and the second phase would finish it, but we can actually do it all with a single phase that does both. So starting with just the regular depth-first post-ordering that is a suspend:

        suspend(root)
        {
                for_each_child(root)
			suspend(child);
		suspend_one_node(root)
        }

the rule would be that for something like USB that wants to do the suspend asynchronously, the node suspend routine would do

	usb_node_suspend(node)
	{
		// Make sure parent doesn't suspend: this will not block, 
		// because we'll call the 'suspend' function for all nodes
		// before we call it for the parent.
		down_read(node->parent->lock);

		// Do the part that may block asynchronously
		async_schedule(do_usb_node_suspend, node);
	}

	do_usb_node_suspend(node)
	{
		// Start out suspend. This will block if we have any
		// children that are still busy suspending (they will
		// have done a down_read() in their suspend).
		down_write(node->lock);
		node->suspend(node);
		up_write(node->lock);

		// This lets our parent continue
		up_read(node->parent->lock);
	}

and it looks like we don't even need a second phase at all.

IOW, I think USB could do this on its own right now, with no extra infrastructure from the device layer AT ALL, except for one small thing:
that new "rwsem" lock in the device data structure, and then we'd need the "wait for everybody to have completed" loop, ie

	for_each_dev(dev) {
		down_write(dev->lock);
		up_write(dev->lock);
	}

thing at the end of the suspend loop (same thing as I mentioned about resuming).

So I think even that whole two-phase thing was unnecessarily complicated.

Linus

#36 lore raw
In reply to Linus Torvalds
Quoted 9 lines · Linus Torvalds

And during phase #1, C and Q won't do anything at all. We _could_ do them during this phase, and it would actually all work out fine, but we wouldn't want to do that for a simple reason: we _want_ the pre_suspend and post_resume phases to be total mirror images, because if we end up doing error handling for the pre-suspend case, then the post-resume phase would be the "fixup" for it, so we actually want leaf things to happen during phase #2 - not because it would screw up locking or ordering, but because of other issues.

Ho humm.

This part made me think. Since I started mulling over the fact that we could do the resume thing in a single phase (and really only wanted the second phase in order to be a mirror image to the suspend), I started thinking that we could perhaps do even the suspend with a single phase, and avoid introducing that pre-suspend/post-resume phase at all.

And now that I think about it, we can do that by simply changing the locking just a tiny bit.

I originally envisioned that two-pase suspend because I was thinking that the first phase would start off the suspend, and the second phase would finish it, but we can actually do it all with a single phase that does both. So starting with just the regular depth-first post-ordering that is a suspend:

        suspend(root)
        {
                for_each_child(root)
			suspend(child);
		suspend_one_node(root)
        }

the rule would be that for something like USB that wants to do the suspend asynchronously, the node suspend routine would do

	usb_node_suspend(node)
	{
		// Make sure parent doesn't suspend: this will not block, 
		// because we'll call the 'suspend' function for all nodes
		// before we call it for the parent.
		down_read(node->parent->lock);

		// Do the part that may block asynchronously
		async_schedule(do_usb_node_suspend, node);
	}

	do_usb_node_suspend(node)
	{
		// Start out suspend. This will block if we have any
		// children that are still busy suspending (they will
		// have done a down_read() in their suspend).
		down_write(node->lock);
		node->suspend(node);
		up_write(node->lock);

		// This lets our parent continue
		up_read(node->parent->lock);
	}

and it looks like we don't even need a second phase at all.

IOW, I think USB could do this on its own right now, with no extra infrastructure from the device layer AT ALL, except for one small thing:
that new "rwsem" lock in the device data structure, and then we'd need the "wait for everybody to have completed" loop, ie

	for_each_dev(dev) {
		down_write(dev->lock);
		up_write(dev->lock);
	}

thing at the end of the suspend loop (same thing as I mentioned about resuming).

So I think even that whole two-phase thing was unnecessarily complicated.

Linus

#37 lore raw
In reply to Linus Torvalds
Quoted 12 lines · Linus Torvalds

No, I haven't overlooked resume at all. I just assumed that it was obvious. It's the exact same thing, except in reverse (the locking ends up being slightly different, but the changes are actually fairly straightforward).

And by reverse, I mean that you walk the tree in the reverse order too, exactly like we already do - on suspend we walk it children-first, on resume we walk it parents-first (small detail: we actually just walk a simple linked list, but the list is topologically ordered, so walking it forwards/backwards is topologically the same thing as doing that depth-first search).

Quoted 6 lines

Notice? It's _exactly_ the same thing as suspend - except all turned around. We do the nodes before the children ("walk the list backwards"), and we also do the locking the other way around (ie on suspend we'd lock the _parent_ if we wanted to do async stuff - to keep it around - but on resume we lock _ourselves_, so that the children can have something to wait on. Also note how we take a _write_ lock rather than a read lock).

Okay, I think I've got it. But you're wrong about one thing: Resume isn't _exactly_ the reverse of suspend. For both of them we have to start the async thread in the first pass. So instead of resume/post_resume we would have pre_resume/resume, just like pre_suspend/suspend.

During the pre- pass, the driver launches an async thread and takes the appropriate locks. The thread does its work as appropriate (with locking to insure that it first waits for children or parents), and then in the second pass the driver waits for the async thread to finish.

A non-async driver (i.e., most of them) would ignore the pre- pass entirely and do all its work in the second pass.

An async-aware driver would look like this:

	pre_suspend(dev)
	{
		/* Prevent parent from suspending until we are ready */
		down_read(dev->parent->lock);
		dev->pm_cookie = async_schedule(async_suspend, dev);
	}

	async_suspend(dev)
	{
		/* Wait until all children are fully suspended */
		down_write(dev->lock);
		Suspend dev, taking as much time as needed
		up_write(dev->lock);

		/* Allow parent to suspend */
		up_read(dev->parent->lock);
	}

	suspend(dev)
	{
		/* Wait until the suspend is complete */
		async_synchronize_cookie(dev->pm_cookie);
	}


	pre_resume(dev)
	{
		/* Prevent children from resuming */
		down_write(dev->lock);
		dev->pm_cookie = async_schedule(async_resume, dev);
	}

	async_resume(dev)
	{
		/* Wait until parent is fully resumed */
		down_read(dev->parent->lock);
		Resume dev, taking as much time as needed
		up_read(dev->parent->lock);

		/* Allow children to resume */
		up_write(dev->lock);
	}

	resume(dev)
	{
		/* Wait until resume is complete */
		async_synchronize_cookie(dev->pm_cookie);
	}

So there's some time symmetry here, but it isn't perfect. This is probably what you had in mind all along, but I needed to get it straight.

There's some question about what to do if a suspend or resume fails. A bunch of async threads will have been launched for other devices, but now there won't be anything to wait for them. It's not clear how this should be handled.

Alan Stern

#38 lore raw
Quoted 3 lines · Alan Stern

Okay, I think I've got it. But you're wrong about one thing: Resume isn't _exactly_ the reverse of suspend.

Yeah, no. But I think I made it much closer by getting rid of pre-suspend and post-resume (my next email to the one you quoted).

And yeah, I started thinking along those lines exactly because it wasn't as clean a mirror image as I thought it should be able to be.

Quoted 2 lines

A non-async driver (i.e., most of them) would ignore the pre- pass entirely and do all its work in the second pass.

See my second email, where I think I can get rid of the whole second pass thing. I think you'll agree that it's an even nicer mirror image.

Quoted 4 lines

There's some question about what to do if a suspend or resume fails. A bunch of async threads will have been launched for other devices, but now there won't be anything to wait for them. It's not clear how this should be handled.

I think the rule for "suspend fails" is very simple: you can't fail in the async codepath. There's no sane way to return errors, and trying to would be too complex anyway. What would you do?

In fact, even though we _can_ fail in the synchronous path, I personally consider a device driver that ever fails its suspend to be terminally broken. We're practically always better off suspending and simply turning off the power than saying "uh, I failed the suspend".

I've occasionally hit a few drivers that caused suspend failures, and each and every time it was a driver bug, and the right thing to do was to just ignore the error and suspend anyway - returning an error code and trying to undo the suspend is not what anybody ever really wants, even if our model _allows_ for it.

(And the rule for "resume fails" is even simpler: there's nothing we can really do if something fails to resume - and that's true whether the failure is synchronous or asynchronous. The device is dead. Try to reset it, or remove it from the device tree. Tough).

Linus

#39 lore raw
In reply to Alan Stern
Quoted 3 lines · Alan Stern

Okay, I think I've got it. But you're wrong about one thing: Resume isn't _exactly_ the reverse of suspend.

Yeah, no. But I think I made it much closer by getting rid of pre-suspend and post-resume (my next email to the one you quoted).

And yeah, I started thinking along those lines exactly because it wasn't as clean a mirror image as I thought it should be able to be.

Quoted 2 lines

A non-async driver (i.e., most of them) would ignore the pre- pass entirely and do all its work in the second pass.

See my second email, where I think I can get rid of the whole second pass thing. I think you'll agree that it's an even nicer mirror image.

Quoted 4 lines

There's some question about what to do if a suspend or resume fails. A bunch of async threads will have been launched for other devices, but now there won't be anything to wait for them. It's not clear how this should be handled.

I think the rule for "suspend fails" is very simple: you can't fail in the async codepath. There's no sane way to return errors, and trying to would be too complex anyway. What would you do?

In fact, even though we _can_ fail in the synchronous path, I personally consider a device driver that ever fails its suspend to be terminally broken. We're practically always better off suspending and simply turning off the power than saying "uh, I failed the suspend".

I've occasionally hit a few drivers that caused suspend failures, and each and every time it was a driver bug, and the right thing to do was to just ignore the error and suspend anyway - returning an error code and trying to undo the suspend is not what anybody ever really wants, even if our model _allows_ for it.

(And the rule for "resume fails" is even simpler: there's nothing we can really do if something fails to resume - and that's true whether the failure is synchronous or asynchronous. The device is dead. Try to reset it, or remove it from the device tree. Tough).

Linus

#40 lore raw
Quoted 2 lines · Linus Torvalds

See my second email, where I think I can get rid of the whole second pass thing. I think you'll agree that it's an even nicer mirror image.

Yes, I like this approach better and better.

There is still a problem. In your code outlines, you have presented a classic depth-first (suspend) or depth-last (resume) tree algorithm. But that's not how the PM core works. Instead it maintains dpm_list, a list of all devices in order of registration. Suspends and resumes are carried out by iterating along this list, in the reverse and forward directions respectively.

There are two advantages. The matter of stack usage, of course. But more importantly, this order of devices is guaranteed to work. For any device D, we _know_ that the system can function properly in circumstances where everything on dpm_list before D is active and everything after D is inactive -- because that's the state the system was in when D was registered. Any other order risks errors because of unknown dependencies.

The consequence is that there's no way to hand off an entire subtree to an async thread. And as a result, your single-pass algorithm runs into the kind of "stall" problem I described before.

(In theory we could convert over to a tree algorithm. IMO that would be nearly as dangerous as going to a full-fledged totally async scheme.)

But all is not lost. We can still get what we want using a two-pass list algorithm, where one of the passes is contained within the PM core -- no extra callbacks are needed. Here's how suspend would work:

	dpm_suspend()	/* Suspend all devices on dpm_list */
	{
		list_for_each_entry_reverse(dev, dpm_list, ...) {

			/* Make the parent wait for dev */
			down_read(dev->parent->lock);
			if (dev->async_pm)
				async_schedule(device_suspend, dev);
		}
		list_for_each_entry_reverse(dev, dpm_list, ...) {
			if (!dev->async_pm)
				device_suspend(dev);
		}
		async_synchronize_full();
	}

	device_suspend(dev)	/* Suspend a single device */
	{
		/* Wait until all the children are suspended */
		down_write(dev->lock);
		dev->bus->suspend(dev);
		up_write(dev->lock);

		/* Tell the parent we are finished */
		up_read(dev->parent->lock);
	}

I have glossed over a bunch of details, such as the fact that device_suspend() really takes two arguments. And it's necessary to be more careful with the list operations than shown here, because devices can be unregistered while all this is going on.

Still, this seems reasonable. Bus subsystems and drivers can set the dev->async_pm flag as desired, and they can use the new rwsems to handle special dependencies without involving the PM core. No new callbacks are needed, nor any changes to existing methods. (Convincing lockdep that all this fancy footwork is valid may require some effort, though.)

By the way, this bears a striking resemblance to Rafael's patch. The biggest difference is the use of the new rwsem for dependency resolution, instead his somewhat cumbersome constraint structures.

Quoted 8 lines
Quoted 4 lines

There's some question about what to do if a suspend or resume fails. A bunch of async threads will have been launched for other devices, but now there won't be anything to wait for them. It's not clear how this should be handled.

I think the rule for "suspend fails" is very simple: you can't fail in the async codepath. There's no sane way to return errors, and trying to would be too complex anyway. What would you do?

You could prevent the suspend procedure from going any further and abort the entire system sleep. If you wanted to.

Quoted 10 lines

In fact, even though we _can_ fail in the synchronous path, I personally consider a device driver that ever fails its suspend to be terminally broken. We're practically always better off suspending and simply turning off the power than saying "uh, I failed the suspend".

I've occasionally hit a few drivers that caused suspend failures, and each and every time it was a driver bug, and the right thing to do was to just ignore the error and suspend anyway - returning an error code and trying to undo the suspend is not what anybody ever really wants, even if our model _allows_ for it.

There is a valid reason for aborting a sleep transition: the driver has received a remote wakeup request. Wakeup requests race with sleep, of course. A request coming after the system is asleep will wake it up; one coming before the system is asleep should either cause it to wake up immediately after shutting down or prevent the sleep entirely.

Causing the system to wake up immediately needs hardware support. But by the time the kernel is aware of a wakeup request, the request is generally no longer present in the hardware. (For example, an interrupt has been delivered and the IRQ line is no longer active.) So the only remaining choice is to abort the sleep transition.

Quoted 4 lines

(And the rule for "resume fails" is even simpler: there's nothing we can really do if something fails to resume - and that's true whether the failure is synchronous or asynchronous. The device is dead. Try to reset it, or remove it from the device tree. Tough).

Right.

Alan Stern

#41 lore raw
Quoted 5 lines · Alan Stern

Yes, I like this approach better and better.

There is still a problem. In your code outlines, you have presented a classic depth-first (suspend) or depth-last (resume) tree algorithm.

Yes, I did that because that clarifies the locking rules (ie "we traverse parents nodes last/first"), not because it was actually relevant to anything else.

And the whole pre-order vs post-order is important, and really only shows up when you show the pseudo-code as a tree walk.

Quoted 2 lines

But that's not how the PM core works. Instead it maintains dpm_list, a list of all devices in order of registration.

Right. I did mention that in a couple of the asides, I'm well aware that we don't actually traverse the tree as a tree.

But the "traverse list forward" is logically the same thing as doing a pre-order DFS, while going backwards is equivalent to doing a post-order DFS, since all we really care about is the whole "parent first" or "children first" part of the ordering.

So I wanted to show the logic in pseudo-code using the tree walk (because that explains the logic _conceptually_ much better), but the actual code would just do the list traversal.

Quoted 3 lines

The consequence is that there's no way to hand off an entire subtree to an async thread. And as a result, your single-pass algorithm runs into the kind of "stall" problem I described before.

No, look again. There's no stall in the thing, because all it really depends on is (for the suspend path) is that it sees all children before the parent (because the child will do a "down_read()" on the parent node and that should not stall), and for the resume path it depends on seeing the parent node before any children (because the parent node does that "down_write()" on its own node).

Everything else is _entirely_ asynchronous, including all the other locks it takes. So there are no stalls (except, of course, if we then hit limits on numbers of outstanding async work and refuse to create too many outstanding async things, but that's a separate issue, and intentional, of course).

You're right that my first one (two-phase suspend) had a stall situation.

Linus

#42 lore raw
In reply to Alan Stern
Quoted 5 lines · Alan Stern

Yes, I like this approach better and better.

There is still a problem. In your code outlines, you have presented a classic depth-first (suspend) or depth-last (resume) tree algorithm.

Yes, I did that because that clarifies the locking rules (ie "we traverse parents nodes last/first"), not because it was actually relevant to anything else.

And the whole pre-order vs post-order is important, and really only shows up when you show the pseudo-code as a tree walk.

Quoted 2 lines

But that's not how the PM core works. Instead it maintains dpm_list, a list of all devices in order of registration.

Right. I did mention that in a couple of the asides, I'm well aware that we don't actually traverse the tree as a tree.

But the "traverse list forward" is logically the same thing as doing a pre-order DFS, while going backwards is equivalent to doing a post-order DFS, since all we really care about is the whole "parent first" or "children first" part of the ordering.

So I wanted to show the logic in pseudo-code using the tree walk (because that explains the logic _conceptually_ much better), but the actual code would just do the list traversal.

Quoted 3 lines

The consequence is that there's no way to hand off an entire subtree to an async thread. And as a result, your single-pass algorithm runs into the kind of "stall" problem I described before.

No, look again. There's no stall in the thing, because all it really depends on is (for the suspend path) is that it sees all children before the parent (because the child will do a "down_read()" on the parent node and that should not stall), and for the resume path it depends on seeing the parent node before any children (because the parent node does that "down_write()" on its own node).

Everything else is _entirely_ asynchronous, including all the other locks it takes. So there are no stalls (except, of course, if we then hit limits on numbers of outstanding async work and refuse to create too many outstanding async things, but that's a separate issue, and intentional, of course).

You're right that my first one (two-phase suspend) had a stall situation.

Linus

#43 lore raw
Quoted 11 lines · Linus Torvalds
Quoted 3 lines

The consequence is that there's no way to hand off an entire subtree to an async thread. And as a result, your single-pass algorithm runs into the kind of "stall" problem I described before.

No, look again. There's no stall in the thing, because all it really depends on is (for the suspend path) is that it sees all children before the parent (because the child will do a "down_read()" on the parent node and that should not stall), and for the resume path it depends on seeing the parent node before any children (because the parent node does that "down_write()" on its own node).

Quoted 5 lines

Everything else is _entirely_ asynchronous, including all the other locks it takes. So there are no stalls (except, of course, if we then hit limits on numbers of outstanding async work and refuse to create too many outstanding async things, but that's a separate issue, and intentional, of course).

It only seems that way because you didn't take into account devices that suspend synchronously but whose children suspend asynchronously.

A synchronous suspend routine for a device with async child suspends would have to look just like your usb_node_suspend():

	suspend_one_node(dev)
	{
		/* Wait until the children are suspended */
		down_write(dev->lock);
		Suspend dev
		up_write(dev->lock);

		/* Allow the parent to suspend */
		up_read(dev->parent->lock);
	}

So now suppose we've got two USB host controllers, A and B. They are PCI devices, so they suspend synchronously. Each has a root hub child (P and Q respectively) which is a USB device and therefore suspends asynchronously. dpm_list contains: A, P, B, Q. (In fact A doesn't enter into this discussion; you can ignore it.)

In your one-pass algorithm, we start with usb_node_suspend(Q). It does down_read(B->lock) and starts an async task for Q. Then we move on to suspend_one_node(B). It does down_write(B->lock) and blocks until the async task finishes; then it suspends B. Finally we move on to usb_node_suspend(P), which does down_read(A->lock) and starts an async task for P.

The upshot is that P is stuck waiting for Q to suspend, even though it should have been able to suspend in parallel. This is simply because P precedes B in the list, and B is synchronous and must wait for Q to finish.

With my two-pass algorithm, we start with Q. The first loop does down_read(B->lock) and starts an async task for Q. We move on to B and do down_read(B->parent->lock), nothing more. Then we move to to P, with down_read(A->lock) and start an async task for P. Finally we do down_read(A->parent->lock). Notice that now there are two async tasks, for P and Q, running in parallel.

The second pass waits for Q to finish before suspending B synchronously, and waits for P to finish before suspending A synchronously. This is unavoidable. The point is that it allows P and Q to suspend at the same time, not one after the other as in the one-pass scheme.

Alan Stern

#44 lore raw
Quoted 3 lines · Alan Stern

It only seems that way because you didn't take into account devices that suspend synchronously but whose children suspend asynchronously.

But why would I care? If somebody suspends synchronously, then that's what he wants.

Quoted 2 lines

A synchronous suspend routine for a device with async child suspends would have to look just like your usb_node_suspend():

Sure. But that sounds like a "Doctor, it hurts when I do this" situation. Don't do that.

Make the USB host controller do its suspend asynchronously. We don't suspend PCI bridges anyway, iirc (but I didn't actually check). And at worst, we can make the PCI _bridges_ know about async suspends, and solve it that way - without actually making any normal PCI drivers do it.

Linus

#45 lore raw
In reply to Alan Stern
Quoted 3 lines · Alan Stern

It only seems that way because you didn't take into account devices that suspend synchronously but whose children suspend asynchronously.

But why would I care? If somebody suspends synchronously, then that's what he wants.

Quoted 2 lines

A synchronous suspend routine for a device with async child suspends would have to look just like your usb_node_suspend():

Sure. But that sounds like a "Doctor, it hurts when I do this" situation. Don't do that.

Make the USB host controller do its suspend asynchronously. We don't suspend PCI bridges anyway, iirc (but I didn't actually check). And at worst, we can make the PCI _bridges_ know about async suspends, and solve it that way - without actually making any normal PCI drivers do it.

Linus

#46 lore raw
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · Alan Stern

It only seems that way because you didn't take into account devices that suspend synchronously but whose children suspend asynchronously.

But why would I care? If somebody suspends synchronously, then that's what he wants.

Quoted 8 lines
Quoted 2 lines

A synchronous suspend routine for a device with async child suspends would have to look just like your usb_node_suspend():

Sure. But that sounds like a "Doctor, it hurts when I do this" situation. Don't do that.

Make the USB host controller do its suspend asynchronously. We don't suspend PCI bridges anyway, iirc (but I didn't actually check).

That's correct, we don't.

Rafael

#47 lore raw
In reply to Linus Torvalds
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · Alan Stern

It only seems that way because you didn't take into account devices that suspend synchronously but whose children suspend asynchronously.

But why would I care? If somebody suspends synchronously, then that's what he wants.

Quoted 8 lines
Quoted 2 lines

A synchronous suspend routine for a device with async child suspends would have to look just like your usb_node_suspend():

Sure. But that sounds like a "Doctor, it hurts when I do this" situation. Don't do that.

Make the USB host controller do its suspend asynchronously. We don't suspend PCI bridges anyway, iirc (but I didn't actually check).

That's correct, we don't.

Rafael

#48 lore raw
In reply to Linus Torvalds
Quoted 7 lines · Linus Torvalds
Quoted 3 lines · Alan Stern

It only seems that way because you didn't take into account devices that suspend synchronously but whose children suspend asynchronously.

But why would I care? If somebody suspends synchronously, then that's what he wants.

It doesn't mean he wants to block unrelated devices from suspending asynchronously, merely because they happen to come earlier in the list.

Quoted 10 lines
Quoted 2 lines

A synchronous suspend routine for a device with async child suspends would have to look just like your usb_node_suspend():

Sure. But that sounds like a "Doctor, it hurts when I do this" situation. Don't do that.

Make the USB host controller do its suspend asynchronously. We don't suspend PCI bridges anyway, iirc (but I didn't actually check). And at worst, we can make the PCI _bridges_ know about async suspends, and solve it that way - without actually making any normal PCI drivers do it.

This sounds suspiciously like pushing the problem up a level and hoping it will go away. (Sometimes that even works.)

In the end it isn't a very big issue. Using one vs. two passes in dpm_suspend() is pretty unimportant.

Alan Stern

P.S.: In fact I planned all along to handle USB host controllers asynchronously anyway, since their resume routines contain some long delays. I was merely using them as an example.

#49 lore raw
Quoted 8 lines · Alan Stern
Quoted 5 lines

Make the USB host controller do its suspend asynchronously. We don't suspend PCI bridges anyway, iirc (but I didn't actually check). And at worst, we can make the PCI _bridges_ know about async suspends, and solve it that way - without actually making any normal PCI drivers do it.

This sounds suspiciously like pushing the problem up a level and hoping it will go away. (Sometimes that even works.)

The "we don't suspend bridges anyway" is definitely a "hoping it will go away" issue. I think we did suspend bridges for a short while during the PM switch-over some time ago, and it worked most of the time, and then on some machines it just didn't work at all. Probably because ACPI ends up touching registers behind bridges that we closed down etc.

So PCI bridges are kind of special. Right now we don't touch them, and if we ever do, that will be another issue.

Quoted 2 lines

In the end it isn't a very big issue. Using one vs. two passes in dpm_suspend() is pretty unimportant.

I also suspect that even if you do the USB host controller suspend synchronously, doing the actual USB devices asynchronously would still help - even if it's only "asynchronously per bus" thing.

So in fact, it's probably a good first step to start off doing only the USB devices, not the controller.

Linus

#50 lore raw
In reply to Alan Stern
Quoted 8 lines · Alan Stern
Quoted 5 lines

Make the USB host controller do its suspend asynchronously. We don't suspend PCI bridges anyway, iirc (but I didn't actually check). And at worst, we can make the PCI _bridges_ know about async suspends, and solve it that way - without actually making any normal PCI drivers do it.

This sounds suspiciously like pushing the problem up a level and hoping it will go away. (Sometimes that even works.)

The "we don't suspend bridges anyway" is definitely a "hoping it will go away" issue. I think we did suspend bridges for a short while during the PM switch-over some time ago, and it worked most of the time, and then on some machines it just didn't work at all. Probably because ACPI ends up touching registers behind bridges that we closed down etc.

So PCI bridges are kind of special. Right now we don't touch them, and if we ever do, that will be another issue.

Quoted 2 lines

In the end it isn't a very big issue. Using one vs. two passes in dpm_suspend() is pretty unimportant.

I also suspect that even if you do the USB host controller suspend synchronously, doing the actual USB devices asynchronously would still help - even if it's only "asynchronously per bus" thing.

So in fact, it's probably a good first step to start off doing only the USB devices, not the controller.

Linus

#51 lore raw
Quoted 6 lines · Linus Torvalds

I also suspect that even if you do the USB host controller suspend synchronously, doing the actual USB devices asynchronously would still help - even if it's only "asynchronously per bus" thing.

So in fact, it's probably a good first step to start off doing only the USB devices, not the controller.

Interesting you should say that. The patch I asked Arjan to test involved not suspending USB devices at all (root hubs being the exception). That is in fact just what we do when CONFIG_USB_SUSPEND isn't set.

There's no need to suspend the individual devices when the whole system is going down. They will automatically suspend when the controller stops sending out SOF packets, which occurs when the root hub is suspended. The USB spec describes this, grandiosely, as a "global suspend".

But yes, I agree. Doing just the USB devices is a good first step.

Alan Stern

#52 lore raw
Quoted 6 lines · Alan Stern

There's no need to suspend the individual devices when the whole system is going down. They will automatically suspend when the controller stops sending out SOF packets, which occurs when the root hub is suspended. The USB spec describes this, grandiosely, as a "global suspend".

Ahh, but the sync vs async would then still matter on resume. No?

Linus

#53 lore raw
Quoted 11 lines · Linus Torvalds
Quoted 6 lines · Alan Stern

There's no need to suspend the individual devices when the whole system is going down. They will automatically suspend when the controller stops sending out SOF packets, which occurs when the root hub is suspended. The USB spec describes this, grandiosely, as a "global suspend".

Ahh, but the sync vs async would then still matter on resume. No?

That's complicated. If we assume the devices weren't runtime-suspended before the sleep began, then they would automatically resume themselves when the controller started transmitting EOF packets. So in that case resume would be fast and async wouldn't matter.

But if the devices were runtime-suspended, then what? The safest course is to resume them during the system-wide resume. In that case yes, the sync vs async would matter.

And if (as happens on many machines) the firmware messes up the controller settings during resume, then all the USB devices would have to be reset -- another slow procedure.

Alan Stern

#54 lore raw
In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 6 lines · Alan Stern

There's no need to suspend the individual devices when the whole system is going down. They will automatically suspend when the controller stops sending out SOF packets, which occurs when the root hub is suspended. The USB spec describes this, grandiosely, as a "global suspend".

Ahh, but the sync vs async would then still matter on resume. No?

That's complicated. If we assume the devices weren't runtime-suspended before the sleep began, then they would automatically resume themselves when the controller started transmitting EOF packets. So in that case resume would be fast and async wouldn't matter.

But if the devices were runtime-suspended, then what? The safest course is to resume them during the system-wide resume. In that case yes, the sync vs async would matter.

And if (as happens on many machines) the firmware messes up the controller settings during resume, then all the USB devices would have to be reset -- another slow procedure.

Alan Stern

#55 lore raw
In reply to Alan Stern
Quoted 6 lines · Alan Stern

There's no need to suspend the individual devices when the whole system is going down. They will automatically suspend when the controller stops sending out SOF packets, which occurs when the root hub is suspended. The USB spec describes this, grandiosely, as a "global suspend".

Ahh, but the sync vs async would then still matter on resume. No?

Linus

#56 lore raw
In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds

I also suspect that even if you do the USB host controller suspend synchronously, doing the actual USB devices asynchronously would still help - even if it's only "asynchronously per bus" thing.

So in fact, it's probably a good first step to start off doing only the USB devices, not the controller.

Interesting you should say that. The patch I asked Arjan to test involved not suspending USB devices at all (root hubs being the exception). That is in fact just what we do when CONFIG_USB_SUSPEND isn't set.

There's no need to suspend the individual devices when the whole system is going down. They will automatically suspend when the controller stops sending out SOF packets, which occurs when the root hub is suspended. The USB spec describes this, grandiosely, as a "global suspend".

But yes, I agree. Doing just the USB devices is a good first step.

Alan Stern

#57 lore raw
In reply to Linus Torvalds
Quoted 7 lines · Linus Torvalds
Quoted 3 lines · Alan Stern

It only seems that way because you didn't take into account devices that suspend synchronously but whose children suspend asynchronously.

But why would I care? If somebody suspends synchronously, then that's what he wants.

It doesn't mean he wants to block unrelated devices from suspending asynchronously, merely because they happen to come earlier in the list.

Quoted 10 lines
Quoted 2 lines

A synchronous suspend routine for a device with async child suspends would have to look just like your usb_node_suspend():

Sure. But that sounds like a "Doctor, it hurts when I do this" situation. Don't do that.

Make the USB host controller do its suspend asynchronously. We don't suspend PCI bridges anyway, iirc (but I didn't actually check). And at worst, we can make the PCI _bridges_ know about async suspends, and solve it that way - without actually making any normal PCI drivers do it.

This sounds suspiciously like pushing the problem up a level and hoping it will go away. (Sometimes that even works.)

In the end it isn't a very big issue. Using one vs. two passes in dpm_suspend() is pretty unimportant.

Alan Stern

P.S.: In fact I planned all along to handle USB host controllers asynchronously anyway, since their resume routines contain some long delays. I was merely using them as an example.

#58 lore raw
In reply to Linus Torvalds
Quoted 12 lines · Linus Torvalds
Quoted 3 lines · Alan Stern

It only seems that way because you didn't take into account devices that suspend synchronously but whose children suspend asynchronously.

But why would I care? If somebody suspends synchronously, then that's what he wants.

Quoted 2 lines

A synchronous suspend routine for a device with async child suspends would have to look just like your usb_node_suspend():

Quoted 7 lines

Sure. But that sounds like a "Doctor, it hurts when I do this" situation. Don't do that.

Make the USB host controller do its suspend asynchronously. We don't suspend PCI bridges anyway, iirc (but I didn't actually check). And at worst, we can make the PCI _bridges_ know about async suspends, and solve it that way - without actually making any normal PCI drivers do it.

BTW, I still don't quite understand why not to put the parent's down_write operation into the core. It's not going to hurt for the "synchronous" devices and the "asynchronous" ones will need to do it anyway.

Also it looks like that's something to do unconditionally for all nodes having children, because the parent need not know if the children do async operations.

Rafael

#59 lore raw
Quoted 4 lines · Rafael J. Wysocki

BTW, I still don't quite understand why not to put the parent's down_write operation into the core. It's not going to hurt for the "synchronous" devices and the "asynchronous" ones will need to do it anyway.

That's what I started out doing (see the first pseudo-code with the two phases). But it _does_ actually hurt.

Because it will hurt exactly for the "multiple hubs" case: if you have two USB hubs in parallel (and the case that Alan pointed out about a USB host bridge is the exact same deal), then you want to be able to suspend and resume those two independent hubs in parallel too.

But if you do the "down_write()" synchronously in the core, that means that you are also stopping the whole "traverse the tree" thing - so now you aren't handling the hubs in parallel even if you are handling all the devices _behind_ them asynchronously.

This "serialize while traversing the tree" was what I was initially trying to avoid with the two-phase approach, but that I realized (after writing the resume path) that I could avoid much better by just moving the parents down_write into the asynchronous path.

Quoted 3 lines

Also it looks like that's something to do unconditionally for all nodes having children, because the parent need not know if the children do async operations.

True, and that was (again) the first iteration. But see above: in order to allow way more concurrency, you don't want to introduce the false dependency between the write-lock and the traversal of the tree (or, as Alan points out - just a list - but that doesn't really change anything) that is introduced by taking the lock synchronously.

So by moving the write-lock to the asynchronous work that also shuts down the parent, you avoid that whole unnecessary serialization. But that means that you can't do the lock in generic code.

Unless you want to do _all_ of the async logic in generic code and re-introduce the "dev->async_suspend" flag. I would be ok with that now that the infrastructure seems so simple.

Linus

#60 lore raw
In reply to Rafael J. Wysocki
Quoted 4 lines · Rafael J. Wysocki

BTW, I still don't quite understand why not to put the parent's down_write operation into the core. It's not going to hurt for the "synchronous" devices and the "asynchronous" ones will need to do it anyway.

That's what I started out doing (see the first pseudo-code with the two phases). But it _does_ actually hurt.

Because it will hurt exactly for the "multiple hubs" case: if you have two USB hubs in parallel (and the case that Alan pointed out about a USB host bridge is the exact same deal), then you want to be able to suspend and resume those two independent hubs in parallel too.

But if you do the "down_write()" synchronously in the core, that means that you are also stopping the whole "traverse the tree" thing - so now you aren't handling the hubs in parallel even if you are handling all the devices _behind_ them asynchronously.

This "serialize while traversing the tree" was what I was initially trying to avoid with the two-phase approach, but that I realized (after writing the resume path) that I could avoid much better by just moving the parents down_write into the asynchronous path.

Quoted 3 lines

Also it looks like that's something to do unconditionally for all nodes having children, because the parent need not know if the children do async operations.

True, and that was (again) the first iteration. But see above: in order to allow way more concurrency, you don't want to introduce the false dependency between the write-lock and the traversal of the tree (or, as Alan points out - just a list - but that doesn't really change anything) that is introduced by taking the lock synchronously.

So by moving the write-lock to the asynchronous work that also shuts down the parent, you avoid that whole unnecessary serialization. But that means that you can't do the lock in generic code.

Unless you want to do _all_ of the async logic in generic code and re-introduce the "dev->async_suspend" flag. I would be ok with that now that the infrastructure seems so simple.

Linus

#61 lore raw
Quoted 9 lines · Linus Torvalds
Quoted 4 lines · Rafael J. Wysocki

BTW, I still don't quite understand why not to put the parent's down_write operation into the core. It's not going to hurt for the "synchronous" devices and the "asynchronous" ones will need to do it anyway.

That's what I started out doing (see the first pseudo-code with the two phases). But it _does_ actually hurt.

Hmm. If no one calls down_read() on the "synchronous" devices, their down_write()s will be nops. In turn, if somebody does call down_read(), it means they really need to wait for someone. They presumably don't need to wait for each other, but we don't really know that (otherwise they would have been "asynchronous").

Quoted 10 lines

Because it will hurt exactly for the "multiple hubs" case: if you have two USB hubs in parallel (and the case that Alan pointed out about a USB host bridge is the exact same deal), then you want to be able to suspend and resume those two independent hubs in parallel too.

But if you do the "down_write()" synchronously in the core, that means that you are also stopping the whole "traverse the tree" thing - so now you aren't handling the hubs in parallel even if you are handling all the devices _behind_ them asynchronously.

Quoted 4 lines

This "serialize while traversing the tree" was what I was initially trying to avoid with the two-phase approach, but that I realized (after writing the resume path) that I could avoid much better by just moving the parents down_write into the asynchronous path.

But the asynchronous path has to be started somewhere. Basically, there are three possible places: the core itself, the bus type's suspend routine called by the core (same goes for resume of course), and the device driver's suspend routine called by the bus type.

Now, I don't really see how we can put the the parent's down_write() in a child's suspend routine, for multiple reasons (one of them being that there can be multiple asynchronous children the parent needs to wait for), so it looks like it needs to be above the driver's suspend.

However, the parent can be on a different bus type than the children, so it looks like we can only start the asynchronous path at the core level.

Quoted 10 lines
Quoted 3 lines

Also it looks like that's something to do unconditionally for all nodes having children, because the parent need not know if the children do async operations.

True, and that was (again) the first iteration. But see above: in order to allow way more concurrency, you don't want to introduce the false dependency between the write-lock and the traversal of the tree (or, as Alan points out - just a list - but that doesn't really change anything) that is introduced by taking the lock synchronously.

Quoted 6 lines

So by moving the write-lock to the asynchronous work that also shuts down the parent, you avoid that whole unnecessary serialization. But that means that you can't do the lock in generic code.

Unless you want to do _all_ of the async logic in generic code and re-introduce the "dev->async_suspend" flag.

Quite frankly, I would like to.

Quoted 1 line

I would be ok with that now that the infrastructure seems so simple.

Well, perhaps I should dig out my original async suspend/resume patches that didn't contain all of the non-essential stuff and post them here for discussion, after all ...

Rafael

#62 lore raw
Quoted 2 lines · Rafael J. Wysocki

However, the parent can be on a different bus type than the children, so it looks like we can only start the asynchronous path at the core level.

Agreed.

Quoted 10 lines
Quoted 2 lines

Unless you want to do _all_ of the async logic in generic code and re-introduce the "dev->async_suspend" flag.

Quite frankly, I would like to.

Quoted 1 line

I would be ok with that now that the infrastructure seems so simple.

Well, perhaps I should dig out my original async suspend/resume patches that didn't contain all of the non-essential stuff and post them here for discussion, after all ...

That seems like a very good idea. IIRC they were quite similar to what we have been discussing.

Alan Stern

#63 lore raw

Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 12 lines · Alan Stern
Quoted 2 lines · Rafael J. Wysocki

However, the parent can be on a different bus type than the children, so it looks like we can only start the asynchronous path at the core level.

Agreed.

Quoted 5 lines
Quoted 2 lines

Unless you want to do _all_ of the async logic in generic code and re-introduce the "dev->async_suspend" flag.

Quite frankly, I would like to.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

I would be ok with that now that the infrastructure seems so simple.

Well, perhaps I should dig out my original async suspend/resume patches that didn't contain all of the non-essential stuff and post them here for discussion, after all ...

That seems like a very good idea. IIRC they were quite similar to what we have been discussing.

There you go.

Below is the resume part. I have reworked the original patch a bit so that it's even simpler. I'll post the suspend part in a reply to this message.

The idea is basically that if a device has the power.async_suspend flag set, we schedule the execution of it's resume callback asynchronously, but we wait for the device's parent to finish resume before the device's suspend is actually executed.

The wait queue plus the op_complete flag combo plays the role of the locking in the Linus' picture, and it's essentially equivalent, since the devices being waited for during resume will have to wait during suspend, so for example if A has to wait for B during suspend, then B will have to wait for A during resume (thus they both need to know in advance who's going to wait for them and whom they need to wait for).

Of course, the code in this patch has the problem that if there are two "asynchronous" devices in dpm_list separated by a series of "synchronous" devices, then they usually won't be resumed in parallel (which is what we ultimately want). That can be optimised in a couple of ways, but such optimisations add quite some details to the code, so let's just omit them for now.

BTW, thanks to the discussion with Linus I've realized that the off-tree dependences may be (relatively easily) taken into account by making the interested drivers directly execute dpm_wait() for the extra devices they need to wait for, so the entire PM links thing is simply unnecessary. So it looks like the only thing this patch is missing are the optimisations mentioned above.

[This version of the patch has only been slightly tested.]

---
 drivers/base/power/main.c    |  129 +++++++++++++++++++++++++++++++++++++++----
 include/linux/device.h       |    6 ++
 include/linux/pm.h           |    4 +
 include/linux/resume-trace.h |    7 ++
 4 files changed, 134 insertions(+), 12 deletions(-)
include/linux/pm.h +3 -1
Index: linux-2.6/include/linux/pm.h
===================================================================
@@ -412,15 +412,17 @@ struct dev_pm_info {
 	pm_message_t		power_state;
 	unsigned int		can_wakeup:1;
 	unsigned int		should_wakeup:1;
+	unsigned		async_suspend:1;
 	enum dpm_state		status;		/* Owned by the PM core */
+	wait_queue_head_t	wait_queue;
 #ifdef CONFIG_PM_SLEEP
 	struct list_head	entry;
+	unsigned int            op_complete:1;
 #endif
 #ifdef CONFIG_PM_RUNTIME
 	struct timer_list	suspend_timer;
 	unsigned long		timer_expires;
 	struct work_struct	work;
-	wait_queue_head_t	wait_queue;
 	spinlock_t		lock;
 	atomic_t		usage_count;
 	atomic_t		child_count;
include/linux/device.h +6 -0
Index: linux-2.6/include/linux/device.h
===================================================================
@@ -472,6 +472,12 @@ static inline int device_is_registered(s
 	return dev->kobj.state_in_sysfs;
 }
 
+static inline void device_enable_async_suspend(struct device *dev, bool enable)
+{
+	if (dev->power.status == DPM_ON)
+		dev->power.async_suspend = enable;
+}
+
 void driver_init(void);
 
 /*
drivers/base/power/main.c +118 -11

231 lines not on this page · raw message

include/linux/resume-trace.h +7 -0
Index: linux-2.6/include/linux/resume-trace.h
===================================================================
@@ -6,6 +6,11 @@
 
 extern int pm_trace_enabled;
 
+static inline int pm_trace_is_enabled(void)
+{
+       return pm_trace_enabled;
+}
+
 struct device;
 extern void set_trace_device(struct device *);
 extern void generate_resume_trace(const void *tracedata, unsigned int user);
@@ -17,6 +22,8 @@ extern void generate_resume_trace(const 
 
 #else
 
+static inline int pm_trace_is_enabled(void) { return 0; }
+
 #define TRACE_DEVICE(dev) do { } while (0)
 #define TRACE_RESUME(dev) do { } while (0)
 
#64 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 15 lines · Rafael J. Wysocki
Quoted 14 lines · Alan Stern
Quoted 2 lines · Rafael J. Wysocki

However, the parent can be on a different bus type than the children, so it looks like we can only start the asynchronous path at the core level.

Agreed.

Quoted 7 lines
Quoted 2 lines

Unless you want to do _all_ of the async logic in generic code and re-introduce the "dev->async_suspend" flag.

Quite frankly, I would like to.

Quoted 1 line

I would be ok with that now that the infrastructure seems so simple.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Well, perhaps I should dig out my original async suspend/resume patches that didn't contain all of the non-essential stuff and post them here for discussion, after all ...

That seems like a very good idea. IIRC they were quite similar to what we have been discussing.

There you go.

Below is the suspend part. It contains some extra code for rolling back the suspend if one of the asynchronous callbacks returns error code, but apart from this it's completely analogous to the resume part.

[This patch has only been slightly tested.]

---
 drivers/base/power/main.c |  113 ++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 104 insertions(+), 9 deletions(-)
drivers/base/power/main.c +104 -9
Index: linux-2.6/drivers/base/power/main.c
===================================================================
@@ -202,6 +202,17 @@ static void dpm_wait(struct device *dev,
                wait_event(dev->power.wait_queue, !!dev->power.op_complete);
 }
 
+static int device_pm_wait_fn(struct device *dev, void *async_ptr)
+{
+	dpm_wait(dev, *((bool *)async_ptr));
+	return 0;
+}
+
+static void dpm_wait_for_children(struct device *dev, bool async)
+{
+	device_for_each_child(dev, &async, device_pm_wait_fn);
+}
+
 /**
  * dpm_synchronize - Wait for PM callbacks of all devices to complete.
  */
@@ -638,6 +649,8 @@ static void dpm_complete(pm_message_t st
 	mutex_unlock(&dpm_list_mtx);
 }
 
+static int async_error;
+
 /**
  * dpm_resume_end - Execute "resume" callbacks and complete system transition.
  * @state: PM transition of the system being carried out.
@@ -685,20 +698,52 @@ static pm_message_t resume_event(pm_mess
  * The driver of @dev will not receive interrupts while this function is being
  * executed.
  */
-static int device_suspend_noirq(struct device *dev, pm_message_t state)
+static int __device_suspend_noirq(struct device *dev, pm_message_t state)
 {
 	int error = 0;
 
-	if (!dev->bus)
-		return 0;
-
-	if (dev->bus->pm) {
+	if (dev->bus && dev->bus->pm) {
 		pm_dev_dbg(dev, state, "LATE ");
 		error = pm_noirq_op(dev, dev->bus->pm, state);
 	}
+
+	dpm_finish(dev);
+
 	return error;
 }
 
+static void async_suspend_noirq(void *data, async_cookie_t cookie)
+{
+	struct device *dev = (struct device *)data;
+	int error = async_error;
+
+	if (error)
+		return;
+
+	dpm_wait_for_children(dev, true);
+	error = __device_suspend_noirq(dev, pm_transition);
+	if (error) {
+		pm_dev_err(dev, pm_transition, " async LATE", error);
+		dev->power.status = DPM_OFF;
+	}
+	put_device(dev);
+
+	if (error && !async_error)
+		async_error = error;
+}
+
+static int device_suspend_noirq(struct device *dev)
+{
+	if (dev->power.async_suspend) {
+		get_device(dev);
+		async_schedule(async_suspend_noirq, dev);
+		return 0;
+	}
+
+	dpm_wait_for_children(dev, false);
+	return __device_suspend_noirq(dev, pm_transition);
+}
+
 /**
  * dpm_suspend_noirq - Execute "late suspend" callbacks for non-sysdev devices.
  * @state: PM transition of the system being carried out.
@@ -713,14 +758,21 @@ int dpm_suspend_noirq(pm_message_t state
 
 	suspend_device_irqs();
 	mutex_lock(&dpm_list_mtx);
+	pm_transition = state;
 	list_for_each_entry_reverse(dev, &dpm_list, power.entry) {
-		error = device_suspend_noirq(dev, state);
+		dev->power.status = DPM_OFF_IRQ;
+		error = device_suspend_noirq(dev);
 		if (error) {
 			pm_dev_err(dev, state, " late", error);
+			dev->power.status = DPM_OFF;
+			break;
+		}
+		if (async_error) {
+			error = async_error;
 			break;
 		}
-		dev->power.status = DPM_OFF_IRQ;
 	}
+	dpm_synchronize();
 	mutex_unlock(&dpm_list_mtx);
 	if (error)
 		dpm_resume_noirq(resume_event(state));
@@ -733,7 +785,7 @@ EXPORT_SYMBOL_GPL(dpm_suspend_noirq);
  * @dev: Device to handle.
  * @state: PM transition of the system being carried out.
  */
-static int device_suspend(struct device *dev, pm_message_t state)
+static int __device_suspend(struct device *dev, pm_message_t state)
 {
 	int error = 0;
 
@@ -773,10 +825,45 @@ static int device_suspend(struct device 
 	}
  End:
 	up(&dev->sem);
+	dpm_finish(dev);
 
 	return error;
 }
 
+static void async_suspend(void *data, async_cookie_t cookie)
+{
+	struct device *dev = (struct device *)data;
+	int error = async_error;
+
+	if (error)
+		goto End;
+
+	dpm_wait_for_children(dev, true);
+	error = __device_suspend(dev, pm_transition);
+	if (error) {
+		pm_dev_err(dev, pm_transition, " async", error);
+
+		dev->power.status = DPM_SUSPENDING;
+		if (!async_error)
+			async_error = error;
+	}
+
+ End:
+	put_device(dev);
+}
+
+static int device_suspend(struct device *dev, pm_message_t state)
+{
+	if (dev->power.async_suspend) {
+		get_device(dev);
+		async_schedule(async_suspend, dev);
+		return 0;
+	}
+
+	dpm_wait_for_children(dev, false);
+	return __device_suspend(dev, pm_transition);
+}
+
 /**
  * dpm_suspend - Execute "suspend" callbacks for all non-sysdev devices.
  * @state: PM transition of the system being carried out.
@@ -788,10 +875,12 @@ static int dpm_suspend(pm_message_t stat
 
 	INIT_LIST_HEAD(&list);
 	mutex_lock(&dpm_list_mtx);
+	pm_transition = state;
 	while (!list_empty(&dpm_list)) {
 		struct device *dev = to_device(dpm_list.prev);
 
 		get_device(dev);
+		dev->power.status = DPM_OFF;
 		mutex_unlock(&dpm_list_mtx);
 
 		error = device_suspend(dev, state);
@@ -799,16 +888,21 @@ static int dpm_suspend(pm_message_t stat
 		mutex_lock(&dpm_list_mtx);
 		if (error) {
 			pm_dev_err(dev, state, "", error);
+			dev->power.status = DPM_SUSPENDING;
 			put_device(dev);
 			break;
 		}
-		dev->power.status = DPM_OFF;
 		if (!list_empty(&dev->power.entry))
 			list_move(&dev->power.entry, &list);
 		put_device(dev);
+		if (async_error)
+			break;
 	}
 	list_splice(&list, dpm_list.prev);
+	dpm_synchronize();
 	mutex_unlock(&dpm_list_mtx);
+	if (!error)
+		error = async_error;
 	return error;
 }
 
@@ -867,6 +961,7 @@ static int dpm_prepare(pm_message_t stat
 	INIT_LIST_HEAD(&list);
 	mutex_lock(&dpm_list_mtx);
 	transition_started = true;
+	async_error = 0;
 	while (!list_empty(&dpm_list)) {
 		struct device *dev = to_device(dpm_list.next);
 
#65 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Rafael J. Wysocki
Quoted 15 lines · Rafael J. Wysocki
Quoted 14 lines · Alan Stern
Quoted 2 lines · Rafael J. Wysocki

However, the parent can be on a different bus type than the children, so it looks like we can only start the asynchronous path at the core level.

Agreed.

Quoted 7 lines
Quoted 2 lines

Unless you want to do _all_ of the async logic in generic code and re-introduce the "dev->async_suspend" flag.

Quite frankly, I would like to.

Quoted 1 line

I would be ok with that now that the infrastructure seems so simple.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Well, perhaps I should dig out my original async suspend/resume patches that didn't contain all of the non-essential stuff and post them here for discussion, after all ...

That seems like a very good idea. IIRC they were quite similar to what we have been discussing.

There you go.

Below is the suspend part. It contains some extra code for rolling back the suspend if one of the asynchronous callbacks returns error code, but apart from this it's completely analogous to the resume part.

[This patch has only been slightly tested.]

---
 drivers/base/power/main.c |  113 ++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 104 insertions(+), 9 deletions(-)
drivers/base/power/main.c +104 -9
Index: linux-2.6/drivers/base/power/main.c
===================================================================
@@ -202,6 +202,17 @@ static void dpm_wait(struct device *dev,
                wait_event(dev->power.wait_queue, !!dev->power.op_complete);
 }
 
+static int device_pm_wait_fn(struct device *dev, void *async_ptr)
+{
+	dpm_wait(dev, *((bool *)async_ptr));
+	return 0;
+}
+
+static void dpm_wait_for_children(struct device *dev, bool async)
+{
+	device_for_each_child(dev, &async, device_pm_wait_fn);
+}
+
 /**
  * dpm_synchronize - Wait for PM callbacks of all devices to complete.
  */
@@ -638,6 +649,8 @@ static void dpm_complete(pm_message_t st
 	mutex_unlock(&dpm_list_mtx);
 }
 
+static int async_error;
+
 /**
  * dpm_resume_end - Execute "resume" callbacks and complete system transition.
  * @state: PM transition of the system being carried out.
@@ -685,20 +698,52 @@ static pm_message_t resume_event(pm_mess
  * The driver of @dev will not receive interrupts while this function is being
  * executed.
  */
-static int device_suspend_noirq(struct device *dev, pm_message_t state)
+static int __device_suspend_noirq(struct device *dev, pm_message_t state)
 {
 	int error = 0;
 
-	if (!dev->bus)
-		return 0;
-
-	if (dev->bus->pm) {
+	if (dev->bus && dev->bus->pm) {
 		pm_dev_dbg(dev, state, "LATE ");
 		error = pm_noirq_op(dev, dev->bus->pm, state);
 	}
+
+	dpm_finish(dev);
+
 	return error;
 }
 
+static void async_suspend_noirq(void *data, async_cookie_t cookie)
+{
+	struct device *dev = (struct device *)data;
+	int error = async_error;
+
+	if (error)
+		return;
+
+	dpm_wait_for_children(dev, true);
+	error = __device_suspend_noirq(dev, pm_transition);
+	if (error) {
+		pm_dev_err(dev, pm_transition, " async LATE", error);
+		dev->power.status = DPM_OFF;
+	}
+	put_device(dev);
+
+	if (error && !async_error)
+		async_error = error;
+}
+
+static int device_suspend_noirq(struct device *dev)
+{
+	if (dev->power.async_suspend) {
+		get_device(dev);
+		async_schedule(async_suspend_noirq, dev);
+		return 0;
+	}
+
+	dpm_wait_for_children(dev, false);
+	return __device_suspend_noirq(dev, pm_transition);
+}
+
 /**
  * dpm_suspend_noirq - Execute "late suspend" callbacks for non-sysdev devices.
  * @state: PM transition of the system being carried out.
@@ -713,14 +758,21 @@ int dpm_suspend_noirq(pm_message_t state
 
 	suspend_device_irqs();
 	mutex_lock(&dpm_list_mtx);
+	pm_transition = state;
 	list_for_each_entry_reverse(dev, &dpm_list, power.entry) {
-		error = device_suspend_noirq(dev, state);
+		dev->power.status = DPM_OFF_IRQ;
+		error = device_suspend_noirq(dev);
 		if (error) {
 			pm_dev_err(dev, state, " late", error);
+			dev->power.status = DPM_OFF;
+			break;
+		}
+		if (async_error) {
+			error = async_error;
 			break;
 		}
-		dev->power.status = DPM_OFF_IRQ;
 	}
+	dpm_synchronize();
 	mutex_unlock(&dpm_list_mtx);
 	if (error)
 		dpm_resume_noirq(resume_event(state));
@@ -733,7 +785,7 @@ EXPORT_SYMBOL_GPL(dpm_suspend_noirq);
  * @dev: Device to handle.
  * @state: PM transition of the system being carried out.
  */
-static int device_suspend(struct device *dev, pm_message_t state)
+static int __device_suspend(struct device *dev, pm_message_t state)
 {
 	int error = 0;
 
@@ -773,10 +825,45 @@ static int device_suspend(struct device 
 	}
  End:
 	up(&dev->sem);
+	dpm_finish(dev);
 
 	return error;
 }
 
+static void async_suspend(void *data, async_cookie_t cookie)
+{
+	struct device *dev = (struct device *)data;
+	int error = async_error;
+
+	if (error)
+		goto End;
+
+	dpm_wait_for_children(dev, true);
+	error = __device_suspend(dev, pm_transition);
+	if (error) {
+		pm_dev_err(dev, pm_transition, " async", error);
+
+		dev->power.status = DPM_SUSPENDING;
+		if (!async_error)
+			async_error = error;
+	}
+
+ End:
+	put_device(dev);
+}
+
+static int device_suspend(struct device *dev, pm_message_t state)
+{
+	if (dev->power.async_suspend) {
+		get_device(dev);
+		async_schedule(async_suspend, dev);
+		return 0;
+	}
+
+	dpm_wait_for_children(dev, false);
+	return __device_suspend(dev, pm_transition);
+}
+
 /**
  * dpm_suspend - Execute "suspend" callbacks for all non-sysdev devices.
  * @state: PM transition of the system being carried out.
@@ -788,10 +875,12 @@ static int dpm_suspend(pm_message_t stat
 
 	INIT_LIST_HEAD(&list);
 	mutex_lock(&dpm_list_mtx);
+	pm_transition = state;
 	while (!list_empty(&dpm_list)) {
 		struct device *dev = to_device(dpm_list.prev);
 
 		get_device(dev);
+		dev->power.status = DPM_OFF;
 		mutex_unlock(&dpm_list_mtx);
 
 		error = device_suspend(dev, state);
@@ -799,16 +888,21 @@ static int dpm_suspend(pm_message_t stat
 		mutex_lock(&dpm_list_mtx);
 		if (error) {
 			pm_dev_err(dev, state, "", error);
+			dev->power.status = DPM_SUSPENDING;
 			put_device(dev);
 			break;
 		}
-		dev->power.status = DPM_OFF;
 		if (!list_empty(&dev->power.entry))
 			list_move(&dev->power.entry, &list);
 		put_device(dev);
+		if (async_error)
+			break;
 	}
 	list_splice(&list, dpm_list.prev);
+	dpm_synchronize();
 	mutex_unlock(&dpm_list_mtx);
+	if (!error)
+		error = async_error;
 	return error;
 }
 
@@ -867,6 +961,7 @@ static int dpm_prepare(pm_message_t stat
 	INIT_LIST_HEAD(&list);
 	mutex_lock(&dpm_list_mtx);
 	transition_started = true;
+	async_error = 0;
 	while (!list_empty(&dpm_list)) {
 		struct device *dev = to_device(dpm_list.next);
 
#66 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Rafael J. Wysocki
Quoted 3 lines · Rafael J. Wysocki

The wait queue plus the op_complete flag combo plays the role of the locking in the Linus' picture

Please just use the lock. Don't make up your own locking crap. Really.

Your patch is horrible. Exactly because your locking is horribly mis-designed. You can't say things are complete from an interrupt, for example, since you made it some random bitfield, which has unknown characteristics (ie non-atomic read-modify-write etc).

The fact is, any time anybody makes up a new locking mechanism, THEY ALWAYS GET IT WRONG. Don't do it.

I suggested using the rwsem locking for a good reason. It made sense. It was simpler. Just do it that way, stop making up crap.

Linus

#67 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Rafael J. Wysocki
Quoted 3 lines · Rafael J. Wysocki

The wait queue plus the op_complete flag combo plays the role of the locking in the Linus' picture

Please just use the lock. Don't make up your own locking crap. Really.

Your patch is horrible. Exactly because your locking is horribly mis-designed. You can't say things are complete from an interrupt, for example, since you made it some random bitfield, which has unknown characteristics (ie non-atomic read-modify-write etc).

The fact is, any time anybody makes up a new locking mechanism, THEY ALWAYS GET IT WRONG. Don't do it.

I suggested using the rwsem locking for a good reason. It made sense. It was simpler. Just do it that way, stop making up crap.

Linus

#68 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 7 lines · Linus Torvalds

Please just use the lock. Don't make up your own locking crap. Really.

Your patch is horrible. Exactly because your locking is horribly mis-designed. You can't say things are complete from an interrupt, for example, since you made it some random bitfield, which has unknown characteristics (ie non-atomic read-modify-write etc).

Quoted 5 lines

The fact is, any time anybody makes up a new locking mechanism, THEY ALWAYS GET IT WRONG. Don't do it.

I suggested using the rwsem locking for a good reason. It made sense. It was simpler. Just do it that way, stop making up crap.

The semantics needed for this kind of lock aren't really the same as for an rwsem (although obviously an rwsem will do the job). Basically it needs to have the capability for multiple users to lock it (no blocking when acquiring a lock) and the capability for a user to wait until it is totally unlocked. It could be implemented trivially using an atomic_t counter and a waitqueue head.

Is this a standard sort of lock? It's a lot simpler than most others. I don't recall seeing anything quite like it anywhere; the closest thing might be some kind of barrier.

Alan Stern

#69 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 9 lines · Alan Stern

The semantics needed for this kind of lock aren't really the same as for an rwsem (although obviously an rwsem will do the job). Basically it needs to have the capability for multiple users to lock it (no blocking when acquiring a lock) and the capability for a user to wait until it is totally unlocked. It could be implemented trivially using an atomic_t counter and a waitqueue head.

Is this a standard sort of lock?

Yes it is.

It's called a rwlock. The counter is for readers, the exclusion is for writers.

Really.

And the thing is, you actually do want the rwlock semantics, because on the resume side you want the parent to lock it for writing first (so that the children can wait for the parent to have completed its resume.

So we actually _want_ the full rwlock semantics.

See the code I posted earlier. Here condensed into one email:

- resume:

        usb_node_resume(node)
        {
		// Wait for parent to finish resume
                down_read(node->parent->lock);
		// .. before resuming outselves
                node->resume(node)

		// Now we're all done
                up_read(node->parent->lock);
                up_write(node->lock);
        }

	/* caller: */
	..
	// This won't block, because we resume parents before children,
	// and the children will take the read lock. 
        down_write(leaf->lock);
	// Do the blocking part asynchronously
        async_schedule(usb_node_resume, leaf);
	..

- suspend:

        usb_node_suspend(node)
        {
                // Start our suspend. This will block if we have any
                // children that are still busy suspending (they will
                // have done a down_read() in their suspend).
                down_write(node->lock);
                node->suspend(node);
                up_write(node->lock);

                // This lets our parent continue
                up_read(node->parent->lock);
        }

	/* caller: */

	// This won't block, because we suspend nodes before parents
        down_read(node->parent->lock);
	// Do the part that may block asynchronously
	async_schedule(do_usb_node_suspend, node);

It really should be that simple. Nothing more, nothing less. And with the above, finishing the suspend (or resume) from interrupts is fine, and you don't have any new lock that has undefined memory ordering issues etc.

Linus

#70 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 9 lines · Alan Stern

The semantics needed for this kind of lock aren't really the same as for an rwsem (although obviously an rwsem will do the job). Basically it needs to have the capability for multiple users to lock it (no blocking when acquiring a lock) and the capability for a user to wait until it is totally unlocked. It could be implemented trivially using an atomic_t counter and a waitqueue head.

Is this a standard sort of lock?

Yes it is.

It's called a rwlock. The counter is for readers, the exclusion is for writers.

Really.

And the thing is, you actually do want the rwlock semantics, because on the resume side you want the parent to lock it for writing first (so that the children can wait for the parent to have completed its resume.

So we actually _want_ the full rwlock semantics.

See the code I posted earlier. Here condensed into one email:

- resume:

        usb_node_resume(node)
        {
		// Wait for parent to finish resume
                down_read(node->parent->lock);
		// .. before resuming outselves
                node->resume(node)

		// Now we're all done
                up_read(node->parent->lock);
                up_write(node->lock);
        }

	/* caller: */
	..
	// This won't block, because we resume parents before children,
	// and the children will take the read lock. 
        down_write(leaf->lock);
	// Do the blocking part asynchronously
        async_schedule(usb_node_resume, leaf);
	..

- suspend:

        usb_node_suspend(node)
        {
                // Start our suspend. This will block if we have any
                // children that are still busy suspending (they will
                // have done a down_read() in their suspend).
                down_write(node->lock);
                node->suspend(node);
                up_write(node->lock);

                // This lets our parent continue
                up_read(node->parent->lock);
        }

	/* caller: */

	// This won't block, because we suspend nodes before parents
        down_read(node->parent->lock);
	// Do the part that may block asynchronously
	async_schedule(do_usb_node_suspend, node);

It really should be that simple. Nothing more, nothing less. And with the above, finishing the suspend (or resume) from interrupts is fine, and you don't have any new lock that has undefined memory ordering issues etc.

Linus

#71 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 16 lines · Linus Torvalds
Quoted 9 lines · Alan Stern

The semantics needed for this kind of lock aren't really the same as for an rwsem (although obviously an rwsem will do the job). Basically it needs to have the capability for multiple users to lock it (no blocking when acquiring a lock) and the capability for a user to wait until it is totally unlocked. It could be implemented trivially using an atomic_t counter and a waitqueue head.

Is this a standard sort of lock?

Yes it is.

It's called a rwlock. The counter is for readers, the exclusion is for writers.

Quoted 7 lines

Really.

And the thing is, you actually do want the rwlock semantics, because on the resume side you want the parent to lock it for writing first (so that the children can wait for the parent to have completed its resume.

So we actually _want_ the full rwlock semantics.

I'm not convinced. Condense the description a little farther:

Suspend: Children lock the parent first. When they are
finished they unlock the parent, allowing it to proceed.

Resume: Parent locks itself first. When it is finished
it unlocks itself, allowing the children to proceed.

The whole readers vs. writers thing is a non-sequitur. (For instance, this never uses the fact that writers exclude each other.) In each case a lock is taken and eventually released, allowing someone else to stop waiting and move forward. In the suspend case we have multiple lockers and one waiter, whereas in the resume case we have one locker and multiple waiters.

The simplest generalization is to allow both multiple lockers and multiple waiters. Call it a waitlock, for want of a better name:

	wait_lock(wl)
	{
		atomic_inc(&wl->count);
	}

	wait_unlock(wl)
	{
		if (atomic_dec_and_test(&wl->count)) {
			smp_mb__after_atomic_dec();
			wake_up_all(wl->wqh);
		}
	}

	wait_for_lock(wl)
	{
		wait_event(wl->wqh, atomic_read(&wl->count) == 0);
		smp_rmb();
	}

Note that both wait_lock() and wait_unlock() can be called in_interrupt.

Quoted 16 lines

See the code I posted earlier. Here condensed into one email:

- resume:

        usb_node_resume(node)
        {
		// Wait for parent to finish resume
                down_read(node->parent->lock);
		// .. before resuming outselves
                node->resume(node)

		// Now we're all done
                up_read(node->parent->lock);
                up_write(node->lock);
        }
Quoted 8 lines
	/* caller: */
	..
	// This won't block, because we resume parents before children,
	// and the children will take the read lock. 
        down_write(leaf->lock);
	// Do the blocking part asynchronously
        async_schedule(usb_node_resume, leaf);
	..

This becomes:

	usb_node_resume(node)
	{
		// Wait for parent to finish resume
		wait_for_lock(node->parent->lock);
		// .. before resuming outselves
                node->resume(node)

		// Now we're all done
		wait_unlock(node->lock);
	}

	/* caller: */
	..
	// This can't block, because wait_lock() is non-blocking.
	wait_lock(node->lock);
	// Do the blocking part asynchronously
	async_schedule(usb_node_resume, leaf);
	..
Quoted 13 lines

- suspend:

        usb_node_suspend(node)
        {
                // Start our suspend. This will block if we have any
                // children that are still busy suspending (they will
                // have done a down_read() in their suspend).
                down_write(node->lock);
                node->suspend(node);
                up_write(node->lock);

                // This lets our parent continue
                up_read(node->parent->lock);
Quoted 8 lines
        }

	/* caller: */

	// This won't block, because we suspend nodes before parents
        down_read(node->parent->lock);
	// Do the part that may block asynchronously
	async_schedule(do_usb_node_suspend, node);
	usb_node_suspend(node)
	{
		// Start our suspend.  This will block if we have any
		// children that are still busy suspending (they will
		// have done a wait_lock() in their suspend).
		wait_for_lock(node->lock);
		node->suspend(node);

		// This lets our parent continue
		wait_unlock(node->parent->lock);
	}

	/* caller: */
	..
	// This can't block, because wait_lock is non-blocking.
	wait_lock(node->parent->lock);
	// Do the part that may block asynchronously
	async_schedule(do_usb_node_suspend, node);
	..
Quoted 3 lines

It really should be that simple. Nothing more, nothing less. And with the above, finishing the suspend (or resume) from interrupts is fine, and you don't have any new lock that has undefined memory ordering issues etc.

Aren't waitlocks simpler than rwsems? Not as generally useful, perhaps. But just as correct in this situation.

Alan Stern

#72 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 11 lines · Alan Stern
Quoted 2 lines

So we actually _want_ the full rwlock semantics.

I'm not convinced. Condense the description a little farther:

Suspend: Children lock the parent first. When they are
finished they unlock the parent, allowing it to proceed.

Resume: Parent locks itself first. When it is finished
it unlocks itself, allowing the children to proceed.

Yes. You can implement it with a simple lock with a count. Nobody debates that.

But a simple counting lock _is_ a rwlock. Really. They are 100% semantically equivalent. There is no difference.

Quoted 1 line

The whole readers vs. writers thing is a non-sequitur.

No it's not.

It's a 100% equivalent problem. It's purely a change of wording. The end result is the same.

Quoted 2 lines

The simplest generalization is to allow both multiple lockers and multiple waiters. Call it a waitlock, for want of a better name:

But we have that. It _has_ a better name: rwlocks.

And the reason the name is better is because now the name describes all the semantics to anybody who has ever taken a course in operating systems or in parallelism.

It's also a better implementation, because it actually _works_.

Quoted 19 lines
	wait_lock(wl)
	{
		atomic_inc(&wl->count);
	}

	wait_unlock(wl)
	{
		if (atomic_dec_and_test(&wl->count)) {
			smp_mb__after_atomic_dec();
			wake_up_all(wl->wqh);
		}
	}

	wait_for_lock(wl)
	{
		wait_event(wl->wqh, atomic_read(&wl->count) == 0);
		smp_rmb();
	}
Quoted 2 lines

Note that both wait_lock() and wait_unlock() can be called in_interrupt.

And note how even though you sprinkled random memory barriers around, you still got it wrong.

So you just implemented a buggy lock, and for what gain? Tell me exactly why your buggy lock (assuming you'd know enough about memory ordering to actually fix it) is better than just using the existing one?

It's certainly not smaller. It's not faster. It doesn't have support for lockdep. And it's BUGGY.

Really. Tell me why you want to re-implement an existing lock - badly.

[ Hint: you need a smp_mb() *before* the atomic_dec() in wait-unlock, so
that anybody else who sees the new value will be guaranteed to have seen anything else the unlocker did.

You also need a smp_mb() in the wait_for_lock(), not a smp_rmb(). Can't allow writes to migrate up either. 'atomic_read()' does not imply any barriers.

But most architectures can optimize these things for their particular memory ordering model, and do so in their rwsem implementation. ]

Quoted 13 lines

This becomes:

	usb_node_resume(node)
	{
		// Wait for parent to finish resume
		wait_for_lock(node->parent->lock);
		// .. before resuming outselves
                node->resume(node)

		// Now we're all done
		wait_unlock(node->lock);
	}
Quoted 7 lines
	/* caller: */
	..
	// This can't block, because wait_lock() is non-blocking.
	wait_lock(node->lock);
	// Do the blocking part asynchronously
	async_schedule(usb_node_resume, leaf);
	..

Umm? Same thing, different words?

That "wait_for_lock()" is equivalent to a 'read_lock()+read_unlock()'. We _could_ expose such a mechanism for rwsem's too, but why do it? It's actually nicer to use a real read-lock - and do it _around_ the operation, because now the locking also automatically gets things like overlapping suspends and resumes right.

(Which you'd obviously hope never happens, but it's nice from a conceptual standpoint to know that the locking is robust).

Quoted 2 lines

Aren't waitlocks simpler than rwsems? Not as generally useful, perhaps. But just as correct in this situation.

NO!

Dammit. I started this whole rant with this comment to Rafael:

"The fact is, any time anybody makes up a new locking mechanism, THEY
ALWAYS GET IT WRONG. Don't do it."

Take heed. You got it wrong. Admit it. Locking is _hard_. SMP memory ordering is HARD.

So leave locking to the pro's. They _also_ got it wrong, but they got it wrong several years ago, and fixed up their sh*t.

This is why you use generic locking. ALWAYS.

Linus

#73 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 11 lines · Alan Stern
Quoted 2 lines

So we actually _want_ the full rwlock semantics.

I'm not convinced. Condense the description a little farther:

Suspend: Children lock the parent first. When they are
finished they unlock the parent, allowing it to proceed.

Resume: Parent locks itself first. When it is finished
it unlocks itself, allowing the children to proceed.

Yes. You can implement it with a simple lock with a count. Nobody debates that.

But a simple counting lock _is_ a rwlock. Really. They are 100% semantically equivalent. There is no difference.

Quoted 1 line

The whole readers vs. writers thing is a non-sequitur.

No it's not.

It's a 100% equivalent problem. It's purely a change of wording. The end result is the same.

Quoted 2 lines

The simplest generalization is to allow both multiple lockers and multiple waiters. Call it a waitlock, for want of a better name:

But we have that. It _has_ a better name: rwlocks.

And the reason the name is better is because now the name describes all the semantics to anybody who has ever taken a course in operating systems or in parallelism.

It's also a better implementation, because it actually _works_.

Quoted 19 lines
	wait_lock(wl)
	{
		atomic_inc(&wl->count);
	}

	wait_unlock(wl)
	{
		if (atomic_dec_and_test(&wl->count)) {
			smp_mb__after_atomic_dec();
			wake_up_all(wl->wqh);
		}
	}

	wait_for_lock(wl)
	{
		wait_event(wl->wqh, atomic_read(&wl->count) == 0);
		smp_rmb();
	}
Quoted 2 lines

Note that both wait_lock() and wait_unlock() can be called in_interrupt.

And note how even though you sprinkled random memory barriers around, you still got it wrong.

So you just implemented a buggy lock, and for what gain? Tell me exactly why your buggy lock (assuming you'd know enough about memory ordering to actually fix it) is better than just using the existing one?

It's certainly not smaller. It's not faster. It doesn't have support for lockdep. And it's BUGGY.

Really. Tell me why you want to re-implement an existing lock - badly.

[ Hint: you need a smp_mb() *before* the atomic_dec() in wait-unlock, so
that anybody else who sees the new value will be guaranteed to have seen anything else the unlocker did.

You also need a smp_mb() in the wait_for_lock(), not a smp_rmb(). Can't allow writes to migrate up either. 'atomic_read()' does not imply any barriers.

But most architectures can optimize these things for their particular memory ordering model, and do so in their rwsem implementation. ]

Quoted 13 lines

This becomes:

	usb_node_resume(node)
	{
		// Wait for parent to finish resume
		wait_for_lock(node->parent->lock);
		// .. before resuming outselves
                node->resume(node)

		// Now we're all done
		wait_unlock(node->lock);
	}
Quoted 7 lines
	/* caller: */
	..
	// This can't block, because wait_lock() is non-blocking.
	wait_lock(node->lock);
	// Do the blocking part asynchronously
	async_schedule(usb_node_resume, leaf);
	..

Umm? Same thing, different words?

That "wait_for_lock()" is equivalent to a 'read_lock()+read_unlock()'. We _could_ expose such a mechanism for rwsem's too, but why do it? It's actually nicer to use a real read-lock - and do it _around_ the operation, because now the locking also automatically gets things like overlapping suspends and resumes right.

(Which you'd obviously hope never happens, but it's nice from a conceptual standpoint to know that the locking is robust).

Quoted 2 lines

Aren't waitlocks simpler than rwsems? Not as generally useful, perhaps. But just as correct in this situation.

NO!

Dammit. I started this whole rant with this comment to Rafael:

"The fact is, any time anybody makes up a new locking mechanism, THEY
ALWAYS GET IT WRONG. Don't do it."

Take heed. You got it wrong. Admit it. Locking is _hard_. SMP memory ordering is HARD.

So leave locking to the pro's. They _also_ got it wrong, but they got it wrong several years ago, and fixed up their sh*t.

This is why you use generic locking. ALWAYS.

Linus

#74 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 11 lines · Linus Torvalds

[ Hint: you need a smp_mb() *before* the atomic_dec() in wait-unlock, so
that anybody else who sees the new value will be guaranteed to have seen anything else the unlocker did.

You also need a smp_mb() in the wait_for_lock(), not a smp_rmb(). Can't allow writes to migrate up either. 'atomic_read()' does not imply any barriers.

But most architectures can optimize these things for their particular memory ordering model, and do so in their rwsem implementation. ]

Side note: if this was a real lock, you'd also needed an smp_wmb() in the 'wait_lock()' path after the atomic_inc(), to make sure that others see the atomic lock was seen by other people before the suspend started.

In your usage scenario, I don't think it would ever be noticeable, since the other users are always going to start running from the same thread that did the wait_lock(), so even if they run on other CPU's, we'll have scheduled _to_ those other CPU's and done enough memory ordering to guarantee that they will see the thing.

So it would be ok in this situation, simply because it acts as an initializer and never sees any real SMP issues.

But it's an example of how you now don't just depend on the locking primitives themselves doing the right thing, you end up depending very subtly on exactly how the lock is used. The standard locks do have the same kind of issue for initializers, but we avoid it elsewhere because it's so risky.

Linus

#75 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds

[ Hint: you need a smp_mb() *before* the atomic_dec() in wait-unlock, so
that anybody else who sees the new value will be guaranteed to have seen anything else the unlocker did.

You also need a smp_mb() in the wait_for_lock(), not a smp_rmb(). Can't allow writes to migrate up either. 'atomic_read()' does not imply any barriers.

But most architectures can optimize these things for their particular memory ordering model, and do so in their rwsem implementation. ]

Side note: if this was a real lock, you'd also needed an smp_wmb() in the 'wait_lock()' path after the atomic_inc(), to make sure that others see the atomic lock was seen by other people before the suspend started.

In your usage scenario, I don't think it would ever be noticeable, since the other users are always going to start running from the same thread that did the wait_lock(), so even if they run on other CPU's, we'll have scheduled _to_ those other CPU's and done enough memory ordering to guarantee that they will see the thing.

So it would be ok in this situation, simply because it acts as an initializer and never sees any real SMP issues.

But it's an example of how you now don't just depend on the locking primitives themselves doing the right thing, you end up depending very subtly on exactly how the lock is used. The standard locks do have the same kind of issue for initializers, but we avoid it elsewhere because it's so risky.

Linus

#76 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 4 lines · Linus Torvalds

Side note: if this was a real lock, you'd also needed an smp_wmb() in the 'wait_lock()' path after the atomic_inc(), to make sure that others see the atomic lock was seen by other people before the suspend started.

Quoted 8 lines

In your usage scenario, I don't think it would ever be noticeable, since the other users are always going to start running from the same thread that did the wait_lock(), so even if they run on other CPU's, we'll have scheduled _to_ those other CPU's and done enough memory ordering to guarantee that they will see the thing.

So it would be ok in this situation, simply because it acts as an initializer and never sees any real SMP issues.

Yes. I would have brought this up, but you made the point for me.

Quoted 5 lines

But it's an example of how you now don't just depend on the locking primitives themselves doing the right thing, you end up depending very subtly on exactly how the lock is used. The standard locks do have the same kind of issue for initializers, but we avoid it elsewhere because it's so risky.

No doubt there are other reasons why the "wait-lock" pattern doesn't get used enough to be noticed.

Alan Stern

#77 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Linus Torvalds
Quoted 4 lines · Linus Torvalds

Side note: if this was a real lock, you'd also needed an smp_wmb() in the 'wait_lock()' path after the atomic_inc(), to make sure that others see the atomic lock was seen by other people before the suspend started.

Quoted 8 lines

In your usage scenario, I don't think it would ever be noticeable, since the other users are always going to start running from the same thread that did the wait_lock(), so even if they run on other CPU's, we'll have scheduled _to_ those other CPU's and done enough memory ordering to guarantee that they will see the thing.

So it would be ok in this situation, simply because it acts as an initializer and never sees any real SMP issues.

Yes. I would have brought this up, but you made the point for me.

Quoted 5 lines

But it's an example of how you now don't just depend on the locking primitives themselves doing the right thing, you end up depending very subtly on exactly how the lock is used. The standard locks do have the same kind of issue for initializers, but we avoid it elsewhere because it's so risky.

No doubt there are other reasons why the "wait-lock" pattern doesn't get used enough to be noticed.

Alan Stern

#78 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds
Quoted 1 line

The whole readers vs. writers thing is a non-sequitur.

No it's not.

It's a 100% equivalent problem. It's purely a change of wording. The end result is the same.

Well, of course the end result is the same (ignoring bugs) -- that was the point. It doesn't follow that the two locking mechanisms are 100% equivalent.

Quoted 2 lines

And note how even though you sprinkled random memory barriers around, you still got it wrong.

Yes. That comes of trying to think at the keyboard.

Quoted 2 lines

It's certainly not smaller. It's not faster. It doesn't have support for lockdep. And it's BUGGY.

Lockdep will choke on the rwsem approach anyway. It has never been very good at handling tree-structured locking, especially when there are non-parent-child interactions. But never mind.

Quoted 1 line

Really. Tell me why you want to re-implement an existing lock - badly.

I didn't want to. The whole exercise was intended to make a point -- that rwsems do more than we really need here.

Quoted 3 lines

[ Hint: you need a smp_mb() *before* the atomic_dec() in wait-unlock, so
that anybody else who sees the new value will be guaranteed to have seen anything else the unlocker did.

Yes.

Quoted 3 lines

You also need a smp_mb() in the wait_for_lock(), not a smp_rmb(). Can't allow writes to migrate up either. 'atomic_read()' does not imply any barriers.

No, that's not needed. Unlike reads, writes can't move in front of data or control dependencies. Or so I've been lead to believe...

Quoted 1 line

That "wait_for_lock()" is equivalent to a 'read_lock()+read_unlock()'.

Not really. It also corresponds to a 'write_lock()+write_unlock()' (in the suspend routine). Are you claiming these two compound operations are equivalent?

Quoted 11 lines

We
_could_ expose such a mechanism for rwsem's too, but why do it? It's actually nicer to use a real read-lock - and do it _around_ the operation, because now the locking also automatically gets things like overlapping suspends and resumes right.

(Which you'd obviously hope never happens, but it's nice from a conceptual standpoint to know that the locking is robust).

Take heed. You got it wrong. Admit it. Locking is _hard_. SMP memory ordering is HARD.

Oh, there's no question about that. I never seriously intended this stuff to be adopted. It was just for discussion.

Alan Stern

#79 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 7 lines · Alan Stern
Quoted 4 lines

You also need a smp_mb() in the wait_for_lock(), not a smp_rmb(). Can't allow writes to migrate up either. 'atomic_read()' does not imply any barriers.

No, that's not needed. Unlike reads, writes can't move in front of data or control dependencies. Or so I've been lead to believe...

Sure they can. Control dependencies are trivial - it's called "branch prediction", and everybody does it, and data dependencies don't exist on many CPU architectures (even to the point of reading through a pointer that you loaded).

But yes, on x86, stores only move down. But that's an x86-specific thing.

[ Not that it's also not very common - write buffering is easy and matters
for performance, so any in-order implementation will generally do it. In contrast, writes moving up doesn't really help peformance and is harder to do, but can happen with a weakly ordered memory subsystem especially if you have multi-way caches where some ways are busy and end up being congested.

So the _common_ case is definitely about delaying writes and doing reads early if possible. But it's not necessarily at all guaranteed in general. ]

Quoted 5 lines
Quoted 1 line

That "wait_for_lock()" is equivalent to a 'read_lock()+read_unlock()'.

Not really. It also corresponds to a 'write_lock()+write_unlock()' (in the suspend routine). Are you claiming these two compound operations are equivalent?

They have separate semantics, and you just want to pick the one that suits you. Your counting lock doesn't have the "read_lock+read_unlock" version, it only has the write_lock/unlock one (ie it requires totally unlocked thing).

The point being, rwsem's can do everything your counting lock does. And they already exist. And they already know about all the subtleties of architecture-specific memory ordering etc.

Linus

#80 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 7 lines · Alan Stern
Quoted 4 lines

You also need a smp_mb() in the wait_for_lock(), not a smp_rmb(). Can't allow writes to migrate up either. 'atomic_read()' does not imply any barriers.

No, that's not needed. Unlike reads, writes can't move in front of data or control dependencies. Or so I've been lead to believe...

Sure they can. Control dependencies are trivial - it's called "branch prediction", and everybody does it, and data dependencies don't exist on many CPU architectures (even to the point of reading through a pointer that you loaded).

But yes, on x86, stores only move down. But that's an x86-specific thing.

[ Not that it's also not very common - write buffering is easy and matters
for performance, so any in-order implementation will generally do it. In contrast, writes moving up doesn't really help peformance and is harder to do, but can happen with a weakly ordered memory subsystem especially if you have multi-way caches where some ways are busy and end up being congested.

So the _common_ case is definitely about delaying writes and doing reads early if possible. But it's not necessarily at all guaranteed in general. ]

Quoted 5 lines
Quoted 1 line

That "wait_for_lock()" is equivalent to a 'read_lock()+read_unlock()'.

Not really. It also corresponds to a 'write_lock()+write_unlock()' (in the suspend routine). Are you claiming these two compound operations are equivalent?

They have separate semantics, and you just want to pick the one that suits you. Your counting lock doesn't have the "read_lock+read_unlock" version, it only has the write_lock/unlock one (ie it requires totally unlocked thing).

The point being, rwsem's can do everything your counting lock does. And they already exist. And they already know about all the subtleties of architecture-specific memory ordering etc.

Linus

#81 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 7 lines · Linus Torvalds
Quoted 2 lines

No, that's not needed. Unlike reads, writes can't move in front of data or control dependencies. Or so I've been lead to believe...

Sure they can. Control dependencies are trivial - it's called "branch prediction", and everybody does it, and data dependencies don't exist on many CPU architectures (even to the point of reading through a pointer that you loaded).

Wait a second. Are you saying that with code like this:

	if (x == 1)
		y = 5;

the CPU may write to y before it has finished reading the value of x? And this write is visible to other CPUs, so that if x was initially 0 and a second CPU sets x to 1, the second CPU may see y == 5 before it executes the write to x (whatever that may mean)?

Alan Stern

#82 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 7 lines
Quoted 4 lines

Sure they can. Control dependencies are trivial - it's called "branch prediction", and everybody does it, and data dependencies don't exist on many CPU architectures (even to the point of reading through a pointer that you loaded).

Wait a second. Are you saying that with code like this:

Quoted 7 lines
	if (x == 1)
		y = 5;

the CPU may write to y before it has finished reading the value of x? And this write is visible to other CPUs, so that if x was initially 0 and a second CPU sets x to 1, the second CPU may see y == 5 before it executes the write to x (whatever that may mean)?

No, the write really depends on x being 1 at any time before the comparison. On the other hand x being != 0 during the comparison does not prevent the write without proper locking or barriers.

Have a look at

http://www.linuxjournal.com/article/8211
http://www.linuxjournal.com/article/8212

especially at the alpha part what can happen when dealing with pointer accesses.

Christian

#83 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 7 lines
Quoted 4 lines

Sure they can. Control dependencies are trivial - it's called "branch prediction", and everybody does it, and data dependencies don't exist on many CPU architectures (even to the point of reading through a pointer that you loaded).

Wait a second. Are you saying that with code like this:

Quoted 7 lines
	if (x == 1)
		y = 5;

the CPU may write to y before it has finished reading the value of x? And this write is visible to other CPUs, so that if x was initially 0 and a second CPU sets x to 1, the second CPU may see y == 5 before it executes the write to x (whatever that may mean)?

No, the write really depends on x being 1 at any time before the comparison. On the other hand x being != 0 during the comparison does not prevent the write without proper locking or barriers.

Have a look at

http://www.linuxjournal.com/article/8211
http://www.linuxjournal.com/article/8212

especially at the alpha part what can happen when dealing with pointer accesses.

Christian

#84 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 6 lines · Alan Stern
Quoted 5 lines

Sure they can. Control dependencies are trivial - it's called "branch prediction", and everybody does it, and data dependencies don't exist on many CPU architectures (even to the point of reading through a pointer that you loaded).

Quoted 6 lines

Wait a second. Are you saying that with code like this:

	if (x == 1)
		y = 5;

the CPU may write to y before it has finished reading the value of x?

Well, in a way. The branch may have been predicted, and the CPU can _internally_ have done the 'y=5' thing into a write buffer before it even did the read.

Some time later it will have to _verify_ the prediction and then perhaps kill the write before it makes it to a data structure that is visible to others, but internally from the CPU standpoint, yes, the write could have happened before the read.

Now, whether that write is "before" or "after" the read is debatable. But one way of looking at it is certainly that the write took place earlier, and the read might have just caused it to be undone.

And there are real effects of this - looking at the bus, you might have a bus transaction to get the cacheline that contains 'y' for exclusive access happen _before_ the bus transaction that reads in the value of 'x' (but you'd never see the writeout of that '5' before).

Quoted 3 lines

And this write is visible to other CPUs, so that if x was initially 0 and a second CPU sets x to 1, the second CPU may see y == 5 before it executes the write to x (whatever that may mean)?

Well, yes and no. CPU1 above won't release the '5' until it has confirmed the '1' (even if it does so by reading it late). but assuming the other CPU also does speculation, then yes, the situation you describe could happen. If the other CPU does

		z = y;
		x = 1;

then it's certainly possible that 'z' contains 5 at the end (even if both x and y started out zero). Because now the read of 'y' on that other CPU might be delayed, and the write of 'x' goes ahead, CPU1 sees the 1, and commits its write of 5, sp when CPU2 gets the cacheline, z will now contain 5.

Is it likely? No. CPU microarchitectures aim to do reads early, and writes late. Reads are on the critical path, writes can be buffered. But you can basically get into "impossible" situations where a write that was _later_ in the instruction stream than a read (on CPU2, the 'store 1 to x' would be after the load of 'y' from memory) could show up in the other order on another CPU.

Linus

#85 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 6 lines · Alan Stern
Quoted 5 lines

Sure they can. Control dependencies are trivial - it's called "branch prediction", and everybody does it, and data dependencies don't exist on many CPU architectures (even to the point of reading through a pointer that you loaded).

Quoted 6 lines

Wait a second. Are you saying that with code like this:

	if (x == 1)
		y = 5;

the CPU may write to y before it has finished reading the value of x?

Well, in a way. The branch may have been predicted, and the CPU can _internally_ have done the 'y=5' thing into a write buffer before it even did the read.

Some time later it will have to _verify_ the prediction and then perhaps kill the write before it makes it to a data structure that is visible to others, but internally from the CPU standpoint, yes, the write could have happened before the read.

Now, whether that write is "before" or "after" the read is debatable. But one way of looking at it is certainly that the write took place earlier, and the read might have just caused it to be undone.

And there are real effects of this - looking at the bus, you might have a bus transaction to get the cacheline that contains 'y' for exclusive access happen _before_ the bus transaction that reads in the value of 'x' (but you'd never see the writeout of that '5' before).

Quoted 3 lines

And this write is visible to other CPUs, so that if x was initially 0 and a second CPU sets x to 1, the second CPU may see y == 5 before it executes the write to x (whatever that may mean)?

Well, yes and no. CPU1 above won't release the '5' until it has confirmed the '1' (even if it does so by reading it late). but assuming the other CPU also does speculation, then yes, the situation you describe could happen. If the other CPU does

		z = y;
		x = 1;

then it's certainly possible that 'z' contains 5 at the end (even if both x and y started out zero). Because now the read of 'y' on that other CPU might be delayed, and the write of 'x' goes ahead, CPU1 sees the 1, and commits its write of 5, sp when CPU2 gets the cacheline, z will now contain 5.

Is it likely? No. CPU microarchitectures aim to do reads early, and writes late. Reads are on the critical path, writes can be buffered. But you can basically get into "impossible" situations where a write that was _later_ in the instruction stream than a read (on CPU2, the 'store 1 to x' would be after the load of 'y' from memory) could show up in the other order on another CPU.

Linus

#86 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 16 lines · Linus Torvalds
Quoted 10 lines

Wait a second. Are you saying that with code like this:

	if (x == 1)
		y = 5;

the CPU may write to y before it has finished reading the value of x?

And this write is visible to other CPUs, so that if x was initially 0 and a second CPU sets x to 1, the second CPU may see y == 5 before it executes the write to x (whatever that may mean)?

Well, yes and no. CPU1 above won't release the '5' until it has confirmed the '1' (even if it does so by reading it late). but assuming the other CPU also does speculation, then yes, the situation you describe could happen. If the other CPU does

Quoted 8 lines
		z = y;
		x = 1;

then it's certainly possible that 'z' contains 5 at the end (even if both x and y started out zero). Because now the read of 'y' on that other CPU might be delayed, and the write of 'x' goes ahead, CPU1 sees the 1, and commits its write of 5, sp when CPU2 gets the cacheline, z will now contain 5.

That could be attributed to reordering on CPU2, so let's take CPU2's peculiarities out of the picture (initially everything is set to 0):

	CPU1			CPU2
	----			----
	if (x == 1)		z = y;
		y = 5;		mb();
				x = 1;

This gets at the heart of the question: Can a write move up past a control dependency? Similar questions apply to the two types of data dependency:

	CPU1			CPU2
	----			----
	y = x + 4;		z = y;
				mb();
				x = 1;

(Initially p points to x, not y):

	CPU1			CPU2
	----			----
	*p = 5;			z = y;
				mb();
				p = &y;

Can z end up equal to 5 in any of these examples?

Alan Stern

#87 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 10 lines · Alan Stern

That could be attributed to reordering on CPU2, so let's take CPU2's peculiarities out of the picture (initially everything is set to 0):

	CPU1			CPU2
	----			----
	if (x == 1)		z = y;
		y = 5;		mb();
				x = 1;
Quoted 4 lines

This gets at the heart of the question: Can a write move up past a control dependency?
[ .. ]
Can z end up equal to 5 in any of these examples?

In any _practical_ microarchitecture I know of, the above will never result in 'z' being 5, even though CPU1 doesn't really have a memory barrier. But if I read the alpha memory ordering guarantees rigth, then at least in theory you really can end up with z=5.

Let me write that as five events (with the things in brackets being what the alpha memory ordering manual calls them):

- A is "read of x returns 1" on CPU1 [ P1:R(x,1) ]
- B is "write of value 5 to y" on CPU1 [ P1:W(y,5) ]
- C is "read of y returns 5" on CPU2 [ P2:R(y,5) ]
- D is "write of value 1 to x" on CPU2 [ P2:W(x,1) ]
- 'MB' is the mb() on CPU2 [ P2:MB ]

(The write of 'z' is irrelevant, we can think of it as a register, the end result is the same).

And yes, if I read the alpha memory ordering rules correctly, you really can end up with z=5, although I don't think you will ever find an alpha _implementation_ that does it.

Why?

The alpha memory ordering literally defines ordering in two ways:

- "location access order". But that is _only_ defined per actual location, so while 'x' can have a location access order specified by seeing certain values, there is no "location access order" for two different memory locations (x and y).

The alpha architecture manual uses "A << B" to say "event A" is before "event B" when there is a defined ordering.

So in the example above, there is a location access ordering between

	P2:W(x,1) << P1:R(x, 1)

and

	P2:R(y,5) << P1:W(y,5)

ie you have D << A and B << C.

Good so far, but that doesn't define anything else: there's only ordering between the pairs (D,A) and (B,C), nothing between them.

- "Processor issue order" for two instruction is _only_ defined by either
(a) memory barriers or (b) accesses to the _same_ locations. The alpha architecture manual uses "A < B" to say that "event A" is before "event B" in processor issue order.

So there is a "Processor issue order" on CPU2 due to the memory
barrier: P2:R(y,5) < P2:MB < P2:W(x,1), or put another way C < MB < D:
C < D.

Now, the question is, can we actually get the behaviour of reading 5 on CPU2 (ie P2:R(y,5)), and that is only possible if we can find an ordering that satisfies all the constraints. We have

	D << A
	B << C
	C < D

and it seems to be that it is a possible situation: "B C D A" really does satisfy all the constraints afaik.

So yes, according to the actual alpha architecture memory ordering rules, you can see '5' from that first read of 'y'. DESPITE having a mb() on CPU2.

In order to not see 5, you need to also specify "A < B", and the _only_ way to do that processor issue order specification is with a memory barrier (or if the locations are the same, which they aren't).

"Causality" simply is nowhere in the officially defined alpha memory ordering. The fact that we test 'x == 1' and conditionally do the write simply doesn't enter the picture. I suspect you'd have a really hard time not having causality in practice, but there _are_ things that can break causality (value prediction etc), so it's not like you'd have to actually violate physics of reality to do it.

IOW, you could at least in theory implement a CPU that does every instruction speculatively in parallel, and then validates the end result afterwards according to the architecture rules. And that CPU would require the memory barrier on alpha.

(On x86, 'causality' is defined to be part of the memory ordering rules, so on x86, you _do_ have a 'A < B' relationship. But not on alpha).

Linus

#88 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 10 lines · Alan Stern

That could be attributed to reordering on CPU2, so let's take CPU2's peculiarities out of the picture (initially everything is set to 0):

	CPU1			CPU2
	----			----
	if (x == 1)		z = y;
		y = 5;		mb();
				x = 1;
Quoted 4 lines

This gets at the heart of the question: Can a write move up past a control dependency?
[ .. ]
Can z end up equal to 5 in any of these examples?

In any _practical_ microarchitecture I know of, the above will never result in 'z' being 5, even though CPU1 doesn't really have a memory barrier. But if I read the alpha memory ordering guarantees rigth, then at least in theory you really can end up with z=5.

Let me write that as five events (with the things in brackets being what the alpha memory ordering manual calls them):

- A is "read of x returns 1" on CPU1 [ P1:R(x,1) ]
- B is "write of value 5 to y" on CPU1 [ P1:W(y,5) ]
- C is "read of y returns 5" on CPU2 [ P2:R(y,5) ]
- D is "write of value 1 to x" on CPU2 [ P2:W(x,1) ]
- 'MB' is the mb() on CPU2 [ P2:MB ]

(The write of 'z' is irrelevant, we can think of it as a register, the end result is the same).

And yes, if I read the alpha memory ordering rules correctly, you really can end up with z=5, although I don't think you will ever find an alpha _implementation_ that does it.

Why?

The alpha memory ordering literally defines ordering in two ways:

- "location access order". But that is _only_ defined per actual location, so while 'x' can have a location access order specified by seeing certain values, there is no "location access order" for two different memory locations (x and y).

The alpha architecture manual uses "A << B" to say "event A" is before "event B" when there is a defined ordering.

So in the example above, there is a location access ordering between

	P2:W(x,1) << P1:R(x, 1)

and

	P2:R(y,5) << P1:W(y,5)

ie you have D << A and B << C.

Good so far, but that doesn't define anything else: there's only ordering between the pairs (D,A) and (B,C), nothing between them.

- "Processor issue order" for two instruction is _only_ defined by either
(a) memory barriers or (b) accesses to the _same_ locations. The alpha architecture manual uses "A < B" to say that "event A" is before "event B" in processor issue order.

So there is a "Processor issue order" on CPU2 due to the memory
barrier: P2:R(y,5) < P2:MB < P2:W(x,1), or put another way C < MB < D:
C < D.

Now, the question is, can we actually get the behaviour of reading 5 on CPU2 (ie P2:R(y,5)), and that is only possible if we can find an ordering that satisfies all the constraints. We have

	D << A
	B << C
	C < D

and it seems to be that it is a possible situation: "B C D A" really does satisfy all the constraints afaik.

So yes, according to the actual alpha architecture memory ordering rules, you can see '5' from that first read of 'y'. DESPITE having a mb() on CPU2.

In order to not see 5, you need to also specify "A < B", and the _only_ way to do that processor issue order specification is with a memory barrier (or if the locations are the same, which they aren't).

"Causality" simply is nowhere in the officially defined alpha memory ordering. The fact that we test 'x == 1' and conditionally do the write simply doesn't enter the picture. I suspect you'd have a really hard time not having causality in practice, but there _are_ things that can break causality (value prediction etc), so it's not like you'd have to actually violate physics of reality to do it.

IOW, you could at least in theory implement a CPU that does every instruction speculatively in parallel, and then validates the end result afterwards according to the architecture rules. And that CPU would require the memory barrier on alpha.

(On x86, 'causality' is defined to be part of the memory ordering rules, so on x86, you _do_ have a 'A < B' relationship. But not on alpha).

Linus

#89 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Linus Torvalds
Quoted 16 lines · Linus Torvalds
Quoted 10 lines

Wait a second. Are you saying that with code like this:

	if (x == 1)
		y = 5;

the CPU may write to y before it has finished reading the value of x?

And this write is visible to other CPUs, so that if x was initially 0 and a second CPU sets x to 1, the second CPU may see y == 5 before it executes the write to x (whatever that may mean)?

Well, yes and no. CPU1 above won't release the '5' until it has confirmed the '1' (even if it does so by reading it late). but assuming the other CPU also does speculation, then yes, the situation you describe could happen. If the other CPU does

Quoted 8 lines
		z = y;
		x = 1;

then it's certainly possible that 'z' contains 5 at the end (even if both x and y started out zero). Because now the read of 'y' on that other CPU might be delayed, and the write of 'x' goes ahead, CPU1 sees the 1, and commits its write of 5, sp when CPU2 gets the cacheline, z will now contain 5.

That could be attributed to reordering on CPU2, so let's take CPU2's peculiarities out of the picture (initially everything is set to 0):

	CPU1			CPU2
	----			----
	if (x == 1)		z = y;
		y = 5;		mb();
				x = 1;

This gets at the heart of the question: Can a write move up past a control dependency? Similar questions apply to the two types of data dependency:

	CPU1			CPU2
	----			----
	y = x + 4;		z = y;
				mb();
				x = 1;

(Initially p points to x, not y):

	CPU1			CPU2
	----			----
	*p = 5;			z = y;
				mb();
				p = &y;

Can z end up equal to 5 in any of these examples?

Alan Stern

#90 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Linus Torvalds
Quoted 7 lines · Linus Torvalds
Quoted 2 lines

No, that's not needed. Unlike reads, writes can't move in front of data or control dependencies. Or so I've been lead to believe...

Sure they can. Control dependencies are trivial - it's called "branch prediction", and everybody does it, and data dependencies don't exist on many CPU architectures (even to the point of reading through a pointer that you loaded).

Wait a second. Are you saying that with code like this:

	if (x == 1)
		y = 5;

the CPU may write to y before it has finished reading the value of x? And this write is visible to other CPUs, so that if x was initially 0 and a second CPU sets x to 1, the second CPU may see y == 5 before it executes the write to x (whatever that may mean)?

Alan Stern

#91 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds
Quoted 1 line

The whole readers vs. writers thing is a non-sequitur.

No it's not.

It's a 100% equivalent problem. It's purely a change of wording. The end result is the same.

Well, of course the end result is the same (ignoring bugs) -- that was the point. It doesn't follow that the two locking mechanisms are 100% equivalent.

Quoted 2 lines

And note how even though you sprinkled random memory barriers around, you still got it wrong.

Yes. That comes of trying to think at the keyboard.

Quoted 2 lines

It's certainly not smaller. It's not faster. It doesn't have support for lockdep. And it's BUGGY.

Lockdep will choke on the rwsem approach anyway. It has never been very good at handling tree-structured locking, especially when there are non-parent-child interactions. But never mind.

Quoted 1 line

Really. Tell me why you want to re-implement an existing lock - badly.

I didn't want to. The whole exercise was intended to make a point -- that rwsems do more than we really need here.

Quoted 3 lines

[ Hint: you need a smp_mb() *before* the atomic_dec() in wait-unlock, so
that anybody else who sees the new value will be guaranteed to have seen anything else the unlocker did.

Yes.

Quoted 3 lines

You also need a smp_mb() in the wait_for_lock(), not a smp_rmb(). Can't allow writes to migrate up either. 'atomic_read()' does not imply any barriers.

No, that's not needed. Unlike reads, writes can't move in front of data or control dependencies. Or so I've been lead to believe...

Quoted 1 line

That "wait_for_lock()" is equivalent to a 'read_lock()+read_unlock()'.

Not really. It also corresponds to a 'write_lock()+write_unlock()' (in the suspend routine). Are you claiming these two compound operations are equivalent?

Quoted 11 lines

We
_could_ expose such a mechanism for rwsem's too, but why do it? It's actually nicer to use a real read-lock - and do it _around_ the operation, because now the locking also automatically gets things like overlapping suspends and resumes right.

(Which you'd obviously hope never happens, but it's nice from a conceptual standpoint to know that the locking is robust).

Take heed. You got it wrong. Admit it. Locking is _hard_. SMP memory ordering is HARD.

Oh, there's no question about that. I never seriously intended this stuff to be adopted. It was just for discussion.

Alan Stern

#92 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Linus Torvalds
Quoted 16 lines · Linus Torvalds
Quoted 9 lines · Alan Stern

The semantics needed for this kind of lock aren't really the same as for an rwsem (although obviously an rwsem will do the job). Basically it needs to have the capability for multiple users to lock it (no blocking when acquiring a lock) and the capability for a user to wait until it is totally unlocked. It could be implemented trivially using an atomic_t counter and a waitqueue head.

Is this a standard sort of lock?

Yes it is.

It's called a rwlock. The counter is for readers, the exclusion is for writers.

Quoted 7 lines

Really.

And the thing is, you actually do want the rwlock semantics, because on the resume side you want the parent to lock it for writing first (so that the children can wait for the parent to have completed its resume.

So we actually _want_ the full rwlock semantics.

I'm not convinced. Condense the description a little farther:

Suspend: Children lock the parent first. When they are
finished they unlock the parent, allowing it to proceed.

Resume: Parent locks itself first. When it is finished
it unlocks itself, allowing the children to proceed.

The whole readers vs. writers thing is a non-sequitur. (For instance, this never uses the fact that writers exclude each other.) In each case a lock is taken and eventually released, allowing someone else to stop waiting and move forward. In the suspend case we have multiple lockers and one waiter, whereas in the resume case we have one locker and multiple waiters.

The simplest generalization is to allow both multiple lockers and multiple waiters. Call it a waitlock, for want of a better name:

	wait_lock(wl)
	{
		atomic_inc(&wl->count);
	}

	wait_unlock(wl)
	{
		if (atomic_dec_and_test(&wl->count)) {
			smp_mb__after_atomic_dec();
			wake_up_all(wl->wqh);
		}
	}

	wait_for_lock(wl)
	{
		wait_event(wl->wqh, atomic_read(&wl->count) == 0);
		smp_rmb();
	}

Note that both wait_lock() and wait_unlock() can be called in_interrupt.

Quoted 16 lines

See the code I posted earlier. Here condensed into one email:

- resume:

        usb_node_resume(node)
        {
		// Wait for parent to finish resume
                down_read(node->parent->lock);
		// .. before resuming outselves
                node->resume(node)

		// Now we're all done
                up_read(node->parent->lock);
                up_write(node->lock);
        }
Quoted 8 lines
	/* caller: */
	..
	// This won't block, because we resume parents before children,
	// and the children will take the read lock. 
        down_write(leaf->lock);
	// Do the blocking part asynchronously
        async_schedule(usb_node_resume, leaf);
	..

This becomes:

	usb_node_resume(node)
	{
		// Wait for parent to finish resume
		wait_for_lock(node->parent->lock);
		// .. before resuming outselves
                node->resume(node)

		// Now we're all done
		wait_unlock(node->lock);
	}

	/* caller: */
	..
	// This can't block, because wait_lock() is non-blocking.
	wait_lock(node->lock);
	// Do the blocking part asynchronously
	async_schedule(usb_node_resume, leaf);
	..
Quoted 13 lines

- suspend:

        usb_node_suspend(node)
        {
                // Start our suspend. This will block if we have any
                // children that are still busy suspending (they will
                // have done a down_read() in their suspend).
                down_write(node->lock);
                node->suspend(node);
                up_write(node->lock);

                // This lets our parent continue
                up_read(node->parent->lock);
Quoted 8 lines
        }

	/* caller: */

	// This won't block, because we suspend nodes before parents
        down_read(node->parent->lock);
	// Do the part that may block asynchronously
	async_schedule(do_usb_node_suspend, node);
	usb_node_suspend(node)
	{
		// Start our suspend.  This will block if we have any
		// children that are still busy suspending (they will
		// have done a wait_lock() in their suspend).
		wait_for_lock(node->lock);
		node->suspend(node);

		// This lets our parent continue
		wait_unlock(node->parent->lock);
	}

	/* caller: */
	..
	// This can't block, because wait_lock is non-blocking.
	wait_lock(node->parent->lock);
	// Do the part that may block asynchronously
	async_schedule(do_usb_node_suspend, node);
	..
Quoted 3 lines

It really should be that simple. Nothing more, nothing less. And with the above, finishing the suspend (or resume) from interrupts is fine, and you don't have any new lock that has undefined memory ordering issues etc.

Aren't waitlocks simpler than rwsems? Not as generally useful, perhaps. But just as correct in this situation.

Alan Stern

#93 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Linus Torvalds
Quoted 7 lines · Linus Torvalds

Please just use the lock. Don't make up your own locking crap. Really.

Your patch is horrible. Exactly because your locking is horribly mis-designed. You can't say things are complete from an interrupt, for example, since you made it some random bitfield, which has unknown characteristics (ie non-atomic read-modify-write etc).

Quoted 5 lines

The fact is, any time anybody makes up a new locking mechanism, THEY ALWAYS GET IT WRONG. Don't do it.

I suggested using the rwsem locking for a good reason. It made sense. It was simpler. Just do it that way, stop making up crap.

The semantics needed for this kind of lock aren't really the same as for an rwsem (although obviously an rwsem will do the job). Basically it needs to have the capability for multiple users to lock it (no blocking when acquiring a lock) and the capability for a user to wait until it is totally unlocked. It could be implemented trivially using an atomic_t counter and a waitqueue head.

Is this a standard sort of lock? It's a lot simpler than most others. I don't recall seeing anything quite like it anywhere; the closest thing might be some kind of barrier.

Alan Stern

#94 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds
Quoted 3 lines · Rafael J. Wysocki

The wait queue plus the op_complete flag combo plays the role of the locking in the Linus' picture

Quoted 6 lines

Please just use the lock. Don't make up your own locking crap. Really.

Your patch is horrible. Exactly because your locking is horribly mis-designed. You can't say things are complete from an interrupt, for example, since you made it some random bitfield, which has unknown characteristics (ie non-atomic read-modify-write etc).

I didn't assume anyone would check it from an interrupt, because I didn't see a point. In fact I didn't assume anyone except for the PM core would check it. In case this assumption is wrong, it can be easily put under the dev->sem that we take anyway before calling the bus type (etc.) callbacks.

Anyway, if we use an rwsem, it won't be checkable from interrupt context just as well.

Quoted 5 lines

The fact is, any time anybody makes up a new locking mechanism, THEY ALWAYS GET IT WRONG. Don't do it.

I suggested using the rwsem locking for a good reason. It made sense. It was simpler. Just do it that way, stop making up crap.

Suppose we use rwsem and during suspend each child uses a down_read() on a parent and then the parent uses down_write() on itself. What if, whatever the reason, the parent is a bit early and does the down_write() before one of the children has a chance to do the down_read()? Aren't we toast?

Do we need any direct protection against that or does it just work itself out in a way I just don't see right now?

Rafael

#95 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 7 lines · Rafael J. Wysocki

Suppose we use rwsem and during suspend each child uses a down_read() on a parent and then the parent uses down_write() on itself. What if, whatever the reason, the parent is a bit early and does the down_write() before one of the children has a chance to do the down_read()? Aren't we toast?

Do we need any direct protection against that or does it just work itself out in a way I just don't see right now?

That's not the way it should be done. Linus had children taking their parents' locks during suspend, which is simple but leads to difficulties.

Instead, the PM core should do a down_write() on each device before starting the device's async suspend routine, and an up_write() when the routine finishes. Parents should, at the start of their async routine, do down_read() on each of their children plus whatever other devices they need to wait for. The core can do the waiting for children part and the driver's suspend routine can handle any other waiting.

This is a little more awkward because it requires the parent to iterate through its children. But it does solve the off-tree dependency problem for suspends.

Alan Stern

#96 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 21 lines · Alan Stern
Quoted 7 lines · Rafael J. Wysocki

Suppose we use rwsem and during suspend each child uses a down_read() on a parent and then the parent uses down_write() on itself. What if, whatever the reason, the parent is a bit early and does the down_write() before one of the children has a chance to do the down_read()? Aren't we toast?

Do we need any direct protection against that or does it just work itself out in a way I just don't see right now?

That's not the way it should be done. Linus had children taking their parents' locks during suspend, which is simple but leads to difficulties.

Instead, the PM core should do a down_write() on each device before starting the device's async suspend routine, and an up_write() when the routine finishes. Parents should, at the start of their async routine, do down_read() on each of their children plus whatever other devices they need to wait for. The core can do the waiting for children part and the driver's suspend routine can handle any other waiting.

Quoted 2 lines

This is a little more awkward because it requires the parent to iterate through its children.

I can live with that.

Quoted 1 line

But it does solve the off-tree dependency problem for suspends.

That's a plus, but I still think we're trying to create a barrier-alike mechanism using lock.

There's one more possibility to consider, though. What if we use a completion instead of the flag + wait queue? It surely is a standard synchronization mechanism and it seems it might work here.

Rafael

#97 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 5 lines · Rafael J. Wysocki
Quoted 2 lines

This is a little more awkward because it requires the parent to iterate through its children.

I can live with that.

Quoted 8 lines
Quoted 1 line

But it does solve the off-tree dependency problem for suspends.

That's a plus, but I still think we're trying to create a barrier-alike mechanism using lock.

There's one more possibility to consider, though. What if we use a completion instead of the flag + wait queue? It surely is a standard synchronization mechanism and it seems it might work here.

You're right. I should have thought of that. Linus's original approach couldn't use a completion because during suspend it needed to make one task (the parent) wait for a bunch of others (the children). But if you iterate through the children by hand, that objection no longer applies.

Alan Stern

#98 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 16 lines · Alan Stern
Quoted 5 lines · Rafael J. Wysocki
Quoted 2 lines

This is a little more awkward because it requires the parent to iterate through its children.

I can live with that.

Quoted 8 lines
Quoted 1 line

But it does solve the off-tree dependency problem for suspends.

That's a plus, but I still think we're trying to create a barrier-alike mechanism using lock.

There's one more possibility to consider, though. What if we use a completion instead of the flag + wait queue? It surely is a standard synchronization mechanism and it seems it might work here.

Quoted 5 lines

You're right. I should have thought of that. Linus's original approach couldn't use a completion because during suspend it needed to make one task (the parent) wait for a bunch of others (the children). But if you iterate through the children by hand, that objection no longer applies.

BTW, is there a good reason why completion_done() doesn't use spin_lock_irqsave and spin_unlock_irqrestore? complete() and complete_all() use them, so why not here?

Rafael

#99 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 3 lines · Rafael J. Wysocki

BTW, is there a good reason why completion_done() doesn't use spin_lock_irqsave and spin_unlock_irqrestore? complete() and complete_all() use them, so why not here?

And likewise in try_wait_for_completion(). It looks like a bug. Maybe these routines were not intended to be called with interrupts disabled, but that requirement doesn't seem to be documented. And it isn't a natural requirement anyway.

Alan Stern

#100 lore raw

spinlock in completion_done() (was: Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33))

Quoted 10 lines · Alan Stern
Quoted 3 lines · Rafael J. Wysocki

BTW, is there a good reason why completion_done() doesn't use spin_lock_irqsave and spin_unlock_irqrestore? complete() and complete_all() use them, so why not here?

And likewise in try_wait_for_completion(). It looks like a bug. Maybe these routines were not intended to be called with interrupts disabled, but that requirement doesn't seem to be documented. And it isn't a natural requirement anyway.

OK, let's ask Ingo about that.

Ingo, is there any particular reason why completion_done() and try_wait_for_completion() don't use spin_lock_irqsave() and spin_unlock_irqrestore()?

Rafael

#101 lore raw

Re: spinlock in completion_done() (was: Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33))

Quoted 12 lines · Rafael J. Wysocki
Quoted 10 lines · Alan Stern
Quoted 3 lines · Rafael J. Wysocki

BTW, is there a good reason why completion_done() doesn't use spin_lock_irqsave and spin_unlock_irqrestore? complete() and complete_all() use them, so why not here?

And likewise in try_wait_for_completion(). It looks like a bug. Maybe these routines were not intended to be called with interrupts disabled, but that requirement doesn't seem to be documented. And it isn't a natural requirement anyway.

Quoted 5 lines

OK, let's ask Ingo about that.

Ingo, is there any particular reason why completion_done() and try_wait_for_completion() don't use spin_lock_irqsave() and spin_unlock_irqrestore()?

that's a bug that should be fixed - all the wakeup side (and atomic) variants of completetion API should be irq safe.

It appears that these new completion APIs were added via the XFS tree about a year ago:

39d2f1a: [XFS] extend completions to provide XFS object flush requirements

Please Cc: scheduler folks to all scheduler patches.

Ingo

#102 lore raw

Re: spinlock in completion_done() (was: Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33))

In reply to Rafael J. Wysocki
Quoted 12 lines · Rafael J. Wysocki
Quoted 10 lines · Alan Stern
Quoted 3 lines · Rafael J. Wysocki

BTW, is there a good reason why completion_done() doesn't use spin_lock_irqsave and spin_unlock_irqrestore? complete() and complete_all() use them, so why not here?

And likewise in try_wait_for_completion(). It looks like a bug. Maybe these routines were not intended to be called with interrupts disabled, but that requirement doesn't seem to be documented. And it isn't a natural requirement anyway.

Quoted 5 lines

OK, let's ask Ingo about that.

Ingo, is there any particular reason why completion_done() and try_wait_for_completion() don't use spin_lock_irqsave() and spin_unlock_irqrestore()?

that's a bug that should be fixed - all the wakeup side (and atomic) variants of completetion API should be irq safe.

It appears that these new completion APIs were added via the XFS tree about a year ago:

39d2f1a: [XFS] extend completions to provide XFS object flush requirements

Please Cc: scheduler folks to all scheduler patches.

Ingo

#103 lore raw

Re: spinlock in completion_done() (was: Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33))

Quoted 24 lines · Ingo Molnar
Quoted 12 lines · Rafael J. Wysocki
Quoted 10 lines · Alan Stern
Quoted 3 lines · Rafael J. Wysocki

BTW, is there a good reason why completion_done() doesn't use spin_lock_irqsave and spin_unlock_irqrestore? complete() and complete_all() use them, so why not here?

And likewise in try_wait_for_completion(). It looks like a bug. Maybe these routines were not intended to be called with interrupts disabled, but that requirement doesn't seem to be documented. And it isn't a natural requirement anyway.

Quoted 5 lines

OK, let's ask Ingo about that.

Ingo, is there any particular reason why completion_done() and try_wait_for_completion() don't use spin_lock_irqsave() and spin_unlock_irqrestore()?

that's a bug that should be fixed - all the wakeup side (and atomic) variants of completetion API should be irq safe.

Quoted 6 lines

It appears that these new completion APIs were added via the XFS tree about a year ago:

39d2f1a: [XFS] extend completions to provide XFS object flush requirements

Please Cc: scheduler folks to all scheduler patches.

If you haven't fixed it locally yet, would you mind me posting a fix?

Rafael

#104 lore raw

Re: spinlock in completion_done() (was: Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33))

Quoted 25 lines · Rafael J. Wysocki
Quoted 24 lines · Ingo Molnar
Quoted 12 lines · Rafael J. Wysocki
Quoted 10 lines · Alan Stern
Quoted 3 lines · Rafael J. Wysocki

BTW, is there a good reason why completion_done() doesn't use spin_lock_irqsave and spin_unlock_irqrestore? complete() and complete_all() use them, so why not here?

And likewise in try_wait_for_completion(). It looks like a bug. Maybe these routines were not intended to be called with interrupts disabled, but that requirement doesn't seem to be documented. And it isn't a natural requirement anyway.

Quoted 5 lines

OK, let's ask Ingo about that.

Ingo, is there any particular reason why completion_done() and try_wait_for_completion() don't use spin_lock_irqsave() and spin_unlock_irqrestore()?

that's a bug that should be fixed - all the wakeup side (and atomic) variants of completetion API should be irq safe.

Quoted 8 lines
Quoted 6 lines

It appears that these new completion APIs were added via the XFS tree about a year ago:

39d2f1a: [XFS] extend completions to provide XFS object flush requirements

Please Cc: scheduler folks to all scheduler patches.

If you haven't fixed it locally yet, would you mind me posting a fix?

I wouldnt mind it at all.

Thanks,

Ingo

#105 lore raw

Re: spinlock in completion_done() (was: Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33))

In reply to Rafael J. Wysocki
Quoted 25 lines · Rafael J. Wysocki
Quoted 24 lines · Ingo Molnar
Quoted 12 lines · Rafael J. Wysocki
Quoted 10 lines · Alan Stern
Quoted 3 lines · Rafael J. Wysocki

BTW, is there a good reason why completion_done() doesn't use spin_lock_irqsave and spin_unlock_irqrestore? complete() and complete_all() use them, so why not here?

And likewise in try_wait_for_completion(). It looks like a bug. Maybe these routines were not intended to be called with interrupts disabled, but that requirement doesn't seem to be documented. And it isn't a natural requirement anyway.

Quoted 5 lines

OK, let's ask Ingo about that.

Ingo, is there any particular reason why completion_done() and try_wait_for_completion() don't use spin_lock_irqsave() and spin_unlock_irqrestore()?

that's a bug that should be fixed - all the wakeup side (and atomic) variants of completetion API should be irq safe.

Quoted 8 lines
Quoted 6 lines

It appears that these new completion APIs were added via the XFS tree about a year ago:

39d2f1a: [XFS] extend completions to provide XFS object flush requirements

Please Cc: scheduler folks to all scheduler patches.

If you haven't fixed it locally yet, would you mind me posting a fix?

I wouldnt mind it at all.

Thanks,

Ingo

#106 lore raw

Re: spinlock in completion_done() (was: Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33))

Quoted 6 lines · Ingo Molnar
Quoted 5 lines · Rafael J. Wysocki
Quoted 4 lines · Ingo Molnar
Quoted 3 lines · Rafael J. Wysocki
Quoted 2 lines · Alan Stern

On Tue, 8 Dec 2009, Rafael J. Wysocki wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

BTW, is there a good reason why completion_done() doesn't use spin_lock_irqsave and spin_unlock_irqrestore? complete() and complete_all() use them, so why not here?

And likewise in try_wait_for_completion(). It looks like a bug. Maybe these routines were not intended to be called with interrupts disabled, but that requirement doesn't seem to be documented. And it isn't a natural requirement anyway.

When I implemented them they were not called from anywhere that disabled interrupts. IIRC the main reason I used spin_lock_irq() was because that is what wait_for_completion() used at the time....

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

that's a bug that should be fixed - all the wakeup side (and atomic) variants of completetion API should be irq safe.

I see no problems with that ;)

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#107 lore raw

Re: spinlock in completion_done() (was: Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33))

Quoted 11 lines · Dave Chinner
Quoted 10 lines · Ingo Molnar
Quoted 9 lines · Rafael J. Wysocki
Quoted 8 lines · Ingo Molnar
Quoted 7 lines · Rafael J. Wysocki
Quoted 6 lines · Alan Stern
Quoted 3 lines · Rafael J. Wysocki

BTW, is there a good reason why completion_done() doesn't use spin_lock_irqsave and spin_unlock_irqrestore? complete() and complete_all() use them, so why not here?

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

And likewise in try_wait_for_completion(). It looks like a bug. Maybe these routines were not intended to be called with interrupts disabled, but that requirement doesn't seem to be documented. And it isn't a natural requirement anyway.

When I implemented them they were not called from anywhere that disabled interrupts. IIRC the main reason I used spin_lock_irq() was because that is what wait_for_completion() used at the time....

Obviously wait_for_competion() as a non-atomic API that can block will (and should) use _irq() - but atomic variants (complete, but also the try-wait thing) use irqsafe methods. A fair portion of completions happen in IRQ context.

Ingo

#108 lore raw

Re: spinlock in completion_done() (was: Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33))

In reply to Dave Chinner
Quoted 11 lines · Dave Chinner
Quoted 10 lines · Ingo Molnar
Quoted 9 lines · Rafael J. Wysocki
Quoted 8 lines · Ingo Molnar
Quoted 7 lines · Rafael J. Wysocki
Quoted 6 lines · Alan Stern
Quoted 3 lines · Rafael J. Wysocki

BTW, is there a good reason why completion_done() doesn't use spin_lock_irqsave and spin_unlock_irqrestore? complete() and complete_all() use them, so why not here?

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

And likewise in try_wait_for_completion(). It looks like a bug. Maybe these routines were not intended to be called with interrupts disabled, but that requirement doesn't seem to be documented. And it isn't a natural requirement anyway.

When I implemented them they were not called from anywhere that disabled interrupts. IIRC the main reason I used spin_lock_irq() was because that is what wait_for_completion() used at the time....

Obviously wait_for_competion() as a non-atomic API that can block will (and should) use _irq() - but atomic variants (complete, but also the try-wait thing) use irqsafe methods. A fair portion of completions happen in IRQ context.

Ingo

#109 lore raw

Re: spinlock in completion_done() (was: Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33))

In reply to Ingo Molnar
Quoted 6 lines · Ingo Molnar
Quoted 5 lines · Rafael J. Wysocki
Quoted 4 lines · Ingo Molnar
Quoted 3 lines · Rafael J. Wysocki
Quoted 2 lines · Alan Stern

On Tue, 8 Dec 2009, Rafael J. Wysocki wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

BTW, is there a good reason why completion_done() doesn't use spin_lock_irqsave and spin_unlock_irqrestore? complete() and complete_all() use them, so why not here?

And likewise in try_wait_for_completion(). It looks like a bug. Maybe these routines were not intended to be called with interrupts disabled, but that requirement doesn't seem to be documented. And it isn't a natural requirement anyway.

When I implemented them they were not called from anywhere that disabled interrupts. IIRC the main reason I used spin_lock_irq() was because that is what wait_for_completion() used at the time....

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

that's a bug that should be fixed - all the wakeup side (and atomic) variants of completetion API should be irq safe.

I see no problems with that ;)

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#110 lore raw

PATCH 1/1 sched: Make wakeup side variants of completion API irq safe (was: Re: spinlock in completion_done())

In reply to Ingo Molnar
Quoted 31 lines · Ingo Molnar
Quoted 28 lines · Rafael J. Wysocki
Quoted 27 lines · Ingo Molnar
Quoted 12 lines · Rafael J. Wysocki
Quoted 10 lines · Alan Stern
Quoted 3 lines · Rafael J. Wysocki

BTW, is there a good reason why completion_done() doesn't use spin_lock_irqsave and spin_unlock_irqrestore? complete() and complete_all() use them, so why not here?

And likewise in try_wait_for_completion(). It looks like a bug. Maybe these routines were not intended to be called with interrupts disabled, but that requirement doesn't seem to be documented. And it isn't a natural requirement anyway.

Quoted 5 lines

OK, let's ask Ingo about that.

Ingo, is there any particular reason why completion_done() and try_wait_for_completion() don't use spin_lock_irqsave() and spin_unlock_irqrestore()?

that's a bug that should be fixed - all the wakeup side (and atomic) variants of completetion API should be irq safe.

It appears that these new completion APIs were added via the XFS tree about a year ago:

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

39d2f1a: [XFS] extend completions to provide XFS object flush requirements

Please Cc: scheduler folks to all scheduler patches.

If you haven't fixed it locally yet, would you mind me posting a fix?

I wouldnt mind it at all.

Is appended.

Thanks,
Rafael

---
From: Rafael J. Wysocki <[email protected]>
Subject: sched: Make wakeup side variants of completion API irq safe

All the wakeup side variants of the completion API shoild be irq safe, but completion_done() and try_wait_for_completion() aren't. Fix the problem by making them use spin_lock_irqsave() and spin_lock_irqrestore().

---
 kernel/sched.c |   10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
kernel/sched.c +6 -4
Index: linux-2.6/kernel/sched.c
===================================================================
@@ -5931,14 +5931,15 @@ EXPORT_SYMBOL(wait_for_completion_killab
  */
 bool try_wait_for_completion(struct completion *x)
 {
+	unsigned long flags;
 	int ret = 1;
 
-	spin_lock_irq(&x->wait.lock);
+	spin_lock_irqsave(&x->wait.lock, flags);
 	if (!x->done)
 		ret = 0;
 	else
 		x->done--;
-	spin_unlock_irq(&x->wait.lock);
+	spin_unlock_irqrestore(&x->wait.lock, flags);
 	return ret;
 }
 EXPORT_SYMBOL(try_wait_for_completion);
@@ -5953,12 +5954,13 @@ EXPORT_SYMBOL(try_wait_for_completion);
  */
 bool completion_done(struct completion *x)
 {
+	unsigned long flags;
 	int ret = 1;
 
-	spin_lock_irq(&x->wait.lock);
+	spin_lock_irqsave(&x->wait.lock, flags);
 	if (!x->done)
 		ret = 0;
-	spin_unlock_irq(&x->wait.lock);
+	spin_unlock_irqrestore(&x->wait.lock, flags);
 	return ret;
 }
 EXPORT_SYMBOL(completion_done);
#111 lore raw

[tip:sched/urgent] sched: Make wakeup side and atomic variants of completion API irq safe

Commit-ID:  7539a3b3d1f892dd97eaf094134d7de55c13befe
Gitweb:     http://git.kernel.org/tip/7539a3b3d1f892dd97eaf094134d7de55c13befe
Author:     Rafael J. Wysocki <[email protected]>
AuthorDate: Sun, 13 Dec 2009 00:07:30 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Sun, 13 Dec 2009 08:12:46 +0100

sched: Make wakeup side and atomic variants of completion API irq safe

Alan Stern noticed that all the wakeup side (and atomic) variants of the completion APIs should be irq safe, but the newly introduced completion_done() and try_wait_for_completion() aren't. The use of the irq unsafe variants in IRQ contexts can cause crashes/hangs.

Fix the problem by making them use spin_lock_irqsave() and spin_lock_irqrestore().

Cc Linus Torvalds <[email protected]>
Cc Zhang Rui <[email protected]>
Cc pm list <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc David Chinner <[email protected]>
Cc Lachlan McIlroy <[email protected]>

LKML-Reference: <[email protected]>

---
 kernel/sched.c |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)
kernel/sched.c +6 -4
index ff39cad..8b3532f 100644
@@ -5908,14 +5908,15 @@ EXPORT_SYMBOL(wait_for_completion_killable);
  */
 bool try_wait_for_completion(struct completion *x)
 {
+	unsigned long flags;
 	int ret = 1;
 
-	spin_lock_irq(&x->wait.lock);
+	spin_lock_irqsave(&x->wait.lock, flags);
 	if (!x->done)
 		ret = 0;
 	else
 		x->done--;
-	spin_unlock_irq(&x->wait.lock);
+	spin_unlock_irqrestore(&x->wait.lock, flags);
 	return ret;
 }
 EXPORT_SYMBOL(try_wait_for_completion);
@@ -5930,12 +5931,13 @@ EXPORT_SYMBOL(try_wait_for_completion);
  */
 bool completion_done(struct completion *x)
 {
+	unsigned long flags;
 	int ret = 1;
 
-	spin_lock_irq(&x->wait.lock);
+	spin_lock_irqsave(&x->wait.lock, flags);
 	if (!x->done)
 		ret = 0;
-	spin_unlock_irq(&x->wait.lock);
+	spin_unlock_irqrestore(&x->wait.lock, flags);
 	return ret;
 }
 EXPORT_SYMBOL(completion_done);
#112 lore raw

Re: spinlock in completion_done() (was: Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33))

In reply to Ingo Molnar
Quoted 24 lines · Ingo Molnar
Quoted 12 lines · Rafael J. Wysocki
Quoted 10 lines · Alan Stern
Quoted 3 lines · Rafael J. Wysocki

BTW, is there a good reason why completion_done() doesn't use spin_lock_irqsave and spin_unlock_irqrestore? complete() and complete_all() use them, so why not here?

And likewise in try_wait_for_completion(). It looks like a bug. Maybe these routines were not intended to be called with interrupts disabled, but that requirement doesn't seem to be documented. And it isn't a natural requirement anyway.

Quoted 5 lines

OK, let's ask Ingo about that.

Ingo, is there any particular reason why completion_done() and try_wait_for_completion() don't use spin_lock_irqsave() and spin_unlock_irqrestore()?

that's a bug that should be fixed - all the wakeup side (and atomic) variants of completetion API should be irq safe.

Quoted 6 lines

It appears that these new completion APIs were added via the XFS tree about a year ago:

39d2f1a: [XFS] extend completions to provide XFS object flush requirements

Please Cc: scheduler folks to all scheduler patches.

If you haven't fixed it locally yet, would you mind me posting a fix?

Rafael

#113 lore raw

spinlock in completion_done() (was: Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33))

In reply to Alan Stern
Quoted 10 lines · Alan Stern
Quoted 3 lines · Rafael J. Wysocki

BTW, is there a good reason why completion_done() doesn't use spin_lock_irqsave and spin_unlock_irqrestore? complete() and complete_all() use them, so why not here?

And likewise in try_wait_for_completion(). It looks like a bug. Maybe these routines were not intended to be called with interrupts disabled, but that requirement doesn't seem to be documented. And it isn't a natural requirement anyway.

OK, let's ask Ingo about that.

Ingo, is there any particular reason why completion_done() and try_wait_for_completion() don't use spin_lock_irqsave() and spin_unlock_irqrestore()?

Rafael

#114 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 5 lines · Alan Stern

And likewise in try_wait_for_completion(). It looks like a bug. Maybe these routines were not intended to be called with interrupts disabled, but that requirement doesn't seem to be documented. And it isn't a natural requirement anyway.

'complete()' is supposed to be callable from interrupts, but the waiting ones aren't. But 'complete()' is all you should need to call from interrupts, so that's fine.

So I think completions should work, if done right. That whole "make the parent wait for all the children to complete" is fine in that sense. And I'll happily take such an approach if my rwlock thing doesn't work.

Linus

#115 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 5 lines · Alan Stern

And likewise in try_wait_for_completion(). It looks like a bug. Maybe these routines were not intended to be called with interrupts disabled, but that requirement doesn't seem to be documented. And it isn't a natural requirement anyway.

'complete()' is supposed to be callable from interrupts, but the waiting ones aren't. But 'complete()' is all you should need to call from interrupts, so that's fine.

So I think completions should work, if done right. That whole "make the parent wait for all the children to complete" is fine in that sense. And I'll happily take such an approach if my rwlock thing doesn't work.

Linus

#116 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 10 lines · Linus Torvalds
Quoted 5 lines · Alan Stern

And likewise in try_wait_for_completion(). It looks like a bug. Maybe these routines were not intended to be called with interrupts disabled, but that requirement doesn't seem to be documented. And it isn't a natural requirement anyway.

'complete()' is supposed to be callable from interrupts, but the waiting ones aren't. But 'complete()' is all you should need to call from interrupts, so that's fine.

And try_wait_for_completion()? The fact that it doesn't block makes it interrupt-safe. What's the point of having an interrupt-safe routine that you can't call from within interrupt handlers?

Even if nobody uses it that way now, there's no guarantee somebody won't attempt it in the future.

Quoted 3 lines

So I think completions should work, if done right. That whole "make the parent wait for all the children to complete" is fine in that sense. And I'll happily take such an approach if my rwlock thing doesn't work.

In principle the two approaches could be combined: Add an rwsem for use by children and a completion for off-tree[*] use. But that would certainly be overkill. Looping over children doesn't take a tremendous amount of time compared to a full system suspend.

Alan Stern

[*] "Off-tree" isn't really an appropriate term; these devices aren't "off" the tree. "Non-tree" would be better.

#117 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Linus Torvalds
Quoted 10 lines · Linus Torvalds
Quoted 5 lines · Alan Stern

And likewise in try_wait_for_completion(). It looks like a bug. Maybe these routines were not intended to be called with interrupts disabled, but that requirement doesn't seem to be documented. And it isn't a natural requirement anyway.

'complete()' is supposed to be callable from interrupts, but the waiting ones aren't. But 'complete()' is all you should need to call from interrupts, so that's fine.

And try_wait_for_completion()? The fact that it doesn't block makes it interrupt-safe. What's the point of having an interrupt-safe routine that you can't call from within interrupt handlers?

Even if nobody uses it that way now, there's no guarantee somebody won't attempt it in the future.

Quoted 3 lines

So I think completions should work, if done right. That whole "make the parent wait for all the children to complete" is fine in that sense. And I'll happily take such an approach if my rwlock thing doesn't work.

In principle the two approaches could be combined: Add an rwsem for use by children and a completion for off-tree[*] use. But that would certainly be overkill. Looping over children doesn't take a tremendous amount of time compared to a full system suspend.

Alan Stern

[*] "Off-tree" isn't really an appropriate term; these devices aren't "off" the tree. "Non-tree" would be better.

#118 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Rafael J. Wysocki
Quoted 3 lines · Rafael J. Wysocki

BTW, is there a good reason why completion_done() doesn't use spin_lock_irqsave and spin_unlock_irqrestore? complete() and complete_all() use them, so why not here?

And likewise in try_wait_for_completion(). It looks like a bug. Maybe these routines were not intended to be called with interrupts disabled, but that requirement doesn't seem to be documented. And it isn't a natural requirement anyway.

Alan Stern

#119 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 16 lines · Alan Stern
Quoted 5 lines · Rafael J. Wysocki
Quoted 2 lines

This is a little more awkward because it requires the parent to iterate through its children.

I can live with that.

Quoted 8 lines
Quoted 1 line

But it does solve the off-tree dependency problem for suspends.

That's a plus, but I still think we're trying to create a barrier-alike mechanism using lock.

There's one more possibility to consider, though. What if we use a completion instead of the flag + wait queue? It surely is a standard synchronization mechanism and it seems it might work here.

Quoted 5 lines

You're right. I should have thought of that. Linus's original approach couldn't use a completion because during suspend it needed to make one task (the parent) wait for a bunch of others (the children). But if you iterate through the children by hand, that objection no longer applies.

BTW, is there a good reason why completion_done() doesn't use spin_lock_irqsave and spin_unlock_irqrestore? complete() and complete_all() use them, so why not here?

Rafael

#120 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Rafael J. Wysocki
Quoted 5 lines · Rafael J. Wysocki
Quoted 2 lines

This is a little more awkward because it requires the parent to iterate through its children.

I can live with that.

Quoted 8 lines
Quoted 1 line

But it does solve the off-tree dependency problem for suspends.

That's a plus, but I still think we're trying to create a barrier-alike mechanism using lock.

There's one more possibility to consider, though. What if we use a completion instead of the flag + wait queue? It surely is a standard synchronization mechanism and it seems it might work here.

You're right. I should have thought of that. Linus's original approach couldn't use a completion because during suspend it needed to make one task (the parent) wait for a bunch of others (the children). But if you iterate through the children by hand, that objection no longer applies.

Alan Stern

#121 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 21 lines · Alan Stern
Quoted 7 lines · Rafael J. Wysocki

Suppose we use rwsem and during suspend each child uses a down_read() on a parent and then the parent uses down_write() on itself. What if, whatever the reason, the parent is a bit early and does the down_write() before one of the children has a chance to do the down_read()? Aren't we toast?

Do we need any direct protection against that or does it just work itself out in a way I just don't see right now?

That's not the way it should be done. Linus had children taking their parents' locks during suspend, which is simple but leads to difficulties.

Instead, the PM core should do a down_write() on each device before starting the device's async suspend routine, and an up_write() when the routine finishes. Parents should, at the start of their async routine, do down_read() on each of their children plus whatever other devices they need to wait for. The core can do the waiting for children part and the driver's suspend routine can handle any other waiting.

Quoted 2 lines

This is a little more awkward because it requires the parent to iterate through its children.

I can live with that.

Quoted 1 line

But it does solve the off-tree dependency problem for suspends.

That's a plus, but I still think we're trying to create a barrier-alike mechanism using lock.

There's one more possibility to consider, though. What if we use a completion instead of the flag + wait queue? It surely is a standard synchronization mechanism and it seems it might work here.

Rafael

#122 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 4 lines · Alan Stern

That's not the way it should be done. Linus had children taking their parents' locks during suspend, which is simple but leads to difficulties.

No it doesn't. Name them.

Quoted 3 lines

Instead, the PM core should do a down_write() on each device before starting the device's async suspend routine, and an up_write() when the routine finishes.

No you should NOT do that. If you do that, you serialize the suspend incorrectly and much too early. IOW, think a topology like this:

	a -> b -> c
	  \
	   > d -> e

where you'd want to suspend 'c' and 'e' asynchronously. If we do a 'down-write()' on b, then we'll delay until 'c' has suspended, an if we have ordered the nodes in the obvious depth-first order, we'll walk the PM device list in the order:

c b e d a

and now we'll serialize on 'b', waiting for 'c' to suspend. Which we do _not_ want to do, because the whole point was to suspend 'c' and 'e' together.

Quoted 4 lines

Parents should, at the start of their async routine,
do down_read() on each of their children plus whatever other devices they need to wait for. The core can do the waiting for children part and the driver's suspend routine can handle any other waiting.

Why?

That just complicates things. Compare to my simple locking scheme I've quoted several times.

Linus

#123 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 4 lines · Alan Stern

That's not the way it should be done. Linus had children taking their parents' locks during suspend, which is simple but leads to difficulties.

No it doesn't. Name them.

Quoted 3 lines

Instead, the PM core should do a down_write() on each device before starting the device's async suspend routine, and an up_write() when the routine finishes.

No you should NOT do that. If you do that, you serialize the suspend incorrectly and much too early. IOW, think a topology like this:

	a -> b -> c
	  \
	   > d -> e

where you'd want to suspend 'c' and 'e' asynchronously. If we do a 'down-write()' on b, then we'll delay until 'c' has suspended, an if we have ordered the nodes in the obvious depth-first order, we'll walk the PM device list in the order:

c b e d a

and now we'll serialize on 'b', waiting for 'c' to suspend. Which we do _not_ want to do, because the whole point was to suspend 'c' and 'e' together.

Quoted 4 lines

Parents should, at the start of their async routine,
do down_read() on each of their children plus whatever other devices they need to wait for. The core can do the waiting for children part and the driver's suspend routine can handle any other waiting.

Why?

That just complicates things. Compare to my simple locking scheme I've quoted several times.

Linus

#124 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 7 lines · Linus Torvalds
Quoted 4 lines · Alan Stern

That's not the way it should be done. Linus had children taking their parents' locks during suspend, which is simple but leads to difficulties.

No it doesn't. Name them.

Really.

Let me put this simply: I've told you guys how to do it simply, with _zero_ crap. No "iterating over children". No games. No data structures. No new infrastructure. Just a single new rwlock per device, and _trivial_ code.

So here's the challenge: try it my simple way first. I've quoted the code about five million times already. If you _actually_ see some problems, explain them. Don't make up stupid "iterate over each child" things. Don't claim totally made-up "leads to difficulties". Don't make it any more complicated than it needs to be.

Keep it simple. And once you have tried that simple approach, and you really can show why it doesn't work, THEN you can try something else.

But before you try the simple approach and explain why it wouldn't work, I simply will not pull anything more complex. Understood and agreed?

Linus

#125 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Linus Torvalds
Quoted 7 lines · Linus Torvalds
Quoted 4 lines · Alan Stern

That's not the way it should be done. Linus had children taking their parents' locks during suspend, which is simple but leads to difficulties.

No it doesn't. Name them.

Really.

Let me put this simply: I've told you guys how to do it simply, with _zero_ crap. No "iterating over children". No games. No data structures. No new infrastructure. Just a single new rwlock per device, and _trivial_ code.

So here's the challenge: try it my simple way first. I've quoted the code about five million times already. If you _actually_ see some problems, explain them. Don't make up stupid "iterate over each child" things. Don't claim totally made-up "leads to difficulties". Don't make it any more complicated than it needs to be.

Keep it simple. And once you have tried that simple approach, and you really can show why it doesn't work, THEN you can try something else.

But before you try the simple approach and explain why it wouldn't work, I simply will not pull anything more complex. Understood and agreed?

Linus

#126 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Linus Torvalds
Quoted 7 lines · Linus Torvalds
Quoted 4 lines · Alan Stern

That's not the way it should be done. Linus had children taking their parents' locks during suspend, which is simple but leads to difficulties.

No it doesn't. Name them.

Well, one difficulty. It arises only because we are contemplating having the PM core fire up the async tasks, rather than having the drivers' suspend routines launch them (the way your original proposal did -- the difficulty does not arise there).

Suppose A and B are unrelated devices and we need to impose the off-tree constraint that A suspends after B. With children taking their parent's lock, the way to prevent A from suspending too soon is by having B's suspend routine acquire A's lock.

But B's suspend routine runs entirely in an async task, because that task is started by the PM core and it does the method call. Hence by the time B's suspend routine is called, A may already have begun suspending -- it's too late to take A's lock. To make the locking work, B would have to acquire A's lock _before_ B's async task starts. Since the PM core is unaware of the off-tree dependency, there's no simple way to make it work.

Quoted 16 lines
Quoted 3 lines

Instead, the PM core should do a down_write() on each device before starting the device's async suspend routine, and an up_write() when the routine finishes.

No you should NOT do that. If you do that, you serialize the suspend incorrectly and much too early. IOW, think a topology like this:

	a -> b -> c
	  \
	   > d -> e

where you'd want to suspend 'c' and 'e' asynchronously. If we do a 'down-write()' on b, then we'll delay until 'c' has suspended, an if we have ordered the nodes in the obvious depth-first order, we'll walk the PM device list in the order:

Quoted 5 lines

c b e d a

and now we'll serialize on 'b', waiting for 'c' to suspend. Which we do _not_ want to do, because the whole point was to suspend 'c' and 'e' together.

You misunderstand. The suspend algorithm will look like this:

	dpm_suspend()
	{
		list_for_each_entry_reverse(dpm_list, dev) {
			down_write(dev->lock);
			async_schedule(device_suspend, dev);
		}
	}

	device_suspend(dev)
	{
		device_for_each_child(dev, child) {
			down_read(child->lock);
			up_read(child->lock);
		}
		dev->suspend(dev);	/* May do off-tree down+up pairs */
		up_write(dev->lock);
	}

With completions instead of rwsems, the down_write() changes to init_completion(), the up_write() changes to complete_all(), and the down_read()+up_read() pairs change to wait_for_completion().

So 'b' will wait for 'c' to suspend, as it must, but 'e' won't wait for anything.

Quoted 9 lines
Quoted 4 lines

Parents should, at the start of their async routine,
do down_read() on each of their children plus whatever other devices they need to wait for. The core can do the waiting for children part and the driver's suspend routine can handle any other waiting.

Why?

That just complicates things. Compare to my simple locking scheme I've quoted several times.

It is a little more complicated in that it involves explicitly iterating over children. But it is simpler in that it can use completions instead of rwsems and it avoids the off-tree dependency problem described above.

Alan Stern

#127 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 20 lines · Alan Stern
Quoted 7 lines · Linus Torvalds
Quoted 4 lines · Alan Stern

That's not the way it should be done. Linus had children taking their parents' locks during suspend, which is simple but leads to difficulties.

No it doesn't. Name them.

Well, one difficulty. It arises only because we are contemplating having the PM core fire up the async tasks, rather than having the drivers' suspend routines launch them (the way your original proposal did -- the difficulty does not arise there).

Suppose A and B are unrelated devices and we need to impose the off-tree constraint that A suspends after B. With children taking their parent's lock, the way to prevent A from suspending too soon is by having B's suspend routine acquire A's lock.

Quoted 7 lines

But B's suspend routine runs entirely in an async task, because that task is started by the PM core and it does the method call. Hence by the time B's suspend routine is called, A may already have begun suspending -- it's too late to take A's lock. To make the locking work, B would have to acquire A's lock _before_ B's async task starts. Since the PM core is unaware of the off-tree dependency, there's no simple way to make it work.

Do not set async_suspend for B and instead start your own async thread from its suspend callback. The parent-children synchronization is done by the core anyway (at least I'd do it that way), so the only thing you need to worry about is the extra dependency.

Quoted 7 lines
Quoted 2 lines

That just complicates things. Compare to my simple locking scheme I've quoted several times.

It is a little more complicated in that it involves explicitly iterating over children. But it is simpler in that it can use completions instead of rwsems and it avoids the off-tree dependency problem described above.

I would be slightly more comfortable using completions, but the rwsem-based approach is fine with me as well.

Rafael

#128 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 18 lines · Rafael J. Wysocki
Quoted 10 lines

Well, one difficulty. It arises only because we are contemplating having the PM core fire up the async tasks, rather than having the drivers' suspend routines launch them (the way your original proposal did -- the difficulty does not arise there).

Suppose A and B are unrelated devices and we need to impose the off-tree constraint that A suspends after B. With children taking their parent's lock, the way to prevent A from suspending too soon is by having B's suspend routine acquire A's lock.

Quoted 7 lines

But B's suspend routine runs entirely in an async task, because that task is started by the PM core and it does the method call. Hence by the time B's suspend routine is called, A may already have begun suspending -- it's too late to take A's lock. To make the locking work, B would have to acquire A's lock _before_ B's async task starts. Since the PM core is unaware of the off-tree dependency, there's no simple way to make it work.

Quoted 4 lines

Do not set async_suspend for B and instead start your own async thread from its suspend callback. The parent-children synchronization is done by the core anyway (at least I'd do it that way), so the only thing you need to worry about is the extra dependency.

I don't like that because it introduces "artificial" dependencies: It makes B depend on all the preceding synchronous suspends, even totally unrelated ones. But yes, it would work.

Quoted 2 lines

I would be slightly more comfortable using completions, but the rwsem-based approach is fine with me as well.

On the principle of making things as easy and foolproof as possible for driver authors, I also favor completions since it makes dealing with non-tree dependencies easier.

However either way would be okay. I do have to handle some non-tree dependencies in USB, but oddly enough they affect only resume, not suspend. So this "who starts the async task" issue doesn't apply.

Alan Stern

#129 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 20 lines · Alan Stern
Quoted 18 lines · Rafael J. Wysocki
Quoted 17 lines

Well, one difficulty. It arises only because we are contemplating having the PM core fire up the async tasks, rather than having the drivers' suspend routines launch them (the way your original proposal did -- the difficulty does not arise there).

Suppose A and B are unrelated devices and we need to impose the off-tree constraint that A suspends after B. With children taking their parent's lock, the way to prevent A from suspending too soon is by having B's suspend routine acquire A's lock.

But B's suspend routine runs entirely in an async task, because that task is started by the PM core and it does the method call. Hence by the time B's suspend routine is called, A may already have begun suspending -- it's too late to take A's lock. To make the locking work, B would have to acquire A's lock _before_ B's async task starts. Since the PM core is unaware of the off-tree dependency, there's no simple way to make it work.

Quoted 8 lines
Quoted 4 lines

Do not set async_suspend for B and instead start your own async thread from its suspend callback. The parent-children synchronization is done by the core anyway (at least I'd do it that way), so the only thing you need to worry about is the extra dependency.

I don't like that because it introduces "artificial" dependencies: It makes B depend on all the preceding synchronous suspends, even totally unrelated ones. But yes, it would work.

Well, unfortunately, it wouldn't, because (at least in the context of my last patch) the core would release the rwsems as soon as your suspend had returned. So you'd have to make your suspend wait for the async thread and that would make it pointless. So scratch that, it wasn't a good idea at all.

This leaves us with basically two options, where the first one is to use rwsems in a way that you've proposed (with iterating over children), and the second one is to use completions. In my opinion rwsems don't give us any advantage in this case, so I'd very much prefer to use completions.

Rafael

#130 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 9 lines · Rafael J. Wysocki
Quoted 3 lines

I don't like that because it introduces "artificial" dependencies: It makes B depend on all the preceding synchronous suspends, even totally unrelated ones. But yes, it would work.

Well, unfortunately, it wouldn't, because (at least in the context of my last patch) the core would release the rwsems as soon as your suspend had returned. So you'd have to make your suspend wait for the async thread and that would make it pointless. So scratch that, it wasn't a good idea at all.

Quoted 4 lines

This leaves us with basically two options, where the first one is to use rwsems in a way that you've proposed (with iterating over children), and the second one is to use completions. In my opinion rwsems don't give us any advantage in this case, so I'd very much prefer to use completions.

If you really want to add support for async suspend constraints, then completions are clearer than rwsems. If you don't care (and it's unlikely that anyone will need them in the near future) then you might as well stick with the current rwsem implementation and avoid iterating over children.

Alan Stern

#131 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Rafael J. Wysocki
Quoted 9 lines · Rafael J. Wysocki
Quoted 3 lines

I don't like that because it introduces "artificial" dependencies: It makes B depend on all the preceding synchronous suspends, even totally unrelated ones. But yes, it would work.

Well, unfortunately, it wouldn't, because (at least in the context of my last patch) the core would release the rwsems as soon as your suspend had returned. So you'd have to make your suspend wait for the async thread and that would make it pointless. So scratch that, it wasn't a good idea at all.

Quoted 4 lines

This leaves us with basically two options, where the first one is to use rwsems in a way that you've proposed (with iterating over children), and the second one is to use completions. In my opinion rwsems don't give us any advantage in this case, so I'd very much prefer to use completions.

If you really want to add support for async suspend constraints, then completions are clearer than rwsems. If you don't care (and it's unlikely that anyone will need them in the near future) then you might as well stick with the current rwsem implementation and avoid iterating over children.

Alan Stern

#132 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 20 lines · Alan Stern
Quoted 18 lines · Rafael J. Wysocki
Quoted 17 lines

Well, one difficulty. It arises only because we are contemplating having the PM core fire up the async tasks, rather than having the drivers' suspend routines launch them (the way your original proposal did -- the difficulty does not arise there).

Suppose A and B are unrelated devices and we need to impose the off-tree constraint that A suspends after B. With children taking their parent's lock, the way to prevent A from suspending too soon is by having B's suspend routine acquire A's lock.

But B's suspend routine runs entirely in an async task, because that task is started by the PM core and it does the method call. Hence by the time B's suspend routine is called, A may already have begun suspending -- it's too late to take A's lock. To make the locking work, B would have to acquire A's lock _before_ B's async task starts. Since the PM core is unaware of the off-tree dependency, there's no simple way to make it work.

Quoted 8 lines
Quoted 4 lines

Do not set async_suspend for B and instead start your own async thread from its suspend callback. The parent-children synchronization is done by the core anyway (at least I'd do it that way), so the only thing you need to worry about is the extra dependency.

I don't like that because it introduces "artificial" dependencies: It makes B depend on all the preceding synchronous suspends, even totally unrelated ones. But yes, it would work.

Well, unfortunately, it wouldn't, because (at least in the context of my last patch) the core would release the rwsems as soon as your suspend had returned. So you'd have to make your suspend wait for the async thread and that would make it pointless. So scratch that, it wasn't a good idea at all.

This leaves us with basically two options, where the first one is to use rwsems in a way that you've proposed (with iterating over children), and the second one is to use completions. In my opinion rwsems don't give us any advantage in this case, so I'd very much prefer to use completions.

Rafael

#133 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Rafael J. Wysocki
Quoted 18 lines · Rafael J. Wysocki
Quoted 10 lines

Well, one difficulty. It arises only because we are contemplating having the PM core fire up the async tasks, rather than having the drivers' suspend routines launch them (the way your original proposal did -- the difficulty does not arise there).

Suppose A and B are unrelated devices and we need to impose the off-tree constraint that A suspends after B. With children taking their parent's lock, the way to prevent A from suspending too soon is by having B's suspend routine acquire A's lock.

Quoted 7 lines

But B's suspend routine runs entirely in an async task, because that task is started by the PM core and it does the method call. Hence by the time B's suspend routine is called, A may already have begun suspending -- it's too late to take A's lock. To make the locking work, B would have to acquire A's lock _before_ B's async task starts. Since the PM core is unaware of the off-tree dependency, there's no simple way to make it work.

Quoted 4 lines

Do not set async_suspend for B and instead start your own async thread from its suspend callback. The parent-children synchronization is done by the core anyway (at least I'd do it that way), so the only thing you need to worry about is the extra dependency.

I don't like that because it introduces "artificial" dependencies: It makes B depend on all the preceding synchronous suspends, even totally unrelated ones. But yes, it would work.

Quoted 2 lines

I would be slightly more comfortable using completions, but the rwsem-based approach is fine with me as well.

On the principle of making things as easy and foolproof as possible for driver authors, I also favor completions since it makes dealing with non-tree dependencies easier.

However either way would be okay. I do have to handle some non-tree dependencies in USB, but oddly enough they affect only resume, not suspend. So this "who starts the async task" issue doesn't apply.

Alan Stern

#134 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 20 lines · Alan Stern
Quoted 7 lines · Linus Torvalds
Quoted 4 lines · Alan Stern

That's not the way it should be done. Linus had children taking their parents' locks during suspend, which is simple but leads to difficulties.

No it doesn't. Name them.

Well, one difficulty. It arises only because we are contemplating having the PM core fire up the async tasks, rather than having the drivers' suspend routines launch them (the way your original proposal did -- the difficulty does not arise there).

Suppose A and B are unrelated devices and we need to impose the off-tree constraint that A suspends after B. With children taking their parent's lock, the way to prevent A from suspending too soon is by having B's suspend routine acquire A's lock.

Quoted 7 lines

But B's suspend routine runs entirely in an async task, because that task is started by the PM core and it does the method call. Hence by the time B's suspend routine is called, A may already have begun suspending -- it's too late to take A's lock. To make the locking work, B would have to acquire A's lock _before_ B's async task starts. Since the PM core is unaware of the off-tree dependency, there's no simple way to make it work.

Do not set async_suspend for B and instead start your own async thread from its suspend callback. The parent-children synchronization is done by the core anyway (at least I'd do it that way), so the only thing you need to worry about is the extra dependency.

Quoted 7 lines
Quoted 2 lines

That just complicates things. Compare to my simple locking scheme I've quoted several times.

It is a little more complicated in that it involves explicitly iterating over children. But it is simpler in that it can use completions instead of rwsems and it avoids the off-tree dependency problem described above.

I would be slightly more comfortable using completions, but the rwsem-based approach is fine with me as well.

Rafael

#135 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 3 lines · Alan Stern

Suppose A and B are unrelated devices and we need to impose the off-tree constraint that A suspends after B.

Ah. Ok, I can imagine the off-tree constraints, but part of my "keep it simple" was to simply not do them. If there are constraints that aren't in the topology of the tree, then I simply don't think that async is worth it in the first place.

Quoted 11 lines

You misunderstand. The suspend algorithm will look like this:

	dpm_suspend()
	{
		list_for_each_entry_reverse(dpm_list, dev) {
			down_write(dev->lock);
			async_schedule(device_suspend, dev);
		}
	}

	device_suspend(dev)
Quoted 8 lines
	{
		device_for_each_child(dev, child) {
			down_read(child->lock);
			up_read(child->lock);
		}
		dev->suspend(dev);	/* May do off-tree down+up pairs */
		up_write(dev->lock);
	}

Ok, so the above I think work (and see my previous email: I think completions would be workable there too).

It's just that I think the "looping over children" is ugly, when I think that by doing it the other way around you can make the code simpler and only depend on the PM device list and a simple parent pointer access.

I also think that you are wrong that the above somehow protects against non-topological dependencies. If the device you want to keep delay yourself suspending for is after you in the list, the down_read() on that may succeed simply because it hasn't even done its down_write() yet and you got scheduled early.

But I guess you could do that by walking the list twice (first to lock them all, then to actually call the suspend function). That whole two-phase thing, except the first phase _only_ locks, and doesn't do any callbacks.

Linus

#136 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 3 lines · Linus Torvalds

It's just that I think the "looping over children" is ugly, when I think that by doing it the other way around you can make the code simpler and only depend on the PM device list and a simple parent pointer access.

I agree that it is uglier. The only advantage is in handling asynchronous non-tree suspend dependencies, of which we probably won't have very many. In fact, I don't know of _any_ offhand.

Interestingly, this non-tree dependency problem does not affect resume.

Quoted 5 lines

I also think that you are wrong that the above somehow protects against non-topological dependencies. If the device you want to keep delay yourself suspending for is after you in the list, the down_read() on that may succeed simply because it hasn't even done its down_write() yet and you got scheduled early.

You mean, if A comes before B in the list and A must suspend after B? Then A's down_read() on B _can't_ occur before B's down_write() on itself. The down_write() on B happens before the list_for_each_entry_reverse() iteration reaches A; it even happens before B's async task is launched.

Quoted 4 lines

But I guess you could do that by walking the list twice (first to lock them all, then to actually call the suspend function). That whole two-phase thing, except the first phase _only_ locks, and doesn't do any callbacks.

Not necessary.

Alan Stern

#137 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 2 lines · Alan Stern

You mean, if A comes before B in the list and A must suspend after B?

But if they are not topologically ordered, then A wouldn't necessarily be before B on the list in the first place.

Of course, if we've mucked with the list by hand and made sure the ordering is ok, then that's a different issue. But your whole point seemed to be that the device could impose its own ordering in its suspend callback, which is not true on its own without external ordering.

Linus

#138 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 6 lines · Linus Torvalds
Quoted 2 lines · Alan Stern

You mean, if A comes before B in the list and A must suspend after B?

But if they are not topologically ordered, then A wouldn't necessarily be before B on the list in the first place.

Okay, I see what you're getting at. Yes, this is quite true -- if A doesn't precede B in dpm_list then A can't safely wait for B to suspend. To put it another way, only list-compatible constraints are feasible.

This shouldn't be a problem. If it were we'd be seeing it right now, because A would _always_ suspend before B.

Quoted 4 lines

Of course, if we've mucked with the list by hand and made sure the ordering is ok, then that's a different issue. But your whole point seemed to be that the device could impose its own ordering in its suspend callback, which is not true on its own without external ordering.

No, sorry for not making it clearer. I was assuming all long that the non-tree constraints were compatible with the list ordering.

In fact these considerations already affect the USB resume operations, even without asynchronous resume. The code relies on the fact that the PCI layer registers sibling devices on a slot in order of increasing function number. There's no guarantee this will remain true in the future (it may already be wrong AFAIK), so putting in some explicit list manipulation is the prudent thing to do.

Alan Stern

#139 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 7 lines · Alan Stern

In fact these considerations already affect the USB resume operations, even without asynchronous resume. The code relies on the fact that the PCI layer registers sibling devices on a slot in order of increasing function number. There's no guarantee this will remain true in the future (it may already be wrong AFAIK), so putting in some explicit list manipulation is the prudent thing to do.

I do think we want to keep the slot ordering.

One of the silent issues that the device management code has always had is the whole notion of naming stability. Now, udev and various fancy naming schemes solve that at a higher level, but it is still the case that we _really_ want basic things like your PCI controllers to show up in stable order.

For example, it is _very_ inconvenient if things like PCI probing ends up allocating different bus numbers (or resource allocations) across reboots even if the hardware hasn't been changed. Just from a debuggability standpoint, that just ends up being a total disaster.

For example, we continually hit odd special cases where PCI resource allocation has some unexplained problem because there is some motherboard resource that is hidden and invisible to our allocator. They are rare in the sense that it's usually just a couple of odd laptops or something, but they are not rare in the sense that pretty much _every_ single time we change some resource allocation logic, we find one or two machines that have some issue.

Things like that would be total disasters if the core device layer then ended up also not having well-defined ordering. This is why I don't want to do asynchronous PCI device probing, for example (ie we probe the hardware synchronously, the PCI driver sets it all up synchronously, and the asynchronous portion is the non-PCI part if any - things like PHY detection, disk spinup etc).

So async things are fine, but they have _huge_ disadvantages, and I'll personally take reliability and a stable serial algorithm over an async one as far as possible.

That's partly why I realy did suggest that we do the async stuff purely in the USB layer, rather than try to put it deeper in the device layer. And if we do support it "natively" in the device layer like Rafael's latest patch, I still think we should be very very nervous about making devices async unless there is a measured - and very noticeable - advantage.

So I really don't want to push things any further than absolutely necessary. I do not think that something like "embedded audio" is a reason for async, for example.

Linus

#140 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 7 lines · Alan Stern

In fact these considerations already affect the USB resume operations, even without asynchronous resume. The code relies on the fact that the PCI layer registers sibling devices on a slot in order of increasing function number. There's no guarantee this will remain true in the future (it may already be wrong AFAIK), so putting in some explicit list manipulation is the prudent thing to do.

I do think we want to keep the slot ordering.

One of the silent issues that the device management code has always had is the whole notion of naming stability. Now, udev and various fancy naming schemes solve that at a higher level, but it is still the case that we _really_ want basic things like your PCI controllers to show up in stable order.

For example, it is _very_ inconvenient if things like PCI probing ends up allocating different bus numbers (or resource allocations) across reboots even if the hardware hasn't been changed. Just from a debuggability standpoint, that just ends up being a total disaster.

For example, we continually hit odd special cases where PCI resource allocation has some unexplained problem because there is some motherboard resource that is hidden and invisible to our allocator. They are rare in the sense that it's usually just a couple of odd laptops or something, but they are not rare in the sense that pretty much _every_ single time we change some resource allocation logic, we find one or two machines that have some issue.

Things like that would be total disasters if the core device layer then ended up also not having well-defined ordering. This is why I don't want to do asynchronous PCI device probing, for example (ie we probe the hardware synchronously, the PCI driver sets it all up synchronously, and the asynchronous portion is the non-PCI part if any - things like PHY detection, disk spinup etc).

So async things are fine, but they have _huge_ disadvantages, and I'll personally take reliability and a stable serial algorithm over an async one as far as possible.

That's partly why I realy did suggest that we do the async stuff purely in the USB layer, rather than try to put it deeper in the device layer. And if we do support it "natively" in the device layer like Rafael's latest patch, I still think we should be very very nervous about making devices async unless there is a measured - and very noticeable - advantage.

So I really don't want to push things any further than absolutely necessary. I do not think that something like "embedded audio" is a reason for async, for example.

Linus

#141 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 5 lines · Linus Torvalds

That's partly why I realy did suggest that we do the async stuff purely in the USB layer, rather than try to put it deeper in the device layer. And if we do support it "natively" in the device layer like Rafael's latest patch, I still think we should be very very nervous about making devices async unless there is a measured - and very noticeable - advantage.

Agreed. Arjan's measurements indicated that USB was one of the biggest offenders; everything else other than the PS/2 mouse was much faster. Given these results there isn't much incentive to do anything else asynchronously.

(However other devices not present on Arjan's machine may be a different story. Spinning up multiple external disks is a good example -- although here it may be necessary for the driver to take charge, because spinning up a disk requires a lot of power and doing too many of them at the same time could be bad.)

Alan Stern

#142 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Hi!

Quoted 11 lines
Quoted 5 lines

That's partly why I realy did suggest that we do the async stuff purely in the USB layer, rather than try to put it deeper in the device layer. And if we do support it "natively" in the device layer like Rafael's latest patch, I still think we should be very very nervous about making devices async unless there is a measured - and very noticeable - advantage.

Agreed. Arjan's measurements indicated that USB was one of the biggest offenders; everything else other than the PS/2 mouse was much faster. Given these results there isn't much incentive to do anything else asynchronously.

Quoted 5 lines

(However other devices not present on Arjan's machine may be a different story. Spinning up multiple external disks is a good example -- although here it may be necessary for the driver to take charge, because spinning up a disk requires a lot of power and doing too many of them at the same time could be bad.)

Well, system would better be able to supply enough current... because usb disks auto-sleep on their own, and then something like async ls -l /*/* would kill your machine...
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#143 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern

Hi!

Quoted 11 lines
Quoted 5 lines

That's partly why I realy did suggest that we do the async stuff purely in the USB layer, rather than try to put it deeper in the device layer. And if we do support it "natively" in the device layer like Rafael's latest patch, I still think we should be very very nervous about making devices async unless there is a measured - and very noticeable - advantage.

Agreed. Arjan's measurements indicated that USB was one of the biggest offenders; everything else other than the PS/2 mouse was much faster. Given these results there isn't much incentive to do anything else asynchronously.

Quoted 5 lines

(However other devices not present on Arjan's machine may be a different story. Spinning up multiple external disks is a good example -- although here it may be necessary for the driver to take charge, because spinning up a disk requires a lot of power and doing too many of them at the same time could be bad.)

Well, system would better be able to supply enough current... because usb disks auto-sleep on their own, and then something like async ls -l /*/* would kill your machine...
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#144 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds

That's partly why I realy did suggest that we do the async stuff purely in the USB layer, rather than try to put it deeper in the device layer. And if we do support it "natively" in the device layer like Rafael's latest patch, I still think we should be very very nervous about making devices async unless there is a measured - and very noticeable - advantage.

Agreed. Arjan's measurements indicated that USB was one of the biggest offenders; everything else other than the PS/2 mouse was much faster. Given these results there isn't much incentive to do anything else asynchronously.

(However other devices not present on Arjan's machine may be a different story. Spinning up multiple external disks is a good example -- although here it may be necessary for the driver to take charge, because spinning up a disk requires a lot of power and doing too many of them at the same time could be bad.)

Alan Stern

#145 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds
Quoted 2 lines · Alan Stern

You mean, if A comes before B in the list and A must suspend after B?

But if they are not topologically ordered, then A wouldn't necessarily be before B on the list in the first place.

Okay, I see what you're getting at. Yes, this is quite true -- if A doesn't precede B in dpm_list then A can't safely wait for B to suspend. To put it another way, only list-compatible constraints are feasible.

This shouldn't be a problem. If it were we'd be seeing it right now, because A would _always_ suspend before B.

Quoted 4 lines

Of course, if we've mucked with the list by hand and made sure the ordering is ok, then that's a different issue. But your whole point seemed to be that the device could impose its own ordering in its suspend callback, which is not true on its own without external ordering.

No, sorry for not making it clearer. I was assuming all long that the non-tree constraints were compatible with the list ordering.

In fact these considerations already affect the USB resume operations, even without asynchronous resume. The code relies on the fact that the PCI layer registers sibling devices on a slot in order of increasing function number. There's no guarantee this will remain true in the future (it may already be wrong AFAIK), so putting in some explicit list manipulation is the prudent thing to do.

Alan Stern

#146 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 2 lines · Alan Stern

You mean, if A comes before B in the list and A must suspend after B?

But if they are not topologically ordered, then A wouldn't necessarily be before B on the list in the first place.

Of course, if we've mucked with the list by hand and made sure the ordering is ok, then that's a different issue. But your whole point seemed to be that the device could impose its own ordering in its suspend callback, which is not true on its own without external ordering.

Linus

#147 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 9 lines · Alan Stern
Quoted 3 lines · Linus Torvalds

It's just that I think the "looping over children" is ugly, when I think that by doing it the other way around you can make the code simpler and only depend on the PM device list and a simple parent pointer access.

I agree that it is uglier. The only advantage is in handling asynchronous non-tree suspend dependencies, of which we probably won't have very many. In fact, I don't know of _any_ offhand.

There's some potential for this in embedded audio - it wants to bring down the entire embedded audio subsystem at once before the individual devices (and their parents) get suspended since bringing them down out of sync can result in audible artifacts. Depending on the system the suspend may take a noticable amount of time so it'd be nice to be able to run it asynchronously, though we don't currently do so.

At the minute we get away with this mostly through not being able to represent the cases that are likely to actually trip up over it.

Quoted 1 line

Interestingly, this non-tree dependency problem does not affect resume.

Embedded audio does potentially - the resume needs all the individual devices in the subsystem and can take a substantial proportion of the overall resume time. Currently we get away with a combination of assuming that all the drivers are live when we decide to start resuming them and using the ALSA userspace API to deal with bringing the resume out of line, but it's not ideal.

#148 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 9 lines · Alan Stern
Quoted 3 lines · Linus Torvalds

It's just that I think the "looping over children" is ugly, when I think that by doing it the other way around you can make the code simpler and only depend on the PM device list and a simple parent pointer access.

I agree that it is uglier. The only advantage is in handling asynchronous non-tree suspend dependencies, of which we probably won't have very many. In fact, I don't know of _any_ offhand.

There's some potential for this in embedded audio - it wants to bring down the entire embedded audio subsystem at once before the individual devices (and their parents) get suspended since bringing them down out of sync can result in audible artifacts. Depending on the system the suspend may take a noticable amount of time so it'd be nice to be able to run it asynchronously, though we don't currently do so.

At the minute we get away with this mostly through not being able to represent the cases that are likely to actually trip up over it.

Quoted 1 line

Interestingly, this non-tree dependency problem does not affect resume.

Embedded audio does potentially - the resume needs all the individual devices in the subsystem and can take a substantial proportion of the overall resume time. Currently we get away with a combination of assuming that all the drivers are live when we decide to start resuming them and using the ALSA userspace API to deal with bringing the resume out of line, but it's not ideal.

#149 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 11 lines · Mark Brown
Quoted 9 lines · Alan Stern
Quoted 3 lines · Linus Torvalds

It's just that I think the "looping over children" is ugly, when I think that by doing it the other way around you can make the code simpler and only depend on the PM device list and a simple parent pointer access.

I agree that it is uglier. The only advantage is in handling asynchronous non-tree suspend dependencies, of which we probably won't have very many. In fact, I don't know of _any_ offhand.

Quoted 6 lines

There's some potential for this in embedded audio - it wants to bring down the entire embedded audio subsystem at once before the individual devices (and their parents) get suspended since bringing them down out of sync can result in audible artifacts. Depending on the system the suspend may take a noticable amount of time so it'd be nice to be able to run it asynchronously, though we don't currently do so.

For something like bringing down the entire embedded audio subsystem, which isn't directly tied to a single device, you would probably be better off doing it when the PM core broadcasts a suspend notification (see register_pm_notifier() in include/linux/suspend.h). This occurs before any devices are suspended, so synchronization isn't an issue.

Quoted 11 lines

At the minute we get away with this mostly through not being able to represent the cases that are likely to actually trip up over it.

Quoted 1 line

Interestingly, this non-tree dependency problem does not affect resume.

Embedded audio does potentially - the resume needs all the individual devices in the subsystem and can take a substantial proportion of the overall resume time. Currently we get away with a combination of assuming that all the drivers are live when we decide to start resuming them and using the ALSA userspace API to deal with bringing the resume out of line, but it's not ideal.

You can do the same thing with the resume notifier.

Alan Stern

#150 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 11 lines · Alan Stern
Quoted 3 lines · Mark Brown

There's some potential for this in embedded audio - it wants to bring down the entire embedded audio subsystem at once before the individual devices (and their parents) get suspended since bringing them down out

For something like bringing down the entire embedded audio subsystem, which isn't directly tied to a single device, you would probably be better off doing it when the PM core broadcasts a suspend notification (see register_pm_notifier() in include/linux/suspend.h). This occurs before any devices are suspended, so synchronization isn't an issue.

I'm not convinced that helps with the fact that the suspend may take a long time - ideally we'd be able to start the suspend process off but let other things carry on while it completes without having to worry about something we're relying on getting suspended underneath us.

Quoted 4 lines
Quoted 2 lines

Embedded audio does potentially - the resume needs all the individual overall resume time. Currently we get away with a combination of

You can do the same thing with the resume notifier.

Similarly, the length of time the resume may take to complete means it'd be nice to start as soon as we've got the devices and complete it at our leisure. This is less pressing since we can tell the PM core we've resumed but still block userspace.

#151 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 13 lines · Mark Brown
Quoted 11 lines · Alan Stern
Quoted 3 lines · Mark Brown

There's some potential for this in embedded audio - it wants to bring down the entire embedded audio subsystem at once before the individual devices (and their parents) get suspended since bringing them down out

For something like bringing down the entire embedded audio subsystem, which isn't directly tied to a single device, you would probably be better off doing it when the PM core broadcasts a suspend notification (see register_pm_notifier() in include/linux/suspend.h). This occurs before any devices are suspended, so synchronization isn't an issue.

Quoted 4 lines

I'm not convinced that helps with the fact that the suspend may take a long time - ideally we'd be able to start the suspend process off but let other things carry on while it completes without having to worry about something we're relying on getting suspended underneath us.

The suspend procedure is oriented around device structures, and what you're talking about isn't. It's something separate which has to be finished before _any_ of the audio devices are suspended.

How long does it take to bring down the entire embedded audio subsystem? And how critical is the timing for typical systems?

Alan Stern

#152 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 10 lines · Alan Stern
Quoted 4 lines · Mark Brown

I'm not convinced that helps with the fact that the suspend may take a long time - ideally we'd be able to start the suspend process off but let other things carry on while it completes without having to worry about something we're relying on getting suspended underneath us.

The suspend procedure is oriented around device structures, and what you're talking about isn't. It's something separate which has to be finished before _any_ of the audio devices are suspended.

In this context the "subsystem" actually has a struct device associated with it so does appear in the device flow.

Quoted 2 lines

How long does it take to bring down the entire embedded audio subsystem? And how critical is the timing for typical systems?

Worst case is about a second for both resume and suspend which means two seconds total but it's very hardware dependant.

The latency budget for suspend and resume are both zero in an ideal world, users want to be able to suspend as much as possible which means they'd like it to take no perceptible time at the human level. Some hardware is at the point where that's getting realistic but the folks on older hardware still want to get as close to that as they can.

#153 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 10 lines · Alan Stern
Quoted 4 lines · Mark Brown

I'm not convinced that helps with the fact that the suspend may take a long time - ideally we'd be able to start the suspend process off but let other things carry on while it completes without having to worry about something we're relying on getting suspended underneath us.

The suspend procedure is oriented around device structures, and what you're talking about isn't. It's something separate which has to be finished before _any_ of the audio devices are suspended.

In this context the "subsystem" actually has a struct device associated with it so does appear in the device flow.

Quoted 2 lines

How long does it take to bring down the entire embedded audio subsystem? And how critical is the timing for typical systems?

Worst case is about a second for both resume and suspend which means two seconds total but it's very hardware dependant.

The latency budget for suspend and resume are both zero in an ideal world, users want to be able to suspend as much as possible which means they'd like it to take no perceptible time at the human level. Some hardware is at the point where that's getting realistic but the folks on older hardware still want to get as close to that as they can.

#154 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 5 lines · Mark Brown
Quoted 2 lines

How long does it take to bring down the entire embedded audio subsystem? And how critical is the timing for typical systems?

Worst case is about a second for both resume and suspend which means two seconds total but it's very hardware dependant.

I would seriously suggest just looking at the code itself.

Maybe the code is just plain sh*t? If we're talking embedded audio, we're generally talking SoC chips (maybe some external audio daughtercard), and quite frankly, it sounds to me like you're just wasting your own time. There is no way that kind of hardware really needs that much time.

We should not design the device infrastructure for crap coding.

Now, I can easily see one-second delays in code that simply has never been thought about or cared about it. We used to have things like that in the serial code where just probing for non-existent serial ports took half a second per port because there was a timeout.

But christ, using that as an argument for "we should do things asynchronously" sounds like a crazy idea. Why not just take a hard look at the driver in question, asking hard questions like "does it really need to do something horrible like that"?

Because bad coding is much more likely to be the real reason.

Linus

#155 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Mark Brown
Quoted 5 lines · Mark Brown
Quoted 2 lines

How long does it take to bring down the entire embedded audio subsystem? And how critical is the timing for typical systems?

Worst case is about a second for both resume and suspend which means two seconds total but it's very hardware dependant.

I would seriously suggest just looking at the code itself.

Maybe the code is just plain sh*t? If we're talking embedded audio, we're generally talking SoC chips (maybe some external audio daughtercard), and quite frankly, it sounds to me like you're just wasting your own time. There is no way that kind of hardware really needs that much time.

We should not design the device infrastructure for crap coding.

Now, I can easily see one-second delays in code that simply has never been thought about or cared about it. We used to have things like that in the serial code where just probing for non-existent serial ports took half a second per port because there was a timeout.

But christ, using that as an argument for "we should do things asynchronously" sounds like a crazy idea. Why not just take a hard look at the driver in question, asking hard questions like "does it really need to do something horrible like that"?

Because bad coding is much more likely to be the real reason.

Linus

#156 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 9 lines · Linus Torvalds
Quoted 2 lines · Mark Brown

Worst case is about a second for both resume and suspend which means two seconds total but it's very hardware dependant.

I would seriously suggest just looking at the code itself.

Maybe the code is just plain sh*t? If we're talking embedded audio, we're generally talking SoC chips (maybe some external audio daughtercard), and

Yes, usually this is a SoC plus one or more external devices handling the mixed signal parts of things all soldered down onto a board.

Quoted 2 lines

quite frankly, it sounds to me like you're just wasting your own time. There is no way that kind of hardware really needs that much time.

Some of the older hardware really does need that much time, sadly. More recent hardware got that down much lower (into the low hundreds of ms where it's much less of an issue but still present) and current generations basically don't have the problem any more but for worst case a second is a good approximation.

The problem comes when you've got audio outputs referenced to something other than ground which used to happen because no negative supplies were available in these systems. To bring these up from cold you need to bring the outputs up to the reference level but if you do that by just turning on the power you get an audible (often loud) noise in the output from the square(ish) waveform that results which users don't find acceptable.

The initial solution was to ramp the voltage on the outputs in such a way that the waveform that appears on the outputs isn't audible, which broadly boils down to ramping it slowly. People were very aware of the problems so later generations of devices added features which allowed this to happen much more quickly than the original implementations had, but still noticably slow in terms of the timescales people need.

Current generation hardware solves the problem by using charge pumps to provide a negative supply, allowing ground referenced outputs which are just a win all round for this and other reasons. They're fast enough to allow the power up to be brought completely in line with the start of the audio stream, taking this out of suspend and resume entirely.

Quoted 4 lines

Now, I can easily see one-second delays in code that simply has never been thought about or cared about it. We used to have things like that in the serial code where just probing for non-existent serial ports took half a second per port because there was a timeout.

It's a deliberate delay waiting for the voltages to ramp, there's plenty of things that need to be fixed or optimised in the code but those that are causing issues these days really are just explicitly inserted delays waiting for things to happen in hardware that do actually take that long.

Quoted 1 line

Because bad coding is much more likely to be the real reason.

Would that it were - you wouldn't believe the amount of time that's been spent over the years tuning for this.

#157 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 8 lines · Mark Brown

The problem comes when you've got audio outputs referenced to something other than ground which used to happen because no negative supplies were available in these systems. To bring these up from cold you need to bring the outputs up to the reference level but if you do that by just turning on the power you get an audible (often loud) noise in the output from the square(ish) waveform that results which users don't find acceptable.

Ouch. A second still sounds way too long - but whatever.

However, it sounds like the nice way to do that isn't by doing it synchronously in the suspend/resume code itself, but simply ramping it down (and up) from a timer. It would be asynchronous, but not because the suspend itself is in any way asynchronous.

Done right, it might even result in a nice volume fade of the sound (ie if the hw allows for it, stop the actual sound engine late on suspend, and start it early on resume, so that sound works _while_ the whole reference volume rampdown/up is going on)

Linus

#158 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Mark Brown
Quoted 8 lines · Mark Brown

The problem comes when you've got audio outputs referenced to something other than ground which used to happen because no negative supplies were available in these systems. To bring these up from cold you need to bring the outputs up to the reference level but if you do that by just turning on the power you get an audible (often loud) noise in the output from the square(ish) waveform that results which users don't find acceptable.

Ouch. A second still sounds way too long - but whatever.

However, it sounds like the nice way to do that isn't by doing it synchronously in the suspend/resume code itself, but simply ramping it down (and up) from a timer. It would be asynchronous, but not because the suspend itself is in any way asynchronous.

Done right, it might even result in a nice volume fade of the sound (ie if the hw allows for it, stop the actual sound engine late on suspend, and start it early on resume, so that sound works _while_ the whole reference volume rampdown/up is going on)

Linus

#159 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 11 lines · Linus Torvalds
Quoted 7 lines · Mark Brown

The problem comes when you've got audio outputs referenced to something other than ground which used to happen because no negative supplies were available in these systems. To bring these up from cold you need to bring the outputs up to the reference level but if you do that by just turning on the power you get an audible (often loud) noise in the output from the square(ish) waveform that results which users don't find acceptable.

Ouch. A second still sounds way too long - but whatever.

Yes, I think there's pretty much universal agreement on that :)

Hardware that needs a few hundred miliseconds is much more common at the minute (and like I say current generation hardware is basically unaffected), but it's the number I keep in mind when considering how bad things might be.

Quoted 4 lines

However, it sounds like the nice way to do that isn't by doing it synchronously in the suspend/resume code itself, but simply ramping it down (and up) from a timer. It would be asynchronous, but not because the suspend itself is in any way asynchronous.

We don't actually need a timer for most of this - generally the ramp is done by charging or discharging a capacitor through a resistor so you just set it going then wait, possibly in several stages with a little bit twiddling in the middle to speed things up which could be done off a timer.

Quoted 4 lines

Done right, it might even result in a nice volume fade of the sound (ie if the hw allows for it, stop the actual sound engine late on suspend, and start it early on resume, so that sound works _while_ the whole reference volume rampdown/up is going on)

The big issue with running off a partially ramped supply is that it can upset the analogue components - for example, if an amplifier is trying to handle a signal with an amplitude outside the supply range then it'll clip. But sometimes that approach does work and it does get used.

For resume we're pretty much taking care of it already by moving the resume out of the main device resume and using ALSA-specific stuff to keep audio streams stopped until we're done but for suspend we don't know the system is going down until the suspend starts and we do want to make sure we got the analogue into a known poweroff state so that we can control powerup properly.

#160 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 7 lines · Mark Brown

The problem comes when you've got audio outputs referenced to something other than ground which used to happen because no negative supplies were available in these systems. To bring these up from cold you need to bring the outputs up to the reference level but if you do that by just turning on the power you get an audible (often loud) noise in the output from the square(ish) waveform that results which users don't find acceptable.

Ouch. A second still sounds way too long - but whatever.

Yes, I think there's pretty much universal agreement on that :)

Hardware that needs a few hundred miliseconds is much more common at the minute (and like I say current generation hardware is basically unaffected), but it's the number I keep in mind when considering how bad things might be.

Quoted 4 lines

However, it sounds like the nice way to do that isn't by doing it synchronously in the suspend/resume code itself, but simply ramping it down (and up) from a timer. It would be asynchronous, but not because the suspend itself is in any way asynchronous.

We don't actually need a timer for most of this - generally the ramp is done by charging or discharging a capacitor through a resistor so you just set it going then wait, possibly in several stages with a little bit twiddling in the middle to speed things up which could be done off a timer.

Quoted 4 lines

Done right, it might even result in a nice volume fade of the sound (ie if the hw allows for it, stop the actual sound engine late on suspend, and start it early on resume, so that sound works _while_ the whole reference volume rampdown/up is going on)

The big issue with running off a partially ramped supply is that it can upset the analogue components - for example, if an amplifier is trying to handle a signal with an amplitude outside the supply range then it'll clip. But sometimes that approach does work and it does get used.

For resume we're pretty much taking care of it already by moving the resume out of the main device resume and using ALSA-specific stuff to keep audio streams stopped until we're done but for suspend we don't know the system is going down until the suspend starts and we do want to make sure we got the analogue into a known poweroff state so that we can control powerup properly.

#161 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Linus Torvalds
Quoted 9 lines · Linus Torvalds
Quoted 2 lines · Mark Brown

Worst case is about a second for both resume and suspend which means two seconds total but it's very hardware dependant.

I would seriously suggest just looking at the code itself.

Maybe the code is just plain sh*t? If we're talking embedded audio, we're generally talking SoC chips (maybe some external audio daughtercard), and

Yes, usually this is a SoC plus one or more external devices handling the mixed signal parts of things all soldered down onto a board.

Quoted 2 lines

quite frankly, it sounds to me like you're just wasting your own time. There is no way that kind of hardware really needs that much time.

Some of the older hardware really does need that much time, sadly. More recent hardware got that down much lower (into the low hundreds of ms where it's much less of an issue but still present) and current generations basically don't have the problem any more but for worst case a second is a good approximation.

The problem comes when you've got audio outputs referenced to something other than ground which used to happen because no negative supplies were available in these systems. To bring these up from cold you need to bring the outputs up to the reference level but if you do that by just turning on the power you get an audible (often loud) noise in the output from the square(ish) waveform that results which users don't find acceptable.

The initial solution was to ramp the voltage on the outputs in such a way that the waveform that appears on the outputs isn't audible, which broadly boils down to ramping it slowly. People were very aware of the problems so later generations of devices added features which allowed this to happen much more quickly than the original implementations had, but still noticably slow in terms of the timescales people need.

Current generation hardware solves the problem by using charge pumps to provide a negative supply, allowing ground referenced outputs which are just a win all round for this and other reasons. They're fast enough to allow the power up to be brought completely in line with the start of the audio stream, taking this out of suspend and resume entirely.

Quoted 4 lines

Now, I can easily see one-second delays in code that simply has never been thought about or cared about it. We used to have things like that in the serial code where just probing for non-existent serial ports took half a second per port because there was a timeout.

It's a deliberate delay waiting for the voltages to ramp, there's plenty of things that need to be fixed or optimised in the code but those that are causing issues these days really are just explicitly inserted delays waiting for things to happen in hardware that do actually take that long.

Quoted 1 line

Because bad coding is much more likely to be the real reason.

Would that it were - you wouldn't believe the amount of time that's been spent over the years tuning for this.

#162 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Mark Brown
Quoted 5 lines · Mark Brown
Quoted 2 lines

How long does it take to bring down the entire embedded audio subsystem? And how critical is the timing for typical systems?

Worst case is about a second for both resume and suspend which means two seconds total but it's very hardware dependant.

A second seems awfully long. What happens if audio isn't being played when the suspend occurs? Can't you shorten things with no artifacts in that case?

If audio _is_ being played when a suspend occurs, users probably don't mind audible artifacts. In fact, they probably expect some.

Alan Stern

#163 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 3 lines · Alan Stern

If audio _is_ being played when a suspend occurs, users probably don't mind audible artifacts. In fact, they probably expect some.

I'd say it's physically impossible not to get them. If you're really suspending your audio hardware, it _will_ be quiet ;)

I suspect somebody is draining existing queues or something, or just probing for an external analog part. Neither of which is really sensible or absolutely required in an embedded suspend/resume kind of situation.

Especially for STR, just "leave all the data structures around, and just stop the DMA engine" is often a perfectly fine solution - but drivers don't do it, exactly because we've often had the mentality that you re-initialize everything under the sun.

I can see _why_ a driver would do that ("we re-use the same code that we use on close/open or module unload/reload"), but it doesn't change the fact that it's stupid to do if you worry about latency.

And yeah, turning it async might hide the problem. But the code word there is "hide" rather than "fix".

Linus

#164 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 3 lines · Alan Stern

If audio _is_ being played when a suspend occurs, users probably don't mind audible artifacts. In fact, they probably expect some.

I'd say it's physically impossible not to get them. If you're really suspending your audio hardware, it _will_ be quiet ;)

I suspect somebody is draining existing queues or something, or just probing for an external analog part. Neither of which is really sensible or absolutely required in an embedded suspend/resume kind of situation.

Especially for STR, just "leave all the data structures around, and just stop the DMA engine" is often a perfectly fine solution - but drivers don't do it, exactly because we've often had the mentality that you re-initialize everything under the sun.

I can see _why_ a driver would do that ("we re-use the same code that we use on close/open or module unload/reload"), but it doesn't change the fact that it's stupid to do if you worry about latency.

And yeah, turning it async might hide the problem. But the code word there is "hide" rather than "fix".

Linus

#165 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 8 lines · Alan Stern
Quoted 2 lines · Mark Brown

Worst case is about a second for both resume and suspend which means two seconds total but it's very hardware dependant.

A second seems awfully long. What happens if audio isn't being played when the suspend occurs? Can't you shorten things with no artifacts in that case?

For the affected hardware the problem is basically the same with or without audio being played. As I said in my reply to Linus this is delays caused by ramping reference voltages. These delays are sufficiently long that the reference voltages have to be maintained all the time so that they don't delay the start of audio streams which means that having or not having an audio stream at suspend time doesn't affect the reference voltage ramps since we don't turn them off when not in use. There is a win from other stuff having been shut off already, but it's already being exploited.

On suspend the problem is the same as for resume - we need to ramp the voltages quietly, this time down to zero. We want to make sure they're actually at zero to ensure that the ramp at resume time starts from a known hardware state.

#166 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 8 lines · Alan Stern
Quoted 2 lines · Mark Brown

Worst case is about a second for both resume and suspend which means two seconds total but it's very hardware dependant.

A second seems awfully long. What happens if audio isn't being played when the suspend occurs? Can't you shorten things with no artifacts in that case?

For the affected hardware the problem is basically the same with or without audio being played. As I said in my reply to Linus this is delays caused by ramping reference voltages. These delays are sufficiently long that the reference voltages have to be maintained all the time so that they don't delay the start of audio streams which means that having or not having an audio stream at suspend time doesn't affect the reference voltage ramps since we don't turn them off when not in use. There is a win from other stuff having been shut off already, but it's already being exploited.

On suspend the problem is the same as for resume - we need to ramp the voltages quietly, this time down to zero. We want to make sure they're actually at zero to ensure that the ramp at resume time starts from a known hardware state.

#167 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Mark Brown
Quoted 5 lines · Mark Brown
Quoted 2 lines

How long does it take to bring down the entire embedded audio subsystem? And how critical is the timing for typical systems?

Worst case is about a second for both resume and suspend which means two seconds total but it's very hardware dependant.

A second seems awfully long. What happens if audio isn't being played when the suspend occurs? Can't you shorten things with no artifacts in that case?

If audio _is_ being played when a suspend occurs, users probably don't mind audible artifacts. In fact, they probably expect some.

Alan Stern

#168 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Mark Brown
Quoted 13 lines · Mark Brown
Quoted 11 lines · Alan Stern
Quoted 3 lines · Mark Brown

There's some potential for this in embedded audio - it wants to bring down the entire embedded audio subsystem at once before the individual devices (and their parents) get suspended since bringing them down out

For something like bringing down the entire embedded audio subsystem, which isn't directly tied to a single device, you would probably be better off doing it when the PM core broadcasts a suspend notification (see register_pm_notifier() in include/linux/suspend.h). This occurs before any devices are suspended, so synchronization isn't an issue.

Quoted 4 lines

I'm not convinced that helps with the fact that the suspend may take a long time - ideally we'd be able to start the suspend process off but let other things carry on while it completes without having to worry about something we're relying on getting suspended underneath us.

The suspend procedure is oriented around device structures, and what you're talking about isn't. It's something separate which has to be finished before _any_ of the audio devices are suspended.

How long does it take to bring down the entire embedded audio subsystem? And how critical is the timing for typical systems?

Alan Stern

#169 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 11 lines · Alan Stern
Quoted 3 lines · Mark Brown

There's some potential for this in embedded audio - it wants to bring down the entire embedded audio subsystem at once before the individual devices (and their parents) get suspended since bringing them down out

For something like bringing down the entire embedded audio subsystem, which isn't directly tied to a single device, you would probably be better off doing it when the PM core broadcasts a suspend notification (see register_pm_notifier() in include/linux/suspend.h). This occurs before any devices are suspended, so synchronization isn't an issue.

I'm not convinced that helps with the fact that the suspend may take a long time - ideally we'd be able to start the suspend process off but let other things carry on while it completes without having to worry about something we're relying on getting suspended underneath us.

Quoted 4 lines
Quoted 2 lines

Embedded audio does potentially - the resume needs all the individual overall resume time. Currently we get away with a combination of

You can do the same thing with the resume notifier.

Similarly, the length of time the resume may take to complete means it'd be nice to start as soon as we've got the devices and complete it at our leisure. This is less pressing since we can tell the PM core we've resumed but still block userspace.

#170 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Mark Brown
Quoted 11 lines · Mark Brown
Quoted 9 lines · Alan Stern
Quoted 3 lines · Linus Torvalds

It's just that I think the "looping over children" is ugly, when I think that by doing it the other way around you can make the code simpler and only depend on the PM device list and a simple parent pointer access.

I agree that it is uglier. The only advantage is in handling asynchronous non-tree suspend dependencies, of which we probably won't have very many. In fact, I don't know of _any_ offhand.

Quoted 6 lines

There's some potential for this in embedded audio - it wants to bring down the entire embedded audio subsystem at once before the individual devices (and their parents) get suspended since bringing them down out of sync can result in audible artifacts. Depending on the system the suspend may take a noticable amount of time so it'd be nice to be able to run it asynchronously, though we don't currently do so.

For something like bringing down the entire embedded audio subsystem, which isn't directly tied to a single device, you would probably be better off doing it when the PM core broadcasts a suspend notification (see register_pm_notifier() in include/linux/suspend.h). This occurs before any devices are suspended, so synchronization isn't an issue.

Quoted 11 lines

At the minute we get away with this mostly through not being able to represent the cases that are likely to actually trip up over it.

Quoted 1 line

Interestingly, this non-tree dependency problem does not affect resume.

Embedded audio does potentially - the resume needs all the individual devices in the subsystem and can take a substantial proportion of the overall resume time. Currently we get away with a combination of assuming that all the drivers are live when we decide to start resuming them and using the ALSA userspace API to deal with bringing the resume out of line, but it's not ideal.

You can do the same thing with the resume notifier.

Alan Stern

#171 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

It's just that I think the "looping over children" is ugly, when I think that by doing it the other way around you can make the code simpler and only depend on the PM device list and a simple parent pointer access.

I agree that it is uglier. The only advantage is in handling asynchronous non-tree suspend dependencies, of which we probably won't have very many. In fact, I don't know of _any_ offhand.

Interestingly, this non-tree dependency problem does not affect resume.

Quoted 5 lines

I also think that you are wrong that the above somehow protects against non-topological dependencies. If the device you want to keep delay yourself suspending for is after you in the list, the down_read() on that may succeed simply because it hasn't even done its down_write() yet and you got scheduled early.

You mean, if A comes before B in the list and A must suspend after B? Then A's down_read() on B _can't_ occur before B's down_write() on itself. The down_write() on B happens before the list_for_each_entry_reverse() iteration reaches A; it even happens before B's async task is launched.

Quoted 4 lines

But I guess you could do that by walking the list twice (first to lock them all, then to actually call the suspend function). That whole two-phase thing, except the first phase _only_ locks, and doesn't do any callbacks.

Not necessary.

Alan Stern

#172 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Alan Stern
Quoted 3 lines · Alan Stern

Suppose A and B are unrelated devices and we need to impose the off-tree constraint that A suspends after B.

Ah. Ok, I can imagine the off-tree constraints, but part of my "keep it simple" was to simply not do them. If there are constraints that aren't in the topology of the tree, then I simply don't think that async is worth it in the first place.

Quoted 11 lines

You misunderstand. The suspend algorithm will look like this:

	dpm_suspend()
	{
		list_for_each_entry_reverse(dpm_list, dev) {
			down_write(dev->lock);
			async_schedule(device_suspend, dev);
		}
	}

	device_suspend(dev)
Quoted 8 lines
	{
		device_for_each_child(dev, child) {
			down_read(child->lock);
			up_read(child->lock);
		}
		dev->suspend(dev);	/* May do off-tree down+up pairs */
		up_write(dev->lock);
	}

Ok, so the above I think work (and see my previous email: I think completions would be workable there too).

It's just that I think the "looping over children" is ugly, when I think that by doing it the other way around you can make the code simpler and only depend on the PM device list and a simple parent pointer access.

I also think that you are wrong that the above somehow protects against non-topological dependencies. If the device you want to keep delay yourself suspending for is after you in the list, the down_read() on that may succeed simply because it hasn't even done its down_write() yet and you got scheduled early.

But I guess you could do that by walking the list twice (first to lock them all, then to actually call the suspend function). That whole two-phase thing, except the first phase _only_ locks, and doesn't do any callbacks.

Linus

#173 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Linus Torvalds
Quoted 7 lines · Linus Torvalds
Quoted 4 lines · Alan Stern

That's not the way it should be done. Linus had children taking their parents' locks during suspend, which is simple but leads to difficulties.

No it doesn't. Name them.

Well, one difficulty. It arises only because we are contemplating having the PM core fire up the async tasks, rather than having the drivers' suspend routines launch them (the way your original proposal did -- the difficulty does not arise there).

Suppose A and B are unrelated devices and we need to impose the off-tree constraint that A suspends after B. With children taking their parent's lock, the way to prevent A from suspending too soon is by having B's suspend routine acquire A's lock.

But B's suspend routine runs entirely in an async task, because that task is started by the PM core and it does the method call. Hence by the time B's suspend routine is called, A may already have begun suspending -- it's too late to take A's lock. To make the locking work, B would have to acquire A's lock _before_ B's async task starts. Since the PM core is unaware of the off-tree dependency, there's no simple way to make it work.

Quoted 16 lines
Quoted 3 lines

Instead, the PM core should do a down_write() on each device before starting the device's async suspend routine, and an up_write() when the routine finishes.

No you should NOT do that. If you do that, you serialize the suspend incorrectly and much too early. IOW, think a topology like this:

	a -> b -> c
	  \
	   > d -> e

where you'd want to suspend 'c' and 'e' asynchronously. If we do a 'down-write()' on b, then we'll delay until 'c' has suspended, an if we have ordered the nodes in the obvious depth-first order, we'll walk the PM device list in the order:

Quoted 5 lines

c b e d a

and now we'll serialize on 'b', waiting for 'c' to suspend. Which we do _not_ want to do, because the whole point was to suspend 'c' and 'e' together.

You misunderstand. The suspend algorithm will look like this:

	dpm_suspend()
	{
		list_for_each_entry_reverse(dpm_list, dev) {
			down_write(dev->lock);
			async_schedule(device_suspend, dev);
		}
	}

	device_suspend(dev)
	{
		device_for_each_child(dev, child) {
			down_read(child->lock);
			up_read(child->lock);
		}
		dev->suspend(dev);	/* May do off-tree down+up pairs */
		up_write(dev->lock);
	}

With completions instead of rwsems, the down_write() changes to init_completion(), the up_write() changes to complete_all(), and the down_read()+up_read() pairs change to wait_for_completion().

So 'b' will wait for 'c' to suspend, as it must, but 'e' won't wait for anything.

Quoted 9 lines
Quoted 4 lines

Parents should, at the start of their async routine,
do down_read() on each of their children plus whatever other devices they need to wait for. The core can do the waiting for children part and the driver's suspend routine can handle any other waiting.

Why?

That just complicates things. Compare to my simple locking scheme I've quoted several times.

It is a little more complicated in that it involves explicitly iterating over children. But it is simpler in that it can use completions instead of rwsems and it avoids the off-tree dependency problem described above.

Alan Stern

#174 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Rafael J. Wysocki
Quoted 7 lines · Rafael J. Wysocki

Suppose we use rwsem and during suspend each child uses a down_read() on a parent and then the parent uses down_write() on itself. What if, whatever the reason, the parent is a bit early and does the down_write() before one of the children has a chance to do the down_read()? Aren't we toast?

Do we need any direct protection against that or does it just work itself out in a way I just don't see right now?

That's not the way it should be done. Linus had children taking their parents' locks during suspend, which is simple but leads to difficulties.

Instead, the PM core should do a down_write() on each device before starting the device's async suspend routine, and an up_write() when the routine finishes. Parents should, at the start of their async routine, do down_read() on each of their children plus whatever other devices they need to wait for. The core can do the waiting for children part and the driver's suspend routine can handle any other waiting.

This is a little more awkward because it requires the parent to iterate through its children. But it does solve the off-tree dependency problem for suspends.

Alan Stern

#175 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Rafael J. Wysocki
Quoted 3 lines · Rafael J. Wysocki

Anyway, if we use an rwsem, it won't be checkable from interrupt context just as well.

You can't do a lock() from an interrupt, but the unlocks should be irq-safe.

Quoted 4 lines

Suppose we use rwsem and during suspend each child uses a down_read() on a parent and then the parent uses down_write() on itself. What if, whatever the reason, the parent is a bit early and does the down_write() before one of the children has a chance to do the down_read()? Aren't we toast?

We're toast, but we're toast for a totally unrealted reason: it means that you tried to resume a child before a parent, which would be a major bug to begin with.

Look, I even wrote out the comments, so let me repeat the code one more time.

- suspend time calling:

        // This won't block, because we suspend nodes before parents
        down_read(node->parent->lock);
        // Do the part that may block asynchronously
        async_schedule(do_usb_node_suspend, node);

- resume time calling:

        // This won't block, because we resume parents before children,
        // and the children will take the read lock. 
        down_write(leaf->lock);
        // Do the blocking part asynchronously
        async_schedule(usb_node_resume, leaf);

See? So when we take the parent lock for suspend, we are guaranteed to do so _before_ the parent node itself suspends. And conversely, when we take the parent lock (asynchronously) for resume, we're guaranteed to do that _after_ the parent node has done its own down_write.

And that all depends on just one trivial thing; that the suspend and resume is called in the right order (children first vs parent first respectively). And that is such a _major_ correctness issue that if that isn't correct, your suspend isn't going to work _anyway_.

Linus

#176 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 39 lines · Linus Torvalds
Quoted 3 lines · Rafael J. Wysocki

Anyway, if we use an rwsem, it won't be checkable from interrupt context just as well.

You can't do a lock() from an interrupt, but the unlocks should be irq-safe.

Quoted 4 lines

Suppose we use rwsem and during suspend each child uses a down_read() on a parent and then the parent uses down_write() on itself. What if, whatever the reason, the parent is a bit early and does the down_write() before one of the children has a chance to do the down_read()? Aren't we toast?

We're toast, but we're toast for a totally unrealted reason: it means that you tried to resume a child before a parent, which would be a major bug to begin with.

Look, I even wrote out the comments, so let me repeat the code one more time.

- suspend time calling:

        // This won't block, because we suspend nodes before parents
        down_read(node->parent->lock);
        // Do the part that may block asynchronously
        async_schedule(do_usb_node_suspend, node);

- resume time calling:

        // This won't block, because we resume parents before children,
        // and the children will take the read lock. 
        down_write(leaf->lock);
        // Do the blocking part asynchronously
        async_schedule(usb_node_resume, leaf);

See? So when we take the parent lock for suspend, we are guaranteed to do so _before_ the parent node itself suspends. And conversely, when we take the parent lock (asynchronously) for resume, we're guaranteed to do that _after_ the parent node has done its own down_write.

Quoted 4 lines

And that all depends on just one trivial thing; that the suspend and resume is called in the right order (children first vs parent first respectively). And that is such a _major_ correctness issue that if that isn't correct, your suspend isn't going to work _anyway_.

Understood (I think).

Let's try it, then. Below is the resume patch based on my previous one in this thread (I have only verified that it builds). Is that along the lines you want?

Rafael

---
 drivers/base/power/main.c    |   78 ++++++++++++++++++++++++++++++++++++++-----
 include/linux/device.h       |    6 +++
 include/linux/pm.h           |    3 +
 include/linux/resume-trace.h |    7 +++
 4 files changed, 85 insertions(+), 9 deletions(-)
include/linux/pm.h +3 -0
Index: linux-2.6/include/linux/pm.h
===================================================================
@@ -26,6 +26,7 @@
 #include <linux/spinlock.h>
 #include <linux/wait.h>
 #include <linux/timer.h>
+#include <linux/rwsem.h>
 
 /*
  * Callbacks for platform drivers to implement.
@@ -412,9 +413,11 @@ struct dev_pm_info {
 	pm_message_t		power_state;
 	unsigned int		can_wakeup:1;
 	unsigned int		should_wakeup:1;
+	unsigned		async_suspend:1;
 	enum dpm_state		status;		/* Owned by the PM core */
 #ifdef CONFIG_PM_SLEEP
 	struct list_head	entry;
+	struct rw_semaphore	rwsem;
 #endif
 #ifdef CONFIG_PM_RUNTIME
 	struct timer_list	suspend_timer;
include/linux/device.h +6 -0
Index: linux-2.6/include/linux/device.h
===================================================================
@@ -472,6 +472,12 @@ static inline int device_is_registered(s
 	return dev->kobj.state_in_sysfs;
 }
 
+static inline void device_enable_async_suspend(struct device *dev, bool enable)
+{
+	if (dev->power.status == DPM_ON)
+		dev->power.async_suspend = enable;
+}
+
 void driver_init(void);
 
 /*
drivers/base/power/main.c +69 -9

174 lines not on this page · raw message

include/linux/resume-trace.h +7 -0
Index: linux-2.6/include/linux/resume-trace.h
===================================================================
@@ -6,6 +6,11 @@
 
 extern int pm_trace_enabled;
 
+static inline int pm_trace_is_enabled(void)
+{
+       return pm_trace_enabled;
+}
+
 struct device;
 extern void set_trace_device(struct device *);
 extern void generate_resume_trace(const void *tracedata, unsigned int user);
@@ -17,6 +22,8 @@ extern void generate_resume_trace(const 
 
 #else
 
+static inline int pm_trace_is_enabled(void) { return 0; }
+
 #define TRACE_DEVICE(dev) do { } while (0)
 #define TRACE_RESUME(dev) do { } while (0)
 
#177 lore raw

Re: Async resume patch (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 45 lines · Rafael J. Wysocki
Quoted 39 lines · Linus Torvalds
Quoted 3 lines · Rafael J. Wysocki

Anyway, if we use an rwsem, it won't be checkable from interrupt context just as well.

You can't do a lock() from an interrupt, but the unlocks should be irq-safe.

Quoted 4 lines

Suppose we use rwsem and during suspend each child uses a down_read() on a parent and then the parent uses down_write() on itself. What if, whatever the reason, the parent is a bit early and does the down_write() before one of the children has a chance to do the down_read()? Aren't we toast?

We're toast, but we're toast for a totally unrealted reason: it means that you tried to resume a child before a parent, which would be a major bug to begin with.

Look, I even wrote out the comments, so let me repeat the code one more time.

- suspend time calling:

        // This won't block, because we suspend nodes before parents
        down_read(node->parent->lock);
        // Do the part that may block asynchronously
        async_schedule(do_usb_node_suspend, node);

- resume time calling:

        // This won't block, because we resume parents before children,
        // and the children will take the read lock. 
        down_write(leaf->lock);
        // Do the blocking part asynchronously
        async_schedule(usb_node_resume, leaf);

See? So when we take the parent lock for suspend, we are guaranteed to do so _before_ the parent node itself suspends. And conversely, when we take the parent lock (asynchronously) for resume, we're guaranteed to do that _after_ the parent node has done its own down_write.

Quoted 4 lines

And that all depends on just one trivial thing; that the suspend and resume is called in the right order (children first vs parent first respectively). And that is such a _major_ correctness issue that if that isn't correct, your suspend isn't going to work _anyway_.

Quoted 4 lines

Understood (I think).

Let's try it, then. Below is the resume patch based on my previous one in this thread (I have only verified that it builds).

Ah, I need to check if dev->parent is not NULL before trying to lock it, but apart from this it doesn't break things at least.

Rafael

#178 lore raw

Async suspend-resume patch w/ rwsems (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 46 lines · Rafael J. Wysocki
Quoted 45 lines · Rafael J. Wysocki
Quoted 43 lines · Linus Torvalds
Quoted 3 lines · Rafael J. Wysocki

Anyway, if we use an rwsem, it won't be checkable from interrupt context just as well.

You can't do a lock() from an interrupt, but the unlocks should be irq-safe.

Quoted 4 lines

Suppose we use rwsem and during suspend each child uses a down_read() on a parent and then the parent uses down_write() on itself. What if, whatever the reason, the parent is a bit early and does the down_write() before one of the children has a chance to do the down_read()? Aren't we toast?

We're toast, but we're toast for a totally unrealted reason: it means that you tried to resume a child before a parent, which would be a major bug to begin with.

Look, I even wrote out the comments, so let me repeat the code one more time.

- suspend time calling:

        // This won't block, because we suspend nodes before parents
        down_read(node->parent->lock);
        // Do the part that may block asynchronously
        async_schedule(do_usb_node_suspend, node);

- resume time calling:

        // This won't block, because we resume parents before children,
        // and the children will take the read lock. 
        down_write(leaf->lock);
        // Do the blocking part asynchronously
        async_schedule(usb_node_resume, leaf);

See? So when we take the parent lock for suspend, we are guaranteed to do so _before_ the parent node itself suspends. And conversely, when we take the parent lock (asynchronously) for resume, we're guaranteed to do that _after_ the parent node has done its own down_write.

And that all depends on just one trivial thing; that the suspend and resume is called in the right order (children first vs parent first respectively). And that is such a _major_ correctness issue that if that isn't correct, your suspend isn't going to work _anyway_.

Quoted 7 lines
Quoted 4 lines

Understood (I think).

Let's try it, then. Below is the resume patch based on my previous one in this thread (I have only verified that it builds).

Ah, I need to check if dev->parent is not NULL before trying to lock it, but apart from this it doesn't break things at least.

For completness, below is the full async suspend/resume patch with rwlocks, that has been (very slightly) tested and doesn't seem to break things.

[Note to Alan: lockdep doesn't seem to complain about the not annotated nested locks.]

Thanks,
Rafael

---
 drivers/base/power/main.c    |  195 +++++++++++++++++++++++++++++++++++++++----
 include/linux/device.h       |    6 +
 include/linux/pm.h           |    3 
 include/linux/resume-trace.h |    7 +
 4 files changed, 194 insertions(+), 17 deletions(-)
include/linux/pm.h +3 -0
Index: linux-2.6/include/linux/pm.h
===================================================================
@@ -26,6 +26,7 @@
 #include <linux/spinlock.h>
 #include <linux/wait.h>
 #include <linux/timer.h>
+#include <linux/rwsem.h>
 
 /*
  * Callbacks for platform drivers to implement.
@@ -412,9 +413,11 @@ struct dev_pm_info {
 	pm_message_t		power_state;
 	unsigned int		can_wakeup:1;
 	unsigned int		should_wakeup:1;
+	unsigned		async_suspend:1;
 	enum dpm_state		status;		/* Owned by the PM core */
 #ifdef CONFIG_PM_SLEEP
 	struct list_head	entry;
+	struct rw_semaphore	rwsem;
 #endif
 #ifdef CONFIG_PM_RUNTIME
 	struct timer_list	suspend_timer;
include/linux/device.h +6 -0
Index: linux-2.6/include/linux/device.h
===================================================================
@@ -472,6 +472,12 @@ static inline int device_is_registered(s
 	return dev->kobj.state_in_sysfs;
 }
 
+static inline void device_enable_async_suspend(struct device *dev, bool enable)
+{
+	if (dev->power.status == DPM_ON)
+		dev->power.async_suspend = enable;
+}
+
 void driver_init(void);
 
 /*
drivers/base/power/main.c +178 -17

376 lines not on this page · raw message

include/linux/resume-trace.h +7 -0
Index: linux-2.6/include/linux/resume-trace.h
===================================================================
@@ -6,6 +6,11 @@
 
 extern int pm_trace_enabled;
 
+static inline int pm_trace_is_enabled(void)
+{
+       return pm_trace_enabled;
+}
+
 struct device;
 extern void set_trace_device(struct device *);
 extern void generate_resume_trace(const void *tracedata, unsigned int user);
@@ -17,6 +22,8 @@ extern void generate_resume_trace(const 
 
 #else
 
+static inline int pm_trace_is_enabled(void) { return 0; }
+
 #define TRACE_DEVICE(dev) do { } while (0)
 #define TRACE_RESUME(dev) do { } while (0)
 
#179 lore raw

Re: Async suspend-resume patch w/ rwsems (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 52 lines · Rafael J. Wysocki
Quoted 51 lines · Rafael J. Wysocki
Quoted 49 lines · Rafael J. Wysocki
Quoted 39 lines · Linus Torvalds
Quoted 3 lines · Rafael J. Wysocki

Anyway, if we use an rwsem, it won't be checkable from interrupt context just as well.

You can't do a lock() from an interrupt, but the unlocks should be irq-safe.

Quoted 4 lines

Suppose we use rwsem and during suspend each child uses a down_read() on a parent and then the parent uses down_write() on itself. What if, whatever the reason, the parent is a bit early and does the down_write() before one of the children has a chance to do the down_read()? Aren't we toast?

We're toast, but we're toast for a totally unrealted reason: it means that you tried to resume a child before a parent, which would be a major bug to begin with.

Look, I even wrote out the comments, so let me repeat the code one more time.

- suspend time calling:

        // This won't block, because we suspend nodes before parents
        down_read(node->parent->lock);
        // Do the part that may block asynchronously
        async_schedule(do_usb_node_suspend, node);

- resume time calling:

        // This won't block, because we resume parents before children,
        // and the children will take the read lock. 
        down_write(leaf->lock);
        // Do the blocking part asynchronously
        async_schedule(usb_node_resume, leaf);

See? So when we take the parent lock for suspend, we are guaranteed to do so _before_ the parent node itself suspends. And conversely, when we take the parent lock (asynchronously) for resume, we're guaranteed to do that _after_ the parent node has done its own down_write.

Quoted 4 lines

And that all depends on just one trivial thing; that the suspend and resume is called in the right order (children first vs parent first respectively). And that is such a _major_ correctness issue that if that isn't correct, your suspend isn't going to work _anyway_.

Understood (I think).

Let's try it, then. Below is the resume patch based on my previous one in this thread (I have only verified that it builds).

Quoted 8 lines
Quoted 2 lines

Ah, I need to check if dev->parent is not NULL before trying to lock it, but apart from this it doesn't break things at least.

For completness, below is the full async suspend/resume patch with rwlocks, that has been (very slightly) tested and doesn't seem to break things.

[Note to Alan: lockdep doesn't seem to complain about the not annotated nested locks.]

BTW, I can easily change it so that it uses completions for synchronization, but I'm not sure if that's worth spending time on, so please let me know.

Rafael

#180 lore raw

Re: Async suspend-resume patch w/ rwsems (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Rafael J. Wysocki
Quoted 52 lines · Rafael J. Wysocki
Quoted 51 lines · Rafael J. Wysocki
Quoted 49 lines · Rafael J. Wysocki
Quoted 39 lines · Linus Torvalds
Quoted 3 lines · Rafael J. Wysocki

Anyway, if we use an rwsem, it won't be checkable from interrupt context just as well.

You can't do a lock() from an interrupt, but the unlocks should be irq-safe.

Quoted 4 lines

Suppose we use rwsem and during suspend each child uses a down_read() on a parent and then the parent uses down_write() on itself. What if, whatever the reason, the parent is a bit early and does the down_write() before one of the children has a chance to do the down_read()? Aren't we toast?

We're toast, but we're toast for a totally unrealted reason: it means that you tried to resume a child before a parent, which would be a major bug to begin with.

Look, I even wrote out the comments, so let me repeat the code one more time.

- suspend time calling:

        // This won't block, because we suspend nodes before parents
        down_read(node->parent->lock);
        // Do the part that may block asynchronously
        async_schedule(do_usb_node_suspend, node);

- resume time calling:

        // This won't block, because we resume parents before children,
        // and the children will take the read lock. 
        down_write(leaf->lock);
        // Do the blocking part asynchronously
        async_schedule(usb_node_resume, leaf);

See? So when we take the parent lock for suspend, we are guaranteed to do so _before_ the parent node itself suspends. And conversely, when we take the parent lock (asynchronously) for resume, we're guaranteed to do that _after_ the parent node has done its own down_write.

Quoted 4 lines

And that all depends on just one trivial thing; that the suspend and resume is called in the right order (children first vs parent first respectively). And that is such a _major_ correctness issue that if that isn't correct, your suspend isn't going to work _anyway_.

Understood (I think).

Let's try it, then. Below is the resume patch based on my previous one in this thread (I have only verified that it builds).

Quoted 8 lines
Quoted 2 lines

Ah, I need to check if dev->parent is not NULL before trying to lock it, but apart from this it doesn't break things at least.

For completness, below is the full async suspend/resume patch with rwlocks, that has been (very slightly) tested and doesn't seem to break things.

[Note to Alan: lockdep doesn't seem to complain about the not annotated nested locks.]

BTW, I can easily change it so that it uses completions for synchronization, but I'm not sure if that's worth spending time on, so please let me know.

Rafael

#181 lore raw

Re: Async suspend-resume patch w/ rwsems (was: Re: [GIT PULL] PM updates for 2.6.33)

In reply to Rafael J. Wysocki
Quoted 5 lines · Rafael J. Wysocki

For completness, below is the full async suspend/resume patch with rwlocks, that has been (very slightly) tested and doesn't seem to break things.

[Note to Alan: lockdep doesn't seem to complain about the not annotated nested locks.]

I can't imagine why not. And wouldn't lockdep get confused by the fact that in the async case, the rwsems are released by a different process from the one that acquired them?

drivers/base/power/main.c

Quoted 4 lines

Index: linux-2.6/drivers/base/power/main.c
===================================================================

drivers/base/power/main.c +0 -0

Should we have an attribute under /sys/power to disable async suspend/resume? It would make testing easier and give people a way to work around problems.

drivers/base/power/main.c static void pm_dev_err(struct device *de · line 341

Quoted 7 lines
drivers/base/power/main.c +1 -1
@@ -334,25 +337,53 @@ static void pm_dev_err(struct device *de
  * The driver of @dev will not receive interrupts while this function is being
  * executed.
  */
-static int device_resume_noirq(struct device *dev, pm_message_t state)
+static int __device_resume_noirq(struct device *dev, pm_message_t state)
 {

Do you want to use async tasks in the late-suspend/early-resume stages? I know that USB won't use it, not even for the PCI host controllers -- not unless the PCI core specifically wants it. Doing just the regular suspend/resume stages may be enough.

drivers/base/power/main.c static void pm_dev_err(struct device *de · line 378

Quoted 5 lines
drivers/base/power/main.c +5 -0
+static int device_resume_noirq(struct device *dev)
+{
+	down_write(&dev->power.rwsem);
+
+	if (dev->power.async_suspend && !pm_trace_is_enabled()) {

If the sysfs attribute exists, then maybe we _should_ allow async with PM tracing enabled. I don't know; it's your decision.

			atomic_set(&async_error, error);
	}

drivers/base/power/main.c static int dpm_suspend(pm_message_t stat · line 843

Quoted 10 lines
drivers/base/power/main.c +2 -0
@@ -683,10 +835,12 @@ static int dpm_suspend(pm_message_t stat
 
 	INIT_LIST_HEAD(&list);
 	mutex_lock(&dpm_list_mtx);
+	pm_transition = state;
 	while (!list_empty(&dpm_list)) {
 		struct device *dev = to_device(dpm_list.prev);
 
 		get_device(dev);
+		dev->power.status = DPM_OFF;

What's that for? dev->power.status is supposed to be DPM_SUSPENDING until the suspend method is successfully completed.

drivers/base/power/main.c static int dpm_suspend(pm_message_t stat · line 851

Quoted 8 lines
drivers/base/power/main.c +1 -0
 		mutex_unlock(&dpm_list_mtx);
 
 		error = device_suspend(dev, state);
@@ -694,16 +848,22 @@ static int dpm_suspend(pm_message_t stat
 		mutex_lock(&dpm_list_mtx);
 		if (error) {
 			pm_dev_err(dev, state, "", error);
+			dev->power.status = DPM_SUSPENDING;

And then this isn't needed.

drivers/base/power/main.c static int dpm_suspend(pm_message_t stat · line 700

Quoted 4 lines
drivers/base/power/main.c +0 -1
 			put_device(dev);
 			break;
 		}
-		dev->power.status = DPM_OFF;

This line has to be moved into __device_suspend(), even though it won't be protected by dpm_list_mtx. The same sort of thing applies to dpm_suspend_noirq() (although nothing needs to be moved if you don't make it async).

The rest looks okay.

How about exporting a wait_for_device_to_resume() routine? Drivers could call it for non-tree resume constraints:

	void wait_for_device_to_resume(struct device *other)
	{
		down_read(&other->power.rwsem);
		up_read(&other->power.rwsem);
	}

Unfortunately there is no equivalent for non-tree suspend constraints.

Alan Stern

#182 lore raw

Re: Async suspend-resume patch w/ rwsems (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 11 lines · Alan Stern
Quoted 5 lines · Rafael J. Wysocki

For completness, below is the full async suspend/resume patch with rwlocks, that has been (very slightly) tested and doesn't seem to break things.

[Note to Alan: lockdep doesn't seem to complain about the not annotated nested locks.]

I can't imagine why not. And wouldn't lockdep get confused by the fact that in the async case, the rwsems are released by a different process from the one that acquired them?

/me looks at the .config

I have CONFIG_LOCKDEP_SUPPORT set, is there anything else I need to set in .config?

Quoted 8 lines

drivers/base/power/main.c

Quoted 4 lines

Index: linux-2.6/drivers/base/power/main.c
===================================================================

drivers/base/power/main.c +0 -0

Should we have an attribute under /sys/power to disable async suspend/resume? It would make testing easier and give people a way to work around problems.

I have a separate patch adding that, but I'd prefer to focus on the core feature first, if possible.

Quoted 4 lines
Quoted 4 lines
drivers/base/power/main.c +0 -0
@@ -334,25 +337,53 @@ static void pm_dev_err(struct device *de
  * The driver of @dev will not receive interrupts while this function is being
  * executed.
  */
Quoted 8 lines

drivers/base/power/main.c static void pm_dev_err(struct device *de · line 341

Quoted 3 lines
drivers/base/power/main.c +1 -1
-static int device_resume_noirq(struct device *dev, pm_message_t state)
+static int __device_resume_noirq(struct device *dev, pm_message_t state)
 {

Do you want to use async tasks in the late-suspend/early-resume stages? I know that USB won't use it, not even for the PCI host controllers -- not unless the PCI core specifically wants it. Doing just the regular suspend/resume stages may be enough.

I guess so. It's a leftover from the time I thought PCI might use async suspend, but it didn't really speed up things at all AFAICS.

I think I'll remove it for now and it's going to be trivial to add it back if desired.

Quoted 8 lines

drivers/base/power/main.c static void pm_dev_err(struct device *de · line 378

Quoted 5 lines
drivers/base/power/main.c +5 -0
+static int device_resume_noirq(struct device *dev)
+{
+	down_write(&dev->power.rwsem);
+
+	if (dev->power.async_suspend && !pm_trace_is_enabled()) {

If the sysfs attribute exists, then maybe we _should_ allow async with PM tracing enabled. I don't know; it's your decision.

I don't think it would be reliable in that case, because the RTC might be written to by two concurrent threads at the same time.

Quoted 12 lines
			atomic_set(&async_error, error);
	}
Quoted 8 lines
drivers/base/power/main.c +1 -0
@@ -683,10 +835,12 @@ static int dpm_suspend(pm_message_t stat
 
 	INIT_LIST_HEAD(&list);
 	mutex_lock(&dpm_list_mtx);
+	pm_transition = state;
 	while (!list_empty(&dpm_list)) {
 		struct device *dev = to_device(dpm_list.prev);
 
Quoted 5 lines

drivers/base/power/main.c static int dpm_suspend(pm_message_t stat · line 843

Quoted 2 lines
drivers/base/power/main.c +1 -0
 		get_device(dev);
+		dev->power.status = DPM_OFF;

What's that for? dev->power.status is supposed to be DPM_SUSPENDING until the suspend method is successfully completed.

If the suspend is run asynchronoysly, the main thread will always get a "success" from device_suspend(), so it can't change power.status on this basis. I thought we could set power.status to DPM_OFF upfront and change it back when error is returned.

The alternative would be to move the modification of power.status to device_suspend() and async_suspend(). Well, maybe that's better.

Quoted 21 lines

drivers/base/power/main.c static int dpm_suspend(pm_message_t stat · line 851

Quoted 8 lines
drivers/base/power/main.c +1 -0
 		mutex_unlock(&dpm_list_mtx);
 
 		error = device_suspend(dev, state);
@@ -694,16 +848,22 @@ static int dpm_suspend(pm_message_t stat
 		mutex_lock(&dpm_list_mtx);
 		if (error) {
 			pm_dev_err(dev, state, "", error);
+			dev->power.status = DPM_SUSPENDING;

And then this isn't needed.

drivers/base/power/main.c static int dpm_suspend(pm_message_t stat · line 700

Quoted 4 lines
drivers/base/power/main.c +0 -1
 			put_device(dev);
 			break;
 		}
-		dev->power.status = DPM_OFF;

This line has to be moved into __device_suspend(), even though it won't be protected by dpm_list_mtx. The same sort of thing applies to dpm_suspend_noirq() (although nothing needs to be moved if you don't make it async).

Quoted 1 line

The rest looks okay.

Still, I think I'd rework it to use completions for the reason described in the message I've just sent (in short, because of the off-tree dependencies problem).

Quoted 10 lines

How about exporting a wait_for_device_to_resume() routine? Drivers could call it for non-tree resume constraints:

	void wait_for_device_to_resume(struct device *other)
	{
		down_read(&other->power.rwsem);
		up_read(&other->power.rwsem);
	}

Unfortunately there is no equivalent for non-tree suspend constraints.

If we use completions, it will be possible to just export something like

dpm_wait(dev)

{
        if (dev)
                wait_for_completion(dev->power.completion);
}

I think. It appears that will also work for suspend, unless I'm missing something.

Rafael

#183 lore raw

Re: Async suspend-resume patch w/ rwsems (was: Re: [GIT PULL] PM updates for 2.6.33)

Quoted 9 lines · Rafael J. Wysocki
Quoted 8 lines · Alan Stern
Quoted 5 lines · Rafael J. Wysocki

For completness, below is the full async suspend/resume patch with rwlocks, that has been (very slightly) tested and doesn't seem to break things.

[Note to Alan: lockdep doesn't seem to complain about the not annotated nested locks.]

Quoted 8 lines
Quoted 3 lines

I can't imagine why not. And wouldn't lockdep get confused by the fact that in the async case, the rwsems are released by a different process from the one that acquired them?

/me looks at the .config

I have CONFIG_LOCKDEP_SUPPORT set, is there anything else I need to set in .config?

How about CONFIG_PROVE_LOCKING? If lockdep really does start complaining then switching to completions would be a simple way to appease it.

Quoted 14 lines
Quoted 8 lines
Quoted 8 lines
drivers/base/power/main.c +1 -0
@@ -683,10 +835,12 @@ static int dpm_suspend(pm_message_t stat
 
 	INIT_LIST_HEAD(&list);
 	mutex_lock(&dpm_list_mtx);
+	pm_transition = state;
 	while (!list_empty(&dpm_list)) {
 		struct device *dev = to_device(dpm_list.prev);
 
Quoted 5 lines

drivers/base/power/main.c static int dpm_suspend(pm_message_t stat · line 843

Quoted 2 lines
drivers/base/power/main.c +1 -0
 		get_device(dev);
+		dev->power.status = DPM_OFF;

What's that for? dev->power.status is supposed to be DPM_SUSPENDING until the suspend method is successfully completed.

Quoted 7 lines

If the suspend is run asynchronoysly, the main thread will always get a "success" from device_suspend(), so it can't change power.status on this basis. I thought we could set power.status to DPM_OFF upfront and change it back when error is returned.

The alternative would be to move the modification of power.status to device_suspend() and async_suspend(). Well, maybe that's better.

Yes, I think so. Or into __device_suspend(). And the same thing in dpm_suspend_noirq().

Quoted 13 lines
Quoted 10 lines

How about exporting a wait_for_device_to_resume() routine? Drivers could call it for non-tree resume constraints:

	void wait_for_device_to_resume(struct device *other)
	{
		down_read(&other->power.rwsem);
		up_read(&other->power.rwsem);
	}

Unfortunately there is no equivalent for non-tree suspend constraints.

If we use completions, it will be possible to just export something like

Quoted 8 lines

dpm_wait(dev)

{
        if (dev)
                wait_for_completion(dev->power.completion);
}

I think. It appears that will also work for suspend, unless I'm missing something.

It will.

Alan Stern

#184 lore raw

Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 15 lines · Alan Stern
Quoted 13 lines · Rafael J. Wysocki
Quoted 11 lines · Alan Stern
Quoted 5 lines · Rafael J. Wysocki

For completness, below is the full async suspend/resume patch with rwlocks, that has been (very slightly) tested and doesn't seem to break things.

[Note to Alan: lockdep doesn't seem to complain about the not annotated nested locks.]

I can't imagine why not. And wouldn't lockdep get confused by the fact that in the async case, the rwsems are released by a different process from the one that acquired them?

Quoted 8 lines
Quoted 4 lines

/me looks at the .config

I have CONFIG_LOCKDEP_SUPPORT set, is there anything else I need to set in .config?

How about CONFIG_PROVE_LOCKING? If lockdep really does start complaining then switching to completions would be a simple way to appease it.

Ah, that one is not set. I guess I'll try it later, although I've already decided to use completions anyway.

...

Quoted 19 lines
Quoted 19 lines
Quoted 10 lines

How about exporting a wait_for_device_to_resume() routine? Drivers could call it for non-tree resume constraints:

	void wait_for_device_to_resume(struct device *other)
	{
		down_read(&other->power.rwsem);
		up_read(&other->power.rwsem);
	}

Unfortunately there is no equivalent for non-tree suspend constraints.

If we use completions, it will be possible to just export something like

dpm_wait(dev)

{
        if (dev)
                wait_for_completion(dev->power.completion);
}
Quoted 4 lines
Quoted 2 lines

I think. It appears that will also work for suspend, unless I'm missing something.

It will.

Completions it is, then.

Additionally, I've removed the async support from the _noirq parts and moved the setting of power.status on suspend to __device_suspend(). The result is appended.

Rafael

---
 drivers/base/power/main.c    |  124 ++++++++++++++++++++++++++++++++++++++++---
 include/linux/device.h       |    6 ++
 include/linux/pm.h           |   12 ++++
 include/linux/resume-trace.h |    7 ++
 4 files changed, 143 insertions(+), 6 deletions(-)
include/linux/pm.h +12 -0
Index: linux-2.6/include/linux/pm.h
===================================================================
@@ -26,6 +26,7 @@
 #include <linux/spinlock.h>
 #include <linux/wait.h>
 #include <linux/timer.h>
+#include <linux/completion.h>
 
 /*
  * Callbacks for platform drivers to implement.
@@ -412,9 +413,11 @@ struct dev_pm_info {
 	pm_message_t		power_state;
 	unsigned int		can_wakeup:1;
 	unsigned int		should_wakeup:1;
+	unsigned		async_suspend:1;
 	enum dpm_state		status;		/* Owned by the PM core */
 #ifdef CONFIG_PM_SLEEP
 	struct list_head	entry;
+	struct completion	completion;
 #endif
 #ifdef CONFIG_PM_RUNTIME
 	struct timer_list	suspend_timer;
@@ -508,6 +511,13 @@ extern void __suspend_report_result(cons
 		__suspend_report_result(__func__, fn, ret);		\
 	} while (0)
 
+extern int __dpm_wait(struct device *dev, void *ign);
+
+static inline void dpm_wait(struct device *dev)
+{
+	__dpm_wait(dev, NULL);
+}
+
 #else /* !CONFIG_PM_SLEEP */
 
 #define device_pm_lock() do {} while (0)
@@ -520,6 +530,8 @@ static inline int dpm_suspend_start(pm_m
 
 #define suspend_report_result(fn, ret)		do {} while (0)
 
+static inline void dpm_wait(struct device *dev) {}
+
 #endif /* !CONFIG_PM_SLEEP */
 
 /* How to reorder dpm_list after device_move() */
drivers/base/power/main.c +118 -6

250 lines not on this page · raw message

include/linux/resume-trace.h +7 -0
Index: linux-2.6/include/linux/resume-trace.h
===================================================================
@@ -6,6 +6,11 @@
 
 extern int pm_trace_enabled;
 
+static inline int pm_trace_is_enabled(void)
+{
+       return pm_trace_enabled;
+}
+
 struct device;
 extern void set_trace_device(struct device *);
 extern void generate_resume_trace(const void *tracedata, unsigned int user);
@@ -17,6 +22,8 @@ extern void generate_resume_trace(const 
 
 #else
 
+static inline int pm_trace_is_enabled(void) { return 0; }
+
 #define TRACE_DEVICE(dev) do { } while (0)
 #define TRACE_RESUME(dev) do { } while (0)
 
include/linux/device.h +6 -0
Index: linux-2.6/include/linux/device.h
===================================================================
@@ -472,6 +472,12 @@ static inline int device_is_registered(s
 	return dev->kobj.state_in_sysfs;
 }
 
+static inline void device_enable_async_suspend(struct device *dev, bool enable)
+{
+	if (dev->power.status == DPM_ON)
+		dev->power.async_suspend = enable;
+}
+
 void driver_init(void);
 
 /*
#185 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 2 lines · Rafael J. Wysocki

Completions it is, then.

What was so hard with the "Try the simple one first" to understand? You had a simpler working patch, why are you making this more complex one without ever having had any problems with the simpler one?

Btw, your 'atomic_set()' with errors is pure voodoo programming. That's not how atomics work. They do SMP-atomic addition etc, the 'atomic_set()' and 'atomic_read()' things are not in any way more atomic than any other access.

They are meant for racy reads (atomic_read()) and for initializations (atomic_set()), and the way you use them that 'atomic' part is entirely pointless, because it really isn't anything different from an 'int', except that it may be very very expensive on some architectures due to hashed spinlocks etc.

So stop this overdesign thing. Start simple. If you _ever_ see real problems, that's when you add stuff. As it is, any time you add complexity, you just add bugs.

Quoted 4 lines
drivers/base/power/main.c +4 -0
+/**
+ * dpm_synchronize - Wait for PM callbacks of all devices to complete.
+ */
+static void dpm_synchronize(void)

drivers/base/power/main.c void device_pm_move_last(struct device * · line 198

Quoted 10 lines
drivers/base/power/main.c +10 -0
+{
+	struct device *dev;
+
+	async_synchronize_full();
+
+	mutex_lock(&dpm_list_mtx);
+	list_for_each_entry(dev, &dpm_list, power.entry)
+		INIT_COMPLETION(dev->power.completion);
+	mutex_unlock(&dpm_list_mtx);
+}

And this, for example, is pretty disgusting. Not only is that INIT_COMPLETION purely brought on by the whole problem with completions (they are fundamentally one-shot, but you want to use them over and over so you need to re-initialize them: a nice lock wouldn't have that problem to begin with), but the comment isn't even accurate. Sure, it waits for any async jobs, but that's the _least_ of what the function actually does, so the comment is actively misleading, isn't it?

Linus

#186 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

In reply to Rafael J. Wysocki
Quoted 2 lines · Rafael J. Wysocki

Completions it is, then.

What was so hard with the "Try the simple one first" to understand? You had a simpler working patch, why are you making this more complex one without ever having had any problems with the simpler one?

Btw, your 'atomic_set()' with errors is pure voodoo programming. That's not how atomics work. They do SMP-atomic addition etc, the 'atomic_set()' and 'atomic_read()' things are not in any way more atomic than any other access.

They are meant for racy reads (atomic_read()) and for initializations (atomic_set()), and the way you use them that 'atomic' part is entirely pointless, because it really isn't anything different from an 'int', except that it may be very very expensive on some architectures due to hashed spinlocks etc.

So stop this overdesign thing. Start simple. If you _ever_ see real problems, that's when you add stuff. As it is, any time you add complexity, you just add bugs.

Quoted 4 lines
drivers/base/power/main.c +4 -0
+/**
+ * dpm_synchronize - Wait for PM callbacks of all devices to complete.
+ */
+static void dpm_synchronize(void)

drivers/base/power/main.c void device_pm_move_last(struct device * · line 198

Quoted 10 lines
drivers/base/power/main.c +10 -0
+{
+	struct device *dev;
+
+	async_synchronize_full();
+
+	mutex_lock(&dpm_list_mtx);
+	list_for_each_entry(dev, &dpm_list, power.entry)
+		INIT_COMPLETION(dev->power.completion);
+	mutex_unlock(&dpm_list_mtx);
+}

And this, for example, is pretty disgusting. Not only is that INIT_COMPLETION purely brought on by the whole problem with completions (they are fundamentally one-shot, but you want to use them over and over so you need to re-initialize them: a nice lock wouldn't have that problem to begin with), but the comment isn't even accurate. Sure, it waits for any async jobs, but that's the _least_ of what the function actually does, so the comment is actively misleading, isn't it?

Linus

#187 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 8 lines · Linus Torvalds
Quoted 2 lines · Rafael J. Wysocki

Completions it is, then.

What was so hard with the "Try the simple one first" to understand? You had a simpler working patch, why are you making this more complex one without ever having had any problems with the simpler one?

OK, why don't you just say you won't merge anything that doesn't use rwsems (although you said before that completions would be fine with you)? That would make things clear, but also it would mean we gave up handling the off-tree dependencies in general.

Quoted 11 lines

Btw, your 'atomic_set()' with errors is pure voodoo programming. That's not how atomics work. They do SMP-atomic addition etc, the 'atomic_set()' and 'atomic_read()' things are not in any way more atomic than any other access.

They are meant for racy reads (atomic_read()) and for initializations (atomic_set()), and the way you use them that 'atomic' part is entirely pointless, because it really isn't anything different from an 'int', except that it may be very very expensive on some architectures due to hashed spinlocks etc.

Quoted 3 lines

So stop this overdesign thing. Start simple. If you _ever_ see real problems, that's when you add stuff. As it is, any time you add complexity, you just add bugs.

OK, so that need not be atomic.

Quoted 10 lines
Quoted 10 lines
drivers/base/power/main.c +10 -0
+/**
+ * dpm_synchronize - Wait for PM callbacks of all devices to complete.
+ */
+static void dpm_synchronize(void)
+{
+	struct device *dev;
+
+	async_synchronize_full();
+
+	mutex_lock(&dpm_list_mtx);
Quoted 8 lines

drivers/base/power/main.c void device_pm_move_last(struct device * · line 198

Quoted 4 lines
drivers/base/power/main.c +4 -0
+	list_for_each_entry(dev, &dpm_list, power.entry)
+		INIT_COMPLETION(dev->power.completion);
+	mutex_unlock(&dpm_list_mtx);
+}

And this, for example, is pretty disgusting. Not only is that INIT_COMPLETION purely brought on by the whole problem with completions (they are fundamentally one-shot, but you want to use them over and over

Actually, twice. However, since I don't want to do any async handling in the _noirq phases any more, I can get rid of this whole function. Thanks for pointing that out to me.

Rafael

#188 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 2 lines · Rafael J. Wysocki

OK, why don't you just say you won't merge anything that doesn't use rwsems

I did! Here's a quote (and it's pretty much the whole email, so it's not like it was hidden):

- [email protected]:

"Let me put this simply: I've told you guys how to do it simply, with
_zero_ crap. No "iterating over children". No games. No data structures. No new infrastructure. Just a single new rwlock per device, and _trivial_ code.

So here's the challenge: try it my simple way first. I've quoted the code about five million times already. If you _actually_ see some problems, explain them. Don't make up stupid "iterate over each child" things. Don't claim totally made-up "leads to difficulties". Don't make it any more complicated than it needs to be.

Keep it simple. And once you have tried that simple approach, and you really can show why it doesn't work, THEN you can try something else.

But before you try the simple approach and explain why it wouldn't work, I simply will not pull anything more complex. Understood and agreed?"

And then later about completions:

- [email protected]:

"So I think completions should work, if done right. That whole "make the
parent wait for all the children to complete" is fine in that sense. And I'll happily take such an approach if my rwlock thing doesn't work."

IOW, I'll happily take the completions version, but dammit, I refuse to take it when there is a simpler approach that does NOT need to iterate, and does NOT need to re-initialize the data structures each round etc.

That's what I've been arguing against the whole time. It started as arguing against complex and unnecessary infrastructure, and trying to show that it _can_ be done so much simpler using existing basic locking.

And I get annoyed when you guys continually seem to want to make it more complex than it needs to be.

Quoted 7 lines
Quoted 3 lines

And this, for example, is pretty disgusting. Not only is that INIT_COMPLETION purely brought on by the whole problem with completions (they are fundamentally one-shot, but you want to use them over and over

Actually, twice. However, since I don't want to do any async handling in the _noirq phases any more, I can get rid of this whole function. Thanks for pointing that out to me.

Well, my point was that you'll need to do that

	INIT_COMPLETION(dev->power.completion);

thing each suspend and each resume. Exactly because completions are designed to be "onw-way" things, so you end up having to reset them each cycle (you just reset them even _more_ than you needed).

Again, my point was that using locks is actually a very _natural_ thing to do. I really don't understand what problems you and Alan have with just using locks - we have way more locks in the kernel than we have completions, so they are the "default" thing to do, and they really are very natural to use.

[ Ok, so admittedly the actual use of 'struct rw_semaphore' is pretty
unusual, but my point is that people are used to locking semantics in general, more so than the semantics of completions ]

Completions were literally designed to be used for one-off things - one of the most common uses is that the 'struct completion' is on the _stack_. It doesn't get much more one-off than that - and the completions are really very explicitly designed so that you can do a 'complete()' on something that will literally disappear from under you as you do it (because the struct completion might be on the stack of the thing that is waiting for it, and gets de-allocated when the waiter goes ahead).

That is why 'wait_for_completion()' always has to take the spinlock, for example - there is no fastpath for completion, because the races for the waiter releasing things too early are too nasty.

So completions are actually very subtle things - and you don't need any of that subtlety. I realize that from a user perspective, completions look very simple, but in many ways they actually have subtler semantics than a regular lock has.

Linus

#189 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 2 lines · Linus Torvalds

On Thu, 10 Dec 2009, Rafael J. Wysocki wrote:

...

Quoted 4 lines

IOW, I'll happily take the completions version, but dammit, I refuse to take it when there is a simpler approach that does NOT need to iterate, and does NOT need to re-initialize the data structures each round etc.

I don't think it really is that simple. For example, the fact that the outer lock has to be taken by one thread and released by another is not exactly straightforward. [One might ask what's the critical section in this case.]

Besides, suppose a device driver wants some off-tree constraints to be satisfied. What's the driver writer supposed to do? He only can lock the other device, but that will cause lockdep to complain, because this lock is going to be nested. Moreover, it's already too late, because his async thread has started and there's no guarantee that the other device hasn't acquired its rwsem yet.

With completions, the driver doesn't have to take any action to prevent another one from suspending too early. Instead, the other one has to wait for its suspend to complete, and for me personally this is a much more natural thing to do. IOW, if I were a driver writed, I'd probably prefer to wait on a completion than to use a lock in a tricky manner.

Quoted 15 lines

That's what I've been arguing against the whole time. It started as arguing against complex and unnecessary infrastructure, and trying to show that it _can_ be done so much simpler using existing basic locking.

And I get annoyed when you guys continually seem to want to make it more complex than it needs to be.

Quoted 7 lines
Quoted 3 lines

And this, for example, is pretty disgusting. Not only is that INIT_COMPLETION purely brought on by the whole problem with completions (they are fundamentally one-shot, but you want to use them over and over

Actually, twice. However, since I don't want to do any async handling in the _noirq phases any more, I can get rid of this whole function. Thanks for pointing that out to me.

Quoted 7 lines

Well, my point was that you'll need to do that

	INIT_COMPLETION(dev->power.completion);

thing each suspend and each resume. Exactly because completions are designed to be "onw-way" things, so you end up having to reset them each cycle (you just reset them even _more_ than you needed).

Well, why actually do we need to preserve the state of the data structure from one cycle to another? There's no need whatsoever.

Quoted 9 lines

Again, my point was that using locks is actually a very _natural_ thing to do. I really don't understand what problems you and Alan have with just using locks - we have way more locks in the kernel than we have completions, so they are the "default" thing to do, and they really are very natural to use.

[ Ok, so admittedly the actual use of 'struct rw_semaphore' is pretty
unusual, but my point is that people are used to locking semantics in general, more so than the semantics of completions ]

I still don't think there are many places where locks are used in a way you're suggesting. I would even say it's quite unusual to use locks this way.

Quoted 7 lines

Completions were literally designed to be used for one-off things - one of the most common uses is that the 'struct completion' is on the _stack_. It doesn't get much more one-off than that - and the completions are really very explicitly designed so that you can do a 'complete()' on something that will literally disappear from under you as you do it (because the struct completion might be on the stack of the thing that is waiting for it, and gets de-allocated when the waiter goes ahead).

We could literally throw away a completion after all of the potentially waiting threads have finished their operations and then allocate it back again when necessary. We only need the synchronization in this particular phase of suspend or resume and it doesn't need to extend to the other phases or other cycles, because all of the concurrent threads we need to synchronize will only live during this one particular phase of suspend or resume. They will all exit when it's finished anyway.

Quoted 8 lines

That is why 'wait_for_completion()' always has to take the spinlock, for example - there is no fastpath for completion, because the races for the waiter releasing things too early are too nasty.

So completions are actually very subtle things - and you don't need any of that subtlety. I realize that from a user perspective, completions look very simple, but in many ways they actually have subtler semantics than a regular lock has.

Well, I guess your point is that the implementation of completions is much more complicated that we really need, but I'm not sure if that really hurts.

Rafael

#190 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 4 lines · Rafael J. Wysocki

I don't think it really is that simple. For example, the fact that the outer lock has to be taken by one thread and released by another is not exactly straightforward. [One might ask what's the critical section in this case.]

Why is that any different from initializing the completion in one thread, and completing it in another?

It's exactly equivalent.

Completions really are "locks that were initialized to locked". That is, in fact, how completions came to be: we literally used to use semaphores for them, and the reason for completions is literally the magic lifetime rules they have.

So when you do

	INIT_COMPLETION(dev->power.completion);

that really is historically, logically, and conceptually exactly the same thing as initializing a lock to the locked state. We literally used to do it with the equivalent of

init_MUTEX_LOCKED()

way back when (well, except we didn't have mutexes back then, we had only counting semaphores) and instead of "complete()", we had "up()" on the semaphore to complete it.

Quoted 2 lines

Besides, suppose a device driver wants some off-tree constraints to be satisfied.

.. and I've told you several times that we should simply not do such devices asynchronously. At least not unless there is some _overriding_ reason to. And so far, nobody has suggested anything even remotely likely for that.

Again - KISS: Keep It Simple, Stupid!

Don't try to make up problems. The _only_ subsystem we know wants this is USB, and we know USB is purely a tree.

Quoted 8 lines
Quoted 5 lines
	INIT_COMPLETION(dev->power.completion);

thing each suspend and each resume. Exactly because completions are designed to be "onw-way" things, so you end up having to reset them each cycle (you just reset them even _more_ than you needed).

Well, why actually do we need to preserve the state of the data structure from one cycle to another? There's no need whatsoever.

My point is, with locks, none of that is necessary. Because they automatically do the right thing.

By picking the right concept, you don't have any of those "oh, we need to re-initialize things" issues. They just work.

Quoted 2 lines

I still don't think there are many places where locks are used in a way you're suggesting. I would even say it's quite unusual to use locks this way.

See above. It's what completions _are_.

Quoted 2 lines

Well, I guess your point is that the implementation of completions is much more complicated that we really need, but I'm not sure if that really hurts.

No. The implementation of completions is actually pretty simple, exactly because they have that spinlock that is required to protect them.

That wasn't the point. The point was that locks are actually the "normal" thing to use.

You are arguing as if completions are somehow the simpler model. That's simply not true. Completions are just a _special_case_of_locking_.

So why not just use regular locks instead, when it's actually the natural way to do it, and results in simpler code?

Linus

#191 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Up front: This is my personal view of the matter. Which probably isn't of much interest to anybody, so I won't bother to defend these views or comment any further on them. The decision about what version to use is up to the two of you. The fact is, either implementation would get the job done.

Quoted 13 lines · Linus Torvalds

Completions really are "locks that were initialized to locked". That is, in fact, how completions came to be: we literally used to use semaphores for them, and the reason for completions is literally the magic lifetime rules they have.

So when you do

	INIT_COMPLETION(dev->power.completion);

that really is historically, logically, and conceptually exactly the same thing as initializing a lock to the locked state. We literally used to do it with the equivalent of

Quoted 5 lines

init_MUTEX_LOCKED()

way back when (well, except we didn't have mutexes back then, we had only counting semaphores) and instead of "complete()", we had "up()" on the semaphore to complete it.

You think of it that way because you have been closely involved in the development of the various kinds of locks. Speaking as an outsider who has relatively little interest in the internal details, completions appear simpler than rwsems. Mostly because they have a smaller API:
complete() (or complete_all()) and wait_for_completion() as opposed to down_read(), up_read(), down_write(), and up_write().

Quoted 7 lines
Quoted 2 lines

Besides, suppose a device driver wants some off-tree constraints to be satisfied.

.. and I've told you several times that we should simply not do such devices asynchronously. At least not unless there is some _overriding_ reason to. And so far, nobody has suggested anything even remotely likely for that.

Agreed. The fact that async non-tree suspend constraints are difficult with rwsems isn't a drawback if nobody needs to use them.

Quoted 8 lines
Quoted 2 lines

Well, why actually do we need to preserve the state of the data structure from one cycle to another? There's no need whatsoever.

My point is, with locks, none of that is necessary. Because they automatically do the right thing.

By picking the right concept, you don't have any of those "oh, we need to re-initialize things" issues. They just work.

That's true, but it's not entirely clear. There are subtle questions about what happens if you stop in the middle or a device gets unregistered or registered in the middle. They require careful thought in both approaches.

Having to reinitialize a completion each time doesn't bother me. It's merely an indication that each suspend & resume is independent of all the others.

Quoted 4 lines
Quoted 2 lines

I still don't think there are many places where locks are used in a way you're suggesting. I would even say it's quite unusual to use locks this way.

See above. It's what completions _are_.

This is almost a philosophical issue. If each A_i must wait for some B_j's, is the onus on each A_i to test the B_j's it's interested in? Or is the onus on each B_j to tell the A_i's waiting for it that they may proceed? As Humpty-Dumpty said, "The question is which is to be master -- that's all".

Quoted 11 lines
Quoted 2 lines

Well, I guess your point is that the implementation of completions is much more complicated that we really need, but I'm not sure if that really hurts.

No. The implementation of completions is actually pretty simple, exactly because they have that spinlock that is required to protect them.

That wasn't the point. The point was that locks are actually the "normal" thing to use.

You are arguing as if completions are somehow the simpler model. That's simply not true. Completions are just a _special_case_of_locking_.

Doesn't that make them simpler by definition? Special cases always have less to worry about than the general case.

Quoted 2 lines

So why not just use regular locks instead, when it's actually the natural way to do it, and results in simpler code?

Simpler but also more subtle, IMO. If you didn't already know how the algorithm worked, figuring it out from the code would be harder with rwsems than with completions. Partly because of the way readers and writers exchange roles in suspend vs. resume, and partly because sometimes devices lock themselves and sometimes they lock other devices. With completions each device has its own, and each device waits for other devices' completions -- easier to keep track of mentally.

(I still think this whole readers vs. writers thing is a red herring. The essential property is that there are two opposing classes of lock holders. The fact that multiple writers can't hold the lock at the same time whereas multiple readers can is of no importance; the algorithm would work just as well if multiple writers _could_ hold the lock simultaneously.)

Balancing the additional conceptual complexity of the rwsem approach is the conceptual simplicity afforded by not needing to check all the children. To me this makes it pretty much a toss-up.

Alan Stern

#192 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 27 lines · Alan Stern

Up front: This is my personal view of the matter. Which probably isn't of much interest to anybody, so I won't bother to defend these views or comment any further on them. The decision about what version to use is up to the two of you. The fact is, either implementation would get the job done.

Quoted 13 lines · Linus Torvalds

Completions really are "locks that were initialized to locked". That is, in fact, how completions came to be: we literally used to use semaphores for them, and the reason for completions is literally the magic lifetime rules they have.

So when you do

	INIT_COMPLETION(dev->power.completion);

that really is historically, logically, and conceptually exactly the same thing as initializing a lock to the locked state. We literally used to do it with the equivalent of

Quoted 5 lines

init_MUTEX_LOCKED()

way back when (well, except we didn't have mutexes back then, we had only counting semaphores) and instead of "complete()", we had "up()" on the semaphore to complete it.

Quoted 6 lines

You think of it that way because you have been closely involved in the development of the various kinds of locks. Speaking as an outsider who has relatively little interest in the internal details, completions appear simpler than rwsems. Mostly because they have a smaller API:
complete() (or complete_all()) and wait_for_completion() as opposed to down_read(), up_read(), down_write(), and up_write().

Agreed.

Quoted 10 lines
Quoted 7 lines
Quoted 2 lines

Besides, suppose a device driver wants some off-tree constraints to be satisfied.

.. and I've told you several times that we should simply not do such devices asynchronously. At least not unless there is some _overriding_ reason to. And so far, nobody has suggested anything even remotely likely for that.

Agreed. The fact that async non-tree suspend constraints are difficult with rwsems isn't a drawback if nobody needs to use them.

Well, see my reply to Linus. The only thing that bothers me is that if we use rwsems, there's no way to handle that even if it turns out that someone needs them after all.

Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

Well, why actually do we need to preserve the state of the data structure from one cycle to another? There's no need whatsoever.

My point is, with locks, none of that is necessary. Because they automatically do the right thing.

By picking the right concept, you don't have any of those "oh, we need to re-initialize things" issues. They just work.

Quoted 8 lines

That's true, but it's not entirely clear. There are subtle questions about what happens if you stop in the middle or a device gets unregistered or registered in the middle. They require careful thought in both approaches.

Having to reinitialize a completion each time doesn't bother me. It's merely an indication that each suspend & resume is independent of all the others.

YES!

Quoted 10 lines
Quoted 4 lines
Quoted 2 lines

I still don't think there are many places where locks are used in a way you're suggesting. I would even say it's quite unusual to use locks this way.

See above. It's what completions _are_.

This is almost a philosophical issue. If each A_i must wait for some B_j's, is the onus on each A_i to test the B_j's it's interested in? Or is the onus on each B_j to tell the A_i's waiting for it that they may proceed? As Humpty-Dumpty said, "The question is which is to be master -- that's all".

Agreed.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Well, I guess your point is that the implementation of completions is much more complicated that we really need, but I'm not sure if that really hurts.

No. The implementation of completions is actually pretty simple, exactly because they have that spinlock that is required to protect them.

Quoted 8 lines
Quoted 5 lines

That wasn't the point. The point was that locks are actually the "normal" thing to use.

You are arguing as if completions are somehow the simpler model. That's simply not true. Completions are just a _special_case_of_locking_.

Doesn't that make them simpler by definition? Special cases always have less to worry about than the general case.

Heh, good point.

Quoted 6 lines
Quoted 2 lines

So why not just use regular locks instead, when it's actually the natural way to do it, and results in simpler code?

Simpler but also more subtle, IMO. If you didn't already know how the algorithm worked, figuring it out from the code would be harder with rwsems than with completions.

Indeed.

Quoted 6 lines

Partly because of the way readers and
writers exchange roles in suspend vs. resume, and partly because sometimes devices lock themselves and sometimes they lock other devices. With completions each device has its own, and each device waits for other devices' completions -- easier to keep track of mentally.

Agreed again.

Quoted 10 lines

(I still think this whole readers vs. writers thing is a red herring. The essential property is that there are two opposing classes of lock holders. The fact that multiple writers can't hold the lock at the same time whereas multiple readers can is of no importance; the algorithm would work just as well if multiple writers _could_ hold the lock simultaneously.)

Balancing the additional conceptual complexity of the rwsem approach is the conceptual simplicity afforded by not needing to check all the children. To me this makes it pretty much a toss-up.

Yup.

Thanks!
Rafael

#193 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 11 lines · Rafael J. Wysocki
Quoted 7 lines
Quoted 4 lines

.. and I've told you several times that we should simply not do such devices asynchronously. At least not unless there is some _overriding_ reason to. And so far, nobody has suggested anything even remotely likely for that.

Agreed. The fact that async non-tree suspend constraints are difficult with rwsems isn't a drawback if nobody needs to use them.

Well, see my reply to Linus. The only thing that bothers me is that if we use rwsems, there's no way to handle that even if it turns out that someone needs them after all.

This is now a totally moot point, but I want to make it anyway just to show how perverse life can be. It turns out that by combining some of the worst parts of the rwsem approach and the completion approach, it _is_ possible to have async non-tree suspend constraints with rwsems. The key is to imitate the way the completions work.

The resume algorithm doesn't change, but the suspend algorithm does. Currently, when suspending a device you first read-lock the parent (to prevent it from suspending too soon), then you asynchronously write-lock the device and suspend it, and finally read-unlock the parent.

Instead, you could first write-lock the device (to prevent the parent and any other dependents from suspending too soon), then asynchronously read-lock each of the children and anything else the device needs to wait for, then suspend the device, and finally write-unlock it. This really is analogous to completions: down_write() is like init_completion(), up_write() is like complete_all(), and down_read()+up_read() is like wait_for_completion(). I got the idea from Linus's comment that completions really are nothing but locks initialized in the "locked" state.

Of course, you would have to iterate over all the children and deal with lockdep complaints. So this obviously is not to be considered as a serious proposal.

Alan Stern

#194 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

In reply to Rafael J. Wysocki
Quoted 11 lines · Rafael J. Wysocki
Quoted 7 lines
Quoted 4 lines

.. and I've told you several times that we should simply not do such devices asynchronously. At least not unless there is some _overriding_ reason to. And so far, nobody has suggested anything even remotely likely for that.

Agreed. The fact that async non-tree suspend constraints are difficult with rwsems isn't a drawback if nobody needs to use them.

Well, see my reply to Linus. The only thing that bothers me is that if we use rwsems, there's no way to handle that even if it turns out that someone needs them after all.

This is now a totally moot point, but I want to make it anyway just to show how perverse life can be. It turns out that by combining some of the worst parts of the rwsem approach and the completion approach, it _is_ possible to have async non-tree suspend constraints with rwsems. The key is to imitate the way the completions work.

The resume algorithm doesn't change, but the suspend algorithm does. Currently, when suspending a device you first read-lock the parent (to prevent it from suspending too soon), then you asynchronously write-lock the device and suspend it, and finally read-unlock the parent.

Instead, you could first write-lock the device (to prevent the parent and any other dependents from suspending too soon), then asynchronously read-lock each of the children and anything else the device needs to wait for, then suspend the device, and finally write-unlock it. This really is analogous to completions: down_write() is like init_completion(), up_write() is like complete_all(), and down_read()+up_read() is like wait_for_completion(). I got the idea from Linus's comment that completions really are nothing but locks initialized in the "locked" state.

Of course, you would have to iterate over all the children and deal with lockdep complaints. So this obviously is not to be considered as a serious proposal.

Alan Stern

#195 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

In reply to Alan Stern
Quoted 27 lines · Alan Stern

Up front: This is my personal view of the matter. Which probably isn't of much interest to anybody, so I won't bother to defend these views or comment any further on them. The decision about what version to use is up to the two of you. The fact is, either implementation would get the job done.

Quoted 13 lines · Linus Torvalds

Completions really are "locks that were initialized to locked". That is, in fact, how completions came to be: we literally used to use semaphores for them, and the reason for completions is literally the magic lifetime rules they have.

So when you do

	INIT_COMPLETION(dev->power.completion);

that really is historically, logically, and conceptually exactly the same thing as initializing a lock to the locked state. We literally used to do it with the equivalent of

Quoted 5 lines

init_MUTEX_LOCKED()

way back when (well, except we didn't have mutexes back then, we had only counting semaphores) and instead of "complete()", we had "up()" on the semaphore to complete it.

Quoted 6 lines

You think of it that way because you have been closely involved in the development of the various kinds of locks. Speaking as an outsider who has relatively little interest in the internal details, completions appear simpler than rwsems. Mostly because they have a smaller API:
complete() (or complete_all()) and wait_for_completion() as opposed to down_read(), up_read(), down_write(), and up_write().

Agreed.

Quoted 10 lines
Quoted 7 lines
Quoted 2 lines

Besides, suppose a device driver wants some off-tree constraints to be satisfied.

.. and I've told you several times that we should simply not do such devices asynchronously. At least not unless there is some _overriding_ reason to. And so far, nobody has suggested anything even remotely likely for that.

Agreed. The fact that async non-tree suspend constraints are difficult with rwsems isn't a drawback if nobody needs to use them.

Well, see my reply to Linus. The only thing that bothers me is that if we use rwsems, there's no way to handle that even if it turns out that someone needs them after all.

Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

Well, why actually do we need to preserve the state of the data structure from one cycle to another? There's no need whatsoever.

My point is, with locks, none of that is necessary. Because they automatically do the right thing.

By picking the right concept, you don't have any of those "oh, we need to re-initialize things" issues. They just work.

Quoted 8 lines

That's true, but it's not entirely clear. There are subtle questions about what happens if you stop in the middle or a device gets unregistered or registered in the middle. They require careful thought in both approaches.

Having to reinitialize a completion each time doesn't bother me. It's merely an indication that each suspend & resume is independent of all the others.

YES!

Quoted 10 lines
Quoted 4 lines
Quoted 2 lines

I still don't think there are many places where locks are used in a way you're suggesting. I would even say it's quite unusual to use locks this way.

See above. It's what completions _are_.

This is almost a philosophical issue. If each A_i must wait for some B_j's, is the onus on each A_i to test the B_j's it's interested in? Or is the onus on each B_j to tell the A_i's waiting for it that they may proceed? As Humpty-Dumpty said, "The question is which is to be master -- that's all".

Agreed.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Well, I guess your point is that the implementation of completions is much more complicated that we really need, but I'm not sure if that really hurts.

No. The implementation of completions is actually pretty simple, exactly because they have that spinlock that is required to protect them.

Quoted 8 lines
Quoted 5 lines

That wasn't the point. The point was that locks are actually the "normal" thing to use.

You are arguing as if completions are somehow the simpler model. That's simply not true. Completions are just a _special_case_of_locking_.

Doesn't that make them simpler by definition? Special cases always have less to worry about than the general case.

Heh, good point.

Quoted 6 lines
Quoted 2 lines

So why not just use regular locks instead, when it's actually the natural way to do it, and results in simpler code?

Simpler but also more subtle, IMO. If you didn't already know how the algorithm worked, figuring it out from the code would be harder with rwsems than with completions.

Indeed.

Quoted 6 lines

Partly because of the way readers and
writers exchange roles in suspend vs. resume, and partly because sometimes devices lock themselves and sometimes they lock other devices. With completions each device has its own, and each device waits for other devices' completions -- easier to keep track of mentally.

Agreed again.

Quoted 10 lines

(I still think this whole readers vs. writers thing is a red herring. The essential property is that there are two opposing classes of lock holders. The fact that multiple writers can't hold the lock at the same time whereas multiple readers can is of no importance; the algorithm would work just as well if multiple writers _could_ hold the lock simultaneously.)

Balancing the additional conceptual complexity of the rwsem approach is the conceptual simplicity afforded by not needing to check all the children. To me this makes it pretty much a toss-up.

Yup.

Thanks!
Rafael

#196 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

In reply to Linus Torvalds

Up front: This is my personal view of the matter. Which probably isn't of much interest to anybody, so I won't bother to defend these views or comment any further on them. The decision about what version to use is up to the two of you. The fact is, either implementation would get the job done.

Quoted 13 lines · Linus Torvalds

Completions really are "locks that were initialized to locked". That is, in fact, how completions came to be: we literally used to use semaphores for them, and the reason for completions is literally the magic lifetime rules they have.

So when you do

	INIT_COMPLETION(dev->power.completion);

that really is historically, logically, and conceptually exactly the same thing as initializing a lock to the locked state. We literally used to do it with the equivalent of

Quoted 5 lines

init_MUTEX_LOCKED()

way back when (well, except we didn't have mutexes back then, we had only counting semaphores) and instead of "complete()", we had "up()" on the semaphore to complete it.

You think of it that way because you have been closely involved in the development of the various kinds of locks. Speaking as an outsider who has relatively little interest in the internal details, completions appear simpler than rwsems. Mostly because they have a smaller API:
complete() (or complete_all()) and wait_for_completion() as opposed to down_read(), up_read(), down_write(), and up_write().

Quoted 7 lines
Quoted 2 lines

Besides, suppose a device driver wants some off-tree constraints to be satisfied.

.. and I've told you several times that we should simply not do such devices asynchronously. At least not unless there is some _overriding_ reason to. And so far, nobody has suggested anything even remotely likely for that.

Agreed. The fact that async non-tree suspend constraints are difficult with rwsems isn't a drawback if nobody needs to use them.

Quoted 8 lines
Quoted 2 lines

Well, why actually do we need to preserve the state of the data structure from one cycle to another? There's no need whatsoever.

My point is, with locks, none of that is necessary. Because they automatically do the right thing.

By picking the right concept, you don't have any of those "oh, we need to re-initialize things" issues. They just work.

That's true, but it's not entirely clear. There are subtle questions about what happens if you stop in the middle or a device gets unregistered or registered in the middle. They require careful thought in both approaches.

Having to reinitialize a completion each time doesn't bother me. It's merely an indication that each suspend & resume is independent of all the others.

Quoted 4 lines
Quoted 2 lines

I still don't think there are many places where locks are used in a way you're suggesting. I would even say it's quite unusual to use locks this way.

See above. It's what completions _are_.

This is almost a philosophical issue. If each A_i must wait for some B_j's, is the onus on each A_i to test the B_j's it's interested in? Or is the onus on each B_j to tell the A_i's waiting for it that they may proceed? As Humpty-Dumpty said, "The question is which is to be master -- that's all".

Quoted 11 lines
Quoted 2 lines

Well, I guess your point is that the implementation of completions is much more complicated that we really need, but I'm not sure if that really hurts.

No. The implementation of completions is actually pretty simple, exactly because they have that spinlock that is required to protect them.

That wasn't the point. The point was that locks are actually the "normal" thing to use.

You are arguing as if completions are somehow the simpler model. That's simply not true. Completions are just a _special_case_of_locking_.

Doesn't that make them simpler by definition? Special cases always have less to worry about than the general case.

Quoted 2 lines

So why not just use regular locks instead, when it's actually the natural way to do it, and results in simpler code?

Simpler but also more subtle, IMO. If you didn't already know how the algorithm worked, figuring it out from the code would be harder with rwsems than with completions. Partly because of the way readers and writers exchange roles in suspend vs. resume, and partly because sometimes devices lock themselves and sometimes they lock other devices. With completions each device has its own, and each device waits for other devices' completions -- easier to keep track of mentally.

(I still think this whole readers vs. writers thing is a red herring. The essential property is that there are two opposing classes of lock holders. The fact that multiple writers can't hold the lock at the same time whereas multiple readers can is of no importance; the algorithm would work just as well if multiple writers _could_ hold the lock simultaneously.)

Balancing the additional conceptual complexity of the rwsem approach is the conceptual simplicity afforded by not needing to check all the children. To me this makes it pretty much a toss-up.

Alan Stern

#197 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

In reply to Linus Torvalds
Quoted 10 lines · Linus Torvalds
Quoted 4 lines · Rafael J. Wysocki

I don't think it really is that simple. For example, the fact that the outer lock has to be taken by one thread and released by another is not exactly straightforward. [One might ask what's the critical section in this case.]

Why is that any different from initializing the completion in one thread, and completing it in another?

Quoted 6 lines

It's exactly equivalent.

Completions really are "locks that were initialized to locked". That is, in fact, how completions came to be: we literally used to use semaphores for them, and the reason for completions is literally the magic lifetime rules they have.

I don't know how they emerged historically and that's why I look a them in a different way than you do, probably.

But fine, say we use the approach based on rwsems and consider suspend and the inner lock. We acquire it using down_write(), because we want to wait for multiple other dirvers. Now, in fact we could do literally

down_write(dev->power.rwsem);
up_write(dev->power.rwsem);

because the lock doesn't really protect anything from anyone. What it does is to prevent _us_ from doing something too early. To me, personally, it's not a usual use of locks.

Moreover, if you think completions should be treated like locks, the up_write() above plays the role of the INIT_COMPLETION() in my last patch (or vice versa), so we reinitialize the data structure to the previous state in this case too, only earlier (and we could do that later just as well).

The only real drawback of using completions I can see is that we have to iterate over the children during suspend, but if async suspend is going to save us any time at all, we can easily afford it (resume with completions is actually simpler than with rwsems, because we only have to wait for one device each time).

Quoted 8 lines
Quoted 2 lines

Besides, suppose a device driver wants some off-tree constraints to be satisfied.

.. and I've told you several times that we should simply not do such devices asynchronously. At least not unless there is some _overriding_ reason to. And so far, nobody has suggested anything even remotely likely for that.

Quoted 4 lines

Again - KISS: Keep It Simple, Stupid!

Don't try to make up problems. The _only_ subsystem we know wants this is USB, and we know USB is purely a tree.

Not really.

I've already said it once, but let me repeat. Some device objects have those ACPI "shadow" device objects that represent the ACPI view of given "physical" device and have their own suspend and resume routines. It turns out that these ACPI "shadow" devices have to be suspended after their "physical" counterparts and resumed before them, or else things beak really badly. I don't know the reason for that, I only verified it experimentally (I also don't like that design, but I didn't invent it and I have to live with it at least for now). So if we don't enforce these constraints doing async suspend and resume, we won't be able to handle _any_ devices with those ACPI "shadow" things asynchronously. Ever. [That includes the majority PCI devices, at least the "planar" ones (which is unfortunate, but that's how it goes).]

If we had a clean way of representing off-tree constraints during asynchronous suspend and resume, we'd be able to handle this issue at the bus type level.

And even if we don't anticipate it right now, I think the iteration over children during suspend is a fair price for a clean interface that bus types or drivers can use in future. YMMV.

Quoted 10 lines
Quoted 2 lines

Well, I guess your point is that the implementation of completions is much more complicated that we really need, but I'm not sure if that really hurts.

No. The implementation of completions is actually pretty simple, exactly because they have that spinlock that is required to protect them.

That wasn't the point. The point was that locks are actually the "normal" thing to use.

You are arguing as if completions are somehow the simpler model.

That's because I think so.

Quoted 1 line

That's simply not true. Completions are just a _special_case_of_locking_.

Which doesn't necessarily prevent them from being conceptually simpler that the locking scheme based on rwsems.

Quoted 2 lines

So why not just use regular locks instead, when it's actually the natural way to do it, and results in simpler code?

Well, to me, it's way not natural and, quite frankly, in my not so humble opinion, it's a matter of personal preference.

But, since your personal preference is what matters in this case, I'm not going to argue any more, because that just plain doesn't make sense.

So, if you're not fine with the last patch I sent
(http://patchwork.kernel.org/patch/66375/), I'll send one using rwsems instead of completions just to make _you_ happy, not because I think that's what we should do objectively.

Rafael

#198 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 11 lines · Rafael J. Wysocki

But fine, say we use the approach based on rwsems and consider suspend and the inner lock. We acquire it using down_write(), because we want to wait for multiple other dirvers. Now, in fact we could do literally

down_write(dev->power.rwsem);
up_write(dev->power.rwsem);

because the lock doesn't really protect anything from anyone. What it does is to prevent _us_ from doing something too early. To me, personally, it's not a usual use of locks.

I agree that it's fairly unusual, but on the other hand, it's unusual only because you contrieved it to be.

If you instead do

	down_write(dev->power.rwsem);
	.. do the actual suspend ..
	up_write(dev->power.rwsem);

it doesn't look odd any more, does it? And while you don't _need_ to hold the power lock over the suspend call, it actually does make sense, and gives you some nicer guarantees.

For an example of the kinds of guarantees it would give you - I think that you might actually be able to do a partial suspend and then a resume without any other locks, and you'd know that just the per-device locking would already guarantee that no device is ever tried to resume before it has finished its asynchronous suspend.

Think about it.

In the completion model, the "async_synchronize_full()" will synchronize all async work, and as a result you think that you don't need that level of robustness from the locking itself.

But think about it this way: if you could abort a failed suspend, and start resuming devices immediately, without doing that "async_synchronize_full()" in between - simply because you know that the node locking itself will just "do the right thing".

To me, that's a sign of a _good_ design. Using a rwsem is simply just more robust and natural for the problem in question. Exactly because it's a real lock.

Quoted 5 lines
Quoted 2 lines

Don't try to make up problems. The _only_ subsystem we know wants this is USB, and we know USB is purely a tree.

Not really.

Quoted 12 lines

I've already said it once, but let me repeat. Some device objects have those ACPI "shadow" device objects that represent the ACPI view of given "physical" device and have their own suspend and resume routines. It turns out that these ACPI "shadow" devices have to be suspended after their "physical" counterparts and resumed before them, or else things beak really badly. I don't know the reason for that, I only verified it experimentally (I also don't like that design, but I didn't invent it and I have to live with it at least for now). So if we don't enforce these constraints doing async suspend and resume, we won't be able to handle _any_ devices with those ACPI "shadow" things asynchronously. Ever. [That includes the majority PCI devices, at least the "planar" ones (which is unfortunate, but that's how it goes).]

So?

First off, you're wrong. It's not "ever". I'm happy to add complexity later, I just don't want to start out with a complex model. Adding complexity too early "just because we migth need it" is the wrong thing to do.

Secondly, I repeat: we don't want to do those PCI devices asynchronously anyway. You're again digging yourself deeper by just continually bringing up this total non-issue. I realize you did it for testing, but I'm serious when I say that we should limit these things as much as possible, rather than see it as an opportunity to do crazy things.

Solve the problem at hand _first_. Solve it as simply as you can. And hope that you never ever will need anything more complex.

Linus

#199 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 10 lines · Linus Torvalds
Quoted 8 lines · Rafael J. Wysocki

But fine, say we use the approach based on rwsems and consider suspend and the inner lock. We acquire it using down_write(), because we want to wait for multiple other dirvers. Now, in fact we could do literally

down_write(dev->power.rwsem);
up_write(dev->power.rwsem);

Quoted 6 lines
Quoted 3 lines

because the lock doesn't really protect anything from anyone. What it does is to prevent _us_ from doing something too early. To me, personally, it's not a usual use of locks.

I agree that it's fairly unusual, but on the other hand, it's unusual only because you contrieved it to be.

Whatever. The very fact that you can freely move the up_write() (as long as it's after the down_write()) is fairly unusual.

Quoted 4 lines

But think about it this way: if you could abort a failed suspend, and start resuming devices immediately, without doing that "async_synchronize_full()" in between - simply because you know that the node locking itself will just "do the right thing".

I'd rather not. :-)

Quoted 3 lines

To me, that's a sign of a _good_ design. Using a rwsem is simply just more robust and natural for the problem in question. Exactly because it's a real lock.

...

Quoted 2 lines

Solve the problem at hand _first_. Solve it as simply as you can. And hope that you never ever will need anything more complex.

Below is a patch I've just tested, but there's a lockdep problem in it I don't know how to solve. Namely, lockdep is apparently unhappy with us not releasing the lock taken in device_suspend() and it complains we take it twice in a row (which we do, but for another device). I need to use down_read_non_owner() to make it shut up and then I also need to use up_read_non_owner() in __device_suspend(), although there's the comment in include/linux/rwsem.h saying exatly this about that:

/*
* Take/release a lock when not the owner will release it. *
* [ This API should be avoided as much as possible - the
* proper abstraction for this case is completions. ]
*/

(I'd like to know your opinion about that). Yet, that's not all, because next it complains during resume that __device_resume() releases a lock it didn't acquire, which it clearly does, but that is intentional. Unfortunately, there's no up_write_non_owner() ...

So, what am I supposed to do about that?

Rafael

---
 drivers/base/power/main.c    |  107 +++++++++++++++++++++++++++++++++++++++----
 include/linux/device.h       |    6 ++
 include/linux/pm.h           |    3 +
 include/linux/resume-trace.h |    7 ++
 4 files changed, 114 insertions(+), 9 deletions(-)
include/linux/pm.h +3 -0
Index: linux-2.6/include/linux/pm.h
===================================================================
@@ -26,6 +26,7 @@
 #include <linux/spinlock.h>
 #include <linux/wait.h>
 #include <linux/timer.h>
+#include <linux/rwsem.h>
 
 /*
  * Callbacks for platform drivers to implement.
@@ -412,9 +413,11 @@ struct dev_pm_info {
 	pm_message_t		power_state;
 	unsigned int		can_wakeup:1;
 	unsigned int		should_wakeup:1;
+	unsigned		async_suspend:1;
 	enum dpm_state		status;		/* Owned by the PM core */
 #ifdef CONFIG_PM_SLEEP
 	struct list_head	entry;
+	struct rw_semaphore	rwsem;
 #endif
 #ifdef CONFIG_PM_RUNTIME
 	struct timer_list	suspend_timer;
drivers/base/power/main.c +98 -9

230 lines not on this page · raw message

include/linux/resume-trace.h +7 -0
Index: linux-2.6/include/linux/resume-trace.h
===================================================================
@@ -6,6 +6,11 @@
 
 extern int pm_trace_enabled;
 
+static inline int pm_trace_is_enabled(void)
+{
+       return pm_trace_enabled;
+}
+
 struct device;
 extern void set_trace_device(struct device *);
 extern void generate_resume_trace(const void *tracedata, unsigned int user);
@@ -17,6 +22,8 @@ extern void generate_resume_trace(const 
 
 #else
 
+static inline int pm_trace_is_enabled(void) { return 0; }
+
 #define TRACE_DEVICE(dev) do { } while (0)
 #define TRACE_RESUME(dev) do { } while (0)
 
include/linux/device.h +6 -0
Index: linux-2.6/include/linux/device.h
===================================================================
@@ -472,6 +472,12 @@ static inline int device_is_registered(s
 	return dev->kobj.state_in_sysfs;
 }
 
+static inline void device_enable_async_suspend(struct device *dev, bool enable)
+{
+	if (dev->power.status == DPM_ON)
+		dev->power.async_suspend = enable;
+}
+
 void driver_init(void);
 
 /*
#200 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 7 lines · Rafael J. Wysocki

Below is a patch I've just tested, but there's a lockdep problem in it I don't know how to solve. Namely, lockdep is apparently unhappy with us not releasing the lock taken in device_suspend() and it complains we take it twice in a row (which we do, but for another device). I need to use down_read_non_owner() to make it shut up and then I also need to use up_read_non_owner() in __device_suspend(),

Ok, that I admit is actually a problem.

Ok, ok, I'll accept that completion() version, even though I think it's inferior.

Linus

#201 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

In reply to Rafael J. Wysocki
Quoted 7 lines · Rafael J. Wysocki

Below is a patch I've just tested, but there's a lockdep problem in it I don't know how to solve. Namely, lockdep is apparently unhappy with us not releasing the lock taken in device_suspend() and it complains we take it twice in a row (which we do, but for another device). I need to use down_read_non_owner() to make it shut up and then I also need to use up_read_non_owner() in __device_suspend(),

Ok, that I admit is actually a problem.

Ok, ok, I'll accept that completion() version, even though I think it's inferior.

Linus

#202 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 10 lines · Linus Torvalds
Quoted 7 lines · Rafael J. Wysocki

Below is a patch I've just tested, but there's a lockdep problem in it I don't know how to solve. Namely, lockdep is apparently unhappy with us not releasing the lock taken in device_suspend() and it complains we take it twice in a row (which we do, but for another device). I need to use down_read_non_owner() to make it shut up and then I also need to use up_read_non_owner() in __device_suspend(),

Quoted 4 lines

Ok, that I admit is actually a problem.

Ok, ok, I'll accept that completion() version, even though I think it's inferior.

Great! :-)

I slightly changed it in the meantime to avoid calling wait_for_completion() when both the parent and the child are "synchronous", which prevents the code from choking on some situations when the ordering of dpm_list is wrong (this happens as a result of bugs, but not necessarily fatal, for example if one of the drivers' suspend and resume callbacks are NULL and the bus type doesn't access the hardware directly, so we shouldn't make things worse than they already are IMO).

I'd like to put it into my tree in this form, if you don't mind.

[Note for Alan: dpm_wait() is not exported for now, we'll export it when there are any users.]

Rafael

---
From: Rafael J. Wysocki <[email protected]>
Subject: PM: Asynchronous suspend and resume of devices

Theoretically, the total time of system sleep transitions (suspend to RAM, hibernation) can be reduced by running suspend and resume callbacks of device drivers in parallel with each other. However, there are dependencies between devices such that we're not allowed to suspend the parent of a device before suspending the device itself. Analogously, we're not allowed to resume a device before resuming its parent.

Thus, to make it possible to execute device drivers' suspend and resume callbacks in parallel with each other, introduce (at the PM core level) a synchronization mechanism preventing the dependencies between devices from being violated.

First, device drivers that want their suspend and resume callbacks to be run asynchronously need to set the power.async_suspend flags of their devices using device_enable_async_suspend().

Second, for each device with the power.async_suspend flag set the PM core will start async threads to execute its suspend and resume callbacks.

The async threads started for different devices are synchronized with each other and with the main suspend (or resume) thread with the help of completions, in the following way:
(1) There is a completion, power.completion, for each device object.
(2) Each device's completion is reset before starting the async suspend (or resume) thread for the device or, in the case of devices whose power.async_suspend flags are not set, before executing the device's suspend and resume callbacks.
(3) During suspend, right before running the bus type, device type and device class suspend callbacks for the device, the PM core waits for the completions of all the device's children to be completed.
(4) During resume, right before running the bus type, device type and device class resume callbacks for the device, the PM core waits for the completion of the device's parent to be completed.
(5) The PM core completes power.completion for each device right after the bus type, device type and device class suspend (or resume) callbacks executed for the device have returned.

---
 drivers/base/power/main.c    |  115 ++++++++++++++++++++++++++++++++++++++++---
 include/linux/device.h       |    6 ++
 include/linux/pm.h           |    3 +
 include/linux/resume-trace.h |    7 ++
 4 files changed, 125 insertions(+), 6 deletions(-)
include/linux/pm.h +3 -0
Index: linux-2.6/include/linux/pm.h
===================================================================
@@ -26,6 +26,7 @@
 #include <linux/spinlock.h>
 #include <linux/wait.h>
 #include <linux/timer.h>
+#include <linux/completion.h>
 
 /*
  * Callbacks for platform drivers to implement.
@@ -412,9 +413,11 @@ struct dev_pm_info {
 	pm_message_t		power_state;
 	unsigned int		can_wakeup:1;
 	unsigned int		should_wakeup:1;
+	unsigned		async_suspend:1;
 	enum dpm_state		status;		/* Owned by the PM core */
 #ifdef CONFIG_PM_SLEEP
 	struct list_head	entry;
+	struct completion	completion;
 #endif
 #ifdef CONFIG_PM_RUNTIME
 	struct timer_list	suspend_timer;
drivers/base/power/main.c +109 -6

241 lines not on this page · raw message

include/linux/resume-trace.h +7 -0
Index: linux-2.6/include/linux/resume-trace.h
===================================================================
@@ -6,6 +6,11 @@
 
 extern int pm_trace_enabled;
 
+static inline int pm_trace_is_enabled(void)
+{
+       return pm_trace_enabled;
+}
+
 struct device;
 extern void set_trace_device(struct device *);
 extern void generate_resume_trace(const void *tracedata, unsigned int user);
@@ -17,6 +22,8 @@ extern void generate_resume_trace(const 
 
 #else
 
+static inline int pm_trace_is_enabled(void) { return 0; }
+
 #define TRACE_DEVICE(dev) do { } while (0)
 #define TRACE_RESUME(dev) do { } while (0)
 
include/linux/device.h +6 -0
Index: linux-2.6/include/linux/device.h
===================================================================
@@ -472,6 +472,12 @@ static inline int device_is_registered(s
 	return dev->kobj.state_in_sysfs;
 }
 
+static inline void device_enable_async_suspend(struct device *dev, bool enable)
+{
+	if (dev->power.status == DPM_ON)
+		dev->power.async_suspend = enable;
+}
+
 void driver_init(void);
 
 /*
#203 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 2 lines · Rafael J. Wysocki

I'd like to put it into my tree in this form, if you don't mind.

This version still has a major problem, which is not related to completions vs rwsems, but simply to the fact that you wanted to do this at the generic device layer level rather than do it at the actual low-level suspend/resume level.

Namely that there's no apparent sane way to say "don't wait for children".

PCI bridges that don't suspend at all - or any other device that only suspends in the 'suspend_late()' thing, for that matter - don't have any reason what-so-ever to wait for children, since they aren't actually suspending in the first place. But you make them wait regardless, which then serializes things unnecessarily (for example, two unrelated USB controllers).

And no, making _everything_ be async is _not_ the answer.

Linus

#204 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 10 lines · Linus Torvalds
Quoted 2 lines · Rafael J. Wysocki

I'd like to put it into my tree in this form, if you don't mind.

This version still has a major problem, which is not related to completions vs rwsems, but simply to the fact that you wanted to do this at the generic device layer level rather than do it at the actual low-level suspend/resume level.

Quoted 8 lines

Namely that there's no apparent sane way to say "don't wait for children".

PCI bridges that don't suspend at all - or any other device that only suspends in the 'suspend_late()' thing, for that matter - don't have any reason what-so-ever to wait for children, since they aren't actually suspending in the first place. But you make them wait regardless, which then serializes things unnecessarily (for example, two unrelated USB controllers).

This is a problem that needs to be solved.

One solution that we have discussed on linux-pm is to start a bunch of async threads searching for async devices that can be suspended and suspending them (assuming suspend is considered) out of order with respect to dpm_list. For example, leaf async devices can always be suspended at the same time regardless of their positions in dpm_list. This way we could get almost the entire gain resulting from suspending or resuming devices in parallel without bothering drivers with the problem of dependencies that need to be honoured.

That's something we can add on top of this patch, though, not to complicate things from the start and it surely requires more discussion.

Quoted 1 line

And no, making _everything_ be async is _not_ the answer.

I'm not sure what you mean, really.

Speaking of PCI bridges, even though they don't "suspend" in the sense of being put into low power states or something, we still need to save their registers on suspend and restore them on resume, and that restore has to be done before we start to access devices below the bridge.

There are devices with totally null suspend and resume routines that even the bus type doesn't really handle, but those can be marked as "async" from the start and they won't really get in the way any more (this creates another issue to solve, namely that we shouldn't really start a new async thread for each of them; we have considered that too).

Even if we move that all to drivers, the constraints won't go away and someone will have to take care of them. Now, since _we_ have problems with reaching an agreement about how to do it, the driver writers will be even less likely to figure that out.

Rafael

#205 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 4 lines · Rafael J. Wysocki
Quoted 3 lines · Linus Torvalds
Quoted 1 line · Rafael J. Wysocki

...

Quoted 9 lines
Quoted 2 lines

And no, making _everything_ be async is _not_ the answer.

I'm not sure what you mean, really.

Speaking of PCI bridges, even though they don't "suspend" in the sense of being put into low power states or something, we still need to save their registers on suspend and restore them on resume, and that restore has to be done before we start to access devices below the bridge.

Of course we restore them at the early stage now so the above remark does't apply to the patch in question, sorry.

But the one below does.

Quoted 4 lines

Even if we move that all to drivers, the constraints won't go away and someone will have to take care of them. Now, since _we_ have problems with reaching an agreement about how to do it, the driver writers will be even less likely to figure that out.

Rafael

#206 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

In reply to Rafael J. Wysocki
Quoted 4 lines · Rafael J. Wysocki

One solution that we have discussed on linux-pm is to start a bunch of async threads searching for async devices that can be suspended and suspending them (assuming suspend is considered) out of order with respect to dpm_list.

Ok, guys, stop the crazy.

That's another of those "ok, that's just ttoally stupid and clearly too complex" ideas that I would never pull.

I should seriously suggest that people just stop discussing architectural details on the pm list if they all end up being this level of crazy.

The sane thing to do is to just totally ignore the async layer on PCI bridges and other things that only have a late-suspend/early-resume thing. No need for the above kind of obviously idiotic crap.

However, my point was really that we wouldn't even have _needed_ that kind of special case if we had just decided to let the subsystems do it. But whatever. At worst, the PCI layer can even just mark such devices with just late/early suspend/resume as being asynchronous, even though that ends up resulting in some totally pointless async work that doesn't do anything.

But please guys - reign in the crazy ideas on the pm list. It's not like our suspend/resume has gotten so stable as to be boring, and we want it to become unreliable again.

Linus

#207 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

In reply to Rafael J. Wysocki
Quoted 4 lines · Rafael J. Wysocki

One solution that we have discussed on linux-pm is to start a bunch of async threads searching for async devices that can be suspended and suspending them (assuming suspend is considered) out of order with respect to dpm_list.

Ok, guys, stop the crazy.

That's another of those "ok, that's just ttoally stupid and clearly too complex" ideas that I would never pull.

I should seriously suggest that people just stop discussing architectural details on the pm list if they all end up being this level of crazy.

The sane thing to do is to just totally ignore the async layer on PCI bridges and other things that only have a late-suspend/early-resume thing. No need for the above kind of obviously idiotic crap.

However, my point was really that we wouldn't even have _needed_ that kind of special case if we had just decided to let the subsystems do it. But whatever. At worst, the PCI layer can even just mark such devices with just late/early suspend/resume as being asynchronous, even though that ends up resulting in some totally pointless async work that doesn't do anything.

But please guys - reign in the crazy ideas on the pm list. It's not like our suspend/resume has gotten so stable as to be boring, and we want it to become unreliable again.

Linus

#208 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 26 lines · Linus Torvalds
Quoted 4 lines · Rafael J. Wysocki

One solution that we have discussed on linux-pm is to start a bunch of async threads searching for async devices that can be suspended and suspending them (assuming suspend is considered) out of order with respect to dpm_list.

Ok, guys, stop the crazy.

That's another of those "ok, that's just ttoally stupid and clearly too complex" ideas that I would never pull.

I should seriously suggest that people just stop discussing architectural details on the pm list if they all end up being this level of crazy.

The sane thing to do is to just totally ignore the async layer on PCI bridges and other things that only have a late-suspend/early-resume thing. No need for the above kind of obviously idiotic crap.

However, my point was really that we wouldn't even have _needed_ that kind of special case if we had just decided to let the subsystems do it. But whatever. At worst, the PCI layer can even just mark such devices with just late/early suspend/resume as being asynchronous, even though that ends up resulting in some totally pointless async work that doesn't do anything.

Quoted 3 lines

But please guys - reign in the crazy ideas on the pm list. It's not like our suspend/resume has gotten so stable as to be boring, and we want it to become unreliable again.

Indeed.

OK, what about a two-pass approach in which the first pass only inits the completions and starts async threads for leaf "async" devices? I think leaf devices are most likely to take much time to suspend, so this will give us a chance to save quite some time.

A more aggressive version of this might start the async threads for all async devices in the first pass and then only handle the sychronous ones in the second pass - as long as there are only a few async devices that should be quite efficient.

Rafael

#209 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 5 lines · Rafael J. Wysocki

OK, what about a two-pass approach in which the first pass only inits the completions and starts async threads for leaf "async" devices? I think leaf devices are most likely to take much time to suspend, so this will give us a chance to save quite some time.

Why?

Really.

Again, stop making it harder than it needs to be.

Why do you make up these crazy schemes that are way more complex than they need to be?

Here's an untested one-liner that has a 10-line comment.

I agree it is ugly, but it is ugly exactly because the generic device layer _forces_ us to wait for children even when we don't want to. With this, that unnecessary wait is now done asynchronously.

I'd rather do it some other way - perhaps having an explicit flag that says "don't wait for children because I'm not going to suspend myself until 'suspend_late' _anyway_". But at least this is _simple_.

Linus

---
 drivers/pci/probe.c |   11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)
drivers/pci/probe.c +11 -0
index 98ffb2d..4e0ad7b 100644
@@ -437,6 +437,17 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent,
 	}
 	bridge->subordinate = child;
 
+	/*
+	 * We don't really suspend PCI buses asyncronously.
+	 *
+	 * However, since we don't actually suspend them at all until
+	 * the late phase, we might as well lie to the device layer
+	 * and it to do our no-op not-suspend asynchronously, so that
+	 * we end up not synchronizing with any of our child devices
+	 * that might want to be asynchronous.
+	 */
+	bridge->dev.power.async_suspend = 1;
+
 	return child;
 }
 
#210 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

In reply to Rafael J. Wysocki
Quoted 5 lines · Rafael J. Wysocki

OK, what about a two-pass approach in which the first pass only inits the completions and starts async threads for leaf "async" devices? I think leaf devices are most likely to take much time to suspend, so this will give us a chance to save quite some time.

Why?

Really.

Again, stop making it harder than it needs to be.

Why do you make up these crazy schemes that are way more complex than they need to be?

Here's an untested one-liner that has a 10-line comment.

I agree it is ugly, but it is ugly exactly because the generic device layer _forces_ us to wait for children even when we don't want to. With this, that unnecessary wait is now done asynchronously.

I'd rather do it some other way - perhaps having an explicit flag that says "don't wait for children because I'm not going to suspend myself until 'suspend_late' _anyway_". But at least this is _simple_.

Linus

---
 drivers/pci/probe.c |   11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)
drivers/pci/probe.c +11 -0
index 98ffb2d..4e0ad7b 100644
@@ -437,6 +437,17 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent,
 	}
 	bridge->subordinate = child;
 
+	/*
+	 * We don't really suspend PCI buses asyncronously.
+	 *
+	 * However, since we don't actually suspend them at all until
+	 * the late phase, we might as well lie to the device layer
+	 * and it to do our no-op not-suspend asynchronously, so that
+	 * we end up not synchronizing with any of our child devices
+	 * that might want to be asynchronous.
+	 */
+	bridge->dev.power.async_suspend = 1;
+
 	return child;
 }
 
#211 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 2 lines · Linus Torvalds

Here's an untested one-liner that has a 10-line comment.

Btw, when I say "untested", in this case I mean that it isn't even compile-tested. I haven't merged your other patches yet, so in my tree that 'async_suspend' flag doesn't even exist, and the patch I sent out definitely doesn't compile.

But it _might_ compile (and perhaps even work) in your tree.

Linus

#212 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

In reply to Linus Torvalds
Quoted 2 lines · Linus Torvalds

Here's an untested one-liner that has a 10-line comment.

Btw, when I say "untested", in this case I mean that it isn't even compile-tested. I haven't merged your other patches yet, so in my tree that 'async_suspend' flag doesn't even exist, and the patch I sent out definitely doesn't compile.

But it _might_ compile (and perhaps even work) in your tree.

Linus

#213 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 5 lines · Rafael J. Wysocki

OK, what about a two-pass approach in which the first pass only inits the completions and starts async threads for leaf "async" devices? I think leaf devices are most likely to take much time to suspend, so this will give us a chance to save quite some time.

Why?

Really.

Because the PCI bridges are not the only case where it matters (I'd say they are really a corner case). Basically, any two async devices separeted by a series of sync ones are likely not to be suspended (or resumed) in parallel with each other, because the parent is usually next to its children in dpm_list. So, if the first device suspends, its "synchronous" parent waits for it and the suspend of the second async device won't be started until the first one's suspend has returned. And it doesn't matter at what level we do the async thing, because dpm_list is there anyway.

As Alan said, the real problem is that we generally can't change the ordering of dpm_list arbitrarily, because we don't know what's going to happen as a result. The async_suspend flag tells us, basically, what devices can be safely moved to different positions in dpm_list without breaking things, as long as they are not moved behind their parents or in front of their children.

Starting the async suspends upfront would effectively work in the same way as moving those devices to the beginning of dpm_list without breaking the parent-child chains, which in turn is likely to allow us to save some extra time.

That's not only about the PCI bridges, it's more general. As far as your one-liner is concerned, I'm going to test it, because I think we could use it anyway.

Rafael

#214 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 5 lines · Rafael J. Wysocki

Because the PCI bridges are not the only case where it matters (I'd say they are really a corner case). Basically, any two async devices separeted by a series of sync ones are likely not to be suspended (or resumed) in parallel with each other, because the parent is usually next to its children in dpm_list.

Give a real example that matters.

Really.

How hard can it be to understand: KISS. Keep It Simple, Stupid.

I get really tired of this whole stupid async discussion, because you're overdesigning it.

To a first approximation, THE ONLY THING THAT MATTERS IS USB.

Linus

#215 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

In reply to Rafael J. Wysocki
Quoted 5 lines · Rafael J. Wysocki

Because the PCI bridges are not the only case where it matters (I'd say they are really a corner case). Basically, any two async devices separeted by a series of sync ones are likely not to be suspended (or resumed) in parallel with each other, because the parent is usually next to its children in dpm_list.

Give a real example that matters.

Really.

How hard can it be to understand: KISS. Keep It Simple, Stupid.

I get really tired of this whole stupid async discussion, because you're overdesigning it.

To a first approximation, THE ONLY THING THAT MATTERS IS USB.

Linus

#216 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 3 lines · Linus Torvalds

I get really tired of this whole stupid async discussion, because you're overdesigning it.

Btw, this is important. I'm not going to pull even your _current_ async stuff if you can't show that you fundamentally UNDERSTAND this fact.

Stop making up idiotic complex interfaces. Look at my one-liner patch, and realize that it gets you 99% there - the 99% that matters.

Linus

#217 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

I get really tired of this whole stupid async discussion, because you're overdesigning it.

Btw, this is important. I'm not going to pull even your _current_ async stuff if you can't show that you fundamentally UNDERSTAND this fact.

Stop making up idiotic complex interfaces. Look at my one-liner patch, and realize that it gets you 99% there - the 99% that matters.

Linus

#218 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 8 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds

I get really tired of this whole stupid async discussion, because you're overdesigning it.

Btw, this is important. I'm not going to pull even your _current_ async stuff if you can't show that you fundamentally UNDERSTAND this fact.

What fact? The only thing that matters is USB? For resume, it is. For suspend, it clearly isn't.

Quoted 2 lines

Stop making up idiotic complex interfaces. Look at my one-liner patch, and realize that it gets you 99% there - the 99% that matters.

I said I was going to use it, but I don't think that's going to be sufficient.

[BTW, I'm not sure what you want to achieve by insulting me. Either you may want to scare me, but I'm not scared, or you may want to try to make me so disgusted that I'll just give up and back off, but this is not going to happen either.]

Insults aside, I'm going to make some measurements to see how much time we can save.

Rafael

#219 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 3 lines · Rafael J. Wysocki

What fact? The only thing that matters is USB? For resume, it is. For suspend, it clearly isn't.

For suspend, the only other case we've seen has been the keyboard and mouse controller, which has exactly the same "we can special case it with a single 'let's do _this_ device asynchronously'". Again, it may not be pretty, but it sure is simple.

Much simpler than talking about some generic infrastructure changes and about doing "let's do leaves of the tree separately" schemes.

And that's why I'm _soo_ unhappy with you, and am insulting you. Because you keep on making the same mistake over and over - overdesigning.

Overdesigning is a SIN. It's the archetypal example of what I call "bad taste". I get really upset when a subsystem maintainer starts overdesigning things.

Linus

#220 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

In reply to Rafael J. Wysocki
Quoted 3 lines · Rafael J. Wysocki

What fact? The only thing that matters is USB? For resume, it is. For suspend, it clearly isn't.

For suspend, the only other case we've seen has been the keyboard and mouse controller, which has exactly the same "we can special case it with a single 'let's do _this_ device asynchronously'". Again, it may not be pretty, but it sure is simple.

Much simpler than talking about some generic infrastructure changes and about doing "let's do leaves of the tree separately" schemes.

And that's why I'm _soo_ unhappy with you, and am insulting you. Because you keep on making the same mistake over and over - overdesigning.

Overdesigning is a SIN. It's the archetypal example of what I call "bad taste". I get really upset when a subsystem maintainer starts overdesigning things.

Linus

#221 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds

I get really tired of this whole stupid async discussion, because you're overdesigning it.

Btw, this is important. I'm not going to pull even your _current_ async stuff if you can't show that you fundamentally UNDERSTAND this fact.

What fact? The only thing that matters is USB? For resume, it is. For suspend, it clearly isn't.

Quoted 2 lines

Stop making up idiotic complex interfaces. Look at my one-liner patch, and realize that it gets you 99% there - the 99% that matters.

I said I was going to use it, but I don't think that's going to be sufficient.

[BTW, I'm not sure what you want to achieve by insulting me. Either you may want to scare me, but I'm not scared, or you may want to try to make me so disgusted that I'll just give up and back off, but this is not going to happen either.]

Insults aside, I'm going to make some measurements to see how much time we can save.

Rafael

#222 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

In reply to Linus Torvalds
Quoted 9 lines · Linus Torvalds
Quoted 5 lines · Rafael J. Wysocki

Because the PCI bridges are not the only case where it matters (I'd say they are really a corner case). Basically, any two async devices separeted by a series of sync ones are likely not to be suspended (or resumed) in parallel with each other, because the parent is usually next to its children in dpm_list.

Give a real example that matters.

I'll try. Let -> denote child-parent relationships and assume dpm_list looks like this:

..., A->B->C, D, E->F->G, ...

where A, B, E, F are all async and C, D, G are sync (E, F, G may be USB and A, B, C may be serio input devices and D is a device that just happens to be in dpm_list between them). Say A and C take the majority of the total suspend time and assume we traverse the dpm_list from left to right.

Now, during suspend, C waits for B that waits for A and G waits for F that waits for E. Moreover, since C is sync, the PM core won't start the suspend of D until the suspend of C has returned. In turn, since D is sync, the suspend of E won't be started until the suspend of D has returned. So in this situation the gain from the async suspends of A, B, E, F is zero.

However, it won't be zero if we start the async suspends of A, B, E, F upfront.

I'm not sure if this is sufficiently "real life" for you, but this is how dpm_list looks on one of my test boxes, more or less.

Quoted 8 lines

Really.

How hard can it be to understand: KISS. Keep It Simple, Stupid.

I get really tired of this whole stupid async discussion, because you're overdesigning it.

To a first approximation, THE ONLY THING THAT MATTERS IS USB.

If this applies to _resume_ only, then I agree, but the Arjan's data clearly show that serio devices take much more time to suspend than USB.

But if we only talk about resume, the PCI bridges don't really matter, because they are resumed before all devices that depend on them, so they don't really need to wait for anyone anyway.

Rafael

#223 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 5 lines · Rafael J. Wysocki
Quoted 2 lines

Give a real example that matters.

I'll try. Let -> denote child-parent relationships and assume dpm_list looks like this:

No.

I mean something real - something like

- if you run on a non-PC with two USB buses behind non-PCI controllers.

- device xyz.

Quoted 2 lines

If this applies to _resume_ only, then I agree, but the Arjan's data clearly show that serio devices take much more time to suspend than USB.

I mean in general - something where you actually have hard data that some device really needs anythign more than my one-liner, and really _needs_ some complex infrastructure.

Not "let's imagine a case like xyz".

Quoted 3 lines

But if we only talk about resume, the PCI bridges don't really matter, because they are resumed before all devices that depend on them, so they don't really need to wait for anyone anyway.

But that's my _point_. That's the whole point of the one-liner patch. Read the comment above that one-liner.

My whole point was that by doing the whole "wait for children" in generic code, you also made devices - such as PCI bridges - have to wait for children, even though they don't need to, and don't want to.

So I suggested an admittedly ugly hack to take care of it - rather than some complex infrastructure.

Linus

#224 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

In reply to Rafael J. Wysocki
Quoted 5 lines · Rafael J. Wysocki
Quoted 2 lines

Give a real example that matters.

I'll try. Let -> denote child-parent relationships and assume dpm_list looks like this:

No.

I mean something real - something like

- if you run on a non-PC with two USB buses behind non-PCI controllers.

- device xyz.

Quoted 2 lines

If this applies to _resume_ only, then I agree, but the Arjan's data clearly show that serio devices take much more time to suspend than USB.

I mean in general - something where you actually have hard data that some device really needs anythign more than my one-liner, and really _needs_ some complex infrastructure.

Not "let's imagine a case like xyz".

Quoted 3 lines

But if we only talk about resume, the PCI bridges don't really matter, because they are resumed before all devices that depend on them, so they don't really need to wait for anyone anyway.

But that's my _point_. That's the whole point of the one-liner patch. Read the comment above that one-liner.

My whole point was that by doing the whole "wait for children" in generic code, you also made devices - such as PCI bridges - have to wait for children, even though they don't need to, and don't want to.

So I suggested an admittedly ugly hack to take care of it - rather than some complex infrastructure.

Linus

#225 lore raw

Re: Async suspend-resume patch w/ completions (was: Re: Async suspend-resume patch w/ rwsems)

Quoted 6 lines · Linus Torvalds

My whole point was that by doing the whole "wait for children" in generic code, you also made devices - such as PCI bridges - have to wait for children, even though they don't need to, and don't want to.

So I suggested an admittedly ugly hack to take care of it - rather than some complex infrastructure.

It doesn't feel like an ugly hack to me. It seems like exactly the Right Thing To Do: Make as many devices as possible use async suspend/resume.

The only reason we don't make every device async is because we don't know whether it's safe. In the case of PCI bridges we _do_ know -- because they don't have any work to do outside of late_suspend/early_resume -- and so they _should_ be async.

The same goes for devices that don't have suspend or resume methods.

There remains a separate question: Should async devices also be forced to wait for their children? I don't see why not. For PCI bridges it won't make any significant difference. As long as the async code doesn't have to do anything, who cares when it runs?

Alan Stern