Skip to contentSkip to conversation

New Driver Model for 2.5

RFC Benjamin Herrenschmidt 17 participants 70 messages Oct 19, 2001 – Oct 26, 2001

lore
Share & tools
lore
#0 lore raw

Reading about the suspend to disk issue, and thinking about some of pmac needs, I tend to stil think we have overlooked that ordering issue.

We should probably add a couple of list_heads to define a second tree in parallell to the device-tree, which is the power tree. A device is by default inserted in both tree as a child of it's bus controller.
But the arch must be able to move it elsewhere. I beleive we have a way around the VM related ordering issues, but we do have other kind of ordering constraints that have to be dealt with when we start broadcasting the callbacks.

Also, I think you didn't state that io_bus is a superset of device. In fact, it's just a device that has childs, and this should probably be more generically viewed in struct device itself.

Any device should be able to have childs, so we really have 2 interleaved trees of devices, the bus tree and the power tree. In fact, to be complete, we could even define the interrupt tree with one more set of links as it's really not related to the bus tree on many archs/machines, and having a tree definition is really useful when you deal with cascaded controllers.

What do you think ?

Ben.

#1 lore raw
Quoted 4 lines · Benjamin Herrenschmidt

Reading about the suspend to disk issue, and thinking about some of pmac needs, I tend to stil think we have overlooked that ordering issue.

Why?

If there is some ordering inherent in the bus, that has to be shown in the bus structure. Why would you EVER care about order between devices that are independent?

Linus

#2 lore raw
Quoted 5 lines

Why?

If there is some ordering inherent in the bus, that has to be shown in the bus structure. Why would you EVER care about order between devices that are independent?

The power tree layout isn't necessarily identical to the bus tree layout. On some macs, for example, we have some ASICs that can control some other chip's clock and power lines, without having a direct parent relationchip.

So I like having the ability to reorder the power tree layout from arch code. But I can work around if this ability is not provided by the struct device. In fact, my main issue here is with Apple's big "mac-io" ASIC (combo of several devices along with various IO lines and clocks), and I beleive I will have to handle it as a special case anyway for other reason (it must really be shut down last as once down, I can't even talk to the power manager chip ;) I think all other devices I have to deal with follow the physical bus ordering.

The problem of suspend-to-disk, which requires, I beleive, that the device used for the memory backup, to be state-saved last, is still a problem I don't know how to solve. Maybe using flags in the device structure indicating it's deferred. That would cause it's parents to be deferred as well. The presence of the flag would prevent the actual "suspend" state to be entered during step 3. Once all devices are suspended, we then know that the bus path to the disk used for suspend-to-disk is still powered and perform the actual suspend-to disk operation.

However, I beleive that requires using non-generic IO functions (as IO queues for the controller have already been blocked by step 2, and the driver would have to deal with it's own saved state carefuly as it can't obviously save state to RAM after it has been used to backup the RAM itself). Maybe that could be a separate message (suspend_to_disk) sent instead of step 3 (suspend) to this device.

That would give us the following scenario:

- The device for suspend-to-disk is identified and a flag is set in it's device structure. This flag (or a different one to make things clear eventually) is "broadcast" all the way up the tree so it's parent brigdes/controllers are marked as well.
- All devices get "suspend_prepare".
- All devices get "suspend_save_state" and block normal IOs
- All devices not marked above get "suspend"
- Last housecleaning is done by the kernel.
- The device marked above get a special "suspend_to_disk" message during which it can perform the actual memory backup and suspend itself.
- The machine is put to sleep.

I currently don't implement suspend-to-disks on Mac, so I may have overlooked something. Also, I'm not too sure about the requirements of x86 laptops regarding those features. I'm lucky, Mac laptops keep the RAM content during suspend ;)

Note also that if not doing suspend-to-disk, I think we should also make sure to sync all buffers after suspend_prepare and before sending the suspend_save_state messages. I noticed that recent 2.4 versions are more sensible about power fail during suspend (that is battery getting empty, or whatever causing lose of RAM content). I used to call fsync_devs(0) between those 2 steps in the Mac PM scheme, but it appears that with recent 2.4's, this doesn't prevent fsck from finding inconsistencies.

Ben.

#3 lore raw
In reply to Linus Torvalds

Hi!

Quoted 3 lines

The problem of suspend-to-disk, which requires, I beleive, that the device used for the memory backup, to be state-saved last, is still a problem I don't know how to solve. Maybe using flags in the device

Don't care about it. Its easy.

Quoted 6 lines

That would give us the following scenario:

- The device for suspend-to-disk is identified and a flag is set in it's device structure. This flag (or a different one to make things clear eventually) is "broadcast" all the way up the tree so it's parent brigdes/controllers are marked as well.

You don't need this.

Quoted 3 lines

- All devices get "suspend_prepare".
- All devices get "suspend_save_state" and block normal IOs
- All devices not marked above get "suspend"

... not needed. You are going powerdown (suspend-to-disk ends in
powerdown, right?), so you don't care about state devices are in. You don't need to suspend them.

You just write state to disk and powerdown, now.
Pavel

Signature · 2 lines
I'm [email protected]. "In my country we have almost anarchy and I don't care."
Panos Katsaloulis describing me w.r.t. patents at [email protected]
#4 lore raw
Quoted 9 lines
Quoted 7 lines

That would give us the following scenario:

- The device for suspend-to-disk is identified and a flag is set in it's device structure. This flag (or a different one to make things clear eventually) is "broadcast" all the way up the tree so it's parent brigdes/controllers are marked as well.

You don't need this.

Correct. Suspend-to-disk is a process, not a discrete action. It goes much like Ben described:

Quoted 1 line

From the system point of view (slightly different from the driver's point

of view):

- notify devices of pending suspension; check for failures
- tell devices to suspend state; write this state to disk
- turn all devices off
- power off (aka suspend system)

Quoted 8 lines
Quoted 3 lines

- All devices get "suspend_prepare".
- All devices get "suspend_save_state" and block normal IOs
- All devices not marked above get "suspend"

... not needed. You are going powerdown (suspend-to-disk ends in
powerdown, right?), so you don't care about state devices are in. You don't need to suspend them.

You just write state to disk and powerdown, now.

Uhm...That would probably work. But, I would rather explicitly turn off all the devices. In the world of ACPI, S4 and S5 (Suspend to disk and soft-off respectively) don't power the system completely off; some power remains to capture wake events.

[ One point to note is that in an ACPI-enabled system, we use S5 to power the system off. But, there are things still running in the system. Everything is not shut off until you perform a mechanical off (unplug it). This means that you can get wake events and cause the system to boot after you think you've turned it off. This is why some people are experiencing reboots when the system should be powering down. ]

By explicitly turning off as many devices as possible, we're playing more on the safe side of things, and possibly reducing the amount of power that is being consumed.

Btw, I updated the model to support an n-stage suspend process, with 3 stages explicitly defined, as per some discussion about it. Those stages are:

SUSPEND_NOTIFY
SUSPEND_SAVE_STATE
SUSPEND_POWER_DOWN

To suspend the device tree, one would do something like:

	/* Tell all the devices we're going to sleep.
	 * This also allows them to allocate memory before the swap
	 * device stops taking orders.
	 */
	device_suspend(3, SUSPEND_NOTIFY);

	/* if someone failed, get out now */

	/* Now tell them to stop I/O and save their state */
	device_suspend(3, SUSPEND_SAVE_STATE);

	/* Write the state to disk... */

	/* Finally, turn all of the devices off. */
	device_suspend(3, SUSPEND_POWER_DOWN);

	/* Now, put the system to sleep. */

I also updated the docs. It can all be found at:

http://kernel.org/pub/linux/kernel/people/mochel/device/

-pat

#5 lore raw
Quoted 2 lines
	/* Now tell them to stop I/O and save their state */
	device_suspend(3, SUSPEND_SAVE_STATE);

I'd very much like this one to be two pass, with the second pass occuring after interrupts are disabled. There are some horrible cases to try and handle otherwise (like devices that like to jam the irq line high).

Ditto on return from suspend where some devices also like to float the irq high as you take them over (eg USB on my Palmax). From comments Ben made ages back I believe ppc has similar issues if not worse

Alan

#6 lore raw
Quoted 6 lines · Alan Cox
Quoted 2 lines
	/* Now tell them to stop I/O and save their state */
	device_suspend(3, SUSPEND_SAVE_STATE);

I'd very much like this one to be two pass, with the second pass occuring after interrupts are disabled. There are some horrible cases to try and handle otherwise (like devices that like to jam the irq line high).

I forgot to mention to disable interrupts after the SUSPEND_NOTIFY call. The idea is to allocate all memory in the first pass, disable interrupts, then save state. Would that work? Or, should some of the state saving take place with interrupts enabled?

Quoted 3 lines

Ditto on return from suspend where some devices also like to float the irq high as you take them over (eg USB on my Palmax). From comments Ben made ages back I believe ppc has similar issues if not worse

Yes, the resume sequence is broken into two stages:

	device_resume(RESUME_POWER_ON);

	/* enable interrupts */

	device_resume(RESUME_RESTORE_STATE);

Do you see a need to break it up further?

-pat

#7 lore raw
Quoted 3 lines

The idea is to allocate all memory in the first pass, disable interrupts, then save state. Would that work? Or, should some of the state saving take place with interrupts enabled?

Imagine the state saving done on a USB device. There you need interrupts on while retrieving the state from say a USB scanner, and in some cases off while killing the USB controller.

Quoted 6 lines
Quoted 3 lines

Ditto on return from suspend where some devices also like to float the irq high as you take them over (eg USB on my Palmax). From comments Ben made ages back I believe ppc has similar issues if not worse

Yes, the resume sequence is broken into two stages:

Quoted 7 lines
	device_resume(RESUME_POWER_ON);

	/* enable interrupts */

	device_resume(RESUME_RESTORE_STATE);

Do you see a need to break it up further?

Nope.

#8 lore raw
Quoted 7 lines · Alan Cox
Quoted 3 lines

The idea is to allocate all memory in the first pass, disable interrupts,
then save state. Would that work? Or, should some of the state saving take place with interrupts enabled?

Imagine the state saving done on a USB device. There you need interrupts on while retrieving the state from say a USB scanner, and in some cases off while killing the USB controller.

Is this a realistic example? That is, is a kernel-side driver likely to be able to meaningfully extract state information from a scanner? And is it necessary?

And for a scanner, if the current operation is a scan generating a GB of data, what happens if the disk subsystem is no longer accepting requests?

As Jeff Garzik pointed out, NIC drivers typically don't need to save any state at all; it's all recreateable from software structures. Perhaps that characteristic can and should be generalized to other devices.

In that case, SUSPEND_SAVE_STATE becomes more like SUSPEND_QUIESCE: stop accepting new requests, and complete current requests.

"Stop accepting new requests" is nontrivial as well, in the general case. New requests that can't be discarded need to be queued somewhere. Whose responsibility is that? Ideally at some point where a queue already exists, possibly in the requester.

Signature · 1 line
/Jonathan Lundell.
#9 lore raw
Quoted 3 lines

Is this a realistic example? That is, is a kernel-side driver likely to be able to meaningfully extract state information from a scanner? And is it necessary?

It may be a bad example - but think about things like page settings. Do you want a resume to scan in colour when you set black and white just before suspend ?

Quoted 3 lines

And for a scanner, if the current operation is a scan generating a GB of data, what happens if the disk subsystem is no longer accepting requests?

It should have refused to suspend because it was active

Quoted 2 lines

In that case, SUSPEND_SAVE_STATE becomes more like SUSPEND_QUIESCE:
stop accepting new requests, and complete current requests.

Maybe. That sounds like nice design and horrible implementation

#10 lore raw
In reply to Jonathan Lundell
Quoted 4 lines

"Stop accepting new requests" is nontrivial as well, in the general case. New requests that can't be discarded need to be queued somewhere. Whose responsibility is that? Ideally at some point where a queue already exists, possibly in the requester.

Some driver already handle queues. In the case of network driver, just stop your network queue and stop accepting incoming packets. If your driver is too simple to have queues, a simple semaphore on entry points can often be enough. You shouldn't deadlock as you are not supposed to re-enter a sleeping driver in step 2.

The above, is ensured by the tree layout which does the dependency ordering. You might have slightly off-tree dependencies, like I have in a couple of case on macs. But I figured that all of them could be handled as special case in some parent nodes without beeing that dirty (in most case, those are Apple specific ASICs containing devices with inter-deps, and the workaround is to move some devices sleep code to the node of the ASIC itself).

Ben.

#11 lore raw
Quoted 10 lines
Quoted 4 lines

"Stop accepting new requests" is nontrivial as well, in the general case. New requests that can't be discarded need to be queued somewhere. Whose responsibility is that? Ideally at some point where a queue already exists, possibly in the requester.

Some driver already handle queues. In the case of network driver, just stop your network queue and stop accepting incoming packets. If your driver is too simple to have queues, a simple semaphore on entry points can often be enough. You shouldn't deadlock as you are not supposed to re-enter a sleeping driver in step 2.

Stop accepting new requests is not simple. To complete existing requests you might need an arbitary other module to complete a new request you submit as part of your shutdown.

Alan

#12 lore raw
Quoted 9 lines
Quoted 5 lines

Some driver already handle queues. In the case of network driver, just stop your network queue and stop accepting incoming packets. If your driver is too simple to have queues, a simple semaphore on entry points can often be enough. You shouldn't deadlock as you are not supposed to re-enter a sleeping driver in step 2.

Stop accepting new requests is not simple. To complete existing requests you might need an arbitary other module to complete a new request you submit as part of your shutdown.

That mean you have an ordering dependency, the driver you rely upon must be stopped after you. That's the point of having a tree here. Patrick and Linus feel the bus tree is enough to handle that dependency, which might well be the case for 99% of drivers.

I have a couple of cases where that's not completely true on pmacs, but nothing that can't worked around simply I beleive.

If you feel more drivers will be affected, then we will probably need to separate the power-tree from the device-tree and provide some hooks so that ordering can be tweaked.

All this assumes you don't have circular dependencies of course ;)

I see a lot of cases where this "block IOs" is easily dealt with in the drivers I maintain on pmac, that might not be that easy on other archs, I can't tell.

Basically, simple drivers can just use a semaphore. I do that for our sound driver for example, I block any app doing an ioctl while the driver is sleeping. (This happens late enough in the sleep process so that userland using /dev/apm_bios already got notified and acked the suspend, letting properly written apps to have stopped themselves already).

Drivers using a request queue usually already have a way to mark themselves busy (they use that to decide if they have to kick the HW or not when getting a new request). In cases where a mid-layer enters the scene, like SCSI, that wants to do timeouts, then well... we can let it timeout (just stop processing requests), or we can have the midlayer go to sleep as well :) That later solution may cause some interesting ordering issues however...

Network drivers can stop their queue or just drop packets... I'd like if they waited for packets received from the network stack before the callback is called are waited to be sent. Those packets may contain the request to a server to send a wake-on-lan magic packet to your machine ;) For now, I just block the output queue and flush the rings on pmac, but I also dont support WOL yet.

For fbdevs, I simply switch them to dummy functions when asleep. This appear to work well. (Well, I do some additional state save and PM, but all I do for "blocking IOs" is to drop them...) Any printk done after they are suspended isn't displayed, but that's not a real issue.

So yes, "blocking IOs" can actually mean "dropping new IOs", that depends very much on the driver.

For USB, for example, we can consider that when a device driver (not a controller driver) suspend has been done, any URB it submits can just be dropped (returned immediately with an error). We don't need blocking here neither. Of course, that means we have the framework to call devices' suspend/resume callbacks when the controller is about to go to sleep.

There might be other examples. I agree it's not a 2 lines fix per driver, but that's the better I could imagine so far to have something reliable.

If you have other ideas, please share.

Ben.

#13 lore raw
Quoted 3 lines

upon must be stopped after you. That's the point of having a tree here. Patrick and Linus feel the bus tree is enough to handle that dependency, which might well be the case for 99% of drivers.

The two trees are certainly closely related - USB devices before USB hub, USB hub before PCI etc. The scanner example works fine there, providing that we are careful about memory issues - remember the USB layer allocates memory to do any transaction, so the scanner has to complete its state save before we do any interrupt disabling/memory alloc freezing.

Thats still just ordering and maybe two passes

Quoted 5 lines

the HW or not when getting a new request). In cases where a mid-layer enters the scene, like SCSI, that wants to do timeouts, then well... we can let it timeout (just stop processing requests), or we can have the midlayer go to sleep as well :) That later solution may cause some interesting ordering issues however...

For scsi you have to complete the pending commands, you don't know what the transaction granularity is in some cases and half completing the sequence won't help you. In addition the upper layers have to queue additional scsi commands to do stuff like cd drawer locking and to ask the drive firmware to enter powerdown modes

Quoted 6 lines

For USB, for example, we can consider that when a device driver (not a controller driver) suspend has been done, any URB it submits can just be dropped (returned immediately with an error). We don't need blocking here neither. Of course, that means we have the framework to call devices' suspend/resume callbacks when the controller is about to go to sleep.

That will scramble large numbers of devices. Randomly erroring pending block writes is -not- civilised.

Alan

#14 lore raw
Quoted 6 lines

The two trees are certainly closely related - USB devices before USB hub, USB hub before PCI etc. The scanner example works fine there, providing that we are careful about memory issues - remember the USB layer allocates memory to do any transaction, so the scanner has to complete its state save before we do any interrupt disabling/memory alloc freezing.

That's why we have that first step that is run before any device is blocked and with interrupts still flowing. Devices that need memory for either state save or for requests used during power down are supposed to allocate them at this point.

Quoted 1 line

Thats still just ordering and maybe two passes

Well, 3 passes actually ;) I'd suggest you re-read Patrick's mochel latest document, which describes the 3 step process. One first pass give a chance to device to "prepare" for sleep, that is allocate anything they will need without actually blocking or suspending anything. On the second pass, devices are asked to suspend IOs, complete pending ones, and save state. This is done with interrupts still enabled. The 3rd pass is called with interrupt disabled and is where the actual shutdown of the device is supposed to happen.

In practice, a lot of drivers will only need to implement 1 or 2 of these 3 passes. But the flexibility has to be there for a device that need both\
to allocate memory and to suspend with interrupts disabled.

An additional idea we had was to make pass 2 somewhat asyncrhonous by allowing some kind of "call my later" result code (basically, a device would block it's queue, and return "call me later" while it still has pending IOs). This is an optional "feature".

Quoted 11 lines
Quoted 5 lines

the HW or not when getting a new request). In cases where a mid-layer enters the scene, like SCSI, that wants to do timeouts, then well... we can let it timeout (just stop processing requests), or we can have the midlayer go to sleep as well :) That later solution may cause some interesting ordering issues however...

For scsi you have to complete the pending commands, you don't know what the transaction granularity is in some cases and half completing the sequence won't help you. In addition the upper layers have to queue additional scsi commands to do stuff like cd drawer locking and to ask the drive firmware to enter powerdown modes

Yes. SCSI is a problem as the current SCSI layer is tricky enough to make this difficult. I beleive that if the SCSI devices are childs of the SCSI controller, then they can take care of suspending the device (that is sending whatever command to lock the drawer or stop the disk spinning) before the controller is actually going to sleep. In that case, there's not much left to the controller, it isn't supposed to have any command in queue nor receive any new one once all it's child drivers have suspended.

Quoted 9 lines
Quoted 6 lines

For USB, for example, we can consider that when a device driver (not a controller driver) suspend has been done, any URB it submits can just be dropped (returned immediately with an error). We don't need blocking here neither. Of course, that means we have the framework to call devices' suspend/resume callbacks when the controller is about to go to sleep.

That will scramble large numbers of devices. Randomly erroring pending block writes is -not- civilised.

The drivers will have to be adapted for PM, whatever scheme we use. The USB case is very similar to SCSI. The controller is a parent of all devices. Devices will get the suspend before the controller, they will have a chance to suspend requests and wait for pending ones to complete (for example, the USB storage will have a chance to block new incoming requests in the queue and wait for pending ones to complete) before the USB controller is put to suspend.
That way, once we are reaching real USB controller suspend, we can safely discard urbs as we are not supposed to get any until devices have been resumed.

I currently don't do that on pmac (I just let URBs be handled by OHCI and let OHCI fill the TD queues in memory, I only prevent the controller from actually handling those queues). It's not good as some drivers will get error messages due to their underlying device beeing suspended, and won't understand why the URBs are going away.

In the case of USB devices that don't support suspend state, it's slightly more tricky as on some HW, I may have to actually turn them off. That mean the driver must deal with re-doing configuration & set-interface on wakeup. That is again a driver matter, and a bit like the PCI case we mentioned previously, we need some way to know if driver for a given device can handle the requested power state or not. If not, we should probably abort the suspend sequence, and find a clean interface to tell userland about which device caused the failure so the user can deal with it (rmmod the driver for example).

Ben.

#15 lore raw
Quoted 3 lines

case, there's not much left to the controller, it isn't supposed to have any command in queue nor receive any new one once all it's child drivers have suspended.

scsi devices are children of the scsi subststem (sd, sg, sr, st, osst) not of the controller. That is how the state flows anyway. Only sr/sd etc know what the state is for a given device on power off as they may issue multiple requests per action true transaction. sg would have to simply refuse any suspend if open (think about cd-burning or even worse firmware download)

So the scsi devices hang off sd, sr etc which in turn hang off scsi and the controllers hang off scsi (and or the bus layers)

This one at least I think I do understand

#16 lore raw
Quoted 11 lines
Quoted 3 lines

case, there's not much left to the controller, it isn't supposed to have any command in queue nor receive any new one once all it's child drivers have suspended.

scsi devices are children of the scsi subststem (sd, sg, sr, st, osst) not of the controller. That is how the state flows anyway. Only sr/sd etc know what the state is for a given device on power off as they may issue multiple requests per action true transaction. sg would have to simply refuse any suspend if open (think about cd-burning or even worse firmware download)

Quoted 4 lines

So the scsi devices hang off sd, sr etc which in turn hang off scsi and the controllers hang off scsi (and or the bus layers)

This one at least I think I do understand

The problem with subsystems is that they don't fit well in the power tree. They aren't "devices" in that sense that they are not exposing a struct device, and they spawn over several controllers which means the dependency can quickly become unmanageable, especially when SCSI starts beeing layered on top of USB or FireWire.

Also, the dependency issue is made worst if you let RAID enter into the dance as I beleive ultimately, nothing would prevent a volume to spawn over several devices from different controllers or even different controller types.

So let's see if I properly understand what is needed in the SCSI case:

The parent is the controller. We can't do much about this since we need that relationchip for ordering. By controller, it can be a real SCSI host, but it can also be a virtual host exposed by an USB storage device or a firewire SBP2 device.

The child of this controller has to be a struct device for each physical device on the bus. (just one in the case of an USB storage). The struct device for this child is +/- generic, possibly created by the generic SCSI probe code.

This device might (must ?) have childs instanciated by whatever "client" attach to a given SCSI device. Clients like sg would effectively refuse suspend, while clients like sd would do standard disk spindown commands. That mean there is not "one" PM node for the SCSI subsystem, but one per instance of a given subsystem module.

Now, I'm not sure what would happen with RAID. If we need to have logical volumes be child of the sd "client", then we have to face the fact that a given child may have multiple parents... welcome to the power graph ! But do we really need logical volumes to be part of the PM tree or can blocking of requests at the sd layer be enough ? Remember we are in pass2, we have already done memory allocation, we are supposed to no longer swap nor do any disk/storage related activity.

A tricky issue indeed...

Ben.

#17 lore raw
Quoted 7 lines

Now, I'm not sure what would happen with RAID. If we need to have logical volumes be child of the sd "client", then we have to face the fact that a given child may have multiple parents... welcome to the power graph ! But do we really need logical volumes to be part of the PM tree or can blocking of requests at the sd layer be enough ? Remember we are in pass2, we have already done memory allocation, we are supposed to no longer swap nor do any disk/storage related activity.

Assuming you want to synchronize the raid before suspend - a reasonably policy but not essential then you'd have to shut down the raid before sd, then sd would let the devices shut down which lets the controller shutdown

#18 lore raw
Quoted 5 lines · Alan Cox

Assuming you want to synchronize the raid before suspend - a reasonably policy but not essential then you'd have to shut down the raid before sd, then sd would let the devices shut down which lets the controller shutdown

I will _refuse_ to have a kernel suspend that synchronizes the raid etc. That would make suspend/resume potentially take a _loong_ time.

If you want to synchronize your raid thing, make the user-level thing that triggers the suspend do it. Same goes for things like "sync network filesystems" etc. This is not a kernel level issue, and the kernel shouldn't even try to do it.

If somebody has pending stuff over NFS and suspends, and when it comes back it's not on the network any more, that is 100% equivalent to removing a PCMCIA network card while running. It's supposed to work - but if you lose data that's YOUR problem, not the kernels.

Linus

#19 lore raw
Quoted 4 lines

If you want to synchronize your raid thing, make the user-level thing that triggers the suspend do it. Same goes for things like "sync network filesystems" etc. This is not a kernel level issue, and the kernel shouldn't even try to do it.

Makes good sense - I agree

#20 lore raw
In reply to Alan Cox
Quoted 9 lines · Linus Torvalds
Quoted 4 lines · Alan Cox

Assuming you want to synchronize the raid before suspend - a reasonably policy but not essential then you'd have to shut down the raid before sd, then sd would let the devices shut down which lets the controller shutdown

I will _refuse_ to have a kernel suspend that synchronizes the raid etc. That would make suspend/resume potentially take a _loong_ time.

If you have Magic SysRq enabled, would that do the job prior
to suspend? Typically with Pavel's swsusp package, I hit the Alt-SysRq-s before hitting Alt-SysRq-d to suspend him. Does Alt-SysRq-s synchronize a raid? Of course, at that point, the choice to take the "_loong_ time" is in user space - meat space, user space - since I chose to hit that key combination.

Quoted 4 lines

If you want to synchronize your raid thing, make the user-level thing that triggers the suspend do it. Same goes for things like "sync network filesystems" etc. This is not a kernel level issue, and the kernel shouldn't even try to do it.

What does the Alt-SysRq-s combination do about networks then?

Quoted 6 lines

If somebody has pending stuff over NFS and suspends, and when it comes back it's not on the network any more, that is 100% equivalent to removing a PCMCIA network card while running. It's supposed to work - but if you lose data that's YOUR problem, not the kernels.

Linus

Mike

Signature · 4 lines
 Michael H. Warfield    |  (770) 985-6132   |  [email protected]
  /\/\|=mhw=|\/\/       |  (678) 463-0932   |  http://www.wittsend.com/mhw/
  NIC whois:  MHW9      |  An optimist believes we live in the best of all
 PGP Key: 0xDF1DD471    |  possible worlds.  A pessimist is sure of it!
#21 lore raw
Quoted 10 lines · Michael H. Warfield
Quoted 3 lines

I will _refuse_ to have a kernel suspend that synchronizes the raid etc. That would make suspend/resume potentially take a _loong_ time.

If you have Magic SysRq enabled, would that do the job prior
to suspend? Typically with Pavel's swsusp package, I hit the Alt-SysRq-s before hitting Alt-SysRq-d to suspend him. Does Alt-SysRq-s synchronize a raid? Of course, at that point, the choice to take the "_loong_ time" is in user space - meat space, user space - since I chose to hit that key combination.

Sure. I only refuse to have it be "integrated" into the suspend - but it's certainly perfectly fine to have "combination events", whether by having special keystrokes that starts them or by having scripts or programs that first do the sync and then do "echo 3 > /proc/acpi/sleep" or whatever.

Quoted 1 line

What does the Alt-SysRq-s combination do about networks then?

I think it just does a "fsync_dev()", which will do the right thing for network filesystems too.

But let's make an example: let's assume that I'm working on my laptop, and the NFS server goes down so I decide to take a break. Should I not be able to suspend, only because the sync won't finish?

That's the wrong answer. By _default_ I should just suspend, and when I come back it will continue to try to write back the data (not by any magical suspend/resume means, but just because that's what NFS does anyway when the server hasn't answered)

Linus

#22 lore raw
In reply to Benjamin Herrenschmidt
Quoted 11 lines · Benjamin Herrenschmidt
Quoted 5 lines

So the scsi devices hang off sd, sr etc which in turn hang off scsi and the controllers hang off scsi (and or the bus layers)

This one at least I think I do understand

The problem with subsystems is that they don't fit well in the power tree. They aren't "devices" in that sense that they are not exposing a struct device, and they spawn over several controllers which means the dependency can quickly become unmanageable, especially when SCSI starts beeing layered on top of USB or FireWire.

Why would you _ever_ get "sg.c" and other crap involved in the suspend process?

The device tree is for _device_ suspend, not for "subsystem suspend". The SCSI subsystem is a piece of cr*p, but even if it was perfect it should never get involved with the act of suspension.

We should not have pending IO, but that's for a totally different reason:
the first thing the much much MUCH higher levels of suspend should be doing is to make sure that user apps are "quiescent". And that isn't done by getting involved with sg.c or anything similar, but by basically stopping all user apps (think of the equivalent of a "kill -STOP -1", but done internally in the kernel without actually using a signal).

Quoted 4 lines

Also, the dependency issue is made worst if you let RAID enter into the dance as I beleive ultimately, nothing would prevent a volume to spawn over several devices from different controllers or even different controller types.

Why would you get RAID involved? There is no _IO_ involved in suspending:
we just stop doing what we're doing, and leave it at that. We don't try to flush state, we just freeze the machine.

The act of "suspend" should basically be: shut off the SCSI controller, screw all devices, reset the bus on resume.

The act of suspend on USB should be to turn off the host controller and remove power from devices. End of story. Nothing fancy.

If somebody removes a disk or equivalent while we're suspended, that's _his_ problem, and is exactly the same as removing a disk while the disk is running. Either the subsystem (like USB) already handles it, or it doesn't. Suspend is _not_ an excuse to do anything that isn't done at run-time.

So suspend is _not_ supposed to be equivalent of a full clean shutdown with just users not seeing it. That's way too expensive to be practical.
Remember: the main point of suspend is to have a laptop go to sleep, and come back up on the order of a few _seconds_.

And if there are desktops which would like to suspend but cannot because they aren't strictly designed for it, then tough - we should not try to design a heavy suspend for hardware that doesn't live with it well.

Also, realize that the act of suspension is STARTED BY THE USER. Which means that before the kernel suspends, you _can_ have user programs that basically take disk arrays off-line etc if that is what you want. But that's not ae kernel suspend issue.

Linus

#23 lore raw
Quoted 1 line · Linus Torvalds

Also, realize that the act of suspension is STARTED BY THE USER. Which

... or triggered by some kind of inactivity timer, or low battery condition.

Xav

#24 lore raw
Quoted 5 lines · Xavier Bestel
Quoted 1 line · Linus Torvalds

Also, realize that the act of suspension is STARTED BY THE USER. Which

... or triggered by some kind of inactivity timer, or low battery condition.

...which should be done in userspace.

-pat

#25 lore raw
In reply to Xavier Bestel
Quoted 6 lines · Xavier Bestel
Quoted 1 line · Linus Torvalds

Also, realize that the act of suspension is STARTED BY THE USER. Which

... or triggered by some kind of inactivity timer, or low battery condition.

Note that even when that happens, it's not supposed to be the kernel start _starts_ the activity of suspension.

An inactivity timer or low battery notification will just notify the proper deamon, and the policy on what to do should be in user space. For example, on low battery you might want to set up a X window warning the user that the machine _will_ suspend in five seconds. And the kernel certainly won't do that.

So as far as the kernel is concerned, a suspend is _always_ started by "the user". Of course, the whole point with computers is that many things can be automated, and "the user" may not be a human sitting at the machine.

Linus

#26 lore raw
Quoted 5 lines

So as far as the kernel is concerned, a suspend is _always_ started by "the user". Of course, the whole point with computers is that many thin= gs
can be automated, and "the user" may not be a human sitting at the machine.

How does that apply to the equivalent of an APM critical shutdown - do we still vector that via userspace ?

#27 lore raw
In reply to Linus Torvalds
Quoted 2 lines

Why would you _ever_ get "sg.c" and other crap involved in the suspend process?

Ahhh... ;)

Quoted 3 lines

The device tree is for _device_ suspend, not for "subsystem suspend". The SCSI subsystem is a piece of cr*p, but even if it was perfect it should never get involved with the act of suspension.

I agree I'd like subsystems to avoid polluting the PM tree (or device tree). If there are a few cases where a subsystem needs to know a driver it's using is asleep, it's probably up to the interface of this susbystem to provide a function to be called by the driver when it's going to suspend mode.

I can't really tell how it would be done for SCSI, I admit I got quickly lost in the current drivers/scsi midlayer trying to figure out how it layered things, but Andre told me it should be rewritten. I beleive the proper way here would be to have an _interface_ (no mid-layer) exposed by drivers who can receive SCSI commands (them beeing either USB, ATAPI, FireWire or real SCSI devices).

However, Alan makes a point about state information. For example, the "generic" CD-ROM driver for SCSI-protocol (those CD-ROMs beeing either ATAPI, real SCSI, ...) would be the only one to know some state information like rotation speed or whatever configuration info a given device may support (placing ourselves in an ideal world of course, where all CDs using the SCSI protocol share the same driver). So it must be some way involved in the PM process.

I don't know what people plan for SCSI & ATAPI overhaul in 2.5, but depending on how that scheme is done will impact the PM process.

Quoted 6 lines

We should not have pending IO, but that's for a totally different reason:
the first thing the much much MUCH higher levels of suspend should be doing is to make sure that user apps are "quiescent". And that isn't done by getting involved with sg.c or anything similar, but by basically stopping all user apps (think of the equivalent of a "kill -STOP -1", but done internally in the kernel without actually using a signal).

Is this necessary ? That would definitely make things easier to implement to forget about incoming requests, but I'm not sure it's the right way. In fact, is it really working ? You could well have in-kernel threads triggering IOs or launching new userland stuffs (what happens if a not yet suspended driver for, let's say USB, see a new device coming in and starts /sbin/hotplug). Some filesystems have garbage collector threads that can do IOs eventually. There are various kind of IOs that can in fact be triggered entirely within the kernel.

Also, if you don't stop userland, wakeup is amazingly fast since userland is up very soon, a sound app (mp3 player for example) will be blocked until the driver it's write()'ing to is woken up, a swapping app is blocked until the IO requests for that page is completed (which means if it was blocked in a block driver queue, once the block driver have resumed operations), etc... Currently, on pmacs, i don't have the tree but I do have some kind of priority mecanism. Userland is woken up as soon as the disk is ready. I think scheduling only have to be disabled between step 2 and 3 of the sleep process, that is after all IOs have been blocked and before shutting down devices (that is before the step that runs with IRQs off).

I really don't think it's _that_ difficult to properly do this blocking. For things like sound drivers, a simple semaphore is plenty enough. For network drivers, stopping the output queue is a one-line thing, for block devices, it usually a matter of marking ourselves busy and letting requests pile on our queue. The should be no risk of blocking the sleep process itself this way since we made sure we did all allocations needed by drivers prior to starting the actual blocking of requests, so we won't block swap out.

Userland apps that hack on hardware directly like X will have been suspended earlier (possibly via the existing /dev/apmbios interface but we should rather define a new cleaner way to have PM control from userland.

I have all of this more or less working on pmac laptops. I don't have the new device model, so I handle dependencies manually with a priority mecanism, but it's already good enough to let me resume userland before ADB and sound, and possibly stuffs. (Which is nice since my sound chips usually need one or 2 second to recalibrate and ADB need a few seconds to probe the bus, all this happens asynchronously).

Quoted 8 lines
Quoted 4 lines

Also, the dependency issue is made worst if you let RAID enter into the dance as I beleive ultimately, nothing would prevent a volume to spawn over several devices from different controllers or even different controller types.

Why would you get RAID involved? There is no _IO_ involved in suspending:
we just stop doing what we're doing, and leave it at that. We don't try to flush state, we just freeze the machine.

Ok.

Quoted 2 lines

The act of "suspend" should basically be: shut off the SCSI controller, screw all devices, reset the bus on resume.

Well, some devices will need some state saving. You may need to save and restore some speed setting for example. You want pending requests to be properly terminated before you shut down, etc...

Quoted 2 lines

The act of suspend on USB should be to turn off the host controller and remove power from devices. End of story. Nothing fancy.

Ehh... no, you want to put the USB bus into suspend state, which isn't the same ;) Or you lose the ability to wake up the machine from the USB keyboard, which on some iMacs is I think, the only way.

So before doing so, you need to iterate all childs of the controller (that's fine, they have struct device entries with PM functions in them), and tell them to suspend. Some devices need to be sent a command for enabling remote wakeup. Others may need other housekeeping before the bus goes to suspend.

But there's nothing magic there, nor anything fancy.
Just USB drivers have a struct device for each USB device, and are notified of suspend normally as part of the device tree walk. They are childs of the USB controller, we are guaranteed the controller won't be down before devices.

I agree that is this is properly done, then we know the controller won't have to deal with pending or new incoming IOs once it's its turn to go sleeping, and can just enter it's suspend state immediately.

Quoted 5 lines

If somebody removes a disk or equivalent while we're suspended, that's _his_ problem, and is exactly the same as removing a disk while the disk is running. Either the subsystem (like USB) already handles it, or it doesn't. Suspend is _not_ an excuse to do anything that isn't done at run-time.

Yup.

Quoted 4 lines

So suspend is _not_ supposed to be equivalent of a full clean shutdown with just users not seeing it. That's way too expensive to be practical.
Remember: the main point of suspend is to have a laptop go to sleep, and come back up on the order of a few _seconds_.

Yup.

Quoted 8 lines

And if there are desktops which would like to suspend but cannot because they aren't strictly designed for it, then tough - we should not try to design a heavy suspend for hardware that doesn't live with it well.

Also, realize that the act of suspension is STARTED BY THE USER. Which means that before the kernel suspends, you _can_ have user programs that basically take disk arrays off-line etc if that is what you want. But that's not ae kernel suspend issue.

Yes. We do that on pmac as well. The lid of the laptop is monitored by a userland daemon, which runs scripts before and after sending the real suspend ioctl to our PM driver.

There is also /dev/apm_bios which allows for the suspend process to notify (and wait for ack) userland apps that requested X (in our case, X so it stop banging the HW, properly save it's own state and properly resume the card on wakeup).

However, that interface could as well be completely userland (X requesting notifications from the PM daemon).

Ben.

#28 lore raw
Quoted 8 lines
Quoted 3 lines

The device tree is for _device_ suspend, not for "subsystem suspend". The SCSI subsystem is a piece of cr*p, but even if it was perfect it should never get involved with the act of suspension.

I agree I'd like subsystems to avoid polluting the PM tree (or device tree). If there are a few cases where a subsystem needs to know a driver it's using is asleep, it's probably up to the interface of this susbystem to provide a function to be called by the driver when it's going to suspend mode.

I don't think it is a big problem. We can add virtual nodes. They way I see it we either
a) put in grungy subsystem hacks
b) register virtual device nodes for subsystems when needed

b feels cleaner

Quoted 2 lines

I really don't think it's _that_ difficult to properly do this blocking. For things like sound drivers, a simple semaphore is plenty enough. For

Sound is more easily handled by not blocking user space but waiting until the final IRQ off moment and grabbing the registers. That avoids a lot of ugly locking gunge. It literally comes down to

case suspending
kmalloc buffer
done
case final suspend point
turn off DMA
readl
readl
readl
readl
...
done

#29 lore raw
Quoted 7 lines · Alan Cox

I don't think it is a big problem. We can add virtual nodes. They way I see it we either
a) put in grungy subsystem hacks
b) register virtual device nodes for subsystems when needed

b feels cleaner

I agree. I would personally see us using _more_ "virtual device node" things already: right now we have things like SuperIO chips that contain both a serial line and a parallel port (and...), and some drivers do really ugly things with them - keep them as one "struct pci_dev", and then have two drivers sharing the device.

It would be much cleaner to have _one_ driver for such SuperIO chips (a "multinode" driver), which just creates two virtual pci_dev structures, and lets the regular serial driver handle the "virtual serial device" etc.

That has the advantage of:
- not needing special hacks in various serial/parallel drivers
- the devices show up naturally and logically in whatever user mode "device m nager" tree

So the device nodes do not have to match the physical tree. The physical device tree only sets up the initial physical scanning, and obviously limits _reality_ ;)

Linus

#30 lore raw
In reply to Alan Cox
Quoted 6 lines
Quoted 2 lines

I really don't think it's _that_ difficult to properly do this blocking. For things like sound drivers, a simple semaphore is plenty enough. For

Sound is more easily handled by not blocking user space but waiting until the final IRQ off moment and grabbing the registers. That avoids a lot of ugly locking gunge. It literally comes down to

My point about using a semaphore was to avoid getting mixer ioctls banging the HW while it is shut down.

Ben.

#31 lore raw
Quoted 6 lines
Quoted 3 lines

Sound is more easily handled by not blocking user space but waiting until the final IRQ off moment and grabbing the registers. That avoids a lot of ugly locking gunge. It literally comes down to

My point about using a semaphore was to avoid getting mixer ioctls banging the HW while it is shut down.

Yes I can follow that - you want to avoid the aclink being shut down while active. That seems to be just part of the ordering. I'd also put the ac97 save/restore in the ac97_codec.c stuff - lets write it once 8)

#32 lore raw
Quoted 6 lines
Quoted 2 lines

My point about using a semaphore was to avoid getting mixer ioctls banging the HW while it is shut down.

Yes I can follow that - you want to avoid the aclink being shut down while active. That seems to be just part of the ordering. I'd also put the ac97 save/restore in the ac97_codec.c stuff - lets write it once 8)

Not exactly ;) Since sleep/resume on pmac is somewhat asynchronous, things like sound (which in my case can take 1 to 2 seconds to come back to to calibration delay of the chip) is done async. So userland is already running again, and may be hitting the driver with various mixer ioctls, while my HW isn't yet ready to get them (nothing fancy here).

I do also have the problem of having the sound chip on an i2c bus on some machiones, and so the problem of dependencies between the i2c controller and the sound driver (samples use a separate i2s bus), but this is also easily fixed by either having the sound chip a child of the i2c controller, or just shutting down the i2c controller as part of the platform code which is what I do now.

I did various experiments doing CD and/or MP3 playback and putting the machine to sleep. The "smoothest" result I obtained was using this semaphore on driver entrypoints. This "cleanly" suspends the sound app on it's next call to the sleeping sound driver and resume it when the sound driver is ready. Since the sound chip takes forever to come back, it's long enough for disk & cd to be fully back up, and the player not to skip when resumed :)

Ben.

#33 lore raw
In reply to Alan Cox
Quoted 7 lines

I don't think it is a big problem. We can add virtual nodes. They way I see it we either
a) put in grungy subsystem hacks
b) register virtual device nodes for subsystems when needed

b feels cleaner

Ok, provided that there is not one big "SCSI subsystem" virtual node, that would screw up the entire dep. hierarchy, but rather virtual nodes created by SCSI "clients" on the fly as childs of their devices. That looks fine to me ;)

An example would be:

* SCSI host
|
|
* SCSI disk device
|
|
* "sd" node

In this case, "sg" could add itself when opened, and eventually cause sleep requests to be rejected for example.

Well, in fact, I don't think there is real need for the "SCSI disk device" node, but that depends pretty much on the new SCSI architecture.

Ben.

#34 lore raw
Quoted 2 lines

In this case, "sg" could add itself when opened, and eventually cause sleep requests to be rejected for example.

Well, looks like Linus won't let this one pass ;) A /proc/sgbusy would eventually be ok, but I'd rather start defining a proper interface to the PM daemon in userland for apps to request that the machine doesn't go to sleep. That would be used, among other things, by CD burners & firmware updaters. No need to hack thousands of apps, I beleive if we get a patch implementing support for that in cdrecord, then all burners software will magically start getting it ;)

Ben.

#35 lore raw
In reply to Benjamin Herrenschmidt
Quoted 2 lines

In this case, "sg" could add itself when opened, and eventually cause sleep requests to be rejected for example.

I think SG is the only really special case one reading over the code. Disk and CD might want to issue a couple of things (cache flush, unlock media type stuff) but nothing tricky.

Quoted 2 lines

Well, in fact, I don't think there is real need for the "SCSI disk device" node, but that depends pretty much on the new SCSI architecture.

Sure

#36 lore raw
In reply to Linus Torvalds
Quoted 6 lines

Why would you _ever_ get "sg.c" and other crap involved in the suspend process?

The device tree is for _device_ suspend, not for "subsystem suspend". The SCSI subsystem is a piece of cr*p, but even if it was perfect it should never get involved with the act of suspension.

Well I don't want my laptop to suspend during a CD burn or firmware update. The device itself doesn't know anything about how busy it is since its just sending packets, only the subsystem driver controller it does

Quoted 3 lines

by getting involved with sg.c or anything similar, but by basically stopping all user apps (think of the equivalent of a "kill -STOP -1", but done internally in the kernel without actually using a signal).

Stopping all user apps really tends to ruin the cd and the firmware update.

Quoted 2 lines

Remember: the main point of suspend is to have a laptop go to sleep, and come back up on the order of a few _seconds_.

It also has to avoid unpleasant situations

Quoted 4 lines

Also, realize that the act of suspension is STARTED BY THE USER. Which means that before the kernel suspends, you _can_ have user programs that basically take disk arrays off-line etc if that is what you want. But that's not ae kernel suspend issue.

There are certain practicalities here with trying to make user space dig around in fuser innards or patching every cd burner. The sg layer is one that has to get involved (be it as a driver call back or a virtual driver)

#37 lore raw
Quoted 4 lines · Alan Cox

Well I don't want my laptop to suspend during a CD burn or firmware update. The device itself doesn't know anything about how busy it is since its just sending packets, only the subsystem driver controller it does

But that's _your_ problem. Not the kernels.

If you have a acpi deamon that decides to make the machine go to sleep while burning a CD, that's nothign to do with the kernel at all.

It has nothing to do with sg.c either, for that matter.

Quoted 4 lines
Quoted 2 lines

Remember: the main point of suspend is to have a laptop go to sleep, and come back up on the order of a few _seconds_.

It also has to avoid unpleasant situations

Absolutely NOT.

The kernel does not set policy. If the user says "suspend now", then we suspend now. Whether a CD burn or anything else is going on is totally irrelevant.

Quoted 3 lines

There are certain practicalities here with trying to make user space dig around in fuser innards or patching every cd burner. The sg layer is one that has to get involved (be it as a driver call back or a virtual driver)

Not a way in hell. If the sg layer wants to export a "/proc/sgbusy", that's its problem.

But if I say "suspend", and the kernel refuses, I will kill the offending piece of crap from sg.c before you can blink an eye.

Linus

#38 lore raw
Quoted 2 lines

If you have a acpi deamon that decides to make the machine go to sleep while burning a CD, that's nothign to do with the kernel at all.

One job kernel drivers have is to say "I can't safely sleep at this moment" Even windows/XP beta gets this right.

Quoted 3 lines

The kernel does not set policy. If the user says "suspend now", then we suspend now. Whether a CD burn or anything else is going on is totally irrelevant.

I know what the end user viewpoint on that would be. In a sense I do agree with you - but that would assume we could re-invent every single scsi generic driver, figure out how to make /proc/sg/%d/... work and the like

Quoted 2 lines

But if I say "suspend", and the kernel refuses, I will kill the offending piece of crap from sg.c before you can blink an eye.

Thats fine by me. Anyone wanting to be able to burn cds safely can run a -ac kernel tree

Alan

#39 lore raw
Quoted 5 lines · Alan Cox
Quoted 2 lines

If you have a acpi deamon that decides to make the machine go to sleep while burning a CD, that's nothign to do with the kernel at all.

One job kernel drivers have is to say "I can't safely sleep at this moment" Even windows/XP beta gets this right.

The other solution (let cdrecord tell I-dunno-how the PM daemon that it's doing something "important") is IMHO better: the PM daemon could judge if it should honor the suspend request depending on its priority (inactivity, power button or low battery) and the running "important" jobs.

Xav

#40 lore raw
Quoted 8 lines
Quoted 2 lines

One job kernel drivers have is to say "I can't safely sleep at this moment" Even windows/XP beta gets this right.

The other solution (let cdrecord tell I-dunno-how the PM daemon that it's doing something "important") is IMHO better: the PM daemon could judge if it should honor the suspend request depending on its priority (inactivity, power button or low battery) and the running "important" jobs.

The cdrecord case is a high level issue, and scsi is a mess ;)

We are not yet at a point where we can be more constructive than what was already said. Ultimately we need to move a bit forward with the real implementation and see how some problems show up. The architecture as it was designed so far is light, and most of the debate is not around it, it's around how it should be used by drivers & kernel subsystems ;)

Also, there will always be corner cases where a kernel driver will be able to override whatever policy was set. Drivers can register to the PM layer and can return errors. It have to be exceptional, but I beleive some critical flash algorithm or whatever would benefit from it.

I would however vote for making that a bit simpler by providing a kernel-global sleep rw semaphore. The idea here is that drivers, file systems, or whatever bits of kernel code that, for a given momment, won't afford beeing put to sleep will take a read lock on it. The sleep code will take a write lock. The read lockers won't make the PM code fail. It will just block a bit, waiting for the critical operations to complete.

I'm thinking about things like jffs2 on embedded. That (nice) flash file system has a background thread that does garbage collecting of your flash in order to be able to erase unused parts. It would benefit greatly embedded boards that implement system sleep if that thread could make sure sleep won't happen in the middle of moving blocks, or whatever atomicity it may want for its operations.

A firmware flashing facility in a driver may use that mecanism too.

It's a corner case, it's not meant to be used a lot, but it nice to have it.

Ben.

#41 lore raw
Quoted 1 line

The cdrecord case is a high level issue, and scsi is a mess ;)

Grin

Quoted 6 lines

We are not yet at a point where we can be more constructive than what was already said. Ultimately we need to move a bit forward with the real implementation and see how some problems show up. The architecture as it was designed so far is light, and most of the debate is not around it, it's around how it should be used by drivers & kernel subsystems ;)

I think I understand how to handle this and avoid races. Linus idea of /proc files so you can ask "what is busy" solves most of it. Then the policy daemon can make a choice about suspending or not.

If we make the proc file a large bitmask of events then the policy daemon call to kernel becomes

"suspend even if [event-mask] set"

This means that a daemon call that races the start of say a CD burn will fail and the daemon can rescan, rethink and if need be reissue the request sanely.

#42 lore raw
In reply to Alan Cox
Quoted 6 lines · Alan Cox
Quoted 3 lines

But if I say "suspend", and the kernel refuses, I will kill the offending piece of crap from sg.c before you can blink an eye.

Thats fine by me. Anyone wanting to be able to burn cds safely can run a -ac kernel tree

Telling the kernel to suspend while burning a CD is on the same level as ejecting the CD while burning. It has to go wrong. Someone explicitly asking for trouble might as well get it.

The really dumb users is probably using a GUI tool for either activity, that one may of course refuse to ruin the burn.

Helge Hafting

#43 lore raw
Quoted 4 lines

Telling the kernel to suspend while burning a CD is on the same level as ejecting the CD while burning. It has to go wrong. Someone explicitly asking for trouble might as well get it.

It need not be someone asking for trouble. It might just be a ten minute "nothing happened" timeout that starts the decision making.

Quoted 3 lines

The really dumb users is probably using a GUI tool for either activity, that one may of course refuse to ruin the burn.

The current GUI tools don't know anything about 2.5 power management, and in some cases don't know when the driver has done needed work or not.

Alan

#44 lore raw
In reply to Linus Torvalds

Hi!

Quoted 7 lines
Quoted 6 lines

We should not have pending IO, but that's for a totally different reason:
the first thing the much much MUCH higher levels of suspend should be doing is to make sure that user apps are "quiescent". And that isn't done by getting involved with sg.c or anything similar, but by basically stopping all user apps (think of the equivalent of a "kill -STOP -1", but done internally in the kernel without actually using a signal).

Quoted 8 lines

Is this necessary ? That would definitely make things easier to implement to forget about incoming requests, but I'm not sure it's the right way. In fact, is it really working ? You could well have in-kernel threads triggering IOs or launching new userland stuffs (what happens if a not yet suspended driver for, let's say USB, see a new device coming in and starts /sbin/hotplug). Some filesystems have garbage collector threads that can do IOs eventually. There are various kind of IOs that can in fact be triggered entirely within the kernel.

I need this for suspend to disk. I really need quiescent system if I want to write to swap image, right?

I implemented a "refrigerator" mechanism, and manually marked threads that should not be frozen. I'm doing my best to stop everyone who might try to wake userland.

[I just mailed the patch as 2.4.13 - swsusp. Tell me if you want a copy.]

Pavel

Signature · 2 lines
STOP THE WAR! Someone killed innocent Americans. That does not give
U.S. right to kill people in Afganistan.
#45 lore raw
In reply to Benjamin Herrenschmidt

"Linus Torvalds" <[email protected]> wrote in message news:cistron.Pine.LNX.4.33.0110240901350.8049-100000@penguin.transmeta.com.. .

Quoted 33 lines · Benjamin Herrenschmidt
Quoted 11 lines
Quoted 5 lines

So the scsi devices hang off sd, sr etc which in turn hang off scsi and the controllers hang off scsi (and or the bus layers)

This one at least I think I do understand

The problem with subsystems is that they don't fit well in the power tree. They aren't "devices" in that sense that they are not exposing a struct device, and they spawn over several controllers which means the dependency can quickly become unmanageable, especially when SCSI starts beeing layered on top of USB or FireWire.

Why would you _ever_ get "sg.c" and other crap involved in the suspend process?

The device tree is for _device_ suspend, not for "subsystem suspend". The SCSI subsystem is a piece of cr*p, but even if it was perfect it should never get involved with the act of suspension.

We should not have pending IO, but that's for a totally different reason:
the first thing the much much MUCH higher levels of suspend should be doing is to make sure that user apps are "quiescent". And that isn't done by getting involved with sg.c or anything similar, but by basically stopping all user apps (think of the equivalent of a "kill -STOP -1", but done internally in the kernel without actually using a signal).

Quoted 4 lines

Also, the dependency issue is made worst if you let RAID enter into the dance as I beleive ultimately, nothing would prevent a volume to spawn over several devices from different controllers or even different controller types.

Quoted 7 lines

Why would you get RAID involved? There is no _IO_ involved in suspending:
we just stop doing what we're doing, and leave it at that. We don't try to flush state, we just freeze the machine.

The act of "suspend" should basically be: shut off the SCSI controller, screw all devices, reset the bus on resume.

Doing so will create havoc on sequential devices, such as tape drives. If your system simply suspends, then all is well. Any data that isn't flushed yet is buffered inside the tapedrive. But when the system resumes and resets the SCSI bus, it will cause all data in the tape drive to be lost, and for most tape systems it will also re-position them at LBOT. Any running tar/dump/whatever tape process would not survive such a suspend-resume cycle.

Another more subtle issue is state information that exists between the SCSI controller and the target devices. At some point they might have negotiated synchronous and/or wide transfer parameters. This information must be preserved, or you'll observe lockups, data corruption and the likes. Since these parameters are maintained at the lowest driver level, they should know about suspend. The low-level driver must know to re-negotiate these parameters when it comes back to life.

Rob

#46 lore raw
Quoted 9 lines

Doing so will create havoc on sequential devices, such as tape drives. If your system simply suspends, then all is well. Any data that isn't flushed yet is buffered inside the tapedrive. But when the system resumes and resets the SCSI bus, it will cause all data in the tape drive to be lost, and for most tape systems it will also re-position them at LBOT. Any running tar/dump/whatever tape process would not survive such a suspend-resume cycle.

Quoted 7 lines

Another more subtle issue is state information that exists between the SCSI controller and the target devices. At some point they might have negotiated synchronous and/or wide transfer parameters. This information must be preserved, or you'll observe lockups, data corruption and the likes. Since these parameters are maintained at the lowest driver level, they should know about suspend. The low-level driver must know to re-negotiate these parameters when it comes back to life.

This can be handled by having st (or sd, or whatever "client driver" decide to take over a SCSI device) register a struct device node that is a child of the actual SCSI device.

In fact, I'm wondering if we need a struct device node at all for the SCSI device on the bus. The SCSI controller (or USB/storage or FireWire SBP2) will expose SCSI devices, that is "interface" to which you can feed SCSI requests, but do those really need to have a structure device associated ? One possibility would be to only do so once attached to a "client" driver like st, sd, sg, ... The "client" would then create that structure.

But...

If we still want "unclaimed" devices to have a representation in the device tree (because, for example, userland wants to know about them, eventually in order to "instanciate" an sg driver), then we could have the SCSI subsystem create a simple skeletton struct device when the devices are probed, and have the client driver just populate this with more infos & PM hooks once attached to the device.
I don't think there's a need to have 2 struct device stacked.

But it's mostly a matter of taste ;)

Thinking more about it, I think I prefer the second solution. That is to have SCSI create "standard" struct device for all devices probed on the bus, thus ensuring they are visible from the userland representation of the device-tree, and then eventually have drivers like sd, st, etc... add entries & PM hooks to those devices if needed when attached to them. But doing that, or just having them create a virtual node as a child of the device is mostly a matter of taste, I beleive.

Ben.

#47 lore raw
In reply to Rob Turk
Quoted 5 lines · Rob Turk
Quoted 3 lines

The act of "suspend" should basically be: shut off the SCSI controller, screw all devices, reset the bus on resume.

Doing so will create havoc on sequential devices, such as tape drives. If

I'm failing to imagine a good case for suspending a system that has a tape drive on it.

#48 lore raw
Quoted 10 lines · Victor Yodaiken
Quoted 5 lines · Rob Turk
Quoted 3 lines

The act of "suspend" should basically be: shut off the SCSI controller, screw all devices, reset the bus on resume.

Doing so will create havoc on sequential devices, such as tape drives. If

I'm failing to imagine a good case for suspending a system that has a tape drive on it.

I've often seen user workstations with tape driver.

I fail to see the need to suspend such a system while using the tape drive, though :)

Signature · 3 lines
Jeff Garzik      | Only so many songs can be sung
Building 1024    | with two lips, two lungs, and one tongue.
MandrakeSoft     |         - nomeansno
#49 lore raw
In reply to Victor Yodaiken
Quoted 10 lines · Victor Yodaiken
Quoted 5 lines · Rob Turk
Quoted 3 lines

The act of "suspend" should basically be: shut off the SCSI controller, screw all devices, reset the bus on resume.

Doing so will create havoc on sequential devices, such as tape drives. If

I'm failing to imagine a good case for suspending a system that has a tape drive on it.

I've often seen user workstations with tape drives. Very uncommon these days, agreed.

I fail to see the need to suspend such a system while using the tape drive, though :)

Signature · 3 lines
Jeff Garzik      | Only so many songs can be sung
Building 1024    | with two lips, two lungs, and one tongue.
MandrakeSoft     |         - nomeansno
#50 lore raw
In reply to Victor Yodaiken

let alone a reason for a suspend to be triggered while running the tape.

David Lang

Quoted 25 lines · Victor Yodaiken

Date: Thu, 25 Oct 2001 08:20:01 -0600
From: Victor Yodaiken <[email protected]>
To: Rob Turk <[email protected]>
Cc: [email protected]
Subject: Re: [RFC] New Driver Model for 2.5

Quoted 5 lines · Rob Turk
Quoted 3 lines

The act of "suspend" should basically be: shut off the SCSI controller, screw all devices, reset the bus on resume.

Doing so will create havoc on sequential devices, such as tape drives. If

I'm failing to imagine a good case for suspending a system that has a tape drive on it.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#51 lore raw
In reply to Rob Turk

"Victor Yodaiken" <[email protected]> wrote in message news:cistron.20011025082001.B764@hq2...

Quoted 10 lines · Rob Turk
Quoted 5 lines
Quoted 3 lines

The act of "suspend" should basically be: shut off the SCSI controller, screw all devices, reset the bus on resume.

Doing so will create havoc on sequential devices, such as tape drives. If

I'm failing to imagine a good case for suspending a system that has a tape drive on it.

Well, maybe the tape example wasn't all that good. The state information (wide/sync negotiation) still needs to be retained for all SCSI devices though.

Rob

#52 lore raw
Quoted 6 lines · Rob Turk
Quoted 3 lines

I'm failing to imagine a good case for suspending a system that has a
tape drive on it.

Well, maybe the tape example wasn't all that good. The state information (wide/sync negotiation) still needs to be retained for all SCSI devices though.

Any driver that uses SCSI bus reset for last-resort error recovery (and I think it's pretty typical) needs to be able to renegotiate the connection. Maybe even after a SCSI device reset; I don't recall. So initiating that negotiation as part of (or after) resume doesn't seem all that burdensome.

You need that anyway for "deep sleep" that powers down devices completely.

Signature · 1 line
/Jonathan Lundell.
#53 lore raw
In reply to Benjamin Herrenschmidt
Quoted 8 lines · Rob Turk

Doing so will create havoc on sequential devices, such as tape drives. If your system simply suspends, then all is well. Any data that isn't flushed yet is buffered inside the tapedrive. But when the system resumes and resets the SCSI bus, it will cause all data in the tape drive to be lost, and for most tape systems it will also re-position them at LBOT. Any running tar/dump/whatever tape process would not survive such a suspend-resume cycle.

Well, why reset the scsi bus on resume then?
That seems unnecessary. At suspend time the devices simply don't get more requests. (Except perhaps spin-down requests for disks.) Then nothing much happens. Eventually the system wakes up, and requests appear again. First spin-up requests, then ordinary io.

Quite a few scsi bioses have an option for not resetting the bus when booting. Less delay, and necessary for those few with a shared scsi bus. Seems a reset won't be necessary for suspend/resume either, which is supposed to be a lighter operation than a reboot.

If your scsi adapter don't support this - it isn't suspend/resume compatible the way I see it.

Helge Hafting

#54 lore raw
In reply to Benjamin Herrenschmidt

Hi!

Quoted 11 lines
Quoted 3 lines

The act of "suspend" should basically be: shut off the SCSI controller, screw all devices, reset the bus on resume.

Doing so will create havoc on sequential devices, such as tape drives. If your system simply suspends, then all is well. Any data that isn't flushed yet is buffered inside the tapedrive. But when the system resumes and resets the SCSI bus, it will cause all data in the tape drive to be lost, and for most tape systems it will also re-position them at LBOT. Any running tar/dump/whatever tape process would not survive such a suspend-resume cycle.

Then there's something wrong with st.

Imagine EMI comes and SCSI gets reset. That should not mean tar failing, right? So you have st broken in first place.
Pavel

Signature · 2 lines
STOP THE WAR! Someone killed innocent Americans. That does not give
U.S. right to kill people in Afganistan.
#55 lore raw
In reply to Benjamin Herrenschmidt

"Pavel Machek" <[email protected]> wrote in message
news:[email protected]...

Quoted 14 lines

Hi!

Quoted 11 lines
Quoted 3 lines

The act of "suspend" should basically be: shut off the SCSI controller, screw all devices, reset the bus on resume.

Doing so will create havoc on sequential devices, such as tape drives. If your system simply suspends, then all is well. Any data that isn't flushed yet is buffered inside the tapedrive. But when the system resumes and resets the SCSI bus, it will cause all data in the tape drive to be lost, and for most tape systems it will also re-position them at LBOT. Any running tar/dump/whatever tape process would not survive such a suspend-resume cycle.

Quoted 6 lines

Then there's something wrong with st.

Imagine EMI comes and SCSI gets reset. That should not mean tar failing, right? So you have st broken in first place. Pavel
--

No, it's an inherant part of the SCSI spec. A SCSI bus reset will cause many, if not all tape devices to rewind to begin of tape. The st driver can detect this (A SCSI Unit Attention will be returned on the next SCSI command), and try to re-position the tape to it's previous location. Doing so is not easy, and on many tape drives even impractical. On an almost full tape, a DLT drive would take up to two hours to get back to where it last was. Too much for most time-out mechanisms...

On a SCSI bus reset, tape related processes are better off passing the condition upward to user space (tar, dump, whatever). Intelligent user space programs may be able to recover, the dumb ones will fail.

By the way, EMI is very unlikely to cause a SCSI bus reset. It may cause repeated parity errorsto the point that a host or devices decides to reset the bus. If there's this much EMI, then you should use a better transport (HVD SCSI, or fibre). But this part of the discussion is probably better helt at comp.periphs.scsi 8-)

Rob

#56 lore raw
In reply to Alan Cox
Quoted 3 lines · Benjamin Herrenschmidt [[email protected]] wrote:

Now, I'm not sure what would happen with RAID. If we need to have logical volumes be child of the sd "client", then we have to face the fact that a given child may have multiple parents... welcome to the power graph !

You do not have to add RAID to need to worry about multiple parents. If we want to correctly represent devices that have multiple paths (i.e twin tailed SCSI, fibre channel, multi-ported devices, etc) we should have a solution to handle this. Some O/S's have moved to directed graphs to address the multiple parent issue. Exposing only one block / character device per real physical device would reduce O/S resources (major / minors, structs) and provide a single request queue.

The current model of a scsi_device having a single parent and being attached to the scsi_host host_queue has made adding multi-path support to Linux below the SCSI lower level driver difficult.

Quoted 9 lines

But do we really need logical volumes to be part of the PM tree or can blocking of requests at the sd layer be enough ? Remember we are in pass2, we have already done memory allocation, we are supposed to no longer swap nor do any disk/storage related activity.

A tricky issue indeed...

Ben.

-Mike

Signature · 2 lines
Michael Anderson
[email protected]
#57 lore raw
In reply to Alan Cox
Quoted 22 lines · Benjamin Herrenschmidt
Quoted 11 lines
Quoted 3 lines

case, there's not much left to the controller, it isn't supposed to have any command in queue nor receive any new one once all it's child drivers have suspended.

scsi devices are children of the scsi subststem (sd, sg, sr, st, osst) not of the controller. That is how the state flows anyway. Only sr/sd etc know what the state is for a given device on power off as they may issue multiple requests per action true transaction. sg would have to simply refuse any suspend if open (think about cd-burning or even worse firmware download)

Quoted 4 lines

So the scsi devices hang off sd, sr etc which in turn hang off scsi and the controllers hang off scsi (and or the bus layers)

This one at least I think I do understand

The problem with subsystems is that they don't fit well in the power tree. They aren't "devices" in that sense that they are not exposing a struct device, and they spawn over several controllers which means the dependency can quickly become unmanageable, especially when SCSI starts beeing layered on top of USB or FireWire.

Quoted 4 lines

Also, the dependency issue is made worst if you let RAID enter into the dance as I beleive ultimately, nothing would prevent a volume to spawn over several devices from different controllers or even different controller types.

On the dependency case for x86 I have a fun common example.
To shut off the cpu, or the whole motherboard I need to talk to the southbridge. To talk to the southbridge, I need to talk to the northbridge.

So at least to some extent shutting down busses is a really different case from shutting down devices. And only in some cases can a tree model it at all.

Equally fun are temperature monitors that appear on both the lpc/isa bus and the i2c bus.

Or another fun common one. To shut down the interrupt controller, I first need to shut down every device that thinks it can generate interrupts. But my interrupt controller is way out on my pci->isa bridge. So I can't shut that device down.

Sorry this whole device tree idea for shutdown ordering doesn't seem to match my idea of reality.

Now I need to take a little time out and see what the code that is being discussed will actually do about situations like the above.

Quoted 1 line

A tricky issue indeed...

Agreed.

Eric

#58 lore raw
Quoted 8 lines · Eric W. Biederman

Or another fun common one. To shut down the interrupt controller, I first need to shut down every device that thinks it can generate interrupts. But my interrupt controller is way out on my pci->isa bridge. So I can't shut that device down.

Sorry this whole device tree idea for shutdown ordering doesn't seem to match my idea of reality.

Your _examples_ do not match any reality.

Don't worry about things like the CPU shutdown: you have to have special code for it anyway.

Let's face it, the device tree is for _devices_. It's for shutting down a network card before we shut down the PCI bridge that is in front of it.

The issue of "core shutdown" is not covered - and isn't _meant_ to be covered. That's the problem of the architecture-specific code. There is no point in having a device tree for that, because it's going to be very much architecture-specific anyway (ie on x86 we may have to just blindly trust some silly APCI table data etc).

Linus

#59 lore raw
Quoted 13 lines
Quoted 7 lines

Or another fun common one. To shut down the interrupt controller, I first need to shut down every device that thinks it can generate interrupts. But my interrupt controller is way out on my pci->isa bridge. So I can't shut that device down.

Sorry this whole device tree idea for shutdown ordering doesn't seem to match my idea of reality.

Your _examples_ do not match any reality.

Don't worry about things like the CPU shutdown: you have to have special code for it anyway.

Quoted 8 lines

Let's face it, the device tree is for _devices_. It's for shutting down a network card before we shut down the PCI bridge that is in front of it.

The issue of "core shutdown" is not covered - and isn't _meant_ to be covered. That's the problem of the architecture-specific code. There is no point in having a device tree for that, because it's going to be very much architecture-specific anyway (ie on x86 we may have to just blindly trust some silly APCI table data etc).

Definitelly. I have similar issues with pmacs, clocks generation and interrupt controller are in Apple's mac-io ASIC which is on PCI, so this ASIC can't be part of the normal PM tree and has to be handled as part of the "core" PM code. This kind of issue will still happen, the new scheme won't "magically" make PM work on every single laptop out there, there will still be some corner cases to deal with, but at least these will be limited to real corner cases and most "normal" drivers will fit in the new, saner, mecanism.

Ben.

#60 lore raw
In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 8 lines · Eric W. Biederman

Or another fun common one. To shut down the interrupt controller, I first need to shut down every device that thinks it can generate interrupts. But my interrupt controller is way out on my pci->isa bridge. So I can't shut that device down.

Sorry this whole device tree idea for shutdown ordering doesn't seem to match my idea of reality.

Your _examples_ do not match any reality.

I'll go as far as agreeing they do not matching any _practical_ reality.

Quoted 2 lines

Don't worry about things like the CPU shutdown: you have to have special code for it anyway.

But that is the case I plan on coding....

Quoted 5 lines

Let's face it, the device tree is for _devices_. It's for shutting down a network card before we shut down the PCI bridge that is in front of it.

The issue of "core shutdown" is not covered - and isn't _meant_ to be covered.

O.k. I'll step back and let you guys handle the normal cases. I rarely get past "core startup" and "core shutdown".

Quoted 4 lines

That's the problem of the architecture-specific code. There is no point in having a device tree for that, because it's going to be very much architecture-specific anyway (ie on x86 we may have to just blindly trust some silly APCI table data etc).

I'm doing my best to provide a real world alternative to ACPI on some boards.

My perspective is coming from linuxBIOS, or in general GPL'd firmware, so it is a little different.

But at this point in the conversation it looks like I should just back off, let the core API get the easy cases correct. And then come back and figure out how to handle the truly weird cases.

Eric

#61 lore raw
Quoted 8 lines · Eric W. Biederman
Quoted 5 lines

That's the problem of the architecture-specific code. There is no point in having a device tree for that, because it's going to be very much architecture-specific anyway (ie on x86 we may have to just blindly trust some silly APCI table data etc).

I'm doing my best to provide a real world alternative to ACPI on some boards.

That will be much appreciated. ACPI is not all that wonderful to say the least. With enough knowledge of the hardware you can mostly do a better job (the problem is "enough knowledge", especially on most laptops where most of the GPIO signals etc are pretty much ad-hoc and not defined by the chipsets but by the board layout person.. And we can't query the board revision even if they gave us the information ;)

Quoted 2 lines

My perspective is coming from linuxBIOS, or in general GPL'd firmware, so it is a little different.

Understood.

Linus

#62 lore raw
In reply to Alan Cox
Quoted 19 lines · Alan Cox
Quoted 5 lines

the HW or not when getting a new request). In cases where a mid-layer
enters the scene, like SCSI, that wants to do timeouts, then well... we can let it timeout (just stop processing requests), or we can have the midlayer go to sleep as well :) That later solution may cause some interesting ordering issues however...

For scsi you have to complete the pending commands, you don't know what the transaction granularity is in some cases and half completing the sequence won't help you. In addition the upper layers have to queue additional scsi commands to do stuff like cd drawer locking and to ask the drive firmware to enter powerdown modes

Quoted 6 lines

For USB, for example, we can consider that when a device driver (not a controller driver) suspend has been done, any URB it submits can just be dropped (returned immediately with an error). We don't need blocking here neither. Of course, that means we have the framework to call devices' suspend/resume callbacks when the
controller is about to go to sleep.

Quoted 2 lines

That will scramble large numbers of devices. Randomly erroring pending block writes is -not- civilised.

In our "extreme prejudice" suspend (this is in the context of masking & recovering from a fault in a fault-tolerant machine) we have cases in which completion of pending commands isn't possible. Our solution is to issue a SCSI bus reset, and terminate all outstanding commands with an appropriate (retryable) error. This is especially easy to implement in drivers that use SCSI bus reset as a routine (though last resort) error recovery mechanism, since the requisite logic is already in place. Not pretty, I suppose, but effective.

One model we've considered (but haven't implemented yet) is to make parents in the device tree responsible for suspending their children, so the suspend propagates down the tree and each node "knows" how to suspend its children, assuming any special action is required. So a SCSI HBA, for example, would be asked by its bus parent to suspend, and in turn would suspend its SCSI device children before suspending itself. I'm not quite sure how virtual device layers like md would fit into this scheme, since they can cut across device and power hierarchies.

Quoted 2 lines · Alan Cox

So the scsi devices hang off sd, sr etc which in turn hang off scsi and the controllers hang off scsi (and or the bus layers)

Our first implementation was under Solaris 2.x (SPARC) in which the parent->child relationship is bus->hba->sd. scsi isn't in the tree; it's more of an interface layer between hba & sd. fwiw.

Signature · 1 line
/Jonathan Lundell.
#63 lore raw
In reply to Alan Cox
Quoted 3 lines · Alan Cox

That will scramble large numbers of devices. Randomly erroring pending block writes is -not- civilised.

Note that one thing in suspending the machine that has _nothing_ to do with the actual device tree is that higher layers have to suspend whatever it is they are doing anyway.

Ie part of the suspend action (which is unrelated to the driver model) is to stop all regularly scheduled activity - not necessarily flushing all dirty buffers, but certainly waiting for all pending IO. That's a much higher level thing that the device though - the devices themselves should never ever see this (except in the sense that they don't see new requests coming in).

There are other "higher-level" issues: while a device "prepare to suspend" call might block for some device information, that does not mean that it can allocate memory with GFP_KERNEL, for example: when we shut off device X, the disk may have been prepared for shutdown already, and the VM layer cannot do any IO. So the suspend (and resume) function have to use GFP_NOIO for their allocations - _regardless_ of any other device issues.

So sure, there are tons of issues here, but none of them have, in my opinion, anything to do with the device model itself. More just normal implementation details.

Linus

#64 lore raw
Quoted 5 lines

call might block for some device information, that does not mean that it can allocate memory with GFP_KERNEL, for example: when we shut off device X, the disk may have been prepared for shutdown already, and the VM layer cannot do any IO. So the suspend (and resume) function have to use GFP_NOIO for their allocations - _regardless_ of any other device issues.

So I have to write a whole extra set of code paths to duplicate normal functionality during power off

Quoted 3 lines

So sure, there are tons of issues here, but none of them have, in my opinion, anything to do with the device model itself. More just normal implementation details.

My concern is that we need to make the implementation details simple. eg so that simple things like "save state" can be done before we get into "no this, no that , no the other" situations. Also so that for the many drivers where freezing the system once we have irqs off is easier (a lot of sound for example is easiest done by disable irq, disable dma engine, copy registers, return) can be done late and with small amounts of code

#65 lore raw
Quoted 9 lines · Alan Cox
Quoted 6 lines

call might block for some device information, that does not mean that it can allocate memory with GFP_KERNEL, for example: when we shut off device X, the disk may have been prepared for shutdown already, and the VM layer cannot do any IO. So the suspend (and resume) function have to use GFP_NOIO for their allocations - _regardless_ of any other device issues.

So I have to write a whole extra set of code paths to duplicate normal functionality during power off

If that ends up being a problem, we can just make alloc_pages turn off the IO bits on suspend. Easy enough..

Although I think you're making the problem bigger than it is. Most of the suspend stuff should not need any "normal functionality" at all.

Linus

#66 lore raw
In reply to Alan Cox

Hi!

Quoted 11 lines
Quoted 6 lines
Quoted 2 lines
	/* Now tell them to stop I/O and save their state */
	device_suspend(3, SUSPEND_SAVE_STATE);

I'd very much like this one to be two pass, with the second pass occuring after interrupts are disabled. There are some horrible cases to try and handle otherwise (like devices that like to jam the irq line high).

I forgot to mention to disable interrupts after the SUSPEND_NOTIFY call. The idea is to allocate all memory in the first pass, disable interrupts, then save state. Would that work? Or, should some of the state saving take place with interrupts enabled?

That looks ugly, because you'd need to add DONT_SUSPEND_NOTIFY, called when SUSPEND_NOTIFY fails.
Pavel

Signature · 2 lines
Casualities in World Trade Center: 6453 dead inside the building,
cryptography in U.S.A. and free speech in Czech Republic.
#67 lore raw
Quoted 8 lines
Quoted 4 lines

I forgot to mention to disable interrupts after the SUSPEND_NOTIFY call. The idea is to allocate all memory in the first pass, disable interrupts, then save state. Would that work? Or, should some of the state saving take place with interrupts enabled?

That looks ugly, because you'd need to add DONT_SUSPEND_NOTIFY, called when SUSPEND_NOTIFY fails.
Pavel

No, interrupts have to be shut down between SUSPEND_SAVE_STATE and SUSPEND_POWER_DOWN, I beleive.

SUSPEND_SAVE_STATE must run with interrupts enabled, as it's supposed to both block new incoming IOs and wait for pending ones to complete (*). It would be sub-efficient to force drivers to implement polled IOs for this case.

SUSPEND_POWER_DOWN itself should perfectly be able to run with interrupts disabled, I beleive, as must of the actual suspend sequence can be done in SUSPEND_SAVE_STATE on most chips.

There is no problem with failure there. Just call RESUME_POWER_ON if SUSPEND_POWER_DOWN failed (or later), then RESUME_RESTORE_STATE in all cases. The driver knows from which state it comes from anyway, and we don't have, I beleive, that strick VM need of separating suspend from free's. Well... let's think more about it... we might actually need to allocate memory in RESUME_RESTORE_STATE to create new requests or whatever the driver need... but we can also do that earlier, inside SUSPEND_NOTIFY (or just use whatever memory we pre-allocated to save state). So it might make sense to have the resume process be an exact mirror of the wakeup one, or not, maybe just a matter of taste.

In most cases, keep in mind that most drivers won't need to implement all of these.

Ben.

#68 lore raw
Quoted 4 lines

SUSPEND_SAVE_STATE must run with interrupts enabled, as it's supposed to both block new incoming IOs and wait for pending ones to complete (*). It would be sub-efficient to force drivers to implement polled IOs for this case.

I forgot...

(*) Did you decide if you allowed that "call me again later" result code from SUSPEND_SAVE_STATE ? If yes, that would mean you must loop notifying all drivers that have not ack'ed it until they all do before going to SUSPEND_POWER_DOWN. It's probably not much bloat to let the feature in, as usual, it doesn't have to be used by drivers, but for those drivers who knows it will take some time for pending async requests to complete, it makes sense to let others perform they job. It would slightly speed up the suspend process, which is not critical, but still nice ;)

Ben.

#69 lore raw
In reply to Alan Cox
Quoted 7 lines

I'd very much like this one to be two pass, with the second pass occuring after interrupts are disabled. There are some horrible cases to try and handle otherwise (like devices that like to jam the irq line high).

Ditto on return from suspend where some devices also like to float the irq high as you take them over (eg USB on my Palmax). From comments Ben made ages back I believe ppc has similar issues if not worse

Well, the idea here was to disable them between the second and third pass. Device that can completely suspend with interrupts enabled can do it at the end of step 2, while more broken devices can do it at step 3. It might be semantically more clear to actually consider step 2 as exclusively "block io & save state", in which case, breaking up the "suspend" state into 2 separate states with and without interrupt makes sense. We just didn't fell that was necessary.

Ben.