Shutting down PCI devices on suspend

The default PCI suspend/resume calls (used if the driver doesn't implement them itself) don't alter the device power state. This seems to cause problems on some Thinkpads - going into S3 doesn't seem to result in the on-board Radeon being put into D3, and as a result the battery is drained at a higher rate than it should be. Who should be responsible for making sure that devices are properly put to sleep:

1) The drivers (not immensely helpful in the case where we don't necessarily have drivers registered for every device)
2) The kernel
3) The hardware itself?

I seem to remember someone mentioning something about Linux not checking for hints from the hardware as to what state devices should be put in, but I know little about this.

In FreeBSD, we do this at the bus layer. So the ACPI and PCI bus code is responsible for powering down/up children. We currently do it:

1. When no driver is attached and on reprobe
2. When suspending/resuming, based on evaluating _SxD

See this commit for details:
http://docs.freebsd.org/cgi/getmsg.cgi?fetch=1604853+0+archive/2004/cvs-all/20041205.cvs-all

Quoted 5 lines · Nate Lawson

In FreeBSD, we do this at the bus layer. So the ACPI and PCI bus code is responsible for powering down/up children. We currently do it:

1. When no driver is attached and on reprobe
2. When suspending/resuming, based on evaluating _SxD

Quoted 2 lines

See this commit for details:
http://docs.freebsd.org/cgi/getmsg.cgi?fetch=1604853+0+archive/2004/cvs-all/20041205.cvs-all

Does _SxD represent the "desired" mode for the hardware, or merely the highest state it can be left in? A mail from Len earlier this year suggested that if S3D returned 0x2, that meant that either D3 or D2 were acceptable.

Is there any reason why adding something like

if (state==3) {
pci_set_power_state(dev,ACPI_STATE_D3COLD)
}

to pci_device_suspend in the non-driver path and reenabling it in pci_default_resume would result in things breaking?

(Possibly with sanity checks as to whether it's a bridge or not)

Quoted 13 lines · Matthew Garrett

Quoted 8 lines · Nate Lawson

In FreeBSD, we do this at the bus layer. So the ACPI and PCI bus code is responsible for powering down/up children. We currently do it:

1. When no driver is attached and on reprobe
2. When suspending/resuming, based on evaluating _SxD

See this commit for details:
http://docs.freebsd.org/cgi/getmsg.cgi?fetch=1604853+0+archive/2004/cvs-all/20041205.cvs-all

Quoted 4 lines

Does _SxD represent the "desired" mode for the hardware, or merely the highest state it can be left in? A mail from Len earlier this year suggested that if S3D returned 0x2, that meant that either D3 or D2 were acceptable.

_SxD is the highest state supported by the device for a given sleep state, yes. We put devices in D3 unless _SxD specifies something higher. The reason to do this is some devices can't wake the system if they are in D3. ACPI 3.0 has ways to indicate a desired power state (not just maximum). I don't have any evidence but fear some devices won't work after resuming if set to D3 and they indicate this with an _SxD so we always use the value in _SxD if present.

Quoted 6 lines

Is there any reason why adding something like

if (state==3) {
pci_set_power_state(dev,ACPI_STATE_D3COLD)
}

Quoted 4 lines

to pci_device_suspend in the non-driver path and reenabling it in pci_default_resume would result in things breaking?

(Possibly with sanity checks as to whether it's a bridge or not)

Not that I can think of. You probably want state == S3 || S4 since it's fine to power down devices for S4. You'll probably find drivers that don't properly resume their device. You have to make sure you power devices down after suspending them and power them up before resuming them.

The default PCI suspend/resume calls (used if the driver doesn't implement them itself) don't alter the device power state. This seems to cause problems on some Thinkpads - going into S3 doesn't seem to result in the on-board Radeon being put into D3, and as a result the battery is drained at a higher rate than it should be. Who should be responsible for making sure that devices are properly put to sleep:

1) The drivers (not immensely helpful in the case where we don't necessarily have drivers registered for every device)
2) The kernel
3) The hardware itself?

I seem to remember someone mentioning something about Linux not checking for hints from the hardware as to what state devices should be put in, but I know little about this.

The default PCI suspend/resume calls (used if the driver doesn't implement them itself) don't alter the device power state. This seems to cause problems on some Thinkpads - going into S3 doesn't seem to result in the on-board Radeon being put into D3, and as a result the battery is drained at a higher rate than it should be. Who should be responsible for making sure that devices are properly put to sleep:

1) The drivers (not immensely helpful in the case where we don't necessarily have drivers registered for every device)

2) The kernel
3) The hardware itself?

I seem to remember someone mentioning something about Linux not checking for hints from the hardware as to what state devices should be put in, but I know little about this.

Please do it in the drivers. Anything else is likely to break currently working setups.

Quoted 2 lines

Please do it in the drivers. Anything else is likely to break currently working setups.

This requires a driver for every PCI device on the system. Is this the best solution?

Hi!

Quoted 2 lines

Quoted 2 lines

Please do it in the drivers. Anything else is likely to break currently working setups.

Quoted 3 lines

This requires a driver for every PCI device on the system. Is this the best solution?

Each PCI device is either:

* supported -- it has a driver, we are okay

* unsupported -- it should not be even turned on, right?

* legacy, like VGA or IDE -- okay, we need drivers for those. You need to reinit them on resume, anyway, so....

...yes, I think this is the best solution.
Pavel

Quoted 9 lines

Each PCI device is either:

* supported -- it has a driver, we are okay

* unsupported -- it should not be even turned on, right?

* legacy, like VGA or IDE -- okay, we need drivers for those. You need to reinit them on resume, anyway, so....

Quoted 1 line

...yes, I think this is the best solution.

Not necessarily. Most unsupported devices start in the on state. Also I think it's difficult to see a consistent power management behavior/policy if each driver is controlling it's devices' power state. I'd prefer to have drivers saving and restoring device state and then having the bus driver take care of the actual power state transitions.

Hi!

Quoted 11 lines

Quoted 10 lines

Each PCI device is either:

* supported -- it has a driver, we are okay

* unsupported -- it should not be even turned on, right?

* legacy, like VGA or IDE -- okay, we need drivers for those. You need to reinit them on resume, anyway, so....

...yes, I think this is the best solution.

Quoted 5 lines

Not necessarily. Most unsupported devices start in the on state. Also I think it's difficult to see a consistent power management behavior/policy if each driver is controlling it's devices' power state. I'd prefer to have drivers saving and restoring device state and then having the bus driver take care of the actual power state transitions.

Why?

Saving/restoring state is way more work than actual power state transitions. Provide nice helpers so that power state transitions are easy, but leave the work on the drivers, so that drivers may do something special if required.

Ouch and btw linux-pm list exists where this stuff is debated...

Because the power management design should be layered:

device power off request -->

(start)

|
| class - disable user space interfaces, stop timers etc
|
| driver - save hardware state, prepare hardware for poweroff
|
| bus - power off physical device
|
| firmware (ex ACPI) - additional power logic (ex external device components
|					       may need to be powered off.
|					       This will typically involve Dx
|					       states)

/
(finish)

| driver (*suspend) - tell class to stop logical device (often large amounts
| of duplicate code between drivers)

|		    - totally ignore ACPI firmware Dx states (how could we
|		      handle them, we don't know enough about the layers below
|		      us.  ACPI may not be compiled into the kernel, the device
|		      may not have ACPI extensions etc.)

Quoted 4 lines

Saving/restoring state is way more work than actual power state transitions. Provide nice helpers so that power state transitions are easy, but leave the work on the drivers, so that drivers may do something special if required.

Could you provide an example of a special case? I'm not sure if we should design our model around the least common denominator of standard compliance.

Hi!

Quoted 16 lines

Because the power management design should be layered:

device power off request -->

(start)

|
| class - disable user space interfaces, stop timers etc
|
| driver - save hardware state, prepare hardware for poweroff
|
| bus - power off physical device
|
| firmware (ex ACPI) - additional power logic (ex external device components
|					       may need to be powered off.
|					       This will typically involve Dx
|					       states)

Quoted 2 lines

/
(finish)

Driver needs to be in control. For every rule, there's exception (buggy hw?). It is fine if classes or buses provide helpers, but ultimately it needs to be driver's suspend using those helpers.

Quoted 2 lines

| driver (*suspend) - tell class to stop logical device (often large amounts
| of duplicate code between drivers)

Well, then feel free to move that duplicate code into helper functions. I do think that I'd notice that large amounts of duplicate code, but feel free to prove me wrong.

Quoted 4 lines

|		    - totally ignore ACPI firmware Dx states (how could we
|		      handle them, we don't know enough about the layers below
|		      us.  ACPI may not be compiled into the kernel, the device
|		      may not have ACPI extensions etc.)

Helpers can do that, right?

Quoted 6 lines

Quoted 5 lines

Saving/restoring state is way more work than actual power state transitions. Provide nice helpers so that power state transitions are easy, but leave the work on the drivers, so that drivers may do something special if required.

Quoted 3 lines

Could you provide an example of a special case? I'm not sure if we should design our model around the least common denominator of standard compliance.

Well, if you want your network card to do wake-on-lan, you need to keep it powered on... And the driver is the piece of code that knows if you want to do wake-on-lan, right?

Quoted 6 lines

Quoted 2 lines

| driver (*suspend) - tell class to stop logical device (often large amounts
| of duplicate code between drivers)

Well, then feel free to move that duplicate code into helper functions. I do think that I'd notice that large amounts of duplicate code, but feel free to prove me wrong.

Quoted 5 lines

Quoted 4 lines

|		    - totally ignore ACPI firmware Dx states (how could we
|		      handle them, we don't know enough about the layers below
|		      us.  ACPI may not be compiled into the kernel, the device
|		      may not have ACPI extensions etc.)

Quoted 1 line

Helpers can do that, right?

Maybe, something really needs to be done about it in the short run.

Why must the driver be in complete control? We can't ensure every driver will behave consistently. Why have a driver model at all if we can't break tasks into layers of responsibility?

Yes, layering has small losses in flexability, but the advantage of ensuring uniform behavior and capabilities is very important. With layering, code repitition is minimal and everything in the kernel is all around more simplistic and easy to maintain. I think we can have a higher average quality of functionality and stability with this approach. Also less work will be

required when developing new drivers or adding new functionality. For example, a developer could update class level code without breaking or needing to change driver and bus level code.

Quoted 3 lines

Well, if you want your network card to do wake-on-lan, you need to keep it powered on... And the driver is the piece of code that knows if you want to do wake-on-lan, right?

Firmware like ACPI will tell us what wake-on-lan capabilities we have. The bus layer also handles much of the physical aspects. The user will say that he or she wants the logical device "eth1", as an example, to wake up the computer if any traffic is incoming. So really all of the layers play some role in wake-on-lan, right? Is a device useful for wake-on-lan without a corresponding logical ethernet device?

* legacy, like VGA or IDE -- okay, we need drivers for those. You need to reinit them on resume, anyway, so....

Quoted 2 lines

* legacy, like VGA or IDE -- okay, we need drivers for those. You need to reinit them on resume, anyway, so....

The current situation is that we can reinit /most/ VGA hardware from userspace, but it's less than trivial to make sure that the appropriate devices are shut down. Surely the situation is more like the following:

1) Some devices don't have drivers. In most cases, it doesn't matter what power state they're in. Therefore, setting them to D3 (or whatever SxD says) shouldn't be a problem.