Query on ASPM driver design
A message that is not in the archive (sent off-list or not yet received): 20210723203206.GA436727@bjorn-Precision-5520
Hi Bjorn,
Quoted 5 lines · Bjorn Helgaas
Quoted 1 line · On Fri, Jul 23, 2021 at 01:11:18PM -0700, [email protected] wrote:
Hi Bjorn,
It's best if you can cc: [email protected] so others can contribute and benefit.
Good idea. Added now.
Quoted 11 lines
Quoted 8 lines
I have a question regarding PCIe ASPM driver in upstream. Looks like current ASPM driver is going to enable ASPM L1 and L1SS based on EP's config space capability register read. Why ASPM driver is enabling L1SS based on capability, instead of that can ASPM honor default control register value (in EP config space) and let pci device driver probe (or later after probe) to make the decision if ASPM needs to be enabled or not.
Are you asking why the PCI core makes the decision about enabling ASPM instead of having each device driver decide?
Yes.
Quoted 2 lines
If you want each driver to decide, what benefit would that have?
Basically if PCI EP has capability to support ASPM L1 and L1SS but power on default control reg values are meant to enumerate with ASPM disabled. Which means EP wants to keep ASPM disabled right from the enumeration, and at some point of time later EP wants to enable the ASPM. Main benefit is to give control to EP to enumerate with what ever its control reg's power on default value is. EP does not want to enable ASPM during its boot up and after entering to mission mode use case it would enable the ASPM.
Quoted 25 lines
Obviously ASPM involves a power vs performance tradeoff, but functionally, ASPM is designed so drivers don't need to be aware of it.
Quoted 3 lines
As far as i know there is an option to update
quirk.cfor a given device id and vendor id to disable L0s/L1/L1SS but that sounds like adding a software workaround to a device specific HW bug.Yes. It is common practice to use software quirks to work around hardware defects.
Quoted 4 lines
Also, i know 5.5 kernel added a patch to control aspm using sysfs per link basis:-
https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/
Yes. This is intended for use by tools like powertop to tune the power vs performance tradeoff.
Quoted 4 lines
Basically point is: it is possible to honor what device control reg reflects power on default and let the pci ep driver running on host to make the decision when to enable/disable the aspm in kernel space pci driver.
Quoted 4 lines
There is a
pci_disable_link_state()interface that drivers can use to disable certain link states. Some drivers use this to work around hardware defects, but it would be better to use quirks in that situation.
Thanks for pointing this API, which quirk also uses. But we just have disable ver which EP driver can call only after enumeration is done. i was thinking of the other way round where EP enumerates and then calls enable API at some point of time. Also, if it decides to again disable and then enable.
Quoted 10 lines
Quoted 4 lines
Sorry if this was already well thought/discussed argument in the design of ASPM driver, i would appreciate, if you can shed some light on the reason for not taking that approach.
I don't know the history of that decision, but in general we try to do things in the core instead of endpoint drivers whenever possible because it reduces the complexity of drivers.
Bjorn
Thanks for taking your time to respond very promptly Bjorn!
-Hemant
---
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Quoted 17 lines · On Fri, Jul 23, 2021 at 03:04:52PM -0700, [email protected] wrote:
Quoted 11 lines · Bjorn Helgaas
Quoted 7 lines · On Fri, Jul 23, 2021 at 01:11:18PM -0700, [email protected] wrote:
I have a question regarding PCIe ASPM driver in upstream. Looks like current ASPM driver is going to enable ASPM L1 and L1SS based on EP's config space capability register read. Why ASPM driver is enabling L1SS based on capability, instead of that can ASPM honor default control register value (in EP config space) and let pci device driver probe (or later after probe) to make the decision if ASPM needs to be enabled or not.
Are you asking why the PCI core makes the decision about enabling ASPM instead of having each device driver decide?
Yes.
Quoted 1 line
If you want each driver to decide, what benefit would that have?
Quoted 8 lines
Basically if PCI EP has capability to support ASPM L1 and L1SS but power on default control reg values are meant to enumerate with ASPM disabled. Which means EP wants to keep ASPM disabled right from the enumeration, and at some point of time later EP wants to enable the ASPM. Main benefit is to give control to EP to enumerate with what ever its control reg's power on default value is. EP does not want to enable ASPM during its boot up and after entering to mission mode use case it would enable the ASPM.
The power-on default value for the "ASPM Control" field in the Link Control register is 00b, which means ASPM is disabled. The current Linux behavior is that when we enumerate the device, we evaluate the L0s and L1 exit latencies and enable ASPM if the device can tolerate them.
It sounds like you want to prevent ASPM from being enabled until the driver explicitly enables it. Why? The device should not be active until a driver claims it, so it should not be a problem to have ASPM enabled.
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines
Basically point is: it is possible to honor what device control reg reflects power on default and let the pci ep driver running on host to make the decision when to enable/disable the aspm in kernel space pci driver.
There is a
pci_disable_link_state()interface that drivers can use to disable certain link states. Some drivers use this to work around hardware defects, but it would be better to use quirks in that situation.
Quoted 5 lines
Thanks for pointing this API, which quirk also uses. But we just have disable ver which EP driver can call only after enumeration is done. i was thinking of the other way round where EP enumerates and then calls enable API at some point of time. Also, if it decides to again disable and then enable.
There is currently no pci_enable_link_state() because nobody has needed it and implemented it. I would push back a little bit on adding this because I don't want to encourage drivers to mess with ASPM.
Bjorn
Hi Bjorn,
I think it makes sense to have the scope of keeping default ASPM policy disable and API pci_enable_link_state() to selectively enable by EP Driver.
sysfs interface for ASPM also does not allow enabling ASPM for a device if the default policy (policy_to_aspm_state()) does not allow it.
Consider a situation, for a platform one wants to utilize ASPM capability of an onboard PCIe device because it is well evaluated, at the same time they want to keep ASPM disabled for other PCIe devices that can be connected on open PCIe slot to avoid possible performance issue.
I see ASPM is broken on many devices, though the device shows ASPM capabilities but has performance issues when it is enabled.
Thanks,
Om
Quoted 64 lines · Bjorn Helgaas
External email: Use caution opening links or attachments
Quoted 17 lines · On Fri, Jul 23, 2021 at 03:04:52PM -0700, [email protected] wrote:
Quoted 11 lines · Bjorn Helgaas
Quoted 7 lines · On Fri, Jul 23, 2021 at 01:11:18PM -0700, [email protected] wrote:
I have a question regarding PCIe ASPM driver in upstream. Looks like current ASPM driver is going to enable ASPM L1 and L1SS based on EP's config space capability register read. Why ASPM driver is enabling L1SS based on capability, instead of that can ASPM honor default control register value (in EP config space) and let pci device driver probe (or later after probe) to make the decision if ASPM needs to be enabled or not.
Are you asking why the PCI core makes the decision about enabling ASPM instead of having each device driver decide?
Yes.
Quoted 1 line
If you want each driver to decide, what benefit would that have?
Quoted 8 lines
Basically if PCI EP has capability to support ASPM L1 and L1SS but power on default control reg values are meant to enumerate with ASPM disabled. Which means EP wants to keep ASPM disabled right from the enumeration, and at some point of time later EP wants to enable the ASPM. Main benefit is to give control to EP to enumerate with what ever its control reg's power on default value is. EP does not want to enable ASPM during its boot up and after entering to mission mode use case it would enable the ASPM.
The power-on default value for the "ASPM Control" field in the Link Control register is 00b, which means ASPM is disabled. The current Linux behavior is that when we enumerate the device, we evaluate the L0s and L1 exit latencies and enable ASPM if the device can tolerate them.
It sounds like you want to prevent ASPM from being enabled until the driver explicitly enables it. Why? The device should not be active until a driver claims it, so it should not be a problem to have ASPM enabled.
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines
Basically point is: it is possible to honor what device control reg reflects power on default and let the pci ep driver running on host to make the decision when to enable/disable the aspm in kernel space pci driver.
There is a
pci_disable_link_state()interface that drivers can use to disable certain link states. Some drivers use this to work around hardware defects, but it would be better to use quirks in that situation.Quoted 5 lines
Thanks for pointing this API, which quirk also uses. But we just have disable ver which EP driver can call only after enumeration is done. i was thinking of the other way round where EP enumerates and then calls enable API at some point of time. Also, if it decides to again disable and then enable.
There is currently no
pci_enable_link_state()because nobody has needed it and implemented it. I would push back a little bit on adding this because I don't want to encourage drivers to mess with ASPM.Bjorn
Quoted 8 lines · Om Prakash Singh
Hi Bjorn,
I think it makes sense to have the scope of keeping default ASPM policy disable and APIpci_enable_link_state()to selectively enable by EP Driver.sysfs interface for ASPM also does not allow enabling ASPM for a device if the default policy (
policy_to_aspm_state()) does not allow it.
The ASPM policy implementation may require changes. I think the current setup where a policy is compiled into the kernel via Kconfig options is seriously flawed.
We need a fail-safe kernel parameter, i.e., "pcie_aspm=off", for cases where devices don't work at all with ASPM. We need quirks to work around devices known to be broken, e.g., those that advertise ASPM support that doesn't actually work, or those that advertise incorrect exit latencies. I think most other configuration should be done via sysfs.
Quoted 8 lines
Consider a situation, for a platform one wants to utilize ASPM capability of an onboard PCIe device because it is well evaluated, at the same time they want to keep ASPM disabled for other PCIe devices that can be connected on open PCIe slot to avoid possible performance issue.
I see ASPM is broken on many devices, though the device shows ASPM capabilities but has performance issues when it is enabled.
I'll wait to see your proposal and use case before commenting on this.
Bjorn
Quoted 30 lines · Bjorn Helgaas
External email: Use caution opening links or attachments
Quoted 8 lines · Om Prakash Singh
Hi Bjorn,
I think it makes sense to have the scope of keeping default ASPM policy disable and APIpci_enable_link_state()to selectively enable by EP Driver.sysfs interface for ASPM also does not allow enabling ASPM for a device if the default policy (
policy_to_aspm_state()) does not allow it.The ASPM policy implementation may require changes. I think the current setup where a policy is compiled into the kernel via
Kconfigoptions is seriously flawed.We need a fail-safe kernel parameter, i.e., "pcie_aspm=off", for cases where devices don't work at all with ASPM. We need quirks to work around devices known to be broken, e.g., those that advertise ASPM support that doesn't actually work, or those that advertise incorrect exit latencies. I think most other configuration should be done via sysfs.
Quoted 6 lines
Consider a situation, for a platform one wants to utilize ASPM capability of an onboard PCIe device because it is well evaluated, at the same time they want to keep ASPM disabled for other PCIe devices that can be connected on open PCIe slot to avoid possible performance issue.
Quoted 4 lines
Quoted 2 lines
I see ASPM is broken on many devices, though the device shows ASPM capabilities but has performance issues when it is enabled.
I'll wait to see your proposal and use case before commenting on this.
few suggestions:
1. We can have a device-tree property to disable ASPM capabilities of a Root port. Corresponding to my example, the device-tree can use be use to enable ASPM capabilities of Root ports that have known/good/onboard PCIe device, and keep ASPM disabled for opens slots
2. sysfs should allow overriding default ASPM policy.
With below change system can boot with default policy ASPM disabled
(performance). But sysfs will still allow to enable ASPM capability of selective device
drivers/pci/pcie/aspm.c +1 -1
@@ -1278,7 +1278,7 @@ link->aspm_disable |= state; } -pcie_config_aspm_link (link,policy_to_aspm_state (link)); +pcie_config_aspm_link (link, state);mutex_unlock (&aspm_lock);up_read (&pci_bus_sem);
3. Add API pci_enable_link_state()
This will give control to driver to change ASPM at runtime depending
on user selected performance mode. For example Android has different performance modes for Wifi chip, for sleep and active state. We are using similar API to overcome some performance issue related to wifi on our internal platform.
Quoted 3 lines
Bjorn
[+cc Rafael, driver-specific vs generic PM at very end]
Quoted 31 lines · Om Prakash Singh
Quoted 30 lines · Bjorn Helgaas
Quoted 8 lines · Om Prakash Singh
Hi Bjorn,
I think it makes sense to have the scope of keeping default ASPM policy disable and APIpci_enable_link_state()to selectively enable by EP Driver.sysfs interface for ASPM also does not allow enabling ASPM for a device if the default policy (
policy_to_aspm_state()) does not allow it.The ASPM policy implementation may require changes. I think the current setup where a policy is compiled into the kernel via
Kconfigoptions is seriously flawed.We need a fail-safe kernel parameter, i.e., "pcie_aspm=off", for cases where devices don't work at all with ASPM. We need quirks to work around devices known to be broken, e.g., those that advertise ASPM support that doesn't actually work, or those that advertise incorrect exit latencies. I think most other configuration should be done via sysfs.
Quoted 8 lines
Consider a situation, for a platform one wants to utilize ASPM capability of an onboard PCIe device because it is well evaluated, at the same time they want to keep ASPM disabled for other PCIe devices that can be connected on open PCIe slot to avoid possible performance issue.
I see ASPM is broken on many devices, though the device shows ASPM capabilities but has performance issues when it is enabled.
Quoted 8 lines
Quoted 1 line
I'll wait to see your proposal and use case before commenting on this.
few suggestions:
1. We can have a device-tree property to disable ASPM capabilities of a Root port. Corresponding to my example, the device-tree can use be use to enable ASPM capabilities of Root ports that have known/good/onboard PCIe device, and keep ASPM disabled for opens slots
ASPM can only be enabled for links between two devices. For empty slots, there's an upstream device (Root Port or Switch Downstream Port) but no downstream device, so ASPM won't be enabled anyway.
Quoted 4 lines
2. sysfs should allow overriding default ASPM policy.
With below change system can boot with default policy ASPM disabled
(performance). But sysfs will still allow to enable ASPM capability of selective device
This sounds like a good idea. If we could get rid of the Kconfig ASPM policy selection, we'd likely make PCIEASPM_POWER_SUPERSAVE the default. Then userspace could use sysfs to disable ASPM states selective as necessary for performance.
Even before removing the Kconfig policy selection (or if we can't do that), I think it make sense to allow sysfs to override it.
Quoted 14 lines
drivers/pci/pcie/aspm.c +1
-1@@ -1278,7 +1278,7 @@ link->aspm_disable |= state; } -pcie_config_aspm_link (link,policy_to_aspm_state (link)); +pcie_config_aspm_link (link, state);mutex_unlock (&aspm_lock);up_read (&pci_bus_sem);
Quoted 6 lines
3. Add API
pci_enable_link_state()
This will give control to driver to change ASPM at runtime depending on
user selected performance mode. For example Android has different performance modes for Wifi chip, for sleep and active state. We are using similar API to overcome some performance issue related to wifi on our internal platform.
I'd prefer to keep performance modes out of the drivers as much as possible because that's not really a scalable approach. I'm not a power management expert, but I suspect the most maintainable approach is to do this in the generic PM core, with selectable overrides via sysfs as necessary. I cc'd Rafael in case he wants to chime in.
Maybe there's a place for drivers to tweak ASPM at runtime, but I'm not convinced yet. I think the intent of ASPM is that devices advertise exit latencies that correspond with their internal buffering, so the L0s and L1 states can be used with no significant performance impact. But without more specifics, we can talk about this all day without getting anywhere.
Bjorn
Quoted 41 lines · Bjorn Helgaas
External email: Use caution opening links or attachments
[+cc Rafael, driver-specific vs generic PM at very end]
Quoted 35 lines · Om Prakash Singh
Quoted 27 lines · Bjorn Helgaas
Quoted 8 lines · Om Prakash Singh
Hi Bjorn,
I think it makes sense to have the scope of keeping default ASPM policy disable and APIpci_enable_link_state()to selectively enable by EP Driver.sysfs interface for ASPM also does not allow enabling ASPM for a device if the default policy (
policy_to_aspm_state()) does not allow it.The ASPM policy implementation may require changes. I think the current setup where a policy is compiled into the kernel via
Kconfigoptions is seriously flawed.We need a fail-safe kernel parameter, i.e., "pcie_aspm=off", for cases where devices don't work at all with ASPM. We need quirks to work around devices known to be broken, e.g., those that advertise ASPM support that doesn't actually work, or those that advertise incorrect exit latencies. I think most other configuration should be done via sysfs.
Quoted 6 lines
Consider a situation, for a platform one wants to utilize ASPM capability of an onboard PCIe device because it is well evaluated, at the same time they want to keep ASPM disabled for other PCIe devices that can be connected on open PCIe slot to avoid possible performance issue.
Quoted 4 lines
Quoted 2 lines
I see ASPM is broken on many devices, though the device shows ASPM capabilities but has performance issues when it is enabled.
I'll wait to see your proposal and use case before commenting on this.
few suggestions:
Quoted 8 lines
Quoted 4 lines
1. We can have a device-tree property to disable ASPM capabilities of a Root port. Corresponding to my example, the device-tree can use be use to enable ASPM capabilities of Root ports that have known/good/onboard PCIe device, and keep ASPM disabled for opens slots
ASPM can only be enabled for links between two devices. For empty slots, there's an upstream device (Root Port or Switch Downstream Port) but no downstream device, so ASPM won't be enabled anyway.
Yes, I meant to say this device-tree property will allow to disable ASPM capabilities of selected root port, ASPM will be disabled even if downstream device has ASPM capability.
Quoted 6 lines
Quoted 5 lines
2. sysfs should allow overriding default ASPM policy.
With below change system can boot with default policy ASPM disabled
(performance). But sysfs will still allow to enable ASPM capability of selective device
Quoted 7 lines
This sounds like a good idea. If we could get rid of the
KconfigASPM policy selection, we'd likely makePCIEASPM_POWER_SUPERSAVEthe default. Then userspace could use sysfs to disable ASPM states selective as necessary for performance.Even before removing the
Kconfigpolicy selection (or if we can't do that), I think it make sense to allow sysfs to override it.
I see Kconfig policy will is useful as different platform owner may want to keep different default policy for their platform.
Quoted 28 lines
Quoted 15 lines
drivers/pci/pcie/aspm.c +1
-1@@ -1278,7 +1278,7 @@ link->aspm_disable |= state; } -pcie_config_aspm_link (link,policy_to_aspm_state (link)); +pcie_config_aspm_link (link, state);mutex_unlock (&aspm_lock);up_read (&pci_bus_sem);Quoted 6 lines
3. Add API
pci_enable_link_state()
This will give control to driver to change ASPM at runtime depending on
user selected performance mode. For example Android has different performance modes for Wifi chip, for sleep and active state. We are using similar API to overcome some performance issue related to wifi on our internal platform.I'd prefer to keep performance modes out of the drivers as much as possible because that's not really a scalable approach. I'm not a power management expert, but I suspect the most maintainable approach is to do this in the generic PM core, with selectable overrides via sysfs as necessary. I cc'd Rafael in case he wants to chime in.
Quoted 6 lines
Maybe there's a place for drivers to tweak ASPM at runtime, but I'm not convinced yet. I think the intent of ASPM is that devices advertise exit latencies that correspond with their internal buffering, so the L0s and L1 states can be used with no significant performance impact. But without more specifics, we can talk about this all day without getting anywhere.
We are discussing about all these options because many times device doesn't performs well with ASPM capabilities what they advertise.
Quoted 3 lines
Bjorn
Quoted 55 lines · Bjorn Helgaas
External email: Use caution opening links or attachments
[+cc Rafael, driver-specific vs generic PM at very end]
Quoted 31 lines · Om Prakash Singh
Quoted 30 lines · Bjorn Helgaas
Quoted 8 lines · Om Prakash Singh
Hi Bjorn,
I think it makes sense to have the scope of keeping default ASPM policy disable and APIpci_enable_link_state()to selectively enable by EP Driver.sysfs interface for ASPM also does not allow enabling ASPM for a device if the default policy (
policy_to_aspm_state()) does not allow it.The ASPM policy implementation may require changes. I think the current setup where a policy is compiled into the kernel via
Kconfigoptions is seriously flawed.We need a fail-safe kernel parameter, i.e., "pcie_aspm=off", for cases where devices don't work at all with ASPM. We need quirks to work around devices known to be broken, e.g., those that advertise ASPM support that doesn't actually work, or those that advertise incorrect exit latencies. I think most other configuration should be done via sysfs.
Quoted 8 lines
Consider a situation, for a platform one wants to utilize ASPM capability of an onboard PCIe device because it is well evaluated, at the same time they want to keep ASPM disabled for other PCIe devices that can be connected on open PCIe slot to avoid possible performance issue.
I see ASPM is broken on many devices, though the device shows ASPM capabilities but has performance issues when it is enabled.
Quoted 8 lines
Quoted 1 line
I'll wait to see your proposal and use case before commenting on this.
few suggestions:
1. We can have a device-tree property to disable ASPM capabilities of a Root port. Corresponding to my example, the device-tree can use be use to enable ASPM capabilities of Root ports that have known/good/onboard PCIe device, and keep ASPM disabled for opens slots
ASPM can only be enabled for links between two devices. For empty slots, there's an upstream device (Root Port or Switch Downstream Port) but no downstream device, so ASPM won't be enabled anyway.
Quoted 4 lines
2. sysfs should allow overriding default ASPM policy.
With below change system can boot with default policy ASPM disabled
(performance). But sysfs will still allow to enable ASPM capability of selective device
Quoted 5 lines
This sounds like a good idea. If we could get rid of the
KconfigASPM policy selection, we'd likely makePCIEASPM_POWER_SUPERSAVEthe default. Then userspace could use sysfs to disable ASPM states selective as necessary for performance.
want to clarify this point, my idea was to have to have all ASPM mode disable by default (PERFORMANCE). And through sysfs user should be enable selected ASPM mode for each device. With current upstream code this is not possible. We can introduce one more policy USER_SELECTABLE if it make more sense.
Keeping ASPM disabled as default policy is important as any rough device connected can impact on overall system performance.
Adding Heiner Kallweit to share his opinion as well
Quoted 39 lines
Even before removing the
Kconfigpolicy selection (or if we can't do that), I think it make sense to allow sysfs to override it.Quoted 14 lines
drivers/pci/pcie/aspm.c +1
-1@@ -1278,7 +1278,7 @@ link->aspm_disable |= state; } -pcie_config_aspm_link (link,policy_to_aspm_state (link)); +pcie_config_aspm_link (link, state);mutex_unlock (&aspm_lock);up_read (&pci_bus_sem);Quoted 6 lines
3. Add API
pci_enable_link_state()
This will give control to driver to change ASPM at runtime depending on
user selected performance mode. For example Android has different performance modes for Wifi chip, for sleep and active state. We are using similar API to overcome some performance issue related to wifi on our internal platform.I'd prefer to keep performance modes out of the drivers as much as possible because that's not really a scalable approach. I'm not a power management expert, but I suspect the most maintainable approach is to do this in the generic PM core, with selectable overrides via sysfs as necessary. I cc'd Rafael in case he wants to chime in.
Maybe there's a place for drivers to tweak ASPM at runtime, but I'm not convinced yet. I think the intent of ASPM is that devices advertise exit latencies that correspond with their internal buffering, so the L0s and L1 states can be used with no significant performance impact. But without more specifics, we can talk about this all day without getting anywhere.
Bjorn