PCI/ASPM: fix unexpected behavior when re-enabling L1
PATCH 1/1 PCI/ASPM: fix unexpected behavior when re-enabling L1
After the referenced commit we may see L1 sub-states being active unexpectedly. Following scenario as an example:
r8169 disables L1 because of known hardware issues on a number of systems. Implicitly L1.1 and L1.2 are disabled too.
On my system L1 and L1.1 work fine, but L1.2 causes missed rx packets. Therefore I write 1 to aspm_l1_1.
This removes ASPM_STATE_L1 from the disabled modes and therefore unexpectedly enables also L1.2. So return to the old behavior.
A comment in the commit message of the referenced change correctly points out that this behavior is inconsistent with aspm_attr_store_common(). So change aspm_attr_store_common() accordingly.
--- drivers/pci/pcie/aspm.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/pci/pcie/aspm.c +3 -1
@@ -1063,7 +1063,7 @@ static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)if (state &PCIE_LINK_STATE_L0S ) link->aspm_disable |=ASPM_STATE_L0S ;if (state &PCIE_LINK_STATE_L1 ) - link->aspm_disable |=ASPM_STATE_L1 ; + link->aspm_disable |=ASPM_STATE_L1 |ASPM_STATE_L1SS ;if (state &PCIE_LINK_STATE_L1_1 ) link->aspm_disable |=ASPM_STATE_L1_1 ;if (state &PCIE_LINK_STATE_L1_2 )
@@ -1251,6 +1251,8 @@ static ssize_t aspm_attr_store_common(struct device *dev, link->aspm_disable &= ~ASPM_STATE_L1 ; }else { link->aspm_disable |= state; +if (state &ASPM_STATE_L1 ) + link->aspm_disable |=ASPM_STATE_L1SS ; }pcie_config_aspm_link (link,policy_to_aspm_state (link));
Signature · 1 line
2.42.0
[+cc Sathy, Lukas]
Quoted 9 lines · Heiner Kallweit
After the referenced commit we may see L1 sub-states being active unexpectedly. Following scenario as an example:
r8169 disables L1 because of known hardware issues on a number of systems. Implicitly L1.1 and L1.2 are disabled too.
On my system L1 and L1.1 work fine, but L1.2 causes missed rx packets. Therefore I write 1 toaspm_l1_1.
This removesASPM_STATE_L1from the disabled modes and therefore unexpectedly enables also L1.2. So return to the old behavior.
Quoted 3 lines
A comment in the commit message of the referenced change correctly points out that this behavior is inconsistent with
aspm_attr_store_common(). So changeaspm_attr_store_common()accordingly.
I think we should split this into a pure revert of fb097dcd5a28 with the description of the unintended consequence, followed by another patch to change aspm_attr_store_common().
I guess the existing aspm_attr_store_common() behavior allows a similar unexpected behavior? For example, in this sequence:
- Write 0 to "l1_aspm" to disable L1
- Write 1 to "l1_1_aspm" to enable L1.1
does L1.2 get implicitly enabled as well even though that's clearly not what the user intended?
There's also the separate question of how the sysfs file and the pci_disable_link_state() API should interact. Drivers use that API when they know about a defect in their device, but the user can override that via syfs.
In [1], we have a similar situation with D3cold support, where we're thinking that we should not allow a user to use sysfs to override that driver knowledge.
Bjorn
[1] https://lore.kernel.org/r/b8a7f4af2b73f6b506ad8ddee59d747cbf834606.1695025365.git.lukas@wunner.de
Quoted 31 lines
Fixes fb097dcd5a28 ("PCI/ASPM: Disable only ASPM_STATE_L1 when driver disables L1")Signed-off-by Heiner Kallweit <[email protected]>--- drivers/pci/pcie/aspm.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/pci/pcie/aspm.c +3
-1@@ -1063,7 +1063,7 @@ static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)if (state &PCIE_LINK_STATE_L0S ) link->aspm_disable |=ASPM_STATE_L0S ;if (state &PCIE_LINK_STATE_L1 ) - link->aspm_disable |=ASPM_STATE_L1 ; + link->aspm_disable |=ASPM_STATE_L1 |ASPM_STATE_L1SS ;if (state &PCIE_LINK_STATE_L1_1 ) link->aspm_disable |=ASPM_STATE_L1_1 ;if (state &PCIE_LINK_STATE_L1_2 )@@ -1251,6 +1251,8 @@ static ssize_t aspm_attr_store_common(struct device *dev, link->aspm_disable &= ~ASPM_STATE_L1 ; }else { link->aspm_disable |= state; +if (state &ASPM_STATE_L1 ) + link->aspm_disable |=ASPM_STATE_L1SS ; }pcie_config_aspm_link (link,policy_to_aspm_state (link));Signature · 1 line
2.42.0
Quoted 11 lines · Bjorn Helgaas
[+cc Sathy, Lukas]
Quoted 8 lines · Heiner Kallweit
After the referenced commit we may see L1 sub-states being active unexpectedly. Following scenario as an example:
r8169 disables L1 because of known hardware issues on a number of systems. Implicitly L1.1 and L1.2 are disabled too.
On my system L1 and L1.1 work fine, but L1.2 causes missed rx packets. Therefore I write 1 toaspm_l1_1.
This removesASPM_STATE_L1from the disabled modes and therefore unexpectedly enables also L1.2. So return to the old behavior.
IIUC, the above-mentioned SysFS issue will be fixed by your change to aspm_attr_store_common(), right?
Can you elaborate why you need the following change?
Quoted 6 lines
Quoted 6 lines
drivers/pci/pcie/aspm.c +1
-1@@ -1063,7 +1063,7 @@ static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)if (state &PCIE_LINK_STATE_L1 ) - link->aspm_disable |=ASPM_STATE_L1 ; + link->aspm_disable |=ASPM_STATE_L1 |ASPM_STATE_L1SS ;
Quoted 8 lines
Quoted 3 lines
A comment in the commit message of the referenced change correctly points out that this behavior is inconsistent with
aspm_attr_store_common(). So changeaspm_attr_store_common()accordingly.I think we should split this into a pure revert of
fb097dcd5a28with the description of the unintended consequence, followed by another patch to changeaspm_attr_store_common().
I agree, the revert and new change should be split into two patches.
Quoted 53 lines
I guess the existing
aspm_attr_store_common()behavior allows a similar unexpected behavior? For example, in this sequence:- Write 0 to "
l1_aspm" to disable L1
- Write 1 to "l1_1_aspm" to enable L1.1does L1.2 get implicitly enabled as well even though that's clearly not what the user intended?
There's also the separate question of how the sysfs file and the
pci_disable_link_state()API should interact. Drivers use that API when they know about a defect in their device, but the user can override that via syfs.In [1], we have a similar situation with D3cold support, where we're thinking that we should not allow a user to use sysfs to override that driver knowledge.
Bjorn
[1] https://lore.kernel.org/r/b8a7f4af2b73f6b506ad8ddee59d747cbf834606.1695025365.git.lukas@wunner.de
Quoted 31 lines
Fixes fb097dcd5a28 ("PCI/ASPM: Disable only ASPM_STATE_L1 when driver disables L1")Signed-off-by Heiner Kallweit <[email protected]>--- drivers/pci/pcie/aspm.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/pci/pcie/aspm.c +3
-1@@ -1063,7 +1063,7 @@ static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)if (state &PCIE_LINK_STATE_L0S ) link->aspm_disable |=ASPM_STATE_L0S ;if (state &PCIE_LINK_STATE_L1 ) - link->aspm_disable |=ASPM_STATE_L1 ; + link->aspm_disable |=ASPM_STATE_L1 |ASPM_STATE_L1SS ;if (state &PCIE_LINK_STATE_L1_1 ) link->aspm_disable |=ASPM_STATE_L1_1 ;if (state &PCIE_LINK_STATE_L1_2 )@@ -1251,6 +1251,8 @@ static ssize_t aspm_attr_store_common(struct device *dev, link->aspm_disable &= ~ASPM_STATE_L1 ; }else { link->aspm_disable |= state; +if (state &ASPM_STATE_L1 ) + link->aspm_disable |=ASPM_STATE_L1SS ; }pcie_config_aspm_link (link,policy_to_aspm_state (link));Signature · 1 line
2.42.0
Signature · 2 lines
Sathyanarayanan Kuppuswamy Linux Kernel Developer
Quoted 15 lines · Kuppuswamy Sathyanarayanan
Quoted 11 lines · Bjorn Helgaas
[+cc Sathy, Lukas]
Quoted 8 lines · Heiner Kallweit
After the referenced commit we may see L1 sub-states being active unexpectedly. Following scenario as an example:
r8169 disables L1 because of known hardware issues on a number of systems. Implicitly L1.1 and L1.2 are disabled too.
On my system L1 and L1.1 work fine, but L1.2 causes missed rx packets. Therefore I write 1 toaspm_l1_1.
This removesASPM_STATE_L1from the disabled modes and therefore unexpectedly enables also L1.2. So return to the old behavior.
Quoted 3 lines
IIUC, the above-mentioned SysFS issue will be fixed by your change to
aspm_attr_store_common(), right?
In my use case the driver uses pci_disable_link_state() to disable L1 and implicitly also L1.1 and L1.2. The following change is needed so that L1.2 remains disabled if L1 is removed from the list of disabled states later. That's the revert we're talking about.
Quoted 73 lines
Can you elaborate why you need the following change?
Quoted 6 lines
Quoted 6 lines
drivers/pci/pcie/aspm.c +1
-1@@ -1063,7 +1063,7 @@ static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)if (state &PCIE_LINK_STATE_L1 ) - link->aspm_disable |=ASPM_STATE_L1 ; + link->aspm_disable |=ASPM_STATE_L1 |ASPM_STATE_L1SS ;Quoted 8 lines
Quoted 3 lines
A comment in the commit message of the referenced change correctly points out that this behavior is inconsistent with
aspm_attr_store_common(). So changeaspm_attr_store_common()accordingly.I think we should split this into a pure revert of
fb097dcd5a28with the description of the unintended consequence, followed by another patch to changeaspm_attr_store_common().I agree, the revert and new change should be split into two patches.
Quoted 53 lines
I guess the existing
aspm_attr_store_common()behavior allows a similar unexpected behavior? For example, in this sequence:- Write 0 to "
l1_aspm" to disable L1
- Write 1 to "l1_1_aspm" to enable L1.1does L1.2 get implicitly enabled as well even though that's clearly not what the user intended?
There's also the separate question of how the sysfs file and the
pci_disable_link_state()API should interact. Drivers use that API when they know about a defect in their device, but the user can override that via syfs.In [1], we have a similar situation with D3cold support, where we're thinking that we should not allow a user to use sysfs to override that driver knowledge.
Bjorn
[1] https://lore.kernel.org/r/b8a7f4af2b73f6b506ad8ddee59d747cbf834606.1695025365.git.lukas@wunner.de
Quoted 31 lines
Fixes fb097dcd5a28 ("PCI/ASPM: Disable only ASPM_STATE_L1 when driver disables L1")Signed-off-by Heiner Kallweit <[email protected]>--- drivers/pci/pcie/aspm.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/pci/pcie/aspm.c +3
-1@@ -1063,7 +1063,7 @@ static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)if (state &PCIE_LINK_STATE_L0S ) link->aspm_disable |=ASPM_STATE_L0S ;if (state &PCIE_LINK_STATE_L1 ) - link->aspm_disable |=ASPM_STATE_L1 ; + link->aspm_disable |=ASPM_STATE_L1 |ASPM_STATE_L1SS ;if (state &PCIE_LINK_STATE_L1_1 ) link->aspm_disable |=ASPM_STATE_L1_1 ;if (state &PCIE_LINK_STATE_L1_2 )@@ -1251,6 +1251,8 @@ static ssize_t aspm_attr_store_common(struct device *dev, link->aspm_disable &= ~ASPM_STATE_L1 ; }else { link->aspm_disable |= state; +if (state &ASPM_STATE_L1 ) + link->aspm_disable |=ASPM_STATE_L1SS ; }pcie_config_aspm_link (link,policy_to_aspm_state (link));Signature · 1 line
2.42.0
Quoted 12 lines · Bjorn Helgaas
[+cc Sathy, Lukas]
Quoted 9 lines · Heiner Kallweit
After the referenced commit we may see L1 sub-states being active unexpectedly. Following scenario as an example:
r8169 disables L1 because of known hardware issues on a number of systems. Implicitly L1.1 and L1.2 are disabled too.
On my system L1 and L1.1 work fine, but L1.2 causes missed rx packets. Therefore I write 1 toaspm_l1_1.
This removesASPM_STATE_L1from the disabled modes and therefore unexpectedly enables also L1.2. So return to the old behavior.
Quoted 8 lines
Quoted 3 lines
A comment in the commit message of the referenced change correctly points out that this behavior is inconsistent with
aspm_attr_store_common(). So changeaspm_attr_store_common()accordingly.I think we should split this into a pure revert of
fb097dcd5a28with the description of the unintended consequence, followed by another patch to changeaspm_attr_store_common().
OK
Quoted 9 lines
I guess the existing
aspm_attr_store_common()behavior allows a similar unexpected behavior? For example, in this sequence:- Write 0 to "
l1_aspm" to disable L1
- Write 1 to "l1_1_aspm" to enable L1.1does L1.2 get implicitly enabled as well even though that's clearly not what the user intended?
Right, it's the same here. Therefore the second change in the patch.
Quoted 9 lines
There's also the separate question of how the sysfs file and the
pci_disable_link_state()API should interact. Drivers use that API when they know about a defect in their device, but the user can override that via syfs.In [1], we have a similar situation with D3cold support, where we're thinking that we should not allow a user to use sysfs to override that driver knowledge.
In my r8169 use case ASPM works fine on one system, but causes issues on another one with same NIC chip version. So it may be a wild mix of NIC hw erratum, BIOS bug, mainboard chipset incompatibilities, etc. Therefore I disable L1 per default, and allow users to re-enable L1/L1.1/L1.2 if their system isn't affected.
Quoted 35 lines
Bjorn
[1] https://lore.kernel.org/r/b8a7f4af2b73f6b506ad8ddee59d747cbf834606.1695025365.git.lukas@wunner.de
Quoted 31 lines
Fixes fb097dcd5a28 ("PCI/ASPM: Disable only ASPM_STATE_L1 when driver disables L1")Signed-off-by Heiner Kallweit <[email protected]>--- drivers/pci/pcie/aspm.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/pci/pcie/aspm.c +3
-1@@ -1063,7 +1063,7 @@ static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)if (state &PCIE_LINK_STATE_L0S ) link->aspm_disable |=ASPM_STATE_L0S ;if (state &PCIE_LINK_STATE_L1 ) - link->aspm_disable |=ASPM_STATE_L1 ; + link->aspm_disable |=ASPM_STATE_L1 |ASPM_STATE_L1SS ;if (state &PCIE_LINK_STATE_L1_1 ) link->aspm_disable |=ASPM_STATE_L1_1 ;if (state &PCIE_LINK_STATE_L1_2 )@@ -1251,6 +1251,8 @@ static ssize_t aspm_attr_store_common(struct device *dev, link->aspm_disable &= ~ASPM_STATE_L1 ; }else { link->aspm_disable |= state; +if (state &ASPM_STATE_L1 ) + link->aspm_disable |=ASPM_STATE_L1SS ; }pcie_config_aspm_link (link,policy_to_aspm_state (link));Signature · 1 line
2.42.0