Skip to contentSkip to conversation

PCI/ASPM: fix unexpected behavior when re-enabling L1

PATCH In review Heiner Kallweit 3 participants 5 messages Aug 26, 2023 – Oct 10, 2023 PCI

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 PCI/ASPM: fix unexpected behavior when re-enabling L1

After the referenced commit we may see L1 sub-states being active unexpectedly. Following scenario as an example:
r8169 disables L1 because of known hardware issues on a number of systems. Implicitly L1.1 and L1.2 are disabled too.
On my system L1 and L1.1 work fine, but L1.2 causes missed rx packets. Therefore I write 1 to aspm_l1_1.
This removes ASPM_STATE_L1 from the disabled modes and therefore unexpectedly enables also L1.2. So return to the old behavior.

A comment in the commit message of the referenced change correctly points out that this behavior is inconsistent with aspm_attr_store_common(). So change aspm_attr_store_common() accordingly.

Fixes fb097dcd5a28 ("PCI/ASPM: Disable only ASPM_STATE_L1 when driver disables L1")
---
 drivers/pci/pcie/aspm.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/pci/pcie/aspm.c +3 -1
index 3dafba0b5..6d3788257 100644
@@ -1063,7 +1063,7 @@ static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)
 	if (state & PCIE_LINK_STATE_L0S)
 		link->aspm_disable |= ASPM_STATE_L0S;
 	if (state & PCIE_LINK_STATE_L1)
-		link->aspm_disable |= ASPM_STATE_L1;
+		link->aspm_disable |= ASPM_STATE_L1 | ASPM_STATE_L1SS;
 	if (state & PCIE_LINK_STATE_L1_1)
 		link->aspm_disable |= ASPM_STATE_L1_1;
 	if (state & PCIE_LINK_STATE_L1_2)
@@ -1251,6 +1251,8 @@ static ssize_t aspm_attr_store_common(struct device *dev,
 			link->aspm_disable &= ~ASPM_STATE_L1;
 	} else {
 		link->aspm_disable |= state;
+		if (state & ASPM_STATE_L1)
+			link->aspm_disable |= ASPM_STATE_L1SS;
 	}
 
 	pcie_config_aspm_link(link, policy_to_aspm_state(link));
Signature · 1 line
2.42.0
#1 lore raw

[+cc Sathy, Lukas]

Quoted 9 lines · Heiner Kallweit

After the referenced commit we may see L1 sub-states being active unexpectedly. Following scenario as an example:
r8169 disables L1 because of known hardware issues on a number of systems. Implicitly L1.1 and L1.2 are disabled too.
On my system L1 and L1.1 work fine, but L1.2 causes missed rx packets. Therefore I write 1 to aspm_l1_1.
This removes ASPM_STATE_L1 from the disabled modes and therefore unexpectedly enables also L1.2. So return to the old behavior.

Quoted 3 lines

A comment in the commit message of the referenced change correctly points out that this behavior is inconsistent with aspm_attr_store_common(). So change aspm_attr_store_common() accordingly.

I think we should split this into a pure revert of fb097dcd5a28 with the description of the unintended consequence, followed by another patch to change aspm_attr_store_common().

I guess the existing aspm_attr_store_common() behavior allows a similar unexpected behavior? For example, in this sequence:

- Write 0 to "l1_aspm" to disable L1
- Write 1 to "l1_1_aspm" to enable L1.1

does L1.2 get implicitly enabled as well even though that's clearly not what the user intended?

There's also the separate question of how the sysfs file and the pci_disable_link_state() API should interact. Drivers use that API when they know about a defect in their device, but the user can override that via syfs.

In [1], we have a similar situation with D3cold support, where we're thinking that we should not allow a user to use sysfs to override that driver knowledge.

Bjorn

[1] https://lore.kernel.org/r/b8a7f4af2b73f6b506ad8ddee59d747cbf834606.1695025365.git.lukas@wunner.de

Quoted 31 lines
Fixes fb097dcd5a28 ("PCI/ASPM: Disable only ASPM_STATE_L1 when driver disables L1")
---
 drivers/pci/pcie/aspm.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/pci/pcie/aspm.c +3 -1
index 3dafba0b5..6d3788257 100644
@@ -1063,7 +1063,7 @@ static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)
 	if (state & PCIE_LINK_STATE_L0S)
 		link->aspm_disable |= ASPM_STATE_L0S;
 	if (state & PCIE_LINK_STATE_L1)
-		link->aspm_disable |= ASPM_STATE_L1;
+		link->aspm_disable |= ASPM_STATE_L1 | ASPM_STATE_L1SS;
 	if (state & PCIE_LINK_STATE_L1_1)
 		link->aspm_disable |= ASPM_STATE_L1_1;
 	if (state & PCIE_LINK_STATE_L1_2)
@@ -1251,6 +1251,8 @@ static ssize_t aspm_attr_store_common(struct device *dev,
 			link->aspm_disable &= ~ASPM_STATE_L1;
 	} else {
 		link->aspm_disable |= state;
+		if (state & ASPM_STATE_L1)
+			link->aspm_disable |= ASPM_STATE_L1SS;
 	}
 
 	pcie_config_aspm_link(link, policy_to_aspm_state(link));
Signature · 1 line
2.42.0
#2 lore raw
Quoted 11 lines · Bjorn Helgaas

[+cc Sathy, Lukas]

Quoted 8 lines · Heiner Kallweit

After the referenced commit we may see L1 sub-states being active unexpectedly. Following scenario as an example:
r8169 disables L1 because of known hardware issues on a number of systems. Implicitly L1.1 and L1.2 are disabled too.
On my system L1 and L1.1 work fine, but L1.2 causes missed rx packets. Therefore I write 1 to aspm_l1_1.
This removes ASPM_STATE_L1 from the disabled modes and therefore unexpectedly enables also L1.2. So return to the old behavior.

IIUC, the above-mentioned SysFS issue will be fixed by your change to aspm_attr_store_common(), right?

Can you elaborate why you need the following change?

Quoted 6 lines
Quoted 6 lines
drivers/pci/pcie/aspm.c +1 -1
@@ -1063,7 +1063,7 @@ static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)
 	if (state & PCIE_LINK_STATE_L1)
-		link->aspm_disable |= ASPM_STATE_L1;
+		link->aspm_disable |= ASPM_STATE_L1 | ASPM_STATE_L1SS;
 
 
Quoted 8 lines
Quoted 3 lines

A comment in the commit message of the referenced change correctly points out that this behavior is inconsistent with aspm_attr_store_common(). So change aspm_attr_store_common() accordingly.

I think we should split this into a pure revert of fb097dcd5a28 with the description of the unintended consequence, followed by another patch to change aspm_attr_store_common().

I agree, the revert and new change should be split into two patches.

Quoted 53 lines

I guess the existing aspm_attr_store_common() behavior allows a similar unexpected behavior? For example, in this sequence:

- Write 0 to "l1_aspm" to disable L1
- Write 1 to "l1_1_aspm" to enable L1.1

does L1.2 get implicitly enabled as well even though that's clearly not what the user intended?

There's also the separate question of how the sysfs file and the pci_disable_link_state() API should interact. Drivers use that API when they know about a defect in their device, but the user can override that via syfs.

In [1], we have a similar situation with D3cold support, where we're thinking that we should not allow a user to use sysfs to override that driver knowledge.

Bjorn

[1] https://lore.kernel.org/r/b8a7f4af2b73f6b506ad8ddee59d747cbf834606.1695025365.git.lukas@wunner.de

Quoted 31 lines
Fixes fb097dcd5a28 ("PCI/ASPM: Disable only ASPM_STATE_L1 when driver disables L1")
---
 drivers/pci/pcie/aspm.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/pci/pcie/aspm.c +3 -1
index 3dafba0b5..6d3788257 100644
@@ -1063,7 +1063,7 @@ static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)
 	if (state & PCIE_LINK_STATE_L0S)
 		link->aspm_disable |= ASPM_STATE_L0S;
 	if (state & PCIE_LINK_STATE_L1)
-		link->aspm_disable |= ASPM_STATE_L1;
+		link->aspm_disable |= ASPM_STATE_L1 | ASPM_STATE_L1SS;
 	if (state & PCIE_LINK_STATE_L1_1)
 		link->aspm_disable |= ASPM_STATE_L1_1;
 	if (state & PCIE_LINK_STATE_L1_2)
@@ -1251,6 +1251,8 @@ static ssize_t aspm_attr_store_common(struct device *dev,
 			link->aspm_disable &= ~ASPM_STATE_L1;
 	} else {
 		link->aspm_disable |= state;
+		if (state & ASPM_STATE_L1)
+			link->aspm_disable |= ASPM_STATE_L1SS;
 	}
 
 	pcie_config_aspm_link(link, policy_to_aspm_state(link));
Signature · 1 line
2.42.0
Signature · 2 lines
Sathyanarayanan Kuppuswamy
Linux Kernel Developer
#3 lore raw
Quoted 15 lines · Kuppuswamy Sathyanarayanan
Quoted 11 lines · Bjorn Helgaas

[+cc Sathy, Lukas]

Quoted 8 lines · Heiner Kallweit

After the referenced commit we may see L1 sub-states being active unexpectedly. Following scenario as an example:
r8169 disables L1 because of known hardware issues on a number of systems. Implicitly L1.1 and L1.2 are disabled too.
On my system L1 and L1.1 work fine, but L1.2 causes missed rx packets. Therefore I write 1 to aspm_l1_1.
This removes ASPM_STATE_L1 from the disabled modes and therefore unexpectedly enables also L1.2. So return to the old behavior.

Quoted 3 lines

IIUC, the above-mentioned SysFS issue will be fixed by your change to aspm_attr_store_common(), right?

In my use case the driver uses pci_disable_link_state() to disable L1 and implicitly also L1.1 and L1.2. The following change is needed so that L1.2 remains disabled if L1 is removed from the list of disabled states later. That's the revert we're talking about.

Quoted 73 lines

Can you elaborate why you need the following change?

Quoted 6 lines
Quoted 6 lines
drivers/pci/pcie/aspm.c +1 -1
@@ -1063,7 +1063,7 @@ static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)
 	if (state & PCIE_LINK_STATE_L1)
-		link->aspm_disable |= ASPM_STATE_L1;
+		link->aspm_disable |= ASPM_STATE_L1 | ASPM_STATE_L1SS;
 
 
Quoted 8 lines
Quoted 3 lines

A comment in the commit message of the referenced change correctly points out that this behavior is inconsistent with aspm_attr_store_common(). So change aspm_attr_store_common() accordingly.

I think we should split this into a pure revert of fb097dcd5a28 with the description of the unintended consequence, followed by another patch to change aspm_attr_store_common().

I agree, the revert and new change should be split into two patches.

Quoted 53 lines

I guess the existing aspm_attr_store_common() behavior allows a similar unexpected behavior? For example, in this sequence:

- Write 0 to "l1_aspm" to disable L1
- Write 1 to "l1_1_aspm" to enable L1.1

does L1.2 get implicitly enabled as well even though that's clearly not what the user intended?

There's also the separate question of how the sysfs file and the pci_disable_link_state() API should interact. Drivers use that API when they know about a defect in their device, but the user can override that via syfs.

In [1], we have a similar situation with D3cold support, where we're thinking that we should not allow a user to use sysfs to override that driver knowledge.

Bjorn

[1] https://lore.kernel.org/r/b8a7f4af2b73f6b506ad8ddee59d747cbf834606.1695025365.git.lukas@wunner.de

Quoted 31 lines
Fixes fb097dcd5a28 ("PCI/ASPM: Disable only ASPM_STATE_L1 when driver disables L1")
---
 drivers/pci/pcie/aspm.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/pci/pcie/aspm.c +3 -1
index 3dafba0b5..6d3788257 100644
@@ -1063,7 +1063,7 @@ static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)
 	if (state & PCIE_LINK_STATE_L0S)
 		link->aspm_disable |= ASPM_STATE_L0S;
 	if (state & PCIE_LINK_STATE_L1)
-		link->aspm_disable |= ASPM_STATE_L1;
+		link->aspm_disable |= ASPM_STATE_L1 | ASPM_STATE_L1SS;
 	if (state & PCIE_LINK_STATE_L1_1)
 		link->aspm_disable |= ASPM_STATE_L1_1;
 	if (state & PCIE_LINK_STATE_L1_2)
@@ -1251,6 +1251,8 @@ static ssize_t aspm_attr_store_common(struct device *dev,
 			link->aspm_disable &= ~ASPM_STATE_L1;
 	} else {
 		link->aspm_disable |= state;
+		if (state & ASPM_STATE_L1)
+			link->aspm_disable |= ASPM_STATE_L1SS;
 	}
 
 	pcie_config_aspm_link(link, policy_to_aspm_state(link));
Signature · 1 line
2.42.0
#4 lore raw
In reply to Bjorn Helgaas
Quoted 12 lines · Bjorn Helgaas

[+cc Sathy, Lukas]

Quoted 9 lines · Heiner Kallweit

After the referenced commit we may see L1 sub-states being active unexpectedly. Following scenario as an example:
r8169 disables L1 because of known hardware issues on a number of systems. Implicitly L1.1 and L1.2 are disabled too.
On my system L1 and L1.1 work fine, but L1.2 causes missed rx packets. Therefore I write 1 to aspm_l1_1.
This removes ASPM_STATE_L1 from the disabled modes and therefore unexpectedly enables also L1.2. So return to the old behavior.

Quoted 8 lines
Quoted 3 lines

A comment in the commit message of the referenced change correctly points out that this behavior is inconsistent with aspm_attr_store_common(). So change aspm_attr_store_common() accordingly.

I think we should split this into a pure revert of fb097dcd5a28 with the description of the unintended consequence, followed by another patch to change aspm_attr_store_common().

OK

Quoted 9 lines

I guess the existing aspm_attr_store_common() behavior allows a similar unexpected behavior? For example, in this sequence:

- Write 0 to "l1_aspm" to disable L1
- Write 1 to "l1_1_aspm" to enable L1.1

does L1.2 get implicitly enabled as well even though that's clearly not what the user intended?

Right, it's the same here. Therefore the second change in the patch.

Quoted 9 lines

There's also the separate question of how the sysfs file and the pci_disable_link_state() API should interact. Drivers use that API when they know about a defect in their device, but the user can override that via syfs.

In [1], we have a similar situation with D3cold support, where we're thinking that we should not allow a user to use sysfs to override that driver knowledge.

In my r8169 use case ASPM works fine on one system, but causes issues on another one with same NIC chip version. So it may be a wild mix of NIC hw erratum, BIOS bug, mainboard chipset incompatibilities, etc. Therefore I disable L1 per default, and allow users to re-enable L1/L1.1/L1.2 if their system isn't affected.

Quoted 35 lines

Bjorn

[1] https://lore.kernel.org/r/b8a7f4af2b73f6b506ad8ddee59d747cbf834606.1695025365.git.lukas@wunner.de

Quoted 31 lines
Fixes fb097dcd5a28 ("PCI/ASPM: Disable only ASPM_STATE_L1 when driver disables L1")
---
 drivers/pci/pcie/aspm.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/pci/pcie/aspm.c +3 -1
index 3dafba0b5..6d3788257 100644
@@ -1063,7 +1063,7 @@ static int __pci_disable_link_state(struct pci_dev *pdev, int state, bool sem)
 	if (state & PCIE_LINK_STATE_L0S)
 		link->aspm_disable |= ASPM_STATE_L0S;
 	if (state & PCIE_LINK_STATE_L1)
-		link->aspm_disable |= ASPM_STATE_L1;
+		link->aspm_disable |= ASPM_STATE_L1 | ASPM_STATE_L1SS;
 	if (state & PCIE_LINK_STATE_L1_1)
 		link->aspm_disable |= ASPM_STATE_L1_1;
 	if (state & PCIE_LINK_STATE_L1_2)
@@ -1251,6 +1251,8 @@ static ssize_t aspm_attr_store_common(struct device *dev,
 			link->aspm_disable &= ~ASPM_STATE_L1;
 	} else {
 		link->aspm_disable |= state;
+		if (state & ASPM_STATE_L1)
+			link->aspm_disable |= ASPM_STATE_L1SS;
 	}
 
 	pcie_config_aspm_link(link, policy_to_aspm_state(link));
Signature · 1 line
2.42.0