Skip to contentSkip to conversation

PCI/ASPM: Don't reconfigure ASPM entering low-power state

PATCH Applied Carlos Bilbao (Lambda) 3 participants 6 messages Apr 28 – Sep 18 PCI

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 PCI/ASPM: Don't reconfigure ASPM entering low-power state

From: Carlos Bilbao <[email protected]>

Reconfiguring ASPM when a device transitions to low-power state can enable L1.1/L1.2 substates on the PCIe link at a time when the device is sleeping and may be unable to exit them. ASPM should be reconfigured on D0 entry (resume), not on the way down.

pci_set_low_power_state() calls pcie_aspm_pm_state_change() after writing D3hot to PCI_PM_CTRL. pcie_aspm_pm_state_change() resets link->aspm_capable to link->aspm_support and then calls pcie_config_aspm_path(), which can enable ASPM L1.1/L1.2 substates on the PCIe link. If the device cannot recover the link from L1.2 while in D3hot, subsequent config space reads return 0xFFFF ("device inaccessible") and pci_power_up() fails with message "Unable to change power state from D3hot to D0, device inaccessible".

This was observed on NVIDIA H100 SXM5 GPUs bound to vfio-pci when Linux runtime PM suspends them to D3hot: the GPU becomes permanently inaccessible and disappears from the PCIe bus.

The call to pcie_aspm_pm_state_change() in pci_set_low_power_state() was restored by commit f93e71aea6c6 ("Revert "PCI/ASPM: Remove pcie_aspm_pm_state_change()""), which reverted
commit 08d0cc5f3426 ("PCI/ASPM: Remove pcie_aspm_pm_state_change()"). The revert was necessary because the
removal broke suspend/resume on certain platforms that required ASPM to be reconfigured on D0 entry. However, the revert restored the call in both pci_set_full_power_state() (D0 entry) and pci_set_low_power_state() (low-power entry).

Only the D0-entry call is needed to fix the suspend/resume regression. The low-power-entry call is harmful: reconfiguring ASPM immediately after putting a device into D3hot can enable link substates that the device or platform cannot exit while the device is sleeping.

Remove the pcie_aspm_pm_state_change() call from pci_set_low_power_state(). ASPM will still be reconfigured correctly when the device returns to D0 via pci_set_full_power_state().

Fixes f93e71aea6c6 ("Revert "PCI/ASPM: Remove pcie_aspm_pm_state_change()"")
---
 drivers/pci/pci.c | 3 ---
 1 file changed, 3 deletions(-)
drivers/pci/pci.c +0 -3
index b2ccb8e122f2..8b47887019f9 100644
@@ -1542,9 +1542,6 @@ static int pci_set_low_power_state(struct pci_dev *dev, pci_power_t state, bool
 				     pci_power_name(dev->current_state),
 				     pci_power_name(state));
 
-	if (dev->bus->self)
-		pcie_aspm_pm_state_change(dev->bus->self, locked);
-
 	return 0;
 }
 
Signature · 1 line
2.50.1 (Apple Git-155)
#1 lore raw

[+cc Kai-Heng, Michael, David, Mani]

Quoted 29 lines · Carlos Bilbao (Lambda)

From: Carlos Bilbao <[email protected]>

Reconfiguring ASPM when a device transitions to low-power state can enable L1.1/L1.2 substates on the PCIe link at a time when the device is sleeping and may be unable to exit them. ASPM should be reconfigured on D0 entry (resume), not on the way down.

pci_set_low_power_state() calls pcie_aspm_pm_state_change() after writing D3hot to PCI_PM_CTRL. pcie_aspm_pm_state_change() resets link->aspm_capable to link->aspm_support and then calls pcie_config_aspm_path(), which can enable ASPM L1.1/L1.2 substates on the PCIe link. If the device cannot recover the link from L1.2 while in D3hot, subsequent config space reads return 0xFFFF ("device inaccessible") and pci_power_up() fails with message "Unable to change power state from D3hot to D0, device inaccessible".

This was observed on NVIDIA H100 SXM5 GPUs bound to vfio-pci when Linux runtime PM suspends them to D3hot: the GPU becomes permanently inaccessible and disappears from the PCIe bus.

The call to pcie_aspm_pm_state_change() in pci_set_low_power_state() was restored by commit f93e71aea6c6 ("Revert "PCI/ASPM: Remove pcie_aspm_pm_state_change()""), which reverted
commit 08d0cc5f3426 ("PCI/ASPM: Remove pcie_aspm_pm_state_change()"). The revert was necessary because the
removal broke suspend/resume on certain platforms that required ASPM to be reconfigured on D0 entry. However, the revert restored the call in both pci_set_full_power_state() (D0 entry) and pci_set_low_power_state() (low-power entry).

Quoted 8 lines

Only the D0-entry call is needed to fix the suspend/resume regression. The low-power-entry call is harmful: reconfiguring ASPM immediately after putting a device into D3hot can enable link substates that the device or platform cannot exit while the device is sleeping.

Remove the pcie_aspm_pm_state_change() call from pci_set_low_power_state(). ASPM will still be reconfigured correctly when the device returns to D0 via pci_set_full_power_state().

Sounds right to me. I don't know why we would want to touch ASPM during suspend to D3hot. I could imagine disabling ASPM states *before* that transition, but enabling new states at that point sounds wrong.

Any comments, Kai-Heng and Michael?

I know your regression report was long ago, Michael,
(https://lore.kernel.org/all/[email protected]/),
so likely not practical for you to test this change, but I would hate to have your system break again.

Quoted 24 lines
Fixes f93e71aea6c6 ("Revert "PCI/ASPM: Remove pcie_aspm_pm_state_change()"")
---
 drivers/pci/pci.c | 3 ---
 1 file changed, 3 deletions(-)
drivers/pci/pci.c +0 -3
index b2ccb8e122f2..8b47887019f9 100644
@@ -1542,9 +1542,6 @@ static int pci_set_low_power_state(struct pci_dev *dev, pci_power_t state, bool
 				     pci_power_name(dev->current_state),
 				     pci_power_name(state));
 
-	if (dev->bus->self)
-		pcie_aspm_pm_state_change(dev->bus->self, locked);
-
 	return 0;
 }
 
Signature · 1 line
2.50.1 (Apple Git-155)
#2 lore raw
In reply to Carlos Bilbao (Lambda)
Quoted 7 lines · Carlos Bilbao (Lambda)

From: Carlos Bilbao <[email protected]>

Reconfiguring ASPM when a device transitions to low-power state can enable L1.1/L1.2 substates on the PCIe link at a time when the device is sleeping and may be unable to exit them. ASPM should be reconfigured on D0 entry (resume), not on the way down.

Quoted 7 lines

pci_set_low_power_state() calls pcie_aspm_pm_state_change() after writing D3hot to PCI_PM_CTRL. pcie_aspm_pm_state_change() resets link->aspm_capable to link->aspm_support and then calls pcie_config_aspm_path(), which can enable ASPM L1.1/L1.2 substates on the PCIe link. If the device cannot recover the link from L1.2 while in D3hot, subsequent config space reads return 0xFFFF ("device inaccessible") and pci_power_up() fails with message "Unable to change power state from D3hot to D0, device inaccessible".

Carlos, do you have a few lines of dmesg showing this issue that we could quote to help people match the issue with this fix?

#3 lore raw

Hey Bjorn.

Quoted 15 lines · Bjorn Helgaas
Quoted 7 lines · Carlos Bilbao (Lambda)

From: Carlos Bilbao <[email protected]>

Reconfiguring ASPM when a device transitions to low-power state can enable L1.1/L1.2 substates on the PCIe link at a time when the device is sleeping and may be unable to exit them. ASPM should be reconfigured on D0 entry (resume), not on the way down.

Quoted 7 lines

pci_set_low_power_state() calls pcie_aspm_pm_state_change() after writing D3hot to PCI_PM_CTRL. pcie_aspm_pm_state_change() resets link->aspm_capable to link->aspm_support and then calls pcie_config_aspm_path(), which can enable ASPM L1.1/L1.2 substates on the PCIe link. If the device cannot recover the link from L1.2 while in D3hot, subsequent config space reads return 0xFFFF ("device inaccessible") and pci_power_up() fails with message "Unable to change power state from D3hot to D0, device inaccessible".

Quoted 2 lines

Carlos, do you have a few lines of dmesg showing this issue that we could quote to help people match the issue with this fix?

Thank you for reviewing this. Only the error message:

[160459.607156] vfio-pci 0000:5d:00.0: Unable to change power state from D3cold to D0, device inaccessible

Thanks,

Carlos

#4 lore raw
In reply to Carlos Bilbao (Lambda)
Quoted 34 lines · Carlos Bilbao (Lambda)

From: Carlos Bilbao <[email protected]>

Reconfiguring ASPM when a device transitions to low-power state can enable L1.1/L1.2 substates on the PCIe link at a time when the device is sleeping and may be unable to exit them. ASPM should be reconfigured on D0 entry (resume), not on the way down.

pci_set_low_power_state() calls pcie_aspm_pm_state_change() after writing D3hot to PCI_PM_CTRL. pcie_aspm_pm_state_change() resets link->aspm_capable to link->aspm_support and then calls pcie_config_aspm_path(), which can enable ASPM L1.1/L1.2 substates on the PCIe link. If the device cannot recover the link from L1.2 while in D3hot, subsequent config space reads return 0xFFFF ("device inaccessible") and pci_power_up() fails with message "Unable to change power state from D3hot to D0, device inaccessible".

This was observed on NVIDIA H100 SXM5 GPUs bound to vfio-pci when Linux runtime PM suspends them to D3hot: the GPU becomes permanently inaccessible and disappears from the PCIe bus.

The call to pcie_aspm_pm_state_change() in pci_set_low_power_state() was restored by commit f93e71aea6c6 ("Revert "PCI/ASPM: Remove pcie_aspm_pm_state_change()""), which reverted
commit 08d0cc5f3426 ("PCI/ASPM: Remove pcie_aspm_pm_state_change()"). The revert was necessary because the
removal broke suspend/resume on certain platforms that required ASPM to be reconfigured on D0 entry. However, the revert restored the call in both pci_set_full_power_state() (D0 entry) and pci_set_low_power_state() (low-power entry).

Only the D0-entry call is needed to fix the suspend/resume regression. The low-power-entry call is harmful: reconfiguring ASPM immediately after putting a device into D3hot can enable link substates that the device or platform cannot exit while the device is sleeping.

Quoted 7 lines

Remove the pcie_aspm_pm_state_change() call from pci_set_low_power_state(). ASPM will still be reconfigured correctly when the device returns to D0 via pci_set_full_power_state().

Fixes f93e71aea6c6 ("Revert "PCI/ASPM: Remove pcie_aspm_pm_state_change()"")

Applied to pci/aspm for v7.2, thanks!

Quoted 21 lines
---
 drivers/pci/pci.c | 3 ---
 1 file changed, 3 deletions(-)
drivers/pci/pci.c +0 -3
index b2ccb8e122f2..8b47887019f9 100644
@@ -1542,9 +1542,6 @@ static int pci_set_low_power_state(struct pci_dev *dev, pci_power_t state, bool
 				     pci_power_name(dev->current_state),
 				     pci_power_name(state));
 
-	if (dev->bus->self)
-		pcie_aspm_pm_state_change(dev->bus->self, locked);
-
 	return 0;
 }
 
Signature · 1 line
2.50.1 (Apple Git-155)
#5 lore raw
Quoted 57 lines · Bjorn Helgaas
Quoted 34 lines · Carlos Bilbao (Lambda)

From: Carlos Bilbao <[email protected]>

Reconfiguring ASPM when a device transitions to low-power state can enable L1.1/L1.2 substates on the PCIe link at a time when the device is sleeping and may be unable to exit them. ASPM should be reconfigured on D0 entry (resume), not on the way down.

pci_set_low_power_state() calls pcie_aspm_pm_state_change() after writing D3hot to PCI_PM_CTRL. pcie_aspm_pm_state_change() resets link->aspm_capable to link->aspm_support and then calls pcie_config_aspm_path(), which can enable ASPM L1.1/L1.2 substates on the PCIe link. If the device cannot recover the link from L1.2 while in D3hot, subsequent config space reads return 0xFFFF ("device inaccessible") and pci_power_up() fails with message "Unable to change power state from D3hot to D0, device inaccessible".

This was observed on NVIDIA H100 SXM5 GPUs bound to vfio-pci when Linux runtime PM suspends them to D3hot: the GPU becomes permanently inaccessible and disappears from the PCIe bus.

The call to pcie_aspm_pm_state_change() in pci_set_low_power_state() was restored by commit f93e71aea6c6 ("Revert "PCI/ASPM: Remove pcie_aspm_pm_state_change()""), which reverted
commit 08d0cc5f3426 ("PCI/ASPM: Remove pcie_aspm_pm_state_change()"). The revert was necessary because the
removal broke suspend/resume on certain platforms that required ASPM to be reconfigured on D0 entry. However, the revert restored the call in both pci_set_full_power_state() (D0 entry) and pci_set_low_power_state() (low-power entry).

Only the D0-entry call is needed to fix the suspend/resume regression. The low-power-entry call is harmful: reconfiguring ASPM immediately after putting a device into D3hot can enable link substates that the device or platform cannot exit while the device is sleeping.

Quoted 7 lines

Remove the pcie_aspm_pm_state_change() call from pci_set_low_power_state(). ASPM will still be reconfigured correctly when the device returns to D0 via pci_set_full_power_state().

Fixes f93e71aea6c6 ("Revert "PCI/ASPM: Remove pcie_aspm_pm_state_change()"")

Applied to pci/aspm for v7.2, thanks!

Quoted 12 lines
---
 drivers/pci/pci.c | 3 ---
 1 file changed, 3 deletions(-)
drivers/pci/pci.c +0 -0
index b2ccb8e122f2..8b47887019f9 100644
@@ -1542,9 +1542,6 @@ static int pci_set_low_power_state(struct pci_dev *dev, pci_power_t state, bool
 				     pci_power_name(dev->current_state),
 				     pci_power_name(state));
 
Quoted 2 lines

drivers/pci/pci.c static int pci_set_low_power_state(struct pci_dev *dev, pci_power_t state, bool · line 1546

Quoted 2 lines
drivers/pci/pci.c +0 -2
-	if (dev->bus->self)
-		pcie_aspm_pm_state_change(dev->bus->self, locked);

While backporting this, Sashiko noticed that "locked" is no longer used and could possibly be removed from the set_low_power paths. I guess ASPM needs "locked" because it holds pci_bus_sem while evaluating all the devices in the path to the device, but since we're not touching ASPM in the set_low_power path, we don't need that.

Quoted 6 lines
Quoted 6 lines
drivers/pci/pci.c +0 -1
 	return 0;
 }
 
-- 

2.50.1 (Apple Git-155)