Skip to contentSkip to conversation

PCI: Disable PTM on Upstream Ports during suspend

PATCH In review Bjorn Helgaas 3 participants 4 messages Aug 30, 2022 – Aug 31, 2022 PCI

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 PCI: Disable PTM on Upstream Ports during suspend

From: Bjorn Helgaas <[email protected]>

During suspend, we want to disable PTM on Root Ports because that allows some chips, e.g., Intel mobile chips since Coffee Lake, to enter a lower-power PM state.

Previously we only disabled PTM for Root Ports, but PCIe r6.0, sec 2.2.8, strongly recommends that functions support generation of Messages in non-D0 states, so we must assume that Switch Upstream Ports or Endpoints may send PTM Request Messages while in D1, D2, and D3hot.

A PTM Message received by a Downstream Port, e.g., a Root Port, with PTM disabled must be treated as an Unsupported Request (sec 6.21.3).

With this topology:

0000:00:1d.0 Root Port to [bus 08-71]
0000:08:00.0 Switch Upstream Port to [bus 09-71]

Kai-Heng reported errors like this:

pcieport 0000:00:1d.0: AER: Uncorrected (Non-Fatal) error received: 0000:00:1d.0
pcieport 0000:00:1d.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)
pcieport 0000:00:1d.0: device [8086:7ab0] error status/mask=00100000/00004000
pcieport 0000:00:1d.0: [20] UnsupReq (First)
pcieport 0000:00:1d.0: AER: TLP Header: 34000000 08000052 00000000 00000000

Decoding this (from PCIe r6.0, sec 2.2.1.1, 2.2.8.10) shows that 00:1d.0 logged an Unsupported Request error when it received a PTM Request with Requester ID 08:00.0.

To prevent this error, disable PTM when suspending Upstream Ports (including those on Endpoints), not just Root Ports.

Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=215453
Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=216210
Based-on: https://lore.kernel.org/r/[email protected]

---
 drivers/pci/pci.c      | 30 ++++++++++++++----------------
 drivers/pci/pcie/ptm.c |  8 +++++++-
 2 files changed, 21 insertions(+), 17 deletions(-)
drivers/pci/pci.c +14 -16
index 95bc329e74c0..96487a9ce5bf 100644
@@ -2707,14 +2707,19 @@ int pci_prepare_to_sleep(struct pci_dev *dev)
 		return -EIO;
 
 	/*
-	 * There are systems (for example, Intel mobile chips since Coffee
-	 * Lake) where the power drawn while suspended can be significantly
-	 * reduced by disabling PTM on PCIe root ports as this allows the
-	 * port to enter a lower-power PM state and the SoC to reach a
-	 * lower-power idle state as a whole.
+	 * We want to disable PTM on Root Ports because that allows some
+	 * chips, e.g., Intel mobile chips since Coffee Lake, to enter a
+	 * lower-power PM state.
+	 *
+	 * PCIe r6.0, sec 2.2.8, strongly recommends that functions support
+	 * generation of messages in non-D0 states, so we assume Switch
+	 * Upstream Ports or Endpoints may send PTM Requests while in D1,
+	 * D2, and D3hot.  A PTM message received by a Downstream Port
+	 * (including a Root Port) with PTM disabled must be treated as an
+	 * Unsupported Request (sec 6.21.3).  To prevent this error,
+	 * disable PTM in *all* devices, not just Root Ports.
 	 */
-	if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT)
-		pci_disable_ptm(dev);
+	pci_disable_ptm(dev);
 
 	pci_enable_wake(dev, target_state, wakeup);
 
@@ -2764,15 +2769,8 @@ int pci_finish_runtime_suspend(struct pci_dev *dev)
 	if (target_state == PCI_POWER_ERROR)
 		return -EIO;
 
-	/*
-	 * There are systems (for example, Intel mobile chips since Coffee
-	 * Lake) where the power drawn while suspended can be significantly
-	 * reduced by disabling PTM on PCIe root ports as this allows the
-	 * port to enter a lower-power PM state and the SoC to reach a
-	 * lower-power idle state as a whole.
-	 */
-	if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT)
-		pci_disable_ptm(dev);
+	/* See rationale above for disabling PTM */
+	pci_disable_ptm(dev);
 
 	__pci_enable_wake(dev, target_state, pci_dev_run_wake(dev));
 
drivers/pci/pcie/ptm.c +7 -1
index 368a254e3124..ec338470d13f 100644
@@ -31,12 +31,18 @@ static void pci_ptm_info(struct pci_dev *dev)
 
 void pci_disable_ptm(struct pci_dev *dev)
 {
-	int ptm;
+	int type, ptm;
 	u16 ctrl;
 
 	if (!pci_is_pcie(dev))
 		return;
 
+	type = pci_pcie_type(dev);
+	if (!(type == PCI_EXP_TYPE_ROOT_PORT ||
+	      type == PCI_EXP_TYPE_UPSTREAM ||
+	      type == PCI_EXP_TYPE_ENDPOINT))
+		return;
+
 	ptm = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_PTM);
 	if (!ptm)
 		return;
Signature · 1 line
2.25.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 37 lines · Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

During suspend, we want to disable PTM on Root Ports because that allows some chips, e.g., Intel mobile chips since Coffee Lake, to enter a lower-power PM state.

Previously we only disabled PTM for Root Ports, but PCIe r6.0, sec 2.2.8, strongly recommends that functions support generation of Messages in non-D0 states, so we must assume that Switch Upstream Ports or Endpoints may send PTM Request Messages while in D1, D2, and D3hot.

A PTM Message received by a Downstream Port, e.g., a Root Port, with PTM disabled must be treated as an Unsupported Request (sec 6.21.3).

With this topology:

0000:00:1d.0 Root Port to [bus 08-71]
0000:08:00.0 Switch Upstream Port to [bus 09-71]

Kai-Heng reported errors like this:

pcieport 0000:00:1d.0: AER: Uncorrected (Non-Fatal) error received: 0000:00:1d.0
pcieport 0000:00:1d.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)
pcieport 0000:00:1d.0: device [8086:7ab0] error status/mask=00100000/00004000
pcieport 0000:00:1d.0: [20] UnsupReq (First)
pcieport 0000:00:1d.0: AER: TLP Header: 34000000 08000052 00000000 00000000

Decoding this (from PCIe r6.0, sec 2.2.1.1, 2.2.8.10) shows that 00:1d.0 logged an Unsupported Request error when it received a PTM Request with Requester ID 08:00.0.

To prevent this error, disable PTM when suspending Upstream Ports (including those on Endpoints), not just Root Ports.

Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=215453
Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=216210
Based-on: https://lore.kernel.org/r/[email protected]

Quoted 7 lines

---

Looks good to me

Quoted 77 lines
 drivers/pci/pci.c      | 30 ++++++++++++++----------------
 drivers/pci/pcie/ptm.c |  8 +++++++-
 2 files changed, 21 insertions(+), 17 deletions(-)
drivers/pci/pci.c +14 -16
index 95bc329e74c0..96487a9ce5bf 100644
@@ -2707,14 +2707,19 @@ int pci_prepare_to_sleep(struct pci_dev *dev)
 		return -EIO;
 
 	/*
-	 * There are systems (for example, Intel mobile chips since Coffee
-	 * Lake) where the power drawn while suspended can be significantly
-	 * reduced by disabling PTM on PCIe root ports as this allows the
-	 * port to enter a lower-power PM state and the SoC to reach a
-	 * lower-power idle state as a whole.
+	 * We want to disable PTM on Root Ports because that allows some
+	 * chips, e.g., Intel mobile chips since Coffee Lake, to enter a
+	 * lower-power PM state.
+	 *
+	 * PCIe r6.0, sec 2.2.8, strongly recommends that functions support
+	 * generation of messages in non-D0 states, so we assume Switch
+	 * Upstream Ports or Endpoints may send PTM Requests while in D1,
+	 * D2, and D3hot.  A PTM message received by a Downstream Port
+	 * (including a Root Port) with PTM disabled must be treated as an
+	 * Unsupported Request (sec 6.21.3).  To prevent this error,
+	 * disable PTM in *all* devices, not just Root Ports.
 	 */
-	if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT)
-		pci_disable_ptm(dev);
+	pci_disable_ptm(dev);
 
 	pci_enable_wake(dev, target_state, wakeup);
 
@@ -2764,15 +2769,8 @@ int pci_finish_runtime_suspend(struct pci_dev *dev)
 	if (target_state == PCI_POWER_ERROR)
 		return -EIO;
 
-	/*
-	 * There are systems (for example, Intel mobile chips since Coffee
-	 * Lake) where the power drawn while suspended can be significantly
-	 * reduced by disabling PTM on PCIe root ports as this allows the
-	 * port to enter a lower-power PM state and the SoC to reach a
-	 * lower-power idle state as a whole.
-	 */
-	if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT)
-		pci_disable_ptm(dev);
+	/* See rationale above for disabling PTM */
+	pci_disable_ptm(dev);
 
 	__pci_enable_wake(dev, target_state, pci_dev_run_wake(dev));
 
drivers/pci/pcie/ptm.c +7 -1
index 368a254e3124..ec338470d13f 100644
@@ -31,12 +31,18 @@ static void pci_ptm_info(struct pci_dev *dev)
 
 void pci_disable_ptm(struct pci_dev *dev)
 {
-	int ptm;
+	int type, ptm;
 	u16 ctrl;
 
 	if (!pci_is_pcie(dev))
 		return;
 
+	type = pci_pcie_type(dev);
+	if (!(type == PCI_EXP_TYPE_ROOT_PORT ||
+	      type == PCI_EXP_TYPE_UPSTREAM ||
+	      type == PCI_EXP_TYPE_ENDPOINT))
+		return;
+
 	ptm = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_PTM);
 	if (!ptm)
 		return;
Signature · 2 lines
Sathyanarayanan Kuppuswamy
Linux Kernel Developer
Reviewed-by on v1 1/1
#2 lore raw
In reply to Bjorn Helgaas

Hi Bjorn,

drivers/pci/pcie/ptm.c static void pci_ptm_info(struct pci_dev *dev) · line 44

Quoted 5 lines · Bjorn Helgaas
drivers/pci/pcie/ptm.c +5 -0
+	type = pci_pcie_type(dev);
+	if (!(type == PCI_EXP_TYPE_ROOT_PORT ||
+	      type == PCI_EXP_TYPE_UPSTREAM ||
+	      type == PCI_EXP_TYPE_ENDPOINT))
+		return;

Perhaps switch () instead?

switch (pci_pcie_type(dev)) {
case PCI_EXP_TYPE_ROOT_PORT:
case PCI_EXP_TYPE_UPSTREAM:
case PCI_EXP_TYPE_ENDPOINT:

	break;
default:
	return;
}

Either way,

#3 lore raw

[+cc Rafael, Rajvi, linux-pm]

Quoted 13 lines · Mika Westerberg

drivers/pci/pcie/ptm.c static void pci_ptm_info(struct pci_dev *dev) · line 44

Quoted 5 lines · Bjorn Helgaas
drivers/pci/pcie/ptm.c +5 -0
+	type = pci_pcie_type(dev);
+	if (!(type == PCI_EXP_TYPE_ROOT_PORT ||
+	      type == PCI_EXP_TYPE_UPSTREAM ||
+	      type == PCI_EXP_TYPE_ENDPOINT))
+		return;

Perhaps switch () instead?

switch (pci_pcie_type(dev)) {
case PCI_EXP_TYPE_ROOT_PORT:
case PCI_EXP_TYPE_UPSTREAM:
case PCI_EXP_TYPE_ENDPOINT:

Quoted 8 lines
	break;
default:
	return;
}

Either way,

Thanks! I dropped this patch for now. I think Rajvi's similar patch [1] is much better because we only need this kind of check in one place -- where we enable PTM.

Also, we need to tighten this up so we *always* disable PTM [2]. The patch I posted still calls pci_disable_ptm() from pci_prepare_to_sleep(), which means we only do it when !pdev->state_saved, so we leave PTM enabled if the driver saves its own state.

Bjorn

[1] https://lore.kernel.org/r/[email protected]
[2] https://lore.kernel.org/r/CAJZ5v0iHckqia4OywKzSNWFCaq7eOkJcm5yXJdT2_sNdd36gDw@mail.gmail.com