Skip to contentSkip to conversation

PCI: Block power management of certain ports with slot implemented bit set

PATCH In review Mika Westerberg 4 participants 8 messages Jan 7, 2019 – Jan 8, 2019 PCI

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 PCI: Block power management of certain ports with slot implemented bit set

Gigabyte X299 DESIGNARE EX motherboard has one PCIe root port that is connected to Alpine Ridge Thunderbolt controller. This port has slot implemented bit set in the config space but other than that it is not hotplug capable in the sense we are expecting in Linux (it has dev->is_hotplug_bridge set to 0):

00:1c.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #5
Bus: primary=00, secondary=05, subordinate=46, sec-latency=0 Memory behind bridge: 78000000-8fffffff [size=384M]
Prefetchable memory behind bridge: 00003800f8000000-00003800ffffffff [size=128M]
...
Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
...
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #8, PowerLimit 25.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet+ LinkState+

This system is using ACPI based hotplug to notify the OS that it needs to rescan the PCI bus (ACPI hotplug).

If there is nothing connected to any of the Thunderbolt ports the root port will not have any runtime PM active children and is thus automatically runtime suspended pretty soon after boot by PCI PM core. Now, when a device is connected the BIOS SMI handler responsible for enumerating newly added devices is not able to find anything because the port is in D3.

For this reason we block power management of PCIe root and downstream ports that have slot implemented set and have node in ACPI namespace.

---
Changes from v1:

- Block PM for all root and downstream ports with slot implemented bit set and has an ACPI companion.

The previous version can be found here:

https://patchwork.kernel.org/patch/10711553/

 drivers/pci/pci.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)
drivers/pci/pci.c +17 -1
index c9d8e3c837de..04bdbcf1dfb7 100644
@@ -2510,10 +2510,14 @@ void pci_config_pm_runtime_put(struct pci_dev *pdev)
  */
 bool pci_bridge_d3_possible(struct pci_dev *bridge)
 {
+	int type;
+
 	if (!pci_is_pcie(bridge))
 		return false;
 
-	switch (pci_pcie_type(bridge)) {
+	type = pci_pcie_type(bridge);
+
+	switch (type) {
 	case PCI_EXP_TYPE_ROOT_PORT:
 	case PCI_EXP_TYPE_UPSTREAM:
 	case PCI_EXP_TYPE_DOWNSTREAM:
@@ -2546,6 +2550,18 @@ bool pci_bridge_d3_possible(struct pci_dev *bridge)
 		if (bridge->is_hotplug_bridge)
 			return false;
 
+		/*
+		 * Some systems such as Gigabyte X299 the root port is
+		 * not marked hotplug capable but ACPI based hotplug is
+		 * still used to bring in the Thunderbolt controller. To
+		 * make sure those ports do not enter D3 and possibly
+		 * confuse the BIOS SMI handler, block D3 for them.
+		 */
+		if (has_acpi_companion(&bridge->dev) &&
+		    type != PCI_EXP_TYPE_UPSTREAM &&
+		    pcie_caps_reg(bridge) & PCI_EXP_FLAGS_SLOT)
+			return false;
+
 		/*
 		 * It should be safe to put PCIe ports from 2015 or newer
 		 * to D3.
Signature · 1 line
2.19.2
Reviewed-by on v2 1/1
#1 lore raw
Quoted 31 lines · Mika Westerberg

Gigabyte X299 DESIGNARE EX motherboard has one PCIe root port that is connected to Alpine Ridge Thunderbolt controller. This port has slot implemented bit set in the config space but other than that it is not hotplug capable in the sense we are expecting in Linux (it has dev->is_hotplug_bridge set to 0):

00:1c.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #5
Bus: primary=00, secondary=05, subordinate=46, sec-latency=0 Memory behind bridge: 78000000-8fffffff [size=384M]
Prefetchable memory behind bridge: 00003800f8000000-00003800ffffffff [size=128M]
...
Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
...
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #8, PowerLimit 25.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet+ LinkState+

This system is using ACPI based hotplug to notify the OS that it needs to rescan the PCI bus (ACPI hotplug).

If there is nothing connected to any of the Thunderbolt ports the root port will not have any runtime PM active children and is thus automatically runtime suspended pretty soon after boot by PCI PM core. Now, when a device is connected the BIOS SMI handler responsible for enumerating newly added devices is not able to find anything because the port is in D3.

Quoted 6 lines

For this reason we block power management of PCIe root and downstream ports that have slot implemented set and have node in ACPI namespace.

Quoted 55 lines

---
Changes from v1:

- Block PM for all root and downstream ports with slot implemented bit set and has an ACPI companion.

The previous version can be found here:

https://patchwork.kernel.org/patch/10711553/

 drivers/pci/pci.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)
drivers/pci/pci.c +17 -2
index c9d8e3c837de..04bdbcf1dfb7 100644
@@ -2510,10 +2510,14 @@ void pci_config_pm_runtime_put(struct pci_dev *pdev)
  */
 bool pci_bridge_d3_possible(struct pci_dev *bridge)
 {
+       int type;
+
        if (!pci_is_pcie(bridge))
                return false;
 
-       switch (pci_pcie_type(bridge)) {
+       type = pci_pcie_type(bridge);
+
+       switch (type) {
        case PCI_EXP_TYPE_ROOT_PORT:
        case PCI_EXP_TYPE_UPSTREAM:
        case PCI_EXP_TYPE_DOWNSTREAM:
@@ -2546,6 +2550,18 @@ bool pci_bridge_d3_possible(struct pci_dev *bridge)
                if (bridge->is_hotplug_bridge)
                        return false;
 
+               /*
+                * Some systems such as Gigabyte X299 the root port is
+                * not marked hotplug capable but ACPI based hotplug is
+                * still used to bring in the Thunderbolt controller. To
+                * make sure those ports do not enter D3 and possibly
+                * confuse the BIOS SMI handler, block D3 for them.
+                */
+               if (has_acpi_companion(&bridge->dev) &&
+                   type != PCI_EXP_TYPE_UPSTREAM &&
+                   pcie_caps_reg(bridge) & PCI_EXP_FLAGS_SLOT)
+                       return false;
+
                /*
                 * It should be safe to put PCIe ports from 2015 or newer
                 * to D3.
--

2.19.2

#2 lore raw
Quoted 33 lines · Rafael J. Wysocki
Quoted 31 lines · Mika Westerberg

Gigabyte X299 DESIGNARE EX motherboard has one PCIe root port that is connected to Alpine Ridge Thunderbolt controller. This port has slot implemented bit set in the config space but other than that it is not hotplug capable in the sense we are expecting in Linux (it has dev->is_hotplug_bridge set to 0):

00:1c.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #5
Bus: primary=00, secondary=05, subordinate=46, sec-latency=0 Memory behind bridge: 78000000-8fffffff [size=384M]
Prefetchable memory behind bridge: 00003800f8000000-00003800ffffffff [size=128M]
...
Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
...
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #8, PowerLimit 25.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet+ LinkState+

This system is using ACPI based hotplug to notify the OS that it needs to rescan the PCI bus (ACPI hotplug).

If there is nothing connected to any of the Thunderbolt ports the root port will not have any runtime PM active children and is thus automatically runtime suspended pretty soon after boot by PCI PM core. Now, when a device is connected the BIOS SMI handler responsible for enumerating newly added devices is not able to find anything because the port is in D3.

Quoted 8 lines
Quoted 6 lines

For this reason we block power management of PCIe root and downstream ports that have slot implemented set and have node in ACPI namespace.

Thanks!

However, I'm having second toughts about this because I remembered that people put a lot of effort getting discrete graphics with power resource attached to the root port powering off properly. If the root port matches the criteria in this patch it will not be able to go into D3 anymore. It might affect others such as M.2 connected NVMe or WiFi chip as well. For that reason I would still prefer blacklist, at least for now.

#3 lore raw
Quoted 44 lines · Mika Westerberg
Quoted 33 lines · Rafael J. Wysocki
Quoted 31 lines · Mika Westerberg

Gigabyte X299 DESIGNARE EX motherboard has one PCIe root port that is connected to Alpine Ridge Thunderbolt controller. This port has slot implemented bit set in the config space but other than that it is not hotplug capable in the sense we are expecting in Linux (it has dev->is_hotplug_bridge set to 0):

00:1c.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #5
Bus: primary=00, secondary=05, subordinate=46, sec-latency=0 Memory behind bridge: 78000000-8fffffff [size=384M]
Prefetchable memory behind bridge: 00003800f8000000-00003800ffffffff [size=128M]
...
Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
...
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #8, PowerLimit 25.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet+ LinkState+

This system is using ACPI based hotplug to notify the OS that it needs to rescan the PCI bus (ACPI hotplug).

If there is nothing connected to any of the Thunderbolt ports the root port will not have any runtime PM active children and is thus automatically runtime suspended pretty soon after boot by PCI PM core. Now, when a device is connected the BIOS SMI handler responsible for enumerating newly added devices is not able to find anything because the port is in D3.

Quoted 8 lines
Quoted 6 lines

For this reason we block power management of PCIe root and downstream ports that have slot implemented set and have node in ACPI namespace.

Quoted 8 lines

Thanks!

However, I'm having second toughts about this because I remembered that people put a lot of effort getting discrete graphics with power resource attached to the root port powering off properly. If the root port matches the criteria in this patch it will not be able to go into D3 anymore. It might affect others such as M.2 connected NVMe or WiFi chip as well. For that reason I would still prefer blacklist, at least for now.

Blacklist would be fine by me too.

#4 lore raw
In reply to Mika Westerberg

Hi Peter,

Quoted 43 lines · Mika Westerberg
Quoted 33 lines · Rafael J. Wysocki
Quoted 31 lines · Mika Westerberg

Gigabyte X299 DESIGNARE EX motherboard has one PCIe root port that is connected to Alpine Ridge Thunderbolt controller. This port has slot implemented bit set in the config space but other than that it is not hotplug capable in the sense we are expecting in Linux (it has dev->is_hotplug_bridge set to 0):

00:1c.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #5
Bus: primary=00, secondary=05, subordinate=46, sec-latency=0 Memory behind bridge: 78000000-8fffffff [size=384M]
Prefetchable memory behind bridge: 00003800f8000000-00003800ffffffff [size=128M]
...
Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
...
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #8, PowerLimit 25.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet+ LinkState+

This system is using ACPI based hotplug to notify the OS that it needs to rescan the PCI bus (ACPI hotplug).

If there is nothing connected to any of the Thunderbolt ports the root port will not have any runtime PM active children and is thus automatically runtime suspended pretty soon after boot by PCI PM core. Now, when a device is connected the BIOS SMI handler responsible for enumerating newly added devices is not able to find anything because the port is in D3.

Quoted 8 lines
Quoted 6 lines

For this reason we block power management of PCIe root and downstream ports that have slot implemented set and have node in ACPI namespace.

Quoted 8 lines

Thanks!

However, I'm having second toughts about this because I remembered that people put a lot of effort getting discrete graphics with power resource attached to the root port powering off properly. If the root port matches the criteria in this patch it will not be able to go into D3 anymore. It might affect others such as M.2 connected NVMe or WiFi chip as well. For that reason I would still prefer blacklist, at least for now.

Would this patch:

    https://patchwork.ozlabs.org/patch/1021317/

break runtime D3cold for the discrete GPU on Optimus laptops such as your Clevo P651RA? Specifically, is the Root Port above the GPU marked "(Slot+)" in lspci -vv? (There doesn't seem to be raw lspci output in https://github.com/Lekensteyn/acpi-stuff)

Thanks,

Lukas

#5 lore raw

Hi Lukas,

Quoted 55 lines · Lukas Wunner

Hi Peter,

Quoted 43 lines · Mika Westerberg
Quoted 33 lines · Rafael J. Wysocki
Quoted 31 lines · Mika Westerberg

Gigabyte X299 DESIGNARE EX motherboard has one PCIe root port that is connected to Alpine Ridge Thunderbolt controller. This port has slot implemented bit set in the config space but other than that it is not hotplug capable in the sense we are expecting in Linux (it has dev->is_hotplug_bridge set to 0):

00:1c.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #5
Bus: primary=00, secondary=05, subordinate=46, sec-latency=0 Memory behind bridge: 78000000-8fffffff [size=384M]
Prefetchable memory behind bridge: 00003800f8000000-00003800ffffffff [size=128M]
...
Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
...
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #8, PowerLimit 25.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet+ LinkState+

This system is using ACPI based hotplug to notify the OS that it needs to rescan the PCI bus (ACPI hotplug).

If there is nothing connected to any of the Thunderbolt ports the root port will not have any runtime PM active children and is thus automatically runtime suspended pretty soon after boot by PCI PM core. Now, when a device is connected the BIOS SMI handler responsible for enumerating newly added devices is not able to find anything because the port is in D3.

Quoted 8 lines
Quoted 6 lines

For this reason we block power management of PCIe root and downstream ports that have slot implemented set and have node in ACPI namespace.

Quoted 8 lines

Thanks!

However, I'm having second toughts about this because I remembered that people put a lot of effort getting discrete graphics with power resource attached to the root port powering off properly. If the root port matches the criteria in this patch it will not be able to go into D3 anymore. It might affect others such as M.2 connected NVMe or WiFi chip as well. For that reason I would still prefer blacklist, at least for now.

Quoted 8 lines

Would this patch:

    https://patchwork.ozlabs.org/patch/1021317/

break runtime D3cold for the discrete GPU on Optimus laptops such as your Clevo P651RA? Specifically, is the Root Port above the GPU marked "(Slot+)" in lspci -vv? (There doesn't seem to be raw lspci output in https://github.com/Lekensteyn/acpi-stuff)

Thanks for bringing this into my attention. There are a couple of full lspci dumps, for example for the Dell XPS 9560.
https://github.com/Lekensteyn/acpi-stuff/blob/master/d3test/XPS9560/lspci-bare-metal.txt
This has upstream port 00:01.0 attached to GPU 01:00.0

    00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05) (prog-if 00 [Normal decode])
        ...
        Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00

and would indeed be negatively affected by this patch. I can observe the same for the Clevo P651RA (for which I can also send the full lspci dump if you need).

Signature · 3 lines
Kind regards,
Peter Wu
https://lekensteyn.nl
#6 lore raw
Quoted 14 lines · Peter Wu
Quoted 8 lines

Would this patch:

    https://patchwork.ozlabs.org/patch/1021317/

break runtime D3cold for the discrete GPU on Optimus laptops such as your Clevo P651RA? Specifically, is the Root Port above the GPU marked "(Slot+)" in lspci -vv? (There doesn't seem to be raw lspci output in https://github.com/Lekensteyn/acpi-stuff)

Thanks for bringing this into my attention. There are a couple of full lspci dumps, for example for the Dell XPS 9560.
https://github.com/Lekensteyn/acpi-stuff/blob/master/d3test/XPS9560/lspci-bare-metal.txt
This has upstream port 00:01.0 attached to GPU 01:00.0

Quoted 7 lines
    00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05) (prog-if 00 [Normal decode])
        ...
        Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00

and would indeed be negatively affected by this patch. I can observe the same for the Clevo P651RA (for which I can also send the full lspci dump if you need).

Thanks both.

I guess that leaves us the blacklist. I can re-send it rebased on top of v5.0-rc1 if there are no objections.

#7 lore raw
Quoted 2 lines · Mika Westerberg

I guess that leaves us the blacklist. I can re-send it rebased on top of v5.0-rc1 if there are no objections.

Please constrain either the blacklist entry for the Gigabyte mainboard or the blacklist check itself to x86 using IS_ENABLED() or #ifdef to avoid code bloat on other arches.

Thanks,

Lukas