Skip to contentSkip to conversation

PCI: Add pci=nobwctrl to disable the PCIe bandwidth controller

PATCH In review Guixin Liu 4 participants 8 messages Jul 29 – Sep 10

Subsystems (3)
lore
Share & tools
lore

Guixin Liu

author
#0 lore raw

cover PCI: Add pci=nobwctrl to disable the PCIe bandwidth controller

Enabling the PCIe bandwidth controller sets the Link Bandwidth Management Interrupt Enable (LBMIE) and Link Autonomous Bandwidth Interrupt Enable (LABIE) bits. On some (typically old) platforms this has been observed to cause boot hangs; commit 46a9f70e93ef ("PCI/bwctrl: Disable BW controller on Intel P45 using a quirk") works around one such platform with a per-device quirk.

A per-device quirk only helps once the offending device is identified and a quirk has shipped, which is awkward for a boot hang since the machine cannot be booted to report the culprit. The only existing escape hatch, pcie_ports=compat, is a blunt instrument that also disables AER, PME, hotplug and DPC.

This series adds a pci=nobwctrl boot option to disable just the bandwidth controller system-wide, giving users an immediate, rebuild-free workaround while a targeted quirk is worked out.

Patch 1 adds the pci_bwctrl_available()/pci_no_bwctrl() infrastructure (mirroring pci_aer_available()/pci_no_aer()) with no functional change. Patch 2 wires up the pci=nobwctrl parameter and documents it.

Tested on a QEMU q35 topology with PCIe ports and switches: without the parameter the bwctrl service devices are present as before; with pci=nobwctrl they all disappear while PME/AER/hotplug services stay intact and boot is clean.

Guixin Liu (2):
PCI/bwctrl: Add pci_bwctrl_available()
PCI: Add pci=nobwctrl parameter to disable the PCIe bandwidth
controller

 Documentation/admin-guide/kernel-parameters.txt |  6 ++++++
 drivers/pci/pci.c                               |  2 ++
 drivers/pci/pci.h                               |  4 ++++
 drivers/pci/pcie/bwctrl.c                       | 12 ++++++++++++
 drivers/pci/pcie/portdrv.c                      |  3 ++-
 5 files changed, 26 insertions(+), 1 deletion(-)
Signature · 1 line
2.43.7
author
#1 lore raw

PATCH 1/2 PCI/bwctrl: Add pci_bwctrl_available()

Enabling the PCIe bandwidth controller sets the Link Bandwidth Management Interrupt Enable (LBMIE) and Link Autonomous Bandwidth Interrupt Enable (LABIE) bits. On some (typically old) platforms this has been observed to cause boot hangs, e.g. commit 46a9f70e93ef ("PCI/bwctrl: Disable BW controller on Intel P45 using a quirk") works around one such case with a per-device quirk.

In preparation for a system-wide switch to turn the bandwidth controller off, add pci_bwctrl_available() and pci_no_bwctrl(), mirroring pci_aer_available()/pci_no_aer(), and consult the former when deciding whether to enable PCIE_PORT_SERVICE_BWCTRL.

No functional change intended: the controller stays available by default.

---
 drivers/pci/pci.h          |  4 ++++
 drivers/pci/pcie/bwctrl.c  | 12 ++++++++++++
 drivers/pci/pcie/portdrv.c |  3 ++-
 3 files changed, 18 insertions(+), 1 deletion(-)
drivers/pci/pci.h +4 -0
index 4469e1a77f3c..7a1d4f624ed8 100644
@@ -1121,8 +1121,12 @@ static inline void pcie_ecrc_get_policy(char *str) { }
 
 #ifdef CONFIG_PCIEPORTBUS
 void pcie_reset_lbms(struct pci_dev *port);
+void pci_no_bwctrl(void);
+bool pci_bwctrl_available(void);
 #else
 static inline void pcie_reset_lbms(struct pci_dev *port) {}
+static inline void pci_no_bwctrl(void) {}
+static inline bool pci_bwctrl_available(void) { return false; }
 #endif
 
 struct pci_dev_reset_methods {
drivers/pci/pcie/bwctrl.c +12 -0
index c4c8d260bf96..947cb4b8697b 100644
@@ -48,6 +48,18 @@ struct pcie_bwctrl_data {
 /* Prevent port removal during Link Speed changes. */
 static DECLARE_RWSEM(pcie_bwctrl_setspeed_rwsem);
 
+static bool pcie_bwctrl_disable;
+
+void pci_no_bwctrl(void)
+{
+	pcie_bwctrl_disable = true;
+}
+
+bool pci_bwctrl_available(void)
+{
+	return !pcie_bwctrl_disable;
+}
+
 static bool pcie_valid_speed(enum pci_bus_speed speed)
 {
 	return (speed >= PCIE_SPEED_2_5GT) && (speed <= PCIE_SPEED_64_0GT);
drivers/pci/pcie/portdrv.c +2 -1
index 2d6aa488fe7b..e212c34f7f4a 100644
@@ -274,7 +274,8 @@ static int get_port_device_capability(struct pci_dev *dev)
 
 		pcie_capability_read_dword(dev, PCI_EXP_LNKCAP, &linkcap);
 		if (linkcap & PCI_EXP_LNKCAP_LBNC &&
-		    hweight8(dev->supported_speeds) > 1)
+		    hweight8(dev->supported_speeds) > 1 &&
+		    pci_bwctrl_available())
 			services |= PCIE_PORT_SERVICE_BWCTRL;
 	}
 
Signature · 1 line
2.43.7
#2 lore raw

Re: [PATCH v1 1/2] PCI/bwctrl: Add pci_bwctrl_available()

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 7 lines

Enabling the PCIe bandwidth controller sets the Link Bandwidth Management Interrupt Enable (LBMIE) and Link Autonomous Bandwidth Interrupt Enable (LABIE) bits. On some (typically old) platforms this has been observed to cause boot hangs, e.g. commit 46a9f70e93ef ("PCI/bwctrl: Disable BW controller on Intel P45 using a quirk") works around one such case with a per-device quirk.

Quoted 8 lines

In preparation for a system-wide switch to turn the bandwidth controller off, add pci_bwctrl_available() and pci_no_bwctrl(), mirroring pci_aer_available()/pci_no_aer(), and consult the former when deciding whether to enable PCIE_PORT_SERVICE_BWCTRL.

No functional change intended: the controller stays available by default.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
author
#3 lore raw

PATCH 2/2 PCI: Add pci=nobwctrl parameter to disable the PCIe bandwidth controller

In reply to Guixin Liu

The PCIe bandwidth controller enables the Link Bandwidth Management Interrupt Enable (LBMIE) and Link Autonomous Bandwidth Interrupt Enable (LABIE) bits. On some (typically old) platforms this has been observed to cause boot hangs; commit 46a9f70e93ef ("PCI/bwctrl: Disable BW controller on Intel P45 using a quirk") addresses one such platform with a per-device quirk.

Per-device quirks only help once the offending device has been identified and a quirk has shipped, which is awkward for a boot hang since the machine cannot be booted to report it. The only existing escape hatch, pcie_ports=compat, is a blunt instrument that also disables AER, PME, hotplug and DPC.

Add a pci=nobwctrl boot option to disable just the bandwidth controller system-wide, giving users an immediate, rebuild-free workaround while a targeted quirk is worked out.

---
 Documentation/admin-guide/kernel-parameters.txt | 6 ++++++
 drivers/pci/pci.c                               | 2 ++
 2 files changed, 8 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +6 -0
index b5493a7f8f22..7621f0b61dc8 100644
@@ -5053,6 +5053,12 @@ Kernel parameters
 		noaer		[PCIE] If the PCIEAER kernel config parameter is
 				enabled, this kernel boot option can be used to
 				disable the use of PCIE advanced error reporting.
+		nobwctrl	[PCIE] Disable the PCIe bandwidth controller,
+				which manages Link Bandwidth Management
+				Interrupts. Enabling these interrupts has been
+				observed to cause boot hangs on some (typically
+				old) platforms; use this option to work around
+				such issues.
 		nodomains	[PCI] Disable support for multiple PCI
 				root domains (aka PCI segments, in ACPI-speak).
 		nommconf	[X86] Disable use of MMCONFIG for PCI
drivers/pci/pci.c +2 -0
index 77b17b13ee61..36c0b3fe4017 100644
@@ -6756,6 +6756,8 @@ static int __init pci_setup(char *str)
 				pcie_ats_disabled = true;
 			} else if (!strcmp(str, "noaer")) {
 				pci_no_aer();
+			} else if (!strcmp(str, "nobwctrl")) {
+				pci_no_bwctrl();
 			} else if (!strcmp(str, "earlydump")) {
 				pci_early_dump = true;
 			} else if (!strncmp(str, "realloc=", 8)) {
Signature · 1 line
2.43.7
#4 lore raw

Re: [PATCH v1 2/2] PCI: Add pci=nobwctrl parameter to disable the PCIe bandwidth controller

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 13 lines

The PCIe bandwidth controller enables the Link Bandwidth Management Interrupt Enable (LBMIE) and Link Autonomous Bandwidth Interrupt Enable (LABIE) bits. On some (typically old) platforms this has been observed to cause boot hangs; commit 46a9f70e93ef ("PCI/bwctrl: Disable BW controller on Intel P45 using a quirk") addresses one such platform with a per-device quirk.

Per-device quirks only help once the offending device has been identified and a quirk has shipped, which is awkward for a boot hang since the machine cannot be booted to report it. The only existing escape hatch, pcie_ports=compat, is a blunt instrument that also disables AER, PME, hotplug and DPC.

Quoted 5 lines

Add a pci=nobwctrl boot option to disable just the bandwidth controller system-wide, giving users an immediate, rebuild-free workaround while a targeted quirk is worked out.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#5 lore raw
In reply to Guixin Liu
Quoted 7 lines · Guixin Liu

Enabling the PCIe bandwidth controller sets the Link Bandwidth Management Interrupt Enable (LBMIE) and Link Autonomous Bandwidth Interrupt Enable (LABIE) bits. On some (typically old) platforms this has been observed to cause boot hangs; commit 46a9f70e93ef ("PCI/bwctrl: Disable BW controller on Intel P45 using a quirk") works around one such platform with a per-device quirk.

Quoted 5 lines

A per-device quirk only helps once the offending device is identified and a quirk has shipped, which is awkward for a boot hang since the machine cannot be booted to report the culprit. The only existing escape hatch, pcie_ports=compat, is a blunt instrument that also disables AER, PME, hotplug and DPC.

Ideally we want to identify all devices that have problems if the bandwidth controller is enabled and add quirks for every one of them.

In a sense, a command line option disincentivizes reporting and fixing affected devices because users can just work around the problem.

Additionally, there is a general dislike in the community for adding new command line options.

Are you seeing issues on different devices than the one that's already been quirked? If so, a patch to add quirks for them would be welcome.

Thanks,

Lukas

author
#6 lore raw
Quoted 22 lines · Lukas Wunner
Quoted 7 lines · Guixin Liu

Enabling the PCIe bandwidth controller sets the Link Bandwidth Management Interrupt Enable (LBMIE) and Link Autonomous Bandwidth Interrupt Enable (LABIE) bits. On some (typically old) platforms this has been observed to cause boot hangs; commit 46a9f70e93ef ("PCI/bwctrl: Disable BW controller on Intel P45 using a quirk") works around one such platform with a per-device quirk.

Quoted 5 lines

A per-device quirk only helps once the offending device is identified and a quirk has shipped, which is awkward for a boot hang since the machine cannot be booted to report the culprit. The only existing escape hatch, pcie_ports=compat, is a blunt instrument that also disables AER, PME, hotplug and DPC.

Ideally we want to identify all devices that have problems if the bandwidth controller is enabled and add quirks for every one of them.

In a sense, a command line option disincentivizes reporting and fixing affected devices because users can just work around the problem.

Additionally, there is a general dislike in the community for adding new command line options.

Quoted 7 lines

Are you seeing issues on different devices than the one that's already been quirked? If so, a patch to add quirks for them would be welcome.

Thanks,

Lukas

Thanks Lukas, that's fair - I fully agree that per-device quirks are the right long-term fix, and I'll gladly send quirks for any device I can reproduce this on.

I'd argue this option isn't really an alternative to quirks, though. It's what lets you *get to* a quirk in the first place, and it closes a gap that is specific to bwctrl:

- To write a bwctrl quirk you first have to prove bwctrl is the culprit. pcie_ports=compat can get a hung machine booted, but it tears down *all* port services at once, so it can't tell you whether the offender was bwctrl, AER, PME or hotplug. pci=nobwctrl isolates exactly one variable, which is precisely the information a quirk needs. In that sense it feeds the quirk pipeline rather than discouraging reports.

- compat is also too blunt as a workaround on modern systems: losing native AER/DPC/hotplug just to silence bwctrl is a large regression in error handling. A surgical switch keeps everything else native.

- bwctrl is currently the only PCIe port service with neither a native/_OSC gate nor a disable of its own. hotplug, AER and PME are all gated on pcie_ports_native || host->native_*, and pci=noaer already exists as precedent. This mostly brings bwctrl up to parity rather than inventing a new class of knob.

If the command-line parameter itself is the sticking point, I'm happy to implement this in whatever form you prefer - e.g. gating bwctrl registration the same way the other services are gated, or a per-port sysfs runtime control. My goal is to close the gap; the mechanism is negotiable.

Best Regards,
Guixin Liu

#7 lore raw
Quoted 58 lines · Guixin Liu
Quoted 22 lines · Lukas Wunner
Quoted 7 lines · Guixin Liu

Enabling the PCIe bandwidth controller sets the Link Bandwidth Management Interrupt Enable (LBMIE) and Link Autonomous Bandwidth Interrupt Enable (LABIE) bits. On some (typically old) platforms this has been observed to cause boot hangs; commit 46a9f70e93ef ("PCI/bwctrl: Disable BW controller on Intel P45 using a quirk") works around one such platform with a per-device quirk.

Quoted 5 lines

A per-device quirk only helps once the offending device is identified and a quirk has shipped, which is awkward for a boot hang since the machine cannot be booted to report the culprit. The only existing escape hatch, pcie_ports=compat, is a blunt instrument that also disables AER, PME, hotplug and DPC.

Ideally we want to identify all devices that have problems if the bandwidth controller is enabled and add quirks for every one of them.

In a sense, a command line option disincentivizes reporting and fixing affected devices because users can just work around the problem.

Additionally, there is a general dislike in the community for adding new command line options.

Quoted 7 lines

Are you seeing issues on different devices than the one that's already been quirked? If so, a patch to add quirks for them would be welcome.

Thanks,

Lukas

Thanks Lukas, that's fair - I fully agree that per-device quirks are the right long-term fix, and I'll gladly send quirks for any device I can reproduce this on.

I'd argue this option isn't really an alternative to quirks, though. It's what lets you *get to* a quirk in the first place, and it closes a gap that is specific to bwctrl:

- To write a bwctrl quirk you first have to prove bwctrl is the culprit. pcie_ports=compat can get a hung machine booted, but it tears down *all* port services at once, so it can't tell you whether the offender was bwctrl, AER, PME or hotplug. pci=nobwctrl isolates exactly one variable, which is precisely the information a quirk needs. In that sense it feeds the quirk pipeline rather than discouraging reports.

- compat is also too blunt as a workaround on modern systems: losing native AER/DPC/hotplug just to silence bwctrl is a large regression in error handling. A surgical switch keeps everything else native.

- bwctrl is currently the only PCIe port service with neither a native/_OSC gate nor a disable of its own. hotplug, AER and PME are all gated on pcie_ports_native || host->native_*, and pci=noaer already exists as precedent. This mostly brings bwctrl up to parity rather than inventing a new class of knob.

Quoted 6 lines

If the command-line parameter itself is the sticking point, I'm happy to implement this in whatever form you prefer - e.g. gating bwctrl registration the same way the other services are gated, or a per-port sysfs runtime control. My goal is to close the gap; the mechanism is negotiable.

I'd argue the other way that developers can still add quirks for testing whether the crash is due to bwctrl or not.

I kind of agree with Lukas that once we add this knob, it may demotivate developers from sending the quirks.

- Mani

Signature · 1 line
மணிவண்ணன் சதாசிவம்