Skip to contentSkip to conversation

PCI: pcie_bus_config can be set at build time

PATCH RESEND Superseded Jim Quinlan 2 participants 4 messages Sep 8, 2020 – Sep 10, 2020 PCI

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH RESEND 1/1 PCI: pcie_bus_config can be set at build time

The Kconfig is modified so that the pcie_bus_config setting can be done at build time in the same manner as the CONFIG_PCIEASPM_XXXX choice. The pci_bus_config setting may still be overridden by the bootline param.

---
 drivers/pci/Kconfig | 40 ++++++++++++++++++++++++++++++++++++++++
 drivers/pci/pci.c   | 12 ++++++++++++
 2 files changed, 52 insertions(+)
drivers/pci/Kconfig +40 -0
index 4bef5c2bae9f..efe69b0d9f7f 100644
@@ -187,6 +187,46 @@ config PCI_HYPERV
 	  The PCI device frontend driver allows the kernel to import arbitrary
 	  PCI devices from a PCI backend to support PCI driver domains.
 
+choice
+	prompt "PCIE default bus config setting"
+	default PCIE_BUS_DEFAULT
+	depends on PCI
+	help
+	  One of the following choices will set the pci_bus_config at
+	  compile time.  This will still be overridden by the appropriate
+	  pci bootline parameter.
+
+config PCIE_BUS_TUNE_OFF
+	bool "Tune Off"
+	depends on PCI
+	help
+	  Use the BIOS defaults; doesn't touch MPS at all.
+
+config PCIE_BUS_DEFAULT
+	bool "Default"
+	depends on PCI
+	help
+	  Ensure MPS matches upstream bridge.
+
+config PCIE_BUS_SAFE
+	bool "Safe"
+	depends on PCI
+	help
+	  Use largest MPS boot-time devices support.
+
+config PCIE_BUS_PERFORMANCE
+	bool "Performance"
+	depends on PCI
+	help
+	  Use MPS and MRRS for best performance.
+
+config PCIE_BUS_PEER2PEER
+	bool "Peer2peer"
+	depends on PCI
+	help
+	  Set MPS = 128 for all devices.
+endchoice
+
 source "drivers/pci/hotplug/Kconfig"
 source "drivers/pci/controller/Kconfig"
 source "drivers/pci/endpoint/Kconfig"
drivers/pci/pci.c +12 -0
index e39c5499770f..dfb52ed4a931 100644
@@ -101,7 +101,19 @@ unsigned long pci_hotplug_mmio_pref_size = DEFAULT_HOTPLUG_MMIO_PREF_SIZE;
 #define DEFAULT_HOTPLUG_BUS_SIZE	1
 unsigned long pci_hotplug_bus_size = DEFAULT_HOTPLUG_BUS_SIZE;
 
+
+/* PCIE bus config, can be overridden by bootline param */
+#ifdef CONFIG_PCIE_BUS_TUNE_OFF
+enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_TUNE_OFF;
+#elif defined CONFIG_PCIE_BUS_SAFE
+enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_SAFE;
+#elif defined CONFIG_PCIE_BUS_PERFORMANCE
+enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_PERFORMANCE;
+#elif defined CONFIG_PCIE_BUS_PEER2PEER
+enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_PEER2PEER;
+#else
 enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_DEFAULT;
+#endif
 
 /*
  * The default CLS is used if arch didn't set CLS explicitly and not
Signature · 1 line
2.17.1
#1 lore raw
Quoted 3 lines · Jim Quinlan

The Kconfig is modified so that the pcie_bus_config setting can be done at build time in the same manner as the CONFIG_PCIEASPM_XXXX choice. The pci_bus_config setting may still be overridden by the bootline param.

I guess... I really hate these build-time config settings for both ASPM and MPS/MRRS. But Linux just isn't smart or flexible enough to do the right thing at run-time, so I guess we're kind of stuck.

I guess you have systems where you need a different default?

It'd be nice if we could put a little more detail in the Kconfig to help users choose the correct one. "Ensure MPS matches upstream bridge" is *accurate*, but it doesn't really tell me why I would choose this rather than a different one.

Maybe we could mention the corresponding command-line parameters, e.g., "This is the same as booting with 'pci=pcie_bus_tune_off'"?

Quoted 84 lines
---
 drivers/pci/Kconfig | 40 ++++++++++++++++++++++++++++++++++++++++
 drivers/pci/pci.c   | 12 ++++++++++++
 2 files changed, 52 insertions(+)
drivers/pci/Kconfig +40 -0
index 4bef5c2bae9f..efe69b0d9f7f 100644
@@ -187,6 +187,46 @@ config PCI_HYPERV
 	  The PCI device frontend driver allows the kernel to import arbitrary
 	  PCI devices from a PCI backend to support PCI driver domains.
 
+choice
+	prompt "PCIE default bus config setting"
+	default PCIE_BUS_DEFAULT
+	depends on PCI
+	help
+	  One of the following choices will set the pci_bus_config at
+	  compile time.  This will still be overridden by the appropriate
+	  pci bootline parameter.
+
+config PCIE_BUS_TUNE_OFF
+	bool "Tune Off"
+	depends on PCI
+	help
+	  Use the BIOS defaults; doesn't touch MPS at all.
+
+config PCIE_BUS_DEFAULT
+	bool "Default"
+	depends on PCI
+	help
+	  Ensure MPS matches upstream bridge.
+
+config PCIE_BUS_SAFE
+	bool "Safe"
+	depends on PCI
+	help
+	  Use largest MPS boot-time devices support.
+
+config PCIE_BUS_PERFORMANCE
+	bool "Performance"
+	depends on PCI
+	help
+	  Use MPS and MRRS for best performance.
+
+config PCIE_BUS_PEER2PEER
+	bool "Peer2peer"
+	depends on PCI
+	help
+	  Set MPS = 128 for all devices.
+endchoice
+
 source "drivers/pci/hotplug/Kconfig"
 source "drivers/pci/controller/Kconfig"
 source "drivers/pci/endpoint/Kconfig"
drivers/pci/pci.c +12 -0
index e39c5499770f..dfb52ed4a931 100644
@@ -101,7 +101,19 @@ unsigned long pci_hotplug_mmio_pref_size = DEFAULT_HOTPLUG_MMIO_PREF_SIZE;
 #define DEFAULT_HOTPLUG_BUS_SIZE	1
 unsigned long pci_hotplug_bus_size = DEFAULT_HOTPLUG_BUS_SIZE;
 
+
+/* PCIE bus config, can be overridden by bootline param */
+#ifdef CONFIG_PCIE_BUS_TUNE_OFF
+enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_TUNE_OFF;
+#elif defined CONFIG_PCIE_BUS_SAFE
+enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_SAFE;
+#elif defined CONFIG_PCIE_BUS_PERFORMANCE
+enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_PERFORMANCE;
+#elif defined CONFIG_PCIE_BUS_PEER2PEER
+enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_PEER2PEER;
+#else
 enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_DEFAULT;
+#endif
 
 /*
  * The default CLS is used if arch didn't set CLS explicitly and not
Signature · 1 line
2.17.1
#2 lore raw

Hi Bjorn,

Quoted 11 lines · Bjorn Helgaas
Quoted 3 lines · Jim Quinlan

The Kconfig is modified so that the pcie_bus_config setting can be done at build time in the same manner as the CONFIG_PCIEASPM_XXXX choice. The pci_bus_config setting may still be overridden by the bootline param.

I guess... I really hate these build-time config settings for both ASPM and MPS/MRRS. But Linux just isn't smart or flexible enough to do the right thing at run-time, so I guess we're kind of stuck.

I guess you have systems where you need a different default?

Yes, we've been shipping our kernel with the DEFAULT and since we do not have FW it is not configured optimally. Some customers have noticed and I tell them to put 'pci=pcie_bus_safe' on their bootline but I'd rather have this setting work for all customers as it yields the option we want.

Quoted 5 lines

It'd be nice if we could put a little more detail in the Kconfig to help users choose the correct one. "Ensure MPS matches upstream bridge" is *accurate*, but it doesn't really tell me why I would choose this rather than a different one.

IIRC I just copied the comments that were in the bootline settings. I'm concerned about there being the same comment in two places; sooner or later someone will update one place and not the other.

Quoted 3 lines

Maybe we could mention the corresponding command-line parameters, e.g., "This is the same as booting with 'pci=pcie_bus_tune_off'"?

Will do and resubmit.

Thanks,
Jim Quinlan
Broadcom STB

Quoted 85 lines
Quoted 85 lines
---
 drivers/pci/Kconfig | 40 ++++++++++++++++++++++++++++++++++++++++
 drivers/pci/pci.c   | 12 ++++++++++++
 2 files changed, 52 insertions(+)
drivers/pci/Kconfig +40 -0
index 4bef5c2bae9f..efe69b0d9f7f 100644
@@ -187,6 +187,46 @@ config PCI_HYPERV
        The PCI device frontend driver allows the kernel to import arbitrary
        PCI devices from a PCI backend to support PCI driver domains.
 
+choice
+     prompt "PCIE default bus config setting"
+     default PCIE_BUS_DEFAULT
+     depends on PCI
+     help
+       One of the following choices will set the pci_bus_config at
+       compile time.  This will still be overridden by the appropriate
+       pci bootline parameter.
+
+config PCIE_BUS_TUNE_OFF
+     bool "Tune Off"
+     depends on PCI
+     help
+       Use the BIOS defaults; doesn't touch MPS at all.
+
+config PCIE_BUS_DEFAULT
+     bool "Default"
+     depends on PCI
+     help
+       Ensure MPS matches upstream bridge.
+
+config PCIE_BUS_SAFE
+     bool "Safe"
+     depends on PCI
+     help
+       Use largest MPS boot-time devices support.
+
+config PCIE_BUS_PERFORMANCE
+     bool "Performance"
+     depends on PCI
+     help
+       Use MPS and MRRS for best performance.
+
+config PCIE_BUS_PEER2PEER
+     bool "Peer2peer"
+     depends on PCI
+     help
+       Set MPS = 128 for all devices.
+endchoice
+
 source "drivers/pci/hotplug/Kconfig"
 source "drivers/pci/controller/Kconfig"
 source "drivers/pci/endpoint/Kconfig"
drivers/pci/pci.c +12 -1
index e39c5499770f..dfb52ed4a931 100644
@@ -101,7 +101,19 @@ unsigned long pci_hotplug_mmio_pref_size = DEFAULT_HOTPLUG_MMIO_PREF_SIZE;
 #define DEFAULT_HOTPLUG_BUS_SIZE     1
 unsigned long pci_hotplug_bus_size = DEFAULT_HOTPLUG_BUS_SIZE;
 
+
+/* PCIE bus config, can be overridden by bootline param */
+#ifdef CONFIG_PCIE_BUS_TUNE_OFF
+enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_TUNE_OFF;
+#elif defined CONFIG_PCIE_BUS_SAFE
+enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_SAFE;
+#elif defined CONFIG_PCIE_BUS_PERFORMANCE
+enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_PERFORMANCE;
+#elif defined CONFIG_PCIE_BUS_PEER2PEER
+enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_PEER2PEER;
+#else
 enum pcie_bus_config_types pcie_bus_config = PCIE_BUS_DEFAULT;
+#endif
 
 /*
  * The default CLS is used if arch didn't set CLS explicitly and not
--

2.17.1

#3 lore raw
Quoted 13 lines · Jim Quinlan

Hi Bjorn,

Quoted 10 lines · Bjorn Helgaas
Quoted 3 lines · Jim Quinlan

The Kconfig is modified so that the pcie_bus_config setting can be done at build time in the same manner as the CONFIG_PCIEASPM_XXXX choice. The pci_bus_config setting may still be overridden by the bootline param.

I guess... I really hate these build-time config settings for both ASPM and MPS/MRRS. But Linux just isn't smart or flexible enough to do the right thing at run-time, so I guess we're kind of stuck.

Quoted 7 lines
Quoted 1 line

I guess you have systems where you need a different default?

Yes, we've been shipping our kernel with the DEFAULT and since we do not have FW it is not configured optimally. Some customers have noticed and I tell them to put 'pci=pcie_bus_safe' on their bootline but I'd rather have this setting work for all customers as it yields the option we want.

I'm guessing you probably don't have any hotplug slots. Seems like we ought to be able to recognize that and pick pcie_bus_safe automatically. Someday.

Maybe that's part of the description: if you have a closed system with no possibility of adding new devices, we can use the largest MPS that's supported by all devices, i.e., pcie_bus_safe.

Quoted 8 lines
Quoted 4 lines

It'd be nice if we could put a little more detail in the Kconfig to help users choose the correct one. "Ensure MPS matches upstream bridge" is *accurate*, but it doesn't really tell me why I would choose this rather than a different one.

IIRC I just copied the comments that were in the bootline settings. I'm concerned about there being the same comment in two places; sooner or later someone will update one place and not the other.

True. It'd be nice if we at least had *one* place with a useful description. I don't think we have any today.

Bjorn