Skip to contentSkip to conversation

PCI: Add "pci=reassign_all_bus" boot parameter

RFC PATCH In review Yao Hongbo 4 participants 6 messages Dec 30, 2021 – Jan 19, 2022

Subsystems (3)
lore
Share & tools
lore

Yao Hongbo

author
#0 lore raw

RFC PATCH 1/1 PCI: Add "pci=reassign_all_bus" boot parameter

PCI bridges may be misconfigured by te system BIOS, and then the OS scan the bridges that need to be reconfigured.
However, the PCI bus topology configured by the bios may be wrong:

[   19.376273] pci 0000:40:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[   19.384443] pci_bus 0000:47: busn_res: can not insert [bus 47-46]

under [bus 40-46] (conflicts with (null) [bus 40-46])

The primary bus number and subordinate bus number written by the bios were wrong, and the OS continues to add bridges on the wrong bus topology.

In order to avoid such problems, a kernel cmdline needs to be added to support the os to fully configure the pci bus.

---
 Documentation/admin-guide/kernel-parameters.txt | 1 +
 drivers/acpi/pci_root.c                         | 3 +++
 drivers/pci/pci.c                               | 5 +++++
 include/linux/pci.h                             | 2 ++
 4 files changed, 11 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +1 -0
index 2fba824..c83a2e5 100644
@@ -4084,6 +4084,7 @@
 		nomio		[S390] Do not use MIO instructions.
 		norid		[S390] ignore the RID field and force use of
 				one PCI domain per PCI function
+		reassign_all_bus	The OS fully configure the PCI bus.
 
 	pcie_aspm=	[PCIE] Forcibly enable or disable PCIe Active State Power
 			Management.
drivers/acpi/pci_root.c +3 -0
index ab2f7df..e21ac25 100644
@@ -592,6 +592,9 @@ static int acpi_pci_root_add(struct acpi_device *device,
 	is_pcie = strcmp(acpi_device_hid(device), "PNP0A08") == 0;
 	negotiate_os_control(root, &no_aspm, is_pcie);
 
+	if (pci_reassign_all_bus)
+		pci_add_flags(PCI_REASSIGN_ALL_BUS);
+
 	/*
 	 * TBD: Need PCI interface for enumeration/configuration of roots.
 	 */
drivers/pci/pci.c +5 -0
index 3d2fb39..5746e88 100644
@@ -142,6 +142,9 @@ bool pci_reset_supported(struct pci_dev *dev)
 /* If set, the PCI config space of each device is printed during boot. */
 bool pci_early_dump;
 
+/* If set, the pci will reassign resources*/
+bool pci_reassign_all_bus;
+
 bool pci_ats_disabled(void)
 {
 	return pcie_ats_disabled;
@@ -6846,6 +6849,8 @@ static int __init pci_setup(char *str)
 				pci_add_flags(PCI_SCAN_ALL_PCIE_DEVS);
 			} else if (!strncmp(str, "disable_acs_redir=", 18)) {
 				disable_acs_redir_param = str + 18;
+			} else if (!strncmp(str, "reassign_all_bus", 16)) {
+				pci_reassign_all_bus = true;
 			} else {
 				pr_err("PCI: Unknown option `%s'\n", str);
 			}
include/linux/pci.h +2 -0
index 18a75c8e..ad0e3e9 100644
@@ -2119,6 +2119,8 @@ int pcim_iomap_regions_request_all(struct pci_dev *pdev, int mask,
 extern u8 pci_dfl_cache_line_size;
 extern u8 pci_cache_line_size;
 
+extern bool pci_reassign_all_bus;
+
 /* Architecture-specific versions may override these (weak) */
 void pcibios_disable_device(struct pci_dev *dev);
 void pcibios_set_master(struct pci_dev *dev);
Signature · 1 line
1.8.3.1
reviewer
#1 lore raw

Hi--

Documentation/admin-guide/kernel-parameters.txt line 4087

Quoted 9 lines · Yao Hongbo
Documentation/admin-guide/kernel-parameters.txt +1 -0
index 2fba824..c83a2e5 100644
@@ -4084,6 +4084,7 @@
 		nomio		[S390] Do not use MIO instructions.
 		norid		[S390] ignore the RID field and force use of
 				one PCI domain per PCI function
+		reassign_all_bus	The OS fully configure the PCI bus.

configures

Documentation/admin-guide/kernel-parameters.txt line 4090

Quoted 3 lines
Documentation/admin-guide/kernel-parameters.txt +0 -0
 	pcie_aspm=	[PCIE] Forcibly enable or disable PCIe Active State Power
 			Management.

thanks.

Signature · 1 line
~Randy
author
#2 lore raw
In reply to Yao Hongbo

Hi, Bjorn,
Gentel ping! Any comments on this patch?

I think this is useful to find whether there is a problem with the PCI enumeration of the bios or the OS.

Quoted 86 lines · Yao Hongbo

PCI bridges may be misconfigured by te system BIOS, and then the OS scan the bridges that need to be reconfigured.
However, the PCI bus topology configured by the bios may be wrong:

[   19.376273] pci 0000:40:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[   19.384443] pci_bus 0000:47: busn_res: can not insert [bus 47-46]

under [bus 40-46] (conflicts with (null) [bus 40-46])

The primary bus number and subordinate bus number written by the bios were wrong, and the OS continues to add bridges on the wrong bus topology.

In order to avoid such problems, a kernel cmdline needs to be added to support the os to fully configure the pci bus.

---
 Documentation/admin-guide/kernel-parameters.txt | 1 +
 drivers/acpi/pci_root.c                         | 3 +++
 drivers/pci/pci.c                               | 5 +++++
 include/linux/pci.h                             | 2 ++
 4 files changed, 11 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +1 -0
index 2fba824..c83a2e5 100644
@@ -4084,6 +4084,7 @@
 		nomio		[S390] Do not use MIO instructions.
 		norid		[S390] ignore the RID field and force use of
 				one PCI domain per PCI function
+		reassign_all_bus	The OS fully configure the PCI bus.
 
 	pcie_aspm=	[PCIE] Forcibly enable or disable PCIe Active State Power
 			Management.
drivers/acpi/pci_root.c +3 -0
index ab2f7df..e21ac25 100644
@@ -592,6 +592,9 @@ static int acpi_pci_root_add(struct acpi_device *device,
 	is_pcie = strcmp(acpi_device_hid(device), "PNP0A08") == 0;
 	negotiate_os_control(root, &no_aspm, is_pcie);
 
+	if (pci_reassign_all_bus)
+		pci_add_flags(PCI_REASSIGN_ALL_BUS);
+
 	/*
 	 * TBD: Need PCI interface for enumeration/configuration of roots.
 	 */
drivers/pci/pci.c +5 -0
index 3d2fb39..5746e88 100644
@@ -142,6 +142,9 @@ bool pci_reset_supported(struct pci_dev *dev)
 /* If set, the PCI config space of each device is printed during boot. */
 bool pci_early_dump;
 
+/* If set, the pci will reassign resources*/
+bool pci_reassign_all_bus;
+
 bool pci_ats_disabled(void)
 {
 	return pcie_ats_disabled;
@@ -6846,6 +6849,8 @@ static int __init pci_setup(char *str)
 				pci_add_flags(PCI_SCAN_ALL_PCIE_DEVS);
 			} else if (!strncmp(str, "disable_acs_redir=", 18)) {
 				disable_acs_redir_param = str + 18;
+			} else if (!strncmp(str, "reassign_all_bus", 16)) {
+				pci_reassign_all_bus = true;
 			} else {
 				pr_err("PCI: Unknown option `%s'\n", str);
 			}
include/linux/pci.h +2 -0
index 18a75c8e..ad0e3e9 100644
@@ -2119,6 +2119,8 @@ int pcim_iomap_regions_request_all(struct pci_dev *pdev, int mask,
 extern u8 pci_dfl_cache_line_size;
 extern u8 pci_cache_line_size;
 
+extern bool pci_reassign_all_bus;
+
 /* Architecture-specific versions may override these (weak) */
 void pcibios_disable_device(struct pci_dev *dev);
 void pcibios_set_master(struct pci_dev *dev);
#3 lore raw
In reply to Yao Hongbo
Quoted 7 lines · Yao Hongbo

PCI bridges may be misconfigured by te system BIOS, and then the OS scan the bridges that need to be reconfigured.
However, the PCI bus topology configured by the bios may be wrong:

[   19.376273] pci 0000:40:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[   19.384443] pci_bus 0000:47: busn_res: can not insert [bus 47-46]
Quoted 8 lines

under [bus 40-46] (conflicts with (null) [bus 40-46])

The primary bus number and subordinate bus number written by the bios were wrong, and the OS continues to add bridges on the wrong bus topology.

In order to avoid such problems, a kernel cmdline needs to be added to support the os to fully configure the pci bus.

Why can't we make Linux smart enough to fix this by itself, without forcing the user to boot with "pci=reassign_all_bus"?

Quoted 73 lines
---
 Documentation/admin-guide/kernel-parameters.txt | 1 +
 drivers/acpi/pci_root.c                         | 3 +++
 drivers/pci/pci.c                               | 5 +++++
 include/linux/pci.h                             | 2 ++
 4 files changed, 11 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +1 -0
index 2fba824..c83a2e5 100644
@@ -4084,6 +4084,7 @@
 		nomio		[S390] Do not use MIO instructions.
 		norid		[S390] ignore the RID field and force use of
 				one PCI domain per PCI function
+		reassign_all_bus	The OS fully configure the PCI bus.
 
 	pcie_aspm=	[PCIE] Forcibly enable or disable PCIe Active State Power
 			Management.
drivers/acpi/pci_root.c +3 -0
index ab2f7df..e21ac25 100644
@@ -592,6 +592,9 @@ static int acpi_pci_root_add(struct acpi_device *device,
 	is_pcie = strcmp(acpi_device_hid(device), "PNP0A08") == 0;
 	negotiate_os_control(root, &no_aspm, is_pcie);
 
+	if (pci_reassign_all_bus)
+		pci_add_flags(PCI_REASSIGN_ALL_BUS);
+
 	/*
 	 * TBD: Need PCI interface for enumeration/configuration of roots.
 	 */
drivers/pci/pci.c +5 -0
index 3d2fb39..5746e88 100644
@@ -142,6 +142,9 @@ bool pci_reset_supported(struct pci_dev *dev)
 /* If set, the PCI config space of each device is printed during boot. */
 bool pci_early_dump;
 
+/* If set, the pci will reassign resources*/
+bool pci_reassign_all_bus;
+
 bool pci_ats_disabled(void)
 {
 	return pcie_ats_disabled;
@@ -6846,6 +6849,8 @@ static int __init pci_setup(char *str)
 				pci_add_flags(PCI_SCAN_ALL_PCIE_DEVS);
 			} else if (!strncmp(str, "disable_acs_redir=", 18)) {
 				disable_acs_redir_param = str + 18;
+			} else if (!strncmp(str, "reassign_all_bus", 16)) {
+				pci_reassign_all_bus = true;
 			} else {
 				pr_err("PCI: Unknown option `%s'\n", str);
 			}
include/linux/pci.h +2 -0
index 18a75c8e..ad0e3e9 100644
@@ -2119,6 +2119,8 @@ int pcim_iomap_regions_request_all(struct pci_dev *pdev, int mask,
 extern u8 pci_dfl_cache_line_size;
 extern u8 pci_cache_line_size;
 
+extern bool pci_reassign_all_bus;
+
 /* Architecture-specific versions may override these (weak) */
 void pcibios_disable_device(struct pci_dev *dev);
 void pcibios_set_master(struct pci_dev *dev);
Signature · 1 line
1.8.3.1
author
#4 lore raw
Quoted 14 lines · Bjorn Helgaas
Quoted 13 lines · Yao Hongbo

PCI bridges may be misconfigured by te system BIOS, and then the OS scan the bridges that need to be reconfigured.
However, the PCI bus topology configured by the bios may be wrong:

[   19.376273] pci 0000:40:00.0: bridge configuration invalid ([bus
00-00]), reconfiguring
[   19.384443] pci_bus 0000:47: busn_res: can not insert [bus 47-46]

under [bus 40-46] (conflicts with (null) [bus 40-46])

The primary bus number and subordinate bus number written by the bios were wrong, and the OS continues to add bridges on the wrong bus topology.

Quoted 6 lines
Quoted 2 lines

In order to avoid such problems, a kernel cmdline needs to be added to support the os to fully configure the pci bus.

Why can't we make Linux smart enough to fix this by itself, without forcing the user to boot with "pci=reassign_all_bus"?

Hi, Bjorn.

You're right, it's better for us to make pci enumeration more smart.

But i think it's better to provide an additional option for users on ACPI systems,

which can quickly distingguish the issues between the BIOS and the OS.

Quoted 73 lines
Quoted 73 lines
---
 Documentation/admin-guide/kernel-parameters.txt | 1 +
 drivers/acpi/pci_root.c                         | 3 +++
 drivers/pci/pci.c                               | 5 +++++
 include/linux/pci.h                             | 2 ++
 4 files changed, 11 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +1 -0
index 2fba824..c83a2e5 100644
@@ -4084,6 +4084,7 @@
 		nomio		[S390] Do not use MIO instructions.
 		norid		[S390] ignore the RID field and force use of
 				one PCI domain per PCI function
+		reassign_all_bus	The OS fully configure the PCI bus.
 
 	pcie_aspm=	[PCIE] Forcibly enable or disable PCIe Active State Power
 			Management.
drivers/acpi/pci_root.c +3 -0
index ab2f7df..e21ac25 100644
@@ -592,6 +592,9 @@ static int acpi_pci_root_add(struct acpi_device *device,
 	is_pcie = strcmp(acpi_device_hid(device), "PNP0A08") == 0;
 	negotiate_os_control(root, &no_aspm, is_pcie);
 
+	if (pci_reassign_all_bus)
+		pci_add_flags(PCI_REASSIGN_ALL_BUS);
+
 	/*
 	 * TBD: Need PCI interface for enumeration/configuration of roots.
 	 */
drivers/pci/pci.c +5 -0
index 3d2fb39..5746e88 100644
@@ -142,6 +142,9 @@ bool pci_reset_supported(struct pci_dev *dev)
 /* If set, the PCI config space of each device is printed during boot. */
 bool pci_early_dump;
 
+/* If set, the pci will reassign resources*/
+bool pci_reassign_all_bus;
+
 bool pci_ats_disabled(void)
 {
 	return pcie_ats_disabled;
@@ -6846,6 +6849,8 @@ static int __init pci_setup(char *str)
 				pci_add_flags(PCI_SCAN_ALL_PCIE_DEVS);
 			} else if (!strncmp(str, "disable_acs_redir=", 18)) {
 				disable_acs_redir_param = str + 18;
+			} else if (!strncmp(str, "reassign_all_bus", 16)) {
+				pci_reassign_all_bus = true;
 			} else {
 				pr_err("PCI: Unknown option `%s'\n", str);
 			}
include/linux/pci.h +2 -0
index 18a75c8e..ad0e3e9 100644
@@ -2119,6 +2119,8 @@ int pcim_iomap_regions_request_all(struct pci_dev *pdev, int mask,
 extern u8 pci_dfl_cache_line_size;
 extern u8 pci_cache_line_size;
 
+extern bool pci_reassign_all_bus;
+
 /* Architecture-specific versions may override these (weak) */
 void pcibios_disable_device(struct pci_dev *dev);
 void pcibios_set_master(struct pci_dev *dev);
Signature · 1 line
1.8.3.1
#5 lore raw
Quoted 8 lines · Yao Hongbo
Quoted 2 lines

Why can't we make Linux smart enough to fix this by itself, without forcing the user to boot with "pci=reassign_all_bus"?

You're right, it's better for us to make pci enumeration more smart.

But i think it's better to provide an additional option for users on ACPI systems,

which can quickly distingguish the issues between the BIOS and the OS.

Even if so, x86 has had `pci=assign-busses' for decades already (i.e.
from Linux 2.4.6), so why not reuse it for whatever your ACPI platform is?

Maciej