Skip to contentSkip to conversation

powerpc/pci: Cleanup unused code and enable 256 PCI buses

PATCH Applied Pali Rohár 4 participants 12 messages Jul 6, 2022 – Aug 17, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover powerpc/pci: Cleanup unused code and enable 256 PCI buses

This patch series cleanup unused code by eliminating it at compile time and then enable usage of all 256 PCI buses per every PCI domain as currently PCI bus numbers have to be unique across all PCI domains. So first bus number of each PCI domain would be zero and not the bus number of the previous domain plus one. As such bus assignment changes BDF address of every PCI device on multi-domain system, this new "feature" is configurable by config option to prevent regressions.

Tested on Freescale P2020 board. Before this patch lspci reports:

8000:00:00.0 PCI bridge: Freescale Semiconductor Inc P2020E (rev 21)
8000:01:00.0 USB controller: Texas Instruments TUSB73x0 SuperSpeed USB 3.0 xHCI Host Controller (rev 02) 9000:02:00.0 PCI bridge: Freescale Semiconductor Inc P2020E (rev 21)
9000:03:00.0 Network controller: Qualcomm Atheros AR93xx Wireless Network Adapter (rev 01) a000:04:00.0 PCI bridge: Freescale Semiconductor Inc P2020E (rev 21)
a000:05:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter

With these patches and enabled config option it reports:

8000:00:00.0 PCI bridge: Freescale Semiconductor Inc P2020E (rev 21)
8000:01:00.0 USB controller: Texas Instruments TUSB73x0 SuperSpeed USB 3.0 xHCI Host Controller (rev 02) 9000:00:00.0 PCI bridge: Freescale Semiconductor Inc P2020E (rev 21)
9000:01:00.0 Network controller: Qualcomm Atheros AR93xx Wireless Network Adapter (rev 01) a000:00:00.0 PCI bridge: Freescale Semiconductor Inc P2020E (rev 21)
a000:01:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter

Now every PCIe Root Port is on bus zero and theoretically with enough multiport PCIe switches connected to every PCIe Root Port, it should be possible to have 256 PCI buses on every PCIe controller (as each is in own PCI domain) and therefore connect more PCIe cards as without these patches.

Pali Rohár (5):
powerpc/pci: Hide pci_device_from_OF_node() for non-powermac code powerpc/pci: Make pcibios_make_OF_bus_map() static powerpc/pci: Hide pci_create_OF_bus_map() for non-chrp code powerpc/pci: Disable filling pci-OF-bus-map for non-chrp/powermac powerpc/pci: Add config option for using all 256 PCI buses

 arch/powerpc/Kconfig                  | 11 +++++++++++
 arch/powerpc/include/asm/pci-bridge.h |  4 ++++
 arch/powerpc/kernel/pci_32.c          | 27 +++++++++++++++++++++------
 arch/powerpc/kernel/pci_64.c          |  2 ++
 4 files changed, 38 insertions(+), 6 deletions(-)
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/5 powerpc/pci: Hide pci_device_from_OF_node() for non-powermac code

Function pci_device_from_OF_node() is used only in powermac code. So hide it from all other platforms as it is unsed.

---
 arch/powerpc/include/asm/pci-bridge.h | 2 ++
 arch/powerpc/kernel/pci_32.c          | 2 ++
 arch/powerpc/kernel/pci_64.c          | 2 ++
 3 files changed, 6 insertions(+)
arch/powerpc/include/asm/pci-bridge.h +2 -0
index c85f901227c9..98156932a1f5 100644
@@ -170,8 +170,10 @@ static inline struct pci_controller *pci_bus_to_host(const struct pci_bus *bus)
 	return bus->sysdata;
 }
 
+#ifdef CONFIG_PPC_PMAC
 extern int pci_device_from_OF_node(struct device_node *node,
 				   u8 *bus, u8 *devfn);
+#endif
 #ifndef CONFIG_PPC64
 
 extern void pci_create_OF_bus_map(void);
arch/powerpc/kernel/pci_32.c +2 -0
index 5a174936c9a0..c3b91fb62a71 100644
@@ -154,6 +154,7 @@ pcibios_make_OF_bus_map(void)
 }
 
 
+#ifdef CONFIG_PPC_PMAC
 /*
  * Returns the PCI device matching a given OF node
  */
@@ -193,6 +194,7 @@ int pci_device_from_OF_node(struct device_node *node, u8 *bus, u8 *devfn)
 	return -ENODEV;
 }
 EXPORT_SYMBOL(pci_device_from_OF_node);
+#endif
 
 /* We create the "pci-OF-bus-map" property now so it appears in the
  * /proc device tree
arch/powerpc/kernel/pci_64.c +2 -0
index 19b03ddf5631..0c7cfb9fab04 100644
@@ -286,6 +286,7 @@ int pcibus_to_node(struct pci_bus *bus)
 EXPORT_SYMBOL(pcibus_to_node);
 #endif
 
+#ifdef CONFIG_PPC_PMAC
 int pci_device_from_OF_node(struct device_node *np, u8 *bus, u8 *devfn)
 {
 	if (!PCI_DN(np))
@@ -294,3 +295,4 @@ int pci_device_from_OF_node(struct device_node *np, u8 *bus, u8 *devfn)
 	*devfn = PCI_DN(np)->devfn;
 	return 0;
 }
+#endif
Signature · 1 line
2.20.1
#2 lore raw

Re: [PATCH 1/5] powerpc/pci: Hide pci_device_from_OF_node() for non-powermac code

Quoted 2 lines · Pali Rohár

Function pci_device_from_OF_node() is used only in powermac code. So hide it from all other platforms as it is unsed.

s/unsed/unused/ (same typo in 3/5 patch)

These are for the powerpc folks, so I'm just kibbitzing here.

Quoted 62 lines
---
 arch/powerpc/include/asm/pci-bridge.h | 2 ++
 arch/powerpc/kernel/pci_32.c          | 2 ++
 arch/powerpc/kernel/pci_64.c          | 2 ++
 3 files changed, 6 insertions(+)
arch/powerpc/include/asm/pci-bridge.h +2 -0
index c85f901227c9..98156932a1f5 100644
@@ -170,8 +170,10 @@ static inline struct pci_controller *pci_bus_to_host(const struct pci_bus *bus)
 	return bus->sysdata;
 }
 
+#ifdef CONFIG_PPC_PMAC
 extern int pci_device_from_OF_node(struct device_node *node,
 				   u8 *bus, u8 *devfn);
+#endif
 #ifndef CONFIG_PPC64
 
 extern void pci_create_OF_bus_map(void);
arch/powerpc/kernel/pci_32.c +2 -0
index 5a174936c9a0..c3b91fb62a71 100644
@@ -154,6 +154,7 @@ pcibios_make_OF_bus_map(void)
 }
 
 
+#ifdef CONFIG_PPC_PMAC
 /*
  * Returns the PCI device matching a given OF node
  */
@@ -193,6 +194,7 @@ int pci_device_from_OF_node(struct device_node *node, u8 *bus, u8 *devfn)
 	return -ENODEV;
 }
 EXPORT_SYMBOL(pci_device_from_OF_node);
+#endif
 
 /* We create the "pci-OF-bus-map" property now so it appears in the
  * /proc device tree
arch/powerpc/kernel/pci_64.c +2 -0
index 19b03ddf5631..0c7cfb9fab04 100644
@@ -286,6 +286,7 @@ int pcibus_to_node(struct pci_bus *bus)
 EXPORT_SYMBOL(pcibus_to_node);
 #endif
 
+#ifdef CONFIG_PPC_PMAC
 int pci_device_from_OF_node(struct device_node *np, u8 *bus, u8 *devfn)
 {
 	if (!PCI_DN(np))
@@ -294,3 +295,4 @@ int pci_device_from_OF_node(struct device_node *np, u8 *bus, u8 *devfn)
 	*devfn = PCI_DN(np)->devfn;
 	return 0;
 }
+#endif
Signature · 1 line
2.20.1
#3 lore raw

PATCH 2/5 powerpc/pci: Make pcibios_make_OF_bus_map() static

In reply to Pali Rohár

Function pcibios_make_OF_bus_map() is used only in pci_32.c file. So make it static and do not export out of pci_32.o unit.

---
 arch/powerpc/kernel/pci_32.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
arch/powerpc/kernel/pci_32.c +1 -3
index c3b91fb62a71..df981294df29 100644
@@ -36,8 +36,6 @@ int pcibios_assign_bus_offset = 1;
 EXPORT_SYMBOL(isa_io_base);
 EXPORT_SYMBOL(pci_dram_offset);
 
-void __init pcibios_make_OF_bus_map(void);
-
 static void fixup_cpc710_pci64(struct pci_dev* dev);
 static u8* pci_to_OF_bus_map;
 
@@ -108,7 +106,7 @@ make_one_node_map(struct device_node* node, u8 pci_bus)
 	}
 }
 	
-void __init
+static void __init
 pcibios_make_OF_bus_map(void)
 {
 	int i;
Signature · 1 line
2.20.1
#4 lore raw

PATCH 3/5 powerpc/pci: Hide pci_create_OF_bus_map() for non-chrp code

In reply to Pali Rohár

Function pci_create_OF_bus_map() is used only in chrp code. So hide it from all other platforms as it is unsed.

---
 arch/powerpc/include/asm/pci-bridge.h | 2 ++
 arch/powerpc/kernel/pci_32.c          | 2 ++
 2 files changed, 4 insertions(+)
arch/powerpc/include/asm/pci-bridge.h +2 -0
index 98156932a1f5..e18c95f4e1d4 100644
@@ -176,7 +176,9 @@ extern int pci_device_from_OF_node(struct device_node *node,
 #endif
 #ifndef CONFIG_PPC64
 
+#ifdef CONFIG_PPC_CHRP
 extern void pci_create_OF_bus_map(void);
+#endif
 
 #else	/* CONFIG_PPC64 */
 
arch/powerpc/kernel/pci_32.c +2 -0
index df981294df29..3291af89cea4 100644
@@ -194,6 +194,7 @@ int pci_device_from_OF_node(struct device_node *node, u8 *bus, u8 *devfn)
 EXPORT_SYMBOL(pci_device_from_OF_node);
 #endif
 
+#ifdef CONFIG_PPC_CHRP
 /* We create the "pci-OF-bus-map" property now so it appears in the
  * /proc device tree
  */
@@ -218,6 +219,7 @@ pci_create_OF_bus_map(void)
 		of_node_put(dn);
 	}
 }
+#endif
 
 void pcibios_setup_phb_io_space(struct pci_controller *hose)
 {
Signature · 1 line
2.20.1
#5 lore raw

PATCH 4/5 powerpc/pci: Disable filling pci-OF-bus-map for non-chrp/powermac

In reply to Pali Rohár

Creating or filling pci-OF-bus-map property in the device-tree is deprecated since May 2006 [1] and was used only in old platforms like PowerMac.

Currently kernel code handles it only for chrp and powermac code. So completely disable filling pci-OF-bus-map property for non-chrp and non-powermac platforms.

[1] - https://lore.kernel.org/linuxppc-dev/[email protected]/

---
 arch/powerpc/kernel/pci_32.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
arch/powerpc/kernel/pci_32.c +10 -3
index 3291af89cea4..2f7284b68f06 100644
@@ -37,15 +37,12 @@ EXPORT_SYMBOL(isa_io_base);
 EXPORT_SYMBOL(pci_dram_offset);
 
 static void fixup_cpc710_pci64(struct pci_dev* dev);
-static u8* pci_to_OF_bus_map;
 
 /* By default, we don't re-assign bus numbers. We do this only on
  * some pmacs
  */
 static int pci_assign_all_buses;
 
-static int pci_bus_count;
-
 /* This will remain NULL for now, until isa-bridge.c is made common
  * to both 32-bit and 64-bit.
  */
@@ -65,6 +62,11 @@ fixup_cpc710_pci64(struct pci_dev* dev)
 }
 DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_IBM,	PCI_DEVICE_ID_IBM_CPC710_PCI64,	fixup_cpc710_pci64);
 
+#if defined(CONFIG_PPC_PMAC) || defined(CONFIG_PPC_CHRP)
+
+static u8* pci_to_OF_bus_map;
+static int pci_bus_count;
+
 /*
  * Functions below are used on OpenFirmware machines.
  */
@@ -221,6 +223,8 @@ pci_create_OF_bus_map(void)
 }
 #endif
 
+#endif /* defined(CONFIG_PPC_PMAC) || defined(CONFIG_PPC_CHRP) */
+
 void pcibios_setup_phb_io_space(struct pci_controller *hose)
 {
 	unsigned long io_offset;
@@ -252,6 +256,8 @@ static int __init pcibios_init(void)
 		if (pci_assign_all_buses || next_busno <= hose->last_busno)
 			next_busno = hose->last_busno + pcibios_assign_bus_offset;
 	}
+
+#if defined(CONFIG_PPC_PMAC) || defined(CONFIG_PPC_CHRP)
 	pci_bus_count = next_busno;
 
 	/* OpenFirmware based machines need a map of OF bus
@@ -260,6 +266,7 @@ static int __init pcibios_init(void)
 	 */
 	if (pci_assign_all_buses)
 		pcibios_make_OF_bus_map();
+#endif
 
 	/* Call common code to handle resource allocation */
 	pcibios_resource_survey();
Signature · 1 line
2.20.1
#6 lore raw

PATCH 5/5 powerpc/pci: Add config option for using all 256 PCI buses

In reply to Pali Rohár

By default on PPC32 are PCI bus numbers unique across all PCI domains. So system could have only 256 PCI buses independently of available PCI domains.

This is due to filling DT property pci-OF-bus-map which does not reflect multi-domain setup.

On all powerpc platforms except chrp and powermac there is no DT property pci-OF-bus-map anymore and therefore it is possible on non-chrp/powermac platforms to avoid this limitation of maximal number of 256 PCI buses in system even on multi-domain setup.

But avoiding this limitation would mean that all PCI and PCIe devices would be present on completely different BDF addresses as every PCI domain starts numbering PCI bueses from zero (instead of the last bus number of previous enumerated PCI domain). Such change could break existing software which expects fixed PCI bus numbers.

So add a new config option CONFIG_PPC_PCI_BUS_NUM_DOMAIN_DEPENDENT which enables this change. By default it is disabled. It cause that initial value of hose->first_busno is zero.

---
 arch/powerpc/Kconfig         | 11 +++++++++++
 arch/powerpc/kernel/pci_32.c |  6 ++++++
 2 files changed, 17 insertions(+)
arch/powerpc/Kconfig +11 -0
index be68c1f02b79..f66084bc1dfe 100644
@@ -370,6 +370,17 @@ config PPC_DCR
 	depends on PPC_DCR_NATIVE || PPC_DCR_MMIO
 	default y
 
+config PPC_PCI_BUS_NUM_DOMAIN_DEPENDENT
+	depends on PPC32
+	depends on !PPC_PMAC && !PPC_CHRP
+	bool "Assign PCI bus numbers from zero individually for each PCI domain"
+	help
+	  By default on PPC32 were PCI bus numbers unique across all PCI domains.
+	  So system could have only 256 PCI buses independently of available
+	  PCI domains. When this option is enabled then PCI bus numbers are
+	  PCI domain dependent and each PCI controller on own domain can have
+	  256 PCI buses, like it is on other Linux architectures.
+
 config PPC_OF_PLATFORM_PCI
 	bool
 	depends on PCI
arch/powerpc/kernel/pci_32.c +6 -0
index 2f7284b68f06..433965bf37b4 100644
@@ -239,7 +239,9 @@ void pcibios_setup_phb_io_space(struct pci_controller *hose)
 static int __init pcibios_init(void)
 {
 	struct pci_controller *hose, *tmp;
+#ifndef CONFIG_PPC_PCI_BUS_NUM_DOMAIN_DEPENDENT
 	int next_busno = 0;
+#endif
 
 	printk(KERN_INFO "PCI: Probing PCI hardware\n");
 
@@ -248,13 +250,17 @@ static int __init pcibios_init(void)
 
 	/* Scan all of the recorded PCI controllers.  */
 	list_for_each_entry_safe(hose, tmp, &hose_list, list_node) {
+#ifndef CONFIG_PPC_PCI_BUS_NUM_DOMAIN_DEPENDENT
 		if (pci_assign_all_buses)
 			hose->first_busno = next_busno;
+#endif
 		hose->last_busno = 0xff;
 		pcibios_scan_phb(hose);
 		pci_bus_add_devices(hose->bus);
+#ifndef CONFIG_PPC_PCI_BUS_NUM_DOMAIN_DEPENDENT
 		if (pci_assign_all_buses || next_busno <= hose->last_busno)
 			next_busno = hose->last_busno + pcibios_assign_bus_offset;
+#endif
 	}
 
 #if defined(CONFIG_PPC_PMAC) || defined(CONFIG_PPC_CHRP)
Signature · 1 line
2.20.1
#7 lore raw

Re: [PATCH 5/5] powerpc/pci: Add config option for using all 256 PCI buses

Quoted 37 lines · On Wednesday 06 July 2022 12:43:08 Pali Rohár wrote:

By default on PPC32 are PCI bus numbers unique across all PCI domains. So system could have only 256 PCI buses independently of available PCI domains.

This is due to filling DT property pci-OF-bus-map which does not reflect multi-domain setup.

On all powerpc platforms except chrp and powermac there is no DT property pci-OF-bus-map anymore and therefore it is possible on non-chrp/powermac platforms to avoid this limitation of maximal number of 256 PCI buses in system even on multi-domain setup.

But avoiding this limitation would mean that all PCI and PCIe devices would be present on completely different BDF addresses as every PCI domain starts numbering PCI bueses from zero (instead of the last bus number of previous enumerated PCI domain). Such change could break existing software which expects fixed PCI bus numbers.

So add a new config option CONFIG_PPC_PCI_BUS_NUM_DOMAIN_DEPENDENT which enables this change. By default it is disabled. It cause that initial value of hose->first_busno is zero.

---
 arch/powerpc/Kconfig         | 11 +++++++++++
 arch/powerpc/kernel/pci_32.c |  6 ++++++
 2 files changed, 17 insertions(+)
arch/powerpc/Kconfig +1 -0
index be68c1f02b79..f66084bc1dfe 100644
@@ -370,6 +370,17 @@ config PPC_DCR
 	depends on PPC_DCR_NATIVE || PPC_DCR_MMIO
 	default y
 
+config PPC_PCI_BUS_NUM_DOMAIN_DEPENDENT

arch/powerpc/Kconfig config PPC_DCR · line 383

Quoted 10 lines
arch/powerpc/Kconfig +10 -0
+	depends on PPC32
+	depends on !PPC_PMAC && !PPC_CHRP
+	bool "Assign PCI bus numbers from zero individually for each PCI domain"
+	help
+	  By default on PPC32 were PCI bus numbers unique across all PCI domains.
+	  So system could have only 256 PCI buses independently of available
+	  PCI domains. When this option is enabled then PCI bus numbers are
+	  PCI domain dependent and each PCI controller on own domain can have
+	  256 PCI buses, like it is on other Linux architectures.
+

What do you think, would it be possible to set default value of this option to enabled?

Quoted 38 lines
arch/powerpc/Kconfig +0 -0
 config PPC_OF_PLATFORM_PCI
 	bool
 	depends on PCI
arch/powerpc/kernel/pci_32.c +6 -0
index 2f7284b68f06..433965bf37b4 100644
@@ -239,7 +239,9 @@ void pcibios_setup_phb_io_space(struct pci_controller *hose)
 static int __init pcibios_init(void)
 {
 	struct pci_controller *hose, *tmp;
+#ifndef CONFIG_PPC_PCI_BUS_NUM_DOMAIN_DEPENDENT
 	int next_busno = 0;
+#endif
 
 	printk(KERN_INFO "PCI: Probing PCI hardware\n");
 
@@ -248,13 +250,17 @@ static int __init pcibios_init(void)
 
 	/* Scan all of the recorded PCI controllers.  */
 	list_for_each_entry_safe(hose, tmp, &hose_list, list_node) {
+#ifndef CONFIG_PPC_PCI_BUS_NUM_DOMAIN_DEPENDENT
 		if (pci_assign_all_buses)
 			hose->first_busno = next_busno;
+#endif
 		hose->last_busno = 0xff;
 		pcibios_scan_phb(hose);
 		pci_bus_add_devices(hose->bus);
+#ifndef CONFIG_PPC_PCI_BUS_NUM_DOMAIN_DEPENDENT
 		if (pci_assign_all_buses || next_busno <= hose->last_busno)
 			next_busno = hose->last_busno + pcibios_assign_bus_offset;
+#endif
 	}
 
 #if defined(CONFIG_PPC_PMAC) || defined(CONFIG_PPC_CHRP)
Signature · 1 line
2.20.1
#8 lore raw

Re: [PATCH 5/5] powerpc/pci: Add config option for using all 256 PCI buses

Quoted 37 lines · Pali Rohár
Quoted 36 lines · On Wednesday 06 July 2022 12:43:08 Pali Rohár wrote:

By default on PPC32 are PCI bus numbers unique across all PCI domains. So system could have only 256 PCI buses independently of available PCI domains.

This is due to filling DT property pci-OF-bus-map which does not reflect multi-domain setup.

On all powerpc platforms except chrp and powermac there is no DT property pci-OF-bus-map anymore and therefore it is possible on non-chrp/powermac platforms to avoid this limitation of maximal number of 256 PCI buses in system even on multi-domain setup.

But avoiding this limitation would mean that all PCI and PCIe devices would be present on completely different BDF addresses as every PCI domain starts numbering PCI bueses from zero (instead of the last bus number of previous enumerated PCI domain). Such change could break existing software which expects fixed PCI bus numbers.

So add a new config option CONFIG_PPC_PCI_BUS_NUM_DOMAIN_DEPENDENT which enables this change. By default it is disabled. It cause that initial value of hose->first_busno is zero.

---
 arch/powerpc/Kconfig         | 11 +++++++++++
 arch/powerpc/kernel/pci_32.c |  6 ++++++
 2 files changed, 17 insertions(+)
arch/powerpc/Kconfig +0 -0
index be68c1f02b79..f66084bc1dfe 100644
@@ -370,6 +370,17 @@ config PPC_DCR
 	depends on PPC_DCR_NATIVE || PPC_DCR_MMIO
 	default y
 
Quoted 14 lines

arch/powerpc/Kconfig config PPC_DCR · line 383

Quoted 11 lines
arch/powerpc/Kconfig +11 -0
+config PPC_PCI_BUS_NUM_DOMAIN_DEPENDENT
+	depends on PPC32
+	depends on !PPC_PMAC && !PPC_CHRP
+	bool "Assign PCI bus numbers from zero individually for each PCI domain"
+	help
+	  By default on PPC32 were PCI bus numbers unique across all PCI domains.
+	  So system could have only 256 PCI buses independently of available
+	  PCI domains. When this option is enabled then PCI bus numbers are
+	  PCI domain dependent and each PCI controller on own domain can have
+	  256 PCI buses, like it is on other Linux architectures.
+

What do you think, would it be possible to set default value of this option to enabled?

My preference would be to not have the option at all, just make it the default behaviour. Every new CONFIG option adds more combinations that need testing, or more likely don't get well tested.

But I don't have a good feel for what could break if we turn it on, so honestly I don't really know.

Also I do most of my 32-bit testing on pmacs, which are not affected by the change.

So I'll probably take the series as-is, and then we can do some more widespread testing and possibly flip the default to enabled, and then maybe remove the option entirely in future.

cheers

#9 lore raw

Re: [PATCH 5/5] powerpc/pci: Add config option for using all 256 PCI buses

Quoted 57 lines · On Tuesday 26 July 2022 21:02:22 Michael Ellerman wrote:
Quoted 37 lines · Pali Rohár
Quoted 36 lines · On Wednesday 06 July 2022 12:43:08 Pali Rohár wrote:

By default on PPC32 are PCI bus numbers unique across all PCI domains. So system could have only 256 PCI buses independently of available PCI domains.

This is due to filling DT property pci-OF-bus-map which does not reflect multi-domain setup.

On all powerpc platforms except chrp and powermac there is no DT property pci-OF-bus-map anymore and therefore it is possible on non-chrp/powermac platforms to avoid this limitation of maximal number of 256 PCI buses in system even on multi-domain setup.

But avoiding this limitation would mean that all PCI and PCIe devices would be present on completely different BDF addresses as every PCI domain starts numbering PCI bueses from zero (instead of the last bus number of previous enumerated PCI domain). Such change could break existing software which expects fixed PCI bus numbers.

So add a new config option CONFIG_PPC_PCI_BUS_NUM_DOMAIN_DEPENDENT which enables this change. By default it is disabled. It cause that initial value of hose->first_busno is zero.

---
 arch/powerpc/Kconfig         | 11 +++++++++++
 arch/powerpc/kernel/pci_32.c |  6 ++++++
 2 files changed, 17 insertions(+)
arch/powerpc/Kconfig +0 -0
index be68c1f02b79..f66084bc1dfe 100644
@@ -370,6 +370,17 @@ config PPC_DCR
 	depends on PPC_DCR_NATIVE || PPC_DCR_MMIO
 	default y
 
Quoted 14 lines

arch/powerpc/Kconfig config PPC_DCR · line 383

Quoted 11 lines
arch/powerpc/Kconfig +11 -0
+config PPC_PCI_BUS_NUM_DOMAIN_DEPENDENT
+	depends on PPC32
+	depends on !PPC_PMAC && !PPC_CHRP
+	bool "Assign PCI bus numbers from zero individually for each PCI domain"
+	help
+	  By default on PPC32 were PCI bus numbers unique across all PCI domains.
+	  So system could have only 256 PCI buses independently of available
+	  PCI domains. When this option is enabled then PCI bus numbers are
+	  PCI domain dependent and each PCI controller on own domain can have
+	  256 PCI buses, like it is on other Linux architectures.
+

What do you think, would it be possible to set default value of this option to enabled?

My preference would be to not have the option at all, just make it the default behaviour. Every new CONFIG option adds more combinations that need testing, or more likely don't get well tested.

Quoted 5 lines

But I don't have a good feel for what could break if we turn it on, so honestly I don't really know.

Also I do most of my 32-bit testing on pmacs, which are not affected by the change.

It is because this change is incompatible with deprecated pci-OF-bus-map which pmac uses. I do not have any pmac box for testing or development, so I let this part as is.

If one day pci-OF-bus-map would be possible to disable on pmac then this pci bus number change can be enabled also for pmac.

Quoted 5 lines

So I'll probably take the series as-is, and then we can do some more widespread testing and possibly flip the default to enabled, and then maybe remove the option entirely in future.

cheers

I have tested it on P2020 board with 3 PCIe devices, each on own bus where each bus is connected to different PCIe controller / domain and it works correctly. Every PCIe device is on bus 1 bus but on different domains.

#10 lore raw

Re: [PATCH 5/5] powerpc/pci: Add config option for using all 256 PCI buses

Quoted 64 lines · On Tuesday 26 July 2022 13:10:01 Pali Rohár wrote:
Quoted 57 lines · On Tuesday 26 July 2022 21:02:22 Michael Ellerman wrote:
Quoted 37 lines · Pali Rohár
Quoted 36 lines · On Wednesday 06 July 2022 12:43:08 Pali Rohár wrote:

By default on PPC32 are PCI bus numbers unique across all PCI domains. So system could have only 256 PCI buses independently of available PCI domains.

This is due to filling DT property pci-OF-bus-map which does not reflect multi-domain setup.

On all powerpc platforms except chrp and powermac there is no DT property pci-OF-bus-map anymore and therefore it is possible on non-chrp/powermac platforms to avoid this limitation of maximal number of 256 PCI buses in system even on multi-domain setup.

But avoiding this limitation would mean that all PCI and PCIe devices would be present on completely different BDF addresses as every PCI domain starts numbering PCI bueses from zero (instead of the last bus number of previous enumerated PCI domain). Such change could break existing software which expects fixed PCI bus numbers.

So add a new config option CONFIG_PPC_PCI_BUS_NUM_DOMAIN_DEPENDENT which enables this change. By default it is disabled. It cause that initial value of hose->first_busno is zero.

---
 arch/powerpc/Kconfig         | 11 +++++++++++
 arch/powerpc/kernel/pci_32.c |  6 ++++++
 2 files changed, 17 insertions(+)
arch/powerpc/Kconfig +0 -0
index be68c1f02b79..f66084bc1dfe 100644
@@ -370,6 +370,17 @@ config PPC_DCR
 	depends on PPC_DCR_NATIVE || PPC_DCR_MMIO
 	default y
 
Quoted 14 lines

arch/powerpc/Kconfig config PPC_DCR · line 383

Quoted 11 lines
arch/powerpc/Kconfig +11 -0
+config PPC_PCI_BUS_NUM_DOMAIN_DEPENDENT
+	depends on PPC32
+	depends on !PPC_PMAC && !PPC_CHRP
+	bool "Assign PCI bus numbers from zero individually for each PCI domain"
+	help
+	  By default on PPC32 were PCI bus numbers unique across all PCI domains.
+	  So system could have only 256 PCI buses independently of available
+	  PCI domains. When this option is enabled then PCI bus numbers are
+	  PCI domain dependent and each PCI controller on own domain can have
+	  256 PCI buses, like it is on other Linux architectures.
+

What do you think, would it be possible to set default value of this option to enabled?

My preference would be to not have the option at all, just make it the default behaviour. Every new CONFIG option adds more combinations that need testing, or more likely don't get well tested.

Quoted 5 lines

But I don't have a good feel for what could break if we turn it on, so honestly I don't really know.

Also I do most of my 32-bit testing on pmacs, which are not affected by the change.

Quoted 6 lines

It is because this change is incompatible with deprecated pci-OF-bus-map which pmac uses. I do not have any pmac box for testing or development, so I let this part as is.

If one day pci-OF-bus-map would be possible to disable on pmac then this pci bus number change can be enabled also for pmac.

Hello! I have created this patch which allows to disable deprecated pci-OF-bus-map on powermac and allow to enable this new config option PPC_PCI_BUS_NUM_DOMAIN_DEPENDENT also on powermac.

So you can test this option too on your powermac boxes.

I'm really not sure if that pci-OF-bus-map is required and for which platforms or software...

Patch for allowing to disable pci-OF-bus-map is here:
https://patchwork.ozlabs.org/project/linuxppc-dev/patch/[email protected]/

Quoted 10 lines
Quoted 5 lines

So I'll probably take the series as-is, and then we can do some more widespread testing and possibly flip the default to enabled, and then maybe remove the option entirely in future.

cheers

I have tested it on P2020 board with 3 PCIe devices, each on own bus where each bus is connected to different PCIe controller / domain and it works correctly. Every PCIe device is on bus 1 bus but on different domains.

#11 lore raw
In reply to Pali Rohár
Quoted 9 lines · Pali Rohár

This patch series cleanup unused code by eliminating it at compile time and then enable usage of all 256 PCI buses per every PCI domain as currently PCI bus numbers have to be unique across all PCI domains. So first bus number of each PCI domain would be zero and not the bus number of the previous domain plus one. As such bus assignment changes BDF address of every PCI device on multi-domain system, this new "feature" is configurable by config option to prevent regressions.

[...]

Applied to powerpc/next.

[1/5] powerpc/pci: Hide pci_device_from_OF_node() for non-powermac code

      https://git.kernel.org/powerpc/c/a2954a7e47b60bb8d8c51f1b4d55af7585a4108a
[2/5] powerpc/pci: Make pcibios_make_OF_bus_map() static
      https://git.kernel.org/powerpc/c/407a767182d3dc87176aabddd7c109b36a5727e9
[3/5] powerpc/pci: Hide pci_create_OF_bus_map() for non-chrp code
      https://git.kernel.org/powerpc/c/704544588735b2e1115212dbba1c210b3687ff7a
[4/5] powerpc/pci: Disable filling pci-OF-bus-map for non-chrp/powermac
      https://git.kernel.org/powerpc/c/7f102d61983275ab68b5ac2715afa35c5f4ffd86
[5/5] powerpc/pci: Add config option for using all 256 PCI buses
      https://git.kernel.org/powerpc/c/5663568130825458a2a8535ccef0db9a1bf7be82

cheers