Skip to contentSkip to conversation

PCI: hv: Support host bridge probing on ARM64

PATCH Applied Boqun Feng 4 participants 28 messages Jul 26, 2021 – Aug 23, 2021

Subsystems (5)
lore
Share & tools
lore

Boqun Feng

author
#0 lore raw

v6 cover PCI: hv: Support host bridge probing on ARM64

Hi,

This is the v6 for the preparation of virtual PCI support on Hyper-V ARM64, Previous versions:

v1: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/lkml/[email protected]/
v3: https://lore.kernel.org/lkml/[email protected]/
v4: https://lore.kernel.org/lkml/[email protected]/
v5: https://lore.kernel.org/lkml/[email protected]/

Changes since last version:

* Rebase to 5.14-rc3

* Comment fixes as suggested by Bjorn.

The basic problem we need to resolve is that ARM64 is an arch with PCI_DOMAINS_GENERIC=y, so the bus sysdata is pci_config_window. However, Hyper-V PCI provides a paravirtualized PCI interface, so there is no actual pci_config_window for a PCI host bridge, so no information can be retrieve from the pci_config_window of a Hyper-V virtual PCI bus. Also there is no corresponding ACPI device for the Hyper-V PCI root bridge, which introduces a special case when trying to find the ACPI device from the sysdata (see patch #3).

With this patchset, we could enable the virtual PCI on Hyper-V ARM64 guest with other code under development.

Comments and suggestions are welcome.

Regards,
Boqun

Arnd Bergmann (1):
PCI: hv: Generify PCI probing

Boqun Feng (7):
PCI: Introduce domain_nr in pci_host_bridge
PCI: Support populating MSI domains of root buses via bridges
arm64: PCI: Restructure pcibios_root_bridge_prepare()
arm64: PCI: Support root bridge preparation for Hyper-V
PCI: hv: Set ->domain_nr of pci_host_bridge at probing time
PCI: hv: Set up MSI domain at bridge probing time
PCI: hv: Turn on the host bridge probing on ARM64

 arch/arm64/kernel/pci.c             | 29 +++++++---
 drivers/pci/controller/pci-hyperv.c | 86 +++++++++++++++++------------
 drivers/pci/probe.c                 | 12 +++-
 include/linux/pci.h                 | 11 ++++
 4 files changed, 93 insertions(+), 45 deletions(-)
Signature · 1 line
2.32.0
author
#1 lore raw

PATCH v6 1/8 PCI: Introduce domain_nr in pci_host_bridge

Currently we retrieve the PCI domain number of the host bridge from the bus sysdata (or pci_config_window if PCI_DOMAINS_GENERIC=y). Actually we have the information at PCI host bridge probing time, and it makes sense that we store it into pci_host_bridge. One benefit of doing so is the requirement for supporting PCI on Hyper-V for ARM64, because the host bridge of Hyper-V doesn't have pci_config_window, whereas ARM64 is a PCI_DOMAINS_GENERIC=y arch, so we cannot retrieve the PCI domain number from pci_config_window on ARM64 Hyper-V guest.

As the preparation for ARM64 Hyper-V PCI support, we introduce the domain_nr in pci_host_bridge and a sentinel value to allow drivers to set domain numbers properly at probing time. Currently CONFIG_PCI_DOMAINS_GENERIC=y archs are only users of this newly-introduced field.

---
 drivers/pci/probe.c |  6 +++++-
 include/linux/pci.h | 11 +++++++++++
 2 files changed, 16 insertions(+), 1 deletion(-)
drivers/pci/probe.c +5 -1
index 79177ac37880..60c50d4f156f 100644
@@ -594,6 +594,7 @@ static void pci_init_host_bridge(struct pci_host_bridge *bridge)
 	bridge->native_pme = 1;
 	bridge->native_ltr = 1;
 	bridge->native_dpc = 1;
+	bridge->domain_nr = PCI_DOMAIN_NR_NOT_SET;
 
 	device_initialize(&bridge->dev);
 }
@@ -898,7 +899,10 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge)
 	bus->ops = bridge->ops;
 	bus->number = bus->busn_res.start = bridge->busnr;
 #ifdef CONFIG_PCI_DOMAINS_GENERIC
-	bus->domain_nr = pci_bus_find_domain_nr(bus, parent);
+	if (bridge->domain_nr == PCI_DOMAIN_NR_NOT_SET)
+		bus->domain_nr = pci_bus_find_domain_nr(bus, parent);
+	else
+		bus->domain_nr = bridge->domain_nr;
 #endif
 
 	b = pci_find_bus(pci_domain_nr(bus), bridge->busnr);
include/linux/pci.h +11 -0
index 540b377ca8f6..01aa201e1df0 100644
@@ -526,6 +526,16 @@ static inline int pci_channel_offline(struct pci_dev *pdev)
 	return (pdev->error_state != pci_channel_io_normal);
 }
 
+/*
+ * Currently in ACPI spec, for each PCI host bridge, PCI Segment
+ * Group number is limited to a 16-bit value, therefore (int)-1 is
+ * not a valid PCI domain number, and can be used as a sentinel
+ * value indicating ->domain_nr is not set by the driver (and
+ * CONFIG_PCI_DOMAINS_GENERIC=y archs will set it with
+ * pci_bus_find_domain_nr()).
+ */
+#define PCI_DOMAIN_NR_NOT_SET (-1)
+
 struct pci_host_bridge {
 	struct device	dev;
 	struct pci_bus	*bus;		/* Root bus */
@@ -533,6 +543,7 @@ struct pci_host_bridge {
 	struct pci_ops	*child_ops;
 	void		*sysdata;
 	int		busnr;
+	int		domain_nr;
 	struct list_head windows;	/* resource_entry */
 	struct list_head dma_ranges;	/* dma ranges resource list */
 	u8 (*swizzle_irq)(struct pci_dev *, u8 *); /* Platform IRQ swizzler */
Signature · 1 line
2.32.0
author
#2 lore raw

PATCH v6 2/8 PCI: Support populating MSI domains of root buses via bridges

In reply to Boqun Feng

Currently, at probing time, the MSI domains of root buses are populated if either the information of MSI domain is available from firmware (DT or ACPI), or arch-specific sysdata is used to pass the fwnode of the MSI domain. These two conditions don't cover all, e.g. Hyper-V virtual PCI on ARM64, which doesn't have the MSI information in the firmware and couldn't use arch-specific sysdata because running on an architecture with PCI_DOMAINS_GENERIC=y.

To support populating MSI domains of the root buses at the probing when neither of the above condition is true, the ->msi_domain of the corresponding bridge device is used: in pci_host_bridge_msi_domain(), which should return the MSI domain of the root bus, the ->msi_domain of the corresponding bridge is fetched first as a potential value of the MSI domain of the root bus.

In order to use the approach to populate MSI domains, the driver needs to dev_set_msi_domain() on the bridge before calling pci_register_host_bridge(), and makes sure GENERIC_MSI_IRQ_DOMAIN=y.

Another advantage of this new approach is providing an arch-independent way to populate MSI domains, which allows sharing the driver code as much as possible between architectures.

---
 drivers/pci/probe.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/pci/probe.c +5 -1
index 60c50d4f156f..ea7f2a57e2f5 100644
@@ -829,11 +829,15 @@ static struct irq_domain *pci_host_bridge_msi_domain(struct pci_bus *bus)
 {
 	struct irq_domain *d;
 
+	/* If the host bridge driver sets a MSI domain of the bridge, use it */
+	d = dev_get_msi_domain(bus->bridge);
+
 	/*
 	 * Any firmware interface that can resolve the msi_domain
 	 * should be called from here.
 	 */
-	d = pci_host_bridge_of_msi_domain(bus);
+	if (!d)
+		d = pci_host_bridge_of_msi_domain(bus);
 	if (!d)
 		d = pci_host_bridge_acpi_msi_domain(bus);
 
Signature · 1 line
2.32.0
Acked-by on v6 2/8
#3 lore raw

Re: [PATCH v6 2/8] PCI: Support populating MSI domains of root buses via bridges

Quoted 19 lines · Boqun Feng

Currently, at probing time, the MSI domains of root buses are populated if either the information of MSI domain is available from firmware (DT or ACPI), or arch-specific sysdata is used to pass the fwnode of the MSI domain. These two conditions don't cover all, e.g. Hyper-V virtual PCI on ARM64, which doesn't have the MSI information in the firmware and couldn't use arch-specific sysdata because running on an architecture with PCI_DOMAINS_GENERIC=y.

To support populating MSI domains of the root buses at the probing when neither of the above condition is true, the ->msi_domain of the corresponding bridge device is used: in pci_host_bridge_msi_domain(), which should return the MSI domain of the root bus, the ->msi_domain of the corresponding bridge is fetched first as a potential value of the MSI domain of the root bus.

In order to use the approach to populate MSI domains, the driver needs to dev_set_msi_domain() on the bridge before calling pci_register_host_bridge(), and makes sure GENERIC_MSI_IRQ_DOMAIN=y.

Quoted 6 lines

Another advantage of this new approach is providing an arch-independent way to populate MSI domains, which allows sharing the driver code as much as possible between architectures.

Quoted 28 lines
---
 drivers/pci/probe.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/pci/probe.c +5 -1
index 60c50d4f156f..ea7f2a57e2f5 100644
@@ -829,11 +829,15 @@ static struct irq_domain *pci_host_bridge_msi_domain(struct pci_bus *bus)
 {
 	struct irq_domain *d;
 
+	/* If the host bridge driver sets a MSI domain of the bridge, use it */
+	d = dev_get_msi_domain(bus->bridge);
+
 	/*
 	 * Any firmware interface that can resolve the msi_domain
 	 * should be called from here.
 	 */
-	d = pci_host_bridge_of_msi_domain(bus);
+	if (!d)
+		d = pci_host_bridge_of_msi_domain(bus);
 	if (!d)
 		d = pci_host_bridge_acpi_msi_domain(bus);
 
Signature · 1 line
2.32.0
author
#4 lore raw

PATCH v6 3/8 arm64: PCI: Restructure pcibios_root_bridge_prepare()

In reply to Boqun Feng

Restructure the pcibios_root_bridge_prepare() as the preparation for supporting cases when no real ACPI device is related to the PCI host bridge.

No functional change.

---
 arch/arm64/kernel/pci.c | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)
arch/arm64/kernel/pci.c +12 -7
index 1006ed2d7c60..5148ae242780 100644
@@ -82,14 +82,19 @@ int acpi_pci_bus_find_domain_nr(struct pci_bus *bus)
 
 int pcibios_root_bridge_prepare(struct pci_host_bridge *bridge)
 {
-	if (!acpi_disabled) {
-		struct pci_config_window *cfg = bridge->bus->sysdata;
-		struct acpi_device *adev = to_acpi_device(cfg->parent);
-		struct device *bus_dev = &bridge->bus->dev;
+	struct pci_config_window *cfg;
+	struct acpi_device *adev;
+	struct device *bus_dev;
 
-		ACPI_COMPANION_SET(&bridge->dev, adev);
-		set_dev_node(bus_dev, acpi_get_node(acpi_device_handle(adev)));
-	}
+	if (acpi_disabled)
+		return 0;
+
+	cfg = bridge->bus->sysdata;
+	adev = to_acpi_device(cfg->parent);
+	bus_dev = &bridge->bus->dev;
+
+	ACPI_COMPANION_SET(&bridge->dev, adev);
+	set_dev_node(bus_dev, acpi_get_node(acpi_device_handle(adev)));
 
 	return 0;
 }
Signature · 1 line
2.32.0
author
#6 lore raw

PATCH v6 4/8 arm64: PCI: Support root bridge preparation for Hyper-V

In reply to Boqun Feng

Currently at root bridge preparation, the corresponding ACPI device will be set as the companion, however for a Hyper-V virtual PCI root bridge, there is no corresponding ACPI device, because a Hyper-V virtual PCI root bridge is discovered via VMBus rather than ACPI table. In order to support this, we need to make pcibios_root_bridge_prepare() work with cfg->parent being NULL.

Use a NULL pointer as the ACPI device if there is no corresponding ACPI device, and this is fine because: 1) ACPI_COMPANION_SET() can work with the second parameter being NULL, 2) semantically, if a NULL pointer is set via ACPI_COMPANION_SET(), ACPI_COMPANION() (the read API for this field) will return NULL, and since ACPI_COMPANION() may return NULL, so users must have handled the cases where it returns NULL, and 3) since there is no corresponding ACPI device, it would be wrong to use any other value here.

---
 arch/arm64/kernel/pci.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
arch/arm64/kernel/pci.c +11 -1
index 5148ae242780..2276689b5411 100644
@@ -90,7 +90,17 @@ int pcibios_root_bridge_prepare(struct pci_host_bridge *bridge)
 		return 0;
 
 	cfg = bridge->bus->sysdata;
-	adev = to_acpi_device(cfg->parent);
+
+	/*
+	 * On Hyper-V there is no corresponding ACPI device for a root bridge,
+	 * therefore ->parent is set as NULL by the driver. And set 'adev' as
+	 * NULL in this case because there is no proper ACPI device.
+	 */
+	if (!cfg->parent)
+		adev = NULL;
+	else
+		adev = to_acpi_device(cfg->parent);
+
 	bus_dev = &bridge->bus->dev;
 
 	ACPI_COMPANION_SET(&bridge->dev, adev);
Signature · 1 line
2.32.0
author
#7 lore raw

Re: [PATCH v6 4/8] arm64: PCI: Support root bridge preparation for Hyper-V

Catalin and Will,

Appreciate it that you can have a look at this one and patch #4, note that there exists an alternative solution at[1].

The difference is the way used to pass the corresponding ACPI device pointers for PCI host bridges: currently pci_config_window->parent is used, and this patch and patch #4 allow the field to be NULL, because Hyper-V's PCI host bridges don't have ACPI devices, while [1] changes to use pci_host_bridge->private. And I'm OK with either way, I don't have a strong opinion here ;-)

Looking forwards to your suggestion, Thanks!

Regards,
Boqun

[1]: https://lore.kernel.org/lkml/[email protected]/

Quoted 47 lines · Boqun Feng

Currently at root bridge preparation, the corresponding ACPI device will be set as the companion, however for a Hyper-V virtual PCI root bridge, there is no corresponding ACPI device, because a Hyper-V virtual PCI root bridge is discovered via VMBus rather than ACPI table. In order to support this, we need to make pcibios_root_bridge_prepare() work with cfg->parent being NULL.

Use a NULL pointer as the ACPI device if there is no corresponding ACPI device, and this is fine because: 1) ACPI_COMPANION_SET() can work with the second parameter being NULL, 2) semantically, if a NULL pointer is set via ACPI_COMPANION_SET(), ACPI_COMPANION() (the read API for this field) will return NULL, and since ACPI_COMPANION() may return NULL, so users must have handled the cases where it returns NULL, and 3) since there is no corresponding ACPI device, it would be wrong to use any other value here.

---
 arch/arm64/kernel/pci.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
arch/arm64/kernel/pci.c +11 -1
index 5148ae242780..2276689b5411 100644
@@ -90,7 +90,17 @@ int pcibios_root_bridge_prepare(struct pci_host_bridge *bridge)
 		return 0;
 
 	cfg = bridge->bus->sysdata;
-	adev = to_acpi_device(cfg->parent);
+
+	/*
+	 * On Hyper-V there is no corresponding ACPI device for a root bridge,
+	 * therefore ->parent is set as NULL by the driver. And set 'adev' as
+	 * NULL in this case because there is no proper ACPI device.
+	 */
+	if (!cfg->parent)
+		adev = NULL;
+	else
+		adev = to_acpi_device(cfg->parent);
+
 	bus_dev = &bridge->bus->dev;
 
 	ACPI_COMPANION_SET(&bridge->dev, adev);
Signature · 1 line
2.32.0
maintainer
#8 lore raw

Re: [PATCH v6 4/8] arm64: PCI: Support root bridge preparation for Hyper-V

Hi Boqun,

Sorry, I just got back from holiday and I'm still in the deleting emails mode.

Quoted 9 lines · Boqun Feng

Appreciate it that you can have a look at this one and patch #4, note that there exists an alternative solution at[1].

The difference is the way used to pass the corresponding ACPI device pointers for PCI host bridges: currently pci_config_window->parent is used, and this patch and patch #4 allow the field to be NULL, because Hyper-V's PCI host bridges don't have ACPI devices, while [1] changes to use pci_host_bridge->private. And I'm OK with either way, I don't have a strong opinion here ;-)

[...]

Quoted 1 line

[1]: https://lore.kernel.org/lkml/[email protected]/

I'm ok with the arm64 bits in this series and the one you linked above. It's up to Lorenzo if he's happy with how pci-hyperv.c ends up looking, I'm not a PCIe expert. My preference would be for a combined series (this and [1] above).

Happy to ack the arm64 patches in a combined series (if you are going to post one), the changes would look even simpler.

Signature · 1 line
Catalin
#9 lore raw

Re: [PATCH v6 4/8] arm64: PCI: Support root bridge preparation for Hyper-V

Quoted 18 lines · Catalin Marinas

Hi Boqun,

Sorry, I just got back from holiday and I'm still in the deleting emails mode.

Quoted 9 lines · Boqun Feng

Appreciate it that you can have a look at this one and patch #4, note that there exists an alternative solution at[1].

The difference is the way used to pass the corresponding ACPI device pointers for PCI host bridges: currently pci_config_window->parent is used, and this patch and patch #4 allow the field to be NULL, because Hyper-V's PCI host bridges don't have ACPI devices, while [1] changes to use pci_host_bridge->private. And I'm OK with either way, I don't have a strong opinion here ;-)

[...]

Quoted 1 line

[1]: https://lore.kernel.org/lkml/[email protected]/

Quoted 7 lines

I'm ok with the arm64 bits in this series and the one you linked above. It's up to Lorenzo if he's happy with how pci-hyperv.c ends up looking, I'm not a PCIe expert. My preference would be for a combined series (this and [1] above).

Happy to ack the arm64 patches in a combined series (if you are going to post one), the changes would look even simpler.

I believe [1] above is an experiment - therefore it is best to stick to this series as it is for the time being, pending refactoring that requires more time, I would not rush it.

If you can ACK the arm64 patches (3,4) please I will pull the series into the PCI tree asap.

Thanks,
Lorenzo

maintainer Acked-by on v6 4/8
#10 lore raw

Re: [PATCH v6 4/8] arm64: PCI: Support root bridge preparation for Hyper-V

In reply to Boqun Feng
Quoted 16 lines · Boqun Feng

Currently at root bridge preparation, the corresponding ACPI device will be set as the companion, however for a Hyper-V virtual PCI root bridge, there is no corresponding ACPI device, because a Hyper-V virtual PCI root bridge is discovered via VMBus rather than ACPI table. In order to support this, we need to make pcibios_root_bridge_prepare() work with cfg->parent being NULL.

Use a NULL pointer as the ACPI device if there is no corresponding ACPI device, and this is fine because: 1) ACPI_COMPANION_SET() can work with the second parameter being NULL, 2) semantically, if a NULL pointer is set via ACPI_COMPANION_SET(), ACPI_COMPANION() (the read API for this field) will return NULL, and since ACPI_COMPANION() may return NULL, so users must have handled the cases where it returns NULL, and 3) since there is no corresponding ACPI device, it would be wrong to use any other value here.

Quoted 1 line
author
#11 lore raw

PATCH v6 5/8 PCI: hv: Generify PCI probing

In reply to Boqun Feng

From: Arnd Bergmann <[email protected]>

In order to support ARM64 Hyper-V PCI, we need to set up the bridge at probing time because ARM64 is a PCI_DOMAIN_GENERIC=y arch and we don't have pci_config_window (ARM64 sysdata) for a PCI root bus on Hyper-V, so it's impossible to retrieve the information (e.g. PCI domains, MSI domains) from bus sysdata on ARM64 after creation.

Originally in create_root_hv_pci_bus(), pci_create_root_bus() is used to create the root bus and the corresponding bridge based on x86 sysdata. Now we create a bridge first and then call pci_scan_root_bus_bridge(), which allows us to do the necessary set-ups for the bridge.

---
 drivers/pci/controller/pci-hyperv.c | 57 +++++++++++++++--------------
 1 file changed, 30 insertions(+), 27 deletions(-)
drivers/pci/controller/pci-hyperv.c +30 -27
index a53bd8728d0d..8d42da5dd1d4 100644
@@ -449,6 +449,7 @@ enum hv_pcibus_state {
 
 struct hv_pcibus_device {
 	struct pci_sysdata sysdata;
+	struct pci_host_bridge *bridge;
 	/* Protocol version negotiated with the host */
 	enum pci_protocol_version_t protocol_version;
 	enum hv_pcibus_state state;
@@ -464,8 +465,6 @@ struct hv_pcibus_device {
 	spinlock_t device_list_lock;	/* Protect lists below */
 	void __iomem *cfg_addr;
 
-	struct list_head resources_for_children;
-
 	struct list_head children;
 	struct list_head dr_list;
 
@@ -1797,7 +1796,7 @@ static void hv_pci_assign_slots(struct hv_pcibus_device *hbus)
 
 		slot_nr = PCI_SLOT(wslot_to_devfn(hpdev->desc.win_slot.slot));
 		snprintf(name, SLOT_NAME_SIZE, "%u", hpdev->desc.ser);
-		hpdev->pci_slot = pci_create_slot(hbus->pci_bus, slot_nr,
+		hpdev->pci_slot = pci_create_slot(hbus->bridge->bus, slot_nr,
 					  name, NULL);
 		if (IS_ERR(hpdev->pci_slot)) {
 			pr_warn("pci_create slot %s failed\n", name);
@@ -1827,7 +1826,7 @@ static void hv_pci_remove_slots(struct hv_pcibus_device *hbus)
 static void hv_pci_assign_numa_node(struct hv_pcibus_device *hbus)
 {
 	struct pci_dev *dev;
-	struct pci_bus *bus = hbus->pci_bus;
+	struct pci_bus *bus = hbus->bridge->bus;
 	struct hv_pci_dev *hv_dev;
 
 	list_for_each_entry(dev, &bus->devices, bus_list) {
@@ -1850,21 +1849,22 @@ static void hv_pci_assign_numa_node(struct hv_pcibus_device *hbus)
  */
 static int create_root_hv_pci_bus(struct hv_pcibus_device *hbus)
 {
-	/* Register the device */
-	hbus->pci_bus = pci_create_root_bus(&hbus->hdev->device,
-					    0, /* bus number is always zero */
-					    &hv_pcifront_ops,
-					    &hbus->sysdata,
-					    &hbus->resources_for_children);
-	if (!hbus->pci_bus)
-		return -ENODEV;
+	int error;
+	struct pci_host_bridge *bridge = hbus->bridge;
+
+	bridge->dev.parent = &hbus->hdev->device;
+	bridge->sysdata = &hbus->sysdata;
+	bridge->ops = &hv_pcifront_ops;
+
+	error = pci_scan_root_bus_bridge(bridge);
+	if (error)
+		return error;
 
 	pci_lock_rescan_remove();
-	pci_scan_child_bus(hbus->pci_bus);
 	hv_pci_assign_numa_node(hbus);
-	pci_bus_assign_resources(hbus->pci_bus);
+	pci_bus_assign_resources(bridge->bus);
 	hv_pci_assign_slots(hbus);
-	pci_bus_add_devices(hbus->pci_bus);
+	pci_bus_add_devices(bridge->bus);
 	pci_unlock_rescan_remove();
 	hbus->state = hv_pcibus_installed;
 	return 0;
@@ -2127,7 +2127,7 @@ static void pci_devices_present_work(struct work_struct *work)
 		 * because there may have been changes.
 		 */
 		pci_lock_rescan_remove();
-		pci_scan_child_bus(hbus->pci_bus);
+		pci_scan_child_bus(hbus->bridge->bus);
 		hv_pci_assign_numa_node(hbus);
 		hv_pci_assign_slots(hbus);
 		pci_unlock_rescan_remove();
@@ -2295,8 +2295,8 @@ static void hv_eject_device_work(struct work_struct *work)
 	/*
 	 * Ejection can come before or after the PCI bus has been set up, so
 	 * attempt to find it and tear down the bus state, if it exists.  This
-	 * must be done without constructs like pci_domain_nr(hbus->pci_bus)
-	 * because hbus->pci_bus may not exist yet.
+	 * must be done without constructs like pci_domain_nr(hbus->bridge->bus)
+	 * because hbus->bridge->bus may not exist yet.
 	 */
 	wslot = wslot_to_devfn(hpdev->desc.win_slot.slot);
 	pdev = pci_get_domain_bus_and_slot(hbus->sysdata.domain, 0, wslot);
@@ -2662,8 +2662,7 @@ static int hv_pci_allocate_bridge_windows(struct hv_pcibus_device *hbus)
 		/* Modify this resource to become a bridge window. */
 		hbus->low_mmio_res->flags |= IORESOURCE_WINDOW;
 		hbus->low_mmio_res->flags &= ~IORESOURCE_BUSY;
-		pci_add_resource(&hbus->resources_for_children,
-				 hbus->low_mmio_res);
+		pci_add_resource(&hbus->bridge->windows, hbus->low_mmio_res);
 	}
 
 	if (hbus->high_mmio_space) {
@@ -2682,8 +2681,7 @@ static int hv_pci_allocate_bridge_windows(struct hv_pcibus_device *hbus)
 		/* Modify this resource to become a bridge window. */
 		hbus->high_mmio_res->flags |= IORESOURCE_WINDOW;
 		hbus->high_mmio_res->flags &= ~IORESOURCE_BUSY;
-		pci_add_resource(&hbus->resources_for_children,
-				 hbus->high_mmio_res);
+		pci_add_resource(&hbus->bridge->windows, hbus->high_mmio_res);
 	}
 
 	return 0;
@@ -3002,6 +3000,7 @@ static void hv_put_dom_num(u16 dom)
 static int hv_pci_probe(struct hv_device *hdev,
 			const struct hv_vmbus_device_id *dev_id)
 {
+	struct pci_host_bridge *bridge;
 	struct hv_pcibus_device *hbus;
 	u16 dom_req, dom;
 	char *name;
@@ -3014,6 +3013,10 @@ static int hv_pci_probe(struct hv_device *hdev,
 	 */
 	BUILD_BUG_ON(sizeof(*hbus) > HV_HYP_PAGE_SIZE);
 
+	bridge = devm_pci_alloc_host_bridge(&hdev->device, 0);
+	if (!bridge)
+		return -ENOMEM;
+
 	/*
 	 * With the recent 59bb47985c1d ("mm, sl[aou]b: guarantee natural
 	 * alignment for kmalloc(power-of-two)"), kzalloc() is able to allocate
@@ -3035,6 +3038,8 @@ static int hv_pci_probe(struct hv_device *hdev,
 	hbus = kzalloc(HV_HYP_PAGE_SIZE, GFP_KERNEL);
 	if (!hbus)
 		return -ENOMEM;
+
+	hbus->bridge = bridge;
 	hbus->state = hv_pcibus_init;
 	hbus->wslot_res_allocated = -1;
 
@@ -3071,7 +3076,6 @@ static int hv_pci_probe(struct hv_device *hdev,
 	hbus->hdev = hdev;
 	INIT_LIST_HEAD(&hbus->children);
 	INIT_LIST_HEAD(&hbus->dr_list);
-	INIT_LIST_HEAD(&hbus->resources_for_children);
 	spin_lock_init(&hbus->config_lock);
 	spin_lock_init(&hbus->device_list_lock);
 	spin_lock_init(&hbus->retarget_msi_interrupt_lock);
@@ -3295,9 +3299,9 @@ static int hv_pci_remove(struct hv_device *hdev)
 
 		/* Remove the bus from PCI's point of view. */
 		pci_lock_rescan_remove();
-		pci_stop_root_bus(hbus->pci_bus);
+		pci_stop_root_bus(hbus->bridge->bus);
 		hv_pci_remove_slots(hbus);
-		pci_remove_root_bus(hbus->pci_bus);
+		pci_remove_root_bus(hbus->bridge->bus);
 		pci_unlock_rescan_remove();
 	}
 
@@ -3307,7 +3311,6 @@ static int hv_pci_remove(struct hv_device *hdev)
 
 	iounmap(hbus->cfg_addr);
 	hv_free_config_window(hbus);
-	pci_free_resource_list(&hbus->resources_for_children);
 	hv_pci_free_bridge_windows(hbus);
 	irq_domain_remove(hbus->irq_domain);
 	irq_domain_free_fwnode(hbus->sysdata.fwnode);
@@ -3390,7 +3393,7 @@ static int hv_pci_restore_msi_msg(struct pci_dev *pdev, void *arg)
  */
 static void hv_pci_restore_msi_state(struct hv_pcibus_device *hbus)
 {
-	pci_walk_bus(hbus->pci_bus, hv_pci_restore_msi_msg, NULL);
+	pci_walk_bus(hbus->bridge->bus, hv_pci_restore_msi_msg, NULL);
 }
 
 static int hv_pci_resume(struct hv_device *hdev)
Signature · 1 line
2.32.0
author
#12 lore raw

PATCH v6 6/8 PCI: hv: Set ->domain_nr of pci_host_bridge at probing time

In reply to Boqun Feng

No functional change, just store and maintain the PCI domain number in the ->domain_nr of pci_host_bridge. Note that we still need to keep the copy of domain number in x86-specific pci_sysdata, because x86 is not a PCI_DOMAINS_GENERIC=y architecture, so the ->domain_nr of pci_host_bridge doesn't work for it yet.

---
 drivers/pci/controller/pci-hyperv.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/pci/controller/pci-hyperv.c +5 -4
index 8d42da5dd1d4..5741b1dd3c14 100644
@@ -2299,7 +2299,7 @@ static void hv_eject_device_work(struct work_struct *work)
 	 * because hbus->bridge->bus may not exist yet.
 	 */
 	wslot = wslot_to_devfn(hpdev->desc.win_slot.slot);
-	pdev = pci_get_domain_bus_and_slot(hbus->sysdata.domain, 0, wslot);
+	pdev = pci_get_domain_bus_and_slot(hbus->bridge->domain_nr, 0, wslot);
 	if (pdev) {
 		pci_lock_rescan_remove();
 		pci_stop_and_remove_bus_device(pdev);
@@ -3071,6 +3071,7 @@ static int hv_pci_probe(struct hv_device *hdev,
 			 "PCI dom# 0x%hx has collision, using 0x%hx",
 			 dom_req, dom);
 
+	hbus->bridge->domain_nr = dom;
 	hbus->sysdata.domain = dom;
 
 	hbus->hdev = hdev;
@@ -3080,7 +3081,7 @@ static int hv_pci_probe(struct hv_device *hdev,
 	spin_lock_init(&hbus->device_list_lock);
 	spin_lock_init(&hbus->retarget_msi_interrupt_lock);
 	hbus->wq = alloc_ordered_workqueue("hv_pci_%x", 0,
-					   hbus->sysdata.domain);
+					   hbus->bridge->domain_nr);
 	if (!hbus->wq) {
 		ret = -ENOMEM;
 		goto free_dom;
@@ -3207,7 +3208,7 @@ static int hv_pci_probe(struct hv_device *hdev,
 destroy_wq:
 	destroy_workqueue(hbus->wq);
 free_dom:
-	hv_put_dom_num(hbus->sysdata.domain);
+	hv_put_dom_num(hbus->bridge->domain_nr);
 free_bus:
 	kfree(hbus);
 	return ret;
@@ -3315,7 +3316,7 @@ static int hv_pci_remove(struct hv_device *hdev)
 	irq_domain_remove(hbus->irq_domain);
 	irq_domain_free_fwnode(hbus->sysdata.fwnode);
 
-	hv_put_dom_num(hbus->sysdata.domain);
+	hv_put_dom_num(hbus->bridge->domain_nr);
 
 	kfree(hbus);
 	return ret;
Signature · 1 line
2.32.0
author
#13 lore raw

PATCH v6 7/8 PCI: hv: Set up MSI domain at bridge probing time

In reply to Boqun Feng

Since PCI_HYPERV depends on PCI_MSI_IRQ_DOMAIN which selects GENERIC_MSI_IRQ_DOMAIN, we can use dev_set_msi_domain() to set up the MSI domain at probing time, and this works for both x86 and ARM64.

Therefore use it as the preparation for ARM64 Hyper-V PCI support.

As a result, no longer need to maintain ->fwnode in x86 specific pci_sysdata, and make hv_pcibus_device own it instead.

---
 drivers/pci/controller/pci-hyperv.c | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)
drivers/pci/controller/pci-hyperv.c +8 -5
index 5741b1dd3c14..e6276aaa4659 100644
@@ -450,6 +450,7 @@ enum hv_pcibus_state {
 struct hv_pcibus_device {
 	struct pci_sysdata sysdata;
 	struct pci_host_bridge *bridge;
+	struct fwnode_handle *fwnode;
 	/* Protocol version negotiated with the host */
 	enum pci_protocol_version_t protocol_version;
 	enum hv_pcibus_state state;
@@ -1565,7 +1566,7 @@ static int hv_pcie_init_irq_domain(struct hv_pcibus_device *hbus)
 	hbus->msi_info.handler = handle_edge_irq;
 	hbus->msi_info.handler_name = "edge";
 	hbus->msi_info.data = hbus;
-	hbus->irq_domain = pci_msi_create_irq_domain(hbus->sysdata.fwnode,
+	hbus->irq_domain = pci_msi_create_irq_domain(hbus->fwnode,
 						     &hbus->msi_info,
 						     x86_vector_domain);
 	if (!hbus->irq_domain) {
@@ -1574,6 +1575,8 @@ static int hv_pcie_init_irq_domain(struct hv_pcibus_device *hbus)
 		return -ENODEV;
 	}
 
+	dev_set_msi_domain(&hbus->bridge->dev, hbus->irq_domain);
+
 	return 0;
 }
 
@@ -3118,9 +3121,9 @@ static int hv_pci_probe(struct hv_device *hdev,
 		goto unmap;
 	}
 
-	hbus->sysdata.fwnode = irq_domain_alloc_named_fwnode(name);
+	hbus->fwnode = irq_domain_alloc_named_fwnode(name);
 	kfree(name);
-	if (!hbus->sysdata.fwnode) {
+	if (!hbus->fwnode) {
 		ret = -ENOMEM;
 		goto unmap;
 	}
@@ -3198,7 +3201,7 @@ static int hv_pci_probe(struct hv_device *hdev,
 free_irq_domain:
 	irq_domain_remove(hbus->irq_domain);
 free_fwnode:
-	irq_domain_free_fwnode(hbus->sysdata.fwnode);
+	irq_domain_free_fwnode(hbus->fwnode);
 unmap:
 	iounmap(hbus->cfg_addr);
 free_config:
@@ -3314,7 +3317,7 @@ static int hv_pci_remove(struct hv_device *hdev)
 	hv_free_config_window(hbus);
 	hv_pci_free_bridge_windows(hbus);
 	irq_domain_remove(hbus->irq_domain);
-	irq_domain_free_fwnode(hbus->sysdata.fwnode);
+	irq_domain_free_fwnode(hbus->fwnode);
 
 	hv_put_dom_num(hbus->bridge->domain_nr);
 
Signature · 1 line
2.32.0
author
#14 lore raw

PATCH v6 8/8 PCI: hv: Turn on the host bridge probing on ARM64

In reply to Boqun Feng

Now we have everything we need, just provide a proper sysdata type for the bus to use on ARM64 and everything else works.

---
 drivers/pci/controller/pci-hyperv.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/pci/controller/pci-hyperv.c +7 -0
index e6276aaa4659..62dbe98d1fe1 100644
@@ -40,6 +40,7 @@
 #include <linux/kernel.h>
 #include <linux/module.h>
 #include <linux/pci.h>
+#include <linux/pci-ecam.h>
 #include <linux/delay.h>
 #include <linux/semaphore.h>
 #include <linux/irqdomain.h>
@@ -448,7 +449,11 @@ enum hv_pcibus_state {
 };
 
 struct hv_pcibus_device {
+#ifdef CONFIG_X86
 	struct pci_sysdata sysdata;
+#elif defined(CONFIG_ARM64)
+	struct pci_config_window sysdata;
+#endif
 	struct pci_host_bridge *bridge;
 	struct fwnode_handle *fwnode;
 	/* Protocol version negotiated with the host */
@@ -3075,7 +3080,9 @@ static int hv_pci_probe(struct hv_device *hdev,
 			 dom_req, dom);
 
 	hbus->bridge->domain_nr = dom;
+#ifdef CONFIG_X86
 	hbus->sysdata.domain = dom;
+#endif
 
 	hbus->hdev = hdev;
 	INIT_LIST_HEAD(&hbus->children);
Signature · 1 line
2.32.0
#15 lore raw

Re: [PATCH v6 8/8] PCI: hv: Turn on the host bridge probing on ARM64

Quoted 21 lines · Boqun Feng

Now we have everything we need, just provide a proper sysdata type for the bus to use on ARM64 and everything else works.

---
 drivers/pci/controller/pci-hyperv.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/pci/controller/pci-hyperv.c +1 -0
index e6276aaa4659..62dbe98d1fe1 100644
@@ -40,6 +40,7 @@
 #include <linux/kernel.h>
 #include <linux/module.h>
 #include <linux/pci.h>
+#include <linux/pci-ecam.h>
 #include <linux/delay.h>
 #include <linux/semaphore.h>
 #include <linux/irqdomain.h>

drivers/pci/controller/pci-hyperv.c enum hv_pcibus_state { · line 455

Quoted 7 lines
drivers/pci/controller/pci-hyperv.c +3 -0
@@ -448,7 +449,11 @@ enum hv_pcibus_state {
 };
 
 struct hv_pcibus_device {
+#ifdef CONFIG_X86
 	struct pci_sysdata sysdata;
+#elif defined(CONFIG_ARM64)
+	struct pci_config_window sysdata;

This is ugly. HV does not need pci_config_window at all right (other than arm64 pcibios_root_bridge_prepare()) ?

The issue is that in HV you have to have *some* sysdata != NULL, it is just some data to retrieve the hv_pcibus_device.

Mmaybe we can rework ARM64 ACPI code to store the acpi_device in struct pci_host_bridge->private instead of retrieving it from pci_config_window so that we decouple HV from the ARM64 back-end.

HV would just set struct pci_host_bridge->private == NULL.

I need to think about this a bit, I don't think it should block this series though but it would be nicer.

Lorenzo

Quoted 17 lines
drivers/pci/controller/pci-hyperv.c +3 -0
+#endif
 	struct pci_host_bridge *bridge;
 	struct fwnode_handle *fwnode;
 	/* Protocol version negotiated with the host */
@@ -3075,7 +3080,9 @@ static int hv_pci_probe(struct hv_device *hdev,
 			 dom_req, dom);
 
 	hbus->bridge->domain_nr = dom;
+#ifdef CONFIG_X86
 	hbus->sysdata.domain = dom;
+#endif
 
 	hbus->hdev = hdev;
 	INIT_LIST_HEAD(&hbus->children);
Signature · 1 line
2.32.0
author
#16 lore raw

Re: [PATCH v6 8/8] PCI: hv: Turn on the host bridge probing on ARM64

Quoted 24 lines · Lorenzo Pieralisi
Quoted 23 lines · Boqun Feng

Now we have everything we need, just provide a proper sysdata type for the bus to use on ARM64 and everything else works.

---
 drivers/pci/controller/pci-hyperv.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/pci/controller/pci-hyperv.c +1 -0
index e6276aaa4659..62dbe98d1fe1 100644
@@ -40,6 +40,7 @@
 #include <linux/kernel.h>
 #include <linux/module.h>
 #include <linux/pci.h>
+#include <linux/pci-ecam.h>
 #include <linux/delay.h>
 #include <linux/semaphore.h>
 #include <linux/irqdomain.h>
@@ -448,7 +449,11 @@ enum hv_pcibus_state {
 };
 
Quoted 9 lines

drivers/pci/controller/pci-hyperv.c enum hv_pcibus_state { · line 455

Quoted 5 lines
drivers/pci/controller/pci-hyperv.c +3 -0
 struct hv_pcibus_device {
+#ifdef CONFIG_X86
 	struct pci_sysdata sysdata;
+#elif defined(CONFIG_ARM64)
+	struct pci_config_window sysdata;

This is ugly. HV does not need pci_config_window at all right (other than arm64 pcibios_root_bridge_prepare()) ?

Right.

Quoted 9 lines

The issue is that in HV you have to have *some* sysdata != NULL, it is just some data to retrieve the hv_pcibus_device.

Mmaybe we can rework ARM64 ACPI code to store the acpi_device in struct pci_host_bridge->private instead of retrieving it from pci_config_window so that we decouple HV from the ARM64 back-end.

HV would just set struct pci_host_bridge->private == NULL.

Works for me, but please note that pci_sysdata is an x86-specific structure, so we still need to define a fake pci_sysdata inside pci-hyperv.c, like:

	#ifndef CONFIG_X86
	struct pci_sysdata { };
	#end
Quoted 2 lines

I need to think about this a bit, I don't think it should block this series though but it would be nicer.

After a quick look into the code, seems that what we need to do is to add an additional parameter for acpi_pci_root_create() and introduce a slightly different version of pci_create_root_bus(). A question is:
should we only do this for ARM64, or should we also do this for other acpi_pci_root_create() users (x86 and ia64)? Another question comes to my mind is, while we are at it, is there anything else that we want to move from sysdata to ->private? These questions are out of scope of this patchset, I think. Maybe it's better that we address them in the future, and I can send out separate RFC patches to start the discussion. Does that sound like a plan to you?

Regards,
Boqun

Quoted 15 lines

Lorenzo

Quoted 12 lines
drivers/pci/controller/pci-hyperv.c +3 -0
+#endif
 	struct pci_host_bridge *bridge;
 	struct fwnode_handle *fwnode;
 	/* Protocol version negotiated with the host */
@@ -3075,7 +3080,9 @@ static int hv_pci_probe(struct hv_device *hdev,
 			 dom_req, dom);
 
 	hbus->bridge->domain_nr = dom;
+#ifdef CONFIG_X86
 	hbus->sysdata.domain = dom;
+#endif
 
Quoted 5 lines
Quoted 5 lines
drivers/pci/controller/pci-hyperv.c +0 -0
 	hbus->hdev = hdev;
 	INIT_LIST_HEAD(&hbus->children);
Signature · 1 line
2.32.0

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#17 lore raw

Re: [PATCH v6 8/8] PCI: hv: Turn on the host bridge probing on ARM64

Quoted 58 lines · Boqun Feng
Quoted 24 lines · Lorenzo Pieralisi
Quoted 23 lines · Boqun Feng

Now we have everything we need, just provide a proper sysdata type for the bus to use on ARM64 and everything else works.

---
 drivers/pci/controller/pci-hyperv.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/pci/controller/pci-hyperv.c +1 -0
index e6276aaa4659..62dbe98d1fe1 100644
@@ -40,6 +40,7 @@
 #include <linux/kernel.h>
 #include <linux/module.h>
 #include <linux/pci.h>
+#include <linux/pci-ecam.h>
 #include <linux/delay.h>
 #include <linux/semaphore.h>
 #include <linux/irqdomain.h>
@@ -448,7 +449,11 @@ enum hv_pcibus_state {
 };
 
Quoted 9 lines

drivers/pci/controller/pci-hyperv.c enum hv_pcibus_state { · line 455

Quoted 5 lines
drivers/pci/controller/pci-hyperv.c +3 -0
 struct hv_pcibus_device {
+#ifdef CONFIG_X86
 	struct pci_sysdata sysdata;
+#elif defined(CONFIG_ARM64)
+	struct pci_config_window sysdata;

This is ugly. HV does not need pci_config_window at all right (other than arm64 pcibios_root_bridge_prepare()) ?

Right.

Quoted 9 lines

The issue is that in HV you have to have *some* sysdata != NULL, it is just some data to retrieve the hv_pcibus_device.

Mmaybe we can rework ARM64 ACPI code to store the acpi_device in struct pci_host_bridge->private instead of retrieving it from pci_config_window so that we decouple HV from the ARM64 back-end.

HV would just set struct pci_host_bridge->private == NULL.

Works for me, but please note that pci_sysdata is an x86-specific structure, so we still need to define a fake pci_sysdata inside pci-hyperv.c, like:

	#ifndef CONFIG_X86
	struct pci_sysdata { };
	#end
Quoted 2 lines

I need to think about this a bit, I don't think it should block this series though but it would be nicer.

Quoted 10 lines

After a quick look into the code, seems that what we need to do is to add an additional parameter for acpi_pci_root_create() and introduce a slightly different version of pci_create_root_bus(). A question is:
should we only do this for ARM64, or should we also do this for other acpi_pci_root_create() users (x86 and ia64)? Another question comes to my mind is, while we are at it, is there anything else that we want to move from sysdata to ->private? These questions are out of scope of this patchset, I think. Maybe it's better that we address them in the future, and I can send out separate RFC patches to start the discussion. Does that sound like a plan to you?

Yes it does and we can start from ARM64 - what I really don't like is the arch/arm64 dependency with the HV controller driver as I described, being forced to have a struct pci_config_window in the driver is not really nice or clean IMO.

Not that I expect any other PCI host bridge driver with ACPI coming anytime soon but even if it is not within set (that we can merge) I'd like to see the decoupling rework done asap, let me put it this way.

Thanks,
Lorenzo

Quoted 23 lines

Regards,
Boqun

Quoted 20 lines

Lorenzo

Quoted 17 lines
drivers/pci/controller/pci-hyperv.c +3 -0
+#endif
 	struct pci_host_bridge *bridge;
 	struct fwnode_handle *fwnode;
 	/* Protocol version negotiated with the host */
@@ -3075,7 +3080,9 @@ static int hv_pci_probe(struct hv_device *hdev,
 			 dom_req, dom);
 
 	hbus->bridge->domain_nr = dom;
+#ifdef CONFIG_X86
 	hbus->sysdata.domain = dom;
+#endif
 
 	hbus->hdev = hdev;
 	INIT_LIST_HEAD(&hbus->children);
Signature · 1 line
2.32.0
author
#18 lore raw

Re: [PATCH v6 8/8] PCI: hv: Turn on the host bridge probing on ARM64

Hi,

Quoted 70 lines · Lorenzo Pieralisi
Quoted 58 lines · Boqun Feng
Quoted 24 lines · Lorenzo Pieralisi
Quoted 23 lines · Boqun Feng

Now we have everything we need, just provide a proper sysdata type for the bus to use on ARM64 and everything else works.

---
 drivers/pci/controller/pci-hyperv.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/pci/controller/pci-hyperv.c +1 -0
index e6276aaa4659..62dbe98d1fe1 100644
@@ -40,6 +40,7 @@
 #include <linux/kernel.h>
 #include <linux/module.h>
 #include <linux/pci.h>
+#include <linux/pci-ecam.h>
 #include <linux/delay.h>
 #include <linux/semaphore.h>
 #include <linux/irqdomain.h>
@@ -448,7 +449,11 @@ enum hv_pcibus_state {
 };
 
Quoted 9 lines

drivers/pci/controller/pci-hyperv.c enum hv_pcibus_state { · line 455

Quoted 5 lines
drivers/pci/controller/pci-hyperv.c +3 -0
 struct hv_pcibus_device {
+#ifdef CONFIG_X86
 	struct pci_sysdata sysdata;
+#elif defined(CONFIG_ARM64)
+	struct pci_config_window sysdata;

This is ugly. HV does not need pci_config_window at all right (other than arm64 pcibios_root_bridge_prepare()) ?

Right.

Quoted 9 lines

The issue is that in HV you have to have *some* sysdata != NULL, it is just some data to retrieve the hv_pcibus_device.

Mmaybe we can rework ARM64 ACPI code to store the acpi_device in struct pci_host_bridge->private instead of retrieving it from pci_config_window so that we decouple HV from the ARM64 back-end.

HV would just set struct pci_host_bridge->private == NULL.

Works for me, but please note that pci_sysdata is an x86-specific structure, so we still need to define a fake pci_sysdata inside pci-hyperv.c, like:

	#ifndef CONFIG_X86
	struct pci_sysdata { };
	#end
Quoted 2 lines

I need to think about this a bit, I don't think it should block this series though but it would be nicer.

Quoted 10 lines

After a quick look into the code, seems that what we need to do is to add an additional parameter for acpi_pci_root_create() and introduce a slightly different version of pci_create_root_bus(). A question is:
should we only do this for ARM64, or should we also do this for other acpi_pci_root_create() users (x86 and ia64)? Another question comes to my mind is, while we are at it, is there anything else that we want to move from sysdata to ->private? These questions are out of scope of this patchset, I think. Maybe it's better that we address them in the future, and I can send out separate RFC patches to start the discussion. Does that sound like a plan to you?

Quoted 9 lines

Yes it does and we can start from ARM64 - what I really don't like is the arch/arm64 dependency with the HV controller driver as I described, being forced to have a struct pci_config_window in the driver is not really nice or clean IMO.

Not that I expect any other PCI host bridge driver with ACPI coming anytime soon but even if it is not within set (that we can merge) I'd like to see the decoupling rework done asap, let me put it this way.

Just want to check whether the following is a good starter for the decoupling rework?

	https://lore.kernel.org/lkml/[email protected]/

If so, is there any other concern about taking this patchset? ;-)

Regards,
Boqun

Quoted 26 lines

Thanks,
Lorenzo

Quoted 23 lines

Regards,
Boqun

Quoted 20 lines

Lorenzo

Quoted 17 lines
drivers/pci/controller/pci-hyperv.c +3 -0
+#endif
 	struct pci_host_bridge *bridge;
 	struct fwnode_handle *fwnode;
 	/* Protocol version negotiated with the host */
@@ -3075,7 +3080,9 @@ static int hv_pci_probe(struct hv_device *hdev,
 			 dom_req, dom);
 
 	hbus->bridge->domain_nr = dom;
+#ifdef CONFIG_X86
 	hbus->sysdata.domain = dom;
+#endif
 
 	hbus->hdev = hdev;
 	INIT_LIST_HEAD(&hbus->children);
Signature · 1 line
2.32.0
#19 lore raw
In reply to Boqun Feng
Quoted 10 lines · Boqun Feng

Hi,

This is the v6 for the preparation of virtual PCI support on Hyper-V ARM64, Previous versions:

v1: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/lkml/[email protected]/
v3: https://lore.kernel.org/lkml/[email protected]/
v4: https://lore.kernel.org/lkml/[email protected]/
v5: https://lore.kernel.org/lkml/[email protected]/

Thanks for these; they're very handy for archaeology later.

Quoted 3 lines

Changes since last version:

* Rebase to 5.14-rc3

Just FYI, it's not a problem that you rebased, but it's not necessary. These will be applied to a topic branch from v5.14-rc1 anyway.

author
#20 lore raw
Quoted 12 lines · Bjorn Helgaas
Quoted 10 lines · Boqun Feng

Hi,

This is the v6 for the preparation of virtual PCI support on Hyper-V ARM64, Previous versions:

v1: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/lkml/[email protected]/
v3: https://lore.kernel.org/lkml/[email protected]/
v4: https://lore.kernel.org/lkml/[email protected]/
v5: https://lore.kernel.org/lkml/[email protected]/

Quoted 8 lines

Thanks for these; they're very handy for archaeology later.

Quoted 3 lines

Changes since last version:

* Rebase to 5.14-rc3

Just FYI, it's not a problem that you rebased, but it's not necessary. These will be applied to a topic branch from v5.14-rc1 anyway.

Got it ;-)

Regards,
Boqun

author
#21 lore raw
In reply to Boqun Feng
Quoted 26 lines · Boqun Feng

Hi,

This is the v6 for the preparation of virtual PCI support on Hyper-V ARM64, Previous versions:

v1: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/lkml/[email protected]/
v3: https://lore.kernel.org/lkml/[email protected]/
v4: https://lore.kernel.org/lkml/[email protected]/
v5: https://lore.kernel.org/lkml/[email protected]/

Changes since last version:

* Rebase to 5.14-rc3

* Comment fixes as suggested by Bjorn.

The basic problem we need to resolve is that ARM64 is an arch with PCI_DOMAINS_GENERIC=y, so the bus sysdata is pci_config_window. However, Hyper-V PCI provides a paravirtualized PCI interface, so there is no actual pci_config_window for a PCI host bridge, so no information can be retrieve from the pci_config_window of a Hyper-V virtual PCI bus. Also there is no corresponding ACPI device for the Hyper-V PCI root bridge, which introduces a special case when trying to find the ACPI device from the sysdata (see patch #3).

Quoted 5 lines

With this patchset, we could enable the virtual PCI on Hyper-V ARM64 guest with other code under development.

Comments and suggestions are welcome.

Ping ;-)

Regards,
Boqun

Quoted 24 lines

Regards,
Boqun

Arnd Bergmann (1):
PCI: hv: Generify PCI probing

Boqun Feng (7):
PCI: Introduce domain_nr in pci_host_bridge
PCI: Support populating MSI domains of root buses via bridges
arm64: PCI: Restructure pcibios_root_bridge_prepare()
arm64: PCI: Support root bridge preparation for Hyper-V
PCI: hv: Set ->domain_nr of pci_host_bridge at probing time
PCI: hv: Set up MSI domain at bridge probing time
PCI: hv: Turn on the host bridge probing on ARM64

 arch/arm64/kernel/pci.c             | 29 +++++++---
 drivers/pci/controller/pci-hyperv.c | 86 +++++++++++++++++------------
 drivers/pci/probe.c                 | 12 +++-
 include/linux/pci.h                 | 11 ++++
 4 files changed, 93 insertions(+), 45 deletions(-)
Signature · 1 line
2.32.0
#22 lore raw
Quoted 27 lines · Boqun Feng
Quoted 26 lines · Boqun Feng

Hi,

This is the v6 for the preparation of virtual PCI support on Hyper-V ARM64, Previous versions:

v1: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/lkml/[email protected]/
v3: https://lore.kernel.org/lkml/[email protected]/
v4: https://lore.kernel.org/lkml/[email protected]/
v5: https://lore.kernel.org/lkml/[email protected]/

Changes since last version:

* Rebase to 5.14-rc3

* Comment fixes as suggested by Bjorn.

The basic problem we need to resolve is that ARM64 is an arch with PCI_DOMAINS_GENERIC=y, so the bus sysdata is pci_config_window. However, Hyper-V PCI provides a paravirtualized PCI interface, so there is no actual pci_config_window for a PCI host bridge, so no information can be retrieve from the pci_config_window of a Hyper-V virtual PCI bus. Also there is no corresponding ACPI device for the Hyper-V PCI root bridge, which introduces a special case when trying to find the ACPI device from the sysdata (see patch #3).

Quoted 6 lines
Quoted 4 lines

With this patchset, we could enable the virtual PCI on Hyper-V ARM64 guest with other code under development.

Comments and suggestions are welcome.

Ping ;-)

Lorenzo normally takes care of the native drivers (including Hyper-V), so I don't want to get in his way. If he's too busy and doesn't get a chance, poke me again after -rc6 or so and I'll pick it up.

Quoted 21 lines
Quoted 21 lines

Arnd Bergmann (1):
PCI: hv: Generify PCI probing

Boqun Feng (7):
PCI: Introduce domain_nr in pci_host_bridge
PCI: Support populating MSI domains of root buses via bridges
arm64: PCI: Restructure pcibios_root_bridge_prepare()
arm64: PCI: Support root bridge preparation for Hyper-V
PCI: hv: Set ->domain_nr of pci_host_bridge at probing time
PCI: hv: Set up MSI domain at bridge probing time
PCI: hv: Turn on the host bridge probing on ARM64

 arch/arm64/kernel/pci.c             | 29 +++++++---
 drivers/pci/controller/pci-hyperv.c | 86 +++++++++++++++++------------
 drivers/pci/probe.c                 | 12 +++-
 include/linux/pci.h                 | 11 ++++
 4 files changed, 93 insertions(+), 45 deletions(-)
Signature · 1 line
2.32.0
#23 lore raw
In reply to Boqun Feng
Quoted 46 lines · Boqun Feng

Hi,

This is the v6 for the preparation of virtual PCI support on Hyper-V ARM64, Previous versions:

v1: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/lkml/[email protected]/
v3: https://lore.kernel.org/lkml/[email protected]/
v4: https://lore.kernel.org/lkml/[email protected]/
v5: https://lore.kernel.org/lkml/[email protected]/

Changes since last version:

* Rebase to 5.14-rc3

* Comment fixes as suggested by Bjorn.

The basic problem we need to resolve is that ARM64 is an arch with PCI_DOMAINS_GENERIC=y, so the bus sysdata is pci_config_window. However, Hyper-V PCI provides a paravirtualized PCI interface, so there is no actual pci_config_window for a PCI host bridge, so no information can be retrieve from the pci_config_window of a Hyper-V virtual PCI bus. Also there is no corresponding ACPI device for the Hyper-V PCI root bridge, which introduces a special case when trying to find the ACPI device from the sysdata (see patch #3).

With this patchset, we could enable the virtual PCI on Hyper-V ARM64 guest with other code under development.

Comments and suggestions are welcome.

Regards,
Boqun

Arnd Bergmann (1):
PCI: hv: Generify PCI probing

Boqun Feng (7):
PCI: Introduce domain_nr in pci_host_bridge
PCI: Support populating MSI domains of root buses via bridges
arm64: PCI: Restructure pcibios_root_bridge_prepare()
arm64: PCI: Support root bridge preparation for Hyper-V
PCI: hv: Set ->domain_nr of pci_host_bridge at probing time
PCI: hv: Set up MSI domain at bridge probing time
PCI: hv: Turn on the host bridge probing on ARM64

Quoted 5 lines
 arch/arm64/kernel/pci.c             | 29 +++++++---
 drivers/pci/controller/pci-hyperv.c | 86 +++++++++++++++++------------
 drivers/pci/probe.c                 | 12 +++-
 include/linux/pci.h                 | 11 ++++
 4 files changed, 93 insertions(+), 45 deletions(-)

If we take this series via the PCI tree we'd need Catalin/Will ACKs on patches 3-4.

I need some time to look into [1] (thanks for that).

Without [1] patch 8 is ugly, that's no news. The question is whether it is worth waiting for a kernel cycle to integrate [1] into this series or not.

Is it really a problem if we postpone this series for another kernel cycle so that we can look into it ?

[1] https://lore.kernel.org/lkml/[email protected]/

author
#24 lore raw
Quoted 47 lines · Lorenzo Pieralisi
Quoted 46 lines · Boqun Feng

Hi,

This is the v6 for the preparation of virtual PCI support on Hyper-V ARM64, Previous versions:

v1: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/lkml/[email protected]/
v3: https://lore.kernel.org/lkml/[email protected]/
v4: https://lore.kernel.org/lkml/[email protected]/
v5: https://lore.kernel.org/lkml/[email protected]/

Changes since last version:

* Rebase to 5.14-rc3

* Comment fixes as suggested by Bjorn.

The basic problem we need to resolve is that ARM64 is an arch with PCI_DOMAINS_GENERIC=y, so the bus sysdata is pci_config_window. However, Hyper-V PCI provides a paravirtualized PCI interface, so there is no actual pci_config_window for a PCI host bridge, so no information can be retrieve from the pci_config_window of a Hyper-V virtual PCI bus. Also there is no corresponding ACPI device for the Hyper-V PCI root bridge, which introduces a special case when trying to find the ACPI device from the sysdata (see patch #3).

With this patchset, we could enable the virtual PCI on Hyper-V ARM64 guest with other code under development.

Comments and suggestions are welcome.

Regards,
Boqun

Arnd Bergmann (1):
PCI: hv: Generify PCI probing

Boqun Feng (7):
PCI: Introduce domain_nr in pci_host_bridge
PCI: Support populating MSI domains of root buses via bridges
arm64: PCI: Restructure pcibios_root_bridge_prepare()
arm64: PCI: Support root bridge preparation for Hyper-V
PCI: hv: Set ->domain_nr of pci_host_bridge at probing time
PCI: hv: Set up MSI domain at bridge probing time
PCI: hv: Turn on the host bridge probing on ARM64

Quoted 9 lines
Quoted 5 lines
 arch/arm64/kernel/pci.c             | 29 +++++++---
 drivers/pci/controller/pci-hyperv.c | 86 +++++++++++++++++------------
 drivers/pci/probe.c                 | 12 +++-
 include/linux/pci.h                 | 11 ++++
 4 files changed, 93 insertions(+), 45 deletions(-)

If we take this series via the PCI tree we'd need Catalin/Will ACKs on patches 3-4.

Got it.

Quoted 9 lines

I need some time to look into [1] (thanks for that).

Without [1] patch 8 is ugly, that's no news. The question is whether it is worth waiting for a kernel cycle to integrate [1] into this series or not.

Is it really a problem if we postpone this series for another kernel cycle so that we can look into it ?

Well, it's definitely better for me that we can have it in 5.15-rc1 ;-), because it's a dependency for Hyper-V virtual PCI support on ARM64 and we plan to send the rest of work in 5.15 cycle. And I can just base on hyperv-next for the rest of the work if this is in 5.15-rc1. But yes, it's not really a problem, since this one still needs to work with other patches to support virtual PCI on ARM64 Hyper-V.

In fact, I personally don't think [1] is better than patch 8 (plus patch 3 & 4): playing with ->private seems dangerous and not very helpful on readiblity, but I agree that we should explore every potential solution, and that's why I send [1].

Regards,
Boqun

Quoted 1 line

[1] https://lore.kernel.org/lkml/[email protected]/

#25 lore raw
Quoted 77 lines · Boqun Feng
Quoted 47 lines · Lorenzo Pieralisi
Quoted 46 lines · Boqun Feng

Hi,

This is the v6 for the preparation of virtual PCI support on Hyper-V ARM64, Previous versions:

v1: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/lkml/[email protected]/
v3: https://lore.kernel.org/lkml/[email protected]/
v4: https://lore.kernel.org/lkml/[email protected]/
v5: https://lore.kernel.org/lkml/[email protected]/

Changes since last version:

* Rebase to 5.14-rc3

* Comment fixes as suggested by Bjorn.

The basic problem we need to resolve is that ARM64 is an arch with PCI_DOMAINS_GENERIC=y, so the bus sysdata is pci_config_window. However, Hyper-V PCI provides a paravirtualized PCI interface, so there is no actual pci_config_window for a PCI host bridge, so no information can be retrieve from the pci_config_window of a Hyper-V virtual PCI bus. Also there is no corresponding ACPI device for the Hyper-V PCI root bridge, which introduces a special case when trying to find the ACPI device from the sysdata (see patch #3).

With this patchset, we could enable the virtual PCI on Hyper-V ARM64 guest with other code under development.

Comments and suggestions are welcome.

Regards,
Boqun

Arnd Bergmann (1):
PCI: hv: Generify PCI probing

Boqun Feng (7):
PCI: Introduce domain_nr in pci_host_bridge
PCI: Support populating MSI domains of root buses via bridges
arm64: PCI: Restructure pcibios_root_bridge_prepare()
arm64: PCI: Support root bridge preparation for Hyper-V
PCI: hv: Set ->domain_nr of pci_host_bridge at probing time
PCI: hv: Set up MSI domain at bridge probing time
PCI: hv: Turn on the host bridge probing on ARM64

Quoted 9 lines
Quoted 5 lines
 arch/arm64/kernel/pci.c             | 29 +++++++---
 drivers/pci/controller/pci-hyperv.c | 86 +++++++++++++++++------------
 drivers/pci/probe.c                 | 12 +++-
 include/linux/pci.h                 | 11 ++++
 4 files changed, 93 insertions(+), 45 deletions(-)

If we take this series via the PCI tree we'd need Catalin/Will ACKs on patches 3-4.

Got it.

Quoted 9 lines

I need some time to look into [1] (thanks for that).

Without [1] patch 8 is ugly, that's no news. The question is whether it is worth waiting for a kernel cycle to integrate [1] into this series or not.

Is it really a problem if we postpone this series for another kernel cycle so that we can look into it ?

Well, it's definitely better for me that we can have it in 5.15-rc1 ;-), because it's a dependency for Hyper-V virtual PCI support on ARM64 and we plan to send the rest of work in 5.15 cycle. And I can just base on hyperv-next for the rest of the work if this is in 5.15-rc1. But yes, it's not really a problem, since this one still needs to work with other patches to support virtual PCI on ARM64 Hyper-V.

Quoted 4 lines

In fact, I personally don't think [1] is better than patch 8 (plus patch 3 & 4): playing with ->private seems dangerous and not very helpful on readiblity, but I agree that we should explore every potential solution, and that's why I send [1].

Pulled the current series - now let's work together to improve it, I will have a look into [1] in the weeks to come and get back to you with some feedback.

Thanks,
Lorenzo

Quoted 4 lines

Regards,
Boqun

Quoted 1 line

[1] https://lore.kernel.org/lkml/[email protected]/

author
#26 lore raw
Quoted 78 lines · Lorenzo Pieralisi
Quoted 77 lines · Boqun Feng
Quoted 47 lines · Lorenzo Pieralisi
Quoted 46 lines · Boqun Feng

Hi,

This is the v6 for the preparation of virtual PCI support on Hyper-V ARM64, Previous versions:

v1: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/lkml/[email protected]/
v3: https://lore.kernel.org/lkml/[email protected]/
v4: https://lore.kernel.org/lkml/[email protected]/
v5: https://lore.kernel.org/lkml/[email protected]/

Changes since last version:

* Rebase to 5.14-rc3

* Comment fixes as suggested by Bjorn.

The basic problem we need to resolve is that ARM64 is an arch with PCI_DOMAINS_GENERIC=y, so the bus sysdata is pci_config_window. However, Hyper-V PCI provides a paravirtualized PCI interface, so there is no actual pci_config_window for a PCI host bridge, so no information can be retrieve from the pci_config_window of a Hyper-V virtual PCI bus. Also there is no corresponding ACPI device for the Hyper-V PCI root bridge, which introduces a special case when trying to find the ACPI device from the sysdata (see patch #3).

With this patchset, we could enable the virtual PCI on Hyper-V ARM64 guest with other code under development.

Comments and suggestions are welcome.

Regards,
Boqun

Arnd Bergmann (1):
PCI: hv: Generify PCI probing

Boqun Feng (7):
PCI: Introduce domain_nr in pci_host_bridge
PCI: Support populating MSI domains of root buses via bridges
arm64: PCI: Restructure pcibios_root_bridge_prepare()
arm64: PCI: Support root bridge preparation for Hyper-V
PCI: hv: Set ->domain_nr of pci_host_bridge at probing time
PCI: hv: Set up MSI domain at bridge probing time
PCI: hv: Turn on the host bridge probing on ARM64

Quoted 9 lines
Quoted 5 lines
 arch/arm64/kernel/pci.c             | 29 +++++++---
 drivers/pci/controller/pci-hyperv.c | 86 +++++++++++++++++------------
 drivers/pci/probe.c                 | 12 +++-
 include/linux/pci.h                 | 11 ++++
 4 files changed, 93 insertions(+), 45 deletions(-)

If we take this series via the PCI tree we'd need Catalin/Will ACKs on patches 3-4.

Got it.

Quoted 9 lines

I need some time to look into [1] (thanks for that).

Without [1] patch 8 is ugly, that's no news. The question is whether it is worth waiting for a kernel cycle to integrate [1] into this series or not.

Is it really a problem if we postpone this series for another kernel cycle so that we can look into it ?

Well, it's definitely better for me that we can have it in 5.15-rc1 ;-), because it's a dependency for Hyper-V virtual PCI support on ARM64 and we plan to send the rest of work in 5.15 cycle. And I can just base on hyperv-next for the rest of the work if this is in 5.15-rc1. But yes, it's not really a problem, since this one still needs to work with other patches to support virtual PCI on ARM64 Hyper-V.

Quoted 9 lines
Quoted 4 lines

In fact, I personally don't think [1] is better than patch 8 (plus patch 3 & 4): playing with ->private seems dangerous and not very helpful on readiblity, but I agree that we should explore every potential solution, and that's why I send [1].

Pulled the current series - now let's work together to improve it, I will have a look into [1] in the weeks to come and get back to you with some feedback.

Thanks a lot! Looking forwards to your feedback ;-)

Regards,
Boqun

Quoted 7 lines

Thanks,
Lorenzo

Quoted 4 lines

Regards,
Boqun

Quoted 1 line

[1] https://lore.kernel.org/lkml/[email protected]/

#27 lore raw
In reply to Boqun Feng
Quoted 10 lines · Boqun Feng

This is the v6 for the preparation of virtual PCI support on Hyper-V ARM64, Previous versions:

v1: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/lkml/[email protected]/
v3: https://lore.kernel.org/lkml/[email protected]/
v4: https://lore.kernel.org/lkml/[email protected]/
v5: https://lore.kernel.org/lkml/[email protected]/

[...]

Applied to pci/hyper-v, thanks!

[1/8] PCI: Introduce domain_nr in pci_host_bridge

      https://git.kernel.org/lpieralisi/pci/c/15d82ca23c
[2/8] PCI: Support populating MSI domains of root buses via bridges
      https://git.kernel.org/lpieralisi/pci/c/41dd40fd71
[3/8] arm64: PCI: Restructure pcibios_root_bridge_prepare()
      https://git.kernel.org/lpieralisi/pci/c/b424d4d426
[4/8] arm64: PCI: Support root bridge preparation for Hyper-V
      https://git.kernel.org/lpieralisi/pci/c/7d40c0f70d
[5/8] PCI: hv: Generify PCI probing
      https://git.kernel.org/lpieralisi/pci/c/418cb6c8e0
[6/8] PCI: hv: Set ->domain_nr of pci_host_bridge at probing time
      https://git.kernel.org/lpieralisi/pci/c/38c0d266dc
[7/8] PCI: hv: Set up MSI domain at bridge probing time
      https://git.kernel.org/lpieralisi/pci/c/9e7f9178ab
[8/8] PCI: hv: Turn on the host bridge probing on ARM64
      https://git.kernel.org/lpieralisi/pci/c/88f94c7f8f

Thanks,
Lorenzo