Skip to contentSkip to conversation

PCI: hv: Hyper-V vPCI for arm64

PATCH Superseded Sunil Muthuswamy 4 participants 8 messages Nov 18, 2021 – Dec 17, 2021

Subsystems (7)

v6 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

v6 cover PCI: hv: Hyper-V vPCI for arm64

From: Sunil Muthuswamy <[email protected]>

Current Hyper-V vPCI code only compiles and works for x86. There are some hardcoded assumptions about the architectural IRQ chip and other arch defines.

Add support for Hyper-V vPCI for arm64 by first breaking the current hard coded dependency using a set of new interfaces and implementing those for x86 first. That is in the first patch. The second patch adds support for Hyper-V vPCI for arm64 by implementing the above mentioned interfaces. That is done by introducing a Hyper-V vPCI specific MSI IRQ domain & chip for allocating SPI vectors.

changes in v1 -> v2:
- Moved the irqchip implementation to drivers/pci as suggested by Marc Zyngier
- Addressed Multi-MSI handling issues identified by Marc Zyngier
- Addressed lock/synchronization primitive as suggested by Marc Zyngier
- Addressed other code feedback from Marc Zyngier

changes in v2 -> v3:
- Addressed comments from Bjorn Helgaas about patch formatting and verbiage
- Using 'git send-email' to ensure that the patch series is correctly threaded. Feedback by Bjorn Helgaas
- Fixed Hyper-V vPCI build break for module build, reported by Boqun Feng

changes in v3 -> v4:
- Removed the separate file for IRQ chip that was there in previous iterations and moved the IRQ chip implementation to pci-hyperv.c. Feedback by Michael Kelley and Marc Zyngier.
- Addressed various comments from Marc Zyngier about structuring and layout.
- Addressed comment from Marc Zyngier about IRQ affinity and other miscellaneous comments.

changes in v4 -> v5:
- Fixed an issue with picking the right cpu for irq affinity, identified by Marc Zyngier.

changes in v5 -> v6:
- Minor comment updates suggested by Michael Kelley.

Sunil Muthuswamy (2):
PCI: hv: Make the code arch neutral by adding arch specific interfaces
arm64: PCI: hv: Add support for Hyper-V vPCI

 arch/arm64/include/asm/hyperv-tlfs.h |   9 +
 arch/x86/include/asm/hyperv-tlfs.h   |  33 ++++
 arch/x86/include/asm/mshyperv.h      |   7 -
 drivers/pci/Kconfig                  |   2 +-
 drivers/pci/controller/Kconfig       |   2 +-
 drivers/pci/controller/pci-hyperv.c  | 281 ++++++++++++++++++++++++---
 include/asm-generic/hyperv-tlfs.h    |  33 ----
 7 files changed, 300 insertions(+), 67 deletions(-)

base-commit: fa55b7dcdc43c1aa1ba12bca9d2dd4318c2a0dbf

Signature · 1 line
2.25.1
#1 lore raw

PATCH v6 1/2 PCI: hv: Make the code arch neutral by adding arch specific interfaces

From: Sunil Muthuswamy <[email protected]>

Encapsulate arch dependencies in Hyper-V vPCI through a set of arch-dependent interfaces. Adding these arch specific interfaces will allow for an implementation for other architectures, such as arm64.

There are no functional changes expected from this patch.

---
In v2, v3, v4, v5 & v6:
Changes are described in the cover letter. No change from v4 -> v5.

 arch/x86/include/asm/hyperv-tlfs.h  | 33 ++++++++++++
 arch/x86/include/asm/mshyperv.h     |  7 ---
 drivers/pci/controller/pci-hyperv.c | 79 ++++++++++++++++++++---------
 include/asm-generic/hyperv-tlfs.h   | 33 ------------
 4 files changed, 87 insertions(+), 65 deletions(-)
arch/x86/include/asm/hyperv-tlfs.h +33 -0
index 381e88122a5f..0a9407dc0859 100644
@@ -602,6 +602,39 @@ enum hv_interrupt_type {
 	HV_X64_INTERRUPT_TYPE_MAXIMUM           = 0x000A,
 };
 
+union hv_msi_address_register {
+	u32 as_uint32;
+	struct {
+		u32 reserved1:2;
+		u32 destination_mode:1;
+		u32 redirection_hint:1;
+		u32 reserved2:8;
+		u32 destination_id:8;
+		u32 msi_base:12;
+	};
+} __packed;
+
+union hv_msi_data_register {
+	u32 as_uint32;
+	struct {
+		u32 vector:8;
+		u32 delivery_mode:3;
+		u32 reserved1:3;
+		u32 level_assert:1;
+		u32 trigger_mode:1;
+		u32 reserved2:16;
+	};
+} __packed;
+
+/* HvRetargetDeviceInterrupt hypercall */
+union hv_msi_entry {
+	u64 as_uint64;
+	struct {
+		union hv_msi_address_register address;
+		union hv_msi_data_register data;
+	} __packed;
+};
+
 #include <asm-generic/hyperv-tlfs.h>
 
 #endif
arch/x86/include/asm/mshyperv.h +0 -7
index da3972fe5a7a..a1c3dceff8eb 100644
@@ -169,13 +169,6 @@ bool hv_vcpu_is_preempted(int vcpu);
 static inline void hv_apic_init(void) {}
 #endif
 
-static inline void hv_set_msi_entry_from_desc(union hv_msi_entry *msi_entry,
-					      struct msi_desc *msi_desc)
-{
-	msi_entry->address.as_uint32 = msi_desc->msg.address_lo;
-	msi_entry->data.as_uint32 = msi_desc->msg.data;
-}
-
 struct irq_domain *hv_create_pci_msi_domain(void);
 
 int hv_map_ioapic_interrupt(int ioapic_id, bool level, int vcpu, int vector,
drivers/pci/controller/pci-hyperv.c +54 -25

181 lines not on this page · raw message

include/asm-generic/hyperv-tlfs.h +0 -33
index 8ed6733d5146..8f97c2927bee 100644
@@ -540,39 +540,6 @@ enum hv_interrupt_source {
 	HV_INTERRUPT_SOURCE_IOAPIC,
 };
 
-union hv_msi_address_register {
-	u32 as_uint32;
-	struct {
-		u32 reserved1:2;
-		u32 destination_mode:1;
-		u32 redirection_hint:1;
-		u32 reserved2:8;
-		u32 destination_id:8;
-		u32 msi_base:12;
-	};
-} __packed;
-
-union hv_msi_data_register {
-	u32 as_uint32;
-	struct {
-		u32 vector:8;
-		u32 delivery_mode:3;
-		u32 reserved1:3;
-		u32 level_assert:1;
-		u32 trigger_mode:1;
-		u32 reserved2:16;
-	};
-} __packed;
-
-/* HvRetargetDeviceInterrupt hypercall */
-union hv_msi_entry {
-	u64 as_uint64;
-	struct {
-		union hv_msi_address_register address;
-		union hv_msi_data_register data;
-	} __packed;
-};
-
 union hv_ioapic_rte {
 	u64 as_uint64;
 
Signature · 1 line
2.25.1
#2 lore raw

PATCH v6 2/2 arm64: PCI: hv: Add support for Hyper-V vPCI

In reply to Sunil Muthuswamy

From: Sunil Muthuswamy <[email protected]>

Add support for Hyper-V vPCI for arm64 by implementing the arch specific interfaces. Introduce an IRQ domain and chip specific to Hyper-v vPCI that is based on SPIs. The IRQ domain parents itself to the arch GIC IRQ domain for basic vector management.

---
In v2, v3, v4, v5 & v6:
Changes are described in the cover letter.

 arch/arm64/include/asm/hyperv-tlfs.h |   9 ++
 drivers/pci/Kconfig                  |   2 +-
 drivers/pci/controller/Kconfig       |   2 +-
 drivers/pci/controller/pci-hyperv.c  | 204 ++++++++++++++++++++++++++-
 4 files changed, 214 insertions(+), 3 deletions(-)
arch/arm64/include/asm/hyperv-tlfs.h +9 -0
index 4d964a7f02ee..bc6c7ac934a1 100644
@@ -64,6 +64,15 @@
 #define HV_REGISTER_STIMER0_CONFIG	0x000B0000
 #define HV_REGISTER_STIMER0_COUNT	0x000B0001
 
+union hv_msi_entry {
+	u64 as_uint64[2];
+	struct {
+		u64 address;
+		u32 data;
+		u32 reserved;
+	} __packed;
+};
+
 #include <asm-generic/hyperv-tlfs.h>
 
 #endif
drivers/pci/Kconfig +1 -1
index 43e615aa12ff..d98fafdd0f99 100644
@@ -184,7 +184,7 @@ config PCI_LABEL
 
 config PCI_HYPERV
 	tristate "Hyper-V PCI Frontend"
-	depends on X86_64 && HYPERV && PCI_MSI && PCI_MSI_IRQ_DOMAIN && SYSFS
+	depends on ((X86 && X86_64) || ARM64) && HYPERV && PCI_MSI && PCI_MSI_IRQ_DOMAIN && SYSFS
 	select PCI_HYPERV_INTERFACE
 	help
 	  The PCI device frontend driver allows the kernel to import arbitrary
drivers/pci/controller/Kconfig +1 -1
index 93b141110537..2536abcc045a 100644
@@ -281,7 +281,7 @@ config PCIE_BRCMSTB
 
 config PCI_HYPERV_INTERFACE
 	tristate "Hyper-V PCI Interface"
-	depends on X86 && HYPERV && PCI_MSI && PCI_MSI_IRQ_DOMAIN && X86_64
+	depends on ((X86 && X86_64) || ARM64) && HYPERV && PCI_MSI && PCI_MSI_IRQ_DOMAIN
 	help
 	  The Hyper-V PCI Interface is a helper driver allows other drivers to
 	  have a common interface with the Hyper-V PCI frontend driver.
drivers/pci/controller/pci-hyperv.c +203 -1

234 lines not on this page · raw message

Signature · 1 line
2.25.1
#3 lore raw

Re: [PATCH v6 2/2] arm64: PCI: hv: Add support for Hyper-V vPCI

Quoted 165 lines · Sunil Muthuswamy

From: Sunil Muthuswamy <[email protected]>

Add support for Hyper-V vPCI for arm64 by implementing the arch specific interfaces. Introduce an IRQ domain and chip specific to Hyper-v vPCI that is based on SPIs. The IRQ domain parents itself to the arch GIC IRQ domain for basic vector management.

---
In v2, v3, v4, v5 & v6:
Changes are described in the cover letter.

 arch/arm64/include/asm/hyperv-tlfs.h |   9 ++
 drivers/pci/Kconfig                  |   2 +-
 drivers/pci/controller/Kconfig       |   2 +-
 drivers/pci/controller/pci-hyperv.c  | 204 ++++++++++++++++++++++++++-
 4 files changed, 214 insertions(+), 3 deletions(-)
arch/arm64/include/asm/hyperv-tlfs.h +9 -0
index 4d964a7f02ee..bc6c7ac934a1 100644
@@ -64,6 +64,15 @@
 #define HV_REGISTER_STIMER0_CONFIG	0x000B0000
 #define HV_REGISTER_STIMER0_COUNT	0x000B0001
 
+union hv_msi_entry {
+	u64 as_uint64[2];
+	struct {
+		u64 address;
+		u32 data;
+		u32 reserved;
+	} __packed;
+};
+
 #include <asm-generic/hyperv-tlfs.h>
 
 #endif
drivers/pci/Kconfig +1 -1
index 43e615aa12ff..d98fafdd0f99 100644
@@ -184,7 +184,7 @@ config PCI_LABEL
 
 config PCI_HYPERV
 	tristate "Hyper-V PCI Frontend"
-	depends on X86_64 && HYPERV && PCI_MSI && PCI_MSI_IRQ_DOMAIN && SYSFS
+	depends on ((X86 && X86_64) || ARM64) && HYPERV && PCI_MSI && PCI_MSI_IRQ_DOMAIN && SYSFS
 	select PCI_HYPERV_INTERFACE
 	help
 	  The PCI device frontend driver allows the kernel to import arbitrary
drivers/pci/controller/Kconfig +1 -1
index 93b141110537..2536abcc045a 100644
@@ -281,7 +281,7 @@ config PCIE_BRCMSTB
 
 config PCI_HYPERV_INTERFACE
 	tristate "Hyper-V PCI Interface"
-	depends on X86 && HYPERV && PCI_MSI && PCI_MSI_IRQ_DOMAIN && X86_64
+	depends on ((X86 && X86_64) || ARM64) && HYPERV && PCI_MSI && PCI_MSI_IRQ_DOMAIN
 	help
 	  The Hyper-V PCI Interface is a helper driver allows other drivers to
 	  have a common interface with the Hyper-V PCI frontend driver.
drivers/pci/controller/pci-hyperv.c +84 -1
index ead7d6cb6bf1..d52e23b1d14b 100644
@@ -47,6 +47,8 @@
 #include <linux/msi.h>
 #include <linux/hyperv.h>
 #include <linux/refcount.h>
+#include <linux/irqdomain.h>
+#include <linux/acpi.h>
 #include <asm/mshyperv.h>
 
 /*
@@ -614,7 +616,202 @@ static int hv_msi_prepare(struct irq_domain *domain, struct device *dev,
 {
 	return pci_msi_prepare(domain, dev, nvec, info);
 }
-#endif /* CONFIG_X86 */
+#elif defined(CONFIG_ARM64)
+/*
+ * SPI vectors to use for vPCI; arch SPIs range is [32, 1019], but leaving a bit
+ * of room at the start to allow for SPIs to be specified through ACPI and
+ * starting with a power of two to satisfy power of 2 multi-MSI requirement.
+ */
+#define HV_PCI_MSI_SPI_START	64
+#define HV_PCI_MSI_SPI_NR	(1020 - HV_PCI_MSI_SPI_START)
+#define DELIVERY_MODE		0
+#define FLOW_HANDLER		NULL
+#define FLOW_NAME		NULL
+#define hv_msi_prepare		NULL
+
+struct hv_pci_chip_data {
+	DECLARE_BITMAP(spi_map, HV_PCI_MSI_SPI_NR);
+	struct mutex	map_lock;
+};
+
+/* Hyper-V vPCI MSI GIC IRQ domain */
+static struct irq_domain *hv_msi_gic_irq_domain;
+
+/* Hyper-V PCI MSI IRQ chip */
+static struct irq_chip hv_arm64_msi_irq_chip = {
+	.name = "MSI",
+	.irq_set_affinity = irq_chip_set_affinity_parent,
+	.irq_eoi = irq_chip_eoi_parent,
+	.irq_mask = irq_chip_mask_parent,
+	.irq_unmask = irq_chip_unmask_parent
+};
+
+static unsigned int hv_msi_get_int_vector(struct irq_data *irqd)
+{
+	return irqd->parent_data->hwirq;
+}
+
+static void hv_set_msi_entry_from_desc(union hv_msi_entry *msi_entry,
+				       struct msi_desc *msi_desc)
+{
+	msi_entry->address = ((u64)msi_desc->msg.address_hi << 32) |
+			      msi_desc->msg.address_lo;
+	msi_entry->data = msi_desc->msg.data;
+}
+
+static void hv_pci_vec_irq_domain_free(struct irq_domain *domain,
+				       unsigned int virq, unsigned int nr_irqs)
+{
+	struct hv_pci_chip_data *chip_data = domain->host_data;
+	struct irq_data *irqd = irq_domain_get_irq_data(domain, virq);
+	int first = irqd->hwirq - HV_PCI_MSI_SPI_START;
+
+	mutex_lock(&chip_data->map_lock);
+	bitmap_release_region(chip_data->spi_map,
+			      first,
+			      get_count_order(nr_irqs));
+	mutex_unlock(&chip_data->map_lock);
+	irq_domain_reset_irq_data(irqd);
+	irq_domain_free_irqs_parent(domain, virq, nr_irqs);
+}
+
+static int hv_pci_vec_alloc_device_irq(struct irq_domain *domain,
+				       unsigned int nr_irqs,
+				       irq_hw_number_t *hwirq)
+{
+	struct hv_pci_chip_data *chip_data = domain->host_data;
+	unsigned int index;
+
+	/* Find and allocate region from the SPI bitmap */
+	mutex_lock(&chip_data->map_lock);
+	index = bitmap_find_free_region(chip_data->spi_map,
+					HV_PCI_MSI_SPI_NR,
+					get_count_order(nr_irqs));
+	mutex_unlock(&chip_data->map_lock);
+	if (index < 0)
+		return -ENOSPC;
+
+	*hwirq = index + HV_PCI_MSI_SPI_START;
+
+	return 0;
+}
+
+static int hv_pci_vec_irq_gic_domain_alloc(struct irq_domain *domain,
+					   unsigned int virq,

drivers/pci/controller/pci-hyperv.c static int hv_msi_prepare(struct irq_domain *domain, struct device *dev, · line 710

Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +10 -0
+					   irq_hw_number_t hwirq)
+{
+	struct irq_fwspec fwspec;
+
+	fwspec.fwnode = domain->parent->fwnode;
+	fwspec.param_count = 2;
+	fwspec.param[0] = hwirq;
+	fwspec.param[1] = IRQ_TYPE_EDGE_RISING;
+
+	return irq_domain_alloc_irqs_parent(domain, virq, 1, &fwspec);

I think you are missing the actual edge configuration here. Since the interrupt specifier doesn't come from either DT or ACPI, nobody will set the trigger type, and you have to do it yourself here. At the moment, you will get whatever is in the GIC configuration.

Quoted 26 lines
drivers/pci/controller/pci-hyperv.c +26 -0
+}
+
+static int hv_pci_vec_irq_domain_alloc(struct irq_domain *domain,
+				       unsigned int virq, unsigned int nr_irqs,
+				       void *args)
+{
+	irq_hw_number_t hwirq;
+	unsigned int i;
+	int ret;
+
+	ret = hv_pci_vec_alloc_device_irq(domain, nr_irqs, &hwirq);
+	if (ret)
+		return ret;
+
+	for (i = 0; i < nr_irqs; i++) {
+		ret = hv_pci_vec_irq_gic_domain_alloc(domain, virq + i,
+						      hwirq + i);
+		if (ret)
+			goto free_irq;
+
+		ret = irq_domain_set_hwirq_and_chip(domain, virq + i,
+						    hwirq + i,
+						    &hv_arm64_msi_irq_chip,
+						    domain->host_data);
+		if (ret)
+			goto free_irq;

drivers/pci/controller/pci-hyperv.c static int hv_msi_prepare(struct irq_domain *domain, struct device *dev, · line 746

Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +10 -0
+
+		pr_debug("pID:%d vID:%u\n", (int)(hwirq + i), virq + i);
+	}
+
+	return 0;
+
+free_irq:
+	hv_pci_vec_irq_domain_free(domain, virq, nr_irqs);
+
+	return ret;

How about the interrupts that have already been allocated?

Quoted 26 lines
drivers/pci/controller/pci-hyperv.c +26 -0
+}
+
+/*
+ * Pick the first online cpu as the irq affinity that can be temporarily used
+ * for composing MSI from the hypervisor. GIC will eventually set the right
+ * affinity for the irq and the 'unmask' will retarget the interrupt to that
+ * cpu.
+ */
+static int hv_pci_vec_irq_domain_activate(struct irq_domain *domain,
+					  struct irq_data *irqd, bool reserve)
+{
+	int cpu = cpumask_first(cpu_online_mask);
+
+	irq_data_update_effective_affinity(irqd, cpumask_of(cpu));
+
+	return 0;
+}
+
+static const struct irq_domain_ops hv_pci_domain_ops = {
+	.alloc	= hv_pci_vec_irq_domain_alloc,
+	.free	= hv_pci_vec_irq_domain_free,
+	.activate = hv_pci_vec_irq_domain_activate,
+};
+
+static int hv_pci_irqchip_init(void)
+{

drivers/pci/controller/pci-hyperv.c static int hv_msi_prepare(struct irq_domain *domain, struct device *dev, · line 782

Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +10 -0
+	static struct hv_pci_chip_data *chip_data;
+	struct fwnode_handle *fn = NULL;
+	int ret = -ENOMEM;
+
+	chip_data = kzalloc(sizeof(*chip_data), GFP_KERNEL);
+	if (!chip_data)
+		return ret;
+
+	mutex_init(&chip_data->map_lock);
+	fn = irq_domain_alloc_named_fwnode("Hyper-V ARM64 vPCI");

This will appear in debugfs. I'd rather you keep it short, sweet and without spaces. "hv_vpci_arm64" seems better to me.

Quoted 54 lines
drivers/pci/controller/pci-hyperv.c +36 -0
+	if (!fn)
+		goto free_chip;
+
+	/*
+	 * IRQ domain once enabled, should not be removed since there is no
+	 * way to ensure that all the corresponding devices are also gone and
+	 * no interrupts will be generated.
+	 */
+	hv_msi_gic_irq_domain = acpi_irq_create_hierarchy(0, HV_PCI_MSI_SPI_NR,
+							  fn, &hv_pci_domain_ops,
+							  chip_data);
+
+	if (!hv_msi_gic_irq_domain) {
+		pr_err("Failed to create Hyper-V arm64 vPCI MSI IRQ domain\n");
+		goto free_chip;
+	}
+
+	return 0;
+
+free_chip:
+	kfree(chip_data);
+	if (fn)
+		irq_domain_free_fwnode(fn);
+
+	return ret;
+}
+
+static struct irq_domain *hv_pci_get_root_domain(void)
+{
+	return hv_msi_gic_irq_domain;
+}
+#endif /* CONFIG_ARM64 */
 
 /**
  * hv_pci_generic_compl() - Invoked for a completion packet
@@ -1227,6 +1424,8 @@ static void hv_msi_free(struct irq_domain *domain, struct msi_domain_info *info,
 static void hv_irq_mask(struct irq_data *data)
 {
 	pci_msi_mask_irq(data);
+	if (data->parent_data->chip->irq_mask)
+		irq_chip_mask_parent(data);
 }
 
 /**
@@ -1343,6 +1542,8 @@ static void hv_irq_unmask(struct irq_data *data)
 		dev_err(&hbus->hdev->device,
 			"%s() failed: %#llx", __func__, res);
 
+	if (data->parent_data->chip->irq_unmask)
+		irq_chip_unmask_parent(data);
 	pci_msi_unmask_irq(data);
 }
 

drivers/pci/controller/pci-hyperv.c static struct irq_chip hv_msi_irq_chip = { · line 1825

Quoted 6 lines
drivers/pci/controller/pci-hyperv.c +1 -0
@@ -1619,6 +1820,7 @@ static struct irq_chip hv_msi_irq_chip = {
 	.irq_compose_msi_msg	= hv_compose_msi_msg,
 	.irq_set_affinity	= irq_chip_set_affinity_parent,
 	.irq_ack		= irq_chip_ack_parent,
+	.irq_eoi		= irq_chip_eoi_parent,
 	.irq_mask		= hv_irq_mask,
 	.irq_unmask		= hv_irq_unmask,

You probably want to avoid unconditionally setting callbacks that may have side effects on another architecture (ack on arm64, eoi on x86).

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#4 lore raw

RE: [EXTERNAL] Re: [PATCH v6 2/2] arm64: PCI: hv: Add support for Hyper-V vPCI

On Friday, November 19, 2021 7:47 AM,
Marc Zyngier <[email protected]> wrote:

[nip..]

Quoted 30 lines
Quoted 30 lines
drivers/pci/controller/pci-hyperv.c +30 -0
+static int hv_pci_vec_alloc_device_irq(struct irq_domain *domain,
+				       unsigned int nr_irqs,
+				       irq_hw_number_t *hwirq)
+{
+	struct hv_pci_chip_data *chip_data = domain->host_data;
+	unsigned int index;
+
+	/* Find and allocate region from the SPI bitmap */
+	mutex_lock(&chip_data->map_lock);
+	index = bitmap_find_free_region(chip_data->spi_map,
+					HV_PCI_MSI_SPI_NR,
+					get_count_order(nr_irqs));
+	mutex_unlock(&chip_data->map_lock);
+	if (index < 0)
+		return -ENOSPC;
+
+	*hwirq = index + HV_PCI_MSI_SPI_START;
+
+	return 0;
+}
+
+static int hv_pci_vec_irq_gic_domain_alloc(struct irq_domain *domain,
+					   unsigned int virq,
+					   irq_hw_number_t hwirq)
+{
+	struct irq_fwspec fwspec;
+
+	fwspec.fwnode = domain->parent->fwnode;
+	fwspec.param_count = 2;
+	fwspec.param[0] = hwirq;
Quoted 9 lines

drivers/pci/controller/pci-hyperv.c static int hv_msi_prepare(struct irq_domain *domain, struct device *dev, · line 710

Quoted 3 lines
drivers/pci/controller/pci-hyperv.c +3 -0
+	fwspec.param[1] = IRQ_TYPE_EDGE_RISING;
+
+	return irq_domain_alloc_irqs_parent(domain, virq, 1, &fwspec);

I think you are missing the actual edge configuration here. Since the interrupt specifier doesn't come from either DT or ACPI, nobody will set the trigger type, and you have to do it yourself here. At the moment, you will get whatever is in the GIC configuration.

I see, thanks. So, just a call of irq_set_irq_type(IRQ_TYPE_EDGE_RISING)?

Quoted 30 lines
Quoted 30 lines
drivers/pci/controller/pci-hyperv.c +30 -0
+}
+
+static int hv_pci_vec_irq_domain_alloc(struct irq_domain *domain,
+				       unsigned int virq, unsigned int nr_irqs,
+				       void *args)
+{
+	irq_hw_number_t hwirq;
+	unsigned int i;
+	int ret;
+
+	ret = hv_pci_vec_alloc_device_irq(domain, nr_irqs, &hwirq);
+	if (ret)
+		return ret;
+
+	for (i = 0; i < nr_irqs; i++) {
+		ret = hv_pci_vec_irq_gic_domain_alloc(domain, virq + i,
+						      hwirq + i);
+		if (ret)
+			goto free_irq;
+
+		ret = irq_domain_set_hwirq_and_chip(domain, virq + i,
+						    hwirq + i,
+						    &hv_arm64_msi_irq_chip,
+						    domain->host_data);
+		if (ret)
+			goto free_irq;
+
+		pr_debug("pID:%d vID:%u\n", (int)(hwirq + i), virq + i);
+	}
+
Quoted 8 lines

drivers/pci/controller/pci-hyperv.c static int hv_msi_prepare(struct irq_domain *domain, struct device *dev, · line 746

Quoted 6 lines
drivers/pci/controller/pci-hyperv.c +6 -0
+	return 0;
+
+free_irq:
+	hv_pci_vec_irq_domain_free(domain, virq, nr_irqs);
+
+	return ret;

How about the interrupts that have already been allocated?

Not sure I am fully following. If you are referring to the failure path and the interrupts that were allocated, then I am calling ' hv_pci_vec_irq_domain_free' which should free the interrupts from the bitmap and the parent irq domain. Can you please clarify?

Quoted 32 lines
Quoted 32 lines
drivers/pci/controller/pci-hyperv.c +31 -0
+}
+
+/*
+ * Pick the first online cpu as the irq affinity that can be temporarily used
+ * for composing MSI from the hypervisor. GIC will eventually set the right
+ * affinity for the irq and the 'unmask' will retarget the interrupt to that
+ * cpu.
+ */
+static int hv_pci_vec_irq_domain_activate(struct irq_domain *domain,
+					  struct irq_data *irqd, bool reserve)
+{
+	int cpu = cpumask_first(cpu_online_mask);
+
+	irq_data_update_effective_affinity(irqd, cpumask_of(cpu));
+
+	return 0;
+}
+
+static const struct irq_domain_ops hv_pci_domain_ops = {
+	.alloc	= hv_pci_vec_irq_domain_alloc,
+	.free	= hv_pci_vec_irq_domain_free,
+	.activate = hv_pci_vec_irq_domain_activate,
+};
+
+static int hv_pci_irqchip_init(void)
+{
+	static struct hv_pci_chip_data *chip_data;
+	struct fwnode_handle *fn = NULL;
+	int ret = -ENOMEM;
+
+	chip_data = kzalloc(sizeof(*chip_data), GFP_KERNEL);
Quoted 8 lines

drivers/pci/controller/pci-hyperv.c static int hv_msi_prepare(struct irq_domain *domain, struct device *dev, · line 782

Quoted 5 lines
drivers/pci/controller/pci-hyperv.c +5 -0
+	if (!chip_data)
+		return ret;
+
+	mutex_init(&chip_data->map_lock);
+	fn = irq_domain_alloc_named_fwnode("Hyper-V ARM64 vPCI");

This will appear in debugfs. I'd rather you keep it short, sweet and without spaces. "hv_vpci_arm64" seems better to me.

Sure, will fix in next version.

Quoted 11 lines

drivers/pci/controller/pci-hyperv.c static struct irq_chip hv_msi_irq_chip = { · line 1825

Quoted 8 lines
drivers/pci/controller/pci-hyperv.c +1 -0
@@ -1619,6 +1820,7 @@ static struct irq_chip hv_msi_irq_chip = {
 	.irq_compose_msi_msg	= hv_compose_msi_msg,
 	.irq_set_affinity	= irq_chip_set_affinity_parent,
 	.irq_ack		= irq_chip_ack_parent,
+	.irq_eoi		= irq_chip_eoi_parent,
 	.irq_mask		= hv_irq_mask,
 	.irq_unmask		= hv_irq_unmask,

You probably want to avoid unconditionally setting callbacks that may have side effects on another architecture (ack on arm64, eoi on x86).

Thanks. Will fix in next version.

Is there some other feedback that would like to see get addressed in the current patch? Trying to close down on all remaining feedback items here.

- Sunil

#5 lore raw

Re: [EXTERNAL] Re: [PATCH v6 2/2] arm64: PCI: hv: Add support for Hyper-V vPCI

Quoted 46 lines · Sunil Muthuswamy

On Friday, November 19, 2021 7:47 AM,
Marc Zyngier <[email protected]> wrote:

[nip..]

Quoted 30 lines
Quoted 30 lines
drivers/pci/controller/pci-hyperv.c +30 -0
+static int hv_pci_vec_alloc_device_irq(struct irq_domain *domain,
+				       unsigned int nr_irqs,
+				       irq_hw_number_t *hwirq)
+{
+	struct hv_pci_chip_data *chip_data = domain->host_data;
+	unsigned int index;
+
+	/* Find and allocate region from the SPI bitmap */
+	mutex_lock(&chip_data->map_lock);
+	index = bitmap_find_free_region(chip_data->spi_map,
+					HV_PCI_MSI_SPI_NR,
+					get_count_order(nr_irqs));
+	mutex_unlock(&chip_data->map_lock);
+	if (index < 0)
+		return -ENOSPC;
+
+	*hwirq = index + HV_PCI_MSI_SPI_START;
+
+	return 0;
+}
+
+static int hv_pci_vec_irq_gic_domain_alloc(struct irq_domain *domain,
+					   unsigned int virq,
+					   irq_hw_number_t hwirq)
+{
+	struct irq_fwspec fwspec;
+
+	fwspec.fwnode = domain->parent->fwnode;
+	fwspec.param_count = 2;
+	fwspec.param[0] = hwirq;
Quoted 9 lines

drivers/pci/controller/pci-hyperv.c static int hv_msi_prepare(struct irq_domain *domain, struct device *dev, · line 710

Quoted 3 lines
drivers/pci/controller/pci-hyperv.c +3 -0
+	fwspec.param[1] = IRQ_TYPE_EDGE_RISING;
+
+	return irq_domain_alloc_irqs_parent(domain, virq, 1, &fwspec);

I think you are missing the actual edge configuration here. Since the interrupt specifier doesn't come from either DT or ACPI, nobody will set the trigger type, and you have to do it yourself here. At the moment, you will get whatever is in the GIC configuration.

Quoted 1 line

I see, thanks. So, just a call of irq_set_irq_type(IRQ_TYPE_EDGE_RISING)?

You are already deep in the irq stack, and calling a high level function here is a pretty bad idea. You'll need something like:

	struct irq_data *d;
	d = irq_domain_get_irq_data(domain->parent, virq);
	d->chip->irq_set_type(d, IRQ_TYPE_EDGE_RISING);

on the return from the parent allocation.

Quoted 39 lines
Quoted 30 lines
Quoted 30 lines
drivers/pci/controller/pci-hyperv.c +30 -0
+}
+
+static int hv_pci_vec_irq_domain_alloc(struct irq_domain *domain,
+				       unsigned int virq, unsigned int nr_irqs,
+				       void *args)
+{
+	irq_hw_number_t hwirq;
+	unsigned int i;
+	int ret;
+
+	ret = hv_pci_vec_alloc_device_irq(domain, nr_irqs, &hwirq);
+	if (ret)
+		return ret;
+
+	for (i = 0; i < nr_irqs; i++) {
+		ret = hv_pci_vec_irq_gic_domain_alloc(domain, virq + i,
+						      hwirq + i);
+		if (ret)
+			goto free_irq;
+
+		ret = irq_domain_set_hwirq_and_chip(domain, virq + i,
+						    hwirq + i,
+						    &hv_arm64_msi_irq_chip,
+						    domain->host_data);
+		if (ret)
+			goto free_irq;
+
+		pr_debug("pID:%d vID:%u\n", (int)(hwirq + i), virq + i);
+	}
+
Quoted 8 lines

drivers/pci/controller/pci-hyperv.c static int hv_msi_prepare(struct irq_domain *domain, struct device *dev, · line 746

Quoted 6 lines
drivers/pci/controller/pci-hyperv.c +6 -0
+	return 0;
+
+free_irq:
+	hv_pci_vec_irq_domain_free(domain, virq, nr_irqs);
+
+	return ret;

How about the interrupts that have already been allocated?

Quoted 4 lines

Not sure I am fully following. If you are referring to the failure path and the interrupts that were allocated, then I am calling ' hv_pci_vec_irq_domain_free' which should free the interrupts from the bitmap and the parent irq domain. Can you please clarify?

I see several problems on the failure path:

- You are freeing more than you actually configured. Not necessary a big deal, but still (it is a common issue, and the core deals with it)

- hv_pci_vec_irq_domain_free() calls irq_domain_reset_irq_data() on a single pointer. Why? Either you wipe them all, or you don't.

Quoted 56 lines
Quoted 33 lines
Quoted 33 lines
drivers/pci/controller/pci-hyperv.c +31 -0
+}
+
+/*
+ * Pick the first online cpu as the irq affinity that can be temporarily used
+ * for composing MSI from the hypervisor. GIC will eventually set the right
+ * affinity for the irq and the 'unmask' will retarget the interrupt to that
+ * cpu.
+ */
+static int hv_pci_vec_irq_domain_activate(struct irq_domain *domain,
+					  struct irq_data *irqd, bool reserve)
+{
+	int cpu = cpumask_first(cpu_online_mask);
+
+	irq_data_update_effective_affinity(irqd, cpumask_of(cpu));
+
+	return 0;
+}
+
+static const struct irq_domain_ops hv_pci_domain_ops = {
+	.alloc	= hv_pci_vec_irq_domain_alloc,
+	.free	= hv_pci_vec_irq_domain_free,
+	.activate = hv_pci_vec_irq_domain_activate,
+};
+
+static int hv_pci_irqchip_init(void)
+{
+	static struct hv_pci_chip_data *chip_data;
+	struct fwnode_handle *fn = NULL;
+	int ret = -ENOMEM;
+
+	chip_data = kzalloc(sizeof(*chip_data), GFP_KERNEL);
Quoted 8 lines

drivers/pci/controller/pci-hyperv.c static int hv_msi_prepare(struct irq_domain *domain, struct device *dev, · line 782

Quoted 5 lines
drivers/pci/controller/pci-hyperv.c +5 -0
+	if (!chip_data)
+		return ret;
+
+	mutex_init(&chip_data->map_lock);
+	fn = irq_domain_alloc_named_fwnode("Hyper-V ARM64 vPCI");

This will appear in debugfs. I'd rather you keep it short, sweet and without spaces. "hv_vpci_arm64" seems better to me.

Sure, will fix in next version.

Quoted 11 lines

drivers/pci/controller/pci-hyperv.c static struct irq_chip hv_msi_irq_chip = { · line 1825

Quoted 8 lines
drivers/pci/controller/pci-hyperv.c +1 -0
@@ -1619,6 +1820,7 @@ static struct irq_chip hv_msi_irq_chip = {
 	.irq_compose_msi_msg	= hv_compose_msi_msg,
 	.irq_set_affinity	= irq_chip_set_affinity_parent,
 	.irq_ack		= irq_chip_ack_parent,
+	.irq_eoi		= irq_chip_eoi_parent,
 	.irq_mask		= hv_irq_mask,
 	.irq_unmask		= hv_irq_unmask,

You probably want to avoid unconditionally setting callbacks that may have side effects on another architecture (ack on arm64, eoi on x86).

Quoted 4 lines

Thanks. Will fix in next version.

Is there some other feedback that would like to see get addressed in the current patch? Trying to close down on all remaining feedback items here.

Not at the moment, as I have paged this out a long time ago. Addressing feedback more often than once a month would definitely help. I usually complain about patches being sent too often, but you're squarely in the opposite camp.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#6 lore raw
In reply to Sunil Muthuswamy
Quoted 3 lines · Sunil Muthuswamy

Sunil Muthuswamy (2):
PCI: hv: Make the code arch neutral by adding arch specific interfaces
arm64: PCI: hv: Add support for Hyper-V vPCI

Both patches are primarily to drivers/pci/controller/pci-hyperv.c, so why do the subject lines look so different?

Instead of making up a new format from scratch, look at the previous history and copy it:

  $ git log --oneline drivers/pci/controller/pci-hyperv.c
  f18312084300 ("PCI: hv: Remove unnecessary use of %hx")
  41608b64b10b ("PCI: hv: Fix sleep while in non-sleep context when removing child devices from the bus")
  88f94c7f8f40 ("PCI: hv: Turn on the host bridge probing on ARM64")
  9e7f9178ab49 ("PCI: hv: Set up MSI domain at bridge probing time")
  38c0d266dc80 ("PCI: hv: Set ->domain_nr of pci_host_bridge at probing time")
  418cb6c8e051 ("PCI: hv: Generify PCI probing")
  8f6a6b3c50ce ("PCI: hv: Support for create interrupt v3")
  7d815f4afa87 ("PCI: hv: Add check for hyperv_initialized in init_hv_pci_drv()")
  326dc2e1e59a ("PCI: hv: Remove bus device removal unused refcount/functions")
  ...

The second patch adds arm64 support, so it *should* mention arm64, but it can be something like this:

PCI: hv: Add arm64 Hyper-V vPCI support

Quoted 8 lines
 arch/arm64/include/asm/hyperv-tlfs.h |   9 +
 arch/x86/include/asm/hyperv-tlfs.h   |  33 ++++
 arch/x86/include/asm/mshyperv.h      |   7 -
 drivers/pci/Kconfig                  |   2 +-
 drivers/pci/controller/Kconfig       |   2 +-
 drivers/pci/controller/pci-hyperv.c  | 281 ++++++++++++++++++++++++---
 include/asm-generic/hyperv-tlfs.h    |  33 ----
 7 files changed, 300 insertions(+), 67 deletions(-)
#7 lore raw
Quoted 27 lines · Bjorn Helgaas
Quoted 3 lines · Sunil Muthuswamy

Sunil Muthuswamy (2):
PCI: hv: Make the code arch neutral by adding arch specific interfaces
arm64: PCI: hv: Add support for Hyper-V vPCI

Both patches are primarily to drivers/pci/controller/pci-hyperv.c, so why do the subject lines look so different?

Instead of making up a new format from scratch, look at the previous history and copy it:

  $ git log --oneline drivers/pci/controller/pci-hyperv.c
  f18312084300 ("PCI: hv: Remove unnecessary use of %hx")
  41608b64b10b ("PCI: hv: Fix sleep while in non-sleep context when removing
child devices from the bus")
  88f94c7f8f40 ("PCI: hv: Turn on the host bridge probing on ARM64")
  9e7f9178ab49 ("PCI: hv: Set up MSI domain at bridge probing time")
  38c0d266dc80 ("PCI: hv: Set ->domain_nr of pci_host_bridge at probing time")
  418cb6c8e051 ("PCI: hv: Generify PCI probing")
  8f6a6b3c50ce ("PCI: hv: Support for create interrupt v3")
  7d815f4afa87 ("PCI: hv: Add check for hyperv_initialized in init_hv_pci_drv()")
  326dc2e1e59a ("PCI: hv: Remove bus device removal unused
refcount/functions")
  ...
Quoted 13 lines

The second patch adds arm64 support, so it *should* mention arm64, but it can be something like this:

PCI: hv: Add arm64 Hyper-V vPCI support

Quoted 8 lines
 arch/arm64/include/asm/hyperv-tlfs.h |   9 +
 arch/x86/include/asm/hyperv-tlfs.h   |  33 ++++
 arch/x86/include/asm/mshyperv.h      |   7 -
 drivers/pci/Kconfig                  |   2 +-
 drivers/pci/controller/Kconfig       |   2 +-
 drivers/pci/controller/pci-hyperv.c  | 281 ++++++++++++++++++++++++---
 include/asm-generic/hyperv-tlfs.h    |  33 ----
 7 files changed, 300 insertions(+), 67 deletions(-)

Thanks. If I am reading your feedback above correctly, the only correction you are suggesting is to rename the subject line of the second patch to fix the format. Otherwise, the subject lines of the patches describe what they do. I will address this in v7.

- Sunil