Skip to contentSkip to conversation

Fix pci-hyperv interrupt code on a nested root partition

PATCH In review [email protected] 4 participants 12 messages Aug 22 – Aug 27

Subsystems (5)
lore
Share & tools
lore

[email protected]

author maintainer
#0 lore raw

cover Fix pci-hyperv interrupt code on a nested root partition

From: Wei Liu <[email protected]>

There are two issues in pci-hyperv when Linux runs as the root partition on a nested setup.

1. Interrupt state between Linux and the hypervisor is out of sync due to pci-hyperv not issuing unmap hypercall.
2. Interrupt migration is broken due to Linux doesn't retarget interrupts correctly.

The first patch is trivial. The two issues are fixed by two separate patches.

Wei

CC "K. Y. Srinivasan" <[email protected]>,
CC Haiyang Zhang <[email protected]>,
CC Dexuan Cui <[email protected]>,
CC Long Li <[email protected]>,

Wei Liu (3):
x86/hyperv: export hv_unmap_msi_interrupt()
PCI: hv: unmap MSI interrupt on the nested root partition teardown
path
PCI: hv: fix interrupt affinity change on the nested root partition

 arch/x86/hyperv/irqdomain.c         |   9 +-
 arch/x86/include/asm/mshyperv.h     |   2 +
 drivers/pci/controller/pci-hyperv.c | 122 ++++++++++++++++++++++++++--
 3 files changed, 122 insertions(+), 11 deletions(-)
Signature · 1 line
2.53.0
author maintainer
#1 lore raw

PATCH 1/3 x86/hyperv: export hv_unmap_msi_interrupt()

From: Wei Liu <[email protected]>

hv_unmap_msi_interrupt() issues the UNMAP_DEVICE_INTERRUPT hypercall that tears down a device-interrupt mapping created by hv_map_msi_interrupt(). The latter is already exported and used by the pci-hyperv driver to map vPCI interrupts on a nested root partition; export the unmap counterpart so the driver can remove those mappings on teardown.

No functional change.

---
 arch/x86/hyperv/irqdomain.c     | 9 +++++----
 arch/x86/include/asm/mshyperv.h | 2 ++
 2 files changed, 7 insertions(+), 4 deletions(-)
arch/x86/hyperv/irqdomain.c +5 -4
index b3ad50a874dc..ecd18e9a97d8 100644
@@ -213,8 +213,8 @@ static void entry_to_msi_msg(struct hv_interrupt_entry *entry,
 	msg->data = entry->msi_entry.data.as_uint32;
 }
 
-static int hv_unmap_msi_interrupt(struct pci_dev *pdev,
-				  struct hv_interrupt_entry *irq_entry);
+int hv_unmap_msi_interrupt(struct pci_dev *pdev,
+			   struct hv_interrupt_entry *irq_entry);
 
 static void hv_irq_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
 {
@@ -266,14 +266,15 @@ static void hv_irq_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
 	entry_to_msi_msg(data->chip_data, msg);
 }
 
-static int hv_unmap_msi_interrupt(struct pci_dev *pdev,
-				  struct hv_interrupt_entry *irq_entry)
+int hv_unmap_msi_interrupt(struct pci_dev *pdev,
+			   struct hv_interrupt_entry *irq_entry)
 {
 	union hv_device_id hv_devid;
 
 	hv_devid = hv_build_devid_type_pci(pdev);
 	return hv_unmap_interrupt(hv_devid.as_uint64, irq_entry);
 }
+EXPORT_SYMBOL_GPL(hv_unmap_msi_interrupt);
 
 /* NB: during map, hv_interrupt_entry is saved via data->chip_data */
 static void hv_teardown_msi_irq(struct pci_dev *pdev, struct irq_data *irqd)
arch/x86/include/asm/mshyperv.h +2 -0
index f64393e853ee..7c84e27d777b 100644
@@ -191,6 +191,8 @@ struct irq_domain *hv_create_pci_msi_domain(void);
 
 int hv_map_msi_interrupt(struct irq_data *data,
 			 struct hv_interrupt_entry *out_entry);
+int hv_unmap_msi_interrupt(struct pci_dev *dev,
+			   struct hv_interrupt_entry *hvirqe);
 int hv_map_ioapic_interrupt(int ioapic_id, bool level, int vcpu, int vector,
 		struct hv_interrupt_entry *entry);
 int hv_unmap_ioapic_interrupt(int ioapic_id, struct hv_interrupt_entry *entry);
Signature · 1 line
2.53.0
author maintainer
#3 lore raw

PATCH 2/3 PCI: hv: unmap MSI interrupt on the nested root partition teardown path

In reply to [email protected]

From: Wei Liu <[email protected]>

On a nested root partition the vPCI MSI/MSI-X interrupts of vmbus devices (e.g. the MANA NIC) are mapped in the hypervisor with a MAP_DEVICE_INTERRUPT hypercall. This is done from hv_arch_irq_unmask() -> hv_map_msi_interrupt() because the nested hypervisor performs the interrupt remapping and a RETARGET_INTERRUPT is not usable there.

The mapping was never removed: hv_arch_irq_unmask() called hv_map_msi_interrupt(data, NULL), so the returned hv_interrupt_entry was discarded, and hv_msi_free() tears the interrupt down with a vmbus PCI_DELETE_INTERRUPT message (hv_int_desc_free()) without issuing UNMAP_DEVICE_INTERRUPT.

This has led to MSHV rejecting already-mapped (vp, vector) pair from being used. When this happens during early boot, the system hangs.

Keep the hypervisor mapping in sync with the kernel's interrupt lifecycle.

The mapping is only created on x86 (hv_arch_irq_unmask() is a stub on arm64), so the unmap hypercall is guarded accordingly.

---
 drivers/pci/controller/pci-hyperv.c | 85 ++++++++++++++++++++++++++---
 1 file changed, 78 insertions(+), 7 deletions(-)
drivers/pci/controller/pci-hyperv.c +78 -7
index cfc8fa403dad..5a36382742bf 100644
@@ -283,6 +283,35 @@ struct tran_int_desc {
 	u64	address;
 } __packed;
 
+/*
+ * On a nested root partition a vPCI MSI is mapped in the hypervisor with a
+ * MAP_DEVICE_INTERRUPT hypercall in hv_arch_irq_unmask().  Keep the entry the
+ * hypervisor returns next to the per-interrupt transaction descriptor so the
+ * mapping can be removed again with UNMAP_DEVICE_INTERRUPT when the interrupt
+ * is torn down.  tran_int_desc stays first: chip_data is used as a struct
+ * tran_int_desc throughout this driver.
+ */
+struct hv_msi_int_entry {
+	struct tran_int_desc		int_desc;
+	struct hv_interrupt_entry	hv_entry;
+};
+
+/* chip_data is passed around as a struct tran_int_desc *, so it must be first. */
+static_assert(offsetof(struct hv_msi_int_entry, int_desc) == 0);
+
+static void hv_vmbus_unmap_msi_interrupt(struct pci_dev *pdev __maybe_unused,
+					 void *chip_data)
+{
+	struct hv_msi_int_entry *ie = chip_data;
+
+	if (!ie || !ie->hv_entry.source)
+		return;
+#ifdef CONFIG_X86
+	hv_unmap_msi_interrupt(pdev, &ie->hv_entry);
+#endif
+	memset(&ie->hv_entry, 0, sizeof(ie->hv_entry));
+}
+
 /*
  * A generic message format for virtual PCI.
  * Specific message formats are defined later in the file.
@@ -715,16 +744,30 @@ static void hv_irq_retarget_interrupt(struct irq_data *data)
 
 static void hv_arch_irq_unmask(struct irq_data *data)
 {
-	if (hv_root_partition())
+	if (hv_root_partition()) {
 		/*
 		 * In case of the nested root partition, the nested hypervisor
 		 * is taking care of interrupt remapping and thus the
 		 * MAP_DEVICE_INTERRUPT hypercall is required instead of
 		 * RETARGET_INTERRUPT.
+		 *
+		 * Keep the returned entry so the mapping can be removed again
+		 * when the interrupt is torn down.
 		 */
-		(void)hv_map_msi_interrupt(data, NULL);
-	else
+		struct hv_msi_int_entry *ie = data->chip_data;
+
+		/*
+		 * A NULL chip_data means hv_compose_msi_msg() failed and the
+		 * interrupt was never set up, so there is nothing to map.
+		 */
+		if (!ie)
+			return;
+
+		if (hv_map_msi_interrupt(data, &ie->hv_entry))
+			memset(&ie->hv_entry, 0, sizeof(ie->hv_entry));
+	} else {
 		hv_irq_retarget_interrupt(data);
+	}
 }
 #elif defined(CONFIG_ARM64)
 /*
@@ -1708,6 +1751,7 @@ static void hv_msi_free(struct irq_domain *domain, unsigned int irq)
 		return;
 	}
 
+	hv_vmbus_unmap_msi_interrupt(pdev, int_desc);
 	hv_int_desc_free(hpdev, int_desc);
 	put_pcichild(hpdev);
 }
@@ -1882,6 +1926,7 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
 	const struct cpumask *dest;
 	struct compose_comp_ctxt comp;
 	struct tran_int_desc *int_desc;
+	struct hv_msi_int_entry *int_entry;
 	struct msi_desc *msi_desc;
 	/*
 	 * vector_count should be u16: see hv_msi_desc, hv_msi_desc2
@@ -1932,9 +1977,10 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
 		hv_int_desc_free(hpdev, int_desc);
 	}
 
-	int_desc = kzalloc_obj(*int_desc, GFP_ATOMIC);
-	if (!int_desc)
+	int_entry = kzalloc_obj(*int_entry, GFP_ATOMIC);
+	if (!int_entry)
 		goto drop_reference;
+	int_desc = &int_entry->int_desc;
 
 	if (multi_msi) {
 		/*
@@ -2184,9 +2230,34 @@ static void hv_pcie_domain_free(struct irq_domain *d, unsigned int virq, unsigne
 	irq_domain_free_irqs_top(d, virq, nr_irqs);
 }
 
+/*
+ * Runs from irq_domain_deactivate_irq() during irq_shutdown(), before the
+ * parent (x86 vector) domain is deactivated and the (cpu, vector) is returned
+ * to the matrix allocator, so a freed vector can never collide with a stale
+ * hypervisor entry when it is reused.
+ */
+static void hv_pcie_domain_deactivate(struct irq_domain *d,
+				      struct irq_data *data)
+{
+	struct msi_desc *msi_desc;
+	struct pci_dev *pdev;
+
+	if (!hv_root_partition())
+		return;
+
+	msi_desc = irq_data_get_msi_desc(data);
+	if (!msi_desc)
+		return;
+
+	pdev = msi_desc_to_pci_dev(msi_desc);
+	if (pdev)
+		hv_vmbus_unmap_msi_interrupt(pdev, data->chip_data);
+}
+
 static const struct irq_domain_ops hv_pcie_domain_ops = {
-	.alloc	= hv_pcie_domain_alloc,
-	.free	= hv_pcie_domain_free,
+	.alloc		= hv_pcie_domain_alloc,
+	.free		= hv_pcie_domain_free,
+	.deactivate	= hv_pcie_domain_deactivate,
 };
 
 /**
Signature · 1 line
2.53.0
author maintainer
#8 lore raw

PATCH 3/3 PCI: hv: fix interrupt affinity change on the nested root partition

In reply to [email protected]

From: Wei Liu <[email protected]>

hv_compose_msi_msg() sends PCI_CREATE_INTERRUPT carrying int_desc.vector, and the vPCI backend programs the device with the address/data it returns for it.

An affinity change only ran irq_chip_set_affinity_parent(), which re-allocates the x86 vector and nothing else, and hv_arch_irq_unmask() then issued MAP_DEVICE_INTERRUPT for the new (VP, vector). Nothing re-composed the interrupt, so the device kept signalling the vector it was created with and the new mapping was never used. This led to loss of interrupts.

Do the re-target where the vector is known and the interrupt is quiescent. If the vector changed, re-compose the VMBus interrupt for it, write the resulting message to the device, and only then map the new (vp, vector); the old mapping is destroyed as the new message is composed.

---
 drivers/pci/controller/pci-hyperv.c | 41 +++++++++++++++++++++++++++--
 1 file changed, 39 insertions(+), 2 deletions(-)
drivers/pci/controller/pci-hyperv.c +39 -2
index 5a36382742bf..5acb8e41567e 100644
@@ -294,6 +294,7 @@ struct tran_int_desc {
 struct hv_msi_int_entry {
 	struct tran_int_desc		int_desc;
 	struct hv_interrupt_entry	hv_entry;
+	unsigned int			mapped_vector;
 };
 
 /* chip_data is passed around as a struct tran_int_desc *, so it must be first. */
@@ -742,6 +743,8 @@ static void hv_irq_retarget_interrupt(struct irq_data *data)
 			"%s() failed: %#llx", __func__, res);
 }
 
+static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg);
+
 static void hv_arch_irq_unmask(struct irq_data *data)
 {
 	if (hv_root_partition()) {
@@ -752,9 +755,17 @@ static void hv_arch_irq_unmask(struct irq_data *data)
 		 * RETARGET_INTERRUPT.
 		 *
 		 * Keep the returned entry so the mapping can be removed again
-		 * when the interrupt is torn down.
+		 * when the interrupt is re-targeted or torn down.
+		 *
+		 * This is also the re-target point.  The core calls us from
+		 * __irq_move_irq() with the interrupt masked once the new
+		 * vector has been assigned, so if the vector changed the vmbus
+		 * interrupt is re-composed for it first -- PCI_CREATE_INTERRUPT
+		 * carries the vector, so the device would otherwise keep
+		 * signalling the one it was created with.
 		 */
 		struct hv_msi_int_entry *ie = data->chip_data;
+		unsigned int vec = hv_msi_get_int_vector(data);
 
 		/*
 		 * A NULL chip_data means hv_compose_msi_msg() failed and the
@@ -763,8 +774,29 @@ static void hv_arch_irq_unmask(struct irq_data *data)
 		if (!ie)
 			return;
 
-		if (hv_map_msi_interrupt(data, &ie->hv_entry))
+		/* Already mapped for this vector, nothing changed. */
+		if (ie->mapped_vector == vec && ie->hv_entry.source)
+			return;
+
+		if (ie->mapped_vector && ie->mapped_vector != vec) {
+			struct msi_msg msg;
+
+			hv_compose_msi_msg(data, &msg);
+
+			ie = data->chip_data;
+			if (!ie)
+				return;
+
+			if (data->chip->irq_write_msi_msg)
+				data->chip->irq_write_msi_msg(data, &msg);
+		}
+
+		if (hv_map_msi_interrupt(data, &ie->hv_entry)) {
 			memset(&ie->hv_entry, 0, sizeof(ie->hv_entry));
+			ie->mapped_vector = 0;
+			return;
+		}
+		ie->mapped_vector = vec;
 	} else {
 		hv_irq_retarget_interrupt(data);
 	}
@@ -1974,6 +2006,11 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
 	if (data->chip_data && !multi_msi) {
 		int_desc = data->chip_data;
 		data->chip_data = NULL;
+		/*
+		 * The descriptor is about to be destroyed, so release the
+		 * hypervisor mapping that belongs to it first.
+		 */
+		hv_vmbus_unmap_msi_interrupt(pdev, int_desc);
 		hv_int_desc_free(hpdev, int_desc);
 	}
 
Signature · 1 line
2.53.0
#4 lore raw

Re: [PATCH 2/3] PCI: hv: unmap MSI interrupt on the nested root partition teardown path

In reply to [email protected]
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 17 lines

From: Wei Liu <[email protected]>

On a nested root partition the vPCI MSI/MSI-X interrupts of vmbus devices (e.g. the MANA NIC) are mapped in the hypervisor with a MAP_DEVICE_INTERRUPT hypercall. This is done from hv_arch_irq_unmask() -> hv_map_msi_interrupt() because the nested hypervisor performs the interrupt remapping and a RETARGET_INTERRUPT is not usable there.

The mapping was never removed: hv_arch_irq_unmask() called hv_map_msi_interrupt(data, NULL), so the returned hv_interrupt_entry was discarded, and hv_msi_free() tears the interrupt down with a vmbus PCI_DELETE_INTERRUPT message (hv_int_desc_free()) without issuing UNMAP_DEVICE_INTERRUPT.

This has led to MSHV rejecting already-mapped (vp, vector) pair from being used. When this happens during early boot, the system hangs.

Quoted 7 lines

Keep the hypervisor mapping in sync with the kernel's interrupt lifecycle.

The mapping is only created on x86 (hv_arch_irq_unmask() is a stub on arm64), so the unmap hypercall is guarded accordingly.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#9 lore raw

Re: [PATCH 3/3] PCI: hv: fix interrupt affinity change on the nested root partition

In reply to [email protected]
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] The newly composed MSI message on affinity change is never written to the PCI device because…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] The newly composed MSI message on affinity change is never written to the PCI device because data->chip->irq_write_msi_msg is NULL.
- [High] VMBus recomposition is permanently skipped when mapped_vector is 0, causing the device to use a stale vector.
- [High] Multi-MSI interrupts leak hypervisor mappings on every affinity change.
--

--- Patch [3]: [PATCH 3/3] PCI: hv: fix interrupt affinity change on the nested root partition --- commit b9428909b0f32d968c5c2cd7f2d85948c6d7893a
Author: Wei Liu <[email protected]>

PCI: hv: fix interrupt affinity change on the nested root partition

This commit updates interrupt handling on affinity changes in the nested root partition. It recomposes the VMBus interrupt with the new vector and programs the device with the resulting message before mapping the new interrupt.

Quoted 4 lines
drivers/pci/controller/pci-hyperv.c +0 -0
index 5a36382742bf7..5acb8e41567e6 100644

[ ... ]

Quoted 20 lines
drivers/pci/controller/pci-hyperv.c +9 -1
@@ -752,9 +755,17 @@ static void hv_arch_irq_unmask(struct irq_data *data)
 		 * RETARGET_INTERRUPT.
 		 *
 		 * Keep the returned entry so the mapping can be removed again
-		 * when the interrupt is torn down.
+		 * when the interrupt is re-targeted or torn down.
+		 *
+		 * This is also the re-target point.  The core calls us from
+		 * __irq_move_irq() with the interrupt masked once the new
+		 * vector has been assigned, so if the vector changed the vmbus
+		 * interrupt is re-composed for it first -- PCI_CREATE_INTERRUPT
+		 * carries the vector, so the device would otherwise keep
+		 * signalling the one it was created with.
 		 */
 		struct hv_msi_int_entry *ie = data->chip_data;
+		unsigned int vec = hv_msi_get_int_vector(data);
 
 		/*
 		 * A NULL chip_data means hv_compose_msi_msg() failed and the

drivers/pci/controller/pci-hyperv.c static void hv_arch_irq_unmask(struct irq_data *data) · line 781

Quoted 9 lines
drivers/pci/controller/pci-hyperv.c +5 -1
@@ -763,8 +774,29 @@ static void hv_arch_irq_unmask(struct irq_data *data)
 		if (!ie)
 			return;
 
-		if (hv_map_msi_interrupt(data, &ie->hv_entry))
+		/* Already mapped for this vector, nothing changed. */
+		if (ie->mapped_vector == vec && ie->hv_entry.source)
+			return;
+
+		if (ie->mapped_vector && ie->mapped_vector != vec) {

[Severity: High]
If hv_map_msi_interrupt() fails previously, ie->mapped_vector is set to 0.

On the next unmask, if the vector has changed due to an affinity change, will this ie->mapped_vector != 0 check skip recomposing the message?

It appears this could leave the device programmed with the old vector while the hypervisor is programmed with the new one.

drivers/pci/controller/pci-hyperv.c static void hv_arch_irq_unmask(struct irq_data *data) · line 791

Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +10 -0
+			struct msi_msg msg;
+
+			hv_compose_msi_msg(data, &msg);
+
+			ie = data->chip_data;
+			if (!ie)
+				return;
+
+			if (data->chip->irq_write_msi_msg)
+				data->chip->irq_write_msi_msg(data, &msg);

[Severity: High]
Does hv_msi_irq_chip actually implement the irq_write_msi_msg callback?

Looking at the hv_msi_irq_chip definition, it seems to lack this callback.

If so, would this check fail and prevent the new MSI message from ever being written to the device, causing it to continue signalling the old unmapped vector?

drivers/pci/controller/pci-hyperv.c static void hv_arch_irq_unmask(struct irq_data *data) · line 794

Quoted 3 lines
drivers/pci/controller/pci-hyperv.c +3 -0
+		}
+
+		if (hv_map_msi_interrupt(data, &ie->hv_entry)) {

[Severity: High]
For multi-MSI devices, does this overwrite ie->hv_entry and leak hypervisor mappings on every affinity change?

When hv_compose_msi_msg() is called for a multi-MSI device, it hits an early return and bypasses the hv_vmbus_unmap_msi_interrupt() cleanup:

hv_compose_msi_msg() {
    ...
    if (data->chip_data && multi_msi) {
        ...
        return;
    }
    ...
}

Then hv_map_msi_interrupt() is called here, which issues a new HVCALL_MAP_DEVICE_INTERRUPT and overwrites ie->hv_entry. Since the previous mapping wasn't unmapped, is it permanently leaked?

Quoted 8 lines
drivers/pci/controller/pci-hyperv.c +4 -0
 			memset(&ie->hv_entry, 0, sizeof(ie->hv_entry));
+			ie->mapped_vector = 0;
+			return;
+		}
+		ie->mapped_vector = vec;
 	} else {
 		hv_irq_retarget_interrupt(data);
 	}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Reviewed-by on v1 1/3
#2 lore raw

RE: [EXTERNAL] [PATCH 1/3] x86/hyperv: export hv_unmap_msi_interrupt()

In reply to [email protected]
Quoted 10 lines

From: Wei Liu <[email protected]>

hv_unmap_msi_interrupt() issues the UNMAP_DEVICE_INTERRUPT hypercall that tears down a device-interrupt mapping created by hv_map_msi_interrupt().
The latter is already exported and used by the pci-hyperv driver to map vPCI interrupts on a nested root partition; export the unmap counterpart so the driver can remove those mappings on teardown.

Quoted 3 lines

No functional change.

Quoted 52 lines
---
 arch/x86/hyperv/irqdomain.c     | 9 +++++----
 arch/x86/include/asm/mshyperv.h | 2 ++
 2 files changed, 7 insertions(+), 4 deletions(-)
arch/x86/hyperv/irqdomain.c +2 -2
index b3ad50a874dc..ecd18e9a97d8 100644
@@ -213,8 +213,8 @@ static void entry_to_msi_msg(struct
 hv_interrupt_entry *entry,
 	msg->data = entry->msi_entry.data.as_uint32;  }
 
-static int hv_unmap_msi_interrupt(struct pci_dev *pdev,
-				  struct hv_interrupt_entry *irq_entry);
+int hv_unmap_msi_interrupt(struct pci_dev *pdev,
+			   struct hv_interrupt_entry *irq_entry);
 
 static void hv_irq_compose_msi_msg(struct irq_data *data, struct msi_msg
 *msg)  { @@ -266,14 +266,15 @@ static void

hv_irq_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)

arch/x86/hyperv/irqdomain.c +3 -2
 	entry_to_msi_msg(data->chip_data, msg);  }
 
-static int hv_unmap_msi_interrupt(struct pci_dev *pdev,
-				  struct hv_interrupt_entry *irq_entry)
+int hv_unmap_msi_interrupt(struct pci_dev *pdev,
+			   struct hv_interrupt_entry *irq_entry)
 {
 	union hv_device_id hv_devid;
 
 	hv_devid = hv_build_devid_type_pci(pdev);
 	return hv_unmap_interrupt(hv_devid.as_uint64, irq_entry);  }
+EXPORT_SYMBOL_GPL(hv_unmap_msi_interrupt);
 
 /* NB: during map, hv_interrupt_entry is saved via data->chip_data */  static

void hv_teardown_msi_irq(struct pci_dev *pdev, struct irq_data *irqd) diff -- git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h index f64393e853ee..7c84e27d777b 100644

arch/x86/include/asm/mshyperv.h +2 -0
@@ -191,6 +191,8 @@ struct irq_domain *hv_create_pci_msi_domain(void);
 
 int hv_map_msi_interrupt(struct irq_data *data,
 			 struct hv_interrupt_entry *out_entry);
+int hv_unmap_msi_interrupt(struct pci_dev *dev,
+			   struct hv_interrupt_entry *hvirqe);
 int hv_map_ioapic_interrupt(int ioapic_id, bool level, int vcpu, int vector,
 		struct hv_interrupt_entry *entry);
 int hv_unmap_ioapic_interrupt(int ioapic_id, struct hv_interrupt_entry

*entry);
--
2.53.0

#5 lore raw

RE: [EXTERNAL] [PATCH 2/3] PCI: hv: unmap MSI interrupt on the nested root partition teardown path

In reply to [email protected]
Quoted 62 lines

From: Wei Liu <[email protected]>

On a nested root partition the vPCI MSI/MSI-X interrupts of vmbus devices (e.g. the MANA NIC) are mapped in the hypervisor with a MAP_DEVICE_INTERRUPT hypercall. This is done from hv_arch_irq_unmask() -> hv_map_msi_interrupt() because the nested hypervisor performs the interrupt remapping and a RETARGET_INTERRUPT is not usable there.

The mapping was never removed: hv_arch_irq_unmask() called hv_map_msi_interrupt(data, NULL), so the returned hv_interrupt_entry was discarded, and hv_msi_free() tears the interrupt down with a vmbus PCI_DELETE_INTERRUPT message (hv_int_desc_free()) without issuing UNMAP_DEVICE_INTERRUPT.

This has led to MSHV rejecting already-mapped (vp, vector) pair from being used. When this happens during early boot, the system hangs.

Keep the hypervisor mapping in sync with the kernel's interrupt lifecycle.

The mapping is only created on x86 (hv_arch_irq_unmask() is a stub on arm64), so the unmap hypercall is guarded accordingly.

---
 drivers/pci/controller/pci-hyperv.c | 85 ++++++++++++++++++++++++++---
 1 file changed, 78 insertions(+), 7 deletions(-)
drivers/pci/controller/pci- renamed from drivers/pci/controller/pci-hyperv.c +0 -0

hyperv.c
index cfc8fa403dad..5a36382742bf 100644

drivers/pci/controller/pci-hyperv.c +8 -0
@@ -283,6 +283,35 @@ struct tran_int_desc {
 	u64	address;
 } __packed;
 
+/*
+ * On a nested root partition a vPCI MSI is mapped in the hypervisor
+with a
+ * MAP_DEVICE_INTERRUPT hypercall in hv_arch_irq_unmask().  Keep the
+entry the
+ * hypervisor returns next to the per-interrupt transaction descriptor
+so the
+ * mapping can be removed again with UNMAP_DEVICE_INTERRUPT when

the

drivers/pci/controller/pci-hyperv.c +15 -0
+interrupt
+ * is torn down.  tran_int_desc stays first: chip_data is used as a
+struct
+ * tran_int_desc throughout this driver.
+ */
+struct hv_msi_int_entry {
+	struct tran_int_desc		int_desc;
+	struct hv_interrupt_entry	hv_entry;
+};
+
+/* chip_data is passed around as a struct tran_int_desc *, so it must
+be first. */ static_assert(offsetof(struct hv_msi_int_entry, int_desc)
+== 0);
+
+static void hv_vmbus_unmap_msi_interrupt(struct pci_dev *pdev

__maybe_unused,

drivers/pci/controller/pci-hyperv.c struct tran_int_desc {

Quoted 8 lines
drivers/pci/controller/pci-hyperv.c +8 -0
+					 void *chip_data)
+{
+	struct hv_msi_int_entry *ie = chip_data;
+
+	if (!ie || !ie->hv_entry.source)
+		return;
+#ifdef CONFIG_X86
+	hv_unmap_msi_interrupt(pdev, &ie->hv_entry); #endif

Since you don't want to check the return value from hv_unmap_msi_interrupt(), add a (void) to the front to prevent some tools from giving warnings.

#10 lore raw

RE: [EXTERNAL] [PATCH 3/3] PCI: hv: fix interrupt affinity change on the nested root partition

In reply to [email protected]
Quoted 78 lines

From: Wei Liu <[email protected]>

hv_compose_msi_msg() sends PCI_CREATE_INTERRUPT carrying int_desc.vector, and the vPCI backend programs the device with the address/data it returns for it.

An affinity change only ran irq_chip_set_affinity_parent(), which re-allocates the x86 vector and nothing else, and hv_arch_irq_unmask() then issued MAP_DEVICE_INTERRUPT for the new (VP, vector). Nothing re-composed the interrupt, so the device kept signalling the vector it was created with and the new mapping was never used. This led to loss of interrupts.

Do the re-target where the vector is known and the interrupt is quiescent. If the vector changed, re-compose the VMBus interrupt for it, write the resulting message to the device, and only then map the new (vp, vector); the old mapping is destroyed as the new message is composed.

---
 drivers/pci/controller/pci-hyperv.c | 41 +++++++++++++++++++++++++++--
 1 file changed, 39 insertions(+), 2 deletions(-)
drivers/pci/controller/pci- renamed from drivers/pci/controller/pci-hyperv.c +0 -0

hyperv.c
index 5a36382742bf..5acb8e41567e 100644

drivers/pci/controller/pci-hyperv.c +1 -0
@@ -294,6 +294,7 @@ struct tran_int_desc {  struct hv_msi_int_entry {
 	struct tran_int_desc		int_desc;
 	struct hv_interrupt_entry	hv_entry;
+	unsigned int			mapped_vector;
 };
 
 /* chip_data is passed around as a struct tran_int_desc *, so it must be first.
 */ @@ -742,6 +743,8 @@ static void hv_irq_retarget_interrupt(struct

irq_data *data)

drivers/pci/controller/pci-hyperv.c +3 -0
 			"%s() failed: %#llx", __func__, res);  }
 
+static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg
+*msg);
+
 static void hv_arch_irq_unmask(struct irq_data *data)  {
 	if (hv_root_partition()) {
@@ -752,9 +755,17 @@ static void hv_arch_irq_unmask(struct irq_data
 *data)
 		 * RETARGET_INTERRUPT.
 		 *
 		 * Keep the returned entry so the mapping can be removed

again

drivers/pci/controller/pci-hyperv.c +5 -1
-		 * when the interrupt is torn down.
+		 * when the interrupt is re-targeted or torn down.
+		 *
+		 * This is also the re-target point.  The core calls us from
+		 * __irq_move_irq() with the interrupt masked once the new
+		 * vector has been assigned, so if the vector changed the

vmbus

drivers/pci/controller/pci-hyperv.c +1 -0
+		 * interrupt is re-composed for it first --

PCI_CREATE_INTERRUPT

drivers/pci/controller/pci-hyperv.c +3 -0
+		 * carries the vector, so the device would otherwise keep
+		 * signalling the one it was created with.
 		 */
 		struct hv_msi_int_entry *ie = data->chip_data;
+		unsigned int vec = hv_msi_get_int_vector(data);
 
 		/*
 		 * A NULL chip_data means hv_compose_msi_msg() failed

and the @@ -763,8 +774,29 @@ static void hv_arch_irq_unmask(struct irq_data *data)

drivers/pci/controller/pci-hyperv.c +5 -1
 		if (!ie)
 			return;
 
-		if (hv_map_msi_interrupt(data, &ie->hv_entry))
+		/* Already mapped for this vector, nothing changed. */
+		if (ie->mapped_vector == vec && ie->hv_entry.source)
+			return;
+
+		if (ie->mapped_vector && ie->mapped_vector != vec) {

drivers/pci/controller/pci-hyperv.c static void hv_arch_irq_unmask(struct irq_data *data) · line 791

Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +10 -0
+			struct msi_msg msg;
+
+			hv_compose_msi_msg(data, &msg);
+
+			ie = data->chip_data;
+			if (!ie)
+				return;
+
+			if (data->chip->irq_write_msi_msg)
+				data->chip->irq_write_msi_msg(data, &msg);

I don't see the function irq_write_msi_msg defined in hv_msi_irq_chip, maybe not looking at the right place. Is this function ever getting called?

author maintainer
#6 lore raw

Re: [EXTERNAL] [PATCH 2/3] PCI: hv: unmap MSI interrupt on the nested root partition teardown path

In reply to Long Li

On Tue, Aug 25, 2026 at 08:12:33PM +0000, Long Li wrote:
[...]

Quoted 6 lines
Quoted 6 lines

+static void hv_vmbus_unmap_msi_interrupt(struct pci_dev *pdev __maybe_unused,

drivers/pci/controller/pci-hyperv.c +4 -0
+					 void *chip_data)
+{
+	struct hv_msi_int_entry *ie = chip_data;
+
Quoted 8 lines

drivers/pci/controller/pci-hyperv.c struct tran_int_desc {

Quoted 4 lines
drivers/pci/controller/pci-hyperv.c +4 -0
+	if (!ie || !ie->hv_entry.source)
+		return;
+#ifdef CONFIG_X86
+	hv_unmap_msi_interrupt(pdev, &ie->hv_entry); #endif

Since you don't want to check the return value from hv_unmap_msi_interrupt(), add a (void) to the front to prevent some tools from giving warnings.

No problem. Thanks for your review.

Wei

#7 lore raw

RE: [PATCH 2/3] PCI: hv: unmap MSI interrupt on the nested root partition teardown path

In reply to [email protected]

From: [email protected] <[email protected]> Sent: Friday, August 21, 2026 5:36 PM

Quoted 2 lines

On a nested root partition the vPCI MSI/MSI-X interrupts of vmbus

s/vmbus/VMBus/ [and other places in this commit msg]

Quoted 6 lines

devices (e.g. the MANA NIC) are mapped in the hypervisor with a MAP_DEVICE_INTERRUPT hypercall. This is done from hv_arch_irq_unmask() -> hv_map_msi_interrupt() because the nested hypervisor performs the interrupt remapping and a RETARGET_INTERRUPT is not usable there.

The mapping was never removed: hv_arch_irq_unmask() called

s/was/is/
s/called/calls/

The wording in this whole paragraph shifts to past tense. I'd suggest keeping present tense for consistency and per general kernel usage.

Quoted 5 lines

hv_map_msi_interrupt(data, NULL), so the returned hv_interrupt_entry was discarded, and hv_msi_free() tears the interrupt down with a vmbus PCI_DELETE_INTERRUPT message (hv_int_desc_free()) without issuing UNMAP_DEVICE_INTERRUPT.

Quoted 8 lines

This has led to MSHV rejecting already-mapped (vp, vector) pair from being used. When this happens during early boot, the system hangs.

Keep the hypervisor mapping in sync with the kernel's interrupt lifecycle.

The mapping is only created on x86 (hv_arch_irq_unmask() is a stub on arm64), so the unmap hypercall is guarded accordingly.

See comment below about the x86-only situation.

Quoted 32 lines
---
 drivers/pci/controller/pci-hyperv.c | 85 ++++++++++++++++++++++++++---
 1 file changed, 78 insertions(+), 7 deletions(-)
drivers/pci/controller/pci-hyperv.c +18 -0
index cfc8fa403dad..5a36382742bf 100644
@@ -283,6 +283,35 @@ struct tran_int_desc {
 	u64	address;
 } __packed;
 
+/*
+ * On a nested root partition a vPCI MSI is mapped in the hypervisor with a
+ * MAP_DEVICE_INTERRUPT hypercall in hv_arch_irq_unmask().  Keep the entry the
+ * hypervisor returns next to the per-interrupt transaction descriptor so the
+ * mapping can be removed again with UNMAP_DEVICE_INTERRUPT when the interrupt
+ * is torn down.  tran_int_desc stays first: chip_data is used as a struct
+ * tran_int_desc throughout this driver.
+ */
+struct hv_msi_int_entry {
+	struct tran_int_desc		int_desc;
+	struct hv_interrupt_entry	hv_entry;
+};
+
+/* chip_data is passed around as a struct tran_int_desc *, so it must be first. */
+static_assert(offsetof(struct hv_msi_int_entry, int_desc) == 0);
+
+static void hv_vmbus_unmap_msi_interrupt(struct pci_dev *pdev __maybe_unused,
+					 void *chip_data)

drivers/pci/controller/pci-hyperv.c struct tran_int_desc { · line 313

Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +10 -0
+{
+	struct hv_msi_int_entry *ie = chip_data;
+
+	if (!ie || !ie->hv_entry.source)
+		return;
+#ifdef CONFIG_X86
+	hv_unmap_msi_interrupt(pdev, &ie->hv_entry);
+#endif
+	memset(&ie->hv_entry, 0, sizeof(ie->hv_entry));
+}

This function seems out-of-place, as it is added in the middle of a bunch of structure definitions. There's large #ifdef CONFIG_X86 ... #elif defined(CONFIG_ARM64) ... #endif block in this source code file. I'd suggest putting this function alongside hv_arch_irq_unmask() in the x86 section, which is where hv_map_msi_interrupt() is called. Then put a stub in the arm64 section -- there's already a stub for hv_arch_irq_unmask(). And maybe name the function hv_arch_unmap_msi_interrupt() since it really doesn't have to do with VMBus stuff like channels, sending ring buffer messages, etc.

Quoted 12 lines
drivers/pci/controller/pci-hyperv.c +2 -1
+
 /*
  * A generic message format for virtual PCI.
  * Specific message formats are defined later in the file.
@@ -715,16 +744,30 @@ static void hv_irq_retarget_interrupt(struct irq_data *data)
 
 static void hv_arch_irq_unmask(struct irq_data *data)
 {
-	if (hv_root_partition())
+	if (hv_root_partition()) {
 		/*
 		 * In case of the nested root partition, the nested hypervisor

drivers/pci/controller/pci-hyperv.c static void hv_irq_retarget_interrupt(struct irq_data *data) · line 757

Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +4 -2
 		 * is taking care of interrupt remapping and thus the
 		 * MAP_DEVICE_INTERRUPT hypercall is required instead of
 		 * RETARGET_INTERRUPT.
+		 *
+		 * Keep the returned entry so the mapping can be removed again
+		 * when the interrupt is torn down.
 		 */
-		(void)hv_map_msi_interrupt(data, NULL);
-	else
+		struct hv_msi_int_entry *ie = data->chip_data;

There's the inline function irq_data_get_irq_chip_data() which seems to be used instead of directly referencing the field (at least most of the time throughout the kernel).

Quoted 61 lines
drivers/pci/controller/pci-hyperv.c +29 -2
+
+		/*
+		 * A NULL chip_data means hv_compose_msi_msg() failed and the
+		 * interrupt was never set up, so there is nothing to map.
+		 */
+		if (!ie)
+			return;
+
+		if (hv_map_msi_interrupt(data, &ie->hv_entry))
+			memset(&ie->hv_entry, 0, sizeof(ie->hv_entry));
+	} else {
 		hv_irq_retarget_interrupt(data);
+	}
 }
 #elif defined(CONFIG_ARM64)
 /*
@@ -1708,6 +1751,7 @@ static void hv_msi_free(struct irq_domain *domain, unsigned int irq)
 		return;
 	}
 
+	hv_vmbus_unmap_msi_interrupt(pdev, int_desc);
 	hv_int_desc_free(hpdev, int_desc);
 	put_pcichild(hpdev);
 }
@@ -1882,6 +1926,7 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
 	const struct cpumask *dest;
 	struct compose_comp_ctxt comp;
 	struct tran_int_desc *int_desc;
+	struct hv_msi_int_entry *int_entry;
 	struct msi_desc *msi_desc;
 	/*
 	 * vector_count should be u16: see hv_msi_desc, hv_msi_desc2
@@ -1932,9 +1977,10 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
 		hv_int_desc_free(hpdev, int_desc);
 	}
 
-	int_desc = kzalloc_obj(*int_desc, GFP_ATOMIC);
-	if (!int_desc)
+	int_entry = kzalloc_obj(*int_entry, GFP_ATOMIC);
+	if (!int_entry)
 		goto drop_reference;
+	int_desc = &int_entry->int_desc;
 
 	if (multi_msi) {
 		/*
@@ -2184,9 +2230,34 @@ static void hv_pcie_domain_free(struct irq_domain *d, unsigned int virq, unsigne
 	irq_domain_free_irqs_top(d, virq, nr_irqs);
 }
 
+/*
+ * Runs from irq_domain_deactivate_irq() during irq_shutdown(), before the
+ * parent (x86 vector) domain is deactivated and the (cpu, vector) is returned
+ * to the matrix allocator, so a freed vector can never collide with a stale
+ * hypervisor entry when it is reused.
+ */
+static void hv_pcie_domain_deactivate(struct irq_domain *d,
+				      struct irq_data *data)
+{
+	struct msi_desc *msi_desc;
+	struct pci_dev *pdev;
+

drivers/pci/controller/pci-hyperv.c static void hv_pcie_domain_free(struct irq_domain *d, unsigned int virq, unsigne · line 2254

Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +10 -0
+	if (!hv_root_partition())
+		return;
+
+	msi_desc = irq_data_get_msi_desc(data);
+	if (!msi_desc)
+		return;
+
+	pdev = msi_desc_to_pci_dev(msi_desc);
+	if (pdev)
+		hv_vmbus_unmap_msi_interrupt(pdev, data->chip_data);

Same here regarding direct reference to the chip_data field.

Quoted 14 lines
drivers/pci/controller/pci-hyperv.c +5 -3
+}
+
 static const struct irq_domain_ops hv_pcie_domain_ops = {
-	.alloc	= hv_pcie_domain_alloc,
-	.free	= hv_pcie_domain_free,
+	.alloc		= hv_pcie_domain_alloc,
+	.free		= hv_pcie_domain_free,
+	.deactivate	= hv_pcie_domain_deactivate,
 };
 
 /**
--

2.53.0

#11 lore raw

RE: [PATCH 3/3] PCI: hv: fix interrupt affinity change on the nested root partition

In reply to [email protected]

From: [email protected] <[email protected]> Sent: Friday, August 21, 2026 5:36 PM

Quoted 11 lines

hv_compose_msi_msg() sends PCI_CREATE_INTERRUPT carrying int_desc.vector, and the vPCI backend programs the device with the address/data it returns for it.

An affinity change only ran irq_chip_set_affinity_parent(), which re-allocates the x86 vector and nothing else, and hv_arch_irq_unmask() then issued MAP_DEVICE_INTERRUPT for the new (VP, vector). Nothing re-composed the interrupt, so the device kept signalling the vector it was created with and the new mapping was never used. This led to loss of interrupts.

Shift the wording in the above paragraph to present tense.

Quoted 3 lines

Do the re-target where the vector is known and the interrupt is quiescent. If the vector changed, re-compose the VMBus interrupt for

I don't think of this as "VMbus interrupt". VMBus interrupts are channel interrupts or control message interrupts tied to the HYPERVISOR_CALLBACK_VECTOR. I think this is just an MSI/MSI-X Interrupt from the guest's standpoint.

Quoted 35 lines

it, write the resulting message to the device, and only then map the new (vp, vector); the old mapping is destroyed as the new message is composed.

---
 drivers/pci/controller/pci-hyperv.c | 41 +++++++++++++++++++++++++++--
 1 file changed, 39 insertions(+), 2 deletions(-)
drivers/pci/controller/pci-hyperv.c +3 -1
index 5a36382742bf..5acb8e41567e 100644
@@ -294,6 +294,7 @@ struct tran_int_desc {
 struct hv_msi_int_entry {
 	struct tran_int_desc		int_desc;
 	struct hv_interrupt_entry	hv_entry;
+	unsigned int			mapped_vector;
 };
 
 /* chip_data is passed around as a struct tran_int_desc *, so it must be first. */
@@ -742,6 +743,8 @@ static void hv_irq_retarget_interrupt(struct irq_data *data)
 			"%s() failed: %#llx", __func__, res);
 }
 
+static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg);
+
 static void hv_arch_irq_unmask(struct irq_data *data)
 {
 	if (hv_root_partition()) {
@@ -752,9 +755,17 @@ static void hv_arch_irq_unmask(struct irq_data *data)
 		 * RETARGET_INTERRUPT.
 		 *
 		 * Keep the returned entry so the mapping can be removed again
-		 * when the interrupt is torn down.

drivers/pci/controller/pci-hyperv.c static void hv_arch_irq_unmask(struct irq_data *data) · line 767

Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +8 -0
+		 * when the interrupt is re-targeted or torn down.
+		 *
+		 * This is also the re-target point.  The core calls us from
+		 * __irq_move_irq() with the interrupt masked once the new
+		 * vector has been assigned, so if the vector changed the vmbus
+		 * interrupt is re-composed for it first -- PCI_CREATE_INTERRUPT
+		 * carries the vector, so the device would otherwise keep
+		 * signalling the one it was created with.
 		 */
 		struct hv_msi_int_entry *ie = data->chip_data;

Use irq_data_to_irq_chip_data()??

Quoted 7 lines
drivers/pci/controller/pci-hyperv.c +1 -0
+		unsigned int vec = hv_msi_get_int_vector(data);
 
 		/*
 		 * A NULL chip_data means hv_compose_msi_msg() failed and the
@@ -763,8 +774,29 @@ static void hv_arch_irq_unmask(struct irq_data *data)
 		if (!ie)
 			return;

drivers/pci/controller/pci-hyperv.c static void hv_arch_irq_unmask(struct irq_data *data) · line 784

Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +8 -1
 
-		if (hv_map_msi_interrupt(data, &ie->hv_entry))
+		/* Already mapped for this vector, nothing changed. */
+		if (ie->mapped_vector == vec && ie->hv_entry.source)
+			return;
+
+		if (ie->mapped_vector && ie->mapped_vector != vec) {
+			struct msi_msg msg;
+
+			hv_compose_msi_msg(data, &msg);

hv_compose_msi_msg() sends a VMBus message, and busy
waits for the ack, which is really ugly, but necessary. I have no idea whether it's OK to do all that from the irq_unmask function.

drivers/pci/controller/pci-hyperv.c static void hv_arch_irq_unmask(struct irq_data *data) · line 788

Quoted 4 lines
drivers/pci/controller/pci-hyperv.c +4 -0
+
+			ie = data->chip_data;
+			if (!ie)
+				return;

Setting ie and checking for NULL appears to be duplicative of code above.

drivers/pci/controller/pci-hyperv.c static void hv_arch_irq_unmask(struct irq_data *data) · line 791

Quoted 3 lines
drivers/pci/controller/pci-hyperv.c +3 -0
+
+			if (data->chip->irq_write_msi_msg)
+				data->chip->irq_write_msi_msg(data, &msg);

Is the irq_write_msi_msg function defaulting to pci_msi_domain_write_msg()? I didn't go follow all the paths to see how this function gets set ..... Is it even necessary to check for NULL before calling? I don't see such checks in the few other places irq_write_msi_msg is invoked.

Quoted 26 lines
drivers/pci/controller/pci-hyperv.c +12 -1
+		}
+
+		if (hv_map_msi_interrupt(data, &ie->hv_entry)) {
 			memset(&ie->hv_entry, 0, sizeof(ie->hv_entry));
+			ie->mapped_vector = 0;
+			return;
+		}
+		ie->mapped_vector = vec;
 	} else {
 		hv_irq_retarget_interrupt(data);
 	}
@@ -1974,6 +2006,11 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
 	if (data->chip_data && !multi_msi) {
 		int_desc = data->chip_data;
 		data->chip_data = NULL;
+		/*
+		 * The descriptor is about to be destroyed, so release the
+		 * hypervisor mapping that belongs to it first.
+		 */
+		hv_vmbus_unmap_msi_interrupt(pdev, int_desc);
 		hv_int_desc_free(hpdev, int_desc);
 	}
 
--

2.53.0