Skip to contentSkip to conversation

vfio/pci: Support dynamic allocation of MSI-X interrupts

RFC PATCH Superseded Reinette Chatre 5 participants 22 messages Mar 15, 2023 – Mar 20, 2023

Subsystems (2)

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

cover vfio/pci: Support dynamic allocation of MSI-X interrupts

== Cover letter ==

Qemu allocates interrupts incrementally at the time the guest unmasks an interrupt, for example each time a Linux guest runs request_irq().

Dynamic allocation of MSI-X interrupts was not possible until v6.2 [1]. This prompted Qemu to, when allocating a new interrupt, first release all previously allocated interrupts (including disable of MSI-X) followed by re-allocation of all interrupts that includes the new interrupt. Please see [2] for a detailed discussion about this issue.

Releasing and re-allocating interrupts may be acceptable if all interrupts are unmasked during device initialization. If unmasking of interrupts occur during runtime this may result in lost interrupts. For example, consider an accelerator device with multiple work queues, each work queue having a dedicated interrupt. A work queue can be enabled at any time with its associated interrupt unmasked while other work queues are already active. Having all interrupts released and MSI-X disabled to enable the new work queue will impact active work queues.

This series builds on the recent interrupt sub-system core changes that added support for dynamic MSI-X allocation after initial MSI-X enabling.

Add support for dynamic MSI-X allocation to vfio-pci. A flag indicating lack of support for dynamic allocation already exist:
VFIO_IRQ_INFO_NORESIZE and has always been set for MSI and MSI-X. With support for dynamic MSI-X the flag is cleared for MSI-X, enabling Qemu to modify its behavior.

== Why is this an RFC ? ==

vfio support for dynamic MSI-X needs to work with existing user space as well as upcoming user space that takes advantage of this feature. I would appreciate guidance on the expectations and requirements surrounding error handling when considering existing user space.

For example, consider the following scenario:
Start: Consider a passthrough device that supports 10 MSI-X interrupts
(0 to 9) and existing Qemu allocated interrupts 0 to 4.

Scenario: Qemu (hypothetically) attempts to associate a new action to
interrupts 0 to 7. Early checking of this range in vfio_set_irqs_validate_and_prepare() will pass since it is a valid range for the device. What happens after the early checking is considered next:

Current behavior (before this series): Since the provided range, 0 - 7,
exceeds the allocated range, no action will be taken on existing allocated interrupts 0 - 4 and the Qemu request will fail without making any state changes.

New behavior (with this series): Since vfio supports dynamic MSI-X new
interrupts will be allocated for vectors 5, 6, and 7. Please note that this changes the behavior encountered by unmodified Qemu: new interrupts are allocated instead of returning an error. Even more, since the range provided by Qemu spans 0 - 7, a failure during allocation of 5, 6, or 7 will result in release of entire range.

This series aims to maintain existing error behavior for MSI (please see "vfio/pci: Remove interrupt context counter") but I would appreciate your guidance on whether existing error behavior should be maintained for MSI-X and how to do so if it is a requirement.

Any feedback is appreciated

Reinette

[1] commit 34026364df8e ("PCI/MSI: Provide post-enable dynamic allocation interfaces for MSI-X")
[2] https://lore.kernel.org/kvm/MWHPR11MB188603D0D809C1079F5817DC8C099@MWHPR11MB1886.namprd11.prod.outlook.com/#t

Reinette Chatre (8):
vfio/pci: Consolidate irq cleanup on MSI/MSI-X disable vfio/pci: Remove negative check on unsigned vector vfio/pci: Prepare for dynamic interrupt context storage vfio/pci: Use xarray for interrupt context storage vfio/pci: Remove interrupt context counter vfio/pci: Move to single error path
vfio/pci: Support dynamic MSI-x
vfio/pci: Clear VFIO_IRQ_INFO_NORESIZE for MSI-X

 drivers/vfio/pci/vfio_pci_core.c  |   3 +-
 drivers/vfio/pci/vfio_pci_intrs.c | 376 ++++++++++++++++++++++--------
 include/linux/vfio_pci_core.h     |   3 +-
 include/uapi/linux/vfio.h         |   3 +
 4 files changed, 286 insertions(+), 99 deletions(-)

base-commit: eeac8ede17557680855031c6f305ece2378af326

Signature · 1 line
2.34.1
#1 lore raw

RFC PATCH 1/8 vfio/pci: Consolidate irq cleanup on MSI/MSI-X disable

vfio_msi_disable() releases all previously allocated state associated with each interrupt before disabling MSI/MSI-X.

vfio_msi_disable() iterates twice over the interrupt state:
first directly with a for loop to do virqfd cleanup, followed by another for loop within vfio_msi_set_block() that releases the interrupt and its state using vfio_msi_set_vector_signal().

Simplify interrupt cleanup by iterating over allocated interrupts once.

---
 drivers/vfio/pci/vfio_pci_intrs.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/vfio/pci/vfio_pci_intrs.c +1 -2
index bffb0741518b..6a9c6a143cc3 100644
@@ -426,10 +426,9 @@ static void vfio_msi_disable(struct vfio_pci_core_device *vdev, bool msix)
 	for (i = 0; i < vdev->num_ctx; i++) {
 		vfio_virqfd_disable(&vdev->ctx[i].unmask);
 		vfio_virqfd_disable(&vdev->ctx[i].mask);
+		vfio_msi_set_vector_signal(vdev, i, -1, msix);
 	}
 
-	vfio_msi_set_block(vdev, 0, vdev->num_ctx, NULL, msix);
-
 	cmd = vfio_pci_memory_lock_and_enable(vdev);
 	pci_free_irq_vectors(pdev);
 	vfio_pci_memory_unlock_and_restore(vdev, cmd);
Signature · 1 line
2.34.1
#2 lore raw

RFC PATCH 2/8 vfio/pci: Remove negative check on unsigned vector

In reply to Reinette Chatre

User space provides the vector as an unsigned int that is checked early for validity (vfio_set_irqs_validate_and_prepare()).

A later negative check of the provided vector is not necessary.

Remove the negative check and ensure the type used for the vector is consistent as an unsigned int.

---
 drivers/vfio/pci/vfio_pci_intrs.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
drivers/vfio/pci/vfio_pci_intrs.c +6 -5
index 6a9c6a143cc3..3f64ccdce69f 100644
@@ -317,14 +317,14 @@ static int vfio_msi_enable(struct vfio_pci_core_device *vdev, int nvec, bool msi
 }
 
 static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev,
-				      int vector, int fd, bool msix)
+				      unsigned int vector, int fd, bool msix)
 {
 	struct pci_dev *pdev = vdev->pdev;
 	struct eventfd_ctx *trigger;
 	int irq, ret;
 	u16 cmd;
 
-	if (vector < 0 || vector >= vdev->num_ctx)
+	if (vector >= vdev->num_ctx)
 		return -EINVAL;
 
 	irq = pci_irq_vector(pdev, vector);
@@ -399,7 +399,8 @@ static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev,
 static int vfio_msi_set_block(struct vfio_pci_core_device *vdev, unsigned start,
 			      unsigned count, int32_t *fds, bool msix)
 {
-	int i, j, ret = 0;
+	int i, ret = 0;
+	unsigned int j;
 
 	if (start >= vdev->num_ctx || start + count > vdev->num_ctx)
 		return -EINVAL;
@@ -420,7 +421,7 @@ static int vfio_msi_set_block(struct vfio_pci_core_device *vdev, unsigned start,
 static void vfio_msi_disable(struct vfio_pci_core_device *vdev, bool msix)
 {
 	struct pci_dev *pdev = vdev->pdev;
-	int i;
+	unsigned int i;
 	u16 cmd;
 
 	for (i = 0; i < vdev->num_ctx; i++) {
@@ -542,7 +543,7 @@ static int vfio_pci_set_msi_trigger(struct vfio_pci_core_device *vdev,
 				    unsigned index, unsigned start,
 				    unsigned count, uint32_t flags, void *data)
 {
-	int i;
+	unsigned int i;
 	bool msix = (index == VFIO_PCI_MSIX_IRQ_INDEX) ? true : false;
 
 	if (irq_is(vdev, index) && !count && (flags & VFIO_IRQ_SET_DATA_NONE)) {
Signature · 1 line
2.34.1
#3 lore raw

RFC PATCH 3/8 vfio/pci: Prepare for dynamic interrupt context storage

In reply to Reinette Chatre

Interrupt context storage is statically allocated at the time interrupts are enabled. Following allocation, the interrupt context is managed by directly accessing the elements of the array using the vector as index.

It is possible to allocate additional MSI-X vectors after MSI-X has been enabled. Dynamic storage of interrupt context is needed to support adding new MSI-X vectors after initial enabling.

Replace direct access of array elements with pointers to the array elements. Doing so reduces impact of moving to a new data structure. Move interactions with the array to helpers to mostly contain changes needed to transition to a dynamic data structure.

No functional change intended.

---
 drivers/vfio/pci/vfio_pci_intrs.c | 206 ++++++++++++++++++++----------
 1 file changed, 140 insertions(+), 66 deletions(-)
drivers/vfio/pci/vfio_pci_intrs.c +140 -66
index 3f64ccdce69f..ece371ebea00 100644
@@ -48,6 +48,31 @@ static bool is_irq_none(struct vfio_pci_core_device *vdev)
 		 vdev->irq_type == VFIO_PCI_MSIX_IRQ_INDEX);
 }
 
+static
+struct vfio_pci_irq_ctx *vfio_irq_ctx_get(struct vfio_pci_core_device *vdev,
+					  unsigned long index)
+{
+	if (index >= vdev->num_ctx)
+		return NULL;
+	return &vdev->ctx[index];
+}
+
+static void vfio_irq_ctx_free_all(struct vfio_pci_core_device *vdev)
+{
+	kfree(vdev->ctx);
+}
+
+static int vfio_irq_ctx_alloc_num(struct vfio_pci_core_device *vdev,
+				  unsigned long num)
+{
+	vdev->ctx = kcalloc(num, sizeof(struct vfio_pci_irq_ctx),
+			    GFP_KERNEL_ACCOUNT);
+	if (!vdev->ctx)
+		return -ENOMEM;
+
+	return 0;
+}
+
 /*
  * INTx
  */
@@ -55,17 +80,28 @@ static void vfio_send_intx_eventfd(void *opaque, void *unused)
 {
 	struct vfio_pci_core_device *vdev = opaque;
 
-	if (likely(is_intx(vdev) && !vdev->virq_disabled))
-		eventfd_signal(vdev->ctx[0].trigger, 1);
+	if (likely(is_intx(vdev) && !vdev->virq_disabled)) {
+		struct vfio_pci_irq_ctx *ctx;
+
+		ctx = vfio_irq_ctx_get(vdev, 0);
+		if (!ctx)
+			return;
+		eventfd_signal(ctx->trigger, 1);
+	}
 }
 
 /* Returns true if the INTx vfio_pci_irq_ctx.masked value is changed. */
 bool vfio_pci_intx_mask(struct vfio_pci_core_device *vdev)
 {
 	struct pci_dev *pdev = vdev->pdev;
+	struct vfio_pci_irq_ctx *ctx;
 	unsigned long flags;
 	bool masked_changed = false;
 
+	ctx = vfio_irq_ctx_get(vdev, 0);
+	if (!ctx)
+		return masked_changed;
+
 	spin_lock_irqsave(&vdev->irqlock, flags);
 
 	/*
@@ -77,7 +113,7 @@ bool vfio_pci_intx_mask(struct vfio_pci_core_device *vdev)
 	if (unlikely(!is_intx(vdev))) {
 		if (vdev->pci_2_3)
 			pci_intx(pdev, 0);
-	} else if (!vdev->ctx[0].masked) {
+	} else if (!ctx->masked) {
 		/*
 		 * Can't use check_and_mask here because we always want to
 		 * mask, not just when something is pending.
@@ -87,7 +123,7 @@ bool vfio_pci_intx_mask(struct vfio_pci_core_device *vdev)
 		else
 			disable_irq_nosync(pdev->irq);
 
-		vdev->ctx[0].masked = true;
+		ctx->masked = true;
 		masked_changed = true;
 	}
 
@@ -105,9 +141,14 @@ static int vfio_pci_intx_unmask_handler(void *opaque, void *unused)
 {
 	struct vfio_pci_core_device *vdev = opaque;
 	struct pci_dev *pdev = vdev->pdev;
+	struct vfio_pci_irq_ctx *ctx;
 	unsigned long flags;
 	int ret = 0;
 
+	ctx = vfio_irq_ctx_get(vdev, 0);
+	if (!ctx)
+		return ret;
+
 	spin_lock_irqsave(&vdev->irqlock, flags);
 
 	/*
@@ -117,7 +158,7 @@ static int vfio_pci_intx_unmask_handler(void *opaque, void *unused)
 	if (unlikely(!is_intx(vdev))) {
 		if (vdev->pci_2_3)
 			pci_intx(pdev, 1);
-	} else if (vdev->ctx[0].masked && !vdev->virq_disabled) {
+	} else if (ctx->masked && !vdev->virq_disabled) {
 		/*
 		 * A pending interrupt here would immediately trigger,
 		 * but we can avoid that overhead by just re-sending
@@ -129,7 +170,7 @@ static int vfio_pci_intx_unmask_handler(void *opaque, void *unused)
 		} else
 			enable_irq(pdev->irq);
 
-		vdev->ctx[0].masked = (ret > 0);
+		ctx->masked = (ret > 0);
 	}
 
 	spin_unlock_irqrestore(&vdev->irqlock, flags);
@@ -146,18 +187,23 @@ void vfio_pci_intx_unmask(struct vfio_pci_core_device *vdev)
 static irqreturn_t vfio_intx_handler(int irq, void *dev_id)
 {
 	struct vfio_pci_core_device *vdev = dev_id;
+	struct vfio_pci_irq_ctx *ctx;
 	unsigned long flags;
 	int ret = IRQ_NONE;
 
+	ctx = vfio_irq_ctx_get(vdev, 0);
+	if (!ctx)
+		return ret;
+
 	spin_lock_irqsave(&vdev->irqlock, flags);
 
 	if (!vdev->pci_2_3) {
 		disable_irq_nosync(vdev->pdev->irq);
-		vdev->ctx[0].masked = true;
+		ctx->masked = true;
 		ret = IRQ_HANDLED;
-	} else if (!vdev->ctx[0].masked &&  /* may be shared */
+	} else if (!ctx->masked &&  /* may be shared */
 		   pci_check_and_mask_intx(vdev->pdev)) {
-		vdev->ctx[0].masked = true;
+		ctx->masked = true;
 		ret = IRQ_HANDLED;
 	}
 
@@ -171,15 +217,24 @@ static irqreturn_t vfio_intx_handler(int irq, void *dev_id)
 
 static int vfio_intx_enable(struct vfio_pci_core_device *vdev)
 {
+	struct vfio_pci_irq_ctx *ctx;
+	int ret;
+
 	if (!is_irq_none(vdev))
 		return -EINVAL;
 
 	if (!vdev->pdev->irq)
 		return -ENODEV;
 
-	vdev->ctx = kzalloc(sizeof(struct vfio_pci_irq_ctx), GFP_KERNEL_ACCOUNT);
-	if (!vdev->ctx)
-		return -ENOMEM;
+	ret = vfio_irq_ctx_alloc_num(vdev, 1);
+	if (ret)
+		return ret;
+
+	ctx = vfio_irq_ctx_get(vdev, 0);
+	if (!ctx) {
+		vfio_irq_ctx_free_all(vdev);
+		return -EINVAL;
+	}
 
 	vdev->num_ctx = 1;
 
@@ -189,9 +244,9 @@ static int vfio_intx_enable(struct vfio_pci_core_device *vdev)
 	 * here, non-PCI-2.3 devices will have to wait until the
 	 * interrupt is enabled.
 	 */
-	vdev->ctx[0].masked = vdev->virq_disabled;
+	ctx->masked = vdev->virq_disabled;
 	if (vdev->pci_2_3)
-		pci_intx(vdev->pdev, !vdev->ctx[0].masked);
+		pci_intx(vdev->pdev, !ctx->masked);
 
 	vdev->irq_type = VFIO_PCI_INTX_IRQ_INDEX;
 
@@ -202,41 +257,46 @@ static int vfio_intx_set_signal(struct vfio_pci_core_device *vdev, int fd)
 {
 	struct pci_dev *pdev = vdev->pdev;
 	unsigned long irqflags = IRQF_SHARED;
+	struct vfio_pci_irq_ctx *ctx;
 	struct eventfd_ctx *trigger;
 	unsigned long flags;
 	int ret;
 
-	if (vdev->ctx[0].trigger) {
+	ctx = vfio_irq_ctx_get(vdev, 0);
+	if (!ctx)
+		return -EINVAL;
+
+	if (ctx->trigger) {
 		free_irq(pdev->irq, vdev);
-		kfree(vdev->ctx[0].name);
-		eventfd_ctx_put(vdev->ctx[0].trigger);
-		vdev->ctx[0].trigger = NULL;
+		kfree(ctx->name);
+		eventfd_ctx_put(ctx->trigger);
+		ctx->trigger = NULL;
 	}
 
 	if (fd < 0) /* Disable only */
 		return 0;
 
-	vdev->ctx[0].name = kasprintf(GFP_KERNEL_ACCOUNT, "vfio-intx(%s)",
-				      pci_name(pdev));
-	if (!vdev->ctx[0].name)
+	ctx->name = kasprintf(GFP_KERNEL_ACCOUNT, "vfio-intx(%s)",
+			      pci_name(pdev));
+	if (!ctx->name)
 		return -ENOMEM;
 
 	trigger = eventfd_ctx_fdget(fd);
 	if (IS_ERR(trigger)) {
-		kfree(vdev->ctx[0].name);
+		kfree(ctx->name);
 		return PTR_ERR(trigger);
 	}
 
-	vdev->ctx[0].trigger = trigger;
+	ctx->trigger = trigger;
 
 	if (!vdev->pci_2_3)
 		irqflags = 0;
 
 	ret = request_irq(pdev->irq, vfio_intx_handler,
-			  irqflags, vdev->ctx[0].name, vdev);
+			  irqflags, ctx->name, vdev);
 	if (ret) {
-		vdev->ctx[0].trigger = NULL;
-		kfree(vdev->ctx[0].name);
+		ctx->trigger = NULL;
+		kfree(ctx->name);
 		eventfd_ctx_put(trigger);
 		return ret;
 	}
@@ -246,7 +306,7 @@ static int vfio_intx_set_signal(struct vfio_pci_core_device *vdev, int fd)
 	 * disable_irq won't.
 	 */
 	spin_lock_irqsave(&vdev->irqlock, flags);
-	if (!vdev->pci_2_3 && vdev->ctx[0].masked)
+	if (!vdev->pci_2_3 && ctx->masked)
 		disable_irq_nosync(pdev->irq);
 	spin_unlock_irqrestore(&vdev->irqlock, flags);
 
@@ -255,12 +315,17 @@ static int vfio_intx_set_signal(struct vfio_pci_core_device *vdev, int fd)
 
 static void vfio_intx_disable(struct vfio_pci_core_device *vdev)
 {
-	vfio_virqfd_disable(&vdev->ctx[0].unmask);
-	vfio_virqfd_disable(&vdev->ctx[0].mask);
+	struct vfio_pci_irq_ctx *ctx;
+
+	ctx = vfio_irq_ctx_get(vdev, 0);
+	if (ctx) {
+		vfio_virqfd_disable(&ctx->unmask);
+		vfio_virqfd_disable(&ctx->mask);
+	}
 	vfio_intx_set_signal(vdev, -1);
 	vdev->irq_type = VFIO_PCI_NUM_IRQS;
 	vdev->num_ctx = 0;
-	kfree(vdev->ctx);
+	vfio_irq_ctx_free_all(vdev);
 }
 
 /*
@@ -284,10 +349,9 @@ static int vfio_msi_enable(struct vfio_pci_core_device *vdev, int nvec, bool msi
 	if (!is_irq_none(vdev))
 		return -EINVAL;
 
-	vdev->ctx = kcalloc(nvec, sizeof(struct vfio_pci_irq_ctx),
-			    GFP_KERNEL_ACCOUNT);
-	if (!vdev->ctx)
-		return -ENOMEM;
+	ret = vfio_irq_ctx_alloc_num(vdev, nvec);
+	if (ret)
+		return ret;
 
 	/* return the number of supported vectors if we can't get all: */
 	cmd = vfio_pci_memory_lock_and_enable(vdev);
@@ -296,7 +360,7 @@ static int vfio_msi_enable(struct vfio_pci_core_device *vdev, int nvec, bool msi
 		if (ret > 0)
 			pci_free_irq_vectors(pdev);
 		vfio_pci_memory_unlock_and_restore(vdev, cmd);
-		kfree(vdev->ctx);
+		vfio_irq_ctx_free_all(vdev);
 		return ret;
 	}
 	vfio_pci_memory_unlock_and_restore(vdev, cmd);
@@ -320,6 +384,7 @@ static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev,
 				      unsigned int vector, int fd, bool msix)
 {
 	struct pci_dev *pdev = vdev->pdev;
+	struct vfio_pci_irq_ctx *ctx;
 	struct eventfd_ctx *trigger;
 	int irq, ret;
 	u16 cmd;
@@ -327,33 +392,33 @@ static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev,
 	if (vector >= vdev->num_ctx)
 		return -EINVAL;
 
+	ctx = vfio_irq_ctx_get(vdev, vector);
+	if (!ctx)
+		return -EINVAL;
 	irq = pci_irq_vector(pdev, vector);
 
-	if (vdev->ctx[vector].trigger) {
-		irq_bypass_unregister_producer(&vdev->ctx[vector].producer);
+	if (ctx->trigger) {
+		irq_bypass_unregister_producer(&ctx->producer);
 
 		cmd = vfio_pci_memory_lock_and_enable(vdev);
-		free_irq(irq, vdev->ctx[vector].trigger);
+		free_irq(irq, ctx->trigger);
 		vfio_pci_memory_unlock_and_restore(vdev, cmd);
-
-		kfree(vdev->ctx[vector].name);
-		eventfd_ctx_put(vdev->ctx[vector].trigger);
-		vdev->ctx[vector].trigger = NULL;
+		kfree(ctx->name);
+		eventfd_ctx_put(ctx->trigger);
+		ctx->trigger = NULL;
 	}
 
 	if (fd < 0)
 		return 0;
 
-	vdev->ctx[vector].name = kasprintf(GFP_KERNEL_ACCOUNT,
-					   "vfio-msi%s[%d](%s)",
-					   msix ? "x" : "", vector,
-					   pci_name(pdev));
-	if (!vdev->ctx[vector].name)
+	ctx->name = kasprintf(GFP_KERNEL_ACCOUNT, "vfio-msi%s[%d](%s)",
+			      msix ? "x" : "", vector, pci_name(pdev));
+	if (!ctx->name)
 		return -ENOMEM;
 
 	trigger = eventfd_ctx_fdget(fd);
 	if (IS_ERR(trigger)) {
-		kfree(vdev->ctx[vector].name);
+		kfree(ctx->name);
 		return PTR_ERR(trigger);
 	}
 
@@ -372,26 +437,25 @@ static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev,
 		pci_write_msi_msg(irq, &msg);
 	}
 
-	ret = request_irq(irq, vfio_msihandler, 0,
-			  vdev->ctx[vector].name, trigger);
+	ret = request_irq(irq, vfio_msihandler, 0, ctx->name, trigger);
 	vfio_pci_memory_unlock_and_restore(vdev, cmd);
 	if (ret) {
-		kfree(vdev->ctx[vector].name);
+		kfree(ctx->name);
 		eventfd_ctx_put(trigger);
 		return ret;
 	}
 
-	vdev->ctx[vector].producer.token = trigger;
-	vdev->ctx[vector].producer.irq = irq;
-	ret = irq_bypass_register_producer(&vdev->ctx[vector].producer);
+	ctx->producer.token = trigger;
+	ctx->producer.irq = irq;
+	ret = irq_bypass_register_producer(&ctx->producer);
 	if (unlikely(ret)) {
 		dev_info(&pdev->dev,
 		"irq bypass producer (token %p) registration fails: %d\n",
-		vdev->ctx[vector].producer.token, ret);
+		ctx->producer.token, ret);
 
-		vdev->ctx[vector].producer.token = NULL;
+		ctx->producer.token = NULL;
 	}
-	vdev->ctx[vector].trigger = trigger;
+	ctx->trigger = trigger;
 
 	return 0;
 }
@@ -421,13 +485,17 @@ static int vfio_msi_set_block(struct vfio_pci_core_device *vdev, unsigned start,
 static void vfio_msi_disable(struct vfio_pci_core_device *vdev, bool msix)
 {
 	struct pci_dev *pdev = vdev->pdev;
+	struct vfio_pci_irq_ctx *ctx;
 	unsigned int i;
 	u16 cmd;
 
 	for (i = 0; i < vdev->num_ctx; i++) {
-		vfio_virqfd_disable(&vdev->ctx[i].unmask);
-		vfio_virqfd_disable(&vdev->ctx[i].mask);
-		vfio_msi_set_vector_signal(vdev, i, -1, msix);
+		ctx = vfio_irq_ctx_get(vdev, i);
+		if (ctx) {
+			vfio_virqfd_disable(&ctx->unmask);
+			vfio_virqfd_disable(&ctx->mask);
+			vfio_msi_set_vector_signal(vdev, i, -1, msix);
+		}
 	}
 
 	cmd = vfio_pci_memory_lock_and_enable(vdev);
@@ -443,7 +511,7 @@ static void vfio_msi_disable(struct vfio_pci_core_device *vdev, bool msix)
 
 	vdev->irq_type = VFIO_PCI_NUM_IRQS;
 	vdev->num_ctx = 0;
-	kfree(vdev->ctx);
+	vfio_irq_ctx_free_all(vdev);
 }
 
 /*
@@ -463,14 +531,18 @@ static int vfio_pci_set_intx_unmask(struct vfio_pci_core_device *vdev,
 		if (unmask)
 			vfio_pci_intx_unmask(vdev);
 	} else if (flags & VFIO_IRQ_SET_DATA_EVENTFD) {
+		struct vfio_pci_irq_ctx *ctx = vfio_irq_ctx_get(vdev, 0);
 		int32_t fd = *(int32_t *)data;
+
+		if (!ctx)
+			return -EINVAL;
 		if (fd >= 0)
 			return vfio_virqfd_enable((void *) vdev,
 						  vfio_pci_intx_unmask_handler,
 						  vfio_send_intx_eventfd, NULL,
-						  &vdev->ctx[0].unmask, fd);
+						  &ctx->unmask, fd);
 
-		vfio_virqfd_disable(&vdev->ctx[0].unmask);
+		vfio_virqfd_disable(&ctx->unmask);
 	}
 
 	return 0;
@@ -543,6 +615,7 @@ static int vfio_pci_set_msi_trigger(struct vfio_pci_core_device *vdev,
 				    unsigned index, unsigned start,
 				    unsigned count, uint32_t flags, void *data)
 {
+	struct vfio_pci_irq_ctx *ctx;
 	unsigned int i;
 	bool msix = (index == VFIO_PCI_MSIX_IRQ_INDEX) ? true : false;
 
@@ -577,14 +650,15 @@ static int vfio_pci_set_msi_trigger(struct vfio_pci_core_device *vdev,
 		return -EINVAL;
 
 	for (i = start; i < start + count; i++) {
-		if (!vdev->ctx[i].trigger)
+		ctx = vfio_irq_ctx_get(vdev, i);
+		if (!ctx || !ctx->trigger)
 			continue;
 		if (flags & VFIO_IRQ_SET_DATA_NONE) {
-			eventfd_signal(vdev->ctx[i].trigger, 1);
+			eventfd_signal(ctx->trigger, 1);
 		} else if (flags & VFIO_IRQ_SET_DATA_BOOL) {
 			uint8_t *bools = data;
 			if (bools[i - start])
-				eventfd_signal(vdev->ctx[i].trigger, 1);
+				eventfd_signal(ctx->trigger, 1);
 		}
 	}
 	return 0;
Signature · 1 line
2.34.1
#4 lore raw

RFC PATCH 4/8 vfio/pci: Use xarray for interrupt context storage

In reply to Reinette Chatre

Interrupt context is statically allocated at the time interrupts are enabled. Following allocation, the context is managed by directly accessing the elements of the array using the vector as index. The storage is released when interrupts are disabled.

It is possible to dynamically allocate a single MSI-X index after MSI-X has been enabled. A dynamic storage for interrupt context is needed to support this. Replace the interrupt context array with an xarray (similar to what the core uses as store for MSI descriptors) that can support the dynamic expansion while maintaining the custom that uses the vector as index.

Use the new data storage to allocate all elements once and free all elements together. Dynamic usage follows.

Create helpers with understanding that it is only possible to (after initial MSI-X enabling) allocate a single MSI-X index at a time. The only time multiple MSI-X are allocated is during initial MSI-X enabling where failure results in no allocations.

---
 drivers/vfio/pci/vfio_pci_core.c  |  1 +
 drivers/vfio/pci/vfio_pci_intrs.c | 63 +++++++++++++++++++++++--------
 include/linux/vfio_pci_core.h     |  2 +-
 3 files changed, 49 insertions(+), 17 deletions(-)
drivers/vfio/pci/vfio_pci_core.c +1 -0
index a5ab416cf476..ae0e161c7fc9 100644
@@ -2102,6 +2102,7 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev)
 	INIT_LIST_HEAD(&vdev->vma_list);
 	INIT_LIST_HEAD(&vdev->sriov_pfs_item);
 	init_rwsem(&vdev->memory_lock);
+	xa_init(&vdev->ctx);
 
 	return 0;
 }
drivers/vfio/pci/vfio_pci_intrs.c +47 -16

88 lines not on this page · raw message

include/linux/vfio_pci_core.h +1 -1
index 367fd79226a3..61d7873a3973 100644
@@ -59,7 +59,7 @@ struct vfio_pci_core_device {
 	struct perm_bits	*msi_perm;
 	spinlock_t		irqlock;
 	struct mutex		igate;
-	struct vfio_pci_irq_ctx	*ctx;
+	struct xarray		ctx;
 	int			num_ctx;
 	int			irq_type;
 	int			num_regions;
Signature · 1 line
2.34.1
#5 lore raw

RFC PATCH 5/8 vfio/pci: Remove interrupt context counter

In reply to Reinette Chatre

struct vfio_pci_core_device::num_ctx counts how many interrupt contexts have been allocated. When all interrupt contexts are allocated simultaneously num_ctx provides the upper bound of all vectors that can be used as indices into the interrupt context array.

With the upcoming support for dynamic MSI-X the number of interrupt contexts does not necessarily span the range of allocated interrupts. Consequently, num_ctx is no longer a trusted upper bound for valid indices.

Stop using num_ctx to determine if a provided vector is valid, use the existence of interrupt context directly.

Introduce a helper that ensures a provided interrupt range is allocated before any user requested action is taken. This maintains existing behavior (early exit without modifications) when user space attempts to modify a range of vectors that includes unallocated interrupts.

The checks that ensure that user space provides a range of vectors that is valid for the device are untouched.

---

Existing behavior on error paths is not maintained for MSI-X when adding support for dynamic MSI-X. Please see maintainer comments associated with "vfio/pci: Support dynamic MSI-x".

 drivers/vfio/pci/vfio_pci_intrs.c | 30 ++++++++++++++++++++----------
 include/linux/vfio_pci_core.h     |  1 -
 2 files changed, 20 insertions(+), 11 deletions(-)
drivers/vfio/pci/vfio_pci_intrs.c +20 -10
index bfcf5cb6b435..187a1ba34a16 100644
@@ -107,6 +107,21 @@ static int vfio_irq_ctx_alloc_num(struct vfio_pci_core_device *vdev,
 	return ret;
 }
 
+static bool vfio_irq_ctx_range_allocated(struct vfio_pci_core_device *vdev,
+					 unsigned int start, unsigned int count)
+{
+	struct vfio_pci_irq_ctx *ctx;
+	unsigned int i;
+
+	for (i = start; i < start + count; i++) {
+		ctx = xa_load(&vdev->ctx, i);
+		if (!ctx)
+			return false;
+	}
+
+	return true;
+}
+
 /*
  * INTx
  */
@@ -270,8 +285,6 @@ static int vfio_intx_enable(struct vfio_pci_core_device *vdev)
 		return -EINVAL;
 	}
 
-	vdev->num_ctx = 1;
-
 	/*
 	 * If the virtual interrupt is masked, restore it.  Devices
 	 * supporting DisINTx can be masked at the hardware level
@@ -358,7 +371,6 @@ static void vfio_intx_disable(struct vfio_pci_core_device *vdev)
 	}
 	vfio_intx_set_signal(vdev, -1);
 	vdev->irq_type = VFIO_PCI_NUM_IRQS;
-	vdev->num_ctx = 0;
 	vfio_irq_ctx_free_all(vdev);
 }
 
@@ -399,7 +411,6 @@ static int vfio_msi_enable(struct vfio_pci_core_device *vdev, int nvec, bool msi
 	}
 	vfio_pci_memory_unlock_and_restore(vdev, cmd);
 
-	vdev->num_ctx = nvec;
 	vdev->irq_type = msix ? VFIO_PCI_MSIX_IRQ_INDEX :
 				VFIO_PCI_MSI_IRQ_INDEX;
 
@@ -423,9 +434,6 @@ static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev,
 	int irq, ret;
 	u16 cmd;
 
-	if (vector >= vdev->num_ctx)
-		return -EINVAL;
-
 	ctx = vfio_irq_ctx_get(vdev, vector);
 	if (!ctx)
 		return -EINVAL;
@@ -500,7 +508,7 @@ static int vfio_msi_set_block(struct vfio_pci_core_device *vdev, unsigned start,
 	int i, ret = 0;
 	unsigned int j;
 
-	if (start >= vdev->num_ctx || start + count > vdev->num_ctx)
+	if (!vfio_irq_ctx_range_allocated(vdev, start, count))
 		return -EINVAL;
 
 	for (i = 0, j = start; i < count && !ret; i++, j++) {
@@ -541,7 +549,6 @@ static void vfio_msi_disable(struct vfio_pci_core_device *vdev, bool msix)
 		pci_intx(pdev, 0);
 
 	vdev->irq_type = VFIO_PCI_NUM_IRQS;
-	vdev->num_ctx = 0;
 	vfio_irq_ctx_free_all(vdev);
 }
 
@@ -677,7 +684,10 @@ static int vfio_pci_set_msi_trigger(struct vfio_pci_core_device *vdev,
 		return ret;
 	}
 
-	if (!irq_is(vdev, index) || start + count > vdev->num_ctx)
+	if (!irq_is(vdev, index))
+		return -EINVAL;
+
+	if (!vfio_irq_ctx_range_allocated(vdev, start, count))
 		return -EINVAL;
 
 	for (i = start; i < start + count; i++) {
include/linux/vfio_pci_core.h +0 -1
index 61d7873a3973..148fd1ae6c1c 100644
@@ -60,7 +60,6 @@ struct vfio_pci_core_device {
 	spinlock_t		irqlock;
 	struct mutex		igate;
 	struct xarray		ctx;
-	int			num_ctx;
 	int			irq_type;
 	int			num_regions;
 	struct vfio_pci_region	*region;
Signature · 1 line
2.34.1
#6 lore raw

RFC PATCH 6/8 vfio/pci: Move to single error path

In reply to Reinette Chatre

The creation and release of interrupt context involves several steps that can fail. Cleanup after failure is done when the error is encountered, resulting in some repetitive code.

Support for dynamic MSI-X will introduce more steps during interrupt context creation and release.

Transition to centralized exit path in preparation for dynamic MSI-X to eliminate duplicate error handling code. Ensure no remaining state refers to freed memory.

---
 drivers/vfio/pci/vfio_pci_intrs.c | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)
drivers/vfio/pci/vfio_pci_intrs.c +11 -7
index 187a1ba34a16..b375a12885ba 100644
@@ -460,8 +460,8 @@ static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev,
 
 	trigger = eventfd_ctx_fdget(fd);
 	if (IS_ERR(trigger)) {
-		kfree(ctx->name);
-		return PTR_ERR(trigger);
+		ret = PTR_ERR(trigger);
+		goto out_free_name;
 	}
 
 	/*
@@ -481,11 +481,8 @@ static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev,
 
 	ret = request_irq(irq, vfio_msihandler, 0, ctx->name, trigger);
 	vfio_pci_memory_unlock_and_restore(vdev, cmd);
-	if (ret) {
-		kfree(ctx->name);
-		eventfd_ctx_put(trigger);
-		return ret;
-	}
+	if (ret)
+		goto out_put_eventfd_ctx;
 
 	ctx->producer.token = trigger;
 	ctx->producer.irq = irq;
@@ -500,6 +497,13 @@ static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev,
 	ctx->trigger = trigger;
 
 	return 0;
+
+out_put_eventfd_ctx:
+	eventfd_ctx_put(trigger);
+out_free_name:
+	kfree(ctx->name);
+	ctx->name = NULL;
+	return ret;
 }
 
 static int vfio_msi_set_block(struct vfio_pci_core_device *vdev, unsigned start,
Signature · 1 line
2.34.1
#7 lore raw

RFC PATCH 7/8 vfio/pci: Support dynamic MSI-x

In reply to Reinette Chatre

Recently introduced pci_msix_alloc_irq_at() and pci_msix_free_irq() enables an individual MSI-X index to be allocated and freed after MSI-X enabling.

Support dynamic MSI-X by keeping the association between allocated interrupt and vfio interrupt context. Allocate new context together with the new interrupt if no interrupt context exist for an MSI-X interrupt. Similarly, release an interrupt with its context.

---

Guidance is appreciated on expectations regarding maintaining existing error behavior. Earlier patch introduced the vfio_irq_ctx_range_allocated() helper to maintain existing error behavior. Now, this helper needs to be disabled for MSI-X. User space not wanting to dynamically allocate MSI-X interrupts, but providing invalid range when providing a new ACTION will now obtain new interrupts or new failures (potentially including freeing of existing interrupts) if the allocation of the new interrupts fail.

 drivers/vfio/pci/vfio_pci_intrs.c | 101 ++++++++++++++++++++++++------
 1 file changed, 83 insertions(+), 18 deletions(-)
drivers/vfio/pci/vfio_pci_intrs.c +83 -18
index b375a12885ba..954a70575802 100644
@@ -55,6 +55,18 @@ struct vfio_pci_irq_ctx *vfio_irq_ctx_get(struct vfio_pci_core_device *vdev,
 	return xa_load(&vdev->ctx, index);
 }
 
+static void vfio_irq_ctx_free(struct vfio_pci_core_device *vdev,
+			      unsigned long index)
+{
+	struct vfio_pci_irq_ctx *ctx;
+
+	ctx = xa_load(&vdev->ctx, index);
+	if (ctx) {
+		xa_erase(&vdev->ctx, index);
+		kfree(ctx);
+	}
+}
+
 static void vfio_irq_ctx_free_all(struct vfio_pci_core_device *vdev)
 {
 	struct vfio_pci_irq_ctx *ctx;
@@ -430,33 +442,63 @@ static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev,
 {
 	struct pci_dev *pdev = vdev->pdev;
 	struct vfio_pci_irq_ctx *ctx;
+	struct msi_map msix_map = {};
 	struct eventfd_ctx *trigger;
+	bool new_ctx;
 	int irq, ret;
 	u16 cmd;
 
 	ctx = vfio_irq_ctx_get(vdev, vector);
-	if (!ctx)
+	/* Only MSI-X allows dynamic allocation. */
+	if (!msix && !ctx)
 		return -EINVAL;
+
 	irq = pci_irq_vector(pdev, vector);
+	/* Context and interrupt are always allocated together. */
+	WARN_ON((ctx && irq == -EINVAL) || (!ctx && irq != -EINVAL));
 
-	if (ctx->trigger) {
+	if (ctx && ctx->trigger) {
 		irq_bypass_unregister_producer(&ctx->producer);
 
 		cmd = vfio_pci_memory_lock_and_enable(vdev);
 		free_irq(irq, ctx->trigger);
+		if (msix) {
+			msix_map.index = vector;
+			msix_map.virq = irq;
+			pci_msix_free_irq(pdev, msix_map);
+			irq = -EINVAL;
+		}
 		vfio_pci_memory_unlock_and_restore(vdev, cmd);
 		kfree(ctx->name);
 		eventfd_ctx_put(ctx->trigger);
 		ctx->trigger = NULL;
+		if (msix) {
+			vfio_irq_ctx_free(vdev, vector);
+			ctx = NULL;
+		}
 	}
 
 	if (fd < 0)
 		return 0;
 
+	if (!ctx) {
+		ret = vfio_irq_ctx_alloc_single(vdev, vector);
+		if (ret)
+			return ret;
+		ctx = vfio_irq_ctx_get(vdev, vector);
+		if (!ctx) {
+			ret = -EINVAL;
+			goto out_free_ctx;
+		}
+		new_ctx = true;
+	}
+
 	ctx->name = kasprintf(GFP_KERNEL_ACCOUNT, "vfio-msi%s[%d](%s)",
 			      msix ? "x" : "", vector, pci_name(pdev));
-	if (!ctx->name)
-		return -ENOMEM;
+	if (!ctx->name) {
+		ret = -ENOMEM;
+		goto out_free_ctx;
+	}
 
 	trigger = eventfd_ctx_fdget(fd);
 	if (IS_ERR(trigger)) {
@@ -464,25 +506,38 @@ static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev,
 		goto out_free_name;
 	}
 
-	/*
-	 * The MSIx vector table resides in device memory which may be cleared
-	 * via backdoor resets. We don't allow direct access to the vector
-	 * table so even if a userspace driver attempts to save/restore around
-	 * such a reset it would be unsuccessful. To avoid this, restore the
-	 * cached value of the message prior to enabling.
-	 */
 	cmd = vfio_pci_memory_lock_and_enable(vdev);
 	if (msix) {
-		struct msi_msg msg;
-
-		get_cached_msi_msg(irq, &msg);
-		pci_write_msi_msg(irq, &msg);
+		if (irq == -EINVAL) {
+			msix_map = pci_msix_alloc_irq_at(pdev, vector, NULL);
+			if (msix_map.index < 0) {
+				vfio_pci_memory_unlock_and_restore(vdev, cmd);
+				ret = msix_map.index;
+				goto out_put_eventfd_ctx;
+			}
+			irq = msix_map.virq;
+		} else {
+			/*
+			 * The MSIx vector table resides in device memory which
+			 * may be cleared via backdoor resets. We don't allow
+			 * direct access to the vector table so even if a
+			 * userspace driver attempts to save/restore around
+			 * such a reset it would be unsuccessful. To avoid
+			 * this, restore the cached value of the message prior
+			 * to enabling.
+			 */
+			struct msi_msg msg;
+
+			get_cached_msi_msg(irq, &msg);
+			pci_write_msi_msg(irq, &msg);
+		}
 	}
 
 	ret = request_irq(irq, vfio_msihandler, 0, ctx->name, trigger);
-	vfio_pci_memory_unlock_and_restore(vdev, cmd);
 	if (ret)
-		goto out_put_eventfd_ctx;
+		goto out_free_irq_locked;
+
+	vfio_pci_memory_unlock_and_restore(vdev, cmd);
 
 	ctx->producer.token = trigger;
 	ctx->producer.irq = irq;
@@ -498,11 +553,21 @@ static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev,
 
 	return 0;
 
+out_free_irq_locked:
+	if (msix && new_ctx) {
+		msix_map.index = vector;
+		msix_map.virq = irq;
+		pci_msix_free_irq(pdev, msix_map);
+	}
+	vfio_pci_memory_unlock_and_restore(vdev, cmd);
 out_put_eventfd_ctx:
 	eventfd_ctx_put(trigger);
 out_free_name:
 	kfree(ctx->name);
 	ctx->name = NULL;
+out_free_ctx:
+	if (msix && new_ctx)
+		vfio_irq_ctx_free(vdev, vector);
 	return ret;
 }
 
@@ -512,7 +577,7 @@ static int vfio_msi_set_block(struct vfio_pci_core_device *vdev, unsigned start,
 	int i, ret = 0;
 	unsigned int j;
 
-	if (!vfio_irq_ctx_range_allocated(vdev, start, count))
+	if (!msix && !vfio_irq_ctx_range_allocated(vdev, start, count))
 		return -EINVAL;
 
 	for (i = 0, j = start; i < count && !ret; i++, j++) {
Signature · 1 line
2.34.1
#8 lore raw

Re: [RFC PATCH 7/8] vfio/pci: Support dynamic MSI-x

Quoted 33 lines · On Wed, 15 Mar 2023 13:59:27 -0700 Reinette Chatre <[email protected]>…

Recently introduced pci_msix_alloc_irq_at() and pci_msix_free_irq() enables an individual MSI-X index to be allocated and freed after MSI-X enabling.

Support dynamic MSI-X by keeping the association between allocated interrupt and vfio interrupt context. Allocate new context together with the new interrupt if no interrupt context exist for an MSI-X interrupt. Similarly, release an interrupt with its context.

---

Guidance is appreciated on expectations regarding maintaining existing error behavior. Earlier patch introduced the vfio_irq_ctx_range_allocated() helper to maintain existing error behavior. Now, this helper needs to be disabled for MSI-X. User space not wanting to dynamically allocate MSI-X interrupts, but providing invalid range when providing a new ACTION will now obtain new interrupts or new failures (potentially including freeing of existing interrupts) if the allocation of the new interrupts fail.

 drivers/vfio/pci/vfio_pci_intrs.c | 101 ++++++++++++++++++++++++------
 1 file changed, 83 insertions(+), 18 deletions(-)
drivers/vfio/pci/vfio_pci_intrs.c +1 -0
index b375a12885ba..954a70575802 100644
@@ -55,6 +55,18 @@ struct vfio_pci_irq_ctx *vfio_irq_ctx_get(struct vfio_pci_core_device *vdev,
 	return xa_load(&vdev->ctx, index);
 }
 
+static void vfio_irq_ctx_free(struct vfio_pci_core_device *vdev,

drivers/vfio/pci/vfio_pci_intrs.c struct vfio_pci_irq_ctx *vfio_irq_ctx_get(struct vfio_pci_core_device *vdev, · line 68

Quoted 10 lines
drivers/vfio/pci/vfio_pci_intrs.c +10 -0
+			      unsigned long index)
+{
+	struct vfio_pci_irq_ctx *ctx;
+
+	ctx = xa_load(&vdev->ctx, index);
+	if (ctx) {
+		xa_erase(&vdev->ctx, index);
+		kfree(ctx);
+	}
+}

The only places calling this have a known valid ctx, so it seems redundant that we xa_load it again. Should ctx be a function arg to reduce this to simply xa_erase() + kfree()?

Quoted 44 lines
drivers/vfio/pci/vfio_pci_intrs.c +19 -2
+
 static void vfio_irq_ctx_free_all(struct vfio_pci_core_device *vdev)
 {
 	struct vfio_pci_irq_ctx *ctx;
@@ -430,33 +442,63 @@ static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev,
 {
 	struct pci_dev *pdev = vdev->pdev;
 	struct vfio_pci_irq_ctx *ctx;
+	struct msi_map msix_map = {};
 	struct eventfd_ctx *trigger;
+	bool new_ctx;
 	int irq, ret;
 	u16 cmd;
 
 	ctx = vfio_irq_ctx_get(vdev, vector);
-	if (!ctx)
+	/* Only MSI-X allows dynamic allocation. */
+	if (!msix && !ctx)
 		return -EINVAL;
+
 	irq = pci_irq_vector(pdev, vector);
+	/* Context and interrupt are always allocated together. */
+	WARN_ON((ctx && irq == -EINVAL) || (!ctx && irq != -EINVAL));
 
-	if (ctx->trigger) {
+	if (ctx && ctx->trigger) {
 		irq_bypass_unregister_producer(&ctx->producer);
 
 		cmd = vfio_pci_memory_lock_and_enable(vdev);
 		free_irq(irq, ctx->trigger);
+		if (msix) {
+			msix_map.index = vector;
+			msix_map.virq = irq;
+			pci_msix_free_irq(pdev, msix_map);
+			irq = -EINVAL;
+		}
 		vfio_pci_memory_unlock_and_restore(vdev, cmd);
 		kfree(ctx->name);
 		eventfd_ctx_put(ctx->trigger);
 		ctx->trigger = NULL;
+		if (msix) {
+			vfio_irq_ctx_free(vdev, vector);
+			ctx = NULL;
+		}

drivers/vfio/pci/vfio_pci_intrs.c static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev, · line 488

Quoted 10 lines
drivers/vfio/pci/vfio_pci_intrs.c +5 -0
 	}
 
 	if (fd < 0)
 		return 0;
 
+	if (!ctx) {
+		ret = vfio_irq_ctx_alloc_single(vdev, vector);
+		if (ret)
+			return ret;
+		ctx = vfio_irq_ctx_get(vdev, vector);

This suggests vfio_irq_ctx_alloc_single() should return ctx.

Quoted 27 lines
drivers/vfio/pci/vfio_pci_intrs.c +11 -7
+		if (!ctx) {
+			ret = -EINVAL;
+			goto out_free_ctx;
+		}
+		new_ctx = true;
+	}
+
 	ctx->name = kasprintf(GFP_KERNEL_ACCOUNT, "vfio-msi%s[%d](%s)",
 			      msix ? "x" : "", vector, pci_name(pdev));
-	if (!ctx->name)
-		return -ENOMEM;
+	if (!ctx->name) {
+		ret = -ENOMEM;
+		goto out_free_ctx;
+	}
 
 	trigger = eventfd_ctx_fdget(fd);
 	if (IS_ERR(trigger)) {
@@ -464,25 +506,38 @@ static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev,
 		goto out_free_name;
 	}
 
-	/*
-	 * The MSIx vector table resides in device memory which may be cleared
-	 * via backdoor resets. We don't allow direct access to the vector
-	 * table so even if a userspace driver attempts to save/restore around
-	 * such a reset it would be unsuccessful. To avoid this, restore the

drivers/vfio/pci/vfio_pci_intrs.c static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev, · line 512

Quoted 10 lines
drivers/vfio/pci/vfio_pci_intrs.c +2 -6
-	 * cached value of the message prior to enabling.
-	 */
 	cmd = vfio_pci_memory_lock_and_enable(vdev);
 	if (msix) {
-		struct msi_msg msg;
-
-		get_cached_msi_msg(irq, &msg);
-		pci_write_msi_msg(irq, &msg);
+		if (irq == -EINVAL) {
+			msix_map = pci_msix_alloc_irq_at(pdev, vector, NULL);

It looks to me like we need to support MSI-X with both NORESIZE behavior and dynamic allocation based on pci_msix_can_alloc_dyn(). It's not entirely clear to me where this is and isn't supported, but the existence of the test helper suggests we can't assume support.

Quoted 11 lines
drivers/vfio/pci/vfio_pci_intrs.c +11 -0
+			if (msix_map.index < 0) {
+				vfio_pci_memory_unlock_and_restore(vdev, cmd);
+				ret = msix_map.index;
+				goto out_put_eventfd_ctx;
+			}
+			irq = msix_map.virq;
+		} else {
+			/*
+			 * The MSIx vector table resides in device memory which
+			 * may be cleared via backdoor resets. We don't allow
+			 * direct access to the vector table so even if a

drivers/vfio/pci/vfio_pci_intrs.c static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev, · line 533

Quoted 10 lines
drivers/vfio/pci/vfio_pci_intrs.c +10 -0
+			 * userspace driver attempts to save/restore around
+			 * such a reset it would be unsuccessful. To avoid
+			 * this, restore the cached value of the message prior
+			 * to enabling.
+			 */
+			struct msi_msg msg;
+
+			get_cached_msi_msg(irq, &msg);
+			pci_write_msi_msg(irq, &msg);
+		}

I don't follow when this latter branch is ever taken in the new flow. It's stated earlier that ctx and irq are coupled, and I believe so is trigger. So if we had a previous ctx and irq (and trigger), we removed it and irq is now always -EINVAL here. Thanks,

Alex

Quoted 43 lines
drivers/vfio/pci/vfio_pci_intrs.c +14 -3
 	}
 
 	ret = request_irq(irq, vfio_msihandler, 0, ctx->name, trigger);
-	vfio_pci_memory_unlock_and_restore(vdev, cmd);
 	if (ret)
-		goto out_put_eventfd_ctx;
+		goto out_free_irq_locked;
+
+	vfio_pci_memory_unlock_and_restore(vdev, cmd);
 
 	ctx->producer.token = trigger;
 	ctx->producer.irq = irq;
@@ -498,11 +553,21 @@ static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev,
 
 	return 0;
 
+out_free_irq_locked:
+	if (msix && new_ctx) {
+		msix_map.index = vector;
+		msix_map.virq = irq;
+		pci_msix_free_irq(pdev, msix_map);
+	}
+	vfio_pci_memory_unlock_and_restore(vdev, cmd);
 out_put_eventfd_ctx:
 	eventfd_ctx_put(trigger);
 out_free_name:
 	kfree(ctx->name);
 	ctx->name = NULL;
+out_free_ctx:
+	if (msix && new_ctx)
+		vfio_irq_ctx_free(vdev, vector);
 	return ret;
 }
 
@@ -512,7 +577,7 @@ static int vfio_msi_set_block(struct vfio_pci_core_device *vdev, unsigned start,
 	int i, ret = 0;
 	unsigned int j;
 
-	if (!vfio_irq_ctx_range_allocated(vdev, start, count))
+	if (!msix && !vfio_irq_ctx_range_allocated(vdev, start, count))
 		return -EINVAL;
 
 	for (i = 0, j = start; i < count && !ret; i++, j++) {
#9 lore raw

Re: [RFC PATCH 7/8] vfio/pci: Support dynamic MSI-x

Hi Alex,

Quoted 3 lines · Alex Williamson

On Wed, 15 Mar 2023 13:59:27 -0700 Reinette Chatre <[email protected]> wrote:

...

Quoted 7 lines
Quoted 7 lines
drivers/vfio/pci/vfio_pci_intrs.c +7 -0
+static void vfio_irq_ctx_free(struct vfio_pci_core_device *vdev,
+			      unsigned long index)
+{
+	struct vfio_pci_irq_ctx *ctx;
+
+	ctx = xa_load(&vdev->ctx, index);
+	if (ctx) {
Quoted 8 lines

drivers/vfio/pci/vfio_pci_intrs.c struct vfio_pci_irq_ctx *vfio_irq_ctx_get(struct vfio_pci_core_device *vdev, · line 68

Quoted 4 lines
drivers/vfio/pci/vfio_pci_intrs.c +4 -0
+		xa_erase(&vdev->ctx, index);
+		kfree(ctx);
+	}
+}

The only places calling this have a known valid ctx, so it seems redundant that we xa_load it again. Should ctx be a function arg to reduce this to simply xa_erase() + kfree()?

Good point. Will do.

...

Quoted 8 lines

drivers/vfio/pci/vfio_pci_intrs.c static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev, · line 488

Quoted 5 lines
drivers/vfio/pci/vfio_pci_intrs.c +5 -0
+	if (!ctx) {
+		ret = vfio_irq_ctx_alloc_single(vdev, vector);
+		if (ret)
+			return ret;
+		ctx = vfio_irq_ctx_get(vdev, vector);

This suggests vfio_irq_ctx_alloc_single() should return ctx.

Thank you. Yes, will do.

Quoted 16 lines
Quoted 16 lines
drivers/vfio/pci/vfio_pci_intrs.c +0 -10
@@ -464,25 +506,38 @@ static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev,
 		goto out_free_name;
 	}
 
-	/*
-	 * The MSIx vector table resides in device memory which may be cleared
-	 * via backdoor resets. We don't allow direct access to the vector
-	 * table so even if a userspace driver attempts to save/restore around
-	 * such a reset it would be unsuccessful. To avoid this, restore the
-	 * cached value of the message prior to enabling.
-	 */
 	cmd = vfio_pci_memory_lock_and_enable(vdev);
 	if (msix) {
-		struct msi_msg msg;
-
-		get_cached_msi_msg(irq, &msg);
Quoted 8 lines

drivers/vfio/pci/vfio_pci_intrs.c static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev, · line 512

Quoted 3 lines
drivers/vfio/pci/vfio_pci_intrs.c +2 -1
-		pci_write_msi_msg(irq, &msg);
+		if (irq == -EINVAL) {
+			msix_map = pci_msix_alloc_irq_at(pdev, vector, NULL);

It looks to me like we need to support MSI-X with both NORESIZE behavior and dynamic allocation based on pci_msix_can_alloc_dyn(). It's not entirely clear to me where this is and isn't supported, but the existence of the test helper suggests we can't assume support.

As I mentioned in my other response ([1]) I cannot see how pci_msix_can_alloc_dyn() can return false. Even so, yes, I can rework this series to support both the original and dynamic MSI-x allocation mechanisms.

Quoted 18 lines
Quoted 18 lines
drivers/vfio/pci/vfio_pci_intrs.c +18 -0
+			if (msix_map.index < 0) {
+				vfio_pci_memory_unlock_and_restore(vdev, cmd);
+				ret = msix_map.index;
+				goto out_put_eventfd_ctx;
+			}
+			irq = msix_map.virq;
+		} else {
+			/*
+			 * The MSIx vector table resides in device memory which
+			 * may be cleared via backdoor resets. We don't allow
+			 * direct access to the vector table so even if a
+			 * userspace driver attempts to save/restore around
+			 * such a reset it would be unsuccessful. To avoid
+			 * this, restore the cached value of the message prior
+			 * to enabling.
+			 */
+			struct msi_msg msg;
+
Quoted 8 lines

drivers/vfio/pci/vfio_pci_intrs.c static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev, · line 533

Quoted 3 lines
drivers/vfio/pci/vfio_pci_intrs.c +3 -0
+			get_cached_msi_msg(irq, &msg);
+			pci_write_msi_msg(irq, &msg);
+		}

I don't follow when this latter branch is ever taken in the new flow. It's stated earlier that ctx and irq are coupled, and I believe so is trigger. So if we had a previous ctx and irq (and trigger), we removed it and irq is now always -EINVAL here. Thanks,

From what I understand MSI-X can be enabled without providing any triggers. That will result in the ctx and irq existing, but not trigger. When a trigger is assigned later, it will run the latter branch.

Thank you very much

Reinette

[1] https://lore.kernel.org/lkml/[email protected]/

#10 lore raw

Re: [RFC PATCH 7/8] vfio/pci: Support dynamic MSI-x

In reply to Reinette Chatre

Hi Reinette,

url:    https://github.com/intel-lab-lkp/linux/commits/Reinette-Chatre/vfio-pci-Consolidate-irq-cleanup-on-MSI-MSI-X-disable/20230316-050150
base:   eeac8ede17557680855031c6f305ece2378af326
patch link:    https://lore.kernel.org/r/591ce11f4a33e022fc9242324ebdc077202bedaf.1678911529.git.reinette.chatre%40intel.com
patch subject: [RFC PATCH 7/8] vfio/pci: Support dynamic MSI-x
config: ia64-randconfig-m031-20230312 (https://download.01.org/0day-ci/archive/20230316/[email protected]/config)
compiler: ia64-linux-gcc (GCC) 12.1.0

If you fix the issue, kindly add following tag where applicable

| Reported-by: kernel test robot <[email protected]>
| Reported-by: Dan Carpenter <[email protected]>
| Link: https://lore.kernel.org/r/[email protected]/

New smatch warnings:
drivers/vfio/pci/vfio_pci_intrs.c:557 vfio_msi_set_vector_signal() error: uninitialized symbol 'new_ctx'.

Old smatch warnings:
drivers/vfio/pci/vfio_pci_intrs.c:569 vfio_msi_set_vector_signal() error: uninitialized symbol 'new_ctx'.

vim +/new_ctx +557 drivers/vfio/pci/vfio_pci_intrs.c

536475109c8284 Max Gurtovoy      2021-08-26  440  static int vfio_msi_set_vector_signal(struct vfio_pci_core_device *vdev,
45d99835bd7f50 Reinette Chatre   2023-03-15  441  				      unsigned int vector, int fd, bool msix)
89e1f7d4c66d85 Alex Williamson   2012-07-31  442  {
89e1f7d4c66d85 Alex Williamson   2012-07-31  443  	struct pci_dev *pdev = vdev->pdev;
d6ddf3a1f66d02 Reinette Chatre   2023-03-15  444  	struct vfio_pci_irq_ctx *ctx;
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  445  	struct msi_map msix_map = {};
89e1f7d4c66d85 Alex Williamson   2012-07-31  446  	struct eventfd_ctx *trigger;
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  447  	bool new_ctx;

Was never intiliazed to false.

b95d9305e8cb8d Alex Williamson   2016-02-22  448  	int irq, ret;
abafbc551fdded Alex Williamson   2020-04-22  449  	u16 cmd;
89e1f7d4c66d85 Alex Williamson   2012-07-31  450  
d6ddf3a1f66d02 Reinette Chatre   2023-03-15  451  	ctx = vfio_irq_ctx_get(vdev, vector);
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  452  	/* Only MSI-X allows dynamic allocation. */
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  453  	if (!msix && !ctx)
d6ddf3a1f66d02 Reinette Chatre   2023-03-15  454  		return -EINVAL;
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  455  
61771468e0a567 Christoph Hellwig 2016-09-11  456  	irq = pci_irq_vector(pdev, vector);
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  457  	/* Context and interrupt are always allocated together. */
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  458  	WARN_ON((ctx && irq == -EINVAL) || (!ctx && irq != -EINVAL));
b95d9305e8cb8d Alex Williamson   2016-02-22  459  
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  460  	if (ctx && ctx->trigger) {
d6ddf3a1f66d02 Reinette Chatre   2023-03-15  461  		irq_bypass_unregister_producer(&ctx->producer);
abafbc551fdded Alex Williamson   2020-04-22  462  
abafbc551fdded Alex Williamson   2020-04-22  463  		cmd = vfio_pci_memory_lock_and_enable(vdev);
d6ddf3a1f66d02 Reinette Chatre   2023-03-15  464  		free_irq(irq, ctx->trigger);
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  465  		if (msix) {
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  466  			msix_map.index = vector;
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  467  			msix_map.virq = irq;
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  468  			pci_msix_free_irq(pdev, msix_map);
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  469  			irq = -EINVAL;
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  470  		}
abafbc551fdded Alex Williamson   2020-04-22  471  		vfio_pci_memory_unlock_and_restore(vdev, cmd);
d6ddf3a1f66d02 Reinette Chatre   2023-03-15  472  		kfree(ctx->name);
d6ddf3a1f66d02 Reinette Chatre   2023-03-15  473  		eventfd_ctx_put(ctx->trigger);
d6ddf3a1f66d02 Reinette Chatre   2023-03-15  474  		ctx->trigger = NULL;
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  475  		if (msix) {
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  476  			vfio_irq_ctx_free(vdev, vector);
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  477  			ctx = NULL;
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  478  		}
89e1f7d4c66d85 Alex Williamson   2012-07-31  479  	}
89e1f7d4c66d85 Alex Williamson   2012-07-31  480  
89e1f7d4c66d85 Alex Williamson   2012-07-31  481  	if (fd < 0)
89e1f7d4c66d85 Alex Williamson   2012-07-31  482  		return 0;
89e1f7d4c66d85 Alex Williamson   2012-07-31  483  
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  484  	if (!ctx) {
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  485  		ret = vfio_irq_ctx_alloc_single(vdev, vector);
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  486  		if (ret)
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  487  			return ret;
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  488  		ctx = vfio_irq_ctx_get(vdev, vector);
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  489  		if (!ctx) {
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  490  			ret = -EINVAL;
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  491  			goto out_free_ctx;
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  492  		}
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  493  		new_ctx = true;
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  494  	}
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  495  
d6ddf3a1f66d02 Reinette Chatre   2023-03-15  496  	ctx->name = kasprintf(GFP_KERNEL_ACCOUNT, "vfio-msi%s[%d](%s)",
d6ddf3a1f66d02 Reinette Chatre   2023-03-15  497  			      msix ? "x" : "", vector, pci_name(pdev));
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  498  	if (!ctx->name) {
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  499  		ret = -ENOMEM;
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  500  		goto out_free_ctx;
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  501  	}
89e1f7d4c66d85 Alex Williamson   2012-07-31  502  
89e1f7d4c66d85 Alex Williamson   2012-07-31  503  	trigger = eventfd_ctx_fdget(fd);
89e1f7d4c66d85 Alex Williamson   2012-07-31  504  	if (IS_ERR(trigger)) {
224474be3986e1 Reinette Chatre   2023-03-15  505  		ret = PTR_ERR(trigger);
224474be3986e1 Reinette Chatre   2023-03-15  506  		goto out_free_name;
89e1f7d4c66d85 Alex Williamson   2012-07-31  507  	}
89e1f7d4c66d85 Alex Williamson   2012-07-31  508  
abafbc551fdded Alex Williamson   2020-04-22  509  	cmd = vfio_pci_memory_lock_and_enable(vdev);
b8f02af096b1fc Gavin Shan        2014-09-29  510  	if (msix) {
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  511  		if (irq == -EINVAL) {
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  512  			msix_map = pci_msix_alloc_irq_at(pdev, vector, NULL);
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  513  			if (msix_map.index < 0) {
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  514  				vfio_pci_memory_unlock_and_restore(vdev, cmd);
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  515  				ret = msix_map.index;
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  516  				goto out_put_eventfd_ctx;
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  517  			}
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  518  			irq = msix_map.virq;
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  519  		} else {
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  520  			/*
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  521  			 * The MSIx vector table resides in device memory which
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  522  			 * may be cleared via backdoor resets. We don't allow
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  523  			 * direct access to the vector table so even if a
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  524  			 * userspace driver attempts to save/restore around
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  525  			 * such a reset it would be unsuccessful. To avoid
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  526  			 * this, restore the cached value of the message prior
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  527  			 * to enabling.
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  528  			 */
b8f02af096b1fc Gavin Shan        2014-09-29  529  			struct msi_msg msg;
b8f02af096b1fc Gavin Shan        2014-09-29  530  
b8f02af096b1fc Gavin Shan        2014-09-29  531  			get_cached_msi_msg(irq, &msg);
83a18912b0e8d2 Jiang Liu         2014-11-09  532  			pci_write_msi_msg(irq, &msg);
b8f02af096b1fc Gavin Shan        2014-09-29  533  		}
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  534  	}
b8f02af096b1fc Gavin Shan        2014-09-29  535  
d6ddf3a1f66d02 Reinette Chatre   2023-03-15  536  	ret = request_irq(irq, vfio_msihandler, 0, ctx->name, trigger);
224474be3986e1 Reinette Chatre   2023-03-15  537  	if (ret)
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  538  		goto out_free_irq_locked;
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  539  
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  540  	vfio_pci_memory_unlock_and_restore(vdev, cmd);
89e1f7d4c66d85 Alex Williamson   2012-07-31  541  
d6ddf3a1f66d02 Reinette Chatre   2023-03-15  542  	ctx->producer.token = trigger;
d6ddf3a1f66d02 Reinette Chatre   2023-03-15  543  	ctx->producer.irq = irq;
d6ddf3a1f66d02 Reinette Chatre   2023-03-15  544  	ret = irq_bypass_register_producer(&ctx->producer);
852b1beecb6ff9 Alex Williamson   2020-10-19  545  	if (unlikely(ret)) {
6d7425f109d262 Feng Wu           2015-09-18  546  		dev_info(&pdev->dev,
6d7425f109d262 Feng Wu           2015-09-18  547  		"irq bypass producer (token %p) registration fails: %d\n",
d6ddf3a1f66d02 Reinette Chatre   2023-03-15  548  		ctx->producer.token, ret);
6d7425f109d262 Feng Wu           2015-09-18  549  
d6ddf3a1f66d02 Reinette Chatre   2023-03-15  550  		ctx->producer.token = NULL;
852b1beecb6ff9 Alex Williamson   2020-10-19  551  	}
d6ddf3a1f66d02 Reinette Chatre   2023-03-15  552  	ctx->trigger = trigger;
89e1f7d4c66d85 Alex Williamson   2012-07-31  553  
89e1f7d4c66d85 Alex Williamson   2012-07-31  554  	return 0;
224474be3986e1 Reinette Chatre   2023-03-15  555  
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  556  out_free_irq_locked:
db5c3d2cbe1a75 Reinette Chatre   2023-03-15 @557  	if (msix && new_ctx) {
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  558  		msix_map.index = vector;
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  559  		msix_map.virq = irq;
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  560  		pci_msix_free_irq(pdev, msix_map);
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  561  	}
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  562  	vfio_pci_memory_unlock_and_restore(vdev, cmd);
224474be3986e1 Reinette Chatre   2023-03-15  563  out_put_eventfd_ctx:
224474be3986e1 Reinette Chatre   2023-03-15  564  	eventfd_ctx_put(trigger);
224474be3986e1 Reinette Chatre   2023-03-15  565  out_free_name:
224474be3986e1 Reinette Chatre   2023-03-15  566  	kfree(ctx->name);
224474be3986e1 Reinette Chatre   2023-03-15  567  	ctx->name = NULL;
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  568  out_free_ctx:
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  569  	if (msix && new_ctx)
db5c3d2cbe1a75 Reinette Chatre   2023-03-15  570  		vfio_irq_ctx_free(vdev, vector);
224474be3986e1 Reinette Chatre   2023-03-15  571  	return ret;
89e1f7d4c66d85 Alex Williamson   2012-07-31  572  }
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests
#11 lore raw

Re: [RFC PATCH 7/8] vfio/pci: Support dynamic MSI-x

Quoted 14 lines · Dan Carpenter

Hi Reinette,

url:    https://github.com/intel-lab-lkp/linux/commits/Reinette-Chatre/vfio-pci-Consolidate-irq-cleanup-on-MSI-MSI-X-disable/20230316-050150
base:   eeac8ede17557680855031c6f305ece2378af326
patch link:    https://lore.kernel.org/r/591ce11f4a33e022fc9242324ebdc077202bedaf.1678911529.git.reinette.chatre%40intel.com
patch subject: [RFC PATCH 7/8] vfio/pci: Support dynamic MSI-x
config: ia64-randconfig-m031-20230312 (https://download.01.org/0day-ci/archive/20230316/[email protected]/config)
compiler: ia64-linux-gcc (GCC) 12.1.0

If you fix the issue, kindly add following tag where applicable

| Reported-by: kernel test robot <[email protected]>
| Reported-by: Dan Carpenter <[email protected]>
| Link: https://lore.kernel.org/r/[email protected]/
Quoted 5 lines

New smatch warnings:
drivers/vfio/pci/vfio_pci_intrs.c:557 vfio_msi_set_vector_signal() error: uninitialized symbol 'new_ctx'.

Old smatch warnings:
drivers/vfio/pci/vfio_pci_intrs.c:569 vfio_msi_set_vector_signal() error: uninitialized symbol 'new_ctx'.

Thank you very much Dan.

Reinette

#12 lore raw

RFC PATCH 8/8 vfio/pci: Clear VFIO_IRQ_INFO_NORESIZE for MSI-X

In reply to Reinette Chatre

Dynamic MSI-X is supported. Clear VFIO_IRQ_INFO_NORESIZE to provide guidance to user space.

---
 drivers/vfio/pci/vfio_pci_core.c | 2 +-
 include/uapi/linux/vfio.h        | 3 +++
 2 files changed, 4 insertions(+), 1 deletion(-)
drivers/vfio/pci/vfio_pci_core.c +1 -1
index ae0e161c7fc9..1d071ee212a7 100644
@@ -1111,7 +1111,7 @@ static int vfio_pci_ioctl_get_irq_info(struct vfio_pci_core_device *vdev,
 	if (info.index == VFIO_PCI_INTX_IRQ_INDEX)
 		info.flags |=
 			(VFIO_IRQ_INFO_MASKABLE | VFIO_IRQ_INFO_AUTOMASKED);
-	else
+	else if (info.index != VFIO_PCI_MSIX_IRQ_INDEX)
 		info.flags |= VFIO_IRQ_INFO_NORESIZE;
 
 	return copy_to_user(arg, &info, minsz) ? -EFAULT : 0;
include/uapi/linux/vfio.h +3 -0
index 0552e8dcf0cb..1a36134cae5c 100644
@@ -511,6 +511,9 @@ struct vfio_region_info_cap_nvlink2_lnkspd {
  * then add and unmask vectors, it's up to userspace to make the decision
  * whether to allocate the maximum supported number of vectors or tear
  * down setup and incrementally increase the vectors as each is enabled.
+ * Absence of the NORESIZE flag indicates that vectors can be enabled
+ * and disabled dynamically without impacting other vectors within the
+ * index.
  */
 struct vfio_irq_info {
 	__u32	argsz;
Signature · 1 line
2.34.1
#13 lore raw

Re: [RFC PATCH 8/8] vfio/pci: Clear VFIO_IRQ_INFO_NORESIZE for MSI-X

Quoted 14 lines · On Wed, 15 Mar 2023 13:59:28 -0700 Reinette Chatre <[email protected]>…

Dynamic MSI-X is supported. Clear VFIO_IRQ_INFO_NORESIZE to provide guidance to user space.

---
 drivers/vfio/pci/vfio_pci_core.c | 2 +-
 include/uapi/linux/vfio.h        | 3 +++
 2 files changed, 4 insertions(+), 1 deletion(-)
drivers/vfio/pci/vfio_pci_core.c +0 -0
index ae0e161c7fc9..1d071ee212a7 100644

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_get_irq_info(struct vfio_pci_core_device *vdev, · line 1116

Quoted 7 lines
drivers/vfio/pci/vfio_pci_core.c +1 -1
@@ -1111,7 +1111,7 @@ static int vfio_pci_ioctl_get_irq_info(struct vfio_pci_core_device *vdev,
 	if (info.index == VFIO_PCI_INTX_IRQ_INDEX)
 		info.flags |=
 			(VFIO_IRQ_INFO_MASKABLE | VFIO_IRQ_INFO_AUTOMASKED);
-	else
+	else if (info.index != VFIO_PCI_MSIX_IRQ_INDEX)
 		info.flags |= VFIO_IRQ_INFO_NORESIZE;
 

I think we need to check pci_msix_can_alloc_dyn(), right? Thanks,

Alex

Quoted 15 lines
drivers/vfio/pci/vfio_pci_core.c +0 -0
 	return copy_to_user(arg, &info, minsz) ? -EFAULT : 0;
include/uapi/linux/vfio.h +3 -0
index 0552e8dcf0cb..1a36134cae5c 100644
@@ -511,6 +511,9 @@ struct vfio_region_info_cap_nvlink2_lnkspd {
  * then add and unmask vectors, it's up to userspace to make the decision
  * whether to allocate the maximum supported number of vectors or tear
  * down setup and incrementally increase the vectors as each is enabled.
+ * Absence of the NORESIZE flag indicates that vectors can be enabled
+ * and disabled dynamically without impacting other vectors within the
+ * index.
  */
 struct vfio_irq_info {
 	__u32	argsz;
#14 lore raw

Re: [RFC PATCH 8/8] vfio/pci: Clear VFIO_IRQ_INFO_NORESIZE for MSI-X

Hi Alex,

Quoted 12 lines · Alex Williamson
Quoted 9 lines · On Wed, 15 Mar 2023 13:59:28 -0700 Reinette Chatre <[email protected]>…

Dynamic MSI-X is supported. Clear VFIO_IRQ_INFO_NORESIZE to provide guidance to user space.

---
 drivers/vfio/pci/vfio_pci_core.c | 2 +-
 include/uapi/linux/vfio.h        | 3 +++
 2 files changed, 4 insertions(+), 1 deletion(-)
Quoted 14 lines

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_get_irq_info(struct vfio_pci_core_device *vdev, · line 1116

Quoted 12 lines
drivers/vfio/pci/vfio_pci_core.c +1 -1
index ae0e161c7fc9..1d071ee212a7 100644
@@ -1111,7 +1111,7 @@ static int vfio_pci_ioctl_get_irq_info(struct vfio_pci_core_device *vdev,
 	if (info.index == VFIO_PCI_INTX_IRQ_INDEX)
 		info.flags |=
 			(VFIO_IRQ_INFO_MASKABLE | VFIO_IRQ_INFO_AUTOMASKED);
-	else
+	else if (info.index != VFIO_PCI_MSIX_IRQ_INDEX)
 		info.flags |= VFIO_IRQ_INFO_NORESIZE;
 

I think we need to check pci_msix_can_alloc_dyn(), right? Thanks,

Can pci_msix_can_alloc_dyn() ever return false?

I cannot see how pci_msix_can_alloc_dyn() can return false when considering the VFIO PCI MSI-X flow:

vfio_msi_enable(..., ..., msix == true)
pci_alloc_irq_vectors(..., ..., ..., flag == PCI_IRQ_MSIX)

    pci_alloc_irq_vectors_affinity() 
      __pci_enable_msix_range() 
        pci_setup_msix_device_domain() 
          pci_create_device_domain(..., &pci_msix_template, ...)

The template used above, pci_msix_template, has MSI_FLAG_PCI_MSIX_ALLOC_DYN hardcoded. This is the flag that pci_msix_can_alloc_dyn() tests for.

If indeed pci_msix_can_alloc_dyn() can return false in the VFIO PCI MSI-X usage then this series needs to be reworked to continue supporting existing allocation mechanism as well as dynamic MSI-X allocation. Which allocation mechanism to use would be depend on pci_msix_can_alloc_dyn().

Alternatively, if you agree that VFIO PCI MSI-X can expect pci_msix_can_alloc_dyn() to always return true then I should perhaps add a test in vfio_msi_enable() that confirms this is the case. This would cause vfio_msi_enable() to fail if dynamic MSI-X is not possible, perhaps even complain loudly with a WARN.

Reinette

#15 lore raw

Re: [RFC PATCH 8/8] vfio/pci: Clear VFIO_IRQ_INFO_NORESIZE for MSI-X

Quoted 50 lines · On Fri, 17 Mar 2023 15:21:09 -0700 Reinette Chatre <[email protected]>…

Hi Alex,

Quoted 12 lines · Alex Williamson
Quoted 9 lines · On Wed, 15 Mar 2023 13:59:28 -0700 Reinette Chatre <[email protected]>…

Dynamic MSI-X is supported. Clear VFIO_IRQ_INFO_NORESIZE to provide guidance to user space.

---
 drivers/vfio/pci/vfio_pci_core.c | 2 +-
 include/uapi/linux/vfio.h        | 3 +++
 2 files changed, 4 insertions(+), 1 deletion(-)
Quoted 14 lines

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_get_irq_info(struct vfio_pci_core_device *vdev, · line 1116

Quoted 12 lines
drivers/vfio/pci/vfio_pci_core.c +1 -1
index ae0e161c7fc9..1d071ee212a7 100644
@@ -1111,7 +1111,7 @@ static int vfio_pci_ioctl_get_irq_info(struct vfio_pci_core_device *vdev,
 	if (info.index == VFIO_PCI_INTX_IRQ_INDEX)
 		info.flags |=
 			(VFIO_IRQ_INFO_MASKABLE | VFIO_IRQ_INFO_AUTOMASKED);
-	else
+	else if (info.index != VFIO_PCI_MSIX_IRQ_INDEX)
 		info.flags |= VFIO_IRQ_INFO_NORESIZE;
 

I think we need to check pci_msix_can_alloc_dyn(), right? Thanks,

Can pci_msix_can_alloc_dyn() ever return false?

I cannot see how pci_msix_can_alloc_dyn() can return false when considering the VFIO PCI MSI-X flow:

vfio_msi_enable(..., ..., msix == true)
pci_alloc_irq_vectors(..., ..., ..., flag == PCI_IRQ_MSIX)

    pci_alloc_irq_vectors_affinity() 
      __pci_enable_msix_range() 
        pci_setup_msix_device_domain() 
          pci_create_device_domain(..., &pci_msix_template, ...)

The template used above, pci_msix_template, has MSI_FLAG_PCI_MSIX_ALLOC_DYN hardcoded. This is the flag that pci_msix_can_alloc_dyn() tests for.

If indeed pci_msix_can_alloc_dyn() can return false in the VFIO PCI MSI-X usage then this series needs to be reworked to continue supporting existing allocation mechanism as well as dynamic MSI-X allocation. Which allocation mechanism to use would be depend on pci_msix_can_alloc_dyn().

Quoted 5 lines

Alternatively, if you agree that VFIO PCI MSI-X can expect pci_msix_can_alloc_dyn() to always return true then I should perhaps add a test in vfio_msi_enable() that confirms this is the case. This would cause vfio_msi_enable() to fail if dynamic MSI-X is not possible, perhaps even complain loudly with a WARN.

pci_setup_msix_device_domain() says it returns true if:

* True when:
* - The device does not have a MSI parent irq domain associated,
* which keeps the legacy architecture specific and the global
* PCI/MSI domain models working
* - The MSI-X domain exists already
* - The MSI-X domain was successfully allocated

That first one seems concerning, dynamic allocation only works on irq domain configurations. What does that exclude and do we care about any of them for vfio-pci? Minimally this suggests a dependency on IRQ_DOMAIN, which we don't currently have, but I'm not sure if supporting irq domains is the same as having irq domains. Thanks,

Alex

#16 lore raw

Re: [RFC PATCH 8/8] vfio/pci: Clear VFIO_IRQ_INFO_NORESIZE for MSI-X

Hi Alex,

Quoted 68 lines · Alex Williamson
Quoted 50 lines · On Fri, 17 Mar 2023 15:21:09 -0700 Reinette Chatre <[email protected]>…

Hi Alex,

Quoted 12 lines · Alex Williamson
Quoted 9 lines · On Wed, 15 Mar 2023 13:59:28 -0700 Reinette Chatre <[email protected]>…

Dynamic MSI-X is supported. Clear VFIO_IRQ_INFO_NORESIZE to provide guidance to user space.

---
 drivers/vfio/pci/vfio_pci_core.c | 2 +-
 include/uapi/linux/vfio.h        | 3 +++
 2 files changed, 4 insertions(+), 1 deletion(-)
Quoted 14 lines

drivers/vfio/pci/vfio_pci_core.c static int vfio_pci_ioctl_get_irq_info(struct vfio_pci_core_device *vdev, · line 1116

Quoted 12 lines
drivers/vfio/pci/vfio_pci_core.c +1 -1
index ae0e161c7fc9..1d071ee212a7 100644
@@ -1111,7 +1111,7 @@ static int vfio_pci_ioctl_get_irq_info(struct vfio_pci_core_device *vdev,
 	if (info.index == VFIO_PCI_INTX_IRQ_INDEX)
 		info.flags |=
 			(VFIO_IRQ_INFO_MASKABLE | VFIO_IRQ_INFO_AUTOMASKED);
-	else
+	else if (info.index != VFIO_PCI_MSIX_IRQ_INDEX)
 		info.flags |= VFIO_IRQ_INFO_NORESIZE;
 

I think we need to check pci_msix_can_alloc_dyn(), right? Thanks,

Can pci_msix_can_alloc_dyn() ever return false?

I cannot see how pci_msix_can_alloc_dyn() can return false when considering the VFIO PCI MSI-X flow:

vfio_msi_enable(..., ..., msix == true)
pci_alloc_irq_vectors(..., ..., ..., flag == PCI_IRQ_MSIX)

    pci_alloc_irq_vectors_affinity() 
      __pci_enable_msix_range() 
        pci_setup_msix_device_domain() 
          pci_create_device_domain(..., &pci_msix_template, ...)

The template used above, pci_msix_template, has MSI_FLAG_PCI_MSIX_ALLOC_DYN hardcoded. This is the flag that pci_msix_can_alloc_dyn() tests for.

If indeed pci_msix_can_alloc_dyn() can return false in the VFIO PCI MSI-X usage then this series needs to be reworked to continue supporting existing allocation mechanism as well as dynamic MSI-X allocation. Which allocation mechanism to use would be depend on pci_msix_can_alloc_dyn().

Quoted 5 lines

Alternatively, if you agree that VFIO PCI MSI-X can expect pci_msix_can_alloc_dyn() to always return true then I should perhaps add a test in vfio_msi_enable() that confirms this is the case. This would cause vfio_msi_enable() to fail if dynamic MSI-X is not possible, perhaps even complain loudly with a WARN.

pci_setup_msix_device_domain() says it returns true if:

* True when:
* - The device does not have a MSI parent irq domain associated,
* which keeps the legacy architecture specific and the global
* PCI/MSI domain models working
* - The MSI-X domain exists already
* - The MSI-X domain was successfully allocated

Quoted 5 lines

That first one seems concerning, dynamic allocation only works on irq domain configurations. What does that exclude and do we care about any of them for vfio-pci? Minimally this suggests a dependency on IRQ_DOMAIN, which we don't currently have, but I'm not sure if supporting irq domains is the same as having irq domains. Thanks,

Just to confirm, as I mentioned in [1] I do plan to rework this solution to support both allocation mechanisms, using pci_msix_can_alloc_dyn() to pick which one to use. Thank you very much for pointing out this gap to me.

Reinette

[1] https://lore.kernel.org/lkml/[email protected]/

#17 lore raw

Re: [RFC PATCH 8/8] vfio/pci: Clear VFIO_IRQ_INFO_NORESIZE for MSI-X

In reply to Alex Williamson
Quoted 9 lines · Alex Williamson

pci_setup_msix_device_domain() says it returns true if:

* True when:
* - The device does not have a MSI parent irq domain associated,
* which keeps the legacy architecture specific and the global
* PCI/MSI domain models working
* - The MSI-X domain exists already
* - The MSI-X domain was successfully allocated

Quoted 3 lines

That first one seems concerning, dynamic allocation only works on irq domain configurations. What does that exclude and do we care about any of them for vfio-pci?

Several archs and other weird things override the MSI-X programming. Eg by turning it into a hypervisor call or something. These were not converted to dynamic mode.

So at the VFIO level you can get end up with MSI support but done through legacy paths that don't support dynamic allocation. Eg on POWER, xen, etc.

Jason

#18 lore raw
In reply to Reinette Chatre
Quoted 47 lines · On Wed, 15 Mar 2023 13:59:20 -0700 Reinette Chatre <[email protected]>…

== Cover letter ==

Qemu allocates interrupts incrementally at the time the guest unmasks an interrupt, for example each time a Linux guest runs request_irq().

Dynamic allocation of MSI-X interrupts was not possible until v6.2 [1]. This prompted Qemu to, when allocating a new interrupt, first release all previously allocated interrupts (including disable of MSI-X) followed by re-allocation of all interrupts that includes the new interrupt. Please see [2] for a detailed discussion about this issue.

Releasing and re-allocating interrupts may be acceptable if all interrupts are unmasked during device initialization. If unmasking of interrupts occur during runtime this may result in lost interrupts. For example, consider an accelerator device with multiple work queues, each work queue having a dedicated interrupt. A work queue can be enabled at any time with its associated interrupt unmasked while other work queues are already active. Having all interrupts released and MSI-X disabled to enable the new work queue will impact active work queues.

This series builds on the recent interrupt sub-system core changes that added support for dynamic MSI-X allocation after initial MSI-X enabling.

Add support for dynamic MSI-X allocation to vfio-pci. A flag indicating lack of support for dynamic allocation already exist:
VFIO_IRQ_INFO_NORESIZE and has always been set for MSI and MSI-X. With support for dynamic MSI-X the flag is cleared for MSI-X, enabling Qemu to modify its behavior.

== Why is this an RFC ? ==

vfio support for dynamic MSI-X needs to work with existing user space as well as upcoming user space that takes advantage of this feature. I would appreciate guidance on the expectations and requirements surrounding error handling when considering existing user space.

For example, consider the following scenario:
Start: Consider a passthrough device that supports 10 MSI-X interrupts
(0 to 9) and existing Qemu allocated interrupts 0 to 4.

Scenario: Qemu (hypothetically) attempts to associate a new action to
interrupts 0 to 7. Early checking of this range in vfio_set_irqs_validate_and_prepare() will pass since it is a valid range for the device. What happens after the early checking is considered next:

Quoted 4 lines

Current behavior (before this series): Since the provided range, 0 - 7,
exceeds the allocated range, no action will be taken on existing allocated interrupts 0 - 4 and the Qemu request will fail without making any state changes.

I must be missing something, it was described correctly earlier that QEMU will disable MSI-X and re-enable with a new vector count. Not only does QEMU not really have a way to fail this change, pretty much nothing would work if we did.

What happens in this case is that the QEMU vfio-pci driver gets a vector_use callback for one of these new vectors {5,6,7}, vfio_msix_vector_do_use() checks that's greater than we have enabled, disables MSI-X, and re-enables it for the new vector count. Worst case we'll do that 3 times if the vectors are presented in ascending order.

Quoted 6 lines

New behavior (with this series): Since vfio supports dynamic MSI-X new
interrupts will be allocated for vectors 5, 6, and 7. Please note that this changes the behavior encountered by unmodified Qemu: new interrupts are allocated instead of returning an error. Even more, since the range provided by Qemu spans 0 - 7, a failure during allocation of 5, 6, or 7 will result in release of entire range.

As above, QEMU doesn't currently return an error here, nor is there actually any means to return an error. MSI-X is not paravirtualized and the PCI spec definition of MSI-X does not define that a driver needs to check whether the device accepted unmasking a vector. All we can do in QEMU if the host fails to configure the device as directed is print an error message as a breadcrumb to help debug why the device stopped working. In practice, I don't think I've ever seen this.

So really the difference should be transparent to the guest. The risk of an error allocating vectors on the host is the same, the only significant difference should be the amount of disruption at the device that we can better approximate the request of the guest.

Quoted 4 lines

This series aims to maintain existing error behavior for MSI (please see "vfio/pci: Remove interrupt context counter") but I would appreciate your guidance on whether existing error behavior should be maintained for MSI-X and how to do so if it is a requirement.

Based on above, there really can never be an error if we expect the device to work, so I think there's a misread of the current status. Dynamic MSI-X support should simply reduce the disruption and chance of lost interrupts at the device, but the points where we risk that the host cannot provide the configuration we need are the same. Thanks,

Alex

#19 lore raw

Hi Alex,

Quoted 3 lines · Alex Williamson

On Wed, 15 Mar 2023 13:59:20 -0700 Reinette Chatre <[email protected]> wrote:

...

Quoted 23 lines
Quoted 18 lines

== Why is this an RFC ? ==

vfio support for dynamic MSI-X needs to work with existing user space as well as upcoming user space that takes advantage of this feature. I would appreciate guidance on the expectations and requirements surrounding error handling when considering existing user space.

For example, consider the following scenario: Start: Consider a passthrough device that supports 10 MSI-X interrupts (0 to 9) and existing Qemu allocated interrupts 0 to 4.

Scenario: Qemu (hypothetically) attempts to associate a new action to interrupts 0 to 7. Early checking of this range in vfio_set_irqs_validate_and_prepare() will pass since it is a valid range for the device. What happens after the early checking is considered next:

Quoted 4 lines

Current behavior (before this series): Since the provided range, 0
- 7, exceeds the allocated range, no action will be taken on existing allocated interrupts 0 - 4 and the Qemu request will fail without making any state changes.

Quoted 4 lines

I must be missing something, it was described correctly earlier that QEMU will disable MSI-X and re-enable with a new vector count. Not only does QEMU not really have a way to fail this change, pretty much nothing would work if we did.

Thank you very much for confirming Qemu behavior.

One of my goals is to ensure that these kernel changes do not break existing user space in any way. The only area I could find where existing user space may encounter new behavior is if user space makes the (pre dynamic MSI-X) mistake of attempting to associate triggers to interrupts that have not been allocated. Thank you for confirming that this is not a valid scenario for Qemu.

Quoted 6 lines

What happens in this case is that the QEMU vfio-pci driver gets a vector_use callback for one of these new vectors {5,6,7}, vfio_msix_vector_do_use() checks that's greater than we have enabled, disables MSI-X, and re-enables it for the new vector count. Worst case we'll do that 3 times if the vectors are presented in ascending order.

Indeed. While testing this work I modified Qemu's vfio_msix_vector_do_use() to consider VFIO_IRQ_INFO_NORESIZE. I saw the vector_use callback arriving for each new vector and having Qemu just associate a new action with the new vector (call vfio_set_irq_signaling()) instead of disabling MSI-X followed by enabling all vectors worked well with the changes I present here. A single interrupt was dynamically allocated and enabled without impacting others.

Quoted 5 lines

Based on above, there really can never be an error if we expect the device to work, so I think there's a misread of the current status. Dynamic MSI-X support should simply reduce the disruption and chance of lost interrupts at the device, but the points where we risk that the host cannot provide the configuration we need are the same.

Thank you very much Alex. In this case, please do consider this submission as a submission for inclusion. I'd be happy to resubmit without the "RFC" prefix if that is preferred.

Reinette

#20 lore raw
Quoted 9 lines

From: Chatre, Reinette <[email protected]>
Sent: Friday, March 17, 2023 7:38 AM

Quoted 5 lines

Based on above, there really can never be an error if we expect the device to work, so I think there's a misread of the current status. Dynamic MSI-X support should simply reduce the disruption and chance of lost interrupts at the device, but the points where we risk that the host cannot provide the configuration we need are the same.

Quoted 4 lines

Thank you very much Alex. In this case, please do consider this submission as a submission for inclusion. I'd be happy to resubmit without the "RFC" prefix if that is preferred.

With that do we still want to keep the error behavior for MSI?

If no patch5 can be simplified e.g. no need of vfio_irq_ctx_range_allocated() and MSI/MSI-X error behaviors become consistent.

#21 lore raw

Hi Kevin,

Quoted 14 lines · Kevin
Quoted 9 lines

From: Chatre, Reinette <[email protected]>
Sent: Friday, March 17, 2023 7:38 AM

Quoted 5 lines

Based on above, there really can never be an error if we expect the device to work, so I think there's a misread of the current status. Dynamic MSI-X support should simply reduce the disruption and chance of lost interrupts at the device, but the points where we risk that the host cannot provide the configuration we need are the same.

Quoted 4 lines

Thank you very much Alex. In this case, please do consider this submission as a submission for inclusion. I'd be happy to resubmit without the "RFC" prefix if that is preferred.

Quoted 4 lines

With that do we still want to keep the error behavior for MSI?

If no patch5 can be simplified e.g. no need of vfio_irq_ctx_range_allocated() and MSI/MSI-X error behaviors become consistent.

Thank you. Yes, if I understand correctly MSI and MSI-X error handling can become consistent. I'll modify patch 5 to remove vfio_irq_ctx_range_allocated().

Reinette