Skip to contentSkip to conversation

nvme/pci: allocate separate interrupt for reserved non-polled IO queue

RFC In review Jeffle Xu 2 participants 4 messages Sep 22, 2020 – Sep 24, 2020 NVMe

lore
Share & tools
lore

Jeffle Xu

author
#0 lore raw

RFC 1/1 nvme/pci: allocate separate interrupt for reserved non-polled IO queue

One queue will be reserved for non-polled IO when nvme.poll_queues is greater or equal than the number of IO queues that the nvme controller can provide. Currently the reserved queue for non-polled IO will reuse the interrupt used by admin queue in this case, e.g, vector 0.

This can work and the performance may not be an issue since the admin queue is used unfrequently. However this behaviour may be inconsistent with that when nvme.poll_queues is smaller than the number of IO queues available.

Thus allocate separate interrupt for this reserved queue, and thus make the behaviour consistent.

---
 drivers/nvme/host/pci.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/nvme/host/pci.c +4 -2
index 899d2f4d7ab6..ff3e28a8a9e0 100644
@@ -2042,12 +2042,14 @@ static int nvme_setup_irqs(struct nvme_dev *dev, unsigned int nr_io_queues)
 
 	/*
 	 * Poll queues don't need interrupts, but we need at least one IO
-	 * queue left over for non-polled IO.
+	 * queue left over for non-polled IO. Also one interrupt for admin
+	 * queue.
 	 */
 	this_p_queues = dev->nr_poll_queues;
 	if (this_p_queues >= nr_io_queues) {
 		this_p_queues = nr_io_queues - 1;
-		irq_queues = 1;
+		/* Allocate separate interrupt for reserved non-polled IO queue. */
+		irq_queues = 2;
 	} else {
 		irq_queues = nr_io_queues - this_p_queues + 1;
 	}
Signature · 7 lines
2.27.0


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
author
#1 lore raw

Any comments?

Quoted 29 lines · Jeffle Xu

One queue will be reserved for non-polled IO when nvme.poll_queues is greater or equal than the number of IO queues that the nvme controller can provide. Currently the reserved queue for non-polled IO will reuse the interrupt used by admin queue in this case, e.g, vector 0.

This can work and the performance may not be an issue since the admin queue is used unfrequently. However this behaviour may be inconsistent with that when nvme.poll_queues is smaller than the number of IO queues available.

Thus allocate separate interrupt for this reserved queue, and thus make the behaviour consistent.

---
  drivers/nvme/host/pci.c | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)
drivers/nvme/host/pci.c +2 -1
index 899d2f4d7ab6..ff3e28a8a9e0 100644
@@ -2042,12 +2042,14 @@ static int nvme_setup_irqs(struct nvme_dev *dev, unsigned int nr_io_queues)
 
  	/*
  	 * Poll queues don't need interrupts, but we need at least one IO
-	 * queue left over for non-polled IO.
+	 * queue left over for non-polled IO. Also one interrupt for admin
+	 * queue.

drivers/nvme/host/pci.c static int nvme_setup_irqs(struct nvme_dev *dev, unsigned int nr_io_queues) · line 2055

Quoted 10 lines
drivers/nvme/host/pci.c +2 -1
  	 */
  	this_p_queues = dev->nr_poll_queues;
  	if (this_p_queues >= nr_io_queues) {
  		this_p_queues = nr_io_queues - 1;
-		irq_queues = 1;
+		/* Allocate separate interrupt for reserved non-polled IO queue. */
+		irq_queues = 2;
  	} else {
  		irq_queues = nr_io_queues - this_p_queues + 1;
  	}

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#2 lore raw
In reply to Jeffle Xu
Quoted 10 lines · Jeffle Xu

One queue will be reserved for non-polled IO when nvme.poll_queues is greater or equal than the number of IO queues that the nvme controller can provide. Currently the reserved queue for non-polled IO will reuse the interrupt used by admin queue in this case, e.g, vector 0.

This can work and the performance may not be an issue since the admin queue is used unfrequently. However this behaviour may be inconsistent with that when nvme.poll_queues is smaller than the number of IO queues available.

Quoted 4 lines

Thus allocate separate interrupt for this reserved queue, and thus make the behaviour consistent.

This code looks good, but the function already is a mess without the addition. What do you think about this variant?

drivers/nvme/host/pci.c +12 -14
index 899d2f4d7ab612..43055138d59a47 100644
@@ -2038,31 +2038,29 @@ static int nvme_setup_irqs(struct nvme_dev *dev, unsigned int nr_io_queues)
 		.calc_sets	= nvme_calc_irq_sets,
 		.priv		= dev,
 	};
-	unsigned int irq_queues, this_p_queues;
+	unsigned int irq_queues, poll_queues;
 
 	/*
-	 * Poll queues don't need interrupts, but we need at least one IO
-	 * queue left over for non-polled IO.
+	 * Poll queues don't need interrupts, but we need at least one I/O queue
+	 * left over for non-polled I/O.
 	 */
-	this_p_queues = dev->nr_poll_queues;
-	if (this_p_queues >= nr_io_queues) {
-		this_p_queues = nr_io_queues - 1;
-		irq_queues = 1;
-	} else {
-		irq_queues = nr_io_queues - this_p_queues + 1;
-	}
-	dev->io_queues[HCTX_TYPE_POLL] = this_p_queues;
+	poll_queues = min(dev->nr_poll_queues, nr_io_queues - 1);
+	dev->io_queues[HCTX_TYPE_POLL] = poll_queues;
 
-	/* Initialize for the single interrupt case */
+	/*
+	 * Initialize for the single interrupt case, will be updated in
+	 * nvme_calc_irq_sets().
+	 */
 	dev->io_queues[HCTX_TYPE_DEFAULT] = 1;
 	dev->io_queues[HCTX_TYPE_READ] = 0;
 
 	/*
-	 * Some Apple controllers require all queues to use the
-	 * first vector.
+	 * Some Apple controllers require all queues to use the first vector.
 	 */
 	if (dev->ctrl.quirks & NVME_QUIRK_SINGLE_VECTOR)
 		irq_queues = 1;
+	else
+		irq_queues = 1 + (nr_io_queues - poll_queues);
 
 	return pci_alloc_irq_vectors_affinity(pdev, 1, irq_queues,
 			      PCI_IRQ_ALL_TYPES | PCI_IRQ_AFFINITY, &affd);

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author
#3 lore raw

Thanks for replying.

Quoted 11 lines · Christoph Hellwig
Quoted 10 lines · Jeffle Xu

One queue will be reserved for non-polled IO when nvme.poll_queues is greater or equal than the number of IO queues that the nvme controller can provide. Currently the reserved queue for non-polled IO will reuse the interrupt used by admin queue in this case, e.g, vector 0.

This can work and the performance may not be an issue since the admin queue is used unfrequently. However this behaviour may be inconsistent with that when nvme.poll_queues is smaller than the number of IO queues available.

Quoted 6 lines
Quoted 4 lines

Thus allocate separate interrupt for this reserved queue, and thus make the behaviour consistent.

This code looks good, but the function already is a mess without the addition. What do you think about this variant?

Looks good to me. If you don't mind, I'd like to send a v2 patch containing your refactored code.

Also some trivial advice below.

Quoted 38 lines
drivers/nvme/host/pci.c +9 -12
index 899d2f4d7ab612..43055138d59a47 100644
@@ -2038,31 +2038,29 @@ static int nvme_setup_irqs(struct nvme_dev *dev, unsigned int nr_io_queues)
  		.calc_sets	= nvme_calc_irq_sets,
  		.priv		= dev,
  	};
-	unsigned int irq_queues, this_p_queues;
+	unsigned int irq_queues, poll_queues;
 
  	/*
-	 * Poll queues don't need interrupts, but we need at least one IO
-	 * queue left over for non-polled IO.
+	 * Poll queues don't need interrupts, but we need at least one I/O queue
+	 * left over for non-polled I/O.
  	 */
-	this_p_queues = dev->nr_poll_queues;
-	if (this_p_queues >= nr_io_queues) {
-		this_p_queues = nr_io_queues - 1;
-		irq_queues = 1;
-	} else {
-		irq_queues = nr_io_queues - this_p_queues + 1;
-	}
-	dev->io_queues[HCTX_TYPE_POLL] = this_p_queues;
+	poll_queues = min(dev->nr_poll_queues, nr_io_queues - 1);
+	dev->io_queues[HCTX_TYPE_POLL] = poll_queues;
 
-	/* Initialize for the single interrupt case */
+	/*
+	 * Initialize for the single interrupt case, will be updated in
+	 * nvme_calc_irq_sets().
+	 */
  	dev->io_queues[HCTX_TYPE_DEFAULT] = 1;
  	dev->io_queues[HCTX_TYPE_READ] = 0;
 

drivers/nvme/host/pci.c static int nvme_setup_irqs(struct nvme_dev *dev, unsigned int nr_io_queues) · line 2064

Quoted 10 lines
drivers/nvme/host/pci.c +3 -2
  	/*
-	 * Some Apple controllers require all queues to use the
-	 * first vector.
+	 * Some Apple controllers require all queues to use the first vector.
  	 */
  	if (dev->ctrl.quirks & NVME_QUIRK_SINGLE_VECTOR)
  		irq_queues = 1;
+	else
 
+		irq_queues = 1 + (nr_io_queues - poll_queues);

Personally it would be better if there's a comment to explicitly point out that there's one interrupt for admin queue.

It would be more friendly to the newborns.

such as

        /*
-        * Some Apple controllers require all queues to use the
-        * first vector.
+        * Some Apple controllers require all queues to use the first vector. */
        if (dev->ctrl.quirks & NVME_QUIRK_SINGLE_VECTOR) irq_queues = 1;

+       else {
+               /* One interrupt for admin queue.*/ +               irq_queues = 1 + (nr_io_queues - poll_queues); +       }

Quoted 11 lines
drivers/nvme/host/pci.c +0 -0
  	return pci_alloc_irq_vectors_affinity(pdev, 1, irq_queues,
  			      PCI_IRQ_ALL_TYPES | PCI_IRQ_AFFINITY, &affd);

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme