Skip to contentSkip to conversation

hw/block/nvme: fix assert on invalid irq vector

PATCH In review Klaus Jensen 4 participants 11 messages Jun 9, 2020 – Jul 7, 2020

lore
Share & tools
lore
#0 lore raw

cover hw/block/nvme: fix assert on invalid irq vector

From: Klaus Jensen <[email protected]>

I goofed up with commit c09794fe40e3 ("hw/block/nvme: allow use of any valid msix vector").

This fixes the goof by adding a new msix_qsize parameter. As a nice side-effect this allows a device with less interrupt vectors available than supported queues. Also, improve the error handling in nvme_init_pci().

Kevin, please consider picking this up for the block branch when reviewed.

Cc Keith Busch <[email protected]>
Cc Max Reitz <[email protected]>
Cc Kevin Wolf <[email protected]>
Cc Klaus Jensen <[email protected]>
Cc Javier Gonzalez <[email protected]>
Cc Maxim Levitsky <[email protected]>
Cc Philippe Mathieu-Daudé <[email protected]>

Klaus Jensen (2):
hw/block/nvme: add msix_qsize parameter hw/block/nvme: verify msix_init_exclusive_bar() return value

 hw/block/nvme.c | 28 ++++++++++++++++++++++------
 hw/block/nvme.h |  1 +
 2 files changed, 23 insertions(+), 6 deletions(-)
Signature · 1 line
2.27.0
#1 lore raw

PATCH 1/2 hw/block/nvme: add msix_qsize parameter

From: Klaus Jensen <[email protected]>

Commit 4eb9f5217a16 ("hw/block/nvme: allow use of any valid msix vector") erronously allows any valid MSI-X vector to be used when creating completion queues. This is bad because MSI-X is initialized with max_ioqpairs + 1 vectors, which defaults to 64 + 1. And since commit ccd579b2ed03 ("hw/block/nvme: Verify msix_vector_use() returned value"), this also causes an assert if the host creates a completion queue with an invalid vector.

Fix this by decoupling the requested maximum number of ioqpairs (param max_ioqpairs) from the number of MSI-X interrupt vectors by introducing a new msix_qsize parameter and initialize MSI-X with that. This allows emulating a device that has fewer vectors than I/O queue pairs and also allows more than 2048 queue pairs. To get the same default behaviour as previously, msix_qsize defaults to 65 (max_ioqpairs default + 1).

This decoupling was actually suggested by Maxim some time ago in a slightly different context, so adding a Suggested-by.

Fixes 4eb9f5217a16 ("hw/block/nvme: allow use of any valid msix vector")
---
 hw/block/nvme.c | 17 +++++++++++++----
 hw/block/nvme.h |  1 +
 2 files changed, 14 insertions(+), 4 deletions(-)
hw/block/nvme.c +13 -4
index 567bce75191a..acc6dbc900e2 100644
@@ -54,6 +54,7 @@
 #include "trace.h"
 #include "nvme.h"
 
+#define NVME_MAX_IOQPAIRS 0xffff
 #define NVME_REG_SIZE 0x1000
 #define NVME_DB_SIZE  4
 #define NVME_CMB_BIR 2
@@ -662,7 +663,7 @@ static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeCmd *cmd)
         trace_pci_nvme_err_invalid_create_cq_vector(vector);
         return NVME_INVALID_IRQ_VECTOR | NVME_DNR;
     }
-    if (unlikely(vector > PCI_MSIX_FLAGS_QSIZE)) {
+    if (unlikely(vector >= n->params.msix_qsize)) {
         trace_pci_nvme_err_invalid_create_cq_vector(vector);
         return NVME_INVALID_IRQ_VECTOR | NVME_DNR;
     }
@@ -1371,9 +1372,16 @@ static void nvme_check_constraints(NvmeCtrl *n, Error **errp)
     }
 
     if (params->max_ioqpairs < 1 ||
-        params->max_ioqpairs > PCI_MSIX_FLAGS_QSIZE) {
+        params->max_ioqpairs > NVME_MAX_IOQPAIRS) {
         error_setg(errp, "max_ioqpairs must be between 1 and %d",
-                   PCI_MSIX_FLAGS_QSIZE);
+                   NVME_MAX_IOQPAIRS);
+        return;
+    }
+
+    if (params->msix_qsize < 1 ||
+        params->msix_qsize > PCI_MSIX_FLAGS_QSIZE + 1) {
+        error_setg(errp, "msix_qsize must be between 1 and %d",
+                   PCI_MSIX_FLAGS_QSIZE + 1);
         return;
     }
 
@@ -1527,7 +1535,7 @@ static void nvme_init_pci(NvmeCtrl *n, PCIDevice *pci_dev)
                           n->reg_size);
     pci_register_bar(pci_dev, 0, PCI_BASE_ADDRESS_SPACE_MEMORY |
                      PCI_BASE_ADDRESS_MEM_TYPE_64, &n->iomem);
-    msix_init_exclusive_bar(pci_dev, n->params.max_ioqpairs + 1, 4, NULL);
+    msix_init_exclusive_bar(pci_dev, n->params.msix_qsize, 4, NULL);
 
     if (n->params.cmb_size_mb) {
         nvme_init_cmb(n, pci_dev);
@@ -1634,6 +1642,7 @@ static Property nvme_props[] = {
     DEFINE_PROP_UINT32("cmb_size_mb", NvmeCtrl, params.cmb_size_mb, 0),
     DEFINE_PROP_UINT32("num_queues", NvmeCtrl, params.num_queues, 0),
     DEFINE_PROP_UINT32("max_ioqpairs", NvmeCtrl, params.max_ioqpairs, 64),
+    DEFINE_PROP_UINT16("msix_qsize", NvmeCtrl, params.msix_qsize, 65),
     DEFINE_PROP_END_OF_LIST(),
 };
 
hw/block/nvme.h +1 -0
index 61dd9b23b81d..1d30c0bca283 100644
@@ -7,6 +7,7 @@ typedef struct NvmeParams {
     char     *serial;
     uint32_t num_queues; /* deprecated since 5.1 */
     uint32_t max_ioqpairs;
+    uint16_t msix_qsize;
     uint32_t cmb_size_mb;
 } NvmeParams;
 
Signature · 1 line
2.27.0
#2 lore raw

PATCH 2/2 hw/block/nvme: verify msix_init_exclusive_bar() return value

In reply to Klaus Jensen

From: Klaus Jensen <[email protected]>

Pass an Error to msix_init_exclusive_bar() and check it.

---
 hw/block/nvme.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
hw/block/nvme.c +10 -3
index acc6dbc900e2..2a2e43f681f9 100644
@@ -1522,7 +1522,7 @@ static void nvme_init_pmr(NvmeCtrl *n, PCIDevice *pci_dev)
                      PCI_BASE_ADDRESS_MEM_PREFETCH, &n->pmrdev->mr);
 }
 
-static void nvme_init_pci(NvmeCtrl *n, PCIDevice *pci_dev)
+static void nvme_init_pci(NvmeCtrl *n, PCIDevice *pci_dev, Error **errp)
 {
     uint8_t *pci_conf = pci_dev->config;
 
@@ -1535,7 +1535,9 @@ static void nvme_init_pci(NvmeCtrl *n, PCIDevice *pci_dev)
                           n->reg_size);
     pci_register_bar(pci_dev, 0, PCI_BASE_ADDRESS_SPACE_MEMORY |
                      PCI_BASE_ADDRESS_MEM_TYPE_64, &n->iomem);
-    msix_init_exclusive_bar(pci_dev, n->params.msix_qsize, 4, NULL);
+    if (msix_init_exclusive_bar(pci_dev, n->params.msix_qsize, 4, errp)) {
+        return;
+    }
 
     if (n->params.cmb_size_mb) {
         nvme_init_cmb(n, pci_dev);
@@ -1603,7 +1605,12 @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)
         return;
     }
 
-    nvme_init_pci(n, pci_dev);
+    nvme_init_pci(n, pci_dev, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
     nvme_init_ctrl(n, pci_dev);
 
     for (i = 0; i < n->num_namespaces; i++) {
Signature · 1 line
2.27.0
#3 lore raw
In reply to Klaus Jensen
Quoted 4 lines · Klaus Jensen

From: Klaus Jensen <[email protected]>

I goofed up with commit c09794fe40e3 ("hw/block/nvme: allow use of any valid msix vector").

Kevin, since your queue isn't merged, can you directly squash the fix?

Quoted 26 lines

This fixes the goof by adding a new msix_qsize parameter. As a nice side-effect this allows a device with less interrupt vectors available than supported queues. Also, improve the error handling in nvme_init_pci().

Kevin, please consider picking this up for the block branch when reviewed.

Cc Keith Busch <[email protected]>
Cc Max Reitz <[email protected]>
Cc Kevin Wolf <[email protected]>
Cc Klaus Jensen <[email protected]>
Cc Javier Gonzalez <[email protected]>
Cc Maxim Levitsky <[email protected]>
Cc Philippe Mathieu-Daudé <[email protected]>

Klaus Jensen (2):
hw/block/nvme: add msix_qsize parameter hw/block/nvme: verify msix_init_exclusive_bar() return value

 hw/block/nvme.c | 28 ++++++++++++++++++++++------
 hw/block/nvme.h |  1 +
 2 files changed, 23 insertions(+), 6 deletions(-)
#4 lore raw
Quoted 7 lines · Philippe Mathieu-Daudé
Quoted 4 lines · Klaus Jensen

From: Klaus Jensen <[email protected]>

I goofed up with commit c09794fe40e3 ("hw/block/nvme: allow use of any valid msix vector").

Kevin, since your queue isn't merged, can you directly squash the fix?

The commit (c09794fe40e3) can just be dropped without conflicts, but it leaves a use of n->params.num_queues in nvme_create_cq() which commit cde74bfd4b87 ("hw/block/nvme: add max_ioqpairs device parameter") must fix.

#5 lore raw

Am 09.06.2020 um 13:46 hat Klaus Jensen geschrieben:

Quoted 7 lines · Philippe Mathieu-Daudé
Quoted 6 lines · Klaus Jensen
Quoted 4 lines

From: Klaus Jensen <[email protected]>

I goofed up with commit c09794fe40e3 ("hw/block/nvme: allow use of any valid msix vector").

Quoted 6 lines
Quoted 1 line

Kevin, since your queue isn't merged, can you directly squash the fix?

The commit (c09794fe40e3) can just be dropped without conflicts, but it leaves a use of n->params.num_queues in nvme_create_cq() which commit cde74bfd4b87 ("hw/block/nvme: add max_ioqpairs device parameter") must fix.

Hm, so it seems this isn't easy to squash in without conflicts (and I would have to rewrite the whole commit message), so I think it's better to just apply the series on top.

One problem with the commit message is that it references commit IDs which aren't stable yet. Maybe it's best if I apply these patches, manually fix up the commit ID references and then immediately do a pull request so that they become stable.

It would be good to have at least one review, though.

Kevin

#6 lore raw
Quoted 15 lines · Kevin Wolf

Am 09.06.2020 um 13:46 hat Klaus Jensen geschrieben:

Quoted 7 lines · Philippe Mathieu-Daudé
Quoted 6 lines · Klaus Jensen
Quoted 4 lines

From: Klaus Jensen <[email protected]>

I goofed up with commit c09794fe40e3 ("hw/block/nvme: allow use of any valid msix vector").

Quoted 6 lines
Quoted 1 line

Kevin, since your queue isn't merged, can you directly squash the fix?

The commit (c09794fe40e3) can just be dropped without conflicts, but it leaves a use of n->params.num_queues in nvme_create_cq() which commit cde74bfd4b87 ("hw/block/nvme: add max_ioqpairs device parameter") must fix.

Quoted 8 lines

Hm, so it seems this isn't easy to squash in without conflicts (and I would have to rewrite the whole commit message), so I think it's better to just apply the series on top.

One problem with the commit message is that it references commit IDs which aren't stable yet. Maybe it's best if I apply these patches, manually fix up the commit ID references and then immediately do a pull request so that they become stable.

This is the friendlier way.

Less friendly way is to drop Klaus's patches and ask him to respin. While this is a valid outcome, if we can avoid it it will save all of us review time.

Quoted 2 lines

It would be good to have at least one review, though.

Maxim catched this issue, I'd feel safer if he acks your pre-merge queue.

Quoted 3 lines

Kevin

#7 lore raw

Am 09.06.2020 um 16:18 hat Philippe Mathieu-Daudé geschrieben:

Quoted 25 lines · Kevin Wolf
Quoted 15 lines

Am 09.06.2020 um 13:46 hat Klaus Jensen geschrieben:

Quoted 7 lines · Philippe Mathieu-Daudé
Quoted 6 lines · Klaus Jensen
Quoted 4 lines

From: Klaus Jensen <[email protected]>

I goofed up with commit c09794fe40e3 ("hw/block/nvme: allow use of any valid msix vector").

Quoted 6 lines
Quoted 1 line

Kevin, since your queue isn't merged, can you directly squash the fix?

The commit (c09794fe40e3) can just be dropped without conflicts, but it leaves a use of n->params.num_queues in nvme_create_cq() which commit cde74bfd4b87 ("hw/block/nvme: add max_ioqpairs device parameter") must fix.

Quoted 8 lines

Hm, so it seems this isn't easy to squash in without conflicts (and I would have to rewrite the whole commit message), so I think it's better to just apply the series on top.

One problem with the commit message is that it references commit IDs which aren't stable yet. Maybe it's best if I apply these patches, manually fix up the commit ID references and then immediately do a pull request so that they become stable.

Quoted 5 lines

This is the friendlier way.

Less friendly way is to drop Klaus's patches and ask him to respin. While this is a valid outcome, if we can avoid it it will save all of us review time.

If Klaus wants to do that, fine with me. I'm just trying to find the easiest solution for all of us.

Quoted 3 lines
Quoted 1 line

It would be good to have at least one review, though.

Maxim catched this issue, I'd feel safer if he acks your pre-merge queue.

Ok. Maxim, can you please review this series then?

Kevin

#8 lore raw
Quoted 26 lines · Kevin Wolf

Am 09.06.2020 um 16:18 hat Philippe Mathieu-Daudé geschrieben:

Quoted 25 lines · Kevin Wolf
Quoted 23 lines

Am 09.06.2020 um 13:46 hat Klaus Jensen geschrieben:

Quoted 7 lines · Philippe Mathieu-Daudé
Quoted 6 lines · Klaus Jensen
Quoted 4 lines

From: Klaus Jensen <[email protected]>

I goofed up with commit c09794fe40e3 ("hw/block/nvme: allow use of any valid msix vector").

Quoted 6 lines
Quoted 1 line

Kevin, since your queue isn't merged, can you directly squash the fix?

The commit (c09794fe40e3) can just be dropped without conflicts, but it leaves a use of n->params.num_queues in nvme_create_cq() which commit cde74bfd4b87 ("hw/block/nvme: add max_ioqpairs device parameter") must fix.

Hm, so it seems this isn't easy to squash in without conflicts (and I would have to rewrite the whole commit message), so I think it's better to just apply the series on top.

One problem with the commit message is that it references commit IDs which aren't stable yet. Maybe it's best if I apply these patches, manually fix up the commit ID references and then immediately do a pull request so that they become stable.

Quoted 9 lines
Quoted 5 lines

This is the friendlier way.

Less friendly way is to drop Klaus's patches and ask him to respin. While this is a valid outcome, if we can avoid it it will save all of us review time.

If Klaus wants to do that, fine with me. I'm just trying to find the easiest solution for all of us.

Sure, I can respin it. I would like to include this series as well though since I think it's a nice addition.

I'll post a v7 that includes Philippes's return value verification patch as well as the patches in this series. We should only need a review or two on those patches then.

#9 lore raw
In reply to Kevin Wolf
Quoted 35 lines · Kevin Wolf

Am 09.06.2020 um 16:18 hat Philippe Mathieu-Daudé geschrieben:

Quoted 25 lines · Kevin Wolf
Quoted 15 lines

Am 09.06.2020 um 13:46 hat Klaus Jensen geschrieben:

Quoted 7 lines · Philippe Mathieu-Daudé
Quoted 6 lines · Klaus Jensen
Quoted 4 lines

From: Klaus Jensen <[email protected]>

I goofed up with commit c09794fe40e3 ("hw/block/nvme: allow use of any valid msix vector").

Quoted 6 lines
Quoted 1 line

Kevin, since your queue isn't merged, can you directly squash the fix?

The commit (c09794fe40e3) can just be dropped without conflicts, but it leaves a use of n->params.num_queues in nvme_create_cq() which commit cde74bfd4b87 ("hw/block/nvme: add max_ioqpairs device parameter") must fix.

Quoted 8 lines

Hm, so it seems this isn't easy to squash in without conflicts (and I would have to rewrite the whole commit message), so I think it's better to just apply the series on top.

One problem with the commit message is that it references commit IDs which aren't stable yet. Maybe it's best if I apply these patches, manually fix up the commit ID references and then immediately do a pull request so that they become stable.

Quoted 5 lines

This is the friendlier way.

Less friendly way is to drop Klaus's patches and ask him to respin. While this is a valid outcome, if we can avoid it it will save all of us review time.

If Klaus wants to do that, fine with me. I'm just trying to find the easiest solution for all of us.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

It would be good to have at least one review, though.

Maxim catched this issue, I'd feel safer if he acks your pre-merge queue.

Ok. Maxim, can you please review this series then?

Kevin

I am slowly getting through the heap of the patches trying to understand the current state of things. I will start reviewing all these patches today.

Best regards,
Maxim Levitsky

#10 lore raw
Quoted 38 lines · Maxim Levitsky
Quoted 37 lines · Kevin Wolf

Am 09.06.2020 um 16:18 hat Philippe Mathieu-Daudé geschrieben:

Quoted 25 lines · Kevin Wolf
Quoted 15 lines

Am 09.06.2020 um 13:46 hat Klaus Jensen geschrieben:

Quoted 7 lines · Philippe Mathieu-Daudé
Quoted 6 lines · Klaus Jensen
Quoted 4 lines

From: Klaus Jensen <[email protected]>

I goofed up with commit c09794fe40e3 ("hw/block/nvme: allow use of any valid msix vector").

Quoted 6 lines
Quoted 1 line

Kevin, since your queue isn't merged, can you directly squash the fix?

The commit (c09794fe40e3) can just be dropped without conflicts, but it leaves a use of n->params.num_queues in nvme_create_cq() which commit cde74bfd4b87 ("hw/block/nvme: add max_ioqpairs device parameter") must fix.

Quoted 8 lines

Hm, so it seems this isn't easy to squash in without conflicts (and I would have to rewrite the whole commit message), so I think it's better to just apply the series on top.

One problem with the commit message is that it references commit IDs which aren't stable yet. Maybe it's best if I apply these patches, manually fix up the commit ID references and then immediately do a pull request so that they become stable.

Quoted 5 lines

This is the friendlier way.

Less friendly way is to drop Klaus's patches and ask him to respin. While this is a valid outcome, if we can avoid it it will save all of us review time.

If Klaus wants to do that, fine with me. I'm just trying to find the easiest solution for all of us.

Quoted 2 lines
Quoted 1 line

It would be good to have at least one review, though.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Maxim catched this issue, I'd feel safer if he acks your pre-merge queue.

Ok. Maxim, can you please review this series then?

Kevin

I am slowly getting through the heap of the patches trying to understand the current state of things. I will start reviewing all these patches today.

Hi Maxim,

Yeah, I bombed it again; sorry! ;)

"[PATCH v3 00/18] hw/block/nvme: bump to v1.3" is the series currently under review.

I also posted:

[PATCH 00/17] hw/block/nvme: AIO and address mapping refactoring, [PATCH 0/2] hw/block/nvme: handle transient dma errors [PATCH 0/3] hw/block/nvme: support scatter gather lists [PATCH 0/4] hw/block/nvme: support multiple namespaces [PATCH] hw/block/nvme: make lba data size configurable [PATCH] hw/block/nvme: add support for dulbe
[PATCH 0/3] hw/block/nvme: bump to v1.4
[PATCH 00/10] hw/block/nvme: namespace types and zoned namespaces

I really appreciate you reviewing! Your R-b's are on a lot of the patches already, thanks for that!