hw/block/nvme: fix assert on invalid irq vector
cover hw/block/nvme: fix assert on invalid irq vector
From: Klaus Jensen <[email protected]>
I goofed up with commit c09794fe40e3 ("hw/block/nvme: allow use of any valid msix vector").
This fixes the goof by adding a new msix_qsize parameter. As a nice side-effect this allows a device with less interrupt vectors available than supported queues. Also, improve the error handling in nvme_init_pci().
Kevin, please consider picking this up for the block branch when reviewed.
Klaus Jensen (2):
hw/block/nvme: add msix_qsize parameter hw/block/nvme: verify msix_init_exclusive_bar() return value
hw/block/nvme.c | 28 ++++++++++++++++++++++------ hw/block/nvme.h | 1 + 2 files changed, 23 insertions(+), 6 deletions(-)
Signature · 1 line
2.27.0
PATCH 1/2 hw/block/nvme: add msix_qsize parameter
From: Klaus Jensen <[email protected]>
Commit 4eb9f5217a16 ("hw/block/nvme: allow use of any valid msix vector") erronously allows any valid MSI-X vector to be used when creating completion queues. This is bad because MSI-X is initialized with max_ioqpairs + 1 vectors, which defaults to 64 + 1. And since commit ccd579b2ed03 ("hw/block/nvme: Verify msix_vector_use() returned value"), this also causes an assert if the host creates a completion queue with an invalid vector.
Fix this by decoupling the requested maximum number of ioqpairs (param max_ioqpairs) from the number of MSI-X interrupt vectors by introducing a new msix_qsize parameter and initialize MSI-X with that. This allows emulating a device that has fewer vectors than I/O queue pairs and also allows more than 2048 queue pairs. To get the same default behaviour as previously, msix_qsize defaults to 65 (max_ioqpairs default + 1).
This decoupling was actually suggested by Maxim some time ago in a slightly different context, so adding a Suggested-by.
--- hw/block/nvme.c | 17 +++++++++++++---- hw/block/nvme.h | 1 + 2 files changed, 14 insertions(+), 4 deletions(-)
hw/block/nvme.c +13 -4
@@ -54,6 +54,7 @@#include "trace.h" #include "nvme.h" +#define NVME_MAX_IOQPAIRS 0xffff#define NVME_REG_SIZE 0x1000#define NVME_DB_SIZE 4#define NVME_CMB_BIR 2
@@ -662,7 +663,7 @@ static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeCmd *cmd)trace_pci_nvme_err_invalid_create_cq_vector (vector);return NVME_INVALID_IRQ_VECTOR |NVME_DNR ; } -if (unlikely (vector >PCI_MSIX_FLAGS_QSIZE )) { +if (unlikely (vector >= n->params.msix_qsize)) {trace_pci_nvme_err_invalid_create_cq_vector (vector);return NVME_INVALID_IRQ_VECTOR |NVME_DNR ; }
@@ -1371,9 +1372,16 @@ static void nvme_check_constraints(NvmeCtrl *n, Error **errp) }if (params->max_ioqpairs <1 || - params->max_ioqpairs >PCI_MSIX_FLAGS_QSIZE ) { + params->max_ioqpairs >NVME_MAX_IOQPAIRS ) {error_setg (errp,"max_ioqpairs must be between 1 and %d" , -PCI_MSIX_FLAGS_QSIZE ); +NVME_MAX_IOQPAIRS ); +return ; + } + +if (params->msix_qsize <1 || + params->msix_qsize >PCI_MSIX_FLAGS_QSIZE +1 ) { +error_setg (errp,"msix_qsize must be between 1 and %d" , +PCI_MSIX_FLAGS_QSIZE +1 );return ; }
@@ -1527,7 +1535,7 @@ static void nvme_init_pci(NvmeCtrl *n, PCIDevice *pci_dev) n->reg_size);pci_register_bar (pci_dev,0 ,PCI_BASE_ADDRESS_SPACE_MEMORY |PCI_BASE_ADDRESS_MEM_TYPE_64 , &n->iomem); -msix_init_exclusive_bar (pci_dev, n->params.max_ioqpairs +1 ,4 ,NULL ); +msix_init_exclusive_bar (pci_dev, n->params.msix_qsize,4 ,NULL );if (n->params.cmb_size_mb) {nvme_init_cmb (n, pci_dev);
@@ -1634,6 +1642,7 @@ static Property nvme_props[] = {DEFINE_PROP_UINT32 ("cmb_size_mb" , NvmeCtrl, params.cmb_size_mb,0 ),DEFINE_PROP_UINT32 ("num_queues" , NvmeCtrl, params.num_queues,0 ),DEFINE_PROP_UINT32 ("max_ioqpairs" , NvmeCtrl, params.max_ioqpairs,64 ), +DEFINE_PROP_UINT16 ("msix_qsize" , NvmeCtrl, params.msix_qsize,65 ), DEFINE_PROP_END_OF_LIST(), };
hw/block/nvme.h +1 -0
@@ -7,6 +7,7 @@ typedef struct NvmeParams {char *serial;uint32_t num_queues;/* deprecated since 5.1 */ uint32_t max_ioqpairs; +uint16_t msix_qsize;uint32_t cmb_size_mb; } NvmeParams;
Signature · 1 line
2.27.0
PATCH 2/2 hw/block/nvme: verify msix_init_exclusive_bar() return value
In reply to Klaus JensenFrom: Klaus Jensen <[email protected]>
Pass an Error to msix_init_exclusive_bar() and check it.
--- hw/block/nvme.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)
hw/block/nvme.c +10 -3
@@ -1522,7 +1522,7 @@ static void nvme_init_pmr(NvmeCtrl *n, PCIDevice *pci_dev) PCI_BASE_ADDRESS_MEM_PREFETCH, &n->pmrdev->mr); } -static void nvme_init_pci (NvmeCtrl *n,PCIDevice *pci_dev) +static void nvme_init_pci (NvmeCtrl *n,PCIDevice *pci_dev,Error **errp) {uint8_t *pci_conf = pci_dev->config;
@@ -1535,7 +1535,9 @@ static void nvme_init_pci(NvmeCtrl *n, PCIDevice *pci_dev) n->reg_size);pci_register_bar (pci_dev,0 ,PCI_BASE_ADDRESS_SPACE_MEMORY |PCI_BASE_ADDRESS_MEM_TYPE_64 , &n->iomem); -msix_init_exclusive_bar (pci_dev, n->params.msix_qsize,4 ,NULL ); +if (msix_init_exclusive_bar (pci_dev, n->params.msix_qsize,4 , errp)) { +return ; + }if (n->params.cmb_size_mb) {nvme_init_cmb (n, pci_dev);
@@ -1603,7 +1605,12 @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)return ; } -nvme_init_pci (n, pci_dev); +nvme_init_pci (n, pci_dev, &local_err); +if (local_err) { +error_propagate (errp, local_err); +return ; + } +nvme_init_ctrl (n, pci_dev);for (i =0 ; i < n->num_namespaces; i++) {
Signature · 1 line
2.27.0
Quoted 4 lines · Klaus Jensen
From: Klaus Jensen <[email protected]>
I goofed up with commit
c09794fe40e3("hw/block/nvme: allow use of any valid msix vector").
Kevin, since your queue isn't merged, can you directly squash the fix?
Quoted 26 lines
This fixes the goof by adding a new
msix_qsizeparameter. As a nice side-effect this allows a device with less interrupt vectors available than supported queues. Also, improve the error handling innvme_init_pci().Kevin, please consider picking this up for the block branch when reviewed.
Cc Keith Busch <[email protected]>Cc Max Reitz <[email protected]>Cc Kevin Wolf <[email protected]>Cc Klaus Jensen <[email protected]>Cc Javier Gonzalez <[email protected]>Cc Maxim Levitsky <[email protected]>Cc Philippe Mathieu-Daudé <[email protected]>Klaus Jensen (2):
hw/block/nvme: addmsix_qsizeparameter hw/block/nvme: verifymsix_init_exclusive_bar()return valuehw/block/nvme.c | 28 ++++++++++++++++++++++------ hw/block/nvme.h | 1 + 2 files changed, 23 insertions(+), 6 deletions(-)
Quoted 7 lines · Philippe Mathieu-Daudé
Quoted 4 lines · Klaus Jensen
From: Klaus Jensen <[email protected]>
I goofed up with commit
c09794fe40e3("hw/block/nvme: allow use of any valid msix vector").Kevin, since your queue isn't merged, can you directly squash the fix?
The commit (c09794fe40e3) can just be dropped without conflicts, but it leaves a use of n->params.num_queues in nvme_create_cq() which commit cde74bfd4b87 ("hw/block/nvme: add max_ioqpairs device parameter") must fix.
Am 09.06.2020 um 13:46 hat Klaus Jensen geschrieben:
Quoted 7 lines · Philippe Mathieu-Daudé
Quoted 6 lines · Klaus Jensen
Quoted 4 lines
From: Klaus Jensen <[email protected]>
I goofed up with commit
c09794fe40e3("hw/block/nvme: allow use of any valid msix vector").
Quoted 6 lines
Quoted 1 line
Kevin, since your queue isn't merged, can you directly squash the fix?
The commit (
c09794fe40e3) can just be dropped without conflicts, but it leaves a use ofn->params.num_queuesinnvme_create_cq()which commitcde74bfd4b87("hw/block/nvme: addmax_ioqpairsdevice parameter") must fix.
Hm, so it seems this isn't easy to squash in without conflicts (and I would have to rewrite the whole commit message), so I think it's better to just apply the series on top.
One problem with the commit message is that it references commit IDs which aren't stable yet. Maybe it's best if I apply these patches, manually fix up the commit ID references and then immediately do a pull request so that they become stable.
It would be good to have at least one review, though.
Kevin
Quoted 15 lines · Kevin Wolf
Am 09.06.2020 um 13:46 hat Klaus Jensen geschrieben:
Quoted 7 lines · Philippe Mathieu-Daudé
Quoted 6 lines · Klaus Jensen
Quoted 4 lines
From: Klaus Jensen <[email protected]>
I goofed up with commit
c09794fe40e3("hw/block/nvme: allow use of any valid msix vector").Quoted 6 lines
Quoted 1 line
Kevin, since your queue isn't merged, can you directly squash the fix?
The commit (
c09794fe40e3) can just be dropped without conflicts, but it leaves a use ofn->params.num_queuesinnvme_create_cq()which commitcde74bfd4b87("hw/block/nvme: addmax_ioqpairsdevice parameter") must fix.
Quoted 8 lines
Hm, so it seems this isn't easy to squash in without conflicts (and I would have to rewrite the whole commit message), so I think it's better to just apply the series on top.
One problem with the commit message is that it references commit IDs which aren't stable yet. Maybe it's best if I apply these patches, manually fix up the commit ID references and then immediately do a pull request so that they become stable.
This is the friendlier way.
Less friendly way is to drop Klaus's patches and ask him to respin. While this is a valid outcome, if we can avoid it it will save all of us review time.
Quoted 2 lines
It would be good to have at least one review, though.
Maxim catched this issue, I'd feel safer if he acks your pre-merge queue.
Quoted 3 lines
Kevin
Am 09.06.2020 um 16:18 hat Philippe Mathieu-Daudé geschrieben:
Quoted 25 lines · Kevin Wolf
Quoted 15 lines
Am 09.06.2020 um 13:46 hat Klaus Jensen geschrieben:
Quoted 7 lines · Philippe Mathieu-Daudé
Quoted 6 lines · Klaus Jensen
Quoted 4 lines
From: Klaus Jensen <[email protected]>
I goofed up with commit
c09794fe40e3("hw/block/nvme: allow use of any valid msix vector").Quoted 6 lines
Quoted 1 line
Kevin, since your queue isn't merged, can you directly squash the fix?
The commit (
c09794fe40e3) can just be dropped without conflicts, but it leaves a use ofn->params.num_queuesinnvme_create_cq()which commitcde74bfd4b87("hw/block/nvme: addmax_ioqpairsdevice parameter") must fix.Quoted 8 lines
Hm, so it seems this isn't easy to squash in without conflicts (and I would have to rewrite the whole commit message), so I think it's better to just apply the series on top.
One problem with the commit message is that it references commit IDs which aren't stable yet. Maybe it's best if I apply these patches, manually fix up the commit ID references and then immediately do a pull request so that they become stable.
Quoted 5 lines
This is the friendlier way.
Less friendly way is to drop Klaus's patches and ask him to respin. While this is a valid outcome, if we can avoid it it will save all of us review time.
If Klaus wants to do that, fine with me. I'm just trying to find the easiest solution for all of us.
Quoted 3 lines
Quoted 1 line
It would be good to have at least one review, though.
Maxim catched this issue, I'd feel safer if he acks your pre-merge queue.
Ok. Maxim, can you please review this series then?
Kevin
Quoted 26 lines · Kevin Wolf
Am 09.06.2020 um 16:18 hat Philippe Mathieu-Daudé geschrieben:
Quoted 25 lines · Kevin Wolf
Quoted 23 lines
Am 09.06.2020 um 13:46 hat Klaus Jensen geschrieben:
Quoted 7 lines · Philippe Mathieu-Daudé
Quoted 6 lines · Klaus Jensen
Quoted 4 lines
From: Klaus Jensen <[email protected]>
I goofed up with commit
c09794fe40e3("hw/block/nvme: allow use of any valid msix vector").Quoted 6 lines
Quoted 1 line
Kevin, since your queue isn't merged, can you directly squash the fix?
The commit (
c09794fe40e3) can just be dropped without conflicts, but it leaves a use ofn->params.num_queuesinnvme_create_cq()which commitcde74bfd4b87("hw/block/nvme: addmax_ioqpairsdevice parameter") must fix.Hm, so it seems this isn't easy to squash in without conflicts (and I would have to rewrite the whole commit message), so I think it's better to just apply the series on top.
One problem with the commit message is that it references commit IDs which aren't stable yet. Maybe it's best if I apply these patches, manually fix up the commit ID references and then immediately do a pull request so that they become stable.
Quoted 9 lines
Quoted 5 lines
This is the friendlier way.
Less friendly way is to drop Klaus's patches and ask him to respin. While this is a valid outcome, if we can avoid it it will save all of us review time.
If Klaus wants to do that, fine with me. I'm just trying to find the easiest solution for all of us.
Sure, I can respin it. I would like to include this series as well though since I think it's a nice addition.
I'll post a v7 that includes Philippes's return value verification patch as well as the patches in this series. We should only need a review or two on those patches then.
Quoted 35 lines · Kevin Wolf
Am 09.06.2020 um 16:18 hat Philippe Mathieu-Daudé geschrieben:
Quoted 25 lines · Kevin Wolf
Quoted 15 lines
Am 09.06.2020 um 13:46 hat Klaus Jensen geschrieben:
Quoted 7 lines · Philippe Mathieu-Daudé
Quoted 6 lines · Klaus Jensen
Quoted 4 lines
From: Klaus Jensen <[email protected]>
I goofed up with commit
c09794fe40e3("hw/block/nvme: allow use of any valid msix vector").Quoted 6 lines
Quoted 1 line
Kevin, since your queue isn't merged, can you directly squash the fix?
The commit (
c09794fe40e3) can just be dropped without conflicts, but it leaves a use ofn->params.num_queuesinnvme_create_cq()which commitcde74bfd4b87("hw/block/nvme: addmax_ioqpairsdevice parameter") must fix.Quoted 8 lines
Hm, so it seems this isn't easy to squash in without conflicts (and I would have to rewrite the whole commit message), so I think it's better to just apply the series on top.
One problem with the commit message is that it references commit IDs which aren't stable yet. Maybe it's best if I apply these patches, manually fix up the commit ID references and then immediately do a pull request so that they become stable.
Quoted 5 lines
This is the friendlier way.
Less friendly way is to drop Klaus's patches and ask him to respin. While this is a valid outcome, if we can avoid it it will save all of us review time.
If Klaus wants to do that, fine with me. I'm just trying to find the easiest solution for all of us.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
It would be good to have at least one review, though.
Maxim catched this issue, I'd feel safer if he acks your pre-merge queue.
Ok. Maxim, can you please review this series then?
Kevin
I am slowly getting through the heap of the patches trying to understand the current state of things. I will start reviewing all these patches today.
Best regards,
Maxim Levitsky
Quoted 38 lines · Maxim Levitsky
Quoted 37 lines · Kevin Wolf
Am 09.06.2020 um 16:18 hat Philippe Mathieu-Daudé geschrieben:
Quoted 25 lines · Kevin Wolf
Quoted 15 lines
Am 09.06.2020 um 13:46 hat Klaus Jensen geschrieben:
Quoted 7 lines · Philippe Mathieu-Daudé
Quoted 6 lines · Klaus Jensen
Quoted 4 lines
From: Klaus Jensen <[email protected]>
I goofed up with commit
c09794fe40e3("hw/block/nvme: allow use of any valid msix vector").Quoted 6 lines
Quoted 1 line
Kevin, since your queue isn't merged, can you directly squash the fix?
The commit (
c09794fe40e3) can just be dropped without conflicts, but it leaves a use ofn->params.num_queuesinnvme_create_cq()which commitcde74bfd4b87("hw/block/nvme: addmax_ioqpairsdevice parameter") must fix.Quoted 8 lines
Hm, so it seems this isn't easy to squash in without conflicts (and I would have to rewrite the whole commit message), so I think it's better to just apply the series on top.
One problem with the commit message is that it references commit IDs which aren't stable yet. Maybe it's best if I apply these patches, manually fix up the commit ID references and then immediately do a pull request so that they become stable.
Quoted 5 lines
This is the friendlier way.
Less friendly way is to drop Klaus's patches and ask him to respin. While this is a valid outcome, if we can avoid it it will save all of us review time.
If Klaus wants to do that, fine with me. I'm just trying to find the easiest solution for all of us.
Quoted 2 lines
Quoted 1 line
It would be good to have at least one review, though.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Maxim catched this issue, I'd feel safer if he acks your pre-merge queue.
Ok. Maxim, can you please review this series then?
Kevin
I am slowly getting through the heap of the patches trying to understand the current state of things. I will start reviewing all these patches today.
Hi Maxim,
Yeah, I bombed it again; sorry! ;)
"[PATCH v3 00/18] hw/block/nvme: bump to v1.3" is the series currently under review.
I also posted:
[PATCH 00/17] hw/block/nvme: AIO and address mapping refactoring, [PATCH 0/2] hw/block/nvme: handle transient dma errors [PATCH 0/3] hw/block/nvme: support scatter gather lists [PATCH 0/4] hw/block/nvme: support multiple namespaces [PATCH] hw/block/nvme: make lba data size configurable [PATCH] hw/block/nvme: add support for dulbe
[PATCH 0/3] hw/block/nvme: bump to v1.4
[PATCH 00/10] hw/block/nvme: namespace types and zoned namespaces
I really appreciate you reviewing! Your R-b's are on a lot of the patches already, thanks for that!