Skip to contentSkip to conversation

hw/nvme: shadow doorbells broken on riscv64

PATCH Superseded Klaus Jensen 2 participants 5 messages Dec 8, 2022

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

cover hw/nvme: shadow doorbells broken on riscv64

From: Klaus Jensen <[email protected]>

Guenter reports[1] that hw/nvme is broken on riscv64.

This is a regression since 7.1, so this does not warrent an rc5 for 7.2. I'm sure Guenter can carry this patch in his tree, and maybe we can get this out in a stable release.

I really wonder why this issue only shows up on riscv64. We have not observed this on other platforms (yet).

[1]: https://lore.kernel.org/qemu-devel/[email protected]/

Klaus Jensen (1):
hw/nvme: fix missing cq eventidx update

 hw/nvme/ctrl.c | 8 ++++++++
 1 file changed, 8 insertions(+)
Signature · 1 line
2.38.1
#1 lore raw

PATCH 1/1 hw/nvme: fix missing cq eventidx update

From: Klaus Jensen <[email protected]>

Prior to reading the shadow doorbell cq head, we have to update the eventidx. Otherwise, we risk that the driver will skip an mmio doorbell write. This happens on riscv64, as reported by Guenter.

Adding the missing update to the cq eventidx fixes the issue.

Fixes 3f7fe8de3d49 ("hw/nvme: Implement shadow doorbell buffer support")
---
 hw/nvme/ctrl.c | 8 ++++++++
 1 file changed, 8 insertions(+)
hw/nvme/ctrl.c +8 -0
index e54276dc1dc7..1192919b4869 100644
@@ -1331,6 +1331,13 @@ static inline void nvme_blk_write(BlockBackend *blk, int64_t offset,
     }
 }
 
+static void nvme_update_cq_eventidx(const NvmeCQueue *cq)
+{
+    pci_dma_write(&cq->ctrl->parent_obj, cq->ei_addr, &cq->head,
+                  sizeof(cq->head));
+    trace_pci_nvme_eventidx_cq(cq->cqid, cq->head);
+}
+
 static void nvme_update_cq_head(NvmeCQueue *cq)
 {
     pci_dma_read(&cq->ctrl->parent_obj, cq->db_addr, &cq->head,
@@ -1351,6 +1358,7 @@ static void nvme_post_cqes(void *opaque)
         hwaddr addr;
 
         if (n->dbbuf_enabled) {
+            nvme_update_cq_eventidx(cq);
             nvme_update_cq_head(cq);
         }
 
Signature · 1 line
2.38.1
#2 lore raw

Re: [PATCH 1/1] hw/nvme: fix missing cq eventidx update

Quoted 20 lines · Klaus Jensen

From: Klaus Jensen <[email protected]>

Prior to reading the shadow doorbell cq head, we have to update the eventidx. Otherwise, we risk that the driver will skip an mmio doorbell write. This happens on riscv64, as reported by Guenter.

Adding the missing update to the cq eventidx fixes the issue.

Fixes 3f7fe8de3d49 ("hw/nvme: Implement shadow doorbell buffer support")
---
  hw/nvme/ctrl.c | 8 ++++++++
  1 file changed, 8 insertions(+)
hw/nvme/ctrl.c +0 -0
index e54276dc1dc7..1192919b4869 100644

hw/nvme/ctrl.c static inline void nvme_blk_write(BlockBackend *blk, int64_t offset, · line 1336

Quoted 6 lines
hw/nvme/ctrl.c +3 -0
@@ -1331,6 +1331,13 @@ static inline void nvme_blk_write(BlockBackend *blk, int64_t offset,
      }
  }
 
+static void nvme_update_cq_eventidx(const NvmeCQueue *cq)
+{
+    pci_dma_write(&cq->ctrl->parent_obj, cq->ei_addr, &cq->head,

'parent_obj' is a private field. Better to use the QOM accessor:

       pci_dma_write(PCI_DEVICE(&cq->ctrl), cq->ei_addr, &cq->head,

hw/nvme/ctrl.c static inline void nvme_blk_write(BlockBackend *blk, int64_t offset, · line 1338

Quoted 2 lines
hw/nvme/ctrl.c +2 -0
+                  sizeof(cq->head));
+    trace_pci_nvme_eventidx_cq(cq->cqid, cq->head);

Surprisingly the trace event format was already present in Jinhao respin...
https://lore.kernel.org/all/YqsIh+OUcWnHU3zp@apples/T/

Could we move the event before the call?

Quoted 13 lines
hw/nvme/ctrl.c +3 -0
+}
+
  static void nvme_update_cq_head(NvmeCQueue *cq)
  {
      pci_dma_read(&cq->ctrl->parent_obj, cq->db_addr, &cq->head,
@@ -1351,6 +1358,7 @@ static void nvme_post_cqes(void *opaque)
          hwaddr addr;
 
          if (n->dbbuf_enabled) {
+            nvme_update_cq_eventidx(cq);
              nvme_update_cq_head(cq);
          }
 
#3 lore raw

Re: [PATCH 1/1] hw/nvme: fix missing cq eventidx update

Quoted 29 lines · Philippe Mathieu-Daudé
Quoted 20 lines · Klaus Jensen

From: Klaus Jensen <[email protected]>

Prior to reading the shadow doorbell cq head, we have to update the eventidx. Otherwise, we risk that the driver will skip an mmio doorbell write. This happens on riscv64, as reported by Guenter.

Adding the missing update to the cq eventidx fixes the issue.

Fixes 3f7fe8de3d49 ("hw/nvme: Implement shadow doorbell buffer support")
---
  hw/nvme/ctrl.c | 8 ++++++++
  1 file changed, 8 insertions(+)
hw/nvme/ctrl.c +0 -0
index e54276dc1dc7..1192919b4869 100644

hw/nvme/ctrl.c static inline void nvme_blk_write(BlockBackend *blk, int64_t offset, · line 1335

Quoted 5 lines
hw/nvme/ctrl.c +3 -0
@@ -1331,6 +1331,13 @@ static inline void nvme_blk_write(BlockBackend *blk, int64_t offset,
      }
  }
+static void nvme_update_cq_eventidx(const NvmeCQueue *cq)
+{
+    pci_dma_write(&cq->ctrl->parent_obj, cq->ei_addr, &cq->head,

'parent_obj' is a private field. Better to use the QOM accessor:

Quoted 1 line
       pci_dma_write(PCI_DEVICE(&cq->ctrl), cq->ei_addr, &cq->head,

Ah yes, of course. I think we have a couple of other ->parent_obj accesses that we should fix also.

Quoted 9 lines

hw/nvme/ctrl.c static inline void nvme_blk_write(BlockBackend *blk, int64_t offset, · line 1338

Quoted 3 lines
hw/nvme/ctrl.c +2 -0
+                  sizeof(cq->head));
+    trace_pci_nvme_eventidx_cq(cq->cqid, cq->head);

Surprisingly the trace event format was already present in Jinhao respin...
https://lore.kernel.org/all/YqsIh+OUcWnHU3zp@apples/T/

Could we move the event before the call?

Makes sense.

Quoted 12 lines
Quoted 11 lines
hw/nvme/ctrl.c +3 -0
+}
+
  static void nvme_update_cq_head(NvmeCQueue *cq)
  {
      pci_dma_read(&cq->ctrl->parent_obj, cq->db_addr, &cq->head,
@@ -1351,6 +1358,7 @@ static void nvme_post_cqes(void *opaque)
          hwaddr addr;
          if (n->dbbuf_enabled) {
+            nvme_update_cq_eventidx(cq);
              nvme_update_cq_head(cq);
          }
#4 lore raw
In reply to Klaus Jensen
Quoted 7 lines · Klaus Jensen

From: Klaus Jensen <[email protected]>

Guenter reports[1] that hw/nvme is broken on riscv64.

This is a regression since 7.1, so this does not warrent an rc5 for 7.2. I'm sure Guenter can carry this patch in his tree, and maybe we can get this out in a stable release.

Delaying to 8.0 and adding a 'Cc: qemu-stable@' tag seems wise.

Cc'ing riscv list in case.

Quoted 11 lines

I really wonder why this issue only shows up on riscv64. We have not observed this on other platforms (yet).

   [1]: https://lore.kernel.org/qemu-devel/[email protected]/

Klaus Jensen (1):
hw/nvme: fix missing cq eventidx update

  hw/nvme/ctrl.c | 8 ++++++++
  1 file changed, 8 insertions(+)