Skip to contentSkip to conversation

NVMe passthrough: Support 64kB page host

RFC In review Eric Auger 2 participants 16 messages Oct 15, 2020 – Oct 20, 2020 Block layer

lore
Share & tools
lore

Eric Auger

author
#0 lore raw

cover NVMe passthrough: Support 64kB page host

This series allows NVMe passthrough on aarch64 with 64kB page host. Addresses and sizes of buffers which are VFIO DMA mapped are aligned with the host page size.

nvme_register_buf() path is taken care of in this series but it does not seem to prevent the use case from working.

Best Regards

Eric

This series can be found at:
https://github.com/eauger/qemu/tree/nvme_64k_rfc

This was tested on ARM only.

Eric Auger (5):
block/nvme: use some NVME_CAP_* macros
block/nvme: Change size and alignment of IDENTIFY response buffer block/nvme: Change size and alignment of queue block/nvme: Change size and alignment of prp_list_pages block/nvme: Align iov's va and size on host page size

 block/nvme.c | 43 +++++++++++++++++++++++--------------------
 1 file changed, 23 insertions(+), 20 deletions(-)
Signature · 1 line
2.21.3
author
#1 lore raw

RFC 1/5 block/nvme: use some NVME_CAP_* macros

let's use NVME_CAP_DSTRD, NVME_CAP_MPSMIN and NVME_CAP_TO macros

---
 block/nvme.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
block/nvme.c +3 -3
index f4f27b6da7..e3d96f20d0 100644
@@ -728,10 +728,10 @@ static int nvme_init(BlockDriverState *bs, const char *device, int namespace,
         goto out;
     }
 
-    s->page_size = MAX(4096, 1 << (12 + ((cap >> 48) & 0xF)));
-    s->doorbell_scale = (4 << (((cap >> 32) & 0xF))) / sizeof(uint32_t);
+    s->page_size = MAX(4096, 1 << (12 + NVME_CAP_MPSMIN(cap)));
+    s->doorbell_scale = (4 << ((NVME_CAP_DSTRD(cap)))) / sizeof(uint32_t);
     bs->bl.opt_mem_alignment = s->page_size;
-    timeout_ms = MIN(500 * ((cap >> 24) & 0xFF), 30000);
+    timeout_ms = MIN(500 * NVME_CAP_TO(cap), 30000);
 
     /* Reset device to get a clean state. */
     s->regs->ctrl.cc = cpu_to_le32(le32_to_cpu(s->regs->ctrl.cc) & 0xFE);
Signature · 1 line
2.21.3
#2 lore raw

Re: [RFC 1/5] block/nvme: use some NVME_CAP_* macros

Quoted 12 lines · Eric Auger

let's use NVME_CAP_DSTRD, NVME_CAP_MPSMIN and NVME_CAP_TO macros

---
  block/nvme.c | 6 +++---
  1 file changed, 3 insertions(+), 3 deletions(-)
block/nvme.c +0 -0
index f4f27b6da7..e3d96f20d0 100644

block/nvme.c static int nvme_init(BlockDriverState *bs, const char *device, int namespace, · line 731

Quoted 6 lines
block/nvme.c +1 -2
@@ -728,10 +728,10 @@ static int nvme_init(BlockDriverState *bs, const char *device, int namespace,
          goto out;
      }
 
-    s->page_size = MAX(4096, 1 << (12 + ((cap >> 48) & 0xF)));
-    s->doorbell_scale = (4 << (((cap >> 32) & 0xF))) / sizeof(uint32_t);
+    s->page_size = MAX(4096, 1 << (12 + NVME_CAP_MPSMIN(cap)));

Are you suggesting commit fad1eb68862 ("block/nvme: Use register definitions from 'block/nvme.h'") is buggy?

Quoted 8 lines
block/nvme.c +2 -1
+    s->doorbell_scale = (4 << ((NVME_CAP_DSTRD(cap)))) / sizeof(uint32_t);
      bs->bl.opt_mem_alignment = s->page_size;
-    timeout_ms = MIN(500 * ((cap >> 24) & 0xFF), 30000);
+    timeout_ms = MIN(500 * NVME_CAP_TO(cap), 30000);
 
      /* Reset device to get a clean state. */
      s->regs->ctrl.cc = cpu_to_le32(le32_to_cpu(s->regs->ctrl.cc) & 0xFE);
#3 lore raw

Re: [RFC 1/5] block/nvme: use some NVME_CAP_* macros

Quoted 8 lines · Philippe Mathieu-Daudé
Quoted 7 lines · Eric Auger

let's use NVME_CAP_DSTRD, NVME_CAP_MPSMIN and NVME_CAP_TO macros

---
  block/nvme.c | 6 +++---

  1 file changed, 3 insertions(+), 3 deletions(-)
Quoted 14 lines

block/nvme.c static int nvme_init(BlockDriverState *bs, const · line 731

Quoted 11 lines
block/nvme.c +1 -2
index f4f27b6da7..e3d96f20d0 100644
@@ -728,10 +728,10 @@ static int nvme_init(BlockDriverState *bs, const 
 char *device, int namespace,
           goto out;
       }
-    s->page_size = MAX(4096, 1 << (12 + ((cap >> 48) & 0xF)));
-    s->doorbell_scale = (4 << (((cap >> 32) & 0xF))) / sizeof(uint32_t);
+    s->page_size = MAX(4096, 1 << (12 + NVME_CAP_MPSMIN(cap)));

Are you suggesting commit fad1eb68862 ("block/nvme: Use register definitions from 'block/nvme.h'") is buggy?

Buh I wonder how we missed that :/

Quoted 10 lines
Quoted 9 lines
block/nvme.c +2 -1
+    s->doorbell_scale = (4 << ((NVME_CAP_DSTRD(cap)))) / 
 sizeof(uint32_t);
       bs->bl.opt_mem_alignment = s->page_size;
-    timeout_ms = MIN(500 * ((cap >> 24) & 0xFF), 30000);
+    timeout_ms = MIN(500 * NVME_CAP_TO(cap), 30000);
       /* Reset device to get a clean state. */
       s->regs->ctrl.cc = cpu_to_le32(le32_to_cpu(s->regs->ctrl.cc) & 0xFE);
#4 lore raw

Re: [RFC 1/5] block/nvme: use some NVME_CAP_* macros

Quoted 25 lines · Philippe Mathieu-Daudé
Quoted 8 lines · Philippe Mathieu-Daudé
Quoted 7 lines · Eric Auger

let's use NVME_CAP_DSTRD, NVME_CAP_MPSMIN and NVME_CAP_TO macros

---
  block/nvme.c | 6 +++---

  1 file changed, 3 insertions(+), 3 deletions(-)
Quoted 15 lines

block/nvme.c static int nvme_init(BlockDriverState *bs, · line 732

Quoted 12 lines
block/nvme.c +1 -2
index f4f27b6da7..e3d96f20d0 100644
@@ -728,10 +728,10 @@ static int nvme_init(BlockDriverState *bs, 
 const char *device, int namespace,
           goto out;
       }
-    s->page_size = MAX(4096, 1 << (12 + ((cap >> 48) & 0xF)));
-    s->doorbell_scale = (4 << (((cap >> 32) & 0xF))) / 
 sizeof(uint32_t);
+    s->page_size = MAX(4096, 1 << (12 + NVME_CAP_MPSMIN(cap)));

Are you suggesting commit fad1eb68862 ("block/nvme: Use register definitions from 'block/nvme.h'") is buggy?

Quoted 1 line

Buh I wonder how we missed that :/

Since your patch doesn't apply anyway, I might fix as:

        s->page_size = 4096 << NVME_CAP_MPSMIN(cap);
Quoted 12 lines
Quoted 11 lines
Quoted 10 lines
block/nvme.c +2 -1
+    s->doorbell_scale = (4 << ((NVME_CAP_DSTRD(cap)))) / 
 sizeof(uint32_t);
       bs->bl.opt_mem_alignment = s->page_size;
-    timeout_ms = MIN(500 * ((cap >> 24) & 0xFF), 30000);
+    timeout_ms = MIN(500 * NVME_CAP_TO(cap), 30000);
       /* Reset device to get a clean state. */

      s->regs->ctrl.cc = cpu_to_le32(le32_to_cpu(s->regs->ctrl.cc) & 0xFE);

author
#5 lore raw

Re: [RFC 1/5] block/nvme: use some NVME_CAP_* macros

Hi Philippe,

Quoted 10 lines · Philippe Mathieu-Daudé
Quoted 9 lines · Philippe Mathieu-Daudé
Quoted 8 lines · Philippe Mathieu-Daudé
Quoted 7 lines · Eric Auger

let's use NVME_CAP_DSTRD, NVME_CAP_MPSMIN and NVME_CAP_TO macros

---
  block/nvme.c | 6 +++---

  1 file changed, 3 insertions(+), 3 deletions(-)
Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
block/nvme.c +0 -0
index f4f27b6da7..e3d96f20d0 100644
@@ -728,10 +728,10 @@ static int nvme_init(BlockDriverState *bs,
 const char *device, int namespace,
           goto out;
Quoted 8 lines
Quoted 8 lines

block/nvme.c static int nvme_init(BlockDriverState *bs, const char *device, int namespace, · line 731

Quoted 5 lines
block/nvme.c +1 -2
       }
-    s->page_size = MAX(4096, 1 << (12 + ((cap >> 48) & 0xF)));
-    s->doorbell_scale = (4 << (((cap >> 32) & 0xF))) /
 sizeof(uint32_t);
+    s->page_size = MAX(4096, 1 << (12 + NVME_CAP_MPSMIN(cap)));

Are you suggesting commit fad1eb68862 ("block/nvme: Use register definitions from 'block/nvme.h'") is buggy?

yes I think so. Sorry I messed up my rebase and failed to grab that patch.

Quoted 6 lines
Quoted 2 lines

Buh I wonder how we missed that :/

Since your patch doesn't apply anyway, I might fix as:

        s->page_size = 4096 << NVME_CAP_MPSMIN(cap);

1 << (12 + NVME_CAP_MPSMIN(cap)) matches the spec text so personally I would keep that.

Thanks

Eric

Quoted 16 lines
Quoted 14 lines
Quoted 13 lines
Quoted 12 lines
block/nvme.c +2 -1
+    s->doorbell_scale = (4 << ((NVME_CAP_DSTRD(cap)))) /
 sizeof(uint32_t);
       bs->bl.opt_mem_alignment = s->page_size;
-    timeout_ms = MIN(500 * ((cap >> 24) & 0xFF), 30000);
+    timeout_ms = MIN(500 * NVME_CAP_TO(cap), 30000);
       /* Reset device to get a clean state. */

      s->regs->ctrl.cc = cpu_to_le32(le32_to_cpu(s->regs->ctrl.cc) & 0xFE);

author
#6 lore raw

RFC 2/5 block/nvme: Change size and alignment of IDENTIFY response buffer

In reply to Eric Auger

In preparation of 64kB host page support, let's change the size and alignment of the IDENTIFY command response buffer so that the VFIO DMA MAP succeeds. We align on the host page size.

---
 block/nvme.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
block/nvme.c +6 -4
index e3d96f20d0..088ff1825a 100644
@@ -513,19 +513,21 @@ static void nvme_identify(BlockDriverState *bs, int namespace, Error **errp)
         .opcode = NVME_ADM_CMD_IDENTIFY,
         .cdw10 = cpu_to_le32(0x1),
     };
+    size_t id_size = QEMU_ALIGN_UP(sizeof(*id), qemu_real_host_page_size);
 
-    id = qemu_try_memalign(s->page_size, sizeof(*id));
+    id = qemu_try_memalign(qemu_real_host_page_size, id_size);
     if (!id) {
         error_setg(errp, "Cannot allocate buffer for identify response");
         goto out;
     }
-    r = qemu_vfio_dma_map(s->vfio, id, sizeof(*id), true, &iova);
+    r = qemu_vfio_dma_map(s->vfio, id, id_size, true, &iova);
     if (r) {
         error_setg(errp, "Cannot map buffer for DMA");
         goto out;
     }
 
-    memset(id, 0, sizeof(*id));
+    memset(id, 0, id_size);
+
     cmd.dptr.prp1 = cpu_to_le64(iova);
     if (nvme_cmd_sync(bs, s->queues[INDEX_ADMIN], &cmd)) {
         error_setg(errp, "Failed to identify controller");
@@ -547,7 +549,7 @@ static void nvme_identify(BlockDriverState *bs, int namespace, Error **errp)
     s->supports_write_zeroes = !!(oncs & NVME_ONCS_WRITE_ZEROES);
     s->supports_discard = !!(oncs & NVME_ONCS_DSM);
 
-    memset(id, 0, sizeof(*id));
+    memset(id, 0, id_size);
     cmd.cdw10 = 0;
     cmd.nsid = cpu_to_le32(namespace);
     if (nvme_cmd_sync(bs, s->queues[INDEX_ADMIN], &cmd)) {
Signature · 1 line
2.21.3
#7 lore raw

Re: [RFC 2/5] block/nvme: Change size and alignment of IDENTIFY response buffer

Quoted 14 lines · Eric Auger

In preparation of 64kB host page support, let's change the size and alignment of the IDENTIFY command response buffer so that the VFIO DMA MAP succeeds. We align on the host page size.

---
  block/nvme.c | 10 ++++++----
  1 file changed, 6 insertions(+), 4 deletions(-)
block/nvme.c +0 -0
index e3d96f20d0..088ff1825a 100644

block/nvme.c static void nvme_identify(BlockDriverState *bs, int namespace, Error **errp) · line 516

Quoted 4 lines
block/nvme.c +1 -0
@@ -513,19 +513,21 @@ static void nvme_identify(BlockDriverState *bs, int namespace, Error **errp)
          .opcode = NVME_ADM_CMD_IDENTIFY,
          .cdw10 = cpu_to_le32(0x1),
      };
+    size_t id_size = QEMU_ALIGN_UP(sizeof(*id), qemu_real_host_page_size);

Do we really need to ALIGN_UP if we then call qemu_try_memalign()?

Quoted 30 lines
block/nvme.c +5 -4
-    id = qemu_try_memalign(s->page_size, sizeof(*id));
+    id = qemu_try_memalign(qemu_real_host_page_size, id_size);
      if (!id) {
          error_setg(errp, "Cannot allocate buffer for identify response");
          goto out;
      }
-    r = qemu_vfio_dma_map(s->vfio, id, sizeof(*id), true, &iova);
+    r = qemu_vfio_dma_map(s->vfio, id, id_size, true, &iova);
      if (r) {
          error_setg(errp, "Cannot map buffer for DMA");
          goto out;
      }
 
-    memset(id, 0, sizeof(*id));
+    memset(id, 0, id_size);
+
      cmd.dptr.prp1 = cpu_to_le64(iova);
      if (nvme_cmd_sync(bs, s->queues[INDEX_ADMIN], &cmd)) {
          error_setg(errp, "Failed to identify controller");
@@ -547,7 +549,7 @@ static void nvme_identify(BlockDriverState *bs, int namespace, Error **errp)
      s->supports_write_zeroes = !!(oncs & NVME_ONCS_WRITE_ZEROES);
      s->supports_discard = !!(oncs & NVME_ONCS_DSM);
 
-    memset(id, 0, sizeof(*id));
+    memset(id, 0, id_size);
      cmd.cdw10 = 0;
      cmd.nsid = cpu_to_le32(namespace);
      if (nvme_cmd_sync(bs, s->queues[INDEX_ADMIN], &cmd)) {
#8 lore raw

Re: [RFC 2/5] block/nvme: Change size and alignment of IDENTIFY response buffer

Quoted 10 lines · Philippe Mathieu-Daudé
Quoted 9 lines · Eric Auger

In preparation of 64kB host page support, let's change the size and alignment of the IDENTIFY command response buffer so that the VFIO DMA MAP succeeds. We align on the host page size.

---
  block/nvme.c | 10 ++++++----

  1 file changed, 6 insertions(+), 4 deletions(-)
Quoted 13 lines

block/nvme.c static void nvme_identify(BlockDriverState *bs, · line 518

Quoted 11 lines
block/nvme.c +1 -0
index e3d96f20d0..088ff1825a 100644
@@ -513,19 +513,21 @@ static void nvme_identify(BlockDriverState *bs, 
 int namespace, Error **errp)
           .opcode = NVME_ADM_CMD_IDENTIFY,
           .cdw10 = cpu_to_le32(0x1),
       };
+    size_t id_size = QEMU_ALIGN_UP(sizeof(*id), 
 qemu_real_host_page_size);

Do we really need to ALIGN_UP if we then call qemu_try_memalign()?

Ah, we need to ALIGN_UP for qemu_vfio_dma_map(), OK.

Quoted 30 lines
Quoted 29 lines
block/nvme.c +5 -4
-    id = qemu_try_memalign(s->page_size, sizeof(*id));
+    id = qemu_try_memalign(qemu_real_host_page_size, id_size);
       if (!id) {
           error_setg(errp, "Cannot allocate buffer for identify response");
           goto out;
       }
-    r = qemu_vfio_dma_map(s->vfio, id, sizeof(*id), true, &iova);
+    r = qemu_vfio_dma_map(s->vfio, id, id_size, true, &iova);
       if (r) {
           error_setg(errp, "Cannot map buffer for DMA");
           goto out;
       }
-    memset(id, 0, sizeof(*id));
+    memset(id, 0, id_size);
+
       cmd.dptr.prp1 = cpu_to_le64(iova);
       if (nvme_cmd_sync(bs, s->queues[INDEX_ADMIN], &cmd)) {
           error_setg(errp, "Failed to identify controller");
@@ -547,7 +549,7 @@ static void nvme_identify(BlockDriverState *bs, 
 int namespace, Error **errp)
       s->supports_write_zeroes = !!(oncs & NVME_ONCS_WRITE_ZEROES);
       s->supports_discard = !!(oncs & NVME_ONCS_DSM);
-    memset(id, 0, sizeof(*id));
+    memset(id, 0, id_size);
       cmd.cdw10 = 0;
       cmd.nsid = cpu_to_le32(namespace);
       if (nvme_cmd_sync(bs, s->queues[INDEX_ADMIN], &cmd)) {
author
#9 lore raw

RFC 3/5 block/nvme: Change size and alignment of queue

In reply to Eric Auger

In preparation of 64kB host page support, let's change the size and alignment of the queue so that the VFIO DMA MAP succeeds. We align on the host page size.

---
 block/nvme.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
block/nvme.c +2 -2
index 088ff1825a..cf8ec200af 100644
@@ -164,9 +164,9 @@ static void nvme_init_queue(BDRVNVMeState *s, NVMeQueue *q,
     size_t bytes;
     int r;
 
-    bytes = ROUND_UP(nentries * entry_bytes, s->page_size);
+    bytes = ROUND_UP(nentries * entry_bytes, qemu_real_host_page_size);
     q->head = q->tail = 0;
-    q->queue = qemu_try_memalign(s->page_size, bytes);
+    q->queue = qemu_try_memalign(qemu_real_host_page_size, bytes);
     if (!q->queue) {
         error_setg(errp, "Cannot allocate queue");
         return;
Signature · 1 line
2.21.3
author
#10 lore raw

RFC 4/5 block/nvme: Change size and alignment of prp_list_pages

In reply to Eric Auger

In preparation of 64kB host page support, let's change the size and alignment of the prp_list_pages so that the VFIO DMA MAP succeeds with 64kB host page size. We align on the host page size.

---
 block/nvme.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
block/nvme.c +5 -5
index cf8ec200af..be8ec48bf2 100644
@@ -210,24 +210,24 @@ static NVMeQueuePair *nvme_create_queue_pair(BDRVNVMeState *s,
     Error *local_err = NULL;
     NVMeQueuePair *q;
     uint64_t prp_list_iova;
+    size_t bytes;
 
     q = g_try_new0(NVMeQueuePair, 1);
     if (!q) {
         return NULL;
     }
-    q->prp_list_pages = qemu_try_memalign(s->page_size,
-                                          s->page_size * NVME_NUM_REQS);
+    bytes = QEMU_ALIGN_UP(s->page_size * NVME_NUM_REQS, qemu_real_host_page_size);
+    q->prp_list_pages = qemu_try_memalign(qemu_real_host_page_size, bytes);
     if (!q->prp_list_pages) {
         goto fail;
     }
-    memset(q->prp_list_pages, 0, s->page_size * NVME_NUM_REQS);
+    memset(q->prp_list_pages, 0, bytes);
     qemu_mutex_init(&q->lock);
     q->s = s;
     q->index = idx;
     qemu_co_queue_init(&q->free_req_queue);
     q->completion_bh = aio_bh_new(aio_context, nvme_process_completion_bh, q);
-    r = qemu_vfio_dma_map(s->vfio, q->prp_list_pages,
-                          s->page_size * NVME_NUM_REQS,
+    r = qemu_vfio_dma_map(s->vfio, q->prp_list_pages, bytes,
                           false, &prp_list_iova);
     if (r) {
         goto fail;
Signature · 1 line
2.21.3
author
#11 lore raw

RFC 5/5 block/nvme: Align iov's va and size on host page size

In reply to Eric Auger

Make sure iov's va and size are properly aligned on the host page size.

---
 block/nvme.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
block/nvme.c +7 -6
index be8ec48bf2..45807ed110 100644
@@ -978,11 +978,12 @@ static coroutine_fn int nvme_cmd_map_qiov(BlockDriverState *bs, NvmeCmd *cmd,
     for (i = 0; i < qiov->niov; ++i) {
         bool retry = true;
         uint64_t iova;
+        size_t len = QEMU_ALIGN_UP(qiov->iov[i].iov_len,
+                                   qemu_real_host_page_size);
 try_map:
         r = qemu_vfio_dma_map(s->vfio,
                               qiov->iov[i].iov_base,
-                              qiov->iov[i].iov_len,
-                              true, &iova);
+                              len, true, &iova);
         if (r == -ENOMEM && retry) {
             retry = false;
             trace_nvme_dma_flush_queue_wait(s);
@@ -1126,8 +1127,8 @@ static inline bool nvme_qiov_aligned(BlockDriverState *bs,
     BDRVNVMeState *s = bs->opaque;
 
     for (i = 0; i < qiov->niov; ++i) {
-        if (!QEMU_PTR_IS_ALIGNED(qiov->iov[i].iov_base, s->page_size) ||
-            !QEMU_IS_ALIGNED(qiov->iov[i].iov_len, s->page_size)) {
+        if (!QEMU_PTR_IS_ALIGNED(qiov->iov[i].iov_base, qemu_real_host_page_size) ||
+            !QEMU_IS_ALIGNED(qiov->iov[i].iov_len, qemu_real_host_page_size)) {
             trace_nvme_qiov_unaligned(qiov, i, qiov->iov[i].iov_base,
                                       qiov->iov[i].iov_len, s->page_size);
             return false;
@@ -1143,7 +1144,7 @@ static int nvme_co_prw(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
     int r;
     uint8_t *buf = NULL;
     QEMUIOVector local_qiov;
-
+    size_t len = QEMU_ALIGN_UP(bytes, qemu_real_host_page_size);
     assert(QEMU_IS_ALIGNED(offset, s->page_size));
     assert(QEMU_IS_ALIGNED(bytes, s->page_size));
     assert(bytes <= s->max_transfer);
@@ -1151,7 +1152,7 @@ static int nvme_co_prw(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
         return nvme_co_prw_aligned(bs, offset, bytes, qiov, is_write, flags);
     }
     trace_nvme_prw_buffered(s, offset, bytes, qiov->niov, is_write);
-    buf = qemu_try_memalign(s->page_size, bytes);
+    buf = qemu_try_memalign(qemu_real_host_page_size, len);
 
     if (!buf) {
         return -ENOMEM;
Signature · 1 line
2.21.3
Reviewed-by on v1 5/5
#12 lore raw

Re: [RFC 5/5] block/nvme: Align iov's va and size on host page size

Quoted 4 lines · Eric Auger

Make sure iov's va and size are properly aligned on the host page size.

Quoted 53 lines
---
  block/nvme.c | 13 +++++++------
  1 file changed, 7 insertions(+), 6 deletions(-)
block/nvme.c +7 -6
index be8ec48bf2..45807ed110 100644
@@ -978,11 +978,12 @@ static coroutine_fn int nvme_cmd_map_qiov(BlockDriverState *bs, NvmeCmd *cmd,
      for (i = 0; i < qiov->niov; ++i) {
          bool retry = true;
          uint64_t iova;
+        size_t len = QEMU_ALIGN_UP(qiov->iov[i].iov_len,
+                                   qemu_real_host_page_size);
  try_map:
          r = qemu_vfio_dma_map(s->vfio,
                                qiov->iov[i].iov_base,
-                              qiov->iov[i].iov_len,
-                              true, &iova);
+                              len, true, &iova);
          if (r == -ENOMEM && retry) {
              retry = false;
              trace_nvme_dma_flush_queue_wait(s);
@@ -1126,8 +1127,8 @@ static inline bool nvme_qiov_aligned(BlockDriverState *bs,
      BDRVNVMeState *s = bs->opaque;
 
      for (i = 0; i < qiov->niov; ++i) {
-        if (!QEMU_PTR_IS_ALIGNED(qiov->iov[i].iov_base, s->page_size) ||
-            !QEMU_IS_ALIGNED(qiov->iov[i].iov_len, s->page_size)) {
+        if (!QEMU_PTR_IS_ALIGNED(qiov->iov[i].iov_base, qemu_real_host_page_size) ||
+            !QEMU_IS_ALIGNED(qiov->iov[i].iov_len, qemu_real_host_page_size)) {
              trace_nvme_qiov_unaligned(qiov, i, qiov->iov[i].iov_base,
                                        qiov->iov[i].iov_len, s->page_size);
              return false;
@@ -1143,7 +1144,7 @@ static int nvme_co_prw(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
      int r;
      uint8_t *buf = NULL;
      QEMUIOVector local_qiov;
-
+    size_t len = QEMU_ALIGN_UP(bytes, qemu_real_host_page_size);
      assert(QEMU_IS_ALIGNED(offset, s->page_size));
      assert(QEMU_IS_ALIGNED(bytes, s->page_size));
      assert(bytes <= s->max_transfer);
@@ -1151,7 +1152,7 @@ static int nvme_co_prw(BlockDriverState *bs, uint64_t offset, uint64_t bytes,
          return nvme_co_prw_aligned(bs, offset, bytes, qiov, is_write, flags);
      }
      trace_nvme_prw_buffered(s, offset, bytes, qiov->niov, is_write);
-    buf = qemu_try_memalign(s->page_size, bytes);
+    buf = qemu_try_memalign(qemu_real_host_page_size, len);
 
      if (!buf) {
          return -ENOMEM;
#13 lore raw
In reply to Eric Auger
Quoted 14 lines · Eric Auger

This series allows NVMe passthrough on aarch64 with 64kB page host. Addresses and sizes of buffers which are VFIO DMA mapped are aligned with the host page size.

nvme_register_buf() path is taken care of in this series but it does not seem to prevent the use case from working.

Best Regards

Eric

This series can be found at:
https://github.com/eauger/qemu/tree/nvme_64k_rfc

Quoted 8 lines

This was tested on ARM only.

Eric Auger (5):
block/nvme: use some NVME_CAP_* macros
block/nvme: Change size and alignment of IDENTIFY response buffer block/nvme: Change size and alignment of queue block/nvme: Change size and alignment of prp_list_pages block/nvme: Align iov's va and size on host page size

Since it is easier for me to rebase on top of your series, I'm including it in my work (fixing the checkpatch errors) and will repost block/nvme/ patches altogether.

Regards,

Phil.

author
#14 lore raw

Hi Philippe,

Quoted 24 lines · Philippe Mathieu-Daudé
Quoted 14 lines · Eric Auger

This series allows NVMe passthrough on aarch64 with 64kB page host. Addresses and sizes of buffers which are VFIO DMA mapped are aligned with the host page size.

nvme_register_buf() path is taken care of in this series but it does not seem to prevent the use case from working.

Best Regards

Eric

This series can be found at:
https://github.com/eauger/qemu/tree/nvme_64k_rfc

Quoted 8 lines

This was tested on ARM only.

Eric Auger (5):
   block/nvme: use some NVME_CAP_* macros
   block/nvme: Change size and alignment of IDENTIFY response buffer block/nvme: Change size and alignment of queue block/nvme: Change size and alignment of prp_list_pages block/nvme: Align iov's va and size on host page size

Quoted 3 lines

Since it is easier for me to rebase on top of your series, I'm including it in my work (fixing the checkpatch errors) and will repost block/nvme/ patches altogether.

There should be one warning (line exceeding 80 chars) but no error. I can easily rebase/respin if you prefer.

Thanks

Eric

Quoted 6 lines

Regards,

Phil.

#15 lore raw
Quoted 27 lines · Auger Eric

Hi Philippe,

Quoted 24 lines · Philippe Mathieu-Daudé
Quoted 22 lines · Eric Auger

This series allows NVMe passthrough on aarch64 with 64kB page host. Addresses and sizes of buffers which are VFIO DMA mapped are aligned with the host page size.

nvme_register_buf() path is taken care of in this series but it does not seem to prevent the use case from working.

Best Regards

Eric

This series can be found at:
https://github.com/eauger/qemu/tree/nvme_64k_rfc

This was tested on ARM only.

Eric Auger (5):
   block/nvme: use some NVME_CAP_* macros
   block/nvme: Change size and alignment of IDENTIFY response buffer block/nvme: Change size and alignment of queue block/nvme: Change size and alignment of prp_list_pages block/nvme: Align iov's va and size on host page size

Quoted 6 lines
Quoted 3 lines

Since it is easier for me to rebase on top of your series, I'm including it in my work (fixing the checkpatch errors) and will repost block/nvme/ patches altogether.

There should be one warning (line exceeding 80 chars) but no error. I can easily rebase/respin if you prefer.

Yes, warnings, no error. No need to respin.

Quoted 11 lines

Thanks

Eric

Quoted 6 lines

Regards,

Phil.