Skip to contentSkip to conversation

ASID support in vhost-vdpa net

PATCH Superseded Eugenio Pérez 3 participants 46 messages Nov 8, 2022 – Nov 16, 2022

Subsystems (2)

v6 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v6 cover ASID support in vhost-vdpa net

Control VQ is the way net devices use to send changes to the device state, like the number of active queues or its mac address.

QEMU needs to intercept this queue so it can track these changes and is able to migrate the device. It can do it from 1576dbb5bbc4 ("vdpa: Add x-svq to NetdevVhostVDPAOptions"). However, to enable x-svq implies to shadow all VirtIO device's virtqueues, which will damage performance.

This series adds address space isolation, so the device and the guest communicate directly with them (passthrough) and CVQ communication is split in
two: The guest communicates with QEMU and QEMU forwards the commands to the device.

Comments are welcome. Thanks!

v6:
- Do not allocate SVQ resources like file descriptors if SVQ cannot be used.
- Disable shadow CVQ if the device does not support it because of net features.

v5:
- Move vring state in vhost_vdpa_get_vring_group instead of using a parameter.
- Rename VHOST_VDPA_NET_CVQ_PASSTHROUGH to VHOST_VDPA_NET_DATA_ASID

v4:
- Rebased on last CVQ start series, that allocated CVQ cmd bufs at load
- Squash vhost_vdpa_cvq_group_is_independent.
- Do not check for cvq index on vhost_vdpa_net_prepare, we only have one that callback registered in that NetClientInfo.
- Add comment specifying behavior if device does not support _F_ASID
- Update headers to a later Linux commit to not to remove SETUP_RNG_SEED

v3:
- Do not return an error but just print a warning if vdpa device initialization returns failure while getting AS num of VQ groups
- Delete extra newline

v2:
- Much as commented on series [1], handle vhost_net backend through NetClientInfo callbacks instead of directly.
- Fix not freeing SVQ properly when device does not support CVQ
- Add BIT_ULL missed checking device's backend feature for _F_ASID.

Eugenio Pérez (10):
vdpa: Use v->shadow_vqs_enabled in vhost_vdpa_svqs_start & stop
vhost: set SVQ device call handler at SVQ start
vhost: Allocate SVQ device file descriptors at device start
vdpa: add vhost_vdpa_net_valid_svq_features
vdpa: move SVQ vring features check to net/
vdpa: Allocate SVQ unconditionally
vdpa: Add asid parameter to vhost_vdpa_dma_map/unmap
vdpa: Store x-svq parameter in VhostVDPAState
vdpa: Add listener_shadow_vq to vhost_vdpa
vdpa: Always start CVQ in SVQ mode

 include/hw/virtio/vhost-vdpa.h     |  10 +-
 hw/virtio/vhost-shadow-virtqueue.c |  35 +-----
 hw/virtio/vhost-vdpa.c             | 114 ++++++++++---------
 net/vhost-vdpa.c                   | 171 ++++++++++++++++++++++++++---
 hw/virtio/trace-events             |   4 +-
 5 files changed, 222 insertions(+), 112 deletions(-)
Signature · 1 line
2.31.1
#1 lore raw

PATCH v6 1/10 vdpa: Use v->shadow_vqs_enabled in vhost_vdpa_svqs_start & stop

This function used to trust in v->shadow_vqs != NULL to know if it must start svq or not.

This is not going to be valid anymore, as qemu is going to allocate svq unconditionally (but it will only start them conditionally).

---
 hw/virtio/vhost-vdpa.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
hw/virtio/vhost-vdpa.c +2 -2
index 7468e44b87..7f0ff4df5b 100644
@@ -1029,7 +1029,7 @@ static bool vhost_vdpa_svqs_start(struct vhost_dev *dev)
     Error *err = NULL;
     unsigned i;
 
-    if (!v->shadow_vqs) {
+    if (!v->shadow_vqs_enabled) {
         return true;
     }
 
@@ -1082,7 +1082,7 @@ static void vhost_vdpa_svqs_stop(struct vhost_dev *dev)
 {
     struct vhost_vdpa *v = dev->opaque;
 
-    if (!v->shadow_vqs) {
+    if (!v->shadow_vqs_enabled) {
         return;
     }
 
Signature · 1 line
2.31.1
#2 lore raw

Re: [PATCH v6 01/10] vdpa: Use v->shadow_vqs_enabled in vhost_vdpa_svqs_start & stop

Quoted 6 lines · Eugenio Pérez

This function used to trust in v->shadow_vqs != NULL to know if it must start svq or not.

This is not going to be valid anymore, as qemu is going to allocate svq unconditionally (but it will only start them conditionally).

It might be a waste of memory if we did this. Any reason for this?

Thanks

Quoted 31 lines
---
 hw/virtio/vhost-vdpa.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
hw/virtio/vhost-vdpa.c +2 -3
index 7468e44b87..7f0ff4df5b 100644
@@ -1029,7 +1029,7 @@ static bool vhost_vdpa_svqs_start(struct vhost_dev *dev)
     Error *err = NULL;
     unsigned i;
 
-    if (!v->shadow_vqs) {
+    if (!v->shadow_vqs_enabled) {
         return true;
     }
 
@@ -1082,7 +1082,7 @@ static void vhost_vdpa_svqs_stop(struct vhost_dev *dev)
 {
     struct vhost_vdpa *v = dev->opaque;
 
-    if (!v->shadow_vqs) {
+    if (!v->shadow_vqs_enabled) {
         return;
     }
 
--

2.31.1

#3 lore raw

Re: [PATCH v6 01/10] vdpa: Use v->shadow_vqs_enabled in vhost_vdpa_svqs_start & stop

Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez

This function used to trust in v->shadow_vqs != NULL to know if it must start svq or not.

This is not going to be valid anymore, as qemu is going to allocate svq unconditionally (but it will only start them conditionally).

It might be a waste of memory if we did this. Any reason for this?

Well, it's modelled after vhost_vdpa notifier member [1].

But sure we can reduce the memory usage if SVQ is not used. The first function that needs it is vhost_set_vring_kick. But I think it is not a good function to place the delayed allocation.

Would it work to move the allocation to vhost_set_features vhost op? It seems unlikely to me to call callbacks that can affect SVQ earlier than that one. Or maybe to create a new one and call it the first on vhost.c:vhost_dev_start?

Thanks!

[1] The notifier member already allocates VIRTIO_QUEUE_MAX VhostVDPAHostNotifier for each vhost_vdpa, It is easy to reduce at least to the number of virtqueues on a vhost_vdpa. Should I send a patch for this one?

Quoted 34 lines

Thanks

Quoted 31 lines
---
 hw/virtio/vhost-vdpa.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
hw/virtio/vhost-vdpa.c +2 -3
index 7468e44b87..7f0ff4df5b 100644
@@ -1029,7 +1029,7 @@ static bool vhost_vdpa_svqs_start(struct vhost_dev *dev)
     Error *err = NULL;
     unsigned i;
 
-    if (!v->shadow_vqs) {
+    if (!v->shadow_vqs_enabled) {
         return true;
     }
 
@@ -1082,7 +1082,7 @@ static void vhost_vdpa_svqs_stop(struct vhost_dev *dev)
 {
     struct vhost_vdpa *v = dev->opaque;
 
-    if (!v->shadow_vqs) {
+    if (!v->shadow_vqs_enabled) {
         return;
     }
 
--

2.31.1

#4 lore raw

Re: [PATCH v6 01/10] vdpa: Use v->shadow_vqs_enabled in vhost_vdpa_svqs_start & stop

Quoted 10 lines · Eugenio Perez Martin
Quoted 8 lines · Jason Wang
Quoted 5 lines · Eugenio Pérez

This function used to trust in v->shadow_vqs != NULL to know if it must start svq or not.

This is not going to be valid anymore, as qemu is going to allocate svq unconditionally (but it will only start them conditionally).

It might be a waste of memory if we did this. Any reason for this?

Well, it's modelled after vhost_vdpa notifier member [1].

Right, this could be optimized in the future as well.

Quoted 9 lines

But sure we can reduce the memory usage if SVQ is not used. The first function that needs it is vhost_set_vring_kick. But I think it is not a good function to place the delayed allocation.

Would it work to move the allocation to vhost_set_features vhost op? It seems unlikely to me to call callbacks that can affect SVQ earlier than that one. Or maybe to create a new one and call it the first on vhost.c:vhost_dev_start?

Rethink about this, so I think we can leave this in the future.

Thanks

Quoted 41 lines

Thanks!

[1] The notifier member already allocates VIRTIO_QUEUE_MAX VhostVDPAHostNotifier for each vhost_vdpa, It is easy to reduce at least to the number of virtqueues on a vhost_vdpa. Should I send a patch for this one?

Quoted 32 lines

Thanks

Quoted 30 lines
---
  hw/virtio/vhost-vdpa.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
hw/virtio/vhost-vdpa.c +2 -3
index 7468e44b87..7f0ff4df5b 100644
@@ -1029,7 +1029,7 @@ static bool vhost_vdpa_svqs_start(struct vhost_dev *dev)
      Error *err = NULL;
      unsigned i;
 
-    if (!v->shadow_vqs) {
+    if (!v->shadow_vqs_enabled) {
          return true;
      }
 
@@ -1082,7 +1082,7 @@ static void vhost_vdpa_svqs_stop(struct vhost_dev *dev)
  {
      struct vhost_vdpa *v = dev->opaque;
 
-    if (!v->shadow_vqs) {
+    if (!v->shadow_vqs_enabled) {
          return;
      }
 
--

2.31.1

#5 lore raw

PATCH v6 2/10 vhost: set SVQ device call handler at SVQ start

In reply to Eugenio Pérez

By the end of this series CVQ is shadowed as long as the features support it.

Since we don't know at the beginning of qemu running if this is supported, move the event notifier handler setting to the start of the SVQ, instead of the start of qemu run.

---
 hw/virtio/vhost-shadow-virtqueue.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
hw/virtio/vhost-shadow-virtqueue.c +2 -2
index 5bd14cad96..264ddc166d 100644
@@ -648,6 +648,7 @@ void vhost_svq_start(VhostShadowVirtqueue *svq, VirtIODevice *vdev,
 {
     size_t desc_size, driver_size, device_size;
 
+    event_notifier_set_handler(&svq->hdev_call, vhost_svq_handle_call);
     svq->next_guest_avail_elem = NULL;
     svq->shadow_avail_idx = 0;
     svq->shadow_used_idx = 0;
@@ -704,6 +705,7 @@ void vhost_svq_stop(VhostShadowVirtqueue *svq)
     g_free(svq->desc_state);
     qemu_vfree(svq->vring.desc);
     qemu_vfree(svq->vring.used);
+    event_notifier_set_handler(&svq->hdev_call, NULL);
 }
 
 /**
@@ -740,7 +742,6 @@ VhostShadowVirtqueue *vhost_svq_new(VhostIOVATree *iova_tree,
     }
 
     event_notifier_init_fd(&svq->svq_kick, VHOST_FILE_UNBIND);
-    event_notifier_set_handler(&svq->hdev_call, vhost_svq_handle_call);
     svq->iova_tree = iova_tree;
     svq->ops = ops;
     svq->ops_opaque = ops_opaque;
@@ -763,7 +764,6 @@ void vhost_svq_free(gpointer pvq)
     VhostShadowVirtqueue *vq = pvq;
     vhost_svq_stop(vq);
     event_notifier_cleanup(&vq->hdev_kick);
-    event_notifier_set_handler(&vq->hdev_call, NULL);
     event_notifier_cleanup(&vq->hdev_call);
     g_free(vq);
 }
Signature · 1 line
2.31.1
Acked-by on v6 2/10
#6 lore raw

Re: [PATCH v6 02/10] vhost: set SVQ device call handler at SVQ start

Quoted 9 lines · Eugenio Pérez

By the end of this series CVQ is shadowed as long as the features support it.

Since we don't know at the beginning of qemu running if this is supported, move the event notifier handler setting to the start of the SVQ, instead of the start of qemu run.

Thanks

Quoted 43 lines
---
 hw/virtio/vhost-shadow-virtqueue.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
hw/virtio/vhost-shadow-virtqueue.c +2 -3
index 5bd14cad96..264ddc166d 100644
@@ -648,6 +648,7 @@ void vhost_svq_start(VhostShadowVirtqueue *svq, VirtIODevice *vdev,
 {
     size_t desc_size, driver_size, device_size;
 
+    event_notifier_set_handler(&svq->hdev_call, vhost_svq_handle_call);
     svq->next_guest_avail_elem = NULL;
     svq->shadow_avail_idx = 0;
     svq->shadow_used_idx = 0;
@@ -704,6 +705,7 @@ void vhost_svq_stop(VhostShadowVirtqueue *svq)
     g_free(svq->desc_state);
     qemu_vfree(svq->vring.desc);
     qemu_vfree(svq->vring.used);
+    event_notifier_set_handler(&svq->hdev_call, NULL);
 }
 
 /**
@@ -740,7 +742,6 @@ VhostShadowVirtqueue *vhost_svq_new(VhostIOVATree *iova_tree,
     }
 
     event_notifier_init_fd(&svq->svq_kick, VHOST_FILE_UNBIND);
-    event_notifier_set_handler(&svq->hdev_call, vhost_svq_handle_call);
     svq->iova_tree = iova_tree;
     svq->ops = ops;
     svq->ops_opaque = ops_opaque;
@@ -763,7 +764,6 @@ void vhost_svq_free(gpointer pvq)
     VhostShadowVirtqueue *vq = pvq;
     vhost_svq_stop(vq);
     event_notifier_cleanup(&vq->hdev_kick);
-    event_notifier_set_handler(&vq->hdev_call, NULL);
     event_notifier_cleanup(&vq->hdev_call);
     g_free(vq);
 }
--

2.31.1

#7 lore raw

PATCH v6 3/10 vhost: Allocate SVQ device file descriptors at device start

In reply to Eugenio Pérez

The next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.

Delay device file descriptors until we know it at device start.

---
 hw/virtio/vhost-shadow-virtqueue.c | 31 ++------------------------
 hw/virtio/vhost-vdpa.c             | 35 ++++++++++++++++++++++++------
 2 files changed, 30 insertions(+), 36 deletions(-)
hw/virtio/vhost-shadow-virtqueue.c +2 -29
index 264ddc166d..3b05bab44d 100644
@@ -715,43 +715,18 @@ void vhost_svq_stop(VhostShadowVirtqueue *svq)
  * @iova_tree: Tree to perform descriptors translations
  * @ops: SVQ owner callbacks
  * @ops_opaque: ops opaque pointer
- *
- * Returns the new virtqueue or NULL.
- *
- * In case of error, reason is reported through error_report.
  */
 VhostShadowVirtqueue *vhost_svq_new(VhostIOVATree *iova_tree,
                                     const VhostShadowVirtqueueOps *ops,
                                     void *ops_opaque)
 {
-    g_autofree VhostShadowVirtqueue *svq = g_new0(VhostShadowVirtqueue, 1);
-    int r;
-
-    r = event_notifier_init(&svq->hdev_kick, 0);
-    if (r != 0) {
-        error_report("Couldn't create kick event notifier: %s (%d)",
-                     g_strerror(errno), errno);
-        goto err_init_hdev_kick;
-    }
-
-    r = event_notifier_init(&svq->hdev_call, 0);
-    if (r != 0) {
-        error_report("Couldn't create call event notifier: %s (%d)",
-                     g_strerror(errno), errno);
-        goto err_init_hdev_call;
-    }
+    VhostShadowVirtqueue *svq = g_new0(VhostShadowVirtqueue, 1);
 
     event_notifier_init_fd(&svq->svq_kick, VHOST_FILE_UNBIND);
     svq->iova_tree = iova_tree;
     svq->ops = ops;
     svq->ops_opaque = ops_opaque;
-    return g_steal_pointer(&svq);
-
-err_init_hdev_call:
-    event_notifier_cleanup(&svq->hdev_kick);
-
-err_init_hdev_kick:
-    return NULL;
+    return svq;
 }
 
 /**
@@ -763,7 +738,5 @@ void vhost_svq_free(gpointer pvq)
 {
     VhostShadowVirtqueue *vq = pvq;
     vhost_svq_stop(vq);
-    event_notifier_cleanup(&vq->hdev_kick);
-    event_notifier_cleanup(&vq->hdev_call);
     g_free(vq);
 }
hw/virtio/vhost-vdpa.c +28 -7
index 7f0ff4df5b..3df2775760 100644
@@ -428,15 +428,11 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v,
 
     shadow_vqs = g_ptr_array_new_full(hdev->nvqs, vhost_svq_free);
     for (unsigned n = 0; n < hdev->nvqs; ++n) {
-        g_autoptr(VhostShadowVirtqueue) svq;
+        VhostShadowVirtqueue *svq;
 
         svq = vhost_svq_new(v->iova_tree, v->shadow_vq_ops,
                             v->shadow_vq_ops_opaque);
-        if (unlikely(!svq)) {
-            error_setg(errp, "Cannot create svq %u", n);
-            return -1;
-        }
-        g_ptr_array_add(shadow_vqs, g_steal_pointer(&svq));
+        g_ptr_array_add(shadow_vqs, svq);
     }
 
     v->shadow_vqs = g_steal_pointer(&shadow_vqs);
@@ -864,11 +860,23 @@ static int vhost_vdpa_svq_set_fds(struct vhost_dev *dev,
     const EventNotifier *event_notifier = &svq->hdev_kick;
     int r;
 
+    r = event_notifier_init(&svq->hdev_kick, 0);
+    if (r != 0) {
+        error_setg_errno(errp, -r, "Couldn't create kick event notifier");
+        goto err_init_hdev_kick;
+    }
+
+    r = event_notifier_init(&svq->hdev_call, 0);
+    if (r != 0) {
+        error_setg_errno(errp, -r, "Couldn't create call event notifier");
+        goto err_init_hdev_call;
+    }
+
     file.fd = event_notifier_get_fd(event_notifier);
     r = vhost_vdpa_set_vring_dev_kick(dev, &file);
     if (unlikely(r != 0)) {
         error_setg_errno(errp, -r, "Can't set device kick fd");
-        return r;
+        goto err_init_set_dev_fd;
     }
 
     event_notifier = &svq->hdev_call;
@@ -876,8 +884,18 @@ static int vhost_vdpa_svq_set_fds(struct vhost_dev *dev,
     r = vhost_vdpa_set_vring_dev_call(dev, &file);
     if (unlikely(r != 0)) {
         error_setg_errno(errp, -r, "Can't set device call fd");
+        goto err_init_set_dev_fd;
     }
 
+    return 0;
+
+err_init_set_dev_fd:
+    event_notifier_set_handler(&svq->hdev_call, NULL);
+
+err_init_hdev_call:
+    event_notifier_cleanup(&svq->hdev_kick);
+
+err_init_hdev_kick:
     return r;
 }
 
@@ -1089,6 +1107,9 @@ static void vhost_vdpa_svqs_stop(struct vhost_dev *dev)
     for (unsigned i = 0; i < v->shadow_vqs->len; ++i) {
         VhostShadowVirtqueue *svq = g_ptr_array_index(v->shadow_vqs, i);
         vhost_vdpa_svq_unmap_rings(dev, svq);
+
+        event_notifier_cleanup(&svq->hdev_kick);
+        event_notifier_cleanup(&svq->hdev_call);
     }
 }
 
Signature · 1 line
2.31.1
Acked-by on v6 3/10
#8 lore raw

Re: [PATCH v6 03/10] vhost: Allocate SVQ device file descriptors at device start

Quoted 7 lines · Eugenio Pérez

The next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.

Delay device file descriptors until we know it at device start.

Thanks

Quoted 142 lines
---
 hw/virtio/vhost-shadow-virtqueue.c | 31 ++------------------------
 hw/virtio/vhost-vdpa.c             | 35 ++++++++++++++++++++++++------
 2 files changed, 30 insertions(+), 36 deletions(-)
hw/virtio/vhost-shadow-virtqueue.c +2 -29
index 264ddc166d..3b05bab44d 100644
@@ -715,43 +715,18 @@ void vhost_svq_stop(VhostShadowVirtqueue *svq)
  * @iova_tree: Tree to perform descriptors translations
  * @ops: SVQ owner callbacks
  * @ops_opaque: ops opaque pointer
- *
- * Returns the new virtqueue or NULL.
- *
- * In case of error, reason is reported through error_report.
  */
 VhostShadowVirtqueue *vhost_svq_new(VhostIOVATree *iova_tree,
                                     const VhostShadowVirtqueueOps *ops,
                                     void *ops_opaque)
 {
-    g_autofree VhostShadowVirtqueue *svq = g_new0(VhostShadowVirtqueue, 1);
-    int r;
-
-    r = event_notifier_init(&svq->hdev_kick, 0);
-    if (r != 0) {
-        error_report("Couldn't create kick event notifier: %s (%d)",
-                     g_strerror(errno), errno);
-        goto err_init_hdev_kick;
-    }
-
-    r = event_notifier_init(&svq->hdev_call, 0);
-    if (r != 0) {
-        error_report("Couldn't create call event notifier: %s (%d)",
-                     g_strerror(errno), errno);
-        goto err_init_hdev_call;
-    }
+    VhostShadowVirtqueue *svq = g_new0(VhostShadowVirtqueue, 1);
 
     event_notifier_init_fd(&svq->svq_kick, VHOST_FILE_UNBIND);
     svq->iova_tree = iova_tree;
     svq->ops = ops;
     svq->ops_opaque = ops_opaque;
-    return g_steal_pointer(&svq);
-
-err_init_hdev_call:
-    event_notifier_cleanup(&svq->hdev_kick);
-
-err_init_hdev_kick:
-    return NULL;
+    return svq;
 }
 
 /**
@@ -763,7 +738,5 @@ void vhost_svq_free(gpointer pvq)
 {
     VhostShadowVirtqueue *vq = pvq;
     vhost_svq_stop(vq);
-    event_notifier_cleanup(&vq->hdev_kick);
-    event_notifier_cleanup(&vq->hdev_call);
     g_free(vq);
 }
hw/virtio/vhost-vdpa.c +28 -8
index 7f0ff4df5b..3df2775760 100644
@@ -428,15 +428,11 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v,
 
     shadow_vqs = g_ptr_array_new_full(hdev->nvqs, vhost_svq_free);
     for (unsigned n = 0; n < hdev->nvqs; ++n) {
-        g_autoptr(VhostShadowVirtqueue) svq;
+        VhostShadowVirtqueue *svq;
 
         svq = vhost_svq_new(v->iova_tree, v->shadow_vq_ops,
                             v->shadow_vq_ops_opaque);
-        if (unlikely(!svq)) {
-            error_setg(errp, "Cannot create svq %u", n);
-            return -1;
-        }
-        g_ptr_array_add(shadow_vqs, g_steal_pointer(&svq));
+        g_ptr_array_add(shadow_vqs, svq);
     }
 
     v->shadow_vqs = g_steal_pointer(&shadow_vqs);
@@ -864,11 +860,23 @@ static int vhost_vdpa_svq_set_fds(struct vhost_dev *dev,
     const EventNotifier *event_notifier = &svq->hdev_kick;
     int r;
 
+    r = event_notifier_init(&svq->hdev_kick, 0);
+    if (r != 0) {
+        error_setg_errno(errp, -r, "Couldn't create kick event notifier");
+        goto err_init_hdev_kick;
+    }
+
+    r = event_notifier_init(&svq->hdev_call, 0);
+    if (r != 0) {
+        error_setg_errno(errp, -r, "Couldn't create call event notifier");
+        goto err_init_hdev_call;
+    }
+
     file.fd = event_notifier_get_fd(event_notifier);
     r = vhost_vdpa_set_vring_dev_kick(dev, &file);
     if (unlikely(r != 0)) {
         error_setg_errno(errp, -r, "Can't set device kick fd");
-        return r;
+        goto err_init_set_dev_fd;
     }
 
     event_notifier = &svq->hdev_call;
@@ -876,8 +884,18 @@ static int vhost_vdpa_svq_set_fds(struct vhost_dev *dev,
     r = vhost_vdpa_set_vring_dev_call(dev, &file);
     if (unlikely(r != 0)) {
         error_setg_errno(errp, -r, "Can't set device call fd");
+        goto err_init_set_dev_fd;
     }
 
+    return 0;
+
+err_init_set_dev_fd:
+    event_notifier_set_handler(&svq->hdev_call, NULL);
+
+err_init_hdev_call:
+    event_notifier_cleanup(&svq->hdev_kick);
+
+err_init_hdev_kick:
     return r;
 }
 
@@ -1089,6 +1107,9 @@ static void vhost_vdpa_svqs_stop(struct vhost_dev *dev)
     for (unsigned i = 0; i < v->shadow_vqs->len; ++i) {
         VhostShadowVirtqueue *svq = g_ptr_array_index(v->shadow_vqs, i);
         vhost_vdpa_svq_unmap_rings(dev, svq);
+
+        event_notifier_cleanup(&svq->hdev_kick);
+        event_notifier_cleanup(&svq->hdev_call);
     }
 }
 
--

2.31.1

#9 lore raw

PATCH v6 4/10 vdpa: add vhost_vdpa_net_valid_svq_features

In reply to Eugenio Pérez

It will be reused at vdpa device start so let's extract in its own function

---
 net/vhost-vdpa.c | 26 +++++++++++++++++---------
 1 file changed, 17 insertions(+), 9 deletions(-)
net/vhost-vdpa.c +17 -9
index e370ecb8eb..d3b1de481b 100644
@@ -106,6 +106,22 @@ VHostNetState *vhost_vdpa_get_vhost_net(NetClientState *nc)
     return s->vhost_net;
 }
 
+static bool vhost_vdpa_net_valid_svq_features(uint64_t features, Error **errp)
+{
+    uint64_t invalid_dev_features =
+        features & ~vdpa_svq_device_features &
+        /* Transport are all accepted at this point */
+        ~MAKE_64BIT_MASK(VIRTIO_TRANSPORT_F_START,
+                         VIRTIO_TRANSPORT_F_END - VIRTIO_TRANSPORT_F_START);
+
+    if (invalid_dev_features) {
+        error_setg(errp, "vdpa svq does not work with features 0x%" PRIx64,
+                   invalid_dev_features);
+    }
+
+    return !invalid_dev_features;
+}
+
 static int vhost_vdpa_net_check_device_id(struct vhost_net *net)
 {
     uint32_t device_id;
@@ -675,15 +691,7 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
     if (opts->x_svq) {
         struct vhost_vdpa_iova_range iova_range;
 
-        uint64_t invalid_dev_features =
-            features & ~vdpa_svq_device_features &
-            /* Transport are all accepted at this point */
-            ~MAKE_64BIT_MASK(VIRTIO_TRANSPORT_F_START,
-                             VIRTIO_TRANSPORT_F_END - VIRTIO_TRANSPORT_F_START);
-
-        if (invalid_dev_features) {
-            error_setg(errp, "vdpa svq does not work with features 0x%" PRIx64,
-                       invalid_dev_features);
+        if (!vhost_vdpa_net_valid_svq_features(features, errp)) {
             goto err_svq;
         }
 
Signature · 1 line
2.31.1
Acked-by on v6 4/10
#10 lore raw

Re: [PATCH v6 04/10] vdpa: add vhost_vdpa_net_valid_svq_features

Quoted 5 lines · Eugenio Pérez

It will be reused at vdpa device start so let's extract in its own function

---

Thanks

Quoted 50 lines
 net/vhost-vdpa.c | 26 +++++++++++++++++---------
 1 file changed, 17 insertions(+), 9 deletions(-)
net/vhost-vdpa.c +17 -10
index e370ecb8eb..d3b1de481b 100644
@@ -106,6 +106,22 @@ VHostNetState *vhost_vdpa_get_vhost_net(NetClientState *nc)
     return s->vhost_net;
 }
 
+static bool vhost_vdpa_net_valid_svq_features(uint64_t features, Error **errp)
+{
+    uint64_t invalid_dev_features =
+        features & ~vdpa_svq_device_features &
+        /* Transport are all accepted at this point */
+        ~MAKE_64BIT_MASK(VIRTIO_TRANSPORT_F_START,
+                         VIRTIO_TRANSPORT_F_END - VIRTIO_TRANSPORT_F_START);
+
+    if (invalid_dev_features) {
+        error_setg(errp, "vdpa svq does not work with features 0x%" PRIx64,
+                   invalid_dev_features);
+    }
+
+    return !invalid_dev_features;
+}
+
 static int vhost_vdpa_net_check_device_id(struct vhost_net *net)
 {
     uint32_t device_id;
@@ -675,15 +691,7 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
     if (opts->x_svq) {
         struct vhost_vdpa_iova_range iova_range;
 
-        uint64_t invalid_dev_features =
-            features & ~vdpa_svq_device_features &
-            /* Transport are all accepted at this point */
-            ~MAKE_64BIT_MASK(VIRTIO_TRANSPORT_F_START,
-                             VIRTIO_TRANSPORT_F_END - VIRTIO_TRANSPORT_F_START);
-
-        if (invalid_dev_features) {
-            error_setg(errp, "vdpa svq does not work with features 0x%" PRIx64,
-                       invalid_dev_features);
+        if (!vhost_vdpa_net_valid_svq_features(features, errp)) {
             goto err_svq;
         }
 
--

2.31.1

#11 lore raw

PATCH v6 5/10 vdpa: move SVQ vring features check to net/

In reply to Eugenio Pérez

The next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.

Since the moved checks will be already evaluated at net/ to know if it is ok to shadow CVQ, move them.

---
 hw/virtio/vhost-vdpa.c | 33 ++-------------------------------
 net/vhost-vdpa.c       |  3 ++-
 2 files changed, 4 insertions(+), 32 deletions(-)
hw/virtio/vhost-vdpa.c +2 -31
index 3df2775760..146f0dcb40 100644
@@ -402,29 +402,9 @@ static int vhost_vdpa_get_dev_features(struct vhost_dev *dev,
     return ret;
 }
 
-static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v,
-                               Error **errp)
+static void vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v)
 {
     g_autoptr(GPtrArray) shadow_vqs = NULL;
-    uint64_t dev_features, svq_features;
-    int r;
-    bool ok;
-
-    if (!v->shadow_vqs_enabled) {
-        return 0;
-    }
-
-    r = vhost_vdpa_get_dev_features(hdev, &dev_features);
-    if (r != 0) {
-        error_setg_errno(errp, -r, "Can't get vdpa device features");
-        return r;
-    }
-
-    svq_features = dev_features;
-    ok = vhost_svq_valid_features(svq_features, errp);
-    if (unlikely(!ok)) {
-        return -1;
-    }
 
     shadow_vqs = g_ptr_array_new_full(hdev->nvqs, vhost_svq_free);
     for (unsigned n = 0; n < hdev->nvqs; ++n) {
@@ -436,7 +416,6 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v,
     }
 
     v->shadow_vqs = g_steal_pointer(&shadow_vqs);
-    return 0;
 }
 
 static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
@@ -461,11 +440,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
     dev->opaque =  opaque ;
     v->listener = vhost_vdpa_memory_listener;
     v->msg_type = VHOST_IOTLB_MSG_V2;
-    ret = vhost_vdpa_init_svq(dev, v, errp);
-    if (ret) {
-        goto err;
-    }
-
+    vhost_vdpa_init_svq(dev, v);
     vhost_vdpa_get_iova_range(v);
 
     if (!vhost_vdpa_first_dev(dev)) {
@@ -476,10 +451,6 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
                                VIRTIO_CONFIG_S_DRIVER);
 
     return 0;
-
-err:
-    ram_block_discard_disable(false);
-    return ret;
 }
 
 static void vhost_vdpa_host_notifier_uninit(struct vhost_dev *dev,
net/vhost-vdpa.c +2 -1
index d3b1de481b..fb35b17ab4 100644
@@ -117,9 +117,10 @@ static bool vhost_vdpa_net_valid_svq_features(uint64_t features, Error **errp)
     if (invalid_dev_features) {
         error_setg(errp, "vdpa svq does not work with features 0x%" PRIx64,
                    invalid_dev_features);
+        return false;
     }
 
-    return !invalid_dev_features;
+    return vhost_svq_valid_features(features, errp);
 }
 
 static int vhost_vdpa_net_check_device_id(struct vhost_net *net)
Signature · 1 line
2.31.1
#12 lore raw

Re: [PATCH v6 05/10] vdpa: move SVQ vring features check to net/

Quoted 2 lines · Eugenio Pérez

The next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.

If I was not wrong, there's no device specific feature that is checked in the function. So it should be general enough to be used by devices other than net. Then I don't see any advantage of doing this.

Thanks

Quoted 93 lines

Since the moved checks will be already evaluated at net/ to know if it is ok to shadow CVQ, move them.

---
  hw/virtio/vhost-vdpa.c | 33 ++-------------------------------
  net/vhost-vdpa.c       |  3 ++-
  2 files changed, 4 insertions(+), 32 deletions(-)
hw/virtio/vhost-vdpa.c +2 -31
index 3df2775760..146f0dcb40 100644
@@ -402,29 +402,9 @@ static int vhost_vdpa_get_dev_features(struct vhost_dev *dev,
      return ret;
  }
 
-static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v,
-                               Error **errp)
+static void vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v)
  {
      g_autoptr(GPtrArray) shadow_vqs = NULL;
-    uint64_t dev_features, svq_features;
-    int r;
-    bool ok;
-
-    if (!v->shadow_vqs_enabled) {
-        return 0;
-    }
-
-    r = vhost_vdpa_get_dev_features(hdev, &dev_features);
-    if (r != 0) {
-        error_setg_errno(errp, -r, "Can't get vdpa device features");
-        return r;
-    }
-
-    svq_features = dev_features;
-    ok = vhost_svq_valid_features(svq_features, errp);
-    if (unlikely(!ok)) {
-        return -1;
-    }
 
      shadow_vqs = g_ptr_array_new_full(hdev->nvqs, vhost_svq_free);
      for (unsigned n = 0; n < hdev->nvqs; ++n) {
@@ -436,7 +416,6 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v,
      }
 
      v->shadow_vqs = g_steal_pointer(&shadow_vqs);
-    return 0;
  }
 
  static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
@@ -461,11 +440,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
      dev->opaque =  opaque ;
      v->listener = vhost_vdpa_memory_listener;
      v->msg_type = VHOST_IOTLB_MSG_V2;
-    ret = vhost_vdpa_init_svq(dev, v, errp);
-    if (ret) {
-        goto err;
-    }
-
+    vhost_vdpa_init_svq(dev, v);
      vhost_vdpa_get_iova_range(v);
 
      if (!vhost_vdpa_first_dev(dev)) {
@@ -476,10 +451,6 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
                                 VIRTIO_CONFIG_S_DRIVER);
 
      return 0;
-
-err:
-    ram_block_discard_disable(false);
-    return ret;
  }
 
  static void vhost_vdpa_host_notifier_uninit(struct vhost_dev *dev,
net/vhost-vdpa.c +2 -1
index d3b1de481b..fb35b17ab4 100644
@@ -117,9 +117,10 @@ static bool vhost_vdpa_net_valid_svq_features(uint64_t features, Error **errp)
      if (invalid_dev_features) {
          error_setg(errp, "vdpa svq does not work with features 0x%" PRIx64,
                     invalid_dev_features);
+        return false;
      }
 
-    return !invalid_dev_features;
+    return vhost_svq_valid_features(features, errp);
  }
 
  static int vhost_vdpa_net_check_device_id(struct vhost_net *net)
#13 lore raw

Re: [PATCH v6 05/10] vdpa: move SVQ vring features check to net/

Quoted 11 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez

The next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.

If I was not wrong, there's no device specific feature that is checked in the function. So it should be general enough to be used by devices other than net. Then I don't see any advantage of doing this.

Because vhost_vdpa_init_svq is called at qemu boot, failing if it is not possible to shadow the Virtqueue.

Now the CVQ will be shadowed if possible, so we need to check this at device start, not at initialization. To store this information at boot time is not valid anymore, because v->shadow_vqs_enabled is not valid at this time anymore.

Thanks!

Quoted 97 lines

Thanks

Quoted 93 lines

Since the moved checks will be already evaluated at net/ to know if it is ok to shadow CVQ, move them.

---
  hw/virtio/vhost-vdpa.c | 33 ++-------------------------------
  net/vhost-vdpa.c       |  3 ++-
  2 files changed, 4 insertions(+), 32 deletions(-)
hw/virtio/vhost-vdpa.c +2 -31
index 3df2775760..146f0dcb40 100644
@@ -402,29 +402,9 @@ static int vhost_vdpa_get_dev_features(struct vhost_dev *dev,
      return ret;
  }
 
-static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v,
-                               Error **errp)
+static void vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v)
  {
      g_autoptr(GPtrArray) shadow_vqs = NULL;
-    uint64_t dev_features, svq_features;
-    int r;
-    bool ok;
-
-    if (!v->shadow_vqs_enabled) {
-        return 0;
-    }
-
-    r = vhost_vdpa_get_dev_features(hdev, &dev_features);
-    if (r != 0) {
-        error_setg_errno(errp, -r, "Can't get vdpa device features");
-        return r;
-    }
-
-    svq_features = dev_features;
-    ok = vhost_svq_valid_features(svq_features, errp);
-    if (unlikely(!ok)) {
-        return -1;
-    }
 
      shadow_vqs = g_ptr_array_new_full(hdev->nvqs, vhost_svq_free);
      for (unsigned n = 0; n < hdev->nvqs; ++n) {
@@ -436,7 +416,6 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v,
      }
 
      v->shadow_vqs = g_steal_pointer(&shadow_vqs);
-    return 0;
  }
 
  static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
@@ -461,11 +440,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
      dev->opaque =  opaque ;
      v->listener = vhost_vdpa_memory_listener;
      v->msg_type = VHOST_IOTLB_MSG_V2;
-    ret = vhost_vdpa_init_svq(dev, v, errp);
-    if (ret) {
-        goto err;
-    }
-
+    vhost_vdpa_init_svq(dev, v);
      vhost_vdpa_get_iova_range(v);
 
      if (!vhost_vdpa_first_dev(dev)) {
@@ -476,10 +451,6 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
                                 VIRTIO_CONFIG_S_DRIVER);
 
      return 0;
-
-err:
-    ram_block_discard_disable(false);
-    return ret;
  }
 
  static void vhost_vdpa_host_notifier_uninit(struct vhost_dev *dev,
net/vhost-vdpa.c +2 -1
index d3b1de481b..fb35b17ab4 100644
@@ -117,9 +117,10 @@ static bool vhost_vdpa_net_valid_svq_features(uint64_t features, Error **errp)
      if (invalid_dev_features) {
          error_setg(errp, "vdpa svq does not work with features 0x%" PRIx64,
                     invalid_dev_features);
+        return false;
      }
 
-    return !invalid_dev_features;
+    return vhost_svq_valid_features(features, errp);
  }
 
  static int vhost_vdpa_net_check_device_id(struct vhost_net *net)
#14 lore raw

Re: [PATCH v6 05/10] vdpa: move SVQ vring features check to net/

Quoted 10 lines · Eugenio Perez Martin
Quoted 9 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez

The next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.

If I was not wrong, there's no device specific feature that is checked in the function. So it should be general enough to be used by devices other than net. Then I don't see any advantage of doing this.

Quoted 5 lines

Because vhost_vdpa_init_svq is called at qemu boot, failing if it is not possible to shadow the Virtqueue.

Now the CVQ will be shadowed if possible, so we need to check this at device start, not at initialization.

Any reason we can't check this at device start? We don't need driver_features and we can do any probing to make sure cvq has an unique group during initialization time.

Quoted 3 lines

To store this information at boot
time is not valid anymore, because v->shadow_vqs_enabled is not valid at this time anymore.

Ok, but this doesn't explain why it is net specific but vhost-vdpa specific.

Thanks

Quoted 98 lines

Thanks!

Quoted 95 lines

Thanks

Quoted 92 lines

Since the moved checks will be already evaluated at net/ to know if it is ok to shadow CVQ, move them.

---
   hw/virtio/vhost-vdpa.c | 33 ++-------------------------------
   net/vhost-vdpa.c       |  3 ++-
   2 files changed, 4 insertions(+), 32 deletions(-)
hw/virtio/vhost-vdpa.c +2 -31
index 3df2775760..146f0dcb40 100644
@@ -402,29 +402,9 @@ static int vhost_vdpa_get_dev_features(struct vhost_dev *dev,
       return ret;
   }
 
-static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v,
-                               Error **errp)
+static void vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v)
   {
       g_autoptr(GPtrArray) shadow_vqs = NULL;
-    uint64_t dev_features, svq_features;
-    int r;
-    bool ok;
-
-    if (!v->shadow_vqs_enabled) {
-        return 0;
-    }
-
-    r = vhost_vdpa_get_dev_features(hdev, &dev_features);
-    if (r != 0) {
-        error_setg_errno(errp, -r, "Can't get vdpa device features");
-        return r;
-    }
-
-    svq_features = dev_features;
-    ok = vhost_svq_valid_features(svq_features, errp);
-    if (unlikely(!ok)) {
-        return -1;
-    }
 
       shadow_vqs = g_ptr_array_new_full(hdev->nvqs, vhost_svq_free);
       for (unsigned n = 0; n < hdev->nvqs; ++n) {
@@ -436,7 +416,6 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v,
       }
 
       v->shadow_vqs = g_steal_pointer(&shadow_vqs);
-    return 0;
   }
 
   static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
@@ -461,11 +440,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
       dev->opaque =  opaque ;
       v->listener = vhost_vdpa_memory_listener;
       v->msg_type = VHOST_IOTLB_MSG_V2;
-    ret = vhost_vdpa_init_svq(dev, v, errp);
-    if (ret) {
-        goto err;
-    }
-
+    vhost_vdpa_init_svq(dev, v);
       vhost_vdpa_get_iova_range(v);
 
       if (!vhost_vdpa_first_dev(dev)) {
@@ -476,10 +451,6 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
                                  VIRTIO_CONFIG_S_DRIVER);
 
       return 0;
-
-err:
-    ram_block_discard_disable(false);
-    return ret;
   }
 
   static void vhost_vdpa_host_notifier_uninit(struct vhost_dev *dev,
net/vhost-vdpa.c +2 -1
index d3b1de481b..fb35b17ab4 100644
@@ -117,9 +117,10 @@ static bool vhost_vdpa_net_valid_svq_features(uint64_t features, Error **errp)
       if (invalid_dev_features) {
           error_setg(errp, "vdpa svq does not work with features 0x%" PRIx64,
                      invalid_dev_features);
+        return false;
       }
 
-    return !invalid_dev_features;
+    return vhost_svq_valid_features(features, errp);
   }
 
   static int vhost_vdpa_net_check_device_id(struct vhost_net *net)
#15 lore raw

Re: [PATCH v6 05/10] vdpa: move SVQ vring features check to net/

Quoted 16 lines · Jason Wang
Quoted 13 lines · Eugenio Perez Martin
Quoted 9 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez

The next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.

If I was not wrong, there's no device specific feature that is checked in the function. So it should be general enough to be used by devices other than net. Then I don't see any advantage of doing this.

Because vhost_vdpa_init_svq is called at qemu boot, failing if it is not possible to shadow the Virtqueue.

Quoted 8 lines
Quoted 2 lines

Now the CVQ will be shadowed if possible, so we need to check this at device start, not at initialization.

Any reason we can't check this at device start? We don't need driver_features and we can do any probing to make sure cvq has an unique group during initialization time.

We need the CVQ index to check if it has an independent group. CVQ index depends on the features the guest's ack:
* If it acks _F_MQ, it is the last one.
* If it doesn't, CVQ idx is 2.

We cannot have acked features at initialization, and they could
change: It is valid for a guest to ack _F_MQ, then reset the device, then not ack it.

Quoted 8 lines
Quoted 4 lines

To store this information at boot
time is not valid anymore, because v->shadow_vqs_enabled is not valid at this time anymore.

Ok, but this doesn't explain why it is net specific but vhost-vdpa specific.

We can try to move it to a vhost op, but we have the same problem as the svq array allocation: We don't have the right place in vhost ops to check this. Maybe vhost_set_features is the right one here?

Thanks!

Quoted 102 lines

Thanks

Quoted 98 lines

Thanks!

Quoted 95 lines

Thanks

Quoted 92 lines

Since the moved checks will be already evaluated at net/ to know if it is ok to shadow CVQ, move them.

---
   hw/virtio/vhost-vdpa.c | 33 ++-------------------------------
   net/vhost-vdpa.c       |  3 ++-
   2 files changed, 4 insertions(+), 32 deletions(-)
hw/virtio/vhost-vdpa.c +2 -31
index 3df2775760..146f0dcb40 100644
@@ -402,29 +402,9 @@ static int vhost_vdpa_get_dev_features(struct vhost_dev *dev,
       return ret;
   }
 
-static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v,
-                               Error **errp)
+static void vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v)
   {
       g_autoptr(GPtrArray) shadow_vqs = NULL;
-    uint64_t dev_features, svq_features;
-    int r;
-    bool ok;
-
-    if (!v->shadow_vqs_enabled) {
-        return 0;
-    }
-
-    r = vhost_vdpa_get_dev_features(hdev, &dev_features);
-    if (r != 0) {
-        error_setg_errno(errp, -r, "Can't get vdpa device features");
-        return r;
-    }
-
-    svq_features = dev_features;
-    ok = vhost_svq_valid_features(svq_features, errp);
-    if (unlikely(!ok)) {
-        return -1;
-    }
 
       shadow_vqs = g_ptr_array_new_full(hdev->nvqs, vhost_svq_free);
       for (unsigned n = 0; n < hdev->nvqs; ++n) {
@@ -436,7 +416,6 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v,
       }
 
       v->shadow_vqs = g_steal_pointer(&shadow_vqs);
-    return 0;
   }
 
   static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
@@ -461,11 +440,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
       dev->opaque =  opaque ;
       v->listener = vhost_vdpa_memory_listener;
       v->msg_type = VHOST_IOTLB_MSG_V2;
-    ret = vhost_vdpa_init_svq(dev, v, errp);
-    if (ret) {
-        goto err;
-    }
-
+    vhost_vdpa_init_svq(dev, v);
       vhost_vdpa_get_iova_range(v);
 
       if (!vhost_vdpa_first_dev(dev)) {
@@ -476,10 +451,6 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
                                  VIRTIO_CONFIG_S_DRIVER);
 
       return 0;
-
-err:
-    ram_block_discard_disable(false);
-    return ret;
   }
 
   static void vhost_vdpa_host_notifier_uninit(struct vhost_dev *dev,
net/vhost-vdpa.c +2 -1
index d3b1de481b..fb35b17ab4 100644
@@ -117,9 +117,10 @@ static bool vhost_vdpa_net_valid_svq_features(uint64_t features, Error **errp)
       if (invalid_dev_features) {
           error_setg(errp, "vdpa svq does not work with features 0x%" PRIx64,
                      invalid_dev_features);
+        return false;
       }
 
-    return !invalid_dev_features;
+    return vhost_svq_valid_features(features, errp);
   }
 
   static int vhost_vdpa_net_check_device_id(struct vhost_net *net)
#16 lore raw

Re: [PATCH v6 05/10] vdpa: move SVQ vring features check to net/

Quoted 27 lines · Eugenio Perez Martin
Quoted 16 lines · Jason Wang
Quoted 13 lines · Eugenio Perez Martin
Quoted 9 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez

The next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.

If I was not wrong, there's no device specific feature that is checked in the function. So it should be general enough to be used by devices other than net. Then I don't see any advantage of doing this.

Because vhost_vdpa_init_svq is called at qemu boot, failing if it is not possible to shadow the Virtqueue.

Quoted 8 lines
Quoted 2 lines

Now the CVQ will be shadowed if possible, so we need to check this at device start, not at initialization.

Any reason we can't check this at device start? We don't need driver_features and we can do any probing to make sure cvq has an unique group during initialization time.

Quoted 8 lines

We need the CVQ index to check if it has an independent group. CVQ index depends on the features the guest's ack:
* If it acks _F_MQ, it is the last one.
* If it doesn't, CVQ idx is 2.

We cannot have acked features at initialization, and they could
change: It is valid for a guest to ack _F_MQ, then reset the device, then not ack it.

Can we do some probing by negotiating _F_MQ if the device offers it, then we can know if cvq has a unique group?

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

To store this information at boot
time is not valid anymore, because v->shadow_vqs_enabled is not valid at this time anymore.

Quoted 6 lines
Quoted 2 lines

Ok, but this doesn't explain why it is net specific but vhost-vdpa specific.

We can try to move it to a vhost op, but we have the same problem as the svq array allocation: We don't have the right place in vhost ops to check this. Maybe vhost_set_features is the right one here?

If we can do all the probing at the initialization phase, we can do everything there.

Thanks

Quoted 106 lines

Thanks!

Quoted 102 lines

Thanks

Quoted 98 lines

Thanks!

Quoted 95 lines

Thanks

Quoted 92 lines

Since the moved checks will be already evaluated at net/ to know if it is ok to shadow CVQ, move them.

---
   hw/virtio/vhost-vdpa.c | 33 ++-------------------------------
   net/vhost-vdpa.c       |  3 ++-
   2 files changed, 4 insertions(+), 32 deletions(-)
hw/virtio/vhost-vdpa.c +2 -31
index 3df2775760..146f0dcb40 100644
@@ -402,29 +402,9 @@ static int vhost_vdpa_get_dev_features(struct vhost_dev *dev,
       return ret;
   }
 
-static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v,
-                               Error **errp)
+static void vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v)
   {
       g_autoptr(GPtrArray) shadow_vqs = NULL;
-    uint64_t dev_features, svq_features;
-    int r;
-    bool ok;
-
-    if (!v->shadow_vqs_enabled) {
-        return 0;
-    }
-
-    r = vhost_vdpa_get_dev_features(hdev, &dev_features);
-    if (r != 0) {
-        error_setg_errno(errp, -r, "Can't get vdpa device features");
-        return r;
-    }
-
-    svq_features = dev_features;
-    ok = vhost_svq_valid_features(svq_features, errp);
-    if (unlikely(!ok)) {
-        return -1;
-    }
 
       shadow_vqs = g_ptr_array_new_full(hdev->nvqs, vhost_svq_free);
       for (unsigned n = 0; n < hdev->nvqs; ++n) {
@@ -436,7 +416,6 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v,
       }
 
       v->shadow_vqs = g_steal_pointer(&shadow_vqs);
-    return 0;
   }
 
   static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
@@ -461,11 +440,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
       dev->opaque =  opaque ;
       v->listener = vhost_vdpa_memory_listener;
       v->msg_type = VHOST_IOTLB_MSG_V2;
-    ret = vhost_vdpa_init_svq(dev, v, errp);
-    if (ret) {
-        goto err;
-    }
-
+    vhost_vdpa_init_svq(dev, v);
       vhost_vdpa_get_iova_range(v);
 
       if (!vhost_vdpa_first_dev(dev)) {
@@ -476,10 +451,6 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
                                  VIRTIO_CONFIG_S_DRIVER);
 
       return 0;
-
-err:
-    ram_block_discard_disable(false);
-    return ret;
   }
 
   static void vhost_vdpa_host_notifier_uninit(struct vhost_dev *dev,
net/vhost-vdpa.c +2 -1
index d3b1de481b..fb35b17ab4 100644
@@ -117,9 +117,10 @@ static bool vhost_vdpa_net_valid_svq_features(uint64_t features, Error **errp)
       if (invalid_dev_features) {
           error_setg(errp, "vdpa svq does not work with features 0x%" PRIx64,
                      invalid_dev_features);
+        return false;
       }
 
-    return !invalid_dev_features;
+    return vhost_svq_valid_features(features, errp);
   }
 
   static int vhost_vdpa_net_check_device_id(struct vhost_net *net)
#17 lore raw

Re: [PATCH v6 05/10] vdpa: move SVQ vring features check to net/

Quoted 35 lines · Jason Wang
Quoted 32 lines · Eugenio Perez Martin
Quoted 16 lines · Jason Wang
Quoted 13 lines · Eugenio Perez Martin
Quoted 9 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez

The next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.

If I was not wrong, there's no device specific feature that is checked in the function. So it should be general enough to be used by devices other than net. Then I don't see any advantage of doing this.

Because vhost_vdpa_init_svq is called at qemu boot, failing if it is not possible to shadow the Virtqueue.

Quoted 8 lines
Quoted 2 lines

Now the CVQ will be shadowed if possible, so we need to check this at device start, not at initialization.

Any reason we can't check this at device start? We don't need driver_features and we can do any probing to make sure cvq has an unique group during initialization time.

We need the CVQ index to check if it has an independent group. CVQ index depends on the features the guest's ack:
* If it acks _F_MQ, it is the last one.
* If it doesn't, CVQ idx is 2.

Quoted 7 lines
Quoted 3 lines

We cannot have acked features at initialization, and they could
change: It is valid for a guest to ack _F_MQ, then reset the device, then not ack it.

Can we do some probing by negotiating _F_MQ if the device offers it, then we can know if cvq has a unique group?

What if the guest does not ack _F_MQ?

To be completed it would go like:

* Probe negotiate _F_MQ, check unique group,
* Probe negotiate !_F_MQ, check unique group,
* Actually negotiate with the guest's feature set.
* React to failures. Probably the same way as if the CVQ is not isolated, disabling SVQ?

To me it seems simpler to specify somehow that the vq must be independent.

Thanks!

Quoted 126 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

To store this information at boot
time is not valid anymore, because v->shadow_vqs_enabled is not valid at this time anymore.

Quoted 6 lines
Quoted 2 lines

Ok, but this doesn't explain why it is net specific but vhost-vdpa specific.

We can try to move it to a vhost op, but we have the same problem as the svq array allocation: We don't have the right place in vhost ops to check this. Maybe vhost_set_features is the right one here?

If we can do all the probing at the initialization phase, we can do everything there.

Thanks

Quoted 106 lines

Thanks!

Quoted 102 lines

Thanks

Quoted 98 lines

Thanks!

Quoted 95 lines

Thanks

Quoted 92 lines

Since the moved checks will be already evaluated at net/ to know if it is ok to shadow CVQ, move them.

---
   hw/virtio/vhost-vdpa.c | 33 ++-------------------------------
   net/vhost-vdpa.c       |  3 ++-
   2 files changed, 4 insertions(+), 32 deletions(-)
hw/virtio/vhost-vdpa.c +2 -31
index 3df2775760..146f0dcb40 100644
@@ -402,29 +402,9 @@ static int vhost_vdpa_get_dev_features(struct vhost_dev *dev,
       return ret;
   }
 
-static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v,
-                               Error **errp)
+static void vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v)
   {
       g_autoptr(GPtrArray) shadow_vqs = NULL;
-    uint64_t dev_features, svq_features;
-    int r;
-    bool ok;
-
-    if (!v->shadow_vqs_enabled) {
-        return 0;
-    }
-
-    r = vhost_vdpa_get_dev_features(hdev, &dev_features);
-    if (r != 0) {
-        error_setg_errno(errp, -r, "Can't get vdpa device features");
-        return r;
-    }
-
-    svq_features = dev_features;
-    ok = vhost_svq_valid_features(svq_features, errp);
-    if (unlikely(!ok)) {
-        return -1;
-    }
 
       shadow_vqs = g_ptr_array_new_full(hdev->nvqs, vhost_svq_free);
       for (unsigned n = 0; n < hdev->nvqs; ++n) {
@@ -436,7 +416,6 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v,
       }
 
       v->shadow_vqs = g_steal_pointer(&shadow_vqs);
-    return 0;
   }
 
   static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
@@ -461,11 +440,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
       dev->opaque =  opaque ;
       v->listener = vhost_vdpa_memory_listener;
       v->msg_type = VHOST_IOTLB_MSG_V2;
-    ret = vhost_vdpa_init_svq(dev, v, errp);
-    if (ret) {
-        goto err;
-    }
-
+    vhost_vdpa_init_svq(dev, v);
       vhost_vdpa_get_iova_range(v);
 
       if (!vhost_vdpa_first_dev(dev)) {
@@ -476,10 +451,6 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
                                  VIRTIO_CONFIG_S_DRIVER);
 
       return 0;
-
-err:
-    ram_block_discard_disable(false);
-    return ret;
   }
 
   static void vhost_vdpa_host_notifier_uninit(struct vhost_dev *dev,
net/vhost-vdpa.c +2 -1
index d3b1de481b..fb35b17ab4 100644
@@ -117,9 +117,10 @@ static bool vhost_vdpa_net_valid_svq_features(uint64_t features, Error **errp)
       if (invalid_dev_features) {
           error_setg(errp, "vdpa svq does not work with features 0x%" PRIx64,
                      invalid_dev_features);
+        return false;
       }
 
-    return !invalid_dev_features;
+    return vhost_svq_valid_features(features, errp);
   }
 
   static int vhost_vdpa_net_check_device_id(struct vhost_net *net)
#18 lore raw

Re: [PATCH v6 05/10] vdpa: move SVQ vring features check to net/

Quoted 35 lines · Eugenio Perez Martin
Quoted 28 lines · Jason Wang
Quoted 26 lines · Eugenio Perez Martin
Quoted 13 lines · Jason Wang
Quoted 11 lines · Eugenio Perez Martin
Quoted 7 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez

The next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.

If I was not wrong, there's no device specific feature that is checked in the function. So it should be general enough to be used by devices other than net. Then I don't see any advantage of doing this.

Because vhost_vdpa_init_svq is called at qemu boot, failing if it is not possible to shadow the Virtqueue.

Quoted 7 lines
Quoted 2 lines

Now the CVQ will be shadowed if possible, so we need to check this at device start, not at initialization.

Any reason we can't check this at device start? We don't need driver_features and we can do any probing to make sure cvq has an unique group during initialization time.

We need the CVQ index to check if it has an independent group. CVQ index depends on the features the guest's ack:
* If it acks _F_MQ, it is the last one.
* If it doesn't, CVQ idx is 2.

Quoted 6 lines
Quoted 3 lines

We cannot have acked features at initialization, and they could
change: It is valid for a guest to ack _F_MQ, then reset the device, then not ack it.

Can we do some probing by negotiating _F_MQ if the device offers it, then we can know if cvq has a unique group?

Quoted 6 lines

What if the guest does not ack _F_MQ?

To be completed it would go like:

* Probe negotiate _F_MQ, check unique group,
* Probe negotiate !_F_MQ, check unique group,

I think it should be a bug if device present a unique virtqueue group that depends on a specific feature. That is to say, we can do a single round of probing instead of try it twice here.

Quoted 5 lines

* Actually negotiate with the guest's feature set.
* React to failures. Probably the same way as if the CVQ is not isolated, disabling SVQ?

To me it seems simpler to specify somehow that the vq must be independent.

It's just a suggestion, if you think doing it at the start, I'm fine. But we need document the reason with a comment maybe.

Thanks

Quoted 119 lines

Thanks!

Quoted 116 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

To store this information at boot
time is not valid anymore, because v->shadow_vqs_enabled is not valid at this time anymore.

Ok, but this doesn't explain why it is net specific but vhost-vdpa specific.

We can try to move it to a vhost op, but we have the same problem as the svq array allocation: We don't have the right place in vhost ops to check this. Maybe vhost_set_features is the right one here?

If we can do all the probing at the initialization phase, we can do everything there.

Thanks

Quoted 102 lines

Thanks!

Quoted 100 lines

Thanks

Quoted 97 lines

Thanks!

Quoted 95 lines

Thanks

Quoted 92 lines

Since the moved checks will be already evaluated at net/ to know if it is ok to shadow CVQ, move them.

---
    hw/virtio/vhost-vdpa.c | 33 ++-------------------------------
    net/vhost-vdpa.c       |  3 ++-
    2 files changed, 4 insertions(+), 32 deletions(-)
hw/virtio/vhost-vdpa.c +2 -31
index 3df2775760..146f0dcb40 100644
@@ -402,29 +402,9 @@ static int vhost_vdpa_get_dev_features(struct vhost_dev *dev,
        return ret;
    }
 
-static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v,
-                               Error **errp)
+static void vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v)
    {
        g_autoptr(GPtrArray) shadow_vqs = NULL;
-    uint64_t dev_features, svq_features;
-    int r;
-    bool ok;
-
-    if (!v->shadow_vqs_enabled) {
-        return 0;
-    }
-
-    r = vhost_vdpa_get_dev_features(hdev, &dev_features);
-    if (r != 0) {
-        error_setg_errno(errp, -r, "Can't get vdpa device features");
-        return r;
-    }
-
-    svq_features = dev_features;
-    ok = vhost_svq_valid_features(svq_features, errp);
-    if (unlikely(!ok)) {
-        return -1;
-    }
 
        shadow_vqs = g_ptr_array_new_full(hdev->nvqs, vhost_svq_free);
        for (unsigned n = 0; n < hdev->nvqs; ++n) {
@@ -436,7 +416,6 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v,
        }
 
        v->shadow_vqs = g_steal_pointer(&shadow_vqs);
-    return 0;
    }
 
    static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
@@ -461,11 +440,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
        dev->opaque =  opaque ;
        v->listener = vhost_vdpa_memory_listener;
        v->msg_type = VHOST_IOTLB_MSG_V2;
-    ret = vhost_vdpa_init_svq(dev, v, errp);
-    if (ret) {
-        goto err;
-    }
-
+    vhost_vdpa_init_svq(dev, v);
        vhost_vdpa_get_iova_range(v);
 
        if (!vhost_vdpa_first_dev(dev)) {
@@ -476,10 +451,6 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
                                   VIRTIO_CONFIG_S_DRIVER);
 
        return 0;
-
-err:
-    ram_block_discard_disable(false);
-    return ret;
    }
 
    static void vhost_vdpa_host_notifier_uninit(struct vhost_dev *dev,
net/vhost-vdpa.c +2 -1
index d3b1de481b..fb35b17ab4 100644
@@ -117,9 +117,10 @@ static bool vhost_vdpa_net_valid_svq_features(uint64_t features, Error **errp)
        if (invalid_dev_features) {
            error_setg(errp, "vdpa svq does not work with features 0x%" PRIx64,
                       invalid_dev_features);
+        return false;
        }
 
-    return !invalid_dev_features;
+    return vhost_svq_valid_features(features, errp);
    }
 
    static int vhost_vdpa_net_check_device_id(struct vhost_net *net)
#19 lore raw

Re: [PATCH v6 05/10] vdpa: move SVQ vring features check to net/

Quoted 42 lines · Jason Wang
Quoted 39 lines · Eugenio Perez Martin
Quoted 28 lines · Jason Wang
Quoted 26 lines · Eugenio Perez Martin
Quoted 13 lines · Jason Wang
Quoted 11 lines · Eugenio Perez Martin
Quoted 7 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez

The next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.

If I was not wrong, there's no device specific feature that is checked in the function. So it should be general enough to be used by devices other than net. Then I don't see any advantage of doing this.

Because vhost_vdpa_init_svq is called at qemu boot, failing if it is not possible to shadow the Virtqueue.

Quoted 7 lines
Quoted 2 lines

Now the CVQ will be shadowed if possible, so we need to check this at device start, not at initialization.

Any reason we can't check this at device start? We don't need driver_features and we can do any probing to make sure cvq has an unique group during initialization time.

We need the CVQ index to check if it has an independent group. CVQ index depends on the features the guest's ack:
* If it acks _F_MQ, it is the last one.
* If it doesn't, CVQ idx is 2.

Quoted 6 lines
Quoted 3 lines

We cannot have acked features at initialization, and they could
change: It is valid for a guest to ack _F_MQ, then reset the device, then not ack it.

Can we do some probing by negotiating _F_MQ if the device offers it, then we can know if cvq has a unique group?

What if the guest does not ack _F_MQ?

To be completed it would go like:

Quoted 8 lines
Quoted 2 lines

* Probe negotiate _F_MQ, check unique group,
* Probe negotiate !_F_MQ, check unique group,

I think it should be a bug if device present a unique virtqueue group that depends on a specific feature. That is to say, we can do a single round of probing instead of try it twice here.

I didn't mean a single virtqueue group but CVQ has its own group.

From vhost POV the valid behavior is feature dependent already: If _F_MQ is not negotiated, the vq in a different group must be the 3rd. If it is negotiated, the vq in its own group must be the last one.

Since the check of the virtqueue group is already dependent on the vq index I think it would be a mistake to change a vq group ASID without checking if it is independent or not.

The consequence of not checking that is that:
* We will move all the dataplane vq group to ASID 1, so the device will not work properly.
* The device will be migratable.

Quoted 9 lines
Quoted 6 lines

* Actually negotiate with the guest's feature set.
* React to failures. Probably the same way as if the CVQ is not isolated, disabling SVQ?

To me it seems simpler to specify somehow that the vq must be independent.

It's just a suggestion, if you think doing it at the start, I'm fine.

Don't get me wrong, we can make changes to go toward that direction and I think it's a good idea, especially to reduce syscalls. I'm just trying to put all the scenarios on the table because maybe I'm missing something to solve them, or we can ignore them

Thanks!

Quoted 125 lines

But we need document the reason with a comment maybe.

Thanks

Quoted 119 lines

Thanks!

Quoted 116 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

To store this information at boot
time is not valid anymore, because v->shadow_vqs_enabled is not valid at this time anymore.

Ok, but this doesn't explain why it is net specific but vhost-vdpa specific.

We can try to move it to a vhost op, but we have the same problem as the svq array allocation: We don't have the right place in vhost ops to check this. Maybe vhost_set_features is the right one here?

If we can do all the probing at the initialization phase, we can do everything there.

Thanks

Quoted 102 lines

Thanks!

Quoted 100 lines

Thanks

Quoted 97 lines

Thanks!

Quoted 95 lines

Thanks

Quoted 92 lines

Since the moved checks will be already evaluated at net/ to know if it is ok to shadow CVQ, move them.

---
    hw/virtio/vhost-vdpa.c | 33 ++-------------------------------
    net/vhost-vdpa.c       |  3 ++-
    2 files changed, 4 insertions(+), 32 deletions(-)
hw/virtio/vhost-vdpa.c +2 -31
index 3df2775760..146f0dcb40 100644
@@ -402,29 +402,9 @@ static int vhost_vdpa_get_dev_features(struct vhost_dev *dev,
        return ret;
    }
 
-static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v,
-                               Error **errp)
+static void vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v)
    {
        g_autoptr(GPtrArray) shadow_vqs = NULL;
-    uint64_t dev_features, svq_features;
-    int r;
-    bool ok;
-
-    if (!v->shadow_vqs_enabled) {
-        return 0;
-    }
-
-    r = vhost_vdpa_get_dev_features(hdev, &dev_features);
-    if (r != 0) {
-        error_setg_errno(errp, -r, "Can't get vdpa device features");
-        return r;
-    }
-
-    svq_features = dev_features;
-    ok = vhost_svq_valid_features(svq_features, errp);
-    if (unlikely(!ok)) {
-        return -1;
-    }
 
        shadow_vqs = g_ptr_array_new_full(hdev->nvqs, vhost_svq_free);
        for (unsigned n = 0; n < hdev->nvqs; ++n) {
@@ -436,7 +416,6 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v,
        }
 
        v->shadow_vqs = g_steal_pointer(&shadow_vqs);
-    return 0;
    }
 
    static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
@@ -461,11 +440,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
        dev->opaque =  opaque ;
        v->listener = vhost_vdpa_memory_listener;
        v->msg_type = VHOST_IOTLB_MSG_V2;
-    ret = vhost_vdpa_init_svq(dev, v, errp);
-    if (ret) {
-        goto err;
-    }
-
+    vhost_vdpa_init_svq(dev, v);
        vhost_vdpa_get_iova_range(v);
 
        if (!vhost_vdpa_first_dev(dev)) {
@@ -476,10 +451,6 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
                                   VIRTIO_CONFIG_S_DRIVER);
 
        return 0;
-
-err:
-    ram_block_discard_disable(false);
-    return ret;
    }
 
    static void vhost_vdpa_host_notifier_uninit(struct vhost_dev *dev,
net/vhost-vdpa.c +2 -1
index d3b1de481b..fb35b17ab4 100644
@@ -117,9 +117,10 @@ static bool vhost_vdpa_net_valid_svq_features(uint64_t features, Error **errp)
        if (invalid_dev_features) {
            error_setg(errp, "vdpa svq does not work with features 0x%" PRIx64,
                       invalid_dev_features);
+        return false;
        }
 
-    return !invalid_dev_features;
+    return vhost_svq_valid_features(features, errp);
    }
 
    static int vhost_vdpa_net_check_device_id(struct vhost_net *net)
#20 lore raw

PATCH v6 6/10 vdpa: Allocate SVQ unconditionally

In reply to Eugenio Pérez

SVQ may run or not in a device depending on runtime conditions (for example, if the device can move CVQ to its own group or not).

Allocate the SVQ array unconditionally at startup, since its hard to move this allocation elsewhere.

---
 hw/virtio/vhost-vdpa.c | 4 ----
 1 file changed, 4 deletions(-)
hw/virtio/vhost-vdpa.c +0 -4
index 146f0dcb40..23efb8f49d 100644
@@ -547,10 +547,6 @@ static void vhost_vdpa_svq_cleanup(struct vhost_dev *dev)
     struct vhost_vdpa *v = dev->opaque;
     size_t idx;
 
-    if (!v->shadow_vqs) {
-        return;
-    }
-
     for (idx = 0; idx < v->shadow_vqs->len; ++idx) {
         vhost_svq_stop(g_ptr_array_index(v->shadow_vqs, idx));
     }
Signature · 1 line
2.31.1
#21 lore raw

PATCH v6 7/10 vdpa: Add asid parameter to vhost_vdpa_dma_map/unmap

In reply to Eugenio Pérez

So the caller can choose which ASID is destined.

No need to update the batch functions as they will always be called from memory listener updates at the moment. Memory listener updates will always update ASID 0, as it's the passthrough ASID.

All vhost devices's ASID are 0 at this moment.

---
v5:
* Solve conflict, now vhost_vdpa_svq_unmap_ring returns void
* Change comment on zero initialization.

v4: Add comment specifying behavior if device does not support _F_ASID

v3: Deleted unneeded space

---
 include/hw/virtio/vhost-vdpa.h |  8 +++++---
 hw/virtio/vhost-vdpa.c         | 29 +++++++++++++++++++----------
 net/vhost-vdpa.c               |  6 +++---
 hw/virtio/trace-events         |  4 ++--
 4 files changed, 29 insertions(+), 18 deletions(-)
include/hw/virtio/vhost-vdpa.h +5 -3
index 1111d85643..6560bb9d78 100644
@@ -29,6 +29,7 @@ typedef struct vhost_vdpa {
     int index;
     uint32_t msg_type;
     bool iotlb_batch_begin_sent;
+    uint32_t address_space_id;
     MemoryListener listener;
     struct vhost_vdpa_iova_range iova_range;
     uint64_t acked_features;
@@ -42,8 +43,9 @@ typedef struct vhost_vdpa {
     VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX];
 } VhostVDPA;
 
-int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,
-                       void *vaddr, bool readonly);
-int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, hwaddr iova, hwaddr size);
+int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                       hwaddr size, void *vaddr, bool readonly);
+int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                         hwaddr size);
 
 #endif
hw/virtio/vhost-vdpa.c +19 -10
index 23efb8f49d..8fd32ba32b 100644
@@ -72,22 +72,24 @@ static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,
     return false;
 }
 
-int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,
-                       void *vaddr, bool readonly)
+int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                       hwaddr size, void *vaddr, bool readonly)
 {
     struct vhost_msg_v2 msg = {};
     int fd = v->device_fd;
     int ret = 0;
 
     msg.type = v->msg_type;
+    msg.asid = asid; /* 0 if vdpa device does not support asid */
     msg.iotlb.iova = iova;
     msg.iotlb.size = size;
     msg.iotlb.uaddr = (uint64_t)(uintptr_t)vaddr;
     msg.iotlb.perm = readonly ? VHOST_ACCESS_RO : VHOST_ACCESS_RW;
     msg.iotlb.type = VHOST_IOTLB_UPDATE;
 
-   trace_vhost_vdpa_dma_map(v, fd, msg.type, msg.iotlb.iova, msg.iotlb.size,
-                            msg.iotlb.uaddr, msg.iotlb.perm, msg.iotlb.type);
+    trace_vhost_vdpa_dma_map(v, fd, msg.type, msg.asid, msg.iotlb.iova,
+                             msg.iotlb.size, msg.iotlb.uaddr, msg.iotlb.perm,
+                             msg.iotlb.type);
 
     if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
         error_report("failed to write, fd=%d, errno=%d (%s)",
@@ -98,18 +100,24 @@ int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,
     return ret;
 }
 
-int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, hwaddr iova, hwaddr size)
+int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                         hwaddr size)
 {
     struct vhost_msg_v2 msg = {};
     int fd = v->device_fd;
     int ret = 0;
 
     msg.type = v->msg_type;
+    /*
+     * The caller must set asid = 0 if the device does not support asid.
+     * This is not an ABI break since it is set to 0 by the initializer anyway.
+     */
+    msg.asid = asid;
     msg.iotlb.iova = iova;
     msg.iotlb.size = size;
     msg.iotlb.type = VHOST_IOTLB_INVALIDATE;
 
-    trace_vhost_vdpa_dma_unmap(v, fd, msg.type, msg.iotlb.iova,
+    trace_vhost_vdpa_dma_unmap(v, fd, msg.type, msg.asid, msg.iotlb.iova,
                                msg.iotlb.size, msg.iotlb.type);
 
     if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
@@ -229,7 +237,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
     }
 
     vhost_vdpa_iotlb_batch_begin_once(v);
-    ret = vhost_vdpa_dma_map(v, iova, int128_get64(llsize),
+    ret = vhost_vdpa_dma_map(v, 0, iova, int128_get64(llsize),
                              vaddr, section->readonly);
     if (ret) {
         error_report("vhost vdpa map fail!");
@@ -303,7 +311,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
         vhost_iova_tree_remove(v->iova_tree, *result);
     }
     vhost_vdpa_iotlb_batch_begin_once(v);
-    ret = vhost_vdpa_dma_unmap(v, iova, int128_get64(llsize));
+    ret = vhost_vdpa_dma_unmap(v, 0, iova, int128_get64(llsize));
     if (ret) {
         error_report("vhost_vdpa dma unmap error!");
     }
@@ -884,7 +892,7 @@ static void vhost_vdpa_svq_unmap_ring(struct vhost_vdpa *v, hwaddr addr)
     }
 
     size = ROUND_UP(result->size, qemu_real_host_page_size());
-    r = vhost_vdpa_dma_unmap(v, result->iova, size);
+    r = vhost_vdpa_dma_unmap(v, v->address_space_id, result->iova, size);
     if (unlikely(r < 0)) {
         error_report("Unable to unmap SVQ vring: %s (%d)", g_strerror(-r), -r);
         return;
@@ -924,7 +932,8 @@ static bool vhost_vdpa_svq_map_ring(struct vhost_vdpa *v, DMAMap *needle,
         return false;
     }
 
-    r = vhost_vdpa_dma_map(v, needle->iova, needle->size + 1,
+    r = vhost_vdpa_dma_map(v, v->address_space_id, needle->iova,
+                           needle->size + 1,
                            (void *)(uintptr_t)needle->translated_addr,
                            needle->perm == IOMMU_RO);
     if (unlikely(r != 0)) {
net/vhost-vdpa.c +3 -3
index fb35b17ab4..ca1acc0410 100644
@@ -258,7 +258,7 @@ static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr)
         return;
     }
 
-    r = vhost_vdpa_dma_unmap(v, map->iova, map->size + 1);
+    r = vhost_vdpa_dma_unmap(v, v->address_space_id, map->iova, map->size + 1);
     if (unlikely(r != 0)) {
         error_report("Device cannot unmap: %s(%d)", g_strerror(r), r);
     }
@@ -298,8 +298,8 @@ static int vhost_vdpa_cvq_map_buf(struct vhost_vdpa *v, void *buf, size_t size,
         return r;
     }
 
-    r = vhost_vdpa_dma_map(v, map.iova, vhost_vdpa_net_cvq_cmd_page_len(), buf,
-                           !write);
+    r = vhost_vdpa_dma_map(v, v->address_space_id, map.iova,
+                           vhost_vdpa_net_cvq_cmd_page_len(), buf, !write);
     if (unlikely(r < 0)) {
         goto dma_map_err;
     }
hw/virtio/trace-events +2 -2
index 820dadc26c..0ad9390307 100644
@@ -30,8 +30,8 @@ vhost_user_write(uint32_t req, uint32_t flags) "req:%d flags:0x%"PRIx32""
 vhost_user_create_notifier(int idx, void *n) "idx:%d n:%p"
 
 # vhost-vdpa.c
-vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8
-vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8
+vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8
+vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8
 vhost_vdpa_listener_begin_batch(void *v, int fd, uint32_t msg_type, uint8_t type)  "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
 vhost_vdpa_listener_commit(void *v, int fd, uint32_t msg_type, uint8_t type)  "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
 vhost_vdpa_listener_region_add(void *vdpa, uint64_t iova, uint64_t llend, void *vaddr, bool readonly) "vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64" vaddr: %p read-only: %d"
Signature · 1 line
2.31.1
#22 lore raw

Re: [PATCH v6 07/10] vdpa: Add asid parameter to vhost_vdpa_dma_map/unmap

Quoted 30 lines · Eugenio Pérez

So the caller can choose which ASID is destined.

No need to update the batch functions as they will always be called from memory listener updates at the moment. Memory listener updates will always update ASID 0, as it's the passthrough ASID.

All vhost devices's ASID are 0 at this moment.

---
v5:
* Solve conflict, now vhost_vdpa_svq_unmap_ring returns void
* Change comment on zero initialization.

v4: Add comment specifying behavior if device does not support _F_ASID

v3: Deleted unneeded space

---
 include/hw/virtio/vhost-vdpa.h |  8 +++++---
 hw/virtio/vhost-vdpa.c         | 29 +++++++++++++++++++----------
 net/vhost-vdpa.c               |  6 +++---
 hw/virtio/trace-events         |  4 ++--
 4 files changed, 29 insertions(+), 18 deletions(-)
include/hw/virtio/vhost-vdpa.h +0 -0
index 1111d85643..6560bb9d78 100644

include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 32

Quoted 4 lines
include/hw/virtio/vhost-vdpa.h +1 -0
@@ -29,6 +29,7 @@ typedef struct vhost_vdpa {
     int index;
     uint32_t msg_type;
     bool iotlb_batch_begin_sent;
+    uint32_t address_space_id;

So the trick is let device specific code to zero this during allocation?

Quoted 25 lines
include/hw/virtio/vhost-vdpa.h +4 -3
     MemoryListener listener;
     struct vhost_vdpa_iova_range iova_range;
     uint64_t acked_features;
@@ -42,8 +43,9 @@ typedef struct vhost_vdpa {
     VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX];
 } VhostVDPA;
 
-int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,
-                       void *vaddr, bool readonly);
-int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, hwaddr iova, hwaddr size);
+int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                       hwaddr size, void *vaddr, bool readonly);
+int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                         hwaddr size);
 
 #endif
hw/virtio/vhost-vdpa.c +0 -1
index 23efb8f49d..8fd32ba32b 100644
@@ -72,22 +72,24 @@ static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,
     return false;
 }
 
-int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,

hw/virtio/vhost-vdpa.c static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section, · line 83

Quoted 10 lines
hw/virtio/vhost-vdpa.c +3 -1
-                       void *vaddr, bool readonly)
+int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                       hwaddr size, void *vaddr, bool readonly)
 {
     struct vhost_msg_v2 msg = {};
     int fd = v->device_fd;
     int ret = 0;
 
     msg.type = v->msg_type;
+    msg.asid = asid; /* 0 if vdpa device does not support asid */

The comment here is confusing. If this is a requirement, we need either

1) doc this

or

2) perform necessary checks in the function itself.

Quoted 42 lines
hw/virtio/vhost-vdpa.c +11 -4
     msg.iotlb.iova = iova;
     msg.iotlb.size = size;
     msg.iotlb.uaddr = (uint64_t)(uintptr_t)vaddr;
     msg.iotlb.perm = readonly ? VHOST_ACCESS_RO : VHOST_ACCESS_RW;
     msg.iotlb.type = VHOST_IOTLB_UPDATE;
 
-   trace_vhost_vdpa_dma_map(v, fd, msg.type, msg.iotlb.iova, msg.iotlb.size,
-                            msg.iotlb.uaddr, msg.iotlb.perm, msg.iotlb.type);
+    trace_vhost_vdpa_dma_map(v, fd, msg.type, msg.asid, msg.iotlb.iova,
+                             msg.iotlb.size, msg.iotlb.uaddr, msg.iotlb.perm,
+                             msg.iotlb.type);
 
     if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
         error_report("failed to write, fd=%d, errno=%d (%s)",
@@ -98,18 +100,24 @@ int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,
     return ret;
 }
 
-int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, hwaddr iova, hwaddr size)
+int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                         hwaddr size)
 {
     struct vhost_msg_v2 msg = {};
     int fd = v->device_fd;
     int ret = 0;
 
     msg.type = v->msg_type;
+    /*
+     * The caller must set asid = 0 if the device does not support asid.
+     * This is not an ABI break since it is set to 0 by the initializer anyway.
+     */
+    msg.asid = asid;
     msg.iotlb.iova = iova;
     msg.iotlb.size = size;
     msg.iotlb.type = VHOST_IOTLB_INVALIDATE;
 
-    trace_vhost_vdpa_dma_unmap(v, fd, msg.type, msg.iotlb.iova,
+    trace_vhost_vdpa_dma_unmap(v, fd, msg.type, msg.asid, msg.iotlb.iova,
                                msg.iotlb.size, msg.iotlb.type);
 
     if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {

hw/virtio/vhost-vdpa.c static void vhost_vdpa_listener_region_add(MemoryListener *listener, · line 240

Quoted 5 lines
hw/virtio/vhost-vdpa.c +1 -1
@@ -229,7 +237,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
     }
 
     vhost_vdpa_iotlb_batch_begin_once(v);
-    ret = vhost_vdpa_dma_map(v, iova, int128_get64(llsize),
+    ret = vhost_vdpa_dma_map(v, 0, iova, int128_get64(llsize),

Can we use v->address_space_id here? Then we don't need to modify this line when we support multiple asids logic in the future.

Thanks

Quoted 73 lines
hw/virtio/vhost-vdpa.c +4 -3
                              vaddr, section->readonly);
     if (ret) {
         error_report("vhost vdpa map fail!");
@@ -303,7 +311,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
         vhost_iova_tree_remove(v->iova_tree, *result);
     }
     vhost_vdpa_iotlb_batch_begin_once(v);
-    ret = vhost_vdpa_dma_unmap(v, iova, int128_get64(llsize));
+    ret = vhost_vdpa_dma_unmap(v, 0, iova, int128_get64(llsize));
     if (ret) {
         error_report("vhost_vdpa dma unmap error!");
     }
@@ -884,7 +892,7 @@ static void vhost_vdpa_svq_unmap_ring(struct vhost_vdpa *v, hwaddr addr)
     }
 
     size = ROUND_UP(result->size, qemu_real_host_page_size());
-    r = vhost_vdpa_dma_unmap(v, result->iova, size);
+    r = vhost_vdpa_dma_unmap(v, v->address_space_id, result->iova, size);
     if (unlikely(r < 0)) {
         error_report("Unable to unmap SVQ vring: %s (%d)", g_strerror(-r), -r);
         return;
@@ -924,7 +932,8 @@ static bool vhost_vdpa_svq_map_ring(struct vhost_vdpa *v, DMAMap *needle,
         return false;
     }
 
-    r = vhost_vdpa_dma_map(v, needle->iova, needle->size + 1,
+    r = vhost_vdpa_dma_map(v, v->address_space_id, needle->iova,
+                           needle->size + 1,
                            (void *)(uintptr_t)needle->translated_addr,
                            needle->perm == IOMMU_RO);
     if (unlikely(r != 0)) {
net/vhost-vdpa.c +3 -3
index fb35b17ab4..ca1acc0410 100644
@@ -258,7 +258,7 @@ static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr)
         return;
     }
 
-    r = vhost_vdpa_dma_unmap(v, map->iova, map->size + 1);
+    r = vhost_vdpa_dma_unmap(v, v->address_space_id, map->iova, map->size + 1);
     if (unlikely(r != 0)) {
         error_report("Device cannot unmap: %s(%d)", g_strerror(r), r);
     }
@@ -298,8 +298,8 @@ static int vhost_vdpa_cvq_map_buf(struct vhost_vdpa *v, void *buf, size_t size,
         return r;
     }
 
-    r = vhost_vdpa_dma_map(v, map.iova, vhost_vdpa_net_cvq_cmd_page_len(), buf,
-                           !write);
+    r = vhost_vdpa_dma_map(v, v->address_space_id, map.iova,
+                           vhost_vdpa_net_cvq_cmd_page_len(), buf, !write);
     if (unlikely(r < 0)) {
         goto dma_map_err;
     }
hw/virtio/trace-events +2 -3
index 820dadc26c..0ad9390307 100644
@@ -30,8 +30,8 @@ vhost_user_write(uint32_t req, uint32_t flags) "req:%d flags:0x%"PRIx32""
 vhost_user_create_notifier(int idx, void *n) "idx:%d n:%p"
 
 # vhost-vdpa.c
-vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8
-vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8
+vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8
+vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8
 vhost_vdpa_listener_begin_batch(void *v, int fd, uint32_t msg_type, uint8_t type)  "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
 vhost_vdpa_listener_commit(void *v, int fd, uint32_t msg_type, uint8_t type)  "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
 vhost_vdpa_listener_region_add(void *vdpa, uint64_t iova, uint64_t llend, void *vaddr, bool readonly) "vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64" vaddr: %p read-only: %d"
--

2.31.1

#23 lore raw

Re: [PATCH v6 07/10] vdpa: Add asid parameter to vhost_vdpa_dma_map/unmap

Quoted 27 lines · Jason Wang
Quoted 25 lines · Eugenio Pérez

So the caller can choose which ASID is destined.

No need to update the batch functions as they will always be called from memory listener updates at the moment. Memory listener updates will always update ASID 0, as it's the passthrough ASID.

All vhost devices's ASID are 0 at this moment.

---
v5:
* Solve conflict, now vhost_vdpa_svq_unmap_ring returns void
* Change comment on zero initialization.

v4: Add comment specifying behavior if device does not support _F_ASID

v3: Deleted unneeded space

---
 include/hw/virtio/vhost-vdpa.h |  8 +++++---
 hw/virtio/vhost-vdpa.c         | 29 +++++++++++++++++++----------
 net/vhost-vdpa.c               |  6 +++---
 hw/virtio/trace-events         |  4 ++--
 4 files changed, 29 insertions(+), 18 deletions(-)
Quoted 12 lines

include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 32

Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1 -0
index 1111d85643..6560bb9d78 100644
@@ -29,6 +29,7 @@ typedef struct vhost_vdpa {
     int index;
     uint32_t msg_type;
     bool iotlb_batch_begin_sent;
+    uint32_t address_space_id;

So the trick is let device specific code to zero this during allocation?

Yes, but I don't see how that is a trick :). All other parameters also trust it to be 0 at allocation.

Quoted 38 lines
Quoted 25 lines
include/hw/virtio/vhost-vdpa.h +4 -3
     MemoryListener listener;
     struct vhost_vdpa_iova_range iova_range;
     uint64_t acked_features;
@@ -42,8 +43,9 @@ typedef struct vhost_vdpa {
     VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX];
 } VhostVDPA;
 
-int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,
-                       void *vaddr, bool readonly);
-int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, hwaddr iova, hwaddr size);
+int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                       hwaddr size, void *vaddr, bool readonly);
+int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                         hwaddr size);
 
 #endif
hw/virtio/vhost-vdpa.c +0 -1
index 23efb8f49d..8fd32ba32b 100644
@@ -72,22 +72,24 @@ static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,
     return false;
 }
 
-int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,

hw/virtio/vhost-vdpa.c static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section, · line 83

Quoted 10 lines
hw/virtio/vhost-vdpa.c +3 -1
-                       void *vaddr, bool readonly)
+int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                       hwaddr size, void *vaddr, bool readonly)
 {
     struct vhost_msg_v2 msg = {};
     int fd = v->device_fd;
     int ret = 0;
 
     msg.type = v->msg_type;
+    msg.asid = asid; /* 0 if vdpa device does not support asid */

The comment here is confusing. If this is a requirement, we need either

Quoted 6 lines

1) doc this

or

2) perform necessary checks in the function itself.

I only documented it in vhost_vdpa_dma_unmap and now I realize it. Would it work to just copy that comment here?

Quoted 44 lines
Quoted 44 lines
hw/virtio/vhost-vdpa.c +11 -4
     msg.iotlb.iova = iova;
     msg.iotlb.size = size;
     msg.iotlb.uaddr = (uint64_t)(uintptr_t)vaddr;
     msg.iotlb.perm = readonly ? VHOST_ACCESS_RO : VHOST_ACCESS_RW;
     msg.iotlb.type = VHOST_IOTLB_UPDATE;
 
-   trace_vhost_vdpa_dma_map(v, fd, msg.type, msg.iotlb.iova, msg.iotlb.size,
-                            msg.iotlb.uaddr, msg.iotlb.perm, msg.iotlb.type);
+    trace_vhost_vdpa_dma_map(v, fd, msg.type, msg.asid, msg.iotlb.iova,
+                             msg.iotlb.size, msg.iotlb.uaddr, msg.iotlb.perm,
+                             msg.iotlb.type);
 
     if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
         error_report("failed to write, fd=%d, errno=%d (%s)",
@@ -98,18 +100,24 @@ int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,
     return ret;
 }
 
-int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, hwaddr iova, hwaddr size)
+int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                         hwaddr size)
 {
     struct vhost_msg_v2 msg = {};
     int fd = v->device_fd;
     int ret = 0;
 
     msg.type = v->msg_type;
+    /*
+     * The caller must set asid = 0 if the device does not support asid.
+     * This is not an ABI break since it is set to 0 by the initializer anyway.
+     */
+    msg.asid = asid;
     msg.iotlb.iova = iova;
     msg.iotlb.size = size;
     msg.iotlb.type = VHOST_IOTLB_INVALIDATE;
 
-    trace_vhost_vdpa_dma_unmap(v, fd, msg.type, msg.iotlb.iova,
+    trace_vhost_vdpa_dma_unmap(v, fd, msg.type, msg.asid, msg.iotlb.iova,
                                msg.iotlb.size, msg.iotlb.type);
 
     if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
@@ -229,7 +237,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
     }
 
Quoted 7 lines

hw/virtio/vhost-vdpa.c static void vhost_vdpa_listener_region_add(MemoryListener *listener, · line 240

Quoted 3 lines
hw/virtio/vhost-vdpa.c +1 -1
     vhost_vdpa_iotlb_batch_begin_once(v);
-    ret = vhost_vdpa_dma_map(v, iova, int128_get64(llsize),
+    ret = vhost_vdpa_dma_map(v, 0, iova, int128_get64(llsize),

Can we use v->address_space_id here? Then we don't need to modify this line when we support multiple asids logic in the future.

The registered memory listener is the one of the last vhost_vdpa, the one that handles the last queue.

If all data virtqueues are not shadowed but CVQ is, v->address_space_id is 1 with the current code. But the listener is actually mapping the ASID 0, not 1.

Another alternative is to register it to the last data virtqueue, not the last queue of vhost_vdpa. But it is hard to express it in a generic way at virtio/vhost-vdpa.c . To have a boolean indicating the vhost_vdpa we want to register its memory listener?

It seems easier to me to simply assign 0 at GPA translations. If SVQ is enabled for all queues, then 0 is GPA to qemu's VA + SVQ stuff. If it is not, 0 is always GPA to qemu's VA.

Thanks!

Quoted 76 lines

Thanks

Quoted 73 lines
hw/virtio/vhost-vdpa.c +4 -3
                              vaddr, section->readonly);
     if (ret) {
         error_report("vhost vdpa map fail!");
@@ -303,7 +311,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
         vhost_iova_tree_remove(v->iova_tree, *result);
     }
     vhost_vdpa_iotlb_batch_begin_once(v);
-    ret = vhost_vdpa_dma_unmap(v, iova, int128_get64(llsize));
+    ret = vhost_vdpa_dma_unmap(v, 0, iova, int128_get64(llsize));
     if (ret) {
         error_report("vhost_vdpa dma unmap error!");
     }
@@ -884,7 +892,7 @@ static void vhost_vdpa_svq_unmap_ring(struct vhost_vdpa *v, hwaddr addr)
     }
 
     size = ROUND_UP(result->size, qemu_real_host_page_size());
-    r = vhost_vdpa_dma_unmap(v, result->iova, size);
+    r = vhost_vdpa_dma_unmap(v, v->address_space_id, result->iova, size);
     if (unlikely(r < 0)) {
         error_report("Unable to unmap SVQ vring: %s (%d)", g_strerror(-r), -r);
         return;
@@ -924,7 +932,8 @@ static bool vhost_vdpa_svq_map_ring(struct vhost_vdpa *v, DMAMap *needle,
         return false;
     }
 
-    r = vhost_vdpa_dma_map(v, needle->iova, needle->size + 1,
+    r = vhost_vdpa_dma_map(v, v->address_space_id, needle->iova,
+                           needle->size + 1,
                            (void *)(uintptr_t)needle->translated_addr,
                            needle->perm == IOMMU_RO);
     if (unlikely(r != 0)) {
net/vhost-vdpa.c +3 -3
index fb35b17ab4..ca1acc0410 100644
@@ -258,7 +258,7 @@ static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr)
         return;
     }
 
-    r = vhost_vdpa_dma_unmap(v, map->iova, map->size + 1);
+    r = vhost_vdpa_dma_unmap(v, v->address_space_id, map->iova, map->size + 1);
     if (unlikely(r != 0)) {
         error_report("Device cannot unmap: %s(%d)", g_strerror(r), r);
     }
@@ -298,8 +298,8 @@ static int vhost_vdpa_cvq_map_buf(struct vhost_vdpa *v, void *buf, size_t size,
         return r;
     }
 
-    r = vhost_vdpa_dma_map(v, map.iova, vhost_vdpa_net_cvq_cmd_page_len(), buf,
-                           !write);
+    r = vhost_vdpa_dma_map(v, v->address_space_id, map.iova,
+                           vhost_vdpa_net_cvq_cmd_page_len(), buf, !write);
     if (unlikely(r < 0)) {
         goto dma_map_err;
     }
hw/virtio/trace-events +2 -3
index 820dadc26c..0ad9390307 100644
@@ -30,8 +30,8 @@ vhost_user_write(uint32_t req, uint32_t flags) "req:%d flags:0x%"PRIx32""
 vhost_user_create_notifier(int idx, void *n) "idx:%d n:%p"
 
 # vhost-vdpa.c
-vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8
-vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8
+vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8
+vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8
 vhost_vdpa_listener_begin_batch(void *v, int fd, uint32_t msg_type, uint8_t type)  "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
 vhost_vdpa_listener_commit(void *v, int fd, uint32_t msg_type, uint8_t type)  "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
 vhost_vdpa_listener_region_add(void *vdpa, uint64_t iova, uint64_t llend, void *vaddr, bool readonly) "vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64" vaddr: %p read-only: %d"
--

2.31.1

#24 lore raw

Re: [PATCH v6 07/10] vdpa: Add asid parameter to vhost_vdpa_dma_map/unmap

Quoted 77 lines · Eugenio Perez Martin
Quoted 25 lines · Jason Wang
Quoted 24 lines · Eugenio Pérez

So the caller can choose which ASID is destined.

No need to update the batch functions as they will always be called from memory listener updates at the moment. Memory listener updates will always update ASID 0, as it's the passthrough ASID.

All vhost devices's ASID are 0 at this moment.

---
v5:
* Solve conflict, now vhost_vdpa_svq_unmap_ring returns void
* Change comment on zero initialization.

v4: Add comment specifying behavior if device does not support _F_ASID

v3: Deleted unneeded space

---
  include/hw/virtio/vhost-vdpa.h |  8 +++++---
  hw/virtio/vhost-vdpa.c         | 29 +++++++++++++++++++----------
  net/vhost-vdpa.c               |  6 +++---
  hw/virtio/trace-events         |  4 ++--
  4 files changed, 29 insertions(+), 18 deletions(-)
Quoted 11 lines

include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 32

Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1 -0
index 1111d85643..6560bb9d78 100644
@@ -29,6 +29,7 @@ typedef struct vhost_vdpa {
      int index;
      uint32_t msg_type;
      bool iotlb_batch_begin_sent;
+    uint32_t address_space_id;

So the trick is let device specific code to zero this during allocation?

Yes, but I don't see how that is a trick :). All other parameters also trust it to be 0 at allocation.

Quoted 37 lines
Quoted 25 lines
include/hw/virtio/vhost-vdpa.h +4 -3
      MemoryListener listener;
      struct vhost_vdpa_iova_range iova_range;
      uint64_t acked_features;
@@ -42,8 +43,9 @@ typedef struct vhost_vdpa {
      VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX];
  } VhostVDPA;
 
-int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,
-                       void *vaddr, bool readonly);
-int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, hwaddr iova, hwaddr size);
+int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                       hwaddr size, void *vaddr, bool readonly);
+int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                         hwaddr size);
 
  #endif
hw/virtio/vhost-vdpa.c +0 -1
index 23efb8f49d..8fd32ba32b 100644
@@ -72,22 +72,24 @@ static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,
      return false;
  }
 
-int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,

hw/virtio/vhost-vdpa.c static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section, · line 83

Quoted 10 lines
hw/virtio/vhost-vdpa.c +3 -1
-                       void *vaddr, bool readonly)
+int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                       hwaddr size, void *vaddr, bool readonly)
  {
      struct vhost_msg_v2 msg = {};
      int fd = v->device_fd;
      int ret = 0;
 
      msg.type = v->msg_type;
+    msg.asid = asid; /* 0 if vdpa device does not support asid */

The comment here is confusing. If this is a requirement, we need either

Quoted 8 lines
Quoted 6 lines

1) doc this

or

2) perform necessary checks in the function itself.

I only documented it in vhost_vdpa_dma_unmap and now I realize it. Would it work to just copy that comment here?

Probably, and let's move the comment above the function definition.

Quoted 51 lines
Quoted 45 lines
Quoted 45 lines
hw/virtio/vhost-vdpa.c +11 -4
      msg.iotlb.iova = iova;
      msg.iotlb.size = size;
      msg.iotlb.uaddr = (uint64_t)(uintptr_t)vaddr;
      msg.iotlb.perm = readonly ? VHOST_ACCESS_RO : VHOST_ACCESS_RW;
      msg.iotlb.type = VHOST_IOTLB_UPDATE;
 
-   trace_vhost_vdpa_dma_map(v, fd, msg.type, msg.iotlb.iova, msg.iotlb.size,
-                            msg.iotlb.uaddr, msg.iotlb.perm, msg.iotlb.type);
+    trace_vhost_vdpa_dma_map(v, fd, msg.type, msg.asid, msg.iotlb.iova,
+                             msg.iotlb.size, msg.iotlb.uaddr, msg.iotlb.perm,
+                             msg.iotlb.type);
 
      if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
          error_report("failed to write, fd=%d, errno=%d (%s)",
@@ -98,18 +100,24 @@ int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,
      return ret;
  }
 
-int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, hwaddr iova, hwaddr size)
+int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                         hwaddr size)
  {
      struct vhost_msg_v2 msg = {};
      int fd = v->device_fd;
      int ret = 0;
 
      msg.type = v->msg_type;
+    /*
+     * The caller must set asid = 0 if the device does not support asid.
+     * This is not an ABI break since it is set to 0 by the initializer anyway.
+     */
+    msg.asid = asid;
      msg.iotlb.iova = iova;
      msg.iotlb.size = size;
      msg.iotlb.type = VHOST_IOTLB_INVALIDATE;
 
-    trace_vhost_vdpa_dma_unmap(v, fd, msg.type, msg.iotlb.iova,
+    trace_vhost_vdpa_dma_unmap(v, fd, msg.type, msg.asid, msg.iotlb.iova,
                                 msg.iotlb.size, msg.iotlb.type);
 
      if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
@@ -229,7 +237,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
      }
 
Quoted 6 lines

hw/virtio/vhost-vdpa.c static void vhost_vdpa_listener_region_add(MemoryListener *listener, · line 240

Quoted 3 lines
hw/virtio/vhost-vdpa.c +1 -1
      vhost_vdpa_iotlb_batch_begin_once(v);
-    ret = vhost_vdpa_dma_map(v, iova, int128_get64(llsize),
+    ret = vhost_vdpa_dma_map(v, 0, iova, int128_get64(llsize),

Can we use v->address_space_id here? Then we don't need to modify this line when we support multiple asids logic in the future.

Quoted 5 lines

The registered memory listener is the one of the last vhost_vdpa, the one that handles the last queue.

If all data virtqueues are not shadowed but CVQ is, v->address_space_id is 1 with the current code.

Ok, right. So let's add a comment here. It would be even better to define the macro for data vq asid in this patch.

Thanks

Quoted 89 lines

But the listener is
actually mapping the ASID 0, not 1.

Another alternative is to register it to the last data virtqueue, not the last queue of vhost_vdpa. But it is hard to express it in a generic way at virtio/vhost-vdpa.c . To have a boolean indicating the vhost_vdpa we want to register its memory listener?

It seems easier to me to simply assign 0 at GPA translations. If SVQ is enabled for all queues, then 0 is GPA to qemu's VA + SVQ stuff. If it is not, 0 is always GPA to qemu's VA.

Thanks!

Quoted 75 lines

Thanks

Quoted 73 lines
hw/virtio/vhost-vdpa.c +4 -3
                               vaddr, section->readonly);
      if (ret) {
          error_report("vhost vdpa map fail!");
@@ -303,7 +311,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
          vhost_iova_tree_remove(v->iova_tree, *result);
      }
      vhost_vdpa_iotlb_batch_begin_once(v);
-    ret = vhost_vdpa_dma_unmap(v, iova, int128_get64(llsize));
+    ret = vhost_vdpa_dma_unmap(v, 0, iova, int128_get64(llsize));
      if (ret) {
          error_report("vhost_vdpa dma unmap error!");
      }
@@ -884,7 +892,7 @@ static void vhost_vdpa_svq_unmap_ring(struct vhost_vdpa *v, hwaddr addr)
      }
 
      size = ROUND_UP(result->size, qemu_real_host_page_size());
-    r = vhost_vdpa_dma_unmap(v, result->iova, size);
+    r = vhost_vdpa_dma_unmap(v, v->address_space_id, result->iova, size);
      if (unlikely(r < 0)) {
          error_report("Unable to unmap SVQ vring: %s (%d)", g_strerror(-r), -r);
          return;
@@ -924,7 +932,8 @@ static bool vhost_vdpa_svq_map_ring(struct vhost_vdpa *v, DMAMap *needle,
          return false;
      }
 
-    r = vhost_vdpa_dma_map(v, needle->iova, needle->size + 1,
+    r = vhost_vdpa_dma_map(v, v->address_space_id, needle->iova,
+                           needle->size + 1,
                             (void *)(uintptr_t)needle->translated_addr,
                             needle->perm == IOMMU_RO);
      if (unlikely(r != 0)) {
net/vhost-vdpa.c +3 -3
index fb35b17ab4..ca1acc0410 100644
@@ -258,7 +258,7 @@ static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr)
          return;
      }
 
-    r = vhost_vdpa_dma_unmap(v, map->iova, map->size + 1);
+    r = vhost_vdpa_dma_unmap(v, v->address_space_id, map->iova, map->size + 1);
      if (unlikely(r != 0)) {
          error_report("Device cannot unmap: %s(%d)", g_strerror(r), r);
      }
@@ -298,8 +298,8 @@ static int vhost_vdpa_cvq_map_buf(struct vhost_vdpa *v, void *buf, size_t size,
          return r;
      }
 
-    r = vhost_vdpa_dma_map(v, map.iova, vhost_vdpa_net_cvq_cmd_page_len(), buf,
-                           !write);
+    r = vhost_vdpa_dma_map(v, v->address_space_id, map.iova,
+                           vhost_vdpa_net_cvq_cmd_page_len(), buf, !write);
      if (unlikely(r < 0)) {
          goto dma_map_err;
      }
hw/virtio/trace-events +2 -3
index 820dadc26c..0ad9390307 100644
@@ -30,8 +30,8 @@ vhost_user_write(uint32_t req, uint32_t flags) "req:%d flags:0x%"PRIx32""
  vhost_user_create_notifier(int idx, void *n) "idx:%d n:%p"
 
  # vhost-vdpa.c
-vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8
-vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8
+vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8
+vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8
  vhost_vdpa_listener_begin_batch(void *v, int fd, uint32_t msg_type, uint8_t type)  "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
  vhost_vdpa_listener_commit(void *v, int fd, uint32_t msg_type, uint8_t type)  "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
  vhost_vdpa_listener_region_add(void *vdpa, uint64_t iova, uint64_t llend, void *vaddr, bool readonly) "vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64" vaddr: %p read-only: %d"
--

2.31.1

#25 lore raw

Re: [PATCH v6 07/10] vdpa: Add asid parameter to vhost_vdpa_dma_map/unmap

Quoted 147 lines · Jason Wang
Quoted 77 lines · Eugenio Perez Martin
Quoted 25 lines · Jason Wang
Quoted 24 lines · Eugenio Pérez

So the caller can choose which ASID is destined.

No need to update the batch functions as they will always be called from memory listener updates at the moment. Memory listener updates will always update ASID 0, as it's the passthrough ASID.

All vhost devices's ASID are 0 at this moment.

---
v5:
* Solve conflict, now vhost_vdpa_svq_unmap_ring returns void
* Change comment on zero initialization.

v4: Add comment specifying behavior if device does not support _F_ASID

v3: Deleted unneeded space

---
  include/hw/virtio/vhost-vdpa.h |  8 +++++---
  hw/virtio/vhost-vdpa.c         | 29 +++++++++++++++++++----------
  net/vhost-vdpa.c               |  6 +++---
  hw/virtio/trace-events         |  4 ++--
  4 files changed, 29 insertions(+), 18 deletions(-)
Quoted 11 lines

include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 32

Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1 -0
index 1111d85643..6560bb9d78 100644
@@ -29,6 +29,7 @@ typedef struct vhost_vdpa {
      int index;
      uint32_t msg_type;
      bool iotlb_batch_begin_sent;
+    uint32_t address_space_id;

So the trick is let device specific code to zero this during allocation?

Yes, but I don't see how that is a trick :). All other parameters also trust it to be 0 at allocation.

Quoted 37 lines
Quoted 25 lines
include/hw/virtio/vhost-vdpa.h +4 -3
      MemoryListener listener;
      struct vhost_vdpa_iova_range iova_range;
      uint64_t acked_features;
@@ -42,8 +43,9 @@ typedef struct vhost_vdpa {
      VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX];
  } VhostVDPA;
 
-int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,
-                       void *vaddr, bool readonly);
-int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, hwaddr iova, hwaddr size);
+int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                       hwaddr size, void *vaddr, bool readonly);
+int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                         hwaddr size);
 
  #endif
hw/virtio/vhost-vdpa.c +0 -1
index 23efb8f49d..8fd32ba32b 100644
@@ -72,22 +72,24 @@ static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,
      return false;
  }
 
-int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,

hw/virtio/vhost-vdpa.c static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section, · line 83

Quoted 10 lines
hw/virtio/vhost-vdpa.c +3 -1
-                       void *vaddr, bool readonly)
+int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                       hwaddr size, void *vaddr, bool readonly)
  {
      struct vhost_msg_v2 msg = {};
      int fd = v->device_fd;
      int ret = 0;
 
      msg.type = v->msg_type;
+    msg.asid = asid; /* 0 if vdpa device does not support asid */

The comment here is confusing. If this is a requirement, we need either

Quoted 8 lines
Quoted 6 lines

1) doc this

or

2) perform necessary checks in the function itself.

I only documented it in vhost_vdpa_dma_unmap and now I realize it. Would it work to just copy that comment here?

Probably, and let's move the comment above the function definition.

Quoted 54 lines
Quoted 45 lines
Quoted 45 lines
hw/virtio/vhost-vdpa.c +11 -4
      msg.iotlb.iova = iova;
      msg.iotlb.size = size;
      msg.iotlb.uaddr = (uint64_t)(uintptr_t)vaddr;
      msg.iotlb.perm = readonly ? VHOST_ACCESS_RO : VHOST_ACCESS_RW;
      msg.iotlb.type = VHOST_IOTLB_UPDATE;
 
-   trace_vhost_vdpa_dma_map(v, fd, msg.type, msg.iotlb.iova, msg.iotlb.size,
-                            msg.iotlb.uaddr, msg.iotlb.perm, msg.iotlb.type);
+    trace_vhost_vdpa_dma_map(v, fd, msg.type, msg.asid, msg.iotlb.iova,
+                             msg.iotlb.size, msg.iotlb.uaddr, msg.iotlb.perm,
+                             msg.iotlb.type);
 
      if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
          error_report("failed to write, fd=%d, errno=%d (%s)",
@@ -98,18 +100,24 @@ int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,
      return ret;
  }
 
-int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, hwaddr iova, hwaddr size)
+int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                         hwaddr size)
  {
      struct vhost_msg_v2 msg = {};
      int fd = v->device_fd;
      int ret = 0;
 
      msg.type = v->msg_type;
+    /*
+     * The caller must set asid = 0 if the device does not support asid.
+     * This is not an ABI break since it is set to 0 by the initializer anyway.
+     */
+    msg.asid = asid;
      msg.iotlb.iova = iova;
      msg.iotlb.size = size;
      msg.iotlb.type = VHOST_IOTLB_INVALIDATE;
 
-    trace_vhost_vdpa_dma_unmap(v, fd, msg.type, msg.iotlb.iova,
+    trace_vhost_vdpa_dma_unmap(v, fd, msg.type, msg.asid, msg.iotlb.iova,
                                 msg.iotlb.size, msg.iotlb.type);
 
      if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
@@ -229,7 +237,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
      }
 
Quoted 6 lines

hw/virtio/vhost-vdpa.c static void vhost_vdpa_listener_region_add(MemoryListener *listener, · line 240

Quoted 3 lines
hw/virtio/vhost-vdpa.c +1 -1
      vhost_vdpa_iotlb_batch_begin_once(v);
-    ret = vhost_vdpa_dma_map(v, iova, int128_get64(llsize),
+    ret = vhost_vdpa_dma_map(v, 0, iova, int128_get64(llsize),

Can we use v->address_space_id here? Then we don't need to modify this line when we support multiple asids logic in the future.

The registered memory listener is the one of the last vhost_vdpa, the one that handles the last queue.

Quoted 7 lines
Quoted 2 lines

If all data virtqueues are not shadowed but CVQ is, v->address_space_id is 1 with the current code.

Ok, right. So let's add a comment here. It would be even better to define the macro for data vq asid in this patch.

I agree that it must be changed to a macro.

However, currently the _ASID macros belong to net/ . Maybe to declare VHOST_VDPA_GPA_ASID in include/hw/virtio/vhost-vdpa.h and just let VHOST_VDPA_NET_CVQ_ASID in net/vhost-vdpa.c?

Thanks!

Quoted 94 lines

Thanks

Quoted 89 lines

But the listener is
actually mapping the ASID 0, not 1.

Another alternative is to register it to the last data virtqueue, not the last queue of vhost_vdpa. But it is hard to express it in a generic way at virtio/vhost-vdpa.c . To have a boolean indicating the vhost_vdpa we want to register its memory listener?

It seems easier to me to simply assign 0 at GPA translations. If SVQ is enabled for all queues, then 0 is GPA to qemu's VA + SVQ stuff. If it is not, 0 is always GPA to qemu's VA.

Thanks!

Quoted 75 lines

Thanks

Quoted 73 lines
hw/virtio/vhost-vdpa.c +4 -3
                               vaddr, section->readonly);
      if (ret) {
          error_report("vhost vdpa map fail!");
@@ -303,7 +311,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
          vhost_iova_tree_remove(v->iova_tree, *result);
      }
      vhost_vdpa_iotlb_batch_begin_once(v);
-    ret = vhost_vdpa_dma_unmap(v, iova, int128_get64(llsize));
+    ret = vhost_vdpa_dma_unmap(v, 0, iova, int128_get64(llsize));
      if (ret) {
          error_report("vhost_vdpa dma unmap error!");
      }
@@ -884,7 +892,7 @@ static void vhost_vdpa_svq_unmap_ring(struct vhost_vdpa *v, hwaddr addr)
      }
 
      size = ROUND_UP(result->size, qemu_real_host_page_size());
-    r = vhost_vdpa_dma_unmap(v, result->iova, size);
+    r = vhost_vdpa_dma_unmap(v, v->address_space_id, result->iova, size);
      if (unlikely(r < 0)) {
          error_report("Unable to unmap SVQ vring: %s (%d)", g_strerror(-r), -r);
          return;
@@ -924,7 +932,8 @@ static bool vhost_vdpa_svq_map_ring(struct vhost_vdpa *v, DMAMap *needle,
          return false;
      }
 
-    r = vhost_vdpa_dma_map(v, needle->iova, needle->size + 1,
+    r = vhost_vdpa_dma_map(v, v->address_space_id, needle->iova,
+                           needle->size + 1,
                             (void *)(uintptr_t)needle->translated_addr,
                             needle->perm == IOMMU_RO);
      if (unlikely(r != 0)) {
net/vhost-vdpa.c +3 -3
index fb35b17ab4..ca1acc0410 100644
@@ -258,7 +258,7 @@ static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr)
          return;
      }
 
-    r = vhost_vdpa_dma_unmap(v, map->iova, map->size + 1);
+    r = vhost_vdpa_dma_unmap(v, v->address_space_id, map->iova, map->size + 1);
      if (unlikely(r != 0)) {
          error_report("Device cannot unmap: %s(%d)", g_strerror(r), r);
      }
@@ -298,8 +298,8 @@ static int vhost_vdpa_cvq_map_buf(struct vhost_vdpa *v, void *buf, size_t size,
          return r;
      }
 
-    r = vhost_vdpa_dma_map(v, map.iova, vhost_vdpa_net_cvq_cmd_page_len(), buf,
-                           !write);
+    r = vhost_vdpa_dma_map(v, v->address_space_id, map.iova,
+                           vhost_vdpa_net_cvq_cmd_page_len(), buf, !write);
      if (unlikely(r < 0)) {
          goto dma_map_err;
      }
hw/virtio/trace-events +2 -3
index 820dadc26c..0ad9390307 100644
@@ -30,8 +30,8 @@ vhost_user_write(uint32_t req, uint32_t flags) "req:%d flags:0x%"PRIx32""
  vhost_user_create_notifier(int idx, void *n) "idx:%d n:%p"
 
  # vhost-vdpa.c
-vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8
-vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8
+vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8
+vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8
  vhost_vdpa_listener_begin_batch(void *v, int fd, uint32_t msg_type, uint8_t type)  "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
  vhost_vdpa_listener_commit(void *v, int fd, uint32_t msg_type, uint8_t type)  "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
  vhost_vdpa_listener_region_add(void *vdpa, uint64_t iova, uint64_t llend, void *vaddr, bool readonly) "vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64" vaddr: %p read-only: %d"
--

2.31.1

#26 lore raw

Re: [PATCH v6 07/10] vdpa: Add asid parameter to vhost_vdpa_dma_map/unmap

Quoted 151 lines · Eugenio Perez Martin
Quoted 141 lines · Jason Wang
Quoted 77 lines · Eugenio Perez Martin
Quoted 25 lines · Jason Wang
Quoted 24 lines · Eugenio Pérez

So the caller can choose which ASID is destined.

No need to update the batch functions as they will always be called from memory listener updates at the moment. Memory listener updates will always update ASID 0, as it's the passthrough ASID.

All vhost devices's ASID are 0 at this moment.

---
v5:
* Solve conflict, now vhost_vdpa_svq_unmap_ring returns void
* Change comment on zero initialization.

v4: Add comment specifying behavior if device does not support _F_ASID

v3: Deleted unneeded space

---
   include/hw/virtio/vhost-vdpa.h |  8 +++++---
   hw/virtio/vhost-vdpa.c         | 29 +++++++++++++++++++----------
   net/vhost-vdpa.c               |  6 +++---
   hw/virtio/trace-events         |  4 ++--
   4 files changed, 29 insertions(+), 18 deletions(-)
Quoted 11 lines

include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 32

Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1 -0
index 1111d85643..6560bb9d78 100644
@@ -29,6 +29,7 @@ typedef struct vhost_vdpa {
       int index;
       uint32_t msg_type;
       bool iotlb_batch_begin_sent;
+    uint32_t address_space_id;

So the trick is let device specific code to zero this during allocation?

Yes, but I don't see how that is a trick :). All other parameters also trust it to be 0 at allocation.

Quoted 37 lines
Quoted 25 lines
include/hw/virtio/vhost-vdpa.h +4 -3
       MemoryListener listener;
       struct vhost_vdpa_iova_range iova_range;
       uint64_t acked_features;
@@ -42,8 +43,9 @@ typedef struct vhost_vdpa {
       VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX];
   } VhostVDPA;
 
-int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,
-                       void *vaddr, bool readonly);
-int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, hwaddr iova, hwaddr size);
+int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                       hwaddr size, void *vaddr, bool readonly);
+int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                         hwaddr size);
 
   #endif
hw/virtio/vhost-vdpa.c +0 -1
index 23efb8f49d..8fd32ba32b 100644
@@ -72,22 +72,24 @@ static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,
       return false;
   }
 
-int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,

hw/virtio/vhost-vdpa.c static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section, · line 83

Quoted 10 lines
hw/virtio/vhost-vdpa.c +3 -1
-                       void *vaddr, bool readonly)
+int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                       hwaddr size, void *vaddr, bool readonly)
   {
       struct vhost_msg_v2 msg = {};
       int fd = v->device_fd;
       int ret = 0;
 
       msg.type = v->msg_type;
+    msg.asid = asid; /* 0 if vdpa device does not support asid */

The comment here is confusing. If this is a requirement, we need either

Quoted 8 lines
Quoted 6 lines

1) doc this

or

2) perform necessary checks in the function itself.

I only documented it in vhost_vdpa_dma_unmap and now I realize it. Would it work to just copy that comment here?

Probably, and let's move the comment above the function definition.

Quoted 50 lines
Quoted 50 lines
Quoted 42 lines
hw/virtio/vhost-vdpa.c +11 -4
       msg.iotlb.iova = iova;
       msg.iotlb.size = size;
       msg.iotlb.uaddr = (uint64_t)(uintptr_t)vaddr;
       msg.iotlb.perm = readonly ? VHOST_ACCESS_RO : VHOST_ACCESS_RW;
       msg.iotlb.type = VHOST_IOTLB_UPDATE;
 
-   trace_vhost_vdpa_dma_map(v, fd, msg.type, msg.iotlb.iova, msg.iotlb.size,
-                            msg.iotlb.uaddr, msg.iotlb.perm, msg.iotlb.type);
+    trace_vhost_vdpa_dma_map(v, fd, msg.type, msg.asid, msg.iotlb.iova,
+                             msg.iotlb.size, msg.iotlb.uaddr, msg.iotlb.perm,
+                             msg.iotlb.type);
 
       if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
           error_report("failed to write, fd=%d, errno=%d (%s)",
@@ -98,18 +100,24 @@ int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,
       return ret;
   }
 
-int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, hwaddr iova, hwaddr size)
+int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+                         hwaddr size)
   {
       struct vhost_msg_v2 msg = {};
       int fd = v->device_fd;
       int ret = 0;
 
       msg.type = v->msg_type;
+    /*
+     * The caller must set asid = 0 if the device does not support asid.
+     * This is not an ABI break since it is set to 0 by the initializer anyway.
+     */
+    msg.asid = asid;
       msg.iotlb.iova = iova;
       msg.iotlb.size = size;
       msg.iotlb.type = VHOST_IOTLB_INVALIDATE;
 
-    trace_vhost_vdpa_dma_unmap(v, fd, msg.type, msg.iotlb.iova,
+    trace_vhost_vdpa_dma_unmap(v, fd, msg.type, msg.asid, msg.iotlb.iova,
                                  msg.iotlb.size, msg.iotlb.type);
 
       if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {

hw/virtio/vhost-vdpa.c static void vhost_vdpa_listener_region_add(MemoryListener *listener, · line 240

Quoted 5 lines
hw/virtio/vhost-vdpa.c +1 -1
@@ -229,7 +237,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
       }
 
       vhost_vdpa_iotlb_batch_begin_once(v);
-    ret = vhost_vdpa_dma_map(v, iova, int128_get64(llsize),
+    ret = vhost_vdpa_dma_map(v, 0, iova, int128_get64(llsize),

Can we use v->address_space_id here? Then we don't need to modify this line when we support multiple asids logic in the future.

Quoted 9 lines
Quoted 5 lines

The registered memory listener is the one of the last vhost_vdpa, the one that handles the last queue.

If all data virtqueues are not shadowed but CVQ is, v->address_space_id is 1 with the current code.

Ok, right. So let's add a comment here. It would be even better to define the macro for data vq asid in this patch.

Quoted 7 lines

I agree that it must be changed to a macro.

However, currently the _ASID macros belong to net/ . Maybe to declare VHOST_VDPA_GPA_ASID in include/hw/virtio/vhost-vdpa.h and just let VHOST_VDPA_NET_CVQ_ASID in net/vhost-vdpa.c?

Thanks!

That should be fine.

Thanks

Quoted 93 lines
Quoted 93 lines

Thanks

Quoted 89 lines

But the listener is
actually mapping the ASID 0, not 1.

Another alternative is to register it to the last data virtqueue, not the last queue of vhost_vdpa. But it is hard to express it in a generic way at virtio/vhost-vdpa.c . To have a boolean indicating the vhost_vdpa we want to register its memory listener?

It seems easier to me to simply assign 0 at GPA translations. If SVQ is enabled for all queues, then 0 is GPA to qemu's VA + SVQ stuff. If it is not, 0 is always GPA to qemu's VA.

Thanks!

Quoted 75 lines

Thanks

Quoted 73 lines
hw/virtio/vhost-vdpa.c +4 -3
                                vaddr, section->readonly);
       if (ret) {
           error_report("vhost vdpa map fail!");
@@ -303,7 +311,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
           vhost_iova_tree_remove(v->iova_tree, *result);
       }
       vhost_vdpa_iotlb_batch_begin_once(v);
-    ret = vhost_vdpa_dma_unmap(v, iova, int128_get64(llsize));
+    ret = vhost_vdpa_dma_unmap(v, 0, iova, int128_get64(llsize));
       if (ret) {
           error_report("vhost_vdpa dma unmap error!");
       }
@@ -884,7 +892,7 @@ static void vhost_vdpa_svq_unmap_ring(struct vhost_vdpa *v, hwaddr addr)
       }
 
       size = ROUND_UP(result->size, qemu_real_host_page_size());
-    r = vhost_vdpa_dma_unmap(v, result->iova, size);
+    r = vhost_vdpa_dma_unmap(v, v->address_space_id, result->iova, size);
       if (unlikely(r < 0)) {
           error_report("Unable to unmap SVQ vring: %s (%d)", g_strerror(-r), -r);
           return;
@@ -924,7 +932,8 @@ static bool vhost_vdpa_svq_map_ring(struct vhost_vdpa *v, DMAMap *needle,
           return false;
       }
 
-    r = vhost_vdpa_dma_map(v, needle->iova, needle->size + 1,
+    r = vhost_vdpa_dma_map(v, v->address_space_id, needle->iova,
+                           needle->size + 1,
                              (void *)(uintptr_t)needle->translated_addr,
                              needle->perm == IOMMU_RO);
       if (unlikely(r != 0)) {
net/vhost-vdpa.c +3 -3
index fb35b17ab4..ca1acc0410 100644
@@ -258,7 +258,7 @@ static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr)
           return;
       }
 
-    r = vhost_vdpa_dma_unmap(v, map->iova, map->size + 1);
+    r = vhost_vdpa_dma_unmap(v, v->address_space_id, map->iova, map->size + 1);
       if (unlikely(r != 0)) {
           error_report("Device cannot unmap: %s(%d)", g_strerror(r), r);
       }
@@ -298,8 +298,8 @@ static int vhost_vdpa_cvq_map_buf(struct vhost_vdpa *v, void *buf, size_t size,
           return r;
       }
 
-    r = vhost_vdpa_dma_map(v, map.iova, vhost_vdpa_net_cvq_cmd_page_len(), buf,
-                           !write);
+    r = vhost_vdpa_dma_map(v, v->address_space_id, map.iova,
+                           vhost_vdpa_net_cvq_cmd_page_len(), buf, !write);
       if (unlikely(r < 0)) {
           goto dma_map_err;
       }
hw/virtio/trace-events +2 -3
index 820dadc26c..0ad9390307 100644
@@ -30,8 +30,8 @@ vhost_user_write(uint32_t req, uint32_t flags) "req:%d flags:0x%"PRIx32""
   vhost_user_create_notifier(int idx, void *n) "idx:%d n:%p"
 
   # vhost-vdpa.c
-vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8
-vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8
+vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8
+vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8
   vhost_vdpa_listener_begin_batch(void *v, int fd, uint32_t msg_type, uint8_t type)  "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
   vhost_vdpa_listener_commit(void *v, int fd, uint32_t msg_type, uint8_t type)  "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
   vhost_vdpa_listener_region_add(void *vdpa, uint64_t iova, uint64_t llend, void *vaddr, bool readonly) "vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64" vaddr: %p read-only: %d"
--

2.31.1

#27 lore raw

PATCH v6 8/10 vdpa: Store x-svq parameter in VhostVDPAState

In reply to Eugenio Pérez

CVQ can be shadowed two ways:
- Device has x-svq=on parameter (current way)
- The device can isolate CVQ in its own vq group

QEMU needs to check for the second condition dynamically, because CVQ index is not known at initialization time. Since this is dynamic, the CVQ isolation could vary with different conditions, making it possible to go from "not isolated group" to "isolated".

Saving the cmdline parameter in an extra field so we never disable CVQ SVQ in case the device was started with cmdline.

---
 net/vhost-vdpa.c | 3 +++
 1 file changed, 3 insertions(+)
net/vhost-vdpa.c +3 -0
index ca1acc0410..85a318faca 100644
@@ -38,6 +38,8 @@ typedef struct VhostVDPAState {
     void *cvq_cmd_out_buffer;
     virtio_net_ctrl_ack *status;
 
+    /* The device always have SVQ enabled */
+    bool always_svq;
     bool started;
 } VhostVDPAState;
 
@@ -566,6 +568,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
 
     s->vhost_vdpa.device_fd = vdpa_device_fd;
     s->vhost_vdpa.index = queue_pair_index;
+    s->always_svq = svq;
     s->vhost_vdpa.shadow_vqs_enabled = svq;
     s->vhost_vdpa.iova_tree = iova_tree;
     if (!is_datapath) {
Signature · 1 line
2.31.1
#28 lore raw

PATCH v6 9/10 vdpa: Add listener_shadow_vq to vhost_vdpa

In reply to Eugenio Pérez

The memory listener that thells the device how to convert GPA to qemu's va is registered against CVQ vhost_vdpa. This series try to map the memory listener translations to ASID 0, while it maps the CVQ ones to ASID 1.

Let's tell the listener if it needs to register them on iova tree or not.

---
v5: Solve conflict about vhost_iova_tree_remove accepting mem_region by
value.

---
 include/hw/virtio/vhost-vdpa.h | 2 ++
 hw/virtio/vhost-vdpa.c         | 6 +++---
 net/vhost-vdpa.c               | 1 +
 3 files changed, 6 insertions(+), 3 deletions(-)
include/hw/virtio/vhost-vdpa.h +2 -0
index 6560bb9d78..0c3ed2d69b 100644
@@ -34,6 +34,8 @@ typedef struct vhost_vdpa {
     struct vhost_vdpa_iova_range iova_range;
     uint64_t acked_features;
     bool shadow_vqs_enabled;
+    /* The listener must send iova tree addresses, not GPA */
+    bool listener_shadow_vq;
     /* IOVA mapping used by the Shadow Virtqueue */
     VhostIOVATree *iova_tree;
     GPtrArray *shadow_vqs;
hw/virtio/vhost-vdpa.c +3 -3
index 8fd32ba32b..e3914fa40e 100644
@@ -220,7 +220,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
                                          vaddr, section->readonly);
 
     llsize = int128_sub(llend, int128_make64(iova));
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
         int r;
 
         mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,
@@ -247,7 +247,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
     return;
 
 fail_map:
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
         vhost_iova_tree_remove(v->iova_tree, mem_region);
     }
 
@@ -292,7 +292,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
 
     llsize = int128_sub(llend, int128_make64(iova));
 
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
         const DMAMap *result;
         const void *vaddr = memory_region_get_ram_ptr(section->mr) +
             section->offset_within_region +
net/vhost-vdpa.c +1 -0
index 85a318faca..02780ee37b 100644
@@ -570,6 +570,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
     s->vhost_vdpa.index = queue_pair_index;
     s->always_svq = svq;
     s->vhost_vdpa.shadow_vqs_enabled = svq;
+    s->vhost_vdpa.listener_shadow_vq = svq;
     s->vhost_vdpa.iova_tree = iova_tree;
     if (!is_datapath) {
         s->cvq_cmd_out_buffer = qemu_memalign(qemu_real_host_page_size(),
Signature · 1 line
2.31.1
#29 lore raw

Re: [PATCH v6 09/10] vdpa: Add listener_shadow_vq to vhost_vdpa

Quoted 68 lines · Eugenio Pérez

The memory listener that thells the device how to convert GPA to qemu's va is registered against CVQ vhost_vdpa. This series try to map the memory listener translations to ASID 0, while it maps the CVQ ones to ASID 1.

Let's tell the listener if it needs to register them on iova tree or not.

---
v5: Solve conflict about vhost_iova_tree_remove accepting mem_region by
value.

---
 include/hw/virtio/vhost-vdpa.h | 2 ++
 hw/virtio/vhost-vdpa.c         | 6 +++---
 net/vhost-vdpa.c               | 1 +
 3 files changed, 6 insertions(+), 3 deletions(-)
include/hw/virtio/vhost-vdpa.h +2 -0
index 6560bb9d78..0c3ed2d69b 100644
@@ -34,6 +34,8 @@ typedef struct vhost_vdpa {
     struct vhost_vdpa_iova_range iova_range;
     uint64_t acked_features;
     bool shadow_vqs_enabled;
+    /* The listener must send iova tree addresses, not GPA */
+    bool listener_shadow_vq;
     /* IOVA mapping used by the Shadow Virtqueue */
     VhostIOVATree *iova_tree;
     GPtrArray *shadow_vqs;
hw/virtio/vhost-vdpa.c +3 -3
index 8fd32ba32b..e3914fa40e 100644
@@ -220,7 +220,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
                                          vaddr, section->readonly);
 
     llsize = int128_sub(llend, int128_make64(iova));
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
         int r;
 
         mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,
@@ -247,7 +247,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
     return;
 
 fail_map:
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
         vhost_iova_tree_remove(v->iova_tree, mem_region);
     }
 
@@ -292,7 +292,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
 
     llsize = int128_sub(llend, int128_make64(iova));
 
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
         const DMAMap *result;
         const void *vaddr = memory_region_get_ram_ptr(section->mr) +
             section->offset_within_region +
net/vhost-vdpa.c +0 -0
index 85a318faca..02780ee37b 100644

net/vhost-vdpa.c static NetClientState *net_vhost_vdpa_init(NetClientState *peer, · line 573

Quoted 4 lines
net/vhost-vdpa.c +1 -0
@@ -570,6 +570,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
     s->vhost_vdpa.index = queue_pair_index;
     s->always_svq = svq;
     s->vhost_vdpa.shadow_vqs_enabled = svq;
+    s->vhost_vdpa.listener_shadow_vq = svq;

Any chance those above two can differ?

Thanks

Quoted 6 lines
net/vhost-vdpa.c +0 -1
     s->vhost_vdpa.iova_tree = iova_tree;
     if (!is_datapath) {
         s->cvq_cmd_out_buffer = qemu_memalign(qemu_real_host_page_size(),
--

2.31.1

#30 lore raw

Re: [PATCH v6 09/10] vdpa: Add listener_shadow_vq to vhost_vdpa

Quoted 60 lines · Jason Wang
Quoted 58 lines · Eugenio Pérez

The memory listener that thells the device how to convert GPA to qemu's va is registered against CVQ vhost_vdpa. This series try to map the memory listener translations to ASID 0, while it maps the CVQ ones to ASID 1.

Let's tell the listener if it needs to register them on iova tree or not.

---
v5: Solve conflict about vhost_iova_tree_remove accepting mem_region by
value.

---
 include/hw/virtio/vhost-vdpa.h | 2 ++
 hw/virtio/vhost-vdpa.c         | 6 +++---
 net/vhost-vdpa.c               | 1 +
 3 files changed, 6 insertions(+), 3 deletions(-)
include/hw/virtio/vhost-vdpa.h +2 -0
index 6560bb9d78..0c3ed2d69b 100644
@@ -34,6 +34,8 @@ typedef struct vhost_vdpa {
     struct vhost_vdpa_iova_range iova_range;
     uint64_t acked_features;
     bool shadow_vqs_enabled;
+    /* The listener must send iova tree addresses, not GPA */
+    bool listener_shadow_vq;
     /* IOVA mapping used by the Shadow Virtqueue */
     VhostIOVATree *iova_tree;
     GPtrArray *shadow_vqs;
hw/virtio/vhost-vdpa.c +2 -2
index 8fd32ba32b..e3914fa40e 100644
@@ -220,7 +220,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
                                          vaddr, section->readonly);
 
     llsize = int128_sub(llend, int128_make64(iova));
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
         int r;
 
         mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,
@@ -247,7 +247,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
     return;
 
 fail_map:
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
         vhost_iova_tree_remove(v->iova_tree, mem_region);
     }
 
@@ -292,7 +292,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
 
     llsize = int128_sub(llend, int128_make64(iova));
 
Quoted 17 lines
Quoted 10 lines
hw/virtio/vhost-vdpa.c +1 -1
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
         const DMAMap *result;
         const void *vaddr = memory_region_get_ram_ptr(section->mr) +
             section->offset_within_region +
net/vhost-vdpa.c +0 -0
index 85a318faca..02780ee37b 100644

net/vhost-vdpa.c static NetClientState *net_vhost_vdpa_init(NetClientState *peer, · line 573

Quoted 4 lines
net/vhost-vdpa.c +1 -0
@@ -570,6 +570,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
     s->vhost_vdpa.index = queue_pair_index;
     s->always_svq = svq;
     s->vhost_vdpa.shadow_vqs_enabled = svq;
+    s->vhost_vdpa.listener_shadow_vq = svq;

Any chance those above two can differ?

If CVQ is shadowed but data VQs are not, shadow_vqs_enabled is true but listener_shadow_vq is not.

It is more clear in the next commit, where only shadow_vqs_enabled is set to true at vhost_vdpa_net_cvq_start.

Thanks!

Quoted 9 lines

Thanks

Quoted 6 lines
net/vhost-vdpa.c +0 -1
     s->vhost_vdpa.iova_tree = iova_tree;
     if (!is_datapath) {
         s->cvq_cmd_out_buffer = qemu_memalign(qemu_real_host_page_size(),
--

2.31.1

#31 lore raw

Re: [PATCH v6 09/10] vdpa: Add listener_shadow_vq to vhost_vdpa

Quoted 20 lines · Eugenio Perez Martin
Quoted 19 lines · Jason Wang
Quoted 18 lines · Eugenio Pérez

The memory listener that thells the device how to convert GPA to qemu's va is registered against CVQ vhost_vdpa. This series try to map the memory listener translations to ASID 0, while it maps the CVQ ones to ASID 1.

Let's tell the listener if it needs to register them on iova tree or not.

---
v5: Solve conflict about vhost_iova_tree_remove accepting mem_region by
value.

---
  include/hw/virtio/vhost-vdpa.h | 2 ++
  hw/virtio/vhost-vdpa.c         | 6 +++---
  net/vhost-vdpa.c               | 1 +
  3 files changed, 6 insertions(+), 3 deletions(-)
Quoted 9 lines
Quoted 9 lines

include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 37

Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1 -0
index 6560bb9d78..0c3ed2d69b 100644
@@ -34,6 +34,8 @@ typedef struct vhost_vdpa {
      struct vhost_vdpa_iova_range iova_range;
      uint64_t acked_features;
      bool shadow_vqs_enabled;
+    /* The listener must send iova tree addresses, not GPA */

Btw, cindy's vIOMMU series will make it not necessarily GPA any more.

Quoted 46 lines
Quoted 30 lines
Quoted 30 lines
include/hw/virtio/vhost-vdpa.h +1 -0
+    bool listener_shadow_vq;
      /* IOVA mapping used by the Shadow Virtqueue */
      VhostIOVATree *iova_tree;
      GPtrArray *shadow_vqs;
hw/virtio/vhost-vdpa.c +2 -2
index 8fd32ba32b..e3914fa40e 100644
@@ -220,7 +220,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
                                           vaddr, section->readonly);
 
      llsize = int128_sub(llend, int128_make64(iova));
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
          int r;
 
          mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,
@@ -247,7 +247,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
      return;
 
  fail_map:
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
          vhost_iova_tree_remove(v->iova_tree, mem_region);
      }
 
@@ -292,7 +292,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
 
      llsize = int128_sub(llend, int128_make64(iova));
 
Quoted 16 lines
Quoted 10 lines
hw/virtio/vhost-vdpa.c +1 -1
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
          const DMAMap *result;
          const void *vaddr = memory_region_get_ram_ptr(section->mr) +
              section->offset_within_region +
net/vhost-vdpa.c +0 -0
index 85a318faca..02780ee37b 100644

net/vhost-vdpa.c static NetClientState *net_vhost_vdpa_init(NetClientState *peer, · line 573

Quoted 4 lines
net/vhost-vdpa.c +1 -0
@@ -570,6 +570,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
      s->vhost_vdpa.index = queue_pair_index;
      s->always_svq = svq;
      s->vhost_vdpa.shadow_vqs_enabled = svq;
+    s->vhost_vdpa.listener_shadow_vq = svq;

Any chance those above two can differ?

Quoted 5 lines

If CVQ is shadowed but data VQs are not, shadow_vqs_enabled is true but listener_shadow_vq is not.

It is more clear in the next commit, where only shadow_vqs_enabled is set to true at vhost_vdpa_net_cvq_start.

Ok, the name looks a little bit confusing. I wonder if it's better to use shadow_cvq and shadow_data ?

Thanks

Quoted 11 lines

Thanks!

Quoted 8 lines

Thanks

Quoted 6 lines
net/vhost-vdpa.c +0 -1
      s->vhost_vdpa.iova_tree = iova_tree;
      if (!is_datapath) {
          s->cvq_cmd_out_buffer = qemu_memalign(qemu_real_host_page_size(),
--

2.31.1

#32 lore raw

Re: [PATCH v6 09/10] vdpa: Add listener_shadow_vq to vhost_vdpa

Quoted 23 lines · Jason Wang
Quoted 20 lines · Eugenio Perez Martin
Quoted 19 lines · Jason Wang
Quoted 18 lines · Eugenio Pérez

The memory listener that thells the device how to convert GPA to qemu's va is registered against CVQ vhost_vdpa. This series try to map the memory listener translations to ASID 0, while it maps the CVQ ones to ASID 1.

Let's tell the listener if it needs to register them on iova tree or not.

---
v5: Solve conflict about vhost_iova_tree_remove accepting mem_region by
value.

---
  include/hw/virtio/vhost-vdpa.h | 2 ++
  hw/virtio/vhost-vdpa.c         | 6 +++---
  net/vhost-vdpa.c               | 1 +
  3 files changed, 6 insertions(+), 3 deletions(-)
Quoted 13 lines
Quoted 9 lines
Quoted 9 lines

include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 37

Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1 -0
index 6560bb9d78..0c3ed2d69b 100644
@@ -34,6 +34,8 @@ typedef struct vhost_vdpa {
      struct vhost_vdpa_iova_range iova_range;
      uint64_t acked_features;
      bool shadow_vqs_enabled;
+    /* The listener must send iova tree addresses, not GPA */

Btw, cindy's vIOMMU series will make it not necessarily GPA any more.

Yes, this comment should be tuned then. But the SVQ iova_tree will not be equal to vIOMMU one because shadow vrings.

But maybe SVQ can inspect both instead of having all the duplicated entries.

Quoted 50 lines
Quoted 50 lines
Quoted 31 lines
Quoted 31 lines
include/hw/virtio/vhost-vdpa.h +1 -0
+    bool listener_shadow_vq;
      /* IOVA mapping used by the Shadow Virtqueue */
      VhostIOVATree *iova_tree;
      GPtrArray *shadow_vqs;
hw/virtio/vhost-vdpa.c +2 -2
index 8fd32ba32b..e3914fa40e 100644
@@ -220,7 +220,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
                                           vaddr, section->readonly);
 
      llsize = int128_sub(llend, int128_make64(iova));
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
          int r;
 
          mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,
@@ -247,7 +247,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
      return;
 
  fail_map:
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
          vhost_iova_tree_remove(v->iova_tree, mem_region);
      }
 
@@ -292,7 +292,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
 
      llsize = int128_sub(llend, int128_make64(iova));
 
Quoted 16 lines
Quoted 10 lines
hw/virtio/vhost-vdpa.c +1 -1
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
          const DMAMap *result;
          const void *vaddr = memory_region_get_ram_ptr(section->mr) +
              section->offset_within_region +
net/vhost-vdpa.c +0 -0
index 85a318faca..02780ee37b 100644

net/vhost-vdpa.c static NetClientState *net_vhost_vdpa_init(NetClientState *peer, · line 573

Quoted 4 lines
net/vhost-vdpa.c +1 -0
@@ -570,6 +570,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
      s->vhost_vdpa.index = queue_pair_index;
      s->always_svq = svq;
      s->vhost_vdpa.shadow_vqs_enabled = svq;
+    s->vhost_vdpa.listener_shadow_vq = svq;

Any chance those above two can differ?

If CVQ is shadowed but data VQs are not, shadow_vqs_enabled is true but listener_shadow_vq is not.

Quoted 7 lines
Quoted 2 lines

It is more clear in the next commit, where only shadow_vqs_enabled is set to true at vhost_vdpa_net_cvq_start.

Ok, the name looks a little bit confusing. I wonder if it's better to use shadow_cvq and shadow_data ?

I'm ok with renaming it, but struct vhost_vdpa is generic across all kind of devices, and it does not know if it is a datapath or not for the moment.

Maybe listener_uses_iova_tree?

Thanks!

#33 lore raw

Re: [PATCH v6 09/10] vdpa: Add listener_shadow_vq to vhost_vdpa

Quoted 92 lines · Eugenio Perez Martin
Quoted 22 lines · Jason Wang
Quoted 20 lines · Eugenio Perez Martin
Quoted 19 lines · Jason Wang
Quoted 18 lines · Eugenio Pérez

The memory listener that thells the device how to convert GPA to qemu's va is registered against CVQ vhost_vdpa. This series try to map the memory listener translations to ASID 0, while it maps the CVQ ones to ASID 1.

Let's tell the listener if it needs to register them on iova tree or not.

---
v5: Solve conflict about vhost_iova_tree_remove accepting mem_region by
value.

---
   include/hw/virtio/vhost-vdpa.h | 2 ++
   hw/virtio/vhost-vdpa.c         | 6 +++---
   net/vhost-vdpa.c               | 1 +
   3 files changed, 6 insertions(+), 3 deletions(-)
Quoted 12 lines
Quoted 9 lines
Quoted 9 lines

include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 37

Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1 -0
index 6560bb9d78..0c3ed2d69b 100644
@@ -34,6 +34,8 @@ typedef struct vhost_vdpa {
       struct vhost_vdpa_iova_range iova_range;
       uint64_t acked_features;
       bool shadow_vqs_enabled;
+    /* The listener must send iova tree addresses, not GPA */

Btw, cindy's vIOMMU series will make it not necessarily GPA any more.

Yes, this comment should be tuned then. But the SVQ iova_tree will not be equal to vIOMMU one because shadow vrings.

But maybe SVQ can inspect both instead of having all the duplicated entries.

Quoted 52 lines
Quoted 51 lines
Quoted 30 lines
Quoted 30 lines
include/hw/virtio/vhost-vdpa.h +1 -0
+    bool listener_shadow_vq;
       /* IOVA mapping used by the Shadow Virtqueue */
       VhostIOVATree *iova_tree;
       GPtrArray *shadow_vqs;
hw/virtio/vhost-vdpa.c +2 -2
index 8fd32ba32b..e3914fa40e 100644
@@ -220,7 +220,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
                                            vaddr, section->readonly);
 
       llsize = int128_sub(llend, int128_make64(iova));
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
           int r;
 
           mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,
@@ -247,7 +247,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
       return;
 
   fail_map:
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
           vhost_iova_tree_remove(v->iova_tree, mem_region);
       }
 
@@ -292,7 +292,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
 
       llsize = int128_sub(llend, int128_make64(iova));
 
Quoted 16 lines
Quoted 10 lines
hw/virtio/vhost-vdpa.c +1 -1
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
           const DMAMap *result;
           const void *vaddr = memory_region_get_ram_ptr(section->mr) +
               section->offset_within_region +
net/vhost-vdpa.c +0 -0
index 85a318faca..02780ee37b 100644

net/vhost-vdpa.c static NetClientState *net_vhost_vdpa_init(NetClientState *peer, · line 573

Quoted 4 lines
net/vhost-vdpa.c +1 -0
@@ -570,6 +570,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
       s->vhost_vdpa.index = queue_pair_index;
       s->always_svq = svq;
       s->vhost_vdpa.shadow_vqs_enabled = svq;
+    s->vhost_vdpa.listener_shadow_vq = svq;

Any chance those above two can differ?

If CVQ is shadowed but data VQs are not, shadow_vqs_enabled is true but listener_shadow_vq is not.

It is more clear in the next commit, where only shadow_vqs_enabled is set to true at vhost_vdpa_net_cvq_start.

Quoted 8 lines
Quoted 3 lines

Ok, the name looks a little bit confusing. I wonder if it's better to use shadow_cvq and shadow_data ?

I'm ok with renaming it, but struct vhost_vdpa is generic across all kind of devices, and it does not know if it is a datapath or not for the moment.

Maybe listener_uses_iova_tree?

I think "iova_tree" is something that is internal to svq implementation, it's better to define the name from the view of vhost_vdpa level.

Thanks

Quoted 3 lines

Thanks!

#34 lore raw

Re: [PATCH v6 09/10] vdpa: Add listener_shadow_vq to vhost_vdpa

Quoted 102 lines · Jason Wang
Quoted 99 lines · Eugenio Perez Martin
Quoted 22 lines · Jason Wang
Quoted 20 lines · Eugenio Perez Martin
Quoted 19 lines · Jason Wang
Quoted 18 lines · Eugenio Pérez

The memory listener that thells the device how to convert GPA to qemu's va is registered against CVQ vhost_vdpa. This series try to map the memory listener translations to ASID 0, while it maps the CVQ ones to ASID 1.

Let's tell the listener if it needs to register them on iova tree or not.

---
v5: Solve conflict about vhost_iova_tree_remove accepting mem_region by
value.

---
   include/hw/virtio/vhost-vdpa.h | 2 ++
   hw/virtio/vhost-vdpa.c         | 6 +++---
   net/vhost-vdpa.c               | 1 +
   3 files changed, 6 insertions(+), 3 deletions(-)
Quoted 12 lines
Quoted 9 lines
Quoted 9 lines

include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 37

Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1 -0
index 6560bb9d78..0c3ed2d69b 100644
@@ -34,6 +34,8 @@ typedef struct vhost_vdpa {
       struct vhost_vdpa_iova_range iova_range;
       uint64_t acked_features;
       bool shadow_vqs_enabled;
+    /* The listener must send iova tree addresses, not GPA */

Btw, cindy's vIOMMU series will make it not necessarily GPA any more.

Yes, this comment should be tuned then. But the SVQ iova_tree will not be equal to vIOMMU one because shadow vrings.

But maybe SVQ can inspect both instead of having all the duplicated entries.

Quoted 46 lines
Quoted 46 lines
Quoted 46 lines
Quoted 40 lines
include/hw/virtio/vhost-vdpa.h +1 -0
+    bool listener_shadow_vq;
       /* IOVA mapping used by the Shadow Virtqueue */
       VhostIOVATree *iova_tree;
       GPtrArray *shadow_vqs;
hw/virtio/vhost-vdpa.c +3 -3
index 8fd32ba32b..e3914fa40e 100644
@@ -220,7 +220,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
                                            vaddr, section->readonly);
 
       llsize = int128_sub(llend, int128_make64(iova));
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
           int r;
 
           mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,
@@ -247,7 +247,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
       return;
 
   fail_map:
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
           vhost_iova_tree_remove(v->iova_tree, mem_region);
       }
 
@@ -292,7 +292,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
 
       llsize = int128_sub(llend, int128_make64(iova));
 
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
           const DMAMap *result;
           const void *vaddr = memory_region_get_ram_ptr(section->mr) +
               section->offset_within_region +
net/vhost-vdpa.c +0 -0
index 85a318faca..02780ee37b 100644

net/vhost-vdpa.c static NetClientState *net_vhost_vdpa_init(NetClientState *peer, · line 573

Quoted 4 lines
net/vhost-vdpa.c +1 -0
@@ -570,6 +570,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
       s->vhost_vdpa.index = queue_pair_index;
       s->always_svq = svq;
       s->vhost_vdpa.shadow_vqs_enabled = svq;
+    s->vhost_vdpa.listener_shadow_vq = svq;

Any chance those above two can differ?

Quoted 9 lines
Quoted 5 lines

If CVQ is shadowed but data VQs are not, shadow_vqs_enabled is true but listener_shadow_vq is not.

It is more clear in the next commit, where only shadow_vqs_enabled is set to true at vhost_vdpa_net_cvq_start.

Ok, the name looks a little bit confusing. I wonder if it's better to use shadow_cvq and shadow_data ?

I'm ok with renaming it, but struct vhost_vdpa is generic across all kind of devices, and it does not know if it is a datapath or not for the moment.

Quoted 6 lines
Quoted 1 line

Maybe listener_uses_iova_tree?

I think "iova_tree" is something that is internal to svq implementation, it's better to define the name from the view of vhost_vdpa level.

I don't get this, vhost_vdpa struct already has a pointer to its iova_tree.

Thanks!

#35 lore raw

Re: [PATCH v6 09/10] vdpa: Add listener_shadow_vq to vhost_vdpa

Quoted 104 lines · Eugenio Perez Martin
Quoted 102 lines · Jason Wang
Quoted 99 lines · Eugenio Perez Martin
Quoted 22 lines · Jason Wang
Quoted 20 lines · Eugenio Perez Martin
Quoted 19 lines · Jason Wang
Quoted 18 lines · Eugenio Pérez

The memory listener that thells the device how to convert GPA to qemu's va is registered against CVQ vhost_vdpa. This series try to map the memory listener translations to ASID 0, while it maps the CVQ ones to ASID 1.

Let's tell the listener if it needs to register them on iova tree or not.

---
v5: Solve conflict about vhost_iova_tree_remove accepting mem_region by
value.

---
   include/hw/virtio/vhost-vdpa.h | 2 ++
   hw/virtio/vhost-vdpa.c         | 6 +++---
   net/vhost-vdpa.c               | 1 +
   3 files changed, 6 insertions(+), 3 deletions(-)
Quoted 12 lines
Quoted 9 lines
Quoted 9 lines

include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 37

Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1 -0
index 6560bb9d78..0c3ed2d69b 100644
@@ -34,6 +34,8 @@ typedef struct vhost_vdpa {
       struct vhost_vdpa_iova_range iova_range;
       uint64_t acked_features;
       bool shadow_vqs_enabled;
+    /* The listener must send iova tree addresses, not GPA */

Btw, cindy's vIOMMU series will make it not necessarily GPA any more.

Yes, this comment should be tuned then. But the SVQ iova_tree will not be equal to vIOMMU one because shadow vrings.

But maybe SVQ can inspect both instead of having all the duplicated entries.

Quoted 46 lines
Quoted 46 lines
Quoted 46 lines
Quoted 40 lines
include/hw/virtio/vhost-vdpa.h +1 -0
+    bool listener_shadow_vq;
       /* IOVA mapping used by the Shadow Virtqueue */
       VhostIOVATree *iova_tree;
       GPtrArray *shadow_vqs;
hw/virtio/vhost-vdpa.c +3 -3
index 8fd32ba32b..e3914fa40e 100644
@@ -220,7 +220,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
                                            vaddr, section->readonly);
 
       llsize = int128_sub(llend, int128_make64(iova));
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
           int r;
 
           mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,
@@ -247,7 +247,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
       return;
 
   fail_map:
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
           vhost_iova_tree_remove(v->iova_tree, mem_region);
       }
 
@@ -292,7 +292,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
 
       llsize = int128_sub(llend, int128_make64(iova));
 
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
           const DMAMap *result;
           const void *vaddr = memory_region_get_ram_ptr(section->mr) +
               section->offset_within_region +
net/vhost-vdpa.c +0 -0
index 85a318faca..02780ee37b 100644

net/vhost-vdpa.c static NetClientState *net_vhost_vdpa_init(NetClientState *peer, · line 573

Quoted 4 lines
net/vhost-vdpa.c +1 -0
@@ -570,6 +570,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
       s->vhost_vdpa.index = queue_pair_index;
       s->always_svq = svq;
       s->vhost_vdpa.shadow_vqs_enabled = svq;
+    s->vhost_vdpa.listener_shadow_vq = svq;

Any chance those above two can differ?

Quoted 9 lines
Quoted 5 lines

If CVQ is shadowed but data VQs are not, shadow_vqs_enabled is true but listener_shadow_vq is not.

It is more clear in the next commit, where only shadow_vqs_enabled is set to true at vhost_vdpa_net_cvq_start.

Ok, the name looks a little bit confusing. I wonder if it's better to use shadow_cvq and shadow_data ?

I'm ok with renaming it, but struct vhost_vdpa is generic across all kind of devices, and it does not know if it is a datapath or not for the moment.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Maybe listener_uses_iova_tree?

I think "iova_tree" is something that is internal to svq implementation, it's better to define the name from the view of vhost_vdpa level.

I don't get this, vhost_vdpa struct already has a pointer to its iova_tree.

Yes, this is a suggestion to improve the readability of the code. So what I meant is to have a name to demonstrate why we need to use iova_tree instead of "uses_iova_tree".

Thanks

Quoted 3 lines

Thanks!

#36 lore raw

Re: [PATCH v6 09/10] vdpa: Add listener_shadow_vq to vhost_vdpa

Quoted 114 lines · Jason Wang
Quoted 111 lines · Eugenio Perez Martin
Quoted 108 lines · Jason Wang
Quoted 92 lines · Eugenio Perez Martin
Quoted 22 lines · Jason Wang
Quoted 20 lines · Eugenio Perez Martin
Quoted 19 lines · Jason Wang
Quoted 18 lines · Eugenio Pérez

The memory listener that thells the device how to convert GPA to qemu's va is registered against CVQ vhost_vdpa. This series try to map the memory listener translations to ASID 0, while it maps the CVQ ones to ASID 1.

Let's tell the listener if it needs to register them on iova tree or not.

---
v5: Solve conflict about vhost_iova_tree_remove accepting mem_region by
value.

---
   include/hw/virtio/vhost-vdpa.h | 2 ++
   hw/virtio/vhost-vdpa.c         | 6 +++---
   net/vhost-vdpa.c               | 1 +
   3 files changed, 6 insertions(+), 3 deletions(-)
Quoted 12 lines
Quoted 9 lines
Quoted 9 lines

include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 37

Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1 -0
index 6560bb9d78..0c3ed2d69b 100644
@@ -34,6 +34,8 @@ typedef struct vhost_vdpa {
       struct vhost_vdpa_iova_range iova_range;
       uint64_t acked_features;
       bool shadow_vqs_enabled;
+    /* The listener must send iova tree addresses, not GPA */

Btw, cindy's vIOMMU series will make it not necessarily GPA any more.

Yes, this comment should be tuned then. But the SVQ iova_tree will not be equal to vIOMMU one because shadow vrings.

But maybe SVQ can inspect both instead of having all the duplicated entries.

Quoted 52 lines
Quoted 51 lines
Quoted 30 lines
Quoted 30 lines
include/hw/virtio/vhost-vdpa.h +1 -0
+    bool listener_shadow_vq;
       /* IOVA mapping used by the Shadow Virtqueue */
       VhostIOVATree *iova_tree;
       GPtrArray *shadow_vqs;
hw/virtio/vhost-vdpa.c +2 -2
index 8fd32ba32b..e3914fa40e 100644
@@ -220,7 +220,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
                                            vaddr, section->readonly);
 
       llsize = int128_sub(llend, int128_make64(iova));
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
           int r;
 
           mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,
@@ -247,7 +247,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
       return;
 
   fail_map:
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
           vhost_iova_tree_remove(v->iova_tree, mem_region);
       }
 
@@ -292,7 +292,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
 
       llsize = int128_sub(llend, int128_make64(iova));
 
Quoted 16 lines
Quoted 10 lines
hw/virtio/vhost-vdpa.c +1 -1
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
           const DMAMap *result;
           const void *vaddr = memory_region_get_ram_ptr(section->mr) +
               section->offset_within_region +
net/vhost-vdpa.c +0 -0
index 85a318faca..02780ee37b 100644

net/vhost-vdpa.c static NetClientState *net_vhost_vdpa_init(NetClientState *peer, · line 573

Quoted 4 lines
net/vhost-vdpa.c +1 -0
@@ -570,6 +570,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
       s->vhost_vdpa.index = queue_pair_index;
       s->always_svq = svq;
       s->vhost_vdpa.shadow_vqs_enabled = svq;
+    s->vhost_vdpa.listener_shadow_vq = svq;

Any chance those above two can differ?

If CVQ is shadowed but data VQs are not, shadow_vqs_enabled is true but listener_shadow_vq is not.

It is more clear in the next commit, where only shadow_vqs_enabled is set to true at vhost_vdpa_net_cvq_start.

Quoted 8 lines
Quoted 3 lines

Ok, the name looks a little bit confusing. I wonder if it's better to use shadow_cvq and shadow_data ?

I'm ok with renaming it, but struct vhost_vdpa is generic across all kind of devices, and it does not know if it is a datapath or not for the moment.

Maybe listener_uses_iova_tree?

I think "iova_tree" is something that is internal to svq implementation, it's better to define the name from the view of vhost_vdpa level.

Quoted 6 lines
Quoted 1 line

I don't get this, vhost_vdpa struct already has a pointer to its iova_tree.

Yes, this is a suggestion to improve the readability of the code. So what I meant is to have a name to demonstrate why we need to use iova_tree instead of "uses_iova_tree".

I understand.

Knowing that the listener will be always bound to data vqs (being net, blk, ...), I think it is ok to rename it to shadow_data.

But I think there is no way to add shadow_cvq properly from hw/virtio/vhost-vdpa.c , since we don't know if the vhost_vdpa belongs to a datapath or not. Would it work just to rename listener_shadow_vq to shadow_data?

Thanks!

#37 lore raw

Re: [PATCH v6 09/10] vdpa: Add listener_shadow_vq to vhost_vdpa

Quoted 125 lines · Eugenio Perez Martin
Quoted 114 lines · Jason Wang
Quoted 111 lines · Eugenio Perez Martin
Quoted 108 lines · Jason Wang
Quoted 92 lines · Eugenio Perez Martin
Quoted 22 lines · Jason Wang
Quoted 20 lines · Eugenio Perez Martin
Quoted 19 lines · Jason Wang
Quoted 18 lines · Eugenio Pérez

The memory listener that thells the device how to convert GPA to qemu's va is registered against CVQ vhost_vdpa. This series try to map the memory listener translations to ASID 0, while it maps the CVQ ones to ASID 1.

Let's tell the listener if it needs to register them on iova tree or not.

---
v5: Solve conflict about vhost_iova_tree_remove accepting mem_region by
value.

---
   include/hw/virtio/vhost-vdpa.h | 2 ++
   hw/virtio/vhost-vdpa.c         | 6 +++---
   net/vhost-vdpa.c               | 1 +
   3 files changed, 6 insertions(+), 3 deletions(-)
Quoted 12 lines
Quoted 9 lines
Quoted 9 lines

include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 37

Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1 -0
index 6560bb9d78..0c3ed2d69b 100644
@@ -34,6 +34,8 @@ typedef struct vhost_vdpa {
       struct vhost_vdpa_iova_range iova_range;
       uint64_t acked_features;
       bool shadow_vqs_enabled;
+    /* The listener must send iova tree addresses, not GPA */

Btw, cindy's vIOMMU series will make it not necessarily GPA any more.

Yes, this comment should be tuned then. But the SVQ iova_tree will not be equal to vIOMMU one because shadow vrings.

But maybe SVQ can inspect both instead of having all the duplicated entries.

Quoted 52 lines
Quoted 51 lines
Quoted 30 lines
Quoted 30 lines
include/hw/virtio/vhost-vdpa.h +1 -0
+    bool listener_shadow_vq;
       /* IOVA mapping used by the Shadow Virtqueue */
       VhostIOVATree *iova_tree;
       GPtrArray *shadow_vqs;
hw/virtio/vhost-vdpa.c +2 -2
index 8fd32ba32b..e3914fa40e 100644
@@ -220,7 +220,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
                                            vaddr, section->readonly);
 
       llsize = int128_sub(llend, int128_make64(iova));
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
           int r;
 
           mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,
@@ -247,7 +247,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
       return;
 
   fail_map:
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
           vhost_iova_tree_remove(v->iova_tree, mem_region);
       }
 
@@ -292,7 +292,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
 
       llsize = int128_sub(llend, int128_make64(iova));
 
Quoted 16 lines
Quoted 10 lines
hw/virtio/vhost-vdpa.c +1 -1
-    if (v->shadow_vqs_enabled) {
+    if (v->listener_shadow_vq) {
           const DMAMap *result;
           const void *vaddr = memory_region_get_ram_ptr(section->mr) +
               section->offset_within_region +
net/vhost-vdpa.c +0 -0
index 85a318faca..02780ee37b 100644

net/vhost-vdpa.c static NetClientState *net_vhost_vdpa_init(NetClientState *peer, · line 573

Quoted 4 lines
net/vhost-vdpa.c +1 -0
@@ -570,6 +570,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
       s->vhost_vdpa.index = queue_pair_index;
       s->always_svq = svq;
       s->vhost_vdpa.shadow_vqs_enabled = svq;
+    s->vhost_vdpa.listener_shadow_vq = svq;

Any chance those above two can differ?

If CVQ is shadowed but data VQs are not, shadow_vqs_enabled is true but listener_shadow_vq is not.

It is more clear in the next commit, where only shadow_vqs_enabled is set to true at vhost_vdpa_net_cvq_start.

Quoted 8 lines
Quoted 3 lines

Ok, the name looks a little bit confusing. I wonder if it's better to use shadow_cvq and shadow_data ?

I'm ok with renaming it, but struct vhost_vdpa is generic across all kind of devices, and it does not know if it is a datapath or not for the moment.

Maybe listener_uses_iova_tree?

I think "iova_tree" is something that is internal to svq implementation, it's better to define the name from the view of vhost_vdpa level.

Quoted 6 lines
Quoted 1 line

I don't get this, vhost_vdpa struct already has a pointer to its iova_tree.

Yes, this is a suggestion to improve the readability of the code. So what I meant is to have a name to demonstrate why we need to use iova_tree instead of "uses_iova_tree".

I understand.

Quoted 7 lines

Knowing that the listener will be always bound to data vqs (being net, blk, ...), I think it is ok to rename it to shadow_data.

But I think there is no way to add shadow_cvq properly from hw/virtio/vhost-vdpa.c , since we don't know if the vhost_vdpa belongs to a datapath or not. Would it work just to rename listener_shadow_vq to shadow_data?

This should work.

Thanks

Quoted 3 lines

Thanks!

#38 lore raw

PATCH v6 10/10 vdpa: Always start CVQ in SVQ mode

In reply to Eugenio Pérez

Isolate control virtqueue in its own group, allowing to intercept control commands but letting dataplane run totally passthrough to the guest.

---
v6:
* Disable control SVQ if the device does not support it because of features.

v5:
* Fixing the not adding cvq buffers when x-svq=on is specified.
* Move vring state in vhost_vdpa_get_vring_group instead of using a parameter.
* Rename VHOST_VDPA_NET_CVQ_PASSTHROUGH to VHOST_VDPA_NET_DATA_ASID

v4:
* Squash vhost_vdpa_cvq_group_is_independent.
* Rebased on last CVQ start series, that allocated CVQ cmd bufs at load
* Do not check for cvq index on vhost_vdpa_net_prepare, we only have one that callback registered in that NetClientInfo.

v3:
* Make asid related queries print a warning instead of returning an error and stop the start of qemu.

---
 hw/virtio/vhost-vdpa.c |   3 +-
 net/vhost-vdpa.c       | 138 ++++++++++++++++++++++++++++++++++++++---
 2 files changed, 132 insertions(+), 9 deletions(-)
hw/virtio/vhost-vdpa.c +2 -1
index e3914fa40e..6401e7efb1 100644
@@ -648,7 +648,8 @@ static int vhost_vdpa_set_backend_cap(struct vhost_dev *dev)
 {
     uint64_t features;
     uint64_t f = 0x1ULL << VHOST_BACKEND_F_IOTLB_MSG_V2 |
-        0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH;
+        0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH |
+        0x1ULL << VHOST_BACKEND_F_IOTLB_ASID;
     int r;
 
     if (vhost_vdpa_call(dev, VHOST_GET_BACKEND_FEATURES, &features)) {
net/vhost-vdpa.c +130 -8

210 lines not on this page · raw message

Signature · 1 line
2.31.1
#39 lore raw

Re: [PATCH v6 10/10] vdpa: Always start CVQ in SVQ mode

Quoted 2 lines · Eugenio Pérez

Isolate control virtqueue in its own group, allowing to intercept control commands but letting dataplane run totally passthrough to the guest.

I think we need to tweak the title to "vdpa: Always start CVQ in SVQ mode if possible". Since SVQ for CVQ can't be enabled without ASID support?

Quoted 46 lines

---
v6:
* Disable control SVQ if the device does not support it because of features.

v5:
* Fixing the not adding cvq buffers when x-svq=on is specified.
* Move vring state in vhost_vdpa_get_vring_group instead of using a parameter.
* Rename VHOST_VDPA_NET_CVQ_PASSTHROUGH to VHOST_VDPA_NET_DATA_ASID

v4:
* Squash vhost_vdpa_cvq_group_is_independent.
* Rebased on last CVQ start series, that allocated CVQ cmd bufs at load
* Do not check for cvq index on vhost_vdpa_net_prepare, we only have one that callback registered in that NetClientInfo.

v3:
* Make asid related queries print a warning instead of returning an error and stop the start of qemu.

---
  hw/virtio/vhost-vdpa.c |   3 +-
  net/vhost-vdpa.c       | 138 ++++++++++++++++++++++++++++++++++++++---
  2 files changed, 132 insertions(+), 9 deletions(-)
hw/virtio/vhost-vdpa.c +2 -1
index e3914fa40e..6401e7efb1 100644
@@ -648,7 +648,8 @@ static int vhost_vdpa_set_backend_cap(struct vhost_dev *dev)
  {
      uint64_t features;
      uint64_t f = 0x1ULL << VHOST_BACKEND_F_IOTLB_MSG_V2 |
-        0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH;
+        0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH |
+        0x1ULL << VHOST_BACKEND_F_IOTLB_ASID;
      int r;
 
      if (vhost_vdpa_call(dev, VHOST_GET_BACKEND_FEATURES, &features)) {
net/vhost-vdpa.c +0 -0
index 02780ee37b..7245ea70c6 100644

net/vhost-vdpa.c typedef struct VhostVDPAState { · line 42

Quoted 5 lines
net/vhost-vdpa.c +2 -0
@@ -38,6 +38,9 @@ typedef struct VhostVDPAState {
      void *cvq_cmd_out_buffer;
      virtio_net_ctrl_ack *status;
 
+    /* Number of address spaces supported by the device */
+    unsigned address_space_num;

I'm not sure this is the best place to store thing like this since it can cause confusion. We will have multiple VhostVDPAState when multiqueue is enabled.

Quoted 16 lines
net/vhost-vdpa.c +4 -0
+
      /* The device always have SVQ enabled */
      bool always_svq;
      bool started;
@@ -101,6 +104,9 @@ static const uint64_t vdpa_svq_device_features =
      BIT_ULL(VIRTIO_NET_F_RSC_EXT) |
      BIT_ULL(VIRTIO_NET_F_STANDBY);
 
+#define VHOST_VDPA_NET_DATA_ASID 0
+#define VHOST_VDPA_NET_CVQ_ASID 1
+
  VHostNetState *vhost_vdpa_get_vhost_net(NetClientState *nc)
  {
      VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc);
@@ -242,6 +248,34 @@ static NetClientInfo net_vhost_vdpa_info = {
          .check_peer_type = vhost_vdpa_check_peer_type,

net/vhost-vdpa.c static NetClientInfo net_vhost_vdpa_info = { · line 258

Quoted 10 lines
net/vhost-vdpa.c +8 -0
  };
 
+static uint32_t vhost_vdpa_get_vring_group(int device_fd, unsigned vq_index)
+{
+    struct vhost_vring_state state = {
+        .index = vq_index,
+    };
+    int r = ioctl(device_fd, VHOST_VDPA_GET_VRING_GROUP, &state);
+
+    return r < 0 ? 0 : state.num;

Assume 0 when ioctl() fail is probably not a good idea: errors in ioctl might be hidden. It would be better to fallback to 0 when ASID is not supported.

Quoted 42 lines
net/vhost-vdpa.c +30 -1
+}
+
+static int vhost_vdpa_set_address_space_id(struct vhost_vdpa *v,
+                                           unsigned vq_group,
+                                           unsigned asid_num)
+{
+    struct vhost_vring_state asid = {
+        .index = vq_group,
+        .num = asid_num,
+    };
+    int ret;
+
+    ret = ioctl(v->device_fd, VHOST_VDPA_SET_GROUP_ASID, &asid);
+    if (unlikely(ret < 0)) {
+        warn_report("Can't set vq group %u asid %u, errno=%d (%s)",
+            asid.index, asid.num, errno, g_strerror(errno));
+    }
+    return ret;
+}
+
  static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr)
  {
      VhostIOVATree *tree = v->iova_tree;
@@ -316,11 +350,54 @@ dma_map_err:
  static int vhost_vdpa_net_cvq_start(NetClientState *nc)
  {
      VhostVDPAState *s;
-    int r;
+    struct vhost_vdpa *v;
+    uint32_t cvq_group;
+    int cvq_index, r;
 
      assert(nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA);
 
      s = DO_UPCAST(VhostVDPAState, nc, nc);
+    v = &s->vhost_vdpa;
+
+    v->listener_shadow_vq = s->always_svq;
+    v->shadow_vqs_enabled = s->always_svq;
+    s->vhost_vdpa.address_space_id = VHOST_VDPA_NET_DATA_ASID;
+
+    if (s->always_svq) {

net/vhost-vdpa.c dma_map_err: · line 376

Quoted 10 lines
net/vhost-vdpa.c +10 -0
+        goto out;
+    }
+
+    if (s->address_space_num < 2) {
+        return 0;
+    }
+
+    if (!vhost_vdpa_net_valid_svq_features(v->dev->features, NULL)) {
+        return 0;
+    }

Any reason we do the above check during the start/stop? It should be easier to do that in the initialization.

Quoted 41 lines
net/vhost-vdpa.c +34 -0
+
+    /**
+     * Check if all the virtqueues of the virtio device are in a different vq
+     * than the last vq. VQ group of last group passed in cvq_group.
+     */
+    cvq_index = v->dev->vq_index_end - 1;
+    cvq_group = vhost_vdpa_get_vring_group(v->device_fd, cvq_index);
+    for (int i = 0; i < cvq_index; ++i) {
+        uint32_t group = vhost_vdpa_get_vring_group(v->device_fd, i);
+
+        if (unlikely(group == cvq_group)) {
+            warn_report("CVQ %u group is the same as VQ %u one (%u)", cvq_group,
+                        i, group);
+            return 0;
+        }
+    }
+
+    r = vhost_vdpa_set_address_space_id(v, cvq_group, VHOST_VDPA_NET_CVQ_ASID);
+    if (r == 0) {
+        v->shadow_vqs_enabled = true;
+        s->vhost_vdpa.address_space_id = VHOST_VDPA_NET_CVQ_ASID;
+    }
+
+out:
      if (!s->vhost_vdpa.shadow_vqs_enabled) {
          return 0;
      }
@@ -542,12 +619,38 @@ static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = {
      .avail_handler = vhost_vdpa_net_handle_ctrl_avail,
  };
 
+static uint32_t vhost_vdpa_get_as_num(int vdpa_device_fd)
+{
+    uint64_t features;
+    unsigned num_as;
+    int r;
+
+    r = ioctl(vdpa_device_fd, VHOST_GET_BACKEND_FEATURES, &features);
+    if (unlikely(r < 0)) {
+        warn_report("Cannot get backend features");
+        return 1;

net/vhost-vdpa.c static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = { · line 641

Quoted 10 lines
net/vhost-vdpa.c +10 -0
+    }
+
+    if (!(features & BIT_ULL(VHOST_BACKEND_F_IOTLB_ASID))) {
+        return 1;
+    }
+
+    r = ioctl(vdpa_device_fd, VHOST_VDPA_GET_AS_NUM, &num_as);
+    if (unlikely(r < 0)) {
+        warn_report("Cannot retrieve number of supported ASs");
+        return 1;

Let's return error here. This help to identify bugs of qemu or kernel.

Quoted 33 lines
net/vhost-vdpa.c +9 -0
+    }
+
+    return num_as;
+}
+
  static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
                                             const char *device,
                                             const char *name,
                                             int vdpa_device_fd,
                                             int queue_pair_index,
                                             int nvqs,
+                                           unsigned nas,
                                             bool is_datapath,
                                             bool svq,
                                             VhostIOVATree *iova_tree)
@@ -566,6 +669,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
      qemu_set_info_str(nc, TYPE_VHOST_VDPA);
      s = DO_UPCAST(VhostVDPAState, nc, nc);
 
+    s->address_space_num = nas;
      s->vhost_vdpa.device_fd = vdpa_device_fd;
      s->vhost_vdpa.index = queue_pair_index;
      s->always_svq = svq;
@@ -652,6 +756,8 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
      g_autoptr(VhostIOVATree) iova_tree = NULL;
      NetClientState *nc;
      int queue_pairs, r, i = 0, has_cvq = 0;
+    unsigned num_as = 1;
+    bool svq_cvq;
 
      assert(netdev->type == NET_CLIENT_DRIVER_VHOST_VDPA);
      opts = &netdev->u.vhost_vdpa;

net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 806

Quoted 10 lines
net/vhost-vdpa.c +5 -2
@@ -693,12 +799,28 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
          return queue_pairs;
      }
 
-    if (opts->x_svq) {
-        struct vhost_vdpa_iova_range iova_range;
+    svq_cvq = opts->x_svq;
+    if (has_cvq && !opts->x_svq) {
+        num_as = vhost_vdpa_get_as_num(vdpa_device_fd);
+        svq_cvq = num_as > 1;
+    }

The above check is not easy to follow, how about?

svq_cvq = vhost_vdpa_get_as_num() > 1 ? true : opts->x_svq;

net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 808

Quoted 2 lines
net/vhost-vdpa.c +2 -0
+
+    if (opts->x_svq || svq_cvq) {

Any chance we can have opts->x_svq = true but svq_cvq = false? Checking svq_cvq seems sufficient here.

net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 813

Quoted 7 lines
net/vhost-vdpa.c +4 -2
+        Error *warn = NULL;
 
-        if (!vhost_vdpa_net_valid_svq_features(features, errp)) {
-            goto err_svq;
+        svq_cvq = vhost_vdpa_net_valid_svq_features(features,
+                                                   opts->x_svq ? errp : &warn);
+        if (!svq_cvq) {

Same question as above.

Quoted 14 lines
net/vhost-vdpa.c +9 -0
+            if (opts->x_svq) {
+                goto err_svq;
+            } else {
+                warn_reportf_err(warn, "Cannot shadow CVQ: ");
+            }
          }
+    }
+
+    if (opts->x_svq || svq_cvq) {
+        struct vhost_vdpa_iova_range iova_range;
 
          vhost_vdpa_get_iova_range(vdpa_device_fd, &iova_range);
          iova_tree = vhost_iova_tree_new(iova_range.first, iova_range.last);

net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 833

Quoted 6 lines
net/vhost-vdpa.c +1 -2
@@ -708,15 +830,15 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
 
      for (i = 0; i < queue_pairs; i++) {
          ncs[i] = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name,
-                                     vdpa_device_fd, i, 2, true, opts->x_svq,
-                                     iova_tree);
+                                     vdpa_device_fd, i, 2, num_as, true,

I don't get why we need pass num_as to a specific vhost_vdpa structure. It should be sufficient to pass asid there.

Thanks

Quoted 12 lines
net/vhost-vdpa.c +2 -1
+                                     opts->x_svq, iova_tree);
          if (!ncs[i])
              goto err;
      }
 
      if (has_cvq) {
          nc = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name,
-                                 vdpa_device_fd, i, 1, false,
+                                 vdpa_device_fd, i, 1, num_as, false,
                                   opts->x_svq, iova_tree);
          if (!nc)
              goto err;
#40 lore raw

Re: [PATCH v6 10/10] vdpa: Always start CVQ in SVQ mode

Quoted 10 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez

Isolate control virtqueue in its own group, allowing to intercept control commands but letting dataplane run totally passthrough to the guest.

I think we need to tweak the title to "vdpa: Always start CVQ in SVQ mode if possible". Since SVQ for CVQ can't be enabled without ASID support?

Yes, I totally agree.

Quoted 50 lines
Quoted 50 lines

---
v6:
* Disable control SVQ if the device does not support it because of features.

v5:
* Fixing the not adding cvq buffers when x-svq=on is specified.
* Move vring state in vhost_vdpa_get_vring_group instead of using a parameter.
* Rename VHOST_VDPA_NET_CVQ_PASSTHROUGH to VHOST_VDPA_NET_DATA_ASID

v4:
* Squash vhost_vdpa_cvq_group_is_independent.
* Rebased on last CVQ start series, that allocated CVQ cmd bufs at load
* Do not check for cvq index on vhost_vdpa_net_prepare, we only have one that callback registered in that NetClientInfo.

v3:
* Make asid related queries print a warning instead of returning an error and stop the start of qemu.

---
  hw/virtio/vhost-vdpa.c |   3 +-
  net/vhost-vdpa.c       | 138 ++++++++++++++++++++++++++++++++++++++---
  2 files changed, 132 insertions(+), 9 deletions(-)
hw/virtio/vhost-vdpa.c +2 -1
index e3914fa40e..6401e7efb1 100644
@@ -648,7 +648,8 @@ static int vhost_vdpa_set_backend_cap(struct vhost_dev *dev)
  {
      uint64_t features;
      uint64_t f = 0x1ULL << VHOST_BACKEND_F_IOTLB_MSG_V2 |
-        0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH;
+        0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH |
+        0x1ULL << VHOST_BACKEND_F_IOTLB_ASID;
      int r;
 
      if (vhost_vdpa_call(dev, VHOST_GET_BACKEND_FEATURES, &features)) {
net/vhost-vdpa.c +0 -0
index 02780ee37b..7245ea70c6 100644
@@ -38,6 +38,9 @@ typedef struct VhostVDPAState {
      void *cvq_cmd_out_buffer;
      virtio_net_ctrl_ack *status;
 
Quoted 8 lines

net/vhost-vdpa.c typedef struct VhostVDPAState { · line 42

Quoted 2 lines
net/vhost-vdpa.c +2 -0
+    /* Number of address spaces supported by the device */
+    unsigned address_space_num;

I'm not sure this is the best place to store thing like this since it can cause confusion. We will have multiple VhostVDPAState when multiqueue is enabled.

I think we can delete this and ask it on each device start.

Quoted 19 lines
Quoted 19 lines
net/vhost-vdpa.c +4 -0
+
      /* The device always have SVQ enabled */
      bool always_svq;
      bool started;
@@ -101,6 +104,9 @@ static const uint64_t vdpa_svq_device_features =
      BIT_ULL(VIRTIO_NET_F_RSC_EXT) |
      BIT_ULL(VIRTIO_NET_F_STANDBY);
 
+#define VHOST_VDPA_NET_DATA_ASID 0
+#define VHOST_VDPA_NET_CVQ_ASID 1
+
  VHostNetState *vhost_vdpa_get_vhost_net(NetClientState *nc)
  {
      VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc);
@@ -242,6 +248,34 @@ static NetClientInfo net_vhost_vdpa_info = {
          .check_peer_type = vhost_vdpa_check_peer_type,
  };
 
Quoted 14 lines

net/vhost-vdpa.c static NetClientInfo net_vhost_vdpa_info = { · line 258

Quoted 8 lines
net/vhost-vdpa.c +8 -0
+static uint32_t vhost_vdpa_get_vring_group(int device_fd, unsigned vq_index)
+{
+    struct vhost_vring_state state = {
+        .index = vq_index,
+    };
+    int r = ioctl(device_fd, VHOST_VDPA_GET_VRING_GROUP, &state);
+
+    return r < 0 ? 0 : state.num;

Assume 0 when ioctl() fail is probably not a good idea: errors in ioctl might be hidden. It would be better to fallback to 0 when ASID is not supported.

Did I misunderstand you on [1]?

Quoted 49 lines
Quoted 49 lines
net/vhost-vdpa.c +36 -1
+}
+
+static int vhost_vdpa_set_address_space_id(struct vhost_vdpa *v,
+                                           unsigned vq_group,
+                                           unsigned asid_num)
+{
+    struct vhost_vring_state asid = {
+        .index = vq_group,
+        .num = asid_num,
+    };
+    int ret;
+
+    ret = ioctl(v->device_fd, VHOST_VDPA_SET_GROUP_ASID, &asid);
+    if (unlikely(ret < 0)) {
+        warn_report("Can't set vq group %u asid %u, errno=%d (%s)",
+            asid.index, asid.num, errno, g_strerror(errno));
+    }
+    return ret;
+}
+
  static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr)
  {
      VhostIOVATree *tree = v->iova_tree;
@@ -316,11 +350,54 @@ dma_map_err:
  static int vhost_vdpa_net_cvq_start(NetClientState *nc)
  {
      VhostVDPAState *s;
-    int r;
+    struct vhost_vdpa *v;
+    uint32_t cvq_group;
+    int cvq_index, r;
 
      assert(nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA);
 
      s = DO_UPCAST(VhostVDPAState, nc, nc);
+    v = &s->vhost_vdpa;
+
+    v->listener_shadow_vq = s->always_svq;
+    v->shadow_vqs_enabled = s->always_svq;
+    s->vhost_vdpa.address_space_id = VHOST_VDPA_NET_DATA_ASID;
+
+    if (s->always_svq) {
+        goto out;
+    }
+
+    if (s->address_space_num < 2) {
+        return 0;
+    }
Quoted 9 lines

net/vhost-vdpa.c dma_map_err: · line 376

Quoted 4 lines
net/vhost-vdpa.c +4 -0
+
+    if (!vhost_vdpa_net_valid_svq_features(v->dev->features, NULL)) {
+        return 0;
+    }

Any reason we do the above check during the start/stop? It should be easier to do that in the initialization.

We can store it as a member of VhostVDPAState maybe? They will be duplicated like the current number of AS.

Quoted 47 lines
Quoted 47 lines
net/vhost-vdpa.c +39 -0
+
+    /**
+     * Check if all the virtqueues of the virtio device are in a different vq
+     * than the last vq. VQ group of last group passed in cvq_group.
+     */
+    cvq_index = v->dev->vq_index_end - 1;
+    cvq_group = vhost_vdpa_get_vring_group(v->device_fd, cvq_index);
+    for (int i = 0; i < cvq_index; ++i) {
+        uint32_t group = vhost_vdpa_get_vring_group(v->device_fd, i);
+
+        if (unlikely(group == cvq_group)) {
+            warn_report("CVQ %u group is the same as VQ %u one (%u)", cvq_group,
+                        i, group);
+            return 0;
+        }
+    }
+
+    r = vhost_vdpa_set_address_space_id(v, cvq_group, VHOST_VDPA_NET_CVQ_ASID);
+    if (r == 0) {
+        v->shadow_vqs_enabled = true;
+        s->vhost_vdpa.address_space_id = VHOST_VDPA_NET_CVQ_ASID;
+    }
+
+out:
      if (!s->vhost_vdpa.shadow_vqs_enabled) {
          return 0;
      }
@@ -542,12 +619,38 @@ static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = {
      .avail_handler = vhost_vdpa_net_handle_ctrl_avail,
  };
 
+static uint32_t vhost_vdpa_get_as_num(int vdpa_device_fd)
+{
+    uint64_t features;
+    unsigned num_as;
+    int r;
+
+    r = ioctl(vdpa_device_fd, VHOST_GET_BACKEND_FEATURES, &features);
+    if (unlikely(r < 0)) {
+        warn_report("Cannot get backend features");
+        return 1;
+    }
+
+    if (!(features & BIT_ULL(VHOST_BACKEND_F_IOTLB_ASID))) {
+        return 1;
+    }
Quoted 9 lines

net/vhost-vdpa.c static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = { · line 641

Quoted 5 lines
net/vhost-vdpa.c +5 -0
+
+    r = ioctl(vdpa_device_fd, VHOST_VDPA_GET_AS_NUM, &num_as);
+    if (unlikely(r < 0)) {
+        warn_report("Cannot retrieve number of supported ASs");
+        return 1;

Let's return error here. This help. to identify bugs of qemu or kernel.

Same comment as with VHOST_VDPA_GET_VRING_GROUP.

Quoted 48 lines
Quoted 34 lines
net/vhost-vdpa.c +9 -0
+    }
+
+    return num_as;
+}
+
  static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
                                             const char *device,
                                             const char *name,
                                             int vdpa_device_fd,
                                             int queue_pair_index,
                                             int nvqs,
+                                           unsigned nas,
                                             bool is_datapath,
                                             bool svq,
                                             VhostIOVATree *iova_tree)
@@ -566,6 +669,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
      qemu_set_info_str(nc, TYPE_VHOST_VDPA);
      s = DO_UPCAST(VhostVDPAState, nc, nc);
 
+    s->address_space_num = nas;
      s->vhost_vdpa.device_fd = vdpa_device_fd;
      s->vhost_vdpa.index = queue_pair_index;
      s->always_svq = svq;
@@ -652,6 +756,8 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
      g_autoptr(VhostIOVATree) iova_tree = NULL;
      NetClientState *nc;
      int queue_pairs, r, i = 0, has_cvq = 0;
+    unsigned num_as = 1;
+    bool svq_cvq;
 
      assert(netdev->type == NET_CLIENT_DRIVER_VHOST_VDPA);
      opts = &netdev->u.vhost_vdpa;

net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 806

Quoted 10 lines
net/vhost-vdpa.c +5 -2
@@ -693,12 +799,28 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
          return queue_pairs;
      }
 
-    if (opts->x_svq) {
-        struct vhost_vdpa_iova_range iova_range;
+    svq_cvq = opts->x_svq;
+    if (has_cvq && !opts->x_svq) {
+        num_as = vhost_vdpa_get_as_num(vdpa_device_fd);
+        svq_cvq = num_as > 1;
+    }

The above check is not easy to follow, how about?

Quoted 2 lines

svq_cvq = vhost_vdpa_get_as_num() > 1 ? true : opts->x_svq;

That would allocate the iova tree even if CVQ is not used in the guest. And num_as is reused later, although we can ask it to the device at device start to avoid this.

If any, the linear conversion would be:
svq_cvq = opts->x_svq || (has_cvq && vhost_vdpa_get_as_num(vdpa_device_fd))

So we avoid the AS_NUM ioctl if not needed.

Quoted 8 lines

net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 808

Quoted 3 lines
net/vhost-vdpa.c +2 -0
+
+    if (opts->x_svq || svq_cvq) {

Any chance we can have opts->x_svq = true but svq_cvq = false? Checking svq_cvq seems sufficient here.

The reverse is possible, to have svq_cvq but no opts->x_svq.

Depending on that, this code emits a warning or a fatal error.

Quoted 28 lines

net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 813

Quoted 8 lines
net/vhost-vdpa.c +4 -2
+        Error *warn = NULL;
 
-        if (!vhost_vdpa_net_valid_svq_features(features, errp)) {
-            goto err_svq;
+        svq_cvq = vhost_vdpa_net_valid_svq_features(features,
+                                                   opts->x_svq ? errp : &warn);
+        if (!svq_cvq) {

Same question as above.

Quoted 15 lines
net/vhost-vdpa.c +9 -0
+            if (opts->x_svq) {
+                goto err_svq;
+            } else {
+                warn_reportf_err(warn, "Cannot shadow CVQ: ");
+            }
          }
+    }
+
+    if (opts->x_svq || svq_cvq) {
+        struct vhost_vdpa_iova_range iova_range;
 
          vhost_vdpa_get_iova_range(vdpa_device_fd, &iova_range);
          iova_tree = vhost_iova_tree_new(iova_range.first, iova_range.last);
@@ -708,15 +830,15 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
 
Quoted 10 lines

net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 833

Quoted 5 lines
net/vhost-vdpa.c +1 -2
      for (i = 0; i < queue_pairs; i++) {
          ncs[i] = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name,
-                                     vdpa_device_fd, i, 2, true, opts->x_svq,
-                                     iova_tree);
+                                     vdpa_device_fd, i, 2, num_as, true,

I don't get why we need pass num_as to a specific vhost_vdpa structure. It should be sufficient to pass asid there.

ASID is not known at this time, but at device's start. This is because we cannot ask if the CVQ is in its own vq group, because we don't know the control virtqueue index until the guest acknowledges the different features.

Thanks!

[1] https://www.mail-archive.com/[email protected]/msg901685.html

Quoted 14 lines
Quoted 13 lines
net/vhost-vdpa.c +2 -1
+                                     opts->x_svq, iova_tree);
          if (!ncs[i])
              goto err;
      }
 
      if (has_cvq) {
          nc = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name,
-                                 vdpa_device_fd, i, 1, false,
+                                 vdpa_device_fd, i, 1, num_as, false,
                                   opts->x_svq, iova_tree);
          if (!nc)
              goto err;
#41 lore raw

Re: [PATCH v6 10/10] vdpa: Always start CVQ in SVQ mode

Quoted 10 lines · Eugenio Perez Martin
Quoted 8 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez

Isolate control virtqueue in its own group, allowing to intercept control commands but letting dataplane run totally passthrough to the guest.

I think we need to tweak the title to "vdpa: Always start CVQ in SVQ mode if possible". Since SVQ for CVQ can't be enabled without ASID support?

Yes, I totally agree.

Btw, I wonder if it's worth to remove the "x-" prefix for the shadow virtqueue. It can help for the devices without ASID support but want do live migration.

Quoted 89 lines
Quoted 49 lines
Quoted 49 lines

---
v6:
* Disable control SVQ if the device does not support it because of features.

v5:
* Fixing the not adding cvq buffers when x-svq=on is specified.
* Move vring state in vhost_vdpa_get_vring_group instead of using a parameter.
* Rename VHOST_VDPA_NET_CVQ_PASSTHROUGH to VHOST_VDPA_NET_DATA_ASID

v4:
* Squash vhost_vdpa_cvq_group_is_independent.
* Rebased on last CVQ start series, that allocated CVQ cmd bufs at load
* Do not check for cvq index on vhost_vdpa_net_prepare, we only have one that callback registered in that NetClientInfo.

v3:
* Make asid related queries print a warning instead of returning an error and stop the start of qemu.

---
   hw/virtio/vhost-vdpa.c |   3 +-
   net/vhost-vdpa.c       | 138 ++++++++++++++++++++++++++++++++++++++---
   2 files changed, 132 insertions(+), 9 deletions(-)
hw/virtio/vhost-vdpa.c +2 -1
index e3914fa40e..6401e7efb1 100644
@@ -648,7 +648,8 @@ static int vhost_vdpa_set_backend_cap(struct vhost_dev *dev)
   {
       uint64_t features;
       uint64_t f = 0x1ULL << VHOST_BACKEND_F_IOTLB_MSG_V2 |
-        0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH;
+        0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH |
+        0x1ULL << VHOST_BACKEND_F_IOTLB_ASID;
       int r;
 
       if (vhost_vdpa_call(dev, VHOST_GET_BACKEND_FEATURES, &features)) {
net/vhost-vdpa.c +0 -0
index 02780ee37b..7245ea70c6 100644
@@ -38,6 +38,9 @@ typedef struct VhostVDPAState {
       void *cvq_cmd_out_buffer;
       virtio_net_ctrl_ack *status;
 
Quoted 7 lines

net/vhost-vdpa.c typedef struct VhostVDPAState { · line 42

Quoted 2 lines
net/vhost-vdpa.c +2 -0
+    /* Number of address spaces supported by the device */
+    unsigned address_space_num;

I'm not sure this is the best place to store thing like this since it can cause confusion. We will have multiple VhostVDPAState when multiqueue is enabled.

I think we can delete this and ask it on each device start.

Quoted 18 lines
Quoted 18 lines
net/vhost-vdpa.c +4 -0
+
       /* The device always have SVQ enabled */
       bool always_svq;
       bool started;
@@ -101,6 +104,9 @@ static const uint64_t vdpa_svq_device_features =
       BIT_ULL(VIRTIO_NET_F_RSC_EXT) |
       BIT_ULL(VIRTIO_NET_F_STANDBY);
 
+#define VHOST_VDPA_NET_DATA_ASID 0
+#define VHOST_VDPA_NET_CVQ_ASID 1
+
   VHostNetState *vhost_vdpa_get_vhost_net(NetClientState *nc)
   {
       VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc);
@@ -242,6 +248,34 @@ static NetClientInfo net_vhost_vdpa_info = {
           .check_peer_type = vhost_vdpa_check_peer_type,
   };
 
Quoted 13 lines

net/vhost-vdpa.c static NetClientInfo net_vhost_vdpa_info = { · line 258

Quoted 8 lines
net/vhost-vdpa.c +8 -0
+static uint32_t vhost_vdpa_get_vring_group(int device_fd, unsigned vq_index)
+{
+    struct vhost_vring_state state = {
+        .index = vq_index,
+    };
+    int r = ioctl(device_fd, VHOST_VDPA_GET_VRING_GROUP, &state);
+
+    return r < 0 ? 0 : state.num;

Assume 0 when ioctl() fail is probably not a good idea: errors in ioctl might be hidden. It would be better to fallback to 0 when ASID is not supported.

Quoted 1 line

Did I misunderstand you on [1]?

Nope. I think I was wrong at that time then :( Sorry for that.

We should differ from the case

1) no ASID support so 0 is assumed

2) something wrong in the case of ioctl, it's not necessarily a ENOTSUPP.

Quoted 57 lines
Quoted 48 lines
Quoted 48 lines
net/vhost-vdpa.c +35 -1
+}
+
+static int vhost_vdpa_set_address_space_id(struct vhost_vdpa *v,
+                                           unsigned vq_group,
+                                           unsigned asid_num)
+{
+    struct vhost_vring_state asid = {
+        .index = vq_group,
+        .num = asid_num,
+    };
+    int ret;
+
+    ret = ioctl(v->device_fd, VHOST_VDPA_SET_GROUP_ASID, &asid);
+    if (unlikely(ret < 0)) {
+        warn_report("Can't set vq group %u asid %u, errno=%d (%s)",
+            asid.index, asid.num, errno, g_strerror(errno));
+    }
+    return ret;
+}
+
   static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr)
   {
       VhostIOVATree *tree = v->iova_tree;
@@ -316,11 +350,54 @@ dma_map_err:
   static int vhost_vdpa_net_cvq_start(NetClientState *nc)
   {
       VhostVDPAState *s;
-    int r;
+    struct vhost_vdpa *v;
+    uint32_t cvq_group;
+    int cvq_index, r;
 
       assert(nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA);
 
       s = DO_UPCAST(VhostVDPAState, nc, nc);
+    v = &s->vhost_vdpa;
+
+    v->listener_shadow_vq = s->always_svq;
+    v->shadow_vqs_enabled = s->always_svq;
+    s->vhost_vdpa.address_space_id = VHOST_VDPA_NET_DATA_ASID;
+
+    if (s->always_svq) {
+        goto out;
+    }
+
+    if (s->address_space_num < 2) {
+        return 0;
Quoted 9 lines

net/vhost-vdpa.c dma_map_err: · line 376

Quoted 5 lines
net/vhost-vdpa.c +5 -0
+    }
+
+    if (!vhost_vdpa_net_valid_svq_features(v->dev->features, NULL)) {
+        return 0;
+    }

Any reason we do the above check during the start/stop? It should be easier to do that in the initialization.

Quoted 2 lines

We can store it as a member of VhostVDPAState maybe? They will be duplicated like the current number of AS.

I meant each VhostVDPAState just need to know the ASID it needs to use. There's no need to know the total number of address spaces or do the validation on it during start (the validation could be done during initialization).

Quoted 103 lines
Quoted 46 lines
Quoted 46 lines
net/vhost-vdpa.c +38 -0
+
+    /**
+     * Check if all the virtqueues of the virtio device are in a different vq
+     * than the last vq. VQ group of last group passed in cvq_group.
+     */
+    cvq_index = v->dev->vq_index_end - 1;
+    cvq_group = vhost_vdpa_get_vring_group(v->device_fd, cvq_index);
+    for (int i = 0; i < cvq_index; ++i) {
+        uint32_t group = vhost_vdpa_get_vring_group(v->device_fd, i);
+
+        if (unlikely(group == cvq_group)) {
+            warn_report("CVQ %u group is the same as VQ %u one (%u)", cvq_group,
+                        i, group);
+            return 0;
+        }
+    }
+
+    r = vhost_vdpa_set_address_space_id(v, cvq_group, VHOST_VDPA_NET_CVQ_ASID);
+    if (r == 0) {
+        v->shadow_vqs_enabled = true;
+        s->vhost_vdpa.address_space_id = VHOST_VDPA_NET_CVQ_ASID;
+    }
+
+out:
       if (!s->vhost_vdpa.shadow_vqs_enabled) {
           return 0;
       }
@@ -542,12 +619,38 @@ static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = {
       .avail_handler = vhost_vdpa_net_handle_ctrl_avail,
   };
 
+static uint32_t vhost_vdpa_get_as_num(int vdpa_device_fd)
+{
+    uint64_t features;
+    unsigned num_as;
+    int r;
+
+    r = ioctl(vdpa_device_fd, VHOST_GET_BACKEND_FEATURES, &features);
+    if (unlikely(r < 0)) {
+        warn_report("Cannot get backend features");
+        return 1;
+    }
+
+    if (!(features & BIT_ULL(VHOST_BACKEND_F_IOTLB_ASID))) {
+        return 1;
Quoted 9 lines

net/vhost-vdpa.c static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = { · line 641

Quoted 6 lines
net/vhost-vdpa.c +6 -0
+    }
+
+    r = ioctl(vdpa_device_fd, VHOST_VDPA_GET_AS_NUM, &num_as);
+    if (unlikely(r < 0)) {
+        warn_report("Cannot retrieve number of supported ASs");
+        return 1;

Let's return error here. This help. to identify bugs of qemu or kernel.

Same comment as with VHOST_VDPA_GET_VRING_GROUP.

Quoted 46 lines
Quoted 33 lines
net/vhost-vdpa.c +9 -0
+    }
+
+    return num_as;
+}
+
   static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
                                              const char *device,
                                              const char *name,
                                              int vdpa_device_fd,
                                              int queue_pair_index,
                                              int nvqs,
+                                           unsigned nas,
                                              bool is_datapath,
                                              bool svq,
                                              VhostIOVATree *iova_tree)
@@ -566,6 +669,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
       qemu_set_info_str(nc, TYPE_VHOST_VDPA);
       s = DO_UPCAST(VhostVDPAState, nc, nc);
 
+    s->address_space_num = nas;
       s->vhost_vdpa.device_fd = vdpa_device_fd;
       s->vhost_vdpa.index = queue_pair_index;
       s->always_svq = svq;
@@ -652,6 +756,8 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
       g_autoptr(VhostIOVATree) iova_tree = NULL;
       NetClientState *nc;
       int queue_pairs, r, i = 0, has_cvq = 0;
+    unsigned num_as = 1;
+    bool svq_cvq;
 
       assert(netdev->type == NET_CLIENT_DRIVER_VHOST_VDPA);
       opts = &netdev->u.vhost_vdpa;

net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 806

Quoted 10 lines
net/vhost-vdpa.c +5 -2
@@ -693,12 +799,28 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
           return queue_pairs;
       }
 
-    if (opts->x_svq) {
-        struct vhost_vdpa_iova_range iova_range;
+    svq_cvq = opts->x_svq;
+    if (has_cvq && !opts->x_svq) {
+        num_as = vhost_vdpa_get_as_num(vdpa_device_fd);
+        svq_cvq = num_as > 1;
+    }

The above check is not easy to follow, how about?

Quoted 5 lines
Quoted 2 lines

svq_cvq = vhost_vdpa_get_as_num() > 1 ? true : opts->x_svq;

That would allocate the iova tree even if CVQ is not used in the guest. And num_as is reused later, although we can ask it to the device at device start to avoid this.

Ok.

Quoted 5 lines

If any, the linear conversion would be:
svq_cvq = opts->x_svq || (has_cvq && vhost_vdpa_get_as_num(vdpa_device_fd))

So we avoid the AS_NUM ioctl if not needed.

So when !opts->x_svq, we need to check num_as is at least 2?

Quoted 10 lines
Quoted 7 lines

net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 808

Quoted 3 lines
net/vhost-vdpa.c +2 -0
+
+    if (opts->x_svq || svq_cvq) {

Any chance we can have opts->x_svq = true but svq_cvq = false? Checking svq_cvq seems sufficient here.

The reverse is possible, to have svq_cvq but no opts->x_svq.

Depending on that, this code emits a warning or a fatal error.

Ok, as replied in the previous patch, I think we need a better name for those ones.

if (opts->x_svq) {
        shadow_data_vq = true;
        if(has_cvq) shadow_cvq = true;
} else if (num_as >= 2 && has_cvq) {
        shadow_cvq = true;
}

The other logic can just check shadow_cvq or shadow_data_vq individually.

Quoted 36 lines
Quoted 27 lines

net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 813

Quoted 8 lines
net/vhost-vdpa.c +4 -2
+        Error *warn = NULL;
 
-        if (!vhost_vdpa_net_valid_svq_features(features, errp)) {
-            goto err_svq;
+        svq_cvq = vhost_vdpa_net_valid_svq_features(features,
+                                                   opts->x_svq ? errp : &warn);
+        if (!svq_cvq) {

Same question as above.

Quoted 15 lines
net/vhost-vdpa.c +9 -0
+            if (opts->x_svq) {
+                goto err_svq;
+            } else {
+                warn_reportf_err(warn, "Cannot shadow CVQ: ");
+            }
           }
+    }
+
+    if (opts->x_svq || svq_cvq) {
+        struct vhost_vdpa_iova_range iova_range;
 
           vhost_vdpa_get_iova_range(vdpa_device_fd, &iova_range);
           iova_tree = vhost_iova_tree_new(iova_range.first, iova_range.last);
@@ -708,15 +830,15 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
 
Quoted 9 lines

net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 833

Quoted 5 lines
net/vhost-vdpa.c +1 -2
       for (i = 0; i < queue_pairs; i++) {
           ncs[i] = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name,
-                                     vdpa_device_fd, i, 2, true, opts->x_svq,
-                                     iova_tree);
+                                     vdpa_device_fd, i, 2, num_as, true,

I don't get why we need pass num_as to a specific vhost_vdpa structure. It should be sufficient to pass asid there.

Quoted 4 lines

ASID is not known at this time, but at device's start. This is because we cannot ask if the CVQ is in its own vq group, because we don't know the control virtqueue index until the guest acknowledges the different features.

We can probe those during initialization I think. E.g doing some negotiation in the initialization phase.

Thanks

Quoted 18 lines

Thanks!

[1] https://www.mail-archive.com/[email protected]/msg901685.html

Quoted 12 lines
Quoted 12 lines
net/vhost-vdpa.c +2 -1
+                                     opts->x_svq, iova_tree);
           if (!ncs[i])
               goto err;
       }
 
       if (has_cvq) {
           nc = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name,
-                                 vdpa_device_fd, i, 1, false,
+                                 vdpa_device_fd, i, 1, num_as, false,
                                    opts->x_svq, iova_tree);
           if (!nc)
               goto err;
#42 lore raw

Re: [PATCH v6 10/10] vdpa: Always start CVQ in SVQ mode

Quoted 12 lines · Jason Wang
Quoted 9 lines · Eugenio Perez Martin
Quoted 8 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez

Isolate control virtqueue in its own group, allowing to intercept control commands but letting dataplane run totally passthrough to the guest.

I think we need to tweak the title to "vdpa: Always start CVQ in SVQ mode if possible". Since SVQ for CVQ can't be enabled without ASID support?

Quoted 7 lines
Quoted 1 line

Yes, I totally agree.

Btw, I wonder if it's worth to remove the "x-" prefix for the shadow virtqueue. It can help for the devices without ASID support but want do live migration.

Sure I can propose on top.

Quoted 93 lines
Quoted 90 lines
Quoted 50 lines
Quoted 50 lines

---
v6:
* Disable control SVQ if the device does not support it because of features.

v5:
* Fixing the not adding cvq buffers when x-svq=on is specified.
* Move vring state in vhost_vdpa_get_vring_group instead of using a parameter.
* Rename VHOST_VDPA_NET_CVQ_PASSTHROUGH to VHOST_VDPA_NET_DATA_ASID

v4:
* Squash vhost_vdpa_cvq_group_is_independent.
* Rebased on last CVQ start series, that allocated CVQ cmd bufs at load
* Do not check for cvq index on vhost_vdpa_net_prepare, we only have one that callback registered in that NetClientInfo.

v3:
* Make asid related queries print a warning instead of returning an error and stop the start of qemu.

---
   hw/virtio/vhost-vdpa.c |   3 +-
   net/vhost-vdpa.c       | 138 ++++++++++++++++++++++++++++++++++++++---
   2 files changed, 132 insertions(+), 9 deletions(-)
hw/virtio/vhost-vdpa.c +2 -1
index e3914fa40e..6401e7efb1 100644
@@ -648,7 +648,8 @@ static int vhost_vdpa_set_backend_cap(struct vhost_dev *dev)
   {
       uint64_t features;
       uint64_t f = 0x1ULL << VHOST_BACKEND_F_IOTLB_MSG_V2 |
-        0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH;
+        0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH |
+        0x1ULL << VHOST_BACKEND_F_IOTLB_ASID;
       int r;
 
       if (vhost_vdpa_call(dev, VHOST_GET_BACKEND_FEATURES, &features)) {
net/vhost-vdpa.c +0 -0
index 02780ee37b..7245ea70c6 100644
@@ -38,6 +38,9 @@ typedef struct VhostVDPAState {
       void *cvq_cmd_out_buffer;
       virtio_net_ctrl_ack *status;
 
Quoted 7 lines

net/vhost-vdpa.c typedef struct VhostVDPAState { · line 42

Quoted 2 lines
net/vhost-vdpa.c +2 -0
+    /* Number of address spaces supported by the device */
+    unsigned address_space_num;

I'm not sure this is the best place to store thing like this since it can cause confusion. We will have multiple VhostVDPAState when multiqueue is enabled.

I think we can delete this and ask it on each device start.

Quoted 18 lines
Quoted 18 lines
net/vhost-vdpa.c +4 -0
+
       /* The device always have SVQ enabled */
       bool always_svq;
       bool started;
@@ -101,6 +104,9 @@ static const uint64_t vdpa_svq_device_features =
       BIT_ULL(VIRTIO_NET_F_RSC_EXT) |
       BIT_ULL(VIRTIO_NET_F_STANDBY);
 
+#define VHOST_VDPA_NET_DATA_ASID 0
+#define VHOST_VDPA_NET_CVQ_ASID 1
+
   VHostNetState *vhost_vdpa_get_vhost_net(NetClientState *nc)
   {
       VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc);
@@ -242,6 +248,34 @@ static NetClientInfo net_vhost_vdpa_info = {
           .check_peer_type = vhost_vdpa_check_peer_type,
   };
 
Quoted 13 lines

net/vhost-vdpa.c static NetClientInfo net_vhost_vdpa_info = { · line 258

Quoted 8 lines
net/vhost-vdpa.c +8 -0
+static uint32_t vhost_vdpa_get_vring_group(int device_fd, unsigned vq_index)
+{
+    struct vhost_vring_state state = {
+        .index = vq_index,
+    };
+    int r = ioctl(device_fd, VHOST_VDPA_GET_VRING_GROUP, &state);
+
+    return r < 0 ? 0 : state.num;

Assume 0 when ioctl() fail is probably not a good idea: errors in ioctl might be hidden. It would be better to fallback to 0 when ASID is not supported.

Quoted 1 line

Did I misunderstand you on [1]?

Quoted 8 lines

Nope. I think I was wrong at that time then :( Sorry for that.

We should differ from the case

1) no ASID support so 0 is assumed

2) something wrong in the case of ioctl, it's not necessarily a ENOTSUPP.

What action should we take here? Isn't it better to disable SVQ and let the device run?

Quoted 58 lines
Quoted 58 lines
Quoted 58 lines
Quoted 44 lines
net/vhost-vdpa.c +30 -1
+}
+
+static int vhost_vdpa_set_address_space_id(struct vhost_vdpa *v,
+                                           unsigned vq_group,
+                                           unsigned asid_num)
+{
+    struct vhost_vring_state asid = {
+        .index = vq_group,
+        .num = asid_num,
+    };
+    int ret;
+
+    ret = ioctl(v->device_fd, VHOST_VDPA_SET_GROUP_ASID, &asid);
+    if (unlikely(ret < 0)) {
+        warn_report("Can't set vq group %u asid %u, errno=%d (%s)",
+            asid.index, asid.num, errno, g_strerror(errno));
+    }
+    return ret;
+}
+
   static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr)
   {
       VhostIOVATree *tree = v->iova_tree;
@@ -316,11 +350,54 @@ dma_map_err:
   static int vhost_vdpa_net_cvq_start(NetClientState *nc)
   {
       VhostVDPAState *s;
-    int r;
+    struct vhost_vdpa *v;
+    uint32_t cvq_group;
+    int cvq_index, r;
 
       assert(nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA);
 
       s = DO_UPCAST(VhostVDPAState, nc, nc);
+    v = &s->vhost_vdpa;
+
+    v->listener_shadow_vq = s->always_svq;
+    v->shadow_vqs_enabled = s->always_svq;
+    s->vhost_vdpa.address_space_id = VHOST_VDPA_NET_DATA_ASID;
+
+    if (s->always_svq) {

net/vhost-vdpa.c dma_map_err: · line 376

Quoted 10 lines
net/vhost-vdpa.c +10 -0
+        goto out;
+    }
+
+    if (s->address_space_num < 2) {
+        return 0;
+    }
+
+    if (!vhost_vdpa_net_valid_svq_features(v->dev->features, NULL)) {
+        return 0;
+    }

Any reason we do the above check during the start/stop? It should be easier to do that in the initialization.

Quoted 9 lines
Quoted 2 lines

We can store it as a member of VhostVDPAState maybe? They will be duplicated like the current number of AS.

I meant each VhostVDPAState just need to know the ASID it needs to use. There's no need to know the total number of address spaces or do the validation on it during start (the validation could be done during initialization).

I thought we were talking about the virtio features.

So let's omit this check and simply try to set ASID? The worst case is an -ENOTSUPP or -EINVAL, so the actions to take are the same as if we don't have enough AS.

Quoted 115 lines
Quoted 104 lines
Quoted 47 lines
Quoted 47 lines
net/vhost-vdpa.c +38 -0
+
+    /**
+     * Check if all the virtqueues of the virtio device are in a different vq
+     * than the last vq. VQ group of last group passed in cvq_group.
+     */
+    cvq_index = v->dev->vq_index_end - 1;
+    cvq_group = vhost_vdpa_get_vring_group(v->device_fd, cvq_index);
+    for (int i = 0; i < cvq_index; ++i) {
+        uint32_t group = vhost_vdpa_get_vring_group(v->device_fd, i);
+
+        if (unlikely(group == cvq_group)) {
+            warn_report("CVQ %u group is the same as VQ %u one (%u)", cvq_group,
+                        i, group);
+            return 0;
+        }
+    }
+
+    r = vhost_vdpa_set_address_space_id(v, cvq_group, VHOST_VDPA_NET_CVQ_ASID);
+    if (r == 0) {
+        v->shadow_vqs_enabled = true;
+        s->vhost_vdpa.address_space_id = VHOST_VDPA_NET_CVQ_ASID;
+    }
+
+out:
       if (!s->vhost_vdpa.shadow_vqs_enabled) {
           return 0;
       }
@@ -542,12 +619,38 @@ static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = {
       .avail_handler = vhost_vdpa_net_handle_ctrl_avail,
   };
 
+static uint32_t vhost_vdpa_get_as_num(int vdpa_device_fd)
+{
+    uint64_t features;
+    unsigned num_as;
+    int r;
+
+    r = ioctl(vdpa_device_fd, VHOST_GET_BACKEND_FEATURES, &features);
+    if (unlikely(r < 0)) {
+        warn_report("Cannot get backend features");
+        return 1;
+    }
+
+    if (!(features & BIT_ULL(VHOST_BACKEND_F_IOTLB_ASID))) {
+        return 1;
Quoted 9 lines

net/vhost-vdpa.c static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = { · line 641

Quoted 6 lines
net/vhost-vdpa.c +6 -0
+    }
+
+    r = ioctl(vdpa_device_fd, VHOST_VDPA_GET_AS_NUM, &num_as);
+    if (unlikely(r < 0)) {
+        warn_report("Cannot retrieve number of supported ASs");
+        return 1;

Let's return error here. This help. to identify bugs of qemu or kernel.

Same comment as with VHOST_VDPA_GET_VRING_GROUP.

Quoted 46 lines
Quoted 33 lines
net/vhost-vdpa.c +9 -0
+    }
+
+    return num_as;
+}
+
   static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
                                              const char *device,
                                              const char *name,
                                              int vdpa_device_fd,
                                              int queue_pair_index,
                                              int nvqs,
+                                           unsigned nas,
                                              bool is_datapath,
                                              bool svq,
                                              VhostIOVATree *iova_tree)
@@ -566,6 +669,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
       qemu_set_info_str(nc, TYPE_VHOST_VDPA);
       s = DO_UPCAST(VhostVDPAState, nc, nc);
 
+    s->address_space_num = nas;
       s->vhost_vdpa.device_fd = vdpa_device_fd;
       s->vhost_vdpa.index = queue_pair_index;
       s->always_svq = svq;
@@ -652,6 +756,8 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
       g_autoptr(VhostIOVATree) iova_tree = NULL;
       NetClientState *nc;
       int queue_pairs, r, i = 0, has_cvq = 0;
+    unsigned num_as = 1;
+    bool svq_cvq;
 
       assert(netdev->type == NET_CLIENT_DRIVER_VHOST_VDPA);
       opts = &netdev->u.vhost_vdpa;

net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 806

Quoted 10 lines
net/vhost-vdpa.c +5 -2
@@ -693,12 +799,28 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
           return queue_pairs;
       }
 
-    if (opts->x_svq) {
-        struct vhost_vdpa_iova_range iova_range;
+    svq_cvq = opts->x_svq;
+    if (has_cvq && !opts->x_svq) {
+        num_as = vhost_vdpa_get_as_num(vdpa_device_fd);
+        svq_cvq = num_as > 1;
+    }

The above check is not easy to follow, how about?

Quoted 5 lines
Quoted 2 lines

svq_cvq = vhost_vdpa_get_as_num() > 1 ? true : opts->x_svq;

That would allocate the iova tree even if CVQ is not used in the guest. And num_as is reused later, although we can ask it to the device at device start to avoid this.

Ok.

Quoted 1 line
Quoted 8 lines
Quoted 4 lines

If any, the linear conversion would be:
svq_cvq = opts->x_svq || (has_cvq && vhost_vdpa_get_as_num(vdpa_device_fd))

So we avoid the AS_NUM ioctl if not needed.

So when !opts->x_svq, we need to check num_as is at least 2?

As I think you proposed, we can simply try to set CVQ ASID and react to -EINVAL.

But this code here is trying to not to allocate iova_tree if we're sure it will not be needed. Maybe it is easier to always allocate the empty iova tree and only fill it if needed?

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 808

Quoted 4 lines
net/vhost-vdpa.c +2 -0
+
+    if (opts->x_svq || svq_cvq) {

Any chance we can have opts->x_svq = true but svq_cvq = false? Checking svq_cvq seems sufficient here.

Quoted 8 lines
Quoted 3 lines

The reverse is possible, to have svq_cvq but no opts->x_svq.

Depending on that, this code emits a warning or a fatal error.

Ok, as replied in the previous patch, I think we need a better name for those ones.

cvq_svq can be renamed for sure. x_svq can be aliased with other variable if needed too.

Quoted 9 lines
if (opts->x_svq) {
         shadow_data_vq = true;
         if(has_cvq) shadow_cvq = true;
} else if (num_as >= 2 && has_cvq) {
         shadow_cvq = true;
}

The other logic can just check shadow_cvq or shadow_data_vq individually.

Not sure if shadow_data_vq is accurate. It sounds to me as "Only shadow data virtqueues but not CVQ".

shadow_device and shadow_cvq?

Quoted 37 lines
Quoted 37 lines
Quoted 37 lines

net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 813

Quoted 9 lines
net/vhost-vdpa.c +4 -2
+        Error *warn = NULL;
 
-        if (!vhost_vdpa_net_valid_svq_features(features, errp)) {
-            goto err_svq;
+        svq_cvq = vhost_vdpa_net_valid_svq_features(features,
+                                                   opts->x_svq ? errp : &warn);
+        if (!svq_cvq) {

Same question as above.

Quoted 14 lines
net/vhost-vdpa.c +9 -0
+            if (opts->x_svq) {
+                goto err_svq;
+            } else {
+                warn_reportf_err(warn, "Cannot shadow CVQ: ");
+            }
           }
+    }
+
+    if (opts->x_svq || svq_cvq) {
+        struct vhost_vdpa_iova_range iova_range;
 
           vhost_vdpa_get_iova_range(vdpa_device_fd, &iova_range);
           iova_tree = vhost_iova_tree_new(iova_range.first, iova_range.last);

net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 833

Quoted 6 lines
net/vhost-vdpa.c +1 -2
@@ -708,15 +830,15 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
 
       for (i = 0; i < queue_pairs; i++) {
           ncs[i] = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name,
-                                     vdpa_device_fd, i, 2, true, opts->x_svq,
-                                     iova_tree);
+                                     vdpa_device_fd, i, 2, num_as, true,

I don't get why we need pass num_as to a specific vhost_vdpa structure. It should be sufficient to pass asid there.

Quoted 9 lines
Quoted 4 lines

ASID is not known at this time, but at device's start. This is because we cannot ask if the CVQ is in its own vq group, because we don't know the control virtqueue index until the guest acknowledges the different features.

We can probe those during initialization I think. E.g doing some negotiation in the initialization phase.

We've developed this idea in other threads, let's continue there better.

Thanks!

Quoted 22 lines

Thanks

Quoted 18 lines

Thanks!

[1] https://www.mail-archive.com/[email protected]/msg901685.html

Quoted 12 lines
Quoted 12 lines
net/vhost-vdpa.c +2 -1
+                                     opts->x_svq, iova_tree);
           if (!ncs[i])
               goto err;
       }
 
       if (has_cvq) {
           nc = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name,
-                                 vdpa_device_fd, i, 1, false,
+                                 vdpa_device_fd, i, 1, num_as, false,
                                    opts->x_svq, iova_tree);
           if (!nc)
               goto err;
#43 lore raw

Re: [PATCH v6 10/10] vdpa: Always start CVQ in SVQ mode

Quoted 108 lines · Eugenio Perez Martin
Quoted 6 lines · Jason Wang
Quoted 4 lines · Eugenio Perez Martin
Quoted 3 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez

Isolate control virtqueue in its own group, allowing to intercept control commands but letting dataplane run totally passthrough to the guest.

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines

I think we need to tweak the title to "vdpa: Always start CVQ in SVQ mode if possible". Since SVQ for CVQ can't be enabled without ASID support?

Yes, I totally agree.

Btw, I wonder if it's worth to remove the "x-" prefix for the shadow virtqueue. It can help for the devices without ASID support but want do live migration.

Sure I can propose on top.

Quoted 90 lines
Quoted 86 lines
Quoted 48 lines
Quoted 48 lines

---
v6:
* Disable control SVQ if the device does not support it because of features.

v5:
* Fixing the not adding cvq buffers when x-svq=on is specified.
* Move vring state in vhost_vdpa_get_vring_group instead of using a parameter.
* Rename VHOST_VDPA_NET_CVQ_PASSTHROUGH to VHOST_VDPA_NET_DATA_ASID

v4:
* Squash vhost_vdpa_cvq_group_is_independent.
* Rebased on last CVQ start series, that allocated CVQ cmd bufs at load
* Do not check for cvq index on vhost_vdpa_net_prepare, we only have one that callback registered in that NetClientInfo.

v3:
* Make asid related queries print a warning instead of returning an error and stop the start of qemu.

---
    hw/virtio/vhost-vdpa.c |   3 +-
    net/vhost-vdpa.c       | 138 ++++++++++++++++++++++++++++++++++++++---
    2 files changed, 132 insertions(+), 9 deletions(-)
hw/virtio/vhost-vdpa.c +2 -1
index e3914fa40e..6401e7efb1 100644
@@ -648,7 +648,8 @@ static int vhost_vdpa_set_backend_cap(struct vhost_dev *dev)
    {
        uint64_t features;
        uint64_t f = 0x1ULL << VHOST_BACKEND_F_IOTLB_MSG_V2 |
-        0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH;
+        0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH |
+        0x1ULL << VHOST_BACKEND_F_IOTLB_ASID;
        int r;
 
        if (vhost_vdpa_call(dev, VHOST_GET_BACKEND_FEATURES, &features)) {
net/vhost-vdpa.c +0 -0
index 02780ee37b..7245ea70c6 100644
@@ -38,6 +38,9 @@ typedef struct VhostVDPAState {
        void *cvq_cmd_out_buffer;
        virtio_net_ctrl_ack *status;
 
Quoted 6 lines

net/vhost-vdpa.c typedef struct VhostVDPAState { · line 42

Quoted 2 lines
net/vhost-vdpa.c +2 -0
+    /* Number of address spaces supported by the device */
+    unsigned address_space_num;

I'm not sure this is the best place to store thing like this since it can cause confusion. We will have multiple VhostVDPAState when multiqueue is enabled.

I think we can delete this and ask it on each device start.

Quoted 18 lines
Quoted 18 lines
net/vhost-vdpa.c +4 -0
+
        /* The device always have SVQ enabled */
        bool always_svq;
        bool started;
@@ -101,6 +104,9 @@ static const uint64_t vdpa_svq_device_features =
        BIT_ULL(VIRTIO_NET_F_RSC_EXT) |
        BIT_ULL(VIRTIO_NET_F_STANDBY);
 
+#define VHOST_VDPA_NET_DATA_ASID 0
+#define VHOST_VDPA_NET_CVQ_ASID 1
+
    VHostNetState *vhost_vdpa_get_vhost_net(NetClientState *nc)
    {
        VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc);
@@ -242,6 +248,34 @@ static NetClientInfo net_vhost_vdpa_info = {
            .check_peer_type = vhost_vdpa_check_peer_type,
    };
 
Quoted 12 lines

net/vhost-vdpa.c static NetClientInfo net_vhost_vdpa_info = { · line 258

Quoted 8 lines
net/vhost-vdpa.c +8 -0
+static uint32_t vhost_vdpa_get_vring_group(int device_fd, unsigned vq_index)
+{
+    struct vhost_vring_state state = {
+        .index = vq_index,
+    };
+    int r = ioctl(device_fd, VHOST_VDPA_GET_VRING_GROUP, &state);
+
+    return r < 0 ? 0 : state.num;

Assume 0 when ioctl() fail is probably not a good idea: errors in ioctl might be hidden. It would be better to fallback to 0 when ASID is not supported.

Quoted 1 line

Did I misunderstand you on [1]?

Nope. I think I was wrong at that time then :( Sorry for that.

Quoted 8 lines
Quoted 6 lines

We should differ from the case

1) no ASID support so 0 is assumed

2) something wrong in the case of ioctl, it's not necessarily a ENOTSUPP.

What action should we take here? Isn't it better to disable SVQ and let the device run?

It should fail the function instead of assuming 0 like how other vhost ioctl work.

Quoted 64 lines
Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
Quoted 43 lines
net/vhost-vdpa.c +30 -1
+}
+
+static int vhost_vdpa_set_address_space_id(struct vhost_vdpa *v,
+                                           unsigned vq_group,
+                                           unsigned asid_num)
+{
+    struct vhost_vring_state asid = {
+        .index = vq_group,
+        .num = asid_num,
+    };
+    int ret;
+
+    ret = ioctl(v->device_fd, VHOST_VDPA_SET_GROUP_ASID, &asid);
+    if (unlikely(ret < 0)) {
+        warn_report("Can't set vq group %u asid %u, errno=%d (%s)",
+            asid.index, asid.num, errno, g_strerror(errno));
+    }
+    return ret;
+}
+
    static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr)
    {
        VhostIOVATree *tree = v->iova_tree;
@@ -316,11 +350,54 @@ dma_map_err:
    static int vhost_vdpa_net_cvq_start(NetClientState *nc)
    {
        VhostVDPAState *s;
-    int r;
+    struct vhost_vdpa *v;
+    uint32_t cvq_group;
+    int cvq_index, r;
 
        assert(nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA);
 
        s = DO_UPCAST(VhostVDPAState, nc, nc);
+    v = &s->vhost_vdpa;
+
+    v->listener_shadow_vq = s->always_svq;
+    v->shadow_vqs_enabled = s->always_svq;
+    s->vhost_vdpa.address_space_id = VHOST_VDPA_NET_DATA_ASID;
+
+    if (s->always_svq) {

net/vhost-vdpa.c dma_map_err: · line 376

Quoted 10 lines
net/vhost-vdpa.c +10 -0
+        goto out;
+    }
+
+    if (s->address_space_num < 2) {
+        return 0;
+    }
+
+    if (!vhost_vdpa_net_valid_svq_features(v->dev->features, NULL)) {
+        return 0;
+    }

Any reason we do the above check during the start/stop? It should be easier to do that in the initialization.

Quoted 8 lines
Quoted 2 lines

We can store it as a member of VhostVDPAState maybe? They will be duplicated like the current number of AS.

I meant each VhostVDPAState just need to know the ASID it needs to use. There's no need to know the total number of address spaces or do the validation on it during start (the validation could be done during initialization).

Quoted 5 lines

I thought we were talking about the virtio features.

So let's omit this check and simply try to set ASID? The worst case is an -ENOTSUPP or -EINVAL, so the actions to take are the same as if we don't have enough AS.

See the discussion in other thread:

1) do the probing of asid stuffs during initalization, we can simply try to set the ASID here and fail the vhost dev start
2) do the probing each time during vhost start

Personally I like 1, but if you want to go with 2 it should also fine. (need some comments)

Quoted 115 lines
Quoted 115 lines
Quoted 101 lines
Quoted 45 lines
Quoted 45 lines
net/vhost-vdpa.c +37 -0
+
+    /**
+     * Check if all the virtqueues of the virtio device are in a different vq
+     * than the last vq. VQ group of last group passed in cvq_group.
+     */
+    cvq_index = v->dev->vq_index_end - 1;
+    cvq_group = vhost_vdpa_get_vring_group(v->device_fd, cvq_index);
+    for (int i = 0; i < cvq_index; ++i) {
+        uint32_t group = vhost_vdpa_get_vring_group(v->device_fd, i);
+
+        if (unlikely(group == cvq_group)) {
+            warn_report("CVQ %u group is the same as VQ %u one (%u)", cvq_group,
+                        i, group);
+            return 0;
+        }
+    }
+
+    r = vhost_vdpa_set_address_space_id(v, cvq_group, VHOST_VDPA_NET_CVQ_ASID);
+    if (r == 0) {
+        v->shadow_vqs_enabled = true;
+        s->vhost_vdpa.address_space_id = VHOST_VDPA_NET_CVQ_ASID;
+    }
+
+out:
        if (!s->vhost_vdpa.shadow_vqs_enabled) {
            return 0;
        }
@@ -542,12 +619,38 @@ static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = {
        .avail_handler = vhost_vdpa_net_handle_ctrl_avail,
    };
 
+static uint32_t vhost_vdpa_get_as_num(int vdpa_device_fd)
+{
+    uint64_t features;
+    unsigned num_as;
+    int r;
+
+    r = ioctl(vdpa_device_fd, VHOST_GET_BACKEND_FEATURES, &features);
+    if (unlikely(r < 0)) {
+        warn_report("Cannot get backend features");
+        return 1;
+    }
+
+    if (!(features & BIT_ULL(VHOST_BACKEND_F_IOTLB_ASID))) {
Quoted 9 lines

net/vhost-vdpa.c static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = { · line 641

Quoted 7 lines
net/vhost-vdpa.c +7 -0
+        return 1;
+    }
+
+    r = ioctl(vdpa_device_fd, VHOST_VDPA_GET_AS_NUM, &num_as);
+    if (unlikely(r < 0)) {
+        warn_report("Cannot retrieve number of supported ASs");
+        return 1;

Let's return error here. This help. to identify bugs of qemu or kernel.

Same comment as with VHOST_VDPA_GET_VRING_GROUP.

Quoted 45 lines
Quoted 33 lines
net/vhost-vdpa.c +9 -0
+    }
+
+    return num_as;
+}
+
    static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
                                               const char *device,
                                               const char *name,
                                               int vdpa_device_fd,
                                               int queue_pair_index,
                                               int nvqs,
+                                           unsigned nas,
                                               bool is_datapath,
                                               bool svq,
                                               VhostIOVATree *iova_tree)
@@ -566,6 +669,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
        qemu_set_info_str(nc, TYPE_VHOST_VDPA);
        s = DO_UPCAST(VhostVDPAState, nc, nc);
 
+    s->address_space_num = nas;
        s->vhost_vdpa.device_fd = vdpa_device_fd;
        s->vhost_vdpa.index = queue_pair_index;
        s->always_svq = svq;
@@ -652,6 +756,8 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
        g_autoptr(VhostIOVATree) iova_tree = NULL;
        NetClientState *nc;
        int queue_pairs, r, i = 0, has_cvq = 0;
+    unsigned num_as = 1;
+    bool svq_cvq;
 
        assert(netdev->type == NET_CLIENT_DRIVER_VHOST_VDPA);
        opts = &netdev->u.vhost_vdpa;

net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 806

Quoted 10 lines
net/vhost-vdpa.c +5 -2
@@ -693,12 +799,28 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
            return queue_pairs;
        }
 
-    if (opts->x_svq) {
-        struct vhost_vdpa_iova_range iova_range;
+    svq_cvq = opts->x_svq;
+    if (has_cvq && !opts->x_svq) {
+        num_as = vhost_vdpa_get_as_num(vdpa_device_fd);
+        svq_cvq = num_as > 1;
+    }

The above check is not easy to follow, how about?

Quoted 5 lines
Quoted 2 lines

svq_cvq = vhost_vdpa_get_as_num() > 1 ? true : opts->x_svq;

That would allocate the iova tree even if CVQ is not used in the guest. And num_as is reused later, although we can ask it to the device at device start to avoid this.

Ok.

Quoted 4 lines

If any, the linear conversion would be:
svq_cvq = opts->x_svq || (has_cvq && vhost_vdpa_get_as_num(vdpa_device_fd))

So we avoid the AS_NUM ioctl if not needed.

Quoted 7 lines
Quoted 2 lines

So when !opts->x_svq, we need to check num_as is at least 2?

As I think you proposed, we can simply try to set CVQ ASID and react to -EINVAL.

But this code here is trying to not to allocate iova_tree if we're sure it will not be needed. Maybe it is easier to always allocate the empty iova tree and only fill it if needed?

I feel it should work.

Quoted 23 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 808

Quoted 3 lines
net/vhost-vdpa.c +2 -0
+
+    if (opts->x_svq || svq_cvq) {

Any chance we can have opts->x_svq = true but svq_cvq = false? Checking svq_cvq seems sufficient here.

Quoted 7 lines
Quoted 3 lines

The reverse is possible, to have svq_cvq but no opts->x_svq.

Depending on that, this code emits a warning or a fatal error.

Ok, as replied in the previous patch, I think we need a better name for those ones.

cvq_svq can be renamed for sure. x_svq can be aliased with other variable if needed too.

Quoted 7 lines
if (opts->x_svq) {
          shadow_data_vq = true;
          if(has_cvq) shadow_cvq = true;
} else if (num_as >= 2 && has_cvq) {
          shadow_cvq = true;
}
Quoted 6 lines
Quoted 2 lines

The other logic can just check shadow_cvq or shadow_data_vq individually.

Not sure if shadow_data_vq is accurate. It sounds to me as "Only shadow data virtqueues but not CVQ".

shadow_device and shadow_cvq?

Should work, let's see.

Thanks

Quoted 66 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines

net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 813

Quoted 8 lines
net/vhost-vdpa.c +4 -2
+        Error *warn = NULL;
 
-        if (!vhost_vdpa_net_valid_svq_features(features, errp)) {
-            goto err_svq;
+        svq_cvq = vhost_vdpa_net_valid_svq_features(features,
+                                                   opts->x_svq ? errp : &warn);
+        if (!svq_cvq) {

Same question as above.

Quoted 14 lines
net/vhost-vdpa.c +9 -0
+            if (opts->x_svq) {
+                goto err_svq;
+            } else {
+                warn_reportf_err(warn, "Cannot shadow CVQ: ");
+            }
            }
+    }
+
+    if (opts->x_svq || svq_cvq) {
+        struct vhost_vdpa_iova_range iova_range;
 
            vhost_vdpa_get_iova_range(vdpa_device_fd, &iova_range);
            iova_tree = vhost_iova_tree_new(iova_range.first, iova_range.last);

net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 833

Quoted 6 lines
net/vhost-vdpa.c +1 -2
@@ -708,15 +830,15 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
 
        for (i = 0; i < queue_pairs; i++) {
            ncs[i] = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name,
-                                     vdpa_device_fd, i, 2, true, opts->x_svq,
-                                     iova_tree);
+                                     vdpa_device_fd, i, 2, num_as, true,

I don't get why we need pass num_as to a specific vhost_vdpa structure. It should be sufficient to pass asid there.

Quoted 8 lines
Quoted 4 lines

ASID is not known at this time, but at device's start. This is because we cannot ask if the CVQ is in its own vq group, because we don't know the control virtqueue index until the guest acknowledges the different features.

We can probe those during initialization I think. E.g doing some negotiation in the initialization phase.

We've developed this idea in other threads, let's continue there better.

Thanks!

Quoted 20 lines

Thanks

Quoted 17 lines

Thanks!

[1] https://www.mail-archive.com/[email protected]/msg901685.html

Quoted 12 lines
Quoted 12 lines
net/vhost-vdpa.c +2 -1
+                                     opts->x_svq, iova_tree);
            if (!ncs[i])
                goto err;
        }
 
        if (has_cvq) {
            nc = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name,
-                                 vdpa_device_fd, i, 1, false,
+                                 vdpa_device_fd, i, 1, num_as, false,
                                     opts->x_svq, iova_tree);
            if (!nc)
                goto err;
#44 lore raw
In reply to Eugenio Pérez
Quoted 8 lines · Eugenio Pérez

Control VQ is the way net devices use to send changes to the device state, like the number of active queues or its mac address.

QEMU needs to intercept this queue so it can track these changes and is able to migrate the device. It can do it from 1576dbb5bbc4 ("vdpa: Add x-svq to NetdevVhostVDPAOptions"). However, to enable x-svq implies to shadow all VirtIO device's virtqueues, which will damage performance.

Quoted 6 lines

This series adds address space isolation, so the device and the guest communicate directly with them (passthrough) and CVQ communication is split in
two: The guest communicates with QEMU and QEMU forwards the commands to the device.

Comments are welcome. Thanks!

This is not 7.2 material, right?

Quoted 51 lines

v6:
- Do not allocate SVQ resources like file descriptors if SVQ cannot be used.
- Disable shadow CVQ if the device does not support it because of net features.

v5:
- Move vring state in vhost_vdpa_get_vring_group instead of using a parameter.
- Rename VHOST_VDPA_NET_CVQ_PASSTHROUGH to VHOST_VDPA_NET_DATA_ASID

v4:
- Rebased on last CVQ start series, that allocated CVQ cmd bufs at load
- Squash vhost_vdpa_cvq_group_is_independent.
- Do not check for cvq index on vhost_vdpa_net_prepare, we only have one that callback registered in that NetClientInfo.
- Add comment specifying behavior if device does not support _F_ASID
- Update headers to a later Linux commit to not to remove SETUP_RNG_SEED

v3:
- Do not return an error but just print a warning if vdpa device initialization returns failure while getting AS num of VQ groups
- Delete extra newline

v2:
- Much as commented on series [1], handle vhost_net backend through NetClientInfo callbacks instead of directly.
- Fix not freeing SVQ properly when device does not support CVQ
- Add BIT_ULL missed checking device's backend feature for _F_ASID.

Eugenio Pérez (10):
vdpa: Use v->shadow_vqs_enabled in vhost_vdpa_svqs_start & stop
vhost: set SVQ device call handler at SVQ start
vhost: Allocate SVQ device file descriptors at device start
vdpa: add vhost_vdpa_net_valid_svq_features
vdpa: move SVQ vring features check to net/
vdpa: Allocate SVQ unconditionally
vdpa: Add asid parameter to vhost_vdpa_dma_map/unmap
vdpa: Store x-svq parameter in VhostVDPAState
vdpa: Add listener_shadow_vq to vhost_vdpa
vdpa: Always start CVQ in SVQ mode

 include/hw/virtio/vhost-vdpa.h     |  10 +-
 hw/virtio/vhost-shadow-virtqueue.c |  35 +-----
 hw/virtio/vhost-vdpa.c             | 114 ++++++++++---------
 net/vhost-vdpa.c                   | 171 ++++++++++++++++++++++++++---
 hw/virtio/trace-events             |   4 +-
 5 files changed, 222 insertions(+), 112 deletions(-)
Signature · 1 line
2.31.1
#45 lore raw
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Eugenio Pérez

Control VQ is the way net devices use to send changes to the device state, like the number of active queues or its mac address.

QEMU needs to intercept this queue so it can track these changes and is able to migrate the device. It can do it from 1576dbb5bbc4 ("vdpa: Add x-svq to NetdevVhostVDPAOptions"). However, to enable x-svq implies to shadow all VirtIO device's virtqueues, which will damage performance.

This series adds address space isolation, so the device and the guest communicate directly with them (passthrough) and CVQ communication is split in
two: The guest communicates with QEMU and QEMU forwards the commands to the device.

Quoted 5 lines
Quoted 1 line

Comments are welcome. Thanks!

This is not 7.2 material, right?

No it is not.

I should have noted it after PATCH in the subject, sorry.

Quoted 52 lines
Quoted 51 lines

v6:
- Do not allocate SVQ resources like file descriptors if SVQ cannot be used.
- Disable shadow CVQ if the device does not support it because of net features.

v5:
- Move vring state in vhost_vdpa_get_vring_group instead of using a parameter.
- Rename VHOST_VDPA_NET_CVQ_PASSTHROUGH to VHOST_VDPA_NET_DATA_ASID

v4:
- Rebased on last CVQ start series, that allocated CVQ cmd bufs at load
- Squash vhost_vdpa_cvq_group_is_independent.
- Do not check for cvq index on vhost_vdpa_net_prepare, we only have one that callback registered in that NetClientInfo.
- Add comment specifying behavior if device does not support _F_ASID
- Update headers to a later Linux commit to not to remove SETUP_RNG_SEED

v3:
- Do not return an error but just print a warning if vdpa device initialization returns failure while getting AS num of VQ groups
- Delete extra newline

v2:
- Much as commented on series [1], handle vhost_net backend through NetClientInfo callbacks instead of directly.
- Fix not freeing SVQ properly when device does not support CVQ
- Add BIT_ULL missed checking device's backend feature for _F_ASID.

Eugenio Pérez (10):
vdpa: Use v->shadow_vqs_enabled in vhost_vdpa_svqs_start & stop
vhost: set SVQ device call handler at SVQ start
vhost: Allocate SVQ device file descriptors at device start
vdpa: add vhost_vdpa_net_valid_svq_features
vdpa: move SVQ vring features check to net/
vdpa: Allocate SVQ unconditionally
vdpa: Add asid parameter to vhost_vdpa_dma_map/unmap
vdpa: Store x-svq parameter in VhostVDPAState
vdpa: Add listener_shadow_vq to vhost_vdpa
vdpa: Always start CVQ in SVQ mode

 include/hw/virtio/vhost-vdpa.h     |  10 +-
 hw/virtio/vhost-shadow-virtqueue.c |  35 +-----
 hw/virtio/vhost-vdpa.c             | 114 ++++++++++---------
 net/vhost-vdpa.c                   | 171 ++++++++++++++++++++++++++---
 hw/virtio/trace-events             |   4 +-
 5 files changed, 222 insertions(+), 112 deletions(-)

--
2.31.1