ASID support in vhost-vdpa net
Subsystems (2)
v6 cover ASID support in vhost-vdpa net
Control VQ is the way net devices use to send changes to the device state, like the number of active queues or its mac address.
QEMU needs to intercept this queue so it can track these changes and is able to migrate the device. It can do it from 1576dbb5bbc4 ("vdpa: Add x-svq to NetdevVhostVDPAOptions"). However, to enable x-svq implies to shadow all VirtIO device's virtqueues, which will damage performance.
This series adds address space isolation, so the device and the guest communicate directly with them (passthrough) and CVQ communication is split in
two: The guest communicates with QEMU and QEMU forwards the commands to the device.
Comments are welcome. Thanks!
v6:
- Do not allocate SVQ resources like file descriptors if SVQ cannot be used.
- Disable shadow CVQ if the device does not support it because of net features.
v5:
- Move vring state in vhost_vdpa_get_vring_group instead of using a parameter.
- Rename VHOST_VDPA_NET_CVQ_PASSTHROUGH to VHOST_VDPA_NET_DATA_ASID
v4:
- Rebased on last CVQ start series, that allocated CVQ cmd bufs at load
- Squash vhost_vdpa_cvq_group_is_independent.
- Do not check for cvq index on vhost_vdpa_net_prepare, we only have one that callback registered in that NetClientInfo.
- Add comment specifying behavior if device does not support _F_ASID
- Update headers to a later Linux commit to not to remove SETUP_RNG_SEED
v3:
- Do not return an error but just print a warning if vdpa device initialization returns failure while getting AS num of VQ groups
- Delete extra newline
v2:
- Much as commented on series [1], handle vhost_net backend through NetClientInfo callbacks instead of directly.
- Fix not freeing SVQ properly when device does not support CVQ
- Add BIT_ULL missed checking device's backend feature for _F_ASID.
Eugenio Pérez (10):
vdpa: Use v->shadow_vqs_enabled in vhost_vdpa_svqs_start & stop
vhost: set SVQ device call handler at SVQ start
vhost: Allocate SVQ device file descriptors at device start
vdpa: add vhost_vdpa_net_valid_svq_features
vdpa: move SVQ vring features check to net/
vdpa: Allocate SVQ unconditionally
vdpa: Add asid parameter to vhost_vdpa_dma_map/unmap
vdpa: Store x-svq parameter in VhostVDPAState
vdpa: Add listener_shadow_vq to vhost_vdpa
vdpa: Always start CVQ in SVQ mode
include/hw/virtio/vhost-vdpa.h | 10 +- hw/virtio/vhost-shadow-virtqueue.c | 35 +----- hw/virtio/vhost-vdpa.c | 114 ++++++++++--------- net/vhost-vdpa.c | 171 ++++++++++++++++++++++++++--- hw/virtio/trace-events | 4 +- 5 files changed, 222 insertions(+), 112 deletions(-)
Signature · 1 line
2.31.1
PATCH v6 1/10 vdpa: Use v->shadow_vqs_enabled in vhost_vdpa_svqs_start & stop
This function used to trust in v->shadow_vqs != NULL to know if it must start svq or not.
This is not going to be valid anymore, as qemu is going to allocate svq unconditionally (but it will only start them conditionally).
--- hw/virtio/vhost-vdpa.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
hw/virtio/vhost-vdpa.c +2 -2
@@ -1029,7 +1029,7 @@ static bool vhost_vdpa_svqs_start(struct vhost_dev *dev)Error *err =NULL ;unsigned i; -if (!v->shadow_vqs) { +if (!v->shadow_vqs_enabled) {return true; }
@@ -1082,7 +1082,7 @@ static void vhost_vdpa_svqs_stop(struct vhost_dev *dev) {struct vhost_vdpa *v = dev->opaque; -if (!v->shadow_vqs) { +if (!v->shadow_vqs_enabled) {return ; }
Signature · 1 line
2.31.1
Re: [PATCH v6 01/10] vdpa: Use v->shadow_vqs_enabled in vhost_vdpa_svqs_start & stop
Quoted 6 lines · Eugenio Pérez
This function used to trust in
v->shadow_vqs!= NULL to know if it must start svq or not.This is not going to be valid anymore, as qemu is going to allocate svq unconditionally (but it will only start them conditionally).
It might be a waste of memory if we did this. Any reason for this?
Thanks
Quoted 31 lines
Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-vdpa.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)hw/virtio/vhost-vdpa.c +2
-3@@ -1029,7 +1029,7 @@ static bool vhost_vdpa_svqs_start(struct vhost_dev *dev)Error *err =NULL ;unsigned i; -if (!v->shadow_vqs) { +if (!v->shadow_vqs_enabled) {return true; }@@ -1082,7 +1082,7 @@ static void vhost_vdpa_svqs_stop(struct vhost_dev *dev) {struct vhost_vdpa *v = dev->opaque; -if (!v->shadow_vqs) { +if (!v->shadow_vqs_enabled) {return ; } --2.31.1
Re: [PATCH v6 01/10] vdpa: Use v->shadow_vqs_enabled in vhost_vdpa_svqs_start & stop
Quoted 11 lines · Jason Wang
Quoted 6 lines · Eugenio Pérez
This function used to trust in
v->shadow_vqs!= NULL to know if it must start svq or not.This is not going to be valid anymore, as qemu is going to allocate svq unconditionally (but it will only start them conditionally).
It might be a waste of memory if we did this. Any reason for this?
Well, it's modelled after vhost_vdpa notifier member [1].
But sure we can reduce the memory usage if SVQ is not used. The first function that needs it is vhost_set_vring_kick. But I think it is not a good function to place the delayed allocation.
Would it work to move the allocation to vhost_set_features vhost op? It seems unlikely to me to call callbacks that can affect SVQ earlier than that one. Or maybe to create a new one and call it the first on vhost.c:vhost_dev_start?
Thanks!
[1] The notifier member already allocates VIRTIO_QUEUE_MAX VhostVDPAHostNotifier for each vhost_vdpa, It is easy to reduce at least to the number of virtqueues on a vhost_vdpa. Should I send a patch for this one?
Quoted 34 lines
Thanks
Quoted 31 lines
Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-vdpa.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)hw/virtio/vhost-vdpa.c +2
-3@@ -1029,7 +1029,7 @@ static bool vhost_vdpa_svqs_start(struct vhost_dev *dev)Error *err =NULL ;unsigned i; -if (!v->shadow_vqs) { +if (!v->shadow_vqs_enabled) {return true; }@@ -1082,7 +1082,7 @@ static void vhost_vdpa_svqs_stop(struct vhost_dev *dev) {struct vhost_vdpa *v = dev->opaque; -if (!v->shadow_vqs) { +if (!v->shadow_vqs_enabled) {return ; } --2.31.1
Re: [PATCH v6 01/10] vdpa: Use v->shadow_vqs_enabled in vhost_vdpa_svqs_start & stop
Quoted 10 lines · Eugenio Perez Martin
Quoted 8 lines · Jason Wang
Quoted 5 lines · Eugenio Pérez
This function used to trust in
v->shadow_vqs!= NULL to know if it must start svq or not.This is not going to be valid anymore, as qemu is going to allocate svq unconditionally (but it will only start them conditionally).
It might be a waste of memory if we did this. Any reason for this?
Well, it's modelled after
vhost_vdpanotifier member [1].
Right, this could be optimized in the future as well.
Quoted 9 lines
But sure we can reduce the memory usage if SVQ is not used. The first function that needs it is
vhost_set_vring_kick. But I think it is not a good function to place the delayed allocation.Would it work to move the allocation to
vhost_set_featuresvhost op? It seems unlikely to me to call callbacks that can affect SVQ earlier than that one. Or maybe to create a new one and call it the first on vhost.c:vhost_dev_start?
Rethink about this, so I think we can leave this in the future.
Thanks
Quoted 41 lines
Thanks!
[1] The notifier member already allocates
VIRTIO_QUEUE_MAXVhostVDPAHostNotifier for eachvhost_vdpa, It is easy to reduce at least to the number of virtqueues on avhost_vdpa. Should I send a patch for this one?Quoted 32 lines
Thanks
Quoted 30 lines
Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-vdpa.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)hw/virtio/vhost-vdpa.c +2
-3@@ -1029,7 +1029,7 @@ static bool vhost_vdpa_svqs_start(struct vhost_dev *dev)Error *err =NULL ;unsigned i; -if (!v->shadow_vqs) { +if (!v->shadow_vqs_enabled) {return true; }@@ -1082,7 +1082,7 @@ static void vhost_vdpa_svqs_stop(struct vhost_dev *dev) {struct vhost_vdpa *v = dev->opaque; -if (!v->shadow_vqs) { +if (!v->shadow_vqs_enabled) {return ; } --2.31.1
PATCH v6 2/10 vhost: set SVQ device call handler at SVQ start
In reply to Eugenio PérezBy the end of this series CVQ is shadowed as long as the features support it.
Since we don't know at the beginning of qemu running if this is supported, move the event notifier handler setting to the start of the SVQ, instead of the start of qemu run.
--- hw/virtio/vhost-shadow-virtqueue.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
hw/virtio/vhost-shadow-virtqueue.c +2 -2
@@ -648,6 +648,7 @@ void vhost_svq_start(VhostShadowVirtqueue *svq, VirtIODevice *vdev, {size_t desc_size, driver_size, device_size; +event_notifier_set_handler (&svq->hdev_call, vhost_svq_handle_call); svq->next_guest_avail_elem =NULL ; svq->shadow_avail_idx =0 ; svq->shadow_used_idx =0 ;
@@ -704,6 +705,7 @@ void vhost_svq_stop(VhostShadowVirtqueue *svq)g_free (svq->desc_state);qemu_vfree (svq->vring.desc);qemu_vfree (svq->vring.used); +event_notifier_set_handler (&svq->hdev_call,NULL ); }/**
@@ -740,7 +742,6 @@ VhostShadowVirtqueue *vhost_svq_new(VhostIOVATree *iova_tree, }event_notifier_init_fd (&svq->svq_kick,VHOST_FILE_UNBIND ); -event_notifier_set_handler (&svq->hdev_call, vhost_svq_handle_call); svq->iova_tree = iova_tree; svq->ops = ops; svq->ops_opaque = ops_opaque;
@@ -763,7 +764,6 @@ void vhost_svq_free(gpointer pvq)VhostShadowVirtqueue *vq = pvq;vhost_svq_stop (vq);event_notifier_cleanup (&vq->hdev_kick); -event_notifier_set_handler (&vq->hdev_call,NULL );event_notifier_cleanup (&vq->hdev_call);g_free (vq); }
Signature · 1 line
2.31.1
Re: [PATCH v6 02/10] vhost: set SVQ device call handler at SVQ start
Quoted 9 lines · Eugenio Pérez
By the end of this series CVQ is shadowed as long as the features support it.
Since we don't know at the beginning of qemu running if this is supported, move the event notifier handler setting to the start of the SVQ, instead of the start of qemu run.
Signed-off-by Eugenio Pérez <[email protected]>
Thanks
Quoted 43 lines
--- hw/virtio/vhost-shadow-virtqueue.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)hw/virtio/vhost-shadow-virtqueue.c +2
-3@@ -648,6 +648,7 @@ void vhost_svq_start(VhostShadowVirtqueue *svq, VirtIODevice *vdev, {size_t desc_size, driver_size, device_size; +event_notifier_set_handler (&svq->hdev_call, vhost_svq_handle_call); svq->next_guest_avail_elem =NULL ; svq->shadow_avail_idx =0 ; svq->shadow_used_idx =0 ;@@ -704,6 +705,7 @@ void vhost_svq_stop(VhostShadowVirtqueue *svq)g_free (svq->desc_state);qemu_vfree (svq->vring.desc);qemu_vfree (svq->vring.used); +event_notifier_set_handler (&svq->hdev_call,NULL ); }/** @@ -740,7 +742,6 @@ VhostShadowVirtqueue *vhost_svq_new(VhostIOVATree *iova_tree, }event_notifier_init_fd (&svq->svq_kick,VHOST_FILE_UNBIND ); -event_notifier_set_handler (&svq->hdev_call, vhost_svq_handle_call); svq->iova_tree = iova_tree; svq->ops = ops; svq->ops_opaque = ops_opaque;@@ -763,7 +764,6 @@ void vhost_svq_free(gpointer pvq)VhostShadowVirtqueue *vq = pvq;vhost_svq_stop (vq);event_notifier_cleanup (&vq->hdev_kick); -event_notifier_set_handler (&vq->hdev_call,NULL );event_notifier_cleanup (&vq->hdev_call);g_free (vq); } --2.31.1
PATCH v6 3/10 vhost: Allocate SVQ device file descriptors at device start
In reply to Eugenio PérezThe next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.
Delay device file descriptors until we know it at device start.
--- hw/virtio/vhost-shadow-virtqueue.c | 31 ++------------------------ hw/virtio/vhost-vdpa.c | 35 ++++++++++++++++++++++++------ 2 files changed, 30 insertions(+), 36 deletions(-)
hw/virtio/vhost-shadow-virtqueue.c +2 -29
@@ -715,43 +715,18 @@ void vhost_svq_stop(VhostShadowVirtqueue *svq)* @iova_tree: Tree to perform descriptors translations * @ops: SVQ owner callbacks * @ops_opaque: ops opaque pointer -* -* Returns the new virtqueue or NULL. -* -* In case of error, reason is reported through error_report. */ VhostShadowVirtqueue *vhost_svq_new (VhostIOVATree *iova_tree,const VhostShadowVirtqueueOps *ops,void *ops_opaque) { - g_autofree VhostShadowVirtqueue *svq = g_new0(VhostShadowVirtqueue, 1); -int r; - - r =event_notifier_init (&svq->hdev_kick,0 ); -if (r !=0 ) { -error_report ("Couldn't create kick event notifier: %s (%d)" , -g_strerror (errno), errno); - goto err_init_hdev_kick; - } - - r =event_notifier_init (&svq->hdev_call,0 ); -if (r !=0 ) { -error_report ("Couldn't create call event notifier: %s (%d)" , -g_strerror (errno), errno); - goto err_init_hdev_call; - } +VhostShadowVirtqueue *svq =g_new0 (VhostShadowVirtqueue,1 );event_notifier_init_fd (&svq->svq_kick,VHOST_FILE_UNBIND ); svq->iova_tree = iova_tree; svq->ops = ops; svq->ops_opaque = ops_opaque; -return g_steal_pointer (&svq); - -err_init_hdev_call: -event_notifier_cleanup (&svq->hdev_kick); - -err_init_hdev_kick: -return NULL ; +return svq; }/**
@@ -763,7 +738,5 @@ void vhost_svq_free(gpointer pvq) {VhostShadowVirtqueue *vq = pvq;vhost_svq_stop (vq); -event_notifier_cleanup (&vq->hdev_kick); -event_notifier_cleanup (&vq->hdev_call);g_free (vq); }
hw/virtio/vhost-vdpa.c +28 -7
@@ -428,15 +428,11 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v, shadow_vqs =g_ptr_array_new_full (hdev->nvqs, vhost_svq_free);for (unsigned n =0 ; n < hdev->nvqs; ++n) { -g_autoptr (VhostShadowVirtqueue) svq; +VhostShadowVirtqueue *svq; svq =vhost_svq_new (v->iova_tree, v->shadow_vq_ops, v->shadow_vq_ops_opaque); -if (unlikely (!svq)) { -error_setg (errp,"Cannot create svq %u" , n); -return -1 ; - } -g_ptr_array_add (shadow_vqs,g_steal_pointer (&svq)); +g_ptr_array_add (shadow_vqs, svq); } v->shadow_vqs =g_steal_pointer (&shadow_vqs);
@@ -864,11 +860,23 @@ static int vhost_vdpa_svq_set_fds(struct vhost_dev *dev,const EventNotifier *event_notifier = &svq->hdev_kick;int r; + r =event_notifier_init (&svq->hdev_kick,0 ); +if (r !=0 ) { +error_setg_errno (errp, -r,"Couldn't create kick event notifier" ); + goto err_init_hdev_kick; + } + + r =event_notifier_init (&svq->hdev_call,0 ); +if (r !=0 ) { +error_setg_errno (errp, -r,"Couldn't create call event notifier" ); + goto err_init_hdev_call; + } + file.fd =event_notifier_get_fd (event_notifier); r =vhost_vdpa_set_vring_dev_kick (dev, &file);if (unlikely (r !=0 )) {error_setg_errno (errp, -r,"Can't set device kick fd" ); -return r; + goto err_init_set_dev_fd; } event_notifier = &svq->hdev_call;
@@ -876,8 +884,18 @@ static int vhost_vdpa_svq_set_fds(struct vhost_dev *dev, r =vhost_vdpa_set_vring_dev_call (dev, &file);if (unlikely (r !=0 )) {error_setg_errno (errp, -r,"Can't set device call fd" ); + goto err_init_set_dev_fd; } +return 0 ; + +err_init_set_dev_fd: +event_notifier_set_handler (&svq->hdev_call,NULL ); + +err_init_hdev_call: +event_notifier_cleanup (&svq->hdev_kick); + +err_init_hdev_kick:return r; }
@@ -1089,6 +1107,9 @@ static void vhost_vdpa_svqs_stop(struct vhost_dev *dev)for (unsigned i =0 ; i < v->shadow_vqs->len; ++i) {VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, i);vhost_vdpa_svq_unmap_rings (dev, svq); + +event_notifier_cleanup (&svq->hdev_kick); +event_notifier_cleanup (&svq->hdev_call); } }
Signature · 1 line
2.31.1
Re: [PATCH v6 03/10] vhost: Allocate SVQ device file descriptors at device start
Quoted 7 lines · Eugenio Pérez
The next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.
Delay device file descriptors until we know it at device start.
Signed-off-by Eugenio Pérez <[email protected]>
Thanks
Quoted 142 lines
--- hw/virtio/vhost-shadow-virtqueue.c | 31 ++------------------------ hw/virtio/vhost-vdpa.c | 35 ++++++++++++++++++++++++------ 2 files changed, 30 insertions(+), 36 deletions(-)hw/virtio/vhost-shadow-virtqueue.c +2
-29@@ -715,43 +715,18 @@ void vhost_svq_stop(VhostShadowVirtqueue *svq)* @iova_tree: Tree to perform descriptors translations * @ops: SVQ owner callbacks * @ops_opaque: ops opaque pointer -* -* Returns the new virtqueue or NULL. -* -* In case of error, reason is reported through error_report. */ VhostShadowVirtqueue *vhost_svq_new (VhostIOVATree *iova_tree,const VhostShadowVirtqueueOps *ops,void *ops_opaque) { - g_autofree VhostShadowVirtqueue *svq = g_new0(VhostShadowVirtqueue, 1); -int r; - - r =event_notifier_init (&svq->hdev_kick,0 ); -if (r !=0 ) { -error_report ("Couldn't create kick event notifier: %s (%d)" , -g_strerror (errno), errno); - goto err_init_hdev_kick; - } - - r =event_notifier_init (&svq->hdev_call,0 ); -if (r !=0 ) { -error_report ("Couldn't create call event notifier: %s (%d)" , -g_strerror (errno), errno); - goto err_init_hdev_call; - } +VhostShadowVirtqueue *svq =g_new0 (VhostShadowVirtqueue,1 );event_notifier_init_fd (&svq->svq_kick,VHOST_FILE_UNBIND ); svq->iova_tree = iova_tree; svq->ops = ops; svq->ops_opaque = ops_opaque; -return g_steal_pointer (&svq); - -err_init_hdev_call: -event_notifier_cleanup (&svq->hdev_kick); - -err_init_hdev_kick: -return NULL ; +return svq; }/** @@ -763,7 +738,5 @@ void vhost_svq_free(gpointer pvq) {VhostShadowVirtqueue *vq = pvq;vhost_svq_stop (vq); -event_notifier_cleanup (&vq->hdev_kick); -event_notifier_cleanup (&vq->hdev_call);g_free (vq); }hw/virtio/vhost-vdpa.c +28
-8@@ -428,15 +428,11 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v, shadow_vqs =g_ptr_array_new_full (hdev->nvqs, vhost_svq_free);for (unsigned n =0 ; n < hdev->nvqs; ++n) { -g_autoptr (VhostShadowVirtqueue) svq; +VhostShadowVirtqueue *svq; svq =vhost_svq_new (v->iova_tree, v->shadow_vq_ops, v->shadow_vq_ops_opaque); -if (unlikely (!svq)) { -error_setg (errp,"Cannot create svq %u" , n); -return -1 ; - } -g_ptr_array_add (shadow_vqs,g_steal_pointer (&svq)); +g_ptr_array_add (shadow_vqs, svq); } v->shadow_vqs =g_steal_pointer (&shadow_vqs);@@ -864,11 +860,23 @@ static int vhost_vdpa_svq_set_fds(struct vhost_dev *dev,const EventNotifier *event_notifier = &svq->hdev_kick;int r; + r =event_notifier_init (&svq->hdev_kick,0 ); +if (r !=0 ) { +error_setg_errno (errp, -r,"Couldn't create kick event notifier" ); + goto err_init_hdev_kick; + } + + r =event_notifier_init (&svq->hdev_call,0 ); +if (r !=0 ) { +error_setg_errno (errp, -r,"Couldn't create call event notifier" ); + goto err_init_hdev_call; + } + file.fd =event_notifier_get_fd (event_notifier); r =vhost_vdpa_set_vring_dev_kick (dev, &file);if (unlikely (r !=0 )) {error_setg_errno (errp, -r,"Can't set device kick fd" ); -return r; + goto err_init_set_dev_fd; } event_notifier = &svq->hdev_call;@@ -876,8 +884,18 @@ static int vhost_vdpa_svq_set_fds(struct vhost_dev *dev, r =vhost_vdpa_set_vring_dev_call (dev, &file);if (unlikely (r !=0 )) {error_setg_errno (errp, -r,"Can't set device call fd" ); + goto err_init_set_dev_fd; } +return 0 ; + +err_init_set_dev_fd: +event_notifier_set_handler (&svq->hdev_call,NULL ); + +err_init_hdev_call: +event_notifier_cleanup (&svq->hdev_kick); + +err_init_hdev_kick:return r; }@@ -1089,6 +1107,9 @@ static void vhost_vdpa_svqs_stop(struct vhost_dev *dev)for (unsigned i =0 ; i < v->shadow_vqs->len; ++i) {VhostShadowVirtqueue *svq =g_ptr_array_index (v->shadow_vqs, i);vhost_vdpa_svq_unmap_rings (dev, svq); + +event_notifier_cleanup (&svq->hdev_kick); +event_notifier_cleanup (&svq->hdev_call); } } --2.31.1
PATCH v6 4/10 vdpa: add vhost_vdpa_net_valid_svq_features
In reply to Eugenio PérezIt will be reused at vdpa device start so let's extract in its own function
--- net/vhost-vdpa.c | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-)
net/vhost-vdpa.c +17 -9
@@ -106,6 +106,22 @@ VHostNetState *vhost_vdpa_get_vhost_net(NetClientState *nc)return s->vhost_net; } +static bool vhost_vdpa_net_valid_svq_features (uint64_t features,Error **errp) +{ +uint64_t invalid_dev_features = + features & ~vdpa_svq_device_features & +/* Transport are all accepted at this point */ + ~MAKE_64BIT_MASK (VIRTIO_TRANSPORT_F_START , +VIRTIO_TRANSPORT_F_END -VIRTIO_TRANSPORT_F_START ); + +if (invalid_dev_features) { +error_setg (errp,"vdpa svq does not work with features 0x%" PRIx64, + invalid_dev_features); + } + +return !invalid_dev_features; +} +static int vhost_vdpa_net_check_device_id (struct vhost_net *net) {uint32_t device_id;
@@ -675,15 +691,7 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,if (opts->x_svq) {struct vhost_vdpa_iova_range iova_range; -uint64_t invalid_dev_features = - features & ~vdpa_svq_device_features & -/* Transport are all accepted at this point */ - ~MAKE_64BIT_MASK (VIRTIO_TRANSPORT_F_START , -VIRTIO_TRANSPORT_F_END -VIRTIO_TRANSPORT_F_START ); - -if (invalid_dev_features) { -error_setg (errp,"vdpa svq does not work with features 0x%" PRIx64, - invalid_dev_features); +if (!vhost_vdpa_net_valid_svq_features (features, errp)) { goto err_svq; }
Signature · 1 line
2.31.1
Re: [PATCH v6 04/10] vdpa: add vhost_vdpa_net_valid_svq_features
Quoted 5 lines · Eugenio Pérez
It will be reused at vdpa device start so let's extract in its own function
Signed-off-by Eugenio Pérez <[email protected]>---
Thanks
Quoted 50 lines
net/vhost-vdpa.c | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-)net/vhost-vdpa.c +17
-10@@ -106,6 +106,22 @@ VHostNetState *vhost_vdpa_get_vhost_net(NetClientState *nc)return s->vhost_net; } +static bool vhost_vdpa_net_valid_svq_features (uint64_t features,Error **errp) +{ +uint64_t invalid_dev_features = + features & ~vdpa_svq_device_features & +/* Transport are all accepted at this point */ + ~MAKE_64BIT_MASK (VIRTIO_TRANSPORT_F_START , +VIRTIO_TRANSPORT_F_END -VIRTIO_TRANSPORT_F_START ); + +if (invalid_dev_features) { +error_setg (errp,"vdpa svq does not work with features 0x%" PRIx64, + invalid_dev_features); + } + +return !invalid_dev_features; +} +static int vhost_vdpa_net_check_device_id (struct vhost_net *net) {uint32_t device_id;@@ -675,15 +691,7 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,if (opts->x_svq) {struct vhost_vdpa_iova_range iova_range; -uint64_t invalid_dev_features = - features & ~vdpa_svq_device_features & -/* Transport are all accepted at this point */ - ~MAKE_64BIT_MASK (VIRTIO_TRANSPORT_F_START , -VIRTIO_TRANSPORT_F_END -VIRTIO_TRANSPORT_F_START ); - -if (invalid_dev_features) { -error_setg (errp,"vdpa svq does not work with features 0x%" PRIx64, - invalid_dev_features); +if (!vhost_vdpa_net_valid_svq_features (features, errp)) { goto err_svq; } --2.31.1
PATCH v6 5/10 vdpa: move SVQ vring features check to net/
In reply to Eugenio PérezThe next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.
Since the moved checks will be already evaluated at net/ to know if it is ok to shadow CVQ, move them.
--- hw/virtio/vhost-vdpa.c | 33 ++------------------------------- net/vhost-vdpa.c | 3 ++- 2 files changed, 4 insertions(+), 32 deletions(-)
hw/virtio/vhost-vdpa.c +2 -31
@@ -402,29 +402,9 @@ static int vhost_vdpa_get_dev_features(struct vhost_dev *dev,return ret; } -static int vhost_vdpa_init_svq (struct vhost_dev *hdev,struct vhost_vdpa *v, -Error **errp) +static void vhost_vdpa_init_svq (struct vhost_dev *hdev,struct vhost_vdpa *v) {g_autoptr (GPtrArray) shadow_vqs =NULL ; -uint64_t dev_features, svq_features; -int r; -bool ok; - -if (!v->shadow_vqs_enabled) { -return 0 ; - } - - r =vhost_vdpa_get_dev_features (hdev, &dev_features); -if (r !=0 ) { -error_setg_errno (errp, -r,"Can't get vdpa device features" ); -return r; - } - - svq_features = dev_features; - ok =vhost_svq_valid_features (svq_features, errp); -if (unlikely (!ok)) { -return -1 ; - } shadow_vqs =g_ptr_array_new_full (hdev->nvqs, vhost_svq_free);for (unsigned n =0 ; n < hdev->nvqs; ++n) {
@@ -436,7 +416,6 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v, } v->shadow_vqs =g_steal_pointer (&shadow_vqs); -return 0 ; }static int vhost_vdpa_init (struct vhost_dev *dev,void *opaque,Error **errp)
@@ -461,11 +440,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) dev->opaque = opaque ; v->listener = vhost_vdpa_memory_listener; v->msg_type =VHOST_IOTLB_MSG_V2 ; - ret =vhost_vdpa_init_svq (dev, v, errp); -if (ret) { - goto err; - } - +vhost_vdpa_init_svq (dev, v);vhost_vdpa_get_iova_range (v);if (!vhost_vdpa_first_dev (dev)) {
@@ -476,10 +451,6 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) VIRTIO_CONFIG_S_DRIVER);return 0 ; - -err: -ram_block_discard_disable (false); -return ret; } static void vhost_vdpa_host_notifier_uninit(struct vhost_dev *dev,
net/vhost-vdpa.c +2 -1
@@ -117,9 +117,10 @@ static bool vhost_vdpa_net_valid_svq_features(uint64_t features, Error **errp)if (invalid_dev_features) {error_setg (errp,"vdpa svq does not work with features 0x%" PRIx64, invalid_dev_features); +return false; } -return !invalid_dev_features; +return vhost_svq_valid_features (features, errp); }static int vhost_vdpa_net_check_device_id (struct vhost_net *net)
Signature · 1 line
2.31.1
Re: [PATCH v6 05/10] vdpa: move SVQ vring features check to net/
Quoted 2 lines · Eugenio Pérez
The next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.
If I was not wrong, there's no device specific feature that is checked in the function. So it should be general enough to be used by devices other than net. Then I don't see any advantage of doing this.
Thanks
Quoted 93 lines
Since the moved checks will be already evaluated at
net/to know if it is ok to shadow CVQ, move them.Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-vdpa.c | 33 ++------------------------------- net/vhost-vdpa.c | 3 ++- 2 files changed, 4 insertions(+), 32 deletions(-)hw/virtio/vhost-vdpa.c +2
-31@@ -402,29 +402,9 @@ static int vhost_vdpa_get_dev_features(struct vhost_dev *dev,return ret; } -static int vhost_vdpa_init_svq (struct vhost_dev *hdev,struct vhost_vdpa *v, -Error **errp) +static void vhost_vdpa_init_svq (struct vhost_dev *hdev,struct vhost_vdpa *v) {g_autoptr (GPtrArray) shadow_vqs =NULL ; -uint64_t dev_features, svq_features; -int r; -bool ok; - -if (!v->shadow_vqs_enabled) { -return 0 ; - } - - r =vhost_vdpa_get_dev_features (hdev, &dev_features); -if (r !=0 ) { -error_setg_errno (errp, -r,"Can't get vdpa device features" ); -return r; - } - - svq_features = dev_features; - ok =vhost_svq_valid_features (svq_features, errp); -if (unlikely (!ok)) { -return -1 ; - } shadow_vqs =g_ptr_array_new_full (hdev->nvqs, vhost_svq_free);for (unsigned n =0 ; n < hdev->nvqs; ++n) {@@ -436,7 +416,6 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v, } v->shadow_vqs =g_steal_pointer (&shadow_vqs); -return 0 ; }static int vhost_vdpa_init (struct vhost_dev *dev,void *opaque,Error **errp)@@ -461,11 +440,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) dev->opaque = opaque ; v->listener = vhost_vdpa_memory_listener; v->msg_type =VHOST_IOTLB_MSG_V2 ; - ret =vhost_vdpa_init_svq (dev, v, errp); -if (ret) { - goto err; - } - +vhost_vdpa_init_svq (dev, v);vhost_vdpa_get_iova_range (v);if (!vhost_vdpa_first_dev (dev)) {@@ -476,10 +451,6 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) VIRTIO_CONFIG_S_DRIVER);return 0 ; - -err: -ram_block_discard_disable (false); -return ret; } static void vhost_vdpa_host_notifier_uninit(struct vhost_dev *dev,net/vhost-vdpa.c +2
-1@@ -117,9 +117,10 @@ static bool vhost_vdpa_net_valid_svq_features(uint64_t features, Error **errp)if (invalid_dev_features) {error_setg (errp,"vdpa svq does not work with features 0x%" PRIx64, invalid_dev_features); +return false; } -return !invalid_dev_features; +return vhost_svq_valid_features (features, errp); }static int vhost_vdpa_net_check_device_id (struct vhost_net *net)
Re: [PATCH v6 05/10] vdpa: move SVQ vring features check to net/
Quoted 11 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez
The next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.
If I was not wrong, there's no device specific feature that is checked in the function. So it should be general enough to be used by devices other than net. Then I don't see any advantage of doing this.
Because vhost_vdpa_init_svq is called at qemu boot, failing if it is not possible to shadow the Virtqueue.
Now the CVQ will be shadowed if possible, so we need to check this at device start, not at initialization. To store this information at boot time is not valid anymore, because v->shadow_vqs_enabled is not valid at this time anymore.
Thanks!
Quoted 97 lines
Thanks
Quoted 93 lines
Since the moved checks will be already evaluated at
net/to know if it is ok to shadow CVQ, move them.Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-vdpa.c | 33 ++------------------------------- net/vhost-vdpa.c | 3 ++- 2 files changed, 4 insertions(+), 32 deletions(-)hw/virtio/vhost-vdpa.c +2
-31@@ -402,29 +402,9 @@ static int vhost_vdpa_get_dev_features(struct vhost_dev *dev,return ret; } -static int vhost_vdpa_init_svq (struct vhost_dev *hdev,struct vhost_vdpa *v, -Error **errp) +static void vhost_vdpa_init_svq (struct vhost_dev *hdev,struct vhost_vdpa *v) {g_autoptr (GPtrArray) shadow_vqs =NULL ; -uint64_t dev_features, svq_features; -int r; -bool ok; - -if (!v->shadow_vqs_enabled) { -return 0 ; - } - - r =vhost_vdpa_get_dev_features (hdev, &dev_features); -if (r !=0 ) { -error_setg_errno (errp, -r,"Can't get vdpa device features" ); -return r; - } - - svq_features = dev_features; - ok =vhost_svq_valid_features (svq_features, errp); -if (unlikely (!ok)) { -return -1 ; - } shadow_vqs =g_ptr_array_new_full (hdev->nvqs, vhost_svq_free);for (unsigned n =0 ; n < hdev->nvqs; ++n) {@@ -436,7 +416,6 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v, } v->shadow_vqs =g_steal_pointer (&shadow_vqs); -return 0 ; }static int vhost_vdpa_init (struct vhost_dev *dev,void *opaque,Error **errp)@@ -461,11 +440,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) dev->opaque = opaque ; v->listener = vhost_vdpa_memory_listener; v->msg_type =VHOST_IOTLB_MSG_V2 ; - ret =vhost_vdpa_init_svq (dev, v, errp); -if (ret) { - goto err; - } - +vhost_vdpa_init_svq (dev, v);vhost_vdpa_get_iova_range (v);if (!vhost_vdpa_first_dev (dev)) {@@ -476,10 +451,6 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) VIRTIO_CONFIG_S_DRIVER);return 0 ; - -err: -ram_block_discard_disable (false); -return ret; } static void vhost_vdpa_host_notifier_uninit(struct vhost_dev *dev,net/vhost-vdpa.c +2
-1@@ -117,9 +117,10 @@ static bool vhost_vdpa_net_valid_svq_features(uint64_t features, Error **errp)if (invalid_dev_features) {error_setg (errp,"vdpa svq does not work with features 0x%" PRIx64, invalid_dev_features); +return false; } -return !invalid_dev_features; +return vhost_svq_valid_features (features, errp); }static int vhost_vdpa_net_check_device_id (struct vhost_net *net)
Re: [PATCH v6 05/10] vdpa: move SVQ vring features check to net/
Quoted 10 lines · Eugenio Perez Martin
Quoted 9 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez
The next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.
If I was not wrong, there's no device specific feature that is checked in the function. So it should be general enough to be used by devices other than net. Then I don't see any advantage of doing this.
Quoted 5 lines
Because
vhost_vdpa_init_svqis called at qemu boot, failing if it is not possible to shadow the Virtqueue.Now the CVQ will be shadowed if possible, so we need to check this at device start, not at initialization.
Any reason we can't check this at device start? We don't need driver_features and we can do any probing to make sure cvq has an unique group during initialization time.
Quoted 3 lines
To store this information at boot
time is not valid anymore, becausev->shadow_vqs_enabledis not valid at this time anymore.
Ok, but this doesn't explain why it is net specific but vhost-vdpa specific.
Thanks
Quoted 98 lines
Thanks!
Quoted 95 lines
Thanks
Quoted 92 lines
Since the moved checks will be already evaluated at
net/to know if it is ok to shadow CVQ, move them.Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-vdpa.c | 33 ++------------------------------- net/vhost-vdpa.c | 3 ++- 2 files changed, 4 insertions(+), 32 deletions(-)hw/virtio/vhost-vdpa.c +2
-31@@ -402,29 +402,9 @@ static int vhost_vdpa_get_dev_features(struct vhost_dev *dev,return ret; } -static int vhost_vdpa_init_svq (struct vhost_dev *hdev,struct vhost_vdpa *v, -Error **errp) +static void vhost_vdpa_init_svq (struct vhost_dev *hdev,struct vhost_vdpa *v) {g_autoptr (GPtrArray) shadow_vqs =NULL ; -uint64_t dev_features, svq_features; -int r; -bool ok; - -if (!v->shadow_vqs_enabled) { -return 0 ; - } - - r =vhost_vdpa_get_dev_features (hdev, &dev_features); -if (r !=0 ) { -error_setg_errno (errp, -r,"Can't get vdpa device features" ); -return r; - } - - svq_features = dev_features; - ok =vhost_svq_valid_features (svq_features, errp); -if (unlikely (!ok)) { -return -1 ; - } shadow_vqs =g_ptr_array_new_full (hdev->nvqs, vhost_svq_free);for (unsigned n =0 ; n < hdev->nvqs; ++n) {@@ -436,7 +416,6 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v, } v->shadow_vqs =g_steal_pointer (&shadow_vqs); -return 0 ; }static int vhost_vdpa_init (struct vhost_dev *dev,void *opaque,Error **errp)@@ -461,11 +440,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) dev->opaque = opaque ; v->listener = vhost_vdpa_memory_listener; v->msg_type =VHOST_IOTLB_MSG_V2 ; - ret =vhost_vdpa_init_svq (dev, v, errp); -if (ret) { - goto err; - } - +vhost_vdpa_init_svq (dev, v);vhost_vdpa_get_iova_range (v);if (!vhost_vdpa_first_dev (dev)) {@@ -476,10 +451,6 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) VIRTIO_CONFIG_S_DRIVER);return 0 ; - -err: -ram_block_discard_disable (false); -return ret; } static void vhost_vdpa_host_notifier_uninit(struct vhost_dev *dev,net/vhost-vdpa.c +2
-1@@ -117,9 +117,10 @@ static bool vhost_vdpa_net_valid_svq_features(uint64_t features, Error **errp)if (invalid_dev_features) {error_setg (errp,"vdpa svq does not work with features 0x%" PRIx64, invalid_dev_features); +return false; } -return !invalid_dev_features; +return vhost_svq_valid_features (features, errp); }static int vhost_vdpa_net_check_device_id (struct vhost_net *net)
Re: [PATCH v6 05/10] vdpa: move SVQ vring features check to net/
Quoted 16 lines · Jason Wang
Quoted 13 lines · Eugenio Perez Martin
Quoted 9 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez
The next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.
If I was not wrong, there's no device specific feature that is checked in the function. So it should be general enough to be used by devices other than net. Then I don't see any advantage of doing this.
Because
vhost_vdpa_init_svqis called at qemu boot, failing if it is not possible to shadow the Virtqueue.
Quoted 8 lines
Quoted 2 lines
Now the CVQ will be shadowed if possible, so we need to check this at device start, not at initialization.
Any reason we can't check this at device start? We don't need
driver_featuresand we can do any probing to make sure cvq has an unique group during initialization time.
We need the CVQ index to check if it has an independent group. CVQ index depends on the features the guest's ack:
* If it acks _F_MQ, it is the last one.
* If it doesn't, CVQ idx is 2.
We cannot have acked features at initialization, and they could
change: It is valid for a guest to ack _F_MQ, then reset the device, then not ack it.
Quoted 8 lines
Quoted 4 lines
To store this information at boot
time is not valid anymore, becausev->shadow_vqs_enabledis not valid at this time anymore.Ok, but this doesn't explain why it is net specific but vhost-vdpa specific.
We can try to move it to a vhost op, but we have the same problem as the svq array allocation: We don't have the right place in vhost ops to check this. Maybe vhost_set_features is the right one here?
Thanks!
Quoted 102 lines
Thanks
Quoted 98 lines
Thanks!
Quoted 95 lines
Thanks
Quoted 92 lines
Since the moved checks will be already evaluated at
net/to know if it is ok to shadow CVQ, move them.Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-vdpa.c | 33 ++------------------------------- net/vhost-vdpa.c | 3 ++- 2 files changed, 4 insertions(+), 32 deletions(-)hw/virtio/vhost-vdpa.c +2
-31@@ -402,29 +402,9 @@ static int vhost_vdpa_get_dev_features(struct vhost_dev *dev,return ret; } -static int vhost_vdpa_init_svq (struct vhost_dev *hdev,struct vhost_vdpa *v, -Error **errp) +static void vhost_vdpa_init_svq (struct vhost_dev *hdev,struct vhost_vdpa *v) {g_autoptr (GPtrArray) shadow_vqs =NULL ; -uint64_t dev_features, svq_features; -int r; -bool ok; - -if (!v->shadow_vqs_enabled) { -return 0 ; - } - - r =vhost_vdpa_get_dev_features (hdev, &dev_features); -if (r !=0 ) { -error_setg_errno (errp, -r,"Can't get vdpa device features" ); -return r; - } - - svq_features = dev_features; - ok =vhost_svq_valid_features (svq_features, errp); -if (unlikely (!ok)) { -return -1 ; - } shadow_vqs =g_ptr_array_new_full (hdev->nvqs, vhost_svq_free);for (unsigned n =0 ; n < hdev->nvqs; ++n) {@@ -436,7 +416,6 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v, } v->shadow_vqs =g_steal_pointer (&shadow_vqs); -return 0 ; }static int vhost_vdpa_init (struct vhost_dev *dev,void *opaque,Error **errp)@@ -461,11 +440,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) dev->opaque = opaque ; v->listener = vhost_vdpa_memory_listener; v->msg_type =VHOST_IOTLB_MSG_V2 ; - ret =vhost_vdpa_init_svq (dev, v, errp); -if (ret) { - goto err; - } - +vhost_vdpa_init_svq (dev, v);vhost_vdpa_get_iova_range (v);if (!vhost_vdpa_first_dev (dev)) {@@ -476,10 +451,6 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) VIRTIO_CONFIG_S_DRIVER);return 0 ; - -err: -ram_block_discard_disable (false); -return ret; } static void vhost_vdpa_host_notifier_uninit(struct vhost_dev *dev,net/vhost-vdpa.c +2
-1@@ -117,9 +117,10 @@ static bool vhost_vdpa_net_valid_svq_features(uint64_t features, Error **errp)if (invalid_dev_features) {error_setg (errp,"vdpa svq does not work with features 0x%" PRIx64, invalid_dev_features); +return false; } -return !invalid_dev_features; +return vhost_svq_valid_features (features, errp); }static int vhost_vdpa_net_check_device_id (struct vhost_net *net)
Re: [PATCH v6 05/10] vdpa: move SVQ vring features check to net/
Quoted 27 lines · Eugenio Perez Martin
Quoted 16 lines · Jason Wang
Quoted 13 lines · Eugenio Perez Martin
Quoted 9 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez
The next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.
If I was not wrong, there's no device specific feature that is checked in the function. So it should be general enough to be used by devices other than net. Then I don't see any advantage of doing this.
Because
vhost_vdpa_init_svqis called at qemu boot, failing if it is not possible to shadow the Virtqueue.Quoted 8 lines
Quoted 2 lines
Now the CVQ will be shadowed if possible, so we need to check this at device start, not at initialization.
Any reason we can't check this at device start? We don't need
driver_featuresand we can do any probing to make sure cvq has an unique group during initialization time.
Quoted 8 lines
We need the CVQ index to check if it has an independent group. CVQ index depends on the features the guest's ack:
* If it acks_F_MQ, it is the last one.
* If it doesn't, CVQ idx is 2.We cannot have acked features at initialization, and they could
change: It is valid for a guest to ack_F_MQ, then reset the device, then not ack it.
Can we do some probing by negotiating _F_MQ if the device offers it, then we can know if cvq has a unique group?
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
To store this information at boot
time is not valid anymore, becausev->shadow_vqs_enabledis not valid at this time anymore.
Quoted 6 lines
Quoted 2 lines
Ok, but this doesn't explain why it is net specific but vhost-vdpa specific.
We can try to move it to a vhost op, but we have the same problem as the svq array allocation: We don't have the right place in vhost ops to check this. Maybe
vhost_set_featuresis the right one here?
If we can do all the probing at the initialization phase, we can do everything there.
Thanks
Quoted 106 lines
Thanks!
Quoted 102 lines
Thanks
Quoted 98 lines
Thanks!
Quoted 95 lines
Thanks
Quoted 92 lines
Since the moved checks will be already evaluated at
net/to know if it is ok to shadow CVQ, move them.Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-vdpa.c | 33 ++------------------------------- net/vhost-vdpa.c | 3 ++- 2 files changed, 4 insertions(+), 32 deletions(-)hw/virtio/vhost-vdpa.c +2
-31@@ -402,29 +402,9 @@ static int vhost_vdpa_get_dev_features(struct vhost_dev *dev,return ret; } -static int vhost_vdpa_init_svq (struct vhost_dev *hdev,struct vhost_vdpa *v, -Error **errp) +static void vhost_vdpa_init_svq (struct vhost_dev *hdev,struct vhost_vdpa *v) {g_autoptr (GPtrArray) shadow_vqs =NULL ; -uint64_t dev_features, svq_features; -int r; -bool ok; - -if (!v->shadow_vqs_enabled) { -return 0 ; - } - - r =vhost_vdpa_get_dev_features (hdev, &dev_features); -if (r !=0 ) { -error_setg_errno (errp, -r,"Can't get vdpa device features" ); -return r; - } - - svq_features = dev_features; - ok =vhost_svq_valid_features (svq_features, errp); -if (unlikely (!ok)) { -return -1 ; - } shadow_vqs =g_ptr_array_new_full (hdev->nvqs, vhost_svq_free);for (unsigned n =0 ; n < hdev->nvqs; ++n) {@@ -436,7 +416,6 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v, } v->shadow_vqs =g_steal_pointer (&shadow_vqs); -return 0 ; }static int vhost_vdpa_init (struct vhost_dev *dev,void *opaque,Error **errp)@@ -461,11 +440,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) dev->opaque = opaque ; v->listener = vhost_vdpa_memory_listener; v->msg_type =VHOST_IOTLB_MSG_V2 ; - ret =vhost_vdpa_init_svq (dev, v, errp); -if (ret) { - goto err; - } - +vhost_vdpa_init_svq (dev, v);vhost_vdpa_get_iova_range (v);if (!vhost_vdpa_first_dev (dev)) {@@ -476,10 +451,6 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) VIRTIO_CONFIG_S_DRIVER);return 0 ; - -err: -ram_block_discard_disable (false); -return ret; } static void vhost_vdpa_host_notifier_uninit(struct vhost_dev *dev,net/vhost-vdpa.c +2
-1@@ -117,9 +117,10 @@ static bool vhost_vdpa_net_valid_svq_features(uint64_t features, Error **errp)if (invalid_dev_features) {error_setg (errp,"vdpa svq does not work with features 0x%" PRIx64, invalid_dev_features); +return false; } -return !invalid_dev_features; +return vhost_svq_valid_features (features, errp); }static int vhost_vdpa_net_check_device_id (struct vhost_net *net)
Re: [PATCH v6 05/10] vdpa: move SVQ vring features check to net/
Quoted 35 lines · Jason Wang
Quoted 32 lines · Eugenio Perez Martin
Quoted 16 lines · Jason Wang
Quoted 13 lines · Eugenio Perez Martin
Quoted 9 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez
The next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.
If I was not wrong, there's no device specific feature that is checked in the function. So it should be general enough to be used by devices other than net. Then I don't see any advantage of doing this.
Because
vhost_vdpa_init_svqis called at qemu boot, failing if it is not possible to shadow the Virtqueue.Quoted 8 lines
Quoted 2 lines
Now the CVQ will be shadowed if possible, so we need to check this at device start, not at initialization.
Any reason we can't check this at device start? We don't need
driver_featuresand we can do any probing to make sure cvq has an unique group during initialization time.We need the CVQ index to check if it has an independent group. CVQ index depends on the features the guest's ack:
* If it acks_F_MQ, it is the last one.
* If it doesn't, CVQ idx is 2.
Quoted 7 lines
Quoted 3 lines
We cannot have acked features at initialization, and they could
change: It is valid for a guest to ack_F_MQ, then reset the device, then not ack it.Can we do some probing by negotiating
_F_MQif the device offers it, then we can know if cvq has a unique group?
What if the guest does not ack _F_MQ?
To be completed it would go like:
* Probe negotiate _F_MQ, check unique group,
* Probe negotiate !_F_MQ, check unique group,
* Actually negotiate with the guest's feature set.
* React to failures. Probably the same way as if the CVQ is not isolated, disabling SVQ?
To me it seems simpler to specify somehow that the vq must be independent.
Thanks!
Quoted 126 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
To store this information at boot
time is not valid anymore, becausev->shadow_vqs_enabledis not valid at this time anymore.Quoted 6 lines
Quoted 2 lines
Ok, but this doesn't explain why it is net specific but vhost-vdpa specific.
We can try to move it to a vhost op, but we have the same problem as the svq array allocation: We don't have the right place in vhost ops to check this. Maybe
vhost_set_featuresis the right one here?If we can do all the probing at the initialization phase, we can do everything there.
Thanks
Quoted 106 lines
Thanks!
Quoted 102 lines
Thanks
Quoted 98 lines
Thanks!
Quoted 95 lines
Thanks
Quoted 92 lines
Since the moved checks will be already evaluated at
net/to know if it is ok to shadow CVQ, move them.Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-vdpa.c | 33 ++------------------------------- net/vhost-vdpa.c | 3 ++- 2 files changed, 4 insertions(+), 32 deletions(-)hw/virtio/vhost-vdpa.c +2
-31@@ -402,29 +402,9 @@ static int vhost_vdpa_get_dev_features(struct vhost_dev *dev,return ret; } -static int vhost_vdpa_init_svq (struct vhost_dev *hdev,struct vhost_vdpa *v, -Error **errp) +static void vhost_vdpa_init_svq (struct vhost_dev *hdev,struct vhost_vdpa *v) {g_autoptr (GPtrArray) shadow_vqs =NULL ; -uint64_t dev_features, svq_features; -int r; -bool ok; - -if (!v->shadow_vqs_enabled) { -return 0 ; - } - - r =vhost_vdpa_get_dev_features (hdev, &dev_features); -if (r !=0 ) { -error_setg_errno (errp, -r,"Can't get vdpa device features" ); -return r; - } - - svq_features = dev_features; - ok =vhost_svq_valid_features (svq_features, errp); -if (unlikely (!ok)) { -return -1 ; - } shadow_vqs =g_ptr_array_new_full (hdev->nvqs, vhost_svq_free);for (unsigned n =0 ; n < hdev->nvqs; ++n) {@@ -436,7 +416,6 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v, } v->shadow_vqs =g_steal_pointer (&shadow_vqs); -return 0 ; }static int vhost_vdpa_init (struct vhost_dev *dev,void *opaque,Error **errp)@@ -461,11 +440,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) dev->opaque = opaque ; v->listener = vhost_vdpa_memory_listener; v->msg_type =VHOST_IOTLB_MSG_V2 ; - ret =vhost_vdpa_init_svq (dev, v, errp); -if (ret) { - goto err; - } - +vhost_vdpa_init_svq (dev, v);vhost_vdpa_get_iova_range (v);if (!vhost_vdpa_first_dev (dev)) {@@ -476,10 +451,6 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) VIRTIO_CONFIG_S_DRIVER);return 0 ; - -err: -ram_block_discard_disable (false); -return ret; } static void vhost_vdpa_host_notifier_uninit(struct vhost_dev *dev,net/vhost-vdpa.c +2
-1@@ -117,9 +117,10 @@ static bool vhost_vdpa_net_valid_svq_features(uint64_t features, Error **errp)if (invalid_dev_features) {error_setg (errp,"vdpa svq does not work with features 0x%" PRIx64, invalid_dev_features); +return false; } -return !invalid_dev_features; +return vhost_svq_valid_features (features, errp); }static int vhost_vdpa_net_check_device_id (struct vhost_net *net)
Re: [PATCH v6 05/10] vdpa: move SVQ vring features check to net/
Quoted 35 lines · Eugenio Perez Martin
Quoted 28 lines · Jason Wang
Quoted 26 lines · Eugenio Perez Martin
Quoted 13 lines · Jason Wang
Quoted 11 lines · Eugenio Perez Martin
Quoted 7 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez
The next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.
If I was not wrong, there's no device specific feature that is checked in the function. So it should be general enough to be used by devices other than net. Then I don't see any advantage of doing this.
Because
vhost_vdpa_init_svqis called at qemu boot, failing if it is not possible to shadow the Virtqueue.Quoted 7 lines
Quoted 2 lines
Now the CVQ will be shadowed if possible, so we need to check this at device start, not at initialization.
Any reason we can't check this at device start? We don't need
driver_featuresand we can do any probing to make sure cvq has an unique group during initialization time.We need the CVQ index to check if it has an independent group. CVQ index depends on the features the guest's ack:
* If it acks_F_MQ, it is the last one.
* If it doesn't, CVQ idx is 2.Quoted 6 lines
Quoted 3 lines
We cannot have acked features at initialization, and they could
change: It is valid for a guest to ack_F_MQ, then reset the device, then not ack it.Can we do some probing by negotiating
_F_MQif the device offers it, then we can know if cvq has a unique group?
Quoted 6 lines
What if the guest does not ack
_F_MQ?To be completed it would go like:
* Probe negotiate
_F_MQ, check unique group,
* Probe negotiate !_F_MQ, check unique group,
I think it should be a bug if device present a unique virtqueue group that depends on a specific feature. That is to say, we can do a single round of probing instead of try it twice here.
Quoted 5 lines
* Actually negotiate with the guest's feature set.
* React to failures. Probably the same way as if the CVQ is not isolated, disabling SVQ?To me it seems simpler to specify somehow that the vq must be independent.
It's just a suggestion, if you think doing it at the start, I'm fine. But we need document the reason with a comment maybe.
Thanks
Quoted 119 lines
Thanks!
Quoted 116 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
To store this information at boot
time is not valid anymore, becausev->shadow_vqs_enabledis not valid at this time anymore.Ok, but this doesn't explain why it is net specific but vhost-vdpa specific.
We can try to move it to a vhost op, but we have the same problem as the svq array allocation: We don't have the right place in vhost ops to check this. Maybe
vhost_set_featuresis the right one here?If we can do all the probing at the initialization phase, we can do everything there.
Thanks
Quoted 102 lines
Thanks!
Quoted 100 lines
Thanks
Quoted 97 lines
Thanks!
Quoted 95 lines
Thanks
Quoted 92 lines
Since the moved checks will be already evaluated at
net/to know if it is ok to shadow CVQ, move them.Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-vdpa.c | 33 ++------------------------------- net/vhost-vdpa.c | 3 ++- 2 files changed, 4 insertions(+), 32 deletions(-)hw/virtio/vhost-vdpa.c +2
-31@@ -402,29 +402,9 @@ static int vhost_vdpa_get_dev_features(struct vhost_dev *dev,return ret; } -static int vhost_vdpa_init_svq (struct vhost_dev *hdev,struct vhost_vdpa *v, -Error **errp) +static void vhost_vdpa_init_svq (struct vhost_dev *hdev,struct vhost_vdpa *v) {g_autoptr (GPtrArray) shadow_vqs =NULL ; -uint64_t dev_features, svq_features; -int r; -bool ok; - -if (!v->shadow_vqs_enabled) { -return 0 ; - } - - r =vhost_vdpa_get_dev_features (hdev, &dev_features); -if (r !=0 ) { -error_setg_errno (errp, -r,"Can't get vdpa device features" ); -return r; - } - - svq_features = dev_features; - ok =vhost_svq_valid_features (svq_features, errp); -if (unlikely (!ok)) { -return -1 ; - } shadow_vqs =g_ptr_array_new_full (hdev->nvqs, vhost_svq_free);for (unsigned n =0 ; n < hdev->nvqs; ++n) {@@ -436,7 +416,6 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v, } v->shadow_vqs =g_steal_pointer (&shadow_vqs); -return 0 ; }static int vhost_vdpa_init (struct vhost_dev *dev,void *opaque,Error **errp)@@ -461,11 +440,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) dev->opaque = opaque ; v->listener = vhost_vdpa_memory_listener; v->msg_type =VHOST_IOTLB_MSG_V2 ; - ret =vhost_vdpa_init_svq (dev, v, errp); -if (ret) { - goto err; - } - +vhost_vdpa_init_svq (dev, v);vhost_vdpa_get_iova_range (v);if (!vhost_vdpa_first_dev (dev)) {@@ -476,10 +451,6 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) VIRTIO_CONFIG_S_DRIVER);return 0 ; - -err: -ram_block_discard_disable (false); -return ret; } static void vhost_vdpa_host_notifier_uninit(struct vhost_dev *dev,net/vhost-vdpa.c +2
-1@@ -117,9 +117,10 @@ static bool vhost_vdpa_net_valid_svq_features(uint64_t features, Error **errp)if (invalid_dev_features) {error_setg (errp,"vdpa svq does not work with features 0x%" PRIx64, invalid_dev_features); +return false; } -return !invalid_dev_features; +return vhost_svq_valid_features (features, errp); }static int vhost_vdpa_net_check_device_id (struct vhost_net *net)
Re: [PATCH v6 05/10] vdpa: move SVQ vring features check to net/
Quoted 42 lines · Jason Wang
Quoted 39 lines · Eugenio Perez Martin
Quoted 28 lines · Jason Wang
Quoted 26 lines · Eugenio Perez Martin
Quoted 13 lines · Jason Wang
Quoted 11 lines · Eugenio Perez Martin
Quoted 7 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez
The next patches will start control SVQ if possible. However, we don't know if that will be possible at qemu boot anymore.
If I was not wrong, there's no device specific feature that is checked in the function. So it should be general enough to be used by devices other than net. Then I don't see any advantage of doing this.
Because
vhost_vdpa_init_svqis called at qemu boot, failing if it is not possible to shadow the Virtqueue.Quoted 7 lines
Quoted 2 lines
Now the CVQ will be shadowed if possible, so we need to check this at device start, not at initialization.
Any reason we can't check this at device start? We don't need
driver_featuresand we can do any probing to make sure cvq has an unique group during initialization time.We need the CVQ index to check if it has an independent group. CVQ index depends on the features the guest's ack:
* If it acks_F_MQ, it is the last one.
* If it doesn't, CVQ idx is 2.Quoted 6 lines
Quoted 3 lines
We cannot have acked features at initialization, and they could
change: It is valid for a guest to ack_F_MQ, then reset the device, then not ack it.Can we do some probing by negotiating
_F_MQif the device offers it, then we can know if cvq has a unique group?What if the guest does not ack
_F_MQ?To be completed it would go like:
Quoted 8 lines
Quoted 2 lines
* Probe negotiate
_F_MQ, check unique group,
* Probe negotiate !_F_MQ, check unique group,I think it should be a bug if device present a unique virtqueue group that depends on a specific feature. That is to say, we can do a single round of probing instead of try it twice here.
I didn't mean a single virtqueue group but CVQ has its own group.
From vhost POV the valid behavior is feature dependent already: If _F_MQ is not negotiated, the vq in a different group must be the 3rd. If it is negotiated, the vq in its own group must be the last one.
Since the check of the virtqueue group is already dependent on the vq index I think it would be a mistake to change a vq group ASID without checking if it is independent or not.
The consequence of not checking that is that:
* We will move all the dataplane vq group to ASID 1, so the device will not work properly.
* The device will be migratable.
Quoted 9 lines
Quoted 6 lines
* Actually negotiate with the guest's feature set.
* React to failures. Probably the same way as if the CVQ is not isolated, disabling SVQ?To me it seems simpler to specify somehow that the vq must be independent.
It's just a suggestion, if you think doing it at the start, I'm fine.
Don't get me wrong, we can make changes to go toward that direction and I think it's a good idea, especially to reduce syscalls. I'm just trying to put all the scenarios on the table because maybe I'm missing something to solve them, or we can ignore them
Thanks!
Quoted 125 lines
But we need document the reason with a comment maybe.
Thanks
Quoted 119 lines
Thanks!
Quoted 116 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
To store this information at boot
time is not valid anymore, becausev->shadow_vqs_enabledis not valid at this time anymore.Ok, but this doesn't explain why it is net specific but vhost-vdpa specific.
We can try to move it to a vhost op, but we have the same problem as the svq array allocation: We don't have the right place in vhost ops to check this. Maybe
vhost_set_featuresis the right one here?If we can do all the probing at the initialization phase, we can do everything there.
Thanks
Quoted 102 lines
Thanks!
Quoted 100 lines
Thanks
Quoted 97 lines
Thanks!
Quoted 95 lines
Thanks
Quoted 92 lines
Since the moved checks will be already evaluated at
net/to know if it is ok to shadow CVQ, move them.Signed-off-by Eugenio Pérez <[email protected]>--- hw/virtio/vhost-vdpa.c | 33 ++------------------------------- net/vhost-vdpa.c | 3 ++- 2 files changed, 4 insertions(+), 32 deletions(-)hw/virtio/vhost-vdpa.c +2
-31@@ -402,29 +402,9 @@ static int vhost_vdpa_get_dev_features(struct vhost_dev *dev,return ret; } -static int vhost_vdpa_init_svq (struct vhost_dev *hdev,struct vhost_vdpa *v, -Error **errp) +static void vhost_vdpa_init_svq (struct vhost_dev *hdev,struct vhost_vdpa *v) {g_autoptr (GPtrArray) shadow_vqs =NULL ; -uint64_t dev_features, svq_features; -int r; -bool ok; - -if (!v->shadow_vqs_enabled) { -return 0 ; - } - - r =vhost_vdpa_get_dev_features (hdev, &dev_features); -if (r !=0 ) { -error_setg_errno (errp, -r,"Can't get vdpa device features" ); -return r; - } - - svq_features = dev_features; - ok =vhost_svq_valid_features (svq_features, errp); -if (unlikely (!ok)) { -return -1 ; - } shadow_vqs =g_ptr_array_new_full (hdev->nvqs, vhost_svq_free);for (unsigned n =0 ; n < hdev->nvqs; ++n) {@@ -436,7 +416,6 @@ static int vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v, } v->shadow_vqs =g_steal_pointer (&shadow_vqs); -return 0 ; }static int vhost_vdpa_init (struct vhost_dev *dev,void *opaque,Error **errp)@@ -461,11 +440,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) dev->opaque = opaque ; v->listener = vhost_vdpa_memory_listener; v->msg_type =VHOST_IOTLB_MSG_V2 ; - ret =vhost_vdpa_init_svq (dev, v, errp); -if (ret) { - goto err; - } - +vhost_vdpa_init_svq (dev, v);vhost_vdpa_get_iova_range (v);if (!vhost_vdpa_first_dev (dev)) {@@ -476,10 +451,6 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp) VIRTIO_CONFIG_S_DRIVER);return 0 ; - -err: -ram_block_discard_disable (false); -return ret; } static void vhost_vdpa_host_notifier_uninit(struct vhost_dev *dev,net/vhost-vdpa.c +2
-1@@ -117,9 +117,10 @@ static bool vhost_vdpa_net_valid_svq_features(uint64_t features, Error **errp)if (invalid_dev_features) {error_setg (errp,"vdpa svq does not work with features 0x%" PRIx64, invalid_dev_features); +return false; } -return !invalid_dev_features; +return vhost_svq_valid_features (features, errp); }static int vhost_vdpa_net_check_device_id (struct vhost_net *net)
PATCH v6 6/10 vdpa: Allocate SVQ unconditionally
In reply to Eugenio PérezSVQ may run or not in a device depending on runtime conditions (for example, if the device can move CVQ to its own group or not).
Allocate the SVQ array unconditionally at startup, since its hard to move this allocation elsewhere.
--- hw/virtio/vhost-vdpa.c | 4 ---- 1 file changed, 4 deletions(-)
hw/virtio/vhost-vdpa.c +0 -4
@@ -547,10 +547,6 @@ static void vhost_vdpa_svq_cleanup(struct vhost_dev *dev)struct vhost_vdpa *v = dev->opaque;size_t idx; -if (!v->shadow_vqs) { -return ; - } -for (idx =0 ; idx < v->shadow_vqs->len; ++idx) {vhost_svq_stop (g_ptr_array_index (v->shadow_vqs, idx)); }
Signature · 1 line
2.31.1
PATCH v6 7/10 vdpa: Add asid parameter to vhost_vdpa_dma_map/unmap
In reply to Eugenio PérezSo the caller can choose which ASID is destined.
No need to update the batch functions as they will always be called from memory listener updates at the moment. Memory listener updates will always update ASID 0, as it's the passthrough ASID.
All vhost devices's ASID are 0 at this moment.
---
v5:
* Solve conflict, now vhost_vdpa_svq_unmap_ring returns void
* Change comment on zero initialization.
v4: Add comment specifying behavior if device does not support _F_ASID
v3: Deleted unneeded space
--- include/hw/virtio/vhost-vdpa.h | 8 +++++--- hw/virtio/vhost-vdpa.c | 29 +++++++++++++++++++---------- net/vhost-vdpa.c | 6 +++--- hw/virtio/trace-events | 4 ++-- 4 files changed, 29 insertions(+), 18 deletions(-)
include/hw/virtio/vhost-vdpa.h +5 -3
@@ -29,6 +29,7 @@ typedef struct vhost_vdpa {int index;uint32_t msg_type;bool iotlb_batch_begin_sent; +uint32_t address_space_id;MemoryListener listener;struct vhost_vdpa_iova_range iova_range;uint64_t acked_features;
@@ -42,8 +43,9 @@ typedef struct vhost_vdpa { VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX]; } VhostVDPA; -int vhost_vdpa_dma_map (struct vhost_vdpa *v,hwaddr iova,hwaddr size, -void *vaddr,bool readonly); -int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,hwaddr iova,hwaddr size); +int vhost_vdpa_dma_map (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size,void *vaddr,bool readonly); +int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size);#endif
hw/virtio/vhost-vdpa.c +19 -10
@@ -72,22 +72,24 @@ static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,return false; } -int vhost_vdpa_dma_map (struct vhost_vdpa *v,hwaddr iova,hwaddr size, -void *vaddr,bool readonly) +int vhost_vdpa_dma_map (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size,void *vaddr,bool readonly) {struct vhost_msg_v2 msg = {};int fd = v->device_fd;int ret =0 ; msg.type = v->msg_type; + msg.asid = asid;/* 0 if vdpa device does not support asid */ msg.iotlb.iova = iova; msg.iotlb.size = size; msg.iotlb.uaddr = (uint64_t )(uintptr_t )vaddr; msg.iotlb.perm = readonly ?VHOST_ACCESS_RO :VHOST_ACCESS_RW ; msg.iotlb.type =VHOST_IOTLB_UPDATE ; -trace_vhost_vdpa_dma_map (v, fd, msg.type, msg.iotlb.iova, msg.iotlb.size, - msg.iotlb.uaddr, msg.iotlb.perm, msg.iotlb.type); +trace_vhost_vdpa_dma_map (v, fd, msg.type, msg.asid, msg.iotlb.iova, + msg.iotlb.size, msg.iotlb.uaddr, msg.iotlb.perm, + msg.iotlb.type);if (write (fd, &msg,sizeof (msg)) !=sizeof (msg)) { error_report("failed to write, fd=%d, errno=%d (%s)",
@@ -98,18 +100,24 @@ int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,return ret; } -int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,hwaddr iova,hwaddr size) +int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size) {struct vhost_msg_v2 msg = {};int fd = v->device_fd;int ret =0 ; msg.type = v->msg_type; +/* +* The caller must set asid = 0 if the device does not support asid. +* This is not an ABI break since it is set to 0 by the initializer anyway. +*/ + msg.asid = asid; msg.iotlb.iova = iova; msg.iotlb.size = size; msg.iotlb.type =VHOST_IOTLB_INVALIDATE ; -trace_vhost_vdpa_dma_unmap (v, fd, msg.type, msg.iotlb.iova, +trace_vhost_vdpa_dma_unmap (v, fd, msg.type, msg.asid, msg.iotlb.iova, msg.iotlb.size, msg.iotlb.type);if (write (fd, &msg,sizeof (msg)) !=sizeof (msg)) {
@@ -229,7 +237,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, }vhost_vdpa_iotlb_batch_begin_once (v); - ret =vhost_vdpa_dma_map (v, iova,int128_get64 (llsize), + ret =vhost_vdpa_dma_map (v,0 , iova,int128_get64 (llsize), vaddr, section->readonly);if (ret) {error_report ("vhost vdpa map fail!" );
@@ -303,7 +311,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,vhost_iova_tree_remove (v->iova_tree, *result); }vhost_vdpa_iotlb_batch_begin_once (v); - ret =vhost_vdpa_dma_unmap (v, iova,int128_get64 (llsize)); + ret =vhost_vdpa_dma_unmap (v,0 , iova,int128_get64 (llsize));if (ret) {error_report ("vhost_vdpa dma unmap error!" ); }
@@ -884,7 +892,7 @@ static void vhost_vdpa_svq_unmap_ring(struct vhost_vdpa *v, hwaddr addr) } size =ROUND_UP (result->size,qemu_real_host_page_size ()); - r =vhost_vdpa_dma_unmap (v, result->iova, size); + r =vhost_vdpa_dma_unmap (v, v->address_space_id, result->iova, size);if (unlikely (r <0 )) {error_report ("Unable to unmap SVQ vring: %s (%d)" ,g_strerror (-r), -r);return ;
@@ -924,7 +932,8 @@ static bool vhost_vdpa_svq_map_ring(struct vhost_vdpa *v, DMAMap *needle,return false; } - r =vhost_vdpa_dma_map (v, needle->iova, needle->size +1 , + r =vhost_vdpa_dma_map (v, v->address_space_id, needle->iova, + needle->size +1 , (void *)(uintptr_t )needle->translated_addr, needle->perm ==IOMMU_RO );if (unlikely (r !=0 )) {
net/vhost-vdpa.c +3 -3
@@ -258,7 +258,7 @@ static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr)return ; } - r =vhost_vdpa_dma_unmap (v, map->iova, map->size +1 ); + r =vhost_vdpa_dma_unmap (v, v->address_space_id, map->iova, map->size +1 );if (unlikely (r !=0 )) {error_report ("Device cannot unmap: %s(%d)" ,g_strerror (r), r); }
@@ -298,8 +298,8 @@ static int vhost_vdpa_cvq_map_buf(struct vhost_vdpa *v, void *buf, size_t size,return r; } - r =vhost_vdpa_dma_map (v, map.iova,vhost_vdpa_net_cvq_cmd_page_len (), buf, - !write); + r =vhost_vdpa_dma_map (v, v->address_space_id, map.iova, +vhost_vdpa_net_cvq_cmd_page_len (), buf, !write);if (unlikely (r <0 )) { goto dma_map_err; }
hw/virtio/trace-events +2 -2
@@ -30,8 +30,8 @@ vhost_user_write(uint32_t req, uint32_t flags) "req:%d flags:0x%"PRIx32"" vhost_user_create_notifier(int idx, void *n) "idx:%d n:%p" # vhost-vdpa.c -vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8 -vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8 +vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8 +vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8 vhost_vdpa_listener_begin_batch(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 vhost_vdpa_listener_commit(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 vhost_vdpa_listener_region_add(void *vdpa, uint64_t iova, uint64_t llend, void *vaddr, bool readonly) "vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64" vaddr: %p read-only: %d"
Signature · 1 line
2.31.1
Re: [PATCH v6 07/10] vdpa: Add asid parameter to vhost_vdpa_dma_map/unmap
Quoted 30 lines · Eugenio Pérez
So the caller can choose which ASID is destined.
No need to update the batch functions as they will always be called from memory listener updates at the moment. Memory listener updates will always update ASID 0, as it's the passthrough ASID.
All vhost devices's ASID are 0 at this moment.
Signed-off-by Eugenio Pérez <[email protected]>---
v5:
* Solve conflict, nowvhost_vdpa_svq_unmap_ringreturns void
* Change comment on zero initialization.v4: Add comment specifying behavior if device does not support
_F_ASIDv3: Deleted unneeded space
--- include/hw/virtio/vhost-vdpa.h | 8 +++++--- hw/virtio/vhost-vdpa.c | 29 +++++++++++++++++++---------- net/vhost-vdpa.c | 6 +++--- hw/virtio/trace-events | 4 ++-- 4 files changed, 29 insertions(+), 18 deletions(-)include/hw/virtio/vhost-vdpa.h +0
-0
include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 32
Quoted 4 lines
include/hw/virtio/vhost-vdpa.h +1
-0@@ -29,6 +29,7 @@ typedef struct vhost_vdpa {int index;uint32_t msg_type;bool iotlb_batch_begin_sent; +uint32_t address_space_id;
So the trick is let device specific code to zero this during allocation?
Quoted 25 lines
include/hw/virtio/vhost-vdpa.h +4
-3MemoryListener listener;struct vhost_vdpa_iova_range iova_range;uint64_t acked_features;@@ -42,8 +43,9 @@ typedef struct vhost_vdpa { VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX]; } VhostVDPA; -int vhost_vdpa_dma_map (struct vhost_vdpa *v,hwaddr iova,hwaddr size, -void *vaddr,bool readonly); -int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,hwaddr iova,hwaddr size); +int vhost_vdpa_dma_map (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size,void *vaddr,bool readonly); +int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size);#endif hw/virtio/vhost-vdpa.c +0
-1@@ -72,22 +72,24 @@ static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,return false; } -int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,
hw/virtio/vhost-vdpa.c static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section, · line 83
Quoted 10 lines
hw/virtio/vhost-vdpa.c +3
-1- void *vaddr, bool readonly) +int vhost_vdpa_dma_map (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size,void *vaddr,bool readonly) {struct vhost_msg_v2 msg = {};int fd = v->device_fd;int ret =0 ; msg.type = v->msg_type; + msg.asid = asid;/* 0 if vdpa device does not support asid */
The comment here is confusing. If this is a requirement, we need either
1) doc this
or
2) perform necessary checks in the function itself.
Quoted 42 lines
hw/virtio/vhost-vdpa.c +11
-4msg.iotlb.iova = iova; msg.iotlb.size = size; msg.iotlb.uaddr = (uint64_t )(uintptr_t )vaddr; msg.iotlb.perm = readonly ?VHOST_ACCESS_RO :VHOST_ACCESS_RW ; msg.iotlb.type =VHOST_IOTLB_UPDATE ; -trace_vhost_vdpa_dma_map (v, fd, msg.type, msg.iotlb.iova, msg.iotlb.size, - msg.iotlb.uaddr, msg.iotlb.perm, msg.iotlb.type); +trace_vhost_vdpa_dma_map (v, fd, msg.type, msg.asid, msg.iotlb.iova, + msg.iotlb.size, msg.iotlb.uaddr, msg.iotlb.perm, + msg.iotlb.type);if (write (fd, &msg,sizeof (msg)) !=sizeof (msg)) { error_report("failed to write, fd=%d, errno=%d (%s)",@@ -98,18 +100,24 @@ int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,return ret; } -int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,hwaddr iova,hwaddr size) +int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size) {struct vhost_msg_v2 msg = {};int fd = v->device_fd;int ret =0 ; msg.type = v->msg_type; +/* +* The caller must set asid = 0 if the device does not support asid. +* This is not an ABI break since it is set to 0 by the initializer anyway. +*/ + msg.asid = asid; msg.iotlb.iova = iova; msg.iotlb.size = size; msg.iotlb.type =VHOST_IOTLB_INVALIDATE ; -trace_vhost_vdpa_dma_unmap (v, fd, msg.type, msg.iotlb.iova, +trace_vhost_vdpa_dma_unmap (v, fd, msg.type, msg.asid, msg.iotlb.iova, msg.iotlb.size, msg.iotlb.type);if (write (fd, &msg,sizeof (msg)) !=sizeof (msg)) {
hw/virtio/vhost-vdpa.c static void vhost_vdpa_listener_region_add(MemoryListener *listener, · line 240
Quoted 5 lines
hw/virtio/vhost-vdpa.c +1
-1@@ -229,7 +237,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, }vhost_vdpa_iotlb_batch_begin_once (v); - ret = vhost_vdpa_dma_map(v, iova, int128_get64(llsize), + ret = vhost_vdpa_dma_map(v, 0, iova, int128_get64(llsize),
Can we use v->address_space_id here? Then we don't need to modify this line when we support multiple asids logic in the future.
Thanks
Quoted 73 lines
hw/virtio/vhost-vdpa.c +4
-3vaddr, section->readonly);if (ret) {error_report ("vhost vdpa map fail!" );@@ -303,7 +311,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,vhost_iova_tree_remove (v->iova_tree, *result); }vhost_vdpa_iotlb_batch_begin_once (v); - ret =vhost_vdpa_dma_unmap (v, iova,int128_get64 (llsize)); + ret =vhost_vdpa_dma_unmap (v,0 , iova,int128_get64 (llsize));if (ret) {error_report ("vhost_vdpa dma unmap error!" ); }@@ -884,7 +892,7 @@ static void vhost_vdpa_svq_unmap_ring(struct vhost_vdpa *v, hwaddr addr) } size =ROUND_UP (result->size,qemu_real_host_page_size ()); - r =vhost_vdpa_dma_unmap (v, result->iova, size); + r =vhost_vdpa_dma_unmap (v, v->address_space_id, result->iova, size);if (unlikely (r <0 )) {error_report ("Unable to unmap SVQ vring: %s (%d)" ,g_strerror (-r), -r);return ;@@ -924,7 +932,8 @@ static bool vhost_vdpa_svq_map_ring(struct vhost_vdpa *v, DMAMap *needle,return false; } - r =vhost_vdpa_dma_map (v, needle->iova, needle->size +1 , + r =vhost_vdpa_dma_map (v, v->address_space_id, needle->iova, + needle->size +1 , (void *)(uintptr_t )needle->translated_addr, needle->perm ==IOMMU_RO );if (unlikely (r !=0 )) {net/vhost-vdpa.c +3
-3@@ -258,7 +258,7 @@ static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr)return ; } - r =vhost_vdpa_dma_unmap (v, map->iova, map->size +1 ); + r =vhost_vdpa_dma_unmap (v, v->address_space_id, map->iova, map->size +1 );if (unlikely (r !=0 )) {error_report ("Device cannot unmap: %s(%d)" ,g_strerror (r), r); }@@ -298,8 +298,8 @@ static int vhost_vdpa_cvq_map_buf(struct vhost_vdpa *v, void *buf, size_t size,return r; } - r =vhost_vdpa_dma_map (v, map.iova,vhost_vdpa_net_cvq_cmd_page_len (), buf, - !write); + r =vhost_vdpa_dma_map (v, v->address_space_id, map.iova, +vhost_vdpa_net_cvq_cmd_page_len (), buf, !write);if (unlikely (r <0 )) { goto dma_map_err; }hw/virtio/trace-events +2
-3@@ -30,8 +30,8 @@ vhost_user_write(uint32_t req, uint32_t flags) "req:%d flags:0x%"PRIx32"" vhost_user_create_notifier(int idx, void *n) "idx:%d n:%p" # vhost-vdpa.c -vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8 -vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8 +vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8 +vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8 vhost_vdpa_listener_begin_batch(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 vhost_vdpa_listener_commit(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 vhost_vdpa_listener_region_add(void *vdpa, uint64_t iova, uint64_t llend, void *vaddr, bool readonly) "vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64" vaddr: %p read-only: %d" --2.31.1
Re: [PATCH v6 07/10] vdpa: Add asid parameter to vhost_vdpa_dma_map/unmap
Quoted 27 lines · Jason Wang
Quoted 25 lines · Eugenio Pérez
So the caller can choose which ASID is destined.
No need to update the batch functions as they will always be called from memory listener updates at the moment. Memory listener updates will always update ASID 0, as it's the passthrough ASID.
All vhost devices's ASID are 0 at this moment.
Signed-off-by Eugenio Pérez <[email protected]>---
v5:
* Solve conflict, nowvhost_vdpa_svq_unmap_ringreturns void
* Change comment on zero initialization.v4: Add comment specifying behavior if device does not support
_F_ASIDv3: Deleted unneeded space
--- include/hw/virtio/vhost-vdpa.h | 8 +++++--- hw/virtio/vhost-vdpa.c | 29 +++++++++++++++++++---------- net/vhost-vdpa.c | 6 +++--- hw/virtio/trace-events | 4 ++-- 4 files changed, 29 insertions(+), 18 deletions(-)
Quoted 12 lines
include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 32
Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1
-0@@ -29,6 +29,7 @@ typedef struct vhost_vdpa {int index;uint32_t msg_type;bool iotlb_batch_begin_sent; +uint32_t address_space_id;So the trick is let device specific code to zero this during allocation?
Yes, but I don't see how that is a trick :). All other parameters also trust it to be 0 at allocation.
Quoted 38 lines
Quoted 25 lines
include/hw/virtio/vhost-vdpa.h +4
-3MemoryListener listener;struct vhost_vdpa_iova_range iova_range;uint64_t acked_features;@@ -42,8 +43,9 @@ typedef struct vhost_vdpa { VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX]; } VhostVDPA; -int vhost_vdpa_dma_map (struct vhost_vdpa *v,hwaddr iova,hwaddr size, -void *vaddr,bool readonly); -int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,hwaddr iova,hwaddr size); +int vhost_vdpa_dma_map (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size,void *vaddr,bool readonly); +int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size);#endif hw/virtio/vhost-vdpa.c +0
-1@@ -72,22 +72,24 @@ static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,return false; } -int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,hw/virtio/vhost-vdpa.c static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section, · line 83
Quoted 10 lines
hw/virtio/vhost-vdpa.c +3
-1- void *vaddr, bool readonly) +int vhost_vdpa_dma_map (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size,void *vaddr,bool readonly) {struct vhost_msg_v2 msg = {};int fd = v->device_fd;int ret =0 ; msg.type = v->msg_type; + msg.asid = asid;/* 0 if vdpa device does not support asid */ The comment here is confusing. If this is a requirement, we need either
Quoted 6 lines
1) doc this
or
2) perform necessary checks in the function itself.
I only documented it in vhost_vdpa_dma_unmap and now I realize it. Would it work to just copy that comment here?
Quoted 44 lines
Quoted 44 lines
hw/virtio/vhost-vdpa.c +11
-4msg.iotlb.iova = iova; msg.iotlb.size = size; msg.iotlb.uaddr = (uint64_t )(uintptr_t )vaddr; msg.iotlb.perm = readonly ?VHOST_ACCESS_RO :VHOST_ACCESS_RW ; msg.iotlb.type =VHOST_IOTLB_UPDATE ; -trace_vhost_vdpa_dma_map (v, fd, msg.type, msg.iotlb.iova, msg.iotlb.size, - msg.iotlb.uaddr, msg.iotlb.perm, msg.iotlb.type); +trace_vhost_vdpa_dma_map (v, fd, msg.type, msg.asid, msg.iotlb.iova, + msg.iotlb.size, msg.iotlb.uaddr, msg.iotlb.perm, + msg.iotlb.type);if (write (fd, &msg,sizeof (msg)) !=sizeof (msg)) { error_report("failed to write, fd=%d, errno=%d (%s)",@@ -98,18 +100,24 @@ int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,return ret; } -int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,hwaddr iova,hwaddr size) +int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size) {struct vhost_msg_v2 msg = {};int fd = v->device_fd;int ret =0 ; msg.type = v->msg_type; +/* +* The caller must set asid = 0 if the device does not support asid. +* This is not an ABI break since it is set to 0 by the initializer anyway. +*/ + msg.asid = asid; msg.iotlb.iova = iova; msg.iotlb.size = size; msg.iotlb.type =VHOST_IOTLB_INVALIDATE ; -trace_vhost_vdpa_dma_unmap (v, fd, msg.type, msg.iotlb.iova, +trace_vhost_vdpa_dma_unmap (v, fd, msg.type, msg.asid, msg.iotlb.iova, msg.iotlb.size, msg.iotlb.type);if (write (fd, &msg,sizeof (msg)) !=sizeof (msg)) {@@ -229,7 +237,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, }
Quoted 7 lines
hw/virtio/vhost-vdpa.c static void vhost_vdpa_listener_region_add(MemoryListener *listener, · line 240
Quoted 3 lines
hw/virtio/vhost-vdpa.c +1
-1vhost_vdpa_iotlb_batch_begin_once (v); - ret = vhost_vdpa_dma_map(v, iova, int128_get64(llsize), + ret = vhost_vdpa_dma_map(v, 0, iova, int128_get64(llsize),Can we use
v->address_space_idhere? Then we don't need to modify this line when we support multiple asids logic in the future.
The registered memory listener is the one of the last vhost_vdpa, the one that handles the last queue.
If all data virtqueues are not shadowed but CVQ is, v->address_space_id is 1 with the current code. But the listener is actually mapping the ASID 0, not 1.
Another alternative is to register it to the last data virtqueue, not the last queue of vhost_vdpa. But it is hard to express it in a generic way at virtio/vhost-vdpa.c . To have a boolean indicating the vhost_vdpa we want to register its memory listener?
It seems easier to me to simply assign 0 at GPA translations. If SVQ is enabled for all queues, then 0 is GPA to qemu's VA + SVQ stuff. If it is not, 0 is always GPA to qemu's VA.
Thanks!
Quoted 76 lines
Thanks
Quoted 73 lines
hw/virtio/vhost-vdpa.c +4
-3vaddr, section->readonly);if (ret) {error_report ("vhost vdpa map fail!" );@@ -303,7 +311,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,vhost_iova_tree_remove (v->iova_tree, *result); }vhost_vdpa_iotlb_batch_begin_once (v); - ret =vhost_vdpa_dma_unmap (v, iova,int128_get64 (llsize)); + ret =vhost_vdpa_dma_unmap (v,0 , iova,int128_get64 (llsize));if (ret) {error_report ("vhost_vdpa dma unmap error!" ); }@@ -884,7 +892,7 @@ static void vhost_vdpa_svq_unmap_ring(struct vhost_vdpa *v, hwaddr addr) } size =ROUND_UP (result->size,qemu_real_host_page_size ()); - r =vhost_vdpa_dma_unmap (v, result->iova, size); + r =vhost_vdpa_dma_unmap (v, v->address_space_id, result->iova, size);if (unlikely (r <0 )) {error_report ("Unable to unmap SVQ vring: %s (%d)" ,g_strerror (-r), -r);return ;@@ -924,7 +932,8 @@ static bool vhost_vdpa_svq_map_ring(struct vhost_vdpa *v, DMAMap *needle,return false; } - r =vhost_vdpa_dma_map (v, needle->iova, needle->size +1 , + r =vhost_vdpa_dma_map (v, v->address_space_id, needle->iova, + needle->size +1 , (void *)(uintptr_t )needle->translated_addr, needle->perm ==IOMMU_RO );if (unlikely (r !=0 )) {net/vhost-vdpa.c +3
-3@@ -258,7 +258,7 @@ static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr)return ; } - r =vhost_vdpa_dma_unmap (v, map->iova, map->size +1 ); + r =vhost_vdpa_dma_unmap (v, v->address_space_id, map->iova, map->size +1 );if (unlikely (r !=0 )) {error_report ("Device cannot unmap: %s(%d)" ,g_strerror (r), r); }@@ -298,8 +298,8 @@ static int vhost_vdpa_cvq_map_buf(struct vhost_vdpa *v, void *buf, size_t size,return r; } - r =vhost_vdpa_dma_map (v, map.iova,vhost_vdpa_net_cvq_cmd_page_len (), buf, - !write); + r =vhost_vdpa_dma_map (v, v->address_space_id, map.iova, +vhost_vdpa_net_cvq_cmd_page_len (), buf, !write);if (unlikely (r <0 )) { goto dma_map_err; }hw/virtio/trace-events +2
-3@@ -30,8 +30,8 @@ vhost_user_write(uint32_t req, uint32_t flags) "req:%d flags:0x%"PRIx32"" vhost_user_create_notifier(int idx, void *n) "idx:%d n:%p" # vhost-vdpa.c -vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8 -vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8 +vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8 +vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8 vhost_vdpa_listener_begin_batch(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 vhost_vdpa_listener_commit(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 vhost_vdpa_listener_region_add(void *vdpa, uint64_t iova, uint64_t llend, void *vaddr, bool readonly) "vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64" vaddr: %p read-only: %d" --2.31.1
Re: [PATCH v6 07/10] vdpa: Add asid parameter to vhost_vdpa_dma_map/unmap
Quoted 77 lines · Eugenio Perez Martin
Quoted 25 lines · Jason Wang
Quoted 24 lines · Eugenio Pérez
So the caller can choose which ASID is destined.
No need to update the batch functions as they will always be called from memory listener updates at the moment. Memory listener updates will always update ASID 0, as it's the passthrough ASID.
All vhost devices's ASID are 0 at this moment.
Signed-off-by Eugenio Pérez <[email protected]>---
v5:
* Solve conflict, nowvhost_vdpa_svq_unmap_ringreturns void
* Change comment on zero initialization.v4: Add comment specifying behavior if device does not support
_F_ASIDv3: Deleted unneeded space
--- include/hw/virtio/vhost-vdpa.h | 8 +++++--- hw/virtio/vhost-vdpa.c | 29 +++++++++++++++++++---------- net/vhost-vdpa.c | 6 +++--- hw/virtio/trace-events | 4 ++-- 4 files changed, 29 insertions(+), 18 deletions(-)Quoted 11 lines
include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 32
Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1
-0@@ -29,6 +29,7 @@ typedef struct vhost_vdpa {int index;uint32_t msg_type;bool iotlb_batch_begin_sent; +uint32_t address_space_id;So the trick is let device specific code to zero this during allocation?
Yes, but I don't see how that is a trick :). All other parameters also trust it to be 0 at allocation.
Quoted 37 lines
Quoted 25 lines
include/hw/virtio/vhost-vdpa.h +4
-3MemoryListener listener;struct vhost_vdpa_iova_range iova_range;uint64_t acked_features;@@ -42,8 +43,9 @@ typedef struct vhost_vdpa { VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX]; } VhostVDPA; -int vhost_vdpa_dma_map (struct vhost_vdpa *v,hwaddr iova,hwaddr size, -void *vaddr,bool readonly); -int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,hwaddr iova,hwaddr size); +int vhost_vdpa_dma_map (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size,void *vaddr,bool readonly); +int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size);#endif hw/virtio/vhost-vdpa.c +0
-1@@ -72,22 +72,24 @@ static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,return false; } -int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,hw/virtio/vhost-vdpa.c static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section, · line 83
Quoted 10 lines
hw/virtio/vhost-vdpa.c +3
-1- void *vaddr, bool readonly) +int vhost_vdpa_dma_map (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size,void *vaddr,bool readonly) {struct vhost_msg_v2 msg = {};int fd = v->device_fd;int ret =0 ; msg.type = v->msg_type; + msg.asid = asid;/* 0 if vdpa device does not support asid */ The comment here is confusing. If this is a requirement, we need either
Quoted 8 lines
Quoted 6 lines
1) doc this
or
2) perform necessary checks in the function itself.
I only documented it in
vhost_vdpa_dma_unmapand now I realize it. Would it work to just copy that comment here?
Probably, and let's move the comment above the function definition.
Quoted 51 lines
Quoted 45 lines
Quoted 45 lines
hw/virtio/vhost-vdpa.c +11
-4msg.iotlb.iova = iova; msg.iotlb.size = size; msg.iotlb.uaddr = (uint64_t )(uintptr_t )vaddr; msg.iotlb.perm = readonly ?VHOST_ACCESS_RO :VHOST_ACCESS_RW ; msg.iotlb.type =VHOST_IOTLB_UPDATE ; -trace_vhost_vdpa_dma_map (v, fd, msg.type, msg.iotlb.iova, msg.iotlb.size, - msg.iotlb.uaddr, msg.iotlb.perm, msg.iotlb.type); +trace_vhost_vdpa_dma_map (v, fd, msg.type, msg.asid, msg.iotlb.iova, + msg.iotlb.size, msg.iotlb.uaddr, msg.iotlb.perm, + msg.iotlb.type);if (write (fd, &msg,sizeof (msg)) !=sizeof (msg)) { error_report("failed to write, fd=%d, errno=%d (%s)",@@ -98,18 +100,24 @@ int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,return ret; } -int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,hwaddr iova,hwaddr size) +int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size) {struct vhost_msg_v2 msg = {};int fd = v->device_fd;int ret =0 ; msg.type = v->msg_type; +/* +* The caller must set asid = 0 if the device does not support asid. +* This is not an ABI break since it is set to 0 by the initializer anyway. +*/ + msg.asid = asid; msg.iotlb.iova = iova; msg.iotlb.size = size; msg.iotlb.type =VHOST_IOTLB_INVALIDATE ; -trace_vhost_vdpa_dma_unmap (v, fd, msg.type, msg.iotlb.iova, +trace_vhost_vdpa_dma_unmap (v, fd, msg.type, msg.asid, msg.iotlb.iova, msg.iotlb.size, msg.iotlb.type);if (write (fd, &msg,sizeof (msg)) !=sizeof (msg)) {@@ -229,7 +237,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, }Quoted 6 lines
hw/virtio/vhost-vdpa.c static void vhost_vdpa_listener_region_add(MemoryListener *listener, · line 240
Quoted 3 lines
hw/virtio/vhost-vdpa.c +1
-1vhost_vdpa_iotlb_batch_begin_once (v); - ret = vhost_vdpa_dma_map(v, iova, int128_get64(llsize), + ret = vhost_vdpa_dma_map(v, 0, iova, int128_get64(llsize),Can we use
v->address_space_idhere? Then we don't need to modify this line when we support multiple asids logic in the future.
Quoted 5 lines
The registered memory listener is the one of the last
vhost_vdpa, the one that handles the last queue.If all data virtqueues are not shadowed but CVQ is,
v->address_space_idis 1 with the current code.
Ok, right. So let's add a comment here. It would be even better to define the macro for data vq asid in this patch.
Thanks
Quoted 89 lines
But the listener is
actually mapping the ASID 0, not 1.Another alternative is to register it to the last data virtqueue, not the last queue of
vhost_vdpa. But it is hard to express it in a generic way atvirtio/vhost-vdpa.c. To have a boolean indicating thevhost_vdpawe want to register its memory listener?It seems easier to me to simply assign 0 at GPA translations. If SVQ is enabled for all queues, then 0 is GPA to qemu's VA + SVQ stuff. If it is not, 0 is always GPA to qemu's VA.
Thanks!
Quoted 75 lines
Thanks
Quoted 73 lines
hw/virtio/vhost-vdpa.c +4
-3vaddr, section->readonly);if (ret) {error_report ("vhost vdpa map fail!" );@@ -303,7 +311,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,vhost_iova_tree_remove (v->iova_tree, *result); }vhost_vdpa_iotlb_batch_begin_once (v); - ret =vhost_vdpa_dma_unmap (v, iova,int128_get64 (llsize)); + ret =vhost_vdpa_dma_unmap (v,0 , iova,int128_get64 (llsize));if (ret) {error_report ("vhost_vdpa dma unmap error!" ); }@@ -884,7 +892,7 @@ static void vhost_vdpa_svq_unmap_ring(struct vhost_vdpa *v, hwaddr addr) } size =ROUND_UP (result->size,qemu_real_host_page_size ()); - r =vhost_vdpa_dma_unmap (v, result->iova, size); + r =vhost_vdpa_dma_unmap (v, v->address_space_id, result->iova, size);if (unlikely (r <0 )) {error_report ("Unable to unmap SVQ vring: %s (%d)" ,g_strerror (-r), -r);return ;@@ -924,7 +932,8 @@ static bool vhost_vdpa_svq_map_ring(struct vhost_vdpa *v, DMAMap *needle,return false; } - r =vhost_vdpa_dma_map (v, needle->iova, needle->size +1 , + r =vhost_vdpa_dma_map (v, v->address_space_id, needle->iova, + needle->size +1 , (void *)(uintptr_t )needle->translated_addr, needle->perm ==IOMMU_RO );if (unlikely (r !=0 )) {net/vhost-vdpa.c +3
-3@@ -258,7 +258,7 @@ static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr)return ; } - r =vhost_vdpa_dma_unmap (v, map->iova, map->size +1 ); + r =vhost_vdpa_dma_unmap (v, v->address_space_id, map->iova, map->size +1 );if (unlikely (r !=0 )) {error_report ("Device cannot unmap: %s(%d)" ,g_strerror (r), r); }@@ -298,8 +298,8 @@ static int vhost_vdpa_cvq_map_buf(struct vhost_vdpa *v, void *buf, size_t size,return r; } - r =vhost_vdpa_dma_map (v, map.iova,vhost_vdpa_net_cvq_cmd_page_len (), buf, - !write); + r =vhost_vdpa_dma_map (v, v->address_space_id, map.iova, +vhost_vdpa_net_cvq_cmd_page_len (), buf, !write);if (unlikely (r <0 )) { goto dma_map_err; }hw/virtio/trace-events +2
-3@@ -30,8 +30,8 @@ vhost_user_write(uint32_t req, uint32_t flags) "req:%d flags:0x%"PRIx32"" vhost_user_create_notifier(int idx, void *n) "idx:%d n:%p" # vhost-vdpa.c -vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8 -vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8 +vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8 +vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8 vhost_vdpa_listener_begin_batch(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 vhost_vdpa_listener_commit(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 vhost_vdpa_listener_region_add(void *vdpa, uint64_t iova, uint64_t llend, void *vaddr, bool readonly) "vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64" vaddr: %p read-only: %d" --2.31.1
Re: [PATCH v6 07/10] vdpa: Add asid parameter to vhost_vdpa_dma_map/unmap
Quoted 147 lines · Jason Wang
Quoted 77 lines · Eugenio Perez Martin
Quoted 25 lines · Jason Wang
Quoted 24 lines · Eugenio Pérez
So the caller can choose which ASID is destined.
No need to update the batch functions as they will always be called from memory listener updates at the moment. Memory listener updates will always update ASID 0, as it's the passthrough ASID.
All vhost devices's ASID are 0 at this moment.
Signed-off-by Eugenio Pérez <[email protected]>---
v5:
* Solve conflict, nowvhost_vdpa_svq_unmap_ringreturns void
* Change comment on zero initialization.v4: Add comment specifying behavior if device does not support
_F_ASIDv3: Deleted unneeded space
--- include/hw/virtio/vhost-vdpa.h | 8 +++++--- hw/virtio/vhost-vdpa.c | 29 +++++++++++++++++++---------- net/vhost-vdpa.c | 6 +++--- hw/virtio/trace-events | 4 ++-- 4 files changed, 29 insertions(+), 18 deletions(-)Quoted 11 lines
include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 32
Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1
-0@@ -29,6 +29,7 @@ typedef struct vhost_vdpa {int index;uint32_t msg_type;bool iotlb_batch_begin_sent; +uint32_t address_space_id;So the trick is let device specific code to zero this during allocation?
Yes, but I don't see how that is a trick :). All other parameters also trust it to be 0 at allocation.
Quoted 37 lines
Quoted 25 lines
include/hw/virtio/vhost-vdpa.h +4
-3MemoryListener listener;struct vhost_vdpa_iova_range iova_range;uint64_t acked_features;@@ -42,8 +43,9 @@ typedef struct vhost_vdpa { VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX]; } VhostVDPA; -int vhost_vdpa_dma_map (struct vhost_vdpa *v,hwaddr iova,hwaddr size, -void *vaddr,bool readonly); -int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,hwaddr iova,hwaddr size); +int vhost_vdpa_dma_map (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size,void *vaddr,bool readonly); +int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size);#endif hw/virtio/vhost-vdpa.c +0
-1@@ -72,22 +72,24 @@ static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,return false; } -int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,hw/virtio/vhost-vdpa.c static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section, · line 83
Quoted 10 lines
hw/virtio/vhost-vdpa.c +3
-1- void *vaddr, bool readonly) +int vhost_vdpa_dma_map (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size,void *vaddr,bool readonly) {struct vhost_msg_v2 msg = {};int fd = v->device_fd;int ret =0 ; msg.type = v->msg_type; + msg.asid = asid;/* 0 if vdpa device does not support asid */ The comment here is confusing. If this is a requirement, we need either
Quoted 8 lines
Quoted 6 lines
1) doc this
or
2) perform necessary checks in the function itself.
I only documented it in
vhost_vdpa_dma_unmapand now I realize it. Would it work to just copy that comment here?Probably, and let's move the comment above the function definition.
Quoted 54 lines
Quoted 45 lines
Quoted 45 lines
hw/virtio/vhost-vdpa.c +11
-4msg.iotlb.iova = iova; msg.iotlb.size = size; msg.iotlb.uaddr = (uint64_t )(uintptr_t )vaddr; msg.iotlb.perm = readonly ?VHOST_ACCESS_RO :VHOST_ACCESS_RW ; msg.iotlb.type =VHOST_IOTLB_UPDATE ; -trace_vhost_vdpa_dma_map (v, fd, msg.type, msg.iotlb.iova, msg.iotlb.size, - msg.iotlb.uaddr, msg.iotlb.perm, msg.iotlb.type); +trace_vhost_vdpa_dma_map (v, fd, msg.type, msg.asid, msg.iotlb.iova, + msg.iotlb.size, msg.iotlb.uaddr, msg.iotlb.perm, + msg.iotlb.type);if (write (fd, &msg,sizeof (msg)) !=sizeof (msg)) { error_report("failed to write, fd=%d, errno=%d (%s)",@@ -98,18 +100,24 @@ int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,return ret; } -int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,hwaddr iova,hwaddr size) +int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size) {struct vhost_msg_v2 msg = {};int fd = v->device_fd;int ret =0 ; msg.type = v->msg_type; +/* +* The caller must set asid = 0 if the device does not support asid. +* This is not an ABI break since it is set to 0 by the initializer anyway. +*/ + msg.asid = asid; msg.iotlb.iova = iova; msg.iotlb.size = size; msg.iotlb.type =VHOST_IOTLB_INVALIDATE ; -trace_vhost_vdpa_dma_unmap (v, fd, msg.type, msg.iotlb.iova, +trace_vhost_vdpa_dma_unmap (v, fd, msg.type, msg.asid, msg.iotlb.iova, msg.iotlb.size, msg.iotlb.type);if (write (fd, &msg,sizeof (msg)) !=sizeof (msg)) {@@ -229,7 +237,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, }Quoted 6 lines
hw/virtio/vhost-vdpa.c static void vhost_vdpa_listener_region_add(MemoryListener *listener, · line 240
Quoted 3 lines
hw/virtio/vhost-vdpa.c +1
-1vhost_vdpa_iotlb_batch_begin_once (v); - ret = vhost_vdpa_dma_map(v, iova, int128_get64(llsize), + ret = vhost_vdpa_dma_map(v, 0, iova, int128_get64(llsize),Can we use
v->address_space_idhere? Then we don't need to modify this line when we support multiple asids logic in the future.The registered memory listener is the one of the last
vhost_vdpa, the one that handles the last queue.
Quoted 7 lines
Quoted 2 lines
If all data virtqueues are not shadowed but CVQ is,
v->address_space_idis 1 with the current code.Ok, right. So let's add a comment here. It would be even better to define the macro for data vq asid in this patch.
I agree that it must be changed to a macro.
However, currently the _ASID macros belong to net/ . Maybe to declare VHOST_VDPA_GPA_ASID in include/hw/virtio/vhost-vdpa.h and just let VHOST_VDPA_NET_CVQ_ASID in net/vhost-vdpa.c?
Thanks!
Quoted 94 lines
Thanks
Quoted 89 lines
But the listener is
actually mapping the ASID 0, not 1.Another alternative is to register it to the last data virtqueue, not the last queue of
vhost_vdpa. But it is hard to express it in a generic way atvirtio/vhost-vdpa.c. To have a boolean indicating thevhost_vdpawe want to register its memory listener?It seems easier to me to simply assign 0 at GPA translations. If SVQ is enabled for all queues, then 0 is GPA to qemu's VA + SVQ stuff. If it is not, 0 is always GPA to qemu's VA.
Thanks!
Quoted 75 lines
Thanks
Quoted 73 lines
hw/virtio/vhost-vdpa.c +4
-3vaddr, section->readonly);if (ret) {error_report ("vhost vdpa map fail!" );@@ -303,7 +311,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,vhost_iova_tree_remove (v->iova_tree, *result); }vhost_vdpa_iotlb_batch_begin_once (v); - ret =vhost_vdpa_dma_unmap (v, iova,int128_get64 (llsize)); + ret =vhost_vdpa_dma_unmap (v,0 , iova,int128_get64 (llsize));if (ret) {error_report ("vhost_vdpa dma unmap error!" ); }@@ -884,7 +892,7 @@ static void vhost_vdpa_svq_unmap_ring(struct vhost_vdpa *v, hwaddr addr) } size =ROUND_UP (result->size,qemu_real_host_page_size ()); - r =vhost_vdpa_dma_unmap (v, result->iova, size); + r =vhost_vdpa_dma_unmap (v, v->address_space_id, result->iova, size);if (unlikely (r <0 )) {error_report ("Unable to unmap SVQ vring: %s (%d)" ,g_strerror (-r), -r);return ;@@ -924,7 +932,8 @@ static bool vhost_vdpa_svq_map_ring(struct vhost_vdpa *v, DMAMap *needle,return false; } - r =vhost_vdpa_dma_map (v, needle->iova, needle->size +1 , + r =vhost_vdpa_dma_map (v, v->address_space_id, needle->iova, + needle->size +1 , (void *)(uintptr_t )needle->translated_addr, needle->perm ==IOMMU_RO );if (unlikely (r !=0 )) {net/vhost-vdpa.c +3
-3@@ -258,7 +258,7 @@ static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr)return ; } - r =vhost_vdpa_dma_unmap (v, map->iova, map->size +1 ); + r =vhost_vdpa_dma_unmap (v, v->address_space_id, map->iova, map->size +1 );if (unlikely (r !=0 )) {error_report ("Device cannot unmap: %s(%d)" ,g_strerror (r), r); }@@ -298,8 +298,8 @@ static int vhost_vdpa_cvq_map_buf(struct vhost_vdpa *v, void *buf, size_t size,return r; } - r =vhost_vdpa_dma_map (v, map.iova,vhost_vdpa_net_cvq_cmd_page_len (), buf, - !write); + r =vhost_vdpa_dma_map (v, v->address_space_id, map.iova, +vhost_vdpa_net_cvq_cmd_page_len (), buf, !write);if (unlikely (r <0 )) { goto dma_map_err; }hw/virtio/trace-events +2
-3@@ -30,8 +30,8 @@ vhost_user_write(uint32_t req, uint32_t flags) "req:%d flags:0x%"PRIx32"" vhost_user_create_notifier(int idx, void *n) "idx:%d n:%p" # vhost-vdpa.c -vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8 -vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8 +vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8 +vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8 vhost_vdpa_listener_begin_batch(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 vhost_vdpa_listener_commit(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 vhost_vdpa_listener_region_add(void *vdpa, uint64_t iova, uint64_t llend, void *vaddr, bool readonly) "vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64" vaddr: %p read-only: %d" --2.31.1
Re: [PATCH v6 07/10] vdpa: Add asid parameter to vhost_vdpa_dma_map/unmap
Quoted 151 lines · Eugenio Perez Martin
Quoted 141 lines · Jason Wang
Quoted 77 lines · Eugenio Perez Martin
Quoted 25 lines · Jason Wang
Quoted 24 lines · Eugenio Pérez
So the caller can choose which ASID is destined.
No need to update the batch functions as they will always be called from memory listener updates at the moment. Memory listener updates will always update ASID 0, as it's the passthrough ASID.
All vhost devices's ASID are 0 at this moment.
Signed-off-by Eugenio Pérez <[email protected]>---
v5:
* Solve conflict, nowvhost_vdpa_svq_unmap_ringreturns void
* Change comment on zero initialization.v4: Add comment specifying behavior if device does not support
_F_ASIDv3: Deleted unneeded space
--- include/hw/virtio/vhost-vdpa.h | 8 +++++--- hw/virtio/vhost-vdpa.c | 29 +++++++++++++++++++---------- net/vhost-vdpa.c | 6 +++--- hw/virtio/trace-events | 4 ++-- 4 files changed, 29 insertions(+), 18 deletions(-)Quoted 11 lines
include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 32
Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1
-0@@ -29,6 +29,7 @@ typedef struct vhost_vdpa {int index;uint32_t msg_type;bool iotlb_batch_begin_sent; +uint32_t address_space_id;So the trick is let device specific code to zero this during allocation?
Yes, but I don't see how that is a trick :). All other parameters also trust it to be 0 at allocation.
Quoted 37 lines
Quoted 25 lines
include/hw/virtio/vhost-vdpa.h +4
-3MemoryListener listener;struct vhost_vdpa_iova_range iova_range;uint64_t acked_features;@@ -42,8 +43,9 @@ typedef struct vhost_vdpa { VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX]; } VhostVDPA; -int vhost_vdpa_dma_map (struct vhost_vdpa *v,hwaddr iova,hwaddr size, -void *vaddr,bool readonly); -int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,hwaddr iova,hwaddr size); +int vhost_vdpa_dma_map (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size,void *vaddr,bool readonly); +int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size);#endif hw/virtio/vhost-vdpa.c +0
-1@@ -72,22 +72,24 @@ static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,return false; } -int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,hw/virtio/vhost-vdpa.c static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section, · line 83
Quoted 10 lines
hw/virtio/vhost-vdpa.c +3
-1- void *vaddr, bool readonly) +int vhost_vdpa_dma_map (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size,void *vaddr,bool readonly) {struct vhost_msg_v2 msg = {};int fd = v->device_fd;int ret =0 ; msg.type = v->msg_type; + msg.asid = asid;/* 0 if vdpa device does not support asid */ The comment here is confusing. If this is a requirement, we need either
Quoted 8 lines
Quoted 6 lines
1) doc this
or
2) perform necessary checks in the function itself.
I only documented it in
vhost_vdpa_dma_unmapand now I realize it. Would it work to just copy that comment here?Probably, and let's move the comment above the function definition.
Quoted 50 lines
Quoted 50 lines
Quoted 42 lines
hw/virtio/vhost-vdpa.c +11
-4msg.iotlb.iova = iova; msg.iotlb.size = size; msg.iotlb.uaddr = (uint64_t )(uintptr_t )vaddr; msg.iotlb.perm = readonly ?VHOST_ACCESS_RO :VHOST_ACCESS_RW ; msg.iotlb.type =VHOST_IOTLB_UPDATE ; -trace_vhost_vdpa_dma_map (v, fd, msg.type, msg.iotlb.iova, msg.iotlb.size, - msg.iotlb.uaddr, msg.iotlb.perm, msg.iotlb.type); +trace_vhost_vdpa_dma_map (v, fd, msg.type, msg.asid, msg.iotlb.iova, + msg.iotlb.size, msg.iotlb.uaddr, msg.iotlb.perm, + msg.iotlb.type);if (write (fd, &msg,sizeof (msg)) !=sizeof (msg)) { error_report("failed to write, fd=%d, errno=%d (%s)",@@ -98,18 +100,24 @@ int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,return ret; } -int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,hwaddr iova,hwaddr size) +int vhost_vdpa_dma_unmap (struct vhost_vdpa *v,uint32_t asid,hwaddr iova, +hwaddr size) {struct vhost_msg_v2 msg = {};int fd = v->device_fd;int ret =0 ; msg.type = v->msg_type; +/* +* The caller must set asid = 0 if the device does not support asid. +* This is not an ABI break since it is set to 0 by the initializer anyway. +*/ + msg.asid = asid; msg.iotlb.iova = iova; msg.iotlb.size = size; msg.iotlb.type =VHOST_IOTLB_INVALIDATE ; -trace_vhost_vdpa_dma_unmap (v, fd, msg.type, msg.iotlb.iova, +trace_vhost_vdpa_dma_unmap (v, fd, msg.type, msg.asid, msg.iotlb.iova, msg.iotlb.size, msg.iotlb.type);if (write (fd, &msg,sizeof (msg)) !=sizeof (msg)) {hw/virtio/vhost-vdpa.c static void vhost_vdpa_listener_region_add(MemoryListener *listener, · line 240
Quoted 5 lines
hw/virtio/vhost-vdpa.c +1
-1@@ -229,7 +237,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, }vhost_vdpa_iotlb_batch_begin_once (v); - ret = vhost_vdpa_dma_map(v, iova, int128_get64(llsize), + ret = vhost_vdpa_dma_map(v, 0, iova, int128_get64(llsize),Can we use
v->address_space_idhere? Then we don't need to modify this line when we support multiple asids logic in the future.Quoted 9 lines
Quoted 5 lines
The registered memory listener is the one of the last
vhost_vdpa, the one that handles the last queue.If all data virtqueues are not shadowed but CVQ is,
v->address_space_idis 1 with the current code.Ok, right. So let's add a comment here. It would be even better to define the macro for data vq asid in this patch.
Quoted 7 lines
I agree that it must be changed to a macro.
However, currently the
_ASIDmacros belong tonet/. Maybe to declareVHOST_VDPA_GPA_ASIDininclude/hw/virtio/vhost-vdpa.hand just letVHOST_VDPA_NET_CVQ_ASIDinnet/vhost-vdpa.c?Thanks!
That should be fine.
Thanks
Quoted 93 lines
Quoted 93 lines
Thanks
Quoted 89 lines
But the listener is
actually mapping the ASID 0, not 1.Another alternative is to register it to the last data virtqueue, not the last queue of
vhost_vdpa. But it is hard to express it in a generic way atvirtio/vhost-vdpa.c. To have a boolean indicating thevhost_vdpawe want to register its memory listener?It seems easier to me to simply assign 0 at GPA translations. If SVQ is enabled for all queues, then 0 is GPA to qemu's VA + SVQ stuff. If it is not, 0 is always GPA to qemu's VA.
Thanks!
Quoted 75 lines
Thanks
Quoted 73 lines
hw/virtio/vhost-vdpa.c +4
-3vaddr, section->readonly);if (ret) {error_report ("vhost vdpa map fail!" );@@ -303,7 +311,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,vhost_iova_tree_remove (v->iova_tree, *result); }vhost_vdpa_iotlb_batch_begin_once (v); - ret =vhost_vdpa_dma_unmap (v, iova,int128_get64 (llsize)); + ret =vhost_vdpa_dma_unmap (v,0 , iova,int128_get64 (llsize));if (ret) {error_report ("vhost_vdpa dma unmap error!" ); }@@ -884,7 +892,7 @@ static void vhost_vdpa_svq_unmap_ring(struct vhost_vdpa *v, hwaddr addr) } size =ROUND_UP (result->size,qemu_real_host_page_size ()); - r =vhost_vdpa_dma_unmap (v, result->iova, size); + r =vhost_vdpa_dma_unmap (v, v->address_space_id, result->iova, size);if (unlikely (r <0 )) {error_report ("Unable to unmap SVQ vring: %s (%d)" ,g_strerror (-r), -r);return ;@@ -924,7 +932,8 @@ static bool vhost_vdpa_svq_map_ring(struct vhost_vdpa *v, DMAMap *needle,return false; } - r =vhost_vdpa_dma_map (v, needle->iova, needle->size +1 , + r =vhost_vdpa_dma_map (v, v->address_space_id, needle->iova, + needle->size +1 , (void *)(uintptr_t )needle->translated_addr, needle->perm ==IOMMU_RO );if (unlikely (r !=0 )) {net/vhost-vdpa.c +3
-3@@ -258,7 +258,7 @@ static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr)return ; } - r =vhost_vdpa_dma_unmap (v, map->iova, map->size +1 ); + r =vhost_vdpa_dma_unmap (v, v->address_space_id, map->iova, map->size +1 );if (unlikely (r !=0 )) {error_report ("Device cannot unmap: %s(%d)" ,g_strerror (r), r); }@@ -298,8 +298,8 @@ static int vhost_vdpa_cvq_map_buf(struct vhost_vdpa *v, void *buf, size_t size,return r; } - r =vhost_vdpa_dma_map (v, map.iova,vhost_vdpa_net_cvq_cmd_page_len (), buf, - !write); + r =vhost_vdpa_dma_map (v, v->address_space_id, map.iova, +vhost_vdpa_net_cvq_cmd_page_len (), buf, !write);if (unlikely (r <0 )) { goto dma_map_err; }hw/virtio/trace-events +2
-3@@ -30,8 +30,8 @@ vhost_user_write(uint32_t req, uint32_t flags) "req:%d flags:0x%"PRIx32"" vhost_user_create_notifier(int idx, void *n) "idx:%d n:%p" # vhost-vdpa.c -vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8 -vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8 +vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8 +vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8 vhost_vdpa_listener_begin_batch(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 vhost_vdpa_listener_commit(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 vhost_vdpa_listener_region_add(void *vdpa, uint64_t iova, uint64_t llend, void *vaddr, bool readonly) "vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64" vaddr: %p read-only: %d" --2.31.1
PATCH v6 8/10 vdpa: Store x-svq parameter in VhostVDPAState
In reply to Eugenio PérezCVQ can be shadowed two ways:
- Device has x-svq=on parameter (current way)
- The device can isolate CVQ in its own vq group
QEMU needs to check for the second condition dynamically, because CVQ index is not known at initialization time. Since this is dynamic, the CVQ isolation could vary with different conditions, making it possible to go from "not isolated group" to "isolated".
Saving the cmdline parameter in an extra field so we never disable CVQ SVQ in case the device was started with cmdline.
--- net/vhost-vdpa.c | 3 +++ 1 file changed, 3 insertions(+)
net/vhost-vdpa.c +3 -0
@@ -38,6 +38,8 @@ typedef struct VhostVDPAState {void *cvq_cmd_out_buffer;virtio_net_ctrl_ack *status; +/* The device always have SVQ enabled */ +bool always_svq;bool started; } VhostVDPAState;
@@ -566,6 +568,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, s->vhost_vdpa.device_fd = vdpa_device_fd; s->vhost_vdpa.index = queue_pair_index; + s->always_svq = svq; s->vhost_vdpa.shadow_vqs_enabled = svq; s->vhost_vdpa.iova_tree = iova_tree;if (!is_datapath) {
Signature · 1 line
2.31.1
PATCH v6 9/10 vdpa: Add listener_shadow_vq to vhost_vdpa
In reply to Eugenio PérezThe memory listener that thells the device how to convert GPA to qemu's va is registered against CVQ vhost_vdpa. This series try to map the memory listener translations to ASID 0, while it maps the CVQ ones to ASID 1.
Let's tell the listener if it needs to register them on iova tree or not.
---
v5: Solve conflict about vhost_iova_tree_remove accepting mem_region by
value.
--- include/hw/virtio/vhost-vdpa.h | 2 ++ hw/virtio/vhost-vdpa.c | 6 +++--- net/vhost-vdpa.c | 1 + 3 files changed, 6 insertions(+), 3 deletions(-)
include/hw/virtio/vhost-vdpa.h +2 -0
@@ -34,6 +34,8 @@ typedef struct vhost_vdpa {struct vhost_vdpa_iova_range iova_range;uint64_t acked_features;bool shadow_vqs_enabled; +/* The listener must send iova tree addresses, not GPA */ +bool listener_shadow_vq;/* IOVA mapping used by the Shadow Virtqueue */ VhostIOVATree *iova_tree;GPtrArray *shadow_vqs;
hw/virtio/vhost-vdpa.c +3 -3
@@ -220,7 +220,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, vaddr, section->readonly); llsize =int128_sub (llend,int128_make64 (iova)); -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {int r; mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,
@@ -247,7 +247,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,return ; fail_map: -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {vhost_iova_tree_remove (v->iova_tree, mem_region); }
@@ -292,7 +292,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener, llsize =int128_sub (llend,int128_make64 (iova)); -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {const DMAMap *result;const void *vaddr =memory_region_get_ram_ptr (section->mr) + section->offset_within_region +
net/vhost-vdpa.c +1 -0
@@ -570,6 +570,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, s->vhost_vdpa.index = queue_pair_index; s->always_svq = svq; s->vhost_vdpa.shadow_vqs_enabled = svq; + s->vhost_vdpa.listener_shadow_vq = svq; s->vhost_vdpa.iova_tree = iova_tree;if (!is_datapath) { s->cvq_cmd_out_buffer = qemu_memalign(qemu_real_host_page_size(),
Signature · 1 line
2.31.1
Re: [PATCH v6 09/10] vdpa: Add listener_shadow_vq to vhost_vdpa
Quoted 68 lines · Eugenio Pérez
The memory listener that thells the device how to convert GPA to qemu's va is registered against CVQ
vhost_vdpa. This series try to map the memory listener translations to ASID 0, while it maps the CVQ ones to ASID 1.Let's tell the listener if it needs to register them on iova tree or not.
Signed-off-by Eugenio Pérez <[email protected]>---
v5: Solve conflict aboutvhost_iova_tree_removeacceptingmem_regionby
value.--- include/hw/virtio/vhost-vdpa.h | 2 ++ hw/virtio/vhost-vdpa.c | 6 +++--- net/vhost-vdpa.c | 1 + 3 files changed, 6 insertions(+), 3 deletions(-)include/hw/virtio/vhost-vdpa.h +2
-0@@ -34,6 +34,8 @@ typedef struct vhost_vdpa {struct vhost_vdpa_iova_range iova_range;uint64_t acked_features;bool shadow_vqs_enabled; +/* The listener must send iova tree addresses, not GPA */ +bool listener_shadow_vq;/* IOVA mapping used by the Shadow Virtqueue */ VhostIOVATree *iova_tree;GPtrArray *shadow_vqs;hw/virtio/vhost-vdpa.c +3
-3@@ -220,7 +220,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, vaddr, section->readonly); llsize =int128_sub (llend,int128_make64 (iova)); -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {int r; mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,@@ -247,7 +247,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,return ; fail_map: -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {vhost_iova_tree_remove (v->iova_tree, mem_region); }@@ -292,7 +292,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener, llsize =int128_sub (llend,int128_make64 (iova)); -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {const DMAMap *result;const void *vaddr =memory_region_get_ram_ptr (section->mr) + section->offset_within_region +net/vhost-vdpa.c +0
-0
net/vhost-vdpa.c static NetClientState *net_vhost_vdpa_init(NetClientState *peer, · line 573
Quoted 4 lines
net/vhost-vdpa.c +1
-0@@ -570,6 +570,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, s->vhost_vdpa.index = queue_pair_index; s->always_svq = svq; s->vhost_vdpa.shadow_vqs_enabled = svq; + s->vhost_vdpa.listener_shadow_vq = svq;
Any chance those above two can differ?
Thanks
Quoted 6 lines
net/vhost-vdpa.c +0
-1s->vhost_vdpa.iova_tree = iova_tree;if (!is_datapath) { s->cvq_cmd_out_buffer = qemu_memalign(qemu_real_host_page_size(), --2.31.1
Re: [PATCH v6 09/10] vdpa: Add listener_shadow_vq to vhost_vdpa
Quoted 60 lines · Jason Wang
Quoted 58 lines · Eugenio Pérez
The memory listener that thells the device how to convert GPA to qemu's va is registered against CVQ
vhost_vdpa. This series try to map the memory listener translations to ASID 0, while it maps the CVQ ones to ASID 1.Let's tell the listener if it needs to register them on iova tree or not.
Signed-off-by Eugenio Pérez <[email protected]>---
v5: Solve conflict aboutvhost_iova_tree_removeacceptingmem_regionby
value.--- include/hw/virtio/vhost-vdpa.h | 2 ++ hw/virtio/vhost-vdpa.c | 6 +++--- net/vhost-vdpa.c | 1 + 3 files changed, 6 insertions(+), 3 deletions(-)include/hw/virtio/vhost-vdpa.h +2
-0@@ -34,6 +34,8 @@ typedef struct vhost_vdpa {struct vhost_vdpa_iova_range iova_range;uint64_t acked_features;bool shadow_vqs_enabled; +/* The listener must send iova tree addresses, not GPA */ +bool listener_shadow_vq;/* IOVA mapping used by the Shadow Virtqueue */ VhostIOVATree *iova_tree;GPtrArray *shadow_vqs;hw/virtio/vhost-vdpa.c +2
-2@@ -220,7 +220,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, vaddr, section->readonly); llsize =int128_sub (llend,int128_make64 (iova)); -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {int r; mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,@@ -247,7 +247,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,return ; fail_map: -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {vhost_iova_tree_remove (v->iova_tree, mem_region); }@@ -292,7 +292,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener, llsize =int128_sub (llend,int128_make64 (iova));
Quoted 17 lines
Quoted 10 lines
hw/virtio/vhost-vdpa.c +1
-1-if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {const DMAMap *result;const void *vaddr =memory_region_get_ram_ptr (section->mr) + section->offset_within_region +net/vhost-vdpa.c +0
-0net/vhost-vdpa.c static NetClientState *net_vhost_vdpa_init(NetClientState *peer, · line 573
Quoted 4 lines
net/vhost-vdpa.c +1
-0@@ -570,6 +570,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, s->vhost_vdpa.index = queue_pair_index; s->always_svq = svq; s->vhost_vdpa.shadow_vqs_enabled = svq; + s->vhost_vdpa.listener_shadow_vq = svq;Any chance those above two can differ?
If CVQ is shadowed but data VQs are not, shadow_vqs_enabled is true but listener_shadow_vq is not.
It is more clear in the next commit, where only shadow_vqs_enabled is set to true at vhost_vdpa_net_cvq_start.
Thanks!
Quoted 9 lines
Thanks
Quoted 6 lines
net/vhost-vdpa.c +0
-1s->vhost_vdpa.iova_tree = iova_tree;if (!is_datapath) { s->cvq_cmd_out_buffer = qemu_memalign(qemu_real_host_page_size(), --2.31.1
Re: [PATCH v6 09/10] vdpa: Add listener_shadow_vq to vhost_vdpa
Quoted 20 lines · Eugenio Perez Martin
Quoted 19 lines · Jason Wang
Quoted 18 lines · Eugenio Pérez
The memory listener that thells the device how to convert GPA to qemu's va is registered against CVQ
vhost_vdpa. This series try to map the memory listener translations to ASID 0, while it maps the CVQ ones to ASID 1.Let's tell the listener if it needs to register them on iova tree or not.
Signed-off-by Eugenio Pérez <[email protected]>---
v5: Solve conflict aboutvhost_iova_tree_removeacceptingmem_regionby
value.--- include/hw/virtio/vhost-vdpa.h | 2 ++ hw/virtio/vhost-vdpa.c | 6 +++--- net/vhost-vdpa.c | 1 + 3 files changed, 6 insertions(+), 3 deletions(-)
Quoted 9 lines
Quoted 9 lines
include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 37
Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1
-0@@ -34,6 +34,8 @@ typedef struct vhost_vdpa {struct vhost_vdpa_iova_range iova_range;uint64_t acked_features;bool shadow_vqs_enabled; +/* The listener must send iova tree addresses, not GPA */
Btw, cindy's vIOMMU series will make it not necessarily GPA any more.
Quoted 46 lines
Quoted 30 lines
Quoted 30 lines
include/hw/virtio/vhost-vdpa.h +1
-0+bool listener_shadow_vq;/* IOVA mapping used by the Shadow Virtqueue */ VhostIOVATree *iova_tree;GPtrArray *shadow_vqs;hw/virtio/vhost-vdpa.c +2
-2@@ -220,7 +220,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, vaddr, section->readonly); llsize =int128_sub (llend,int128_make64 (iova)); -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {int r; mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,@@ -247,7 +247,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,return ; fail_map: -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {vhost_iova_tree_remove (v->iova_tree, mem_region); }@@ -292,7 +292,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener, llsize =int128_sub (llend,int128_make64 (iova));Quoted 16 lines
Quoted 10 lines
hw/virtio/vhost-vdpa.c +1
-1-if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {const DMAMap *result;const void *vaddr =memory_region_get_ram_ptr (section->mr) + section->offset_within_region +net/vhost-vdpa.c +0
-0net/vhost-vdpa.c static NetClientState *net_vhost_vdpa_init(NetClientState *peer, · line 573
Quoted 4 lines
net/vhost-vdpa.c +1
-0@@ -570,6 +570,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, s->vhost_vdpa.index = queue_pair_index; s->always_svq = svq; s->vhost_vdpa.shadow_vqs_enabled = svq; + s->vhost_vdpa.listener_shadow_vq = svq;Any chance those above two can differ?
Quoted 5 lines
If CVQ is shadowed but data VQs are not,
shadow_vqs_enabledis true butlistener_shadow_vqis not.It is more clear in the next commit, where only
shadow_vqs_enabledis set to true atvhost_vdpa_net_cvq_start.
Ok, the name looks a little bit confusing. I wonder if it's better to use shadow_cvq and shadow_data ?
Thanks
Quoted 11 lines
Thanks!
Quoted 8 lines
Thanks
Quoted 6 lines
net/vhost-vdpa.c +0
-1s->vhost_vdpa.iova_tree = iova_tree;if (!is_datapath) { s->cvq_cmd_out_buffer = qemu_memalign(qemu_real_host_page_size(), --2.31.1
Re: [PATCH v6 09/10] vdpa: Add listener_shadow_vq to vhost_vdpa
Quoted 23 lines · Jason Wang
Quoted 20 lines · Eugenio Perez Martin
Quoted 19 lines · Jason Wang
Quoted 18 lines · Eugenio Pérez
The memory listener that thells the device how to convert GPA to qemu's va is registered against CVQ
vhost_vdpa. This series try to map the memory listener translations to ASID 0, while it maps the CVQ ones to ASID 1.Let's tell the listener if it needs to register them on iova tree or not.
Signed-off-by Eugenio Pérez <[email protected]>---
v5: Solve conflict aboutvhost_iova_tree_removeacceptingmem_regionby
value.--- include/hw/virtio/vhost-vdpa.h | 2 ++ hw/virtio/vhost-vdpa.c | 6 +++--- net/vhost-vdpa.c | 1 + 3 files changed, 6 insertions(+), 3 deletions(-)
Quoted 13 lines
Quoted 9 lines
Quoted 9 lines
include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 37
Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1
-0@@ -34,6 +34,8 @@ typedef struct vhost_vdpa {struct vhost_vdpa_iova_range iova_range;uint64_t acked_features;bool shadow_vqs_enabled; +/* The listener must send iova tree addresses, not GPA */ Btw, cindy's vIOMMU series will make it not necessarily GPA any more.
Yes, this comment should be tuned then. But the SVQ iova_tree will not be equal to vIOMMU one because shadow vrings.
But maybe SVQ can inspect both instead of having all the duplicated entries.
Quoted 50 lines
Quoted 50 lines
Quoted 31 lines
Quoted 31 lines
include/hw/virtio/vhost-vdpa.h +1
-0+bool listener_shadow_vq;/* IOVA mapping used by the Shadow Virtqueue */ VhostIOVATree *iova_tree;GPtrArray *shadow_vqs;hw/virtio/vhost-vdpa.c +2
-2@@ -220,7 +220,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, vaddr, section->readonly); llsize =int128_sub (llend,int128_make64 (iova)); -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {int r; mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,@@ -247,7 +247,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,return ; fail_map: -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {vhost_iova_tree_remove (v->iova_tree, mem_region); }@@ -292,7 +292,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener, llsize =int128_sub (llend,int128_make64 (iova));Quoted 16 lines
Quoted 10 lines
hw/virtio/vhost-vdpa.c +1
-1-if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {const DMAMap *result;const void *vaddr =memory_region_get_ram_ptr (section->mr) + section->offset_within_region +net/vhost-vdpa.c +0
-0net/vhost-vdpa.c static NetClientState *net_vhost_vdpa_init(NetClientState *peer, · line 573
Quoted 4 lines
net/vhost-vdpa.c +1
-0@@ -570,6 +570,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, s->vhost_vdpa.index = queue_pair_index; s->always_svq = svq; s->vhost_vdpa.shadow_vqs_enabled = svq; + s->vhost_vdpa.listener_shadow_vq = svq;Any chance those above two can differ?
If CVQ is shadowed but data VQs are not,
shadow_vqs_enabledis true butlistener_shadow_vqis not.
Quoted 7 lines
Quoted 2 lines
It is more clear in the next commit, where only
shadow_vqs_enabledis set to true atvhost_vdpa_net_cvq_start.Ok, the name looks a little bit confusing. I wonder if it's better to use
shadow_cvqandshadow_data?
I'm ok with renaming it, but struct vhost_vdpa is generic across all kind of devices, and it does not know if it is a datapath or not for the moment.
Maybe listener_uses_iova_tree?
Thanks!
Re: [PATCH v6 09/10] vdpa: Add listener_shadow_vq to vhost_vdpa
Quoted 92 lines · Eugenio Perez Martin
Quoted 22 lines · Jason Wang
Quoted 20 lines · Eugenio Perez Martin
Quoted 19 lines · Jason Wang
Quoted 18 lines · Eugenio Pérez
The memory listener that thells the device how to convert GPA to qemu's va is registered against CVQ
vhost_vdpa. This series try to map the memory listener translations to ASID 0, while it maps the CVQ ones to ASID 1.Let's tell the listener if it needs to register them on iova tree or not.
Signed-off-by Eugenio Pérez <[email protected]>---
v5: Solve conflict aboutvhost_iova_tree_removeacceptingmem_regionby
value.--- include/hw/virtio/vhost-vdpa.h | 2 ++ hw/virtio/vhost-vdpa.c | 6 +++--- net/vhost-vdpa.c | 1 + 3 files changed, 6 insertions(+), 3 deletions(-)Quoted 12 lines
Quoted 9 lines
Quoted 9 lines
include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 37
Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1
-0@@ -34,6 +34,8 @@ typedef struct vhost_vdpa {struct vhost_vdpa_iova_range iova_range;uint64_t acked_features;bool shadow_vqs_enabled; +/* The listener must send iova tree addresses, not GPA */ Btw, cindy's vIOMMU series will make it not necessarily GPA any more.
Yes, this comment should be tuned then. But the SVQ
iova_treewill not be equal to vIOMMU one because shadow vrings.But maybe SVQ can inspect both instead of having all the duplicated entries.
Quoted 52 lines
Quoted 51 lines
Quoted 30 lines
Quoted 30 lines
include/hw/virtio/vhost-vdpa.h +1
-0+bool listener_shadow_vq;/* IOVA mapping used by the Shadow Virtqueue */ VhostIOVATree *iova_tree;GPtrArray *shadow_vqs;hw/virtio/vhost-vdpa.c +2
-2@@ -220,7 +220,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, vaddr, section->readonly); llsize =int128_sub (llend,int128_make64 (iova)); -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {int r; mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,@@ -247,7 +247,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,return ; fail_map: -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {vhost_iova_tree_remove (v->iova_tree, mem_region); }@@ -292,7 +292,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener, llsize =int128_sub (llend,int128_make64 (iova));Quoted 16 lines
Quoted 10 lines
hw/virtio/vhost-vdpa.c +1
-1-if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {const DMAMap *result;const void *vaddr =memory_region_get_ram_ptr (section->mr) + section->offset_within_region +net/vhost-vdpa.c +0
-0net/vhost-vdpa.c static NetClientState *net_vhost_vdpa_init(NetClientState *peer, · line 573
Quoted 4 lines
net/vhost-vdpa.c +1
-0@@ -570,6 +570,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, s->vhost_vdpa.index = queue_pair_index; s->always_svq = svq; s->vhost_vdpa.shadow_vqs_enabled = svq; + s->vhost_vdpa.listener_shadow_vq = svq;Any chance those above two can differ?
If CVQ is shadowed but data VQs are not,
shadow_vqs_enabledis true butlistener_shadow_vqis not.It is more clear in the next commit, where only
shadow_vqs_enabledis set to true atvhost_vdpa_net_cvq_start.
Quoted 8 lines
Quoted 3 lines
Ok, the name looks a little bit confusing. I wonder if it's better to use
shadow_cvqandshadow_data?I'm ok with renaming it, but struct
vhost_vdpais generic across all kind of devices, and it does not know if it is a datapath or not for the moment.Maybe
listener_uses_iova_tree?
I think "iova_tree" is something that is internal to svq implementation, it's better to define the name from the view of vhost_vdpa level.
Thanks
Quoted 3 lines
Thanks!
Re: [PATCH v6 09/10] vdpa: Add listener_shadow_vq to vhost_vdpa
Quoted 102 lines · Jason Wang
Quoted 99 lines · Eugenio Perez Martin
Quoted 22 lines · Jason Wang
Quoted 20 lines · Eugenio Perez Martin
Quoted 19 lines · Jason Wang
Quoted 18 lines · Eugenio Pérez
The memory listener that thells the device how to convert GPA to qemu's va is registered against CVQ
vhost_vdpa. This series try to map the memory listener translations to ASID 0, while it maps the CVQ ones to ASID 1.Let's tell the listener if it needs to register them on iova tree or not.
Signed-off-by Eugenio Pérez <[email protected]>---
v5: Solve conflict aboutvhost_iova_tree_removeacceptingmem_regionby
value.--- include/hw/virtio/vhost-vdpa.h | 2 ++ hw/virtio/vhost-vdpa.c | 6 +++--- net/vhost-vdpa.c | 1 + 3 files changed, 6 insertions(+), 3 deletions(-)Quoted 12 lines
Quoted 9 lines
Quoted 9 lines
include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 37
Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1
-0@@ -34,6 +34,8 @@ typedef struct vhost_vdpa {struct vhost_vdpa_iova_range iova_range;uint64_t acked_features;bool shadow_vqs_enabled; +/* The listener must send iova tree addresses, not GPA */ Btw, cindy's vIOMMU series will make it not necessarily GPA any more.
Yes, this comment should be tuned then. But the SVQ
iova_treewill not be equal to vIOMMU one because shadow vrings.But maybe SVQ can inspect both instead of having all the duplicated entries.
Quoted 46 lines
Quoted 46 lines
Quoted 46 lines
Quoted 40 lines
include/hw/virtio/vhost-vdpa.h +1
-0+bool listener_shadow_vq;/* IOVA mapping used by the Shadow Virtqueue */ VhostIOVATree *iova_tree;GPtrArray *shadow_vqs;hw/virtio/vhost-vdpa.c +3
-3@@ -220,7 +220,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, vaddr, section->readonly); llsize =int128_sub (llend,int128_make64 (iova)); -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {int r; mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,@@ -247,7 +247,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,return ; fail_map: -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {vhost_iova_tree_remove (v->iova_tree, mem_region); }@@ -292,7 +292,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener, llsize =int128_sub (llend,int128_make64 (iova)); -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {const DMAMap *result;const void *vaddr =memory_region_get_ram_ptr (section->mr) + section->offset_within_region +net/vhost-vdpa.c +0
-0net/vhost-vdpa.c static NetClientState *net_vhost_vdpa_init(NetClientState *peer, · line 573
Quoted 4 lines
net/vhost-vdpa.c +1
-0@@ -570,6 +570,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, s->vhost_vdpa.index = queue_pair_index; s->always_svq = svq; s->vhost_vdpa.shadow_vqs_enabled = svq; + s->vhost_vdpa.listener_shadow_vq = svq;Any chance those above two can differ?
Quoted 9 lines
Quoted 5 lines
If CVQ is shadowed but data VQs are not,
shadow_vqs_enabledis true butlistener_shadow_vqis not.It is more clear in the next commit, where only
shadow_vqs_enabledis set to true atvhost_vdpa_net_cvq_start.Ok, the name looks a little bit confusing. I wonder if it's better to use
shadow_cvqandshadow_data?I'm ok with renaming it, but struct
vhost_vdpais generic across all kind of devices, and it does not know if it is a datapath or not for the moment.
Quoted 6 lines
Quoted 1 line
Maybe
listener_uses_iova_tree?I think "
iova_tree" is something that is internal to svq implementation, it's better to define the name from the view ofvhost_vdpalevel.
I don't get this, vhost_vdpa struct already has a pointer to its iova_tree.
Thanks!
Re: [PATCH v6 09/10] vdpa: Add listener_shadow_vq to vhost_vdpa
Quoted 104 lines · Eugenio Perez Martin
Quoted 102 lines · Jason Wang
Quoted 99 lines · Eugenio Perez Martin
Quoted 22 lines · Jason Wang
Quoted 20 lines · Eugenio Perez Martin
Quoted 19 lines · Jason Wang
Quoted 18 lines · Eugenio Pérez
The memory listener that thells the device how to convert GPA to qemu's va is registered against CVQ
vhost_vdpa. This series try to map the memory listener translations to ASID 0, while it maps the CVQ ones to ASID 1.Let's tell the listener if it needs to register them on iova tree or not.
Signed-off-by Eugenio Pérez <[email protected]>---
v5: Solve conflict aboutvhost_iova_tree_removeacceptingmem_regionby
value.--- include/hw/virtio/vhost-vdpa.h | 2 ++ hw/virtio/vhost-vdpa.c | 6 +++--- net/vhost-vdpa.c | 1 + 3 files changed, 6 insertions(+), 3 deletions(-)Quoted 12 lines
Quoted 9 lines
Quoted 9 lines
include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 37
Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1
-0@@ -34,6 +34,8 @@ typedef struct vhost_vdpa {struct vhost_vdpa_iova_range iova_range;uint64_t acked_features;bool shadow_vqs_enabled; +/* The listener must send iova tree addresses, not GPA */ Btw, cindy's vIOMMU series will make it not necessarily GPA any more.
Yes, this comment should be tuned then. But the SVQ
iova_treewill not be equal to vIOMMU one because shadow vrings.But maybe SVQ can inspect both instead of having all the duplicated entries.
Quoted 46 lines
Quoted 46 lines
Quoted 46 lines
Quoted 40 lines
include/hw/virtio/vhost-vdpa.h +1
-0+bool listener_shadow_vq;/* IOVA mapping used by the Shadow Virtqueue */ VhostIOVATree *iova_tree;GPtrArray *shadow_vqs;hw/virtio/vhost-vdpa.c +3
-3@@ -220,7 +220,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, vaddr, section->readonly); llsize =int128_sub (llend,int128_make64 (iova)); -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {int r; mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,@@ -247,7 +247,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,return ; fail_map: -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {vhost_iova_tree_remove (v->iova_tree, mem_region); }@@ -292,7 +292,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener, llsize =int128_sub (llend,int128_make64 (iova)); -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {const DMAMap *result;const void *vaddr =memory_region_get_ram_ptr (section->mr) + section->offset_within_region +net/vhost-vdpa.c +0
-0net/vhost-vdpa.c static NetClientState *net_vhost_vdpa_init(NetClientState *peer, · line 573
Quoted 4 lines
net/vhost-vdpa.c +1
-0@@ -570,6 +570,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, s->vhost_vdpa.index = queue_pair_index; s->always_svq = svq; s->vhost_vdpa.shadow_vqs_enabled = svq; + s->vhost_vdpa.listener_shadow_vq = svq;Any chance those above two can differ?
Quoted 9 lines
Quoted 5 lines
If CVQ is shadowed but data VQs are not,
shadow_vqs_enabledis true butlistener_shadow_vqis not.It is more clear in the next commit, where only
shadow_vqs_enabledis set to true atvhost_vdpa_net_cvq_start.Ok, the name looks a little bit confusing. I wonder if it's better to use
shadow_cvqandshadow_data?I'm ok with renaming it, but struct
vhost_vdpais generic across all kind of devices, and it does not know if it is a datapath or not for the moment.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Maybe
listener_uses_iova_tree?I think "
iova_tree" is something that is internal to svq implementation, it's better to define the name from the view ofvhost_vdpalevel.I don't get this,
vhost_vdpastruct already has a pointer to itsiova_tree.
Yes, this is a suggestion to improve the readability of the code. So what I meant is to have a name to demonstrate why we need to use iova_tree instead of "uses_iova_tree".
Thanks
Quoted 3 lines
Thanks!
Re: [PATCH v6 09/10] vdpa: Add listener_shadow_vq to vhost_vdpa
Quoted 114 lines · Jason Wang
Quoted 111 lines · Eugenio Perez Martin
Quoted 108 lines · Jason Wang
Quoted 92 lines · Eugenio Perez Martin
Quoted 22 lines · Jason Wang
Quoted 20 lines · Eugenio Perez Martin
Quoted 19 lines · Jason Wang
Quoted 18 lines · Eugenio Pérez
The memory listener that thells the device how to convert GPA to qemu's va is registered against CVQ
vhost_vdpa. This series try to map the memory listener translations to ASID 0, while it maps the CVQ ones to ASID 1.Let's tell the listener if it needs to register them on iova tree or not.
Signed-off-by Eugenio Pérez <[email protected]>---
v5: Solve conflict aboutvhost_iova_tree_removeacceptingmem_regionby
value.--- include/hw/virtio/vhost-vdpa.h | 2 ++ hw/virtio/vhost-vdpa.c | 6 +++--- net/vhost-vdpa.c | 1 + 3 files changed, 6 insertions(+), 3 deletions(-)Quoted 12 lines
Quoted 9 lines
Quoted 9 lines
include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 37
Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1
-0@@ -34,6 +34,8 @@ typedef struct vhost_vdpa {struct vhost_vdpa_iova_range iova_range;uint64_t acked_features;bool shadow_vqs_enabled; +/* The listener must send iova tree addresses, not GPA */ Btw, cindy's vIOMMU series will make it not necessarily GPA any more.
Yes, this comment should be tuned then. But the SVQ
iova_treewill not be equal to vIOMMU one because shadow vrings.But maybe SVQ can inspect both instead of having all the duplicated entries.
Quoted 52 lines
Quoted 51 lines
Quoted 30 lines
Quoted 30 lines
include/hw/virtio/vhost-vdpa.h +1
-0+bool listener_shadow_vq;/* IOVA mapping used by the Shadow Virtqueue */ VhostIOVATree *iova_tree;GPtrArray *shadow_vqs;hw/virtio/vhost-vdpa.c +2
-2@@ -220,7 +220,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, vaddr, section->readonly); llsize =int128_sub (llend,int128_make64 (iova)); -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {int r; mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,@@ -247,7 +247,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,return ; fail_map: -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {vhost_iova_tree_remove (v->iova_tree, mem_region); }@@ -292,7 +292,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener, llsize =int128_sub (llend,int128_make64 (iova));Quoted 16 lines
Quoted 10 lines
hw/virtio/vhost-vdpa.c +1
-1-if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {const DMAMap *result;const void *vaddr =memory_region_get_ram_ptr (section->mr) + section->offset_within_region +net/vhost-vdpa.c +0
-0net/vhost-vdpa.c static NetClientState *net_vhost_vdpa_init(NetClientState *peer, · line 573
Quoted 4 lines
net/vhost-vdpa.c +1
-0@@ -570,6 +570,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, s->vhost_vdpa.index = queue_pair_index; s->always_svq = svq; s->vhost_vdpa.shadow_vqs_enabled = svq; + s->vhost_vdpa.listener_shadow_vq = svq;Any chance those above two can differ?
If CVQ is shadowed but data VQs are not,
shadow_vqs_enabledis true butlistener_shadow_vqis not.It is more clear in the next commit, where only
shadow_vqs_enabledis set to true atvhost_vdpa_net_cvq_start.Quoted 8 lines
Quoted 3 lines
Ok, the name looks a little bit confusing. I wonder if it's better to use
shadow_cvqandshadow_data?I'm ok with renaming it, but struct
vhost_vdpais generic across all kind of devices, and it does not know if it is a datapath or not for the moment.Maybe
listener_uses_iova_tree?I think "
iova_tree" is something that is internal to svq implementation, it's better to define the name from the view ofvhost_vdpalevel.
Quoted 6 lines
Quoted 1 line
I don't get this,
vhost_vdpastruct already has a pointer to itsiova_tree.Yes, this is a suggestion to improve the readability of the code. So what I meant is to have a name to demonstrate why we need to use
iova_treeinstead of "uses_iova_tree".
I understand.
Knowing that the listener will be always bound to data vqs (being net, blk, ...), I think it is ok to rename it to shadow_data.
But I think there is no way to add shadow_cvq properly from hw/virtio/vhost-vdpa.c , since we don't know if the vhost_vdpa belongs to a datapath or not. Would it work just to rename listener_shadow_vq to shadow_data?
Thanks!
Re: [PATCH v6 09/10] vdpa: Add listener_shadow_vq to vhost_vdpa
Quoted 125 lines · Eugenio Perez Martin
Quoted 114 lines · Jason Wang
Quoted 111 lines · Eugenio Perez Martin
Quoted 108 lines · Jason Wang
Quoted 92 lines · Eugenio Perez Martin
Quoted 22 lines · Jason Wang
Quoted 20 lines · Eugenio Perez Martin
Quoted 19 lines · Jason Wang
Quoted 18 lines · Eugenio Pérez
The memory listener that thells the device how to convert GPA to qemu's va is registered against CVQ
vhost_vdpa. This series try to map the memory listener translations to ASID 0, while it maps the CVQ ones to ASID 1.Let's tell the listener if it needs to register them on iova tree or not.
Signed-off-by Eugenio Pérez <[email protected]>---
v5: Solve conflict aboutvhost_iova_tree_removeacceptingmem_regionby
value.--- include/hw/virtio/vhost-vdpa.h | 2 ++ hw/virtio/vhost-vdpa.c | 6 +++--- net/vhost-vdpa.c | 1 + 3 files changed, 6 insertions(+), 3 deletions(-)Quoted 12 lines
Quoted 9 lines
Quoted 9 lines
include/hw/virtio/vhost-vdpa.h typedef struct vhost_vdpa { · line 37
Quoted 9 lines
include/hw/virtio/vhost-vdpa.h +1
-0@@ -34,6 +34,8 @@ typedef struct vhost_vdpa {struct vhost_vdpa_iova_range iova_range;uint64_t acked_features;bool shadow_vqs_enabled; +/* The listener must send iova tree addresses, not GPA */ Btw, cindy's vIOMMU series will make it not necessarily GPA any more.
Yes, this comment should be tuned then. But the SVQ
iova_treewill not be equal to vIOMMU one because shadow vrings.But maybe SVQ can inspect both instead of having all the duplicated entries.
Quoted 52 lines
Quoted 51 lines
Quoted 30 lines
Quoted 30 lines
include/hw/virtio/vhost-vdpa.h +1
-0+bool listener_shadow_vq;/* IOVA mapping used by the Shadow Virtqueue */ VhostIOVATree *iova_tree;GPtrArray *shadow_vqs;hw/virtio/vhost-vdpa.c +2
-2@@ -220,7 +220,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, vaddr, section->readonly); llsize =int128_sub (llend,int128_make64 (iova)); -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {int r; mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,@@ -247,7 +247,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,return ; fail_map: -if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {vhost_iova_tree_remove (v->iova_tree, mem_region); }@@ -292,7 +292,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener, llsize =int128_sub (llend,int128_make64 (iova));Quoted 16 lines
Quoted 10 lines
hw/virtio/vhost-vdpa.c +1
-1-if (v->shadow_vqs_enabled) { +if (v->listener_shadow_vq) {const DMAMap *result;const void *vaddr =memory_region_get_ram_ptr (section->mr) + section->offset_within_region +net/vhost-vdpa.c +0
-0net/vhost-vdpa.c static NetClientState *net_vhost_vdpa_init(NetClientState *peer, · line 573
Quoted 4 lines
net/vhost-vdpa.c +1
-0@@ -570,6 +570,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, s->vhost_vdpa.index = queue_pair_index; s->always_svq = svq; s->vhost_vdpa.shadow_vqs_enabled = svq; + s->vhost_vdpa.listener_shadow_vq = svq;Any chance those above two can differ?
If CVQ is shadowed but data VQs are not,
shadow_vqs_enabledis true butlistener_shadow_vqis not.It is more clear in the next commit, where only
shadow_vqs_enabledis set to true atvhost_vdpa_net_cvq_start.Quoted 8 lines
Quoted 3 lines
Ok, the name looks a little bit confusing. I wonder if it's better to use
shadow_cvqandshadow_data?I'm ok with renaming it, but struct
vhost_vdpais generic across all kind of devices, and it does not know if it is a datapath or not for the moment.Maybe
listener_uses_iova_tree?I think "
iova_tree" is something that is internal to svq implementation, it's better to define the name from the view ofvhost_vdpalevel.Quoted 6 lines
Quoted 1 line
I don't get this,
vhost_vdpastruct already has a pointer to itsiova_tree.Yes, this is a suggestion to improve the readability of the code. So what I meant is to have a name to demonstrate why we need to use
iova_treeinstead of "uses_iova_tree".I understand.
Quoted 7 lines
Knowing that the listener will be always bound to data vqs (being net, blk, ...), I think it is ok to rename it to
shadow_data.But I think there is no way to add
shadow_cvqproperly fromhw/virtio/vhost-vdpa.c, since we don't know if thevhost_vdpabelongs to a datapath or not. Would it work just to renamelistener_shadow_vqtoshadow_data?
This should work.
Thanks
Quoted 3 lines
Thanks!
PATCH v6 10/10 vdpa: Always start CVQ in SVQ mode
In reply to Eugenio PérezIsolate control virtqueue in its own group, allowing to intercept control commands but letting dataplane run totally passthrough to the guest.
---
v6:
* Disable control SVQ if the device does not support it because of features.
v5:
* Fixing the not adding cvq buffers when x-svq=on is specified.
* Move vring state in vhost_vdpa_get_vring_group instead of using a parameter.
* Rename VHOST_VDPA_NET_CVQ_PASSTHROUGH to VHOST_VDPA_NET_DATA_ASID
v4:
* Squash vhost_vdpa_cvq_group_is_independent.
* Rebased on last CVQ start series, that allocated CVQ cmd bufs at load
* Do not check for cvq index on vhost_vdpa_net_prepare, we only have one that callback registered in that NetClientInfo.
v3:
* Make asid related queries print a warning instead of returning an error and stop the start of qemu.
--- hw/virtio/vhost-vdpa.c | 3 +- net/vhost-vdpa.c | 138 ++++++++++++++++++++++++++++++++++++++--- 2 files changed, 132 insertions(+), 9 deletions(-)
hw/virtio/vhost-vdpa.c +2 -1
@@ -648,7 +648,8 @@ static int vhost_vdpa_set_backend_cap(struct vhost_dev *dev) {uint64_t features;uint64_t f =0x1ULL <<VHOST_BACKEND_F_IOTLB_MSG_V2 | -0x1ULL <<VHOST_BACKEND_F_IOTLB_BATCH ; +0x1ULL <<VHOST_BACKEND_F_IOTLB_BATCH | +0x1ULL <<VHOST_BACKEND_F_IOTLB_ASID ;int r;if (vhost_vdpa_call (dev,VHOST_GET_BACKEND_FEATURES , &features)) {
net/vhost-vdpa.c +130 -8
210 lines not on this page · raw message
Signature · 1 line
2.31.1
Re: [PATCH v6 10/10] vdpa: Always start CVQ in SVQ mode
Quoted 2 lines · Eugenio Pérez
Isolate control virtqueue in its own group, allowing to intercept control commands but letting dataplane run totally passthrough to the guest.
I think we need to tweak the title to "vdpa: Always start CVQ in SVQ mode if possible". Since SVQ for CVQ can't be enabled without ASID support?
Quoted 46 lines
Signed-off-by Eugenio Pérez <[email protected]>---
v6:
* Disable control SVQ if the device does not support it because of features.v5:
* Fixing the not adding cvq buffers when x-svq=on is specified.
* Move vring state invhost_vdpa_get_vring_groupinstead of using a parameter.
* RenameVHOST_VDPA_NET_CVQ_PASSTHROUGHtoVHOST_VDPA_NET_DATA_ASIDv4:
* Squashvhost_vdpa_cvq_group_is_independent.
* Rebased on last CVQ start series, that allocated CVQ cmd bufs at load
* Do not check for cvq index onvhost_vdpa_net_prepare, we only have one that callback registered in that NetClientInfo.v3:
* Make asid related queries print a warning instead of returning an error and stop the start of qemu.--- hw/virtio/vhost-vdpa.c | 3 +- net/vhost-vdpa.c | 138 ++++++++++++++++++++++++++++++++++++++--- 2 files changed, 132 insertions(+), 9 deletions(-)hw/virtio/vhost-vdpa.c +2
-1@@ -648,7 +648,8 @@ static int vhost_vdpa_set_backend_cap(struct vhost_dev *dev) {uint64_t features;uint64_t f =0x1ULL <<VHOST_BACKEND_F_IOTLB_MSG_V2 | -0x1ULL <<VHOST_BACKEND_F_IOTLB_BATCH ; +0x1ULL <<VHOST_BACKEND_F_IOTLB_BATCH | +0x1ULL <<VHOST_BACKEND_F_IOTLB_ASID ;int r;if (vhost_vdpa_call (dev,VHOST_GET_BACKEND_FEATURES , &features)) {net/vhost-vdpa.c +0
-0
net/vhost-vdpa.c typedef struct VhostVDPAState { · line 42
Quoted 5 lines
net/vhost-vdpa.c +2
-0@@ -38,6 +38,9 @@ typedef struct VhostVDPAState {void *cvq_cmd_out_buffer;virtio_net_ctrl_ack *status; +/* Number of address spaces supported by the device */ +unsigned address_space_num;
I'm not sure this is the best place to store thing like this since it can cause confusion. We will have multiple VhostVDPAState when multiqueue is enabled.
Quoted 16 lines
net/vhost-vdpa.c +4
-0+/* The device always have SVQ enabled */ bool always_svq;bool started;@@ -101,6 +104,9 @@ static const uint64_t vdpa_svq_device_features =BIT_ULL (VIRTIO_NET_F_RSC_EXT ) |BIT_ULL (VIRTIO_NET_F_STANDBY ); +#define VHOST_VDPA_NET_DATA_ASID 0 +#define VHOST_VDPA_NET_CVQ_ASID 1 +VHostNetState *vhost_vdpa_get_vhost_net (NetClientState *nc) {VhostVDPAState *s =DO_UPCAST (VhostVDPAState, nc, nc);@@ -242,6 +248,34 @@ static NetClientInfo net_vhost_vdpa_info = { .check_peer_type = vhost_vdpa_check_peer_type,
net/vhost-vdpa.c static NetClientInfo net_vhost_vdpa_info = { · line 258
Quoted 10 lines
net/vhost-vdpa.c +8
-0}; +static uint32_t vhost_vdpa_get_vring_group (int device_fd,unsigned vq_index) +{ +struct vhost_vring_state state = { + .index = vq_index, + }; +int r =ioctl (device_fd,VHOST_VDPA_GET_VRING_GROUP , &state); + +return r <0 ?0 : state.num;
Assume 0 when ioctl() fail is probably not a good idea: errors in ioctl might be hidden. It would be better to fallback to 0 when ASID is not supported.
Quoted 42 lines
net/vhost-vdpa.c +30
-1+} + +static int vhost_vdpa_set_address_space_id (struct vhost_vdpa *v, +unsigned vq_group, +unsigned asid_num) +{ +struct vhost_vring_state asid = { + .index = vq_group, + .num = asid_num, + }; +int ret; + + ret =ioctl (v->device_fd,VHOST_VDPA_SET_GROUP_ASID , &asid); +if (unlikely (ret <0 )) { +warn_report ("Can't set vq group %u asid %u, errno=%d (%s)" , + asid.index, asid.num, errno,g_strerror (errno)); + } +return ret; +} +static void vhost_vdpa_cvq_unmap_buf (struct vhost_vdpa *v,void *addr) {VhostIOVATree *tree = v->iova_tree;@@ -316,11 +350,54 @@ dma_map_err:static int vhost_vdpa_net_cvq_start (NetClientState *nc) {VhostVDPAState *s; -int r; +struct vhost_vdpa *v; +uint32_t cvq_group; +int cvq_index, r;assert (nc->info->type ==NET_CLIENT_DRIVER_VHOST_VDPA ); s =DO_UPCAST (VhostVDPAState, nc, nc); + v = &s->vhost_vdpa; + + v->listener_shadow_vq = s->always_svq; + v->shadow_vqs_enabled = s->always_svq; + s->vhost_vdpa.address_space_id =VHOST_VDPA_NET_DATA_ASID ; + +if (s->always_svq) {
net/vhost-vdpa.c dma_map_err: · line 376
Quoted 10 lines
net/vhost-vdpa.c +10
-0+ goto out; + } + +if (s->address_space_num <2 ) { +return 0 ; + } + +if (!vhost_vdpa_net_valid_svq_features (v->dev->features,NULL )) { +return 0 ; + }
Any reason we do the above check during the start/stop? It should be easier to do that in the initialization.
Quoted 41 lines
net/vhost-vdpa.c +34
-0+ +/** +* Check if all the virtqueues of the virtio device are in a different vq +* than the last vq. VQ group of last group passed in cvq_group. +*/ + cvq_index = v->dev->vq_index_end -1 ; + cvq_group =vhost_vdpa_get_vring_group (v->device_fd, cvq_index); +for (int i =0 ; i < cvq_index; ++i) { +uint32_t group =vhost_vdpa_get_vring_group (v->device_fd, i); + +if (unlikely (group == cvq_group)) { +warn_report ("CVQ %u group is the same as VQ %u one (%u)" , cvq_group, + i, group); +return 0 ; + } + } + + r =vhost_vdpa_set_address_space_id (v, cvq_group,VHOST_VDPA_NET_CVQ_ASID ); +if (r ==0 ) { + v->shadow_vqs_enabled = true; + s->vhost_vdpa.address_space_id =VHOST_VDPA_NET_CVQ_ASID ; + } + +out:if (!s->vhost_vdpa.shadow_vqs_enabled) {return 0 ; }@@ -542,12 +619,38 @@ static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = { .avail_handler = vhost_vdpa_net_handle_ctrl_avail, }; +static uint32_t vhost_vdpa_get_as_num (int vdpa_device_fd) +{ +uint64_t features; +unsigned num_as; +int r; + + r =ioctl (vdpa_device_fd,VHOST_GET_BACKEND_FEATURES , &features); +if (unlikely (r <0 )) { +warn_report ("Cannot get backend features" ); +return 1 ;
net/vhost-vdpa.c static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = { · line 641
Quoted 10 lines
net/vhost-vdpa.c +10
-0+ } + +if (!(features &BIT_ULL (VHOST_BACKEND_F_IOTLB_ASID ))) { +return 1 ; + } + + r =ioctl (vdpa_device_fd,VHOST_VDPA_GET_AS_NUM , &num_as); +if (unlikely (r <0 )) { +warn_report ("Cannot retrieve number of supported ASs" ); +return 1 ;
Let's return error here. This help to identify bugs of qemu or kernel.
Quoted 33 lines
net/vhost-vdpa.c +9
-0+ } + +return num_as; +} +static NetClientState *net_vhost_vdpa_init (NetClientState *peer,const char *device,const char *name,int vdpa_device_fd,int queue_pair_index,int nvqs, +unsigned nas,bool is_datapath,bool svq,VhostIOVATree *iova_tree)@@ -566,6 +669,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,qemu_set_info_str (nc,TYPE_VHOST_VDPA ); s =DO_UPCAST (VhostVDPAState, nc, nc); + s->address_space_num = nas; s->vhost_vdpa.device_fd = vdpa_device_fd; s->vhost_vdpa.index = queue_pair_index; s->always_svq = svq;@@ -652,6 +756,8 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,g_autoptr (VhostIOVATree) iova_tree =NULL ;NetClientState *nc;int queue_pairs, r, i =0 , has_cvq =0 ; +unsigned num_as =1 ; +bool svq_cvq;assert (netdev->type ==NET_CLIENT_DRIVER_VHOST_VDPA ); opts = &netdev->u.vhost_vdpa;
net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 806
Quoted 10 lines
net/vhost-vdpa.c +5
-2@@ -693,12 +799,28 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,return queue_pairs; } -if (opts->x_svq) { -struct vhost_vdpa_iova_range iova_range; + svq_cvq = opts->x_svq; +if (has_cvq && !opts->x_svq) { + num_as =vhost_vdpa_get_as_num (vdpa_device_fd); + svq_cvq = num_as >1 ; + }
The above check is not easy to follow, how about?
svq_cvq = vhost_vdpa_get_as_num() > 1 ? true : opts->x_svq;
net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 808
Quoted 2 lines
net/vhost-vdpa.c +2
-0+ +if (opts->x_svq || svq_cvq) {
Any chance we can have opts->x_svq = true but svq_cvq = false? Checking svq_cvq seems sufficient here.
net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 813
Quoted 7 lines
net/vhost-vdpa.c +4
-2+Error *warn =NULL ; -if (!vhost_vdpa_net_valid_svq_features (features, errp)) { - goto err_svq; + svq_cvq =vhost_vdpa_net_valid_svq_features (features, + opts->x_svq ? errp : &warn); +if (!svq_cvq) {
Same question as above.
Quoted 14 lines
net/vhost-vdpa.c +9
-0+if (opts->x_svq) { + goto err_svq; + }else { +warn_reportf_err (warn,"Cannot shadow CVQ: " ); + } } + } + +if (opts->x_svq || svq_cvq) { +struct vhost_vdpa_iova_range iova_range;vhost_vdpa_get_iova_range (vdpa_device_fd, &iova_range); iova_tree =vhost_iova_tree_new (iova_range.first, iova_range.last);
net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 833
Quoted 6 lines
net/vhost-vdpa.c +1
-2@@ -708,15 +830,15 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,for (i =0 ; i < queue_pairs; i++) { ncs[i] = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name, - vdpa_device_fd, i,2 , true, opts->x_svq, - iova_tree); + vdpa_device_fd, i, 2, num_as, true,
I don't get why we need pass num_as to a specific vhost_vdpa structure. It should be sufficient to pass asid there.
Thanks
Quoted 12 lines
net/vhost-vdpa.c +2
-1+ opts->x_svq, iova_tree);if (!ncs[i]) goto err; }if (has_cvq) { nc =net_vhost_vdpa_init (peer,TYPE_VHOST_VDPA , name, - vdpa_device_fd, i,1 , false, + vdpa_device_fd, i,1 , num_as, false, opts->x_svq, iova_tree);if (!nc) goto err;
Re: [PATCH v6 10/10] vdpa: Always start CVQ in SVQ mode
Quoted 10 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez
Isolate control virtqueue in its own group, allowing to intercept control commands but letting dataplane run totally passthrough to the guest.
I think we need to tweak the title to "vdpa: Always start CVQ in SVQ mode if possible". Since SVQ for CVQ can't be enabled without ASID support?
Yes, I totally agree.
Quoted 50 lines
Quoted 50 lines
Signed-off-by Eugenio Pérez <[email protected]>---
v6:
* Disable control SVQ if the device does not support it because of features.v5:
* Fixing the not adding cvq buffers when x-svq=on is specified.
* Move vring state invhost_vdpa_get_vring_groupinstead of using a parameter.
* RenameVHOST_VDPA_NET_CVQ_PASSTHROUGHtoVHOST_VDPA_NET_DATA_ASIDv4:
* Squashvhost_vdpa_cvq_group_is_independent.
* Rebased on last CVQ start series, that allocated CVQ cmd bufs at load
* Do not check for cvq index onvhost_vdpa_net_prepare, we only have one that callback registered in that NetClientInfo.v3:
* Make asid related queries print a warning instead of returning an error and stop the start of qemu.--- hw/virtio/vhost-vdpa.c | 3 +- net/vhost-vdpa.c | 138 ++++++++++++++++++++++++++++++++++++++--- 2 files changed, 132 insertions(+), 9 deletions(-)hw/virtio/vhost-vdpa.c +2
-1@@ -648,7 +648,8 @@ static int vhost_vdpa_set_backend_cap(struct vhost_dev *dev) {uint64_t features;uint64_t f =0x1ULL <<VHOST_BACKEND_F_IOTLB_MSG_V2 | -0x1ULL <<VHOST_BACKEND_F_IOTLB_BATCH ; +0x1ULL <<VHOST_BACKEND_F_IOTLB_BATCH | +0x1ULL <<VHOST_BACKEND_F_IOTLB_ASID ;int r;if (vhost_vdpa_call (dev,VHOST_GET_BACKEND_FEATURES , &features)) {net/vhost-vdpa.c +0
-0@@ -38,6 +38,9 @@ typedef struct VhostVDPAState {void *cvq_cmd_out_buffer;virtio_net_ctrl_ack *status;
Quoted 8 lines
net/vhost-vdpa.c typedef struct VhostVDPAState { · line 42
Quoted 2 lines
net/vhost-vdpa.c +2
-0+/* Number of address spaces supported by the device */ +unsigned address_space_num;I'm not sure this is the best place to store thing like this since it can cause confusion. We will have multiple VhostVDPAState when multiqueue is enabled.
I think we can delete this and ask it on each device start.
Quoted 19 lines
Quoted 19 lines
net/vhost-vdpa.c +4
-0+/* The device always have SVQ enabled */ bool always_svq;bool started;@@ -101,6 +104,9 @@ static const uint64_t vdpa_svq_device_features =BIT_ULL (VIRTIO_NET_F_RSC_EXT ) |BIT_ULL (VIRTIO_NET_F_STANDBY ); +#define VHOST_VDPA_NET_DATA_ASID 0 +#define VHOST_VDPA_NET_CVQ_ASID 1 +VHostNetState *vhost_vdpa_get_vhost_net (NetClientState *nc) {VhostVDPAState *s =DO_UPCAST (VhostVDPAState, nc, nc);@@ -242,6 +248,34 @@ static NetClientInfo net_vhost_vdpa_info = { .check_peer_type = vhost_vdpa_check_peer_type, };
Quoted 14 lines
net/vhost-vdpa.c static NetClientInfo net_vhost_vdpa_info = { · line 258
Quoted 8 lines
net/vhost-vdpa.c +8
-0+static uint32_t vhost_vdpa_get_vring_group (int device_fd,unsigned vq_index) +{ +struct vhost_vring_state state = { + .index = vq_index, + }; +int r =ioctl (device_fd,VHOST_VDPA_GET_VRING_GROUP , &state); + +return r <0 ?0 : state.num;Assume 0 when
ioctl()fail is probably not a good idea: errors in ioctl might be hidden. It would be better to fallback to 0 when ASID is not supported.
Did I misunderstand you on [1]?
Quoted 49 lines
Quoted 49 lines
net/vhost-vdpa.c +36
-1+} + +static int vhost_vdpa_set_address_space_id (struct vhost_vdpa *v, +unsigned vq_group, +unsigned asid_num) +{ +struct vhost_vring_state asid = { + .index = vq_group, + .num = asid_num, + }; +int ret; + + ret =ioctl (v->device_fd,VHOST_VDPA_SET_GROUP_ASID , &asid); +if (unlikely (ret <0 )) { +warn_report ("Can't set vq group %u asid %u, errno=%d (%s)" , + asid.index, asid.num, errno,g_strerror (errno)); + } +return ret; +} +static void vhost_vdpa_cvq_unmap_buf (struct vhost_vdpa *v,void *addr) {VhostIOVATree *tree = v->iova_tree;@@ -316,11 +350,54 @@ dma_map_err:static int vhost_vdpa_net_cvq_start (NetClientState *nc) {VhostVDPAState *s; -int r; +struct vhost_vdpa *v; +uint32_t cvq_group; +int cvq_index, r;assert (nc->info->type ==NET_CLIENT_DRIVER_VHOST_VDPA ); s =DO_UPCAST (VhostVDPAState, nc, nc); + v = &s->vhost_vdpa; + + v->listener_shadow_vq = s->always_svq; + v->shadow_vqs_enabled = s->always_svq; + s->vhost_vdpa.address_space_id =VHOST_VDPA_NET_DATA_ASID ; + +if (s->always_svq) { + goto out; + } + +if (s->address_space_num <2 ) { +return 0 ; + }
Quoted 9 lines
net/vhost-vdpa.c dma_map_err: · line 376
Quoted 4 lines
net/vhost-vdpa.c +4
-0+ +if (!vhost_vdpa_net_valid_svq_features (v->dev->features,NULL )) { +return 0 ; + }Any reason we do the above check during the start/stop? It should be easier to do that in the initialization.
We can store it as a member of VhostVDPAState maybe? They will be duplicated like the current number of AS.
Quoted 47 lines
Quoted 47 lines
net/vhost-vdpa.c +39
-0+ +/** +* Check if all the virtqueues of the virtio device are in a different vq +* than the last vq. VQ group of last group passed in cvq_group. +*/ + cvq_index = v->dev->vq_index_end -1 ; + cvq_group =vhost_vdpa_get_vring_group (v->device_fd, cvq_index); +for (int i =0 ; i < cvq_index; ++i) { +uint32_t group =vhost_vdpa_get_vring_group (v->device_fd, i); + +if (unlikely (group == cvq_group)) { +warn_report ("CVQ %u group is the same as VQ %u one (%u)" , cvq_group, + i, group); +return 0 ; + } + } + + r =vhost_vdpa_set_address_space_id (v, cvq_group,VHOST_VDPA_NET_CVQ_ASID ); +if (r ==0 ) { + v->shadow_vqs_enabled = true; + s->vhost_vdpa.address_space_id =VHOST_VDPA_NET_CVQ_ASID ; + } + +out:if (!s->vhost_vdpa.shadow_vqs_enabled) {return 0 ; }@@ -542,12 +619,38 @@ static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = { .avail_handler = vhost_vdpa_net_handle_ctrl_avail, }; +static uint32_t vhost_vdpa_get_as_num (int vdpa_device_fd) +{ +uint64_t features; +unsigned num_as; +int r; + + r =ioctl (vdpa_device_fd,VHOST_GET_BACKEND_FEATURES , &features); +if (unlikely (r <0 )) { +warn_report ("Cannot get backend features" ); +return 1 ; + } + +if (!(features &BIT_ULL (VHOST_BACKEND_F_IOTLB_ASID ))) { +return 1 ; + }
Quoted 9 lines
net/vhost-vdpa.c static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = { · line 641
Quoted 5 lines
net/vhost-vdpa.c +5
-0+ + r =ioctl (vdpa_device_fd,VHOST_VDPA_GET_AS_NUM , &num_as); +if (unlikely (r <0 )) { +warn_report ("Cannot retrieve number of supported ASs" ); +return 1 ;Let's return error here. This help. to identify bugs of qemu or kernel.
Same comment as with VHOST_VDPA_GET_VRING_GROUP.
Quoted 48 lines
Quoted 34 lines
net/vhost-vdpa.c +9
-0+ } + +return num_as; +} +static NetClientState *net_vhost_vdpa_init (NetClientState *peer,const char *device,const char *name,int vdpa_device_fd,int queue_pair_index,int nvqs, +unsigned nas,bool is_datapath,bool svq,VhostIOVATree *iova_tree)@@ -566,6 +669,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,qemu_set_info_str (nc,TYPE_VHOST_VDPA ); s =DO_UPCAST (VhostVDPAState, nc, nc); + s->address_space_num = nas; s->vhost_vdpa.device_fd = vdpa_device_fd; s->vhost_vdpa.index = queue_pair_index; s->always_svq = svq;@@ -652,6 +756,8 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,g_autoptr (VhostIOVATree) iova_tree =NULL ;NetClientState *nc;int queue_pairs, r, i =0 , has_cvq =0 ; +unsigned num_as =1 ; +bool svq_cvq;assert (netdev->type ==NET_CLIENT_DRIVER_VHOST_VDPA ); opts = &netdev->u.vhost_vdpa;net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 806
Quoted 10 lines
net/vhost-vdpa.c +5
-2@@ -693,12 +799,28 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,return queue_pairs; } -if (opts->x_svq) { -struct vhost_vdpa_iova_range iova_range; + svq_cvq = opts->x_svq; +if (has_cvq && !opts->x_svq) { + num_as =vhost_vdpa_get_as_num (vdpa_device_fd); + svq_cvq = num_as >1 ; + }The above check is not easy to follow, how about?
Quoted 2 lines
svq_cvq=vhost_vdpa_get_as_num()> 1 ? true :opts->x_svq;
That would allocate the iova tree even if CVQ is not used in the guest. And num_as is reused later, although we can ask it to the device at device start to avoid this.
If any, the linear conversion would be:svq_cvq = opts->x_svq || (has_cvq && vhost_vdpa_get_as_num(vdpa_device_fd))
So we avoid the AS_NUM ioctl if not needed.
Quoted 8 lines
net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 808
Quoted 3 lines
net/vhost-vdpa.c +2
-0+ +if (opts->x_svq || svq_cvq) {Any chance we can have
opts->x_svq= true butsvq_cvq= false? Checkingsvq_cvqseems sufficient here.
The reverse is possible, to have svq_cvq but no opts->x_svq.
Depending on that, this code emits a warning or a fatal error.
Quoted 28 lines
net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 813
Quoted 8 lines
net/vhost-vdpa.c +4
-2+Error *warn =NULL ; -if (!vhost_vdpa_net_valid_svq_features (features, errp)) { - goto err_svq; + svq_cvq =vhost_vdpa_net_valid_svq_features (features, + opts->x_svq ? errp : &warn); +if (!svq_cvq) {Same question as above.
Quoted 15 lines
net/vhost-vdpa.c +9
-0+if (opts->x_svq) { + goto err_svq; + }else { +warn_reportf_err (warn,"Cannot shadow CVQ: " ); + } } + } + +if (opts->x_svq || svq_cvq) { +struct vhost_vdpa_iova_range iova_range;vhost_vdpa_get_iova_range (vdpa_device_fd, &iova_range); iova_tree =vhost_iova_tree_new (iova_range.first, iova_range.last);@@ -708,15 +830,15 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
Quoted 10 lines
net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 833
Quoted 5 lines
net/vhost-vdpa.c +1
-2for (i =0 ; i < queue_pairs; i++) { ncs[i] = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name, - vdpa_device_fd, i,2 , true, opts->x_svq, - iova_tree); + vdpa_device_fd, i, 2, num_as, true,I don't get why we need pass
num_asto a specificvhost_vdpastructure. It should be sufficient to pass asid there.
ASID is not known at this time, but at device's start. This is because we cannot ask if the CVQ is in its own vq group, because we don't know the control virtqueue index until the guest acknowledges the different features.
Thanks!
[1] https://www.mail-archive.com/[email protected]/msg901685.html
Quoted 14 lines
Quoted 13 lines
net/vhost-vdpa.c +2
-1+ opts->x_svq, iova_tree);if (!ncs[i]) goto err; }if (has_cvq) { nc =net_vhost_vdpa_init (peer,TYPE_VHOST_VDPA , name, - vdpa_device_fd, i,1 , false, + vdpa_device_fd, i,1 , num_as, false, opts->x_svq, iova_tree);if (!nc) goto err;
Re: [PATCH v6 10/10] vdpa: Always start CVQ in SVQ mode
Quoted 10 lines · Eugenio Perez Martin
Quoted 8 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez
Isolate control virtqueue in its own group, allowing to intercept control commands but letting dataplane run totally passthrough to the guest.
I think we need to tweak the title to "vdpa: Always start CVQ in SVQ mode if possible". Since SVQ for CVQ can't be enabled without ASID support?
Yes, I totally agree.
Btw, I wonder if it's worth to remove the "x-" prefix for the shadow virtqueue. It can help for the devices without ASID support but want do live migration.
Quoted 89 lines
Quoted 49 lines
Quoted 49 lines
Signed-off-by Eugenio Pérez <[email protected]>---
v6:
* Disable control SVQ if the device does not support it because of features.v5:
* Fixing the not adding cvq buffers when x-svq=on is specified.
* Move vring state invhost_vdpa_get_vring_groupinstead of using a parameter.
* RenameVHOST_VDPA_NET_CVQ_PASSTHROUGHtoVHOST_VDPA_NET_DATA_ASIDv4:
* Squashvhost_vdpa_cvq_group_is_independent.
* Rebased on last CVQ start series, that allocated CVQ cmd bufs at load
* Do not check for cvq index onvhost_vdpa_net_prepare, we only have one that callback registered in that NetClientInfo.v3:
* Make asid related queries print a warning instead of returning an error and stop the start of qemu.--- hw/virtio/vhost-vdpa.c | 3 +- net/vhost-vdpa.c | 138 ++++++++++++++++++++++++++++++++++++++--- 2 files changed, 132 insertions(+), 9 deletions(-)hw/virtio/vhost-vdpa.c +2
-1@@ -648,7 +648,8 @@ static int vhost_vdpa_set_backend_cap(struct vhost_dev *dev) {uint64_t features;uint64_t f =0x1ULL <<VHOST_BACKEND_F_IOTLB_MSG_V2 | -0x1ULL <<VHOST_BACKEND_F_IOTLB_BATCH ; +0x1ULL <<VHOST_BACKEND_F_IOTLB_BATCH | +0x1ULL <<VHOST_BACKEND_F_IOTLB_ASID ;int r;if (vhost_vdpa_call (dev,VHOST_GET_BACKEND_FEATURES , &features)) {net/vhost-vdpa.c +0
-0@@ -38,6 +38,9 @@ typedef struct VhostVDPAState {void *cvq_cmd_out_buffer;virtio_net_ctrl_ack *status;Quoted 7 lines
net/vhost-vdpa.c typedef struct VhostVDPAState { · line 42
Quoted 2 lines
net/vhost-vdpa.c +2
-0+/* Number of address spaces supported by the device */ +unsigned address_space_num;I'm not sure this is the best place to store thing like this since it can cause confusion. We will have multiple VhostVDPAState when multiqueue is enabled.
I think we can delete this and ask it on each device start.
Quoted 18 lines
Quoted 18 lines
net/vhost-vdpa.c +4
-0+/* The device always have SVQ enabled */ bool always_svq;bool started;@@ -101,6 +104,9 @@ static const uint64_t vdpa_svq_device_features =BIT_ULL (VIRTIO_NET_F_RSC_EXT ) |BIT_ULL (VIRTIO_NET_F_STANDBY ); +#define VHOST_VDPA_NET_DATA_ASID 0 +#define VHOST_VDPA_NET_CVQ_ASID 1 +VHostNetState *vhost_vdpa_get_vhost_net (NetClientState *nc) {VhostVDPAState *s =DO_UPCAST (VhostVDPAState, nc, nc);@@ -242,6 +248,34 @@ static NetClientInfo net_vhost_vdpa_info = { .check_peer_type = vhost_vdpa_check_peer_type, };Quoted 13 lines
net/vhost-vdpa.c static NetClientInfo net_vhost_vdpa_info = { · line 258
Quoted 8 lines
net/vhost-vdpa.c +8
-0+static uint32_t vhost_vdpa_get_vring_group (int device_fd,unsigned vq_index) +{ +struct vhost_vring_state state = { + .index = vq_index, + }; +int r =ioctl (device_fd,VHOST_VDPA_GET_VRING_GROUP , &state); + +return r <0 ?0 : state.num;Assume 0 when
ioctl()fail is probably not a good idea: errors in ioctl might be hidden. It would be better to fallback to 0 when ASID is not supported.
Quoted 1 line
Did I misunderstand you on [1]?
Nope. I think I was wrong at that time then :( Sorry for that.
We should differ from the case
1) no ASID support so 0 is assumed
2) something wrong in the case of ioctl, it's not necessarily a ENOTSUPP.
Quoted 57 lines
Quoted 48 lines
Quoted 48 lines
net/vhost-vdpa.c +35
-1+} + +static int vhost_vdpa_set_address_space_id (struct vhost_vdpa *v, +unsigned vq_group, +unsigned asid_num) +{ +struct vhost_vring_state asid = { + .index = vq_group, + .num = asid_num, + }; +int ret; + + ret =ioctl (v->device_fd,VHOST_VDPA_SET_GROUP_ASID , &asid); +if (unlikely (ret <0 )) { +warn_report ("Can't set vq group %u asid %u, errno=%d (%s)" , + asid.index, asid.num, errno,g_strerror (errno)); + } +return ret; +} +static void vhost_vdpa_cvq_unmap_buf (struct vhost_vdpa *v,void *addr) {VhostIOVATree *tree = v->iova_tree;@@ -316,11 +350,54 @@ dma_map_err:static int vhost_vdpa_net_cvq_start (NetClientState *nc) {VhostVDPAState *s; -int r; +struct vhost_vdpa *v; +uint32_t cvq_group; +int cvq_index, r;assert (nc->info->type ==NET_CLIENT_DRIVER_VHOST_VDPA ); s =DO_UPCAST (VhostVDPAState, nc, nc); + v = &s->vhost_vdpa; + + v->listener_shadow_vq = s->always_svq; + v->shadow_vqs_enabled = s->always_svq; + s->vhost_vdpa.address_space_id =VHOST_VDPA_NET_DATA_ASID ; + +if (s->always_svq) { + goto out; + } + +if (s->address_space_num <2 ) { +return 0 ;Quoted 9 lines
net/vhost-vdpa.c dma_map_err: · line 376
Quoted 5 lines
net/vhost-vdpa.c +5
-0+ } + +if (!vhost_vdpa_net_valid_svq_features (v->dev->features,NULL )) { +return 0 ; + }Any reason we do the above check during the start/stop? It should be easier to do that in the initialization.
Quoted 2 lines
We can store it as a member of VhostVDPAState maybe? They will be duplicated like the current number of AS.
I meant each VhostVDPAState just need to know the ASID it needs to use. There's no need to know the total number of address spaces or do the validation on it during start (the validation could be done during initialization).
Quoted 103 lines
Quoted 46 lines
Quoted 46 lines
net/vhost-vdpa.c +38
-0+ +/** +* Check if all the virtqueues of the virtio device are in a different vq +* than the last vq. VQ group of last group passed in cvq_group. +*/ + cvq_index = v->dev->vq_index_end -1 ; + cvq_group =vhost_vdpa_get_vring_group (v->device_fd, cvq_index); +for (int i =0 ; i < cvq_index; ++i) { +uint32_t group =vhost_vdpa_get_vring_group (v->device_fd, i); + +if (unlikely (group == cvq_group)) { +warn_report ("CVQ %u group is the same as VQ %u one (%u)" , cvq_group, + i, group); +return 0 ; + } + } + + r =vhost_vdpa_set_address_space_id (v, cvq_group,VHOST_VDPA_NET_CVQ_ASID ); +if (r ==0 ) { + v->shadow_vqs_enabled = true; + s->vhost_vdpa.address_space_id =VHOST_VDPA_NET_CVQ_ASID ; + } + +out:if (!s->vhost_vdpa.shadow_vqs_enabled) {return 0 ; }@@ -542,12 +619,38 @@ static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = { .avail_handler = vhost_vdpa_net_handle_ctrl_avail, }; +static uint32_t vhost_vdpa_get_as_num (int vdpa_device_fd) +{ +uint64_t features; +unsigned num_as; +int r; + + r =ioctl (vdpa_device_fd,VHOST_GET_BACKEND_FEATURES , &features); +if (unlikely (r <0 )) { +warn_report ("Cannot get backend features" ); +return 1 ; + } + +if (!(features &BIT_ULL (VHOST_BACKEND_F_IOTLB_ASID ))) { +return 1 ;Quoted 9 lines
net/vhost-vdpa.c static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = { · line 641
Quoted 6 lines
net/vhost-vdpa.c +6
-0+ } + + r =ioctl (vdpa_device_fd,VHOST_VDPA_GET_AS_NUM , &num_as); +if (unlikely (r <0 )) { +warn_report ("Cannot retrieve number of supported ASs" ); +return 1 ;Let's return error here. This help. to identify bugs of qemu or kernel.
Same comment as with
VHOST_VDPA_GET_VRING_GROUP.Quoted 46 lines
Quoted 33 lines
net/vhost-vdpa.c +9
-0+ } + +return num_as; +} +static NetClientState *net_vhost_vdpa_init (NetClientState *peer,const char *device,const char *name,int vdpa_device_fd,int queue_pair_index,int nvqs, +unsigned nas,bool is_datapath,bool svq,VhostIOVATree *iova_tree)@@ -566,6 +669,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,qemu_set_info_str (nc,TYPE_VHOST_VDPA ); s =DO_UPCAST (VhostVDPAState, nc, nc); + s->address_space_num = nas; s->vhost_vdpa.device_fd = vdpa_device_fd; s->vhost_vdpa.index = queue_pair_index; s->always_svq = svq;@@ -652,6 +756,8 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,g_autoptr (VhostIOVATree) iova_tree =NULL ;NetClientState *nc;int queue_pairs, r, i =0 , has_cvq =0 ; +unsigned num_as =1 ; +bool svq_cvq;assert (netdev->type ==NET_CLIENT_DRIVER_VHOST_VDPA ); opts = &netdev->u.vhost_vdpa;net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 806
Quoted 10 lines
net/vhost-vdpa.c +5
-2@@ -693,12 +799,28 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,return queue_pairs; } -if (opts->x_svq) { -struct vhost_vdpa_iova_range iova_range; + svq_cvq = opts->x_svq; +if (has_cvq && !opts->x_svq) { + num_as =vhost_vdpa_get_as_num (vdpa_device_fd); + svq_cvq = num_as >1 ; + }The above check is not easy to follow, how about?
Quoted 5 lines
Quoted 2 lines
svq_cvq=vhost_vdpa_get_as_num()> 1 ? true :opts->x_svq;That would allocate the iova tree even if CVQ is not used in the guest. And
num_asis reused later, although we can ask it to the device at device start to avoid this.
Ok.
Quoted 5 lines
If any, the linear conversion would be:
svq_cvq=opts->x_svq|| (has_cvq&&vhost_vdpa_get_as_num(vdpa_device_fd))So we avoid the
AS_NUMioctl if not needed.
So when !opts->x_svq, we need to check num_as is at least 2?
Quoted 10 lines
Quoted 7 lines
net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 808
Quoted 3 lines
net/vhost-vdpa.c +2
-0+ +if (opts->x_svq || svq_cvq) {Any chance we can have
opts->x_svq= true butsvq_cvq= false? Checkingsvq_cvqseems sufficient here.The reverse is possible, to have
svq_cvqbut noopts->x_svq.Depending on that, this code emits a warning or a fatal error.
Ok, as replied in the previous patch, I think we need a better name for those ones.
if (opts->x_svq) {
shadow_data_vq = true;
if(has_cvq) shadow_cvq = true;
} else if (num_as >= 2 && has_cvq) {
shadow_cvq = true;
}
The other logic can just check shadow_cvq or shadow_data_vq individually.
Quoted 36 lines
Quoted 27 lines
net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 813
Quoted 8 lines
net/vhost-vdpa.c +4
-2+Error *warn =NULL ; -if (!vhost_vdpa_net_valid_svq_features (features, errp)) { - goto err_svq; + svq_cvq =vhost_vdpa_net_valid_svq_features (features, + opts->x_svq ? errp : &warn); +if (!svq_cvq) {Same question as above.
Quoted 15 lines
net/vhost-vdpa.c +9
-0+if (opts->x_svq) { + goto err_svq; + }else { +warn_reportf_err (warn,"Cannot shadow CVQ: " ); + } } + } + +if (opts->x_svq || svq_cvq) { +struct vhost_vdpa_iova_range iova_range;vhost_vdpa_get_iova_range (vdpa_device_fd, &iova_range); iova_tree =vhost_iova_tree_new (iova_range.first, iova_range.last);@@ -708,15 +830,15 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,Quoted 9 lines
net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 833
Quoted 5 lines
net/vhost-vdpa.c +1
-2for (i =0 ; i < queue_pairs; i++) { ncs[i] = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name, - vdpa_device_fd, i,2 , true, opts->x_svq, - iova_tree); + vdpa_device_fd, i, 2, num_as, true,I don't get why we need pass
num_asto a specificvhost_vdpastructure. It should be sufficient to pass asid there.
Quoted 4 lines
ASID is not known at this time, but at device's start. This is because we cannot ask if the CVQ is in its own vq group, because we don't know the control virtqueue index until the guest acknowledges the different features.
We can probe those during initialization I think. E.g doing some negotiation in the initialization phase.
Thanks
Quoted 18 lines
Thanks!
[1] https://www.mail-archive.com/[email protected]/msg901685.html
Quoted 12 lines
Quoted 12 lines
net/vhost-vdpa.c +2
-1+ opts->x_svq, iova_tree);if (!ncs[i]) goto err; }if (has_cvq) { nc =net_vhost_vdpa_init (peer,TYPE_VHOST_VDPA , name, - vdpa_device_fd, i,1 , false, + vdpa_device_fd, i,1 , num_as, false, opts->x_svq, iova_tree);if (!nc) goto err;
Re: [PATCH v6 10/10] vdpa: Always start CVQ in SVQ mode
Quoted 12 lines · Jason Wang
Quoted 9 lines · Eugenio Perez Martin
Quoted 8 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez
Isolate control virtqueue in its own group, allowing to intercept control commands but letting dataplane run totally passthrough to the guest.
I think we need to tweak the title to "vdpa: Always start CVQ in SVQ mode if possible". Since SVQ for CVQ can't be enabled without ASID support?
Quoted 7 lines
Quoted 1 line
Yes, I totally agree.
Btw, I wonder if it's worth to remove the "x-" prefix for the shadow virtqueue. It can help for the devices without ASID support but want do live migration.
Sure I can propose on top.
Quoted 93 lines
Quoted 90 lines
Quoted 50 lines
Quoted 50 lines
Signed-off-by Eugenio Pérez <[email protected]>---
v6:
* Disable control SVQ if the device does not support it because of features.v5:
* Fixing the not adding cvq buffers when x-svq=on is specified.
* Move vring state invhost_vdpa_get_vring_groupinstead of using a parameter.
* RenameVHOST_VDPA_NET_CVQ_PASSTHROUGHtoVHOST_VDPA_NET_DATA_ASIDv4:
* Squashvhost_vdpa_cvq_group_is_independent.
* Rebased on last CVQ start series, that allocated CVQ cmd bufs at load
* Do not check for cvq index onvhost_vdpa_net_prepare, we only have one that callback registered in that NetClientInfo.v3:
* Make asid related queries print a warning instead of returning an error and stop the start of qemu.--- hw/virtio/vhost-vdpa.c | 3 +- net/vhost-vdpa.c | 138 ++++++++++++++++++++++++++++++++++++++--- 2 files changed, 132 insertions(+), 9 deletions(-)hw/virtio/vhost-vdpa.c +2
-1@@ -648,7 +648,8 @@ static int vhost_vdpa_set_backend_cap(struct vhost_dev *dev) {uint64_t features;uint64_t f =0x1ULL <<VHOST_BACKEND_F_IOTLB_MSG_V2 | -0x1ULL <<VHOST_BACKEND_F_IOTLB_BATCH ; +0x1ULL <<VHOST_BACKEND_F_IOTLB_BATCH | +0x1ULL <<VHOST_BACKEND_F_IOTLB_ASID ;int r;if (vhost_vdpa_call (dev,VHOST_GET_BACKEND_FEATURES , &features)) {net/vhost-vdpa.c +0
-0@@ -38,6 +38,9 @@ typedef struct VhostVDPAState {void *cvq_cmd_out_buffer;virtio_net_ctrl_ack *status;Quoted 7 lines
net/vhost-vdpa.c typedef struct VhostVDPAState { · line 42
Quoted 2 lines
net/vhost-vdpa.c +2
-0+/* Number of address spaces supported by the device */ +unsigned address_space_num;I'm not sure this is the best place to store thing like this since it can cause confusion. We will have multiple VhostVDPAState when multiqueue is enabled.
I think we can delete this and ask it on each device start.
Quoted 18 lines
Quoted 18 lines
net/vhost-vdpa.c +4
-0+/* The device always have SVQ enabled */ bool always_svq;bool started;@@ -101,6 +104,9 @@ static const uint64_t vdpa_svq_device_features =BIT_ULL (VIRTIO_NET_F_RSC_EXT ) |BIT_ULL (VIRTIO_NET_F_STANDBY ); +#define VHOST_VDPA_NET_DATA_ASID 0 +#define VHOST_VDPA_NET_CVQ_ASID 1 +VHostNetState *vhost_vdpa_get_vhost_net (NetClientState *nc) {VhostVDPAState *s =DO_UPCAST (VhostVDPAState, nc, nc);@@ -242,6 +248,34 @@ static NetClientInfo net_vhost_vdpa_info = { .check_peer_type = vhost_vdpa_check_peer_type, };Quoted 13 lines
net/vhost-vdpa.c static NetClientInfo net_vhost_vdpa_info = { · line 258
Quoted 8 lines
net/vhost-vdpa.c +8
-0+static uint32_t vhost_vdpa_get_vring_group (int device_fd,unsigned vq_index) +{ +struct vhost_vring_state state = { + .index = vq_index, + }; +int r =ioctl (device_fd,VHOST_VDPA_GET_VRING_GROUP , &state); + +return r <0 ?0 : state.num;Assume 0 when
ioctl()fail is probably not a good idea: errors in ioctl might be hidden. It would be better to fallback to 0 when ASID is not supported.Quoted 1 line
Did I misunderstand you on [1]?
Quoted 8 lines
Nope. I think I was wrong at that time then :( Sorry for that.
We should differ from the case
1) no ASID support so 0 is assumed
2) something wrong in the case of ioctl, it's not necessarily a ENOTSUPP.
What action should we take here? Isn't it better to disable SVQ and let the device run?
Quoted 58 lines
Quoted 58 lines
Quoted 58 lines
Quoted 44 lines
net/vhost-vdpa.c +30
-1+} + +static int vhost_vdpa_set_address_space_id (struct vhost_vdpa *v, +unsigned vq_group, +unsigned asid_num) +{ +struct vhost_vring_state asid = { + .index = vq_group, + .num = asid_num, + }; +int ret; + + ret =ioctl (v->device_fd,VHOST_VDPA_SET_GROUP_ASID , &asid); +if (unlikely (ret <0 )) { +warn_report ("Can't set vq group %u asid %u, errno=%d (%s)" , + asid.index, asid.num, errno,g_strerror (errno)); + } +return ret; +} +static void vhost_vdpa_cvq_unmap_buf (struct vhost_vdpa *v,void *addr) {VhostIOVATree *tree = v->iova_tree;@@ -316,11 +350,54 @@ dma_map_err:static int vhost_vdpa_net_cvq_start (NetClientState *nc) {VhostVDPAState *s; -int r; +struct vhost_vdpa *v; +uint32_t cvq_group; +int cvq_index, r;assert (nc->info->type ==NET_CLIENT_DRIVER_VHOST_VDPA ); s =DO_UPCAST (VhostVDPAState, nc, nc); + v = &s->vhost_vdpa; + + v->listener_shadow_vq = s->always_svq; + v->shadow_vqs_enabled = s->always_svq; + s->vhost_vdpa.address_space_id =VHOST_VDPA_NET_DATA_ASID ; + +if (s->always_svq) {net/vhost-vdpa.c dma_map_err: · line 376
Quoted 10 lines
net/vhost-vdpa.c +10
-0+ goto out; + } + +if (s->address_space_num <2 ) { +return 0 ; + } + +if (!vhost_vdpa_net_valid_svq_features (v->dev->features,NULL )) { +return 0 ; + }Any reason we do the above check during the start/stop? It should be easier to do that in the initialization.
Quoted 9 lines
Quoted 2 lines
We can store it as a member of VhostVDPAState maybe? They will be duplicated like the current number of AS.
I meant each VhostVDPAState just need to know the ASID it needs to use. There's no need to know the total number of address spaces or do the validation on it during start (the validation could be done during initialization).
I thought we were talking about the virtio features.
So let's omit this check and simply try to set ASID? The worst case is an -ENOTSUPP or -EINVAL, so the actions to take are the same as if we don't have enough AS.
Quoted 115 lines
Quoted 104 lines
Quoted 47 lines
Quoted 47 lines
net/vhost-vdpa.c +38
-0+ +/** +* Check if all the virtqueues of the virtio device are in a different vq +* than the last vq. VQ group of last group passed in cvq_group. +*/ + cvq_index = v->dev->vq_index_end -1 ; + cvq_group =vhost_vdpa_get_vring_group (v->device_fd, cvq_index); +for (int i =0 ; i < cvq_index; ++i) { +uint32_t group =vhost_vdpa_get_vring_group (v->device_fd, i); + +if (unlikely (group == cvq_group)) { +warn_report ("CVQ %u group is the same as VQ %u one (%u)" , cvq_group, + i, group); +return 0 ; + } + } + + r =vhost_vdpa_set_address_space_id (v, cvq_group,VHOST_VDPA_NET_CVQ_ASID ); +if (r ==0 ) { + v->shadow_vqs_enabled = true; + s->vhost_vdpa.address_space_id =VHOST_VDPA_NET_CVQ_ASID ; + } + +out:if (!s->vhost_vdpa.shadow_vqs_enabled) {return 0 ; }@@ -542,12 +619,38 @@ static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = { .avail_handler = vhost_vdpa_net_handle_ctrl_avail, }; +static uint32_t vhost_vdpa_get_as_num (int vdpa_device_fd) +{ +uint64_t features; +unsigned num_as; +int r; + + r =ioctl (vdpa_device_fd,VHOST_GET_BACKEND_FEATURES , &features); +if (unlikely (r <0 )) { +warn_report ("Cannot get backend features" ); +return 1 ; + } + +if (!(features &BIT_ULL (VHOST_BACKEND_F_IOTLB_ASID ))) { +return 1 ;Quoted 9 lines
net/vhost-vdpa.c static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = { · line 641
Quoted 6 lines
net/vhost-vdpa.c +6
-0+ } + + r =ioctl (vdpa_device_fd,VHOST_VDPA_GET_AS_NUM , &num_as); +if (unlikely (r <0 )) { +warn_report ("Cannot retrieve number of supported ASs" ); +return 1 ;Let's return error here. This help. to identify bugs of qemu or kernel.
Same comment as with
VHOST_VDPA_GET_VRING_GROUP.Quoted 46 lines
Quoted 33 lines
net/vhost-vdpa.c +9
-0+ } + +return num_as; +} +static NetClientState *net_vhost_vdpa_init (NetClientState *peer,const char *device,const char *name,int vdpa_device_fd,int queue_pair_index,int nvqs, +unsigned nas,bool is_datapath,bool svq,VhostIOVATree *iova_tree)@@ -566,6 +669,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,qemu_set_info_str (nc,TYPE_VHOST_VDPA ); s =DO_UPCAST (VhostVDPAState, nc, nc); + s->address_space_num = nas; s->vhost_vdpa.device_fd = vdpa_device_fd; s->vhost_vdpa.index = queue_pair_index; s->always_svq = svq;@@ -652,6 +756,8 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,g_autoptr (VhostIOVATree) iova_tree =NULL ;NetClientState *nc;int queue_pairs, r, i =0 , has_cvq =0 ; +unsigned num_as =1 ; +bool svq_cvq;assert (netdev->type ==NET_CLIENT_DRIVER_VHOST_VDPA ); opts = &netdev->u.vhost_vdpa;net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 806
Quoted 10 lines
net/vhost-vdpa.c +5
-2@@ -693,12 +799,28 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,return queue_pairs; } -if (opts->x_svq) { -struct vhost_vdpa_iova_range iova_range; + svq_cvq = opts->x_svq; +if (has_cvq && !opts->x_svq) { + num_as =vhost_vdpa_get_as_num (vdpa_device_fd); + svq_cvq = num_as >1 ; + }The above check is not easy to follow, how about?
Quoted 5 lines
Quoted 2 lines
svq_cvq=vhost_vdpa_get_as_num()> 1 ? true :opts->x_svq;That would allocate the iova tree even if CVQ is not used in the guest. And
num_asis reused later, although we can ask it to the device at device start to avoid this.Ok.
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
If any, the linear conversion would be:
svq_cvq=opts->x_svq|| (has_cvq&&vhost_vdpa_get_as_num(vdpa_device_fd))So we avoid the
AS_NUMioctl if not needed.So when !opts->x_svq, we need to check
num_asis at least 2?
As I think you proposed, we can simply try to set CVQ ASID and react to -EINVAL.
But this code here is trying to not to allocate iova_tree if we're sure it will not be needed. Maybe it is easier to always allocate the empty iova tree and only fill it if needed?
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 808
Quoted 4 lines
net/vhost-vdpa.c +2
-0+ +if (opts->x_svq || svq_cvq) {Any chance we can have
opts->x_svq= true butsvq_cvq= false? Checkingsvq_cvqseems sufficient here.
Quoted 8 lines
Quoted 3 lines
The reverse is possible, to have
svq_cvqbut noopts->x_svq.Depending on that, this code emits a warning or a fatal error.
Ok, as replied in the previous patch, I think we need a better name for those ones.
cvq_svq can be renamed for sure. x_svq can be aliased with other variable if needed too.
Quoted 9 lines
if (opts->x_svq) { shadow_data_vq = true; if(has_cvq) shadow_cvq = true; } else if (num_as >= 2 && has_cvq) { shadow_cvq = true; }The other logic can just check
shadow_cvqorshadow_data_vqindividually.
Not sure if shadow_data_vq is accurate. It sounds to me as "Only shadow data virtqueues but not CVQ".
shadow_device and shadow_cvq?
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 813
Quoted 9 lines
net/vhost-vdpa.c +4
-2+Error *warn =NULL ; -if (!vhost_vdpa_net_valid_svq_features (features, errp)) { - goto err_svq; + svq_cvq =vhost_vdpa_net_valid_svq_features (features, + opts->x_svq ? errp : &warn); +if (!svq_cvq) {Same question as above.
Quoted 14 lines
net/vhost-vdpa.c +9
-0+if (opts->x_svq) { + goto err_svq; + }else { +warn_reportf_err (warn,"Cannot shadow CVQ: " ); + } } + } + +if (opts->x_svq || svq_cvq) { +struct vhost_vdpa_iova_range iova_range;vhost_vdpa_get_iova_range (vdpa_device_fd, &iova_range); iova_tree =vhost_iova_tree_new (iova_range.first, iova_range.last);net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 833
Quoted 6 lines
net/vhost-vdpa.c +1
-2@@ -708,15 +830,15 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,for (i =0 ; i < queue_pairs; i++) { ncs[i] = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name, - vdpa_device_fd, i,2 , true, opts->x_svq, - iova_tree); + vdpa_device_fd, i, 2, num_as, true,I don't get why we need pass
num_asto a specificvhost_vdpastructure. It should be sufficient to pass asid there.
Quoted 9 lines
Quoted 4 lines
ASID is not known at this time, but at device's start. This is because we cannot ask if the CVQ is in its own vq group, because we don't know the control virtqueue index until the guest acknowledges the different features.
We can probe those during initialization I think. E.g doing some negotiation in the initialization phase.
We've developed this idea in other threads, let's continue there better.
Thanks!
Quoted 22 lines
Thanks
Quoted 18 lines
Thanks!
[1] https://www.mail-archive.com/[email protected]/msg901685.html
Quoted 12 lines
Quoted 12 lines
net/vhost-vdpa.c +2
-1+ opts->x_svq, iova_tree);if (!ncs[i]) goto err; }if (has_cvq) { nc =net_vhost_vdpa_init (peer,TYPE_VHOST_VDPA , name, - vdpa_device_fd, i,1 , false, + vdpa_device_fd, i,1 , num_as, false, opts->x_svq, iova_tree);if (!nc) goto err;
Re: [PATCH v6 10/10] vdpa: Always start CVQ in SVQ mode
Quoted 108 lines · Eugenio Perez Martin
Quoted 6 lines · Jason Wang
Quoted 4 lines · Eugenio Perez Martin
Quoted 3 lines · Jason Wang
Quoted 2 lines · Eugenio Pérez
Isolate control virtqueue in its own group, allowing to intercept control commands but letting dataplane run totally passthrough to the guest.
Quoted 9 lines
Quoted 4 lines
Quoted 3 lines
I think we need to tweak the title to "vdpa: Always start CVQ in SVQ mode if possible". Since SVQ for CVQ can't be enabled without ASID support?
Yes, I totally agree.
Btw, I wonder if it's worth to remove the "x-" prefix for the shadow virtqueue. It can help for the devices without ASID support but want do live migration.
Sure I can propose on top.
Quoted 90 lines
Quoted 86 lines
Quoted 48 lines
Quoted 48 lines
Signed-off-by Eugenio Pérez <[email protected]>---
v6:
* Disable control SVQ if the device does not support it because of features.v5:
* Fixing the not adding cvq buffers when x-svq=on is specified.
* Move vring state invhost_vdpa_get_vring_groupinstead of using a parameter.
* RenameVHOST_VDPA_NET_CVQ_PASSTHROUGHtoVHOST_VDPA_NET_DATA_ASIDv4:
* Squashvhost_vdpa_cvq_group_is_independent.
* Rebased on last CVQ start series, that allocated CVQ cmd bufs at load
* Do not check for cvq index onvhost_vdpa_net_prepare, we only have one that callback registered in that NetClientInfo.v3:
* Make asid related queries print a warning instead of returning an error and stop the start of qemu.--- hw/virtio/vhost-vdpa.c | 3 +- net/vhost-vdpa.c | 138 ++++++++++++++++++++++++++++++++++++++--- 2 files changed, 132 insertions(+), 9 deletions(-)hw/virtio/vhost-vdpa.c +2
-1@@ -648,7 +648,8 @@ static int vhost_vdpa_set_backend_cap(struct vhost_dev *dev) {uint64_t features;uint64_t f =0x1ULL <<VHOST_BACKEND_F_IOTLB_MSG_V2 | -0x1ULL <<VHOST_BACKEND_F_IOTLB_BATCH ; +0x1ULL <<VHOST_BACKEND_F_IOTLB_BATCH | +0x1ULL <<VHOST_BACKEND_F_IOTLB_ASID ;int r;if (vhost_vdpa_call (dev,VHOST_GET_BACKEND_FEATURES , &features)) {net/vhost-vdpa.c +0
-0@@ -38,6 +38,9 @@ typedef struct VhostVDPAState {void *cvq_cmd_out_buffer;virtio_net_ctrl_ack *status;Quoted 6 lines
net/vhost-vdpa.c typedef struct VhostVDPAState { · line 42
Quoted 2 lines
net/vhost-vdpa.c +2
-0+/* Number of address spaces supported by the device */ +unsigned address_space_num;I'm not sure this is the best place to store thing like this since it can cause confusion. We will have multiple VhostVDPAState when multiqueue is enabled.
I think we can delete this and ask it on each device start.
Quoted 18 lines
Quoted 18 lines
net/vhost-vdpa.c +4
-0+/* The device always have SVQ enabled */ bool always_svq;bool started;@@ -101,6 +104,9 @@ static const uint64_t vdpa_svq_device_features =BIT_ULL (VIRTIO_NET_F_RSC_EXT ) |BIT_ULL (VIRTIO_NET_F_STANDBY ); +#define VHOST_VDPA_NET_DATA_ASID 0 +#define VHOST_VDPA_NET_CVQ_ASID 1 +VHostNetState *vhost_vdpa_get_vhost_net (NetClientState *nc) {VhostVDPAState *s =DO_UPCAST (VhostVDPAState, nc, nc);@@ -242,6 +248,34 @@ static NetClientInfo net_vhost_vdpa_info = { .check_peer_type = vhost_vdpa_check_peer_type, };Quoted 12 lines
net/vhost-vdpa.c static NetClientInfo net_vhost_vdpa_info = { · line 258
Quoted 8 lines
net/vhost-vdpa.c +8
-0+static uint32_t vhost_vdpa_get_vring_group (int device_fd,unsigned vq_index) +{ +struct vhost_vring_state state = { + .index = vq_index, + }; +int r =ioctl (device_fd,VHOST_VDPA_GET_VRING_GROUP , &state); + +return r <0 ?0 : state.num;Assume 0 when
ioctl()fail is probably not a good idea: errors in ioctl might be hidden. It would be better to fallback to 0 when ASID is not supported.Quoted 1 line
Did I misunderstand you on [1]?
Nope. I think I was wrong at that time then :( Sorry for that.
Quoted 8 lines
Quoted 6 lines
We should differ from the case
1) no ASID support so 0 is assumed
2) something wrong in the case of ioctl, it's not necessarily a ENOTSUPP.
What action should we take here? Isn't it better to disable SVQ and let the device run?
It should fail the function instead of assuming 0 like how other vhost ioctl work.
Quoted 64 lines
Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
Quoted 43 lines
net/vhost-vdpa.c +30
-1+} + +static int vhost_vdpa_set_address_space_id (struct vhost_vdpa *v, +unsigned vq_group, +unsigned asid_num) +{ +struct vhost_vring_state asid = { + .index = vq_group, + .num = asid_num, + }; +int ret; + + ret =ioctl (v->device_fd,VHOST_VDPA_SET_GROUP_ASID , &asid); +if (unlikely (ret <0 )) { +warn_report ("Can't set vq group %u asid %u, errno=%d (%s)" , + asid.index, asid.num, errno,g_strerror (errno)); + } +return ret; +} +static void vhost_vdpa_cvq_unmap_buf (struct vhost_vdpa *v,void *addr) {VhostIOVATree *tree = v->iova_tree;@@ -316,11 +350,54 @@ dma_map_err:static int vhost_vdpa_net_cvq_start (NetClientState *nc) {VhostVDPAState *s; -int r; +struct vhost_vdpa *v; +uint32_t cvq_group; +int cvq_index, r;assert (nc->info->type ==NET_CLIENT_DRIVER_VHOST_VDPA ); s =DO_UPCAST (VhostVDPAState, nc, nc); + v = &s->vhost_vdpa; + + v->listener_shadow_vq = s->always_svq; + v->shadow_vqs_enabled = s->always_svq; + s->vhost_vdpa.address_space_id =VHOST_VDPA_NET_DATA_ASID ; + +if (s->always_svq) {net/vhost-vdpa.c dma_map_err: · line 376
Quoted 10 lines
net/vhost-vdpa.c +10
-0+ goto out; + } + +if (s->address_space_num <2 ) { +return 0 ; + } + +if (!vhost_vdpa_net_valid_svq_features (v->dev->features,NULL )) { +return 0 ; + }Any reason we do the above check during the start/stop? It should be easier to do that in the initialization.
Quoted 8 lines
Quoted 2 lines
We can store it as a member of VhostVDPAState maybe? They will be duplicated like the current number of AS.
I meant each VhostVDPAState just need to know the ASID it needs to use. There's no need to know the total number of address spaces or do the validation on it during start (the validation could be done during initialization).
Quoted 5 lines
I thought we were talking about the virtio features.
So let's omit this check and simply try to set ASID? The worst case is an
-ENOTSUPPor-EINVAL, so the actions to take are the same as if we don't have enough AS.
See the discussion in other thread:
1) do the probing of asid stuffs during initalization, we can simply try to set the ASID here and fail the vhost dev start
2) do the probing each time during vhost start
Personally I like 1, but if you want to go with 2 it should also fine. (need some comments)
Quoted 115 lines
Quoted 115 lines
Quoted 101 lines
Quoted 45 lines
Quoted 45 lines
net/vhost-vdpa.c +37
-0+ +/** +* Check if all the virtqueues of the virtio device are in a different vq +* than the last vq. VQ group of last group passed in cvq_group. +*/ + cvq_index = v->dev->vq_index_end -1 ; + cvq_group =vhost_vdpa_get_vring_group (v->device_fd, cvq_index); +for (int i =0 ; i < cvq_index; ++i) { +uint32_t group =vhost_vdpa_get_vring_group (v->device_fd, i); + +if (unlikely (group == cvq_group)) { +warn_report ("CVQ %u group is the same as VQ %u one (%u)" , cvq_group, + i, group); +return 0 ; + } + } + + r =vhost_vdpa_set_address_space_id (v, cvq_group,VHOST_VDPA_NET_CVQ_ASID ); +if (r ==0 ) { + v->shadow_vqs_enabled = true; + s->vhost_vdpa.address_space_id =VHOST_VDPA_NET_CVQ_ASID ; + } + +out:if (!s->vhost_vdpa.shadow_vqs_enabled) {return 0 ; }@@ -542,12 +619,38 @@ static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = { .avail_handler = vhost_vdpa_net_handle_ctrl_avail, }; +static uint32_t vhost_vdpa_get_as_num (int vdpa_device_fd) +{ +uint64_t features; +unsigned num_as; +int r; + + r =ioctl (vdpa_device_fd,VHOST_GET_BACKEND_FEATURES , &features); +if (unlikely (r <0 )) { +warn_report ("Cannot get backend features" ); +return 1 ; + } + +if (!(features &BIT_ULL (VHOST_BACKEND_F_IOTLB_ASID ))) {Quoted 9 lines
net/vhost-vdpa.c static const VhostShadowVirtqueueOps vhost_vdpa_net_svq_ops = { · line 641
Quoted 7 lines
net/vhost-vdpa.c +7
-0+return 1 ; + } + + r =ioctl (vdpa_device_fd,VHOST_VDPA_GET_AS_NUM , &num_as); +if (unlikely (r <0 )) { +warn_report ("Cannot retrieve number of supported ASs" ); +return 1 ;Let's return error here. This help. to identify bugs of qemu or kernel.
Same comment as with
VHOST_VDPA_GET_VRING_GROUP.Quoted 45 lines
Quoted 33 lines
net/vhost-vdpa.c +9
-0+ } + +return num_as; +} +static NetClientState *net_vhost_vdpa_init (NetClientState *peer,const char *device,const char *name,int vdpa_device_fd,int queue_pair_index,int nvqs, +unsigned nas,bool is_datapath,bool svq,VhostIOVATree *iova_tree)@@ -566,6 +669,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,qemu_set_info_str (nc,TYPE_VHOST_VDPA ); s =DO_UPCAST (VhostVDPAState, nc, nc); + s->address_space_num = nas; s->vhost_vdpa.device_fd = vdpa_device_fd; s->vhost_vdpa.index = queue_pair_index; s->always_svq = svq;@@ -652,6 +756,8 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,g_autoptr (VhostIOVATree) iova_tree =NULL ;NetClientState *nc;int queue_pairs, r, i =0 , has_cvq =0 ; +unsigned num_as =1 ; +bool svq_cvq;assert (netdev->type ==NET_CLIENT_DRIVER_VHOST_VDPA ); opts = &netdev->u.vhost_vdpa;net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 806
Quoted 10 lines
net/vhost-vdpa.c +5
-2@@ -693,12 +799,28 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,return queue_pairs; } -if (opts->x_svq) { -struct vhost_vdpa_iova_range iova_range; + svq_cvq = opts->x_svq; +if (has_cvq && !opts->x_svq) { + num_as =vhost_vdpa_get_as_num (vdpa_device_fd); + svq_cvq = num_as >1 ; + }The above check is not easy to follow, how about?
Quoted 5 lines
Quoted 2 lines
svq_cvq=vhost_vdpa_get_as_num()> 1 ? true :opts->x_svq;That would allocate the iova tree even if CVQ is not used in the guest. And
num_asis reused later, although we can ask it to the device at device start to avoid this.Ok.
Quoted 4 lines
If any, the linear conversion would be:
svq_cvq=opts->x_svq|| (has_cvq&&vhost_vdpa_get_as_num(vdpa_device_fd))So we avoid the
AS_NUMioctl if not needed.
Quoted 7 lines
Quoted 2 lines
So when !opts->x_svq, we need to check
num_asis at least 2?As I think you proposed, we can simply try to set CVQ ASID and react to
-EINVAL.But this code here is trying to not to allocate
iova_treeif we're sure it will not be needed. Maybe it is easier to always allocate the empty iova tree and only fill it if needed?
I feel it should work.
Quoted 23 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 808
Quoted 3 lines
net/vhost-vdpa.c +2
-0+ +if (opts->x_svq || svq_cvq) {Any chance we can have
opts->x_svq= true butsvq_cvq= false? Checkingsvq_cvqseems sufficient here.Quoted 7 lines
Quoted 3 lines
The reverse is possible, to have
svq_cvqbut noopts->x_svq.Depending on that, this code emits a warning or a fatal error.
Ok, as replied in the previous patch, I think we need a better name for those ones.
cvq_svqcan be renamed for sure.x_svqcan be aliased with other variable if needed too.Quoted 7 lines
if (opts->x_svq) { shadow_data_vq = true; if(has_cvq) shadow_cvq = true; } else if (num_as >= 2 && has_cvq) { shadow_cvq = true; }
Quoted 6 lines
Quoted 2 lines
The other logic can just check
shadow_cvqorshadow_data_vqindividually.Not sure if
shadow_data_vqis accurate. It sounds to me as "Only shadow data virtqueues but not CVQ".
shadow_deviceandshadow_cvq?
Should work, let's see.
Thanks
Quoted 66 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 813
Quoted 8 lines
net/vhost-vdpa.c +4
-2+Error *warn =NULL ; -if (!vhost_vdpa_net_valid_svq_features (features, errp)) { - goto err_svq; + svq_cvq =vhost_vdpa_net_valid_svq_features (features, + opts->x_svq ? errp : &warn); +if (!svq_cvq) {Same question as above.
Quoted 14 lines
net/vhost-vdpa.c +9
-0+if (opts->x_svq) { + goto err_svq; + }else { +warn_reportf_err (warn,"Cannot shadow CVQ: " ); + } } + } + +if (opts->x_svq || svq_cvq) { +struct vhost_vdpa_iova_range iova_range;vhost_vdpa_get_iova_range (vdpa_device_fd, &iova_range); iova_tree =vhost_iova_tree_new (iova_range.first, iova_range.last);net/vhost-vdpa.c int net_init_vhost_vdpa(const Netdev *netdev, const char *name, · line 833
Quoted 6 lines
net/vhost-vdpa.c +1
-2@@ -708,15 +830,15 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,for (i =0 ; i < queue_pairs; i++) { ncs[i] = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name, - vdpa_device_fd, i,2 , true, opts->x_svq, - iova_tree); + vdpa_device_fd, i, 2, num_as, true,I don't get why we need pass
num_asto a specificvhost_vdpastructure. It should be sufficient to pass asid there.Quoted 8 lines
Quoted 4 lines
ASID is not known at this time, but at device's start. This is because we cannot ask if the CVQ is in its own vq group, because we don't know the control virtqueue index until the guest acknowledges the different features.
We can probe those during initialization I think. E.g doing some negotiation in the initialization phase.
We've developed this idea in other threads, let's continue there better.
Thanks!
Quoted 20 lines
Thanks
Quoted 17 lines
Thanks!
[1] https://www.mail-archive.com/[email protected]/msg901685.html
Quoted 12 lines
Quoted 12 lines
net/vhost-vdpa.c +2
-1+ opts->x_svq, iova_tree);if (!ncs[i]) goto err; }if (has_cvq) { nc =net_vhost_vdpa_init (peer,TYPE_VHOST_VDPA , name, - vdpa_device_fd, i,1 , false, + vdpa_device_fd, i,1 , num_as, false, opts->x_svq, iova_tree);if (!nc) goto err;
Quoted 8 lines · Eugenio Pérez
Control VQ is the way net devices use to send changes to the device state, like the number of active queues or its mac address.
QEMU needs to intercept this queue so it can track these changes and is able to migrate the device. It can do it from
1576dbb5bbc4("vdpa: Add x-svq to NetdevVhostVDPAOptions"). However, to enable x-svq implies to shadow all VirtIO device's virtqueues, which will damage performance.
Quoted 6 lines
This series adds address space isolation, so the device and the guest communicate directly with them (passthrough) and CVQ communication is split in
two: The guest communicates with QEMU and QEMU forwards the commands to the device.Comments are welcome. Thanks!
This is not 7.2 material, right?
Quoted 51 lines
v6:
- Do not allocate SVQ resources like file descriptors if SVQ cannot be used.
- Disable shadow CVQ if the device does not support it because of net features.v5:
- Move vring state invhost_vdpa_get_vring_groupinstead of using a parameter.
- RenameVHOST_VDPA_NET_CVQ_PASSTHROUGHtoVHOST_VDPA_NET_DATA_ASIDv4:
- Rebased on last CVQ start series, that allocated CVQ cmd bufs at load
- Squashvhost_vdpa_cvq_group_is_independent.
- Do not check for cvq index onvhost_vdpa_net_prepare, we only have one that callback registered in that NetClientInfo.
- Add comment specifying behavior if device does not support_F_ASID
- Update headers to a later Linux commit to not to removeSETUP_RNG_SEEDv3:
- Do not return an error but just print a warning if vdpa device initialization returns failure while getting AS num of VQ groups
- Delete extra newlinev2:
- Much as commented on series [1], handlevhost_netbackend through NetClientInfo callbacks instead of directly.
- Fix not freeing SVQ properly when device does not support CVQ
- AddBIT_ULLmissed checking device's backend feature for_F_ASID.Eugenio Pérez (10):
vdpa: Usev->shadow_vqs_enabledinvhost_vdpa_svqs_start& stop
vhost: set SVQ device call handler at SVQ start
vhost: Allocate SVQ device file descriptors at device start
vdpa: addvhost_vdpa_net_valid_svq_features
vdpa: move SVQ vring features check tonet/
vdpa: Allocate SVQ unconditionally
vdpa: Add asid parameter to vhost_vdpa_dma_map/unmap
vdpa: Store x-svq parameter in VhostVDPAState
vdpa: Addlistener_shadow_vqtovhost_vdpa
vdpa: Always start CVQ in SVQ modeinclude/hw/virtio/vhost-vdpa.h | 10 +- hw/virtio/vhost-shadow-virtqueue.c | 35 +----- hw/virtio/vhost-vdpa.c | 114 ++++++++++--------- net/vhost-vdpa.c | 171 ++++++++++++++++++++++++++--- hw/virtio/trace-events | 4 +- 5 files changed, 222 insertions(+), 112 deletions(-)Signature · 1 line
2.31.1
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Eugenio Pérez
Control VQ is the way net devices use to send changes to the device state, like the number of active queues or its mac address.
QEMU needs to intercept this queue so it can track these changes and is able to migrate the device. It can do it from
1576dbb5bbc4("vdpa: Add x-svq to NetdevVhostVDPAOptions"). However, to enable x-svq implies to shadow all VirtIO device's virtqueues, which will damage performance.This series adds address space isolation, so the device and the guest communicate directly with them (passthrough) and CVQ communication is split in
two: The guest communicates with QEMU and QEMU forwards the commands to the device.
Quoted 5 lines
Quoted 1 line
Comments are welcome. Thanks!
This is not 7.2 material, right?
No it is not.
I should have noted it after PATCH in the subject, sorry.
Quoted 52 lines
Quoted 51 lines
v6:
- Do not allocate SVQ resources like file descriptors if SVQ cannot be used.
- Disable shadow CVQ if the device does not support it because of net features.v5:
- Move vring state invhost_vdpa_get_vring_groupinstead of using a parameter.
- RenameVHOST_VDPA_NET_CVQ_PASSTHROUGHtoVHOST_VDPA_NET_DATA_ASIDv4:
- Rebased on last CVQ start series, that allocated CVQ cmd bufs at load
- Squashvhost_vdpa_cvq_group_is_independent.
- Do not check for cvq index onvhost_vdpa_net_prepare, we only have one that callback registered in that NetClientInfo.
- Add comment specifying behavior if device does not support_F_ASID
- Update headers to a later Linux commit to not to removeSETUP_RNG_SEEDv3:
- Do not return an error but just print a warning if vdpa device initialization returns failure while getting AS num of VQ groups
- Delete extra newlinev2:
- Much as commented on series [1], handlevhost_netbackend through NetClientInfo callbacks instead of directly.
- Fix not freeing SVQ properly when device does not support CVQ
- AddBIT_ULLmissed checking device's backend feature for_F_ASID.Eugenio Pérez (10):
vdpa: Usev->shadow_vqs_enabledinvhost_vdpa_svqs_start& stop
vhost: set SVQ device call handler at SVQ start
vhost: Allocate SVQ device file descriptors at device start
vdpa: addvhost_vdpa_net_valid_svq_features
vdpa: move SVQ vring features check tonet/
vdpa: Allocate SVQ unconditionally
vdpa: Add asid parameter to vhost_vdpa_dma_map/unmap
vdpa: Store x-svq parameter in VhostVDPAState
vdpa: Addlistener_shadow_vqtovhost_vdpa
vdpa: Always start CVQ in SVQ modeinclude/hw/virtio/vhost-vdpa.h | 10 +- hw/virtio/vhost-shadow-virtqueue.c | 35 +----- hw/virtio/vhost-vdpa.c | 114 ++++++++++--------- net/vhost-vdpa.c | 171 ++++++++++++++++++++++++++--- hw/virtio/trace-events | 4 +- 5 files changed, 222 insertions(+), 112 deletions(-)--
2.31.1