Skip to contentSkip to conversation

Fix expected set_vq_state behavior on vdpa_sim

PATCH In review Eugenio Pérez 4 participants 17 messages Jan 18, 2023 – Feb 13, 2023

Subsystems (2)
lore
Share & tools
lore

Eugenio Pérez

author maintainer
#0 lore raw

cover Fix expected set_vq_state behavior on vdpa_sim

The use of set_vq_state is to indicate vdpa device the state of a virtqueue. In the case of split, it means the avail_idx. This is mandatory for use cases like live migration.

However, vdpa_sim reset the vq state at vdpasim_queue_ready since it calls vringh_init_iotlb.

Also, to starting from an used_idx different than 0 is needed in use cases like virtual machine migration. Not doing so and letting the caller set an avail idx different than 0 causes destination device to try to use old buffers that source driver already recover and are not available anymore.

This series fixes both problems allowing to migrate to a vdpa_sim_net device.

Eugenio Pérez (2):
vdpa_sim: not reset state in vdpasim_queue_ready
vringh: fetch used_idx from vring at vringh_init_iotlb

 drivers/vdpa/vdpa_sim/vdpa_sim.c |  2 ++
 drivers/vhost/vringh.c           | 25 +++++++++++++++++++++++--
 2 files changed, 25 insertions(+), 2 deletions(-)
Signature · 1 line
2.31.1
author maintainer
#1 lore raw

PATCH 1/2 vdpa_sim: not reset state in vdpasim_queue_ready

vdpasim_queue_ready calls vringh_init_iotlb, which resets split indexes. But it can be called after setting a ring base with vdpasim_set_vq_state.

Fix it by stashing them. They're still resetted in vdpasim_vq_reset.

This was discovered and tested live migrating the vdpa_sim_net device.

Fixes 2c53d0f64c06 ("vdpasim: vDPA device simulator")
---
 drivers/vdpa/vdpa_sim/vdpa_sim.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/vdpa/vdpa_sim/vdpa_sim.c +2 -0
index cb88891b44a8..8839232a3fcb 100644
@@ -66,6 +66,7 @@ static void vdpasim_vq_notify(struct vringh *vring)
 static void vdpasim_queue_ready(struct vdpasim *vdpasim, unsigned int idx)
 {
 	struct vdpasim_virtqueue *vq = &vdpasim->vqs[idx];
+	uint16_t last_avail_idx = vq->vring.last_avail_idx;
 
 	vringh_init_iotlb(&vq->vring, vdpasim->features, vq->num, false,
 			  (struct vring_desc *)(uintptr_t)vq->desc_addr,
@@ -74,6 +75,7 @@ static void vdpasim_queue_ready(struct vdpasim *vdpasim, unsigned int idx)
 			  (struct vring_used *)
 			  (uintptr_t)vq->device_addr);
 
+	vq->vring.last_avail_idx = last_avail_idx;
 	vq->vring.notify = vdpasim_vq_notify;
 }
 
Signature · 1 line
2.31.1
#2 lore raw

Re: [PATCH 1/2] vdpa_sim: not reset state in vdpasim_queue_ready

Quoted 28 lines · Eugenio Pérez

vdpasim_queue_ready calls vringh_init_iotlb, which resets split indexes. But it can be called after setting a ring base with vdpasim_set_vq_state.

Fix it by stashing them. They're still resetted in vdpasim_vq_reset.

This was discovered and tested live migrating the vdpa_sim_net device.

Fixes 2c53d0f64c06 ("vdpasim: vDPA device simulator")
---
 drivers/vdpa/vdpa_sim/vdpa_sim.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/vdpa/vdpa_sim/vdpa_sim.c +1 -0
index cb88891b44a8..8839232a3fcb 100644
@@ -66,6 +66,7 @@ static void vdpasim_vq_notify(struct vringh *vring)
 static void vdpasim_queue_ready(struct vdpasim *vdpasim, unsigned int idx)
 {
        struct vdpasim_virtqueue *vq = &vdpasim->vqs[idx];
+       uint16_t last_avail_idx = vq->vring.last_avail_idx;
 
        vringh_init_iotlb(&vq->vring, vdpasim->features, vq->num, false,
                          (struct vring_desc *)(uintptr_t)vq->desc_addr,

drivers/vdpa/vdpa_sim/vdpa_sim.c static void vdpasim_queue_ready(struct vdpasim *vdpasim, unsigned int idx) · line 78

Quoted 4 lines
drivers/vdpa/vdpa_sim/vdpa_sim.c +1 -0
@@ -74,6 +75,7 @@ static void vdpasim_queue_ready(struct vdpasim *vdpasim, unsigned int idx)
                          (struct vring_used *)
                          (uintptr_t)vq->device_addr);
 
+       vq->vring.last_avail_idx = last_avail_idx;

Does this need to be serialized with the datapath?

E.g in set_vq_state() we do:

spin_lock(&vdpasim->lock);
vrh->last_avail_idx = state->split.avail_index;
spin_unlock(&vdpasim->lock);

Thanks

Quoted 6 lines
drivers/vdpa/vdpa_sim/vdpa_sim.c +0 -1
        vq->vring.notify = vdpasim_vq_notify;
 }
 
--

2.31.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author maintainer
#3 lore raw

Re: [PATCH 1/2] vdpa_sim: not reset state in vdpasim_queue_ready

Quoted 37 lines · Jason Wang
Quoted 28 lines · Eugenio Pérez

vdpasim_queue_ready calls vringh_init_iotlb, which resets split indexes. But it can be called after setting a ring base with vdpasim_set_vq_state.

Fix it by stashing them. They're still resetted in vdpasim_vq_reset.

This was discovered and tested live migrating the vdpa_sim_net device.

Fixes 2c53d0f64c06 ("vdpasim: vDPA device simulator")
---
 drivers/vdpa/vdpa_sim/vdpa_sim.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/vdpa/vdpa_sim/vdpa_sim.c +1 -0
index cb88891b44a8..8839232a3fcb 100644
@@ -66,6 +66,7 @@ static void vdpasim_vq_notify(struct vringh *vring)
 static void vdpasim_queue_ready(struct vdpasim *vdpasim, unsigned int idx)
 {
        struct vdpasim_virtqueue *vq = &vdpasim->vqs[idx];
+       uint16_t last_avail_idx = vq->vring.last_avail_idx;
 
        vringh_init_iotlb(&vq->vring, vdpasim->features, vq->num, false,
                          (struct vring_desc *)(uintptr_t)vq->desc_addr,

drivers/vdpa/vdpa_sim/vdpa_sim.c static void vdpasim_queue_ready(struct vdpasim *vdpasim, unsigned int idx) · line 78

Quoted 4 lines
drivers/vdpa/vdpa_sim/vdpa_sim.c +1 -0
@@ -74,6 +75,7 @@ static void vdpasim_queue_ready(struct vdpasim *vdpasim, unsigned int idx)
                          (struct vring_used *)
                          (uintptr_t)vq->device_addr);
 
+       vq->vring.last_avail_idx = last_avail_idx;

Does this need to be serialized with the datapath?

Quoted 6 lines

E.g in set_vq_state() we do:

spin_lock(&vdpasim->lock);
vrh->last_avail_idx = state->split.avail_index;
spin_unlock(&vdpasim->lock);

vdpasim_queue_ready is called from vdpasim_set_vq_ready, which holds these locks.

Maybe it's too much indirection and to embed vdpasim_queue_ready in vdpasim_set_vq_ready would be clearer for the future?

Thanks!

Acked-by
#4 lore raw

Re: [PATCH 1/2] vdpa_sim: not reset state in vdpasim_queue_ready

Quoted 41 lines · Eugenio Perez Martin
Quoted 31 lines · Jason Wang
Quoted 30 lines · Eugenio Pérez

vdpasim_queue_ready calls vringh_init_iotlb, which resets split indexes. But it can be called after setting a ring base with vdpasim_set_vq_state.

Fix it by stashing them. They're still resetted in vdpasim_vq_reset.

This was discovered and tested live migrating the vdpa_sim_net device.

Fixes 2c53d0f64c06 ("vdpasim: vDPA device simulator")
---
  drivers/vdpa/vdpa_sim/vdpa_sim.c | 2 ++
  1 file changed, 2 insertions(+)
drivers/vdpa/vdpa_sim/vdpa_sim.c +1 -0
index cb88891b44a8..8839232a3fcb 100644
@@ -66,6 +66,7 @@ static void vdpasim_vq_notify(struct vringh *vring)
  static void vdpasim_queue_ready(struct vdpasim *vdpasim, unsigned int idx)
  {
         struct vdpasim_virtqueue *vq = &vdpasim->vqs[idx];
+       uint16_t last_avail_idx = vq->vring.last_avail_idx;
 
         vringh_init_iotlb(&vq->vring, vdpasim->features, vq->num, false,
                           (struct vring_desc *)(uintptr_t)vq->desc_addr,
@@ -74,6 +75,7 @@ static void vdpasim_queue_ready(struct vdpasim *vdpasim, unsigned int idx)
                           (struct vring_used *)
                           (uintptr_t)vq->device_addr);
 
Quoted 9 lines

drivers/vdpa/vdpa_sim/vdpa_sim.c static void vdpasim_queue_ready(struct vdpasim *vdpasim, unsigned int idx) · line 78

Quoted 1 line
drivers/vdpa/vdpa_sim/vdpa_sim.c +1 -0
+       vq->vring.last_avail_idx = last_avail_idx;

Does this need to be serialized with the datapath?

E.g in set_vq_state() we do:

spin_lock(&vdpasim->lock);
vrh->last_avail_idx = state->split.avail_index;
spin_unlock(&vdpasim->lock);

Quoted 5 lines

vdpasim_queue_ready is called from vdpasim_set_vq_ready, which holds these locks.

Maybe it's too much indirection and to embed vdpasim_queue_ready in vdpasim_set_vq_ready would be clearer for the future?

Nope, I miss the caller.

Thanks

Quoted 3 lines

Thanks!

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Tested-by
#5 lore raw

Re: [PATCH 1/2] vdpa_sim: not reset state in vdpasim_queue_ready

The patch was tested by QE in a test environment and regression tested using vdpa_sim device with virtio_vdpa and vhost_vdpa;There are no new issues caused by this patch.

Quoted 62 lines · Jason Wang <[email protected]> 于2023年1月29日周日 13:56写道:
Quoted 41 lines · Eugenio Perez Martin
Quoted 31 lines · Jason Wang
Quoted 30 lines · Eugenio Pérez

vdpasim_queue_ready calls vringh_init_iotlb, which resets split indexes. But it can be called after setting a ring base with vdpasim_set_vq_state.

Fix it by stashing them. They're still resetted in vdpasim_vq_reset.

This was discovered and tested live migrating the vdpa_sim_net device.

Fixes 2c53d0f64c06 ("vdpasim: vDPA device simulator")
---
  drivers/vdpa/vdpa_sim/vdpa_sim.c | 2 ++
  1 file changed, 2 insertions(+)
drivers/vdpa/vdpa_sim/vdpa_sim.c +1 -0
index cb88891b44a8..8839232a3fcb 100644
@@ -66,6 +66,7 @@ static void vdpasim_vq_notify(struct vringh *vring)
  static void vdpasim_queue_ready(struct vdpasim *vdpasim, unsigned int idx)
  {
         struct vdpasim_virtqueue *vq = &vdpasim->vqs[idx];
+       uint16_t last_avail_idx = vq->vring.last_avail_idx;
 
         vringh_init_iotlb(&vq->vring, vdpasim->features, vq->num, false,
                           (struct vring_desc *)(uintptr_t)vq->desc_addr,
@@ -74,6 +75,7 @@ static void vdpasim_queue_ready(struct vdpasim *vdpasim, unsigned int idx)
                           (struct vring_used *)
                           (uintptr_t)vq->device_addr);
 
Quoted 9 lines

drivers/vdpa/vdpa_sim/vdpa_sim.c static void vdpasim_queue_ready(struct vdpasim *vdpasim, unsigned int idx) · line 78

Quoted 1 line
drivers/vdpa/vdpa_sim/vdpa_sim.c +1 -0
+       vq->vring.last_avail_idx = last_avail_idx;

Does this need to be serialized with the datapath?

E.g in set_vq_state() we do:

spin_lock(&vdpasim->lock);
vrh->last_avail_idx = state->split.avail_index;
spin_unlock(&vdpasim->lock);

Quoted 5 lines

vdpasim_queue_ready is called from vdpasim_set_vq_ready, which holds these locks.

Maybe it's too much indirection and to embed vdpasim_queue_ready in vdpasim_set_vq_ready would be clearer for the future?

Nope, I miss the caller.

Thanks

Quoted 3 lines

Thanks!

author maintainer
#6 lore raw

PATCH 2/2 vringh: fetch used_idx from vring at vringh_init_iotlb

In reply to Eugenio Pérez

Starting from an used_idx different than 0 is needed in use cases like virtual machine migration. Not doing so and letting the caller set an avail idx different than 0 causes destination device to try to use old buffers that source driver already recover and are not available anymore.

While callers like vdpa_sim set avail_idx directly it does not set used_idx. Instead of let the caller do the assignment, fetch it from the guest at initialization like vhost-kernel do.

To perform the same at vring_kernel_init and vring_user_init is left for the future.

---
 drivers/vhost/vringh.c | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)
drivers/vhost/vringh.c +23 -2
index 33eb941fcf15..0eed825197f2 100644
@@ -1301,6 +1301,17 @@ static inline int putused_iotlb(const struct vringh *vrh,
 	return 0;
 }
 
+/**
+ * vringh_update_used_idx - fetch used idx from driver's used split vring
+ * @vrh: The vring.
+ *
+ * Returns -errno or 0.
+ */
+static inline int vringh_update_used_idx(struct vringh *vrh)
+{
+	return getu16_iotlb(vrh, &vrh->last_used_idx, &vrh->vring.used->idx);
+}
+
 /**
  * vringh_init_iotlb - initialize a vringh for a ring with IOTLB.
  * @vrh: the vringh to initialize.
@@ -1319,8 +1330,18 @@ int vringh_init_iotlb(struct vringh *vrh, u64 features,
 		      struct vring_avail *avail,
 		      struct vring_used *used)
 {
-	return vringh_init_kern(vrh, features, num, weak_barriers,
-				desc, avail, used);
+	int r = vringh_init_kern(vrh, features, num, weak_barriers, desc,
+				 avail, used);
+
+	if (r != 0)
+		return r;
+
+	/* Consider the ring not initialized */
+	if ((void *)desc == used)
+		return 0;
+
+	return vringh_update_used_idx(vrh);
+
 }
 EXPORT_SYMBOL(vringh_init_iotlb);
 
Signature · 1 line
2.31.1
#7 lore raw

Re: [PATCH 2/2] vringh: fetch used_idx from vring at vringh_init_iotlb

Quoted 42 lines · Eugenio Pérez

Starting from an used_idx different than 0 is needed in use cases like virtual machine migration. Not doing so and letting the caller set an avail idx different than 0 causes destination device to try to use old buffers that source driver already recover and are not available anymore.

While callers like vdpa_sim set avail_idx directly it does not set used_idx. Instead of let the caller do the assignment, fetch it from the guest at initialization like vhost-kernel do.

To perform the same at vring_kernel_init and vring_user_init is left for the future.

---
 drivers/vhost/vringh.c | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)
drivers/vhost/vringh.c +11 -0
index 33eb941fcf15..0eed825197f2 100644
@@ -1301,6 +1301,17 @@ static inline int putused_iotlb(const struct vringh *vrh,
        return 0;
 }
 
+/**
+ * vringh_update_used_idx - fetch used idx from driver's used split vring
+ * @vrh: The vring.
+ *
+ * Returns -errno or 0.
+ */
+static inline int vringh_update_used_idx(struct vringh *vrh)
+{
+       return getu16_iotlb(vrh, &vrh->last_used_idx, &vrh->vring.used->idx);
+}
+
 /**
  * vringh_init_iotlb - initialize a vringh for a ring with IOTLB.
  * @vrh: the vringh to initialize.

drivers/vhost/vringh.c int vringh_init_iotlb(struct vringh *vrh, u64 features, · line 1332

Quoted 3 lines
drivers/vhost/vringh.c +0 -0
@@ -1319,8 +1330,18 @@ int vringh_init_iotlb(struct vringh *vrh, u64 features,
                      struct vring_avail *avail,
                      struct vring_used *used)
 {

While at this, I wonder if it's better to have a dedicated parameter for last_avail_idx?

drivers/vhost/vringh.c int vringh_init_iotlb(struct vringh *vrh, u64 features, · line 1341

Quoted 11 lines
drivers/vhost/vringh.c +9 -2
-       return vringh_init_kern(vrh, features, num, weak_barriers,
-                               desc, avail, used);
+       int r = vringh_init_kern(vrh, features, num, weak_barriers, desc,
+                                avail, used);
+
+       if (r != 0)
+               return r;
+
+       /* Consider the ring not initialized */
+       if ((void *)desc == used)
+               return 0;

I don't understand when we can get this (actually it should be a bug of the caller).

Thanks

Quoted 9 lines
drivers/vhost/vringh.c +3 -1
+
+       return vringh_update_used_idx(vrh);
+
 }
 EXPORT_SYMBOL(vringh_init_iotlb);
 
--

2.31.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author maintainer
#8 lore raw

Re: [PATCH 2/2] vringh: fetch used_idx from vring at vringh_init_iotlb

Quoted 42 lines · Jason Wang
Quoted 40 lines · Eugenio Pérez

Starting from an used_idx different than 0 is needed in use cases like virtual machine migration. Not doing so and letting the caller set an avail idx different than 0 causes destination device to try to use old buffers that source driver already recover and are not available anymore.

While callers like vdpa_sim set avail_idx directly it does not set used_idx. Instead of let the caller do the assignment, fetch it from the guest at initialization like vhost-kernel do.

To perform the same at vring_kernel_init and vring_user_init is left for the future.

---
 drivers/vhost/vringh.c | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)
drivers/vhost/vringh.c +11 -0
index 33eb941fcf15..0eed825197f2 100644
@@ -1301,6 +1301,17 @@ static inline int putused_iotlb(const struct vringh *vrh,
        return 0;
 }
 
+/**
+ * vringh_update_used_idx - fetch used idx from driver's used split vring
+ * @vrh: The vring.
+ *
+ * Returns -errno or 0.
+ */
+static inline int vringh_update_used_idx(struct vringh *vrh)
+{
+       return getu16_iotlb(vrh, &vrh->last_used_idx, &vrh->vring.used->idx);
+}
+
 /**
  * vringh_init_iotlb - initialize a vringh for a ring with IOTLB.
Quoted 9 lines

drivers/vhost/vringh.c int vringh_init_iotlb(struct vringh *vrh, u64 features, · line 1332

Quoted 5 lines
drivers/vhost/vringh.c +0 -0
  * @vrh: the vringh to initialize.
@@ -1319,8 +1330,18 @@ int vringh_init_iotlb(struct vringh *vrh, u64 features,
                      struct vring_avail *avail,
                      struct vring_used *used)
 {

While at this, I wonder if it's better to have a dedicated parameter for last_avail_idx?

I also had that thought. To directly assign last_avail_idx is not a specially elegant API IMO.

Maybe expose a way to fetch used_idx from device vring and pass used_idx as parameter too?

Quoted 6 lines
Quoted 6 lines
drivers/vhost/vringh.c +4 -2
-       return vringh_init_kern(vrh, features, num, weak_barriers,
-                               desc, avail, used);
+       int r = vringh_init_kern(vrh, features, num, weak_barriers, desc,
+                                avail, used);
+
+       if (r != 0)
Quoted 9 lines

drivers/vhost/vringh.c int vringh_init_iotlb(struct vringh *vrh, u64 features, · line 1341

Quoted 5 lines
drivers/vhost/vringh.c +5 -0
+               return r;
+
+       /* Consider the ring not initialized */
+       if ((void *)desc == used)
+               return 0;

I don't understand when we can get this (actually it should be a bug of the caller).

You can see it in vdpasim_vq_reset.

Note that to consider desc == 0 to be an uninitialized ring is a bug IMO. QEMU considers it that way also, but the standard does not forbid any ring to be at address 0. Especially if we use vIOMMU.

So I think the best way to know if we can use the vringh is either this way, or provide an explicit "initialized" boolean attribute. Maybe a new "bool is_initialized(vrh)" is enough, if we don't want to add new attributes.

Thanks!

Quoted 12 lines

Thanks

Quoted 9 lines
drivers/vhost/vringh.c +3 -1
+
+       return vringh_update_used_idx(vrh);
+
 }
 EXPORT_SYMBOL(vringh_init_iotlb);
 
--

2.31.1

#9 lore raw

Re: [PATCH 2/2] vringh: fetch used_idx from vring at vringh_init_iotlb

Quoted 54 lines · Eugenio Perez Martin
Quoted 42 lines · Jason Wang
Quoted 40 lines · Eugenio Pérez

Starting from an used_idx different than 0 is needed in use cases like virtual machine migration. Not doing so and letting the caller set an avail idx different than 0 causes destination device to try to use old buffers that source driver already recover and are not available anymore.

While callers like vdpa_sim set avail_idx directly it does not set used_idx. Instead of let the caller do the assignment, fetch it from the guest at initialization like vhost-kernel do.

To perform the same at vring_kernel_init and vring_user_init is left for the future.

---
 drivers/vhost/vringh.c | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)
drivers/vhost/vringh.c +11 -0
index 33eb941fcf15..0eed825197f2 100644
@@ -1301,6 +1301,17 @@ static inline int putused_iotlb(const struct vringh *vrh,
        return 0;
 }
 
+/**
+ * vringh_update_used_idx - fetch used idx from driver's used split vring
+ * @vrh: The vring.
+ *
+ * Returns -errno or 0.
+ */
+static inline int vringh_update_used_idx(struct vringh *vrh)
+{
+       return getu16_iotlb(vrh, &vrh->last_used_idx, &vrh->vring.used->idx);
+}
+
 /**
  * vringh_init_iotlb - initialize a vringh for a ring with IOTLB.
Quoted 9 lines

drivers/vhost/vringh.c int vringh_init_iotlb(struct vringh *vrh, u64 features, · line 1332

Quoted 5 lines
drivers/vhost/vringh.c +0 -0
  * @vrh: the vringh to initialize.
@@ -1319,8 +1330,18 @@ int vringh_init_iotlb(struct vringh *vrh, u64 features,
                      struct vring_avail *avail,
                      struct vring_used *used)
 {

While at this, I wonder if it's better to have a dedicated parameter for last_avail_idx?

Quoted 5 lines

I also had that thought. To directly assign last_avail_idx is not a specially elegant API IMO.

Maybe expose a way to fetch used_idx from device vring and pass used_idx as parameter too?

If I was not wrong, we can start from last_avail_idx, for used_idx it is only needed for inflight descriptors which might require other APIs?

(All the current vDPA user of vringh is doing in order processing)

Quoted 19 lines
Quoted 16 lines

drivers/vhost/vringh.c int vringh_init_iotlb(struct vringh *vrh, u64 features, · line 1341

Quoted 12 lines
drivers/vhost/vringh.c +9 -2
-       return vringh_init_kern(vrh, features, num, weak_barriers,
-                               desc, avail, used);
+       int r = vringh_init_kern(vrh, features, num, weak_barriers, desc,
+                                avail, used);
+
+       if (r != 0)
+               return r;
+
+       /* Consider the ring not initialized */
+       if ((void *)desc == used)
+               return 0;

I don't understand when we can get this (actually it should be a bug of the caller).

You can see it in vdpasim_vq_reset.

Quoted 8 lines

Note that to consider desc == 0 to be an uninitialized ring is a bug IMO. QEMU considers it that way also, but the standard does not forbid any ring to be at address 0. Especially if we use vIOMMU.

So I think the best way to know if we can use the vringh is either this way, or provide an explicit "initialized" boolean attribute. Maybe a new "bool is_initialized(vrh)" is enough, if we don't want to add new attributes.

I wonder if we can avoid this in the simulator level instead of the vringh (anyhow it only exposes a vringh_init_xxx() helper now).

Thanks

Quoted 16 lines

Thanks!

Quoted 12 lines

Thanks

Quoted 9 lines
drivers/vhost/vringh.c +3 -1
+
+       return vringh_update_used_idx(vrh);
+
 }
 EXPORT_SYMBOL(vringh_init_iotlb);
 
--

2.31.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author maintainer
#10 lore raw

Re: [PATCH 2/2] vringh: fetch used_idx from vring at vringh_init_iotlb

Quoted 60 lines · Jason Wang
Quoted 57 lines · Eugenio Perez Martin
Quoted 42 lines · Jason Wang
Quoted 40 lines · Eugenio Pérez

Starting from an used_idx different than 0 is needed in use cases like virtual machine migration. Not doing so and letting the caller set an avail idx different than 0 causes destination device to try to use old buffers that source driver already recover and are not available anymore.

While callers like vdpa_sim set avail_idx directly it does not set used_idx. Instead of let the caller do the assignment, fetch it from the guest at initialization like vhost-kernel do.

To perform the same at vring_kernel_init and vring_user_init is left for the future.

---
 drivers/vhost/vringh.c | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)
drivers/vhost/vringh.c +11 -0
index 33eb941fcf15..0eed825197f2 100644
@@ -1301,6 +1301,17 @@ static inline int putused_iotlb(const struct vringh *vrh,
        return 0;
 }
 
+/**
+ * vringh_update_used_idx - fetch used idx from driver's used split vring
+ * @vrh: The vring.
+ *
+ * Returns -errno or 0.
+ */
+static inline int vringh_update_used_idx(struct vringh *vrh)
+{
+       return getu16_iotlb(vrh, &vrh->last_used_idx, &vrh->vring.used->idx);
+}
+
 /**
  * vringh_init_iotlb - initialize a vringh for a ring with IOTLB.
Quoted 9 lines

drivers/vhost/vringh.c int vringh_init_iotlb(struct vringh *vrh, u64 features, · line 1332

Quoted 5 lines
drivers/vhost/vringh.c +0 -0
  * @vrh: the vringh to initialize.
@@ -1319,8 +1330,18 @@ int vringh_init_iotlb(struct vringh *vrh, u64 features,
                      struct vring_avail *avail,
                      struct vring_used *used)
 {

While at this, I wonder if it's better to have a dedicated parameter for last_avail_idx?

I also had that thought. To directly assign last_avail_idx is not a specially elegant API IMO.

Quoted 9 lines
Quoted 2 lines

Maybe expose a way to fetch used_idx from device vring and pass used_idx as parameter too?

If I was not wrong, we can start from last_avail_idx, for used_idx it is only needed for inflight descriptors which might require other APIs?

(All the current vDPA user of vringh is doing in order processing)

That was actually my first attempt and it works equally well for the moment, but it diverges from vhost-kernel behavior for little benefit.

To assign both values at set_vring_base mean that if vDPA introduces an (hypothetical) VHOST_VDPA_F_INFLIGHT backend feature in the future, the initialization process would vary a lot:
* Without that feature, the used_idx starts with 0, and the avail one is 0 or whatever value the user set with vring_set_base.
* With that feature, the device will read guest's used_idx as vhost-kernel? We would enable a new ioctl to set it, or expand set_base to include used_idx, effectively diverting from vhost-kernel?

To me the wisest option is to move this with vhost-kernel. Maybe we need to add a feature bit to know that the hypervisor can trust the device will do "the right thing" (VHOST_VDPA_F_FETCH_USED_AT_ENABLE?), but we should keep it orthogonal to inflight descriptor migration in my opinion.

Having said that, I'm totally ok to do it otherwise (or to expand the patch message if needed).

Quoted 23 lines
Quoted 23 lines
Quoted 16 lines

drivers/vhost/vringh.c int vringh_init_iotlb(struct vringh *vrh, u64 features, · line 1341

Quoted 12 lines
drivers/vhost/vringh.c +9 -2
-       return vringh_init_kern(vrh, features, num, weak_barriers,
-                               desc, avail, used);
+       int r = vringh_init_kern(vrh, features, num, weak_barriers, desc,
+                                avail, used);
+
+       if (r != 0)
+               return r;
+
+       /* Consider the ring not initialized */
+       if ((void *)desc == used)
+               return 0;

I don't understand when we can get this (actually it should be a bug of the caller).

You can see it in vdpasim_vq_reset.

Note that to consider desc == 0 to be an uninitialized ring is a bug IMO. QEMU considers it that way also, but the standard does not forbid any ring to be at address 0. Especially if we use vIOMMU.

Quoted 8 lines
Quoted 4 lines

So I think the best way to know if we can use the vringh is either this way, or provide an explicit "initialized" boolean attribute. Maybe a new "bool is_initialized(vrh)" is enough, if we don't want to add new attributes.

I wonder if we can avoid this in the simulator level instead of the vringh (anyhow it only exposes a vringh_init_xxx() helper now).

In my opinion that is a mistake if other drivers will use it to implement the emulated control virtqueue. And it requires more changes. But it is doable for sure.

Thanks!

#11 lore raw

Re: [PATCH 2/2] vringh: fetch used_idx from vring at vringh_init_iotlb

Quoted 75 lines · Eugenio Perez Martin
Quoted 60 lines · Jason Wang
Quoted 57 lines · Eugenio Perez Martin
Quoted 42 lines · Jason Wang
Quoted 40 lines · Eugenio Pérez

Starting from an used_idx different than 0 is needed in use cases like virtual machine migration. Not doing so and letting the caller set an avail idx different than 0 causes destination device to try to use old buffers that source driver already recover and are not available anymore.

While callers like vdpa_sim set avail_idx directly it does not set used_idx. Instead of let the caller do the assignment, fetch it from the guest at initialization like vhost-kernel do.

To perform the same at vring_kernel_init and vring_user_init is left for the future.

---
 drivers/vhost/vringh.c | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)
drivers/vhost/vringh.c +11 -0
index 33eb941fcf15..0eed825197f2 100644
@@ -1301,6 +1301,17 @@ static inline int putused_iotlb(const struct vringh *vrh,
        return 0;
 }
 
+/**
+ * vringh_update_used_idx - fetch used idx from driver's used split vring
+ * @vrh: The vring.
+ *
+ * Returns -errno or 0.
+ */
+static inline int vringh_update_used_idx(struct vringh *vrh)
+{
+       return getu16_iotlb(vrh, &vrh->last_used_idx, &vrh->vring.used->idx);
+}
+
 /**
  * vringh_init_iotlb - initialize a vringh for a ring with IOTLB.
Quoted 9 lines

drivers/vhost/vringh.c int vringh_init_iotlb(struct vringh *vrh, u64 features, · line 1332

Quoted 5 lines
drivers/vhost/vringh.c +0 -0
  * @vrh: the vringh to initialize.
@@ -1319,8 +1330,18 @@ int vringh_init_iotlb(struct vringh *vrh, u64 features,
                      struct vring_avail *avail,
                      struct vring_used *used)
 {

While at this, I wonder if it's better to have a dedicated parameter for last_avail_idx?

I also had that thought. To directly assign last_avail_idx is not a specially elegant API IMO.

Quoted 9 lines
Quoted 2 lines

Maybe expose a way to fetch used_idx from device vring and pass used_idx as parameter too?

If I was not wrong, we can start from last_avail_idx, for used_idx it is only needed for inflight descriptors which might require other APIs?

(All the current vDPA user of vringh is doing in order processing)

That was actually my first attempt and it works equally well for the moment, but it diverges from vhost-kernel behavior for little benefit.

Quoted 8 lines

To assign both values at set_vring_base mean that if vDPA introduces an (hypothetical) VHOST_VDPA_F_INFLIGHT backend feature in the future, the initialization process would vary a lot:
* Without that feature, the used_idx starts with 0, and the avail one is 0 or whatever value the user set with vring_set_base.
* With that feature, the device will read guest's used_idx as vhost-kernel? We would enable a new ioctl to set it, or expand set_base to include used_idx, effectively diverting from vhost-kernel?

Adding Longpeng who is looking at this.

We can leave this to the caller to decide.

Btw, a question, at which case the device used index does not equal to the used index in the vring when the device is suspended? I think the correct way is to flush the pending used indexes before suspending. Otherwise we need an API to get pending used indices?

Quoted 6 lines

To me the wisest option is to move this with vhost-kernel. Maybe we need to add a feature bit to know that the hypervisor can trust the device will do "the right thing" (VHOST_VDPA_F_FETCH_USED_AT_ENABLE?), but we should keep it orthogonal to inflight descriptor migration in my opinion.

I think we need to understand if there are any other possible use cases for setting used idx other than inflight stuff.

Quoted 3 lines

Having said that, I'm totally ok to do it otherwise (or to expand the patch message if needed).

I tend to do that in another series (not mix with the fixes).

Quoted 29 lines
Quoted 29 lines
Quoted 28 lines
Quoted 17 lines

drivers/vhost/vringh.c int vringh_init_iotlb(struct vringh *vrh, u64 features, · line 1341

Quoted 13 lines
drivers/vhost/vringh.c +9 -2
-       return vringh_init_kern(vrh, features, num, weak_barriers,
-                               desc, avail, used);
+       int r = vringh_init_kern(vrh, features, num, weak_barriers, desc,
+                                avail, used);
+
+       if (r != 0)
+               return r;
+
+       /* Consider the ring not initialized */
+       if ((void *)desc == used)
+               return 0;

I don't understand when we can get this (actually it should be a bug of the caller).

You can see it in vdpasim_vq_reset.

Note that to consider desc == 0 to be an uninitialized ring is a bug IMO. QEMU considers it that way also, but the standard does not forbid any ring to be at address 0. Especially if we use vIOMMU.

So I think the best way to know if we can use the vringh is either this way, or provide an explicit "initialized" boolean attribute. Maybe a new "bool is_initialized(vrh)" is enough, if we don't want to add new attributes.

Quoted 7 lines
Quoted 3 lines

I wonder if we can avoid this in the simulator level instead of the vringh (anyhow it only exposes a vringh_init_xxx() helper now).

In my opinion that is a mistake if other drivers will use it to implement the emulated control virtqueue. And it requires more changes. But it is doable for sure.

The problem is, there's no reset API in vringh, that's why you need to do if ((void *)desc == used) which depends on behaviour of the vringh user.

So I think we should either:

1) move that check in vdpa_sim (since it's not guaranteed that all the vringh users will make desc equal to used during reset)

or

2) introduce a vringh_reset_xxx()

1) seems a good step for -stable.

Thanks

Quoted 3 lines

Thanks!

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author maintainer
#12 lore raw

Re: [PATCH 2/2] vringh: fetch used_idx from vring at vringh_init_iotlb

Quoted 87 lines · Jason Wang
Quoted 75 lines · Eugenio Perez Martin
Quoted 60 lines · Jason Wang
Quoted 57 lines · Eugenio Perez Martin
Quoted 42 lines · Jason Wang
Quoted 40 lines · Eugenio Pérez

Starting from an used_idx different than 0 is needed in use cases like virtual machine migration. Not doing so and letting the caller set an avail idx different than 0 causes destination device to try to use old buffers that source driver already recover and are not available anymore.

While callers like vdpa_sim set avail_idx directly it does not set used_idx. Instead of let the caller do the assignment, fetch it from the guest at initialization like vhost-kernel do.

To perform the same at vring_kernel_init and vring_user_init is left for the future.

---
 drivers/vhost/vringh.c | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)
drivers/vhost/vringh.c +11 -0
index 33eb941fcf15..0eed825197f2 100644
@@ -1301,6 +1301,17 @@ static inline int putused_iotlb(const struct vringh *vrh,
        return 0;
 }
 
+/**
+ * vringh_update_used_idx - fetch used idx from driver's used split vring
+ * @vrh: The vring.
+ *
+ * Returns -errno or 0.
+ */
+static inline int vringh_update_used_idx(struct vringh *vrh)
+{
+       return getu16_iotlb(vrh, &vrh->last_used_idx, &vrh->vring.used->idx);
+}
+
 /**
  * vringh_init_iotlb - initialize a vringh for a ring with IOTLB.
Quoted 9 lines

drivers/vhost/vringh.c int vringh_init_iotlb(struct vringh *vrh, u64 features, · line 1332

Quoted 5 lines
drivers/vhost/vringh.c +0 -0
  * @vrh: the vringh to initialize.
@@ -1319,8 +1330,18 @@ int vringh_init_iotlb(struct vringh *vrh, u64 features,
                      struct vring_avail *avail,
                      struct vring_used *used)
 {

While at this, I wonder if it's better to have a dedicated parameter for last_avail_idx?

I also had that thought. To directly assign last_avail_idx is not a specially elegant API IMO.

Quoted 9 lines
Quoted 2 lines

Maybe expose a way to fetch used_idx from device vring and pass used_idx as parameter too?

If I was not wrong, we can start from last_avail_idx, for used_idx it is only needed for inflight descriptors which might require other APIs?

(All the current vDPA user of vringh is doing in order processing)

That was actually my first attempt and it works equally well for the moment, but it diverges from vhost-kernel behavior for little benefit.

Quoted 8 lines

To assign both values at set_vring_base mean that if vDPA introduces an (hypothetical) VHOST_VDPA_F_INFLIGHT backend feature in the future, the initialization process would vary a lot:
* Without that feature, the used_idx starts with 0, and the avail one is 0 or whatever value the user set with vring_set_base.
* With that feature, the device will read guest's used_idx as vhost-kernel? We would enable a new ioctl to set it, or expand set_base to include used_idx, effectively diverting from vhost-kernel?

Quoted 2 lines

Adding Longpeng who is looking at this.

Sorry, I'll CC [email protected] in future series like this one.

Quoted 8 lines

We can leave this to the caller to decide.

Btw, a question, at which case the device used index does not equal to the used index in the vring when the device is suspended? I think the correct way is to flush the pending used indexes before suspending. Otherwise we need an API to get pending used indices?

Quoted 1 line
Quoted 9 lines
Quoted 5 lines

To me the wisest option is to move this with vhost-kernel. Maybe we need to add a feature bit to know that the hypervisor can trust the device will do "the right thing" (VHOST_VDPA_F_FETCH_USED_AT_ENABLE?), but we should keep it orthogonal to inflight descriptor migration in my opinion.

I think we need to understand if there are any other possible use cases for setting used idx other than inflight stuff.

Answering this and the previous comment, I cannot think in any case outside of inflight migration. I'm just trying to avoid different behavior between vhost-kernel and vhost-vdpa, and to make features as orthogonal as possible.

Quoted 39 lines
Quoted 3 lines

Having said that, I'm totally ok to do it otherwise (or to expand the patch message if needed).

I tend to do that in another series (not mix with the fixes).

Quoted 33 lines
Quoted 32 lines
Quoted 20 lines
Quoted 17 lines

drivers/vhost/vringh.c int vringh_init_iotlb(struct vringh *vrh, u64 features, · line 1341

Quoted 13 lines
drivers/vhost/vringh.c +9 -2
-       return vringh_init_kern(vrh, features, num, weak_barriers,
-                               desc, avail, used);
+       int r = vringh_init_kern(vrh, features, num, weak_barriers, desc,
+                                avail, used);
+
+       if (r != 0)
+               return r;
+
+       /* Consider the ring not initialized */
+       if ((void *)desc == used)
+               return 0;

I don't understand when we can get this (actually it should be a bug of the caller).

You can see it in vdpasim_vq_reset.

Quoted 8 lines

Note that to consider desc == 0 to be an uninitialized ring is a bug IMO. QEMU considers it that way also, but the standard does not forbid any ring to be at address 0. Especially if we use vIOMMU.

So I think the best way to know if we can use the vringh is either this way, or provide an explicit "initialized" boolean attribute. Maybe a new "bool is_initialized(vrh)" is enough, if we don't want to add new attributes.

I wonder if we can avoid this in the simulator level instead of the vringh (anyhow it only exposes a vringh_init_xxx() helper now).

Quoted 8 lines
Quoted 3 lines

In my opinion that is a mistake if other drivers will use it to implement the emulated control virtqueue. And it requires more changes. But it is doable for sure.

The problem is, there's no reset API in vringh, that's why you need to do if ((void *)desc == used) which depends on behaviour of the vringh user.

That's a very good point indeed.

Quoted 11 lines

So I think we should either:

1) move that check in vdpa_sim (since it's not guaranteed that all the vringh users will make desc equal to used during reset)

or

2) introduce a vringh_reset_xxx()

1) seems a good step for -stable.

We can go to 1 for sure. So let's set last_used_idx at vdpasim_set_vq_state, same value as last_avail_idx, and stash it at vdpasim_queue_ready.

Do I need to resend the previous patch in this series?

Do we need to offer a new feature flag indicating we will set used_idx with avail_idx?

Thanks!

#13 lore raw

Re: [PATCH 2/2] vringh: fetch used_idx from vring at vringh_init_iotlb

Quoted 182 lines · Eugenio Perez Martin
Quoted 87 lines · Jason Wang
Quoted 75 lines · Eugenio Perez Martin
Quoted 60 lines · Jason Wang
Quoted 57 lines · Eugenio Perez Martin
Quoted 42 lines · Jason Wang
Quoted 40 lines · Eugenio Pérez

Starting from an used_idx different than 0 is needed in use cases like virtual machine migration. Not doing so and letting the caller set an avail idx different than 0 causes destination device to try to use old buffers that source driver already recover and are not available anymore.

While callers like vdpa_sim set avail_idx directly it does not set used_idx. Instead of let the caller do the assignment, fetch it from the guest at initialization like vhost-kernel do.

To perform the same at vring_kernel_init and vring_user_init is left for the future.

---
 drivers/vhost/vringh.c | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)
drivers/vhost/vringh.c +11 -0
index 33eb941fcf15..0eed825197f2 100644
@@ -1301,6 +1301,17 @@ static inline int putused_iotlb(const struct vringh *vrh,
        return 0;
 }
 
+/**
+ * vringh_update_used_idx - fetch used idx from driver's used split vring
+ * @vrh: The vring.
+ *
+ * Returns -errno or 0.
+ */
+static inline int vringh_update_used_idx(struct vringh *vrh)
+{
+       return getu16_iotlb(vrh, &vrh->last_used_idx, &vrh->vring.used->idx);
+}
+
 /**
  * vringh_init_iotlb - initialize a vringh for a ring with IOTLB.
Quoted 9 lines

drivers/vhost/vringh.c int vringh_init_iotlb(struct vringh *vrh, u64 features, · line 1332

Quoted 5 lines
drivers/vhost/vringh.c +0 -0
  * @vrh: the vringh to initialize.
@@ -1319,8 +1330,18 @@ int vringh_init_iotlb(struct vringh *vrh, u64 features,
                      struct vring_avail *avail,
                      struct vring_used *used)
 {

While at this, I wonder if it's better to have a dedicated parameter for last_avail_idx?

I also had that thought. To directly assign last_avail_idx is not a specially elegant API IMO.

Quoted 9 lines
Quoted 2 lines

Maybe expose a way to fetch used_idx from device vring and pass used_idx as parameter too?

If I was not wrong, we can start from last_avail_idx, for used_idx it is only needed for inflight descriptors which might require other APIs?

(All the current vDPA user of vringh is doing in order processing)

That was actually my first attempt and it works equally well for the moment, but it diverges from vhost-kernel behavior for little benefit.

Quoted 8 lines

To assign both values at set_vring_base mean that if vDPA introduces an (hypothetical) VHOST_VDPA_F_INFLIGHT backend feature in the future, the initialization process would vary a lot:
* Without that feature, the used_idx starts with 0, and the avail one is 0 or whatever value the user set with vring_set_base.
* With that feature, the device will read guest's used_idx as vhost-kernel? We would enable a new ioctl to set it, or expand set_base to include used_idx, effectively diverting from vhost-kernel?

Quoted 2 lines

Adding Longpeng who is looking at this.

Sorry, I'll CC [email protected] in future series like this one.

Quoted 8 lines

We can leave this to the caller to decide.

Btw, a question, at which case the device used index does not equal to the used index in the vring when the device is suspended? I think the correct way is to flush the pending used indexes before suspending. Otherwise we need an API to get pending used indices?

Quoted 1 line
Quoted 9 lines
Quoted 5 lines

To me the wisest option is to move this with vhost-kernel. Maybe we need to add a feature bit to know that the hypervisor can trust the device will do "the right thing" (VHOST_VDPA_F_FETCH_USED_AT_ENABLE?), but we should keep it orthogonal to inflight descriptor migration in my opinion.

I think we need to understand if there are any other possible use cases for setting used idx other than inflight stuff.

Answering this and the previous comment, I cannot think in any case outside of inflight migration. I'm just trying to avoid different behavior between vhost-kernel and vhost-vdpa, and to make features as orthogonal as possible.

Quoted 39 lines
Quoted 3 lines

Having said that, I'm totally ok to do it otherwise (or to expand the patch message if needed).

I tend to do that in another series (not mix with the fixes).

Quoted 33 lines
Quoted 32 lines
Quoted 20 lines
Quoted 17 lines

drivers/vhost/vringh.c int vringh_init_iotlb(struct vringh *vrh, u64 features, · line 1341

Quoted 13 lines
drivers/vhost/vringh.c +9 -2
-       return vringh_init_kern(vrh, features, num, weak_barriers,
-                               desc, avail, used);
+       int r = vringh_init_kern(vrh, features, num, weak_barriers, desc,
+                                avail, used);
+
+       if (r != 0)
+               return r;
+
+       /* Consider the ring not initialized */
+       if ((void *)desc == used)
+               return 0;

I don't understand when we can get this (actually it should be a bug of the caller).

You can see it in vdpasim_vq_reset.

Quoted 8 lines

Note that to consider desc == 0 to be an uninitialized ring is a bug IMO. QEMU considers it that way also, but the standard does not forbid any ring to be at address 0. Especially if we use vIOMMU.

So I think the best way to know if we can use the vringh is either this way, or provide an explicit "initialized" boolean attribute. Maybe a new "bool is_initialized(vrh)" is enough, if we don't want to add new attributes.

I wonder if we can avoid this in the simulator level instead of the vringh (anyhow it only exposes a vringh_init_xxx() helper now).

Quoted 8 lines
Quoted 3 lines

In my opinion that is a mistake if other drivers will use it to implement the emulated control virtqueue. And it requires more changes. But it is doable for sure.

The problem is, there's no reset API in vringh, that's why you need to do if ((void *)desc == used) which depends on behaviour of the vringh user.

That's a very good point indeed.

Quoted 11 lines

So I think we should either:

1) move that check in vdpa_sim (since it's not guaranteed that all the vringh users will make desc equal to used during reset)

or

2) introduce a vringh_reset_xxx()

1) seems a good step for -stable.

We can go to 1 for sure. So let's set last_used_idx at vdpasim_set_vq_state, same value as last_avail_idx, and stash it at vdpasim_queue_ready.

Quoted 6 lines

Do I need to resend the previous patch in this series?

Do we need to offer a new feature flag indicating we will set used_idx with avail_idx?

Thanks!

Jason did you forget to answer or did I miss it?

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#14 lore raw

Re: [PATCH 2/2] vringh: fetch used_idx from vring at vringh_init_iotlb

In reply to Eugenio Pérez
Quoted 6 lines · Eugenio Pérez

Starting from an used_idx different than 0 is needed in use cases like virtual machine migration. Not doing so and letting the caller set an avail idx different than 0 causes destination device to try to use old buffers that source driver already recover and are not available anymore.

Quoted 8 lines

While callers like vdpa_sim set avail_idx directly it does not set used_idx. Instead of let the caller do the assignment, fetch it from the guest at initialization like vhost-kernel do.

To perform the same at vring_kernel_init and vring_user_init is left for the future.

So I applied 1/2 and dropped 2/2 for now, right?

Quoted 49 lines
---
 drivers/vhost/vringh.c | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)
drivers/vhost/vringh.c +23 -2
index 33eb941fcf15..0eed825197f2 100644
@@ -1301,6 +1301,17 @@ static inline int putused_iotlb(const struct vringh *vrh,
 	return 0;
 }
 
+/**
+ * vringh_update_used_idx - fetch used idx from driver's used split vring
+ * @vrh: The vring.
+ *
+ * Returns -errno or 0.
+ */
+static inline int vringh_update_used_idx(struct vringh *vrh)
+{
+	return getu16_iotlb(vrh, &vrh->last_used_idx, &vrh->vring.used->idx);
+}
+
 /**
  * vringh_init_iotlb - initialize a vringh for a ring with IOTLB.
  * @vrh: the vringh to initialize.
@@ -1319,8 +1330,18 @@ int vringh_init_iotlb(struct vringh *vrh, u64 features,
 		      struct vring_avail *avail,
 		      struct vring_used *used)
 {
-	return vringh_init_kern(vrh, features, num, weak_barriers,
-				desc, avail, used);
+	int r = vringh_init_kern(vrh, features, num, weak_barriers, desc,
+				 avail, used);
+
+	if (r != 0)
+		return r;
+
+	/* Consider the ring not initialized */
+	if ((void *)desc == used)
+		return 0;
+
+	return vringh_update_used_idx(vrh);
+
 }
 EXPORT_SYMBOL(vringh_init_iotlb);
 
Signature · 1 line
2.31.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author maintainer
#15 lore raw

Re: [PATCH 2/2] vringh: fetch used_idx from vring at vringh_init_iotlb

Quoted 12 lines · Michael S. Tsirkin
Quoted 10 lines · Eugenio Pérez

Starting from an used_idx different than 0 is needed in use cases like virtual machine migration. Not doing so and letting the caller set an avail idx different than 0 causes destination device to try to use old buffers that source driver already recover and are not available anymore.

While callers like vdpa_sim set avail_idx directly it does not set used_idx. Instead of let the caller do the assignment, fetch it from the guest at initialization like vhost-kernel do.

Quoted 8 lines
Quoted 4 lines

To perform the same at vring_kernel_init and vring_user_init is left for the future.

So I applied 1/2 and dropped 2/2 for now, right?

Yes, please. 2/2 needs tweaking, I'll address them ASAP.

Thanks!

Quoted 50 lines
Quoted 49 lines
---
 drivers/vhost/vringh.c | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)
drivers/vhost/vringh.c +23 -3
index 33eb941fcf15..0eed825197f2 100644
@@ -1301,6 +1301,17 @@ static inline int putused_iotlb(const struct vringh *vrh,
      return 0;
 }
 
+/**
+ * vringh_update_used_idx - fetch used idx from driver's used split vring
+ * @vrh: The vring.
+ *
+ * Returns -errno or 0.
+ */
+static inline int vringh_update_used_idx(struct vringh *vrh)
+{
+     return getu16_iotlb(vrh, &vrh->last_used_idx, &vrh->vring.used->idx);
+}
+
 /**
  * vringh_init_iotlb - initialize a vringh for a ring with IOTLB.
  * @vrh: the vringh to initialize.
@@ -1319,8 +1330,18 @@ int vringh_init_iotlb(struct vringh *vrh, u64 features,
                    struct vring_avail *avail,
                    struct vring_used *used)
 {
-     return vringh_init_kern(vrh, features, num, weak_barriers,
-                             desc, avail, used);
+     int r = vringh_init_kern(vrh, features, num, weak_barriers, desc,
+                              avail, used);
+
+     if (r != 0)
+             return r;
+
+     /* Consider the ring not initialized */
+     if ((void *)desc == used)
+             return 0;
+
+     return vringh_update_used_idx(vrh);
+
 }
 EXPORT_SYMBOL(vringh_init_iotlb);
 
--

2.31.1

#16 lore raw
In reply to Eugenio Pérez
Quoted 7 lines · Eugenio Pérez

The use of set_vq_state is to indicate vdpa device the state of a virtqueue. In the case of split, it means the avail_idx. This is mandatory for use cases like live migration.

However, vdpa_sim reset the vq state at vdpasim_queue_ready since it calls vringh_init_iotlb.

Quoted 6 lines

Also, to starting from an used_idx different than 0 is needed in use cases like virtual machine migration. Not doing so and letting the caller set an avail idx different than 0 causes destination device to try to use old buffers that source driver already recover and are not available anymore.

This series fixes both problems allowing to migrate to a vdpa_sim_net device.

Jason problems you pointed out are all consmetic do you ack the patchset? Or expect another revision?

Quoted 11 lines

Eugenio Pérez (2):
vdpa_sim: not reset state in vdpasim_queue_ready
vringh: fetch used_idx from vring at vringh_init_iotlb

 drivers/vdpa/vdpa_sim/vdpa_sim.c |  2 ++
 drivers/vhost/vringh.c           | 25 +++++++++++++++++++++++--
 2 files changed, 25 insertions(+), 2 deletions(-)
Signature · 1 line
2.31.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization