Skip to contentSkip to conversation

vhost_vdpa: better PACKED support

PATCH In review Shannon Nelson via Virtualization 4 participants 22 messages Apr 24, 2023 – Jun 5, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover vhost_vdpa: better PACKED support

While testing our vDPA driver with QEMU we found that vhost_vdpa was missing some support for PACKED VQs. Adding these helped us get further in our testing. The first patch makes sure that the vhost_vdpa vqs are given the feature flags, as done in other vhost variants. The second and third patches use the feature flags to properly format and pass set/get_vq requests.

v1:
- included wrap counter in packing answers for VHOST_GET_VRING_BASE
- added comments to vhost_virtqueue fields last_avail_idx and last_used_idx

v0:
https://lists.linuxfoundation.org/pipermail/virtualization/2023-April/066253.html
- initial version

Shannon Nelson (3):
vhost_vdpa: tell vqs about the negotiated
vhost: support PACKED when setting-getting vring_base
vhost_vdpa: support PACKED when setting-getting vring_base

 drivers/vhost/vdpa.c  | 34 ++++++++++++++++++++++++++++++----
 drivers/vhost/vhost.c | 18 +++++++++++++-----
 drivers/vhost/vhost.h |  8 ++++++--
 3 files changed, 49 insertions(+), 11 deletions(-)
Signature · 6 lines
2.17.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
#1 lore raw

PATCH v2 1/3 vhost_vdpa: tell vqs about the negotiated

As is done in the net, iscsi, and vsock vhost support, let the vdpa vqs know about the features that have been negotiated. This allows vhost to more safely make decisions based on the features, such as when using PACKED vs split queues.

---
 drivers/vhost/vdpa.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
drivers/vhost/vdpa.c +13 -0
index 7be9d9d8f01c..599b8cc238c7 100644
@@ -385,7 +385,10 @@ static long vhost_vdpa_set_features(struct vhost_vdpa *v, u64 __user *featurep)
 {
 	struct vdpa_device *vdpa = v->vdpa;
 	const struct vdpa_config_ops *ops = vdpa->config;
+	struct vhost_dev *d = &v->vdev;
+	u64 actual_features;
 	u64 features;
+	int i;
 
 	/*
 	 * It's not allowed to change the features after they have
@@ -400,6 +403,16 @@ static long vhost_vdpa_set_features(struct vhost_vdpa *v, u64 __user *featurep)
 	if (vdpa_set_features(vdpa, features))
 		return -EINVAL;
 
+	/* let the vqs know what has been configured */
+	actual_features = ops->get_driver_features(vdpa);
+	for (i = 0; i < d->nvqs; ++i) {
+		struct vhost_virtqueue *vq = d->vqs[i];
+
+		mutex_lock(&vq->mutex);
+		vq->acked_features = actual_features;
+		mutex_unlock(&vq->mutex);
+	}
+
 	return 0;
 }
 
Signature · 6 lines
2.17.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
#2 lore raw

PATCH v2 2/3 vhost: support PACKED when setting-getting vring_base

In reply to Shannon Nelson via Virtualization

Use the right structs for PACKED or split vqs when setting and getting the vring base.

---
 drivers/vhost/vhost.c | 18 +++++++++++++-----
 drivers/vhost/vhost.h |  8 ++++++--
 2 files changed, 19 insertions(+), 7 deletions(-)
drivers/vhost/vhost.c +13 -5
index f11bdbe4c2c5..f64efda48f21 100644
@@ -1633,17 +1633,25 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
 			r = -EFAULT;
 			break;
 		}
-		if (s.num > 0xffff) {
-			r = -EINVAL;
-			break;
+		if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED)) {
+			vq->last_avail_idx = s.num & 0xffff;
+			vq->last_used_idx = (s.num >> 16) & 0xffff;
+		} else {
+			if (s.num > 0xffff) {
+				r = -EINVAL;
+				break;
+			}
+			vq->last_avail_idx = s.num;
 		}
-		vq->last_avail_idx = s.num;
 		/* Forget the cached index value. */
 		vq->avail_idx = vq->last_avail_idx;
 		break;
 	case VHOST_GET_VRING_BASE:
 		s.index = idx;
-		s.num = vq->last_avail_idx;
+		if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED))
+			s.num = (u32)vq->last_avail_idx | ((u32)vq->last_used_idx << 16);
+		else
+			s.num = vq->last_avail_idx;
 		if (copy_to_user(argp, &s, sizeof s))
 			r = -EFAULT;
 		break;
drivers/vhost/vhost.h +6 -2
index 1647b750169c..6f73f29d5979 100644
@@ -85,13 +85,17 @@ struct vhost_virtqueue {
 	/* The routine to call when the Guest pings us, or timeout. */
 	vhost_work_fn_t handle_kick;
 
-	/* Last available index we saw. */
+	/* Last available index we saw.
+	 * Values are limited to 0x7fff, and the high bit is used as
+	 * a wrap counter when using VIRTIO_F_RING_PACKED. */
 	u16 last_avail_idx;
 
 	/* Caches available index value from user. */
 	u16 avail_idx;
 
-	/* Last index we used. */
+	/* Last index we used.
+	 * Values are limited to 0x7fff, and the high bit is used as
+	 * a wrap counter when using VIRTIO_F_RING_PACKED. */
 	u16 last_used_idx;
 
 	/* Used flags */
Signature · 6 lines
2.17.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Acked-by on v2 2/3
#3 lore raw

Re: [PATCH v2 2/3] vhost: support PACKED when setting-getting vring_base

Quoted 5 lines · Shannon Nelson

Use the right structs for PACKED or split vqs when setting and getting the vring base.

Thanks

Quoted 65 lines
---
 drivers/vhost/vhost.c | 18 +++++++++++++-----
 drivers/vhost/vhost.h |  8 ++++++--
 2 files changed, 19 insertions(+), 7 deletions(-)
drivers/vhost/vhost.c +13 -5
index f11bdbe4c2c5..f64efda48f21 100644
@@ -1633,17 +1633,25 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
                        r = -EFAULT;
                        break;
                }
-               if (s.num > 0xffff) {
-                       r = -EINVAL;
-                       break;
+               if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED)) {
+                       vq->last_avail_idx = s.num & 0xffff;
+                       vq->last_used_idx = (s.num >> 16) & 0xffff;
+               } else {
+                       if (s.num > 0xffff) {
+                               r = -EINVAL;
+                               break;
+                       }
+                       vq->last_avail_idx = s.num;
                }
-               vq->last_avail_idx = s.num;
                /* Forget the cached index value. */
                vq->avail_idx = vq->last_avail_idx;
                break;
        case VHOST_GET_VRING_BASE:
                s.index = idx;
-               s.num = vq->last_avail_idx;
+               if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED))
+                       s.num = (u32)vq->last_avail_idx | ((u32)vq->last_used_idx << 16);
+               else
+                       s.num = vq->last_avail_idx;
                if (copy_to_user(argp, &s, sizeof s))
                        r = -EFAULT;
                break;
drivers/vhost/vhost.h +6 -3
index 1647b750169c..6f73f29d5979 100644
@@ -85,13 +85,17 @@ struct vhost_virtqueue {
        /* The routine to call when the Guest pings us, or timeout. */
        vhost_work_fn_t handle_kick;
 
-       /* Last available index we saw. */
+       /* Last available index we saw.
+        * Values are limited to 0x7fff, and the high bit is used as
+        * a wrap counter when using VIRTIO_F_RING_PACKED. */
        u16 last_avail_idx;
 
        /* Caches available index value from user. */
        u16 avail_idx;
 
-       /* Last index we used. */
+       /* Last index we used.
+        * Values are limited to 0x7fff, and the high bit is used as
+        * a wrap counter when using VIRTIO_F_RING_PACKED. */
        u16 last_used_idx;
 
        /* Used flags */
--
Quoted 2 lines

2.17.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#4 lore raw

Re: [PATCH v2 2/3] vhost: support PACKED when setting-getting vring_base

In reply to Shannon Nelson via Virtualization
Quoted 31 lines · Shannon Nelson via Virtualization

Use the right structs for PACKED or split vqs when setting and getting the vring base.

---
drivers/vhost/vhost.c | 18 +++++++++++++-----
drivers/vhost/vhost.h | 8 ++++++--

2 files changed, 19 insertions(+), 7 deletions(-)
drivers/vhost/vhost.c +9 -4
index f11bdbe4c2c5..f64efda48f21 100644
@@ -1633,17 +1633,25 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg
 			r = -EFAULT;
 			break;
 		}
-		if (s.num > 0xffff) {
-			r = -EINVAL;
-			break;
+		if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED)) {
+			vq->last_avail_idx = s.num & 0xffff;
+			vq->last_used_idx = (s.num >> 16) & 0xffff;
+		} else {
+			if (s.num > 0xffff) {
+				r = -EINVAL;
+				break;
+			}
+			vq->last_avail_idx = s.num;
 		}
-		vq->last_avail_idx = s.num;

drivers/vhost/vhost.c long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg · line 1654

Quoted 10 lines
drivers/vhost/vhost.c +4 -1
 		/* Forget the cached index value. */
 		vq->avail_idx = vq->last_avail_idx;
 		break;
 	case VHOST_GET_VRING_BASE:
 		s.index = idx;
-		s.num = vq->last_avail_idx;
+		if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED))
+			s.num = (u32)vq->last_avail_idx | ((u32)vq->last_used_idx << 16);
+		else
+			s.num = vq->last_avail_idx;

The changes LGTM, but since we are changing the UAPI, should we update the documentation of VHOST_SET_VRING_BASE and VHOST_GET_VRING_BASE in include/uapi/linux/vhost.h?

Thanks,
Stefano

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#5 lore raw

Re: [PATCH v2 2/3] vhost: support PACKED when setting-getting vring_base

Quoted 40 lines · Stefano Garzarella
Quoted 39 lines · Shannon Nelson via Virtualization

Use the right structs for PACKED or split vqs when setting and getting the vring base.

---
drivers/vhost/vhost.c | 18 +++++++++++++-----
drivers/vhost/vhost.h |  8 ++++++--

2 files changed, 19 insertions(+), 7 deletions(-)
drivers/vhost/vhost.c +0 -0
index f11bdbe4c2c5..f64efda48f21 100644
@@ -1633,17 +1633,25 @@ long vhost_vring_ioctl(struct vhost_dev *d, 

unsigned int ioctl, void __user *arg
                      r = -EFAULT;
                      break;
              }
-              if (s.num > 0xffff) {
-                      r = -EINVAL;
-                      break;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED)) { +                      vq->last_avail_idx = s.num & 0xffff; +                      vq->last_used_idx = (s.num >> 16) & 0xffff; +              } else {
+                      if (s.num > 0xffff) {
+                              r = -EINVAL;
+                              break;
+                      }
+                      vq->last_avail_idx = s.num;
              }
-              vq->last_avail_idx = s.num;
              /* Forget the cached index value. */ vq->avail_idx = vq->last_avail_idx; break;
      case VHOST_GET_VRING_BASE:
              s.index = idx;
-              s.num = vq->last_avail_idx;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED))

Quoted 8 lines
Quoted 4 lines

+                      s.num = (u32)vq->last_avail_idx | ((u32)vq->last_used_idx << 16);
+              else
+                      s.num = vq->last_avail_idx;

The changes LGTM, but since we are changing the UAPI, should we update the documentation of VHOST_SET_VRING_BASE and VHOST_GET_VRING_BASE in include/uapi/linux/vhost.h?

Correct me if I'm wrong, but I don't think we're changing anything in the UAPI here, just fixing code to work correctly with what is already happening.

sln

Quoted 4 lines

Thanks,
Stefano

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#6 lore raw

Re: [PATCH v2 2/3] vhost: support PACKED when setting-getting vring_base

Quoted 50 lines · Shannon Nelson
Quoted 40 lines · Stefano Garzarella
Quoted 39 lines · Shannon Nelson via Virtualization

Use the right structs for PACKED or split vqs when setting and getting the vring base.

---
drivers/vhost/vhost.c | 18 +++++++++++++-----
drivers/vhost/vhost.h |  8 ++++++--

2 files changed, 19 insertions(+), 7 deletions(-)
drivers/vhost/vhost.c +9 -4
index f11bdbe4c2c5..f64efda48f21 100644
@@ -1633,17 +1633,25 @@ long vhost_vring_ioctl(struct vhost_dev 
 *d, unsigned int ioctl, void __user *arg
                       r = -EFAULT;
                       break;
               }
-              if (s.num > 0xffff) {
-                      r = -EINVAL;
-                      break;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED)) {
+                      vq->last_avail_idx = s.num & 0xffff;
+                      vq->last_used_idx = (s.num >> 16) & 0xffff;
+              } else {
+                      if (s.num > 0xffff) {
+                              r = -EINVAL;
+                              break;
+                      }
+                      vq->last_avail_idx = s.num;
               }
-              vq->last_avail_idx = s.num;
               /* Forget the cached index value. */
               vq->avail_idx = vq->last_avail_idx;
               break;

      case VHOST_GET_VRING_BASE:
              s.index = idx;
-              s.num = vq->last_avail_idx;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED))

Quoted 8 lines
Quoted 4 lines

+                      s.num = (u32)vq->last_avail_idx | ((u32)vq->last_used_idx << 16);
+              else
+                      s.num = vq->last_avail_idx;

The changes LGTM, but since we are changing the UAPI, should we update the documentation of VHOST_SET_VRING_BASE and VHOST_GET_VRING_BASE in include/uapi/linux/vhost.h?

Quoted 1 line

Correct me if I'm wrong, but I don't think we're changing anything in the UAPI here, just fixing code to work correctly with what is already happening.

IIUC before this patch VHOST_GET_VRING_BASE and VHOST_SET_VRING_BASE never worked with packed virtqueue, since we were only handling last_avail_idx. Now we are supporting packed virtqueue, handling in vhost_vring_state.num both last_avail_idx and last_used_idx (with wrap counters).

For example for VHOST_GET_VRING_BASE where is documented that the first 15 bits are last_avail_idx, the 16th the avail_wrap_counter, and the others are last_used_idx and used_wrap_counter?

Maybe I missed something, but since this is UAPI, IMHO we should document the parameters of ioctls at least in include/uapi/linux/vhost.h.

Thanks,
Stefano

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#7 lore raw

Re: [PATCH v2 2/3] vhost: support PACKED when setting-getting vring_base

Quoted 62 lines · Stefano Garzarella
Quoted 51 lines · Shannon Nelson
Quoted 41 lines · Stefano Garzarella
Quoted 39 lines · Shannon Nelson via Virtualization

Use the right structs for PACKED or split vqs when setting and getting the vring base.

---
drivers/vhost/vhost.c | 18 +++++++++++++-----
drivers/vhost/vhost.h |  8 ++++++--

2 files changed, 19 insertions(+), 7 deletions(-)
drivers/vhost/vhost.c +9 -4
index f11bdbe4c2c5..f64efda48f21 100644
@@ -1633,17 +1633,25 @@ long vhost_vring_ioctl(struct vhost_dev
 *d, unsigned int ioctl, void __user *arg
                       r = -EFAULT;
                       break;
               }
-              if (s.num > 0xffff) {
-                      r = -EINVAL;
-                      break;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED)) {
+                      vq->last_avail_idx = s.num & 0xffff;
+                      vq->last_used_idx = (s.num >> 16) & 0xffff;
+              } else {
+                      if (s.num > 0xffff) {
+                              r = -EINVAL;
+                              break;
+                      }
+                      vq->last_avail_idx = s.num;
               }
-              vq->last_avail_idx = s.num;
               /* Forget the cached index value. */
               vq->avail_idx = vq->last_avail_idx;
               break;

      case VHOST_GET_VRING_BASE:
              s.index = idx;
-              s.num = vq->last_avail_idx;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED))

Quoted 8 lines
Quoted 4 lines

+                      s.num = (u32)vq->last_avail_idx | ((u32)vq->last_used_idx << 16);
+              else
+                      s.num = vq->last_avail_idx;

The changes LGTM, but since we are changing the UAPI, should we update the documentation of VHOST_SET_VRING_BASE and VHOST_GET_VRING_BASE in include/uapi/linux/vhost.h?

Quoted 3 lines

Correct me if I'm wrong, but I don't think we're changing anything in the UAPI here, just fixing code to work correctly with what is already happening.

IIUC before this patch VHOST_GET_VRING_BASE and VHOST_SET_VRING_BASE never worked with packed virtqueue, since we were only handling last_avail_idx. Now we are supporting packed virtqueue, handling in vhost_vring_state.num both last_avail_idx and last_used_idx (with wrap counters).

Quoted 7 lines

For example for VHOST_GET_VRING_BASE where is documented that the first 15 bits are last_avail_idx, the 16th the avail_wrap_counter, and the others are last_used_idx and used_wrap_counter?

Maybe I missed something, but since this is UAPI, IMHO we should document the parameters of ioctls at least in include/uapi/linux/vhost.h.

Perhaps Jason already has something written up that could be put in here from when he first added the wrap_counter a couple of years ago?

sln

Quoted 8 lines

Thanks,
Stefano

Signature · 3 lines
You received this message because you are subscribed to the Google Groups "Pensando Drivers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit https://groups.google.com/a/pensando.io/d/msgid/drivers/q6cmfha36sdkgflwwd3pr4sw7rgajag4ahgjbpfjrr76w4o2b6%403yc7zs5u65s4.

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#8 lore raw

Re: [PATCH v2 2/3] vhost: support PACKED when setting-getting vring_base

Quoted 68 lines · Shannon Nelson
Quoted 66 lines · Stefano Garzarella
Quoted 51 lines · Shannon Nelson
Quoted 41 lines · Stefano Garzarella
Quoted 39 lines · Shannon Nelson via Virtualization

Use the right structs for PACKED or split vqs when setting and getting the vring base.

---
drivers/vhost/vhost.c | 18 +++++++++++++-----
drivers/vhost/vhost.h | 8 ++++++--

2 files changed, 19 insertions(+), 7 deletions(-)
drivers/vhost/vhost.c +10 -5
index f11bdbe4c2c5..f64efda48f21 100644
@@ -1633,17 +1633,25 @@ long vhost_vring_ioctl(struct vhost_dev
 *d, unsigned int ioctl, void __user *arg
                      r = -EFAULT;
                      break;
              }
-              if (s.num > 0xffff) {
-                      r = -EINVAL;
-                      break;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED)) {
+                      vq->last_avail_idx = s.num & 0xffff;
+                      vq->last_used_idx = (s.num >> 16) & 0xffff;
+              } else {
+                      if (s.num > 0xffff) {
+                              r = -EINVAL;
+                              break;
+                      }
+                      vq->last_avail_idx = s.num;
              }
-              vq->last_avail_idx = s.num;
              /* Forget the cached index value. */
              vq->avail_idx = vq->last_avail_idx;
              break;
      case VHOST_GET_VRING_BASE:
              s.index = idx;
-              s.num = vq->last_avail_idx;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED))
Quoted 8 lines

drivers/vhost/vhost.c long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg · line 1654

Quoted 4 lines
drivers/vhost/vhost.c +3 -0
+                      s.num = (u32)vq->last_avail_idx |
 ((u32)vq->last_used_idx << 16);
+              else
+                      s.num = vq->last_avail_idx;

The changes LGTM, but since we are changing the UAPI, should we update the documentation of VHOST_SET_VRING_BASE and VHOST_GET_VRING_BASE in include/uapi/linux/vhost.h?

Quoted 3 lines

Correct me if I'm wrong, but I don't think we're changing anything in the UAPI here, just fixing code to work correctly with what is already happening.

IIUC before this patch VHOST_GET_VRING_BASE and VHOST_SET_VRING_BASE never worked with packed virtqueue, since we were only handling last_avail_idx. Now we are supporting packed virtqueue, handling in vhost_vring_state.num both last_avail_idx and last_used_idx (with wrap counters).

For example for VHOST_GET_VRING_BASE where is documented that the first 15 bits are last_avail_idx, the 16th the avail_wrap_counter, and the others are last_used_idx and used_wrap_counter?

Quoted 6 lines
Quoted 3 lines

Maybe I missed something, but since this is UAPI, IMHO we should document the parameters of ioctls at least in include/uapi/linux/vhost.h.

Perhaps Jason already has something written up that could be put in here from when he first added the wrap_counter a couple of years ago?

If you meant the virtio driver support for packed, I think it's different from the context which is vhost here.

I agree with Stefano that we need to update the comments around GET_VRING_BASE and SET_VRING_BASE, then we are fine.

Thanks

Quoted 15 lines

sln

Quoted 11 lines

Thanks,
Stefano

--
You received this message because you are subscribed to the Google Groups "Pensando Drivers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit
https://groups.google.com/a/pensando.io/d/msgid/drivers/q6cmfha36sdkgflwwd3pr4sw7rgajag4ahgjbpfjrr76w4o2b6%403yc7zs5u65s4.

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#9 lore raw

Re: [PATCH v2 2/3] vhost: support PACKED when setting-getting vring_base

Quoted 74 lines · Jason Wang
Quoted 72 lines · Shannon Nelson
Quoted 69 lines · Stefano Garzarella
Quoted 51 lines · Shannon Nelson
Quoted 41 lines · Stefano Garzarella
Quoted 39 lines · Shannon Nelson via Virtualization

Use the right structs for PACKED or split vqs when setting and getting the vring base.

---
drivers/vhost/vhost.c | 18 +++++++++++++-----
drivers/vhost/vhost.h | 8 ++++++--

2 files changed, 19 insertions(+), 7 deletions(-)
drivers/vhost/vhost.c +10 -5
index f11bdbe4c2c5..f64efda48f21 100644
@@ -1633,17 +1633,25 @@ long vhost_vring_ioctl(struct vhost_dev
 *d, unsigned int ioctl, void __user *arg
                      r = -EFAULT;
                      break;
              }
-              if (s.num > 0xffff) {
-                      r = -EINVAL;
-                      break;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED)) {
+                      vq->last_avail_idx = s.num & 0xffff;
+                      vq->last_used_idx = (s.num >> 16) & 0xffff;
+              } else {
+                      if (s.num > 0xffff) {
+                              r = -EINVAL;
+                              break;
+                      }
+                      vq->last_avail_idx = s.num;
              }
-              vq->last_avail_idx = s.num;
              /* Forget the cached index value. */
              vq->avail_idx = vq->last_avail_idx;
              break;
      case VHOST_GET_VRING_BASE:
              s.index = idx;
-              s.num = vq->last_avail_idx;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED))
Quoted 8 lines

drivers/vhost/vhost.c long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg · line 1654

Quoted 4 lines
drivers/vhost/vhost.c +3 -0
+                      s.num = (u32)vq->last_avail_idx |
 ((u32)vq->last_used_idx << 16);
+              else
+                      s.num = vq->last_avail_idx;

The changes LGTM, but since we are changing the UAPI, should we update the documentation of VHOST_SET_VRING_BASE and VHOST_GET_VRING_BASE in include/uapi/linux/vhost.h?

Quoted 3 lines

Correct me if I'm wrong, but I don't think we're changing anything in the UAPI here, just fixing code to work correctly with what is already happening.

IIUC before this patch VHOST_GET_VRING_BASE and VHOST_SET_VRING_BASE never worked with packed virtqueue, since we were only handling last_avail_idx. Now we are supporting packed virtqueue, handling in vhost_vring_state.num both last_avail_idx and last_used_idx (with wrap counters).

For example for VHOST_GET_VRING_BASE where is documented that the first 15 bits are last_avail_idx, the 16th the avail_wrap_counter, and the others are last_used_idx and used_wrap_counter?

Maybe I missed something, but since this is UAPI, IMHO we should document the parameters of ioctls at least in include/uapi/linux/vhost.h.

Quoted 8 lines
Quoted 2 lines

Perhaps Jason already has something written up that could be put in here from when he first added the wrap_counter a couple of years ago?

If you meant the virtio driver support for packed, I think it's different from the context which is vhost here.

I agree with Stefano that we need to update the comments around GET_VRING_BASE and SET_VRING_BASE, then we are fine.

I'm thinking if we should also add a new VHOST_BACKEND_F_RING_PACKED feature (or something similar) to inform the user space that now we are able to handle packed virtqueue through vhost IOCTLs, otherwise how can the userspace know if it is supported or not?

Thanks,
Stefano

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#10 lore raw

Re: [PATCH v2 2/3] vhost: support PACKED when setting-getting vring_base

Quoted 82 lines · Stefano Garzarella
Quoted 80 lines · Jason Wang
Quoted 68 lines · Shannon Nelson
Quoted 66 lines · Stefano Garzarella
Quoted 51 lines · Shannon Nelson
Quoted 41 lines · Stefano Garzarella
Quoted 39 lines · Shannon Nelson via Virtualization

Use the right structs for PACKED or split vqs when setting and getting the vring base.

---
drivers/vhost/vhost.c | 18 +++++++++++++-----
drivers/vhost/vhost.h | 8 ++++++--

2 files changed, 19 insertions(+), 7 deletions(-)
drivers/vhost/vhost.c +10 -5
index f11bdbe4c2c5..f64efda48f21 100644
@@ -1633,17 +1633,25 @@ long vhost_vring_ioctl(struct vhost_dev
 *d, unsigned int ioctl, void __user *arg
                      r = -EFAULT;
                      break;
              }
-              if (s.num > 0xffff) {
-                      r = -EINVAL;
-                      break;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED)) {
+                      vq->last_avail_idx = s.num & 0xffff;
+                      vq->last_used_idx = (s.num >> 16) & 0xffff;
+              } else {
+                      if (s.num > 0xffff) {
+                              r = -EINVAL;
+                              break;
+                      }
+                      vq->last_avail_idx = s.num;
              }
-              vq->last_avail_idx = s.num;
              /* Forget the cached index value. */
              vq->avail_idx = vq->last_avail_idx;
              break;
      case VHOST_GET_VRING_BASE:
              s.index = idx;
-              s.num = vq->last_avail_idx;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED))
Quoted 8 lines

drivers/vhost/vhost.c long vhost_vring_ioctl(struct vhost_dev · line 1656

Quoted 4 lines
drivers/vhost/vhost.c +3 -0
+                      s.num = (u32)vq->last_avail_idx |
 ((u32)vq->last_used_idx << 16);
+              else
+                      s.num = vq->last_avail_idx;

The changes LGTM, but since we are changing the UAPI, should we update the documentation of VHOST_SET_VRING_BASE and VHOST_GET_VRING_BASE in include/uapi/linux/vhost.h?

Quoted 3 lines

Correct me if I'm wrong, but I don't think we're changing anything in the UAPI here, just fixing code to work correctly with what is already happening.

IIUC before this patch VHOST_GET_VRING_BASE and VHOST_SET_VRING_BASE never worked with packed virtqueue, since we were only handling last_avail_idx. Now we are supporting packed virtqueue, handling in vhost_vring_state.num both last_avail_idx and last_used_idx (with wrap counters).

For example for VHOST_GET_VRING_BASE where is documented that the first 15 bits are last_avail_idx, the 16th the avail_wrap_counter, and the others are last_used_idx and used_wrap_counter?

Quoted 6 lines
Quoted 3 lines

Maybe I missed something, but since this is UAPI, IMHO we should document the parameters of ioctls at least in include/uapi/linux/vhost.h.

Perhaps Jason already has something written up that could be put in here from when he first added the wrap_counter a couple of years ago?

If you meant the virtio driver support for packed, I think it's different from the context which is vhost here.

Quoted 7 lines
Quoted 2 lines

I agree with Stefano that we need to update the comments around GET_VRING_BASE and SET_VRING_BASE, then we are fine.

I'm thinking if we should also add a new VHOST_BACKEND_F_RING_PACKED feature (or something similar) to inform the user space that now we are able to handle packed virtqueue through vhost IOCTLs, otherwise how can the userspace know if it is supported or not?

I probably understand this but I think it should be done via VHOST_GET_FEAETURES. It would be a burden if we matianing duplicated features.

Thanks

Quoted 4 lines

Thanks,
Stefano

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#11 lore raw

Re: [PATCH v2 2/3] vhost: support PACKED when setting-getting vring_base

Quoted 87 lines · Jason Wang
Quoted 85 lines · Stefano Garzarella
Quoted 82 lines · Jason Wang
Quoted 68 lines · Shannon Nelson
Quoted 66 lines · Stefano Garzarella
Quoted 51 lines · Shannon Nelson
Quoted 41 lines · Stefano Garzarella
Quoted 39 lines · Shannon Nelson via Virtualization

Use the right structs for PACKED or split vqs when setting and getting the vring base.

---
drivers/vhost/vhost.c | 18 +++++++++++++-----
drivers/vhost/vhost.h | 8 ++++++--

2 files changed, 19 insertions(+), 7 deletions(-)
drivers/vhost/vhost.c +10 -5
index f11bdbe4c2c5..f64efda48f21 100644
@@ -1633,17 +1633,25 @@ long vhost_vring_ioctl(struct vhost_dev
 *d, unsigned int ioctl, void __user *arg
                      r = -EFAULT;
                      break;
              }
-              if (s.num > 0xffff) {
-                      r = -EINVAL;
-                      break;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED)) {
+                      vq->last_avail_idx = s.num & 0xffff;
+                      vq->last_used_idx = (s.num >> 16) & 0xffff;
+              } else {
+                      if (s.num > 0xffff) {
+                              r = -EINVAL;
+                              break;
+                      }
+                      vq->last_avail_idx = s.num;
              }
-              vq->last_avail_idx = s.num;
              /* Forget the cached index value. */
              vq->avail_idx = vq->last_avail_idx;
              break;
      case VHOST_GET_VRING_BASE:
              s.index = idx;
-              s.num = vq->last_avail_idx;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED))
Quoted 8 lines

drivers/vhost/vhost.c long vhost_vring_ioctl(struct vhost_dev · line 1656

Quoted 4 lines
drivers/vhost/vhost.c +3 -0
+                      s.num = (u32)vq->last_avail_idx |
 ((u32)vq->last_used_idx << 16);
+              else
+                      s.num = vq->last_avail_idx;

The changes LGTM, but since we are changing the UAPI, should we update the documentation of VHOST_SET_VRING_BASE and VHOST_GET_VRING_BASE in include/uapi/linux/vhost.h?

Quoted 3 lines

Correct me if I'm wrong, but I don't think we're changing anything in the UAPI here, just fixing code to work correctly with what is already happening.

IIUC before this patch VHOST_GET_VRING_BASE and VHOST_SET_VRING_BASE never worked with packed virtqueue, since we were only handling last_avail_idx. Now we are supporting packed virtqueue, handling in vhost_vring_state.num both last_avail_idx and last_used_idx (with wrap counters).

For example for VHOST_GET_VRING_BASE where is documented that the first 15 bits are last_avail_idx, the 16th the avail_wrap_counter, and the others are last_used_idx and used_wrap_counter?

Quoted 6 lines
Quoted 3 lines

Maybe I missed something, but since this is UAPI, IMHO we should document the parameters of ioctls at least in include/uapi/linux/vhost.h.

Perhaps Jason already has something written up that could be put in here from when he first added the wrap_counter a couple of years ago?

If you meant the virtio driver support for packed, I think it's different from the context which is vhost here.

I agree with Stefano that we need to update the comments around GET_VRING_BASE and SET_VRING_BASE, then we are fine.

Quoted 8 lines
Quoted 4 lines

I'm thinking if we should also add a new VHOST_BACKEND_F_RING_PACKED feature (or something similar) to inform the user space that now we are able to handle packed virtqueue through vhost IOCTLs, otherwise how can the userspace know if it is supported or not?

I probably understand this but I think it should be done via VHOST_GET_FEAETURES. It would be a burden if we matianing duplicated features.

Good point, I see.

I think we should do one of these things, though:
- mask VIRTIO_F_RING_PACKED in the stable kernels when VHOST_GET_FEAETURES is called
- backport this patch on all stable kernels that support vhost-vdpa

Maybe the last one makes more sense.

Thanks,
Stefano

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#12 lore raw

Re: [PATCH v2 2/3] vhost: support PACKED when setting-getting vring_base

Quoted 98 lines · Stefano Garzarella
Quoted 87 lines · Jason Wang
Quoted 85 lines · Stefano Garzarella
Quoted 82 lines · Jason Wang
Quoted 68 lines · Shannon Nelson
Quoted 66 lines · Stefano Garzarella
Quoted 51 lines · Shannon Nelson
Quoted 41 lines · Stefano Garzarella
Quoted 39 lines · Shannon Nelson via Virtualization

Use the right structs for PACKED or split vqs when setting and getting the vring base.

---
drivers/vhost/vhost.c | 18 +++++++++++++-----
drivers/vhost/vhost.h | 8 ++++++--

2 files changed, 19 insertions(+), 7 deletions(-)
drivers/vhost/vhost.c +10 -5
index f11bdbe4c2c5..f64efda48f21 100644
@@ -1633,17 +1633,25 @@ long vhost_vring_ioctl(struct vhost_dev
 *d, unsigned int ioctl, void __user *arg
                      r = -EFAULT;
                      break;
              }
-              if (s.num > 0xffff) {
-                      r = -EINVAL;
-                      break;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED)) {
+                      vq->last_avail_idx = s.num & 0xffff;
+                      vq->last_used_idx = (s.num >> 16) & 0xffff;
+              } else {
+                      if (s.num > 0xffff) {
+                              r = -EINVAL;
+                              break;
+                      }
+                      vq->last_avail_idx = s.num;
              }
-              vq->last_avail_idx = s.num;
              /* Forget the cached index value. */
              vq->avail_idx = vq->last_avail_idx;
              break;
      case VHOST_GET_VRING_BASE:
              s.index = idx;
-              s.num = vq->last_avail_idx;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED))
Quoted 8 lines

drivers/vhost/vhost.c long vhost_vring_ioctl(struct vhost_dev · line 1656

Quoted 4 lines
drivers/vhost/vhost.c +3 -0
+                      s.num = (u32)vq->last_avail_idx |
 ((u32)vq->last_used_idx << 16);
+              else
+                      s.num = vq->last_avail_idx;

The changes LGTM, but since we are changing the UAPI, should we update the documentation of VHOST_SET_VRING_BASE and VHOST_GET_VRING_BASE in include/uapi/linux/vhost.h?

Quoted 3 lines

Correct me if I'm wrong, but I don't think we're changing anything in the UAPI here, just fixing code to work correctly with what is already happening.

IIUC before this patch VHOST_GET_VRING_BASE and VHOST_SET_VRING_BASE never worked with packed virtqueue, since we were only handling last_avail_idx. Now we are supporting packed virtqueue, handling in vhost_vring_state.num both last_avail_idx and last_used_idx (with wrap counters).

For example for VHOST_GET_VRING_BASE where is documented that the first 15 bits are last_avail_idx, the 16th the avail_wrap_counter, and the others are last_used_idx and used_wrap_counter?

Quoted 6 lines
Quoted 3 lines

Maybe I missed something, but since this is UAPI, IMHO we should document the parameters of ioctls at least in include/uapi/linux/vhost.h.

Perhaps Jason already has something written up that could be put in here from when he first added the wrap_counter a couple of years ago?

If you meant the virtio driver support for packed, I think it's different from the context which is vhost here.

I agree with Stefano that we need to update the comments around GET_VRING_BASE and SET_VRING_BASE, then we are fine.

Quoted 8 lines
Quoted 4 lines

I'm thinking if we should also add a new VHOST_BACKEND_F_RING_PACKED feature (or something similar) to inform the user space that now we are able to handle packed virtqueue through vhost IOCTLs, otherwise how can the userspace know if it is supported or not?

I probably understand this but I think it should be done via VHOST_GET_FEAETURES. It would be a burden if we matianing duplicated features.

Quoted 8 lines

Good point, I see.

I think we should do one of these things, though:
- mask VIRTIO_F_RING_PACKED in the stable kernels when VHOST_GET_FEAETURES is called
- backport this patch on all stable kernels that support vhost-vdpa

Maybe the last one makes more sense.

Not sure, it looks to me the packed support for vDPA was first added by Gautam. So it probably means that except for vp_vdpa, we don't have a vDPA parent that can do the packed virtuque now. Adding the relevant people here for more comment

Thanks

Quoted 4 lines

Thanks,
Stefano

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#13 lore raw

Re: [PATCH v2 2/3] vhost: support PACKED when setting-getting vring_base

Quoted 106 lines · Jason Wang
Quoted 105 lines · Stefano Garzarella
Quoted 87 lines · Jason Wang
Quoted 85 lines · Stefano Garzarella
Quoted 82 lines · Jason Wang
Quoted 68 lines · Shannon Nelson
Quoted 66 lines · Stefano Garzarella
Quoted 51 lines · Shannon Nelson
Quoted 41 lines · Stefano Garzarella
Quoted 39 lines · Shannon Nelson via Virtualization

Use the right structs for PACKED or split vqs when setting and getting the vring base.

---
drivers/vhost/vhost.c | 18 +++++++++++++-----
drivers/vhost/vhost.h | 8 ++++++--

2 files changed, 19 insertions(+), 7 deletions(-)
drivers/vhost/vhost.c +10 -5
index f11bdbe4c2c5..f64efda48f21 100644
@@ -1633,17 +1633,25 @@ long vhost_vring_ioctl(struct vhost_dev
 *d, unsigned int ioctl, void __user *arg
                      r = -EFAULT;
                      break;
              }
-              if (s.num > 0xffff) {
-                      r = -EINVAL;
-                      break;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED)) {
+                      vq->last_avail_idx = s.num & 0xffff;
+                      vq->last_used_idx = (s.num >> 16) & 0xffff;
+              } else {
+                      if (s.num > 0xffff) {
+                              r = -EINVAL;
+                              break;
+                      }
+                      vq->last_avail_idx = s.num;
              }
-              vq->last_avail_idx = s.num;
              /* Forget the cached index value. */
              vq->avail_idx = vq->last_avail_idx;
              break;
      case VHOST_GET_VRING_BASE:
              s.index = idx;
-              s.num = vq->last_avail_idx;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED))
Quoted 8 lines

drivers/vhost/vhost.c long vhost_vring_ioctl(struct vhost_dev · line 1656

Quoted 4 lines
drivers/vhost/vhost.c +3 -0
+                      s.num = (u32)vq->last_avail_idx |
 ((u32)vq->last_used_idx << 16);
+              else
+                      s.num = vq->last_avail_idx;

The changes LGTM, but since we are changing the UAPI, should we update the documentation of VHOST_SET_VRING_BASE and VHOST_GET_VRING_BASE in include/uapi/linux/vhost.h?

Quoted 3 lines

Correct me if I'm wrong, but I don't think we're changing anything in the UAPI here, just fixing code to work correctly with what is already happening.

IIUC before this patch VHOST_GET_VRING_BASE and VHOST_SET_VRING_BASE never worked with packed virtqueue, since we were only handling last_avail_idx. Now we are supporting packed virtqueue, handling in vhost_vring_state.num both last_avail_idx and last_used_idx (with wrap counters).

For example for VHOST_GET_VRING_BASE where is documented that the first 15 bits are last_avail_idx, the 16th the avail_wrap_counter, and the others are last_used_idx and used_wrap_counter?

Quoted 6 lines
Quoted 3 lines

Maybe I missed something, but since this is UAPI, IMHO we should document the parameters of ioctls at least in include/uapi/linux/vhost.h.

Perhaps Jason already has something written up that could be put in here from when he first added the wrap_counter a couple of years ago?

If you meant the virtio driver support for packed, I think it's different from the context which is vhost here.

I agree with Stefano that we need to update the comments around GET_VRING_BASE and SET_VRING_BASE, then we are fine.

Quoted 8 lines
Quoted 4 lines

I'm thinking if we should also add a new VHOST_BACKEND_F_RING_PACKED feature (or something similar) to inform the user space that now we are able to handle packed virtqueue through vhost IOCTLs, otherwise how can the userspace know if it is supported or not?

I probably understand this but I think it should be done via VHOST_GET_FEAETURES. It would be a burden if we matianing duplicated features.

Good point, I see.

I think we should do one of these things, though:
- mask VIRTIO_F_RING_PACKED in the stable kernels when VHOST_GET_FEAETURES is called
- backport this patch on all stable kernels that support vhost-vdpa

Quoted 8 lines
Quoted 1 line

Maybe the last one makes more sense.

Not sure, it looks to me the packed support for vDPA was first added by Gautam. So it probably means that except for vp_vdpa, we don't have a vDPA parent that can do the packed virtuque now. Adding the relevant people here for more comment

Thanks

BTW should we fix up vhost.c to support packed rings too? E.g. so we can migrate to vhost?
There's an old patchset of mine that started work on this:

https://lore.kernel.org/all/20200407011612.478226-1-mst%40redhat.com

Is there need for this now?

Quoted 4 lines
Quoted 4 lines

Thanks,
Stefano

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#14 lore raw

Re: [PATCH v2 2/3] vhost: support PACKED when setting-getting vring_base

Quoted 115 lines · Michael S. Tsirkin
Quoted 113 lines · Jason Wang
Quoted 98 lines · Stefano Garzarella
Quoted 87 lines · Jason Wang
Quoted 85 lines · Stefano Garzarella
Quoted 82 lines · Jason Wang
Quoted 68 lines · Shannon Nelson
Quoted 66 lines · Stefano Garzarella
Quoted 51 lines · Shannon Nelson
Quoted 41 lines · Stefano Garzarella
Quoted 39 lines · Shannon Nelson via Virtualization

Use the right structs for PACKED or split vqs when setting and getting the vring base.

---
drivers/vhost/vhost.c | 18 +++++++++++++-----
drivers/vhost/vhost.h | 8 ++++++--

2 files changed, 19 insertions(+), 7 deletions(-)
drivers/vhost/vhost.c +10 -5
index f11bdbe4c2c5..f64efda48f21 100644
@@ -1633,17 +1633,25 @@ long vhost_vring_ioctl(struct vhost_dev
 *d, unsigned int ioctl, void __user *arg
                      r = -EFAULT;
                      break;
              }
-              if (s.num > 0xffff) {
-                      r = -EINVAL;
-                      break;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED)) {
+                      vq->last_avail_idx = s.num & 0xffff;
+                      vq->last_used_idx = (s.num >> 16) & 0xffff;
+              } else {
+                      if (s.num > 0xffff) {
+                              r = -EINVAL;
+                              break;
+                      }
+                      vq->last_avail_idx = s.num;
              }
-              vq->last_avail_idx = s.num;
              /* Forget the cached index value. */
              vq->avail_idx = vq->last_avail_idx;
              break;
      case VHOST_GET_VRING_BASE:
              s.index = idx;
-              s.num = vq->last_avail_idx;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED))
Quoted 8 lines

drivers/vhost/vhost.c long vhost_vring_ioctl(struct vhost_dev · line 1656

Quoted 4 lines
drivers/vhost/vhost.c +3 -0
+                      s.num = (u32)vq->last_avail_idx |
 ((u32)vq->last_used_idx << 16);
+              else
+                      s.num = vq->last_avail_idx;

The changes LGTM, but since we are changing the UAPI, should we update the documentation of VHOST_SET_VRING_BASE and VHOST_GET_VRING_BASE in include/uapi/linux/vhost.h?

Quoted 3 lines

Correct me if I'm wrong, but I don't think we're changing anything in the UAPI here, just fixing code to work correctly with what is already happening.

IIUC before this patch VHOST_GET_VRING_BASE and VHOST_SET_VRING_BASE never worked with packed virtqueue, since we were only handling last_avail_idx. Now we are supporting packed virtqueue, handling in vhost_vring_state.num both last_avail_idx and last_used_idx (with wrap counters).

For example for VHOST_GET_VRING_BASE where is documented that the first 15 bits are last_avail_idx, the 16th the avail_wrap_counter, and the others are last_used_idx and used_wrap_counter?

Quoted 6 lines
Quoted 3 lines

Maybe I missed something, but since this is UAPI, IMHO we should document the parameters of ioctls at least in include/uapi/linux/vhost.h.

Perhaps Jason already has something written up that could be put in here from when he first added the wrap_counter a couple of years ago?

If you meant the virtio driver support for packed, I think it's different from the context which is vhost here.

I agree with Stefano that we need to update the comments around GET_VRING_BASE and SET_VRING_BASE, then we are fine.

Quoted 8 lines
Quoted 4 lines

I'm thinking if we should also add a new VHOST_BACKEND_F_RING_PACKED feature (or something similar) to inform the user space that now we are able to handle packed virtqueue through vhost IOCTLs, otherwise how can the userspace know if it is supported or not?

I probably understand this but I think it should be done via VHOST_GET_FEAETURES. It would be a burden if we matianing duplicated features.

Quoted 8 lines

Good point, I see.

I think we should do one of these things, though:
- mask VIRTIO_F_RING_PACKED in the stable kernels when VHOST_GET_FEAETURES is called
- backport this patch on all stable kernels that support vhost-vdpa

Maybe the last one makes more sense.

Not sure, it looks to me the packed support for vDPA was first added by Gautam. So it probably means that except for vp_vdpa, we don't have a vDPA parent that can do the packed virtuque now. Adding the relevant people here for more comment

Quoted 4 lines
Quoted 1 line

Thanks

BTW should we fix up vhost.c to support packed rings too? E.g. so we can migrate to vhost?

Better to have.

Quoted 5 lines

There's an old patchset of mine that started work on this:

https://lore.kernel.org/all/20200407011612.478226-1-mst%40redhat.com

Is there need for this now?

Is there any benchmark for that?

Thanks

Quoted 6 lines
Quoted 5 lines
Quoted 5 lines

Thanks,
Stefano

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#15 lore raw

Re: [PATCH v2 2/3] vhost: support PACKED when setting-getting vring_base

Quoted 125 lines · Jason Wang
Quoted 115 lines · Michael S. Tsirkin
Quoted 113 lines · Jason Wang
Quoted 98 lines · Stefano Garzarella
Quoted 87 lines · Jason Wang
Quoted 85 lines · Stefano Garzarella
Quoted 82 lines · Jason Wang
Quoted 68 lines · Shannon Nelson
Quoted 66 lines · Stefano Garzarella
Quoted 51 lines · Shannon Nelson
Quoted 41 lines · Stefano Garzarella
Quoted 39 lines · Shannon Nelson via Virtualization

Use the right structs for PACKED or split vqs when setting and getting the vring base.

---
drivers/vhost/vhost.c | 18 +++++++++++++-----
drivers/vhost/vhost.h | 8 ++++++--

2 files changed, 19 insertions(+), 7 deletions(-)
drivers/vhost/vhost.c +10 -5
index f11bdbe4c2c5..f64efda48f21 100644
@@ -1633,17 +1633,25 @@ long vhost_vring_ioctl(struct vhost_dev
 *d, unsigned int ioctl, void __user *arg
                      r = -EFAULT;
                      break;
              }
-              if (s.num > 0xffff) {
-                      r = -EINVAL;
-                      break;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED)) {
+                      vq->last_avail_idx = s.num & 0xffff;
+                      vq->last_used_idx = (s.num >> 16) & 0xffff;
+              } else {
+                      if (s.num > 0xffff) {
+                              r = -EINVAL;
+                              break;
+                      }
+                      vq->last_avail_idx = s.num;
              }
-              vq->last_avail_idx = s.num;
              /* Forget the cached index value. */
              vq->avail_idx = vq->last_avail_idx;
              break;
      case VHOST_GET_VRING_BASE:
              s.index = idx;
-              s.num = vq->last_avail_idx;
+              if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED))
Quoted 8 lines

drivers/vhost/vhost.c long vhost_vring_ioctl(struct vhost_dev · line 1656

Quoted 4 lines
drivers/vhost/vhost.c +3 -0
+                      s.num = (u32)vq->last_avail_idx |
 ((u32)vq->last_used_idx << 16);
+              else
+                      s.num = vq->last_avail_idx;

The changes LGTM, but since we are changing the UAPI, should we update the documentation of VHOST_SET_VRING_BASE and VHOST_GET_VRING_BASE in include/uapi/linux/vhost.h?

Quoted 3 lines

Correct me if I'm wrong, but I don't think we're changing anything in the UAPI here, just fixing code to work correctly with what is already happening.

IIUC before this patch VHOST_GET_VRING_BASE and VHOST_SET_VRING_BASE never worked with packed virtqueue, since we were only handling last_avail_idx. Now we are supporting packed virtqueue, handling in vhost_vring_state.num both last_avail_idx and last_used_idx (with wrap counters).

For example for VHOST_GET_VRING_BASE where is documented that the first 15 bits are last_avail_idx, the 16th the avail_wrap_counter, and the others are last_used_idx and used_wrap_counter?

Quoted 6 lines
Quoted 3 lines

Maybe I missed something, but since this is UAPI, IMHO we should document the parameters of ioctls at least in include/uapi/linux/vhost.h.

Perhaps Jason already has something written up that could be put in here from when he first added the wrap_counter a couple of years ago?

If you meant the virtio driver support for packed, I think it's different from the context which is vhost here.

I agree with Stefano that we need to update the comments around GET_VRING_BASE and SET_VRING_BASE, then we are fine.

Quoted 8 lines
Quoted 4 lines

I'm thinking if we should also add a new VHOST_BACKEND_F_RING_PACKED feature (or something similar) to inform the user space that now we are able to handle packed virtqueue through vhost IOCTLs, otherwise how can the userspace know if it is supported or not?

I probably understand this but I think it should be done via VHOST_GET_FEAETURES. It would be a burden if we matianing duplicated features.

Quoted 8 lines

Good point, I see.

I think we should do one of these things, though:
- mask VIRTIO_F_RING_PACKED in the stable kernels when VHOST_GET_FEAETURES is called
- backport this patch on all stable kernels that support vhost-vdpa

Maybe the last one makes more sense.

Not sure, it looks to me the packed support for vDPA was first added by Gautam. So it probably means that except for vp_vdpa, we don't have a vDPA parent that can do the packed virtuque now. Adding the relevant people here for more comment

Quoted 4 lines
Quoted 1 line

Thanks

BTW should we fix up vhost.c to support packed rings too? E.g. so we can migrate to vhost?

Better to have.

Quoted 2 lines

There's an old patchset of mine that started work on this:

Quoted 7 lines
Quoted 3 lines

https://lore.kernel.org/all/20200407011612.478226-1-mst%40redhat.com

Is there need for this now?

Is there any benchmark for that?

Thanks

I used one in tools/virtio, there was no real change. Might make sense to remake it since we fixed some bugs in EVENT_IDX support and things might have changed.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines

Thanks,
Stefano

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#16 lore raw

Re: [PATCH v2 2/3] vhost: support PACKED when setting-getting vring_base

In reply to Stefano Garzarella
Quoted 9 lines · Stefano Garzarella

I think we should do one of these things, though:
- mask VIRTIO_F_RING_PACKED in the stable kernels when VHOST_GET_FEAETURES is called
- backport this patch on all stable kernels that support vhost-vdpa

Maybe the last one makes more sense.

Thanks,
Stefano

OK which patches do you want to go to stable exactly?

Signature · 6 lines
MST

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
#17 lore raw

Re: [PATCH v2 2/3] vhost: support PACKED when setting-getting vring_base

Quoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Stefano Garzarella

I think we should do one of these things, though:
- mask VIRTIO_F_RING_PACKED in the stable kernels when VHOST_GET_FEAETURES is called
- backport this patch on all stable kernels that support vhost-vdpa

Quoted 6 lines
Quoted 4 lines

Maybe the last one makes more sense.

Thanks,
Stefano

OK which patches do you want to go to stable exactly?

Initially I was thinking this entire series, but I think it's too risky, so on second thoughtwhat do you think about this:

drivers/vhost/vdpa.c +6 -0
index 8c1aefc865f0..ac2152135b23 100644
@@ -397,6 +397,12 @@ static long vhost_vdpa_get_features(struct vhost_vdpa *v, u64 __user *featurep)
 
        features = ops->get_device_features(vdpa);
 
+       /*
+        * IOCTLs (eg. VHOST_GET_VRING_BASE, VHOST_SET_VRING_BASE) don't support
+        * packed virtqueue well yet, so let's filter the feature for now.
+        */
+       features &= ~BIT_ULL(VIRTIO_F_RING_PACKED);
+
        if (copy_to_user(featurep, &features, sizeof(features)))
                return -EFAULT;
 

I can send the patch ASAP and we can apply it before this series.

Thanks,
Stefano

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#18 lore raw

PATCH v2 3/3 vhost_vdpa: support PACKED when setting-getting vring_base

In reply to Shannon Nelson via Virtualization

Use the right structs for PACKED or split vqs when setting and getting the vring base.

---
 drivers/vhost/vdpa.c | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)
drivers/vhost/vdpa.c +17 -4
index 599b8cc238c7..fe392b67d5be 100644
@@ -585,7 +585,14 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
 		if (r)
 			return r;
 
-		vq->last_avail_idx = vq_state.split.avail_index;
+		if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED)) {
+			vq->last_avail_idx = vq_state.packed.last_avail_idx |
+					     (vq_state.packed.last_avail_counter << 15);
+			vq->last_used_idx = vq_state.packed.last_used_idx |
+					    (vq_state.packed.last_used_counter << 15);
+		} else {
+			vq->last_avail_idx = vq_state.split.avail_index;
+		}
 		break;
 	}
 
@@ -603,9 +610,15 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
 		break;
 
 	case VHOST_SET_VRING_BASE:
-		vq_state.split.avail_index = vq->last_avail_idx;
-		if (ops->set_vq_state(vdpa, idx, &vq_state))
-			r = -EINVAL;
+		if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED)) {
+			vq_state.packed.last_avail_idx = vq->last_avail_idx & 0x7fff;
+			vq_state.packed.last_avail_counter = !!(vq->last_avail_idx & 0x8000);
+			vq_state.packed.last_used_idx = vq->last_used_idx & 0x7fff;
+			vq_state.packed.last_used_counter = !!(vq->last_used_idx & 0x8000);
+		} else {
+			vq_state.split.avail_index = vq->last_avail_idx;
+		}
+		r = ops->set_vq_state(vdpa, idx, &vq_state);
 		break;
 
 	case VHOST_SET_VRING_CALL:
Signature · 6 lines
2.17.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Acked-by on v2 3/3
#19 lore raw

Re: [PATCH v2 3/3] vhost_vdpa: support PACKED when setting-getting vring_base

Quoted 5 lines · Shannon Nelson

Use the right structs for PACKED or split vqs when setting and getting the vring base.

Thanks

Quoted 44 lines
---
 drivers/vhost/vdpa.c | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)
drivers/vhost/vdpa.c +17 -5
index 599b8cc238c7..fe392b67d5be 100644
@@ -585,7 +585,14 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
                if (r)
                        return r;
 
-               vq->last_avail_idx = vq_state.split.avail_index;
+               if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED)) {
+                       vq->last_avail_idx = vq_state.packed.last_avail_idx |
+                                            (vq_state.packed.last_avail_counter << 15);
+                       vq->last_used_idx = vq_state.packed.last_used_idx |
+                                           (vq_state.packed.last_used_counter << 15);
+               } else {
+                       vq->last_avail_idx = vq_state.split.avail_index;
+               }
                break;
        }
 
@@ -603,9 +610,15 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
                break;
 
        case VHOST_SET_VRING_BASE:
-               vq_state.split.avail_index = vq->last_avail_idx;
-               if (ops->set_vq_state(vdpa, idx, &vq_state))
-                       r = -EINVAL;
+               if (vhost_has_feature(vq, VIRTIO_F_RING_PACKED)) {
+                       vq_state.packed.last_avail_idx = vq->last_avail_idx & 0x7fff;
+                       vq_state.packed.last_avail_counter = !!(vq->last_avail_idx & 0x8000);
+                       vq_state.packed.last_used_idx = vq->last_used_idx & 0x7fff;
+                       vq_state.packed.last_used_counter = !!(vq->last_used_idx & 0x8000);
+               } else {
+                       vq_state.split.avail_index = vq->last_avail_idx;
+               }
+               r = ops->set_vq_state(vdpa, idx, &vq_state);
                break;
 
        case VHOST_SET_VRING_CALL:
--
Quoted 2 lines

2.17.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#20 lore raw
In reply to Shannon Nelson via Virtualization
Quoted 6 lines · Shannon Nelson

While testing our vDPA driver with QEMU we found that vhost_vdpa was missing some support for PACKED VQs. Adding these helped us get further in our testing. The first patch makes sure that the vhost_vdpa vqs are given the feature flags, as done in other vhost variants. The second and third patches use the feature flags to properly format and pass set/get_vq requests.

This missed the merge window most likely, unless there's a reason to send a second pull towards the end of it. We'll see.

Quoted 13 lines

v1:
- included wrap counter in packing answers for VHOST_GET_VRING_BASE
- added comments to vhost_virtqueue fields last_avail_idx and last_used_idx

v0:

   https://lists.linuxfoundation.org/pipermail/virtualization/2023-April/066253.html

- initial version

Shannon Nelson (3):
vhost_vdpa: tell vqs about the negotiated
vhost: support PACKED when setting-getting vring_base
vhost_vdpa: support PACKED when setting-getting vring_base

Quoted 7 lines
 drivers/vhost/vdpa.c  | 34 ++++++++++++++++++++++++++++++----
 drivers/vhost/vhost.c | 18 +++++++++++++-----
 drivers/vhost/vhost.h |  8 ++++++--
 3 files changed, 49 insertions(+), 11 deletions(-)
Signature · 1 line
2.17.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#21 lore raw
Quoted 11 lines · Michael S. Tsirkin
Quoted 6 lines · Shannon Nelson

While testing our vDPA driver with QEMU we found that vhost_vdpa was missing some support for PACKED VQs. Adding these helped us get further in our testing. The first patch makes sure that the vhost_vdpa vqs are given the feature flags, as done in other vhost variants. The second and third patches use the feature flags to properly format and pass set/get_vq requests.

This missed the merge window most likely, unless there's a reason to send a second pull towards the end of it. We'll see.

Understood - we'll keep our fingers crossed for good luck :-) Thanks for the help.

sln

Quoted 14 lines
Quoted 14 lines

v1:
- included wrap counter in packing answers for VHOST_GET_VRING_BASE
- added comments to vhost_virtqueue fields last_avail_idx and last_used_idx

v0:

    https://lists.linuxfoundation.org/pipermail/virtualization/2023-April/066253.html

- initial version

Shannon Nelson (3):
vhost_vdpa: tell vqs about the negotiated
vhost: support PACKED when setting-getting vring_base
vhost_vdpa: support PACKED when setting-getting vring_base

Quoted 8 lines
Quoted 7 lines
  drivers/vhost/vdpa.c  | 34 ++++++++++++++++++++++++++++++----
  drivers/vhost/vhost.c | 18 +++++++++++++-----
  drivers/vhost/vhost.h |  8 ++++++--
  3 files changed, 49 insertions(+), 11 deletions(-)

--
2.17.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization