Skip to contentSkip to conversation

Implement vdpasim stop operation

PATCH Superseded Eugenio Pérez 6 participants 42 messages May 26, 2022 – Jun 17, 2022

Subsystems (5)

v4 of 8 Superseded Latest is v8 →

lore
Share & tools
lore

Eugenio Pérez

author maintainer
#0 lore raw

v4 cover Implement vdpasim stop operation

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration, since the device could modify them after userland gets them. There are individual ways to perform that action for some devices (VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any pending operations like in flight requests. It must also preserve all the necessary state (the virtqueue vring base plus the possible device specific states) that is required for restoring in the future. The device must not change its configuration after that point.

After the return of ioctl with stop == 0, the device can continue processing buffers as long as typical conditions are met (vq is enabled, DRIVER_OK status bit is enabled, etc).

In the future, we will provide features similar to VHOST_USER_GET_INFLIGHT_FD so the device can save pending operations.

Comments are welcome.

v4:
* Replace VHOST_STOP to VHOST_VDPA_STOP in vhost ioctl switch case too.

v3:
* s/VHOST_STOP/VHOST_VDPA_STOP/
* Add documentation and requirements of the ioctl above its definition.

v2:
* Replace raw _F_STOP with BIT_ULL(_F_STOP).
* Fix obtaining of stop ioctl arg (it was not obtained but written).
* Add stop to vdpa_sim_blk.

Eugenio Pérez (4):
vdpa: Add stop operation
vhost-vdpa: introduce STOP backend feature bit
vhost-vdpa: uAPI to stop the device
vdpa_sim: Implement stop vdpa op

 drivers/vdpa/vdpa_sim/vdpa_sim.c     | 21 +++++++++++++++++
 drivers/vdpa/vdpa_sim/vdpa_sim.h     |  1 +
 drivers/vdpa/vdpa_sim/vdpa_sim_blk.c |  3 +++
 drivers/vdpa/vdpa_sim/vdpa_sim_net.c |  3 +++
 drivers/vhost/vdpa.c                 | 34 +++++++++++++++++++++++++++-
 include/linux/vdpa.h                 |  6 +++++
 include/uapi/linux/vhost.h           | 14 ++++++++++++
 include/uapi/linux/vhost_types.h     |  2 ++
 8 files changed, 83 insertions(+), 1 deletion(-)
Signature · 1 line
2.31.1
author maintainer
#1 lore raw

PATCH v4 1/4 vdpa: Add stop operation

This operation is optional: It it's not implemented, backend feature bit will not be exposed.

---
 include/linux/vdpa.h | 6 ++++++
 1 file changed, 6 insertions(+)
include/linux/vdpa.h +6 -0
index 15af802d41c4..ddfebc4e1e01 100644
@@ -215,6 +215,11 @@ struct vdpa_map_file {
  * @reset:			Reset device
  *				@vdev: vdpa device
  *				Returns integer: success (0) or error (< 0)
+ * @stop:			Stop or resume the device (optional, but it must
+ *				be implemented if require device stop)
+ *				@vdev: vdpa device
+ *				@stop: stop (true), not stop (false)
+ *				Returns integer: success (0) or error (< 0)
  * @get_config_size:		Get the size of the configuration space includes
  *				fields that are conditional on feature bits.
  *				@vdev: vdpa device
@@ -316,6 +321,7 @@ struct vdpa_config_ops {
 	u8 (*get_status)(struct vdpa_device *vdev);
 	void (*set_status)(struct vdpa_device *vdev, u8 status);
 	int (*reset)(struct vdpa_device *vdev);
+	int (*stop)(struct vdpa_device *vdev, bool stop);
 	size_t (*get_config_size)(struct vdpa_device *vdev);
 	void (*get_config)(struct vdpa_device *vdev, unsigned int offset,
 			   void *buf, unsigned int len);
Signature · 1 line
2.31.1
#2 lore raw

Re: [PATCH v4 1/4] vdpa: Add stop operation

Quoted 13 lines · Eugenio Pérez

This operation is optional: It it's not implemented, backend feature bit will not be exposed.

---
include/linux/vdpa.h | 6 ++++++

1 file changed, 6 insertions(+)
include/linux/vdpa.h +0 -0
index 15af802d41c4..ddfebc4e1e01 100644

include/linux/vdpa.h struct vdpa_map_file { · line 221

Quoted 7 lines
include/linux/vdpa.h +4 -0
@@ -215,6 +215,11 @@ struct vdpa_map_file {
 * @reset:			Reset device
 *				@vdev: vdpa device
 *				Returns integer: success (0) or error (< 0)
+ * @stop:			Stop or resume the device (optional, but it must
+ *				be implemented if require device stop)
+ *				@vdev: vdpa device
+ *				@stop: stop (true), not stop (false)

Sorry for just seeing this now, but if you have to send a v5, maybe we could use "resume" here instead of "not stop".

Thanks,
Stefano

Quoted 15 lines
include/linux/vdpa.h +2 -0
+ *				Returns integer: success (0) or error (< 0)
 * @get_config_size:		Get the size of the configuration space includes
 *				fields that are conditional on feature bits.
 *				@vdev: vdpa device
@@ -316,6 +321,7 @@ struct vdpa_config_ops {
 	u8 (*get_status)(struct vdpa_device *vdev);
 	void (*set_status)(struct vdpa_device *vdev, u8 status);
 	int (*reset)(struct vdpa_device *vdev);
+	int (*stop)(struct vdpa_device *vdev, bool stop);
 	size_t (*get_config_size)(struct vdpa_device *vdev);
 	void (*get_config)(struct vdpa_device *vdev, unsigned int offset,
 			   void *buf, unsigned int len);
Signature · 1 line
2.31.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author maintainer
#3 lore raw

Re: [PATCH v4 1/4] vdpa: Add stop operation

Quoted 10 lines · Stefano Garzarella
Quoted 8 lines · Eugenio Pérez

This operation is optional: It it's not implemented, backend feature bit will not be exposed.

---
include/linux/vdpa.h | 6 ++++++

1 file changed, 6 insertions(+)
Quoted 16 lines

include/linux/vdpa.h struct vdpa_map_file { · line 221

Quoted 12 lines
include/linux/vdpa.h +4 -0
index 15af802d41c4..ddfebc4e1e01 100644
@@ -215,6 +215,11 @@ struct vdpa_map_file {
 * @reset:                    Reset device
 *                            @vdev: vdpa device
 *                            Returns integer: success (0) or error (< 0)
+ * @stop:                     Stop or resume the device (optional, but it must
+ *                            be implemented if require device stop)
+ *                            @vdev: vdpa device
+ *                            @stop: stop (true), not stop (false)

Sorry for just seeing this now, but if you have to send a v5, maybe we could use "resume" here instead of "not stop".

I agree it fits way better, I'll queue for the next :). Thanks!

Quoted 19 lines

Thanks,
Stefano

Quoted 15 lines
include/linux/vdpa.h +2 -1
+ *                            Returns integer: success (0) or error (< 0)
 * @get_config_size:          Get the size of the configuration space includes
 *                            fields that are conditional on feature bits.
 *                            @vdev: vdpa device
@@ -316,6 +321,7 @@ struct vdpa_config_ops {
      u8 (*get_status)(struct vdpa_device *vdev);
      void (*set_status)(struct vdpa_device *vdev, u8 status);
      int (*reset)(struct vdpa_device *vdev);
+      int (*stop)(struct vdpa_device *vdev, bool stop);
      size_t (*get_config_size)(struct vdpa_device *vdev);
      void (*get_config)(struct vdpa_device *vdev, unsigned int offset,
                         void *buf, unsigned int len);
--

2.31.1

#4 lore raw

RE: [PATCH v4 1/4] vdpa: Add stop operation

In reply to Eugenio Pérez
Quoted 25 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, May 26, 2022 3:44 PM
To: Michael S. Tsirkin <[email protected]>; [email protected]; [email protected]; [email protected]; Jason Wang <[email protected]>; [email protected]
Cc: [email protected]; Stefano Garzarella <[email protected]>; [email protected]; [email protected]; [email protected]; Parav Pandit <[email protected]>; Eli Cohen <[email protected]>; Dan Carpenter <[email protected]>; Xie Yongji <[email protected]>; Christophe JAILLET <[email protected]>; Zhang Min <[email protected]>; Wu Zongyong <[email protected]>; [email protected]; Zhu Lingshan <[email protected]>; [email protected]; Si-Wei Liu <si- [email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Longpeng <[email protected]>
Subject: [PATCH v4 1/4] vdpa: Add stop operation

This operation is optional: It it's not implemented, backend feature bit will not be exposed.

---
 include/linux/vdpa.h | 6 ++++++
 1 file changed, 6 insertions(+)
include/linux/vdpa.h +0 -0
index 15af802d41c4..ddfebc4e1e01 100644

include/linux/vdpa.h struct vdpa_map_file { · line 222

Quoted 8 lines
include/linux/vdpa.h +5 -0
@@ -215,6 +215,11 @@ struct vdpa_map_file {
  * @reset:			Reset device
  *				@vdev: vdpa device
  *				Returns integer: success (0) or error (< 0)
+ * @stop:			Stop or resume the device (optional, but it must
+ *				be implemented if require device stop)
+ *				@vdev: vdpa device
+ *				@stop: stop (true), not stop (false)
+ *				Returns integer: success (0) or error (< 0)

I assume after successful "stop" the device is guaranteed to stop processing descriptors and after resume it may process descriptors? If that is so, I think it should be clear in the change log.

Quoted 13 lines
include/linux/vdpa.h +1 -1
  * @get_config_size:		Get the size of the configuration space includes
  *				fields that are conditional on feature bits.
  *				@vdev: vdpa device
@@ -316,6 +321,7 @@ struct vdpa_config_ops {
 	u8 (*get_status)(struct vdpa_device *vdev);
 	void (*set_status)(struct vdpa_device *vdev, u8 status);
 	int (*reset)(struct vdpa_device *vdev);
+	int (*stop)(struct vdpa_device *vdev, bool stop);
 	size_t (*get_config_size)(struct vdpa_device *vdev);
 	void (*get_config)(struct vdpa_device *vdev, unsigned int offset,
 			   void *buf, unsigned int len);
--

2.31.1

author maintainer
#5 lore raw

Re: [PATCH v4 1/4] vdpa: Add stop operation

Quoted 21 lines · Eli Cohen
Quoted 21 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, May 26, 2022 3:44 PM
To: Michael S. Tsirkin <[email protected]>; [email protected]; [email protected]; [email protected]; Jason Wang <[email protected]>; [email protected]
Cc: [email protected]; Stefano Garzarella <[email protected]>; [email protected]; [email protected]; [email protected]; Parav Pandit <[email protected]>; Eli Cohen <[email protected]>; Dan Carpenter <[email protected]>; Xie Yongji <[email protected]>; Christophe JAILLET <[email protected]>; Zhang Min <[email protected]>; Wu Zongyong <[email protected]>; [email protected]; Zhu Lingshan <[email protected]>; [email protected]; Si-Wei Liu <si- [email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Longpeng <[email protected]>
Subject: [PATCH v4 1/4] vdpa: Add stop operation

This operation is optional: It it's not implemented, backend feature bit will not be exposed.

---
 include/linux/vdpa.h | 6 ++++++
 1 file changed, 6 insertions(+)
Quoted 17 lines

include/linux/vdpa.h struct vdpa_map_file { · line 222

Quoted 13 lines
include/linux/vdpa.h +5 -0
index 15af802d41c4..ddfebc4e1e01 100644
@@ -215,6 +215,11 @@ struct vdpa_map_file {
  * @reset:                   Reset device
  *                           @vdev: vdpa device
  *                           Returns integer: success (0) or error (< 0)
+ * @stop:                    Stop or resume the device (optional, but it must
+ *                           be implemented if require device stop)
+ *                           @vdev: vdpa device
+ *                           @stop: stop (true), not stop (false)
+ *                           Returns integer: success (0) or error (< 0)

I assume after successful "stop" the device is guaranteed to stop processing descriptors and after resume it may process descriptors? If that is so, I think it should be clear in the change log.

Yes.

It's better described in the changelog of vdpa sim change, maybe it's better to move here.

Thanks!

Quoted 14 lines
Quoted 13 lines
include/linux/vdpa.h +1 -1
  * @get_config_size:         Get the size of the configuration space includes
  *                           fields that are conditional on feature bits.
  *                           @vdev: vdpa device
@@ -316,6 +321,7 @@ struct vdpa_config_ops {
      u8 (*get_status)(struct vdpa_device *vdev);
      void (*set_status)(struct vdpa_device *vdev, u8 status);
      int (*reset)(struct vdpa_device *vdev);
+     int (*stop)(struct vdpa_device *vdev, bool stop);
      size_t (*get_config_size)(struct vdpa_device *vdev);
      void (*get_config)(struct vdpa_device *vdev, unsigned int offset,
                         void *buf, unsigned int len);
--

2.31.1

author maintainer
#6 lore raw

PATCH v4 2/4 vhost-vdpa: introduce STOP backend feature bit

In reply to Eugenio Pérez

Userland knows if it can stop the device or not by checking this feature bit.

It's only offered if the vdpa driver backend implements the stop() operation callback, and try to set it if the backend does not offer that callback is an error.

---
 drivers/vhost/vdpa.c             | 16 +++++++++++++++-
 include/uapi/linux/vhost_types.h |  2 ++
 2 files changed, 17 insertions(+), 1 deletion(-)
drivers/vhost/vdpa.c +15 -1
index 1f1d1c425573..32713db5831d 100644
@@ -347,6 +347,14 @@ static long vhost_vdpa_set_config(struct vhost_vdpa *v,
 	return 0;
 }
 
+static bool vhost_vdpa_can_stop(const struct vhost_vdpa *v)
+{
+	struct vdpa_device *vdpa = v->vdpa;
+	const struct vdpa_config_ops *ops = vdpa->config;
+
+	return ops->stop;
+}
+
 static long vhost_vdpa_get_features(struct vhost_vdpa *v, u64 __user *featurep)
 {
 	struct vdpa_device *vdpa = v->vdpa;
@@ -575,7 +583,11 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
 	if (cmd == VHOST_SET_BACKEND_FEATURES) {
 		if (copy_from_user(&features, featurep, sizeof(features)))
 			return -EFAULT;
-		if (features & ~VHOST_VDPA_BACKEND_FEATURES)
+		if (features & ~(VHOST_VDPA_BACKEND_FEATURES |
+				 BIT_ULL(VHOST_BACKEND_F_STOP)))
+			return -EOPNOTSUPP;
+		if ((features & BIT_ULL(VHOST_BACKEND_F_STOP)) &&
+		     !vhost_vdpa_can_stop(v))
 			return -EOPNOTSUPP;
 		vhost_set_backend_features(&v->vdev, features);
 		return 0;
@@ -624,6 +636,8 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
 		break;
 	case VHOST_GET_BACKEND_FEATURES:
 		features = VHOST_VDPA_BACKEND_FEATURES;
+		if (vhost_vdpa_can_stop(v))
+			features |= BIT_ULL(VHOST_BACKEND_F_STOP);
 		if (copy_to_user(featurep, &features, sizeof(features)))
 			r = -EFAULT;
 		break;
include/uapi/linux/vhost_types.h +2 -0
index 634cee485abb..2758e665791b 100644
@@ -161,5 +161,7 @@ struct vhost_vdpa_iova_range {
  * message
  */
 #define VHOST_BACKEND_F_IOTLB_ASID  0x3
+/* Stop device from processing virtqueue buffers */
+#define VHOST_BACKEND_F_STOP  0x4
 
 #endif
Signature · 1 line
2.31.1
author maintainer
#7 lore raw

PATCH v4 3/4 vhost-vdpa: uAPI to stop the device

In reply to Eugenio Pérez

The ioctl adds support for stop the device from userspace.

---
 drivers/vhost/vdpa.c       | 18 ++++++++++++++++++
 include/uapi/linux/vhost.h | 14 ++++++++++++++
 2 files changed, 32 insertions(+)
drivers/vhost/vdpa.c +18 -0
index 32713db5831d..d1d19555c4b7 100644
@@ -478,6 +478,21 @@ static long vhost_vdpa_get_vqs_count(struct vhost_vdpa *v, u32 __user *argp)
 	return 0;
 }
 
+static long vhost_vdpa_stop(struct vhost_vdpa *v, u32 __user *argp)
+{
+	struct vdpa_device *vdpa = v->vdpa;
+	const struct vdpa_config_ops *ops = vdpa->config;
+	int stop;
+
+	if (!ops->stop)
+		return -EOPNOTSUPP;
+
+	if (copy_from_user(&stop, argp, sizeof(stop)))
+		return -EFAULT;
+
+	return ops->stop(vdpa, stop);
+}
+
 static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
 				   void __user *argp)
 {
@@ -650,6 +665,9 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
 	case VHOST_VDPA_GET_VQS_COUNT:
 		r = vhost_vdpa_get_vqs_count(v, argp);
 		break;
+	case VHOST_VDPA_STOP:
+		r = vhost_vdpa_stop(v, argp);
+		break;
 	default:
 		r = vhost_dev_ioctl(&v->vdev, cmd, argp);
 		if (r == -ENOIOCTLCMD)
include/uapi/linux/vhost.h +14 -0
index cab645d4a645..c7e47b29bf61 100644
@@ -171,4 +171,18 @@
 #define VHOST_VDPA_SET_GROUP_ASID	_IOW(VHOST_VIRTIO, 0x7C, \
 					     struct vhost_vring_state)
 
+/* Stop or resume a device so it does not process virtqueue requests anymore
+ *
+ * After the return of ioctl with stop != 0, the device must finish any
+ * pending operations like in flight requests. It must also preserve all
+ * the necessary state (the virtqueue vring base plus the possible device
+ * specific states) that is required for restoring in the future. The
+ * device must not change its configuration after that point.
+ *
+ * After the return of ioctl with stop == 0, the device can continue
+ * processing buffers as long as typical conditions are met (vq is enabled,
+ * DRIVER_OK status bit is enabled, etc).
+ */
+#define VHOST_VDPA_STOP			_IOW(VHOST_VIRTIO, 0x7D, int)
+
 #endif
Signature · 1 line
2.31.1
#8 lore raw

Re: [PATCH v4 3/4] vhost-vdpa: uAPI to stop the device

Quoted 57 lines · Eugenio Pérez

The ioctl adds support for stop the device from userspace.

---
 drivers/vhost/vdpa.c       | 18 ++++++++++++++++++
 include/uapi/linux/vhost.h | 14 ++++++++++++++
 2 files changed, 32 insertions(+)
drivers/vhost/vdpa.c +18 -0
index 32713db5831d..d1d19555c4b7 100644
@@ -478,6 +478,21 @@ static long vhost_vdpa_get_vqs_count(struct vhost_vdpa *v, u32 __user *argp)
 	return 0;
 }
 
+static long vhost_vdpa_stop(struct vhost_vdpa *v, u32 __user *argp)
+{
+	struct vdpa_device *vdpa = v->vdpa;
+	const struct vdpa_config_ops *ops = vdpa->config;
+	int stop;
+
+	if (!ops->stop)
+		return -EOPNOTSUPP;
+
+	if (copy_from_user(&stop, argp, sizeof(stop)))
+		return -EFAULT;
+
+	return ops->stop(vdpa, stop);
+}
+
 static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
 				   void __user *argp)
 {
@@ -650,6 +665,9 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
 	case VHOST_VDPA_GET_VQS_COUNT:
 		r = vhost_vdpa_get_vqs_count(v, argp);
 		break;
+	case VHOST_VDPA_STOP:
+		r = vhost_vdpa_stop(v, argp);
+		break;
 	default:
 		r = vhost_dev_ioctl(&v->vdev, cmd, argp);
 		if (r == -ENOIOCTLCMD)
include/uapi/linux/vhost.h +5 -0
index cab645d4a645..c7e47b29bf61 100644
@@ -171,4 +171,18 @@
 #define VHOST_VDPA_SET_GROUP_ASID	_IOW(VHOST_VIRTIO, 0x7C, \
 					     struct vhost_vring_state)
 
+/* Stop or resume a device so it does not process virtqueue requests anymore
+ *
+ * After the return of ioctl with stop != 0, the device must finish any
+ * pending operations like in flight requests. It must also preserve all
+ * the necessary state (the virtqueue vring base plus the possible device

include/uapi/linux/vhost.h line 188

Quoted 10 lines
include/uapi/linux/vhost.h +9 -0
+ * specific states) that is required for restoring in the future. The
+ * device must not change its configuration after that point.
+ *
+ * After the return of ioctl with stop == 0, the device can continue
+ * processing buffers as long as typical conditions are met (vq is enabled,
+ * DRIVER_OK status bit is enabled, etc).
+ */
+#define VHOST_VDPA_STOP			_IOW(VHOST_VIRTIO, 0x7D, int)
+
 #endif

I wonder how does this interact with the admin vq idea. I.e. if we stop all VQs then apparently admin vq can't work either ...
Thoughts?

Quoted 2 lines
Signature · 1 line
2.31.1
author maintainer
#9 lore raw

Re: [PATCH v4 3/4] vhost-vdpa: uAPI to stop the device

Quoted 66 lines · Michael S. Tsirkin
Quoted 64 lines · Eugenio Pérez

The ioctl adds support for stop the device from userspace.

---
 drivers/vhost/vdpa.c       | 18 ++++++++++++++++++
 include/uapi/linux/vhost.h | 14 ++++++++++++++
 2 files changed, 32 insertions(+)
drivers/vhost/vdpa.c +18 -0
index 32713db5831d..d1d19555c4b7 100644
@@ -478,6 +478,21 @@ static long vhost_vdpa_get_vqs_count(struct vhost_vdpa *v, u32 __user *argp)
      return 0;
 }
 
+static long vhost_vdpa_stop(struct vhost_vdpa *v, u32 __user *argp)
+{
+     struct vdpa_device *vdpa = v->vdpa;
+     const struct vdpa_config_ops *ops = vdpa->config;
+     int stop;
+
+     if (!ops->stop)
+             return -EOPNOTSUPP;
+
+     if (copy_from_user(&stop, argp, sizeof(stop)))
+             return -EFAULT;
+
+     return ops->stop(vdpa, stop);
+}
+
 static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
                                 void __user *argp)
 {
@@ -650,6 +665,9 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
      case VHOST_VDPA_GET_VQS_COUNT:
              r = vhost_vdpa_get_vqs_count(v, argp);
              break;
+     case VHOST_VDPA_STOP:
+             r = vhost_vdpa_stop(v, argp);
+             break;
      default:
              r = vhost_dev_ioctl(&v->vdev, cmd, argp);
              if (r == -ENOIOCTLCMD)
include/uapi/linux/vhost.h +12 -0
index cab645d4a645..c7e47b29bf61 100644
@@ -171,4 +171,18 @@
 #define VHOST_VDPA_SET_GROUP_ASID    _IOW(VHOST_VIRTIO, 0x7C, \
                                           struct vhost_vring_state)
 
+/* Stop or resume a device so it does not process virtqueue requests anymore
+ *
+ * After the return of ioctl with stop != 0, the device must finish any
+ * pending operations like in flight requests. It must also preserve all
+ * the necessary state (the virtqueue vring base plus the possible device
+ * specific states) that is required for restoring in the future. The
+ * device must not change its configuration after that point.
+ *
+ * After the return of ioctl with stop == 0, the device can continue
+ * processing buffers as long as typical conditions are met (vq is enabled,
+ * DRIVER_OK status bit is enabled, etc).
+ */
Quoted 9 lines

include/uapi/linux/vhost.h line 188

Quoted 3 lines
include/uapi/linux/vhost.h +2 -0
+#define VHOST_VDPA_STOP                      _IOW(VHOST_VIRTIO, 0x7D, int)
+
 #endif

I wonder how does this interact with the admin vq idea. I.e. if we stop all VQs then apparently admin vq can't work either ...
Thoughts?

Copying here the answer to Parav, feel free to answer to any thread or highlight if I missed something :). Using the admin vq proposal terminology of "device group".

--
This would stop a device of a device
group, but not the whole virtqueue group. If the admin VQ is offered by the PF (since it's not exposed to the guest), it will continue accepting requests as normal. If it's exposed in the VF, I think the best bet is to shadow it, since guest and host requests could conflict.

Since this is offered through vdpa, the device backend driver can route it to whatever method works better for the hardware. For example, to send an admin vq command to the PF. That's why it's important to keep the feature as self-contained and orthogonal to others as possible.
--

Quoted 3 lines
Quoted 2 lines

--
2.31.1

#10 lore raw

RE: [PATCH v4 3/4] vhost-vdpa: uAPI to stop the device

Quoted 75 lines

From: Eugenio Perez Martin <[email protected]>
Sent: Wednesday, June 1, 2022 7:15 AM

Quoted 10 lines · Michael S. Tsirkin
Quoted 8 lines · Eugenio Pérez

The ioctl adds support for stop the device from userspace.

---
 drivers/vhost/vdpa.c       | 18 ++++++++++++++++++
 include/uapi/linux/vhost.h | 14 ++++++++++++++
 2 files changed, 32 insertions(+)
Quoted 5 lines

drivers/vhost/vdpa.c static long vhost_vdpa_get_vqs_count(struct

Quoted 5 lines
drivers/vhost/vdpa.c index renamed from drivers/vhost/vdpa.c +0 -0

32713db5831d..d1d19555c4b7 100644

drivers/vhost/vdpa.c +0 -0
@@ -478,6 +478,21 @@ static long vhost_vdpa_get_vqs_count(struct

vhost_vdpa *v, u32 __user *argp)

Quoted 19 lines
Quoted 9 lines
drivers/vhost/vdpa.c +6 -0
      return 0;
 }
 
+static long vhost_vdpa_stop(struct vhost_vdpa *v, u32 __user *argp)
+{
+     struct vdpa_device *vdpa = v->vdpa;
+     const struct vdpa_config_ops *ops = vdpa->config;
+     int stop;
+

drivers/vhost/vdpa.c static long vhost_vdpa_get_vqs_count(struct vhost_vdpa *v, u32 __user *argp)

Quoted 10 lines
drivers/vhost/vdpa.c +9 -0
+     if (!ops->stop)
+             return -EOPNOTSUPP;
+
+     if (copy_from_user(&stop, argp, sizeof(stop)))
+             return -EFAULT;
+
+     return ops->stop(vdpa, stop);
+}
+
 static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int

cmd,

Quoted 9 lines
Quoted 9 lines
drivers/vhost/vdpa.c +4 -0
                                 void __user *argp)  { @@ -650,6
+665,9 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
      case VHOST_VDPA_GET_VQS_COUNT:
              r = vhost_vdpa_get_vqs_count(v, argp);
              break;
+     case VHOST_VDPA_STOP:
+             r = vhost_vdpa_stop(v, argp);
+             break;
      default:
Quoted 8 lines

include/uapi/linux/vhost.h line 171

Quoted 8 lines
              r = vhost_dev_ioctl(&v->vdev, cmd, argp);
              if (r == -ENOIOCTLCMD) diff --git

a/include/uapi/linux/vhost.h b/include/uapi/linux/vhost.h index cab645d4a645..c7e47b29bf61 100644

include/uapi/linux/vhost.h +0 -0
@@ -171,4 +171,18 @@
 #define VHOST_VDPA_SET_GROUP_ASID    _IOW(VHOST_VIRTIO, 0x7C,

\

Quoted 15 lines
Quoted 15 lines
include/uapi/linux/vhost.h +13 -0
                                           struct vhost_vring_state)
 
+/* Stop or resume a device so it does not process virtqueue
+requests anymore
+ *
+ * After the return of ioctl with stop != 0, the device must finish
+any
+ * pending operations like in flight requests. It must also
+preserve all
+ * the necessary state (the virtqueue vring base plus the possible
+device
+ * specific states) that is required for restoring in the future.
+The
+ * device must not change its configuration after that point.
+ *
Quoted 8 lines
Quoted 8 lines

include/uapi/linux/vhost.h line 187

Quoted 8 lines
include/uapi/linux/vhost.h +8 -0
+ * After the return of ioctl with stop == 0, the device can
+continue
+ * processing buffers as long as typical conditions are met (vq is
+enabled,
+ * DRIVER_OK status bit is enabled, etc).
+ */
+#define VHOST_VDPA_STOP                      _IOW(VHOST_VIRTIO, 0x7D, int)
+

A better name is VHOST_VDPA_SET_STATE
State = stop/suspend
State = start/resume

Suspend/resume seems more logical, as opposed start/stop, because it more clearly indicates that the resume (start) is from some programmed beginning (and not first boot).

Quoted 11 lines
Quoted 6 lines
Quoted 1 line

#endif

I wonder how does this interact with the admin vq idea. I.e. if we stop all VQs then apparently admin vq can't work either ... Thoughts?

Copying here the answer to Parav, feel free to answer to any thread or highlight if I missed something :). Using the admin vq proposal terminology of "device group".

Quoted 7 lines

--
This would stop a device of a device
group, but not the whole virtqueue group. If the admin VQ is offered by the PF (since it's not exposed to the guest), it will continue accepting requests as normal. If it's exposed in the VF, I think the best bet is to shadow it, since guest and host requests could conflict.

vhost-vdpa device is exposed for a VF through vp-vdpa driver to user land.
Now vp-vdpa driver will have to choose between using config register vs using AQ to suspend/resume the device.

Why not always begin with more superior interface of AQ that address multiple of these needs for LM case?

For LM case, more you explore, we realize that either VF relying on PF's AQ for query/config/setup/restore makes more sense or have its own dedicated AQ.

VM's suspend/resume operation can be handled through the shadow Q.

Quoted 9 lines

Since this is offered through vdpa, the device backend driver can route it to whatever method works better for the hardware. For example, to send an admin vq command to the PF. That's why it's important to keep the feature as self-contained and orthogonal to others as possible.
--

Quoted 3 lines
Quoted 2 lines

--
2.31.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author maintainer
#11 lore raw

Re: [PATCH v4 3/4] vhost-vdpa: uAPI to stop the device

Quoted 88 lines · Parav Pandit
Quoted 76 lines

From: Eugenio Perez Martin <[email protected]>
Sent: Wednesday, June 1, 2022 7:15 AM

Quoted 10 lines · Michael S. Tsirkin
Quoted 8 lines · Eugenio Pérez

The ioctl adds support for stop the device from userspace.

---
 drivers/vhost/vdpa.c       | 18 ++++++++++++++++++
 include/uapi/linux/vhost.h | 14 ++++++++++++++
 2 files changed, 32 insertions(+)
Quoted 5 lines

drivers/vhost/vdpa.c static long vhost_vdpa_get_vqs_count(struct

Quoted 5 lines
drivers/vhost/vdpa.c index renamed from drivers/vhost/vdpa.c +0 -0

32713db5831d..d1d19555c4b7 100644

drivers/vhost/vdpa.c +0 -0
@@ -478,6 +478,21 @@ static long vhost_vdpa_get_vqs_count(struct

vhost_vdpa *v, u32 __user *argp)

Quoted 19 lines
Quoted 9 lines
drivers/vhost/vdpa.c +6 -0
      return 0;
 }
 
+static long vhost_vdpa_stop(struct vhost_vdpa *v, u32 __user *argp)
+{
+     struct vdpa_device *vdpa = v->vdpa;
+     const struct vdpa_config_ops *ops = vdpa->config;
+     int stop;
+

drivers/vhost/vdpa.c static long vhost_vdpa_get_vqs_count(struct vhost_vdpa *v, u32 __user *argp)

Quoted 10 lines
drivers/vhost/vdpa.c +9 -0
+     if (!ops->stop)
+             return -EOPNOTSUPP;
+
+     if (copy_from_user(&stop, argp, sizeof(stop)))
+             return -EFAULT;
+
+     return ops->stop(vdpa, stop);
+}
+
 static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int

cmd,

Quoted 9 lines
Quoted 9 lines
drivers/vhost/vdpa.c +4 -0
                                 void __user *argp)  { @@ -650,6
+665,9 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
      case VHOST_VDPA_GET_VQS_COUNT:
              r = vhost_vdpa_get_vqs_count(v, argp);
              break;
+     case VHOST_VDPA_STOP:
+             r = vhost_vdpa_stop(v, argp);
+             break;
      default:
Quoted 8 lines

include/uapi/linux/vhost.h line 171

Quoted 8 lines
              r = vhost_dev_ioctl(&v->vdev, cmd, argp);
              if (r == -ENOIOCTLCMD) diff --git

a/include/uapi/linux/vhost.h b/include/uapi/linux/vhost.h index cab645d4a645..c7e47b29bf61 100644

include/uapi/linux/vhost.h +0 -0
@@ -171,4 +171,18 @@
 #define VHOST_VDPA_SET_GROUP_ASID    _IOW(VHOST_VIRTIO, 0x7C,

\

Quoted 15 lines
Quoted 15 lines
include/uapi/linux/vhost.h +13 -0
                                           struct vhost_vring_state)
 
+/* Stop or resume a device so it does not process virtqueue
+requests anymore
+ *
+ * After the return of ioctl with stop != 0, the device must finish
+any
+ * pending operations like in flight requests. It must also
+preserve all
+ * the necessary state (the virtqueue vring base plus the possible
+device
+ * specific states) that is required for restoring in the future.
+The
+ * device must not change its configuration after that point.
+ *
Quoted 8 lines
Quoted 8 lines

include/uapi/linux/vhost.h line 187

Quoted 8 lines
include/uapi/linux/vhost.h +8 -0
+ * After the return of ioctl with stop == 0, the device can
+continue
+ * processing buffers as long as typical conditions are met (vq is
+enabled,
+ * DRIVER_OK status bit is enabled, etc).
+ */
+#define VHOST_VDPA_STOP                      _IOW(VHOST_VIRTIO, 0x7D, int)
+

A better name is VHOST_VDPA_SET_STATE
State = stop/suspend
State = start/resume

Quoted 2 lines

Suspend/resume seems more logical, as opposed start/stop, because it more clearly indicates that the resume (start) is from some programmed beginning (and not first boot).

It's fine to move to that nomenclature in my opinion.

Quoted 19 lines
Quoted 11 lines
Quoted 6 lines
Quoted 1 line

#endif

I wonder how does this interact with the admin vq idea. I.e. if we stop all VQs then apparently admin vq can't work either ... Thoughts?

Copying here the answer to Parav, feel free to answer to any thread or highlight if I missed something :). Using the admin vq proposal terminology of "device group".

Quoted 7 lines

--
This would stop a device of a device
group, but not the whole virtqueue group. If the admin VQ is offered by the PF (since it's not exposed to the guest), it will continue accepting requests as normal. If it's exposed in the VF, I think the best bet is to shadow it, since guest and host requests could conflict.

Quoted 3 lines

vhost-vdpa device is exposed for a VF through vp-vdpa driver to user land.
Now vp-vdpa driver will have to choose between using config register vs using AQ to suspend/resume the device.

vp_vdpa cannot choose if the virtio device has an admin vq or any other feature, it just wraps the virtio device. If that virtio device does not expose AQ, vp_vdpa cannot expose it.

Quoted 2 lines

Why not always begin with more superior interface of AQ that address multiple of these needs for LM case?

Because it doesn't address valid use cases like vp_vdpa with no AQ, devices that are not VF, or nested virtualization.

VHOST_VDPA_STOP / VHOST_VDPA_SET_STATE does not replace AQ commands:
It's just the way vhost-vdpa exposes that capability to qemu. vdpa backend is free to choose whatever methods it finds better to implement it.

Quoted 2 lines

For LM case, more you explore, we realize that either VF relying on PF's AQ for query/config/setup/restore makes more sense or have its own dedicated AQ.

This ioctl does not mandate that the device cannot implement it through AQ, or that the device has to be a VF.

Thanks!

Quoted 12 lines

VM's suspend/resume operation can be handled through the shadow Q.

Quoted 9 lines

Since this is offered through vdpa, the device backend driver can route it to whatever method works better for the hardware. For example, to send an admin vq command to the PF. That's why it's important to keep the feature as self-contained and orthogonal to others as possible.
--

Quoted 3 lines
Quoted 2 lines

--
2.31.1

author maintainer
#12 lore raw

PATCH v4 4/4 vdpa_sim: Implement stop vdpa op

In reply to Eugenio Pérez

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration, since the device could modify them after userland gets them. There are individual ways to perform that action for some devices (VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no way to perform it for any vhost device (and, in particular, vhost-vdpa).

---
 drivers/vdpa/vdpa_sim/vdpa_sim.c     | 21 +++++++++++++++++++++
 drivers/vdpa/vdpa_sim/vdpa_sim.h     |  1 +
 drivers/vdpa/vdpa_sim/vdpa_sim_blk.c |  3 +++
 drivers/vdpa/vdpa_sim/vdpa_sim_net.c |  3 +++
 4 files changed, 28 insertions(+)
drivers/vdpa/vdpa_sim/vdpa_sim.c +21 -0
index 50d721072beb..0515cf314bed 100644
@@ -107,6 +107,7 @@ static void vdpasim_do_reset(struct vdpasim *vdpasim)
 	for (i = 0; i < vdpasim->dev_attr.nas; i++)
 		vhost_iotlb_reset(&vdpasim->iommu[i]);
 
+	vdpasim->running = true;
 	spin_unlock(&vdpasim->iommu_lock);
 
 	vdpasim->features = 0;
@@ -505,6 +506,24 @@ static int vdpasim_reset(struct vdpa_device *vdpa)
 	return 0;
 }
 
+static int vdpasim_stop(struct vdpa_device *vdpa, bool stop)
+{
+	struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
+	int i;
+
+	spin_lock(&vdpasim->lock);
+	vdpasim->running = !stop;
+	if (vdpasim->running) {
+		/* Check for missed buffers */
+		for (i = 0; i < vdpasim->dev_attr.nvqs; ++i)
+			vdpasim_kick_vq(vdpa, i);
+
+	}
+	spin_unlock(&vdpasim->lock);
+
+	return 0;
+}
+
 static size_t vdpasim_get_config_size(struct vdpa_device *vdpa)
 {
 	struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
@@ -694,6 +713,7 @@ static const struct vdpa_config_ops vdpasim_config_ops = {
 	.get_status             = vdpasim_get_status,
 	.set_status             = vdpasim_set_status,
 	.reset			= vdpasim_reset,
+	.stop			= vdpasim_stop,
 	.get_config_size        = vdpasim_get_config_size,
 	.get_config             = vdpasim_get_config,
 	.set_config             = vdpasim_set_config,
@@ -726,6 +746,7 @@ static const struct vdpa_config_ops vdpasim_batch_config_ops = {
 	.get_status             = vdpasim_get_status,
 	.set_status             = vdpasim_set_status,
 	.reset			= vdpasim_reset,
+	.stop			= vdpasim_stop,
 	.get_config_size        = vdpasim_get_config_size,
 	.get_config             = vdpasim_get_config,
 	.set_config             = vdpasim_set_config,
drivers/vdpa/vdpa_sim/vdpa_sim.h +1 -0
index 622782e92239..061986f30911 100644
@@ -66,6 +66,7 @@ struct vdpasim {
 	u32 generation;
 	u64 features;
 	u32 groups;
+	bool running;
 	/* spinlock to synchronize iommu table */
 	spinlock_t iommu_lock;
 };
drivers/vdpa/vdpa_sim/vdpa_sim_blk.c +3 -0
index 42d401d43911..bcdb1982c378 100644
@@ -204,6 +204,9 @@ static void vdpasim_blk_work(struct work_struct *work)
 	if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK))
 		goto out;
 
+	if (!vdpasim->running)
+		goto out;
+
 	for (i = 0; i < VDPASIM_BLK_VQ_NUM; i++) {
 		struct vdpasim_virtqueue *vq = &vdpasim->vqs[i];
 
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +3 -0
index 5125976a4df8..886449e88502 100644
@@ -154,6 +154,9 @@ static void vdpasim_net_work(struct work_struct *work)
 
 	spin_lock(&vdpasim->lock);
 
+	if (!vdpasim->running)
+		goto out;
+
 	if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK))
 		goto out;
 
Signature · 1 line
2.31.1
Reviewed-by on v4 4/4
#13 lore raw

Re: [PATCH v4 4/4] vdpa_sim: Implement stop vdpa op

Quoted 9 lines · Eugenio Pérez

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration, since the device could modify them after userland gets them. There are individual ways to perform that action for some devices (VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no way to perform it for any vhost device (and, in particular, vhost-vdpa).

Quoted 7 lines

---

drivers/vdpa/vdpa_sim/vdpa_sim.c     | 21 +++++++++++++++++++++
drivers/vdpa/vdpa_sim/vdpa_sim.h     |  1 +
drivers/vdpa/vdpa_sim/vdpa_sim_blk.c |  3 +++
drivers/vdpa/vdpa_sim/vdpa_sim_net.c |  3 +++
4 files changed, 28 insertions(+)
Quoted 97 lines
drivers/vdpa/vdpa_sim/vdpa_sim.c +21 -0
index 50d721072beb..0515cf314bed 100644
@@ -107,6 +107,7 @@ static void vdpasim_do_reset(struct vdpasim *vdpasim)
 	for (i = 0; i < vdpasim->dev_attr.nas; i++)
 		vhost_iotlb_reset(&vdpasim->iommu[i]);
 
+	vdpasim->running = true;
 	spin_unlock(&vdpasim->iommu_lock);
 
 	vdpasim->features = 0;
@@ -505,6 +506,24 @@ static int vdpasim_reset(struct vdpa_device *vdpa)
 	return 0;
 }
 
+static int vdpasim_stop(struct vdpa_device *vdpa, bool stop)
+{
+	struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
+	int i;
+
+	spin_lock(&vdpasim->lock);
+	vdpasim->running = !stop;
+	if (vdpasim->running) {
+		/* Check for missed buffers */
+		for (i = 0; i < vdpasim->dev_attr.nvqs; ++i)
+			vdpasim_kick_vq(vdpa, i);
+
+	}
+	spin_unlock(&vdpasim->lock);
+
+	return 0;
+}
+
 static size_t vdpasim_get_config_size(struct vdpa_device *vdpa)
 {
 	struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
@@ -694,6 +713,7 @@ static const struct vdpa_config_ops vdpasim_config_ops = {
 	.get_status             = vdpasim_get_status,
 	.set_status             = vdpasim_set_status,
 	.reset			= vdpasim_reset,
+	.stop			= vdpasim_stop,
 	.get_config_size        = vdpasim_get_config_size,
 	.get_config             = vdpasim_get_config,
 	.set_config             = vdpasim_set_config,
@@ -726,6 +746,7 @@ static const struct vdpa_config_ops vdpasim_batch_config_ops = {
 	.get_status             = vdpasim_get_status,
 	.set_status             = vdpasim_set_status,
 	.reset			= vdpasim_reset,
+	.stop			= vdpasim_stop,
 	.get_config_size        = vdpasim_get_config_size,
 	.get_config             = vdpasim_get_config,
 	.set_config             = vdpasim_set_config,
drivers/vdpa/vdpa_sim/vdpa_sim.h +1 -0
index 622782e92239..061986f30911 100644
@@ -66,6 +66,7 @@ struct vdpasim {
 	u32 generation;
 	u64 features;
 	u32 groups;
+	bool running;
 	/* spinlock to synchronize iommu table */
 	spinlock_t iommu_lock;
 };
drivers/vdpa/vdpa_sim/vdpa_sim_blk.c +3 -0
index 42d401d43911..bcdb1982c378 100644
@@ -204,6 +204,9 @@ static void vdpasim_blk_work(struct work_struct *work)
 	if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK))
 		goto out;
 
+	if (!vdpasim->running)
+		goto out;
+
 	for (i = 0; i < VDPASIM_BLK_VQ_NUM; i++) {
 		struct vdpasim_virtqueue *vq = &vdpasim->vqs[i];
 
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +3 -0
index 5125976a4df8..886449e88502 100644
@@ -154,6 +154,9 @@ static void vdpasim_net_work(struct work_struct *work)
 
 	spin_lock(&vdpasim->lock);
 
+	if (!vdpasim->running)
+		goto out;
+
 	if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK))
 		goto out;
 
Signature · 1 line
2.31.1
#14 lore raw
In reply to Eugenio Pérez
Quoted 36 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, May 26, 2022 8:44 AM

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer

that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration,

since the device could modify them after userland gets them. There are

individual ways to perform that action for some devices

(VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no

way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any

pending operations like in flight requests. It must also preserve all

the necessary state (the virtqueue vring base plus the possible device

specific states) that is required for restoring in the future. The

device must not change its configuration after that point.

Quoted 5 lines

After the return of ioctl with stop == 0, the device can continue

processing buffers as long as typical conditions are met (vq is enabled,

DRIVER_OK status bit is enabled, etc).

Just to be clear, we are adding vdpa level new ioctl() that doesn’t map to any mechanism in the virtio spec.

Why can't we use this ioctl() to indicate driver to start/stop the device instead of driving it through the driver_ok? This is in the context of other discussion we had in the LM series.

#15 lore raw
Quoted 37 lines · Parav Pandit
Quoted 37 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, May 26, 2022 8:44 AM

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer

that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration,

since the device could modify them after userland gets them. There are

individual ways to perform that action for some devices

(VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no

way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any

pending operations like in flight requests. It must also preserve all

the necessary state (the virtqueue vring base plus the possible device

specific states) that is required for restoring in the future. The

device must not change its configuration after that point.

Quoted 7 lines
Quoted 5 lines

After the return of ioctl with stop == 0, the device can continue

processing buffers as long as typical conditions are met (vq is enabled,

DRIVER_OK status bit is enabled, etc).

Just to be clear, we are adding vdpa level new ioctl() that doesn’t map to any mechanism in the virtio spec.

We try to provide forward compatibility to VIRTIO_CONFIG_S_STOP. That means it is expected to implement at least a subset of VIRTIO_CONFIG_S_STOP.

Quoted 2 lines

Why can't we use this ioctl() to indicate driver to start/stop the device instead of driving it through the driver_ok?

So the idea is to add capability that does not exist in the spec. Then came the stop/resume which can't be done via DRIVER_OK. I think we should only allow the stop/resume to succeed after DRIVER_OK is set.

Quoted 1 line

This is in the context of other discussion we had in the LM series.

Do you see any issue that blocks the live migration?

Thanks

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author maintainer
#16 lore raw
Quoted 43 lines · Jason Wang
Quoted 41 lines · Parav Pandit
Quoted 41 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, May 26, 2022 8:44 AM

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer

that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration,

since the device could modify them after userland gets them. There are

individual ways to perform that action for some devices

(VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no

way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any

pending operations like in flight requests. It must also preserve all

the necessary state (the virtqueue vring base plus the possible device

specific states) that is required for restoring in the future. The

device must not change its configuration after that point.

After the return of ioctl with stop == 0, the device can continue

processing buffers as long as typical conditions are met (vq is enabled,

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

DRIVER_OK status bit is enabled, etc).

Just to be clear, we are adding vdpa level new ioctl() that doesn’t map to any mechanism in the virtio spec.

We try to provide forward compatibility to VIRTIO_CONFIG_S_STOP. That means it is expected to implement at least a subset of VIRTIO_CONFIG_S_STOP.

Appending a link to the proposal, just for reference [1].

Quoted 3 lines
Quoted 2 lines

Why can't we use this ioctl() to indicate driver to start/stop the device instead of driving it through the driver_ok?

Parav, I'm not sure I follow you here.

By the proposal, the resume of the device is (From qemu POV):
1. To configure all data vqs and cvq (addr, num, ...)
2. To enable only CVQ, not data vqs
3. To send DRIVER_OK
4. Wait for all buffers of CVQ to be used
5. To enable all others data vqs (individual ioctl at the moment)

Where can we fit the resume (as "stop(false)") here? If the device is stopped (as if we send stop(true) before DRIVER_OK), we don't read CVQ first. If we send it right after (or instead) DRIVER_OK, data buffers can reach data vqs before configuring RSS.

Quoted 10 lines

So the idea is to add capability that does not exist in the spec. Then came the stop/resume which can't be done via DRIVER_OK. I think we should only allow the stop/resume to succeed after DRIVER_OK is set.

Quoted 1 line

This is in the context of other discussion we had in the LM series.

Do you see any issue that blocks the live migration?

Thanks

#17 lore raw
Quoted 81 lines

From: Eugenio Perez Martin <[email protected]>
Sent: Friday, May 27, 2022 3:55 AM

Quoted 18 lines · Jason Wang
Quoted 16 lines · Parav Pandit
Quoted 16 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, May 26, 2022 8:44 AM

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer

that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration,

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

since the device could modify them after userland gets them. There are

individual ways to perform that action for some devices

(VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but

there

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

was no

way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any

pending operations like in flight requests. It must also preserve all

the necessary state (the virtqueue vring base plus the possible device

specific states) that is required for restoring in the future. The

device must not change its configuration after that point.

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

After the return of ioctl with stop == 0, the device can continue

processing buffers as long as typical conditions are met (vq is enabled,

DRIVER_OK status bit is enabled, etc).

Just to be clear, we are adding vdpa level new ioctl() that doesn’t map to

any mechanism in the virtio spec.

Quoted 5 lines

We try to provide forward compatibility to VIRTIO_CONFIG_S_STOP. That means it is expected to implement at least a subset of VIRTIO_CONFIG_S_STOP.

Appending a link to the proposal, just for reference [1].

Quoted 2 lines
Quoted 2 lines

Why can't we use this ioctl() to indicate driver to start/stop the device

instead of driving it through the driver_ok?

Quoted 1 line

Parav, I'm not sure I follow you here.

By the proposal, the resume of the device is (From qemu POV):
1. To configure all data vqs and cvq (addr, num, ...) 2. To enable only CVQ, not data vqs 3. To send DRIVER_OK 4. Wait for all buffers of CVQ to be used 5. To enable all others data vqs (individual ioctl at the moment)

Quoted 5 lines

Where can we fit the resume (as "stop(false)") here? If the device is stopped (as if we send stop(true) before DRIVER_OK), we don't read CVQ first. If we send it right after (or instead) DRIVER_OK, data buffers can reach data vqs before configuring RSS.

It doesn’t make sense with currently proposed way of using cvq to replay the config.
Need to continue with currently proposed temporary method that subsequently to be replaced with optimized flow as we discussed.

author maintainer
#18 lore raw
Quoted 82 lines · Parav Pandit
Quoted 82 lines

From: Eugenio Perez Martin <[email protected]>
Sent: Friday, May 27, 2022 3:55 AM

Quoted 18 lines · Jason Wang
Quoted 16 lines · Parav Pandit
Quoted 16 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, May 26, 2022 8:44 AM

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer

that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration,

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

since the device could modify them after userland gets them. There are

individual ways to perform that action for some devices

(VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but

there

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

was no

way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any

pending operations like in flight requests. It must also preserve all

the necessary state (the virtqueue vring base plus the possible device

specific states) that is required for restoring in the future. The

device must not change its configuration after that point.

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

After the return of ioctl with stop == 0, the device can continue

processing buffers as long as typical conditions are met (vq is enabled,

DRIVER_OK status bit is enabled, etc).

Just to be clear, we are adding vdpa level new ioctl() that doesn’t map to

any mechanism in the virtio spec.

Quoted 5 lines

We try to provide forward compatibility to VIRTIO_CONFIG_S_STOP. That means it is expected to implement at least a subset of VIRTIO_CONFIG_S_STOP.

Appending a link to the proposal, just for reference [1].

Quoted 2 lines
Quoted 2 lines

Why can't we use this ioctl() to indicate driver to start/stop the device

instead of driving it through the driver_ok?

Quoted 1 line

Parav, I'm not sure I follow you here.

By the proposal, the resume of the device is (From qemu POV):
1. To configure all data vqs and cvq (addr, num, ...) 2. To enable only CVQ, not data vqs 3. To send DRIVER_OK 4. Wait for all buffers of CVQ to be used 5. To enable all others data vqs (individual ioctl at the moment)

Quoted 6 lines
Quoted 5 lines

Where can we fit the resume (as "stop(false)") here? If the device is stopped (as if we send stop(true) before DRIVER_OK), we don't read CVQ first. If we send it right after (or instead) DRIVER_OK, data buffers can reach data vqs before configuring RSS.

It doesn’t make sense with currently proposed way of using cvq to replay the config.

The stop/resume part is not intended to restore the config through the CVQ. The stop call is issued to be able to retrieve the vq status (base, in vhost terminology). The symmetric operation (resume) was added on demand, it was never intended to be part neither of the config restore or the virtqueue state restore workflow.

The configuration restore workflow was modelled after the device initialization, so each part needed to add the less things the better, and only qemu needed to be changed. From the device POV, there is no need to learn new tricks for this. The support of .set_vq_ready and .get_vq_ready is already in the kernel in every vdpa backend driver.

Quoted 1 line

Need to continue with currently proposed temporary method that subsequently to be replaced with optimized flow as we discussed.

Back then, it was noted by you that enabling each data vq individually after DRIVER_OK is slow on mlx5 devices. The solution was to batch these enable calls accounting in the kernel, achieving no growth in the vdpa uAPI layer. The proposed solution did not involve the resume operation.

After that, you proposed in this thread "Why can't we use this ioctl() to indicate driver to start/stop the device instead of driving it through the driver_ok?". As I understand, that is a mistake, since it requires the device, the vdpa layer, etc... to learn new tricks. It requires qemu to duplicate the initialization layer (it's now common for start and restore config). But I might have not seen the whole picture, missing advantages of using the resume call for this workflow. Can you describe the workflow you have in mind? How does that new workflow affect this proposal?

I'm ok to change the proposal as long as we find we obtain a net gain.

Thanks!

#19 lore raw
In reply to Parav Pandit
Quoted 38 lines · Parav Pandit
Quoted 38 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, May 26, 2022 8:44 AM

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer

that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration,

since the device could modify them after userland gets them. There are

individual ways to perform that action for some devices

(VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no

way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any

pending operations like in flight requests. It must also preserve all

the necessary state (the virtqueue vring base plus the possible device

specific states) that is required for restoring in the future. The

device must not change its configuration after that point.

After the return of ioctl with stop == 0, the device can continue

Quoted 8 lines
Quoted 3 lines

processing buffers as long as typical conditions are met (vq is enabled,

DRIVER_OK status bit is enabled, etc).

Just to be clear, we are adding vdpa level new ioctl() that doesn’t map to any mechanism in the virtio spec.

Why can't we use this ioctl() to indicate driver to start/stop the device instead of driving it through the driver_ok? This is in the context of other discussion we had in the LM series.

If there's something in the spec that does this then let's use that. Unfortunately the LM series seems to be stuck on moving bits around with the admin virtqueue ...

Signature · 6 lines
MST

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
#20 lore raw
Quoted 42 lines · Michael S. Tsirkin
Quoted 40 lines · Parav Pandit
Quoted 40 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, May 26, 2022 8:44 AM

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer

that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration,

since the device could modify them after userland gets them. There are

individual ways to perform that action for some devices

(VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no

way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any

pending operations like in flight requests. It must also preserve all

the necessary state (the virtqueue vring base plus the possible device

specific states) that is required for restoring in the future. The

device must not change its configuration after that point.

After the return of ioctl with stop == 0, the device can continue

processing buffers as long as typical conditions are met (vq is enabled,

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

DRIVER_OK status bit is enabled, etc).

Just to be clear, we are adding vdpa level new ioctl() that doesn’t map to any mechanism in the virtio spec.

Why can't we use this ioctl() to indicate driver to start/stop the device instead of driving it through the driver_ok? This is in the context of other discussion we had in the LM series.

If there's something in the spec that does this then let's use that.

Actually, we try to propose a independent feature here:

https://lists.oasis-open.org/archives/virtio-dev/202111/msg00020.html

Does it make sense to you?

Thanks

Quoted 6 lines

Unfortunately the LM series seems to be stuck on moving bits around with the admin virtqueue ...

--
MST

#21 lore raw
Quoted 52 lines · Jason Wang
Quoted 42 lines · Michael S. Tsirkin
Quoted 40 lines · Parav Pandit
Quoted 40 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, May 26, 2022 8:44 AM

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer

that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration,

since the device could modify them after userland gets them. There are

individual ways to perform that action for some devices

(VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no

way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any

pending operations like in flight requests. It must also preserve all

the necessary state (the virtqueue vring base plus the possible device

specific states) that is required for restoring in the future. The

device must not change its configuration after that point.

After the return of ioctl with stop == 0, the device can continue

processing buffers as long as typical conditions are met (vq is enabled,

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

DRIVER_OK status bit is enabled, etc).

Just to be clear, we are adding vdpa level new ioctl() that doesn’t map to any mechanism in the virtio spec.

Why can't we use this ioctl() to indicate driver to start/stop the device instead of driving it through the driver_ok? This is in the context of other discussion we had in the LM series.

If there's something in the spec that does this then let's use that.

Quoted 7 lines

Actually, we try to propose a independent feature here:

https://lists.oasis-open.org/archives/virtio-dev/202111/msg00020.html

Does it make sense to you?

Thanks

But I thought the LM patches are trying to replace all that?

Quoted 6 lines
Quoted 6 lines

Unfortunately the LM series seems to be stuck on moving bits around with the admin virtqueue ...

--
MST

#22 lore raw
Quoted 56 lines · Michael S. Tsirkin
Quoted 54 lines · Jason Wang
Quoted 42 lines · Michael S. Tsirkin
Quoted 40 lines · Parav Pandit
Quoted 40 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, May 26, 2022 8:44 AM

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer

that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration,

since the device could modify them after userland gets them. There are

individual ways to perform that action for some devices

(VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no

way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any

pending operations like in flight requests. It must also preserve all

the necessary state (the virtqueue vring base plus the possible device

specific states) that is required for restoring in the future. The

device must not change its configuration after that point.

After the return of ioctl with stop == 0, the device can continue

processing buffers as long as typical conditions are met (vq is enabled,

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

DRIVER_OK status bit is enabled, etc).

Just to be clear, we are adding vdpa level new ioctl() that doesn’t map to any mechanism in the virtio spec.

Why can't we use this ioctl() to indicate driver to start/stop the device instead of driving it through the driver_ok? This is in the context of other discussion we had in the LM series.

If there's something in the spec that does this then let's use that.

Actually, we try to propose a independent feature here:

Quoted 7 lines
Quoted 5 lines

https://lists.oasis-open.org/archives/virtio-dev/202111/msg00020.html

Does it make sense to you?

Thanks

But I thought the LM patches are trying to replace all that?

I'm not sure, and actually I think they are orthogonal. We need a new state and the command to set the state could be transport specific or a virtqueue.

As far as I know, most of the vendors have implemented this semantic.

Thanks

Quoted 9 lines
Quoted 8 lines
Quoted 8 lines

Unfortunately the LM series seems to be stuck on moving bits around with the admin virtqueue ...

--
MST

#23 lore raw
In reply to Jason Wang
Quoted 61 lines

From: Jason Wang <[email protected]>
Sent: Sunday, May 29, 2022 11:39 PM

Quoted 17 lines · Michael S. Tsirkin
Quoted 15 lines · Parav Pandit
Quoted 15 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, May 26, 2022 8:44 AM

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer

that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration,

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

since the device could modify them after userland gets them. There are

individual ways to perform that action for some devices

(VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but

there

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

was no

way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any

pending operations like in flight requests. It must also preserve all

the necessary state (the virtqueue vring base plus the possible device

specific states) that is required for restoring in the future. The

device must not change its configuration after that point.

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

After the return of ioctl with stop == 0, the device can continue

processing buffers as long as typical conditions are met (vq is enabled,

DRIVER_OK status bit is enabled, etc).

Just to be clear, we are adding vdpa level new ioctl() that doesn’t map to

any mechanism in the virtio spec.

Quoted 2 lines
Quoted 2 lines

Why can't we use this ioctl() to indicate driver to start/stop the device

Quoted 9 lines

instead of driving it through the driver_ok?

Quoted 3 lines
Quoted 1 line

This is in the context of other discussion we had in the LM series.

If there's something in the spec that does this then let's use that.

Actually, we try to propose a independent feature here:

https://lists.oasis-open.org/archives/virtio-dev/202111/msg00020.html

This will stop the device for all the operations.
Once the device is stopped, its state cannot be queried further as device won't respond. It has limited use case.
What we need is to stop non admin queue related portion of the device.

Quoted 10 lines

Does it make sense to you?

Thanks

Quoted 6 lines

Unfortunately the LM series seems to be stuck on moving bits around with the admin virtqueue ...

--
MST

#24 lore raw
Quoted 65 lines · Parav Pandit
Quoted 65 lines

From: Jason Wang <[email protected]>
Sent: Sunday, May 29, 2022 11:39 PM

Quoted 17 lines · Michael S. Tsirkin
Quoted 15 lines · Parav Pandit
Quoted 15 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, May 26, 2022 8:44 AM

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer

that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration,

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

since the device could modify them after userland gets them. There are

individual ways to perform that action for some devices

(VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but

there

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

was no

way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any

pending operations like in flight requests. It must also preserve all

the necessary state (the virtqueue vring base plus the possible device

specific states) that is required for restoring in the future. The

device must not change its configuration after that point.

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

After the return of ioctl with stop == 0, the device can continue

processing buffers as long as typical conditions are met (vq is enabled,

DRIVER_OK status bit is enabled, etc).

Just to be clear, we are adding vdpa level new ioctl() that doesn’t map to

any mechanism in the virtio spec.

Quoted 2 lines
Quoted 2 lines

Why can't we use this ioctl() to indicate driver to start/stop the device

instead of driving it through the driver_ok?

Quoted 2 lines
Quoted 1 line

This is in the context of other discussion we had in the LM series.

Quoted 7 lines
Quoted 6 lines
Quoted 1 line

If there's something in the spec that does this then let's use that.

Actually, we try to propose a independent feature here:

https://lists.oasis-open.org/archives/virtio-dev/202111/msg00020.html

This will stop the device for all the operations.

Well, the ability to query the virtqueue state was proposed as another feature (Eugenio, please correct me). This should be sufficient for making virtio-net to be live migrated.

https://lists.oasis-open.org/archives/virtio-comment/202103/msg00008.html

Quoted 3 lines

Once the device is stopped, its state cannot be queried further as device won't respond. It has limited use case.
What we need is to stop non admin queue related portion of the device.

See above.

Thanks

Quoted 12 lines
Quoted 11 lines

Does it make sense to you?

Thanks

Quoted 6 lines

Unfortunately the LM series seems to be stuck on moving bits around with the admin virtqueue ...

--
MST

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#25 lore raw
Quoted 8 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, May 31, 2022 10:42 PM

Well, the ability to query the virtqueue state was proposed as another feature (Eugenio, please correct me). This should be sufficient for making virtio-net to be live migrated.

The device is stopped, it won't answer to this special vq config done here. Programming all of these using cfg registers doesn't scale for on-chip memory and for the speed.

Next would be to program hundreds of statistics of the 64 VQs through giant PCI config space register in some busy polling scheme.

I can clearly see how all these are inefficient for faster LM. We need an efficient AQ to proceed with at minimum.

Quoted 6 lines

https://lists.oasis-open.org/archives/virtio-comment/202103/msg00008.html

Quoted 1 line

Once the device is stopped, its state cannot be queried further as device

won't respond.

Quoted 2 lines

It has limited use case.
What we need is to stop non admin queue related portion of the device.

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#26 lore raw
Quoted 10 lines · Parav Pandit
Quoted 9 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, May 31, 2022 10:42 PM

Well, the ability to query the virtqueue state was proposed as another feature (Eugenio, please correct me). This should be sufficient for making virtio-net to be live migrated.

The device is stopped, it won't answer to this special vq config done here.

This depends on the definition of the stop. Any query to the device state should be allowed otherwise it's meaningless for us.

Quoted 1 line

Programming all of these using cfg registers doesn't scale for on-chip memory and for the speed.

Well, they are orthogonal and what I want to say is, we should first define the semantics of stop and state of the virtqueue.

Such a facility could be accessed by either transport specific method or admin virtqueue, it totally depends on the hardware architecture of the vendor.

Quoted 2 lines

Next would be to program hundreds of statistics of the 64 VQs through a giant PCI config space register in some busy polling scheme.

We don't need giant config space, and this method has been implemented by some vDPA vendors.

Quoted 3 lines

I can clearly see how all these are inefficient for faster LM. We need an efficient AQ to proceed with at minimum.

I'm fine with admin virtqueue, but the stop and state are orthogonal to that. And using admin virtqueue for stop/state will be more natural if we use admin virtqueue as a transport.

Thanks

Quoted 7 lines
Quoted 7 lines

https://lists.oasis-open.org/archives/virtio-comment/202103/msg00008.html

Quoted 1 line

Once the device is stopped, its state cannot be queried further as device

won't respond.

Quoted 2 lines

It has limited use case.
What we need is to stop non admin queue related portion of the device.

#27 lore raw
Quoted 19 lines

From: Jason Wang <[email protected]>
Sent: Wednesday, June 1, 2022 10:00 PM

Quoted 10 lines · Parav Pandit
Quoted 9 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, May 31, 2022 10:42 PM

Well, the ability to query the virtqueue state was proposed as another feature (Eugenio, please correct me). This should be sufficient for making virtio-net to be live migrated.

The device is stopped, it won't answer to this special vq config done here.

This depends on the definition of the stop. Any query to the device state should be allowed otherwise it's meaningless for us.

Quoted 9 lines
Quoted 1 line

Programming all of these using cfg registers doesn't scale for on-chip

memory and for the speed.

Well, they are orthogonal and what I want to say is, we should first define the semantics of stop and state of the virtqueue.

Such a facility could be accessed by either transport specific method or admin virtqueue, it totally depends on the hardware architecture of the vendor.

I find it hard to believe that a vendor can implement a CVQ but not AQ and chose to expose tens of hundreds of registers. But maybe, it fits some specific hw.

I like to learn the advantages of such method other than simplicity.

We can clearly that we are shifting away from such PCI registers with SIOV, IMS and other scalable solutions. virtio drifting in reverse direction by introducing more registers as transport.
I expect it to an optional transport like AQ.

Quoted 7 lines
Quoted 2 lines

Next would be to program hundreds of statistics of the 64 VQs through a

giant PCI config space register in some busy polling scheme.

We don't need giant config space, and this method has been implemented by some vDPA vendors.

There are tens of 64-bit counters per VQs. These needs to programmed on destination side. Programming these via registers requires exposing them on the registers.
In one of the proposals, I see them being queried via CVQ from the device.

Programming them via cfg registers requires large cfg space or synchronous programming until receiving ACK from it. This means one entry at a time...

Programming them via CVQ needs replicate and align cmd values etc on all device types. All duplicate and hard to maintain.

Quoted 7 lines
Quoted 3 lines

I can clearly see how all these are inefficient for faster LM. We need an efficient AQ to proceed with at minimum.

I'm fine with admin virtqueue, but the stop and state are orthogonal to that. And using admin virtqueue for stop/state will be more natural if we use admin virtqueue as a transport.

Ok.
We should have defined it bit earlier that all vendors can use. :(

#28 lore raw
Quoted 23 lines · Parav Pandit
Quoted 23 lines

From: Jason Wang <[email protected]>
Sent: Wednesday, June 1, 2022 10:00 PM

Quoted 10 lines · Parav Pandit
Quoted 9 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, May 31, 2022 10:42 PM

Well, the ability to query the virtqueue state was proposed as another feature (Eugenio, please correct me). This should be sufficient for making virtio-net to be live migrated.

The device is stopped, it won't answer to this special vq config done here.

This depends on the definition of the stop. Any query to the device state should be allowed otherwise it's meaningless for us.

Quoted 1 line

Programming all of these using cfg registers doesn't scale for on-chip

memory and for the speed.

Quoted 8 lines
Quoted 6 lines

Well, they are orthogonal and what I want to say is, we should first define the semantics of stop and state of the virtqueue.

Such a facility could be accessed by either transport specific method or admin virtqueue, it totally depends on the hardware architecture of the vendor.

I find it hard to believe that a vendor can implement a CVQ but not AQ and chose to expose tens of hundreds of registers. But maybe, it fits some specific hw.

You can have a look at the ifcvf dpdk driver as an example.

But another thing that is unrelated to hardware architecture is the nesting support. Having admin virtqueue in a nesting environment looks like an overkill. Presenting a register in L1 and map it to L0's admin should be good enough.

Quoted 6 lines

I like to learn the advantages of such method other than simplicity.

We can clearly that we are shifting away from such PCI registers with SIOV, IMS and other scalable solutions. virtio drifting in reverse direction by introducing more registers as transport.
I expect it to an optional transport like AQ.

Actually, I had a proposal of using admin virtqueue as a transport, it's designed to be SIOV/IMS capable. And it's not hard to extend it with the state/stop support etc.

Quoted 11 lines
Quoted 8 lines
Quoted 3 lines

Next would be to program hundreds of statistics of the 64 VQs through a

giant PCI config space register in some busy polling scheme.

We don't need giant config space, and this method has been implemented by some vDPA vendors.

There are tens of 64-bit counters per VQs. These needs to programmed on destination side. Programming these via registers requires exposing them on the registers.
In one of the proposals, I see them being queried via CVQ from the device.

I didn't see a proposal like this. And I don't think querying general virtio state like idx with a device specific CVQ is a good design.

Quoted 8 lines

Programming them via cfg registers requires large cfg space or synchronous programming until receiving ACK from it. This means one entry at a time...

Programming them via CVQ needs replicate and align cmd values etc on all device types. All duplicate and hard to maintain.

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

I can clearly see how all these are inefficient for faster LM. We need an efficient AQ to proceed with at minimum.

I'm fine with admin virtqueue, but the stop and state are orthogonal to that. And using admin virtqueue for stop/state will be more natural if we use admin virtqueue as a transport.

Ok.
We should have defined it bit earlier that all vendors can use. :(

I agree.

Thanks

#29 lore raw
Quoted 36 lines

From: Jason Wang <[email protected]>
Sent: Wednesday, June 1, 2022 11:54 PM

Quoted 8 lines · Parav Pandit
Quoted 8 lines

From: Jason Wang <[email protected]>
Sent: Wednesday, June 1, 2022 10:00 PM

Quoted 2 lines · Parav Pandit
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, May 31, 2022 10:42 PM

Well, the ability to query the virtqueue state was proposed as another feature (Eugenio, please correct me). This should be sufficient for making virtio-net to be live migrated.

The device is stopped, it won't answer to this special vq config done

here.

Quoted 6 lines
Quoted 6 lines

This depends on the definition of the stop. Any query to the device state should be allowed otherwise it's meaningless for us.

Quoted 2 lines

Programming all of these using cfg registers doesn't scale for on-chip

Quoted 7 lines
Quoted 7 lines

memory and for the speed.

Well, they are orthogonal and what I want to say is, we should first define the semantics of stop and state of the virtqueue.

Such a facility could be accessed by either transport specific method or admin virtqueue, it totally depends on the hardware

Quoted 8 lines

architecture of the vendor.

Quoted 2 lines
Quoted 1 line

I find it hard to believe that a vendor can implement a CVQ but not AQ and

chose to expose tens of hundreds of registers.

Quoted 1 line

But maybe, it fits some specific hw.

You can have a look at the ifcvf dpdk driver as an example.

Ifcvf is an example of using registers.
It is not an answer why AQ is hard for it. :)
virtio spec has definition of queue now and implementing yet another queue shouldn't be a problem.

So far no one seem to have problem with the additional queue. So I take it as AQ is ok.

Quoted 4 lines

But another thing that is unrelated to hardware architecture is the nesting support. Having admin virtqueue in a nesting environment looks like an overkill. Presenting a register in L1 and map it to L0's admin should be good enough.

So may be a optimized interface can be added that fits nested env.
At this point in time real users that we heard are interested in non-nested use cases. Let's enable them first.

Quoted 23 lines
Quoted 5 lines

I like to learn the advantages of such method other than simplicity.

We can clearly that we are shifting away from such PCI registers with SIOV,

IMS and other scalable solutions.

Quoted 1 line

virtio drifting in reverse direction by introducing more registers as

transport.

Quoted 1 line

I expect it to an optional transport like AQ.

Actually, I had a proposal of using admin virtqueue as a transport, it's designed to be SIOV/IMS capable. And it's not hard to extend it with the state/stop support etc.

Quoted 9 lines
Quoted 9 lines
Quoted 4 lines

Next would be to program hundreds of statistics of the 64 VQs through a

giant PCI config space register in some busy polling scheme.

We don't need giant config space, and this method has been implemented by some vDPA vendors.

Quoted 8 lines
Quoted 1 line

There are tens of 64-bit counters per VQs. These needs to programmed on

destination side.

Quoted 2 lines

Programming these via registers requires exposing them on the registers. In one of the proposals, I see them being queried via CVQ from the device.

I didn't see a proposal like this. And I don't think querying general virtio state like idx with a device specific CVQ is a good design.

My example was not for the idx. But for VQ statistics that is queried via CVQ.

Quoted 13 lines
Quoted 2 lines

Programming them via cfg registers requires large cfg space or synchronous

programming until receiving ACK from it.

Quoted 3 lines

This means one entry at a time...

Programming them via CVQ needs replicate and align cmd values etc on all

device types. All duplicate and hard to maintain.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

I can clearly see how all these are inefficient for faster LM. We need an efficient AQ to proceed with at minimum.

Quoted 8 lines
Quoted 1 line
Quoted 1 line

I'm fine with admin virtqueue, but the stop and state are orthogonal to

that.

Quoted 4 lines
Quoted 2 lines

And using admin virtqueue for stop/state will be more natural if we use admin virtqueue as a transport.

Ok.
We should have defined it bit earlier that all vendors can use. :(

I agree.

I remember few months back, you acked in the weekly meeting that TC has approved the AQ direction. And we are still in this circle of debating the AQ.

#30 lore raw
Quoted 39 lines · Parav Pandit
Quoted 39 lines

From: Jason Wang <[email protected]>
Sent: Wednesday, June 1, 2022 11:54 PM

Quoted 8 lines · Parav Pandit
Quoted 8 lines

From: Jason Wang <[email protected]>
Sent: Wednesday, June 1, 2022 10:00 PM

Quoted 2 lines · Parav Pandit
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, May 31, 2022 10:42 PM

Well, the ability to query the virtqueue state was proposed as another feature (Eugenio, please correct me). This should be sufficient for making virtio-net to be live migrated.

The device is stopped, it won't answer to this special vq config done

here.

Quoted 6 lines
Quoted 6 lines

This depends on the definition of the stop. Any query to the device state should be allowed otherwise it's meaningless for us.

Quoted 2 lines

Programming all of these using cfg registers doesn't scale for on-chip

Quoted 7 lines
Quoted 7 lines

memory and for the speed.

Well, they are orthogonal and what I want to say is, we should first define the semantics of stop and state of the virtqueue.

Such a facility could be accessed by either transport specific method or admin virtqueue, it totally depends on the hardware

architecture of the vendor.

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 1 line

I find it hard to believe that a vendor can implement a CVQ but not AQ and

chose to expose tens of hundreds of registers.

Quoted 1 line

But maybe, it fits some specific hw.

You can have a look at the ifcvf dpdk driver as an example.

Ifcvf is an example of using registers.
It is not an answer why AQ is hard for it. :)

Well, it's an example of how vDPA is implemented. I think we agree that for vDPA, vendors have the flexibility to implement their perferrable datapath.

Quoted 11 lines

virtio spec has definition of queue now and implementing yet another queue shouldn't be a problem.

So far no one seem to have problem with the additional queue. So I take it as AQ is ok.

Quoted 4 lines

But another thing that is unrelated to hardware architecture is the nesting support. Having admin virtqueue in a nesting environment looks like an overkill. Presenting a register in L1 and map it to L0's admin should be good enough.

So may be a optimized interface can be added that fits nested env.
At this point in time real users that we heard are interested in non-nested use cases. Let's enable them first.

That's fine. For nests, it's actually really easy, just adding an interface within the existing transport should be sufficient.

Quoted 52 lines
Quoted 25 lines
Quoted 7 lines

I like to learn the advantages of such method other than simplicity.

We can clearly that we are shifting away from such PCI registers with SIOV,

IMS and other scalable solutions.

Quoted 1 line

virtio drifting in reverse direction by introducing more registers as

transport.

Quoted 1 line

I expect it to an optional transport like AQ.

Actually, I had a proposal of using admin virtqueue as a transport, it's designed to be SIOV/IMS capable. And it's not hard to extend it with the state/stop support etc.

Quoted 9 lines
Quoted 9 lines
Quoted 4 lines

Next would be to program hundreds of statistics of the 64 VQs through a

giant PCI config space register in some busy polling scheme.

We don't need giant config space, and this method has been implemented by some vDPA vendors.

Quoted 8 lines
Quoted 1 line

There are tens of 64-bit counters per VQs. These needs to programmed on

destination side.

Quoted 2 lines

Programming these via registers requires exposing them on the registers. In one of the proposals, I see them being queried via CVQ from the device.

I didn't see a proposal like this. And I don't think querying general virtio state like idx with a device specific CVQ is a good design.

My example was not for the idx. But for VQ statistics that is queried via CVQ.

Quoted 17 lines
Quoted 2 lines

Programming them via cfg registers requires large cfg space or synchronous

programming until receiving ACK from it.

Quoted 3 lines

This means one entry at a time...

Programming them via CVQ needs replicate and align cmd values etc on all

device types. All duplicate and hard to maintain.

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

I can clearly see how all these are inefficient for faster LM. We need an efficient AQ to proceed with at minimum.

I'm fine with admin virtqueue, but the stop and state are orthogonal to

that.

Quoted 2 lines
Quoted 2 lines

And using admin virtqueue for stop/state will be more natural if we use admin virtqueue as a transport.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Ok.
We should have defined it bit earlier that all vendors can use. :(

I agree.

I remember few months back, you acked in the weekly meeting that TC has approved the AQ direction. And we are still in this circle of debating the AQ.

I think not. Just to make sure we are on the same page, the proposal here is for vDPA, and hope it can provide forward compatibility to virtio. So in the context of vDPA, admin virtqueue is not a must.

Thanks

#31 lore raw
Quoted 7 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, June 14, 2022 9:29 PM

Well, it's an example of how vDPA is implemented. I think we agree that for vDPA, vendors have the flexibility to implement their perferrable datapath.

Yes for the vdpa level and for the virtio level.

Quoted 8 lines
Quoted 2 lines

I remember few months back, you acked in the weekly meeting that TC has

approved the AQ direction.

Quoted 1 line

And we are still in this circle of debating the AQ.

I think not. Just to make sure we are on the same page, the proposal here is for vDPA, and hope it can provide forward compatibility to virtio. So in the context of vDPA, admin virtqueue is not a must.

In context of vdpa over virtio, an efficient transport interface is needed.
If AQ is not much any other interface such as hundreds to thousands of registers is not must either.

AQ is one interface proposed with multiple benefits.
I haven’t seen any other alternatives that delivers all the benefits. Only one I have seen is synchronous config registers.

If you let vendors progress, handful of sensible interfaces can exist, each with different characteristics. How would we proceed from here?

#32 lore raw
Quoted 21 lines · Parav Pandit
Quoted 8 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, June 14, 2022 9:29 PM

Well, it's an example of how vDPA is implemented. I think we agree that for vDPA, vendors have the flexibility to implement their perferrable datapath.

Yes for the vdpa level and for the virtio level.

Quoted 8 lines
Quoted 2 lines

I remember few months back, you acked in the weekly meeting that TC has

approved the AQ direction.

Quoted 1 line

And we are still in this circle of debating the AQ.

I think not. Just to make sure we are on the same page, the proposal here is for vDPA, and hope it can provide forward compatibility to virtio. So in the context of vDPA, admin virtqueue is not a must.

In context of vdpa over virtio, an efficient transport interface is needed.
If AQ is not much any other interface such as hundreds to thousands of registers is not must either.

Quoted 6 lines

AQ is one interface proposed with multiple benefits.
I haven’t seen any other alternatives that delivers all the benefits. Only one I have seen is synchronous config registers.

If you let vendors progress, handful of sensible interfaces can exist, each with different characteristics. How would we proceed from here?

I'm pretty fine with having admin virtqueue in the virtio spec. If you remember, I've even submitted a proposal to use admin virtqueue as a transport last year.

Let's just proceed in the virtio-dev list.

Thanks

#33 lore raw
Quoted 35 lines

From: Jason Wang <[email protected]>
Sent: Thursday, June 16, 2022 9:15 PM

Quoted 7 lines · Parav Pandit
Quoted 7 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, June 14, 2022 9:29 PM

Well, it's an example of how vDPA is implemented. I think we agree that for vDPA, vendors have the flexibility to implement their perferrable

datapath.

Quoted 6 lines
Quoted 1 line

Yes for the vdpa level and for the virtio level.

Quoted 3 lines
Quoted 3 lines

I remember few months back, you acked in the weekly meeting that TC has

Quoted 8 lines
Quoted 6 lines

approved the AQ direction.

Quoted 1 line

And we are still in this circle of debating the AQ.

I think not. Just to make sure we are on the same page, the proposal here is for vDPA, and hope it can provide forward compatibility to virtio. So in the context of vDPA, admin virtqueue is not a must.

In context of vdpa over virtio, an efficient transport interface is needed. If AQ is not much any other interface such as hundreds to thousands of

registers is not must either.

Quoted 6 lines

AQ is one interface proposed with multiple benefits.
I haven’t seen any other alternatives that delivers all the benefits. Only one I have seen is synchronous config registers.

If you let vendors progress, handful of sensible interfaces can exist, each

Quoted 8 lines

with different characteristics.

Quoted 1 line

How would we proceed from here?

I'm pretty fine with having admin virtqueue in the virtio spec. If you remember, I've even submitted a proposal to use admin virtqueue as a transport last year.

Let's just proceed in the virtio-dev list.

o.k. thanks. I am aligned with your thoughts now.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author maintainer
#34 lore raw
In reply to Parav Pandit
Quoted 45 lines · Parav Pandit
Quoted 20 lines

From: Jason Wang <[email protected]>
Sent: Wednesday, June 1, 2022 10:00 PM

Quoted 10 lines · Parav Pandit
Quoted 9 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, May 31, 2022 10:42 PM

Well, the ability to query the virtqueue state was proposed as another feature (Eugenio, please correct me). This should be sufficient for making virtio-net to be live migrated.

The device is stopped, it won't answer to this special vq config done here.

This depends on the definition of the stop. Any query to the device state should be allowed otherwise it's meaningless for us.

Quoted 9 lines
Quoted 1 line

Programming all of these using cfg registers doesn't scale for on-chip

memory and for the speed.

Well, they are orthogonal and what I want to say is, we should first define the semantics of stop and state of the virtqueue.

Such a facility could be accessed by either transport specific method or admin virtqueue, it totally depends on the hardware architecture of the vendor.

I find it hard to believe that a vendor can implement a CVQ but not AQ and chose to expose tens of hundreds of registers. But maybe, it fits some specific hw.

I like to learn the advantages of such method other than simplicity.

We can clearly that we are shifting away from such PCI registers with SIOV, IMS and other scalable solutions. virtio drifting in reverse direction by introducing more registers as transport.
I expect it to an optional transport like AQ.

Quoted 7 lines
Quoted 2 lines

Next would be to program hundreds of statistics of the 64 VQs through a

giant PCI config space register in some busy polling scheme.

We don't need giant config space, and this method has been implemented by some vDPA vendors.

Quoted 9 lines

There are tens of 64-bit counters per VQs. These needs to programmed on destination side. Programming these via registers requires exposing them on the registers.
In one of the proposals, I see them being queried via CVQ from the device.

Programming them via cfg registers requires large cfg space or synchronous programming until receiving ACK from it. This means one entry at a time...

Programming them via CVQ needs replicate and align cmd values etc on all device types. All duplicate and hard to maintain.

I think this discussion should be moved to the proposals on virtio-comment. In the vdpa context, they should be covered.

This one is about exposing the basic facility of stopping and resuming a device to userland, and it fits equally well if the device implements it via cfg registers, via admin vq, via channel I/O, or via whatever transport the vdpa backend prefers. To ask for the state is already covered in the vhost layer, and this proposal does not affect it.

Given the flexibility of vdpa, we can even ask vq state using backend-specific methods, cache it (knowing that there will be no change of them until resume or DRIVER_OK), and expose them to qemu using config space interface or any other batch method. Same as with the enable_vq problem. And the same applies to stats. And we maintain compatibility with all vendor-specific control plane.

Would that work for devices that cannot or does not want to expose them via config space?

Thanks!

Quoted 10 lines
Quoted 8 lines
Quoted 4 lines

I can clearly see how all these are inefficient for faster LM. We need an efficient AQ to proceed with at minimum.

I'm fine with admin virtqueue, but the stop and state are orthogonal to that. And using admin virtqueue for stop/state will be more natural if we use admin virtqueue as a transport.

Ok.
We should have defined it bit earlier that all vendors can use. :(

author maintainer
#35 lore raw
In reply to Parav Pandit
Quoted 67 lines · Parav Pandit
Quoted 67 lines

From: Jason Wang <[email protected]>
Sent: Sunday, May 29, 2022 11:39 PM

Quoted 17 lines · Michael S. Tsirkin
Quoted 15 lines · Parav Pandit
Quoted 15 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, May 26, 2022 8:44 AM

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer

that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration,

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

since the device could modify them after userland gets them. There are

individual ways to perform that action for some devices

(VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but

there

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

was no

way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any

pending operations like in flight requests. It must also preserve all

the necessary state (the virtqueue vring base plus the possible device

specific states) that is required for restoring in the future. The

device must not change its configuration after that point.

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

After the return of ioctl with stop == 0, the device can continue

processing buffers as long as typical conditions are met (vq is enabled,

DRIVER_OK status bit is enabled, etc).

Just to be clear, we are adding vdpa level new ioctl() that doesn’t map to

any mechanism in the virtio spec.

Quoted 2 lines
Quoted 2 lines

Why can't we use this ioctl() to indicate driver to start/stop the device

instead of driving it through the driver_ok?

Quoted 3 lines
Quoted 1 line

This is in the context of other discussion we had in the LM series.

If there's something in the spec that does this then let's use that.

Quoted 9 lines
Quoted 4 lines

Actually, we try to propose a independent feature here:

https://lists.oasis-open.org/archives/virtio-dev/202111/msg00020.html

This will stop the device for all the operations.
Once the device is stopped, its state cannot be queried further as device won't respond. It has limited use case.
What we need is to stop non admin queue related portion of the device.

Still don't follow this, sorry.

Adding the admin vq to the mix, this would stop a device of a device group, but not the whole virtqueue group. If the admin VQ is offered by the PF (since it's not exposed to the guest), it will continue accepting requests as normal. If it's exposed in the VF, I think the best bet is to shadow it, since guest and host requests could conflict.

Since this is offered through vdpa, the device backend driver can route it to whatever method works better for the hardware. For example, to send an admin vq command to the PF. That's why it's important to keep the feature as self-contained and orthogonal to others as possible.

Quoted 11 lines
Quoted 10 lines

Does it make sense to you?

Thanks

Quoted 6 lines

Unfortunately the LM series seems to be stuck on moving bits around with the admin virtqueue ...

--
MST

#36 lore raw
Quoted 85 lines

From: Eugenio Perez Martin <[email protected]>
Sent: Wednesday, June 1, 2022 5:50 AM

Quoted 6 lines · Parav Pandit
Quoted 6 lines

From: Jason Wang <[email protected]>
Sent: Sunday, May 29, 2022 11:39 PM

On Fri, May 27, 2022 at 6:56 PM Michael S. Tsirkin <[email protected]>

Quoted 17 lines · wrote:
Quoted 17 lines
Quoted 17 lines · Parav Pandit
Quoted 15 lines
Quoted 15 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, May 26, 2022 8:44 AM

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer

that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration,

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

since the device could modify them after userland gets them. There are

individual ways to perform that action for some devices

(VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...)

but

Quoted 4 lines
Quoted 4 lines

there

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

was no

way to perform it for any vhost device (and, in particular, vhost-

vdpa).

Quoted 43 lines
Quoted 43 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines

After the return of ioctl with stop != 0, the device MUST finish any

pending operations like in flight requests. It must also preserve all

the necessary state (the virtqueue vring base plus the possible device

specific states) that is required for restoring in the future. The

device must not change its configuration after that point.

After the return of ioctl with stop == 0, the device can continue

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

processing buffers as long as typical conditions are met (vq is enabled,

DRIVER_OK status bit is enabled, etc).

Just to be clear, we are adding vdpa level new ioctl() that doesn’t map to

any mechanism in the virtio spec.

Quoted 3 lines
Quoted 3 lines

Why can't we use this ioctl() to indicate driver to start/stop the device

instead of driving it through the driver_ok?

Quoted 3 lines
Quoted 1 line

This is in the context of other discussion we had in the LM series.

If there's something in the spec that does this then let's use that.

Actually, we try to propose a independent feature here:

https://lists.oasis-open.org/archives/virtio-dev/202111/msg00020.htm
l

Quoted 8 lines
Quoted 2 lines

This will stop the device for all the operations.
Once the device is stopped, its state cannot be queried further as device

won't respond.

Quoted 3 lines

It has limited use case.
What we need is to stop non admin queue related portion of the device.

Still don't follow this, sorry.

Once a device it stopped its state etc cannot be queried.
if you want to stop and still allow certain operations, a better spec definition is needed that says,

stop A, B, C, but allow D, E, F, G.
A = stop CVQs and save its state somewhere
B = stop data VQs and save it state somewhere
C = stop generic config interrupt

D = query state of multiple VQs
E = query device statistics and other elements/objects in future F = setup/config/restore certain fields
G = resume the device

Quoted 7 lines

Adding the admin vq to the mix, this would stop a device of a device group, but not the whole virtqueue group. If the admin VQ is offered by the PF (since it's not exposed to the guest), it will continue accepting requests as normal. If it's exposed in the VF, I think the best bet is to shadow it, since guest and host requests could conflict.

Quoted 5 lines

Since this is offered through vdpa, the device backend driver can route it to whatever method works better for the hardware. For example, to send an admin vq command to the PF. That's why it's important to keep the feature as self-contained and orthogonal to others as possible.

I replied in other thread to continue there.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#37 lore raw
Quoted 88 lines · Parav Pandit
Quoted 88 lines

From: Eugenio Perez Martin <[email protected]>
Sent: Wednesday, June 1, 2022 5:50 AM

Quoted 6 lines · Parav Pandit
Quoted 6 lines

From: Jason Wang <[email protected]>
Sent: Sunday, May 29, 2022 11:39 PM

On Fri, May 27, 2022 at 6:56 PM Michael S. Tsirkin <[email protected]>

Quoted 17 lines · wrote:
Quoted 17 lines
Quoted 17 lines · Parav Pandit
Quoted 15 lines
Quoted 15 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, May 26, 2022 8:44 AM

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer

that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration,

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

since the device could modify them after userland gets them. There are

individual ways to perform that action for some devices

(VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...)

but

Quoted 4 lines
Quoted 4 lines

there

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

was no

way to perform it for any vhost device (and, in particular, vhost-

vdpa).

Quoted 45 lines
Quoted 34 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines

After the return of ioctl with stop != 0, the device MUST finish any

pending operations like in flight requests. It must also preserve all

the necessary state (the virtqueue vring base plus the possible device

specific states) that is required for restoring in the future. The

device must not change its configuration after that point.

After the return of ioctl with stop == 0, the device can continue

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

processing buffers as long as typical conditions are met (vq is enabled,

DRIVER_OK status bit is enabled, etc).

Just to be clear, we are adding vdpa level new ioctl() that doesn’t map to

any mechanism in the virtio spec.

Quoted 3 lines
Quoted 3 lines

Why can't we use this ioctl() to indicate driver to start/stop the device

instead of driving it through the driver_ok?

Quoted 9 lines
Quoted 3 lines
Quoted 1 line

This is in the context of other discussion we had in the LM series.

If there's something in the spec that does this then let's use that.

Actually, we try to propose a independent feature here:

https://lists.oasis-open.org/archives/virtio-dev/202111/msg00020.htm
l

This will stop the device for all the operations.
Once the device is stopped, its state cannot be queried further as device

Quoted 7 lines
Quoted 6 lines

won't respond.

Quoted 3 lines

It has limited use case.
What we need is to stop non admin queue related portion of the device.

Still don't follow this, sorry.

Once a device it stopped its state etc cannot be queried.

This is not what is proposed here.

Quoted 6 lines

if you want to stop and still allow certain operations, a better spec definition is needed that says,

stop A, B, C, but allow D, E, F, G.
A = stop CVQs and save its state somewhere B = stop data VQs and save it state somewhere C = stop generic config interrupt

Actually, it's the stop of the config space change. And what more, any guest visible state must not be changed.

Quoted 3 lines

D = query state of multiple VQs
E = query device statistics and other elements/objects in future

This is the device state I believe.

Quoted 1 line

F = setup/config/restore certain fields

This is the reverse of D and E, that is setting the state.

Quoted 2 lines

G = resume the device

Thanks

Quoted 7 lines
Quoted 7 lines

Adding the admin vq to the mix, this would stop a device of a device group, but not the whole virtqueue group. If the admin VQ is offered by the PF (since it's not exposed to the guest), it will continue accepting requests as normal. If it's exposed in the VF, I think the best bet is to shadow it, since guest and host requests could conflict.

Quoted 7 lines
Quoted 5 lines

Since this is offered through vdpa, the device backend driver can route it to whatever method works better for the hardware. For example, to send an admin vq command to the PF. That's why it's important to keep the feature as self-contained and orthogonal to others as possible.

I replied in other thread to continue there.

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#38 lore raw
In reply to Eugenio Pérez
Quoted 15 lines · Eugenio Pérez

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration, since the device could modify them after userland gets them. There are individual ways to perform that action for some devices (VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any pending operations like in flight requests. It must also preserve all the necessary state (the virtqueue vring base plus the possible device specific states) that is required for restoring in the future. The device must not change its configuration after that point.

Quoted 8 lines

After the return of ioctl with stop == 0, the device can continue processing buffers as long as typical conditions are met (vq is enabled, DRIVER_OK status bit is enabled, etc).

In the future, we will provide features similar to VHOST_USER_GET_INFLIGHT_FD so the device can save pending operations.

Comments are welcome.

So given this is just for simulator and affects UAPI I think it's fine to make it wait for the next merge window, until there's a consensus. Right?

Quoted 18 lines

v4:
* Replace VHOST_STOP to VHOST_VDPA_STOP in vhost ioctl switch case too.

v3:
* s/VHOST_STOP/VHOST_VDPA_STOP/
* Add documentation and requirements of the ioctl above its definition.

v2:
* Replace raw _F_STOP with BIT_ULL(_F_STOP).
* Fix obtaining of stop ioctl arg (it was not obtained but written).
* Add stop to vdpa_sim_blk.

Eugenio Pérez (4):
vdpa: Add stop operation
vhost-vdpa: introduce STOP backend feature bit
vhost-vdpa: uAPI to stop the device
vdpa_sim: Implement stop vdpa op

Quoted 13 lines
 drivers/vdpa/vdpa_sim/vdpa_sim.c     | 21 +++++++++++++++++
 drivers/vdpa/vdpa_sim/vdpa_sim.h     |  1 +
 drivers/vdpa/vdpa_sim/vdpa_sim_blk.c |  3 +++
 drivers/vdpa/vdpa_sim/vdpa_sim_net.c |  3 +++
 drivers/vhost/vdpa.c                 | 34 +++++++++++++++++++++++++++-
 include/linux/vdpa.h                 |  6 +++++
 include/uapi/linux/vhost.h           | 14 ++++++++++++
 include/uapi/linux/vhost_types.h     |  2 ++
 8 files changed, 83 insertions(+), 1 deletion(-)
Signature · 1 line
2.31.1

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author maintainer
#39 lore raw
Quoted 24 lines · Michael S. Tsirkin
Quoted 22 lines · Eugenio Pérez

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration, since the device could modify them after userland gets them. There are individual ways to perform that action for some devices (VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any pending operations like in flight requests. It must also preserve all the necessary state (the virtqueue vring base plus the possible device specific states) that is required for restoring in the future. The device must not change its configuration after that point.

After the return of ioctl with stop == 0, the device can continue processing buffers as long as typical conditions are met (vq is enabled, DRIVER_OK status bit is enabled, etc).

In the future, we will provide features similar to VHOST_USER_GET_INFLIGHT_FD so the device can save pending operations.

Quoted 7 lines
Quoted 1 line

Comments are welcome.

So given this is just for simulator and affects UAPI I think it's fine to make it wait for the next merge window, until there's a consensus. Right?

While the change is only implemented in the simulator at this moment, it's just the very last missing piece in the kernel to implement complete live migration for net devices with cvq :). All vendor drivers can implement this call with current code, just a little bit of plumbing is needed. And it was accepted in previous meetings.

If it proves it works for every configuration (nested, etc), the implementation can forward the call to the admin vq for example. At the moment, it follows the proposed stop status bit sematic to stop the device, which POC has been tested in these circumstances.

Thanks!

Quoted 32 lines
Quoted 31 lines

v4:
* Replace VHOST_STOP to VHOST_VDPA_STOP in vhost ioctl switch case too.

v3:
* s/VHOST_STOP/VHOST_VDPA_STOP/
* Add documentation and requirements of the ioctl above its definition.

v2:
* Replace raw _F_STOP with BIT_ULL(_F_STOP).
* Fix obtaining of stop ioctl arg (it was not obtained but written).
* Add stop to vdpa_sim_blk.

Eugenio Pérez (4):
vdpa: Add stop operation
vhost-vdpa: introduce STOP backend feature bit
vhost-vdpa: uAPI to stop the device
vdpa_sim: Implement stop vdpa op

 drivers/vdpa/vdpa_sim/vdpa_sim.c     | 21 +++++++++++++++++
 drivers/vdpa/vdpa_sim/vdpa_sim.h     |  1 +
 drivers/vdpa/vdpa_sim/vdpa_sim_blk.c |  3 +++
 drivers/vdpa/vdpa_sim/vdpa_sim_net.c |  3 +++
 drivers/vhost/vdpa.c                 | 34 +++++++++++++++++++++++++++-
 include/linux/vdpa.h                 |  6 +++++
 include/uapi/linux/vhost.h           | 14 ++++++++++++
 include/uapi/linux/vhost_types.h     |  2 ++
 8 files changed, 83 insertions(+), 1 deletion(-)

--
2.31.1

#40 lore raw
Quoted 39 lines · Eugenio Perez Martin
Quoted 24 lines · Michael S. Tsirkin
Quoted 22 lines · Eugenio Pérez

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration, since the device could modify them after userland gets them. There are individual ways to perform that action for some devices (VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any pending operations like in flight requests. It must also preserve all the necessary state (the virtqueue vring base plus the possible device specific states) that is required for restoring in the future. The device must not change its configuration after that point.

After the return of ioctl with stop == 0, the device can continue processing buffers as long as typical conditions are met (vq is enabled, DRIVER_OK status bit is enabled, etc).

In the future, we will provide features similar to VHOST_USER_GET_INFLIGHT_FD so the device can save pending operations.

Quoted 7 lines
Quoted 1 line

Comments are welcome.

So given this is just for simulator and affects UAPI I think it's fine to make it wait for the next merge window, until there's a consensus. Right?

While the change is only implemented in the simulator at this moment, it's just the very last missing piece in the kernel to implement complete live migration for net devices with cvq :). All vendor drivers can implement this call with current code, just a little bit of plumbing is needed. And it was accepted in previous meetings.

Quoted 6 lines

If it proves it works for every configuration (nested, etc), the implementation can forward the call to the admin vq for example. At the moment, it follows the proposed stop status bit sematic to stop the device, which POC has been tested in these circumstances.

Thanks!

Oh absolutely, but I am guessing this plumbing won't be ready for this merge window.

Quoted 32 lines
Quoted 32 lines
Quoted 31 lines

v4:
* Replace VHOST_STOP to VHOST_VDPA_STOP in vhost ioctl switch case too.

v3:
* s/VHOST_STOP/VHOST_VDPA_STOP/
* Add documentation and requirements of the ioctl above its definition.

v2:
* Replace raw _F_STOP with BIT_ULL(_F_STOP).
* Fix obtaining of stop ioctl arg (it was not obtained but written).
* Add stop to vdpa_sim_blk.

Eugenio Pérez (4):
vdpa: Add stop operation
vhost-vdpa: introduce STOP backend feature bit
vhost-vdpa: uAPI to stop the device
vdpa_sim: Implement stop vdpa op

 drivers/vdpa/vdpa_sim/vdpa_sim.c     | 21 +++++++++++++++++
 drivers/vdpa/vdpa_sim/vdpa_sim.h     |  1 +
 drivers/vdpa/vdpa_sim/vdpa_sim_blk.c |  3 +++
 drivers/vdpa/vdpa_sim/vdpa_sim_net.c |  3 +++
 drivers/vhost/vdpa.c                 | 34 +++++++++++++++++++++++++++-
 include/linux/vdpa.h                 |  6 +++++
 include/uapi/linux/vhost.h           | 14 ++++++++++++
 include/uapi/linux/vhost_types.h     |  2 ++
 8 files changed, 83 insertions(+), 1 deletion(-)

--
2.31.1

#41 lore raw
In reply to Eugenio Pérez
Quoted 9 lines · Eugenio Pérez

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration, since the device could modify them after userland gets them. There are individual ways to perform that action for some devices (VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no way to perform it for any vhost device (and, in particular, vhost-vdpa).

Quoted 5 lines

After the return of ioctl with stop != 0, the device MUST finish any pending operations like in flight requests. It must also preserve all the necessary state (the virtqueue vring base plus the possible device specific states) that is required for restoring in the future. The device must not change its configuration after that point.

I think we probably need more accurate definition on the state as Parav suggested.

Besides this, we should also clarify when stop is allowed. E.g should we allow setting stop without DRIVER_OK?

Thanks

Quoted 38 lines

After the return of ioctl with stop == 0, the device can continue processing buffers as long as typical conditions are met (vq is enabled, DRIVER_OK status bit is enabled, etc).

In the future, we will provide features similar to VHOST_USER_GET_INFLIGHT_FD so the device can save pending operations.

Comments are welcome.

v4:
* Replace VHOST_STOP to VHOST_VDPA_STOP in vhost ioctl switch case too.

v3:
* s/VHOST_STOP/VHOST_VDPA_STOP/
* Add documentation and requirements of the ioctl above its definition.

v2:
* Replace raw _F_STOP with BIT_ULL(_F_STOP).
* Fix obtaining of stop ioctl arg (it was not obtained but written).
* Add stop to vdpa_sim_blk.

Eugenio Pérez (4):
vdpa: Add stop operation
vhost-vdpa: introduce STOP backend feature bit
vhost-vdpa: uAPI to stop the device
vdpa_sim: Implement stop vdpa op

  drivers/vdpa/vdpa_sim/vdpa_sim.c     | 21 +++++++++++++++++
  drivers/vdpa/vdpa_sim/vdpa_sim.h     |  1 +
  drivers/vdpa/vdpa_sim/vdpa_sim_blk.c |  3 +++
  drivers/vdpa/vdpa_sim/vdpa_sim_net.c |  3 +++
  drivers/vhost/vdpa.c                 | 34 +++++++++++++++++++++++++++-
  include/linux/vdpa.h                 |  6 +++++
  include/uapi/linux/vhost.h           | 14 ++++++++++++
  include/uapi/linux/vhost_types.h     |  2 ++
  8 files changed, 83 insertions(+), 1 deletion(-)