Skip to contentSkip to conversation

Implement vdpasim stop operation

PATCH Superseded Eugenio Pérez 3 participants 12 messages May 24, 2022 – May 25, 2022

Subsystems (5)

v2 of 8 Superseded Latest is v8 →

lore
Share & tools
lore

Eugenio Pérez

author maintainer
#0 lore raw

v2 cover Implement vdpasim stop operation

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration, since the device could modify them after userland gets them. There are individual ways to perform that action for some devices (VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any pending operations like in flight requests. It must also preserve all the necessary state (the virtqueue vring base plus the possible device specific states) that is required for restoring in the future. The device must not change its configuration after that point.

After the return of ioctl with stop == 0, the device can continue processing buffers as long as typical conditions are met (vq is enabled, DRIVER_OK status bit is enabled, etc).

In the future, we will provide features similar to VHOST_USER_GET_INFLIGHT_FD so the device can save pending operations.

Comments are welcome.

v2:
* Replace raw _F_STOP with BIT_ULL(_F_STOP).
* Fix obtaining of stop ioctl arg (it was not obtained but written).
* Add stop to vdpa_sim_blk.

Eugenio Pérez (4):
vdpa: Add stop operation
vhost-vdpa: introduce STOP backend feature bit
vhost-vdpa: uAPI to stop the device
vdpa_sim: Implement stop vdpa op

 drivers/vdpa/vdpa_sim/vdpa_sim.c     | 21 +++++++++++++++++
 drivers/vdpa/vdpa_sim/vdpa_sim.h     |  1 +
 drivers/vdpa/vdpa_sim/vdpa_sim_blk.c |  3 +++
 drivers/vdpa/vdpa_sim/vdpa_sim_net.c |  3 +++
 drivers/vhost/vdpa.c                 | 34 +++++++++++++++++++++++++++-
 include/linux/vdpa.h                 |  6 +++++
 include/uapi/linux/vhost.h           |  3 +++
 include/uapi/linux/vhost_types.h     |  2 ++
 8 files changed, 72 insertions(+), 1 deletion(-)
Signature · 1 line
2.27.0
author maintainer
#1 lore raw

PATCH v2 1/4 vdpa: Add stop operation

This operation is optional: It it's not implemented, backend feature bit will not be exposed.

---
 include/linux/vdpa.h | 6 ++++++
 1 file changed, 6 insertions(+)
include/linux/vdpa.h +6 -0
index 15af802d41c4..ddfebc4e1e01 100644
@@ -215,6 +215,11 @@ struct vdpa_map_file {
  * @reset:			Reset device
  *				@vdev: vdpa device
  *				Returns integer: success (0) or error (< 0)
+ * @stop:			Stop or resume the device (optional, but it must
+ *				be implemented if require device stop)
+ *				@vdev: vdpa device
+ *				@stop: stop (true), not stop (false)
+ *				Returns integer: success (0) or error (< 0)
  * @get_config_size:		Get the size of the configuration space includes
  *				fields that are conditional on feature bits.
  *				@vdev: vdpa device
@@ -316,6 +321,7 @@ struct vdpa_config_ops {
 	u8 (*get_status)(struct vdpa_device *vdev);
 	void (*set_status)(struct vdpa_device *vdev, u8 status);
 	int (*reset)(struct vdpa_device *vdev);
+	int (*stop)(struct vdpa_device *vdev, bool stop);
 	size_t (*get_config_size)(struct vdpa_device *vdev);
 	void (*get_config)(struct vdpa_device *vdev, unsigned int offset,
 			   void *buf, unsigned int len);
Signature · 1 line
2.27.0
author maintainer
#2 lore raw

PATCH v2 2/4 vhost-vdpa: introduce STOP backend feature bit

In reply to Eugenio Pérez

Userland knows if it can stop the device or not by checking this feature bit.

It's only offered if the vdpa driver backend implements the stop() operation callback, and try to set it if the backend does not offer that callback is an error.

---
 drivers/vhost/vdpa.c             | 16 +++++++++++++++-
 include/uapi/linux/vhost_types.h |  2 ++
 2 files changed, 17 insertions(+), 1 deletion(-)
drivers/vhost/vdpa.c +15 -1
index 1f1d1c425573..32713db5831d 100644
@@ -347,6 +347,14 @@ static long vhost_vdpa_set_config(struct vhost_vdpa *v,
 	return 0;
 }
 
+static bool vhost_vdpa_can_stop(const struct vhost_vdpa *v)
+{
+	struct vdpa_device *vdpa = v->vdpa;
+	const struct vdpa_config_ops *ops = vdpa->config;
+
+	return ops->stop;
+}
+
 static long vhost_vdpa_get_features(struct vhost_vdpa *v, u64 __user *featurep)
 {
 	struct vdpa_device *vdpa = v->vdpa;
@@ -575,7 +583,11 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
 	if (cmd == VHOST_SET_BACKEND_FEATURES) {
 		if (copy_from_user(&features, featurep, sizeof(features)))
 			return -EFAULT;
-		if (features & ~VHOST_VDPA_BACKEND_FEATURES)
+		if (features & ~(VHOST_VDPA_BACKEND_FEATURES |
+				 BIT_ULL(VHOST_BACKEND_F_STOP)))
+			return -EOPNOTSUPP;
+		if ((features & BIT_ULL(VHOST_BACKEND_F_STOP)) &&
+		     !vhost_vdpa_can_stop(v))
 			return -EOPNOTSUPP;
 		vhost_set_backend_features(&v->vdev, features);
 		return 0;
@@ -624,6 +636,8 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
 		break;
 	case VHOST_GET_BACKEND_FEATURES:
 		features = VHOST_VDPA_BACKEND_FEATURES;
+		if (vhost_vdpa_can_stop(v))
+			features |= BIT_ULL(VHOST_BACKEND_F_STOP);
 		if (copy_to_user(featurep, &features, sizeof(features)))
 			r = -EFAULT;
 		break;
include/uapi/linux/vhost_types.h +2 -0
index 634cee485abb..2758e665791b 100644
@@ -161,5 +161,7 @@ struct vhost_vdpa_iova_range {
  * message
  */
 #define VHOST_BACKEND_F_IOTLB_ASID  0x3
+/* Stop device from processing virtqueue buffers */
+#define VHOST_BACKEND_F_STOP  0x4
 
 #endif
Signature · 1 line
2.27.0
author maintainer
#3 lore raw

PATCH v2 3/4 vhost-vdpa: uAPI to stop the device

In reply to Eugenio Pérez

The ioctl adds support for stop the device from userspace.

---
 drivers/vhost/vdpa.c       | 18 ++++++++++++++++++
 include/uapi/linux/vhost.h |  3 +++
 2 files changed, 21 insertions(+)
drivers/vhost/vdpa.c +18 -0
index 32713db5831d..a5d33bad92f9 100644
@@ -478,6 +478,21 @@ static long vhost_vdpa_get_vqs_count(struct vhost_vdpa *v, u32 __user *argp)
 	return 0;
 }
 
+static long vhost_vdpa_stop(struct vhost_vdpa *v, u32 __user *argp)
+{
+	struct vdpa_device *vdpa = v->vdpa;
+	const struct vdpa_config_ops *ops = vdpa->config;
+	int stop;
+
+	if (!ops->stop)
+		return -EOPNOTSUPP;
+
+	if (copy_from_user(&stop, argp, sizeof(stop)))
+		return -EFAULT;
+
+	return ops->stop(vdpa, stop);
+}
+
 static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
 				   void __user *argp)
 {
@@ -650,6 +665,9 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
 	case VHOST_VDPA_GET_VQS_COUNT:
 		r = vhost_vdpa_get_vqs_count(v, argp);
 		break;
+	case VHOST_STOP:
+		r = vhost_vdpa_stop(v, argp);
+		break;
 	default:
 		r = vhost_dev_ioctl(&v->vdev, cmd, argp);
 		if (r == -ENOIOCTLCMD)
include/uapi/linux/vhost.h +3 -0
index cab645d4a645..e7526968ab0c 100644
@@ -171,4 +171,7 @@
 #define VHOST_VDPA_SET_GROUP_ASID	_IOW(VHOST_VIRTIO, 0x7C, \
 					     struct vhost_vring_state)
 
+/* Stop or resume a device so it does not process virtqueue requests anymore */
+#define VHOST_STOP			_IOW(VHOST_VIRTIO, 0x7D, int)
+
 #endif
Signature · 1 line
2.27.0
#4 lore raw

Re: [PATCH v2 3/4] vhost-vdpa: uAPI to stop the device

Quoted 50 lines · Eugenio Pérez

The ioctl adds support for stop the device from userspace.

---
 drivers/vhost/vdpa.c       | 18 ++++++++++++++++++
 include/uapi/linux/vhost.h |  3 +++
 2 files changed, 21 insertions(+)
drivers/vhost/vdpa.c +18 -0
index 32713db5831d..a5d33bad92f9 100644
@@ -478,6 +478,21 @@ static long vhost_vdpa_get_vqs_count(struct vhost_vdpa *v, u32 __user *argp)
        return 0;
 }
 
+static long vhost_vdpa_stop(struct vhost_vdpa *v, u32 __user *argp)
+{
+       struct vdpa_device *vdpa = v->vdpa;
+       const struct vdpa_config_ops *ops = vdpa->config;
+       int stop;
+
+       if (!ops->stop)
+               return -EOPNOTSUPP;
+
+       if (copy_from_user(&stop, argp, sizeof(stop)))
+               return -EFAULT;
+
+       return ops->stop(vdpa, stop);
+}
+
 static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
                                   void __user *argp)
 {
@@ -650,6 +665,9 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
        case VHOST_VDPA_GET_VQS_COUNT:
                r = vhost_vdpa_get_vqs_count(v, argp);
                break;
+       case VHOST_STOP:
+               r = vhost_vdpa_stop(v, argp);
+               break;
        default:
                r = vhost_dev_ioctl(&v->vdev, cmd, argp);
                if (r == -ENOIOCTLCMD)
include/uapi/linux/vhost.h +0 -0
index cab645d4a645..e7526968ab0c 100644

include/uapi/linux/vhost.h line 176

Quoted 6 lines
include/uapi/linux/vhost.h +3 -0
@@ -171,4 +171,7 @@
 #define VHOST_VDPA_SET_GROUP_ASID      _IOW(VHOST_VIRTIO, 0x7C, \
                                             struct vhost_vring_state)
 
+/* Stop or resume a device so it does not process virtqueue requests anymore */
+#define VHOST_STOP                     _IOW(VHOST_VIRTIO, 0x7D, int)
+

Unless we know it's a vhost general uAPI, let's use VHOST_VDPA_STOP here.

Thanks

Quoted 4 lines
include/uapi/linux/vhost.h +0 -1
 #endif
--

2.27.0

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author maintainer
#5 lore raw

Re: [PATCH v2 3/4] vhost-vdpa: uAPI to stop the device

Quoted 55 lines · Jason Wang
Quoted 53 lines · Eugenio Pérez

The ioctl adds support for stop the device from userspace.

---
 drivers/vhost/vdpa.c       | 18 ++++++++++++++++++
 include/uapi/linux/vhost.h |  3 +++
 2 files changed, 21 insertions(+)
drivers/vhost/vdpa.c +18 -0
index 32713db5831d..a5d33bad92f9 100644
@@ -478,6 +478,21 @@ static long vhost_vdpa_get_vqs_count(struct vhost_vdpa *v, u32 __user *argp)
        return 0;
 }
 
+static long vhost_vdpa_stop(struct vhost_vdpa *v, u32 __user *argp)
+{
+       struct vdpa_device *vdpa = v->vdpa;
+       const struct vdpa_config_ops *ops = vdpa->config;
+       int stop;
+
+       if (!ops->stop)
+               return -EOPNOTSUPP;
+
+       if (copy_from_user(&stop, argp, sizeof(stop)))
+               return -EFAULT;
+
+       return ops->stop(vdpa, stop);
+}
+
 static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
                                   void __user *argp)
 {
@@ -650,6 +665,9 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
        case VHOST_VDPA_GET_VQS_COUNT:
                r = vhost_vdpa_get_vqs_count(v, argp);
                break;
+       case VHOST_STOP:
+               r = vhost_vdpa_stop(v, argp);
+               break;
        default:
                r = vhost_dev_ioctl(&v->vdev, cmd, argp);
                if (r == -ENOIOCTLCMD)
include/uapi/linux/vhost.h +0 -0
index cab645d4a645..e7526968ab0c 100644
@@ -171,4 +171,7 @@
 #define VHOST_VDPA_SET_GROUP_ASID      _IOW(VHOST_VIRTIO, 0x7C, \
                                             struct vhost_vring_state)
 
Quoted 6 lines

include/uapi/linux/vhost.h line 176

Quoted 3 lines
include/uapi/linux/vhost.h +3 -0
+/* Stop or resume a device so it does not process virtqueue requests anymore */
+#define VHOST_STOP                     _IOW(VHOST_VIRTIO, 0x7D, int)
+

Unless we know it's a vhost general uAPI, let's use VHOST_VDPA_STOP here.

Ok I'll rename.

Thanks!

Quoted 7 lines

Thanks

Quoted 4 lines
include/uapi/linux/vhost.h +0 -1
 #endif
--

2.27.0

author maintainer
#6 lore raw

PATCH v2 4/4 vdpa_sim: Implement stop vdpa op

In reply to Eugenio Pérez

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration, since the device could modify them after userland gets them. There are individual ways to perform that action for some devices (VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any pending operations like in flight requests. It must also preserve all the necessary state (the virtqueue vring base plus the possible device specific states) that is required for restoring in the future. The device must not change its configuration after that point.

After the return of ioctl with stop == 0, the device can continue processing buffers as long as typical conditions are met (vq is enabled, DRIVER_OK status bit is enabled, etc).

In the future, we will provide features similar to VHOST_USER_GET_INFLIGHT_FD so the device can save pending operations.

---
 drivers/vdpa/vdpa_sim/vdpa_sim.c     | 21 +++++++++++++++++++++
 drivers/vdpa/vdpa_sim/vdpa_sim.h     |  1 +
 drivers/vdpa/vdpa_sim/vdpa_sim_blk.c |  3 +++
 drivers/vdpa/vdpa_sim/vdpa_sim_net.c |  3 +++
 4 files changed, 28 insertions(+)
drivers/vdpa/vdpa_sim/vdpa_sim.c +21 -0
index 50d721072beb..0515cf314bed 100644
@@ -107,6 +107,7 @@ static void vdpasim_do_reset(struct vdpasim *vdpasim)
 	for (i = 0; i < vdpasim->dev_attr.nas; i++)
 		vhost_iotlb_reset(&vdpasim->iommu[i]);
 
+	vdpasim->running = true;
 	spin_unlock(&vdpasim->iommu_lock);
 
 	vdpasim->features = 0;
@@ -505,6 +506,24 @@ static int vdpasim_reset(struct vdpa_device *vdpa)
 	return 0;
 }
 
+static int vdpasim_stop(struct vdpa_device *vdpa, bool stop)
+{
+	struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
+	int i;
+
+	spin_lock(&vdpasim->lock);
+	vdpasim->running = !stop;
+	if (vdpasim->running) {
+		/* Check for missed buffers */
+		for (i = 0; i < vdpasim->dev_attr.nvqs; ++i)
+			vdpasim_kick_vq(vdpa, i);
+
+	}
+	spin_unlock(&vdpasim->lock);
+
+	return 0;
+}
+
 static size_t vdpasim_get_config_size(struct vdpa_device *vdpa)
 {
 	struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
@@ -694,6 +713,7 @@ static const struct vdpa_config_ops vdpasim_config_ops = {
 	.get_status             = vdpasim_get_status,
 	.set_status             = vdpasim_set_status,
 	.reset			= vdpasim_reset,
+	.stop			= vdpasim_stop,
 	.get_config_size        = vdpasim_get_config_size,
 	.get_config             = vdpasim_get_config,
 	.set_config             = vdpasim_set_config,
@@ -726,6 +746,7 @@ static const struct vdpa_config_ops vdpasim_batch_config_ops = {
 	.get_status             = vdpasim_get_status,
 	.set_status             = vdpasim_set_status,
 	.reset			= vdpasim_reset,
+	.stop			= vdpasim_stop,
 	.get_config_size        = vdpasim_get_config_size,
 	.get_config             = vdpasim_get_config,
 	.set_config             = vdpasim_set_config,
drivers/vdpa/vdpa_sim/vdpa_sim.h +1 -0
index 622782e92239..061986f30911 100644
@@ -66,6 +66,7 @@ struct vdpasim {
 	u32 generation;
 	u64 features;
 	u32 groups;
+	bool running;
 	/* spinlock to synchronize iommu table */
 	spinlock_t iommu_lock;
 };
drivers/vdpa/vdpa_sim/vdpa_sim_blk.c +3 -0
index 42d401d43911..bcdb1982c378 100644
@@ -204,6 +204,9 @@ static void vdpasim_blk_work(struct work_struct *work)
 	if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK))
 		goto out;
 
+	if (!vdpasim->running)
+		goto out;
+
 	for (i = 0; i < VDPASIM_BLK_VQ_NUM; i++) {
 		struct vdpasim_virtqueue *vq = &vdpasim->vqs[i];
 
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +3 -0
index 5125976a4df8..886449e88502 100644
@@ -154,6 +154,9 @@ static void vdpasim_net_work(struct work_struct *work)
 
 	spin_lock(&vdpasim->lock);
 
+	if (!vdpasim->running)
+		goto out;
+
 	if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK))
 		goto out;
 
Signature · 1 line
2.27.0
#7 lore raw

Re: [PATCH v2 4/4] vdpa_sim: Implement stop vdpa op

Quoted 115 lines · Eugenio Pérez

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration, since the device could modify them after userland gets them. There are individual ways to perform that action for some devices (VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any pending operations like in flight requests. It must also preserve all the necessary state (the virtqueue vring base plus the possible device specific states) that is required for restoring in the future. The device must not change its configuration after that point.

After the return of ioctl with stop == 0, the device can continue processing buffers as long as typical conditions are met (vq is enabled, DRIVER_OK status bit is enabled, etc).

In the future, we will provide features similar to VHOST_USER_GET_INFLIGHT_FD so the device can save pending operations.

---
 drivers/vdpa/vdpa_sim/vdpa_sim.c     | 21 +++++++++++++++++++++
 drivers/vdpa/vdpa_sim/vdpa_sim.h     |  1 +
 drivers/vdpa/vdpa_sim/vdpa_sim_blk.c |  3 +++
 drivers/vdpa/vdpa_sim/vdpa_sim_net.c |  3 +++
 4 files changed, 28 insertions(+)
drivers/vdpa/vdpa_sim/vdpa_sim.c +21 -0
index 50d721072beb..0515cf314bed 100644
@@ -107,6 +107,7 @@ static void vdpasim_do_reset(struct vdpasim *vdpasim)
        for (i = 0; i < vdpasim->dev_attr.nas; i++)
                vhost_iotlb_reset(&vdpasim->iommu[i]);
 
+       vdpasim->running = true;
        spin_unlock(&vdpasim->iommu_lock);
 
        vdpasim->features = 0;
@@ -505,6 +506,24 @@ static int vdpasim_reset(struct vdpa_device *vdpa)
        return 0;
 }
 
+static int vdpasim_stop(struct vdpa_device *vdpa, bool stop)
+{
+       struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
+       int i;
+
+       spin_lock(&vdpasim->lock);
+       vdpasim->running = !stop;
+       if (vdpasim->running) {
+               /* Check for missed buffers */
+               for (i = 0; i < vdpasim->dev_attr.nvqs; ++i)
+                       vdpasim_kick_vq(vdpa, i);
+
+       }
+       spin_unlock(&vdpasim->lock);
+
+       return 0;
+}
+
 static size_t vdpasim_get_config_size(struct vdpa_device *vdpa)
 {
        struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
@@ -694,6 +713,7 @@ static const struct vdpa_config_ops vdpasim_config_ops = {
        .get_status             = vdpasim_get_status,
        .set_status             = vdpasim_set_status,
        .reset                  = vdpasim_reset,
+       .stop                   = vdpasim_stop,
        .get_config_size        = vdpasim_get_config_size,
        .get_config             = vdpasim_get_config,
        .set_config             = vdpasim_set_config,
@@ -726,6 +746,7 @@ static const struct vdpa_config_ops vdpasim_batch_config_ops = {
        .get_status             = vdpasim_get_status,
        .set_status             = vdpasim_set_status,
        .reset                  = vdpasim_reset,
+       .stop                   = vdpasim_stop,
        .get_config_size        = vdpasim_get_config_size,
        .get_config             = vdpasim_get_config,
        .set_config             = vdpasim_set_config,
drivers/vdpa/vdpa_sim/vdpa_sim.h +1 -0
index 622782e92239..061986f30911 100644
@@ -66,6 +66,7 @@ struct vdpasim {
        u32 generation;
        u64 features;
        u32 groups;
+       bool running;
        /* spinlock to synchronize iommu table */
        spinlock_t iommu_lock;
 };
drivers/vdpa/vdpa_sim/vdpa_sim_blk.c +3 -0
index 42d401d43911..bcdb1982c378 100644
@@ -204,6 +204,9 @@ static void vdpasim_blk_work(struct work_struct *work)
        if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK))
                goto out;
 
+       if (!vdpasim->running)
+               goto out;
+
        for (i = 0; i < VDPASIM_BLK_VQ_NUM; i++) {
                struct vdpasim_virtqueue *vq = &vdpasim->vqs[i];
 
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +0 -0
index 5125976a4df8..886449e88502 100644

drivers/vdpa/vdpa_sim/vdpa_sim_net.c static void vdpasim_net_work(struct work_struct *work) · line 159

Quoted 6 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +3 -0
@@ -154,6 +154,9 @@ static void vdpasim_net_work(struct work_struct *work)
 
        spin_lock(&vdpasim->lock);
 
+       if (!vdpasim->running)
+               goto out;
+

Do we need to check vdpasim->running in vdpasim_kick_vq()?

Thanks

Quoted 6 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +0 -1
        if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK))
                goto out;
 
--

2.27.0

author maintainer
#8 lore raw

Re: [PATCH v2 4/4] vdpa_sim: Implement stop vdpa op

Quoted 118 lines · Jason Wang
Quoted 116 lines · Eugenio Pérez

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration, since the device could modify them after userland gets them. There are individual ways to perform that action for some devices (VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any pending operations like in flight requests. It must also preserve all the necessary state (the virtqueue vring base plus the possible device specific states) that is required for restoring in the future. The device must not change its configuration after that point.

After the return of ioctl with stop == 0, the device can continue processing buffers as long as typical conditions are met (vq is enabled, DRIVER_OK status bit is enabled, etc).

In the future, we will provide features similar to VHOST_USER_GET_INFLIGHT_FD so the device can save pending operations.

---
 drivers/vdpa/vdpa_sim/vdpa_sim.c     | 21 +++++++++++++++++++++
 drivers/vdpa/vdpa_sim/vdpa_sim.h     |  1 +
 drivers/vdpa/vdpa_sim/vdpa_sim_blk.c |  3 +++
 drivers/vdpa/vdpa_sim/vdpa_sim_net.c |  3 +++
 4 files changed, 28 insertions(+)
drivers/vdpa/vdpa_sim/vdpa_sim.c +21 -0
index 50d721072beb..0515cf314bed 100644
@@ -107,6 +107,7 @@ static void vdpasim_do_reset(struct vdpasim *vdpasim)
        for (i = 0; i < vdpasim->dev_attr.nas; i++)
                vhost_iotlb_reset(&vdpasim->iommu[i]);
 
+       vdpasim->running = true;
        spin_unlock(&vdpasim->iommu_lock);
 
        vdpasim->features = 0;
@@ -505,6 +506,24 @@ static int vdpasim_reset(struct vdpa_device *vdpa)
        return 0;
 }
 
+static int vdpasim_stop(struct vdpa_device *vdpa, bool stop)
+{
+       struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
+       int i;
+
+       spin_lock(&vdpasim->lock);
+       vdpasim->running = !stop;
+       if (vdpasim->running) {
+               /* Check for missed buffers */
+               for (i = 0; i < vdpasim->dev_attr.nvqs; ++i)
+                       vdpasim_kick_vq(vdpa, i);
+
+       }
+       spin_unlock(&vdpasim->lock);
+
+       return 0;
+}
+
 static size_t vdpasim_get_config_size(struct vdpa_device *vdpa)
 {
        struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
@@ -694,6 +713,7 @@ static const struct vdpa_config_ops vdpasim_config_ops = {
        .get_status             = vdpasim_get_status,
        .set_status             = vdpasim_set_status,
        .reset                  = vdpasim_reset,
+       .stop                   = vdpasim_stop,
        .get_config_size        = vdpasim_get_config_size,
        .get_config             = vdpasim_get_config,
        .set_config             = vdpasim_set_config,
@@ -726,6 +746,7 @@ static const struct vdpa_config_ops vdpasim_batch_config_ops = {
        .get_status             = vdpasim_get_status,
        .set_status             = vdpasim_set_status,
        .reset                  = vdpasim_reset,
+       .stop                   = vdpasim_stop,
        .get_config_size        = vdpasim_get_config_size,
        .get_config             = vdpasim_get_config,
        .set_config             = vdpasim_set_config,
drivers/vdpa/vdpa_sim/vdpa_sim.h +1 -0
index 622782e92239..061986f30911 100644
@@ -66,6 +66,7 @@ struct vdpasim {
        u32 generation;
        u64 features;
        u32 groups;
+       bool running;
        /* spinlock to synchronize iommu table */
        spinlock_t iommu_lock;
 };
drivers/vdpa/vdpa_sim/vdpa_sim_blk.c +3 -0
index 42d401d43911..bcdb1982c378 100644
@@ -204,6 +204,9 @@ static void vdpasim_blk_work(struct work_struct *work)
        if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK))
                goto out;
 
+       if (!vdpasim->running)
+               goto out;
+
        for (i = 0; i < VDPASIM_BLK_VQ_NUM; i++) {
                struct vdpasim_virtqueue *vq = &vdpasim->vqs[i];
 
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +0 -0
index 5125976a4df8..886449e88502 100644
@@ -154,6 +154,9 @@ static void vdpasim_net_work(struct work_struct *work)
 
Quoted 8 lines

drivers/vdpa/vdpa_sim/vdpa_sim_net.c static void vdpasim_net_work(struct work_struct *work) · line 159

Quoted 5 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +3 -0
        spin_lock(&vdpasim->lock);
 
+       if (!vdpasim->running)
+               goto out;
+

Do we need to check vdpasim->running in vdpasim_kick_vq()?

I'd say that not really: The important part is that we don't process more buffers, and that is more accurate here. To check it here will always avoid it although we queue work.

Maybe we can see it as an optimization: either to check before queuing the work as you propose or simply stop polling kick file descriptors?

Thanks!

Quoted 9 lines

Thanks

Quoted 6 lines
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +0 -1
        if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK))
                goto out;
 
--

2.27.0

#9 lore raw

Re: [PATCH v2 4/4] vdpa_sim: Implement stop vdpa op

In reply to Eugenio Pérez
Quoted 100 lines · Eugenio Pérez

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration, since the device could modify them after userland gets them. There are individual ways to perform that action for some devices (VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any pending operations like in flight requests. It must also preserve all the necessary state (the virtqueue vring base plus the possible device specific states) that is required for restoring in the future. The device must not change its configuration after that point.

After the return of ioctl with stop == 0, the device can continue processing buffers as long as typical conditions are met (vq is enabled, DRIVER_OK status bit is enabled, etc).

In the future, we will provide features similar to VHOST_USER_GET_INFLIGHT_FD so the device can save pending operations.

---

drivers/vdpa/vdpa_sim/vdpa_sim.c     | 21 +++++++++++++++++++++
drivers/vdpa/vdpa_sim/vdpa_sim.h     |  1 +
drivers/vdpa/vdpa_sim/vdpa_sim_blk.c |  3 +++
drivers/vdpa/vdpa_sim/vdpa_sim_net.c |  3 +++
4 files changed, 28 insertions(+)
drivers/vdpa/vdpa_sim/vdpa_sim.c +21 -0
index 50d721072beb..0515cf314bed 100644
@@ -107,6 +107,7 @@ static void vdpasim_do_reset(struct vdpasim *vdpasim)
 	for (i = 0; i < vdpasim->dev_attr.nas; i++)
 		vhost_iotlb_reset(&vdpasim->iommu[i]);
 
+	vdpasim->running = true;
 	spin_unlock(&vdpasim->iommu_lock);
 
 	vdpasim->features = 0;
@@ -505,6 +506,24 @@ static int vdpasim_reset(struct vdpa_device *vdpa)
 	return 0;
 }
 
+static int vdpasim_stop(struct vdpa_device *vdpa, bool stop)
+{
+	struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
+	int i;
+
+	spin_lock(&vdpasim->lock);
+	vdpasim->running = !stop;
+	if (vdpasim->running) {
+		/* Check for missed buffers */
+		for (i = 0; i < vdpasim->dev_attr.nvqs; ++i)
+			vdpasim_kick_vq(vdpa, i);
+
+	}
+	spin_unlock(&vdpasim->lock);
+
+	return 0;
+}
+
 static size_t vdpasim_get_config_size(struct vdpa_device *vdpa)
 {
 	struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
@@ -694,6 +713,7 @@ static const struct vdpa_config_ops vdpasim_config_ops = {
 	.get_status             = vdpasim_get_status,
 	.set_status             = vdpasim_set_status,
 	.reset			= vdpasim_reset,
+	.stop			= vdpasim_stop,
 	.get_config_size        = vdpasim_get_config_size,
 	.get_config             = vdpasim_get_config,
 	.set_config             = vdpasim_set_config,
@@ -726,6 +746,7 @@ static const struct vdpa_config_ops vdpasim_batch_config_ops = {
 	.get_status             = vdpasim_get_status,
 	.set_status             = vdpasim_set_status,
 	.reset			= vdpasim_reset,
+	.stop			= vdpasim_stop,
 	.get_config_size        = vdpasim_get_config_size,
 	.get_config             = vdpasim_get_config,
 	.set_config             = vdpasim_set_config,
drivers/vdpa/vdpa_sim/vdpa_sim.h +1 -0
index 622782e92239..061986f30911 100644
@@ -66,6 +66,7 @@ struct vdpasim {
 	u32 generation;
 	u64 features;
 	u32 groups;
+	bool running;
 	/* spinlock to synchronize iommu table */
 	spinlock_t iommu_lock;
 };
drivers/vdpa/vdpa_sim/vdpa_sim_blk.c +0 -0
index 42d401d43911..bcdb1982c378 100644

drivers/vdpa/vdpa_sim/vdpa_sim_blk.c static void vdpasim_blk_work(struct work_struct *work) · line 209

Quoted 6 lines
drivers/vdpa/vdpa_sim/vdpa_sim_blk.c +3 -0
@@ -204,6 +204,9 @@ static void vdpasim_blk_work(struct work_struct *work)
 	if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK))
 		goto out;
 
+	if (!vdpasim->running)
+		goto out;
+

Not related to this series, but I think in vdpa_sim_blk.c we should implement something similar to what we already do in vdpa_sim_net.c and re-schedule the work after X requests handled, otherwise we risk never stopping if there are always requests to handle.

Also for supporting multiple queues, that could be a problem, but for now we only support one, so there should be no problem.

I have other patches to send for vdpa_sim_blk.c, so if you want I can do that in my series.

Thanks,
Stefano

Quoted 20 lines
drivers/vdpa/vdpa_sim/vdpa_sim_blk.c +0 -0
 	for (i = 0; i < VDPASIM_BLK_VQ_NUM; i++) {
 		struct vdpasim_virtqueue *vq = &vdpasim->vqs[i];
 
drivers/vdpa/vdpa_sim/vdpa_sim_net.c +3 -0
index 5125976a4df8..886449e88502 100644
@@ -154,6 +154,9 @@ static void vdpasim_net_work(struct work_struct *work)
 
 	spin_lock(&vdpasim->lock);
 
+	if (!vdpasim->running)
+		goto out;
+
 	if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK))
 		goto out;
 
Signature · 1 line
2.27.0
#10 lore raw
In reply to Eugenio Pérez
Quoted 10 lines · Eugenio Pérez

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration, since the device could modify them after userland gets them. There are individual ways to perform that action for some devices (VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no way to perform it for any vhost device (and, in particular, vhost-vdpa).

Quoted 5 lines

After the return of ioctl with stop != 0, the device MUST finish any pending operations like in flight requests. It must also preserve all the necessary state (the virtqueue vring base plus the possible device specific states) that is required for restoring in the future. The device must not change its configuration after that point.

I'd suggest documenting this in the code maybe around ops->stop()?

Thanks

Quoted 35 lines

After the return of ioctl with stop == 0, the device can continue processing buffers as long as typical conditions are met (vq is enabled, DRIVER_OK status bit is enabled, etc).

In the future, we will provide features similar to VHOST_USER_GET_INFLIGHT_FD so the device can save pending operations.

Comments are welcome.

v2:
* Replace raw _F_STOP with BIT_ULL(_F_STOP).
* Fix obtaining of stop ioctl arg (it was not obtained but written).
* Add stop to vdpa_sim_blk.

Eugenio Pérez (4):
vdpa: Add stop operation
vhost-vdpa: introduce STOP backend feature bit
vhost-vdpa: uAPI to stop the device
vdpa_sim: Implement stop vdpa op

 drivers/vdpa/vdpa_sim/vdpa_sim.c     | 21 +++++++++++++++++
 drivers/vdpa/vdpa_sim/vdpa_sim.h     |  1 +
 drivers/vdpa/vdpa_sim/vdpa_sim_blk.c |  3 +++
 drivers/vdpa/vdpa_sim/vdpa_sim_net.c |  3 +++
 drivers/vhost/vdpa.c                 | 34 +++++++++++++++++++++++++++-
 include/linux/vdpa.h                 |  6 +++++
 include/uapi/linux/vhost.h           |  3 +++
 include/uapi/linux/vhost_types.h     |  2 ++
 8 files changed, 72 insertions(+), 1 deletion(-)

--
2.27.0

author maintainer
#11 lore raw
Quoted 12 lines · Jason Wang
Quoted 10 lines · Eugenio Pérez

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration, since the device could modify them after userland gets them. There are individual ways to perform that action for some devices (VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no way to perform it for any vhost device (and, in particular, vhost-vdpa).

Quoted 8 lines
Quoted 5 lines

After the return of ioctl with stop != 0, the device MUST finish any pending operations like in flight requests. It must also preserve all the necessary state (the virtqueue vring base plus the possible device specific states) that is required for restoring in the future. The device must not change its configuration after that point.

I'd suggest documenting this in the code maybe around ops->stop()?

I agree it'd be better to put in the source code, but both vdpa_config_ops and ops->stop don't have a lot of space for docs.

Would it work to document at drivers/vdpa/vdpa.c:vhost_vdpa_stop() and redirect config ops like "for more info, see vhost_vdpa_stop"?

Thanks!

Quoted 38 lines

Thanks

Quoted 35 lines

After the return of ioctl with stop == 0, the device can continue processing buffers as long as typical conditions are met (vq is enabled, DRIVER_OK status bit is enabled, etc).

In the future, we will provide features similar to VHOST_USER_GET_INFLIGHT_FD so the device can save pending operations.

Comments are welcome.

v2:
* Replace raw _F_STOP with BIT_ULL(_F_STOP).
* Fix obtaining of stop ioctl arg (it was not obtained but written).
* Add stop to vdpa_sim_blk.

Eugenio Pérez (4):
vdpa: Add stop operation
vhost-vdpa: introduce STOP backend feature bit
vhost-vdpa: uAPI to stop the device
vdpa_sim: Implement stop vdpa op

 drivers/vdpa/vdpa_sim/vdpa_sim.c     | 21 +++++++++++++++++
 drivers/vdpa/vdpa_sim/vdpa_sim.h     |  1 +
 drivers/vdpa/vdpa_sim/vdpa_sim_blk.c |  3 +++
 drivers/vdpa/vdpa_sim/vdpa_sim_net.c |  3 +++
 drivers/vhost/vdpa.c                 | 34 +++++++++++++++++++++++++++-
 include/linux/vdpa.h                 |  6 +++++
 include/uapi/linux/vhost.h           |  3 +++
 include/uapi/linux/vhost_types.h     |  2 ++
 8 files changed, 72 insertions(+), 1 deletion(-)

--
2.27.0