Skip to contentSkip to conversation

vdpa: Add logging operatins

PATCH In review Jiang Dongxu 3 participants 6 messages Aug 26, 2023 – Nov 1, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover vdpa: Add logging operatins

From: jiangdongxu <[email protected]>

Currently, the vdpa device supports suspend and resume operations. To support vdpa device live migration, we need to support logging operations and device state save/load opertions.

These series introduces some new operations for vdpa devices. They allow vdpa to enable logging while vm start live migration.

And I will submit another patches about saving and loading vdpa device state later.

jiangdongxu (2):
vdpa: add log operations
vhost-vdpa: add uAPI for logging

 drivers/vhost/vdpa.c       | 49 ++++++++++++++++++++++++++++++++++++++
 include/linux/vdpa.h       | 14 +++++++++++
 include/uapi/linux/vhost.h |  4 ++++
 3 files changed, 67 insertions(+)
Signature · 1 line
2.27.0
#1 lore raw

PATCH 1/2 vdpa: add log operations

From: jiangdongxu <[email protected]>

Several new interfaces are introduced to allow vdpa device logging guest memory during live migration and return to the VMM.

The set_log_base interface is used to set the base address for buffer storing bitmaps.

The set_log_size interface is used to set the size of buffer used for storing bitmaps.

The log_sync interface is used to copy the bitmaps from kernel space to user space of VMM.

These operations are optional. If they are not implemented, these operations will return EOPNOTSUPP.

---
 include/linux/vdpa.h | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
include/linux/vdpa.h +14 -0
index db1b0eaef4eb..985eed4f05b3 100644
@@ -327,6 +327,15 @@ struct vdpa_map_file {
  * @unbind_mm:			Unbind the device from the address space
  *				bound using the bind_mm callback. (optional)
  *				@vdev: vdpa device
+ * @set_log_base		Set base address for logging. (optional)
+ *				@vdev: vdpa device
+ *				@base: base address
+ * @set_log_size		Set buffer size for logging. (optional)
+ *				@vdev: vdpa device
+ *				@size: logging buffer size
+ * @log_sync			Synchronize logging buffer from kernel space to
+ *				user space. (optional)
+ *				@vdev: vdpa device
  * @free:			Free resources that belongs to vDPA (optional)
  *				@vdev: vdpa device
  */
@@ -396,6 +405,11 @@ struct vdpa_config_ops {
 	int (*bind_mm)(struct vdpa_device *vdev, struct mm_struct *mm);
 	void (*unbind_mm)(struct vdpa_device *vdev);
 
+	/* Log ops */
+	int (*set_log_base)(struct vdpa_device *vdev, uint64_t base);
+	int (*set_log_size)(struct vdpa_device *vdev, uint64_t size);
+	int (*log_sync)(struct vdpa_device *vdev);
+
 	/* Free device resources */
 	void (*free)(struct vdpa_device *vdev);
 };
Signature · 1 line
2.27.0
#2 lore raw

PATCH 2/2 vhost-vdpa: add uAPI for logging

In reply to Jiang Dongxu

From: jiangdongxu <[email protected]>

These new ioctl add support for setting bitmaps config, like base address and buffer size from userspace.

When setup migration, VMM will call VHOST_SET_LOG_BASE and VHOST_SET_LOG_SIZE to set address and size of buffer used for storing bitmaps.

Then VMM start live migration, VMM will enable logging vhost device by set feature VHOST_F_LOG_ALL.

And during live migration iterate, VMM get dirty page info from vhost device by calling VHOST_LOG_SYNC.

---
 drivers/vhost/vdpa.c       | 49 ++++++++++++++++++++++++++++++++++++++
 include/uapi/linux/vhost.h |  4 ++++
 2 files changed, 53 insertions(+)
drivers/vhost/vdpa.c +49 -0
index b43e8680eee8..75e17d9b136b 100644
@@ -556,6 +556,47 @@ static long vhost_vdpa_resume(struct vhost_vdpa *v)
 	return ops->resume(vdpa);
 }
 
+static long vhost_vdpa_set_log_base(struct vhost_vdpa *v, u64 __user *argp)
+{
+	struct vdpa_device *vdpa = v->vdpa;
+	const struct vdpa_config_ops *ops = vdpa->config;
+	u64 log;
+
+	if (!ops->set_log_base)
+		return -EOPNOTSUPP;
+
+	if (copy_from_user(&log, argp, sizeof(uint64_t)))
+		return -EFAULT;
+
+	return ops->set_log_base(vdpa, log);
+}
+
+static long vhost_vdpa_set_log_size(struct vhost_vdpa *v, u64 __user *sizep)
+{
+	struct vdpa_device *vdpa = v->vdpa;
+	const struct vdpa_config_ops *ops = vdpa->config;
+	u64 log_size;
+
+	if (!ops->set_log_size)
+		return -EOPNOTSUPP;
+
+	if (copy_from_user(&log_size, sizep, sizeof(log_size)))
+		return -EFAULT;
+
+	return ops->set_log_size(vdpa, log_size);
+}
+
+static long vhost_vdpa_log_sync(struct vhost_vdpa *v)
+{
+	struct vdpa_device *vdpa = v->vdpa;
+	const struct vdpa_config_ops *ops = vdpa->config;
+
+	if (!ops->log_sync)
+		return -EOPNOTSUPP;
+
+	return ops->log_sync(vdpa);
+}
+
 static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
 				   void __user *argp)
 {
@@ -729,6 +770,14 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
 			r = -EFAULT;
 		break;
 	case VHOST_SET_LOG_BASE:
+		r = vhost_vdpa_set_log_base(v, argp);
+		break;
+	case VHOST_SET_LOG_SIZE:
+		r = vhost_vdpa_set_log_size(v, argp);
+		break;
+	case VHOST_LOG_SYNC:
+		r = vhost_vdpa_log_sync(v);
+		break;
 	case VHOST_SET_LOG_FD:
 		r = -ENOIOCTLCMD;
 		break;
include/uapi/linux/vhost.h +4 -0
index f5c48b61ab62..ce9d187432d1 100644
@@ -43,6 +43,10 @@
  * The bit is set using an atomic 32 bit operation. */
 /* Set base address for logging. */
 #define VHOST_SET_LOG_BASE _IOW(VHOST_VIRTIO, 0x04, __u64)
+/* Set buffer size for logging */
+#define VHOST_SET_LOG_SIZE _IOW(VHOST_VIRTIO, 0x05, __u64)
+/* Synchronize logging buffer from kernel space to user space */
+#define VHOST_LOG_SYNC _IO(VHOST_VIRTIO, 0x06)
 /* Specify an eventfd file descriptor to signal on log write. */
 #define VHOST_SET_LOG_FD _IOW(VHOST_VIRTIO, 0x07, int)
 /* By default, a device gets one vhost_worker that its virtqueues share. This
Signature · 1 line
2.27.0
#3 lore raw
In reply to Jiang Dongxu

Adding Eugenio and Si Wei.

Quoted 7 lines · Jiang Dongxu

From: jiangdongxu <[email protected]>

Currently, the vdpa device supports suspend and resume operations. To support vdpa device live migration, we need to support logging operations and device state save/load opertions.

Quoted 5 lines

These series introduces some new operations for vdpa devices. They allow vdpa to enable logging while vm start live migration.

And I will submit another patches about saving and loading vdpa device state later.

Thanks for working on this, I have several questions:

1) Is there an example implementation of the logging in the drivers? We need a real user in order to merge this.
2) Is the logging based on IOVA or VA? How the DMA isolation is being done with this? Do we need a SET_LOGGING_ASID uAPI for this? (We had some discussion on this in the past).

Thanks

Quoted 5 lines

jiangdongxu (2):
vdpa: add log operations
vhost-vdpa: add uAPI for logging

Quoted 8 lines
 drivers/vhost/vdpa.c       | 49 ++++++++++++++++++++++++++++++++++++++
 include/linux/vdpa.h       | 14 +++++++++++
 include/uapi/linux/vhost.h |  4 ++++
 3 files changed, 67 insertions(+)

--
2.27.0

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#4 lore raw
Quoted 18 lines · Jason Wang

Adding Eugenio and Si Wei.

Quoted 7 lines · Jiang Dongxu

From: jiangdongxu <[email protected]>

Currently, the vdpa device supports suspend and resume operations. To support vdpa device live migration, we need to support logging operations and device state save/load opertions.

Quoted 5 lines

These series introduces some new operations for vdpa devices. They allow vdpa to enable logging while vm start live migration.

And I will submit another patches about saving and loading vdpa device state later.

Thanks for working on this, I have several questions:

Quoted 7 lines

1) Is there an example implementation of the logging in the drivers? We need a real user in order to merge this.
2) Is the logging based on IOVA or VA? How the DMA isolation is being done with this? Do we need a SET_LOGGING_ASID uAPI for this? (We had some discussion on this in the past).

Thanks

Seems to be stuck. Dropped this for now.

Quoted 5 lines
Quoted 5 lines

jiangdongxu (2):
vdpa: add log operations
vhost-vdpa: add uAPI for logging

Quoted 8 lines
Quoted 8 lines
 drivers/vhost/vdpa.c       | 49 ++++++++++++++++++++++++++++++++++++++
 include/linux/vdpa.h       | 14 +++++++++++
 include/uapi/linux/vhost.h |  4 ++++
 3 files changed, 67 insertions(+)

--
2.27.0

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#5 lore raw
In reply to Jason Wang
Quoted 18 lines · Jason Wang

Adding Eugenio and Si Wei.

Quoted 7 lines · Jiang Dongxu

From: jiangdongxu <[email protected]>

Currently, the vdpa device supports suspend and resume operations. To support vdpa device live migration, we need to support logging operations and device state save/load opertions.

Quoted 5 lines

These series introduces some new operations for vdpa devices. They allow vdpa to enable logging while vm start live migration.

And I will submit another patches about saving and loading vdpa device state later.

Thanks for working on this, I have several questions:

Quoted 7 lines

1) Is there an example implementation of the logging in the drivers? We need a real user in order to merge this.
2) Is the logging based on IOVA or VA? How the DMA isolation is being done with this? Do we need a SET_LOGGING_ASID uAPI for this? (We had some discussion on this in the past).

Thanks

No answer so far so I'll drop this for now.

Quoted 5 lines
Quoted 5 lines

jiangdongxu (2):
vdpa: add log operations
vhost-vdpa: add uAPI for logging

Quoted 8 lines
Quoted 8 lines
 drivers/vhost/vdpa.c       | 49 ++++++++++++++++++++++++++++++++++++++
 include/linux/vdpa.h       | 14 +++++++++++
 include/uapi/linux/vhost.h |  4 ++++
 3 files changed, 67 insertions(+)

--
2.27.0

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization