Skip to contentSkip to conversation

Add migration PRE_COPY support for mlx5 driver

PATCH In review Yishai Hadas 4 participants 25 messages Nov 6, 2022 – Nov 14, 2022

Subsystems (7)
lore
Share & tools
lore

Yishai Hadas

author maintainer
#0 lore raw

cover Add migration PRE_COPY support for mlx5 driver

This series adds migration PRE_COPY uAPIs and their implementation as part of mlx5 driver.

The uAPIs follow some discussion that was done in the mailing list [1] in this area.

By the time the patches were sent, there was no driver implementation for the uAPIs, now we have it for mlx5 driver.

The optional PRE_COPY state opens the saving data transfer FD before reaching STOP_COPY and allows the device to dirty track the internal state changes with the general idea to reduce the volume of data transferred in the STOP_COPY stage.

While in PRE_COPY the device remains RUNNING, but the saving FD is open.

A new ioctl VFIO_MIG_GET_PRECOPY_INFO is provided to allow userspace to query the progress of the precopy operation in the driver with the idea it will judge to move to STOP_COPY at least once the initial data set is transferred, and possibly after the dirty size has shrunk appropriately.

User space can detect whether PRE_COPY is supported for a given device by checking the VFIO_MIGRATION_PRE_COPY flag once using the VFIO_DEVICE_FEATURE_MIGRATION ioctl.

Extra details exist as part of the specific uAPI patch from the series.

Finally, we come with mlx5 implementation based on its device specification for PRE_COPY.

To support PRE_COPY, mlx5 driver is transferring multiple states (images) of the device. e.g.: the source VF can save and transfer multiple states, and the target VF will load them by that order.

The device is saving three kinds of states:
1) Initial state - when the device moves to PRE_COPY state.
2) Middle state - during PRE_COPY phase via VFIO_MIG_GET_PRECOPY_INFO, can be multiple such states.
3) Final state - when the device moves to STOP_COPY state.

After moving to PRE_COPY state, the user is holding the saving FD and should use it for transferring the data from the source to the target while the VM is still running. From user point of view, it's a stream of data, however, from mlx5 driver point of view it includes multiple images/states. For that, it sets some headers with metadata on the source to be parsed on the target.

At some point, user may switch the device state from PRE_COPY to STOP_COPY, this will invoke saving of the final state.

As discussed earlier in the mailing list, the data that is returned as part of PRE_COPY is not required to have any bearing relative to the data size available during the STOP_COPY phase.

For this, we have the VFIO_DEVICE_FEATURE_MIG_DATA_SIZE option [2], it was sent also as part of this series as its initial patch.

In mlx5 driver we could gain with this series about 20-30 percent improvement in the downtime compared to the previous code when PRE_COPY wasn't supported.

The series includes some pre-patches to be ready for managing multiple images then it comes with the PRE_COPY implementation itself.

The matching qemu changes can be previewed here [3].
They come on top of the v2 migration protocol patches that were sent already to the mailing list.

Note:
As this series includes a net/mlx5 patch, we may need to send it as a pull request format to VFIO to avoid conflicts before acceptance.

[1] https://lore.kernel.org/kvm/[email protected]/
[2] https://patchwork.kernel.org/project/kvm/patch/[email protected]/
[3] https://github.com/avihai1122/qemu/commits/mig_v2_precopy

Yishai

Jason Gunthorpe (1):
vfio: Extend the device migration protocol with PRE_COPY

Shay Drory (9):
net/mlx5: Introduce ifc bits for pre_copy vfio/mlx5: Refactor total_length name and usage vfio/mlx5: Introduce device transitions of PRE_COPY vfio/mlx5: Introduce vfio precopy ioctl implementation vfio/mlx5: Manage read() of multiple state saves vfio/mlx5: Introduce SW headers for migration states vfio/mlx5: Introduce multiple loads
vfio/mlx5: Fallback to STOP_COPY upon specific PRE_COPY error vfio/mlx5: Enable MIGRATION_PRE_COPY flag

Yishai Hadas (3):
vfio: Add an option to get migration data size vfio/mlx5: Fix a typo in mlx5vf_cmd_load_vhca_state() vfio/mlx5: Enforce a single SAVE command at a time

 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    |   9 +
 drivers/vfio/pci/mlx5/cmd.c                   | 131 +++--
 drivers/vfio/pci/mlx5/cmd.h                   |  32 +-
 drivers/vfio/pci/mlx5/main.c                  | 495 ++++++++++++++++--
 drivers/vfio/pci/vfio_pci_core.c              |   3 +-
 drivers/vfio/vfio_main.c                      | 106 +++-
 include/linux/mlx5/mlx5_ifc.h                 |  14 +-
 include/linux/vfio.h                          |   5 +
 include/uapi/linux/vfio.h                     | 135 ++++-
 9 files changed, 843 insertions(+), 87 deletions(-)
Signature · 1 line
2.18.1
author maintainer
#1 lore raw

PATCH 1/13 vfio: Add an option to get migration data size

Add an option to get migration data size by introducing a new migration feature named VFIO_DEVICE_FEATURE_MIG_DATA_SIZE.

Upon VFIO_DEVICE_FEATURE_GET the estimated data length that will be required to complete STOP_COPY is returned.

This option may better enable user space to consider before moving to STOP_COPY whether it can meet the downtime SLA based on the returned data.

The patch also includes the implementation for mlx5 and hisi for this new option to make it feature complete for the existing drivers in this area.

---
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    |  9 ++++++
 drivers/vfio/pci/mlx5/main.c                  | 18 +++++++++++
 drivers/vfio/pci/vfio_pci_core.c              |  3 +-
 drivers/vfio/vfio_main.c                      | 32 +++++++++++++++++++
 include/linux/vfio.h                          |  5 +++
 include/uapi/linux/vfio.h                     | 13 ++++++++
 6 files changed, 79 insertions(+), 1 deletion(-)
drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c +9 -0
index 39eeca18a0f7..0c0c0c7f0521 100644
@@ -957,6 +957,14 @@ hisi_acc_vfio_pci_set_device_state(struct vfio_device *vdev,
 	return res;
 }
 
+static int
+hisi_acc_vfio_pci_get_data_size(struct vfio_device *vdev,
+				unsigned long *stop_copy_length)
+{
+	*stop_copy_length = sizeof(struct acc_vf_data);
+	return 0;
+}
+
 static int
 hisi_acc_vfio_pci_get_device_state(struct vfio_device *vdev,
 				   enum vfio_device_mig_state *curr_state)
@@ -1213,6 +1221,7 @@ static void hisi_acc_vfio_pci_close_device(struct vfio_device *core_vdev)
 static const struct vfio_migration_ops hisi_acc_vfio_pci_migrn_state_ops = {
 	.migration_set_state = hisi_acc_vfio_pci_set_device_state,
 	.migration_get_state = hisi_acc_vfio_pci_get_device_state,
+	.migration_get_data_size = hisi_acc_vfio_pci_get_data_size,
 };
 
 static int hisi_acc_vfio_pci_migrn_init_dev(struct vfio_device *core_vdev)
drivers/vfio/pci/mlx5/main.c +18 -0
index fd6ccb8454a2..4c7a39ffd247 100644
@@ -512,6 +512,23 @@ mlx5vf_pci_set_device_state(struct vfio_device *vdev,
 	return res;
 }
 
+static int mlx5vf_pci_get_data_size(struct vfio_device *vdev,
+				    unsigned long *stop_copy_length)
+{
+	struct mlx5vf_pci_core_device *mvdev = container_of(
+		vdev, struct mlx5vf_pci_core_device, core_device.vdev);
+	size_t state_size;
+	int ret;
+
+	mutex_lock(&mvdev->state_mutex);
+	ret = mlx5vf_cmd_query_vhca_migration_state(mvdev,
+						    &state_size);
+	if (!ret)
+		*stop_copy_length = state_size;
+	mlx5vf_state_mutex_unlock(mvdev);
+	return ret;
+}
+
 static int mlx5vf_pci_get_device_state(struct vfio_device *vdev,
 				       enum vfio_device_mig_state *curr_state)
 {
@@ -577,6 +594,7 @@ static void mlx5vf_pci_close_device(struct vfio_device *core_vdev)
 static const struct vfio_migration_ops mlx5vf_pci_mig_ops = {
 	.migration_set_state = mlx5vf_pci_set_device_state,
 	.migration_get_state = mlx5vf_pci_get_device_state,
+	.migration_get_data_size = mlx5vf_pci_get_data_size,
 };
 
 static const struct vfio_log_ops mlx5vf_pci_log_ops = {
drivers/vfio/pci/vfio_pci_core.c +2 -1
index badc9d828cac..4d97ca66ba6c 100644
@@ -2128,7 +2128,8 @@ int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev)
 
 	if (vdev->vdev.mig_ops) {
 		if (!(vdev->vdev.mig_ops->migration_get_state &&
-		      vdev->vdev.mig_ops->migration_set_state) ||
+		      vdev->vdev.mig_ops->migration_set_state &&
+		      vdev->vdev.mig_ops->migration_get_data_size) ||
 		    !(vdev->vdev.migration_flags & VFIO_MIGRATION_STOP_COPY))
 			return -EINVAL;
 	}
drivers/vfio/vfio_main.c +32 -0
index 2d168793d4e1..b118e7b1bc59 100644
@@ -1256,6 +1256,34 @@ vfio_ioctl_device_feature_mig_device_state(struct vfio_device *device,
 	return 0;
 }
 
+static int
+vfio_ioctl_device_feature_migration_data_size(struct vfio_device *device,
+					      u32 flags, void __user *arg,
+					      size_t argsz)
+{
+	struct vfio_device_feature_mig_data_size data_size = {};
+	unsigned long stop_copy_length;
+	int ret;
+
+	if (!device->mig_ops)
+		return -ENOTTY;
+
+	ret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_GET,
+				 sizeof(data_size));
+	if (ret != 1)
+		return ret;
+
+	ret = device->mig_ops->migration_get_data_size(device, &stop_copy_length);
+	if (ret)
+		return ret;
+
+	data_size.stop_copy_length = stop_copy_length;
+	if (copy_to_user(arg, &data_size, sizeof(data_size)))
+		return -EFAULT;
+
+	return 0;
+}
+
 static int vfio_ioctl_device_feature_migration(struct vfio_device *device,
 					       u32 flags, void __user *arg,
 					       size_t argsz)
@@ -1483,6 +1511,10 @@ static int vfio_ioctl_device_feature(struct vfio_device *device,
 		return vfio_ioctl_device_feature_logging_report(
 			device, feature.flags, arg->data,
 			feature.argsz - minsz);
+	case VFIO_DEVICE_FEATURE_MIG_DATA_SIZE:
+		return vfio_ioctl_device_feature_migration_data_size(
+			device, feature.flags, arg->data,
+			feature.argsz - minsz);
 	default:
 		if (unlikely(!device->ops->device_feature))
 			return -EINVAL;
include/linux/vfio.h +5 -0
index e7cebeb875dd..5509451ae709 100644
@@ -107,6 +107,9 @@ struct vfio_device_ops {
  * @migration_get_state: Optional callback to get the migration state for
  *         devices that support migration. It's mandatory for
  *         VFIO_DEVICE_FEATURE_MIGRATION migration support.
+ * @migration_get_data_size: Optional callback to get the estimated data
+ *          length that will be required to complete stop copy. It's mandatory for
+ *          VFIO_DEVICE_FEATURE_MIGRATION migration support.
  */
 struct vfio_migration_ops {
 	struct file *(*migration_set_state)(
@@ -114,6 +117,8 @@ struct vfio_migration_ops {
 		enum vfio_device_mig_state new_state);
 	int (*migration_get_state)(struct vfio_device *device,
 				   enum vfio_device_mig_state *curr_state);
+	int (*migration_get_data_size)(struct vfio_device *device,
+				       unsigned long *stop_copy_length);
 };
 
 /**
include/uapi/linux/vfio.h +13 -0
index d7d8e0922376..3e45dbaf190e 100644
@@ -1128,6 +1128,19 @@ struct vfio_device_feature_dma_logging_report {
 
 #define VFIO_DEVICE_FEATURE_DMA_LOGGING_REPORT 8
 
+/*
+ * Upon VFIO_DEVICE_FEATURE_GET read back the estimated data length that will
+ * be required to complete stop copy.
+ *
+ * Note: Can be called on each device state.
+ */
+
+struct vfio_device_feature_mig_data_size {
+	__aligned_u64 stop_copy_length;
+};
+
+#define VFIO_DEVICE_FEATURE_MIG_DATA_SIZE 9
+
 /* -------- API for Type1 VFIO IOMMU -------- */
 
 /**
Signature · 1 line
2.18.1
maintainer Reviewed-by on v1 1/13
#2 lore raw

Re: [PATCH vfio 01/13] vfio: Add an option to get migration data size

On 2022/11/7 1:46, Yishai Hadas Wrote:

Quoted 44 lines

Add an option to get migration data size by introducing a new migration feature named VFIO_DEVICE_FEATURE_MIG_DATA_SIZE.

Upon VFIO_DEVICE_FEATURE_GET the estimated data length that will be required to complete STOP_COPY is returned.

This option may better enable user space to consider before moving to STOP_COPY whether it can meet the downtime SLA based on the returned data.

The patch also includes the implementation for mlx5 and hisi for this new option to make it feature complete for the existing drivers in this area.

---
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    |  9 ++++++
 drivers/vfio/pci/mlx5/main.c                  | 18 +++++++++++
 drivers/vfio/pci/vfio_pci_core.c              |  3 +-
 drivers/vfio/vfio_main.c                      | 32 +++++++++++++++++++
 include/linux/vfio.h                          |  5 +++
 include/uapi/linux/vfio.h                     | 13 ++++++++
 6 files changed, 79 insertions(+), 1 deletion(-)
drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c +8 -0
index 39eeca18a0f7..0c0c0c7f0521 100644
@@ -957,6 +957,14 @@ hisi_acc_vfio_pci_set_device_state(struct vfio_device *vdev,
 	return res;
 }
 
+static int
+hisi_acc_vfio_pci_get_data_size(struct vfio_device *vdev,
+				unsigned long *stop_copy_length)
+{
+	*stop_copy_length = sizeof(struct acc_vf_data);
+	return 0;
+}
+
 static int
 hisi_acc_vfio_pci_get_device_state(struct vfio_device *vdev,
 				   enum vfio_device_mig_state *curr_state)

drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c static void hisi_acc_vfio_pci_close_device(struct vfio_device *core_vdev) · line 1226

Quoted 6 lines
drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c +1 -0
@@ -1213,6 +1221,7 @@ static void hisi_acc_vfio_pci_close_device(struct vfio_device *core_vdev)
 static const struct vfio_migration_ops hisi_acc_vfio_pci_migrn_state_ops = {
 	.migration_set_state = hisi_acc_vfio_pci_set_device_state,
 	.migration_get_state = hisi_acc_vfio_pci_get_device_state,
+	.migration_get_data_size = hisi_acc_vfio_pci_get_data_size,
 };
 

Thanks.

Quoted 149 lines
drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c +0 -0
 static int hisi_acc_vfio_pci_migrn_init_dev(struct vfio_device *core_vdev)
drivers/vfio/pci/mlx5/main.c +18 -0
index fd6ccb8454a2..4c7a39ffd247 100644
@@ -512,6 +512,23 @@ mlx5vf_pci_set_device_state(struct vfio_device *vdev,
 	return res;
 }
 
+static int mlx5vf_pci_get_data_size(struct vfio_device *vdev,
+				    unsigned long *stop_copy_length)
+{
+	struct mlx5vf_pci_core_device *mvdev = container_of(
+		vdev, struct mlx5vf_pci_core_device, core_device.vdev);
+	size_t state_size;
+	int ret;
+
+	mutex_lock(&mvdev->state_mutex);
+	ret = mlx5vf_cmd_query_vhca_migration_state(mvdev,
+						    &state_size);
+	if (!ret)
+		*stop_copy_length = state_size;
+	mlx5vf_state_mutex_unlock(mvdev);
+	return ret;
+}
+
 static int mlx5vf_pci_get_device_state(struct vfio_device *vdev,
 				       enum vfio_device_mig_state *curr_state)
 {
@@ -577,6 +594,7 @@ static void mlx5vf_pci_close_device(struct vfio_device *core_vdev)
 static const struct vfio_migration_ops mlx5vf_pci_mig_ops = {
 	.migration_set_state = mlx5vf_pci_set_device_state,
 	.migration_get_state = mlx5vf_pci_get_device_state,
+	.migration_get_data_size = mlx5vf_pci_get_data_size,
 };
 
 static const struct vfio_log_ops mlx5vf_pci_log_ops = {
drivers/vfio/pci/vfio_pci_core.c +2 -1
index badc9d828cac..4d97ca66ba6c 100644
@@ -2128,7 +2128,8 @@ int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev)
 
 	if (vdev->vdev.mig_ops) {
 		if (!(vdev->vdev.mig_ops->migration_get_state &&
-		      vdev->vdev.mig_ops->migration_set_state) ||
+		      vdev->vdev.mig_ops->migration_set_state &&
+		      vdev->vdev.mig_ops->migration_get_data_size) ||
 		    !(vdev->vdev.migration_flags & VFIO_MIGRATION_STOP_COPY))
 			return -EINVAL;
 	}
drivers/vfio/vfio_main.c +32 -0
index 2d168793d4e1..b118e7b1bc59 100644
@@ -1256,6 +1256,34 @@ vfio_ioctl_device_feature_mig_device_state(struct vfio_device *device,
 	return 0;
 }
 
+static int
+vfio_ioctl_device_feature_migration_data_size(struct vfio_device *device,
+					      u32 flags, void __user *arg,
+					      size_t argsz)
+{
+	struct vfio_device_feature_mig_data_size data_size = {};
+	unsigned long stop_copy_length;
+	int ret;
+
+	if (!device->mig_ops)
+		return -ENOTTY;
+
+	ret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_GET,
+				 sizeof(data_size));
+	if (ret != 1)
+		return ret;
+
+	ret = device->mig_ops->migration_get_data_size(device, &stop_copy_length);
+	if (ret)
+		return ret;
+
+	data_size.stop_copy_length = stop_copy_length;
+	if (copy_to_user(arg, &data_size, sizeof(data_size)))
+		return -EFAULT;
+
+	return 0;
+}
+
 static int vfio_ioctl_device_feature_migration(struct vfio_device *device,
 					       u32 flags, void __user *arg,
 					       size_t argsz)
@@ -1483,6 +1511,10 @@ static int vfio_ioctl_device_feature(struct vfio_device *device,
 		return vfio_ioctl_device_feature_logging_report(
 			device, feature.flags, arg->data,
 			feature.argsz - minsz);
+	case VFIO_DEVICE_FEATURE_MIG_DATA_SIZE:
+		return vfio_ioctl_device_feature_migration_data_size(
+			device, feature.flags, arg->data,
+			feature.argsz - minsz);
 	default:
 		if (unlikely(!device->ops->device_feature))
 			return -EINVAL;
include/linux/vfio.h +5 -0
index e7cebeb875dd..5509451ae709 100644
@@ -107,6 +107,9 @@ struct vfio_device_ops {
  * @migration_get_state: Optional callback to get the migration state for
  *         devices that support migration. It's mandatory for
  *         VFIO_DEVICE_FEATURE_MIGRATION migration support.
+ * @migration_get_data_size: Optional callback to get the estimated data
+ *          length that will be required to complete stop copy. It's mandatory for
+ *          VFIO_DEVICE_FEATURE_MIGRATION migration support.
  */
 struct vfio_migration_ops {
 	struct file *(*migration_set_state)(
@@ -114,6 +117,8 @@ struct vfio_migration_ops {
 		enum vfio_device_mig_state new_state);
 	int (*migration_get_state)(struct vfio_device *device,
 				   enum vfio_device_mig_state *curr_state);
+	int (*migration_get_data_size)(struct vfio_device *device,
+				       unsigned long *stop_copy_length);
 };
 
 /**
include/uapi/linux/vfio.h +13 -0
index d7d8e0922376..3e45dbaf190e 100644
@@ -1128,6 +1128,19 @@ struct vfio_device_feature_dma_logging_report {
 
 #define VFIO_DEVICE_FEATURE_DMA_LOGGING_REPORT 8
 
+/*
+ * Upon VFIO_DEVICE_FEATURE_GET read back the estimated data length that will
+ * be required to complete stop copy.
+ *
+ * Note: Can be called on each device state.
+ */
+
+struct vfio_device_feature_mig_data_size {
+	__aligned_u64 stop_copy_length;
+};
+
+#define VFIO_DEVICE_FEATURE_MIG_DATA_SIZE 9
+
 /* -------- API for Type1 VFIO IOMMU -------- */
 
 /**
reviewer Reviewed-by on v1 1/13
#3 lore raw

Re: [PATCH vfio 01/13] vfio: Add an option to get migration data size

In reply to Yishai Hadas
Quoted 15 lines · Yishai Hadas

Add an option to get migration data size by introducing a new migration feature named VFIO_DEVICE_FEATURE_MIG_DATA_SIZE.

Upon VFIO_DEVICE_FEATURE_GET the estimated data length that will be required to complete STOP_COPY is returned.

This option may better enable user space to consider before moving to STOP_COPY whether it can meet the downtime SLA based on the returned data.

The patch also includes the implementation for mlx5 and hisi for this new option to make it feature complete for the existing drivers in this area.

Quoted 8 lines
---
 .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    |  9 ++++++
 drivers/vfio/pci/mlx5/main.c                  | 18 +++++++++++
 drivers/vfio/pci/vfio_pci_core.c              |  3 +-
 drivers/vfio/vfio_main.c                      | 32 +++++++++++++++++++
 include/linux/vfio.h                          |  5 +++
 include/uapi/linux/vfio.h                     | 13 ++++++++
 6 files changed, 79 insertions(+), 1 deletion(-)

Jason

author maintainer
#4 lore raw

Re: [PATCH vfio 01/13] vfio: Add an option to get migration data size

Quoted 24 lines · Jason Gunthorpe
Quoted 15 lines · Yishai Hadas

Add an option to get migration data size by introducing a new migration feature named VFIO_DEVICE_FEATURE_MIG_DATA_SIZE.

Upon VFIO_DEVICE_FEATURE_GET the estimated data length that will be required to complete STOP_COPY is returned.

This option may better enable user space to consider before moving to STOP_COPY whether it can meet the downtime SLA based on the returned data.

The patch also includes the implementation for mlx5 and hisi for this new option to make it feature complete for the existing drivers in this area.

Quoted 8 lines
---
  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    |  9 ++++++
  drivers/vfio/pci/mlx5/main.c                  | 18 +++++++++++
  drivers/vfio/pci/vfio_pci_core.c              |  3 +-
  drivers/vfio/vfio_main.c                      | 32 +++++++++++++++++++
  include/linux/vfio.h                          |  5 +++
  include/uapi/linux/vfio.h                     | 13 ++++++++
  6 files changed, 79 insertions(+), 1 deletion(-)
Quoted 3 lines

Jason

Alex,

Are we fine with taking the first 2 patches from this series ?

For this one we have reviewed-by from Jason and Longfang Liu, the next patch has also a reviewed-by Jason and is very simple.

Please let me know if you want me to send them separately outside of this pre_copy series and add the mentioned reviewed-by or that you can just collect them out from the list by yourself.

Thanks,

Yishai

#5 lore raw

Re: [PATCH vfio 01/13] vfio: Add an option to get migration data size

Quoted 31 lines · On Sun, 13 Nov 2022 18:58:50 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 24 lines · Jason Gunthorpe
Quoted 15 lines · Yishai Hadas

Add an option to get migration data size by introducing a new migration feature named VFIO_DEVICE_FEATURE_MIG_DATA_SIZE.

Upon VFIO_DEVICE_FEATURE_GET the estimated data length that will be required to complete STOP_COPY is returned.

This option may better enable user space to consider before moving to STOP_COPY whether it can meet the downtime SLA based on the returned data.

The patch also includes the implementation for mlx5 and hisi for this new option to make it feature complete for the existing drivers in this area.

Quoted 8 lines
---
  .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c    |  9 ++++++
  drivers/vfio/pci/mlx5/main.c                  | 18 +++++++++++
  drivers/vfio/pci/vfio_pci_core.c              |  3 +-
  drivers/vfio/vfio_main.c                      | 32 +++++++++++++++++++
  include/linux/vfio.h                          |  5 +++
  include/uapi/linux/vfio.h                     | 13 ++++++++
  6 files changed, 79 insertions(+), 1 deletion(-)  
Quoted 3 lines

Jason

Alex,

Quoted 8 lines

Are we fine with taking the first 2 patches from this series ?

For this one we have reviewed-by from Jason and Longfang Liu, the next patch has also a reviewed-by Jason and is very simple.

Please let me know if you want me to send them separately outside of this pre_copy series and add the mentioned reviewed-by or that you can just collect them out from the list by yourself.

Applied 1 & 2 to vfio next branch for v6.2. Thanks,

Alex

author maintainer
#6 lore raw

PATCH 2/13 vfio/mlx5: Fix a typo in mlx5vf_cmd_load_vhca_state()

In reply to Yishai Hadas

Fix a typo in mlx5vf_cmd_load_vhca_state() to use the 'load' memory layout.

As in/out sizes are equal for save and load commands there wasn't any functional issue.

Fixes f1d98f346ee3 ("vfio/mlx5: Expose migration commands over mlx5 device")
---
 drivers/vfio/pci/mlx5/cmd.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +2 -2
index c604b70437a5..0848bc905d3e 100644
@@ -378,8 +378,8 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev,
 			       struct mlx5_vf_migration_file *migf)
 {
 	struct mlx5_core_dev *mdev;
-	u32 out[MLX5_ST_SZ_DW(save_vhca_state_out)] = {};
-	u32 in[MLX5_ST_SZ_DW(save_vhca_state_in)] = {};
+	u32 out[MLX5_ST_SZ_DW(load_vhca_state_out)] = {};
+	u32 in[MLX5_ST_SZ_DW(load_vhca_state_in)] = {};
 	u32 pdn, mkey;
 	int err;
 
Signature · 1 line
2.18.1
reviewer Reviewed-by on v1 2/13
#7 lore raw

Re: [PATCH vfio 02/13] vfio/mlx5: Fix a typo in mlx5vf_cmd_load_vhca_state()

Quoted 11 lines · Yishai Hadas

Fix a typo in mlx5vf_cmd_load_vhca_state() to use the 'load' memory layout.

As in/out sizes are equal for save and load commands there wasn't any functional issue.

Fixes f1d98f346ee3 ("vfio/mlx5: Expose migration commands over mlx5 device")
---
 drivers/vfio/pci/mlx5/cmd.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Jason

author maintainer
#8 lore raw

PATCH 3/13 net/mlx5: Introduce ifc bits for pre_copy

In reply to Yishai Hadas

From: Shay Drory <[email protected]>

Introduce ifc related stuff to enable PRE_COPY of VF during migration.

---
 include/linux/mlx5/mlx5_ifc.h | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
include/linux/mlx5/mlx5_ifc.h +11 -3
index 06eab92b9fb3..9e35b657866c 100644
@@ -1831,7 +1831,12 @@ struct mlx5_ifc_cmd_hca_cap_2_bits {
 	u8	   max_reformat_remove_size[0x8];
 	u8	   max_reformat_remove_offset[0x8];
 
-	u8	   reserved_at_c0[0x160];
+	u8	   reserved_at_c0[0x8];
+	u8	   migration_multi_load[0x1];
+	u8	   migration_tracking_state[0x1];
+	u8	   reserved_at_ca[0x16];
+
+	u8         reserved_at_e0[0x140];
 
 	u8	   reserved_at_220[0x1];
 	u8	   sw_vhca_id_valid[0x1];
@@ -11744,7 +11749,8 @@ struct mlx5_ifc_query_vhca_migration_state_in_bits {
 	u8         reserved_at_20[0x10];
 	u8         op_mod[0x10];
 
-	u8         reserved_at_40[0x10];
+	u8         incremental[0x1];
+	u8         reserved_at_41[0xf];
 	u8         vhca_id[0x10];
 
 	u8         reserved_at_60[0x20];
@@ -11770,7 +11776,9 @@ struct mlx5_ifc_save_vhca_state_in_bits {
 	u8         reserved_at_20[0x10];
 	u8         op_mod[0x10];
 
-	u8         reserved_at_40[0x10];
+	u8         incremental[0x1];
+	u8         set_track[0x1];
+	u8         reserved_at_42[0xe];
 	u8         vhca_id[0x10];
 
 	u8         reserved_at_60[0x20];
Signature · 1 line
2.18.1
author maintainer
#9 lore raw

PATCH 4/13 vfio: Extend the device migration protocol with PRE_COPY

In reply to Yishai Hadas

From: Jason Gunthorpe <[email protected]>

The optional PRE_COPY states open the saving data transfer FD before reaching STOP_COPY and allows the device to dirty track internal state changes with the general idea to reduce the volume of data transferred in the STOP_COPY stage.

While in PRE_COPY the device remains RUNNING, but the saving FD is open.

Only if the device also supports RUNNING_P2P can it support PRE_COPY_P2P, which halts P2P transfers while continuing the saving FD.

PRE_COPY, with P2P support, requires the driver to implement 7 new arcs and exists as an optional FSM branch between RUNNING and STOP_COPY:

    RUNNING -> PRE_COPY -> PRE_COPY_P2P -> STOP_COPY

A new ioctl VFIO_MIG_GET_PRECOPY_INFO is provided to allow userspace to query the progress of the precopy operation in the driver with the idea it will judge to move to STOP_COPY at least once the initial data set is transferred, and possibly after the dirty size has shrunk appropriately.

This ioctl is valid only in PRE_COPY states and kernel driver should return -EINVAL from any other migration state.

Compared to the v1 clarification, STOP_COPY -> PRE_COPY is blocked and to be defined in future.
We also split the pending_bytes report into the initial and sustaining values, e.g.: initial_bytes and dirty_bytes.
initial_bytes: Amount of initial mandatory precopy data.
dirty_bytes: device state changes relative to data previously retrieved. These fields are not required to have any bearing to STOP_COPY phase.

---
 drivers/vfio/vfio_main.c  |  74 ++++++++++++++++++++++-
 include/uapi/linux/vfio.h | 122 ++++++++++++++++++++++++++++++++++++--
 2 files changed, 190 insertions(+), 6 deletions(-)
drivers/vfio/vfio_main.c +72 -2
index b118e7b1bc59..c58a795510f9 100644
@@ -1056,7 +1056,7 @@ int vfio_mig_get_next_state(struct vfio_device *device,
 			    enum vfio_device_mig_state new_fsm,
 			    enum vfio_device_mig_state *next_fsm)
 {
-	enum { VFIO_DEVICE_NUM_STATES = VFIO_DEVICE_STATE_RUNNING_P2P + 1 };
+	enum { VFIO_DEVICE_NUM_STATES = VFIO_DEVICE_STATE_PRE_COPY_P2P + 1 };
 	/*
 	 * The coding in this table requires the driver to implement the
 	 * following FSM arcs:
@@ -1071,30 +1071,65 @@ int vfio_mig_get_next_state(struct vfio_device *device,
 	 *         RUNNING_P2P -> RUNNING
 	 *         RUNNING_P2P -> STOP
 	 *         STOP -> RUNNING_P2P
-	 * Without P2P the driver must implement:
+	 *
+	 * If precopy is supported then the driver must support these additional
+	 * FSM arcs:
+	 *         RUNNING -> PRE_COPY
+	 *         PRE_COPY -> RUNNING
+	 *         PRE_COPY -> STOP_COPY
+	 * However, if precopy and P2P are supported together then the driver
+	 * must support these additional arcs beyond the P2P arcs above:
+	 *         PRE_COPY -> RUNNING
+	 *         PRE_COPY -> PRE_COPY_P2P
+	 *         PRE_COPY_P2P -> PRE_COPY
+	 *         PRE_COPY_P2P -> RUNNING_P2P
+	 *         PRE_COPY_P2P -> STOP_COPY
+	 *         RUNNING -> PRE_COPY
+	 *         RUNNING_P2P -> PRE_COPY_P2P
+	 *
+	 * Without P2P and precopy the driver must implement:
 	 *         RUNNING -> STOP
 	 *         STOP -> RUNNING
 	 *
 	 * The coding will step through multiple states for some combination
 	 * transitions; if all optional features are supported, this means the
 	 * following ones:
+	 *         PRE_COPY -> PRE_COPY_P2P -> STOP_COPY
+	 *         PRE_COPY -> RUNNING -> RUNNING_P2P
+	 *         PRE_COPY -> RUNNING -> RUNNING_P2P -> STOP
+	 *         PRE_COPY -> RUNNING -> RUNNING_P2P -> STOP -> RESUMING
+	 *         PRE_COPY_P2P -> RUNNING_P2P -> RUNNING
+	 *         PRE_COPY_P2P -> RUNNING_P2P -> STOP
+	 *         PRE_COPY_P2P -> RUNNING_P2P -> STOP -> RESUMING
 	 *         RESUMING -> STOP -> RUNNING_P2P
+	 *         RESUMING -> STOP -> RUNNING_P2P -> PRE_COPY_P2P
 	 *         RESUMING -> STOP -> RUNNING_P2P -> RUNNING
+	 *         RESUMING -> STOP -> RUNNING_P2P -> RUNNING -> PRE_COPY
 	 *         RESUMING -> STOP -> STOP_COPY
+	 *         RUNNING -> RUNNING_P2P -> PRE_COPY_P2P
 	 *         RUNNING -> RUNNING_P2P -> STOP
 	 *         RUNNING -> RUNNING_P2P -> STOP -> RESUMING
 	 *         RUNNING -> RUNNING_P2P -> STOP -> STOP_COPY
+	 *         RUNNING_P2P -> RUNNING -> PRE_COPY
 	 *         RUNNING_P2P -> STOP -> RESUMING
 	 *         RUNNING_P2P -> STOP -> STOP_COPY
+	 *         STOP -> RUNNING_P2P -> PRE_COPY_P2P
 	 *         STOP -> RUNNING_P2P -> RUNNING
+	 *         STOP -> RUNNING_P2P -> RUNNING -> PRE_COPY
 	 *         STOP_COPY -> STOP -> RESUMING
 	 *         STOP_COPY -> STOP -> RUNNING_P2P
 	 *         STOP_COPY -> STOP -> RUNNING_P2P -> RUNNING
+	 *
+	 *  The following transitions are blocked:
+	 *         STOP_COPY -> PRE_COPY
+	 *         STOP_COPY -> PRE_COPY_P2P
 	 */
 	static const u8 vfio_from_fsm_table[VFIO_DEVICE_NUM_STATES][VFIO_DEVICE_NUM_STATES] = {
 		[VFIO_DEVICE_STATE_STOP] = {
 			[VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_STOP,
 			[VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_RUNNING_P2P,
+			[VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_RUNNING_P2P,
+			[VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_RUNNING_P2P,
 			[VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_STOP_COPY,
 			[VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_RESUMING,
 			[VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_RUNNING_P2P,
@@ -1103,14 +1138,38 @@ int vfio_mig_get_next_state(struct vfio_device *device,
 		[VFIO_DEVICE_STATE_RUNNING] = {
 			[VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_RUNNING_P2P,
 			[VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_RUNNING,
+			[VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_PRE_COPY,
+			[VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_RUNNING_P2P,
 			[VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_RUNNING_P2P,
 			[VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_RUNNING_P2P,
 			[VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_RUNNING_P2P,
 			[VFIO_DEVICE_STATE_ERROR] = VFIO_DEVICE_STATE_ERROR,
 		},
+		[VFIO_DEVICE_STATE_PRE_COPY] = {
+			[VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_RUNNING,
+			[VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_RUNNING,
+			[VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_PRE_COPY,
+			[VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_PRE_COPY_P2P,
+			[VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_PRE_COPY_P2P,
+			[VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_RUNNING,
+			[VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_RUNNING,
+			[VFIO_DEVICE_STATE_ERROR] = VFIO_DEVICE_STATE_ERROR,
+		},
+		[VFIO_DEVICE_STATE_PRE_COPY_P2P] = {
+			[VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_RUNNING_P2P,
+			[VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_RUNNING_P2P,
+			[VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_PRE_COPY,
+			[VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_PRE_COPY_P2P,
+			[VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_STOP_COPY,
+			[VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_RUNNING_P2P,
+			[VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_RUNNING_P2P,
+			[VFIO_DEVICE_STATE_ERROR] = VFIO_DEVICE_STATE_ERROR,
+		},
 		[VFIO_DEVICE_STATE_STOP_COPY] = {
 			[VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_STOP,
 			[VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_STOP,
+			[VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_ERROR,
+			[VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_ERROR,
 			[VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_STOP_COPY,
 			[VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_STOP,
 			[VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_STOP,
@@ -1119,6 +1178,8 @@ int vfio_mig_get_next_state(struct vfio_device *device,
 		[VFIO_DEVICE_STATE_RESUMING] = {
 			[VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_STOP,
 			[VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_STOP,
+			[VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_STOP,
+			[VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_STOP,
 			[VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_STOP,
 			[VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_RESUMING,
 			[VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_STOP,
@@ -1127,6 +1188,8 @@ int vfio_mig_get_next_state(struct vfio_device *device,
 		[VFIO_DEVICE_STATE_RUNNING_P2P] = {
 			[VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_STOP,
 			[VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_RUNNING,
+			[VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_RUNNING,
+			[VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_PRE_COPY_P2P,
 			[VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_STOP,
 			[VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_STOP,
 			[VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_RUNNING_P2P,
@@ -1135,6 +1198,8 @@ int vfio_mig_get_next_state(struct vfio_device *device,
 		[VFIO_DEVICE_STATE_ERROR] = {
 			[VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_ERROR,
 			[VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_ERROR,
+			[VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_ERROR,
+			[VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_ERROR,
 			[VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_ERROR,
 			[VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_ERROR,
 			[VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_ERROR,
@@ -1145,6 +1210,11 @@ int vfio_mig_get_next_state(struct vfio_device *device,
 	static const unsigned int state_flags_table[VFIO_DEVICE_NUM_STATES] = {
 		[VFIO_DEVICE_STATE_STOP] = VFIO_MIGRATION_STOP_COPY,
 		[VFIO_DEVICE_STATE_RUNNING] = VFIO_MIGRATION_STOP_COPY,
+		[VFIO_DEVICE_STATE_PRE_COPY] =
+			VFIO_MIGRATION_STOP_COPY | VFIO_MIGRATION_PRE_COPY,
+		[VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_MIGRATION_STOP_COPY |
+						   VFIO_MIGRATION_P2P |
+						   VFIO_MIGRATION_PRE_COPY,
 		[VFIO_DEVICE_STATE_STOP_COPY] = VFIO_MIGRATION_STOP_COPY,
 		[VFIO_DEVICE_STATE_RESUMING] = VFIO_MIGRATION_STOP_COPY,
 		[VFIO_DEVICE_STATE_RUNNING_P2P] =
include/uapi/linux/vfio.h +118 -4

191 lines not on this page · raw message

Signature · 1 line
2.18.1
author maintainer
#10 lore raw

PATCH 5/13 vfio/mlx5: Enforce a single SAVE command at a time

In reply to Yishai Hadas

Enforce a single SAVE command at a time.

As the SAVE command is an asynchronous one, we must enforce running only a single command at a time.

This will preserve ordering between multiple calls and protect from races on the migration file data structure.

This is a must for the next patches from the series where as part of PRE_COPY we may have multiple images to be saved and multiple SAVE commands may be issued from different flows.

---
 drivers/vfio/pci/mlx5/cmd.c  | 5 +++++
 drivers/vfio/pci/mlx5/cmd.h  | 2 ++
 drivers/vfio/pci/mlx5/main.c | 1 +
 3 files changed, 8 insertions(+)
drivers/vfio/pci/mlx5/cmd.c +5 -0
index 0848bc905d3e..b9ed2f1c8689 100644
@@ -281,6 +281,8 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work)
 	dma_unmap_sgtable(mdev->device, &migf->table.sgt, DMA_FROM_DEVICE, 0);
 	mlx5_core_dealloc_pd(mdev, async_data->pdn);
 	kvfree(async_data->out);
+	migf->save_cb_active = false;
+	wake_up(&migf->save_wait);
 	fput(migf->filp);
 }
 
@@ -321,6 +323,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,
 		return -ENOTCONN;
 
 	mdev = mvdev->mdev;
+	wait_event(migf->save_wait, !migf->save_cb_active);
 	err = mlx5_core_alloc_pd(mdev, &pdn);
 	if (err)
 		return err;
@@ -353,6 +356,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,
 	get_file(migf->filp);
 	async_data->mkey = mkey;
 	async_data->pdn = pdn;
+	migf->save_cb_active = true;
 	err = mlx5_cmd_exec_cb(&migf->async_ctx, in, sizeof(in),
 			       async_data->out,
 			       out_size, mlx5vf_save_callback,
@@ -371,6 +375,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,
 	dma_unmap_sgtable(mdev->device, &migf->table.sgt, DMA_FROM_DEVICE, 0);
 err_dma_map:
 	mlx5_core_dealloc_pd(mdev, pdn);
+	migf->save_cb_active = false;
 	return err;
 }
 
drivers/vfio/pci/mlx5/cmd.h +2 -0
index 921d5720a1e5..b1c5dd2ff144 100644
@@ -26,6 +26,7 @@ struct mlx5_vf_migration_file {
 	struct mutex lock;
 	u8 disabled:1;
 	u8 is_err:1;
+	u8 save_cb_active:1;
 
 	struct sg_append_table table;
 	size_t total_length;
@@ -37,6 +38,7 @@ struct mlx5_vf_migration_file {
 	unsigned long last_offset;
 	struct mlx5vf_pci_core_device *mvdev;
 	wait_queue_head_t poll_wait;
+	wait_queue_head_t save_wait;
 	struct mlx5_async_ctx async_ctx;
 	struct mlx5vf_async_data async_data;
 };
drivers/vfio/pci/mlx5/main.c +1 -0
index 4c7a39ffd247..5da278f3c31c 100644
@@ -245,6 +245,7 @@ mlx5vf_pci_save_device_data(struct mlx5vf_pci_core_device *mvdev)
 	stream_open(migf->filp->f_inode, migf->filp);
 	mutex_init(&migf->lock);
 	init_waitqueue_head(&migf->poll_wait);
+	init_waitqueue_head(&migf->save_wait);
 	mlx5_cmd_init_async_ctx(mvdev->mdev, &migf->async_ctx);
 	INIT_WORK(&migf->async_data.work, mlx5vf_mig_file_cleanup_cb);
 	ret = mlx5vf_cmd_query_vhca_migration_state(mvdev,
Signature · 1 line
2.18.1
#11 lore raw

Re: [PATCH vfio 05/13] vfio/mlx5: Enforce a single SAVE command at a time

Quoted 12 lines · Yishai Hadas

Enforce a single SAVE command at a time.

As the SAVE command is an asynchronous one, we must enforce running only a single command at a time.

This will preserve ordering between multiple calls and protect from races on the migration file data structure.

This is a must for the next patches from the series where as part of PRE_COPY we may have multiple images to be saved and multiple SAVE commands may be issued from different flows.

Quoted 6 lines
---
 drivers/vfio/pci/mlx5/cmd.c  | 5 +++++
 drivers/vfio/pci/mlx5/cmd.h  | 2 ++
 drivers/vfio/pci/mlx5/main.c | 1 +
 3 files changed, 8 insertions(+)

This should just use a 'counting completion' instead of open coding one.

drivers/vfio/pci/mlx5/cmd.c void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work) · line 285

Quoted 10 lines
drivers/vfio/pci/mlx5/cmd.c +2 -0
index 0848bc905d3e..b9ed2f1c8689 100644
@@ -281,6 +281,8 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work)
 	dma_unmap_sgtable(mdev->device, &migf->table.sgt, DMA_FROM_DEVICE, 0);
 	mlx5_core_dealloc_pd(mdev, async_data->pdn);
 	kvfree(async_data->out);
+	migf->save_cb_active = false;
+	wake_up(&migf->save_wait);

complete()

drivers/vfio/pci/mlx5/cmd.c int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, · line 326

Quoted 8 lines
drivers/vfio/pci/mlx5/cmd.c +1 -0
 	fput(migf->filp);
 }
 
@@ -321,6 +323,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,
 		return -ENOTCONN;
 
 	mdev = mvdev->mdev;
+	wait_event(migf->save_wait, !migf->save_cb_active);

wait_for_completion_interruptible()

Quoted 12 lines
drivers/vfio/pci/mlx5/cmd.c +1 -0
 	err = mlx5_core_alloc_pd(mdev, &pdn);
 	if (err)
 		return err;
@@ -353,6 +356,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,
 	get_file(migf->filp);
 	async_data->mkey = mkey;
 	async_data->pdn = pdn;
+	migf->save_cb_active = true;
 	err = mlx5_cmd_exec_cb(&migf->async_ctx, in, sizeof(in),
 			       async_data->out,
 			       out_size, mlx5vf_save_callback,

drivers/vfio/pci/mlx5/cmd.c int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, · line 378

Quoted 4 lines
drivers/vfio/pci/mlx5/cmd.c +1 -0
@@ -371,6 +375,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,
 	dma_unmap_sgtable(mdev->device, &migf->table.sgt, DMA_FROM_DEVICE, 0);
 err_dma_map:
 	mlx5_core_dealloc_pd(mdev, pdn);
+	migf->save_cb_active = false;

complete()

Quoted 28 lines
drivers/vfio/pci/mlx5/cmd.c +0 -0
 	return err;
 }
 
drivers/vfio/pci/mlx5/cmd.h +2 -0
index 921d5720a1e5..b1c5dd2ff144 100644
@@ -26,6 +26,7 @@ struct mlx5_vf_migration_file {
 	struct mutex lock;
 	u8 disabled:1;
 	u8 is_err:1;
+	u8 save_cb_active:1;
 
 	struct sg_append_table table;
 	size_t total_length;
@@ -37,6 +38,7 @@ struct mlx5_vf_migration_file {
 	unsigned long last_offset;
 	struct mlx5vf_pci_core_device *mvdev;
 	wait_queue_head_t poll_wait;
+	wait_queue_head_t save_wait;
 	struct mlx5_async_ctx async_ctx;
 	struct mlx5vf_async_data async_data;
 };
drivers/vfio/pci/mlx5/main.c +0 -0
index 4c7a39ffd247..5da278f3c31c 100644

drivers/vfio/pci/mlx5/main.c mlx5vf_pci_save_device_data(struct mlx5vf_pci_core_device *mvdev) · line 248

Quoted 4 lines
drivers/vfio/pci/mlx5/main.c +1 -0
@@ -245,6 +245,7 @@ mlx5vf_pci_save_device_data(struct mlx5vf_pci_core_device *mvdev)
 	stream_open(migf->filp->f_inode, migf->filp);
 	mutex_init(&migf->lock);
 	init_waitqueue_head(&migf->poll_wait);
+	init_waitqueue_head(&migf->save_wait);

init_completion()
complete()

Jason

author maintainer
#12 lore raw

Re: [PATCH vfio 05/13] vfio/mlx5: Enforce a single SAVE command at a time

Quoted 13 lines · Jason Gunthorpe
Quoted 12 lines · Yishai Hadas

Enforce a single SAVE command at a time.

As the SAVE command is an asynchronous one, we must enforce running only a single command at a time.

This will preserve ordering between multiple calls and protect from races on the migration file data structure.

This is a must for the next patches from the series where as part of PRE_COPY we may have multiple images to be saved and multiple SAVE commands may be issued from different flows.

Quoted 8 lines
Quoted 6 lines
---
  drivers/vfio/pci/mlx5/cmd.c  | 5 +++++
  drivers/vfio/pci/mlx5/cmd.h  | 2 ++
  drivers/vfio/pci/mlx5/main.c | 1 +
  3 files changed, 8 insertions(+)

This should just use a 'counting completion' instead of open coding one.

Makes sense, will change accordingly as part of V1.

Yishai

Quoted 77 lines

drivers/vfio/pci/mlx5/cmd.c void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work) · line 285

Quoted 11 lines
drivers/vfio/pci/mlx5/cmd.c +2 -0
index 0848bc905d3e..b9ed2f1c8689 100644
@@ -281,6 +281,8 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work)
  	dma_unmap_sgtable(mdev->device, &migf->table.sgt, DMA_FROM_DEVICE, 0);
  	mlx5_core_dealloc_pd(mdev, async_data->pdn);
  	kvfree(async_data->out);
+	migf->save_cb_active = false;
+	wake_up(&migf->save_wait);

complete()

drivers/vfio/pci/mlx5/cmd.c int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, · line 326

Quoted 8 lines
drivers/vfio/pci/mlx5/cmd.c +1 -0
  	fput(migf->filp);
  }
 
@@ -321,6 +323,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,
  		return -ENOTCONN;
 
  	mdev = mvdev->mdev;
+	wait_event(migf->save_wait, !migf->save_cb_active);

wait_for_completion_interruptible()

Quoted 12 lines
drivers/vfio/pci/mlx5/cmd.c +1 -0
  	err = mlx5_core_alloc_pd(mdev, &pdn);
  	if (err)
  		return err;
@@ -353,6 +356,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,
  	get_file(migf->filp);
  	async_data->mkey = mkey;
  	async_data->pdn = pdn;
+	migf->save_cb_active = true;
  	err = mlx5_cmd_exec_cb(&migf->async_ctx, in, sizeof(in),
  			       async_data->out,
  			       out_size, mlx5vf_save_callback,

drivers/vfio/pci/mlx5/cmd.c int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, · line 378

Quoted 4 lines
drivers/vfio/pci/mlx5/cmd.c +1 -0
@@ -371,6 +375,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,
  	dma_unmap_sgtable(mdev->device, &migf->table.sgt, DMA_FROM_DEVICE, 0);
  err_dma_map:
  	mlx5_core_dealloc_pd(mdev, pdn);
+	migf->save_cb_active = false;

complete()

Quoted 28 lines
drivers/vfio/pci/mlx5/cmd.c +0 -0
  	return err;
  }
 
drivers/vfio/pci/mlx5/cmd.h +2 -0
index 921d5720a1e5..b1c5dd2ff144 100644
@@ -26,6 +26,7 @@ struct mlx5_vf_migration_file {
  	struct mutex lock;
  	u8 disabled:1;
  	u8 is_err:1;
+	u8 save_cb_active:1;
 
  	struct sg_append_table table;
  	size_t total_length;
@@ -37,6 +38,7 @@ struct mlx5_vf_migration_file {
  	unsigned long last_offset;
  	struct mlx5vf_pci_core_device *mvdev;
  	wait_queue_head_t poll_wait;
+	wait_queue_head_t save_wait;
  	struct mlx5_async_ctx async_ctx;
  	struct mlx5vf_async_data async_data;
  };
drivers/vfio/pci/mlx5/main.c +0 -0
index 4c7a39ffd247..5da278f3c31c 100644

drivers/vfio/pci/mlx5/main.c mlx5vf_pci_save_device_data(struct mlx5vf_pci_core_device *mvdev) · line 248

Quoted 4 lines
drivers/vfio/pci/mlx5/main.c +1 -0
@@ -245,6 +245,7 @@ mlx5vf_pci_save_device_data(struct mlx5vf_pci_core_device *mvdev)
  	stream_open(migf->filp->f_inode, migf->filp);
  	mutex_init(&migf->lock);
  	init_waitqueue_head(&migf->poll_wait);
+	init_waitqueue_head(&migf->save_wait);

init_completion()
complete()

Jason

author maintainer
#13 lore raw

PATCH 6/13 vfio/mlx5: Refactor total_length name and usage

In reply to Yishai Hadas

From: Shay Drory <[email protected]>

On the source side flow, change the usage of total_length to keep only the size of the image as returned from the FW and rename it to image_length. This is eliminating the usage of total_length inside the SAVE command.

This is a preparation step for the next patches from the series where more than one image could be managed on the migration file.

---
 drivers/vfio/pci/mlx5/cmd.c  | 24 +++++++++++-------------
 drivers/vfio/pci/mlx5/cmd.h  |  2 +-
 drivers/vfio/pci/mlx5/main.c | 20 ++++++++++----------
 3 files changed, 22 insertions(+), 24 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +11 -13
index b9ed2f1c8689..24c6d2e4c2be 100644
@@ -212,9 +212,9 @@ static int mlx5vf_cmd_get_vhca_id(struct mlx5_core_dev *mdev, u16 function_id,
 static int _create_mkey(struct mlx5_core_dev *mdev, u32 pdn,
 			struct mlx5_vf_migration_file *migf,
 			struct mlx5_vhca_recv_buf *recv_buf,
-			u32 *mkey)
+			u32 *mkey, size_t length)
 {
-	size_t npages = migf ? DIV_ROUND_UP(migf->total_length, PAGE_SIZE) :
+	size_t npages = migf ? DIV_ROUND_UP(length, PAGE_SIZE) :
 				recv_buf->npages;
 	int err = 0, inlen;
 	__be64 *mtt;
@@ -255,8 +255,7 @@ static int _create_mkey(struct mlx5_core_dev *mdev, u32 pdn,
 	MLX5_SET(mkc, mkc, qpn, 0xffffff);
 	MLX5_SET(mkc, mkc, log_page_size, PAGE_SHIFT);
 	MLX5_SET(mkc, mkc, translations_octword_size, DIV_ROUND_UP(npages, 2));
-	MLX5_SET64(mkc, mkc, len,
-		   migf ? migf->total_length : (npages * PAGE_SIZE));
+	MLX5_SET64(mkc, mkc, len, migf ? length : (npages * PAGE_SIZE));
 	err = mlx5_core_create_mkey(mdev, mkey, in, inlen);
 	kvfree(in);
 	return err;
@@ -294,7 +293,7 @@ static void mlx5vf_save_callback(int status, struct mlx5_async_work *context)
 			struct mlx5_vf_migration_file, async_data);
 
 	if (!status) {
-		WRITE_ONCE(migf->total_length,
+		WRITE_ONCE(migf->image_length,
 			   MLX5_GET(save_vhca_state_out, async_data->out,
 				    actual_image_size));
 		wake_up_interruptible(&migf->poll_wait);
@@ -333,7 +332,8 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,
 	if (err)
 		goto err_dma_map;
 
-	err = _create_mkey(mdev, pdn, migf, NULL, &mkey);
+	err = _create_mkey(mdev, pdn, migf, NULL,
+			   &mkey, migf->allocated_length);
 	if (err)
 		goto err_create_mkey;
 
@@ -342,7 +342,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,
 	MLX5_SET(save_vhca_state_in, in, op_mod, 0);
 	MLX5_SET(save_vhca_state_in, in, vhca_id, mvdev->vhca_id);
 	MLX5_SET(save_vhca_state_in, in, mkey, mkey);
-	MLX5_SET(save_vhca_state_in, in, size, migf->total_length);
+	MLX5_SET(save_vhca_state_in, in, size, migf->allocated_length);
 
 	async_data = &migf->async_data;
 	async_data->out = kvzalloc(out_size, GFP_KERNEL);
@@ -351,8 +351,6 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,
 		goto err_out;
 	}
 
-	/* no data exists till the callback comes back */
-	migf->total_length = 0;
 	get_file(migf->filp);
 	async_data->mkey = mkey;
 	async_data->pdn = pdn;
@@ -393,7 +391,7 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev,
 		return -ENOTCONN;
 
 	mutex_lock(&migf->lock);
-	if (!migf->total_length) {
+	if (!migf->image_length) {
 		err = -EINVAL;
 		goto end;
 	}
@@ -407,7 +405,7 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev,
 	if (err)
 		goto err_reg;
 
-	err = _create_mkey(mdev, pdn, migf, NULL, &mkey);
+	err = _create_mkey(mdev, pdn, migf, NULL, &mkey, migf->image_length);
 	if (err)
 		goto err_mkey;
 
@@ -416,7 +414,7 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev,
 	MLX5_SET(load_vhca_state_in, in, op_mod, 0);
 	MLX5_SET(load_vhca_state_in, in, vhca_id, mvdev->vhca_id);
 	MLX5_SET(load_vhca_state_in, in, mkey, mkey);
-	MLX5_SET(load_vhca_state_in, in, size, migf->total_length);
+	MLX5_SET(load_vhca_state_in, in, size, migf->image_length);
 
 	err = mlx5_cmd_exec_inout(mdev, load_vhca_state, in, out);
 
@@ -1047,7 +1045,7 @@ static int mlx5vf_alloc_qp_recv_resources(struct mlx5_core_dev *mdev,
 	if (err)
 		goto end;
 
-	err = _create_mkey(mdev, pdn, NULL, recv_buf, &recv_buf->mkey);
+	err = _create_mkey(mdev, pdn, NULL, recv_buf, &recv_buf->mkey, 0);
 	if (err)
 		goto err_create_mkey;
 
drivers/vfio/pci/mlx5/cmd.h +1 -1
index b1c5dd2ff144..b1fa1a0418a5 100644
@@ -29,7 +29,7 @@ struct mlx5_vf_migration_file {
 	u8 save_cb_active:1;
 
 	struct sg_append_table table;
-	size_t total_length;
+	size_t image_length;
 	size_t allocated_length;
 
 	/* Optimize mlx5vf_get_migration_page() for sequential access */
drivers/vfio/pci/mlx5/main.c +10 -10
index 5da278f3c31c..624b1a99dc21 100644
@@ -116,7 +116,7 @@ static void mlx5vf_disable_fd(struct mlx5_vf_migration_file *migf)
 		__free_page(sg_page_iter_page(&sg_iter));
 	sg_free_append_table(&migf->table);
 	migf->disabled = true;
-	migf->total_length = 0;
+	migf->image_length = 0;
 	migf->allocated_length = 0;
 	migf->filp->f_pos = 0;
 	mutex_unlock(&migf->lock);
@@ -144,16 +144,16 @@ static ssize_t mlx5vf_save_read(struct file *filp, char __user *buf, size_t len,
 
 	if (!(filp->f_flags & O_NONBLOCK)) {
 		if (wait_event_interruptible(migf->poll_wait,
-			     READ_ONCE(migf->total_length) || migf->is_err))
+			     READ_ONCE(migf->image_length) || migf->is_err))
 			return -ERESTARTSYS;
 	}
 
 	mutex_lock(&migf->lock);
-	if ((filp->f_flags & O_NONBLOCK) && !READ_ONCE(migf->total_length)) {
+	if ((filp->f_flags & O_NONBLOCK) && !READ_ONCE(migf->image_length)) {
 		done = -EAGAIN;
 		goto out_unlock;
 	}
-	if (*pos > migf->total_length) {
+	if (*pos > migf->image_length) {
 		done = -EINVAL;
 		goto out_unlock;
 	}
@@ -162,7 +162,7 @@ static ssize_t mlx5vf_save_read(struct file *filp, char __user *buf, size_t len,
 		goto out_unlock;
 	}
 
-	len = min_t(size_t, migf->total_length - *pos, len);
+	len = min_t(size_t, migf->image_length - *pos, len);
 	while (len) {
 		size_t page_offset;
 		struct page *page;
@@ -208,7 +208,7 @@ static __poll_t mlx5vf_save_poll(struct file *filp,
 	mutex_lock(&migf->lock);
 	if (migf->disabled || migf->is_err)
 		pollflags = EPOLLIN | EPOLLRDNORM | EPOLLRDHUP;
-	else if (READ_ONCE(migf->total_length))
+	else if (READ_ONCE(migf->image_length))
 		pollflags = EPOLLIN | EPOLLRDNORM;
 	mutex_unlock(&migf->lock);
 
@@ -227,6 +227,7 @@ static struct mlx5_vf_migration_file *
 mlx5vf_pci_save_device_data(struct mlx5vf_pci_core_device *mvdev)
 {
 	struct mlx5_vf_migration_file *migf;
+	size_t length;
 	int ret;
 
 	migf = kzalloc(sizeof(*migf), GFP_KERNEL);
@@ -248,13 +249,12 @@ mlx5vf_pci_save_device_data(struct mlx5vf_pci_core_device *mvdev)
 	init_waitqueue_head(&migf->save_wait);
 	mlx5_cmd_init_async_ctx(mvdev->mdev, &migf->async_ctx);
 	INIT_WORK(&migf->async_data.work, mlx5vf_mig_file_cleanup_cb);
-	ret = mlx5vf_cmd_query_vhca_migration_state(mvdev,
-						    &migf->total_length);
+	ret = mlx5vf_cmd_query_vhca_migration_state(mvdev, &length);
 	if (ret)
 		goto out_free;
 
 	ret = mlx5vf_add_migration_pages(
-		migf, DIV_ROUND_UP_ULL(migf->total_length, PAGE_SIZE));
+		migf, DIV_ROUND_UP_ULL(length, PAGE_SIZE));
 	if (ret)
 		goto out_free;
 
@@ -328,7 +328,7 @@ static ssize_t mlx5vf_resume_write(struct file *filp, const char __user *buf,
 		len -= page_len;
 		done += page_len;
 		buf += page_len;
-		migf->total_length += page_len;
+		migf->image_length += page_len;
 	}
 out_unlock:
 	mutex_unlock(&migf->lock);
Signature · 1 line
2.18.1
#14 lore raw

Re: [PATCH vfio 06/13] vfio/mlx5: Refactor total_length name and usage

drivers/vfio/pci/mlx5/cmd.c static int mlx5vf_alloc_qp_recv_resources(struct mlx5_core_dev *mdev, · line 1048

Quoted 6 lines · Yishai Hadas
drivers/vfio/pci/mlx5/cmd.c +1 -1
@@ -1047,7 +1045,7 @@ static int mlx5vf_alloc_qp_recv_resources(struct mlx5_core_dev *mdev,
 	if (err)
 		goto end;
 
-	err = _create_mkey(mdev, pdn, NULL, recv_buf, &recv_buf->mkey);
+	err = _create_mkey(mdev, pdn, NULL, recv_buf, &recv_buf->mkey, 0);

This seems pretty goofy, a 0 length means use the length in the recv_buf?

Jason

author maintainer
#15 lore raw

Re: [PATCH vfio 06/13] vfio/mlx5: Refactor total_length name and usage

Quoted 11 lines · Jason Gunthorpe

drivers/vfio/pci/mlx5/cmd.c static int mlx5vf_alloc_qp_recv_resources(struct mlx5_core_dev *mdev, · line 1048

Quoted 6 lines · Yishai Hadas
drivers/vfio/pci/mlx5/cmd.c +1 -1
@@ -1047,7 +1045,7 @@ static int mlx5vf_alloc_qp_recv_resources(struct mlx5_core_dev *mdev,
  	if (err)
  		goto end;
 
-	err = _create_mkey(mdev, pdn, NULL, recv_buf, &recv_buf->mkey);
+	err = _create_mkey(mdev, pdn, NULL, recv_buf, &recv_buf->mkey, 0);

This seems pretty goofy, a 0 length means use the length in the recv_buf?

Right

In V1 will embed here the expected length (i.e. npages * PAGE_SIZE) and use it directly as part of _create_mkey().

Yishai

author maintainer
#16 lore raw

PATCH 7/13 vfio/mlx5: Introduce device transitions of PRE_COPY

In reply to Yishai Hadas

From: Shay Drory <[email protected]>

In order to support PRE_COPY, mlx5 driver is transferring multiple states (images) of the device. e.g.: the source VF can save and transfer multiple states, and the target VF will load them by that order.

The device is saving three kinds of states:
1) Initial state - when the device moves to PRE_COPY state.
2) Middle state - during PRE_COPY phase via VFIO_MIG_GET_PRECOPY_INFO. There can be multiple states of this type.
3) Final state - when the device moves to STOP_COPY state.

After moving to PRE_COPY state, user is holding the saving migf FD and can use it. For example: user can start transferring data via read() callback. Also, user can switch from PRE_COPY to STOP_COPY whenever he sees it fits. This will invoke saving of final state.

This means that mlx5 VFIO device can be switched to STOP_COPY without transferring any data in PRE_COPY state. Therefore, when the device moves to STOP_COPY, mlx5 will store the final state on a dedicated data structure.

---
 drivers/vfio/pci/mlx5/cmd.c  | 56 +++++++++++++++-------
 drivers/vfio/pci/mlx5/cmd.h  | 16 ++++++-
 drivers/vfio/pci/mlx5/main.c | 93 +++++++++++++++++++++++++++++++-----
 3 files changed, 134 insertions(+), 31 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +39 -17
index 24c6d2e4c2be..eb684455c2b2 100644
@@ -14,6 +14,7 @@ _mlx5vf_free_page_tracker_resources(struct mlx5vf_pci_core_device *mvdev);
 
 int mlx5vf_cmd_suspend_vhca(struct mlx5vf_pci_core_device *mvdev, u16 op_mod)
 {
+	struct mlx5_vf_migration_file *migf = mvdev->saving_migf;
 	u32 out[MLX5_ST_SZ_DW(suspend_vhca_out)] = {};
 	u32 in[MLX5_ST_SZ_DW(suspend_vhca_in)] = {};
 
@@ -21,6 +22,14 @@ int mlx5vf_cmd_suspend_vhca(struct mlx5vf_pci_core_device *mvdev, u16 op_mod)
 	if (mvdev->mdev_detach)
 		return -ENOTCONN;
 
+	/*
+	 * In case PRE_COPY is used, saving_migf is exposed while the device is
+	 * running. Make sure to run only once there is no active save command.
+	 * Running both in parallel, might end-up with a failure in the save
+	 * command once it will try to turn on 'tracking' on a suspended device.
+	 */
+	if (migf)
+		wait_event(migf->save_wait, !migf->save_cb_active);
 	MLX5_SET(suspend_vhca_in, in, opcode, MLX5_CMD_OP_SUSPEND_VHCA);
 	MLX5_SET(suspend_vhca_in, in, vhca_id, mvdev->vhca_id);
 	MLX5_SET(suspend_vhca_in, in, op_mod, op_mod);
@@ -45,7 +54,7 @@ int mlx5vf_cmd_resume_vhca(struct mlx5vf_pci_core_device *mvdev, u16 op_mod)
 }
 
 int mlx5vf_cmd_query_vhca_migration_state(struct mlx5vf_pci_core_device *mvdev,
-					  size_t *state_size)
+					  size_t *state_size, u8 query_flags)
 {
 	u32 out[MLX5_ST_SZ_DW(query_vhca_migration_state_out)] = {};
 	u32 in[MLX5_ST_SZ_DW(query_vhca_migration_state_in)] = {};
@@ -59,6 +68,8 @@ int mlx5vf_cmd_query_vhca_migration_state(struct mlx5vf_pci_core_device *mvdev,
 		 MLX5_CMD_OP_QUERY_VHCA_MIGRATION_STATE);
 	MLX5_SET(query_vhca_migration_state_in, in, vhca_id, mvdev->vhca_id);
 	MLX5_SET(query_vhca_migration_state_in, in, op_mod, 0);
+	MLX5_SET(query_vhca_migration_state_in, in, incremental,
+		 query_flags & MLX5VF_QUERY_INC);
 
 	ret = mlx5_cmd_exec_inout(mvdev->mdev, query_vhca_migration_state, in,
 				  out);
@@ -210,11 +221,11 @@ static int mlx5vf_cmd_get_vhca_id(struct mlx5_core_dev *mdev, u16 function_id,
 }
 
 static int _create_mkey(struct mlx5_core_dev *mdev, u32 pdn,
-			struct mlx5_vf_migration_file *migf,
+			struct sg_table *sgt,
 			struct mlx5_vhca_recv_buf *recv_buf,
 			u32 *mkey, size_t length)
 {
-	size_t npages = migf ? DIV_ROUND_UP(length, PAGE_SIZE) :
+	size_t npages = sgt ? DIV_ROUND_UP(length, PAGE_SIZE) :
 				recv_buf->npages;
 	int err = 0, inlen;
 	__be64 *mtt;
@@ -232,10 +243,10 @@ static int _create_mkey(struct mlx5_core_dev *mdev, u32 pdn,
 		 DIV_ROUND_UP(npages, 2));
 	mtt = (__be64 *)MLX5_ADDR_OF(create_mkey_in, in, klm_pas_mtt);
 
-	if (migf) {
+	if (sgt) {
 		struct sg_dma_page_iter dma_iter;
 
-		for_each_sgtable_dma_page(&migf->table.sgt, &dma_iter, 0)
+		for_each_sgtable_dma_page(sgt, &dma_iter, 0)
 			*mtt++ = cpu_to_be64(sg_page_iter_dma_address(&dma_iter));
 	} else {
 		int i;
@@ -255,7 +266,7 @@ static int _create_mkey(struct mlx5_core_dev *mdev, u32 pdn,
 	MLX5_SET(mkc, mkc, qpn, 0xffffff);
 	MLX5_SET(mkc, mkc, log_page_size, PAGE_SHIFT);
 	MLX5_SET(mkc, mkc, translations_octword_size, DIV_ROUND_UP(npages, 2));
-	MLX5_SET64(mkc, mkc, len, migf ? length : (npages * PAGE_SIZE));
+	MLX5_SET64(mkc, mkc, len, sgt ? length : (npages * PAGE_SIZE));
 	err = mlx5_core_create_mkey(mdev, mkey, in, inlen);
 	kvfree(in);
 	return err;
@@ -277,7 +288,7 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work)
 	mutex_unlock(&migf->lock);
 
 	mlx5_core_destroy_mkey(mdev, async_data->mkey);
-	dma_unmap_sgtable(mdev->device, &migf->table.sgt, DMA_FROM_DEVICE, 0);
+	dma_unmap_sgtable(mdev->device, async_data->sgt, DMA_FROM_DEVICE, 0);
 	mlx5_core_dealloc_pd(mdev, async_data->pdn);
 	kvfree(async_data->out);
 	migf->save_cb_active = false;
@@ -293,9 +304,14 @@ static void mlx5vf_save_callback(int status, struct mlx5_async_work *context)
 			struct mlx5_vf_migration_file, async_data);
 
 	if (!status) {
-		WRITE_ONCE(migf->image_length,
-			   MLX5_GET(save_vhca_state_out, async_data->out,
-				    actual_image_size));
+		size_t len = MLX5_GET(save_vhca_state_out, async_data->out,
+				      actual_image_size);
+
+		if (async_data->sgt == &migf->final_table.sgt)
+			WRITE_ONCE(migf->final_length, len);
+		else
+			WRITE_ONCE(migf->image_length, len);
+
 		wake_up_interruptible(&migf->poll_wait);
 	}
 
@@ -308,7 +324,8 @@ static void mlx5vf_save_callback(int status, struct mlx5_async_work *context)
 }
 
 int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,
-			       struct mlx5_vf_migration_file *migf)
+			       struct mlx5_vf_migration_file *migf, bool inc,
+			       bool track)
 {
 	u32 out_size = MLX5_ST_SZ_BYTES(save_vhca_state_out);
 	u32 in[MLX5_ST_SZ_DW(save_vhca_state_in)] = {};
@@ -327,12 +344,15 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,
 	if (err)
 		return err;
 
-	err = dma_map_sgtable(mdev->device, &migf->table.sgt, DMA_FROM_DEVICE,
-			      0);
+	async_data = &migf->async_data;
+	async_data->sgt = (!track && inc) ? &migf->final_table.sgt :
+		&migf->table.sgt;
+	err = dma_map_sgtable(mdev->device, async_data->sgt,
+			      DMA_FROM_DEVICE, 0);
 	if (err)
 		goto err_dma_map;
 
-	err = _create_mkey(mdev, pdn, migf, NULL,
+	err = _create_mkey(mdev, pdn, async_data->sgt, NULL,
 			   &mkey, migf->allocated_length);
 	if (err)
 		goto err_create_mkey;
@@ -343,8 +363,9 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,
 	MLX5_SET(save_vhca_state_in, in, vhca_id, mvdev->vhca_id);
 	MLX5_SET(save_vhca_state_in, in, mkey, mkey);
 	MLX5_SET(save_vhca_state_in, in, size, migf->allocated_length);
+	MLX5_SET(save_vhca_state_in, in, incremental, inc);
+	MLX5_SET(save_vhca_state_in, in, set_track, track);
 
-	async_data = &migf->async_data;
 	async_data->out = kvzalloc(out_size, GFP_KERNEL);
 	if (!async_data->out) {
 		err = -ENOMEM;
@@ -370,7 +391,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,
 err_out:
 	mlx5_core_destroy_mkey(mdev, mkey);
 err_create_mkey:
-	dma_unmap_sgtable(mdev->device, &migf->table.sgt, DMA_FROM_DEVICE, 0);
+	dma_unmap_sgtable(mdev->device, async_data->sgt, DMA_FROM_DEVICE, 0);
 err_dma_map:
 	mlx5_core_dealloc_pd(mdev, pdn);
 	migf->save_cb_active = false;
@@ -405,7 +426,8 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev,
 	if (err)
 		goto err_reg;
 
-	err = _create_mkey(mdev, pdn, migf, NULL, &mkey, migf->image_length);
+	err = _create_mkey(mdev, pdn, &migf->table.sgt, NULL, &mkey,
+			   migf->image_length);
 	if (err)
 		goto err_mkey;
 
drivers/vfio/pci/mlx5/cmd.h +14 -2
index b1fa1a0418a5..c12fa81ba53f 100644
@@ -15,6 +15,7 @@
 struct mlx5vf_async_data {
 	struct mlx5_async_work cb_work;
 	struct work_struct work;
+	struct sg_table *sgt;
 	int status;
 	u32 pdn;
 	u32 mkey;
@@ -31,6 +32,12 @@ struct mlx5_vf_migration_file {
 	struct sg_append_table table;
 	size_t image_length;
 	size_t allocated_length;
+	/*
+	 * The device can be moved to stop_copy before the previous state was
+	 * fully read. Another set of variables is needed to maintain it.
+	 */
+	size_t final_length;
+	struct sg_append_table final_table;
 
 	/* Optimize mlx5vf_get_migration_page() for sequential access */
 	struct scatterlist *last_offset_sg;
@@ -115,17 +122,22 @@ struct mlx5vf_pci_core_device {
 	struct mlx5_core_dev *mdev;
 };
 
+enum {
+	MLX5VF_QUERY_INC = (1UL << 0),
+};
+
 int mlx5vf_cmd_suspend_vhca(struct mlx5vf_pci_core_device *mvdev, u16 op_mod);
 int mlx5vf_cmd_resume_vhca(struct mlx5vf_pci_core_device *mvdev, u16 op_mod);
 int mlx5vf_cmd_query_vhca_migration_state(struct mlx5vf_pci_core_device *mvdev,
-					  size_t *state_size);
+					  size_t *state_size, u8 query_flags);
 void mlx5vf_cmd_set_migratable(struct mlx5vf_pci_core_device *mvdev,
 			       const struct vfio_migration_ops *mig_ops,
 			       const struct vfio_log_ops *log_ops);
 void mlx5vf_cmd_remove_migratable(struct mlx5vf_pci_core_device *mvdev);
 void mlx5vf_cmd_close_migratable(struct mlx5vf_pci_core_device *mvdev);
 int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,
-			       struct mlx5_vf_migration_file *migf);
+			       struct mlx5_vf_migration_file *migf, bool inc,
+			       bool track);
 int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev,
 			       struct mlx5_vf_migration_file *migf);
 void mlx5vf_state_mutex_unlock(struct mlx5vf_pci_core_device *mvdev);
drivers/vfio/pci/mlx5/main.c +81 -12

181 lines not on this page · raw message

Signature · 1 line
2.18.1
author maintainer
#17 lore raw

PATCH 8/13 vfio/mlx5: Introduce vfio precopy ioctl implementation

In reply to Yishai Hadas

From: Shay Drory <[email protected]>

vfio precopy ioctl returns an estimation of data available for transferring from the device.

Whenever a user is using VFIO_MIG_GET_PRECOPY_INFO, track the current state of the device, and if needed, append the dirty data to the transfer FD data. This is done by saving a middle state.

As mlx5 runs the SAVE command asynchronously, make sure to query for incremental data only once there is no active save command. Running both in parallel, might end-up with a failure in the incremental query command on un-tracked vhca.

Also, a middle state will be saved only after the previous state has finished its SAVE command and has been fully transferred, this enables to re-use the resources.

In order to map between FD position and the new saved state data, store the current FD position.

---
 drivers/vfio/pci/mlx5/cmd.c  |   9 +++
 drivers/vfio/pci/mlx5/cmd.h  |   1 +
 drivers/vfio/pci/mlx5/main.c | 131 +++++++++++++++++++++++++++++++++++
 3 files changed, 141 insertions(+)
drivers/vfio/pci/mlx5/cmd.c +9 -0
index eb684455c2b2..2d2171191218 100644
@@ -64,6 +64,15 @@ int mlx5vf_cmd_query_vhca_migration_state(struct mlx5vf_pci_core_device *mvdev,
 	if (mvdev->mdev_detach)
 		return -ENOTCONN;
 
+	/*
+	 * In case PRE_COPY is used, saving_migf is exposed while device is
+	 * running. Make sure to run only once there is no active save command.
+	 * Running both in parallel, might end-up with a failure in the
+	 * incremental query command on un-tracked vhca.
+	 */
+	if (query_flags & MLX5VF_QUERY_INC)
+		wait_event(mvdev->saving_migf->save_wait,
+			   !mvdev->saving_migf->save_cb_active);
 	MLX5_SET(query_vhca_migration_state_in, in, opcode,
 		 MLX5_CMD_OP_QUERY_VHCA_MIGRATION_STATE);
 	MLX5_SET(query_vhca_migration_state_in, in, vhca_id, mvdev->vhca_id);
drivers/vfio/pci/mlx5/cmd.h +1 -0
index c12fa81ba53f..07a2fc54c9d8 100644
@@ -30,6 +30,7 @@ struct mlx5_vf_migration_file {
 	u8 save_cb_active:1;
 
 	struct sg_append_table table;
+	size_t table_start_pos;
 	size_t image_length;
 	size_t allocated_length;
 	/*
drivers/vfio/pci/mlx5/main.c +131 -0

159 lines not on this page · raw message

Signature · 1 line
2.18.1
author maintainer
#18 lore raw

PATCH 9/13 vfio/mlx5: Manage read() of multiple state saves

In reply to Yishai Hadas

From: Shay Drory <[email protected]>

Since all the states that were mentioned in previous patches are transferred over the same FD, on one hand, and mlx5 keeps one data structure per state, on the other hand, mlx5 needs to manage the delta between FD position and the current state (data structure) transferred.

Also, as mentioned in previous patch, user can switch VFIO device to STOP_COPY without transferring any data in PRE_COPY state. Hence, the delta management of the final state has a dedicated data structure.

---
 drivers/vfio/pci/mlx5/main.c | 115 ++++++++++++++++++++++++++++++-----
 1 file changed, 100 insertions(+), 15 deletions(-)
drivers/vfio/pci/mlx5/main.c +100 -15
index 266626066fed..8a5714158e43 100644
@@ -34,7 +34,7 @@ static struct mlx5vf_pci_core_device *mlx5vf_drvdata(struct pci_dev *pdev)
 
 static struct page *
 mlx5vf_get_migration_page(struct mlx5_vf_migration_file *migf,
-			  unsigned long offset)
+			  unsigned long offset, struct sg_append_table *table)
 {
 	unsigned long cur_offset = 0;
 	struct scatterlist *sg;
@@ -43,14 +43,14 @@ mlx5vf_get_migration_page(struct mlx5_vf_migration_file *migf,
 	/* All accesses are sequential */
 	if (offset < migf->last_offset || !migf->last_offset_sg) {
 		migf->last_offset = 0;
-		migf->last_offset_sg = migf->table.sgt.sgl;
+		migf->last_offset_sg = table->sgt.sgl;
 		migf->sg_last_entry = 0;
 	}
 
 	cur_offset = migf->last_offset;
 
 	for_each_sg(migf->last_offset_sg, sg,
-			migf->table.sgt.orig_nents - migf->sg_last_entry, i) {
+			table->sgt.orig_nents - migf->sg_last_entry, i) {
 		if (offset < sg->length + cur_offset) {
 			migf->last_offset_sg = sg;
 			migf->sg_last_entry += i;
@@ -161,10 +161,45 @@ static int mlx5vf_release_file(struct inode *inode, struct file *filp)
 	(mvdev->mig_state == VFIO_DEVICE_STATE_PRE_COPY || \
 	 mvdev->mig_state == VFIO_DEVICE_STATE_PRE_COPY_P2P)
 
+#define VFIO_PRE_COPY_SUPP(mvdev) \
+	(mvdev->core_device.vdev.migration_flags & VFIO_MIGRATION_PRE_COPY)
+
+#define MIGF_HAS_DATA(migf) \
+	(READ_ONCE(migf->image_length) || READ_ONCE(migf->final_length))
+
+static size_t
+mlx5vf_final_table_start_pos(struct mlx5_vf_migration_file *migf)
+{
+	return MIGF_TOTAL_DATA(migf) - migf->final_length;
+}
+
+static size_t mlx5vf_get_table_start_pos(struct mlx5_vf_migration_file *migf)
+{
+	return migf->table_start_pos;
+}
+
+static size_t mlx5vf_get_table_end_pos(struct mlx5_vf_migration_file *migf,
+				       struct sg_append_table *table)
+{
+	if (table == &migf->final_table)
+		return MIGF_TOTAL_DATA(migf);
+	return migf->table_start_pos + migf->image_length;
+}
+
+static struct sg_append_table *
+mlx5vf_get_table(struct mlx5_vf_migration_file *migf, loff_t *pos)
+{
+	if (migf->final_length &&
+	    *pos >= mlx5vf_final_table_start_pos(migf))
+		return &migf->final_table;
+	return &migf->table;
+}
+
 static ssize_t mlx5vf_save_read(struct file *filp, char __user *buf, size_t len,
 			       loff_t *pos)
 {
 	struct mlx5_vf_migration_file *migf = filp->private_data;
+	struct sg_append_table *table;
 	ssize_t done = 0;
 
 	if (pos)
@@ -173,16 +208,16 @@ static ssize_t mlx5vf_save_read(struct file *filp, char __user *buf, size_t len,
 
 	if (!(filp->f_flags & O_NONBLOCK)) {
 		if (wait_event_interruptible(migf->poll_wait,
-			     READ_ONCE(migf->image_length) || migf->is_err))
+			     (MIGF_HAS_DATA(migf) || migf->is_err)))
 			return -ERESTARTSYS;
 	}
 
 	mutex_lock(&migf->lock);
-	if ((filp->f_flags & O_NONBLOCK) && !READ_ONCE(migf->image_length)) {
+	if ((filp->f_flags & O_NONBLOCK) && !MIGF_HAS_DATA(migf)) {
 		done = -EAGAIN;
 		goto out_unlock;
 	}
-	if (*pos > migf->image_length) {
+	if (*pos > MIGF_TOTAL_DATA(migf)) {
 		done = -EINVAL;
 		goto out_unlock;
 	}
@@ -191,16 +226,28 @@ static ssize_t mlx5vf_save_read(struct file *filp, char __user *buf, size_t len,
 		goto out_unlock;
 	}
 
-	len = min_t(size_t, migf->image_length - *pos, len);
+	/* If we reach the end of the PRE_COPY size */
+	if (MIGF_TOTAL_DATA(migf) == *pos &&
+	    VFIO_MIG_STATE_PRE_COPY(migf->mvdev)) {
+		done = -ENOMSG;
+		goto out_unlock;
+	}
+
+	len = min_t(size_t, MIGF_TOTAL_DATA(migf) - *pos, len);
+	table = mlx5vf_get_table(migf, pos);
 	while (len) {
+		struct sg_append_table *tmp = table;
+		unsigned long offset;
 		size_t page_offset;
 		struct page *page;
 		size_t page_len;
 		u8 *from_buff;
 		int ret;
 
-		page_offset = (*pos) % PAGE_SIZE;
-		page = mlx5vf_get_migration_page(migf, *pos - page_offset);
+		offset = *pos - mlx5vf_get_table_start_pos(migf);
+		page_offset = offset % PAGE_SIZE;
+		offset -= page_offset;
+		page = mlx5vf_get_migration_page(migf, offset, table);
 		if (!page) {
 			if (done == 0)
 				done = -EINVAL;
@@ -208,6 +255,12 @@ static ssize_t mlx5vf_save_read(struct file *filp, char __user *buf, size_t len,
 		}
 
 		page_len = min_t(size_t, len, PAGE_SIZE - page_offset);
+		/*
+		 * In case an image is ended in the middle of the page, read
+		 * until the end of the image and manage it.
+		 */
+		page_len = min_t(size_t, page_len,
+				 mlx5vf_get_table_end_pos(migf, table) - *pos);
 		from_buff = kmap_local_page(page);
 		ret = copy_to_user(buf, from_buff + page_offset, page_len);
 		kunmap_local(from_buff);
@@ -219,6 +272,23 @@ static ssize_t mlx5vf_save_read(struct file *filp, char __user *buf, size_t len,
 		len -= page_len;
 		done += page_len;
 		buf += page_len;
+		/*
+		 * In case we moved from PRE_COPY to STOP_COPY we need to prepare
+		 * migf for final state when current state was fully transferred.
+		 * Otherwise we might miss the final table and caller may get EOF
+		 * by next read().
+		 */
+		if (migf->final_table.sgt.sgl &&
+		    *pos == mlx5vf_final_table_start_pos(migf)) {
+			mlx5vf_prep_next_table(migf);
+			table = mlx5vf_get_table(migf, pos);
+			/*
+			 * Check whether the SAVE command has finished and we
+			 * have some extra data.
+			 */
+			if (tmp == table)
+				break;
+		}
 	}
 
 out_unlock:
@@ -237,7 +307,7 @@ static __poll_t mlx5vf_save_poll(struct file *filp,
 	mutex_lock(&migf->lock);
 	if (migf->disabled || migf->is_err)
 		pollflags = EPOLLIN | EPOLLRDNORM | EPOLLRDHUP;
-	else if (READ_ONCE(migf->image_length))
+	else if (MIGF_HAS_DATA(migf))
 		pollflags = EPOLLIN | EPOLLRDNORM;
 	mutex_unlock(&migf->lock);
 
@@ -380,20 +450,34 @@ static int mlx5vf_pci_save_device_inc_data(struct mlx5vf_pci_core_device *mvdev)
 	if (ret)
 		return ret;
 
-	if (migf->is_err)
-		return -ENODEV;
-
+	mutex_lock(&migf->lock);
+	if (migf->is_err) {
+		ret = -ENODEV;
+		goto err;
+	}
+	/*
+	 * We finished transferring the current state, prepare migf for final
+	 * table. Otherwise we might miss the final table and caller may get
+	 * EOF by next read().
+	 */
+	if (migf->filp->f_pos == MIGF_TOTAL_DATA(migf))
+		mlx5vf_prep_next_table(migf);
 	ret = mlx5vf_add_migration_pages(
 		migf, DIV_ROUND_UP_ULL(length, PAGE_SIZE), &migf->final_table);
 	if (ret) {
 		mlx5vf_mark_err(migf);
-		return ret;
+		goto err;
 	}
 
+	mutex_unlock(&migf->lock);
 	ret = mlx5vf_cmd_save_vhca_state(mvdev, migf, true, false);
 	if (ret)
 		mlx5vf_mark_err(migf);
 	return ret;
+
+err:
+	mutex_unlock(&migf->lock);
+	return ret;
 }
 
 static struct mlx5_vf_migration_file *
@@ -482,7 +566,8 @@ static ssize_t mlx5vf_resume_write(struct file *filp, const char __user *buf,
 		int ret;
 
 		page_offset = (*pos) % PAGE_SIZE;
-		page = mlx5vf_get_migration_page(migf, *pos - page_offset);
+		page = mlx5vf_get_migration_page(migf, *pos - page_offset,
+						 &migf->table);
 		if (!page) {
 			if (done == 0)
 				done = -EINVAL;
Signature · 1 line
2.18.1
author maintainer
#19 lore raw

PATCH 10/13 vfio/mlx5: Introduce SW headers for migration states

In reply to Yishai Hadas

From: Shay Drory <[email protected]>

As mentioned in the previous patches, mlx5 is transferring multiple states when the PRE_COPY protocol is used. This states mechanism requires the target VM to know the states' size in order to execute multiple loads.
Therefore, add SW header, with the needed information, for each saved state the source VM is transferring to the target VM.

This patch implements the source VM handling of the headers, following patch will implement the target VM handling of the headers.

---
 drivers/vfio/pci/mlx5/cmd.h  |  7 +++++
 drivers/vfio/pci/mlx5/main.c | 50 +++++++++++++++++++++++++++++++++---
 2 files changed, 54 insertions(+), 3 deletions(-)
drivers/vfio/pci/mlx5/cmd.h +7 -0
index 07a2fc54c9d8..3b0411e4a74e 100644
@@ -22,17 +22,24 @@ struct mlx5vf_async_data {
 	void *out;
 };
 
+struct mlx5_vf_migration_header {
+	u32 image_size;
+	u32 reserved;
+};
+
 struct mlx5_vf_migration_file {
 	struct file *filp;
 	struct mutex lock;
 	u8 disabled:1;
 	u8 is_err:1;
 	u8 save_cb_active:1;
+	u8 header_read:1;
 
 	struct sg_append_table table;
 	size_t table_start_pos;
 	size_t image_length;
 	size_t allocated_length;
+	size_t sw_headers_bytes_sent;
 	/*
 	 * The device can be moved to stop_copy before the previous state was
 	 * fully read. Another set of variables is needed to maintain it.
drivers/vfio/pci/mlx5/main.c +47 -3

86 lines not on this page · raw message

Signature · 1 line
2.18.1
#20 lore raw

Re: [PATCH vfio 10/13] vfio/mlx5: Introduce SW headers for migration states

Quoted 12 lines · Yishai Hadas

From: Shay Drory <[email protected]>

As mentioned in the previous patches, mlx5 is transferring multiple states when the PRE_COPY protocol is used. This states mechanism requires the target VM to know the states' size in order to execute multiple loads.
Therefore, add SW header, with the needed information, for each saved state the source VM is transferring to the target VM.

This patch implements the source VM handling of the headers, following patch will implement the target VM handling of the headers.

Quoted 6 lines
---
 drivers/vfio/pci/mlx5/cmd.h  |  7 +++++
 drivers/vfio/pci/mlx5/main.c | 50 +++++++++++++++++++++++++++++++++---
 2 files changed, 54 insertions(+), 3 deletions(-)

This seems very awkwardly done. If you are going to string together sg_tables, then just do that consistently. Put the header into its own small sg_table as well and have a simple queue of sg_table/starting offset/length chunks to let read figure out on its own. When you start new stuff you stick it to the back of the queue. eg when you get the incremental stick on two sg_tables to the queue. when you get to the final stick on the final sg_table to the back of the queue. Whatever has or hasn't happened just works out naturally.

There are too many special cases trying to figure out which chunk is the right chunk.

drivers/vfio/pci/mlx5/main.c static int mlx5vf_release_file(struct inode *inode, struct file *filp) · line 160

Quoted 4 lines

#define MIGF_TOTAL_DATA(migf) \

drivers/vfio/pci/mlx5/main.c +2 -1
-	(migf->table_start_pos + migf->image_length + migf->final_length)
+	(migf->table_start_pos + migf->image_length + migf->final_length + \
+	 migf->sw_headers_bytes_sent)

And make all these macros static inline functions in all the patches - they don't even have proper argument bracketing..

Quoted 10 lines
drivers/vfio/pci/mlx5/main.c +10 -0
+static void mlx5vf_send_sw_header(struct mlx5_vf_migration_file *migf,
+				  loff_t *pos, char __user **buf, size_t *len,
+				  ssize_t *done)
+{
+	struct mlx5_vf_migration_header header = {};
+	size_t header_size = sizeof(header);
+	void *header_buf = &header;
+	size_t size_to_transfer;
+
+	if (*pos >= mlx5vf_final_table_start_pos(migf))

drivers/vfio/pci/mlx5/main.c static size_t mlx5vf_get_table_end_pos(struct mlx5_vf_migration_file *migf, · line 211

Quoted 10 lines
drivers/vfio/pci/mlx5/main.c +10 -0
+		header.image_size = migf->final_length;
+	else
+		header.image_size = migf->image_length;
+
+	size_to_transfer = header_size -
+			   (migf->sw_headers_bytes_sent % header_size);
+	size_to_transfer = min_t(size_t, size_to_transfer, *len);
+	header_buf += header_size - size_to_transfer;
+	if (copy_to_user(*buf, header_buf, size_to_transfer)) {
+		*done = -EFAULT;

A function that has errors should return a value, not something like this..

Jason

author maintainer
#21 lore raw

PATCH 11/13 vfio/mlx5: Introduce multiple loads

In reply to Yishai Hadas

From: Shay Drory <[email protected]>

In order to support PRE_COPY, mlx5 driver transfers multiple states (images) of the device. e.g.: the source VF can save and transfer multiple states, and the target VF will load them by that order.

This patch implements the changes for the target VF to decompose the header for each state and to write and load multiple states.

---
 drivers/vfio/pci/mlx5/cmd.c  | 12 ++---
 drivers/vfio/pci/mlx5/cmd.h  |  2 +
 drivers/vfio/pci/mlx5/main.c | 98 ++++++++++++++++++++++++++++++------
 3 files changed, 89 insertions(+), 23 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +4 -8
index 2d2171191218..a1b17cd688b9 100644
@@ -420,16 +420,14 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev,
 	if (mvdev->mdev_detach)
 		return -ENOTCONN;
 
-	mutex_lock(&migf->lock);
-	if (!migf->image_length) {
-		err = -EINVAL;
-		goto end;
-	}
+	lockdep_assert_held(&migf->lock);
+	if (!migf->image_length)
+		return -EINVAL;
 
 	mdev = mvdev->mdev;
 	err = mlx5_core_alloc_pd(mdev, &pdn);
 	if (err)
-		goto end;
+		return err;
 
 	err = dma_map_sgtable(mdev->device, &migf->table.sgt, DMA_TO_DEVICE, 0);
 	if (err)
@@ -454,8 +452,6 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev,
 	dma_unmap_sgtable(mdev->device, &migf->table.sgt, DMA_TO_DEVICE, 0);
 err_reg:
 	mlx5_core_dealloc_pd(mdev, pdn);
-end:
-	mutex_unlock(&migf->lock);
 	return err;
 }
 
drivers/vfio/pci/mlx5/cmd.h +2 -0
index 3b0411e4a74e..03f3b5e99879 100644
@@ -39,6 +39,8 @@ struct mlx5_vf_migration_file {
 	size_t table_start_pos;
 	size_t image_length;
 	size_t allocated_length;
+	size_t expected_length;
+	struct mlx5_vf_migration_header header;
 	size_t sw_headers_bytes_sent;
 	/*
 	 * The device can be moved to stop_copy before the previous state was
drivers/vfio/pci/mlx5/main.c +83 -15

165 lines not on this page · raw message

Signature · 1 line
2.18.1
#22 lore raw

Re: [PATCH vfio 11/13] vfio/mlx5: Introduce multiple loads

drivers/vfio/pci/mlx5/main.c static ssize_t mlx5vf_resume_write(struct file *filp, const char __user *buf, · line 641

Quoted 11 lines · Yishai Hadas
drivers/vfio/pci/mlx5/main.c +5 -2
+start_over:
 	if (migf->allocated_length < requested_length) {
-		done = mlx5vf_add_migration_pages(
+		ret = mlx5vf_add_migration_pages(
 			migf,
 			DIV_ROUND_UP(requested_length - migf->allocated_length,
 				     PAGE_SIZE), &migf->table);
-		if (done)
+		if (ret)
+			goto out_unlock;
+	}

This really wants to be coded as a state machine, not a tangle of gotos

drivers/vfio/pci/mlx5/main.c static ssize_t mlx5vf_resume_write(struct file *filp, const char __user *buf, · line 647

Quoted 6 lines
drivers/vfio/pci/mlx5/main.c +5 -0
+
+	if (VFIO_PRE_COPY_SUPP(migf->mvdev)) {
+		if (!migf->header_read)
+			mlx5vf_recv_sw_header(migf, pos, &buf, &len, &done);
+		if (done < 0)
 			goto out_unlock;

And when you make it into a FSM then we can pre-allocate the required sg_table space based on the header instead of having to stream

Jason

author maintainer
#23 lore raw

PATCH 12/13 vfio/mlx5: Fallback to STOP_COPY upon specific PRE_COPY error

In reply to Yishai Hadas

From: Shay Drory <[email protected]>

Before a SAVE command is issued, a QUERY command is issued in order to know the device data size.
In case PRE_COPY is used, the above commands are issued while the device is running. Thus, it is possible that between the QUERY and the SAVE commands the state of the device will be changed significantly and thus the SAVE will fail.

Currently, if a SAVE command is failing, the driver will fail the migration. In the above case, don't fail the migration, but don't allow for new SAVEs to be executed while the device is in a RUNNING state. Once the device will be moved to STOP_COPY, SAVE can be executed again and the full device state will be read.

---
 drivers/vfio/pci/mlx5/cmd.c  | 30 ++++++++++++++++++++++++++++--
 drivers/vfio/pci/mlx5/cmd.h  |  2 ++
 drivers/vfio/pci/mlx5/main.c |  7 ++++---
 3 files changed, 34 insertions(+), 5 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +28 -2
index a1b17cd688b9..ef1c141dc5e0 100644
@@ -70,9 +70,18 @@ int mlx5vf_cmd_query_vhca_migration_state(struct mlx5vf_pci_core_device *mvdev,
 	 * Running both in parallel, might end-up with a failure in the
 	 * incremental query command on un-tracked vhca.
 	 */
-	if (query_flags & MLX5VF_QUERY_INC)
+	if (query_flags & MLX5VF_QUERY_INC) {
 		wait_event(mvdev->saving_migf->save_wait,
 			   !mvdev->saving_migf->save_cb_active);
+		if (mvdev->saving_migf->precopy_err) {
+			/* In case we had a PRE_COPY error, only query full image for final image */
+			if (!(query_flags & MLX5VF_QUERY_FINAL)) {
+				*state_size = 0;
+				return 0;
+			}
+			query_flags &= ~MLX5VF_QUERY_INC;
+		}
+	}
 	MLX5_SET(query_vhca_migration_state_in, in, opcode,
 		 MLX5_CMD_OP_QUERY_VHCA_MIGRATION_STATE);
 	MLX5_SET(query_vhca_migration_state_in, in, vhca_id, mvdev->vhca_id);
@@ -291,7 +300,10 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work)
 
 	mutex_lock(&migf->lock);
 	if (async_data->status) {
-		migf->is_err = true;
+		if (async_data->status == MLX5_CMD_STAT_BAD_RES_STATE_ERR)
+			migf->precopy_err = true;
+		else
+			migf->is_err = true;
 		wake_up_interruptible(&migf->poll_wait);
 	}
 	mutex_unlock(&migf->lock);
@@ -328,6 +340,8 @@ static void mlx5vf_save_callback(int status, struct mlx5_async_work *context)
 	 * The error and the cleanup flows can't run from an
 	 * interrupt context
 	 */
+	if (status == -EREMOTEIO)
+		status = MLX5_GET(save_vhca_state_out, async_data->out, status);
 	async_data->status = status;
 	queue_work(migf->mvdev->cb_wq, &async_data->work);
 }
@@ -356,6 +370,18 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,
 	async_data = &migf->async_data;
 	async_data->sgt = (!track && inc) ? &migf->final_table.sgt :
 		&migf->table.sgt;
+	if (migf->precopy_err) {
+		/*
+		 * In case we had a PRE_COPY error, SAVE is triggered only for
+		 * the final image, read device full image.
+		 */
+		inc = false;
+		/*
+		 * Turn off precopy_err to let reader proceed only once this
+		 * SAVE call is completed, otherwise final state might be lost.
+		 */
+		migf->precopy_err = false;
+	}
 	err = dma_map_sgtable(mdev->device, async_data->sgt,
 			      DMA_FROM_DEVICE, 0);
 	if (err)
drivers/vfio/pci/mlx5/cmd.h +2 -0
index 03f3b5e99879..784670848a7c 100644
@@ -32,6 +32,7 @@ struct mlx5_vf_migration_file {
 	struct mutex lock;
 	u8 disabled:1;
 	u8 is_err:1;
+	u8 precopy_err:1;
 	u8 save_cb_active:1;
 	u8 header_read:1;
 
@@ -134,6 +135,7 @@ struct mlx5vf_pci_core_device {
 
 enum {
 	MLX5VF_QUERY_INC = (1UL << 0),
+	MLX5VF_QUERY_FINAL = (1UL << 1),
 };
 
 int mlx5vf_cmd_suspend_vhca(struct mlx5vf_pci_core_device *mvdev, u16 op_mod);
drivers/vfio/pci/mlx5/main.c +4 -3
index 6cdd4fc93818..db2d0166a0f5 100644
@@ -243,7 +243,8 @@ static ssize_t mlx5vf_save_read(struct file *filp, char __user *buf, size_t len,
 
 	if (!(filp->f_flags & O_NONBLOCK)) {
 		if (wait_event_interruptible(migf->poll_wait,
-			     (MIGF_HAS_DATA(migf) || migf->is_err)))
+			     (MIGF_HAS_DATA(migf) || migf->is_err ||
+			      migf->precopy_err)))
 			return -ERESTARTSYS;
 	}
 
@@ -351,7 +352,7 @@ static __poll_t mlx5vf_save_poll(struct file *filp,
 	mutex_lock(&migf->lock);
 	if (migf->disabled || migf->is_err)
 		pollflags = EPOLLIN | EPOLLRDNORM | EPOLLRDHUP;
-	else if (MIGF_HAS_DATA(migf))
+	else if (MIGF_HAS_DATA(migf) || migf->precopy_err)
 		pollflags = EPOLLIN | EPOLLRDNORM;
 	mutex_unlock(&migf->lock);
 
@@ -490,7 +491,7 @@ static int mlx5vf_pci_save_device_inc_data(struct mlx5vf_pci_core_device *mvdev)
 	int ret;
 
 	ret = mlx5vf_cmd_query_vhca_migration_state(mvdev, &length,
-						    MLX5VF_QUERY_INC);
+					MLX5VF_QUERY_INC | MLX5VF_QUERY_FINAL);
 	if (ret)
 		return ret;
 
Signature · 1 line
2.18.1
author maintainer
#24 lore raw

PATCH 13/13 vfio/mlx5: Enable MIGRATION_PRE_COPY flag

In reply to Yishai Hadas

From: Shay Drory <[email protected]>

Now that everything has been set up for MIGRATION_PRE_COPY, enable it.

---
 drivers/vfio/pci/mlx5/cmd.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/vfio/pci/mlx5/cmd.c +5 -0
index ef1c141dc5e0..8c3bb706f630 100644
@@ -202,6 +202,11 @@ void mlx5vf_cmd_set_migratable(struct mlx5vf_pci_core_device *mvdev,
 	if (MLX5_CAP_GEN(mvdev->mdev, adv_virtualization))
 		mvdev->core_device.vdev.log_ops = log_ops;
 
+	if (MLX5_CAP_GEN_2(mvdev->mdev, migration_multi_load) &&
+	    MLX5_CAP_GEN_2(mvdev->mdev, migration_tracking_state))
+		mvdev->core_device.vdev.migration_flags |=
+			VFIO_MIGRATION_PRE_COPY;
+
 end:
 	mlx5_vf_put_core_dev(mvdev->mdev);
 }
Signature · 1 line
2.18.1