Add migration PRE_COPY support for mlx5 driver
Subsystems (7)
Yishai Hadas
maintainercover Add migration PRE_COPY support for mlx5 driver
This series adds migration PRE_COPY uAPIs and their implementation as part of mlx5 driver.
The uAPIs follow some discussion that was done in the mailing list [1] in this area.
By the time the patches were sent, there was no driver implementation for the uAPIs, now we have it for mlx5 driver.
The optional PRE_COPY state opens the saving data transfer FD before reaching STOP_COPY and allows the device to dirty track the internal state changes with the general idea to reduce the volume of data transferred in the STOP_COPY stage.
While in PRE_COPY the device remains RUNNING, but the saving FD is open.
A new ioctl VFIO_MIG_GET_PRECOPY_INFO is provided to allow userspace to query the progress of the precopy operation in the driver with the idea it will judge to move to STOP_COPY at least once the initial data set is transferred, and possibly after the dirty size has shrunk appropriately.
User space can detect whether PRE_COPY is supported for a given device by checking the VFIO_MIGRATION_PRE_COPY flag once using the VFIO_DEVICE_FEATURE_MIGRATION ioctl.
Extra details exist as part of the specific uAPI patch from the series.
Finally, we come with mlx5 implementation based on its device specification for PRE_COPY.
To support PRE_COPY, mlx5 driver is transferring multiple states (images) of the device. e.g.: the source VF can save and transfer multiple states, and the target VF will load them by that order.
The device is saving three kinds of states:
1) Initial state - when the device moves to PRE_COPY state.
2) Middle state - during PRE_COPY phase via VFIO_MIG_GET_PRECOPY_INFO, can be multiple such states.
3) Final state - when the device moves to STOP_COPY state.
After moving to PRE_COPY state, the user is holding the saving FD and should use it for transferring the data from the source to the target while the VM is still running. From user point of view, it's a stream of data, however, from mlx5 driver point of view it includes multiple images/states. For that, it sets some headers with metadata on the source to be parsed on the target.
At some point, user may switch the device state from PRE_COPY to STOP_COPY, this will invoke saving of the final state.
As discussed earlier in the mailing list, the data that is returned as part of PRE_COPY is not required to have any bearing relative to the data size available during the STOP_COPY phase.
For this, we have the VFIO_DEVICE_FEATURE_MIG_DATA_SIZE option [2], it was sent also as part of this series as its initial patch.
In mlx5 driver we could gain with this series about 20-30 percent improvement in the downtime compared to the previous code when PRE_COPY wasn't supported.
The series includes some pre-patches to be ready for managing multiple images then it comes with the PRE_COPY implementation itself.
The matching qemu changes can be previewed here [3].
They come on top of the v2 migration protocol patches that were sent already to the mailing list.
Note:
As this series includes a net/mlx5 patch, we may need to send it as a pull request format to VFIO to avoid conflicts before acceptance.
[1] https://lore.kernel.org/kvm/[email protected]/
[2] https://patchwork.kernel.org/project/kvm/patch/[email protected]/
[3] https://github.com/avihai1122/qemu/commits/mig_v2_precopy
Yishai
Jason Gunthorpe (1):
vfio: Extend the device migration protocol with PRE_COPY
Shay Drory (9):
net/mlx5: Introduce ifc bits for pre_copy vfio/mlx5: Refactor total_length name and usage vfio/mlx5: Introduce device transitions of PRE_COPY vfio/mlx5: Introduce vfio precopy ioctl implementation vfio/mlx5: Manage read() of multiple state saves vfio/mlx5: Introduce SW headers for migration states vfio/mlx5: Introduce multiple loads
vfio/mlx5: Fallback to STOP_COPY upon specific PRE_COPY error vfio/mlx5: Enable MIGRATION_PRE_COPY flag
Yishai Hadas (3):
vfio: Add an option to get migration data size vfio/mlx5: Fix a typo in mlx5vf_cmd_load_vhca_state() vfio/mlx5: Enforce a single SAVE command at a time
.../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 9 + drivers/vfio/pci/mlx5/cmd.c | 131 +++-- drivers/vfio/pci/mlx5/cmd.h | 32 +- drivers/vfio/pci/mlx5/main.c | 495 ++++++++++++++++-- drivers/vfio/pci/vfio_pci_core.c | 3 +- drivers/vfio/vfio_main.c | 106 +++- include/linux/mlx5/mlx5_ifc.h | 14 +- include/linux/vfio.h | 5 + include/uapi/linux/vfio.h | 135 ++++- 9 files changed, 843 insertions(+), 87 deletions(-)
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH 1/13 vfio: Add an option to get migration data size
Add an option to get migration data size by introducing a new migration feature named VFIO_DEVICE_FEATURE_MIG_DATA_SIZE.
Upon VFIO_DEVICE_FEATURE_GET the estimated data length that will be required to complete STOP_COPY is returned.
This option may better enable user space to consider before moving to STOP_COPY whether it can meet the downtime SLA based on the returned data.
The patch also includes the implementation for mlx5 and hisi for this new option to make it feature complete for the existing drivers in this area.
--- .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 9 ++++++ drivers/vfio/pci/mlx5/main.c | 18 +++++++++++ drivers/vfio/pci/vfio_pci_core.c | 3 +- drivers/vfio/vfio_main.c | 32 +++++++++++++++++++ include/linux/vfio.h | 5 +++ include/uapi/linux/vfio.h | 13 ++++++++ 6 files changed, 79 insertions(+), 1 deletion(-)
drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c +9 -0
@@ -957,6 +957,14 @@ hisi_acc_vfio_pci_set_device_state(struct vfio_device *vdev,return res; } +static int +hisi_acc_vfio_pci_get_data_size (struct vfio_device *vdev, +unsigned long *stop_copy_length) +{ + *stop_copy_length =sizeof (struct acc_vf_data ); +return 0 ; +} +static int hisi_acc_vfio_pci_get_device_state (struct vfio_device *vdev,enum vfio_device_mig_state *curr_state)
@@ -1213,6 +1221,7 @@ static void hisi_acc_vfio_pci_close_device(struct vfio_device *core_vdev)static const struct vfio_migration_ops hisi_acc_vfio_pci_migrn_state_ops = { .migration_set_state = hisi_acc_vfio_pci_set_device_state, .migration_get_state = hisi_acc_vfio_pci_get_device_state, + .migration_get_data_size = hisi_acc_vfio_pci_get_data_size, };static int hisi_acc_vfio_pci_migrn_init_dev (struct vfio_device *core_vdev)
drivers/vfio/pci/mlx5/main.c +18 -0
@@ -512,6 +512,23 @@ mlx5vf_pci_set_device_state(struct vfio_device *vdev,return res; } +static int mlx5vf_pci_get_data_size (struct vfio_device *vdev, +unsigned long *stop_copy_length) +{ +struct mlx5vf_pci_core_device *mvdev =container_of ( + vdev, struct mlx5vf_pci_core_device, core_device.vdev); +size_t state_size; +int ret; + +mutex_lock (&mvdev->state_mutex); + ret =mlx5vf_cmd_query_vhca_migration_state (mvdev, + &state_size); +if (!ret) + *stop_copy_length = state_size; +mlx5vf_state_mutex_unlock (mvdev); +return ret; +} +static int mlx5vf_pci_get_device_state (struct vfio_device *vdev,enum vfio_device_mig_state *curr_state) {
@@ -577,6 +594,7 @@ static void mlx5vf_pci_close_device(struct vfio_device *core_vdev)static const struct vfio_migration_ops mlx5vf_pci_mig_ops = { .migration_set_state = mlx5vf_pci_set_device_state, .migration_get_state = mlx5vf_pci_get_device_state, + .migration_get_data_size = mlx5vf_pci_get_data_size, };static const struct vfio_log_ops mlx5vf_pci_log_ops = {
drivers/vfio/pci/vfio_pci_core.c +2 -1
@@ -2128,7 +2128,8 @@ int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev)if (vdev->vdev.mig_ops) {if (!(vdev->vdev.mig_ops->migration_get_state && - vdev->vdev.mig_ops->migration_set_state) || + vdev->vdev.mig_ops->migration_set_state && + vdev->vdev.mig_ops->migration_get_data_size) || !(vdev->vdev.migration_flags &VFIO_MIGRATION_STOP_COPY ))return -EINVAL ; }
drivers/vfio/vfio_main.c +32 -0
@@ -1256,6 +1256,34 @@ vfio_ioctl_device_feature_mig_device_state(struct vfio_device *device,return 0 ; } +static int +vfio_ioctl_device_feature_migration_data_size (struct vfio_device *device, +u32 flags,void __user *arg, +size_t argsz) +{ +struct vfio_device_feature_mig_data_size data_size = {}; +unsigned long stop_copy_length; +int ret; + +if (!device->mig_ops) +return -ENOTTY ; + + ret =vfio_check_feature (flags, argsz,VFIO_DEVICE_FEATURE_GET , +sizeof (data_size)); +if (ret !=1 ) +return ret; + + ret = device->mig_ops->migration_get_data_size (device, &stop_copy_length); +if (ret) +return ret; + + data_size.stop_copy_length = stop_copy_length; +if (copy_to_user (arg, &data_size,sizeof (data_size))) +return -EFAULT ; + +return 0 ; +} +static int vfio_ioctl_device_feature_migration (struct vfio_device *device,u32 flags,void __user *arg,size_t argsz)
@@ -1483,6 +1511,10 @@ static int vfio_ioctl_device_feature(struct vfio_device *device,return vfio_ioctl_device_feature_logging_report ( device, feature.flags, arg->data, feature.argsz - minsz); +case VFIO_DEVICE_FEATURE_MIG_DATA_SIZE : +return vfio_ioctl_device_feature_migration_data_size ( + device, feature.flags, arg->data, + feature.argsz - minsz);default :if (unlikely (!device->ops->device_feature))return -EINVAL ;
include/linux/vfio.h +5 -0
@@ -107,6 +107,9 @@ struct vfio_device_ops {* @migration_get_state: Optional callback to get the migration state for * devices that support migration. It's mandatory for * VFIO_DEVICE_FEATURE_MIGRATION migration support. +* @migration_get_data_size: Optional callback to get the estimated data +* length that will be required to complete stop copy. It's mandatory for +* VFIO_DEVICE_FEATURE_MIGRATION migration support. */ struct vfio_migration_ops { struct file *(*migration_set_state)(
@@ -114,6 +117,8 @@ struct vfio_migration_ops { enum vfio_device_mig_state new_state);int (*migration_get_state)(struct vfio_device *device,enum vfio_device_mig_state *curr_state); +int (*migration_get_data_size)(struct vfio_device *device, +unsigned long *stop_copy_length); };/**
include/uapi/linux/vfio.h +13 -0
@@ -1128,6 +1128,19 @@ struct vfio_device_feature_dma_logging_report {#define VFIO_DEVICE_FEATURE_DMA_LOGGING_REPORT 8 +/* +* Upon VFIO_DEVICE_FEATURE_GET read back the estimated data length that will +* be required to complete stop copy. +* +* Note: Can be called on each device state. +*/ + +struct vfio_device_feature_mig_data_size { +__aligned_u64 stop_copy_length; +}; + +#define VFIO_DEVICE_FEATURE_MIG_DATA_SIZE 9 +/* -------- API for Type1 VFIO IOMMU -------- */ /**
Signature · 1 line
2.18.1
Re: [PATCH vfio 01/13] vfio: Add an option to get migration data size
On 2022/11/7 1:46, Yishai Hadas Wrote:
Quoted 44 lines
Add an option to get migration data size by introducing a new migration feature named
VFIO_DEVICE_FEATURE_MIG_DATA_SIZE.Upon
VFIO_DEVICE_FEATURE_GETthe estimated data length that will be required to completeSTOP_COPYis returned.This option may better enable user space to consider before moving to
STOP_COPYwhether it can meet the downtime SLA based on the returned data.The patch also includes the implementation for mlx5 and hisi for this new option to make it feature complete for the existing drivers in this area.
Signed-off-by Yishai Hadas <[email protected]>--- .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 9 ++++++ drivers/vfio/pci/mlx5/main.c | 18 +++++++++++ drivers/vfio/pci/vfio_pci_core.c | 3 +- drivers/vfio/vfio_main.c | 32 +++++++++++++++++++ include/linux/vfio.h | 5 +++ include/uapi/linux/vfio.h | 13 ++++++++ 6 files changed, 79 insertions(+), 1 deletion(-)drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c +8
-0@@ -957,6 +957,14 @@ hisi_acc_vfio_pci_set_device_state(struct vfio_device *vdev,return res; } +static int +hisi_acc_vfio_pci_get_data_size (struct vfio_device *vdev, +unsigned long *stop_copy_length) +{ + *stop_copy_length =sizeof (struct acc_vf_data ); +return 0 ; +} +static int hisi_acc_vfio_pci_get_device_state (struct vfio_device *vdev,enum vfio_device_mig_state *curr_state)
drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c static void hisi_acc_vfio_pci_close_device(struct vfio_device *core_vdev) · line 1226
Quoted 6 lines
drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c +1
-0@@ -1213,6 +1221,7 @@ static void hisi_acc_vfio_pci_close_device(struct vfio_device *core_vdev)static const struct vfio_migration_ops hisi_acc_vfio_pci_migrn_state_ops = { .migration_set_state = hisi_acc_vfio_pci_set_device_state, .migration_get_state = hisi_acc_vfio_pci_get_device_state, + .migration_get_data_size = hisi_acc_vfio_pci_get_data_size, };
Thanks.
Quoted 149 lines
drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c +0
-0static int hisi_acc_vfio_pci_migrn_init_dev (struct vfio_device *core_vdev)drivers/vfio/pci/mlx5/main.c +18
-0@@ -512,6 +512,23 @@ mlx5vf_pci_set_device_state(struct vfio_device *vdev,return res; } +static int mlx5vf_pci_get_data_size (struct vfio_device *vdev, +unsigned long *stop_copy_length) +{ +struct mlx5vf_pci_core_device *mvdev =container_of ( + vdev, struct mlx5vf_pci_core_device, core_device.vdev); +size_t state_size; +int ret; + +mutex_lock (&mvdev->state_mutex); + ret =mlx5vf_cmd_query_vhca_migration_state (mvdev, + &state_size); +if (!ret) + *stop_copy_length = state_size; +mlx5vf_state_mutex_unlock (mvdev); +return ret; +} +static int mlx5vf_pci_get_device_state (struct vfio_device *vdev,enum vfio_device_mig_state *curr_state) {@@ -577,6 +594,7 @@ static void mlx5vf_pci_close_device(struct vfio_device *core_vdev)static const struct vfio_migration_ops mlx5vf_pci_mig_ops = { .migration_set_state = mlx5vf_pci_set_device_state, .migration_get_state = mlx5vf_pci_get_device_state, + .migration_get_data_size = mlx5vf_pci_get_data_size, };static const struct vfio_log_ops mlx5vf_pci_log_ops = {drivers/vfio/pci/vfio_pci_core.c +2
-1@@ -2128,7 +2128,8 @@ int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev)if (vdev->vdev.mig_ops) {if (!(vdev->vdev.mig_ops->migration_get_state && - vdev->vdev.mig_ops->migration_set_state) || + vdev->vdev.mig_ops->migration_set_state && + vdev->vdev.mig_ops->migration_get_data_size) || !(vdev->vdev.migration_flags &VFIO_MIGRATION_STOP_COPY ))return -EINVAL ; }drivers/vfio/vfio_main.c +32
-0@@ -1256,6 +1256,34 @@ vfio_ioctl_device_feature_mig_device_state(struct vfio_device *device,return 0 ; } +static int +vfio_ioctl_device_feature_migration_data_size (struct vfio_device *device, +u32 flags,void __user *arg, +size_t argsz) +{ +struct vfio_device_feature_mig_data_size data_size = {}; +unsigned long stop_copy_length; +int ret; + +if (!device->mig_ops) +return -ENOTTY ; + + ret =vfio_check_feature (flags, argsz,VFIO_DEVICE_FEATURE_GET , +sizeof (data_size)); +if (ret !=1 ) +return ret; + + ret = device->mig_ops->migration_get_data_size (device, &stop_copy_length); +if (ret) +return ret; + + data_size.stop_copy_length = stop_copy_length; +if (copy_to_user (arg, &data_size,sizeof (data_size))) +return -EFAULT ; + +return 0 ; +} +static int vfio_ioctl_device_feature_migration (struct vfio_device *device,u32 flags,void __user *arg,size_t argsz)@@ -1483,6 +1511,10 @@ static int vfio_ioctl_device_feature(struct vfio_device *device,return vfio_ioctl_device_feature_logging_report ( device, feature.flags, arg->data, feature.argsz - minsz); +case VFIO_DEVICE_FEATURE_MIG_DATA_SIZE : +return vfio_ioctl_device_feature_migration_data_size ( + device, feature.flags, arg->data, + feature.argsz - minsz);default :if (unlikely (!device->ops->device_feature))return -EINVAL ;include/linux/vfio.h +5
-0@@ -107,6 +107,9 @@ struct vfio_device_ops {* @migration_get_state: Optional callback to get the migration state for * devices that support migration. It's mandatory for * VFIO_DEVICE_FEATURE_MIGRATION migration support. +* @migration_get_data_size: Optional callback to get the estimated data +* length that will be required to complete stop copy. It's mandatory for +* VFIO_DEVICE_FEATURE_MIGRATION migration support. */ struct vfio_migration_ops { struct file *(*migration_set_state)(@@ -114,6 +117,8 @@ struct vfio_migration_ops { enum vfio_device_mig_state new_state);int (*migration_get_state)(struct vfio_device *device,enum vfio_device_mig_state *curr_state); +int (*migration_get_data_size)(struct vfio_device *device, +unsigned long *stop_copy_length); };/** include/uapi/linux/vfio.h +13
-0@@ -1128,6 +1128,19 @@ struct vfio_device_feature_dma_logging_report {#define VFIO_DEVICE_FEATURE_DMA_LOGGING_REPORT 8 +/* +* Upon VFIO_DEVICE_FEATURE_GET read back the estimated data length that will +* be required to complete stop copy. +* +* Note: Can be called on each device state. +*/ + +struct vfio_device_feature_mig_data_size { +__aligned_u64 stop_copy_length; +}; + +#define VFIO_DEVICE_FEATURE_MIG_DATA_SIZE 9 +/* -------- API for Type1 VFIO IOMMU -------- */ /**
Re: [PATCH vfio 01/13] vfio: Add an option to get migration data size
In reply to Yishai HadasQuoted 15 lines · Yishai Hadas
Add an option to get migration data size by introducing a new migration feature named
VFIO_DEVICE_FEATURE_MIG_DATA_SIZE.Upon
VFIO_DEVICE_FEATURE_GETthe estimated data length that will be required to completeSTOP_COPYis returned.This option may better enable user space to consider before moving to
STOP_COPYwhether it can meet the downtime SLA based on the returned data.The patch also includes the implementation for mlx5 and hisi for this new option to make it feature complete for the existing drivers in this area.
Signed-off-by Yishai Hadas <[email protected]>
Quoted 8 lines
--- .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 9 ++++++ drivers/vfio/pci/mlx5/main.c | 18 +++++++++++ drivers/vfio/pci/vfio_pci_core.c | 3 +- drivers/vfio/vfio_main.c | 32 +++++++++++++++++++ include/linux/vfio.h | 5 +++ include/uapi/linux/vfio.h | 13 ++++++++ 6 files changed, 79 insertions(+), 1 deletion(-)
Jason
Yishai Hadas
maintainerRe: [PATCH vfio 01/13] vfio: Add an option to get migration data size
Quoted 24 lines · Jason Gunthorpe
Quoted 15 lines · Yishai Hadas
Add an option to get migration data size by introducing a new migration feature named
VFIO_DEVICE_FEATURE_MIG_DATA_SIZE.Upon
VFIO_DEVICE_FEATURE_GETthe estimated data length that will be required to completeSTOP_COPYis returned.This option may better enable user space to consider before moving to
STOP_COPYwhether it can meet the downtime SLA based on the returned data.The patch also includes the implementation for mlx5 and hisi for this new option to make it feature complete for the existing drivers in this area.
Signed-off-by Yishai Hadas <[email protected]>Quoted 8 lines
--- .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 9 ++++++ drivers/vfio/pci/mlx5/main.c | 18 +++++++++++ drivers/vfio/pci/vfio_pci_core.c | 3 +- drivers/vfio/vfio_main.c | 32 +++++++++++++++++++ include/linux/vfio.h | 5 +++ include/uapi/linux/vfio.h | 13 ++++++++ 6 files changed, 79 insertions(+), 1 deletion(-)
Quoted 3 lines
Reviewed-by Jason Gunthorpe <[email protected]>Jason
Alex,
Are we fine with taking the first 2 patches from this series ?
For this one we have reviewed-by from Jason and Longfang Liu, the next patch has also a reviewed-by Jason and is very simple.
Please let me know if you want me to send them separately outside of this pre_copy series and add the mentioned reviewed-by or that you can just collect them out from the list by yourself.
Thanks,
Yishai
Re: [PATCH vfio 01/13] vfio: Add an option to get migration data size
Quoted 31 lines · On Sun, 13 Nov 2022 18:58:50 +0200 Yishai Hadas <[email protected]> wrote:
Quoted 24 lines · Jason Gunthorpe
Quoted 15 lines · Yishai Hadas
Add an option to get migration data size by introducing a new migration feature named
VFIO_DEVICE_FEATURE_MIG_DATA_SIZE.Upon
VFIO_DEVICE_FEATURE_GETthe estimated data length that will be required to completeSTOP_COPYis returned.This option may better enable user space to consider before moving to
STOP_COPYwhether it can meet the downtime SLA based on the returned data.The patch also includes the implementation for mlx5 and hisi for this new option to make it feature complete for the existing drivers in this area.
Signed-off-by Yishai Hadas <[email protected]>Quoted 8 lines
--- .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 9 ++++++ drivers/vfio/pci/mlx5/main.c | 18 +++++++++++ drivers/vfio/pci/vfio_pci_core.c | 3 +- drivers/vfio/vfio_main.c | 32 +++++++++++++++++++ include/linux/vfio.h | 5 +++ include/uapi/linux/vfio.h | 13 ++++++++ 6 files changed, 79 insertions(+), 1 deletion(-)Quoted 3 lines
Reviewed-by Jason Gunthorpe <[email protected]>Jason
Alex,
Quoted 8 lines
Are we fine with taking the first 2 patches from this series ?
For this one we have reviewed-by from Jason and Longfang Liu, the next patch has also a reviewed-by Jason and is very simple.
Please let me know if you want me to send them separately outside of this
pre_copyseries and add the mentioned reviewed-by or that you can just collect them out from the list by yourself.
Applied 1 & 2 to vfio next branch for v6.2. Thanks,
Alex
Yishai Hadas
maintainerPATCH 2/13 vfio/mlx5: Fix a typo in mlx5vf_cmd_load_vhca_state()
In reply to Yishai HadasFix a typo in mlx5vf_cmd_load_vhca_state() to use the 'load' memory layout.
As in/out sizes are equal for save and load commands there wasn't any functional issue.
--- drivers/vfio/pci/mlx5/cmd.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +2 -2
@@ -378,8 +378,8 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev, struct mlx5_vf_migration_file *migf) {struct mlx5_core_dev *mdev; -u32 out[MLX5_ST_SZ_DW (save_vhca_state_out)] = {}; -u32 in[MLX5_ST_SZ_DW (save_vhca_state_in)] = {}; +u32 out[MLX5_ST_SZ_DW (load_vhca_state_out)] = {}; +u32 in[MLX5_ST_SZ_DW (load_vhca_state_in)] = {};u32 pdn, mkey;int err;
Signature · 1 line
2.18.1
Re: [PATCH vfio 02/13] vfio/mlx5: Fix a typo in mlx5vf_cmd_load_vhca_state()
Quoted 11 lines · Yishai Hadas
Fix a typo in
mlx5vf_cmd_load_vhca_state()to use the 'load' memory layout.As in/out sizes are equal for save and load commands there wasn't any functional issue.
Fixes f1d98f346ee3 ("vfio/mlx5: Expose migration commands over mlx5 device")Signed-off-by Yishai Hadas <[email protected]>--- drivers/vfio/pci/mlx5/cmd.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Jason
Yishai Hadas
maintainerPATCH 3/13 net/mlx5: Introduce ifc bits for pre_copy
In reply to Yishai HadasFrom: Shay Drory <[email protected]>
Introduce ifc related stuff to enable PRE_COPY of VF during migration.
--- include/linux/mlx5/mlx5_ifc.h | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
include/linux/mlx5/mlx5_ifc.h +11 -3
@@ -1831,7 +1831,12 @@ struct mlx5_ifc_cmd_hca_cap_2_bits {u8 max_reformat_remove_size[0x8 ];u8 max_reformat_remove_offset[0x8 ]; -u8 reserved_at_c0[0x160 ]; +u8 reserved_at_c0[0x8 ]; +u8 migration_multi_load[0x1 ]; +u8 migration_tracking_state[0x1 ]; +u8 reserved_at_ca[0x16 ]; + +u8 reserved_at_e0[0x140 ];u8 reserved_at_220[0x1 ];u8 sw_vhca_id_valid[0x1 ];
@@ -11744,7 +11749,8 @@ struct mlx5_ifc_query_vhca_migration_state_in_bits {u8 reserved_at_20[0x10 ];u8 op_mod[0x10 ]; -u8 reserved_at_40[0x10 ]; +u8 incremental[0x1 ]; +u8 reserved_at_41[0xf ];u8 vhca_id[0x10 ];u8 reserved_at_60[0x20 ];
@@ -11770,7 +11776,9 @@ struct mlx5_ifc_save_vhca_state_in_bits {u8 reserved_at_20[0x10 ];u8 op_mod[0x10 ]; -u8 reserved_at_40[0x10 ]; +u8 incremental[0x1 ]; +u8 set_track[0x1 ]; +u8 reserved_at_42[0xe ];u8 vhca_id[0x10 ];u8 reserved_at_60[0x20 ];
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH 4/13 vfio: Extend the device migration protocol with PRE_COPY
In reply to Yishai HadasFrom: Jason Gunthorpe <[email protected]>
The optional PRE_COPY states open the saving data transfer FD before reaching STOP_COPY and allows the device to dirty track internal state changes with the general idea to reduce the volume of data transferred in the STOP_COPY stage.
While in PRE_COPY the device remains RUNNING, but the saving FD is open.
Only if the device also supports RUNNING_P2P can it support PRE_COPY_P2P, which halts P2P transfers while continuing the saving FD.
PRE_COPY, with P2P support, requires the driver to implement 7 new arcs and exists as an optional FSM branch between RUNNING and STOP_COPY:
RUNNING -> PRE_COPY -> PRE_COPY_P2P -> STOP_COPY
A new ioctl VFIO_MIG_GET_PRECOPY_INFO is provided to allow userspace to query the progress of the precopy operation in the driver with the idea it will judge to move to STOP_COPY at least once the initial data set is transferred, and possibly after the dirty size has shrunk appropriately.
This ioctl is valid only in PRE_COPY states and kernel driver should return -EINVAL from any other migration state.
Compared to the v1 clarification, STOP_COPY -> PRE_COPY is blocked and to be defined in future.
We also split the pending_bytes report into the initial and sustaining values, e.g.: initial_bytes and dirty_bytes.initial_bytes: Amount of initial mandatory precopy data.dirty_bytes: device state changes relative to data previously retrieved. These fields are not required to have any bearing to STOP_COPY phase.
--- drivers/vfio/vfio_main.c | 74 ++++++++++++++++++++++- include/uapi/linux/vfio.h | 122 ++++++++++++++++++++++++++++++++++++-- 2 files changed, 190 insertions(+), 6 deletions(-)
drivers/vfio/vfio_main.c +72 -2
@@ -1056,7 +1056,7 @@ int vfio_mig_get_next_state(struct vfio_device *device,enum vfio_device_mig_state new_fsm, enum vfio_device_mig_state *next_fsm) { - enum { VFIO_DEVICE_NUM_STATES = VFIO_DEVICE_STATE_RUNNING_P2P + 1 }; + enum { VFIO_DEVICE_NUM_STATES = VFIO_DEVICE_STATE_PRE_COPY_P2P + 1 };/* * The coding in this table requires the driver to implement the * following FSM arcs:
@@ -1071,30 +1071,65 @@ int vfio_mig_get_next_state(struct vfio_device *device,* RUNNING_P2P -> RUNNING * RUNNING_P2P -> STOP * STOP -> RUNNING_P2P -* Without P2P the driver must implement: +* +* If precopy is supported then the driver must support these additional +* FSM arcs: +* RUNNING -> PRE_COPY +* PRE_COPY -> RUNNING +* PRE_COPY -> STOP_COPY +* However, if precopy and P2P are supported together then the driver +* must support these additional arcs beyond the P2P arcs above: +* PRE_COPY -> RUNNING +* PRE_COPY -> PRE_COPY_P2P +* PRE_COPY_P2P -> PRE_COPY +* PRE_COPY_P2P -> RUNNING_P2P +* PRE_COPY_P2P -> STOP_COPY +* RUNNING -> PRE_COPY +* RUNNING_P2P -> PRE_COPY_P2P +* +* Without P2P and precopy the driver must implement: * RUNNING -> STOP * STOP -> RUNNING * * The coding will step through multiple states for some combination * transitions; if all optional features are supported, this means the * following ones: +* PRE_COPY -> PRE_COPY_P2P -> STOP_COPY +* PRE_COPY -> RUNNING -> RUNNING_P2P +* PRE_COPY -> RUNNING -> RUNNING_P2P -> STOP +* PRE_COPY -> RUNNING -> RUNNING_P2P -> STOP -> RESUMING +* PRE_COPY_P2P -> RUNNING_P2P -> RUNNING +* PRE_COPY_P2P -> RUNNING_P2P -> STOP +* PRE_COPY_P2P -> RUNNING_P2P -> STOP -> RESUMING * RESUMING -> STOP -> RUNNING_P2P +* RESUMING -> STOP -> RUNNING_P2P -> PRE_COPY_P2P * RESUMING -> STOP -> RUNNING_P2P -> RUNNING +* RESUMING -> STOP -> RUNNING_P2P -> RUNNING -> PRE_COPY * RESUMING -> STOP -> STOP_COPY +* RUNNING -> RUNNING_P2P -> PRE_COPY_P2P * RUNNING -> RUNNING_P2P -> STOP * RUNNING -> RUNNING_P2P -> STOP -> RESUMING * RUNNING -> RUNNING_P2P -> STOP -> STOP_COPY +* RUNNING_P2P -> RUNNING -> PRE_COPY * RUNNING_P2P -> STOP -> RESUMING * RUNNING_P2P -> STOP -> STOP_COPY +* STOP -> RUNNING_P2P -> PRE_COPY_P2P * STOP -> RUNNING_P2P -> RUNNING +* STOP -> RUNNING_P2P -> RUNNING -> PRE_COPY * STOP_COPY -> STOP -> RESUMING * STOP_COPY -> STOP -> RUNNING_P2P * STOP_COPY -> STOP -> RUNNING_P2P -> RUNNING +* +* The following transitions are blocked: +* STOP_COPY -> PRE_COPY +* STOP_COPY -> PRE_COPY_P2P */ static const u8 vfio_from_fsm_table[VFIO_DEVICE_NUM_STATES ][VFIO_DEVICE_NUM_STATES ] = { [VFIO_DEVICE_STATE_STOP ] = { [VFIO_DEVICE_STATE_STOP ] =VFIO_DEVICE_STATE_STOP , [VFIO_DEVICE_STATE_RUNNING ] =VFIO_DEVICE_STATE_RUNNING_P2P , + [VFIO_DEVICE_STATE_PRE_COPY ] =VFIO_DEVICE_STATE_RUNNING_P2P , + [VFIO_DEVICE_STATE_PRE_COPY_P2P ] =VFIO_DEVICE_STATE_RUNNING_P2P , [VFIO_DEVICE_STATE_STOP_COPY ] =VFIO_DEVICE_STATE_STOP_COPY , [VFIO_DEVICE_STATE_RESUMING ] =VFIO_DEVICE_STATE_RESUMING , [VFIO_DEVICE_STATE_RUNNING_P2P ] =VFIO_DEVICE_STATE_RUNNING_P2P ,
@@ -1103,14 +1138,38 @@ int vfio_mig_get_next_state(struct vfio_device *device, [VFIO_DEVICE_STATE_RUNNING] = { [VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_RUNNING_P2P, [VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_RUNNING, + [VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_PRE_COPY, + [VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_RUNNING_P2P, [VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_RUNNING_P2P, [VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_RUNNING_P2P, [VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_RUNNING_P2P, [VFIO_DEVICE_STATE_ERROR] = VFIO_DEVICE_STATE_ERROR, }, + [VFIO_DEVICE_STATE_PRE_COPY] = { + [VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_RUNNING, + [VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_RUNNING, + [VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_PRE_COPY, + [VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_PRE_COPY_P2P, + [VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_PRE_COPY_P2P, + [VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_RUNNING, + [VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_RUNNING, + [VFIO_DEVICE_STATE_ERROR] = VFIO_DEVICE_STATE_ERROR, + }, + [VFIO_DEVICE_STATE_PRE_COPY_P2P] = { + [VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_RUNNING_P2P, + [VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_RUNNING_P2P, + [VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_PRE_COPY, + [VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_PRE_COPY_P2P, + [VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_STOP_COPY, + [VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_RUNNING_P2P, + [VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_RUNNING_P2P, + [VFIO_DEVICE_STATE_ERROR] = VFIO_DEVICE_STATE_ERROR, + }, [VFIO_DEVICE_STATE_STOP_COPY] = { [VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_STOP, [VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_STOP, + [VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_ERROR, + [VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_ERROR, [VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_STOP_COPY, [VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_STOP, [VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_STOP,
@@ -1119,6 +1178,8 @@ int vfio_mig_get_next_state(struct vfio_device *device, [VFIO_DEVICE_STATE_RESUMING] = { [VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_STOP, [VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_STOP, + [VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_STOP, + [VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_STOP, [VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_STOP, [VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_RESUMING, [VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_STOP,
@@ -1127,6 +1188,8 @@ int vfio_mig_get_next_state(struct vfio_device *device, [VFIO_DEVICE_STATE_RUNNING_P2P] = { [VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_STOP, [VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_RUNNING, + [VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_RUNNING, + [VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_PRE_COPY_P2P, [VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_STOP, [VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_STOP, [VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_RUNNING_P2P,
@@ -1135,6 +1198,8 @@ int vfio_mig_get_next_state(struct vfio_device *device, [VFIO_DEVICE_STATE_ERROR] = { [VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_ERROR, [VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_ERROR, + [VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_ERROR, + [VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_ERROR, [VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_ERROR, [VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_ERROR, [VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_ERROR,
@@ -1145,6 +1210,11 @@ int vfio_mig_get_next_state(struct vfio_device *device,static const unsigned int state_flags_table[VFIO_DEVICE_NUM_STATES ] = { [VFIO_DEVICE_STATE_STOP ] =VFIO_MIGRATION_STOP_COPY , [VFIO_DEVICE_STATE_RUNNING ] =VFIO_MIGRATION_STOP_COPY , + [VFIO_DEVICE_STATE_PRE_COPY ] = +VFIO_MIGRATION_STOP_COPY |VFIO_MIGRATION_PRE_COPY , + [VFIO_DEVICE_STATE_PRE_COPY_P2P ] =VFIO_MIGRATION_STOP_COPY | +VFIO_MIGRATION_P2P | +VFIO_MIGRATION_PRE_COPY , [VFIO_DEVICE_STATE_STOP_COPY ] =VFIO_MIGRATION_STOP_COPY , [VFIO_DEVICE_STATE_RESUMING ] =VFIO_MIGRATION_STOP_COPY , [VFIO_DEVICE_STATE_RUNNING_P2P] =
include/uapi/linux/vfio.h +118 -4
191 lines not on this page · raw message
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH 5/13 vfio/mlx5: Enforce a single SAVE command at a time
In reply to Yishai HadasEnforce a single SAVE command at a time.
As the SAVE command is an asynchronous one, we must enforce running only a single command at a time.
This will preserve ordering between multiple calls and protect from races on the migration file data structure.
This is a must for the next patches from the series where as part of PRE_COPY we may have multiple images to be saved and multiple SAVE commands may be issued from different flows.
--- drivers/vfio/pci/mlx5/cmd.c | 5 +++++ drivers/vfio/pci/mlx5/cmd.h | 2 ++ drivers/vfio/pci/mlx5/main.c | 1 + 3 files changed, 8 insertions(+)
drivers/vfio/pci/mlx5/cmd.c +5 -0
@@ -281,6 +281,8 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work)dma_unmap_sgtable (mdev->device, &migf->table.sgt,DMA_FROM_DEVICE ,0 );mlx5_core_dealloc_pd (mdev, async_data->pdn);kvfree (async_data->out); + migf->save_cb_active = false; +wake_up (&migf->save_wait);fput (migf->filp); }
@@ -321,6 +323,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,return -ENOTCONN ; mdev = mvdev->mdev; +wait_event (migf->save_wait, !migf->save_cb_active); err =mlx5_core_alloc_pd (mdev, &pdn);if (err)return err;
@@ -353,6 +356,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,get_file (migf->filp); async_data->mkey = mkey; async_data->pdn = pdn; + migf->save_cb_active = true; err = mlx5_cmd_exec_cb(&migf->async_ctx, in, sizeof(in), async_data->out, out_size, mlx5vf_save_callback,
@@ -371,6 +375,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,dma_unmap_sgtable (mdev->device, &migf->table.sgt,DMA_FROM_DEVICE ,0 ); err_dma_map:mlx5_core_dealloc_pd (mdev, pdn); + migf->save_cb_active = false;return err; }
drivers/vfio/pci/mlx5/cmd.h +2 -0
@@ -26,6 +26,7 @@ struct mlx5_vf_migration_file {struct mutex lock; u8 disabled:1; u8 is_err:1; + u8 save_cb_active:1;struct sg_append_table table;size_t total_length;
@@ -37,6 +38,7 @@ struct mlx5_vf_migration_file {unsigned long last_offset;struct mlx5vf_pci_core_device *mvdev;wait_queue_head_t poll_wait; +wait_queue_head_t save_wait;struct mlx5_async_ctx async_ctx;struct mlx5vf_async_data async_data; };
drivers/vfio/pci/mlx5/main.c +1 -0
@@ -245,6 +245,7 @@ mlx5vf_pci_save_device_data(struct mlx5vf_pci_core_device *mvdev)stream_open (migf->filp->f_inode, migf->filp);mutex_init (&migf->lock);init_waitqueue_head (&migf->poll_wait); +init_waitqueue_head (&migf->save_wait);mlx5_cmd_init_async_ctx (mvdev->mdev, &migf->async_ctx);INIT_WORK (&migf->async_data.work, mlx5vf_mig_file_cleanup_cb); ret = mlx5vf_cmd_query_vhca_migration_state(mvdev,
Signature · 1 line
2.18.1
Jason Gunthorpe
reviewerRe: [PATCH vfio 05/13] vfio/mlx5: Enforce a single SAVE command at a time
Quoted 12 lines · Yishai Hadas
Enforce a single SAVE command at a time.
As the SAVE command is an asynchronous one, we must enforce running only a single command at a time.
This will preserve ordering between multiple calls and protect from races on the migration file data structure.
This is a must for the next patches from the series where as part of
PRE_COPYwe may have multiple images to be saved and multiple SAVE commands may be issued from different flows.
Quoted 6 lines
Signed-off-by Yishai Hadas <[email protected]>--- drivers/vfio/pci/mlx5/cmd.c | 5 +++++ drivers/vfio/pci/mlx5/cmd.h | 2 ++ drivers/vfio/pci/mlx5/main.c | 1 + 3 files changed, 8 insertions(+)
This should just use a 'counting completion' instead of open coding one.
drivers/vfio/pci/mlx5/cmd.c void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work) · line 285
Quoted 10 lines
drivers/vfio/pci/mlx5/cmd.c +2
-0@@ -281,6 +281,8 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work)dma_unmap_sgtable (mdev->device, &migf->table.sgt,DMA_FROM_DEVICE ,0 );mlx5_core_dealloc_pd (mdev, async_data->pdn);kvfree (async_data->out); + migf->save_cb_active = false; +wake_up (&migf->save_wait);
complete()
drivers/vfio/pci/mlx5/cmd.c int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, · line 326
Quoted 8 lines
drivers/vfio/pci/mlx5/cmd.c +1
-0fput (migf->filp); }@@ -321,6 +323,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,return -ENOTCONN ; mdev = mvdev->mdev; +wait_event (migf->save_wait, !migf->save_cb_active);
wait_for_completion_interruptible()
Quoted 12 lines
drivers/vfio/pci/mlx5/cmd.c +1
-0err =mlx5_core_alloc_pd (mdev, &pdn);if (err)return err;@@ -353,6 +356,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,get_file (migf->filp); async_data->mkey = mkey; async_data->pdn = pdn; + migf->save_cb_active = true; err = mlx5_cmd_exec_cb(&migf->async_ctx, in, sizeof(in), async_data->out, out_size, mlx5vf_save_callback,
drivers/vfio/pci/mlx5/cmd.c int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, · line 378
Quoted 4 lines
drivers/vfio/pci/mlx5/cmd.c +1
-0@@ -371,6 +375,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,dma_unmap_sgtable (mdev->device, &migf->table.sgt,DMA_FROM_DEVICE ,0 ); err_dma_map:mlx5_core_dealloc_pd (mdev, pdn); + migf->save_cb_active = false;
complete()
Quoted 28 lines
drivers/vfio/pci/mlx5/cmd.c +0
-0return err; }drivers/vfio/pci/mlx5/cmd.h +2
-0@@ -26,6 +26,7 @@ struct mlx5_vf_migration_file {struct mutex lock; u8 disabled:1; u8 is_err:1; + u8 save_cb_active:1;struct sg_append_table table;size_t total_length;@@ -37,6 +38,7 @@ struct mlx5_vf_migration_file {unsigned long last_offset;struct mlx5vf_pci_core_device *mvdev;wait_queue_head_t poll_wait; +wait_queue_head_t save_wait;struct mlx5_async_ctx async_ctx;struct mlx5vf_async_data async_data; };drivers/vfio/pci/mlx5/main.c +0
-0
drivers/vfio/pci/mlx5/main.c mlx5vf_pci_save_device_data(struct mlx5vf_pci_core_device *mvdev) · line 248
Quoted 4 lines
drivers/vfio/pci/mlx5/main.c +1
-0@@ -245,6 +245,7 @@ mlx5vf_pci_save_device_data(struct mlx5vf_pci_core_device *mvdev)stream_open (migf->filp->f_inode, migf->filp);mutex_init (&migf->lock);init_waitqueue_head (&migf->poll_wait); +init_waitqueue_head (&migf->save_wait);
init_completion()complete()
Jason
Yishai Hadas
maintainerRe: [PATCH vfio 05/13] vfio/mlx5: Enforce a single SAVE command at a time
Quoted 13 lines · Jason Gunthorpe
Quoted 12 lines · Yishai Hadas
Enforce a single SAVE command at a time.
As the SAVE command is an asynchronous one, we must enforce running only a single command at a time.
This will preserve ordering between multiple calls and protect from races on the migration file data structure.
This is a must for the next patches from the series where as part of
PRE_COPYwe may have multiple images to be saved and multiple SAVE commands may be issued from different flows.
Quoted 8 lines
Quoted 6 lines
Signed-off-by Yishai Hadas <[email protected]>--- drivers/vfio/pci/mlx5/cmd.c | 5 +++++ drivers/vfio/pci/mlx5/cmd.h | 2 ++ drivers/vfio/pci/mlx5/main.c | 1 + 3 files changed, 8 insertions(+)This should just use a 'counting completion' instead of open coding one.
Makes sense, will change accordingly as part of V1.
Yishai
Quoted 77 lines
drivers/vfio/pci/mlx5/cmd.c void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work) · line 285
Quoted 11 lines
drivers/vfio/pci/mlx5/cmd.c +2
-0@@ -281,6 +281,8 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work)dma_unmap_sgtable (mdev->device, &migf->table.sgt,DMA_FROM_DEVICE ,0 );mlx5_core_dealloc_pd (mdev, async_data->pdn);kvfree (async_data->out); + migf->save_cb_active = false; +wake_up (&migf->save_wait);
complete()drivers/vfio/pci/mlx5/cmd.c int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, · line 326
Quoted 8 lines
drivers/vfio/pci/mlx5/cmd.c +1
-0fput (migf->filp); }@@ -321,6 +323,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,return -ENOTCONN ; mdev = mvdev->mdev; +wait_event (migf->save_wait, !migf->save_cb_active);
wait_for_completion_interruptible()Quoted 12 lines
drivers/vfio/pci/mlx5/cmd.c +1
-0err =mlx5_core_alloc_pd (mdev, &pdn);if (err)return err;@@ -353,6 +356,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,get_file (migf->filp); async_data->mkey = mkey; async_data->pdn = pdn; + migf->save_cb_active = true; err = mlx5_cmd_exec_cb(&migf->async_ctx, in, sizeof(in), async_data->out, out_size, mlx5vf_save_callback,drivers/vfio/pci/mlx5/cmd.c int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, · line 378
Quoted 4 lines
drivers/vfio/pci/mlx5/cmd.c +1
-0@@ -371,6 +375,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,dma_unmap_sgtable (mdev->device, &migf->table.sgt,DMA_FROM_DEVICE ,0 ); err_dma_map:mlx5_core_dealloc_pd (mdev, pdn); + migf->save_cb_active = false;
complete()Quoted 28 lines
drivers/vfio/pci/mlx5/cmd.c +0
-0return err; }drivers/vfio/pci/mlx5/cmd.h +2
-0@@ -26,6 +26,7 @@ struct mlx5_vf_migration_file {struct mutex lock; u8 disabled:1; u8 is_err:1; + u8 save_cb_active:1;struct sg_append_table table;size_t total_length;@@ -37,6 +38,7 @@ struct mlx5_vf_migration_file {unsigned long last_offset;struct mlx5vf_pci_core_device *mvdev;wait_queue_head_t poll_wait; +wait_queue_head_t save_wait;struct mlx5_async_ctx async_ctx;struct mlx5vf_async_data async_data; };drivers/vfio/pci/mlx5/main.c +0
-0drivers/vfio/pci/mlx5/main.c mlx5vf_pci_save_device_data(struct mlx5vf_pci_core_device *mvdev) · line 248
Quoted 4 lines
drivers/vfio/pci/mlx5/main.c +1
-0@@ -245,6 +245,7 @@ mlx5vf_pci_save_device_data(struct mlx5vf_pci_core_device *mvdev)stream_open (migf->filp->f_inode, migf->filp);mutex_init (&migf->lock);init_waitqueue_head (&migf->poll_wait); +init_waitqueue_head (&migf->save_wait);
init_completion()complete()Jason
Yishai Hadas
maintainerPATCH 6/13 vfio/mlx5: Refactor total_length name and usage
In reply to Yishai HadasFrom: Shay Drory <[email protected]>
On the source side flow, change the usage of total_length to keep only the size of the image as returned from the FW and rename it to image_length. This is eliminating the usage of total_length inside the SAVE command.
This is a preparation step for the next patches from the series where more than one image could be managed on the migration file.
--- drivers/vfio/pci/mlx5/cmd.c | 24 +++++++++++------------- drivers/vfio/pci/mlx5/cmd.h | 2 +- drivers/vfio/pci/mlx5/main.c | 20 ++++++++++---------- 3 files changed, 22 insertions(+), 24 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +11 -13
@@ -212,9 +212,9 @@ static int mlx5vf_cmd_get_vhca_id(struct mlx5_core_dev *mdev, u16 function_id,static int _create_mkey (struct mlx5_core_dev *mdev,u32 pdn,struct mlx5_vf_migration_file *migf,struct mlx5_vhca_recv_buf *recv_buf, -u32 *mkey) +u32 *mkey,size_t length) { -size_t npages = migf ?DIV_ROUND_UP (migf->total_length,PAGE_SIZE ) : +size_t npages = migf ?DIV_ROUND_UP (length,PAGE_SIZE ) : recv_buf->npages;int err =0 , inlen;__be64 *mtt;
@@ -255,8 +255,7 @@ static int _create_mkey(struct mlx5_core_dev *mdev, u32 pdn,MLX5_SET (mkc, mkc, qpn,0xffffff );MLX5_SET (mkc, mkc, log_page_size,PAGE_SHIFT );MLX5_SET (mkc, mkc, translations_octword_size,DIV_ROUND_UP (npages,2 )); -MLX5_SET64 (mkc, mkc, len, - migf ? migf->total_length : (npages *PAGE_SIZE )); +MLX5_SET64 (mkc, mkc, len, migf ? length : (npages *PAGE_SIZE )); err =mlx5_core_create_mkey (mdev, mkey, in, inlen);kvfree (in);return err;
@@ -294,7 +293,7 @@ static void mlx5vf_save_callback(int status, struct mlx5_async_work *context) struct mlx5_vf_migration_file, async_data);if (!status) { -WRITE_ONCE (migf->total_length, +WRITE_ONCE (migf->image_length,MLX5_GET (save_vhca_state_out, async_data->out, actual_image_size));wake_up_interruptible (&migf->poll_wait);
@@ -333,7 +332,8 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,if (err) goto err_dma_map; - err =_create_mkey (mdev, pdn, migf,NULL , &mkey); + err =_create_mkey (mdev, pdn, migf,NULL , + &mkey, migf->allocated_length);if (err) goto err_create_mkey;
@@ -342,7 +342,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,MLX5_SET (save_vhca_state_in, in, op_mod,0 );MLX5_SET (save_vhca_state_in, in, vhca_id, mvdev->vhca_id);MLX5_SET (save_vhca_state_in, in, mkey, mkey); -MLX5_SET (save_vhca_state_in, in, size, migf->total_length); +MLX5_SET (save_vhca_state_in, in, size, migf->allocated_length); async_data = &migf->async_data; async_data->out =kvzalloc (out_size,GFP_KERNEL );
@@ -351,8 +351,6 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, goto err_out; } -/* no data exists till the callback comes back */ - migf->total_length =0 ;get_file (migf->filp); async_data->mkey = mkey; async_data->pdn = pdn;
@@ -393,7 +391,7 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev,return -ENOTCONN ;mutex_lock (&migf->lock); -if (!migf->total_length) { +if (!migf->image_length) { err = -EINVAL ; goto end; }
@@ -407,7 +405,7 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev,if (err) goto err_reg; - err =_create_mkey (mdev, pdn, migf,NULL , &mkey); + err =_create_mkey (mdev, pdn, migf,NULL , &mkey, migf->image_length);if (err) goto err_mkey;
@@ -416,7 +414,7 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev,MLX5_SET (load_vhca_state_in, in, op_mod,0 );MLX5_SET (load_vhca_state_in, in, vhca_id, mvdev->vhca_id);MLX5_SET (load_vhca_state_in, in, mkey, mkey); -MLX5_SET (load_vhca_state_in, in, size, migf->total_length); +MLX5_SET (load_vhca_state_in, in, size, migf->image_length); err =mlx5_cmd_exec_inout (mdev, load_vhca_state, in, out);
@@ -1047,7 +1045,7 @@ static int mlx5vf_alloc_qp_recv_resources(struct mlx5_core_dev *mdev,if (err) goto end; - err =_create_mkey (mdev, pdn,NULL , recv_buf, &recv_buf->mkey); + err =_create_mkey (mdev, pdn,NULL , recv_buf, &recv_buf->mkey,0 );if (err) goto err_create_mkey;
drivers/vfio/pci/mlx5/cmd.h +1 -1
@@ -29,7 +29,7 @@ struct mlx5_vf_migration_file { u8 save_cb_active:1;struct sg_append_table table; -size_t total_length; +size_t image_length;size_t allocated_length;/* Optimize mlx5vf_get_migration_page() for sequential access */
drivers/vfio/pci/mlx5/main.c +10 -10
@@ -116,7 +116,7 @@ static void mlx5vf_disable_fd(struct mlx5_vf_migration_file *migf)__free_page (sg_page_iter_page (&sg_iter));sg_free_append_table (&migf->table); migf->disabled = true; - migf->total_length =0 ; + migf->image_length =0 ; migf->allocated_length =0 ; migf->filp->f_pos =0 ;mutex_unlock (&migf->lock);
@@ -144,16 +144,16 @@ static ssize_t mlx5vf_save_read(struct file *filp, char __user *buf, size_t len,if (!(filp->f_flags &O_NONBLOCK )) {if (wait_event_interruptible (migf->poll_wait, -READ_ONCE (migf->total_length) || migf->is_err)) +READ_ONCE (migf->image_length) || migf->is_err))return -ERESTARTSYS ; }mutex_lock (&migf->lock); -if ((filp->f_flags &O_NONBLOCK ) && !READ_ONCE (migf->total_length)) { +if ((filp->f_flags &O_NONBLOCK ) && !READ_ONCE (migf->image_length)) { done = -EAGAIN ; goto out_unlock; } -if (*pos > migf->total_length) { +if (*pos > migf->image_length) { done = -EINVAL ; goto out_unlock; }
@@ -162,7 +162,7 @@ static ssize_t mlx5vf_save_read(struct file *filp, char __user *buf, size_t len, goto out_unlock; } - len =min_t (size_t, migf->total_length - *pos, len); + len =min_t (size_t, migf->image_length - *pos, len);while (len) {size_t page_offset;struct page *page;
@@ -208,7 +208,7 @@ static __poll_t mlx5vf_save_poll(struct file *filp,mutex_lock (&migf->lock);if (migf->disabled || migf->is_err) pollflags =EPOLLIN |EPOLLRDNORM |EPOLLRDHUP ; -else if (READ_ONCE (migf->total_length)) +else if (READ_ONCE (migf->image_length)) pollflags =EPOLLIN |EPOLLRDNORM ;mutex_unlock (&migf->lock);
@@ -227,6 +227,7 @@ static struct mlx5_vf_migration_file * mlx5vf_pci_save_device_data(struct mlx5vf_pci_core_device *mvdev) {struct mlx5_vf_migration_file *migf; +size_t length;int ret; migf =kzalloc (sizeof (*migf),GFP_KERNEL );
@@ -248,13 +249,12 @@ mlx5vf_pci_save_device_data(struct mlx5vf_pci_core_device *mvdev)init_waitqueue_head (&migf->save_wait);mlx5_cmd_init_async_ctx (mvdev->mdev, &migf->async_ctx);INIT_WORK (&migf->async_data.work, mlx5vf_mig_file_cleanup_cb); - ret =mlx5vf_cmd_query_vhca_migration_state (mvdev, - &migf->total_length); + ret =mlx5vf_cmd_query_vhca_migration_state (mvdev, &length);if (ret) goto out_free; ret =mlx5vf_add_migration_pages ( - migf,DIV_ROUND_UP_ULL (migf->total_length,PAGE_SIZE )); + migf,DIV_ROUND_UP_ULL (length,PAGE_SIZE ));if (ret) goto out_free;
@@ -328,7 +328,7 @@ static ssize_t mlx5vf_resume_write(struct file *filp, const char __user *buf, len -= page_len; done += page_len; buf += page_len; - migf->total_length += page_len; + migf->image_length += page_len; } out_unlock:mutex_unlock (&migf->lock);
Signature · 1 line
2.18.1
Jason Gunthorpe
reviewerRe: [PATCH vfio 06/13] vfio/mlx5: Refactor total_length name and usage
drivers/vfio/pci/mlx5/cmd.c static int mlx5vf_alloc_qp_recv_resources(struct mlx5_core_dev *mdev, · line 1048
Quoted 6 lines · Yishai Hadas
drivers/vfio/pci/mlx5/cmd.c +1
-1@@ -1047,7 +1045,7 @@ static int mlx5vf_alloc_qp_recv_resources(struct mlx5_core_dev *mdev,if (err) goto end; - err =_create_mkey (mdev, pdn,NULL , recv_buf, &recv_buf->mkey); + err =_create_mkey (mdev, pdn,NULL , recv_buf, &recv_buf->mkey,0 );
This seems pretty goofy, a 0 length means use the length in the recv_buf?
Jason
Yishai Hadas
maintainerRe: [PATCH vfio 06/13] vfio/mlx5: Refactor total_length name and usage
Quoted 11 lines · Jason Gunthorpe
drivers/vfio/pci/mlx5/cmd.c static int mlx5vf_alloc_qp_recv_resources(struct mlx5_core_dev *mdev, · line 1048
Quoted 6 lines · Yishai Hadas
drivers/vfio/pci/mlx5/cmd.c +1
-1@@ -1047,7 +1045,7 @@ static int mlx5vf_alloc_qp_recv_resources(struct mlx5_core_dev *mdev,if (err) goto end; - err =_create_mkey (mdev, pdn,NULL , recv_buf, &recv_buf->mkey); + err =_create_mkey (mdev, pdn,NULL , recv_buf, &recv_buf->mkey,0 );This seems pretty goofy, a 0 length means use the length in the
recv_buf?
Right
In V1 will embed here the expected length (i.e. npages * PAGE_SIZE) and use it directly as part of _create_mkey().
Yishai
Yishai Hadas
maintainerPATCH 7/13 vfio/mlx5: Introduce device transitions of PRE_COPY
In reply to Yishai HadasFrom: Shay Drory <[email protected]>
In order to support PRE_COPY, mlx5 driver is transferring multiple states (images) of the device. e.g.: the source VF can save and transfer multiple states, and the target VF will load them by that order.
The device is saving three kinds of states:
1) Initial state - when the device moves to PRE_COPY state.
2) Middle state - during PRE_COPY phase via VFIO_MIG_GET_PRECOPY_INFO. There can be multiple states of this type.
3) Final state - when the device moves to STOP_COPY state.
After moving to PRE_COPY state, user is holding the saving migf FD and can use it. For example: user can start transferring data via read() callback. Also, user can switch from PRE_COPY to STOP_COPY whenever he sees it fits. This will invoke saving of final state.
This means that mlx5 VFIO device can be switched to STOP_COPY without transferring any data in PRE_COPY state. Therefore, when the device moves to STOP_COPY, mlx5 will store the final state on a dedicated data structure.
--- drivers/vfio/pci/mlx5/cmd.c | 56 +++++++++++++++------- drivers/vfio/pci/mlx5/cmd.h | 16 ++++++- drivers/vfio/pci/mlx5/main.c | 93 +++++++++++++++++++++++++++++++----- 3 files changed, 134 insertions(+), 31 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +39 -17
@@ -14,6 +14,7 @@ _mlx5vf_free_page_tracker_resources(struct mlx5vf_pci_core_device *mvdev);int mlx5vf_cmd_suspend_vhca (struct mlx5vf_pci_core_device *mvdev,u16 op_mod) { +struct mlx5_vf_migration_file *migf = mvdev->saving_migf;u32 out[MLX5_ST_SZ_DW (suspend_vhca_out)] = {};u32 in[MLX5_ST_SZ_DW (suspend_vhca_in)] = {};
@@ -21,6 +22,14 @@ int mlx5vf_cmd_suspend_vhca(struct mlx5vf_pci_core_device *mvdev, u16 op_mod)if (mvdev->mdev_detach)return -ENOTCONN ; +/* +* In case PRE_COPY is used, saving_migf is exposed while the device is +* running. Make sure to run only once there is no active save command. +* Running both in parallel, might end-up with a failure in the save +* command once it will try to turn on 'tracking' on a suspended device. +*/ +if (migf) +wait_event (migf->save_wait, !migf->save_cb_active);MLX5_SET (suspend_vhca_in, in, opcode,MLX5_CMD_OP_SUSPEND_VHCA );MLX5_SET (suspend_vhca_in, in, vhca_id, mvdev->vhca_id);MLX5_SET (suspend_vhca_in, in, op_mod, op_mod);
@@ -45,7 +54,7 @@ int mlx5vf_cmd_resume_vhca(struct mlx5vf_pci_core_device *mvdev, u16 op_mod) }int mlx5vf_cmd_query_vhca_migration_state (struct mlx5vf_pci_core_device *mvdev, -size_t *state_size) +size_t *state_size,u8 query_flags) {u32 out[MLX5_ST_SZ_DW (query_vhca_migration_state_out)] = {};u32 in[MLX5_ST_SZ_DW (query_vhca_migration_state_in)] = {};
@@ -59,6 +68,8 @@ int mlx5vf_cmd_query_vhca_migration_state(struct mlx5vf_pci_core_device *mvdev, MLX5_CMD_OP_QUERY_VHCA_MIGRATION_STATE);MLX5_SET (query_vhca_migration_state_in, in, vhca_id, mvdev->vhca_id);MLX5_SET (query_vhca_migration_state_in, in, op_mod,0 ); +MLX5_SET (query_vhca_migration_state_in, in, incremental, + query_flags &MLX5VF_QUERY_INC ); ret =mlx5_cmd_exec_inout (mvdev->mdev, query_vhca_migration_state, in, out);
@@ -210,11 +221,11 @@ static int mlx5vf_cmd_get_vhca_id(struct mlx5_core_dev *mdev, u16 function_id, }static int _create_mkey (struct mlx5_core_dev *mdev,u32 pdn, -struct mlx5_vf_migration_file *migf, +struct sg_table *sgt,struct mlx5_vhca_recv_buf *recv_buf,u32 *mkey,size_t length) { -size_t npages = migf ?DIV_ROUND_UP (length,PAGE_SIZE ) : +size_t npages = sgt ?DIV_ROUND_UP (length,PAGE_SIZE ) : recv_buf->npages;int err =0 , inlen;__be64 *mtt;
@@ -232,10 +243,10 @@ static int _create_mkey(struct mlx5_core_dev *mdev, u32 pdn, DIV_ROUND_UP(npages, 2)); mtt = (__be64 *)MLX5_ADDR_OF (create_mkey_in, in, klm_pas_mtt); -if (migf) { +if (sgt) {struct sg_dma_page_iter dma_iter; -for_each_sgtable_dma_page (&migf->table.sgt, &dma_iter,0 ) +for_each_sgtable_dma_page (sgt, &dma_iter,0 ) *mtt++ =cpu_to_be64 (sg_page_iter_dma_address (&dma_iter)); }else {int i;
@@ -255,7 +266,7 @@ static int _create_mkey(struct mlx5_core_dev *mdev, u32 pdn,MLX5_SET (mkc, mkc, qpn,0xffffff );MLX5_SET (mkc, mkc, log_page_size,PAGE_SHIFT );MLX5_SET (mkc, mkc, translations_octword_size,DIV_ROUND_UP (npages,2 )); -MLX5_SET64 (mkc, mkc, len, migf ? length : (npages *PAGE_SIZE )); +MLX5_SET64 (mkc, mkc, len, sgt ? length : (npages *PAGE_SIZE )); err =mlx5_core_create_mkey (mdev, mkey, in, inlen);kvfree (in);return err;
@@ -277,7 +288,7 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work)mutex_unlock (&migf->lock);mlx5_core_destroy_mkey (mdev, async_data->mkey); -dma_unmap_sgtable (mdev->device, &migf->table.sgt,DMA_FROM_DEVICE ,0 ); +dma_unmap_sgtable (mdev->device, async_data->sgt,DMA_FROM_DEVICE ,0 );mlx5_core_dealloc_pd (mdev, async_data->pdn);kvfree (async_data->out); migf->save_cb_active = false;
@@ -293,9 +304,14 @@ static void mlx5vf_save_callback(int status, struct mlx5_async_work *context) struct mlx5_vf_migration_file, async_data);if (!status) { -WRITE_ONCE (migf->image_length, -MLX5_GET (save_vhca_state_out, async_data->out, - actual_image_size)); +size_t len =MLX5_GET (save_vhca_state_out, async_data->out, + actual_image_size); + +if (async_data->sgt == &migf->final_table.sgt) +WRITE_ONCE (migf->final_length, len); +else +WRITE_ONCE (migf->image_length, len); +wake_up_interruptible (&migf->poll_wait); }
@@ -308,7 +324,8 @@ static void mlx5vf_save_callback(int status, struct mlx5_async_work *context) }int mlx5vf_cmd_save_vhca_state (struct mlx5vf_pci_core_device *mvdev, -struct mlx5_vf_migration_file *migf) +struct mlx5_vf_migration_file *migf,bool inc, +bool track) {u32 out_size =MLX5_ST_SZ_BYTES (save_vhca_state_out);u32 in[MLX5_ST_SZ_DW (save_vhca_state_in)] = {};
@@ -327,12 +344,15 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,if (err)return err; - err =dma_map_sgtable (mdev->device, &migf->table.sgt,DMA_FROM_DEVICE , -0 ); + async_data = &migf->async_data; + async_data->sgt = (!track && inc) ? &migf->final_table.sgt : + &migf->table.sgt; + err =dma_map_sgtable (mdev->device, async_data->sgt, +DMA_FROM_DEVICE ,0 );if (err) goto err_dma_map; - err =_create_mkey (mdev, pdn, migf,NULL , + err =_create_mkey (mdev, pdn, async_data->sgt,NULL , &mkey, migf->allocated_length);if (err) goto err_create_mkey;
@@ -343,8 +363,9 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,MLX5_SET (save_vhca_state_in, in, vhca_id, mvdev->vhca_id);MLX5_SET (save_vhca_state_in, in, mkey, mkey);MLX5_SET (save_vhca_state_in, in, size, migf->allocated_length); +MLX5_SET (save_vhca_state_in, in, incremental, inc); +MLX5_SET (save_vhca_state_in, in, set_track, track); - async_data = &migf->async_data; async_data->out =kvzalloc (out_size,GFP_KERNEL );if (!async_data->out) { err = -ENOMEM ;
@@ -370,7 +391,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, err_out:mlx5_core_destroy_mkey (mdev, mkey); err_create_mkey: -dma_unmap_sgtable (mdev->device, &migf->table.sgt,DMA_FROM_DEVICE ,0 ); +dma_unmap_sgtable (mdev->device, async_data->sgt,DMA_FROM_DEVICE ,0 ); err_dma_map:mlx5_core_dealloc_pd (mdev, pdn); migf->save_cb_active = false;
@@ -405,7 +426,8 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev,if (err) goto err_reg; - err =_create_mkey (mdev, pdn, migf,NULL , &mkey, migf->image_length); + err =_create_mkey (mdev, pdn, &migf->table.sgt,NULL , &mkey, + migf->image_length);if (err) goto err_mkey;
drivers/vfio/pci/mlx5/cmd.h +14 -2
@@ -15,6 +15,7 @@struct mlx5vf_async_data {struct mlx5_async_work cb_work;struct work_struct work; +struct sg_table *sgt;int status;u32 pdn;u32 mkey;
@@ -31,6 +32,12 @@ struct mlx5_vf_migration_file {struct sg_append_table table;size_t image_length;size_t allocated_length; +/* +* The device can be moved to stop_copy before the previous state was +* fully read. Another set of variables is needed to maintain it. +*/ +size_t final_length; +struct sg_append_table final_table;/* Optimize mlx5vf_get_migration_page() for sequential access */ struct scatterlist *last_offset_sg;
@@ -115,17 +122,22 @@ struct mlx5vf_pci_core_device {struct mlx5_core_dev *mdev; }; +enum { +MLX5VF_QUERY_INC = (1UL <<0 ), +}; +int mlx5vf_cmd_suspend_vhca (struct mlx5vf_pci_core_device *mvdev,u16 op_mod);int mlx5vf_cmd_resume_vhca (struct mlx5vf_pci_core_device *mvdev,u16 op_mod);int mlx5vf_cmd_query_vhca_migration_state (struct mlx5vf_pci_core_device *mvdev, -size_t *state_size); +size_t *state_size,u8 query_flags);void mlx5vf_cmd_set_migratable (struct mlx5vf_pci_core_device *mvdev,const struct vfio_migration_ops *mig_ops,const struct vfio_log_ops *log_ops);void mlx5vf_cmd_remove_migratable (struct mlx5vf_pci_core_device *mvdev);void mlx5vf_cmd_close_migratable (struct mlx5vf_pci_core_device *mvdev);int mlx5vf_cmd_save_vhca_state (struct mlx5vf_pci_core_device *mvdev, -struct mlx5_vf_migration_file *migf); +struct mlx5_vf_migration_file *migf,bool inc, +bool track);int mlx5vf_cmd_load_vhca_state (struct mlx5vf_pci_core_device *mvdev,struct mlx5_vf_migration_file *migf);void mlx5vf_state_mutex_unlock (struct mlx5vf_pci_core_device *mvdev);
drivers/vfio/pci/mlx5/main.c +81 -12
181 lines not on this page · raw message
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH 8/13 vfio/mlx5: Introduce vfio precopy ioctl implementation
In reply to Yishai HadasFrom: Shay Drory <[email protected]>
vfio precopy ioctl returns an estimation of data available for transferring from the device.
Whenever a user is using VFIO_MIG_GET_PRECOPY_INFO, track the current state of the device, and if needed, append the dirty data to the transfer FD data. This is done by saving a middle state.
As mlx5 runs the SAVE command asynchronously, make sure to query for incremental data only once there is no active save command. Running both in parallel, might end-up with a failure in the incremental query command on un-tracked vhca.
Also, a middle state will be saved only after the previous state has finished its SAVE command and has been fully transferred, this enables to re-use the resources.
In order to map between FD position and the new saved state data, store the current FD position.
--- drivers/vfio/pci/mlx5/cmd.c | 9 +++ drivers/vfio/pci/mlx5/cmd.h | 1 + drivers/vfio/pci/mlx5/main.c | 131 +++++++++++++++++++++++++++++++++++ 3 files changed, 141 insertions(+)
drivers/vfio/pci/mlx5/cmd.c +9 -0
@@ -64,6 +64,15 @@ int mlx5vf_cmd_query_vhca_migration_state(struct mlx5vf_pci_core_device *mvdev,if (mvdev->mdev_detach)return -ENOTCONN ; +/* +* In case PRE_COPY is used, saving_migf is exposed while device is +* running. Make sure to run only once there is no active save command. +* Running both in parallel, might end-up with a failure in the +* incremental query command on un-tracked vhca. +*/ +if (query_flags &MLX5VF_QUERY_INC ) +wait_event (mvdev->saving_migf->save_wait, + !mvdev->saving_migf->save_cb_active);MLX5_SET (query_vhca_migration_state_in, in, opcode,MLX5_CMD_OP_QUERY_VHCA_MIGRATION_STATE );MLX5_SET (query_vhca_migration_state_in, in, vhca_id, mvdev->vhca_id);
drivers/vfio/pci/mlx5/cmd.h +1 -0
@@ -30,6 +30,7 @@ struct mlx5_vf_migration_file { u8 save_cb_active:1;struct sg_append_table table; +size_t table_start_pos;size_t image_length;size_t allocated_length;/*
drivers/vfio/pci/mlx5/main.c +131 -0
159 lines not on this page · raw message
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH 9/13 vfio/mlx5: Manage read() of multiple state saves
In reply to Yishai HadasFrom: Shay Drory <[email protected]>
Since all the states that were mentioned in previous patches are transferred over the same FD, on one hand, and mlx5 keeps one data structure per state, on the other hand, mlx5 needs to manage the delta between FD position and the current state (data structure) transferred.
Also, as mentioned in previous patch, user can switch VFIO device to STOP_COPY without transferring any data in PRE_COPY state. Hence, the delta management of the final state has a dedicated data structure.
--- drivers/vfio/pci/mlx5/main.c | 115 ++++++++++++++++++++++++++++++----- 1 file changed, 100 insertions(+), 15 deletions(-)
drivers/vfio/pci/mlx5/main.c +100 -15
@@ -34,7 +34,7 @@ static struct mlx5vf_pci_core_device *mlx5vf_drvdata(struct pci_dev *pdev)static struct page *mlx5vf_get_migration_page (struct mlx5_vf_migration_file *migf, -unsigned long offset) +unsigned long offset,struct sg_append_table *table) {unsigned long cur_offset =0 ;struct scatterlist *sg;
@@ -43,14 +43,14 @@ mlx5vf_get_migration_page(struct mlx5_vf_migration_file *migf,/* All accesses are sequential */ if (offset < migf->last_offset || !migf->last_offset_sg) { migf->last_offset =0 ; - migf->last_offset_sg = migf->table.sgt.sgl; + migf->last_offset_sg = table->sgt.sgl; migf->sg_last_entry =0 ; } cur_offset = migf->last_offset;for_each_sg (migf->last_offset_sg, sg, - migf->table.sgt.orig_nents - migf->sg_last_entry, i) { + table->sgt.orig_nents - migf->sg_last_entry, i) {if (offset < sg->length + cur_offset) { migf->last_offset_sg = sg; migf->sg_last_entry += i;
@@ -161,10 +161,45 @@ static int mlx5vf_release_file(struct inode *inode, struct file *filp) (mvdev->mig_state ==VFIO_DEVICE_STATE_PRE_COPY || \ mvdev->mig_state == VFIO_DEVICE_STATE_PRE_COPY_P2P) +#define VFIO_PRE_COPY_SUPP (mvdev) \ + (mvdev->core_device.vdev.migration_flags & VFIO_MIGRATION_PRE_COPY) + +#define MIGF_HAS_DATA (migf) \ + (READ_ONCE(migf->image_length) || READ_ONCE(migf->final_length)) + +static size_t +mlx5vf_final_table_start_pos (struct mlx5_vf_migration_file *migf) +{ +return MIGF_TOTAL_DATA (migf) - migf->final_length; +} + +static size_t mlx5vf_get_table_start_pos (struct mlx5_vf_migration_file *migf) +{ +return migf->table_start_pos; +} + +static size_t mlx5vf_get_table_end_pos (struct mlx5_vf_migration_file *migf, +struct sg_append_table *table) +{ +if (table == &migf->final_table) +return MIGF_TOTAL_DATA (migf); +return migf->table_start_pos + migf->image_length; +} + +static struct sg_append_table * +mlx5vf_get_table (struct mlx5_vf_migration_file *migf,loff_t *pos) +{ +if (migf->final_length && + *pos >=mlx5vf_final_table_start_pos (migf)) +return &migf->final_table; +return &migf->table; +} +static ssize_t mlx5vf_save_read (struct file *filp,char __user *buf,size_t len,loff_t *pos) {struct mlx5_vf_migration_file *migf = filp->private_data; +struct sg_append_table *table;ssize_t done =0 ;if (pos)
@@ -173,16 +208,16 @@ static ssize_t mlx5vf_save_read(struct file *filp, char __user *buf, size_t len,if (!(filp->f_flags &O_NONBLOCK )) {if (wait_event_interruptible (migf->poll_wait, -READ_ONCE (migf->image_length) || migf->is_err)) + (MIGF_HAS_DATA (migf) || migf->is_err)))return -ERESTARTSYS ; }mutex_lock (&migf->lock); -if ((filp->f_flags &O_NONBLOCK ) && !READ_ONCE (migf->image_length)) { +if ((filp->f_flags &O_NONBLOCK ) && !MIGF_HAS_DATA (migf)) { done = -EAGAIN ; goto out_unlock; } -if (*pos > migf->image_length) { +if (*pos >MIGF_TOTAL_DATA (migf)) { done = -EINVAL ; goto out_unlock; }
@@ -191,16 +226,28 @@ static ssize_t mlx5vf_save_read(struct file *filp, char __user *buf, size_t len, goto out_unlock; } - len =min_t (size_t, migf->image_length - *pos, len); +/* If we reach the end of the PRE_COPY size */ +if (MIGF_TOTAL_DATA (migf) == *pos && +VFIO_MIG_STATE_PRE_COPY (migf->mvdev)) { + done = -ENOMSG ; + goto out_unlock; + } + + len =min_t (size_t,MIGF_TOTAL_DATA (migf) - *pos, len); + table =mlx5vf_get_table (migf, pos);while (len) { +struct sg_append_table *tmp = table; +unsigned long offset;size_t page_offset;struct page *page;size_t page_len;u8 *from_buff;int ret; - page_offset = (*pos) %PAGE_SIZE ; - page =mlx5vf_get_migration_page (migf, *pos - page_offset); + offset = *pos -mlx5vf_get_table_start_pos (migf); + page_offset = offset %PAGE_SIZE ; + offset -= page_offset; + page =mlx5vf_get_migration_page (migf, offset, table);if (!page) {if (done ==0 ) done = -EINVAL ;
@@ -208,6 +255,12 @@ static ssize_t mlx5vf_save_read(struct file *filp, char __user *buf, size_t len, } page_len =min_t (size_t, len,PAGE_SIZE - page_offset); +/* +* In case an image is ended in the middle of the page, read +* until the end of the image and manage it. +*/ + page_len =min_t (size_t, page_len, +mlx5vf_get_table_end_pos (migf, table) - *pos); from_buff =kmap_local_page (page); ret =copy_to_user (buf, from_buff + page_offset, page_len);kunmap_local (from_buff);
@@ -219,6 +272,23 @@ static ssize_t mlx5vf_save_read(struct file *filp, char __user *buf, size_t len, len -= page_len; done += page_len; buf += page_len; +/* +* In case we moved from PRE_COPY to STOP_COPY we need to prepare +* migf for final state when current state was fully transferred. +* Otherwise we might miss the final table and caller may get EOF +* by next read(). +*/ +if (migf->final_table.sgt.sgl && + *pos ==mlx5vf_final_table_start_pos (migf)) { +mlx5vf_prep_next_table (migf); + table =mlx5vf_get_table (migf, pos); +/* +* Check whether the SAVE command has finished and we +* have some extra data. +*/ +if (tmp == table) +break ; + } } out_unlock:
@@ -237,7 +307,7 @@ static __poll_t mlx5vf_save_poll(struct file *filp,mutex_lock (&migf->lock);if (migf->disabled || migf->is_err) pollflags =EPOLLIN |EPOLLRDNORM |EPOLLRDHUP ; -else if (READ_ONCE (migf->image_length)) +else if (MIGF_HAS_DATA (migf)) pollflags =EPOLLIN |EPOLLRDNORM ;mutex_unlock (&migf->lock);
@@ -380,20 +450,34 @@ static int mlx5vf_pci_save_device_inc_data(struct mlx5vf_pci_core_device *mvdev)if (ret)return ret; -if (migf->is_err) -return -ENODEV ; - +mutex_lock (&migf->lock); +if (migf->is_err) { + ret = -ENODEV ; + goto err; + } +/* +* We finished transferring the current state, prepare migf for final +* table. Otherwise we might miss the final table and caller may get +* EOF by next read(). +*/ +if (migf->filp->f_pos ==MIGF_TOTAL_DATA (migf)) +mlx5vf_prep_next_table (migf); ret =mlx5vf_add_migration_pages ( migf,DIV_ROUND_UP_ULL (length,PAGE_SIZE ), &migf->final_table);if (ret) {mlx5vf_mark_err (migf); -return ret; + goto err; } +mutex_unlock (&migf->lock); ret =mlx5vf_cmd_save_vhca_state (mvdev, migf, true, false);if (ret)mlx5vf_mark_err (migf);return ret; + +err: +mutex_unlock (&migf->lock); +return ret; } static struct mlx5_vf_migration_file *
@@ -482,7 +566,8 @@ static ssize_t mlx5vf_resume_write(struct file *filp, const char __user *buf,int ret; page_offset = (*pos) %PAGE_SIZE ; - page =mlx5vf_get_migration_page (migf, *pos - page_offset); + page =mlx5vf_get_migration_page (migf, *pos - page_offset, + &migf->table);if (!page) {if (done ==0 ) done = -EINVAL ;
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH 10/13 vfio/mlx5: Introduce SW headers for migration states
In reply to Yishai HadasFrom: Shay Drory <[email protected]>
As mentioned in the previous patches, mlx5 is transferring multiple states when the PRE_COPY protocol is used. This states mechanism requires the target VM to know the states' size in order to execute multiple loads.
Therefore, add SW header, with the needed information, for each saved state the source VM is transferring to the target VM.
This patch implements the source VM handling of the headers, following patch will implement the target VM handling of the headers.
--- drivers/vfio/pci/mlx5/cmd.h | 7 +++++ drivers/vfio/pci/mlx5/main.c | 50 +++++++++++++++++++++++++++++++++--- 2 files changed, 54 insertions(+), 3 deletions(-)
drivers/vfio/pci/mlx5/cmd.h +7 -0
@@ -22,17 +22,24 @@ struct mlx5vf_async_data {void *out; }; +struct mlx5_vf_migration_header { +u32 image_size; +u32 reserved; +}; +struct mlx5_vf_migration_file {struct file *filp;struct mutex lock;u8 disabled:1 ;u8 is_err:1 ;u8 save_cb_active:1 ; +u8 header_read:1 ;struct sg_append_table table;size_t table_start_pos;size_t image_length;size_t allocated_length; +size_t sw_headers_bytes_sent;/* * The device can be moved to stop_copy before the previous state was * fully read. Another set of variables is needed to maintain it.
drivers/vfio/pci/mlx5/main.c +47 -3
86 lines not on this page · raw message
Signature · 1 line
2.18.1
Jason Gunthorpe
reviewerRe: [PATCH vfio 10/13] vfio/mlx5: Introduce SW headers for migration states
Quoted 12 lines · Yishai Hadas
From: Shay Drory <[email protected]>
As mentioned in the previous patches, mlx5 is transferring multiple states when the
PRE_COPYprotocol is used. This states mechanism requires the target VM to know the states' size in order to execute multiple loads.
Therefore, add SW header, with the needed information, for each saved state the source VM is transferring to the target VM.This patch implements the source VM handling of the headers, following patch will implement the target VM handling of the headers.
Quoted 6 lines
Signed-off-by Shay Drory <[email protected]>Signed-off-by Yishai Hadas <[email protected]>--- drivers/vfio/pci/mlx5/cmd.h | 7 +++++ drivers/vfio/pci/mlx5/main.c | 50 +++++++++++++++++++++++++++++++++--- 2 files changed, 54 insertions(+), 3 deletions(-)
This seems very awkwardly done. If you are going to string together sg_tables, then just do that consistently. Put the header into its own small sg_table as well and have a simple queue of sg_table/starting offset/length chunks to let read figure out on its own. When you start new stuff you stick it to the back of the queue. eg when you get the incremental stick on two sg_tables to the queue. when you get to the final stick on the final sg_table to the back of the queue. Whatever has or hasn't happened just works out naturally.
There are too many special cases trying to figure out which chunk is the right chunk.
drivers/vfio/pci/mlx5/main.c static int mlx5vf_release_file(struct inode *inode, struct file *filp) · line 160
Quoted 4 lines
#define
MIGF_TOTAL_DATA(migf)\drivers/vfio/pci/mlx5/main.c +2
-1- (migf->table_start_pos + migf->image_length + migf->final_length) + (migf->table_start_pos + migf->image_length + migf->final_length + \ + migf->sw_headers_bytes_sent)
And make all these macros static inline functions in all the patches - they don't even have proper argument bracketing..
Quoted 10 lines
drivers/vfio/pci/mlx5/main.c +10
-0+static void mlx5vf_send_sw_header (struct mlx5_vf_migration_file *migf, +loff_t *pos,char __user **buf,size_t *len, +ssize_t *done) +{ +struct mlx5_vf_migration_header header = {}; +size_t header_size =sizeof (header); +void *header_buf = &header; +size_t size_to_transfer; + +if (*pos >=mlx5vf_final_table_start_pos (migf))
drivers/vfio/pci/mlx5/main.c static size_t mlx5vf_get_table_end_pos(struct mlx5_vf_migration_file *migf, · line 211
Quoted 10 lines
drivers/vfio/pci/mlx5/main.c +10
-0+ header.image_size = migf->final_length; + else + header.image_size = migf->image_length; + + size_to_transfer = header_size - + (migf->sw_headers_bytes_sent % header_size); + size_to_transfer =min_t (size_t, size_to_transfer, *len); + header_buf += header_size - size_to_transfer; +if (copy_to_user (*buf, header_buf, size_to_transfer)) { + *done = -EFAULT ;
A function that has errors should return a value, not something like this..
Jason
Yishai Hadas
maintainerPATCH 11/13 vfio/mlx5: Introduce multiple loads
In reply to Yishai HadasFrom: Shay Drory <[email protected]>
In order to support PRE_COPY, mlx5 driver transfers multiple states (images) of the device. e.g.: the source VF can save and transfer multiple states, and the target VF will load them by that order.
This patch implements the changes for the target VF to decompose the header for each state and to write and load multiple states.
--- drivers/vfio/pci/mlx5/cmd.c | 12 ++--- drivers/vfio/pci/mlx5/cmd.h | 2 + drivers/vfio/pci/mlx5/main.c | 98 ++++++++++++++++++++++++++++++------ 3 files changed, 89 insertions(+), 23 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +4 -8
@@ -420,16 +420,14 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev,if (mvdev->mdev_detach)return -ENOTCONN ; -mutex_lock (&migf->lock); -if (!migf->image_length) { - err = -EINVAL ; - goto end; - } +lockdep_assert_held (&migf->lock); +if (!migf->image_length) +return -EINVAL ; mdev = mvdev->mdev; err =mlx5_core_alloc_pd (mdev, &pdn);if (err) - goto end; +return err; err =dma_map_sgtable (mdev->device, &migf->table.sgt,DMA_TO_DEVICE ,0 );if (err)
@@ -454,8 +452,6 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev,dma_unmap_sgtable (mdev->device, &migf->table.sgt,DMA_TO_DEVICE ,0 ); err_reg:mlx5_core_dealloc_pd (mdev, pdn); -end: -mutex_unlock (&migf->lock);return err; }
drivers/vfio/pci/mlx5/cmd.h +2 -0
@@ -39,6 +39,8 @@ struct mlx5_vf_migration_file {size_t table_start_pos;size_t image_length;size_t allocated_length; +size_t expected_length; +struct mlx5_vf_migration_header header;size_t sw_headers_bytes_sent;/* * The device can be moved to stop_copy before the previous state was
drivers/vfio/pci/mlx5/main.c +83 -15
165 lines not on this page · raw message
Signature · 1 line
2.18.1
Jason Gunthorpe
reviewerRe: [PATCH vfio 11/13] vfio/mlx5: Introduce multiple loads
drivers/vfio/pci/mlx5/main.c static ssize_t mlx5vf_resume_write(struct file *filp, const char __user *buf, · line 641
Quoted 11 lines · Yishai Hadas
drivers/vfio/pci/mlx5/main.c +5
-2+start_over:if (migf->allocated_length < requested_length) { - done =mlx5vf_add_migration_pages ( + ret =mlx5vf_add_migration_pages ( migf,DIV_ROUND_UP (requested_length - migf->allocated_length,PAGE_SIZE ), &migf->table); -if (done) +if (ret) + goto out_unlock; + }
This really wants to be coded as a state machine, not a tangle of gotos
drivers/vfio/pci/mlx5/main.c static ssize_t mlx5vf_resume_write(struct file *filp, const char __user *buf, · line 647
Quoted 6 lines
drivers/vfio/pci/mlx5/main.c +5
-0+ +if (VFIO_PRE_COPY_SUPP (migf->mvdev)) { +if (!migf->header_read) +mlx5vf_recv_sw_header (migf, pos, &buf, &len, &done); +if (done <0 ) goto out_unlock;
And when you make it into a FSM then we can pre-allocate the required sg_table space based on the header instead of having to stream
Jason
Yishai Hadas
maintainerPATCH 12/13 vfio/mlx5: Fallback to STOP_COPY upon specific PRE_COPY error
In reply to Yishai HadasFrom: Shay Drory <[email protected]>
Before a SAVE command is issued, a QUERY command is issued in order to know the device data size.
In case PRE_COPY is used, the above commands are issued while the device is running. Thus, it is possible that between the QUERY and the SAVE commands the state of the device will be changed significantly and thus the SAVE will fail.
Currently, if a SAVE command is failing, the driver will fail the migration. In the above case, don't fail the migration, but don't allow for new SAVEs to be executed while the device is in a RUNNING state. Once the device will be moved to STOP_COPY, SAVE can be executed again and the full device state will be read.
--- drivers/vfio/pci/mlx5/cmd.c | 30 ++++++++++++++++++++++++++++-- drivers/vfio/pci/mlx5/cmd.h | 2 ++ drivers/vfio/pci/mlx5/main.c | 7 ++++--- 3 files changed, 34 insertions(+), 5 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +28 -2
@@ -70,9 +70,18 @@ int mlx5vf_cmd_query_vhca_migration_state(struct mlx5vf_pci_core_device *mvdev,* Running both in parallel, might end-up with a failure in the * incremental query command on un-tracked vhca. */ -if (query_flags &MLX5VF_QUERY_INC ) +if (query_flags &MLX5VF_QUERY_INC ) {wait_event (mvdev->saving_migf->save_wait, !mvdev->saving_migf->save_cb_active); +if (mvdev->saving_migf->precopy_err) { +/* In case we had a PRE_COPY error, only query full image for final image */ +if (!(query_flags &MLX5VF_QUERY_FINAL )) { + *state_size =0 ; +return 0 ; + } + query_flags &= ~MLX5VF_QUERY_INC ; + } + }MLX5_SET (query_vhca_migration_state_in, in, opcode,MLX5_CMD_OP_QUERY_VHCA_MIGRATION_STATE );MLX5_SET (query_vhca_migration_state_in, in, vhca_id, mvdev->vhca_id);
@@ -291,7 +300,10 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work)mutex_lock (&migf->lock);if (async_data->status) { - migf->is_err = true; +if (async_data->status ==MLX5_CMD_STAT_BAD_RES_STATE_ERR ) + migf->precopy_err = true; +else + migf->is_err = true;wake_up_interruptible (&migf->poll_wait); }mutex_unlock (&migf->lock);
@@ -328,6 +340,8 @@ static void mlx5vf_save_callback(int status, struct mlx5_async_work *context)* The error and the cleanup flows can't run from an * interrupt context */ +if (status == -EREMOTEIO ) + status =MLX5_GET (save_vhca_state_out, async_data->out, status); async_data->status = status;queue_work (migf->mvdev->cb_wq, &async_data->work); }
@@ -356,6 +370,18 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, async_data = &migf->async_data; async_data->sgt = (!track && inc) ? &migf->final_table.sgt : &migf->table.sgt; +if (migf->precopy_err) { +/* +* In case we had a PRE_COPY error, SAVE is triggered only for +* the final image, read device full image. +*/ + inc = false; +/* +* Turn off precopy_err to let reader proceed only once this +* SAVE call is completed, otherwise final state might be lost. +*/ + migf->precopy_err = false; + } err =dma_map_sgtable (mdev->device, async_data->sgt,DMA_FROM_DEVICE ,0 );if (err)
drivers/vfio/pci/mlx5/cmd.h +2 -0
@@ -32,6 +32,7 @@ struct mlx5_vf_migration_file {struct mutex lock; u8 disabled:1; u8 is_err:1; + u8 precopy_err:1; u8 save_cb_active:1; u8 header_read:1;
@@ -134,6 +135,7 @@ struct mlx5vf_pci_core_device {enum {MLX5VF_QUERY_INC = (1UL <<0 ), +MLX5VF_QUERY_FINAL = (1UL <<1 ), };int mlx5vf_cmd_suspend_vhca (struct mlx5vf_pci_core_device *mvdev,u16 op_mod);
drivers/vfio/pci/mlx5/main.c +4 -3
@@ -243,7 +243,8 @@ static ssize_t mlx5vf_save_read(struct file *filp, char __user *buf, size_t len,if (!(filp->f_flags &O_NONBLOCK )) {if (wait_event_interruptible (migf->poll_wait, - (MIGF_HAS_DATA (migf) || migf->is_err))) + (MIGF_HAS_DATA (migf) || migf->is_err || + migf->precopy_err)))return -ERESTARTSYS ; }
@@ -351,7 +352,7 @@ static __poll_t mlx5vf_save_poll(struct file *filp,mutex_lock (&migf->lock);if (migf->disabled || migf->is_err) pollflags =EPOLLIN |EPOLLRDNORM |EPOLLRDHUP ; -else if (MIGF_HAS_DATA (migf)) +else if (MIGF_HAS_DATA (migf) || migf->precopy_err) pollflags =EPOLLIN |EPOLLRDNORM ;mutex_unlock (&migf->lock);
@@ -490,7 +491,7 @@ static int mlx5vf_pci_save_device_inc_data(struct mlx5vf_pci_core_device *mvdev)int ret; ret =mlx5vf_cmd_query_vhca_migration_state (mvdev, &length, -MLX5VF_QUERY_INC ); +MLX5VF_QUERY_INC |MLX5VF_QUERY_FINAL );if (ret)return ret;
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH 13/13 vfio/mlx5: Enable MIGRATION_PRE_COPY flag
In reply to Yishai HadasFrom: Shay Drory <[email protected]>
Now that everything has been set up for MIGRATION_PRE_COPY, enable it.
--- drivers/vfio/pci/mlx5/cmd.c | 5 +++++ 1 file changed, 5 insertions(+)
drivers/vfio/pci/mlx5/cmd.c +5 -0
@@ -202,6 +202,11 @@ void mlx5vf_cmd_set_migratable(struct mlx5vf_pci_core_device *mvdev,if (MLX5_CAP_GEN (mvdev->mdev, adv_virtualization)) mvdev->core_device.vdev.log_ops = log_ops; +if (MLX5_CAP_GEN_2 (mvdev->mdev, migration_multi_load) && +MLX5_CAP_GEN_2 (mvdev->mdev, migration_tracking_state)) + mvdev->core_device.vdev.migration_flags |= +VFIO_MIGRATION_PRE_COPY ; + end:mlx5_vf_put_core_dev (mvdev->mdev); }
Signature · 1 line
2.18.1