Add migration PRE_COPY support for mlx5 driver
Subsystems (6)
Yishai Hadas
maintainercover Add migration PRE_COPY support for mlx5 driver
This series adds migration PRE_COPY uAPIs and their implementation as part of mlx5 driver.
The uAPIs follow some discussion that was done in the mailing list [1] in this area.
By the time the patches were sent, there was no driver implementation for the uAPIs, now we have it for mlx5 driver.
The optional PRE_COPY state opens the saving data transfer FD before reaching STOP_COPY and allows the device to dirty track the internal state changes with the general idea to reduce the volume of data transferred in the STOP_COPY stage.
While in PRE_COPY the device remains RUNNING, but the saving FD is open.
A new ioctl VFIO_MIG_GET_PRECOPY_INFO is provided to allow userspace to query the progress of the precopy operation in the driver with the idea it will judge to move to STOP_COPY at least once the initial data set is transferred, and possibly after the dirty size has shrunk appropriately.
User space can detect whether PRE_COPY is supported for a given device by checking the VFIO_MIGRATION_PRE_COPY flag once using the VFIO_DEVICE_FEATURE_MIGRATION ioctl.
Extra details exist as part of the specific uAPI patch from the series.
Finally, we come with mlx5 implementation based on its device specification for PRE_COPY.
To support PRE_COPY, mlx5 driver is transferring multiple states (images) of the device. e.g.: the source VF can save and transfer multiple states, and the target VF will load them by that order.
The device is saving three kinds of states:
1) Initial state - when the device moves to PRE_COPY state.
2) Middle state - during PRE_COPY phase via VFIO_MIG_GET_PRECOPY_INFO, can be multiple such states.
3) Final state - when the device moves to STOP_COPY state.
After moving to PRE_COPY state, the user is holding the saving FD and should use it for transferring the data from the source to the target while the VM is still running. From user point of view, it's a stream of data, however, from mlx5 driver point of view it includes multiple images/states. For that, it sets some headers with metadata on the source to be parsed on the target.
At some point, user may switch the device state from PRE_COPY to STOP_COPY, this will invoke saving of the final state.
As discussed earlier in the mailing list, the data that is returned as part of PRE_COPY is not required to have any bearing relative to the data size available during the STOP_COPY phase.
For this, we have the VFIO_DEVICE_FEATURE_MIG_DATA_SIZE option.
In mlx5 driver we could gain with this series about 20-30 percent improvement in the downtime compared to the previous code when PRE_COPY wasn't supported.
The series includes some pre-patches to be ready for managing multiple images then it comes with the PRE_COPY implementation itself.
The matching qemu changes can be previewed here [2].
They come on top of the v2 migration protocol patches that were sent already to the mailing list.
Note:
As this series includes a net/mlx5 patch, we may need to send it as a pull request format to VFIO to avoid conflicts before acceptance.
[1] https://lore.kernel.org/kvm/[email protected]/
[2] https://github.com/avihai1122/qemu/commits/mig_v2_precopy
Changes from V0: https://www.spinics.net/lists/kvm/msg294247.html
Drop the first 2 patches that Alex merged already.
Refactor mlx5 implementation based on Jason's comments on V0, it includes the below:
* Refactor the PD usage to be aligned with the migration file life cycle.
* Refactor the MKEY usage to be aligned with the migration file life cycle.
* Refactor the migration file state.
* Use queue based data chunks to simplify the driver code.
* Use the FSM model on the target to simplify the driver code.
* Extend the driver pre_copy header for future use.
Yishai
Jason Gunthorpe (1):
vfio: Extend the device migration protocol with PRE_COPY
Shay Drory (3):
net/mlx5: Introduce ifc bits for pre_copy vfio/mlx5: Fallback to STOP_COPY upon specific PRE_COPY error vfio/mlx5: Enable MIGRATION_PRE_COPY flag
Yishai Hadas (10):
vfio/mlx5: Enforce a single SAVE command at a time vfio/mlx5: Refactor PD usage
vfio/mlx5: Refactor MKEY usage
vfio/mlx5: Refactor migration file state
vfio/mlx5: Refactor to use queue based data chunks vfio/mlx5: Introduce device transitions of PRE_COPY vfio/mlx5: Introduce SW headers for migration states vfio/mlx5: Introduce vfio precopy ioctl implementation vfio/mlx5: Consider temporary end of stream as part of PRE_COPY vfio/mlx5: Introduce multiple loads
drivers/vfio/pci/mlx5/cmd.c | 408 ++++++++++++++---- drivers/vfio/pci/mlx5/cmd.h | 93 ++++- drivers/vfio/pci/mlx5/main.c | 750 ++++++++++++++++++++++++++++------ drivers/vfio/vfio_main.c | 74 +++- include/linux/mlx5/mlx5_ifc.h | 14 +- include/uapi/linux/vfio.h | 122 +++++- 6 files changed, 1241 insertions(+), 220 deletions(-)
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH 1/14 net/mlx5: Introduce ifc bits for pre_copy
From: Shay Drory <[email protected]>
Introduce ifc related stuff to enable PRE_COPY of VF during migration.
--- include/linux/mlx5/mlx5_ifc.h | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
include/linux/mlx5/mlx5_ifc.h +11 -3
@@ -1882,7 +1882,12 @@ struct mlx5_ifc_cmd_hca_cap_2_bits {u8 max_reformat_remove_size[0x8 ];u8 max_reformat_remove_offset[0x8 ]; -u8 reserved_at_c0[0xe0 ]; +u8 reserved_at_c0[0x8 ]; +u8 migration_multi_load[0x1 ]; +u8 migration_tracking_state[0x1 ]; +u8 reserved_at_ca[0x16 ]; + +u8 reserved_at_e0[0xc0 ];u8 reserved_at_1a0[0xb ];u8 log_min_mkey_entity_size[0x5 ];
@@ -11918,7 +11923,8 @@ struct mlx5_ifc_query_vhca_migration_state_in_bits {u8 reserved_at_20[0x10 ];u8 op_mod[0x10 ]; -u8 reserved_at_40[0x10 ]; +u8 incremental[0x1 ]; +u8 reserved_at_41[0xf ];u8 vhca_id[0x10 ];u8 reserved_at_60[0x20 ];
@@ -11944,7 +11950,9 @@ struct mlx5_ifc_save_vhca_state_in_bits {u8 reserved_at_20[0x10 ];u8 op_mod[0x10 ]; -u8 reserved_at_40[0x10 ]; +u8 incremental[0x1 ]; +u8 set_track[0x1 ]; +u8 reserved_at_42[0xe ];u8 vhca_id[0x10 ];u8 reserved_at_60[0x20 ];
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH 2/14 vfio: Extend the device migration protocol with PRE_COPY
In reply to Yishai HadasFrom: Jason Gunthorpe <[email protected]>
The optional PRE_COPY states open the saving data transfer FD before reaching STOP_COPY and allows the device to dirty track internal state changes with the general idea to reduce the volume of data transferred in the STOP_COPY stage.
While in PRE_COPY the device remains RUNNING, but the saving FD is open.
Only if the device also supports RUNNING_P2P can it support PRE_COPY_P2P, which halts P2P transfers while continuing the saving FD.
PRE_COPY, with P2P support, requires the driver to implement 7 new arcs and exists as an optional FSM branch between RUNNING and STOP_COPY:
RUNNING -> PRE_COPY -> PRE_COPY_P2P -> STOP_COPY
A new ioctl VFIO_MIG_GET_PRECOPY_INFO is provided to allow userspace to query the progress of the precopy operation in the driver with the idea it will judge to move to STOP_COPY at least once the initial data set is transferred, and possibly after the dirty size has shrunk appropriately.
This ioctl is valid only in PRE_COPY states and kernel driver should return -EINVAL from any other migration state.
Compared to the v1 clarification, STOP_COPY -> PRE_COPY is blocked and to be defined in future.
We also split the pending_bytes report into the initial and sustaining values, e.g.: initial_bytes and dirty_bytes.initial_bytes: Amount of initial mandatory precopy data.dirty_bytes: device state changes relative to data previously retrieved. These fields are not required to have any bearing to STOP_COPY phase.
--- drivers/vfio/vfio_main.c | 74 ++++++++++++++++++++++- include/uapi/linux/vfio.h | 122 ++++++++++++++++++++++++++++++++++++-- 2 files changed, 190 insertions(+), 6 deletions(-)
drivers/vfio/vfio_main.c +72 -2
@@ -1042,7 +1042,7 @@ int vfio_mig_get_next_state(struct vfio_device *device,enum vfio_device_mig_state new_fsm, enum vfio_device_mig_state *next_fsm) { - enum { VFIO_DEVICE_NUM_STATES = VFIO_DEVICE_STATE_RUNNING_P2P + 1 }; + enum { VFIO_DEVICE_NUM_STATES = VFIO_DEVICE_STATE_PRE_COPY_P2P + 1 };/* * The coding in this table requires the driver to implement the * following FSM arcs:
@@ -1057,30 +1057,65 @@ int vfio_mig_get_next_state(struct vfio_device *device,* RUNNING_P2P -> RUNNING * RUNNING_P2P -> STOP * STOP -> RUNNING_P2P -* Without P2P the driver must implement: +* +* If precopy is supported then the driver must support these additional +* FSM arcs: +* RUNNING -> PRE_COPY +* PRE_COPY -> RUNNING +* PRE_COPY -> STOP_COPY +* However, if precopy and P2P are supported together then the driver +* must support these additional arcs beyond the P2P arcs above: +* PRE_COPY -> RUNNING +* PRE_COPY -> PRE_COPY_P2P +* PRE_COPY_P2P -> PRE_COPY +* PRE_COPY_P2P -> RUNNING_P2P +* PRE_COPY_P2P -> STOP_COPY +* RUNNING -> PRE_COPY +* RUNNING_P2P -> PRE_COPY_P2P +* +* Without P2P and precopy the driver must implement: * RUNNING -> STOP * STOP -> RUNNING * * The coding will step through multiple states for some combination * transitions; if all optional features are supported, this means the * following ones: +* PRE_COPY -> PRE_COPY_P2P -> STOP_COPY +* PRE_COPY -> RUNNING -> RUNNING_P2P +* PRE_COPY -> RUNNING -> RUNNING_P2P -> STOP +* PRE_COPY -> RUNNING -> RUNNING_P2P -> STOP -> RESUMING +* PRE_COPY_P2P -> RUNNING_P2P -> RUNNING +* PRE_COPY_P2P -> RUNNING_P2P -> STOP +* PRE_COPY_P2P -> RUNNING_P2P -> STOP -> RESUMING * RESUMING -> STOP -> RUNNING_P2P +* RESUMING -> STOP -> RUNNING_P2P -> PRE_COPY_P2P * RESUMING -> STOP -> RUNNING_P2P -> RUNNING +* RESUMING -> STOP -> RUNNING_P2P -> RUNNING -> PRE_COPY * RESUMING -> STOP -> STOP_COPY +* RUNNING -> RUNNING_P2P -> PRE_COPY_P2P * RUNNING -> RUNNING_P2P -> STOP * RUNNING -> RUNNING_P2P -> STOP -> RESUMING * RUNNING -> RUNNING_P2P -> STOP -> STOP_COPY +* RUNNING_P2P -> RUNNING -> PRE_COPY * RUNNING_P2P -> STOP -> RESUMING * RUNNING_P2P -> STOP -> STOP_COPY +* STOP -> RUNNING_P2P -> PRE_COPY_P2P * STOP -> RUNNING_P2P -> RUNNING +* STOP -> RUNNING_P2P -> RUNNING -> PRE_COPY * STOP_COPY -> STOP -> RESUMING * STOP_COPY -> STOP -> RUNNING_P2P * STOP_COPY -> STOP -> RUNNING_P2P -> RUNNING +* +* The following transitions are blocked: +* STOP_COPY -> PRE_COPY +* STOP_COPY -> PRE_COPY_P2P */ static const u8 vfio_from_fsm_table[VFIO_DEVICE_NUM_STATES ][VFIO_DEVICE_NUM_STATES ] = { [VFIO_DEVICE_STATE_STOP ] = { [VFIO_DEVICE_STATE_STOP ] =VFIO_DEVICE_STATE_STOP , [VFIO_DEVICE_STATE_RUNNING ] =VFIO_DEVICE_STATE_RUNNING_P2P , + [VFIO_DEVICE_STATE_PRE_COPY ] =VFIO_DEVICE_STATE_RUNNING_P2P , + [VFIO_DEVICE_STATE_PRE_COPY_P2P ] =VFIO_DEVICE_STATE_RUNNING_P2P , [VFIO_DEVICE_STATE_STOP_COPY ] =VFIO_DEVICE_STATE_STOP_COPY , [VFIO_DEVICE_STATE_RESUMING ] =VFIO_DEVICE_STATE_RESUMING , [VFIO_DEVICE_STATE_RUNNING_P2P ] =VFIO_DEVICE_STATE_RUNNING_P2P ,
@@ -1089,14 +1124,38 @@ int vfio_mig_get_next_state(struct vfio_device *device, [VFIO_DEVICE_STATE_RUNNING] = { [VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_RUNNING_P2P, [VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_RUNNING, + [VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_PRE_COPY, + [VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_RUNNING_P2P, [VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_RUNNING_P2P, [VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_RUNNING_P2P, [VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_RUNNING_P2P, [VFIO_DEVICE_STATE_ERROR] = VFIO_DEVICE_STATE_ERROR, }, + [VFIO_DEVICE_STATE_PRE_COPY] = { + [VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_RUNNING, + [VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_RUNNING, + [VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_PRE_COPY, + [VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_PRE_COPY_P2P, + [VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_PRE_COPY_P2P, + [VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_RUNNING, + [VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_RUNNING, + [VFIO_DEVICE_STATE_ERROR] = VFIO_DEVICE_STATE_ERROR, + }, + [VFIO_DEVICE_STATE_PRE_COPY_P2P] = { + [VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_RUNNING_P2P, + [VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_RUNNING_P2P, + [VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_PRE_COPY, + [VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_PRE_COPY_P2P, + [VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_STOP_COPY, + [VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_RUNNING_P2P, + [VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_RUNNING_P2P, + [VFIO_DEVICE_STATE_ERROR] = VFIO_DEVICE_STATE_ERROR, + }, [VFIO_DEVICE_STATE_STOP_COPY] = { [VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_STOP, [VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_STOP, + [VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_ERROR, + [VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_ERROR, [VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_STOP_COPY, [VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_STOP, [VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_STOP,
@@ -1105,6 +1164,8 @@ int vfio_mig_get_next_state(struct vfio_device *device, [VFIO_DEVICE_STATE_RESUMING] = { [VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_STOP, [VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_STOP, + [VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_STOP, + [VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_STOP, [VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_STOP, [VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_RESUMING, [VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_STOP,
@@ -1113,6 +1174,8 @@ int vfio_mig_get_next_state(struct vfio_device *device, [VFIO_DEVICE_STATE_RUNNING_P2P] = { [VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_STOP, [VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_RUNNING, + [VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_RUNNING, + [VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_PRE_COPY_P2P, [VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_STOP, [VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_STOP, [VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_RUNNING_P2P,
@@ -1121,6 +1184,8 @@ int vfio_mig_get_next_state(struct vfio_device *device, [VFIO_DEVICE_STATE_ERROR] = { [VFIO_DEVICE_STATE_STOP] = VFIO_DEVICE_STATE_ERROR, [VFIO_DEVICE_STATE_RUNNING] = VFIO_DEVICE_STATE_ERROR, + [VFIO_DEVICE_STATE_PRE_COPY] = VFIO_DEVICE_STATE_ERROR, + [VFIO_DEVICE_STATE_PRE_COPY_P2P] = VFIO_DEVICE_STATE_ERROR, [VFIO_DEVICE_STATE_STOP_COPY] = VFIO_DEVICE_STATE_ERROR, [VFIO_DEVICE_STATE_RESUMING] = VFIO_DEVICE_STATE_ERROR, [VFIO_DEVICE_STATE_RUNNING_P2P] = VFIO_DEVICE_STATE_ERROR,
@@ -1131,6 +1196,11 @@ int vfio_mig_get_next_state(struct vfio_device *device,static const unsigned int state_flags_table[VFIO_DEVICE_NUM_STATES ] = { [VFIO_DEVICE_STATE_STOP ] =VFIO_MIGRATION_STOP_COPY , [VFIO_DEVICE_STATE_RUNNING ] =VFIO_MIGRATION_STOP_COPY , + [VFIO_DEVICE_STATE_PRE_COPY ] = +VFIO_MIGRATION_STOP_COPY |VFIO_MIGRATION_PRE_COPY , + [VFIO_DEVICE_STATE_PRE_COPY_P2P ] =VFIO_MIGRATION_STOP_COPY | +VFIO_MIGRATION_P2P | +VFIO_MIGRATION_PRE_COPY , [VFIO_DEVICE_STATE_STOP_COPY ] =VFIO_MIGRATION_STOP_COPY , [VFIO_DEVICE_STATE_RESUMING ] =VFIO_MIGRATION_STOP_COPY , [VFIO_DEVICE_STATE_RUNNING_P2P] =
include/uapi/linux/vfio.h +118 -4
191 lines not on this page · raw message
Signature · 1 line
2.18.1
Re: [PATCH V1 vfio 02/14] vfio: Extend the device migration protocol with PRE_COPY
Quoted 13 lines · On Thu, 24 Nov 2022 19:39:20 +0200 Yishai Hadas <[email protected]> wrote:
+/**
+ *VFIO_MIG_GET_PRECOPY_INFO-_IO(VFIO_TYPE, VFIO_BASE + 21)
+ *
+ * This ioctl is used on the migration data FD in the precopy phase of the
+ * migration data transfer. It returns an estimate of the current data sizes
+ * remaining to be transferred. It allows the user to judge when it is
+ * appropriate to leavePRE_COPYforSTOP_COPY.
+ *
+ * This ioctl is valid only inPRE_COPYstates and kernel driver should
+ * return-EINVALfrom any other migration state.
+ *
+ * Thevfio_precopy_infodata structure returned by this ioctl provides
+ * estimates of data available from the device during thePRE_COPYstates.
Quoted 8 lines
+ * This estimate is split into two categories,
initial_bytesand
+ *dirty_bytes.
+ *
+ * Theinitial_bytesfield indicates the amount of initial mandatory precopy
+ * data available from the device. This field should have a non-zero initial
+ * value and decrease as migration data is read from the device.
+ * It is a must to leavePRE_COPYforSTOP_COPYonly after this field reach
+ * zero.
Is this actually a requirement that's compatible with current QEMU behavior? It's my impression that a user can force the migration to move to STOP_COPY at any point in time. Thanks,
Alex
Quoted 41 lines
+ *
+ * Thedirty_bytesfield tracks device state changes relative to data
+ * previously retrieved. This field starts at zero and may increase as
+ * the internal device state is modified or decrease as that modified
+ * state is read from the device.
+ *
+ * Userspace may use the combination of these fields to estimate the
+ * potential data size available during thePRE_COPYphases, as well as
+ * trends relative to the rate the device is dirtying its internal
+ * state, but these fields are not required to have any bearing relative
+ * to the data size available during theSTOP_COPYphase.
+ *
+ * Drivers have a lot of flexibility in when and what they transfer during the
+ *PRE_COPYphase, and how they report this fromVFIO_MIG_GET_PRECOPY_INFO.
+ *
+ * During pre-copy the migration data FD has a temporary "end of stream" that is
+ * reached when bothinitial_bytesanddirty_byteare zero. For instance, this
+ * may indicate that the device is idle and not currently dirtying any internal
+ * state. Whenread()is done on this temporary end of stream the kernel driver
+ * should return ENOMSG fromread(). Userspace can wait for more data (which may
+ * never come) by using poll.
+ *
+ * Once inSTOP_COPYthe migration data FD has a permanent end of stream
+ * signaled in the usual way byread()always returning 0 and poll always
+ * returning readable. ENOMSG may not be returned inSTOP_COPY. Support
+ * for this ioctl is optional.
+ *
+ * Return: 0 on success, -1 and errno set on failure.
+ */include/uapi/linux/vfio.h +7
-0+struct vfio_precopy_info { +__u32 argsz; +__u32 flags; +__aligned_u64 initial_bytes; +__aligned_u64 dirty_bytes; }; +#define VFIO_MIG_GET_PRECOPY_INFO _IO(VFIO_TYPE, VFIO_BASE + 21) +/* * Upon VFIO_DEVICE_FEATURE_SET, allow the device to be moved into a low power * state with the platform-based power management. Device use of lower power
Jason Gunthorpe
reviewerRe: [PATCH V1 vfio 02/14] vfio: Extend the device migration protocol with PRE_COPY
Quoted 25 lines · Alex Williamson
Quoted 13 lines · On Thu, 24 Nov 2022 19:39:20 +0200 Yishai Hadas <[email protected]> wrote:
+/**
+ *VFIO_MIG_GET_PRECOPY_INFO-_IO(VFIO_TYPE, VFIO_BASE + 21)
+ *
+ * This ioctl is used on the migration data FD in the precopy phase of the
+ * migration data transfer. It returns an estimate of the current data sizes
+ * remaining to be transferred. It allows the user to judge when it is
+ * appropriate to leavePRE_COPYforSTOP_COPY.
+ *
+ * This ioctl is valid only inPRE_COPYstates and kernel driver should
+ * return-EINVALfrom any other migration state.
+ *
+ * Thevfio_precopy_infodata structure returned by this ioctl provides
+ * estimates of data available from the device during thePRE_COPYstates.Quoted 8 lines
+ * This estimate is split into two categories,
initial_bytesand
+ *dirty_bytes.
+ *
+ * Theinitial_bytesfield indicates the amount of initial mandatory precopy
+ * data available from the device. This field should have a non-zero initial
+ * value and decrease as migration data is read from the device.
+ * It is a must to leavePRE_COPYforSTOP_COPYonly after this field reach
+ * zero.
Quoted 3 lines
Is this actually a requirement that's compatible with current QEMU behavior? It's my impression that a user can force the migration to move to
STOP_COPYat any point in time. Thanks,
I think it is a typo
It should be explaining that leaving PRE_COPY early will make things slower, but is not a functional problem.
Jason
Yishai Hadas
maintainerRe: [PATCH V1 vfio 02/14] vfio: Extend the device migration protocol with PRE_COPY
Quoted 28 lines · Jason Gunthorpe
Quoted 24 lines · Alex Williamson
Quoted 13 lines · On Thu, 24 Nov 2022 19:39:20 +0200 Yishai Hadas <[email protected]> wrote:
+/**
+ *VFIO_MIG_GET_PRECOPY_INFO-_IO(VFIO_TYPE, VFIO_BASE + 21)
+ *
+ * This ioctl is used on the migration data FD in the precopy phase of the
+ * migration data transfer. It returns an estimate of the current data sizes
+ * remaining to be transferred. It allows the user to judge when it is
+ * appropriate to leavePRE_COPYforSTOP_COPY.
+ *
+ * This ioctl is valid only inPRE_COPYstates and kernel driver should
+ * return-EINVALfrom any other migration state.
+ *
+ * Thevfio_precopy_infodata structure returned by this ioctl provides
+ * estimates of data available from the device during thePRE_COPYstates.Quoted 8 lines
+ * This estimate is split into two categories,
initial_bytesand
+ *dirty_bytes.
+ *
+ * Theinitial_bytesfield indicates the amount of initial mandatory precopy
+ * data available from the device. This field should have a non-zero initial
+ * value and decrease as migration data is read from the device.
+ * It is a must to leavePRE_COPYforSTOP_COPYonly after this field reach
+ * zero.Quoted 3 lines
Is this actually a requirement that's compatible with current QEMU behavior? It's my impression that a user can force the migration to move to
STOP_COPYat any point in time. Thanks,
Quoted 6 lines
I think it is a typo
It should be explaining that leaving
PRE_COPYearly will make things slower, but is not a functional problem.Jason
Right
I''ll rephrase as part of V2 as of below.
The initial_bytes field indicates the amount of initial precopy data available from the device. This field should have a non-zero initial value and decrease as migration data is read from the device. It is recommended to leave PRE_COPY for STOP_COPY only after this field reaches zero. Leaving PRE_COPY earlier might make things slower.
Yishai
Yishai Hadas
maintainerPATCH 3/14 vfio/mlx5: Enforce a single SAVE command at a time
In reply to Yishai HadasEnforce a single SAVE command at a time.
As the SAVE command is an asynchronous one, we must enforce running only a single command at a time.
This will preserve ordering between multiple calls and protect from races on the migration file data structure.
This is a must for the next patches from the series where as part of PRE_COPY we may have multiple images to be saved and multiple SAVE commands may be issued from different flows.
--- drivers/vfio/pci/mlx5/cmd.c | 6 ++++++ drivers/vfio/pci/mlx5/cmd.h | 1 + drivers/vfio/pci/mlx5/main.c | 2 ++ 3 files changed, 9 insertions(+)
drivers/vfio/pci/mlx5/cmd.c +6 -0
@@ -281,6 +281,7 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work)dma_unmap_sgtable (mdev->device, &migf->table.sgt,DMA_FROM_DEVICE ,0 );mlx5_core_dealloc_pd (mdev, async_data->pdn);kvfree (async_data->out); +complete (&migf->save_comp);fput (migf->filp); }
@@ -321,6 +322,10 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,return -ENOTCONN ; mdev = mvdev->mdev; + err =wait_for_completion_interruptible (&migf->save_comp); +if (err) +return err; + err =mlx5_core_alloc_pd (mdev, &pdn);if (err)return err;
@@ -371,6 +376,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,dma_unmap_sgtable (mdev->device, &migf->table.sgt,DMA_FROM_DEVICE ,0 ); err_dma_map:mlx5_core_dealloc_pd (mdev, pdn); +complete (&migf->save_comp);return err; }
drivers/vfio/pci/mlx5/cmd.h +1 -0
@@ -37,6 +37,7 @@ struct mlx5_vf_migration_file {unsigned long last_offset;struct mlx5vf_pci_core_device *mvdev;wait_queue_head_t poll_wait; +struct completion save_comp;struct mlx5_async_ctx async_ctx;struct mlx5vf_async_data async_data; };
drivers/vfio/pci/mlx5/main.c +2 -0
@@ -245,6 +245,8 @@ mlx5vf_pci_save_device_data(struct mlx5vf_pci_core_device *mvdev)stream_open (migf->filp->f_inode, migf->filp);mutex_init (&migf->lock);init_waitqueue_head (&migf->poll_wait); +init_completion (&migf->save_comp); +complete (&migf->save_comp);mlx5_cmd_init_async_ctx (mvdev->mdev, &migf->async_ctx);INIT_WORK (&migf->async_data.work, mlx5vf_mig_file_cleanup_cb); ret = mlx5vf_cmd_query_vhca_migration_state(mvdev,
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH 4/14 vfio/mlx5: Refactor PD usage
In reply to Yishai HadasThis patch refactors PD usage such as its life cycle will be as of the migration file instead of allocating/destroying it upon each SAVE/LOAD command.
This is a preparation step towards the PRE_COPY series where multiple images will be SAVED/LOADED and a single PD can be simply reused.
--- drivers/vfio/pci/mlx5/cmd.c | 53 ++++++++++++++++++++++++------------ drivers/vfio/pci/mlx5/cmd.h | 5 +++- drivers/vfio/pci/mlx5/main.c | 44 ++++++++++++++++++++++-------- 3 files changed, 71 insertions(+), 31 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +35 -18
@@ -279,7 +279,6 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work)mlx5_core_destroy_mkey (mdev, async_data->mkey);dma_unmap_sgtable (mdev->device, &migf->table.sgt,DMA_FROM_DEVICE ,0 ); -mlx5_core_dealloc_pd (mdev, async_data->pdn);kvfree (async_data->out);complete (&migf->save_comp);fput (migf->filp);
@@ -314,7 +313,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,u32 in[MLX5_ST_SZ_DW (save_vhca_state_in)] = {};struct mlx5vf_async_data *async_data;struct mlx5_core_dev *mdev; -u32 pdn, mkey; +u32 mkey;int err;lockdep_assert_held (&mvdev->state_mutex);
@@ -326,16 +325,12 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,if (err)return err; - err =mlx5_core_alloc_pd (mdev, &pdn); -if (err) -return err; - err =dma_map_sgtable (mdev->device, &migf->table.sgt,DMA_FROM_DEVICE ,0 );if (err) goto err_dma_map; - err =_create_mkey (mdev, pdn, migf,NULL , &mkey); + err =_create_mkey (mdev, migf->pdn, migf,NULL , &mkey);if (err) goto err_create_mkey;
@@ -357,7 +352,6 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, migf->total_length =0 ;get_file (migf->filp); async_data->mkey = mkey; - async_data->pdn = pdn; err = mlx5_cmd_exec_cb(&migf->async_ctx, in, sizeof(in), async_data->out, out_size, mlx5vf_save_callback,
@@ -375,7 +369,6 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, err_create_mkey:dma_unmap_sgtable (mdev->device, &migf->table.sgt,DMA_FROM_DEVICE ,0 ); err_dma_map: -mlx5_core_dealloc_pd (mdev, pdn);complete (&migf->save_comp);return err; }
@@ -386,7 +379,7 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev,struct mlx5_core_dev *mdev;u32 out[MLX5_ST_SZ_DW (load_vhca_state_out)] = {};u32 in[MLX5_ST_SZ_DW (load_vhca_state_in)] = {}; -u32 pdn, mkey; +u32 mkey;int err;lockdep_assert_held (&mvdev->state_mutex);
@@ -400,15 +393,11 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev, } mdev = mvdev->mdev; - err =mlx5_core_alloc_pd (mdev, &pdn); -if (err) - goto end; - err =dma_map_sgtable (mdev->device, &migf->table.sgt,DMA_TO_DEVICE ,0 );if (err) - goto err_reg; + goto end; - err =_create_mkey (mdev, pdn, migf,NULL , &mkey); + err =_create_mkey (mdev, migf->pdn, migf,NULL , &mkey);if (err) goto err_mkey;
@@ -424,13 +413,41 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev,mlx5_core_destroy_mkey (mdev, mkey); err_mkey:dma_unmap_sgtable (mdev->device, &migf->table.sgt,DMA_TO_DEVICE ,0 ); -err_reg: -mlx5_core_dealloc_pd (mdev, pdn); end:mutex_unlock (&migf->lock);return err; } +int mlx5vf_cmd_alloc_pd (struct mlx5_vf_migration_file *migf) +{ +int err; + +lockdep_assert_held (&migf->mvdev->state_mutex); +if (migf->mvdev->mdev_detach) +return -ENOTCONN ; + + err =mlx5_core_alloc_pd (migf->mvdev->mdev, &migf->pdn); +return err; +} + +void mlx5vf_cmd_dealloc_pd (struct mlx5_vf_migration_file *migf) +{ +lockdep_assert_held (&migf->mvdev->state_mutex); +if (migf->mvdev->mdev_detach) +return ; + +mlx5_core_dealloc_pd (migf->mvdev->mdev, migf->pdn); +} + +void mlx5fv_cmd_clean_migf_resources (struct mlx5_vf_migration_file *migf) +{ +lockdep_assert_held (&migf->mvdev->state_mutex); + +WARN_ON (migf->mvdev->mdev_detach); + +mlx5vf_cmd_dealloc_pd (migf); +} +static void combine_ranges (struct rb_root_cached *root,u32 cur_nodes,u32 req_nodes) {
drivers/vfio/pci/mlx5/cmd.h +4 -1
@@ -16,7 +16,6 @@ struct mlx5vf_async_data {struct mlx5_async_work cb_work;struct work_struct work;int status; -u32 pdn;u32 mkey;void *out; };
@@ -27,6 +26,7 @@ struct mlx5_vf_migration_file { u8 disabled:1; u8 is_err:1; +u32 pdn;struct sg_append_table table;size_t total_length;size_t allocated_length;
@@ -127,6 +127,9 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, struct mlx5_vf_migration_file *migf);int mlx5vf_cmd_load_vhca_state (struct mlx5vf_pci_core_device *mvdev,struct mlx5_vf_migration_file *migf); +int mlx5vf_cmd_alloc_pd (struct mlx5_vf_migration_file *migf); +void mlx5vf_cmd_dealloc_pd (struct mlx5_vf_migration_file *migf); +void mlx5fv_cmd_clean_migf_resources (struct mlx5_vf_migration_file *migf);void mlx5vf_state_mutex_unlock (struct mlx5vf_pci_core_device *mvdev);void mlx5vf_disable_fds (struct mlx5vf_pci_core_device *mvdev);void mlx5vf_mig_file_cleanup_cb (struct work_struct *_work);
drivers/vfio/pci/mlx5/main.c +32 -12
96 lines not on this page · raw message
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH 5/14 vfio/mlx5: Refactor MKEY usage
In reply to Yishai HadasThis patch refactors MKEY usage such as its life cycle will be as of the migration file instead of allocating/destroying it upon each SAVE/LOAD command.
This is a preparation step towards the PRE_COPY series where multiple images will be SAVED/LOADED.
We achieve it by having a new struct named mlx5_vhca_data_buffer which holds the mkey and its related stuff as of sg_append_table, allocated_length, etc.
The above fields were taken out from the migration file main struct, into mlx5_vhca_data_buffer dedicated struct with the proper helpers in place.
For now we have a single mlx5_vhca_data_buffer per migration file. However, in coming patches we'll have multiple of them to support multiple images.
--- drivers/vfio/pci/mlx5/cmd.c | 162 ++++++++++++++++++++++------------- drivers/vfio/pci/mlx5/cmd.h | 37 +++++--- drivers/vfio/pci/mlx5/main.c | 92 +++++++++++--------- 3 files changed, 178 insertions(+), 113 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +102 -60
@@ -210,11 +210,11 @@ static int mlx5vf_cmd_get_vhca_id(struct mlx5_core_dev *mdev, u16 function_id, }static int _create_mkey (struct mlx5_core_dev *mdev,u32 pdn, -struct mlx5_vf_migration_file *migf, +struct mlx5_vhca_data_buffer *buf,struct mlx5_vhca_recv_buf *recv_buf,u32 *mkey) { -size_t npages = migf ?DIV_ROUND_UP (migf->total_length,PAGE_SIZE ) : +size_t npages = buf ?DIV_ROUND_UP (buf->allocated_length,PAGE_SIZE ) : recv_buf->npages;int err =0 , inlen;__be64 *mtt;
@@ -232,10 +232,10 @@ static int _create_mkey(struct mlx5_core_dev *mdev, u32 pdn, DIV_ROUND_UP(npages, 2)); mtt = (__be64 *)MLX5_ADDR_OF (create_mkey_in, in, klm_pas_mtt); -if (migf) { +if (buf) {struct sg_dma_page_iter dma_iter; -for_each_sgtable_dma_page (&migf->table.sgt, &dma_iter,0 ) +for_each_sgtable_dma_page (&buf->table.sgt, &dma_iter,0 ) *mtt++ =cpu_to_be64 (sg_page_iter_dma_address (&dma_iter)); }else {int i;
@@ -255,20 +255,99 @@ static int _create_mkey(struct mlx5_core_dev *mdev, u32 pdn,MLX5_SET (mkc, mkc, qpn,0xffffff );MLX5_SET (mkc, mkc, log_page_size,PAGE_SHIFT );MLX5_SET (mkc, mkc, translations_octword_size,DIV_ROUND_UP (npages,2 )); -MLX5_SET64 (mkc, mkc, len, - migf ? migf->total_length : (npages *PAGE_SIZE )); +MLX5_SET64 (mkc, mkc, len, npages *PAGE_SIZE ); err =mlx5_core_create_mkey (mdev, mkey, in, inlen);kvfree (in);return err; } +static int mlx5vf_dma_data_buffer (struct mlx5_vhca_data_buffer *buf) +{ +struct mlx5vf_pci_core_device *mvdev = buf->migf->mvdev; +struct mlx5_core_dev *mdev = mvdev->mdev; +int ret; + +lockdep_assert_held (&mvdev->state_mutex); +if (mvdev->mdev_detach) +return -ENOTCONN ; + +if (buf->dmaed || !buf->allocated_length) +return -EINVAL ; + + ret =dma_map_sgtable (mdev->device, &buf->table.sgt, buf->dma_dir,0 ); +if (ret) +return ret; + + ret =_create_mkey (mdev, buf->migf->pdn, buf,NULL , &buf->mkey); +if (ret) + goto err; + + buf->dmaed = true; + +return 0 ; +err: +dma_unmap_sgtable (mdev->device, &buf->table.sgt, buf->dma_dir,0 ); +return ret; +} + +void mlx5vf_free_data_buffer (struct mlx5_vhca_data_buffer *buf) +{ +struct mlx5_vf_migration_file *migf = buf->migf; +struct sg_page_iter sg_iter; + +lockdep_assert_held (&migf->mvdev->state_mutex); +WARN_ON (migf->mvdev->mdev_detach); + +if (buf->dmaed) { +mlx5_core_destroy_mkey (migf->mvdev->mdev, buf->mkey); +dma_unmap_sgtable (migf->mvdev->mdev->device, &buf->table.sgt, + buf->dma_dir,0 ); + } + +/* Undo alloc_pages_bulk_array() */ +for_each_sgtable_page (&buf->table.sgt, &sg_iter,0 ) +__free_page (sg_page_iter_page (&sg_iter)); +sg_free_append_table (&buf->table); +kfree (buf); +} + +struct mlx5_vhca_data_buffer * +mlx5vf_alloc_data_buffer (struct mlx5_vf_migration_file *migf, +size_t length, +enum dma_data_direction dma_dir) +{ +struct mlx5_vhca_data_buffer *buf; +int ret; + + buf =kzalloc (sizeof (*buf),GFP_KERNEL ); +if (!buf) +return ERR_PTR (-ENOMEM ); + + buf->dma_dir = dma_dir; + buf->migf = migf; +if (length) { + ret =mlx5vf_add_migration_pages (buf, +DIV_ROUND_UP_ULL (length,PAGE_SIZE )); +if (ret) + goto end; + + ret =mlx5vf_dma_data_buffer (buf); +if (ret) + goto end; + } + +return buf; +end: +mlx5vf_free_data_buffer (buf); +return ERR_PTR (ret); +} +void mlx5vf_mig_file_cleanup_cb (struct work_struct *_work) {struct mlx5vf_async_data *async_data =container_of (_work, struct mlx5vf_async_data, work);struct mlx5_vf_migration_file *migf =container_of (async_data, struct mlx5_vf_migration_file, async_data); -struct mlx5_core_dev *mdev = migf->mvdev->mdev;mutex_lock (&migf->lock);if (async_data->status) {
@@ -276,9 +355,6 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work)wake_up_interruptible (&migf->poll_wait); }mutex_unlock (&migf->lock); - -mlx5_core_destroy_mkey (mdev, async_data->mkey); -dma_unmap_sgtable (mdev->device, &migf->table.sgt,DMA_FROM_DEVICE ,0 );kvfree (async_data->out);complete (&migf->save_comp);fput (migf->filp);
@@ -292,7 +368,7 @@ static void mlx5vf_save_callback(int status, struct mlx5_async_work *context) struct mlx5_vf_migration_file, async_data);if (!status) { -WRITE_ONCE (migf->total_length, +WRITE_ONCE (migf->buf->length,MLX5_GET (save_vhca_state_out, async_data->out, actual_image_size));wake_up_interruptible (&migf->poll_wait);
@@ -307,39 +383,28 @@ static void mlx5vf_save_callback(int status, struct mlx5_async_work *context) }int mlx5vf_cmd_save_vhca_state (struct mlx5vf_pci_core_device *mvdev, -struct mlx5_vf_migration_file *migf) +struct mlx5_vf_migration_file *migf, +struct mlx5_vhca_data_buffer *buf) {u32 out_size =MLX5_ST_SZ_BYTES (save_vhca_state_out);u32 in[MLX5_ST_SZ_DW (save_vhca_state_in)] = {};struct mlx5vf_async_data *async_data; -struct mlx5_core_dev *mdev; -u32 mkey;int err;lockdep_assert_held (&mvdev->state_mutex);if (mvdev->mdev_detach)return -ENOTCONN ; - mdev = mvdev->mdev; err =wait_for_completion_interruptible (&migf->save_comp);if (err)return err; - err =dma_map_sgtable (mdev->device, &migf->table.sgt,DMA_FROM_DEVICE , -0 ); -if (err) - goto err_dma_map; - - err =_create_mkey (mdev, migf->pdn, migf,NULL , &mkey); -if (err) - goto err_create_mkey; -MLX5_SET (save_vhca_state_in, in, opcode,MLX5_CMD_OP_SAVE_VHCA_STATE );MLX5_SET (save_vhca_state_in, in, op_mod,0 );MLX5_SET (save_vhca_state_in, in, vhca_id, mvdev->vhca_id); -MLX5_SET (save_vhca_state_in, in, mkey, mkey); -MLX5_SET (save_vhca_state_in, in, size, migf->total_length); +MLX5_SET (save_vhca_state_in, in, mkey, buf->mkey); +MLX5_SET (save_vhca_state_in, in, size, buf->allocated_length); async_data = &migf->async_data; async_data->out =kvzalloc (out_size,GFP_KERNEL );
@@ -348,10 +413,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, goto err_out; } -/* no data exists till the callback comes back */ - migf->total_length =0 ;get_file (migf->filp); - async_data->mkey = mkey; err = mlx5_cmd_exec_cb(&migf->async_ctx, in, sizeof(in), async_data->out, out_size, mlx5vf_save_callback,
@@ -365,57 +427,33 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,fput (migf->filp);kvfree (async_data->out); err_out: -mlx5_core_destroy_mkey (mdev, mkey); -err_create_mkey: -dma_unmap_sgtable (mdev->device, &migf->table.sgt,DMA_FROM_DEVICE ,0 ); -err_dma_map:complete (&migf->save_comp);return err; }int mlx5vf_cmd_load_vhca_state (struct mlx5vf_pci_core_device *mvdev, -struct mlx5_vf_migration_file *migf) +struct mlx5_vf_migration_file *migf, +struct mlx5_vhca_data_buffer *buf) { -struct mlx5_core_dev *mdev;u32 out[MLX5_ST_SZ_DW (load_vhca_state_out)] = {};u32 in[MLX5_ST_SZ_DW (load_vhca_state_in)] = {}; -u32 mkey;int err;lockdep_assert_held (&mvdev->state_mutex);if (mvdev->mdev_detach)return -ENOTCONN ; -mutex_lock (&migf->lock); -if (!migf->total_length) { - err = -EINVAL ; - goto end; - } - - mdev = mvdev->mdev; - err =dma_map_sgtable (mdev->device, &migf->table.sgt,DMA_TO_DEVICE ,0 ); + err =mlx5vf_dma_data_buffer (buf);if (err) - goto end; - - err =_create_mkey (mdev, migf->pdn, migf,NULL , &mkey); -if (err) - goto err_mkey; +return err;MLX5_SET (load_vhca_state_in, in, opcode,MLX5_CMD_OP_LOAD_VHCA_STATE );MLX5_SET (load_vhca_state_in, in, op_mod,0 );MLX5_SET (load_vhca_state_in, in, vhca_id, mvdev->vhca_id); -MLX5_SET (load_vhca_state_in, in, mkey, mkey); -MLX5_SET (load_vhca_state_in, in, size, migf->total_length); - - err =mlx5_cmd_exec_inout (mdev, load_vhca_state, in, out); - -mlx5_core_destroy_mkey (mdev, mkey); -err_mkey: -dma_unmap_sgtable (mdev->device, &migf->table.sgt,DMA_TO_DEVICE ,0 ); -end: -mutex_unlock (&migf->lock); -return err; +MLX5_SET (load_vhca_state_in, in, mkey, buf->mkey); +MLX5_SET (load_vhca_state_in, in, size, buf->length); +return mlx5_cmd_exec_inout (mvdev->mdev, load_vhca_state, in, out); }int mlx5vf_cmd_alloc_pd (struct mlx5_vf_migration_file *migf)
@@ -445,6 +483,10 @@ void mlx5fv_cmd_clean_migf_resources(struct mlx5_vf_migration_file *migf)WARN_ON (migf->mvdev->mdev_detach); +if (migf->buf) { +mlx5vf_free_data_buffer (migf->buf); + migf->buf =NULL ; + }mlx5vf_cmd_dealloc_pd (migf); }
drivers/vfio/pci/mlx5/cmd.h +26 -11
@@ -12,11 +12,25 @@#include <linux/mlx5/cq.h> #include <linux/mlx5/qp.h> +struct mlx5_vhca_data_buffer { +struct sg_append_table table; +loff_t start_pos; +u64 length; +u64 allocated_length; +u32 mkey; +enum dma_data_direction dma_dir; +u8 dmaed:1 ; +struct mlx5_vf_migration_file *migf; +/* Optimize mlx5vf_get_migration_page() for sequential access */ +struct scatterlist *last_offset_sg; +unsigned int sg_last_entry; +unsigned long last_offset; +}; +struct mlx5vf_async_data {struct mlx5_async_work cb_work;struct work_struct work;int status; -u32 mkey;void *out; };
@@ -27,14 +41,7 @@ struct mlx5_vf_migration_file { u8 is_err:1;u32 pdn; -struct sg_append_table table; -size_t total_length; -size_t allocated_length; - -/* Optimize mlx5vf_get_migration_page() for sequential access */ -struct scatterlist *last_offset_sg; -unsigned int sg_last_entry; -unsigned long last_offset; +struct mlx5_vhca_data_buffer *buf;struct mlx5vf_pci_core_device *mvdev;wait_queue_head_t poll_wait;struct completion save_comp;
@@ -124,12 +131,20 @@ void mlx5vf_cmd_set_migratable(struct mlx5vf_pci_core_device *mvdev,void mlx5vf_cmd_remove_migratable (struct mlx5vf_pci_core_device *mvdev);void mlx5vf_cmd_close_migratable (struct mlx5vf_pci_core_device *mvdev);int mlx5vf_cmd_save_vhca_state (struct mlx5vf_pci_core_device *mvdev, -struct mlx5_vf_migration_file *migf); +struct mlx5_vf_migration_file *migf, +struct mlx5_vhca_data_buffer *buf);int mlx5vf_cmd_load_vhca_state (struct mlx5vf_pci_core_device *mvdev, -struct mlx5_vf_migration_file *migf); +struct mlx5_vf_migration_file *migf, +struct mlx5_vhca_data_buffer *buf);int mlx5vf_cmd_alloc_pd (struct mlx5_vf_migration_file *migf);void mlx5vf_cmd_dealloc_pd (struct mlx5_vf_migration_file *migf);void mlx5fv_cmd_clean_migf_resources (struct mlx5_vf_migration_file *migf); +struct mlx5_vhca_data_buffer * +mlx5vf_alloc_data_buffer (struct mlx5_vf_migration_file *migf, +size_t length,enum dma_data_direction dma_dir); +void mlx5vf_free_data_buffer (struct mlx5_vhca_data_buffer *buf); +int mlx5vf_add_migration_pages (struct mlx5_vhca_data_buffer *buf, +unsigned int npages);void mlx5vf_state_mutex_unlock (struct mlx5vf_pci_core_device *mvdev);void mlx5vf_disable_fds (struct mlx5vf_pci_core_device *mvdev);void mlx5vf_mig_file_cleanup_cb (struct work_struct *_work);
drivers/vfio/pci/mlx5/main.c +50 -42
237 lines not on this page · raw message
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH 6/14 vfio/mlx5: Refactor migration file state
In reply to Yishai HadasRefactor migration file state to be an emum which is mutual exclusive.
As of that dropped the 'disabled' state as 'error' is the same from functional point of view.
Next patches from the series will extend this enum for other relevant states.
--- drivers/vfio/pci/mlx5/cmd.c | 2 +- drivers/vfio/pci/mlx5/cmd.h | 7 +++++-- drivers/vfio/pci/mlx5/main.c | 11 ++++++----- 3 files changed, 12 insertions(+), 8 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +1 -1
@@ -351,7 +351,7 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work)mutex_lock (&migf->lock);if (async_data->status) { - migf->is_err = true; + migf->state =MLX5_MIGF_STATE_ERROR ;wake_up_interruptible (&migf->poll_wait); }mutex_unlock (&migf->lock);
drivers/vfio/pci/mlx5/cmd.h +5 -2
@@ -12,6 +12,10 @@#include <linux/mlx5/cq.h> #include <linux/mlx5/qp.h> +enum mlx5_vf_migf_state { +MLX5_MIGF_STATE_ERROR =1 , +}; +struct mlx5_vhca_data_buffer {struct sg_append_table table;loff_t start_pos;
@@ -37,8 +41,7 @@ struct mlx5vf_async_data {struct mlx5_vf_migration_file {struct file *filp;struct mutex lock; -u8 disabled:1 ; -u8 is_err:1 ; +enum mlx5_vf_migf_state state;u32 pdn;struct mlx5_vhca_data_buffer *buf;
drivers/vfio/pci/mlx5/main.c +6 -5
@@ -109,7 +109,7 @@ int mlx5vf_add_migration_pages(struct mlx5_vhca_data_buffer *buf,static void mlx5vf_disable_fd (struct mlx5_vf_migration_file *migf) {mutex_lock (&migf->lock); - migf->disabled = true; + migf->state =MLX5_MIGF_STATE_ERROR ; migf->filp->f_pos =0 ;mutex_unlock (&migf->lock); }
@@ -137,7 +137,8 @@ static ssize_t mlx5vf_save_read(struct file *filp, char __user *buf, size_t len,if (!(filp->f_flags &O_NONBLOCK )) {if (wait_event_interruptible (migf->poll_wait, -READ_ONCE (vhca_buf->length) || migf->is_err)) +READ_ONCE (vhca_buf->length) || + migf->state ==MLX5_MIGF_STATE_ERROR ))return -ERESTARTSYS ; }
@@ -150,7 +151,7 @@ static ssize_t mlx5vf_save_read(struct file *filp, char __user *buf, size_t len, done = -EINVAL ; goto out_unlock; } -if (migf->disabled || migf->is_err) { +if (migf->state ==MLX5_MIGF_STATE_ERROR ) { done = -ENODEV ; goto out_unlock; }
@@ -199,7 +200,7 @@ static __poll_t mlx5vf_save_poll(struct file *filp,poll_wait (filp, &migf->poll_wait, wait);mutex_lock (&migf->lock); -if (migf->disabled || migf->is_err) +if (migf->state ==MLX5_MIGF_STATE_ERROR ) pollflags =EPOLLIN |EPOLLRDNORM |EPOLLRDHUP ;else if (READ_ONCE (migf->buf->length)) pollflags =EPOLLIN |EPOLLRDNORM ;
@@ -293,7 +294,7 @@ static ssize_t mlx5vf_resume_write(struct file *filp, const char __user *buf,return -ENOMEM ;mutex_lock (&migf->lock); -if (migf->disabled) { +if (migf->state ==MLX5_MIGF_STATE_ERROR ) { done = -ENODEV ; goto out_unlock; }
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH 7/14 vfio/mlx5: Refactor to use queue based data chunks
In reply to Yishai HadasRefactor to use queue based data chunks on the migration file.
The SAVE command adds a chunk to the tail of the queue while the read() API finds the required chunk and returns its data.
In case the queue is empty but the state of the migration file is MLX5_MIGF_STATE_COMPLETE, read() may not be blocked but will return 0 to indicate end of file.
This is a step towards maintaining multiple images and their meta data (i.e. headers) on the migration file as part of next patches from the series.
Note:
At that point, we still use a single chunk on the migration file but becomes ready to support multiple.
--- drivers/vfio/pci/mlx5/cmd.c | 24 +++++- drivers/vfio/pci/mlx5/cmd.h | 5 ++ drivers/vfio/pci/mlx5/main.c | 145 +++++++++++++++++++++++++++-------- 3 files changed, 136 insertions(+), 38 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +20 -4
@@ -351,6 +351,7 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work)mutex_lock (&migf->lock);if (async_data->status) { + migf->buf = async_data->buf; migf->state =MLX5_MIGF_STATE_ERROR ;wake_up_interruptible (&migf->poll_wait); }
@@ -368,9 +369,15 @@ static void mlx5vf_save_callback(int status, struct mlx5_async_work *context) struct mlx5_vf_migration_file, async_data);if (!status) { -WRITE_ONCE (migf->buf->length, -MLX5_GET (save_vhca_state_out, async_data->out, - actual_image_size)); +unsigned long flags; + + async_data->buf->length = +MLX5_GET (save_vhca_state_out, async_data->out, + actual_image_size); +spin_lock_irqsave (&migf->list_lock, flags); +list_add_tail (&async_data->buf->buf_elm, &migf->buf_list); +spin_unlock_irqrestore (&migf->list_lock, flags); + migf->state =MLX5_MIGF_STATE_COMPLETE ;wake_up_interruptible (&migf->poll_wait); }
@@ -407,6 +414,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,MLX5_SET (save_vhca_state_in, in, size, buf->allocated_length); async_data = &migf->async_data; + async_data->buf = buf; async_data->out =kvzalloc (out_size,GFP_KERNEL );if (!async_data->out) { err = -ENOMEM ;
@@ -479,14 +487,22 @@ void mlx5vf_cmd_dealloc_pd(struct mlx5_vf_migration_file *migf)void mlx5fv_cmd_clean_migf_resources (struct mlx5_vf_migration_file *migf) { -lockdep_assert_held (&migf->mvdev->state_mutex); +struct mlx5_vhca_data_buffer *entry; +lockdep_assert_held (&migf->mvdev->state_mutex);WARN_ON (migf->mvdev->mdev_detach);if (migf->buf) {mlx5vf_free_data_buffer (migf->buf); migf->buf =NULL ; } + +while ((entry =list_first_entry_or_null (&migf->buf_list, + struct mlx5_vhca_data_buffer, buf_elm))) { +list_del (&entry->buf_elm); +mlx5vf_free_data_buffer (entry); + } +mlx5vf_cmd_dealloc_pd (migf); }
drivers/vfio/pci/mlx5/cmd.h +5 -0
@@ -14,6 +14,7 @@enum mlx5_vf_migf_state {MLX5_MIGF_STATE_ERROR =1 , +MLX5_MIGF_STATE_COMPLETE , };struct mlx5_vhca_data_buffer {
@@ -24,6 +25,7 @@ struct mlx5_vhca_data_buffer {u32 mkey;enum dma_data_direction dma_dir; u8 dmaed:1; +struct list_head buf_elm;struct mlx5_vf_migration_file *migf;/* Optimize mlx5vf_get_migration_page() for sequential access */ struct scatterlist *last_offset_sg;
@@ -34,6 +36,7 @@ struct mlx5_vhca_data_buffer {struct mlx5vf_async_data {struct mlx5_async_work cb_work;struct work_struct work; +struct mlx5_vhca_data_buffer *buf;int status;void *out; };
@@ -45,6 +48,8 @@ struct mlx5_vf_migration_file {u32 pdn;struct mlx5_vhca_data_buffer *buf; +spinlock_t list_lock; +struct list_head buf_list;struct mlx5vf_pci_core_device *mvdev;wait_queue_head_t poll_wait;struct completion save_comp;
drivers/vfio/pci/mlx5/main.c +111 -34
201 lines not on this page · raw message
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH 8/14 vfio/mlx5: Introduce device transitions of PRE_COPY
In reply to Yishai HadasIn order to support PRE_COPY, mlx5 driver is transferring multiple states (images) of the device. e.g.: the source VF can save and transfer multiple states, and the target VF will load them by that order.
The device is saving three kinds of states:
1) Initial state - when the device moves to PRE_COPY state.
2) Middle state - during PRE_COPY phase via VFIO_MIG_GET_PRECOPY_INFO. There can be multiple states of this type.
3) Final state - when the device moves to STOP_COPY state.
After moving to PRE_COPY state, user is holding the saving migf FD and can use it. For example: user can start transferring data via read() callback. Also, user can switch from PRE_COPY to STOP_COPY whenever he sees it fits. This will invoke saving of final state.
This means that mlx5 VFIO device can be switched to STOP_COPY without transferring any data in PRE_COPY state. Therefore, when the device moves to STOP_COPY, mlx5 will store the final state on a dedicated queue entry on the list.
--- drivers/vfio/pci/mlx5/cmd.c | 96 +++++++++++++++++++++++++++++++++--- drivers/vfio/pci/mlx5/cmd.h | 16 +++++- drivers/vfio/pci/mlx5/main.c | 90 ++++++++++++++++++++++++++++++--- 3 files changed, 184 insertions(+), 18 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +88 -8
@@ -14,18 +14,36 @@ _mlx5vf_free_page_tracker_resources(struct mlx5vf_pci_core_device *mvdev);int mlx5vf_cmd_suspend_vhca (struct mlx5vf_pci_core_device *mvdev,u16 op_mod) { +struct mlx5_vf_migration_file *migf = mvdev->saving_migf;u32 out[MLX5_ST_SZ_DW (suspend_vhca_out)] = {};u32 in[MLX5_ST_SZ_DW (suspend_vhca_in)] = {}; +int err;lockdep_assert_held (&mvdev->state_mutex);if (mvdev->mdev_detach)return -ENOTCONN ; +/* +* In case PRE_COPY is used, saving_migf is exposed while the device is +* running. Make sure to run only once there is no active save command. +* Running both in parallel, might end-up with a failure in the save +* command once it will try to turn on 'tracking' on a suspended device. +*/ +if (migf) { + err =wait_for_completion_interruptible (&migf->save_comp); +if (err) +return err; + } +MLX5_SET (suspend_vhca_in, in, opcode,MLX5_CMD_OP_SUSPEND_VHCA );MLX5_SET (suspend_vhca_in, in, vhca_id, mvdev->vhca_id);MLX5_SET (suspend_vhca_in, in, op_mod, op_mod); -return mlx5_cmd_exec_inout (mvdev->mdev, suspend_vhca, in, out); + err =mlx5_cmd_exec_inout (mvdev->mdev, suspend_vhca, in, out); +if (migf) +complete (&migf->save_comp); + +return err; }int mlx5vf_cmd_resume_vhca (struct mlx5vf_pci_core_device *mvdev,u16 op_mod)
@@ -45,7 +63,7 @@ int mlx5vf_cmd_resume_vhca(struct mlx5vf_pci_core_device *mvdev, u16 op_mod) }int mlx5vf_cmd_query_vhca_migration_state (struct mlx5vf_pci_core_device *mvdev, -size_t *state_size) +size_t *state_size,u8 query_flags) {u32 out[MLX5_ST_SZ_DW (query_vhca_migration_state_out)] = {};u32 in[MLX5_ST_SZ_DW (query_vhca_migration_state_in)] = {};
@@ -59,6 +77,8 @@ int mlx5vf_cmd_query_vhca_migration_state(struct mlx5vf_pci_core_device *mvdev, MLX5_CMD_OP_QUERY_VHCA_MIGRATION_STATE);MLX5_SET (query_vhca_migration_state_in, in, vhca_id, mvdev->vhca_id);MLX5_SET (query_vhca_migration_state_in, in, op_mod,0 ); +MLX5_SET (query_vhca_migration_state_in, in, incremental, + query_flags &MLX5VF_QUERY_INC ); ret =mlx5_cmd_exec_inout (mvdev->mdev, query_vhca_migration_state, in, out);
@@ -342,6 +362,56 @@ mlx5vf_alloc_data_buffer(struct mlx5_vf_migration_file *migf,return ERR_PTR (ret); } +void mlx5vf_put_data_buffer (struct mlx5_vhca_data_buffer *buf) +{ +spin_lock_irq (&buf->migf->list_lock); +list_add_tail (&buf->buf_elm, &buf->migf->avail_list); +spin_unlock_irq (&buf->migf->list_lock); +} + +struct mlx5_vhca_data_buffer * +mlx5vf_get_data_buffer (struct mlx5_vf_migration_file *migf, +size_t length,enum dma_data_direction dma_dir) +{ +struct mlx5_vhca_data_buffer *buf, *temp_buf; +struct list_head free_list; + +lockdep_assert_held (&migf->mvdev->state_mutex); +if (migf->mvdev->mdev_detach) +return ERR_PTR (-ENOTCONN ); + +INIT_LIST_HEAD (&free_list); + +spin_lock_irq (&migf->list_lock); +list_for_each_entry_safe (buf, temp_buf, &migf->avail_list, buf_elm) { +if (buf->dma_dir == dma_dir) { +list_del_init (&buf->buf_elm); +if (buf->allocated_length >= length) { +spin_unlock_irq (&migf->list_lock); + goto found; + } +/* +* Prevent holding redundant buffers. Put in a free +* list and call at the end not under the spin lock +* (&migf->list_lock) to mlx5vf_free_data_buffer which +* might sleep. +*/ +list_add (&buf->buf_elm, &free_list); + } + } +spin_unlock_irq (&migf->list_lock); + buf =mlx5vf_alloc_data_buffer (migf, length, dma_dir); + +found: +while ((temp_buf =list_first_entry_or_null (&free_list, + struct mlx5_vhca_data_buffer, buf_elm))) { +list_del (&temp_buf->buf_elm); +mlx5vf_free_data_buffer (temp_buf); + } + +return buf; +} +void mlx5vf_mig_file_cleanup_cb (struct work_struct *_work) { struct mlx5vf_async_data *async_data = container_of(_work,
@@ -351,7 +421,7 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work)mutex_lock (&migf->lock);if (async_data->status) { - migf->buf = async_data->buf; +mlx5vf_put_data_buffer (async_data->buf); migf->state =MLX5_MIGF_STATE_ERROR ;wake_up_interruptible (&migf->poll_wait); }
@@ -369,15 +439,19 @@ static void mlx5vf_save_callback(int status, struct mlx5_async_work *context) struct mlx5_vf_migration_file, async_data);if (!status) { +size_t image_size;unsigned long flags; - async_data->buf->length = -MLX5_GET (save_vhca_state_out, async_data->out, - actual_image_size); + image_size =MLX5_GET (save_vhca_state_out, async_data->out, + actual_image_size); + async_data->buf->length = image_size; + async_data->buf->start_pos = migf->max_pos; + migf->max_pos += async_data->buf->length;spin_lock_irqsave (&migf->list_lock, flags);list_add_tail (&async_data->buf->buf_elm, &migf->buf_list);spin_unlock_irqrestore (&migf->list_lock, flags); - migf->state =MLX5_MIGF_STATE_COMPLETE ; +if (async_data->last_chunk) + migf->state =MLX5_MIGF_STATE_COMPLETE ;wake_up_interruptible (&migf->poll_wait); }
@@ -391,7 +465,8 @@ static void mlx5vf_save_callback(int status, struct mlx5_async_work *context)int mlx5vf_cmd_save_vhca_state (struct mlx5vf_pci_core_device *mvdev,struct mlx5_vf_migration_file *migf, -struct mlx5_vhca_data_buffer *buf) +struct mlx5_vhca_data_buffer *buf,bool inc, +bool track) {u32 out_size =MLX5_ST_SZ_BYTES (save_vhca_state_out);u32 in[MLX5_ST_SZ_DW (save_vhca_state_in)] = {};
@@ -412,9 +487,12 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,MLX5_SET (save_vhca_state_in, in, vhca_id, mvdev->vhca_id);MLX5_SET (save_vhca_state_in, in, mkey, buf->mkey);MLX5_SET (save_vhca_state_in, in, size, buf->allocated_length); +MLX5_SET (save_vhca_state_in, in, incremental, inc); +MLX5_SET (save_vhca_state_in, in, set_track, track); async_data = &migf->async_data; async_data->buf = buf; + async_data->last_chunk = !track; async_data->out =kvzalloc (out_size,GFP_KERNEL );if (!async_data->out) { err = -ENOMEM ;
@@ -497,6 +575,8 @@ void mlx5fv_cmd_clean_migf_resources(struct mlx5_vf_migration_file *migf) migf->buf =NULL ; } +list_splice (&migf->avail_list, &migf->buf_list); +while ((entry =list_first_entry_or_null (&migf->buf_list, struct mlx5_vhca_data_buffer, buf_elm))) {list_del (&entry->buf_elm);
drivers/vfio/pci/mlx5/cmd.h +14 -2
@@ -38,6 +38,7 @@ struct mlx5vf_async_data {struct work_struct work;struct mlx5_vhca_data_buffer *buf;int status; + u8 last_chunk:1;void *out; };
@@ -47,9 +48,11 @@ struct mlx5_vf_migration_file {enum mlx5_vf_migf_state state;u32 pdn; +loff_t max_pos;struct mlx5_vhca_data_buffer *buf;spinlock_t list_lock;struct list_head buf_list; +struct list_head avail_list;struct mlx5vf_pci_core_device *mvdev;wait_queue_head_t poll_wait;struct completion save_comp;
@@ -129,10 +132,14 @@ struct mlx5vf_pci_core_device {struct mlx5_core_dev *mdev; }; +enum { +MLX5VF_QUERY_INC = (1UL <<0 ), +}; +int mlx5vf_cmd_suspend_vhca (struct mlx5vf_pci_core_device *mvdev,u16 op_mod);int mlx5vf_cmd_resume_vhca (struct mlx5vf_pci_core_device *mvdev,u16 op_mod);int mlx5vf_cmd_query_vhca_migration_state (struct mlx5vf_pci_core_device *mvdev, -size_t *state_size); +size_t *state_size,u8 query_flags);void mlx5vf_cmd_set_migratable (struct mlx5vf_pci_core_device *mvdev,const struct vfio_migration_ops *mig_ops,const struct vfio_log_ops *log_ops);
@@ -140,7 +147,8 @@ void mlx5vf_cmd_remove_migratable(struct mlx5vf_pci_core_device *mvdev);void mlx5vf_cmd_close_migratable (struct mlx5vf_pci_core_device *mvdev);int mlx5vf_cmd_save_vhca_state (struct mlx5vf_pci_core_device *mvdev,struct mlx5_vf_migration_file *migf, -struct mlx5_vhca_data_buffer *buf); +struct mlx5_vhca_data_buffer *buf,bool inc, +bool track);int mlx5vf_cmd_load_vhca_state (struct mlx5vf_pci_core_device *mvdev,struct mlx5_vf_migration_file *migf,struct mlx5_vhca_data_buffer *buf);
@@ -151,6 +159,10 @@ struct mlx5_vhca_data_buffer * mlx5vf_alloc_data_buffer(struct mlx5_vf_migration_file *migf, size_t length, enum dma_data_direction dma_dir);void mlx5vf_free_data_buffer (struct mlx5_vhca_data_buffer *buf); +struct mlx5_vhca_data_buffer * +mlx5vf_get_data_buffer (struct mlx5_vf_migration_file *migf, +size_t length,enum dma_data_direction dma_dir); +void mlx5vf_put_data_buffer (struct mlx5_vhca_data_buffer *buf);int mlx5vf_add_migration_pages (struct mlx5_vhca_data_buffer *buf,unsigned int npages);void mlx5vf_state_mutex_unlock (struct mlx5vf_pci_core_device *mvdev);
drivers/vfio/pci/mlx5/main.c +82 -8
164 lines not on this page · raw message
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH 9/14 vfio/mlx5: Introduce SW headers for migration states
In reply to Yishai HadasAs mentioned in the previous patches, mlx5 is transferring multiple states when the PRE_COPY protocol is used. This states mechanism requires the target VM to know the states' size in order to execute multiple loads. Therefore, add SW header, with the needed information, for each saved state the source VM is transferring to the target VM.
This patch implements the source VM handling of the headers, following patch will implement the target VM handling of the headers.
--- drivers/vfio/pci/mlx5/cmd.c | 56 ++++++++++++++++++++++++++++++++++-- drivers/vfio/pci/mlx5/cmd.h | 10 +++++++ drivers/vfio/pci/mlx5/main.c | 2 +- 3 files changed, 64 insertions(+), 4 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +53 -3
@@ -351,9 +351,11 @@ mlx5vf_alloc_data_buffer(struct mlx5_vf_migration_file *migf,if (ret) goto end; - ret =mlx5vf_dma_data_buffer (buf); -if (ret) - goto end; +if (dma_dir !=DMA_NONE ) { + ret =mlx5vf_dma_data_buffer (buf); +if (ret) + goto end; + } }return buf;
@@ -422,6 +424,8 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work)mutex_lock (&migf->lock);if (async_data->status) {mlx5vf_put_data_buffer (async_data->buf); +if (async_data->header_buf) +mlx5vf_put_data_buffer (async_data->header_buf); migf->state =MLX5_MIGF_STATE_ERROR ;wake_up_interruptible (&migf->poll_wait); }
@@ -431,6 +435,32 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work)fput (migf->filp); } +static int add_buf_header (struct mlx5_vhca_data_buffer *header_buf, +size_t image_size) +{ +struct mlx5_vf_migration_file *migf = header_buf->migf; +struct mlx5_vf_migration_header header = {}; +unsigned long flags; +struct page *page; +u8 *to_buff; + + header.image_size =cpu_to_le64 (image_size); + page =mlx5vf_get_migration_page (header_buf,0 ); +if (!page) +return -EINVAL ; + to_buff =kmap_local_page (page); +memcpy (to_buff, &header,sizeof (header)); +kunmap_local (to_buff); + header_buf->length =sizeof (header); + header_buf->header_image_size = image_size; + header_buf->start_pos = header_buf->migf->max_pos; + migf->max_pos += header_buf->length; +spin_lock_irqsave (&migf->list_lock, flags); +list_add_tail (&header_buf->buf_elm, &migf->buf_list); +spin_unlock_irqrestore (&migf->list_lock, flags); +return 0 ; +} +static void mlx5vf_save_callback (int status,struct mlx5_async_work *context) { struct mlx5vf_async_data *async_data = container_of(context,
@@ -444,6 +474,11 @@ static void mlx5vf_save_callback(int status, struct mlx5_async_work *context) image_size =MLX5_GET (save_vhca_state_out, async_data->out, actual_image_size); +if (async_data->header_buf) { + status =add_buf_header (async_data->header_buf, image_size); +if (status) + goto err; + } async_data->buf->length = image_size; async_data->buf->start_pos = migf->max_pos; migf->max_pos += async_data->buf->length;
@@ -455,6 +490,7 @@ static void mlx5vf_save_callback(int status, struct mlx5_async_work *context)wake_up_interruptible (&migf->poll_wait); } +err:/* * The error and the cleanup flows can't run from an * interrupt context
@@ -470,6 +506,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, {u32 out_size =MLX5_ST_SZ_BYTES (save_vhca_state_out);u32 in[MLX5_ST_SZ_DW (save_vhca_state_in)] = {}; +struct mlx5_vhca_data_buffer *header_buf =NULL ;struct mlx5vf_async_data *async_data;int err;
@@ -499,6 +536,16 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, goto err_out; } +if (track || inc) { + header_buf =mlx5vf_get_data_buffer (migf, +sizeof (struct mlx5_vf_migration_header ),DMA_NONE ); +if (IS_ERR (header_buf)) { + err =PTR_ERR (header_buf); + goto err_free; + } + } + + async_data->header_buf = header_buf;get_file (migf->filp); err = mlx5_cmd_exec_cb(&migf->async_ctx, in, sizeof(in), async_data->out,
@@ -510,7 +557,10 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,return 0 ; err_exec: +if (header_buf) +mlx5vf_put_data_buffer (header_buf);fput (migf->filp); +err_free:kvfree (async_data->out); err_out:complete (&migf->save_comp);
drivers/vfio/pci/mlx5/cmd.h +10 -0
@@ -17,11 +17,18 @@ enum mlx5_vf_migf_state { MLX5_MIGF_STATE_COMPLETE, }; +struct mlx5_vf_migration_header { +__le64 image_size; +/* For future use in case we may need to change the kernel protocol */ +__le64 flags; +}; +struct mlx5_vhca_data_buffer {struct sg_append_table table;loff_t start_pos;u64 length;u64 allocated_length; +u64 header_image_size;u32 mkey;enum dma_data_direction dma_dir;u8 dmaed:1 ;
@@ -37,6 +44,7 @@ struct mlx5vf_async_data {struct mlx5_async_work cb_work;struct work_struct work;struct mlx5_vhca_data_buffer *buf; +struct mlx5_vhca_data_buffer *header_buf;int status; u8 last_chunk:1;void *out;
@@ -165,6 +173,8 @@ mlx5vf_get_data_buffer(struct mlx5_vf_migration_file *migf,void mlx5vf_put_data_buffer (struct mlx5_vhca_data_buffer *buf);int mlx5vf_add_migration_pages (struct mlx5_vhca_data_buffer *buf,unsigned int npages); +struct page *mlx5vf_get_migration_page (struct mlx5_vhca_data_buffer *buf, +unsigned long offset);void mlx5vf_state_mutex_unlock (struct mlx5vf_pci_core_device *mvdev);void mlx5vf_disable_fds (struct mlx5vf_pci_core_device *mvdev);void mlx5vf_mig_file_cleanup_cb (struct work_struct *_work);
drivers/vfio/pci/mlx5/main.c +1 -1
@@ -32,7 +32,7 @@ static struct mlx5vf_pci_core_device *mlx5vf_drvdata(struct pci_dev *pdev) core_device); } -static struct page * +struct page *mlx5vf_get_migration_page (struct mlx5_vhca_data_buffer *buf,unsigned long offset) {
Signature · 1 line
2.18.1
Yishai Hadas
maintainerRe: [PATCH V1 vfio 09/14] vfio/mlx5: Introduce SW headers for migration states
Quoted 106 lines · Yishai Hadas
As mentioned in the previous patches, mlx5 is transferring multiple states when the
PRE_COPYprotocol is used. This states mechanism requires the target VM to know the states' size in order to execute multiple loads. Therefore, add SW header, with the needed information, for each saved state the source VM is transferring to the target VM.This patch implements the source VM handling of the headers, following patch will implement the target VM handling of the headers.
Signed-off-by Yishai Hadas <[email protected]>--- drivers/vfio/pci/mlx5/cmd.c | 56 ++++++++++++++++++++++++++++++++++-- drivers/vfio/pci/mlx5/cmd.h | 10 +++++++ drivers/vfio/pci/mlx5/main.c | 2 +- 3 files changed, 64 insertions(+), 4 deletions(-)drivers/vfio/pci/mlx5/cmd.c +40
-3@@ -351,9 +351,11 @@ mlx5vf_alloc_data_buffer(struct mlx5_vf_migration_file *migf,if (ret) goto end; - ret =mlx5vf_dma_data_buffer (buf); -if (ret) - goto end; +if (dma_dir !=DMA_NONE ) { + ret =mlx5vf_dma_data_buffer (buf); +if (ret) + goto end; + } }return buf;@@ -422,6 +424,8 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work)mutex_lock (&migf->lock);if (async_data->status) {mlx5vf_put_data_buffer (async_data->buf); +if (async_data->header_buf) +mlx5vf_put_data_buffer (async_data->header_buf); migf->state =MLX5_MIGF_STATE_ERROR ;wake_up_interruptible (&migf->poll_wait); }@@ -431,6 +435,32 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work)fput (migf->filp); } +static int add_buf_header (struct mlx5_vhca_data_buffer *header_buf, +size_t image_size) +{ +struct mlx5_vf_migration_file *migf = header_buf->migf; +struct mlx5_vf_migration_header header = {}; +unsigned long flags; +struct page *page; +u8 *to_buff; + + header.image_size =cpu_to_le64 (image_size); + page =mlx5vf_get_migration_page (header_buf,0 ); +if (!page) +return -EINVAL ; + to_buff =kmap_local_page (page); +memcpy (to_buff, &header,sizeof (header)); +kunmap_local (to_buff); + header_buf->length =sizeof (header); + header_buf->header_image_size = image_size; + header_buf->start_pos = header_buf->migf->max_pos; + migf->max_pos += header_buf->length; +spin_lock_irqsave (&migf->list_lock, flags); +list_add_tail (&header_buf->buf_elm, &migf->buf_list); +spin_unlock_irqrestore (&migf->list_lock, flags); +return 0 ; +} +static void mlx5vf_save_callback (int status,struct mlx5_async_work *context) { struct mlx5vf_async_data *async_data = container_of(context,@@ -444,6 +474,11 @@ static void mlx5vf_save_callback(int status, struct mlx5_async_work *context) image_size =MLX5_GET (save_vhca_state_out, async_data->out, actual_image_size); +if (async_data->header_buf) { + status =add_buf_header (async_data->header_buf, image_size); +if (status) + goto err; + } async_data->buf->length = image_size; async_data->buf->start_pos = migf->max_pos; migf->max_pos += async_data->buf->length;@@ -455,6 +490,7 @@ static void mlx5vf_save_callback(int status, struct mlx5_async_work *context)wake_up_interruptible (&migf->poll_wait); } +err:/* * The error and the cleanup flows can't run from an * interrupt context @@ -470,6 +506,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, {u32 out_size =MLX5_ST_SZ_BYTES (save_vhca_state_out);u32 in[MLX5_ST_SZ_DW (save_vhca_state_in)] = {}; +struct mlx5_vhca_data_buffer *header_buf =NULL ;struct mlx5vf_async_data *async_data;int err;
drivers/vfio/pci/mlx5/cmd.c int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, · line 539
Quoted 4 lines
drivers/vfio/pci/mlx5/cmd.c +1
-0@@ -499,6 +536,16 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, goto err_out; } +if (track || inc) {
This check should be replaced by some general check whether PRE_COPY is supported by the mlx5 device.
Otherwise the header data won't be sent in some flows (e.g. user space moved directly to STOP_COPY without being in PRE_COPY) which is wrong.
Will fix it as part of V2.
Yishai
Quoted 76 lines
drivers/vfio/pci/mlx5/cmd.c +12
-0+ header_buf =mlx5vf_get_data_buffer (migf, +sizeof (struct mlx5_vf_migration_header ),DMA_NONE ); +if (IS_ERR (header_buf)) { + err =PTR_ERR (header_buf); + goto err_free; + } + } + + async_data->header_buf = header_buf;get_file (migf->filp); err = mlx5_cmd_exec_cb(&migf->async_ctx, in, sizeof(in), async_data->out,@@ -510,7 +557,10 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,return 0 ; err_exec: +if (header_buf) +mlx5vf_put_data_buffer (header_buf);fput (migf->filp); +err_free:kvfree (async_data->out); err_out:complete (&migf->save_comp);drivers/vfio/pci/mlx5/cmd.h +10
-0@@ -17,11 +17,18 @@ enum mlx5_vf_migf_state { MLX5_MIGF_STATE_COMPLETE, }; +struct mlx5_vf_migration_header { +__le64 image_size; +/* For future use in case we may need to change the kernel protocol */ +__le64 flags; +}; +struct mlx5_vhca_data_buffer {struct sg_append_table table;loff_t start_pos;u64 length;u64 allocated_length; +u64 header_image_size;u32 mkey;enum dma_data_direction dma_dir;u8 dmaed:1 ;@@ -37,6 +44,7 @@ struct mlx5vf_async_data {struct mlx5_async_work cb_work;struct work_struct work;struct mlx5_vhca_data_buffer *buf; +struct mlx5_vhca_data_buffer *header_buf;int status; u8 last_chunk:1;void *out;@@ -165,6 +173,8 @@ mlx5vf_get_data_buffer(struct mlx5_vf_migration_file *migf,void mlx5vf_put_data_buffer (struct mlx5_vhca_data_buffer *buf);int mlx5vf_add_migration_pages (struct mlx5_vhca_data_buffer *buf,unsigned int npages); +struct page *mlx5vf_get_migration_page (struct mlx5_vhca_data_buffer *buf, +unsigned long offset);void mlx5vf_state_mutex_unlock (struct mlx5vf_pci_core_device *mvdev);void mlx5vf_disable_fds (struct mlx5vf_pci_core_device *mvdev);void mlx5vf_mig_file_cleanup_cb (struct work_struct *_work);drivers/vfio/pci/mlx5/main.c +1
-1@@ -32,7 +32,7 @@ static struct mlx5vf_pci_core_device *mlx5vf_drvdata(struct pci_dev *pdev) core_device); } -static struct page * +struct page *mlx5vf_get_migration_page (struct mlx5_vhca_data_buffer *buf,unsigned long offset) {
Yishai Hadas
maintainerPATCH 10/14 vfio/mlx5: Introduce vfio precopy ioctl implementation
In reply to Yishai Hadasvfio precopy ioctl returns an estimation of data available for transferring from the device.
Whenever a user is using VFIO_MIG_GET_PRECOPY_INFO, track the current state of the device, and if needed, append the dirty data to the transfer FD data. This is done by saving a middle state.
As mlx5 runs the SAVE command asynchronously, make sure to query for incremental data only once there is no active save command. Running both in parallel, might end-up with a failure in the incremental query command on un-tracked vhca.
Also, a middle state will be saved only after the previous state has finished its SAVE command and has been fully transferred, this prevents endless use resources.
--- drivers/vfio/pci/mlx5/cmd.c | 16 +++++ drivers/vfio/pci/mlx5/main.c | 111 +++++++++++++++++++++++++++++++++++ 2 files changed, 127 insertions(+)
drivers/vfio/pci/mlx5/cmd.c +16 -0
@@ -67,12 +67,25 @@ int mlx5vf_cmd_query_vhca_migration_state(struct mlx5vf_pci_core_device *mvdev, {u32 out[MLX5_ST_SZ_DW (query_vhca_migration_state_out)] = {};u32 in[MLX5_ST_SZ_DW (query_vhca_migration_state_in)] = {}; +bool inc = query_flags &MLX5VF_QUERY_INC ;int ret;lockdep_assert_held (&mvdev->state_mutex);if (mvdev->mdev_detach)return -ENOTCONN ; +/* +* In case PRE_COPY is used, saving_migf is exposed while device is +* running. Make sure to run only once there is no active save command. +* Running both in parallel, might end-up with a failure in the +* incremental query command on un-tracked vhca. +*/ +if (inc) { + ret =wait_for_completion_interruptible (&mvdev->saving_migf->save_comp); +if (ret) +return ret; + } +MLX5_SET (query_vhca_migration_state_in, in, opcode,MLX5_CMD_OP_QUERY_VHCA_MIGRATION_STATE );MLX5_SET (query_vhca_migration_state_in, in, vhca_id, mvdev->vhca_id);
@@ -82,6 +95,9 @@ int mlx5vf_cmd_query_vhca_migration_state(struct mlx5vf_pci_core_device *mvdev, ret =mlx5_cmd_exec_inout (mvdev->mdev, query_vhca_migration_state, in, out); +if (inc) +complete (&mvdev->saving_migf->save_comp); +if (ret)return ret;
drivers/vfio/pci/mlx5/main.c +111 -0
121 lines not on this page · raw message
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH 11/14 vfio/mlx5: Consider temporary end of stream as part of PRE_COPY
In reply to Yishai HadasDuring PRE_COPY the migration data FD may have a temporary "end of stream" that is reached when the initial_bytes were read and no other dirty data exists yet.
For instance, this may indicate that the device is idle and not currently dirtying any internal state. When read() is done on this temporary end of stream the kernel driver should return ENOMSG from read(). Userspace can wait for more data or consider moving to STOP_COPY.
To not block the user upon read() and let it get ENOMSG we add a new state named MLX5_MIGF_STATE_PRE_COPY on the migration file.
In addition, we add the MLX5_MIGF_STATE_SAVE_LAST state to block the read() once we call the last SAVE upon moving to STOP_COPY.
Any further error will be marked with MLX5_MIGF_STATE_ERROR and the user won't be blocked.
--- drivers/vfio/pci/mlx5/cmd.c | 7 +++++-- drivers/vfio/pci/mlx5/cmd.h | 2 ++ drivers/vfio/pci/mlx5/main.c | 7 +++++++ 3 files changed, 14 insertions(+), 2 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +5 -2
@@ -501,8 +501,8 @@ static void mlx5vf_save_callback(int status, struct mlx5_async_work *context)spin_lock_irqsave (&migf->list_lock, flags);list_add_tail (&async_data->buf->buf_elm, &migf->buf_list);spin_unlock_irqrestore (&migf->list_lock, flags); -if (async_data->last_chunk) - migf->state =MLX5_MIGF_STATE_COMPLETE ; + migf->state = async_data->last_chunk ? +MLX5_MIGF_STATE_COMPLETE :MLX5_MIGF_STATE_PRE_COPY ;wake_up_interruptible (&migf->poll_wait); }
@@ -561,6 +561,9 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, } } +if (async_data->last_chunk) + migf->state =MLX5_MIGF_STATE_SAVE_LAST ; + async_data->header_buf = header_buf;get_file (migf->filp); err = mlx5_cmd_exec_cb(&migf->async_ctx, in, sizeof(in),
drivers/vfio/pci/mlx5/cmd.h +2 -0
@@ -14,6 +14,8 @@enum mlx5_vf_migf_state {MLX5_MIGF_STATE_ERROR =1 , +MLX5_MIGF_STATE_PRE_COPY , +MLX5_MIGF_STATE_SAVE_LAST ,MLX5_MIGF_STATE_COMPLETE , };
drivers/vfio/pci/mlx5/main.c +7 -0
@@ -219,6 +219,7 @@ static ssize_t mlx5vf_save_read(struct file *filp, char __user *buf, size_t len,if (wait_event_interruptible (migf->poll_wait, !list_empty (&migf->buf_list) || migf->state ==MLX5_MIGF_STATE_ERROR || + migf->state ==MLX5_MIGF_STATE_PRE_COPY || migf->state ==MLX5_MIGF_STATE_COMPLETE ))return -ERESTARTSYS ; }
@@ -236,6 +237,12 @@ static ssize_t mlx5vf_save_read(struct file *filp, char __user *buf, size_t len, &end_of_data);if (first_loop_call) { first_loop_call = false; +/* Temporary end of file as part of PRE_COPY */ +if (end_of_data && migf->state ==MLX5_MIGF_STATE_PRE_COPY ) { + done = -ENOMSG ; + goto out_unlock; + } +if (end_of_data && migf->state !=MLX5_MIGF_STATE_COMPLETE ) {if (filp->f_flags &O_NONBLOCK ) { done = -EAGAIN ;
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH 12/14 vfio/mlx5: Introduce multiple loads
In reply to Yishai HadasIn order to support PRE_COPY, mlx5 driver transfers multiple states (images) of the device. e.g.: the source VF can save and transfer multiple states, and the target VF will load them by that order.
This patch implements the changes for the target VF to decompose the header for each state and to write and load multiple states.
--- drivers/vfio/pci/mlx5/cmd.c | 13 +- drivers/vfio/pci/mlx5/cmd.h | 10 ++ drivers/vfio/pci/mlx5/main.c | 282 +++++++++++++++++++++++++++++------ 3 files changed, 260 insertions(+), 45 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +10 -3
@@ -598,9 +598,11 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev,if (mvdev->mdev_detach)return -ENOTCONN ; - err =mlx5vf_dma_data_buffer (buf); -if (err) -return err; +if (!buf->dmaed) { + err =mlx5vf_dma_data_buffer (buf); +if (err) +return err; + }MLX5_SET (load_vhca_state_in, in, opcode,MLX5_CMD_OP_LOAD_VHCA_STATE );
@@ -644,6 +646,11 @@ void mlx5fv_cmd_clean_migf_resources(struct mlx5_vf_migration_file *migf) migf->buf =NULL ; } +if (migf->buf_header) { +mlx5vf_free_data_buffer (migf->buf_header); + migf->buf_header =NULL ; + } +list_splice (&migf->avail_list, &migf->buf_list); while ((entry = list_first_entry_or_null(&migf->buf_list,
drivers/vfio/pci/mlx5/cmd.h +10 -0
@@ -19,6 +19,14 @@ enum mlx5_vf_migf_state { MLX5_MIGF_STATE_COMPLETE, }; +enum mlx5_vf_load_state { +MLX5_VF_LOAD_STATE_READ_IMAGE_NO_HEADER , +MLX5_VF_LOAD_STATE_READ_HEADER , +MLX5_VF_LOAD_STATE_PREP_IMAGE , +MLX5_VF_LOAD_STATE_READ_IMAGE , +MLX5_VF_LOAD_STATE_LOAD_IMAGE , +}; +struct mlx5_vf_migration_header {__le64 image_size;/* For future use in case we may need to change the kernel protocol */
@@ -57,9 +65,11 @@ struct mlx5_vf_migration_file {struct mutex lock;enum mlx5_vf_migf_state state; +enum mlx5_vf_load_state load_state;u32 pdn;loff_t max_pos;struct mlx5_vhca_data_buffer *buf; +struct mlx5_vhca_data_buffer *buf_header;spinlock_t list_lock;struct list_head buf_list;struct list_head avail_list;
drivers/vfio/pci/mlx5/main.c +240 -42
338 lines not on this page · raw message
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH 13/14 vfio/mlx5: Fallback to STOP_COPY upon specific PRE_COPY error
In reply to Yishai HadasFrom: Shay Drory <[email protected]>
Before a SAVE command is issued, a QUERY command is issued in order to know the device data size.
In case PRE_COPY is used, the above commands are issued while the device is running. Thus, it is possible that between the QUERY and the SAVE commands the state of the device will be changed significantly and thus the SAVE will fail.
Currently, if a SAVE command is failing, the driver will fail the migration. In the above case, don't fail the migration, but don't allow for new SAVEs to be executed while the device is in a RUNNING state. Once the device will be moved to STOP_COPY, SAVE can be executed again and the full device state will be read.
--- drivers/vfio/pci/mlx5/cmd.c | 26 +++++++++++++++++++++++++- drivers/vfio/pci/mlx5/cmd.h | 2 ++ drivers/vfio/pci/mlx5/main.c | 6 ++++-- 3 files changed, 31 insertions(+), 3 deletions(-)
drivers/vfio/pci/mlx5/cmd.c +25 -1
@@ -84,6 +84,18 @@ int mlx5vf_cmd_query_vhca_migration_state(struct mlx5vf_pci_core_device *mvdev, ret =wait_for_completion_interruptible (&mvdev->saving_migf->save_comp);if (ret)return ret; +if (mvdev->saving_migf->state == +MLX5_MIGF_STATE_PRE_COPY_ERROR ) { +/* +* In case we had a PRE_COPY error, only query full +* image for final image +*/ +if (!(query_flags &MLX5VF_QUERY_FINAL )) { + *state_size =0 ; +return 0 ; + } + query_flags &= ~MLX5VF_QUERY_INC ; + } } MLX5_SET(query_vhca_migration_state_in, in, opcode,
@@ -442,7 +454,10 @@ void mlx5vf_mig_file_cleanup_cb(struct work_struct *_work)mlx5vf_put_data_buffer (async_data->buf);if (async_data->header_buf)mlx5vf_put_data_buffer (async_data->header_buf); - migf->state =MLX5_MIGF_STATE_ERROR ; +if (async_data->status ==MLX5_CMD_STAT_BAD_RES_STATE_ERR ) + migf->state =MLX5_MIGF_STATE_PRE_COPY_ERROR ; +else + migf->state =MLX5_MIGF_STATE_ERROR ;wake_up_interruptible (&migf->poll_wait); }mutex_unlock (&migf->lock);
@@ -511,6 +526,8 @@ static void mlx5vf_save_callback(int status, struct mlx5_async_work *context)* The error and the cleanup flows can't run from an * interrupt context */ +if (status == -EREMOTEIO ) + status =MLX5_GET (save_vhca_state_out, async_data->out, status); async_data->status = status;queue_work (migf->mvdev->cb_wq, &async_data->work); }
@@ -534,6 +551,13 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev,if (err)return err; +if (migf->state ==MLX5_MIGF_STATE_PRE_COPY_ERROR ) +/* +* In case we had a PRE_COPY error, SAVE is triggered only for +* the final image, read device full image. +*/ + inc = false; +MLX5_SET (save_vhca_state_in, in, opcode,MLX5_CMD_OP_SAVE_VHCA_STATE );MLX5_SET (save_vhca_state_in, in, op_mod,0 );
drivers/vfio/pci/mlx5/cmd.h +2 -0
@@ -14,6 +14,7 @@enum mlx5_vf_migf_state {MLX5_MIGF_STATE_ERROR =1 , +MLX5_MIGF_STATE_PRE_COPY_ERROR ,MLX5_MIGF_STATE_PRE_COPY ,MLX5_MIGF_STATE_SAVE_LAST ,MLX5_MIGF_STATE_COMPLETE ,
@@ -154,6 +155,7 @@ struct mlx5vf_pci_core_device {enum {MLX5VF_QUERY_INC = (1UL <<0 ), +MLX5VF_QUERY_FINAL = (1UL <<1 ), };int mlx5vf_cmd_suspend_vhca (struct mlx5vf_pci_core_device *mvdev,u16 op_mod);
drivers/vfio/pci/mlx5/main.c +4 -2
@@ -222,6 +222,7 @@ static ssize_t mlx5vf_save_read(struct file *filp, char __user *buf, size_t len,if (wait_event_interruptible (migf->poll_wait, !list_empty (&migf->buf_list) || migf->state ==MLX5_MIGF_STATE_ERROR || + migf->state ==MLX5_MIGF_STATE_PRE_COPY_ERROR || migf->state ==MLX5_MIGF_STATE_PRE_COPY || migf->state ==MLX5_MIGF_STATE_COMPLETE ))return -ERESTARTSYS ;
@@ -241,7 +242,8 @@ static ssize_t mlx5vf_save_read(struct file *filp, char __user *buf, size_t len,if (first_loop_call) { first_loop_call = false;/* Temporary end of file as part of PRE_COPY */ -if (end_of_data && migf->state ==MLX5_MIGF_STATE_PRE_COPY ) { +if (end_of_data && (migf->state ==MLX5_MIGF_STATE_PRE_COPY || + migf->state ==MLX5_MIGF_STATE_PRE_COPY_ERROR )) { done = -ENOMSG ; goto out_unlock; }
@@ -434,7 +436,7 @@ static int mlx5vf_pci_save_device_inc_data(struct mlx5vf_pci_core_device *mvdev)return -ENODEV ; ret =mlx5vf_cmd_query_vhca_migration_state (mvdev, &length, -MLX5VF_QUERY_INC ); +MLX5VF_QUERY_INC |MLX5VF_QUERY_FINAL );if (ret) goto err;
Signature · 1 line
2.18.1
Yishai Hadas
maintainerPATCH 14/14 vfio/mlx5: Enable MIGRATION_PRE_COPY flag
In reply to Yishai HadasFrom: Shay Drory <[email protected]>
Now that everything has been set up for MIGRATION_PRE_COPY, enable it.
--- drivers/vfio/pci/mlx5/cmd.c | 5 +++++ 1 file changed, 5 insertions(+)
drivers/vfio/pci/mlx5/cmd.c +5 -0
@@ -221,6 +221,11 @@ void mlx5vf_cmd_set_migratable(struct mlx5vf_pci_core_device *mvdev,if (MLX5_CAP_GEN (mvdev->mdev, adv_virtualization)) mvdev->core_device.vdev.log_ops = log_ops; +if (MLX5_CAP_GEN_2 (mvdev->mdev, migration_multi_load) && +MLX5_CAP_GEN_2 (mvdev->mdev, migration_tracking_state)) + mvdev->core_device.vdev.migration_flags |= +VFIO_MIGRATION_PRE_COPY ; + end:mlx5_vf_put_core_dev (mvdev->mdev); }
Signature · 1 line
2.18.1
Yishai Hadas
maintainerQuoted 110 lines · Yishai Hadas
This series adds migration
PRE_COPYuAPIs and their implementation as part of mlx5 driver.The uAPIs follow some discussion that was done in the mailing list [1] in this area.
By the time the patches were sent, there was no driver implementation for the uAPIs, now we have it for mlx5 driver.
The optional
PRE_COPYstate opens the saving data transfer FD before reachingSTOP_COPYand allows the device to dirty track the internal state changes with the general idea to reduce the volume of data transferred in theSTOP_COPYstage.While in
PRE_COPYthe device remains RUNNING, but the saving FD is open.A new ioctl
VFIO_MIG_GET_PRECOPY_INFOis provided to allow userspace to query the progress of the precopy operation in the driver with the idea it will judge to move toSTOP_COPYat least once the initial data set is transferred, and possibly after the dirty size has shrunk appropriately.User space can detect whether
PRE_COPYis supported for a given device by checking theVFIO_MIGRATION_PRE_COPYflag once using theVFIO_DEVICE_FEATURE_MIGRATIONioctl.Extra details exist as part of the specific uAPI patch from the series.
Finally, we come with mlx5 implementation based on its device specification for
PRE_COPY.To support
PRE_COPY, mlx5 driver is transferring multiple states (images) of the device. e.g.: the source VF can save and transfer multiple states, and the target VF will load them by that order.The device is saving three kinds of states:
1) Initial state - when the device moves toPRE_COPYstate.
2) Middle state - duringPRE_COPYphase viaVFIO_MIG_GET_PRECOPY_INFO, can be multiple such states.
3) Final state - when the device moves toSTOP_COPYstate.After moving to
PRE_COPYstate, the user is holding the saving FD and should use it for transferring the data from the source to the target while the VM is still running. From user point of view, it's a stream of data, however, from mlx5 driver point of view it includes multiple images/states. For that, it sets some headers with metadata on the source to be parsed on the target.At some point, user may switch the device state from
PRE_COPYtoSTOP_COPY, this will invoke saving of the final state.As discussed earlier in the mailing list, the data that is returned as part of
PRE_COPYis not required to have any bearing relative to the data size available during theSTOP_COPYphase.For this, we have the
VFIO_DEVICE_FEATURE_MIG_DATA_SIZEoption.In mlx5 driver we could gain with this series about 20-30 percent improvement in the downtime compared to the previous code when
PRE_COPYwasn't supported.The series includes some pre-patches to be ready for managing multiple images then it comes with the
PRE_COPYimplementation itself.The matching qemu changes can be previewed here [2].
They come on top of the v2 migration protocol patches that were sent already to the mailing list.
Note:
As this series includes a net/mlx5 patch, we may need to send it as a pull request format to VFIO to avoid conflicts before acceptance.[1] https://lore.kernel.org/kvm/[email protected]/
[2] https://github.com/avihai1122/qemu/commits/mig_v2_precopyChanges from V0: https://www.spinics.net/lists/kvm/msg294247.html
Drop the first 2 patches that Alex merged already.
Refactor mlx5 implementation based on Jason's comments on V0, it includes the below:
* Refactor the PD usage to be aligned with the migration file life cycle.
* Refactor the MKEY usage to be aligned with the migration file life cycle.
* Refactor the migration file state.
* Use queue based data chunks to simplify the driver code.
* Use the FSM model on the target to simplify the driver code.
* Extend the driverpre_copyheader for future use.Yishai
Jason Gunthorpe (1):
vfio: Extend the device migration protocol withPRE_COPYShay Drory (3):
net/mlx5: Introduce ifc bits forpre_copyvfio/mlx5: Fallback toSTOP_COPYupon specificPRE_COPYerror vfio/mlx5: EnableMIGRATION_PRE_COPYflagYishai Hadas (10):
vfio/mlx5: Enforce a single SAVE command at a time vfio/mlx5: Refactor PD usage
vfio/mlx5: Refactor MKEY usage
vfio/mlx5: Refactor migration file state
vfio/mlx5: Refactor to use queue based data chunks vfio/mlx5: Introduce device transitions ofPRE_COPYvfio/mlx5: Introduce SW headers for migration states vfio/mlx5: Introduce vfio precopy ioctl implementation vfio/mlx5: Consider temporary end of stream as part ofPRE_COPYvfio/mlx5: Introduce multiple loads
Quoted 8 lines
drivers/vfio/pci/mlx5/cmd.c | 408 ++++++++++++++---- drivers/vfio/pci/mlx5/cmd.h | 93 ++++- drivers/vfio/pci/mlx5/main.c | 750 ++++++++++++++++++++++++++++------ drivers/vfio/vfio_main.c | 74 +++- include/linux/mlx5/mlx5_ifc.h | 14 +- include/uapi/linux/vfio.h | 122 +++++- 6 files changed, 1241 insertions(+), 220 deletions(-)
Hi Alex,
Any comments on V1 ? I would like to send V2 very soon with a small fix that was found in patch #9 (see my note in the list).
As no comments were published on the UAPI patch so far and the other code is mlx5 driver specific, I believe that V2 can be a merge candidate.
Let's try to complete review soon so that we can send a PR for the first net/mlx5 patch, the others should go later on via the vfio tree.
By that, we can have in kernel 6.2 feature complete for migration V2 in the UAPI area that were in V1 and we can focus on completing the QEMU side.
Thanks,
Yishai