s390/vfio-ap: Add live guest migration support
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v13 Latest14 patches 0 repliesNew
- v1214 patches 1 replySuperseded
- v1114 patches 7 repliesSuperseded
Earlier versions · 9
| Version | Posted | Patches | Replies | Status |
|---|---|---|---|---|
| v10 | 15 patches | 8 replies | Superseded | |
| v9 | 15 patches | 4 replies | Superseded | |
| v8 | 15 patches | 5 replies | Superseded | |
| v7 | 15 patches | 35 replies | Superseded | |
| v6 | 15 patches | 11 replies | Superseded | |
| v5 | 15 patches | 0 replies | Superseded | |
| v4 | 15 patches | 0 replies | Superseded | |
| v3 | 15 patches | 0 replies | Superseded | |
| v2 | 16 patches | 6 replies | Superseded |
Changes in v13 from the cover letter
~~~~~~~~~~~~~~~~~~~~~
Patch 12/14: Set up for vfio-ap device migration
* Fixed commit message
~ Removed bullets for initializing mig_state and
resuming_mig_data_complete
* vfio_ap_setup_vfio_device_migration()
~ Moved below the definition of vfio_ap_migration_ops ~ Removed forward declaration of vfio_ap_migration_ops
Patches in v13 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/14 | s390/vfio-ap: Provide function to get the number of queues assigned to mdevdrivers/ drivers/ | +10 | 0 replies | |
| 2/14 | s390/vfio-ap: Data structures for facilitating vfio device migrationdrivers/ drivers/ drivers/ | +131 | 0 replies | |
| 3/14 | s390/vfio-ap: Functions to initialize/release vfio device migration datadrivers/ drivers/ drivers/ | +137 | 0 replies | |
| 4/14 | s390/vfio-ap: Reset migration state on VFIO_DEVICE_RESET and device closedrivers/ drivers/ drivers/ | +49 | 0 replies | |
| 5/14 | s390/vfio-ap: Callback to get/set vfio device mig state during guest migrationdrivers/ | +144 | 0 replies | |
| 6/14 | s390/vfio-ap: Transition guest migration state from STOP to STOP_COPYdrivers/ | +72 | 0 replies | |
| 7/14 | s390/vfio-ap: File ops called to save the vfio device migration statedrivers/ | +264 | 0 replies | |
| 8/14 | s390/vfio-ap: Transition device migration state from STOP to RESUMINGdrivers/ | +72 | 0 replies | |
| 9/14 | s390/vfio-ap: File ops called to resume the vfio device migrationdrivers/ | +599 | 0 replies | |
| 10/14 | s390/vfio-ap: Transition device migration state to from/to STOPdrivers/ | +40 | 0 replies | |
| 11/14 | s390/vfio-ap: Callback to get the size of data to be migrated during guest migrationdrivers/ | +9 | 0 replies | |
| 12/14 | s390/vfio-ap: Set up for vfio-ap device migrationdrivers/ drivers/ drivers/ | +32 | 0 replies | |
| 13/14 | s390/vfio-ap: Add 'migration' feature to sysfs 'features' attributedrivers/ | +13 | 0 replies | |
| 14/14 | s390/vfio-ap: Add live guest migration chapter to vfio-ap.rstDocumentation/ | +544 | 0 replies |
Reviews trailers per patch
4 of 14 patches have a Reviewed-by or Acked-by.
| Patch | Jason J. Herne |
|---|---|
| 1/14 | R-b |
| 2/14 | R-b |
| 3/14 | |
| 4/14 | |
| 5/14 | |
| 6/14 | |
| 7/14 | |
| 8/14 | R-b |
| 9/14 | |
| 10/14 | |
| 11/14 | R-b |
| 12/14 | |
| 13/14 | |
| 14/14 |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [PATCH v13 00/14] s390/vfio-ap: Add live guest migration support
This patch series implements live guest migration support for KVM guests with s390 AP (Adjunct Processor) devices passed through via the VFIO mediated device framework.
Background
~~~~~~~~~~
The vfio-ap device driver differs from typical VFIO device drivers in that it does not virtualize a physical device. Instead, it manages AP configuration metadata identifying the AP adapters, domains, and control domains to which a guest will be granted access. These AP resources are configured by assigning them to a vfio-ap mediated device via its sysfs assignment interfaces. When the fd for the VFIO device is opened by userspace, the vfio_ap device driver sets the guest's AP configuration from the metadata stored with the mediated device. As such, the AP devices are not accessed directly through the vfio_ap driver, so the driver has no internal AP device state to migrate. What it does migrate is the AP configuration metadata of the source guest.
Implementation Approach
~~~~~~~~~~~~~~~~~~~~~~~
This series implements the VFIO migration protocol using the STOP_COPY migration flow. The key aspects are:
1. On transition of the migration state from STOP to STOP_COPY
- The vfio_ap device driver creates a filestream for userspace to use to read the guest's AP configuration from the mdev
2. During the STOP_COPY phase
- Userspace uses the filestream created in #1 to read the source guest's AP configuration
- The vfio_ap device driver copies the source guest's AP configuration information to userspace
3. On transition of the migration state from STOP to RESUMING
- The vfio_ap device driver creates a filestream for userspace to use to write the source guest's AP configuration information so it can be restored to the mdev on the destination host.
4. During the RESUMING phase
- Userspace uses the filestream created in #3 to send the source guest's AP configuration information to the vfio_ap device driver on the destination host.
- The vfio_ap device driver first verifies the source guest's AP configuration is compatible with the destination host's.
- The driver restores AP configuration to the mdev on the destination host which automatically hot plugs the AP resources identified therein.
5. Documentation
- Add live guest migration chapter to vfio-ap.rst
Compatibility Validation
~~~~~~~~~~~~~~~~~~~~~~~~
The series includes comprehensive validation to ensure source and destination AP configurations are compatible. For each queue, the following characteristics must match:
- AP type (target must be same or newer than source)
- Installed facilities (APSC, APQKM, AP4KC, SLCF)
- Operating mode (CCA, Accelerator, XCP)
- APXA facility setting
- Classification (native vs stateless functions)
- Queue usability (binding/associated state)
When incompatibilities are detected, migration fails with detailed error messages identifying the specific queue and characteristic that caused the failure.
Configuration Management
~~~~~~~~~~~~~~~~~~~~~~~~
This implementation does not prevent configuration changes during migration. Configuration stability is an orchestration-layer responsibility, consistent with other VFIO device types. The driver's role is to validate configurations and provide clear diagnostics when incompatibilities are detected, enabling orchestration tools to implement appropriate policies.
QEMU patches exploiting this series:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
https://lore.kernel.org/qemu-devel/[email protected]/
Change log v12 => v13:
~~~~~~~~~~~~~~~~~~~~~
Patch 12/14: Set up for vfio-ap device migration
* Fixed commit message
~ Removed bullets for initializing mig_state and
resuming_mig_data_complete
* vfio_ap_setup_vfio_device_migration()
~ Moved below the definition of vfio_ap_migration_ops ~ Removed forward declaration of vfio_ap_migration_ops
Anthony Krowiak (14):
s390/vfio-ap: Provide function to get the number of queues assigned to
mdev
s390/vfio-ap: Data structures for facilitating vfio device migration s390/vfio-ap: Functions to initialize/release vfio device migration
data
s390/vfio-ap: Reset migration state on VFIO_DEVICE_RESET and device
close
s390/vfio-ap: Callback to get/set vfio device mig state during guest
migration
s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY s390/vfio-ap: File ops called to save the vfio device migration state s390/vfio-ap: Transition device migration state from STOP to RESUMING s390/vfio-ap: File ops called to resume the vfio device migration s390/vfio-ap: Transition device migration state to from/to STOP s390/vfio-ap: Callback to get the size of data to be migrated during
guest migration
s390/vfio-ap: Set up for vfio-ap device migration s390/vfio-ap: Add 'migration' feature to sysfs 'features' attribute s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst
Documentation/arch/s390/vfio-ap.rst | 643 ++++++++-- drivers/s390/crypto/Makefile | 2 +- drivers/s390/crypto/vfio_ap_drv.c | 14 +- drivers/s390/crypto/vfio_ap_migration.c | 1454 +++++++++++++++++++++++ drivers/s390/crypto/vfio_ap_ops.c | 70 +- drivers/s390/crypto/vfio_ap_private.h | 12 + 6 files changed, 2089 insertions(+), 106 deletions(-) create mode 100644 drivers/s390/crypto/vfio_ap_migration.c
Signature · 1 line
2.53.0