s390/vfio-ap: Implement live guest migration of guests using AP devices
Subsystems (7)
Anthony Krowiak
maintainercover s390/vfio-ap: Implement live guest migration of guests using AP devices
This patch series implements live guest migration of a guest to which AP devices have been passed through. To better comprehend this design, one has to understand that VFIO AP mediated device is not used to provide userspace with direct access to a device as is the case with other devices that use the VFIO framework to pass them through to a guest. The sole purpose of the VFIO AP mediated device is to manage an AP configuration for a guest. An AP configuration is comprised of the AP adapter IDs (APID), AP queue indexes (APQI) and domain numbers of the control domains to which a guest will be granted access. Once the VFIO AP mediated device is attached to a guest, its AP configuration is set by the vfio_ap device driver. Once set, all access to the AP devices is handled by the s390 Interpretive Execution facility; in other words, the vfio_ap device driver plays no role in providing direct access to the AP devices in the guest's AP configuration.
The only role that the vfio_ap device driver plays in the migration process is to verify that the AP configuration for the source guest is compatible with the AP configuration of the destination guest. Incompatibility will result in a live guest migration failure. In order to be compatible, the following requirements must be met:
1. The destination guest will be started with the same QEMU command line as the source guest, so the mediated device supplying the AP configuration on both guests must have the same name (UUID).
2. The AP configuration assigned via the VFIO AP mediated device on both guests must be compatible. As such, each AP configuration must meet the following requirements:
* Both guests must have the same number of APQNs
* Each APQN assigned to the source guest must also be assigned to the destination guest
* Each APQN assigned to both guests must reference an AP queue with the same hardware capabilities
Note: There is a forthcoming consumer of this series which will be a QEMU
patch series is entitled:
'hw/vfio/ap: Implement live guest migration of guests using AP devices'
This design also adds a use case for enabling and disabling migration of guests to which AP devices have been passed through. To facilitate this, a new read/write sysfs 'migratable' attribute is added to the mediated device. This attribute specifies whether the vfio device is migratable (1) or not (0). When the value of this attribute is changed, the vfio_ap device driver will signal an eventfd to userspace. It is up to userspace to respond to the change by enabling or disabling migration of the guest to which the mediated device is attached. The operation will be rejected with a 'Device or resource busy' message if a migration is in progress.
Userspace must also have a means for retrieving the value of the sysfs 'migratable' attribute when the guest is started to initialize whether it can be migrated. For this, The VFIO_DEVICE_GET_INFO ioctl is used. The struct vfio_device_info object passed to the ioctl will be extended with a capability specifying the vfio device attributes. One of the attributes will contain the value of the mediated device's 'migratable' attribute.
Anthony Krowiak (23):
s390/vfio-ap: Store queue hardware info when probed s390/vfio-ap: Provide access to queue objects and related info s390/vfio-ap: Add header file for xfer of vfio device caps to
userspace
MAINTAINERS: Add new header file for the S390 VFIO AP DRIVER
maintainers
s390/vfio-ap: A sysfs 'migratable' attribute to enable/disable
migration of guest
s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute s390/vfio-ap: Signal event to enable/disable live guest migration s390/vfio-ap: Return value of sysfs migratable attribute from
VFIO_DEVICE_GET_INFO ioctl
s390/vfio-ap: Data structures for facilitating vfio device migration s390/vfio-ap: Initialize/release vfio device migration data
s390-vfio-ap: Callback to set vfio device mig state during guest
migration
s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY s390/vfio-ap: File ops called to save the vfio device migration state s390/vfio-ap: Transition device migration state from STOP to RESUMING s390/vfio-ap: File ops called to resume the vfio device migration s390/vfio-ap: Transition device migration state from RESUMING to STOP s390/vfio-ap: Transition device migration state from STOP_COPY to STOP s390/vfio-ap: Transition device migration state from STOP to RUNNING
and vice versa
s390-vfio-ap: Callback to get the current vfio device migration state s390/vfio-ap: Callback to get the size of data to be migrated during
guest migration
s390/vfio-ap: Provide API to query whether migration is in progress s390/vfio-ap: Disallow blocking migration in progress s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst
Documentation/arch/s390/vfio-ap.rst | 339 +++++-- MAINTAINERS | 1 + drivers/s390/crypto/Makefile | 2 +- drivers/s390/crypto/vfio_ap_drv.c | 4 +- drivers/s390/crypto/vfio_ap_migration.c | 1131 +++++++++++++++++++++++ drivers/s390/crypto/vfio_ap_ops.c | 263 +++++- drivers/s390/crypto/vfio_ap_private.h | 20 + include/uapi/linux/vfio.h | 2 + include/uapi/linux/vfio_ap.h | 34 + 9 files changed, 1685 insertions(+), 111 deletions(-) create mode 100644 drivers/s390/crypto/vfio_ap_migration.c create mode 100644 include/uapi/linux/vfio_ap.h
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 1/24 fixup! KVM: s390: Remove non-atomic dat_crstep_xchg()
From: Claudio Imbrenda <[email protected]>
--- arch/s390/kvm/gmap.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/s390/kvm/gmap.h +1 -1
@@ -198,7 +198,7 @@ static inline bool __must_check _gmap_crstep_xchg_atomic(struct gmap *gmap, unio union crste oldcrste, union crste newcrste, gfn_t gfn, bool needs_lock) { - unsigned long align = 8 + (is_pmd(*crstep) ? 0 : 11); + unsigned long align = is_pmd(*crstep) ? _PAGE_ENTRIES : _PAGE_ENTRIES * _CRST_ENTRIES;lockdep_assert_held (&gmap->kvm->mmu_lock);if (!needs_lock)
Signature · 1 line
2.52.0
Re: [PATCH v1 01/24] fixup! KVM: s390: Remove non-atomic dat_crstep_xchg()
Quoted 3 lines · Anthony Krowiak
From: Claudio Imbrenda <[email protected]>
Signed-off-by Claudio Imbrenda <[email protected]>
Tony/Claudio, I suspect this was included on accident and will be handled separately and dropped from this series? I don't even see it mentioned in the cover letter.
If intentional and intended to be part of this series, then please add a proper commit message.
Quoted 17 lines
--- arch/s390/kvm/gmap.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/s390/kvm/gmap.h +1
-1@@ -198,7 +198,7 @@ static inline bool __must_check _gmap_crstep_xchg_atomic(struct gmap *gmap, unio union crste oldcrste, union crste newcrste, gfn_t gfn, bool needs_lock) { - unsigned long align = 8 + (is_pmd(*crstep) ? 0 : 11); + unsigned long align = is_pmd(*crstep) ? _PAGE_ENTRIES : _PAGE_ENTRIES * _CRST_ENTRIES;lockdep_assert_held (&gmap->kvm->mmu_lock);if (!needs_lock)
Anthony Krowiak
maintainerRe: [PATCH v1 01/24] fixup! KVM: s390: Remove non-atomic dat_crstep_xchg()
Quoted 10 lines · Matthew Rosato
Quoted 3 lines · Anthony Krowiak
From: Claudio Imbrenda <[email protected]>
Signed-off-by Claudio Imbrenda <[email protected]>Tony/Claudio, I suspect this was included on accident and will be handled separately and dropped from this series? I don't even see it mentioned in the cover letter.
If intentional and intended to be part of this series, then please add a proper commit message.
This definitely does not belong in this series. I accidentally included it when building the patch series.
Quoted 18 lines
Quoted 18 lines
--- arch/s390/kvm/gmap.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/s390/kvm/gmap.h +1
-1@@ -198,7 +198,7 @@ static inline bool __must_check _gmap_crstep_xchg_atomic(struct gmap *gmap, unio union crste oldcrste, union crste newcrste, gfn_t gfn, bool needs_lock) { - unsigned long align = 8 + (is_pmd(*crstep) ? 0 : 11); + unsigned long align = is_pmd(*crstep) ? _PAGE_ENTRIES : _PAGE_ENTRIES * _CRST_ENTRIES;lockdep_assert_held (&gmap->kvm->mmu_lock);if (!needs_lock)
Anthony Krowiak
maintainerPATCH 2/24 s390/vfio-ap: Store queue hardware info when probed
In reply to Anthony KrowiakStore the output of the PQAP(TAPQ) command with the struct vfio_ap_queue object when a queue device is probed. This data can be retrieved from the object passed to the probe callback.
--- drivers/s390/crypto/vfio_ap_ops.c | 5 ++++- drivers/s390/crypto/vfio_ap_private.h | 1 + 2 files changed, 5 insertions(+), 1 deletion(-)
drivers/s390/crypto/vfio_ap_ops.c +4 -1
@@ -2419,6 +2419,7 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev)struct vfio_ap_queue *q;DECLARE_BITMAP (apm_filtered,AP_DEVICES );struct ap_matrix_mdev *matrix_mdev; +struct ap_queue *ap_queue; ret =sysfs_create_group (&apdev->device.kobj, &vfio_queue_attr_group);if (ret)
@@ -2430,8 +2431,10 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev) goto err_remove_group; } - q->apqn =to_ap_queue (&apdev->device)->qid; + ap_queue =to_ap_queue (&apdev->device); + q->apqn = ap_queue->qid; q->saved_isc =VFIO_AP_ISC_INVALID ; +memcpy (&q->hwinfo, &ap_queue->card->hwinfo,sizeof (q->hwinfo));memset (&q->reset_status,0 ,sizeof (q->reset_status));INIT_WORK (&q->reset_work, apq_reset_check); matrix_mdev =get_update_locks_by_apqn (q->apqn);
drivers/s390/crypto/vfio_ap_private.h +1 -0
@@ -150,6 +150,7 @@ struct vfio_ap_queue {struct list_head reset_qnode;struct ap_queue_status reset_status;struct work_struct reset_work; +struct ap_tapq_hwinfo hwinfo; };int vfio_ap_mdev_register (void );
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 3/24 s390/vfio-ap: Provide access to queue objects and related info
In reply to Anthony KrowiakProvide the following:
1. A function that returns a struct vfio_ap_queue object representing a queue that is bound to the vfio_ap device driver and is assigned to an mdev
2. A function that returns the number of queues that are or will be passed through to a guest when the mdev is attached to a guest.
There is already a function for #1; however, it is a static function in drivers/s390/crypto/vfio_ap_ops.c. The function will be defined in the drivers/s390/crypto/vfio_ap_private.h file and made non-static. Note that it would probably make sense to ensure that the mdevs_lock is held for this function, so a call to lockdep_assert_held(&matrix_dev->mdevs_lock) will be added to the function implementation.
--- drivers/s390/crypto/vfio_ap_ops.c | 20 +++++++++++++++++--- drivers/s390/crypto/vfio_ap_private.h | 4 ++++ 2 files changed, 21 insertions(+), 3 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +17 -3
@@ -203,12 +203,12 @@ static inline void get_update_locks_for_queue(struct vfio_ap_queue *q)* Return: the pointer to the vfio_ap_queue struct representing the queue or * NULL if the queue is not assigned to @matrix_mdev */ -static struct vfio_ap_queue *vfio_ap_mdev_get_queue ( -struct ap_matrix_mdev *matrix_mdev, -int apqn) +struct vfio_ap_queue *vfio_ap_mdev_get_queue (struct ap_matrix_mdev *matrix_mdev,int apqn) {struct vfio_ap_queue *q; +lockdep_assert_held (&matrix_dev->mdevs_lock); +hash_for_each_possible (matrix_mdev->qtable.queues, q, mdev_qnode, apqn) {if (q && q->apqn == apqn)
@@ -2098,6 +2098,20 @@ static void vfio_ap_mdev_request(struct vfio_device *vdev, unsigned int count)release_update_locks_for_mdev (matrix_mdev); } +int vfio_ap_mdev_get_num_queues (struct ap_matrix *ap_matrix) +{ +unsigned long apid, apqi; +int num_queues =0 ; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + +for_each_set_bit_inv (apid, ap_matrix->apm,AP_DEVICES ) +for_each_set_bit_inv (apqi, ap_matrix->aqm,AP_DOMAINS ) + num_queues++; + +return num_queues; +} +static int vfio_ap_mdev_get_device_info (unsigned long arg) {unsigned long minsz;
drivers/s390/crypto/vfio_ap_private.h +4 -0
@@ -153,6 +153,10 @@ struct vfio_ap_queue {struct ap_tapq_hwinfo hwinfo; }; +struct vfio_ap_queue *vfio_ap_mdev_get_queue (struct ap_matrix_mdev *matrix_mdev,int apqn); + +int vfio_ap_mdev_get_num_queues (struct ap_matrix *ap_matrix); +int vfio_ap_mdev_register (void );void vfio_ap_mdev_unregister (void );
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 4/24 s390/vfio-ap: Add header file for xfer of vfio device caps to userspace
In reply to Anthony KrowiakAdds the vfio_ap.h file used to transfer vfio device capabilities to userspace in response to the VFIO_DEVICE_GET_INFO ioctl. The device capabilities includes an object specifying the vfio device attributes. This object contains a flag indicating whether the vfio device is migratable.
Note: The migratable flag could have been returned as a capability
unto itself, but returning it as one attribute of a set of attributes allows for this capability to easily be extended to include additional attributes in the future if need be.
--- include/uapi/linux/vfio_ap.h | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 include/uapi/linux/vfio_ap.h
include/uapi/linux/vfio_ap.h new file +34 -0
@@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* +* Interfaces for vfio-ap +* +* Copyright IBM Corp. 2025 +* +* Author(s): Anthony Krowiak <[email protected]> +*/ + +#ifndef INCLUDE_UAPI_LINUX_VFIO_AP_H_ +#define INCLUDE_UAPI_LINUX_VFIO_AP_H_ + +#include <linux/types.h> +#include <linux/vfio.h> + +#define VFIO_DEVINFO_CAP_AP_ATTRS_VERSION 1 +#define VFIO_DEVINFO_CAP_AP_ATTRS_ID 1 + +/** +* struct vfio_device_info_cap_ap_attrs - vfio device capability object used +* supply the device attributes to +* userspace. +* +* @header: the header used to extend the struct vfio_device_info object +* to provide the vfio device attributes to userspace +* @migratable: indicates whether the vfio device is migratable (1) or +* not (0). +*/ +struct vfio_device_info_cap_ap_attrs { +struct vfio_info_cap_header header; +int migratable; +}; + +#endif /* INCLUDE_UAPI_LINUX_VFIO_AP_H_ */
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 5/24 MAINTAINERS: Add new header file for the S390 VFIO AP DRIVER maintainers
In reply to Anthony KrowiakAdd new header file include/uapi/linux/vfio_ap.h for the S390 VFIO AP DRIVER maintainers.
--- MAINTAINERS | 1 + 1 file changed, 1 insertion(+)
MAINTAINERS +1 -0
@@ -23339,6 +23339,7 @@ L: [email protected] S: Supported F: Documentation/arch/s390/vfio-ap* F: drivers/s390/crypto/vfio_ap* +F: include/uapi/linux/vfio_ap.h S390 VFIO-CCW DRIVER M: Eric Farman <[email protected]>
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 6/24 s390/vfio-ap: A sysfs 'migratable' attribute to enable/disable migration of guest
In reply to Anthony KrowiakThis patch creates a new R/W sysfs attribute of the vfio-ap mediated device that will ultimately be used to enable and disable migration of the guest to which the mediated device is attached. The value for this attribute is an int to be stored with the mediated device. A value of 1 indicates the vfio device can be migrated, and a value of 0 indicates migration is blocked. The initial value will be set to 0.
--- drivers/s390/crypto/vfio_ap_ops.c | 46 +++++++++++++++++++++++++++ drivers/s390/crypto/vfio_ap_private.h | 3 ++ 2 files changed, 49 insertions(+)
drivers/s390/crypto/vfio_ap_ops.c +46 -0
@@ -32,6 +32,8 @@#define AP_RESET_INTERVAL 20/* Reset sleep interval (20ms) */ +#define VFIO_AP_DEVICE_MIGRATABLE_DEFAULT 0 +static int vfio_ap_mdev_reset_queues (struct ap_matrix_mdev *matrix_mdev);static int vfio_ap_mdev_reset_qlist (struct list_head *qlist);static struct vfio_ap_queue *vfio_ap_find_queue (int apqn);
@@ -802,6 +804,7 @@ static int vfio_ap_mdev_probe(struct mdev_device *mdev) goto err_put_vdev; matrix_mdev->req_trigger =NULL ; matrix_mdev->cfg_chg_trigger =NULL ; + matrix_mdev->migratable =VFIO_AP_DEVICE_MIGRATABLE_DEFAULT ;dev_set_drvdata (&mdev->dev, matrix_mdev);mutex_lock (&matrix_dev->mdevs_lock);list_add (&matrix_mdev->node, &matrix_dev->mdev_list);
@@ -1785,6 +1788,48 @@ static ssize_t ap_config_store(struct device *dev, struct device_attribute *attr }static DEVICE_ATTR_RW (ap_config); +static ssize_t migratable_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct ap_matrix_mdev *matrix_mdev; +size_t nchars; + +mutex_lock (&matrix_dev->mdevs_lock); + matrix_mdev =dev_get_drvdata (dev); + nchars =sysfs_emit (buf,"%d\n" , matrix_mdev->migratable); +mutex_unlock (&matrix_dev->mdevs_lock); + +return nchars; +} + +static ssize_t migratable_store (struct device *dev,struct device_attribute *attr, +const char *buf,size_t count) +{ +struct ap_matrix_mdev *matrix_mdev; +int migratable; +int ret; + +mutex_lock (&matrix_dev->mdevs_lock); + matrix_mdev =dev_get_drvdata (dev); + + ret =kstrtoint (buf,10 , &migratable); +if (ret) + goto out_unlock; + +if (migratable !=0 && migratable !=1 ) { + ret = -EINVAL ; + goto out_unlock; + } + + matrix_mdev->migratable = migratable; + ret = count; +out_unlock: +mutex_unlock (&matrix_dev->mdevs_lock); + +return ret; +} +static DEVICE_ATTR_RW (migratable); +static struct attribute *vfio_ap_mdev_attrs[] = { &dev_attr_assign_adapter.attr, &dev_attr_unassign_adapter.attr,
@@ -1796,6 +1841,7 @@ static struct attribute *vfio_ap_mdev_attrs[] = { &dev_attr_control_domains.attr, &dev_attr_matrix.attr, &dev_attr_guest_matrix.attr, + &dev_attr_migratable.attr, NULL, };
drivers/s390/crypto/vfio_ap_private.h +3 -0
@@ -104,6 +104,8 @@ struct ap_queue_table {* PQAP(AQIC) instruction. * @mdev: the mediated device * @qtable: table of queues (struct vfio_ap_queue) assigned to the mdev +* @migratable: an int value indicating whether the vfio device can be +* migrated (1) or not (0) * @req_trigger eventfd ctx for signaling userspace to return a device * @cfg_chg_trigger eventfd ctx to signal AP config changed to userspace * @apm_add: bitmap of APIDs added to the host's AP configuration
@@ -120,6 +122,7 @@ struct ap_matrix_mdev {crypto_hook pqap_hook;struct mdev_device *mdev;struct ap_queue_table qtable; +int migratable;struct eventfd_ctx *req_trigger;struct eventfd_ctx *cfg_chg_trigger;DECLARE_BITMAP (apm_add,AP_DEVICES );
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 7/24 s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute
In reply to Anthony KrowiakAdd the 'migratable' feature to the features provided by the sysfs /sys/devices/vfio_ap/matrix/feature attribute to indicate that migration of vfio devices is supported.
--- drivers/s390/crypto/vfio_ap_drv.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/s390/crypto/vfio_ap_drv.c +3 -1
@@ -26,9 +26,11 @@ MODULE_LICENSE("GPL v2");struct ap_matrix_dev *matrix_dev;debug_info_t *vfio_ap_dbf_info; +#define FEATURES "guest_matrix hotplug ap_config migratable" +static ssize_t features_show (struct device *dev,struct device_attribute *attr,char *buf) { -return sysfs_emit (buf,"guest_matrix hotplug ap_config\n" ); +return sysfs_emit (buf,"%s\n" ,FEATURES ); }static DEVICE_ATTR_RO (features);
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 8/24 s390/vfio-ap: Signal event to enable/disable live guest migration
In reply to Anthony KrowiakDefines two new eventfd contexts for signaling userspace to enable and disable vfio device migration. The IRQ indexes defined for these events are defined in an enum in the include/uapi/linux/vfio.h header file:
VFIO_AP_ENABLE_MIG_IRQ_INDEX: enable vfio device migration VFIO_AP_DISABLE_MIG_IRQ_INDEX: disable vfio device migration
Information about those IRQs can be retrieved from userspace via the VFIO_DEVICE_GET_IRQ_INFO ioctl. These events are initialized by the vfio_ap device driver via the in response to the VFIO_DEVICE_SET_IRQ_INFO ioctl. The eventfd for these IRQs will be signaled whenever the sysfs migratable attribute of the mediated device is changed.
--- drivers/s390/crypto/vfio_ap_ops.c | 70 +++++++++++++++++++++++++-- drivers/s390/crypto/vfio_ap_private.h | 6 +++ include/uapi/linux/vfio.h | 2 + 3 files changed, 75 insertions(+), 3 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +67 -3
@@ -658,6 +658,18 @@ static void signal_guest_ap_cfg_changed(struct ap_matrix_mdev *matrix_mdev)eventfd_signal (matrix_mdev->cfg_chg_trigger); } +static void signal_enable_migration (struct ap_matrix_mdev *matrix_mdev) +{ +if (matrix_mdev->enable_mig_trigger) +eventfd_signal (matrix_mdev->enable_mig_trigger); +} + +static void signal_disable_migration (struct ap_matrix_mdev *matrix_mdev) +{ +if (matrix_mdev->disable_mig_trigger) +eventfd_signal (matrix_mdev->disable_mig_trigger); +} +static void vfio_ap_mdev_update_guest_apcb (struct ap_matrix_mdev *matrix_mdev) {if (matrix_mdev->kvm) {
@@ -805,6 +817,8 @@ static int vfio_ap_mdev_probe(struct mdev_device *mdev) matrix_mdev->req_trigger =NULL ; matrix_mdev->cfg_chg_trigger =NULL ; matrix_mdev->migratable =VFIO_AP_DEVICE_MIGRATABLE_DEFAULT ; + matrix_mdev->enable_mig_trigger =NULL ; + matrix_mdev->disable_mig_trigger =NULL ;dev_set_drvdata (&mdev->dev, matrix_mdev);mutex_lock (&matrix_dev->mdevs_lock);list_add (&matrix_mdev->node, &matrix_dev->mdev_list);
@@ -1821,7 +1835,18 @@ static ssize_t migratable_store(struct device *dev, struct device_attribute *att goto out_unlock; } +if (matrix_mdev->migratable == migratable) { + ret = count; + goto out_unlock; + } + matrix_mdev->migratable = migratable; + +if (matrix_mdev->migratable) +signal_enable_migration (matrix_mdev); +else +signal_disable_migration (matrix_mdev); + ret = count; out_unlock:mutex_unlock (&matrix_dev->mdevs_lock);
@@ -2193,10 +2218,9 @@ static ssize_t vfio_ap_get_irq_info(unsigned long arg)switch (info.index) {case VFIO_AP_REQ_IRQ_INDEX : - info.count =1 ; - info.flags =VFIO_IRQ_INFO_EVENTFD ; -break ;case VFIO_AP_CFG_CHG_IRQ_INDEX : +case VFIO_AP_ENABLE_MIG_IRQ_INDEX : +case VFIO_AP_DISABLE_MIG_IRQ_INDEX : info.count =1 ; info.flags =VFIO_IRQ_INFO_EVENTFD ;break ;
@@ -2296,6 +2320,40 @@ static int vfio_ap_set_cfg_change_irq(struct ap_matrix_mdev *matrix_mdev, unsignreturn 0 ; } +static int vfio_ap_set_mig_irq (struct eventfd_ctx **mig_trigger, +unsigned long arg) +{ +s32 fd; +void __user *data; +unsigned long minsz; +struct eventfd_ctx *mig_ctx; + + minsz = offsetofend(struct vfio_irq_set, count); + data = (void __user *)(arg + minsz); + +if (get_user (fd, (s32 __user *)data)) +return -EFAULT ; + +if (fd ==-1 ) { +if (*mig_trigger) +eventfd_ctx_put (*mig_trigger); + *mig_trigger =NULL ; + }else if (fd >=0 ) { + mig_ctx =eventfd_ctx_fdget (fd); +if (IS_ERR (mig_ctx)) +return PTR_ERR (mig_ctx); + +if (*mig_trigger) +eventfd_ctx_put (*mig_trigger); + + *mig_trigger = mig_ctx; + }else { +return -EINVAL ; + } + +return 0 ; +} +static int vfio_ap_set_irqs (struct ap_matrix_mdev *matrix_mdev,unsigned long arg) {
@@ -2313,6 +2371,12 @@ static int vfio_ap_set_irqs(struct ap_matrix_mdev *matrix_mdev,return vfio_ap_set_request_irq (matrix_mdev, arg);case VFIO_AP_CFG_CHG_IRQ_INDEX :return vfio_ap_set_cfg_change_irq (matrix_mdev, arg); +case VFIO_AP_ENABLE_MIG_IRQ_INDEX : +return vfio_ap_set_mig_irq (&matrix_mdev->enable_mig_trigger, + arg); +case VFIO_AP_DISABLE_MIG_IRQ_INDEX : +return vfio_ap_set_mig_irq (&matrix_mdev->disable_mig_trigger, + arg);default :return -EINVAL ; }
drivers/s390/crypto/vfio_ap_private.h +6 -0
@@ -108,6 +108,10 @@ struct ap_queue_table {* migrated (1) or not (0) * @req_trigger eventfd ctx for signaling userspace to return a device * @cfg_chg_trigger eventfd ctx to signal AP config changed to userspace +* @enable_mig_trigger: eventfd ctx to signal userspace to enable vfio device +* migration +* @disable_mig_trigger: eventfd ctx to signal userspace to disable vfio device +* migration * @apm_add: bitmap of APIDs added to the host's AP configuration * @aqm_add: bitmap of APQIs added to the host's AP configuration * @adm_add: bitmap of control domain numbers added to the host's AP
@@ -125,6 +129,8 @@ struct ap_matrix_mdev {int migratable;struct eventfd_ctx *req_trigger;struct eventfd_ctx *cfg_chg_trigger; +struct eventfd_ctx *enable_mig_trigger; +struct eventfd_ctx *disable_mig_trigger;DECLARE_BITMAP (apm_add,AP_DEVICES );DECLARE_BITMAP (aqm_add,AP_DOMAINS );DECLARE_BITMAP (adm_add,AP_DOMAINS );
include/uapi/linux/vfio.h +2 -0
@@ -673,6 +673,8 @@ enum {enum {VFIO_AP_REQ_IRQ_INDEX ,VFIO_AP_CFG_CHG_IRQ_INDEX , +VFIO_AP_ENABLE_MIG_IRQ_INDEX , +VFIO_AP_DISABLE_MIG_IRQ_INDEX ,VFIO_AP_NUM_IRQS };
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 9/24 s390/vfio-ap: Return value of sysfs migratable attribute from VFIO_DEVICE_GET_INFO ioctl
In reply to Anthony KrowiakExtends the 'vfio_device_info' object passed to the VFIO_DEVICE_GET_INFO ioctl to include a vfio device capability specifying the value of the mdev's sysfs 'migratable' attribute. This value may be used to enable or block live guest migration when the vfio device is realized in userspace.
--- drivers/s390/crypto/vfio_ap_ops.c | 75 ++++++++++++++++++++++++++++++- 1 file changed, 73 insertions(+), 2 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +73 -2
@@ -19,6 +19,7 @@#include <linux/uuid.h> #include <asm/kvm.h> #include <asm/zcrypt.h> +#include <uapi/linux/vfio_ap.h> #include "vfio_ap_private.h" #include "vfio_ap_debug.h"
@@ -2183,11 +2184,75 @@ int vfio_ap_mdev_get_num_queues(struct ap_matrix *ap_matrix)return num_queues; } -static int vfio_ap_mdev_get_device_info (unsigned long arg) +static int vfio_ap_add_caps_to_dev_info (struct vfio_device_info *dev_info, +unsigned long arg, +struct vfio_info_cap *caps) { +lockdep_assert_held (&matrix_dev->mdevs_lock); + +if (caps->size) { + dev_info->flags |=VFIO_DEVICE_FLAGS_CAPS ; +if (dev_info->argsz <sizeof (*dev_info) + caps->size) { + dev_info->argsz =sizeof (*dev_info) + caps->size; + }else { +vfio_info_cap_shift (caps,sizeof (*dev_info)); + +if (copy_to_user ((void __user *)arg +sizeof (*dev_info), + caps->buf, caps->size)) { +kfree (caps->buf); +return -EFAULT ; + } + + dev_info->cap_offset =sizeof (*dev_info); + } + +kfree (caps->buf); + } + +return 0 ; +} + +static int vfio_ap_add_attrs_cap_to_caps (struct ap_matrix_mdev *matrix_mdev, +struct vfio_info_cap *caps) +{ +struct vfio_device_info_cap_ap_attrs cap; + +lockdep_assert_held (matrix_dev->mdevs_lock); + cap.migratable = matrix_mdev->migratable; + cap.header.id =VFIO_DEVINFO_CAP_AP_ATTRS_ID ; + cap.header.version =VFIO_DEVINFO_CAP_AP_ATTRS_VERSION ; + +return vfio_info_add_capability (caps, &cap.header,sizeof (cap)); +} + +static int vfio_ap_add_caps (struct ap_matrix_mdev *matrix_mdev, +unsigned long arg, +struct vfio_device_info *dev_info) +{ +struct vfio_info_cap caps = { .buf =NULL , .size =0 }; +int ret =0 ; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + + ret =vfio_ap_add_attrs_cap_to_caps (matrix_mdev, &caps); +if (ret) +return ret; + + ret =vfio_ap_add_caps_to_dev_info (dev_info, arg, &caps); +if (ret) +return ret; + +return 0 ; +} + +static int vfio_ap_mdev_get_device_info (struct ap_matrix_mdev *matrix_mdev, +unsigned long arg) +{ +int ret;unsigned long minsz;struct vfio_device_info info; +lockdep_assert_held (&matrix_dev->mdevs_lock); minsz = offsetofend(struct vfio_device_info, num_irqs);if (copy_from_user (&info, (void __user *)arg, minsz))
@@ -2200,6 +2265,12 @@ static int vfio_ap_mdev_get_device_info(unsigned long arg) info.num_regions =0 ; info.num_irqs =VFIO_AP_NUM_IRQS ; + ret =vfio_ap_add_caps (matrix_mdev, arg, &info); +if (ret) +return ret; + + minsz =min_t (size_t, info.argsz,sizeof (info)); +return copy_to_user ((void __user *)arg, &info, minsz) ? -EFAULT :0 ; }
@@ -2395,7 +2466,7 @@ static ssize_t vfio_ap_mdev_ioctl(struct vfio_device *vdev,mutex_lock (&matrix_dev->mdevs_lock);switch (cmd) {case VFIO_DEVICE_GET_INFO : - ret =vfio_ap_mdev_get_device_info (arg); + ret =vfio_ap_mdev_get_device_info (matrix_mdev, arg);break ;case VFIO_DEVICE_RESET : ret =vfio_ap_mdev_reset_queues (matrix_mdev);
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 10/24 s390/vfio-ap: Data structures for facilitating vfio device migration
In reply to Anthony KrowiakCreates the data structures used to facilitate state transitions during vfio device migration.
--- drivers/s390/crypto/Makefile | 2 +- drivers/s390/crypto/vfio_ap_migration.c | 68 +++++++++++++++++++++++++ drivers/s390/crypto/vfio_ap_private.h | 2 + 3 files changed, 71 insertions(+), 1 deletion(-) create mode 100644 drivers/s390/crypto/vfio_ap_migration.c
drivers/s390/crypto/Makefile +1 -1
@@ -34,5 +34,5 @@ pkey-uv-objs := pkey_uv.o obj-$(CONFIG_PKEY_UV) += pkey-uv.o # adjunct processor matrix -vfio_ap-objs := vfio_ap_drv.o vfio_ap_ops.o +vfio_ap-objs := vfio_ap_drv.o vfio_ap_ops.o vfio_ap_migration.o obj-$(CONFIG_VFIO_AP) += vfio_ap.o
drivers/s390/crypto/vfio_ap_migration.c new file +68 -0
68 lines not on this page · raw message
drivers/s390/crypto/vfio_ap_private.h +2 -0
@@ -116,6 +116,7 @@ struct ap_queue_table {* @aqm_add: bitmap of APQIs added to the host's AP configuration * @adm_add: bitmap of control domain numbers added to the host's AP * configuration +* @mig_data: vfio device migration data */ struct ap_matrix_mdev {struct vfio_device vdev;
@@ -134,6 +135,7 @@ struct ap_matrix_mdev {DECLARE_BITMAP (apm_add,AP_DEVICES );DECLARE_BITMAP (aqm_add,AP_DOMAINS );DECLARE_BITMAP (adm_add,AP_DOMAINS ); +void *mig_data; };/**
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 11/24 s390/vfio-ap: Initialize/release vfio device migration data
In reply to Anthony KrowiakProvides the functions that initialize and release the data structures used during live guest migration. These functions should
be called when the mdev fd is opened and closed respectively. The function to release the data structures shall also be invoked when the mdev is released.
--- drivers/s390/crypto/vfio_ap_migration.c | 94 +++++++++++++++++++++++++ drivers/s390/crypto/vfio_ap_ops.c | 41 +++++++++-- drivers/s390/crypto/vfio_ap_private.h | 3 + 3 files changed, 131 insertions(+), 7 deletions(-)
drivers/s390/crypto/vfio_ap_migration.c +94 -0
@@ -4,6 +4,7 @@* * Copyright IBM Corp. 2025 */ +#include <linux/file.h> #include "uapi/linux/vfio_ap.h" #include "vfio_ap_private.h"
@@ -66,3 +67,96 @@ struct vfio_ap_migration_file {struct vfio_ap_config *ap_config;size_t config_sz; }; + +static void vfio_ap_disable_file (struct vfio_ap_migration_file *migf) +{ +lockdep_assert_held (&matrix_dev->mdevs_lock); + migf->matrix_mdev =NULL ; + migf->disabled = true; + migf->filp->f_pos =0 ; +} + +static void vfio_ap_release_mig_files (struct ap_matrix_mdev *matrix_mdev) +{ +struct vfio_ap_migration_data *mig_data; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + mig_data = matrix_mdev->mig_data; + +if (mig_data->resuming_migf) { +vfio_ap_disable_file (mig_data->resuming_migf); +fput (mig_data->resuming_migf->filp); + mig_data->resuming_migf =NULL ; + } + +if (mig_data->saving_migf) { +vfio_ap_disable_file (mig_data->saving_migf); +fput (mig_data->saving_migf->filp); + mig_data->saving_migf =NULL ; + } +} + +static struct file *vfio_ap_set_state (struct vfio_device *vdev, +enum vfio_device_mig_state new_state) +{ +return NULL ; +} + +static int vfio_ap_get_state (struct vfio_device *vdev, +enum vfio_device_mig_state *current_state) +{ +return -EOPNOTSUPP ; +} + +static int vfio_ap_get_data_size (struct vfio_device *vdev, +unsigned long *stop_copy_length) +{ +return -EOPNOTSUPP ; +} + +static const struct vfio_migration_ops vfio_ap_migration_ops = { + .migration_set_state = vfio_ap_set_state, + .migration_get_state = vfio_ap_get_state, + .migration_get_data_size = vfio_ap_get_data_size, +}; + +/** +* vfio_ap_init_migration_data - initialize migration data and functions +* +* @matrix_mdev: pointer to object containing the mdev state +* +* Return: zero if initialization is successful; otherwise, returns a error. +*/ +int vfio_ap_init_migration_data (struct ap_matrix_mdev *matrix_mdev) +{ +struct vfio_ap_migration_data *mig_data; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + + mig_data = kzalloc_obj(struct vfio_ap_migration_data, GFP_KERNEL); +if (!mig_data) +return -ENOMEM ; + + mig_data->mig_state =VFIO_DEVICE_STATE_STOP ; + matrix_mdev->vdev.migration_flags =VFIO_MIGRATION_STOP_COPY ; + matrix_mdev->vdev.mig_ops = &vfio_ap_migration_ops; + matrix_mdev->mig_data = mig_data; + +return 0 ; +} + +/** +* vfio_ap_release_migration_data: reclaim private migration data +* +* @vdev: pointer to the mdev +*/ +void vfio_ap_release_migration_data (struct ap_matrix_mdev *matrix_mdev) +{ +lockdep_assert_held (&matrix_dev->mdevs_lock); +if (!matrix_mdev->mig_data) +return ; + +vfio_ap_release_mig_files (matrix_mdev); +kfree (matrix_mdev->mig_data); + matrix_mdev->mig_data =NULL ; +}
drivers/s390/crypto/vfio_ap_ops.c +34 -7
77 lines not on this page · raw message
drivers/s390/crypto/vfio_ap_private.h +3 -0
@@ -181,4 +181,7 @@ void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info,void vfio_ap_on_scan_complete (struct ap_config_info *new_config_info,struct ap_config_info *old_config_info); +int vfio_ap_init_migration_data (struct ap_matrix_mdev *matrix_mdev); +void vfio_ap_release_migration_data (struct ap_matrix_mdev *matrix_mdev); +#endif /* _VFIO_AP_PRIVATE_H_ */
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 12/24 s390-vfio-ap: Callback to set vfio device mig state during guest migration
In reply to Anthony KrowiakImplements the callback that sets a new migration state of a vfio device during live migration of guests with pass-through access to AP devices. This callback is mandatory for VFIO_DEVICE_FEATURE_MIGRATION support.
The function pointer for this callback is specified via the 'migration_set_state' field of the 'vfio_migration_ops' structure which is stored with the VFIO device when the 'vfio_device' structure representing the mediated device is initialized.
--- drivers/s390/crypto/vfio_ap_migration.c | 87 ++++++++++++++++++++++++- 1 file changed, 86 insertions(+), 1 deletion(-)
drivers/s390/crypto/vfio_ap_migration.c +86 -1
@@ -96,10 +96,95 @@ static void vfio_ap_release_mig_files(struct ap_matrix_mdev *matrix_mdev) } } +static struct file * +vfio_ap_transition_to_state (struct ap_matrix_mdev *matrix_mdev, +enum vfio_device_mig_state new_state) +{ +struct vfio_ap_migration_data *mig_data; +enum vfio_device_mig_state cur_state; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + mig_data = matrix_mdev->mig_data; + cur_state = mig_data->mig_state; +dev_dbg (matrix_mdev->vdev.dev,"%s: %d -> %d\n" , __func__, cur_state, + new_state); + +if (cur_state ==VFIO_DEVICE_STATE_STOP && + new_state ==VFIO_DEVICE_STATE_STOP_COPY ) { +/* TODO */ +return ERR_PTR (-EOPNOTSUPP ); + } + +if (cur_state ==VFIO_DEVICE_STATE_STOP && + new_state ==VFIO_DEVICE_STATE_RESUMING ) { +/* TODO */ +return ERR_PTR (-EOPNOTSUPP ); + } + +if (cur_state ==VFIO_DEVICE_STATE_RESUMING && + new_state ==VFIO_DEVICE_STATE_STOP ) { +/* TODO */ +return ERR_PTR (-EOPNOTSUPP ); + } + +if (cur_state ==VFIO_DEVICE_STATE_STOP_COPY && + new_state ==VFIO_DEVICE_STATE_STOP ) { +/* TODO */ +return ERR_PTR (-EOPNOTSUPP ); + } + +if ((cur_state ==VFIO_DEVICE_STATE_STOP && + new_state ==VFIO_DEVICE_STATE_RUNNING ) || + (cur_state ==VFIO_DEVICE_STATE_RUNNING && + new_state ==VFIO_DEVICE_STATE_STOP )) { +/* TODO */ +return ERR_PTR (-EOPNOTSUPP ); + } + +/* vfio_mig_get_next_state() does not use arcs other than the above */ +WARN_ON (true); + +return ERR_PTR (-EINVAL ); +} +static struct file *vfio_ap_set_state (struct vfio_device *vdev,enum vfio_device_mig_state new_state) { -return NULL ; +int ret; +struct file *filp =NULL ; +struct ap_matrix_mdev *matrix_mdev; +enum vfio_device_mig_state next_state; +struct vfio_ap_migration_data *mig_data; + +mutex_lock (&matrix_dev->mdevs_lock); + matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev); + mig_data = matrix_mdev->mig_data; +dev_dbg (vdev->dev,"%s -> %d\n" , __func__, new_state); + +while (mig_data->mig_state != new_state) { + ret =vfio_mig_get_next_state (vdev, mig_data->mig_state, + new_state, &next_state); +if (ret) { + filp =ERR_PTR (ret); +break ; + } + + filp =vfio_ap_transition_to_state (matrix_mdev, next_state); +if (IS_ERR (filp)) +break ; + + mig_data->mig_state = next_state; + +if (WARN_ON (filp && new_state != next_state)) { +fput (filp); + filp =ERR_PTR (-EINVAL ); +break ; + } + } + +mutex_unlock (&matrix_dev->mdevs_lock); + +return filp; } static int vfio_ap_get_state(struct vfio_device *vdev,
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 13/24 s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY
In reply to Anthony KrowiakThe transition of the guest migration state from VFIO_DEVICE_STATE_STOP to VFIO_DEVICE_STATE_STOP_COPY begins the process of saving the vfio device state. The vfio device state is comprised of the guest's AP configuration which specifies the adapters, domains and control domains to which the guest will be given access.
To begin the process of saving the vfio device state, the VFIO AP device driver will:
1. Allocate an object in which to store the guest's AP configuration information comprised of the APQN and hardware information for each queue passed through to the source guest.
2. Opens a file stream that will be used to read the AP configuration stored with the object created in #1. The AP configuration will be read from this file stream when the guest is resumed on the target system to verify that the AP configuration of the source and target guests are compatible.
--- drivers/s390/crypto/vfio_ap_migration.c | 166 +++++++++++++++++++++++- 1 file changed, 164 insertions(+), 2 deletions(-)
drivers/s390/crypto/vfio_ap_migration.c +164 -2
@@ -4,6 +4,7 @@* * Copyright IBM Corp. 2025 */ +#include <linux/anon_inodes.h> #include <linux/file.h> #include "uapi/linux/vfio_ap.h"
@@ -96,12 +97,160 @@ static void vfio_ap_release_mig_files(struct ap_matrix_mdev *matrix_mdev) } } +static ssize_t vfio_ap_save_read (struct file *,char __user *,size_t ,loff_t *) +{ +/* TODO */ +return -EOPNOTSUPP ; +} + +static int vfio_ap_release_migf (struct inode *,struct file *) +{ +/* TODO */ +return -EOPNOTSUPP ; +} + +static const struct file_operations vfio_ap_save_fops = { + .owner =THIS_MODULE , + .read = vfio_ap_save_read, + .compat_ioctl = compat_ptr_ioctl, + .release = vfio_ap_release_migf, +}; + +static struct vfio_ap_config +*vfio_ap_allocate_config (struct ap_matrix_mdev *matrix_mdev,size_t *config_sz) +{ +struct vfio_ap_config *ap_config; +size_t qinfo_size; +int num_queues; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + num_queues =vfio_ap_mdev_get_num_queues (&matrix_mdev->shadow_apcb); + qinfo_size = num_queues *sizeof (struct vfio_ap_queue_info ); + *config_sz = qinfo_size +sizeof (struct vfio_ap_config ); + ap_config =kzalloc (*config_sz,GFP_KERNEL_ACCOUNT ); + +if (!ap_config) +return ERR_PTR (-ENOMEM ); + + ap_config->num_queues = num_queues; + +return ap_config; +} + +static void vfio_ap_store_queue_info (struct vfio_ap_migration_file *migf) +{ +unsigned long *apm, *aqm, num_queues, apid, apqi, apqn, data; +struct ap_matrix_mdev *matrix_mdev; +struct vfio_ap_queue *q; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + matrix_mdev = migf->matrix_mdev; + apm = matrix_mdev->shadow_apcb.apm; + aqm = matrix_mdev->shadow_apcb.aqm; + num_queues =0 ; + +for_each_set_bit_inv (apid, apm,AP_DEVICES ) { +for_each_set_bit_inv (apqi, aqm,AP_DOMAINS ) { + apqn =AP_MKQID (apid, apqi); + q =vfio_ap_mdev_get_queue (matrix_mdev, apqn); + +if (!q) +continue ; + + migf->ap_config->qinfo[num_queues].apqn = apqn; + data = q->hwinfo.value; + migf->ap_config->qinfo[num_queues].data = data; + num_queues +=1 ; +dev_dbg (matrix_mdev->vdev.dev, +"%s (%d): qinfo: apqn=%04lx data=%016lx\n" , + __func__, __LINE__, apqn, data); + } + } +} + +static struct vfio_ap_migration_file +*vfio_ap_allocate_migf (struct ap_matrix_mdev *matrix_mdev) +{ +struct vfio_ap_migration_file *migf; +struct vfio_ap_config *ap_config; +size_t config_size; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + + migf = kzalloc_obj(struct vfio_ap_migration_file, GFP_KERNEL_ACCOUNT); +if (!migf) +return ERR_PTR (-ENOMEM ); + + ap_config =vfio_ap_allocate_config (matrix_mdev, &config_size); +if (IS_ERR (ap_config)) { +kfree (migf); +return ERR_CAST (ap_config); + } + + migf->ap_config = ap_config; + migf->config_sz = config_size; + migf->matrix_mdev = matrix_mdev; + +return migf; +} + +static void vfio_ap_deallocate_migf (struct vfio_ap_migration_file *migf) +{ +kfree (migf->ap_config); +kfree (migf); +} + +static struct file *vfio_ap_open_file_stream (struct vfio_ap_migration_file *migf, +const struct file_operations *fops, +int flags) +{ +struct file *filp; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + + filp =anon_inode_getfile ("vfio_ap_migf" , fops, migf, flags); +if (IS_ERR (filp)) +return ERR_CAST (filp); + +stream_open (filp->f_inode, filp); + +return filp; +} + +static struct vfio_ap_migration_file * +vfio_ap_save_mdev_state (struct ap_matrix_mdev *matrix_mdev) +{ +struct vfio_ap_migration_data *mig_data; +struct vfio_ap_migration_file *migf; +struct file *filp; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + mig_data = matrix_mdev->mig_data; + + migf =vfio_ap_allocate_migf (matrix_mdev); +if (IS_ERR (migf)) +return ERR_CAST (migf); + + filp =vfio_ap_open_file_stream (migf, &vfio_ap_save_fops,O_RDONLY ); +if (IS_ERR (filp)) { +vfio_ap_deallocate_migf (migf); +return ERR_CAST (filp); + } + + migf->filp = filp; + mig_data->saving_migf = migf; +vfio_ap_store_queue_info (mig_data->saving_migf); + +return mig_data->saving_migf; +} +static struct file *vfio_ap_transition_to_state (struct ap_matrix_mdev *matrix_mdev,enum vfio_device_mig_state new_state) {struct vfio_ap_migration_data *mig_data;enum vfio_device_mig_state cur_state; +struct vfio_ap_migration_file *migf;lockdep_assert_held (&matrix_dev->mdevs_lock); mig_data = matrix_mdev->mig_data;
@@ -109,10 +258,23 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,dev_dbg (matrix_mdev->vdev.dev,"%s: %d -> %d\n" , __func__, cur_state, new_state); +/* +* Begins the process of saving the vfio device state by creating and +* returning a streaming data_fd to be used to read out the internal +* state of the vfio-ap device on the source host. +*/ if (cur_state ==VFIO_DEVICE_STATE_STOP && new_state ==VFIO_DEVICE_STATE_STOP_COPY ) { -/* TODO */ -return ERR_PTR (-EOPNOTSUPP ); + migf =vfio_ap_save_mdev_state (matrix_mdev); +if (IS_ERR (migf)) +return ERR_CAST (migf); + +if (migf) { +get_file (migf->filp); +return migf->filp; + } + +return NULL ; } if (cur_state == VFIO_DEVICE_STATE_STOP &&
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 14/24 s390/vfio-ap: File ops called to save the vfio device migration state
In reply to Anthony KrowiakImplements two callback functions that were added to the 'file_operations' structure for the file created to save the state of the vfio device when the migration state transitioned from VFIO_DEVICE_STATE_STOP to VFIO_DEVICE_STATE_STOP_COPY:
* Read callback
This function copies the guest's AP configuration information to userspace. The information copied is comprised of the APQN of each queue device passed through to the guest along with its hardware information. This data will be used to verify the source and target guests have compatible AP configurations.
* Release callback
This function deallocates the object used to save the state of the vfio device.
--- drivers/s390/crypto/vfio_ap_migration.c | 78 +++++++++++++++++++++++-- 1 file changed, 72 insertions(+), 6 deletions(-)
drivers/s390/crypto/vfio_ap_migration.c +72 -6
@@ -97,16 +97,82 @@ static void vfio_ap_release_mig_files(struct ap_matrix_mdev *matrix_mdev) } } -static ssize_t vfio_ap_save_read (struct file *,char __user *,size_t ,loff_t *) +static ssize_t validate_save_read_parms (struct vfio_ap_migration_file *migf, +loff_t *pos,size_t len) { -/* TODO */ -return -EOPNOTSUPP ; +lockdep_assert_held (&matrix_dev->mdevs_lock); + +if (migf->disabled) { +dev_err (migf->matrix_mdev->vdev.dev, +"%s (%d): migration file is disabled\n" , + __func__, __LINE__); +return -ENODEV ; + } + +if (*pos > migf->config_sz) { +dev_err (migf->matrix_mdev->vdev.dev, +"%s (%d): file pos (%llu) exceeds migf->config size (%zu)\n" , + __func__, __LINE__, *pos, migf->config_sz); +return -EINVAL ; + } + +return 0 ; } -static int vfio_ap_release_migf (struct inode *,struct file *) +static ssize_t vfio_ap_save_read (struct file *filp,char __user *buf, +size_t len,loff_t *pos) { -/* TODO */ -return -EOPNOTSUPP ; +struct vfio_ap_migration_file *migf; +ssize_t ret =0 ; + +if (pos) +return -ESPIPE ; + +mutex_lock (&matrix_dev->mdevs_lock); + + pos = &filp->f_pos; + migf = filp->private_data; + + ret =validate_save_read_parms (migf, pos, len); +if (ret) + goto out_unlock; + + len =min_t (size_t, migf->config_sz - *pos, len); +if (len) { +if (copy_to_user (buf, (void *)migf->ap_config + *pos, len)) { + ret = -EFAULT ; +dev_err (migf->matrix_mdev->vdev.dev, +"%s (%d): failed to copy config data to user\n" , + __func__, __LINE__); + goto out_unlock; + } + + *pos += len; + ret = len; + } + +dev_dbg (migf->matrix_mdev->vdev.dev, +"%s (%d): copied %zu bytes of AP config data to user\n" , + __func__, __LINE__, len); + +out_unlock: +mutex_unlock (&matrix_dev->mdevs_lock); + +return ret; +} + +static void vfio_ap_deallocate_migf (struct vfio_ap_migration_file *migf); + +static int vfio_ap_release_migf (struct inode *inode,struct file *filp) +{ +struct vfio_ap_migration_file *migf; + +mutex_lock (&matrix_dev->mdevs_lock); + migf = filp->private_data; +vfio_ap_deallocate_migf (migf); +mutex_unlock (&matrix_dev->mdevs_lock); + +return 0 ; }static const struct file_operations vfio_ap_save_fops = {
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 15/24 s390/vfio-ap: Transition device migration state from STOP to RESUMING
In reply to Anthony KrowiakTransitioning from VFIO_DEVICE_STATE_STOP to VFIO_DEVICE_STATE_RESUMING starts a process of restoring the device state of the vfio device on the target system. To prepare for restoring the device state, open a file stream to receive the guest's AP configuration saved when the device state on the source system transitioned from VFIO_DEVICE_STATE_STOP to VFIO_DEVICE_STATE_STOP_COPY.
--- drivers/s390/crypto/vfio_ap_migration.c | 55 ++++++++++++++++++++++++- 1 file changed, 53 insertions(+), 2 deletions(-)
drivers/s390/crypto/vfio_ap_migration.c +53 -2
@@ -310,6 +310,46 @@ vfio_ap_save_mdev_state(struct ap_matrix_mdev *matrix_mdev)return mig_data->saving_migf; } +static ssize_t vfio_ap_resume_write (struct file *filp,const char __user *buf, +size_t len,loff_t *pos) +{ +/* TODO */ +return -EOPNOTSUPP ; +} + +static const struct file_operations vfio_ap_resume_fops = { + .owner =THIS_MODULE , + .write = vfio_ap_resume_write, + .release = vfio_ap_release_migf, +}; + +static struct vfio_ap_migration_file * +vfio_ap_resume_mdev_state (struct ap_matrix_mdev *matrix_mdev) +{ +struct vfio_ap_migration_data *mig_data; +struct vfio_ap_migration_file *migf; +struct file *filp; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + mig_data = matrix_mdev->mig_data; + + migf =vfio_ap_allocate_migf (matrix_mdev); +if (IS_ERR (migf)) +return ERR_CAST (migf); + + filp =vfio_ap_open_file_stream (migf, &vfio_ap_resume_fops,O_WRONLY ); +if (IS_ERR (filp)) { +vfio_ap_deallocate_migf (migf); +return ERR_CAST (filp); + } + + migf->matrix_mdev = matrix_mdev; + migf->filp = filp; + mig_data->resuming_migf = migf; + +return migf; +} +static struct file *vfio_ap_transition_to_state (struct ap_matrix_mdev *matrix_mdev,enum vfio_device_mig_state new_state)
@@ -343,10 +383,21 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,return NULL ; } +/* +* Starts the process of restoring the state of the vfio-ap device +* on the target host by creating a filestream to be used to transfer +* the internal state of the vfio-ap device on the source host that +* was saved during the STOP_COPY phase of the migration. +*/ if (cur_state ==VFIO_DEVICE_STATE_STOP && new_state ==VFIO_DEVICE_STATE_RESUMING ) { -/* TODO */ -return ERR_PTR (-EOPNOTSUPP ); + migf =vfio_ap_resume_mdev_state (matrix_mdev); +if (IS_ERR (migf)) +return ERR_CAST (migf); + +get_file (migf->filp); + +return migf->filp; } if (cur_state == VFIO_DEVICE_STATE_RESUMING &&
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 16/24 s390/vfio-ap: File ops called to resume the vfio device migration
In reply to Anthony KrowiakImplements two callback functions that were added to the 'file_operations' structure for the file created to restore the state of the vfio device when the migration state transitioned from VFIO_DEVICE_STATE_STOP to VFIO_DEVICE_STATE_RESUMING:
* Write callback
The write callback retrieves the vfio device migration state saved to the file stream created when the vfio device state was transitioned from VFIO_DEVICE_STATE_STOP to VFIO_DEVICE_STATE_STOP_COPY. The saved state contains the guest's AP configuration information. Since the guest's AP configuration on the target system is supplied by the mediated device residing thereon, the migration state passed into this callback will not be used to restore the vfio device state, but to verify that the source and target AP configurations are compatible since the mediated devices on the source and target systems may differ.
The verification rules are:
o The source and target AP configurations must have the same number of APQNs
o Each APQN in the source guest's AP configuration must also be in the target guest's configuration
o Each APQN in the source and target guests' AP configurations must reference a queue device with compatible hardware:
- The source and target queues must have the same facilities installed
- APSC facility
- APQKM facility
- AP4KC facility
- The source and target queues must have the same mode
- Coprocessor-mode
- Accelerator-mode
- XCP-mode
- The source and target queues must have the same APXA facility setting
- If the APXA facility is installed on target queue, it must also be installed on the source queue
- If the APXA facility is not installed on target queue, it must also not be installed on the source queue
- The source and target queues must have the same classification
- Full native card function
- Stateless functions
- The binding and associated state
~ At the time of migration, the queue on the target guest will
neither be bound nor associated, so the binding and associated state for both the source and target must indicate that the queue is usable for all messages (i.e., BS bits equal to 00).
- The AP type of the target queue must be the same or newer than the source queue
* Release callback
This function deallocates the object used to restore the state of the vfio device when the file is released.
--- drivers/s390/crypto/vfio_ap_migration.c | 525 +++++++++++++++++++++++- 1 file changed, 523 insertions(+), 2 deletions(-)
drivers/s390/crypto/vfio_ap_migration.c +523 -2
@@ -10,6 +10,36 @@#include "vfio_ap_private.h" +/* +* Masks the fields of the queue information returned from the PQAP(TAPQ) +* command. In order to migrate a guest, it's AP configuration must be +* compatible with AP configuration assigned to the target guest's mdev. +* This mask is used to verify that the queue information for each source and +* target queue is compatible (i.e., the masked fields are equivalent). +* +* The relevant fields covered by this mask are: +* S bit 0: APSC facility installed +* M bit 1: APQKM facility installed +* C bit 2: AP4KC facility installed +* Mode bits 3-5: +* D bit 3: CCA-mode facility +* A bit 4: accelerator-mode facility +* X bit 5: XCP-mode facility +* N bit 6: APXA facility installed +* SL bit 7: SLCF facility installed +* Classification (functional capabilities) bits 8-16 +* bit 8: Native card function +* bit 9: Only stateless functions +* AP Type bits 32-40: +*/ +#define QINFO_DATA_MASK 0xffffc000ff000000 + +/* +* Masks the bit that indicates whether full native card function is available +* from the 8 bits specifying the functional capabilities of a queue +*/ +#define CLASSIFICATION_NATIVE_FCN_MASK 0x80 +/** * vfio_ap_migration_data - the data needed to migrate a guest with pass-through * access to AP devices
@@ -234,6 +264,53 @@ static void vfio_ap_store_queue_info(struct vfio_ap_migration_file *migf) } } +static int validate_resume_write_parms (struct vfio_ap_migration_file *migf, +size_t len,loff_t *pos) +{ +loff_t total_len; +int ret = -EIO ; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + +if (!migf->matrix_mdev) { +pr_err ("migration failed: matrix_mdev object not linked to migration file" ); + goto done; + } + +if (*pos <0 ) { +dev_err (migf->matrix_mdev->vdev.dev, +"migration failed: invalid migration file position (%lli) for write\n" , + *pos); + goto done; + } + +if (check_add_overflow ((loff_t )len, *pos, &total_len)) { +dev_err (migf->matrix_mdev->vdev.dev, +"migration failed: pos (%llu) plus len (%zu) operation overflowed loff_t precision\n" , + *pos, len); + goto done; + } + +if (total_len > migf->config_sz) { +dev_err (migf->matrix_mdev->vdev.dev, +"migration failed: source guest's AP config size (%llu) larger than target's (%lu)" , + total_len, migf->config_sz); + goto done; + } + +if (migf->disabled) { +dev_err (migf->matrix_mdev->vdev.dev, +"migration failed: migration file is disabled" ); + goto done; + } + +dev_dbg (migf->matrix_mdev->vdev.dev,"resume write parameters validated\n" ); + ret =0 ; + +done: +return ret; +} +static struct vfio_ap_migration_file *vfio_ap_allocate_migf (struct ap_matrix_mdev *matrix_mdev) {
@@ -310,11 +387,455 @@ vfio_ap_save_mdev_state(struct ap_matrix_mdev *matrix_mdev)return mig_data->saving_migf; } +static void report_facilities_compatibility (struct ap_matrix_mdev *matrix_mdev, +unsigned long apqn, +struct ap_tapq_hwinfo *src_hwinfo, +struct ap_tapq_hwinfo *target_hwinfo) +{ +if (src_hwinfo->apsc != target_hwinfo->apsc) { +if (src_hwinfo->apsc) { +dev_err (matrix_mdev->vdev.dev, +"APSC facility installed in source queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +dev_err (matrix_mdev->vdev.dev, +"APSC facility not installed in target queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + }else { +dev_err (matrix_mdev->vdev.dev, +"APSC facility not installed in source queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +dev_err (matrix_mdev->vdev.dev, +"APSC facility installed in target queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + } + } + +if (src_hwinfo->mex4k != target_hwinfo->mex4k) { +if (src_hwinfo->mex4k) { +dev_err (matrix_mdev->vdev.dev, +"mex4k facility installed in source queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +dev_err (matrix_mdev->vdev.dev, +"mex4k facility not installed in target queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + }else { +dev_err (matrix_mdev->vdev.dev, +"mex4k facility not installed in source queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +dev_err (matrix_mdev->vdev.dev, +"mex4k facility installed in target queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + } + } + +if (src_hwinfo->crt4k != target_hwinfo->crt4k) { +if (src_hwinfo->crt4k) { +dev_err (matrix_mdev->vdev.dev, +"crt4k facility installed in source queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +dev_err (matrix_mdev->vdev.dev, +"crt4k facility not installed in target queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + }else { +dev_err (matrix_mdev->vdev.dev, +"crt4k facility not installed in source queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +dev_err (matrix_mdev->vdev.dev, +"crt4k facility installed in target queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + } + } +} + +static void report_mode_compatibility (struct ap_matrix_mdev *matrix_mdev, +unsigned long apqn, +struct ap_tapq_hwinfo *src_hwinfo, +struct ap_tapq_hwinfo *target_hwinfo) +{ +if (src_hwinfo->cca != target_hwinfo->cca) { +if (src_hwinfo->cca) { +dev_err (matrix_mdev->vdev.dev, +"Coprocessor-mode facility installed in source queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +dev_err (matrix_mdev->vdev.dev, +"Coprocessor-mode facility not installed target queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + }else { +dev_err (matrix_mdev->vdev.dev, +"Coprocessor-mode facility not installed in source queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +dev_err (matrix_mdev->vdev.dev, +"Coprocessor-mode facility installed target queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + } + } + +if (src_hwinfo->accel != target_hwinfo->accel) { +if (src_hwinfo->accel) { +dev_err (matrix_mdev->vdev.dev, +"Accelerator-mode facility installed source queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +dev_err (matrix_mdev->vdev.dev, +"Accelerator-mode facility not installed target queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + }else { +dev_err (matrix_mdev->vdev.dev, +"Accelerator-mode facility not installed source queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +dev_err (matrix_mdev->vdev.dev, +"Accelerator-mode facility installed target queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + } + } + +if (src_hwinfo->ep11 != target_hwinfo->ep11) { +if (src_hwinfo->ep11) { +dev_err (matrix_mdev->vdev.dev, +"XCP-mode facility installed source queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +dev_err (matrix_mdev->vdev.dev, +"XCP-mode facility not installed target queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + }else { +dev_err (matrix_mdev->vdev.dev, +"XCP-mode facility not installed source queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +dev_err (matrix_mdev->vdev.dev, +"XCP-mode facility installed target queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + } + } +} + +static void report_apxa_compatibility (struct ap_matrix_mdev *matrix_mdev, +unsigned long apqn, +struct ap_tapq_hwinfo *src_hwinfo, +struct ap_tapq_hwinfo *target_hwinfo) +{ +if (src_hwinfo->apxa != target_hwinfo->apxa) { +if (src_hwinfo->apxa) { +dev_err (matrix_mdev->vdev.dev, +"AP-extended-addressing (APXA) facility installed in source queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +dev_err (matrix_mdev->vdev.dev, +"AP-extended-addressing (APXA) facility not installed in target queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + }else { +dev_err (matrix_mdev->vdev.dev, +"AP-extended-addressing (APXA) facility not installed in source queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +dev_err (matrix_mdev->vdev.dev, +"AP-extended-addressing (APXA) facility installed in target queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + } + } +} + +static void report_slcf_compatibility (struct ap_matrix_mdev *matrix_mdev, +unsigned long apqn, +struct ap_tapq_hwinfo *src_hwinfo, +struct ap_tapq_hwinfo *target_hwinfo) +{ +if (src_hwinfo->slcf != target_hwinfo->slcf) { +if (src_hwinfo->slcf) { +dev_err (matrix_mdev->vdev.dev, +"Stateless-command-filtering (SLCF) available in source queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +dev_err (matrix_mdev->vdev.dev, +"Stateless-command-filtering (SLCF) not available in target queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + }else { +dev_err (matrix_mdev->vdev.dev, +"Stateless-command-filtering (SLCF) not available in source queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +dev_err (matrix_mdev->vdev.dev, +"Stateless-command-filtering (SLCF) available in target queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + } + } +} + +static void report_class_compatibility (struct ap_matrix_mdev *matrix_mdev, +unsigned long apqn, +struct ap_tapq_hwinfo *src_hwinfo, +struct ap_tapq_hwinfo *target_hwinfo) +{ +unsigned long src_native, target_native; + + src_native = src_hwinfo->class &CLASSIFICATION_NATIVE_FCN_MASK ; + target_native = target_hwinfo->class &CLASSIFICATION_NATIVE_FCN_MASK ; + +if (src_native != target_native) { +/* +* If the source queue has full native card function and the +* target queue has only stateless functions available, then +* there may be instructions that will not execute on the +* target queue. +* +* If the source queue has only stateless card functions and the +* target queue has full native card function available, then +* we are okay because the target queue can run all card +* functions. +*/ +if (src_native) { +dev_err (matrix_mdev->vdev.dev, +"Full native card function available on source queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +dev_err (matrix_mdev->vdev.dev, +"Only stateless functions available on target queue %02lx.%04lx\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + } + } +} + +static void report_bs_compatibility (struct ap_matrix_mdev *matrix_mdev, +unsigned long apqn, +struct ap_tapq_hwinfo *src_hwinfo, +struct ap_tapq_hwinfo *target_hwinfo) +{ +if (src_hwinfo->bs || target_hwinfo->bs) { +dev_err (matrix_mdev->vdev.dev, +"Bind/associate state for source (%01x) and target (%01x) queue %02lx.%04lx are not compatible\n" , + src_hwinfo->bs, target_hwinfo->bs, +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + } +} + +static void report_aptype_compatibility (struct ap_matrix_mdev *matrix_mdev, +unsigned long apqn, +struct ap_tapq_hwinfo *src_hwinfo, +struct ap_tapq_hwinfo *target_hwinfo) +{ +dev_err (matrix_mdev->vdev.dev, +"AP type of source (%02x) not compatible with target (%02x)\n" , + src_hwinfo->at, target_hwinfo->at); +} + +/* +* Log a device error reporting that migration failed due to queue +* incompatibilities followed by a device error for each incompatible feature. +*/ +static void report_qinfo_incompatibilities (struct ap_matrix_mdev *matrix_mdev, +unsigned long apqn, +struct ap_tapq_hwinfo *src_hwinfo, +struct ap_tapq_hwinfo *target_hwinfo) +{ +dev_err (matrix_mdev->vdev.dev, +"Migration failed: Source and target queue (%02lx.%04lx) not compatible\n" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +report_facilities_compatibility (matrix_mdev, apqn, src_hwinfo, target_hwinfo); +report_mode_compatibility (matrix_mdev, apqn, src_hwinfo, target_hwinfo); +report_apxa_compatibility (matrix_mdev, apqn, src_hwinfo, target_hwinfo); +report_slcf_compatibility (matrix_mdev, apqn, src_hwinfo, target_hwinfo); +report_class_compatibility (matrix_mdev, apqn, src_hwinfo, target_hwinfo); +report_aptype_compatibility (matrix_mdev, apqn, src_hwinfo, target_hwinfo); +report_bs_compatibility (matrix_mdev, apqn, src_hwinfo, target_hwinfo); +} + +static bool qinfo_compatible (struct ap_matrix_mdev *matrix_mdev, +unsigned long apqn, +struct ap_tapq_hwinfo *src_hwinfo, +struct ap_tapq_hwinfo *target_hwinfo) +{ +unsigned long src_bits, target_bits; + + src_bits = src_hwinfo->value &QINFO_DATA_MASK ; + target_bits = target_hwinfo->value &QINFO_DATA_MASK ; + +/* +* If all relevant bits are the same, or only the AP type of the source +* and target queue differ but the source type is older than the target +* type, then no incompatibilities will be reported. The AP types are +* considered compatible even if they differ as long as the source type +* is older than the target type since AP devices are backwards +* compatible. +*/ +if (src_bits == target_bits || + (src_hwinfo->fac == target_hwinfo->fac && + src_hwinfo->at <= target_hwinfo->at)) { +return true; + } + +report_qinfo_incompatibilities (matrix_mdev, apqn, src_hwinfo, + target_hwinfo); + +return false; +} + +static int matrixes_compatible (struct vfio_ap_migration_file *migf) +{ +struct ap_matrix_mdev *matrix_mdev; +struct ap_tapq_hwinfo src_hwinfo; +struct vfio_ap_queue *q; +unsigned long apqn; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + + matrix_mdev = migf->matrix_mdev; + +for (int i =0 ; i < migf->ap_config->num_queues; i++) { + apqn = migf->ap_config->qinfo[i].apqn; + q =vfio_ap_mdev_get_queue (matrix_mdev, apqn); +memcpy (&src_hwinfo, &migf->ap_config->qinfo[i].data, +sizeof (src_hwinfo)); + +if (!qinfo_compatible (matrix_mdev, apqn, &src_hwinfo, &q->hwinfo)) +return -EFAULT ; + } + +return 0 ; +} + +static bool apqns_match (struct vfio_ap_migration_file *migf) +{ +struct ap_matrix_mdev *matrix_mdev; +unsigned long apid, apqi, apqn; +bool ret = true; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + + matrix_mdev = migf->matrix_mdev; + +for (int i =0 ; i < migf->ap_config->num_queues; i++) { + apqn = migf->ap_config->qinfo[i].apqn; + apid =AP_QID_CARD (apqn); + apqi =AP_QID_QUEUE (apqn); + +if (!test_bit_inv (apid, matrix_mdev->shadow_apcb.apm) || + !test_bit_inv (apqi, matrix_mdev->shadow_apcb.aqm)) { +dev_err (matrix_mdev->vdev.dev, +"migration failed: queue %02lx.%04lx not assigned to guest matrix\n" , + apid, apqi); + ret = false; + } + } + +return ret; +} + +static int vfio_ap_validate_num_queues (struct vfio_ap_migration_file *migf) +{ +int num_migf_queues, num_mdev_queues; +struct ap_matrix_mdev *matrix_mdev; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + matrix_mdev = migf->matrix_mdev; + num_mdev_queues =vfio_ap_mdev_get_num_queues (&matrix_mdev->shadow_apcb); + num_migf_queues = migf->ap_config->num_queues; + +if (num_mdev_queues != num_migf_queues) { +dev_err (matrix_mdev->vdev.dev, +"migration failed: number of queues on source (%d) and target (%d) guests differ\n" , + num_migf_queues, num_mdev_queues); +return (num_mdev_queues > num_migf_queues) ? -ENODEV : -E2BIG ; + } + +return 0 ; +} + +static int do_post_copy_validation (struct vfio_ap_migration_file *migf,loff_t pos) +{ +unsigned long nqueues_offset; +int ret; + + nqueues_offset = offsetofend(struct vfio_ap_config, num_queues); +if (pos >= nqueues_offset) { + ret =vfio_ap_validate_num_queues (migf); +if (ret) +return ret; + +if (pos == migf->config_sz) { +if (!apqns_match (migf)) +return -ENODEV ; + ret =matrixes_compatible (migf); +if (ret) +return ret; + } + } + +return 0 ; +} + +/** +* vfio_ap_resume_write - store the AP configuration information sent from the +* source guest into the migration file. +* @filp: the file used to send the AP configuration information from the source +* guest. +* @buf: buffer containing the AP configuration information sent from the +* source guest +* @len: the length of the AP configuration information contained in @buf +* *pos: a pointer to store the file position after retrieving the AP config +* information from @buf +*/ static ssize_t vfio_ap_resume_write (struct file *filp,const char __user *buf,size_t len,loff_t *pos) { -/* TODO */ -return -EOPNOTSUPP ; +struct vfio_ap_migration_data *mig_data; +struct vfio_ap_migration_file *migf; +ssize_t ret =0 ; + +if (pos) +return -ESPIPE ; + +mutex_lock (&matrix_dev->mdevs_lock); + + pos = &filp->f_pos; + migf = filp->private_data; + +if (ap_is_se_guest ()) { +dev_err (migf->matrix_mdev->vdev.dev, +"Migration failed: Target is a secure execution guest\n" ); +return -EOPNOTSUPP ; + } + + mig_data = migf->matrix_mdev->mig_data; + + ret =validate_resume_write_parms (migf, len, pos); +if (ret) + goto out_unlock; + +if (copy_from_user ((void *)migf->ap_config + *pos, buf, len)) { +dev_err (migf->matrix_mdev->vdev.dev, +"%s (%d): failed to copy queue information from userspace" , + __func__, __LINE__); + ret = -EFAULT ; + goto out_unlock; + } + + *pos += len; + + ret =do_post_copy_validation (migf, *pos); +if (ret) + goto out_unlock; + + ret = len; + +dev_dbg (migf->matrix_mdev->vdev.dev, +"%s (%d): %zu bytes of queue information stored in the migration file" , + __func__, __LINE__, len); + +out_unlock: +mutex_unlock (&matrix_dev->mdevs_lock); + +return ret; }static const struct file_operations vfio_ap_resume_fops = {
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 17/24 s390/vfio-ap: Transition device migration state from RESUMING to STOP
In reply to Anthony KrowiakLeaving the RESUMING state terminates a data transfer of the internal state of the vfio-ap device on the source host to the vfio-ap device on the target host. This state transition indicates the vfio_ap driver should complete the incorporation of data written to the data transfer FD into the into the vfio-ap device. Since a vfio-ap device does not virtualize a physical device, there is no internal device state to incorporate; so, the only thing left to do is release the migration files used to help facilitate the migration process.
--- drivers/s390/crypto/vfio_ap_migration.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/s390/crypto/vfio_ap_migration.c +12 -2
@@ -921,10 +921,20 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,return migf->filp; } +/* +* Terminates the data transfer session of the vfio-ap device state +* between the source and target hosts. Since the vfio-ap device does +* not virtualize a DMA device, there is no internal device state to +* incorporate into the vfio-ap device on the target; so, the only +* thing left to do is release the migration files used to process +* the vfio device migration. Note that this state transition is for +* the vfio-ap device on the target host. +*/ if (cur_state ==VFIO_DEVICE_STATE_RESUMING && new_state ==VFIO_DEVICE_STATE_STOP ) { -/* TODO */ -return ERR_PTR (-EOPNOTSUPP ); +vfio_ap_release_mig_files (matrix_mdev); + +return NULL ; } if (cur_state == VFIO_DEVICE_STATE_STOP_COPY &&
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 18/24 s390/vfio-ap: Transition device migration state from STOP_COPY to STOP
In reply to Anthony KrowiakTypically at this point, interrupt processing for the device, DMA and other changes to the device's external state must be prohibited. For VFIO AP, there is no physical DMA device virtualized by the vfio-ap device . It merely manages the guest's AP configuration that identifies the devices passed through to the guest, but does not have access to any of them. Those AP devices are passed through and controlled exclusively by the SIE program used to start the guest. The only thing to do at this point is release all migration objects used to manage migration of the guest.
--- drivers/s390/crypto/vfio_ap_migration.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
drivers/s390/crypto/vfio_ap_migration.c +10 -2
@@ -937,10 +937,18 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,return NULL ; } +/* +* Stop the operation of the vfio-ap device. Since the vfio-ap device +* does not virtualize a DMA device, there is no physical device to +* stop; so, the only thing left to do is release the migration files +* used to process the vfio device migration. Note that this state +* transition is for the vfio-ap device on the source host. +*/ if (cur_state ==VFIO_DEVICE_STATE_STOP_COPY && new_state ==VFIO_DEVICE_STATE_STOP ) { -/* TODO */ -return ERR_PTR (-EOPNOTSUPP ); +vfio_ap_release_mig_files (matrix_mdev); + +return NULL ; } if ((cur_state == VFIO_DEVICE_STATE_STOP &&
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 19/24 s390/vfio-ap: Transition device migration state from STOP to RUNNING and vice versa
In reply to Anthony KrowiakThe transition from STOP to RUNNING and vice versa indicates that the vfio device is operating normally and migration is not currently taking place. In this case, just set the new state and if there are any migrations files used to help facilitate migration hanging around, release them.
--- drivers/s390/crypto/vfio_ap_migration.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
drivers/s390/crypto/vfio_ap_migration.c +8 -2
@@ -951,12 +951,18 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,return NULL ; } +/* +* These states indicates migration has either not been initiated or +* has completed and the vfio-ap device is operating normally; so +* just set the new migration state. Just in case, release the +* migration files used to facilitate migration if any are hanging +* around. +*/ if ((cur_state ==VFIO_DEVICE_STATE_STOP && new_state ==VFIO_DEVICE_STATE_RUNNING ) || (cur_state ==VFIO_DEVICE_STATE_RUNNING && new_state ==VFIO_DEVICE_STATE_STOP )) { -/* TODO */ -return ERR_PTR (-EOPNOTSUPP ); +return NULL ; }/* vfio_mig_get_next_state() does not use arcs other than the above */
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 20/24 s390-vfio-ap: Callback to get the current vfio device migration state
In reply to Anthony KrowiakImplements the callback that returns the current vfio device migration state during live migration of guests with pass-through access to AP devices.
The function pointer for this callback is specified via the 'migration_get_state' field of the 'vfio_migration_ops' structure which is stored with the VFIO device when the 'vfio_device' structure representing the mediated device is initialized.
--- drivers/s390/crypto/vfio_ap_migration.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
drivers/s390/crypto/vfio_ap_migration.c +12 -1
@@ -1014,7 +1014,18 @@ static struct file *vfio_ap_set_state(struct vfio_device *vdev,static int vfio_ap_get_state (struct vfio_device *vdev,enum vfio_device_mig_state *current_state) { -return -EOPNOTSUPP ; +struct ap_matrix_mdev *matrix_mdev; +struct vfio_ap_migration_data *mig_data; + +mutex_lock (&matrix_dev->mdevs_lock); + + matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev); + mig_data = matrix_mdev->mig_data; + *current_state = mig_data->mig_state; + +mutex_unlock (&matrix_dev->mdevs_lock); + +return 0 ; } static int vfio_ap_get_data_size(struct vfio_device *vdev,
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 21/24 s390/vfio-ap: Callback to get the size of data to be migrated during guest migration
In reply to Anthony KrowiakCallback to get the estimated data length of the data that will be required to complete reading of the vfio device's internal state. It's mandatory for VFIO_DEVICE_FEATURE_MIGRATION migration support.
The function pointer for this callback is specified via the 'migration_get_data_size' field of the 'vfio_migration_ops' structure which is stored with the VFIO device when the 'vfio_device' structure representing the mediated device is initialized.
--- drivers/s390/crypto/vfio_ap_migration.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
drivers/s390/crypto/vfio_ap_migration.c +12 -1
@@ -1031,7 +1031,18 @@ static int vfio_ap_get_state(struct vfio_device *vdev,static int vfio_ap_get_data_size (struct vfio_device *vdev,unsigned long *stop_copy_length) { -return -EOPNOTSUPP ; +struct ap_matrix_mdev *matrix_mdev; +size_t qinfo_sz; +int num_queues; + +mutex_lock (&matrix_dev->mdevs_lock); + matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev); + num_queues =vfio_ap_mdev_get_num_queues (&matrix_mdev->shadow_apcb); + qinfo_sz = num_queues *sizeof (struct vfio_ap_queue_info ); + *stop_copy_length = qinfo_sz +sizeof (struct vfio_ap_config ); +mutex_unlock (&matrix_dev->mdevs_lock); + +return 0 ; }static const struct vfio_migration_ops vfio_ap_migration_ops = {
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 22/24 s390/vfio-ap: Provide API to query whether migration is in progress
In reply to Anthony KrowiakProvides an AP to query whether a live guest migration is in progress. This can be called to determine whether an action should be allowed during migration; for example, allowing a system administrator to disable migration while it is in progress. Other uses may be to prevent changing a guest's AP configuration while a migration is in progress.
The only means the vfio_ap device driver has to determine that a migration is in progress is via the callback functions specified in the 'struct vfio_migration_ops' object associated with the vfio device. The very first callback invoked during migration is the callback specified in the 'migration_get_data_size' field of the structure. This callback will set a flag indicating that migration is in progress. The flag will be cleared when the vfio device migration state transitions from VFIO_DEVICE_STATE_STOP_COPY to VFIO_DEVICE_STATE_STOP. At this point, the internal state of the vfio device on the source has been copied and will be restored on the target.
Note that the 'migration_get_data_size' callback occurs after migration has been initiated, so there is a short period of time during which the API will erroneously indicate migration is not in progress. This, unfortunately, is unavoidable given the only means of determining that a migration is in progress is via these callbacks.
--- drivers/s390/crypto/vfio_ap_migration.c | 40 ++++++++++++++++++++++++- drivers/s390/crypto/vfio_ap_private.h | 1 + 2 files changed, 40 insertions(+), 1 deletion(-)
drivers/s390/crypto/vfio_ap_migration.c +39 -1
@@ -47,11 +47,14 @@* @mig_state: the current migration state * @resuming_migf: the object used to resume the target guest * @saving_migf: the object used to save the state of the source guest +* @mig_in_progress flag indicating whether migration is in progress (true) +* or not (false) */ struct vfio_ap_migration_data {enum vfio_device_mig_state mig_state;struct vfio_ap_migration_file *resuming_migf;struct vfio_ap_migration_file *saving_migf; +bool mig_in_progress; };/**
@@ -971,18 +974,30 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,return ERR_PTR (-EINVAL ); } +static void vfio_ap_set_mig_in_progress (struct vfio_ap_migration_data *mig_data, +enum vfio_device_mig_state prev_state, +enum vfio_device_mig_state new_state) +{ +lockdep_assert_held (&matrix_dev->mdevs_lock); +if (prev_state ==VFIO_DEVICE_STATE_STOP_COPY && + new_state ==VFIO_DEVICE_STATE_STOP ) { + mig_data->mig_in_progress = false; + } +} +static struct file *vfio_ap_set_state (struct vfio_device *vdev,enum vfio_device_mig_state new_state) {int ret;struct file *filp =NULL ;struct ap_matrix_mdev *matrix_mdev; -enum vfio_device_mig_state next_state; +enum vfio_device_mig_state prev_state, next_state;struct vfio_ap_migration_data *mig_data;mutex_lock (&matrix_dev->mdevs_lock); matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev); mig_data = matrix_mdev->mig_data; + prev_state = mig_data->mig_state;dev_dbg (vdev->dev,"%s -> %d\n" , __func__, new_state);while (mig_data->mig_state != new_state) {
@@ -1006,6 +1021,7 @@ static struct file *vfio_ap_set_state(struct vfio_device *vdev, } } +vfio_ap_set_mig_in_progress (mig_data, prev_state, new_state);mutex_unlock (&matrix_dev->mdevs_lock);return filp;
@@ -1031,12 +1047,15 @@ static int vfio_ap_get_state(struct vfio_device *vdev,static int vfio_ap_get_data_size (struct vfio_device *vdev,unsigned long *stop_copy_length) { +struct vfio_ap_migration_data *mig_data;struct ap_matrix_mdev *matrix_mdev;size_t qinfo_sz;int num_queues;mutex_lock (&matrix_dev->mdevs_lock); matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev); + mig_data = matrix_mdev->mig_data; + mig_data->mig_in_progress = true; num_queues =vfio_ap_mdev_get_num_queues (&matrix_mdev->shadow_apcb); qinfo_sz = num_queues *sizeof (struct vfio_ap_queue_info ); *stop_copy_length = qinfo_sz +sizeof (struct vfio_ap_config );
@@ -1051,6 +1070,24 @@ static const struct vfio_migration_ops vfio_ap_migration_ops = { .migration_get_data_size = vfio_ap_get_data_size, }; +/** +* vfio_ap_migration_in_progress: Indicates whether a live guest migration is in +* progress or not. +* +* @matrix_mdev: pointer to object maintaining the vfio device state +* +* Return: returns true if a migration is in progress; otherwise, returns false. +*/ +bool vfio_ap_migration_in_progress (struct ap_matrix_mdev *matrix_mdev) +{ +struct vfio_ap_migration_data *mig_data; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + mig_data = matrix_mdev->mig_data; + +return mig_data->mig_in_progress; +} +/** * vfio_ap_init_migration_data - initialize migration data and functions *
@@ -1069,6 +1106,7 @@ int vfio_ap_init_migration_data(struct ap_matrix_mdev *matrix_mdev)return -ENOMEM ; mig_data->mig_state =VFIO_DEVICE_STATE_STOP ; + mig_data->mig_in_progress = false; matrix_mdev->vdev.migration_flags =VFIO_MIGRATION_STOP_COPY ; matrix_mdev->vdev.mig_ops = &vfio_ap_migration_ops; matrix_mdev->mig_data = mig_data;
drivers/s390/crypto/vfio_ap_private.h +1 -0
@@ -183,5 +183,6 @@ void vfio_ap_on_scan_complete(struct ap_config_info *new_config_info,int vfio_ap_init_migration_data (struct ap_matrix_mdev *matrix_mdev);void vfio_ap_release_migration_data (struct ap_matrix_mdev *matrix_mdev); +bool vfio_ap_migration_in_progress (struct ap_matrix_mdev *matrix_mdev);#endif /* _VFIO_AP_PRIVATE_H_ */
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 23/24 s390/vfio-ap: Disallow blocking migration in progress
In reply to Anthony KrowiakThe sysfs 'migratable' attribute allows a system administrator to enable or block live guest migration. If an attempt to block migration is attempted while a migration is in already in progress, the migration will complete leaving the system administrator believing that it was blocked. To prevent this scenario, the operation will fail with the
"Device or resource busy" (-EBUSY) error if a migration is in progress.
--- drivers/s390/crypto/vfio_ap_ops.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/s390/crypto/vfio_ap_ops.c +6 -0
@@ -1853,6 +1853,12 @@ static ssize_t migratable_store(struct device *dev, struct device_attribute *att goto out_unlock; } +if (!migratable && matrix_mdev->migratable && +vfio_ap_migration_in_progress (matrix_mdev)) { + ret = -EBUSY ; + goto out_unlock; + } + matrix_mdev->migratable = migratable;if (matrix_mdev->migratable)
Signature · 1 line
2.52.0
Anthony Krowiak
maintainerPATCH 24/24 s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst
In reply to Anthony KrowiakAdd a 'Live Guest Migration' chapter to the Documentation/arch/s390/vfio-ap.rst document to describe the details for initiating live guest migration for a guest to which AP adapters, domains and control domains have been passed through.
--- Documentation/arch/s390/vfio-ap.rst | 339 ++++++++++++++++++++-------- 1 file changed, 246 insertions(+), 93 deletions(-)
Documentation/arch/s390/vfio-ap.rst +246 -93
@@ -1016,7 +1016,7 @@ guest_matrix dyn ap_config the following features are advertised: ----------------+---------------------------------------------------------------+ ++--------------+---------------------------------------------------------------+ | Flag | Description | +==============+===============================================================+ | guest_matrix | guest_matrix attribute exists. It reports the matrix of |
@@ -1025,105 +1025,258 @@ the following features are advertised: +--------------+---------------------------------------------------------------+ | dyn | Indicates hot plug/unplug of AP adapters, domains and control | | | domains for a guest to which the mdev is attached. | -+------------+-----------------------------------------------------------------+ ++--------------+---------------------------------------------------------------+ | ap_config | ap_config interface for one-shot modifications to mdev config | +--------------+---------------------------------------------------------------+ +| migratable | Indicates that live guest migration is supported for guests | +| | to which crypto devices are passed through | ++--------------+---------------------------------------------------------------+ -Limitations -=========== -Live guest migration is not supported for guests using AP devices without -intervention by a system administrator. Before a KVM guest can be migrated, -the vfio_ap mediated device must be removed. Unfortunately, it can not be -removed manually (i.e., echo 1 > /sys/devices/vfio_ap/matrix/$UUID/remove) while -the mdev is in use by a KVM guest. If the guest is being emulated by QEMU, -its mdev can be hot unplugged from the guest in one of two ways: - -1. If the KVM guest was started with libvirt, you can hot unplug the mdev via - the following commands: - - virsh detach-device <guestname> <path-to-device-xml> - - For example, to hot unplug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 from - the guest named 'my-guest': - - virsh detach-device my-guest ~/config/my-guest-hostdev.xml - - The contents of my-guest-hostdev.xml: - -.. code-block:: xml - - <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'> - <source> - <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/> - </source> - </hostdev> - - - virsh qemu-monitor-command <guest-name> --hmp "device-del <device-id>" - - For example, to hot unplug the vfio_ap mediated device identified on the - qemu command line with 'id=hostdev0' from the guest named 'my-guest': - -.. code-block:: sh - - virsh qemu-monitor-command my-guest --hmp "device_del hostdev0" - -2. A vfio_ap mediated device can be hot unplugged by attaching the qemu monitor - to the guest and using the following qemu monitor command: - - (QEMU) device-del id=<device-id> - - For example, to hot unplug the vfio_ap mediated device that was specified - on the qemu command line with 'id=hostdev0' when the guest was started: - - (QEMU) device-del id=hostdev0 - -After live migration of the KVM guest completes, an AP configuration can be -restored to the KVM guest by hot plugging a vfio_ap mediated device on the target -system into the guest in one of two ways: - -1. If the KVM guest was started with libvirt, you can hot plug a matrix mediated - device into the guest via the following virsh commands: - - virsh attach-device <guestname> <path-to-device-xml> - - For example, to hot plug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 into - the guest named 'my-guest': - - virsh attach-device my-guest ~/config/my-guest-hostdev.xml - - The contents of my-guest-hostdev.xml: - -.. code-block:: xml - - <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'> - <source> - <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/> - </source> - </hostdev> - +Live Guest Migration +==================== +A guest to which crypto devices (i.e., AP adapters, domains and control domains) +have been passed through can be migrated to another system if both the source +and destination guests meet the following requirements: - virsh qemu-monitor-command <guest-name> --hmp \ - "device_add vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>" +* The destination guest will be started with the same QEMU command line as the + source guest, so the mediated device supplying the AP configuration on both + the source and destination guests must have the same name (UUID). - For example, to hot plug the vfio_ap mediated device - 62177883-f1bb-47f0-914d-32a22e3a8804 into the guest named 'my-guest' with - device-id hostdev0: +* The AP configuration passed through to both the source and destination guests + must be compatible. As such, each AP configuration: - virsh qemu-monitor-command my-guest --hmp \ - "device_add vfio-ap,\ - sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\ - id=hostdev0" + * Must have the same number of APQNs -2. A vfio_ap mediated device can be hot plugged by attaching the qemu monitor - to the guest and using the following qemu monitor command: + * Each APQN assigned to the source guest must also be assigned to the + destination guest - (qemu) device_add "vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>" + * Each APQN assigned to the source guest and destination guest must reference + a queue with compatible hardware capabilities: - For example, to plug the vfio_ap mediated device - 62177883-f1bb-47f0-914d-32a22e3a8804 into the guest with the device-id - hostdev0: ++--------------+---------------------------------------------------------------+ +| Hardware | Description | +| Capabilities | | ++==============+===============================================================+ +| facilities | * AP special command facility (APSC) | +| | * AP 4096-bit ME PKU commands facility (AP4KM) | +| | * AP 4096-bit CRT PKU commands (AP4KC) | ++--------------+---------------------------------------------------------------+ +| mode | * CCA-mode | +| | * Accelerator-mode | +| | * XCP-mode (EP11) | ++--------------+---------------------------------------------------------------+ +| AP extended | APXA installed | +| addressing | | ++--------------+---------------------------------------------------------------+ +| command | Command filtering available | +| filtering | | ++--------------+---------------------------------------------------------------+ +| functional | * Full native card function | +| capabilities | * Only stateless functions | ++--------------+---------------------------------------------------------------+ +| secure | The guest running on the source host can not have any queues | +| execution | bound or associated with it | ++--------------+---------------------------------------------------------------+ +| AP type | * No AP type | +| | * PCICC (Leeds-2) | +| | * PCICA (Leeds-2 Lite) | +| | * PCIXCC | +| | * CEX2A | +| | * CEX2C | +| | * CEX3A | +| | * CEX3C | +| | * CEX4S | +| | * CEX5S | +| | * CEX6S | +| | * CEX7S | +| | * CEX8S | +| | | +| | Note: The AP type on the source and destination guests can | +| | differ if the queue passed through to the target guest | +| | is a newer model (backwards compatible) | ++--------------+---------------------------------------------------------------+ - (QEMU) device-add "vfio-ap,\ - sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\ - id=hostdev0" +Enabling and disabling live guest migration +------------------------------------------- +By default, live guest migration will be disabled for guests that use crypto +devices. A r/w sysfs 'migratable' attribute for each mediated device is provided +to enable/disable live guest migration for the guest to which the mediated +device is attached. The valid values for this attribute are 1 (enable) and +0 (disable). + +To enable live migration: + + echo 1 /sys/bus/mdev/devices/$UUID/migratable + +To disable live migration: + + echo 0 /sys/bus/mdev/devices/$UUID/migratable + +The 'migratable' attribute allows migration to be dynamically enabled or +disabled for the guest to which the mediated device is attached; however, +if migration of the vfio-ap device is already in progress, the operation will +be rejected with a 'Device or resource busy' message. + +Live guest migration failures due to AP configuration errors +------------------------------------------------------------ +The VFIO AP mediated device is not used to provide userspace with direct access +to a device as is the case with other devices that use the VFIO framework to +pass them through to a guest. The sole purpose of the VFIO AP mediated device +is to manage an AP configuration for a guest. An AP configuration is comprised +of the AP adapter IDs (APID), AP queue indexes (APQI) and domain numbers of the +control domains to which a guest will be granted access. Once the VFIO AP +mediated device is attached to the guest, its AP configuration is set by the +vfio_ap device driver. Once set, all access to the AP devices is handled +by the s390 Interpretive Execution facility; in other words, the vfio_ap device +driver plays no role in providing direct access to these AP devices. + +The only role that the vfio_ap device driver plays in the migration process is +to verify that the AP configuration for the source guest is compatible with the +AP configuration of the destination guest. Incompatibility will result in a live +guest migration failure. This section describes how to debug this type of +failure. + +The destination host is missing the mediated device with the same name as the +mdev attached to the source guest +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Source host** + ++--------------+--------------------------------------------------------------------+ +| Log | Message | ++==============+====================================================================+ +| Console log: | *error: device not found: mediated device '$UUID' not found* | ++--------------+--------------------------------------------------------------------+ + +The source guest is not enabled for migration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +**Source host** + ++--------------+--------------------------------------------------------------------+ +| Log | Message | ++==============+====================================================================+ +| Console log: | *error: Requested operation is not valid: | +| | cannot migrate domain: $UUID: Migration is disabled for VFIO | +| | device* | ++--------------+--------------------------------------------------------------------+ + + +The AP configuration of the source and destination guests are not compatible +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +**Source host** + ++--------------+--------------------------------------------------------------------+ +| Log | Message | ++==============+====================================================================+ +| Console log: | *error: operation failed: job 'migration in' failed: load of | +| | migration failed: Bad address* | ++--------------+--------------------------------------------------------------------+ +| Kernel log: | N/A | ++--------------+--------------------------------------------------------------------+ +| QEMU log: | *initiating migration* | +| | | +| | *qemu-system-s390x: Sibling indicated error 1* | ++--------------+--------------------------------------------------------------------+ + +**Destination host** + ++--------------+--------------------------------------------------------------------+ +| Log | Message | ++==============+====================================================================+ +| Console log: | N/A | ++--------------+--------------------------------------------------------------------+ +| Kernel log: | *vfio_ap_mdev $UUID: Migration failed: Source and target queues | +| | ($APQN) are not compatible* | +| | | +| | The message above will be followed by one or more messages | +| | enumerating the incompatible features; for example: | +| | | +| | *vfio_ap_mdev $UUID: APSC facility installed in source queue $APQN*| +| | | +| | *vfio_ap_mdev $UUID: APSC facility not installed in target queue | +| | $APQN* | +| | | +| | *AP type of source ($APTYPE) not compatible with target ($APTYPE)* | ++--------------+--------------------------------------------------------------------+ +| QEMU log: | *initiating migration* | +| | | +| | *qemu-system-s390x: error while loading state section id ...* | +| | | +| | *shutting down, reason=failed* | +| | | +| | *terminating on signal 15 from pid 1196 (/usr/sbin/virtqemud)* | ++--------------+--------------------------------------------------------------------+ + +The AP configuration of the source guest has more APQNS than the destination guest +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +**Source host** + ++--------------+--------------------------------------------------------------------+ +| Log | Message | ++==============+====================================================================+ +| Console log: | *error: operation f1e166ee77e6failed: job 'migration in' failed: load of | +| | migration failed: Input/output error* | ++--------------+--------------------------------------------------------------------+ +| Kernel log: | N/A | ++--------------+--------------------------------------------------------------------+ +| QEMU log: | *initiating migration* | +| | | +| | *qemu-system-s390x: Sibling indicated error 1* | ++--------------+--------------------------------------------------------------------+ + +**Destination host** + ++--------------+--------------------------------------------------------------------+ +| Log | Message | ++==============+====================================================================+ +| Console log: | N/A | ++--------------+--------------------------------------------------------------------+ +| Kernel log: | *vfio_ap_mdev $UUID: | +| | migration failed: source guest's AP config size (xx) larger than | +| | target's (yy)* | ++--------------+--------------------------------------------------------------------+ +| QEMU log: | *initiating migration* | +| | | +| | *qemu-system-s390x: error while loading state section id ...* | +| | | +| | *shutting down, reason=failed* | +| | | +| | *terminating on signal 15 from pid 1196 (/usr/sbin/virtqemud)* | ++--------------+--------------------------------------------------------------------+ + +The AP configuration of the source guest has fewer APQNS than the destination guest +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +**Source host** + ++--------------+--------------------------------------------------------------------+ +| Log | Message | ++==============+====================================================================+ +| Console log: | *error: operation failed: job 'migration in' failed: load of | +| | migration failed: No such device* | ++--------------+--------------------------------------------------------------------+ +| Kernel log: | N/A | ++--------------+--------------------------------------------------------------------+ +| QEMU log: | *initiating migration* | +| | | +| | *qemu-system-s390x: Sibling indicated error 1* | ++--------------+--------------------------------------------------------------------+ + +**Destination host** + ++--------------+--------------------------------------------------------------------+ +| Log | Message | ++==============+====================================================================+ +| Console log: | N/A | ++--------------+--------------------------------------------------------------------+ +| Kernel log: | *vfio_ap_mdev $UUID: | +| | migration failed: number of queues on source (x) and target (y) | +| | guests differ* | ++--------------+--------------------------------------------------------------------+ +| QEMU log: | *initiating migration* | +| | | +| | *qemu-system-s390x: error while loading state section id ...* | +| | | +| | *shutting down, reason=failed* | +| | | +| | *terminating on signal 15 from pid 1196 (/usr/sbin/virtqemud)* | ++--------------+--------------------------------------------------------------------+
Signature · 1 line
2.52.0
Alex Williamson
maintainerQuoted 20 lines · On Wed, 25 Mar 2026 17:00:47 -0400 Anthony Krowiak <[email protected]>…
This patch series implements live guest migration of a guest to which AP devices have been passed through. To better comprehend this design, one has to understand that VFIO AP mediated device is not used to provide userspace with direct access to a device as is the case with other devices that use the VFIO framework to pass them through to a guest. The sole purpose of the VFIO AP mediated device is to manage an AP configuration for a guest. An AP configuration is comprised of the AP adapter IDs (APID), AP queue indexes (APQI) and domain numbers of the control domains to which a guest will be granted access. Once the VFIO AP mediated device is attached to a guest, its AP configuration is set by the
vfio_apdevice driver. Once set, all access to the AP devices is handled by the s390 Interpretive Execution facility; in other words, thevfio_apdevice driver plays no role in providing direct access to the AP devices in the guest's AP configuration.The only role that the
vfio_apdevice driver plays in the migration process is to verify that the AP configuration for the source guest is compatible with the AP configuration of the destination guest. Incompatibility will result in a live guest migration failure. In order to be compatible, the following requirements must be met:
Quoted 3 lines
1. The destination guest will be started with the same QEMU command line as the source guest, so the mediated device supplying the AP configuration on both guests must have the same name (UUID).
AFAIK, same UUID is not a requirement for out-of-tree mdev drivers supporting migration. You're really concerned more with the configuration/composition of the mdev device, so requiring the same UUID seems a bit arbitrary.
Quoted 5 lines
2. The AP configuration assigned via the VFIO AP mediated device on both guests must be compatible. As such, each AP configuration must meet the following requirements:
Quoted 7 lines
* Both guests must have the same number of APQNs
* Each APQN assigned to the source guest must also be assigned to the destination guest
* Each APQN assigned to both guests must reference an AP queue with the same hardware capabilities
Why isn't this sufficient vs also requiring the same UUID?
Quoted 5 lines
Note: There is a forthcoming consumer of this series which will be a QEMU
patch series is entitled:
'hw/vfio/ap: Implement live guest migration of guests using AP devices'
Quoted 10 lines
This design also adds a use case for enabling and disabling migration of guests to which AP devices have been passed through. To facilitate this, a new read/write sysfs 'migratable' attribute is added to the mediated device. This attribute specifies whether the vfio device is migratable (1) or not (0). When the value of this attribute is changed, the
vfio_apdevice driver will signal an eventfd to userspace. It is up to userspace to respond to the change by enabling or disabling migration of the guest to which the mediated device is attached. The operation will be rejected with a 'Device or resource busy' message if a migration is in progress.
This seems inherently racy. What happens if the device becomes unmigratable while it's being migrated?
If userspace is deciding that the device is now unmigratable, why does it need to signal this through the kernel driver rather than with the userspace orchestration agent? The entire path seems unnecessary.
Quoted 6 lines
Userspace must also have a means for retrieving the value of the sysfs 'migratable' attribute when the guest is started to initialize whether it can be migrated. For this, The
VFIO_DEVICE_GET_INFOioctl is used. The structvfio_device_infoobject passed to the ioctl will be extended with a capability specifying the vfio device attributes. One of the attributes will contain the value of the mediated device's 'migratable' attribute.
This is just broken, it's redundant to our current device feature mechanism for exposing migration support. If you want the capability to create unmigratable devices statically, can't that be encompassed within the definition of the mdev type? Dynamic migration support just seems like it's involving the kernel in orchestration it shouldn't be a part of. Thanks,
Alex
Quoted 45 lines
Anthony Krowiak (23):
s390/vfio-ap: Store queue hardware info when probed s390/vfio-ap: Provide access to queue objects and related info s390/vfio-ap: Add header file for xfer of vfio device caps to
userspace
MAINTAINERS: Add new header file for the S390 VFIO AP DRIVER
maintainers
s390/vfio-ap: A sysfs 'migratable' attribute to enable/disable
migration of guest
s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute s390/vfio-ap: Signal event to enable/disable live guest migration s390/vfio-ap: Return value of sysfs migratable attribute from
VFIO_DEVICE_GET_INFOioctl
s390/vfio-ap: Data structures for facilitating vfio device migration s390/vfio-ap: Initialize/release vfio device migration data
s390-vfio-ap: Callback to set vfio device mig state during guest
migration
s390/vfio-ap: Transition guest migration state from STOP toSTOP_COPYs390/vfio-ap: File ops called to save the vfio device migration state s390/vfio-ap: Transition device migration state from STOP to RESUMING s390/vfio-ap: File ops called to resume the vfio device migration s390/vfio-ap: Transition device migration state from RESUMING to STOP s390/vfio-ap: Transition device migration state fromSTOP_COPYto STOP s390/vfio-ap: Transition device migration state from STOP to RUNNING
and vice versa
s390-vfio-ap: Callback to get the current vfio device migration state s390/vfio-ap: Callback to get the size of data to be migrated during
guest migration
s390/vfio-ap: Provide API to query whether migration is in progress s390/vfio-ap: Disallow blocking migration in progress s390/vfio-ap: Add live guest migration chapter tovfio-ap.rstDocumentation/arch/s390/vfio-ap.rst | 339 +++++-- MAINTAINERS | 1 + drivers/s390/crypto/Makefile | 2 +- drivers/s390/crypto/vfio_ap_drv.c | 4 +- drivers/s390/crypto/vfio_ap_migration.c | 1131 +++++++++++++++++++++++ drivers/s390/crypto/vfio_ap_ops.c | 263 +++++- drivers/s390/crypto/vfio_ap_private.h | 20 + include/uapi/linux/vfio.h | 2 + include/uapi/linux/vfio_ap.h | 34 + 9 files changed, 1685 insertions(+), 111 deletions(-) create mode 100644 drivers/s390/crypto/vfio_ap_migration.c create mode 100644 include/uapi/linux/vfio_ap.h
Anthony Krowiak
maintainerQuoted 23 lines · Alex Williamson
Quoted 20 lines · On Wed, 25 Mar 2026 17:00:47 -0400 Anthony Krowiak <[email protected]>…
This patch series implements live guest migration of a guest to which AP devices have been passed through. To better comprehend this design, one has to understand that VFIO AP mediated device is not used to provide userspace with direct access to a device as is the case with other devices that use the VFIO framework to pass them through to a guest. The sole purpose of the VFIO AP mediated device is to manage an AP configuration for a guest. An AP configuration is comprised of the AP adapter IDs (APID), AP queue indexes (APQI) and domain numbers of the control domains to which a guest will be granted access. Once the VFIO AP mediated device is attached to a guest, its AP configuration is set by the
vfio_apdevice driver. Once set, all access to the AP devices is handled by the s390 Interpretive Execution facility; in other words, thevfio_apdevice driver plays no role in providing direct access to the AP devices in the guest's AP configuration.The only role that the
vfio_apdevice driver plays in the migration process is to verify that the AP configuration for the source guest is compatible with the AP configuration of the destination guest. Incompatibility will result in a live guest migration failure. In order to be compatible, the following requirements must be met:
Quoted 7 lines
Quoted 3 lines
1. The destination guest will be started with the same QEMU command line as the source guest, so the mediated device supplying the AP configuration on both guests must have the same name (UUID).
AFAIK, same UUID is not a requirement for out-of-tree mdev drivers supporting migration. You're really concerned more with the configuration/composition of the mdev device, so requiring the same UUID seems a bit arbitrary.
As stated above, the destination guest will be started with the same QEMU command line as the source guest. Within that command line will be a '-device' parameter like the following:
-device '{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}
Note that sysfsdev is the path to the mdev named 62177883-f1bb-47f0-914d-32a22e3a8804; therefore, the mdev with that name must exist on the destination guest or the migration will fail with the following error:
error: device not found: mediated device '62177883-f1bb-47f0-914d-32a22e3a8804' not found
Quoted 96 lines
Quoted 5 lines
2. The AP configuration assigned via the VFIO AP mediated device on both guests must be compatible. As such, each AP configuration must meet the following requirements:
Quoted 7 lines
* Both guests must have the same number of APQNs
* Each APQN assigned to the source guest must also be assigned to the destination guest
* Each APQN assigned to both guests must reference an AP queue with the same hardware capabilities
Why isn't this sufficient vs also requiring the same UUID?
Quoted 5 lines
Note: There is a forthcoming consumer of this series which will be a QEMU
patch series is entitled:
'hw/vfio/ap: Implement live guest migration of guests using AP devices'Quoted 10 lines
This design also adds a use case for enabling and disabling migration of guests to which AP devices have been passed through. To facilitate this, a new read/write sysfs 'migratable' attribute is added to the mediated device. This attribute specifies whether the vfio device is migratable (1) or not (0). When the value of this attribute is changed, the
vfio_apdevice driver will signal an eventfd to userspace. It is up to userspace to respond to the change by enabling or disabling migration of the guest to which the mediated device is attached. The operation will be rejected with a 'Device or resource busy' message if a migration is in progress.This seems inherently racy. What happens if the device becomes unmigratable while it's being migrated?
If userspace is deciding that the device is now unmigratable, why does it need to signal this through the kernel driver rather than with the userspace orchestration agent? The entire path seems unnecessary.
Quoted 6 lines
Userspace must also have a means for retrieving the value of the sysfs 'migratable' attribute when the guest is started to initialize whether it can be migrated. For this, The
VFIO_DEVICE_GET_INFOioctl is used. The structvfio_device_infoobject passed to the ioctl will be extended with a capability specifying the vfio device attributes. One of the attributes will contain the value of the mediated device's 'migratable' attribute.This is just broken, it's redundant to our current device feature mechanism for exposing migration support. If you want the capability to create unmigratable devices statically, can't that be encompassed within the definition of the mdev type? Dynamic migration support just seems like it's involving the kernel in orchestration it shouldn't be a part of. Thanks,
Alex
Quoted 45 lines
Anthony Krowiak (23):
s390/vfio-ap: Store queue hardware info when probed s390/vfio-ap: Provide access to queue objects and related info s390/vfio-ap: Add header file for xfer of vfio device caps to
userspace
MAINTAINERS: Add new header file for the S390 VFIO AP DRIVER
maintainers
s390/vfio-ap: A sysfs 'migratable' attribute to enable/disable
migration of guest
s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute s390/vfio-ap: Signal event to enable/disable live guest migration s390/vfio-ap: Return value of sysfs migratable attribute from
VFIO_DEVICE_GET_INFOioctl
s390/vfio-ap: Data structures for facilitating vfio device migration s390/vfio-ap: Initialize/release vfio device migration data
s390-vfio-ap: Callback to set vfio device mig state during guest
migration
s390/vfio-ap: Transition guest migration state from STOP toSTOP_COPYs390/vfio-ap: File ops called to save the vfio device migration state s390/vfio-ap: Transition device migration state from STOP to RESUMING s390/vfio-ap: File ops called to resume the vfio device migration s390/vfio-ap: Transition device migration state from RESUMING to STOP s390/vfio-ap: Transition device migration state fromSTOP_COPYto STOP s390/vfio-ap: Transition device migration state from STOP to RUNNING
and vice versa
s390-vfio-ap: Callback to get the current vfio device migration state s390/vfio-ap: Callback to get the size of data to be migrated during
guest migration
s390/vfio-ap: Provide API to query whether migration is in progress s390/vfio-ap: Disallow blocking migration in progress s390/vfio-ap: Add live guest migration chapter tovfio-ap.rstDocumentation/arch/s390/vfio-ap.rst | 339 +++++-- MAINTAINERS | 1 + drivers/s390/crypto/Makefile | 2 +- drivers/s390/crypto/vfio_ap_drv.c | 4 +- drivers/s390/crypto/vfio_ap_migration.c | 1131 +++++++++++++++++++++++ drivers/s390/crypto/vfio_ap_ops.c | 263 +++++- drivers/s390/crypto/vfio_ap_private.h | 20 + include/uapi/linux/vfio.h | 2 + include/uapi/linux/vfio_ap.h | 34 + 9 files changed, 1685 insertions(+), 111 deletions(-)create mode 100644
drivers/s390/crypto/vfio_ap_migration.ccreate mode 100644include/uapi/linux/vfio_ap.h
Anthony Krowiak
maintainerQuoted 35 lines · Alex Williamson
Quoted 20 lines · On Wed, 25 Mar 2026 17:00:47 -0400 Anthony Krowiak <[email protected]>…
This patch series implements live guest migration of a guest to which AP devices have been passed through. To better comprehend this design, one has to understand that VFIO AP mediated device is not used to provide userspace with direct access to a device as is the case with other devices that use the VFIO framework to pass them through to a guest. The sole purpose of the VFIO AP mediated device is to manage an AP configuration for a guest. An AP configuration is comprised of the AP adapter IDs (APID), AP queue indexes (APQI) and domain numbers of the control domains to which a guest will be granted access. Once the VFIO AP mediated device is attached to a guest, its AP configuration is set by the
vfio_apdevice driver. Once set, all access to the AP devices is handled by the s390 Interpretive Execution facility; in other words, thevfio_apdevice driver plays no role in providing direct access to the AP devices in the guest's AP configuration.The only role that the
vfio_apdevice driver plays in the migration process is to verify that the AP configuration for the source guest is compatible with the AP configuration of the destination guest. Incompatibility will result in a live guest migration failure. In order to be compatible, the following requirements must be met:Quoted 3 lines
1. The destination guest will be started with the same QEMU command line as the source guest, so the mediated device supplying the AP configuration on both guests must have the same name (UUID).
AFAIK, same UUID is not a requirement for out-of-tree mdev drivers supporting migration. You're really concerned more with the configuration/composition of the mdev device, so requiring the same UUID seems a bit arbitrary.
Quoted 4 lines
2. The AP configuration assigned via the VFIO AP mediated device on both guests must be compatible. As such, each AP configuration must meet the following requirements:
Quoted 8 lines
Quoted 7 lines
* Both guests must have the same number of APQNs
* Each APQN assigned to the source guest must also be assigned to the destination guest
* Each APQN assigned to both guests must reference an AP queue with the same hardware capabilities
Why isn't this sufficient vs also requiring the same UUID?
I explained why in my previous response.
Quoted 16 lines
Quoted 6 lines
Note: There is a forthcoming consumer of this series which will be a QEMU
patch series is entitled:
'hw/vfio/ap: Implement live guest migration of guests using AP devices'Quoted 10 lines
This design also adds a use case for enabling and disabling migration of guests to which AP devices have been passed through. To facilitate this, a new read/write sysfs 'migratable' attribute is added to the mediated device. This attribute specifies whether the vfio device is migratable (1) or not (0). When the value of this attribute is changed, the
vfio_apdevice driver will signal an eventfd to userspace. It is up to userspace to respond to the change by enabling or disabling migration of the guest to which the mediated device is attached. The operation will be rejected with a 'Device or resource busy' message if a migration is in progress.
Quoted 6 lines
This seems inherently racy. What happens if the device becomes unmigratable while it's being migrated?
If userspace is deciding that the device is now unmigratable, why does it need to signal this through the kernel driver rather than with the userspace orchestration agent? The entire path seems unnecessary.
I am not familiar with what a userspace orchestration agent is, so I can't address that. Can you please describe how that would work?
Maybe it would help to provide the reason for this. For certain types of crypto operations, a master key must be configured for the crypto card domain being used. This master key must be synchronized between the source and destination crypto device so that in-flight crypto operations can be completed during migration. If these master keys must be changed, migration must be blocked until the master key changes can be synchronized between the source and destination system(s).
Quoted 7 lines
Quoted 7 lines
Userspace must also have a means for retrieving the value of the sysfs 'migratable' attribute when the guest is started to initialize whether it can be migrated. For this, The
VFIO_DEVICE_GET_INFOioctl is used. The structvfio_device_infoobject passed to the ioctl will be extended with a capability specifying the vfio device attributes. One of the attributes will contain the value of the mediated device's 'migratable' attribute.
Quoted 6 lines
This is just broken, it's redundant to our current device feature mechanism for exposing migration support. If you want the capability to create unmigratable devices statically, can't that be encompassed within the definition of the mdev type? Dynamic migration support just seems like it's involving the kernel in orchestration it shouldn't be a part of.
So, it appears you are suggesting the creation of a new mdev type for unmigratable crypto devices. I don't see the value in that. As I stated above, there is a valid reason for wanting to prevent migration while master key synchronization is taking place.
If this feature violates the implicit rules of vfio device migration, then so be it. Maybe we have to figure out another way to ensure migration is not initiated during master key synchronization.
If we can't find an acceptable means to do this programmatically, then maybe it will come down to a matter of documenting the need to ensure migration is not initiated while master key synchronization is taking place. This would put the onus on the various system administrators responsible for host, guest and master key administration to communicate out of band to ensure they are all on the same page with regard to migration.
It would be preferable to be able to do this with a userspace interface, so any suggestions would be greatly appreciated.
Quoted 3 lines
Thanks,
Alex
Thank you for your review.
Quoted 46 lines
Quoted 46 lines
Anthony Krowiak (23):
s390/vfio-ap: Store queue hardware info when probed s390/vfio-ap: Provide access to queue objects and related info s390/vfio-ap: Add header file for xfer of vfio device caps to
userspace
MAINTAINERS: Add new header file for the S390 VFIO AP DRIVER
maintainers
s390/vfio-ap: A sysfs 'migratable' attribute to enable/disable
migration of guest
s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute s390/vfio-ap: Signal event to enable/disable live guest migration s390/vfio-ap: Return value of sysfs migratable attribute from
VFIO_DEVICE_GET_INFOioctl
s390/vfio-ap: Data structures for facilitating vfio device migration s390/vfio-ap: Initialize/release vfio device migration data
s390-vfio-ap: Callback to set vfio device mig state during guest
migration
s390/vfio-ap: Transition guest migration state from STOP toSTOP_COPYs390/vfio-ap: File ops called to save the vfio device migration state s390/vfio-ap: Transition device migration state from STOP to RESUMING s390/vfio-ap: File ops called to resume the vfio device migration s390/vfio-ap: Transition device migration state from RESUMING to STOP s390/vfio-ap: Transition device migration state fromSTOP_COPYto STOP s390/vfio-ap: Transition device migration state from STOP to RUNNING
and vice versa
s390-vfio-ap: Callback to get the current vfio device migration state s390/vfio-ap: Callback to get the size of data to be migrated during
guest migration
s390/vfio-ap: Provide API to query whether migration is in progress s390/vfio-ap: Disallow blocking migration in progress s390/vfio-ap: Add live guest migration chapter tovfio-ap.rstDocumentation/arch/s390/vfio-ap.rst | 339 +++++-- MAINTAINERS | 1 + drivers/s390/crypto/Makefile | 2 +- drivers/s390/crypto/vfio_ap_drv.c | 4 +- drivers/s390/crypto/vfio_ap_migration.c | 1131 +++++++++++++++++++++++ drivers/s390/crypto/vfio_ap_ops.c | 263 +++++- drivers/s390/crypto/vfio_ap_private.h | 20 + include/uapi/linux/vfio.h | 2 + include/uapi/linux/vfio_ap.h | 34 + 9 files changed, 1685 insertions(+), 111 deletions(-)create mode 100644
drivers/s390/crypto/vfio_ap_migration.ccreate mode 100644include/uapi/linux/vfio_ap.h
Alex Williamson
maintainerQuoted 24 lines · On Tue, 31 Mar 2026 08:07:06 -0400 Anthony Krowiak <[email protected]>…
Quoted 23 lines · Alex Williamson
Quoted 20 lines · On Wed, 25 Mar 2026 17:00:47 -0400 Anthony Krowiak <[email protected]>…
This patch series implements live guest migration of a guest to which AP devices have been passed through. To better comprehend this design, one has to understand that VFIO AP mediated device is not used to provide userspace with direct access to a device as is the case with other devices that use the VFIO framework to pass them through to a guest. The sole purpose of the VFIO AP mediated device is to manage an AP configuration for a guest. An AP configuration is comprised of the AP adapter IDs (APID), AP queue indexes (APQI) and domain numbers of the control domains to which a guest will be granted access. Once the VFIO AP mediated device is attached to a guest, its AP configuration is set by the
vfio_apdevice driver. Once set, all access to the AP devices is handled by the s390 Interpretive Execution facility; in other words, thevfio_apdevice driver plays no role in providing direct access to the AP devices in the guest's AP configuration.The only role that the
vfio_apdevice driver plays in the migration process is to verify that the AP configuration for the source guest is compatible with the AP configuration of the destination guest. Incompatibility will result in a live guest migration failure. In order to be compatible, the following requirements must be met:
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
1. The destination guest will be started with the same QEMU command line as the source guest, so the mediated device supplying the AP configuration on both guests must have the same name (UUID).
AFAIK, same UUID is not a requirement for out-of-tree mdev drivers supporting migration. You're really concerned more with the configuration/composition of the mdev device, so requiring the same UUID seems a bit arbitrary.
Combining replies:
Quoted 8 lines · On Tue, 31 Mar 2026 07:17:08 -0400 Anthony Krowiak <[email protected]>…
As stated above, the destination guest will be started with the same QEMU command line as the source guest. Within that command line will be a '-device' parameter like the following:
-device '{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}
Quoted 7 lines
Note that sysfsdev is the path to the mdev named 62177883-f1bb-47f0-914d-32a22e3a8804;
therefore, the mdev with that name must exist on the destination guest or the migration will fail with the following error:error: device not found: mediated device
'62177883-f1bb-47f0-914d-32a22e3a8804' not found
Then this is a requirement of your tooling, not a kernel requirement, not something the kernel should care about. QEMU matches devices by their virtual bus path, not the sysfsdev or host attributes. In the case of VF migration with vfio-pci variant drivers we cannot require that the source and target devices exist at the same bus address. Ideally the pre-copy data from the source device to the target will include relevant configuration information to validate that the source and target are compatible, regardless of the uuid.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
2. The AP configuration assigned via the VFIO AP mediated device on both guests must be compatible. As such, each AP configuration must meet the following requirements:
* Both guests must have the same number of APQNs
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
* Each APQN assigned to the source guest must also be assigned to the destination guest
* Each APQN assigned to both guests must reference an AP queue with the same hardware capabilities
Why isn't this sufficient vs also requiring the same UUID?
I explained why in my previous response.
See above, userspace tooling requirements don't imply kernel requirement.
Quoted 18 lines
Quoted 18 lines
Quoted 5 lines
Note: There is a forthcoming consumer of this series which will be a QEMU
patch series is entitled:
'hw/vfio/ap: Implement live guest migration of guests using AP devices'Quoted 10 lines
This design also adds a use case for enabling and disabling migration of guests to which AP devices have been passed through. To facilitate this, a new read/write sysfs 'migratable' attribute is added to the mediated device. This attribute specifies whether the vfio device is migratable (1) or not (0). When the value of this attribute is changed, the
vfio_apdevice driver will signal an eventfd to userspace. It is up to userspace to respond to the change by enabling or disabling migration of the guest to which the mediated device is attached. The operation will be rejected with a 'Device or resource busy' message if a migration is in progress.This seems inherently racy. What happens if the device becomes unmigratable while it's being migrated?
Quoted 6 lines
Quoted 3 lines
If userspace is deciding that the device is now unmigratable, why does it need to signal this through the kernel driver rather than with the userspace orchestration agent? The entire path seems unnecessary.
I am not familiar with what a userspace orchestration agent is, so I can't address that. Can you please describe how that would work?
Something in userspace, perhaps libvirt, is managing the VM. It needs to coordinate with a counterpart on another host managing and configuring the target VM to accept the migration data stream. There's likely also another entity that's responsible for deciding this migration should occur and where to place the target. All of this is what I'm referring to as orchestration.
The path of a userspace agent writing into a sysfs attribute to mark the device as becoming non-migratable so that the host driver can send an interrupt to the VM to poll an INFO ioctl to block migration is an over-engineered path at the wrong layer versus some sort of RPC to the managing process, or even to QEMU, to block migrations.
Additionally, if the device actually itself becomes non-migratable, it can simply fail any migration state transition other than returning to RUNNING and can generate errors in the data stream if it needs to abort an in-progress migration.
Quoted 8 lines
Maybe it would help to provide the reason for this. For certain types of crypto operations, a master key must be configured for the crypto card domain being used. This master key must be synchronized between the source and destination crypto device so that in-flight crypto operations can be completed during migration. If these master keys must be changed, migration must be blocked until the master key changes can be synchronized between the source and destination system(s).
This sounds like a userspace orchestration problem, not a kernel problem. It might be a valid choice to use the mechanisms I outline above to abort an in progress migration if a new master key is configured during migration, but it's not the kernel's problem to provide a synchronization point for this through the kernel.
Quoted 13 lines
Quoted 6 lines
Quoted 6 lines
Userspace must also have a means for retrieving the value of the sysfs 'migratable' attribute when the guest is started to initialize whether it can be migrated. For this, The
VFIO_DEVICE_GET_INFOioctl is used. The structvfio_device_infoobject passed to the ioctl will be extended with a capability specifying the vfio device attributes. One of the attributes will contain the value of the mediated device's 'migratable' attribute.Quoted 6 lines
This is just broken, it's redundant to our current device feature mechanism for exposing migration support. If you want the capability to create unmigratable devices statically, can't that be encompassed within the definition of the mdev type? Dynamic migration support just seems like it's involving the kernel in orchestration it shouldn't be a part of.
Quoted 4 lines
So, it appears you are suggesting the creation of a new mdev type for unmigratable crypto devices. I don't see the value in that. As I stated above, there is a valid reason for wanting to prevent migration while master key synchronization is taking place.
Then prevent it in userspace.
Quoted 3 lines
If this feature violates the implicit rules of vfio device migration, then so be it. Maybe we have to figure out another way to ensure migration is not initiated during master key synchronization.
If there's a software entity that has the ability to write to sysfs to declare that a device is not currently available for migration, give it the ability to notify whatever entity is coordinating and scheduling, ie. orchestrating, the migration rather than creating a channel through the device. Combine that with using the existing mechanisms to abort a migration if its already in progress.
Quoted 10 lines
If we can't find an acceptable means to do this programmatically, then maybe it will come down to a matter of documenting the need to ensure migration is not initiated while master key synchronization is taking place. This would put the onus on the various system administrators responsible for host, guest and master key administration to communicate out of band to ensure they are all on the same page with regard to migration.
It would be preferable to be able to do this with a userspace interface, so any suggestions would be greatly appreciated.
A userspace interface can still exist, I just don't find this path through the driver to the VM acceptable with this justification. Mechanisms already exist for the device to refuse a state transition or generate an error for a migration already in progress. IMHO, it would be acceptable for the device to block a key change if the migration is already in progress. If the key change cannot be represented in the migration data stream, then it's up to the orchestration of the migration to make sure they stay synchronized, but I don't see that the vfio uAPI needs to be involved. Thanks,
Alex
Anthony Krowiak
maintainerQuoted 52 lines · Alex Williamson
Quoted 24 lines · On Tue, 31 Mar 2026 08:07:06 -0400 Anthony Krowiak <[email protected]>…
Quoted 23 lines · Alex Williamson
Quoted 20 lines · On Wed, 25 Mar 2026 17:00:47 -0400 Anthony Krowiak <[email protected]>…
This patch series implements live guest migration of a guest to which AP devices have been passed through. To better comprehend this design, one has to understand that VFIO AP mediated device is not used to provide userspace with direct access to a device as is the case with other devices that use the VFIO framework to pass them through to a guest. The sole purpose of the VFIO AP mediated device is to manage an AP configuration for a guest. An AP configuration is comprised of the AP adapter IDs (APID), AP queue indexes (APQI) and domain numbers of the control domains to which a guest will be granted access. Once the VFIO AP mediated device is attached to a guest, its AP configuration is set by the
vfio_apdevice driver. Once set, all access to the AP devices is handled by the s390 Interpretive Execution facility; in other words, thevfio_apdevice driver plays no role in providing direct access to the AP devices in the guest's AP configuration.The only role that the
vfio_apdevice driver plays in the migration process is to verify that the AP configuration for the source guest is compatible with the AP configuration of the destination guest. Incompatibility will result in a live guest migration failure. In order to be compatible, the following requirements must be met:Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
1. The destination guest will be started with the same QEMU command line as the source guest, so the mediated device supplying the AP configuration on both guests must have the same name (UUID).
AFAIK, same UUID is not a requirement for out-of-tree mdev drivers supporting migration. You're really concerned more with the configuration/composition of the mdev device, so requiring the same UUID seems a bit arbitrary.
Combining replies:
Quoted 7 lines · On Tue, 31 Mar 2026 07:17:08 -0400 Anthony Krowiak <[email protected]>…
As stated above, the destination guest will be started with the same QEMU command line as the source guest. Within that command line will be a '-device' parameter like the following:
-device '{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}
Quoted 7 lines
Note that sysfsdev is the path to the mdev named 62177883-f1bb-47f0-914d-32a22e3a8804;
therefore, the mdev with that name must exist on the destination guest or the migration will fail with the following error:error: device not found: mediated device
'62177883-f1bb-47f0-914d-32a22e3a8804' not found
Quoted 8 lines
Then this is a requirement of your tooling, not a kernel requirement, not something the kernel should care about. QEMU matches devices by their virtual bus path, not the sysfsdev or host attributes. In the case of VF migration with vfio-pci variant drivers we cannot require that the source and target devices exist at the same bus address. Ideally the pre-copy data from the source device to the target will include relevant configuration information to validate that the source and target are compatible, regardless of the uuid.
Maybe the problem here is stating that having the same UUID is a requirement in the patch series description. I agree this is not a kernel requirement; however, a live guest migration will not succeed unless the destination host has a mediated device with a UUID that matches that of the UUID of the mediated device used to pass through s390 crypto devices to the source guest for the reason I stated above. Would it help if I removed item #1 as a requirement?
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
2. The AP configuration assigned via the VFIO AP mediated device on both guests must be compatible. As such, each AP configuration must meet the following requirements:
* Both guests must have the same number of APQNs
* Each APQN assigned to the source guest must also be assigned to the destination guest
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
* Each APQN assigned to both guests must reference an AP queue with the same hardware capabilities
Why isn't this sufficient vs also requiring the same UUID?
I explained why in my previous response.
See above, userspace tooling requirements don't imply kernel requirement.
Got it.
Quoted 9 lines
A userspace interface can still exist, I just don't find this path through the driver to the VM acceptable with this justification. Mechanisms already exist for the device to refuse a state transition or generate an error for a migration already in progress. IMHO, it would be acceptable for the device to block a key change if the migration is already in progress. If the key change cannot be represented in the migration data stream, then it's up to the orchestration of the migration to make sure they stay synchronized, but I don't see that the vfio uAPI needs to be involved. Thanks,
As I stated in the cover letter description, all access to the AP devices is handled by the s390 Interpretive Execution facility, so without a complete redesign of the vfio_ap device driver, there is no way to know that a master key change is being requested. I do get your point, however, so I will figure out how best to handle this in userspace. Thanks for you review.
Quoted 2 lines
Alex
Alex Williamson
maintainerQuoted 62 lines · On Wed, 1 Apr 2026 09:38:59 -0400 Anthony Krowiak <[email protected]>…
Quoted 52 lines · Alex Williamson
Quoted 24 lines · On Tue, 31 Mar 2026 08:07:06 -0400 Anthony Krowiak <[email protected]>…
Quoted 23 lines · Alex Williamson
Quoted 20 lines · On Wed, 25 Mar 2026 17:00:47 -0400 Anthony Krowiak <[email protected]>…
This patch series implements live guest migration of a guest to which AP devices have been passed through. To better comprehend this design, one has to understand that VFIO AP mediated device is not used to provide userspace with direct access to a device as is the case with other devices that use the VFIO framework to pass them through to a guest. The sole purpose of the VFIO AP mediated device is to manage an AP configuration for a guest. An AP configuration is comprised of the AP adapter IDs (APID), AP queue indexes (APQI) and domain numbers of the control domains to which a guest will be granted access. Once the VFIO AP mediated device is attached to a guest, its AP configuration is set by the
vfio_apdevice driver. Once set, all access to the AP devices is handled by the s390 Interpretive Execution facility; in other words, thevfio_apdevice driver plays no role in providing direct access to the AP devices in the guest's AP configuration.The only role that the
vfio_apdevice driver plays in the migration process is to verify that the AP configuration for the source guest is compatible with the AP configuration of the destination guest. Incompatibility will result in a live guest migration failure. In order to be compatible, the following requirements must be met:Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
1. The destination guest will be started with the same QEMU command line as the source guest, so the mediated device supplying the AP configuration on both guests must have the same name (UUID).
AFAIK, same UUID is not a requirement for out-of-tree mdev drivers supporting migration. You're really concerned more with the configuration/composition of the mdev device, so requiring the same UUID seems a bit arbitrary.
Combining replies:
Quoted 7 lines · On Tue, 31 Mar 2026 07:17:08 -0400 Anthony Krowiak <[email protected]>…
As stated above, the destination guest will be started with the same QEMU command line as the source guest. Within that command line will be a '-device' parameter like the following:
-device '{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}
Quoted 7 lines
Note that sysfsdev is the path to the mdev named 62177883-f1bb-47f0-914d-32a22e3a8804;
therefore, the mdev with that name must exist on the destination guest or the migration will fail with the following error:error: device not found: mediated device
'62177883-f1bb-47f0-914d-32a22e3a8804' not foundQuoted 8 lines
Then this is a requirement of your tooling, not a kernel requirement, not something the kernel should care about. QEMU matches devices by their virtual bus path, not the sysfsdev or host attributes. In the case of VF migration with vfio-pci variant drivers we cannot require that the source and target devices exist at the same bus address. Ideally the pre-copy data from the source device to the target will include relevant configuration information to validate that the source and target are compatible, regardless of the uuid.
Quoted 8 lines
Maybe the problem here is stating that having the same UUID is a requirement in the patch series description. I agree this is not a kernel requirement; however, a live guest migration will not succeed unless the destination host has a mediated device with a UUID that matches that of the UUID of the mediated device used to pass through s390 crypto devices to the source guest for the reason I stated above. Would it help if I removed item #1 as a requirement?
Yes, AIUI this is an artifact of your tooling. The destination QEMU can be started with any arbitrary UUID and should accept an incoming migration, routing the migration to the correct device by the virtual path, independent of the UUID.
Quoted 28 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
2. The AP configuration assigned via the VFIO AP mediated device on both guests must be compatible. As such, each AP configuration must meet the following requirements:
* Both guests must have the same number of APQNs
* Each APQN assigned to the source guest must also be assigned to the destination guest
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
* Each APQN assigned to both guests must reference an AP queue with the same hardware capabilities
Why isn't this sufficient vs also requiring the same UUID?
I explained why in my previous response.
See above, userspace tooling requirements don't imply kernel requirement.
Got it.
Quoted 9 lines
A userspace interface can still exist, I just don't find this path through the driver to the VM acceptable with this justification. Mechanisms already exist for the device to refuse a state transition or generate an error for a migration already in progress. IMHO, it would be acceptable for the device to block a key change if the migration is already in progress. If the key change cannot be represented in the migration data stream, then it's up to the orchestration of the migration to make sure they stay synchronized, but I don't see that the vfio uAPI needs to be involved. Thanks,
Quoted 6 lines
As I stated in the cover letter description, all access to the AP devices is handled by the s390 Interpretive Execution facility, so without a complete redesign of the
vfio_apdevice driver, there is no way to know that a master key change is being requested. I do get your point, however, so I will figure out how best to handle this in userspace. Thanks for you review.
The cover letter has one sentence describing the Interpretive Execution facility handling access to the device. There are a lot of subtleties there that are not obvious to laypersons.
It seems like the architecture really doesn't have the ability to monitor state changes of the device, and therefore represent those changes in the migration data stream, so you want to fallback to a cooperative scheme with a side-band channel to interrupt or block migration when an asynchronous configuration change happens.
I just don't see that such cooperation needs a new uAPI through the device; either the device itself could perform the same within the existing uAPI or the agent changing the configuration could coordinate with the migration orchestration. Thanks,
Alex
Anthony Krowiak
maintainerQuoted 72 lines · Alex Williamson
Quoted 61 lines · On Wed, 1 Apr 2026 09:38:59 -0400 Anthony Krowiak <[email protected]>…
Quoted 52 lines · Alex Williamson
Quoted 24 lines · On Tue, 31 Mar 2026 08:07:06 -0400 Anthony Krowiak <[email protected]>…
Quoted 23 lines · Alex Williamson
Quoted 20 lines · On Wed, 25 Mar 2026 17:00:47 -0400 Anthony Krowiak <[email protected]>…
This patch series implements live guest migration of a guest to which AP devices have been passed through. To better comprehend this design, one has to understand that VFIO AP mediated device is not used to provide userspace with direct access to a device as is the case with other devices that use the VFIO framework to pass them through to a guest. The sole purpose of the VFIO AP mediated device is to manage an AP configuration for a guest. An AP configuration is comprised of the AP adapter IDs (APID), AP queue indexes (APQI) and domain numbers of the control domains to which a guest will be granted access. Once the VFIO AP mediated device is attached to a guest, its AP configuration is set by the
vfio_apdevice driver. Once set, all access to the AP devices is handled by the s390 Interpretive Execution facility; in other words, thevfio_apdevice driver plays no role in providing direct access to the AP devices in the guest's AP configuration.The only role that the
vfio_apdevice driver plays in the migration process is to verify that the AP configuration for the source guest is compatible with the AP configuration of the destination guest. Incompatibility will result in a live guest migration failure. In order to be compatible, the following requirements must be met:Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
1. The destination guest will be started with the same QEMU command line as the source guest, so the mediated device supplying the AP configuration on both guests must have the same name (UUID).
AFAIK, same UUID is not a requirement for out-of-tree mdev drivers supporting migration. You're really concerned more with the configuration/composition of the mdev device, so requiring the same UUID seems a bit arbitrary.
Combining replies:
Quoted 7 lines · On Tue, 31 Mar 2026 07:17:08 -0400 Anthony Krowiak <[email protected]>…
As stated above, the destination guest will be started with the same QEMU command line as the source guest. Within that command line will be a '-device' parameter like the following:
-device '{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}
Quoted 7 lines
Note that sysfsdev is the path to the mdev named 62177883-f1bb-47f0-914d-32a22e3a8804;
therefore, the mdev with that name must exist on the destination guest or the migration will fail with the following error:error: device not found: mediated device
'62177883-f1bb-47f0-914d-32a22e3a8804' not foundQuoted 8 lines
Then this is a requirement of your tooling, not a kernel requirement, not something the kernel should care about. QEMU matches devices by their virtual bus path, not the sysfsdev or host attributes. In the case of VF migration with vfio-pci variant drivers we cannot require that the source and target devices exist at the same bus address. Ideally the pre-copy data from the source device to the target will include relevant configuration information to validate that the source and target are compatible, regardless of the uuid.
Quoted 8 lines
Maybe the problem here is stating that having the same UUID is a requirement in the patch series description. I agree this is not a kernel requirement; however, a live guest migration will not succeed unless the destination host has a mediated device with a UUID that matches that of the UUID of the mediated device used to pass through s390 crypto devices to the source guest for the reason I stated above. Would it help if I removed item #1 as a requirement?
Quoted 4 lines
Yes, AIUI this is an artifact of your tooling. The destination QEMU can be started with any arbitrary UUID and should accept an incoming migration, routing the migration to the correct device by the virtual path, independent of the UUID.
If by tooling you are referring to libvirt and/or quemu, then I agree; this is an artifact of tooling. According to
https://www.linux-kvm.org/page/Migration#Requirements,
"the guest on the destination must be started the same way it was started on the source". s390 guests are run under qemu, so the qemu command line used to start the source guest will also be used to start the destination guest. To pass through
AP devices to the guest, the mediated device that provides the guest's AP configuration - i.e., the crypto devices to which the guest is granted access - must be identified on the qemu command line. As stated previously, the following qemu command line parameter is included for this purpose:
-device '{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}
When the qemu command is used to start the destination guest, the same qemu command line will be used to start it. As such, qemu will try to open an fd for that device. If the mdev with UUID does not exist, the migration will fail. In other words, I don't think the guest can be started with any arbitrary UUID. Am I wrong about this?
In any case, it is probably not relevant to mention this for this patch series since it is not a KVM/kernel requirement, so I will remove it from the cover-letter description.
Quoted 43 lines
Quoted 27 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
2. The AP configuration assigned via the VFIO AP mediated device on both guests must be compatible. As such, each AP configuration must meet the following requirements:
* Both guests must have the same number of APQNs
* Each APQN assigned to the source guest must also be assigned to the destination guest
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
* Each APQN assigned to both guests must reference an AP queue with the same hardware capabilities
Why isn't this sufficient vs also requiring the same UUID?
I explained why in my previous response.
See above, userspace tooling requirements don't imply kernel requirement.
Got it.
Quoted 9 lines
A userspace interface can still exist, I just don't find this path through the driver to the VM acceptable with this justification. Mechanisms already exist for the device to refuse a state transition or generate an error for a migration already in progress. IMHO, it would be acceptable for the device to block a key change if the migration is already in progress. If the key change cannot be represented in the migration data stream, then it's up to the orchestration of the migration to make sure they stay synchronized, but I don't see that the vfio uAPI needs to be involved. Thanks,
Quoted 6 lines
As I stated in the cover letter description, all access to the AP devices is handled by the s390 Interpretive Execution facility, so without a complete redesign of the
vfio_apdevice driver, there is no way to know that a master key change is being requested. I do get your point, however, so I will figure out how best to handle this in userspace. Thanks for you review.The cover letter has one sentence describing the Interpretive Execution facility handling access to the device. There are a lot of subtleties there that are not obvious to laypersons.
It seems like the architecture really doesn't have the ability to monitor state changes of the device, and therefore represent those changes in the migration data stream, so you want to fallback to a cooperative scheme with a side-band channel to interrupt or block migration when an asynchronous configuration change happens.
Quoted 4 lines
I just don't see that such cooperation needs a new uAPI through the device; either the device itself could perform the same within the existing uAPI or the agent changing the configuration could coordinate with the migration orchestration. Thanks,
I understand your view on this and have already agreed to remove the new uAPI.
Quoted 2 lines
Alex