s390/vfio-ap: Add live guest migration support
Subsystems (5)
Anthony Krowiak
maintainerv4 cover s390/vfio-ap: Add live guest migration support
This patch series implements live guest migration support for KVM guests with s390 AP (Adjunct Processor) devices passed through via the VFIO mediated device framework.
Background
~~~~~~~~~~
The vfio-ap device driver differs from typical VFIO device drivers in that it does not virtualize a physical device. Instead, it manages AP configuration metadata identifying the AP adapters, domains, and control domains to which a guest will be granted access. These AP resources are configured by assigning them to a vfio-ap mediated device via its sysfs assignment interfaces. When the fd for the VFIO device is opened by userspace, the vfio_ap device driver sets the guest's AP configuration from the metadata stored with the mediated device. As such, the AP devices are not accessed directly through the vfio_ap driver, so the driver has no internal AP device state to migrate. What it does migrate is the AP configuration metadata of the source guest.
Implementation Approach
~~~~~~~~~~~~~~~~~~~~~~~
This series implements the VFIO migration protocol using the STOP_COPY migration flow. The key aspects are:
1. On transition of the migration state from STOP to STOP_COPY
- The vfio_ap device driver creates a filestream for userspace to use to read the guest's AP configuration from the mdev
2. During the STOP_COPY phase
- Userspace uses the filestream created in #1 to read the source guest's AP configuration
- The vfio_ap device driver copies the source guest's AP configuration information to userspace
3. On transition of the migration state from STOP to RESUMING
- The vfio_ap device driver creates a filestream for userspace to use to write the source guest's AP configuration information so it can be restored to the mdev on the destination host.
4. During the RESUMING phase
- Userspace uses the filestream created in #3 to send the source guest's AP configuration information to the vfio_ap device driver on the destination host.
- The vfio_ap device driver first verifies the source guest's AP configuration is compatible with the destination host's.
- The driver restores AP configuration to the mdev on the destination host which automatically hot plugs the AP resources identified therein.
5. Documentation
- Add live guest migration chapter to vfio-ap.rst
Compatibility Validation
~~~~~~~~~~~~~~~~~~~~~~~~
The series includes comprehensive validation to ensure source and destination AP configurations are compatible. For each queue, the following characteristics must match:
- AP type (target must be same or newer than source)
- Installed facilities (APSC, APQKM, AP4KC, SLCF)
- Operating mode (CCA, Accelerator, XCP)
- APXA facility setting
- Classification (native vs stateless functions)
- Queue usability (binding/associated state)
When incompatibilities are detected, migration fails with detailed error messages identifying the specific queue and characteristic that caused the failure.
Configuration Management
~~~~~~~~~~~~~~~~~~~~~~~~
This implementation does not prevent configuration changes during migration. Configuration stability is an orchestration-layer responsibility, consistent with other VFIO device types. The driver's role is to validate configurations and provide clear diagnostics when incompatibilities are detected, enabling orchestration tools to implement appropriate policies.
QEMU patches exploiting this series:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
https://lore.kernel.org/qemu-devel/[email protected]/
Change log v3 => v4:
~~~~~~~~~~~~~~~~~~~
Patch 2:
* Fixed kernel-doc fields that didn't match structures
* Removed config_sz field from vfio_ap_config structure; it was a dup of the config_sz in the vfio_ap_migration_data structure.
* Added a vfio_ap_mig_file structure to use to store AP configuration data between calls to the read/write callbacks invoked from userspace during the STOP_COPY (read) and RESUMING (write) phases of migration.
* Added adm field for control domains to the vfio_ap_config structure
Patch 3:
* Free storage for mig_data->resuming_mig_state.ap_config in the vfio_ap_release_migration_data function to fix potential memory leak.
* Release matrix_dev->mdevs_lock in error path before vfio_put_device in the vfio_ap_mdev_probe function to avoid deadlock when the VFIO device is released.
Patch 5:
* Replaced dev_err with dev_err_once for the migration not allowed for SE guests since SE guest status is fixed at boot time and there is no value in printing the message more than once. That circumvents a potential a kernel log flood if the ioctl is called in a tight loop.
* Check for VFIO_DEVICE_STATE_ERROR for while loop in the vfio_ap_set_state function
* Restored vfio_ap_release_mig_files function removed via this patch and again calling it from the vfio_ap_reset_migration_state function.
Patch 6:
* Refactored vfio_ap_release_mig_file function to take the matrix_dev->mdevs_lock for the duration of the function
Patch 7:
* Refactored the vfio_ap_stop_copy_read callback to use the new vfio_ap_mig_file structure (see Patch 2 above) to store the source guest's AP configuration between calls.
* Now migrating control domains
Patch 9:
* Removed filtering_allowed parameter and make it encumbent upon the caller to validate the queues are available and bound to the vfio_ap device driver and the control domains are in the host's AP config
Patch 10:
* Move setting the mig_data->resuming_mig_state.ap_config and mig_data->resuming_mig_state.config_sz values outside of check for completion of the write callback.
* Added cur_cfg_sz parameter to reallocat_ap_config function due to removal of config_sz field in the vfio_ap_migration_data structure (see Patch 2 changes above)
* Refactored the vfio_ap_resuming_write callback to use the new vfio_ap_mig_file structure (see Patch 2 above) to store the source guest's AP configuration between calls.
* Now migrating control domains
* Added check for source and destination BS bits set to 0 if the hardware info for both matches
* Added initialization for guest_matrix before call to setup_ap_matrix_from_ap_config
Patch 15:
* Added debug log messages for unavailable control domains on destination host
* Fixed a few misspellings and format errors.
Anthony Krowiak (15):
s390/vfio-ap: Provide function to get the number of queues assigned to
mdev
s390/vfio-ap: Data structures for facilitating vfio device migration s390/vfio-ap: Functions to initialize/release vfio device migration
data
s390/vfio-ap: Reset migration state in VFIO_DEVICE_RESET ioctl handler
s390-vfio-ap: Callback to get/set vfio device mig state during guest
migration
s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY s390/vfio-ap: File ops called to save the vfio device migration state s390/vfio-ap: Transition device migration state from STOP to RESUMING s390/vfio-ap: Add method to set a new guest AP configuration s390/vfio-ap: File ops called to resume the vfio device migration s390/vfio-ap: Transition device migration state to STOP s390/vfio-ap: Transition device migration state from STOP to RUNNING
and vice versa
s390/vfio-ap: Callback to get the size of data to be migrated during
guest migration
s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst
Documentation/arch/s390/vfio-ap.rst | 514 +++++++-- drivers/s390/crypto/Makefile | 2 +- drivers/s390/crypto/vfio_ap_drv.c | 4 +- drivers/s390/crypto/vfio_ap_migration.c | 1373 +++++++++++++++++++++++ drivers/s390/crypto/vfio_ap_ops.c | 462 ++++++-- drivers/s390/crypto/vfio_ap_private.h | 72 ++ 6 files changed, 2225 insertions(+), 202 deletions(-) create mode 100644 drivers/s390/crypto/vfio_ap_migration.c
--
2.53.0
Anthony Krowiak
maintainerPATCH v4 1/15 s390/vfio-ap: Provide function to get the number of queues assigned to mdev
Provides a function that returns the number of queues that are or will be passed through to a guest when the mdev is attached to a guest.
--- drivers/s390/crypto/vfio_ap_ops.c | 14 ++++++++++++++ drivers/s390/crypto/vfio_ap_private.h | 2 ++ 2 files changed, 16 insertions(+)
drivers/s390/crypto/vfio_ap_ops.c +14 -0
@@ -2098,6 +2098,20 @@ static void vfio_ap_mdev_request(struct vfio_device *vdev, unsigned int count)release_update_locks_for_mdev (matrix_mdev); } +int vfio_ap_mdev_get_num_queues (struct ap_matrix *ap_matrix) +{ +unsigned long apid, apqi; +int num_queues =0 ; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + +for_each_set_bit_inv (apid, ap_matrix->apm,AP_DEVICES ) +for_each_set_bit_inv (apqi, ap_matrix->aqm,AP_DOMAINS ) + num_queues++; + +return num_queues; +} +static int vfio_ap_mdev_get_device_info (unsigned long arg) {unsigned long minsz;
drivers/s390/crypto/vfio_ap_private.h +2 -0
@@ -152,6 +152,8 @@ struct vfio_ap_queue {struct work_struct reset_work; }; +int vfio_ap_mdev_get_num_queues (struct ap_matrix *ap_matrix); +int vfio_ap_mdev_register (void );void vfio_ap_mdev_unregister (void );
Signature · 1 line
2.53.0
Anthony Krowiak
maintainerPATCH v4 2/15 s390/vfio-ap: Data structures for facilitating vfio device migration
In reply to Anthony KrowiakCreates the data structures used to facilitate state transitions during vfio device migration.
--- drivers/s390/crypto/Makefile | 2 +- drivers/s390/crypto/vfio_ap_migration.c | 56 +++++++++++++++++++++++++ drivers/s390/crypto/vfio_ap_private.h | 5 +++ 3 files changed, 62 insertions(+), 1 deletion(-) create mode 100644 drivers/s390/crypto/vfio_ap_migration.c
drivers/s390/crypto/Makefile +1 -1
@@ -34,5 +34,5 @@ pkey-uv-objs := pkey_uv.o obj-$(CONFIG_PKEY_UV) += pkey-uv.o # adjunct processor matrix -vfio_ap-objs := vfio_ap_drv.o vfio_ap_ops.o +vfio_ap-objs := vfio_ap_drv.o vfio_ap_ops.o vfio_ap_migration.o obj-$(CONFIG_VFIO_AP) += vfio_ap.o
drivers/s390/crypto/vfio_ap_migration.c new file +56 -0
56 lines not on this page · raw message
drivers/s390/crypto/vfio_ap_private.h +5 -0
@@ -91,6 +91,9 @@ struct ap_queue_table {DECLARE_HASHTABLE (queues,8 ); }; +/* Forward declaration for migration data structure */ +struct vfio_ap_migration_data ; +/** * struct ap_matrix_mdev - Contains the data associated with a matrix mediated * device.
@@ -110,6 +113,7 @@ struct ap_queue_table {* @aqm_add: bitmap of APQIs added to the host's AP configuration * @adm_add: bitmap of control domain numbers added to the host's AP * configuration +* @mig_data: vfio device migration data */ struct ap_matrix_mdev {struct vfio_device vdev;
@@ -125,6 +129,7 @@ struct ap_matrix_mdev {DECLARE_BITMAP (apm_add,AP_DEVICES );DECLARE_BITMAP (aqm_add,AP_DOMAINS );DECLARE_BITMAP (adm_add,AP_DOMAINS ); +struct vfio_ap_migration_data *mig_data; };/**
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v4 02/15] s390/vfio-ap: Data structures for facilitating vfio device migration
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Low] Kernel-doc comments for the new migration data structures contain mismatched and missing fields.…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Low] Kernel-doc comments for the new migration data structures contain mismatched and missing fields.
- [Low] Use of the architecture-dependent type size_t in a struct intended for a migration data payload. --
--- Patch [2]: [PATCH v4 02/15] s390/vfio-ap: Data structures for facilitating vfio device migration --- commit c51eed4d1a56608445d8a10292fff2170e3cf604
Author: Anthony Krowiak <[email protected]>
s390/vfio-ap: Data structures for facilitating vfio device migration
Creates the data structures used to facilitate state transitions during vfio device migration.
Quoted 5 lines
drivers/s390/crypto/vfio_ap_migration.c new file +0
-0
[ ... ]
Quoted 9 lines
drivers/s390/crypto/vfio_ap_migration.c new file +9
-0+/** +* vfio_ap_migration_data - the data needed to migrate a guest with pass-through +* access to AP devices +* +* @mig_state: the current migration state +* @resuming_migf: the object used to resume the target guest +* @saving_migf: the object used to save the state of the source guest +*/ +struct vfio_ap_migration_data {
drivers/s390/crypto/vfio_ap_migration.c line 27
Quoted 10 lines
drivers/s390/crypto/vfio_ap_migration.c new file +10
-0+enum vfio_device_mig_state mig_state; + +struct { +struct file *filp; +struct vfio_ap_config *ap_config; +size_t config_sz; + } resuming_mig_state; + +struct file *stop_copy_mig_file; +};
[Severity: Low]
This isn't a bug, but the kernel-doc block documents fields (@resuming_migf and @saving_migf) that do not match the actual struct members (resuming_mig_state and stop_copy_mig_file). Should the documentation be updated to reflect the actual member names?
[ ... ]
drivers/s390/crypto/vfio_ap_migration.c line 54
Quoted 12 lines
drivers/s390/crypto/vfio_ap_migration.c new file +12
-0+/** +* vfio_ap_config - the guest's AP configuration +* +* @num_queues: the number of queues passed through to the guest +* @reserved: padding to ensure proper alignment of qinfo array +* @qinfo: an array of vfio_ap_queue_info objects, each specifying the +* queue information for a queue passed through to the guest +*/ +struct vfio_ap_config { +u32 num_queues; +u8 reserved[4 ]; +size_t config_sz;
[Severity: Low]
This isn't a bug, but the kernel-doc block above is missing documentation for the config_sz member.
Also, since this struct is intended for a migration data payload, does the use of size_t present an issue for serialization? While s390x is strictly 64-bit, using an architecture-dependent type in a migration stream is generally an anti-pattern. Would a fixed-width integer type like u64 be more appropriate here to ensure consistent structure size?
Quoted 2 lines
drivers/s390/crypto/vfio_ap_migration.c new file +2
-0+ struct vfio_ap_queue_info qinfo[] __counted_by(num_queues); +};
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Anthony Krowiak
maintainerPATCH v4 3/15 s390/vfio-ap: Functions to initialize/release vfio device migration data
In reply to Anthony KrowiakProvides the functions that initialize and release the data structures used during live guest migration:
* vfio_ap_init_migration_capabilities
Sets the migration flags and vfio_migration_ops structure into the vfio_device object when the mdev is probed.
* vfio_ap_init_migration_data
Allocates and initializes the object used to maintain the state of the VFIO migration. It is called when the VFIO device is opened.
* vfio_ap_release_migration_data
Frees the memory of the object used to maintain the VFIO migration state. It is called when the VFIO device release callback is invoked and when the VFIO device is closed.
--- drivers/s390/crypto/vfio_ap_migration.c | 75 +++++++++++++++++++++++++ drivers/s390/crypto/vfio_ap_ops.c | 60 +++++++++++++++++--- drivers/s390/crypto/vfio_ap_private.h | 4 ++ 3 files changed, 131 insertions(+), 8 deletions(-)
drivers/s390/crypto/vfio_ap_migration.c +75 -0
@@ -4,6 +4,7 @@* * Copyright IBM Corp. 2025 */ +#include <linux/file.h> #include "vfio_ap_private.h" /**
@@ -54,3 +55,77 @@ struct vfio_ap_config {size_t config_sz; struct vfio_ap_queue_info qinfo[] __counted_by(num_queues); }; + +static struct file *vfio_ap_set_state (struct vfio_device *vdev, +enum vfio_device_mig_state new_state) +{ +return NULL ; +} + +static int vfio_ap_get_state (struct vfio_device *vdev, +enum vfio_device_mig_state *current_state) +{ +return -EOPNOTSUPP ; +} + +static int vfio_ap_get_data_size (struct vfio_device *vdev, +unsigned long *stop_copy_length) +{ +return -EOPNOTSUPP ; +} + +static const struct vfio_migration_ops vfio_ap_migration_ops = { + .migration_set_state = vfio_ap_set_state, + .migration_get_state = vfio_ap_get_state, + .migration_get_data_size = vfio_ap_get_data_size, +}; + +/** +* vfio_ap_init_migrations_capabilities - initialize migration capabilities +* +* @matrix_mdev: pointer to object containing the mdev state +*/ +void vfio_ap_init_migration_capabilities (struct ap_matrix_mdev *matrix_mdev) +{ + matrix_mdev->vdev.migration_flags =VFIO_MIGRATION_STOP_COPY ; + matrix_mdev->vdev.mig_ops = &vfio_ap_migration_ops; +} + +/** +* vfio_ap_init_migration_data - initialize migration data and functions +* +* @matrix_mdev: pointer to object containing the mdev state +* +* Return: zero if initialization is successful; otherwise, returns a error. +*/ +int vfio_ap_init_migration_data (struct ap_matrix_mdev *matrix_mdev) +{ +struct vfio_ap_migration_data *mig_data; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + + mig_data = kzalloc_obj(struct vfio_ap_migration_data, GFP_KERNEL); +if (!mig_data) +return -ENOMEM ; + + mig_data->mig_state =VFIO_DEVICE_STATE_RUNNING ; + matrix_mdev->mig_data = mig_data; + +return 0 ; +} + +/** +* vfio_ap_release_migration_data: reclaim private migration data +* +* @vdev: pointer to the mdev +*/ +void vfio_ap_release_migration_data (struct ap_matrix_mdev *matrix_mdev) +{ +lockdep_assert_held (&matrix_dev->mdevs_lock); + +if (!matrix_mdev->mig_data) +return ; + +kfree (matrix_mdev->mig_data); + matrix_mdev->mig_data =NULL ; +}
drivers/s390/crypto/vfio_ap_ops.c +52 -8
114 lines not on this page · raw message
drivers/s390/crypto/vfio_ap_private.h +4 -0
@@ -172,4 +172,8 @@ void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info,void vfio_ap_on_scan_complete (struct ap_config_info *new_config_info,struct ap_config_info *old_config_info); +void vfio_ap_init_migration_capabilities (struct ap_matrix_mdev *matrix_mdev); +int vfio_ap_init_migration_data (struct ap_matrix_mdev *matrix_mdev); +void vfio_ap_release_migration_data (struct ap_matrix_mdev *matrix_mdev); +#endif /* _VFIO_AP_PRIVATE_H_ */
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v4 03/15] s390/vfio-ap: Functions to initialize/release vfio device migration data
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
New issues:
- [High] Self-deadlock in the error path of vfio_ap_mdev_probe() due to holding matrix_dev->mdevs_lock while calling vfio_put_device().
- [High] AB-BA lock inversion between matrix_dev->mdevs_lock and the VFIO core's dev_set->lock.
- [Medium] Advertising VFIO_MIGRATION_STOP_COPY with stub callbacks that return -EOPNOTSUPP breaks userspace expectations.
Pre-existing issues:
- [Critical] If vfio_ap_mdev_set_kvm returns -EPERM, it leaves a dangling pointer in kvm->arch.crypto.pqap_hook, leading to a Use-After-Free. --
--- Patch [3]: [PATCH v4 03/15] s390/vfio-ap: Functions to initialize/release vfio device migration data --- commit fd0c4839e5d26823861a00cf7c060331ae1fd632
Author: Anthony Krowiak <[email protected]>
s390/vfio-ap: Functions to initialize/release vfio device migration data
Provides the functions that initialize and release the data structures used during live guest migration.
Quoted 4 lines
drivers/s390/crypto/vfio_ap_migration.c +0
-0
[ ... ]
Quoted 29 lines
drivers/s390/crypto/vfio_ap_migration.c +25
-0@@ -54,3 +55,77 @@ struct vfio_ap_config {size_t config_sz; struct vfio_ap_queue_info qinfo[] __counted_by(num_queues); }; + +static struct file *vfio_ap_set_state (struct vfio_device *vdev, +enum vfio_device_mig_state new_state) +{ +return NULL ; +} + +static int vfio_ap_get_state (struct vfio_device *vdev, +enum vfio_device_mig_state *current_state) +{ +return -EOPNOTSUPP ; +} + +static int vfio_ap_get_data_size (struct vfio_device *vdev, +unsigned long *stop_copy_length) +{ +return -EOPNOTSUPP ; +} + +static const struct vfio_migration_ops vfio_ap_migration_ops = { + .migration_set_state = vfio_ap_set_state, + .migration_get_state = vfio_ap_get_state, + .migration_get_data_size = vfio_ap_get_data_size, +}; +
drivers/s390/crypto/vfio_ap_migration.c struct vfio_ap_config { · line 92
Quoted 10 lines
drivers/s390/crypto/vfio_ap_migration.c +10
-0+/** +* vfio_ap_init_migrations_capabilities - initialize migration capabilities +* +* @matrix_mdev: pointer to object containing the mdev state +*/ +void vfio_ap_init_migration_capabilities (struct ap_matrix_mdev *matrix_mdev) +{ + matrix_mdev->vdev.migration_flags =VFIO_MIGRATION_STOP_COPY ; + matrix_mdev->vdev.mig_ops = &vfio_ap_migration_ops; +}
[Severity: Medium]
Does advertising VFIO_MIGRATION_STOP_COPY here break userspace expectations when the callbacks like vfio_ap_get_state() unconditionally return -EOPNOTSUPP?
If userspace queries the device state via VFIO_DEVICE_FEATURE_MIG_DEVICE_STATE, wouldn't the -EOPNOTSUPP response cause the setup or migration to unexpectedly abort?
[ ... ]
Quoted 4 lines
drivers/s390/crypto/vfio_ap_ops.c +0
-0
[ ... ]
Quoted 6 lines
drivers/s390/crypto/vfio_ap_ops.c +2
-0@@ -797,19 +809,30 @@ static int vfio_ap_mdev_probe(struct mdev_device *mdev)if (IS_ERR (matrix_mdev))return PTR_ERR (matrix_mdev); +mutex_lock (&matrix_dev->mdevs_lock); +
drivers/s390/crypto/vfio_ap_ops.c static int vfio_ap_mdev_probe(struct mdev_device *mdev) · line 823
Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +9
-0+/* +* Migration capabilities must be initialized before calling +* vfio_register_emulated_iommu_dev; otherwise, the VFIO core +* will see mig_ops as NULL during the registration. This could +* prevent the VFIO core from properly setting up migration +* infrastructure like debugfs entries. +*/ +vfio_ap_init_migration_capabilities (matrix_mdev); + ret =vfio_register_emulated_iommu_dev (&matrix_mdev->vdev);
[Severity: High]
Does acquiring matrix_dev->mdevs_lock before calling vfio_register_emulated_iommu_dev() create a lock inversion?
Since vfio_register_emulated_iommu_dev() acquires dev_set->lock, this sets the lock order to mdevs_lock -> dev_set->lock.
Later, when userspace opens the device, the core holds dev_set->lock before invoking the open_device callback (vfio_ap_mdev_open_device). The callback then attempts to acquire mdevs_lock, which reverses the order to dev_set->lock -> mdevs_lock, potentially leading to a deadlock.
drivers/s390/crypto/vfio_ap_ops.c static int vfio_ap_mdev_probe(struct mdev_device *mdev) · line 834
Quoted 12 lines
drivers/s390/crypto/vfio_ap_ops.c +0
-1if (ret) goto err_put_vdev; matrix_mdev->req_trigger =NULL ; matrix_mdev->cfg_chg_trigger =NULL ;dev_set_drvdata (&mdev->dev, matrix_mdev); -mutex_lock (&matrix_dev->mdevs_lock);list_add (&matrix_mdev->node, &matrix_dev->mdev_list);mutex_unlock (&matrix_dev->mdevs_lock);return 0 ; err_put_vdev:vfio_put_device (&matrix_mdev->vdev);
[Severity: High]
Is there a self-deadlock risk here in the error path?
If vfio_register_emulated_iommu_dev() fails, we jump to err_put_vdev while matrix_dev->mdevs_lock is still held. Calling vfio_put_device() here will synchronously trigger the release callback (vfio_ap_mdev_release_dev), which attempts to acquire the same non-recursive mdevs_lock again.
Quoted 16 lines
drivers/s390/crypto/vfio_ap_ops.c +3
-2+mutex_unlock (&matrix_dev->mdevs_lock);return ret; }@@ -2052,19 +2075,39 @@ static int vfio_ap_mdev_reset_qlist(struct list_head *qlist)static int vfio_ap_mdev_open_device (struct vfio_device *vdev) { -struct ap_matrix_mdev *matrix_mdev = - container_of(vdev, struct ap_matrix_mdev, vdev); +struct ap_matrix_mdev *matrix_mdev; +int ret;if (!vdev->kvm)return -EINVAL ;
drivers/s390/crypto/vfio_ap_ops.c static int vfio_ap_mdev_reset_qlist(struct list_head *qlist) · line 2092
Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +9
-1-return vfio_ap_mdev_set_kvm (matrix_mdev, vdev->kvm); +mutex_lock (&matrix_dev->mdevs_lock); + matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev); + ret =vfio_ap_init_migration_data (matrix_mdev); +mutex_unlock (&matrix_dev->mdevs_lock); + +if (ret) +return ret; + + ret =vfio_ap_mdev_set_kvm (matrix_mdev, vdev->kvm);
[Severity: Critical]
This isn't a bug introduced by this patch, but does a failure in vfio_ap_mdev_set_kvm() leave a dangling pointer that leads to a use-after-free?
In vfio_ap_mdev_set_kvm(), kvm->arch.crypto.pqap_hook is immediately overwritten with &matrix_mdev->pqap_hook:
down_write(&kvm->arch.crypto.pqap_hook_rwsem);
kvm->arch.crypto.pqap_hook = &matrix_mdev->pqap_hook;
up_write(&kvm->arch.crypto.pqap_hook_rwsem);If it then detects that another mdev is already assigned to this KVM, it returns -EPERM but skips resetting the hook. Since matrix_mdev->kvm remains NULL, vfio_ap_mdev_unset_kvm() won't clean this up on teardown.
If the second mdev is freed, wouldn't the active KVM guest trigger a use-after-free if it executes a PQAP instruction using the stale hook?
Quoted 9 lines
drivers/s390/crypto/vfio_ap_ops.c +8
-0+if (ret) { +/* Clean up migration data on failure */ +mutex_lock (&matrix_dev->mdevs_lock); +vfio_ap_release_migration_data (matrix_mdev); +mutex_unlock (&matrix_dev->mdevs_lock); + } + +return ret; }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Anthony Krowiak
maintainerPATCH v4 4/15 s390/vfio-ap: Reset migration state in VFIO_DEVICE_RESET ioctl handler
In reply to Anthony KrowiakAccording to the VFIO migration specifications, userspace must issue a VFIO_DEVICE_RESET ioctl if a vfio device enters an error state or fails a state transition to recover the device and return its migration state to RUNNING. This patch adds a function that is called when a VFIO_DEVICE_RESET is issued that releases the migration files and frees the storage allocated for maintaining the migration state.
--- drivers/s390/crypto/vfio_ap_migration.c | 20 ++++++++++++++++++++ drivers/s390/crypto/vfio_ap_ops.c | 4 ++++ drivers/s390/crypto/vfio_ap_private.h | 1 + 3 files changed, 25 insertions(+)
drivers/s390/crypto/vfio_ap_migration.c +20 -0
@@ -129,3 +129,23 @@ void vfio_ap_release_migration_data(struct ap_matrix_mdev *matrix_mdev)kfree (matrix_mdev->mig_data); matrix_mdev->mig_data =NULL ; } + +/** +* vfio_ap_reset_migration_state - Reset the vfio-ap migration state +* +* @matrix_mdev: pointer to the object maintaining the vfio-ap device state +* +* Called during VFIO_DEVICE_RESET to clean up any active migration +* stte and reset the device to RUNNING state as required by the VFIO +* migration specification. +*/ +void vfio_ap_reset_migration_state (struct ap_matrix_mdev *matrix_mdev) +{ +lockdep_assert_held (&matrix_dev->mdevs_lock); + +if (!matrix_mdev->mig_data) +return ; + +vfio_ap_release_mig_files (matrix_mdev); + matrix_mdev->mig_data->mig_state =VFIO_DEVICE_STATE_RUNNING ; +}
drivers/s390/crypto/vfio_ap_ops.c +4 -0
@@ -2332,6 +2332,10 @@ static ssize_t vfio_ap_mdev_ioctl(struct vfio_device *vdev,break ;case VFIO_DEVICE_RESET : ret =vfio_ap_mdev_reset_queues (matrix_mdev); +if (ret ==0 ) { +/* Reset migration state per VFIO migration spec */ +vfio_ap_reset_migration_state (matrix_mdev); + }break ;case VFIO_DEVICE_GET_IRQ_INFO : ret =vfio_ap_get_irq_info (arg);
drivers/s390/crypto/vfio_ap_private.h +1 -0
@@ -175,5 +175,6 @@ void vfio_ap_on_scan_complete(struct ap_config_info *new_config_info,void vfio_ap_init_migration_capabilities (struct ap_matrix_mdev *matrix_mdev);int vfio_ap_init_migration_data (struct ap_matrix_mdev *matrix_mdev);void vfio_ap_release_migration_data (struct ap_matrix_mdev *matrix_mdev); +void vfio_ap_reset_migration_state (struct ap_matrix_mdev *matrix_mdev);#endif /* _VFIO_AP_PRIVATE_H_ */
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v4 04/15] s390/vfio-ap: Reset migration state in VFIO_DEVICE_RESET ioctl handler
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
New issues:
- [Critical] Build Failure / Undefined Function / Potential NULL Pointer Dereference in vfio_ap_reset_migration_state.
- [Low] Typo in comment: 'stte' instead of 'state'.
Pre-existing issues:
- [High] System-wide Denial of Service via unbounded wait holding the global matrix_dev->mdevs_lock. --
--- Patch [4]: [PATCH v4 04/15] s390/vfio-ap: Reset migration state in VFIO_DEVICE_RESET ioctl handler --- commit 35f281972b36c8fc12cff370107f5d4f61c3aa01
Author: Anthony Krowiak <[email protected]>
s390/vfio-ap: Reset migration state in VFIO_DEVICE_RESET ioctl handler
This patch adds a function called during VFIO_DEVICE_RESET to release migration files and free allocated storage for migration state. It resets the device migration state to RUNNING as required by the specification.
Quoted 4 lines
drivers/s390/crypto/vfio_ap_migration.c +0
-0
[ ... ]
drivers/s390/crypto/vfio_ap_migration.c void vfio_ap_release_migration_data(struct ap_matrix_mdev *matrix_mdev) · line 139
Quoted 12 lines
drivers/s390/crypto/vfio_ap_migration.c +8
-0@@ -129,3 +129,23 @@ void vfio_ap_release_migration_data(struct ap_matrix_mdev *matrix_mdev)kfree (matrix_mdev->mig_data); matrix_mdev->mig_data =NULL ; } + +/** +* vfio_ap_reset_migration_state - Reset the vfio-ap migration state +* +* @matrix_mdev: pointer to the object maintaining the vfio-ap device state +* +* Called during VFIO_DEVICE_RESET to clean up any active migration +* stte and reset the device to RUNNING state as required by the VFIO
[Severity: Low]
This isn't a bug, but there is a typo here. Did you mean "state" instead of "stte"?
drivers/s390/crypto/vfio_ap_migration.c void vfio_ap_release_migration_data(struct ap_matrix_mdev *matrix_mdev) · line 149
Quoted 10 lines
drivers/s390/crypto/vfio_ap_migration.c +10
-0+* migration specification. +*/ +void vfio_ap_reset_migration_state (struct ap_matrix_mdev *matrix_mdev) +{ +lockdep_assert_held (&matrix_dev->mdevs_lock); + +if (!matrix_mdev->mig_data) +return ; + +vfio_ap_release_mig_files (matrix_mdev);
[Severity: Critical]
Does vfio_ap_release_mig_files() exist in the kernel? I could not find a definition or declaration for it.
If the intention was to call vfio_ap_release_migration_data() to free the storage as mentioned in the commit message, wouldn't that set matrix_mdev->mig_data to NULL?
drivers/s390/crypto/vfio_ap_migration.c void vfio_ap_release_migration_data(struct ap_matrix_mdev *matrix_mdev) · line 150
Quoted 1 line
drivers/s390/crypto/vfio_ap_migration.c +1
-0+ matrix_mdev->mig_data->mig_state =VFIO_DEVICE_STATE_RUNNING ;
If matrix_mdev->mig_data is freed and set to NULL just prior to this, won't this line unconditionally dereference the newly NULL pointer and cause a crash?
drivers/s390/crypto/vfio_ap_ops.c static ssize_t vfio_ap_mdev_ioctl(struct vfio_device *vdev, · line 2334
Quoted 9 lines
drivers/s390/crypto/vfio_ap_migration.c +1
-0+}drivers/s390/crypto/vfio_ap_ops.c +0
-0@@ -2332,6 +2332,10 @@ static ssize_t vfio_ap_mdev_ioctl(struct vfio_device *vdev,break ;case VFIO_DEVICE_RESET : ret =vfio_ap_mdev_reset_queues (matrix_mdev);
[Severity: High]
This is a pre-existing issue, but can this lead to a system-wide deadlock?
In vfio_ap_mdev_ioctl(), the global matrix_dev->mdevs_lock is acquired before calling vfio_ap_mdev_reset_queues().
Inside vfio_ap_mdev_reset_queues(), flush_work() is called, which synchronously waits for apq_reset_check() to complete:
vfio_ap_mdev_reset_queues()
flush_work(&q->reset_work);
If the hardware faults, apq_reset_check() polls indefinitely in its infinite while (true) loop. Because the global mdevs_lock is held during this time, would this permanently block all other operations across all vfio-ap mediated devices?
Quoted 5 lines
drivers/s390/crypto/vfio_ap_ops.c +4
-0+if (ret ==0 ) { +/* Reset migration state per VFIO migration spec */ +vfio_ap_reset_migration_state (matrix_mdev); + }break ;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Anthony Krowiak
maintainerPATCH v4 5/15 s390-vfio-ap: Callback to get/set vfio device mig state during guest migration
In reply to Anthony KrowiakImplements two callbacks related to the current state of a vfio-ap device:
* The callback that sets a new migration state of a vfio-ap device during live migration of guests with pass-through access to AP devices. This callback is mandatory for VFIO_DEVICE_FEATURE_MIGRATION support.
The function pointer for this callback is specified via the 'migration_set_state' field of the 'vfio_migration_ops' structure which is stored with the VFIO device when the 'vfio_device' structure representing the mediated device is initialized.
* The callback that returns the current vfio device migration state during live migration of guests with pass-through access to AP devices.
The function pointer for this callback is specified via the 'migration_get_state' field of the 'vfio_migration_ops' structure which is stored with the VFIO device when the 'vfio_device' structure representing the mediated device is initialized.
--- drivers/s390/crypto/vfio_ap_migration.c | 130 +++++++++++++++++++++++- 1 file changed, 127 insertions(+), 3 deletions(-)
drivers/s390/crypto/vfio_ap_migration.c +127 -3
@@ -5,6 +5,7 @@* Copyright IBM Corp. 2025 */ #include <linux/file.h> +#include "ap_bus.h" #include "vfio_ap_private.h" /**
@@ -56,16 +57,140 @@ struct vfio_ap_config { struct vfio_ap_queue_info qinfo[] __counted_by(num_queues); }; +static struct file * +vfio_ap_transition_to_state (struct ap_matrix_mdev *matrix_mdev, +enum vfio_device_mig_state new_state) +{ +struct vfio_ap_migration_data *mig_data; +enum vfio_device_mig_state cur_state; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + mig_data = matrix_mdev->mig_data; + cur_state = mig_data->mig_state; +dev_dbg (matrix_mdev->vdev.dev,"%s: %d -> %d\n" , __func__, cur_state, + new_state); + +if (cur_state ==VFIO_DEVICE_STATE_STOP && + new_state ==VFIO_DEVICE_STATE_STOP_COPY ) { +/* TODO */ +return ERR_PTR (-EOPNOTSUPP ); + } + +if (cur_state ==VFIO_DEVICE_STATE_STOP && + new_state ==VFIO_DEVICE_STATE_RESUMING ) { +/* TODO */ +return ERR_PTR (-EOPNOTSUPP ); + } + +if ((cur_state ==VFIO_DEVICE_STATE_RESUMING && + new_state ==VFIO_DEVICE_STATE_STOP ) || + (cur_state ==VFIO_DEVICE_STATE_STOP_COPY && + new_state ==VFIO_DEVICE_STATE_STOP )) { +/* TODO */ +return ERR_PTR (-EOPNOTSUPP ); + } + +if ((cur_state ==VFIO_DEVICE_STATE_STOP && + new_state ==VFIO_DEVICE_STATE_RUNNING ) || + (cur_state ==VFIO_DEVICE_STATE_RUNNING && + new_state ==VFIO_DEVICE_STATE_STOP )) { +/* TODO */ +return ERR_PTR (-EOPNOTSUPP ); + } + +/* vfio_mig_get_next_state() does not use arcs other than the above */ +WARN_ON (true); + +return ERR_PTR (-EINVAL ); +} +static struct file *vfio_ap_set_state (struct vfio_device *vdev,enum vfio_device_mig_state new_state) { -return NULL ; +int ret; +struct file *filp =NULL ; +struct ap_matrix_mdev *matrix_mdev; +enum vfio_device_mig_state next_state; +struct vfio_ap_migration_data *mig_data; + + matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev); + +mutex_lock (&matrix_dev->mdevs_lock); +if (ap_is_se_guest ()) { +dev_err (matrix_mdev->vdev.dev, +"Migration not allowed from or to a Secure Execution guest\n" ); +mutex_unlock (&matrix_dev->mdevs_lock); +return ERR_PTR (-EPERM ); + } + + mig_data = matrix_mdev->mig_data; + +/* +* The mig_data pointer is set in the vfio_ap_init_migration_data +* function which is called when the vfio-ap device fd is opened. +* Since the implicit pre-open state is RUNNING, a request to set +* RUNNING is a no-op. Any other state transition is invalid before +* open_device. +*/ +if (!mig_data) { +mutex_unlock (&matrix_dev->mdevs_lock); +if (new_state ==VFIO_DEVICE_STATE_RUNNING ) +return NULL ; +return ERR_PTR (-ENODEV ); + } + +dev_dbg (vdev->dev,"%s -> %d\n" , __func__, new_state); + +while (mig_data->mig_state != new_state) { + ret =vfio_mig_get_next_state (vdev, mig_data->mig_state, + new_state, &next_state); +if (ret) { + filp =ERR_PTR (ret); +break ; + } + + filp =vfio_ap_transition_to_state (matrix_mdev, next_state); +if (IS_ERR (filp)) +break ; + + mig_data->mig_state = next_state; + +if (WARN_ON (filp && new_state != next_state)) { +fput (filp); + filp =ERR_PTR (-EINVAL ); +break ; + } + } + +mutex_unlock (&matrix_dev->mdevs_lock); + +return filp; }static int vfio_ap_get_state (struct vfio_device *vdev,enum vfio_device_mig_state *current_state) { -return -EOPNOTSUPP ; +struct ap_matrix_mdev *matrix_mdev; +struct vfio_ap_migration_data *mig_data; + +mutex_lock (&matrix_dev->mdevs_lock); + + matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev); + mig_data = matrix_mdev->mig_data; + +/* +* The mig_data pointer is set in the vfio_ap_init_migration_data +* function which is called when the vfio-ap device fd is opened. +* If mig_data is NULL, report RUNNING as the implicit pre-open state +* so userspace doesn't need to perform any state transition before the +* device becomes active. +*/ + *current_state = (mig_data) ? mig_data->mig_state : +VFIO_DEVICE_STATE_RUNNING ; + +mutex_unlock (&matrix_dev->mdevs_lock); + +return 0 ; } static int vfio_ap_get_data_size(struct vfio_device *vdev,
@@ -146,6 +271,5 @@ void vfio_ap_reset_migration_state(struct ap_matrix_mdev *matrix_mdev)if (!matrix_mdev->mig_data)return ; -vfio_ap_release_mig_files (matrix_mdev); matrix_mdev->mig_data->mig_state =VFIO_DEVICE_STATE_RUNNING ; }
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v4 05/15] s390-vfio-ap: Callback to get/set vfio device mig state during guest migration
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] The migration state machine is completely stubbed out, resulting in unconditional migration…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] The migration state machine is completely stubbed out, resulting in unconditional migration failures despite advertising migration capabilities.
- [High] A self-deadlock occurs in the error path of vfio_ap_mdev_probe() when vfio_register_emulated_iommu_dev() fails.
- [Medium] Migration capabilities are unconditionally advertised to userspace, but runtime transitions are blocked for Secure Execution (SE) guests. --
--- Patch [5]: [PATCH v4 05/15] s390-vfio-ap: Callback to get/set vfio device mig state during guest migration --- commit c722c4dee661c005442dd72e3de8888c3305b360
Author: Anthony Krowiak <[email protected]>
s390-vfio-ap: Callback to get/set vfio device mig state during guest migration
This commit implements two callbacks, migration_set_state and migration_get_state, for the vfio_migration_ops structure. These callbacks manage the migration state of a vfio-ap device during live migration of guests with pass-through access.
Quoted 4 lines
drivers/s390/crypto/vfio_ap_migration.c +0
-0
[ ... ]
Quoted 33 lines
drivers/s390/crypto/vfio_ap_migration.c +29
-0@@ -56,16 +57,140 @@ struct vfio_ap_config { struct vfio_ap_queue_info qinfo[] __counted_by(num_queues); }; +static struct file * +vfio_ap_transition_to_state (struct ap_matrix_mdev *matrix_mdev, +enum vfio_device_mig_state new_state) +{ +struct vfio_ap_migration_data *mig_data; +enum vfio_device_mig_state cur_state; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + mig_data = matrix_mdev->mig_data; + cur_state = mig_data->mig_state; +dev_dbg (matrix_mdev->vdev.dev,"%s: %d -> %d\n" , __func__, cur_state, + new_state); + +if (cur_state ==VFIO_DEVICE_STATE_STOP && + new_state ==VFIO_DEVICE_STATE_STOP_COPY ) { +/* TODO */ +return ERR_PTR (-EOPNOTSUPP ); + } + +if (cur_state ==VFIO_DEVICE_STATE_STOP && + new_state ==VFIO_DEVICE_STATE_RESUMING ) { +/* TODO */ +return ERR_PTR (-EOPNOTSUPP ); + } + +if ((cur_state ==VFIO_DEVICE_STATE_RESUMING && + new_state ==VFIO_DEVICE_STATE_STOP ) || + (cur_state ==VFIO_DEVICE_STATE_STOP_COPY && + new_state ==VFIO_DEVICE_STATE_STOP )) {
drivers/s390/crypto/vfio_ap_migration.c struct vfio_ap_config { · line 98
Quoted 10 lines
drivers/s390/crypto/vfio_ap_migration.c +10
-0+/* TODO */ +return ERR_PTR (-EOPNOTSUPP ); + } + +if ((cur_state ==VFIO_DEVICE_STATE_STOP && + new_state ==VFIO_DEVICE_STATE_RUNNING ) || + (cur_state ==VFIO_DEVICE_STATE_RUNNING && + new_state ==VFIO_DEVICE_STATE_STOP )) { +/* TODO */ +return ERR_PTR (-EOPNOTSUPP );
[Severity: High]
Does this code cause a regression by unconditionally returning -EOPNOTSUPP?
Since all valid migration state transitions in vfio_ap_transition_to_state() are caught and unconditionally return -EOPNOTSUPP, could this cause orchestrators (like libvirt or QEMU) to fail at runtime? Orchestrators querying the device will see migration as supported, but might unconditionally fail when they attempt to change the state.
Quoted 16 lines
drivers/s390/crypto/vfio_ap_migration.c +12
-1+ } + +/* vfio_mig_get_next_state() does not use arcs other than the above */ +WARN_ON (true); + +return ERR_PTR (-EINVAL ); +} +static struct file *vfio_ap_set_state (struct vfio_device *vdev,enum vfio_device_mig_state new_state) { -return NULL ; +int ret; +struct file *filp =NULL ; +struct ap_matrix_mdev *matrix_mdev; +enum vfio_device_mig_state next_state;
drivers/s390/crypto/vfio_ap_migration.c struct vfio_ap_config { · line 123
Quoted 10 lines
drivers/s390/crypto/vfio_ap_migration.c +10
-0+struct vfio_ap_migration_data *mig_data; + + matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev); + +mutex_lock (&matrix_dev->mdevs_lock); +if (ap_is_se_guest ()) { +dev_err (matrix_mdev->vdev.dev, +"Migration not allowed from or to a Secure Execution guest\n" ); +mutex_unlock (&matrix_dev->mdevs_lock); +return ERR_PTR (-EPERM );
[Severity: Medium]
Could this runtime check cause a regression with orchestration software?
By advertising migration capabilities unconditionally to userspace, but blocking transitions for Secure Execution (SE) guests here at runtime, orchestrators might incorrectly believe migration is supported for SE guests.
Could this be rejected earlier during capability checks rather than failing during the actual migration attempt?
drivers/s390/crypto/vfio_ap_migration.c struct vfio_ap_config { · line 126
Quoted 3 lines
drivers/s390/crypto/vfio_ap_migration.c +3
-0+ } + + mig_data = matrix_mdev->mig_data;
[ ... ]
[Severity: High]
Is there a regression involving a self-deadlock in the error path of vfio_ap_mdev_probe() when vfio_register_emulated_iommu_dev() fails?
The error path in drivers/s390/crypto/vfio_ap_ops.c:vfio_ap_mdev_probe() holds the mdevs_lock while invoking vfio_put_device():
err_put_vdev:
vfio_put_device(&matrix_mdev->vdev); mutex_unlock(&matrix_dev->mdevs_lock);
This triggers the synchronous release callback vfio_ap_mdev_release_dev() which attempts to acquire the same lock:
mutex_lock(&matrix_dev->mdevs_lock); matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev); vfio_ap_release_migration_data(matrix_mdev); mutex_unlock(&matrix_dev->mdevs_lock);
Could this cause a kernel thread freeze when creating an mdev device fails?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Anthony Krowiak
maintainerPATCH v4 6/15 s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY
In reply to Anthony KrowiakThe transition of the guest migration state from VFIO_DEVICE_STATE_STOP to VFIO_DEVICE_STATE_STOP_COPY begins the process of saving the vfio device state. The vfio device state is comprised of the guest's AP configuration which specifies the adapters, domains and control domains to which the guest will be given access.
To begin the process of saving the vfio device state, the VFIO AP device driver opens a file stream that will be used to read the source guest's AP configuration.
Note:
This commit introduces the file operations structure for migration state files with a stub implementation of the read operation. The complete implementation is provided in the subsequent commit.
This incremental approach allows for easier review of the migration infrastructure setup separate from the actual implementation logic.
--- drivers/s390/crypto/vfio_ap_migration.c | 68 ++++++++++++++++++++++++- 1 file changed, 66 insertions(+), 2 deletions(-)
drivers/s390/crypto/vfio_ap_migration.c +66 -2
@@ -4,6 +4,7 @@* * Copyright IBM Corp. 2025 */ +#include <linux/anon_inodes.h> #include <linux/file.h> #include "ap_bus.h" #include "vfio_ap_private.h"
@@ -57,6 +58,57 @@ struct vfio_ap_config { struct vfio_ap_queue_info qinfo[] __counted_by(num_queues); }; +static void +vfio_ap_release_stop_copy_file (struct vfio_ap_migration_data *mig_data) +{ +if (mig_data->stop_copy_mig_file) + mig_data->stop_copy_mig_file =NULL ; +} + +static ssize_t +vfio_ap_stop_copy_read (struct file *,char __user *,size_t ,loff_t *) +{ +/* TODO */ +return -EOPNOTSUPP ; +} + +static int vfio_ap_release_mig_file (struct inode *file_inode,struct file *filp) +{ +struct ap_matrix_mdev *matrix_mdev = filp->private_data; + +if (!matrix_mdev || !matrix_mdev->mig_data) +return -ENODEV ; + +if (filp == matrix_mdev->mig_data->stop_copy_mig_file) +vfio_ap_release_stop_copy_file (matrix_mdev->mig_data); +else +return -ENOENT ; + +return 0 ; +} + +static const struct file_operations vfio_ap_stop_copy_fops = { + .owner =THIS_MODULE , + .read = vfio_ap_stop_copy_read, + .compat_ioctl = compat_ptr_ioctl, + .release = vfio_ap_release_mig_file, +}; + +static struct file *vfio_ap_open_file_stream (struct ap_matrix_mdev *matrix_mdev, +const struct file_operations *fops, +int flags) +{ +struct file *filp; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + + filp =anon_inode_getfile ("vfio_ap_mig_file" , fops, matrix_mdev, flags); +if (!IS_ERR (filp)) +stream_open (filp->f_inode, filp); + +return filp; +} +static struct file *vfio_ap_transition_to_state (struct ap_matrix_mdev *matrix_mdev,enum vfio_device_mig_state new_state)
@@ -70,10 +122,22 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,dev_dbg (matrix_mdev->vdev.dev,"%s: %d -> %d\n" , __func__, cur_state, new_state); +/* +* Begins the process of saving the vfio device state by creating and +* returning a streaming data_fd to be used to read out the internal +* state of the vfio-ap device on the source host. +*/ if (cur_state ==VFIO_DEVICE_STATE_STOP && new_state ==VFIO_DEVICE_STATE_STOP_COPY ) { -/* TODO */ -return ERR_PTR (-EOPNOTSUPP ); +struct file *filp =vfio_ap_open_file_stream (matrix_mdev, + &vfio_ap_stop_copy_fops, +O_RDONLY ); +if (IS_ERR (filp)) +return ERR_CAST (filp); + + mig_data->stop_copy_mig_file = filp; + +return filp; } if (cur_state == VFIO_DEVICE_STATE_STOP &&
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v4 06/15] s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Critical] The migration file descriptor's lifecycle is not decoupled from the VFIO device, and its…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] The migration file descriptor's lifecycle is not decoupled from the VFIO device, and its release callback accesses device structures locklessly. This causes a Use-After-Free and TOCTOU races if the migration file outlives the VFIO device or if it is closed concurrently with device teardown. --
--- Patch [6]: [PATCH v4 06/15] s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY --- commit 2960f99fd69c88ca78cea015d8ad75148e325f63
Author: Anthony Krowiak <[email protected]>
s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY
This commit introduces the transition to the STOP_COPY migration state for vfio-ap devices. It creates a migration file descriptor using an anonymous inode to stream the guest's AP configuration, providing a stub read implementation and a release callback.
Quoted 3 lines
drivers/s390/crypto/vfio_ap_migration.c +0
-0
[ ... ]
drivers/s390/crypto/vfio_ap_migration.c struct vfio_ap_config { · line 83
Quoted 9 lines
drivers/s390/crypto/vfio_ap_migration.c +9
-0+static int vfio_ap_release_mig_file (struct inode *file_inode,struct file *filp) +{ +struct ap_matrix_mdev *matrix_mdev = filp->private_data; + +if (!matrix_mdev || !matrix_mdev->mig_data) +return -ENODEV ; + +if (filp == matrix_mdev->mig_data->stop_copy_mig_file) +vfio_ap_release_stop_copy_file (matrix_mdev->mig_data);
[Severity: Critical]
Is there a risk of a use-after-free here if the main VFIO device is closed or unbound concurrently while the migration file descriptor is still open?
Since this release callback accesses matrix_mdev and its mig_data locklessly, could matrix_mdev have already been freed during device teardown if userspace closes the device but keeps the migration file descriptor open?
Quoted 5 lines
drivers/s390/crypto/vfio_ap_migration.c +5
-0+else +return -ENOENT ; + +return 0 ; +}
[ ... ]
drivers/s390/crypto/vfio_ap_migration.c struct vfio_ap_config { · line 105
Quoted 9 lines
drivers/s390/crypto/vfio_ap_migration.c +9
-0+static struct file *vfio_ap_open_file_stream (struct ap_matrix_mdev *matrix_mdev, +const struct file_operations *fops, +int flags) +{ +struct file *filp; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + + filp =anon_inode_getfile ("vfio_ap_mig_file" , fops, matrix_mdev, flags);
[Severity: Critical]
Does this code safely couple the migration file descriptor's lifecycle to the VFIO device?
Because matrix_mdev is passed as private_data without taking a registration reference on the VFIO device or wrapping it in an independent lifecycle object, it seems the VFIO core will not wait for this migration fd to close before tearing down the device.
Quoted 5 lines
drivers/s390/crypto/vfio_ap_migration.c +5
-0+if (!IS_ERR (filp)) +stream_open (filp->f_inode, filp); + +return filp; +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Anthony Krowiak
maintainerPATCH v4 7/15 s390/vfio-ap: File ops called to save the vfio device migration state
In reply to Anthony KrowiakImplements the read callback function that was added to the file_operations structure for the file created to save the state of the vfio-ap device when the migration state transitioned from STOP to to the STOP_COPY state.
This function copies the guest's AP configuration information to userspace. The information copied is comprised of the APQN of each queue device passed through to the guest along with its hardware information. This state data will be transferred to the vfio_ap device driver on the destination host when the state is transitioned to RESUMING.
--- drivers/s390/crypto/vfio_ap_migration.c | 216 +++++++++++++++++++++++- 1 file changed, 209 insertions(+), 7 deletions(-)
drivers/s390/crypto/vfio_ap_migration.c +209 -7
@@ -65,13 +65,6 @@ vfio_ap_release_stop_copy_file(struct vfio_ap_migration_data *mig_data) mig_data->stop_copy_mig_file =NULL ; } -static ssize_t -vfio_ap_stop_copy_read (struct file *,char __user *,size_t ,loff_t *) -{ -/* TODO */ -return -EOPNOTSUPP ; -} -static int vfio_ap_release_mig_file (struct inode *file_inode,struct file *filp) {struct ap_matrix_mdev *matrix_mdev = filp->private_data;
@@ -87,6 +80,215 @@ static int vfio_ap_release_mig_file(struct inode *file_inode, struct file *filp)return 0 ; } +/** +* validate_stop_copy_read_parms: Validate the input parameters to the +* vfio_ap_stop_copy_read function +* +* @matrix_mdev: The object device containing the state to be read +* @filp: Pointer to the file stream used to read the vfio-ap device state +* @pos: The file offset from which to start reading data +* @len: The length of the data to be read +* +* Verify the following: +* - @filp private data is an ap_matrix_mdev instance +* - @filp is the instance opened when state transitioned from STOP to STOP_COPY +* - @pos + @len does not cause integer overflow +* +* Returns: 0 if the parameters pass validation; otherwise returns an error +*/ +static int validate_stop_copy_read_parms (struct file *filp,loff_t *pos, +size_t len) +{ +struct vfio_ap_migration_data *mig_data; +struct ap_matrix_mdev *matrix_mdev; +loff_t total_len; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + +if (check_add_overflow ((loff_t )len, *pos, &total_len)) +return -EIO ; + + matrix_mdev = filp->private_data; + +if (!matrix_mdev || !matrix_mdev->mig_data) +return -ENODEV ; + + mig_data = matrix_mdev->mig_data; + +if (mig_data->stop_copy_mig_file != filp) +return -EINVAL ; + +return 0 ; +} + +static size_t vfio_ap_config_size (struct ap_matrix_mdev *matrix_mdev, +int *num_queues) +{ +size_t qinfo_size; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + + *num_queues =vfio_ap_mdev_get_num_queues (&matrix_mdev->shadow_apcb); + qinfo_size = *num_queues *sizeof (struct vfio_ap_queue_info ); + +return qinfo_size +sizeof (struct vfio_ap_config ); +} + +static int get_hardware_info_for_queue (struct ap_matrix_mdev *matrix_mdev, +struct ap_tapq_hwinfo *hwinfo, +unsigned long apqn) +{ +struct ap_queue_status status; + + status =ap_tapq (apqn, hwinfo); + +switch (status.response_code) { +case AP_RESPONSE_NORMAL : +case AP_RESPONSE_RESET_IN_PROGRESS : +case AP_RESPONSE_DECONFIGURED : +case AP_RESPONSE_CHECKSTOPPED : +case AP_RESPONSE_BUSY : +/* For all these RCs the tapq info should be available */ +return 0 ; +case AP_RESPONSE_Q_NOT_AVAIL : +dev_err (matrix_mdev->vdev.dev, +"migration failed: Failed to get hwinfo for queue %02lx.%04lx on target host: TAPQ rc=%d" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn), status.response_code); +return -ENODEV ; +default : +/* On a pending async error the tapq info should be available */ +if (!status.async) +return 0 ; + +dev_err (matrix_mdev->vdev.dev, +"Failed to get hwinfo for queue %02lx.%04lx: TAPQ rc=%d" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn), status.response_code); +return -EIO ; + } + +return -EINVAL ; +} + +static int vfio_ap_store_queue_info (struct ap_matrix_mdev *matrix_mdev, +struct vfio_ap_config *ap_config) +{ +unsigned long *apm, *aqm, num_queues, apid, apqi, apqn; +struct ap_tapq_hwinfo source_hwinfo; +int ret; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + + apm = matrix_mdev->shadow_apcb.apm; + aqm = matrix_mdev->shadow_apcb.aqm; + num_queues =0 ; + +for_each_set_bit_inv (apid, apm,AP_DEVICES ) { +for_each_set_bit_inv (apqi, aqm,AP_DOMAINS ) { + apqn =AP_MKQID (apid, apqi); + + ret =get_hardware_info_for_queue (matrix_mdev, + &source_hwinfo, apqn); +if (ret) +return ret; + + ap_config->qinfo[num_queues].apqn = apqn; + ap_config->qinfo[num_queues].data = source_hwinfo.value; + num_queues +=1 ; + } + } + +return (num_queues != ap_config->num_queues) ? -EINVAL :0 ; +} + +static int +vfio_ap_get_config (struct ap_matrix_mdev *matrix_mdev, +struct vfio_ap_config **ap_config,size_t *ap_config_size) +{ +struct vfio_ap_config *ap_configuration; +int num_queues, ret; + + *ap_config_size =vfio_ap_config_size (matrix_mdev, &num_queues); + + ap_configuration =kzalloc (*ap_config_size,GFP_KERNEL_ACCOUNT ); +if (!ap_configuration) +return -ENOMEM ; + + ap_configuration->num_queues = num_queues; + + ret =vfio_ap_store_queue_info (matrix_mdev, ap_configuration); +if (ret) { +kfree (ap_configuration); +return ret; + } + + *ap_config = ap_configuration; + +return 0 ; +} + +static ssize_t vfio_ap_stop_copy_read (struct file *filp,char __user *buf, +size_t len,loff_t *pos) +{ +struct ap_matrix_mdev *matrix_mdev; +size_t ret =0 , ap_config_size; +struct vfio_ap_config *ap_config; + +/* +* When userspace calls read() with an explicit offset (pread), pos is +* non-NULL and the function rejects it with -ESPIPE (illegal seek). For +* normal read() calls, pos is NULL, so we'll use the file's internal +* position filp->f_pos +*/ +if (pos) +return -ESPIPE ; + +mutex_lock (&matrix_dev->mdevs_lock); + + pos = &filp->f_pos; + + ret =validate_stop_copy_read_parms (filp, pos, len); +if (ret) { +mutex_unlock (&matrix_dev->mdevs_lock); +return ret; + } + + matrix_mdev = filp->private_data; + + ret =vfio_ap_get_config (matrix_mdev, &ap_config, &ap_config_size); +if (ret) { +mutex_unlock (&matrix_dev->mdevs_lock); +return ret; + } + +/* +* If the position exceeds the size of the AP configuration data, +* then indicate EOF; otherwise calculate the length of the data to +* read such that a buffer overrun is prevented. +*/ +if (*pos >= ap_config_size) + len =0 ; +else + len =min_t (size_t, ap_config_size - *pos, len); + +/* If we've reached an EOF condition, let the caller know */ +if (len ==0 ) { +kfree (ap_config); +mutex_unlock (&matrix_dev->mdevs_lock); +return 0 ; + } + +mutex_unlock (&matrix_dev->mdevs_lock); + +if (copy_to_user (buf, (char *)ap_config + *pos, len)) { +kfree (ap_config); +return -EFAULT ; + } + +kfree (ap_config); + *pos += len; +return len; +} +static const struct file_operations vfio_ap_stop_copy_fops = { .owner =THIS_MODULE , .read = vfio_ap_stop_copy_read,
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v4 07/15] s390/vfio-ap: File ops called to save the vfio device migration state
Thank you for your contribution! Sashiko AI review found 9 potential issue(s) to consider: - [Critical] The migration stream file does not hold a reference to the `matrix_mdev` device, allowing a…
Thank you for your contribution! Sashiko AI review found 9 potential issue(s) to consider:
- [Critical] The migration stream file does not hold a reference to the matrix_mdev device, allowing a trivial Use-After-Free if the file descriptor outlives the VFIO device.
- [High] Migration state is dynamically regenerated from hardware on every chunked read(), causing state tearing and massive performance overhead.
- [High] The config_sz field in the UAPI vfio_ap_config structure is never initialized.
- [High] Using kzalloc for potentially large variable-sized migration buffers can cause memory allocation failures.
- [High] Inverted condition when handling pending async errors in get_hardware_info_for_queue leads to false migration failures and silent data corruption.
- [Medium] Concurrent reads to the migration file race on filp->f_pos, corrupting the stream position.
- [Low] A size_t variable is used to store negative error codes, creating brittle type conversions.
- [Low] The error log in get_hardware_info_for_queue misleadingly states it is executing on the target host.
- [Low] Use of architecture-dependent size_t in a migration data structure.
--
--- Patch [7]: [PATCH v4 07/15] s390/vfio-ap: File ops called to save the vfio device migration state --- commit eb8c5880a35e9533d3246f99a48030464bf1844e
Author: Anthony Krowiak <[email protected]>
Subject: s390/vfio-ap: File ops called to save the vfio device migration state
This commit implements the read callback function for the vfio-ap device's migration state file. It copies the guest's AP configuration and hardware information to userspace when the migration state transitions to STOP_COPY. The state is then transferred to the destination host.
Quoted 8 lines
drivers/s390/crypto/vfio_ap_migration.c +0
-0@@ -87,6 +80,215 @@ static int vfio_ap_release_mig_file(struct inode *file_inode, struct file *filp)return 0 ; }
[ ... ]
Quoted 9 lines
drivers/s390/crypto/vfio_ap_migration.c +9
-0+static int get_hardware_info_for_queue (struct ap_matrix_mdev *matrix_mdev, +struct ap_tapq_hwinfo *hwinfo, +unsigned long apqn) +{ +struct ap_queue_status status; + + status =ap_tapq (apqn, hwinfo); + +switch (status.response_code) {
drivers/s390/crypto/vfio_ap_migration.c static int vfio_ap_release_mig_file(struct inode *file_inode, struct file *filp) · line 155
Quoted 10 lines
drivers/s390/crypto/vfio_ap_migration.c +10
-0+case AP_RESPONSE_NORMAL : +case AP_RESPONSE_RESET_IN_PROGRESS : +case AP_RESPONSE_DECONFIGURED : +case AP_RESPONSE_CHECKSTOPPED : +case AP_RESPONSE_BUSY : +/* For all these RCs the tapq info should be available */ +return 0 ; +case AP_RESPONSE_Q_NOT_AVAIL : + dev_err(matrix_mdev->vdev.dev, + "migration failed: Failed to get hwinfo for queue %02lx.%04lx on target host: TAPQ rc=%d",
[Severity: Low]
Is the target host string in this error message misleading?
Since this function is used to read the configuration during the STOP_COPY state, it executes on the source host rather than the target host.
drivers/s390/crypto/vfio_ap_migration.c static int vfio_ap_release_mig_file(struct inode *file_inode, struct file *filp) · line 161
Quoted 6 lines
drivers/s390/crypto/vfio_ap_migration.c +6
-0+ AP_QID_CARD(apqn), AP_QID_QUEUE(apqn), status.response_code); +return -ENODEV ; +default : +/* On a pending async error the tapq info should be available */ +if (!status.async) +return 0 ;
[Severity: High]
Does this logic invert the handling of pending async errors?
The comment indicates that on a pending async error the tapq info should be available (implying a return of 0). However, the code checks !status.async to return 0. If an async error is pending (status.async == 1), this will skip the return and incorrectly log an error and return -EIO. Conversely, if an unrecognized fatal error occurs (status.async == 0), it will silently return 0, potentially corrupting the migration state.
Quoted 9 lines
drivers/s390/crypto/vfio_ap_migration.c +9
-0+ +dev_err (matrix_mdev->vdev.dev, +"Failed to get hwinfo for queue %02lx.%04lx: TAPQ rc=%d" , +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn), status.response_code); +return -EIO ; + } + +return -EINVAL ; +}
[ ... ]
drivers/s390/crypto/vfio_ap_migration.c static int vfio_ap_release_mig_file(struct inode *file_inode, struct file *filp) · line 212
Quoted 10 lines
drivers/s390/crypto/vfio_ap_migration.c +10
-0+static int +vfio_ap_get_config (struct ap_matrix_mdev *matrix_mdev, +struct vfio_ap_config **ap_config,size_t *ap_config_size) +{ +struct vfio_ap_config *ap_configuration; +int num_queues, ret; + + *ap_config_size =vfio_ap_config_size (matrix_mdev, &num_queues); + + ap_configuration =kzalloc (*ap_config_size,GFP_KERNEL_ACCOUNT );
[Severity: High]
Could this kzalloc() fail on long-running systems when scaling to a large number of queues?
The total contiguous physical allocation size can exceed 1MB (up to 65,536 queues). Requesting such a large memory block (e.g., order-8) via kzalloc() with GFP_KERNEL is highly susceptible to physical memory fragmentation, which could result in -ENOMEM and abort the migration. Would kvzalloc() be more appropriate here to allow fallback to vmalloc?
[Severity: Low]
Does the underlying struct vfio_ap_config use an architecture-dependent type for UAPI/migration?
The struct uses a size_t config_sz field. Since this structure is serialized and sent to the destination host, using size_t (which is architecture-dependent) in an external ABI or migration stream could cause binary layout inconsistencies, even if s390x is exclusively 64-bit.
drivers/s390/crypto/vfio_ap_migration.c static int vfio_ap_release_mig_file(struct inode *file_inode, struct file *filp) · line 218
Quoted 6 lines
drivers/s390/crypto/vfio_ap_migration.c +6
-0+if (!ap_configuration) +return -ENOMEM ; + + ap_configuration->num_queues = num_queues; + + ret =vfio_ap_store_queue_info (matrix_mdev, ap_configuration);
[Severity: High]
Is the config_sz field of ap_configuration intentionally left uninitialized?
Since kzalloc() zeroes the memory, the config_sz field remains 0. When this state payload is received by the destination host, the stream might be rejected or misparsed if it attempts to validate the size of the incoming migration data against config_sz.
Quoted 5 lines
drivers/s390/crypto/vfio_ap_migration.c +5
-0+if (ret) { +kfree (ap_configuration); +return ret; + } +
drivers/s390/crypto/vfio_ap_migration.c static int vfio_ap_release_mig_file(struct inode *file_inode, struct file *filp) · line 233
Quoted 10 lines
drivers/s390/crypto/vfio_ap_migration.c +10
-0+ *ap_config = ap_configuration; + +return 0 ; +} + +static ssize_t vfio_ap_stop_copy_read (struct file *filp,char __user *buf, +size_t len,loff_t *pos) +{ +struct ap_matrix_mdev *matrix_mdev; +size_t ret =0 , ap_config_size;
[Severity: Low]
Is it safe to use size_t (an unsigned type) to store negative error codes?
The ret variable receives the results of validate_stop_copy_read_parms() and vfio_ap_get_config(), which return negative integers like -ENODEV. While the implicit cast back to ssize_t masks this currently, it creates a brittle pattern that could introduce bugs if ret is ever checked in a conditional statement (e.g., if (ret < 0)).
Quoted 12 lines
drivers/s390/crypto/vfio_ap_migration.c +12
-0+struct vfio_ap_config *ap_config; + +/* +* When userspace calls read() with an explicit offset (pread), pos is +* non-NULL and the function rejects it with -ESPIPE (illegal seek). For +* normal read() calls, pos is NULL, so we'll use the file's internal +* position filp->f_pos +*/ +if (pos) +return -ESPIPE ; + +mutex_lock (&matrix_dev->mdevs_lock);
drivers/s390/crypto/vfio_ap_migration.c static int vfio_ap_release_mig_file(struct inode *file_inode, struct file *filp) · line 255
Quoted 10 lines
drivers/s390/crypto/vfio_ap_migration.c +10
-0+ + pos = &filp->f_pos; + + ret =validate_stop_copy_read_parms (filp, pos, len); +if (ret) { +mutex_unlock (&matrix_dev->mdevs_lock); +return ret; + } + + matrix_mdev = filp->private_data;
[Severity: Critical]
Does the migration file descriptor hold a reference to matrix_mdev?
When the migration file descriptor is created, filp->private_data points to matrix_mdev. If a user keeps this file descriptor open and unbinds the device from sysfs, matrix_mdev is freed. Subsequent VFS operations (like read() here or close() in vfio_ap_release_mig_file()) will dereference the freed pointer, leading to a Use-After-Free.
drivers/s390/crypto/vfio_ap_migration.c
Quoted 2 lines
drivers/s390/crypto/vfio_ap_migration.c +2
-0+ + ret =vfio_ap_get_config (matrix_mdev, &ap_config, &ap_config_size);
[Severity: High]
Does dynamically regenerating the migration state on every read() cause tearing and performance issues?
If userspace reads the state in chunks (e.g., 4KB at a time), any hardware or configuration changes between chunks will shift the data layout, causing the migration stream to contain a torn and inconsistent snapshot.
Additionally, for a large number of queues, repeatedly polling the hardware via ap_tapq() on every chunk introduces massive O(N^2) overhead, which could stall the CPU and time out the migration. Should the driver cache a snapshot upon entering the STOP_COPY state instead?
Quoted 21 lines
drivers/s390/crypto/vfio_ap_migration.c +21
-0+if (ret) { +mutex_unlock (&matrix_dev->mdevs_lock); +return ret; + } + +/* +* If the position exceeds the size of the AP configuration data, +* then indicate EOF; otherwise calculate the length of the data to +* read such that a buffer overrun is prevented. +*/ +if (*pos >= ap_config_size) + len =0 ; +else + len =min_t (size_t, ap_config_size - *pos, len); + +/* If we've reached an EOF condition, let the caller know */ +if (len ==0 ) { +kfree (ap_config); +mutex_unlock (&matrix_dev->mdevs_lock); +return 0 ; + }
drivers/s390/crypto/vfio_ap_migration.c static int vfio_ap_release_mig_file(struct inode *file_inode, struct file *filp) · line 288
Quoted 10 lines
drivers/s390/crypto/vfio_ap_migration.c +10
-0+ +mutex_unlock (&matrix_dev->mdevs_lock); + +if (copy_to_user (buf, (char *)ap_config + *pos, len)) { +kfree (ap_config); +return -EFAULT ; + } + +kfree (ap_config); + *pos += len;
[Severity: Medium]
Can concurrent reads corrupt the stream position here?
The migration file is opened with stream_open(), so the VFS f_pos_lock is not held during read(). Because mdevs_lock is dropped before copying to userspace, concurrent reads will race on evaluating and incrementing *pos (which points to filp->f_pos), leading to duplicated data and a corrupted stream offset.
Quoted 2 lines
drivers/s390/crypto/vfio_ap_migration.c +2
-0+return len; +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
Anthony Krowiak
maintainerPATCH v4 8/15 s390/vfio-ap: Transition device migration state from STOP to RESUMING
In reply to Anthony KrowiakTransitioning from VFIO_DEVICE_STATE_STOP to VFIO_DEVICE_STATE_RESUMING starts a process of restoring the device state of the vfio device on the target system. To prepare for restoring the device state, open a file stream to receive the guest's AP configuration saved when the device state on the source system transitioned from VFIO_DEVICE_STATE_STOP to VFIO_DEVICE_STATE_STOP_COPY.
Note:
This commit introduces the file operations structure for migration state files with a stub implementation of the write operation. The complete implementation is provided in the subsequent commit.
This incremental approach allows for easier review of the migration infrastructure setup separate from the actual implementation logic.
--- drivers/s390/crypto/vfio_ap_migration.c | 54 ++++++++++++++++++++++++- 1 file changed, 52 insertions(+), 2 deletions(-)
drivers/s390/crypto/vfio_ap_migration.c +52 -2
@@ -65,6 +65,17 @@ vfio_ap_release_stop_copy_file(struct vfio_ap_migration_data *mig_data) mig_data->stop_copy_mig_file =NULL ; } +static void +vfio_ap_release_resuming_file (struct vfio_ap_migration_data *mig_data) +{ +if (mig_data->resuming_mig_state.filp) + mig_data->resuming_mig_state.filp =NULL ; + +kfree (mig_data->resuming_mig_state.ap_config); + mig_data->resuming_mig_state.ap_config =NULL ; + mig_data->resuming_mig_state.config_sz =0 ; +} +static int vfio_ap_release_mig_file (struct inode *file_inode,struct file *filp) {struct ap_matrix_mdev *matrix_mdev = filp->private_data;
@@ -74,6 +85,8 @@ static int vfio_ap_release_mig_file(struct inode *file_inode, struct file *filp)if (filp == matrix_mdev->mig_data->stop_copy_mig_file)vfio_ap_release_stop_copy_file (matrix_mdev->mig_data); +else if (filp == matrix_mdev->mig_data->resuming_mig_state.filp) +vfio_ap_release_resuming_file (matrix_mdev->mig_data);else return -ENOENT ;
@@ -311,6 +324,32 @@ static struct file *vfio_ap_open_file_stream(struct ap_matrix_mdev *matrix_mdev,return filp; } +static ssize_t vfio_ap_resuming_write (struct file *filp,const char __user *buf, +size_t len,loff_t *pos) +{ +/* TODO */ +return -EOPNOTSUPP ; +} + +static const struct file_operations vfio_ap_resume_fops = { + .owner =THIS_MODULE , + .write = vfio_ap_resuming_write, + .release = vfio_ap_release_mig_file, +}; + +static struct file *vfio_ap_resuming_init (struct ap_matrix_mdev *matrix_mdev) +{ +struct vfio_ap_migration_data *mig_data; +struct file *filp; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + + mig_data = matrix_mdev->mig_data; + filp =vfio_ap_open_file_stream (matrix_mdev, &vfio_ap_resume_fops,O_WRONLY ); + +return filp; +} +static struct file *vfio_ap_transition_to_state (struct ap_matrix_mdev *matrix_mdev,enum vfio_device_mig_state new_state)
@@ -342,10 +381,21 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,return filp; } +/* +* Begins the process of restoring the vfio device state by creating and +* returning a streaming data_fd to be used to read in the internal +* state of the vfio-ap device on the destination host. +*/ if (cur_state ==VFIO_DEVICE_STATE_STOP && new_state ==VFIO_DEVICE_STATE_RESUMING ) { -/* TODO */ -return ERR_PTR (-EOPNOTSUPP ); +struct file *filp =vfio_ap_resuming_init (matrix_mdev); + +if (IS_ERR (filp)) +return ERR_CAST (filp); + + mig_data->resuming_mig_state.filp = filp; + +return filp; } if ((cur_state == VFIO_DEVICE_STATE_RESUMING &&
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v4 08/15] s390/vfio-ap: Transition device migration state from STOP to RESUMING
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Migration file streams do not decouple their lifecycle from the vfio_device, allowing…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Migration file streams do not decouple their lifecycle from the vfio_device, allowing matrix_mdev to be accessed after being freed.
- [Medium] Concurrent reads on the migration state file cause a data race on filp->f_pos.
--
--- Patch [8]: [PATCH v4 08/15] s390/vfio-ap: Transition device migration state from STOP to RESUMING --- commit df7f5fdcda2503aec3804ab2b04e4b736cd6f0a1
Author: Anthony Krowiak <[email protected]>
s390/vfio-ap: Transition device migration state from STOP to RESUMING
Transitioning from VFIO_DEVICE_STATE_STOP to VFIO_DEVICE_STATE_RESUMING starts a process of restoring the device state of the vfio device on the target system. To prepare for restoring the device state, open a file stream to receive the guest's AP configuration saved when the device state on the source system transitioned from VFIO_DEVICE_STATE_STOP to VFIO_DEVICE_STATE_STOP_COPY.
Quoted 4 lines
drivers/s390/crypto/vfio_ap_migration.c +0
-0
[ ... ]
drivers/s390/crypto/vfio_ap_migration.c static int vfio_ap_release_mig_file(struct inode *file_inode, struct file *filp) · line 91
Quoted 8 lines
drivers/s390/crypto/vfio_ap_migration.c +2
-0@@ -74,6 +85,8 @@ static int vfio_ap_release_mig_file(struct inode *file_inode, struct file *filp)if (filp == matrix_mdev->mig_data->stop_copy_mig_file)vfio_ap_release_stop_copy_file (matrix_mdev->mig_data); +else if (filp == matrix_mdev->mig_data->resuming_mig_state.filp) +vfio_ap_release_resuming_file (matrix_mdev->mig_data);else return -ENOENT ;
[Severity: High]
Does this expose the matrix_mdev lifecycle to the file descriptor?
Migration file streams do not seem to decouple their lifecycle from the vfio_device. The migration state file descriptor is created using anon_inode_getfile() with matrix_mdev set as its private_data.
If a user closes the main device file descriptor but keeps the migration file descriptor open, and the administrator unbinds the mdev device, the core VFIO layer unregisters and frees matrix_mdev.
When the user later closes the migration file descriptor, vfio_ap_release_mig_file() dereferences filp->private_data to access the already freed matrix_mdev:
static int vfio_ap_release_mig_file(...)
{
struct ap_matrix_mdev *matrix_mdev = filp->private_data;
...
mutex_lock(&matrix_dev->mdevs_lock);
if (filp == matrix_mdev->mig_data->stop_copy_mig_file)Could this lead to a use-after-free of matrix_mdev when attempting to clean up the migration state?
[ ... ]
drivers/s390/crypto/vfio_ap_migration.c static struct file *vfio_ap_open_file_stream(struct ap_matrix_mdev *matrix_mdev, · line 325
Quoted 3 lines
drivers/s390/crypto/vfio_ap_migration.c +0
-0@@ -311,6 +324,32 @@ static struct file *vfio_ap_open_file_stream(struct ap_matrix_mdev *matrix_mdev,return filp; }
[Severity: Medium]
Is it possible for concurrent operations on these migration state files to cause a data race on filp->f_pos?
vfio_ap_open_file_stream() initializes the file with stream_open(), which sets FMODE_STREAM and clears FMODE_ATOMIC_POS. In the VFS layer, this skips acquiring f_pos_lock for stream files.
Looking at vfio_ap_stop_copy_read() which uses these stream files:
pos = &filp->f_pos;
...
mutex_unlock(&matrix_dev->mdevs_lock);
if (copy_to_user(buf, (char *)ap_config + *pos, len)) {
kfree(ap_config);
return -EFAULT;
}
kfree(ap_config);
*pos += len;Since matrix_dev->mdevs_lock is released before copy_to_user() and before *pos += len is executed, wouldn't concurrent read() calls read from the same *pos offset, perform copy_to_user() with duplicate data, and concurrently update *pos?
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Anthony Krowiak
maintainerPATCH v4 9/15 s390/vfio-ap: Add method to set a new guest AP configuration
In reply to Anthony KrowiakAdds a new vfio_ap_set_new_config function to set a guest's AP configuration. This is needed in order to set the state of the mdev when it is migrated from a remote host system during the RESUMING phase.
Key changes:
* Refactored code from the ap_config_store function - handles changes to the sysfs ap_config attribute - into a new, non-static function which is callable from the ap_config_store function as well as the live guest migration code.
* A flag parameter indicating whether the update to the guest's AP configuration can proceed if APIDs must be filtered out because one or more of the queues are not bound to the vfio_ap device driver. This is primarily for calls from the migration code because we don't want code that is running in the guest to lose access. The ap_config_store function will continue to operate as it did before and allow filtering.
--- drivers/s390/crypto/vfio_ap_ops.c | 358 +++++++++++++++++++++----- drivers/s390/crypto/vfio_ap_private.h | 4 + 2 files changed, 298 insertions(+), 64 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +294 -64
@@ -105,6 +105,27 @@ static inline void get_update_locks_for_mdev(struct ap_matrix_mdev *matrix_mdev)mutex_lock (&matrix_dev->mdevs_lock); } +/** +* assert_has_update_locks_for_mdev: +* +* Assert the locks required to dynamically update a KVM guest's APCB are +* currently held. +* +* @matrix_mdev: a pointer to a struct ap_matrix_mdev object containing the AP +* configuration data to use to update a KVM guest's APCB. +* +* Note: If @matrix_mdev is NULL or is not attached to a KVM guest, the KVM +* lock will not be taken. +*/ +static inline void +assert_has_update_locks_for_mdev (struct ap_matrix_mdev *matrix_mdev) +{ +lockdep_assert_held (&matrix_dev->guests_lock); +if (matrix_mdev && matrix_mdev->kvm) +lockdep_assert_held (&matrix_mdev->kvm->lock); +lockdep_assert_held (&matrix_dev->mdevs_lock); +} +/** * release_update_locks_for_mdev: Release the locks used to dynamically update a * KVM guest's APCB in the proper order.
@@ -875,7 +896,40 @@ static void vfio_ap_mdev_unlink_fr_queues(struct ap_matrix_mdev *matrix_mdev) q =vfio_ap_mdev_get_queue (matrix_mdev,AP_MKQID (apid, apqi));if (q) - q->matrix_mdev =NULL ; +vfio_ap_mdev_link_queue (matrix_mdev, q); + } + } +} + +static void vfio_ap_unlink_queues (struct ap_matrix_mdev *matrix_mdev) +{ +struct vfio_ap_queue *q; +unsigned long apid, apqi; + +for_each_set_bit_inv (apid, matrix_mdev->matrix.apm,AP_DEVICES ) { +for_each_set_bit_inv (apqi, matrix_mdev->matrix.aqm, +AP_DOMAINS ) { + q =vfio_ap_mdev_get_queue (matrix_mdev, +AP_MKQID (apid, apqi)); +if (q) { +vfio_ap_unlink_queue_fr_mdev (q); +vfio_ap_unlink_mdev_fr_queue (q); + } + } + } +} + +static void vfio_ap_link_queues (struct ap_matrix_mdev *matrix_mdev) +{ +struct vfio_ap_queue *q; +unsigned long apid, apqi; + +for_each_set_bit_inv (apid, matrix_mdev->matrix.apm,AP_DEVICES ) { +for_each_set_bit_inv (apqi, matrix_mdev->matrix.aqm, +AP_DOMAINS ) { + q =vfio_ap_find_queue (AP_MKQID (apid, apqi)); +if (q) +vfio_ap_mdev_link_queue (matrix_mdev, q); } } }
@@ -1014,19 +1068,36 @@ static void vfio_ap_mdev_link_adapter(struct ap_matrix_mdev *matrix_mdev,unsigned long apqi;for_each_set_bit_inv (apqi, matrix_mdev->matrix.aqm,AP_DOMAINS ) -vfio_ap_mdev_link_apqn (matrix_mdev, -AP_MKQID (apid, apqi)); +vfio_ap_mdev_link_apqn (matrix_mdev,AP_MKQID (apid, apqi)); } -static void collect_queues_to_reset (struct ap_matrix_mdev *matrix_mdev, -unsigned long apid, -struct list_head *qlist) +static void collect_queues_by_apid (struct ap_matrix_mdev *matrix_mdev, +unsigned long apid, +struct list_head *qlist) {struct vfio_ap_queue *q;unsigned long apqi;for_each_set_bit_inv (apqi, matrix_mdev->shadow_apcb.aqm,AP_DOMAINS ) { - q =vfio_ap_mdev_get_queue (matrix_mdev,AP_MKQID (apid, apqi)); + q = matrix_mdev ? +vfio_ap_mdev_get_queue (matrix_mdev,AP_MKQID (apid, apqi)) : +vfio_ap_find_queue (AP_MKQID (apid, apqi)); +if (q) +list_add_tail (&q->reset_qnode, qlist); + } +} + +static void collect_queues_by_apqi (struct ap_matrix_mdev *matrix_mdev, +unsigned long apqi, +struct list_head *qlist) +{ +struct vfio_ap_queue *q; +unsigned long apid; + +for_each_set_bit_inv (apid, matrix_mdev->shadow_apcb.apm,AP_DEVICES ) { + q = matrix_mdev ? +vfio_ap_mdev_get_queue (matrix_mdev,AP_MKQID (apid, apqi)) : +vfio_ap_find_queue (AP_MKQID (apid, apqi));if (q)list_add_tail (&q->reset_qnode, qlist); }
@@ -1038,7 +1109,7 @@ static void reset_queues_for_apid(struct ap_matrix_mdev *matrix_mdev,struct list_head qlist;INIT_LIST_HEAD (&qlist); -collect_queues_to_reset (matrix_mdev, apid, &qlist); +collect_queues_by_apid (matrix_mdev, apid, &qlist);vfio_ap_mdev_reset_qlist (&qlist); }
@@ -1054,7 +1125,7 @@ static int reset_queues_for_apids(struct ap_matrix_mdev *matrix_mdev,INIT_LIST_HEAD (&qlist);for_each_set_bit_inv (apid, apm_reset,AP_DEVICES ) -collect_queues_to_reset (matrix_mdev, apid, &qlist); +collect_queues_by_apid (matrix_mdev, apid, &qlist);return vfio_ap_mdev_reset_qlist (&qlist); }
@@ -1725,82 +1796,241 @@ static void ap_matrix_copy(struct ap_matrix *dst, struct ap_matrix *src)bitmap_copy (dst->adm, src->adm,AP_DOMAINS ); } -static ssize_t ap_config_store (struct device *dev,struct device_attribute *attr, -const char *buf,size_t count) +static int validate_ap_matrix (struct ap_matrix_mdev *matrix_mdev) { -struct ap_matrix_mdev *matrix_mdev =dev_get_drvdata (dev); -struct ap_matrix m_new, m_old, m_added, m_removed; -DECLARE_BITMAP (apm_filtered,AP_DEVICES ); -unsigned long newbit; -char *newbuf, *rest; -int rc = count; -bool do_update; +int rc; - newbuf =kstrndup (buf,AP_CONFIG_STRLEN ,GFP_KERNEL ); -if (!newbuf) -return -ENOMEM ; - rest = newbuf; +lockdep_assert_held (&ap_attr_mutex); +lockdep_assert_held (&matrix_dev->mdevs_lock); -mutex_lock (&ap_attr_mutex); -get_update_locks_for_mdev (matrix_mdev); + rc =vfio_ap_mdev_validate_masks (matrix_mdev); +if (rc) +return rc; + rc =ap_matrix_overflow_check (matrix_mdev); +if (rc) +return rc; -/* Save old state */ -ap_matrix_copy (&m_old, &matrix_mdev->matrix); -if (parse_bitmap (&rest, m_new.apm,AP_DEVICES ) || -parse_bitmap (&rest, m_new.aqm,AP_DOMAINS ) || -parse_bitmap (&rest, m_new.adm,AP_DOMAINS )) { - rc = -EINVAL ; - goto out; +return 0 ; +} + +static void get_removed_matrixes (struct ap_matrix *m_removed, +struct ap_matrix *m_old, +struct ap_matrix *m_new) +{ +bitmap_andnot (m_removed->apm, m_old->apm, m_new->apm,AP_DEVICES ); +bitmap_andnot (m_removed->aqm, m_old->aqm, m_new->aqm,AP_DOMAINS ); +bitmap_andnot (m_removed->adm, m_old->adm, m_new->adm,AP_DOMAINS ); +} + +static void reset_removed_queues_by_apid (unsigned long *apm_removed, +unsigned long *apm_filtered, +struct list_head *qlist) +{ +DECLARE_BITMAP (apids_removed,AP_DEVICES ); +unsigned long apid; + +for_each_set_bit_inv (apid, apm_filtered,AP_DEVICES ) +set_bit_inv (apid, apids_removed); + +for_each_set_bit_inv (apid, apm_removed,AP_DEVICES ) { +if (!test_bit_inv (apid, apids_removed)) +set_bit_inv (apid, apids_removed); } -bitmap_andnot (m_removed.apm, m_old.apm, m_new.apm,AP_DEVICES ); -bitmap_andnot (m_removed.aqm, m_old.aqm, m_new.aqm,AP_DOMAINS ); -bitmap_andnot (m_added.apm, m_new.apm, m_old.apm,AP_DEVICES ); -bitmap_andnot (m_added.aqm, m_new.aqm, m_old.aqm,AP_DOMAINS ); +if (!bitmap_empty (apids_removed,AP_DEVICES )) { +for_each_set_bit_inv (apid, apids_removed,AP_DEVICES ) +collect_queues_by_apid (NULL , apid, qlist); + } -/* Need new bitmaps in matrix_mdev for validation */ -ap_matrix_copy (&matrix_mdev->matrix, &m_new); +if (!list_empty (qlist)) +vfio_ap_mdev_reset_qlist (qlist); +} -/* Ensure new state is valid, else undo new state */ - rc =vfio_ap_mdev_validate_masks (matrix_mdev); -if (rc) { -ap_matrix_copy (&matrix_mdev->matrix, &m_old); - goto out; +/** +* remove_queues_already_reset: +* +* Remove the queues that have already beeen reset from a list of queues that +* have yet to be reset. +* +* @qlist_rst: A list of queues that have already been reset +* @qlist_rem: A list of queues from which already reset queues are to be +* removed. +*/ +static void remove_queues_already_reset (struct list_head *qlist_rst, +struct list_head *qlist_rem) +{ +struct vfio_ap_queue *rq, *qr, *trq, *tqr; + +if (list_empty (qlist_rst)) +return ; + +/* +* Each queue in qlist_reset has already been reset, so remove the +* matching queues from qlist_reset so they don't get reset again +*/ +list_for_each_entry_safe (qr, tqr, qlist_rem, reset_qnode) { +list_for_each_entry_safe (rq, trq, qlist_rst, reset_qnode) { +if (qr->apqn == rq->apqn) +list_del (&qr->reset_qnode); + } } - rc =ap_matrix_overflow_check (matrix_mdev); -if (rc) { -ap_matrix_copy (&matrix_mdev->matrix, &m_old); - goto out; +} + +static void reset_removed_queues (struct ap_matrix *m_removed, +unsigned long *apm_filtered) +{ +struct list_head qlist_by_apid, qlist_by_apqi; +DECLARE_BITMAP (apqis,AP_DOMAINS ); +unsigned long apqi; + +INIT_LIST_HEAD (&qlist_by_apid); +INIT_LIST_HEAD (&qlist_by_apqi); +bitmap_clear (apqis,0 ,AP_DOMAINS ); + +reset_removed_queues_by_apid (m_removed->apm, apm_filtered, &qlist_by_apid); + +for_each_set_bit_inv (apqi, m_removed->aqm,AP_DEVICES ) { +set_bit_inv (apqi, apqis); +collect_queues_by_apqi (NULL , apqi, &qlist_by_apqi); } - rc = count; -/* Need old bitmaps in matrix_mdev for unplug/unlink */ -ap_matrix_copy (&matrix_mdev->matrix, &m_old); +if (list_empty (&qlist_by_apqi)) +return ; -/* Unlink removed adapters/domains */ -vfio_ap_mdev_hot_unplug_adapters (matrix_mdev, m_removed.apm); -vfio_ap_mdev_hot_unplug_domains (matrix_mdev, m_removed.aqm); +remove_queues_already_reset (&qlist_by_apid, &qlist_by_apqi); -/* Need new bitmaps in matrix_mdev for linking new adapters/domains */ -ap_matrix_copy (&matrix_mdev->matrix, &m_new); +if (!list_empty (&qlist_by_apqi)) +vfio_ap_mdev_reset_qlist (&qlist_by_apid); +} -/* Link newly added adapters */ -for_each_set_bit_inv (newbit, m_added.apm,AP_DEVICES ) -vfio_ap_mdev_link_adapter (matrix_mdev, newbit); +/** +* restore_mdev_state: +* +* Restore the mdev to its previous state: +* - Unlink the queues from the updated mdev +* - Copy the previous matrix and shadow_apcb to the mdev +* - Re-link the original queues to the mdev +* +* @matrix_mdev: The object that maintains the AP configuration for a guest +* @m_old: The object containing the bitmaps specifying the guest AP +* configuration profile as it was prior to the attempt to set +* a new one +* @m_old_shadow: The object containing the bitmaps specifying the guest AP +* configuration as it was prior to the attempt to set a new +* one +*/ +static void restore_mdev_state (struct ap_matrix_mdev *matrix_mdev, +struct ap_matrix *m_old, +struct ap_matrix *m_old_shadow) +{ +vfio_ap_unlink_queues (matrix_mdev); +ap_matrix_copy (&matrix_mdev->matrix, m_old); +ap_matrix_copy (&matrix_mdev->shadow_apcb, m_old_shadow); +vfio_ap_link_queues (matrix_mdev); +} -for_each_set_bit_inv (newbit, m_added.aqm,AP_DOMAINS ) -vfio_ap_mdev_link_domain (matrix_mdev, newbit); +/** +* vfio_ap_set_new_guest_config: +* +* Set a new AP configuration for a guest. +* +* @matrix_mdev: Object used to maintain the AP configuration for a guest +* @m_new: Object used to set the new AP configuration +* @filtering_allowable: A boolean value indicating whether any filtering of +* new AP configuration is acceptable. If the configuration +* needs to be filtered before it can be passed through to +* the guest and this flag is set to false, then +* the operation shall be terminated with an error. +* +* Returns: zero (0) if the new AP configuration is successfully set; otherwise, +* returns an error: +* +* ~ EADDRNOTAVAIL One or more APQNs are reserved for host use +* ~ EADDRINUSE One or more APQNs are assigned to another mdev +* ~ ENODEV An adapter, domain or control domain in the new +* AP configuration exceeds the max architected value +* ~ ECANCELED @filtering_allowed was specified as false and the +* AP configuration needs to be filtered. +*/ +int vfio_ap_set_new_guest_config (struct ap_matrix_mdev *matrix_mdev, +struct ap_matrix *m_new, +bool filtering_allowable) +{ +DECLARE_BITMAP (apm_filtered,AP_DEVICES ); +struct ap_matrix m_old, m_old_shadow, m_removed; +bool do_update; +int rc; -/* filter resources not bound to vfio-ap */ +lockdep_assert_held (&ap_attr_mutex); +assert_has_update_locks_for_mdev (matrix_mdev); + +/* Save old state */ +ap_matrix_copy (&m_old, &matrix_mdev->matrix); +ap_matrix_copy (&m_old_shadow, &matrix_mdev->shadow_apcb); + +/* Reset mdev state */ +vfio_ap_unlink_queues (matrix_mdev); +ap_matrix_copy (&matrix_mdev->matrix, m_new); +vfio_ap_link_queues (matrix_mdev); + + rc =validate_ap_matrix (matrix_mdev); +if (rc) { +restore_mdev_state (matrix_mdev, &m_old, &m_old_shadow); +return rc; + } + +/* +* If APIDs need to be filtered from the guest AP config and filtering +* is not allowable according to the caller, then terminate the operation. +*/ do_update =vfio_ap_mdev_filter_matrix (matrix_mdev, apm_filtered); +if (!bitmap_empty (apm_filtered,AP_DEVICES ) && !filtering_allowable) { +restore_mdev_state (matrix_mdev, &m_old, &m_old_shadow); +return -ECANCELED ; + } + do_update |=vfio_ap_mdev_filter_cdoms (matrix_mdev); -/* Apply changes to shadow apbc if things changed */ -if (do_update) { +if (do_update)vfio_ap_mdev_update_guest_apcb (matrix_mdev); -reset_queues_for_apids (matrix_mdev, apm_filtered); + +get_removed_matrixes (&m_removed, &m_old, m_new); +if (!bitmap_empty (m_removed.apm,AP_DEVICES ) || + !bitmap_empty (apm_filtered,AP_DEVICES )) +reset_removed_queues (&m_removed, apm_filtered); + +return 0 ; +} + +static ssize_t ap_config_store (struct device *dev,struct device_attribute *attr, +const char *buf,size_t count) +{ +struct ap_matrix_mdev *matrix_mdev =dev_get_drvdata (dev); +struct ap_matrix m_new; +char *newbuf, *rest; +ssize_t rc; + + newbuf =kstrndup (buf,AP_CONFIG_STRLEN ,GFP_KERNEL ); +if (!newbuf) +return -ENOMEM ; + rest = newbuf; + +mutex_lock (&ap_attr_mutex); +get_update_locks_for_mdev (matrix_mdev); + +if (parse_bitmap (&rest, m_new.apm,AP_DEVICES ) || +parse_bitmap (&rest, m_new.aqm,AP_DOMAINS ) || +parse_bitmap (&rest, m_new.adm,AP_DOMAINS )) { +kfree (newbuf); +release_update_locks_for_mdev (matrix_mdev); +mutex_unlock (&ap_attr_mutex); +return -EINVAL ; } -out: + + rc =vfio_ap_set_new_guest_config (matrix_mdev, &m_new, true); +if (!rc) + rc = count; +release_update_locks_for_mdev (matrix_mdev);mutex_unlock (&ap_attr_mutex);kfree (newbuf);
drivers/s390/crypto/vfio_ap_private.h +4 -0
@@ -177,4 +177,8 @@ int vfio_ap_init_migration_data(struct ap_matrix_mdev *matrix_mdev);void vfio_ap_release_migration_data (struct ap_matrix_mdev *matrix_mdev);void vfio_ap_reset_migration_state (struct ap_matrix_mdev *matrix_mdev); +int vfio_ap_set_new_guest_config (struct ap_matrix_mdev *matrix_mdev, +struct ap_matrix *m_new, +bool filtering_allowable); +#endif /* _VFIO_AP_PRIVATE_H_ */
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v4 09/15] s390/vfio-ap: Add method to set a new guest AP configuration
Thank you for your contribution! Sashiko AI review found 8 potential issue(s) to consider: - [Critical] Unconditional NULL pointer dereference of `matrix_mdev` when gathering queues by APID and…
Thank you for your contribution! Sashiko AI review found 8 potential issue(s) to consider:
- [Critical] Unconditional NULL pointer dereference of matrix_mdev when gathering queues by APID and APQI.
- [Critical] Adding the same queue to two independent linked lists corrupts the reset_qnode list pointers.
- [Critical] vfio_ap_mdev_unlink_fr_queues links queues instead of unlinking them, leading to a UAF on mdev removal.
- [Critical] Unlinking queues from the old configuration and aggressively linking them to the new configuration before validating exclusivity leads to queue hijacking.
- [High] Kernel stack data leak and logic error due to uninitialized apids_removed bitmap.
- [High] AP hardware queues associated with removed AP domains are never reset, leaking cryptographic state to other guests.
- [High] Attempting to reset removed queues by passing a NULL matrix_mdev forces a global queue lookup, destructively resetting queues assigned to other guests.
- [Low] Iterating over AP_DOMAINS sized bitmap using AP_DEVICES size macro.
--
--- Patch [9]: [PATCH v4 09/15] s390/vfio-ap: Add method to set a new guest AP configuration --- commit ac6b1dacfdb4c478ea23f906be7720936567f0d9
Author: Anthony Krowiak <[email protected]>
s390/vfio-ap: Add method to set a new guest AP configuration
Adds a new vfio_ap_set_new_config function to set a guest's AP configuration. This is needed to set the state of the mdev when it is migrated from a remote host system during the RESUMING phase.
drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_mdev_unlink_fr_queues(struct ap_matrix_mdev *matrix_mdev) · line 899
Quoted 8 lines
drivers/s390/crypto/vfio_ap_ops.c +1
-1@@ -875,7 +896,40 @@ static void vfio_ap_mdev_unlink_fr_queues(struct ap_matrix_mdev *matrix_mdev) q =vfio_ap_mdev_get_queue (matrix_mdev,AP_MKQID (apid, apqi));if (q) - q->matrix_mdev =NULL ; +vfio_ap_mdev_link_queue (matrix_mdev, q);
[Severity: Critical]
Does this introduce a use-after-free risk when the mdev is removed?
By replacing the unlinking statement with vfio_ap_mdev_link_queue(), the queues retain pointers to the matrix_mdev struct. When an mdev is later destroyed, subsequent hardware interrupts or queue accesses could trigger memory corruption.
[ ... ]
drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_mdev_link_adapter(struct ap_matrix_mdev *matrix_mdev, · line 1081
Quoted 11 lines
drivers/s390/crypto/vfio_ap_ops.c +3
-3-static void collect_queues_to_reset (struct ap_matrix_mdev *matrix_mdev, -unsigned long apid, -struct list_head *qlist) +static void collect_queues_by_apid (struct ap_matrix_mdev *matrix_mdev, +unsigned long apid, +struct list_head *qlist) {struct vfio_ap_queue *q;unsigned long apqi;for_each_set_bit_inv (apqi, matrix_mdev->shadow_apcb.aqm,AP_DOMAINS ) {
[Severity: Critical]
Can this cause a NULL pointer dereference?
If matrix_mdev is passed as NULL (which happens when called from reset_removed_queues_by_apid()), the loop header will unconditionally dereference matrix_mdev->shadow_apcb.aqm before the NULL check occurs inside the loop.
drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_mdev_link_adapter(struct ap_matrix_mdev *matrix_mdev, · line 1084
Quoted 3 lines
drivers/s390/crypto/vfio_ap_ops.c +3
-0+ q = matrix_mdev ? +vfio_ap_mdev_get_queue (matrix_mdev,AP_MKQID (apid, apqi)) : +vfio_ap_find_queue (AP_MKQID (apid, apqi));
[Severity: High]
When matrix_mdev is NULL, could falling back to vfio_ap_find_queue() destructively reset queues assigned to other guests?
Since APIDs can be shared across multiple guests using disjoint domains, a global search might discover and indiscriminately reset hardware queues actively serving other guests.
drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_mdev_link_adapter(struct ap_matrix_mdev *matrix_mdev, · line 1097
Quoted 13 lines
drivers/s390/crypto/vfio_ap_ops.c +13
-0+if (q) +list_add_tail (&q->reset_qnode, qlist); + } +} + +static void collect_queues_by_apqi (struct ap_matrix_mdev *matrix_mdev, +unsigned long apqi, +struct list_head *qlist) +{ +struct vfio_ap_queue *q; +unsigned long apid; + +for_each_set_bit_inv (apid, matrix_mdev->shadow_apcb.apm,AP_DEVICES ) {
[Severity: Critical]
Is there a similar NULL pointer dereference risk here?
Similar to collect_queues_by_apid(), if matrix_mdev is NULL, the loop header evaluates matrix_mdev->shadow_apcb.apm before the ternary check.
[ ... ]
drivers/s390/crypto/vfio_ap_ops.c static void ap_matrix_copy(struct ap_matrix *dst, struct ap_matrix *src) · line 1833
Quoted 9 lines
drivers/s390/crypto/vfio_ap_ops.c +9
-0+static void reset_removed_queues_by_apid (unsigned long *apm_removed, +unsigned long *apm_filtered, +struct list_head *qlist) +{ +DECLARE_BITMAP (apids_removed,AP_DEVICES ); +unsigned long apid; + +for_each_set_bit_inv (apid, apm_filtered,AP_DEVICES ) +set_bit_inv (apid, apids_removed);
[Severity: High]
Can this lead to unintended hardware queues being reset?
DECLARE_BITMAP allocates array storage on the kernel stack but does not zero-initialize it. The set_bit_inv() calls flip specific bits, but the unmodified bits retain their previous stack garbage, which is later iterated over as removed APIDs.
[ ... ]
Quoted 4 lines
drivers/s390/crypto/vfio_ap_ops.c +4
-0+static void reset_removed_queues (struct ap_matrix *m_removed, +unsigned long *apm_filtered) +{ +struct list_head qlist_by_apid, qlist_by_apqi;
drivers/s390/crypto/vfio_ap_ops.c static void ap_matrix_copy(struct ap_matrix *dst, struct ap_matrix *src) · line 1892
Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +10
-0+DECLARE_BITMAP (apqis,AP_DOMAINS ); +unsigned long apqi; + +INIT_LIST_HEAD (&qlist_by_apid); +INIT_LIST_HEAD (&qlist_by_apqi); +bitmap_clear (apqis,0 ,AP_DOMAINS ); + +reset_removed_queues_by_apid (m_removed->apm, apm_filtered, &qlist_by_apid); + +for_each_set_bit_inv (apqi, m_removed->aqm,AP_DEVICES ) {
[Severity: Low]
Is AP_DEVICES the correct limit here?
It looks like this is iterating over m_removed->aqm, so using AP_DOMAINS would be the correct limit macro.
drivers/s390/crypto/vfio_ap_ops.c static void ap_matrix_copy(struct ap_matrix *dst, struct ap_matrix *src) · line 1904
Quoted 3 lines
drivers/s390/crypto/vfio_ap_ops.c +3
-0+set_bit_inv (apqi, apqis); +collect_queues_by_apqi (NULL , apqi, &qlist_by_apqi); + }
[Severity: Critical]
Could this corrupt the reset_qnode list pointers?
If an AP configuration update removes both an APID and an APQI matching a single queue, the same queue object is fetched first by reset_removed_queues_by_apid() and then by collect_queues_by_apqi().
Adding the same queue's reset_qnode to two independent list heads (qlist_by_apid and qlist_by_apqi) will blindly overwrite its internal next and prev pointers.
drivers/s390/crypto/vfio_ap_ops.c static void ap_matrix_copy(struct ap_matrix *dst, struct ap_matrix *src) · line 1903
Quoted 8 lines
drivers/s390/crypto/vfio_ap_ops.c +8
-0+ +if (list_empty (&qlist_by_apqi)) +return ; + +remove_queues_already_reset (&qlist_by_apid, &qlist_by_apqi); + +if (!list_empty (&qlist_by_apqi)) +vfio_ap_mdev_reset_qlist (&qlist_by_apid);
[Severity: High]
Should this pass &qlist_by_apqi instead of &qlist_by_apid?
If the domains list is not empty, passing &qlist_by_apid fails to reset the removed domains.
[ ... ]
Quoted 16 lines
drivers/s390/crypto/vfio_ap_ops.c +15
-0+int vfio_ap_set_new_guest_config (struct ap_matrix_mdev *matrix_mdev, +struct ap_matrix *m_new, +bool filtering_allowable) +{ +DECLARE_BITMAP (apm_filtered,AP_DEVICES ); +struct ap_matrix m_old, m_old_shadow, m_removed; +bool do_update; +int rc; +lockdep_assert_held (&ap_attr_mutex); +assert_has_update_locks_for_mdev (matrix_mdev); + +/* Save old state */ +ap_matrix_copy (&m_old, &matrix_mdev->matrix); +ap_matrix_copy (&m_old_shadow, &matrix_mdev->shadow_apcb); +
drivers/s390/crypto/vfio_ap_ops.c static void ap_matrix_copy(struct ap_matrix *dst, struct ap_matrix *src) · line 1980
Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +10
-0+/* Reset mdev state */ +vfio_ap_unlink_queues (matrix_mdev); +ap_matrix_copy (&matrix_mdev->matrix, m_new); +vfio_ap_link_queues (matrix_mdev); + + rc =validate_ap_matrix (matrix_mdev); +if (rc) { +restore_mdev_state (matrix_mdev, &m_old, &m_old_shadow); +return rc; + }
[Severity: Critical]
Could linking queues to the new configuration before validating exclusivity lead to queue hijacking?
By executing vfio_ap_link_queues() before validate_ap_matrix() enforces assignment conflicts, the code forcefully overwrites q->matrix_mdev on queues that might be assigned to other active guests.
If validate_ap_matrix() fails, the rollback in restore_mdev_state() unlinks the queue and leaves the original owner disconnected from their active resource.
[ ... ]
drivers/s390/crypto/vfio_ap_ops.c static void ap_matrix_copy(struct ap_matrix *dst, struct ap_matrix *src) · line 2000
Quoted 4 lines
drivers/s390/crypto/vfio_ap_ops.c +4
-0+get_removed_matrixes (&m_removed, &m_old, m_new); +if (!bitmap_empty (m_removed.apm,AP_DEVICES ) || + !bitmap_empty (apm_filtered,AP_DEVICES )) +reset_removed_queues (&m_removed, apm_filtered);
[Severity: High]
Does this conditional miss checking m_removed.aqm?
If an AP configuration update purely removes domains (aqm), this check skips reset_removed_queues() entirely. This leaves the hardware queues associated with removed domains un-reset, potentially leaking cryptographic state to subsequent guests.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
Anthony Krowiak
maintainerPATCH v4 10/15 s390/vfio-ap: File ops called to resume the vfio device migration
In reply to Anthony KrowiakImplements the 'write' callback function that was added to the 'file_operations' structure for the file stream created to restore the state of the vfio-ap device on the destination system when the migration state transitioned from STOP to RESUMING
The write callback retrieves the vfio device migration state saved to the file stream created when the vfio device state was transitioned from STOP to STOP_COPY. The saved state contains the source guest's AP configuration information. This data is copied from the userspace buffer passed to the 'write' callback and stored in the vfio_ap_config structure used to set the state of the vfio-ap device on the destination host. If the source guest's AP configuration is compatible with the AP configuration on the destination host, it will be hot plugged into the destination guest.
In order for the source guest's and destination host's AP configurations to be considered compatible:
* Each APQN in the source guest's AP configuration must also be in the destination host's AP configuration
* Each matching APQN in the destination host's AP configuration must be bound to the vfio_ap device driver
* Each matching APQN in the destination host's AP configuration must reference a queue device with compatible hardware:
- The source and destination queues must have the same facilities installed:
~ APSC facility
~ APQKM facility
~ AP4KC facility
- The source and destination queues must have the same mode:
~ Coprocessor-mode
~ Accelerator-mode
~ XCP-mode
- The source and destination queues must have the same APXA facility setting
~ If the APXA facility is installed on source queue, it must also
be installed on the destination queue and vice versa
- The source and destination queues must have a compatible classification setting. If the source queue has full native card function, then the destination queue must also have full native card function. If the source queue has stateless functions, then the destination queue can have stateless functions or full native card function because the latter includes the stateless functions.
- The binding and associated state for both the source and destination queues must indicate that the queue is usable for all messages (i.e., BS bits equal to 00).
- The AP type of the destination queue must be the same as or newer than the source queue (backward compatibility)
Note: The get_hardware_info_for_queue function that was created in
a previous patch was modified to take a mediated device name rather than an ap_matrix_mdev object because that is what is needed for this patch so the function can be executed without holding the matrix_dev->mdevs_lock.
--- drivers/s390/crypto/vfio_ap_migration.c | 785 +++++++++++++++++++++++- drivers/s390/crypto/vfio_ap_ops.c | 68 -- drivers/s390/crypto/vfio_ap_private.h | 56 ++ 3 files changed, 829 insertions(+), 80 deletions(-)
drivers/s390/crypto/vfio_ap_migration.c +773 -12
@@ -9,6 +9,40 @@#include "ap_bus.h" #include "vfio_ap_private.h" +/* +* Masks the fields of the queue information returned from the PQAP(TAPQ) +* command. In order to migrate a guest, it's AP configuration must be +* compatible with AP configuration assigned to the target guest's mdev. +* This mask is used to verify that the queue information for each source and +* target queue is compatible (i.e., the masked fields are equivalent). +* +* The relevant fields covered by this mask are: +* S bit 0: APSC facility installed +* M bit 1: APQKM facility installed +* C bit 2: AP4KC facility installed +* Mode bits 3-5: +* D bit 3: CCA-mode facility +* A bit 4: accelerator-mode facility +* X bit 5: XCP-mode facility +* N bit 6: APXA facility installed +* SL bit 7: SLCF facility installed +* Classification (functional capabilities) bits 8-16 +* bit 8: Native card function +* bit 9: Only stateless functions +* BS bits 16-17: +* AP Type bits 32-40: +*/ +#define QINFO_DATA_MASK 0xffffc000ff000000 + +/* +* Masks the bit that indicates whether full native card function is available +* from the 8 bits specifying the functional capabilities of a queue +*/ +#define CLASSIFICATION_NATIVE_FCN_MASK 0x80 + +/* The maximum number of queues that can be installed in an s390 system */ +#define MAX_AP_QUEUES (AP_DEVICES * AP_DOMAINS) +/** * vfio_ap_migration_data - the data needed to migrate a guest with pass-through * access to AP devices
@@ -147,7 +181,7 @@ static size_t vfio_ap_config_size(struct ap_matrix_mdev *matrix_mdev,return qinfo_size +sizeof (struct vfio_ap_config ); } -static int get_hardware_info_for_queue (struct ap_matrix_mdev *matrix_mdev, +static int get_hardware_info_for_queue (const char *mdev_name,struct ap_tapq_hwinfo *hwinfo,unsigned long apqn) {
@@ -164,22 +198,20 @@ static int get_hardware_info_for_queue(struct ap_matrix_mdev *matrix_mdev,/* For all these RCs the tapq info should be available */ return 0 ;case AP_RESPONSE_Q_NOT_AVAIL : -dev_err (matrix_mdev->vdev.dev, -"migration failed: Failed to get hwinfo for queue %02lx.%04lx on target host: TAPQ rc=%d" , -AP_QID_CARD (apqn),AP_QID_QUEUE (apqn), status.response_code); +pr_err ("vfio_ap_mdev %s: migration failed: Failed to get hwinfo for queue %02lx.%04lx on target host: TAPQ rc=%d" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn), status.response_code); +return -ENODEV ;default :/* On a pending async error the tapq info should be available */ if (!status.async)return 0 ; -dev_err (matrix_mdev->vdev.dev, -"Failed to get hwinfo for queue %02lx.%04lx: TAPQ rc=%d" , -AP_QID_CARD (apqn),AP_QID_QUEUE (apqn), status.response_code); +pr_err ("vfio_ap_mdev %s: Failed to get hwinfo for queue %02lx.%04lx: TAPQ rc=%d" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn), status.response_code); +return -EIO ; } - -return -EINVAL ; } static int vfio_ap_store_queue_info(struct ap_matrix_mdev *matrix_mdev,
@@ -187,10 +219,12 @@ static int vfio_ap_store_queue_info(struct ap_matrix_mdev *matrix_mdev, {unsigned long *apm, *aqm, num_queues, apid, apqi, apqn;struct ap_tapq_hwinfo source_hwinfo; +const char *mdev_name;int ret;lockdep_assert_held (&matrix_dev->mdevs_lock); + mdev_name =dev_name (matrix_mdev->vdev.dev); apm = matrix_mdev->shadow_apcb.apm; aqm = matrix_mdev->shadow_apcb.aqm; num_queues =0 ;
@@ -199,7 +233,7 @@ static int vfio_ap_store_queue_info(struct ap_matrix_mdev *matrix_mdev,for_each_set_bit_inv (apqi, aqm,AP_DOMAINS ) { apqn =AP_MKQID (apid, apqi); - ret =get_hardware_info_for_queue (matrix_mdev, + ret =get_hardware_info_for_queue (mdev_name, &source_hwinfo, apqn);if (ret)return ret;
@@ -324,11 +358,738 @@ static struct file *vfio_ap_open_file_stream(struct ap_matrix_mdev *matrix_mdev,return filp; } +static int validate_resuming_write_parms (struct file *filp, +size_t len,loff_t *pos) +{ +struct ap_matrix_mdev *matrix_mdev; +loff_t total_len; +int ret; + +lockdep_assert_held (&matrix_dev->mdevs_lock); + + matrix_mdev = filp->private_data; +if (!matrix_mdev || !matrix_mdev->mig_data) { + ret = -ENODEV ; + goto done; + } + +if (filp != matrix_mdev->mig_data->resuming_mig_state.filp) { + ret = -ENXIO ; + goto done; + } + +if (*pos <0 ) { + ret = -EINVAL ; + goto done; + } + +if (check_add_overflow ((loff_t )len, *pos, &total_len)) { + ret = -ERANGE ; + goto done; + } + +/* +* If the ap_config has not yet been allocated and the file position +* indicates this is not the first write, or the ap_config has been allocated +* but the file position indicates this is the first write, then this is an +* error condition. +*/ +if ((!matrix_mdev->mig_data->resuming_mig_state.ap_config && *pos !=0 ) || + (matrix_mdev->mig_data->resuming_mig_state.ap_config && *pos ==0 )) { + ret = -EFAULT ; + + goto done; + } + + ret =0 ; + +done: +return ret; +} + +static ssize_t calculate_ap_config_size (unsigned int num_queues) +{ +size_t qinfo_size; + +if (num_queues >MAX_AP_QUEUES ) +return -EINVAL ; + + qinfo_size = num_queues *sizeof (struct vfio_ap_queue_info ); +return qinfo_size +sizeof (struct vfio_ap_config ); +} + +/** +* allocate_ap_config: +* +* Allocate storage for the source guest's AP configuration data sent from +* userspace. +* +* @ap_config: The location in which to store the pointer to the storage +* allocated for the AP configuration data. +* @buf: The userspace buffer containing some or all of the source +* guest's AP configuration data +* @len: The number of bytes of data to copy from @buf +* +* Returns: The number of bytes of storage allocated for the config data or +* an error: +* +* +* -EIO: failed to copy data from @buf +* -EINVAL: the number of queues specified exceeds the max allowed +* -ENOMEM: the allocation of storage failed +*/ +static ssize_t allocate_ap_config (struct vfio_ap_config **ap_config, +const char __user *buf,size_t len) +{ +struct vfio_ap_config tmp_ap_config; +ssize_t config_size; +size_t copy_size; + +/* +* If the length of the data sent exceeds the size of the vfio_ap_config +* structure, then we will copy enough data from userspace to get the +* number of queues which we can use to allocate enough space all of +* the queue information. +*/ + copy_size =min (len,sizeof (tmp_ap_config)); + +if (copy_from_user (&tmp_ap_config, buf, copy_size)) +return -EIO ; + +/* +* If the length of data sent includes the number of queues +* in the AP configuration, then calculate its size; otherwise +* set config_size to the length of data sent. +*/ +if (len >=sizeof (struct vfio_ap_config )) { + config_size =calculate_ap_config_size (tmp_ap_config.num_queues); + +/* If the calculation returned an error */ +if (config_size <0 ) +return config_size; + }else { + config_size = len; + } + + *ap_config =kzalloc (config_size,GFP_KERNEL_ACCOUNT ); +if (!*ap_config) +return -ENOMEM ; + +return config_size; +} + +/** +* reallocate_ap_config: +* +* Reallocate the storage buffer so it is large enough to store the source +* guest's AP configuration data sent from userspace. +* +* @mig_ap_config: The location in which to store the pointer to the storage +* reallocated for the AP configuration data. +* len: The length of the data to be stored +* +* Returns: The size of the memory allocated for the source guest's +* AP configuration data or an error: +* +* -ENOMEM: The call to krealloc failed +* -EINVAL: The guest's AP configuration size changed between +* calls to the vfio_ap_resuming_write function. +* +*/ +static ssize_t reallocate_ap_config (struct vfio_ap_config **mig_ap_config, +size_t len) +{ +struct vfio_ap_config *ap_config = *mig_ap_config; +struct vfio_ap_config *new_ap_config; +size_t new_cfg_sz, cur_cfg_sz; +unsigned int num_queues; + + cur_cfg_sz = ap_config->config_sz; + num_queues = ap_config->num_queues; + +/* +* If the current configuration size is greater than the +* size of a vfio_ap_config structure (i.e., contains the num_queues +* field), then there should already be enough storage allocated +* to store the source guest's AP configuration. Let's verify that the +* amount of storage allocated is what we expect based on the number of +* vfio_ap_queue_info objects that must be stored. +*/ +if (cur_cfg_sz >=sizeof (struct vfio_ap_config )) { + new_cfg_sz =calculate_ap_config_size (num_queues); +if (cur_cfg_sz != new_cfg_sz) +return -EINVAL ; + }else { + new_cfg_sz = cur_cfg_sz + len; + } + + new_ap_config =krealloc (ap_config, new_cfg_sz,GFP_KERNEL_ACCOUNT ); +if (!new_ap_config) +return -ENOMEM ; + + *mig_ap_config = new_ap_config; + +return new_cfg_sz; +} + +/** +* qdev_is_bound_to_vfio_ap: +* +* Query to determine whether a queue with the specified APQN is available on +* the host system and bound to the vfio_ap device driver. +* +* @apqn: The APQN of the queue device being queried +* +* Returns: True if there is a queue device with the specified @apqn installed +* in the system and is bound to the vfio_ap device driver; otherwise, +* returns false. +*/ +static bool qdev_is_bound_to_vfio_ap (unsigned int apqn) +{ +struct ap_queue *queue; +bool is_bound = true; + + queue =ap_get_qdev (apqn); +if (!queue) +return false; + +if (queue->ap_dev.device.driver != &matrix_dev->vfio_ap_drv->driver) + is_bound = false; + +put_device (&queue->ap_dev.device); + +return is_bound; +} + +/** +* queues_available_on_target_system: +* +* Query whether each queue from the source guest's AP configuration is +* available and bound to the vfio_ap device driver; if not, log an error +* message. +* +* @mdev_name: The mdev name to use in error messages +* @source_config: The object specifying the source guest's AP configuration +* +* Returns: true if each queue identified in @source_config is available and +* bound to the vfio_ap device driver; otherwise, returns false. +*/ +static bool +queues_available_on_target_system (const char *mdev_name, +struct vfio_ap_config *source_config) +{ +unsigned long apqn; +bool ret = true; + +for (int i =0 ; i < source_config->num_queues; i++) { + apqn = source_config->qinfo[i].apqn; + +/* +* Find the queue device bound to the vfio_ap device driver. If it is +* not found, log an error and continue so users see all problems +* at once, not one-at-a-time through retries of the migration. +*/ +if (!qdev_is_bound_to_vfio_ap (apqn)) { +pr_err ("vfio_ap_mdev %s: Queue %02lx.%04lx not available to vfio_ap driver on target host\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + ret = false; + } + } + +return ret; +} + +static void report_facilities_compatibility (const char *mdev_name, +unsigned long apqn, +struct ap_tapq_hwinfo *src_hwinfo, +struct ap_tapq_hwinfo *target_hwinfo) +{ +if (src_hwinfo->apsc != target_hwinfo->apsc) { +if (src_hwinfo->apsc) { +pr_err ("vfio_ap_mdev %s: APSC facility installed in source queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +pr_err ("vfio_ap_mdev %s: APSC facility not installed in target queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + }else { +pr_err ("vfio_ap_mdev %s: APSC facility not installed in source queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +pr_err ("vfio_ap_mdev %s APSC facility installed in target queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + } + } + +if (src_hwinfo->mex4k != target_hwinfo->mex4k) { +if (src_hwinfo->mex4k) { +pr_err ("vfio_ap_mdev %s: mex4k facility installed in source queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +pr_err ("vfio_ap_mdev %s: mex4k facility not installed in target queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + }else { +pr_err ("vfio_ap_mdev %s: mex4k facility not installed in source queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +pr_err ("vfio_ap_mdev %s: mex4k facility installed in target queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + } + } + +if (src_hwinfo->crt4k != target_hwinfo->crt4k) { +if (src_hwinfo->crt4k) { +pr_err ("vfio_ap_mdev %s: crt4k facility installed in source queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +pr_err ("vfio_ap_mdev %s: crt4k facility not installed in target queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + }else { +pr_err ("vfio_ap_mdev %s: crt4k facility not installed in source queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +pr_err ("vfio_ap_mdev %s: crt4k facility installed in target queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + } + } +} + +static void report_mode_compatibility (const char *mdev_name, +unsigned long apqn, +struct ap_tapq_hwinfo *src_hwinfo, +struct ap_tapq_hwinfo *target_hwinfo) +{ +if (src_hwinfo->cca != target_hwinfo->cca) { +if (src_hwinfo->cca) { +pr_err ("vfio_ap_mdev %s: Coprocessor-mode facility installed in source queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +pr_err ("vfio_ap_mdev %s: Coprocessor-mode facility not installed target queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + }else { +pr_err ("vfio_ap_mdev %s: Coprocessor-mode facility not installed in source queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +pr_err ("vfio_ap_mdev %s: Coprocessor-mode facility installed target queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + } + } + +if (src_hwinfo->accel != target_hwinfo->accel) { +if (src_hwinfo->accel) { +pr_err ("vfio_ap_mdev %s: Accelerator-mode facility installed source queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +pr_err ("vfio_ap_mdev %s: Accelerator-mode facility not installed target queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + }else { +pr_err ("vfio_ap_mdev %s: Accelerator-mode facility not installed source queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +pr_err ("vfio_ap_mdev %s: Accelerator-mode facility installed target queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + } + } + +if (src_hwinfo->ep11 != target_hwinfo->ep11) { +if (src_hwinfo->ep11) { +pr_err ("vfio_ap_mdev %s: XCP-mode facility installed source queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +pr_err ("vfio_ap_mdev %s: XCP-mode facility not installed target queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + }else { +pr_err ("vfio_ap_mdev %s: XCP-mode facility not installed source queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +pr_err ("vfio_ap_mdev %s: XCP-mode facility installed target queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + } + } +} + +static void report_apxa_compatibility (const char *mdev_name, +unsigned long apqn, +struct ap_tapq_hwinfo *src_hwinfo, +struct ap_tapq_hwinfo *target_hwinfo) +{ +if (src_hwinfo->apxa != target_hwinfo->apxa) { +if (src_hwinfo->apxa) { +pr_err ("vfio_ap_mdev %s: AP-extended-addressing (APXA) facility installed in source queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +pr_err ("vfio_ap_mdev %s: AP-extended-addressing (APXA) facility not installed in target queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + }else { +pr_err ("vfio_ap_mdev %s: AP-extended-addressing (APXA) facility not installed in source queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +pr_err ("vfio_ap_mdev %s: AP-extended-addressing (APXA) facility installed in target queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + } + } +} + +static void report_slcf_compatibility (const char *mdev_name, +unsigned long apqn, +struct ap_tapq_hwinfo *src_hwinfo, +struct ap_tapq_hwinfo *target_hwinfo) +{ +if (src_hwinfo->slcf != target_hwinfo->slcf) { +if (src_hwinfo->slcf) { +pr_err ("vfio_ap_mdev %s: Stateless-command-filtering (SLCF) available in source queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +pr_err ("vfio_ap_mdev %s: Stateless-command-filtering (SLCF) not available in target queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + }else { +pr_err ("vfio_ap_mdev %s: Stateless-command-filtering (SLCF) not available in source queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +pr_err ("vfio_ap_mdev %s: Stateless-command-filtering (SLCF) available in target queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + } + } +} + +static void report_bs_compatibility (const char *mdev_name, +unsigned long apqn, +struct ap_tapq_hwinfo *src_hwinfo, +struct ap_tapq_hwinfo *target_hwinfo) +{ +/* +* The BS field on both the source and destination must be 0, so if one of +* them is not, then report an error. +*/ +if (src_hwinfo->bs || target_hwinfo->bs) { +pr_err ("vfio_ap_mdev %s: Bind/associate state for source (%01x) and target (%01x) queue %02lx.%04lx must be 0\n" , + mdev_name, src_hwinfo->bs, target_hwinfo->bs, +AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + } +} + +static void report_aptype_compatibility (const char *mdev_name, +unsigned long apqn, +struct ap_tapq_hwinfo *src_hwinfo, +struct ap_tapq_hwinfo *target_hwinfo) +{ +if (src_hwinfo->at > target_hwinfo->at) { +pr_err ("vfio_ap_mdev %s: AP type of source (%02x) not compatible with target (%02x)\n" , + mdev_name, src_hwinfo->at, target_hwinfo->at); + } +} + +static bool classes_compatible (struct ap_tapq_hwinfo *src_hwinfo, +struct ap_tapq_hwinfo *target_hwinfo) +{ +unsigned long src_native, target_native; + + src_native = src_hwinfo->class &CLASSIFICATION_NATIVE_FCN_MASK ; + target_native = target_hwinfo->class &CLASSIFICATION_NATIVE_FCN_MASK ; + +/* +* If the source queue has full native card function and the +* target queue has only stateless functions available, then +* there may be instructions that will not execute on the +* target queue. This shall be reported as an error. +* +* If the source queue has only stateless card functions and the +* target queue has full native card function available, then +* we are okay because the target queue can run all stateless card +* functions. +*/ +return (src_native != target_native) ? !src_native : true; +} + +static void report_class_compatibility (const char *mdev_name, +unsigned long apqn, +struct ap_tapq_hwinfo *src_hwinfo, +struct ap_tapq_hwinfo *target_hwinfo) +{ +if (!classes_compatible (src_hwinfo, target_hwinfo)) { +pr_err ("vfio_ap_mdev %s: Full native card function available on source queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +pr_err ("vfio_ap_mdev %s: Only stateless functions available on target queue %02lx.%04lx\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + } +} + +/* +* Log a device error reporting that migration failed due to queue +* incompatibilities followed by a device error for each incompatible feature. +*/ +static void report_qinfo_incompatibilities (const char *mdev_name, +unsigned long apqn, +struct ap_tapq_hwinfo *src_hwinfo, +struct ap_tapq_hwinfo *target_hwinfo) +{ +pr_err ("vfio_ap_mdev %s: Migration failed: Source and target queue (%02lx.%04lx) not compatible\n" , + mdev_name,AP_QID_CARD (apqn),AP_QID_QUEUE (apqn)); + +report_facilities_compatibility (mdev_name, apqn, src_hwinfo, target_hwinfo); +report_mode_compatibility (mdev_name, apqn, src_hwinfo, target_hwinfo); +report_apxa_compatibility (mdev_name, apqn, src_hwinfo, target_hwinfo); +report_slcf_compatibility (mdev_name, apqn, src_hwinfo, target_hwinfo); +report_aptype_compatibility (mdev_name, apqn, src_hwinfo, target_hwinfo); +report_bs_compatibility (mdev_name, apqn, src_hwinfo, target_hwinfo); +report_class_compatibility (mdev_name, apqn, src_hwinfo, target_hwinfo); +} + +/** +* queue_hardware_info_is_compatible: +* +* Verify whether the hardware information for a source queue is compatible with +* the hardware info for the corresponding queue on this system. +* +* In order to be compatible, the hardware information for each queue must +* meet the following requirements: +* +* 1. The hardware facilities bits much match +* 2. The AP type of the source queue must be the same as or older than that +* of the target queue (target is backwards compatible) +* 3. The classification bits must indicate: +* - Both queues have full native card function or both have stateless +* functions available +* - If the classification bits don't match, then the only acceptable +* configuration is stateless functions for the source queue and +* full native function for the target queue +* 4. The BS bits for both queues must be 0 (Queue usable for all messages +* supported by the adapter) +* +* @mdev_name: The mdev name to use in error messages +* @apqn: The APQN for the queues +* @src_hwinfo: The hardware info for the source queue +* @target_hwinfo: The hardware info for the corresponding queue on this system +* +* Returns: true if the hardware info for the two queues is compatible; +* otherwise, returns false. +*/ +static bool queue_hardware_info_is_compatible (const char *mdev_name, +unsigned long apqn, +struct ap_tapq_hwinfo *src_hwinfo, +struct ap_tapq_hwinfo *target_hwinfo) +{ +unsigned long src_bits, target_bits; + + src_bits = src_hwinfo->value &QINFO_DATA_MASK ; + target_bits = target_hwinfo->value &QINFO_DATA_MASK ; + +/* If all bits match the queues are compatible */ +if (src_bits == target_bits) +return true; + +if (src_hwinfo->fac == target_hwinfo->fac && + src_hwinfo->at <= target_hwinfo->at && +classes_compatible (src_hwinfo, target_hwinfo) && + (src_hwinfo->bs ==0 && target_hwinfo->bs ==0 )) { +return true; + } + +report_qinfo_incompatibilities (mdev_name, apqn, src_hwinfo, target_hwinfo); + +return false; +} + +/** +* verify_ap_configs_are_compatible: +* +* Verifies that the queues in the source guest's AP configuration are +* compatible with the corresponding queues on this system. +* +* @mdev_name: The mdev name to use in error messages +* @source_config: The object specifying the source guest's AP configuration +* +* Returns: an error indicating either a failure to retrieve a queue's +* hardware information or one or more source queues are not +* compatible with the corresponding queue on this system; otherwise, +* returns zero to indicate compatibility. +*/ +static int verify_ap_configs_are_compatible (const char *mdev_name, +struct vfio_ap_config *source_config) +{ +struct ap_tapq_hwinfo src_hwinfo, dest_hwinfo; +unsigned long apqn; +int ret =0 , rc; + +for (int i =0 ; i < source_config->num_queues; i++) { + apqn = source_config->qinfo[i].apqn; + +/* +* If we can't get the hardware info for a particular queue, then let's +* capture the function return code and continue so we can log all +* errors to aid in debugging of migration. +*/ + rc =get_hardware_info_for_queue (mdev_name, &dest_hwinfo, apqn); +if (rc) { + ret = rc; +continue ; + } + + src_hwinfo.value = source_config->qinfo[i].data; + +if (!queue_hardware_info_is_compatible (mdev_name, apqn, + &src_hwinfo, + &dest_hwinfo)) + ret = -EFAULT ; + } + +return ret; +} + +static int do_post_copy_validation (const char *mdev_name, +struct vfio_ap_config *source_config) +{ +if (!queues_available_on_target_system (mdev_name, source_config)) +return -ENODEV ; + +return verify_ap_configs_are_compatible (mdev_name, source_config); +} + +/** +* setup_ap_matrix_from_ap_config: +* +* Set the bits corresponding to the adapters, domains and control domains +* in the source guest's AP configuration into an ap_matrix object to be +* used to update the target guest to run on this host. An error message will +* be logged for each adapter, domain or control domain that is not available +* on this host. Returning an error after the each error may result in needing +* to initiate multiple migrations in order to find and fix each of them. +* +* Returns: zero (0) if each adapter, domain and control domain from the +* source guest's ap configuration is available on this host; +* otherwise, returns -ENODEV. +*/ +static void setup_ap_matrix_from_ap_config (const char *mdev_name, +struct vfio_ap_config *ap_config, +struct ap_matrix *guest_matrix) +{ +struct vfio_ap_queue_info qinfo; +unsigned long apid, apqi; + +for (int i =0 ; i < ap_config->num_queues; i++) { + qinfo = ap_config->qinfo[i]; + apid =AP_QID_CARD (qinfo.apqn); + apqi =AP_QID_QUEUE (qinfo.apqn); + +if (!test_bit_inv (apid, guest_matrix->apm)) +set_bit_inv (apid, guest_matrix->apm); +if (!test_bit_inv (apqi, guest_matrix->aqm)) +set_bit_inv (apqi, guest_matrix->aqm); + } +} +static ssize_t vfio_ap_resuming_write (struct file *filp,const char __user *buf,size_t len,loff_t *pos) { -/* TODO */ -return -EOPNOTSUPP ; +struct vfio_ap_migration_data *mig_data; +struct ap_matrix_mdev *matrix_mdev; +struct vfio_ap_config *ap_config; +struct ap_matrix guest_matrix; +ssize_t ret =0 , cfg_sz; +const char *mdev_name; + +/* +* When userspace calls write() with an explicit offset (pwrite), pos is +* non-NULL and the function rejects it with -ESPIPE (illegal seek). For +* normal write() calls, pos is NULL, so we'll use the file's internal +* position filp->f_pos +*/ +if (pos) +return -ESPIPE ; + +mutex_lock (&matrix_dev->mdevs_lock); + pos = &filp->f_pos; + + ret =validate_resuming_write_parms (filp, len, pos); +if (ret) + goto done; + + matrix_mdev = filp->private_data; + mig_data = matrix_mdev->mig_data; + mdev_name =dev_name (matrix_mdev->vdev.dev); + +/* +* If this is the first write operation, then allocate storage for the +* AP configuration information; otherwise, reallocate the +* struct vfio_ap_config object used to store the AP configuration data +* sent from userspace. +*/ +if (*pos ==0 ) { + ret =allocate_ap_config (&ap_config, buf, len); + +/* If the allocation failed, we'll return the error */ +if (ret <0 ) + goto done; + + cfg_sz = ret; + }else { + ap_config = mig_data->resuming_mig_state.ap_config; + + ret =reallocate_ap_config (&ap_config, len); +if (ret <0 ) + goto cleanup; + + cfg_sz = ret; + } + +if (*pos + len > cfg_sz) { + ret = -EIO ; + goto cleanup; + } + +/* +* We don't want to lock all mdevs while copying data from userspace so +* we don't block all other mdevs in case the I/O takes a long time. +* From here on out we don't need this lock because we are not +* accessing the matrix_mdev until we need to get the update +* locks to set the new destination guest's AP configuration in which +* case this lock will be taken then. +*/ +mutex_unlock (&matrix_dev->mdevs_lock); + +if (copy_from_user ((char *)ap_config + *pos, buf, len)) { + ret = -EIO ; + goto cleanup; + } + +/* Check if we've completed writing the entire configuration */ +if (*pos + len == cfg_sz) { + ret =do_post_copy_validation (mdev_name, ap_config); +if (ret <0 ) + goto cleanup; + +setup_ap_matrix_from_ap_config (mdev_name, ap_config, + &guest_matrix); + +/* Acquire locks required to update the guest's AP config */ +mutex_lock (&ap_attr_mutex); +get_update_locks_for_mdev (matrix_mdev); + + ret =vfio_ap_set_new_guest_config (matrix_mdev, &guest_matrix, + false); +if (!ret) { + mig_data->resuming_mig_state.ap_config = ap_config; + mig_data->resuming_mig_state.config_sz = cfg_sz; + } + +release_update_locks_for_mdev (matrix_mdev); +mutex_unlock (&ap_attr_mutex); + +if (ret) + goto cleanup; + } + + ret = len; + *pos += len; + goto done; + +cleanup: +kfree (ap_config); +done: +if (mutex_is_locked (&matrix_dev->mdevs_lock)) +mutex_unlock (&matrix_dev->mdevs_lock); + +return ret; }static const struct file_operations vfio_ap_resume_fops = {
drivers/s390/crypto/vfio_ap_ops.c +0 -68
74 lines not on this page · raw message
drivers/s390/crypto/vfio_ap_private.h +56 -0
62 lines not on this page · raw message
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v4 10/15] s390/vfio-ap: File ops called to resume the vfio device migration
Thank you for your contribution! Sashiko AI review found 8 potential issue(s) to consider: - [Critical] A time-of-check to time-of-use (TOCTOU) vulnerability exists due to a double fetch of the…
Thank you for your contribution! Sashiko AI review found 8 potential issue(s) to consider:
- [Critical] A time-of-check to time-of-use (TOCTOU) vulnerability exists due to a double fetch of the migration configuration, leading to out-of-bounds memory access.
- [Critical] The stack-allocated guest_matrix is used without initialization, leading to random AP devices and domains being assigned to the guest.
- [Critical] Error and reallocation paths free the ap_config buffer while leaving a dangling pointer in the persistent migration state, causing a Use-After-Free or Double Free.
- [Critical] Misuse of mutex_is_locked to conditionally release mdevs_lock wrongfully unlocks a mutex actively held by a concurrent thread.
- [High] vfio_ap_resuming_write unconditionally aborts normal write operations because the pos argument is never NULL.
- [High] Writing a payload smaller than sizeof(struct vfio_ap_config) allocates an undersized buffer but triggers configuration validation, causing an out-of-bounds heap read.
- [High] Writing the migration stream in chunks skips saving the configuration pointer, causing a memory leak and completely breaking stream resumption.
- [Medium] A strict equality check on the fac bitfield incorrectly rejects compatible device classifications, breaking the intended backward compatibility logic. --
--- Patch [10]: [PATCH v4 10/15] s390/vfio-ap: File ops called to resume the vfio device migration --- commit 2ee7fca0e8467c278538f305a2be0b05605513a9
Author: Anthony Krowiak <[email protected]>
s390/vfio-ap: File ops called to resume the vfio device migration
Implements the 'write' callback function that was added to the 'file_operations' structure for the file stream created to restore the state of the vfio-ap device on the destination system when the migration state transitioned from STOP to RESUMING.
Quoted 3 lines
drivers/s390/crypto/vfio_ap_migration.c +0
-0
[ ... ]
Quoted 7 lines
drivers/s390/crypto/vfio_ap_migration.c +7
-0+static ssize_t allocate_ap_config (struct vfio_ap_config **ap_config, +const char __user *buf,size_t len) +{ +struct vfio_ap_config tmp_ap_config; +ssize_t config_size; +size_t copy_size; +
drivers/s390/crypto/vfio_ap_migration.c static struct file *vfio_ap_open_file_stream(struct ap_matrix_mdev *matrix_mdev, · line 457
Quoted 10 lines
drivers/s390/crypto/vfio_ap_migration.c +10
-0+/* +* If the length of the data sent exceeds the size of the vfio_ap_config +* structure, then we will copy enough data from userspace to get the +* number of queues which we can use to allocate enough space all of +* the queue information. +*/ + copy_size =min (len,sizeof (tmp_ap_config)); + +if (copy_from_user (&tmp_ap_config, buf, copy_size)) +return -EIO ;
[Severity: Critical]
Could this initial fetch allow a time-of-check to time-of-use (TOCTOU) race?
The first copy_from_user() here reads tmp_ap_config to calculate the allocation size based on num_queues. Later in vfio_ap_resuming_write():
if (copy_from_user((char *)ap_config + *pos, buf, len)) {The entire payload is copied again from userspace. If a concurrent userspace thread modifies num_queues between these two fetches, it can bypass the size calculation and inject an arbitrarily large num_queues into the heap buffer, which will then be used during the validation loops causing out-of-bounds access.
Quoted 5 lines
drivers/s390/crypto/vfio_ap_migration.c +5
-0+ +/* +* If the length of data sent includes the number of queues +* in the AP configuration, then calculate its size; otherwise +* set config_size to the length of data sent.
drivers/s390/crypto/vfio_ap_migration.c static struct file *vfio_ap_open_file_stream(struct ap_matrix_mdev *matrix_mdev, · line 472
Quoted 10 lines
drivers/s390/crypto/vfio_ap_migration.c +10
-0+*/ +if (len >=sizeof (struct vfio_ap_config )) { + config_size =calculate_ap_config_size (tmp_ap_config.num_queues); + +/* If the calculation returned an error */ +if (config_size <0 ) +return config_size; + }else { + config_size = len; + }
[Severity: High]
If userspace writes a payload smaller than sizeof(struct vfio_ap_config), could this allocate an undersized buffer?
If config_size is set to len here, and then *pos + len == cfg_sz evaluates to true in vfio_ap_resuming_write(), the code proceeds to verify_ap_configs_are_compatible().
verify_ap_configs_are_compatible() will then iterate:
for (int i = 0; i < source_config->num_queues; i++) {
apqn = source_config->qinfo[i].apqn;using the num_queues value read from the small buffer, potentially reading far beyond the allocated memory.
[ ... ]
Quoted 10 lines
drivers/s390/crypto/vfio_ap_migration.c +10
-0+static bool queue_hardware_info_is_compatible (const char *mdev_name, +unsigned long apqn, +struct ap_tapq_hwinfo *src_hwinfo, +struct ap_tapq_hwinfo *target_hwinfo) +{ +unsigned long src_bits, target_bits; + + src_bits = src_hwinfo->value &QINFO_DATA_MASK ; + target_bits = target_hwinfo->value &QINFO_DATA_MASK ; +
drivers/s390/crypto/vfio_ap_migration.c static struct file *vfio_ap_open_file_stream(struct ap_matrix_mdev *matrix_mdev, · line 886
Quoted 10 lines
drivers/s390/crypto/vfio_ap_migration.c +10
-0+/* If all bits match the queues are compatible */ +if (src_bits == target_bits) +return true; + +if (src_hwinfo->fac == target_hwinfo->fac && + src_hwinfo->at <= target_hwinfo->at && +classes_compatible (src_hwinfo, target_hwinfo) && + (src_hwinfo->bs ==0 && target_hwinfo->bs ==0 )) { +return true; + }
[Severity: Medium]
Will this strict equality check on the fac bitfield reject compatible device classifications?
The 32-bit fac union field encompasses the 8-bit class bitfield. If the source queue has stateless functions and the target has full native functions, their class sub-fields differ, causing their encompassing fac fields to differ. This causes the strict equality check on fac to fail, bypassing the fallback logic in classes_compatible().
[ ... ]
drivers/s390/crypto/vfio_ap_migration.c static struct file *vfio_ap_open_file_stream(struct ap_matrix_mdev *matrix_mdev, · line 987
Quoted 7 lines
drivers/s390/crypto/vfio_ap_migration.c +7
-0+static ssize_t vfio_ap_resuming_write (struct file *filp,const char __user *buf, +size_t len,loff_t *pos) +{ +struct vfio_ap_migration_data *mig_data; +struct ap_matrix_mdev *matrix_mdev; +struct vfio_ap_config *ap_config; +struct ap_matrix guest_matrix;
[Severity: Critical]
Is it possible this passes uninitialized stack memory as the guest matrix?
guest_matrix is declared on the stack without initialization here. setup_ap_matrix_from_ap_config() sets bits in guest_matrix using set_bit_inv() but never zeros the rest of the struct:
if (!test_bit_inv(apid, guest_matrix->apm))
set_bit_inv(apid, guest_matrix->apm);This uninitialized matrix is then passed to vfio_ap_set_new_guest_config(), which might grant the guest access to random AP hardware resources based on the uninitialized stack memory.
drivers/s390/crypto/vfio_ap_migration.c static struct file *vfio_ap_open_file_stream(struct ap_matrix_mdev *matrix_mdev, · line 998
Quoted 11 lines
drivers/s390/crypto/vfio_ap_migration.c +11
-0+ssize_t ret =0 , cfg_sz; +const char *mdev_name; + +/* +* When userspace calls write() with an explicit offset (pwrite), pos is +* non-NULL and the function rejects it with -ESPIPE (illegal seek). For +* normal write() calls, pos is NULL, so we'll use the file's internal +* position filp->f_pos +*/ +if (pos) +return -ESPIPE ;
[Severity: High]
Does this unconditionally abort normal write operations?
The VFS layer (vfs_write) always passes a valid loff_t *pos pointer to the write callback, so pos will never be NULL here, resulting in -ESPIPE for every write attempt.
[ ... ]
Quoted 20 lines
drivers/s390/crypto/vfio_ap_migration.c +20
-0+/* Check if we've completed writing the entire configuration */ +if (*pos + len == cfg_sz) { + ret =do_post_copy_validation (mdev_name, ap_config); +if (ret <0 ) + goto cleanup; + +setup_ap_matrix_from_ap_config (mdev_name, ap_config, + &guest_matrix); + +/* Acquire locks required to update the guest's AP config */ +mutex_lock (&ap_attr_mutex); +get_update_locks_for_mdev (matrix_mdev); + + ret =vfio_ap_set_new_guest_config (matrix_mdev, &guest_matrix, + false); +if (!ret) { + mig_data->resuming_mig_state.ap_config = ap_config; + mig_data->resuming_mig_state.config_sz = cfg_sz; + } +
drivers/s390/crypto/vfio_ap_migration.c static struct file *vfio_ap_open_file_stream(struct ap_matrix_mdev *matrix_mdev, · line 1084
Quoted 10 lines
drivers/s390/crypto/vfio_ap_migration.c +10
-0+release_update_locks_for_mdev (matrix_mdev); +mutex_unlock (&ap_attr_mutex); + +if (ret) + goto cleanup; + } + + ret = len; + *pos += len; + goto done;
[Severity: High]
Does writing the migration stream in partial chunks result in a memory leak and break stream resumption?
When *pos + len < cfg_sz, the code jumps directly to done here. This skips saving the allocated pointer to mig_data->resuming_mig_state.ap_config, leaking the allocation.
On the next sequential write, validate_resuming_write_parms() executes:
if ((!matrix_mdev->mig_data->resuming_mig_state.ap_config && *pos != 0) ||
...
ret = -EFAULT;Since the pointer was never saved, the first condition is met, permanently aborting the migration stream.
drivers/s390/crypto/vfio_ap_migration.c static struct file *vfio_ap_open_file_stream(struct ap_matrix_mdev *matrix_mdev, · line 1087
Quoted 3 lines
drivers/s390/crypto/vfio_ap_migration.c +3
-0+ +cleanup: +kfree (ap_config);
[Severity: Critical]
Does this leave a dangling pointer in the migration state if an error occurs after the configuration pointer was saved?
When a complete configuration is written, the pointer is saved:
mig_data->resuming_mig_state.ap_config = ap_config;
If a subsequent write triggers a reallocate_ap_config() failure, or an I/O error, the code jumps to cleanup and frees the memory. However, mig_data->resuming_mig_state.ap_config is not cleared. Releasing the file descriptor later or subsequent operations might access or double free this pointer.
drivers/s390/crypto/vfio_ap_migration.c static struct file *vfio_ap_open_file_stream(struct ap_matrix_mdev *matrix_mdev, · line 1090
Quoted 3 lines
drivers/s390/crypto/vfio_ap_migration.c +3
-0+done: +if (mutex_is_locked (&matrix_dev->mdevs_lock)) +mutex_unlock (&matrix_dev->mdevs_lock);
[Severity: Critical]
Can this mistakenly release a mutex actively held by a concurrent thread?
mutex_is_locked() only checks if the mutex is held by any thread, not strictly the current one. Since the lock was explicitly released earlier in the function (before copy_from_user), if another thread acquired mdevs_lock in the meantime, this will return true and unlock the mutex while the other thread still owns it.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
Anthony Krowiak
maintainerPATCH v4 11/15 s390/vfio-ap: Transition device migration state to STOP
In reply to Anthony KrowiakTypically at this point, interrupt processing for the device, DMA and other changes to the device's external state must be prohibited. For VFIO AP, there is no physical DMA device virtualized by the vfio-ap device . It merely manages the guest's AP configuration that identifies the devices passed through to the guest, but does not have access to any of them. Those AP devices are passed through and controlled exclusively by the SIE program used to start the guest. There only thing to do at this point is return NULL to the caller since the file streams and associated data will get cleaned up as a result of the VFIO migration framework releasing the files.
--- drivers/s390/crypto/vfio_ap_migration.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/s390/crypto/vfio_ap_migration.c +7 -2
@@ -1159,12 +1159,17 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,return filp; } +/* +* Terminates the data transfer session of the vfio-ap device state +* between the source and target hosts. Since the vfio-ap device does +* not virtualize a DMA device, there is no internal device state to +* incorporate into the vfio-ap device on the target. +*/ if ((cur_state ==VFIO_DEVICE_STATE_RESUMING && new_state ==VFIO_DEVICE_STATE_STOP ) || (cur_state ==VFIO_DEVICE_STATE_STOP_COPY && new_state ==VFIO_DEVICE_STATE_STOP )) { -/* TODO */ -return ERR_PTR (-EOPNOTSUPP ); +return NULL ; } if ((cur_state == VFIO_DEVICE_STATE_STOP &&
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v4 11/15] s390/vfio-ap: Transition device migration state to STOP
Thank you for your contribution! Sashiko AI review found 8 potential issue(s) to consider: - [Critical] Fragmented or malformed writes to the resuming file descriptor trigger severe memory…
Thank you for your contribution! Sashiko AI review found 8 potential issue(s) to consider:
- [Critical] Fragmented or malformed writes to the resuming file descriptor trigger severe memory corruption bugs (Double Free, NULL Dereference, Leak).
- [Critical] Insufficient length check in allocate_ap_config allows out-of-bounds heap read during post-copy validation.
- [Critical] Uninitialized stack variable guest_matrix is used to configure the guest AP matrix.
- [High] Improper handling of RESUMING -> STOP transition enables state machine bypass, lack of validation, and memory leaks.
- [High] The config_sz field in the serialized struct vfio_ap_config is never set during the STOP_COPY phase, breaking allocation logic on the target host.
- [High] Transitions between RUNNING and STOP states incorrectly return -EOPNOTSUPP.
- [High] Missing locking in vfio_ap_release_mig_file leads to a TOCTOU data race and Use-After-Free of migration state pointers.
- [Medium] vfio_ap_release_migration_data leaks ap_config when the device is closed.
--
--- Patch [11]: [PATCH v4 11/15] s390/vfio-ap: Transition device migration state to STOP --- commit 07eb7dcf0820cdb3df175eeebfcfafc28e35382d
Author: Anthony Krowiak <[email protected]>
s390/vfio-ap: Transition device migration state to STOP
This commit modifies the VFIO AP migration state machine to return NULL when transitioning to the STOP state from RESUMING or STOP_COPY. It removes the unsupported error return to allow the migration framework to clean up the file streams.
Quoted 11 lines
drivers/s390/crypto/vfio_ap_migration.c +3
-0@@ -1159,12 +1159,17 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,return filp; } +/* +* Terminates the data transfer session of the vfio-ap device state +* between the source and target hosts. Since the vfio-ap device does
drivers/s390/crypto/vfio_ap_migration.c vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev, · line 1172
Quoted 10 lines
drivers/s390/crypto/vfio_ap_migration.c +4
-2+* not virtualize a DMA device, there is no internal device state to +* incorporate into the vfio-ap device on the target. +*/ if ((cur_state ==VFIO_DEVICE_STATE_RESUMING && new_state ==VFIO_DEVICE_STATE_STOP ) || (cur_state ==VFIO_DEVICE_STATE_STOP_COPY && new_state ==VFIO_DEVICE_STATE_STOP )) { -/* TODO */ -return ERR_PTR (-EOPNOTSUPP ); +return NULL ;
[Severity: High]
Does returning NULL here without explicitly closing the file or clearing mig_data->resuming_mig_state.filp allow a state machine bypass?
If a misbehaving userspace keeps the file descriptor open and transitions the device back to RUNNING, it appears it could bypass migration state machine constraints. Because vfio_ap_resuming_write() only checks that the file descriptor matches resuming_mig_state.filp rather than verifying the current mig_state, could userspace overwrite the active AP configuration while the device is fully running?
Also, would initiating another migration cycle blindly overwrite filp and cause a memory leak?
drivers/s390/crypto/vfio_ap_migration.c vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev, · line 1175
Quoted 3 lines
drivers/s390/crypto/vfio_ap_migration.c +0
-0} if ((cur_state == VFIO_DEVICE_STATE_STOP &&
[Severity: High]
Are transitions between RUNNING and STOP states incorrectly returning -EOPNOTSUPP in this adjacent block?
drivers/s390/crypto/vfio_ap_migration.c:vfio_ap_transition_to_state() {
...
if ((cur_state == VFIO_DEVICE_STATE_STOP &&
new_state == VFIO_DEVICE_STATE_RUNNING) ||
(cur_state == VFIO_DEVICE_STATE_RUNNING &&
new_state == VFIO_DEVICE_STATE_STOP)) {
/* TODO */
return ERR_PTR(-EOPNOTSUPP);
}
...
}The VFIO migration specification mandates that devices must support transitions between VFIO_DEVICE_STATE_RUNNING and VFIO_DEVICE_STATE_STOP. When vfio_mig_get_next_state() attempts to transition the device through the state machine, will this stub logic instantly fail and render the device migration feature broken?
[Severity: Critical]
Can fragmented or malformed writes to the resuming file descriptor trigger memory corruption?
In vfio_ap_resuming_write():
if (*pos + len == cfg_sz) {
ret = do_post_copy_validation(mdev_name, ap_config);
...
mig_data->resuming_mig_state.ap_config = ap_config;
}If a short write occurs where len < sizeof(config), it appears ap_config is allocated locally but not saved to mig_data because *pos + len == cfg_sz evaluates to false, causing a memory leak.
Then, if a subsequent write fetches the unsaved (NULL) pointer and passes it to reallocate_ap_config():
cur_cfg_sz = ap_config->config_sz;
Could this unconditionally dereference ap_config, causing a kernel crash?
Alternatively, if a full write is followed by an extra write, reallocate_ap_config() executes krealloc(). If the bounds check fails and cleanup frees the local ap_config pointer, does mig_data->resuming_mig_state.ap_config still hold the freed pointer, leading to a Double Free when vfio_ap_release_resuming_file() is called?
[Severity: Critical]
Is allocate_ap_config() allocating an undersized buffer if the input payload is smaller than expected?
drivers/s390/crypto/vfio_ap_migration.c:allocate_ap_config() {
...
} else {
config_size = len;
}
*ap_config = kzalloc(config_size, GFP_KERNEL_ACCOUNT);
...
}If userspace writes a payload strictly smaller than sizeof(struct vfio_ap_config), and vfio_ap_resuming_write() considers the configuration fully written, the undersized buffer is passed to do_post_copy_validation().
In queues_available_on_target_system():
for (int i = 0; i < source_config->num_queues; i++) {
apqn = source_config->qinfo[i].apqn;Does this allow an out-of-bounds heap read based on the untrusted and undersized config buffer?
[Severity: Critical]
Could the stack variable guest_matrix be used without initialization?
drivers/s390/crypto/vfio_ap_migration.c:vfio_ap_resuming_write() {
...
struct ap_matrix guest_matrix;
...
setup_ap_matrix_from_ap_config(mdev_name, ap_config,
&guest_matrix);
...
}In setup_ap_matrix_from_ap_config():
if (!test_bit_inv(apid, guest_matrix->apm)) set_bit_inv(apid, guest_matrix->apm);
It looks like guest_matrix is declared on the stack without being zero-initialized, and setup_ap_matrix_from_ap_config() only sets individual bits. Does this leave uninitialized bits in the apm, aqm, and adm bitmaps, potentially granting the target guest hardware access to unauthorized AP adapters and domains?
[Severity: High]
Are we missing the initialization of the config_sz field during the STOP_COPY phase?
drivers/s390/crypto/vfio_ap_migration.c:vfio_ap_get_config() {
ap_configuration = kzalloc(*ap_config_size, GFP_KERNEL_ACCOUNT);
if (!ap_configuration)
return -ENOMEM;
ap_configuration->num_queues = num_queues;
...
}The allocation succeeds, but it doesn't look like ap_configuration->config_sz is initialized before returning it to be copied to user space. If this structure is transferred and the target host receives config_sz == 0, would a fragmented write on the target erroneously call krealloc() to shrink the buffer to merely len bytes, breaking fragmented migrations?
[Severity: High]
Is there a locking issue in vfio_ap_release_mig_file() that could lead to a data race and Use-After-Free?
drivers/s390/crypto/vfio_ap_migration.c:vfio_ap_release_mig_file() {
...
if (!matrix_mdev || !matrix_mdev->mig_data)
return -ENODEV;
if (filp == matrix_mdev->mig_data->stop_copy_mig_file)
vfio_ap_release_stop_copy_file(matrix_mdev->mig_data);
...
}This accesses matrix_mdev->mig_data entirely locklessly. Since vfio_ap_release_migration_data() and vfio_ap_transition_to_state() modify and free this data under matrix_dev->mdevs_lock, could a concurrent close of the migration file descriptor result in a Time-of-Check to Time-of-Use race?
If vfio_ap_release_mig_file() evaluates matrix_mdev->mig_data as valid but a concurrent teardown thread kfrees it, would this result in a Use-After-Free when dereferencing mig_data->stop_copy_mig_file?
[Severity: Medium]
Will vfio_ap_release_migration_data() leak ap_config when the device is closed?
drivers/s390/crypto/vfio_ap_migration.c:vfio_ap_release_migration_data() {
if (!matrix_mdev->mig_data)
return;
kfree(matrix_mdev->mig_data);
matrix_mdev->mig_data = NULL;
}This frees mig_data without cleaning up the dynamically allocated mig_data->resuming_mig_state.ap_config buffer inside it.
Because the pointer is nulled, when VFS eventually drops the final reference to the RESUMING migration file, vfio_ap_release_mig_file() will abort early with -ENODEV. Will this bypass the kfree call within vfio_ap_release_resuming_file() and permanently leak the allocated ap_config memory?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
Anthony Krowiak
maintainerPATCH v4 12/15 s390/vfio-ap: Transition device migration state from STOP to RUNNING and vice versa
In reply to Anthony KrowiakThe transition from STOP to RUNNING and vice versa indicates that the vfio device is operating normally and migration is not currently taking place. In this case, just set the new state.
--- drivers/s390/crypto/vfio_ap_migration.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/s390/crypto/vfio_ap_migration.c +7 -2
@@ -1172,12 +1172,17 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,return NULL ; } +/* +* These states indicate migration has either not been initiated or +* has completed and the vfio-ap device is operating normally.Since the +* vfio-ap device does not virtualize a DMA device, there is no internal +* device state to incorporate into the vfio-ap device on the target. +*/ if ((cur_state ==VFIO_DEVICE_STATE_STOP && new_state ==VFIO_DEVICE_STATE_RUNNING ) || (cur_state ==VFIO_DEVICE_STATE_RUNNING && new_state ==VFIO_DEVICE_STATE_STOP )) { -/* TODO */ -return ERR_PTR (-EOPNOTSUPP ); +return NULL ; }/* vfio_mig_get_next_state() does not use arcs other than the above */
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v4 12/15] s390/vfio-ap: Transition device migration state from STOP to RUNNING and vice versa
Thank you for your contribution! Sashiko AI review found 7 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 7 potential issue(s) to consider:
New issues:
- [High] The transition to VFIO_DEVICE_STATE_STOP does not quiesce the AP hardware queues, leaving the device running during and after migration.
Pre-existing issues:
- [Critical] Writing fewer bytes than sizeof(struct vfio_ap_config) leads to a ZERO_SIZE_PTR panic or Slab Out-of-Bounds read.
- [Critical] A partial write of the AP configuration at pos == 0 permanently leaks the allocated memory and leads to a NULL pointer dereference crash.
- [Critical] Error paths and reallocation logic in vfio_ap_resuming_write free ap_config without clearing the global pointer in mig_data.
- [High] The guest_matrix stack variable is used without initialization when setting up the AP matrix.
- [High] Concurrent write operations on the migration file descriptors cause a data race on filp->f_pos, bypassing bounds checks and corrupting the migration stream state.
- [High] vfio_ap_release_mig_file accesses and modifies mig_data internal pointers locklessly, racing with concurrent device state transitions. --
--- Patch [12]: [PATCH v4 12/15] s390/vfio-ap: Transition device migration state from STOP to RUNNING and vice versa --- commit 6d2ef5d46b7ff5343fd3f1a4ec81fec852894cf9
Author: Anthony Krowiak <[email protected]>
s390/vfio-ap: Transition device migration state from STOP to RUNNING and vice versa
This commit handles the transition between the STOP and RUNNING states for the vfio-ap device during migration. It indicates that the device is operating normally and sets the new state by returning NULL instead of an error.
Quoted 11 lines
drivers/s390/crypto/vfio_ap_migration.c +3
-0@@ -1172,12 +1172,17 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev,return NULL ; } +/* +* These states indicate migration has either not been initiated or +* has completed and the vfio-ap device is operating normally.Since the
drivers/s390/crypto/vfio_ap_migration.c vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev, · line 1185
Quoted 10 lines
drivers/s390/crypto/vfio_ap_migration.c +4
-2+* vfio-ap device does not virtualize a DMA device, there is no internal +* device state to incorporate into the vfio-ap device on the target. +*/ if ((cur_state ==VFIO_DEVICE_STATE_STOP && new_state ==VFIO_DEVICE_STATE_RUNNING ) || (cur_state ==VFIO_DEVICE_STATE_RUNNING && new_state ==VFIO_DEVICE_STATE_STOP )) { -/* TODO */ -return ERR_PTR (-EOPNOTSUPP ); +return NULL ;
[Severity: High]
Does this transition to VFIO_DEVICE_STATE_STOP fully quiesce the hardware?
The VFIO migration specification mandates that VFIO_DEVICE_STATE_STOP must fully stop the device from executing or performing DMA. By just returning NULL here without calling vfio_ap_mdev_reset_queues() or similar, won't the AP queues continue to asynchronously process in-flight requests and perform DMA writes to the guest's NIB physical address?
drivers/s390/crypto/vfio_ap_migration.c vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev, · line 1188
Quoted 3 lines
drivers/s390/crypto/vfio_ap_migration.c +0
-0}/* vfio_mig_get_next_state() does not use arcs other than the above */
While reviewing this migration code, I also noticed several pre-existing issues in the vfio_ap_migration.c implementation:
[Severity: High]
This is a pre-existing issue, but does setup_ap_matrix_from_ap_config() use an uninitialized stack variable?
In vfio_ap_resuming_write(), guest_matrix is declared on the stack without initialization:
struct ap_matrix guest_matrix;
...
setup_ap_matrix_from_ap_config(mdev_name, ap_config, &guest_matrix);Inside setup_ap_matrix_from_ap_config(), it only sets bits to 1:
if (!test_bit_inv(apid, guest_matrix->apm))
set_bit_inv(apid, guest_matrix->apm);Can this leave uninitialized stack garbage in the arrays (like the adm array), causing the target guest to be configured with unintended AP adapter access?
[Severity: Critical]
This is a pre-existing issue, but does allocate_ap_config() handle small lengths correctly?
If a user writes a small chunk where len < sizeof(struct vfio_ap_config), the code falls back to allocating len bytes:
if (len >= sizeof(struct vfio_ap_config)) {
config_size = calculate_ap_config_size(tmp_ap_config.num_queues);
...
} else {
config_size = len;
}
*ap_config = kzalloc(config_size, GFP_KERNEL_ACCOUNT);If a user writes 0 bytes, won't kzalloc(0) return ZERO_SIZE_PTR, causing a kernel panic upon dereference?
Also, if len is small (e.g., 8 bytes), this bypasses the minimum size check. Could this allow the tiny buffer to be passed to do_post_copy_validation(), which reads num_queues from the heap, potentially leading to an out-of-bounds memory access?
[Severity: Critical]
This is a pre-existing issue, but does a partial write leak memory and cause a NULL pointer dereference?
In vfio_ap_resuming_write(), if the first write is partial, the newly allocated pointer is not saved to the global state until the entire write is complete:
if (*pos + len == cfg_sz) {
...
mig_data->resuming_mig_state.ap_config = ap_config;
}On the next write chunk, the code retrieves the pointer from mig_data:
ap_config = mig_data->resuming_mig_state.ap_config;
ret = reallocate_ap_config(&ap_config, len);Since the pointer was never stored on the partial write, it retrieves a NULL pointer. Inside reallocate_ap_config(), it immediately dereferences ap_config to read config_sz. Won't this cause a guaranteed NULL pointer dereference crash? Also, doesn't the initially allocated buffer leak entirely?
[Severity: Critical]
This is a pre-existing issue, but do error paths in vfio_ap_resuming_write() leave dangling pointers?
If an error occurs on a subsequent chunk (such as *pos + len > cfg_sz), the cleanup path frees the pointer:
ap_config = mig_data->resuming_mig_state.ap_config;
...
cleanup:
kfree(ap_config);However, mig_data->resuming_mig_state.ap_config is not cleared to NULL. When the file descriptor is subsequently closed, vfio_ap_release_mig_file() will unconditionally call kfree() on it again. Can this trigger a double free?
[Severity: High]
This is a pre-existing issue, but is there a data race on the file position?
vfio_ap_resuming_write() uses the file's internal position and locklessly updates it after dropping mdevs_lock:
mutex_unlock(&matrix_dev->mdevs_lock);
if (copy_from_user((char *)ap_config + *pos, buf, len)) {
...
ret = len;
*pos += len;Since the file stream is opened with stream_open(), FMODE_ATOMIC_POS is cleared, and the VFS layer does not serialize writes with f_pos_lock. Could concurrent write syscalls race to read and update *pos, bypassing bounds checks and corrupting the migration state?
[Severity: High]
This is a pre-existing issue, but does vfio_ap_release_mig_file() safely handle concurrent device state transitions?
The VFS release callback checks filp and frees ap_config locklessly:
else if (filp == matrix_mdev->mig_data->resuming_mig_state.filp)
vfio_ap_release_resuming_file(matrix_mdev->mig_data);If a concurrent VFIO ioctl transitions the device state and overwrites mig_data->resuming_mig_state.filp before the old file descriptor is released, won't this callback fail to match the old filp? Could this bypass cleanup and leak memory, or improperly free memory belonging to a new migration session?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
Anthony Krowiak
maintainerPATCH v4 13/15 s390/vfio-ap: Callback to get the size of data to be migrated during guest migration
In reply to Anthony KrowiakCallback to get the estimated data length of the data that will be required to complete reading of the vfio device's internal state. It's mandatory for VFIO_DEVICE_FEATURE_MIGRATION migration support.
The function pointer for this callback is specified via the 'migration_get_data_size' field of the 'vfio_migration_ops' structure which is stored with the VFIO device when the 'vfio_device' structure representing the mediated device is initialized.
--- drivers/s390/crypto/vfio_ap_migration.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
drivers/s390/crypto/vfio_ap_migration.c +12 -1
@@ -1283,7 +1283,18 @@ static int vfio_ap_get_state(struct vfio_device *vdev,static int vfio_ap_get_data_size (struct vfio_device *vdev,unsigned long *stop_copy_length) { -return -EOPNOTSUPP ; +struct ap_matrix_mdev *matrix_mdev; +size_t qinfo_sz; +int num_queues; + +mutex_lock (&matrix_dev->mdevs_lock); + matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev); + num_queues =vfio_ap_mdev_get_num_queues (&matrix_mdev->shadow_apcb); + qinfo_sz = num_queues *sizeof (struct vfio_ap_queue_info ); + *stop_copy_length = qinfo_sz +sizeof (struct vfio_ap_config ); +mutex_unlock (&matrix_dev->mdevs_lock); + +return 0 ; }static const struct vfio_migration_ops vfio_ap_migration_ops = {
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v4 13/15] s390/vfio-ap: Callback to get the size of data to be migrated during guest migration
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] vfio_ap_resuming_write leaks memory and fails subsequent writes if userspace performs a partial…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] vfio_ap_resuming_write leaks memory and fails subsequent writes if userspace performs a partial write during migration resuming.
- [High] vfio_ap_get_config does not initialize the config_sz field of the payload, sending 0 to the migration target. --
--- Patch [13]: [PATCH v4 13/15] s390/vfio-ap: Callback to get the size of data to be migrated during guest migration --- commit cf31fd21666e32de812c2582ed7358837c47d2bd
Author: Anthony Krowiak <[email protected]>
s390/vfio-ap: Callback to get the size of data to be migrated during guest migration
This commit adds a callback to get the estimated data length required to complete reading of the vfio device's internal state. This is mandatory for VFIO_DEVICE_FEATURE_MIGRATION support.
Quoted 14 lines
drivers/s390/crypto/vfio_ap_migration.c +5
-1@@ -1283,7 +1283,18 @@ static int vfio_ap_get_state(struct vfio_device *vdev,static int vfio_ap_get_data_size (struct vfio_device *vdev,unsigned long *stop_copy_length) { -return -EOPNOTSUPP ; +struct ap_matrix_mdev *matrix_mdev; +size_t qinfo_sz; +int num_queues; + +mutex_lock (&matrix_dev->mdevs_lock);
drivers/s390/crypto/vfio_ap_migration.c static int vfio_ap_get_state(struct vfio_device *vdev, · line 1300
Quoted 10 lines
drivers/s390/crypto/vfio_ap_migration.c +7
-0+ matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev); + num_queues =vfio_ap_mdev_get_num_queues (&matrix_mdev->shadow_apcb); + qinfo_sz = num_queues *sizeof (struct vfio_ap_queue_info ); + *stop_copy_length = qinfo_sz +sizeof (struct vfio_ap_config ); +mutex_unlock (&matrix_dev->mdevs_lock); + +return 0 ; }static const struct vfio_migration_ops vfio_ap_migration_ops = {
[Severity: High]
Does vfio_ap_resuming_write() leak the newly allocated ap_config and fail subsequent writes if userspace performs a partial write during migration resuming?
If a partial write occurs during vfio_ap_resuming_write() (i.e. *pos == 0 but len < cfg_sz), the newly allocated ap_config is not saved to mig_data->resuming_mig_state.ap_config because the check
if (*pos + len == cfg_sz) is false.
The function then bypasses the cleanup label:
drivers/s390/crypto/vfio_ap_migration.c:vfio_ap_resuming_write() {
...
ret = len; *pos += len; goto done;
cleanup:
kfree(ap_config);
done:
...
}
Because the pointer was never saved in the first partial write, the subsequent partial write fails this validation, returning -EFAULT, which permanently fails the migration:
drivers/s390/crypto/vfio_ap_migration.c:validate_resuming_write_parms() {
...
if ((!matrix_mdev->mig_data->resuming_mig_state.ap_config && *pos != 0) ||
(matrix_mdev->mig_data->resuming_mig_state.ap_config && *pos == 0)) {
ret = -EFAULT;...
}
[Severity: High]
Can vfio_ap_get_config() send an uninitialized config_sz field to the migration target, causing the target to fail?
In vfio_ap_get_config(), a new struct vfio_ap_config is allocated and populated, but its config_sz field is never initialized and remains 0. This structure is sent verbatim over the migration stream to the target host:
drivers/s390/crypto/vfio_ap_migration.c:vfio_ap_get_config() {
...
ap_configuration = kzalloc(*ap_config_size, GFP_KERNEL_ACCOUNT); if (!ap_configuration) return -ENOMEM; ap_configuration->num_queues = num_queues; ret = vfio_ap_store_queue_info(matrix_mdev, ap_configuration);
...
}
If the target host receives a partial write and calls reallocate_ap_config(), it reads cur_cfg_sz = ap_config->config_sz, which is 0.
This causes the target to take the wrong reallocation path, shrinking the buffer incorrectly:
drivers/s390/crypto/vfio_ap_migration.c:reallocate_ap_config() {
...
cur_cfg_sz = ap_config->config_sz;
num_queues = ap_config->num_queues;
/* ... */
if (cur_cfg_sz >= sizeof(struct vfio_ap_config)) {
new_cfg_sz = calculate_ap_config_size(num_queues);
// ...
} else {
new_cfg_sz = cur_cfg_sz + len;
}...
}
The validation if (*pos + len > cfg_sz) then fails because cfg_sz evaluates to len and *pos > 0, causing the buffer to be freed and migration to fail with -EIO.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
Anthony Krowiak
maintainerPATCH v4 14/15 s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute
In reply to Anthony KrowiakAdd the 'migratable' feature to the features provided by the sysfs /sys/devices/vfio_ap/matrix/feature attribute to indicate that migration of vfio devices is supported.
--- drivers/s390/crypto/vfio_ap_drv.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/s390/crypto/vfio_ap_drv.c +3 -1
@@ -26,9 +26,11 @@ MODULE_LICENSE("GPL v2");struct ap_matrix_dev *matrix_dev;debug_info_t *vfio_ap_dbf_info; +#define FEATURES "guest_matrix hotplug ap_config migratable" +static ssize_t features_show (struct device *dev,struct device_attribute *attr,char *buf) { -return sysfs_emit (buf,"guest_matrix hotplug ap_config\n" ); +return sysfs_emit (buf,"%s\n" ,FEATURES ); }static DEVICE_ATTR_RO (features);
Signature · 1 line
2.53.0
Anthony Krowiak
maintainerPATCH v4 15/15 s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst
In reply to Anthony KrowiakAdd a 'Live Guest Migration' chapter to the Documentation/arch/s390/vfio-ap.rst document to describe the details for initiating live guest migration for a guest to which AP adapters, domains and control domains have been passed through.
--- Documentation/arch/s390/vfio-ap.rst | 514 +++++++++++++++++++++++----- 1 file changed, 433 insertions(+), 81 deletions(-)
Documentation/arch/s390/vfio-ap.rst +433 -81
@@ -1012,7 +1012,7 @@ guest_matrix dyn ap_config the following features are advertised: ----------------+---------------------------------------------------------------+ ++--------------+---------------------------------------------------------------+ | Flag | Description | +==============+===============================================================+ | guest_matrix | guest_matrix attribute exists. It reports the matrix of |
@@ -1021,105 +1021,457 @@ the following features are advertised: +--------------+---------------------------------------------------------------+ | dyn | Indicates hot plug/unplug of AP adapters, domains and control | | | domains for a guest to which the mdev is attached. | -+------------+-----------------------------------------------------------------+ ++--------------+---------------------------------------------------------------+ | ap_config | ap_config interface for one-shot modifications to mdev config | +--------------+---------------------------------------------------------------+ +| migratable | Indicates that live guest migration is supported for guests | +| | to which crypto devices are passed through | ++--------------+---------------------------------------------------------------+ -Limitations -=========== -Live guest migration is not supported for guests using AP devices without -intervention by a system administrator. Before a KVM guest can be migrated, -the vfio_ap mediated device must be removed. Unfortunately, it can not be -removed manually (i.e., echo 1 > /sys/devices/vfio_ap/matrix/$UUID/remove) while -the mdev is in use by a KVM guest. If the guest is being emulated by QEMU, -its mdev can be hot unplugged from the guest in one of two ways: - -1. If the KVM guest was started with libvirt, you can hot unplug the mdev via - the following commands: - - virsh detach-device <guestname> <path-to-device-xml> - - For example, to hot unplug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 from - the guest named 'my-guest': - - virsh detach-device my-guest ~/config/my-guest-hostdev.xml - - The contents of my-guest-hostdev.xml: - -.. code-block:: xml - - <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'> - <source> - <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/> - </source> - </hostdev> - - - virsh qemu-monitor-command <guest-name> --hmp "device-del <device-id>" - - For example, to hot unplug the vfio_ap mediated device identified on the - qemu command line with 'id=hostdev0' from the guest named 'my-guest': - -.. code-block:: sh - - virsh qemu-monitor-command my-guest --hmp "device_del hostdev0" - -2. A vfio_ap mediated device can be hot unplugged by attaching the qemu monitor - to the guest and using the following qemu monitor command: +Live Guest Migration +==================== +The VFIO AP mediated device is not used to provide userspace with direct +access to a device as is the case with other devices that use the VFIO +framework for device pass-through. Instead, it manages AP +configuration metadata identifying the adapters, domains, and control +domains to which a guest will be granted access. These AP resources are +configured by assigning them to a vfio-ap mediated device via its sysfs +assignment interfaces. When the guest is started, the vfio_ap device driver +sets the guest's AP configuration from the metadata stored with the mediated +device. The AP devices are not accessed directly through the vfio_ap driver, +so the driver does not migrate the state of the AP devices themselves. +Instead, it migrates the AP configuration metadata. The vfio_ap device +driver on the destination host will then ensure that the AP configuration +metadata migrated from the source host is compatible with the AP +configuration available on the destination host and if +so, will set the destination guest's AP configuration accordingly. - (QEMU) device-del id=<device-id> +To be considered compatible, the AP configuration for the destination host must +meet these requirements: - For example, to hot unplug the vfio_ap mediated device that was specified - on the qemu command line with 'id=hostdev0' when the guest was started: +* Each AP adapter, domain and control domain number assigned to the source + guest's AP configuration must be in the destination host's AP + configuration. This can be verified as follows: - (QEMU) device-del id=hostdev0 + * On the source host, display the AP configuration of the source guest: -After live migration of the KVM guest completes, an AP configuration can be -restored to the KVM guest by hot plugging a vfio_ap mediated device on the target -system into the guest in one of two ways: + .. code-block:: bash -1. If the KVM guest was started with libvirt, you can hot plug a matrix mediated - device into the guest via the following virsh commands: + cat /sys/devices/vfio_ap/matrix/$UUID/guest_matrix + cat /sys/devices/vfio_ap/matrix/$UUID/control_domains - virsh attach-device <guestname> <path-to-device-xml> + .. note:: + Replace ``$UUID`` with the actual UUID of your mediated device. - For example, to hot plug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 into - the guest named 'my-guest': + * Display the AP configuration of the destination host: - virsh attach-device my-guest ~/config/my-guest-hostdev.xml + .. code-block:: bash - The contents of my-guest-hostdev.xml: + lszcrypt -V + cat /sys/bus/ap/ap_control_domain_mask -.. code-block:: xml + Note that each bit in the ap_control_domain_mask corresponds to + correspond to domain numbers 0-255. If a bit is set to one, then + the control domain is available on the destination host. For + example: - <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'> - <source> - <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/> - </source> - </hostdev> + .. code-block:: bash + 0x0000000000001020000000000000000000000000000000000000000000000000 - virsh qemu-monitor-command <guest-name> --hmp \ - "device_add vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>" + This mask indicates that AP adapters 51 (0x33) and 58 (0x58) are + available on the host system. - For example, to hot plug the vfio_ap mediated device - 62177883-f1bb-47f0-914d-32a22e3a8804 into the guest named 'my-guest' with - device-id hostdev0: + * Verify the configurations: - virsh qemu-monitor-command my-guest --hmp \ - "device_add vfio-ap,\ - sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\ - id=hostdev0" + * Each APQN in the guest's sysfs ``guest_matrix`` file must match an + APQN returned from the lszcrypt command on the destination host. It + is important to note the driver to which the queue is bound which + is displayed in the ``driver`` column. Queues not bound to the vfio_ap + driver are not available to the destination guest. That, however, does + not guarantee they can be used by the guest being migrated; for example, + migration will fail if any of these APQNs are assigned to a mediated + device other than the one being migrated. -2. A vfio_ap mediated device can be hot plugged by attaching the qemu monitor - to the guest and using the following qemu monitor command: + * Each domain number from the source guest's sysfs + ``control_domains`` attribute must correspond to a bit set to one in + the destination host's ``ap_control_domain_mask``. - (qemu) device_add "vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>" +* Each APQN assigned to the source guest and destination guest must + reference a queue device with compatible hardware capabilities: - For example, to plug the vfio_ap mediated device - 62177883-f1bb-47f0-914d-32a22e3a8804 into the guest with the device-id - hostdev0: ++--------------+---------------------------------------------------------------+ +| Hardware | Description | +| Capabilities | | ++==============+===============================================================+ +| [1] | * AP special command facility (APSC) | +| Facilities | * AP 4096-bit ME PKU commands facility (AP4KM) | +| | * AP 4096-bit CRT PKU commands (AP4KC) | ++--------------+---------------------------------------------------------------+ +| [1] Mode | * CCA-mode | +| | * Accelerator-mode | +| | * XCP-mode (EP11) | ++--------------+---------------------------------------------------------------+ +| [1] AP | APXA installed | +| extended | | +| addressing | | ++--------------+---------------------------------------------------------------+ +| [1] Command | Command filtering available | +| filtering | | ++--------------+---------------------------------------------------------------+ +| [2] | * Full native card function | +| Functional | * Only stateless functions | +| capabilities | | ++--------------+---------------------------------------------------------------+ +| [3] AP type | * 10 (0x0a): CEX4 | +| | * 11 (0x0b): CEX5 | +| | * 12 (0x0c) CEX6 | +| | * 13 (0x0d) CEX7 | +| | * 14 (0x0e) CEX8 | ++--------------+---------------------------------------------------------------+ - (QEMU) device-add "vfio-ap,\ - sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\ - id=hostdev0" ++-----------------------------------------------------------------------------+ +| Table Keys | ++=============================================================================+ +| | +| [1] The hardware capability must be the same for both the source | +| and destination queue device. | +| | +| [2] The source and destination queues must either both have | +| full native card function or both have stateless functions. | +| If the functional capabilities don't match, then the source | +| queue can have stateless capabilities since full native card | +| function can handle stateless functions; otherwise, migration | +| will fail. | +| | +| [3] The AP type on the source and destination guests can | +| differ if the queue passed through to the target guest is a | +| newer model (backwards compatible) | ++--------------+--------------------------------------------------------------+ + +* To verify the hardware capabilities are compatible: + + * On both the source and destination hosts, display the hardware capabilities + for each AP adapter in the AP configuration for that host:: + + cat /sys/bus/ap/devices/card$APID/ap_functions + + .. note:: + The ``$APID`` is the two-character adapter number in hexidecimal format; + for example, card02 or card1f. The ``/sys/bus/ap/devices`` directory + also lists the APQNs of the queue devices installed in the respective + host system with the first two characters being the APID. + + * Verify the hardware capabilities for each AP adapter device on the source + and destination host are compatible. The ``ap_functions`` attribute is a + bitmask. The bits in the mask read from left to right starting with bit 0. + Each bit that is set to one indicates the corresponding hardware capability + is installed: + + * Bits 0-3 are the facilities bits. Each bit value must match for the AP + devices on the source and destination systems. The values indicate the + following: + * bit-0: APSC is installed + * bit-1: AP4KM is installed + * bit-2: AP4KC is installed + + * Bits 3-5 specify the adapter Modes. Only one of these bits will be set to + one and must be the same for both the source and destination adapter + devices: + * bit-3: CCA-mode + * bit-4: Accelerator-mode + * bit-5: XCP-mode (EP11) + + * Bit 6 specifies whether APXA is installed and must be the same for both + the source and destination adapter devices. + + * Bit 7 specifies command whether command filtering is installed + and must be the same for both the source and destination adapter devices. + + * Bits 8-15 are the classification bits. There are only two relevant + bits and only one of them will be set to one. For the source and destination + adapter devices to be considered compatible: + + * Bit 8 may be set to 1 on both hosts (full native card function) + * Bit 9 may be set to 1 on both hosts (only stateless functions available) + * Bit 9 may be set to 1 on the source host and bit 8 may be set to 1 on + the destination host + + * Bits 16-17 are the binding state bits which must be 00 + +* To verify the AP types are compatible: + + * On both the source and destination hosts, display the hardware type + for each AP adapter in the AP configuration for that host to ensure the + source and destination adapter types are the same, or the source type is + less than the destination type:: + + cat /sys/bus/ap/devices/card$APID/hwtype + + .. note:: + The ``$APID`` is the two-character adapter number in hexidecimal format; + for example, card02 or card1f. + +Setting up for live guest migration on the destination host +----------------------------------------------------------- +To set up a host as the destination for live guest migration, do the +following: + +Create a mediated device +~~~~~~~~~~~~~~~~~~~~~~~~ +Create a mediated device with the same name (UUID) as the mediated device +used to supply the AP configuration to the source guest. For example, if +the mediated device on the source guest is named +62177883-f1bb-47f0-914d-32a22e3a8804: + +.. code-block:: bash + + echo 62177883-f1bb-47f0-914d-32a22e3a8804 > \ + /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/create + +Reserve adapters and domains for the vfio_ap device driver's use +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Reserve the same adapters and domains passed through to the source guest +for the vfio_ap device driver on the destination host by clearing the +adapter numbers from the ``/sys/bus/ap/ap_mask`` and/or the domain numbers +from the ``/sys/bus/ap/aqmask`` files. + +For example, suppose adapters 0, 20 (0x14) and 22 (0x16) as well as +domains 51 (0x33) and 58 (0x3a) are passed through to the source guest. To +ensure they can be passed through to the destination guest, they must be +made available to the vfio_ap device driver. To make them available, do one +or both of the following on the destination host: + +.. code-block:: bash + + echo -0,-0x14,-22 > /sys/bus/ap/apmask + echo -0,-0x33,-0x3a > /sys/bus/ap/aqmask + +.. note:: + + * It is not necessary to reserve both the adapters and domains. + Reserving an adapter implicitly allows the vfio_ap driver to assign + each of the domains available on the system along with the reserved + adapters to a guest's AP configuration and vice versa. + + * Reserving adapters and/or domains will result in binding the queue + devices referenced by the APQNs derived from the Cartesian product of + the adapters and domains made available to the vfio_ap device driver + with one caveat; an APQN does not necessarily have to reference a + queue device installed in the system in order to reserve it for the + vfio_ap device driver; however, if the APQN does not reference a + queue device installed on the destination host, migration will fail. + + * It is not necessary to configure the destination mediated device with + adapters, domains and control domains because the source guest's AP + configuration will overlay the destination guest's during migration. + + * To verify that the queue devices are reserved for the vfio_ap + device driver, issue the ``lszcrypt -V`` command and you should + see ``vfio_ap`` in the ``driver`` column of the output. + +Live guest migration failures due to AP configuration errors +------------------------------------------------------------ +This section describes the errors that may occur during live guest migration +when the AP configuration of the source guest and destination host are not +compatible and where to look for problem determination. + +The destination host is missing the mediated device +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This error occurs when a mediated device with the same name (UUID) as the +mediated device being migrated has not been created on the destination host. + +**Source host** + ++--------------+----------------------------------------------------------------+ +| Log | Message | ++==============+================================================================+ +| Console log: | error: device not found: mediated device '$UUID' not | +| | found | ++--------------+----------------------------------------------------------------+ +| QEMU log: | initiating migration | +| | qemu-system-s390x: Sibling indicated error 1 | ++--------------+----------------------------------------------------------------+ + +.. note:: + $UUID is the UUID of the mediated device being migrated from the source host. + + +One or more queues are not bound to the vfio_ap driver on the destination host +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The queue devices passed through to the source guest are not installed in the +destination host or are not bound to its vfio_ap device driver. + +**Source host** + ++--------------+----------------------------------------------------------------+ +| Log | Message | ++==============+================================================================+ +| Console log: | error: operation failed: migration failed. Message from the | +| | source host: operation failed: job 'migration out' failed: | +| | Sibling indicated error 1. Message from the destination host: | +| | operation failed: job 'migration in' failed: load of migration | +| | failed: No such device: Failed to load vmstate version_id: 1, | +| | ret: -19 | ++--------------+----------------------------------------------------------------+ + +**Destinaton host** + ++--------------+----------------------------------------------------------------+ +| Kernel log: | vfio_ap_mdev $UUID: Queue $APID.$APQI not available to | +| | vfio_ap driver on target host | ++--------------+----------------------------------------------------------------+ +| QEMU log: | shutting down, reason=failed | +| | qemu-system-s390x: terminating on signal $SIGNAL_NUM from | +| | pid $PID $DAEMON | ++--------------+----------------------------------------------------------------+ + + +.. note:: + * $UUID is the UUID of the mediated device being migrated from the source host. + * $APID.$APQI is adapter number and queue index (domain number) comprising the + $APQN (for example, 00.0033) of the queue device that is not available. + * $SIGNAL_NUM is the signal number that caused the QEMU process to terminate. + * $PID is the process ID of the QEMU process on the destination host. + * $DAEMON is the name of the QEMU daemon process. + * This message will be logged for each queue for which this error is detected. + +Failure to get hardware information for a queue device on destination host +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The vfio_ap device driver on the destination host will execute an AP instruction +to get the hardware information for a queue device to perform queue compatibility +checks. The instruction can fail for a number of reasons in which case the +migration will be terminated. + +**Source host** + ++--------------+----------------------------------------------------------------+ +| Log | Message | ++==============+================================================================+ +| Console log: | error: operation failed: migration failed. Message from the | +| | source host: operation failed: job 'migration out' failed: | +| | Sibling indicated error 1. Message from the destination host: | +| | operation failed: job 'migration in' failed: load of migration | +| | failed: failed to get hardware information for queue device | +| | $APQN | ++--------------+----------------------------------------------------------------+ + +**Destination host** + ++--------------+----------------------------------------------------------------+ +| Log | Message | ++==============+================================================================+ +| Console log: | vfio_ap_mdev $UUID: migration failed: Failed to get hwinfo for | +| | queue $APQN on target host: rc=$RESPONSE-CODE" | ++--------------+----------------------------------------------------------------+ +| QEMU log: | shutting down, reason=failed | +| | qemu-system-s390x: terminating on signal $SIGNAL-ID from pid | +| | $PID $DAEMON | ++--------------+----------------------------------------------------------------+ + +.. note:: + * $UUID is the UUID of the mediated device being migrated from the source host. + * $APQN (for example, 00.0033) of the queue device that is not compatible. + * $RESPONSE_CODE is the response code from the AP instruction. + * $SIGNAL_NUM is the signal number of the signal that caused the QEMU process + to terminate. + * $PID is the process ID of the QEMU process on the destination host. + * $DAEMON is the name of the QEMU daemon process. + * This message will be logged for each queue for which this error is detected. + +The AP configurations of the source guest and destination host not incompatible +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The functional capabilities or facilities available on one or more of the queues +passed through to the source guest are not compatible with the queue device with +the same APQN on the destination system (see the Hardware Capabilities table above) + +**Source host** + ++--------------+----------------------------------------------------------------+ +| Log | Message | ++==============+================================================================+ +| Console log: | error: operation failed: migration failed. Message from the | +| | source host: operation failed: job 'migration out' failed: | +| | Sibling indicated error 1. Message from the destination host: | +| | operation failed: job 'migration in' failed: load of migration | +| | failed: No such device: Failed to load vmstate version_id: 1, | +| | ret: -19 | ++--------------+----------------------------------------------------------------+ +| Kernel log: | N/A | ++--------------+----------------------------------------------------------------+ +| QEMU log: | *initiating migration* | +| | | +| | *qemu-system-s390x: Sibling indicated error 1* | ++--------------+----------------------------------------------------------------+ + +**Destination host** + ++--------------+----------------------------------------------------------------+ +| Log | Message | ++==============+================================================================+ +| Console log: | N/A | ++--------------+----------------------------------------------------------------+ +| Kernel log: | *vfio_ap_mdev $UUID: Migration failed: Source and target | +| | queue ($APQN) not compatible* | +| | | +| | The message above will be followed by one or more messages | +| | enumerating the incompatible features; for example: | +| | | +| | ``vfio_ap_mdev $UUID: APSC facility installed in source queue | +| | $APQN`` | +| | ``vfio_ap_mdev $UUID: APSC facility not installed in target | +| | queue $APQN`` | +| | | +| | ``AP type of source ($APTYPE1) not compatible with target | +| | ($APTYPE2)`` ... | ++--------------+----------------------------------------------------------------+ +| QEMU log: | *initiating migration* | +| | | +| | *qemu-system-s390x: error while loading state section id ...* | +| | | +| | *shutting down, reason=failed* | +| | | +| | *terminating on signal $SIGNAL-ID from pid $PID $DAEMON* | ++--------------+----------------------------------------------------------------+ + +.. note:: + * $UUID is the UUID of the mediated device being migrated from the source host. + * $APQN (for example, 00.0033) of the queue device that is incompatible. + * $APTYPE1 is the type of the source queue device. + * $APTYPE2 is the type of the target queue device. + * $SIGNAL-ID is the signal number of the signal that caused the QEMU process + to terminate. + * $PROCESS-ID is the process ID of the QEMU process on the source host. + * $PID is the process ID of the QEMU process on the destination host. + * $DAEMON is the name of the QEMU daemon process. + * Each follow-on message logging the incompatibility will be logged for each + error detected for the specified queue device. + +AP Configuration Management +--------------------------- +The AP configurations of the source guest and destination host must be kept +in synchronization or live guest migration will likely fail due to potential +incompatibilities introduced by such changes. In particular, changing adapter +or domain reservations via the sysfs ``apmask`` or ``aqmask`` attributes on +either the source or destination host without coordinating such changes between +the two hosts -especially while a migration is taking place - can result in +migration failures. + +Configuration stability is an orchestration-layer or system administrator +responsibility, which is consistent with other VFIO device types. The vfio_ap +device driver will validate source and destination configurations and provide +clear diagnostics when incompatibilities are detected, enabling orchestration +tools to implement appropriate policies; for example, blocking migration when +such changes are to be made. + +Master Key administration +-------------------------- +It is imperative that s390 Common Cryptographic Architecture (CCA) master key +administration be performed on both the source and destination AP +devices to synchronize the key values prior to allowing live guest +migration. If the master keys do not match, then crypto applications that +rely on secure keys wrapped by a CCA master key will fail when the guest on +which they are running is migrated to the destination host. Again, this is +an orchestration-layer or system administrator responsibility and migration must +be blocked while master key administration is being performed.
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v4 15/15] s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] The newly added documentation for live guest migration contains multiple typos, grammatical issues…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] The newly added documentation for live guest migration contains multiple typos, grammatical issues, factual errors, and ReST formatting problems. --
--- Patch [15]: [PATCH v4 15/15] s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst --- commit e1db7dba8604c95334563aa2130ba1fa4ead1210
Author: Anthony Krowiak <[email protected]>
s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst
Add a 'Live Guest Migration' chapter to the Documentation/arch/s390/vfio-ap.rst document to describe the details for initiating live guest migration for a guest to which AP adapters, domains and control domains have been passed through.
Quoted 4 lines
Documentation/arch/s390/vfio-ap.rst +0
-0
[ ... ]
Quoted 8 lines
Documentation/arch/s390/vfio-ap.rst +8
-0+ * Display the AP configuration of the destination host: + + .. code-block:: bash + + lszcrypt -V + cat /sys/bus/ap/ap_control_domain_mask + + Note that each bit in the ap_control_domain_mask corresponds to
Documentation/arch/s390/vfio-ap.rst the following features are advertised: · line 1083
Quoted 10 lines
Documentation/arch/s390/vfio-ap.rst +10
-0+ correspond to domain numbers 0-255. If a bit is set to one, then + the control domain is available on the destination host. For + example: + + .. code-block:: bash + + 0x0000000000001020000000000000000000000000000000000000000000000000 + + This mask indicates that AP adapters 51 (0x33) and 58 (0x58) are + available on the host system.
[Severity: Low]
This isn't a bug, but there are a couple of inaccuracies in this sentence.
Since we are examining ap_control_domain_mask, shouldn't this refer to "AP control domains" rather than "AP adapters"?
Also, the hexadecimal value for 58 is 0x3a, not 0x58.
[ ... ]
Documentation/arch/s390/vfio-ap.rst the following features are advertised: · line 1176
Quoted 12 lines
Documentation/arch/s390/vfio-ap.rst +12
-0+ * Verify the hardware capabilities for each AP adapter device on the source + and destination host are compatible. The ``ap_functions`` attribute is a + bitmask. The bits in the mask read from left to right starting with bit 0. + Each bit that is set to one indicates the corresponding hardware capability + is installed: + + * Bits 0-3 are the facilities bits. Each bit value must match for the AP + devices on the source and destination systems. The values indicate the + following: + * bit-0: APSC is installed + * bit-1: AP4KM is installed + * bit-2: AP4KC is installed
[Severity: Low]
This isn't a bug, but there is a missing blank line before this nested list that might cause incorrect Sphinx rendering.
Should there be a blank line after "The values indicate the following:"?
Documentation/arch/s390/vfio-ap.rst
Quoted 2 lines
Documentation/arch/s390/vfio-ap.rst +2
-0+ + * Bits 3-5 specify the adapter Modes. Only one of these bits will be set to
[Severity: Low]
This isn't a bug, but there is an overlap in the bit ranges described here.
The facilities bits are described as Bits 0-3, but the adapter modes start at Bit 3.
Should the facilities bits be 0-2?
Documentation/arch/s390/vfio-ap.rst the following features are advertised: · line 1188
Quoted 10 lines
Documentation/arch/s390/vfio-ap.rst +10
-0+ one and must be the same for both the source and destination adapter + devices: + * bit-3: CCA-mode + * bit-4: Accelerator-mode + * bit-5: XCP-mode (EP11) + + * Bit 6 specifies whether APXA is installed and must be the same for both + the source and destination adapter devices. + + * Bit 7 specifies command whether command filtering is installed
[Severity: Low]
This isn't a bug, but there is a grammatical error here.
Could this be updated to remove the extra "command" ("specifies whether command filtering")?
Quoted 1 line
+ and must be the same for both the source and destination adapter devices.
[ ... ]
Documentation/arch/s390/vfio-ap.rst the following features are advertised: · line 1236
Quoted 5 lines
Documentation/arch/s390/vfio-ap.rst +5
-0+Reserve adapters and domains for the vfio_ap device driver's use +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Reserve the same adapters and domains passed through to the source guest +for the vfio_ap device driver on the destination host by clearing the +adapter numbers from the ``/sys/bus/ap/ap_mask`` and/or the domain numbers
[Severity: Low]
This isn't a bug, but the sysfs path here seems incorrect.
Should this be /sys/bus/ap/apmask instead of /sys/bus/ap/ap_mask to match the actual sysfs entry?
Quoted 1 line
+from the ``/sys/bus/ap/aqmask`` files.
[ ... ]
Quoted 1 line
+**Destinaton host**
[Severity: Low]
This isn't a bug, but there is a typo here ("Destinaton" instead of "Destination").
[ ... ]
Quoted 2 lines
+The AP configurations of the source guest and destination host not incompatible
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[Severity: Low]
This isn't a bug, but the section heading might be confusing.
Should it read "are incompatible" instead of "not incompatible"?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15