s390/vfio-ap: fix hang when mdev attached to guest is removed
Subsystems (5)
Tony Krowiak
maintainercover s390/vfio-ap: fix hang when mdev attached to guest is removed
When a user attempts to remove a vfio-ap mediated device attached to a guest, the operation hangs until the mdev's fd is closed by the guest (i.e., the hostdev is detached or the guest is shut down). This patch series provides kernel-side code that allows userspace to set up a communication channel that will allow the vfio_ap device driver to notify userspace when a request to release the mdev is received, so that userspace can close the mdev fd and avoid the hang. The patch series provides the following:
1. Introduces code to handle the VFIO_DEVICE_GET_IRQ_INFO and VFIO_DEVICE_SET_IRQS ioctl calls to set the eventfd_ctx for signaling a device request to userspace.
2. Wires up the VFIO bus driver callback to request a release of the mdev. When invoked, the vfio_ap device driver will use the eventfd_ctx set up in #1 to signal a request to userspace to release the mdev.
Note:
----
If a user subsequently attempts to restart the guest or re-attach the mdev, the operation will fail with a message indicating the domain is already active. This is a libvirt problem resolved with the following commit:
commit ebd004a03dbd ("security: do not remember/recall labels for VFIO MDEVs")
Tony Krowiak (3):
vfio: ap: realize the VFIO_DEVICE_GET_IRQ_INFO ioctl
vfio: ap: realize the VFIO_DEVICE_SET_IRQS ioctl s390/vfio-ap: Wire in the vfio_device_ops request callback
drivers/s390/crypto/vfio_ap_ops.c | 134 +++++++++++++++++++++++++- drivers/s390/crypto/vfio_ap_private.h | 3 + include/uapi/linux/vfio.h | 9 ++ 3 files changed, 145 insertions(+), 1 deletion(-)
Signature · 1 line
2.31.1
Tony Krowiak
maintainerPATCH 1/3 vfio: ap: realize the VFIO_DEVICE_GET_IRQ_INFO ioctl
Realize the VFIO_DEVICE_GET_IRQ_INFO ioctl to retrieve the information for the VFIO device request IRQ.
--- drivers/s390/crypto/vfio_ap_ops.c | 30 +++++++++++++++++++++++++++++- include/uapi/linux/vfio.h | 9 +++++++++ 2 files changed, 38 insertions(+), 1 deletion(-)
drivers/s390/crypto/vfio_ap_ops.c +29 -1
@@ -1750,7 +1750,32 @@ static int vfio_ap_mdev_get_device_info(unsigned long arg) info.flags =VFIO_DEVICE_FLAGS_AP |VFIO_DEVICE_FLAGS_RESET ; info.num_regions =0 ; - info.num_irqs =0 ; + info.num_irqs =VFIO_AP_NUM_IRQS ; + +return copy_to_user ((void __user *)arg, &info, minsz) ? -EFAULT :0 ; +} + +static ssize_t vfio_ap_get_irq_info (unsigned long arg) +{ +unsigned long minsz; +struct vfio_irq_info info; + + minsz = offsetofend(struct vfio_irq_info, count); + +if (copy_from_user (&info, (void __user *)arg, minsz)) +return -EFAULT ; + +if (info.argsz < minsz || info.index >=VFIO_AP_NUM_IRQS ) +return -EINVAL ; + +switch (info.index) { +case VFIO_AP_REQ_IRQ_INDEX : + info.count =1 ; + info.flags =VFIO_IRQ_INFO_EVENTFD ; +break ; +default : +return -EINVAL ; + }return copy_to_user ((void __user *)arg, &info, minsz) ? -EFAULT :0 ; }
@@ -1770,6 +1795,9 @@ static ssize_t vfio_ap_mdev_ioctl(struct vfio_device *vdev,case VFIO_DEVICE_RESET : ret =vfio_ap_mdev_reset_queues (&matrix_mdev->qtable);break ; +case VFIO_DEVICE_GET_IRQ_INFO : + ret =vfio_ap_get_irq_info (arg); +break ;default : ret = -EOPNOTSUPP ;break ;
include/uapi/linux/vfio.h +9 -0
@@ -646,6 +646,15 @@ enum { VFIO_CCW_NUM_IRQS }; +/* +* The vfio-ap bus driver makes use of the following IRQ index mapping. +* Unimplemented IRQ types return a count of zero. +*/ +enum { +VFIO_AP_REQ_IRQ_INDEX , +VFIO_AP_NUM_IRQS +}; +/** * VFIO_DEVICE_GET_PCI_HOT_RESET_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12, * struct vfio_pci_hot_reset_info)
Signature · 1 line
2.31.1
Re: [PATCH 1/3] vfio: ap: realize the VFIO_DEVICE_GET_IRQ_INFO ioctl
Quoted 4 lines · Tony Krowiak
Realize the
VFIO_DEVICE_GET_IRQ_INFOioctl to retrieve the information for the VFIO device request IRQ.Signed-off-by Tony Krowiak <[email protected]>
Thanks,
C.
Quoted 73 lines
--- drivers/s390/crypto/vfio_ap_ops.c | 30 +++++++++++++++++++++++++++++- include/uapi/linux/vfio.h | 9 +++++++++ 2 files changed, 38 insertions(+), 1 deletion(-)drivers/s390/crypto/vfio_ap_ops.c +29
-1@@ -1750,7 +1750,32 @@ static int vfio_ap_mdev_get_device_info(unsigned long arg) info.flags =VFIO_DEVICE_FLAGS_AP |VFIO_DEVICE_FLAGS_RESET ; info.num_regions =0 ; - info.num_irqs =0 ; + info.num_irqs =VFIO_AP_NUM_IRQS ; + +return copy_to_user ((void __user *)arg, &info, minsz) ? -EFAULT :0 ; +} + +static ssize_t vfio_ap_get_irq_info (unsigned long arg) +{ +unsigned long minsz; +struct vfio_irq_info info; + + minsz = offsetofend(struct vfio_irq_info, count); + +if (copy_from_user (&info, (void __user *)arg, minsz)) +return -EFAULT ; + +if (info.argsz < minsz || info.index >=VFIO_AP_NUM_IRQS ) +return -EINVAL ; + +switch (info.index) { +case VFIO_AP_REQ_IRQ_INDEX : + info.count =1 ; + info.flags =VFIO_IRQ_INFO_EVENTFD ; +break ; +default : +return -EINVAL ; + }return copy_to_user ((void __user *)arg, &info, minsz) ? -EFAULT :0 ; }@@ -1770,6 +1795,9 @@ static ssize_t vfio_ap_mdev_ioctl(struct vfio_device *vdev,case VFIO_DEVICE_RESET : ret =vfio_ap_mdev_reset_queues (&matrix_mdev->qtable);break ; +case VFIO_DEVICE_GET_IRQ_INFO : + ret =vfio_ap_get_irq_info (arg); +break ;default : ret = -EOPNOTSUPP ;break ;include/uapi/linux/vfio.h +9
-0@@ -646,6 +646,15 @@ enum { VFIO_CCW_NUM_IRQS }; +/* +* The vfio-ap bus driver makes use of the following IRQ index mapping. +* Unimplemented IRQ types return a count of zero. +*/ +enum { +VFIO_AP_REQ_IRQ_INDEX , +VFIO_AP_NUM_IRQS +}; +/** * VFIO_DEVICE_GET_PCI_HOT_RESET_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12, * struct vfio_pci_hot_reset_info)
Anthony Krowiak
maintainerRe: [PATCH 1/3] vfio: ap: realize the VFIO_DEVICE_GET_IRQ_INFO ioctl
Quoted 1 line · Cédric Le Goater
Reviewed-by Cédric Le Goater <[email protected]>
Thank you for the review.
Re: [PATCH 1/3] vfio: ap: realize the VFIO_DEVICE_GET_IRQ_INFO ioctl
Quoted 7 lines · Anthony Krowiak
Quoted 1 line · Cédric Le Goater
Reviewed-by Cédric Le Goater <[email protected]>Thank you for the review.
I also ran a few tests with the QEMU part on guests with passthrough crypto devices. This is probably a v6.5 candidate.
C.
Re: [PATCH 1/3] vfio: ap: realize the VFIO_DEVICE_GET_IRQ_INFO ioctl
In reply to Tony KrowiakQuoted 8 lines · On Tue, 30 May 2023 18:35:36 -0400 Tony Krowiak <[email protected]> wrote:
Realize the
VFIO_DEVICE_GET_IRQ_INFOioctl to retrieve the information for the VFIO device request IRQ.Signed-off-by Tony Krowiak <[email protected]>--- drivers/s390/crypto/vfio_ap_ops.c | 30 +++++++++++++++++++++++++++++- include/uapi/linux/vfio.h | 9 +++++++++ 2 files changed, 38 insertions(+), 1 deletion(-)
Tony Krowiak
maintainerPATCH 2/3 vfio: ap: realize the VFIO_DEVICE_SET_IRQS ioctl
In reply to Tony KrowiakRealize the VFIO_DEVICE_SET_IRQS ioctl to set an eventfd file descriptor to be used by the vfio_ap device driver to signal a device request to userspace.
--- drivers/s390/crypto/vfio_ap_ops.c | 83 +++++++++++++++++++++++++++ drivers/s390/crypto/vfio_ap_private.h | 3 + 2 files changed, 86 insertions(+)
drivers/s390/crypto/vfio_ap_ops.c +83 -0
@@ -716,6 +716,7 @@ static int vfio_ap_mdev_probe(struct mdev_device *mdev) ret =vfio_register_emulated_iommu_dev (&matrix_mdev->vdev);if (ret) goto err_put_vdev; + matrix_mdev->req_trigger =NULL ;dev_set_drvdata (&mdev->dev, matrix_mdev);mutex_lock (&matrix_dev->mdevs_lock);list_add (&matrix_mdev->node, &matrix_dev->mdev_list);
@@ -1780,6 +1781,85 @@ static ssize_t vfio_ap_get_irq_info(unsigned long arg)return copy_to_user ((void __user *)arg, &info, minsz) ? -EFAULT :0 ; } +static int vfio_ap_irq_set_init (struct vfio_irq_set *irq_set,unsigned long arg) +{ +int ret; +size_t data_size; +unsigned long minsz; + + minsz = offsetofend(struct vfio_irq_set, count); + +if (copy_from_user (irq_set, (void __user *)arg, minsz)) +return -EFAULT ; + + ret =vfio_set_irqs_validate_and_prepare (irq_set,1 ,VFIO_AP_NUM_IRQS , + &data_size); +if (ret) +return ret; + +if (!(irq_set->flags &VFIO_IRQ_SET_ACTION_TRIGGER )) +return -EINVAL ; + +return 0 ; +} + +static int vfio_ap_set_request_irq (struct ap_matrix_mdev *matrix_mdev, +unsigned long arg) +{ +s32 fd; +void __user *data; +unsigned long minsz; +struct eventfd_ctx *req_trigger; + + minsz = offsetofend(struct vfio_irq_set, count); + data = (void __user *)(arg + minsz); + +if (get_user (fd, (s32 __user *)data)) +return -EFAULT ; + +if (fd ==-1 ) { +if (matrix_mdev->req_trigger) +eventfd_ctx_put (matrix_mdev->req_trigger); + matrix_mdev->req_trigger =NULL ; + }else if (fd >=0 ) { + req_trigger =eventfd_ctx_fdget (fd); +if (IS_ERR (req_trigger)) +return PTR_ERR (req_trigger); + +if (matrix_mdev->req_trigger) +eventfd_ctx_put (matrix_mdev->req_trigger); + + matrix_mdev->req_trigger = req_trigger; + }else { +return -EINVAL ; + } + +return 0 ; +} + +static int vfio_ap_set_irqs (struct ap_matrix_mdev *matrix_mdev, +unsigned long arg) +{ +int ret; +struct vfio_irq_set irq_set; + + ret =vfio_ap_irq_set_init (&irq_set, arg); +if (ret) +return ret; + +switch (irq_set.flags &VFIO_IRQ_SET_DATA_TYPE_MASK ) { +case VFIO_IRQ_SET_DATA_EVENTFD : +switch (irq_set.index) { +case VFIO_AP_REQ_IRQ_INDEX : +return vfio_ap_set_request_irq (matrix_mdev, arg); +default : +return -EINVAL ; + } +default : +return -EINVAL ; + } +} +static ssize_t vfio_ap_mdev_ioctl (struct vfio_device *vdev,unsigned int cmd,unsigned long arg) {
@@ -1798,6 +1878,9 @@ static ssize_t vfio_ap_mdev_ioctl(struct vfio_device *vdev,case VFIO_DEVICE_GET_IRQ_INFO : ret =vfio_ap_get_irq_info (arg);break ; +case VFIO_DEVICE_SET_IRQS : + ret =vfio_ap_set_irqs (matrix_mdev, arg); +break ;default : ret = -EOPNOTSUPP ;break ;
drivers/s390/crypto/vfio_ap_private.h +3 -0
@@ -15,6 +15,7 @@#include <linux/types.h> #include <linux/mdev.h> #include <linux/delay.h> +#include <linux/eventfd.h> #include <linux/mutex.h> #include <linux/kvm_host.h> #include <linux/vfio.h>
@@ -103,6 +104,7 @@ struct ap_queue_table {* PQAP(AQIC) instruction. * @mdev: the mediated device * @qtable: table of queues (struct vfio_ap_queue) assigned to the mdev +* @req_trigger eventfd ctx for signaling userspace to return a device * @apm_add: bitmap of APIDs added to the host's AP configuration * @aqm_add: bitmap of APQIs added to the host's AP configuration * @adm_add: bitmap of control domain numbers added to the host's AP
@@ -117,6 +119,7 @@ struct ap_matrix_mdev {crypto_hook pqap_hook;struct mdev_device *mdev;struct ap_queue_table qtable; +struct eventfd_ctx *req_trigger;DECLARE_BITMAP (apm_add,AP_DEVICES );DECLARE_BITMAP (aqm_add,AP_DOMAINS );DECLARE_BITMAP (adm_add,AP_DOMAINS );
Signature · 1 line
2.31.1
Re: [PATCH 2/3] vfio: ap: realize the VFIO_DEVICE_SET_IRQS ioctl
Quoted 5 lines · Tony Krowiak
Realize the
VFIO_DEVICE_SET_IRQSioctl to set an eventfd file descriptor to be used by thevfio_apdevice driver to signal a device request to userspace.Signed-off-by Tony Krowiak <[email protected]>
Thanks,
C.
Quoted 141 lines
--- drivers/s390/crypto/vfio_ap_ops.c | 83 +++++++++++++++++++++++++++ drivers/s390/crypto/vfio_ap_private.h | 3 + 2 files changed, 86 insertions(+)drivers/s390/crypto/vfio_ap_ops.c +83
-0@@ -716,6 +716,7 @@ static int vfio_ap_mdev_probe(struct mdev_device *mdev) ret =vfio_register_emulated_iommu_dev (&matrix_mdev->vdev);if (ret) goto err_put_vdev; + matrix_mdev->req_trigger =NULL ;dev_set_drvdata (&mdev->dev, matrix_mdev);mutex_lock (&matrix_dev->mdevs_lock);list_add (&matrix_mdev->node, &matrix_dev->mdev_list);@@ -1780,6 +1781,85 @@ static ssize_t vfio_ap_get_irq_info(unsigned long arg)return copy_to_user ((void __user *)arg, &info, minsz) ? -EFAULT :0 ; } +static int vfio_ap_irq_set_init (struct vfio_irq_set *irq_set,unsigned long arg) +{ +int ret; +size_t data_size; +unsigned long minsz; + + minsz = offsetofend(struct vfio_irq_set, count); + +if (copy_from_user (irq_set, (void __user *)arg, minsz)) +return -EFAULT ; + + ret =vfio_set_irqs_validate_and_prepare (irq_set,1 ,VFIO_AP_NUM_IRQS , + &data_size); +if (ret) +return ret; + +if (!(irq_set->flags &VFIO_IRQ_SET_ACTION_TRIGGER )) +return -EINVAL ; + +return 0 ; +} + +static int vfio_ap_set_request_irq (struct ap_matrix_mdev *matrix_mdev, +unsigned long arg) +{ +s32 fd; +void __user *data; +unsigned long minsz; +struct eventfd_ctx *req_trigger; + + minsz = offsetofend(struct vfio_irq_set, count); + data = (void __user *)(arg + minsz); + +if (get_user (fd, (s32 __user *)data)) +return -EFAULT ; + +if (fd ==-1 ) { +if (matrix_mdev->req_trigger) +eventfd_ctx_put (matrix_mdev->req_trigger); + matrix_mdev->req_trigger =NULL ; + }else if (fd >=0 ) { + req_trigger =eventfd_ctx_fdget (fd); +if (IS_ERR (req_trigger)) +return PTR_ERR (req_trigger); + +if (matrix_mdev->req_trigger) +eventfd_ctx_put (matrix_mdev->req_trigger); + + matrix_mdev->req_trigger = req_trigger; + }else { +return -EINVAL ; + } + +return 0 ; +} + +static int vfio_ap_set_irqs (struct ap_matrix_mdev *matrix_mdev, +unsigned long arg) +{ +int ret; +struct vfio_irq_set irq_set; + + ret =vfio_ap_irq_set_init (&irq_set, arg); +if (ret) +return ret; + +switch (irq_set.flags &VFIO_IRQ_SET_DATA_TYPE_MASK ) { +case VFIO_IRQ_SET_DATA_EVENTFD : +switch (irq_set.index) { +case VFIO_AP_REQ_IRQ_INDEX : +return vfio_ap_set_request_irq (matrix_mdev, arg); +default : +return -EINVAL ; + } +default : +return -EINVAL ; + } +} +static ssize_t vfio_ap_mdev_ioctl (struct vfio_device *vdev,unsigned int cmd,unsigned long arg) {@@ -1798,6 +1878,9 @@ static ssize_t vfio_ap_mdev_ioctl(struct vfio_device *vdev,case VFIO_DEVICE_GET_IRQ_INFO : ret =vfio_ap_get_irq_info (arg);break ; +case VFIO_DEVICE_SET_IRQS : + ret =vfio_ap_set_irqs (matrix_mdev, arg); +break ;default : ret = -EOPNOTSUPP ;break ;drivers/s390/crypto/vfio_ap_private.h +3
-0@@ -15,6 +15,7 @@#include <linux/types.h> #include <linux/mdev.h> #include <linux/delay.h> +#include <linux/eventfd.h> #include <linux/mutex.h> #include <linux/kvm_host.h> #include <linux/vfio.h> @@ -103,6 +104,7 @@ struct ap_queue_table {* PQAP(AQIC) instruction. * @mdev: the mediated device * @qtable: table of queues (struct vfio_ap_queue) assigned to the mdev +* @req_trigger eventfd ctx for signaling userspace to return a device * @apm_add: bitmap of APIDs added to the host's AP configuration * @aqm_add: bitmap of APQIs added to the host's AP configuration * @adm_add: bitmap of control domain numbers added to the host's AP @@ -117,6 +119,7 @@ struct ap_matrix_mdev {crypto_hook pqap_hook;struct mdev_device *mdev;struct ap_queue_table qtable; +struct eventfd_ctx *req_trigger;DECLARE_BITMAP (apm_add,AP_DEVICES );DECLARE_BITMAP (aqm_add,AP_DOMAINS );DECLARE_BITMAP (adm_add,AP_DOMAINS );
Tony Krowiak
maintainerPATCH 3/3 s390/vfio-ap: Wire in the vfio_device_ops request callback
In reply to Tony KrowiakThe mdev device is being removed, so pass the request to userspace to ask for a graceful cleanup. This should free up the thread that would otherwise loop waiting for the device to be fully released.
--- drivers/s390/crypto/vfio_ap_ops.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+)
drivers/s390/crypto/vfio_ap_ops.c +21 -0
@@ -1736,6 +1736,26 @@ static void vfio_ap_mdev_close_device(struct vfio_device *vdev)vfio_ap_mdev_unset_kvm (matrix_mdev); } +static void vfio_ap_mdev_request (struct vfio_device *vdev,unsigned int count) +{ +struct device *dev = vdev->dev; +struct ap_matrix_mdev *matrix_mdev; + + matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev); + +if (matrix_mdev->req_trigger) { +if (!(count %10 )) +dev_notice_ratelimited (dev, +"Relaying device request to user (#%u)\n" , + count); + +eventfd_signal (matrix_mdev->req_trigger,1 ); + }else if (count ==0 ) { +dev_notice (dev, +"No device request registered, blocked until released by user\n" ); + } +} +static int vfio_ap_mdev_get_device_info (unsigned long arg) {unsigned long minsz;
@@ -1955,6 +1975,7 @@ static const struct vfio_device_ops vfio_ap_matrix_dev_ops = { .bind_iommufd = vfio_iommufd_emulated_bind, .unbind_iommufd = vfio_iommufd_emulated_unbind, .attach_ioas = vfio_iommufd_emulated_attach_ioas, + .request = vfio_ap_mdev_request };static struct mdev_driver vfio_ap_matrix_driver = {
Signature · 1 line
2.31.1
Re: [PATCH 3/3] s390/vfio-ap: Wire in the vfio_device_ops request callback
Quoted 5 lines · Tony Krowiak
The mdev device is being removed, so pass the request to userspace to ask for a graceful cleanup. This should free up the thread that would otherwise loop waiting for the device to be fully released.
Signed-off-by Tony Krowiak <[email protected]>
Thanks,
C.
Quoted 43 lines
--- drivers/s390/crypto/vfio_ap_ops.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+)drivers/s390/crypto/vfio_ap_ops.c +21
-0@@ -1736,6 +1736,26 @@ static void vfio_ap_mdev_close_device(struct vfio_device *vdev)vfio_ap_mdev_unset_kvm (matrix_mdev); } +static void vfio_ap_mdev_request (struct vfio_device *vdev,unsigned int count) +{ +struct device *dev = vdev->dev; +struct ap_matrix_mdev *matrix_mdev; + + matrix_mdev = container_of(vdev, struct ap_matrix_mdev, vdev); + +if (matrix_mdev->req_trigger) { +if (!(count %10 )) +dev_notice_ratelimited (dev, +"Relaying device request to user (#%u)\n" , + count); + +eventfd_signal (matrix_mdev->req_trigger,1 ); + }else if (count ==0 ) { +dev_notice (dev, +"No device request registered, blocked until released by user\n" ); + } +} +static int vfio_ap_mdev_get_device_info (unsigned long arg) {unsigned long minsz;@@ -1955,6 +1975,7 @@ static const struct vfio_device_ops vfio_ap_matrix_dev_ops = { .bind_iommufd = vfio_iommufd_emulated_bind, .unbind_iommufd = vfio_iommufd_emulated_unbind, .attach_ioas = vfio_iommufd_emulated_attach_ioas, + .request = vfio_ap_mdev_request };static struct mdev_driver vfio_ap_matrix_driver = {
Quoted 18 lines · Tony Krowiak
When a user attempts to remove a vfio-ap mediated device attached to a guest, the operation hangs until the mdev's fd is closed by the guest (i.e., the hostdev is detached or the guest is shut down). This patch series provides kernel-side code that allows userspace to set up a communication channel that will allow the
vfio_apdevice driver to notify userspace when a request to release the mdev is received, so that userspace can close the mdev fd and avoid the hang. The patch series provides the following:1. Introduces code to handle the
VFIO_DEVICE_GET_IRQ_INFOandVFIO_DEVICE_SET_IRQSioctl calls to set theeventfd_ctxfor signaling a device request to userspace.2. Wires up the VFIO bus driver callback to request a release of the mdev. When invoked, the
vfio_apdevice driver will use theeventfd_ctxset up in #1 to signal a request to userspace to release the mdev.
Quoted 8 lines
Note:
----
If a user subsequently attempts to restart the guest or re-attach the mdev, the operation will fail with a message indicating the domain is already active. This is a libvirt problem resolved with the following commit:commit
ebd004a03dbd("security: do not remember/recall labels for VFIO MDEVs")
For the series:
I also did some testing using the companion qemu series at
https://lore.kernel.org/qemu-devel/[email protected]
Before kernel+qemu changes:
1. mdevctl start -u <uuid>, where <uuid> references a vfio-ap mdev
2. start a qemu guest with <uuid> attached
3. mdvectl stop -u <uuid>
4. -mdevctl will now hang indefinitely; the mdev remains in-use by the guest-
Note: detaching the device or powering off the guest will allow the mdevctl command to complete.
After kernel+qemu changes:
1. mdevctl start -u <uuid>, where <uuid> references a vfio-ap mdev
2. start a qemu guest with <uuid> attached
3. mdvectl stop -u <uuid>
4. -device is detached from the guest and stopped-
5. Using a libvirt that includes ebd004a03dbd I also verified that the mdev can be started again and re-attached to the running guest without error.
Quoted 11 lines
Tony Krowiak (3):
vfio: ap: realize theVFIO_DEVICE_GET_IRQ_INFOioctl
vfio: ap: realize theVFIO_DEVICE_SET_IRQSioctl s390/vfio-ap: Wire in thevfio_device_opsrequest callbackdrivers/s390/crypto/vfio_ap_ops.c | 134 +++++++++++++++++++++++++- drivers/s390/crypto/vfio_ap_private.h | 3 + include/uapi/linux/vfio.h | 9 ++ 3 files changed, 145 insertions(+), 1 deletion(-)
Anthony Krowiak
maintainerQuoted 25 lines · Matthew Rosato
Quoted 24 lines · Tony Krowiak
When a user attempts to remove a vfio-ap mediated device attached to a guest, the operation hangs until the mdev's fd is closed by the guest (i.e., the hostdev is detached or the guest is shut down). This patch series provides kernel-side code that allows userspace to set up a communication channel that will allow the
vfio_apdevice driver to notify userspace when a request to release the mdev is received, so that userspace can close the mdev fd and avoid the hang. The patch series provides the following:1. Introduces code to handle the
VFIO_DEVICE_GET_IRQ_INFOandVFIO_DEVICE_SET_IRQSioctl calls to set theeventfd_ctxfor signaling a device request to userspace.2. Wires up the VFIO bus driver callback to request a release of the mdev. When invoked, the
vfio_apdevice driver will use theeventfd_ctxset up in #1 to signal a request to userspace to release the mdev.Note:
----
If a user subsequently attempts to restart the guest or re-attach the mdev, the operation will fail with a message indicating the domain is already active. This is a libvirt problem resolved with the following commit:
Quoted 6 lines
Quoted 2 lines
commit
ebd004a03dbd("security: do not remember/recall labels for VFIO MDEVs")For the series:
Reviewed-by Matthew Rosato <[email protected]>
Thanks for the review.
Quoted 4 lines
I also did some testing using the companion qemu series at
https://lore.kernel.org/qemu-devel/[email protected]
Shall I credit you with Tested-by also?
Quoted 28 lines
Before kernel+qemu changes:
1. mdevctl start -u <uuid>, where <uuid> references a vfio-ap mdev
2. start a qemu guest with <uuid> attached
3. mdvectl stop -u <uuid>
4. -mdevctl will now hang indefinitely; the mdev remains in-use by the guest-
Note: detaching the device or powering off the guest will allow the mdevctl command to complete.After kernel+qemu changes:
1. mdevctl start -u <uuid>, where <uuid> references a vfio-ap mdev
2. start a qemu guest with <uuid> attached
3. mdvectl stop -u <uuid>
4. -device is detached from the guest and stopped-
5. Using a libvirt that includesebd004a03dbdI also verified that the mdev can be started again and re-attached to the running guest without error.Quoted 11 lines
Tony Krowiak (3):
vfio: ap: realize theVFIO_DEVICE_GET_IRQ_INFOioctl
vfio: ap: realize theVFIO_DEVICE_SET_IRQSioctl s390/vfio-ap: Wire in thevfio_device_opsrequest callbackdrivers/s390/crypto/vfio_ap_ops.c | 134 +++++++++++++++++++++++++- drivers/s390/crypto/vfio_ap_private.h | 3 + include/uapi/linux/vfio.h | 9 ++ 3 files changed, 145 insertions(+), 1 deletion(-)
Quoted 9 lines · Anthony Krowiak
Quoted 2 lines · Matthew Rosato
I also did some testing using the companion qemu series at
https://lore.kernel.org/qemu-devel/[email protected]Shall I credit you with Tested-by also?
Sure.
Thanks,
Matt
Quoted 9 lines · Tony Krowiak
When a user attempts to remove a vfio-ap mediated device attached to a guest, the operation hangs until the mdev's fd is closed by the guest (i.e., the hostdev is detached or the guest is shut down). This patch series provides kernel-side code that allows userspace to set up a communication channel that will allow the
vfio_apdevice driver to notify userspace when a request to release the mdev is received, so that userspace can close the mdev fd and avoid the hang. The patch series provides the following:
Quoted 8 lines
1. Introduces code to handle the
VFIO_DEVICE_GET_IRQ_INFOandVFIO_DEVICE_SET_IRQSioctl calls to set theeventfd_ctxfor signaling a device request to userspace.2. Wires up the VFIO bus driver callback to request a release of the mdev. When invoked, the
vfio_apdevice driver will use theeventfd_ctxset up in #1 to signal a request to userspace to release the mdev.
As to how this series eventually reaches master... It touches both s390 and vfio.
@Alex/@s390 maintainers -- I suggest it go through s390 given the diffstat, it's almost completely in s390 drivers code. However there is a uapi hit to vfio.h (in patch 1) that should get at least an ACK from Alex beforehand.
Quoted 19 lines · On Wed, 31 May 2023 10:51:54 -0400 Matthew Rosato <[email protected]>…
Quoted 9 lines · Tony Krowiak
When a user attempts to remove a vfio-ap mediated device attached to a guest, the operation hangs until the mdev's fd is closed by the guest (i.e., the hostdev is detached or the guest is shut down). This patch series provides kernel-side code that allows userspace to set up a communication channel that will allow the
vfio_apdevice driver to notify userspace when a request to release the mdev is received, so that userspace can close the mdev fd and avoid the hang. The patch series provides the following:Quoted 8 lines
1. Introduces code to handle the
VFIO_DEVICE_GET_IRQ_INFOandVFIO_DEVICE_SET_IRQSioctl calls to set theeventfd_ctxfor signaling a device request to userspace.2. Wires up the VFIO bus driver callback to request a release of the mdev. When invoked, the
vfio_apdevice driver will use theeventfd_ctxset up in #1 to signal a request to userspace to release the mdev.
Quoted 6 lines
As to how this series eventually reaches master... It touches both s390 and vfio.
@Alex/@s390 maintainers -- I suggest it go through s390 given the diffstat, it's almost completely in s390 drivers code. However there is a uapi hit to
vfio.h(in patch 1) that should get at least an ACK from Alex beforehand.
Ack'd, I'll expect this to go through the s390 tree. Thanks,
Alex
On Thu, Jun 01, 2023 at 02:47:22PM -0600, Alex Williamson wrote:
...
Quoted 8 lines
Quoted 6 lines
As to how this series eventually reaches master... It touches both s390 and vfio.
@Alex/@s390 maintainers -- I suggest it go through s390 given the diffstat, it's almost completely in s390 drivers code. However there is a uapi hit to
vfio.h(in patch 1) that should get at least an ACK from Alex beforehand.Ack'd, I'll expect this to go through the s390 tree. Thanks,
Applied, thanks!
Quoted 1 line
Alex