s390x/ap: fix hang when mdev attached to guest is removed
cover s390x/ap: fix hang when mdev attached to guest is removed
When a user attempts to remove a vfio-ap mediated device attached to a guest, the operation hangs until the mdev's fd is closed by the guest (i.e., the guest is shut down). This patch series provides userspace (i.e., qemu) code to handle device unplug requests for AP. When notified that the mdev is being removed, the handler will unplug the device, thus avoiding the hang condition.
Tony Krowiak (2):
linux-headers: Update with vfio_ap IRQ index mapping s390x/ap: Wire up the device request notifier interface
hw/vfio/ap.c | 113 +++++++++++++++++++++++++++++++++++++ linux-headers/linux/vfio.h | 9 +++ 2 files changed, 122 insertions(+)
Signature · 1 line
2.31.1
PATCH 1/2 linux-headers: Update with vfio_ap IRQ index mapping
--- linux-headers/linux/vfio.h | 9 +++++++++ 1 file changed, 9 insertions(+)
linux-headers/linux/vfio.h +9 -0
@@ -646,6 +646,15 @@ enum { VFIO_CCW_NUM_IRQS }; +/* +* The vfio-ap bus driver makes use of the following IRQ index mapping. +* Unimplemented IRQ types return a count of zero. +*/ +enum { +VFIO_AP_REQ_IRQ_INDEX , +VFIO_AP_NUM_IRQS +}; +/** * VFIO_DEVICE_GET_PCI_HOT_RESET_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12, * struct vfio_pci_hot_reset_info)
Signature · 1 line
2.31.1
Re: [PATCH 1/2] linux-headers: Update with vfio_ap IRQ index mapping
Quoted 4 lines · Tony Krowiak
Signed-off-by Tony Krowiak <[email protected]>--- linux-headers/linux/vfio.h | 9 +++++++++ 1 file changed, 9 insertions(+)
Worth nothing here that linux-headers patches should be generated using scripts/update-linux-headers.sh.
Since this linux-headers update includes changes that aren't merged into the kernel yet, I would still use update-linux-headers.sh -- but also include in the commit message that this is a placeholder patch that includes unmerged uapi changes. Then once the kernel changes merge you can just have a proper linux-headers update patch in a subsequent qemu series.
Quoted 21 lines
linux-headers/linux/vfio.h +9
-0@@ -646,6 +646,15 @@ enum { VFIO_CCW_NUM_IRQS }; +/* +* The vfio-ap bus driver makes use of the following IRQ index mapping. +* Unimplemented IRQ types return a count of zero. +*/ +enum { +VFIO_AP_REQ_IRQ_INDEX , +VFIO_AP_NUM_IRQS +}; +/** * VFIO_DEVICE_GET_PCI_HOT_RESET_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12, * struct vfio_pci_hot_reset_info)
Re: [PATCH 1/2] linux-headers: Update with vfio_ap IRQ index mapping
Quoted 9 lines · Matthew Rosato
Quoted 4 lines · Tony Krowiak
Signed-off-by Tony Krowiak <[email protected]>--- linux-headers/linux/vfio.h | 9 +++++++++ 1 file changed, 9 insertions(+)Worth nothing here that linux-headers patches should be generated using
scripts/update-linux-headers.sh.Since this linux-headers update includes changes that aren't merged into the kernel yet, I would still use
update-linux-headers.sh-- but also include in the commit message that this is a placeholder patch that includes unmerged uapi changes. Then once the kernel changes merge you can just have a proper linux-headers update patch in a subsequent qemu series.
I guess I do not understand the procedure here. I first determined the latest kernel release in which the vfio.h file was updated with the following command:
git log --oneline origin/master -- linux-headers/linux/vfio.h
According to the git log, the vfio.h file was last updated in kernel v6.3-rc5. I cloned that kernel from git.kernel.org/pub/scm/linux/kernel/git/stable and checked out kernel 6.3-rc5. I then made the changes to the linux-headers/linux/vfio.h file and ran the update-linux-headers.sh script and created this patch from that. Where did I go wrong?
Quoted 23 lines
Quoted 22 lines
linux-headers/linux/vfio.h +9
-0@@ -646,6 +646,15 @@ enum { VFIO_CCW_NUM_IRQS }; +/* +* The vfio-ap bus driver makes use of the following IRQ index mapping. +* Unimplemented IRQ types return a count of zero. +*/ +enum { +VFIO_AP_REQ_IRQ_INDEX , +VFIO_AP_NUM_IRQS +}; +/** * VFIO_DEVICE_GET_PCI_HOT_RESET_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12, * struct vfio_pci_hot_reset_info)
Re: [PATCH 1/2] linux-headers: Update with vfio_ap IRQ index mapping
Quoted 9 lines · Anthony Krowiak
Quoted 6 lines · Matthew Rosato
Quoted 4 lines · Tony Krowiak
Signed-off-by Tony Krowiak <[email protected]>---
linux-headers/linux/vfio.h| 9 +++++++++1 file changed, 9 insertions(+)
Quoted 8 lines
Quoted 3 lines
Worth nothing here that linux-headers patches should be generated using
scripts/update-linux-headers.sh.Since this linux-headers update includes changes that aren't merged into the kernel yet, I would still use
update-linux-headers.sh-- but also include in the commit message that this is a placeholder patch that includes unmerged uapi changes. Then once the kernel changes merge you can just have a proper linux-headers update patch in a subsequent qemu series.I guess I do not understand the procedure here. I first determined the latest kernel release in which the
vfio.hfile was updated with the following command:
git log --oneline origin/master --linux-headers/linux/vfio.hAccording to the git log, the
vfio.hfile was last updated in kernel v6.3-rc5. I cloned that kernel from git.kernel.org/pub/scm/linux/kernel/git/stable and checked out kernel 6.3-rc5. I then made the changes to thelinux-headers/linux/vfio.hfile and ran theupdate-linux-headers.shscript and created this patch from that. Where did I go wrong?
Presumably your kernel series that you just posted was built on top of 6.4-rc4, not v6.3-rc5 (if it's not, you should rebase onto a recent kernel like 6.4-rc4). Then, you want to point update-linux-headers.sh at that source repository which includes your changes. This will pull in all of the changes to the uapi up to kernel 6.4-rc* + your additional unmerged changes. FWIW, I just pointed update-linux-headers.sh at kernel master from today and I got the following:
--- include/standard-headers/linux/const.h | 2 +- include/standard-headers/linux/virtio_blk.h | 18 +++---- .../standard-headers/linux/virtio_config.h | 6 +++ include/standard-headers/linux/virtio_net.h | 1 + linux-headers/asm-arm64/kvm.h | 33 ++++++++++++ linux-headers/asm-riscv/kvm.h | 53 ++++++++++++++++++- linux-headers/asm-riscv/unistd.h | 9 ++++ linux-headers/asm-s390/unistd_32.h | 1 + linux-headers/asm-s390/unistd_64.h | 1 + linux-headers/asm-x86/kvm.h | 3 ++ linux-headers/linux/const.h | 2 +- linux-headers/linux/kvm.h | 12 +++-- linux-headers/linux/psp-sev.h | 7 +++ linux-headers/linux/userfaultfd.h | 17 +++++- 14 files changed, 149 insertions(+), 16 deletions(-)
---
In your case you would also see an additional line for linux-headers/linux/vfio.h, which would be your unmerged kernel uapi changes.
Then you can include a cover letter something like:
This is a placeholder that pulls in 6.4-rc4 + unmerged kernel changes required by this series. A proper header sync can be done once the associated kernel code merges.
Here's an example from an old series where I did this before:
https://lore.kernel.org/qemu-devel/[email protected]/
One of the main advantages of doing it this way is that if there are any uapi breakages unrelated to your patch we catch them now. That helps whoever might take your series (e.g. Thomas) avoid having to deal with the fallout later when sending a pull request.
Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
linux-headers/linux/vfio.h +0
-0@@ -646,6 +646,15 @@ enum {
VFIO_CCW_NUM_IRQS
};
+/*
+ * The vfio-ap bus driver makes use of the following IRQ index mapping.
+ * Unimplemented IRQ types return a count of zero.
+ */linux-headers/linux/vfio.h +5
-0+enum { + VFIO_AP_REQ_IRQ_INDEX, + VFIO_AP_NUM_IRQS +}; +/**
*VFIO_DEVICE_GET_PCI_HOT_RESET_INFO- _IOWR(VFIO_TYPE,VFIO_BASE+ 12, * structvfio_pci_hot_reset_info)
Re: [PATCH 1/2] linux-headers: Update with vfio_ap IRQ index mapping
Quoted 47 lines · Matthew Rosato
Quoted 9 lines · Anthony Krowiak
Quoted 6 lines · Matthew Rosato
Quoted 4 lines · Tony Krowiak
Signed-off-by Tony Krowiak <[email protected]>--- linux-headers/linux/vfio.h | 9 +++++++++ 1 file changed, 9 insertions(+)Quoted 8 lines
Quoted 3 lines
Worth nothing here that linux-headers patches should be generated using
scripts/update-linux-headers.sh.Since this linux-headers update includes changes that aren't merged into the kernel yet, I would still use
update-linux-headers.sh-- but also include in the commit message that this is a placeholder patch that includes unmerged uapi changes. Then once the kernel changes merge you can just have a proper linux-headers update patch in a subsequent qemu series.I guess I do not understand the procedure here. I first determined the latest kernel release in which the
vfio.hfile was updated with the following command:
git log --oneline origin/master --linux-headers/linux/vfio.hAccording to the git log, the
vfio.hfile was last updated in kernel v6.3-rc5. I cloned that kernel from git.kernel.org/pub/scm/linux/kernel/git/stable and checked out kernel 6.3-rc5. I then made the changes to thelinux-headers/linux/vfio.hfile and ran theupdate-linux-headers.shscript and created this patch from that. Where did I go wrong?Presumably your kernel series that you just posted was built on top of 6.4-rc4, not v6.3-rc5 (if it's not, you should rebase onto a recent kernel like 6.4-rc4). Then, you want to point
update-linux-headers.shat that source repository which includes your changes. This will pull in all of the changes to the uapi up to kernel 6.4-rc* + your additional unmerged changes. FWIW, I just pointedupdate-linux-headers.shat kernel master from today and I got the following:--- include/standard-headers/linux/const.h | 2 +- include/standard-headers/linux/virtio_blk.h | 18 +++---- .../standard-headers/linux/virtio_config.h | 6 +++ include/standard-headers/linux/virtio_net.h | 1 + linux-headers/asm-arm64/kvm.h | 33 ++++++++++++ linux-headers/asm-riscv/kvm.h | 53 ++++++++++++++++++- linux-headers/asm-riscv/unistd.h | 9 ++++ linux-headers/asm-s390/unistd_32.h | 1 + linux-headers/asm-s390/unistd_64.h | 1 + linux-headers/asm-x86/kvm.h | 3 ++ linux-headers/linux/const.h | 2 +- linux-headers/linux/kvm.h | 12 +++-- linux-headers/linux/psp-sev.h | 7 +++ linux-headers/linux/userfaultfd.h | 17 +++++- 14 files changed, 149 insertions(+), 16 deletions(-)---
In your case you would also see an additional line for
linux-headers/linux/vfio.h, which would be your unmerged kernel uapi changes.Then you can include a cover letter something like:
This is a placeholder that pulls in 6.4-rc4 + unmerged kernel changes required by this series. A proper header sync can be done once the associated kernel code merges.
Quoted 5 lines
Here's an example from an old series where I did this before:
https://lore.kernel.org/qemu-devel/[email protected]/
One of the main advantages of doing it this way is that if there are any uapi breakages unrelated to your patch we catch them now. That helps whoever might take your series (e.g. Thomas) avoid having to deal with the fallout later when sending a pull request.
Thanks Matt, I'll do this for v2.
Quoted 25 lines
Quoted 24 lines
Quoted 24 lines
Quoted 23 lines
linux-headers/linux/vfio.h +8
-0@@ -646,6 +646,15 @@ enum { VFIO_CCW_NUM_IRQS }; +/* +* The vfio-ap bus driver makes use of the following IRQ index mapping. +* Unimplemented IRQ types return a count of zero. +*/ +enum { + VFIO_AP_REQ_IRQ_INDEX, + VFIO_AP_NUM_IRQS +}; + /*** VFIO_DEVICE_GET_PCI_HOT_RESET_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12, * struct vfio_pci_hot_reset_info)
Re: [PATCH 1/2] linux-headers: Update with vfio_ap IRQ index mapping
In reply to Anthony KrowiakQuoted 16 lines · Anthony Krowiak
Quoted 9 lines · Matthew Rosato
Quoted 4 lines · Tony Krowiak
Signed-off-by Tony Krowiak <[email protected]>--- linux-headers/linux/vfio.h | 9 +++++++++ 1 file changed, 9 insertions(+)Worth nothing here that linux-headers patches should be generated using
scripts/update-linux-headers.sh.Since this linux-headers update includes changes that aren't merged into the kernel yet, I would still use
update-linux-headers.sh-- but also include in the commit message that this is a placeholder patch that includes unmerged uapi changes. Then once the kernel changes merge you can just have a proper linux-headers update patch in a subsequent qemu series.I guess I do not understand the procedure here. I first determined the latest kernel release in which the
vfio.hfile was updated with the following command:
git log --oneline origin/master --linux-headers/linux/vfio.h
Quoted 6 lines
According to the git log, the
vfio.hfile was last updated in kernel v6.3-rc5. I cloned that kernel from git.kernel.org/pub/scm/linux/kernel/git/stable and checked out kernel 6.3-rc5. I then made the changes to thelinux-headers/linux/vfio.hfile and ran theupdate-linux-headers.shscript and created this patch from that. Where did I go wrong?
I think your procedure is fine for changes that are local to a single header file. The one thing I'd recommend is to put an explicit "dummy patch, to be replaced by a proper headers update" note into the patch, so that it doesn't get merged by accident.
(For complex changes, headers update + explicit note might be better, but the simple approach works in most cases.)
Re: [PATCH 1/2] linux-headers: Update with vfio_ap IRQ index mapping
Quoted 25 lines · Cornelia Huck
Quoted 16 lines · Anthony Krowiak
Quoted 9 lines · Matthew Rosato
Quoted 4 lines · Tony Krowiak
Signed-off-by Tony Krowiak <[email protected]>--- linux-headers/linux/vfio.h | 9 +++++++++ 1 file changed, 9 insertions(+)Worth nothing here that linux-headers patches should be generated using
scripts/update-linux-headers.sh.Since this linux-headers update includes changes that aren't merged into the kernel yet, I would still use
update-linux-headers.sh-- but also include in the commit message that this is a placeholder patch that includes unmerged uapi changes. Then once the kernel changes merge you can just have a proper linux-headers update patch in a subsequent qemu series.I guess I do not understand the procedure here. I first determined the latest kernel release in which the
vfio.hfile was updated with the following command:
git log --oneline origin/master --linux-headers/linux/vfio.hQuoted 6 lines
According to the git log, the
vfio.hfile was last updated in kernel v6.3-rc5. I cloned that kernel from git.kernel.org/pub/scm/linux/kernel/git/stable and checked out kernel 6.3-rc5. I then made the changes to thelinux-headers/linux/vfio.hfile and ran theupdate-linux-headers.shscript and created this patch from that. Where did I go wrong?
Quoted 7 lines
I think your procedure is fine for changes that are local to a single header file. The one thing I'd recommend is to put an explicit "dummy patch, to be replaced by a proper headers update" note into the patch, so that it doesn't get merged by accident.
(For complex changes, headers update + explicit note might be better, but the simple approach works in most cases.)
Will do.
Quoted 1 line
PATCH 2/2 s390x/ap: Wire up the device request notifier interface
In reply to Tony KrowiakLet's wire up the device request notifier interface to handle device unplug requests for AP.
--- hw/vfio/ap.c | 113 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+)
hw/vfio/ap.c +113 -0
@@ -18,6 +18,8 @@#include "hw/vfio/vfio-common.h" #include "hw/s390x/ap-device.h" #include "qemu/error-report.h" +#include "qemu/event_notifier.h" +#include "qemu/main-loop.h" #include "qemu/module.h" #include "qemu/option.h" #include "qemu/config-file.h"
@@ -33,6 +35,7 @@struct VFIOAPDevice {APDevice apdev;VFIODevice vdev; +EventNotifier req_notifier; };OBJECT_DECLARE_SIMPLE_TYPE (VFIOAPDevice,VFIO_AP_DEVICE )
@@ -84,10 +87,110 @@ static VFIOGroup *vfio_ap_get_group(VFIOAPDevice *vapdev, Error **errp)return vfio_get_group (groupid, &address_space_memory, errp); } +static void vfio_ap_req_notifier_handler (void *opaque) +{ +VFIOAPDevice *vapdev = opaque; +Error *err =NULL ; + +if (!event_notifier_test_and_clear (&vapdev->req_notifier)) { +return ; + } + +qdev_unplug (DEVICE (vapdev), &err); + +if (err) { +warn_reportf_err (err,VFIO_MSG_PREFIX , vapdev->vdev.name); + } +} + +static void vfio_ap_register_irq_notifier (VFIOAPDevice *vapdev, +unsigned int irq,Error **errp) +{ +int fd; +size_t argsz; +IOHandler *fd_read; +EventNotifier *notifier; +struct vfio_irq_info *irq_info; +VFIODevice *vdev = &vapdev->vdev; + +switch (irq) { +case VFIO_AP_REQ_IRQ_INDEX : + notifier = &vapdev->req_notifier; + fd_read = vfio_ap_req_notifier_handler; +break ; +default : +error_setg (errp,"vfio: Unsupported device irq(%d)" , irq); +return ; + } + +if (vdev->num_irqs < irq +1 ) { +error_setg (errp,"vfio: IRQ %u not available (number of irqs %u)" , + irq, vdev->num_irqs); +return ; + } + + argsz =sizeof (*irq_info); + irq_info =g_malloc0 (argsz); + irq_info->index = irq; + irq_info->argsz = argsz; + +if (ioctl (vdev->fd,VFIO_DEVICE_GET_IRQ_INFO , + irq_info) <0 || irq_info->count <1 ) { +error_setg_errno (errp, errno,"vfio: Error getting irq info" ); + goto out_free_info; + } + +if (event_notifier_init (notifier,0 )) { +error_setg_errno (errp, errno, +"vfio: Unable to init event notifier for irq (%d)" , + irq); + goto out_free_info; + } + + fd =event_notifier_get_fd (notifier); +qemu_set_fd_handler (fd, fd_read,NULL , vapdev); + +if (vfio_set_irq_signaling (vdev, irq,0 ,VFIO_IRQ_SET_ACTION_TRIGGER , fd, + errp)) { +qemu_set_fd_handler (fd,NULL ,NULL , vapdev); +event_notifier_cleanup (notifier); + } + +out_free_info: +g_free (irq_info); + +} + +static void vfio_ap_unregister_irq_notifier (VFIOAPDevice *vapdev, +unsigned int irq) +{ +Error *err =NULL ; +EventNotifier *notifier; + +switch (irq) { +case VFIO_AP_REQ_IRQ_INDEX : + notifier = &vapdev->req_notifier; +break ; +default : +error_report ("vfio: Unsupported device irq(%d)" , irq); +return ; + } + +if (vfio_set_irq_signaling (&vapdev->vdev, irq,0 , +VFIO_IRQ_SET_ACTION_TRIGGER ,-1 , &err)) { +warn_reportf_err (err,VFIO_MSG_PREFIX , vapdev->vdev.name); + } + +qemu_set_fd_handler (event_notifier_get_fd (notifier), +NULL ,NULL , vapdev); +event_notifier_cleanup (notifier); +} +static void vfio_ap_realize (DeviceState *dev,Error **errp) {int ret;char *mdevid; +Error *err =NULL ;VFIOGroup *vfio_group;APDevice *apdev =AP_DEVICE (dev);VFIOAPDevice *vapdev =VFIO_AP_DEVICE (apdev);
@@ -116,6 +219,15 @@ static void vfio_ap_realize(DeviceState *dev, Error **errp) goto out_get_dev_err; } +vfio_ap_register_irq_notifier (vapdev,VFIO_AP_REQ_IRQ_INDEX , &err); +if (err) { +/* +* Report this error, but do not make it a failing condition. +* Lack of this IRQ in the host does not prevent normal operation. +*/ +error_report_err (err); + } +return ; out_get_dev_err:
@@ -129,6 +241,7 @@ static void vfio_ap_unrealize(DeviceState *dev)VFIOAPDevice *vapdev =VFIO_AP_DEVICE (apdev);VFIOGroup *group = vapdev->vdev.group; +vfio_ap_unregister_irq_notifier (vapdev,VFIO_AP_REQ_IRQ_INDEX );vfio_ap_put_device (vapdev);vfio_put_group (group); }
Signature · 1 line
2.31.1
The required kernel changes associated with the patch are here:
https://lore.kernel.org/linux-s390/[email protected]/
Quoted 15 lines · Tony Krowiak
When a user attempts to remove a vfio-ap mediated device attached to a guest, the operation hangs until the mdev's fd is closed by the guest (i.e., the guest is shut down). This patch series provides userspace (i.e., qemu) code to handle device unplug requests for AP. When notified that the mdev is being removed, the handler will unplug the device, thus avoiding the hang condition.
Tony Krowiak (2):
linux-headers: Update withvfio_apIRQ index mapping s390x/ap: Wire up the device request notifier interfacehw/vfio/ap.c | 113 +++++++++++++++++++++++++++++++++++++ linux-headers/linux/vfio.h | 9 +++ 2 files changed, 122 insertions(+)