vfio: fix deadlock between group lock and kvm lock
Subsystems (3)
PATCH v3 1/1 vfio: fix deadlock between group lock and kvm lock
After 51cdc8bc120e, we have another deadlock scenario between the kvm->lock and the vfio group_lock with two different codepaths acquiring the locks in different order. Specifically in vfio_open_device, vfio holds the vfio group_lock when issuing device->ops->open_device but some drivers (like vfio-ap) need to acquire kvm->lock during their open_device routine; Meanwhile, kvm_vfio_release will acquire the kvm->lock first before calling vfio_file_set_kvm which will acquire the vfio group_lock.
To resolve this, let's remove the need for the vfio group_lock from the kvm_vfio_release codepath. This is done by introducing a new spinlock to protect modifications to the vfio group kvm pointer, and acquiring a kvm ref from within vfio while holding this spinlock, with the reference held until the last close for the device in question.
---
Changes from v2:
* Relocate the new functions back to vfio_main and externalize to call from group (Kevin) since cdev will need this too
* s/vfio_kvm_*_kvm/vfio_device_*_kvm/ and only pass device as input. Handle new kvm_ref_lock directly inside vfio_device_get_kvm (Alex)
* Add assert_lockdep_held for dev_set lock (Alex)
* Internalize error paths for vfio_device_get_kvm_safe and now return void - either device->kvm is set with a ref taken or is NULL (Alex)
* Other flow suggestions to make the call path cleaner - Thanks! (Alex)
* Can't pass group->kvm to vfio_device_open, as it references the value outside of new lock. Pass device->kvm to minimize changes in this fix (Alex, Yi)
Changes from v1:
* use spin_lock instead of spin_lock_irqsave (Jason)
* clear device->kvm_put as part of vfio_kvm_put_kvm (Yi)
* Re-arrange code to avoid referencing the group contents from within vfio_main (Kevin) which meant moving most of the code in this patch to group.c along with getting/dropping of the dev_set lock
--- drivers/vfio/group.c | 32 ++++++++++++++---- drivers/vfio/vfio.h | 14 ++++++++ drivers/vfio/vfio_main.c | 70 ++++++++++++++++++++++++++++++++++++---- include/linux/vfio.h | 2 +- 4 files changed, 103 insertions(+), 15 deletions(-)
drivers/vfio/group.c +25 -7
@@ -164,13 +164,23 @@ static int vfio_device_group_open(struct vfio_device *device) goto out_unlock; } +mutex_lock (&device->dev_set->lock); +/* -* Here we pass the KVM pointer with the group under the lock. If the -* device driver will use it, it must obtain a reference and release it -* during close_device. +* Before the first device open, get the KVM pointer currently +* associated with the group (if there is one) and obtain a reference +* now that will be held until the open_count reaches 0 again. Save +* the pointer in the device for use by drivers. */ - ret =vfio_device_open (device, device->group->iommufd, - device->group->kvm); +if (device->open_count ==0 ) +vfio_device_get_kvm_safe (device); + + ret =vfio_device_open (device, device->group->iommufd, device->kvm); + +if (device->open_count ==0 ) +vfio_device_put_kvm (device); + +mutex_unlock (&device->dev_set->lock); out_unlock:mutex_unlock (&device->group->group_lock);
@@ -180,7 +190,14 @@ static int vfio_device_group_open(struct vfio_device *device)void vfio_device_group_close (struct vfio_device *device) {mutex_lock (&device->group->group_lock); +mutex_lock (&device->dev_set->lock); +vfio_device_close (device, device->group->iommufd); + +if (device->open_count ==0 ) +vfio_device_put_kvm (device); + +mutex_unlock (&device->dev_set->lock);mutex_unlock (&device->group->group_lock); }
@@ -450,6 +467,7 @@ static struct vfio_group *vfio_group_alloc(struct iommu_group *iommu_group,refcount_set (&group->drivers,1 );mutex_init (&group->group_lock); +spin_lock_init (&group->kvm_ref_lock);INIT_LIST_HEAD (&group->device_list);mutex_init (&group->device_lock); group->iommu_group = iommu_group;
@@ -803,9 +821,9 @@ void vfio_file_set_kvm(struct file *file, struct kvm *kvm)if (!vfio_file_is_group (file))return ; -mutex_lock (&group->group_lock); +spin_lock (&group->kvm_ref_lock); group->kvm = kvm; -mutex_unlock (&group->group_lock); +spin_unlock (&group->kvm_ref_lock); }EXPORT_SYMBOL_GPL (vfio_file_set_kvm);
drivers/vfio/vfio.h +14 -0
@@ -74,6 +74,7 @@ struct vfio_group {struct file *opened_file;struct blocking_notifier_head notifier;struct iommufd_ctx *iommufd; +spinlock_t kvm_ref_lock; }; int vfio_device_set_group(struct vfio_device *device,
@@ -251,4 +252,17 @@ extern bool vfio_noiommu __read_mostly;enum { vfio_noiommu = false };#endif +#ifdef CONFIG_HAVE_KVM +void vfio_device_get_kvm_safe (struct vfio_device *device); +void vfio_device_put_kvm (struct vfio_device *device); +#else +static inline void vfio_device_get_kvm_safe (struct vfio_device *device) +{ +} + +static inline void vfio_device_put_kvm (struct vfio_device *device) +{ +} +#endif +#endif
drivers/vfio/vfio_main.c +63 -7
122 lines not on this page · raw message
include/linux/vfio.h +1 -1
@@ -46,7 +46,6 @@ struct vfio_device {struct vfio_device_set *dev_set;struct list_head dev_set_list;unsigned int migration_flags; -/* Driver must reference the kvm during open_device or never touch it */ struct kvm *kvm;/* Members below here are private, not for driver use */
@@ -58,6 +57,7 @@ struct vfio_device {struct list_head group_next;struct list_head iommu_entry;struct iommufd_access *iommufd_access; +void (*put_kvm)(struct kvm *kvm);#if IS_ENABLED (CONFIG_IOMMUFD )struct iommufd_device *iommufd_device;struct iommufd_ctx *iommufd_ictx;
Signature · 1 line
2.39.1
Patchwork
CI[Intel-gfx] ✓ Fi.CI.BAT: success for vfio: fix deadlock between group lock and kvm lock (rev5)
== Series Details ==
== Series Details ==
Series: vfio: fix deadlock between group lock and kvm lock (rev5)
URL : https://patchwork.freedesktop.org/series/113535/
State : success
== Summary ==
CI Bug Log - changes from CI_DRM_12684 -> Patchwork_113535v5
====================================================
Summary
-------
**SUCCESS**
No regressions found.
External URL: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/index.html
Participating hosts (27 -> 25)
------------------------------
Missing (2): fi-kbl-soraka fi-snb-2520m
Known issues
------------
Here are the changes found in Patchwork_113535v5 that come from known issues:
### IGT changes ###
#### Issues hit ####
* igt@kms_chamelium_hpd@common-hpd-after-suspend:
- bat-dg1-6: NOTRUN -> [SKIP][1] ([i915#7828])
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/bat-dg1-6/igt@[email protected]
* igt@kms_cursor_legacy@basic-busy-flip-before-cursor@atomic-transitions-varying-size:
- fi-bsw-n3050: [PASS][2] -> [FAIL][3] ([i915#6298])
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/fi-bsw-n3050/igt@kms_cursor_legacy@basic-busy-flip-before-cursor@atomic-transitions-varying-size.html
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/fi-bsw-n3050/igt@kms_cursor_legacy@basic-busy-flip-before-cursor@atomic-transitions-varying-size.html
#### Possible fixes ####
* igt@i915_selftest@live@gt_lrc:
- bat-dg1-6: [ABORT][4] ([i915#4983]) -> [PASS][5]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/bat-dg1-6/igt@i915_selftest@live@gt_lrc.html
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/bat-dg1-6/igt@i915_selftest@live@gt_lrc.html
* igt@i915_selftest@live@reset:
- {bat-rpls-2}: [ABORT][6] ([i915#4983]) -> [PASS][7]
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/bat-rpls-2/igt@i915_selftest@[email protected]
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/bat-rpls-2/igt@i915_selftest@[email protected]
{name}: This element is suppressed. This means it is ignored when computing
the status of the difference (SUCCESS, WARNING, or FAILURE).
[i915#1845]: https://gitlab.freedesktop.org/drm/intel/issues/1845
[i915#4983]: https://gitlab.freedesktop.org/drm/intel/issues/4983
[i915#6298]: https://gitlab.freedesktop.org/drm/intel/issues/6298
[i915#6367]: https://gitlab.freedesktop.org/drm/intel/issues/6367
[i915#7699]: https://gitlab.freedesktop.org/drm/intel/issues/7699
[i915#7828]: https://gitlab.freedesktop.org/drm/intel/issues/7828
[i915#7977]: https://gitlab.freedesktop.org/drm/intel/issues/7977
[i915#7996]: https://gitlab.freedesktop.org/drm/intel/issues/7996
Build changes
-------------
* Linux: CI_DRM_12684 -> Patchwork_113535v5
CI-20190529: 20190529
CI_DRM_12684: 57eb8680d3ef1213de1d9c607b95e522035036e6 @ git://anongit.freedesktop.org/gfx-ci/linux
IGT_7144: cda71bf809b981a646270963d6b1ccee4fd4643b @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
Patchwork_113535v5: 57eb8680d3ef1213de1d9c607b95e522035036e6 @ git://anongit.freedesktop.org/gfx-ci/linux
### Linux commits
9764937dc1e4 vfio: fix deadlock between group lock and kvm lock
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/index.html
Quoted 37 lines · On Thu, 2 Feb 2023 11:24:42 -0500 Matthew Rosato <[email protected]>…
After
51cdc8bc120e, we have another deadlock scenario between thekvm->lockand the vfiogroup_lockwith two different codepaths acquiring the locks in different order. Specifically invfio_open_device, vfio holds the vfiogroup_lockwhen issuingdevice->ops->open_devicebut some drivers (like vfio-ap) need to acquirekvm->lockduring theiropen_deviceroutine; Meanwhile,kvm_vfio_releasewill acquire thekvm->lockfirst before callingvfio_file_set_kvmwhich will acquire the vfiogroup_lock.To resolve this, let's remove the need for the vfio
group_lockfrom thekvm_vfio_releasecodepath. This is done by introducing a new spinlock to protect modifications to the vfio group kvm pointer, and acquiring a kvm ref from within vfio while holding this spinlock, with the reference held until the last close for the device in question.Fixes 51cdc8bc120e ("kvm/vfio: Fix potential deadlock on vfio group_lock")Reported-by Anthony Krowiak <[email protected]>Suggested-by Jason Gunthorpe <[email protected]>Signed-off-by Matthew Rosato <[email protected]>---
Changes from v2:
* Relocate the new functions back tovfio_mainand externalize to call from group (Kevin) since cdev will need this too
* s/vfio_kvm_*_kvm/vfio_device_*_kvm/ and only pass device as input. Handle newkvm_ref_lockdirectly insidevfio_device_get_kvm(Alex)
* Addassert_lockdep_heldfordev_setlock (Alex)
* Internalize error paths forvfio_device_get_kvm_safeand now return void - eitherdevice->kvmis set with a ref taken or is NULL (Alex)
* Other flow suggestions to make the call path cleaner - Thanks! (Alex)
* Can't passgroup->kvmtovfio_device_open, as it references the value outside of new lock. Passdevice->kvmto minimize changes in this fix (Alex, Yi)
Changes from v1:
* usespin_lockinstead ofspin_lock_irqsave(Jason)
* cleardevice->kvm_putas part ofvfio_kvm_put_kvm(Yi)
* Re-arrange code to avoid referencing the group contents from withinvfio_main(Kevin) which meant moving most of the code in this patch togroup.calong with getting/dropping of thedev_setlock
Quoted 6 lines
--- drivers/vfio/group.c | 32 ++++++++++++++---- drivers/vfio/vfio.h | 14 ++++++++ drivers/vfio/vfio_main.c | 70 ++++++++++++++++++++++++++++++++++++---- include/linux/vfio.h | 2 +- 4 files changed, 103 insertions(+), 15 deletions(-)
LGTM. I'm not sure moving the functions to vfio_main really buys us anything since we're making so much use of group fields. The cdev approach will necessarily be different, so the bulk of the get code will likely need to move back to group.c anyway.
I'll wait for further comments and reviews by others before applying. Thanks,
Alex
Quoted 251 lines
drivers/vfio/group.c +25
-7@@ -164,13 +164,23 @@ static int vfio_device_group_open(struct vfio_device *device) goto out_unlock; } +mutex_lock (&device->dev_set->lock); +/* -* Here we pass the KVM pointer with the group under the lock. If the -* device driver will use it, it must obtain a reference and release it -* during close_device. +* Before the first device open, get the KVM pointer currently +* associated with the group (if there is one) and obtain a reference +* now that will be held until the open_count reaches 0 again. Save +* the pointer in the device for use by drivers. */ - ret =vfio_device_open (device, device->group->iommufd, - device->group->kvm); +if (device->open_count ==0 ) +vfio_device_get_kvm_safe (device); + + ret =vfio_device_open (device, device->group->iommufd, device->kvm); + +if (device->open_count ==0 ) +vfio_device_put_kvm (device); + +mutex_unlock (&device->dev_set->lock); out_unlock:mutex_unlock (&device->group->group_lock);@@ -180,7 +190,14 @@ static int vfio_device_group_open(struct vfio_device *device)void vfio_device_group_close (struct vfio_device *device) {mutex_lock (&device->group->group_lock); +mutex_lock (&device->dev_set->lock); +vfio_device_close (device, device->group->iommufd); + +if (device->open_count ==0 ) +vfio_device_put_kvm (device); + +mutex_unlock (&device->dev_set->lock);mutex_unlock (&device->group->group_lock); }@@ -450,6 +467,7 @@ static struct vfio_group *vfio_group_alloc(struct iommu_group *iommu_group,refcount_set (&group->drivers,1 );mutex_init (&group->group_lock); +spin_lock_init (&group->kvm_ref_lock);INIT_LIST_HEAD (&group->device_list);mutex_init (&group->device_lock); group->iommu_group = iommu_group;@@ -803,9 +821,9 @@ void vfio_file_set_kvm(struct file *file, struct kvm *kvm)if (!vfio_file_is_group (file))return ; -mutex_lock (&group->group_lock); +spin_lock (&group->kvm_ref_lock); group->kvm = kvm; -mutex_unlock (&group->group_lock); +spin_unlock (&group->kvm_ref_lock); }EXPORT_SYMBOL_GPL (vfio_file_set_kvm);drivers/vfio/vfio.h +14
-0@@ -74,6 +74,7 @@ struct vfio_group {struct file *opened_file;struct blocking_notifier_head notifier;struct iommufd_ctx *iommufd; +spinlock_t kvm_ref_lock; }; int vfio_device_set_group(struct vfio_device *device,@@ -251,4 +252,17 @@ extern bool vfio_noiommu __read_mostly;enum { vfio_noiommu = false };#endif +#ifdef CONFIG_HAVE_KVM +void vfio_device_get_kvm_safe (struct vfio_device *device); +void vfio_device_put_kvm (struct vfio_device *device); +#else +static inline void vfio_device_get_kvm_safe (struct vfio_device *device) +{ +} + +static inline void vfio_device_put_kvm (struct vfio_device *device) +{ +} +#endif +#endif drivers/vfio/vfio_main.c +63
-7@@ -16,6 +16,9 @@#include <linux/fs.h> #include <linux/idr.h> #include <linux/iommu.h> +#ifdef CONFIG_HAVE_KVM +#include <linux/kvm_host.h> +#endif #include <linux/list.h> #include <linux/miscdevice.h> #include <linux/module.h> @@ -338,6 +341,62 @@ void vfio_unregister_group_dev(struct vfio_device *device) }EXPORT_SYMBOL_GPL (vfio_unregister_group_dev); +#ifdef CONFIG_HAVE_KVM +void vfio_device_get_kvm_safe (struct vfio_device *device) +{ +void (*pfn)(struct kvm *kvm); +bool (*fn)(struct kvm *kvm); +bool ret; + +lockdep_assert_held (&device->dev_set->lock); + +spin_lock (&device->group->kvm_ref_lock); +if (!device->group->kvm) + goto unlock; + + pfn =symbol_get (kvm_put_kvm); +if (WARN_ON (!pfn)) + goto unlock; + + fn =symbol_get (kvm_get_kvm_safe); +if (WARN_ON (!fn)) { +symbol_put (kvm_put_kvm); + goto unlock; + } + + ret =fn (device->group->kvm); +symbol_put (kvm_get_kvm_safe); +if (!ret) { +symbol_put (kvm_put_kvm); + goto unlock; + } + + device->put_kvm = pfn; + device->kvm = device->group->kvm; + +unlock: +spin_unlock (&device->group->kvm_ref_lock); +} + +void vfio_device_put_kvm (struct vfio_device *device) +{ +lockdep_assert_held (&device->dev_set->lock); + +if (!device->kvm) +return ; + +if (WARN_ON (!device->put_kvm)) + goto clear; + + device->put_kvm (device->kvm); + device->put_kvm =NULL ; +symbol_put (kvm_put_kvm); + +clear: + device->kvm =NULL ; +} +#endif +/* true if the vfio_device has open_device() called but not close_device() */ static bool vfio_assert_device_open (struct vfio_device *device) {@@ -361,7 +420,6 @@ static int vfio_device_first_open(struct vfio_device *device,if (ret) goto err_module_put; - device->kvm = kvm;if (device->ops->open_device) { ret = device->ops->open_device (device);if (ret)@@ -370,7 +428,6 @@ static int vfio_device_first_open(struct vfio_device *device,return 0 ; err_unuse_iommu: - device->kvm =NULL ;if (iommufd)vfio_iommufd_unbind (device);else @@ -387,7 +444,6 @@ static void vfio_device_last_close(struct vfio_device *device,if (device->ops->close_device) device->ops->close_device (device); - device->kvm =NULL ;if (iommufd)vfio_iommufd_unbind (device);else @@ -400,14 +456,14 @@ int vfio_device_open(struct vfio_device *device, {int ret =0 ; -mutex_lock (&device->dev_set->lock); +lockdep_assert_held (&device->dev_set->lock); + device->open_count++;if (device->open_count ==1 ) { ret =vfio_device_first_open (device, iommufd, kvm);if (ret) device->open_count--; } -mutex_unlock (&device->dev_set->lock);return ret; }@@ -415,12 +471,12 @@ int vfio_device_open(struct vfio_device *device,void vfio_device_close (struct vfio_device *device,struct iommufd_ctx *iommufd) { -mutex_lock (&device->dev_set->lock); +lockdep_assert_held (&device->dev_set->lock); +vfio_assert_device_open (device);if (device->open_count ==1 )vfio_device_last_close (device, iommufd); device->open_count--; -mutex_unlock (&device->dev_set->lock); }/* include/linux/vfio.h +1
-1@@ -46,7 +46,6 @@ struct vfio_device {struct vfio_device_set *dev_set;struct list_head dev_set_list;unsigned int migration_flags; -/* Driver must reference the kvm during open_device or never touch it */ struct kvm *kvm;/* Members below here are private, not for driver use */ @@ -58,6 +57,7 @@ struct vfio_device {struct list_head group_next;struct list_head iommu_entry;struct iommufd_access *iommufd_access; +void (*put_kvm)(struct kvm *kvm);#if IS_ENABLED (CONFIG_IOMMUFD )struct iommufd_device *iommufd_device;struct iommufd_ctx *iommufd_ictx;
Quoted 48 lines
From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 3:42 AMQuoted 4 lines · On Thu, 2 Feb 2023 11:24:42 -0500 Matthew Rosato <[email protected]>…
After
51cdc8bc120e, we have another deadlock scenario between thekvm->lockand the vfiogroup_lockwith two different codepaths acquiring the locks in different order. Specifically invfio_open_device, vfio holds the vfiogroup_lockwhen issuingdevice->ops->open_devicebutsome
Quoted 33 lines
drivers (like vfio-ap) need to acquire
kvm->lockduring theiropen_deviceroutine; Meanwhile,kvm_vfio_releasewill acquire thekvm->lockfirst before callingvfio_file_set_kvmwhich will acquire the vfiogroup_lock.To resolve this, let's remove the need for the vfio
group_lockfrom thekvm_vfio_releasecodepath. This is done by introducing a new spinlock to protect modifications to the vfio group kvm pointer, and acquiring a kvm ref from within vfio while holding this spinlock, with the reference held until the last close for the device in question.Fixes 51cdc8bc120e ("kvm/vfio: Fix potential deadlock on vfio group_lock")Reported-by Anthony Krowiak <[email protected]>Suggested-by Jason Gunthorpe <[email protected]>Signed-off-by Matthew Rosato <[email protected]>---
Changes from v2:
* Relocate the new functions back tovfio_mainand externalize to call from group (Kevin) since cdev will need this too
* s/vfio_kvm_*_kvm/vfio_device_*_kvm/ and only pass device as input. Handle newkvm_ref_lockdirectly insidevfio_device_get_kvm(Alex)
* Addassert_lockdep_heldfordev_setlock (Alex)
* Internalize error paths forvfio_device_get_kvm_safeand now return void - eitherdevice->kvmis set with a ref taken or is NULL (Alex)
* Other flow suggestions to make the call path cleaner - Thanks! (Alex)
* Can't passgroup->kvmtovfio_device_open, as it references the value outside of new lock. Passdevice->kvmto minimize changes in this fix (Alex, Yi)
Changes from v1:
* usespin_lockinstead ofspin_lock_irqsave(Jason)
* cleardevice->kvm_putas part ofvfio_kvm_put_kvm(Yi)
* Re-arrange code to avoid referencing the group contents from withinvfio_main(Kevin) which meant moving most of the code in this patch togroup.calong with getting/dropping of thedev_setlockQuoted 4 lines
--- drivers/vfio/group.c | 32 ++++++++++++++---- drivers/vfio/vfio.h | 14 ++++++++ drivers/vfio/vfio_main.c | 70 ++++++++++++++++++++++++++++++++++++--
Quoted 9 lines
--
Quoted 2 lines
include/linux/vfio.h | 2 +- 4 files changed, 103 insertions(+), 15 deletions(-)LGTM. I'm not sure moving the functions to
vfio_mainreally buys us anything since we're making so much use of group fields. The cdev approach will necessarily be different, so the bulk of the get code will likely need to move back togroup.canyway.
well my last comment was based on Matthew's v2 where the get code gets a kvm passed in instead of implicitly retrieving group ref_lock internally. In that case the get/put helpers only contain device logic thus fit in vfio_main.c.
with v3 then they have to be in group.c since we don't want to use group fields in vfio_main.c.
but I still think v2 of the helpers is slightly better. The only difference between cdev and group when handling this race is using different ref_lock. the symbol get/put part is exactly same. So even if we merge v3 like this, very likely Yi has to change it back to v2 style to share the get/put helpers while just leaving the ref_lock part handled differently between the two path.
Thanks
Kevin
Quoted 66 lines · Kevin
Quoted 40 lines
From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 3:42 AMQuoted 4 lines · On Thu, 2 Feb 2023 11:24:42 -0500 Matthew Rosato <[email protected]>…
After
51cdc8bc120e, we have another deadlock scenario between thekvm->lockand the vfiogroup_lockwith two different codepaths acquiring the locks in different order. Specifically invfio_open_device, vfio holds the vfiogroup_lockwhen issuingdevice->ops->open_devicebutsome
Quoted 29 lines
drivers (like vfio-ap) need to acquire
kvm->lockduring theiropen_deviceroutine; Meanwhile,kvm_vfio_releasewill acquire thekvm->lockfirst before callingvfio_file_set_kvmwhich will acquire the vfiogroup_lock.To resolve this, let's remove the need for the vfio
group_lockfrom thekvm_vfio_releasecodepath. This is done by introducing a new spinlock to protect modifications to the vfio group kvm pointer, and acquiring a kvm ref from within vfio while holding this spinlock, with the reference held until the last close for the device in question.Fixes 51cdc8bc120e ("kvm/vfio: Fix potential deadlock on vfio group_lock")Reported-by Anthony Krowiak <[email protected]>Suggested-by Jason Gunthorpe <[email protected]>Signed-off-by Matthew Rosato <[email protected]>---
Changes from v2:
* Relocate the new functions back tovfio_mainand externalize to call from group (Kevin) since cdev will need this too
* s/vfio_kvm_*_kvm/vfio_device_*_kvm/ and only pass device as input. Handle newkvm_ref_lockdirectly insidevfio_device_get_kvm(Alex)
* Addassert_lockdep_heldfordev_setlock (Alex)
* Internalize error paths forvfio_device_get_kvm_safeand now return void - eitherdevice->kvmis set with a ref taken or is NULL (Alex)
* Other flow suggestions to make the call path cleaner - Thanks! (Alex)
* Can't passgroup->kvmtovfio_device_open, as it references the value outside of new lock. Passdevice->kvmto minimize changes in this fix (Alex, Yi)
Changes from v1:
* usespin_lockinstead ofspin_lock_irqsave(Jason)Quoted 17 lines
Quoted 8 lines
* clear
device->kvm_putas part ofvfio_kvm_put_kvm(Yi)
* Re-arrange code to avoid referencing the group contents from withinvfio_main(Kevin) which meant moving most of the code in this patch togroup.calong with getting/dropping of thedev_setlock--- drivers/vfio/group.c | 32 ++++++++++++++---- drivers/vfio/vfio.h | 14 ++++++++ drivers/vfio/vfio_main.c | 70 ++++++++++++++++++++++++++++++++++++--Quoted 2 lines
include/linux/vfio.h | 2 +- 4 files changed, 103 insertions(+), 15 deletions(-)LGTM. I'm not sure moving the functions to
vfio_mainreally buys us anything since we're making so much use of group fields. The cdev approach will necessarily be different, so the bulk of the get code will likely need to move back togroup.canyway.well my last comment was based on Matthew's v2 where the get code gets a kvm passed in instead of implicitly retrieving group
ref_lockinternally. In that case the get/put helpers only contain device logic thus fit invfio_main.c.with v3 then they have to be in
group.csince we don't want to use group fields invfio_main.c.
Quoted 6 lines
but I still think v2 of the helpers is slightly better. The only difference between cdev and group when handling this race is using different
ref_lock. the symbol get/put part is exactly same. So even if we merge v3 like this, very likely Yi has to change it back to v2 style to share the get/put helpers while just leaving theref_lockpart handled differently between the two path.
I'm not really a fan of the asymmetry of the v2 version where the get helper needs to be called under the new kvm_ref_lock, but the put helper does not. Having the get helper handle that makes the caller much cleaner. Thanks,
Alex
Quoted 31 lines
From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 7:13 AMQuoted 18 lines · Kevin
Quoted 9 lines
From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 3:42 AMLGTM. I'm not sure moving the functions to
vfio_mainreally buys us anything since we're making so much use of group fields. The cdev approach will necessarily be different, so the bulk of the get code will likely need to move back togroup.canyway.well my last comment was based on Matthew's v2 where the get code gets a kvm passed in instead of implicitly retrieving group
ref_lockinternally. In that case the get/put helpers only contain device logic thus fit invfio_main.c.with v3 then they have to be in
group.csince we don't want to use group fields invfio_main.c.Quoted 6 lines
but I still think v2 of the helpers is slightly better. The only difference between cdev and group when handling this race is using different
ref_lock. the symbol get/put part is exactly same. So even if we merge v3 like this, very likely Yi has to change it back to v2 style to share the get/put helpers while just leaving theref_lockpart handled differently between the two path.
Quoted 5 lines
I'm not really a fan of the asymmetry of the v2 version where the get helper needs to be called under the new
kvm_ref_lock, but the put helper does not. Having the get helper handle that makes the caller much cleaner. Thanks,
What about passing the lock pointer into the helper? it's still slightly asymmetry as the put helper doesn't carry the lock pointer but it could also be interpreted as if the pointer has been saved in the get then if it needs to be referenced by the put there is no need to pass it in again.
Quoted 40 lines
From: Tian, Kevin <[email protected]>
Sent: Friday, February 3, 2023 10:00 AMQuoted 31 lines
From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 7:13 AMQuoted 18 lines · Kevin
Quoted 9 lines
From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 3:42 AMLGTM. I'm not sure moving the functions to
vfio_mainreally buys us anything since we're making so much use of group fields. The cdev approach will necessarily be different, so the bulk of the get code will likely need to move back togroup.canyway.well my last comment was based on Matthew's v2 where the get code gets a kvm passed in instead of implicitly retrieving group
ref_lockinternally. In that case the get/put helpers only contain device logic thus fit invfio_main.c.with v3 then they have to be in
group.csince we don't want to use group fields invfio_main.c.Quoted 6 lines
but I still think v2 of the helpers is slightly better. The only difference between cdev and group when handling this race is using different
ref_lock. the symbol get/put part is exactly same. So even if we merge v3 like this, very likely Yi has to change it back to v2 style to share the get/put helpers while just leaving theref_lockpart handled differently between the two path.Quoted 5 lines
I'm not really a fan of the asymmetry of the v2 version where the get helper needs to be called under the new
kvm_ref_lock, but the put helper does not. Having the get helper handle that makes the caller much cleaner. Thanks,
Quoted 5 lines
What about passing the lock pointer into the helper? it's still slightly asymmetry as the put helper doesn't carry the lock pointer but it could also be interpreted as if the pointer has been saved in the get then if it needs to be referenced by the put there is no need to pass it in again.
For cdev, I may modify vfio_device_get_kvm_safe() to accept struct kvm and let its caller hold a kvm_ref_lock (field within struct vfio_device_file). Meanwhile, the group path holds the group->kvm_ref_lock before invoking vfio_device_get_kvm_safe(). vfio_device_get_kvm_safe() just includes the symbol get/put and the device->kvm and put_kvm set.
Regards,
Yi Liu
Quoted 46 lines · Yi L
Quoted 40 lines
From: Tian, Kevin <[email protected]>
Sent: Friday, February 3, 2023 10:00 AMQuoted 31 lines
From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 7:13 AMQuoted 18 lines · Kevin
Quoted 9 lines
From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 3:42 AMLGTM. I'm not sure moving the functions to
vfio_mainreally buys us anything since we're making so much use of group fields. The cdev approach will necessarily be different, so the bulk of the get code will likely need to move back togroup.canyway.well my last comment was based on Matthew's v2 where the get code gets a kvm passed in instead of implicitly retrieving group
ref_lockinternally. In that case the get/put helpers only contain device logic thus fit invfio_main.c.with v3 then they have to be in
group.csince we don't want to use group fields invfio_main.c.Quoted 6 lines
but I still think v2 of the helpers is slightly better. The only difference between cdev and group when handling this race is using different
ref_lock. the symbol get/put part is exactly same. So even if we merge v3 like this, very likely Yi has to change it back to v2 style to share the get/put helpers while just leaving theref_lockpart handled differently between the two path.Quoted 5 lines
I'm not really a fan of the asymmetry of the v2 version where the get helper needs to be called under the new
kvm_ref_lock, but the put helper does not. Having the get helper handle that makes the caller much cleaner. Thanks,Quoted 5 lines
What about passing the lock pointer into the helper? it's still slightly asymmetry as the put helper doesn't carry the lock pointer but it could also be interpreted as if the pointer has been saved in the get then if it needs to be referenced by the put there is no need to pass it in again.
Quoted 6 lines
For cdev, I may modify
vfio_device_get_kvm_safe()to accept struct kvm and let its caller hold akvm_ref_lock(field within structvfio_device_file). Meanwhile, the group path holds thegroup->kvm_ref_lockbefore invokingvfio_device_get_kvm_safe().vfio_device_get_kvm_safe()just includes the symbol get/put and thedevice->kvmandput_kvmset.
Sounds a lot like v2 :-\ I'd look more towards group and cdev specific helpers that handle the locking so that the callers aren't exposed to the asymmetry of get vs put, and reduce a new _vfio_device_get_kvm_safe() in common code that only does the symbol work. Thanks,
Alex
Quoted 55 lines
From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 9:50 PMQuoted 6 lines · Yi L
Quoted 6 lines
From: Tian, Kevin <[email protected]>
Sent: Friday, February 3, 2023 10:00 AMQuoted 3 lines
From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 7:13 AMQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines · Kevin
Quoted 5 lines
From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 3:42 AMLGTM. I'm not sure moving the functions to
vfio_mainreally buysus
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
anything since we're making so much use of group fields. The cdev approach will necessarily be different, so the bulk of the get code
will
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
likely need to move back to
group.canyway.well my last comment was based on Matthew's v2 where the get
code
Quoted 26 lines
Quoted 25 lines
Quoted 14 lines
Quoted 7 lines
gets a kvm passed in instead of implicitly retrieving group
ref_lockinternally. In that case the get/put helpers only contain device logic thus fit invfio_main.c.with v3 then they have to be in
group.csince we don't want to use group fields invfio_main.c.Quoted 6 lines
but I still think v2 of the helpers is slightly better. The only difference between cdev and group when handling this race is using different
ref_lock. the symbol get/put part is exactly same. So even if we merge v3 like this, very likely Yi has to change it back to v2 style to share the get/put helpers while just leaving theref_lockpart handled differently between the two path.Quoted 5 lines
I'm not really a fan of the asymmetry of the v2 version where the get helper needs to be called under the new
kvm_ref_lock, but the put helper does not. Having the get helper handle that makes the caller much cleaner. Thanks,What about passing the lock pointer into the helper? it's still slightly asymmetry as the put helper doesn't carry the lock pointer but it could also be interpreted as if the pointer has been saved in the get then if it needs to be referenced by the put there is no need to pass it in again.
Quoted 8 lines
Quoted 6 lines
For cdev, I may modify
vfio_device_get_kvm_safe()to accept struct kvm and let its caller hold akvm_ref_lock(field within structvfio_device_file). Meanwhile, the group path holds thegroup->kvm_ref_lockbefore invokingvfio_device_get_kvm_safe().vfio_device_get_kvm_safe()just includes the symbol get/put and thedevice->kvmandput_kvmset.Sounds a lot like v2 :-\
Yes, like v2. 😊
Quoted 5 lines
I'd look more towards group and cdev specific
helpers that handle the locking so that the callers aren't exposed to the asymmetry of get vs put, and reduce a new_vfio_device_get_kvm_safe()in common code that only does the symbol work. Thanks,
If so, looks like Matthew needs a v4. I'm waiting for the final version of this patch and sending a new cdev series based on it. wish to see it soon ^_^.
Regards,
Yi Liu
Quoted 72 lines · Yi L
Quoted 55 lines
From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 9:50 PMQuoted 6 lines · Yi L
Quoted 6 lines
From: Tian, Kevin <[email protected]>
Sent: Friday, February 3, 2023 10:00 AMQuoted 3 lines
From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 7:13 AMQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines · Kevin
Quoted 5 lines
From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 3:42 AMLGTM. I'm not sure moving the functions to
vfio_mainreally buysus
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
anything since we're making so much use of group fields. The cdev approach will necessarily be different, so the bulk of the get code
will
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
likely need to move back to
group.canyway.well my last comment was based on Matthew's v2 where the get
code
Quoted 26 lines
Quoted 25 lines
Quoted 14 lines
Quoted 7 lines
gets a kvm passed in instead of implicitly retrieving group
ref_lockinternally. In that case the get/put helpers only contain device logic thus fit invfio_main.c.with v3 then they have to be in
group.csince we don't want to use group fields invfio_main.c.Quoted 6 lines
but I still think v2 of the helpers is slightly better. The only difference between cdev and group when handling this race is using different
ref_lock. the symbol get/put part is exactly same. So even if we merge v3 like this, very likely Yi has to change it back to v2 style to share the get/put helpers while just leaving theref_lockpart handled differently between the two path.Quoted 5 lines
I'm not really a fan of the asymmetry of the v2 version where the get helper needs to be called under the new
kvm_ref_lock, but the put helper does not. Having the get helper handle that makes the caller much cleaner. Thanks,What about passing the lock pointer into the helper? it's still slightly asymmetry as the put helper doesn't carry the lock pointer but it could also be interpreted as if the pointer has been saved in the get then if it needs to be referenced by the put there is no need to pass it in again.
Quoted 8 lines
Quoted 6 lines
For cdev, I may modify
vfio_device_get_kvm_safe()to accept struct kvm and let its caller hold akvm_ref_lock(field within structvfio_device_file). Meanwhile, the group path holds thegroup->kvm_ref_lockbefore invokingvfio_device_get_kvm_safe().vfio_device_get_kvm_safe()just includes the symbol get/put and thedevice->kvmandput_kvmset.Sounds a lot like v2 :-\
Yes, like v2. 😊
Quoted 5 lines
I'd look more towards group and cdev specific
helpers that handle the locking so that the callers aren't exposed to the asymmetry of get vs put, and reduce a new_vfio_device_get_kvm_safe()in common code that only does the symbol work. Thanks,
Quoted 3 lines
If so, looks like Matthew needs a v4. I'm waiting for the final version of this patch and sending a new cdev series based on it. wish to see it soon ^_^.
cdev support is a future feature, why does it become a requirement for a fix to the current base? The refactoring could also happen in the cdev series. Thanks,
Alex
Quoted 79 lines · Alex Williamson
Quoted 72 lines · Yi L
Quoted 55 lines
From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 9:50 PMQuoted 6 lines · Yi L
Quoted 6 lines
From: Tian, Kevin <[email protected]>
Sent: Friday, February 3, 2023 10:00 AMQuoted 3 lines
From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 7:13 AMQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines · Kevin
Quoted 5 lines
From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 3:42 AMLGTM. I'm not sure moving the functions to
vfio_mainreally buysus
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
anything since we're making so much use of group fields. The cdev approach will necessarily be different, so the bulk of the get code
will
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
likely need to move back to
group.canyway.well my last comment was based on Matthew's v2 where the get
code
Quoted 26 lines
Quoted 25 lines
Quoted 14 lines
Quoted 7 lines
gets a kvm passed in instead of implicitly retrieving group
ref_lockinternally. In that case the get/put helpers only contain device logic thus fit invfio_main.c.with v3 then they have to be in
group.csince we don't want to use group fields invfio_main.c.Quoted 6 lines
but I still think v2 of the helpers is slightly better. The only difference between cdev and group when handling this race is using different
ref_lock. the symbol get/put part is exactly same. So even if we merge v3 like this, very likely Yi has to change it back to v2 style to share the get/put helpers while just leaving theref_lockpart handled differently between the two path.Quoted 5 lines
I'm not really a fan of the asymmetry of the v2 version where the get helper needs to be called under the new
kvm_ref_lock, but the put helper does not. Having the get helper handle that makes the caller much cleaner. Thanks,What about passing the lock pointer into the helper? it's still slightly asymmetry as the put helper doesn't carry the lock pointer but it could also be interpreted as if the pointer has been saved in the get then if it needs to be referenced by the put there is no need to pass it in again.
Quoted 8 lines
Quoted 6 lines
For cdev, I may modify
vfio_device_get_kvm_safe()to accept struct kvm and let its caller hold akvm_ref_lock(field within structvfio_device_file). Meanwhile, the group path holds thegroup->kvm_ref_lockbefore invokingvfio_device_get_kvm_safe().vfio_device_get_kvm_safe()just includes the symbol get/put and thedevice->kvmandput_kvmset.Sounds a lot like v2 :-\
Yes, like v2. 😊
Quoted 5 lines
I'd look more towards group and cdev specific
helpers that handle the locking so that the callers aren't exposed to the asymmetry of get vs put, and reduce a new_vfio_device_get_kvm_safe()in common code that only does the symbol work. Thanks,Quoted 3 lines
If so, looks like Matthew needs a v4. I'm waiting for the final version of this patch and sending a new cdev series based on it. wish to see it soon ^_^.
Quoted 6 lines
cdev support is a future feature, why does it become a requirement for a fix to the current base? The refactoring could also happen in the cdev series. Thanks,
Alex
FWIW, that would bloat the fix by a bit and it's already a decent-sized fix... I took a quick stab and such a v4 would end up with a total of 120 insertions(+), 15 deletions(-). See below for a diff of what I tried on top of v3. The idea would be to move the serialization and setting/clearing of device->kvm into group/cdev and use device->kvm as the value within vfio_device_get_kvm_safe for getting the kvm ref. Wrappers in group/cdev would then be responsible for backing that device->kvm setting out on ref failure. Each side (group/cdev) can wrap their own serialization / load device->kvm from the appropriate location / handle the failed get / clear device->kvm when done (since get doesn't set it, put shouldn't clear it).
If Alex wants, I can wrap something like this into a v4 -- Or, if we don't want to include this kind of infrastructure in the fix, then Yi please feel free to use it as a starting point for what you need in cdev.
Thanks,
Matt
drivers/vfio/group.c +28 -3
@@ -154,6 +154,31 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,return ret; } +static void vfio_device_group_get_kvm (struct vfio_device *device) +{ +lockdep_assert_held (&device->dev_set->lock); + +spin_lock (&device->group->kvm_ref_lock); + +if (!device->group->kvm) + goto unlock; + + device->kvm = device->group->kvm; +if (!vfio_device_get_kvm_safe (device)) + device->kvm =NULL ; + +unlock: +spin_unlock (&device->group->kvm_ref_lock); +} + +static void vfio_device_group_put_kvm (struct vfio_device *device) +{ +lockdep_assert_held (&device->dev_set->lock); + +vfio_device_put_kvm (device); + device->kvm =NULL ; +} +static int vfio_device_group_open (struct vfio_device *device) {int ret;
@@ -173,12 +198,12 @@ static int vfio_device_group_open(struct vfio_device *device)* the pointer in the device for use by drivers. */ if (device->open_count ==0 ) -vfio_device_get_kvm_safe (device); +vfio_device_group_get_kvm (device); ret =vfio_device_open (device, device->group->iommufd, device->kvm);if (device->open_count ==0 ) -vfio_device_put_kvm (device); +vfio_device_group_put_kvm (device);mutex_unlock (&device->dev_set->lock);
@@ -195,7 +220,7 @@ void vfio_device_group_close(struct vfio_device *device)vfio_device_close (device, device->group->iommufd);if (device->open_count ==0 ) -vfio_device_put_kvm (device); +vfio_device_group_put_kvm (device);mutex_unlock (&device->dev_set->lock);mutex_unlock (&device->group->group_lock);
drivers/vfio/vfio.h +3 -2
@@ -253,11 +253,12 @@ enum { vfio_noiommu = false };#endif #ifdef CONFIG_HAVE_KVM -void vfio_device_get_kvm_safe (struct vfio_device *device); +bool vfio_device_get_kvm_safe (struct vfio_device *device);void vfio_device_put_kvm (struct vfio_device *device);#else -static inline void vfio_device_get_kvm_safe (struct vfio_device *device) +static inline bool vfio_device_get_kvm_safe (struct vfio_device *device) { +return false; }static inline void vfio_device_put_kvm (struct vfio_device *device)
drivers/vfio/vfio_main.c +7 -16
@@ -342,7 +342,7 @@ void vfio_unregister_group_dev(struct vfio_device *device)EXPORT_SYMBOL_GPL (vfio_unregister_group_dev);#ifdef CONFIG_HAVE_KVM -void vfio_device_get_kvm_safe (struct vfio_device *device) +bool vfio_device_get_kvm_safe (struct vfio_device *device) {void (*pfn)(struct kvm *kvm);bool (*fn)(struct kvm *kvm);
@@ -350,32 +350,26 @@ void vfio_device_get_kvm_safe(struct vfio_device *device)lockdep_assert_held (&device->dev_set->lock); -spin_lock (&device->group->kvm_ref_lock); -if (!device->group->kvm) - goto unlock; - pfn =symbol_get (kvm_put_kvm);if (WARN_ON (!pfn)) - goto unlock; +return false; fn =symbol_get (kvm_get_kvm_safe);if (WARN_ON (!fn)) {symbol_put (kvm_put_kvm); - goto unlock; +return false; } - ret =fn (device->group->kvm); + ret =fn (device->kvm);symbol_put (kvm_get_kvm_safe);if (!ret) {symbol_put (kvm_put_kvm); - goto unlock; +return false; } device->put_kvm = pfn; - device->kvm = device->group->kvm; -unlock: -spin_unlock (&device->group->kvm_ref_lock); +return true; }void vfio_device_put_kvm (struct vfio_device *device)
@@ -386,14 +380,11 @@ void vfio_device_put_kvm(struct vfio_device *device)return ;if (WARN_ON (!device->put_kvm)) - goto clear; +return ; device->put_kvm (device->kvm); device->put_kvm =NULL ; symbol_put(kvm_put_kvm); - -clear: - device->kvm = NULL; }#endif
Quoted 104 lines · On Fri, 3 Feb 2023 12:29:01 -0500 Matthew Rosato <[email protected]> wrote:
Quoted 79 lines · Alex Williamson
Quoted 72 lines · Yi L
Quoted 55 lines
From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 9:50 PMQuoted 6 lines · Yi L
Quoted 6 lines
From: Tian, Kevin <[email protected]>
Sent: Friday, February 3, 2023 10:00 AMQuoted 3 lines
From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 7:13 AMQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines · Kevin
Quoted 5 lines
From: Alex Williamson <[email protected]>
Sent: Friday, February 3, 2023 3:42 AMLGTM. I'm not sure moving the functions to
vfio_mainreally buysus
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
anything since we're making so much use of group fields. The cdev approach will necessarily be different, so the bulk of the get code
will
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
likely need to move back to
group.canyway.well my last comment was based on Matthew's v2 where the get
code
Quoted 26 lines
Quoted 25 lines
Quoted 14 lines
Quoted 7 lines
gets a kvm passed in instead of implicitly retrieving group
ref_lockinternally. In that case the get/put helpers only contain device logic thus fit invfio_main.c.with v3 then they have to be in
group.csince we don't want to use group fields invfio_main.c.Quoted 6 lines
but I still think v2 of the helpers is slightly better. The only difference between cdev and group when handling this race is using different
ref_lock. the symbol get/put part is exactly same. So even if we merge v3 like this, very likely Yi has to change it back to v2 style to share the get/put helpers while just leaving theref_lockpart handled differently between the two path.Quoted 5 lines
I'm not really a fan of the asymmetry of the v2 version where the get helper needs to be called under the new
kvm_ref_lock, but the put helper does not. Having the get helper handle that makes the caller much cleaner. Thanks,What about passing the lock pointer into the helper? it's still slightly asymmetry as the put helper doesn't carry the lock pointer but it could also be interpreted as if the pointer has been saved in the get then if it needs to be referenced by the put there is no need to pass it in again.
Quoted 8 lines
Quoted 6 lines
For cdev, I may modify
vfio_device_get_kvm_safe()to accept struct kvm and let its caller hold akvm_ref_lock(field within structvfio_device_file). Meanwhile, the group path holds thegroup->kvm_ref_lockbefore invokingvfio_device_get_kvm_safe().vfio_device_get_kvm_safe()just includes the symbol get/put and thedevice->kvmandput_kvmset.Sounds a lot like v2 :-\
Yes, like v2. 😊
Quoted 5 lines
I'd look more towards group and cdev specific
helpers that handle the locking so that the callers aren't exposed to the asymmetry of get vs put, and reduce a new_vfio_device_get_kvm_safe()in common code that only does the symbol work. Thanks,Quoted 3 lines
If so, looks like Matthew needs a v4. I'm waiting for the final version of this patch and sending a new cdev series based on it. wish to see it soon ^_^.
Quoted 6 lines
cdev support is a future feature, why does it become a requirement for a fix to the current base? The refactoring could also happen in the cdev series. Thanks,
Alex
FWIW, that would bloat the fix by a bit and it's already a decent-sized fix... I took a quick stab and such a v4 would end up with a total of 120 insertions(+), 15 deletions(-). See below for a diff of what I tried on top of v3. The idea would be to move the serialization and setting/clearing of
device->kvminto group/cdev and usedevice->kvmas the value withinvfio_device_get_kvm_safefor getting the kvm ref. Wrappers in group/cdev would then be responsible for backing thatdevice->kvmsetting out on ref failure. Each side (group/cdev) can wrap their own serialization / loaddevice->kvmfrom the appropriate location / handle the failed get / cleardevice->kvmwhen done (since get doesn't set it, put shouldn't clear it).If Alex wants, I can wrap something like this into a v4 -- Or, if we don't want to include this kind of infrastructure in the fix, then Yi please feel free to use it as a starting point for what you need in cdev.
Thanks,
Mattdrivers/vfio/group.c +1
-0@@ -154,6 +154,31 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,return ret; } +static void vfio_device_group_get_kvm (struct vfio_device *device)
drivers/vfio/group.c static int vfio_group_ioctl_set_container(struct vfio_group *group, · line 167
Quoted 10 lines
drivers/vfio/group.c +10
-0+{ +lockdep_assert_held (&device->dev_set->lock); + +spin_lock (&device->group->kvm_ref_lock); + +if (!device->group->kvm) + goto unlock; + + device->kvm = device->group->kvm; +if (!vfio_device_get_kvm_safe (device))
I'd probably go back to making this:
void _vfio_device_get_kvm_safe(struct vfio_device *device, struct kvm *kvm);
so the vfio_main function would handle setting and clearing device->kvm. That way we could also move the lockdep into the vfio_main functions. Once we do that, there's no reason to have a group vs cdev put function and we end up with only a wrapper on the get function, which should really never be used directly, so we prefix it with an underscore.
At that point (see incremental diff below), it's about a wash. Current v3:
drivers/vfio/group.c | 32 +++++++++++++---- drivers/vfio/vfio.h | 14 +++++++ drivers/vfio/vfio_main.c | 70 +++++++++++++++++++++++++++++++++++---- include/linux/vfio.h | 2 - 4 files changed, 103 insertions(+), 15 deletions(-)
Folding in below:
drivers/vfio/group.c | 44 ++++++++++++++++++++++----- drivers/vfio/vfio.h | 15 +++++++++ drivers/vfio/vfio_main.c | 63 ++++++++++++++++++++++++++++++++++----- include/linux/vfio.h | 2 - 4 files changed, 109 insertions(+), 15 deletions(-)
Unfortunately it seems I've talked myself into the answer that we should maybe just pre-enable cdev by not adding a group reference in vfio_main. Thanks,
Alex
drivers/vfio/group.c +13 -1
@@ -154,6 +154,18 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,return ret; } +static void vfio_device_group_get_kvm_safe (struct vfio_device *device) +{ +spin_lock (&device->group->kvm_ref_lock); +if (!device->group->kvm) + goto unlock; + +_vfio_device_get_kvm_safe (device, device->group->kvm); + +unlock: +spin_unlock (&device->group->kvm_ref_lock); +} +static int vfio_device_group_open (struct vfio_device *device) {int ret;
@@ -173,7 +185,7 @@ static int vfio_device_group_open(struct vfio_device *device)* the pointer in the device for use by drivers. */ if (device->open_count ==0 ) -vfio_device_get_kvm_safe (device); +vfio_device_group_get_kvm_safe (device); ret =vfio_device_open (device, device->group->iommufd, device->kvm);
drivers/vfio/vfio.h +3 -2
@@ -253,10 +253,11 @@ enum { vfio_noiommu = false };#endif #ifdef CONFIG_HAVE_KVM -void vfio_device_get_kvm_safe (struct vfio_device *device); +void _vfio_device_get_kvm_safe (struct vfio_device *device,struct kvm *kvm);void vfio_device_put_kvm (struct vfio_device *device);#else -static inline void vfio_device_get_kvm_safe (struct vfio_device *device) +static inline void _vfio_device_get_kvm_safe (struct vfio_device *device, +struct kvm *kvm) { }
drivers/vfio/vfio_main.c +5 -12
@@ -342,7 +342,7 @@ void vfio_unregister_group_dev(struct vfio_device *device)EXPORT_SYMBOL_GPL (vfio_unregister_group_dev);#ifdef CONFIG_HAVE_KVM -void vfio_device_get_kvm_safe (struct vfio_device *device) +void _vfio_device_get_kvm_safe (struct vfio_device *device,struct kvm *kvm) {void (*pfn)(struct kvm *kvm);bool (*fn)(struct kvm *kvm);
@@ -350,32 +350,25 @@ void vfio_device_get_kvm_safe(struct vfio_device *device)lockdep_assert_held (&device->dev_set->lock); -spin_lock (&device->group->kvm_ref_lock); -if (!device->group->kvm) - goto unlock; - pfn =symbol_get (kvm_put_kvm);if (WARN_ON (!pfn)) - goto unlock; +return ; fn =symbol_get (kvm_get_kvm_safe);if (WARN_ON (!fn)) {symbol_put (kvm_put_kvm); - goto unlock; +return ; } ret =fn (device->group->kvm);symbol_put (kvm_get_kvm_safe);if (!ret) {symbol_put (kvm_put_kvm); - goto unlock; +return ; } device->put_kvm = pfn; - device->kvm = device->group->kvm; - -unlock: -spin_unlock (&device->group->kvm_ref_lock); + device->kvm = kvm; }void vfio_device_put_kvm (struct vfio_device *device)
Quoted 2 lines · Alex Williamson
On Fri, 3 Feb 2023 12:29:01 -0500
Matthew Rosato <[email protected]> wrote:
...
Quoted 106 lines
I'd probably go back to making this:
void
_vfio_device_get_kvm_safe(struct vfio_device *device, struct kvm *kvm);so the
vfio_mainfunction would handle setting and clearingdevice->kvm. That way we could also move the lockdep into thevfio_mainfunctions. Once we do that, there's no reason to have a group vs cdev put function and we end up with only a wrapper on the get function, which should really never be used directly, so we prefix it with an underscore.At that point (see incremental diff below), it's about a wash. Current v3:
drivers/vfio/group.c | 32 +++++++++++++---- drivers/vfio/vfio.h | 14 +++++++ drivers/vfio/vfio_main.c | 70 +++++++++++++++++++++++++++++++++++---- include/linux/vfio.h | 2 - 4 files changed, 103 insertions(+), 15 deletions(-)Folding in below:
drivers/vfio/group.c | 44 ++++++++++++++++++++++----- drivers/vfio/vfio.h | 15 +++++++++ drivers/vfio/vfio_main.c | 63 ++++++++++++++++++++++++++++++++++----- include/linux/vfio.h | 2 - 4 files changed, 109 insertions(+), 15 deletions(-)Unfortunately it seems I've talked myself into the answer that we should maybe just pre-enable cdev by not adding a group reference in
vfio_main. Thanks,Alex
drivers/vfio/group.c +13
-1@@ -154,6 +154,18 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,return ret; } +static void vfio_device_group_get_kvm_safe (struct vfio_device *device) +{ +spin_lock (&device->group->kvm_ref_lock); +if (!device->group->kvm) + goto unlock; + +_vfio_device_get_kvm_safe (device, device->group->kvm); + +unlock: +spin_unlock (&device->group->kvm_ref_lock); +} +static int vfio_device_group_open (struct vfio_device *device) {int ret;@@ -173,7 +185,7 @@ static int vfio_device_group_open(struct vfio_device *device)* the pointer in the device for use by drivers. */ if (device->open_count ==0 ) -vfio_device_get_kvm_safe (device); +vfio_device_group_get_kvm_safe (device); ret =vfio_device_open (device, device->group->iommufd, device->kvm);drivers/vfio/vfio.h +3
-2@@ -253,10 +253,11 @@ enum { vfio_noiommu = false };#endif #ifdef CONFIG_HAVE_KVM -void vfio_device_get_kvm_safe (struct vfio_device *device); +void _vfio_device_get_kvm_safe (struct vfio_device *device,struct kvm *kvm);void vfio_device_put_kvm (struct vfio_device *device);#else -static inline void vfio_device_get_kvm_safe (struct vfio_device *device) +static inline void _vfio_device_get_kvm_safe (struct vfio_device *device, +struct kvm *kvm) { }drivers/vfio/vfio_main.c +1
-5@@ -342,7 +342,7 @@ void vfio_unregister_group_dev(struct vfio_device *device)EXPORT_SYMBOL_GPL (vfio_unregister_group_dev);#ifdef CONFIG_HAVE_KVM -void vfio_device_get_kvm_safe (struct vfio_device *device) +void _vfio_device_get_kvm_safe (struct vfio_device *device,struct kvm *kvm) {void (*pfn)(struct kvm *kvm);bool (*fn)(struct kvm *kvm);@@ -350,32 +350,25 @@ void vfio_device_get_kvm_safe(struct vfio_device *device)lockdep_assert_held (&device->dev_set->lock); -spin_lock (&device->group->kvm_ref_lock); -if (!device->group->kvm) - goto unlock; - pfn =symbol_get (kvm_put_kvm);if (WARN_ON (!pfn))
drivers/vfio/vfio_main.c void vfio_device_get_kvm_safe(struct vfio_device *device) · line 362
Quoted 10 lines
drivers/vfio/vfio_main.c +2
-2- goto unlock; +return ; fn =symbol_get (kvm_get_kvm_safe);if (WARN_ON (!fn)) {symbol_put (kvm_put_kvm); - goto unlock; +return ; } > ret = fn(device->group->kvm);
s/device->group->kvm/kvm/
With that small change, this looks good to me too (and testing looks good too). Do you want me to send a v4 for one last round of review?
Thanks,
Matt
Quoted 13 lines · On Fri, 3 Feb 2023 16:19:10 -0500 Matthew Rosato <[email protected]> wrote:
Quoted 13 lines
drivers/vfio/vfio_main.c +1
-5@@ -350,32 +350,25 @@ void vfio_device_get_kvm_safe(struct vfio_device *device)lockdep_assert_held (&device->dev_set->lock); -spin_lock (&device->group->kvm_ref_lock); -if (!device->group->kvm) - goto unlock; - pfn =symbol_get (kvm_put_kvm);if (WARN_ON (!pfn)) - goto unlock; +return ;
Quoted 9 lines
drivers/vfio/vfio_main.c void vfio_device_get_kvm_safe(struct vfio_device *device)
Quoted 7 lines
drivers/vfio/vfio_main.c +1
-1fn =symbol_get (kvm_get_kvm_safe);if (WARN_ON (!fn)) {symbol_put (kvm_put_kvm); - goto unlock; +return ; } > ret = fn(device->group->kvm);s/device->group->kvm/kvm/
Oops, yes.
Quoted 2 lines
With that small change, this looks good to me too (and testing looks good too). Do you want me to send a v4 for one last round of review?
Please do. Thanks,
Alex
Patchwork
CI[Intel-gfx] ✓ Fi.CI.IGT: success for vfio: fix deadlock between group lock and kvm lock (rev5)
In reply to Matthew Rosato== Series Details ==
== Series Details ==
Series: vfio: fix deadlock between group lock and kvm lock (rev5)
URL : https://patchwork.freedesktop.org/series/113535/
State : success
== Summary ==
CI Bug Log - changes from CI_DRM_12684_full -> Patchwork_113535v5_full
====================================================
Summary
-------
**SUCCESS**
No regressions found.
External URL: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/index.html
Participating hosts (10 -> 10)
------------------------------
No changes in participating hosts
Possible new issues
-------------------
Here are the unknown changes that may have been introduced in Patchwork_113535v5_full:
### IGT changes ###
#### Suppressed ####
The following results come from untrusted machines, tests, or statuses. They do not affect the overall result.
* igt@i915_pm_rpm@gem-execbuf@smem0:
- {shard-tglu}: [PASS][1] -> [FAIL][2]
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/shard-tglu-7/igt@i915_pm_rpm@[email protected]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/shard-tglu-8/igt@i915_pm_rpm@[email protected]
* igt@perf_pmu@module-unload:
- {shard-tglu}: [PASS][3] -> [DMESG-WARN][4]
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/shard-tglu-7/igt@[email protected]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/shard-tglu-8/igt@[email protected]
Known issues
------------
Here are the changes found in Patchwork_113535v5_full that come from known issues:
### IGT changes ###
#### Issues hit ####
* igt@gem_exec_fair@basic-deadline:
- shard-glk: [PASS][5] -> [FAIL][6] ([i915#2846])
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/shard-glk1/igt@[email protected]
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/shard-glk9/igt@[email protected]
* igt@gem_exec_fair@basic-pace-share@rcs0:
- shard-glk: [PASS][7] -> [FAIL][8] ([i915#2842])
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/shard-glk2/igt@gem_exec_fair@[email protected]
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/shard-glk8/igt@gem_exec_fair@[email protected]
* igt@gen9_exec_parse@allowed-single:
- shard-glk: [PASS][9] -> [ABORT][10] ([i915#5566])
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/shard-glk6/igt@[email protected]
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/shard-glk3/igt@[email protected]
* igt@kms_cursor_legacy@flip-vs-cursor@atomic-transitions:
- shard-glk: [PASS][11] -> [FAIL][12] ([i915#2346])
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/shard-glk2/igt@kms_cursor_legacy@[email protected]
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/shard-glk6/igt@kms_cursor_legacy@[email protected]
#### Possible fixes ####
* igt@gem_ctx_isolation@preservation-s3@bcs0:
- {shard-rkl}: [DMESG-WARN][13] ([i915#5122]) -> [PASS][14]
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/shard-rkl-4/igt@gem_ctx_isolation@[email protected]
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/shard-rkl-1/igt@gem_ctx_isolation@[email protected]
* igt@gem_ctx_isolation@preservation-s3@rcs0:
- {shard-rkl}: [ABORT][15] -> [PASS][16]
[15]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/shard-rkl-4/igt@gem_ctx_isolation@[email protected]
[16]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/shard-rkl-1/igt@gem_ctx_isolation@[email protected]
* igt@gem_ctx_persistence@many-contexts:
- {shard-rkl}: [INCOMPLETE][17] -> [PASS][18]
[17]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/shard-rkl-3/igt@[email protected]
[18]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/shard-rkl-4/igt@[email protected]
* igt@gem_exec_fair@basic-none@vcs0:
- {shard-rkl}: [FAIL][19] ([i915#2842]) -> [PASS][20] +3 similar issues
[19]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/shard-rkl-1/igt@gem_exec_fair@[email protected]
[20]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/shard-rkl-5/igt@gem_exec_fair@[email protected]
* igt@gem_exec_reloc@basic-wc-gtt:
- {shard-rkl}: [SKIP][21] ([i915#3281]) -> [PASS][22] +4 similar issues
[21]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/shard-rkl-3/igt@[email protected]
[22]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/shard-rkl-5/igt@[email protected]
* igt@gem_set_tiling_vs_pwrite:
- {shard-rkl}: [SKIP][23] ([i915#3282]) -> [PASS][24] +2 similar issues
[23]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/shard-rkl-1/igt@gem_set_tiling_vs_pwrite.html
[24]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/shard-rkl-5/igt@gem_set_tiling_vs_pwrite.html
* igt@gen9_exec_parse@bb-start-param:
- {shard-rkl}: [SKIP][25] ([i915#2527]) -> [PASS][26] +1 similar issue
[25]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/shard-rkl-1/igt@[email protected]
[26]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/shard-rkl-5/igt@[email protected]
* igt@i915_pm_dc@dc5-psr:
- {shard-rkl}: [SKIP][27] ([i915#658]) -> [PASS][28]
[27]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/shard-rkl-2/igt@[email protected]
[28]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/shard-rkl-6/igt@[email protected]
* igt@i915_pm_rpm@modeset-lpsp-stress:
- {shard-dg1}: [SKIP][29] ([i915#1397]) -> [PASS][30]
[29]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/shard-dg1-16/igt@[email protected]
[30]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/shard-dg1-14/igt@[email protected]
* igt@i915_pm_sseu@full-enable:
- {shard-rkl}: [SKIP][31] ([i915#4387]) -> [PASS][32]
[31]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/shard-rkl-1/igt@[email protected]
[32]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/shard-rkl-5/igt@[email protected]
* igt@kms_vblank@pipe-a-ts-continuation-idle:
- {shard-rkl}: [SKIP][33] ([i915#1845] / [i915#4098]) -> [PASS][34] +1 similar issue
[33]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/shard-rkl-2/igt@[email protected]
[34]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/shard-rkl-6/igt@[email protected]
* igt@perf_pmu@idle@rcs0:
- {shard-rkl}: [FAIL][35] ([i915#4349]) -> [PASS][36]
[35]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/shard-rkl-6/igt@perf_pmu@[email protected]
[36]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/shard-rkl-1/igt@perf_pmu@[email protected]
* igt@sysfs_timeslice_duration@timeout@rcs0:
- {shard-rkl}: [FAIL][37] ([i915#1755]) -> [PASS][38] +1 similar issue
[37]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12684/shard-rkl-3/igt@sysfs_timeslice_duration@[email protected]
[38]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/shard-rkl-3/igt@sysfs_timeslice_duration@[email protected]
{name}: This element is suppressed. This means it is ignored when computing
the status of the difference (SUCCESS, WARNING, or FAILURE).
[fdo#103375]: https://bugs.freedesktop.org/show_bug.cgi?id=103375
[fdo#109274]: https://bugs.freedesktop.org/show_bug.cgi?id=109274
[fdo#109279]: https://bugs.freedesktop.org/show_bug.cgi?id=109279
[fdo#109280]: https://bugs.freedesktop.org/show_bug.cgi?id=109280
[fdo#109283]: https://bugs.freedesktop.org/show_bug.cgi?id=109283
[fdo#109285]: https://bugs.freedesktop.org/show_bug.cgi?id=109285
[fdo#109289]: https://bugs.freedesktop.org/show_bug.cgi?id=109289
[fdo#109291]: https://bugs.freedesktop.org/show_bug.cgi?id=109291
[fdo#109295]: https://bugs.freedesktop.org/show_bug.cgi?id=109295
[fdo#109303]: https://bugs.freedesktop.org/show_bug.cgi?id=109303
[fdo#109308]: https://bugs.freedesktop.org/show_bug.cgi?id=109308
[fdo#109309]: https://bugs.freedesktop.org/show_bug.cgi?id=109309
[fdo#109313]: https://bugs.freedesktop.org/show_bug.cgi?id=109313
[fdo#109315]: https://bugs.freedesktop.org/show_bug.cgi?id=109315
[fdo#109506]: https://bugs.freedesktop.org/show_bug.cgi?id=109506
[fdo#109642]: https://bugs.freedesktop.org/show_bug.cgi?id=109642
[fdo#110189]: https://bugs.freedesktop.org/show_bug.cgi?id=110189
[fdo#110542]: https://bugs.freedesktop.org/show_bug.cgi?id=110542
[fdo#110723]: https://bugs.freedesktop.org/show_bug.cgi?id=110723
[fdo#111068]: https://bugs.freedesktop.org/show_bug.cgi?id=111068
[fdo#111614]: https://bugs.freedesktop.org/show_bug.cgi?id=111614
[fdo#111615]: https://bugs.freedesktop.org/show_bug.cgi?id=111615
[fdo#111644]: https://bugs.freedesktop.org/show_bug.cgi?id=111644
[fdo#111825]: https://bugs.freedesktop.org/show_bug.cgi?id=111825
[fdo#111827]: https://bugs.freedesktop.org/show_bug.cgi?id=111827
[fdo#112054]: https://bugs.freedesktop.org/show_bug.cgi?id=112054
[fdo#112283]: https://bugs.freedesktop.org/show_bug.cgi?id=112283
[i915#1072]: https://gitlab.freedesktop.org/drm/intel/issues/1072
[i915#132]: https://gitlab.freedesktop.org/drm/intel/issues/132
[i915#1397]: https://gitlab.freedesktop.org/drm/intel/issues/1397
[i915#1755]: https://gitlab.freedesktop.org/drm/intel/issues/1755
[i915#1825]: https://gitlab.freedesktop.org/drm/intel/issues/1825
[i915#1839]: https://gitlab.freedesktop.org/drm/intel/issues/1839
[i915#1845]: https://gitlab.freedesktop.org/drm/intel/issues/1845
[i915#1849]: https://gitlab.freedesktop.org/drm/intel/issues/1849
[i915#1850]: https://gitlab.freedesktop.org/drm/intel/issues/1850
[i915#2346]: https://gitlab.freedesktop.org/drm/intel/issues/2346
[i915#2433]: https://gitlab.freedesktop.org/drm/intel/issues/2433
[i915#2436]: https://gitlab.freedesktop.org/drm/intel/issues/2436
[i915#2437]: https://gitlab.freedesktop.org/drm/intel/issues/2437
[i915#2527]: https://gitlab.freedesktop.org/drm/intel/issues/2527
[i915#2575]: https://gitlab.freedesktop.org/drm/intel/issues/2575
[i915#2582]: https://gitlab.freedesktop.org/drm/intel/issues/2582
[i915#2587]: https://gitlab.freedesktop.org/drm/intel/issues/2587
[i915#2658]: https://gitlab.freedesktop.org/drm/intel/issues/2658
[i915#2672]: https://gitlab.freedesktop.org/drm/intel/issues/2672
[i915#2681]: https://gitlab.freedesktop.org/drm/intel/issues/2681
[i915#2705]: https://gitlab.freedesktop.org/drm/intel/issues/2705
[i915#280]: https://gitlab.freedesktop.org/drm/intel/issues/280
[i915#2842]: https://gitlab.freedesktop.org/drm/intel/issues/2842
[i915#2846]: https://gitlab.freedesktop.org/drm/intel/issues/2846
[i915#2856]: https://gitlab.freedesktop.org/drm/intel/issues/2856
[i915#2920]: https://gitlab.freedesktop.org/drm/intel/issues/2920
[i915#3063]: https://gitlab.freedesktop.org/drm/intel/issues/3063
[i915#3116]: https://gitlab.freedesktop.org/drm/intel/issues/3116
[i915#3281]: https://gitlab.freedesktop.org/drm/intel/issues/3281
[i915#3282]: https://gitlab.freedesktop.org/drm/intel/issues/3282
[i915#3291]: https://gitlab.freedesktop.org/drm/intel/issues/3291
[i915#3297]: https://gitlab.freedesktop.org/drm/intel/issues/3297
[i915#3299]: https://gitlab.freedesktop.org/drm/intel/issues/3299
[i915#3301]: https://gitlab.freedesktop.org/drm/intel/issues/3301
[i915#3359]: https://gitlab.freedesktop.org/drm/intel/issues/3359
[i915#3361]: https://gitlab.freedesktop.org/drm/intel/issues/3361
[i915#3458]: https://gitlab.freedesktop.org/drm/intel/issues/3458
[i915#3469]: https://gitlab.freedesktop.org/drm/intel/issues/3469
[i915#3539]: https://gitlab.freedesktop.org/drm/intel/issues/3539
[i915#3546]: https://gitlab.freedesktop.org/drm/intel/issues/3546
[i915#3555]: https://gitlab.freedesktop.org/drm/intel/issues/3555
[i915#3558]: https://gitlab.freedesktop.org/drm/intel/issues/3558
[i915#3591]: https://gitlab.freedesktop.org/drm/intel/issues/3591
[i915#3637]: https://gitlab.freedesktop.org/drm/intel/issues/3637
[i915#3638]: https://gitlab.freedesktop.org/drm/intel/issues/3638
[i915#3689]: https://gitlab.freedesktop.org/drm/intel/issues/3689
[i915#3708]: https://gitlab.freedesktop.org/drm/intel/issues/3708
[i915#3734]: https://gitlab.freedesktop.org/drm/intel/issues/3734
[i915#3742]: https://gitlab.freedesktop.org/drm/intel/issues/3742
[i915#3840]: https://gitlab.freedesktop.org/drm/intel/issues/3840
[i915#3886]: https://gitlab.freedesktop.org/drm/intel/issues/3886
[i915#3938]: https://gitlab.freedesktop.org/drm/intel/issues/3938
[i915#3955]: https://gitlab.freedesktop.org/drm/intel/issues/3955
[i915#404]: https://gitlab.freedesktop.org/drm/intel/issues/404
[i915#4070]: https://gitlab.freedesktop.org/drm/intel/issues/4070
[i915#4077]: https://gitlab.freedesktop.org/drm/intel/issues/4077
[i915#4078]: https://gitlab.freedesktop.org/drm/intel/issues/4078
[i915#4079]: https://gitlab.freedesktop.org/drm/intel/issues/4079
[i915#4083]: https://gitlab.freedesktop.org/drm/intel/issues/4083
[i915#4098]: https://gitlab.freedesktop.org/drm/intel/issues/4098
[i915#4212]: https://gitlab.freedesktop.org/drm/intel/issues/4212
[i915#4270]: https://gitlab.freedesktop.org/drm/intel/issues/4270
[i915#4349]: https://gitlab.freedesktop.org/drm/intel/issues/4349
[i915#4387]: https://gitlab.freedesktop.org/drm/intel/issues/4387
[i915#4525]: https://gitlab.freedesktop.org/drm/intel/issues/4525
[i915#4538]: https://gitlab.freedesktop.org/drm/intel/issues/4538
[i915#4613]: https://gitlab.freedesktop.org/drm/intel/issues/4613
[i915#4812]: https://gitlab.freedesktop.org/drm/intel/issues/4812
[i915#4833]: https://gitlab.freedesktop.org/drm/intel/issues/4833
[i915#4852]: https://gitlab.freedesktop.org/drm/intel/issues/4852
[i915#4859]: https://gitlab.freedesktop.org/drm/intel/issues/4859
[i915#4860]: https://gitlab.freedesktop.org/drm/intel/issues/4860
[i915#4877]: https://gitlab.freedesktop.org/drm/intel/issues/4877
[i915#4880]: https://gitlab.freedesktop.org/drm/intel/issues/4880
[i915#5122]: https://gitlab.freedesktop.org/drm/intel/issues/5122
[i915#5176]: https://gitlab.freedesktop.org/drm/intel/issues/5176
[i915#5235]: https://gitlab.freedesktop.org/drm/intel/issues/5235
[i915#5286]: https://gitlab.freedesktop.org/drm/intel/issues/5286
[i915#5288]: https://gitlab.freedesktop.org/drm/intel/issues/5288
[i915#5289]: https://gitlab.freedesktop.org/drm/intel/issues/5289
[i915#5325]: https://gitlab.freedesktop.org/drm/intel/issues/5325
[i915#533]: https://gitlab.freedesktop.org/drm/intel/issues/533
[i915#5439]: https://gitlab.freedesktop.org/drm/intel/issues/5439
[i915#5461]: https://gitlab.freedesktop.org/drm/intel/issues/5461
[i915#5566]: https://gitlab.freedesktop.org/drm/intel/issues/5566
[i915#6095]: https://gitlab.freedesktop.org/drm/intel/issues/6095
[i915#6117]: https://gitlab.freedesktop.org/drm/intel/issues/6117
[i915#6227]: https://gitlab.freedesktop.org/drm/intel/issues/6227
[i915#6230]: https://gitlab.freedesktop.org/drm/intel/issues/6230
[i915#6245]: https://gitlab.freedesktop.org/drm/intel/issues/6245
[i915#6247]: https://gitlab.freedesktop.org/drm/intel/issues/6247
[i915#6248]: https://gitlab.freedesktop.org/drm/intel/issues/6248
[i915#6268]: https://gitlab.freedesktop.org/drm/intel/issues/6268
[i915#6301]: https://gitlab.freedesktop.org/drm/intel/issues/6301
[i915#6334]: https://gitlab.freedesktop.org/drm/intel/issues/6334
[i915#6344]: https://gitlab.freedesktop.org/drm/intel/issues/6344
[i915#6497]: https://gitlab.freedesktop.org/drm/intel/issues/6497
[i915#6524]: https://gitlab.freedesktop.org/drm/intel/issues/6524
[i915#658]: https://gitlab.freedesktop.org/drm/intel/issues/658
[i915#6590]: https://gitlab.freedesktop.org/drm/intel/issues/6590
[i915#6621]: https://gitlab.freedesktop.org/drm/intel/issues/6621
[i915#6768]: https://gitlab.freedesktop.org/drm/intel/issues/6768
[i915#6944]: https://gitlab.freedesktop.org/drm/intel/issues/6944
[i915#6946]: https://gitlab.freedesktop.org/drm/intel/issues/6946
[i915#7116]: https://gitlab.freedesktop.org/drm/intel/issues/7116
[i915#7118]: https://gitlab.freedesktop.org/drm/intel/issues/7118
[i915#7128]: https://gitlab.freedesktop.org/drm/intel/issues/7128
[i915#7294]: https://gitlab.freedesktop.org/drm/intel/issues/7294
[i915#7561]: https://gitlab.freedesktop.org/drm/intel/issues/7561
[i915#7582]: https://gitlab.freedesktop.org/drm/intel/issues/7582
[i915#7651]: https://gitlab.freedesktop.org/drm/intel/issues/7651
[i915#7697]: https://gitlab.freedesktop.org/drm/intel/issues/7697
[i915#7701]: https://gitlab.freedesktop.org/drm/intel/issues/7701
[i915#7707]: https://gitlab.freedesktop.org/drm/intel/issues/7707
[i915#7711]: https://gitlab.freedesktop.org/drm/intel/issues/7711
[i915#7742]: https://gitlab.freedesktop.org/drm/intel/issues/7742
[i915#7828]: https://gitlab.freedesktop.org/drm/intel/issues/7828
[i915#7949]: https://gitlab.freedesktop.org/drm/intel/issues/7949
Build changes
-------------
* Linux: CI_DRM_12684 -> Patchwork_113535v5
CI-20190529: 20190529
CI_DRM_12684: 57eb8680d3ef1213de1d9c607b95e522035036e6 @ git://anongit.freedesktop.org/gfx-ci/linux
IGT_7144: cda71bf809b981a646270963d6b1ccee4fd4643b @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
Patchwork_113535v5: 57eb8680d3ef1213de1d9c607b95e522035036e6 @ git://anongit.freedesktop.org/gfx-ci/linux
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113535v5/index.html
Hi Matthew,
Quoted 233 lines
From: Matthew Rosato <[email protected]>
Sent: Friday, February 3, 2023 12:25 AMAfter
51cdc8bc120e, we have another deadlock scenario between thekvm->lockand the vfiogroup_lockwith two different codepaths acquiring the locks in different order. Specifically invfio_open_device, vfio holds the vfiogroup_lockwhen issuingdevice->ops->open_devicebut some
drivers (like vfio-ap) need to acquirekvm->lockduring theiropen_deviceroutine; Meanwhile,kvm_vfio_releasewill acquire thekvm->lockfirst before callingvfio_file_set_kvmwhich will acquire the vfiogroup_lock.To resolve this, let's remove the need for the vfio
group_lockfrom thekvm_vfio_releasecodepath. This is done by introducing a new spinlock to protect modifications to the vfio group kvm pointer, and acquiring a kvm ref from within vfio while holding this spinlock, with the reference held until the last close for the device in question.Fixes 51cdc8bc120e ("kvm/vfio: Fix potential deadlock on vfio group_lock")Reported-by Anthony Krowiak <[email protected]>Suggested-by Jason Gunthorpe <[email protected]>Signed-off-by Matthew Rosato <[email protected]>---
Changes from v2:
* Relocate the new functions back tovfio_mainand externalize to call from group (Kevin) since cdev will need this too
* s/vfio_kvm_*_kvm/vfio_device_*_kvm/ and only pass device as input. Handle newkvm_ref_lockdirectly insidevfio_device_get_kvm(Alex)
* Addassert_lockdep_heldfordev_setlock (Alex)
* Internalize error paths forvfio_device_get_kvm_safeand now return void - eitherdevice->kvmis set with a ref taken or is NULL (Alex)
* Other flow suggestions to make the call path cleaner - Thanks! (Alex)
* Can't passgroup->kvmtovfio_device_open, as it references the value outside of new lock. Passdevice->kvmto minimize changes in this fix (Alex, Yi)
Changes from v1:
* usespin_lockinstead ofspin_lock_irqsave(Jason)
* cleardevice->kvm_putas part ofvfio_kvm_put_kvm(Yi)
* Re-arrange code to avoid referencing the group contents from withinvfio_main(Kevin) which meant moving most of the code in this patch togroup.calong with getting/dropping of thedev_setlock--- drivers/vfio/group.c | 32 ++++++++++++++---- drivers/vfio/vfio.h | 14 ++++++++ drivers/vfio/vfio_main.c | 70 ++++++++++++++++++++++++++++++++++++---- include/linux/vfio.h | 2 +- 4 files changed, 103 insertions(+), 15 deletions(-)drivers/vfio/group.c +2
-1@@ -164,13 +164,23 @@ static int vfio_device_group_open(struct vfio_device *device) goto out_unlock; } +mutex_lock (&device->dev_set->lock); +/* -* Here we pass the KVM pointer with the group under the lock. If the
drivers/vfio/group.c +8
-4-* device driver will use it, it must obtain a reference and release it -* during close_device. +* Before the first device open, get the KVM pointer currently +* associated with the group (if there is one) and obtain a reference +* now that will be held until the open_count reaches 0 again. Save +* the pointer in the device for use by drivers. */ - ret =vfio_device_open (device, device->group->iommufd, - device->group->kvm); +if (device->open_count ==0 ) +vfio_device_get_kvm_safe (device); + + ret = vfio_device_open(device, device->group->iommufd, device-Quoted 1 line
kvm);
drivers/vfio/group.c +13
-0+ +if (device->open_count ==0 ) +vfio_device_put_kvm (device); + +mutex_unlock (&device->dev_set->lock); out_unlock:mutex_unlock (&device->group->group_lock);@@ -180,7 +190,14 @@ static int vfio_device_group_open(struct vfio_device *device)void vfio_device_group_close (struct vfio_device *device) {mutex_lock (&device->group->group_lock); +mutex_lock (&device->dev_set->lock); +vfio_device_close (device, device->group->iommufd); + +if (device->open_count ==0 ) +vfio_device_put_kvm (device); + +mutex_unlock (&device->dev_set->lock);mutex_unlock (&device->group->group_lock); }@@ -450,6 +467,7 @@ static struct vfio_group *vfio_group_alloc(struct iommu_group *iommu_group,refcount_set (&group->drivers,1 );mutex_init (&group->group_lock); +spin_lock_init (&group->kvm_ref_lock);INIT_LIST_HEAD (&group->device_list);mutex_init (&group->device_lock);drivers/vfio/group.c +2
-2group->iommu_group = iommu_group;@@ -803,9 +821,9 @@ void vfio_file_set_kvm(struct file *file, struct kvm *kvm)if (!vfio_file_is_group (file))return ; -mutex_lock (&group->group_lock); +spin_lock (&group->kvm_ref_lock); group->kvm = kvm; -mutex_unlock (&group->group_lock); +spin_unlock (&group->kvm_ref_lock); }EXPORT_SYMBOL_GPL (vfio_file_set_kvm);drivers/vfio/vfio.h +14
-0@@ -74,6 +74,7 @@ struct vfio_group {struct file *opened_file;struct blocking_notifier_head notifier;struct iommufd_ctx *iommufd; +spinlock_t kvm_ref_lock; }; int vfio_device_set_group(struct vfio_device *device,@@ -251,4 +252,17 @@ extern bool vfio_noiommu __read_mostly;enum { vfio_noiommu = false };#endif +#ifdef CONFIG_HAVE_KVM +void vfio_device_get_kvm_safe (struct vfio_device *device); +void vfio_device_put_kvm (struct vfio_device *device); +#else +static inline void vfio_device_get_kvm_safe (struct vfio_device *device) +{ +} + +static inline void vfio_device_put_kvm (struct vfio_device *device) +{ +} +#endif +#endif drivers/vfio/vfio_main.c +59
-0@@ -16,6 +16,9 @@#include <linux/fs.h> #include <linux/idr.h> #include <linux/iommu.h> +#ifdef CONFIG_HAVE_KVM +#include <linux/kvm_host.h> +#endif #include <linux/list.h> #include <linux/miscdevice.h> #include <linux/module.h> @@ -338,6 +341,62 @@ void vfio_unregister_group_dev(struct vfio_device *device) }EXPORT_SYMBOL_GPL (vfio_unregister_group_dev); +#ifdef CONFIG_HAVE_KVM +void vfio_device_get_kvm_safe (struct vfio_device *device) +{ +void (*pfn)(struct kvm *kvm); +bool (*fn)(struct kvm *kvm); +bool ret; + +lockdep_assert_held (&device->dev_set->lock); + +spin_lock (&device->group->kvm_ref_lock); +if (!device->group->kvm) + goto unlock; + + pfn =symbol_get (kvm_put_kvm); +if (WARN_ON (!pfn)) + goto unlock; + + fn =symbol_get (kvm_get_kvm_safe); +if (WARN_ON (!fn)) { +symbol_put (kvm_put_kvm); + goto unlock; + } + + ret =fn (device->group->kvm); +symbol_put (kvm_get_kvm_safe); +if (!ret) { +symbol_put (kvm_put_kvm); + goto unlock; + } + + device->put_kvm = pfn; + device->kvm = device->group->kvm; + +unlock: +spin_unlock (&device->group->kvm_ref_lock); +} + +void vfio_device_put_kvm (struct vfio_device *device) +{ +lockdep_assert_held (&device->dev_set->lock); + +if (!device->kvm) +return ; + +if (WARN_ON (!device->put_kvm)) + goto clear; + + device->put_kvm (device->kvm); + device->put_kvm =NULL ; +symbol_put (kvm_put_kvm); + +clear: + device->kvm =NULL ; +} +#endif +/* true if the vfio_device has open_device() called but not close_device() */ static bool
vfio_assert_device_open(struct vfio_device *device){
drivers/vfio/vfio_main.c static int vfio_device_first_open(struct vfio_device · line 365
Quoted 5 lines
drivers/vfio/vfio_main.c +0
-1@@ -361,7 +420,6 @@ static int vfio_device_first_open(struct vfio_device *device, if (ret) goto err_module_put; - device->kvm = kvm;
Since you've deleted the only usage of kvm pointer in this function, I guess you can remove the kvm parameter from vfio_device_open() and vfio_device_first_open(). :-) if it makes this patch too big, may just have another patch to do it.
Regards,
Yi Liu
Quoted 76 lines
drivers/vfio/vfio_main.c +0
-0if (device->ops->open_device) { ret = device->ops->open_device (device);drivers/vfio/vfio_main.c +0
-1if (ret)@@ -370,7 +428,6 @@ static int vfio_device_first_open(struct vfio_device *device, return 0; err_unuse_iommu: - device->kvm =NULL ;if (iommufd)vfio_iommufd_unbind (device);drivers/vfio/vfio_main.c +0
-1else @@ -387,7 +444,6 @@ static void vfio_device_last_close(struct vfio_device *device, if (device->ops->close_device) device->ops->close_device(device); - device->kvm =NULL ;if (iommufd)vfio_iommufd_unbind (device);drivers/vfio/vfio_main.c +4
-4else @@ -400,14 +456,14 @@ int vfio_device_open(struct vfio_device *device, {int ret =0 ; -mutex_lock (&device->dev_set->lock); +lockdep_assert_held (&device->dev_set->lock); + device->open_count++;if (device->open_count ==1 ) { ret =vfio_device_first_open (device, iommufd, kvm);if (ret) device->open_count--; } -mutex_unlock (&device->dev_set->lock);return ret; }@@ -415,12 +471,12 @@ int vfio_device_open(struct vfio_device *device,void vfio_device_close (struct vfio_device *device,struct iommufd_ctx *iommufd) { -mutex_lock (&device->dev_set->lock); +lockdep_assert_held (&device->dev_set->lock); +vfio_assert_device_open (device);if (device->open_count ==1 )vfio_device_last_close (device, iommufd); device->open_count--; -mutex_unlock (&device->dev_set->lock); }/* include/linux/vfio.h +0
-1@@ -46,7 +46,6 @@ struct vfio_device {struct vfio_device_set *dev_set;struct list_head dev_set_list;unsigned int migration_flags; -/* Driver must reference the kvm during open_device or never touch it */
include/linux/vfio.h +1
-1struct kvm *kvm;/* Members below here are private, not for driver use */ @@ -58,6 +57,7 @@ struct vfio_device {struct list_head group_next;struct list_head iommu_entry;struct iommufd_access *iommufd_access; +void (*put_kvm)(struct kvm *kvm);#if IS_ENABLED (CONFIG_IOMMUFD )struct iommufd_device *iommufd_device;struct iommufd_ctx *iommufd_ictx; --2.39.1
Quoted 2 lines · Yi L
Hi Matthew,
...
Quoted 3 lines
Quoted 3 lines
* Can't pass
group->kvmtovfio_device_open, as it references the value outside of new lock. Passdevice->kvmto minimize changes in this fix (Alex, Yi)
...
Quoted 5 lines
Quoted 5 lines
drivers/vfio/vfio_main.c +0
-0@@ -361,7 +420,6 @@ static int vfio_device_first_open(struct vfio_device *device, if (ret) goto err_module_put;
Quoted 7 lines
drivers/vfio/vfio_main.c static int vfio_device_first_open(struct vfio_device *device, · line 364
Quoted 1 line
drivers/vfio/vfio_main.c +0
-1- device->kvm = kvm;Since you've deleted the only usage of kvm pointer in this function, I guess you can remove the kvm parameter from
vfio_device_open()andvfio_device_first_open(). :-) if it makes this patch too big, may just have another patch to do it.
Hi Yi,
Yeah, I mentioned it briefly (and vaguely I guess) in the cover, that was intentionally left out to reduce the patch size since this is a fix. I thought that was the consensus from the v2 comments anyway.
If I end up doing a v4 for this I can just include the removal as a 2nd patch (without a fixes tag) and Alex can squash or keep separate as preferred -- if not you can feel free to do that removal with your cdev follow-up that exploits this work.
Thanks,
Matt
Quoted 27 lines
From: Matthew Rosato <[email protected]>
Sent: Friday, February 3, 2023 10:26 PMQuoted 2 lines · Yi L
Hi Matthew,
...
Quoted 3 lines
Quoted 3 lines
* Can't pass
group->kvmtovfio_device_open, as it references the value outside of new lock. Passdevice->kvmto minimize changes in this fix (Alex, Yi)...
Quoted 1 line
Review static int vfio_device_first_open(struct
Quoted 1 line
(quoted diff) +0
-0@@ -361,7 +420,6 @@ static int vfio_device_first_open(struct
vfio_deviceQuoted 11 lines
drivers/vfio/vfio_main.c static int vfio_device_first_open(struct vfio_device *device, · line 364
Quoted 5 lines
drivers/vfio/vfio_main.c +0
-1*device, if (ret) goto err_module_put; - device->kvm = kvm;Since you've deleted the only usage of kvm pointer in this function, I guess you can remove the kvm parameter from
vfio_device_open()andvfio_device_first_open(). :-) if it makes this patch too big, may just have another patch to do it.Hi Yi,
Quoted 8 lines
Yeah, I mentioned it briefly (and vaguely I guess) in the cover, that was intentionally left out to reduce the patch size since this is a fix. I thought that was the consensus from the v2 comments anyway.
If I end up doing a v4 for this I can just include the removal as a 2nd patch (without a fixes tag) and Alex can squash or keep separate as preferred -- if not you can feel free to do that removal with your cdev follow-up that exploits this work.
Sure. 😊
Regards,
Yi Liu
Patchwork
CI[Intel-gfx] ✗ Fi.CI.BUILD: failure for vfio: fix deadlock between group lock and kvm lock (rev6)
In reply to Matthew Rosato== Series Details ==
== Series Details ==
Series: vfio: fix deadlock between group lock and kvm lock (rev6)
URL : https://patchwork.freedesktop.org/series/113535/
State : failure
== Summary ==
Error: patch https://patchwork.freedesktop.org/api/1.0/series/113535/revisions/6/mbox/ not applied
Applying: vfio: fix deadlock between group lock and kvm lock
Using index info to reconstruct a base tree...
M drivers/vfio/group.c
M drivers/vfio/vfio.h
M drivers/vfio/vfio_main.c
Falling back to patching base and 3-way merge...
Auto-merging drivers/vfio/vfio_main.c
CONFLICT (content): Merge conflict in drivers/vfio/vfio_main.c
Auto-merging drivers/vfio/vfio.h
CONFLICT (content): Merge conflict in drivers/vfio/vfio.h
Auto-merging drivers/vfio/group.c
CONFLICT (content): Merge conflict in drivers/vfio/group.c
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 vfio: fix deadlock between group lock and kvm lock
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Patchwork
CI[Intel-gfx] ✗ Fi.CI.BUILD: failure for vfio: fix deadlock between group lock and kvm lock (rev7)
In reply to Matthew Rosato== Series Details ==
== Series Details ==
Series: vfio: fix deadlock between group lock and kvm lock (rev7)
URL : https://patchwork.freedesktop.org/series/113535/
State : failure
== Summary ==
Error: patch https://patchwork.freedesktop.org/api/1.0/series/113535/revisions/7/mbox/ not applied
Applying: vfio: fix deadlock between group lock and kvm lock
Using index info to reconstruct a base tree...
M drivers/vfio/group.c
M drivers/vfio/vfio.h
M drivers/vfio/vfio_main.c
Falling back to patching base and 3-way merge...
Auto-merging drivers/vfio/vfio_main.c
CONFLICT (content): Merge conflict in drivers/vfio/vfio_main.c
Auto-merging drivers/vfio/vfio.h
CONFLICT (content): Merge conflict in drivers/vfio/vfio.h
Auto-merging drivers/vfio/group.c
CONFLICT (content): Merge conflict in drivers/vfio/group.c
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 vfio: fix deadlock between group lock and kvm lock
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".