vfio: circular locking dependency in pci_dev_reset_iommu_prepare()
Subsystems (5)
cover vfio: circular locking dependency in pci_dev_reset_iommu_prepare()
Hello,
I have encountered a reproducible circular locking dependency (lockdep) warning when performing a device reset via VFIO (e.g. VFIO_DEVICE_RESET ioctl or kexec live update) on a device whose BARs have been mmapped and faulted.
This issue was introduced in commit f5b16b802174 ("PCI: Suspend iommu function prior to resetting a device"), which added calls to pci_dev_reset_iommu_prepare() / pci_dev_reset_iommu_done() inside core PCI reset routines (pcie_flr(), pci_pm_reset(), etc.).
Patch 1 adds a new selftest in tools/testing/selftests/vfio/ that triggers this warning reliably on systems with ATS and reset support.
I used Intel DSA device 8086:0b25 with the test in Patch 1.
This issue was identified in the VFIO Live Update series
https://lore.kernel.org/kvm/[email protected]/
Since issue is independent of Live Update, I am sending this out separately to not mix two things.
I have uploaded the patch to github also:
https://github.com/shvipin/linux vfio/lockdep-mmap-fault-device-reset
================================================================================
Lock Hierarchy Analysis (Helped by AI)
================================================================================
1. In VFIO, handling VFIO_DEVICE_RESET (vfio_pci_ioctl_reset()) or bus/hot reset acquires down_write(&vdev->memory_lock) to zap active MMIO BAR mappings and prevent concurrent MMIO faults during reset.
Inside memory_lock, it calls pci_try_reset_function() / pcie_flr().
2. Commit f5b16b802174 added pci_dev_reset_iommu_prepare() into pcie_flr(). pci_dev_reset_iommu_prepare() acquires guard(mutex)(&group->mutex). This establishes:
&vdev->memory_lock --> &group->mutex
3. Meanwhile, during kernel boot or device discovery (iommu_probe_device() / bus_iommu_probe()), iommu.c holds &group->mutex while calling iommu_setup_dma_ops(), which calls iova_domain_init_rcaches(). iova_domain_init_rcaches() registers a multi-instance cpuhp handler via cpuhp_state_add_instance_nocalls(), acquiring cpus_read_lock(). This establishes:
&group->mutex --> cpu_hotplug_lock
4. cpu_hotplug_lock is an outer lock for various kernel subsystems that execute callbacks or sysfs/firmware/perf interactions, which eventually perform copy_to_user(), copy_from_user(), or filldir() (which can trigger a page fault in user memory under mmap_lock).
This establishes:
cpu_hotplug_lock --> (kernfs_rwsem / inode->i_rwsem / cpuctx_mutex)
--> &mm->mmap_lock5. When userspace touches an mmapped VFIO BAR, the page fault handler (vfio_pci_mmap_huge_fault() / vfio_pci_mmap_page_fault()) executes under &mm->mmap_lock and acquires down_read(&vdev->memory_lock). This establishes:
&mm->mmap_lock --> &vdev->memory_lock
Combining (1) through (5) yields the circular dependency cycle:
&group->mutex
--> cpu_hotplug_lock
--> [kernfs_rwsem | i_mutex_dir_key | cpuctx_mutex]
--> &mm->mmap_lock
--> &vdev->memory_lock
--> &group->mutex <== DEADLOCK DETECTED================================================================================
Reproductions / Lockdep Traces
================================================================================
I observed this lockdep warning closing via multiple distinct paths:
--- Trace 1: via sysfs / kernfs readdir ---
[ 438.457974] WARNING: possible circular locking dependency detected
[ 438.477952] vfio_cdev_lockd/17678 is trying to acquire lock:
[ 438.484320] (&group->mutex){+.+.}-{4:4}, at: pci_dev_reset_iommu_prepare+0x6e/0x200
[ 438.494169] but task is already holding lock:
[ 438.501401] (&vdev->memory_lock){++++}-{4:4}, at: vfio_pci_core_ioctl+0x460/0xb20
[ 438.511064] which lock already depends on the new lock.
...
[ 438.919866] Chain exists of:
[ 438.919866] &group->mutex --> &mm->mmap_lock --> &vdev->memory_lock
...
[ 438.999115] __mutex_lock+0x8c/0xd80
[ 438.999122] pci_dev_reset_iommu_prepare+0x6e/0x200
[ 438.999123] pcie_flr+0x32/0xc0
[ 438.999125] __pci_reset_function_locked+0x84/0x120
[ 438.999126] pci_try_reset_function+0x51/0xe0
[ 438.999128] vfio_pci_core_ioctl+0x72b/0xb20--- Trace 2: via VFS lookup / microcode loading during kexec ---
[ 584.552264] WARNING: possible circular locking dependency detected
[ 584.572247] kexec/14239 is trying to acquire lock:
[ 584.577754] (&group->mutex){+.+.}-{4:4}, at: pci_dev_reset_iommu_prepare+0x6e/0x1a0
[ 584.587602] but task is already holding lock:
[ 584.594838] (&vdev->memory_lock){++++}-{4:4}, at: vfio_pci_liveupdate_freeze+0x51/0x100
...
[ 584.751130] -> #2 (&type->i_mutex_dir_key#4):
[ 584.760319] lookup_slow+0x26/0x50
[ 584.786970] _request_firmware+0x4ab/0x8c0
[ 584.797979] request_microcode_fw+0xf2/0x510
[ 584.834601] -> #1 (cpu_hotplug_lock):
[ 584.842183] cpus_read_lock+0x3b/0xd0
[ 584.858957] iommu_setup_dma_ops+0x175/0x540--- Trace 3: via perf read / watchdog cpuhp callback ---
[ 233.859740] WARNING: possible circular locking dependency detected
[ 233.879725] vfio_cdev_lockd/14999 is trying to acquire lock:
[ 233.886094] (&group->mutex){+.+.}-{4:4}, at: pci_dev_reset_iommu_prepare+0x6e/0x200
[ 233.895948] but task is already holding lock:
[ 233.903183] (&vdev->memory_lock){++++}-{4:4}, at: vfio_pci_core_ioctl+0x460/0xb20
...
[ 234.030037] -> #3 (&cpuctx_mutex):
[ 234.040960] perf_event_enable+0x18/0xa0
[ 234.046126] lockup_detector_online_cpu+0x22/0x30
[ 234.057658] cpuhp_thread_fun+0x164/0x1e0
[ 234.115402] -> #1 (cpu_hotplug_lock):
[ 234.122985] cpus_read_lock+0x3b/0xd0
[ 234.139757] iommu_setup_dma_ops+0x18a/0x560================================================================================
Potential Solutions Suggested by AI
================================================================================
1. Decouple iommu_setup_dma_ops() from group->mutex in drivers/iommu/iommu.c:
iommu_setup_dma_ops() only requires struct device * and the domain pointer (group->default_domain); it does not mutate any fields in struct iommu_group. Moving the iommu_setup_dma_ops() calls after mutex_unlock(&group->mutex) in iommu_probe_device(), bus_iommu_probe(), and iommu_group_store_type() breaks the initial &group->mutex -> cpu_hotplug_lock dependency.
2. Avoid holding down_write(&vdev->memory_lock) across pci_try_reset_function() in VFIO:
vfio-pci could zap active BAR mappings under memory_lock and set a state flag / disable memory decoding, drop memory_lock before calling pci_try_reset_function(), and then re-acquire memory_lock to re-enable memory. While resetting, any concurrent user fault will see the memory disabled condition and return VM_FAULT_SIGBUS safely.
3. Refine synchronization in pci_dev_reset_iommu_prepare():
Evaluate if attaching to the blocking domain and pausing ATS during device reset can be protected using more fine-grained locking or atomic state flags without holding the coarse &group->mutex.
Feedback and suggestions on the preferred direction to resolve this would be greatly appreciated.
Thanks,
Vipin Sharma
Vipin Sharma (1):
selftests: vfio: Add mmap fault and device reset test
tools/testing/selftests/vfio/Makefile | 1 + .../selftests/vfio/vfio_pci_mmap_reset_test.c | 60 +++++++++++++++++++ 2 files changed, 61 insertions(+) create mode 100644 tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c
Signature · 1 line
2.55.0.766.g2966f0265a-goog
RFC PATCH 1/1 selftests: vfio: Add mmap fault and device reset test
Add a selftest to verify VFIO PCI device reset on an mmapped and faulted device. The test maps all available BARs on the device, faults them, and triggers a device reset via the VFIO_DEVICE_RESET ioctl.
This exercise uncovers a circular locking dependency introduced in commit f5b16b802174 ("PCI: Suspend iommu function prior to resetting a device"), where pci_dev_reset_iommu_prepare() acquires group->mutex under vfio's vdev->memory_lock.
--- tools/testing/selftests/vfio/Makefile | 1 + .../selftests/vfio/vfio_pci_mmap_reset_test.c | 60 +++++++++++++++++++ 2 files changed, 61 insertions(+) create mode 100644 tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c
tools/testing/selftests/vfio/Makefile +1 -0
@@ -13,6 +13,7 @@ TEST_GEN_PROGS += vfio_pci_device_test TEST_GEN_PROGS += vfio_pci_device_init_perf_test TEST_GEN_PROGS += vfio_pci_driver_test TEST_GEN_PROGS += vfio_pci_sriov_uapi_test +TEST_GEN_PROGS += vfio_pci_mmap_reset_test TEST_FILES += scripts/cleanup.sh TEST_FILES += scripts/lib.sh
tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c new file +60 -0
60 lines not on this page · raw message
Signature · 1 line
2.55.0.766.g2966f0265a-goog
[email protected]
AI reviewRe: [RFC PATCH 1/1] selftests: vfio: Add mmap fault and device reset test
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] A circular locking dependency exists between mmap_lock, vdev->memory_lock, and group->mutex, leading to a potential deadlock. --
--- Patch [1]: [RFC PATCH 1/1] selftests: vfio: Add mmap fault and device reset test --- commit 3fb77a6912d6cc2309856fef1f82986ea816df96
Author: Vipin Sharma <[email protected]>
Subject: selftests: vfio: Add mmap fault and device reset test
This commit adds a selftest to verify VFIO PCI device reset on an mmapped and faulted device. The test maps all available BARs, triggers a page fault on them, and then initiates a device reset via the VFIO_DEVICE_RESET ioctl to expose a circular locking dependency.
Quoted 4 lines
tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c new file +0
-0
[ ... ]
Quoted 2 lines
tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c new file +2
-0+TEST_F(vfio_pci_mmap_reset_test, mmap_fault_and_reset) +{
[ ... ]
tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c line 53
Quoted 5 lines
tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c new file +5
-0+if (!has_mmap) +SKIP (return,"No mmapable BAR found on device\n" ); + +/* Trigger device reset under memory_lock */ +vfio_pci_device_reset (self->device);
[Severity: High]
This is a pre-existing issue, but does this test sequence reliably trigger a host kernel deadlock by exposing a circular locking dependency between mmap_lock, vdev->memory_lock, and group->mutex?
The lockdep cycle appears to start when a page fault holds mmap_lock and acquires vdev->memory_lock:
drivers/vfio/pci/vfio_pci_core.c:vfio_pci_mmap_huge_fault()
scoped_guard(rwsem_read, &vdev->memory_lock)
Separately, a device reset acquires vdev->memory_lock before initiating the reset:
drivers/vfio/pci/vfio_pci_core.c:vfio_pci_ioctl_reset()
vfio_pci_zap_and_down_write_memory_lock(vdev);
While held, it calls into the PCI reset functions which eventually prepare the IOMMU and acquire group->mutex:
drivers/iommu/iommu.c:pci_dev_reset_iommu_prepare()
guard(mutex)(&group->mutex);
The cycle completes when another kernel path holding group->mutex waits on mmap_lock or memory_lock. Can a malicious guest VM or local user with VFIO access trigger concurrent page faults and device resets to exploit this cycle?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
David Matlack
maintainerQuoted 10 lines · Vipin Sharma
================================================================================
Potential Solutions Suggested by AI
================================================================================1. Decouple
iommu_setup_dma_ops()fromgroup->mutexindrivers/iommu/iommu.c:
iommu_setup_dma_ops()only requires struct device * and the domain pointer (group->default_domain); it does not mutate any fields in structiommu_group. Moving theiommu_setup_dma_ops()calls aftermutex_unlock(&group->mutex)iniommu_probe_device(),bus_iommu_probe(), andiommu_group_store_type()breaks the initial&group->mutex->cpu_hotplug_lockdependency.
Are there any other code paths that rely on group->mutex --> mm->mmap_lock ordering? If so fixing this one case wouldn't help.
Quoted 7 lines
2. Avoid holding
down_write(&vdev->memory_lock)acrosspci_try_reset_function()in VFIO:
vfio-pci could zap active BAR mappings undermemory_lockand set a state flag / disable memory decoding, dropmemory_lockbefore callingpci_try_reset_function(), and then re-acquirememory_lockto re-enable memory. While resetting, any concurrent user fault will see the memory disabled condition and returnVM_FAULT_SIGBUSsafely.
This would change the userspace-visible behavior of faulting on a VFIO device BAR from "block until reset is done and the succeed" to "fail with SIGBUS". And it would allow VFIO to access VFIO device BARs during the reset through vfio_pci_core_iowrite*().
But I think we can extend this idea to solve those problems by introducing a wait queue for tasks to sit on while a device is being reset.
e.g. Something like this (completely untested and partially written by AI):
From: David Matlack <[email protected]>
Date: Tue, 25 Aug 2026 18:37:52 +0000
Subject: [PATCH] vfio/pci: Avoid circular locking dependency during device reset
Avoid a circular locking dependency during VFIO device reset by dropping vdev->memory_lock prior to calling PCI reset functions (pci_try_reset_function() and pci_reset_bus()). Introduce an explicit reset state flag (vdev->resetting) and wait queue (vdev->reset_done_wq) to stall concurrent BAR page faults and MMIO accesses during reset without holding vdev->memory_lock across PCI reset operations.
Export a new helper function vfio_pci_core_try_reset() to consolidate the common reset sequence across vfio_pci_ioctl_reset() and PCI config space FLR handlers.
Export a new helper function vfio_pci_core_down_read_memory_lock() to encapsulate acquiring down_read(&vdev->memory_lock) while waiting on vdev->reset_done_wq if a reset is in progress.
Holding vdev->memory_lock across pci_try_reset_function() or pci_reset_bus() causes a lock inversion between vdev->memory_lock and the IOMMU group mutex (group->mutex). PCI reset functions invoke pci_dev_reset_iommu_prepare(), which acquires group->mutex. However, vdev->memory_lock is acquired inside page fault context (vfio_pci_mmap_huge_fault()), placing vdev->memory_lock below mm->mmap_lock in the locking hierarchy. Meanwhile, operations holding group->mutex (such as sysfs interactions or IOMMU domain operations) can fault on user memory, placing group->mutex above mm->mmap_lock. This establishes the circular locking chain:
group->mutex --> mm->mmap_lock --> vdev->memory_lock --> group->mutex
Simply dropping vdev->memory_lock before calling PCI reset routines would alter userspace-visible behavior by causing concurrent BAR page faults during reset to fail with VM_FAULT_SIGBUS rather than stalling until reset completes.
Preserve the stalling behavior while resolving the deadlock:
1. In vfio_pci_core_try_reset(), set vdev->resetting = true under vdev->memory_lock, zap BAR mmaps, and drop vdev->memory_lock before calling pci_try_reset_function().
2. In vfio_pci_core_down_read_memory_lock(), if vdev->resetting is true, drop memory_lock and sleep uninterruptibly on vdev->reset_done_wq via wait_event().
3. Upon reset completion, clear vdev->resetting = false and wake up the wait queue.
--- drivers/vfio/pci/vfio_pci_config.c | 20 +++--------- drivers/vfio/pci/vfio_pci_core.c | 51 ++++++++++++++++++++++++++---- drivers/vfio/pci/vfio_pci_rdwr.c | 4 +-- include/linux/vfio_pci_core.h | 4 +++ 4 files changed, 54 insertions(+), 25 deletions(-)
drivers/vfio/pci/vfio_pci_config.c +4 -16
@@ -907,14 +907,8 @@ static int vfio_exp_config_write(struct vfio_pci_core_device *vdev, int pos, pos - offset +PCI_EXP_DEVCAP , &cap); -if (!ret && (cap &PCI_EXP_DEVCAP_FLR )) { -vfio_pci_zap_and_down_write_memory_lock (vdev); -vfio_pci_dma_buf_move (vdev, true); -pci_try_reset_function (vdev->pdev); -if (__vfio_pci_memory_enabled (vdev)) -vfio_pci_dma_buf_move (vdev, false); -up_write (&vdev->memory_lock); - } +if (!ret && (cap &PCI_EXP_DEVCAP_FLR )) +vfio_pci_core_try_reset (vdev); }/*
@@ -992,14 +986,8 @@ static int vfio_af_config_write(struct vfio_pci_core_device *vdev, int pos, pos - offset +PCI_AF_CAP , &cap); -if (!ret && (cap &PCI_AF_CAP_FLR ) && (cap &PCI_AF_CAP_TP )) { -vfio_pci_zap_and_down_write_memory_lock (vdev); -vfio_pci_dma_buf_move (vdev, true); -pci_try_reset_function (vdev->pdev); -if (__vfio_pci_memory_enabled (vdev)) -vfio_pci_dma_buf_move (vdev, false); -up_write (&vdev->memory_lock); - } +if (!ret && (cap &PCI_AF_CAP_FLR ) && (cap &PCI_AF_CAP_TP )) +vfio_pci_core_try_reset (vdev); }return count;
drivers/vfio/pci/vfio_pci_core.c +44 -7
112 lines not on this page · raw message
drivers/vfio/pci/vfio_pci_rdwr.c +2 -2
@@ -43,7 +43,7 @@ int vfio_pci_core_iowrite##size(struct vfio_pci_core_device *vdev, \ bool test_mem, u##size val, void __iomem *io) \ { \ if (test_mem) { \ - down_read(&vdev->memory_lock); \ + vfio_pci_core_down_read_memory_lock(vdev); \ if (!__vfio_pci_memory_enabled(vdev)) { \ up_read(&vdev->memory_lock); \ return -EIO; \
@@ -69,7 +69,7 @@ int vfio_pci_core_ioread##size(struct vfio_pci_core_device *vdev, \ bool test_mem, u##size *val, void __iomem *io) \ { \ if (test_mem) { \ - down_read(&vdev->memory_lock); \ + vfio_pci_core_down_read_memory_lock(vdev); \ if (!__vfio_pci_memory_enabled(vdev)) { \ up_read(&vdev->memory_lock); \ return -EIO; \
include/linux/vfio_pci_core.h +4 -0
@@ -148,6 +148,8 @@ struct vfio_pci_core_device {struct vfio_pci_core_device *sriov_pf_core_dev;struct notifier_block nb;struct rw_semaphore memory_lock; +bool resetting; +wait_queue_head_t reset_done_wq;struct list_head dmabufs; };
@@ -171,6 +173,8 @@ void vfio_pci_core_unregister_device(struct vfio_pci_core_device *vdev);extern const struct pci_error_handlers vfio_pci_core_err_handlers;int vfio_pci_core_sriov_configure (struct vfio_pci_core_device *vdev,int nr_virtfn); +int vfio_pci_core_try_reset (struct vfio_pci_core_device *vdev); +void vfio_pci_core_down_read_memory_lock (struct vfio_pci_core_device *vdev);long vfio_pci_core_ioctl (struct vfio_device *core_vdev,unsigned int cmd,unsigned long arg); int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags,
Quoted 5 lines
3. Refine synchronization in
pci_dev_reset_iommu_prepare():
Evaluate if attaching to the blocking domain and pausing ATS during device reset can be protected using more fine-grained locking or atomic state flags without holding the coarse&group->mutex.
I don't know enough about this part of the kernel to say, but this would directly address the new lock ordering dependency vdev->memory_lock --> group->mutex introduced by commit f5b16b802174 ("PCI: Suspend iommu function prior to resetting a device"), which is what led to this lockdep error.
David Matlack
maintainerQuoted 40 lines · David Matlack
Quoted 10 lines · Vipin Sharma
================================================================================
Potential Solutions Suggested by AI
================================================================================1. Decouple
iommu_setup_dma_ops()fromgroup->mutexindrivers/iommu/iommu.c:
iommu_setup_dma_ops()only requires struct device * and the domain pointer (group->default_domain); it does not mutate any fields in structiommu_group. Moving theiommu_setup_dma_ops()calls aftermutex_unlock(&group->mutex)iniommu_probe_device(),bus_iommu_probe(), andiommu_group_store_type()breaks the initial&group->mutex->cpu_hotplug_lockdependency.Are there any other code paths that rely on
group->mutex-->mm->mmap_lockordering? If so fixing this one case wouldn't help.Quoted 7 lines
2. Avoid holding
down_write(&vdev->memory_lock)acrosspci_try_reset_function()in VFIO:
vfio-pci could zap active BAR mappings undermemory_lockand set a state flag / disable memory decoding, dropmemory_lockbefore callingpci_try_reset_function(), and then re-acquirememory_lockto re-enable memory. While resetting, any concurrent user fault will see the memory disabled condition and returnVM_FAULT_SIGBUSsafely.This would change the userspace-visible behavior of faulting on a VFIO device BAR from "block until reset is done and the succeed" to "fail with SIGBUS". And it would allow VFIO to access VFIO device BARs during the reset through vfio_pci_core_iowrite*().
But I think we can extend this idea to solve those problems by introducing a wait queue for tasks to sit on while a device is being reset.
e.g. Something like this (completely untested and partially written by AI):
From: David Matlack <[email protected]>
Date: Tue, 25 Aug 2026 18:37:52 +0000
Subject: [PATCH] vfio/pci: Avoid circular locking dependency during device reset
Quoted 6 lines
Avoid a circular locking dependency during VFIO device reset by dropping
vdev->memory_lockprior to calling PCI reset functions (pci_try_reset_function()andpci_reset_bus()). Introduce an explicit reset state flag (vdev->resetting) and wait queue (vdev->reset_done_wq) to stall concurrent BAR page faults and MMIO accesses during reset without holdingvdev->memory_lockacross PCI reset operations.
...
This approach does not look ideal. The implementation has a bug where concurrent resets can lead to vdev->resetting being cleared too early. And from a maintainability perspective, there are more call sites that currently take memory_lock that would probably also have to be updated to wait for vdev->resetting to become false.
Quoted 10 lines
Quoted 5 lines
3. Refine synchronization in
pci_dev_reset_iommu_prepare():
Evaluate if attaching to the blocking domain and pausing ATS during device reset can be protected using more fine-grained locking or atomic state flags without holding the coarse&group->mutex.I don't know enough about this part of the kernel to say, but this would directly address the new lock ordering dependency
vdev->memory_lock-->group->mutexintroduced by commitf5b16b802174("PCI: Suspend iommu function prior to resetting a device"), which is what led to this lockdep error.
David Matlack
maintainerQuoted 48 lines · David Matlack
Quoted 40 lines · David Matlack
Quoted 10 lines · Vipin Sharma
================================================================================
Potential Solutions Suggested by AI
================================================================================1. Decouple
iommu_setup_dma_ops()fromgroup->mutexindrivers/iommu/iommu.c:
iommu_setup_dma_ops()only requires struct device * and the domain pointer (group->default_domain); it does not mutate any fields in structiommu_group. Moving theiommu_setup_dma_ops()calls aftermutex_unlock(&group->mutex)iniommu_probe_device(),bus_iommu_probe(), andiommu_group_store_type()breaks the initial&group->mutex->cpu_hotplug_lockdependency.Are there any other code paths that rely on
group->mutex-->mm->mmap_lockordering? If so fixing this one case wouldn't help.Quoted 7 lines
2. Avoid holding
down_write(&vdev->memory_lock)acrosspci_try_reset_function()in VFIO:
vfio-pci could zap active BAR mappings undermemory_lockand set a state flag / disable memory decoding, dropmemory_lockbefore callingpci_try_reset_function(), and then re-acquirememory_lockto re-enable memory. While resetting, any concurrent user fault will see the memory disabled condition and returnVM_FAULT_SIGBUSsafely.This would change the userspace-visible behavior of faulting on a VFIO device BAR from "block until reset is done and the succeed" to "fail with SIGBUS". And it would allow VFIO to access VFIO device BARs during the reset through vfio_pci_core_iowrite*().
But I think we can extend this idea to solve those problems by introducing a wait queue for tasks to sit on while a device is being reset.
e.g. Something like this (completely untested and partially written by AI):
From: David Matlack <[email protected]>
Date: Tue, 25 Aug 2026 18:37:52 +0000
Subject: [PATCH] vfio/pci: Avoid circular locking dependency during device resetQuoted 6 lines
Avoid a circular locking dependency during VFIO device reset by dropping
vdev->memory_lockprior to calling PCI reset functions (pci_try_reset_function()andpci_reset_bus()). Introduce an explicit reset state flag (vdev->resetting) and wait queue (vdev->reset_done_wq) to stall concurrent BAR page faults and MMIO accesses during reset without holdingvdev->memory_lockacross PCI reset operations.
Quoted 18 lines
...
This approach does not look ideal. The implementation has a bug where concurrent resets can lead to
vdev->resettingbeing cleared too early. And from a maintainability perspective, there are more call sites that currently takememory_lockthat would probably also have to be updated to wait forvdev->resettingto become false.Quoted 10 lines
Quoted 5 lines
3. Refine synchronization in
pci_dev_reset_iommu_prepare():
Evaluate if attaching to the blocking domain and pausing ATS during device reset can be protected using more fine-grained locking or atomic state flags without holding the coarse&group->mutex.I don't know enough about this part of the kernel to say, but this would directly address the new lock ordering dependency
vdev->memory_lock-->group->mutexintroduced by commitf5b16b802174("PCI: Suspend iommu function prior to resetting a device"), which is what led to this lockdep error.
I sent a proposed fix that takes this approach here:
Quoted 8 lines · David Matlack
Quoted 7 lines · David Matlack
Quoted 4 lines · Vipin Sharma
================================================================================
Potential Solutions Suggested by AI
================================================================================
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
1. Decouple
iommu_setup_dma_ops()fromgroup->mutexindrivers/iommu/iommu.c:
iommu_setup_dma_ops()only requires struct device * and the domain pointer (group->default_domain); it does not mutate any fields in structiommu_group. Moving theiommu_setup_dma_ops()calls aftermutex_unlock(&group->mutex)iniommu_probe_device(),bus_iommu_probe(), andiommu_group_store_type()breaks the initial&group->mutex->cpu_hotplug_lockdependency.
The group->mutex is needed here also since it sets up the dma_ops on the default_domain that is currently attached to the device. And those attachments are protected with group->mutex.
Quoted 33 lines
Quoted 27 lines
Are there any other code paths that rely on
group->mutex-->mm->mmap_lockordering? If so fixing this one case wouldn't help.Quoted 7 lines
2. Avoid holding
down_write(&vdev->memory_lock)acrosspci_try_reset_function()in VFIO:
vfio-pci could zap active BAR mappings undermemory_lockand set a state flag / disable memory decoding, dropmemory_lockbefore callingpci_try_reset_function(), and then re-acquirememory_lockto re-enable memory. While resetting, any concurrent user fault will see the memory disabled condition and returnVM_FAULT_SIGBUSsafely.This would change the userspace-visible behavior of faulting on a VFIO device BAR from "block until reset is done and the succeed" to "fail with SIGBUS". And it would allow VFIO to access VFIO device BARs during the reset through vfio_pci_core_iowrite*().
But I think we can extend this idea to solve those problems by introducing a wait queue for tasks to sit on while a device is being reset.
e.g. Something like this (completely untested and partially written by AI):
From: David Matlack <[email protected]>
Date: Tue, 25 Aug 2026 18:37:52 +0000
Subject: [PATCH] vfio/pci: Avoid circular locking dependency during device resetQuoted 6 lines
Avoid a circular locking dependency during VFIO device reset by dropping
vdev->memory_lockprior to calling PCI reset functions (pci_try_reset_function()andpci_reset_bus()). Introduce an explicit reset state flag (vdev->resetting) and wait queue (vdev->reset_done_wq) to stall concurrent BAR page faults and MMIO accesses during reset without holdingvdev->memory_lockacross PCI reset operations.
Quoted 13 lines
...
This approach does not look ideal. The implementation has a bug where concurrent resets can lead to
vdev->resettingbeing cleared too early. And from a maintainability perspective, there are more call sites that currently takememory_lockthat would probably also have to be updated to wait forvdev->resettingto become false.Quoted 5 lines
Quoted 5 lines
3. Refine synchronization in
pci_dev_reset_iommu_prepare():
Evaluate if attaching to the blocking domain and pausing ATS during device reset can be protected using more fine-grained locking or atomic state flags without holding the coarse&group->mutex.
This might not work as reset_iommu_prepare() changes the domain of the device being reset and those things protected by the group->mutex.
Quoted 5 lines
Quoted 5 lines
I don't know enough about this part of the kernel to say, but this would directly address the new lock ordering dependency
vdev->memory_lock-->group->mutexintroduced by commitf5b16b802174("PCI: Suspend iommu function prior to resetting a device"), which is what led to this lockdep error.
Thanks,
Sami