Skip to contentSkip to conversation

vfio: circular locking dependency in pci_dev_reset_iommu_prepare()

RFC PATCH In review Vipin Sharma 4 participants 7 messages Aug 21 – Aug 27

Subsystems (5)
KVM Kselftest PCI VFIO driver +1 more subsystems
lore
Share & tools
lore
#0 lore raw

cover vfio: circular locking dependency in pci_dev_reset_iommu_prepare()

Hello,

I have encountered a reproducible circular locking dependency (lockdep) warning when performing a device reset via VFIO (e.g. VFIO_DEVICE_RESET ioctl or kexec live update) on a device whose BARs have been mmapped and faulted.

This issue was introduced in commit f5b16b802174 ("PCI: Suspend iommu function prior to resetting a device"), which added calls to pci_dev_reset_iommu_prepare() / pci_dev_reset_iommu_done() inside core PCI reset routines (pcie_flr(), pci_pm_reset(), etc.).

Patch 1 adds a new selftest in tools/testing/selftests/vfio/ that triggers this warning reliably on systems with ATS and reset support.

I used Intel DSA device 8086:0b25 with the test in Patch 1.

This issue was identified in the VFIO Live Update series
https://lore.kernel.org/kvm/[email protected]/

Since issue is independent of Live Update, I am sending this out separately to not mix two things.

I have uploaded the patch to github also:
https://github.com/shvipin/linux vfio/lockdep-mmap-fault-device-reset

================================================================================
Lock Hierarchy Analysis (Helped by AI)
================================================================================

1. In VFIO, handling VFIO_DEVICE_RESET (vfio_pci_ioctl_reset()) or bus/hot reset acquires down_write(&vdev->memory_lock) to zap active MMIO BAR mappings and prevent concurrent MMIO faults during reset.
Inside memory_lock, it calls pci_try_reset_function() / pcie_flr().

2. Commit f5b16b802174 added pci_dev_reset_iommu_prepare() into pcie_flr(). pci_dev_reset_iommu_prepare() acquires guard(mutex)(&group->mutex). This establishes:

       &vdev->memory_lock --> &group->mutex

3. Meanwhile, during kernel boot or device discovery (iommu_probe_device() / bus_iommu_probe()), iommu.c holds &group->mutex while calling iommu_setup_dma_ops(), which calls iova_domain_init_rcaches(). iova_domain_init_rcaches() registers a multi-instance cpuhp handler via cpuhp_state_add_instance_nocalls(), acquiring cpus_read_lock(). This establishes:

       &group->mutex --> cpu_hotplug_lock

4. cpu_hotplug_lock is an outer lock for various kernel subsystems that execute callbacks or sysfs/firmware/perf interactions, which eventually perform copy_to_user(), copy_from_user(), or filldir() (which can trigger a page fault in user memory under mmap_lock).
This establishes:

       cpu_hotplug_lock --> (kernfs_rwsem / inode->i_rwsem / cpuctx_mutex)
                        --> &mm->mmap_lock

5. When userspace touches an mmapped VFIO BAR, the page fault handler (vfio_pci_mmap_huge_fault() / vfio_pci_mmap_page_fault()) executes under &mm->mmap_lock and acquires down_read(&vdev->memory_lock). This establishes:

       &mm->mmap_lock --> &vdev->memory_lock

Combining (1) through (5) yields the circular dependency cycle:

    &group->mutex
      --> cpu_hotplug_lock
        --> [kernfs_rwsem | i_mutex_dir_key | cpuctx_mutex]
          --> &mm->mmap_lock
            --> &vdev->memory_lock
              --> &group->mutex  <== DEADLOCK DETECTED

================================================================================
Reproductions / Lockdep Traces
================================================================================

I observed this lockdep warning closing via multiple distinct paths:

--- Trace 1: via sysfs / kernfs readdir ---

[  438.457974] WARNING: possible circular locking dependency detected
[  438.477952] vfio_cdev_lockd/17678 is trying to acquire lock:
[  438.484320] (&group->mutex){+.+.}-{4:4}, at: pci_dev_reset_iommu_prepare+0x6e/0x200
[  438.494169] but task is already holding lock:
[  438.501401] (&vdev->memory_lock){++++}-{4:4}, at: vfio_pci_core_ioctl+0x460/0xb20
[  438.511064] which lock already depends on the new lock.
...
[  438.919866] Chain exists of:
[  438.919866]   &group->mutex --> &mm->mmap_lock --> &vdev->memory_lock
...
[  438.999115]  __mutex_lock+0x8c/0xd80
[  438.999122]  pci_dev_reset_iommu_prepare+0x6e/0x200
[  438.999123]  pcie_flr+0x32/0xc0
[  438.999125]  __pci_reset_function_locked+0x84/0x120
[  438.999126]  pci_try_reset_function+0x51/0xe0
[  438.999128]  vfio_pci_core_ioctl+0x72b/0xb20

--- Trace 2: via VFS lookup / microcode loading during kexec ---

[  584.552264] WARNING: possible circular locking dependency detected
[  584.572247] kexec/14239 is trying to acquire lock:
[  584.577754] (&group->mutex){+.+.}-{4:4}, at: pci_dev_reset_iommu_prepare+0x6e/0x1a0
[  584.587602] but task is already holding lock:
[  584.594838] (&vdev->memory_lock){++++}-{4:4}, at: vfio_pci_liveupdate_freeze+0x51/0x100
...
[  584.751130] -> #2 (&type->i_mutex_dir_key#4):
[  584.760319]        lookup_slow+0x26/0x50
[  584.786970]        _request_firmware+0x4ab/0x8c0
[  584.797979]        request_microcode_fw+0xf2/0x510
[  584.834601] -> #1 (cpu_hotplug_lock):
[  584.842183]        cpus_read_lock+0x3b/0xd0
[  584.858957]        iommu_setup_dma_ops+0x175/0x540

--- Trace 3: via perf read / watchdog cpuhp callback ---

[  233.859740] WARNING: possible circular locking dependency detected
[  233.879725] vfio_cdev_lockd/14999 is trying to acquire lock:
[  233.886094] (&group->mutex){+.+.}-{4:4}, at: pci_dev_reset_iommu_prepare+0x6e/0x200
[  233.895948] but task is already holding lock:
[  233.903183] (&vdev->memory_lock){++++}-{4:4}, at: vfio_pci_core_ioctl+0x460/0xb20
...
[  234.030037] -> #3 (&cpuctx_mutex):
[  234.040960]        perf_event_enable+0x18/0xa0
[  234.046126]        lockup_detector_online_cpu+0x22/0x30
[  234.057658]        cpuhp_thread_fun+0x164/0x1e0
[  234.115402] -> #1 (cpu_hotplug_lock):
[  234.122985]        cpus_read_lock+0x3b/0xd0
[  234.139757]        iommu_setup_dma_ops+0x18a/0x560

================================================================================
Potential Solutions Suggested by AI
================================================================================

1. Decouple iommu_setup_dma_ops() from group->mutex in drivers/iommu/iommu.c:
iommu_setup_dma_ops() only requires struct device * and the domain pointer (group->default_domain); it does not mutate any fields in struct iommu_group. Moving the iommu_setup_dma_ops() calls after mutex_unlock(&group->mutex) in iommu_probe_device(), bus_iommu_probe(), and iommu_group_store_type() breaks the initial &group->mutex -> cpu_hotplug_lock dependency.

2. Avoid holding down_write(&vdev->memory_lock) across pci_try_reset_function() in VFIO:
vfio-pci could zap active BAR mappings under memory_lock and set a state flag / disable memory decoding, drop memory_lock before calling pci_try_reset_function(), and then re-acquire memory_lock to re-enable memory. While resetting, any concurrent user fault will see the memory disabled condition and return VM_FAULT_SIGBUS safely.

3. Refine synchronization in pci_dev_reset_iommu_prepare():
Evaluate if attaching to the blocking domain and pausing ATS during device reset can be protected using more fine-grained locking or atomic state flags without holding the coarse &group->mutex.

Feedback and suggestions on the preferred direction to resolve this would be greatly appreciated.

Thanks,
Vipin Sharma

Vipin Sharma (1):
selftests: vfio: Add mmap fault and device reset test

 tools/testing/selftests/vfio/Makefile         |  1 +
 .../selftests/vfio/vfio_pci_mmap_reset_test.c | 60 +++++++++++++++++++
 2 files changed, 61 insertions(+)
 create mode 100644 tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c
Signature · 1 line
2.55.0.766.g2966f0265a-goog
#1 lore raw

RFC PATCH 1/1 selftests: vfio: Add mmap fault and device reset test

Add a selftest to verify VFIO PCI device reset on an mmapped and faulted device. The test maps all available BARs on the device, faults them, and triggers a device reset via the VFIO_DEVICE_RESET ioctl.

This exercise uncovers a circular locking dependency introduced in commit f5b16b802174 ("PCI: Suspend iommu function prior to resetting a device"), where pci_dev_reset_iommu_prepare() acquires group->mutex under vfio's vdev->memory_lock.

Assisted-by Jetski:gemini-3.1-pro
---
 tools/testing/selftests/vfio/Makefile         |  1 +
 .../selftests/vfio/vfio_pci_mmap_reset_test.c | 60 +++++++++++++++++++
 2 files changed, 61 insertions(+)
 create mode 100644 tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c
tools/testing/selftests/vfio/Makefile +1 -0
index 2c32c48db509..17b3a2fe215c 100644
@@ -13,6 +13,7 @@ TEST_GEN_PROGS += vfio_pci_device_test
 TEST_GEN_PROGS += vfio_pci_device_init_perf_test
 TEST_GEN_PROGS += vfio_pci_driver_test
 TEST_GEN_PROGS += vfio_pci_sriov_uapi_test
+TEST_GEN_PROGS += vfio_pci_mmap_reset_test
 
 TEST_FILES += scripts/cleanup.sh
 TEST_FILES += scripts/lib.sh
tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c new file +60 -0

60 lines not on this page · raw message

Signature · 1 line
2.55.0.766.g2966f0265a-goog
#2 lore raw

Re: [RFC PATCH 1/1] selftests: vfio: Add mmap fault and device reset test

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] A circular locking dependency exists between mmap_lock, vdev->memory_lock, and group->mutex, leading to a potential deadlock. --

--- Patch [1]: [RFC PATCH 1/1] selftests: vfio: Add mmap fault and device reset test --- commit 3fb77a6912d6cc2309856fef1f82986ea816df96
Author: Vipin Sharma <[email protected]>
Subject: selftests: vfio: Add mmap fault and device reset test

This commit adds a selftest to verify VFIO PCI device reset on an mmapped and faulted device. The test maps all available BARs, triggers a page fault on them, and then initiates a device reset via the VFIO_DEVICE_RESET ioctl to expose a circular locking dependency.

Quoted 4 lines
tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c new file +0 -0
new file mode 100644

[ ... ]

Quoted 2 lines
tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c new file +2 -0
+TEST_F(vfio_pci_mmap_reset_test, mmap_fault_and_reset)
+{

[ ... ]

tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c line 53

Quoted 5 lines
tools/testing/selftests/vfio/vfio_pci_mmap_reset_test.c new file +5 -0
+	if (!has_mmap)
+		SKIP(return, "No mmapable BAR found on device\n");
+
+	/* Trigger device reset under memory_lock */
+	vfio_pci_device_reset(self->device);

[Severity: High]
This is a pre-existing issue, but does this test sequence reliably trigger a host kernel deadlock by exposing a circular locking dependency between mmap_lock, vdev->memory_lock, and group->mutex?

The lockdep cycle appears to start when a page fault holds mmap_lock and acquires vdev->memory_lock:

drivers/vfio/pci/vfio_pci_core.c:vfio_pci_mmap_huge_fault()

    scoped_guard(rwsem_read, &vdev->memory_lock)

Separately, a device reset acquires vdev->memory_lock before initiating the reset:

drivers/vfio/pci/vfio_pci_core.c:vfio_pci_ioctl_reset()

    vfio_pci_zap_and_down_write_memory_lock(vdev);

While held, it calls into the PCI reset functions which eventually prepare the IOMMU and acquire group->mutex:

drivers/iommu/iommu.c:pci_dev_reset_iommu_prepare()

    guard(mutex)(&group->mutex);

The cycle completes when another kernel path holding group->mutex waits on mmap_lock or memory_lock. Can a malicious guest VM or local user with VFIO access trigger concurrent page faults and device resets to exploit this cycle?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
maintainer
#3 lore raw
In reply to Vipin Sharma
Quoted 10 lines · Vipin Sharma

================================================================================
Potential Solutions Suggested by AI
================================================================================

1. Decouple iommu_setup_dma_ops() from group->mutex in drivers/iommu/iommu.c:
iommu_setup_dma_ops() only requires struct device * and the domain pointer (group->default_domain); it does not mutate any fields in struct iommu_group. Moving the iommu_setup_dma_ops() calls after mutex_unlock(&group->mutex) in iommu_probe_device(), bus_iommu_probe(), and iommu_group_store_type() breaks the initial &group->mutex -> cpu_hotplug_lock dependency.

Are there any other code paths that rely on group->mutex --> mm->mmap_lock ordering? If so fixing this one case wouldn't help.

Quoted 7 lines

2. Avoid holding down_write(&vdev->memory_lock) across pci_try_reset_function() in VFIO:
vfio-pci could zap active BAR mappings under memory_lock and set a state flag / disable memory decoding, drop memory_lock before calling pci_try_reset_function(), and then re-acquire memory_lock to re-enable memory. While resetting, any concurrent user fault will see the memory disabled condition and return VM_FAULT_SIGBUS safely.

This would change the userspace-visible behavior of faulting on a VFIO device BAR from "block until reset is done and the succeed" to "fail with SIGBUS". And it would allow VFIO to access VFIO device BARs during the reset through vfio_pci_core_iowrite*().

But I think we can extend this idea to solve those problems by introducing a wait queue for tasks to sit on while a device is being reset.

e.g. Something like this (completely untested and partially written by AI):

From: David Matlack <[email protected]>
Date: Tue, 25 Aug 2026 18:37:52 +0000
Subject: [PATCH] vfio/pci: Avoid circular locking dependency during device reset

Avoid a circular locking dependency during VFIO device reset by dropping vdev->memory_lock prior to calling PCI reset functions (pci_try_reset_function() and pci_reset_bus()). Introduce an explicit reset state flag (vdev->resetting) and wait queue (vdev->reset_done_wq) to stall concurrent BAR page faults and MMIO accesses during reset without holding vdev->memory_lock across PCI reset operations.

Export a new helper function vfio_pci_core_try_reset() to consolidate the common reset sequence across vfio_pci_ioctl_reset() and PCI config space FLR handlers.

Export a new helper function vfio_pci_core_down_read_memory_lock() to encapsulate acquiring down_read(&vdev->memory_lock) while waiting on vdev->reset_done_wq if a reset is in progress.

Holding vdev->memory_lock across pci_try_reset_function() or pci_reset_bus() causes a lock inversion between vdev->memory_lock and the IOMMU group mutex (group->mutex). PCI reset functions invoke pci_dev_reset_iommu_prepare(), which acquires group->mutex. However, vdev->memory_lock is acquired inside page fault context (vfio_pci_mmap_huge_fault()), placing vdev->memory_lock below mm->mmap_lock in the locking hierarchy. Meanwhile, operations holding group->mutex (such as sysfs interactions or IOMMU domain operations) can fault on user memory, placing group->mutex above mm->mmap_lock. This establishes the circular locking chain:

group->mutex --> mm->mmap_lock --> vdev->memory_lock --> group->mutex

Simply dropping vdev->memory_lock before calling PCI reset routines would alter userspace-visible behavior by causing concurrent BAR page faults during reset to fail with VM_FAULT_SIGBUS rather than stalling until reset completes.

Preserve the stalling behavior while resolving the deadlock:
1. In vfio_pci_core_try_reset(), set vdev->resetting = true under vdev->memory_lock, zap BAR mmaps, and drop vdev->memory_lock before calling pci_try_reset_function().
2. In vfio_pci_core_down_read_memory_lock(), if vdev->resetting is true, drop memory_lock and sleep uninterruptibly on vdev->reset_done_wq via wait_event().
3. Upon reset completion, clear vdev->resetting = false and wake up the wait queue.

Fixes f5b16b802174 ("PCI: Suspend iommu function prior to resetting a device")
---
 drivers/vfio/pci/vfio_pci_config.c | 20 +++---------
 drivers/vfio/pci/vfio_pci_core.c   | 51 ++++++++++++++++++++++++++----
 drivers/vfio/pci/vfio_pci_rdwr.c   |  4 +--
 include/linux/vfio_pci_core.h      |  4 +++
 4 files changed, 54 insertions(+), 25 deletions(-)
drivers/vfio/pci/vfio_pci_config.c +4 -16
index 9914f3ac69ae..4980f53c21ca 100644
@@ -907,14 +907,8 @@ static int vfio_exp_config_write(struct vfio_pci_core_device *vdev, int pos,
 						 pos - offset + PCI_EXP_DEVCAP,
 						 &cap);
 
-		if (!ret && (cap & PCI_EXP_DEVCAP_FLR)) {
-			vfio_pci_zap_and_down_write_memory_lock(vdev);
-			vfio_pci_dma_buf_move(vdev, true);
-			pci_try_reset_function(vdev->pdev);
-			if (__vfio_pci_memory_enabled(vdev))
-				vfio_pci_dma_buf_move(vdev, false);
-			up_write(&vdev->memory_lock);
-		}
+		if (!ret && (cap & PCI_EXP_DEVCAP_FLR))
+			vfio_pci_core_try_reset(vdev);
 	}
 
 	/*
@@ -992,14 +986,8 @@ static int vfio_af_config_write(struct vfio_pci_core_device *vdev, int pos,
 						pos - offset + PCI_AF_CAP,
 						&cap);
 
-		if (!ret && (cap & PCI_AF_CAP_FLR) && (cap & PCI_AF_CAP_TP)) {
-			vfio_pci_zap_and_down_write_memory_lock(vdev);
-			vfio_pci_dma_buf_move(vdev, true);
-			pci_try_reset_function(vdev->pdev);
-			if (__vfio_pci_memory_enabled(vdev))
-				vfio_pci_dma_buf_move(vdev, false);
-			up_write(&vdev->memory_lock);
-		}
+		if (!ret && (cap & PCI_AF_CAP_FLR) && (cap & PCI_AF_CAP_TP))
+			vfio_pci_core_try_reset(vdev);
 	}
 
 	return count;
drivers/vfio/pci/vfio_pci_core.c +44 -7

112 lines not on this page · raw message

drivers/vfio/pci/vfio_pci_rdwr.c +2 -2
index 7f14dd46de17..4313972e35ef 100644
@@ -43,7 +43,7 @@ int vfio_pci_core_iowrite##size(struct vfio_pci_core_device *vdev,	\
 			bool test_mem, u##size val, void __iomem *io)	\
 {									\
 	if (test_mem) {							\
-		down_read(&vdev->memory_lock);				\
+		vfio_pci_core_down_read_memory_lock(vdev);		\
 		if (!__vfio_pci_memory_enabled(vdev)) {			\
 			up_read(&vdev->memory_lock);			\
 			return -EIO;					\
@@ -69,7 +69,7 @@ int vfio_pci_core_ioread##size(struct vfio_pci_core_device *vdev,	\
 			bool test_mem, u##size *val, void __iomem *io)	\
 {									\
 	if (test_mem) {							\
-		down_read(&vdev->memory_lock);				\
+		vfio_pci_core_down_read_memory_lock(vdev);		\
 		if (!__vfio_pci_memory_enabled(vdev)) {			\
 			up_read(&vdev->memory_lock);			\
 			return -EIO;					\
include/linux/vfio_pci_core.h +4 -0
index 9a1674c152aa..152662be3795 100644
@@ -148,6 +148,8 @@ struct vfio_pci_core_device {
 	struct vfio_pci_core_device	*sriov_pf_core_dev;
 	struct notifier_block	nb;
 	struct rw_semaphore	memory_lock;
+	bool			resetting;
+	wait_queue_head_t	reset_done_wq;
 	struct list_head	dmabufs;
 };
 
@@ -171,6 +173,8 @@ void vfio_pci_core_unregister_device(struct vfio_pci_core_device *vdev);
 extern const struct pci_error_handlers vfio_pci_core_err_handlers;
 int vfio_pci_core_sriov_configure(struct vfio_pci_core_device *vdev,
 				  int nr_virtfn);
+int vfio_pci_core_try_reset(struct vfio_pci_core_device *vdev);
+void vfio_pci_core_down_read_memory_lock(struct vfio_pci_core_device *vdev);
 long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd,
 		unsigned long arg);
 int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags,
Quoted 5 lines

3. Refine synchronization in pci_dev_reset_iommu_prepare():
Evaluate if attaching to the blocking domain and pausing ATS during device reset can be protected using more fine-grained locking or atomic state flags without holding the coarse &group->mutex.

I don't know enough about this part of the kernel to say, but this would directly address the new lock ordering dependency vdev->memory_lock --> group->mutex introduced by commit f5b16b802174 ("PCI: Suspend iommu function prior to resetting a device"), which is what led to this lockdep error.

maintainer
#4 lore raw
Quoted 40 lines · David Matlack
Quoted 10 lines · Vipin Sharma

================================================================================
Potential Solutions Suggested by AI
================================================================================

1. Decouple iommu_setup_dma_ops() from group->mutex in drivers/iommu/iommu.c:
iommu_setup_dma_ops() only requires struct device * and the domain pointer (group->default_domain); it does not mutate any fields in struct iommu_group. Moving the iommu_setup_dma_ops() calls after mutex_unlock(&group->mutex) in iommu_probe_device(), bus_iommu_probe(), and iommu_group_store_type() breaks the initial &group->mutex -> cpu_hotplug_lock dependency.

Are there any other code paths that rely on group->mutex --> mm->mmap_lock ordering? If so fixing this one case wouldn't help.

Quoted 7 lines

2. Avoid holding down_write(&vdev->memory_lock) across pci_try_reset_function() in VFIO:
vfio-pci could zap active BAR mappings under memory_lock and set a state flag / disable memory decoding, drop memory_lock before calling pci_try_reset_function(), and then re-acquire memory_lock to re-enable memory. While resetting, any concurrent user fault will see the memory disabled condition and return VM_FAULT_SIGBUS safely.

This would change the userspace-visible behavior of faulting on a VFIO device BAR from "block until reset is done and the succeed" to "fail with SIGBUS". And it would allow VFIO to access VFIO device BARs during the reset through vfio_pci_core_iowrite*().

But I think we can extend this idea to solve those problems by introducing a wait queue for tasks to sit on while a device is being reset.

e.g. Something like this (completely untested and partially written by AI):

From: David Matlack <[email protected]>
Date: Tue, 25 Aug 2026 18:37:52 +0000
Subject: [PATCH] vfio/pci: Avoid circular locking dependency during device reset

Quoted 6 lines

Avoid a circular locking dependency during VFIO device reset by dropping vdev->memory_lock prior to calling PCI reset functions (pci_try_reset_function() and pci_reset_bus()). Introduce an explicit reset state flag (vdev->resetting) and wait queue (vdev->reset_done_wq) to stall concurrent BAR page faults and MMIO accesses during reset without holding vdev->memory_lock across PCI reset operations.

...

This approach does not look ideal. The implementation has a bug where concurrent resets can lead to vdev->resetting being cleared too early. And from a maintainability perspective, there are more call sites that currently take memory_lock that would probably also have to be updated to wait for vdev->resetting to become false.

Quoted 10 lines
Quoted 5 lines

3. Refine synchronization in pci_dev_reset_iommu_prepare():
Evaluate if attaching to the blocking domain and pausing ATS during device reset can be protected using more fine-grained locking or atomic state flags without holding the coarse &group->mutex.

I don't know enough about this part of the kernel to say, but this would directly address the new lock ordering dependency vdev->memory_lock --> group->mutex introduced by commit f5b16b802174 ("PCI: Suspend iommu function prior to resetting a device"), which is what led to this lockdep error.

maintainer
#5 lore raw
Quoted 48 lines · David Matlack
Quoted 40 lines · David Matlack
Quoted 10 lines · Vipin Sharma

================================================================================
Potential Solutions Suggested by AI
================================================================================

1. Decouple iommu_setup_dma_ops() from group->mutex in drivers/iommu/iommu.c:
iommu_setup_dma_ops() only requires struct device * and the domain pointer (group->default_domain); it does not mutate any fields in struct iommu_group. Moving the iommu_setup_dma_ops() calls after mutex_unlock(&group->mutex) in iommu_probe_device(), bus_iommu_probe(), and iommu_group_store_type() breaks the initial &group->mutex -> cpu_hotplug_lock dependency.

Are there any other code paths that rely on group->mutex --> mm->mmap_lock ordering? If so fixing this one case wouldn't help.

Quoted 7 lines

2. Avoid holding down_write(&vdev->memory_lock) across pci_try_reset_function() in VFIO:
vfio-pci could zap active BAR mappings under memory_lock and set a state flag / disable memory decoding, drop memory_lock before calling pci_try_reset_function(), and then re-acquire memory_lock to re-enable memory. While resetting, any concurrent user fault will see the memory disabled condition and return VM_FAULT_SIGBUS safely.

This would change the userspace-visible behavior of faulting on a VFIO device BAR from "block until reset is done and the succeed" to "fail with SIGBUS". And it would allow VFIO to access VFIO device BARs during the reset through vfio_pci_core_iowrite*().

But I think we can extend this idea to solve those problems by introducing a wait queue for tasks to sit on while a device is being reset.

e.g. Something like this (completely untested and partially written by AI):

From: David Matlack <[email protected]>
Date: Tue, 25 Aug 2026 18:37:52 +0000
Subject: [PATCH] vfio/pci: Avoid circular locking dependency during device reset

Quoted 6 lines

Avoid a circular locking dependency during VFIO device reset by dropping vdev->memory_lock prior to calling PCI reset functions (pci_try_reset_function() and pci_reset_bus()). Introduce an explicit reset state flag (vdev->resetting) and wait queue (vdev->reset_done_wq) to stall concurrent BAR page faults and MMIO accesses during reset without holding vdev->memory_lock across PCI reset operations.

Quoted 18 lines

...

This approach does not look ideal. The implementation has a bug where concurrent resets can lead to vdev->resetting being cleared too early. And from a maintainability perspective, there are more call sites that currently take memory_lock that would probably also have to be updated to wait for vdev->resetting to become false.

Quoted 10 lines
Quoted 5 lines

3. Refine synchronization in pci_dev_reset_iommu_prepare():
Evaluate if attaching to the blocking domain and pausing ATS during device reset can be protected using more fine-grained locking or atomic state flags without holding the coarse &group->mutex.

I don't know enough about this part of the kernel to say, but this would directly address the new lock ordering dependency vdev->memory_lock --> group->mutex introduced by commit f5b16b802174 ("PCI: Suspend iommu function prior to resetting a device"), which is what led to this lockdep error.

I sent a proposed fix that takes this approach here:

https://lore.kernel.org/linux-iommu/[email protected]/

#6 lore raw
In reply to David Matlack
Quoted 8 lines · David Matlack
Quoted 7 lines · David Matlack
Quoted 4 lines · Vipin Sharma

================================================================================
Potential Solutions Suggested by AI
================================================================================

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

1. Decouple iommu_setup_dma_ops() from group->mutex in drivers/iommu/iommu.c:
iommu_setup_dma_ops() only requires struct device * and the domain pointer (group->default_domain); it does not mutate any fields in struct iommu_group. Moving the iommu_setup_dma_ops() calls after mutex_unlock(&group->mutex) in iommu_probe_device(), bus_iommu_probe(), and iommu_group_store_type() breaks the initial &group->mutex -> cpu_hotplug_lock dependency.

The group->mutex is needed here also since it sets up the dma_ops on the default_domain that is currently attached to the device. And those attachments are protected with group->mutex.

Quoted 33 lines
Quoted 27 lines

Are there any other code paths that rely on group->mutex --> mm->mmap_lock ordering? If so fixing this one case wouldn't help.

Quoted 7 lines

2. Avoid holding down_write(&vdev->memory_lock) across pci_try_reset_function() in VFIO:
vfio-pci could zap active BAR mappings under memory_lock and set a state flag / disable memory decoding, drop memory_lock before calling pci_try_reset_function(), and then re-acquire memory_lock to re-enable memory. While resetting, any concurrent user fault will see the memory disabled condition and return VM_FAULT_SIGBUS safely.

This would change the userspace-visible behavior of faulting on a VFIO device BAR from "block until reset is done and the succeed" to "fail with SIGBUS". And it would allow VFIO to access VFIO device BARs during the reset through vfio_pci_core_iowrite*().

But I think we can extend this idea to solve those problems by introducing a wait queue for tasks to sit on while a device is being reset.

e.g. Something like this (completely untested and partially written by AI):

From: David Matlack <[email protected]>
Date: Tue, 25 Aug 2026 18:37:52 +0000
Subject: [PATCH] vfio/pci: Avoid circular locking dependency during device reset

Quoted 6 lines

Avoid a circular locking dependency during VFIO device reset by dropping vdev->memory_lock prior to calling PCI reset functions (pci_try_reset_function() and pci_reset_bus()). Introduce an explicit reset state flag (vdev->resetting) and wait queue (vdev->reset_done_wq) to stall concurrent BAR page faults and MMIO accesses during reset without holding vdev->memory_lock across PCI reset operations.

Quoted 13 lines

...

This approach does not look ideal. The implementation has a bug where concurrent resets can lead to vdev->resetting being cleared too early. And from a maintainability perspective, there are more call sites that currently take memory_lock that would probably also have to be updated to wait for vdev->resetting to become false.

Quoted 5 lines
Quoted 5 lines

3. Refine synchronization in pci_dev_reset_iommu_prepare():
Evaluate if attaching to the blocking domain and pausing ATS during device reset can be protected using more fine-grained locking or atomic state flags without holding the coarse &group->mutex.

This might not work as reset_iommu_prepare() changes the domain of the device being reset and those things protected by the group->mutex.

Quoted 5 lines
Quoted 5 lines

I don't know enough about this part of the kernel to say, but this would directly address the new lock ordering dependency vdev->memory_lock --> group->mutex introduced by commit f5b16b802174 ("PCI: Suspend iommu function prior to resetting a device"), which is what led to this lockdep error.

Thanks,
Sami