Skip to contentSkip to conversation

cxl/memdev: Fix poison debugfs vs unbind deadlock

PATCH Superseded Guixin Liu 4 participants 16 messages Sep 10 – Sep 16

Subsystems (2)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Guixin Liu

author
#0 lore raw

v3 cover cxl/memdev: Fix poison debugfs vs unbind deadlock

Writing a memdev poison debugfs file while cxl_mem is being unbound deadlocks. Patch 2 fixes it by not waiting for the device lock in those handlers. Patch 1 adds the trylock guard it uses.

Patch 2 does not build without patch 1, so the two need to travel together.

Testing:

Reproduced on a QEMU CXL topology whose type3 devices advertise poison inject support:

while :; do echo 0 > /sys/kernel/debug/cxl/mem0/inject_poison; done & while :; do
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/unbind
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/bind
done

Without the fix the unbind wedges within seconds. The three tasks involved, from /proc/<pid>/stack:

writer, state S, holds the debugfs reference and waits for the lock

    cxl_debugfs_poison_inject+0x25/0xa0 [cxl_mem]
    debugfs_attr_write+0x61/0xb0
    full_proxy_write+0xfc/0x1c0
    vfs_write+0x1d4/0xe60

unbind, state D, holds the lock and waits for the reference to drain

    remove_one+0x27f/0x3d0
    debugfs_remove+0x44/0x60
    release_nodes+0xfa/0x2c0
    devres_release_all+0x113/0x1a0
    device_unbind_cleanup+0x76/0x260
    device_release_driver_internal+0x3eb/0x540
    unbind_store+0xde/0x100

cxl_port workqueue, state D, blocked on the same lock

    device_release_driver_internal+0x96/0x540
    detach_memdev+0x79/0xb0 [cxl_core]
    process_one_work+0x6b0/0xfb0

The writer is in interruptible sleep and can be killed; the unbind cannot, and because cxl_bus_wq is an ordered workqueue the wedged detach_memdev() blocks every other CXL bus work item behind it.

With both patches applied, 46 unbind/bind cycles against the same writer loop all completed, no task was left in D state, and the writer collected 10920 EBUSY returns from the contended trylock. Note that lockdep stays quiet either way: one leg of the cycle is the debugfs active_users completion rather than a lock it tracks.

v1 -> v2:
- add the device_trylock() guard and use ACQUIRE(device_try, ...) instead of open-coding device_trylock()/device_unlock(), keeping the style the
Fixes: commit established (Shaikh Kamaluddin)
- cut the changelog down to the failing condition, the consequence and the fix; the call graph and the reproducer live here instead
- say how the issue was found and how it was tested

v2 -> v3:
- rebase onto v7.3-rc2 (master), per Dave's request to send the series against Linus's tags rather than cxl/next

v1:
https://lore.kernel.org/linux-cxl/[email protected]/

v2:
https://lore.kernel.org/linux-cxl/[email protected]/

Guixin Liu (2):
driver core: Add conditional guard support for device_trylock() cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

 drivers/cxl/mem.c      | 14 ++++++++++----
 include/linux/device.h |  1 +
 2 files changed, 11 insertions(+), 4 deletions(-)

base-commit: 50d05c7c76c96b90462f24debacca971d2e86713 --
2.43.7

author
#1 lore raw

PATCH v3 1/2 driver core: Add conditional guard support for device_trylock()

Introduce a conditional guard version of device_trylock() for scenarios that must not block on the device lock. device_trylock() returns 1 on success like mutex_trylock(), so the default binary condition applies and ACQUIRE_ERR() reports -EBUSY on contention.

---
 include/linux/device.h | 1 +
 1 file changed, 1 insertion(+)
include/linux/device.h +1 -0
index aee79fd6b32b..a8d6cf76d137 100644
@@ -1121,6 +1121,7 @@ static inline void device_unlock(struct device *dev)
 
 DEFINE_GUARD(device, struct device *, device_lock(_T), device_unlock(_T))
 DEFINE_GUARD_COND(device, _intr, device_lock_interruptible(_T), _RET == 0)
+DEFINE_GUARD_COND(device, _try, device_trylock(_T))
 
 static inline void device_lock_assert(struct device *dev)
 {
Signature · 1 line
2.43.7
maintainer Reviewed-by on v3 1/2
#2 lore raw

Re: [PATCH v3 1/2] driver core: Add conditional guard support for device_trylock()

Quoted 7 lines · On Thu, 10 Sep 2026 17:40:16 +0800 Guixin Liu <[email protected]> wrote:

Introduce a conditional guard version of device_trylock() for scenarios that must not block on the device lock. device_trylock() returns 1 on success like mutex_trylock(), so the default binary condition applies and ACQUIRE_ERR() reports -EBUSY on contention.

author
#3 lore raw

PATCH v3 2/2 cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

In reply to Guixin Liu

The poison debugfs handlers take the memdev device lock so that the region lookup sees a stable cxlmd->dev.driver. debugfs holds a reference on the file across the handler, and cxl_mem unbind removes that file while holding the very same device lock, so a handler that waits for the lock deadlocks against a concurrent unbind.

Both tasks then hang. The unbind side is uninterruptible, and it also blocks the memdev detach work, which runs on an ordered workqueue and so stalls every other CXL bus work item.

Take the lock with the trylock guard and return -EBUSY instead of waiting. An unbind that wins the race removes the file first and the write fails with -ENOENT.

Found by code inspection. Reproduced by writing inject_poison in a loop while unbinding and rebinding cxl_mem, and confirmed fixed by the same test.

Fixes 574eda81d0a7 ("cxl/memdev: Hold memdev lock during memdev poison injection/clear")

---
checkpatch reports "do not use assignment in if condition" twice, on the two ACQUIRE_ERR() lines. Those are pre-existing: the unpatched file and the Fixes: commit report the same two, this patch only swaps the lock class on them, and the combined form is what all 40 ACQUIRE_ERR() call sites in drivers/cxl use.

 drivers/cxl/mem.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
drivers/cxl/mem.c +10 -4
index 798e5c369cfc..3959ec963026 100644
@@ -50,8 +50,13 @@ static int cxl_debugfs_poison_inject(void *data, u64 dpa)
 	struct cxl_memdev *cxlmd = data;
 	int rc;
 
-	ACQUIRE(device_intr, devlock)(&cxlmd->dev);
-	if ((rc = ACQUIRE_ERR(device_intr, &devlock)))
+	/*
+	 * Never wait for this lock: the debugfs proxy holds a file reference
+	 * across the callback and unbind removes the file under the same
+	 * device lock, so waiting here deadlocks against unbind.
+	 */
+	ACQUIRE(device_try, devlock)(&cxlmd->dev);
+	if ((rc = ACQUIRE_ERR(device_try, &devlock)))
 		return rc;
 
 	return cxl_inject_poison(cxlmd, dpa);
@@ -65,8 +70,9 @@ static int cxl_debugfs_poison_clear(void *data, u64 dpa)
 	struct cxl_memdev *cxlmd = data;
 	int rc;
 
-	ACQUIRE(device_intr, devlock)(&cxlmd->dev);
-	if ((rc = ACQUIRE_ERR(device_intr, &devlock)))
+	/* Never wait, per the inject path above. */
+	ACQUIRE(device_try, devlock)(&cxlmd->dev);
+	if ((rc = ACQUIRE_ERR(device_try, &devlock)))
 		return rc;
 
 	return cxl_clear_poison(cxlmd, dpa);
Signature · 1 line
2.43.7
#4 lore raw

Re: [PATCH v3 2/2] cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

Quoted 10 lines · Guixin Liu

The poison debugfs handlers take the memdev device lock so that the region lookup sees a stable cxlmd->dev.driver. debugfs holds a reference on the file across the handler, and cxl_mem unbind removes that file while holding the very same device lock, so a handler that waits for the lock deadlocks against a concurrent unbind.

Both tasks then hang. The unbind side is uninterruptible, and it also blocks the memdev detach work, which runs on an ordered workqueue and so stalls every other CXL bus work item.

Quoted 7 lines

Take the lock with the trylock guard and return -EBUSY instead of waiting. An unbind that wins the race removes the file first and the write fails with -ENOENT.

Found by code inspection. Reproduced by writing inject_poison in a loop while unbinding and rebinding cxl_mem, and confirmed fixed by the same test.

Why would anyone normally "unbind" cxl_mem at all? That will taint kernels soon, so you don't normally want to do that, right?

And debugfs is root-only, so this is a "root did something bad, and gets to keep the mess", right? This should not ever be a normal operation.

thanks,

greg k-h

author
#5 lore raw

Re: [PATCH v3 2/2] cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

Quoted 15 lines · Greg Kroah-Hartman
Quoted 14 lines · Guixin Liu

The poison debugfs handlers take the memdev device lock so that the region lookup sees a stable cxlmd->dev.driver. debugfs holds a reference on the file across the handler, and cxl_mem unbind removes that file while holding the very same device lock, so a handler that waits for the lock deadlocks against a concurrent unbind.

Both tasks then hang. The unbind side is uninterruptible, and it also blocks the memdev detach work, which runs on an ordered workqueue and so stalls every other CXL bus work item.

Take the lock with the trylock guard and return -EBUSY instead of waiting. An unbind that wins the race removes the file first and the write fails with -ENOENT.

Quoted 8 lines
Quoted 3 lines

Found by code inspection. Reproduced by writing inject_poison in a loop while unbinding and rebinding cxl_mem, and confirmed fixed by the same test.

Why would anyone normally "unbind" cxl_mem at all? That will taint kernels soon, so you don't normally want to do that, right?

And debugfs is root-only, so this is a "root did something bad, and gets to keep the mess", right? This should not ever be a normal operation.

Agreed, nobody should unbind cxl_mem in production. The sysfs unbind is in the reproducer only because it's the cheapest trigger.

The window itself is not sysfs-unbind specific.
The debugfs directory is torn down from a devm action, so every path that ends in device_release_driver_internal() hits the same wait: rmmod cxl_mem, and the memdev detach work that PCI hot-remove schedules. That detach path is the third task in the reproducer stack, the one wedged on cxl_bus_wq.
A poison write racing an rmmod or a hot-remove is not root misbehaving, and no taint flags it either.

Best Regards,
Guixin Liu

Quoted 3 lines

thanks,

greg k-h

#6 lore raw

Re: [PATCH v3 2/2] cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

Quoted 29 lines · Guixin Liu
Quoted 15 lines · Greg Kroah-Hartman
Quoted 14 lines · Guixin Liu

The poison debugfs handlers take the memdev device lock so that the region lookup sees a stable cxlmd->dev.driver. debugfs holds a reference on the file across the handler, and cxl_mem unbind removes that file while holding the very same device lock, so a handler that waits for the lock deadlocks against a concurrent unbind.

Both tasks then hang. The unbind side is uninterruptible, and it also blocks the memdev detach work, which runs on an ordered workqueue and so stalls every other CXL bus work item.

Take the lock with the trylock guard and return -EBUSY instead of waiting. An unbind that wins the race removes the file first and the write fails with -ENOENT.

Quoted 8 lines
Quoted 3 lines

Found by code inspection. Reproduced by writing inject_poison in a loop while unbinding and rebinding cxl_mem, and confirmed fixed by the same test.

Why would anyone normally "unbind" cxl_mem at all? That will taint kernels soon, so you don't normally want to do that, right?

And debugfs is root-only, so this is a "root did something bad, and gets to keep the mess", right? This should not ever be a normal operation.

Agreed, nobody should unbind cxl_mem in production. The sysfs unbind is in the reproducer only because it's the cheapest trigger.

Quoted 10 lines

The window itself is not sysfs-unbind specific.
The debugfs directory is torn down from a devm action, so every path that ends
in device_release_driver_internal() hits the same wait: rmmod cxl_mem, and the
memdev detach work that PCI hot-remove schedules. That detach path is the third
task in the reproducer stack, the one wedged on cxl_bus_wq.
A poison write racing an rmmod or a hot-remove is not root misbehaving, and no taint flags it either.

But how can cxl_mem ever be removed, it doesn't live on a bus that is hot-removable, does it?

rmmod doesn't count either, that never happens except by developers.

thanks,

greg k-h

maintainer
#7 lore raw

Re: [PATCH v3 2/2] cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

Quoted 41 lines · On Thu, 10 Sep 2026 13:58:28 +0200 Greg Kroah-Hartman…
Quoted 29 lines · Guixin Liu
Quoted 15 lines · Greg Kroah-Hartman
Quoted 14 lines · Guixin Liu

The poison debugfs handlers take the memdev device lock so that the region lookup sees a stable cxlmd->dev.driver. debugfs holds a reference on the file across the handler, and cxl_mem unbind removes that file while holding the very same device lock, so a handler that waits for the lock deadlocks against a concurrent unbind.

Both tasks then hang. The unbind side is uninterruptible, and it also blocks the memdev detach work, which runs on an ordered workqueue and so stalls every other CXL bus work item.

Take the lock with the trylock guard and return -EBUSY instead of waiting. An unbind that wins the race removes the file first and the write fails with -ENOENT.

Quoted 8 lines
Quoted 3 lines

Found by code inspection. Reproduced by writing inject_poison in a loop while unbinding and rebinding cxl_mem, and confirmed fixed by the same test.

Why would anyone normally "unbind" cxl_mem at all? That will taint kernels soon, so you don't normally want to do that, right?

And debugfs is root-only, so this is a "root did something bad, and gets to keep the mess", right? This should not ever be a normal operation.

Agreed, nobody should unbind cxl_mem in production. The sysfs unbind is in the reproducer only because it's the cheapest trigger.

Quoted 10 lines

The window itself is not sysfs-unbind specific.
The debugfs directory is torn down from a devm action, so every path that ends
in device_release_driver_internal() hits the same wait: rmmod cxl_mem, and the
memdev detach work that PCI hot-remove schedules. That detach path is the third
task in the reproducer stack, the one wedged on cxl_bus_wq.
A poison write racing an rmmod or a hot-remove is not root misbehaving, and no taint flags it either.

Quoted 2 lines

But how can cxl_mem ever be removed, it doesn't live on a bus that is hot-removable, does it?

It is effectively a child of cxl_pci and that lives on the pci bus and definitely is hotpluggable. Those flows should work fine so I see this as a real if somewhat obscure bug to fix.

Jonathan

Quoted 6 lines

rmmod doesn't count either, that never happens except by developers.

thanks,

greg k-h

maintainer Reviewed-by on v3 2/2
#8 lore raw

Re: [PATCH v3 2/2] cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

In reply to Guixin Liu
Quoted 14 lines · On Thu, 10 Sep 2026 17:40:17 +0800 Guixin Liu <[email protected]> wrote:

The poison debugfs handlers take the memdev device lock so that the region lookup sees a stable cxlmd->dev.driver. debugfs holds a reference on the file across the handler, and cxl_mem unbind removes that file while holding the very same device lock, so a handler that waits for the lock deadlocks against a concurrent unbind.

Both tasks then hang. The unbind side is uninterruptible, and it also blocks the memdev detach work, which runs on an ordered workqueue and so stalls every other CXL bus work item.

Take the lock with the trylock guard and return -EBUSY instead of waiting. An unbind that wins the race removes the file first and the write fails with -ENOENT.

Quoted 7 lines

Found by code inspection. Reproduced by writing inject_poison in a loop while unbinding and rebinding cxl_mem, and confirmed fixed by the same test.

Fixes 574eda81d0a7 ("cxl/memdev: Hold memdev lock during memdev poison injection/clear")

Probably not one to rush in but nice to clean up the deadlock even if it is a little hard to hit. If it is possible to test with a hot remove flow even better. You should be able to do that with emulation in qemu if you don't have hardware capable of safe hotplug operations.

Quoted 6 lines

---
checkpatch reports "do not use assignment in if condition" twice, on the two ACQUIRE_ERR() lines. Those are pre-existing: the unpatched file and the Fixes: commit report the same two, this patch only swaps the lock class on them, and the combined form is what all 40 ACQUIRE_ERR() call sites in drivers/cxl use.

We should fix that up. Oddly I thought we had, but guess not.

Quoted 36 lines
 drivers/cxl/mem.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
drivers/cxl/mem.c +10 -4
index 798e5c369cfc..3959ec963026 100644
@@ -50,8 +50,13 @@ static int cxl_debugfs_poison_inject(void *data, u64 dpa)
 	struct cxl_memdev *cxlmd = data;
 	int rc;
 
-	ACQUIRE(device_intr, devlock)(&cxlmd->dev);
-	if ((rc = ACQUIRE_ERR(device_intr, &devlock)))
+	/*
+	 * Never wait for this lock: the debugfs proxy holds a file reference
+	 * across the callback and unbind removes the file under the same
+	 * device lock, so waiting here deadlocks against unbind.
+	 */
+	ACQUIRE(device_try, devlock)(&cxlmd->dev);
+	if ((rc = ACQUIRE_ERR(device_try, &devlock)))
 		return rc;
 
 	return cxl_inject_poison(cxlmd, dpa);
@@ -65,8 +70,9 @@ static int cxl_debugfs_poison_clear(void *data, u64 dpa)
 	struct cxl_memdev *cxlmd = data;
 	int rc;
 
-	ACQUIRE(device_intr, devlock)(&cxlmd->dev);
-	if ((rc = ACQUIRE_ERR(device_intr, &devlock)))
+	/* Never wait, per the inject path above. */
+	ACQUIRE(device_try, devlock)(&cxlmd->dev);
+	if ((rc = ACQUIRE_ERR(device_try, &devlock)))
 		return rc;
 
 	return cxl_clear_poison(cxlmd, dpa);
author
#9 lore raw

Re: [PATCH v3 2/2] cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

Quoted 24 lines · Jonathan Cameron
Quoted 14 lines · On Thu, 10 Sep 2026 17:40:17 +0800 Guixin Liu <[email protected]> wrote:

The poison debugfs handlers take the memdev device lock so that the region lookup sees a stable cxlmd->dev.driver. debugfs holds a reference on the file across the handler, and cxl_mem unbind removes that file while holding the very same device lock, so a handler that waits for the lock deadlocks against a concurrent unbind.

Both tasks then hang. The unbind side is uninterruptible, and it also blocks the memdev detach work, which runs on an ordered workqueue and so stalls every other CXL bus work item.

Take the lock with the trylock guard and return -EBUSY instead of waiting. An unbind that wins the race removes the file first and the write fails with -ENOENT.

Quoted 7 lines

Found by code inspection. Reproduced by writing inject_poison in a loop while unbinding and rebinding cxl_mem, and confirmed fixed by the same test.

Fixes 574eda81d0a7 ("cxl/memdev: Hold memdev lock during memdev poison injection/clear")
Quoted 6 lines

Probably not one to rush in but nice to clean up the deadlock even if it is a little hard to hit. If it is possible to test with a hot remove flow even better. You should be able to do that with emulation in qemu if you don't have hardware capable of safe hotplug operations.

Sure, I reproduced this on hot-remove situation:

debugfs writer, state S, waits for the memdev lock

      cxl_debugfs_poison_inject+0x25/0xa0 [cxl_mem]
      simple_attr_write_xsigned.isra.0+0x1ca/0x2c0
      debugfs_attr_write+0x61/0xb0
      full_proxy_write+0xfc/0x1c0
      vfs_write+0x1d4/0xe60
      ksys_write+0x11f/0x250
      do_syscall_64+0xe2/0x560
      entry_SYSCALL_64_after_hwframe+0x76/0x7e

irq/28-pciehp, state D, holds the memdev lock, waits for the debugfs reference to drain

      remove_one+0x27f/0x3d0
      __simple_recursive_removal+0x183/0x4a0
      debugfs_remove+0x44/0x60
      release_nodes+0xfa/0x2c0
      devres_release_all+0x113/0x1a0
      device_unbind_cleanup+0x76/0x260
      device_release_driver_internal+0x3eb/0x540
      bus_remove_device+0x28a/0x540
      device_del+0x371/0x930
      cdev_device_del+0x1d/0xf0
      cxl_memdev_unregister+0x1c/0x70 [cxl_core]
      release_nodes+0xfa/0x2c0
      devres_release_all+0x113/0x1a0
      device_unbind_cleanup+0x76/0x260
      device_release_driver_internal+0x3eb/0x540
      pci_stop_bus_device+0x122/0x170
      pci_stop_and_remove_bus_device+0x16/0x30
      pciehp_unconfigure_device+0x1b4/0x3b0
      pciehp_disable_slot+0xf9/0x2e0
      pciehp_handle_disable_request+0x81/0x100
      pciehp_ist+0x29f/0x410
      irq_thread_fn+0x8b/0x160
      irq_thread+0x189/0x320
      kthread+0x329/0x410
      ret_from_fork+0x33b/0x670
      ret_from_fork_asm+0x1a/0x30

cxl_port workqueue, state D, waits for the memdev lock

      device_release_driver_internal+0x96/0x540
      detach_memdev+0x79/0xb0 [cxl_core]
      process_one_work+0x6b0/0xfb0
      worker_thread+0x4dd/0xd30
      kthread+0x329/0x410
      ret_from_fork+0x33b/0x670
      ret_from_fork_asm+0x1a/0x30

With this patch, the hot-remove flow completed normally.

Quoted 8 lines
Quoted 7 lines

---
checkpatch reports "do not use assignment in if condition" twice, on the two ACQUIRE_ERR() lines. Those are pre-existing: the unpatched file and the Fixes: commit report the same two, this patch only swaps the lock class on them, and the combined form is what all 40 ACQUIRE_ERR() call sites in drivers/cxl use.

We should fix that up. Oddly I thought we had, but guess not.

I think we should fix this in checkpatch.pl, like this:
    if ($c =~ /\bif\s*\(.*[^<>!=]=[^=].*/s && +    $c !~ /=\s*ACQUIRE_ERR\s*\(/) {

Best Regards,
Guixin Liu

Quoted 35 lines
Quoted 35 lines
  drivers/cxl/mem.c | 14 ++++++++++----
  1 file changed, 10 insertions(+), 4 deletions(-)
drivers/cxl/mem.c +10 -4
index 798e5c369cfc..3959ec963026 100644
@@ -50,8 +50,13 @@ static int cxl_debugfs_poison_inject(void *data, u64 dpa)
  	struct cxl_memdev *cxlmd = data;
  	int rc;
 
-	ACQUIRE(device_intr, devlock)(&cxlmd->dev);
-	if ((rc = ACQUIRE_ERR(device_intr, &devlock)))
+	/*
+	 * Never wait for this lock: the debugfs proxy holds a file reference
+	 * across the callback and unbind removes the file under the same
+	 * device lock, so waiting here deadlocks against unbind.
+	 */
+	ACQUIRE(device_try, devlock)(&cxlmd->dev);
+	if ((rc = ACQUIRE_ERR(device_try, &devlock)))
  		return rc;
 
  	return cxl_inject_poison(cxlmd, dpa);
@@ -65,8 +70,9 @@ static int cxl_debugfs_poison_clear(void *data, u64 dpa)
  	struct cxl_memdev *cxlmd = data;
  	int rc;
 
-	ACQUIRE(device_intr, devlock)(&cxlmd->dev);
-	if ((rc = ACQUIRE_ERR(device_intr, &devlock)))
+	/* Never wait, per the inject path above. */
+	ACQUIRE(device_try, devlock)(&cxlmd->dev);
+	if ((rc = ACQUIRE_ERR(device_try, &devlock)))
  		return rc;
 
  	return cxl_clear_poison(cxlmd, dpa);
maintainer
#10 lore raw

Re: [PATCH v3 2/2] cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

Quoted 35 lines · On Fri, 11 Sep 2026 11:04:47 +0800 Guixin Liu <[email protected]> wrote:
Quoted 24 lines · Jonathan Cameron
Quoted 14 lines · On Thu, 10 Sep 2026 17:40:17 +0800 Guixin Liu <[email protected]> wrote:

The poison debugfs handlers take the memdev device lock so that the region lookup sees a stable cxlmd->dev.driver. debugfs holds a reference on the file across the handler, and cxl_mem unbind removes that file while holding the very same device lock, so a handler that waits for the lock deadlocks against a concurrent unbind.

Both tasks then hang. The unbind side is uninterruptible, and it also blocks the memdev detach work, which runs on an ordered workqueue and so stalls every other CXL bus work item.

Take the lock with the trylock guard and return -EBUSY instead of waiting. An unbind that wins the race removes the file first and the write fails with -ENOENT.

Quoted 7 lines

Found by code inspection. Reproduced by writing inject_poison in a loop while unbinding and rebinding cxl_mem, and confirmed fixed by the same test.

Fixes 574eda81d0a7 ("cxl/memdev: Hold memdev lock during memdev poison injection/clear")
Quoted 6 lines

Probably not one to rush in but nice to clean up the deadlock even if it is a little hard to hit. If it is possible to test with a hot remove flow even better. You should be able to do that with emulation in qemu if you don't have hardware capable of safe hotplug operations.

Sure, I reproduced this on hot-remove situation:

debugfs writer, state S, waits for the memdev lock
      cxl_debugfs_poison_inject+0x25/0xa0 [cxl_mem]

...

Quoted 1 line

With this patch, the hot-remove flow completed normally.

Nice. Thanks for doing that.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

---
checkpatch reports "do not use assignment in if condition" twice, on the two ACQUIRE_ERR() lines. Those are pre-existing: the unpatched file and the Fixes: commit report the same two, this patch only swaps the lock class on them, and the combined form is what all 40 ACQUIRE_ERR() call sites in drivers/cxl use.

Quoted 4 lines
Quoted 1 line

We should fix that up. Oddly I thought we had, but guess not.

I think we should fix this in checkpatch.pl, like this:
    if ($c =~ /\bif\s*\(.*[^<>!=]=[^=].*/s && +    $c !~ /=\s*ACQUIRE_ERR\s*\(/) {

There are a few other macros that are wrappers of ACQUIRE_ERR that should be covered in such a patch as well. If you have time send a patch!

Thanks,

Jonathan

author
#11 lore raw

Re: [PATCH v3 2/2] cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

Quoted 54 lines · Jonathan Cameron
Quoted 35 lines · On Fri, 11 Sep 2026 11:04:47 +0800 Guixin Liu <[email protected]> wrote:
Quoted 24 lines · Jonathan Cameron
Quoted 14 lines · On Thu, 10 Sep 2026 17:40:17 +0800 Guixin Liu <[email protected]> wrote:

The poison debugfs handlers take the memdev device lock so that the region lookup sees a stable cxlmd->dev.driver. debugfs holds a reference on the file across the handler, and cxl_mem unbind removes that file while holding the very same device lock, so a handler that waits for the lock deadlocks against a concurrent unbind.

Both tasks then hang. The unbind side is uninterruptible, and it also blocks the memdev detach work, which runs on an ordered workqueue and so stalls every other CXL bus work item.

Take the lock with the trylock guard and return -EBUSY instead of waiting. An unbind that wins the race removes the file first and the write fails with -ENOENT.

Quoted 7 lines

Found by code inspection. Reproduced by writing inject_poison in a loop while unbinding and rebinding cxl_mem, and confirmed fixed by the same test.

Fixes 574eda81d0a7 ("cxl/memdev: Hold memdev lock during memdev poison injection/clear")
Quoted 6 lines

Probably not one to rush in but nice to clean up the deadlock even if it is a little hard to hit. If it is possible to test with a hot remove flow even better. You should be able to do that with emulation in qemu if you don't have hardware capable of safe hotplug operations.

Sure, I reproduced this on hot-remove situation:

debugfs writer, state S, waits for the memdev lock
      cxl_debugfs_poison_inject+0x25/0xa0 [cxl_mem]

...

Quoted 1 line

With this patch, the hot-remove flow completed normally.

Nice. Thanks for doing that.

Quoted 11 lines
Quoted 8 lines
Quoted 7 lines

---
checkpatch reports "do not use assignment in if condition" twice, on the two ACQUIRE_ERR() lines. Those are pre-existing: the unpatched file and the Fixes: commit report the same two, this patch only swaps the lock class on them, and the combined form is what all 40 ACQUIRE_ERR() call sites in drivers/cxl use.

We should fix that up. Oddly I thought we had, but guess not.

I think we should fix this in checkpatch.pl, like this:
    if ($c =~ /\bif\s*\(.*[^<>!=]=[^=].*/s && +    $c !~ /=\s*ACQUIRE_ERR\s*\(/) {

Quoted 7 lines

There are a few other macros that are wrappers of ACQUIRE_ERR that should be covered in such a patch as well. If you have time send a patch!

Thanks,

Jonathan

Sure, I have already done that, please see: [PATCH] checkpatch: don't flag ACQUIRE_ERR() assignments in if conditions

Best Regards,
Guixin Liu

maintainer
#12 lore raw

Re: [PATCH v3 2/2] cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

Quoted 59 lines · Guixin Liu
Quoted 56 lines · Jonathan Cameron
Quoted 34 lines · On Fri, 11 Sep 2026 11:04:47 +0800 Guixin Liu <[email protected]> wrote:
Quoted 23 lines · Jonathan Cameron
Quoted 14 lines · On Thu, 10 Sep 2026 17:40:17 +0800 Guixin Liu <[email protected]> wrote:

The poison debugfs handlers take the memdev device lock so that the region lookup sees a stable cxlmd->dev.driver. debugfs holds a reference on the file across the handler, and cxl_mem unbind removes that file while holding the very same device lock, so a handler that waits for the lock deadlocks against a concurrent unbind.

Both tasks then hang. The unbind side is uninterruptible, and it also blocks the memdev detach work, which runs on an ordered workqueue and so stalls every other CXL bus work item.

Take the lock with the trylock guard and return -EBUSY instead of waiting. An unbind that wins the race removes the file first and the write fails with -ENOENT.

Quoted 7 lines

Found by code inspection. Reproduced by writing inject_poison in a loop while unbinding and rebinding cxl_mem, and confirmed fixed by the same test.

Fixes 574eda81d0a7 ("cxl/memdev: Hold memdev lock during memdev poison injection/clear")
Quoted 6 lines

Probably not one to rush in but nice to clean up the deadlock even if it is a little hard to hit. If it is possible to test with a hot remove flow even better. You should be able to do that with emulation in qemu if you don't have hardware capable of safe hotplug operations.

Sure, I reproduced this on hot-remove situation:

debugfs writer, state S, waits for the memdev lock
      cxl_debugfs_poison_inject+0x25/0xa0 [cxl_mem]

...

Quoted 1 line

With this patch, the hot-remove flow completed normally.

Nice. Thanks for doing that.

Quoted 10 lines
Quoted 7 lines
Quoted 6 lines

---
checkpatch reports "do not use assignment in if condition" twice, on the two ACQUIRE_ERR() lines. Those are pre-existing: the unpatched file and the Fixes: commit report the same two, this patch only swaps the lock class on them, and the combined form is what all 40 ACQUIRE_ERR() call sites in drivers/cxl use.

We should fix that up. Oddly I thought we had, but guess not.

I think we should fix this in checkpatch.pl, like this:
    if ($c =~ /\bif\s*\(.*[^<>!=]=[^=].*/s && +    $c !~ /=\s*ACQUIRE_ERR\s*\(/) {

There are a few other macros that are wrappers of ACQUIRE_ERR that should be covered in such a patch as well. If you have time send a patch!

Quoted 5 lines
Quoted 3 lines

Thanks,

Jonathan

Sure, I have already done that, please see: [PATCH] checkpatch: don't flag ACQUIRE_ERR() assignments in if conditions

Hi Guixin,

I tried same about a year ago and it was not merged, find it here:

https://lore.kernel.org/linux-cxl/[email protected]/

Note that we in CXL land decided to keep using this syntax and ignore those checkpatch 'suggestions'.

Send me link to your checkpatch patch (can't find it? ) and I will review it.

-- Alison

Quoted 4 lines

Best Regards,
Guixin Liu

author
#13 lore raw

Re: [PATCH v3 2/2] cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

Quoted 68 lines · Alison Schofield
Quoted 58 lines · Guixin Liu
Quoted 56 lines · Jonathan Cameron
Quoted 34 lines · On Fri, 11 Sep 2026 11:04:47 +0800 Guixin Liu <[email protected]> wrote:
Quoted 23 lines · Jonathan Cameron
Quoted 14 lines · On Thu, 10 Sep 2026 17:40:17 +0800 Guixin Liu <[email protected]> wrote:

The poison debugfs handlers take the memdev device lock so that the region lookup sees a stable cxlmd->dev.driver. debugfs holds a reference on the file across the handler, and cxl_mem unbind removes that file while holding the very same device lock, so a handler that waits for the lock deadlocks against a concurrent unbind.

Both tasks then hang. The unbind side is uninterruptible, and it also blocks the memdev detach work, which runs on an ordered workqueue and so stalls every other CXL bus work item.

Take the lock with the trylock guard and return -EBUSY instead of waiting. An unbind that wins the race removes the file first and the write fails with -ENOENT.

Quoted 7 lines

Found by code inspection. Reproduced by writing inject_poison in a loop while unbinding and rebinding cxl_mem, and confirmed fixed by the same test.

Fixes 574eda81d0a7 ("cxl/memdev: Hold memdev lock during memdev poison injection/clear")
Quoted 6 lines

Probably not one to rush in but nice to clean up the deadlock even if it is a little hard to hit. If it is possible to test with a hot remove flow even better. You should be able to do that with emulation in qemu if you don't have hardware capable of safe hotplug operations.

Sure, I reproduced this on hot-remove situation:

debugfs writer, state S, waits for the memdev lock
      cxl_debugfs_poison_inject+0x25/0xa0 [cxl_mem]

...

Quoted 1 line

With this patch, the hot-remove flow completed normally.

Nice. Thanks for doing that.

Quoted 10 lines
Quoted 7 lines
Quoted 6 lines

---
checkpatch reports "do not use assignment in if condition" twice, on the two ACQUIRE_ERR() lines. Those are pre-existing: the unpatched file and the Fixes: commit report the same two, this patch only swaps the lock class on them, and the combined form is what all 40 ACQUIRE_ERR() call sites in drivers/cxl use.

We should fix that up. Oddly I thought we had, but guess not.

I think we should fix this in checkpatch.pl, like this:

       if ($c =~ /\bif\s*\(.*[^<>!=]=[^=].*/s &&
      +    $c !~ /=\s*ACQUIRE_ERR\s*\(/) {

There are a few other macros that are wrappers of ACQUIRE_ERR that should be covered in such a patch as well. If you have time send a patch!

Quoted 5 lines
Quoted 3 lines

Thanks,

Jonathan

Sure, I have already done that, please see: [PATCH] checkpatch: don't flag ACQUIRE_ERR() assignments in if conditions

Hi Guixin,

I tried same about a year ago and it was not merged, find it here:

Quoted 8 lines

https://lore.kernel.org/linux-cxl/[email protected]/

Note that we in CXL land decided to keep using this syntax and ignore those checkpatch 'suggestions'.

Send me link to your checkpatch patch (can't find it? ) and I will review it.

-- Alison

Here is my patch:

https://lore.kernel.org/all/[email protected]/

Best Regards,
Guixin Liu

Quoted 4 lines
Quoted 4 lines

Best Regards,
Guixin Liu

maintainer
#14 lore raw

Re: [PATCH v3 2/2] cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

In reply to Guixin Liu
Quoted 10 lines · Guixin Liu

The poison debugfs handlers take the memdev device lock so that the region lookup sees a stable cxlmd->dev.driver. debugfs holds a reference on the file across the handler, and cxl_mem unbind removes that file while holding the very same device lock, so a handler that waits for the lock deadlocks against a concurrent unbind.

Both tasks then hang. The unbind side is uninterruptible, and it also blocks the memdev detach work, which runs on an ordered workqueue and so stalls every other CXL bus work item.

Quoted 7 lines

Take the lock with the trylock guard and return -EBUSY instead of waiting. An unbind that wins the race removes the file first and the write fails with -ENOENT.

Found by code inspection. Reproduced by writing inject_poison in a loop while unbinding and rebinding cxl_mem, and confirmed fixed by the same test.

Hi Guixin,

I'm good w closing this gap, however, I'm cautious about claiming more robust support for this debugfs interface than we intend. We offer this poison inject and clear ABI for expert users, mostly device vendors, to test their CXL device capabilities. If those users take the step to do something like 'hotplugging while poisoning', well then they are just being reckless.

This patch commit message should make it clear the user scenario that would hit the deadlock, and that users should NOT do that intentionally! But if they do, this fixup prevents deadlock.

BTW - I really do appreciate that you replicated this as opposed to many other patches that both begin and end with LLM suggestions and give no consideration to, nor demonstrate an understanding of, the actual user visible impact.

-- ALison

Quoted 49 lines
Fixes 574eda81d0a7 ("cxl/memdev: Hold memdev lock during memdev poison injection/clear")

---
checkpatch reports "do not use assignment in if condition" twice, on the two ACQUIRE_ERR() lines. Those are pre-existing: the unpatched file and the Fixes: commit report the same two, this patch only swaps the lock class on them, and the combined form is what all 40 ACQUIRE_ERR() call sites in drivers/cxl use.

 drivers/cxl/mem.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
drivers/cxl/mem.c +10 -4
index 798e5c369cfc..3959ec963026 100644
@@ -50,8 +50,13 @@ static int cxl_debugfs_poison_inject(void *data, u64 dpa)
 	struct cxl_memdev *cxlmd = data;
 	int rc;
 
-	ACQUIRE(device_intr, devlock)(&cxlmd->dev);
-	if ((rc = ACQUIRE_ERR(device_intr, &devlock)))
+	/*
+	 * Never wait for this lock: the debugfs proxy holds a file reference
+	 * across the callback and unbind removes the file under the same
+	 * device lock, so waiting here deadlocks against unbind.
+	 */
+	ACQUIRE(device_try, devlock)(&cxlmd->dev);
+	if ((rc = ACQUIRE_ERR(device_try, &devlock)))
 		return rc;
 
 	return cxl_inject_poison(cxlmd, dpa);
@@ -65,8 +70,9 @@ static int cxl_debugfs_poison_clear(void *data, u64 dpa)
 	struct cxl_memdev *cxlmd = data;
 	int rc;
 
-	ACQUIRE(device_intr, devlock)(&cxlmd->dev);
-	if ((rc = ACQUIRE_ERR(device_intr, &devlock)))
+	/* Never wait, per the inject path above. */
+	ACQUIRE(device_try, devlock)(&cxlmd->dev);
+	if ((rc = ACQUIRE_ERR(device_try, &devlock)))
 		return rc;
 
 	return cxl_clear_poison(cxlmd, dpa);
Signature · 1 line
2.43.7
author
#15 lore raw

Re: [PATCH v3 2/2] cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

Quoted 26 lines · Alison Schofield
Quoted 10 lines · Guixin Liu

The poison debugfs handlers take the memdev device lock so that the region lookup sees a stable cxlmd->dev.driver. debugfs holds a reference on the file across the handler, and cxl_mem unbind removes that file while holding the very same device lock, so a handler that waits for the lock deadlocks against a concurrent unbind.

Both tasks then hang. The unbind side is uninterruptible, and it also blocks the memdev detach work, which runs on an ordered workqueue and so stalls every other CXL bus work item.

Quoted 7 lines

Take the lock with the trylock guard and return -EBUSY instead of waiting. An unbind that wins the race removes the file first and the write fails with -ENOENT.

Found by code inspection. Reproduced by writing inject_poison in a loop while unbinding and rebinding cxl_mem, and confirmed fixed by the same test.

Hi Guixin,

I'm good w closing this gap, however, I'm cautious about claiming more robust support for this debugfs interface than we intend. We offer this poison inject and clear ABI for expert users, mostly device vendors, to test their CXL device capabilities. If those users take the step to do something like 'hotplugging while poisoning', well then they are just being reckless.

Quoted 3 lines

This patch commit message should make it clear the user scenario that would hit the deadlock, and that users should NOT do that intentionally! But if they do, this fixup prevents deadlock.

Sure, I will change this in v4, thanks.

Best Regards,
Guixin Liu

Quoted 56 lines

BTW - I really do appreciate that you replicated this as opposed to many other patches that both begin and end with LLM suggestions and give no consideration to, nor demonstrate an understanding of, the actual user visible impact.

-- ALison

Quoted 48 lines
Fixes 574eda81d0a7 ("cxl/memdev: Hold memdev lock during memdev poison injection/clear")

---
checkpatch reports "do not use assignment in if condition" twice, on the two ACQUIRE_ERR() lines. Those are pre-existing: the unpatched file and the Fixes: commit report the same two, this patch only swaps the lock class on them, and the combined form is what all 40 ACQUIRE_ERR() call sites in drivers/cxl use.

  drivers/cxl/mem.c | 14 ++++++++++----
  1 file changed, 10 insertions(+), 4 deletions(-)
drivers/cxl/mem.c +10 -4
index 798e5c369cfc..3959ec963026 100644
@@ -50,8 +50,13 @@ static int cxl_debugfs_poison_inject(void *data, u64 dpa)
  	struct cxl_memdev *cxlmd = data;
  	int rc;
 
-	ACQUIRE(device_intr, devlock)(&cxlmd->dev);
-	if ((rc = ACQUIRE_ERR(device_intr, &devlock)))
+	/*
+	 * Never wait for this lock: the debugfs proxy holds a file reference
+	 * across the callback and unbind removes the file under the same
+	 * device lock, so waiting here deadlocks against unbind.
+	 */
+	ACQUIRE(device_try, devlock)(&cxlmd->dev);
+	if ((rc = ACQUIRE_ERR(device_try, &devlock)))
  		return rc;
 
  	return cxl_inject_poison(cxlmd, dpa);
@@ -65,8 +70,9 @@ static int cxl_debugfs_poison_clear(void *data, u64 dpa)
  	struct cxl_memdev *cxlmd = data;
  	int rc;
 
-	ACQUIRE(device_intr, devlock)(&cxlmd->dev);
-	if ((rc = ACQUIRE_ERR(device_intr, &devlock)))
+	/* Never wait, per the inject path above. */
+	ACQUIRE(device_try, devlock)(&cxlmd->dev);
+	if ((rc = ACQUIRE_ERR(device_try, &devlock)))
  		return rc;
 
  	return cxl_clear_poison(cxlmd, dpa);
Signature · 1 line
2.43.7