Skip to contentSkip to conversation

cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

PATCH Superseded Guixin Liu 2 participants 3 messages Aug 26 – Aug 31 CXL

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Guixin Liu

author
#0 lore raw

PATCH 1/1 cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

Writing to the memdev poison debugfs files while cxl_mem is being unbound deadlocks. The unbind path takes the memdev device lock and then waits for the debugfs file to drain, while the writer holds the debugfs reference and waits for that same lock:

unbind_store() echo 0 > .../mem0/inject_poison

    device_release_driver_internal()   full_proxy_write()
      <takes cxlmd->dev lock>            <takes debugfs_file_get() ref>
      device_unbind_cleanup()            debugfs_attr_write()
        devres_release_all()               cxl_debugfs_poison_inject()
          remove_debugfs()                   <waits for cxlmd->dev lock>
            debugfs_remove_recursive()
              __debugfs_file_removed()
                <waits for the ref to drain>

Neither side can make progress. A third task piles up behind the held device lock: detach_memdev() runs on cxl_bus_wq, and since that is an ordered workqueue the wedged work item stalls every other CXL bus work item, so unrelated devices stop being rescanned or detached.

The write side only needs the device lock so that cxl_dpa_to_region() sees a stable cxlmd->dev.driver. Nothing requires the caller to wait for the lock, so use device_trylock() and report -EBUSY instead. That keeps the debugfs reference from ever being held across a wait for the lock, which breaks the cycle. A concurrent unbind now either completes before the write starts, in which case the write fails with -ENOENT because the file is gone, or loses the race and the write reports -EBUSY.

The unbind task sits in uninterruptible sleep and cannot be killed. The writer can be signalled, which is the only reason the current code is recoverable at all.

Reproducer, on a device that advertises poison inject support:

while :; do echo 0 > /sys/kernel/debug/cxl/mem0/inject_poison; done & while :; do
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/unbind
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/bind
done

This hits within milliseconds. Note that lockdep stays quiet: one end of the cycle is the debugfs active_users completion rather than a lock it tracks. The hung task detector does report both blocked tasks.

Fixes 574eda81d0a7 ("cxl/memdev: Hold memdev lock during memdev poison injection/clear")
---
 drivers/cxl/mem.c | 28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)
drivers/cxl/mem.c +20 -8
index 798e5c369cfc..01d2d5855694 100644
@@ -50,11 +50,20 @@ static int cxl_debugfs_poison_inject(void *data, u64 dpa)
 	struct cxl_memdev *cxlmd = data;
 	int rc;
 
-	ACQUIRE(device_intr, devlock)(&cxlmd->dev);
-	if ((rc = ACQUIRE_ERR(device_intr, &devlock)))
-		return rc;
+	/*
+	 * The debugfs proxy holds a debugfs_file_get() reference across this
+	 * callback, and cxl_mem unbind removes this file from devres while
+	 * holding the same device lock. Blocking here would let unbind wait
+	 * in debugfs_remove() for a reference that cannot be dropped until
+	 * unbind releases the lock, so never wait for the lock.
+	 */
+	if (!device_trylock(&cxlmd->dev))
+		return -EBUSY;
 
-	return cxl_inject_poison(cxlmd, dpa);
+	rc = cxl_inject_poison(cxlmd, dpa);
+	device_unlock(&cxlmd->dev);
+
+	return rc;
 }
 
 DEFINE_DEBUGFS_ATTRIBUTE(cxl_poison_inject_fops, NULL,
@@ -65,11 +74,14 @@ static int cxl_debugfs_poison_clear(void *data, u64 dpa)
 	struct cxl_memdev *cxlmd = data;
 	int rc;
 
-	ACQUIRE(device_intr, devlock)(&cxlmd->dev);
-	if ((rc = ACQUIRE_ERR(device_intr, &devlock)))
-		return rc;
+	/* Avoid the unbind deadlock described in the inject path above. */
+	if (!device_trylock(&cxlmd->dev))
+		return -EBUSY;
+
+	rc = cxl_clear_poison(cxlmd, dpa);
+	device_unlock(&cxlmd->dev);
 
-	return cxl_clear_poison(cxlmd, dpa);
+	return rc;
 }
 
 DEFINE_DEBUGFS_ATTRIBUTE(cxl_poison_clear_fops, NULL,
Signature · 1 line
2.43.7
#1 lore raw
Quoted 86 lines · Guixin Liu

Writing to the memdev poison debugfs files while cxl_mem is being unbound deadlocks. The unbind path takes the memdev device lock and then waits for the debugfs file to drain, while the writer holds the debugfs reference and waits for that same lock:

unbind_store() echo 0 > .../mem0/inject_poison

    device_release_driver_internal()   full_proxy_write()
      <takes cxlmd->dev lock>            <takes debugfs_file_get() ref>
      device_unbind_cleanup()            debugfs_attr_write()
        devres_release_all()               cxl_debugfs_poison_inject()
          remove_debugfs()                   <waits for cxlmd->dev lock>
            debugfs_remove_recursive()
              __debugfs_file_removed()
                <waits for the ref to drain>

Neither side can make progress. A third task piles up behind the held device lock: detach_memdev() runs on cxl_bus_wq, and since that is an ordered workqueue the wedged work item stalls every other CXL bus work item, so unrelated devices stop being rescanned or detached.

The write side only needs the device lock so that cxl_dpa_to_region() sees a stable cxlmd->dev.driver. Nothing requires the caller to wait for the lock, so use device_trylock() and report -EBUSY instead. That keeps the debugfs reference from ever being held across a wait for the lock, which breaks the cycle. A concurrent unbind now either completes before the write starts, in which case the write fails with -ENOENT because the file is gone, or loses the race and the write reports -EBUSY.

The unbind task sits in uninterruptible sleep and cannot be killed. The writer can be signalled, which is the only reason the current code is recoverable at all.

Reproducer, on a device that advertises poison inject support:

while :; do echo 0 > /sys/kernel/debug/cxl/mem0/inject_poison; done & while :; do
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/unbind
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/bind
done

This hits within milliseconds. Note that lockdep stays quiet: one end of the cycle is the debugfs active_users completion rather than a lock it tracks. The hung task detector does report both blocked tasks.

Fixes 574eda81d0a7 ("cxl/memdev: Hold memdev lock during memdev poison injection/clear")
---
 drivers/cxl/mem.c | 28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)
drivers/cxl/mem.c +13 -7
index 798e5c369cfc..01d2d5855694 100644
@@ -50,11 +50,20 @@ static int cxl_debugfs_poison_inject(void *data, u64 dpa)
 	struct cxl_memdev *cxlmd = data;
 	int rc;
 
-	ACQUIRE(device_intr, devlock)(&cxlmd->dev);
-	if ((rc = ACQUIRE_ERR(device_intr, &devlock)))
-		return rc;
+	/*
+	 * The debugfs proxy holds a debugfs_file_get() reference across this
+	 * callback, and cxl_mem unbind removes this file from devres while
+	 * holding the same device lock. Blocking here would let unbind wait
+	 * in debugfs_remove() for a reference that cannot be dropped until
+	 * unbind releases the lock, so never wait for the lock.
+	 */
+	if (!device_trylock(&cxlmd->dev))
+		return -EBUSY;
 
-	return cxl_inject_poison(cxlmd, dpa);
+	rc = cxl_inject_poison(cxlmd, dpa);
+	device_unlock(&cxlmd->dev);
+
+	return rc;
 }
 
 DEFINE_DEBUGFS_ATTRIBUTE(cxl_poison_inject_fops, NULL,
@@ -65,11 +74,14 @@ static int cxl_debugfs_poison_clear(void *data, u64 dpa)
 	struct cxl_memdev *cxlmd = data;
 	int rc;
 
-	ACQUIRE(device_intr, devlock)(&cxlmd->dev);
-	if ((rc = ACQUIRE_ERR(device_intr, &devlock)))
-		return rc;

drivers/cxl/mem.c static int cxl_debugfs_poison_clear(void *data, u64 dpa) · line 85

Quoted 10 lines
drivers/cxl/mem.c +7 -1
+	/* Avoid the unbind deadlock described in the inject path above. */
+	if (!device_trylock(&cxlmd->dev))
+		return -EBUSY;
+
+	rc = cxl_clear_poison(cxlmd, dpa);
+	device_unlock(&cxlmd->dev);
 
-	return cxl_clear_poison(cxlmd, dpa);
+	return rc;
 }

Hi Guixin

The analysis and the fix look correct to me. The cycle reduces to:

Poison writer: holds debugfs reference -> waits for device lock
Unbind: holds device lock -> waits for debugfs refs to drain

Since one leg is the debugfs active_users completion rather than a tracked lock, lockdep cannot see it and only trylock semantics can break it -- an interruptible lock would still sleep on the device lock while holding the debugfs reference, leaving the cycle intact.

Two suggestions:

1. This wants:

Fixes 574eda81d0a7 ("cxl/memdev: Hold memdev lock during memdev poison injection/clear")

since that commit introduced taking the memdev device lock in these debugfs entry points, which is what closed the cycle against debugfs_remove_recursive() on the unbind path.

2. That same commit used the ACQUIRE(device_intr, ...) conditional guard, so rather than open-coding device_trylock()/device_unlock(), consider keeping the existing style by adding the trylock sibling of commit bb729bf1d6fd ("driver core: Add conditional guard support for device_lock()"):

	--- a/include/linux/device.h
	+++ b/include/linux/device.h
	@@
	 DEFINE_GUARD(device, struct device *, device_lock(_T), device_unlock(_T))
	 DEFINE_GUARD_COND(device, _intr, device_lock_interruptible(_T), _RET == 0)
	+DEFINE_GUARD_COND(device, _try, device_trylock(_T))

and switching the lock class in the debugfs handlers:

	ACQUIRE(device_try, devlock)(&cxlmd->dev);
	if ((rc = ACQUIRE_ERR(device_try, &devlock)))
		return rc;

device_trylock() returns 1 on success like mutex_trylock(), so the default condition applies and ACQUIRE_ERR() yields -EBUSY on contention, matching the behavior described above. The device.h hunk would need a Cc to the driver core maintainers, same routing as bb729bf1d6fd.

Best regards,
Shaikh Kamaluddin

Quoted 5 lines
drivers/cxl/mem.c +0 -0
 DEFINE_DEBUGFS_ATTRIBUTE(cxl_poison_clear_fops, NULL,
Signature · 1 line
2.43.7
author
#2 lore raw
Quoted 111 lines · Shaikh Kamaluddin
Quoted 86 lines · Guixin Liu

Writing to the memdev poison debugfs files while cxl_mem is being unbound deadlocks. The unbind path takes the memdev device lock and then waits for the debugfs file to drain, while the writer holds the debugfs reference and waits for that same lock:

unbind_store() echo 0 > .../mem0/inject_poison

     device_release_driver_internal()   full_proxy_write()
       <takes cxlmd->dev lock>            <takes debugfs_file_get() ref>
       device_unbind_cleanup()            debugfs_attr_write()
         devres_release_all()               cxl_debugfs_poison_inject()
           remove_debugfs()                   <waits for cxlmd->dev lock>
             debugfs_remove_recursive()
               __debugfs_file_removed()
                 <waits for the ref to drain>

Neither side can make progress. A third task piles up behind the held device lock: detach_memdev() runs on cxl_bus_wq, and since that is an ordered workqueue the wedged work item stalls every other CXL bus work item, so unrelated devices stop being rescanned or detached.

The write side only needs the device lock so that cxl_dpa_to_region() sees a stable cxlmd->dev.driver. Nothing requires the caller to wait for the lock, so use device_trylock() and report -EBUSY instead. That keeps the debugfs reference from ever being held across a wait for the lock, which breaks the cycle. A concurrent unbind now either completes before the write starts, in which case the write fails with -ENOENT because the file is gone, or loses the race and the write reports -EBUSY.

The unbind task sits in uninterruptible sleep and cannot be killed. The writer can be signalled, which is the only reason the current code is recoverable at all.

Reproducer, on a device that advertises poison inject support:

while :; do echo 0 > /sys/kernel/debug/cxl/mem0/inject_poison; done & while :; do
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/unbind
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/bind
done

This hits within milliseconds. Note that lockdep stays quiet: one end of the cycle is the debugfs active_users completion rather than a lock it tracks. The hung task detector does report both blocked tasks.

Fixes 574eda81d0a7 ("cxl/memdev: Hold memdev lock during memdev poison injection/clear")
---
  drivers/cxl/mem.c | 28 ++++++++++++++++++++--------
  1 file changed, 20 insertions(+), 8 deletions(-)
drivers/cxl/mem.c +13 -7
index 798e5c369cfc..01d2d5855694 100644
@@ -50,11 +50,20 @@ static int cxl_debugfs_poison_inject(void *data, u64 dpa)
  	struct cxl_memdev *cxlmd = data;
  	int rc;
 
-	ACQUIRE(device_intr, devlock)(&cxlmd->dev);
-	if ((rc = ACQUIRE_ERR(device_intr, &devlock)))
-		return rc;
+	/*
+	 * The debugfs proxy holds a debugfs_file_get() reference across this
+	 * callback, and cxl_mem unbind removes this file from devres while
+	 * holding the same device lock. Blocking here would let unbind wait
+	 * in debugfs_remove() for a reference that cannot be dropped until
+	 * unbind releases the lock, so never wait for the lock.
+	 */
+	if (!device_trylock(&cxlmd->dev))
+		return -EBUSY;
 
-	return cxl_inject_poison(cxlmd, dpa);
+	rc = cxl_inject_poison(cxlmd, dpa);
+	device_unlock(&cxlmd->dev);
+
+	return rc;
  }
 
  DEFINE_DEBUGFS_ATTRIBUTE(cxl_poison_inject_fops, NULL,
@@ -65,11 +74,14 @@ static int cxl_debugfs_poison_clear(void *data, u64 dpa)
  	struct cxl_memdev *cxlmd = data;
  	int rc;
 
-	ACQUIRE(device_intr, devlock)(&cxlmd->dev);
-	if ((rc = ACQUIRE_ERR(device_intr, &devlock)))
-		return rc;

drivers/cxl/mem.c static int cxl_debugfs_poison_clear(void *data, u64 dpa) · line 85

Quoted 10 lines
drivers/cxl/mem.c +7 -1
+	/* Avoid the unbind deadlock described in the inject path above. */
+	if (!device_trylock(&cxlmd->dev))
+		return -EBUSY;
+
+	rc = cxl_clear_poison(cxlmd, dpa);
+	device_unlock(&cxlmd->dev);
 
-	return cxl_clear_poison(cxlmd, dpa);
+	return rc;
  }

Hi Guixin

The analysis and the fix look correct to me. The cycle reduces to:

Poison writer: holds debugfs reference -> waits for device lock
Unbind: holds device lock -> waits for debugfs refs to drain

Since one leg is the debugfs active_users completion rather than a tracked lock, lockdep cannot see it and only trylock semantics can break it -- an interruptible lock would still sleep on the device lock while holding the debugfs reference, leaving the cycle intact.

Two suggestions:

Quoted 7 lines

1. This wants:

Fixes 574eda81d0a7 ("cxl/memdev: Hold memdev lock during memdev poison injection/clear")

since that commit introduced taking the memdev device lock in these debugfs entry points, which is what closed the cycle against debugfs_remove_recursive() on the unbind path.

Well, this fix tag is already added in this version.

Quoted 19 lines

2. That same commit used the ACQUIRE(device_intr, ...) conditional guard, so rather than open-coding device_trylock()/device_unlock(), consider keeping the existing style by adding the trylock sibling of commit bb729bf1d6fd ("driver core: Add conditional guard support for device_lock()"):

--- a/include/linux/device.h
+++ b/include/linux/device.h
@@
DEFINE_GUARD(device, struct device *, device_lock(_T), device_unlock(_T)) DEFINE_GUARD_COND(device, _intr, device_lock_interruptible(_T), _RET == 0)
+DEFINE_GUARD_COND(device, _try, device_trylock(_T))

and switching the lock class in the debugfs handlers:

ACQUIRE(device_try, devlock)(&cxlmd->dev); if ((rc = ACQUIRE_ERR(device_try, &devlock)))
return rc;

Quoted 5 lines

device_trylock() returns 1 on success like mutex_trylock(), so the default condition applies and ACQUIRE_ERR() yields -EBUSY on contention, matching the behavior described above. The device.h hunk would need a Cc to the driver core maintainers, same routing as bb729bf1d6fd.

Sure, I will change to new style in v2, thanks.

Best Regards,
Guixin Liu

Quoted 9 lines

Best regards,
Shaikh Kamaluddin

Quoted 5 lines
drivers/cxl/mem.c +0 -0
  DEFINE_DEBUGFS_ATTRIBUTE(cxl_poison_clear_fops, NULL,
Signature · 1 line
2.43.7