cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind
PATCH 1/1 cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind
Writing to the memdev poison debugfs files while cxl_mem is being unbound deadlocks. The unbind path takes the memdev device lock and then waits for the debugfs file to drain, while the writer holds the debugfs reference and waits for that same lock:
unbind_store() echo 0 > .../mem0/inject_poison
device_release_driver_internal() full_proxy_write()
<takes cxlmd->dev lock> <takes debugfs_file_get() ref>
device_unbind_cleanup() debugfs_attr_write()
devres_release_all() cxl_debugfs_poison_inject()
remove_debugfs() <waits for cxlmd->dev lock>
debugfs_remove_recursive()
__debugfs_file_removed()
<waits for the ref to drain>Neither side can make progress. A third task piles up behind the held device lock: detach_memdev() runs on cxl_bus_wq, and since that is an ordered workqueue the wedged work item stalls every other CXL bus work item, so unrelated devices stop being rescanned or detached.
The write side only needs the device lock so that cxl_dpa_to_region() sees a stable cxlmd->dev.driver. Nothing requires the caller to wait for the lock, so use device_trylock() and report -EBUSY instead. That keeps the debugfs reference from ever being held across a wait for the lock, which breaks the cycle. A concurrent unbind now either completes before the write starts, in which case the write fails with -ENOENT because the file is gone, or loses the race and the write reports -EBUSY.
The unbind task sits in uninterruptible sleep and cannot be killed. The writer can be signalled, which is the only reason the current code is recoverable at all.
Reproducer, on a device that advertises poison inject support:
while :; do echo 0 > /sys/kernel/debug/cxl/mem0/inject_poison; done & while :; do
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/unbind
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/bind
done
This hits within milliseconds. Note that lockdep stays quiet: one end of the cycle is the debugfs active_users completion rather than a lock it tracks. The hung task detector does report both blocked tasks.
--- drivers/cxl/mem.c | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-)
drivers/cxl/mem.c +20 -8
@@ -50,11 +50,20 @@ static int cxl_debugfs_poison_inject(void *data, u64 dpa)struct cxl_memdev *cxlmd = data;int rc; -ACQUIRE (device_intr, devlock)(&cxlmd->dev); -if ((rc =ACQUIRE_ERR (device_intr, &devlock))) -return rc; +/* +* The debugfs proxy holds a debugfs_file_get() reference across this +* callback, and cxl_mem unbind removes this file from devres while +* holding the same device lock. Blocking here would let unbind wait +* in debugfs_remove() for a reference that cannot be dropped until +* unbind releases the lock, so never wait for the lock. +*/ +if (!device_trylock (&cxlmd->dev)) +return -EBUSY ; -return cxl_inject_poison (cxlmd, dpa); + rc =cxl_inject_poison (cxlmd, dpa); +device_unlock (&cxlmd->dev); + +return rc; } DEFINE_DEBUGFS_ATTRIBUTE(cxl_poison_inject_fops, NULL,
@@ -65,11 +74,14 @@ static int cxl_debugfs_poison_clear(void *data, u64 dpa)struct cxl_memdev *cxlmd = data;int rc; -ACQUIRE (device_intr, devlock)(&cxlmd->dev); -if ((rc =ACQUIRE_ERR (device_intr, &devlock))) -return rc; +/* Avoid the unbind deadlock described in the inject path above. */ +if (!device_trylock (&cxlmd->dev)) +return -EBUSY ; + + rc =cxl_clear_poison (cxlmd, dpa); +device_unlock (&cxlmd->dev); -return cxl_clear_poison (cxlmd, dpa); +return rc; } DEFINE_DEBUGFS_ATTRIBUTE(cxl_poison_clear_fops, NULL,
Signature · 1 line
2.43.7
Quoted 86 lines · Guixin Liu
Writing to the memdev poison debugfs files while
cxl_memis being unbound deadlocks. The unbind path takes the memdev device lock and then waits for the debugfs file to drain, while the writer holds the debugfs reference and waits for that same lock:
unbind_store()echo 0 > .../mem0/inject_poisondevice_release_driver_internal() full_proxy_write() <takes cxlmd->dev lock> <takes debugfs_file_get() ref> device_unbind_cleanup() debugfs_attr_write() devres_release_all() cxl_debugfs_poison_inject() remove_debugfs() <waits for cxlmd->dev lock> debugfs_remove_recursive() __debugfs_file_removed() <waits for the ref to drain>Neither side can make progress. A third task piles up behind the held device lock:
detach_memdev()runs oncxl_bus_wq, and since that is an ordered workqueue the wedged work item stalls every other CXL bus work item, so unrelated devices stop being rescanned or detached.The write side only needs the device lock so that
cxl_dpa_to_region()sees a stablecxlmd->dev.driver. Nothing requires the caller to wait for the lock, so usedevice_trylock()and report-EBUSYinstead. That keeps the debugfs reference from ever being held across a wait for the lock, which breaks the cycle. A concurrent unbind now either completes before the write starts, in which case the write fails with-ENOENTbecause the file is gone, or loses the race and the write reports-EBUSY.The unbind task sits in uninterruptible sleep and cannot be killed. The writer can be signalled, which is the only reason the current code is recoverable at all.
Reproducer, on a device that advertises poison inject support:
while :; do echo 0 >
/sys/kernel/debug/cxl/mem0/inject_poison; done & while :; do
echo mem0 >/sys/bus/cxl/drivers/cxl_mem/unbind
echo mem0 >/sys/bus/cxl/drivers/cxl_mem/bind
doneThis hits within milliseconds. Note that lockdep stays quiet: one end of the cycle is the debugfs
active_userscompletion rather than a lock it tracks. The hung task detector does report both blocked tasks.Fixes 574eda81d0a7 ("cxl/memdev: Hold memdev lock during memdev poison injection/clear")Signed-off-by Guixin Liu <[email protected]>--- drivers/cxl/mem.c | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-)drivers/cxl/mem.c +13
-7@@ -50,11 +50,20 @@ static int cxl_debugfs_poison_inject(void *data, u64 dpa)struct cxl_memdev *cxlmd = data;int rc; -ACQUIRE (device_intr, devlock)(&cxlmd->dev); -if ((rc =ACQUIRE_ERR (device_intr, &devlock))) -return rc; +/* +* The debugfs proxy holds a debugfs_file_get() reference across this +* callback, and cxl_mem unbind removes this file from devres while +* holding the same device lock. Blocking here would let unbind wait +* in debugfs_remove() for a reference that cannot be dropped until +* unbind releases the lock, so never wait for the lock. +*/ +if (!device_trylock (&cxlmd->dev)) +return -EBUSY ; -return cxl_inject_poison (cxlmd, dpa); + rc =cxl_inject_poison (cxlmd, dpa); +device_unlock (&cxlmd->dev); + +return rc; } DEFINE_DEBUGFS_ATTRIBUTE(cxl_poison_inject_fops, NULL,@@ -65,11 +74,14 @@ static int cxl_debugfs_poison_clear(void *data, u64 dpa)struct cxl_memdev *cxlmd = data;int rc; -ACQUIRE (device_intr, devlock)(&cxlmd->dev); -if ((rc =ACQUIRE_ERR (device_intr, &devlock))) -return rc;
drivers/cxl/mem.c static int cxl_debugfs_poison_clear(void *data, u64 dpa) · line 85
Quoted 10 lines
drivers/cxl/mem.c +7
-1+/* Avoid the unbind deadlock described in the inject path above. */ +if (!device_trylock (&cxlmd->dev)) +return -EBUSY ; + + rc =cxl_clear_poison (cxlmd, dpa); +device_unlock (&cxlmd->dev); -return cxl_clear_poison (cxlmd, dpa); +return rc; }
Hi Guixin
The analysis and the fix look correct to me. The cycle reduces to:
Poison writer: holds debugfs reference -> waits for device lock
Unbind: holds device lock -> waits for debugfs refs to drain
Since one leg is the debugfs active_users completion rather than a tracked lock, lockdep cannot see it and only trylock semantics can break it -- an interruptible lock would still sleep on the device lock while holding the debugfs reference, leaving the cycle intact.
Two suggestions:
1. This wants:
since that commit introduced taking the memdev device lock in these debugfs entry points, which is what closed the cycle against debugfs_remove_recursive() on the unbind path.
2. That same commit used the ACQUIRE(device_intr, ...) conditional guard, so rather than open-coding device_trylock()/device_unlock(), consider keeping the existing style by adding the trylock sibling of commit bb729bf1d6fd ("driver core: Add conditional guard support for device_lock()"):
--- a/include/linux/device.h +++ b/include/linux/device.h @@ DEFINE_GUARD(device, struct device *, device_lock(_T), device_unlock(_T)) DEFINE_GUARD_COND(device, _intr, device_lock_interruptible(_T), _RET == 0) +DEFINE_GUARD_COND(device, _try, device_trylock(_T))
and switching the lock class in the debugfs handlers:
ACQUIRE(device_try, devlock)(&cxlmd->dev); if ((rc = ACQUIRE_ERR(device_try, &devlock))) return rc;
device_trylock() returns 1 on success like mutex_trylock(), so the default condition applies and ACQUIRE_ERR() yields -EBUSY on contention, matching the behavior described above. The device.h hunk would need a Cc to the driver core maintainers, same routing as bb729bf1d6fd.
Best regards,
Shaikh Kamaluddin
Quoted 5 lines
drivers/cxl/mem.c +0
-0DEFINE_DEBUGFS_ATTRIBUTE(cxl_poison_clear_fops, NULL,Signature · 1 line
2.43.7
Quoted 111 lines · Shaikh Kamaluddin
Quoted 86 lines · Guixin Liu
Writing to the memdev poison debugfs files while
cxl_memis being unbound deadlocks. The unbind path takes the memdev device lock and then waits for the debugfs file to drain, while the writer holds the debugfs reference and waits for that same lock:
unbind_store()echo 0 > .../mem0/inject_poisondevice_release_driver_internal() full_proxy_write() <takes cxlmd->dev lock> <takes debugfs_file_get() ref> device_unbind_cleanup() debugfs_attr_write() devres_release_all() cxl_debugfs_poison_inject() remove_debugfs() <waits for cxlmd->dev lock> debugfs_remove_recursive() __debugfs_file_removed() <waits for the ref to drain>Neither side can make progress. A third task piles up behind the held device lock:
detach_memdev()runs oncxl_bus_wq, and since that is an ordered workqueue the wedged work item stalls every other CXL bus work item, so unrelated devices stop being rescanned or detached.The write side only needs the device lock so that
cxl_dpa_to_region()sees a stablecxlmd->dev.driver. Nothing requires the caller to wait for the lock, so usedevice_trylock()and report-EBUSYinstead. That keeps the debugfs reference from ever being held across a wait for the lock, which breaks the cycle. A concurrent unbind now either completes before the write starts, in which case the write fails with-ENOENTbecause the file is gone, or loses the race and the write reports-EBUSY.The unbind task sits in uninterruptible sleep and cannot be killed. The writer can be signalled, which is the only reason the current code is recoverable at all.
Reproducer, on a device that advertises poison inject support:
while :; do echo 0 >
/sys/kernel/debug/cxl/mem0/inject_poison; done & while :; do
echo mem0 >/sys/bus/cxl/drivers/cxl_mem/unbind
echo mem0 >/sys/bus/cxl/drivers/cxl_mem/bind
doneThis hits within milliseconds. Note that lockdep stays quiet: one end of the cycle is the debugfs
active_userscompletion rather than a lock it tracks. The hung task detector does report both blocked tasks.Fixes 574eda81d0a7 ("cxl/memdev: Hold memdev lock during memdev poison injection/clear")Signed-off-by Guixin Liu <[email protected]>--- drivers/cxl/mem.c | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-)drivers/cxl/mem.c +13
-7@@ -50,11 +50,20 @@ static int cxl_debugfs_poison_inject(void *data, u64 dpa)struct cxl_memdev *cxlmd = data;int rc; -ACQUIRE (device_intr, devlock)(&cxlmd->dev); -if ((rc =ACQUIRE_ERR (device_intr, &devlock))) -return rc; +/* +* The debugfs proxy holds a debugfs_file_get() reference across this +* callback, and cxl_mem unbind removes this file from devres while +* holding the same device lock. Blocking here would let unbind wait +* in debugfs_remove() for a reference that cannot be dropped until +* unbind releases the lock, so never wait for the lock. +*/ +if (!device_trylock (&cxlmd->dev)) +return -EBUSY ; -return cxl_inject_poison (cxlmd, dpa); + rc =cxl_inject_poison (cxlmd, dpa); +device_unlock (&cxlmd->dev); + +return rc; } DEFINE_DEBUGFS_ATTRIBUTE(cxl_poison_inject_fops, NULL,@@ -65,11 +74,14 @@ static int cxl_debugfs_poison_clear(void *data, u64 dpa)struct cxl_memdev *cxlmd = data;int rc; -ACQUIRE (device_intr, devlock)(&cxlmd->dev); -if ((rc =ACQUIRE_ERR (device_intr, &devlock))) -return rc;drivers/cxl/mem.c static int cxl_debugfs_poison_clear(void *data, u64 dpa) · line 85
Quoted 10 lines
drivers/cxl/mem.c +7
-1+/* Avoid the unbind deadlock described in the inject path above. */ +if (!device_trylock (&cxlmd->dev)) +return -EBUSY ; + + rc =cxl_clear_poison (cxlmd, dpa); +device_unlock (&cxlmd->dev); -return cxl_clear_poison (cxlmd, dpa); +return rc; }Hi Guixin
The analysis and the fix look correct to me. The cycle reduces to:
Poison writer: holds debugfs reference -> waits for device lock
Unbind: holds device lock -> waits for debugfs refs to drainSince one leg is the debugfs
active_userscompletion rather than a tracked lock, lockdep cannot see it and only trylock semantics can break it -- an interruptible lock would still sleep on the device lock while holding the debugfs reference, leaving the cycle intact.Two suggestions:
Quoted 7 lines
1. This wants:
Fixes 574eda81d0a7 ("cxl/memdev: Hold memdev lock during memdev poison injection/clear")since that commit introduced taking the memdev device lock in these debugfs entry points, which is what closed the cycle against
debugfs_remove_recursive()on the unbind path.
Well, this fix tag is already added in this version.
Quoted 19 lines
2. That same commit used the
ACQUIRE(device_intr, ...)conditional guard, so rather than open-coding device_trylock()/device_unlock(), consider keeping the existing style by adding the trylock sibling of commitbb729bf1d6fd("driver core: Add conditional guard support fordevice_lock()"):---
a/include/linux/device.h
+++b/include/linux/device.h
@@
DEFINE_GUARD(device, struct device *, device_lock(_T), device_unlock(_T))DEFINE_GUARD_COND(device, _intr, device_lock_interruptible(_T), _RET == 0)
+DEFINE_GUARD_COND(device,_try,device_trylock(_T))and switching the lock class in the debugfs handlers:
ACQUIRE(device_try, devlock)(&cxlmd->dev); if ((rc =ACQUIRE_ERR(device_try, &devlock)))
return rc;
Quoted 5 lines
device_trylock()returns 1 on success likemutex_trylock(), so the default condition applies andACQUIRE_ERR()yields-EBUSYon contention, matching the behavior described above. Thedevice.hhunk would need a Cc to the driver core maintainers, same routing asbb729bf1d6fd.
Sure, I will change to new style in v2, thanks.
Best Regards,
Guixin Liu
Quoted 9 lines
Best regards,
Shaikh KamaluddinQuoted 5 lines
drivers/cxl/mem.c +0
-0DEFINE_DEBUGFS_ATTRIBUTE(cxl_poison_clear_fops, NULL,Signature · 1 line
2.43.7