cxl/memdev: Fix poison debugfs vs unbind deadlock
Subsystems (2)
v2 cover cxl/memdev: Fix poison debugfs vs unbind deadlock
Writing a memdev poison debugfs file while cxl_mem is being unbound deadlocks. Patch 2 fixes it by not waiting for the device lock in those handlers. Patch 1 adds the trylock guard it uses.
Patch 2 does not build without patch 1, so the two need to travel together.
Testing:
Reproduced on a QEMU CXL topology whose type3 devices advertise poison inject support:
while :; do echo 0 > /sys/kernel/debug/cxl/mem0/inject_poison; done & while :; do
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/unbind
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/bind
done
Without the fix the unbind wedges within seconds. The three tasks involved, from /proc/<pid>/stack:
writer, state S, holds the debugfs reference and waits for the lock
cxl_debugfs_poison_inject+0x25/0xa0 [cxl_mem]
debugfs_attr_write+0x61/0xb0
full_proxy_write+0xfc/0x1c0
vfs_write+0x1d4/0xe60unbind, state D, holds the lock and waits for the reference to drain
remove_one+0x27f/0x3d0
debugfs_remove+0x44/0x60
release_nodes+0xfa/0x2c0
devres_release_all+0x113/0x1a0
device_unbind_cleanup+0x76/0x260
device_release_driver_internal+0x3eb/0x540
unbind_store+0xde/0x100 cxl_port workqueue, state D, blocked on the same lock
device_release_driver_internal+0x96/0x540
detach_memdev+0x79/0xb0 [cxl_core]
process_one_work+0x6b0/0xfb0The writer is in interruptible sleep and can be killed; the unbind cannot, and because cxl_bus_wq is an ordered workqueue the wedged detach_memdev() blocks every other CXL bus work item behind it.
With both patches applied, 46 unbind/bind cycles against the same writer loop all completed, no task was left in D state, and the writer collected 10920 EBUSY returns from the contended trylock. Note that lockdep stays quiet either way: one leg of the cycle is the debugfs active_users completion rather than a lock it tracks.
v1 -> v2:
- add the device_trylock() guard and use ACQUIRE(device_try, ...) instead of open-coding device_trylock()/device_unlock(), keeping the style the
Fixes: commit established (Shaikh Kamaluddin)
- cut the changelog down to the failing condition, the consequence and the fix; the call graph and the reproducer live here instead
- say how the issue was found and how it was tested
v1:
https://lore.kernel.org/linux-cxl/\
[email protected]/
Guixin Liu (2):
driver core: Add conditional guard support for device_trylock() cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind
drivers/cxl/mem.c | 14 ++++++++++---- include/linux/device.h | 1 + 2 files changed, 11 insertions(+), 4 deletions(-)
base-commit: 7098e9cd98a05c0c5de2fae0c2465f9d966fdd07
Signature · 1 line
2.43.7
PATCH v2 1/2 driver core: Add conditional guard support for device_trylock()
Introduce a conditional guard version of device_trylock() for scenarios that must not block on the device lock. device_trylock() returns 1 on success like mutex_trylock(), so the default binary condition applies and ACQUIRE_ERR() reports -EBUSY on contention.
--- include/linux/device.h | 1 + 1 file changed, 1 insertion(+)
include/linux/device.h +1 -0
@@ -1119,6 +1119,7 @@ static inline void device_unlock(struct device *dev) DEFINE_GUARD(device, struct device *, device_lock(_T), device_unlock(_T)) DEFINE_GUARD_COND(device, _intr, device_lock_interruptible(_T), _RET == 0) +DEFINE_GUARD_COND(device, _try, device_trylock(_T))static inline void device_lock_assert (struct device *dev) {
base-commit: 7098e9cd98a05c0c5de2fae0c2465f9d966fdd07
Signature · 1 line
2.43.7
PATCH v2 2/2 cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind
In reply to Guixin LiuThe poison debugfs handlers take the memdev device lock so that the region lookup sees a stable cxlmd->dev.driver. debugfs holds a reference on the file across the handler, and cxl_mem unbind removes that file while holding the very same device lock, so a handler that waits for the lock deadlocks against a concurrent unbind.
Both tasks then hang. The unbind side is uninterruptible, and it also blocks the memdev detach work, which runs on an ordered workqueue and so stalls every other CXL bus work item.
Take the lock with the trylock guard and return -EBUSY instead of waiting. An unbind that wins the race removes the file first and the write fails with -ENOENT.
Found by code inspection. Reproduced by writing inject_poison in a loop while unbinding and rebinding cxl_mem, and confirmed fixed by the same test.
---
checkpatch reports "do not use assignment in if condition" twice, on the two ACQUIRE_ERR() lines. Those are pre-existing: the unpatched file and the Fixes: commit report the same two, this patch only swaps the lock class on them, and the combined form is what all 40 ACQUIRE_ERR() call sites in drivers/cxl use.
drivers/cxl/mem.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
drivers/cxl/mem.c +10 -4
@@ -50,8 +50,13 @@ static int cxl_debugfs_poison_inject(void *data, u64 dpa)struct cxl_memdev *cxlmd = data;int rc; -ACQUIRE (device_intr, devlock)(&cxlmd->dev); -if ((rc =ACQUIRE_ERR (device_intr, &devlock))) +/* +* Never wait for this lock: the debugfs proxy holds a file reference +* across the callback and unbind removes the file under the same +* device lock, so waiting here deadlocks against unbind. +*/ +ACQUIRE (device_try, devlock)(&cxlmd->dev); +if ((rc =ACQUIRE_ERR (device_try, &devlock)))return rc;return cxl_inject_poison (cxlmd, dpa);
@@ -65,8 +70,9 @@ static int cxl_debugfs_poison_clear(void *data, u64 dpa)struct cxl_memdev *cxlmd = data;int rc; -ACQUIRE (device_intr, devlock)(&cxlmd->dev); -if ((rc =ACQUIRE_ERR (device_intr, &devlock))) +/* Never wait, per the inject path above. */ +ACQUIRE (device_try, devlock)(&cxlmd->dev); +if ((rc =ACQUIRE_ERR (device_try, &devlock)))return rc;return cxl_clear_poison (cxlmd, dpa);
Signature · 1 line
2.43.7
Hi, gently ping on this patchset, Any feedback would be appreciated. thanks.
Best Regards,
Guixin Liu
Quoted 73 lines · Guixin Liu
Writing a memdev poison debugfs file while
cxl_memis being unbound deadlocks. Patch 2 fixes it by not waiting for the device lock in those handlers. Patch 1 adds the trylock guard it uses.Patch 2 does not build without patch 1, so the two need to travel together.
Testing:
Reproduced on a QEMU CXL topology whose type3 devices advertise poison inject support:
while :; do echo 0 >
/sys/kernel/debug/cxl/mem0/inject_poison; done & while :; do
echo mem0 >/sys/bus/cxl/drivers/cxl_mem/unbind
echo mem0 >/sys/bus/cxl/drivers/cxl_mem/bind
doneWithout the fix the unbind wedges within seconds. The three tasks involved, from /proc/<pid>/stack:
writer, state S, holds the debugfs reference and waits for the lock cxl_debugfs_poison_inject+0x25/0xa0 [cxl_mem] debugfs_attr_write+0x61/0xb0 full_proxy_write+0xfc/0x1c0 vfs_write+0x1d4/0xe60 unbind, state D, holds the lock and waits for the reference to drain remove_one+0x27f/0x3d0 debugfs_remove+0x44/0x60 release_nodes+0xfa/0x2c0 devres_release_all+0x113/0x1a0 device_unbind_cleanup+0x76/0x260 device_release_driver_internal+0x3eb/0x540 unbind_store+0xde/0x100 cxl_port workqueue, state D, blocked on the same lock device_release_driver_internal+0x96/0x540 detach_memdev+0x79/0xb0 [cxl_core] process_one_work+0x6b0/0xfb0The writer is in interruptible sleep and can be killed; the unbind cannot, and because
cxl_bus_wqis an ordered workqueue the wedgeddetach_memdev()blocks every other CXL bus work item behind it.With both patches applied, 46 unbind/bind cycles against the same writer loop all completed, no task was left in D state, and the writer collected 10920 EBUSY returns from the contended trylock. Note that lockdep stays quiet either way: one leg of the cycle is the debugfs
active_userscompletion rather than a lock it tracks.v1 -> v2:
- add thedevice_trylock()guard and useACQUIRE(device_try, ...)instead of open-coding device_trylock()/device_unlock(), keeping the style the
Fixes: commit established (Shaikh Kamaluddin)
- cut the changelog down to the failing condition, the consequence and the fix; the call graph and the reproducer live here instead
- say how the issue was found and how it was testedv1:
https://lore.kernel.org/linux-cxl/\
[email protected]/Guixin Liu (2):
driver core: Add conditional guard support fordevice_trylock()cxl/memdev: Fix deadlock between poison debugfs andcxl_memunbinddrivers/cxl/mem.c | 14 ++++++++++---- include/linux/device.h | 1 + 2 files changed, 11 insertions(+), 4 deletions(-)base-commit:
7098e9cd98a05c0c5de2fae0c2465f9d966fdd07
Dave Jiang
maintainerQuoted 67 lines · Guixin Liu
Writing a memdev poison debugfs file while
cxl_memis being unbound deadlocks. Patch 2 fixes it by not waiting for the device lock in those handlers. Patch 1 adds the trylock guard it uses.Patch 2 does not build without patch 1, so the two need to travel together.
Testing:
Reproduced on a QEMU CXL topology whose type3 devices advertise poison inject support:
while :; do echo 0 >
/sys/kernel/debug/cxl/mem0/inject_poison; done & while :; do
echo mem0 >/sys/bus/cxl/drivers/cxl_mem/unbind
echo mem0 >/sys/bus/cxl/drivers/cxl_mem/bind
doneWithout the fix the unbind wedges within seconds. The three tasks involved, from /proc/<pid>/stack:
writer, state S, holds the debugfs reference and waits for the lock
cxl_debugfs_poison_inject+0x25/0xa0 [cxl_mem] debugfs_attr_write+0x61/0xb0 full_proxy_write+0xfc/0x1c0 vfs_write+0x1d4/0xe60unbind, state D, holds the lock and waits for the reference to drain
remove_one+0x27f/0x3d0 debugfs_remove+0x44/0x60 release_nodes+0xfa/0x2c0 devres_release_all+0x113/0x1a0 device_unbind_cleanup+0x76/0x260 device_release_driver_internal+0x3eb/0x540 unbind_store+0xde/0x100
cxl_portworkqueue, state D, blocked on the same lockdevice_release_driver_internal+0x96/0x540 detach_memdev+0x79/0xb0 [cxl_core] process_one_work+0x6b0/0xfb0The writer is in interruptible sleep and can be killed; the unbind cannot, and because
cxl_bus_wqis an ordered workqueue the wedgeddetach_memdev()blocks every other CXL bus work item behind it.With both patches applied, 46 unbind/bind cycles against the same writer loop all completed, no task was left in D state, and the writer collected 10920 EBUSY returns from the contended trylock. Note that lockdep stays quiet either way: one leg of the cycle is the debugfs
active_userscompletion rather than a lock it tracks.v1 -> v2:
- add thedevice_trylock()guard and useACQUIRE(device_try, ...)instead of open-coding device_trylock()/device_unlock(), keeping the style the
Fixes: commit established (Shaikh Kamaluddin)
- cut the changelog down to the failing condition, the consequence and the fix; the call graph and the reproducer live here instead
- say how the issue was found and how it was testedv1:
https://lore.kernel.org/linux-cxl/\
[email protected]/Guixin Liu (2):
driver core: Add conditional guard support fordevice_trylock()cxl/memdev: Fix deadlock between poison debugfs andcxl_memunbind
Quoted 6 lines
drivers/cxl/mem.c | 14 ++++++++++---- include/linux/device.h | 1 + 2 files changed, 11 insertions(+), 4 deletions(-)base-commit:
7098e9cd98a05c0c5de2fae0c2465f9d966fdd07
Please rebase the patch series on Linus tags rather than cxl/next. Currently v7.3-rc2. It would be nice to get sashiko coverage.
Quoted 69 lines · Dave Jiang
Quoted 67 lines · Guixin Liu
Writing a memdev poison debugfs file while
cxl_memis being unbound deadlocks. Patch 2 fixes it by not waiting for the device lock in those handlers. Patch 1 adds the trylock guard it uses.Patch 2 does not build without patch 1, so the two need to travel together.
Testing:
Reproduced on a QEMU CXL topology whose type3 devices advertise poison inject support:
while :; do echo 0 >
/sys/kernel/debug/cxl/mem0/inject_poison; done & while :; do
echo mem0 >/sys/bus/cxl/drivers/cxl_mem/unbind
echo mem0 >/sys/bus/cxl/drivers/cxl_mem/bind
doneWithout the fix the unbind wedges within seconds. The three tasks involved, from /proc/<pid>/stack:
writer, state S, holds the debugfs reference and waits for the lock cxl_debugfs_poison_inject+0x25/0xa0 [cxl_mem] debugfs_attr_write+0x61/0xb0 full_proxy_write+0xfc/0x1c0 vfs_write+0x1d4/0xe60 unbind, state D, holds the lock and waits for the reference to drain remove_one+0x27f/0x3d0 debugfs_remove+0x44/0x60 release_nodes+0xfa/0x2c0 devres_release_all+0x113/0x1a0 device_unbind_cleanup+0x76/0x260 device_release_driver_internal+0x3eb/0x540 unbind_store+0xde/0x100 cxl_port workqueue, state D, blocked on the same lock device_release_driver_internal+0x96/0x540 detach_memdev+0x79/0xb0 [cxl_core] process_one_work+0x6b0/0xfb0The writer is in interruptible sleep and can be killed; the unbind cannot, and because
cxl_bus_wqis an ordered workqueue the wedgeddetach_memdev()blocks every other CXL bus work item behind it.With both patches applied, 46 unbind/bind cycles against the same writer loop all completed, no task was left in D state, and the writer collected 10920 EBUSY returns from the contended trylock. Note that lockdep stays quiet either way: one leg of the cycle is the debugfs
active_userscompletion rather than a lock it tracks.v1 -> v2:
- add thedevice_trylock()guard and useACQUIRE(device_try, ...)instead of open-coding device_trylock()/device_unlock(), keeping the style the
Fixes: commit established (Shaikh Kamaluddin)
- cut the changelog down to the failing condition, the consequence and the fix; the call graph and the reproducer live here instead
- say how the issue was found and how it was testedv1:
https://lore.kernel.org/linux-cxl/\
[email protected]/Guixin Liu (2):
driver core: Add conditional guard support fordevice_trylock()cxl/memdev: Fix deadlock between poison debugfs andcxl_memunbind
Quoted 7 lines
Quoted 6 lines
drivers/cxl/mem.c | 14 ++++++++++---- include/linux/device.h | 1 + 2 files changed, 11 insertions(+), 4 deletions(-)base-commit:
7098e9cd98a05c0c5de2fae0c2465f9d966fdd07Please rebase the patch series on Linus tags rather than cxl/next. Currently v7.3-rc2. It would be nice to get sashiko coverage.
OK, changed in v3.
Best Regards,
Guixin Liu