Skip to contentSkip to conversation

cxl/memdev: Fix poison debugfs vs unbind deadlock

PATCH Applied Guixin Liu 4 participants 7 messages Sep 16 – Sep 21

Subsystems (2)
lore
Share & tools
lore

Guixin Liu

author
#0 lore raw

v4 cover cxl/memdev: Fix poison debugfs vs unbind deadlock

Writing a memdev poison debugfs file while cxl_mem is being unbound deadlocks. Patch 2 fixes it by not waiting for the device lock in those handlers. Patch 1 adds the trylock guard it uses.

Patch 2 does not build without patch 1, so the two need to travel together.

Testing:

Reproduced on a QEMU CXL topology whose type3 devices advertise poison inject support:

while :; do echo 0 > /sys/kernel/debug/cxl/mem0/inject_poison; done & while :; do
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/unbind
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/bind
done

Without the fix the unbind wedges within seconds. The three tasks involved, from /proc/<pid>/stack:

writer, state S, holds the debugfs reference and waits for the lock

    cxl_debugfs_poison_inject+0x25/0xa0 [cxl_mem]
    debugfs_attr_write+0x61/0xb0
    full_proxy_write+0xfc/0x1c0
    vfs_write+0x1d4/0xe60

unbind, state D, holds the lock and waits for the reference to drain

    remove_one+0x27f/0x3d0
    debugfs_remove+0x44/0x60
    release_nodes+0xfa/0x2c0
    devres_release_all+0x113/0x1a0
    device_unbind_cleanup+0x76/0x260
    device_release_driver_internal+0x3eb/0x540
    unbind_store+0xde/0x100

cxl_port workqueue, state D, blocked on the same lock

    device_release_driver_internal+0x96/0x540
    detach_memdev+0x79/0xb0 [cxl_core]
    process_one_work+0x6b0/0xfb0

The writer is in interruptible sleep and can be killed; the unbind cannot, and because cxl_bus_wq is an ordered workqueue the wedged detach_memdev() blocks every other CXL bus work item behind it.

The same deadlock shows up with a pciehp hot-remove racing the writer
loop: the pciehp thread hits the same debugfs_remove() drain holding the memdev lock, and the hot-remove wedges the same way. With both patches applied the hot-remove flow completes normally.

With both patches applied, 46 unbind/bind cycles against the same writer loop all completed, no task was left in D state, and the writer collected 10920 EBUSY returns from the contended trylock. Note that lockdep stays quiet either way: one leg of the cycle is the debugfs active_users completion rather than a lock it tracks.

v3 -> v4:
- reword the patch 2 commit message per Alison: enumerate the teardown paths that race a poison write into an unbind, and state that mixing poison writes with cxl_mem teardown is not a supported use of this debug ABI, though the fix keeps the cost of doing so to a failed write
- add the pciehp hot-remove reproduction, reported during v3 review, to the patch 2 commit message
- collect Jonathan's Reviewed-by on both patches (no code change)

v1 -> v2:
- add the device_trylock() guard and use ACQUIRE(device_try, ...) instead of open-coding device_trylock()/device_unlock(), keeping the style the
Fixes: commit established (Shaikh Kamaluddin)
- cut the changelog down to the failing condition, the consequence and the fix; the call graph and the reproducer live here instead
- say how the issue was found and how it was tested

v2 -> v3:
- rebase onto v7.3-rc2 (master), per Dave's request to send the series against Linus's tags rather than cxl/next

v1:
https://lore.kernel.org/linux-cxl/[email protected]/

v2:
https://lore.kernel.org/linux-cxl/[email protected]/

v3:
https://lore.kernel.org/linux-cxl/[email protected]/

Guixin Liu (2):
driver core: Add conditional guard support for device_trylock() cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

 drivers/cxl/mem.c      | 14 ++++++++++----
 include/linux/device.h |  1 +
 2 files changed, 11 insertions(+), 4 deletions(-)
Signature · 1 line
2.43.7
author
#1 lore raw

PATCH v4 1/2 driver core: Add conditional guard support for device_trylock()

Introduce a conditional guard version of device_trylock() for scenarios that must not block on the device lock. device_trylock() returns 1 on success like mutex_trylock(), so the default binary condition applies and ACQUIRE_ERR() reports -EBUSY on contention.

---
 include/linux/device.h | 1 +
 1 file changed, 1 insertion(+)
include/linux/device.h +1 -0
index aee79fd6b32b..a8d6cf76d137 100644
@@ -1121,6 +1121,7 @@ static inline void device_unlock(struct device *dev)
 
 DEFINE_GUARD(device, struct device *, device_lock(_T), device_unlock(_T))
 DEFINE_GUARD_COND(device, _intr, device_lock_interruptible(_T), _RET == 0)
+DEFINE_GUARD_COND(device, _try, device_trylock(_T))
 
 static inline void device_lock_assert(struct device *dev)
 {
Signature · 1 line
2.43.7
author
#2 lore raw

PATCH v4 2/2 cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

In reply to Guixin Liu

The poison debugfs handlers take the memdev device lock so that the region lookup sees a stable cxlmd->dev.driver. debugfs holds a reference on the file across the handler, and cxl_mem unbind removes that file while holding the very same device lock, so a handler that waits for the lock deadlocks against a concurrent unbind.

The trigger is any teardown that detaches cxl_mem from the memdev while a poison write is in flight: unbinding or unloading cxl_mem or cxl_pci, or removing the endpoint PCI device, including hot-remove.

Both tasks then hang. The unbind side is uninterruptible, and it also blocks the memdev detach work, which runs on an ordered workqueue and so stalls every other CXL bus work item.

Take the lock with the trylock guard and return -EBUSY instead of waiting. An unbind that wins the race removes the file first and the write fails with -ENOENT.

The poison inject and clear files are a debug ABI for expert users, mostly device vendors, to test the poison capabilities of their devices. Mixing a poison write with cxl_mem teardown is not a supported use of the interface, but when it happens anyway the cost should be a failed write, not a hung kernel.

Found by code inspection. Reproduced by writing inject_poison in a loop while unbinding and rebinding cxl_mem, and confirmed fixed by the same test. Also reproduced the same deadlock with those writes racing a pciehp hot-remove of the memdev; with this patch the hot-remove flow completes normally.

Fixes 574eda81d0a7 ("cxl/memdev: Hold memdev lock during memdev poison injection/clear")
---
 drivers/cxl/mem.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
drivers/cxl/mem.c +10 -4
index 798e5c369cfc..3959ec963026 100644
@@ -50,8 +50,13 @@ static int cxl_debugfs_poison_inject(void *data, u64 dpa)
 	struct cxl_memdev *cxlmd = data;
 	int rc;
 
-	ACQUIRE(device_intr, devlock)(&cxlmd->dev);
-	if ((rc = ACQUIRE_ERR(device_intr, &devlock)))
+	/*
+	 * Never wait for this lock: the debugfs proxy holds a file reference
+	 * across the callback and unbind removes the file under the same
+	 * device lock, so waiting here deadlocks against unbind.
+	 */
+	ACQUIRE(device_try, devlock)(&cxlmd->dev);
+	if ((rc = ACQUIRE_ERR(device_try, &devlock)))
 		return rc;
 
 	return cxl_inject_poison(cxlmd, dpa);
@@ -65,8 +70,9 @@ static int cxl_debugfs_poison_clear(void *data, u64 dpa)
 	struct cxl_memdev *cxlmd = data;
 	int rc;
 
-	ACQUIRE(device_intr, devlock)(&cxlmd->dev);
-	if ((rc = ACQUIRE_ERR(device_intr, &devlock)))
+	/* Never wait, per the inject path above. */
+	ACQUIRE(device_try, devlock)(&cxlmd->dev);
+	if ((rc = ACQUIRE_ERR(device_try, &devlock)))
 		return rc;
 
 	return cxl_clear_poison(cxlmd, dpa);
Signature · 1 line
2.43.7
maintainer Reviewed-by on v4 2/2
#3 lore raw

Re: [PATCH v4 2/2] cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

Quoted 24 lines · Guixin Liu

The poison debugfs handlers take the memdev device lock so that the region lookup sees a stable cxlmd->dev.driver. debugfs holds a reference on the file across the handler, and cxl_mem unbind removes that file while holding the very same device lock, so a handler that waits for the lock deadlocks against a concurrent unbind.

The trigger is any teardown that detaches cxl_mem from the memdev while a poison write is in flight: unbinding or unloading cxl_mem or cxl_pci, or removing the endpoint PCI device, including hot-remove.

Both tasks then hang. The unbind side is uninterruptible, and it also blocks the memdev detach work, which runs on an ordered workqueue and so stalls every other CXL bus work item.

Take the lock with the trylock guard and return -EBUSY instead of waiting. An unbind that wins the race removes the file first and the write fails with -ENOENT.

The poison inject and clear files are a debug ABI for expert users, mostly device vendors, to test the poison capabilities of their devices. Mixing a poison write with cxl_mem teardown is not a supported use of the interface, but when it happens anyway the cost should be a failed write, not a hung kernel.

Quoted 5 lines

Found by code inspection. Reproduced by writing inject_poison in a loop while unbinding and rebinding cxl_mem, and confirmed fixed by the same test. Also reproduced the same deadlock with those writes racing a pciehp hot-remove of the memdev; with this patch the hot-remove flow completes normally.

maintainer Reviewed-by on the v4 series
#4 lore raw
In reply to Guixin Liu
Quoted 86 lines · Guixin Liu

Writing a memdev poison debugfs file while cxl_mem is being unbound deadlocks. Patch 2 fixes it by not waiting for the device lock in those handlers. Patch 1 adds the trylock guard it uses.

Patch 2 does not build without patch 1, so the two need to travel together.

Testing:

Reproduced on a QEMU CXL topology whose type3 devices advertise poison inject support:

while :; do echo 0 > /sys/kernel/debug/cxl/mem0/inject_poison; done & while :; do
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/unbind
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/bind
done

Without the fix the unbind wedges within seconds. The three tasks involved, from /proc/<pid>/stack:

writer, state S, holds the debugfs reference and waits for the lock

    cxl_debugfs_poison_inject+0x25/0xa0 [cxl_mem]
    debugfs_attr_write+0x61/0xb0
    full_proxy_write+0xfc/0x1c0
    vfs_write+0x1d4/0xe60

unbind, state D, holds the lock and waits for the reference to drain

    remove_one+0x27f/0x3d0
    debugfs_remove+0x44/0x60
    release_nodes+0xfa/0x2c0
    devres_release_all+0x113/0x1a0
    device_unbind_cleanup+0x76/0x260
    device_release_driver_internal+0x3eb/0x540
    unbind_store+0xde/0x100

cxl_port workqueue, state D, blocked on the same lock

    device_release_driver_internal+0x96/0x540
    detach_memdev+0x79/0xb0 [cxl_core]
    process_one_work+0x6b0/0xfb0

The writer is in interruptible sleep and can be killed; the unbind cannot, and because cxl_bus_wq is an ordered workqueue the wedged detach_memdev() blocks every other CXL bus work item behind it.

The same deadlock shows up with a pciehp hot-remove racing the writer
loop: the pciehp thread hits the same debugfs_remove() drain holding the memdev lock, and the hot-remove wedges the same way. With both patches applied the hot-remove flow completes normally.

With both patches applied, 46 unbind/bind cycles against the same writer loop all completed, no task was left in D state, and the writer collected 10920 EBUSY returns from the contended trylock. Note that lockdep stays quiet either way: one leg of the cycle is the debugfs active_users completion rather than a lock it tracks.

v3 -> v4:
- reword the patch 2 commit message per Alison: enumerate the teardown paths that race a poison write into an unbind, and state that mixing poison writes with cxl_mem teardown is not a supported use of this debug ABI, though the fix keeps the cost of doing so to a failed write
- add the pciehp hot-remove reproduction, reported during v3 review, to the patch 2 commit message
- collect Jonathan's Reviewed-by on both patches (no code change)

v1 -> v2:
- add the device_trylock() guard and use ACQUIRE(device_try, ...) instead of open-coding device_trylock()/device_unlock(), keeping the style the
Fixes: commit established (Shaikh Kamaluddin)
- cut the changelog down to the failing condition, the consequence and the fix; the call graph and the reproducer live here instead
- say how the issue was found and how it was tested

v2 -> v3:
- rebase onto v7.3-rc2 (master), per Dave's request to send the series against Linus's tags rather than cxl/next

v1:
https://lore.kernel.org/linux-cxl/[email protected]/

v2:
https://lore.kernel.org/linux-cxl/[email protected]/

v3:
https://lore.kernel.org/linux-cxl/[email protected]/

Quoted 8 lines

Guixin Liu (2):
driver core: Add conditional guard support for device_trylock() cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

 drivers/cxl/mem.c      | 14 ++++++++++----
 include/linux/device.h |  1 +
 2 files changed, 11 insertions(+), 4 deletions(-)

For the series

Greg,
I can take the patches through the CXL tree if you ack the first patch. Thanks!

maintainer Acked-by
#5 lore raw
Quoted 98 lines · Dave Jiang
Quoted 86 lines · Guixin Liu

Writing a memdev poison debugfs file while cxl_mem is being unbound deadlocks. Patch 2 fixes it by not waiting for the device lock in those handlers. Patch 1 adds the trylock guard it uses.

Patch 2 does not build without patch 1, so the two need to travel together.

Testing:

Reproduced on a QEMU CXL topology whose type3 devices advertise poison inject support:

while :; do echo 0 > /sys/kernel/debug/cxl/mem0/inject_poison; done & while :; do
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/unbind
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/bind
done

Without the fix the unbind wedges within seconds. The three tasks involved, from /proc/<pid>/stack:

writer, state S, holds the debugfs reference and waits for the lock

    cxl_debugfs_poison_inject+0x25/0xa0 [cxl_mem]
    debugfs_attr_write+0x61/0xb0
    full_proxy_write+0xfc/0x1c0
    vfs_write+0x1d4/0xe60

unbind, state D, holds the lock and waits for the reference to drain

    remove_one+0x27f/0x3d0
    debugfs_remove+0x44/0x60
    release_nodes+0xfa/0x2c0
    devres_release_all+0x113/0x1a0
    device_unbind_cleanup+0x76/0x260
    device_release_driver_internal+0x3eb/0x540
    unbind_store+0xde/0x100

cxl_port workqueue, state D, blocked on the same lock

    device_release_driver_internal+0x96/0x540
    detach_memdev+0x79/0xb0 [cxl_core]
    process_one_work+0x6b0/0xfb0

The writer is in interruptible sleep and can be killed; the unbind cannot, and because cxl_bus_wq is an ordered workqueue the wedged detach_memdev() blocks every other CXL bus work item behind it.

The same deadlock shows up with a pciehp hot-remove racing the writer
loop: the pciehp thread hits the same debugfs_remove() drain holding the memdev lock, and the hot-remove wedges the same way. With both patches applied the hot-remove flow completes normally.

With both patches applied, 46 unbind/bind cycles against the same writer loop all completed, no task was left in D state, and the writer collected 10920 EBUSY returns from the contended trylock. Note that lockdep stays quiet either way: one leg of the cycle is the debugfs active_users completion rather than a lock it tracks.

v3 -> v4:
- reword the patch 2 commit message per Alison: enumerate the teardown paths that race a poison write into an unbind, and state that mixing poison writes with cxl_mem teardown is not a supported use of this debug ABI, though the fix keeps the cost of doing so to a failed write
- add the pciehp hot-remove reproduction, reported during v3 review, to the patch 2 commit message
- collect Jonathan's Reviewed-by on both patches (no code change)

v1 -> v2:
- add the device_trylock() guard and use ACQUIRE(device_try, ...) instead of open-coding device_trylock()/device_unlock(), keeping the style the
Fixes: commit established (Shaikh Kamaluddin)
- cut the changelog down to the failing condition, the consequence and the fix; the call graph and the reproducer live here instead
- say how the issue was found and how it was tested

v2 -> v3:
- rebase onto v7.3-rc2 (master), per Dave's request to send the series against Linus's tags rather than cxl/next

v1:
https://lore.kernel.org/linux-cxl/[email protected]/

v2:
https://lore.kernel.org/linux-cxl/[email protected]/

v3:
https://lore.kernel.org/linux-cxl/[email protected]/

Quoted 8 lines

Guixin Liu (2):
driver core: Add conditional guard support for device_trylock() cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

 drivers/cxl/mem.c      | 14 ++++++++++----
 include/linux/device.h |  1 +
 2 files changed, 11 insertions(+), 4 deletions(-)
Quoted 5 lines

For the series

Greg,
I can take the patches through the CXL tree if you ack the first patch. Thanks!

Please do!

maintainer Applied
#6 lore raw
In reply to Guixin Liu
Quoted 86 lines · Guixin Liu

Writing a memdev poison debugfs file while cxl_mem is being unbound deadlocks. Patch 2 fixes it by not waiting for the device lock in those handlers. Patch 1 adds the trylock guard it uses.

Patch 2 does not build without patch 1, so the two need to travel together.

Testing:

Reproduced on a QEMU CXL topology whose type3 devices advertise poison inject support:

while :; do echo 0 > /sys/kernel/debug/cxl/mem0/inject_poison; done & while :; do
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/unbind
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/bind
done

Without the fix the unbind wedges within seconds. The three tasks involved, from /proc/<pid>/stack:

writer, state S, holds the debugfs reference and waits for the lock

    cxl_debugfs_poison_inject+0x25/0xa0 [cxl_mem]
    debugfs_attr_write+0x61/0xb0
    full_proxy_write+0xfc/0x1c0
    vfs_write+0x1d4/0xe60

unbind, state D, holds the lock and waits for the reference to drain

    remove_one+0x27f/0x3d0
    debugfs_remove+0x44/0x60
    release_nodes+0xfa/0x2c0
    devres_release_all+0x113/0x1a0
    device_unbind_cleanup+0x76/0x260
    device_release_driver_internal+0x3eb/0x540
    unbind_store+0xde/0x100

cxl_port workqueue, state D, blocked on the same lock

    device_release_driver_internal+0x96/0x540
    detach_memdev+0x79/0xb0 [cxl_core]
    process_one_work+0x6b0/0xfb0

The writer is in interruptible sleep and can be killed; the unbind cannot, and because cxl_bus_wq is an ordered workqueue the wedged detach_memdev() blocks every other CXL bus work item behind it.

The same deadlock shows up with a pciehp hot-remove racing the writer
loop: the pciehp thread hits the same debugfs_remove() drain holding the memdev lock, and the hot-remove wedges the same way. With both patches applied the hot-remove flow completes normally.

With both patches applied, 46 unbind/bind cycles against the same writer loop all completed, no task was left in D state, and the writer collected 10920 EBUSY returns from the contended trylock. Note that lockdep stays quiet either way: one leg of the cycle is the debugfs active_users completion rather than a lock it tracks.

v3 -> v4:
- reword the patch 2 commit message per Alison: enumerate the teardown paths that race a poison write into an unbind, and state that mixing poison writes with cxl_mem teardown is not a supported use of this debug ABI, though the fix keeps the cost of doing so to a failed write
- add the pciehp hot-remove reproduction, reported during v3 review, to the patch 2 commit message
- collect Jonathan's Reviewed-by on both patches (no code change)

v1 -> v2:
- add the device_trylock() guard and use ACQUIRE(device_try, ...) instead of open-coding device_trylock()/device_unlock(), keeping the style the
Fixes: commit established (Shaikh Kamaluddin)
- cut the changelog down to the failing condition, the consequence and the fix; the call graph and the reproducer live here instead
- say how the issue was found and how it was tested

v2 -> v3:
- rebase onto v7.3-rc2 (master), per Dave's request to send the series against Linus's tags rather than cxl/next

v1:
https://lore.kernel.org/linux-cxl/[email protected]/

v2:
https://lore.kernel.org/linux-cxl/[email protected]/

v3:
https://lore.kernel.org/linux-cxl/[email protected]/

Quoted 8 lines

Guixin Liu (2):
driver core: Add conditional guard support for device_trylock() cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

 drivers/cxl/mem.c      | 14 ++++++++++----
 include/linux/device.h |  1 +
 2 files changed, 11 insertions(+), 4 deletions(-)

Applied to cxl/next:
18f269585447
391bc7ab9be6