Skip to content

cxl/memdev: Fix poison debugfs vs unbind deadlock

Guixin Liu driver-core linux-cxl 4 versions – Applied

Status from this message: Applied to cxl/next:

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v4 Latest2 patches 4 repliesApplied
  2. v32 patches 13 repliesSuperseded
  3. v22 patches 3 repliesSuperseded
Earlier versions · 1
VersionPostedPatchesRepliesStatus
v11 patch2 repliesSuperseded

Changes in v4 from the cover letter

- reword the patch 2 commit message per Alison: enumerate the teardown paths that race a poison write into an unbind, and state that mixing poison writes with cxl_mem teardown is not a supported use of this debug ABI, though the fix keeps the cost of doing so to a failed write
- add the pciehp hot-remove reproduction, reported during v3 review, to the patch 2 commit message
- collect Jonathan's Reviewed-by on both patches (no code change)

Patches in v4 Applied

#Subject and filesSizeRepliesApplied
1/2driver core: Add conditional guard support for device_trylock()include/linux/device.h+1 -00 replies
2/2cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbinddrivers/cxl/mem.c+10 -41 reply

Reviews trailers per patch

All 2 patches have a Reviewed-by or Acked-by of their own; the series has one for all of them.

PatchJonathan CameronDave JiangGreg Kroah-HartmanAlison Schofield
SeriesR-b A-b
1/2R-b
2/2R-b R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 4 reviewers.

Read the v4 thread

Cover letter [PATCH v4 0/2] cxl/memdev: Fix poison debugfs vs unbind deadlock

Writing a memdev poison debugfs file while cxl_mem is being unbound deadlocks. Patch 2 fixes it by not waiting for the device lock in those handlers. Patch 1 adds the trylock guard it uses.

Patch 2 does not build without patch 1, so the two need to travel together.

Testing:

Reproduced on a QEMU CXL topology whose type3 devices advertise poison inject support:

while :; do echo 0 > /sys/kernel/debug/cxl/mem0/inject_poison; done & while :; do
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/unbind
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/bind
done

Without the fix the unbind wedges within seconds. The three tasks involved, from /proc/<pid>/stack:

writer, state S, holds the debugfs reference and waits for the lock

    cxl_debugfs_poison_inject+0x25/0xa0 [cxl_mem]
    debugfs_attr_write+0x61/0xb0
    full_proxy_write+0xfc/0x1c0
    vfs_write+0x1d4/0xe60

unbind, state D, holds the lock and waits for the reference to drain

    remove_one+0x27f/0x3d0
    debugfs_remove+0x44/0x60
    release_nodes+0xfa/0x2c0
    devres_release_all+0x113/0x1a0
    device_unbind_cleanup+0x76/0x260
    device_release_driver_internal+0x3eb/0x540
    unbind_store+0xde/0x100

cxl_port workqueue, state D, blocked on the same lock

    device_release_driver_internal+0x96/0x540
    detach_memdev+0x79/0xb0 [cxl_core]
    process_one_work+0x6b0/0xfb0

The writer is in interruptible sleep and can be killed; the unbind cannot, and because cxl_bus_wq is an ordered workqueue the wedged detach_memdev() blocks every other CXL bus work item behind it.

The same deadlock shows up with a pciehp hot-remove racing the writer
loop: the pciehp thread hits the same debugfs_remove() drain holding the memdev lock, and the hot-remove wedges the same way. With both patches applied the hot-remove flow completes normally.

With both patches applied, 46 unbind/bind cycles against the same writer loop all completed, no task was left in D state, and the writer collected 10920 EBUSY returns from the contended trylock. Note that lockdep stays quiet either way: one leg of the cycle is the debugfs active_users completion rather than a lock it tracks.

v3 -> v4:
- reword the patch 2 commit message per Alison: enumerate the teardown paths that race a poison write into an unbind, and state that mixing poison writes with cxl_mem teardown is not a supported use of this debug ABI, though the fix keeps the cost of doing so to a failed write
- add the pciehp hot-remove reproduction, reported during v3 review, to the patch 2 commit message
- collect Jonathan's Reviewed-by on both patches (no code change)

v1 -> v2:
- add the device_trylock() guard and use ACQUIRE(device_try, ...) instead of open-coding device_trylock()/device_unlock(), keeping the style the
Fixes: commit established (Shaikh Kamaluddin)
- cut the changelog down to the failing condition, the consequence and the fix; the call graph and the reproducer live here instead
- say how the issue was found and how it was tested

v2 -> v3:
- rebase onto v7.3-rc2 (master), per Dave's request to send the series against Linus's tags rather than cxl/next

v1:
https://lore.kernel.org/linux-cxl/[email protected]/

v2:
https://lore.kernel.org/linux-cxl/[email protected]/

v3:
https://lore.kernel.org/linux-cxl/[email protected]/

Guixin Liu (2):
driver core: Add conditional guard support for device_trylock() cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind

 drivers/cxl/mem.c      | 14 ++++++++++----
 include/linux/device.h |  1 +
 2 files changed, 11 insertions(+), 4 deletions(-)
Signature · 1 line
2.43.7