cxl/memdev: Fix poison debugfs vs unbind deadlock
Status from this message: Applied to cxl/next:
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v4 Latest2 patches 4 repliesApplied
- v32 patches 13 repliesSuperseded
- v22 patches 3 repliesSuperseded
Earlier versions · 1
| Version | Posted | Patches | Replies | Status |
|---|---|---|---|---|
| v1 | 1 patch | 2 replies | Superseded |
Changes in v4 from the cover letter
- reword the patch 2 commit message per Alison: enumerate the teardown paths that race a poison write into an unbind, and state that mixing poison writes with cxl_mem teardown is not a supported use of this debug ABI, though the fix keeps the cost of doing so to a failed write
- add the pciehp hot-remove reproduction, reported during v3 review, to the patch 2 commit message
- collect Jonathan's Reviewed-by on both patches (no code change)
Patches in v4 Applied
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/2 | driver core: Add conditional guard support for device_trylock()include/ | +1 | 0 replies | |
| 2/2 | cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbinddrivers/ | +10 | 1 reply |
Reviews trailers per patch
All 2 patches have a Reviewed-by or Acked-by of their own; the series has one for all of them.
| Patch | Jonathan Cameron | Dave Jiang | Greg Kroah-Hartman | Alison Schofield |
|---|---|---|---|---|
| Series | R-b | A-b | ||
| 1/2 | R-b | |||
| 2/2 | R-b | R-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 4 reviewers.
Cover letter [PATCH v4 0/2] cxl/memdev: Fix poison debugfs vs unbind deadlock
Writing a memdev poison debugfs file while cxl_mem is being unbound deadlocks. Patch 2 fixes it by not waiting for the device lock in those handlers. Patch 1 adds the trylock guard it uses.
Patch 2 does not build without patch 1, so the two need to travel together.
Testing:
Reproduced on a QEMU CXL topology whose type3 devices advertise poison inject support:
while :; do echo 0 > /sys/kernel/debug/cxl/mem0/inject_poison; done & while :; do
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/unbind
echo mem0 > /sys/bus/cxl/drivers/cxl_mem/bind
done
Without the fix the unbind wedges within seconds. The three tasks involved, from /proc/<pid>/stack:
writer, state S, holds the debugfs reference and waits for the lock
cxl_debugfs_poison_inject+0x25/0xa0 [cxl_mem]
debugfs_attr_write+0x61/0xb0
full_proxy_write+0xfc/0x1c0
vfs_write+0x1d4/0xe60unbind, state D, holds the lock and waits for the reference to drain
remove_one+0x27f/0x3d0
debugfs_remove+0x44/0x60
release_nodes+0xfa/0x2c0
devres_release_all+0x113/0x1a0
device_unbind_cleanup+0x76/0x260
device_release_driver_internal+0x3eb/0x540
unbind_store+0xde/0x100 cxl_port workqueue, state D, blocked on the same lock
device_release_driver_internal+0x96/0x540
detach_memdev+0x79/0xb0 [cxl_core]
process_one_work+0x6b0/0xfb0The writer is in interruptible sleep and can be killed; the unbind cannot, and because cxl_bus_wq is an ordered workqueue the wedged detach_memdev() blocks every other CXL bus work item behind it.
The same deadlock shows up with a pciehp hot-remove racing the writer
loop: the pciehp thread hits the same debugfs_remove() drain holding the memdev lock, and the hot-remove wedges the same way. With both patches applied the hot-remove flow completes normally.
With both patches applied, 46 unbind/bind cycles against the same writer loop all completed, no task was left in D state, and the writer collected 10920 EBUSY returns from the contended trylock. Note that lockdep stays quiet either way: one leg of the cycle is the debugfs active_users completion rather than a lock it tracks.
v3 -> v4:
- reword the patch 2 commit message per Alison: enumerate the teardown paths that race a poison write into an unbind, and state that mixing poison writes with cxl_mem teardown is not a supported use of this debug ABI, though the fix keeps the cost of doing so to a failed write
- add the pciehp hot-remove reproduction, reported during v3 review, to the patch 2 commit message
- collect Jonathan's Reviewed-by on both patches (no code change)
v1 -> v2:
- add the device_trylock() guard and use ACQUIRE(device_try, ...) instead of open-coding device_trylock()/device_unlock(), keeping the style the
Fixes: commit established (Shaikh Kamaluddin)
- cut the changelog down to the failing condition, the consequence and the fix; the call graph and the reproducer live here instead
- say how the issue was found and how it was tested
v2 -> v3:
- rebase onto v7.3-rc2 (master), per Dave's request to send the series against Linus's tags rather than cxl/next
v1:
https://lore.kernel.org/linux-cxl/[email protected]/
v2:
https://lore.kernel.org/linux-cxl/[email protected]/
v3:
https://lore.kernel.org/linux-cxl/[email protected]/
Guixin Liu (2):
driver core: Add conditional guard support for device_trylock() cxl/memdev: Fix deadlock between poison debugfs and cxl_mem unbind
drivers/cxl/mem.c | 14 ++++++++++---- include/linux/device.h | 1 + 2 files changed, 11 insertions(+), 4 deletions(-)
Signature · 1 line
2.43.7