Please backport 49f06cff50a4 ("block: skip sync_blockdev() on surprise removal in bdev_mark_dead()") to 6.6.y, 6.12.y, 6.18.y
Hi stable team,
Please consider the following mainline commit for the stable trees:
commit 49f06cff50a4ccf3b7a1a662ceb892b3b21a527a
Author: Chao Shi <[email protected]>
"block: skip sync_blockdev() on surprise removal in bdev_mark_dead()"
Why it should be applied:
On surprise removal (@surprise == true) the device is already gone, but the bare block-device path in bdev_mark_dead() (no ->mark_dead holder op) calls sync_blockdev() unconditionally. It can then hang forever in folio_wait_writeback() waiting on writeback that can never complete. We hit this via nvme_reset_work()'s "I/O queues lost" path (nvme_mark_namespaces_dead -> blk_mark_disk_dead -> bdev_mark_dead(bdev, true)), which wedges the reset worker and every task serialized behind it -- an unrecoverable hung-task/DoS (multiple tasks blocked >120s, reproduced several times under fuzzing). The fix simply skips the futile sync on surprise removal, matching fs_bdev_mark_dead(); invalidate_bdev() still runs and orderly removal is unchanged.
Affected versions:
which first shipped in v6.6 (it dropped the pre-existing !surprise guard from the bare-bdev path). So the bug is present in v6.6 through the fix. v7.0+ already
carries the fix, and pre-6.6 trees still have the original guard, so this is only
needed for the 6.6.y, 6.12.y and 6.18.y stable trees.
The change is a self-contained one-line guard (plus a comment) in bdev_mark_dead()
and should cherry-pick cleanly onto all three; happy to send adjusted backports if any tree conflicts.
Thanks,
Chao Shi
Quoted 30 lines · Chao S
Hi stable team,
Please consider the following mainline commit for the stable trees:
commit
49f06cff50a4ccf3b7a1a662ceb892b3b21a527a
Author: Chao Shi <[email protected]>
"block: skipsync_blockdev()on surprise removal inbdev_mark_dead()"Why it should be applied:
On surprise removal (@surprise == true) the device is already gone, but the bare block-device path inbdev_mark_dead()(no ->mark_dead holder op) callssync_blockdev()unconditionally. It can then hang forever infolio_wait_writeback()waiting on writeback that can never complete. We hit this via nvme_reset_work()'s "I/O queues lost" path (nvme_mark_namespaces_dead->blk_mark_disk_dead->bdev_mark_dead(bdev, true)), which wedges the reset worker and every task serialized behind it -- an unrecoverable hung-task/DoS (multiple tasks blocked >120s, reproduced several times under fuzzing). The fix simply skips the futile sync on surprise removal, matchingfs_bdev_mark_dead();invalidate_bdev()still runs and orderly removal is unchanged.Affected versions:
Fixes d8530de5a6e8 ("block: call into the file system for bdev_mark_dead")which first shipped in v6.6 (it dropped the pre-existing !surprise guard from the bare-bdev path). So the bug is present in v6.6 through the fix. v7.0+ already
carries the fix, and pre-6.6 trees still have the original guard, so this is only
needed for the 6.6.y, 6.12.y and 6.18.y stable trees.
Quoted 4 lines
The change is a self-contained one-line guard (plus a comment) in
bdev_mark_dead()
and should cherry-pick cleanly onto all three; happy to send adjusted backports if any tree conflicts.
We need a 6.6.y backport as it does not apply cleanly there.
thanks,
greg k-h
Hi Greg,
Thanks for your email. Sure, working on it.
Best,
Chao Shi
Quoted 42 lines · Greg KH
Quoted 30 lines · Chao S
Hi stable team,
Please consider the following mainline commit for the stable trees:
commit
49f06cff50a4ccf3b7a1a662ceb892b3b21a527a
Author: Chao Shi <[email protected]>
"block: skipsync_blockdev()on surprise removal inbdev_mark_dead()"Why it should be applied:
On surprise removal (@surprise == true) the device is already gone, but the bare block-device path inbdev_mark_dead()(no ->mark_dead holder op) callssync_blockdev()unconditionally. It can then hang forever infolio_wait_writeback()waiting on writeback that can never complete. We hit this via nvme_reset_work()'s "I/O queues lost" path (nvme_mark_namespaces_dead->blk_mark_disk_dead->bdev_mark_dead(bdev, true)), which wedges the reset worker and every task serialized behind it -- an unrecoverable hung-task/DoS (multiple tasks blocked >120s, reproduced several times under fuzzing). The fix simply skips the futile sync on surprise removal, matchingfs_bdev_mark_dead();invalidate_bdev()still runs and orderly removal is unchanged.Affected versions:
Fixes d8530de5a6e8 ("block: call into the file system for bdev_mark_dead")which first shipped in v6.6 (it dropped the pre-existing !surprise guard from the bare-bdev path). So the bug is present in v6.6 through the fix. v7.0+ already
carries the fix, and pre-6.6 trees still have the original guard, so this is only
needed for the 6.6.y, 6.12.y and 6.18.y stable trees.Quoted 4 lines
The change is a self-contained one-line guard (plus a comment) in
bdev_mark_dead()
and should cherry-pick cleanly onto all three; happy to send adjusted backports if any tree conflicts.We need a 6.6.y backport as it does not apply cleanly there.
thanks,
greg k-h
PATCH 1/1 block: skip sync_blockdev() on surprise removal in bdev_mark_dead()
In reply to Greg KHcommit 49f06cff50a4ccf3b7a1a662ceb892b3b21a527a upstream.
bdev_mark_dead()'s @surprise == true means the device is already gone. The filesystem callback fs_bdev_mark_dead() honours this and skips sync_filesystem(), but the bare block device path (no ->mark_dead op) lost its !surprise guard when the holder ->mark_dead callback was wired up (see Fixes), and now calls sync_blockdev() unconditionally, which can hang forever waiting on writeback that can no longer complete.
syzkaller hit this via nvme_reset_work()'s "I/O queues lost" path:nvme_mark_namespaces_dead() -> blk_mark_disk_dead() -> bdev_mark_dead(bdev, true) -> sync_blockdev() blocks in folio_wait_writeback(), wedging the reset worker and every task waiting on it.
Skip the sync on surprise removal, matching fs_bdev_mark_dead(); invalidate_bdev() still runs. Orderly removal (surprise == false) is unchanged.
Found by FuzzNvme.
[ adjusted for 6.6.y: bdev_mark_dead() there still unlocks bd_holder_lock
after the if/else and the else arm is a bare sync_blockdev(), so the !surprise guard is added as "else if (!surprise)" instead of inside the mainline else block. ]
--- block/bdev.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
block/bdev.c +6 -1
@@ -961,7 +961,12 @@ void bdev_mark_dead(struct block_device *bdev, bool surprise)mutex_lock (&bdev->bd_holder_lock);if (bdev->bd_holder_ops && bdev->bd_holder_ops->mark_dead) bdev->bd_holder_ops->mark_dead (bdev, surprise); -else +/* +* On surprise removal the device is already gone; syncing is +* futile and can hang forever waiting on I/O that will never +* complete. Match fs_bdev_mark_dead(), which also skips it. +*/ +else if (!surprise)sync_blockdev (bdev);mutex_unlock (&bdev->bd_holder_lock);
Signature · 1 line
2.43.0
Re: [PATCH 6.6.y] block: skip sync_blockdev() on surprise removal in bdev_mark_dead()
Quoted 6 lines · Chao Shi
bdev_mark_dead()'s @surprise == true means the device is already gone. The filesystem callback
fs_bdev_mark_dead()honours this and skipssync_filesystem(), but the bare block device path (no ->mark_dead op) lost its !surprise guard when the holder ->mark_dead callback was wired up (see Fixes), and now callssync_blockdev()unconditionally, which can hang forever waiting on writeback that can no longer complete.
Queued for 6.6, thanks.
Signature · 2 lines
Thanks, Sasha
Re: [PATCH 6.6.y] block: skip sync_blockdev() on surprise removal in bdev_mark_dead()
Thank you!
Best,
Chao
Quoted 14 lines · Sasha Levin
Quoted 6 lines · Chao Shi
bdev_mark_dead()'s @surprise == true means the device is already gone. The filesystem callback
fs_bdev_mark_dead()honours this and skipssync_filesystem(), but the bare block device path (no ->mark_dead op) lost its !surprise guard when the holder ->mark_dead callback was wired up (see Fixes), and now callssync_blockdev()unconditionally, which can hang forever waiting on writeback that can no longer complete.Queued for 6.6, thanks.
--
Thanks,
Sasha