Skip to contentSkip to conversation

Please backport 49f06cff50a4 ("block: skip sync_blockdev() on surprise removal in bdev_mark_dead()") to 6.6.y, 6.12.y, 6.18.y

PATCH Applied Chao S 3 participants 6 messages Jul 14 – Jul 18 Stable branch

lore
Share & tools
lore

Chao S

author
#0 lore raw

Hi stable team,

Please consider the following mainline commit for the stable trees:

commit 49f06cff50a4ccf3b7a1a662ceb892b3b21a527a
Author: Chao Shi <[email protected]>
"block: skip sync_blockdev() on surprise removal in bdev_mark_dead()"

Why it should be applied:
On surprise removal (@surprise == true) the device is already gone, but the bare block-device path in bdev_mark_dead() (no ->mark_dead holder op) calls sync_blockdev() unconditionally. It can then hang forever in folio_wait_writeback() waiting on writeback that can never complete. We hit this via nvme_reset_work()'s "I/O queues lost" path (nvme_mark_namespaces_dead -> blk_mark_disk_dead -> bdev_mark_dead(bdev, true)), which wedges the reset worker and every task serialized behind it -- an unrecoverable hung-task/DoS (multiple tasks blocked >120s, reproduced several times under fuzzing). The fix simply skips the futile sync on surprise removal, matching fs_bdev_mark_dead(); invalidate_bdev() still runs and orderly removal is unchanged.

Affected versions:

Fixes d8530de5a6e8 ("block: call into the file system for bdev_mark_dead")

which first shipped in v6.6 (it dropped the pre-existing !surprise guard from the bare-bdev path). So the bug is present in v6.6 through the fix. v7.0+ already
carries the fix, and pre-6.6 trees still have the original guard, so this is only
needed for the 6.6.y, 6.12.y and 6.18.y stable trees.

The change is a self-contained one-line guard (plus a comment) in bdev_mark_dead()
and should cherry-pick cleanly onto all three; happy to send adjusted backports if any tree conflicts.

Thanks,
Chao Shi

#1 lore raw
Quoted 30 lines · Chao S

Hi stable team,

Please consider the following mainline commit for the stable trees:

commit 49f06cff50a4ccf3b7a1a662ceb892b3b21a527a
Author: Chao Shi <[email protected]>
"block: skip sync_blockdev() on surprise removal in bdev_mark_dead()"

Why it should be applied:
On surprise removal (@surprise == true) the device is already gone, but the bare block-device path in bdev_mark_dead() (no ->mark_dead holder op) calls sync_blockdev() unconditionally. It can then hang forever in folio_wait_writeback() waiting on writeback that can never complete. We hit this via nvme_reset_work()'s "I/O queues lost" path (nvme_mark_namespaces_dead -> blk_mark_disk_dead -> bdev_mark_dead(bdev, true)), which wedges the reset worker and every task serialized behind it -- an unrecoverable hung-task/DoS (multiple tasks blocked >120s, reproduced several times under fuzzing). The fix simply skips the futile sync on surprise removal, matching fs_bdev_mark_dead(); invalidate_bdev() still runs and orderly removal is unchanged.

Affected versions:

Fixes d8530de5a6e8 ("block: call into the file system for bdev_mark_dead")

which first shipped in v6.6 (it dropped the pre-existing !surprise guard from the bare-bdev path). So the bug is present in v6.6 through the fix. v7.0+ already
carries the fix, and pre-6.6 trees still have the original guard, so this is only
needed for the 6.6.y, 6.12.y and 6.18.y stable trees.

Quoted 4 lines

The change is a self-contained one-line guard (plus a comment) in bdev_mark_dead()
and should cherry-pick cleanly onto all three; happy to send adjusted backports if any tree conflicts.

We need a 6.6.y backport as it does not apply cleanly there.

thanks,

greg k-h

author
#2 lore raw

Hi Greg,

Thanks for your email. Sure, working on it.

Best,
Chao Shi

Quoted 42 lines · Greg KH
Quoted 30 lines · Chao S

Hi stable team,

Please consider the following mainline commit for the stable trees:

commit 49f06cff50a4ccf3b7a1a662ceb892b3b21a527a
Author: Chao Shi <[email protected]>
"block: skip sync_blockdev() on surprise removal in bdev_mark_dead()"

Why it should be applied:
On surprise removal (@surprise == true) the device is already gone, but the bare block-device path in bdev_mark_dead() (no ->mark_dead holder op) calls sync_blockdev() unconditionally. It can then hang forever in folio_wait_writeback() waiting on writeback that can never complete. We hit this via nvme_reset_work()'s "I/O queues lost" path (nvme_mark_namespaces_dead -> blk_mark_disk_dead -> bdev_mark_dead(bdev, true)), which wedges the reset worker and every task serialized behind it -- an unrecoverable hung-task/DoS (multiple tasks blocked >120s, reproduced several times under fuzzing). The fix simply skips the futile sync on surprise removal, matching fs_bdev_mark_dead(); invalidate_bdev() still runs and orderly removal is unchanged.

Affected versions:

Fixes d8530de5a6e8 ("block: call into the file system for bdev_mark_dead")

which first shipped in v6.6 (it dropped the pre-existing !surprise guard from the bare-bdev path). So the bug is present in v6.6 through the fix. v7.0+ already
carries the fix, and pre-6.6 trees still have the original guard, so this is only
needed for the 6.6.y, 6.12.y and 6.18.y stable trees.

Quoted 4 lines

The change is a self-contained one-line guard (plus a comment) in bdev_mark_dead()
and should cherry-pick cleanly onto all three; happy to send adjusted backports if any tree conflicts.

We need a 6.6.y backport as it does not apply cleanly there.

thanks,

greg k-h

author
#3 lore raw

PATCH 1/1 block: skip sync_blockdev() on surprise removal in bdev_mark_dead()

In reply to Greg KH

commit 49f06cff50a4ccf3b7a1a662ceb892b3b21a527a upstream.

bdev_mark_dead()'s @surprise == true means the device is already gone. The filesystem callback fs_bdev_mark_dead() honours this and skips sync_filesystem(), but the bare block device path (no ->mark_dead op) lost its !surprise guard when the holder ->mark_dead callback was wired up (see Fixes), and now calls sync_blockdev() unconditionally, which can hang forever waiting on writeback that can no longer complete.

syzkaller hit this via nvme_reset_work()'s "I/O queues lost" path:
nvme_mark_namespaces_dead() -> blk_mark_disk_dead() -> bdev_mark_dead(bdev, true) -> sync_blockdev() blocks in folio_wait_writeback(), wedging the reset worker and every task waiting on it.

Skip the sync on surprise removal, matching fs_bdev_mark_dead(); invalidate_bdev() still runs. Orderly removal (surprise == false) is unchanged.

Found by FuzzNvme.

[ adjusted for 6.6.y: bdev_mark_dead() there still unlocks bd_holder_lock
after the if/else and the else arm is a bare sync_blockdev(), so the !surprise guard is added as "else if (!surprise)" instead of inside the mainline else block. ]

---
 block/bdev.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
block/bdev.c +6 -1
index f3b13aa1b7d4..2bbe0408dac2 100644
@@ -961,7 +961,12 @@ void bdev_mark_dead(struct block_device *bdev, bool surprise)
 	mutex_lock(&bdev->bd_holder_lock);
 	if (bdev->bd_holder_ops && bdev->bd_holder_ops->mark_dead)
 		bdev->bd_holder_ops->mark_dead(bdev, surprise);
-	else
+	/*
+	 * On surprise removal the device is already gone; syncing is
+	 * futile and can hang forever waiting on I/O that will never
+	 * complete.  Match fs_bdev_mark_dead(), which also skips it.
+	 */
+	else if (!surprise)
 		sync_blockdev(bdev);
 	mutex_unlock(&bdev->bd_holder_lock);
 
Signature · 1 line
2.43.0
maintainer Applied
#4 lore raw

Re: [PATCH 6.6.y] block: skip sync_blockdev() on surprise removal in bdev_mark_dead()

Quoted 6 lines · Chao Shi

bdev_mark_dead()'s @surprise == true means the device is already gone. The filesystem callback fs_bdev_mark_dead() honours this and skips sync_filesystem(), but the bare block device path (no ->mark_dead op) lost its !surprise guard when the holder ->mark_dead callback was wired up (see Fixes), and now calls sync_blockdev() unconditionally, which can hang forever waiting on writeback that can no longer complete.

Queued for 6.6, thanks.

Signature · 2 lines
Thanks,
Sasha
author
#5 lore raw

Re: [PATCH 6.6.y] block: skip sync_blockdev() on surprise removal in bdev_mark_dead()

Thank you!

Best,
Chao

Quoted 14 lines · Sasha Levin
Quoted 6 lines · Chao Shi

bdev_mark_dead()'s @surprise == true means the device is already gone. The filesystem callback fs_bdev_mark_dead() honours this and skips sync_filesystem(), but the bare block device path (no ->mark_dead op) lost its !surprise guard when the holder ->mark_dead callback was wired up (see Fixes), and now calls sync_blockdev() unconditionally, which can hang forever waiting on writeback that can no longer complete.

Queued for 6.6, thanks.

--
Thanks,
Sasha