ublk: fix infinite loop in ublk server teardown
Subsystems (3)
v2 cover ublk: fix infinite loop in ublk server teardown
Fix an infinite loop in ublk server teardown which can arise when a ublk server dies while recovering a device. Also add a test which demonstrates the problem and verifies the fix.
---
Changes in v2:
- Add support for batched fetch in pre_fetch_io (Ming Lei)
- Link to v1: https://lore.kernel.org/r/[email protected]
---
Uday Shankar (2):
ublk: reset per-IO canceled flag on each fetch
selftests: ublk: test that teardown after incomplete recovery completes
drivers/block/ublk_drv.c | 21 ++++++---- tools/testing/selftests/ublk/Makefile | 1 + tools/testing/selftests/ublk/fault_inject.c | 52 +++++++++++++++++++++++-- tools/testing/selftests/ublk/kublk.c | 7 ++++ tools/testing/selftests/ublk/kublk.h | 3 ++ tools/testing/selftests/ublk/test_generic_17.sh | 35 +++++++++++++++++ 6 files changed, 108 insertions(+), 11 deletions(-)
---
base-commit: cdd71b7feb3674d14c5edd9c133242ddf9a363f2
change-id: 20260403-cancel-bb6a1292534c
Best regards,
Signature · 1 line
Uday Shankar <[email protected]>
PATCH v2 1/2 ublk: reset per-IO canceled flag on each fetch
If a ublk server starts recovering devices but dies before issuing fetch commands for all IOs, cancellation of the fetch commands that were successfully issued may never complete. This is because the per-IO canceled flag can remain set even after the fetch for that IO has been submitted - the per-IO canceled flags for all IOs in a queue are reset together only once all IOs for that queue have been fetched. So if a nonempty proper subset of the IOs for a queue are fetched when the ublk server dies, the IOs in that subset will never successfully be canceled, as their canceled flags remain set, and this prevents ublk_cancel_cmd from actually calling io_uring_cmd_done on the commands, despite the fact that they are outstanding.
Fix this by resetting the per-IO cancel flags immediately when each IO is fetched instead of waiting for all IOs for the queue (which may never happen).
--- drivers/block/ublk_drv.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)
drivers/block/ublk_drv.c +13 -8
@@ -2916,22 +2916,26 @@ static void ublk_stop_dev(struct ublk_device *ub)ublk_cancel_dev (ub); } +static void ublk_reset_io_flags (struct ublk_queue *ubq,struct ublk_io *io) +{ +/* UBLK_IO_FLAG_CANCELED can be cleared now */ +spin_lock (&ubq->cancel_lock); + io->flags &= ~UBLK_IO_FLAG_CANCELED ; +spin_unlock (&ubq->cancel_lock); +} +/* reset per-queue io flags */ static void ublk_queue_reset_io_flags (struct ublk_queue *ubq) { -int j; - -/* UBLK_IO_FLAG_CANCELED can be cleared now */ spin_lock (&ubq->cancel_lock); -for (j =0 ; j < ubq->q_depth; j++) - ubq->ios[j].flags &= ~UBLK_IO_FLAG_CANCELED ; ubq->canceling = false;spin_unlock (&ubq->cancel_lock); ubq->fail_io = false; }/* device can only be started after all IOs are ready */ -static void ublk_mark_io_ready (struct ublk_device *ub,u16 q_id) +static void ublk_mark_io_ready (struct ublk_device *ub,u16 q_id, +struct ublk_io *io) __must_hold(&ub->mutex) {struct ublk_queue *ubq =ublk_get_queue (ub, q_id);
@@ -2940,6 +2944,7 @@ static void ublk_mark_io_ready(struct ublk_device *ub, u16 q_id) ub->unprivileged_daemons = true; ubq->nr_io_ready++; +ublk_reset_io_flags (ubq, io);/* Check if this specific queue is now fully ready */ if (ublk_queue_ready (ubq)) {
@@ -3202,7 +3207,7 @@ static int ublk_fetch(struct io_uring_cmd *cmd, struct ublk_device *ub,if (!ret) ret =ublk_config_io_buf (ub, io, cmd, buf_addr,NULL );if (!ret) -ublk_mark_io_ready (ub, q_id); +ublk_mark_io_ready (ub, q_id, io);mutex_unlock (&ub->mutex);return ret; }
@@ -3610,7 +3615,7 @@ static int ublk_batch_prep_io(struct ublk_queue *ubq,ublk_io_unlock (io);if (!ret) -ublk_mark_io_ready (data->ub, ubq->q_id); +ublk_mark_io_ready (data->ub, ubq->q_id, io);return ret; }
Signature · 1 line
2.34.1
Re: [PATCH v2 1/2] ublk: reset per-IO canceled flag on each fetch
Quoted 30 lines · Uday Shankar
If a ublk server starts recovering devices but dies before issuing fetch commands for all IOs, cancellation of the fetch commands that were successfully issued may never complete. This is because the per-IO canceled flag can remain set even after the fetch for that IO has been submitted - the per-IO canceled flags for all IOs in a queue are reset together only once all IOs for that queue have been fetched. So if a nonempty proper subset of the IOs for a queue are fetched when the ublk server dies, the IOs in that subset will never successfully be canceled, as their canceled flags remain set, and this prevents
ublk_cancel_cmdfrom actually callingio_uring_cmd_doneon the commands, despite the fact that they are outstanding.Fix this by resetting the per-IO cancel flags immediately when each IO is fetched instead of waiting for all IOs for the queue (which may never happen).
Signed-off-by Uday Shankar <[email protected]>Fixes 728cbac5fe21 ("ublk: move device reset into ublk_ch_release()")Reviewed-by Ming Lei <[email protected]>Reviewed-by zhang, the-essence-of-life <[email protected]>--- drivers/block/ublk_drv.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)drivers/block/ublk_drv.c +0
-0@@ -2916,22 +2916,26 @@ static void ublk_stop_dev(struct ublk_device *ub)ublk_cancel_dev (ub);
drivers/block/ublk_drv.c static void ublk_stop_dev(struct ublk_device *ub) · line 2926
Quoted 10 lines
drivers/block/ublk_drv.c +8
-0} +static void ublk_reset_io_flags (struct ublk_queue *ubq,struct ublk_io *io) +{ +/* UBLK_IO_FLAG_CANCELED can be cleared now */ +spin_lock (&ubq->cancel_lock); + io->flags &= ~UBLK_IO_FLAG_CANCELED ; +spin_unlock (&ubq->cancel_lock); +} +
One wonders why we can't use 'set_bit' here, or, rather, convert 'flags' usage to set_bit().
The spinlock feels a bit silly as it's now per-io, and one would think that we don't have concurrent accesses to the same io...
drivers/block/ublk_drv.c static void ublk_stop_dev(struct ublk_device *ub) · line 2934
Quoted 13 lines
drivers/block/ublk_drv.c +0
-5/* reset per-queue io flags */ static void ublk_queue_reset_io_flags (struct ublk_queue *ubq) { -int j; - -/* UBLK_IO_FLAG_CANCELED can be cleared now */ spin_lock (&ubq->cancel_lock); -for (j =0 ; j < ubq->q_depth; j++) - ubq->ios[j].flags &= ~UBLK_IO_FLAG_CANCELED ; ubq->canceling = false;spin_unlock (&ubq->cancel_lock); ubq->fail_io = false; }
Similar here; as we don't loop anymore, why do we need the spinlock? Isn't WRITE_ONCE() sufficient here?
Quoted 27 lines
drivers/block/ublk_drv.c +4
-2/* device can only be started after all IOs are ready */ -static void ublk_mark_io_ready (struct ublk_device *ub,u16 q_id) +static void ublk_mark_io_ready (struct ublk_device *ub,u16 q_id, +struct ublk_io *io) __must_hold(&ub->mutex) {struct ublk_queue *ubq =ublk_get_queue (ub, q_id);@@ -2940,6 +2944,7 @@ static void ublk_mark_io_ready(struct ublk_device *ub, u16 q_id) ub->unprivileged_daemons = true; ubq->nr_io_ready++; +ublk_reset_io_flags (ubq, io);/* Check if this specific queue is now fully ready */ if (ublk_queue_ready (ubq)) {@@ -3202,7 +3207,7 @@ static int ublk_fetch(struct io_uring_cmd *cmd, struct ublk_device *ub,if (!ret) ret =ublk_config_io_buf (ub, io, cmd, buf_addr,NULL );if (!ret) -ublk_mark_io_ready (ub, q_id); +ublk_mark_io_ready (ub, q_id, io);mutex_unlock (&ub->mutex);return ret; }
drivers/block/ublk_drv.c static int ublk_batch_prep_io(struct ublk_queue *ubq, · line 3621
Quoted 8 lines
drivers/block/ublk_drv.c +1
-1@@ -3610,7 +3615,7 @@ static int ublk_batch_prep_io(struct ublk_queue *ubq,ublk_io_unlock (io);if (!ret) -ublk_mark_io_ready (data->ub, ubq->q_id); +ublk_mark_io_ready (data->ub, ubq->q_id, io);return ret; }
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [PATCH v2 1/2] ublk: reset per-IO canceled flag on each fetch
Quoted 34 lines · Hannes Reinecke
Quoted 33 lines · Uday Shankar
If a ublk server starts recovering devices but dies before issuing fetch commands for all IOs, cancellation of the fetch commands that were successfully issued may never complete. This is because the per-IO canceled flag can remain set even after the fetch for that IO has been submitted - the per-IO canceled flags for all IOs in a queue are reset together only once all IOs for that queue have been fetched. So if a nonempty proper subset of the IOs for a queue are fetched when the ublk server dies, the IOs in that subset will never successfully be canceled, as their canceled flags remain set, and this prevents
ublk_cancel_cmdfrom actually callingio_uring_cmd_doneon the commands, despite the fact that they are outstanding.Fix this by resetting the per-IO cancel flags immediately when each IO is fetched instead of waiting for all IOs for the queue (which may never happen).
Signed-off-by Uday Shankar <[email protected]>Fixes 728cbac5fe21 ("ublk: move device reset into ublk_ch_release()")Reviewed-by Ming Lei <[email protected]>Reviewed-by zhang, the-essence-of-life <[email protected]>--- drivers/block/ublk_drv.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)drivers/block/ublk_drv.c +2
-0@@ -2916,22 +2916,26 @@ static void ublk_stop_dev(struct ublk_device *ub)ublk_cancel_dev (ub); } +static void ublk_reset_io_flags (struct ublk_queue *ubq,struct ublk_io *io) +{
Quoted 8 lines
drivers/block/ublk_drv.c static void ublk_stop_dev(struct ublk_device *ub) · line 2926
Quoted 6 lines
drivers/block/ublk_drv.c +6
-0+/* UBLK_IO_FLAG_CANCELED can be cleared now */ +spin_lock (&ubq->cancel_lock); + io->flags &= ~UBLK_IO_FLAG_CANCELED ; +spin_unlock (&ubq->cancel_lock); +} +One wonders why we can't use '
set_bit' here, or, rather, convert 'flags' usage toset_bit().
It isn't necessary, because UBLK_F_PER_IO_DAEMON is enabled.
Quoted 2 lines
The spinlock feels a bit silly as it's now per-io, and one would think that we don't have concurrent accesses to the same io...
UBLK_IO_FLAG_CANCELED is only used in slow path, yes, it is supposed to be accessed concurrently.
It could be moved out of io->flags, but we do want to make struct ublk_io held in single cache line.
Quoted 16 lines
Quoted 4 lines
drivers/block/ublk_drv.c +0
-0/* reset per-queue io flags */ static void ublk_queue_reset_io_flags (struct ublk_queue *ubq) {drivers/block/ublk_drv.c static void ublk_stop_dev(struct ublk_device *ub) · line 2934
Quoted 10 lines
drivers/block/ublk_drv.c +0
-5-int j; - -/* UBLK_IO_FLAG_CANCELED can be cleared now */ spin_lock (&ubq->cancel_lock); -for (j =0 ; j < ubq->q_depth; j++) - ubq->ios[j].flags &= ~UBLK_IO_FLAG_CANCELED ; ubq->canceling = false;spin_unlock (&ubq->cancel_lock); ubq->fail_io = false; }Similar here; as we don't loop anymore, why do we need the spinlock? Isn't
WRITE_ONCE()sufficient here?
WRITE_ONCE() isn't enough, because we have to make sure that io->cmd is only completed once, please see ublk_cancel_cmd().
Anyway, all these comments should belong to improvement or new issue, not a blocker for current bug fix.
Thanks,
Ming
PATCH v2 2/2 selftests: ublk: test that teardown after incomplete recovery completes
In reply to Uday ShankarBefore the fix, teardown of a ublk server that was attempting to recover a device, but died when it had submitted a nonempty proper subset of the fetch commands to any queue would loop forever. Add a test to verify that, after the fix, teardown completes. This is done by:
- Adding a new argument to the fault_inject target that causes it die after fetching a nonempty proper subset of the IOs to a queue
- Using that argument in a new test while trying to recover an already-created device
- Attempting to delete the ublk device at the end of the test; this hangs forever if teardown from the fault-injected ublk server never completed.
It was manually verified that the test passes with the fix and hangs without it.
--- tools/testing/selftests/ublk/Makefile | 1 + tools/testing/selftests/ublk/fault_inject.c | 52 +++++++++++++++++++++++-- tools/testing/selftests/ublk/kublk.c | 7 ++++ tools/testing/selftests/ublk/kublk.h | 3 ++ tools/testing/selftests/ublk/test_generic_17.sh | 35 +++++++++++++++++ 5 files changed, 95 insertions(+), 3 deletions(-)
tools/testing/selftests/ublk/Makefile +1 -0
@@ -18,6 +18,7 @@ TEST_PROGS += test_generic_10.sh TEST_PROGS += test_generic_12.sh TEST_PROGS += test_generic_13.sh TEST_PROGS += test_generic_16.sh +TEST_PROGS += test_generic_17.sh TEST_PROGS += test_batch_01.sh TEST_PROGS += test_batch_02.sh
tools/testing/selftests/ublk/fault_inject.c +49 -3
106 lines not on this page · raw message
tools/testing/selftests/ublk/kublk.c +7 -0
@@ -796,6 +796,8 @@ static void ublk_submit_fetch_commands(struct ublk_thread *t) q = &t->dev->q[q_id]; io = &q->ios[tag]; io->buf_index = j++; +if (q->tgt_ops->pre_fetch_io) + q->tgt_ops->pre_fetch_io (t, q, tag, false);ublk_queue_io_cmd (t, io); } }else {
@@ -807,6 +809,8 @@ static void ublk_submit_fetch_commands(struct ublk_thread *t)for (i =0 ; i < q->q_depth; i++) { io = &q->ios[i]; io->buf_index = i; +if (q->tgt_ops->pre_fetch_io) + q->tgt_ops->pre_fetch_io (t, q, i, false);ublk_queue_io_cmd (t, io); } }
@@ -983,6 +987,9 @@ static void ublk_batch_setup_queues(struct ublk_thread *t)if (t->q_map[i] ==0 )continue ; +if (q->tgt_ops->pre_fetch_io) + q->tgt_ops->pre_fetch_io (t, q,0 , true); + ret =ublk_batch_queue_prep_io_cmds (t, q);ublk_assert (ret >=0 ); }
tools/testing/selftests/ublk/kublk.h +3 -0
@@ -60,6 +60,7 @@ struct stripe_ctx {struct fault_inject_ctx {/* fault_inject */ unsigned long delay_us; +bool die_during_fetch; };struct dev_ctx {
@@ -138,6 +139,8 @@ struct ublk_tgt_ops {int (*init_tgt)(const struct dev_ctx *ctx,struct ublk_dev *);void (*deinit_tgt)(struct ublk_dev *); +void (*pre_fetch_io)(struct ublk_thread *t,struct ublk_queue *q, +int tag,bool batch);int (*queue_io)(struct ublk_thread *,struct ublk_queue *,int tag);void (*tgt_io_done)(struct ublk_thread *,struct ublk_queue *,const struct io_uring_cqe *);
tools/testing/selftests/ublk/test_generic_17.sh new file +35 -0
@@ -0,0 +1,35 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +. "$(cd "$(dirname "$0")" && pwd)"/test_common.sh + +ERR_CODE=0 + +_prep_test "fault_inject" "teardown after incomplete recovery" + +# First start and stop a ublk server with device configured for recovery +dev_id=$(_add_ublk_dev -t fault_inject -r 1) +_check_add_dev $TID $? +state=$(__ublk_kill_daemon "${dev_id}" "QUIESCED") +if [ "$state" != "QUIESCED" ]; then + echo "device isn't quiesced($state) after $action" + ERR_CODE=255 +fi + +# Then recover the device, but use --die_during_fetch to have the ublk +# server die while a queue has some (but not all) I/Os fetched +${UBLK_PROG} recover -n "${dev_id}" --foreground -t fault_inject --die_during_fetch 1 +RECOVER_RES=$? +# 137 is the result when dying of SIGKILL +if (( RECOVER_RES != 137 )); then + echo "recover command exited with unexpected code ${RECOVER_RES}!" + ERR_CODE=255 +fi + +# Clean up the device. This can only succeed once teardown of the above +# exited ublk server completes. So if teardown never completes, we will +# time out here +_ublk_del_dev "${dev_id}" + +_cleanup_test "fault_inject" +_show_result $TID $ERR_CODE
Signature · 1 line
2.34.1
Re: [PATCH v2 2/2] selftests: ublk: test that teardown after incomplete recovery completes
Quoted 13 lines · Uday Shankar
Before the fix, teardown of a ublk server that was attempting to recover a device, but died when it had submitted a nonempty proper subset of the fetch commands to any queue would loop forever. Add a test to verify that, after the fix, teardown completes. This is done by:
- Adding a new argument to the
fault_injecttarget that causes it die after fetching a nonempty proper subset of the IOs to a queue
- Using that argument in a new test while trying to recover an already-created device
- Attempting to delete the ublk device at the end of the test; this hangs forever if teardown from the fault-injected ublk server never completed.
Quoted 4 lines
It was manually verified that the test passes with the fix and hangs without it.
Signed-off-by Uday Shankar <[email protected]>
Thanks,
Ming
Quoted 3 lines · Uday Shankar
Fix an infinite loop in ublk server teardown which can arise when a ublk server dies while recovering a device. Also add a test which demonstrates the problem and verifies the fix.
Applied, thanks!
[1/2] ublk: reset per-IO canceled flag on each fetch
commit: 0842186d2c4e67d2f8c8c2d1d779e8acffd41b5b
[2/2] selftests: ublk: test that teardown after incomplete recovery completes
commit: 320f9b1c6a942a73c26be56742ee1da04f893a4fBest regards,
Signature · 1 line
Jens Axboe