Skip to content

nvme-loop: avoid cancelling/aborting I/O and admin tagset

Nilay Shroff linux-nvme 1 version Applied

Status from this message: Thanks, applied to nvme-7.1.

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest1 patch 2 repliesApplied

Patches in v1 Applied

#Subject and filesSizeRepliesApplied
1/1nvme-loop: do not cancel I/O and admin tagset during ctrl reset/shutdowndrivers/nvme/target/loop.c+0 -22 replies

Reviews trailers per patch

The patch has a Reviewed-by or Acked-by.

PatchChristoph Hellwig
1/1R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH 0/1] nvme-loop: avoid cancelling/aborting I/O and admin tagset

Hi,

During nvme-loop controller reset or shutdown, the current code first cancels/aborts the I/O and admin tagsets and then proceeds to destroy the corresponding I/O and admin queues.

For the loop controller this cancellation is unnecessary. The queue destruction path already waits for all in-flight target I/O and admin operations to complete, which ensures that no outstanding operations remain before the queues are torn down.

Cancelling the tagsets first also introduces a small race window where a late completion from the target may arrive after the corresponding request tag has been cancelled but before the queues are destroyed. If this occurs, the completion path may attempt to access a request whose tag has already been cancelled or freed, which can lead to a kernel crash. So the patch in this patchset, avoids cancelling/aborting the I/O and admin tagsets for nvme-loop target, as this step is redundant and can expose the race described above.

This issue was observed while running blktests nvme/040. The kernel crash encountered is shown below:

run blktests nvme/040 at 2026-03-08 06:34:27
loop0: detected capacity change from 0 to 2097152
nvmet: adding nsid 1 to subsystem blktests-subsystem-1
nvmet: Created nvm controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349. nvme nvme6: creating 96 I/O queues.
nvme nvme6: new ctrl: "blktests-subsystem-1"
nvme_log_error: 1 callbacks suppressed
block nvme6n1: no usable path - requeuing I/O
nvme6c6n1: Read(0x2) @ LBA 2096384, 128 blocks, Host Aborted Command (sct 0x3 / sc 0x71)
blk_print_req_error: 1 callbacks suppressed
I/O error, dev nvme6c6n1, sector 2096384 op 0x0:(READ) flags 0x2880700 phys_seg 1 prio class 2
block nvme6n1: no usable path - requeuing I/O
Kernel attempted to read user page (286) - exploit attempt? (uid: 0)

BUG: Kernel NULL pointer dereference on read at 0x00000286
Faulting instruction address: 0xc00000000090ca18
Oops: Kernel access of bad area, sig: 11 [#1]

[...]
[...]
NIP [c000000000961274] blk_mq_complete_request_remote+0x28/0x2d4 LR [c008000009af1808] nvme_loop_queue_response+0x110/0x290 [nvme_loop]

    Call Trace:
     0xc00000000502c640 (unreliable)
     nvme_loop_queue_response+0x104/0x290 [nvme_loop]
     __nvmet_req_complete+0x80/0x498 [nvmet]
     nvmet_req_complete+0x24/0xf8 [nvmet]
     nvmet_bio_done+0x58/0xcc [nvmet]
     bio_endio+0x250/0x390
     blk_update_request+0x2e8/0x68c
     blk_mq_end_request+0x30/0x5c
     lo_complete_rq+0x94/0x110 [loop]
     blk_complete_reqs+0x78/0x98
     handle_softirqs+0x148/0x454
     do_softirq_own_stack+0x3c/0x50
     __irq_exit_rcu+0x18c/0x1b4
     irq_exit+0x1c/0x34
     do_IRQ+0x114/0x278
     hardware_interrupt_common_virt+0x28c/0x290

The above kernel oops occured in blk_mq_complete_request_remote():
1319 bool blk_mq_complete_request_remote(struct request *rq) 1320 {
1321 WRITE_ONCE(rq->state, MQ_RQ_COMPLETE);
1322
1323 /*

1324          * For request which hctx has only one ctx mapping,
1325          * or a polled request, always complete locally,
1326          * it's pointless to redirect the completion.
1327          */

1328 if ((rq->mq_hctx->nr_ctx == 1 &&
1329 rq->mq_ctx->cpu == raw_smp_processor_id()) ||
1330 rq->cmd_flags & REQ_POLLED)
1331 return false;

In the above code on line #1328, when kernel attempts to dereference rq->mq_hctx->nr_ctx it triggers carsh because rq->mq_hctx is NULL. This request has been already aborted/cancelled while loop controller reset is initiated.

Nilay Shroff (1):
nvme-loop: do not cancel I/O and admin tagset during ctrl
reset/shutdown

 drivers/nvme/target/loop.c | 2 --
 1 file changed, 2 deletions(-)
Signature · 1 line
2.53.0