nvme-loop: avoid cancelling/aborting I/O and admin tagset
Status from this message: Thanks, applied to nvme-7.1.
Get series with b4
b4 am [email protected]Versions newest first
Patches in v1 Applied
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/1 | nvme-loop: do not cancel I/O and admin tagset during ctrl reset/shutdowndrivers/ | +0 | 2 replies |
Reviews trailers per patch
The patch has a Reviewed-by or Acked-by.
| Patch | Christoph Hellwig |
|---|---|
| 1/1 | R-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [PATCH 0/1] nvme-loop: avoid cancelling/aborting I/O and admin tagset
Hi,
During nvme-loop controller reset or shutdown, the current code first cancels/aborts the I/O and admin tagsets and then proceeds to destroy the corresponding I/O and admin queues.
For the loop controller this cancellation is unnecessary. The queue destruction path already waits for all in-flight target I/O and admin operations to complete, which ensures that no outstanding operations remain before the queues are torn down.
Cancelling the tagsets first also introduces a small race window where a late completion from the target may arrive after the corresponding request tag has been cancelled but before the queues are destroyed. If this occurs, the completion path may attempt to access a request whose tag has already been cancelled or freed, which can lead to a kernel crash. So the patch in this patchset, avoids cancelling/aborting the I/O and admin tagsets for nvme-loop target, as this step is redundant and can expose the race described above.
This issue was observed while running blktests nvme/040. The kernel crash encountered is shown below:
run blktests nvme/040 at 2026-03-08 06:34:27
loop0: detected capacity change from 0 to 2097152
nvmet: adding nsid 1 to subsystem blktests-subsystem-1
nvmet: Created nvm controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349. nvme nvme6: creating 96 I/O queues.
nvme nvme6: new ctrl: "blktests-subsystem-1"nvme_log_error: 1 callbacks suppressed
block nvme6n1: no usable path - requeuing I/O
nvme6c6n1: Read(0x2) @ LBA 2096384, 128 blocks, Host Aborted Command (sct 0x3 / sc 0x71)blk_print_req_error: 1 callbacks suppressed
I/O error, dev nvme6c6n1, sector 2096384 op 0x0:(READ) flags 0x2880700 phys_seg 1 prio class 2
block nvme6n1: no usable path - requeuing I/O
Kernel attempted to read user page (286) - exploit attempt? (uid: 0)
BUG: Kernel NULL pointer dereference on read at 0x00000286 Faulting instruction address: 0xc00000000090ca18 Oops: Kernel access of bad area, sig: 11 [#1]
[...]
[...]
NIP [c000000000961274] blk_mq_complete_request_remote+0x28/0x2d4 LR [c008000009af1808] nvme_loop_queue_response+0x110/0x290 [nvme_loop]
Call Trace:
0xc00000000502c640 (unreliable)
nvme_loop_queue_response+0x104/0x290 [nvme_loop]
__nvmet_req_complete+0x80/0x498 [nvmet]
nvmet_req_complete+0x24/0xf8 [nvmet]
nvmet_bio_done+0x58/0xcc [nvmet]
bio_endio+0x250/0x390
blk_update_request+0x2e8/0x68c
blk_mq_end_request+0x30/0x5c
lo_complete_rq+0x94/0x110 [loop]
blk_complete_reqs+0x78/0x98
handle_softirqs+0x148/0x454
do_softirq_own_stack+0x3c/0x50
__irq_exit_rcu+0x18c/0x1b4
irq_exit+0x1c/0x34
do_IRQ+0x114/0x278
hardware_interrupt_common_virt+0x28c/0x290The above kernel oops occured in blk_mq_complete_request_remote():
1319 bool blk_mq_complete_request_remote(struct request *rq) 1320 {
1321 WRITE_ONCE(rq->state, MQ_RQ_COMPLETE);
1322
1323 /*
1324 * For request which hctx has only one ctx mapping, 1325 * or a polled request, always complete locally, 1326 * it's pointless to redirect the completion. 1327 */
1328 if ((rq->mq_hctx->nr_ctx == 1 &&
1329 rq->mq_ctx->cpu == raw_smp_processor_id()) ||
1330 rq->cmd_flags & REQ_POLLED)
1331 return false;
In the above code on line #1328, when kernel attempts to dereference rq->mq_hctx->nr_ctx it triggers carsh because rq->mq_hctx is NULL. This request has been already aborted/cancelled while loop controller reset is initiated.
Nilay Shroff (1):
nvme-loop: do not cancel I/O and admin tagset during ctrl
reset/shutdown
drivers/nvme/target/loop.c | 2 -- 1 file changed, 2 deletions(-)
Signature · 1 line
2.53.0