Skip to contentSkip to conversation

nvme: avoid missing db ring during reset

PATCH In review Linjun Bao 2 participants 3 messages Apr 25, 2025 NVMe

lore
Share & tools
lore

Linjun Bao

author
#0 lore raw

PATCH 1/1 nvme: avoid missing db ring during reset

During nvme reset, there is a rare case, when user admin cmd such as smart-log and nvme_admin_create_sq from nvme_setup_io_queues happen to in the same blk_mq dispatch list, and the user cmd is the last one. nvme_admin_create_sq is dispatched first in nvme_queue_rq(), nvme_write_sq_db() is called but immediately returns without writing the doorbell because it's not masked "last". The subsequent smart-log ioctl fails fast hitting nvme_fail_nonready_cmd(), skipping both nvme_sq_copy_cmd() and nvme_write_sq_db(), so no doorbell write ever occurs. The nvme_admin_create_sq fails timeout finally.

The proposal is that do not treat user admin cmd during RECONNECTING as non-ready, through it to the drive, thus no doorbell missing happens in case above.

---
 drivers/nvme/host/core.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +2 -1
index 3cc79817e4d7..fc550226ed77 100644
@@ -750,7 +750,8 @@ bool __nvme_check_ready(struct nvme_ctrl *ctrl, struct request *rq,
 	 * sequence. until the controller will be LIVE, fail with
 	 * BLK_STS_RESOURCE so that they will be rescheduled.
 	 */
-	if (rq->q == ctrl->admin_q && (req->flags & NVME_REQ_USERCMD))
+	if (rq->q == ctrl->admin_q && (req->flags & NVME_REQ_USERCMD) &&
+	    (nvme_ctrl_state(ctrl) != NVME_CTRL_CONNECTING))
 		return false;
 
 	if (ctrl->ops->flags & NVME_F_FABRICS) {
Signature · 1 line
2.25.1
maintainer
#1 lore raw
Quoted 10 lines · Linjun Bao

During nvme reset, there is a rare case, when user admin cmd such as smart-log and nvme_admin_create_sq from nvme_setup_io_queues happen to in the same blk_mq dispatch list, and the user cmd is the last one. nvme_admin_create_sq is dispatched first in nvme_queue_rq(), nvme_write_sq_db() is called but immediately returns without writing the doorbell because it's not masked "last". The subsequent smart-log ioctl fails fast hitting nvme_fail_nonready_cmd(), skipping both nvme_sq_copy_cmd() and nvme_write_sq_db(), so no doorbell write ever occurs. The nvme_admin_create_sq fails timeout finally.

The block layer is supposed to call the driver's commit_rqs() function if anything in the dispatch list wasn't successful, which should notify the controller of any pending SQEs. Is that not happening here?

author
#2 lore raw
Quoted 12 lines · Keith Busch
Quoted 10 lines · Linjun Bao

During nvme reset, there is a rare case, when user admin cmd such as smart-log and nvme_admin_create_sq from nvme_setup_io_queues happen to in the same blk_mq dispatch list, and the user cmd is the last one. nvme_admin_create_sq is dispatched first in nvme_queue_rq(), nvme_write_sq_db() is called but immediately returns without writing the doorbell because it's not masked "last". The subsequent smart-log ioctl fails fast hitting nvme_fail_nonready_cmd(), skipping both nvme_sq_copy_cmd() and nvme_write_sq_db(), so no doorbell write ever occurs. The nvme_admin_create_sq fails timeout finally.

Quoted 3 lines

The block layer is supposed to call the driver's commit_rqs() function if anything in the dispatch list wasn't successful, which should notify the controller of any pending SQEs. Is that not happening here?

Yes, in this case, the last user admin cmd will fail nvme_host_path_errror finally, but ret BLK_STS_OK, which will let blk_mq_dispatch_rq_list skips the commit_rqs, thus missing updating the SQ doorbell.