Skip to contentSkip to conversation

nvme-pci: enhance timeout kernel log

PATCH Superseded Keith Busch 5 participants 6 messages Dec 18, 2023 – Dec 19, 2023 NVMe

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 nvme-pci: enhance timeout kernel log

From: Keith Busch <[email protected]>

Kernel configs don't necessarily have opcode decoding, and some opcodes are not even decodable. It is still interesting for debugging SSD issues to know what opcode is timing out, what request type it came from, and the data size (if applicable).

---
v1->v2:

Even more verbose annotations in the messages (Jens and Chaitanya)

 drivers/nvme/host/pci.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
drivers/nvme/host/pci.c +10 -8
index 507bc149046dc..f71ec41489a73 100644
@@ -1284,6 +1284,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req)
 	struct request *abort_req;
 	struct nvme_command cmd = { };
 	u32 csts = readl(dev->bar + NVME_REG_CSTS);
+	u8 opcode;
 
 	/* If PCI error recovery process is happening, we cannot reset or
 	 * the recovery mechanism will surely fail.
@@ -1310,7 +1311,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req)
 
 	if (blk_mq_rq_state(req) != MQ_RQ_IN_FLIGHT) {
 		dev_warn(dev->ctrl.device,
-			 "I/O %d QID %d timeout, completion polled\n",
+			 "I/O tag %d QID %d timeout, completion polled\n",
 			 req->tag, nvmeq->qid);
 		return BLK_EH_DONE;
 	}
@@ -1327,7 +1328,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req)
 		fallthrough;
 	case NVME_CTRL_DELETING:
 		dev_warn_ratelimited(dev->ctrl.device,
-			 "I/O %d QID %d timeout, disable controller\n",
+			 "I/O tag %d QID %d timeout, disable controller\n",
 			 req->tag, nvmeq->qid);
 		nvme_req(req)->flags |= NVME_REQ_CANCELLED;
 		nvme_dev_disable(dev, true);
@@ -1343,10 +1344,12 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req)
 	 * command was already aborted once before and still hasn't been
 	 * returned to the driver, or if this is the admin queue.
 	 */
+	opcode = nvme_req(req)->cmd->common.opcode;
 	if (!nvmeq->qid || iod->aborted) {
 		dev_warn(dev->ctrl.device,
-			 "I/O %d QID %d timeout, reset controller\n",
-			 req->tag, nvmeq->qid);
+			 "I/O tag %d (opcode:%s %d) QID %d timeout, reset controller\n",
+			 req->tag, nvme_opcode_str(nvmeq->qid, opcode, 0),
+			 opcode, nvmeq->qid);
 		nvme_req(req)->flags |= NVME_REQ_CANCELLED;
 		goto disable;
 	}
@@ -1362,10 +1365,9 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req)
 	cmd.abort.sqid = cpu_to_le16(nvmeq->qid);
 
 	dev_warn(nvmeq->dev->ctrl.device,
-		"I/O %d (%s) QID %d timeout, aborting\n",
-		 req->tag,
-		 nvme_get_opcode_str(nvme_req(req)->cmd->common.opcode),
-		 nvmeq->qid);
+		 "I/O tag %d (opcode:%s %d) QID %d timeout, aborting req_op:%s(%u) size:%u\n",
+		 req->tag, nvme_get_opcode_str(opcode), opcode, nvmeq->qid,
+		 blk_op_str(req_op(req)), req_op(req), blk_rq_bytes(req));
 
 	abort_req = blk_mq_alloc_request(dev->ctrl.admin_q, nvme_req_op(&cmd),
 					 BLK_MQ_REQ_NOWAIT);
Signature · 1 line
2.34.1
#1 lore raw

drivers/nvme/host/pci.c static enum blk_eh_timer_return nvme_timeout(struct request *req) · line 1314

Quoted 2 lines · Keith Busch
drivers/nvme/host/pci.c +1 -1
-			 "I/O %d QID %d timeout, completion polled\n",
+			 "I/O tag %d QID %d timeout, completion polled\n",

Q: should we use the Linux term tag or the NVMe term command ID (or command_id) ? I don't really care, just thinking out loud.

drivers/nvme/host/pci.c static enum blk_eh_timer_return nvme_timeout(struct request *req) · line 1352

Quoted 8 lines
drivers/nvme/host/pci.c +4 -2
+	opcode = nvme_req(req)->cmd->common.opcode;
 	if (!nvmeq->qid || iod->aborted) {
 		dev_warn(dev->ctrl.device,
-			 "I/O %d QID %d timeout, reset controller\n",
-			 req->tag, nvmeq->qid);
+			 "I/O tag %d (opcode:%s %d) QID %d timeout, reset controller\n",
+			 req->tag, nvme_opcode_str(nvmeq->qid, opcode, 0),
+			 opcode, nvmeq->qid);

space after the ":" ?

drivers/nvme/host/pci.c static enum blk_eh_timer_return nvme_timeout(struct request *req) · line 1370

Quoted 3 lines
drivers/nvme/host/pci.c +3 -0
+		 "I/O tag %d (opcode:%s %d) QID %d timeout, aborting req_op:%s(%u) size:%u\n",
+		 req->tag, nvme_get_opcode_str(opcode), opcode, nvmeq->qid,
+		 blk_op_str(req_op(req)), req_op(req), blk_rq_bytes(req));

Same.

maintainer
#2 lore raw
Quoted 6 lines

drivers/nvme/host/pci.c static enum blk_eh_timer_return nvme_timeout(struct request *req) · line 1314

Quoted 3 lines
drivers/nvme/host/pci.c +1 -1
-			 "I/O %d QID %d timeout, completion polled\n",
+			 "I/O tag %d QID %d timeout, completion polled\n",

Q: should we use the Linux term tag or the NVMe term command ID (or command_id) ? I don't really care, just thinking out loud.

I was fine with neither tbh, but tag is better than command_id I think.

#3 lore raw
Quoted 10 lines · Sagi Grimberg
Quoted 6 lines
Quoted 3 lines

-             "I/O %d QID %d timeout, completion polled\n", +             "I/O tag %d QID %d timeout, completion polled\n",

Q: should we use the Linux term tag or the NVMe term command ID (or command_id) ?  I don't really care, just thinking out loud.

I was fine with neither tbh, but tag is better than command_id I think.

In the current Keith code it is actually the tag. If you would like to print the cid please use: nvme_cid(req).

I guess the cid will be the useful value that we can compare to the wireshark or the target prints.

maintainer
#4 lore raw
Quoted 14 lines · Max Gurtovoy
Quoted 10 lines · Sagi Grimberg
Quoted 6 lines

drivers/nvme/host/pci.c static enum blk_eh_timer_return nvme_timeout(struct request *req) · line 1314

Quoted 3 lines
drivers/nvme/host/pci.c +1 -1
-             "I/O %d QID %d timeout, completion polled\n",
+             "I/O tag %d QID %d timeout, completion polled\n",

Q: should we use the Linux term tag or the NVMe term command ID (or command_id) ? I don't really care, just thinking out loud.

I was fine with neither tbh, but tag is better than command_id I think.

Quoted 5 lines

In the current Keith code it is actually the tag. If you would like to print the cid please use: nvme_cid(req).

I guess the cid will be the useful value that we can compare to the wireshark or the target prints.

Yes! This currently prints the "tag" from blk-mq. I can change it to print the encoded command_id in the SQE. Or both!

#5 lore raw
Quoted 15 lines · Keith Busch
Quoted 14 lines · Max Gurtovoy
Quoted 10 lines · Sagi Grimberg
Quoted 6 lines

drivers/nvme/host/pci.c static enum blk_eh_timer_return nvme_timeout(struct request *req) · line 1314

Quoted 3 lines
drivers/nvme/host/pci.c +1 -1
-             "I/O %d QID %d timeout, completion polled\n",
+             "I/O tag %d QID %d timeout, completion polled\n",

Q: should we use the Linux term tag or the NVMe term command ID (or command_id) ? I don't really care, just thinking out loud.

I was fine with neither tbh, but tag is better than command_id I think.

Quoted 8 lines
Quoted 5 lines

In the current Keith code it is actually the tag. If you would like to print the cid please use: nvme_cid(req).

I guess the cid will be the useful value that we can compare to the wireshark or the target prints.

Yes! This currently prints the "tag" from blk-mq. I can change it to print the encoded command_id in the SQE. Or both!

I tend to say command_id is more interesting. So better use the option of cid or both.

Can you please align all the timeout functions in different transports to print same value (tag or cid) ? each transport today prints different values. For example tcp prints cid and rdma prints tag :)