Skip to contentSkip to conversation

nvme-pci: enhance timeout kernel log

PATCH Superseded Keith Busch 4 participants 4 messages Dec 7, 2023 NVMe

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 nvme-pci: enhance timeout kernel log

From: Keith Busch <[email protected]>

Kernel configs don't necessarily have opcode decoding, and some opcodes are not even decodable. It is still interesting for debugging SSD issues to know what opcode is timing out, what request type it came from, and the data size (if applicable).

---
 drivers/nvme/host/pci.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/nvme/host/pci.c +5 -4
index fad4cccce745c..49771919b01f1 100644
@@ -1284,6 +1284,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req)
 	struct request *abort_req;
 	struct nvme_command cmd = { };
 	u32 csts = readl(dev->bar + NVME_REG_CSTS);
+	u8 opcode;
 
 	/* If PCI error recovery process is happening, we cannot reset or
 	 * the recovery mechanism will surely fail.
@@ -1361,11 +1362,11 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req)
 	cmd.abort.cid = nvme_cid(req);
 	cmd.abort.sqid = cpu_to_le16(nvmeq->qid);
 
+	opcode = nvme_req(req)->cmd->common.opcode,
 	dev_warn(nvmeq->dev->ctrl.device,
-		"I/O %d (%s) QID %d timeout, aborting\n",
-		 req->tag,
-		 nvme_get_opcode_str(nvme_req(req)->cmd->common.opcode),
-		 nvmeq->qid);
+		 "I/O %d (%s %d) QID %d timeout, aborting req_op:%u size:%u\n",
+		 req->tag, nvme_get_opcode_str(opcode), opcode, nvmeq->qid,
+		 req_op(req), blk_rq_bytes(req));
 
 	abort_req = blk_mq_alloc_request(dev->ctrl.admin_q, nvme_req_op(&cmd),
 					 BLK_MQ_REQ_NOWAIT);
Signature · 1 line
2.34.1
maintainer
#1 lore raw
Quoted 27 lines · Keith Busch

From: Keith Busch <[email protected]>

Kernel configs don't necessarily have opcode decoding, and some opcodes are not even decodable. It is still interesting for debugging SSD issues to know what opcode is timing out, what request type it came from, and the data size (if applicable).

---
 drivers/nvme/host/pci.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/nvme/host/pci.c +1 -0
index fad4cccce745c..49771919b01f1 100644
@@ -1284,6 +1284,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req)
 	struct request *abort_req;
 	struct nvme_command cmd = { };
 	u32 csts = readl(dev->bar + NVME_REG_CSTS);
+	u8 opcode;
 
 	/* If PCI error recovery process is happening, we cannot reset or
 	 * the recovery mechanism will surely fail.
@@ -1361,11 +1362,11 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req)
 	cmd.abort.cid = nvme_cid(req);
 	cmd.abort.sqid = cpu_to_le16(nvmeq->qid);

drivers/nvme/host/pci.c static enum blk_eh_timer_return nvme_timeout(struct request *req) · line 1369

Quoted 10 lines
drivers/nvme/host/pci.c +4 -4
 
+	opcode = nvme_req(req)->cmd->common.opcode,
 	dev_warn(nvmeq->dev->ctrl.device,
-		"I/O %d (%s) QID %d timeout, aborting\n",
-		 req->tag,
-		 nvme_get_opcode_str(nvme_req(req)->cmd->common.opcode),
-		 nvmeq->qid);
+		 "I/O %d (%s %d) QID %d timeout, aborting req_op:%u size:%u\n",
+		 req->tag, nvme_get_opcode_str(opcode), opcode, nvmeq->qid,
+		 req_op(req), blk_rq_bytes(req));

Your additions look good to me, but I do wish that we'd be equally verbose on what the other values are. Ala:

	I/O tag %d (opcode: %s %d) ...

would be a lot more useful, for example. I find myself having to dig into the source for a specific kernel sometimes when looking at various nvme errors, which is really annoying.

Signature · 1 line
Jens Axboe
maintainer
#2 lore raw
Quoted 42 lines · Jens Axboe
Quoted 27 lines · Keith Busch

From: Keith Busch <[email protected]>

Kernel configs don't necessarily have opcode decoding, and some opcodes are not even decodable. It is still interesting for debugging SSD issues to know what opcode is timing out, what request type it came from, and the data size (if applicable).

---
 drivers/nvme/host/pci.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/nvme/host/pci.c +1 -0
index fad4cccce745c..49771919b01f1 100644
@@ -1284,6 +1284,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req)
 	struct request *abort_req;
 	struct nvme_command cmd = { };
 	u32 csts = readl(dev->bar + NVME_REG_CSTS);
+	u8 opcode;
 
 	/* If PCI error recovery process is happening, we cannot reset or
 	 * the recovery mechanism will surely fail.
@@ -1361,11 +1362,11 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req)
 	cmd.abort.cid = nvme_cid(req);
 	cmd.abort.sqid = cpu_to_le16(nvmeq->qid);

drivers/nvme/host/pci.c static enum blk_eh_timer_return nvme_timeout(struct request *req) · line 1369

Quoted 10 lines
drivers/nvme/host/pci.c +4 -4
 
+	opcode = nvme_req(req)->cmd->common.opcode,
 	dev_warn(nvmeq->dev->ctrl.device,
-		"I/O %d (%s) QID %d timeout, aborting\n",
-		 req->tag,
-		 nvme_get_opcode_str(nvme_req(req)->cmd->common.opcode),
-		 nvmeq->qid);
+		 "I/O %d (%s %d) QID %d timeout, aborting req_op:%u size:%u\n",
+		 req->tag, nvme_get_opcode_str(opcode), opcode, nvmeq->qid,
+		 req_op(req), blk_rq_bytes(req));

Your additions look good to me, but I do wish that we'd be equally verbose on what the other values are. Ala:

Quoted 5 lines
	I/O tag %d (opcode: %s %d) ...

would be a lot more useful, for example. I find myself having to dig into the source for a specific kernel sometimes when looking at various nvme errors, which is really annoying.

That is annoying. I'll add your suggestion in.

#3 lore raw
Quoted 43 lines · Keith Busch
Quoted 42 lines · Jens Axboe
Quoted 27 lines · Keith Busch

From: Keith Busch <[email protected]>

Kernel configs don't necessarily have opcode decoding, and some opcodes are not even decodable. It is still interesting for debugging SSD issues to know what opcode is timing out, what request type it came from, and the data size (if applicable).

---
  drivers/nvme/host/pci.c | 9 +++++----
  1 file changed, 5 insertions(+), 4 deletions(-)
drivers/nvme/host/pci.c +1 -0
index fad4cccce745c..49771919b01f1 100644
@@ -1284,6 +1284,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req)
  	struct request *abort_req;
  	struct nvme_command cmd = { };
  	u32 csts = readl(dev->bar + NVME_REG_CSTS);
+	u8 opcode;
 
  	/* If PCI error recovery process is happening, we cannot reset or
  	 * the recovery mechanism will surely fail.
@@ -1361,11 +1362,11 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req)
  	cmd.abort.cid = nvme_cid(req);
  	cmd.abort.sqid = cpu_to_le16(nvmeq->qid);

drivers/nvme/host/pci.c static enum blk_eh_timer_return nvme_timeout(struct request *req) · line 1369

Quoted 10 lines
drivers/nvme/host/pci.c +4 -4
 
+	opcode = nvme_req(req)->cmd->common.opcode,
  	dev_warn(nvmeq->dev->ctrl.device,
-		"I/O %d (%s) QID %d timeout, aborting\n",
-		 req->tag,
-		 nvme_get_opcode_str(nvme_req(req)->cmd->common.opcode),
-		 nvmeq->qid);
+		 "I/O %d (%s %d) QID %d timeout, aborting req_op:%u size:%u\n",
+		 req->tag, nvme_get_opcode_str(opcode), opcode, nvmeq->qid,
+		 req_op(req), blk_rq_bytes(req));

Your additions look good to me, but I do wish that we'd be equally verbose on what the other values are. Ala:

Quoted 8 lines
Quoted 5 lines
	I/O tag %d (opcode: %s %d) ...

would be a lot more useful, for example. I find myself having to dig into the source for a specific kernel sometimes when looking at various nvme errors, which is really annoying.

That is annoying. I'll add your suggestion in.

while doing that is it worth printing the blk_op_str(req_op(req)) along with the req_op(req) since we are doing verbose anyway ? if it is not worth disregard my suggestion ...

-ck