nvme-pci: enhance timeout kernel log
PATCH v2 1/1 nvme-pci: enhance timeout kernel log
From: Keith Busch <[email protected]>
Kernel configs don't necessarily have opcode decoding, and some opcodes are not even decodable. It is still interesting for debugging SSD issues to know what opcode is timing out, what request type it came from, and the data size (if applicable).
---v1->v2:
Even more verbose annotations in the messages (Jens and Chaitanya)
drivers/nvme/host/pci.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)
drivers/nvme/host/pci.c +10 -8
@@ -1284,6 +1284,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req)struct request *abort_req;struct nvme_command cmd = { };u32 csts =readl (dev->bar +NVME_REG_CSTS ); +u8 opcode;/* If PCI error recovery process is happening, we cannot reset or * the recovery mechanism will surely fail.
@@ -1310,7 +1311,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req)if (blk_mq_rq_state (req) !=MQ_RQ_IN_FLIGHT ) {dev_warn (dev->ctrl.device, -"I/O %d QID %d timeout, completion polled\n" , +"I/O tag %d QID %d timeout, completion polled\n" , req->tag, nvmeq->qid);return BLK_EH_DONE ; }
@@ -1327,7 +1328,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req) fallthrough;case NVME_CTRL_DELETING :dev_warn_ratelimited (dev->ctrl.device, -"I/O %d QID %d timeout, disable controller\n" , +"I/O tag %d QID %d timeout, disable controller\n" , req->tag, nvmeq->qid);nvme_req (req)->flags |=NVME_REQ_CANCELLED ;nvme_dev_disable (dev, true);
@@ -1343,10 +1344,12 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req)* command was already aborted once before and still hasn't been * returned to the driver, or if this is the admin queue. */ + opcode =nvme_req (req)->cmd->common.opcode;if (!nvmeq->qid || iod->aborted) {dev_warn (dev->ctrl.device, -"I/O %d QID %d timeout, reset controller\n" , - req->tag, nvmeq->qid); +"I/O tag %d (opcode:%s %d) QID %d timeout, reset controller\n" , + req->tag,nvme_opcode_str (nvmeq->qid, opcode,0 ), + opcode, nvmeq->qid);nvme_req (req)->flags |=NVME_REQ_CANCELLED ; goto disable; }
@@ -1362,10 +1365,9 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req) cmd.abort.sqid =cpu_to_le16 (nvmeq->qid);dev_warn (nvmeq->dev->ctrl.device, -"I/O %d (%s) QID %d timeout, aborting\n" , - req->tag, -nvme_get_opcode_str (nvme_req (req)->cmd->common.opcode), - nvmeq->qid); +"I/O tag %d (opcode:%s %d) QID %d timeout, aborting req_op:%s(%u) size:%u\n" , + req->tag,nvme_get_opcode_str (opcode), opcode, nvmeq->qid, +blk_op_str (req_op (req)),req_op (req),blk_rq_bytes (req)); abort_req =blk_mq_alloc_request (dev->ctrl.admin_q,nvme_req_op (&cmd),BLK_MQ_REQ_NOWAIT );
Signature · 1 line
2.34.1
Christoph Hellwig
maintainerdrivers/nvme/host/pci.c static enum blk_eh_timer_return nvme_timeout(struct request *req) · line 1314
Quoted 2 lines · Keith Busch
drivers/nvme/host/pci.c +1
-1- "I/O %d QID %d timeout, completion polled\n", + "I/O tag %d QID %d timeout, completion polled\n",
Q: should we use the Linux term tag or the NVMe term command ID (or command_id) ? I don't really care, just thinking out loud.
drivers/nvme/host/pci.c static enum blk_eh_timer_return nvme_timeout(struct request *req) · line 1352
Quoted 8 lines
drivers/nvme/host/pci.c +4
-2+ opcode =nvme_req (req)->cmd->common.opcode;if (!nvmeq->qid || iod->aborted) {dev_warn (dev->ctrl.device, -"I/O %d QID %d timeout, reset controller\n" , - req->tag, nvmeq->qid); +"I/O tag %d (opcode:%s %d) QID %d timeout, reset controller\n" , + req->tag,nvme_opcode_str (nvmeq->qid, opcode,0 ), + opcode, nvmeq->qid);
space after the ":" ?
drivers/nvme/host/pci.c static enum blk_eh_timer_return nvme_timeout(struct request *req) · line 1370
Quoted 3 lines
drivers/nvme/host/pci.c +3
-0+"I/O tag %d (opcode:%s %d) QID %d timeout, aborting req_op:%s(%u) size:%u\n" , + req->tag,nvme_get_opcode_str (opcode), opcode, nvmeq->qid, + blk_op_str(req_op(req)), req_op(req), blk_rq_bytes(req));
Same.
Sagi Grimberg
maintainerQuoted 6 lines
drivers/nvme/host/pci.c static enum blk_eh_timer_return nvme_timeout(struct request *req) · line 1314
Quoted 3 lines
drivers/nvme/host/pci.c +1
-1- "I/O %d QID %d timeout, completion polled\n", + "I/O tag %d QID %d timeout, completion polled\n",Q: should we use the Linux term tag or the NVMe term command ID (or
command_id) ? I don't really care, just thinking out loud.
I was fine with neither tbh, but tag is better than command_id I think.
Quoted 10 lines · Sagi Grimberg
Quoted 6 lines
Quoted 3 lines
- "I/O %d QID %d timeout, completion polled\n", + "I/O tag %d QID %d timeout, completion polled\n",
Q: should we use the Linux term tag or the NVMe term command ID (or
command_id) ? I don't really care, just thinking out loud.I was fine with neither tbh, but tag is better than
command_idI think.
In the current Keith code it is actually the tag. If you would like to print the cid please use: nvme_cid(req).
I guess the cid will be the useful value that we can compare to the wireshark or the target prints.
Keith Busch
maintainerQuoted 14 lines · Max Gurtovoy
Quoted 10 lines · Sagi Grimberg
Quoted 6 lines
drivers/nvme/host/pci.c static enum blk_eh_timer_return nvme_timeout(struct request *req) · line 1314
Quoted 3 lines
drivers/nvme/host/pci.c +1
-1- "I/O %d QID %d timeout, completion polled\n", + "I/O tag %d QID %d timeout, completion polled\n",Q: should we use the Linux term tag or the NVMe term command ID (or
command_id) ? I don't really care, just thinking out loud.I was fine with neither tbh, but tag is better than
command_idI think.
Quoted 5 lines
In the current Keith code it is actually the tag. If you would like to print the cid please use:
nvme_cid(req).I guess the cid will be the useful value that we can compare to the wireshark or the target prints.
Yes! This currently prints the "tag" from blk-mq. I can change it to print the encoded command_id in the SQE. Or both!
Quoted 15 lines · Keith Busch
Quoted 14 lines · Max Gurtovoy
Quoted 10 lines · Sagi Grimberg
Quoted 6 lines
drivers/nvme/host/pci.c static enum blk_eh_timer_return nvme_timeout(struct request *req) · line 1314
Quoted 3 lines
drivers/nvme/host/pci.c +1
-1- "I/O %d QID %d timeout, completion polled\n", + "I/O tag %d QID %d timeout, completion polled\n",Q: should we use the Linux term tag or the NVMe term command ID (or
command_id) ? I don't really care, just thinking out loud.I was fine with neither tbh, but tag is better than
command_idI think.
Quoted 8 lines
Quoted 5 lines
In the current Keith code it is actually the tag. If you would like to print the cid please use:
nvme_cid(req).I guess the cid will be the useful value that we can compare to the wireshark or the target prints.
Yes! This currently prints the "tag" from blk-mq. I can change it to print the encoded
command_idin the SQE. Or both!
I tend to say command_id is more interesting. So better use the option of cid or both.
Can you please align all the timeout functions in different transports to print same value (tag or cid) ? each transport today prints different values. For example tcp prints cid and rdma prints tag :)