Skip to contentSkip to conversation

nvme: ratelimit the completion-path messages driven by device data

PATCH In review Chao Shi 2 participants 3 messages Aug 10 – Aug 11 NVMe

lore
Share & tools
lore

Chao Shi

author
#0 lore raw

PATCH 1/1 nvme: ratelimit the completion-path messages driven by device data

nvme_find_rq() and nvme_handle_cqe() print an unratelimited message for every completion queue entry whose command id does not resolve to an in-flight request. Both are reached from the completion interrupt path (nvme_irq() -> nvme_poll_cq() -> nvme_handle_cqe()) and the decision to print is made entirely from device-supplied data, so a controller that posts a stream of bogus command ids drives unbounded printk from hard interrupt context.

This is not hypothetical. A single boot under an emulated controller that posts invalid completions produced 846 "could not locate request for tag 0x0", 846 "invalid id 0 completed on queue 2" and 123 "genctr mismatch" lines. Once the tag set has been torn down every subsequent completion resolves to nothing, so the print rate is bounded only by how fast the device can post entries.

Ratelimit the three messages. The information they carry is diagnostic and repeats, so the suppression count printed by the ratelimit helpers is enough to tell that the condition persists. This matches how the other device-driven error prints in the driver are already handled, for example the status messages in nvme_log_error() and nvme_log_err_passthru().

nvme_find_rq() lives in nvme.h and is shared by pci, tcp, rdma, apple and target-loop, so all transports are covered.

Found by FuzzNvme.

---
 drivers/nvme/host/nvme.h | 11 ++++++-----
 drivers/nvme/host/pci.c  |  6 +++---
 2 files changed, 9 insertions(+), 8 deletions(-)
drivers/nvme/host/nvme.h +6 -5
index ccd5e05dac98..31e771e1b721 100644
@@ -666,14 +666,15 @@ static inline struct request *nvme_find_rq(struct blk_mq_tags *tags,
 
 	rq = blk_mq_tag_to_rq(tags, tag);
 	if (unlikely(!rq)) {
-		pr_err("could not locate request for tag %#x\n",
-			tag);
+		pr_err_ratelimited("could not locate request for tag %#x\n",
+				   tag);
 		return NULL;
 	}
 	if (unlikely(nvme_genctr_mask(nvme_req(rq)->genctr) != genctr)) {
-		dev_err(nvme_req(rq)->ctrl->device,
-			"request %#x genctr mismatch (got %#x expected %#x)\n",
-			tag, genctr, nvme_genctr_mask(nvme_req(rq)->genctr));
+		dev_err_ratelimited(nvme_req(rq)->ctrl->device,
+				    "request %#x genctr mismatch (got %#x expected %#x)\n",
+				    tag, genctr,
+				    nvme_genctr_mask(nvme_req(rq)->genctr));
 		return NULL;
 	}
 	return rq;
drivers/nvme/host/pci.c +3 -3
index db5fc9bf6627..93c0cb47bcaf 100644
@@ -1549,9 +1549,9 @@ static inline void nvme_handle_cqe(struct nvme_queue *nvmeq,
 
 	req = nvme_find_rq(nvme_queue_tagset(nvmeq), command_id);
 	if (unlikely(!req)) {
-		dev_warn(nvmeq->dev->ctrl.device,
-			"invalid id %d completed on queue %d\n",
-			command_id, le16_to_cpu(cqe->sq_id));
+		dev_warn_ratelimited(nvmeq->dev->ctrl.device,
+				     "invalid id %d completed on queue %d\n",
+				     command_id, le16_to_cpu(cqe->sq_id));
 		return;
 	}
 

base-commit: 8541d8f725c673db3bd741947f27974358b2e163

Signature · 1 line
2.43.0
maintainer
#1 lore raw
Quoted 7 lines · Chao Shi

nvme_find_rq() and nvme_handle_cqe() print an unratelimited message for every completion queue entry whose command id does not resolve to an in-flight request. Both are reached from the completion interrupt path (nvme_irq() -> nvme_poll_cq() -> nvme_handle_cqe()) and the decision to print is made entirely from device-supplied data, so a controller that posts a stream of bogus command ids drives unbounded printk from hard interrupt context.

Yeah, if a controller is so badly malfunctioning that only garbage is getting posted to the completion queues, then we don't really get any more information by dumping every single instance compared to rate limiting it. So applied to nvme-7.3, but with a minor fix to an overly long line.

author
#2 lore raw

Thank you so much! I will take care of the later patch submission; avoid long lines. Appreciate!

Quoted 15 lines · Keith Busch
Quoted 7 lines · Chao Shi

nvme_find_rq() and nvme_handle_cqe() print an unratelimited message for every completion queue entry whose command id does not resolve to an in-flight request. Both are reached from the completion interrupt path (nvme_irq() -> nvme_poll_cq() -> nvme_handle_cqe()) and the decision to print is made entirely from device-supplied data, so a controller that posts a stream of bogus command ids drives unbounded printk from hard interrupt context.

Yeah, if a controller is so badly malfunctioning that only garbage is getting posted to the completion queues, then we don't really get any more information by dumping every single instance compared to rate limiting it. So applied to nvme-7.3, but with a minor fix to an overly long line.