nvme: ratelimit the completion-path messages driven by device data
PATCH 1/1 nvme: ratelimit the completion-path messages driven by device data
nvme_find_rq() and nvme_handle_cqe() print an unratelimited message for every completion queue entry whose command id does not resolve to an in-flight request. Both are reached from the completion interrupt path (nvme_irq() -> nvme_poll_cq() -> nvme_handle_cqe()) and the decision to print is made entirely from device-supplied data, so a controller that posts a stream of bogus command ids drives unbounded printk from hard interrupt context.
This is not hypothetical. A single boot under an emulated controller that posts invalid completions produced 846 "could not locate request for tag 0x0", 846 "invalid id 0 completed on queue 2" and 123 "genctr mismatch" lines. Once the tag set has been torn down every subsequent completion resolves to nothing, so the print rate is bounded only by how fast the device can post entries.
Ratelimit the three messages. The information they carry is diagnostic and repeats, so the suppression count printed by the ratelimit helpers is enough to tell that the condition persists. This matches how the other device-driven error prints in the driver are already handled, for example the status messages in nvme_log_error() and nvme_log_err_passthru().
nvme_find_rq() lives in nvme.h and is shared by pci, tcp, rdma, apple and target-loop, so all transports are covered.
Found by FuzzNvme.
--- drivers/nvme/host/nvme.h | 11 ++++++----- drivers/nvme/host/pci.c | 6 +++--- 2 files changed, 9 insertions(+), 8 deletions(-)
drivers/nvme/host/nvme.h +6 -5
@@ -666,14 +666,15 @@ static inline struct request *nvme_find_rq(struct blk_mq_tags *tags, rq =blk_mq_tag_to_rq (tags, tag);if (unlikely (!rq)) { -pr_err ("could not locate request for tag %#x\n" , - tag); +pr_err_ratelimited ("could not locate request for tag %#x\n" , + tag);return NULL ; }if (unlikely (nvme_genctr_mask (nvme_req (rq)->genctr) != genctr)) { -dev_err (nvme_req (rq)->ctrl->device, -"request %#x genctr mismatch (got %#x expected %#x)\n" , - tag, genctr,nvme_genctr_mask (nvme_req (rq)->genctr)); +dev_err_ratelimited (nvme_req (rq)->ctrl->device, +"request %#x genctr mismatch (got %#x expected %#x)\n" , + tag, genctr, +nvme_genctr_mask (nvme_req (rq)->genctr));return NULL ; }return rq;
drivers/nvme/host/pci.c +3 -3
@@ -1549,9 +1549,9 @@ static inline void nvme_handle_cqe(struct nvme_queue *nvmeq, req =nvme_find_rq (nvme_queue_tagset (nvmeq), command_id);if (unlikely (!req)) { -dev_warn (nvmeq->dev->ctrl.device, -"invalid id %d completed on queue %d\n" , - command_id,le16_to_cpu (cqe->sq_id)); +dev_warn_ratelimited (nvmeq->dev->ctrl.device, +"invalid id %d completed on queue %d\n" , + command_id,le16_to_cpu (cqe->sq_id));return ; }
base-commit: 8541d8f725c673db3bd741947f27974358b2e163
Signature · 1 line
2.43.0
Keith Busch
maintainerQuoted 7 lines · Chao Shi
nvme_find_rq()andnvme_handle_cqe()print an unratelimited message for every completion queue entry whose command id does not resolve to an in-flight request. Both are reached from the completion interrupt path (nvme_irq()->nvme_poll_cq()->nvme_handle_cqe()) and the decision to print is made entirely from device-supplied data, so a controller that posts a stream of bogus command ids drives unbounded printk from hard interrupt context.
Yeah, if a controller is so badly malfunctioning that only garbage is getting posted to the completion queues, then we don't really get any more information by dumping every single instance compared to rate limiting it. So applied to nvme-7.3, but with a minor fix to an overly long line.
Thank you so much! I will take care of the later patch submission; avoid long lines. Appreciate!
Quoted 15 lines · Keith Busch
Quoted 7 lines · Chao Shi
nvme_find_rq()andnvme_handle_cqe()print an unratelimited message for every completion queue entry whose command id does not resolve to an in-flight request. Both are reached from the completion interrupt path (nvme_irq()->nvme_poll_cq()->nvme_handle_cqe()) and the decision to print is made entirely from device-supplied data, so a controller that posts a stream of bogus command ids drives unbounded printk from hard interrupt context.Yeah, if a controller is so badly malfunctioning that only garbage is getting posted to the completion queues, then we don't really get any more information by dumping every single instance compared to rate limiting it. So applied to nvme-7.3, but with a minor fix to an overly long line.