Skip to contentSkip to conversation

scsi: ufs: core: Fix unsafe MMIO reads and redundant CQ sweeps in MCQ reset

PATCH Applied Stanley Jhu 4 participants 6 messages Sep 28 – Oct 6

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v3 cover scsi: ufs: core: Fix unsafe MMIO reads and redundant CQ sweeps in MCQ reset

During Multi-Circular Queue (MCQ) error recovery and host reset, ufshcd_mcq_compl_pending_transfer() sweeps or polls completion queues to reap pending transfers. Two bugs exist in this path:

1. Unsafe MMIO read and spurious errors while HCE = 0 (Patch 1/2):
ufshcd_host_reset_and_restore() stops the controller (HCE = 0) before calling ufshcd_mcq_compl_all_cqes_lock(). Calling ufshcd_mcq_update_cq_tail_slot() at the end of the sweep reads CQTPy over MMIO while HCE = 0, directly contradicting the function's own documented contract that reading host controller registers may not be safe when the controller is disabled. In addition, passing expected empty slots during a full-ring sweep into ufshcd_mcq_process_cqe() prints spurious "Abnormal CQ entry!" errors.

2. Redundant per-request CQ sweeps and polls (Patch 2/2):
ufshcd_mcq_compl_pending_transfer() runs hardware queue completion sweeps (force_compl == true) or CQTPy polls (force_compl == false) inside blk_mq_tagset_busy_iter() callbacks, repeating whole-queue operations once per busy request instead of once per hardware queue.

Patch 1/2 removes the unsafe ufshcd_mcq_update_cq_tail_slot() call and redundant slot assignment in ufshcd_mcq_compl_all_cqes_lock(), and extracts ufshcd_mcq_compl_cqe() so full-ring sweeps skip empty slots silently. Patch 2/2 sweeps or polls each hardware queue once before iterating residual requests and removes ufshcd_mcq_compl_one().

Differences from v2:
- Fix the grammar of the CQE comment in ufshcd_mcq_compl_cqe() (Bart).
- Drop the redundant cq_tail_slot assignment in ufshcd_mcq_compl_all_cqes_lock() (Bart).
- Add Reviewed-by tags from Peter and Bart.

Differences from v1:
- Split into a two-patch series separating ring sweep safety from per-request tagset iteration.
- Extract ufshcd_mcq_compl_cqe() to skip empty slots without double CQE checks.
- Decouple hardware queue polling/sweeping for both force_compl paths and remove ufshcd_mcq_compl_one().

Tested: QEMU ARM64 MCQ/SDB host reset and I/O without CQE error logs.

Stanley Jhu (2):
scsi: ufs: core: Avoid unsafe MMIO reads in
ufshcd_mcq_compl_all_cqes_lock()
scsi: ufs: core: Decouple CQ sweep from request iterator in MCQ

 drivers/ufs/core/ufs-mcq.c | 30 +++++++++++++++++-------------
 drivers/ufs/core/ufshcd.c  | 31 ++++++++++++-------------------
 2 files changed, 29 insertions(+), 32 deletions(-)

base-commit: f09d2c7485b32adb82336d0d748935c8237a649e

Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#1 lore raw

PATCH v3 1/2 scsi: ufs: core: Avoid unsafe MMIO reads in ufshcd_mcq_compl_all_cqes_lock()

During MCQ host reset, ufshcd_host_reset_and_restore() stops the host controller via ufshcd_hba_stop() (HCE = 0) before calling ufshcd_complete_requests(hba, true) -> ufshcd_mcq_compl_pending_transfer(hba, true) -> ufshcd_mcq_force_compl_one() -> ufshcd_mcq_compl_all_cqes_lock(). Because ufshcd_mcq_force_compl_one() is its sole caller, ufshcd_mcq_compl_all_cqes_lock() always runs with HCE = 0.

Despite the comment above ufshcd_mcq_compl_all_cqes_lock() stating that reading CQTPy may not be safe with the controller disabled, the function still calls ufshcd_mcq_update_cq_tail_slot() at the end of its sweep:

1. Unsafe CQTPy MMIO read:
Calling ufshcd_mcq_update_cq_tail_slot() at the end of the sweep reads CQTPy over MMIO while HCE = 0, directly contradicting the function's documented contract (commit 1373df88d535 ("scsi: ufs:
core: Add a comment block above ufshcd_mcq_compl_all_cqes_lock()")) that reading CQTPy may not be safe with the controller disabled.

2. Spurious error logs on empty slots:
Sweeping all max_entries slots visits empty entries where command_desc_base_addr is 0, causing ufshcd_mcq_process_cqe() to log unguarded dev_err(hba->dev, "Abnormal CQ entry!\n") messages.

Fix both issues in ufshcd_mcq_compl_all_cqes_lock():
- Remove the ufshcd_mcq_update_cq_tail_slot() call and the redundant hwq->cq_head_slot = hwq->cq_tail_slot assignment without replacement. The two indices are already equal after the sweep: they are equal when the sweep starts, since ufshcd_mcq_poll_cqe_lock() consumes entries until cq_head_slot reaches cq_tail_slot, and the sweep advances cq_head_slot by exactly one full ring. Both indices are also reinitialized before the queue is reused.
- Extract ufshcd_mcq_compl_cqe() and invoke it only on non-empty slots during full-ring sweeps, keeping "Abnormal CQ entry!" logging strictly for unexpected empty entries in ufshcd_mcq_poll_cqe_lock().

Fixes ab248643d3d6 ("scsi: ufs: core: Add error handling for MCQ mode")
---
 drivers/ufs/core/ufs-mcq.c | 30 +++++++++++++++++-------------
 1 file changed, 17 insertions(+), 13 deletions(-)
drivers/ufs/core/ufs-mcq.c +17 -13
index 8106d55f4041..df293b66350d 100644
@@ -312,20 +312,24 @@ static int ufshcd_mcq_get_tag(struct ufs_hba *hba, struct cq_entry *cqe)
 		UFSHCD_NUM_RESERVED;
 }
 
+static void ufshcd_mcq_compl_cqe(struct ufs_hba *hba, struct cq_entry *cqe)
+{
+	int tag = ufshcd_mcq_get_tag(hba, cqe);
+
+	ufshcd_compl_one_cqe(hba, tag, cqe);
+	/* After processing the CQE, mark it as an empty (invalid) entry. */
+	cqe->command_desc_base_addr = 0;
+}
+
 static void ufshcd_mcq_process_cqe(struct ufs_hba *hba,
 				   struct ufs_hw_queue *hwq)
 {
 	struct cq_entry *cqe = ufshcd_mcq_cur_cqe(hwq);
 
-	if (cqe->command_desc_base_addr) {
-		int tag = ufshcd_mcq_get_tag(hba, cqe);
-
-		ufshcd_compl_one_cqe(hba, tag, cqe);
-		/* After processed the cqe, mark it empty (invalid) entry */
-		cqe->command_desc_base_addr = 0;
-	} else {
+	if (cqe->command_desc_base_addr)
+		ufshcd_mcq_compl_cqe(hba, cqe);
+	else
 		dev_err(hba->dev, "Abnormal CQ entry!\n");
-	}
 }
 
 /*
@@ -333,7 +337,7 @@ static void ufshcd_mcq_process_cqe(struct ufs_hba *hba,
  * controller disabled (HCE = 0). Reading host controller registers, e.g. the
  * CQ tail pointer (CQTPy), may not be safe with the host controller disabled.
  * Hence, iterate over all completion queue entries. This won't result in
- * double completions because ufshcd_mcq_process_cqe() clears a CQE after it
+ * double completions because ufshcd_mcq_compl_cqe() clears a CQE after it
  * has been processed.
  */
 void ufshcd_mcq_compl_all_cqes_lock(struct ufs_hba *hba,
@@ -344,13 +348,13 @@ void ufshcd_mcq_compl_all_cqes_lock(struct ufs_hba *hba,
 
 	spin_lock_irqsave(&hwq->cq_lock, flags);
 	while (entries > 0) {
-		ufshcd_mcq_process_cqe(hba, hwq);
+		struct cq_entry *cqe = ufshcd_mcq_cur_cqe(hwq);
+
+		if (cqe->command_desc_base_addr)
+			ufshcd_mcq_compl_cqe(hba, cqe);
 		ufshcd_mcq_inc_cq_head_slot(hwq);
 		entries--;
 	}
-
-	ufshcd_mcq_update_cq_tail_slot(hwq);
-	hwq->cq_head_slot = hwq->cq_tail_slot;
 	spin_unlock_irqrestore(&hwq->cq_lock, flags);
 }
 
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
reviewer Reviewed-by on v3 1/2
#2 lore raw

Re: [PATCH v3 1/2] scsi: ufs: core: Avoid unsafe MMIO reads in ufshcd_mcq_compl_all_cqes_lock()

Quoted 11 lines · Stanley Jhu

Fix both issues in ufshcd_mcq_compl_all_cqes_lock():
- Remove the ufshcd_mcq_update_cq_tail_slot() call and the redundant hwq->cq_head_slot = hwq->cq_tail_slot assignment without replacement. The two indices are already equal after the sweep: they are equal when the sweep starts, since ufshcd_mcq_poll_cqe_lock() consumes entries until cq_head_slot reaches cq_tail_slot, and the sweep advances cq_head_slot by exactly one full ring. Both indices are also reinitialized before the queue is reused.
- Extract ufshcd_mcq_compl_cqe() and invoke it only on non-empty slots during full-ring sweeps, keeping "Abnormal CQ entry!" logging strictly for unexpected empty entries in ufshcd_mcq_poll_cqe_lock().

#3 lore raw

PATCH v3 2/2 scsi: ufs: core: Decouple CQ sweep from request iterator in MCQ

In reply to Stanley Jhu

In MCQ mode, ufshcd_mcq_compl_pending_transfer() uses blk_mq_tagset_busy_iter() to iterate over busy requests during error recovery and host reset. However, both iterator callbacks perform whole-queue operations redundantly for each visited request:

- force_compl == true: ufshcd_mcq_force_compl_one() calls ufshcd_mcq_compl_all_cqes_lock() on every busy request, sweeping the entire completion ring (hwq->max_entries slots) once per active request under spin_lock_irqsave even though the first sweep already cleared all completion entries.
- force_compl == false: ufshcd_mcq_compl_one() acquires cq_lock and polls CQTPy over MMIO via ufshcd_mcq_poll_cqe_lock() for every busy request without doing any per-request work.

Sweep or poll each hardware queue (hba->uhq[i]) once at the start of ufshcd_mcq_compl_pending_transfer(). When force_compl is true, run blk_mq_tagset_busy_iter() afterward to complete residual in-flight requests with DID_REQUEUE, and remove the now-unused ufshcd_mcq_compl_one() callback.

Fixes ab248643d3d6 ("scsi: ufs: core: Add error handling for MCQ mode")
---
 drivers/ufs/core/ufshcd.c | 31 ++++++++++++-------------------
 1 file changed, 12 insertions(+), 19 deletions(-)
drivers/ufs/core/ufshcd.c +12 -19
index 234e18b5078f..f50431f35e58 100644
@@ -6068,8 +6068,6 @@ static bool ufshcd_mcq_force_compl_one(struct request *rq, void *priv)
 	if (blk_mq_is_reserved_rq(rq) || !hwq)
 		return true;
 
-	ufshcd_mcq_compl_all_cqes_lock(hba, hwq);
-
 	/*
 	 * For those cmds of which the cqes are not present in the cq, complete
 	 * them explicitly.
@@ -6085,19 +6083,6 @@ static bool ufshcd_mcq_force_compl_one(struct request *rq, void *priv)
 	return true;
 }
 
-static bool ufshcd_mcq_compl_one(struct request *rq, void *priv)
-{
-	struct scsi_device *sdev = rq->q->queuedata;
-	struct Scsi_Host *shost = sdev->host;
-	struct ufs_hba *hba = shost_priv(shost);
-	struct ufs_hw_queue *hwq = ufshcd_mcq_req_to_hwq(hba, rq);
-
-	if (!blk_mq_is_reserved_rq(rq) && hwq)
-		ufshcd_mcq_poll_cqe_lock(hba, hwq);
-
-	return true;
-}
-
 /**
  * ufshcd_mcq_compl_pending_transfer - MCQ mode function. It is
  * invoked from the error handler context or ufshcd_host_reset_and_restore()
@@ -6112,10 +6097,18 @@ static bool ufshcd_mcq_compl_one(struct request *rq, void *priv)
 static void ufshcd_mcq_compl_pending_transfer(struct ufs_hba *hba,
 					      bool force_compl)
 {
-	blk_mq_tagset_busy_iter(&hba->host->tag_set,
-				force_compl ? ufshcd_mcq_force_compl_one :
-					      ufshcd_mcq_compl_one,
-				NULL);
+	int i;
+
+	for (i = 0; i < hba->nr_hw_queues; i++) {
+		if (force_compl)
+			ufshcd_mcq_compl_all_cqes_lock(hba, &hba->uhq[i]);
+		else
+			ufshcd_mcq_poll_cqe_lock(hba, &hba->uhq[i]);
+	}
+
+	if (force_compl)
+		blk_mq_tagset_busy_iter(&hba->host->tag_set,
+					ufshcd_mcq_force_compl_one, NULL);
 }
 
 /**
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Reviewed-by on v3 2/2
#4 lore raw

Re: [PATCH v3 2/2] scsi: ufs: core: Decouple CQ sweep from request iterator in MCQ

Quoted 21 lines · Stanley Jhu

In MCQ mode, ufshcd_mcq_compl_pending_transfer() uses blk_mq_tagset_busy_iter() to iterate over busy requests during error recovery and host reset. However, both iterator callbacks perform whole-queue operations redundantly for each visited request:

- force_compl == true: ufshcd_mcq_force_compl_one() calls ufshcd_mcq_compl_all_cqes_lock() on every busy request, sweeping the
  entire completion ring (hwq->max_entries slots) once per active request under spin_lock_irqsave even though the first sweep already cleared all completion entries.
- force_compl == false: ufshcd_mcq_compl_one() acquires cq_lock and polls CQTPy over MMIO via ufshcd_mcq_poll_cqe_lock() for every busy request without doing any per-request work.

Sweep or poll each hardware queue (hba->uhq[i]) once at the start of ufshcd_mcq_compl_pending_transfer(). When force_compl is true, run blk_mq_tagset_busy_iter() afterward to complete residual in-flight requests with DID_REQUEUE, and remove the now-unused ufshcd_mcq_compl_one() callback.

Quoted 5 lines
Fixes ab248643d3d6 ("scsi: ufs: core: Add error handling for MCQ

mode")

maintainer Applied
#5 lore raw
In reply to Stanley Jhu
Quoted 14 lines · Stanley Jhu

During Multi-Circular Queue (MCQ) error recovery and host reset, ufshcd_mcq_compl_pending_transfer() sweeps or polls completion queues to reap pending transfers. Two bugs exist in this path:

1. Unsafe MMIO read and spurious errors while HCE = 0 (Patch 1/2):
ufshcd_host_reset_and_restore() stops the controller (HCE = 0) before calling ufshcd_mcq_compl_all_cqes_lock(). Calling ufshcd_mcq_update_cq_tail_slot() at the end of the sweep reads CQTPy over MMIO while HCE = 0, directly contradicting the function's own documented contract that reading host controller registers may not be safe when the controller is disabled. In addition, passing expected empty slots during a full-ring sweep into ufshcd_mcq_process_cqe() prints spurious "Abnormal CQ entry!" errors.

Quoted 1 line

[...]

Applied to 7.3/scsi-fixes, thanks!

[1/2] scsi: ufs: core: Avoid unsafe MMIO reads in ufshcd_mcq_compl_all_cqes_lock()

      https://git.kernel.org/mkp/scsi/c/375f3a5691dd
[2/2] scsi: ufs: core: Decouple CQ sweep from request iterator in MCQ
      https://git.kernel.org/mkp/scsi/c/469fa055664b
Signature · 1 line
Martin K. Petersen