scsi: ufs: core: Fix unsafe MMIO reads and redundant CQ sweeps in MCQ reset
Subsystems (3)
v3 cover scsi: ufs: core: Fix unsafe MMIO reads and redundant CQ sweeps in MCQ reset
During Multi-Circular Queue (MCQ) error recovery and host reset, ufshcd_mcq_compl_pending_transfer() sweeps or polls completion queues to reap pending transfers. Two bugs exist in this path:
1. Unsafe MMIO read and spurious errors while HCE = 0 (Patch 1/2):
ufshcd_host_reset_and_restore() stops the controller (HCE = 0) before calling ufshcd_mcq_compl_all_cqes_lock(). Calling ufshcd_mcq_update_cq_tail_slot() at the end of the sweep reads CQTPy over MMIO while HCE = 0, directly contradicting the function's own documented contract that reading host controller registers may not be safe when the controller is disabled. In addition, passing expected empty slots during a full-ring sweep into ufshcd_mcq_process_cqe() prints spurious "Abnormal CQ entry!" errors.
2. Redundant per-request CQ sweeps and polls (Patch 2/2):
ufshcd_mcq_compl_pending_transfer() runs hardware queue completion sweeps (force_compl == true) or CQTPy polls (force_compl == false) inside blk_mq_tagset_busy_iter() callbacks, repeating whole-queue operations once per busy request instead of once per hardware queue.
Patch 1/2 removes the unsafe ufshcd_mcq_update_cq_tail_slot() call and redundant slot assignment in ufshcd_mcq_compl_all_cqes_lock(), and extracts ufshcd_mcq_compl_cqe() so full-ring sweeps skip empty slots silently. Patch 2/2 sweeps or polls each hardware queue once before iterating residual requests and removes ufshcd_mcq_compl_one().
Differences from v2:
- Fix the grammar of the CQE comment in ufshcd_mcq_compl_cqe() (Bart).
- Drop the redundant cq_tail_slot assignment in ufshcd_mcq_compl_all_cqes_lock() (Bart).
- Add Reviewed-by tags from Peter and Bart.
Differences from v1:
- Split into a two-patch series separating ring sweep safety from per-request tagset iteration.
- Extract ufshcd_mcq_compl_cqe() to skip empty slots without double CQE checks.
- Decouple hardware queue polling/sweeping for both force_compl paths and remove ufshcd_mcq_compl_one().
Tested: QEMU ARM64 MCQ/SDB host reset and I/O without CQE error logs.
Stanley Jhu (2):
scsi: ufs: core: Avoid unsafe MMIO reads in
ufshcd_mcq_compl_all_cqes_lock()
scsi: ufs: core: Decouple CQ sweep from request iterator in MCQ
drivers/ufs/core/ufs-mcq.c | 30 +++++++++++++++++------------- drivers/ufs/core/ufshcd.c | 31 ++++++++++++------------------- 2 files changed, 29 insertions(+), 32 deletions(-)
base-commit: f09d2c7485b32adb82336d0d748935c8237a649e
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
PATCH v3 1/2 scsi: ufs: core: Avoid unsafe MMIO reads in ufshcd_mcq_compl_all_cqes_lock()
During MCQ host reset, ufshcd_host_reset_and_restore() stops the host controller via ufshcd_hba_stop() (HCE = 0) before calling ufshcd_complete_requests(hba, true) -> ufshcd_mcq_compl_pending_transfer(hba, true) -> ufshcd_mcq_force_compl_one() -> ufshcd_mcq_compl_all_cqes_lock(). Because ufshcd_mcq_force_compl_one() is its sole caller, ufshcd_mcq_compl_all_cqes_lock() always runs with HCE = 0.
Despite the comment above ufshcd_mcq_compl_all_cqes_lock() stating that reading CQTPy may not be safe with the controller disabled, the function still calls ufshcd_mcq_update_cq_tail_slot() at the end of its sweep:
1. Unsafe CQTPy MMIO read:
Calling ufshcd_mcq_update_cq_tail_slot() at the end of the sweep reads CQTPy over MMIO while HCE = 0, directly contradicting the function's documented contract (commit 1373df88d535 ("scsi: ufs:
core: Add a comment block above ufshcd_mcq_compl_all_cqes_lock()")) that reading CQTPy may not be safe with the controller disabled.
2. Spurious error logs on empty slots:
Sweeping all max_entries slots visits empty entries where command_desc_base_addr is 0, causing ufshcd_mcq_process_cqe() to log unguarded dev_err(hba->dev, "Abnormal CQ entry!\n") messages.
Fix both issues in ufshcd_mcq_compl_all_cqes_lock():
- Remove the ufshcd_mcq_update_cq_tail_slot() call and the redundant hwq->cq_head_slot = hwq->cq_tail_slot assignment without replacement. The two indices are already equal after the sweep: they are equal when the sweep starts, since ufshcd_mcq_poll_cqe_lock() consumes entries until cq_head_slot reaches cq_tail_slot, and the sweep advances cq_head_slot by exactly one full ring. Both indices are also reinitialized before the queue is reused.
- Extract ufshcd_mcq_compl_cqe() and invoke it only on non-empty slots during full-ring sweeps, keeping "Abnormal CQ entry!" logging strictly for unexpected empty entries in ufshcd_mcq_poll_cqe_lock().
--- drivers/ufs/core/ufs-mcq.c | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-)
drivers/ufs/core/ufs-mcq.c +17 -13
@@ -312,20 +312,24 @@ static int ufshcd_mcq_get_tag(struct ufs_hba *hba, struct cq_entry *cqe)UFSHCD_NUM_RESERVED ; } +static void ufshcd_mcq_compl_cqe (struct ufs_hba *hba,struct cq_entry *cqe) +{ +int tag =ufshcd_mcq_get_tag (hba, cqe); + +ufshcd_compl_one_cqe (hba, tag, cqe); +/* After processing the CQE, mark it as an empty (invalid) entry. */ + cqe->command_desc_base_addr =0 ; +} +static void ufshcd_mcq_process_cqe (struct ufs_hba *hba,struct ufs_hw_queue *hwq) {struct cq_entry *cqe =ufshcd_mcq_cur_cqe (hwq); -if (cqe->command_desc_base_addr) { -int tag =ufshcd_mcq_get_tag (hba, cqe); - -ufshcd_compl_one_cqe (hba, tag, cqe); -/* After processed the cqe, mark it empty (invalid) entry */ - cqe->command_desc_base_addr =0 ; - }else { +if (cqe->command_desc_base_addr) +ufshcd_mcq_compl_cqe (hba, cqe); +else dev_err (hba->dev,"Abnormal CQ entry!\n" ); - } }/*
@@ -333,7 +337,7 @@ static void ufshcd_mcq_process_cqe(struct ufs_hba *hba,* controller disabled (HCE = 0). Reading host controller registers, e.g. the * CQ tail pointer (CQTPy), may not be safe with the host controller disabled. * Hence, iterate over all completion queue entries. This won't result in -* double completions because ufshcd_mcq_process_cqe() clears a CQE after it +* double completions because ufshcd_mcq_compl_cqe() clears a CQE after it * has been processed. */ void ufshcd_mcq_compl_all_cqes_lock(struct ufs_hba *hba,
@@ -344,13 +348,13 @@ void ufshcd_mcq_compl_all_cqes_lock(struct ufs_hba *hba,spin_lock_irqsave (&hwq->cq_lock, flags);while (entries >0 ) { -ufshcd_mcq_process_cqe (hba, hwq); +struct cq_entry *cqe =ufshcd_mcq_cur_cqe (hwq); + +if (cqe->command_desc_base_addr) +ufshcd_mcq_compl_cqe (hba, cqe);ufshcd_mcq_inc_cq_head_slot (hwq); entries--; } - -ufshcd_mcq_update_cq_tail_slot (hwq); - hwq->cq_head_slot = hwq->cq_tail_slot;spin_unlock_irqrestore (&hwq->cq_lock, flags); }
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Re: [PATCH v3 1/2] scsi: ufs: core: Avoid unsafe MMIO reads in ufshcd_mcq_compl_all_cqes_lock()
Quoted 11 lines · Stanley Jhu
Fix both issues in
ufshcd_mcq_compl_all_cqes_lock():
- Remove theufshcd_mcq_update_cq_tail_slot()call and the redundanthwq->cq_head_slot=hwq->cq_tail_slotassignment without replacement. The two indices are already equal after the sweep: they are equal when the sweep starts, sinceufshcd_mcq_poll_cqe_lock()consumes entries untilcq_head_slotreachescq_tail_slot, and the sweep advancescq_head_slotby exactly one full ring. Both indices are also reinitialized before the queue is reused.
- Extractufshcd_mcq_compl_cqe()and invoke it only on non-empty slots during full-ring sweeps, keeping "Abnormal CQ entry!" logging strictly for unexpected empty entries inufshcd_mcq_poll_cqe_lock().
PATCH v3 2/2 scsi: ufs: core: Decouple CQ sweep from request iterator in MCQ
In reply to Stanley JhuIn MCQ mode, ufshcd_mcq_compl_pending_transfer() uses blk_mq_tagset_busy_iter() to iterate over busy requests during error recovery and host reset. However, both iterator callbacks perform whole-queue operations redundantly for each visited request:
- force_compl == true: ufshcd_mcq_force_compl_one() calls ufshcd_mcq_compl_all_cqes_lock() on every busy request, sweeping the entire completion ring (hwq->max_entries slots) once per active request under spin_lock_irqsave even though the first sweep already cleared all completion entries.
- force_compl == false: ufshcd_mcq_compl_one() acquires cq_lock and polls CQTPy over MMIO via ufshcd_mcq_poll_cqe_lock() for every busy request without doing any per-request work.
Sweep or poll each hardware queue (hba->uhq[i]) once at the start of ufshcd_mcq_compl_pending_transfer(). When force_compl is true, run blk_mq_tagset_busy_iter() afterward to complete residual in-flight requests with DID_REQUEUE, and remove the now-unused ufshcd_mcq_compl_one() callback.
--- drivers/ufs/core/ufshcd.c | 31 ++++++++++++------------------- 1 file changed, 12 insertions(+), 19 deletions(-)
drivers/ufs/core/ufshcd.c +12 -19
@@ -6068,8 +6068,6 @@ static bool ufshcd_mcq_force_compl_one(struct request *rq, void *priv)if (blk_mq_is_reserved_rq (rq) || !hwq)return true; -ufshcd_mcq_compl_all_cqes_lock (hba, hwq); -/* * For those cmds of which the cqes are not present in the cq, complete * them explicitly.
@@ -6085,19 +6083,6 @@ static bool ufshcd_mcq_force_compl_one(struct request *rq, void *priv)return true; } -static bool ufshcd_mcq_compl_one (struct request *rq,void *priv) -{ -struct scsi_device *sdev = rq->q->queuedata; -struct Scsi_Host *shost = sdev->host; -struct ufs_hba *hba =shost_priv (shost); -struct ufs_hw_queue *hwq =ufshcd_mcq_req_to_hwq (hba, rq); - -if (!blk_mq_is_reserved_rq (rq) && hwq) -ufshcd_mcq_poll_cqe_lock (hba, hwq); - -return true; -} -/** * ufshcd_mcq_compl_pending_transfer - MCQ mode function. It is * invoked from the error handler context or ufshcd_host_reset_and_restore()
@@ -6112,10 +6097,18 @@ static bool ufshcd_mcq_compl_one(struct request *rq, void *priv)static void ufshcd_mcq_compl_pending_transfer (struct ufs_hba *hba,bool force_compl) { -blk_mq_tagset_busy_iter (&hba->host->tag_set, - force_compl ? ufshcd_mcq_force_compl_one : - ufshcd_mcq_compl_one, -NULL ); +int i; + +for (i =0 ; i < hba->nr_hw_queues; i++) { +if (force_compl) +ufshcd_mcq_compl_all_cqes_lock (hba, &hba->uhq[i]); +else +ufshcd_mcq_poll_cqe_lock (hba, &hba->uhq[i]); + } + +if (force_compl) +blk_mq_tagset_busy_iter (&hba->host->tag_set, + ufshcd_mcq_force_compl_one,NULL ); }/**
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Re: [PATCH v3 2/2] scsi: ufs: core: Decouple CQ sweep from request iterator in MCQ
Quoted 21 lines · Stanley Jhu
In MCQ mode,
ufshcd_mcq_compl_pending_transfer()usesblk_mq_tagset_busy_iter()to iterate over busy requests during error recovery and host reset. However, both iterator callbacks perform whole-queue operations redundantly for each visited request:-
force_compl== true:ufshcd_mcq_force_compl_one()callsufshcd_mcq_compl_all_cqes_lock()on every busy request, sweeping the
entire completion ring (hwq->max_entriesslots) once per active request underspin_lock_irqsaveeven though the first sweep already cleared all completion entries.
-force_compl== false:ufshcd_mcq_compl_one()acquirescq_lockand polls CQTPy over MMIO viaufshcd_mcq_poll_cqe_lock()for every busy request without doing any per-request work.Sweep or poll each hardware queue (hba->uhq[i]) once at the start of
ufshcd_mcq_compl_pending_transfer(). Whenforce_complis true, runblk_mq_tagset_busy_iter()afterward to complete residual in-flight requests withDID_REQUEUE, and remove the now-unusedufshcd_mcq_compl_one()callback.
Quoted 5 lines
Fixes ab248643d3d6 ("scsi: ufs: core: Add error handling for MCQmode")
Reviewed-by Bart Van Assche <[email protected]>Signed-off-by Stanley Jhu <[email protected]>
Quoted 14 lines · Stanley Jhu
During Multi-Circular Queue (MCQ) error recovery and host reset,
ufshcd_mcq_compl_pending_transfer()sweeps or polls completion queues to reap pending transfers. Two bugs exist in this path:1. Unsafe MMIO read and spurious errors while HCE = 0 (Patch 1/2):
ufshcd_host_reset_and_restore()stops the controller (HCE = 0) before callingufshcd_mcq_compl_all_cqes_lock(). Callingufshcd_mcq_update_cq_tail_slot()at the end of the sweep reads CQTPy over MMIO while HCE = 0, directly contradicting the function's own documented contract that reading host controller registers may not be safe when the controller is disabled. In addition, passing expected empty slots during a full-ring sweep intoufshcd_mcq_process_cqe()prints spurious "Abnormal CQ entry!" errors.
Quoted 1 line
[...]
Applied to 7.3/scsi-fixes, thanks!
[1/2] scsi: ufs: core: Avoid unsafe MMIO reads in ufshcd_mcq_compl_all_cqes_lock()
https://git.kernel.org/mkp/scsi/c/375f3a5691dd [2/2] scsi: ufs: core: Decouple CQ sweep from request iterator in MCQ https://git.kernel.org/mkp/scsi/c/469fa055664b
Signature · 1 line
Martin K. Petersen