scsi: ufs: core: Factor hba->outstanding_reqs helper and fix double completions
Subsystems (3)
v3 cover scsi: ufs: core: Factor hba->outstanding_reqs helper and fix double completions
In Single Doorbell (SDB) mode, multiple completion and error handling paths can race on the same transfer request when bits in hba->outstanding_reqs are not claimed atomically with completing the request. A stale bit in hba->outstanding_reqs lets a later poll or reset complete the request again and unmap its DMA buffers twice.
Patch 1 removes a WARN_ONCE() in ufshcd_poll() whose condition is always false.
Patch 2 adds __ufshcd_clear_outstanding_reqs() and ufshcd_clear_outstanding_reqs() to share the bit-claim logic across ufshcd_poll() and ufshcd_abort() with no functional change.
Patch 3 uses the helper in ufshcd_clear_lu_cmds() to fix a double completion during LU reset.
Patch 4 converts ufshcd_eh_timed_out() to the helper as a cleanup.
Patch 5 uses the helper return value in ufshcd_eh_timed_out() to prevent releasing driver resources for a request already completed by another context.
Patch 6 uses the helper in ufshcd_abort() to fix a double release in the missing-interrupt path.
Patch 7 skips releasing reserved requests at the end of ufshcd_abort().
This series is based on mkp/for-next (db8b4d2bf2d3).
Tested on QEMU ARM64 UFS in SDB mode across LU reset, abort, and PM timeout paths without double DMA unmaps.
Changes since v2:
- Move WARN_ONCE() removal to 1/7 and use __must_hold()/guard() (Bart).
- Add patch 5 to fix a double release in ufshcd_eh_timed_out().
- Add patch 6 to fix a double release in ufshcd_abort() (Bean, Bart).
- Add patch 7 to skip releasing reserved requests in ufshcd_abort().
- Link to v2: https://lore.kernel.org/r/[email protected]
Stanley Jhu (7):
scsi: ufs: core: Remove an always-false WARN_ONCE()
scsi: ufs: core: Add a helper for clearing hba->outstanding_reqs
scsi: ufs: core: Fix double completion in ufshcd_clear_lu_cmds()
scsi: ufs: core: Use ufshcd_clear_outstanding_reqs() in
ufshcd_eh_timed_out()
scsi: ufs: core: Fix double release in ufshcd_eh_timed_out()
scsi: ufs: core: Fix double release in ufshcd_abort()
scsi: ufs: core: Do not release a reserved request in ufshcd_abort()
drivers/ufs/core/ufshcd.c | 56 ++++++++++++++++++++++++--------------- 1 file changed, 35 insertions(+), 21 deletions(-)
base-commit: db8b4d2bf2d3e02b4ff78bbf69d00e11795840de
Signature · 1 line
2.56.0.212.g129ede20d9-goog
PATCH v3 1/7 scsi: ufs: core: Remove an always-false WARN_ONCE()
The WARN_ONCE() in ufshcd_poll() can never fire. completed_reqs is set to ~tr_doorbell & hba->outstanding_reqs on the line above, and hba->outstanding_reqs cannot change in between because ufshcd_poll() holds hba->outstanding_lock. So completed_reqs & ~hba->outstanding_reqs is always zero. Remove it. No functional change.
--- drivers/ufs/core/ufshcd.c | 3 --- 1 file changed, 3 deletions(-)
drivers/ufs/core/ufshcd.c +0 -3
@@ -6072,9 +6072,6 @@ static int ufshcd_poll(struct Scsi_Host *shost, unsigned int queue_num)spin_lock_irqsave (&hba->outstanding_lock, flags); tr_doorbell =ufshcd_readl (hba,REG_UTP_TRANSFER_REQ_DOOR_BELL ); completed_reqs = ~tr_doorbell & hba->outstanding_reqs; -WARN_ONCE (completed_reqs & ~hba->outstanding_reqs, -"completed: %#lx; outstanding: %#lx\n" , completed_reqs, - hba->outstanding_reqs);if (queue_num ==UFSHCD_POLL_FROM_INTERRUPT_CONTEXT ) {/* Do not complete polled requests from interrupt context. */ ufshcd_clear_polled (hba, &completed_reqs);
Signature · 1 line
2.56.0.212.g129ede20d9-goog
Re: [PATCH v3 1/7] scsi: ufs: core: Remove an always-false WARN_ONCE()
Quoted 5 lines · Stanley Jhu
The
WARN_ONCE()inufshcd_poll()can never fire.completed_reqsis set to ~tr_doorbell &hba->outstanding_reqson the line above, andhba->outstanding_reqscannot change in between becauseufshcd_poll()holdshba->outstanding_lock. Socompleted_reqs& ~hba->outstanding_reqs is always zero. Remove it. No functional change.
Patch 2/7 has a Cc: stable tag so this patch should have that tag too.
Anyway:
PATCH v3 2/7 scsi: ufs: core: Add a helper for clearing hba->outstanding_reqs
In reply to Stanley Jhuufshcd_poll() and ufshcd_abort() each have their own copy of the code that clears bits in hba->outstanding_reqs under hba->outstanding_lock.
Move that code into ufshcd_clear_outstanding_reqs() and its locked variant __ufshcd_clear_outstanding_reqs(), and use them in both functions. No functional change.
---
v3: Add __must_hold() and use guard(spinlock_irqsave) (Bart Van Assche).
drivers/ufs/core/ufshcd.c | 39 +++++++++++++++++++++++++++------------ 1 file changed, 27 insertions(+), 12 deletions(-)
drivers/ufs/core/ufshcd.c +27 -12
@@ -6052,6 +6052,29 @@ static void ufshcd_clear_polled(struct ufs_hba *hba, } } +static unsigned long __ufshcd_clear_outstanding_reqs (struct ufs_hba *hba, +unsigned long mask, +bool from_irq) + __must_hold(&hba->outstanding_lock) +{ +unsigned long completed_reqs = mask & hba->outstanding_reqs; + +if (from_irq) { +/* Do not complete polled requests from interrupt context. */ +ufshcd_clear_polled (hba, &completed_reqs); + } + hba->outstanding_reqs &= ~completed_reqs; + +return completed_reqs; +} + +static unsigned long ufshcd_clear_outstanding_reqs (struct ufs_hba *hba, +unsigned long mask) +{ +guard (spinlock_irqsave)(&hba->outstanding_lock); +return __ufshcd_clear_outstanding_reqs (hba, mask, false); +} +/* * Return: > 0 if one or more commands have been completed or 0 if no * requests have been completed.
@@ -6071,12 +6094,9 @@ static int ufshcd_poll(struct Scsi_Host *shost, unsigned int queue_num)spin_lock_irqsave (&hba->outstanding_lock, flags); tr_doorbell =ufshcd_readl (hba,REG_UTP_TRANSFER_REQ_DOOR_BELL ); - completed_reqs = ~tr_doorbell & hba->outstanding_reqs; -if (queue_num ==UFSHCD_POLL_FROM_INTERRUPT_CONTEXT ) { -/* Do not complete polled requests from interrupt context. */ -ufshcd_clear_polled (hba, &completed_reqs); - } - hba->outstanding_reqs &= ~completed_reqs; + completed_reqs = +__ufshcd_clear_outstanding_reqs (hba, ~tr_doorbell, + queue_num ==UFSHCD_POLL_FROM_INTERRUPT_CONTEXT );spin_unlock_irqrestore (&hba->outstanding_lock, flags);if (completed_reqs)
@@ -8111,7 +8131,6 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)struct ufshcd_lrb *lrbp =scsi_cmd_priv (cmd);unsigned long flags;int err =FAILED ; -bool outstanding;u32 reg;ufshcd_hold (hba);
@@ -8206,11 +8225,7 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)* Clear the corresponding bit from outstanding_reqs since the command * has been aborted successfully. */ -spin_lock_irqsave (&hba->outstanding_lock, flags); - outstanding =__test_and_clear_bit (tag, &hba->outstanding_reqs); -spin_unlock_irqrestore (&hba->outstanding_lock, flags); - -if (outstanding) +if (ufshcd_clear_outstanding_reqs (hba,BIT (tag)))ufshcd_release_scsi_cmd (hba, cmd); err =SUCCESS ;
Signature · 1 line
2.56.0.212.g129ede20d9-goog
Bart Van Assche
reviewerRe: [PATCH v3 2/7] scsi: ufs: core: Add a helper for clearing hba->outstanding_reqs
Quoted 12 lines · Stanley Jhu
drivers/ufs/core/ufshcd.c +12
-0+static unsigned long __ufshcd_clear_outstanding_reqs (struct ufs_hba *hba, +unsigned long mask, +bool from_irq) + __must_hold(&hba->outstanding_lock) +{ +unsigned long completed_reqs = mask & hba->outstanding_reqs; + +if (from_irq) { +/* Do not complete polled requests from interrupt context. */ +ufshcd_clear_polled (hba, &completed_reqs); + } + hba->outstanding_reqs &= ~completed_reqs;
drivers/ufs/core/ufshcd.c static void ufshcd_clear_polled(struct ufs_hba *hba, · line 6076
Quoted 10 lines
drivers/ufs/core/ufshcd.c +10
-0+ +return completed_reqs; +} + +static unsigned long ufshcd_clear_outstanding_reqs (struct ufs_hba *hba, +unsigned long mask) +{ +guard (spinlock_irqsave)(&hba->outstanding_lock); +return __ufshcd_clear_outstanding_reqs (hba, mask, false); +}
Please add a short comment above the above two functions that explains the meaning of the return value.
Quoted 6 lines
/*
* Return: > 0 if one or more commands have been completed or 0 if no
* requests have been completed.drivers/ufs/core/ufshcd.c +0
-0@@ -6071,12 +6094,9 @@ static int ufshcd_poll(struct Scsi_Host *shost, unsigned int queue_num)spin_lock_irqsave (&hba->outstanding_lock, flags);
drivers/ufs/core/ufshcd.c static int ufshcd_poll(struct Scsi_Host *shost, unsigned int queue_num) · line 6099
Quoted 10 lines
drivers/ufs/core/ufshcd.c +3
-6tr_doorbell =ufshcd_readl (hba,REG_UTP_TRANSFER_REQ_DOOR_BELL ); - completed_reqs = ~tr_doorbell & hba->outstanding_reqs; -if (queue_num ==UFSHCD_POLL_FROM_INTERRUPT_CONTEXT ) { -/* Do not complete polled requests from interrupt context. */ -ufshcd_clear_polled (hba, &completed_reqs); - } - hba->outstanding_reqs &= ~completed_reqs; + completed_reqs = +__ufshcd_clear_outstanding_reqs (hba, ~tr_doorbell, + queue_num ==UFSHCD_POLL_FROM_INTERRUPT_CONTEXT );
The above line is about 98 columns wide. That's too much - all other code in this driver is 80 columns wide. One way to address this is by introducing a local "from_irq" boolean variable:
+ from_irq = queue_num == UFSHCD_POLL_FROM_INTERRUPT_CONTEXT;
+ completed_reqs = __ufshcd_clear_outstanding_reqs(hba, ~tr_doorbell,
+ from_irq);
Otherwise this patch looks good to me.
Bart.
PATCH v3 3/7 scsi: ufs: core: Fix double completion in ufshcd_clear_lu_cmds()
In reply to Stanley JhuIn SDB mode, ufshcd_clear_lu_cmds() clears the request slot in the controller and completes the request, but leaves its bit set in hba->outstanding_reqs. ufshcd_poll() treats a request whose bit is set in hba->outstanding_reqs and clear in the doorbell register as finished, so the next ufshcd_poll() completes the same request again and unmaps its DMA buffers twice.
Clear the bit with ufshcd_clear_outstanding_reqs() before completing the request, and complete it only if the bit was still set. If ufshcd_poll() has already completed the request, the bit is already clear and ufshcd_clear_lu_cmds() skips it.
--- drivers/ufs/core/ufshcd.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/ufs/core/ufshcd.c +2 -1
@@ -7978,7 +7978,8 @@ static bool ufshcd_clear_lu_cmds(struct request *req, void *priv)return true; } -ufshcd_compl_one_cqe (hba, tag,NULL ); +if (ufshcd_clear_outstanding_reqs (hba,BIT (tag))) +ufshcd_compl_one_cqe (hba, tag,NULL );return true; }
Signature · 1 line
2.56.0.212.g129ede20d9-goog
Re: [PATCH v3 3/7] scsi: ufs: core: Fix double completion in ufshcd_clear_lu_cmds()
Quoted 6 lines · Stanley Jhu
In SDB mode,
ufshcd_clear_lu_cmds()clears the request slot in the controller and completes the request, but leaves its bit set inhba->outstanding_reqs.ufshcd_poll()treats a request whose bit is set inhba->outstanding_reqsand clear in the doorbell register as finished, so the nextufshcd_poll()completes the same request again and unmaps its DMA buffers twice.
PATCH v3 4/7 scsi: ufs: core: Use ufshcd_clear_outstanding_reqs() in ufshcd_eh_timed_out()
In reply to Stanley Jhuufshcd_eh_timed_out() has its own copy of the code that clears the timed-out tag in hba->outstanding_reqs. Use ufshcd_clear_outstanding_reqs() instead. No functional change.
--- drivers/ufs/core/ufshcd.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
drivers/ufs/core/ufshcd.c +1 -4
@@ -9651,12 +9651,9 @@ static enum scsi_timeout_action ufshcd_eh_timed_out(struct scsi_cmnd *scmd)*/ if (!test_bit (SCMD_STATE_COMPLETE , &scmd->state)) {if (!hba->mcq_enabled) { -unsigned long flags;struct request *rq =scsi_cmd_to_rq (scmd); -spin_lock_irqsave (&hba->outstanding_lock, flags); -__clear_bit (rq->tag, &hba->outstanding_reqs); -spin_unlock_irqrestore (&hba->outstanding_lock, flags); +ufshcd_clear_outstanding_reqs (hba,BIT (rq->tag)); }if (ufshcd_is_scsi_cmd (scmd)) {
Signature · 1 line
2.56.0.212.g129ede20d9-goog
PATCH v3 5/7 scsi: ufs: core: Fix double release in ufshcd_eh_timed_out()
In reply to Stanley JhuIn SDB mode, a command that times out during a power management operation can be released twice:
1. The command times out while its normal completion is running.
2. The completion path clears BIT(rq->tag) in hba->outstanding_reqs and releases the command. It has not called scsi_done() yet, so SCMD_STATE_COMPLETE is still clear.
3. ufshcd_eh_timed_out() recovers the link, sees SCMD_STATE_COMPLETE clear, clears BIT(rq->tag) in hba->outstanding_reqs without checking whether it was still set, and releases the same command a second time.
Return SCSI_EH_DONE when ufshcd_clear_outstanding_reqs() returns 0, as another path has already claimed the command.
--- drivers/ufs/core/ufshcd.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/ufs/core/ufshcd.c +2 -1
@@ -9653,7 +9653,8 @@ static enum scsi_timeout_action ufshcd_eh_timed_out(struct scsi_cmnd *scmd)if (!hba->mcq_enabled) {struct request *rq =scsi_cmd_to_rq (scmd); -ufshcd_clear_outstanding_reqs (hba,BIT (rq->tag)); +if (!ufshcd_clear_outstanding_reqs (hba,BIT (rq->tag))) +return SCSI_EH_DONE ; }if (ufshcd_is_scsi_cmd (scmd)) {
Signature · 1 line
2.56.0.212.g129ede20d9-goog
Bart Van Assche
reviewerRe: [PATCH v3 5/7] scsi: ufs: core: Fix double release in ufshcd_eh_timed_out()
Quoted 2 lines · Stanley Jhu
Fixes 01d5e237b339 ("scsi: ufs: core: Handle PM commands timeout before SCSI EH")
This patch modifies the same code as patch 4/7 while patch 4/7 does not have a Cc: stable tag. Hence, this patch can't be applied on stable kernels without patch 4/7. Please combine both patches into a single patch. Otherwise this patch looks good to me.
Thanks,
Bart.
PATCH v3 6/7 scsi: ufs: core: Fix double release in ufshcd_abort()
In reply to Stanley JhuIn SDB mode, a timed-out command whose completion interrupt has not been handled yet can be released twice:
1. The command times out and scsi_timeout() schedules ufshcd_abort().
2. The command finishes in hardware. Its doorbell bit is clear, but the completion interrupt has not been handled, so BIT(tag) is still set in hba->outstanding_reqs.
3. ufshcd_abort() sees the doorbell bit clear and calls __ufshcd_transfer_req_compl(). That releases the command but leaves BIT(tag) set in hba->outstanding_reqs. ufshcd_abort() returns FAILED.
4. The next ufshcd_poll() call, or the logical unit reset that follows the FAILED, sees BIT(tag) set in hba->outstanding_reqs and releases the same command again.
Claim BIT(tag) in hba->outstanding_reqs with ufshcd_clear_outstanding_reqs() and release the command only if ufshcd_abort() claimed it and it is a SCSI command.
--- drivers/ufs/core/ufshcd.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/ufs/core/ufshcd.c +3 -1
@@ -8176,7 +8176,9 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)dev_err (hba->dev,"%s: cmd was completed, but without a notifying intr, tag = %d" , __func__, tag); -__ufshcd_transfer_req_compl (hba,1UL << tag); +if (ufshcd_clear_outstanding_reqs (hba,BIT (tag)) && +ufshcd_is_scsi_cmd (cmd)) +ufshcd_release_scsi_cmd (hba, cmd); goto release; }
Signature · 1 line
2.56.0.212.g129ede20d9-goog
Bart Van Assche
reviewerRe: [PATCH v3 6/7] scsi: ufs: core: Fix double release in ufshcd_abort()
Quoted 5 lines · Stanley Jhu
drivers/ufs/core/ufshcd.c +0
-0
drivers/ufs/core/ufshcd.c static int ufshcd_abort(struct scsi_cmnd *cmd) · line 8183
Quoted 9 lines
drivers/ufs/core/ufshcd.c +3
-1@@ -8176,7 +8176,9 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)dev_err (hba->dev,"%s: cmd was completed, but without a notifying intr, tag = %d" , __func__, tag); -__ufshcd_transfer_req_compl (hba,1UL << tag); +if (ufshcd_clear_outstanding_reqs (hba,BIT (tag)) && +ufshcd_is_scsi_cmd (cmd)) +ufshcd_release_scsi_cmd (hba, cmd); goto release; }
This change is incomplete. "err = SUCCESS" is missing above "goto release". Without this change, the SCSI error handler will escalate error handling unnecessarily to the next level.
Bart.
Re: [PATCH v3 6/7] scsi: ufs: core: Fix double release in ufshcd_abort()
Quoted 3 lines · Bart Van Assche
This change is incomplete. "err = SUCCESS" is missing above "goto release". Without this change, the SCSI error handler will escalate error handling unnecessarily to the next level.
Fixed in v4 by setting err = SUCCESS when ufshcd_clear_outstanding_reqs() claims the tag. If the claim fails, a concurrent ufshcd_poll() may still be completing the command, so the handler keeps returning FAILED.
PATCH v3 7/7 scsi: ufs: core: Do not release a reserved request in ufshcd_abort()
In reply to Stanley JhuIn SDB mode, aborting a timed-out device management request drops a clock hold that the request never took:
1. ufshcd_dev_man_lock() takes one ufshcd_hold() for the whole device management command. ufshcd_queue_reserved_command() submits the request without taking a hold of its own.
2. The request times out and ufshcd_abort() clears its slot with ufshcd_clear_cmd().
3. ufshcd_abort() claims BIT(tag) in hba->outstanding_reqs and calls ufshcd_release_scsi_cmd(), which calls ufshcd_release() for a hold the request never took.
4. When ufshcd_dev_man_unlock() drops the hold from step 1, hba->clk_gating.active_reqs goes below zero.
Release the command only if it is a SCSI command.
--- drivers/ufs/core/ufshcd.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/ufs/core/ufshcd.c +2 -1
@@ -8228,7 +8228,8 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)* Clear the corresponding bit from outstanding_reqs since the command * has been aborted successfully. */ -if (ufshcd_clear_outstanding_reqs (hba,BIT (tag))) +if (ufshcd_clear_outstanding_reqs (hba,BIT (tag)) && +ufshcd_is_scsi_cmd (cmd))ufshcd_release_scsi_cmd (hba, cmd); err =SUCCESS ;
Signature · 1 line
2.56.0.212.g129ede20d9-goog
Re: [PATCH v3 7/7] scsi: ufs: core: Do not release a reserved request in ufshcd_abort()
Quoted 5 lines · Stanley Jhu
drivers/ufs/core/ufshcd.c +0
-0
drivers/ufs/core/ufshcd.c static int ufshcd_abort(struct scsi_cmnd *cmd) · line 8235
Quoted 9 lines
drivers/ufs/core/ufshcd.c +2
-1@@ -8228,7 +8228,8 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)* Clear the corresponding bit from outstanding_reqs since the command * has been aborted successfully. */ -if (ufshcd_clear_outstanding_reqs (hba,BIT (tag))) +if (ufshcd_clear_outstanding_reqs (hba,BIT (tag)) && +ufshcd_is_scsi_cmd (cmd))ufshcd_release_scsi_cmd (hba, cmd); err =SUCCESS ;