scsi: ufs: core: Factor hba->outstanding_reqs helper and fix double completion
Subsystems (3)
v2 cover scsi: ufs: core: Factor hba->outstanding_reqs helper and fix double completion
In Single Doorbell (SDB) mode, ufshcd_clear_lu_cmds() clears a transfer request slot in the controller and completes the request without claiming its bit in hba->outstanding_reqs. A subsequent ufshcd_poll() call sees the stale bit with its doorbell bit cleared and completes the request a second time, causing a double DMA unmap.
Patch 1 adds __ufshcd_clear_outstanding_reqs() and ufshcd_clear_outstanding_reqs() to share the bit-claim logic across ufshcd_poll() and ufshcd_abort() with no functional change. Patch 2 uses the new helper in ufshcd_clear_lu_cmds() to fix the SDB double completion. Patch 3 uses the new helper in ufshcd_eh_timed_out(). It is a follow-on cleanup and is not needed for the fix. Patch 4 removes a WARN_ONCE() whose condition is always false.
This series is based on mkp/for-next (db8b4d2bf2d3).
Tested on QEMU ARM64 UFS in SDB and MCQ modes across LU reset, polled and interrupt I/O, and task abort without double completion or DMA unmap warnings.
Changes since v1:
- Add a helper for clearing hba->outstanding_reqs and use it in ufshcd_poll(), ufshcd_abort(), and ufshcd_clear_lu_cmds()
(Bart Van Assche).
- Convert ufshcd_eh_timed_out() to use the new helper in patch 3.
- Remove an always-false WARN_ONCE() in patch 4.
- Link to v1: https://lore.kernel.org/r/[email protected]
Stanley Jhu (4):
scsi: ufs: core: Add a helper for clearing hba->outstanding_reqs
scsi: ufs: core: Fix double completion in ufshcd_clear_lu_cmds()
scsi: ufs: core: Use ufshcd_clear_outstanding_reqs() in
ufshcd_eh_timed_out()
scsi: ufs: core: Remove an always-false WARN_ONCE()
drivers/ufs/core/ufshcd.c | 55 +++++++++++++++++++++++++-------------- 1 file changed, 35 insertions(+), 20 deletions(-)
base-commit: db8b4d2bf2d3e02b4ff78bbf69d00e11795840de
Signature · 1 line
2.56.0.176.g3575e65682-goog
PATCH v2 1/4 scsi: ufs: core: Add a helper for clearing hba->outstanding_reqs
ufshcd_poll() and ufshcd_abort() each have their own copy of the code that clears bits in hba->outstanding_reqs under hba->outstanding_lock.
Move that code into ufshcd_clear_outstanding_reqs() and its locked variant __ufshcd_clear_outstanding_reqs(), and use them in both functions. No functional change.
--- drivers/ufs/core/ufshcd.c | 50 +++++++++++++++++++++++++++------------ 1 file changed, 35 insertions(+), 15 deletions(-)
drivers/ufs/core/ufshcd.c +35 -15
@@ -6052,6 +6052,37 @@ static void ufshcd_clear_polled(struct ufs_hba *hba, } } +static unsigned long __ufshcd_clear_outstanding_reqs (struct ufs_hba *hba, +unsigned long mask, +bool from_irq) +{ +unsigned long completed_reqs = mask & hba->outstanding_reqs; + +lockdep_assert_held (&hba->outstanding_lock); +WARN_ONCE (completed_reqs & ~hba->outstanding_reqs, +"completed: %#lx; outstanding: %#lx\n" , completed_reqs, + hba->outstanding_reqs); +if (from_irq) { +/* Do not complete polled requests from interrupt context. */ +ufshcd_clear_polled (hba, &completed_reqs); + } + hba->outstanding_reqs &= ~completed_reqs; + +return completed_reqs; +} + +static unsigned long ufshcd_clear_outstanding_reqs (struct ufs_hba *hba, +unsigned long mask) +{ +unsigned long completed_reqs, flags; + +spin_lock_irqsave (&hba->outstanding_lock, flags); + completed_reqs =__ufshcd_clear_outstanding_reqs (hba, mask, false); +spin_unlock_irqrestore (&hba->outstanding_lock, flags); + +return completed_reqs; +} +/* * Return: > 0 if one or more commands have been completed or 0 if no * requests have been completed.
@@ -6071,15 +6102,9 @@ static int ufshcd_poll(struct Scsi_Host *shost, unsigned int queue_num)spin_lock_irqsave (&hba->outstanding_lock, flags); tr_doorbell =ufshcd_readl (hba,REG_UTP_TRANSFER_REQ_DOOR_BELL ); - completed_reqs = ~tr_doorbell & hba->outstanding_reqs; -WARN_ONCE (completed_reqs & ~hba->outstanding_reqs, -"completed: %#lx; outstanding: %#lx\n" , completed_reqs, - hba->outstanding_reqs); -if (queue_num ==UFSHCD_POLL_FROM_INTERRUPT_CONTEXT ) { -/* Do not complete polled requests from interrupt context. */ -ufshcd_clear_polled (hba, &completed_reqs); - } - hba->outstanding_reqs &= ~completed_reqs; + completed_reqs = +__ufshcd_clear_outstanding_reqs (hba, ~tr_doorbell, + queue_num ==UFSHCD_POLL_FROM_INTERRUPT_CONTEXT );spin_unlock_irqrestore (&hba->outstanding_lock, flags);if (completed_reqs)
@@ -8114,7 +8139,6 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)struct ufshcd_lrb *lrbp =scsi_cmd_priv (cmd);unsigned long flags;int err =FAILED ; -bool outstanding;u32 reg;ufshcd_hold (hba);
@@ -8209,11 +8233,7 @@ static int ufshcd_abort(struct scsi_cmnd *cmd)* Clear the corresponding bit from outstanding_reqs since the command * has been aborted successfully. */ -spin_lock_irqsave (&hba->outstanding_lock, flags); - outstanding =__test_and_clear_bit (tag, &hba->outstanding_reqs); -spin_unlock_irqrestore (&hba->outstanding_lock, flags); - -if (outstanding) +if (ufshcd_clear_outstanding_reqs (hba,BIT (tag)))ufshcd_release_scsi_cmd (hba, cmd); err =SUCCESS ;
Signature · 1 line
2.56.0.176.g3575e65682-goog
Bart Van Assche
reviewerRe: [PATCH v2 1/4] scsi: ufs: core: Add a helper for clearing hba->outstanding_reqs
Quoted 16 lines · Stanley Jhu
drivers/ufs/core/ufshcd.c +8
-0@@ -6052,6 +6052,37 @@ static void ufshcd_clear_polled(struct ufs_hba *hba, } } +static unsigned long __ufshcd_clear_outstanding_reqs (struct ufs_hba *hba, +unsigned long mask, +bool from_irq) +{ +unsigned long completed_reqs = mask & hba->outstanding_reqs; + +lockdep_assert_held (&hba->outstanding_lock); + WARN_ONCE(completed_reqs & ~hba->outstanding_reqs,
drivers/ufs/core/ufshcd.c static void ufshcd_clear_polled(struct ufs_hba *hba, · line 6072
Quoted 10 lines
drivers/ufs/core/ufshcd.c +10
-0+"completed: %#lx; outstanding: %#lx\n" , completed_reqs, + hba->outstanding_reqs); +if (from_irq) { +/* Do not complete polled requests from interrupt context. */ +ufshcd_clear_polled (hba, &completed_reqs); + } + hba->outstanding_reqs &= ~completed_reqs; + +return completed_reqs; +}
Please use __must_hold() instead of lockdep_assert_held(). __must_hold() is evaluated at compile time (by Clang) while lockdep_assert_held() is evaluated at run-time. If you don't want to depend on Clang, feel free to use both.
drivers/ufs/core/ufshcd.c static void ufshcd_clear_polled(struct ufs_hba *hba, · line 6084
Quoted 11 lines
drivers/ufs/core/ufshcd.c +11
-0+static unsigned long ufshcd_clear_outstanding_reqs (struct ufs_hba *hba, +unsigned long mask) +{ +unsigned long completed_reqs, flags; + +spin_lock_irqsave (&hba->outstanding_lock, flags); + completed_reqs =__ufshcd_clear_outstanding_reqs (hba, mask, false); +spin_unlock_irqrestore (&hba->outstanding_lock, flags); + +return completed_reqs; +}
The seven-line body of the above function can be reduced to two lines by using guard():
static unsigned long ufshcd_clear_outstanding_reqs(struct ufs_hba *hba,
unsigned long mask)
{
guard(spinlock_irqsave)(&hba->outstanding_lock);
return __ufshcd_clear_outstanding_reqs(hba, mask, false);
}Otherwise this patch looks good to me.
Thanks,
Bart.
PATCH v2 2/4 scsi: ufs: core: Fix double completion in ufshcd_clear_lu_cmds()
In reply to Stanley JhuIn SDB mode, ufshcd_clear_lu_cmds() clears the request slot in the controller and completes the request, but leaves its bit set in hba->outstanding_reqs. ufshcd_poll() treats a request whose bit is set in hba->outstanding_reqs and clear in the doorbell register as finished, so the next ufshcd_poll() completes the same request again and unmaps its DMA buffers twice.
Clear the bit with ufshcd_clear_outstanding_reqs() before completing the request, and complete it only if the bit was still set. If ufshcd_poll() has already completed the request, the bit is already clear and ufshcd_clear_lu_cmds() skips it.
--- drivers/ufs/core/ufshcd.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/ufs/core/ufshcd.c +2 -1
@@ -7986,7 +7986,8 @@ static bool ufshcd_clear_lu_cmds(struct request *req, void *priv)return true; } -ufshcd_compl_one_cqe (hba, tag,NULL ); +if (ufshcd_clear_outstanding_reqs (hba,BIT (tag))) +ufshcd_compl_one_cqe (hba, tag,NULL );return true; }
Signature · 1 line
2.56.0.176.g3575e65682-goog
PATCH v2 3/4 scsi: ufs: core: Use ufshcd_clear_outstanding_reqs() in ufshcd_eh_timed_out()
In reply to Stanley Jhuufshcd_eh_timed_out() has its own copy of the code that clears the timed-out tag in hba->outstanding_reqs. Use ufshcd_clear_outstanding_reqs() instead. No functional change.
--- drivers/ufs/core/ufshcd.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
drivers/ufs/core/ufshcd.c +1 -4
@@ -9659,12 +9659,9 @@ static enum scsi_timeout_action ufshcd_eh_timed_out(struct scsi_cmnd *scmd)*/ if (!test_bit (SCMD_STATE_COMPLETE , &scmd->state)) {if (!hba->mcq_enabled) { -unsigned long flags;struct request *rq =scsi_cmd_to_rq (scmd); -spin_lock_irqsave (&hba->outstanding_lock, flags); -__clear_bit (rq->tag, &hba->outstanding_reqs); -spin_unlock_irqrestore (&hba->outstanding_lock, flags); +ufshcd_clear_outstanding_reqs (hba,BIT (rq->tag)); }if (ufshcd_is_scsi_cmd (scmd)) {
Signature · 1 line
2.56.0.176.g3575e65682-goog
PATCH v2 4/4 scsi: ufs: core: Remove an always-false WARN_ONCE()
In reply to Stanley JhuThe WARN_ONCE() in __ufshcd_clear_outstanding_reqs() can never fire. completed_reqs is set to mask & hba->outstanding_reqs on the line above, and hba->outstanding_reqs cannot change in between because the caller holds hba->outstanding_lock. So completed_reqs & ~hba->outstanding_reqs is always zero. Remove it. No functional change.
--- drivers/ufs/core/ufshcd.c | 3 --- 1 file changed, 3 deletions(-)
drivers/ufs/core/ufshcd.c +0 -3
@@ -6059,9 +6059,6 @@ static unsigned long __ufshcd_clear_outstanding_reqs(struct ufs_hba *hba,unsigned long completed_reqs = mask & hba->outstanding_reqs;lockdep_assert_held (&hba->outstanding_lock); -WARN_ONCE (completed_reqs & ~hba->outstanding_reqs, -"completed: %#lx; outstanding: %#lx\n" , completed_reqs, - hba->outstanding_reqs);if (from_irq) {/* Do not complete polled requests from interrupt context. */ ufshcd_clear_polled (hba, &completed_reqs);
Signature · 1 line
2.56.0.176.g3575e65682-goog
Bart Van Assche
reviewerRe: [PATCH v2 4/4] scsi: ufs: core: Remove an always-false WARN_ONCE()
Quoted 5 lines · Stanley Jhu
drivers/ufs/core/ufshcd.c +0
-0
drivers/ufs/core/ufshcd.c static unsigned long __ufshcd_clear_outstanding_reqs(struct ufs_hba *hba, · line 6064
Quoted 9 lines
drivers/ufs/core/ufshcd.c +0
-3@@ -6059,9 +6059,6 @@ static unsigned long __ufshcd_clear_outstanding_reqs(struct ufs_hba *hba,unsigned long completed_reqs = mask & hba->outstanding_reqs;lockdep_assert_held (&hba->outstanding_lock); -WARN_ONCE (completed_reqs & ~hba->outstanding_reqs, -"completed: %#lx; outstanding: %#lx\n" , completed_reqs, - hba->outstanding_reqs);if (from_irq) {/* Do not complete polled requests from interrupt context. */ ufshcd_clear_polled (hba, &completed_reqs);
This patch should either be the first patch in this series or should be squashed into patch 1/4 from this series.
Thanks,
Bart.
Quoted 29 lines · Stanley Jhu
In Single Doorbell (SDB) mode,
ufshcd_clear_lu_cmds()clears a transfer request slot in the controller and completes the request without claiming its bit inhba->outstanding_reqs. A subsequentufshcd_poll()call sees the stale bit with its doorbell bit cleared and completes the request a second time, causing a double DMA unmap.Patch 1 adds
__ufshcd_clear_outstanding_reqs()andufshcd_clear_outstanding_reqs()to share the bit-claim logic acrossufshcd_poll()andufshcd_abort()with no functional change. Patch 2 uses the new helper inufshcd_clear_lu_cmds()to fix the SDB double completion. Patch 3 uses the new helper inufshcd_eh_timed_out(). It is a follow-on cleanup and is not needed for the fix. Patch 4 removes aWARN_ONCE()whose condition is always false.This series is based on mkp/for-next (
db8b4d2bf2d3).Tested on QEMU ARM64 UFS in SDB and MCQ modes across LU reset, polled and interrupt I/O, and task abort without double completion or DMA unmap warnings.
Changes since v1:
- Add a helper for clearinghba->outstanding_reqsand use it inufshcd_poll(),ufshcd_abort(), andufshcd_clear_lu_cmds()(Bart Van Assche).
- Convertufshcd_eh_timed_out()to use the new helper in patch 3.
- Remove an always-falseWARN_ONCE()in patch 4.
- Link to v1:
https://lore.kernel.org/r/[email protected]
Quoted 7 lines
Stanley Jhu (4):
scsi: ufs: core: Add a helper for clearinghba->outstanding_reqs
scsi: ufs: core: Fix double completion inufshcd_clear_lu_cmds()
scsi: ufs: core: Useufshcd_clear_outstanding_reqs()inufshcd_eh_timed_out()
scsi: ufs: core: Remove an always-falseWARN_ONCE()
Hi Stanley,
For the whole series:
One more place has the same problem. In ufshcd_abort(), the "cmd was completed, but without a notifying intr" path completes the request but leaves its bit set in hba->outstanding_reqs:
if (!hba->mcq_enabled && !(reg & (1 << tag))) {
...
__ufshcd_transfer_req_compl(hba, 1UL << tag);
...
}could use the new helper in the same way as patch 2?
if (ufshcd_clear_outstanding_reqs(hba, BIT(tag)))
ufshcd_compl_one_cqe(hba, tag, NULL);
Kind regards,
Bean
Quoted 3 lines · Bean Huo
One more place has the same problem. In
ufshcd_abort(), the "cmd was completed, but without a notifying intr" path completes the request but leaves its bit set inhba->outstanding_reqs:
Good catch, thanks. I can confirm it. I will add a separate patch to fix it in v3.
Bart Van Assche
reviewerQuoted 11 lines · Bean Huo
One more place has the same problem. In
ufshcd_abort(), the "cmd was completed, but without a notifying intr" path completes the request but leaves its bit set inhba->outstanding_reqs:if (!hba->mcq_enabled && !(reg & (1 << tag))) { ... __ufshcd_transfer_req_compl(hba, 1UL << tag); ... }
Quoted 5 lines
could use the new helper in the same way as patch 2?
if (
ufshcd_clear_outstanding_reqs(hba, BIT(tag)))ufshcd_compl_one_cqe(hba, tag, NULL);
Hi Bean,
There are two problems with calling ufshcd_compl_one_cqe() from ufshcd_abort():
* ufshcd_compl_one_cqe() must only be called for SCSI commands while 'tag' may refer to a device management command.
* SCSI commands must not be freed by a SCSI abort handler. Otherwise a use-after-free may be triggered in scmd_eh_abort_handler() when it tries to call scsi_queue_insert() or scsi_finish_command().
Bart.
Quoted 11 lines · Bart Van Assche
Hi Bean,
There are two problems with calling
ufshcd_compl_one_cqe()fromufshcd_abort():
*ufshcd_compl_one_cqe()must only be called for SCSI commands while 'tag' may refer to a device management command.
* SCSI commands must not be freed by a SCSI abort handler. Otherwise a use-after-free may be triggered inscmd_eh_abort_handler()when it
tries to callscsi_queue_insert()orscsi_finish_command().
Quoted 1 line
Bart.
Hi Bart,
So, should we remove the call to __ufshcd_transfer_req_compl from ufshcd_abort, which was added by commit64180742605f (“scsi: ufs: Fix the SCSI abort handler”)?
Thanks.
Peter
Quoted 3 lines · Peter Wang
So, should we remove the call to
__ufshcd_transfer_req_complfromufshcd_abort, which was added by commit64180742605f("scsi: ufs: Fix the SCSI abort handler")?
__ufshcd_transfer_req_compl() is a wrapper around ufshcd_compl_one_cqe(), so calling it from ufshcd_abort() has the two issues Bart noted. In addition, ufshcd_abort() called it without clearing BIT(tag) in hba->outstanding_reqs, which left the command to be released a second time by a later ufshcd_poll() or LU reset.
In v3 6/7 [1], we replace that __ufshcd_transfer_req_compl() call with claiming BIT(tag) via ufshcd_clear_outstanding_reqs() and calling ufshcd_release_scsi_cmd() if it is a SCSI command, matching the end of ufshcd_abort().
Bart Van Assche
reviewerQuoted 3 lines · Peter Wang
So, should we remove the call to
__ufshcd_transfer_req_complfromufshcd_abort, which was added by commit64180742605f(“scsi: ufs: Fix the SCSI abort handler”)?
Removing the __ufshcd_transfer_req_compl() call from ufshcd_abort()
sounds good to me. Please note that there was already such a call before
my patch. I think this is the first commit that introduced a completion
call in the ufshcd_abort() handler: 8bb2dde069d8 ("scsi: ufs: Properly release resources if a task is aborted successfully").
Best regards,
Bart.
Bart Van Assche
reviewerQuoted 7 lines · Bart Van Assche
Quoted 3 lines · Peter Wang
So, should we remove the call to
__ufshcd_transfer_req_complfromufshcd_abort, which was added by commit64180742605f(“scsi: ufs: Fix the SCSI abort handler”)?Removing the
__ufshcd_transfer_req_compl()call fromufshcd_abort()sounds good to me.
(replying to my above email)
Please ignore what I wrote - the __ufshcd_transfer_req_compl() call should be retained in ufshcd_abort().
Bart.
Quoted 41 lines · Stanley Jhu
In Single Doorbell (SDB) mode,
ufshcd_clear_lu_cmds()clears a transfer
request slot in the controller and completes the request without claiming its bit inhba->outstanding_reqs. A subsequentufshcd_poll()call sees the stale bit with its doorbell bit cleared and completes the
request a second time, causing a double DMA unmap.Patch 1 adds
__ufshcd_clear_outstanding_reqs()andufshcd_clear_outstanding_reqs()to share the bit-claim logic acrossufshcd_poll()andufshcd_abort()with no functional change. Patch 2 uses
the new helper inufshcd_clear_lu_cmds()to fix the SDB double completion. Patch 3 uses the new helper inufshcd_eh_timed_out(). It is
a follow-on cleanup and is not needed for the fix. Patch 4 removes aWARN_ONCE()whose condition is always false.This series is based on mkp/for-next (
db8b4d2bf2d3).Tested on QEMU ARM64 UFS in SDB and MCQ modes across LU reset, polled and interrupt I/O, and task abort without double completion or DMA unmap warnings.
Changes since v1:
- Add a helper for clearinghba->outstanding_reqsand use it inufshcd_poll(),ufshcd_abort(), andufshcd_clear_lu_cmds()(Bart Van Assche).
- Convertufshcd_eh_timed_out()to use the new helper in patch 3.
- Remove an always-falseWARN_ONCE()in patch 4.
- Link to v1:
https://lore.kernel.org/r/[email protected]Stanley Jhu (4):
scsi: ufs: core: Add a helper for clearinghba->outstanding_reqs
scsi: ufs: core: Fix double completion inufshcd_clear_lu_cmds()
scsi: ufs: core: Useufshcd_clear_outstanding_reqs()inufshcd_eh_timed_out()
scsi: ufs: core: Remove an always-falseWARN_ONCE()
Quoted 6 lines
drivers/ufs/core/ufshcd.c | 55 +++++++++++++++++++++++++------------ --
1 file changed, 35 insertions(+), 20 deletions(-)base-commit:
db8b4d2bf2d3e02b4ff78bbf69d00e11795840de
For the series: