ufs: core: wlun suspend SSU fail recovery
Subsystems (3)
PATCH 1/1 ufs: core: wlun suspend SSU fail recovery
From: Peter Wang <[email protected]>
When SSU fail in wlun suspend flow, trigger error handlder and return busy to break the suspend.
If not, wlun runtime pm status become error and the consumer will stuck in runtime suspend status.
--- drivers/ufs/core/ufshcd.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)
drivers/ufs/core/ufshcd.c +15 -1
@@ -8970,6 +8970,7 @@ static int __ufshcd_wl_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op)enum ufs_pm_level pm_lvl;enum ufs_dev_pwr_mode req_dev_pwr_mode;enum uic_link_state req_link_state; +unsigned long flags; hba->pm_op_in_progress = true;if (pm_op !=UFS_SHUTDOWN_PM ) {
@@ -9049,8 +9050,21 @@ static int __ufshcd_wl_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op)if (!hba->dev_info.b_rpm_dev_flush_capable) { ret =ufshcd_set_dev_pwr_mode (hba, req_dev_pwr_mode); -if (ret) +if (ret) { +/* +* If retrun err in suspend flow, IO will hang. +* Trigger error handler and break suspend for +* error recovery. +*/ +spin_lock_irqsave (hba->host->host_lock, flags); + hba->force_reset = true; +ufshcd_schedule_eh_work (hba); +spin_unlock_irqrestore (hba->host->host_lock, + flags); + + ret = -EBUSY ; goto enable_scaling; + } } }
Signature · 1 line
2.18.0
Quoted 3 lines · On 1/11/22 16:24, [email protected] wrote:
From: Peter Wang <[email protected]>
When SSU fail in wlun suspend flow, trigger error handlder and
handlder -> handler
Why / how does SSU fail?
Quoted 22 lines
return busy to break the suspend.
If not, wlun runtime pm status become error and the consumer will stuck in runtime suspend status.Signed-off-by Peter Wang <[email protected]>--- drivers/ufs/core/ufshcd.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)drivers/ufs/core/ufshcd.c +1
-0@@ -8970,6 +8970,7 @@ static int __ufshcd_wl_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op)enum ufs_pm_level pm_lvl;enum ufs_dev_pwr_mode req_dev_pwr_mode;enum uic_link_state req_link_state; +unsigned long flags; hba->pm_op_in_progress = true;if (pm_op !=UFS_SHUTDOWN_PM ) {
drivers/ufs/core/ufshcd.c static int __ufshcd_wl_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op) · line 9055
Quoted 7 lines
drivers/ufs/core/ufshcd.c +3
-1@@ -9049,8 +9050,21 @@ static int __ufshcd_wl_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op)if (!hba->dev_info.b_rpm_dev_flush_capable) { ret =ufshcd_set_dev_pwr_mode (hba, req_dev_pwr_mode); -if (ret) +if (ret) { +/* +* If retrun err in suspend flow, IO will hang.
retrun -> return
drivers/ufs/core/ufshcd.c static int __ufshcd_wl_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op) · line 9061
Quoted 6 lines
drivers/ufs/core/ufshcd.c +6
-0+* Trigger error handler and break suspend for +* error recovery. +*/ +spin_lock_irqsave (hba->host->host_lock, flags); + hba->force_reset = true; +ufshcd_schedule_eh_work (hba);
__ufshcd_wl_suspend() is also used for shutdown and poweroff where scheduling EH is not appropriate.
Quoted 9 lines
drivers/ufs/core/ufshcd.c +5
-0+spin_unlock_irqrestore (hba->host->host_lock, + flags); + + ret = -EBUSY ; goto enable_scaling; + } } }
Bart Van Assche
reviewerQuoted 8 lines · Adrian Hunter
Quoted 3 lines · On 1/11/22 16:24, [email protected] wrote:
From: Peter Wang <[email protected]>
When SSU fail in wlun suspend flow, trigger error handlder and
handlder -> handler
Why / how does SSU fail?
I'm not sure but the issue that Peter is trying to fix with this patch may already have been fixed by my patch series "Fix a deadlock in the UFS driver".
Thanks,
Bart.
Quoted 16 lines · Bart Van Assche
Quoted 8 lines · Adrian Hunter
Quoted 3 lines · On 1/11/22 16:24, [email protected] wrote:
From: Peter Wang <[email protected]>
When SSU fail in wlun suspend flow, trigger error handlder and
handlder -> handler
Why / how does SSU fail?
I'm not sure but the issue that Peter is trying to fix with this patch
may already have been fixed by my patch series "Fix a deadlock in the
UFS driver".
Quoted 4 lines
Thanks,
Bart.
Hi Bart,
Yes, it may also can fix by
"[v4,07/10] scsi: ufs: Try harder to change the power mode"
But I am not sure if any outher corner case SSU will got antother err which retry canoot fix? (ex, device always return same error sense code which host is not expect to receive again, again and again let retry sill fail)
By the way, in wl suspend flow, not only SSU, but also enter hibern8(0x17) could got fail. I think it is better fix in both fail case.
Hera is our 0x17 timeout fail log and runtime suspend retrun timeout.
473-c(1), 21140.355276017,31742,18, rs, ret=-110, time_us= 2892755, pwr_mode=2, link_status=3
472-u(1), 21139.829423708,31742, 7,0x17,arg1=0x0,arg2=0x0,arg3=0x0, 0
471-r(1), 21139.829361861, 0,
1,0x1b,t=46,db:0x 0,is:0x 80000,crypt:0,0,lba= 0,len=
-1, 1738384
470-r(1), 21139.827623708,31742,
0,0x1b,t=46,db:0x 0,is:0x 80000,crypt:0,0,lba= 0,len=
-1, 0
Thanks.
Peter
Quoted 7 lines · Bart Van Assche
Quoted 5 lines · Adrian Hunter
Quoted 3 lines · On 1/11/22 16:24, [email protected] wrote:
From: Peter Wang <[email protected]>
When SSU fail in wlun suspend flow, trigger error handlder and
Quoted 7 lines
Quoted 3 lines
handlder -> handler
Why / how does SSU fail?
I'm not sure but the issue that Peter is trying to fix with this patch may already have been fixed by my patch series "Fix a deadlock in the UFS driver".
We observe a similar failure scenario when a device tries to change power mode (e.g. due to autosuspend) while "purge" is in progress. It appears that in this case the "pm_only" counter is increased on the device queue prior to the attempt of entering a runtime suspended state, but not reduced upon it failing due to the device being busy with an ongoing purge. That leads to a deadlock of subsequent IO operations to the device.
--Daniil
Quoted 27 lines · Daniil Lunev
Quoted 9 lines · Bart Van Assche
Quoted 7 lines · Adrian Hunter
Quoted 3 lines · On 1/11/22 16:24, [email protected] wrote:
From: Peter Wang <[email protected]>
When SSU fail in wlun suspend flow, trigger error handlder and
handlder -> handler
Quoted 7 lines
Quoted 1 line
Why / how does SSU fail?
I'm not sure but the issue that Peter is trying to fix with this patch
may already have been fixed by my patch series "Fix a deadlock in the
UFS driver".We observe a similar failure scenario when a device tries to change power mode (e.g. due to autosuspend) while "purge" is in progress. It appears that in this case the "
pm_only" counter is increased on the device queue prior to the attempt of entering a runtime suspended state, but not reduced upon it failing due to the device being busy with an ongoing purge. That leads to a deadlock of subsequent IO operations to the device.
Quoted 1 line
--Daniil
Hi Daniil,
May I have a question, which dievce you use in this failure scenario? I think it is same issue due to SSU fail, and wlun devcie pm enter error state. So the cunsomer(scsi lu device) is block in suspend state and connot resume to reduce pm_only lead to IO hang.
Thanks.
BR
Peter
Quoted 8 lines
May I have a question, which dievce you use in this failure scenario? I think it is same issue due to SSU fail, and wlun devcie pm enter error state. So the cunsomer(scsi lu device) is block in suspend state and connot resume to reduce
pm_onlylead to IO hang.Thanks.
BR
Peter
I don't think it is device specific. I am pretty sure it is the same problem, for it traces to the same origin and your patch fixes the issue (though with a lot of log spam which is a bit unfortunate, but we can live with that)
--Daniil
Hi Daniil,
Thanks your feedback.
I am glad to hear your problem can be solved by this patch. Please also help have a review of my v3 patch
https://patchwork.kernel.org/project/linux-scsi/patch/[email protected]/
Thanks.
BR
Peter
Quoted 17 lines · Daniil Lunev
Quoted 10 lines
May I have a question, which dievce you use in this failure scenario?
I think it is same issue due to SSU fail, and wlun devcie pm enter error state. So the cunsomer(scsi lu device) is block in suspend state
and connot resume to reducepm_onlylead to IO hang.Thanks.
BR
PeterI don't think it is device specific. I am pretty sure it is the same problem, for it traces to the same origin and your patch fixes the issue (though with a lot of log spam which is a bit unfortunate, but we can live with that)
--Daniil
Quoted 6 lines · Adrian Hunter
Quoted 3 lines · On 1/11/22 16:24, [email protected] wrote:
From: Peter Wang <[email protected]>
When SSU fail in wlun suspend flow, trigger error handlder and
handlder -> handler
will fix next versio, thanks.
Quoted 3 lines
Why / how does SSU fail?
Unknow, it is seldom and we cannot catch fail the failure scenarios. We suspect is maybe the device problem which response wrong sense code. So, I think the driver is better to cover this error.
Here is our cmd histroy log with 3 times SSU fail, and retrun busy to do error handler.
26-c(1), 2576.085601525, 9396,18, rs, ret=-16, time_us= 41136,
pwr_mode=1, link_status=1
25-r(1), 2576.085027371, 9396,
1,0x1b,t=53,db:0x 0,is:0x 80000,crypt:0,0,lba= 0,len=
-1, 31308
24-r(1), 2576.084996910, 9396,
0,0x1b,t=53,db:0x 0,is:0x 80000,crypt:0,0,lba= 0,len=
-1, 0
23-r(1), 2576.084919679, 9396,
1,0x1b,t=52,db:0x 0,is:0x 80000,crypt:0,0,lba= 0,len=
-1, 32846
22-r(1), 2576.084887679, 9396,
0,0x1b,t=52,db:0x 0,is:0x 80000,crypt:0,0,lba= 0,len=
-1, 0
21-r(1), 2576.084704295, 0,
1,0x1b,t=55,db:0x 0,is:0x 80000,crypt:0,0,lba= 0,len=
-1, 39777539
20-r(1), 2576.044927679, 9396,
0,0x1b,t=55,db:0x 0,is:0x 80000,crypt:0,0,lba= 0,len=
-1, 0
Quoted 25 lines
Quoted 25 lines
return busy to break the suspend.
If not, wlun runtime pm status become error and the consumer will stuck in runtime suspend status.Signed-off-by Peter Wang <[email protected]>--- drivers/ufs/core/ufshcd.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)drivers/ufs/core/ufshcd.c +1
-0@@ -8970,6 +8970,7 @@ static int __ufshcd_wl_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op)enum ufs_pm_level pm_lvl;enum ufs_dev_pwr_mode req_dev_pwr_mode;enum uic_link_state req_link_state; +unsigned long flags; hba->pm_op_in_progress = true;drivers/ufs/core/ufshcd.c +0
-0if (pm_op !=UFS_SHUTDOWN_PM ) {@@ -9049,8 +9050,21 @@ static int __ufshcd_wl_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op)
Quoted 10 lines
Quoted 8 lines
drivers/ufs/core/ufshcd.c +3
-1if (!hba->dev_info.b_rpm_dev_flush_capable) { ret =ufshcd_set_dev_pwr_mode (hba, req_dev_pwr_mode); -if (ret) +if (ret) { +/* +* If retrun err in suspend flow, IO will hang.
retrun -> return
will fix next version, thanks
Quoted 12 lines
drivers/ufs/core/ufshcd.c static int __ufshcd_wl_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op) · line 9061
Quoted 9 lines
drivers/ufs/core/ufshcd.c +1
-0+* Trigger error handler and break suspend for
drivers/ufs/core/ufshcd.c +3
-0+* error recovery. +*/ + spin_lock_irqsave(hba->host->host_lock,flags);
drivers/ufs/core/ufshcd.c +2
-0+ hba->force_reset = true; +ufshcd_schedule_eh_work (hba);
__ufshcd_wl_suspend()is also used for shutdown and poweroff where scheduling EH is not appropriate.
Will check pm_op and bypass UFS_SHUTDOWN_PM next version, thanks.
Quoted 13 lines
Quoted 11 lines
drivers/ufs/core/ufshcd.c +1
-0+ spin_unlock_irqrestore(hba->host-Quoted 1 line
host_lock,drivers/ufs/core/ufshcd.c +4
-0+ flags); + + ret = -EBUSY ; goto enable_scaling; + } } }