Skip to contentSkip to conversation

ufs: core: wlun suspend SSU fail recovery

PATCH Superseded [email protected] 4 participants 9 messages Nov 1, 2022 – Dec 5, 2022

Subsystems (3)

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ufs: core: wlun suspend SSU fail recovery

From: Peter Wang <[email protected]>

When SSU fail in wlun suspend flow, trigger error handlder and return busy to break the suspend.
If not, wlun runtime pm status become error and the consumer will stuck in runtime suspend status.

---
 drivers/ufs/core/ufshcd.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
drivers/ufs/core/ufshcd.c +15 -1
index b1f59a5fe632..2f2d3d5d8684 100644
@@ -8970,6 +8970,7 @@ static int __ufshcd_wl_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op)
 	enum ufs_pm_level pm_lvl;
 	enum ufs_dev_pwr_mode req_dev_pwr_mode;
 	enum uic_link_state req_link_state;
+	unsigned long flags;
 
 	hba->pm_op_in_progress = true;
 	if (pm_op != UFS_SHUTDOWN_PM) {
@@ -9049,8 +9050,21 @@ static int __ufshcd_wl_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op)
 
 		if (!hba->dev_info.b_rpm_dev_flush_capable) {
 			ret = ufshcd_set_dev_pwr_mode(hba, req_dev_pwr_mode);
-			if (ret)
+			if (ret) {
+				/*
+				 * If retrun err in suspend flow, IO will hang.
+				 * Trigger error handler and break suspend for
+				 * error recovery.
+				 */
+				spin_lock_irqsave(hba->host->host_lock, flags);
+				hba->force_reset = true;
+				ufshcd_schedule_eh_work(hba);
+				spin_unlock_irqrestore(hba->host->host_lock,
+					flags);
+
+				ret = -EBUSY;
 				goto enable_scaling;
+			}
 		}
 	}
 
Signature · 1 line
2.18.0
#1 lore raw
Quoted 3 lines · On 1/11/22 16:24, [email protected] wrote:

From: Peter Wang <[email protected]>

When SSU fail in wlun suspend flow, trigger error handlder and

handlder -> handler

Why / how does SSU fail?

Quoted 22 lines

return busy to break the suspend.
If not, wlun runtime pm status become error and the consumer will stuck in runtime suspend status.

---
 drivers/ufs/core/ufshcd.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
drivers/ufs/core/ufshcd.c +1 -0
index b1f59a5fe632..2f2d3d5d8684 100644
@@ -8970,6 +8970,7 @@ static int __ufshcd_wl_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op)
 	enum ufs_pm_level pm_lvl;
 	enum ufs_dev_pwr_mode req_dev_pwr_mode;
 	enum uic_link_state req_link_state;
+	unsigned long flags;
 
 	hba->pm_op_in_progress = true;
 	if (pm_op != UFS_SHUTDOWN_PM) {

drivers/ufs/core/ufshcd.c static int __ufshcd_wl_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op) · line 9055

Quoted 7 lines
drivers/ufs/core/ufshcd.c +3 -1
@@ -9049,8 +9050,21 @@ static int __ufshcd_wl_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op)
 
 		if (!hba->dev_info.b_rpm_dev_flush_capable) {
 			ret = ufshcd_set_dev_pwr_mode(hba, req_dev_pwr_mode);
-			if (ret)
+			if (ret) {
+				/*
+				 * If retrun err in suspend flow, IO will hang.

retrun -> return

drivers/ufs/core/ufshcd.c static int __ufshcd_wl_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op) · line 9061

Quoted 6 lines
drivers/ufs/core/ufshcd.c +6 -0
+				 * Trigger error handler and break suspend for
+				 * error recovery.
+				 */
+				spin_lock_irqsave(hba->host->host_lock, flags);
+				hba->force_reset = true;
+				ufshcd_schedule_eh_work(hba);

__ufshcd_wl_suspend() is also used for shutdown and poweroff where scheduling EH is not appropriate.

Quoted 9 lines
drivers/ufs/core/ufshcd.c +5 -0
+				spin_unlock_irqrestore(hba->host->host_lock,
+					flags);
+
+				ret = -EBUSY;
 				goto enable_scaling;
+			}
 		}
 	}
 
#2 lore raw
Quoted 8 lines · Adrian Hunter
Quoted 3 lines · On 1/11/22 16:24, [email protected] wrote:

From: Peter Wang <[email protected]>

When SSU fail in wlun suspend flow, trigger error handlder and

handlder -> handler

Why / how does SSU fail?

I'm not sure but the issue that Peter is trying to fix with this patch may already have been fixed by my patch series "Fix a deadlock in the UFS driver".

Thanks,

Bart.

#3 lore raw
Quoted 16 lines · Bart Van Assche
Quoted 8 lines · Adrian Hunter
Quoted 3 lines · On 1/11/22 16:24, [email protected] wrote:

From: Peter Wang <[email protected]>

When SSU fail in wlun suspend flow, trigger error handlder and

handlder -> handler

Why / how does SSU fail?

I'm not sure but the issue that Peter is trying to fix with this patch
may already have been fixed by my patch series "Fix a deadlock in the
UFS driver".

Quoted 4 lines

Thanks,

Bart.

Hi Bart,

Yes, it may also can fix by
"[v4,07/10] scsi: ufs: Try harder to change the power mode"

But I am not sure if any outher corner case SSU will got antother err which retry canoot fix? (ex, device always return same error sense code which host is not expect to receive again, again and again let retry sill fail)

By the way, in wl suspend flow, not only SSU, but also enter hibern8(0x17) could got fail. I think it is better fix in both fail case.
Hera is our 0x17 timeout fail log and runtime suspend retrun timeout.

473-c(1), 21140.355276017,31742,18, rs, ret=-110, time_us= 2892755, pwr_mode=2, link_status=3
472-u(1), 21139.829423708,31742, 7,0x17,arg1=0x0,arg2=0x0,arg3=0x0, 0
471-r(1), 21139.829361861, 0,
1,0x1b,t=46,db:0x 0,is:0x 80000,crypt:0,0,lba= 0,len=
-1, 1738384
470-r(1), 21139.827623708,31742,
0,0x1b,t=46,db:0x 0,is:0x 80000,crypt:0,0,lba= 0,len=
-1, 0

Thanks.
Peter

#4 lore raw
In reply to Bart Van Assche
Quoted 7 lines · Bart Van Assche
Quoted 5 lines · Adrian Hunter
Quoted 3 lines · On 1/11/22 16:24, [email protected] wrote:

From: Peter Wang <[email protected]>

When SSU fail in wlun suspend flow, trigger error handlder and

Quoted 7 lines
Quoted 3 lines

handlder -> handler

Why / how does SSU fail?

I'm not sure but the issue that Peter is trying to fix with this patch may already have been fixed by my patch series "Fix a deadlock in the UFS driver".

We observe a similar failure scenario when a device tries to change power mode (e.g. due to autosuspend) while "purge" is in progress. It appears that in this case the "pm_only" counter is increased on the device queue prior to the attempt of entering a runtime suspended state, but not reduced upon it failing due to the device being busy with an ongoing purge. That leads to a deadlock of subsequent IO operations to the device.

--Daniil

#5 lore raw
Quoted 27 lines · Daniil Lunev
Quoted 9 lines · Bart Van Assche
Quoted 7 lines · Adrian Hunter
Quoted 3 lines · On 1/11/22 16:24, [email protected] wrote:

From: Peter Wang <[email protected]>

When SSU fail in wlun suspend flow, trigger error handlder and

handlder -> handler

Quoted 7 lines
Quoted 1 line

Why / how does SSU fail?

I'm not sure but the issue that Peter is trying to fix with this patch
may already have been fixed by my patch series "Fix a deadlock in the
UFS driver".

We observe a similar failure scenario when a device tries to change power mode (e.g. due to autosuspend) while "purge" is in progress. It appears that in this case the "pm_only" counter is increased on the device queue prior to the attempt of entering a runtime suspended state, but not reduced upon it failing due to the device being busy with an ongoing purge. That leads to a deadlock of subsequent IO operations to the device.

Quoted 1 line

--Daniil

Hi Daniil,

May I have a question, which dievce you use in this failure scenario? I think it is same issue due to SSU fail, and wlun devcie pm enter error state. So the cunsomer(scsi lu device) is block in suspend state and connot resume to reduce pm_only lead to IO hang.

Thanks.
BR
Peter

#6 lore raw
Quoted 8 lines

May I have a question, which dievce you use in this failure scenario? I think it is same issue due to SSU fail, and wlun devcie pm enter error state. So the cunsomer(scsi lu device) is block in suspend state and connot resume to reduce pm_only lead to IO hang.

Thanks.
BR
Peter

I don't think it is device specific. I am pretty sure it is the same problem, for it traces to the same origin and your patch fixes the issue (though with a lot of log spam which is a bit unfortunate, but we can live with that)

--Daniil

#7 lore raw

Hi Daniil,

Thanks your feedback.
I am glad to hear your problem can be solved by this patch. Please also help have a review of my v3 patch

https://patchwork.kernel.org/project/linux-scsi/patch/[email protected]/

Thanks.
BR
Peter

Quoted 17 lines · Daniil Lunev
Quoted 10 lines

May I have a question, which dievce you use in this failure scenario?
I think it is same issue due to SSU fail, and wlun devcie pm enter error state. So the cunsomer(scsi lu device) is block in suspend state
and connot resume to reduce pm_only lead to IO hang.

Thanks.
BR
Peter

I don't think it is device specific. I am pretty sure it is the same problem, for it traces to the same origin and your patch fixes the issue (though with a lot of log spam which is a bit unfortunate, but we can live with that)

--Daniil

#8 lore raw
In reply to Adrian Hunter
Quoted 6 lines · Adrian Hunter
Quoted 3 lines · On 1/11/22 16:24, [email protected] wrote:

From: Peter Wang <[email protected]>

When SSU fail in wlun suspend flow, trigger error handlder and

handlder -> handler

will fix next versio, thanks.

Quoted 3 lines

Why / how does SSU fail?

Unknow, it is seldom and we cannot catch fail the failure scenarios. We suspect is maybe the device problem which response wrong sense code. So, I think the driver is better to cover this error.
Here is our cmd histroy log with 3 times SSU fail, and retrun busy to do error handler.

26-c(1), 2576.085601525, 9396,18, rs, ret=-16, time_us= 41136,
pwr_mode=1, link_status=1
25-r(1), 2576.085027371, 9396,
1,0x1b,t=53,db:0x 0,is:0x 80000,crypt:0,0,lba= 0,len=
-1, 31308
24-r(1), 2576.084996910, 9396,
0,0x1b,t=53,db:0x 0,is:0x 80000,crypt:0,0,lba= 0,len=
-1, 0
23-r(1), 2576.084919679, 9396,
1,0x1b,t=52,db:0x 0,is:0x 80000,crypt:0,0,lba= 0,len=
-1, 32846
22-r(1), 2576.084887679, 9396,
0,0x1b,t=52,db:0x 0,is:0x 80000,crypt:0,0,lba= 0,len=
-1, 0
21-r(1), 2576.084704295, 0,
1,0x1b,t=55,db:0x 0,is:0x 80000,crypt:0,0,lba= 0,len=
-1, 39777539
20-r(1), 2576.044927679, 9396,
0,0x1b,t=55,db:0x 0,is:0x 80000,crypt:0,0,lba= 0,len=
-1, 0

Quoted 25 lines
Quoted 25 lines

return busy to break the suspend.
If not, wlun runtime pm status become error and the consumer will stuck in runtime suspend status.

---
 drivers/ufs/core/ufshcd.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
drivers/ufs/core/ufshcd.c +1 -0
index b1f59a5fe632..2f2d3d5d8684 100644
@@ -8970,6 +8970,7 @@ static int __ufshcd_wl_suspend(struct ufs_hba
 *hba, enum ufs_pm_op pm_op)
 	enum ufs_pm_level pm_lvl;
 	enum ufs_dev_pwr_mode req_dev_pwr_mode;
 	enum uic_link_state req_link_state;
+	unsigned long flags;
 
 	hba->pm_op_in_progress = true;
drivers/ufs/core/ufshcd.c +0 -0
 	if (pm_op != UFS_SHUTDOWN_PM) {
@@ -9049,8 +9050,21 @@ static int __ufshcd_wl_suspend(struct
 ufs_hba *hba, enum ufs_pm_op pm_op)
 
Quoted 10 lines
Quoted 8 lines
drivers/ufs/core/ufshcd.c +3 -1
 		if (!hba->dev_info.b_rpm_dev_flush_capable) {
 			ret = ufshcd_set_dev_pwr_mode(hba,
 req_dev_pwr_mode);
-			if (ret)
+			if (ret) {
+				/*
+				 * If retrun err in suspend flow, IO

will hang.

retrun -> return

will fix next version, thanks

Quoted 12 lines

drivers/ufs/core/ufshcd.c static int __ufshcd_wl_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op) · line 9061

Quoted 9 lines
drivers/ufs/core/ufshcd.c +1 -0
+				 * Trigger error handler and break

suspend for

drivers/ufs/core/ufshcd.c +3 -0
+				 * error recovery.
+				 */
+				spin_lock_irqsave(hba->host->host_lock, 

flags);

drivers/ufs/core/ufshcd.c +2 -0
+				hba->force_reset = true;
+				ufshcd_schedule_eh_work(hba);

__ufshcd_wl_suspend() is also used for shutdown and poweroff where scheduling EH is not appropriate.

Will check pm_op and bypass UFS_SHUTDOWN_PM next version, thanks.

Quoted 13 lines
Quoted 11 lines
drivers/ufs/core/ufshcd.c +1 -0
+				spin_unlock_irqrestore(hba->host-
Quoted 1 line

host_lock,

drivers/ufs/core/ufshcd.c +4 -0
+					flags);
+
+				ret = -EBUSY;
 				goto enable_scaling;
+			}
 		}
 	}