ufs: core: wlun resume SSU(Acitve) fail recovery
Subsystems (3)
PATCH v3 1/1 ufs: core: wlun resume SSU(Acitve) fail recovery
From: Peter Wang <[email protected]>
When wlun resume SSU(Active) timeout, scsi try eh_host_reset_handler. But ufshcd_eh_host_reset_handler hang at wait flush_work(&hba->eh_work). And ufshcd_err_handler hang at wait rpm resume.
Do link recovery only in this case. Below is IO hang stack dump.
<ffffffdd78e02b34> schedule+0x110/0x204 <ffffffdd78e0be60> schedule_timeout+0x98/0x138 <ffffffdd78e040e8> wait_for_common_io+0x130/0x2d0 <ffffffdd77d6a000> blk_execute_rq+0x10c/0x16c <ffffffdd78126d90> __scsi_execute+0xfc/0x278 <ffffffdd7813891c> ufshcd_set_dev_pwr_mode+0x1c8/0x40c <ffffffdd78137d1c> __ufshcd_wl_resume+0xf0/0x5cc <ffffffdd78137ae0> ufshcd_wl_runtime_resume+0x40/0x18c <ffffffdd78136108> scsi_runtime_resume+0x88/0x104 <ffffffdd7809a4f8> __rpm_callback+0x1a0/0xaec <ffffffdd7809b624> rpm_resume+0x7e0/0xcd0 <ffffffdd7809a788> __rpm_callback+0x430/0xaec <ffffffdd7809b644> rpm_resume+0x800/0xcd0 <ffffffdd780a0778> pm_runtime_work+0x148/0x198
<ffffffdd78e02b34> schedule+0x110/0x204 <ffffffdd78e0be10> schedule_timeout+0x48/0x138 <ffffffdd78e03d9c> wait_for_common+0x144/0x2dc <ffffffdd7758bba4> __flush_work+0x3d0/0x508 <ffffffdd7815572c> ufshcd_eh_host_reset_handler+0x134/0x3a8 <ffffffdd781216f4> scsi_try_host_reset+0x54/0x204 <ffffffdd78120594> scsi_eh_ready_devs+0xb30/0xd48 <ffffffdd7812373c> scsi_error_handler+0x260/0x874
<ffffffdd78e02b34> schedule+0x110/0x204 <ffffffdd7809af64> rpm_resume+0x120/0xcd0 <ffffffdd7809fde8> __pm_runtime_resume+0xa0/0x17c <ffffffdd7815193c> ufshcd_err_handling_prepare+0x40/0x430 <ffffffdd7814cce8> ufshcd_err_handler+0x1c4/0xd4c
--- drivers/ufs/core/ufshcd.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
drivers/ufs/core/ufshcd.c +17 -0
@@ -7366,6 +7366,23 @@ static int ufshcd_eh_host_reset_handler(struct scsi_cmnd *cmd) hba =shost_priv (cmd->device->host); +/* +* If pm op resume fail and wait err recovery, do link recovery only. +* Because schedule eh work will get dead lock in ufshcd_rpm_get_sync +* and wait wlun resume, but wlun resume error wait eh work finish. +*/ +if (hba->pm_op_in_progress && + hba->curr_dev_pwr_mode !=UFS_ACTIVE_PWR_MODE ) { + err =ufshcd_link_recovery (hba); +if (err) { +dev_err (hba->dev,"%s: power state %d pm_progress %d" , + __func__, + hba->curr_dev_pwr_mode, + hba->pm_op_in_progress); + } +return err; + } +spin_lock_irqsave (hba->host->host_lock, flags); hba->force_reset = true;ufshcd_schedule_eh_work (hba);
Signature · 1 line
2.18.0
Bart Van Assche
reviewerQuoted 1 line · On 12/27/22 22:01, [email protected] wrote:
When wlun resume SSU(Active) timeout, scsi try
eh_host_reset_handler.
^^^^^^^^^^^
Please use the same spelling in the patch subject (Acitve -> Active).
timeout -> times out
scsi try -> the SCSI core invokes
Quoted 1 line
But
ufshcd_eh_host_reset_handlerhang at waitflush_work(&hba->eh_work).
hang at -> hangs in
Quoted 1 line
And
ufshcd_err_handlerhang at wait rpm resume.
hang at wait rpm resume -> hangs in rpm_resume().
Quoted 24 lines
<
ffffffdd78e02b34>schedule+0x110/0x204<ffffffdd78e0be60>schedule_timeout+0x98/0x138<ffffffdd78e040e8>wait_for_common_io+0x130/0x2d0<ffffffdd77d6a000>blk_execute_rq+0x10c/0x16c<ffffffdd78126d90>__scsi_execute+0xfc/0x278<ffffffdd7813891c>ufshcd_set_dev_pwr_mode+0x1c8/0x40c<ffffffdd78137d1c>__ufshcd_wl_resume+0xf0/0x5cc<ffffffdd78137ae0>ufshcd_wl_runtime_resume+0x40/0x18c<ffffffdd78136108>scsi_runtime_resume+0x88/0x104<ffffffdd7809a4f8>__rpm_callback+0x1a0/0xaec<ffffffdd7809b624>rpm_resume+0x7e0/0xcd0<ffffffdd7809a788>__rpm_callback+0x430/0xaec<ffffffdd7809b644>rpm_resume+0x800/0xcd0<ffffffdd780a0778>pm_runtime_work+0x148/0x198<
ffffffdd78e02b34>schedule+0x110/0x204<ffffffdd78e0be10>schedule_timeout+0x48/0x138<ffffffdd78e03d9c>wait_for_common+0x144/0x2dc<ffffffdd7758bba4>__flush_work+0x3d0/0x508<ffffffdd7815572c>ufshcd_eh_host_reset_handler+0x134/0x3a8<ffffffdd781216f4>scsi_try_host_reset+0x54/0x204<ffffffdd78120594>scsi_eh_ready_devs+0xb30/0xd48<ffffffdd7812373c>scsi_error_handler+0x260/0x874
Quoted 5 lines
<
ffffffdd78e02b34>schedule+0x110/0x204<ffffffdd7809af64>rpm_resume+0x120/0xcd0<ffffffdd7809fde8>__pm_runtime_resume+0xa0/0x17c<ffffffdd7815193c>ufshcd_err_handling_prepare+0x40/0x430<ffffffdd7814cce8>ufshcd_err_handler+0x1c4/0xd4c
On top of which kernel version has this patch been developed?
I think this deadlock has already been fixed by commit 7029e2151a7c ("scsi: ufs: Fix a deadlock between PM and the SCSI error handler"). Please check whether that commit by itself (without this patch) is sufficient to fix the reported deadlock.
Quoted 3 lines
--- drivers/ufs/core/ufshcd.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
The changelog is missing. Please include a changelog when posting v2 or a later version of a patch.
drivers/ufs/core/ufshcd.c static int ufshcd_eh_host_reset_handler(struct scsi_cmnd *cmd) · line 7373
Quoted 13 lines
drivers/ufs/core/ufshcd.c +5
-0@@ -7366,6 +7366,23 @@ static int ufshcd_eh_host_reset_handler(struct scsi_cmnd *cmd) hba =shost_priv (cmd->device->host); +/* +* If pm op resume fail and wait err recovery, do link recovery only. +* Because schedule eh work will get dead lock in ufshcd_rpm_get_sync +* and wait wlun resume, but wlun resume error wait eh work finish. +*/
The above comment has grammar issues and some parts are incomprehensible. What does e.g. "wait err recovery" mean? Please improve this source code comment.
Thanks,
Bart.
Quoted 49 lines · Bart Van Assche
Quoted 2 lines · On 12/27/22 22:01, [email protected] wrote:
When wlun resume SSU(Active) timeout, scsi try
eh_host_reset_handler.^^^^^^^^^^^Please use the same spelling in the patch subject (Acitve -> Active).
timeout -> times out
scsi try -> the SCSI core invokesQuoted 2 lines
But
ufshcd_eh_host_reset_handlerhang at wait flush_work(&hba-Quoted 1 line
eh_work).hang at -> hangs in
Quoted 1 line
And
ufshcd_err_handlerhang at wait rpm resume.hang at wait rpm resume -> hangs in
rpm_resume().Quoted 24 lines
<
ffffffdd78e02b34>schedule+0x110/0x204<ffffffdd78e0be60>schedule_timeout+0x98/0x138<ffffffdd78e040e8>wait_for_common_io+0x130/0x2d0<ffffffdd77d6a000>blk_execute_rq+0x10c/0x16c<ffffffdd78126d90>__scsi_execute+0xfc/0x278<ffffffdd7813891c>ufshcd_set_dev_pwr_mode+0x1c8/0x40c<ffffffdd78137d1c>__ufshcd_wl_resume+0xf0/0x5cc<ffffffdd78137ae0>ufshcd_wl_runtime_resume+0x40/0x18c<ffffffdd78136108>scsi_runtime_resume+0x88/0x104<ffffffdd7809a4f8>__rpm_callback+0x1a0/0xaec<ffffffdd7809b624>rpm_resume+0x7e0/0xcd0<ffffffdd7809a788>__rpm_callback+0x430/0xaec<ffffffdd7809b644>rpm_resume+0x800/0xcd0<ffffffdd780a0778>pm_runtime_work+0x148/0x198<
ffffffdd78e02b34>schedule+0x110/0x204<ffffffdd78e0be10>schedule_timeout+0x48/0x138<ffffffdd78e03d9c>wait_for_common+0x144/0x2dc<ffffffdd7758bba4>__flush_work+0x3d0/0x508<ffffffdd7815572c>ufshcd_eh_host_reset_handler+0x134/0x3a8<ffffffdd781216f4>scsi_try_host_reset+0x54/0x204<ffffffdd78120594>scsi_eh_ready_devs+0xb30/0xd48<ffffffdd7812373c>scsi_error_handler+0x260/0x874Quoted 5 lines
<
ffffffdd78e02b34>schedule+0x110/0x204<ffffffdd7809af64>rpm_resume+0x120/0xcd0<ffffffdd7809fde8>__pm_runtime_resume+0xa0/0x17c<ffffffdd7815193c>ufshcd_err_handling_prepare+0x40/0x430<ffffffdd7814cce8>ufshcd_err_handler+0x1c4/0xd4c
Quoted 6 lines
On top of which kernel version has this patch been developed? I think this deadlock has already been fixed by commit
7029e2151a7c("scsi: ufs: Fix a deadlock between PM and the SCSI error handler"). Please check whether that commit by itself (without this patch) is sufficient to fix the reported deadlock.
Hi Bart,
7029e2151a7c is not fix this dead lock, we still found this issue happen in kernel-6.1
I will update this patch and correct comment.
Please have a look again.
Thanks.
Peter
Quoted 33 lines
Quoted 3 lines
--- drivers/ufs/core/ufshcd.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)The changelog is missing. Please include a changelog when posting v2 or
a later version of a patch.Quoted 16 lines
drivers/ufs/core/ufshcd.c +2
-0@@ -7366,6 +7366,23 @@ static int ufshcd_eh_host_reset_handler(struct scsi_cmnd *cmd) hba =shost_priv (cmd->device->host); +/* +* If pm op resume fail and wait err recovery, do link recovery only.
drivers/ufs/core/ufshcd.c +1
-0+* Because schedule eh work will get dead lock in
ufshcd_rpm_get_syncdrivers/ufs/core/ufshcd.c +1
-0+* and wait wlun resume, but wlun resume error wait eh work finish.
drivers/ufs/core/ufshcd.c static int ufshcd_eh_host_reset_handler(struct scsi_cmnd *cmd) · line 7373
Quoted 1 line
drivers/ufs/core/ufshcd.c +1
-0+*/ The above comment has grammar issues and some parts are incomprehensible. What does e.g. "wait err recovery" mean? Please improve this source code comment.
Thanks,
Bart.