ufs: core: Fix a deadlock in the frequency scaling code
Subsystems (2)
Bart Van Assche
reviewerPATCH 1/1 ufs: core: Fix a deadlock in the frequency scaling code
Commit 08b12cda6c44 ("scsi: ufs: core: Switch to scsi_get_internal_cmd()") accidentally introduced a deadlock in the frequency scaling code. ufshcd_clock_scaling_unprepare() may submit a device management command while SCSI command processing is blocked. The deadlock was introduced by using the SCSI core for submitting device management commands (scsi_get_internal_cmd() + blk_execute_rq()). Fix this deadlock by calling blk_mq_unquiesce_tagset() before any device management commands are submitted by ufshcd_clock_scaling_unprepare().
--- drivers/ufs/core/ufshcd.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
drivers/ufs/core/ufshcd.c +3 -4
@@ -1455,15 +1455,14 @@ static int ufshcd_clock_scaling_prepare(struct ufs_hba *hba, u64 timeout_us)static void ufshcd_clock_scaling_unprepare (struct ufs_hba *hba,int err) {up_write (&hba->clk_scaling_lock); +mutex_unlock (&hba->wb_mutex); +blk_mq_unquiesce_tagset (&hba->host->tag_set); +mutex_unlock (&hba->host->scan_mutex);/* Enable Write Booster if current gear requires it else disable it */ if (ufshcd_enable_wb_if_scaling_up (hba) && !err)ufshcd_wb_toggle (hba, hba->pwr_info.gear_rx >= hba->clk_scaling.wb_gear); -mutex_unlock (&hba->wb_mutex); - -blk_mq_unquiesce_tagset (&hba->host->tag_set); -mutex_unlock (&hba->host->scan_mutex);ufshcd_release (hba); }
Quoted 15 lines · Bart Van Assche
Commit
08b12cda6c44("scsi: ufs: core: Switch toscsi_get_internal_cmd()")
accidentally introduced a deadlock in the frequency scaling code.ufshcd_clock_scaling_unprepare()may submit a device management command
while SCSI command processing is blocked. The deadlock was introduced by
using the SCSI core for submitting device management commands (scsi_get_internal_cmd()+blk_execute_rq()). Fix this deadlock by callingblk_mq_unquiesce_tagset()before any device management commands are submitted byufshcd_clock_scaling_unprepare().Fixes 08b12cda6c44 ("scsi: ufs: core: Switch to
scsi_get_internal_cmd()")
Quoted 8 lines
Reported-by Manivannan Sadhasivam <[email protected]>Reported-by Roger Shimizu <[email protected]>Closes:
https://lore.kernel.org/linux-scsi/ehorjaflathzab5oekx2nae2zss5vi2r36yqkqsfjb2fgsifz2@yk3us5g3igow/Tested-by Roger Shimizu <[email protected]>Cc Nitin Rawat <[email protected]>Signed-off-by Bart Van Assche <[email protected]>---
Quoted 9 lines · Bart Van Assche
Commit
08b12cda6c44("scsi: ufs: core: Switch toscsi_get_internal_cmd()") accidentally introduced a deadlock in the frequency scaling code.ufshcd_clock_scaling_unprepare()may submit a device management command while SCSI command processing is blocked. The deadlock was introduced by using the SCSI core for submitting device management commands (scsi_get_internal_cmd()+blk_execute_rq()). Fix this deadlock by callingblk_mq_unquiesce_tagset()before any device management commands are submitted byufshcd_clock_scaling_unprepare().
Quoted 7 lines
Fixes 08b12cda6c44 ("scsi: ufs: core: Switch to scsi_get_internal_cmd()")Reported-by Manivannan Sadhasivam <[email protected]>Reported-by Roger Shimizu <[email protected]>Closes https://lore.kernel.org/linux-scsi/ehorjaflathzab5oekx2nae2zss5vi2r36yqkqsfjb2fgsifz2@yk3us5g3igow/ open hereTested-by Roger Shimizu <[email protected]>Cc Nitin Rawat <[email protected]>Signed-off-by Bart Van Assche <[email protected]>
Quoted 9 lines · Bart Van Assche
Commit
08b12cda6c44("scsi: ufs: core: Switch toscsi_get_internal_cmd()") accidentally introduced a deadlock in the frequency scaling code.ufshcd_clock_scaling_unprepare()may submit a device management command while SCSI command processing is blocked. The deadlock was introduced by using the SCSI core for submitting device management commands (scsi_get_internal_cmd()+blk_execute_rq()). Fix this deadlock by callingblk_mq_unquiesce_tagset()before any device management commands are submitted byufshcd_clock_scaling_unprepare().
Quoted 7 lines
Fixes 08b12cda6c44 ("scsi: ufs: core: Switch to scsi_get_internal_cmd()")Reported-by Manivannan Sadhasivam <[email protected]>Reported-by Roger Shimizu <[email protected]>Closes https://lore.kernel.org/linux-scsi/ehorjaflathzab5oekx2nae2zss5vi2r36yqkqsfjb2fgsifz2@yk3us5g3igow/ open hereTested-by Roger Shimizu <[email protected]>Cc Nitin Rawat <[email protected]>Signed-off-by Bart Van Assche <[email protected]>
This fixes the regression of freezing/hanging boot in initramfs on Qcom RB5 board with recent linux-nexts. Thank you!
Best regards,
Alexey
Martin K. Petersen
AppliedBart,
Quoted 9 lines
Commit
08b12cda6c44("scsi: ufs: core: Switch toscsi_get_internal_cmd()") accidentally introduced a deadlock in the frequency scaling code.ufshcd_clock_scaling_unprepare()may submit a device management command while SCSI command processing is blocked. The deadlock was introduced by using the SCSI core for submitting device management commands (scsi_get_internal_cmd()+blk_execute_rq()). Fix this deadlock by callingblk_mq_unquiesce_tagset()before any device management commands are submitted byufshcd_clock_scaling_unprepare().
Applied to 6.19/scsi-staging, thanks!
Signature · 1 line
Martin K. Petersen
Quoted 9 lines · Bart Van Assche
Commit
08b12cda6c44("scsi: ufs: core: Switch toscsi_get_internal_cmd()") accidentally introduced a deadlock in the frequency scaling code.ufshcd_clock_scaling_unprepare()may submit a device management command while SCSI command processing is blocked. The deadlock was introduced by using the SCSI core for submitting device management commands (scsi_get_internal_cmd()+blk_execute_rq()). Fix this deadlock by callingblk_mq_unquiesce_tagset()before any device management commands are submitted byufshcd_clock_scaling_unprepare().
Quoted 7 lines
Fixes 08b12cda6c44 ("scsi: ufs: core: Switch to scsi_get_internal_cmd()")Reported-by Manivannan Sadhasivam <[email protected]>Reported-by Roger Shimizu <[email protected]>Closes https://lore.kernel.org/linux-scsi/ehorjaflathzab5oekx2nae2zss5vi2r36yqkqsfjb2fgsifz2@yk3us5g3igow/ open hereTested-by Roger Shimizu <[email protected]>Cc Nitin Rawat <[email protected]>Signed-off-by Bart Van Assche <[email protected]>
Thanks for the quirk fix. While it seems to have fixed the boot hang, I'm quite skeptical about the fix. What is the guarantee that another device management command won't be submitted before blk_mq_unquiesce_tagset() in the future? IOW, the developer would have no idea that doing such will result in a deadlock as nothing in the UFS code makes it clear, like using the same lock and such.
- Mani
Quoted 27 lines
--- drivers/ufs/core/ufshcd.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)drivers/ufs/core/ufshcd.c +3
-4@@ -1455,15 +1455,14 @@ static int ufshcd_clock_scaling_prepare(struct ufs_hba *hba, u64 timeout_us)static void ufshcd_clock_scaling_unprepare (struct ufs_hba *hba,int err) {up_write (&hba->clk_scaling_lock); +mutex_unlock (&hba->wb_mutex); +blk_mq_unquiesce_tagset (&hba->host->tag_set); +mutex_unlock (&hba->host->scan_mutex);/* Enable Write Booster if current gear requires it else disable it */ if (ufshcd_enable_wb_if_scaling_up (hba) && !err)ufshcd_wb_toggle (hba, hba->pwr_info.gear_rx >= hba->clk_scaling.wb_gear); -mutex_unlock (&hba->wb_mutex); - -blk_mq_unquiesce_tagset (&hba->host->tag_set); -mutex_unlock (&hba->host->scan_mutex);ufshcd_release (hba); }
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Bart Van Assche
reviewerQuoted 6 lines · Manivannan Sadhasivam
Thanks for the quirk fix. While it seems to have fixed the boot hang, I'm quite skeptical about the fix. What is the guarantee that another device management command won't be submitted before
blk_mq_unquiesce_tagset()in the future? IOW, the developer would have no idea that doing such will result in a deadlock as nothing in the UFS code makes it clear, like using the same lock and such.
I think that the recent changes can be considered as a bug fix. With previous kernel versions, only SCSI commands were paused during clock frequency transitions but device management commands not. With the
approach that landed in Linus' master branch, device management commands
are also paused during clock frequency transitions. Device management
commands should also be paused because these also use the communication
link between host controller and UFS device.
Regarding debugging potential future deadlocks caused by submitting a device management command while the tag set is quiesced, a call trace is sufficient to discover the root cause (echo w > /proc/sysrq-trigger).
Regarding future changes in ufshcd_clock_scaling_unprepare(), I think there are two cases: changes made by developers who have access to a test setup that supports frequency scaling and changes made by developers who don't have access to a test setup that supports frequency scaling. Has it already been considered to insert calls to ufshcd_clock_scaling_prepare() and ufshcd_clock_scaling_unprepare() near the end of ufshcd_async_scan() to make sure that this code gets executed on test setups that do not support frequency scaling?
Thanks,
Bart.