Skip to contentSkip to conversation

scsi: pm: Balance pm_only counter of request queue during system resume

PATCH Superseded Can Guo 3 participants 12 messages Apr 30, 2020 – May 2, 2020 SCSI

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Can Guo

author
#0 lore raw

PATCH v3 1/1 scsi: pm: Balance pm_only counter of request queue during system resume

During system resume, scsi_resume_device() decreases a request queue's pm_only counter if the scsi device was quiesced before. But after that, if the scsi device's RPM status is RPM_SUSPENDED, the pm_only counter is still held (non-zero). Current scsi resume hook only sets the RPM status of the scsi device and its request queue to RPM_ACTIVE, but leaves the pm_only counter unchanged. This may make the request queue's pm_only counter remain non-zero after resume hook returns, hence those who are waiting on the mq_freeze_wq would never be woken up. Fix this by calling blk_post_runtime_resume() if pm_only is non-zero to balance the pm_only counter which is held by the scsi device's RPM ops.

(struct request_queue)0xFFFFFF815B69E938

	pm_only = (counter = 2),
	rpm_status = 0,
	dev = 0xFFFFFF815B0511A0,

((struct device)0xFFFFFF815B0511A0)).power
is_suspended = FALSE,
runtime_status = RPM_ACTIVE,

(struct scsi_device)0xFFFFFF815b051000

	request_queue = 0xFFFFFF815B69E938,
	sdev_state = SDEV_RUNNING,
	quiesced_by = 0x0,

B::v.f_/task_

-000|__switch_to
-001|context_switch
-001|__schedule
-002|schedule
-003|blk_queue_enter(q = 0xFFFFFF815B69E938, flags = 0)
-004|generic_make_request
-005|submit_bio

---

Change since v2:
- Rebased on 5.8-scsi-queue

Change since v1:
- Added more debugging context info

 drivers/scsi/scsi_pm.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/scsi/scsi_pm.c +4 -2
index 3717eea..4804029 100644
@@ -93,8 +93,10 @@ static int scsi_dev_type_resume(struct device *dev,
 		 */
 		if (!err && scsi_is_sdev_device(dev)) {
 			struct scsi_device *sdev = to_scsi_device(dev);
-
-			blk_set_runtime_active(sdev->request_queue);
+			if (blk_queue_pm_only(sdev->request_queue))
+				blk_post_runtime_resume(sdev->request_queue, 0);
+			else
+				blk_set_runtime_active(sdev->request_queue);
 		}
 	}
 
Signature · 1 line
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project.
#1 lore raw
Quoted 10 lines · Can Guo

During system resume, scsi_resume_device() decreases a request queue's pm_only counter if the scsi device was quiesced before. But after that, if the scsi device's RPM status is RPM_SUSPENDED, the pm_only counter is still held (non-zero). Current scsi resume hook only sets the RPM status of the scsi device and its request queue to RPM_ACTIVE, but leaves the pm_only counter unchanged. This may make the request queue's pm_only counter remain non-zero after resume hook returns, hence those who are waiting on the mq_freeze_wq would never be woken up. Fix this by calling blk_post_runtime_resume() if pm_only is non-zero to balance the pm_only counter which is held by the scsi device's RPM ops.

How was this issue discovered? How has this patch been tested?

Thanks,

Bart.

author
#2 lore raw

Hi Bart,

Quoted 12 lines · Bart Van Assche
Quoted 10 lines · Can Guo

During system resume, scsi_resume_device() decreases a request queue's pm_only counter if the scsi device was quiesced before. But after that, if the scsi device's RPM status is RPM_SUSPENDED, the pm_only counter is still held (non-zero). Current scsi resume hook only sets the RPM status of the scsi device and its request queue to RPM_ACTIVE, but leaves the pm_only counter unchanged. This may make the request queue's pm_only counter remain non-zero after resume hook returns, hence those who are waiting on the mq_freeze_wq would never be woken up. Fix this by calling blk_post_runtime_resume() if pm_only is non-zero to balance the pm_only counter which is held by the scsi device's RPM ops.

Quoted 5 lines

How was this issue discovered? How has this patch been tested?

Thanks,

Bart.

As the issue was found after system resumes, so the issue was discovered during system suspend/resume test, and it is very easy to be replicated. After system resumes, if this issue hits some scsi devices, all bios sent to their request queues are blocked, which may cause a system hang if the scsi devices are vital to system functionality.

To make sure the patch work well, we have tested system suspend/resume and made sure no system hang happen due to request queues got blocked by imbalanced pm_only counter.

Thanks,

Can Guo.

#10 lore raw
In reply to Bart Van Assche
Quoted 14 lines · Can Guo
Quoted 10 lines

During system resume, scsi_resume_device() decreases a request queue's pm_only counter if the scsi device was quiesced before. But after that, if the scsi device's RPM status is RPM_SUSPENDED, the pm_only counter is still held (non-zero). Current scsi resume hook only sets the RPM status of the scsi device and its request queue to RPM_ACTIVE, but leaves the pm_only counter unchanged. This may make the request queue's pm_only counter remain non-zero after resume hook returns, hence those who are waiting on the mq_freeze_wq would never be woken up. Fix this by calling blk_post_runtime_resume() if pm_only is non-zero to balance the pm_only counter which is held by the scsi device's RPM ops.

Quoted 1 line

How was this issue discovered? How has this patch been tested?

I think this insight was originally gained as part of commit fb276f770118 (scsi: ufs: Enable block layer runtime PM for well-known logical units)

But I will let Can reply on that.

Thanks,
Avri

Quoted 4 lines

Thanks,

Bart.

author
#11 lore raw

Hi Avri,

Quoted 15 lines · Avri Altman
Quoted 13 lines · Can Guo
Quoted 10 lines

During system resume, scsi_resume_device() decreases a request queue's pm_only counter if the scsi device was quiesced before. But after that, if the scsi device's RPM status is RPM_SUSPENDED, the pm_only counter is still held (non-zero). Current scsi resume hook only sets the RPM status of the scsi device and its request queue to RPM_ACTIVE, but leaves the pm_only counter unchanged. This may make the request queue's pm_only counter remain non-zero after resume hook returns, hence those who are waiting on the mq_freeze_wq would never be woken up. Fix this by calling blk_post_runtime_resume() if pm_only is non-zero to balance the pm_only counter which is held by the scsi device's RPM ops.

Quoted 1 line

How was this issue discovered? How has this patch been tested?

Quoted 8 lines

I think this insight was originally gained as part of commit fb276f770118 (scsi: ufs: Enable block layer runtime PM for well-known logical units)

But I will let Can reply on that.

Thanks,
Avri

Thanks for pointing to that commit, but this is a different story here. SCSI devices, which have block layer runtime PM enabled, can hit this issue during system resume. In the contratry, those which have block layer runtime PM disabled are immune to this issue.

Thanks,

Can Guo.

Quoted 4 lines
Quoted 4 lines

Thanks,

Bart.

#3 lore raw
In reply to Can Guo
Quoted 21 lines · Can Guo
Quoted 12 lines · Bart Van Assche
Quoted 10 lines · Can Guo

During system resume, scsi_resume_device() decreases a request queue's pm_only counter if the scsi device was quiesced before. But after that, if the scsi device's RPM status is RPM_SUSPENDED, the pm_only counter is still held (non-zero). Current scsi resume hook only sets the RPM status of the scsi device and its request queue to RPM_ACTIVE, but leaves the pm_only counter unchanged. This may make the request queue's pm_only counter remain non-zero after resume hook returns, hence those who are waiting on the mq_freeze_wq would never be woken up. Fix this by calling blk_post_runtime_resume() if pm_only is non-zero to balance the pm_only counter which is held by the scsi device's RPM ops.

Quoted 1 line

How was this issue discovered? How has this patch been tested?

As the issue was found after system resumes, so the issue was discovered during system suspend/resume test, and it is very easy to be replicated. After system resumes, if this issue hits some scsi devices, all bios sent to their request queues are blocked, which may cause a system hang if the scsi devices are vital to system functionality.

Quoted 3 lines

To make sure the patch work well, we have tested system suspend/resume and made sure no system hang happen due to request queues got blocked by imbalanced pm_only counter.

Thanks, that's very interesting information. My concern with this patch is that the power management code is not the only caller of blk_set_pm_only() / blk_clear_pm_only(). E.g. the SCSI SPI code also calls scsi_device_quiesce() and scsi_device_resume(). These last functions call blk_set_pm_only() and blk_clear_pm_only(). More calls of scsi_device_quiesce() and scsi_device_resume() might be added in the future.

Has it been considered to test directly whether a SCSI device has been runtime suspended instead of relying on blk_queue_pm_only()? How about using pm_runtime_status_suspended() or adding a function in block/blk-pm.h that checks whether q->rpm_status == RPM_SUSPENDED?

Thanks,

Bart.

author
#4 lore raw
Quoted 33 lines · Bart Van Assche
Quoted 21 lines · Can Guo
Quoted 12 lines · Bart Van Assche
Quoted 10 lines · Can Guo

During system resume, scsi_resume_device() decreases a request queue's pm_only counter if the scsi device was quiesced before. But after that, if the scsi device's RPM status is RPM_SUSPENDED, the pm_only counter is still held (non-zero). Current scsi resume hook only sets the RPM status of the scsi device and its request queue to RPM_ACTIVE, but leaves the pm_only counter unchanged. This may make the request queue's pm_only counter remain non-zero after resume hook returns, hence those who are waiting on the mq_freeze_wq would never be woken up. Fix this by calling blk_post_runtime_resume() if pm_only is non-zero to balance the pm_only counter which is held by the scsi device's RPM ops.

Quoted 1 line

How was this issue discovered? How has this patch been tested?

As the issue was found after system resumes, so the issue was discovered during system suspend/resume test, and it is very easy to be replicated. After system resumes, if this issue hits some scsi devices, all bios sent to their request queues are blocked, which may cause a system hang if the scsi devices are vital to system functionality.

Quoted 3 lines

To make sure the patch work well, we have tested system suspend/resume and made sure no system hang happen due to request queues got blocked by imbalanced pm_only counter.

Thanks, that's very interesting information. My concern with this patch is that the power management code is not the only caller of blk_set_pm_only() / blk_clear_pm_only(). E.g. the SCSI SPI code also calls scsi_device_quiesce() and scsi_device_resume(). These last functions call blk_set_pm_only() and blk_clear_pm_only(). More calls of scsi_device_quiesce() and scsi_device_resume() might be added in the future.

Quoted 8 lines

Has it been considered to test directly whether a SCSI device has been runtime suspended instead of relying on blk_queue_pm_only()? How about using pm_runtime_status_suspended() or adding a function in block/blk-pm.h that checks whether q->rpm_status == RPM_SUSPENDED?

Thanks,

Bart.

Hi Bart,

Please let me address your concern.

First of all, it is allowed to call scsi_device_quiesce() multiple times, but one sdev's request queue's pm_only counter can only be increased once by scsi_device_quiesce(), because if a sdev has already been quiesced, in scsi_device_quiesce(), scsi_device_set_state(sdev, SDEV_QUIESCE) would return -ENIVAL (illegal state transform), then blk_clear_pm_only() shall be called to decrease pm_only once, so no matter how many times scsi_device_quiesce() is called, it can only increase pm_only once.

scsi_device_resume() is same, it calls blk_clear_pm_only only once and only if the sdev was quiesced().

So, in a word, after scsi_device_resume() returns in scsi_dev_type_resume(), pm_only counter should be 1 (if the sdev's runtime power status is RPM_SUSPENDED) or 0 (if the sdev's runtime power status is RPM_ACTIVE).

Quoted 4 lines

Has it been considered to test directly whether a SCSI device has been runtime suspended instead of relying on blk_queue_pm_only()? How about using pm_runtime_status_suspended() or adding a function in block/blk-pm.h that checks whether q->rpm_status == RPM_SUSPENDED?

Yes, I used to make the patch like that way, and it also worked well, as both ways are equal actually. I kinda like the current code because we should be confident that after scsi_dev_type_resume() returns, pm_only must be 0. Different reviewers may have different opionions, either way works well anyways.

Thanks,

Can Guo.

author
#5 lore raw
In reply to Bart Van Assche
Quoted 33 lines · Bart Van Assche
Quoted 21 lines · Can Guo
Quoted 12 lines · Bart Van Assche
Quoted 10 lines · Can Guo

During system resume, scsi_resume_device() decreases a request queue's pm_only counter if the scsi device was quiesced before. But after that, if the scsi device's RPM status is RPM_SUSPENDED, the pm_only counter is still held (non-zero). Current scsi resume hook only sets the RPM status of the scsi device and its request queue to RPM_ACTIVE, but leaves the pm_only counter unchanged. This may make the request queue's pm_only counter remain non-zero after resume hook returns, hence those who are waiting on the mq_freeze_wq would never be woken up. Fix this by calling blk_post_runtime_resume() if pm_only is non-zero to balance the pm_only counter which is held by the scsi device's RPM ops.

Quoted 1 line

How was this issue discovered? How has this patch been tested?

As the issue was found after system resumes, so the issue was discovered during system suspend/resume test, and it is very easy to be replicated. After system resumes, if this issue hits some scsi devices, all bios sent to their request queues are blocked, which may cause a system hang if the scsi devices are vital to system functionality.

Quoted 3 lines

To make sure the patch work well, we have tested system suspend/resume and made sure no system hang happen due to request queues got blocked by imbalanced pm_only counter.

Thanks, that's very interesting information. My concern with this patch is that the power management code is not the only caller of blk_set_pm_only() / blk_clear_pm_only(). E.g. the SCSI SPI code also calls scsi_device_quiesce() and scsi_device_resume(). These last functions call blk_set_pm_only() and blk_clear_pm_only(). More calls of scsi_device_quiesce() and scsi_device_resume() might be added in the future.

Quoted 8 lines

Has it been considered to test directly whether a SCSI device has been runtime suspended instead of relying on blk_queue_pm_only()? How about using pm_runtime_status_suspended() or adding a function in block/blk-pm.h that checks whether q->rpm_status == RPM_SUSPENDED?

Thanks,

Bart.

Hi Bart,

Slightly revised my previous mail.

Please let me address your concern.

First of all, it is allowed to call scsi_device_quiesce() multiple times, but one sdev's request queue's pm_only counter can only be increased once by scsi_device_quiesce(), because if a sdev has already been quiesced, in scsi_device_quiesce(), scsi_device_set_state(sdev, SDEV_QUIESCE) would return -ENIVAL (illegal state transform), then blk_clear_pm_only() shall be called to decrease pm_only once, so no matter how many times scsi_device_quiesce() is called, it can only increase pm_only once.

scsi_device_resume() is same, it calls blk_clear_pm_only only once and only if the sdev was quiesced().

So, in a word, after scsi_device_resume() returns in scsi_dev_type_resume(), if a sdev has block layer runtime PM enabled (sdev->request_queue->dev is not NULL), its queue's pm_only counter should be 1 (if the sdev's runtime power status is RPM_SUSPENDED) or 0 (if the sdev's runtime power status is RPM_ACTIVE). If a sdev has block layer runtime PM disabled (sdev->request_queue->dev is NULL), its queue's pm_only counter should be 0.

Has it been considered to test directly whether a SCSI device has been runtime suspended instead of relying on blk_queue_pm_only()? How about using pm_runtime_status_suspended() or adding a function in block/blk-pm.h that checks whether q->rpm_status == RPM_SUSPENDED?

Yes, I used to make the patch like that way, and it also worked well, as both ways are equal actually. I kinda like the current code because we should be confident that after scsi_dev_type_resume() returns, pm_only must be 0. Different reviewers may have different opinions, either way works well anyways.

Thanks,

Can Guo.

#6 lore raw
Quoted 11 lines · Can Guo
Quoted 4 lines · Bart Van Assche

Has it been considered to test directly whether a SCSI device has been runtime suspended instead of relying on blk_queue_pm_only()? How about using pm_runtime_status_suspended() or adding a function in block/blk-pm.h that checks whether q->rpm_status == RPM_SUSPENDED?

Yes, I used to make the patch like that way, and it also worked well, as both ways are equal actually. I kinda like the current code because we should be confident that after scsi_dev_type_resume() returns, pm_only must be 0. Different reviewers may have different opinions, either way works well anyways.

Hi Can,

Please note that this is not a matter of personal preferences of a reviewer but a matter of correctness. blk_queue_pm_only() does not only return a value > 0 if a SCSI device has been runtime suspended but also returns true if scsi_device_quiesce() was called for another reason. Hence my request to test the "runtime suspended" status directly and not to rely on blk_queue_pm_only().

Thanks,

Bart.

author
#7 lore raw
Quoted 22 lines · Bart Van Assche
Quoted 11 lines · Can Guo
Quoted 4 lines · Bart Van Assche

Has it been considered to test directly whether a SCSI device has been runtime suspended instead of relying on blk_queue_pm_only()? How about using pm_runtime_status_suspended() or adding a function in block/blk-pm.h that checks whether q->rpm_status == RPM_SUSPENDED?

Yes, I used to make the patch like that way, and it also worked well, as both ways are equal actually. I kinda like the current code because we should be confident that after scsi_dev_type_resume() returns, pm_only must be 0. Different reviewers may have different opinions, either way works well anyways.

Hi Can,

Please note that this is not a matter of personal preferences of a reviewer but a matter of correctness. blk_queue_pm_only() does not only return a value > 0 if a SCSI device has been runtime suspended but also returns true if scsi_device_quiesce() was called for another reason. Hence my request to test the "runtime suspended" status directly and not to rely on blk_queue_pm_only().

Quoted 3 lines

Thanks,

Bart.

Hi Bart,

I agree we are pursuing correctness here, but as I said, I think both way are equally correct. I also agree with you that the alternative way, see [2], is much easier to be understood, we can take the alternative way if you are OK with it.

[1] Currently, scsi_dev_type_resume() is the hooker for resume, thaw and restore. Per my understanding, when scsi_dev_type_resume() is running, it is not possible that scsi_device_quiesce() can be called to this sdev, at least not possible in current code base. So it is OK to rely on blk_queue_pm_only() in scsi_dev_type_resume().

[2] The alternative way which I have tested with is like below. I think it is what you requested for if my understanding is right, please correct me if I am wrong.

drivers/scsi/scsi_pm.c +0 -0
index 3717eea..d18271d 100644
@@ -74,12 +74,15 @@ static int scsi_dev_type_resume(struct device *dev,
 {
        const struct dev_pm_ops *pm = dev->driver ? dev->driver->pm : 
NULL;
        int err = 0;

+ bool was_rpm_suspended = false;

        err = cb(dev, pm);
        scsi_device_resume(to_scsi_device(dev));
        dev_dbg(dev, "scsi resume: %d\n", err);

        if (err == 0) {
+               was_rpm_suspended = pm_runtime_suspended(dev);
+
                pm_runtime_disable(dev);
                err = pm_runtime_set_active(dev);
                pm_runtime_enable(dev);
@@ -93,8 +96,10 @@ static int scsi_dev_type_resume(struct device *dev,
                 */
                if (!err && scsi_is_sdev_device(dev)) {
                        struct scsi_device *sdev = to_scsi_device(dev);

-
- blk_set_runtime_active(sdev->request_queue);
+ if (was_rpm_suspended)
+ blk_post_runtime_resume(sdev->request_queue, 0);
+ else

+                               blk_set_runtime_active(sdev->request_queue);
                }
        }

Thanks,

Can Guo

#8 lore raw
Quoted 7 lines · Can Guo
drivers/scsi/scsi_pm.c +0 -0
index 3717eea..d18271d 100644
@@ -74,12 +74,15 @@ static int scsi_dev_type_resume(struct device *dev,
  {
         const struct dev_pm_ops *pm = dev->driver ? dev->driver->pm : NULL;

drivers/scsi/scsi_pm.c static int scsi_dev_type_resume(struct device *dev, · line 85

Quoted 10 lines
drivers/scsi/scsi_pm.c +3 -0
         int err = 0;
+       bool was_rpm_suspended = false;
 
         err = cb(dev, pm);
         scsi_device_resume(to_scsi_device(dev));
         dev_dbg(dev, "scsi resume: %d\n", err);
 
         if (err == 0) {
+               was_rpm_suspended = pm_runtime_suspended(dev);
+

How about renaming this variable into "was_runtime_suspended"? How about moving the declaration of that variable inside the if-statement?

Quoted 5 lines
drivers/scsi/scsi_pm.c +0 -0
                 pm_runtime_disable(dev);
                 err = pm_runtime_set_active(dev);
                 pm_runtime_enable(dev);
@@ -93,8 +96,10 @@ static int scsi_dev_type_resume(struct device *dev,
                  */

drivers/scsi/scsi_pm.c static int scsi_dev_type_resume(struct device *dev, · line 104

Quoted 10 lines
drivers/scsi/scsi_pm.c +4 -2
                 if (!err && scsi_is_sdev_device(dev)) {
                         struct scsi_device *sdev = to_scsi_device(dev);
-
-                       blk_set_runtime_active(sdev->request_queue);
+                       if (was_rpm_suspended)
+                              blk_post_runtime_resume(sdev->request_queue, 0);
+                       else
+                              blk_set_runtime_active(sdev->request_queue);
                 }
         }

Does other code always call both blk_pre_runtime_resume() and blk_post_runtime_resume() upon runtime resume? How about adding a blk_pre_runtime_resume() call before the blk_post_runtime_resume() call?

Thanks,

Bart.

author
#9 lore raw
Quoted 16 lines · Bart Van Assche
Quoted 15 lines · Can Guo
drivers/scsi/scsi_pm.c +1 -0
index 3717eea..d18271d 100644
@@ -74,12 +74,15 @@ static int scsi_dev_type_resume(struct device 
 *dev,
  {
         const struct dev_pm_ops *pm = dev->driver ? dev->driver->pm : NULL;
         int err = 0;
+       bool was_rpm_suspended = false;
 
         err = cb(dev, pm);
         scsi_device_resume(to_scsi_device(dev));
         dev_dbg(dev, "scsi resume: %d\n", err);
 
Quoted 7 lines

drivers/scsi/scsi_pm.c static int scsi_dev_type_resume(struct device · line 86

Quoted 3 lines
drivers/scsi/scsi_pm.c +2 -0
         if (err == 0) {
+               was_rpm_suspended = pm_runtime_suspended(dev);
+

How about renaming this variable into "was_runtime_suspended"? How about moving the declaration of that variable inside the if-statement?

Sure, shall do, this patch was just a prototype which I made for testing. If you are OK with this idea, I will send it as the next version.

Quoted 22 lines
Quoted 16 lines
drivers/scsi/scsi_pm.c +0 -0
                 pm_runtime_disable(dev);
                 err = pm_runtime_set_active(dev);

                pm_runtime_enable(dev);

drivers/scsi/scsi_pm.c +4 -2
@@ -93,8 +96,10 @@ static int scsi_dev_type_resume(struct device *dev,
                  */
                 if (!err && scsi_is_sdev_device(dev)) {
                         struct scsi_device *sdev = to_scsi_device(dev);
-
-                       blk_set_runtime_active(sdev->request_queue);
+                       if (was_rpm_suspended)
+                              
 blk_post_runtime_resume(sdev->request_queue, 0);
+                       else
+                              
 blk_set_runtime_active(sdev->request_queue);
                 }
Quoted 1 line

        }

Does other code always call both blk_pre_runtime_resume() and blk_post_runtime_resume() upon runtime resume? How about adding a blk_pre_runtime_resume() call before the blk_post_runtime_resume() call?

Quoted 3 lines

Thanks,

Bart.

Yes, but adding a blk_pre_runtime_resume() here is meaningless, it only sets the q->rpm_status to RPM_RESUMING, blk_post_runtime_resume() overrides it to RPM_ACTIVE for sure. Besides, this place comes after the call of pm_runtime_set_active(), meaning sdev is already runtime active, in contrast with the real runtime resume routine, we can think it as the sdev's runtime resume ops has returned 0, so we can just call blk_post_runtime_resume(err=0).

Thanks,

Can Guo.