scsi: sd: add runtime pm to open / release
Subsystems (2)
PATCH 1/1 scsi: sd: add runtime pm to open / release
This add a very conservative but simple implementation for runtime PM to the sd scsi driver:
Resume when opened (mounted) and suspend when released (unmounted).
Improvements that allow suspending while a device is "open" can be added later, but now we save power when no filesystem is mounted and runtime PM is enabled.
--- drivers/scsi/sd.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/scsi/sd.c +6 -0
@@ -1372,6 +1372,7 @@ static int sd_open(struct block_device *bdev, fmode_t mode)SCSI_LOG_HLQUEUE (3 ,sd_printk (KERN_INFO , sdkp,"sd_open\n" )); sdev = sdkp->device; +scsi_autopm_get_device (sdev);/* * If the device is in error recovery, wait until it is done.
@@ -1418,6 +1419,9 @@ static int sd_open(struct block_device *bdev, fmode_t mode) error_out:scsi_disk_put (sdkp); + +scsi_autopm_put_device (sdev); +return retval; }
@@ -1441,6 +1445,8 @@ static void sd_release(struct gendisk *disk, fmode_t mode)SCSI_LOG_HLQUEUE (3 ,sd_printk (KERN_INFO , sdkp,"sd_release\n" )); +scsi_autopm_put_device (sdev); +if (atomic_dec_return (&sdkp->openers) ==0 && sdev->removable) {if (scsi_block_when_processing_errors (sdev))scsi_set_medium_removal (sdev,SCSI_REMOVAL_ALLOW );
Signature · 1 line
2.20.1
Quoted 36 lines · Martin Kepplinger
This add a very conservative but simple implementation for runtime PM to the sd scsi driver:
Resume when opened (mounted) and suspend when released (unmounted).Improvements that allow suspending while a device is "open" can be added later, but now we save power when no filesystem is mounted and runtime PM is enabled.
Signed-off-by Martin Kepplinger <[email protected]>--- drivers/scsi/sd.c | 6 ++++++ 1 file changed, 6 insertions(+)drivers/scsi/sd.c +4
-0@@ -1372,6 +1372,7 @@ static int sd_open(struct block_device *bdev, fmode_t mode)SCSI_LOG_HLQUEUE (3 ,sd_printk (KERN_INFO , sdkp,"sd_open\n" )); sdev = sdkp->device; +scsi_autopm_get_device (sdev);/* * If the device is in error recovery, wait until it is done. @@ -1418,6 +1419,9 @@ static int sd_open(struct block_device *bdev, fmode_t mode) error_out:scsi_disk_put (sdkp); + +scsi_autopm_put_device (sdev); +return retval; }
drivers/scsi/sd.c static void sd_release(struct gendisk *disk, fmode_t mode) · line 1452
Quoted 8 lines
drivers/scsi/sd.c +2
-0@@ -1441,6 +1445,8 @@ static void sd_release(struct gendisk *disk, fmode_t mode)SCSI_LOG_HLQUEUE (3 ,sd_printk (KERN_INFO , sdkp,"sd_release\n" )); +scsi_autopm_put_device (sdev); +if (atomic_dec_return (&sdkp->openers) ==0 && sdev->removable) {if (scsi_block_when_processing_errors (sdev))scsi_set_medium_removal (sdev,SCSI_REMOVAL_ALLOW );
My understanding of the above patch is that it introduces a regression, namely by disabling runtime suspend as long as an sd device is held open.
Bart.
Quoted 49 lines · Bart Van Assche
Quoted 36 lines · Martin Kepplinger
This add a very conservative but simple implementation for runtime PM to the sd scsi driver:
Resume when opened (mounted) and suspend when released (unmounted).Improvements that allow suspending while a device is "open" can be added later, but now we save power when no filesystem is mounted and runtime PM is enabled.
Signed-off-by Martin Kepplinger <[email protected]>--- drivers/scsi/sd.c | 6 ++++++ 1 file changed, 6 insertions(+)drivers/scsi/sd.c +4
-0@@ -1372,6 +1372,7 @@ static int sd_open(struct block_device *bdev, fmode_t mode)SCSI_LOG_HLQUEUE (3 ,sd_printk (KERN_INFO , sdkp,"sd_open\n" )); sdev = sdkp->device; +scsi_autopm_get_device (sdev);/* * If the device is in error recovery, wait until it is done. @@ -1418,6 +1419,9 @@ static int sd_open(struct block_device *bdev, fmode_t mode) error_out:scsi_disk_put (sdkp); + +scsi_autopm_put_device (sdev); +return retval; }drivers/scsi/sd.c static void sd_release(struct gendisk *disk, fmode_t mode) · line 1452
Quoted 8 lines
drivers/scsi/sd.c +2
-0@@ -1441,6 +1445,8 @@ static void sd_release(struct gendisk *disk, fmode_t mode)SCSI_LOG_HLQUEUE (3 ,sd_printk (KERN_INFO , sdkp,"sd_release\n" )); +scsi_autopm_put_device (sdev); +if (atomic_dec_return (&sdkp->openers) ==0 && sdev->removable) {if (scsi_block_when_processing_errors (sdev))scsi_set_medium_removal (sdev,SCSI_REMOVAL_ALLOW );My understanding of the above patch is that it introduces a regression, namely by disabling runtime suspend as long as an sd device is held open.
Quoted 3 lines
Bart.
hi Bart,
Alan says the same (on block request, the block layer should initiate a runtime resume), so merging with the thread from
https://lore.kernel.org/linux-usb/[email protected]/T/
now and answer to both Bart and Alan here:]
I see scsi-pm.c using the blk-pm.c API but I'm not sure how the block layer would itself resume the scsi device (I use it via usb_storage, so that usb_stor_resume() follows in my case but I guess that doesn't matter here):
my understanding of "sd" is: enable runtime pm in probe(), so *allow* the device to be suspended (if enabled by the user), but never resume(?). Also, why isn't "autopm" used in its ioctl() implementation (as opposed to in "sr")?
here's roughly what happens when enabling runtime PM in sysfs (again, because sd_probe() calls autopm_put() and thus allows it:
[ 27.384446] sd 0:0:0:0: scsi_runtime_suspend [ 27.432282] blk_pre_runtime_suspend [ 27.435783] sd_suspend_common [ 27.438782] blk_post_runtime_suspend [ 27.442427] scsi target0:0:0: scsi_runtime_suspend [ 27.447303] scsi host0: scsi_runtime_suspend
then I "mount /dev/sda1 /mnt" and none of the resume() functions get called. To me it looks like the sd driver should initiate resuming, and that's not implemented.
what am I doing wrong or overlooking? how exactly does (or should) the block layer initiate resume here?
thanks again for your time,
martin
Quoted 61 lines · Martin Kepplinger
Quoted 49 lines · Bart Van Assche
Quoted 36 lines · Martin Kepplinger
This add a very conservative but simple implementation for runtime PM to the sd scsi driver:
Resume when opened (mounted) and suspend when released (unmounted).Improvements that allow suspending while a device is "open" can be added later, but now we save power when no filesystem is mounted and runtime PM is enabled.
Signed-off-by Martin Kepplinger <[email protected]>--- drivers/scsi/sd.c | 6 ++++++ 1 file changed, 6 insertions(+)drivers/scsi/sd.c +4
-0@@ -1372,6 +1372,7 @@ static int sd_open(struct block_device *bdev, fmode_t mode)SCSI_LOG_HLQUEUE (3 ,sd_printk (KERN_INFO , sdkp,"sd_open\n" )); sdev = sdkp->device; +scsi_autopm_get_device (sdev);/* * If the device is in error recovery, wait until it is done. @@ -1418,6 +1419,9 @@ static int sd_open(struct block_device *bdev, fmode_t mode) error_out:scsi_disk_put (sdkp); + +scsi_autopm_put_device (sdev); +return retval; }drivers/scsi/sd.c static void sd_release(struct gendisk *disk, fmode_t mode) · line 1452
Quoted 8 lines
drivers/scsi/sd.c +2
-0@@ -1441,6 +1445,8 @@ static void sd_release(struct gendisk *disk, fmode_t mode)SCSI_LOG_HLQUEUE (3 ,sd_printk (KERN_INFO , sdkp,"sd_release\n" )); +scsi_autopm_put_device (sdev); +if (atomic_dec_return (&sdkp->openers) ==0 && sdev->removable) {if (scsi_block_when_processing_errors (sdev))scsi_set_medium_removal (sdev,SCSI_REMOVAL_ALLOW );My understanding of the above patch is that it introduces a regression, namely by disabling runtime suspend as long as an sd device is held open.
Quoted 3 lines
Bart.
hi Bart,
Alan says the same (on block request, the block layer should initiate a runtime resume), so merging with the thread from
https://lore.kernel.org/linux-usb/[email protected]/T/
now and answer to both Bart and Alan here:]
Quoted 4 lines
I see
scsi-pm.cusing theblk-pm.cAPI but I'm not sure how the block layer would itself resume the scsi device (I use it viausb_storage, so thatusb_stor_resume()follows in my case but I guess that doesn't matter here):
The block layer does this in block/blk-core.c:blk_queue_enter(), as part of the condition check in the call to wait_event() near the end of the function. The blk_pm_request_resume() inline routine calls pm_request_resume().
At least, that's what is _supposed_ to happen. See commit 0d25bd072b49 ("block: Schedule runtime resume earlier").
Quoted 4 lines
my understanding of "sd" is: enable runtime pm in
probe(), so *allow* the device to be suspended (if enabled by the user), but never resume(?). Also, why isn't "autopm" used in itsioctl()implementation (as opposed to in "sr")?
I don't remember the reason. It may be that the code in sr.c isn't needed.
Quoted 10 lines
here's roughly what happens when enabling runtime PM in sysfs (again, because
sd_probe()callsautopm_put()and thus allows it:[ 27.384446] sd 0:0:0:0: scsi_runtime_suspend [ 27.432282] blk_pre_runtime_suspend [ 27.435783] sd_suspend_common [ 27.438782] blk_post_runtime_suspend [ 27.442427] scsi target0:0:0: scsi_runtime_suspend [ 27.447303] scsi host0: scsi_runtime_suspend
Quoted 6 lines
then I "mount
/dev/sda1/mnt" and none of theresume()functions get called. To me it looks like the sd driver should initiate resuming, and that's not implemented.what am I doing wrong or overlooking? how exactly does (or should) the block layer initiate resume here?
I don't know what's going wrong. Bart, can you look into it? As far as I can tell, you're the last person to touch the block-layer's runtime PM code.
Alan Stern
Quoted 2 lines · Martin Kepplinger
Also, why isn't "autopm" used in its
ioctl()implementation (as opposed to in "sr")?
Some of the scsi_autopm_{get,put}_device() calls in the sr driver have been introduced by me before I fully understood runtime pm. I will have a another look to see whether these are really necessary and if not, post a patch to remove these again.
Bart.
Quoted 10 lines · Martin Kepplinger
here's roughly what happens when enabling runtime PM in sysfs (again, because
sd_probe()callsautopm_put()and thus allows it:[ 27.384446] sd 0:0:0:0: scsi_runtime_suspend [ 27.432282] blk_pre_runtime_suspend [ 27.435783] sd_suspend_common [ 27.438782] blk_post_runtime_suspend [ 27.442427] scsi target0:0:0: scsi_runtime_suspend [ 27.447303] scsi host0: scsi_runtime_suspend
Quoted 6 lines
then I "mount
/dev/sda1/mnt" and none of theresume()functions get called. To me it looks like the sd driver should initiate resuming, and that's not implemented.what am I doing wrong or overlooking? how exactly does (or should) the block layer initiate resume here?
As far as I know runtime power management support in the sd driver is working fine and is being used intensively by the UFS driver. The following commit was submitted to fix a bug encountered by an UFS developer: 05d18ae1cc8a ("scsi:
pm: Balance pm_only counter of request queue during system resume") # v5.7. I'm not sure which bug is causing trouble on your setup but I think it's likely that the root cause is somewhere else than in the block layer, the SCSI core or the SCSI sd driver.
Bart.
Quoted 15 lines · Bart Van Assche
Quoted 14 lines · Martin Kepplinger
here's roughly what happens when enabling runtime PM in sysfs (again, because
sd_probe()callsautopm_put()and thus allows it:[ 27.384446] sd 0:0:0:0: scsi_runtime_suspend [ 27.432282] blk_pre_runtime_suspend [ 27.435783] sd_suspend_common [ 27.438782] blk_post_runtime_suspend [ 27.442427] scsi target0:0:0: scsi_runtime_suspend [ 27.447303] scsi host0: scsi_runtime_suspendthen I "mount
/dev/sda1/mnt" and none of theresume()functions get called. To me it looks like the sd driver should initiate resuming, and that's not implemented.
Quoted 7 lines
Quoted 2 lines
what am I doing wrong or overlooking? how exactly does (or should) the block layer initiate resume here?
As far as I know runtime power management support in the sd driver is working fine and is being used intensively by the UFS driver. The following commit was submitted to fix a bug encountered by an UFS developer:
05d18ae1cc8a("scsi:
pm: Balancepm_onlycounter of request queue during system resume") # v5.7.
I just looked at that commit for the first time.
Instead of making the SCSI driver do the work of deciding what routine to call, why not redefine blk_set_runtime_active(q) to simply call blk_post_runtime_resume(q, 0)? Or vice versa: if err == 0 have blk_post_runtime_resume call blk_set_runtime_active?
After all, the two routines do almost the same thing -- and the bug addressed by this commit was caused by the difference in their behaviors.
If the device was already runtime-active during the system suspend, doing an extra clear of the pm_only counter won't hurt anything.
Quoted 5 lines
I'm not sure which bug is causing trouble on your setup but I think it's likely that the root cause is somewhere else than in the block layer, the SCSI core or the SCSI sd driver.
Bart.
Martin's best approach would be to add some debugging code to find out why blk_queue_enter() isn't calling bkl_pm_request_resume(), or why that call doesn't lead to pm_request_resume().
Alan Stern
Quoted 13 lines · Alan Stern
Quoted 4 lines · Bart Van Assche
As far as I know runtime power management support in the sd driver is working fine and is being used intensively by the UFS driver. The following commit was submitted to fix a bug encountered by an UFS developer:
05d18ae1cc8a("scsi:
pm: Balancepm_onlycounter of request queue during system resume") # v5.7.I just looked at that commit for the first time.
Instead of making the SCSI driver do the work of deciding what routine to call, why not redefine
blk_set_runtime_active(q)to simply callblk_post_runtime_resume(q, 0)? Or vice versa: if err == 0 haveblk_post_runtime_resumecallblk_set_runtime_active?
Quoted 5 lines
After all, the two routines do almost the same thing -- and the bug addressed by this commit was caused by the difference in their behaviors.
If the device was already runtime-active during the system suspend, doing an extra clear of the
pm_onlycounter won't hurt anything.
Hi Alan,
Do you want to submit a patch that implements this change or do you perhaps expect me to do that?
Thanks,
Bart.
Quoted 17 lines · Bart Van Assche
Quoted 16 lines · Alan Stern
Quoted 4 lines · Bart Van Assche
As far as I know runtime power management support in the sd driver is working fine and is being used intensively by the UFS driver. The following commit was submitted to fix a bug encountered by an UFS developer:
05d18ae1cc8a("scsi:
pm: Balancepm_onlycounter of request queue during system resume") # v5.7.I just looked at that commit for the first time.
Instead of making the SCSI driver do the work of deciding what routine to call, why not redefine
blk_set_runtime_active(q)to simply callblk_post_runtime_resume(q, 0)? Or vice versa: if err == 0 haveblk_post_runtime_resumecallblk_set_runtime_active?After all, the two routines do almost the same thing -- and the bug addressed by this commit was caused by the difference in their behaviors.
Quoted 7 lines
Quoted 2 lines
If the device was already runtime-active during the system suspend, doing an extra clear of the
pm_onlycounter won't hurt anything.Hi Alan,
Do you want to submit a patch that implements this change or do you perhaps expect me to do that?
I'll submit a patch in a few days. I just wanted to check first that the idea was sound.
Alan Stern
Quoted 41 lines · Alan Stern
Quoted 15 lines · Bart Van Assche
Quoted 14 lines · Martin Kepplinger
here's roughly what happens when enabling runtime PM in sysfs (again, because
sd_probe()callsautopm_put()and thus allows it:[ 27.384446] sd 0:0:0:0: scsi_runtime_suspend [ 27.432282] blk_pre_runtime_suspend [ 27.435783] sd_suspend_common [ 27.438782] blk_post_runtime_suspend [ 27.442427] scsi target0:0:0: scsi_runtime_suspend [ 27.447303] scsi host0: scsi_runtime_suspendthen I "mount
/dev/sda1/mnt" and none of theresume()functions get called. To me it looks like the sd driver should initiate resuming, and that's not implemented.Quoted 7 lines
Quoted 2 lines
what am I doing wrong or overlooking? how exactly does (or should) the block layer initiate resume here?
As far as I know runtime power management support in the sd driver is working fine and is being used intensively by the UFS driver. The following commit was submitted to fix a bug encountered by an UFS developer:
05d18ae1cc8a("scsi:
pm: Balancepm_onlycounter of request queue during system resume") # v5.7.I just looked at that commit for the first time.
Instead of making the SCSI driver do the work of deciding what routine to call, why not redefine
blk_set_runtime_active(q)to simply callblk_post_runtime_resume(q, 0)? Or vice versa: if err == 0 haveblk_post_runtime_resumecallblk_set_runtime_active?After all, the two routines do almost the same thing -- and the bug addressed by this commit was caused by the difference in their behaviors.
If the device was already runtime-active during the system suspend, doing an extra clear of the
pm_onlycounter won't hurt anything.Quoted 4 lines
I'm not sure which bug is causing trouble on your setup but I think it's likely that the root cause is somewhere else than in the block layer, the SCSI core or the SCSI sd driver.
Quoted 8 lines
Quoted 1 line
Bart.
Martin's best approach would be to add some debugging code to find out why
blk_queue_enter()isn't callingbkl_pm_request_resume(), or why that call doesn't lead topm_request_resume().Alan Stern
Hi Alan,
blk_queue_enter() always - especially when sd is runtime suspended and I try to mount as above - sets success to be true for me, so never continues down to bkl_pm_request_resume(). All I see is "PM: Removing info for No Bus:sda1".
thanks,
martin
Quoted 10 lines · Martin Kepplinger
Quoted 6 lines · Alan Stern
Martin's best approach would be to add some debugging code to find out why
blk_queue_enter()isn't callingbkl_pm_request_resume(), or why that call doesn't lead topm_request_resume().Alan Stern
Quoted 6 lines
Hi Alan,
blk_queue_enter()always - especially when sd is runtime suspended and I try to mount as above - sets success to be true for me, so never continues down tobkl_pm_request_resume(). All I see is "PM: Removing info for No Bus:sda1".
Aha. Looking at this more closely, it's apparent that the code in blk-core.c contains a logic bug: It assumes that if the BLK_MQ_REQ_PREEMPT flag is set then the request can be issued regardless of the queue's runtime status. That is not correct when the queue is suspended.
Below is my attempt to fix this up. I'm not sure that the patch is entirely correct, but it should fix this logic bug. I would appreciate a critical review.
Martin, does this fix the problem?
Alan Stern
block/blk-core.c +3 -3
@@ -423,7 +423,8 @@ int blk_queue_enter(struct request_queue* responsible for ensuring that that counter is * globally visible before the queue is unfrozen. */ -if (pm || !blk_queue_pm_only (q)) { +if ((pm && q->rpm_status !=RPM_SUSPENDED ) || + !blk_queue_pm_only (q)) { success = true; }else {percpu_ref_put (&q->q_usage_counter);
@@ -448,8 +449,7 @@ int blk_queue_enter(struct request_queuewait_event (q->mq_freeze_wq, (!q->mq_freeze_depth && - (pm || (blk_pm_request_resume (q), - !blk_queue_pm_only (q)))) || +blk_pm_resume_queue (pm, q)) ||blk_queue_dying (q));if (blk_queue_dying (q))return -ENODEV ;
block/blk-pm.h +9 -5
@@ -6,11 +6,14 @@#include <linux/pm_runtime.h> #ifdef CONFIG_PM -static inline void blk_pm_request_resume (struct request_queue *q) +static inline int blk_pm_resume_queue (const bool pm,struct request_queue *q) { -if (q->dev && (q->rpm_status ==RPM_SUSPENDED || - q->rpm_status ==RPM_SUSPENDING )) -pm_request_resume (q->dev); +if (!q->dev || !blk_queue_pm_only (q)) +return 1 ;/* Nothing to do */ +if (pm && q->rpm_status !=RPM_SUSPENDED ) +return 1 ;/* Request allowed */ +pm_request_resume (q->dev); +return 0 ; }static inline void blk_pm_mark_last_busy (struct request *rq)
@@ -44,8 +47,9 @@ static inline void blk_pm_put_request(st --rq->q->nr_pending; }#else -static inline void blk_pm_request_resume (struct request_queue *q) +static inline int blk_pm_resume_queue (const bool pm,struct request_queue *q) { +return 1 ; }static inline void blk_pm_mark_last_busy (struct request *rq)
Quoted 4 lines · Alan Stern
Aha. Looking at this more closely, it's apparent that the code in
blk-core.ccontains a logic bug: It assumes that if theBLK_MQ_REQ_PREEMPTflag is set then the request can be issued regardless of the queue's runtime status. That is not correct when the queue is suspended.
Please clarify why this is not correct.
Quoted 5 lines
Index:
usb-devel/block/blk-core.c
===================================================================
block/blk-core.c int blk_queue_enter(struct request_queue · line 430
Quoted 9 lines
block/blk-core.c +2
-1@@ -423,7 +423,8 @@ int blk_queue_enter(struct request_queue* responsible for ensuring that that counter is * globally visible before the queue is unfrozen. */ -if (pm || !blk_queue_pm_only (q)) { +if ((pm && q->rpm_status !=RPM_SUSPENDED ) || + !blk_queue_pm_only (q)) { success = true; }else {percpu_ref_put (&q->q_usage_counter);
Does the above change make it impossible to bring a suspended device back to the RPM_ACTIVE state if the BLK_MQ_REQ_NOWAIT flag is set?
Quoted 21 lines
block/blk-core.c +1
-2@@ -448,8 +449,7 @@ int blk_queue_enter(struct request_queuewait_event (q->mq_freeze_wq, (!q->mq_freeze_depth && - (pm || (blk_pm_request_resume (q), - !blk_queue_pm_only (q)))) || +blk_pm_resume_queue (pm, q)) ||blk_queue_dying (q));if (blk_queue_dying (q))return -ENODEV ;Index:
usb-devel/block/blk-pm.h
===================================================================block/blk-pm.h +1
-1@@ -6,11 +6,14 @@#include <linux/pm_runtime.h> #ifdef CONFIG_PM -static inline void blk_pm_request_resume (struct request_queue *q) +static inline int blk_pm_resume_queue (const bool pm,struct request_queue *q) {
block/blk-pm.h line 17
Quoted 10 lines
block/blk-pm.h +6
-3-if (q->dev && (q->rpm_status ==RPM_SUSPENDED || - q->rpm_status ==RPM_SUSPENDING )) -pm_request_resume (q->dev); +if (!q->dev || !blk_queue_pm_only (q)) +return 1 ;/* Nothing to do */ +if (pm && q->rpm_status !=RPM_SUSPENDED ) +return 1 ;/* Request allowed */ +pm_request_resume (q->dev); +return 0 ; }
Does the above change, especially the " && q->rpm_status != RPM_SUSPENDED" part, make it impossible to bring a suspended device back to the RPM_ACTIVE state?
Thanks,
Bart.
Quoted 7 lines · Bart Van Assche
Quoted 4 lines · Alan Stern
Aha. Looking at this more closely, it's apparent that the code in
blk-core.ccontains a logic bug: It assumes that if theBLK_MQ_REQ_PREEMPTflag is set then the request can be issued regardless of the queue's runtime status. That is not correct when the queue is suspended.Please clarify why this is not correct.
As I understand it, BLK_MQ_REQ_PREEMPT is supposed to mean (among other things) that this request may be issued as part of the procedure for putting a device into a low-power state or returning it to a high-power state. Consequently, requests with that flag set must be allowed while the queue is in the RPM_SUSPENDING or RPM_RESUMING runtime states -- as opposed to ordinary requests, which are allowed only in the RPM_ACTIVE state.
In the RPM_SUSPENDED state, however, the queue is entirely inactive. Even if a request were to be issued somehow, it would fail because the system would not be able to transmit it to the device. In other words, when the queue is in the RPM_SUSPENDED state, a resume must be requested before _any_ request can be issued.
Quoted 9 lines
Quoted 9 lines
Index:
usb-devel/block/blk-core.c
===================================================================block/blk-core.c +0
-1@@ -423,7 +423,8 @@ int blk_queue_enter(struct request_queue* responsible for ensuring that that counter is * globally visible before the queue is unfrozen. */ -if (pm || !blk_queue_pm_only (q)) {
Quoted 8 lines
block/blk-core.c int blk_queue_enter(struct request_queue · line 430
Quoted 5 lines
block/blk-core.c +2
-0+if ((pm && q->rpm_status !=RPM_SUSPENDED ) || + !blk_queue_pm_only (q)) { success = true; }else {percpu_ref_put (&q->q_usage_counter);Does the above change make it impossible to bring a suspended device back to the
RPM_ACTIVEstate if theBLK_MQ_REQ_NOWAITflag is set?
The only case affected by this change is when BLK_MQ_REQ_PREEMPT is set and the queue is in the RPM_SUSPENDED state. If BLK_MQ_REQ_NOWAIT was also set, the original code would set "success" to true, allowing the request to proceed even though it could not be carried out immediately -- a bug.
With the patch, such a request will fail without resuming the device. I don't know whether that is the desired behavior or not, but at least it's not obviously a bug.
It does seem odd that blk_queue_enter() tests the queue's pm_only status and the request flag in two different spots (here and below). Why does it do this? It seems like an invitation for bugs.
Quoted 27 lines
Quoted 27 lines
block/blk-core.c +1
-2@@ -448,8 +449,7 @@ int blk_queue_enter(struct request_queuewait_event (q->mq_freeze_wq, (!q->mq_freeze_depth && - (pm || (blk_pm_request_resume (q), - !blk_queue_pm_only (q)))) || +blk_pm_resume_queue (pm, q)) ||blk_queue_dying (q));if (blk_queue_dying (q))return -ENODEV ;Index:
usb-devel/block/blk-pm.h
===================================================================block/blk-pm.h +4
-4@@ -6,11 +6,14 @@#include <linux/pm_runtime.h> #ifdef CONFIG_PM -static inline void blk_pm_request_resume (struct request_queue *q) +static inline int blk_pm_resume_queue (const bool pm,struct request_queue *q) { -if (q->dev && (q->rpm_status ==RPM_SUSPENDED || - q->rpm_status ==RPM_SUSPENDING )) -pm_request_resume (q->dev); +if (!q->dev || !blk_queue_pm_only (q)) +return 1 ;/* Nothing to do */ +if (pm && q->rpm_status !=RPM_SUSPENDED )
Quoted 8 lines
block/blk-pm.h line 17
Quoted 4 lines
block/blk-pm.h +3
-0+return 1 ;/* Request allowed */ +pm_request_resume (q->dev); +return 0 ; }Does the above change, especially the " &&
q->rpm_status!=RPM_SUSPENDED" part, make it impossible to bring a suspended device back to theRPM_ACTIVEstate?
Just the opposite -- the change makes it _possible_ for a BLK_MQ_REQ_PREEMPT request to bring a suspended device back to the RPM_ACTIVE state.
Look at the existing code: If pm is true then blk_pm_request_resume() will be skipped, so the device won't be resumed. With this patch -- in particular with the "&& q->rpm_status != RPM_SUSPENDED" part added -- the call won't be skipped and so the resume will take place.
The rather complicated syntax of the wait_event() call in the existing code contributes to this confusion. One of the things my patch tries to do is make the code more straightforward and easier to grasp.
I admit that there are parts to this thing I don't understand. The wait_event() call in blk_queue_enter(), for example: If we are waiting for the device to leave the RPM_SUSPENDED state (or enter the RPM_ACTIVE state), where does q->mq_freeze_wq get woken up? There's no obvious spot in blk_{pre|post}_runtime_resume().
Alan Stern
Quoted 23 lines · Alan Stern
Quoted 10 lines · Martin Kepplinger
Quoted 6 lines · Alan Stern
Martin's best approach would be to add some debugging code to find out why
blk_queue_enter()isn't callingbkl_pm_request_resume(), or why that call doesn't lead topm_request_resume().Alan Stern
Quoted 6 lines
Hi Alan,
blk_queue_enter()always - especially when sd is runtime suspended and I try to mount as above - sets success to be true for me, so never continues down tobkl_pm_request_resume(). All I see is "PM: Removing info for No Bus:sda1".Aha. Looking at this more closely, it's apparent that the code in
blk-core.ccontains a logic bug: It assumes that if theBLK_MQ_REQ_PREEMPTflag is set then the request can be issued regardless of the queue's runtime status. That is not correct when the queue is suspended.
Quoted 6 lines
Below is my attempt to fix this up. I'm not sure that the patch is entirely correct, but it should fix this logic bug. I would appreciate a critical review.
Martin, does this fix the problem?
not quite: mounting works and resuming itself indeed happens now when copying a file, but the I/O itself doesn't, but says "device offline or changed":
[ 167.167615] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 cmd_age=0s [ 167.167630] sd 0:0:0:0: [sda] tag#0 Sense Key : 0x6 [current] [ 167.167638] sd 0:0:0:0: [sda] tag#0 ASC=0x28 ASCQ=0x0 [ 167.167648] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 00 24 c2 00 00 01 00 [ 167.167658] blk_update_request: I/O error, dev sda, sector 9410 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 167.178327] FAT-fs (sda1): FAT read failed (blocknr 1218) [ 167.183895] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 167.189695] blk_update_request: I/O error, dev sda, sector 5101888 op 0x0:(READ) flags 0x80700 phys_seg 8 prio class 0 [ 167.200510] sd 0:0:0:0: [sda] tag#0 device offline or changed
and a later try to copy a file only yields (mostly my own debug prints):
[ 371.110798] blk_queue_enter: wait_event: pm=0 [ 371.300666] scsi_runtime_resume [ 371.303834] scsi_runtime_resume [ 371.307007] scsi_runtime_resume [ 371.310213] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 371.316011] blk_update_request: I/O error, dev sda, sector 5101888 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 372.560690] scsi_runtime_suspend [ 372.563968] scsi_runtime_suspend [ 372.567237] scsi_runtime_suspend
thanks Alan for taking the time and trying to fix this! you're close. what is missing?
martin
Quoted 11 lines · Martin Kepplinger
Quoted 2 lines
Martin, does this fix the problem?
not quite: mounting works and resuming itself indeed happens now when copying a file, but the I/O itself doesn't, but says "device offline or changed":
[ 167.167615] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 cmd_age=0s [ 167.167630] sd 0:0:0:0: [sda] tag#0 Sense Key : 0x6 [current] [ 167.167638] sd 0:0:0:0: [sda] tag#0 ASC=0x28 ASCQ=0x0
That code stands for "Not-ready to ready transition". It isn't really an error, just a notification. The command should have been retried.
Quoted 5 lines
[ 167.167648] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 00 24 c2 00 00 01 00 [ 167.167658] blk_update_request: I/O error, dev sda, sector 9410 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 167.178327] FAT-fs (sda1): FAT read failed (blocknr 1218)
And it should not have failed. Martin or James, any ideas about this?
Quoted 9 lines
[ 167.183895] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 167.189695] blk_update_request: I/O error, dev sda, sector 5101888 op 0x0:(READ) flags 0x80700 phys_seg 8 prio class 0 [ 167.200510] sd 0:0:0:0: [sda] tag#0 device offline or changedand a later try to copy a file only yields (mostly my own debug prints):
Quoted 7 lines
[ 371.110798] blk_queue_enter: wait_event: pm=0 [ 371.300666] scsi_runtime_resume [ 371.303834] scsi_runtime_resume [ 371.307007] scsi_runtime_resume [ 371.310213] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 371.316011] blk_update_request: I/O error, dev sda, sector 5101888 op0x0:(READ) flags
0x0phys_seg1 prio class 0
No way to tell from the log what caused this error.
Quoted 6 lines
[ 372.560690] scsi_runtime_suspend [ 372.563968] scsi_runtime_suspend [ 372.567237] scsi_runtime_suspendthanks Alan for taking the time and trying to fix this! you're close. what is missing?
At this point I don't know.
Alan Stern
Quoted 4 lines · Alan Stern
Aha. Looking at this more closely, it's apparent that the code in
blk-core.ccontains a logic bug: It assumes that if theBLK_MQ_REQ_PREEMPTflag is set then the request can be issued regardless of the queue's runtime status. That is not correct when the queue is suspended.
Are you sure of this? In the past (legacy block layer) no requests were processed for queues in state RPM_SUSPENDED. However, that function and its successor blk_pm_allow_request() are gone. The following code was removed by commit 7cedffec8e75 ("block: Make blk_get_request() block for non-PM requests while suspended").
static struct request *blk_pm_peek_request(struct request_queue *q,
struct request *rq)
{
if (q->dev && (q->rpm_status == RPM_SUSPENDED ||
(q->rpm_status != RPM_ACTIVE && !(rq->cmd_flags & REQ_PM))))
return NULL;
else
return rq;
}Bart.
Quoted 14 lines · Bart Van Assche
Quoted 4 lines · Alan Stern
Aha. Looking at this more closely, it's apparent that the code in
blk-core.ccontains a logic bug: It assumes that if theBLK_MQ_REQ_PREEMPTflag is set then the request can be issued regardless of the queue's runtime status. That is not correct when the queue is suspended.Are you sure of this? In the past (legacy block layer) no requests were processed for queues in state
RPM_SUSPENDED. However, that function and its successorblk_pm_allow_request()are gone. The following code was removed by commit7cedffec8e75("block: Makeblk_get_request()block for non-PM requests while suspended").static struct request *blk_pm_peek_request(struct
request_queue*q,
struct request *rq)
Quoted 7 lines
{
if (q->dev && (q->rpm_status == RPM_SUSPENDED ||
(q->rpm_status != RPM_ACTIVE && !(rq->cmd_flags & REQ_PM))))
return NULL;
else
return rq;
}No, it wasn't. Another routine, blk_pm_allow_request(), was removed by that commit, but almost no other code was deleted. Maybe you're thinking of a different commit?
In any case, I don't understand the point you're trying to make.
Here's what I _do_ understand: While the queue is in the RPM_SUSPENDED state, it can't carry out any requests at all. If a request is added to the queue, the queue must be resumed before the request can be issued to the lower-layer drivers -- no matter what flags are set in the request.
Right now there doesn't seem to be any mechanism for resuming the queue if an REQ_PREEMPT request is added while the queue is suspended.
Alan Stern
Quoted 2 lines · Alan Stern
Right now there doesn't seem to be any mechanism for resuming the queue if an
REQ_PREEMPTrequest is added while the queue is suspended.
I do not agree with the above statement. My understanding is that resuming happens as follows if a request is submitted against a runtime suspended queue owned by a SCSI LLD:
blk_queue_enter()
-> blk_pm_request_resume()
-> pm_request_resume(dev)
-> __pm_runtime_resume(dev, RPM_ASYNC)
-> rpm_resume(dev, RPM_ASYNC)
-> dev->power.request = RPM_REQ_RESUME;
-> queue_work(pm_wq, &dev->power.work)
-> pm_runtime_work()
-> rpm_resume(dev, RPM_NOWAIT)
-> callback = scsi_runtime_resume;
-> rpm_callback(callback, dev);
-> scsi_runtime_resume(dev);
-> sdev_runtime_resume(dev);
-> blk_pre_runtime_resume(sdev->request_queue);
-> q->rpm_status = RPM_RESUMING;
-> sd_resume(dev);
-> sd_start_stop_device(sdkp);
-> sd_pm_ops.runtime_resume == scsi_execute(sdp, START);
-> blk_get_request(..., ..., BLK_MQ_REQ_PREEMPT)
-> blk_mq_alloc_request()
-> blk_queue_enter()
-> __blk_mq_alloc_request()
-> blk_execute_rq()
-> blk_put_request()
-> blk_post_runtime_resume(sdev->request_queue);
-> q->rpm_status = RPM_ACTIVE;
-> pm_runtime_mark_last_busy(q->dev);
-> pm_request_autosuspend(q->dev);Bart.
Quoted 10 lines · Bart Van Assche
Quoted 2 lines · Alan Stern
Right now there doesn't seem to be any mechanism for resuming the queue if an
REQ_PREEMPTrequest is added while the queue is suspended.I do not agree with the above statement. My understanding is that resuming happens as follows if a request is submitted against a runtime suspended queue owned by a SCSI LLD:
blk_queue_enter()
->blk_pm_request_resume()
Assume that BLK_MQ_REQ_PREEMPT is set in flags. Then where exactly does blk_queue_enter(q, flags) call blk_pm_request_resume(q)?
Nowhere, as far as I can see.
Alan Stern
Quoted 28 lines
-> pm_request_resume(dev) -> __pm_runtime_resume(dev, RPM_ASYNC) -> rpm_resume(dev, RPM_ASYNC) -> dev->power.request = RPM_REQ_RESUME; -> queue_work(pm_wq, &dev->power.work) -> pm_runtime_work() -> rpm_resume(dev, RPM_NOWAIT) -> callback = scsi_runtime_resume; -> rpm_callback(callback, dev); -> scsi_runtime_resume(dev); -> sdev_runtime_resume(dev); -> blk_pre_runtime_resume(sdev->request_queue); -> q->rpm_status = RPM_RESUMING; -> sd_resume(dev); -> sd_start_stop_device(sdkp); -> sd_pm_ops.runtime_resume == scsi_execute(sdp, START); -> blk_get_request(..., ..., BLK_MQ_REQ_PREEMPT) -> blk_mq_alloc_request() -> blk_queue_enter() -> __blk_mq_alloc_request() -> blk_execute_rq() -> blk_put_request() -> blk_post_runtime_resume(sdev->request_queue); -> q->rpm_status = RPM_ACTIVE; -> pm_runtime_mark_last_busy(q->dev); -> pm_request_autosuspend(q->dev);Bart.
Quoted 2 lines · Alan Stern
Assume that
BLK_MQ_REQ_PREEMPTis set in flags. Then where exactly doesblk_queue_enter(q, flags)callblk_pm_request_resume(q)?
Please take a look at how the *current* implementation of runtime power management works. Your question is relevant for the old implementation of runtime power management but not for the current implementation.
Bart.
Quoted 7 lines · Bart Van Assche
Quoted 2 lines · Alan Stern
Assume that
BLK_MQ_REQ_PREEMPTis set in flags. Then where exactly doesblk_queue_enter(q, flags)callblk_pm_request_resume(q)?Please take a look at how the *current* implementation of runtime power management works. Your question is relevant for the old implementation of runtime power management but not for the current implementation.
What do you mean by "current"? I have been looking at the implementation in 5.8-rc3 from Linus's tree. Should I look somewhere else?
Alan Stern
Quoted 11 lines · Alan Stern
Quoted 7 lines · Bart Van Assche
Quoted 2 lines · Alan Stern
Assume that
BLK_MQ_REQ_PREEMPTis set in flags. Then where exactly doesblk_queue_enter(q, flags)callblk_pm_request_resume(q)?Please take a look at how the *current* implementation of runtime power management works. Your question is relevant for the old implementation of runtime power management but not for the current implementation.
What do you mean by "current"? I have been looking at the implementation in 5.8-rc3 from Linus's tree. Should I look somewhere else?
Any reply to this, or further concerns about the proposed patch?
I'd like to fix up this API, and it appears that you are the only person to have worked on it significantly in the last two years.
Alan Stern
Quoted 13 lines · Alan Stern
Quoted 11 lines · Alan Stern
Quoted 7 lines · Bart Van Assche
Quoted 2 lines · Alan Stern
Assume that
BLK_MQ_REQ_PREEMPTis set in flags. Then where exactly doesblk_queue_enter(q, flags)callblk_pm_request_resume(q)?Please take a look at how the *current* implementation of runtime power management works. Your question is relevant for the old implementation of runtime power management but not for the current implementation.
What do you mean by "current"? I have been looking at the implementation in 5.8-rc3 from Linus's tree. Should I look somewhere else?
Quoted 7 lines
Any reply to this, or further concerns about the proposed patch?
I'd like to fix up this API, and it appears that you are the only person to have worked on it significantly in the last two years.
Alan Stern
Hi Alan,
Any API cleanup is of course welcome. I just wanted to remind you that the underlying problem: broken block device runtime pm. Your initial proposed fix "almost" did it and mounting works but during file access, it still just looks like a runtime_resume is missing somewhere.
As we need to have that working at some point, I might look into it, but someone who has experience in the block layer can surely do it more efficiently.
to test, turn off polling:
echo 0 > /sys/module/block/parameters/events_dfl_poll_msecs
and enable runtime pm in your (scsi) device: power/autosuspend_delay_ms and of course power/control (set to auto).
thanks,
martin
Quoted 6 lines · Martin Kepplinger
Hi Alan,
Any API cleanup is of course welcome. I just wanted to remind you that the underlying problem: broken block device runtime pm. Your initial proposed fix "almost" did it and mounting works but during file access, it still just looks like a
runtime_resumeis missing somewhere.
Well, I have tested that proposed fix several times, and on my system it's working perfectly. When I stop accessing a drive it autosuspends, and when I access it again it gets resumed and works -- as you would expect.
Quoted 3 lines
As we need to have that working at some point, I might look into it, but someone who has experience in the block layer can surely do it more efficiently.
I suspect that any problems you still face are caused by something else.
Alan Stern
Quoted 8 lines · Alan Stern
Quoted 6 lines · Martin Kepplinger
Hi Alan,
Any API cleanup is of course welcome. I just wanted to remind you that the underlying problem: broken block device runtime pm. Your initial proposed fix "almost" did it and mounting works but during file access, it still just looks like a
runtime_resumeis missing somewhere.
Quoted 4 lines
Well, I have tested that proposed fix several times, and on my system it's working perfectly. When I stop accessing a drive it autosuspends, and when I access it again it gets resumed and works -- as you would expect.
that's weird. when I mount, everything looks good, "sda1". But as soon as I cd to the mountpoint and do "ls" (on another SD card "ls" works but actual file reading leads to the exact same errors), I get:
[ 77.474632] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 cmd_age=0s [ 77.474647] sd 0:0:0:0: [sda] tag#0 Sense Key : 0x6 [current] [ 77.474655] sd 0:0:0:0: [sda] tag#0 ASC=0x28 ASCQ=0x0 [ 77.474667] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 00 60 40 00 00 01 00 [ 77.474678] blk_update_request: I/O error, dev sda, sector 24640 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 77.485836] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 77.491628] blk_update_request: I/O error, dev sda, sector 24641 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 77.502275] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 77.508051] blk_update_request: I/O error, dev sda, sector 24642 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 77.518651] sd 0:0:0:0: [sda] tag#0 device offline or changed (...) [ 77.947653] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 77.953434] FAT-fs (sda1): Directory bread(block 16448) failed [ 77.959333] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 77.965118] FAT-fs (sda1): Directory bread(block 16449) failed [ 77.971014] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 77.976802] FAT-fs (sda1): Directory bread(block 16450) failed [ 77.982698] sd 0:0:0:0: [sda] tag#0 device offline or changed (...) [ 78.384929] FAT-fs (sda1): Filesystem has been set read-only [ 103.070973] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 103.076751] print_req_error: 118 callbacks suppressed [ 103.076760] blk_update_request: I/O error, dev sda, sector 9748 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0 [ 103.087428] Buffer I/O error on dev sda1, logical block 1556, lost async page write [ 103.095309] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 103.101123] blk_update_request: I/O error, dev sda, sector 17162 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0 [ 103.111883] Buffer I/O error on dev sda1, logical block 8970, lost
async page write
Quoted 7 lines
Quoted 4 lines
As we need to have that working at some point, I might look into it, but someone who has experience in the block layer can surely do it more efficiently.
I suspect that any problems you still face are caused by something else.
I then formatted sda1 to ext2 (on the runtime suspend system testing your patch) and that seems to have worked!
Again accessing the mountpoint then yield the very same "device offline or changed" errors.
What kind of device are you testing? You should be easily able to reproduce this using an "sd" device.
The problems must lie in the different other drivers we use I guess.
martin
Quoted 2 lines
Alan Stern
Quoted 18 lines · Martin Kepplinger
Quoted 8 lines · Alan Stern
Quoted 6 lines · Martin Kepplinger
Hi Alan,
Any API cleanup is of course welcome. I just wanted to remind you that the underlying problem: broken block device runtime pm. Your initial proposed fix "almost" did it and mounting works but during file access, it still just looks like a
runtime_resumeis missing somewhere.Quoted 4 lines
Well, I have tested that proposed fix several times, and on my system it's working perfectly. When I stop accessing a drive it autosuspends, and when I access it again it gets resumed and works -- as you would expect.
that's weird. when I mount, everything looks good, "sda1". But as soon as I cd to the mountpoint and do "ls" (on another SD card "ls" works but actual file reading leads to the exact same errors), I get:
Quoted 6 lines
[ 77.474632] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 cmd_age=0s [ 77.474647] sd 0:0:0:0: [sda] tag#0 Sense Key : 0x6 [current] [ 77.474655] sd 0:0:0:0: [sda] tag#0 ASC=0x28 ASCQ=0x0 [ 77.474667] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 00 6040 00 00 01 00
This error report comes from the SCSI layer, not the block layer.
Quoted 29 lines
[ 77.474678] blk_update_request: I/O error, dev sda, sector 24640 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 77.485836] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 77.491628] blk_update_request: I/O error, dev sda, sector 24641 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 77.502275] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 77.508051] blk_update_request: I/O error, dev sda, sector 24642 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 77.518651] sd 0:0:0:0: [sda] tag#0 device offline or changed (...) [ 77.947653] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 77.953434] FAT-fs (sda1): Directory bread(block 16448) failed [ 77.959333] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 77.965118] FAT-fs (sda1): Directory bread(block 16449) failed [ 77.971014] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 77.976802] FAT-fs (sda1): Directory bread(block 16450) failed [ 77.982698] sd 0:0:0:0: [sda] tag#0 device offline or changed (...) [ 78.384929] FAT-fs (sda1): Filesystem has been set read-only [ 103.070973] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 103.076751] print_req_error: 118 callbacks suppressed [ 103.076760] blk_update_request: I/O error, dev sda, sector 9748 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0 [ 103.087428] Buffer I/O error on dev sda1, logical block 1556, lost async page write [ 103.095309] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 103.101123] blk_update_request: I/O error, dev sda, sector 17162 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0 [ 103.111883] Buffer I/O error on dev sda1, logical block 8970, lost
Quoted 1 line
async page write
I can't tell why you're getting that error. In one of my tests the device returned the same kind of error status (Sense Key = 6, ASC = 0x28) but the operation was then retried successfully. Perhaps the problem lies in the device you are testing.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
As we need to have that working at some point, I might look into it, but someone who has experience in the block layer can surely do it more efficiently.
I suspect that any problems you still face are caused by something else.
Quoted 8 lines
I then formatted sda1 to ext2 (on the runtime suspend system testing your patch) and that seems to have worked!
Again accessing the mountpoint then yield the very same "device offline or changed" errors.
What kind of device are you testing? You should be easily able to reproduce this using an "sd" device.
I tested two devices: a SanDisk Cruzer USB flash drive and a g-mass-storage gadget running under dummy-hcd. They each showed up as /dev/sdb on my system.
I haven't tried testing with an SD card. If you have any specific sequence of commands you would like me to run, let me know.
Quoted 1 line
The problems must lie in the different other drivers we use I guess.
Or the devices. Have you tried testing with a USB flash drive?
Alan Stern
Alan,
Quoted 8 lines
Quoted 6 lines
[ 77.474632] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 cmd_age=0s [ 77.474647] sd 0:0:0:0: [sda] tag#0 Sense Key : 0x6 [current] [ 77.474655] sd 0:0:0:0: [sda] tag#0 ASC=0x28 ASCQ=0x0 [ 77.474667] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 00 6040 00 00 01 00
This error report comes from the SCSI layer, not the block layer.
This the device telling us that the media (SD card?) has changed.
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Quoted 10 lines · Martin K. Petersen
Alan,
Quoted 7 lines
Quoted 6 lines
[ 77.474632] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 cmd_age=0s [ 77.474647] sd 0:0:0:0: [sda] tag#0 Sense Key : 0x6 [current] [ 77.474655] sd 0:0:0:0: [sda] tag#0 ASC=0x28 ASCQ=0x0 [ 77.474667] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 00 6040 00 00 01 00
Quoted 3 lines
Quoted 1 line
This error report comes from the SCSI layer, not the block layer.
This the device telling us that the media (SD card?) has changed.
Ah yes, thank you. I knew that SK=6 ASC=0x28 meant "Not Ready to Ready Transition", but I had forgotten the "(Media May Have Changed)" part.
This makes sense and is a reasonable thing to see, since many SD card readers lose track of whether or not the card has been changed when they go into suspend.
Alan Stern
Quoted 22 lines · Alan Stern
Quoted 21 lines · Martin Kepplinger
Quoted 10 lines · Alan Stern
Quoted 7 lines · Martin Kepplinger
Hi Alan,
Any API cleanup is of course welcome. I just wanted to remind you that the underlying problem: broken block device runtime pm. Your initial proposed fix "almost" did it and mounting works but during file access, it still just looks like a
runtime_resumeis missing somewhere.Quoted 4 lines
Well, I have tested that proposed fix several times, and on my system it's working perfectly. When I stop accessing a drive it autosuspends, and when I access it again it gets resumed and works -- as you would expect.
that's weird. when I mount, everything looks good, "sda1". But as soon as I cd to the mountpoint and do "ls" (on another SD card "ls" works but actual file reading leads to the exact same errors), I get:
Quoted 8 lines
Quoted 6 lines
[ 77.474632] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 cmd_age=0s [ 77.474647] sd 0:0:0:0: [sda] tag#0 Sense Key : 0x6 [current] [ 77.474655] sd 0:0:0:0: [sda] tag#0 ASC=0x28 ASCQ=0x0 [ 77.474667] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 0060 40 00 00 01 00
This error report comes from the SCSI layer, not the block layer.
That sense code means "NOT READY TO READY CHANGE, MEDIUM MAY HAVE CHANGED" so it sounds like it something we should be ignoring. Usually this signals a problem, like you changed the medium manually (ejected the CD). But in this case you can tell us to expect this by setting
sdev->expecting_cc_ua
And we'll retry. I think you need to set this on all resumed devices.
James
James Bottomley
maintainerQuoted 38 lines · James Bottomley
Quoted 22 lines · Alan Stern
Quoted 21 lines · Martin Kepplinger
Quoted 10 lines · Alan Stern
Quoted 7 lines · Martin Kepplinger
Hi Alan,
Any API cleanup is of course welcome. I just wanted to remind you that the underlying problem: broken block device runtime pm. Your initial proposed fix "almost" did it and mounting works but during file access, it still just looks like a
runtime_resumeis missing somewhere.Quoted 4 lines
Well, I have tested that proposed fix several times, and on my system it's working perfectly. When I stop accessing a drive it autosuspends, and when I access it again it gets resumed and works -- as you would expect.
that's weird. when I mount, everything looks good, "sda1". But as soon as I cd to the mountpoint and do "ls" (on another SD card "ls" works but actual file reading leads to the exact same errors), I get:
Quoted 8 lines
Quoted 6 lines
[ 77.474632] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 cmd_age=0s [ 77.474647] sd 0:0:0:0: [sda] tag#0 Sense Key : 0x6 [current] [ 77.474655] sd 0:0:0:0: [sda] tag#0 ASC=0x28 ASCQ=0x0 [ 77.474667] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 0000 60 40 00 00 01 00
This error report comes from the SCSI layer, not the block layer.
That sense code means "NOT READY TO READY CHANGE, MEDIUM MAY HAVE CHANGED" so it sounds like it something we should be ignoring. Usually this signals a problem, like you changed the medium manually (ejected the CD). But in this case you can tell us to expect this by setting
Quoted 4 lines
sdev->expecting_cc_uaAnd we'll retry. I think you need to set this on all resumed devices.
Actually, it's not quite that easy, we filter out this ASC/ASCQ combination from the check because we should never ignore medium might have changed events on running devices. We could ignore it if we had a flag to say the power has been yanked (perhaps an additional sdev flag you set on resume) but we would still miss the case where you really had powered off the drive and then changed the media ... if you can regard this as the user's problem, then we might have a solution.
James
Quoted 52 lines · James Bottomley
Quoted 38 lines · James Bottomley
Quoted 22 lines · Alan Stern
Quoted 21 lines · Martin Kepplinger
Quoted 10 lines · Alan Stern
Quoted 7 lines · Martin Kepplinger
Hi Alan,
Any API cleanup is of course welcome. I just wanted to remind you that the underlying problem: broken block device runtime pm. Your initial proposed fix "almost" did it and mounting works but during file access, it still just looks like a
runtime_resumeis missing somewhere.Quoted 4 lines
Well, I have tested that proposed fix several times, and on my system it's working perfectly. When I stop accessing a drive it autosuspends, and when I access it again it gets resumed and works -- as you would expect.
that's weird. when I mount, everything looks good, "sda1". But as soon as I cd to the mountpoint and do "ls" (on another SD card "ls" works but actual file reading leads to the exact same errors), I get:
Quoted 8 lines
Quoted 6 lines
[ 77.474632] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 cmd_age=0s [ 77.474647] sd 0:0:0:0: [sda] tag#0 Sense Key : 0x6 [current] [ 77.474655] sd 0:0:0:0: [sda] tag#0 ASC=0x28 ASCQ=0x0 [ 77.474667] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 0000 60 40 00 00 01 00
This error report comes from the SCSI layer, not the block layer.
That sense code means "NOT READY TO READY CHANGE, MEDIUM MAY HAVE CHANGED" so it sounds like it something we should be ignoring. Usually this signals a problem, like you changed the medium manually (ejected the CD). But in this case you can tell us to expect this by setting
Quoted 4 lines
sdev->expecting_cc_uaAnd we'll retry. I think you need to set this on all resumed devices.
Actually, it's not quite that easy, we filter out this ASC/ASCQ combination from the check because we should never ignore medium might have changed events on running devices. We could ignore it if we had a flag to say the power has been yanked (perhaps an additional sdev flag you set on resume) but we would still miss the case where you really had powered off the drive and then changed the media ... if you can regard this as the user's problem, then we might have a solution.
Quoted 2 lines
James
oh I see what you mean now, thanks for the ellaboration.
if I do the following change, things all look normal and runtime pm works. I'm not 100% sure if just setting expecting_cc_ua in resume() is "correct" but that looks like it is what you're talking about:
(note that this is of course with the one block layer diff applied that Alan posted a few emails back)
drivers/scsi/scsi_error.c +2 -10
@@ -554,16 +554,8 @@ int scsi_check_sense(struct scsi_cmnd *scmd)* so that we can deal with it there. */ if (scmd->device->expecting_cc_ua) { -/* -* Because some device does not queue unit -* attentions correctly, we carefully check -* additional sense code and qualifier so as -* not to squash media change unit attention. -*/ -if (sshdr.asc !=0x28 || sshdr.ascq !=0x00 ) { - scmd->device->expecting_cc_ua =0 ; -return NEEDS_RETRY ; - } + scmd->device->expecting_cc_ua =0 ; +return NEEDS_RETRY ; }/* * we might also expect a cc/ua if another LUN on the target
drivers/scsi/sd.c +0 -0
@@ -3642,6 +3642,8 @@ static int sd_resume(struct device *dev)if (!sdkp)/* E.g.: runtime resume at the start of
sd_probe() */
return 0;
+ sdkp->device->expecting_cc_ua = 1;
+
if (!sdkp->device->manage_start_stop)
return 0;James Bottomley
maintainerQuoted 3 lines · Martin Kepplinger
Quoted 2 lines · James Bottomley
Quoted 1 line · James Bottomley
On Wed, 2020-07-29 at 10:32 -0400, Alan Stern wrote:
[...]
Quoted 45 lines
Quoted 23 lines
Quoted 9 lines
Quoted 2 lines
This error report comes from the SCSI layer, not the block layer.
That sense code means "NOT READY TO READY CHANGE, MEDIUM MAY HAVE CHANGED" so it sounds like it something we should be ignoring. Usually this signals a problem, like you changed the medium manually (ejected the CD). But in this case you can tell us to expect this by setting
Quoted 4 lines
sdev->expecting_cc_uaAnd we'll retry. I think you need to set this on all resumed devices.
Actually, it's not quite that easy, we filter out this ASC/ASCQ combination from the check because we should never ignore medium might have changed events on running devices. We could ignore it if we had a flag to say the power has been yanked (perhaps an additional sdev flag you set on resume) but we would still miss the case where you really had powered off the drive and then changed the media ... if you can regard this as the user's problem, then we might have a solution.
Quoted 2 lines
James
oh I see what you mean now, thanks for the ellaboration.
if I do the following change, things all look normal and runtime pm works. I'm not 100% sure if just setting
expecting_cc_uainresume()is "correct" but that looks like it is what you're talking about:(note that this is of course with the one block layer diff applied that Alan posted a few emails back)
drivers/scsi/scsi_error.c +0
-3@@ -554,16 +554,8 @@ int scsi_check_sense(struct scsi_cmnd *scmd)* so that we can deal with it there. */ if (scmd->device->expecting_cc_ua) { -/* -* Because some device does not queue unit -* attentions correctly, we carefully check
drivers/scsi/scsi_error.c int scsi_check_sense(struct scsi_cmnd *scmd) · line 559
Quoted 10 lines
drivers/scsi/scsi_error.c +2
-7-* additional sense code and qualifier so as -* not to squash media change unit attention. -*/ -if (sshdr.asc !=0x28 || sshdr.ascq !=0x00 ) { - scmd->device->expecting_cc_ua =0 ; -return NEEDS_RETRY ; - } + scmd->device->expecting_cc_ua =0 ; +return NEEDS_RETRY ;
Well, yes, but you can't do this because it would lose us media change events in the non-suspend/resume case which we really don't want. That's why I was suggesting a new flag.
James
Am 29. Juli 2020 17:44:42 MESZ schrieb James Bottomley <[email protected]>:
Quoted 65 lines · Martin Kepplinger
Quoted 3 lines · James Bottomley
Quoted 2 lines · James Bottomley
Quoted 1 line · Alan Stern
[...]
Quoted 45 lines
Quoted 23 lines
Quoted 9 lines
Quoted 2 lines
This error report comes from the SCSI layer, not the block layer.
That sense code means "NOT READY TO READY CHANGE, MEDIUM MAY HAVE CHANGED" so it sounds like it something we should be ignoring. Usually this signals a problem, like you changed the medium manually (ejected the CD). But in this case you can tell us to expect this by setting
Quoted 4 lines
sdev->expecting_cc_uaAnd we'll retry. I think you need to set this on all resumed devices.
Actually, it's not quite that easy, we filter out this ASC/ASCQ combination from the check because we should never ignore medium might have changed events on running devices. We could ignore it if we had a flag to say the power has been yanked (perhaps an additional sdev flag you set on resume) but we would still miss the case where you really had powered off the drive and then changed the media ... if you can regard this as the user's problem, then we might have a solution.
Quoted 2 lines
James
oh I see what you mean now, thanks for the ellaboration.
if I do the following change, things all look normal and runtime pm works. I'm not 100% sure if just setting
expecting_cc_uainresume()is "correct" but that looks like it is what you're talking about:(note that this is of course with the one block layer diff applied that Alan posted a few emails back)
drivers/scsi/scsi_error.c +0
-3@@ -554,16 +554,8 @@ int scsi_check_sense(struct scsi_cmnd *scmd)* so that we can deal with it there. */ if (scmd->device->expecting_cc_ua) { -/* -* Because some device does not queue unit -* attentions correctly, we carefully check drivers/scsi/scsi_error.c int scsi_check_sense(struct scsi_cmnd *scmd) · line 559
Quoted 10 lines
drivers/scsi/scsi_error.c +2
-7-* additional sense code and qualifier so as -* not to squash media change unit attention. -*/ -if (sshdr.asc !=0x28 || sshdr.ascq !=0x00 ) { - scmd->device->expecting_cc_ua =0 ; -return NEEDS_RETRY ; - } + scmd->device->expecting_cc_ua =0 ; +return NEEDS_RETRY ;Well, yes, but you can't do this because it would lose us media change events in the non-suspend/resume case which we really don't want. That's why I was suggesting a new flag.
Quoted 1 line
James
also if I set expecting_cc_ua in resume() only, like I did?
Signature · 3 lines
Martin Kepplinger xmpp: [email protected] Sent from mobile.
Quoted 68 lines · Martin Kepplinger
Am 29. Juli 2020 17:44:42 MESZ schrieb James Bottomley <[email protected]>:
Quoted 65 lines · Martin Kepplinger
Quoted 3 lines · James Bottomley
Quoted 2 lines · James Bottomley
Quoted 1 line · Alan Stern
[...]
Quoted 45 lines
Quoted 23 lines
Quoted 9 lines
Quoted 2 lines
This error report comes from the SCSI layer, not the block layer.
That sense code means "NOT READY TO READY CHANGE, MEDIUM MAY HAVE CHANGED" so it sounds like it something we should be ignoring. Usually this signals a problem, like you changed the medium manually (ejected the CD). But in this case you can tell us to expect this by setting
Quoted 4 lines
sdev->expecting_cc_uaAnd we'll retry. I think you need to set this on all resumed devices.
Actually, it's not quite that easy, we filter out this ASC/ASCQ combination from the check because we should never ignore medium might have changed events on running devices. We could ignore it if we had a flag to say the power has been yanked (perhaps an additional sdev flag you set on resume) but we would still miss the case where you really had powered off the drive and then changed the media ... if you can regard this as the user's problem, then we might have a solution.
Quoted 2 lines
James
oh I see what you mean now, thanks for the ellaboration.
if I do the following change, things all look normal and runtime pm works. I'm not 100% sure if just setting
expecting_cc_uainresume()is "correct" but that looks like it is what you're talking about:(note that this is of course with the one block layer diff applied that Alan posted a few emails back)
drivers/scsi/scsi_error.c +0
-3@@ -554,16 +554,8 @@ int scsi_check_sense(struct scsi_cmnd *scmd)* so that we can deal with it there. */ if (scmd->device->expecting_cc_ua) { -/* -* Because some device does not queue unit -* attentions correctly, we carefully check drivers/scsi/scsi_error.c int scsi_check_sense(struct scsi_cmnd *scmd) · line 559
Quoted 10 lines
drivers/scsi/scsi_error.c +2
-7-* additional sense code and qualifier so as -* not to squash media change unit attention. -*/ -if (sshdr.asc !=0x28 || sshdr.ascq !=0x00 ) { - scmd->device->expecting_cc_ua =0 ; -return NEEDS_RETRY ; - } + scmd->device->expecting_cc_ua =0 ; +return NEEDS_RETRY ;Well, yes, but you can't do this because it would lose us media change events in the non-suspend/resume case which we really don't want. That's why I was suggesting a new flag.
Quoted 3 lines
Quoted 1 line
James
also if I set
expecting_cc_uainresume()only, like I did?
That wouldn't make any difference. The information sent by your card reader has sshdr.asc == 0x28 and sshdr.ascq == 0x00 (you can see it in the log). So because of the code here in scsi_check_sense(), which you can't change, the Unit Attention sent by the card reader would not be retried even if you do set the flag in resume().
Alan Stern
James Bottomley
maintainerQuoted 1 line · Alan Stern
On Wed, Jul 29, 2020 at 06:43:48PM +0200, Martin Kepplinger wrote:
[...]
Quoted 33 lines
Quoted 33 lines
Quoted 32 lines
Quoted 22 lines
drivers/scsi/scsi_error.c +0
-2@@ -554,16 +554,8 @@ int scsi_check_sense(struct scsi_cmnd*scmd) * so that we can deal with it there. */ if (scmd->device->expecting_cc_ua) { -/* -* Because some device does not queue unit
- * attentions correctly, we carefully
check
- * additional sense code and qualifier
so as
- * not to squash media change unit
attention.drivers/scsi/scsi_error.c +0
-2-*/ - if (sshdr.asc != 0x28 || sshdr.ascq !=
0x00)
{
-scmd->device->expecting_cc_ua=
0;drivers/scsi/scsi_error.c int scsi_check_sense(struct scsi_cmnd *scmd) · line 558
Quoted 4 lines
drivers/scsi/scsi_error.c +2
-2-return NEEDS_RETRY ; - } + scmd->device->expecting_cc_ua =0 ; +return NEEDS_RETRY ;Well, yes, but you can't do this because it would lose us media change events in the non-suspend/resume case which we really don't want. That's why I was suggesting a new flag.
James
Quoted 7 lines
Quoted 1 line
also if I set
expecting_cc_uainresume()only, like I did?That wouldn't make any difference. The information sent by your card reader has sshdr.asc ==
0x28and sshdr.ascq ==0x00(you can see it in the log). So because of the code here inscsi_check_sense(), which you can't change, the Unit Attention sent by the card reader would not be retried even if you do set the flag inresume().
But if we had a new flag, like expecting_media_change, you could set that in resume and we could condition the above test in the code on it and reset it and do a retry if it gets set. I'm not saying we have to do it this way, but it sounds like we have to do something in the kernel, so I think the flag will become necessary but there might be a bit of policy based dance around how it gets set in the kernel (to avoid losing accurate media change events).
James
Quoted 52 lines · James Bottomley
Quoted 1 line · Alan Stern
On Wed, Jul 29, 2020 at 06:43:48PM +0200, Martin Kepplinger wrote:
[...]
Quoted 33 lines
Quoted 33 lines
Quoted 32 lines
Quoted 22 lines
drivers/scsi/scsi_error.c +0
-2@@ -554,16 +554,8 @@ int scsi_check_sense(struct scsi_cmnd*scmd) * so that we can deal with it there. */ if (scmd->device->expecting_cc_ua) { -/* -* Because some device does not queue unit
- * attentions correctly, we carefully
check
- * additional sense code and qualifier
so as
- * not to squash media change unit
attention.drivers/scsi/scsi_error.c +0
-2-*/ - if (sshdr.asc != 0x28 || sshdr.ascq !=
0x00)
{
-scmd->device->expecting_cc_ua=
0;drivers/scsi/scsi_error.c
Quoted 4 lines
drivers/scsi/scsi_error.c +2
-2-return NEEDS_RETRY ; - } + scmd->device->expecting_cc_ua =0 ; +return NEEDS_RETRY ;Well, yes, but you can't do this because it would lose us media change events in the non-suspend/resume case which we really don't want. That's why I was suggesting a new flag.
James
Quoted 7 lines
Quoted 1 line
also if I set
expecting_cc_uainresume()only, like I did?That wouldn't make any difference. The information sent by your card reader has sshdr.asc ==
0x28and sshdr.ascq ==0x00(you can see it in the log). So because of the code here inscsi_check_sense(), which you can't change, the Unit Attention sent by the card reader would not be retried even if you do set the flag inresume().But if we had a new flag, like
expecting_media_change, you could set that in resume and we could condition the above test in the code on it and reset it and do a retry if it gets set. I'm not saying we have to do it this way, but it sounds like we have to do something in the kernel, so I think the flag will become necessary but there might be a bit of policy based dance around how it gets set in the kernel (to avoid losing accurate media change events).
Quoted 2 lines
James
Maybe I should just start a new discussion with a patch, but the below is what makes sense to me (when I understand you correctly) and seems to work. I basically add a new flag, so that the old flags behave unchanged and only call it during *runtime* resume for SD cards:
drivers/scsi/scsi_error.c +0 -0
@@ -553,15 +553,21 @@ int scsi_check_sense(struct scsi_cmnd *scmd)* information that we should pass up to the upper-level
driver
* so that we can deal with it there.
*/
- if (scmd->device->expecting_cc_ua) {
+ if (scmd->device->expecting_cc_ua ||
+ scmd->device->expecting_media_change) {
/*
* Because some device does not queue unit
* attentions correctly, we carefully check
* additional sense code and qualifier so as
- * not to squash media change unit attention.
+ * not to squash media change unit attention;
+ * unless expecting_media_change is set, indicating
+ * that the media (most likely) didn't change
+ * but a device only believes so (for example
+ * because of suspend/resume).
*/- if (sshdr.asc != 0x28 || sshdr.ascq != 0x00) {
- scmd->device->expecting_cc_ua = 0;
+ if ((sshdr.asc != 0x28 || sshdr.ascq != 0x00) ||
+ scmd->device->expecting_media_change) {
+ scmd->device->expecting_media_change = 0;
return NEEDS_RETRY;
}
}drivers/scsi/sd.c +7 -1
@@ -114,6 +114,7 @@ static void sd_shutdown(struct device *);static int sd_suspend_system (struct device *);static int sd_suspend_runtime (struct device *);static int sd_resume (struct device *); +static int sd_resume_runtime (struct device *);static void sd_rescan (struct device *);static blk_status_t sd_init_command (struct scsi_cmnd *SCpnt);static void sd_uninit_command (struct scsi_cmnd *SCpnt);
@@ -574,7 +575,7 @@ static const struct dev_pm_ops sd_pm_ops = { .poweroff = sd_suspend_system, .restore = sd_resume, .runtime_suspend = sd_suspend_runtime, - .runtime_resume = sd_resume, + .runtime_resume = sd_resume_runtime, };static struct scsi_driver sd_template = {
@@ -3652,6 +3653,21 @@ static int sd_resume(struct device *dev)return ret; } +static int sd_resume_runtime (struct device *dev) +{ +struct scsi_disk *sdkp =dev_get_drvdata (dev); + +/* Some SD cardreaders report media change when resuming from
suspend
+ * because they can't keep track during suspend. */
+
+ /* XXX This is not unproblematic though: We won't notice when a card
+ * was really changed during runtime suspend! We basically rely
on users
+ * to unmount or suspend before doing so. */
+ sdkp->device->expecting_media_change = 1;
+
+ return sd_resume(dev);
+}
+
/**
* init_sd - entry point for this driver (both when built in or when
* a module).
include/scsi/scsi_device.h +1 -0
@@ -169,6 +169,8 @@ struct scsi_device {* this device */ unsigned expecting_cc_ua:1; /* Expecting a CHECK_CONDITION/UNIT_ATTN* because we did a bus reset. */ + unsigned expecting_media_change:1; /* Expecting media change
ASC/ASCQ
+ when it actually doesn't
change */
unsigned use_10_for_rw:1; /* first try 10-byte read / write */ unsigned use_10_for_ms:1; /* first try 10-byte mode sense/select */ unsigned set_dbd_for_ms:1; /* Set "DBD" field in mode sense */
Quoted 78 lines · Martin Kepplinger
Quoted 52 lines · James Bottomley
Quoted 1 line · Alan Stern
On Wed, Jul 29, 2020 at 06:43:48PM +0200, Martin Kepplinger wrote:
[...]
Quoted 33 lines
Quoted 33 lines
Quoted 32 lines
Quoted 22 lines
drivers/scsi/scsi_error.c +0
-2@@ -554,16 +554,8 @@ int scsi_check_sense(struct scsi_cmnd*scmd) * so that we can deal with it there. */ if (scmd->device->expecting_cc_ua) { -/* -* Because some device does not queue unit
- * attentions correctly, we carefully
check
- * additional sense code and qualifier
so as
- * not to squash media change unit
attention.drivers/scsi/scsi_error.c +0
-2-*/ - if (sshdr.asc != 0x28 || sshdr.ascq !=
0x00)
{
-scmd->device->expecting_cc_ua=
0;drivers/scsi/scsi_error.c
Quoted 4 lines
drivers/scsi/scsi_error.c +2
-2-return NEEDS_RETRY ; - } + scmd->device->expecting_cc_ua =0 ; +return NEEDS_RETRY ;Well, yes, but you can't do this because it would lose us media change events in the non-suspend/resume case which we really don't want. That's why I was suggesting a new flag.
James
Quoted 7 lines
Quoted 1 line
also if I set
expecting_cc_uainresume()only, like I did?That wouldn't make any difference. The information sent by your card reader has sshdr.asc ==
0x28and sshdr.ascq ==0x00(you can see it in the log). So because of the code here inscsi_check_sense(), which you can't change, the Unit Attention sent by the card reader would not be retried even if you do set the flag inresume().But if we had a new flag, like
expecting_media_change, you could set that in resume and we could condition the above test in the code on it and reset it and do a retry if it gets set. I'm not saying we have to do it this way, but it sounds like we have to do something in the kernel, so I think the flag will become necessary but there might be a bit of policy based dance around how it gets set in the kernel (to avoid losing accurate media change events).Quoted 2 lines
James
Maybe I should just start a new discussion with a patch, but the below is what makes sense to me (when I understand you correctly) and seems to work. I basically add a new flag, so that the old flags behave unchanged and only call it during *runtime* resume for SD cards:
drivers/scsi/scsi_error.c +0
-0@@ -553,15 +553,21 @@ int scsi_check_sense(struct scsi_cmnd *scmd)* information that we should pass up to the upper-level driver
* so that we can deal with it there. */drivers/scsi/scsi_error.c +3
-2-if (scmd->device->expecting_cc_ua) { +if (scmd->device->expecting_cc_ua || + scmd->device->expecting_media_change) {/* * Because some device does not queue unit * attentions correctly, we carefully check * additional sense code and qualifier so as -* not to squash media change unit attention. +* not to squash media change unit attention;
drivers/scsi/scsi_error.c
Quoted 10 lines
drivers/scsi/scsi_error.c +7
-2+* unless expecting_media_change is set, indicating +* that the media (most likely) didn't change +* but a device only believes so (for example +* because of suspend/resume). */ -if (sshdr.asc !=0x28 || sshdr.ascq !=0x00 ) { - scmd->device->expecting_cc_ua =0 ; +if ((sshdr.asc !=0x28 || sshdr.ascq !=0x00 ) || + scmd->device->expecting_media_change) { + scmd->device->expecting_media_change =0 ;
oops, I missed expecting_cc_ua here, but you get the idea.
Quoted 64 lines
return NEEDS_RETRY; } }drivers/scsi/sd.c +7
-1@@ -114,6 +114,7 @@ static void sd_shutdown(struct device *);static int sd_suspend_system (struct device *);static int sd_suspend_runtime (struct device *);static int sd_resume (struct device *); +static int sd_resume_runtime (struct device *);static void sd_rescan (struct device *);static blk_status_t sd_init_command (struct scsi_cmnd *SCpnt);static void sd_uninit_command (struct scsi_cmnd *SCpnt);@@ -574,7 +575,7 @@ static const struct dev_pm_ops sd_pm_ops = { .poweroff = sd_suspend_system, .restore = sd_resume, .runtime_suspend = sd_suspend_runtime, - .runtime_resume = sd_resume, + .runtime_resume = sd_resume_runtime, };static struct scsi_driver sd_template = {@@ -3652,6 +3653,21 @@ static int sd_resume(struct device *dev)return ret; } +static int sd_resume_runtime (struct device *dev) +{ +struct scsi_disk *sdkp =dev_get_drvdata (dev); + +/* Some SD cardreaders report media change when resuming from suspend
drivers/scsi/sd.c +4
-0+* because they can't keep track during suspend. */ + +/* XXX This is not unproblematic though: We won't notice when a card +* was really changed during runtime suspend! We basically rely on users
drivers/scsi/sd.c +6
-0+* to unmount or suspend before doing so. */ + sdkp->device->expecting_media_change =1 ; + +return sd_resume (dev); +} +/** * init_sd - entry point for this driver (both when built in or when * a module). include/scsi/scsi_device.h +1
-0@@ -169,6 +169,8 @@ struct scsi_device {* this device */ unsigned expecting_cc_ua:1; /* Expecting a CHECK_CONDITION/UNIT_ATTN* because we did a bus reset. */ + unsigned expecting_media_change:1; /* Expecting media changeASC/ASCQ
+ when it actually doesn't
change */
unsigned use_10_for_rw:1; /* first try 10-byte read / write */ unsigned use_10_for_ms:1; /* first try 10-byte mode sense/select */ unsigned set_dbd_for_ms:1; /* Set "DBD" field in mode sense */
Quoted 89 lines · Martin Kepplinger
Maybe I should just start a new discussion with a patch, but the below is what makes sense to me (when I understand you correctly) and seems to work. I basically add a new flag, so that the old flags behave unchanged and only call it during *runtime* resume for SD cards:
drivers/scsi/scsi_error.c +0
-0@@ -553,15 +553,21 @@ int scsi_check_sense(struct scsi_cmnd *scmd)* information that we should pass up to the upper-level driver
* so that we can deal with it there. */drivers/scsi/scsi_error.c +10
-4-if (scmd->device->expecting_cc_ua) { +if (scmd->device->expecting_cc_ua || + scmd->device->expecting_media_change) {/* * Because some device does not queue unit * attentions correctly, we carefully check * additional sense code and qualifier so as -* not to squash media change unit attention. +* not to squash media change unit attention; +* unless expecting_media_change is set, indicating +* that the media (most likely) didn't change +* but a device only believes so (for example +* because of suspend/resume). */ -if (sshdr.asc !=0x28 || sshdr.ascq !=0x00 ) { - scmd->device->expecting_cc_ua =0 ; +if ((sshdr.asc !=0x28 || sshdr.ascq !=0x00 ) || + scmd->device->expecting_media_change) { + scmd->device->expecting_media_change =0 ;return NEEDS_RETRY ; } }drivers/scsi/sd.c +7
-1@@ -114,6 +114,7 @@ static void sd_shutdown(struct device *);static int sd_suspend_system (struct device *);static int sd_suspend_runtime (struct device *);static int sd_resume (struct device *); +static int sd_resume_runtime (struct device *);static void sd_rescan (struct device *);static blk_status_t sd_init_command (struct scsi_cmnd *SCpnt);static void sd_uninit_command (struct scsi_cmnd *SCpnt);@@ -574,7 +575,7 @@ static const struct dev_pm_ops sd_pm_ops = { .poweroff = sd_suspend_system, .restore = sd_resume, .runtime_suspend = sd_suspend_runtime, - .runtime_resume = sd_resume, + .runtime_resume = sd_resume_runtime, };static struct scsi_driver sd_template = {@@ -3652,6 +3653,21 @@ static int sd_resume(struct device *dev)return ret; } +static int sd_resume_runtime (struct device *dev) +{ +struct scsi_disk *sdkp =dev_get_drvdata (dev); + +/* Some SD cardreaders report media change when resuming from suspend
drivers/scsi/sd.c +4
-0+* because they can't keep track during suspend. */ + +/* XXX This is not unproblematic though: We won't notice when a card +* was really changed during runtime suspend! We basically rely on users
drivers/scsi/sd.c +6
-0+* to unmount or suspend before doing so. */ + sdkp->device->expecting_media_change =1 ; + +return sd_resume (dev); +} +/** * init_sd - entry point for this driver (both when built in or when * a module). include/scsi/scsi_device.h +1
-0@@ -169,6 +169,8 @@ struct scsi_device {* this device */ unsigned expecting_cc_ua:1; /* Expecting a CHECK_CONDITION/UNIT_ATTN* because we did a bus reset. */ + unsigned expecting_media_change:1; /* Expecting media change
Quoted 6 lines
ASC/ASCQ
+ when it actually doesn't
change */
unsigned use_10_for_rw:1; /* first try 10-byte read / write */ unsigned use_10_for_ms:1; /* first try 10-byte mode sense/select */ unsigned set_dbd_for_ms:1; /* Set "DBD" field in mode sense */
That's pretty much what James was suggesting, except for one thing: You must not set sdkp->device->expecting_media_change to 1 for all devices in sd_runtime_resume(). Only for devices which may generate a spurious Unit Attention following runtime resume -- and maybe not even for all of them, depending on what the user wants.
Alan Stern
Quoted 103 lines · Alan Stern
Quoted 89 lines · Martin Kepplinger
Maybe I should just start a new discussion with a patch, but the below is what makes sense to me (when I understand you correctly) and seems to work. I basically add a new flag, so that the old flags behave unchanged and only call it during *runtime* resume for SD cards:
drivers/scsi/scsi_error.c +0
-0@@ -553,15 +553,21 @@ int scsi_check_sense(struct scsi_cmnd *scmd)* information that we should pass up to the upper-level driver
* so that we can deal with it there. */drivers/scsi/scsi_error.c +10
-4-if (scmd->device->expecting_cc_ua) { +if (scmd->device->expecting_cc_ua || + scmd->device->expecting_media_change) {/* * Because some device does not queue unit * attentions correctly, we carefully check * additional sense code and qualifier so as -* not to squash media change unit attention. +* not to squash media change unit attention; +* unless expecting_media_change is set, indicating +* that the media (most likely) didn't change +* but a device only believes so (for example +* because of suspend/resume). */ -if (sshdr.asc !=0x28 || sshdr.ascq !=0x00 ) { - scmd->device->expecting_cc_ua =0 ; +if ((sshdr.asc !=0x28 || sshdr.ascq !=0x00 ) || + scmd->device->expecting_media_change) { + scmd->device->expecting_media_change =0 ;return NEEDS_RETRY ; } }drivers/scsi/sd.c +7
-1@@ -114,6 +114,7 @@ static void sd_shutdown(struct device *);static int sd_suspend_system (struct device *);static int sd_suspend_runtime (struct device *);static int sd_resume (struct device *); +static int sd_resume_runtime (struct device *);static void sd_rescan (struct device *);static blk_status_t sd_init_command (struct scsi_cmnd *SCpnt);static void sd_uninit_command (struct scsi_cmnd *SCpnt);@@ -574,7 +575,7 @@ static const struct dev_pm_ops sd_pm_ops = { .poweroff = sd_suspend_system, .restore = sd_resume, .runtime_suspend = sd_suspend_runtime, - .runtime_resume = sd_resume, + .runtime_resume = sd_resume_runtime, };static struct scsi_driver sd_template = {@@ -3652,6 +3653,21 @@ static int sd_resume(struct device *dev)return ret; } +static int sd_resume_runtime (struct device *dev) +{ +struct scsi_disk *sdkp =dev_get_drvdata (dev); + +/* Some SD cardreaders report media change when resuming from suspend
drivers/scsi/sd.c +4
-0+* because they can't keep track during suspend. */ + +/* XXX This is not unproblematic though: We won't notice when a card +* was really changed during runtime suspend! We basically rely on users
drivers/scsi/sd.c +6
-0+* to unmount or suspend before doing so. */ + sdkp->device->expecting_media_change =1 ; + +return sd_resume (dev); +} +/** * init_sd - entry point for this driver (both when built in or when * a module). include/scsi/scsi_device.h +1
-0@@ -169,6 +169,8 @@ struct scsi_device {* this device */ unsigned expecting_cc_ua:1; /* Expecting a CHECK_CONDITION/UNIT_ATTN* because we did a bus reset. */ + unsigned expecting_media_change:1; /* Expecting media changeQuoted 6 lines
ASC/ASCQ
+ when it actually doesn't
change */
unsigned use_10_for_rw:1; /* first try 10-byte read / write */ unsigned use_10_for_ms:1; /* first try 10-byte mode sense/select */ unsigned set_dbd_for_ms:1; /* Set "DBD" field in mode sense */That's pretty much what James was suggesting, except for one thing: You must not set
sdkp->device->expecting_media_changeto 1 for all devices insd_runtime_resume(). Only for devices which may generate a spurious Unit Attention following runtime resume -- and maybe not even for all of them, depending on what the user wants.
Quoted 2 lines
Alan Stern
when I mount the SD card myself or via Nautilus, things work. When I put sth like:
/dev/sda1 /mnt/sda1 auto auto,nofail 0 2
into fstab, I *still* get (constantly) when accessing the files:
[ 50.838061] sd 0:0:0:0: [sda] tag#0 device offline or changed
why could that be? is there another place we would add such a new flag (not only resume())?
martin
Quoted 112 lines · Martin Kepplinger
Quoted 103 lines · Alan Stern
Quoted 89 lines · Martin Kepplinger
Maybe I should just start a new discussion with a patch, but the below is what makes sense to me (when I understand you correctly) and seems to work. I basically add a new flag, so that the old flags behave unchanged and only call it during *runtime* resume for SD cards:
drivers/scsi/scsi_error.c +0
-0@@ -553,15 +553,21 @@ int scsi_check_sense(struct scsi_cmnd *scmd)* information that we should pass up to the upper-level driver
* so that we can deal with it there. */drivers/scsi/scsi_error.c +10
-4-if (scmd->device->expecting_cc_ua) { +if (scmd->device->expecting_cc_ua || + scmd->device->expecting_media_change) {/* * Because some device does not queue unit * attentions correctly, we carefully check * additional sense code and qualifier so as -* not to squash media change unit attention. +* not to squash media change unit attention; +* unless expecting_media_change is set, indicating +* that the media (most likely) didn't change +* but a device only believes so (for example +* because of suspend/resume). */ -if (sshdr.asc !=0x28 || sshdr.ascq !=0x00 ) { - scmd->device->expecting_cc_ua =0 ; +if ((sshdr.asc !=0x28 || sshdr.ascq !=0x00 ) || + scmd->device->expecting_media_change) { + scmd->device->expecting_media_change =0 ;return NEEDS_RETRY ; } }drivers/scsi/sd.c +7
-1@@ -114,6 +114,7 @@ static void sd_shutdown(struct device *);static int sd_suspend_system (struct device *);static int sd_suspend_runtime (struct device *);static int sd_resume (struct device *); +static int sd_resume_runtime (struct device *);static void sd_rescan (struct device *);static blk_status_t sd_init_command (struct scsi_cmnd *SCpnt);static void sd_uninit_command (struct scsi_cmnd *SCpnt);@@ -574,7 +575,7 @@ static const struct dev_pm_ops sd_pm_ops = { .poweroff = sd_suspend_system, .restore = sd_resume, .runtime_suspend = sd_suspend_runtime, - .runtime_resume = sd_resume, + .runtime_resume = sd_resume_runtime, };static struct scsi_driver sd_template = {@@ -3652,6 +3653,21 @@ static int sd_resume(struct device *dev)return ret; } +static int sd_resume_runtime (struct device *dev) +{ +struct scsi_disk *sdkp =dev_get_drvdata (dev); + +/* Some SD cardreaders report media change when resuming from suspend
drivers/scsi/sd.c +4
-0+* because they can't keep track during suspend. */ + +/* XXX This is not unproblematic though: We won't notice when a card +* was really changed during runtime suspend! We basically rely on users
drivers/scsi/sd.c +6
-0+* to unmount or suspend before doing so. */ + sdkp->device->expecting_media_change =1 ; + +return sd_resume (dev); +} +/** * init_sd - entry point for this driver (both when built in or when * a module). include/scsi/scsi_device.h +1
-0@@ -169,6 +169,8 @@ struct scsi_device {* this device */ unsigned expecting_cc_ua:1; /* Expecting a CHECK_CONDITION/UNIT_ATTN* because we did a bus reset. */ + unsigned expecting_media_change:1; /* Expecting media changeQuoted 6 lines
ASC/ASCQ
+ when it actually doesn't
change */
unsigned use_10_for_rw:1; /* first try 10-byte read / write */ unsigned use_10_for_ms:1; /* first try 10-byte mode sense/select */ unsigned set_dbd_for_ms:1; /* Set "DBD" field in mode sense */That's pretty much what James was suggesting, except for one thing: You must not set
sdkp->device->expecting_media_changeto 1 for all devices insd_runtime_resume(). Only for devices which may generate a spurious Unit Attention following runtime resume -- and maybe not even for all of them, depending on what the user wants.Quoted 2 lines
Alan Stern
when I mount the SD card myself or via Nautilus, things work. When I put sth like:
/dev/sda1/mnt/sda1auto auto,nofail 0 2
Quoted 9 lines
into fstab, I *still* get (constantly) when accessing the files:
[ 50.838061] sd 0:0:0:0: [sda] tag#0 device offline or changedwhy could that be? is there another place we would add such a new flag (not only
resume())?martin
it's really strange: below is the change I'm trying. Of course that's only for testing the functionality, nothing how a patch could look like.
While I remember it had worked, now (weirdly since I tried that mounting via fstab) it doesn't anymore!
What I understand (not much): I handle the error with "retry" via the new flag, but scsi_decide_disposition() returns SUCCESS because of "no more retries"; but it's the first and only time it's called.
How can this be? What am I missing?
drivers/scsi/scsi_error.c +7 -0
@@ -565,6 +565,13 @@ int scsi_check_sense(struct scsi_cmnd *scmd)return NEEDS_RETRY ; } } +if (scmd->device->expecting_media_change) { +if (sshdr.asc ==0x28 && sshdr.ascq ==0x00 ) { + scmd->device->expecting_media_change =0 ; +return NEEDS_RETRY ; + } + } +/* * we might also expect a cc/ua if another LUN on the target * reported a UA with an ASC/ASCQ of 3F 0E -
drivers/scsi/sd.c +2 -0
@@ -3642,6 +3642,8 @@ static int sd_resume(struct device *dev)if (!sdkp)/* E.g.: runtime resume at the start of sd_probe() */ return 0 ; + sdkp->device->expecting_media_change =1 ; +if (!sdkp->device->manage_start_stop)return 0 ;
include/scsi/scsi_device.h +1 -0
@@ -169,6 +169,7 @@ struct scsi_device {* this device */ unsigned expecting_cc_ua:1; /* Expecting a CHECK_CONDITION/UNIT_ATTN* because we did a bus reset. */ + unsigned expecting_media_change:1; unsigned use_10_for_rw:1; /* first try 10-byte read / write */ unsigned use_10_for_ms:1; /* first try 10-byte mode sense/select */ unsigned set_dbd_for_ms:1; /* Set "DBD" field in mode sense */
Quoted 9 lines · Martin Kepplinger
it's really strange: below is the change I'm trying. Of course that's only for testing the functionality, nothing how a patch could look like.
While I remember it had worked, now (weirdly since I tried that mounting via fstab) it doesn't anymore!
What I understand (not much): I handle the error with "retry" via the new flag, but
scsi_decide_disposition()returns SUCCESS because of "no more retries"; but it's the first and only time it's called.
Are you saying that scmd->allowed is set to 0? Or is scsi_notry_cmd() returning a nonzero value? Whichever is true, why does it happen that way?
What is the failing command? Is it a READ(10)?
Quoted 1 line
How can this be? What am I missing?
It's kind of hard to tell without seeing the error messages or system log or any debugging information.
Alan Stern
Quoted 41 lines
drivers/scsi/scsi_error.c +7
-0@@ -565,6 +565,13 @@ int scsi_check_sense(struct scsi_cmnd *scmd)return NEEDS_RETRY ; } } +if (scmd->device->expecting_media_change) { +if (sshdr.asc ==0x28 && sshdr.ascq ==0x00 ) { + scmd->device->expecting_media_change =0 ; +return NEEDS_RETRY ; + } + } +/* * we might also expect a cc/ua if another LUN on the target * reported a UA with an ASC/ASCQ of 3F 0E - drivers/scsi/sd.c +2
-0@@ -3642,6 +3642,8 @@ static int sd_resume(struct device *dev)if (!sdkp)/* E.g.: runtime resume at the start of sd_probe() */ return 0 ; + sdkp->device->expecting_media_change =1 ; +if (!sdkp->device->manage_start_stop)return 0 ;include/scsi/scsi_device.h +1
-0@@ -169,6 +169,7 @@ struct scsi_device {* this device */ unsigned expecting_cc_ua:1; /* Expecting a CHECK_CONDITION/UNIT_ATTN* because we did a bus reset. */ + unsigned expecting_media_change:1; unsigned use_10_for_rw:1; /* first try 10-byte read / write */ unsigned use_10_for_ms:1; /* first try 10-byte mode sense/select */ unsigned set_dbd_for_ms:1; /* Set "DBD" field in mode sense */
Quoted 7 lines · Alan Stern
Quoted 6 lines · Martin Kepplinger
it's really strange: below is the change I'm trying. Of course that's only for testing the functionality, nothing how a patch could look like.
While I remember it had worked, now (weirdly since I tried that mounting via fstab) it doesn't anymore!
Quoted 7 lines
Quoted 3 lines
What I understand (not much): I handle the error with "retry" via the new flag, but
scsi_decide_disposition()returns SUCCESS because of "no more retries"; but it's the first and only time it's called.Are you saying that
scmd->allowedis set to 0? Or isscsi_notry_cmd()returning a nonzero value? Whichever is true, why does it happen that way?
scsi_notry_cmd() is returning 1. (it's retry 1 of 5 allowed).
why is it returning 1? REQ_FAILFAST_DEV is set. It's DID_OK, then "if (status_byte(scmd->result) != CHECK_CONDITION)" appearently is not true, then at the end it returns 1 because of REQ_FAILFAST_DEV.
that seems to come from the block layer. why and when? could I change that so that the scsi error handling stays in control?
Quoted 2 lines
What is the failing command? Is it a READ(10)?
Not sure how I'd answer that, but here's the test to trigger the error:
mount /dev/sda1 /mnt
cd /mnt
ls
cp file ~/ (if ls "works" and doesn't yet trigger the error)
and that's the (familiar looking) logs when doing so. again: despite the mentioned workaround in scsi_error and the new expected_media_change flag *is* set and gets cleared, as it should be. REQ_FAILFAST_DEV seems to override what I want to do is scsi_error:
[ 55.557629] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 cmd_age=0s [ 55.557639] sd 0:0:0:0: [sda] tag#0 Sense Key : 0x6 [current] [ 55.557646] sd 0:0:0:0: [sda] tag#0 ASC=0x28 ASCQ=0x0 [ 55.557657] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 08 fc e0 00 00 01 00 [ 55.557666] blk_update_request: I/O error, dev sda, sector 589024 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 55.568899] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 55.574691] blk_update_request: I/O error, dev sda, sector 589025 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 55.585756] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 55.591562] blk_update_request: I/O error, dev sda, sector 589026 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 55.602274] sd 0:0:0:0: [sda] tag#0 device offline or changed
(... goes on with the same)
Quoted 5 lines
Quoted 2 lines
How can this be? What am I missing?
It's kind of hard to tell without seeing the error messages or system log or any debugging information.
thanks a lot for getting back to me,
martin
Quoted 44 lines
Alan Stern
Quoted 41 lines
drivers/scsi/scsi_error.c +7
-0@@ -565,6 +565,13 @@ int scsi_check_sense(struct scsi_cmnd *scmd)return NEEDS_RETRY ; } } +if (scmd->device->expecting_media_change) { +if (sshdr.asc ==0x28 && sshdr.ascq ==0x00 ) { + scmd->device->expecting_media_change =0 ; +return NEEDS_RETRY ; + } + } +/* * we might also expect a cc/ua if another LUN on the target * reported a UA with an ASC/ASCQ of 3F 0E - drivers/scsi/sd.c +2
-0@@ -3642,6 +3642,8 @@ static int sd_resume(struct device *dev)if (!sdkp)/* E.g.: runtime resume at the start of sd_probe() */ return 0 ; + sdkp->device->expecting_media_change =1 ; +if (!sdkp->device->manage_start_stop)return 0 ;include/scsi/scsi_device.h +1
-0@@ -169,6 +169,7 @@ struct scsi_device {* this device */ unsigned expecting_cc_ua:1; /* Expecting a CHECK_CONDITION/UNIT_ATTN* because we did a bus reset. */ + unsigned expecting_media_change:1; unsigned use_10_for_rw:1; /* first try 10-byte read / write */ unsigned use_10_for_ms:1; /* first try 10-byte mode sense/select */ unsigned set_dbd_for_ms:1; /* Set "DBD" field in mode sense */
Quoted 16 lines · Martin Kepplinger
Quoted 7 lines · Alan Stern
Quoted 6 lines · Martin Kepplinger
it's really strange: below is the change I'm trying. Of course that's only for testing the functionality, nothing how a patch could look like.
While I remember it had worked, now (weirdly since I tried that mounting via fstab) it doesn't anymore!
Quoted 7 lines
Quoted 3 lines
What I understand (not much): I handle the error with "retry" via the new flag, but
scsi_decide_disposition()returns SUCCESS because of "no more retries"; but it's the first and only time it's called.Are you saying that
scmd->allowedis set to 0? Or isscsi_notry_cmd()returning a nonzero value? Whichever is true, why does it happen that way?
Quoted 8 lines
scsi_notry_cmd()is returning 1. (it's retry 1 of 5 allowed).why is it returning 1?
REQ_FAILFAST_DEVis set. It'sDID_OK, then "if (status_byte(scmd->result)!=CHECK_CONDITION)" appearently is not true, then at the end it returns 1 because ofREQ_FAILFAST_DEV.that seems to come from the block layer. why and when? could I change that so that the scsi error handling stays in control?
The only place I see where that flag might get set is in blk_mq_bio_to_request() in block/blk-mq.c, which does:
if (bio->bi_opf & REQ_RAHEAD) rq->cmd_flags |= REQ_FAILFAST_MASK;
So apparently read-ahead reads are supposed to fail fast (i.e., without retries), presumably because they are optional after all.
Quoted 14 lines
Quoted 1 line
What is the failing command? Is it a READ(10)?
Not sure how I'd answer that, but here's the test to trigger the error:
mount
/dev/sda1/mnt
cd /mnt
ls
cp file ~/ (if ls "works" and doesn't yet trigger the error)and that's the (familiar looking) logs when doing so. again: despite the mentioned workaround in
scsi_errorand the newexpected_media_changeflag *is* set and gets cleared, as it should be.REQ_FAILFAST_DEVseems to override what I want to do isscsi_error:
Quoted 6 lines
[ 55.557629] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 cmd_age=0s [ 55.557639] sd 0:0:0:0: [sda] tag#0 Sense Key : 0x6 [current] [ 55.557646] sd 0:0:0:0: [sda] tag#0 ASC=0x28 ASCQ=0x0 [ 55.557657] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 08 fce0 00 00 01 00
Yes, 0x28 is READ(10). Likely this is a read-ahead request, although I don't know how we can tell for sure.
Quoted 10 lines
[ 55.557666] blk_update_request: I/O error, dev sda, sector 589024 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 55.568899] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 55.574691] blk_update_request: I/O error, dev sda, sector 589025 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 55.585756] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 55.591562] blk_update_request: I/O error, dev sda, sector 589026 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 55.602274] sd 0:0:0:0: [sda] tag#0 device offline or changed(... goes on with the same)
Is such a drastic response really appropriate for the failure of a read-ahead request? It seems like a more logical response would be to let the request fail but keep the device online.
Of course, that would only solve part of your problem -- your log would still get filled with those "I/O error" messages even though they wouldn't be fatal. Probably a better approach would be to make the new expecting_media_change flag override scsi_no_retry_cmd().
But this is not my area of expertise. Maybe someone else will have more to say.
Alan Stern
Quoted 79 lines · Alan Stern
Quoted 16 lines · Martin Kepplinger
Quoted 7 lines · Alan Stern
Quoted 6 lines · Martin Kepplinger
it's really strange: below is the change I'm trying. Of course that's only for testing the functionality, nothing how a patch could look like.
While I remember it had worked, now (weirdly since I tried that mounting via fstab) it doesn't anymore!
Quoted 7 lines
Quoted 3 lines
What I understand (not much): I handle the error with "retry" via the new flag, but
scsi_decide_disposition()returns SUCCESS because of "no more retries"; but it's the first and only time it's called.Are you saying that
scmd->allowedis set to 0? Or isscsi_notry_cmd()returning a nonzero value? Whichever is true, why does it happen that way?Quoted 8 lines
scsi_notry_cmd()is returning 1. (it's retry 1 of 5 allowed).why is it returning 1?
REQ_FAILFAST_DEVis set. It'sDID_OK, then "if (status_byte(scmd->result)!=CHECK_CONDITION)" appearently is not true, then at the end it returns 1 because ofREQ_FAILFAST_DEV.that seems to come from the block layer. why and when? could I change that so that the scsi error handling stays in control?
The only place I see where that flag might get set is in
blk_mq_bio_to_request()inblock/blk-mq.c, which does:if (bio->bi_opf & REQ_RAHEAD) rq->cmd_flags |= REQ_FAILFAST_MASK;So apparently read-ahead reads are supposed to fail fast (i.e., without retries), presumably because they are optional after all.
Quoted 14 lines
Quoted 1 line
What is the failing command? Is it a READ(10)?
Not sure how I'd answer that, but here's the test to trigger the error:
mount
/dev/sda1/mnt
cd /mnt
ls
cp file ~/ (if ls "works" and doesn't yet trigger the error)and that's the (familiar looking) logs when doing so. again: despite the mentioned workaround in
scsi_errorand the newexpected_media_changeflag *is* set and gets cleared, as it should be.REQ_FAILFAST_DEVseems to override what I want to do isscsi_error:Quoted 6 lines
[ 55.557629] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 cmd_age=0s [ 55.557639] sd 0:0:0:0: [sda] tag#0 Sense Key : 0x6 [current] [ 55.557646] sd 0:0:0:0: [sda] tag#0 ASC=0x28 ASCQ=0x0 [ 55.557657] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 08 fce0 00 00 01 00
Yes,
0x28is READ(10). Likely this is a read-ahead request, although I don't know how we can tell for sure.Quoted 10 lines
[ 55.557666] blk_update_request: I/O error, dev sda, sector 589024 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 55.568899] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 55.574691] blk_update_request: I/O error, dev sda, sector 589025 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 55.585756] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 55.591562] blk_update_request: I/O error, dev sda, sector 589026 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 55.602274] sd 0:0:0:0: [sda] tag#0 device offline or changed(... goes on with the same)
Is such a drastic response really appropriate for the failure of a read-ahead request? It seems like a more logical response would be to let the request fail but keep the device online.
Of course, that would only solve part of your problem -- your log would still get filled with those "I/O error" messages even though they wouldn't be fatal. Probably a better approach would be to make the new
expecting_media_changeflag overridescsi_no_retry_cmd().
Quoted 5 lines
But this is not my area of expertise. Maybe someone else will have more to say.
Alan Stern
Hey Alan, I'm really glad for that, I suspected some of this but I have little experience in scsi/block layers, so that is super helpful.
I'd appreciate an opinion on the below workaround that *seems* to work now (let's see, I had thought so before :)
Whether or not this helps to find a real solution, let's see. But integration of such a flag in the error handling paths is what's interesting for now:
drivers/scsi/scsi_error.c +20 -0
@@ -565,6 +565,17 @@ int scsi_check_sense(struct scsi_cmnd *scmd)return NEEDS_RETRY ; } } +if (scmd->device->expecting_media_change) { +if (sshdr.asc ==0x28 && sshdr.ascq ==0x00 ) { +/* clear expecting_media_change in +* scsi_noretry_cmd() because we need +* to override possible "failfast" overrides +* that block readahead can cause. +*/ +return NEEDS_RETRY ; + } + } +/* * we might also expect a cc/ua if another LUN on the target * reported a UA with an ASC/ASCQ of 3F 0E -
@@ -1744,6 +1755,15 @@ int scsi_noretry_cmd(struct scsi_cmnd *scmd)return (scmd->request->cmd_flags &REQ_FAILFAST_DRIVER ); } +/* +* We need to have retries when expecting_media_change is set. +* So we need to return success and clear the flag here. +*/ +if (scmd->device->expecting_media_change) { + scmd->device->expecting_media_change =0 ; +return 0 ; + } +if (status_byte (scmd->result) !=CHECK_CONDITION )return 0 ;
drivers/scsi/sd.c +2 -0
@@ -3642,6 +3642,8 @@ static int sd_resume(struct device *dev)if (!sdkp)/* E.g.: runtime resume at the start of sd_probe() */ return 0 ; + sdkp->device->expecting_media_change =1 ; +if (!sdkp->device->manage_start_stop)return 0 ;
include/scsi/scsi_device.h +1 -0
@@ -169,6 +169,7 @@ struct scsi_device {* this device */ unsigned expecting_cc_ua:1; /* Expecting a CHECK_CONDITION/UNIT_ATTN* because we did a bus reset. */ + unsigned expecting_media_change:1; unsigned use_10_for_rw:1; /* first try 10-byte read / write */ unsigned use_10_for_ms:1; /* first try 10-byte mode sense/select */ unsigned set_dbd_for_ms:1; /* Set "DBD" field in mode sense */
--
Quoted 15 lines · Martin Kepplinger
Hey Alan, I'm really glad for that, I suspected some of this but I have little experience in scsi/block layers, so that is super helpful.
I'd appreciate an opinion on the below workaround that *seems* to work now (let's see, I had thought so before :)
Whether or not this helps to find a real solution, let's see. But integration of such a flag in the error handling paths is what's interesting for now:
drivers/scsi/scsi_error.c +0
-0@@ -565,6 +565,17 @@ int scsi_check_sense(struct scsi_cmnd *scmd)return NEEDS_RETRY ;
drivers/scsi/scsi_error.c int scsi_check_sense(struct scsi_cmnd *scmd) · line 575
Quoted 10 lines
drivers/scsi/scsi_error.c +8
-0} } +if (scmd->device->expecting_media_change) { +if (sshdr.asc ==0x28 && sshdr.ascq ==0x00 ) { +/* clear expecting_media_change in +* scsi_noretry_cmd() because we need +* to override possible "failfast" overrides +* that block readahead can cause. +*/ +return NEEDS_RETRY ;
This is a somewhat fragile approach. You don't know for certain that scsi_noretry_cmd will be called. Also, scsi_noretry_cmd can be called from other places.
It would be better to clear the expecting_media_change flag just before returning from scsi_decide_disposition. That way its use is localized to one routine, not spread out between two.
Alan Stern
Quoted 31 lines · Alan Stern
Quoted 15 lines · Martin Kepplinger
Hey Alan, I'm really glad for that, I suspected some of this but I have little experience in scsi/block layers, so that is super helpful.
I'd appreciate an opinion on the below workaround that *seems* to work now (let's see, I had thought so before :)
Whether or not this helps to find a real solution, let's see. But integration of such a flag in the error handling paths is what's interesting for now:
drivers/scsi/scsi_error.c +0
-0@@ -565,6 +565,17 @@ int scsi_check_sense(struct scsi_cmnd *scmd)return NEEDS_RETRY ;drivers/scsi/scsi_error.c int scsi_check_sense(struct scsi_cmnd *scmd) · line 575
Quoted 10 lines
drivers/scsi/scsi_error.c +8
-0} } +if (scmd->device->expecting_media_change) { +if (sshdr.asc ==0x28 && sshdr.ascq ==0x00 ) { +/* clear expecting_media_change in +* scsi_noretry_cmd() because we need +* to override possible "failfast" overrides +* that block readahead can cause. +*/ +return NEEDS_RETRY ;This is a somewhat fragile approach. You don't know for certain that
scsi_noretry_cmdwill be called. Also,scsi_noretry_cmdcan be called from other places.
Quoted 6 lines
It would be better to clear the
expecting_media_changeflag just before returning fromscsi_decide_disposition. That way its use is localized to one routine, not spread out between two.Alan Stern
Hi Alan,
maybe you're right. I initially just thought that I'd allow for specific error codes in scsi_noretry_cmd() to return non-NULL (BUS_BUSY, PARITY, ERROR) despite having the flag set.
The below version works equally fine for me but I'm not sure if it's actually more safe.
James, when exposing a new writable sysfs option like "suspend_no_media_change"(?) that drivers can check before setting the new "expecting_media_change" flag (during resume), would this addition make sense to you?
thanks,
martin
drivers/scsi/scsi_error.c +25 -3
@@ -565,6 +565,18 @@ int scsi_check_sense(struct scsi_cmnd *scmd)return NEEDS_RETRY ; } } +if (scmd->device->expecting_media_change) { +if (sshdr.asc ==0x28 && sshdr.ascq ==0x00 ) { +/* +* clear the expecting_media_change in +* scsi_decide_disposition() because we +* need to catch possible "fail fast" overrides +* that block readahead can cause. +*/ +return NEEDS_RETRY ; + } + } +/* * we might also expect a cc/ua if another LUN on the target * reported a UA with an ASC/ASCQ of 3F 0E -
@@ -1944,9 +1956,19 @@ int scsi_decide_disposition(struct scsi_cmnd *scmd)* the request was not marked fast fail. Note that above, * even if the request is marked fast fail, we still requeue * for queue congestion conditions (QUEUE_FULL or BUSY) */ -if ((++scmd->retries) <= scmd->allowed - && !scsi_noretry_cmd (scmd)) { -return NEEDS_RETRY ; +if ((++scmd->retries) <= scmd->allowed) { +/* +* but scsi_noretry_cmd() cannot override the +* expecting_media_change flag. +*/ +if (!scsi_noretry_cmd (scmd) || + scmd->device->expecting_media_change) { + scmd->device->expecting_media_change =0 ; +return NEEDS_RETRY ; + }else { +/* marked fast fail and not expected. */ +return SUCCESS ; + } }else {/* * no more retries - report this one back to upper level.
drivers/scsi/sd.c +2 -0
@@ -3642,6 +3642,8 @@ static int sd_resume(struct device *dev)if (!sdkp)/* E.g.: runtime resume at the start of sd_probe() */ return 0 ; + sdkp->device->expecting_media_change =1 ; +if (!sdkp->device->manage_start_stop)return 0 ;
include/scsi/scsi_device.h +1 -0
@@ -169,6 +169,7 @@ struct scsi_device {* this device */ unsigned expecting_cc_ua:1; /* Expecting a CHECK_CONDITION/UNIT_ATTN* because we did a bus reset. */ + unsigned expecting_media_change:1; unsigned use_10_for_rw:1; /* first try 10-byte read / write */ unsigned use_10_for_ms:1; /* first try 10-byte mode sense/select */ unsigned set_dbd_for_ms:1; /* Set "DBD" field in mode sense */
Quoted 64 lines · Martin Kepplinger
Quoted 10 lines · Alan Stern
This is a somewhat fragile approach. You don't know for certain that
scsi_noretry_cmdwill be called. Also,scsi_noretry_cmdcan be called from other places.It would be better to clear the
expecting_media_changeflag just before returning fromscsi_decide_disposition. That way its use is localized to one routine, not spread out between two.Alan Stern
Hi Alan,
maybe you're right. I initially just thought that I'd allow for specific error codes in
scsi_noretry_cmd()to return non-NULL (BUS_BUSY, PARITY, ERROR) despite having the flag set.The below version works equally fine for me but I'm not sure if it's actually more safe.
James, when exposing a new writable sysfs option like "suspend_no_media_change"(?) that drivers can check before setting the new "
expecting_media_change" flag (during resume), would this addition make sense to you?thanks,
martin
drivers/scsi/scsi_error.c +16
-3@@ -565,6 +565,18 @@ int scsi_check_sense(struct scsi_cmnd *scmd)return NEEDS_RETRY ; } } +if (scmd->device->expecting_media_change) { +if (sshdr.asc ==0x28 && sshdr.ascq ==0x00 ) { +/* +* clear the expecting_media_change in +* scsi_decide_disposition() because we +* need to catch possible "fail fast" overrides +* that block readahead can cause. +*/ +return NEEDS_RETRY ; + } + } +/* * we might also expect a cc/ua if another LUN on the target * reported a UA with an ASC/ASCQ of 3F 0E - @@ -1944,9 +1956,19 @@ int scsi_decide_disposition(struct scsi_cmnd *scmd)* the request was not marked fast fail. Note that above, * even if the request is marked fast fail, we still requeue * for queue congestion conditions (QUEUE_FULL or BUSY) */ -if ((++scmd->retries) <= scmd->allowed - && !scsi_noretry_cmd (scmd)) { -return NEEDS_RETRY ; +if ((++scmd->retries) <= scmd->allowed) { +/* +* but scsi_noretry_cmd() cannot override the +* expecting_media_change flag.
drivers/scsi/scsi_error.c int scsi_decide_disposition(struct scsi_cmnd *scmd) · line 1972
Quoted 10 lines
drivers/scsi/scsi_error.c +9
-0+*/ +if (!scsi_noretry_cmd (scmd) || + scmd->device->expecting_media_change) { + scmd->device->expecting_media_change =0 ; +return NEEDS_RETRY ; + }else { +/* marked fast fail and not expected. */ +return SUCCESS ; + } }else {
This may not matter... but it's worth pointing out that expecting_media_change doesn't get cleared if ++scmd->retries > scmd->allowed.
It also doesn't get cleared in cases where the device _doesn't_ report a Unit Attention.
Alan Stern
Quoted 71 lines · Alan Stern
Quoted 70 lines · Martin Kepplinger
Quoted 10 lines · Alan Stern
This is a somewhat fragile approach. You don't know for certain that
scsi_noretry_cmdwill be called. Also,scsi_noretry_cmdcan be called from other places.It would be better to clear the
expecting_media_changeflag just before returning fromscsi_decide_disposition. That way its use is localized to one routine, not spread out between two.Alan Stern
Hi Alan,
maybe you're right. I initially just thought that I'd allow for specific error codes in
scsi_noretry_cmd()to return non-NULL (BUS_BUSY, PARITY, ERROR) despite having the flag set.The below version works equally fine for me but I'm not sure if it's actually more safe.
James, when exposing a new writable sysfs option like "suspend_no_media_change"(?) that drivers can check before setting the new "
expecting_media_change" flag (during resume), would this addition make sense to you?thanks,
martin
drivers/scsi/scsi_error.c +22
-3@@ -565,6 +565,18 @@ int scsi_check_sense(struct scsi_cmnd *scmd)return NEEDS_RETRY ; } } +if (scmd->device->expecting_media_change) { +if (sshdr.asc ==0x28 && sshdr.ascq ==0x00 ) { +/* +* clear the expecting_media_change in +* scsi_decide_disposition() because we +* need to catch possible "fail fast" overrides +* that block readahead can cause. +*/ +return NEEDS_RETRY ; + } + } +/* * we might also expect a cc/ua if another LUN on the target * reported a UA with an ASC/ASCQ of 3F 0E - @@ -1944,9 +1956,19 @@ int scsi_decide_disposition(struct scsi_cmnd *scmd)* the request was not marked fast fail. Note that above, * even if the request is marked fast fail, we still requeue * for queue congestion conditions (QUEUE_FULL or BUSY) */ -if ((++scmd->retries) <= scmd->allowed - && !scsi_noretry_cmd (scmd)) { -return NEEDS_RETRY ; +if ((++scmd->retries) <= scmd->allowed) { +/* +* but scsi_noretry_cmd() cannot override the +* expecting_media_change flag. +*/ +if (!scsi_noretry_cmd (scmd) || + scmd->device->expecting_media_change) { + scmd->device->expecting_media_change =0 ; +return NEEDS_RETRY ; + }else {
Quoted 8 lines
drivers/scsi/scsi_error.c int scsi_decide_disposition(struct scsi_cmnd *scmd) · line 1972
Quoted 4 lines
drivers/scsi/scsi_error.c +3
-0+/* marked fast fail and not expected. */ +return SUCCESS ; + } }else {This may not matter... but it's worth pointing out that
expecting_media_changedoesn't get cleared if ++scmd->retries >scmd->allowed.
absolutely worth pointing out and I'm not yet sure about that one.
Quoted 3 lines
It also doesn't get cleared in cases where the device _doesn't_ report a Unit Attention.
true. but don't we set the flag for a future UA we don't yet know of? If we would want to clear it outside of a UA, I think we'd need to keep track of a suspend/resume cycle and if we see that we *had* successfully "done requests" after resuming, we could clear it...
Quoted 3 lines
Alan Stern
Quoted 8 lines · Martin Kepplinger
Quoted 3 lines · Alan Stern
This may not matter... but it's worth pointing out that
expecting_media_changedoesn't get cleared if ++scmd->retries >scmd->allowed.absolutely worth pointing out and I'm not yet sure about that one.
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
It also doesn't get cleared in cases where the device _doesn't_ report a Unit Attention.
true. but don't we set the flag for a future UA we don't yet know of? If we would want to clear it outside of a UA, I think we'd need to keep track of a suspend/resume cycle and if we see that we *had* successfully "done requests" after resuming, we could clear it...
The point is that expecting_media_change should apply only to the _next_ command. It should be cleared after _every_ command. You do not want to leave it hanging around -- if you do then you will miss real media changes.
There's a potential issue when a bunch of commands get sent and completed all at once, but hopefully it won't matter here.
Alan Stern
PATCH 1/1 block: Fix bug in runtime-resume handling
In reply to Martin KepplingerRuntime power-management support in the block layer has somehow gotten messed up in the past few years. This area of code has been through a lot of churn and it's not easy to track exactly when the bug addressed here was introduced; it was present for a while, then fixed for a while, and then it returned.
At any rate, the problem is that the block layer code thinks that it's okay to issue a request with the BLK_MQ_REQ_PREEMPT flag set at any time, even when the queue and underlying device are runtime suspended. This belief is wrong; the flag merely indicates that it's okay to issue the request while the queue and device are in the process of suspending or resuming. When they are actually suspended, no requests may be issued.
The symptom of this bug is that a runtime-suspended block device doesn't resume as it should. The request which should cause a runtime resume instead gets issued directly, without resuming the device first. Of course the device can't handle it properly, the I/O fails, and the device remains suspended.
The problem is fixed by checking that the queue's runtime-PM status isn't RPM_SUSPENDED before allowing a request to be issued, and queuing a runtime-resume request if it is. In particular, the inline blk_pm_request_resume() routine is renamed blk_pm_resume_queue() and the code is unified by merging the surrounding checks into the routine. If the queue isn't set up for runtime PM, or there currently is no restriction on allowed requests, the request is allowed. Likewise if the BLK_MQ_REQ_PREEMPT flag is set and the status isn't RPM_SUSPENDED. Otherwise a runtime resume is queued and the request is blocked until conditions are more suitable.
---
The bug goes back way before 5.2, but other changes will prevent the patch from applying directly to earlier kernels, so I'm limiting the @stable updates to 5.2 and after.
[as1941]
block/blk-core.c | 6 +++--- block/blk-pm.h | 14 +++++++++----- 2 files changed, 12 insertions(+), 8 deletions(-)
block/blk-core.c +3 -3
@@ -423,7 +423,8 @@ int blk_queue_enter(struct request_queue* responsible for ensuring that that counter is * globally visible before the queue is unfrozen. */ -if (pm || !blk_queue_pm_only (q)) { +if ((pm && q->rpm_status !=RPM_SUSPENDED ) || + !blk_queue_pm_only (q)) { success = true; }else {percpu_ref_put (&q->q_usage_counter);
@@ -448,8 +449,7 @@ int blk_queue_enter(struct request_queuewait_event (q->mq_freeze_wq, (!q->mq_freeze_depth && - (pm || (blk_pm_request_resume (q), - !blk_queue_pm_only (q)))) || +blk_pm_resume_queue (pm, q)) ||blk_queue_dying (q));if (blk_queue_dying (q))return -ENODEV ;
block/blk-pm.h +9 -5
@@ -6,11 +6,14 @@#include <linux/pm_runtime.h> #ifdef CONFIG_PM -static inline void blk_pm_request_resume (struct request_queue *q) +static inline int blk_pm_resume_queue (const bool pm,struct request_queue *q) { -if (q->dev && (q->rpm_status ==RPM_SUSPENDED || - q->rpm_status ==RPM_SUSPENDING )) -pm_request_resume (q->dev); +if (!q->dev || !blk_queue_pm_only (q)) +return 1 ;/* Nothing to do */ +if (pm && q->rpm_status !=RPM_SUSPENDED ) +return 1 ;/* Request allowed */ +pm_request_resume (q->dev); +return 0 ; }static inline void blk_pm_mark_last_busy (struct request *rq)
@@ -44,8 +47,9 @@ static inline void blk_pm_put_request(st --rq->q->nr_pending; }#else -static inline void blk_pm_request_resume (struct request_queue *q) +static inline int blk_pm_resume_queue (const bool pm,struct request_queue *q) { +return 1 ; }static inline void blk_pm_mark_last_busy (struct request *rq)
Re: [PATCH] block: Fix bug in runtime-resume handling
Quoted 2 lines · Alan Stern
The problem is fixed by checking that the queue's runtime-PM status isn't
RPM_SUSPENDEDbefore allowing a request to be issued, [ ... ]
Can, can you help with testing this patch?
Thanks,
Bart.
Re: [PATCH] block: Fix bug in runtime-resume handling
Quoted 11 lines · Bart Van Assche
Quoted 2 lines · Alan Stern
The problem is fixed by checking that the queue's runtime-PM status isn't
RPM_SUSPENDEDbefore allowing a request to be issued, [ ... ]Reviewed-by Bart Van Assche <[email protected]>Can, can you help with testing this patch?
Thanks,
Bart.
Martin:
(I forgot to ask this question several weeks ago, while you were running your tests. Better ask it now before I forget again...)
I suspect the old runtime-PM code in the block layer would have worked okay in your SD cardreader test if the BLK_MQ_REQ_PREEMPT flag had not been set. Do you know why the flag was set, or what line of code caused it to be set?
As I recall, the request that failed was a read-ahead. It's not clear to me why such a request would need to have BLK_MQ_REQ_PREEMPT set. Although there may be other reasons for having that flag, at this point we're concerned with requests that are part of the runtime-resume procedure. A read-ahead does not fall into that category.
Do you think you can find the answer? Perhaps we should add a separate BLK_MQ_REQ_PM flag.
Alan Stern
Re: [PATCH] block: Fix bug in runtime-resume handling
Quoted 15 lines · Alan Stern
Quoted 11 lines · Bart Van Assche
Quoted 2 lines · Alan Stern
The problem is fixed by checking that the queue's runtime-PM status isn't
RPM_SUSPENDEDbefore allowing a request to be issued, [ ... ]Reviewed-by Bart Van Assche <[email protected]>Can, can you help with testing this patch?
Thanks,
Bart.
Martin:
Quoted 7 lines
(I forgot to ask this question several weeks ago, while you were running your tests. Better ask it now before I forget again...)
I suspect the old runtime-PM code in the block layer would have worked okay in your SD cardreader test if the
BLK_MQ_REQ_PREEMPTflag had not been set. Do you know why the flag was set, or what line of code caused it to be set?
Correct. if not set, I could handle all I need in the scsi error path.
Quoted 9 lines
As I recall, the request that failed was a read-ahead. It's not clear to me why such a request would need to have
BLK_MQ_REQ_PREEMPTset. Although there may be other reasons for having that flag, at this point we're concerned with requests that are part of the runtime-resume procedure. A read-ahead does not fall into that category.Do you think you can find the answer? Perhaps we should add a separate
BLK_MQ_REQ_PMflag.
I guess I can but give me some time as I'm on vacations currently.
Quoted 3 lines
Alan Stern
Re: [PATCH] block: Fix bug in runtime-resume handling
Quoted 19 lines · Martin Kepplinger
Quoted 18 lines · Alan Stern
Quoted 11 lines · Bart Van Assche
Quoted 2 lines · Alan Stern
The problem is fixed by checking that the queue's runtime-PM status isn't
RPM_SUSPENDEDbefore allowing a request to be issued, [ ... ]Reviewed-by Bart Van Assche <[email protected]>Can, can you help with testing this patch?
Thanks,
Bart.
Martin:
(I forgot to ask this question several weeks ago, while you were running your tests. Better ask it now before I forget again...)
Quoted 6 lines
Quoted 4 lines
I suspect the old runtime-PM code in the block layer would have worked okay in your SD cardreader test if the
BLK_MQ_REQ_PREEMPTflag had not been set. Do you know why the flag was set, or what line of code caused it to be set?Correct. if not set, I could handle all I need in the scsi error path.
this thread becomes a bit confusing. I thought about REQ_FAILFAST_DEV but you're talking about something different.
the only place I see BLK_MQ_REQ_PREEMPT getting passed on is in __scsi_execute() which is the case when mounting/unmounting. At least that about the only place I can find.
I remember *only* your block pm fix would let me mount/unmount, but not use files yet (REQ_FAILFAST_DEV and so on).
When I revert your fix and remove BLK_MQ_REQ_PREEMPT from being passed on to blk_get_request() in __scsi_execute(), that line gets executed exactly once during startup and I'm missing the /dev/sda device from the cardreader then.
Is this what you're asking?
Quoted 10 lines
Quoted 10 lines
As I recall, the request that failed was a read-ahead. It's not clear to me why such a request would need to have
BLK_MQ_REQ_PREEMPTset. Although there may be other reasons for having that flag, at this point we're concerned with requests that are part of the runtime-resume procedure. A read-ahead does not fall into that category.Do you think you can find the answer? Perhaps we should add a separate
BLK_MQ_REQ_PMflag.
Re: [PATCH] block: Fix bug in runtime-resume handling
Quoted 13 lines · Martin Kepplinger
Quoted 12 lines · Martin Kepplinger
Quoted 9 lines · Alan Stern
Martin:
(I forgot to ask this question several weeks ago, while you were running your tests. Better ask it now before I forget again...)
I suspect the old runtime-PM code in the block layer would have worked okay in your SD cardreader test if the
BLK_MQ_REQ_PREEMPTflag had not been set. Do you know why the flag was set, or what line of code caused it to be set?
Quoted 8 lines
Quoted 1 line
Correct. if not set, I could handle all I need in the scsi error path.
this thread becomes a bit confusing. I thought about
REQ_FAILFAST_DEVbut you're talking about something different.the only place I see
BLK_MQ_REQ_PREEMPTgetting passed on is in__scsi_execute()which is the case when mounting/unmounting. At least that about the only place I can find.
Ah yes, I see what you mean.
Quoted 9 lines
I remember *only* your block pm fix would let me mount/unmount, but not use files yet (
REQ_FAILFAST_DEVand so on).When I revert your fix and remove
BLK_MQ_REQ_PREEMPTfrom being passed on toblk_get_request()in__scsi_execute(), that line gets executed exactly once during startup and I'm missing the/dev/sdadevice from the cardreader then.Is this what you're asking?
Not quite sure, but it doesn't matter. Removing BLK_MQ_REQ_PREEMPT in __scsi_execute() is probably not a safe thing to do.
Instead, look at sd_resume(). That routine calls __scsi_execute() indirectly through sd_start_stop_device(), and the only reason it does this is because the sdkp->device->manage_start_stop flag is set. You ought to be able to clear this flag in sysfs, by writing to /sys/block/sda/device/scsi_disk/*/manage_start_stop. If you do this before allowing the card reader to go into runtime suspend, does it then resume okay?
(Yes, I know you still won't be able to read it because of the FAILFAST flag. I just want to know if the runtime resume actually takes place.)
Alan Stern
Re: [PATCH] block: Fix bug in runtime-resume handling
Quoted 38 lines · Alan Stern
Quoted 13 lines · Martin Kepplinger
Quoted 12 lines · Martin Kepplinger
Quoted 9 lines · Alan Stern
Martin:
(I forgot to ask this question several weeks ago, while you were running your tests. Better ask it now before I forget again...)
I suspect the old runtime-PM code in the block layer would have worked okay in your SD cardreader test if the
BLK_MQ_REQ_PREEMPTflag had not been set. Do you know why the flag was set, or what line of code caused it to be set?Quoted 8 lines
Quoted 1 line
Correct. if not set, I could handle all I need in the scsi error path.
this thread becomes a bit confusing. I thought about
REQ_FAILFAST_DEVbut you're talking about something different.the only place I see
BLK_MQ_REQ_PREEMPTgetting passed on is in__scsi_execute()which is the case when mounting/unmounting. At least that about the only place I can find.Ah yes, I see what you mean.
Quoted 9 lines
I remember *only* your block pm fix would let me mount/unmount, but not use files yet (
REQ_FAILFAST_DEVand so on).When I revert your fix and remove
BLK_MQ_REQ_PREEMPTfrom being passed on toblk_get_request()in__scsi_execute(), that line gets executed exactly once during startup and I'm missing the/dev/sdadevice from the cardreader then.Is this what you're asking?
Not quite sure, but it doesn't matter. Removing
BLK_MQ_REQ_PREEMPTin__scsi_execute()is probably not a safe thing to do.
Quoted 7 lines
Instead, look at
sd_resume(). That routine calls__scsi_execute()indirectly throughsd_start_stop_device(), and the only reason it does this is because thesdkp->device->manage_start_stopflag is set. You ought to be able to clear this flag in sysfs, by writing to/sys/block/sda/device/scsi_disk/*/manage_start_stop. If you do this before allowing the card reader to go into runtime suspend, does it then resume okay?
manage_start_stop in sysfs is 0 here.
Quoted 6 lines
(Yes, I know you still won't be able to read it because of the FAILFAST flag. I just want to know if the runtime resume actually takes place.)
Alan Stern
Re: [PATCH] block: Fix bug in runtime-resume handling
Quoted 10 lines · Martin Kepplinger
Quoted 7 lines · Alan Stern
Instead, look at
sd_resume(). That routine calls__scsi_execute()indirectly throughsd_start_stop_device(), and the only reason it does this is because thesdkp->device->manage_start_stopflag is set. You ought to be able to clear this flag in sysfs, by writing to/sys/block/sda/device/scsi_disk/*/manage_start_stop. If you do this before allowing the card reader to go into runtime suspend, does it then resume okay?
manage_start_stopin sysfs is 0 here.
Hmmm. I'm wondering about something you wrote back in June
(https://marc.info/?l=linux-scsi&m=159345778431615&w=2):
blk_queue_enter() always - especially when sd is runtime suspended and I try to mount as above - sets success to be true for me, so never continues down to bkl_pm_request_resume(). All I see is "PM: Removing info for No Bus:sda1".
blk_queue_enter() would always set success to be true because pm (derived from the BLK_MQ_REQ_PREEMPT flag) is true. But why was the BLK_MQ_REQ_PREEMPT flag set? In other words, where was blk_queue_enter() called from?
Can you get a stack trace (i.e., call dump_stack()) at exactly this point, that is, when pm is true and q->rpm_status is RPM_SUSPENDED? Or do you already know the answer?
Alan Stern
Re: [PATCH] block: Fix bug in runtime-resume handling
Quoted 20 lines · Alan Stern
Quoted 10 lines · Martin Kepplinger
Quoted 7 lines · Alan Stern
Instead, look at
sd_resume(). That routine calls__scsi_execute()indirectly throughsd_start_stop_device(), and the only reason it does this is because thesdkp->device->manage_start_stopflag is set. You ought to be able to clear this flag in sysfs, by writing to/sys/block/sda/device/scsi_disk/*/manage_start_stop. If you do this before allowing the card reader to go into runtime suspend, does it then resume okay?
manage_start_stopin sysfs is 0 here.Hmmm. I'm wondering about something you wrote back in June
(https://marc.info/?l=linux-scsi&m=159345778431615&w=2):
blk_queue_enter()always - especially when sd is runtime suspended and I try to mount as above - sets success to be true for me, so never continues down tobkl_pm_request_resume(). All I see is "PM: Removing info for No Bus:sda1".
Quoted 10 lines
blk_queue_enter()would always set success to be true because pm (derived from theBLK_MQ_REQ_PREEMPTflag) is true. But why was theBLK_MQ_REQ_PREEMPTflag set? In other words, where wasblk_queue_enter()called from?Can you get a stack trace (i.e., call
dump_stack()) at exactly this point, that is, when pm is true andq->rpm_statusisRPM_SUSPENDED? Or do you already know the answer?
I reverted any scsi/block out-of-tree fixes for this.
when I try to mount, pm is TRUE (BLK_MQ_REQ_PREEMT set) and that's the first stack trace I get in this condition, inside of blk_queue_enter():
There is more, but I don't know if that's interesting.
[ 38.642202] CPU: 2 PID: 1522 Comm: mount Not tainted 5.8.0-1-librem5 #487 [ 38.642207] Hardware name: Purism Librem 5r3 (DT) [ 38.642213] Call trace: [ 38.642233] dump_backtrace+0x0/0x210 [ 38.642242] show_stack+0x20/0x30 [ 38.642252] dump_stack+0xc8/0x128 [ 38.642262] blk_queue_enter+0x1b8/0x2d8 [ 38.642271] blk_mq_alloc_request+0x54/0xb0 [ 38.642277] blk_get_request+0x34/0x78 [ 38.642286] __scsi_execute+0x60/0x1c8 [ 38.642291] scsi_test_unit_ready+0x88/0x118 [ 38.642298] sd_check_events+0x110/0x158 [ 38.642306] disk_check_events+0x68/0x188 [ 38.642312] disk_clear_events+0x84/0x198 [ 38.642320] check_disk_change+0x38/0x90 [ 38.642325] sd_open+0x60/0x148 [ 38.642330] __blkdev_get+0xcc/0x4c8 [ 38.642335] __blkdev_get+0x278/0x4c8 [ 38.642339] blkdev_get+0x128/0x1a8 [ 38.642345] blkdev_open+0x98/0xb0 [ 38.642354] do_dentry_open+0x130/0x3c8 [ 38.642359] vfs_open+0x34/0x40 [ 38.642366] path_openat+0xa30/0xe40 [ 38.642372] do_filp_open+0x84/0x100 [ 38.642377] do_sys_openat2+0x1f4/0x2b0 [ 38.642382] do_sys_open+0x60/0xa8
(...)
and of course it doesn't work and /dev/sda1 disappears, see the initial discussion that led to your fix.
Re: [PATCH] block: Fix bug in runtime-resume handling
Quoted 53 lines · Martin Kepplinger
Quoted 8 lines · Alan Stern
Hmmm. I'm wondering about something you wrote back in June
(https://marc.info/?l=linux-scsi&m=159345778431615&w=2):
blk_queue_enter()always - especially when sd is runtime suspended and I try to mount as above - sets success to be true for me, so never continues down tobkl_pm_request_resume(). All I see is "PM: Removing info for No Bus:sda1".Quoted 10 lines
blk_queue_enter()would always set success to be true because pm (derived from theBLK_MQ_REQ_PREEMPTflag) is true. But why was theBLK_MQ_REQ_PREEMPTflag set? In other words, where wasblk_queue_enter()called from?Can you get a stack trace (i.e., call
dump_stack()) at exactly this point, that is, when pm is true andq->rpm_statusisRPM_SUSPENDED? Or do you already know the answer?I reverted any scsi/block out-of-tree fixes for this.
when I try to mount, pm is TRUE (
BLK_MQ_REQ_PREEMTset) and that's the first stack trace I get in this condition, inside ofblk_queue_enter():There is more, but I don't know if that's interesting.
[ 38.642202] CPU: 2 PID: 1522 Comm: mount Not tainted 5.8.0-1-librem5 #487 [ 38.642207] Hardware name: Purism Librem 5r3 (DT) [ 38.642213] Call trace: [ 38.642233] dump_backtrace+0x0/0x210 [ 38.642242] show_stack+0x20/0x30 [ 38.642252] dump_stack+0xc8/0x128 [ 38.642262] blk_queue_enter+0x1b8/0x2d8 [ 38.642271] blk_mq_alloc_request+0x54/0xb0 [ 38.642277] blk_get_request+0x34/0x78 [ 38.642286] __scsi_execute+0x60/0x1c8 [ 38.642291] scsi_test_unit_ready+0x88/0x118 [ 38.642298] sd_check_events+0x110/0x158 [ 38.642306] disk_check_events+0x68/0x188 [ 38.642312] disk_clear_events+0x84/0x198 [ 38.642320] check_disk_change+0x38/0x90 [ 38.642325] sd_open+0x60/0x148 [ 38.642330] __blkdev_get+0xcc/0x4c8 [ 38.642335] __blkdev_get+0x278/0x4c8 [ 38.642339] blkdev_get+0x128/0x1a8 [ 38.642345] blkdev_open+0x98/0xb0 [ 38.642354] do_dentry_open+0x130/0x3c8 [ 38.642359] vfs_open+0x34/0x40 [ 38.642366] path_openat+0xa30/0xe40 [ 38.642372] do_filp_open+0x84/0x100 [ 38.642377] do_sys_openat2+0x1f4/0x2b0 [ 38.642382] do_sys_open+0x60/0xa8
Quoted 4 lines
(...)
and of course it doesn't work and
/dev/sda1disappears, see the initial discussion that led to your fix.
Great! That's exactly what I was looking for, thank you.
Bart, this is a perfect example of the potential race I've been talking about in the other email thread. Suppose thread 0 is carrying out a runtime suspend of a SCSI disk and at the same time, thread 1 is opening the disk's block device (as we see in the stack trace here). Then we could have the following:
Thread 0 Thread 1
-------- --------
Start runtime suspend
blk_pre_runtime_suspend calls
blk_set_pm_only and sets
q->rpm_status to RPM_SUSPENDING
Call sd_open -> ... -> scsi_test_unit_ready
-> __scsi_execute -> ...
-> blk_queue_enter
Sees BLK_MQ_REQ_PREEMPT set and
RPM_SUSPENDING queue status, so does not postpone the request
blk_post_runtime_suspend sets
q->rpm_status to RPM_SUSPENDED
The drive goes into runtime suspend
Issues the TEST UNIT READY request Request fails because the drive is suspended
One way to avoid this race is mutual exclusion: We could make sd_open prevent the drive from being runtime suspended until it returns. However I don't like this approach; it would mean tracking down every possible pathway to __scsi_execute and making sure that runtime suspend is blocked.
A more fine-grained approach would be to have __scsi_execute itself call scsi_autopm_get/put_device whenever the rq_flags argument doesn't contain RQF_PM. This way we wouldn't have to worry about missing any possiible pathways. But it relies on an implicit assumption that __scsi_execute is the only place where the PREEMPT flag gets set.
A third possibility is the approach I outlined before, adding a BLK_MQ_REQ_PM flag. But to avoid the deadlock you pointed out, I would make blk_queue_enter smarter about whether to postpone a request. The logic would go like this:
If !blk_queue_pm_only: Allow If !BLK_MQ_REQ_PREEMPT: Postpone If q->rpm_status is RPM_ACTIVE: Allow If !BLK_MQ_REQ_PM: Postpone If q->rpm_status is RPM_SUSPENDED: Postpone Else: Allow
The assumption here is that the PREEMPT flag is set whenever the PM flag is.
I believe either the second or third possibility would work. The second looks to be the simplest
What do you think?
Alan Stern
Re: [PATCH] block: Fix bug in runtime-resume handling
Quoted 18 lines · Alan Stern
A third possibility is the approach I outlined before, adding a
BLK_MQ_REQ_PMflag. But to avoid the deadlock you pointed out, I would makeblk_queue_entersmarter about whether to postpone a request. The logic would go like this:If !blk_queue_pm_only: Allow If !BLK_MQ_REQ_PREEMPT: Postpone If q->rpm_status is RPM_ACTIVE: Allow If !BLK_MQ_REQ_PM: Postpone If q->rpm_status is RPM_SUSPENDED: Postpone Else: Allow
Quoted 2 lines
The assumption here is that the PREEMPT flag is set whenever the PM flag is.
Hi Alan,
Although interesting, these solutions sound like workarounds to me. How about fixing the root cause by modifying the SCSI DV implementation such that it doesn't use scsi_device_quiesce() anymore()? That change would allow to remove BLK_MQ_REQ_PREEMPT / RQF_PREEMPT from the block layer and move these flags into the SCSI and IDE code.
Thanks,
Bart.
Re: [PATCH] block: Fix bug in runtime-resume handling
Quoted 22 lines · Bart Van Assche
Quoted 18 lines · Alan Stern
A third possibility is the approach I outlined before, adding a
BLK_MQ_REQ_PMflag. But to avoid the deadlock you pointed out, I would makeblk_queue_entersmarter about whether to postpone a request. The logic would go like this:If !blk_queue_pm_only: Allow If !BLK_MQ_REQ_PREEMPT: Postpone If q->rpm_status is RPM_ACTIVE: Allow If !BLK_MQ_REQ_PM: Postpone If q->rpm_status is RPM_SUSPENDED: Postpone Else: AllowQuoted 2 lines
The assumption here is that the PREEMPT flag is set whenever the PM flag is.
Quoted 7 lines
Hi Alan,
Although interesting, these solutions sound like workarounds to me. How about fixing the root cause by modifying the SCSI DV implementation such that it doesn't use
scsi_device_quiesce()anymore()? That change would allow to removeBLK_MQ_REQ_PREEMPT/RQF_PREEMPTfrom the block layer and move these flags into the SCSI and IDE code.
That's a perfectly reasonable approach, but I have no idea how to do it. Any suggestions?
Alan Stern
Quoted 44 lines · James Bottomley
Quoted 38 lines · James Bottomley
Quoted 22 lines · Alan Stern
Quoted 21 lines · Martin Kepplinger
Quoted 10 lines · Alan Stern
Quoted 7 lines · Martin Kepplinger
Hi Alan,
Any API cleanup is of course welcome. I just wanted to remind you that the underlying problem: broken block device runtime pm. Your initial proposed fix "almost" did it and mounting works but during file access, it still just looks like a
runtime_resumeis missing somewhere.Quoted 4 lines
Well, I have tested that proposed fix several times, and on my system it's working perfectly. When I stop accessing a drive it autosuspends, and when I access it again it gets resumed and works -- as you would expect.
that's weird. when I mount, everything looks good, "sda1". But as soon as I cd to the mountpoint and do "ls" (on another SD card "ls" works but actual file reading leads to the exact same errors), I get:
Quoted 8 lines
Quoted 6 lines
[ 77.474632] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 cmd_age=0s [ 77.474647] sd 0:0:0:0: [sda] tag#0 Sense Key : 0x6 [current] [ 77.474655] sd 0:0:0:0: [sda] tag#0 ASC=0x28 ASCQ=0x0 [ 77.474667] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 0000 60 40 00 00 01 00
This error report comes from the SCSI layer, not the block layer.
That sense code means "NOT READY TO READY CHANGE, MEDIUM MAY HAVE CHANGED" so it sounds like it something we should be ignoring. Usually this signals a problem, like you changed the medium manually (ejected the CD). But in this case you can tell us to expect this by setting
Quoted 4 lines
sdev->expecting_cc_uaAnd we'll retry. I think you need to set this on all resumed devices.
Quoted 7 lines
Actually, it's not quite that easy, we filter out this ASC/ASCQ combination from the check because we should never ignore medium might have changed events on running devices. We could ignore it if we had a flag to say the power has been yanked (perhaps an additional sdev flag you set on resume) but we would still miss the case where you really had powered off the drive and then changed the media ... if you can regard this as the user's problem, then we might have a solution.
Indeed, I was going to make the same point.
The only reasonable conclusion is that suspending these SD card readers isn't safe unless they don't contain a card -- or maybe just if the device file isn't open or mounted.
Although support for this sort of thing could be added to the kernel, for now it's best to rely on userspace doing the right thing. The kernel doesn't even know which devices suffer from this problem.
Alan Stern
James Bottomley
maintainerQuoted 2 lines · Alan Stern
Quoted 1 line · James Bottomley
On Wed, 2020-07-29 at 07:46 -0700, James Bottomley wrote:
[...]
Quoted 20 lines
Quoted 11 lines
Quoted 10 lines
That sense code means "NOT READY TO READY CHANGE, MEDIUM MAY HAVE CHANGED" so it sounds like it something we should be ignoring. Usually this signals a problem, like you changed the medium manually (ejected the CD). But in this case you can tell us to expect this by setting
sdev->expecting_cc_uaAnd we'll retry. I think you need to set this on all resumed devices.
Quoted 8 lines
Actually, it's not quite that easy, we filter out this ASC/ASCQ combination from the check because we should never ignore medium might have changed events on running devices. We could ignore it if we had a flag to say the power has been yanked (perhaps an additional sdev flag you set on resume) but we would still miss the case where you really had powered off the drive and then changed the media ... if you can regard this as the user's problem, then we might have a solution.
Quoted 5 lines
Indeed, I was going to make the same point.
The only reasonable conclusion is that suspending these SD card readers isn't safe unless they don't contain a card -- or maybe just if the device file isn't open or mounted.
For standard removable media, I could see issuing an eject before suspend to emphasize the point. However, sd cards don't work like that ... they're not ejectable except manually and mostly used as the HDD of embedded, so the 99% use case is that the user will suspend and resume the device with the same sdd insert. It does sound like a use case we should support.
Quoted 4 lines
Although support for this sort of thing could be added to the kernel, for now it's best to rely on userspace doing the right thing. The kernel doesn't even know which devices suffer from this problem.
Can we solve from userspace the case where the user hasn't changed the media and the resume fails because we don't know?
James
Quoted 24 lines · James Bottomley
Quoted 2 lines · Alan Stern
Quoted 1 line · James Bottomley
On Wed, 2020-07-29 at 07:46 -0700, James Bottomley wrote:
[...]
Quoted 20 lines
Quoted 19 lines
Quoted 10 lines
That sense code means "NOT READY TO READY CHANGE, MEDIUM MAY HAVE CHANGED" so it sounds like it something we should be ignoring. Usually this signals a problem, like you changed the medium manually (ejected the CD). But in this case you can tell us to expect this by setting
sdev->expecting_cc_uaAnd we'll retry. I think you need to set this on all resumed devices.
Actually, it's not quite that easy, we filter out this ASC/ASCQ combination from the check because we should never ignore medium might have changed events on running devices. We could ignore it if we had a flag to say the power has been yanked (perhaps an additional sdev flag you set on resume) but we would still miss the case where you really had powered off the drive and then changed the media ... if you can regard this as the user's problem, then we might have a solution.
Quoted 8 lines
Quoted 5 lines
Indeed, I was going to make the same point.
The only reasonable conclusion is that suspending these SD card readers isn't safe unless they don't contain a card -- or maybe just if the device file isn't open or mounted.
For standard removable media, I could see issuing an eject before suspend to emphasize the point.
That's not a workable approach in general. For example, you wouldn't want to eject a CD just because it hadn't been used for a couple of minutes and the drive was therefore about to be suspended.
Quoted 5 lines
However, sd cards don't work like that
... they're not ejectable except manually and mostly used as the HDD of embedded, so the 99% use case is that the user will suspend and resume the device with the same sdd insert. It does sound like a use case we should support.
Agreed.
Quoted 7 lines
Quoted 4 lines
Although support for this sort of thing could be added to the kernel, for now it's best to rely on userspace doing the right thing. The kernel doesn't even know which devices suffer from this problem.
Can we solve from userspace the case where the user hasn't changed the media and the resume fails because we don't know?
The difficulty is recognizing situations where the media really was changed while the device was suspended. Most devices, AFAIK, don't have any problem with this, but card readers often do.
I suppose the kernel could just rely on users not to change media in drives while they are mounted. Then any problems that arise would be the users' fault. Yes, this is nothing more than passing the buck, but it's hard to come up with any better approach. Doesn't Windows do something like this?
Alan Stern
Quoted 55 lines · Alan Stern
Quoted 44 lines · James Bottomley
Quoted 38 lines · James Bottomley
Quoted 22 lines · Alan Stern
Quoted 21 lines · Martin Kepplinger
Quoted 10 lines · Alan Stern
Quoted 7 lines · Martin Kepplinger
Hi Alan,
Any API cleanup is of course welcome. I just wanted to remind you that the underlying problem: broken block device runtime pm. Your initial proposed fix "almost" did it and mounting works but during file access, it still just looks like a
runtime_resumeis missing somewhere.Quoted 4 lines
Well, I have tested that proposed fix several times, and on my system it's working perfectly. When I stop accessing a drive it autosuspends, and when I access it again it gets resumed and works -- as you would expect.
that's weird. when I mount, everything looks good, "sda1". But as soon as I cd to the mountpoint and do "ls" (on another SD card "ls" works but actual file reading leads to the exact same errors), I get:
Quoted 8 lines
Quoted 6 lines
[ 77.474632] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 cmd_age=0s [ 77.474647] sd 0:0:0:0: [sda] tag#0 Sense Key : 0x6 [current] [ 77.474655] sd 0:0:0:0: [sda] tag#0 ASC=0x28 ASCQ=0x0 [ 77.474667] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 0000 60 40 00 00 01 00
This error report comes from the SCSI layer, not the block layer.
That sense code means "NOT READY TO READY CHANGE, MEDIUM MAY HAVE CHANGED" so it sounds like it something we should be ignoring. Usually this signals a problem, like you changed the medium manually (ejected the CD). But in this case you can tell us to expect this by setting
Quoted 4 lines
sdev->expecting_cc_uaAnd we'll retry. I think you need to set this on all resumed devices.
Quoted 7 lines
Actually, it's not quite that easy, we filter out this ASC/ASCQ combination from the check because we should never ignore medium might have changed events on running devices. We could ignore it if we had a flag to say the power has been yanked (perhaps an additional sdev flag you set on resume) but we would still miss the case where you really had powered off the drive and then changed the media ... if you can regard this as the user's problem, then we might have a solution.
Indeed, I was going to make the same point.
Quoted 9 lines
The only reasonable conclusion is that suspending these SD card readers isn't safe unless they don't contain a card -- or maybe just if the device file isn't open or mounted.
Although support for this sort of thing could be added to the kernel, for now it's best to rely on userspace doing the right thing. The kernel doesn't even know which devices suffer from this problem.
well, userspace can do something like "automatically unmount if not used" but that is *way* more inefficient than if the kernel would support this is some way or another, for SD cards.
Quoted 19 lines · Alan Stern
Quoted 18 lines · Martin Kepplinger
Quoted 8 lines · Alan Stern
Quoted 6 lines · Martin Kepplinger
Hi Alan,
Any API cleanup is of course welcome. I just wanted to remind you that the underlying problem: broken block device runtime pm. Your initial proposed fix "almost" did it and mounting works but during file access, it still just looks like a
runtime_resumeis missing somewhere.Quoted 4 lines
Well, I have tested that proposed fix several times, and on my system it's working perfectly. When I stop accessing a drive it autosuspends, and when I access it again it gets resumed and works -- as you would expect.
that's weird. when I mount, everything looks good, "sda1". But as soon as I cd to the mountpoint and do "ls" (on another SD card "ls" works but actual file reading leads to the exact same errors), I get:
Quoted 8 lines
Quoted 6 lines
[ 77.474632] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08 cmd_age=0s [ 77.474647] sd 0:0:0:0: [sda] tag#0 Sense Key : 0x6 [current] [ 77.474655] sd 0:0:0:0: [sda] tag#0 ASC=0x28 ASCQ=0x0 [ 77.474667] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 00 6040 00 00 01 00
This error report comes from the SCSI layer, not the block layer.
SCSI's first 11 byte command! I'm guessing the first byte is being repeated and it's actually:
28 00 00 00 60 40 00 00 01 00 [READ(10)]
That should be fixed. It should be something like: "...CDB in hex: 28 00 ...".
Doug Gilbert
Quoted 65 lines
Quoted 29 lines
[ 77.474678] blk_update_request: I/O error, dev sda, sector 24640 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 77.485836] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 77.491628] blk_update_request: I/O error, dev sda, sector 24641 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 77.502275] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 77.508051] blk_update_request: I/O error, dev sda, sector 24642 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 [ 77.518651] sd 0:0:0:0: [sda] tag#0 device offline or changed (...) [ 77.947653] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 77.953434] FAT-fs (sda1): Directory bread(block 16448) failed [ 77.959333] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 77.965118] FAT-fs (sda1): Directory bread(block 16449) failed [ 77.971014] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 77.976802] FAT-fs (sda1): Directory bread(block 16450) failed [ 77.982698] sd 0:0:0:0: [sda] tag#0 device offline or changed (...) [ 78.384929] FAT-fs (sda1): Filesystem has been set read-only [ 103.070973] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 103.076751] print_req_error: 118 callbacks suppressed [ 103.076760] blk_update_request: I/O error, dev sda, sector 9748 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0 [ 103.087428] Buffer I/O error on dev sda1, logical block 1556, lost async page write [ 103.095309] sd 0:0:0:0: [sda] tag#0 device offline or changed [ 103.101123] blk_update_request: I/O error, dev sda, sector 17162 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0 [ 103.111883] Buffer I/O error on dev sda1, logical block 8970, lostQuoted 1 line
async page write
I can't tell why you're getting that error. In one of my tests the device returned the same kind of error status (Sense Key = 6, ASC =
0x28) but the operation was then retried successfully. Perhaps the problem lies in the device you are testing.Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
As we need to have that working at some point, I might look into it, but someone who has experience in the block layer can surely do it more efficiently.
I suspect that any problems you still face are caused by something else.
Quoted 8 lines
I then formatted sda1 to ext2 (on the runtime suspend system testing your patch) and that seems to have worked!
Again accessing the mountpoint then yield the very same "device offline or changed" errors.
What kind of device are you testing? You should be easily able to reproduce this using an "sd" device.
I tested two devices: a SanDisk Cruzer USB flash drive and a g-mass-storage gadget running under dummy-hcd. They each showed up as
/dev/sdbon my system.I haven't tried testing with an SD card. If you have any specific sequence of commands you would like me to run, let me know.
Quoted 1 line
The problems must lie in the different other drivers we use I guess.
Or the devices. Have you tried testing with a USB flash drive?
Alan Stern
Martin Kepplinger
Tested-byQuoted 23 lines · Alan Stern
Quoted 10 lines · Martin Kepplinger
Quoted 6 lines · Alan Stern
Martin's best approach would be to add some debugging code to find out why
blk_queue_enter()isn't callingbkl_pm_request_resume(), or why that call doesn't lead topm_request_resume().Alan Stern
Quoted 6 lines
Hi Alan,
blk_queue_enter()always - especially when sd is runtime suspended and I try to mount as above - sets success to be true for me, so never continues down tobkl_pm_request_resume(). All I see is "PM: Removing info for No Bus:sda1".Aha. Looking at this more closely, it's apparent that the code in
blk-core.ccontains a logic bug: It assumes that if theBLK_MQ_REQ_PREEMPTflag is set then the request can be issued regardless of the queue's runtime status. That is not correct when the queue is suspended.
Quoted 7 lines
Below is my attempt to fix this up. I'm not sure that the patch is entirely correct, but it should fix this logic bug. I would appreciate a critical review.
Martin, does this fix the problem?
Alan Stern
Hi Alan,
So in the block layer your change below indeed fixes the problem and if you want to submit that 1:1 feel free to add
thanks for your help in this!
martin
Quoted 62 lines
Index:
usb-devel/block/blk-core.c
===================================================================block/blk-core.c +3
-3@@ -423,7 +423,8 @@ int blk_queue_enter(struct request_queue* responsible for ensuring that that counter is * globally visible before the queue is unfrozen. */ -if (pm || !blk_queue_pm_only (q)) { +if ((pm && q->rpm_status !=RPM_SUSPENDED ) || + !blk_queue_pm_only (q)) { success = true; }else {percpu_ref_put (&q->q_usage_counter);@@ -448,8 +449,7 @@ int blk_queue_enter(struct request_queuewait_event (q->mq_freeze_wq, (!q->mq_freeze_depth && - (pm || (blk_pm_request_resume (q), - !blk_queue_pm_only (q)))) || +blk_pm_resume_queue (pm, q)) ||blk_queue_dying (q));if (blk_queue_dying (q))return -ENODEV ;Index:
usb-devel/block/blk-pm.h
===================================================================block/blk-pm.h +9
-5@@ -6,11 +6,14 @@#include <linux/pm_runtime.h> #ifdef CONFIG_PM -static inline void blk_pm_request_resume (struct request_queue *q) +static inline int blk_pm_resume_queue (const bool pm,struct request_queue *q) { -if (q->dev && (q->rpm_status ==RPM_SUSPENDED || - q->rpm_status ==RPM_SUSPENDING )) -pm_request_resume (q->dev); +if (!q->dev || !blk_queue_pm_only (q)) +return 1 ;/* Nothing to do */ +if (pm && q->rpm_status !=RPM_SUSPENDED ) +return 1 ;/* Request allowed */ +pm_request_resume (q->dev); +return 0 ; }static inline void blk_pm_mark_last_busy (struct request *rq)@@ -44,8 +47,9 @@ static inline void blk_pm_put_request(st --rq->q->nr_pending; }#else -static inline void blk_pm_request_resume (struct request_queue *q) +static inline int blk_pm_resume_queue (const bool pm,struct request_queue *q) { +return 1 ; }static inline void blk_pm_mark_last_busy (struct request *rq)
Quoted 32 lines · Martin Kepplinger
Quoted 23 lines · Alan Stern
Quoted 10 lines · Martin Kepplinger
Quoted 6 lines · Alan Stern
Martin's best approach would be to add some debugging code to find out why
blk_queue_enter()isn't callingbkl_pm_request_resume(), or why that call doesn't lead topm_request_resume().Alan Stern
Quoted 6 lines
Hi Alan,
blk_queue_enter()always - especially when sd is runtime suspended and I try to mount as above - sets success to be true for me, so never continues down tobkl_pm_request_resume(). All I see is "PM: Removing info for No Bus:sda1".Aha. Looking at this more closely, it's apparent that the code in
blk-core.ccontains a logic bug: It assumes that if theBLK_MQ_REQ_PREEMPTflag is set then the request can be issued regardless of the queue's runtime status. That is not correct when the queue is suspended.Quoted 7 lines
Below is my attempt to fix this up. I'm not sure that the patch is entirely correct, but it should fix this logic bug. I would appreciate a critical review.
Martin, does this fix the problem?
Alan Stern
Quoted 8 lines
Hi Alan,
So in the block layer your change below indeed fixes the problem and if you want to submit that 1:1 feel free to add
Tested-by Martin Kepplinger <[email protected]>thanks for your help in this!
Thank you for _your_ help!
The next merge window is coming up soon. I think I'll wait until it is over before submitting the patch (maintainers tend to be too busy to consider new patches during a merge window).
But I am still open to comments or criticism of the patch in the meantime. There hasn't been any feedback since Bart's initial set of questions.
Alan Stern