Reduce ATA disk resume time
Subsystems (3)
v2 cover Reduce ATA disk resume time
Hi Martin,
Recently it was reported that patch "scsi: core: pm: Rely on the device driver core for async power management" causes resuming to take longer if an ATA disk is present. This patch series fixes that regression. Please consider this patch series for kernel v5.20.
Thanks,
Bart.
Changes compared to v1: addressed John Garry's two comments.
Bart Van Assche (2):
scsi: core: Move the definition of SCSI_QUEUE_DELAY
scsi: sd: Rework asynchronous resume support
drivers/scsi/scsi_lib.c | 14 +++---- drivers/scsi/sd.c | 84 ++++++++++++++++++++++++++++++++--------- drivers/scsi/sd.h | 5 +++ 3 files changed, 78 insertions(+), 25 deletions(-)
PATCH v2 1/2 scsi: core: Move the definition of SCSI_QUEUE_DELAY
Move the definition of SCSI_QUEUE_DELAY to just above the function that uses it.
--- drivers/scsi/scsi_lib.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
drivers/scsi/scsi_lib.c +7 -7
@@ -75,13 +75,6 @@ int scsi_init_sense_cache(struct Scsi_Host *shost)return ret; } -/* -* When to reinvoke queueing after a resource shortage. It's 3 msecs to -* not change behaviour from the previous unplug mechanism, experimentation -* may prove this needs changing. -*/ -#define SCSI_QUEUE_DELAY 3 -static void scsi_set_blocked (struct scsi_cmnd *cmd,int reason) {
@@ -1648,6 +1641,13 @@ static void scsi_mq_put_budget(struct request_queue *q, int budget_token)sbitmap_put (&sdev->budget_map, budget_token); } +/* +* When to reinvoke queueing after a resource shortage. It's 3 msecs to +* not change behaviour from the previous unplug mechanism, experimentation +* may prove this needs changing. +*/ +#define SCSI_QUEUE_DELAY 3 +static int scsi_mq_get_budget (struct request_queue *q) {struct scsi_device *sdev = q->queuedata;
PATCH v2 2/2 scsi: sd: Rework asynchronous resume support
In reply to Bart Van AsscheFor some technologies, e.g. an ATA bus, resuming can take multiple seconds. Waiting for resume to finish can cause a very noticeable delay. Hence this patch that restores the behavior from before patch "scsi:
core: pm: Rely on the device driver core for async power management" for most SCSI devices.
This patch introduces a behavior change: if the START command fails, do not consider this as a SCSI disk resume failure.
--- drivers/scsi/sd.c | 84 +++++++++++++++++++++++++++++++++++++---------- drivers/scsi/sd.h | 5 +++ 2 files changed, 71 insertions(+), 18 deletions(-)
drivers/scsi/sd.c +66 -18
@@ -103,6 +103,7 @@ static void sd_config_discard(struct scsi_disk *, unsigned int);static void sd_config_write_same (struct scsi_disk *);static int sd_revalidate_disk (struct gendisk *);static void sd_unlock_native_capacity (struct gendisk *disk); +static void sd_start_done_work (struct work_struct *work);static int sd_probe (struct device *);static int sd_remove (struct device *);static void sd_shutdown (struct device *);
@@ -3463,6 +3464,7 @@ static int sd_probe(struct device *dev) sdkp->max_retries =SD_MAX_RETRIES ;atomic_set (&sdkp->openers,0 );atomic_set (&sdkp->device->ioerr_cnt,0 ); +INIT_WORK (&sdkp->start_done_work, sd_start_done_work);if (!sdp->request_queue->rq_timeout) {if (sdp->type !=TYPE_MOD )
@@ -3585,12 +3587,69 @@ static void scsi_disk_release(struct device *dev)kfree (sdkp); } +/* Process sense data after a START command finished. */ +static void sd_start_done_work (struct work_struct *work) +{ +struct scsi_disk *sdkp =container_of (work,typeof (*sdkp), + start_done_work); +struct scsi_sense_hdr sshdr; +int res = sdkp->start_result; + +if (res ==0 ) +return ; + +sd_print_result (sdkp,"Start/Stop Unit failed" , res); + +if (res <0 ) +return ; + +if (scsi_normalize_sense (sdkp->start_sense_buffer, + sdkp->start_sense_len, &sshdr)) +sd_print_sense_hdr (sdkp, &sshdr); +} + +/* A START command finished. May be called from interrupt context. */ +static void sd_start_done (struct request *req,blk_status_t status) +{ +const struct scsi_cmnd *scmd =blk_mq_rq_to_pdu (req); +struct scsi_disk *sdkp =scsi_disk (req->q->disk); + + sdkp->start_result = scmd->result; +WARN_ON_ONCE (scmd->sense_len >SCSI_SENSE_BUFFERSIZE ); + sdkp->start_sense_len = scmd->sense_len; +memcpy (sdkp->start_sense_buffer, scmd->sense_buffer, +ARRAY_SIZE (sdkp->start_sense_buffer)); +WARN_ON_ONCE (!schedule_work (&sdkp->start_done_work)); +} + +/* Submit a START command asynchronously. */ +static int sd_submit_start (struct scsi_disk *sdkp,u8 cmd[],u8 cmd_len) +{ +struct scsi_device *sdev = sdkp->device; +struct request_queue *q = sdev->request_queue; +struct request *req; +struct scsi_cmnd *scmd; + + req =scsi_alloc_request (q,REQ_OP_DRV_IN ,BLK_MQ_REQ_PM ); +if (IS_ERR (req)) +return PTR_ERR (req); + + scmd =blk_mq_rq_to_pdu (req); + scmd->cmd_len = cmd_len; +memcpy (scmd->cmnd, cmd, cmd_len); + scmd->allowed = sdkp->max_retries; + req->timeout =SD_TIMEOUT ; + req->rq_flags |=RQF_PM |RQF_QUIET ; + req->end_io = sd_start_done; +blk_execute_rq_nowait (req,/*at_head=*/ true); + +return 0 ; +} +static int sd_start_stop_device (struct scsi_disk *sdkp,int start) {unsigned char cmd[6 ] = {START_STOP };/* START_VALID */ -struct scsi_sense_hdr sshdr;struct scsi_device *sdp = sdkp->device; -int res;if (start) cmd[4 ] |=1 ;/* START */
@@ -3601,23 +3660,10 @@ static int sd_start_stop_device(struct scsi_disk *sdkp, int start)if (!scsi_device_online (sdp))return -ENODEV ; - res =scsi_execute (sdp, cmd,DMA_NONE ,NULL ,0 ,NULL , &sshdr, -SD_TIMEOUT , sdkp->max_retries,0 ,RQF_PM ,NULL ); -if (res) { -sd_print_result (sdkp,"Start/Stop Unit failed" , res); -if (res >0 &&scsi_sense_valid (&sshdr)) { -sd_print_sense_hdr (sdkp, &sshdr); -/* 0x3a is medium not present */ -if (sshdr.asc ==0x3a ) - res =0 ; - } - } +/* Wait until processing of sense data has finished. */ +flush_work (&sdkp->start_done_work); -/* SCSI error codes must not go to the generic layer */ -if (res) -return -EIO ; - -return 0 ; +return sd_submit_start (sdkp, cmd,sizeof (cmd)); }/*
@@ -3644,6 +3690,8 @@ static void sd_shutdown(struct device *dev)sd_printk (KERN_NOTICE , sdkp,"Stopping disk\n" );sd_start_stop_device (sdkp,0 ); } + +flush_work (&sdkp->start_done_work); }static int sd_suspend_common (struct device *dev,bool ignore_stop_errors)
drivers/scsi/sd.h +5 -0
@@ -150,6 +150,11 @@ struct scsi_disk { unsigned urswrz : 1; unsigned security : 1; unsigned ignore_medium_access_errors : 1; + +int start_result; +u32 start_sense_len; +u8 start_sense_buffer[SCSI_SENSE_BUFFERSIZE ]; +struct work_struct start_done_work; };#define to_scsi_disk (obj) container_of(obj, struct scsi_disk, disk_dev)
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support
Hoi Bart,
Quoted 9 lines · Bart Van Assche
For some technologies, e.g. an ATA bus, resuming can take multiple seconds. Waiting for resume to finish can cause a very noticeable delay. Hence this patch that restores the behavior from before patch "scsi:
core: pm: Rely on the device driver core for async power management" for most SCSI devices.This patch introduces a behavior change: if the START command fails, do not consider this as a SCSI disk resume failure.
Quoted 9 lines
Cc Ming Lei <[email protected]>Cc Hannes Reinecke <[email protected]>Cc John Garry <[email protected]>Tested-by [email protected]Fixes a19a93e4c6a9 ("scsi: core: pm: Rely on the device driver core for async power management")Signed-off-by Bart Van Assche <[email protected]>
Thanks for your patch, which is now commit 88f1669019bd62b3 ("scsi: sd:
Rework asynchronous resume support") in scsi/for-next.
On the Salvator-XS development board[1] with a SATA hard drive connected, accessing the hard drive after resume from s2idle hangs. I have bisected this to the aformentioned commit, and reverting this commit fixes the issue.
[1] arch/arm64/boot/dts/renesas/r8a77951-salvator-xs.dts
Quoted 139 lines
---
drivers/scsi/sd.c| 84 +++++++++++++++++++++++++++++++++++++----------drivers/scsi/sd.h| 5 +++2 files changed, 71 insertions(+), 18 deletions(-)drivers/scsi/sd.c +66
-18@@ -103,6 +103,7 @@ static void sd_config_discard(struct scsi_disk *, unsigned int);static void sd_config_write_same (struct scsi_disk *);static int sd_revalidate_disk (struct gendisk *);static void sd_unlock_native_capacity (struct gendisk *disk); +static void sd_start_done_work (struct work_struct *work);static int sd_probe (struct device *);static int sd_remove (struct device *);static void sd_shutdown (struct device *);@@ -3463,6 +3464,7 @@ static int sd_probe(struct device *dev) sdkp->max_retries =SD_MAX_RETRIES ;atomic_set (&sdkp->openers,0 );atomic_set (&sdkp->device->ioerr_cnt,0 ); +INIT_WORK (&sdkp->start_done_work, sd_start_done_work);if (!sdp->request_queue->rq_timeout) {if (sdp->type !=TYPE_MOD )@@ -3585,12 +3587,69 @@ static void scsi_disk_release(struct device *dev)kfree (sdkp); } +/* Process sense data after a START command finished. */ +static void sd_start_done_work (struct work_struct *work) +{ +struct scsi_disk *sdkp =container_of (work,typeof (*sdkp), + start_done_work); +struct scsi_sense_hdr sshdr; +int res = sdkp->start_result; + +if (res ==0 ) +return ; + +sd_print_result (sdkp,"Start/Stop Unit failed" , res); + +if (res <0 ) +return ; + +if (scsi_normalize_sense (sdkp->start_sense_buffer, + sdkp->start_sense_len, &sshdr)) +sd_print_sense_hdr (sdkp, &sshdr); +} + +/* A START command finished. May be called from interrupt context. */ +static void sd_start_done (struct request *req,blk_status_t status) +{ +const struct scsi_cmnd *scmd =blk_mq_rq_to_pdu (req); +struct scsi_disk *sdkp =scsi_disk (req->q->disk); + + sdkp->start_result = scmd->result; +WARN_ON_ONCE (scmd->sense_len >SCSI_SENSE_BUFFERSIZE ); + sdkp->start_sense_len = scmd->sense_len; +memcpy (sdkp->start_sense_buffer, scmd->sense_buffer, +ARRAY_SIZE (sdkp->start_sense_buffer)); +WARN_ON_ONCE (!schedule_work (&sdkp->start_done_work)); +} + +/* Submit a START command asynchronously. */ +static int sd_submit_start (struct scsi_disk *sdkp,u8 cmd[],u8 cmd_len) +{ +struct scsi_device *sdev = sdkp->device; +struct request_queue *q = sdev->request_queue; +struct request *req; +struct scsi_cmnd *scmd; + + req =scsi_alloc_request (q,REQ_OP_DRV_IN ,BLK_MQ_REQ_PM ); +if (IS_ERR (req)) +return PTR_ERR (req); + + scmd =blk_mq_rq_to_pdu (req); + scmd->cmd_len = cmd_len; +memcpy (scmd->cmnd, cmd, cmd_len); + scmd->allowed = sdkp->max_retries; + req->timeout =SD_TIMEOUT ; + req->rq_flags |=RQF_PM |RQF_QUIET ; + req->end_io = sd_start_done; +blk_execute_rq_nowait (req,/*at_head=*/ true); + +return 0 ; +} +static int sd_start_stop_device (struct scsi_disk *sdkp,int start) {unsigned char cmd[6 ] = {START_STOP };/* START_VALID */ -struct scsi_sense_hdr sshdr;struct scsi_device *sdp = sdkp->device; -int res;if (start) cmd[4 ] |=1 ;/* START */ @@ -3601,23 +3660,10 @@ static int sd_start_stop_device(struct scsi_disk *sdkp, int start)if (!scsi_device_online (sdp))return -ENODEV ; - res =scsi_execute (sdp, cmd,DMA_NONE ,NULL ,0 ,NULL , &sshdr, -SD_TIMEOUT , sdkp->max_retries,0 ,RQF_PM ,NULL ); -if (res) { -sd_print_result (sdkp,"Start/Stop Unit failed" , res); -if (res >0 &&scsi_sense_valid (&sshdr)) { -sd_print_sense_hdr (sdkp, &sshdr); -/* 0x3a is medium not present */ -if (sshdr.asc ==0x3a ) - res =0 ; - } - } +/* Wait until processing of sense data has finished. */ +flush_work (&sdkp->start_done_work); -/* SCSI error codes must not go to the generic layer */ -if (res) -return -EIO ; - -return 0 ; +return sd_submit_start (sdkp, cmd,sizeof (cmd)); }/* @@ -3644,6 +3690,8 @@ static void sd_shutdown(struct device *dev)sd_printk (KERN_NOTICE , sdkp,"Stopping disk\n" );sd_start_stop_device (sdkp,0 ); } + +flush_work (&sdkp->start_done_work); }static int sd_suspend_common (struct device *dev,bool ignore_stop_errors)drivers/scsi/sd.h +0
-0@@ -150,6 +150,11 @@ struct scsi_disk { unsigned urswrz : 1;
drivers/scsi/sd.h struct scsi_disk { · line 160
Quoted 10 lines
drivers/scsi/sd.h +5
-0unsigned security : 1; unsigned ignore_medium_access_errors : 1; + +int start_result; +u32 start_sense_len; +u8 start_sense_buffer[SCSI_SENSE_BUFFERSIZE ]; +struct work_struct start_done_work; };#define to_scsi_disk (obj) container_of(obj, struct scsi_disk, disk_dev)
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support
Quoted 9 lines · Geert Uytterhoeven
Thanks for your patch, which is now commit
88f1669019bd62b3("scsi: sd:
Rework asynchronous resume support") in scsi/for-next.On the Salvator-XS development board[1] with a SATA hard drive connected, accessing the hard drive after resume from s2idle hangs. I have bisected this to the aformentioned commit, and reverting this commit fixes the issue.
[1]
arch/arm64/boot/dts/renesas/r8a77951-salvator-xs.dts
Hi Geert,
Thank you for having reported this. How about creating a bug report on https://bugzilla.kernel.org/ and attaching the kernel logs with and without this patch? It's probably better to upload the kernel logs somewhere rather than to post these on the linux-scsi mailing list.
Thanks,
Bart.
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support
Hoi Bart,
Quoted 9 lines · Bart Van Assche
Quoted 8 lines · Geert Uytterhoeven
Thanks for your patch, which is now commit
88f1669019bd62b3("scsi: sd:
Rework asynchronous resume support") in scsi/for-next.On the Salvator-XS development board[1] with a SATA hard drive connected, accessing the hard drive after resume from s2idle hangs. I have bisected this to the aformentioned commit, and reverting this commit fixes the issue.
Quoted 4 lines
Quoted 1 line
[1]
arch/arm64/boot/dts/renesas/r8a77951-salvator-xs.dtsThank you for having reported this. How about creating a bug report on
https://bugzilla.kernel.org/ and attaching the kernel logs with and
I never created a bug report on bko before...
Quoted 1 line
without this patch? It's probably better to upload the kernel logs
Unfortunately the logs do not indicate anything went wrong. I.e. no difference before/after.
Quoted 1 line
somewhere rather than to post these on the linux-scsi mailing list.
Lore links are great for after-the-fact reporting. If you follow the lore link in the buggy commit, you arrive in this thread ;-)
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support
Quoted 15 lines · Geert Uytterhoeven
Quoted 9 lines · Bart Van Assche
Quoted 8 lines · Geert Uytterhoeven
Thanks for your patch, which is now commit
88f1669019bd62b3("scsi: sd:
Rework asynchronous resume support") in scsi/for-next.On the Salvator-XS development board[1] with a SATA hard drive connected, accessing the hard drive after resume from s2idle hangs. I have bisected this to the aformentioned commit, and reverting this commit fixes the issue.
Quoted 4 lines
Quoted 1 line
[1]
arch/arm64/boot/dts/renesas/r8a77951-salvator-xs.dtsThank you for having reported this. How about creating a bug report on
https://bugzilla.kernel.org/ and attaching the kernel logs with and
Quoted 6 lines
I never created a bug report on bko before...
Quoted 1 line
without this patch? It's probably better to upload the kernel logs
Unfortunately the logs do not indicate anything went wrong. I.e. no difference before/after.
With more debug options enabled, it prints:
INFO: task kworker/0:7:283 blocked for more than 120 seconds.
Not tainted 5.19.0-rc7-salvator-x-00794-g6780eb02b605 #1287
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/0:7 state:D stack: 0 pid: 283 ppid: 2 flags:0x00000008
Workqueue: events ata_scsi_dev_rescan
Call trace:
__switch_to+0xbc/0x124
__schedule+0x540/0x71c
schedule+0x58/0xa0
io_schedule+0x18/0x34
blk_mq_get_tag+0x138/0x244
__blk_mq_alloc_requests+0x130/0x2f0
blk_mq_alloc_request+0x74/0xa8
scsi_alloc_request+0x10/0x30
__scsi_execute+0x5c/0x18c
scsi_vpd_inquiry+0x7c/0xdc
scsi_get_vpd_size+0x34/0xa8
scsi_get_vpd_buf+0x28/0xf4
scsi_attach_vpd+0x44/0x170
scsi_rescan_device+0x30/0x98
ata_scsi_dev_rescan+0xc8/0xfc
process_one_work+0x2e0/0x474
worker_thread+0x1cc/0x270
kthread+0xd8/0xe8
ret_from_fork+0x10/0x20This doesn't look like it's blocked in the R-Car SATA driver, but on some I/O scheduling event in the block core?
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support
Quoted 28 lines · Geert Uytterhoeven
With more debug options enabled, it prints:
INFO: task kworker/0:7:283 blocked for more than 120 seconds. Not tainted 5.19.0-rc7-salvator-x-00794-g6780eb02b605 #1287 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:kworker/0:7 state:D stack: 0 pid: 283 ppid: 2 flags:0x00000008 Workqueue: events ata_scsi_dev_rescan Call trace: __switch_to+0xbc/0x124 __schedule+0x540/0x71c schedule+0x58/0xa0 io_schedule+0x18/0x34 blk_mq_get_tag+0x138/0x244 __blk_mq_alloc_requests+0x130/0x2f0 blk_mq_alloc_request+0x74/0xa8 scsi_alloc_request+0x10/0x30 __scsi_execute+0x5c/0x18c scsi_vpd_inquiry+0x7c/0xdc scsi_get_vpd_size+0x34/0xa8 scsi_get_vpd_buf+0x28/0xf4 scsi_attach_vpd+0x44/0x170 scsi_rescan_device+0x30/0x98 ata_scsi_dev_rescan+0xc8/0xfc process_one_work+0x2e0/0x474 worker_thread+0x1cc/0x270 kthread+0xd8/0xe8 ret_from_fork+0x10/0x20
Quoted 2 lines
This doesn't look like it's blocked in the R-Car SATA driver, but on some I/O scheduling event in the block core?
I'm not familiar with the SATA code but from a quick look it seems like the above code is only triggered from inside the ATA error handler (ata_do_eh() -> ata_eh_recover() -> ata_eh_revalidate_and_attach() -> schedule_work(&(ap->scsi_rescan_task) -> ata_scsi_dev_rescan()). It doesn't seem normal to me that the ATA error handler gets invoked during a resume. How about testing the following two code changes?
* In sd_start_stop_device(), change "return sd_submit_start(sdkp, cmd, sizeof(cmd))" into "sd_submit_start(sdkp, cmd, sizeof(cmd))" and below that call add "flush_work(&sdkp->start_done_work)". This makes sd_start_stop_device() again synchronous. This will learn us whether the behavior change is caused by submitting the START command from another context or by not waiting until the START command has finished.
* Back out the above change, change "return sd_submit_start(sdkp, cmd, sizeof(cmd))" again into "sd_submit_start(sdkp, cmd, sizeof(cmd))" and below that statement add a call to scsi_run_queue(sdkp->device->request_queue). If this change helps it means that the scsi_run_queue() call is necessary to prevent reordering of the START command with other SCSI commands.
Thanks,
Bart.
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support
Hi Bart,
Quoted 32 lines · Bart Van Assche
Quoted 28 lines · Geert Uytterhoeven
With more debug options enabled, it prints:
INFO: task kworker/0:7:283 blocked for more than 120 seconds. Not tainted 5.19.0-rc7-salvator-x-00794-g6780eb02b605 #1287 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:kworker/0:7 state:D stack: 0 pid: 283 ppid: 2 flags:0x00000008 Workqueue: events ata_scsi_dev_rescan Call trace: __switch_to+0xbc/0x124 __schedule+0x540/0x71c schedule+0x58/0xa0 io_schedule+0x18/0x34 blk_mq_get_tag+0x138/0x244 __blk_mq_alloc_requests+0x130/0x2f0 blk_mq_alloc_request+0x74/0xa8 scsi_alloc_request+0x10/0x30 __scsi_execute+0x5c/0x18c scsi_vpd_inquiry+0x7c/0xdc scsi_get_vpd_size+0x34/0xa8 scsi_get_vpd_buf+0x28/0xf4 scsi_attach_vpd+0x44/0x170 scsi_rescan_device+0x30/0x98 ata_scsi_dev_rescan+0xc8/0xfc process_one_work+0x2e0/0x474 worker_thread+0x1cc/0x270 kthread+0xd8/0xe8 ret_from_fork+0x10/0x20Quoted 2 lines
This doesn't look like it's blocked in the R-Car SATA driver, but on some I/O scheduling event in the block core?
Quoted 6 lines
I'm not familiar with the SATA code but from a quick look it seems like the above code is only triggered from inside the ATA error handler (
ata_do_eh()->ata_eh_recover()->ata_eh_revalidate_and_attach()->schedule_work(&(ap->scsi_rescan_task) -> ata_scsi_dev_rescan()). It doesn't seem normal to me that the ATA error handler gets invoked during a resume. How about testing the following two code changes?
Thanks for your suggestions!
Quoted 6 lines
* In
sd_start_stop_device(), change "returnsd_submit_start(sdkp, cmd, sizeof(cmd))" into "sd_submit_start(sdkp, cmd, sizeof(cmd))" and below that call add "flush_work(&sdkp->start_done_work)". This makessd_start_stop_device()again synchronous. This will learn us whether the behavior change is caused by submitting the START command from another context or by not waiting until the START command has finished.
Unfortunately this doesn't have any impact.
Quoted 4 lines
* Back out the above change, change "return
sd_submit_start(sdkp, cmd, sizeof(cmd))" again into "sd_submit_start(sdkp, cmd, sizeof(cmd))" and below that statement add a call toscsi_run_queue(sdkp->device->request_queue). If this change helps it
(that's the static scsi_run_queue() in drivers/scsi/scsi_lib.c?)
Quoted 2 lines
means that the
scsi_run_queue()call is necessary to prevent reordering of the START command with other SCSI commands.
Unfortunately this doesn't have any impact either.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support
Quoted 24 lines · Geert Uytterhoeven
Quoted 6 lines · Bart Van Assche
I'm not familiar with the SATA code but from a quick look it seems like the above code is only triggered from inside the ATA error handler (
ata_do_eh()->ata_eh_recover()->ata_eh_revalidate_and_attach()->schedule_work(&(ap->scsi_rescan_task) -> ata_scsi_dev_rescan()). It doesn't seem normal to me that the ATA error handler gets invoked during a resume. How about testing the following two code changes?Thanks for your suggestions!
Quoted 6 lines
* In
sd_start_stop_device(), change "returnsd_submit_start(sdkp, cmd, sizeof(cmd))" into "sd_submit_start(sdkp, cmd, sizeof(cmd))" and below that call add "flush_work(&sdkp->start_done_work)". This makessd_start_stop_device()again synchronous. This will learn us whether the behavior change is caused by submitting the START command from another context or by not waiting until the START command has finished.Unfortunately this doesn't have any impact.
Quoted 4 lines
* Back out the above change, change "return
sd_submit_start(sdkp, cmd, sizeof(cmd))" again into "sd_submit_start(sdkp, cmd, sizeof(cmd))" and below that statement add a call toscsi_run_queue(sdkp->device->request_queue). If this change helps it
Quoted 6 lines
(that's the static
scsi_run_queue()indrivers/scsi/scsi_lib.c?)Quoted 2 lines
means that the
scsi_run_queue()call is necessary to prevent reordering of the START command with other SCSI commands.Unfortunately this doesn't have any impact either.
That's surprising. Is there anything unusual about the test setup that I should know, e.g. very small number of CPU cores or a very small queue depth of the SATA device? How about adding pr_info() statements at the start and end of the following functions and also before the return statements in these functions to determine where execution of the START command hangs?
* sd_start_done().
* sd_start_done_work().
Thanks,
Bart.
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support
Hoi Bart,
Quoted 32 lines · Bart Van Assche
Quoted 24 lines · Geert Uytterhoeven
Quoted 6 lines · Bart Van Assche
I'm not familiar with the SATA code but from a quick look it seems like the above code is only triggered from inside the ATA error handler (
ata_do_eh()->ata_eh_recover()->ata_eh_revalidate_and_attach()->schedule_work(&(ap->scsi_rescan_task) -> ata_scsi_dev_rescan()). It doesn't seem normal to me that the ATA error handler gets invoked during a resume. How about testing the following two code changes?Thanks for your suggestions!
Quoted 6 lines
* In
sd_start_stop_device(), change "returnsd_submit_start(sdkp, cmd, sizeof(cmd))" into "sd_submit_start(sdkp, cmd, sizeof(cmd))" and below that call add "flush_work(&sdkp->start_done_work)". This makessd_start_stop_device()again synchronous. This will learn us whether the behavior change is caused by submitting the START command from another context or by not waiting until the START command has finished.Unfortunately this doesn't have any impact.
Quoted 4 lines
* Back out the above change, change "return
sd_submit_start(sdkp, cmd, sizeof(cmd))" again into "sd_submit_start(sdkp, cmd, sizeof(cmd))" and below that statement add a call toscsi_run_queue(sdkp->device->request_queue). If this change helps itQuoted 6 lines
(that's the static
scsi_run_queue()indrivers/scsi/scsi_lib.c?)Quoted 2 lines
means that the
scsi_run_queue()call is necessary to prevent reordering of the START command with other SCSI commands.Unfortunately this doesn't have any impact either.
Quoted 8 lines
That's surprising. Is there anything unusual about the test setup that I should know, e.g. very small number of CPU cores or a very small queue depth of the SATA device? How about adding
pr_info()statements at the start and end of the following functions and also before the return statements in these functions to determine where execution of the START command hangs?
*sd_start_done().
*sd_start_done_work().
None of these functions seem to be called at all?
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support
Quoted 11 lines · Geert Uytterhoeven
Quoted 8 lines · Bart Van Assche
That's surprising. Is there anything unusual about the test setup that I should know, e.g. very small number of CPU cores or a very small queue depth of the SATA device? How about adding
pr_info()statements at the start and end of the following functions and also before the return statements in these functions to determine where execution of the START command hangs?
*sd_start_done().
*sd_start_done_work().None of these functions seem to be called at all?
That's weird. This means that either sd_submit_start() hangs or that the execution of the START command never finishes. The latter is unlikely since the SCSI error handler is assumed to abort commands that hang. It would also be weird if sd_submit_start() would hang before the START command is submitted since the code flow for submitting the START command is very similar to the code flow for submitting the START command without patch "scsi: sd: Rework asynchronous resume support" (calling scsi_execute()).
What is also weird is that there are at least two SATA setups on which this code works fine, including my Qemu setup.
Although it is possible to enable tracing at boot time, adding the following parameters to the kernel command line would generate too much logging data:
tp_printk trace_event=block_rq_complete,block_rq_error,block_rq_insert,block_rq_issue,block_rq_merge,block_rq_remap,block_rq_requeue,scsi_dispatch_cmd_done,scsi_dispatch_cmd_start,scsi_eh_wakeup,scsi_dispatch_cmd_error,scsi_dispatch_cmd_timeout scsi_mod.scsi_logging_level=32256
I'm not sure what the best way is to proceed since I cannot reproduce this issue.
Bart.
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support
Hoi Bart,
Quoted 13 lines · Bart Van Assche
Quoted 11 lines · Geert Uytterhoeven
Quoted 8 lines · Bart Van Assche
That's surprising. Is there anything unusual about the test setup that I should know, e.g. very small number of CPU cores or a very small queue depth of the SATA device? How about adding
pr_info()statements at the start and end of the following functions and also before the return statements in these functions to determine where execution of the START command hangs?
*sd_start_done().
*sd_start_done_work().None of these functions seem to be called at all?
Quoted 8 lines
That's weird. This means that either
sd_submit_start()hangs or that the execution of the START command never finishes. The latter is unlikely since the SCSI error handler is assumed to abort commands that hang. It would also be weird ifsd_submit_start()would hang before the START command is submitted since the code flow for submitting the START command is very similar to the code flow for submitting the START command without patch "scsi: sd: Rework asynchronous resume support" (callingscsi_execute()).
I think you misunderstood: none of these functions seem to be called, even when reading from hard drive works fine.
Quoted 7 lines
What is also weird is that there are at least two SATA setups on which this code works fine, including my Qemu setup.
Although it is possible to enable tracing at boot time, adding the following parameters to the kernel command line would generate too much logging data:
Quoted 6 lines
tp_printktrace_event=block_rq_complete,block_rq_error,block_rq_insert,block_rq_issue,block_rq_merge,block_rq_remap,block_rq_requeue,scsi_dispatch_cmd_done,scsi_dispatch_cmd_start,scsi_eh_wakeup,scsi_dispatch_cmd_error,scsi_dispatch_cmd_timeout scsi_mod.scsi_logging_level=32256I'm not sure what the best way is to proceed since I cannot reproduce this issue.
During s2idle, the following trace data is generated:
kworker/u16:9-325 [000] ...2. 230.478731: block_rq_issue: 8,0
N 0 () 0 + 0 [kworker/u16:9]
kworker/u16:9-325 [000] ...2. 230.478745:scsi_dispatch_cmd_start: host_no=0 channel=0 id=0 lun=0 data_sgl=0 prot_sgl=0 prot_op=SCSI_PROT_NORMAL driver_tag=0 scheduler_tag=0 cmnd=(SYNCHRONIZE_CACHE - raw=35 00 00 00 00 00 00 00 00 00)
<idle>-0 [007] d.h3. 230.478832:
scsi_dispatch_cmd_done: host_no=0 channel=0 id=0 lun=0 data_sgl=0 prot_sgl=0 prot_op=SCSI_PROT_NORMAL driver_tag=0 scheduler_tag=0 cmnd=(SYNCHRONIZE_CACHE - raw=35 00 00 00 00 00 00 00 00 00) result=(driver=DRIVER_OK host=DID_OK message=COMMAND_COMPLETE status=SAM_STAT_GOOD)
<idle>-0 [000] ..s2. 230.478851: block_rq_complete:
8,0 N () 18446744073709551615 + 0 [0]
kworker/u16:9-325 [000] ...2. 230.483134: block_rq_issue: 8,0
N 0 () 0 + 0 [kworker/u16:9]
kworker/u16:9-325 [000] ...2. 230.483136:scsi_dispatch_cmd_start: host_no=0 channel=0 id=0 lun=0 data_sgl=0 prot_sgl=0 prot_op=SCSI_PROT_NORMAL driver_tag=0 scheduler_tag=1 cmnd=(START_STOP - raw=1b 00 00 00 00 00)
<idle>-0 [007] d.h3. 230.624530:
scsi_dispatch_cmd_done: host_no=0 channel=0 id=0 lun=0 data_sgl=0 prot_sgl=0 prot_op=SCSI_PROT_NORMAL driver_tag=0 scheduler_tag=1 cmnd=(START_STOP - raw=1b 00 00 00 00 00) result=(driver=DRIVER_OK host=DID_OK message=COMMAND_COMPLETE status=SAM_STAT_GOOD)
<idle>-0 [000] d.s4. 230.624634: scsi_eh_wakeup: host_no=0
<idle>-0 [000] ..s2. 230.624642: block_rq_complete:8,0 N () 18446744073709551615 + 0 [0]
kworker/u16:14-1027 [007] d..3. 231.393642: scsi_eh_wakeup: host_no=0
When reading from hard drive after s2idle, no more trace data is generated.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support
Quoted 27 lines · Geert Uytterhoeven
During s2idle, the following trace data is generated:
kworker/u16:9-325 [000] ...2. 230.478731: block_rq_issue: 8,0 N 0 () 0 + 0 [kworker/u16:9] kworker/u16:9-325 [000] ...2. 230.478745: scsi_dispatch_cmd_start: host_no=0 channel=0 id=0 lun=0 data_sgl=0 prot_sgl=0 prot_op=SCSI_PROT_NORMAL driver_tag=0 scheduler_tag=0 cmnd=(SYNCHRONIZE_CACHE - raw=35 00 00 00 00 00 00 00 00 00) <idle>-0 [007] d.h3. 230.478832:
scsi_dispatch_cmd_done: host_no=0 channel=0 id=0 lun=0 data_sgl=0 prot_sgl=0 prot_op=SCSI_PROT_NORMAL driver_tag=0 scheduler_tag=0 cmnd=(SYNCHRONIZE_CACHE - raw=35 00 00 00 00 00 00 00 00 00) result=(driver=DRIVER_OK host=DID_OK message=COMMAND_COMPLETE status=SAM_STAT_GOOD)<idle>-0 [000] ..s2. 230.478851: block_rq_complete: 8,0 N () 18446744073709551615 + 0 [0] kworker/u16:9-325 [000] ...2. 230.483134: block_rq_issue: 8,0 N 0 () 0 + 0 [kworker/u16:9] kworker/u16:9-325 [000] ...2. 230.483136: scsi_dispatch_cmd_start: host_no=0 channel=0 id=0 lun=0 data_sgl=0 prot_sgl=0 prot_op=SCSI_PROT_NORMAL driver_tag=0 scheduler_tag=1 cmnd=(START_STOP - raw=1b 00 00 00 00 00) <idle>-0 [007] d.h3. 230.624530:
scsi_dispatch_cmd_done: host_no=0 channel=0 id=0 lun=0 data_sgl=0 prot_sgl=0 prot_op=SCSI_PROT_NORMAL driver_tag=0 scheduler_tag=1 cmnd=(START_STOP - raw=1b 00 00 00 00 00) result=(driver=DRIVER_OK host=DID_OK message=COMMAND_COMPLETE status=SAM_STAT_GOOD)
Quoted 7 lines
<idle>-0 [000] d.s4. 230.624634: scsi_eh_wakeup: host_no=0 <idle>-0 [000] ..s2. 230.624642: block_rq_complete:8,0 N () 18446744073709551615 + 0 [0]
kworker/u16:14-1027 [007] d..3. 231.393642:scsi_eh_wakeup: host_no=0When reading from hard drive after s2idle, no more trace data is generated.
I think the above commands come from the suspend sequence. '1b 00 00 00 00 00' stops a block device. The lowest bit in byte 4 needs to be set to start a block device.
Something that is not yet clear is whether or not sd_submit_start() hangs during the resume process. How about verifying whether or not sd_submit_start() hangs by either issuing SysRq-t or by adding pr_info() statements in that function?
Thanks,
Bart.
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support
Hi Bart,
CC linux-ide
Quoted 36 lines · Bart Van Assche
Quoted 27 lines · Geert Uytterhoeven
During s2idle, the following trace data is generated:
kworker/u16:9-325 [000] ...2. 230.478731: block_rq_issue: 8,0 N 0 () 0 + 0 [kworker/u16:9] kworker/u16:9-325 [000] ...2. 230.478745: scsi_dispatch_cmd_start: host_no=0 channel=0 id=0 lun=0 data_sgl=0 prot_sgl=0 prot_op=SCSI_PROT_NORMAL driver_tag=0 scheduler_tag=0 cmnd=(SYNCHRONIZE_CACHE - raw=35 00 00 00 00 00 00 00 00 00) <idle>-0 [007] d.h3. 230.478832:
scsi_dispatch_cmd_done: host_no=0 channel=0 id=0 lun=0 data_sgl=0 prot_sgl=0 prot_op=SCSI_PROT_NORMAL driver_tag=0 scheduler_tag=0 cmnd=(SYNCHRONIZE_CACHE - raw=35 00 00 00 00 00 00 00 00 00) result=(driver=DRIVER_OK host=DID_OK message=COMMAND_COMPLETE status=SAM_STAT_GOOD)<idle>-0 [000] ..s2. 230.478851: block_rq_complete: 8,0 N () 18446744073709551615 + 0 [0] kworker/u16:9-325 [000] ...2. 230.483134: block_rq_issue: 8,0 N 0 () 0 + 0 [kworker/u16:9] kworker/u16:9-325 [000] ...2. 230.483136: scsi_dispatch_cmd_start: host_no=0 channel=0 id=0 lun=0 data_sgl=0 prot_sgl=0 prot_op=SCSI_PROT_NORMAL driver_tag=0 scheduler_tag=1 cmnd=(START_STOP - raw=1b 00 00 00 00 00) <idle>-0 [007] d.h3. 230.624530:
scsi_dispatch_cmd_done: host_no=0 channel=0 id=0 lun=0 data_sgl=0 prot_sgl=0 prot_op=SCSI_PROT_NORMAL driver_tag=0 scheduler_tag=1 cmnd=(START_STOP - raw=1b 00 00 00 00 00) result=(driver=DRIVER_OK host=DID_OK message=COMMAND_COMPLETE status=SAM_STAT_GOOD)Quoted 7 lines
<idle>-0 [000] d.s4. 230.624634: scsi_eh_wakeup: host_no=0 <idle>-0 [000] ..s2. 230.624642: block_rq_complete:8,0 N () 18446744073709551615 + 0 [0]
kworker/u16:14-1027 [007] d..3. 231.393642:scsi_eh_wakeup: host_no=0When reading from hard drive after s2idle, no more trace data is generated.
Quoted 8 lines
I think the above commands come from the suspend sequence. '1b 00 00 00 00 00' stops a block device. The lowest bit in byte 4 needs to be set to start a block device.
Something that is not yet clear is whether or not
sd_submit_start()hangs during the resume process. How about verifying whether or notsd_submit_start()hangs by either issuing SysRq-t or by addingpr_info()statements in that function?
sd_submit_start() is called once during suspend, and once during resume. It does not hang.
Reading from /dev/sda hangs after resume (not in sd_submit_start(), which is never called for reading).
Two tasks are blocked in blk_mq_get_tag() calling io_schedule():
task:kworker/7:1 state:D stack: 0 pid: 122 ppid: 2 flags:0x00000008
Workqueue: events ata_scsi_dev_rescan Call trace: __switch_to+0xbc/0x124 __schedule+0x540/0x71c schedule+0x58/0xa0 io_schedule+0x18/0x34 blk_mq_get_tag+0x138/0x244 __blk_mq_alloc_requests+0x130/0x2f0 blk_mq_alloc_request+0x74/0xa8 scsi_alloc_request+0x10/0x30 __scsi_execute+0x5c/0x18c scsi_vpd_inquiry+0x7c/0xdc scsi_get_vpd_size+0x34/0xa8 scsi_get_vpd_buf+0x28/0xf4 scsi_attach_vpd+0x44/0x170 scsi_rescan_device+0x30/0x98 ata_scsi_dev_rescan+0xc8/0xfc process_one_work+0x2e0/0x474 worker_thread+0x1cc/0x270 kthread+0xd8/0xe8 ret_from_fork+0x10/0x20
task:hd state:D stack: 0 pid: 1163 ppid: 1076 flags:0x00000000
Call trace: __switch_to+0xbc/0x124 __schedule+0x540/0x71c schedule+0x58/0xa0 io_schedule+0x18/0x34 blk_mq_get_tag+0x138/0x244 __blk_mq_alloc_requests+0x130/0x2f0 blk_mq_submit_bio+0x44c/0x5b4 __submit_bio+0x24/0x5c submit_bio_noacct_nocheck+0x8c/0x178 submit_bio_noacct+0x380/0x3b0 submit_bio+0x34/0x3c mpage_bio_submit+0x28/0x38 mpage_readahead+0xa8/0x178 blkdev_readahead+0x14/0x1c read_pages+0x4c/0x158 page_cache_ra_unbounded+0xd8/0x174 do_page_cache_ra+0x40/0x4c page_cache_ra_order+0x14/0x1c ondemand_readahead+0x124/0x2fc page_cache_sync_ra+0x50/0x54 filemap_read+0x130/0x6e8 blkdev_read_iter+0xf0/0x164 new_sync_read+0x74/0xc0 vfs_read+0xbc/0xd8 ksys_read+0x6c/0xd4 __arm64_sys_read+0x14/0x1c invoke_syscall+0x70/0xf4 el0_svc_common.constprop.0+0xbc/0xf0 do_el0_svc+0x18/0x20 el0_svc+0x30/0x84 el0t_64_sync_handler+0x90/0xf8 el0t_64_sync+0x14c/0x150
I hope this helps.
Thanks!
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support
Quoted 58 lines · Geert Uytterhoeven
sd_submit_start()is called once during suspend, and once during resume. It does not hang.Reading from
/dev/sdahangs after resume (not insd_submit_start(), which is never called for reading).Two tasks are blocked in
blk_mq_get_tag()callingio_schedule():task:kworker/7:1 state:D stack: 0 pid: 122 ppid: 2 flags:0x00000008
Workqueue: events ata_scsi_dev_rescan Call trace: __switch_to+0xbc/0x124 __schedule+0x540/0x71c schedule+0x58/0xa0 io_schedule+0x18/0x34 blk_mq_get_tag+0x138/0x244 __blk_mq_alloc_requests+0x130/0x2f0 blk_mq_alloc_request+0x74/0xa8 scsi_alloc_request+0x10/0x30 __scsi_execute+0x5c/0x18c scsi_vpd_inquiry+0x7c/0xdc scsi_get_vpd_size+0x34/0xa8 scsi_get_vpd_buf+0x28/0xf4 scsi_attach_vpd+0x44/0x170 scsi_rescan_device+0x30/0x98 ata_scsi_dev_rescan+0xc8/0xfc process_one_work+0x2e0/0x474 worker_thread+0x1cc/0x270 kthread+0xd8/0xe8 ret_from_fork+0x10/0x20task:hd state:D stack: 0 pid: 1163 ppid: 1076 flags:0x00000000
Call trace: __switch_to+0xbc/0x124 __schedule+0x540/0x71c schedule+0x58/0xa0 io_schedule+0x18/0x34 blk_mq_get_tag+0x138/0x244 __blk_mq_alloc_requests+0x130/0x2f0 blk_mq_submit_bio+0x44c/0x5b4 __submit_bio+0x24/0x5c submit_bio_noacct_nocheck+0x8c/0x178 submit_bio_noacct+0x380/0x3b0 submit_bio+0x34/0x3c mpage_bio_submit+0x28/0x38 mpage_readahead+0xa8/0x178 blkdev_readahead+0x14/0x1c read_pages+0x4c/0x158 page_cache_ra_unbounded+0xd8/0x174 do_page_cache_ra+0x40/0x4c page_cache_ra_order+0x14/0x1c ondemand_readahead+0x124/0x2fc page_cache_sync_ra+0x50/0x54 filemap_read+0x130/0x6e8 blkdev_read_iter+0xf0/0x164 new_sync_read+0x74/0xc0 vfs_read+0xbc/0xd8
Quoted 8 lines
ksys_read+0x6c/0xd4 __arm64_sys_read+0x14/0x1c invoke_syscall+0x70/0xf4 el0_svc_common.constprop.0+0xbc/0xf0 do_el0_svc+0x18/0x20 el0_svc+0x30/0x84 el0t_64_sync_handler+0x90/0xf8 el0t_64_sync+0x14c/0x150
Hi Geert,
All that can be concluded from the above is that blk_mq_get_tag() is waiting for other I/O request(s) to finish. One or more other requests are in progress and either scsi_done() has not been called for these requests or the error handler got stuck. Since the issue reported above is not observed with other ATA interfaces, this may be related to the ATA interface driver used in your test setup.
Bart.
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support
Hoi Bart,
Quoted 6 lines · Bart Van Assche
Quoted 5 lines · Geert Uytterhoeven
sd_submit_start()is called once during suspend, and once during resume. It does not hang.Reading from
/dev/sdahangs after resume (not insd_submit_start(), which is never called for reading).
FTR, this issue is now present in v6.0-rc1. Reverting commit 88f1669019bd62b3 ("scsi: sd: Rework asynchronous resume support") fixes it.
Quoted 61 lines
Quoted 52 lines
Two tasks are blocked in
blk_mq_get_tag()callingio_schedule():task:kworker/7:1 state:D stack: 0 pid: 122 ppid: 2 flags:0x00000008
Workqueue: events ata_scsi_dev_rescan Call trace: __switch_to+0xbc/0x124 __schedule+0x540/0x71c schedule+0x58/0xa0 io_schedule+0x18/0x34 blk_mq_get_tag+0x138/0x244 __blk_mq_alloc_requests+0x130/0x2f0 blk_mq_alloc_request+0x74/0xa8 scsi_alloc_request+0x10/0x30 __scsi_execute+0x5c/0x18c scsi_vpd_inquiry+0x7c/0xdc scsi_get_vpd_size+0x34/0xa8 scsi_get_vpd_buf+0x28/0xf4 scsi_attach_vpd+0x44/0x170 scsi_rescan_device+0x30/0x98 ata_scsi_dev_rescan+0xc8/0xfc process_one_work+0x2e0/0x474 worker_thread+0x1cc/0x270 kthread+0xd8/0xe8 ret_from_fork+0x10/0x20task:hd state:D stack: 0 pid: 1163 ppid: 1076 flags:0x00000000
Call trace: __switch_to+0xbc/0x124 __schedule+0x540/0x71c schedule+0x58/0xa0 io_schedule+0x18/0x34 blk_mq_get_tag+0x138/0x244 __blk_mq_alloc_requests+0x130/0x2f0 blk_mq_submit_bio+0x44c/0x5b4 __submit_bio+0x24/0x5c submit_bio_noacct_nocheck+0x8c/0x178 submit_bio_noacct+0x380/0x3b0 submit_bio+0x34/0x3c mpage_bio_submit+0x28/0x38 mpage_readahead+0xa8/0x178 blkdev_readahead+0x14/0x1c read_pages+0x4c/0x158 page_cache_ra_unbounded+0xd8/0x174 do_page_cache_ra+0x40/0x4c page_cache_ra_order+0x14/0x1c ondemand_readahead+0x124/0x2fc page_cache_sync_ra+0x50/0x54 filemap_read+0x130/0x6e8 blkdev_read_iter+0xf0/0x164 new_sync_read+0x74/0xc0 vfs_read+0xbc/0xd8Quoted 8 lines
ksys_read+0x6c/0xd4 __arm64_sys_read+0x14/0x1c invoke_syscall+0x70/0xf4 el0_svc_common.constprop.0+0xbc/0xf0 do_el0_svc+0x18/0x20 el0_svc+0x30/0x84 el0t_64_sync_handler+0x90/0xf8 el0t_64_sync+0x14c/0x150
Quoted 6 lines
All that can be concluded from the above is that
blk_mq_get_tag()is waiting for other I/O request(s) to finish. One or more other requests are in progress and eitherscsi_done()has not been called for these requests or the error handler got stuck. Since the issue reported above is not observed with other ATA interfaces, this may be related to the ATA interface driver used in your test setup.
I have added debug prints to all ata_port_operations in sata_rcar_port_ops. After s2idle, running "hd /dev/sda | head -70" hangs before any of these functions are called.
Showing all locks held in the system:
1 lock held by rcu_tasks_kthre/10:
#0: ffff800009575c38 (rcu_tasks.tasks_gp_mutex){+.+.}-{3:3}, at:
rcu_tasks_one_gp+0x34/0x4c8
4 locks held by kworker/0:10/104:
#0: ffff0004c0008738 ((wq_completion)events){+.+.}-{0:0}, at:
process_one_work+0x1f4/0x6a0
#1: ffff80000a90bde0
((work_completion)(&ap->scsi_rescan_task)){+.+.}-{0:0}, at:
process_one_work+0x1f4/0x6a0
#2: ffff0004c2b6bf60 (&ap->scsi_scan_mutex){+.+.}-{3:3}, at:
ata_scsi_dev_rescan+0x28/0x118
#3: ffff0004c2902368 (&dev->mutex){....}-{3:3}, at:
scsi_rescan_device+0x28/0x78
1 lock held by in:imklog/636:
#0: ffff0004c5ee86e8 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0x54/0x68
1 lock held by hd/1013:
#0: ffff0004c06388b8 (mapping.invalidate_lock#2){.+.+}-{3:3}, at:
page_cache_ra_unbounded+0x64/0x1a8I've just tried with a USB storage device on the same platform, and it can be read fine after s2idle. So it looks like the issue is related to SATA.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support
Quoted 11 lines · Geert Uytterhoeven
Showing all locks held in the system:
1 lock held by rcu_tasks_kthre/10:
#0: ffff800009575c38 (rcu_tasks.tasks_gp_mutex){+.+.}-{3:3}, at:
rcu_tasks_one_gp+0x34/0x4c8
4 locks held by kworker/0:10/104:
#0: ffff0004c0008738 ((wq_completion)events){+.+.}-{0:0}, at:
process_one_work+0x1f4/0x6a0
#1: ffff80000a90bde0
((work_completion)(&ap->scsi_rescan_task)){+.+.}-{0:0}, at:
process_one_work+0x1f4/0x6a0
#2: ffff0004c2b6bf60 (&ap->scsi_scan_mutex){+.+.}-{3:3}, at:Quoted 8 lines
ata_scsi_dev_rescan+0x28/0x118
#3: ffff0004c2902368 (&dev->mutex){....}-{3:3}, at:
scsi_rescan_device+0x28/0x78
1 lock held by in:imklog/636:
#0: ffff0004c5ee86e8 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0x54/0x68
1 lock held by hd/1013:
#0: ffff0004c06388b8 (mapping.invalidate_lock#2){.+.+}-{3:3}, at:
page_cache_ra_unbounded+0x64/0x1a8Thank you for having shared this information. I will take a closer look and see what I can derive from the above information.
Quoted 3 lines
I've just tried with a USB storage device on the same platform, and it can be read fine after s2idle. So it looks like the issue is related to SATA.
Unfortunately the above does not learn us anything new. The code modified by commit 88f1669019bd ("scsi: sd: Rework asynchronous resume support") is only called if sdev->manage_start_stop != 1. Only the SATA code, the Firewire code and the manage_start_stop sysfs attribute store method set that member variable:
$ git grep -nH 'manage_start_stop = ' drivers/ata/libata-scsi.c:1083: sdev->manage_start_stop = 1; drivers/firewire/sbp2.c:1521: sdev->manage_start_stop = 1; drivers/scsi/sd.c:240: sdp->manage_start_stop = v;
Would it be possible to share the output of the command below? That should reveal which ATA driver is active on the test setup.
find /sys -name proc_name | xargs grep -aH .
Thanks,
Bart.
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support
Hoi Bart,
Quoted 39 lines · Bart Van Assche
Quoted 11 lines · Geert Uytterhoeven
Showing all locks held in the system: 1 lock held by rcu_tasks_kthre/10: #0: ffff800009575c38 (rcu_tasks.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x34/0x4c8 4 locks held by kworker/0:10/104: #0: ffff0004c0008738 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0x1f4/0x6a0 #1: ffff80000a90bde0 ((work_completion)(&ap->scsi_rescan_task)){+.+.}-{0:0}, at: process_one_work+0x1f4/0x6a0 #2: ffff0004c2b6bf60 (&ap->scsi_scan_mutex){+.+.}-{3:3}, at:Quoted 8 lines
ata_scsi_dev_rescan+0x28/0x118 #3: ffff0004c2902368 (&dev->mutex){....}-{3:3}, at: scsi_rescan_device+0x28/0x78 1 lock held by in:imklog/636: #0: ffff0004c5ee86e8 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0x54/0x68 1 lock held by hd/1013: #0: ffff0004c06388b8 (mapping.invalidate_lock#2){.+.+}-{3:3}, at: page_cache_ra_unbounded+0x64/0x1a8Thank you for having shared this information. I will take a closer look and see what I can derive from the above information.
Quoted 3 lines
I've just tried with a USB storage device on the same platform, and it can be read fine after s2idle. So it looks like the issue is related to SATA.
Unfortunately the above does not learn us anything new. The code modified by commit
88f1669019bd("scsi: sd: Rework asynchronous resume support") is only called ifsdev->manage_start_stop!= 1. Only the SATA code, the Firewire code and themanage_start_stopsysfs attribute store method set that member variable:$ git grep -nH 'manage_start_stop = ' drivers/ata/libata-scsi.c:1083: sdev->manage_start_stop = 1; drivers/firewire/sbp2.c:1521: sdev->manage_start_stop = 1; drivers/scsi/sd.c:240: sdp->manage_start_stop = v;
Quoted 4 lines
Would it be possible to share the output of the command below? That should reveal which ATA driver is active on the test setup.
find /sys -name
proc_name| xargs grep -aH .
/sys/devices/platform/soc/ee300000.sata/ata1/host0/scsi_host/host0/proc_name:sata_rcar
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support
Quoted 7 lines · Geert Uytterhoeven
Quoted 4 lines · Bart Van Assche
Would it be possible to share the output of the command below? That should reveal which ATA driver is active on the test setup.
find /sys -name
proc_name| xargs grep -aH ./sys/devices/platform/soc/ee300000.sata/ata1/host0/scsi_host/host0/proc_name:sata_rcar
Thanks Geert for the help. Although I already posted a revert, I'm still trying to root-cause this issue. Do you perhaps know whether sata_rcar controllers support NCQ and if so, what queue depth these controllers support? I think that information is available in sysfs. Here is an example for a VM:
# (cd /sys/class/scsi_device && for a in */device/*/*/ncq_prio_enable; do p=${a%/ncq_prio_enable}; grep -qi ata $p/inquiry || continue; grep -aH . $p/{queue_depth,ncq*}; done) 2:0:0:0/device/driver/2:0:0:0/queue_depth:32
2:0:0:0/device/driver/2:0:0:0/ncq_prio_enable:0
2:0:0:0/device/driver/2:0:0:0/ncq_prio_supported:0
2:0:0:0/device/generic/device/queue_depth:32
2:0:0:0/device/generic/device/ncq_prio_enable:0
2:0:0:0/device/generic/device/ncq_prio_supported:0
6:0:0:1/device/driver/2:0:0:0/queue_depth:32
6:0:0:1/device/driver/2:0:0:0/ncq_prio_enable:0
6:0:0:1/device/driver/2:0:0:0/ncq_prio_supported:0
Thanks,
Bart.
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support
Hello!
Quoted 10 lines · Bart Van Assche
Quoted 7 lines · Bart Van Assche
Quoted 4 lines
Would it be possible to share the output of the command below? That should reveal which ATA driver is active on the test setup.
find /sys -name
proc_name| xargs grep -aH ./sys/devices/platform/soc/ee300000.sata/ata1/host0/scsi_host/host0/proc_name:sata_rcar
Thanks Geert for the help. Although I already posted a revert, I'm still trying to root-cause this issue. Do you perhaps know whether
sata_rcarcontrollers support NCQ
They don't. :-)
Quoted 13 lines
and if so, what queue depth these controllers support? I think that information is available in sysfs. Here is an example for a VM:
# (cd
/sys/class/scsi_device&& for a in */device/*/*/ncq_prio_enable; do p=${a%/ncq_prio_enable}; grep -qi ata $p/inquiry || continue; grep -aH . $p/{queue_depth,ncq*}; done) 2:0:0:0/device/driver/2:0:0:0/queue_depth:32
2:0:0:0/device/driver/2:0:0:0/ncq_prio_enable:0
2:0:0:0/device/driver/2:0:0:0/ncq_prio_supported:0
2:0:0:0/device/generic/device/queue_depth:32
2:0:0:0/device/generic/device/ncq_prio_enable:0
2:0:0:0/device/generic/device/ncq_prio_supported:0
6:0:0:1/device/driver/2:0:0:0/queue_depth:32
6:0:0:1/device/driver/2:0:0:0/ncq_prio_enable:0
6:0:0:1/device/driver/2:0:0:0/ncq_prio_supported:0
Quoted 3 lines
Thanks,
Bart.
MBR, Sergey
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support
In reply to Bart Van AsscheHi, I have a T460 hanging on resume from suspend to ram in 6.0-rc1 that I bisected to this commit.
Quoted 11 lines
Unfortunately the above does not learn us anything new. The code modified by commit
88f1669019bd("scsi: sd: Rework asynchronous resume support") is only called ifsdev->manage_start_stop!= 1. Only the SATA code, the Firewire code and themanage_start_stopsysfs attribute store method set that member variable:$ git grep -nH 'manage_start_stop = ' drivers/ata/libata-scsi.c:1083: sdev->manage_start_stop = 1; drivers/firewire/sbp2.c:1521: sdev->manage_start_stop = 1; drivers/scsi/sd.c:240: sdp->manage_start_stop = v;
Quoted 4 lines
Would it be possible to share the output of the command below? That should reveal which ATA driver is active on the test setup.
find /sys -name
proc_name| xargs grep -aH .
In my case it's /sys/devices/pci0000:00/0000:00:17.0/ata1/host0/scsi_host/host0/proc_name:ahci /sys/devices/pci0000:00/0000:00:17.0/ata2/host1/scsi_host/host1/proc_name:ahci
Some more details from dmesg
[ 0.849373] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 0.852849] ata2.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 0.854671] ata2.00: supports DRM functions and may not be fully accessible [ 0.856181] ata2.00: ATA-9: SAMSUNG MZ7LN512HMJP-000L7, MAV01L6Q, max UDMA/133 [ 0.858115] ata2.00: 1000215216 sectors, multi 1: LBA48 NCQ (depth 32), AA [ 0.861584] ata2.00: Features: Trust Dev-Sleep NCQ-sndrcv [ 0.863749] ata2.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 0.865481] ata2.00: supports DRM functions and may not be fully accessible [ 0.870043] ata2.00: configured for UDMA/133 [ 0.871871] scsi 1:0:0:0: Direct-Access ATA SAMSUNG MZ7LN512 1L6Q PQ: 0 ANSI: 5
Please Cc me on further questions/steps to try/patches to test.
#regzbot introduced: 88f1669019bd62b3
#regzbot monitor: https://lore.kernel.org/all/[email protected]/
Quoted 3 lines
Thanks,
Bart.
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support
Quoted 17 lines · Vlastimil Babka
In my case it's /sys/devices/pci0000:00/0000:00:17.0/ata1/host0/scsi_host/host0/proc_name:ahci /sys/devices/pci0000:00/0000:00:17.0/ata2/host1/scsi_host/host1/proc_name:ahci
Some more details from dmesg
[ 0.849373] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 0.852849] ata2.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 0.854671] ata2.00: supports DRM functions and may not be fully accessible [ 0.856181] ata2.00: ATA-9: SAMSUNG MZ7LN512HMJP-000L7, MAV01L6Q, max UDMA/133 [ 0.858115] ata2.00: 1000215216 sectors, multi 1: LBA48 NCQ (depth 32), AA [ 0.861584] ata2.00: Features: Trust Dev-Sleep NCQ-sndrcv [ 0.863749] ata2.00: ACPI cmd f5/00:00:00:00:00:a0(SECURITY FREEZE LOCK) filtered out [ 0.865481] ata2.00: supports DRM functions and may not be fully accessible [ 0.870043] ata2.00: configured for UDMA/133 [ 0.871871] scsi 1:0:0:0: Direct-Access ATA SAMSUNG MZ7LN512 1L6Q PQ: 0 ANSI: 5
Quoted 1 line
Please Cc me on further questions/steps to try/patches to test.
Hi Vlastimil,
Thank you for having provided the above information. The root cause of the hang is not yet clear to me. I was wondering whether the hang perhaps would be triggered by controllers that only support queue depth 1. However, in the above output I see "depth 32".
As already reported in this email thread a revert for commit 88f1669019bd62b3 has been posted on the linux-scsi mailing list. Additionally, Greg KH has been asked to drop that patch from the stable trees.
Thanks,
Bart.
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support #forregzbot
In reply to Vlastimil BabkaTWIMC: this mail is primarily send for documentation purposes and for regzbot, my Linux kernel regression tracking bot. These mails usually contain '#forregzbot' in the subject, to make them easy to spot and filter.
Quoted 11 lines · Vlastimil Babka
Hi, I have a T460 hanging on resume from suspend to ram in 6.0-rc1 that I bisected to this commit.
Quoted 5 lines
Unfortunately the above does not learn us anything new. The code modified by commit
88f1669019bd("scsi: sd: Rework asynchronous resume support") is only called ifsdev->manage_start_stop!= 1. Only the SATA code, the Firewire code and themanage_start_stopsysfs attribute store method set that member variable:[...]
#regzbot introduced:88f1669019bd62b3
#regzbot monitor: https://lore.kernel.org/all/[email protected]/
#regzbot fixed-by: 785538bfdd682c8e962341d585f9b88262a0475
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
P.S.: As the Linux kernel's regression tracker I deal with a lot of reports and sometimes miss something important when writing mails like this. If that's the case here, don't hesitate to tell me in a public reply, it's in everyone's interest to set the public record straight.
Re: [PATCH v2 2/2] scsi: sd: Rework asynchronous resume support
In reply to Bart Van AsscheQuoted 29 lines · Bart Van Assche
Quoted 28 lines · Geert Uytterhoeven
With more debug options enabled, it prints:
INFO: task kworker/0:7:283 blocked for more than 120 seconds. Not tainted 5.19.0-rc7-salvator-x-00794-g6780eb02b605 #1287 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:kworker/0:7 state:D stack: 0 pid: 283 ppid: 2 flags:0x00000008 Workqueue: events ata_scsi_dev_rescan Call trace: __switch_to+0xbc/0x124 __schedule+0x540/0x71c schedule+0x58/0xa0 io_schedule+0x18/0x34 blk_mq_get_tag+0x138/0x244 __blk_mq_alloc_requests+0x130/0x2f0 blk_mq_alloc_request+0x74/0xa8 scsi_alloc_request+0x10/0x30 __scsi_execute+0x5c/0x18c scsi_vpd_inquiry+0x7c/0xdc scsi_get_vpd_size+0x34/0xa8 scsi_get_vpd_buf+0x28/0xf4 scsi_attach_vpd+0x44/0x170 scsi_rescan_device+0x30/0x98 ata_scsi_dev_rescan+0xc8/0xfc process_one_work+0x2e0/0x474 worker_thread+0x1cc/0x270 kthread+0xd8/0xe8 ret_from_fork+0x10/0x20
Quoted 4 lines
Quoted 2 lines
This doesn't look like it's blocked in the R-Car SATA driver, but on some I/O scheduling event in the block core?
I'm not familiar with the SATA code but from a quick look it seems like the above code is only triggered from inside the ATA error handler (
ata_do_eh()->ata_eh_recover()->ata_eh_revalidate_and_attach()->schedule_work(&(ap->scsi_rescan_task) -> ata_scsi_dev_rescan()). It doesn't seem normal to me that the ATA error handler gets invoked during a resume. How about testing the following two code changes?
Sadly, that _is_ the normal workflow. SATA EH gets invoked regularly, and is handling the device resets after resume.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Martin K. Petersen
AppliedBart,
Quoted 4 lines
Recently it was reported that patch "scsi: core: pm: Rely on the device driver core for async power management" causes resuming to take longer if an ATA disk is present. This patch series fixes that regression. Please consider this patch series for kernel v5.20.
Applied to 5.20/scsi-staging, thanks!
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Martin K. Petersen
AppliedQuoted 8 lines · Bart Van Assche
Recently it was reported that patch "scsi: core: pm: Rely on the device driver core for async power management" causes resuming to take longer if an ATA disk is present. This patch series fixes that regression. Please consider this patch series for kernel v5.20.
Thanks,
[...]
Applied to 5.20/scsi-queue, thanks!
[1/2] scsi: core: Move the definition of SCSI_QUEUE_DELAY
https://git.kernel.org/mkp/scsi/c/90552cd2d1f9 [2/2] scsi: sd: Rework asynchronous resume support https://git.kernel.org/mkp/scsi/c/88f1669019bd
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering