scsi: Don't wait for completion of in-flight requests
PATCH 1/1 scsi: Don't wait for completion of in-flight requests
Problem:
When the system disk uses the scsi disk bus, The main qemu command line includes:
...
-device virtio-scsi-pci,id=scsi0 \
-device scsi-hd,scsi-id=1,drive=drive-virtio-disk -drive id=drive-virtio-disk,if=none,file=/home/kvm/test.qcow2
...
The dmesg log is as follows::
[ 50.304591][ T4382] sd 0:0:0:0: [sda] Synchronizing SCSI cache [ 50.377002][ T4382] kexec_core: Starting new kernel [ 50.669775][ T194] psci: CPU1 killed (polled 0 ms) [ 50.849665][ T194] psci: CPU2 killed (polled 0 ms) [ 51.109625][ T194] psci: CPU3 killed (polled 0 ms) [ 51.319594][ T194] psci: CPU4 killed (polled 0 ms) [ 51.489667][ T194] psci: CPU5 killed (polled 0 ms) [ 51.709582][ T194] psci: CPU6 killed (polled 0 ms) [ 51.949508][ T10] psci: CPU7 killed (polled 0 ms) [ 52.139499][ T10] psci: CPU8 killed (polled 0 ms) [ 52.289426][ T10] psci: CPU9 killed (polled 0 ms) [ 52.439552][ T10] psci: CPU10 killed (polled 0 ms) [ 52.579525][ T10] psci: CPU11 killed (polled 0 ms) [ 52.709501][ T10] psci: CPU12 killed (polled 0 ms) [ 52.819509][ T194] psci: CPU13 killed (polled 0 ms) [ 52.919509][ T194] psci: CPU14 killed (polled 0 ms) [ 243.214009][ T115] INFO: task kworker/0:1:10 blocked for more than 122 seconds. [ 243.214810][ T115] Not tainted 6.6.0+ #1 [ 243.215517][ T115] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 243.216390][ T115] task:kworker/0:1 state:D stack:0 pid:10 ppid:2 flags:0x00000008 [ 243.217299][ T115] Workqueue: events vmstat_shepherd [ 243.217816][ T115] Call trace: [ 243.218133][ T115] __switch_to+0x130/0x1e8 [ 243.218568][ T115] __schedule+0x660/0xcf8 [ 243.219013][ T115] schedule+0x58/0xf0 [ 243.219402][ T115] percpu_rwsem_wait+0xb0/0x1d0 [ 243.219880][ T115] __percpu_down_read+0x40/0xe0 [ 243.220353][ T115] cpus_read_lock+0x5c/0x70 [ 243.220795][ T115] vmstat_shepherd+0x40/0x140 [ 243.221250][ T115] process_one_work+0x170/0x3c0 [ 243.221726][ T115] worker_thread+0x234/0x3b8 [ 243.222176][ T115] kthread+0xf0/0x108 [ 243.222564][ T115] ret_from_fork+0x10/0x20 ... [ 243.254080][ T115] INFO: task kworker/0:2:194 blocked for more than 122 seconds. [ 243.254834][ T115] Not tainted 6.6.0+ #1 [ 243.255529][ T115] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 243.256378][ T115] task:kworker/0:2 state:D stack:0 pid:194 ppid:2 flags:0x00000008 [ 243.257284][ T115] Workqueue: events work_for_cpu_fn [ 243.257793][ T115] Call trace: [ 243.258111][ T115] __switch_to+0x130/0x1e8 [ 243.258541][ T115] __schedule+0x660/0xcf8 [ 243.258971][ T115] schedule+0x58/0xf0 [ 243.259360][ T115] schedule_timeout+0x280/0x2f0 [ 243.259832][ T115] wait_for_common+0xcc/0x2d8 [ 243.260287][ T115] wait_for_completion+0x20/0x38 [ 243.260767][ T115] cpuhp_kick_ap+0xe8/0x278 [ 243.261207][ T115] cpuhp_kick_ap_work+0x5c/0x188 [ 243.261688][ T115] _cpu_down+0x120/0x378 [ 243.262103][ T115] __cpu_down_maps_locked+0x20/0x38 [ 243.262609][ T115] work_for_cpu_fn+0x24/0x40 [ 243.263059][ T115] process_one_work+0x170/0x3c0 [ 243.263533][ T115] worker_thread+0x234/0x3b8 [ 243.263981][ T115] kthread+0xf0/0x108 [ 243.264405][ T115] ret_from_fork+0x10/0x20 [ 243.264846][ T115] INFO: task kworker/15:2:639 blocked for more than 122 seconds. [ 243.265602][ T115] Not tainted 6.6.0+ #1 [ 243.266296][ T115] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 243.267143][ T115] task:kworker/15:2 state:D stack:0 pid:639 ppid:2 flags:0x00000008 [ 243.268044][ T115] Workqueue: events_freezable_power_ disk_events_workfn [ 243.268727][ T115] Call trace: [ 243.269051][ T115] __switch_to+0x130/0x1e8 [ 243.269481][ T115] __schedule+0x660/0xcf8 [ 243.269903][ T115] schedule+0x58/0xf0 [ 243.270293][ T115] schedule_timeout+0x280/0x2f0 [ 243.270763][ T115] io_schedule_timeout+0x50/0x70 [ 243.271245][ T115] wait_for_common_io.constprop.0+0xb0/0x298 [ 243.271830][ T115] wait_for_completion_io+0x1c/0x30 [ 243.272335][ T115] blk_execute_rq+0x1d8/0x278 [ 243.272793][ T115] scsi_execute_cmd+0x114/0x238 [ 243.273267][ T115] sr_check_events+0xc8/0x310 [sr_mod] [ 243.273808][ T115] cdrom_check_events+0x2c/0x50 [cdrom] [ 243.274408][ T115] sr_block_check_events+0x34/0x48 [sr_mod] [ 243.274994][ T115] disk_check_events+0x44/0x1b0 [ 243.275468][ T115] disk_events_workfn+0x20/0x38 [ 243.275939][ T115] process_one_work+0x170/0x3c0 [ 243.276410][ T115] worker_thread+0x234/0x3b8 [ 243.276855][ T115] kthread+0xf0/0x108 [ 243.277241][ T115] ret_from_fork+0x10/0x20
ftrace finds that it enters an endless loop, code as follows:
if (percpu_ref_tryget(&hctx->queue->q_usage_counter)) {
while (blk_mq_hctx_has_requests(hctx))
msleep(5);
percpu_ref_put(&hctx->queue->q_usage_counter);
}Solution:
Refer to the loop and dm-rq in patch commit bf0beec0607d ("blk-mq: drain I/O when all CPUs in a hctx are offline"), add a BLK_MQ_F_STACKING and set it for scsi, so we don't need to wait for completion of in-flight requests to avoid a potential hung task.
--- drivers/scsi/scsi_lib.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/scsi/scsi_lib.c +1 -1
@@ -2065,7 +2065,7 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost) tag_set->queue_depth = shost->can_queue; tag_set->cmd_size = cmd_size; tag_set->numa_node =dev_to_node (shost->dma_dev); - tag_set->flags =BLK_MQ_F_SHOULD_MERGE ; + tag_set->flags =BLK_MQ_F_SHOULD_MERGE |BLK_MQ_F_STACKING ; tag_set->flags |=BLK_ALLOC_POLICY_TO_MQ_FLAG (shost->hostt->tag_alloc_policy);if (shost->queuecommand_may_block)
Signature · 1 line
2.20.1
Quoted 118 lines · Qiang Ma
Problem:
When the system disk uses the scsi disk bus, The main qemu command line includes:
...(quoted diff) +0
-3-device virtio-scsi-pci,id=scsi0 \ -device scsi-hd,scsi-id=1,drive=drive-virtio-disk -drive id=drive-virtio-disk,if=none,file=/home/kvm/test.qcow2The dmesg log is as follows::
[ 50.304591][ T4382] sd 0:0:0:0: [sda] Synchronizing SCSI cache [ 50.377002][ T4382] kexec_core: Starting new kernel [ 50.669775][ T194] psci: CPU1 killed (polled 0 ms) [ 50.849665][ T194] psci: CPU2 killed (polled 0 ms) [ 51.109625][ T194] psci: CPU3 killed (polled 0 ms) [ 51.319594][ T194] psci: CPU4 killed (polled 0 ms) [ 51.489667][ T194] psci: CPU5 killed (polled 0 ms) [ 51.709582][ T194] psci: CPU6 killed (polled 0 ms) [ 51.949508][ T10] psci: CPU7 killed (polled 0 ms) [ 52.139499][ T10] psci: CPU8 killed (polled 0 ms) [ 52.289426][ T10] psci: CPU9 killed (polled 0 ms) [ 52.439552][ T10] psci: CPU10 killed (polled 0 ms) [ 52.579525][ T10] psci: CPU11 killed (polled 0 ms) [ 52.709501][ T10] psci: CPU12 killed (polled 0 ms) [ 52.819509][ T194] psci: CPU13 killed (polled 0 ms) [ 52.919509][ T194] psci: CPU14 killed (polled 0 ms) [ 243.214009][ T115] INFO: task kworker/0:1:10 blocked for more than 122 seconds. [ 243.214810][ T115] Not tainted 6.6.0+ #1 [ 243.215517][ T115] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 243.216390][ T115] task:kworker/0:1 state:D stack:0 pid:10 ppid:2 flags:0x00000008 [ 243.217299][ T115] Workqueue: events vmstat_shepherd [ 243.217816][ T115] Call trace: [ 243.218133][ T115] __switch_to+0x130/0x1e8 [ 243.218568][ T115] __schedule+0x660/0xcf8 [ 243.219013][ T115] schedule+0x58/0xf0 [ 243.219402][ T115] percpu_rwsem_wait+0xb0/0x1d0 [ 243.219880][ T115] __percpu_down_read+0x40/0xe0 [ 243.220353][ T115] cpus_read_lock+0x5c/0x70 [ 243.220795][ T115] vmstat_shepherd+0x40/0x140 [ 243.221250][ T115] process_one_work+0x170/0x3c0 [ 243.221726][ T115] worker_thread+0x234/0x3b8 [ 243.222176][ T115] kthread+0xf0/0x108 [ 243.222564][ T115] ret_from_fork+0x10/0x20 ... [ 243.254080][ T115] INFO: task kworker/0:2:194 blocked for more than 122 seconds. [ 243.254834][ T115] Not tainted 6.6.0+ #1 [ 243.255529][ T115] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 243.256378][ T115] task:kworker/0:2 state:D stack:0 pid:194 ppid:2 flags:0x00000008 [ 243.257284][ T115] Workqueue: events work_for_cpu_fn [ 243.257793][ T115] Call trace: [ 243.258111][ T115] __switch_to+0x130/0x1e8 [ 243.258541][ T115] __schedule+0x660/0xcf8 [ 243.258971][ T115] schedule+0x58/0xf0 [ 243.259360][ T115] schedule_timeout+0x280/0x2f0 [ 243.259832][ T115] wait_for_common+0xcc/0x2d8 [ 243.260287][ T115] wait_for_completion+0x20/0x38 [ 243.260767][ T115] cpuhp_kick_ap+0xe8/0x278 [ 243.261207][ T115] cpuhp_kick_ap_work+0x5c/0x188 [ 243.261688][ T115] _cpu_down+0x120/0x378 [ 243.262103][ T115] __cpu_down_maps_locked+0x20/0x38 [ 243.262609][ T115] work_for_cpu_fn+0x24/0x40 [ 243.263059][ T115] process_one_work+0x170/0x3c0 [ 243.263533][ T115] worker_thread+0x234/0x3b8 [ 243.263981][ T115] kthread+0xf0/0x108 [ 243.264405][ T115] ret_from_fork+0x10/0x20 [ 243.264846][ T115] INFO: task kworker/15:2:639 blocked for more than 122 seconds. [ 243.265602][ T115] Not tainted 6.6.0+ #1 [ 243.266296][ T115] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 243.267143][ T115] task:kworker/15:2 state:D stack:0 pid:639 ppid:2 flags:0x00000008 [ 243.268044][ T115] Workqueue: events_freezable_power_ disk_events_workfn [ 243.268727][ T115] Call trace: [ 243.269051][ T115] __switch_to+0x130/0x1e8 [ 243.269481][ T115] __schedule+0x660/0xcf8 [ 243.269903][ T115] schedule+0x58/0xf0 [ 243.270293][ T115] schedule_timeout+0x280/0x2f0 [ 243.270763][ T115] io_schedule_timeout+0x50/0x70 [ 243.271245][ T115] wait_for_common_io.constprop.0+0xb0/0x298 [ 243.271830][ T115] wait_for_completion_io+0x1c/0x30 [ 243.272335][ T115] blk_execute_rq+0x1d8/0x278 [ 243.272793][ T115] scsi_execute_cmd+0x114/0x238 [ 243.273267][ T115] sr_check_events+0xc8/0x310 [sr_mod] [ 243.273808][ T115] cdrom_check_events+0x2c/0x50 [cdrom] [ 243.274408][ T115] sr_block_check_events+0x34/0x48 [sr_mod] [ 243.274994][ T115] disk_check_events+0x44/0x1b0 [ 243.275468][ T115] disk_events_workfn+0x20/0x38 [ 243.275939][ T115] process_one_work+0x170/0x3c0 [ 243.276410][ T115] worker_thread+0x234/0x3b8 [ 243.276855][ T115] kthread+0xf0/0x108 [ 243.277241][ T115] ret_from_fork+0x10/0x20ftrace finds that it enters an endless loop, code as follows:
if (percpu_ref_tryget(&hctx->queue->q_usage_counter)) { while (blk_mq_hctx_has_requests(hctx)) msleep(5); percpu_ref_put(&hctx->queue->q_usage_counter); }Solution:
Refer to the loop and dm-rq in patch commitbf0beec0607d("blk-mq: drain I/O when all CPUs in a hctx are offline"), add aBLK_MQ_F_STACKINGand set it for scsi, so we don't need to wait for completion of in-flight requests to avoid a potential hung task.Fixes bf0beec0607d ("blk-mq: drain I/O when all CPUs in a hctx are offline")Signed-off-by Qiang Ma <[email protected]>--- drivers/scsi/scsi_lib.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/scsi/scsi_lib.c +0
-0
drivers/scsi/scsi_lib.c int scsi_mq_setup_tags(struct Scsi_Host *shost) · line 2068
Quoted 5 lines
drivers/scsi/scsi_lib.c +1
-1@@ -2065,7 +2065,7 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost) tag_set->queue_depth = shost->can_queue; tag_set->cmd_size = cmd_size; tag_set->numa_node =dev_to_node (shost->dma_dev); - tag_set->flags =BLK_MQ_F_SHOULD_MERGE ; + tag_set->flags =BLK_MQ_F_SHOULD_MERGE |BLK_MQ_F_STACKING ;
This should not be set for all SCSI hosts. Some SCSI hosts rely on bf0beec0607d.
Quoted 3 lines
drivers/scsi/scsi_lib.c +0
-0tag_set->flags |=BLK_ALLOC_POLICY_TO_MQ_FLAG (shost->hostt->tag_alloc_policy);if (shost->queuecommand_may_block)
Quoted 13 lines · John Garry
drivers/scsi/scsi_lib.c int scsi_mq_setup_tags(struct Scsi_Host *shost) · line 2068
Quoted 10 lines · Qiang Ma
drivers/scsi/scsi_lib.c +1
-1@@ -2065,7 +2065,7 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost) tag_set->queue_depth = shost->can_queue; tag_set->cmd_size = cmd_size; tag_set->numa_node = dev_to_node(shost->dma_dev); - tag_set->flags = BLK_MQ_F_SHOULD_MERGE; + tag_set->flags = BLK_MQ_F_SHOULD_MERGE | BLK_MQ_F_STACKING;This should not be set for all SCSI hosts. Some SCSI hosts rely on
bf0beec0607d.
Are there any SCSI hosts for which it is safe to set this flag? To me the above change looks like a hack that should not be merged at all. Did I perhaps overlook something?
Thanks,
Bart.
Quoted 121 lines · On Tue, 26 Nov 2024 12:21:24 +0000 John Garry <[email protected]> wrote:
Quoted 120 lines · Qiang Ma
Problem:
When the system disk uses the scsi disk bus, The main qemu command line includes:
...(quoted diff) +0
-3-device virtio-scsi-pci,id=scsi0 \ -device scsi-hd,scsi-id=1,drive=drive-virtio-disk -drive id=drive-virtio-disk,if=none,file=/home/kvm/test.qcow2The dmesg log is as follows::
[ 50.304591][ T4382] sd 0:0:0:0: [sda] Synchronizing SCSI cache [ 50.377002][ T4382] kexec_core: Starting new kernel [ 50.669775][ T194] psci: CPU1 killed (polled 0 ms) [ 50.849665][ T194] psci: CPU2 killed (polled 0 ms) [ 51.109625][ T194] psci: CPU3 killed (polled 0 ms) [ 51.319594][ T194] psci: CPU4 killed (polled 0 ms) [ 51.489667][ T194] psci: CPU5 killed (polled 0 ms) [ 51.709582][ T194] psci: CPU6 killed (polled 0 ms) [ 51.949508][ T10] psci: CPU7 killed (polled 0 ms) [ 52.139499][ T10] psci: CPU8 killed (polled 0 ms) [ 52.289426][ T10] psci: CPU9 killed (polled 0 ms) [ 52.439552][ T10] psci: CPU10 killed (polled 0 ms) [ 52.579525][ T10] psci: CPU11 killed (polled 0 ms) [ 52.709501][ T10] psci: CPU12 killed (polled 0 ms) [ 52.819509][ T194] psci: CPU13 killed (polled 0 ms) [ 52.919509][ T194] psci: CPU14 killed (polled 0 ms) [ 243.214009][ T115] INFO: task kworker/0:1:10 blocked for more than 122 seconds. [ 243.214810][ T115] Not tainted 6.6.0+ #1 [ 243.215517][ T115] "echo 0Quoted 3 lines
/proc/sys/kernel/hung_task_timeout_secs" disables this message.[ 243.216390][ T115] task:kworker/0:1 state:D stack:0pid:10 ppid:2 flags:0x00000008
[ 243.217299][ T115] Workqueue: events vmstat_shepherd [ 243.217816][ T115] Call trace: [ 243.218133][ T115] __switch_to+0x130/0x1e8 [ 243.218568][ T115] __schedule+0x660/0xcf8 [ 243.219013][ T115] schedule+0x58/0xf0 [ 243.219402][ T115] percpu_rwsem_wait+0xb0/0x1d0 [ 243.219880][ T115] __percpu_down_read+0x40/0xe0 [ 243.220353][ T115] cpus_read_lock+0x5c/0x70 [ 243.220795][ T115] vmstat_shepherd+0x40/0x140 [ 243.221250][ T115] process_one_work+0x170/0x3c0 [ 243.221726][ T115] worker_thread+0x234/0x3b8 [ 243.222176][ T115] kthread+0xf0/0x108 [ 243.222564][ T115] ret_from_fork+0x10/0x20 ... [ 243.254080][ T115] INFO: task kworker/0:2:194 blocked for more than 122 seconds. [ 243.254834][ T115] Not tainted 6.6.0+ #1 [ 243.255529][ T115] "echo 0Quoted 3 lines
/proc/sys/kernel/hung_task_timeout_secs" disables this message.[ 243.256378][ T115] task:kworker/0:2 state:D stack:0pid:194 ppid:2 flags:0x00000008
[ 243.257284][ T115] Workqueue: events work_for_cpu_fn [ 243.257793][ T115] Call trace: [ 243.258111][ T115] __switch_to+0x130/0x1e8 [ 243.258541][ T115] __schedule+0x660/0xcf8 [ 243.258971][ T115] schedule+0x58/0xf0 [ 243.259360][ T115] schedule_timeout+0x280/0x2f0 [ 243.259832][ T115] wait_for_common+0xcc/0x2d8 [ 243.260287][ T115] wait_for_completion+0x20/0x38 [ 243.260767][ T115] cpuhp_kick_ap+0xe8/0x278 [ 243.261207][ T115] cpuhp_kick_ap_work+0x5c/0x188 [ 243.261688][ T115] _cpu_down+0x120/0x378 [ 243.262103][ T115] __cpu_down_maps_locked+0x20/0x38 [ 243.262609][ T115] work_for_cpu_fn+0x24/0x40 [ 243.263059][ T115] process_one_work+0x170/0x3c0 [ 243.263533][ T115] worker_thread+0x234/0x3b8 [ 243.263981][ T115] kthread+0xf0/0x108 [ 243.264405][ T115] ret_from_fork+0x10/0x20 [ 243.264846][ T115] INFO: task kworker/15:2:639 blocked for more than 122 seconds. [ 243.265602][ T115] Not tainted 6.6.0+ #1 [ 243.266296][ T115] "echo 0Quoted 3 lines
/proc/sys/kernel/hung_task_timeout_secs" disables this message.[ 243.267143][ T115] task:kworker/15:2 state:D stack:0pid:639 ppid:2 flags:0x00000008
[ 243.268044][ T115] Workqueue: events_freezable_power_ disk_events_workfn [ 243.268727][ T115] Call trace: [ 243.269051][ T115] __switch_to+0x130/0x1e8 [ 243.269481][ T115] __schedule+0x660/0xcf8 [ 243.269903][ T115] schedule+0x58/0xf0 [ 243.270293][ T115] schedule_timeout+0x280/0x2f0 [ 243.270763][ T115] io_schedule_timeout+0x50/0x70 [ 243.271245][ T115] wait_for_common_io.constprop.0+0xb0/0x298 [ 243.271830][ T115] wait_for_completion_io+0x1c/0x30 [ 243.272335][ T115] blk_execute_rq+0x1d8/0x278 [ 243.272793][ T115] scsi_execute_cmd+0x114/0x238 [ 243.273267][ T115] sr_check_events+0xc8/0x310 [sr_mod] [ 243.273808][ T115] cdrom_check_events+0x2c/0x50 [cdrom] [ 243.274408][ T115] sr_block_check_events+0x34/0x48 [sr_mod] [ 243.274994][ T115] disk_check_events+0x44/0x1b0 [ 243.275468][ T115] disk_events_workfn+0x20/0x38 [ 243.275939][ T115] process_one_work+0x170/0x3c0 [ 243.276410][ T115] worker_thread+0x234/0x3b8 [ 243.276855][ T115] kthread+0xf0/0x108 [ 243.277241][ T115] ret_from_fork+0x10/0x20ftrace finds that it enters an endless loop, code as follows:
if (percpu_ref_tryget(&hctx->queue->q_usage_counter)) { while (blk_mq_hctx_has_requests(hctx)) msleep(5); percpu_ref_put(&hctx->queue->q_usage_counter); }Solution:
Refer to the loop and dm-rq in patch commitbf0beec0607d("blk-mq: drain I/O when all CPUs in a hctx are offline"), add aBLK_MQ_F_STACKINGand set it for scsi, so we don't need to wait for completion of in-flight requests to avoid a potential hung task.Fixes bf0beec0607d ("blk-mq: drain I/O when all CPUs in a hctx areoffline")
Signed-off-by Qiang Ma <[email protected]>--- drivers/scsi/scsi_lib.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 16 lines
drivers/scsi/scsi_lib.c int scsi_mq_setup_tags(struct Scsi_Host · line 2069
Quoted 11 lines
drivers/scsi/scsi_lib.c +1
-1@@ -2065,7 +2065,7 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost) tag_set->queue_depth = shost->can_queue; tag_set->cmd_size = cmd_size; tag_set->numa_node =dev_to_node (shost->dma_dev); - tag_set->flags =BLK_MQ_F_SHOULD_MERGE ; + tag_set->flags =BLK_MQ_F_SHOULD_MERGE |BLK_MQ_F_STACKING ;This should not be set for all SCSI hosts. Some SCSI hosts rely on
bf0beec0607d.
What are the side effects of setting this up for some scsi hosts?
Quoted 5 lines
Quoted 3 lines
drivers/scsi/scsi_lib.c +0
-0tag_set->flags |=BLK_ALLOC_POLICY_TO_MQ_FLAG (shost->hostt->tag_alloc_policy);if (
shost->queuecommand_may_block)
Quoted 82 lines · Qiang Ma
Problem:
When the system disk uses the scsi disk bus, The main qemu command line includes:
...(quoted diff) +0
-3-device virtio-scsi-pci,id=scsi0 \ -device scsi-hd,scsi-id=1,drive=drive-virtio-disk -drive id=drive-virtio-disk,if=none,file=/home/kvm/test.qcow2The dmesg log is as follows::
[ 50.304591][ T4382] sd 0:0:0:0: [sda] Synchronizing SCSI cache [ 50.377002][ T4382] kexec_core: Starting new kernel [ 50.669775][ T194] psci: CPU1 killed (polled 0 ms) [ 50.849665][ T194] psci: CPU2 killed (polled 0 ms) [ 51.109625][ T194] psci: CPU3 killed (polled 0 ms) [ 51.319594][ T194] psci: CPU4 killed (polled 0 ms) [ 51.489667][ T194] psci: CPU5 killed (polled 0 ms) [ 51.709582][ T194] psci: CPU6 killed (polled 0 ms) [ 51.949508][ T10] psci: CPU7 killed (polled 0 ms) [ 52.139499][ T10] psci: CPU8 killed (polled 0 ms) [ 52.289426][ T10] psci: CPU9 killed (polled 0 ms) [ 52.439552][ T10] psci: CPU10 killed (polled 0 ms) [ 52.579525][ T10] psci: CPU11 killed (polled 0 ms) [ 52.709501][ T10] psci: CPU12 killed (polled 0 ms) [ 52.819509][ T194] psci: CPU13 killed (polled 0 ms) [ 52.919509][ T194] psci: CPU14 killed (polled 0 ms) [ 243.214009][ T115] INFO: task kworker/0:1:10 blocked for more than 122 seconds. [ 243.214810][ T115] Not tainted 6.6.0+ #1 [ 243.215517][ T115] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 243.216390][ T115] task:kworker/0:1 state:D stack:0 pid:10 ppid:2 flags:0x00000008 [ 243.217299][ T115] Workqueue: events vmstat_shepherd [ 243.217816][ T115] Call trace: [ 243.218133][ T115] __switch_to+0x130/0x1e8 [ 243.218568][ T115] __schedule+0x660/0xcf8 [ 243.219013][ T115] schedule+0x58/0xf0 [ 243.219402][ T115] percpu_rwsem_wait+0xb0/0x1d0 [ 243.219880][ T115] __percpu_down_read+0x40/0xe0 [ 243.220353][ T115] cpus_read_lock+0x5c/0x70 [ 243.220795][ T115] vmstat_shepherd+0x40/0x140 [ 243.221250][ T115] process_one_work+0x170/0x3c0 [ 243.221726][ T115] worker_thread+0x234/0x3b8 [ 243.222176][ T115] kthread+0xf0/0x108 [ 243.222564][ T115] ret_from_fork+0x10/0x20 ... [ 243.254080][ T115] INFO: task kworker/0:2:194 blocked for more than 122 seconds. [ 243.254834][ T115] Not tainted 6.6.0+ #1 [ 243.255529][ T115] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 243.256378][ T115] task:kworker/0:2 state:D stack:0 pid:194 ppid:2 flags:0x00000008 [ 243.257284][ T115] Workqueue: events work_for_cpu_fn [ 243.257793][ T115] Call trace: [ 243.258111][ T115] __switch_to+0x130/0x1e8 [ 243.258541][ T115] __schedule+0x660/0xcf8 [ 243.258971][ T115] schedule+0x58/0xf0 [ 243.259360][ T115] schedule_timeout+0x280/0x2f0 [ 243.259832][ T115] wait_for_common+0xcc/0x2d8 [ 243.260287][ T115] wait_for_completion+0x20/0x38 [ 243.260767][ T115] cpuhp_kick_ap+0xe8/0x278 [ 243.261207][ T115] cpuhp_kick_ap_work+0x5c/0x188 [ 243.261688][ T115] _cpu_down+0x120/0x378 [ 243.262103][ T115] __cpu_down_maps_locked+0x20/0x38 [ 243.262609][ T115] work_for_cpu_fn+0x24/0x40 [ 243.263059][ T115] process_one_work+0x170/0x3c0 [ 243.263533][ T115] worker_thread+0x234/0x3b8 [ 243.263981][ T115] kthread+0xf0/0x108 [ 243.264405][ T115] ret_from_fork+0x10/0x20 [ 243.264846][ T115] INFO: task kworker/15:2:639 blocked for more than 122 seconds. [ 243.265602][ T115] Not tainted 6.6.0+ #1 [ 243.266296][ T115] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 243.267143][ T115] task:kworker/15:2 state:D stack:0 pid:639 ppid:2 flags:0x00000008 [ 243.268044][ T115] Workqueue: events_freezable_power_ disk_events_workfn [ 243.268727][ T115] Call trace: [ 243.269051][ T115] __switch_to+0x130/0x1e8 [ 243.269481][ T115] __schedule+0x660/0xcf8 [ 243.269903][ T115] schedule+0x58/0xf0 [ 243.270293][ T115] schedule_timeout+0x280/0x2f0 [ 243.270763][ T115] io_schedule_timeout+0x50/0x70 [ 243.271245][ T115] wait_for_common_io.constprop.0+0xb0/0x298 [ 243.271830][ T115] wait_for_completion_io+0x1c/0x30 [ 243.272335][ T115] blk_execute_rq+0x1d8/0x278 [ 243.272793][ T115] scsi_execute_cmd+0x114/0x238 [ 243.273267][ T115] sr_check_events+0xc8/0x310 [sr_mod]
Quoted 8 lines
[ 243.273808][ T115] cdrom_check_events+0x2c/0x50 [cdrom] [ 243.274408][ T115] sr_block_check_events+0x34/0x48 [sr_mod] [ 243.274994][ T115] disk_check_events+0x44/0x1b0 [ 243.275468][ T115] disk_events_workfn+0x20/0x38 [ 243.275939][ T115] process_one_work+0x170/0x3c0 [ 243.276410][ T115] worker_thread+0x234/0x3b8 [ 243.276855][ T115] kthread+0xf0/0x108 [ 243.277241][ T115] ret_from_fork+0x10/0x20
Question is why this scsi command can't be completed?
When blk_mq_hctx_notify_offline() is called, the CPU isn't shutdown yet, and it still can handle interrupt of this SCSI command.
Thanks,
Ming
Quoted 98 lines · On Tue, 26 Nov 2024 22:15:02 +0800 Ming Lei <[email protected]> wrote:
Quoted 88 lines · Qiang Ma
Problem:
When the system disk uses the scsi disk bus, The main qemu command line includes:
...(quoted diff) +0
-3-device virtio-scsi-pci,id=scsi0 \ -device scsi-hd,scsi-id=1,drive=drive-virtio-disk -drive id=drive-virtio-disk,if=none,file=/home/kvm/test.qcow2The dmesg log is as follows::
[ 50.304591][ T4382] sd 0:0:0:0: [sda] Synchronizing SCSI cache [ 50.377002][ T4382] kexec_core: Starting new kernel [ 50.669775][ T194] psci: CPU1 killed (polled 0 ms) [ 50.849665][ T194] psci: CPU2 killed (polled 0 ms) [ 51.109625][ T194] psci: CPU3 killed (polled 0 ms) [ 51.319594][ T194] psci: CPU4 killed (polled 0 ms) [ 51.489667][ T194] psci: CPU5 killed (polled 0 ms) [ 51.709582][ T194] psci: CPU6 killed (polled 0 ms) [ 51.949508][ T10] psci: CPU7 killed (polled 0 ms) [ 52.139499][ T10] psci: CPU8 killed (polled 0 ms) [ 52.289426][ T10] psci: CPU9 killed (polled 0 ms) [ 52.439552][ T10] psci: CPU10 killed (polled 0 ms) [ 52.579525][ T10] psci: CPU11 killed (polled 0 ms) [ 52.709501][ T10] psci: CPU12 killed (polled 0 ms) [ 52.819509][ T194] psci: CPU13 killed (polled 0 ms) [ 52.919509][ T194] psci: CPU14 killed (polled 0 ms) [ 243.214009][ T115] INFO: task kworker/0:1:10 blocked for more than 122 seconds. [ 243.214810][ T115] Not tainted 6.6.0+ #1 [ 243.215517][ T115] "echo 0Quoted 3 lines
/proc/sys/kernel/hung_task_timeout_secs" disables this message.[ 243.216390][ T115] task:kworker/0:1 state:D stack:0pid:10 ppid:2 flags:0x00000008
[ 243.217299][ T115] Workqueue: events vmstat_shepherd [ 243.217816][ T115] Call trace: [ 243.218133][ T115] __switch_to+0x130/0x1e8 [ 243.218568][ T115] __schedule+0x660/0xcf8 [ 243.219013][ T115] schedule+0x58/0xf0 [ 243.219402][ T115] percpu_rwsem_wait+0xb0/0x1d0 [ 243.219880][ T115] __percpu_down_read+0x40/0xe0 [ 243.220353][ T115] cpus_read_lock+0x5c/0x70 [ 243.220795][ T115] vmstat_shepherd+0x40/0x140 [ 243.221250][ T115] process_one_work+0x170/0x3c0 [ 243.221726][ T115] worker_thread+0x234/0x3b8 [ 243.222176][ T115] kthread+0xf0/0x108 [ 243.222564][ T115] ret_from_fork+0x10/0x20 ... [ 243.254080][ T115] INFO: task kworker/0:2:194 blocked for more than 122 seconds. [ 243.254834][ T115] Not tainted 6.6.0+ #1 [ 243.255529][ T115] "echo 0Quoted 3 lines
/proc/sys/kernel/hung_task_timeout_secs" disables this message.[ 243.256378][ T115] task:kworker/0:2 state:D stack:0pid:194 ppid:2 flags:0x00000008
[ 243.257284][ T115] Workqueue: events work_for_cpu_fn [ 243.257793][ T115] Call trace: [ 243.258111][ T115] __switch_to+0x130/0x1e8 [ 243.258541][ T115] __schedule+0x660/0xcf8 [ 243.258971][ T115] schedule+0x58/0xf0 [ 243.259360][ T115] schedule_timeout+0x280/0x2f0 [ 243.259832][ T115] wait_for_common+0xcc/0x2d8 [ 243.260287][ T115] wait_for_completion+0x20/0x38 [ 243.260767][ T115] cpuhp_kick_ap+0xe8/0x278 [ 243.261207][ T115] cpuhp_kick_ap_work+0x5c/0x188 [ 243.261688][ T115] _cpu_down+0x120/0x378 [ 243.262103][ T115] __cpu_down_maps_locked+0x20/0x38 [ 243.262609][ T115] work_for_cpu_fn+0x24/0x40 [ 243.263059][ T115] process_one_work+0x170/0x3c0 [ 243.263533][ T115] worker_thread+0x234/0x3b8 [ 243.263981][ T115] kthread+0xf0/0x108 [ 243.264405][ T115] ret_from_fork+0x10/0x20 [ 243.264846][ T115] INFO: task kworker/15:2:639 blocked for more than 122 seconds. [ 243.265602][ T115] Not tainted 6.6.0+ #1 [ 243.266296][ T115] "echo 0Quoted 3 lines
/proc/sys/kernel/hung_task_timeout_secs" disables this message.[ 243.267143][ T115] task:kworker/15:2 state:D stack:0pid:639 ppid:2 flags:0x00000008
[ 243.268044][ T115] Workqueue: events_freezable_power_ disk_events_workfn [ 243.268727][ T115] Call trace: [ 243.269051][ T115] __switch_to+0x130/0x1e8 [ 243.269481][ T115] __schedule+0x660/0xcf8 [ 243.269903][ T115] schedule+0x58/0xf0 [ 243.270293][ T115] schedule_timeout+0x280/0x2f0 [ 243.270763][ T115] io_schedule_timeout+0x50/0x70 [ 243.271245][ T115] wait_for_common_io.constprop.0+0xb0/0x298 [ 243.271830][ T115] wait_for_completion_io+0x1c/0x30 [ 243.272335][ T115] blk_execute_rq+0x1d8/0x278 [ 243.272793][ T115] scsi_execute_cmd+0x114/0x238 [ 243.273267][ T115] sr_check_events+0xc8/0x310 [sr_mod]Quoted 8 lines
[ 243.273808][ T115] cdrom_check_events+0x2c/0x50 [cdrom] [ 243.274408][ T115] sr_block_check_events+0x34/0x48 [sr_mod] [ 243.274994][ T115] disk_check_events+0x44/0x1b0 [ 243.275468][ T115] disk_events_workfn+0x20/0x38 [ 243.275939][ T115] process_one_work+0x170/0x3c0 [ 243.276410][ T115] worker_thread+0x234/0x3b8 [ 243.276855][ T115] kthread+0xf0/0x108 [ 243.277241][ T115] ret_from_fork+0x10/0x20
Quoted 10 lines
Question is why this scsi command can't be completed?
When
blk_mq_hctx_notify_offline()is called, the CPU isn't shutdown yet, and it still can handle interrupt of this SCSI command.Thanks,
Ming
Sorry, at the moment I don't know why it can't be finished, just provides a solution like loop and dm-rq.
Currently see the call stack:
=> blk_mq_run_hw_queue
=>__blk_mq_sched_restart
=> blk_mq_run_hw_queue
=> __blk_mq_sched_restart
=> __blk_mq_free_request
=> blk_mq_free_request
=> blk_mq_end_request
=> blk_flush_complete_seq
=> flush_end_io
=> __blk_mq_end_request
=> scsi_end_request
=> scsi_io_completion
=> scsi_finish_command
=> scsi_complete
=> blk_mq_complete_request
=> scsi_done_internal
=> scsi_done
=> virtscsi_complete_cmd
=> virtscsi_req_done
=> vring_interrupt
In finction blk_mq_run_hw_queue():
__blk_mq_run_dispatch_ops(hctx->queue, false,
need_run = !blk_queue_quiesced(hctx->queue) && blk_mq_hctx_has_pending(hctx)); if (!need_run) return;
Come here and return directly.
Quoted 13 lines · Qiang Ma
Quoted 10 lines · On Tue, 26 Nov 2024 22:15:02 +0800 Ming Lei <[email protected]> wrote:
Quoted 9 lines · Qiang Ma
Problem:
When the system disk uses the scsi disk bus, The main qemu command line includes:
...(quoted diff) +0
-3-device virtio-scsi-pci,id=scsi0 \ -device scsi-hd,scsi-id=1,drive=drive-virtio-disk -drive id=drive-virtio-disk,if=none,file=/home/kvm/test.qcow2
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
The dmesg log is as follows::
[ 50.304591][ T4382] sd 0:0:0:0: [sda] Synchronizing SCSI cache [ 50.377002][ T4382] kexec_core: Starting new kernel
Why is one new kernel started? what event triggers kernel_kexec()?
machine_shutdown() follows, probably the scsi controller is dead.
Quoted 121 lines
Quoted 84 lines
Quoted 75 lines
[ 50.669775][ T194] psci: CPU1 killed (polled 0 ms) [ 50.849665][ T194] psci: CPU2 killed (polled 0 ms) [ 51.109625][ T194] psci: CPU3 killed (polled 0 ms) [ 51.319594][ T194] psci: CPU4 killed (polled 0 ms) [ 51.489667][ T194] psci: CPU5 killed (polled 0 ms) [ 51.709582][ T194] psci: CPU6 killed (polled 0 ms) [ 51.949508][ T10] psci: CPU7 killed (polled 0 ms) [ 52.139499][ T10] psci: CPU8 killed (polled 0 ms) [ 52.289426][ T10] psci: CPU9 killed (polled 0 ms) [ 52.439552][ T10] psci: CPU10 killed (polled 0 ms) [ 52.579525][ T10] psci: CPU11 killed (polled 0 ms) [ 52.709501][ T10] psci: CPU12 killed (polled 0 ms) [ 52.819509][ T194] psci: CPU13 killed (polled 0 ms) [ 52.919509][ T194] psci: CPU14 killed (polled 0 ms) [ 243.214009][ T115] INFO: task kworker/0:1:10 blocked for more than 122 seconds. [ 243.214810][ T115] Not tainted 6.6.0+ #1 [ 243.215517][ T115] "echo 0Quoted 3 lines
/proc/sys/kernel/hung_task_timeout_secs" disables this message.[ 243.216390][ T115] task:kworker/0:1 state:D stack:0pid:10 ppid:2 flags:0x00000008
[ 243.217299][ T115] Workqueue: events vmstat_shepherd [ 243.217816][ T115] Call trace: [ 243.218133][ T115] __switch_to+0x130/0x1e8 [ 243.218568][ T115] __schedule+0x660/0xcf8 [ 243.219013][ T115] schedule+0x58/0xf0 [ 243.219402][ T115] percpu_rwsem_wait+0xb0/0x1d0 [ 243.219880][ T115] __percpu_down_read+0x40/0xe0 [ 243.220353][ T115] cpus_read_lock+0x5c/0x70 [ 243.220795][ T115] vmstat_shepherd+0x40/0x140 [ 243.221250][ T115] process_one_work+0x170/0x3c0 [ 243.221726][ T115] worker_thread+0x234/0x3b8 [ 243.222176][ T115] kthread+0xf0/0x108 [ 243.222564][ T115] ret_from_fork+0x10/0x20 ... [ 243.254080][ T115] INFO: task kworker/0:2:194 blocked for more than 122 seconds. [ 243.254834][ T115] Not tainted 6.6.0+ #1 [ 243.255529][ T115] "echo 0Quoted 3 lines
/proc/sys/kernel/hung_task_timeout_secs" disables this message.[ 243.256378][ T115] task:kworker/0:2 state:D stack:0pid:194 ppid:2 flags:0x00000008
[ 243.257284][ T115] Workqueue: events work_for_cpu_fn [ 243.257793][ T115] Call trace: [ 243.258111][ T115] __switch_to+0x130/0x1e8 [ 243.258541][ T115] __schedule+0x660/0xcf8 [ 243.258971][ T115] schedule+0x58/0xf0 [ 243.259360][ T115] schedule_timeout+0x280/0x2f0 [ 243.259832][ T115] wait_for_common+0xcc/0x2d8 [ 243.260287][ T115] wait_for_completion+0x20/0x38 [ 243.260767][ T115] cpuhp_kick_ap+0xe8/0x278 [ 243.261207][ T115] cpuhp_kick_ap_work+0x5c/0x188 [ 243.261688][ T115] _cpu_down+0x120/0x378 [ 243.262103][ T115] __cpu_down_maps_locked+0x20/0x38 [ 243.262609][ T115] work_for_cpu_fn+0x24/0x40 [ 243.263059][ T115] process_one_work+0x170/0x3c0 [ 243.263533][ T115] worker_thread+0x234/0x3b8 [ 243.263981][ T115] kthread+0xf0/0x108 [ 243.264405][ T115] ret_from_fork+0x10/0x20 [ 243.264846][ T115] INFO: task kworker/15:2:639 blocked for more than 122 seconds. [ 243.265602][ T115] Not tainted 6.6.0+ #1 [ 243.266296][ T115] "echo 0Quoted 3 lines
/proc/sys/kernel/hung_task_timeout_secs" disables this message.[ 243.267143][ T115] task:kworker/15:2 state:D stack:0pid:639 ppid:2 flags:0x00000008
[ 243.268044][ T115] Workqueue: events_freezable_power_ disk_events_workfn [ 243.268727][ T115] Call trace: [ 243.269051][ T115] __switch_to+0x130/0x1e8 [ 243.269481][ T115] __schedule+0x660/0xcf8 [ 243.269903][ T115] schedule+0x58/0xf0 [ 243.270293][ T115] schedule_timeout+0x280/0x2f0 [ 243.270763][ T115] io_schedule_timeout+0x50/0x70 [ 243.271245][ T115] wait_for_common_io.constprop.0+0xb0/0x298 [ 243.271830][ T115] wait_for_completion_io+0x1c/0x30 [ 243.272335][ T115] blk_execute_rq+0x1d8/0x278 [ 243.272793][ T115] scsi_execute_cmd+0x114/0x238 [ 243.273267][ T115] sr_check_events+0xc8/0x310 [sr_mod]Quoted 8 lines
[ 243.273808][ T115] cdrom_check_events+0x2c/0x50 [cdrom] [ 243.274408][ T115] sr_block_check_events+0x34/0x48 [sr_mod] [ 243.274994][ T115] disk_check_events+0x44/0x1b0 [ 243.275468][ T115] disk_events_workfn+0x20/0x38 [ 243.275939][ T115] process_one_work+0x170/0x3c0 [ 243.276410][ T115] worker_thread+0x234/0x3b8 [ 243.276855][ T115] kthread+0xf0/0x108 [ 243.277241][ T115] ret_from_fork+0x10/0x20Quoted 10 lines
Question is why this scsi command can't be completed?
When
blk_mq_hctx_notify_offline()is called, the CPU isn't shutdown yet, and it still can handle interrupt of this SCSI command.Thanks,
MingSorry, at the moment I don't know why it can't be finished, just provides a solution like loop and dm-rq.
Currently see the call stack:
=>blk_mq_run_hw_queue
=>__blk_mq_sched_restart
=>blk_mq_run_hw_queue
=>__blk_mq_sched_restart
=>__blk_mq_free_request
=>blk_mq_free_request
=>blk_mq_end_request
=>blk_flush_complete_seq
=>flush_end_io
=>__blk_mq_end_request
=>scsi_end_request
=>scsi_io_completion
=>scsi_finish_command
=>scsi_complete
=>blk_mq_complete_request
=>scsi_done_internal
=>scsi_done
=>virtscsi_complete_cmd
=>virtscsi_req_done
=>vring_interruptIn finction
blk_mq_run_hw_queue():
__blk_mq_run_dispatch_ops(hctx->queue, false,
Quoted 7 lines
need_run = !blk_queue_quiesced(hctx->queue) && blk_mq_hctx_has_pending(hctx)); if (!need_run) return;Come here and return directly.
Does blk_queue_quiesced() return true?
Thanks,
Ming
Quoted 14 lines · On Wed, 27 Nov 2024 23:20:28 +0800 Ming Lei <[email protected]> wrote:
Quoted 13 lines · Qiang Ma
Quoted 10 lines · On Tue, 26 Nov 2024 22:15:02 +0800 Ming Lei <[email protected]> wrote:
Quoted 9 lines · Qiang Ma
Problem:
When the system disk uses the scsi disk bus, The main qemu command line includes:
...(quoted diff) +0
-3-device virtio-scsi-pci,id=scsi0 \ -device scsi-hd,scsi-id=1,drive=drive-virtio-disk -drive id=drive-virtio-disk,if=none,file=/home/kvm/test.qcow2
Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
The dmesg log is as follows::
[ 50.304591][ T4382] sd 0:0:0:0: [sda] Synchronizing SCSIcache [ 50.377002][ T4382]
kexec_core: Starting new kernelWhy is one new kernel started? what event triggers
kernel_kexec()?
machine_shutdown()follows, probably the scsi controller is dead.
Yes, triggered by kexec, manually execute the following command:
kexec -l /boot/vmlinuz-6.6.0+ --reuse-cmdline --initrd=/boot/initramfs-6.6.0+.img kexec -e
Quoted 126 lines
Quoted 126 lines
Quoted 87 lines
Quoted 76 lines
[ 50.669775][ T194] psci: CPU1 killed (polled 0 ms) [ 50.849665][ T194] psci: CPU2 killed (polled 0 ms) [ 51.109625][ T194] psci: CPU3 killed (polled 0 ms) [ 51.319594][ T194] psci: CPU4 killed (polled 0 ms) [ 51.489667][ T194] psci: CPU5 killed (polled 0 ms) [ 51.709582][ T194] psci: CPU6 killed (polled 0 ms) [ 51.949508][ T10] psci: CPU7 killed (polled 0 ms) [ 52.139499][ T10] psci: CPU8 killed (polled 0 ms) [ 52.289426][ T10] psci: CPU9 killed (polled 0 ms) [ 52.439552][ T10] psci: CPU10 killed (polled 0 ms) [ 52.579525][ T10] psci: CPU11 killed (polled 0 ms) [ 52.709501][ T10] psci: CPU12 killed (polled 0 ms) [ 52.819509][ T194] psci: CPU13 killed (polled 0 ms) [ 52.919509][ T194] psci: CPU14 killed (polled 0 ms) [ 243.214009][ T115] INFO: task kworker/0:1:10 blocked formore than 122 seconds. [ 243.214810][ T115] Not tainted
6.6.0+ #1 [ 243.215517][ T115] "echo 0Quoted 3 lines
/proc/sys/kernel/hung_task_timeout_secs" disables this
message. [ 243.216390][ T115] task:kworker/0:1 state:D
stack:0 pid:10 ppid:2 flags:0x00000008[ 243.217299][ T115] Workqueue: events vmstat_shepherd [ 243.217816][ T115] Call trace: [ 243.218133][ T115] __switch_to+0x130/0x1e8 [ 243.218568][ T115] __schedule+0x660/0xcf8 [ 243.219013][ T115] schedule+0x58/0xf0 [ 243.219402][ T115] percpu_rwsem_wait+0xb0/0x1d0 [ 243.219880][ T115] __percpu_down_read+0x40/0xe0 [ 243.220353][ T115] cpus_read_lock+0x5c/0x70 [ 243.220795][ T115] vmstat_shepherd+0x40/0x140 [ 243.221250][ T115] process_one_work+0x170/0x3c0 [ 243.221726][ T115] worker_thread+0x234/0x3b8 [ 243.222176][ T115] kthread+0xf0/0x108 [ 243.222564][ T115] ret_from_fork+0x10/0x20 ... [ 243.254080][ T115] INFO: task kworker/0:2:194 blocked formore than 122 seconds. [ 243.254834][ T115] Not tainted
6.6.0+ #1 [ 243.255529][ T115] "echo 0Quoted 3 lines
/proc/sys/kernel/hung_task_timeout_secs" disables this
message. [ 243.256378][ T115] task:kworker/0:2 state:D
stack:0 pid:194 ppid:2 flags:0x00000008[ 243.257284][ T115] Workqueue: events work_for_cpu_fn [ 243.257793][ T115] Call trace: [ 243.258111][ T115] __switch_to+0x130/0x1e8 [ 243.258541][ T115] __schedule+0x660/0xcf8 [ 243.258971][ T115] schedule+0x58/0xf0 [ 243.259360][ T115] schedule_timeout+0x280/0x2f0 [ 243.259832][ T115] wait_for_common+0xcc/0x2d8 [ 243.260287][ T115] wait_for_completion+0x20/0x38 [ 243.260767][ T115] cpuhp_kick_ap+0xe8/0x278 [ 243.261207][ T115] cpuhp_kick_ap_work+0x5c/0x188 [ 243.261688][ T115] _cpu_down+0x120/0x378 [ 243.262103][ T115] __cpu_down_maps_locked+0x20/0x38 [ 243.262609][ T115] work_for_cpu_fn+0x24/0x40 [ 243.263059][ T115] process_one_work+0x170/0x3c0 [ 243.263533][ T115] worker_thread+0x234/0x3b8 [ 243.263981][ T115] kthread+0xf0/0x108 [ 243.264405][ T115] ret_from_fork+0x10/0x20 [ 243.264846][ T115] INFO: task kworker/15:2:639 blocked formore than 122 seconds. [ 243.265602][ T115] Not tainted
6.6.0+ #1 [ 243.266296][ T115] "echo 0Quoted 3 lines
/proc/sys/kernel/hung_task_timeout_secs" disables this
message. [ 243.267143][ T115] task:kworker/15:2 state:D
stack:0 pid:639 ppid:2 flags:0x00000008[ 243.268044][ T115] Workqueue: events_freezable_power_ disk_events_workfn [ 243.268727][ T115] Call trace: [ 243.269051][ T115] __switch_to+0x130/0x1e8 [ 243.269481][ T115] __schedule+0x660/0xcf8 [ 243.269903][ T115] schedule+0x58/0xf0 [ 243.270293][ T115] schedule_timeout+0x280/0x2f0 [ 243.270763][ T115] io_schedule_timeout+0x50/0x70 [ 243.271245][ T115] wait_for_common_io.constprop.0+0xb0/0x298 [ 243.271830][ T115] wait_for_completion_io+0x1c/0x30 [ 243.272335][ T115] blk_execute_rq+0x1d8/0x278 [ 243.272793][ T115] scsi_execute_cmd+0x114/0x238 [ 243.273267][ T115] sr_check_events+0xc8/0x310 [sr_mod]Quoted 8 lines
[ 243.273808][ T115] cdrom_check_events+0x2c/0x50 [cdrom] [ 243.274408][ T115] sr_block_check_events+0x34/0x48 [sr_mod] [ 243.274994][ T115] disk_check_events+0x44/0x1b0 [ 243.275468][ T115] disk_events_workfn+0x20/0x38 [ 243.275939][ T115] process_one_work+0x170/0x3c0 [ 243.276410][ T115] worker_thread+0x234/0x3b8 [ 243.276855][ T115] kthread+0xf0/0x108 [ 243.277241][ T115] ret_from_fork+0x10/0x20Question is why this scsi command can't be completed?
Quoted 9 lines
When
blk_mq_hctx_notify_offline()is called, the CPU isn't shutdown yet, and it still can handle interrupt of this SCSI command.Thanks,
MingSorry, at the moment I don't know why it can't be finished, just provides a solution like loop and dm-rq.
Currently see the call stack:
=>blk_mq_run_hw_queue
=>__blk_mq_sched_restart
=>blk_mq_run_hw_queue
=>__blk_mq_sched_restart
=>__blk_mq_free_request
=>blk_mq_free_request
=>blk_mq_end_request
=>blk_flush_complete_seq
=>flush_end_io
=>__blk_mq_end_request
=>scsi_end_request
=>scsi_io_completion
=>scsi_finish_command
=>scsi_complete
=>blk_mq_complete_request
=>scsi_done_internal
=>scsi_done
=>virtscsi_complete_cmd
=>virtscsi_req_done
=>vring_interruptIn finction
blk_mq_run_hw_queue():
__blk_mq_run_dispatch_ops(hctx->queue, false,need_run = !blk_queue_quiesced(hctx->queue) && blk_mq_hctx_has_pending(hctx));
Quoted 7 lines
Quoted 4 lines
if (!need_run) return;Come here and return directly.
Does
blk_queue_quiesced()return true?
blk_queue_quiesced() return false
Sorry, the calltrace above is not the one that is stuck this time. The hang is caused by the wait_for_completion_io(), in blk_execute_rq():
blk_status_t blk_execute_rq(struct request *rq, bool at_head)
{
...
rq->end_io_data = &wait;
rq->end_io = blk_end_sync_rq;
...
blk_account_io_start(rq);
blk_mq_insert_request(rq, at_head ? BLK_MQ_INSERT_AT_HEAD : 0);
blk_mq_run_hw_queue(hctx, false);
if (blk_rq_is_poll(rq)) {
blk_rq_poll_completion(rq, &wait.done);
} else {
...
wait_for_completion_io(&wait.done);
}
}In this case, end_io is blk_end_sync_rq, which is not executed.
The process for sending scsi commands is as follows:
kworker/7:1-134 [007] ..... 32.772727: vp_notify
<-virtqueue_notify kworker/7:1-134 [007] ..... 32.772729:
<stack trace> => vp_notify => virtqueue_notify
=> virtscsi_add_cmd
=> virtscsi_queuecommand
=> scsi_dispatch_cmd
=> scsi_queue_rq
=> blk_mq_dispatch_rq_list
=> __blk_mq_sched_dispatch_requests
=> blk_mq_sched_dispatch_requests
=> blk_mq_run_hw_queue
=> blk_execute_rq
=> scsi_execute_cmd
=> sr_check_events
=> cdrom_check_events
=> sr_block_check_events
=> disk_check_events
=> disk_events_workfn
=> process_one_work
=> worker_thread
=> kthread
=> ret_from_fork
In qemu:
static void virtio_scsi_handle_cmd_vq(VirtIOSCSI *s, VirtQueue *vq)
{
...
do {
if (suppress_notifications) {
virtio_queue_set_notification(vq, 0);
}
...
} while (ret != -EINVAL && !virtio_queue_empty(vq));
...
QTAILQ_FOREACH_SAFE(req, &reqs, next, next) {
virtio_scsi_handle_cmd_req_submit(s, req);
}
}virtio_queue_empty() always return true.
As a result, the virtio_scsi_handle_cmd_req_submit() was not executed, and the io command was never submitted.
The reason is that virtio_device_disabled returns true in virtio_queue_split_empty, the code is as follows:
int virtio_queue_empty(VirtQueue *vq)
{
if (virtio_vdev_has_feature(vq->vdev, VIRTIO_F_RING_PACKED)) {
return virtio_queue_packed_empty(vq);
} else {
return virtio_queue_split_empty(vq);
}
}static int virtio_queue_split_empty(VirtQueue *vq)
{
bool empty;
if (virtio_device_disabled(vq->vdev)) {
return 1;
...
}This function was introduced in the following patch:
commit 9d7bd0826f2d19f88631ad7078662668148f7b5f
Author: Michael Roth <[email protected]> Date: Tue Nov 19 18:50:03 2019 -0600
virtio-pci: disable vring processing when bus-mastering is disabled
Currently the SLOF firmware for pseries guests will
disable/re-enable a PCI device multiple times via IO/MEM/MASTER
bits of PCI_COMMAND register after the initial probe/feature
negotiation, as it tends to work with a single device at a time at
various stages like probing and running block/network bootloaders
without doing a full reset in-between.
In QEMU, when PCI_COMMAND_MASTER is disabled we disable the
corresponding IOMMU memory region, so DMA accesses (including to
vring fields like idx/flags) will no longer undergo the necessary
translation. Normally we wouldn't expect this to happen since it
would be misbehavior on the driver side to continue driving DMA
requests.
However, in the case of pseries, with iommu_platform=on, we trigger
the following sequence when tearing down the virtio-blk dataplane
ioeventfd in response to the guest unsetting PCI_COMMAND_MASTER:
#2 0x0000555555922651 in virtqueue_map_desc
(vdev=vdev@entry=0x555556dbcfb0,
p_num_sg=p_num_sg@entry=0x7fffe657e1a8,
addr=addr@entry=0x7fffe657e240, iov=iov@entry=0x7fffe6580240,
max_num_sg=max_num_sg@entry=1024, is_write=is_write@entry=false,
pa=0, sz=0) at /home/mdroth/w/qemu.git/hw/virtio/virtio.c:757 #3
0x0000555555922a89 in virtqueue_pop (vq=vq@entry=0x555556dc8660,
sz=sz@entry=184) at /home/mdroth/w/qemu.git/hw/virtio/virtio.c:950
#4 0x00005555558d3eca in virtio_blk_get_request
(vq=0x555556dc8660, s=0x555556dbcfb0)
at /home/mdroth/w/qemu.git/hw/block/virtio-blk.c:255 #5
0x00005555558d3eca in virtio_blk_handle_vq (s=0x555556dbcfb0,
vq=0x555556dc8660)
at /home/mdroth/w/qemu.git/hw/block/virtio-blk.c:776 #6
0x000055555591dd66 in virtio_queue_notify_aio_vq
(vq=vq@entry=0x555556dc8660)
at /home/mdroth/w/qemu.git/hw/virtio/virtio.c:1550 #7
0x000055555591ecef in virtio_queue_notify_aio_vq
(vq=0x555556dc8660)
at /home/mdroth/w/qemu.git/hw/virtio/virtio.c:1546 #8
0x000055555591ecef in virtio_queue_host_notifier_aio_poll
(opaque=0x555556dc86c8)
at /home/mdroth/w/qemu.git/hw/virtio/virtio.c:2527 #9
0x0000555555d02164 in run_poll_handlers_once
(ctx=ctx@entry=0x55555688bfc0,
timeout=timeout@entry=0x7fffe65844a8)
at /home/mdroth/w/qemu.git/util/aio-posix.c:520 #10
0x0000555555d02d1b in try_poll_mode (timeout=0x7fffe65844a8,
ctx=0x55555688bfc0) at /home/mdroth/w/qemu.git/util/aio-posix.c:607
#11 0x0000555555d02d1b in aio_poll (ctx=ctx@entry=0x55555688bfc0,
blocking=blocking@entry=true)
at /home/mdroth/w/qemu.git/util/aio-posix.c:639 #12
0x0000555555d0004d in aio_wait_bh_oneshot (ctx=0x55555688bfc0,
cb=cb@entry=0x5555558d5130 <virtio_blk_data_plane_stop_bh>,
opaque=opaque@entry=0x555556de86f0)
at /home/mdroth/w/qemu.git/util/aio-wait.c:71 #13
0x00005555558d59bf in virtio_blk_data_plane_stop (vdev=<optimized
out>)
at /home/mdroth/w/qemu.git/hw/block/dataplane/virtio-blk.c:288 #14
0x0000555555b906a1 in virtio_bus_stop_ioeventfd
(bus=bus@entry=0x555556dbcf38)
at /home/mdroth/w/qemu.git/hw/virtio/virtio-bus.c:245 #15
0x0000555555b90dbb in virtio_bus_stop_ioeventfd
(bus=bus@entry=0x555556dbcf38)
at /home/mdroth/w/qemu.git/hw/virtio/virtio-bus.c:237 #16
0x0000555555b92a8e in virtio_pci_stop_ioeventfd
(proxy=0x555556db4e40)
at /home/mdroth/w/qemu.git/hw/virtio/virtio-pci.c:292 #17
0x0000555555b92a8e in virtio_write_config (pci_dev=0x555556db4e40,
address=<optimized out>, val=1048832, len=<optimized out>)
at /home/mdroth/w/qemu.git/hw/virtio/virtio-pci.c:613 I.e. the
calling code is only scheduling a one-shot BH for
virtio_blk_data_plane_stop_bh, but somehow we end up trying to
process an additional virtqueue entry before we get there. This is
likely due to the following check in
virtio_queue_host_notifier_aio_poll: static bool
virtio_queue_host_notifier_aio_poll(void *opaque) { EventNotifier
*n = opaque; VirtQueue *vq = container_of(n, VirtQueue,
host_notifier); bool progress; if (!vq->vring.desc ||
virtio_queue_empty(vq)) { return false; } progress =
virtio_queue_notify_aio_vq(vq); namely the call to
virtio_queue_empty(). In this case, since no new requests have
actually been issued, shadow_avail_idx == last_avail_idx, so we
actually try to access the vring via vring_avail_idx() to get the
latest non-shadowed idx: int virtio_queue_empty(VirtQueue *vq)
{ bool empty; ... if (vq->shadow_avail_idx != vq->last_avail_idx)
{ return 0; } rcu_read_lock(); empty = vring_avail_idx(vq) ==
vq->last_avail_idx; rcu_read_unlock(); return empty;
but since the IOMMU region has been disabled we get a bogus value (0
usually), which causes virtio_queue_empty() to falsely report that
there are entries to be processed, which causes errors such as:
"virtio: zero sized buffers are not allowed"
or
"virtio-blk missing headers"
and puts the device in an error state.
This patch works around the issue by introducing
virtio_set_disabled(), which sets a 'disabled' flag to bypass
checks like virtio_queue_empty() when bus-mastering is disabled.
Since we'd check this flag at all the same sites as vdev->broken,
we replace those checks with an inline function which checks for
either vdev->broken or vdev->disabled.
The 'disabled' flag is only migrated when set, which should be
fairly rare, but to maintain migration compatibility we disable
it's use for older machine types. Users requiring the use of the
flag in conjunction with older machine types can set it explicitly
as a virtio-device option.
NOTES:
- This leaves some other oddities in play, like the fact that
DRIVER_OK also gets unset in response to bus-mastering being
disabled, but not restored (however the device seems to continue
working)
- Similarly, we disable the host notifier via
virtio_bus_stop_ioeventfd(), which seems to move the handling out
of virtio-blk dataplane and back into the main IO thread, and it
ends up staying there till a reset (but otherwise continues
working normally)
Cc: David Gibson <[email protected]>,
Cc: Alexey Kardashevskiy <[email protected]>
Cc: "Michael S. Tsirkin" <[email protected]>
Signed-off-by: Michael Roth <[email protected]>
Message-Id: <[email protected]>
Reviewed-by: Michael S. Tsirkin <[email protected]>
Signed-off-by: Michael S. Tsirkin <[email protected]>Quoted 4 lines
Thanks,
Ming
Quoted 258 lines · Qiang Ma
Quoted 14 lines · On Wed, 27 Nov 2024 23:20:28 +0800 Ming Lei <[email protected]> wrote:
Quoted 13 lines · Qiang Ma
Quoted 10 lines · On Tue, 26 Nov 2024 22:15:02 +0800 Ming Lei <[email protected]> wrote:
Quoted 9 lines · Qiang Ma
Problem:
When the system disk uses the scsi disk bus, The main qemu command line includes:
...(quoted diff) +0
-3-device virtio-scsi-pci,id=scsi0 \ -device scsi-hd,scsi-id=1,drive=drive-virtio-disk -drive id=drive-virtio-disk,if=none,file=/home/kvm/test.qcow2Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
The dmesg log is as follows::
[ 50.304591][ T4382] sd 0:0:0:0: [sda] Synchronizing SCSIcache [ 50.377002][ T4382]
kexec_core: Starting new kernelWhy is one new kernel started? what event triggers
kernel_kexec()?
machine_shutdown()follows, probably the scsi controller is dead.Yes, triggered by kexec, manually execute the following command:
kexec -l/boot/vmlinuz-6.6.0+--reuse-cmdline --initrd=/boot/initramfs-6.6.0+.img kexec -eQuoted 126 lines
Quoted 126 lines
Quoted 87 lines
Quoted 76 lines
[ 50.669775][ T194] psci: CPU1 killed (polled 0 ms) [ 50.849665][ T194] psci: CPU2 killed (polled 0 ms) [ 51.109625][ T194] psci: CPU3 killed (polled 0 ms) [ 51.319594][ T194] psci: CPU4 killed (polled 0 ms) [ 51.489667][ T194] psci: CPU5 killed (polled 0 ms) [ 51.709582][ T194] psci: CPU6 killed (polled 0 ms) [ 51.949508][ T10] psci: CPU7 killed (polled 0 ms) [ 52.139499][ T10] psci: CPU8 killed (polled 0 ms) [ 52.289426][ T10] psci: CPU9 killed (polled 0 ms) [ 52.439552][ T10] psci: CPU10 killed (polled 0 ms) [ 52.579525][ T10] psci: CPU11 killed (polled 0 ms) [ 52.709501][ T10] psci: CPU12 killed (polled 0 ms) [ 52.819509][ T194] psci: CPU13 killed (polled 0 ms) [ 52.919509][ T194] psci: CPU14 killed (polled 0 ms) [ 243.214009][ T115] INFO: task kworker/0:1:10 blocked formore than 122 seconds. [ 243.214810][ T115] Not tainted
6.6.0+ #1 [ 243.215517][ T115] "echo 0Quoted 3 lines
/proc/sys/kernel/hung_task_timeout_secs" disables this
message. [ 243.216390][ T115] task:kworker/0:1 state:D
stack:0 pid:10 ppid:2 flags:0x00000008[ 243.217299][ T115] Workqueue: events vmstat_shepherd [ 243.217816][ T115] Call trace: [ 243.218133][ T115] __switch_to+0x130/0x1e8 [ 243.218568][ T115] __schedule+0x660/0xcf8 [ 243.219013][ T115] schedule+0x58/0xf0 [ 243.219402][ T115] percpu_rwsem_wait+0xb0/0x1d0 [ 243.219880][ T115] __percpu_down_read+0x40/0xe0 [ 243.220353][ T115] cpus_read_lock+0x5c/0x70 [ 243.220795][ T115] vmstat_shepherd+0x40/0x140 [ 243.221250][ T115] process_one_work+0x170/0x3c0 [ 243.221726][ T115] worker_thread+0x234/0x3b8 [ 243.222176][ T115] kthread+0xf0/0x108 [ 243.222564][ T115] ret_from_fork+0x10/0x20 ... [ 243.254080][ T115] INFO: task kworker/0:2:194 blocked formore than 122 seconds. [ 243.254834][ T115] Not tainted
6.6.0+ #1 [ 243.255529][ T115] "echo 0Quoted 3 lines
/proc/sys/kernel/hung_task_timeout_secs" disables this
message. [ 243.256378][ T115] task:kworker/0:2 state:D
stack:0 pid:194 ppid:2 flags:0x00000008[ 243.257284][ T115] Workqueue: events work_for_cpu_fn [ 243.257793][ T115] Call trace: [ 243.258111][ T115] __switch_to+0x130/0x1e8 [ 243.258541][ T115] __schedule+0x660/0xcf8 [ 243.258971][ T115] schedule+0x58/0xf0 [ 243.259360][ T115] schedule_timeout+0x280/0x2f0 [ 243.259832][ T115] wait_for_common+0xcc/0x2d8 [ 243.260287][ T115] wait_for_completion+0x20/0x38 [ 243.260767][ T115] cpuhp_kick_ap+0xe8/0x278 [ 243.261207][ T115] cpuhp_kick_ap_work+0x5c/0x188 [ 243.261688][ T115] _cpu_down+0x120/0x378 [ 243.262103][ T115] __cpu_down_maps_locked+0x20/0x38 [ 243.262609][ T115] work_for_cpu_fn+0x24/0x40 [ 243.263059][ T115] process_one_work+0x170/0x3c0 [ 243.263533][ T115] worker_thread+0x234/0x3b8 [ 243.263981][ T115] kthread+0xf0/0x108 [ 243.264405][ T115] ret_from_fork+0x10/0x20 [ 243.264846][ T115] INFO: task kworker/15:2:639 blocked formore than 122 seconds. [ 243.265602][ T115] Not tainted
6.6.0+ #1 [ 243.266296][ T115] "echo 0Quoted 3 lines
/proc/sys/kernel/hung_task_timeout_secs" disables this
message. [ 243.267143][ T115] task:kworker/15:2 state:D
stack:0 pid:639 ppid:2 flags:0x00000008[ 243.268044][ T115] Workqueue: events_freezable_power_ disk_events_workfn [ 243.268727][ T115] Call trace: [ 243.269051][ T115] __switch_to+0x130/0x1e8 [ 243.269481][ T115] __schedule+0x660/0xcf8 [ 243.269903][ T115] schedule+0x58/0xf0 [ 243.270293][ T115] schedule_timeout+0x280/0x2f0 [ 243.270763][ T115] io_schedule_timeout+0x50/0x70 [ 243.271245][ T115] wait_for_common_io.constprop.0+0xb0/0x298 [ 243.271830][ T115] wait_for_completion_io+0x1c/0x30 [ 243.272335][ T115] blk_execute_rq+0x1d8/0x278 [ 243.272793][ T115] scsi_execute_cmd+0x114/0x238 [ 243.273267][ T115] sr_check_events+0xc8/0x310 [sr_mod]Quoted 8 lines
[ 243.273808][ T115] cdrom_check_events+0x2c/0x50 [cdrom] [ 243.274408][ T115] sr_block_check_events+0x34/0x48 [sr_mod] [ 243.274994][ T115] disk_check_events+0x44/0x1b0 [ 243.275468][ T115] disk_events_workfn+0x20/0x38 [ 243.275939][ T115] process_one_work+0x170/0x3c0 [ 243.276410][ T115] worker_thread+0x234/0x3b8 [ 243.276855][ T115] kthread+0xf0/0x108 [ 243.277241][ T115] ret_from_fork+0x10/0x20Question is why this scsi command can't be completed?
Quoted 9 lines
When
blk_mq_hctx_notify_offline()is called, the CPU isn't shutdown yet, and it still can handle interrupt of this SCSI command.Thanks,
MingSorry, at the moment I don't know why it can't be finished, just provides a solution like loop and dm-rq.
Currently see the call stack:
=>blk_mq_run_hw_queue
=>__blk_mq_sched_restart
=>blk_mq_run_hw_queue
=>__blk_mq_sched_restart
=>__blk_mq_free_request
=>blk_mq_free_request
=>blk_mq_end_request
=>blk_flush_complete_seq
=>flush_end_io
=>__blk_mq_end_request
=>scsi_end_request
=>scsi_io_completion
=>scsi_finish_command
=>scsi_complete
=>blk_mq_complete_request
=>scsi_done_internal
=>scsi_done
=>virtscsi_complete_cmd
=>virtscsi_req_done
=>vring_interruptIn finction
blk_mq_run_hw_queue():
__blk_mq_run_dispatch_ops(hctx->queue, false,need_run = !blk_queue_quiesced(hctx->queue) && blk_mq_hctx_has_pending(hctx));Quoted 7 lines
Quoted 4 lines
if (!need_run) return;Come here and return directly.
Does
blk_queue_quiesced()return true?
blk_queue_quiesced()return falseSorry, the calltrace above is not the one that is stuck this time. The hang is caused by the
wait_for_completion_io(), inblk_execute_rq():
blk_status_tblk_execute_rq(struct request *rq, bool at_head){ ... rq->end_io_data = &wait; rq->end_io = blk_end_sync_rq; ... blk_account_io_start(rq); blk_mq_insert_request(rq, at_head ? BLK_MQ_INSERT_AT_HEAD : 0); blk_mq_run_hw_queue(hctx, false); if (blk_rq_is_poll(rq)) { blk_rq_poll_completion(rq, &wait.done); } else { ... wait_for_completion_io(&wait.done); } }In this case,
end_ioisblk_end_sync_rq, which is not executed.The process for sending scsi commands is as follows:
kworker/7:1-134 [007] ..... 32.772727: vp_notify <-virtqueue_notify kworker/7:1-134 [007] ..... 32.772729: <stack trace> => vp_notify=>
virtqueue_notify
=>virtscsi_add_cmd
=>virtscsi_queuecommand
=>scsi_dispatch_cmd
=>scsi_queue_rq
=>blk_mq_dispatch_rq_list
=>__blk_mq_sched_dispatch_requests
=>blk_mq_sched_dispatch_requests
=>blk_mq_run_hw_queue
=>blk_execute_rq
=>scsi_execute_cmd
=>sr_check_events
=>cdrom_check_events
=>sr_block_check_events
=>disk_check_events
=>disk_events_workfn
=>process_one_work
=>worker_thread
=> kthread
=>ret_from_forkIn qemu:
static voidvirtio_scsi_handle_cmd_vq(VirtIOSCSI *s, VirtQueue *vq){ ... do { if (suppress_notifications) { virtio_queue_set_notification(vq, 0); } ... } while (ret != -EINVAL && !virtio_queue_empty(vq)); ... QTAILQ_FOREACH_SAFE(req, &reqs, next, next) { virtio_scsi_handle_cmd_req_submit(s, req); } }
virtio_queue_empty()always return true.As a result, the
virtio_scsi_handle_cmd_req_submit()was not executed, and the io command was never submitted.The reason is that
virtio_device_disabledreturns true invirtio_queue_split_empty, the code is as follows:int
virtio_queue_empty(VirtQueue *vq){ if (virtio_vdev_has_feature(vq->vdev, VIRTIO_F_RING_PACKED)) { return virtio_queue_packed_empty(vq); } else { return virtio_queue_split_empty(vq); } }static int
virtio_queue_split_empty(VirtQueue *vq){ bool empty; if (virtio_device_disabled(vq->vdev)) { return 1; ... }
Quoted 7 lines
This function was introduced in the following patch:
commit
9d7bd0826f2d19f88631ad7078662668148f7b5fAuthor: Michael Roth <[email protected]> Date: Tue Nov 19 18:50:03 2019 -0600virtio-pci: disable vring processing when bus-mastering is disabled
OK, then it is obviously one qemu issue, and it can explain why IO isn't completed in this case.
Thanks,
Ming
Quoted 268 lines · On Tue, 3 Dec 2024 22:36:47 +0800 Ming Lei <[email protected]> wrote:
Quoted 267 lines · Qiang Ma
Quoted 16 lines · On Wed, 27 Nov 2024 23:20:28 +0800 Ming Lei <[email protected]> wrote:
Quoted 15 lines · Qiang Ma
Quoted 12 lines · On Tue, 26 Nov 2024 22:15:02 +0800 Ming Lei <[email protected]> wrote:
Quoted 11 lines · Qiang Ma
Problem:
When the system disk uses the scsi disk bus, The main qemu command line includes:
...(quoted diff) +0
-3-device virtio-scsi-pci,id=scsi0 \ -device scsi-hd,scsi-id=1,drive=drive-virtio-disk -driveid=drive-virtio-disk,if=none,file=/home/kvm/test.qcow2 ...
The dmesg log is as follows::
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
[ 50.304591][ T4382] sd 0:0:0:0: [sda] Synchronizing SCSIcache [ 50.377002][ T4382]
kexec_core: Starting new
kernelWhy is one new kernel started? what event triggers
kernel_kexec()?
machine_shutdown()follows, probably the scsi controller is dead.Yes, triggered by kexec, manually execute the following command:
kexec -l/boot/vmlinuz-6.6.0+--reuse-cmdline --initrd=/boot/initramfs-6.6.0+.img kexec -eQuoted 127 lines
Quoted 127 lines
Quoted 88 lines
Quoted 77 lines
[ 50.669775][ T194] psci: CPU1 killed (polled 0 ms) [ 50.849665][ T194] psci: CPU2 killed (polled 0 ms) [ 51.109625][ T194] psci: CPU3 killed (polled 0 ms) [ 51.319594][ T194] psci: CPU4 killed (polled 0 ms) [ 51.489667][ T194] psci: CPU5 killed (polled 0 ms) [ 51.709582][ T194] psci: CPU6 killed (polled 0 ms) [ 51.949508][ T10] psci: CPU7 killed (polled 0 ms) [ 52.139499][ T10] psci: CPU8 killed (polled 0 ms) [ 52.289426][ T10] psci: CPU9 killed (polled 0 ms) [ 52.439552][ T10] psci: CPU10 killed (polled 0 ms) [ 52.579525][ T10] psci: CPU11 killed (polled 0 ms) [ 52.709501][ T10] psci: CPU12 killed (polled 0 ms) [ 52.819509][ T194] psci: CPU13 killed (polled 0 ms) [ 52.919509][ T194] psci: CPU14 killed (polled 0 ms) [ 243.214009][ T115] INFO: task kworker/0:1:10 blocked formore than 122 seconds. [ 243.214810][ T115] Not
tainted 6.6.0+ #1 [ 243.215517][ T115] "echo 0Quoted 3 lines
/proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 243.216390][ T115] task:kworker/0:1
state:D stack:0 pid:10 ppid:2 flags:0x00000008[ 243.217299][ T115] Workqueue: events vmstat_shepherd [ 243.217816][ T115] Call trace: [ 243.218133][ T115] __switch_to+0x130/0x1e8 [ 243.218568][ T115] __schedule+0x660/0xcf8 [ 243.219013][ T115] schedule+0x58/0xf0 [ 243.219402][ T115] percpu_rwsem_wait+0xb0/0x1d0 [ 243.219880][ T115] __percpu_down_read+0x40/0xe0 [ 243.220353][ T115] cpus_read_lock+0x5c/0x70 [ 243.220795][ T115] vmstat_shepherd+0x40/0x140 [ 243.221250][ T115] process_one_work+0x170/0x3c0 [ 243.221726][ T115] worker_thread+0x234/0x3b8 [ 243.222176][ T115] kthread+0xf0/0x108 [ 243.222564][ T115] ret_from_fork+0x10/0x20 ... [ 243.254080][ T115] INFO: task kworker/0:2:194 blockedfor more than 122 seconds. [ 243.254834][ T115] Not
tainted 6.6.0+ #1 [ 243.255529][ T115] "echo 0Quoted 3 lines
/proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 243.256378][ T115] task:kworker/0:2
state:D stack:0 pid:194 ppid:2 flags:0x00000008[ 243.257284][ T115] Workqueue: events work_for_cpu_fn [ 243.257793][ T115] Call trace: [ 243.258111][ T115] __switch_to+0x130/0x1e8 [ 243.258541][ T115] __schedule+0x660/0xcf8 [ 243.258971][ T115] schedule+0x58/0xf0 [ 243.259360][ T115] schedule_timeout+0x280/0x2f0 [ 243.259832][ T115] wait_for_common+0xcc/0x2d8 [ 243.260287][ T115] wait_for_completion+0x20/0x38 [ 243.260767][ T115] cpuhp_kick_ap+0xe8/0x278 [ 243.261207][ T115] cpuhp_kick_ap_work+0x5c/0x188 [ 243.261688][ T115] _cpu_down+0x120/0x378 [ 243.262103][ T115] __cpu_down_maps_locked+0x20/0x38 [ 243.262609][ T115] work_for_cpu_fn+0x24/0x40 [ 243.263059][ T115] process_one_work+0x170/0x3c0 [ 243.263533][ T115] worker_thread+0x234/0x3b8 [ 243.263981][ T115] kthread+0xf0/0x108 [ 243.264405][ T115] ret_from_fork+0x10/0x20 [ 243.264846][ T115] INFO: task kworker/15:2:639 blockedfor more than 122 seconds. [ 243.265602][ T115] Not
tainted 6.6.0+ #1 [ 243.266296][ T115] "echo 0Quoted 3 lines
/proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 243.267143][ T115] task:kworker/15:2
state:D stack:0 pid:639 ppid:2 flags:0x00000008[ 243.268044][ T115] Workqueue: events_freezable_power_ disk_events_workfn [ 243.268727][ T115] Call trace: [ 243.269051][ T115] __switch_to+0x130/0x1e8 [ 243.269481][ T115] __schedule+0x660/0xcf8 [ 243.269903][ T115] schedule+0x58/0xf0 [ 243.270293][ T115] schedule_timeout+0x280/0x2f0 [ 243.270763][ T115] io_schedule_timeout+0x50/0x70 [ 243.271245][ T115] wait_for_common_io.constprop.0+0xb0/0x298 [ 243.271830][ T115] wait_for_completion_io+0x1c/0x30 [ 243.272335][ T115] blk_execute_rq+0x1d8/0x278 [ 243.272793][ T115] scsi_execute_cmd+0x114/0x238 [ 243.273267][ T115] sr_check_events+0xc8/0x310 [sr_mod] [ 243.273808][ T115] cdrom_check_events+0x2c/0x50 [cdrom]Quoted 8 lines
[ 243.274408][ T115] sr_block_check_events+0x34/0x48 [sr_mod] [ 243.274994][ T115] disk_check_events+0x44/0x1b0 [ 243.275468][ T115] disk_events_workfn+0x20/0x38 [ 243.275939][ T115] process_one_work+0x170/0x3c0 [ 243.276410][ T115] worker_thread+0x234/0x3b8 [ 243.276855][ T115] kthread+0xf0/0x108 [ 243.277241][ T115] ret_from_fork+0x10/0x20Question is why this scsi command can't be completed?
Quoted 9 lines
When
blk_mq_hctx_notify_offline()is called, the CPU isn't shutdown yet, and it still can handle interrupt of this SCSI command.Thanks,
MingSorry, at the moment I don't know why it can't be finished, just provides a solution like loop and dm-rq.
Currently see the call stack:
=>blk_mq_run_hw_queue
=>__blk_mq_sched_restart
=>blk_mq_run_hw_queue
=>__blk_mq_sched_restart
=>__blk_mq_free_request
=>blk_mq_free_request
=>blk_mq_end_request
=>blk_flush_complete_seq
=>flush_end_io
=>__blk_mq_end_request
=>scsi_end_request
=>scsi_io_completion
=>scsi_finish_command
=>scsi_complete
=>blk_mq_complete_request
=>scsi_done_internal
=>scsi_done
=>virtscsi_complete_cmd
=>virtscsi_req_done
=>vring_interruptIn finction
blk_mq_run_hw_queue():
__blk_mq_run_dispatch_ops(hctx->queue, false,need_run = !blk_queue_quiesced(hctx->queue) && blk_mq_hctx_has_pending(hctx));Quoted 7 lines
Quoted 4 lines
if (!need_run) return;Come here and return directly.
Does
blk_queue_quiesced()return true?
blk_queue_quiesced()return falseSorry, the calltrace above is not the one that is stuck this time. The hang is caused by the
wait_for_completion_io(), inblk_execute_rq():
blk_status_tblk_execute_rq(struct request *rq, bool at_head){ ... rq->end_io_data = &wait; rq->end_io = blk_end_sync_rq; ... blk_account_io_start(rq); blk_mq_insert_request(rq, at_head ? BLK_MQ_INSERT_AT_HEAD :0);
blk_mq_run_hw_queue(hctx, false);if (blk_rq_is_poll(rq)) { blk_rq_poll_completion(rq, &wait.done); } else { ... wait_for_completion_io(&wait.done); } }In this case,
end_ioisblk_end_sync_rq, which is not executed.The process for sending scsi commands is as follows:
kworker/7:1-134 [007] ..... 32.772727: vp_notify <-virtqueue_notify kworker/7:1-134 [007] .....32.772729: <stack trace> =>
vp_notify
=>virtqueue_notify
=>virtscsi_add_cmd
=>virtscsi_queuecommand
=>scsi_dispatch_cmd
=>scsi_queue_rq
=>blk_mq_dispatch_rq_list
=>__blk_mq_sched_dispatch_requests
=>blk_mq_sched_dispatch_requests
=>blk_mq_run_hw_queue
=>blk_execute_rq
=>scsi_execute_cmd
=>sr_check_events
=>cdrom_check_events
=>sr_block_check_events
=>disk_check_events
=>disk_events_workfn
=>process_one_work
=>worker_thread
=> kthread
=>ret_from_forkIn qemu:
static voidvirtio_scsi_handle_cmd_vq(VirtIOSCSI *s, VirtQueue *vq){ ... do { if (suppress_notifications) { virtio_queue_set_notification(vq, 0); } ... } while (ret != -EINVAL && !virtio_queue_empty(vq)); ... QTAILQ_FOREACH_SAFE(req, &reqs, next, next) { virtio_scsi_handle_cmd_req_submit(s, req); } }
virtio_queue_empty()always return true.As a result, the
virtio_scsi_handle_cmd_req_submit()was not executed, and the io command was never submitted.The reason is that
virtio_device_disabledreturns true invirtio_queue_split_empty, the code is as follows:int
virtio_queue_empty(VirtQueue *vq){ if (virtio_vdev_has_feature(vq->vdev, VIRTIO_F_RING_PACKED)) { return virtio_queue_packed_empty(vq); } else { return virtio_queue_split_empty(vq); } }static int
virtio_queue_split_empty(VirtQueue *vq){ bool empty; if (virtio_device_disabled(vq->vdev)) { return 1; ... }This function was introduced in the following patch:
commit
9d7bd0826f2d19f88631ad7078662668148f7b5fAuthor: Michael Roth <[email protected]> Date: Tue Nov 19 18:50:03 2019 -0600
Quoted 7 lines
Quoted 2 lines
virtio-pci: disable vring processing when bus-mastering is
disabledOK, then it is obviously one qemu issue, and it can explain why IO isn't completed in this case.
However, in fact, the current pci-master state is disabled, it is shutdown by executing the syscore_shutdown() in kernel_kexec(), and the call stack is as follows:kernel_kexec
=> syscore_shutdown
=> ops->shutdown
=> pci_device_shutdown
=> pci_clear_master
=> __pci_set_master
Therefore, qemu determines the pci-master status and disable virtio-pci when tthe state of the pci-master is disabled.
static void virtio_write_config(PCIDevice *pci_dev, uint32_t address,
uint32_t val, int len)
{
...
if (!(pci_dev->config[PCI_COMMAND] & PCI_COMMAND_MASTER)) {
virtio_set_disabled(vdev, true);
virtio_pci_stop_ioeventfd(proxy);
virtio_set_status(vdev,
vdev->status&~VIRTIO_CONFIG_S_DRIVER_OK);
} else {
virtio_set_disabled(vdev, false);
}
...
}After opening pci_dbg of initcall_debug and pci.c, the kernel log is as follows:
[root@localhost ~]# [ 41.490716][ T8227] platform
regulatory.0: shutdown [ 41.491726][ T8227] sd 0:0:0:0: shutdown
[ 41.492471][ T8227] sd 0:0:0:0: [sda] Synchronizing SCSI cache [ 41.496945][ T8227] snd_aloop snd_aloop.0: shutdown [ 41.497885][ T8227] platform smccc_trng: shutdown [ 41.498797][ T8227] platform platform-framebuffer.0: shutdown [ 41.499874][ T8227] alarmtimer alarmtimer.0.auto: shutdown [ 41.500905][ T8227] usb usb2-port15: shutdown [ 41.501713][ T8227] usb usb2-port14: shutdown [ 41.502527][ T8227] usb usb2-port13: shutdown [ 41.503367][ T8227] usb usb2-port12: shutdown [ 41.504188][ T8227] usb usb2-port11: shutdown [ 41.505002][ T8227] usb usb2-port10: shutdown [ 41.505816][ T8227] usb usb2-port9: shutdown [ 41.506615][ T8227] usb usb2-port8: shutdown [ 41.507409][ T8227] usb usb2-port7: shutdown [ 41.508205][ T8227] usb usb2-port6: shutdown [ 41.509053][ T8227] usb usb2-port5: shutdown [ 41.509853][ T8227] usb usb2-port4: shutdown [ 41.510652][ T8227] usb usb2-port3: shutdown [ 41.511448][ T8227] usb usb2-port2: shutdown [ 41.512249][ T8227] usb usb2-port1: shutdown [ 41.513053][ T8227] usb usb1-port15: shutdown [ 41.513836][ T8227] usb usb1-port14: shutdown [ 41.514622][ T8227] usb usb1-port13: shutdown [ 41.515408][ T8227] usb usb1-port12: shutdown [ 41.516191][ T8227] usb usb1-port11: shutdown [ 41.516970][ T8227] usb usb1-port10: shutdown [ 41.517754][ T8227] usb usb1-port9: shutdown [ 41.518519][ T8227] usb usb1-port8: shutdown [ 41.519341][ T8227] usb usb1-port7: shutdown [ 41.520102][ T8227] usb usb1-port6: shutdown [ 41.520868][ T8227] usb usb1-port5: shutdown [ 41.521630][ T8227] usb usb1-port4: shutdown [ 41.522396][ T8227] usb usb1-port3: shutdown [ 41.523169][ T8227] usb usb1-port2: shutdown [ 41.523929][ T8227] usb usb1-port1: shutdown [ 41.524698][ T8227] platform Fixed MDIO bus.0: shutdown [ 41.525697][ T8227] kgdboc kgdboc: shutdown [ 41.526451][ T8227] serial8250 serial8250: shutdown [ 41.527335][ T8227] pciehp 0000:00:01.7:pcie004: shutdown [ 41.528295][ T8227] aer 0000:00:01.7:pcie002: shutdown [ 41.529274][ T8227] pcie_pme 0000:00:01.7:pcie001: shutdown [ 41.530278][ T8227] pciehp 0000:00:01.6:pcie004: shutdown [ 41.531243][ T8227] aer 0000:00:01.6:pcie002: shutdown [ 41.532161][ T8227] pcie_pme 0000:00:01.6:pcie001: shutdown [ 41.533164][ T8227] pciehp 0000:00:01.5:pcie004: shutdown [ 41.534132][ T8227] aer 0000:00:01.5:pcie002: shutdown [ 41.535052][ T8227] pcie_pme 0000:00:01.5:pcie001: shutdown [ 41.536051][ T8227] pciehp 0000:00:01.4:pcie004: shutdown [ 41.537011][ T8227] aer 0000:00:01.4:pcie002: shutdown [ 41.537928][ T8227] pcie_pme 0000:00:01.4:pcie001: shutdown [ 41.538971][ T8227] pciehp 0000:00:01.3:pcie004: shutdown [ 41.539939][ T8227] aer 0000:00:01.3:pcie002: shutdown [ 41.540855][ T8227] pcie_pme 0000:00:01.3:pcie001: shutdown [ 41.541847][ T8227] pciehp 0000:00:01.2:pcie004: shutdown [ 41.542823][ T8227] aer 0000:00:01.2:pcie002: shutdown [ 41.543743][ T8227] pcie_pme 0000:00:01.2:pcie001: shutdown [ 41.544742][ T8227] pciehp 0000:00:01.1:pcie004: shutdown [ 41.545705][ T8227] aer 0000:00:01.1:pcie002: shutdown [ 41.546625][ T8227] pcie_pme 0000:00:01.1:pcie001: shutdown [ 41.547616][ T8227] pciehp 0000:00:01.0:pcie004: shutdown [ 41.548579][ T8227] aer 0000:00:01.0:pcie002: shutdown [ 41.549571][ T8227] pcie_pme 0000:00:01.0:pcie001: shutdown [ 41.550645][ T8227] system 00:00: shutdown [ 41.551381][ T8227] platform efivars.0: shutdown [ 41.552208][ T8227] rtc-efi rtc-efi.0: shutdown [ 41.553029][ T8227] platform PNP0C0C:00: shutdown [ 41.553865][ T8227] acpi-ged ACPI0013:00: shutdown [ 41.554829][ T8227] ahci 0000:09:01.0: shutdown [ 41.555911][ T8227] ahci 0000:09:01.0: disabling bus mastering [ 41.557683][ T8227] shpchp 0000:08:00.0: shutdown [ 41.558555][ T8227] shpchp 0000:08:00.0: disabling bus mastering [ 41.560917][ T8227] xhci_hcd 0000:07:00.0: shutdown [ 41.563837][ T8227] virtio-pci 0000:06:00.0: shutdown [ 41.564769][ T8227] virtio-pci 0000:06:00.0: disabling bus mastering [ 41.566490][ T8227] virtio-pci 0000:05:00.0: shutdown [ 41.567436][ T8227] virtio-pci 0000:05:00.0: disabling bus mastering [ 41.569188][ T8227] virtio-pci 0000:04:00.0: shutdown [ 41.570126][ T8227] virtio-pci 0000:04:00.0: disabling bus mastering [ 41.571903][ T8227] virtio-pci 0000:03:00.0: shutdown [ 41.572856][ T8227] virtio-pci 0000:03:00.0: disabling bus mastering [ 41.576969][ T8227] virtio-pci 0000:02:00.0: shutdown [ 41.578341][ T8227] virtio-pci 0000:02:00.0: disabling bus mastering [ 41.902816][ T8227] virtio-pci 0000:01:00.0: shutdown [ 41.905600][ T8227] virtio-pci 0000:01:00.0: disabling bus mastering [ 41.942715][ T8227] pcieport 0000:00:01.7: shutdown [ 41.944723][ T8227] pcieport 0000:00:01.7: disabling bus mastering [ 41.946667][ T8227] pcieport 0000:00:01.6: shutdown [ 41.948481][ T8227] pcieport 0000:00:01.6: disabling bus mastering [ 41.950693][ T8227] pcieport 0000:00:01.5: shutdown [ 41.953068][ T8227] pcieport 0000:00:01.5: disabling bus mastering [ 41.955722][ T8227] pcieport 0000:00:01.4: shutdown [ 41.957888][ T8227] pcieport 0000:00:01.4: disabling bus mastering [ 41.960272][ T8227] pcieport 0000:00:01.3: shutdown [ 41.962422][ T8227] pcieport 0000:00:01.3: disabling bus mastering [ 41.964726][ T8227] pcieport 0000:00:01.2: shutdown [ 41.967348][ T8227] pcieport 0000:00:01.2: disabling bus mastering [ 41.969762][ T8227] pcieport 0000:00:01.1: shutdown [ 41.972004][ T8227] pcieport 0000:00:01.1: disabling bus mastering [ 41.974365][ T8227] pcieport 0000:00:01.0: shutdown [ 41.977924][ T8227] pcieport 0000:00:01.0: disabling bus mastering [ 41.979974][ T8227] pci 0000:00:00.0: shutdown [ 41.980836][ T8227] platform LNRO0005:1f: shutdown [ 41.981739][ T8227] platform LNRO0005:1e: shutdown [ 41.982654][ T8227] platform LNRO0005:1d: shutdown [ 41.983555][ T8227] platform LNRO0005:1c: shutdown [ 41.984454][ T8227] platform LNRO0005:1b: shutdown [ 41.985523][ T8227] platform LNRO0005:1a: shutdown [ 41.986416][ T8227] platform LNRO0005:19: shutdown [ 41.987310][ T8227] platform LNRO0005:18: shutdown [ 41.988205][ T8227] platform LNRO0005:17: shutdown [ 41.989159][ T8227] platform LNRO0005:16: shutdown [ 41.990059][ T8227] platform LNRO0005:15: shutdown [ 41.990956][ T8227] platform LNRO0005:14: shutdown [ 41.991848][ T8227] platform LNRO0005:13: shutdown [ 41.992753][ T8227] platform LNRO0005:12: shutdown [ 41.993638][ T8227] platform LNRO0005:11: shutdown [ 41.994517][ T8227] platform LNRO0005:10: shutdown [ 41.995396][ T8227] platform LNRO0005:0f: shutdown [ 41.996266][ T8227] platform LNRO0005:0e: shutdown [ 41.997135][ T8227] platform LNRO0005:0d: shutdown [ 41.998002][ T8227] platform LNRO0005:0c: shutdown [ 41.998901][ T8227] platform LNRO0005:0b: shutdown [ 41.999766][ T8227] platform LNRO0005:0a: shutdown [ 42.000629][ T8227] platform LNRO0005:09: shutdown [ 42.001490][ T8227] platform LNRO0005:08: shutdown [ 42.002351][ T8227] platform LNRO0005:07: shutdown [ 42.003224][ T8227] platform LNRO0005:06: shutdown [ 42.004087][ T8227] platform LNRO0005:05: shutdown [ 42.004950][ T8227] platform LNRO0005:04: shutdown [ 42.005812][ T8227] platform LNRO0005:03: shutdown [ 42.006675][ T8227] platform LNRO0005:02: shutdown [ 42.007537][ T8227] platform LNRO0005:01: shutdown [ 42.008401][ T8227] platform LNRO0005:00: shutdown [ 42.009320][ T8227] fw_cfg QEMU0002:00: shutdown [ 42.010159][ T8227] sbsa-uart ARMH0011:00: shutdown [ 42.011097][ T8227] reg-dummy reg-dummy: shutdown [ 42.012042][ T8227] kexec_core: Starting new kernel [ 42.252819][ T10] psci: CPU1 killed (polled 0 ms) [ 42.352823][ T304] psci: CPU2 killed (polled 0 ms) [ 42.442807][ T304] psci: CPU3 killed (polled 0 ms) [ 42.532842][ T304] psci: CPU4 killed (polled 0 ms) [ 42.652807][ T304] psci: CPU5 killed (polled 0 ms) [ 42.759163][ T304] psci: CPU6 killed (polled 0 ms) [ 42.839104][ T304] psci: CPU7 killed (polled 0 ms)
Quoted 5 lines
Thanks,
Ming
Quoted 289 lines · Qiang Ma
Quoted 268 lines · On Tue, 3 Dec 2024 22:36:47 +0800 Ming Lei <[email protected]> wrote:
Quoted 267 lines · Qiang Ma
Quoted 16 lines · On Wed, 27 Nov 2024 23:20:28 +0800 Ming Lei <[email protected]> wrote:
Quoted 15 lines · Qiang Ma
Quoted 12 lines · On Tue, 26 Nov 2024 22:15:02 +0800 Ming Lei <[email protected]> wrote:
Quoted 11 lines · Qiang Ma
Problem:
When the system disk uses the scsi disk bus, The main qemu command line includes:
...(quoted diff) +0
-3-device virtio-scsi-pci,id=scsi0 \ -device scsi-hd,scsi-id=1,drive=drive-virtio-disk -driveid=drive-virtio-disk,if=none,file=/home/kvm/test.qcow2 ...
The dmesg log is as follows::
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
[ 50.304591][ T4382] sd 0:0:0:0: [sda] Synchronizing SCSIcache [ 50.377002][ T4382]
kexec_core: Starting new
kernelWhy is one new kernel started? what event triggers
kernel_kexec()?
machine_shutdown()follows, probably the scsi controller is dead.Yes, triggered by kexec, manually execute the following command:
kexec -l/boot/vmlinuz-6.6.0+--reuse-cmdline --initrd=/boot/initramfs-6.6.0+.img kexec -eQuoted 127 lines
Quoted 127 lines
Quoted 88 lines
Quoted 77 lines
[ 50.669775][ T194] psci: CPU1 killed (polled 0 ms) [ 50.849665][ T194] psci: CPU2 killed (polled 0 ms) [ 51.109625][ T194] psci: CPU3 killed (polled 0 ms) [ 51.319594][ T194] psci: CPU4 killed (polled 0 ms) [ 51.489667][ T194] psci: CPU5 killed (polled 0 ms) [ 51.709582][ T194] psci: CPU6 killed (polled 0 ms) [ 51.949508][ T10] psci: CPU7 killed (polled 0 ms) [ 52.139499][ T10] psci: CPU8 killed (polled 0 ms) [ 52.289426][ T10] psci: CPU9 killed (polled 0 ms) [ 52.439552][ T10] psci: CPU10 killed (polled 0 ms) [ 52.579525][ T10] psci: CPU11 killed (polled 0 ms) [ 52.709501][ T10] psci: CPU12 killed (polled 0 ms) [ 52.819509][ T194] psci: CPU13 killed (polled 0 ms) [ 52.919509][ T194] psci: CPU14 killed (polled 0 ms) [ 243.214009][ T115] INFO: task kworker/0:1:10 blocked formore than 122 seconds. [ 243.214810][ T115] Not
tainted 6.6.0+ #1 [ 243.215517][ T115] "echo 0Quoted 3 lines
/proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 243.216390][ T115] task:kworker/0:1
state:D stack:0 pid:10 ppid:2 flags:0x00000008[ 243.217299][ T115] Workqueue: events vmstat_shepherd [ 243.217816][ T115] Call trace: [ 243.218133][ T115] __switch_to+0x130/0x1e8 [ 243.218568][ T115] __schedule+0x660/0xcf8 [ 243.219013][ T115] schedule+0x58/0xf0 [ 243.219402][ T115] percpu_rwsem_wait+0xb0/0x1d0 [ 243.219880][ T115] __percpu_down_read+0x40/0xe0 [ 243.220353][ T115] cpus_read_lock+0x5c/0x70 [ 243.220795][ T115] vmstat_shepherd+0x40/0x140 [ 243.221250][ T115] process_one_work+0x170/0x3c0 [ 243.221726][ T115] worker_thread+0x234/0x3b8 [ 243.222176][ T115] kthread+0xf0/0x108 [ 243.222564][ T115] ret_from_fork+0x10/0x20 ... [ 243.254080][ T115] INFO: task kworker/0:2:194 blockedfor more than 122 seconds. [ 243.254834][ T115] Not
tainted 6.6.0+ #1 [ 243.255529][ T115] "echo 0Quoted 3 lines
/proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 243.256378][ T115] task:kworker/0:2
state:D stack:0 pid:194 ppid:2 flags:0x00000008[ 243.257284][ T115] Workqueue: events work_for_cpu_fn [ 243.257793][ T115] Call trace: [ 243.258111][ T115] __switch_to+0x130/0x1e8 [ 243.258541][ T115] __schedule+0x660/0xcf8 [ 243.258971][ T115] schedule+0x58/0xf0 [ 243.259360][ T115] schedule_timeout+0x280/0x2f0 [ 243.259832][ T115] wait_for_common+0xcc/0x2d8 [ 243.260287][ T115] wait_for_completion+0x20/0x38 [ 243.260767][ T115] cpuhp_kick_ap+0xe8/0x278 [ 243.261207][ T115] cpuhp_kick_ap_work+0x5c/0x188 [ 243.261688][ T115] _cpu_down+0x120/0x378 [ 243.262103][ T115] __cpu_down_maps_locked+0x20/0x38 [ 243.262609][ T115] work_for_cpu_fn+0x24/0x40 [ 243.263059][ T115] process_one_work+0x170/0x3c0 [ 243.263533][ T115] worker_thread+0x234/0x3b8 [ 243.263981][ T115] kthread+0xf0/0x108 [ 243.264405][ T115] ret_from_fork+0x10/0x20 [ 243.264846][ T115] INFO: task kworker/15:2:639 blockedfor more than 122 seconds. [ 243.265602][ T115] Not
tainted 6.6.0+ #1 [ 243.266296][ T115] "echo 0Quoted 3 lines
/proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 243.267143][ T115] task:kworker/15:2
state:D stack:0 pid:639 ppid:2 flags:0x00000008[ 243.268044][ T115] Workqueue: events_freezable_power_ disk_events_workfn [ 243.268727][ T115] Call trace: [ 243.269051][ T115] __switch_to+0x130/0x1e8 [ 243.269481][ T115] __schedule+0x660/0xcf8 [ 243.269903][ T115] schedule+0x58/0xf0 [ 243.270293][ T115] schedule_timeout+0x280/0x2f0 [ 243.270763][ T115] io_schedule_timeout+0x50/0x70 [ 243.271245][ T115] wait_for_common_io.constprop.0+0xb0/0x298 [ 243.271830][ T115] wait_for_completion_io+0x1c/0x30 [ 243.272335][ T115] blk_execute_rq+0x1d8/0x278 [ 243.272793][ T115] scsi_execute_cmd+0x114/0x238 [ 243.273267][ T115] sr_check_events+0xc8/0x310 [sr_mod] [ 243.273808][ T115] cdrom_check_events+0x2c/0x50 [cdrom]Quoted 8 lines
[ 243.274408][ T115] sr_block_check_events+0x34/0x48 [sr_mod] [ 243.274994][ T115] disk_check_events+0x44/0x1b0 [ 243.275468][ T115] disk_events_workfn+0x20/0x38 [ 243.275939][ T115] process_one_work+0x170/0x3c0 [ 243.276410][ T115] worker_thread+0x234/0x3b8 [ 243.276855][ T115] kthread+0xf0/0x108 [ 243.277241][ T115] ret_from_fork+0x10/0x20Question is why this scsi command can't be completed?
Quoted 9 lines
When
blk_mq_hctx_notify_offline()is called, the CPU isn't shutdown yet, and it still can handle interrupt of this SCSI command.Thanks,
MingSorry, at the moment I don't know why it can't be finished, just provides a solution like loop and dm-rq.
Currently see the call stack:
=>blk_mq_run_hw_queue
=>__blk_mq_sched_restart
=>blk_mq_run_hw_queue
=>__blk_mq_sched_restart
=>__blk_mq_free_request
=>blk_mq_free_request
=>blk_mq_end_request
=>blk_flush_complete_seq
=>flush_end_io
=>__blk_mq_end_request
=>scsi_end_request
=>scsi_io_completion
=>scsi_finish_command
=>scsi_complete
=>blk_mq_complete_request
=>scsi_done_internal
=>scsi_done
=>virtscsi_complete_cmd
=>virtscsi_req_done
=>vring_interruptIn finction
blk_mq_run_hw_queue():
__blk_mq_run_dispatch_ops(hctx->queue, false,need_run = !blk_queue_quiesced(hctx->queue) && blk_mq_hctx_has_pending(hctx));Quoted 7 lines
Quoted 4 lines
if (!need_run) return;Come here and return directly.
Does
blk_queue_quiesced()return true?
blk_queue_quiesced()return falseSorry, the calltrace above is not the one that is stuck this time. The hang is caused by the
wait_for_completion_io(), inblk_execute_rq():
blk_status_tblk_execute_rq(struct request *rq, bool at_head){ ... rq->end_io_data = &wait; rq->end_io = blk_end_sync_rq; ... blk_account_io_start(rq); blk_mq_insert_request(rq, at_head ? BLK_MQ_INSERT_AT_HEAD :0);
blk_mq_run_hw_queue(hctx, false);if (blk_rq_is_poll(rq)) { blk_rq_poll_completion(rq, &wait.done); } else { ... wait_for_completion_io(&wait.done); } }In this case,
end_ioisblk_end_sync_rq, which is not executed.The process for sending scsi commands is as follows:
kworker/7:1-134 [007] ..... 32.772727: vp_notify <-virtqueue_notify kworker/7:1-134 [007] .....32.772729: <stack trace> =>
vp_notify
=>virtqueue_notify
=>virtscsi_add_cmd
=>virtscsi_queuecommand
=>scsi_dispatch_cmd
=>scsi_queue_rq
=>blk_mq_dispatch_rq_list
=>__blk_mq_sched_dispatch_requests
=>blk_mq_sched_dispatch_requests
=>blk_mq_run_hw_queue
=>blk_execute_rq
=>scsi_execute_cmd
=>sr_check_events
=>cdrom_check_events
=>sr_block_check_events
=>disk_check_events
=>disk_events_workfn
=>process_one_work
=>worker_thread
=> kthread
=>ret_from_forkIn qemu:
static voidvirtio_scsi_handle_cmd_vq(VirtIOSCSI *s, VirtQueue *vq){ ... do { if (suppress_notifications) { virtio_queue_set_notification(vq, 0); } ... } while (ret != -EINVAL && !virtio_queue_empty(vq)); ... QTAILQ_FOREACH_SAFE(req, &reqs, next, next) { virtio_scsi_handle_cmd_req_submit(s, req); } }
virtio_queue_empty()always return true.As a result, the
virtio_scsi_handle_cmd_req_submit()was not executed, and the io command was never submitted.The reason is that
virtio_device_disabledreturns true invirtio_queue_split_empty, the code is as follows:int
virtio_queue_empty(VirtQueue *vq){ if (virtio_vdev_has_feature(vq->vdev, VIRTIO_F_RING_PACKED)) { return virtio_queue_packed_empty(vq); } else { return virtio_queue_split_empty(vq); } }static int
virtio_queue_split_empty(VirtQueue *vq){ bool empty; if (virtio_device_disabled(vq->vdev)) { return 1; ... }This function was introduced in the following patch:
commit
9d7bd0826f2d19f88631ad7078662668148f7b5fAuthor: Michael Roth <[email protected]> Date: Tue Nov 19 18:50:03 2019 -0600Quoted 7 lines
Quoted 2 lines
virtio-pci: disable vring processing when bus-mastering is
disabledOK, then it is obviously one qemu issue, and it can explain why IO isn't completed in this case.
However, in fact, the current pci-master state is disabled, it is shutdown by executing the
syscore_shutdown()inkernel_kexec(), and the call stack is as follows:kernel_kexec
=>syscore_shutdown
=>ops->shutdown
=>pci_device_shutdown
=>pci_clear_master
=>__pci_set_master
Quoted 2 lines
Therefore, qemu determines the pci-master status and disable virtio-pci when tthe state of the pci-master is disabled.
But either virtio-scsi driver or qemu knows clearly that there is one in-flight request not completed, and is just ignored silently.
Anyway, it is not one issue in block layer or scsi-core code.
Thanks,
Ming