virtio_net: Add timeout handler to avoid kernel hang
Subsystems (4)
PATCH 1/1 virtio_net: Add timeout handler to avoid kernel hang
From: Zhu Yanjun <[email protected]>
Some devices emulate the virtio_net hardwares. When virtio_net driver sends commands to the emulated hardware, normally the hardware needs time to response. Sometimes the time is very long. Thus, the following will appear. Then the whole system will hang.
The similar problems also occur in Intel NICs and Mellanox NICs. As such, the similar solution is borrowed from them. A timeout value is added and the timeout value as large as possible is set to ensure that the driver gets the maximum possible response from the hardware.
"
[ 213.795860] watchdog: BUG: soft lockup - CPU#108 stuck for 26s! [(udev-worker):3157] [ 213.796114] Modules linked in: virtio_net(+) net_failover failover qrtr rfkill sunrpc intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common intel_ifs i10nm_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt rapl intel_pmc_bxt dax_hmem iTCO_vendor_support vfat cxl_acpi intel_cstate pmt_telemetry pmt_class intel_sdsi joydev intel_uncore cxl_core fat pcspkr mei_me isst_if_mbox_pci isst_if_mmio idxd i2c_i801 isst_if_common mei intel_vsec idxd_bus i2c_smbus i2c_ismt ipmi_ssif acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter pfr_telemetry pfr_update fuse loop zram xfs crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 bnxt_en sha256_ssse3 sha1_ssse3 nvme ast nvme_core i2c_algo_bit wmi pinctrl_emmitsburg scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_multipath [ 213.796194] irq event stamp: 67740 [ 213.796195] hardirqs last enabled at (67739): [<ffffffff8c2015ca>] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 213.796203] hardirqs last disabled at (67740): [<ffffffff8c14108e>] sysvec_apic_timer_interrupt+0xe/0x90 [ 213.796208] softirqs last enabled at (67686): [<ffffffff8b12115e>] __irq_exit_rcu+0xbe/0xe0 [ 213.796214] softirqs last disabled at (67681): [<ffffffff8b12115e>] __irq_exit_rcu+0xbe/0xe0 [ 213.796217] CPU: 108 PID: 3157 Comm: (udev-worker) Kdump: loaded Not tainted 6.7.0+ #9 [ 213.796220] Hardware name: Intel Corporation M50FCP2SBSTD/M50FCP2SBSTD, BIOS SE5C741.86B.01.01.0001.2211140926 11/14/2022 [ 213.796221] RIP: 0010:virtqueue_get_buf_ctx_split+0x8d/0x110 [ 213.796228] Code: 89 df e8 26 fe ff ff 0f b7 43 50 83 c0 01 66 89 43 50 f6 43 78 01 75 12 80 7b 42 00 48 8b 4b 68 8b 53 58 74 0f 66 87 44 51 04 <48> 89 e8 5b 5d c3 cc cc cc cc 66 89 44 51 04 0f ae f0 48 89 e8 5b [ 213.796230] RSP: 0018:ff4bbb362306f9b0 EFLAGS: 00000246 [ 213.796233] RAX: 0000000000000000 RBX: ff2f15095896f000 RCX: 0000000000000001 [ 213.796235] RDX: 0000000000000000 RSI: ff4bbb362306f9cc RDI: ff2f15095896f000 [ 213.796236] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 213.796237] R10: 0000000000000003 R11: ff2f15095893cc40 R12: 0000000000000002 [ 213.796239] R13: 0000000000000004 R14: 0000000000000000 R15: ff2f1509534f3000 [ 213.796240] FS: 00007f775847d0c0(0000) GS:ff2f1528bac00000(0000) knlGS:0000000000000000 [ 213.796242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 213.796243] CR2: 0000557f987b6e70 CR3: 0000002098602006 CR4: 0000000000f71ef0 [ 213.796245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 213.796246] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 [ 213.796247] PKRU: 55555554 [ 213.796249] Call Trace: [ 213.796250] <IRQ> [ 213.796252] ? watchdog_timer_fn+0x1c0/0x220 [ 213.796258] ? __pfx_watchdog_timer_fn+0x10/0x10 [ 213.796261] ? __hrtimer_run_queues+0x1af/0x380 [ 213.796269] ? hrtimer_interrupt+0xf8/0x230 [ 213.796274] ? __sysvec_apic_timer_interrupt+0x64/0x1a0 [ 213.796279] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 213.796282] </IRQ> [ 213.796284] <TASK> [ 213.796285] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 213.796293] ? virtqueue_get_buf_ctx_split+0x8d/0x110 [ 213.796297] virtnet_send_command+0x18a/0x1f0 [virtio_net] [ 213.796310] _virtnet_set_queues+0xc6/0x120 [virtio_net] [ 213.796319] virtnet_probe+0xa06/0xd50 [virtio_net] [ 213.796328] virtio_dev_probe+0x195/0x230 [ 213.796333] really_probe+0x19f/0x400 [ 213.796338] ? __pfx___driver_attach+0x10/0x10 [ 213.796340] __driver_probe_device+0x78/0x160 [ 213.796343] driver_probe_device+0x1f/0x90 [ 213.796346] __driver_attach+0xd6/0x1d0 [ 213.796349] bus_for_each_dev+0x8c/0xe0 [ 213.796355] bus_add_driver+0x119/0x220 [ 213.796359] driver_register+0x59/0x100 [ 213.796362] ? __pfx_virtio_net_driver_init+0x10/0x10 [virtio_net] [ 213.796369] virtio_net_driver_init+0x8e/0xff0 [virtio_net] [ 213.796375] do_one_initcall+0x6f/0x380 [ 213.796384] do_init_module+0x60/0x240 [ 213.796388] init_module_from_file+0x86/0xc0 [ 213.796396] idempotent_init_module+0x129/0x2c0 [ 213.796406] __x64_sys_finit_module+0x5e/0xb0 [ 213.796409] do_syscall_64+0x60/0xe0 [ 213.796415] ? do_syscall_64+0x6f/0xe0 [ 213.796418] ? lockdep_hardirqs_on_prepare+0xe4/0x1a0 [ 213.796424] ? do_syscall_64+0x6f/0xe0 [ 213.796427] ? do_syscall_64+0x6f/0xe0 [ 213.796431] entry_SYSCALL_64_after_hwframe+0x6e/0x76 [ 213.796435] RIP: 0033:0x7f7758f279cd [ 213.796465] Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 33 e4 0c 00 f7 d8 64 89 01 48 [ 213.796467] RSP: 002b:00007ffe2cad8738 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 213.796469] RAX: ffffffffffffffda RBX: 0000557f987a8180 RCX: 00007f7758f279cd [ 213.796471] RDX: 0000000000000000 RSI: 00007f77593e5453 RDI: 000000000000000f [ 213.796472] RBP: 00007f77593e5453 R08: 0000000000000000 R09: 00007ffe2cad8860 [ 213.796473] R10: 000000000000000f R11: 0000000000000246 R12: 0000000000020000 [ 213.796475] R13: 0000557f9879f8e0 R14: 0000000000000000 R15: 0000557f98783aa0 [ 213.796482] </TASK>
"
--- drivers/net/virtio_net.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
drivers/net/virtio_net.c +8 -2
@@ -2468,7 +2468,7 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, {struct scatterlist *sgs[4 ], hdr, stat;unsigned out_num =0 , tmp; -int ret; +int ret, timeout =200 ;/* Caller should know better */ BUG_ON (!virtio_has_feature (vi->vdev,VIRTIO_NET_F_CTRL_VQ ));
@@ -2502,8 +2502,14 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +if (timeout) + timeout--; +else +return false;/* long time no response */ +cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; }
Signature · 1 line
2.42.0
Quoted 96 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
Some devices emulate the
virtio_nethardwares. Whenvirtio_netdriver sends commands to the emulated hardware, normally the hardware needs time to response. Sometimes the time is very long. Thus, the following will appear. Then the whole system will hang.
The similar problems also occur in Intel NICs and Mellanox NICs. As such, the similar solution is borrowed from them. A timeout value is added and the timeout value as large as possible is set to ensure that the driver gets the maximum possible response from the hardware."
[ 213.795860] watchdog: BUG: soft lockup - CPU#108 stuck for 26s! [(udev-worker):3157] [ 213.796114] Modules linked in: virtio_net(+) net_failover failover qrtr rfkill sunrpc intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common intel_ifs i10nm_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt rapl intel_pmc_bxt dax_hmem iTCO_vendor_support vfat cxl_acpi intel_cstate pmt_telemetry pmt_class intel_sdsi joydev intel_uncore cxl_core fat pcspkr mei_me isst_if_mbox_pci isst_if_mmio idxd i2c_i801 isst_if_common mei intel_vsec idxd_bus i2c_smbus i2c_ismt ipmi_ssif acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter pfr_telemetry pfr_update fuse loop zram xfs crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 bnxt_en sha256_ssse3 sha1_ssse3 nvme ast nvme_core i2c_algo_bit wmi pinctrl_emmitsburg scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_multipath [ 213.796194] irq event stamp: 67740 [ 213.796195] hardirqs last enabled at (67739): [<ffffffff8c2015ca>] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 213.796203] hardirqs last disabled at (67740): [<ffffffff8c14108e>] sysvec_apic_timer_interrupt+0xe/0x90 [ 213.796208] softirqs last enabled at (67686): [<ffffffff8b12115e>] __irq_exit_rcu+0xbe/0xe0 [ 213.796214] softirqs last disabled at (67681): [<ffffffff8b12115e>] __irq_exit_rcu+0xbe/0xe0 [ 213.796217] CPU: 108 PID: 3157 Comm: (udev-worker) Kdump: loaded Not tainted 6.7.0+ #9 [ 213.796220] Hardware name: Intel Corporation M50FCP2SBSTD/M50FCP2SBSTD, BIOS SE5C741.86B.01.01.0001.2211140926 11/14/2022 [ 213.796221] RIP: 0010:virtqueue_get_buf_ctx_split+0x8d/0x110 [ 213.796228] Code: 89 df e8 26 fe ff ff 0f b7 43 50 83 c0 01 66 89 43 50 f6 43 78 01 75 12 80 7b 42 00 48 8b 4b 68 8b 53 58 74 0f 66 87 44 51 04 <48> 89 e8 5b 5d c3 cc cc cc cc 66 89 44 51 04 0f ae f0 48 89 e8 5b [ 213.796230] RSP: 0018:ff4bbb362306f9b0 EFLAGS: 00000246 [ 213.796233] RAX: 0000000000000000 RBX: ff2f15095896f000 RCX: 0000000000000001 [ 213.796235] RDX: 0000000000000000 RSI: ff4bbb362306f9cc RDI: ff2f15095896f000 [ 213.796236] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 213.796237] R10: 0000000000000003 R11: ff2f15095893cc40 R12: 0000000000000002 [ 213.796239] R13: 0000000000000004 R14: 0000000000000000 R15: ff2f1509534f3000 [ 213.796240] FS: 00007f775847d0c0(0000) GS:ff2f1528bac00000(0000) knlGS:0000000000000000 [ 213.796242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 213.796243] CR2: 0000557f987b6e70 CR3: 0000002098602006 CR4: 0000000000f71ef0 [ 213.796245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 213.796246] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 [ 213.796247] PKRU: 55555554 [ 213.796249] Call Trace: [ 213.796250] <IRQ> [ 213.796252] ? watchdog_timer_fn+0x1c0/0x220 [ 213.796258] ? __pfx_watchdog_timer_fn+0x10/0x10 [ 213.796261] ? __hrtimer_run_queues+0x1af/0x380 [ 213.796269] ? hrtimer_interrupt+0xf8/0x230 [ 213.796274] ? __sysvec_apic_timer_interrupt+0x64/0x1a0 [ 213.796279] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 213.796282] </IRQ> [ 213.796284] <TASK> [ 213.796285] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 213.796293] ? virtqueue_get_buf_ctx_split+0x8d/0x110 [ 213.796297] virtnet_send_command+0x18a/0x1f0 [virtio_net] [ 213.796310] _virtnet_set_queues+0xc6/0x120 [virtio_net] [ 213.796319] virtnet_probe+0xa06/0xd50 [virtio_net] [ 213.796328] virtio_dev_probe+0x195/0x230 [ 213.796333] really_probe+0x19f/0x400 [ 213.796338] ? __pfx___driver_attach+0x10/0x10 [ 213.796340] __driver_probe_device+0x78/0x160 [ 213.796343] driver_probe_device+0x1f/0x90 [ 213.796346] __driver_attach+0xd6/0x1d0 [ 213.796349] bus_for_each_dev+0x8c/0xe0 [ 213.796355] bus_add_driver+0x119/0x220 [ 213.796359] driver_register+0x59/0x100 [ 213.796362] ? __pfx_virtio_net_driver_init+0x10/0x10 [virtio_net] [ 213.796369] virtio_net_driver_init+0x8e/0xff0 [virtio_net] [ 213.796375] do_one_initcall+0x6f/0x380 [ 213.796384] do_init_module+0x60/0x240 [ 213.796388] init_module_from_file+0x86/0xc0 [ 213.796396] idempotent_init_module+0x129/0x2c0 [ 213.796406] __x64_sys_finit_module+0x5e/0xb0 [ 213.796409] do_syscall_64+0x60/0xe0 [ 213.796415] ? do_syscall_64+0x6f/0xe0 [ 213.796418] ? lockdep_hardirqs_on_prepare+0xe4/0x1a0 [ 213.796424] ? do_syscall_64+0x6f/0xe0 [ 213.796427] ? do_syscall_64+0x6f/0xe0 [ 213.796431] entry_SYSCALL_64_after_hwframe+0x6e/0x76 [ 213.796435] RIP: 0033:0x7f7758f279cd [ 213.796465] Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 33 e4 0c 00 f7 d8 64 89 01 48 [ 213.796467] RSP: 002b:00007ffe2cad8738 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 213.796469] RAX: ffffffffffffffda RBX: 0000557f987a8180 RCX: 00007f7758f279cd [ 213.796471] RDX: 0000000000000000 RSI: 00007f77593e5453 RDI: 000000000000000f [ 213.796472] RBP: 00007f77593e5453 R08: 0000000000000000 R09: 00007ffe2cad8860 [ 213.796473] R10: 000000000000000f R11: 0000000000000246 R12: 0000000000020000 [ 213.796475] R13: 0000557f9879f8e0 R14: 0000000000000000 R15: 0000557f98783aa0 [ 213.796482] </TASK>"
Signed-off-by Zhu Yanjun <[email protected]>--- drivers/net/virtio_net.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)drivers/net/virtio_net.c +0
-0
drivers/net/virtio_net.c static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, · line 2471
Quoted 5 lines
drivers/net/virtio_net.c +1
-1@@ -2468,7 +2468,7 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, {struct scatterlist *sgs[4 ], hdr, stat;unsigned out_num =0 , tmp; -int ret; +int ret, timeout =200 ;
Any reason we choose this value or how can we know it works for all types of devices?
A more easy way is to use cond_resched() but it may have side effects as well[1]. But it seems less intrusive anyhow than the proposal here?
Thanks
[1] https://www.mail-archive.com/[email protected]/msg60297.html
Quoted 22 lines
drivers/net/virtio_net.c +7
-2/* Caller should know better */ BUG_ON (!virtio_has_feature (vi->vdev,VIRTIO_NET_F_CTRL_VQ ));@@ -2502,8 +2502,14 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +if (timeout) + timeout--; +else +return false;/* long time no response */ +cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.42.0
Quoted 91 lines · Jason Wang
Quoted 90 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
Some devices emulate the
virtio_nethardwares. Whenvirtio_netdriver sends commands to the emulated hardware, normally the hardware needs time to response. Sometimes the time is very long. Thus, the following will appear. Then the whole system will hang.
The similar problems also occur in Intel NICs and Mellanox NICs. As such, the similar solution is borrowed from them. A timeout value is added and the timeout value as large as possible is set to ensure that the driver gets the maximum possible response from the hardware."
[ 213.795860] watchdog: BUG: soft lockup - CPU#108 stuck for 26s! [(udev-worker):3157] [ 213.796114] Modules linked in: virtio_net(+) net_failover failover qrtr rfkill sunrpc intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common intel_ifs i10nm_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt rapl intel_pmc_bxt dax_hmem iTCO_vendor_support vfat cxl_acpi intel_cstate pmt_telemetry pmt_class intel_sdsi joydev intel_uncore cxl_core fat pcspkr mei_me isst_if_mbox_pci isst_if_mmio idxd i2c_i801 isst_if_common mei intel_vsec idxd_bus i2c_smbus i2c_ismt ipmi_ssif acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter pfr_telemetry pfr_update fuse loop zram xfs crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 bnxt_en sha256_ssse3 sha1_ssse3 nvme ast nvme_core i2c_algo_bit wmi pinctrl_emmitsburg scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_multipath [ 213.796194] irq event stamp: 67740 [ 213.796195] hardirqs last enabled at (67739): [<ffffffff8c2015ca>] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 213.796203] hardirqs last disabled at (67740): [<ffffffff8c14108e>] sysvec_apic_timer_interrupt+0xe/0x90 [ 213.796208] softirqs last enabled at (67686): [<ffffffff8b12115e>] __irq_exit_rcu+0xbe/0xe0 [ 213.796214] softirqs last disabled at (67681): [<ffffffff8b12115e>] __irq_exit_rcu+0xbe/0xe0 [ 213.796217] CPU: 108 PID: 3157 Comm: (udev-worker) Kdump: loaded Not tainted 6.7.0+ #9 [ 213.796220] Hardware name: Intel Corporation M50FCP2SBSTD/M50FCP2SBSTD, BIOS SE5C741.86B.01.01.0001.2211140926 11/14/2022 [ 213.796221] RIP: 0010:virtqueue_get_buf_ctx_split+0x8d/0x110 [ 213.796228] Code: 89 df e8 26 fe ff ff 0f b7 43 50 83 c0 01 66 89 43 50 f6 43 78 01 75 12 80 7b 42 00 48 8b 4b 68 8b 53 58 74 0f 66 87 44 51 04 <48> 89 e8 5b 5d c3 cc cc cc cc 66 89 44 51 04 0f ae f0 48 89 e8 5b [ 213.796230] RSP: 0018:ff4bbb362306f9b0 EFLAGS: 00000246 [ 213.796233] RAX: 0000000000000000 RBX: ff2f15095896f000 RCX: 0000000000000001 [ 213.796235] RDX: 0000000000000000 RSI: ff4bbb362306f9cc RDI: ff2f15095896f000 [ 213.796236] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 213.796237] R10: 0000000000000003 R11: ff2f15095893cc40 R12: 0000000000000002 [ 213.796239] R13: 0000000000000004 R14: 0000000000000000 R15: ff2f1509534f3000 [ 213.796240] FS: 00007f775847d0c0(0000) GS:ff2f1528bac00000(0000) knlGS:0000000000000000 [ 213.796242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 213.796243] CR2: 0000557f987b6e70 CR3: 0000002098602006 CR4: 0000000000f71ef0 [ 213.796245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 213.796246] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 [ 213.796247] PKRU: 55555554 [ 213.796249] Call Trace: [ 213.796250] <IRQ> [ 213.796252] ? watchdog_timer_fn+0x1c0/0x220 [ 213.796258] ? __pfx_watchdog_timer_fn+0x10/0x10 [ 213.796261] ? __hrtimer_run_queues+0x1af/0x380 [ 213.796269] ? hrtimer_interrupt+0xf8/0x230 [ 213.796274] ? __sysvec_apic_timer_interrupt+0x64/0x1a0 [ 213.796279] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 213.796282] </IRQ> [ 213.796284] <TASK> [ 213.796285] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 213.796293] ? virtqueue_get_buf_ctx_split+0x8d/0x110 [ 213.796297] virtnet_send_command+0x18a/0x1f0 [virtio_net] [ 213.796310] _virtnet_set_queues+0xc6/0x120 [virtio_net] [ 213.796319] virtnet_probe+0xa06/0xd50 [virtio_net] [ 213.796328] virtio_dev_probe+0x195/0x230 [ 213.796333] really_probe+0x19f/0x400 [ 213.796338] ? __pfx___driver_attach+0x10/0x10 [ 213.796340] __driver_probe_device+0x78/0x160 [ 213.796343] driver_probe_device+0x1f/0x90 [ 213.796346] __driver_attach+0xd6/0x1d0 [ 213.796349] bus_for_each_dev+0x8c/0xe0 [ 213.796355] bus_add_driver+0x119/0x220 [ 213.796359] driver_register+0x59/0x100 [ 213.796362] ? __pfx_virtio_net_driver_init+0x10/0x10 [virtio_net] [ 213.796369] virtio_net_driver_init+0x8e/0xff0 [virtio_net] [ 213.796375] do_one_initcall+0x6f/0x380 [ 213.796384] do_init_module+0x60/0x240 [ 213.796388] init_module_from_file+0x86/0xc0 [ 213.796396] idempotent_init_module+0x129/0x2c0 [ 213.796406] __x64_sys_finit_module+0x5e/0xb0 [ 213.796409] do_syscall_64+0x60/0xe0 [ 213.796415] ? do_syscall_64+0x6f/0xe0 [ 213.796418] ? lockdep_hardirqs_on_prepare+0xe4/0x1a0 [ 213.796424] ? do_syscall_64+0x6f/0xe0 [ 213.796427] ? do_syscall_64+0x6f/0xe0 [ 213.796431] entry_SYSCALL_64_after_hwframe+0x6e/0x76 [ 213.796435] RIP: 0033:0x7f7758f279cd [ 213.796465] Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 33 e4 0c 00 f7 d8 64 89 01 48 [ 213.796467] RSP: 002b:00007ffe2cad8738 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 213.796469] RAX: ffffffffffffffda RBX: 0000557f987a8180 RCX: 00007f7758f279cd [ 213.796471] RDX: 0000000000000000 RSI: 00007f77593e5453 RDI: 000000000000000f [ 213.796472] RBP: 00007f77593e5453 R08: 0000000000000000 R09: 00007ffe2cad8860 [ 213.796473] R10: 000000000000000f R11: 0000000000000246 R12: 0000000000020000 [ 213.796475] R13: 0000557f9879f8e0 R14: 0000000000000000 R15: 0000557f98783aa0 [ 213.796482] </TASK>"
Signed-off-by Zhu Yanjun <[email protected]>--- drivers/net/virtio_net.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
Quoted 12 lines
drivers/net/virtio_net.c static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, · line 2471
Quoted 10 lines
drivers/net/virtio_net.c +1
-1@@ -2468,7 +2468,7 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, {struct scatterlist *sgs[4 ], hdr, stat;unsigned out_num =0 , tmp; -int ret; +int ret, timeout =200 ;Any reason we choose this value or how can we know it works for all types of devices?
Sorry. Resend it again because it is rejected by netdev.
As I mentioned in the commit log, the similar problem also occurs in Intel NIC driver and Mellanox NIC driver.
This commit is borrowed from the solution of Intel NIC driver. So the value is also from Intel NIC driver solution.
Quoted 3 lines
A more easy way is to use
cond_resched()but it may have side effects as well[1]. But it seems less intrusive anyhow than the proposal here?
Thanks a lot for your suggestions. I have made tests with the commits in the link https://www.mail-archive.com/[email protected]/msg60297.html.
Because virtio_net driver spins waiting for the response of hardware, virtio_net driver can not be unloaded and kernel hang still occurs when running "ip link" or unloading virtio_net module.
Zhu Yanjun
Quoted 26 lines
Thanks
[1] https://www.mail-archive.com/[email protected]/msg60297.html
Quoted 21 lines
drivers/net/virtio_net.c +7
-2/* Caller should know better */ BUG_ON (!virtio_has_feature (vi->vdev,VIRTIO_NET_F_CTRL_VQ ));@@ -2502,8 +2502,14 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +if (timeout) + timeout--; +else +return false;/* long time no response */ +cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.42.0
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 110 lines · Zhu Yanjun
在 2024/1/15 10:20, Jason Wang 写道:
On Mon, Jan 15, 2024 at 9:35 AM Zhu Yanjun <[email protected]> wrote:
From: Zhu Yanjun <[email protected]>
Some devices emulate the
virtio_nethardwares. Whenvirtio_netdriver sends commands to the emulated hardware, normally the hardware needs time to response. Sometimes the time is very long. Thus, the following will appear. Then the whole system will hang.
The similar problems also occur in Intel NICs and Mellanox NICs. As such, the similar solution is borrowed from them. A timeout value is added and the timeout value as large as possible is set to ensure that the driver gets the maximum possible response from the hardware."
[ 213.795860] watchdog: BUG: soft lockup - CPU#108 stuck for 26s! [(udev-worker):3157] [ 213.796114] Modules linked in: virtio_net(+) net_failover failover qrtr rfkill sunrpc intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common intel_ifs i10nm_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt rapl intel_pmc_bxt dax_hmem iTCO_vendor_support vfat cxl_acpi intel_cstate pmt_telemetry pmt_class intel_sdsi joydev intel_uncore cxl_core fat pcspkr mei_me isst_if_mbox_pci isst_if_mmio idxd i2c_i801 isst_if_common mei intel_vsec idxd_bus i2c_smbus i2c_ismt ipmi_ssif acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter pfr_telemetry pfr_update fuse loop zram xfs crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 bnxt_en sha256_ssse3 sha1_ssse3 nvme ast nvme_core i2c_algo_bit wmi pinctrl_emmitsburg scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_multipath [ 213.796194] irq event stamp: 67740 [ 213.796195] hardirqs last enabled at (67739): [<ffffffff8c2015ca>] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 213.796203] hardirqs last disabled at (67740): [<ffffffff8c14108e>] sysvec_apic_timer_interrupt+0xe/0x90 [ 213.796208] softirqs last enabled at (67686): [<ffffffff8b12115e>] __irq_exit_rcu+0xbe/0xe0 [ 213.796214] softirqs last disabled at (67681): [<ffffffff8b12115e>] __irq_exit_rcu+0xbe/0xe0 [ 213.796217] CPU: 108 PID: 3157 Comm: (udev-worker) Kdump: loaded Not tainted 6.7.0+ #9 [ 213.796220] Hardware name: Intel Corporation M50FCP2SBSTD/M50FCP2SBSTD, BIOS SE5C741.86B.01.01.0001.2211140926 11/14/2022 [ 213.796221] RIP: 0010:virtqueue_get_buf_ctx_split+0x8d/0x110 [ 213.796228] Code: 89 df e8 26 fe ff ff 0f b7 43 50 83 c0 01 66 89 43 50 f6 43 78 01 75 12 80 7b 42 00 48 8b 4b 68 8b 53 58 74 0f 66 87 44 51 04 <48> 89 e8 5b 5d c3 cc cc cc cc 66 89 44 51 04 0f ae f0 48 89 e8 5b [ 213.796230] RSP: 0018:ff4bbb362306f9b0 EFLAGS: 00000246 [ 213.796233] RAX: 0000000000000000 RBX: ff2f15095896f000 RCX: 0000000000000001 [ 213.796235] RDX: 0000000000000000 RSI: ff4bbb362306f9cc RDI: ff2f15095896f000 [ 213.796236] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 213.796237] R10: 0000000000000003 R11: ff2f15095893cc40 R12: 0000000000000002 [ 213.796239] R13: 0000000000000004 R14: 0000000000000000 R15: ff2f1509534f3000 [ 213.796240] FS: 00007f775847d0c0(0000) GS:ff2f1528bac00000(0000) knlGS:0000000000000000 [ 213.796242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 213.796243] CR2: 0000557f987b6e70 CR3: 0000002098602006 CR4: 0000000000f71ef0 [ 213.796245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 213.796246] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 [ 213.796247] PKRU: 55555554 [ 213.796249] Call Trace: [ 213.796250] <IRQ> [ 213.796252] ? watchdog_timer_fn+0x1c0/0x220 [ 213.796258] ? __pfx_watchdog_timer_fn+0x10/0x10 [ 213.796261] ? __hrtimer_run_queues+0x1af/0x380 [ 213.796269] ? hrtimer_interrupt+0xf8/0x230 [ 213.796274] ? __sysvec_apic_timer_interrupt+0x64/0x1a0 [ 213.796279] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 213.796282] </IRQ> [ 213.796284] <TASK> [ 213.796285] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 213.796293] ? virtqueue_get_buf_ctx_split+0x8d/0x110 [ 213.796297] virtnet_send_command+0x18a/0x1f0 [virtio_net] [ 213.796310] _virtnet_set_queues+0xc6/0x120 [virtio_net] [ 213.796319] virtnet_probe+0xa06/0xd50 [virtio_net] [ 213.796328] virtio_dev_probe+0x195/0x230 [ 213.796333] really_probe+0x19f/0x400 [ 213.796338] ? __pfx___driver_attach+0x10/0x10 [ 213.796340] __driver_probe_device+0x78/0x160 [ 213.796343] driver_probe_device+0x1f/0x90 [ 213.796346] __driver_attach+0xd6/0x1d0 [ 213.796349] bus_for_each_dev+0x8c/0xe0 [ 213.796355] bus_add_driver+0x119/0x220 [ 213.796359] driver_register+0x59/0x100 [ 213.796362] ? __pfx_virtio_net_driver_init+0x10/0x10 [virtio_net] [ 213.796369] virtio_net_driver_init+0x8e/0xff0 [virtio_net] [ 213.796375] do_one_initcall+0x6f/0x380 [ 213.796384] do_init_module+0x60/0x240 [ 213.796388] init_module_from_file+0x86/0xc0 [ 213.796396] idempotent_init_module+0x129/0x2c0 [ 213.796406] __x64_sys_finit_module+0x5e/0xb0 [ 213.796409] do_syscall_64+0x60/0xe0 [ 213.796415] ? do_syscall_64+0x6f/0xe0 [ 213.796418] ? lockdep_hardirqs_on_prepare+0xe4/0x1a0 [ 213.796424] ? do_syscall_64+0x6f/0xe0 [ 213.796427] ? do_syscall_64+0x6f/0xe0 [ 213.796431] entry_SYSCALL_64_after_hwframe+0x6e/0x76 [ 213.796435] RIP: 0033:0x7f7758f279cd [ 213.796465] Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 33 e4 0c 00 f7 d8 64 89 01 48 [ 213.796467] RSP: 002b:00007ffe2cad8738 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 213.796469] RAX: ffffffffffffffda RBX: 0000557f987a8180 RCX: 00007f7758f279cd [ 213.796471] RDX: 0000000000000000 RSI: 00007f77593e5453 RDI: 000000000000000f [ 213.796472] RBP: 00007f77593e5453 R08: 0000000000000000 R09: 00007ffe2cad8860 [ 213.796473] R10: 000000000000000f R11: 0000000000000246 R12: 0000000000020000 [ 213.796475] R13: 0000557f9879f8e0 R14: 0000000000000000 R15: 0000557f98783aa0 [ 213.796482] </TASK>"
Signed-off-by Zhu Yanjun <[email protected]>--- drivers/net/virtio_net.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)drivers/net/virtio_net.c +1
-1@@ -2468,7 +2468,7 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, {struct scatterlist *sgs[4 ], hdr, stat;unsigned out_num =0 , tmp; -int ret; +int ret, timeout =200 ;Any reason we choose this value or how can we know it works for all types of devices?
Quoted 3 lines
As I mentioned in the commit log, the similar problem also occurs in Intel NIC driver and Mellanox NIC driver.
This commit is borrowed from the solution of Intel NIC driver. So the value is also from Intel NIC driver solution.
Right, so basically I meant we need a solution that works for all vendors.
Quoted 7 lines
A more easy way is to use
cond_resched()but it may have side effects as well[1]. But it seems less intrusive anyhow than the proposal here?Thanks a lot for your suggestions. I have made tests with the commits in the link https://www.mail-archive.com/[email protected]/msg60297.html.
Because
virtio_netdriver spins waiting for the response of hardware,virtio_netdriver can not be unloaded and kernel hang still occurs when running "ip link" or unloadingvirtio_netmodule.
Well, yes. It aims to solve the lockups as described in this commit log. It doesn't solve the issue of infinite waiting etc.
Thanks
Quoted 28 lines
Zhu Yanjun
Thanks
[1] https://www.mail-archive.com/[email protected]/msg60297.html
/* Caller should know better */ BUG_ON(!virtio_has_feature(vi->vdev, VIRTIO_NET_F_CTRL_VQ));drivers/net/virtio_net.c +7
-2@@ -2502,8 +2502,14 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +if (timeout) + timeout--; +else +return false;/* long time no response */ +cpu_relax (); + }return vi->ctrl->status ==VIRTIO_NET_OK ; } --2.42.0
Paolo Abeni
maintainerQuoted 104 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
Some devices emulate the
virtio_nethardwares. Whenvirtio_netdriver sends commands to the emulated hardware, normally the hardware needs time to response. Sometimes the time is very long. Thus, the following will appear. Then the whole system will hang.
The similar problems also occur in Intel NICs and Mellanox NICs. As such, the similar solution is borrowed from them. A timeout value is added and the timeout value as large as possible is set to ensure that the driver gets the maximum possible response from the hardware."
[ 213.795860] watchdog: BUG: soft lockup - CPU#108 stuck for 26s! [(udev-worker):3157] [ 213.796114] Modules linked in: virtio_net(+) net_failover failover qrtr rfkill sunrpc intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common intel_ifs i10nm_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt rapl intel_pmc_bxt dax_hmem iTCO_vendor_support vfat cxl_acpi intel_cstate pmt_telemetry pmt_class intel_sdsi joydev intel_uncore cxl_core fat pcspkr mei_me isst_if_mbox_pci isst_if_mmio idxd i2c_i801 isst_if_common mei intel_vsec idxd_bus i2c_smbus i2c_ismt ipmi_ssif acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter pfr_telemetry pfr_update fuse loop zram xfs crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 bnxt_en sha256_ssse3 sha1_ssse3 nvme ast nvme_core i2c_algo_bit wmi pinctrl_emmitsburg scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_multipath [ 213.796194] irq event stamp: 67740 [ 213.796195] hardirqs last enabled at (67739): [<ffffffff8c2015ca>] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 213.796203] hardirqs last disabled at (67740): [<ffffffff8c14108e>] sysvec_apic_timer_interrupt+0xe/0x90 [ 213.796208] softirqs last enabled at (67686): [<ffffffff8b12115e>] __irq_exit_rcu+0xbe/0xe0 [ 213.796214] softirqs last disabled at (67681): [<ffffffff8b12115e>] __irq_exit_rcu+0xbe/0xe0 [ 213.796217] CPU: 108 PID: 3157 Comm: (udev-worker) Kdump: loaded Not tainted 6.7.0+ #9 [ 213.796220] Hardware name: Intel Corporation M50FCP2SBSTD/M50FCP2SBSTD, BIOS SE5C741.86B.01.01.0001.2211140926 11/14/2022 [ 213.796221] RIP: 0010:virtqueue_get_buf_ctx_split+0x8d/0x110 [ 213.796228] Code: 89 df e8 26 fe ff ff 0f b7 43 50 83 c0 01 66 89 43 50 f6 43 78 01 75 12 80 7b 42 00 48 8b 4b 68 8b 53 58 74 0f 66 87 44 51 04 <48> 89 e8 5b 5d c3 cc cc cc cc 66 89 44 51 04 0f ae f0 48 89 e8 5b [ 213.796230] RSP: 0018:ff4bbb362306f9b0 EFLAGS: 00000246 [ 213.796233] RAX: 0000000000000000 RBX: ff2f15095896f000 RCX: 0000000000000001 [ 213.796235] RDX: 0000000000000000 RSI: ff4bbb362306f9cc RDI: ff2f15095896f000 [ 213.796236] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 213.796237] R10: 0000000000000003 R11: ff2f15095893cc40 R12: 0000000000000002 [ 213.796239] R13: 0000000000000004 R14: 0000000000000000 R15: ff2f1509534f3000 [ 213.796240] FS: 00007f775847d0c0(0000) GS:ff2f1528bac00000(0000) knlGS:0000000000000000 [ 213.796242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 213.796243] CR2: 0000557f987b6e70 CR3: 0000002098602006 CR4: 0000000000f71ef0 [ 213.796245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 213.796246] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 [ 213.796247] PKRU: 55555554 [ 213.796249] Call Trace: [ 213.796250] <IRQ> [ 213.796252] ? watchdog_timer_fn+0x1c0/0x220 [ 213.796258] ? __pfx_watchdog_timer_fn+0x10/0x10 [ 213.796261] ? __hrtimer_run_queues+0x1af/0x380 [ 213.796269] ? hrtimer_interrupt+0xf8/0x230 [ 213.796274] ? __sysvec_apic_timer_interrupt+0x64/0x1a0 [ 213.796279] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 213.796282] </IRQ> [ 213.796284] <TASK> [ 213.796285] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 213.796293] ? virtqueue_get_buf_ctx_split+0x8d/0x110 [ 213.796297] virtnet_send_command+0x18a/0x1f0 [virtio_net] [ 213.796310] _virtnet_set_queues+0xc6/0x120 [virtio_net] [ 213.796319] virtnet_probe+0xa06/0xd50 [virtio_net] [ 213.796328] virtio_dev_probe+0x195/0x230 [ 213.796333] really_probe+0x19f/0x400 [ 213.796338] ? __pfx___driver_attach+0x10/0x10 [ 213.796340] __driver_probe_device+0x78/0x160 [ 213.796343] driver_probe_device+0x1f/0x90 [ 213.796346] __driver_attach+0xd6/0x1d0 [ 213.796349] bus_for_each_dev+0x8c/0xe0 [ 213.796355] bus_add_driver+0x119/0x220 [ 213.796359] driver_register+0x59/0x100 [ 213.796362] ? __pfx_virtio_net_driver_init+0x10/0x10 [virtio_net] [ 213.796369] virtio_net_driver_init+0x8e/0xff0 [virtio_net] [ 213.796375] do_one_initcall+0x6f/0x380 [ 213.796384] do_init_module+0x60/0x240 [ 213.796388] init_module_from_file+0x86/0xc0 [ 213.796396] idempotent_init_module+0x129/0x2c0 [ 213.796406] __x64_sys_finit_module+0x5e/0xb0 [ 213.796409] do_syscall_64+0x60/0xe0 [ 213.796415] ? do_syscall_64+0x6f/0xe0 [ 213.796418] ? lockdep_hardirqs_on_prepare+0xe4/0x1a0 [ 213.796424] ? do_syscall_64+0x6f/0xe0 [ 213.796427] ? do_syscall_64+0x6f/0xe0 [ 213.796431] entry_SYSCALL_64_after_hwframe+0x6e/0x76 [ 213.796435] RIP: 0033:0x7f7758f279cd [ 213.796465] Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 33 e4 0c 00 f7 d8 64 89 01 48 [ 213.796467] RSP: 002b:00007ffe2cad8738 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 213.796469] RAX: ffffffffffffffda RBX: 0000557f987a8180 RCX: 00007f7758f279cd [ 213.796471] RDX: 0000000000000000 RSI: 00007f77593e5453 RDI: 000000000000000f [ 213.796472] RBP: 00007f77593e5453 R08: 0000000000000000 R09: 00007ffe2cad8860 [ 213.796473] R10: 000000000000000f R11: 0000000000000246 R12: 0000000000020000 [ 213.796475] R13: 0000557f9879f8e0 R14: 0000000000000000 R15: 0000557f98783aa0 [ 213.796482] </TASK>"
Signed-off-by Zhu Yanjun <[email protected]>--- drivers/net/virtio_net.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)drivers/net/virtio_net.c +1
-1@@ -2468,7 +2468,7 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, {struct scatterlist *sgs[4 ], hdr, stat;unsigned out_num =0 , tmp; -int ret; +int ret, timeout =200 ;/* Caller should know better */ BUG_ON (!virtio_has_feature (vi->vdev,VIRTIO_NET_F_CTRL_VQ ));
drivers/net/virtio_net.c static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, · line 2507
Quoted 7 lines
drivers/net/virtio_net.c +3
-1@@ -2502,8 +2502,14 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +if (timeout) + timeout--;
This is not really a timeout, just a loop counter. 200 iterations could be a very short time on reasonable H/W. I guess this avoid the soft lockup, but possibly (likely?) breaks the functionality when we need to loop for some non negligible time.
I fear we need a more complex solution, as mentioned by Micheal in the thread you quoted.
Cheers,
Paolo
Quoted 117 lines · Paolo Abeni
Quoted 104 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
Some devices emulate the
virtio_nethardwares. Whenvirtio_netdriver sends commands to the emulated hardware, normally the hardware needs time to response. Sometimes the time is very long. Thus, the following will appear. Then the whole system will hang.
The similar problems also occur in Intel NICs and Mellanox NICs. As such, the similar solution is borrowed from them. A timeout value is added and the timeout value as large as possible is set to ensure that the driver gets the maximum possible response from the hardware."
[ 213.795860] watchdog: BUG: soft lockup - CPU#108 stuck for 26s! [(udev-worker):3157] [ 213.796114] Modules linked in: virtio_net(+) net_failover failover qrtr rfkill sunrpc intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common intel_ifs i10nm_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt rapl intel_pmc_bxt dax_hmem iTCO_vendor_support vfat cxl_acpi intel_cstate pmt_telemetry pmt_class intel_sdsi joydev intel_uncore cxl_core fat pcspkr mei_me isst_if_mbox_pci isst_if_mmio idxd i2c_i801 isst_if_common mei intel_vsec idxd_bus i2c_smbus i2c_ismt ipmi_ssif acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter pfr_telemetry pfr_update fuse loop zram xfs crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 bnxt_en sha256_ssse3 sha1_ssse3 nvme ast nvme_core i2c_algo_bit wmi pinctrl_emmitsburg scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_multipath [ 213.796194] irq event stamp: 67740 [ 213.796195] hardirqs last enabled at (67739): [<ffffffff8c2015ca>] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 213.796203] hardirqs last disabled at (67740): [<ffffffff8c14108e>] sysvec_apic_timer_interrupt+0xe/0x90 [ 213.796208] softirqs last enabled at (67686): [<ffffffff8b12115e>] __irq_exit_rcu+0xbe/0xe0 [ 213.796214] softirqs last disabled at (67681): [<ffffffff8b12115e>] __irq_exit_rcu+0xbe/0xe0 [ 213.796217] CPU: 108 PID: 3157 Comm: (udev-worker) Kdump: loaded Not tainted 6.7.0+ #9 [ 213.796220] Hardware name: Intel Corporation M50FCP2SBSTD/M50FCP2SBSTD, BIOS SE5C741.86B.01.01.0001.2211140926 11/14/2022 [ 213.796221] RIP: 0010:virtqueue_get_buf_ctx_split+0x8d/0x110 [ 213.796228] Code: 89 df e8 26 fe ff ff 0f b7 43 50 83 c0 01 66 89 43 50 f6 43 78 01 75 12 80 7b 42 00 48 8b 4b 68 8b 53 58 74 0f 66 87 44 51 04 <48> 89 e8 5b 5d c3 cc cc cc cc 66 89 44 51 04 0f ae f0 48 89 e8 5b [ 213.796230] RSP: 0018:ff4bbb362306f9b0 EFLAGS: 00000246 [ 213.796233] RAX: 0000000000000000 RBX: ff2f15095896f000 RCX: 0000000000000001 [ 213.796235] RDX: 0000000000000000 RSI: ff4bbb362306f9cc RDI: ff2f15095896f000 [ 213.796236] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 213.796237] R10: 0000000000000003 R11: ff2f15095893cc40 R12: 0000000000000002 [ 213.796239] R13: 0000000000000004 R14: 0000000000000000 R15: ff2f1509534f3000 [ 213.796240] FS: 00007f775847d0c0(0000) GS:ff2f1528bac00000(0000) knlGS:0000000000000000 [ 213.796242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 213.796243] CR2: 0000557f987b6e70 CR3: 0000002098602006 CR4: 0000000000f71ef0 [ 213.796245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 213.796246] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 [ 213.796247] PKRU: 55555554 [ 213.796249] Call Trace: [ 213.796250] <IRQ> [ 213.796252] ? watchdog_timer_fn+0x1c0/0x220 [ 213.796258] ? __pfx_watchdog_timer_fn+0x10/0x10 [ 213.796261] ? __hrtimer_run_queues+0x1af/0x380 [ 213.796269] ? hrtimer_interrupt+0xf8/0x230 [ 213.796274] ? __sysvec_apic_timer_interrupt+0x64/0x1a0 [ 213.796279] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 213.796282] </IRQ> [ 213.796284] <TASK> [ 213.796285] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 213.796293] ? virtqueue_get_buf_ctx_split+0x8d/0x110 [ 213.796297] virtnet_send_command+0x18a/0x1f0 [virtio_net] [ 213.796310] _virtnet_set_queues+0xc6/0x120 [virtio_net] [ 213.796319] virtnet_probe+0xa06/0xd50 [virtio_net] [ 213.796328] virtio_dev_probe+0x195/0x230 [ 213.796333] really_probe+0x19f/0x400 [ 213.796338] ? __pfx___driver_attach+0x10/0x10 [ 213.796340] __driver_probe_device+0x78/0x160 [ 213.796343] driver_probe_device+0x1f/0x90 [ 213.796346] __driver_attach+0xd6/0x1d0 [ 213.796349] bus_for_each_dev+0x8c/0xe0 [ 213.796355] bus_add_driver+0x119/0x220 [ 213.796359] driver_register+0x59/0x100 [ 213.796362] ? __pfx_virtio_net_driver_init+0x10/0x10 [virtio_net] [ 213.796369] virtio_net_driver_init+0x8e/0xff0 [virtio_net] [ 213.796375] do_one_initcall+0x6f/0x380 [ 213.796384] do_init_module+0x60/0x240 [ 213.796388] init_module_from_file+0x86/0xc0 [ 213.796396] idempotent_init_module+0x129/0x2c0 [ 213.796406] __x64_sys_finit_module+0x5e/0xb0 [ 213.796409] do_syscall_64+0x60/0xe0 [ 213.796415] ? do_syscall_64+0x6f/0xe0 [ 213.796418] ? lockdep_hardirqs_on_prepare+0xe4/0x1a0 [ 213.796424] ? do_syscall_64+0x6f/0xe0 [ 213.796427] ? do_syscall_64+0x6f/0xe0 [ 213.796431] entry_SYSCALL_64_after_hwframe+0x6e/0x76 [ 213.796435] RIP: 0033:0x7f7758f279cd [ 213.796465] Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 33 e4 0c 00 f7 d8 64 89 01 48 [ 213.796467] RSP: 002b:00007ffe2cad8738 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 213.796469] RAX: ffffffffffffffda RBX: 0000557f987a8180 RCX: 00007f7758f279cd [ 213.796471] RDX: 0000000000000000 RSI: 00007f77593e5453 RDI: 000000000000000f [ 213.796472] RBP: 00007f77593e5453 R08: 0000000000000000 R09: 00007ffe2cad8860 [ 213.796473] R10: 000000000000000f R11: 0000000000000246 R12: 0000000000020000 [ 213.796475] R13: 0000557f9879f8e0 R14: 0000000000000000 R15: 0000557f98783aa0 [ 213.796482] </TASK>"
Signed-off-by Zhu Yanjun <[email protected]>--- drivers/net/virtio_net.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)drivers/net/virtio_net.c +1
-1@@ -2468,7 +2468,7 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, {struct scatterlist *sgs[4 ], hdr, stat;unsigned out_num =0 , tmp; -int ret; +int ret, timeout =200 ;/* Caller should know better */ BUG_ON (!virtio_has_feature (vi->vdev,VIRTIO_NET_F_CTRL_VQ ));drivers/net/virtio_net.c static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, · line 2507
Quoted 7 lines
drivers/net/virtio_net.c +3
-1@@ -2502,8 +2502,14 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +if (timeout) + timeout--;This is not really a timeout, just a loop counter. 200 iterations could be a very short time on reasonable H/W. I guess this avoid the soft lockup, but possibly (likely?) breaks the functionality when we need to loop for some non negligible time.
Quoted 2 lines
I fear we need a more complex solution, as mentioned by Micheal in the thread you quoted.
Got it. I also look forward to the more complex solution to this problem.
Zhu Yanjun
Quoted 5 lines
Cheers,
Paolo
Quoted 120 lines · Zhu Yanjun
Quoted 118 lines · Paolo Abeni
Quoted 107 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
Some devices emulate the
virtio_nethardwares. Whenvirtio_netdriver sends commands to the emulated hardware, normally the hardware needs time to response. Sometimes the time is very long. Thus, the following will appear. Then the whole system will hang.
The similar problems also occur in Intel NICs and Mellanox NICs. As such, the similar solution is borrowed from them. A timeout value is added and the timeout value as large as possible is set to ensure that the driver gets the maximum possible response from the hardware."
[ 213.795860] watchdog: BUG: soft lockup - CPU#108 stuck for 26s! [(udev-worker):3157]
[ 213.796114] Modules linked in:virtio_net(+)net_failoverfailover qrtr rfkill sunrpcintel_rapl_msrintel_rapl_commonintel_uncore_frequencyintel_uncore_frequency_commonintel_ifsi10nm_edacnfit libnvdimmx86_pkg_temp_thermalintel_powerclampcoretempiTCO_wdtraplintel_pmc_bxtdax_hmemiTCO_vendor_supportvfatcxl_acpiintel_cstatepmt_telemetrypmt_classintel_sdsijoydevintel_uncorecxl_corefat pcspkrmei_meisst_if_mbox_pciisst_if_mmioidxdi2c_i801isst_if_commonmeiintel_vsecidxd_busi2c_smbusi2c_ismtipmi_ssifacpi_ipmiipmi_siipmi_devintfipmi_msghandleracpi_padacpi_power_meterpfr_telemetrypfr_updatefuse loop zram xfscrct10dif_pclmulcrc32_pclmulcrc32c_intelpolyval_clmulnipolyval_genericghash_clmulni_intelsha512_ssse3bnxt_ensha256_ssse3sha1_ssse3nvme astnvme_corei2c_algo_bitwmipinctrl_emmitsburgscsi_dh_rdacscsi_dh_emcscsi_dh_aluadm_multipath[ 213.796194] irq event stamp: 67740
[ 213.796195] hardirqs last enabled at (67739): [<ffffffff8c2015ca>]asm_sysvec_apic_timer_interrupt+0x1a/0x20
[ 213.796203] hardirqs last disabled at (67740): [<ffffffff8c14108e>]sysvec_apic_timer_interrupt+0xe/0x90
[ 213.796208] softirqs last enabled at (67686): [<ffffffff8b12115e>]__irq_exit_rcu+0xbe/0xe0
[ 213.796214] softirqs last disabled at (67681): [<ffffffff8b12115e>]__irq_exit_rcu+0xbe/0xe0
[ 213.796217] CPU: 108 PID: 3157 Comm: (udev-worker) Kdump: loaded Not tainted 6.7.0+ #9
[ 213.796220] Hardware name: Intel Corporation M50FCP2SBSTD/M50FCP2SBSTD, BIOS SE5C741.86B.01.01.0001.2211140926 11/14/2022
[ 213.796221] RIP: 0010:virtqueue_get_buf_ctx_split+0x8d/0x110
[ 213.796228] Code: 89 df e8 26 fe ff ff 0f b7 43 50 83 c0 01 66 89 43 50 f6 43 78 01 75 12 80 7b 42 00 48 8b 4b 68 8b 53 58 74 0f 66 87 44 51 04 <48> 89 e8 5b 5d c3 cc cc cc cc 66 89 44 51 04 0f ae f0 48 89 e8 5b
[ 213.796230] RSP: 0018:ff4bbb362306f9b0 EFLAGS: 00000246
[ 213.796233] RAX: 0000000000000000 RBX:ff2f15095896f000RCX: 0000000000000001
[ 213.796235] RDX: 0000000000000000 RSI:ff4bbb362306f9ccRDI:ff2f15095896f000
[ 213.796236] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[ 213.796237] R10: 0000000000000003 R11:ff2f15095893cc40R12: 0000000000000002
[ 213.796239] R13: 0000000000000004 R14: 0000000000000000 R15:ff2f1509534f3000
[ 213.796240] FS: 00007f775847d0c0(0000) GS:ff2f1528bac00000(0000) knlGS:0000000000000000
[ 213.796242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 213.796243] CR2:0000557f987b6e70CR3: 0000002098602006 CR4:0000000000f71ef0
[ 213.796245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 213.796246] DR3: 0000000000000000 DR6:00000000fffe07f0DR7: 0000000000000400
[ 213.796247] PKRU: 55555554
[ 213.796249] Call Trace:
[ 213.796250] <IRQ>
[ 213.796252] ?watchdog_timer_fn+0x1c0/0x220
[ 213.796258] ?__pfx_watchdog_timer_fn+0x10/0x10
[ 213.796261] ?__hrtimer_run_queues+0x1af/0x380
[ 213.796269] ?hrtimer_interrupt+0xf8/0x230
[ 213.796274] ?__sysvec_apic_timer_interrupt+0x64/0x1a0
[ 213.796279] ?sysvec_apic_timer_interrupt+0x6d/0x90
[ 213.796282] </IRQ>
[ 213.796284] <TASK>
[ 213.796285] ?asm_sysvec_apic_timer_interrupt+0x1a/0x20
[ 213.796293] ?virtqueue_get_buf_ctx_split+0x8d/0x110
[ 213.796297]virtnet_send_command+0x18a/0x1f0[virtio_net]
[ 213.796310]_virtnet_set_queues+0xc6/0x120[virtio_net]
[ 213.796319]virtnet_probe+0xa06/0xd50[virtio_net]
[ 213.796328]virtio_dev_probe+0x195/0x230
[ 213.796333]really_probe+0x19f/0x400
[ 213.796338] ?__pfx___driver_attach+0x10/0x10
[ 213.796340]__driver_probe_device+0x78/0x160
[ 213.796343]driver_probe_device+0x1f/0x90
[ 213.796346]__driver_attach+0xd6/0x1d0
[ 213.796349]bus_for_each_dev+0x8c/0xe0
[ 213.796355]bus_add_driver+0x119/0x220
[ 213.796359]driver_register+0x59/0x100
[ 213.796362] ?__pfx_virtio_net_driver_init+0x10/0x10[virtio_net]
[ 213.796369]virtio_net_driver_init+0x8e/0xff0[virtio_net]
[ 213.796375]do_one_initcall+0x6f/0x380
[ 213.796384]do_init_module+0x60/0x240
[ 213.796388]init_module_from_file+0x86/0xc0
[ 213.796396]idempotent_init_module+0x129/0x2c0
[ 213.796406]__x64_sys_finit_module+0x5e/0xb0
[ 213.796409]do_syscall_64+0x60/0xe0
[ 213.796415] ?do_syscall_64+0x6f/0xe0
[ 213.796418] ?lockdep_hardirqs_on_prepare+0xe4/0x1a0
[ 213.796424] ?do_syscall_64+0x6f/0xe0
[ 213.796427] ?do_syscall_64+0x6f/0xe0
[ 213.796431]entry_SYSCALL_64_after_hwframe+0x6e/0x76
[ 213.796435] RIP: 0033:0x7f7758f279cd
[ 213.796465] Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 33 e4 0c 00 f7 d8 64 89 01 48
[ 213.796467] RSP: 002b:00007ffe2cad8738 EFLAGS: 00000246ORIG_RAX: 0000000000000139
[ 213.796469] RAX: ffffffffffffffda RBX:0000557f987a8180RCX:00007f7758f279cd
[ 213.796471] RDX: 0000000000000000 RSI:00007f77593e5453RDI:000000000000000f
[ 213.796472] RBP:00007f77593e5453R08: 0000000000000000 R09:00007ffe2cad8860
[ 213.796473] R10:000000000000000fR11: 0000000000000246 R12: 0000000000020000
[ 213.796475] R13:0000557f9879f8e0R14: 0000000000000000 R15:0000557f98783aa0
[ 213.796482] </TASK>
"Signed-off-by Zhu Yanjun <[email protected]>---
drivers/net/virtio_net.c| 10 ++++++++--1 file changed, 8 insertions(+), 2 deletions(-)drivers/net/virtio_net.c +1
-1@@ -2468,7 +2468,7 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, { struct scatterlist *sgs[4], hdr, stat; unsigned out_num = 0, tmp; - int ret; + int ret, timeout = 200; /* Caller should know better */ BUG_ON(!virtio_has_feature(vi->vdev, VIRTIO_NET_F_CTRL_VQ));@@ -2502,8 +2502,14 @@ static bool virtnet_send_command(structvirtnet_info *vi, u8 class, u8 cmd, * into the hypervisor, so the request should be handled immediately. */ Quoted 5 lines
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
- !virtqueue_is_broken(vi->cvq))
+ !virtqueue_is_broken(vi->cvq)) {
+ if (timeout)
+ timeout--;This is not really a timeout, just a loop counter. 200 iterations could be a very short time on reasonable H/W. I guess this avoid the soft lockup, but possibly (likely?) breaks the functionality when we need to loop for some non negligible time.
Quoted 4 lines
Quoted 2 lines
I fear we need a more complex solution, as mentioned by Micheal in the thread you quoted.
Got it. I also look forward to the more complex solution to this problem.
Can we add a device capability (new feature bit) such as ctrq_wait_timeout to get a reasonable timeout?
Thanks,
Heng
Quoted 8 lines
Zhu Yanjun
Quoted 5 lines
Cheers,
Paolo
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
- !virtqueue_is_broken(vi->cvq))
+ !virtqueue_is_broken(vi->cvq)) {
+ if (timeout)
+ timeout--;This is not really a timeout, just a loop counter. 200 iterations could be a very short time on reasonable H/W. I guess this avoid the soft lockup, but possibly (likely?) breaks the functionality when we need to loop for some non negligible time.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
I fear we need a more complex solution, as mentioned by Micheal in the thread you quoted.
Got it. I also look forward to the more complex solution to this problem.
Can we add a device capability (new feature bit) such as
ctrq_wait_timeoutto get a reasonable timeout?
The usual solution to this is include/linux/iopoll.h. If you can sleep read_poll_timeout() otherwise read_poll_timeout_atomic().
Andrew
Quoted 10 lines · Andrew Lunn
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
- !virtqueue_is_broken(vi->cvq)) + !virtqueue_is_broken(vi->cvq)) { + if (timeout)
+ timeout--;This is not really a timeout, just a loop counter. 200 iterations could be a very short time on reasonable H/W. I guess this avoid the soft lockup, but possibly (likely?) breaks the functionality when we need to loop for some non negligible time.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
I fear we need a more complex solution, as mentioned by Micheal in the thread you quoted.
Got it. I also look forward to the more complex solution to this problem.
Can we add a device capability (new feature bit) such as
ctrq_wait_timeoutto get a reasonable timeout?The usual solution to this is
include/linux/iopoll.h. If you can sleepread_poll_timeout()otherwiseread_poll_timeout_atomic().
Thanks. The 2 functions read_poll_timeout() and read_poll_timeout_atomic() are interesting.
Zhu Yanjun
Quoted 2 lines
Andrew
Quoted 10 lines · Andrew Lunn
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
- !virtqueue_is_broken(vi->cvq)) + !virtqueue_is_broken(vi->cvq)) { + if (timeout)
+ timeout--;This is not really a timeout, just a loop counter. 200 iterations could be a very short time on reasonable H/W. I guess this avoid the soft lockup, but possibly (likely?) breaks the functionality when we need to loop for some non negligible time.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
I fear we need a more complex solution, as mentioned by Micheal in the thread you quoted.
Got it. I also look forward to the more complex solution to this problem.
Can we add a device capability (new feature bit) such as
ctrq_wait_timeoutto get a reasonable timeout?The usual solution to this is
include/linux/iopoll.h. If you can sleepread_poll_timeout()otherwiseread_poll_timeout_atomic().
I read carefully the functions read_poll_timeout() and read_poll_timeout_atomic(). The timeout is set by the caller of the 2 functions.
As such, can we add a module parameter to customize this timeout value by the user?
Or this timeout value is stored in device register, virtio_net driver will read this timeout value at initialization?
Zhu Yanjun
Quoted 2 lines
Andrew
Quoted 16 lines · Zhu Yanjun
Quoted 13 lines · Andrew Lunn
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
drivers/net/virtio_net.c static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, · line 2507
Quoted 5 lines
while (!virtqueue_get_buf(vi->cvq, &tmp) &&drivers/net/virtio_net.c +3
-1- !virtqueue_is_broken(vi->cvq)) + !virtqueue_is_broken(vi->cvq)) { +if (timeout) + timeout--;Quoted 7 lines
This is not really a timeout, just a loop counter. 200 iterations could be a very short time on reasonable H/W. I guess this avoid the soft lockup, but possibly (likely?) breaks the functionality when we need to loop for some non negligible time.
I fear we need a more complex solution, as mentioned by Micheal in the thread you quoted.
Got it. I also look forward to the more complex solution to this problem.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Can we add a device capability (new feature bit) such as
ctrq_wait_timeoutto get a reasonable timeout?The usual solution to this is
include/linux/iopoll.h. If you can sleepread_poll_timeout()otherwiseread_poll_timeout_atomic().I read carefully the functions
read_poll_timeout()andread_poll_timeout_atomic(). The timeout is set by the caller of the 2 functions.
FYI, in order to avoid a swtich of atomic or not, we need convert rx mode setting to workqueue first:
https://www.mail-archive.com/[email protected]/msg60298.html
Quoted 3 lines
As such, can we add a module parameter to customize this timeout value by the user?
Who is the "user" here, or how can the "user" know the value?
Quoted 3 lines
Or this timeout value is stored in device register,
virtio_netdriver will read this timeout value at initialization?
See another thread. The design needs to be general, or you can post a RFC.
In another thought, we've already had a tx watchdog, maybe we can have something similar to cvq and use timeout + reset in that case.
Thans
Quoted 6 lines
Zhu Yanjun
Quoted 2 lines
Andrew
Xuan Zhuo
reviewerQuoted 38 lines · Jason Wang
Quoted 16 lines · Zhu Yanjun
Quoted 13 lines · Andrew Lunn
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
Review
Quoted 5 lines
while (!virtqueue_get_buf(vi->cvq, &tmp) &&(quoted diff) +3
-1- !virtqueue_is_broken(vi->cvq)) + !virtqueue_is_broken(vi->cvq)) { + if (timeout) + timeout--;Quoted 7 lines
This is not really a timeout, just a loop counter. 200 iterations could be a very short time on reasonable H/W. I guess this avoid the soft lockup, but possibly (likely?) breaks the functionality when we need to loop for some non negligible time.
I fear we need a more complex solution, as mentioned by Micheal in the thread you quoted.
Got it. I also look forward to the more complex solution to this problem.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Can we add a device capability (new feature bit) such as
ctrq_wait_timeoutto get a reasonable timeout?The usual solution to this is
include/linux/iopoll.h. If you can sleepread_poll_timeout()otherwiseread_poll_timeout_atomic().I read carefully the functions
read_poll_timeout()andread_poll_timeout_atomic(). The timeout is set by the caller of the 2 functions.FYI, in order to avoid a swtich of atomic or not, we need convert rx mode setting to workqueue first:
https://www.mail-archive.com/[email protected]/msg60298.html
Quoted 3 lines
As such, can we add a module parameter to customize this timeout value by the user?
Who is the "user" here, or how can the "user" know the value?
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
Or this timeout value is stored in device register,
virtio_netdriver will read this timeout value at initialization?See another thread. The design needs to be general, or you can post a RFC.
In another thought, we've already had a tx watchdog, maybe we can have something similar to cvq and use timeout + reset in that case.
But we may block by the reset ^_^ if the device is broken?
Thanks.
Quoted 10 lines
Thans
Quoted 6 lines
Zhu Yanjun
Quoted 2 lines
Andrew
Quoted 43 lines · Xuan Zhuo
Quoted 41 lines · Jason Wang
Quoted 16 lines · Zhu Yanjun
Quoted 13 lines · Andrew Lunn
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
Review
Quoted 5 lines
while (!virtqueue_get_buf(vi->cvq, &tmp) &&(quoted diff) +3
-1- !virtqueue_is_broken(vi->cvq)) + !virtqueue_is_broken(vi->cvq)) { + if (timeout) + timeout--;Quoted 7 lines
This is not really a timeout, just a loop counter. 200 iterations could be a very short time on reasonable H/W. I guess this avoid the soft lockup, but possibly (likely?) breaks the functionality when we need to loop for some non negligible time.
I fear we need a more complex solution, as mentioned by Micheal in the thread you quoted.
Got it. I also look forward to the more complex solution to this problem.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Can we add a device capability (new feature bit) such as
ctrq_wait_timeoutto get a reasonable timeout?The usual solution to this is
include/linux/iopoll.h. If you can sleepread_poll_timeout()otherwiseread_poll_timeout_atomic().I read carefully the functions
read_poll_timeout()andread_poll_timeout_atomic(). The timeout is set by the caller of the 2 functions.FYI, in order to avoid a swtich of atomic or not, we need convert rx mode setting to workqueue first:
https://www.mail-archive.com/[email protected]/msg60298.html
Quoted 3 lines
As such, can we add a module parameter to customize this timeout value by the user?
Who is the "user" here, or how can the "user" know the value?
Quoted 3 lines
Or this timeout value is stored in device register,
virtio_netdriver will read this timeout value at initialization?
Quoted 6 lines
Quoted 4 lines
See another thread. The design needs to be general, or you can post a RFC.
In another thought, we've already had a tx watchdog, maybe we can have something similar to cvq and use timeout + reset in that case.
But we may block by the reset ^_^ if the device is broken?
I mean vq reset here.
It looks like we have multiple goals here
1) avoid lockups, using workqueue + cond_resched() seems to be sufficient, it has issue but nothing new
2) recover from the unresponsive device, the issue for timeout is that it needs to deal with false positives
Thanks
Quoted 15 lines
Thanks.
Quoted 10 lines
Thans
Quoted 6 lines
Zhu Yanjun
Quoted 2 lines
Andrew
Xuan Zhuo
reviewerQuoted 44 lines · Jason Wang
Quoted 43 lines · Xuan Zhuo
Quoted 41 lines · Jason Wang
Quoted 16 lines · Zhu Yanjun
Quoted 13 lines · Andrew Lunn
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
Review
Quoted 5 lines
while (!virtqueue_get_buf(vi->cvq, &tmp) &&(quoted diff) +3
-1- !virtqueue_is_broken(vi->cvq)) + !virtqueue_is_broken(vi->cvq)) { + if (timeout) + timeout--;Quoted 7 lines
This is not really a timeout, just a loop counter. 200 iterations could be a very short time on reasonable H/W. I guess this avoid the soft lockup, but possibly (likely?) breaks the functionality when we need to loop for some non negligible time.
I fear we need a more complex solution, as mentioned by Micheal in the thread you quoted.
Got it. I also look forward to the more complex solution to this problem.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Can we add a device capability (new feature bit) such as
ctrq_wait_timeoutto get a reasonable timeout?The usual solution to this is
include/linux/iopoll.h. If you can sleepread_poll_timeout()otherwiseread_poll_timeout_atomic().I read carefully the functions
read_poll_timeout()andread_poll_timeout_atomic(). The timeout is set by the caller of the 2 functions.FYI, in order to avoid a swtich of atomic or not, we need convert rx mode setting to workqueue first:
https://www.mail-archive.com/[email protected]/msg60298.html
Quoted 3 lines
As such, can we add a module parameter to customize this timeout value by the user?
Who is the "user" here, or how can the "user" know the value?
Quoted 3 lines
Or this timeout value is stored in device register,
virtio_netdriver will read this timeout value at initialization?
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
See another thread. The design needs to be general, or you can post a RFC.
In another thought, we've already had a tx watchdog, maybe we can have something similar to cvq and use timeout + reset in that case.
But we may block by the reset ^_^ if the device is broken?
I mean vq reset here.
I see.
I mean when the deivce is broken, the vq reset also many be blocked.
void vp_modern_set_queue_reset(struct virtio_pci_modern_device *mdev, u16 index)
{
struct virtio_pci_modern_common_cfg __iomem *cfg;
cfg = (struct virtio_pci_modern_common_cfg __iomem *)mdev->common;
vp_iowrite16(index, &cfg->cfg.queue_select);
vp_iowrite16(1, &cfg->queue_reset);
while (vp_ioread16(&cfg->queue_reset))
msleep(1);
while (vp_ioread16(&cfg->cfg.queue_enable))
msleep(1);
}
EXPORT_SYMBOL_GPL(vp_modern_set_queue_reset);In this function, for the broken device, we can not expect something.
Quoted 7 lines
It looks like we have multiple goals here
1) avoid lockups, using workqueue +
cond_resched()seems to be sufficient, it has issue but nothing new
2) recover from the unresponsive device, the issue for timeout is that it needs to deal with false positives
I agree.
But I want to add a new goal, cvq async. In the netdim, we will send many requests via the cvq, so the cvq async will be nice.
Thanks.
Quoted 19 lines
Thanks
Quoted 15 lines
Thanks.
Quoted 10 lines
Thans
Quoted 6 lines
Zhu Yanjun
Quoted 2 lines
Andrew
Quoted 76 lines · Xuan Zhuo
Quoted 44 lines · Jason Wang
Quoted 43 lines · Xuan Zhuo
Quoted 41 lines · Jason Wang
Quoted 16 lines · Zhu Yanjun
Quoted 13 lines · Andrew Lunn
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
Review
Quoted 5 lines
while (!virtqueue_get_buf(vi->cvq, &tmp) &&(quoted diff) +3
-1- !virtqueue_is_broken(vi->cvq)) + !virtqueue_is_broken(vi->cvq)) { + if (timeout) + timeout--;Quoted 7 lines
This is not really a timeout, just a loop counter. 200 iterations could be a very short time on reasonable H/W. I guess this avoid the soft lockup, but possibly (likely?) breaks the functionality when we need to loop for some non negligible time.
I fear we need a more complex solution, as mentioned by Micheal in the thread you quoted.
Got it. I also look forward to the more complex solution to this problem.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Can we add a device capability (new feature bit) such as
ctrq_wait_timeoutto get a reasonable timeout?The usual solution to this is
include/linux/iopoll.h. If you can sleepread_poll_timeout()otherwiseread_poll_timeout_atomic().I read carefully the functions
read_poll_timeout()andread_poll_timeout_atomic(). The timeout is set by the caller of the 2 functions.FYI, in order to avoid a swtich of atomic or not, we need convert rx mode setting to workqueue first:
https://www.mail-archive.com/[email protected]/msg60298.html
Quoted 3 lines
As such, can we add a module parameter to customize this timeout value by the user?
Who is the "user" here, or how can the "user" know the value?
Quoted 3 lines
Or this timeout value is stored in device register,
virtio_netdriver will read this timeout value at initialization?Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
See another thread. The design needs to be general, or you can post a RFC.
In another thought, we've already had a tx watchdog, maybe we can have something similar to cvq and use timeout + reset in that case.
But we may block by the reset ^_^ if the device is broken?
I mean vq reset here.
I see.
I mean when the deivce is broken, the vq reset also many be blocked.
void vp_modern_set_queue_reset(struct virtio_pci_modern_device *mdev, u16 index) { struct virtio_pci_modern_common_cfg __iomem *cfg; cfg = (struct virtio_pci_modern_common_cfg __iomem *)mdev->common; vp_iowrite16(index, &cfg->cfg.queue_select); vp_iowrite16(1, &cfg->queue_reset); while (vp_ioread16(&cfg->queue_reset)) msleep(1); while (vp_ioread16(&cfg->cfg.queue_enable)) msleep(1); } EXPORT_SYMBOL_GPL(vp_modern_set_queue_reset);
Quoted 1 line
In this function, for the broken device, we can not expect something.
Yes, it's best effort, there's no guarantee then. But it doesn't harm to try.
Thanks
Quoted 39 lines
Quoted 9 lines
It looks like we have multiple goals here
1) avoid lockups, using workqueue +
cond_resched()seems to be sufficient, it has issue but nothing new
2) recover from the unresponsive device, the issue for timeout is that it needs to deal with false positivesI agree.
But I want to add a new goal, cvq async. In the netdim, we will send many requests via the cvq, so the cvq async will be nice.
Thanks.
Quoted 19 lines
Thanks
Quoted 15 lines
Thanks.
Quoted 10 lines
Thans
Quoted 6 lines
Zhu Yanjun
Quoted 2 lines
Andrew
Quoted 78 lines · Jason Wang
Quoted 76 lines · Xuan Zhuo
Quoted 44 lines · Jason Wang
Quoted 43 lines · Xuan Zhuo
Quoted 41 lines · Jason Wang
Quoted 16 lines · Zhu Yanjun
Quoted 13 lines · Andrew Lunn
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
Review
Quoted 5 lines
while (!virtqueue_get_buf(vi->cvq, &tmp) &&(quoted diff) +3
-1- !virtqueue_is_broken(vi->cvq)) + !virtqueue_is_broken(vi->cvq)) { + if (timeout) + timeout--;Quoted 7 lines
This is not really a timeout, just a loop counter. 200 iterations could be a very short time on reasonable H/W. I guess this avoid the soft lockup, but possibly (likely?) breaks the functionality when we need to loop for some non negligible time.
I fear we need a more complex solution, as mentioned by Micheal in the thread you quoted.
Got it. I also look forward to the more complex solution to this problem.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Can we add a device capability (new feature bit) such as
ctrq_wait_timeoutto get a reasonable timeout?The usual solution to this is
include/linux/iopoll.h. If you can sleepread_poll_timeout()otherwiseread_poll_timeout_atomic().I read carefully the functions
read_poll_timeout()andread_poll_timeout_atomic(). The timeout is set by the caller of the 2 functions.FYI, in order to avoid a swtich of atomic or not, we need convert rx mode setting to workqueue first:
https://www.mail-archive.com/[email protected]/msg60298.html
Quoted 3 lines
As such, can we add a module parameter to customize this timeout value by the user?
Who is the "user" here, or how can the "user" know the value?
Quoted 3 lines
Or this timeout value is stored in device register,
virtio_netdriver will read this timeout value at initialization?Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
See another thread. The design needs to be general, or you can post a RFC.
In another thought, we've already had a tx watchdog, maybe we can have something similar to cvq and use timeout + reset in that case.
But we may block by the reset ^_^ if the device is broken?
I mean vq reset here.
I see.
I mean when the deivce is broken, the vq reset also many be blocked.
void vp_modern_set_queue_reset(struct virtio_pci_modern_device *mdev, u16 index) { struct virtio_pci_modern_common_cfg __iomem *cfg; cfg = (struct virtio_pci_modern_common_cfg __iomem *)mdev->common; vp_iowrite16(index, &cfg->cfg.queue_select); vp_iowrite16(1, &cfg->queue_reset); while (vp_ioread16(&cfg->queue_reset)) msleep(1); while (vp_ioread16(&cfg->cfg.queue_enable)) msleep(1); } EXPORT_SYMBOL_GPL(vp_modern_set_queue_reset);
Quoted 21 lines
Quoted 1 line
In this function, for the broken device, we can not expect something.
Yes, it's best effort, there's no guarantee then. But it doesn't harm to try.
Thanks
Quoted 15 lines
Quoted 9 lines
It looks like we have multiple goals here
1) avoid lockups, using workqueue +
cond_resched()seems to be sufficient, it has issue but nothing new
2) recover from the unresponsive device, the issue for timeout is that it needs to deal with false positivesI agree.
But I want to add a new goal, cvq async. In the netdim, we will send many requests via the cvq, so the cvq async will be nice.
Then you need an interrupt for cvq.
FYI, I've posted a series that use interrupt for cvq in the past:
https://lore.kernel.org/lkml/[email protected]/t/
Haven't found time in working on this anymore, maybe we can start from this or not.
Thanks
Quoted 24 lines
Quoted 24 lines
Thanks.
Quoted 19 lines
Thanks
Quoted 15 lines
Thanks.
Quoted 10 lines
Thans
Quoted 6 lines
Zhu Yanjun
Quoted 2 lines
Andrew
Xuan Zhuo
reviewerQuoted 98 lines · Jason Wang
Quoted 97 lines · Jason Wang
Quoted 76 lines · Xuan Zhuo
Quoted 44 lines · Jason Wang
Quoted 43 lines · Xuan Zhuo
Quoted 41 lines · Jason Wang
Quoted 16 lines · Zhu Yanjun
Quoted 13 lines · Andrew Lunn
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
Review
Quoted 5 lines
while (!virtqueue_get_buf(vi->cvq, &tmp) &&(quoted diff) +3
-1- !virtqueue_is_broken(vi->cvq)) + !virtqueue_is_broken(vi->cvq)) { + if (timeout) + timeout--;Quoted 7 lines
This is not really a timeout, just a loop counter. 200 iterations could be a very short time on reasonable H/W. I guess this avoid the soft lockup, but possibly (likely?) breaks the functionality when we need to loop for some non negligible time.
I fear we need a more complex solution, as mentioned by Micheal in the thread you quoted.
Got it. I also look forward to the more complex solution to this problem.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Can we add a device capability (new feature bit) such as
ctrq_wait_timeoutto get a reasonable timeout?The usual solution to this is
include/linux/iopoll.h. If you can sleepread_poll_timeout()otherwiseread_poll_timeout_atomic().I read carefully the functions
read_poll_timeout()andread_poll_timeout_atomic(). The timeout is set by the caller of the 2 functions.FYI, in order to avoid a swtich of atomic or not, we need convert rx mode setting to workqueue first:
https://www.mail-archive.com/[email protected]/msg60298.html
Quoted 3 lines
As such, can we add a module parameter to customize this timeout value by the user?
Who is the "user" here, or how can the "user" know the value?
Quoted 3 lines
Or this timeout value is stored in device register,
virtio_netdriver will read this timeout value at initialization?Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
See another thread. The design needs to be general, or you can post a RFC.
In another thought, we've already had a tx watchdog, maybe we can have something similar to cvq and use timeout + reset in that case.
But we may block by the reset ^_^ if the device is broken?
I mean vq reset here.
I see.
I mean when the deivce is broken, the vq reset also many be blocked.
void vp_modern_set_queue_reset(struct virtio_pci_modern_device *mdev, u16 index) { struct virtio_pci_modern_common_cfg __iomem *cfg; cfg = (struct virtio_pci_modern_common_cfg __iomem *)mdev->common; vp_iowrite16(index, &cfg->cfg.queue_select); vp_iowrite16(1, &cfg->queue_reset); while (vp_ioread16(&cfg->queue_reset)) msleep(1); while (vp_ioread16(&cfg->cfg.queue_enable)) msleep(1); } EXPORT_SYMBOL_GPL(vp_modern_set_queue_reset);Quoted 1 line
In this function, for the broken device, we can not expect something.
Yes, it's best effort, there's no guarantee then. But it doesn't harm to try.
Thanks
Quoted 13 lines
Quoted 9 lines
It looks like we have multiple goals here
1) avoid lockups, using workqueue +
cond_resched()seems to be sufficient, it has issue but nothing new
2) recover from the unresponsive device, the issue for timeout is that it needs to deal with false positivesI agree.
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
But I want to add a new goal, cvq async. In the netdim, we will send many requests via the cvq, so the cvq async will be nice.
Then you need an interrupt for cvq.
FYI, I've posted a series that use interrupt for cvq in the past:
I know this. But the interrupt maybe not a good solution without new space.
Quoted 3 lines
Haven't found time in working on this anymore, maybe we can start from this or not.
I said async, but my aim is to put many requests to the cvq before getting the response.
Heng Qi posted this https://lore.kernel.org/all/[email protected]/
Thanks.
Quoted 28 lines
Thanks
Quoted 24 lines
Quoted 24 lines
Thanks.
Quoted 19 lines
Thanks
Quoted 15 lines
Thanks.
Quoted 10 lines
Thans
Quoted 6 lines
Zhu Yanjun
Quoted 2 lines
Andrew
Quoted 103 lines · Xuan Zhuo
Quoted 101 lines · Jason Wang
Quoted 99 lines · Jason Wang
Quoted 76 lines · Xuan Zhuo
Quoted 44 lines · Jason Wang
Quoted 43 lines · Xuan Zhuo
Quoted 41 lines · Jason Wang
Quoted 16 lines · Zhu Yanjun
Quoted 13 lines · Andrew Lunn
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
- !virtqueue_is_broken(vi->cvq))
+ !virtqueue_is_broken(vi->cvq)) {
+ if (timeout)
+ timeout--;Quoted 7 lines
This is not really a timeout, just a loop counter. 200 iterations could be a very short time on reasonable H/W. I guess this avoid the soft lockup, but possibly (likely?) breaks the functionality when we need to loop for some non negligible time.
I fear we need a more complex solution, as mentioned by Micheal in the thread you quoted.
Got it. I also look forward to the more complex solution to this problem.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Can we add a device capability (new feature bit) such as
ctrq_wait_timeoutto get a reasonable timeout?The usual solution to this is
include/linux/iopoll.h. If you can sleepread_poll_timeout()otherwiseread_poll_timeout_atomic().I read carefully the functions
read_poll_timeout()andread_poll_timeout_atomic(). The timeout is set by the caller of the 2 functions.FYI, in order to avoid a swtich of atomic or not, we need convert rx mode setting to workqueue first:
https://www.mail-archive.com/[email protected]/msg60298.html
Quoted 3 lines
As such, can we add a module parameter to customize this timeout value by the user?
Who is the "user" here, or how can the "user" know the value?
Quoted 3 lines
Or this timeout value is stored in device register,
virtio_netdriver will read this timeout value at initialization?Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
See another thread. The design needs to be general, or you can post a RFC.
In another thought, we've already had a tx watchdog, maybe we can have something similar to cvq and use timeout + reset in that case.
But we may block by the reset ^_^ if the device is broken?
I mean vq reset here.
I see.
I mean when the deivce is broken, the vq reset also many be blocked.
void vp_modern_set_queue_reset(struct virtio_pci_modern_device *mdev, u16 index) { struct virtio_pci_modern_common_cfg __iomem *cfg; cfg = (struct virtio_pci_modern_common_cfg __iomem *)mdev->common; vp_iowrite16(index, &cfg->cfg.queue_select); vp_iowrite16(1, &cfg->queue_reset); while (vp_ioread16(&cfg->queue_reset)) msleep(1); while (vp_ioread16(&cfg->cfg.queue_enable)) msleep(1); } EXPORT_SYMBOL_GPL(vp_modern_set_queue_reset);Quoted 1 line
In this function, for the broken device, we can not expect something.
Yes, it's best effort, there's no guarantee then. But it doesn't harm to try.
Thanks
Quoted 15 lines
Quoted 9 lines
It looks like we have multiple goals here
1) avoid lockups, using workqueue +
cond_resched()seems to be sufficient, it has issue but nothing new
2) recover from the unresponsive device, the issue for timeout is that it needs to deal with false positivesI agree.
But I want to add a new goal, cvq async. In the netdim, we will send many requests via the cvq, so the cvq async will be nice.
Quoted 7 lines
Quoted 5 lines
Then you need an interrupt for cvq.
FYI, I've posted a series that use interrupt for cvq in the past:
I know this. But the interrupt maybe not a good solution without new space.
What do you mean by "new space"?
We can introduce something like enable_cb_delayed(), then you will only get notified after several requests.
Quoted 8 lines
Quoted 4 lines
Haven't found time in working on this anymore, maybe we can start from this or not.
I said async, but my aim is to put many requests to the cvq before getting the response.
It doesn't differ from TX/RX in this case.
Quoted 3 lines
Heng Qi posted this https://lore.kernel.org/all/[email protected]/
This seems like a hack, if interrupt is used, you can simply do that in the callback.
Thanks
Quoted 32 lines
Thanks.
Quoted 28 lines
Thanks
Quoted 24 lines
Quoted 24 lines
Thanks.
Quoted 19 lines
Thanks
Quoted 15 lines
Thanks.
Quoted 10 lines
Thans
Quoted 6 lines
Zhu Yanjun
Quoted 2 lines
Andrew
Xuan Zhuo
reviewerQuoted 106 lines · Jason Wang
Quoted 105 lines · Xuan Zhuo
Quoted 103 lines · Jason Wang
Quoted 78 lines · Jason Wang
Quoted 76 lines · Xuan Zhuo
Quoted 44 lines · Jason Wang
Quoted 43 lines · Xuan Zhuo
Quoted 41 lines · Jason Wang
Quoted 16 lines · Zhu Yanjun
Quoted 13 lines · Andrew Lunn
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
- !virtqueue_is_broken(vi->cvq))
+ !virtqueue_is_broken(vi->cvq)) {
+ if (timeout)
+ timeout--;
This is not really a timeout, just a loop counter. 200 iterations could
be a very short time on reasonable H/W. I guess this avoid the soft
lockup, but possibly (likely?) breaks the functionality when we need to
loop for some non negligible time.I fear we need a more complex solution, as mentioned by Micheal in the
thread you quoted.Got it. I also look forward to the more complex solution to this problem.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Can we add a device capability (new feature bit) such as
ctrq_wait_timeoutto get a reasonable timeout?The usual solution to this is
include/linux/iopoll.h. If you can sleepread_poll_timeout()otherwiseread_poll_timeout_atomic().I read carefully the functions
read_poll_timeout()andread_poll_timeout_atomic(). The timeout is set by the caller of the 2 functions.FYI, in order to avoid a swtich of atomic or not, we need convert rx mode setting to workqueue first:
https://www.mail-archive.com/[email protected]/msg60298.html
Quoted 3 lines
As such, can we add a module parameter to customize this timeout value by the user?
Who is the "user" here, or how can the "user" know the value?
Quoted 3 lines
Or this timeout value is stored in device register,
virtio_netdriver will read this timeout value at initialization?Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
See another thread. The design needs to be general, or you can post a RFC.
In another thought, we've already had a tx watchdog, maybe we can have something similar to cvq and use timeout + reset in that case.
But we may block by the reset ^_^ if the device is broken?
I mean vq reset here.
I see.
I mean when the deivce is broken, the vq reset also many be blocked.
void vp_modern_set_queue_reset(struct virtio_pci_modern_device *mdev, u16 index) { struct virtio_pci_modern_common_cfg __iomem *cfg; cfg = (struct virtio_pci_modern_common_cfg __iomem *)mdev->common; vp_iowrite16(index, &cfg->cfg.queue_select); vp_iowrite16(1, &cfg->queue_reset); while (vp_ioread16(&cfg->queue_reset)) msleep(1); while (vp_ioread16(&cfg->cfg.queue_enable)) msleep(1); } EXPORT_SYMBOL_GPL(vp_modern_set_queue_reset);Quoted 21 lines
Quoted 1 line
In this function, for the broken device, we can not expect something.
Yes, it's best effort, there's no guarantee then. But it doesn't harm to try.
Thanks
Quoted 15 lines
Quoted 9 lines
It looks like we have multiple goals here
1) avoid lockups, using workqueue +
cond_resched()seems to be sufficient, it has issue but nothing new
2) recover from the unresponsive device, the issue for timeout is that it needs to deal with false positivesI agree.
But I want to add a new goal, cvq async. In the netdim, we will send many requests via the cvq, so the cvq async will be nice.
Then you need an interrupt for cvq.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
FYI, I've posted a series that use interrupt for cvq in the past:
I know this. But the interrupt maybe not a good solution without new space.
What do you mean by "new space"?
Yes, I know, the cvq can work with interrupt by the virtio spec. But as I know, many hypervisors implement the cvq without supporting interrupt. If we let the cvq work with interrupt without negotiation then many hypervisors will hang on the new kernel.
Quoted 13 lines
We can introduce something like
enable_cb_delayed(), then you will only get notified after several requests.Quoted 8 lines
Quoted 4 lines
Haven't found time in working on this anymore, maybe we can start from this or not.
I said async, but my aim is to put many requests to the cvq before getting the response.
Quoted 8 lines
It doesn't differ from TX/RX in this case.
Quoted 3 lines
Heng Qi posted this https://lore.kernel.org/all/[email protected]/
This seems like a hack, if interrupt is used, you can simply do that in the callback.
YES.
I also want to change the code, I just want to say the async is a goal.
For the rx mode, we have introduce a work queue, I want to move the sending command job to the work queue. The caller just wakeup the work queue.
If the caller want to got the result sync, then the caller can wait for it. If not, the caller can register an function to the work queue.
And I think it will be easy to implement the timeout inside the workqueue.
Thanks.
Quoted 36 lines
Thanks
Quoted 32 lines
Thanks.
Quoted 28 lines
Thanks
Quoted 24 lines
Quoted 24 lines
Thanks.
Quoted 19 lines
Thanks
Quoted 15 lines
Thanks.
Quoted 10 lines
Thans
Quoted 6 lines
Zhu Yanjun
Quoted 2 lines
Andrew
Quoted 110 lines · Xuan Zhuo
Quoted 108 lines · Jason Wang
Quoted 107 lines · Xuan Zhuo
Quoted 105 lines · Jason Wang
Quoted 78 lines · Jason Wang
Quoted 76 lines · Xuan Zhuo
Quoted 44 lines · Jason Wang
Quoted 43 lines · Xuan Zhuo
Quoted 41 lines · Jason Wang
Quoted 16 lines · Zhu Yanjun
Quoted 13 lines · Andrew Lunn
Quoted 13 lines
Quoted 13 lines
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
- !virtqueue_is_broken(vi->cvq))
+ !virtqueue_is_broken(vi->cvq)) {
+ if (timeout)
+ timeout--;
This is not really a timeout, just a loop counter. 200 iterations could
be a very short time on reasonable H/W. I guess this avoid the soft
lockup, but possibly (likely?) breaks the functionality when we need to
loop for some non negligible time.I fear we need a more complex solution, as mentioned by Micheal in the
thread you quoted.
Got it. I also look forward to the more complex solution to this problem.Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Can we add a device capability (new feature bit) such as
ctrq_wait_timeoutto get a reasonable timeout?The usual solution to this is
include/linux/iopoll.h. If you can sleepread_poll_timeout()otherwiseread_poll_timeout_atomic().I read carefully the functions
read_poll_timeout()andread_poll_timeout_atomic(). The timeout is set by the caller of the 2 functions.FYI, in order to avoid a swtich of atomic or not, we need convert rx mode setting to workqueue first:
https://www.mail-archive.com/[email protected]/msg60298.html
Quoted 3 lines
As such, can we add a module parameter to customize this timeout value by the user?
Who is the "user" here, or how can the "user" know the value?
Quoted 3 lines
Or this timeout value is stored in device register,
virtio_netdriver will read this timeout value at initialization?Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
See another thread. The design needs to be general, or you can post a RFC.
In another thought, we've already had a tx watchdog, maybe we can have something similar to cvq and use timeout + reset in that case.
But we may block by the reset ^_^ if the device is broken?
I mean vq reset here.
I see.
I mean when the deivce is broken, the vq reset also many be blocked.
void vp_modern_set_queue_reset(struct virtio_pci_modern_device *mdev, u16 index) { struct virtio_pci_modern_common_cfg __iomem *cfg; cfg = (struct virtio_pci_modern_common_cfg __iomem *)mdev->common; vp_iowrite16(index, &cfg->cfg.queue_select); vp_iowrite16(1, &cfg->queue_reset); while (vp_ioread16(&cfg->queue_reset)) msleep(1); while (vp_ioread16(&cfg->cfg.queue_enable)) msleep(1); } EXPORT_SYMBOL_GPL(vp_modern_set_queue_reset);Quoted 21 lines
Quoted 1 line
In this function, for the broken device, we can not expect something.
Yes, it's best effort, there's no guarantee then. But it doesn't harm to try.
Thanks
Quoted 15 lines
Quoted 9 lines
It looks like we have multiple goals here
1) avoid lockups, using workqueue +
cond_resched()seems to be sufficient, it has issue but nothing new
2) recover from the unresponsive device, the issue for timeout is that it needs to deal with false positivesI agree.
But I want to add a new goal, cvq async. In the netdim, we will send many requests via the cvq, so the cvq async will be nice.
Then you need an interrupt for cvq.
FYI, I've posted a series that use interrupt for cvq in the past:
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
I know this. But the interrupt maybe not a good solution without new space.
What do you mean by "new space"?
Yes, I know, the cvq can work with interrupt by the virtio spec. But as I know, many hypervisors implement the cvq without supporting interrupt.
It's a bug of the hypervisor that needs to be fix. Interrupt is provided by transport not the virtio itself.
Otherwise it can only support for Linux but not other OSes.
Quoted 29 lines
If we let the cvq work with interrupt without negotiation then many hypervisors will hang on the new kernel.
Quoted 13 lines
We can introduce something like
enable_cb_delayed(), then you will only get notified after several requests.Quoted 8 lines
Quoted 4 lines
Haven't found time in working on this anymore, maybe we can start from this or not.
I said async, but my aim is to put many requests to the cvq before getting the response.
Quoted 8 lines
It doesn't differ from TX/RX in this case.
Quoted 3 lines
Heng Qi posted this https://lore.kernel.org/all/[email protected]/
This seems like a hack, if interrupt is used, you can simply do that in the callback.
YES.
I also want to change the code, I just want to say the async is a goal.
Quoted 8 lines
For the rx mode, we have introduce a work queue, I want to move the sending command job to the work queue. The caller just wakeup the work queue.
If the caller want to got the result sync, then the caller can wait for it. If not, the caller can register an function to the work queue.
And I think it will be easy to implement the timeout inside the workqueue.
Looks much more complicated than a simple interrupt + timer/watchdog etc.
Thanks
Quoted 41 lines
Thanks.
Quoted 36 lines
Thanks
Quoted 32 lines
Thanks.
Quoted 28 lines
Thanks
Quoted 24 lines
Quoted 24 lines
Thanks.
Quoted 19 lines
Thanks
Quoted 15 lines
Thanks.
Quoted 10 lines
Thans
Quoted 6 lines
Zhu Yanjun
Quoted 2 lines
Andrew
Xuan Zhuo
reviewerQuoted 115 lines · Jason Wang
Quoted 114 lines · Xuan Zhuo
Quoted 112 lines · Jason Wang
Quoted 110 lines · Xuan Zhuo
Quoted 98 lines · Jason Wang
Quoted 97 lines · Jason Wang
Quoted 76 lines · Xuan Zhuo
Quoted 44 lines · Jason Wang
Quoted 43 lines · Xuan Zhuo
Quoted 41 lines · Jason Wang
Quoted 16 lines · Zhu Yanjun
Quoted 13 lines · Andrew Lunn
Quoted 13 lines
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
- !virtqueue_is_broken(vi->cvq))
+ !virtqueue_is_broken(vi->cvq)) {
+ if (timeout)
+ timeout--;
This is not really a timeout, just a loop counter. 200 iterations could
be a very short time on reasonable H/W. I guess this avoid the soft
lockup, but possibly (likely?) breaks the functionality when we need to
loop for some non negligible time.I fear we need a more complex solution, as mentioned by Micheal in the
thread you quoted.
Got it. I also look forward to the more complex solution to this problem.Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Can we add a device capability (new feature bit) such as ctrq_wait_timeout
to get a reasonable timeout?The usual solution to this is
include/linux/iopoll.h. If you can sleepread_poll_timeout()otherwiseread_poll_timeout_atomic().I read carefully the functions
read_poll_timeout()andread_poll_timeout_atomic(). The timeout is set by the caller of the 2 functions.FYI, in order to avoid a swtich of atomic or not, we need convert rx mode setting to workqueue first:
https://www.mail-archive.com/[email protected]/msg60298.html
Quoted 3 lines
As such, can we add a module parameter to customize this timeout value by the user?
Who is the "user" here, or how can the "user" know the value?
Quoted 3 lines
Or this timeout value is stored in device register,
virtio_netdriver will read this timeout value at initialization?Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
See another thread. The design needs to be general, or you can post a RFC.
In another thought, we've already had a tx watchdog, maybe we can have something similar to cvq and use timeout + reset in that case.
But we may block by the reset ^_^ if the device is broken?
I mean vq reset here.
I see.
I mean when the deivce is broken, the vq reset also many be blocked.
void vp_modern_set_queue_reset(struct virtio_pci_modern_device *mdev, u16 index) { struct virtio_pci_modern_common_cfg __iomem *cfg; cfg = (struct virtio_pci_modern_common_cfg __iomem *)mdev->common; vp_iowrite16(index, &cfg->cfg.queue_select); vp_iowrite16(1, &cfg->queue_reset); while (vp_ioread16(&cfg->queue_reset)) msleep(1); while (vp_ioread16(&cfg->cfg.queue_enable)) msleep(1); } EXPORT_SYMBOL_GPL(vp_modern_set_queue_reset);Quoted 1 line
In this function, for the broken device, we can not expect something.
Yes, it's best effort, there's no guarantee then. But it doesn't harm to try.
Thanks
Quoted 13 lines
Quoted 9 lines
It looks like we have multiple goals here
1) avoid lockups, using workqueue +
cond_resched()seems to be sufficient, it has issue but nothing new
2) recover from the unresponsive device, the issue for timeout is that it needs to deal with false positivesI agree.
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
But I want to add a new goal, cvq async. In the netdim, we will send many requests via the cvq, so the cvq async will be nice.
Then you need an interrupt for cvq.
FYI, I've posted a series that use interrupt for cvq in the past:
I know this. But the interrupt maybe not a good solution without new space.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
What do you mean by "new space"?
Yes, I know, the cvq can work with interrupt by the virtio spec. But as I know, many hypervisors implement the cvq without supporting interrupt.
It's a bug of the hypervisor that needs to be fix. Interrupt is provided by transport not the virtio itself.
YES. I agree.
But I still think we should not work with interrupt without any negotiation directly. I more like to introduce a new feature to enable this.
Thanks.
Quoted 90 lines
Otherwise it can only support for Linux but not other OSes.
Quoted 29 lines
If we let the cvq work with interrupt without negotiation then many hypervisors will hang on the new kernel.
Quoted 13 lines
We can introduce something like
enable_cb_delayed(), then you will only get notified after several requests.Quoted 8 lines
Quoted 4 lines
Haven't found time in working on this anymore, maybe we can start from this or not.
I said async, but my aim is to put many requests to the cvq before getting the response.
Quoted 8 lines
It doesn't differ from TX/RX in this case.
Quoted 3 lines
Heng Qi posted this https://lore.kernel.org/all/[email protected]/
This seems like a hack, if interrupt is used, you can simply do that in the callback.
YES.
I also want to change the code, I just want to say the async is a goal.
Quoted 8 lines
For the rx mode, we have introduce a work queue, I want to move the sending command job to the work queue. The caller just wakeup the work queue.
If the caller want to got the result sync, then the caller can wait for it. If not, the caller can register an function to the work queue.
And I think it will be easy to implement the timeout inside the workqueue.
Looks much more complicated than a simple interrupt + timer/watchdog etc.
Thanks
Quoted 41 lines
Thanks.
Quoted 36 lines
Thanks
Quoted 32 lines
Thanks.
Quoted 28 lines
Thanks
Quoted 24 lines
Quoted 24 lines
Thanks.
Quoted 19 lines
Thanks
Quoted 15 lines
Thanks.
Quoted 10 lines
Thans
Quoted 6 lines
Zhu Yanjun
Quoted 2 lines
Andrew
Quoted 122 lines · Xuan Zhuo
Quoted 120 lines · Jason Wang
Quoted 118 lines · Xuan Zhuo
Quoted 106 lines · Jason Wang
Quoted 105 lines · Xuan Zhuo
Quoted 103 lines · Jason Wang
Quoted 78 lines · Jason Wang
Quoted 76 lines · Xuan Zhuo
Quoted 44 lines · Jason Wang
Quoted 43 lines · Xuan Zhuo
Quoted 41 lines · Jason Wang
Quoted 16 lines · Zhu Yanjun
Quoted 13 lines · Andrew Lunn
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
- !virtqueue_is_broken(vi->cvq))
+ !virtqueue_is_broken(vi->cvq)) {
+ if (timeout)
+ timeout--;
This is not really a timeout, just a loop counter. 200 iterations could
be a very short time on reasonable H/W. I guess this avoid the soft
lockup, but possibly (likely?) breaks the functionality when we need to
loop for some non negligible time.I fear we need a more complex solution, as mentioned by Micheal in the
thread you quoted.
Got it. I also look forward to the more complex solution to this problem.Quoted 8 lines
Quoted 4 lines
Can we add a device capability (new feature bit) such as ctrq_wait_timeout
to get a reasonable timeout?
The usual solution to this is include/linux/iopoll.h. If you can sleep
read_poll_timeout() otherwise read_poll_timeout_atomic().I read carefully the functions
read_poll_timeout()andread_poll_timeout_atomic(). The timeout is set by the caller of the 2 functions.FYI, in order to avoid a swtich of atomic or not, we need convert rx mode setting to workqueue first:
https://www.mail-archive.com/[email protected]/msg60298.html
Quoted 3 lines
As such, can we add a module parameter to customize this timeout value by the user?
Who is the "user" here, or how can the "user" know the value?
Quoted 3 lines
Or this timeout value is stored in device register,
virtio_netdriver will read this timeout value at initialization?Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
See another thread. The design needs to be general, or you can post a RFC.
In another thought, we've already had a tx watchdog, maybe we can have something similar to cvq and use timeout + reset in that case.
But we may block by the reset ^_^ if the device is broken?
I mean vq reset here.
I see.
I mean when the deivce is broken, the vq reset also many be blocked.
void vp_modern_set_queue_reset(struct virtio_pci_modern_device *mdev, u16 index) { struct virtio_pci_modern_common_cfg __iomem *cfg; cfg = (struct virtio_pci_modern_common_cfg __iomem *)mdev->common; vp_iowrite16(index, &cfg->cfg.queue_select); vp_iowrite16(1, &cfg->queue_reset); while (vp_ioread16(&cfg->queue_reset)) msleep(1); while (vp_ioread16(&cfg->cfg.queue_enable)) msleep(1); } EXPORT_SYMBOL_GPL(vp_modern_set_queue_reset);Quoted 21 lines
Quoted 1 line
In this function, for the broken device, we can not expect something.
Yes, it's best effort, there's no guarantee then. But it doesn't harm to try.
Thanks
Quoted 15 lines
Quoted 9 lines
It looks like we have multiple goals here
1) avoid lockups, using workqueue +
cond_resched()seems to be sufficient, it has issue but nothing new
2) recover from the unresponsive device, the issue for timeout is that it needs to deal with false positivesI agree.
But I want to add a new goal, cvq async. In the netdim, we will send many requests via the cvq, so the cvq async will be nice.
Then you need an interrupt for cvq.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
FYI, I've posted a series that use interrupt for cvq in the past:
I know this. But the interrupt maybe not a good solution without new space.
What do you mean by "new space"?
Yes, I know, the cvq can work with interrupt by the virtio spec. But as I know, many hypervisors implement the cvq without supporting interrupt.
Quoted 7 lines
Quoted 2 lines
It's a bug of the hypervisor that needs to be fix. Interrupt is provided by transport not the virtio itself.
YES. I agree.
But I still think we should not work with interrupt without any negotiation directly. I more like to introduce a new feature to enable this.
I can hardly believe we need to workaround the issue of specific hypervisors like this...
Thanks
Quoted 4 lines
Thanks.
Xuan Zhuo
reviewerQuoted 126 lines · Jason Wang
Quoted 125 lines · Xuan Zhuo
Quoted 122 lines · Jason Wang
Quoted 110 lines · Xuan Zhuo
Quoted 108 lines · Jason Wang
Quoted 107 lines · Xuan Zhuo
Quoted 105 lines · Jason Wang
Quoted 78 lines · Jason Wang
Quoted 76 lines · Xuan Zhuo
Quoted 44 lines · Jason Wang
Quoted 43 lines · Xuan Zhuo
Quoted 41 lines · Jason Wang
Quoted 24 lines · Zhu Yanjun
在 2024/1/20 1:29, Andrew Lunn 写道:
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
- !virtqueue_is_broken(vi->cvq))
+ !virtqueue_is_broken(vi->cvq)) {
+ if (timeout)
+ timeout--;
This is not really a timeout, just a loop counter. 200 iterations could
be a very short time on reasonable H/W. I guess this avoid the soft
lockup, but possibly (likely?) breaks the functionality when we need to
loop for some non negligible time.I fear we need a more complex solution, as mentioned by Micheal in the
thread you quoted.
Got it. I also look forward to the more complex solution to this problem.
Can we add a device capability (new feature bit) such as ctrq_wait_timeout
to get a reasonable timeout?
The usual solution to this is include/linux/iopoll.h. If you can sleep
read_poll_timeout() otherwise read_poll_timeout_atomic().I read carefully the functions read_poll_timeout() and
read_poll_timeout_atomic(). The timeout is set by the caller of the 2
functions.FYI, in order to avoid a swtich of atomic or not, we need convert rx mode setting to workqueue first:
https://www.mail-archive.com/[email protected]/msg60298.html
Quoted 3 lines
As such, can we add a module parameter to customize this timeout value
by the user?Who is the "user" here, or how can the "user" know the value?
Quoted 3 lines
Or this timeout value is stored in device register, virtio_net driver
will read this timeout value at initialization?Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
See another thread. The design needs to be general, or you can post a RFC.
In another thought, we've already had a tx watchdog, maybe we can have something similar to cvq and use timeout + reset in that case.
But we may block by the reset ^_^ if the device is broken?
I mean vq reset here.
I see.
I mean when the deivce is broken, the vq reset also many be blocked.
void vp_modern_set_queue_reset(struct virtio_pci_modern_device *mdev, u16 index) { struct virtio_pci_modern_common_cfg __iomem *cfg; cfg = (struct virtio_pci_modern_common_cfg __iomem *)mdev->common; vp_iowrite16(index, &cfg->cfg.queue_select); vp_iowrite16(1, &cfg->queue_reset); while (vp_ioread16(&cfg->queue_reset)) msleep(1); while (vp_ioread16(&cfg->cfg.queue_enable)) msleep(1); } EXPORT_SYMBOL_GPL(vp_modern_set_queue_reset);Quoted 21 lines
Quoted 1 line
In this function, for the broken device, we can not expect something.
Yes, it's best effort, there's no guarantee then. But it doesn't harm to try.
Thanks
Quoted 15 lines
Quoted 9 lines
It looks like we have multiple goals here
1) avoid lockups, using workqueue +
cond_resched()seems to be sufficient, it has issue but nothing new
2) recover from the unresponsive device, the issue for timeout is that it needs to deal with false positivesI agree.
But I want to add a new goal, cvq async. In the netdim, we will send many requests via the cvq, so the cvq async will be nice.
Then you need an interrupt for cvq.
FYI, I've posted a series that use interrupt for cvq in the past:
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
I know this. But the interrupt maybe not a good solution without new space.
What do you mean by "new space"?
Yes, I know, the cvq can work with interrupt by the virtio spec. But as I know, many hypervisors implement the cvq without supporting interrupt.
It's a bug of the hypervisor that needs to be fix. Interrupt is provided by transport not the virtio itself.
Quoted 7 lines
Quoted 4 lines
YES. I agree.
But I still think we should not work with interrupt without any negotiation directly. I more like to introduce a new feature to enable this.
I can hardly believe we need to workaround the issue of specific hypervisors like this...
Maybe we should hear others.
We are ok for this. I just think for other hypervisors.
Thanks.
Quoted 8 lines
Thanks
Quoted 4 lines
Thanks.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 89 lines · Zhu Yanjun
Quoted 86 lines · Xuan Zhuo
Quoted 85 lines · Jason Wang
Quoted 64 lines · Jason Wang
Quoted 63 lines · Xuan Zhuo
Quoted 33 lines · Jason Wang
Quoted 32 lines · Xuan Zhuo
Quoted 31 lines · Jason Wang
Quoted 13 lines · Zhu Yanjun
Quoted 12 lines · Andrew Lunn
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
- !virtqueue_is_broken(vi->cvq))
+ !virtqueue_is_broken(vi->cvq)) {
+ if (timeout)
+ timeout--;
This is not really a timeout, just a loop counter. 200 iterations could
be a very short time on reasonable H/W. I guess this avoid the soft
lockup, but possibly (likely?) breaks the functionality when we need to
loop for some non negligible time.I fear we need a more complex solution, as mentioned by Micheal in the
thread you quoted.Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Got it. I also look forward to the more complex solution to this problem.
Can we add a device capability (new feature bit) such as
ctrq_wait_timeoutto get a reasonable timeout?The usual solution to this is
include/linux/iopoll.h. If you can sleepread_poll_timeout()otherwiseread_poll_timeout_atomic().I read carefully the functions
read_poll_timeout()andread_poll_timeout_atomic(). The timeout is set by the caller of the 2 functions.FYI, in order to avoid a swtich of atomic or not, we need convert rx mode setting to workqueue first:
https://www.mail-archive.com/[email protected]/msg60298.html
Quoted 2 lines
As such, can we add a module parameter to customize this timeout value by the user?
Who is the "user" here, or how can the "user" know the value?
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
Or this timeout value is stored in device register,
virtio_netdriver will read this timeout value at initialization?See another thread. The design needs to be general, or you can post a RFC.
In another thought, we've already had a tx watchdog, maybe we can have something similar to cvq and use timeout + reset in that case.
But we may block by the reset ^_^ if the device is broken?
I mean vq reset here.
I see.
I mean when the deivce is broken, the vq reset also many be blocked.
void vp_modern_set_queue_reset(struct virtio_pci_modern_device *mdev, u16 index) { struct virtio_pci_modern_common_cfg __iomem *cfg; cfg = (struct virtio_pci_modern_common_cfg __iomem *)mdev->common; vp_iowrite16(index, &cfg->cfg.queue_select); vp_iowrite16(1, &cfg->queue_reset); while (vp_ioread16(&cfg->queue_reset)) msleep(1); while (vp_ioread16(&cfg->cfg.queue_enable)) msleep(1); } EXPORT_SYMBOL_GPL(vp_modern_set_queue_reset);Quoted 15 lines
Quoted 1 line
In this function, for the broken device, we can not expect something.
Yes, it's best effort, there's no guarantee then. But it doesn't harm to try.
Thanks
Quoted 10 lines
Quoted 6 lines
It looks like we have multiple goals here
1) avoid lockups, using workqueue +
cond_resched()seems to be sufficient, it has issue but nothing new
2) recover from the unresponsive device, the issue for timeout is that it needs to deal with false positivesI agree.
But I want to add a new goal, cvq async. In the netdim, we will send many requests via the cvq, so the cvq async will be nice.
Then you need an interrupt for cvq.
FYI, I've posted a series that use interrupt for cvq in the past:
Quoted 8 lines
Quoted 8 lines
I know this. But the interrupt maybe not a good solution without new space.
Quoted 2 lines
Haven't found time in working on this anymore, maybe we can start from this or not.
I said async, but my aim is to put many requests to the cvq before getting the response.
Heng Qi posted thishttps://lore.kernel.org/all/[email protected]/
Sorry. This mail is rejected by netdev maillist. So I have to resend it.
Thanks a lot. I read Heng Qi's commits carefully. This patch series are similiar with the NIC feature xmit_more.
But if cvq command is urgent, can we let this urgent cvq command be passed ASAP?
I mean, can we set a flag similar to xmit_more? if cvq command is not urgent, it can be queued. If it is urgent,
this cvq command is passed ASAP.
Zhu Yanjun
Quoted 20 lines
Zhu Yanjun
Quoted 18 lines
Thanks.
Quoted 15 lines
Thanks
Quoted 13 lines
Quoted 13 lines
Thanks.
Quoted 10 lines
Thanks
Quoted 8 lines
Thanks.
Quoted 5 lines
Thans
Quoted 3 lines
Zhu Yanjun
Quoted 1 line
Andrew
Xuan Zhuo
reviewerQuoted 72 lines · Jason Wang
Quoted 71 lines · Xuan Zhuo
Quoted 44 lines · Jason Wang
Quoted 43 lines · Xuan Zhuo
Quoted 41 lines · Jason Wang
Quoted 16 lines · Zhu Yanjun
Quoted 13 lines · Andrew Lunn
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
Review
Quoted 5 lines
while (!virtqueue_get_buf(vi->cvq, &tmp) &&(quoted diff) +3
-1- !virtqueue_is_broken(vi->cvq)) + !virtqueue_is_broken(vi->cvq)) { + if (timeout) + timeout--;Quoted 7 lines
This is not really a timeout, just a loop counter. 200 iterations could be a very short time on reasonable H/W. I guess this avoid the soft lockup, but possibly (likely?) breaks the functionality when we need to loop for some non negligible time.
I fear we need a more complex solution, as mentioned by Micheal in the thread you quoted.
Got it. I also look forward to the more complex solution to this problem.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Can we add a device capability (new feature bit) such as
ctrq_wait_timeoutto get a reasonable timeout?The usual solution to this is
include/linux/iopoll.h. If you can sleepread_poll_timeout()otherwiseread_poll_timeout_atomic().I read carefully the functions
read_poll_timeout()andread_poll_timeout_atomic(). The timeout is set by the caller of the 2 functions.FYI, in order to avoid a swtich of atomic or not, we need convert rx mode setting to workqueue first:
https://www.mail-archive.com/[email protected]/msg60298.html
Quoted 3 lines
As such, can we add a module parameter to customize this timeout value by the user?
Who is the "user" here, or how can the "user" know the value?
Quoted 3 lines
Or this timeout value is stored in device register,
virtio_netdriver will read this timeout value at initialization?Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
See another thread. The design needs to be general, or you can post a RFC.
In another thought, we've already had a tx watchdog, maybe we can have something similar to cvq and use timeout + reset in that case.
But we may block by the reset ^_^ if the device is broken?
I mean vq reset here.
I see.
I mean when the deivce is broken, the vq reset also many be blocked.
void vp_modern_set_queue_reset(struct virtio_pci_modern_device *mdev, u16 index) { struct virtio_pci_modern_common_cfg __iomem *cfg; cfg = (struct virtio_pci_modern_common_cfg __iomem *)mdev->common; vp_iowrite16(index, &cfg->cfg.queue_select); vp_iowrite16(1, &cfg->queue_reset); while (vp_ioread16(&cfg->queue_reset)) msleep(1);
Quoted 9 lines
Quoted 6 lines
while (vp_ioread16(&cfg->cfg.queue_enable)) msleep(1); } EXPORT_SYMBOL_GPL(vp_modern_set_queue_reset);In this function, for the broken device, we can not expect something.
Yes, it's best effort, there's no guarantee then. But it doesn't harm to try.
YES. I agree.
Thanks.
Quoted 42 lines
Thanks
Quoted 39 lines
Quoted 9 lines
It looks like we have multiple goals here
1) avoid lockups, using workqueue +
cond_resched()seems to be sufficient, it has issue but nothing new
2) recover from the unresponsive device, the issue for timeout is that it needs to deal with false positivesI agree.
But I want to add a new goal, cvq async. In the netdim, we will send many requests via the cvq, so the cvq async will be nice.
Thanks.
Quoted 19 lines
Thanks
Quoted 15 lines
Thanks.
Quoted 10 lines
Thans
Quoted 6 lines
Zhu Yanjun
Quoted 2 lines
Andrew
Quoted 167 lines · Heng Qi
Quoted 163 lines · Zhu Yanjun
Quoted 161 lines · Paolo Abeni
Quoted 151 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
Some devices emulate the
virtio_nethardwares. Whenvirtio_netdriver sends commands to the emulated hardware, normally the hardware needs time to response. Sometimes the time is very long. Thus, the following will appear. Then the whole system will hang.
The similar problems also occur in Intel NICs and Mellanox NICs. As such, the similar solution is borrowed from them. A timeout value is added and the timeout value as large as possible is set to ensure that the driver gets the maximum possible response from the hardware."
[ 213.795860] watchdog: BUG: soft lockup - CPU#108 stuck for 26s! [(udev-worker):3157] [ 213.796114] Modules linked in: virtio_net(+) net_failoverfailover qrtr rfkill sunrpc
intel_rapl_msrintel_rapl_commonintel_uncore_frequencyintel_uncore_frequency_commonintel_ifsi10nm_edacnfit libnvdimmx86_pkg_temp_thermalintel_powerclampcoretempiTCO_wdtraplintel_pmc_bxtdax_hmemiTCO_vendor_supportvfatcxl_acpiintel_cstatepmt_telemetrypmt_classintel_sdsijoydevintel_uncorecxl_corefat pcspkrmei_meisst_if_mbox_pciisst_if_mmioidxdi2c_i801isst_if_commonmeiintel_vsecidxd_busi2c_smbusi2c_ismtipmi_ssifacpi_ipmiipmi_siipmi_devintfipmi_msghandleracpi_padacpi_power_meterpfr_telemetrypfr_updatefuse loop zram xfscrct10dif_pclmulcrc32_pclmulcrc32c_intelpolyval_clmulnipolyval_genericghash_clmulni_intelsha512_ssse3bnxt_ensha256_ssse3sha1_ssse3nvme astnvme_corei2c_algo_bitwmipinctrl_emmitsburgscsi_dh_rdacscsi_dh_emcscsi_dh_aluadm_multipath[ 213.796194] irq event stamp: 67740 [ 213.796195] hardirqs last enabled at (67739): [<ffffffff8c2015ca>] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 213.796203] hardirqs last disabled at (67740): [<ffffffff8c14108e>] sysvec_apic_timer_interrupt+0xe/0x90 [ 213.796208] softirqs last enabled at (67686): [<ffffffff8b12115e>] __irq_exit_rcu+0xbe/0xe0 [ 213.796214] softirqs last disabled at (67681): [<ffffffff8b12115e>] __irq_exit_rcu+0xbe/0xe0 [ 213.796217] CPU: 108 PID: 3157 Comm: (udev-worker) Kdump: loaded Not tainted 6.7.0+ #9 [ 213.796220] Hardware name: Intel Corporation M50FCP2SBSTD/M50FCP2SBSTD, BIOS SE5C741.86B.01.01.0001.2211140926 11/14/2022 [ 213.796221] RIP: 0010:virtqueue_get_buf_ctx_split+0x8d/0x110 [ 213.796228] Code: 89 df e8 26 fe ff ff 0f b7 43 50 83 c0 01 66 89 43 50 f6 43 78 01 75 12 80 7b 42 00 48 8b 4b 68 8b 53 58 74 0f 66 87 44 51 04 <48> 89 e8 5b 5d c3 cc cc cc cc 66 89 44 51 04 0f ae f0 48 89 e8 5b [ 213.796230] RSP: 0018:ff4bbb362306f9b0 EFLAGS: 00000246 [ 213.796233] RAX: 0000000000000000 RBX: ff2f15095896f000 RCX: 0000000000000001 [ 213.796235] RDX: 0000000000000000 RSI: ff4bbb362306f9cc RDI: ff2f15095896f000 [ 213.796236] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 213.796237] R10: 0000000000000003 R11: ff2f15095893cc40 R12: 0000000000000002 [ 213.796239] R13: 0000000000000004 R14: 0000000000000000 R15: ff2f1509534f3000 [ 213.796240] FS: 00007f775847d0c0(0000) GS:ff2f1528bac00000(0000) knlGS:0000000000000000 [ 213.796242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 213.796243] CR2: 0000557f987b6e70 CR3: 0000002098602006 CR4: 0000000000f71ef0 [ 213.796245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 213.796246] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 [ 213.796247] PKRU: 55555554 [ 213.796249] Call Trace: [ 213.796250] <IRQ> [ 213.796252] ? watchdog_timer_fn+0x1c0/0x220 [ 213.796258] ? __pfx_watchdog_timer_fn+0x10/0x10 [ 213.796261] ? __hrtimer_run_queues+0x1af/0x380 [ 213.796269] ? hrtimer_interrupt+0xf8/0x230 [ 213.796274] ? __sysvec_apic_timer_interrupt+0x64/0x1a0 [ 213.796279] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 213.796282] </IRQ> [ 213.796284] <TASK> [ 213.796285] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 213.796293] ? virtqueue_get_buf_ctx_split+0x8d/0x110 [ 213.796297] virtnet_send_command+0x18a/0x1f0 [virtio_net] [ 213.796310] _virtnet_set_queues+0xc6/0x120 [virtio_net] [ 213.796319] virtnet_probe+0xa06/0xd50 [virtio_net] [ 213.796328] virtio_dev_probe+0x195/0x230 [ 213.796333] really_probe+0x19f/0x400 [ 213.796338] ? __pfx___driver_attach+0x10/0x10 [ 213.796340] __driver_probe_device+0x78/0x160 [ 213.796343] driver_probe_device+0x1f/0x90 [ 213.796346] __driver_attach+0xd6/0x1d0 [ 213.796349] bus_for_each_dev+0x8c/0xe0 [ 213.796355] bus_add_driver+0x119/0x220 [ 213.796359] driver_register+0x59/0x100 [ 213.796362] ? __pfx_virtio_net_driver_init+0x10/0x10 [virtio_net] [ 213.796369] virtio_net_driver_init+0x8e/0xff0 [virtio_net] [ 213.796375] do_one_initcall+0x6f/0x380 [ 213.796384] do_init_module+0x60/0x240 [ 213.796388] init_module_from_file+0x86/0xc0 [ 213.796396] idempotent_init_module+0x129/0x2c0 [ 213.796406] __x64_sys_finit_module+0x5e/0xb0 [ 213.796409] do_syscall_64+0x60/0xe0 [ 213.796415] ? do_syscall_64+0x6f/0xe0 [ 213.796418] ? lockdep_hardirqs_on_prepare+0xe4/0x1a0 [ 213.796424] ? do_syscall_64+0x6f/0xe0 [ 213.796427] ? do_syscall_64+0x6f/0xe0 [ 213.796431] entry_SYSCALL_64_after_hwframe+0x6e/0x76 [ 213.796435] RIP: 0033:0x7f7758f279cd [ 213.796465] Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 33 e4 0c 00 f7 d8 64 89 01 48 [ 213.796467] RSP: 002b:00007ffe2cad8738 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 213.796469] RAX: ffffffffffffffda RBX: 0000557f987a8180 RCX: 00007f7758f279cd [ 213.796471] RDX: 0000000000000000 RSI: 00007f77593e5453 RDI: 000000000000000f [ 213.796472] RBP: 00007f77593e5453 R08: 0000000000000000 R09: 00007ffe2cad8860 [ 213.796473] R10: 000000000000000f R11: 0000000000000246 R12: 0000000000020000 [ 213.796475] R13: 0000557f9879f8e0 R14: 0000000000000000 R15: 0000557f98783aa0 [ 213.796482] </TASK>"
Signed-off-by Zhu Yanjun <[email protected]>--- drivers/net/virtio_net.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)drivers/net/virtio_net.c +1
-1@@ -2468,7 +2468,7 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, { struct scatterlist *sgs[4], hdr, stat;unsigned out_num =0 , tmp; -int ret; +int ret, timeout =200 ;/* Caller should know better */ BUG_ON (!virtio_has_feature (vi->vdev,VIRTIO_NET_F_CTRL_VQ ));@@ -2502,8 +2502,14 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, * into the hypervisor, so the request should be handledimmediately.
*/ while (!virtqueue_get_buf(vi->cvq, &tmp) &&drivers/net/virtio_net.c static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, · line 2507
Quoted 4 lines
drivers/net/virtio_net.c +3
-1- !virtqueue_is_broken(vi->cvq)) + !virtqueue_is_broken(vi->cvq)) { +if (timeout) + timeout--;This is not really a timeout, just a loop counter. 200 iterations could be a very short time on reasonable H/W. I guess this avoid the soft lockup, but possibly (likely?) breaks the functionality when we need to loop for some non negligible time.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
I fear we need a more complex solution, as mentioned by Micheal in the thread you quoted.
Got it. I also look forward to the more complex solution to this problem.
Can we add a device capability (new feature bit) such as
ctrq_wait_timeoutto get a reasonable timeout?
This adds another kind of complexity for migration compatibility.
And we need to make it more general, e.g
1) it should not be cvq specific
2) or we can have a timeout that works for all queues
?
Thanks
Quoted 13 lines
Thanks,
HengQuoted 8 lines
Zhu Yanjun
Quoted 5 lines
Cheers,
Paolo
Quoted 166 lines · Jason Wang
Quoted 165 lines · Heng Qi
Quoted 162 lines · Zhu Yanjun
Quoted 161 lines · Paolo Abeni
Quoted 151 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
Some devices emulate the
virtio_nethardwares. Whenvirtio_netdriver sends commands to the emulated hardware, normally the hardware needs time to response. Sometimes the time is very long. Thus, the following will appear. Then the whole system will hang.
The similar problems also occur in Intel NICs and Mellanox NICs. As such, the similar solution is borrowed from them. A timeout value is added and the timeout value as large as possible is set to ensure that the driver gets the maximum possible response from the hardware."
[ 213.795860] watchdog: BUG: soft lockup - CPU#108 stuck for 26s! [(udev-worker):3157] [ 213.796114] Modules linked in: virtio_net(+) net_failoverfailover qrtr rfkill sunrpc
intel_rapl_msrintel_rapl_commonintel_uncore_frequencyintel_uncore_frequency_commonintel_ifsi10nm_edacnfit libnvdimmx86_pkg_temp_thermalintel_powerclampcoretempiTCO_wdtraplintel_pmc_bxtdax_hmemiTCO_vendor_supportvfatcxl_acpiintel_cstatepmt_telemetrypmt_classintel_sdsijoydevintel_uncorecxl_corefat pcspkrmei_meisst_if_mbox_pciisst_if_mmioidxdi2c_i801isst_if_commonmeiintel_vsecidxd_busi2c_smbusi2c_ismtipmi_ssifacpi_ipmiipmi_siipmi_devintfipmi_msghandleracpi_padacpi_power_meterpfr_telemetrypfr_updatefuse loop zram xfscrct10dif_pclmulcrc32_pclmulcrc32c_intelpolyval_clmulnipolyval_genericghash_clmulni_intelsha512_ssse3bnxt_ensha256_ssse3sha1_ssse3nvme astnvme_corei2c_algo_bitwmipinctrl_emmitsburgscsi_dh_rdacscsi_dh_emcscsi_dh_aluadm_multipath[ 213.796194] irq event stamp: 67740 [ 213.796195] hardirqs last enabled at (67739): [<ffffffff8c2015ca>] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 213.796203] hardirqs last disabled at (67740): [<ffffffff8c14108e>] sysvec_apic_timer_interrupt+0xe/0x90 [ 213.796208] softirqs last enabled at (67686): [<ffffffff8b12115e>] __irq_exit_rcu+0xbe/0xe0 [ 213.796214] softirqs last disabled at (67681): [<ffffffff8b12115e>] __irq_exit_rcu+0xbe/0xe0 [ 213.796217] CPU: 108 PID: 3157 Comm: (udev-worker) Kdump: loaded Not tainted 6.7.0+ #9 [ 213.796220] Hardware name: Intel Corporation M50FCP2SBSTD/M50FCP2SBSTD, BIOS SE5C741.86B.01.01.0001.2211140926 11/14/2022 [ 213.796221] RIP: 0010:virtqueue_get_buf_ctx_split+0x8d/0x110 [ 213.796228] Code: 89 df e8 26 fe ff ff 0f b7 43 50 83 c0 01 66 89 43 50 f6 43 78 01 75 12 80 7b 42 00 48 8b 4b 68 8b 53 58 74 0f 66 87 44 51 04 <48> 89 e8 5b 5d c3 cc cc cc cc 66 89 44 51 04 0f ae f0 48 89 e8 5b [ 213.796230] RSP: 0018:ff4bbb362306f9b0 EFLAGS: 00000246 [ 213.796233] RAX: 0000000000000000 RBX: ff2f15095896f000 RCX: 0000000000000001 [ 213.796235] RDX: 0000000000000000 RSI: ff4bbb362306f9cc RDI: ff2f15095896f000 [ 213.796236] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 213.796237] R10: 0000000000000003 R11: ff2f15095893cc40 R12: 0000000000000002 [ 213.796239] R13: 0000000000000004 R14: 0000000000000000 R15: ff2f1509534f3000 [ 213.796240] FS: 00007f775847d0c0(0000) GS:ff2f1528bac00000(0000) knlGS:0000000000000000 [ 213.796242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 213.796243] CR2: 0000557f987b6e70 CR3: 0000002098602006 CR4: 0000000000f71ef0 [ 213.796245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 213.796246] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 [ 213.796247] PKRU: 55555554 [ 213.796249] Call Trace: [ 213.796250] <IRQ> [ 213.796252] ? watchdog_timer_fn+0x1c0/0x220 [ 213.796258] ? __pfx_watchdog_timer_fn+0x10/0x10 [ 213.796261] ? __hrtimer_run_queues+0x1af/0x380 [ 213.796269] ? hrtimer_interrupt+0xf8/0x230 [ 213.796274] ? __sysvec_apic_timer_interrupt+0x64/0x1a0 [ 213.796279] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 213.796282] </IRQ> [ 213.796284] <TASK> [ 213.796285] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 213.796293] ? virtqueue_get_buf_ctx_split+0x8d/0x110 [ 213.796297] virtnet_send_command+0x18a/0x1f0 [virtio_net] [ 213.796310] _virtnet_set_queues+0xc6/0x120 [virtio_net] [ 213.796319] virtnet_probe+0xa06/0xd50 [virtio_net] [ 213.796328] virtio_dev_probe+0x195/0x230 [ 213.796333] really_probe+0x19f/0x400 [ 213.796338] ? __pfx___driver_attach+0x10/0x10 [ 213.796340] __driver_probe_device+0x78/0x160 [ 213.796343] driver_probe_device+0x1f/0x90 [ 213.796346] __driver_attach+0xd6/0x1d0 [ 213.796349] bus_for_each_dev+0x8c/0xe0 [ 213.796355] bus_add_driver+0x119/0x220 [ 213.796359] driver_register+0x59/0x100 [ 213.796362] ? __pfx_virtio_net_driver_init+0x10/0x10 [virtio_net] [ 213.796369] virtio_net_driver_init+0x8e/0xff0 [virtio_net] [ 213.796375] do_one_initcall+0x6f/0x380 [ 213.796384] do_init_module+0x60/0x240 [ 213.796388] init_module_from_file+0x86/0xc0 [ 213.796396] idempotent_init_module+0x129/0x2c0 [ 213.796406] __x64_sys_finit_module+0x5e/0xb0 [ 213.796409] do_syscall_64+0x60/0xe0 [ 213.796415] ? do_syscall_64+0x6f/0xe0 [ 213.796418] ? lockdep_hardirqs_on_prepare+0xe4/0x1a0 [ 213.796424] ? do_syscall_64+0x6f/0xe0 [ 213.796427] ? do_syscall_64+0x6f/0xe0 [ 213.796431] entry_SYSCALL_64_after_hwframe+0x6e/0x76 [ 213.796435] RIP: 0033:0x7f7758f279cd [ 213.796465] Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 33 e4 0c 00 f7 d8 64 89 01 48 [ 213.796467] RSP: 002b:00007ffe2cad8738 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 213.796469] RAX: ffffffffffffffda RBX: 0000557f987a8180 RCX: 00007f7758f279cd [ 213.796471] RDX: 0000000000000000 RSI: 00007f77593e5453 RDI: 000000000000000f [ 213.796472] RBP: 00007f77593e5453 R08: 0000000000000000 R09: 00007ffe2cad8860 [ 213.796473] R10: 000000000000000f R11: 0000000000000246 R12: 0000000000020000 [ 213.796475] R13: 0000557f9879f8e0 R14: 0000000000000000 R15: 0000557f98783aa0 [ 213.796482] </TASK>"
Signed-off-by Zhu Yanjun <[email protected]>--- drivers/net/virtio_net.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)drivers/net/virtio_net.c +1
-1@@ -2468,7 +2468,7 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, { struct scatterlist *sgs[4], hdr, stat;unsigned out_num =0 , tmp; -int ret; +int ret, timeout =200 ;/* Caller should know better */ BUG_ON (!virtio_has_feature (vi->vdev,VIRTIO_NET_F_CTRL_VQ ));@@ -2502,8 +2502,14 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, * into the hypervisor, so the request should be handledimmediately.
*/ while (!virtqueue_get_buf(vi->cvq, &tmp) &&drivers/net/virtio_net.c static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, · line 2507
Quoted 4 lines
drivers/net/virtio_net.c +3
-1- !virtqueue_is_broken(vi->cvq)) + !virtqueue_is_broken(vi->cvq)) { +if (timeout) + timeout--;This is not really a timeout, just a loop counter. 200 iterations could be a very short time on reasonable H/W. I guess this avoid the soft lockup, but possibly (likely?) breaks the functionality when we need to loop for some non negligible time.
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
I fear we need a more complex solution, as mentioned by Micheal in the thread you quoted.
Got it. I also look forward to the more complex solution to this problem.
Can we add a device capability (new feature bit) such as
ctrq_wait_timeoutto get a reasonable timeout?This adds another kind of complexity for migration compatibility.
Yes, this requires device adaptation. But the value will be more reasonable, different devices have different requirements for timeout.
Quoted 4 lines
And we need to make it more general, e.g
1) it should not be cvq specific
Agree. Especially considering avq also has this problem.
Quoted 1 line
2) or we can have a timeout that works for all queues
I remember Parav mentioned that they were also making some bulk improvements to ctrlq/avq, or we should look at a combined solution.
rxq timeout can be used for our detection aliveness needs, especially after a live migration. txq timeout can be used with watchdog + vq reset.
Thanks,
Heng
Quoted 13 lines
?
Thanks
Quoted 9 lines
Thanks,
HengQuoted 6 lines
Zhu Yanjun
Quoted 4 lines
Cheers,
Paolo
Michael S. Tsirkin
maintainerQuoted 118 lines · Paolo Abeni
Quoted 104 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
Some devices emulate the
virtio_nethardwares. Whenvirtio_netdriver sends commands to the emulated hardware, normally the hardware needs time to response. Sometimes the time is very long. Thus, the following will appear. Then the whole system will hang.
The similar problems also occur in Intel NICs and Mellanox NICs. As such, the similar solution is borrowed from them. A timeout value is added and the timeout value as large as possible is set to ensure that the driver gets the maximum possible response from the hardware."
[ 213.795860] watchdog: BUG: soft lockup - CPU#108 stuck for 26s! [(udev-worker):3157] [ 213.796114] Modules linked in: virtio_net(+) net_failover failover qrtr rfkill sunrpc intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common intel_ifs i10nm_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt rapl intel_pmc_bxt dax_hmem iTCO_vendor_support vfat cxl_acpi intel_cstate pmt_telemetry pmt_class intel_sdsi joydev intel_uncore cxl_core fat pcspkr mei_me isst_if_mbox_pci isst_if_mmio idxd i2c_i801 isst_if_common mei intel_vsec idxd_bus i2c_smbus i2c_ismt ipmi_ssif acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter pfr_telemetry pfr_update fuse loop zram xfs crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 bnxt_en sha256_ssse3 sha1_ssse3 nvme ast nvme_core i2c_algo_bit wmi pinctrl_emmitsburg scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_multipath [ 213.796194] irq event stamp: 67740 [ 213.796195] hardirqs last enabled at (67739): [<ffffffff8c2015ca>] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 213.796203] hardirqs last disabled at (67740): [<ffffffff8c14108e>] sysvec_apic_timer_interrupt+0xe/0x90 [ 213.796208] softirqs last enabled at (67686): [<ffffffff8b12115e>] __irq_exit_rcu+0xbe/0xe0 [ 213.796214] softirqs last disabled at (67681): [<ffffffff8b12115e>] __irq_exit_rcu+0xbe/0xe0 [ 213.796217] CPU: 108 PID: 3157 Comm: (udev-worker) Kdump: loaded Not tainted 6.7.0+ #9 [ 213.796220] Hardware name: Intel Corporation M50FCP2SBSTD/M50FCP2SBSTD, BIOS SE5C741.86B.01.01.0001.2211140926 11/14/2022 [ 213.796221] RIP: 0010:virtqueue_get_buf_ctx_split+0x8d/0x110 [ 213.796228] Code: 89 df e8 26 fe ff ff 0f b7 43 50 83 c0 01 66 89 43 50 f6 43 78 01 75 12 80 7b 42 00 48 8b 4b 68 8b 53 58 74 0f 66 87 44 51 04 <48> 89 e8 5b 5d c3 cc cc cc cc 66 89 44 51 04 0f ae f0 48 89 e8 5b [ 213.796230] RSP: 0018:ff4bbb362306f9b0 EFLAGS: 00000246 [ 213.796233] RAX: 0000000000000000 RBX: ff2f15095896f000 RCX: 0000000000000001 [ 213.796235] RDX: 0000000000000000 RSI: ff4bbb362306f9cc RDI: ff2f15095896f000 [ 213.796236] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 213.796237] R10: 0000000000000003 R11: ff2f15095893cc40 R12: 0000000000000002 [ 213.796239] R13: 0000000000000004 R14: 0000000000000000 R15: ff2f1509534f3000 [ 213.796240] FS: 00007f775847d0c0(0000) GS:ff2f1528bac00000(0000) knlGS:0000000000000000 [ 213.796242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 213.796243] CR2: 0000557f987b6e70 CR3: 0000002098602006 CR4: 0000000000f71ef0 [ 213.796245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 213.796246] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 [ 213.796247] PKRU: 55555554 [ 213.796249] Call Trace: [ 213.796250] <IRQ> [ 213.796252] ? watchdog_timer_fn+0x1c0/0x220 [ 213.796258] ? __pfx_watchdog_timer_fn+0x10/0x10 [ 213.796261] ? __hrtimer_run_queues+0x1af/0x380 [ 213.796269] ? hrtimer_interrupt+0xf8/0x230 [ 213.796274] ? __sysvec_apic_timer_interrupt+0x64/0x1a0 [ 213.796279] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 213.796282] </IRQ> [ 213.796284] <TASK> [ 213.796285] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 213.796293] ? virtqueue_get_buf_ctx_split+0x8d/0x110 [ 213.796297] virtnet_send_command+0x18a/0x1f0 [virtio_net] [ 213.796310] _virtnet_set_queues+0xc6/0x120 [virtio_net] [ 213.796319] virtnet_probe+0xa06/0xd50 [virtio_net] [ 213.796328] virtio_dev_probe+0x195/0x230 [ 213.796333] really_probe+0x19f/0x400 [ 213.796338] ? __pfx___driver_attach+0x10/0x10 [ 213.796340] __driver_probe_device+0x78/0x160 [ 213.796343] driver_probe_device+0x1f/0x90 [ 213.796346] __driver_attach+0xd6/0x1d0 [ 213.796349] bus_for_each_dev+0x8c/0xe0 [ 213.796355] bus_add_driver+0x119/0x220 [ 213.796359] driver_register+0x59/0x100 [ 213.796362] ? __pfx_virtio_net_driver_init+0x10/0x10 [virtio_net] [ 213.796369] virtio_net_driver_init+0x8e/0xff0 [virtio_net] [ 213.796375] do_one_initcall+0x6f/0x380 [ 213.796384] do_init_module+0x60/0x240 [ 213.796388] init_module_from_file+0x86/0xc0 [ 213.796396] idempotent_init_module+0x129/0x2c0 [ 213.796406] __x64_sys_finit_module+0x5e/0xb0 [ 213.796409] do_syscall_64+0x60/0xe0 [ 213.796415] ? do_syscall_64+0x6f/0xe0 [ 213.796418] ? lockdep_hardirqs_on_prepare+0xe4/0x1a0 [ 213.796424] ? do_syscall_64+0x6f/0xe0 [ 213.796427] ? do_syscall_64+0x6f/0xe0 [ 213.796431] entry_SYSCALL_64_after_hwframe+0x6e/0x76 [ 213.796435] RIP: 0033:0x7f7758f279cd [ 213.796465] Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 33 e4 0c 00 f7 d8 64 89 01 48 [ 213.796467] RSP: 002b:00007ffe2cad8738 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 213.796469] RAX: ffffffffffffffda RBX: 0000557f987a8180 RCX: 00007f7758f279cd [ 213.796471] RDX: 0000000000000000 RSI: 00007f77593e5453 RDI: 000000000000000f [ 213.796472] RBP: 00007f77593e5453 R08: 0000000000000000 R09: 00007ffe2cad8860 [ 213.796473] R10: 000000000000000f R11: 0000000000000246 R12: 0000000000020000 [ 213.796475] R13: 0000557f9879f8e0 R14: 0000000000000000 R15: 0000557f98783aa0 [ 213.796482] </TASK>"
Signed-off-by Zhu Yanjun <[email protected]>--- drivers/net/virtio_net.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)drivers/net/virtio_net.c +1
-1@@ -2468,7 +2468,7 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, {struct scatterlist *sgs[4 ], hdr, stat;unsigned out_num =0 , tmp; -int ret; +int ret, timeout =200 ;/* Caller should know better */ BUG_ON (!virtio_has_feature (vi->vdev,VIRTIO_NET_F_CTRL_VQ ));drivers/net/virtio_net.c static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, · line 2507
Quoted 7 lines
drivers/net/virtio_net.c +3
-1@@ -2502,8 +2502,14 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +if (timeout) + timeout--;This is not really a timeout, just a loop counter. 200 iterations could be a very short time on reasonable H/W. I guess this avoid the soft lockup, but possibly (likely?) breaks the functionality when we need to loop for some non negligible time.
Quoted 6 lines
I fear we need a more complex solution, as mentioned by Micheal in the thread you quoted.
Cheers,
Paolo
So a bunch of devices just queue a rquest on a workqueue. Problem is, errors are not reported then. I really feel this is something netdev core should support better: return a "deferred" status and have netdev core wait outside a lock on a waitqueue.
Signature · 1 line
MST
Xuan Zhuo
reviewerQuoted 122 lines · Michael S. Tsirkin
Quoted 121 lines · Paolo Abeni
Quoted 104 lines · Zhu Yanjun
From: Zhu Yanjun <[email protected]>
Some devices emulate the
virtio_nethardwares. Whenvirtio_netdriver sends commands to the emulated hardware, normally the hardware needs time to response. Sometimes the time is very long. Thus, the following will appear. Then the whole system will hang.
The similar problems also occur in Intel NICs and Mellanox NICs. As such, the similar solution is borrowed from them. A timeout value is added and the timeout value as large as possible is set to ensure that the driver gets the maximum possible response from the hardware."
[ 213.795860] watchdog: BUG: soft lockup - CPU#108 stuck for 26s! [(udev-worker):3157] [ 213.796114] Modules linked in: virtio_net(+) net_failover failover qrtr rfkill sunrpc intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common intel_ifs i10nm_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt rapl intel_pmc_bxt dax_hmem iTCO_vendor_support vfat cxl_acpi intel_cstate pmt_telemetry pmt_class intel_sdsi joydev intel_uncore cxl_core fat pcspkr mei_me isst_if_mbox_pci isst_if_mmio idxd i2c_i801 isst_if_common mei intel_vsec idxd_bus i2c_smbus i2c_ismt ipmi_ssif acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter pfr_telemetry pfr_update fuse loop zram xfs crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 bnxt_en sha256_ssse3 sha1_ssse3 nvme ast nvme_core i2c_algo_bit wmi pinctrl_emmitsburg scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_multipath [ 213.796194] irq event stamp: 67740 [ 213.796195] hardirqs last enabled at (67739): [<ffffffff8c2015ca>] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 213.796203] hardirqs last disabled at (67740): [<ffffffff8c14108e>] sysvec_apic_timer_interrupt+0xe/0x90 [ 213.796208] softirqs last enabled at (67686): [<ffffffff8b12115e>] __irq_exit_rcu+0xbe/0xe0 [ 213.796214] softirqs last disabled at (67681): [<ffffffff8b12115e>] __irq_exit_rcu+0xbe/0xe0 [ 213.796217] CPU: 108 PID: 3157 Comm: (udev-worker) Kdump: loaded Not tainted 6.7.0+ #9 [ 213.796220] Hardware name: Intel Corporation M50FCP2SBSTD/M50FCP2SBSTD, BIOS SE5C741.86B.01.01.0001.2211140926 11/14/2022 [ 213.796221] RIP: 0010:virtqueue_get_buf_ctx_split+0x8d/0x110 [ 213.796228] Code: 89 df e8 26 fe ff ff 0f b7 43 50 83 c0 01 66 89 43 50 f6 43 78 01 75 12 80 7b 42 00 48 8b 4b 68 8b 53 58 74 0f 66 87 44 51 04 <48> 89 e8 5b 5d c3 cc cc cc cc 66 89 44 51 04 0f ae f0 48 89 e8 5b [ 213.796230] RSP: 0018:ff4bbb362306f9b0 EFLAGS: 00000246 [ 213.796233] RAX: 0000000000000000 RBX: ff2f15095896f000 RCX: 0000000000000001 [ 213.796235] RDX: 0000000000000000 RSI: ff4bbb362306f9cc RDI: ff2f15095896f000 [ 213.796236] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 213.796237] R10: 0000000000000003 R11: ff2f15095893cc40 R12: 0000000000000002 [ 213.796239] R13: 0000000000000004 R14: 0000000000000000 R15: ff2f1509534f3000 [ 213.796240] FS: 00007f775847d0c0(0000) GS:ff2f1528bac00000(0000) knlGS:0000000000000000 [ 213.796242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 213.796243] CR2: 0000557f987b6e70 CR3: 0000002098602006 CR4: 0000000000f71ef0 [ 213.796245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 213.796246] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 [ 213.796247] PKRU: 55555554 [ 213.796249] Call Trace: [ 213.796250] <IRQ> [ 213.796252] ? watchdog_timer_fn+0x1c0/0x220 [ 213.796258] ? __pfx_watchdog_timer_fn+0x10/0x10 [ 213.796261] ? __hrtimer_run_queues+0x1af/0x380 [ 213.796269] ? hrtimer_interrupt+0xf8/0x230 [ 213.796274] ? __sysvec_apic_timer_interrupt+0x64/0x1a0 [ 213.796279] ? sysvec_apic_timer_interrupt+0x6d/0x90 [ 213.796282] </IRQ> [ 213.796284] <TASK> [ 213.796285] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 213.796293] ? virtqueue_get_buf_ctx_split+0x8d/0x110 [ 213.796297] virtnet_send_command+0x18a/0x1f0 [virtio_net] [ 213.796310] _virtnet_set_queues+0xc6/0x120 [virtio_net] [ 213.796319] virtnet_probe+0xa06/0xd50 [virtio_net] [ 213.796328] virtio_dev_probe+0x195/0x230 [ 213.796333] really_probe+0x19f/0x400 [ 213.796338] ? __pfx___driver_attach+0x10/0x10 [ 213.796340] __driver_probe_device+0x78/0x160 [ 213.796343] driver_probe_device+0x1f/0x90 [ 213.796346] __driver_attach+0xd6/0x1d0 [ 213.796349] bus_for_each_dev+0x8c/0xe0 [ 213.796355] bus_add_driver+0x119/0x220 [ 213.796359] driver_register+0x59/0x100 [ 213.796362] ? __pfx_virtio_net_driver_init+0x10/0x10 [virtio_net] [ 213.796369] virtio_net_driver_init+0x8e/0xff0 [virtio_net] [ 213.796375] do_one_initcall+0x6f/0x380 [ 213.796384] do_init_module+0x60/0x240 [ 213.796388] init_module_from_file+0x86/0xc0 [ 213.796396] idempotent_init_module+0x129/0x2c0 [ 213.796406] __x64_sys_finit_module+0x5e/0xb0 [ 213.796409] do_syscall_64+0x60/0xe0 [ 213.796415] ? do_syscall_64+0x6f/0xe0 [ 213.796418] ? lockdep_hardirqs_on_prepare+0xe4/0x1a0 [ 213.796424] ? do_syscall_64+0x6f/0xe0 [ 213.796427] ? do_syscall_64+0x6f/0xe0 [ 213.796431] entry_SYSCALL_64_after_hwframe+0x6e/0x76 [ 213.796435] RIP: 0033:0x7f7758f279cd [ 213.796465] Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 33 e4 0c 00 f7 d8 64 89 01 48 [ 213.796467] RSP: 002b:00007ffe2cad8738 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 213.796469] RAX: ffffffffffffffda RBX: 0000557f987a8180 RCX: 00007f7758f279cd [ 213.796471] RDX: 0000000000000000 RSI: 00007f77593e5453 RDI: 000000000000000f [ 213.796472] RBP: 00007f77593e5453 R08: 0000000000000000 R09: 00007ffe2cad8860 [ 213.796473] R10: 000000000000000f R11: 0000000000000246 R12: 0000000000020000 [ 213.796475] R13: 0000557f9879f8e0 R14: 0000000000000000 R15: 0000557f98783aa0 [ 213.796482] </TASK>"
Signed-off-by Zhu Yanjun <[email protected]>--- drivers/net/virtio_net.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)drivers/net/virtio_net.c +1
-1@@ -2468,7 +2468,7 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, {struct scatterlist *sgs[4 ], hdr, stat;unsigned out_num =0 , tmp; -int ret; +int ret, timeout =200 ;/* Caller should know better */ BUG_ON (!virtio_has_feature (vi->vdev,VIRTIO_NET_F_CTRL_VQ ));drivers/net/virtio_net.c static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd, · line 2507
Quoted 7 lines
drivers/net/virtio_net.c +3
-1@@ -2502,8 +2502,14 @@ static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,* into the hypervisor, so the request should be handled immediately. */ while (!virtqueue_get_buf (vi->cvq, &tmp) && - !virtqueue_is_broken (vi->cvq)) + !virtqueue_is_broken (vi->cvq)) { +if (timeout) + timeout--;This is not really a timeout, just a loop counter. 200 iterations could be a very short time on reasonable H/W. I guess this avoid the soft lockup, but possibly (likely?) breaks the functionality when we need to loop for some non negligible time.
I fear we need a more complex solution, as mentioned by Micheal in the thread you quoted.
Quoted 8 lines
Quoted 3 lines
Cheers,
Paolo
So a bunch of devices just queue a rquest on a workqueue. Problem is, errors are not reported then. I really feel this is something netdev core should support better: return a "deferred" status and have netdev core wait outside a lock on a waitqueue.
And on the other side, more and more requests will be post to the cvq, we should refactor this to async mode.
Now the netdim has this request.
@Heng Qi
Thanks.
Quoted 5 lines
--
MST