blktest nvme_trtyp3=tcp nvme/003 is failing on nvme-6.7
Hi,
blktests nvme/003 is failing with following error message :-
In case this has been already discussed and fixed please ignore this message.
[ 3081.633737] run blktests nvme/003 at 2023-10-11 20:21:13 [ 3081.649812] loop0: detected capacity change from 0 to 2097152 [ 3081.661590] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 3081.666424] nvmet_tcp: enabling port 0 (127.0.0.1:4420) [ 3081.676553] nvmet: creating discovery controller 1 for subsystem nqn.2014-08.org.nvmexpress.discovery for NQN nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349. [ 3081.676954] nvme nvme1: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 127.0.0.1:4420 [ 3092.796012] nvme nvme1: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery" [ 3092.796342] ------------[ cut here ]------------ [ 3092.796344] WARNING: CPU: 34 PID: 19842 at kernel/workqueue.c:3400 __flush_work.isra.0+0x25d/0x270 [ 3092.796354] Modules linked in: nvmet_tcp(O) nvme_tcp(O)
nvme_fabrics(O) nvmet(O) loop nvme(O) nvme_core(O) nvme_common snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore ip6table_mangle ip6table_raw ip6table_security iptable_mangle iptable_raw iptable_security ip_set nf_tables rfkill nfnetlink ip6table_filter ip6_tables iptable_filter tun sunrpc intel_rapl_msr intel_rapl_common xfs kvm_amd ccp ppdev kvm parport_pc parport irqbypass joydev i2c_piix4 pcspkr ip_tables bochs drm_vram_helper crct10dif_pclmul crc32_pclmul drm_kms_helper crc32c_intel drm_ttm_helper ttm ghash_clmulni_intel virtio_net net_failover sha512_ssse3 drm serio_raw virtio_blk failover ata_generic pata_acpi qemu_fw_cfg fuse [last
unloaded: nvmet_tcp(O)]
[ 3092.796419] CPU: 34 PID: 19842 Comm: kworker/34:0 Tainted: G W O N 6.6.0-rc2nvme+ #18 [ 3092.796423] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014 [ 3092.796425] Workqueue: nvmet-wq nvmet_tcp_release_queue_work [nvmet_tcp] [ 3092.796436] RIP: 0010:__flush_work.isra.0+0x25d/0x270 [ 3092.796440] Code: 25 00 dc 02 00 48 89 44 24 40 8b 43 30 4c 8b 43 40 89 c1 e9 e5 fe ff ff 40 30 f6 4c 8b 36 e9 24 fe ff ff 0f 0b e9 03 ff ff ff <0f> 0b 45 31 ed e9 f9 fe ff ff e8 94 a1 d3 00 0f 1f 40 00 90 90 90 [ 3092.796443] RSP: 0018:ffffc90005107d68 EFLAGS: 00010246 [ 3092.796446] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 3092.796449] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88810d33e230 [ 3092.796450] RBP: ffff88810d33e230 R08: 0000000000000000 R09: ffff88810d33e3c0 [ 3092.796452] R10: 0000000000000018 R11: 0000000000000018 R12: ffff88810d33e230 [ 3092.796454] R13: 0000000000000001 R14: ffff88810d33e010 R15: ffff88810d33e3a0 [ 3092.796459] FS: 0000000000000000(0000) GS:ffff8897dfa80000(0000) knlGS:0000000000000000 [ 3092.796461] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3092.796463] CR2: 00000000004c0438 CR3: 0000000235de0000 CR4: 0000000000350ee0 [ 3092.796468] DR0: ffffffff8378f188 DR1: ffffffff8378f189 DR2: ffffffff8378f18a [ 3092.796470] DR3: ffffffff8378f18b DR6: 00000000ffff0ff0 DR7: 0000000000000600 [ 3092.796472] Call Trace: [ 3092.796475] <TASK> [ 3092.796478] ? __warn+0x80/0x130 [ 3092.796483] ? __flush_work.isra.0+0x25d/0x270 [ 3092.796487] ? report_bug+0x1c3/0x1d0 [ 3092.796492] ? handle_bug+0x42/0x70 [ 3092.796496] ? exc_invalid_op+0x14/0x70 [ 3092.796499] ? asm_exc_invalid_op+0x16/0x20 [ 3092.796505] ? __flush_work.isra.0+0x25d/0x270 [ 3092.796508] ? _raw_spin_lock+0x13/0x40 [ 3092.796513] ? raw_spin_rq_lock_nested+0x1b/0x30 [ 3092.796518] ? newidle_balance+0x26e/0x400 [ 3092.796525] __cancel_work_timer+0x108/0x190 [ 3092.796530] nvmet_tcp_release_queue_work+0xd8/0x480 [nvmet_tcp] [ 3092.796538] process_one_work+0x147/0x300 [ 3092.796541] worker_thread+0x27e/0x390 [ 3092.796544] ? __pfx_worker_thread+0x10/0x10 [ 3092.796547] kthread+0xf0/0x120 [ 3092.796552] ? __pfx_kthread+0x10/0x10 [ 3092.796556] ret_from_fork+0x30/0x50 [ 3092.796560] ? __pfx_kthread+0x10/0x10 [ 3092.796564] ret_from_fork_asm+0x1b/0x30 [ 3092.796571] </TASK> [ 3092.796572] ---[ end trace 0000000000000000 ]---
-ck
Quoted 6 lines · Chaitanya Kulkarni
Hi,
blktests nvme/003 is failing with following error message :-
In case this has been already discussed and fixed please ignore this message.
This failure symptom looks similar as the failure I observed and reported for rdma and tcp transports [1]. What I observed was a WARN that lockdep reported. But this time, it is a WARN that workqueue reports. So, this could be a new, different failure.
[1] https://lore.kernel.org/linux-block/dycg2iltyeezp6y7bqfhfke6bb3dantkk5nsyk6qjfg2o55esu@liixlkhgtqy3/
Quoted 27 lines
[ 3081.633737] run blktests nvme/003 at 2023-10-11 20:21:13 [ 3081.649812] loop0: detected capacity change from 0 to 2097152 [ 3081.661590] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 3081.666424] nvmet_tcp: enabling port 0 (127.0.0.1:4420) [ 3081.676553] nvmet: creating discovery controller 1 for subsystem nqn.2014-08.org.nvmexpress.discovery for NQN nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349. [ 3081.676954] nvme nvme1: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 127.0.0.1:4420 [ 3092.796012] nvme nvme1: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery" [ 3092.796342] ------------[ cut here ]------------ [ 3092.796344] WARNING: CPU: 34 PID: 19842 at kernel/workqueue.c:3400 __flush_work.isra.0+0x25d/0x270 [ 3092.796354] Modules linked in: nvmet_tcp(O) nvme_tcp(O)
nvme_fabrics(O)nvmet(O) loop nvme(O)nvme_core(O)nvme_commonsnd_seq_dummysnd_hrtimersnd_seqsnd_seq_devicesnd_timersnd soundcoreip6table_mangleip6table_rawip6table_securityiptable_mangleiptable_rawiptable_securityip_setnf_tablesrfkill nfnetlinkip6table_filterip6_tablesiptable_filtertun sunrpcintel_rapl_msrintel_rapl_commonxfskvm_amdccp ppdev kvmparport_pcparport irqbypass joydevi2c_piix4pcspkrip_tablesbochsdrm_vram_helpercrct10dif_pclmulcrc32_pclmuldrm_kms_helpercrc32c_inteldrm_ttm_helperttmghash_clmulni_intelvirtio_netnet_failoversha512_ssse3drmserio_rawvirtio_blkfailoverata_genericpata_acpiqemu_fw_cfgfuse [last
unloaded:nvmet_tcp(O)]
Quoted 2 lines
[ 3092.796419] CPU: 34 PID: 19842 Comm: kworker/34:0 Tainted: GW O N 6.6.0-rc2nvme+ #18
Is this the kernel at nvme-6.7 branch in infradead nvme repo?