9pfs issues on 6.12-rc1
Subsystems (6)
Hi!
Starting with Linux 6.12-rc1 the automatic VM tests of NixOS don't boot anymore and fail like this:
mounting nix-store on /nix/.ro-store...
[ 1.604781] 9p: Installing v9fs 9p2000 file system support
mounting tmpfs on /nix/.rw-store...
mounting overlay on /nix/store...
mounting shared on /tmp/shared...
mounting xchg on /tmp/xchg...
switch_root: can't execute '/nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/init': Exec format error
[ 1.734997] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[ 1.736002] CPU: 0 UID: 0 PID: 1 Comm: switch_root Not tainted 6.12.0-rc1 #1-NixOS
[ 1.736965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[ 1.738309] Call Trace:
[ 1.738698] <TASK>
[ 1.739034] panic+0x324/0x340
[ 1.739458] do_exit+0x92e/0xa90
[ 1.739919] ? count_memcg_events.constprop.0+0x1a/0x40
[ 1.740568] ? srso_return_thunk+0x5/0x5f
[ 1.741095] ? handle_mm_fault+0xb0/0x2e0
[ 1.741709] do_group_exit+0x30/0x80
[ 1.742229] __x64_sys_exit_group+0x18/0x20
[ 1.742800] x64_sys_call+0x17f3/0x1800
[ 1.743326] do_syscall_64+0xb7/0x210
[ 1.743895] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 1.744530] RIP: 0033:0x7f8e1a7b9d1d
[ 1.745061] Code: 45 31 c0 45 31 d2 45 31 db c3 0f 1f 00 f3 0f 1e fa 48 8b 35 e5 e0 10 00 ba e7 00 00 00 eb 07 66 0f 1f 44 00 00 f4 89 d0 0f 05 <48> 3d 00 f0 ff ff 76 f3 f7 d8 64 89 06 eb ec 0f 1f 40 00 f3 0f 1e
[ 1.747263] RSP: 002b:00007ffcb56d63b8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
[ 1.748250] RAX: ffffffffffffffda RBX: 00007f8e1a8c9fa8 RCX: 00007f8e1a7b9d1d
[ 1.749187] RDX: 00000000000000e7 RSI: ffffffffffffff88 RDI: 0000000000000001
[ 1.750050] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000
[ 1.750891] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[ 1.751706] R13: 0000000000000001 R14: 00007f8e1a8c8680 R15: 00007f8e1a8c9fc0
[ 1.752583] </TASK>
[ 1.753010] Kernel Offset: 0xb800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)The failing script here is the initrd's /init when it tries to perform a switch_root to /sysroot:
exec env -i $(type -P switch_root) "$targetRoot" "$stage2Init"
Said "$stage2Init" file consistently gets a different hash when doing sha256sum on it in the initrd script, but looks & behaves correct on the host. I reproduced the test failures on 4 different build machines and two architectures (x86_64-linux, aarch64-linux) now.
The "$stage2Init" script is a shell-script itself. When trying to start the interpreter from its shebang inside the initrd (via $targetRoot/nix/store/...-bash-5.2p32/bin/bash) and do the switch_root I get a different error:
+ exec env -i /nix/store/akm69s5sngxyvqrzys326dss9rsrvbpy-extra-utils/bin/switch_root /mnt-root /nix/store/k3pm4iv44y7x7p74kky6cwxiswmr6kpi-nixos-system-machine-test/init
[ 1.912859] list_del corruption. prev->next should be ffffc5cf80be0248, but was ffffc5cf80bd9208. (prev=ffffc5cf80bb4d48)
[ 1.914237] ------------[ cut here ]------------
[ 1.915059] kernel BUG at lib/list_debug.c:62!
[ 1.915854] Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[ 1.916739] CPU: 0 UID: 0 PID: 17 Comm: ksoftirqd/0 Not tainted 6.12.0-rc1 #1-NixOS
[ 1.917837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[ 1.919354] RIP: 0010:__list_del_entry_valid_or_report+0xb4/0xd0
[ 1.920180] Code: 0f 0b 48 89 fe 48 89 ca 48 c7 c7 38 52 41 9f e8 42 91 ac ff 90 0f 0b 48 89 fe 48 89 c2 48 c7 c7 70 52 41 9f e8 2d 91 ac ff 90 <0f> 0b 48 89 d1 48 c7 c7 c0 52 41 9f 48 89 f2 48 89 c6 e8 15 91 ac
[ 1.922636] RSP: 0018:ffff96f800093c00 EFLAGS: 00010046
[ 1.923563] RAX: 000000000000006d RBX: 0000000000000001 RCX: 0000000000000000
[ 1.924692] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 1.925664] RBP: 0000000000000341 R08: 0000000000000000 R09: 0000000000000000
[ 1.926646] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8fbebd83dc90
[ 1.927584] R13: ffffc5cf80be0240 R14: ffff8fbebd83dc80 R15: 000000000002f809
[ 1.928533] FS: 0000000000000000(0000) GS:ffff8fbebd800000(0000) knlGS:0000000000000000
[ 1.929647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1.930431] CR2: 00007fed6f09b000 CR3: 0000000001e02000 CR4: 0000000000350ef0
[ 1.931333] Call Trace:
[ 1.931727] <TASK>
[ 1.932088] ? die+0x36/0x90
[ 1.932595] ? do_trap+0xed/0x110
[ 1.933047] ? __list_del_entry_valid_or_report+0xb4/0xd0
[ 1.933757] ? do_error_trap+0x6a/0xa0
[ 1.934390] ? __list_del_entry_valid_or_report+0xb4/0xd0
[ 1.935073] ? exc_invalid_op+0x51/0x80
[ 1.935627] ? __list_del_entry_valid_or_report+0xb4/0xd0
[ 1.936326] ? asm_exc_invalid_op+0x1a/0x20
[ 1.936904] ? __list_del_entry_valid_or_report+0xb4/0xd0
[ 1.937622] free_pcppages_bulk+0x130/0x280
[ 1.938151] free_unref_page_commit+0x21c/0x380
[ 1.938753] free_unref_page+0x472/0x4f0
[ 1.939343] __put_partials+0xee/0x130
[ 1.939921] ? rcu_do_batch+0x1f2/0x800
[ 1.940471] kmem_cache_free+0x2c3/0x370
[ 1.940990] rcu_do_batch+0x1f2/0x800
[ 1.941508] ? rcu_do_batch+0x180/0x800
[ 1.942031] rcu_core+0x182/0x340
[ 1.942500] handle_softirqs+0xe4/0x2f0
[ 1.943034] run_ksoftirqd+0x33/0x40
[ 1.943522] smpboot_thread_fn+0xdd/0x1d0
[ 1.944056] ? __pfx_smpboot_thread_fn+0x10/0x10
[ 1.944679] kthread+0xd0/0x100
[ 1.945126] ? __pfx_kthread+0x10/0x10
[ 1.945656] ret_from_fork+0x34/0x50
[ 1.946151] ? __pfx_kthread+0x10/0x10
[ 1.946680] ret_from_fork_asm+0x1a/0x30
[ 1.947269] </TASK>
[ 1.947622] Modules linked in: overlay 9p ext4 crc32c_generic crc16 mbcache jbd2 hid_generic usbhid hid 9pnet_virtio 9pnet netfs sr_mod virtio_net cdrom virtio_blk net_failover atkbd failover libps2 vivaldi_fmap crc32c_intel ata_piix libata uhci_hcd scsi_mod ehci_hcd virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev scsi_common i8042 serio rtc_cmos dm_mod dax virtio_gpu virtio_dma_buf virtio_rng rng_core virtio_console virtio_balloon virtio virtio_ring
[ 1.952291] ---[ end trace 0000000000000000 ]---
[ 1.952893] RIP: 0010:__list_del_entry_valid_or_report+0xb4/0xd0
[ 1.953678] Code: 0f 0b 48 89 fe 48 89 ca 48 c7 c7 38 52 41 9f e8 42 91 ac ff 90 0f 0b 48 89 fe 48 89 c2 48 c7 c7 70 52 41 9f e8 2d 91 ac ff 90 <0f> 0b 48 89 d1 48 c7 c7 c0 52 41 9f 48 89 f2 48 89 c6 e8 15 91 ac
[ 1.955888] RSP: 0018:ffff96f800093c00 EFLAGS: 00010046
[ 1.956548] RAX: 000000000000006d RBX: 0000000000000001 RCX: 0000000000000000
[ 1.957436] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 1.958328] RBP: 0000000000000341 R08: 0000000000000000 R09: 0000000000000000
[ 1.959166] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8fbebd83dc90
[ 1.960044] R13: ffffc5cf80be0240 R14: ffff8fbebd83dc80 R15: 000000000002f809
[ 1.960905] FS: 0000000000000000(0000) GS:ffff8fbebd800000(0000) knlGS:0000000000000000
[ 1.961926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1.962693] CR2: 00007fed6f09b000 CR3: 0000000001e02000 CR4: 0000000000350ef0
[ 1.963548] Kernel panic - not syncing: Fatal exception in interrupt
[ 1.964417] Kernel Offset: 0x1ce00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)On a subsequent run to verify this, it failed earlier while reading $targetRoot/.../bash like this:
[ 1.871810] BUG: Bad page state in process cat pfn:2e74a
[ 1.872481] page: refcount:1 mapcount:0 mapping:0000000000000000 index:0x1e5 pfn:0x2e74a
[ 1.873499] flags: 0xffffc000000000(node=0|zone=1|lastcpupid=0x1ffff)
[ 1.874260] raw: 00ffffc000000000 dead000000000100 dead000000000122 0000000000000000
[ 1.875250] raw: 00000000000001e5 0000000000000000 00000001ffffffff 0000000000000000
[ 1.876295] page dumped because: nonzero _refcount
[ 1.876910] Modules linked in: overlay 9p ext4 crc32c_generic crc16 mbcache jbd2 hid_generic usbhid hid 9pnet_virtio 9pnet netfs sr_mod virtio_net cdrom virtio_blk net_failover atkbd failover libps2 vivaldi_fmap crc32c_intel ata_piix libata scsi_mod uhci_hcd ehci_hcd virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev scsi_common i8042 serio rtc_cmos dm_mod dax virtio_gpu virtio_dma_buf virtio_rng rng_core virtio_console virtio_balloon virtio virtio_ring
[ 1.881465] CPU: 0 UID: 0 PID: 315 Comm: cat Not tainted 6.12.0-rc1 #1-NixOS
[ 1.882326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[ 1.883684] Call Trace:
[ 1.884103] <TASK>
[ 1.884440] dump_stack_lvl+0x64/0x90
[ 1.884954] bad_page+0x70/0x110
[ 1.885468] __rmqueue_pcplist+0x209/0xd00
[ 1.886029] ? srso_return_thunk+0x5/0x5f
[ 1.886572] ? pdu_read+0x36/0x50 [9pnet]
[ 1.887177] get_page_from_freelist+0x2df/0x1910
[ 1.887788] ? srso_return_thunk+0x5/0x5f
[ 1.888324] ? enqueue_entity+0xce/0x510
[ 1.888881] ? srso_return_thunk+0x5/0x5f
[ 1.889415] ? pick_eevdf+0x76/0x1a0
[ 1.889970] ? update_curr+0x35/0x270
[ 1.890476] __alloc_pages_noprof+0x1a3/0x1150
[ 1.891158] ? srso_return_thunk+0x5/0x5f
[ 1.891712] ? __mod_memcg_lruvec_state+0xa9/0x160
[ 1.892346] ? srso_return_thunk+0x5/0x5f
[ 1.892919] ? __lruvec_stat_mod_folio+0x83/0xd0
[ 1.893521] alloc_pages_mpol_noprof+0x8f/0x1f0
[ 1.894148] folio_alloc_noprof+0x5b/0xb0
[ 1.894671] page_cache_ra_unbounded+0x11f/0x200
[ 1.895270] filemap_get_pages+0x538/0x6d0
[ 1.895813] ? srso_return_thunk+0x5/0x5f
[ 1.896361] filemap_splice_read+0x136/0x320
[ 1.896948] backing_file_splice_read+0x52/0xa0
[ 1.897522] ovl_splice_read+0xd2/0xf0 [overlay]
[ 1.898160] ? __pfx_ovl_file_accessed+0x10/0x10 [overlay]
[ 1.898817] splice_direct_to_actor+0xb4/0x270
[ 1.899404] ? __pfx_direct_splice_actor+0x10/0x10
[ 1.900103] do_splice_direct+0x77/0xd0
[ 1.900627] ? __pfx_direct_file_splice_eof+0x10/0x10
[ 1.901308] do_sendfile+0x359/0x410
[ 1.901788] __x64_sys_sendfile64+0xb9/0xd0
[ 1.902370] do_syscall_64+0xb7/0x210
[ 1.902904] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 1.903604] RIP: 0033:0x7fa9f3a7289e
[ 1.904214] Code: 75 0e 00 f7 d8 64 89 02 b8 ff ff ff ff 31 d2 31 c9 31 ff 45 31 db c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 12 31 d2 31 c9 31 f6 31 ff 45 31 d2 45 31 db
[ 1.906436] RSP: 002b:00007ffe6a82bde8 EFLAGS: 00000246 ORIG_RAX: 0000000000000028
[ 1.907400] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa9f3a7289e
[ 1.908241] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000001
[ 1.909184] RBP: 00007ffe6a82be50 R08: 0000000000000000 R09: 0000000000000000
[ 1.910212] R10: 0000000001000000 R11: 0000000000000246 R12: 0000000000000001
[ 1.911117] R13: 0000000001000000 R14: 0000000000000001 R15: 0000000000000000
[ 1.911998] </TASK>
[ 1.912376] Disabling lock debugging due to kernel taint
[ 1.913479] list_del corruption. next->prev should be ffffc80e40b9d948, but was ffffc80e40b9d0c8. (next=ffffc80e40b9c7c8)
[ 1.914823] ------------[ cut here ]------------
[ 1.915408] kernel BUG at lib/list_debug.c:65!
[ 1.916050] Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[ 1.916785] CPU: 0 UID: 0 PID: 315 Comm: cat Tainted: G B 6.12.0-rc1 #1-NixOS
[ 1.917877] Tainted: [B]=BAD_PAGE
[ 1.918350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[ 1.919996] RIP: 0010:__list_del_entry_valid_or_report+0xcc/0xd0
[ 1.920903] Code: 89 fe 48 89 c2 48 c7 c7 70 52 41 ba e8 2d 91 ac ff 90 0f 0b 48 89 d1 48 c7 c7 c0 52 41 ba 48 89 f2 48 89 c6 e8 15 91 ac ff 90 <0f> 0b 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f
[ 1.923423] RSP: 0018:ffff9ed880187748 EFLAGS: 00010246
[ 1.924210] RAX: 000000000000006d RBX: ffff94db3d83dc80 RCX: 0000000000000000
[ 1.925147] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 1.926051] RBP: ffffc80e40b9d940 R08: 0000000000000000 R09: 0000000000000000
[ 1.926940] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[ 1.927809] R13: ffff94db3d83dc80 R14: ffffc80e40b9d948 R15: ffff94db3ffd6180
[ 1.928695] FS: 00007fa9f396eb80(0000) GS:ffff94db3d800000(0000) knlGS:0000000000000000
[ 1.929728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1.930540] CR2: 00000000004d1829 CR3: 0000000001dd2000 CR4: 0000000000350ef0
[ 1.931444] Call Trace:
[ 1.931916] <TASK>
[ 1.932357] ? die+0x36/0x90
[ 1.932831] ? do_trap+0xed/0x110
[ 1.933385] ? __list_del_entry_valid_or_report+0xcc/0xd0
[ 1.934073] ? do_error_trap+0x6a/0xa0
[ 1.934583] ? __list_del_entry_valid_or_report+0xcc/0xd0
[ 1.935242] ? exc_invalid_op+0x51/0x80
[ 1.935781] ? __list_del_entry_valid_or_report+0xcc/0xd0
[ 1.936484] ? asm_exc_invalid_op+0x1a/0x20
[ 1.937174] ? __list_del_entry_valid_or_report+0xcc/0xd0
[ 1.937926] ? __list_del_entry_valid_or_report+0xcb/0xd0
[ 1.938685] __rmqueue_pcplist+0xa5/0xd00
[ 1.939292] ? srso_return_thunk+0x5/0x5f
[ 1.940004] ? __mod_memcg_lruvec_state+0xa9/0x160
[ 1.940758] ? srso_return_thunk+0x5/0x5f
[ 1.941417] ? update_load_avg+0x7e/0x7f0
[ 1.942133] ? srso_return_thunk+0x5/0x5f
[ 1.942838] ? srso_return_thunk+0x5/0x5f
[ 1.943508] get_page_from_freelist+0x2df/0x1910
[ 1.944143] ? srso_return_thunk+0x5/0x5f
[ 1.944696] ? check_preempt_wakeup_fair+0x1ee/0x240
[ 1.945335] ? srso_return_thunk+0x5/0x5f
[ 1.945905] __alloc_pages_noprof+0x1a3/0x1150
[ 1.946489] ? __blk_flush_plug+0xf5/0x150
[ 1.947105] ? srso_return_thunk+0x5/0x5f
[ 1.947629] ? __dquot_alloc_space+0x2a8/0x3a0
[ 1.948404] ? srso_return_thunk+0x5/0x5f
[ 1.949116] ? __mod_memcg_lruvec_state+0xa9/0x160
[ 1.949888] alloc_pages_mpol_noprof+0x8f/0x1f0
[ 1.950514] folio_alloc_mpol_noprof+0x14/0x40
[ 1.951153] shmem_alloc_folio+0xa7/0xd0
[ 1.951692] ? shmem_recalc_inode+0x20/0x90
[ 1.952272] shmem_alloc_and_add_folio+0x109/0x490
[ 1.952940] ? filemap_get_entry+0x10f/0x1a0
[ 1.953570] ? srso_return_thunk+0x5/0x5f
[ 1.954185] shmem_get_folio_gfp+0x248/0x610
[ 1.954791] shmem_write_begin+0x64/0x110
[ 1.955484] generic_perform_write+0xdf/0x2a0
[ 1.956239] shmem_file_write_iter+0x8a/0x90
[ 1.956882] iter_file_splice_write+0x33f/0x580
[ 1.957577] direct_splice_actor+0x54/0x140
[ 1.958178] splice_direct_to_actor+0xec/0x270
[ 1.958813] ? __pfx_direct_splice_actor+0x10/0x10
[ 1.959442] do_splice_direct+0x77/0xd0
[ 1.960018] ? __pfx_direct_file_splice_eof+0x10/0x10
[ 1.960726] do_sendfile+0x359/0x410
[ 1.961248] __x64_sys_sendfile64+0xb9/0xd0
[ 1.961905] do_syscall_64+0xb7/0x210
[ 1.962467] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 1.963211] RIP: 0033:0x7fa9f3a7289e
[ 1.963711] Code: 75 0e 00 f7 d8 64 89 02 b8 ff ff ff ff 31 d2 31 c9 31 ff 45 31 db c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 12 31 d2 31 c9 31 f6 31 ff 45 31 d2 45 31 db
[ 1.965846] RSP: 002b:00007ffe6a82bde8 EFLAGS: 00000246 ORIG_RAX: 0000000000000028
[ 1.966788] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa9f3a7289e
[ 1.967644] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000001
[ 1.968480] RBP: 00007ffe6a82be50 R08: 0000000000000000 R09: 0000000000000000
[ 1.969396] R10: 0000000001000000 R11: 0000000000000246 R12: 0000000000000001
[ 1.970315] R13: 0000000001000000 R14: 0000000000000001 R15: 0000000000000000
[ 1.971214] </TASK>
[ 1.971572] Modules linked in: overlay 9p ext4 crc32c_generic crc16 mbcache jbd2 hid_generic usbhid hid 9pnet_virtio 9pnet netfs sr_mod virtio_net cdrom virtio_blk net_failover atkbd failover libps2 vivaldi_fmap crc32c_intel ata_piix libata scsi_mod uhci_hcd ehci_hcd virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev scsi_common i8042 serio rtc_cmos dm_mod dax virtio_gpu virtio_dma_buf virtio_rng rng_core virtio_console virtio_balloon virtio virtio_ring
[ 1.976558] ---[ end trace 0000000000000000 ]---
[ 1.977219] RIP: 0010:__list_del_entry_valid_or_report+0xcc/0xd0
[ 1.978033] Code: 89 fe 48 89 c2 48 c7 c7 70 52 41 ba e8 2d 91 ac ff 90 0f 0b 48 89 d1 48 c7 c7 c0 52 41 ba 48 89 f2 48 89 c6 e8 15 91 ac ff 90 <0f> 0b 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f
[ 1.980179] RSP: 0018:ffff9ed880187748 EFLAGS: 00010246
[ 1.980847] RAX: 000000000000006d RBX: ffff94db3d83dc80 RCX: 0000000000000000
[ 1.981705] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 1.982584] RBP: ffffc80e40b9d940 R08: 0000000000000000 R09: 0000000000000000
[ 1.983464] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[ 1.984358] R13: ffff94db3d83dc80 R14: ffffc80e40b9d948 R15: ffff94db3ffd6180
[ 1.987765] FS: 00007fa9f396eb80(0000) GS:ffff94db3d800000(0000) knlGS:0000000000000000
[ 1.988805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1.989497] CR2: 00000000004d1829 CR3: 0000000001dd2000 CR4: 0000000000350ef0
[ 1.990418] note: cat[315] exited with preempt_count 2I bisected it back to ee4cdf7ba857a894ad1650d6ab77669cbbfa329e which also seems to touch part of the 9p VFS code.
Unfortunately the revert didn't apply cleanly on 6.12-rc1, so I couldn't meaningfully test whether a simple revert solves the problem.
The VMs get the Nix store mounted via 9p. In the store are basically all build artifacts including the stage-2 init script of the system that is booted into in the VM test.
The invocation basically looks like this:
qemu-system-x86_64 -cpu max \
-name machine \
-m 1024 \
-smp 1 \
-device virtio-rng-pci \
-net nic,netdev=user.0,model=virtio -netdev user,id=user.0,"$QEMU_NET_OPTS" \
-virtfs local,path=/nix/store,security_model=none,mount_tag=nix-store \
-virtfs local,path="${SHARED_DIR:-$TMPDIR/xchg}",security_model=none,mount_tag=shared \
-virtfs local,path="$TMPDIR"/xchg,security_model=none,mount_tag=xchg \
-drive cache=writeback,file="$NIX_DISK_IMAGE",id=drive1,if=none,index=1,werror=report -device virtio-blk-pci,bootindex=1,drive=drive1,serial=root \
-device virtio-net-pci,netdev=vlan1,mac=52:54:00:12:01:01 \
-netdev vde,id=vlan1,sock="$QEMU_VDE_SOCKET_1" \
-device virtio-keyboard \
-usb \
-device usb-tablet,bus=usb-bus.0 \
-kernel ${NIXPKGS_QEMU_KERNEL_machine:-/nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/kernel} \
-initrd /nix/store/qqalw1iq1wbgq3ndx0cvqn3bfypn56w2-initrd-linux-6.12-rc1/initrd \
-append "$(cat /nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/kernel-params) init=/nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/init regInfo=/nix/store/5izvfal6xm2rk51v0r1h2cxcng33paby-closure-info/registration console=ttyS0 $QEMU_KERNEL_PARAMS" \
$QEMU_OPTSIf you're using Nix, you can also reproduce this by running
nix-build nixos/tests/kernel-generic.nix -A linux_testing
on 5c19646b81db43dd7f4b6954f17d71a523009706 from https://github.com/nixos/nixpkgs.
To me, this seems like a regression in rc1.
Is there anything else I can do to help troubleshooting this?
With best regards
Maximilian
Hi, Thorsten here, the Linux kernel's regression tracker. Top-posting for once, to make this easily accessible to everyone.
Thx for the report. Not my area of expertise (so everyone: corrent me if I'm wrong), but I suspect your problem might be a duplicate of the following report, which was bisected to the same commit from dhowells (ee4cdf7ba857a8 ("netfs: Speed up buffered reading") [v6.12-rc1]):
https://lore.kernel.org/all/[email protected]/
A fix for it is already pending in the vfs.fixes branch and -next:
https://lore.kernel.org/all/cbaf141ba6c0e2e209717d02746584072844841a.1727722269.git.osandov@fb.com/
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.
Quoted 312 lines · Maximilian Bosch
Starting with Linux 6.12-rc1 the automatic VM tests of NixOS don't boot anymore and fail like this:
Quoted 1 line
mounting nix-store on /nix/.ro-store...
[ 1.604781] 9p: Installing v9fs 9p2000 file system support mounting tmpfs on /nix/.rw-store... mounting overlay on /nix/store... mounting shared on /tmp/shared... mounting xchg on /tmp/xchg... switch_root: can't execute '/nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/init': Exec format error [ 1.734997] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100 [ 1.736002] CPU: 0 UID: 0 PID: 1 Comm: switch_root Not tainted 6.12.0-rc1 #1-NixOS [ 1.736965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 1.738309] Call Trace: [ 1.738698] <TASK> [ 1.739034] panic+0x324/0x340 [ 1.739458] do_exit+0x92e/0xa90 [ 1.739919] ? count_memcg_events.constprop.0+0x1a/0x40 [ 1.740568] ? srso_return_thunk+0x5/0x5f [ 1.741095] ? handle_mm_fault+0xb0/0x2e0 [ 1.741709] do_group_exit+0x30/0x80 [ 1.742229] __x64_sys_exit_group+0x18/0x20 [ 1.742800] x64_sys_call+0x17f3/0x1800 [ 1.743326] do_syscall_64+0xb7/0x210 [ 1.743895] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1.744530] RIP: 0033:0x7f8e1a7b9d1d [ 1.745061] Code: 45 31 c0 45 31 d2 45 31 db c3 0f 1f 00 f3 0f 1e fa 48 8b 35 e5 e0 10 00 ba e7 00 00 00 eb 07 66 0f 1f 44 00 00 f4 89 d0 0f 05 <48> 3d 00 f0 ff ff 76 f3 f7 d8 64 89 06 eb ec 0f 1f 40 00 f3 0f 1e [ 1.747263] RSP: 002b:00007ffcb56d63b8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7 [ 1.748250] RAX: ffffffffffffffda RBX: 00007f8e1a8c9fa8 RCX: 00007f8e1a7b9d1d [ 1.749187] RDX: 00000000000000e7 RSI: ffffffffffffff88 RDI: 0000000000000001 [ 1.750050] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000 [ 1.750891] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 1.751706] R13: 0000000000000001 R14: 00007f8e1a8c8680 R15: 00007f8e1a8c9fc0 [ 1.752583] </TASK> [ 1.753010] Kernel Offset: 0xb800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)The failing script here is the initrd's /init when it tries to perform a
switch_rootto/sysroot:exec env -i $(type -P switch_root) "$targetRoot" "$stage2Init"Said "$stage2Init" file consistently gets a different hash when doing
sha256sumon it in the initrd script, but looks & behaves correct on the host. I reproduced the test failures on 4 different build machines and two architectures (x86_64-linux, aarch64-linux) now.The "$stage2Init" script is a shell-script itself. When trying to start the interpreter from its shebang inside the initrd (via
$targetRoot/nix/store/...-bash-5.2p32/bin/bash) and do theswitch_rootI get a different error:+ exec env -i /nix/store/akm69s5sngxyvqrzys326dss9rsrvbpy-extra-utils/bin/switch_root /mnt-root /nix/store/k3pm4iv44y7x7p74kky6cwxiswmr6kpi-nixos-system-machine-test/init [ 1.912859] list_del corruption. prev->next should be ffffc5cf80be0248, but was ffffc5cf80bd9208. (prev=ffffc5cf80bb4d48) [ 1.914237] ------------[ cut here ]------------ [ 1.915059] kernel BUG at lib/list_debug.c:62! [ 1.915854] Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 1.916739] CPU: 0 UID: 0 PID: 17 Comm: ksoftirqd/0 Not tainted 6.12.0-rc1 #1-NixOS [ 1.917837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 1.919354] RIP: 0010:__list_del_entry_valid_or_report+0xb4/0xd0 [ 1.920180] Code: 0f 0b 48 89 fe 48 89 ca 48 c7 c7 38 52 41 9f e8 42 91 ac ff 90 0f 0b 48 89 fe 48 89 c2 48 c7 c7 70 52 41 9f e8 2d 91 ac ff 90 <0f> 0b 48 89 d1 48 c7 c7 c0 52 41 9f 48 89 f2 48 89 c6 e8 15 91 ac [ 1.922636] RSP: 0018:ffff96f800093c00 EFLAGS: 00010046 [ 1.923563] RAX: 000000000000006d RBX: 0000000000000001 RCX: 0000000000000000 [ 1.924692] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 1.925664] RBP: 0000000000000341 R08: 0000000000000000 R09: 0000000000000000 [ 1.926646] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8fbebd83dc90 [ 1.927584] R13: ffffc5cf80be0240 R14: ffff8fbebd83dc80 R15: 000000000002f809 [ 1.928533] FS: 0000000000000000(0000) GS:ffff8fbebd800000(0000) knlGS:0000000000000000 [ 1.929647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.930431] CR2: 00007fed6f09b000 CR3: 0000000001e02000 CR4: 0000000000350ef0 [ 1.931333] Call Trace: [ 1.931727] <TASK> [ 1.932088] ? die+0x36/0x90 [ 1.932595] ? do_trap+0xed/0x110 [ 1.933047] ? __list_del_entry_valid_or_report+0xb4/0xd0 [ 1.933757] ? do_error_trap+0x6a/0xa0 [ 1.934390] ? __list_del_entry_valid_or_report+0xb4/0xd0 [ 1.935073] ? exc_invalid_op+0x51/0x80 [ 1.935627] ? __list_del_entry_valid_or_report+0xb4/0xd0 [ 1.936326] ? asm_exc_invalid_op+0x1a/0x20 [ 1.936904] ? __list_del_entry_valid_or_report+0xb4/0xd0 [ 1.937622] free_pcppages_bulk+0x130/0x280 [ 1.938151] free_unref_page_commit+0x21c/0x380 [ 1.938753] free_unref_page+0x472/0x4f0 [ 1.939343] __put_partials+0xee/0x130 [ 1.939921] ? rcu_do_batch+0x1f2/0x800 [ 1.940471] kmem_cache_free+0x2c3/0x370 [ 1.940990] rcu_do_batch+0x1f2/0x800 [ 1.941508] ? rcu_do_batch+0x180/0x800 [ 1.942031] rcu_core+0x182/0x340 [ 1.942500] handle_softirqs+0xe4/0x2f0 [ 1.943034] run_ksoftirqd+0x33/0x40 [ 1.943522] smpboot_thread_fn+0xdd/0x1d0 [ 1.944056] ? __pfx_smpboot_thread_fn+0x10/0x10 [ 1.944679] kthread+0xd0/0x100 [ 1.945126] ? __pfx_kthread+0x10/0x10 [ 1.945656] ret_from_fork+0x34/0x50 [ 1.946151] ? __pfx_kthread+0x10/0x10 [ 1.946680] ret_from_fork_asm+0x1a/0x30 [ 1.947269] </TASK> [ 1.947622] Modules linked in: overlay 9p ext4 crc32c_generic crc16 mbcache jbd2 hid_generic usbhid hid 9pnet_virtio 9pnet netfs sr_mod virtio_net cdrom virtio_blk net_failover atkbd failover libps2 vivaldi_fmap crc32c_intel ata_piix libata uhci_hcd scsi_mod ehci_hcd virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev scsi_common i8042 serio rtc_cmos dm_mod dax virtio_gpu virtio_dma_buf virtio_rng rng_core virtio_console virtio_balloon virtio virtio_ring [ 1.952291] ---[ end trace 0000000000000000 ]--- [ 1.952893] RIP: 0010:__list_del_entry_valid_or_report+0xb4/0xd0 [ 1.953678] Code: 0f 0b 48 89 fe 48 89 ca 48 c7 c7 38 52 41 9f e8 42 91 ac ff 90 0f 0b 48 89 fe 48 89 c2 48 c7 c7 70 52 41 9f e8 2d 91 ac ff 90 <0f> 0b 48 89 d1 48 c7 c7 c0 52 41 9f 48 89 f2 48 89 c6 e8 15 91 ac [ 1.955888] RSP: 0018:ffff96f800093c00 EFLAGS: 00010046 [ 1.956548] RAX: 000000000000006d RBX: 0000000000000001 RCX: 0000000000000000 [ 1.957436] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 1.958328] RBP: 0000000000000341 R08: 0000000000000000 R09: 0000000000000000 [ 1.959166] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8fbebd83dc90 [ 1.960044] R13: ffffc5cf80be0240 R14: ffff8fbebd83dc80 R15: 000000000002f809 [ 1.960905] FS: 0000000000000000(0000) GS:ffff8fbebd800000(0000) knlGS:0000000000000000 [ 1.961926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.962693] CR2: 00007fed6f09b000 CR3: 0000000001e02000 CR4: 0000000000350ef0 [ 1.963548] Kernel panic - not syncing: Fatal exception in interrupt [ 1.964417] Kernel Offset: 0x1ce00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)On a subsequent run to verify this, it failed earlier while reading $targetRoot/.../bash like this:
[ 1.871810] BUG: Bad page state in process cat pfn:2e74a [ 1.872481] page: refcount:1 mapcount:0 mapping:0000000000000000 index:0x1e5 pfn:0x2e74a [ 1.873499] flags: 0xffffc000000000(node=0|zone=1|lastcpupid=0x1ffff) [ 1.874260] raw: 00ffffc000000000 dead000000000100 dead000000000122 0000000000000000 [ 1.875250] raw: 00000000000001e5 0000000000000000 00000001ffffffff 0000000000000000 [ 1.876295] page dumped because: nonzero _refcount [ 1.876910] Modules linked in: overlay 9p ext4 crc32c_generic crc16 mbcache jbd2 hid_generic usbhid hid 9pnet_virtio 9pnet netfs sr_mod virtio_net cdrom virtio_blk net_failover atkbd failover libps2 vivaldi_fmap crc32c_intel ata_piix libata scsi_mod uhci_hcd ehci_hcd virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev scsi_common i8042 serio rtc_cmos dm_mod dax virtio_gpu virtio_dma_buf virtio_rng rng_core virtio_console virtio_balloon virtio virtio_ring [ 1.881465] CPU: 0 UID: 0 PID: 315 Comm: cat Not tainted 6.12.0-rc1 #1-NixOS [ 1.882326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 1.883684] Call Trace: [ 1.884103] <TASK> [ 1.884440] dump_stack_lvl+0x64/0x90 [ 1.884954] bad_page+0x70/0x110 [ 1.885468] __rmqueue_pcplist+0x209/0xd00 [ 1.886029] ? srso_return_thunk+0x5/0x5f [ 1.886572] ? pdu_read+0x36/0x50 [9pnet] [ 1.887177] get_page_from_freelist+0x2df/0x1910 [ 1.887788] ? srso_return_thunk+0x5/0x5f [ 1.888324] ? enqueue_entity+0xce/0x510 [ 1.888881] ? srso_return_thunk+0x5/0x5f [ 1.889415] ? pick_eevdf+0x76/0x1a0 [ 1.889970] ? update_curr+0x35/0x270 [ 1.890476] __alloc_pages_noprof+0x1a3/0x1150 [ 1.891158] ? srso_return_thunk+0x5/0x5f [ 1.891712] ? __mod_memcg_lruvec_state+0xa9/0x160 [ 1.892346] ? srso_return_thunk+0x5/0x5f [ 1.892919] ? __lruvec_stat_mod_folio+0x83/0xd0 [ 1.893521] alloc_pages_mpol_noprof+0x8f/0x1f0 [ 1.894148] folio_alloc_noprof+0x5b/0xb0 [ 1.894671] page_cache_ra_unbounded+0x11f/0x200 [ 1.895270] filemap_get_pages+0x538/0x6d0 [ 1.895813] ? srso_return_thunk+0x5/0x5f [ 1.896361] filemap_splice_read+0x136/0x320 [ 1.896948] backing_file_splice_read+0x52/0xa0 [ 1.897522] ovl_splice_read+0xd2/0xf0 [overlay] [ 1.898160] ? __pfx_ovl_file_accessed+0x10/0x10 [overlay] [ 1.898817] splice_direct_to_actor+0xb4/0x270 [ 1.899404] ? __pfx_direct_splice_actor+0x10/0x10 [ 1.900103] do_splice_direct+0x77/0xd0 [ 1.900627] ? __pfx_direct_file_splice_eof+0x10/0x10 [ 1.901308] do_sendfile+0x359/0x410 [ 1.901788] __x64_sys_sendfile64+0xb9/0xd0 [ 1.902370] do_syscall_64+0xb7/0x210 [ 1.902904] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1.903604] RIP: 0033:0x7fa9f3a7289e [ 1.904214] Code: 75 0e 00 f7 d8 64 89 02 b8 ff ff ff ff 31 d2 31 c9 31 ff 45 31 db c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 12 31 d2 31 c9 31 f6 31 ff 45 31 d2 45 31 db [ 1.906436] RSP: 002b:00007ffe6a82bde8 EFLAGS: 00000246 ORIG_RAX: 0000000000000028 [ 1.907400] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa9f3a7289e [ 1.908241] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000001 [ 1.909184] RBP: 00007ffe6a82be50 R08: 0000000000000000 R09: 0000000000000000 [ 1.910212] R10: 0000000001000000 R11: 0000000000000246 R12: 0000000000000001 [ 1.911117] R13: 0000000001000000 R14: 0000000000000001 R15: 0000000000000000 [ 1.911998] </TASK> [ 1.912376] Disabling lock debugging due to kernel taint [ 1.913479] list_del corruption. next->prev should be ffffc80e40b9d948, but was ffffc80e40b9d0c8. (next=ffffc80e40b9c7c8) [ 1.914823] ------------[ cut here ]------------ [ 1.915408] kernel BUG at lib/list_debug.c:65! [ 1.916050] Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 1.916785] CPU: 0 UID: 0 PID: 315 Comm: cat Tainted: G B 6.12.0-rc1 #1-NixOS [ 1.917877] Tainted: [B]=BAD_PAGE [ 1.918350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 1.919996] RIP: 0010:__list_del_entry_valid_or_report+0xcc/0xd0 [ 1.920903] Code: 89 fe 48 89 c2 48 c7 c7 70 52 41 ba e8 2d 91 ac ff 90 0f 0b 48 89 d1 48 c7 c7 c0 52 41 ba 48 89 f2 48 89 c6 e8 15 91 ac ff 90 <0f> 0b 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f [ 1.923423] RSP: 0018:ffff9ed880187748 EFLAGS: 00010246 [ 1.924210] RAX: 000000000000006d RBX: ffff94db3d83dc80 RCX: 0000000000000000 [ 1.925147] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 1.926051] RBP: ffffc80e40b9d940 R08: 0000000000000000 R09: 0000000000000000 [ 1.926940] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [ 1.927809] R13: ffff94db3d83dc80 R14: ffffc80e40b9d948 R15: ffff94db3ffd6180 [ 1.928695] FS: 00007fa9f396eb80(0000) GS:ffff94db3d800000(0000) knlGS:0000000000000000 [ 1.929728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.930540] CR2: 00000000004d1829 CR3: 0000000001dd2000 CR4: 0000000000350ef0 [ 1.931444] Call Trace: [ 1.931916] <TASK> [ 1.932357] ? die+0x36/0x90 [ 1.932831] ? do_trap+0xed/0x110 [ 1.933385] ? __list_del_entry_valid_or_report+0xcc/0xd0 [ 1.934073] ? do_error_trap+0x6a/0xa0 [ 1.934583] ? __list_del_entry_valid_or_report+0xcc/0xd0 [ 1.935242] ? exc_invalid_op+0x51/0x80 [ 1.935781] ? __list_del_entry_valid_or_report+0xcc/0xd0 [ 1.936484] ? asm_exc_invalid_op+0x1a/0x20 [ 1.937174] ? __list_del_entry_valid_or_report+0xcc/0xd0 [ 1.937926] ? __list_del_entry_valid_or_report+0xcb/0xd0 [ 1.938685] __rmqueue_pcplist+0xa5/0xd00 [ 1.939292] ? srso_return_thunk+0x5/0x5f [ 1.940004] ? __mod_memcg_lruvec_state+0xa9/0x160 [ 1.940758] ? srso_return_thunk+0x5/0x5f [ 1.941417] ? update_load_avg+0x7e/0x7f0 [ 1.942133] ? srso_return_thunk+0x5/0x5f [ 1.942838] ? srso_return_thunk+0x5/0x5f [ 1.943508] get_page_from_freelist+0x2df/0x1910 [ 1.944143] ? srso_return_thunk+0x5/0x5f [ 1.944696] ? check_preempt_wakeup_fair+0x1ee/0x240 [ 1.945335] ? srso_return_thunk+0x5/0x5f [ 1.945905] __alloc_pages_noprof+0x1a3/0x1150 [ 1.946489] ? __blk_flush_plug+0xf5/0x150 [ 1.947105] ? srso_return_thunk+0x5/0x5f [ 1.947629] ? __dquot_alloc_space+0x2a8/0x3a0 [ 1.948404] ? srso_return_thunk+0x5/0x5f [ 1.949116] ? __mod_memcg_lruvec_state+0xa9/0x160 [ 1.949888] alloc_pages_mpol_noprof+0x8f/0x1f0 [ 1.950514] folio_alloc_mpol_noprof+0x14/0x40 [ 1.951153] shmem_alloc_folio+0xa7/0xd0 [ 1.951692] ? shmem_recalc_inode+0x20/0x90 [ 1.952272] shmem_alloc_and_add_folio+0x109/0x490 [ 1.952940] ? filemap_get_entry+0x10f/0x1a0 [ 1.953570] ? srso_return_thunk+0x5/0x5f [ 1.954185] shmem_get_folio_gfp+0x248/0x610 [ 1.954791] shmem_write_begin+0x64/0x110 [ 1.955484] generic_perform_write+0xdf/0x2a0 [ 1.956239] shmem_file_write_iter+0x8a/0x90 [ 1.956882] iter_file_splice_write+0x33f/0x580 [ 1.957577] direct_splice_actor+0x54/0x140 [ 1.958178] splice_direct_to_actor+0xec/0x270 [ 1.958813] ? __pfx_direct_splice_actor+0x10/0x10 [ 1.959442] do_splice_direct+0x77/0xd0 [ 1.960018] ? __pfx_direct_file_splice_eof+0x10/0x10 [ 1.960726] do_sendfile+0x359/0x410 [ 1.961248] __x64_sys_sendfile64+0xb9/0xd0 [ 1.961905] do_syscall_64+0xb7/0x210 [ 1.962467] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1.963211] RIP: 0033:0x7fa9f3a7289e [ 1.963711] Code: 75 0e 00 f7 d8 64 89 02 b8 ff ff ff ff 31 d2 31 c9 31 ff 45 31 db c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 12 31 d2 31 c9 31 f6 31 ff 45 31 d2 45 31 db [ 1.965846] RSP: 002b:00007ffe6a82bde8 EFLAGS: 00000246 ORIG_RAX: 0000000000000028 [ 1.966788] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa9f3a7289e [ 1.967644] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000001 [ 1.968480] RBP: 00007ffe6a82be50 R08: 0000000000000000 R09: 0000000000000000 [ 1.969396] R10: 0000000001000000 R11: 0000000000000246 R12: 0000000000000001 [ 1.970315] R13: 0000000001000000 R14: 0000000000000001 R15: 0000000000000000 [ 1.971214] </TASK> [ 1.971572] Modules linked in: overlay 9p ext4 crc32c_generic crc16 mbcache jbd2 hid_generic usbhid hid 9pnet_virtio 9pnet netfs sr_mod virtio_net cdrom virtio_blk net_failover atkbd failover libps2 vivaldi_fmap crc32c_intel ata_piix libata scsi_mod uhci_hcd ehci_hcd virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev scsi_common i8042 serio rtc_cmos dm_mod dax virtio_gpu virtio_dma_buf virtio_rng rng_core virtio_console virtio_balloon virtio virtio_ring [ 1.976558] ---[ end trace 0000000000000000 ]--- [ 1.977219] RIP: 0010:__list_del_entry_valid_or_report+0xcc/0xd0 [ 1.978033] Code: 89 fe 48 89 c2 48 c7 c7 70 52 41 ba e8 2d 91 ac ff 90 0f 0b 48 89 d1 48 c7 c7 c0 52 41 ba 48 89 f2 48 89 c6 e8 15 91 ac ff 90 <0f> 0b 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f [ 1.980179] RSP: 0018:ffff9ed880187748 EFLAGS: 00010246 [ 1.980847] RAX: 000000000000006d RBX: ffff94db3d83dc80 RCX: 0000000000000000 [ 1.981705] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 1.982584] RBP: ffffc80e40b9d940 R08: 0000000000000000 R09: 0000000000000000 [ 1.983464] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [ 1.984358] R13: ffff94db3d83dc80 R14: ffffc80e40b9d948 R15: ffff94db3ffd6180 [ 1.987765] FS: 00007fa9f396eb80(0000) GS:ffff94db3d800000(0000) knlGS:0000000000000000 [ 1.988805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.989497] CR2: 00000000004d1829 CR3: 0000000001dd2000 CR4: 0000000000350ef0 [ 1.990418] note: cat[315] exited with preempt_count 2I bisected it back to
ee4cdf7ba857a894ad1650d6ab77669cbbfa329ewhich also seems to touch part of the 9p VFS code.Unfortunately the revert didn't apply cleanly on 6.12-rc1, so I couldn't meaningfully test whether a simple revert solves the problem.
The VMs get the Nix store mounted via 9p. In the store are basically all build artifacts including the stage-2 init script of the system that is booted into in the VM test.
The invocation basically looks like this:
qemu-system-x86_64 -cpu max \ -name machine \ -m 1024 \ -smp 1 \ -device virtio-rng-pci \ -net nic,netdev=user.0,model=virtio -netdev user,id=user.0,"$QEMU_NET_OPTS" \ -virtfs local,path=/nix/store,security_model=none,mount_tag=nix-store \ -virtfs local,path="${SHARED_DIR:-$TMPDIR/xchg}",security_model=none,mount_tag=shared \ -virtfs local,path="$TMPDIR"/xchg,security_model=none,mount_tag=xchg \ -drive cache=writeback,file="$NIX_DISK_IMAGE",id=drive1,if=none,index=1,werror=report -device virtio-blk-pci,bootindex=1,drive=drive1,serial=root \ -device virtio-net-pci,netdev=vlan1,mac=52:54:00:12:01:01 \ -netdev vde,id=vlan1,sock="$QEMU_VDE_SOCKET_1" \ -device virtio-keyboard \ -usb \ -device usb-tablet,bus=usb-bus.0 \ -kernel ${NIXPKGS_QEMU_KERNEL_machine:-/nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/kernel} \ -initrd /nix/store/qqalw1iq1wbgq3ndx0cvqn3bfypn56w2-initrd-linux-6.12-rc1/initrd \ -append "$(cat /nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/kernel-params) init=/nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/init regInfo=/nix/store/5izvfal6xm2rk51v0r1h2cxcng33paby-closure-info/registration console=ttyS0 $QEMU_KERNEL_PARAMS" \ $QEMU_OPTSIf you're using Nix, you can also reproduce this by running
nix-build nixos/tests/kernel-generic.nix -A
linux_testingon
5c19646b81db43dd7f4b6954f17d71a523009706from https://github.com/nixos/nixpkgs.To me, this seems like a regression in rc1.
Is there anything else I can do to help troubleshooting this?
With best regards
Maximilian
Good evening,
thanks a lot for the quick reply!
Quoted 2 lines
A fix for it is already pending in the vfs.fixes branch and -next:
https://lore.kernel.org/all/cbaf141ba6c0e2e209717d02746584072844841a.1727722269.git.osandov@fb.com/
I applied the patch on top of Linux 6.12-rc1 locally and I can confirm that this resolves the issue, thanks!
With best regards
Maximilian
Quoted 331 lines · Linux regression tracking (Thorsten Leemhuis)
Hi, Thorsten here, the Linux kernel's regression tracker. Top-posting for once, to make this easily accessible to everyone.
Thx for the report. Not my area of expertise (so everyone: corrent me if I'm wrong), but I suspect your problem might be a duplicate of the following report, which was bisected to the same commit from dhowells (
ee4cdf7ba857a8("netfs: Speed up buffered reading") [v6.12-rc1]):
https://lore.kernel.org/all/[email protected]/A fix for it is already pending in the vfs.fixes branch and -next:
https://lore.kernel.org/all/cbaf141ba6c0e2e209717d02746584072844841a.1727722269.git.osandov@fb.com/Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.Quoted 312 lines · Maximilian Bosch
Starting with Linux 6.12-rc1 the automatic VM tests of NixOS don't boot anymore and fail like this:
Quoted 1 line
mounting nix-store on /nix/.ro-store...
[ 1.604781] 9p: Installing v9fs 9p2000 file system support mounting tmpfs on /nix/.rw-store... mounting overlay on /nix/store... mounting shared on /tmp/shared... mounting xchg on /tmp/xchg... switch_root: can't execute '/nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/init': Exec format error [ 1.734997] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100 [ 1.736002] CPU: 0 UID: 0 PID: 1 Comm: switch_root Not tainted 6.12.0-rc1 #1-NixOS [ 1.736965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 1.738309] Call Trace: [ 1.738698] <TASK> [ 1.739034] panic+0x324/0x340 [ 1.739458] do_exit+0x92e/0xa90 [ 1.739919] ? count_memcg_events.constprop.0+0x1a/0x40 [ 1.740568] ? srso_return_thunk+0x5/0x5f [ 1.741095] ? handle_mm_fault+0xb0/0x2e0 [ 1.741709] do_group_exit+0x30/0x80 [ 1.742229] __x64_sys_exit_group+0x18/0x20 [ 1.742800] x64_sys_call+0x17f3/0x1800 [ 1.743326] do_syscall_64+0xb7/0x210 [ 1.743895] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1.744530] RIP: 0033:0x7f8e1a7b9d1d [ 1.745061] Code: 45 31 c0 45 31 d2 45 31 db c3 0f 1f 00 f3 0f 1e fa 48 8b 35 e5 e0 10 00 ba e7 00 00 00 eb 07 66 0f 1f 44 00 00 f4 89 d0 0f 05 <48> 3d 00 f0 ff ff 76 f3 f7 d8 64 89 06 eb ec 0f 1f 40 00 f3 0f 1e [ 1.747263] RSP: 002b:00007ffcb56d63b8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7 [ 1.748250] RAX: ffffffffffffffda RBX: 00007f8e1a8c9fa8 RCX: 00007f8e1a7b9d1d [ 1.749187] RDX: 00000000000000e7 RSI: ffffffffffffff88 RDI: 0000000000000001 [ 1.750050] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000 [ 1.750891] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 1.751706] R13: 0000000000000001 R14: 00007f8e1a8c8680 R15: 00007f8e1a8c9fc0 [ 1.752583] </TASK> [ 1.753010] Kernel Offset: 0xb800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)The failing script here is the initrd's /init when it tries to perform a
switch_rootto/sysroot:exec env -i $(type -P switch_root) "$targetRoot" "$stage2Init"Said "$stage2Init" file consistently gets a different hash when doing
sha256sumon it in the initrd script, but looks & behaves correct on the host. I reproduced the test failures on 4 different build machines and two architectures (x86_64-linux, aarch64-linux) now.The "$stage2Init" script is a shell-script itself. When trying to start the interpreter from its shebang inside the initrd (via
$targetRoot/nix/store/...-bash-5.2p32/bin/bash) and do theswitch_rootI get a different error:+ exec env -i /nix/store/akm69s5sngxyvqrzys326dss9rsrvbpy-extra-utils/bin/switch_root /mnt-root /nix/store/k3pm4iv44y7x7p74kky6cwxiswmr6kpi-nixos-system-machine-test/init [ 1.912859] list_del corruption. prev->next should be ffffc5cf80be0248, but was ffffc5cf80bd9208. (prev=ffffc5cf80bb4d48) [ 1.914237] ------------[ cut here ]------------ [ 1.915059] kernel BUG at lib/list_debug.c:62! [ 1.915854] Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 1.916739] CPU: 0 UID: 0 PID: 17 Comm: ksoftirqd/0 Not tainted 6.12.0-rc1 #1-NixOS [ 1.917837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 1.919354] RIP: 0010:__list_del_entry_valid_or_report+0xb4/0xd0 [ 1.920180] Code: 0f 0b 48 89 fe 48 89 ca 48 c7 c7 38 52 41 9f e8 42 91 ac ff 90 0f 0b 48 89 fe 48 89 c2 48 c7 c7 70 52 41 9f e8 2d 91 ac ff 90 <0f> 0b 48 89 d1 48 c7 c7 c0 52 41 9f 48 89 f2 48 89 c6 e8 15 91 ac [ 1.922636] RSP: 0018:ffff96f800093c00 EFLAGS: 00010046 [ 1.923563] RAX: 000000000000006d RBX: 0000000000000001 RCX: 0000000000000000 [ 1.924692] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 1.925664] RBP: 0000000000000341 R08: 0000000000000000 R09: 0000000000000000 [ 1.926646] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8fbebd83dc90 [ 1.927584] R13: ffffc5cf80be0240 R14: ffff8fbebd83dc80 R15: 000000000002f809 [ 1.928533] FS: 0000000000000000(0000) GS:ffff8fbebd800000(0000) knlGS:0000000000000000 [ 1.929647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.930431] CR2: 00007fed6f09b000 CR3: 0000000001e02000 CR4: 0000000000350ef0 [ 1.931333] Call Trace: [ 1.931727] <TASK> [ 1.932088] ? die+0x36/0x90 [ 1.932595] ? do_trap+0xed/0x110 [ 1.933047] ? __list_del_entry_valid_or_report+0xb4/0xd0 [ 1.933757] ? do_error_trap+0x6a/0xa0 [ 1.934390] ? __list_del_entry_valid_or_report+0xb4/0xd0 [ 1.935073] ? exc_invalid_op+0x51/0x80 [ 1.935627] ? __list_del_entry_valid_or_report+0xb4/0xd0 [ 1.936326] ? asm_exc_invalid_op+0x1a/0x20 [ 1.936904] ? __list_del_entry_valid_or_report+0xb4/0xd0 [ 1.937622] free_pcppages_bulk+0x130/0x280 [ 1.938151] free_unref_page_commit+0x21c/0x380 [ 1.938753] free_unref_page+0x472/0x4f0 [ 1.939343] __put_partials+0xee/0x130 [ 1.939921] ? rcu_do_batch+0x1f2/0x800 [ 1.940471] kmem_cache_free+0x2c3/0x370 [ 1.940990] rcu_do_batch+0x1f2/0x800 [ 1.941508] ? rcu_do_batch+0x180/0x800 [ 1.942031] rcu_core+0x182/0x340 [ 1.942500] handle_softirqs+0xe4/0x2f0 [ 1.943034] run_ksoftirqd+0x33/0x40 [ 1.943522] smpboot_thread_fn+0xdd/0x1d0 [ 1.944056] ? __pfx_smpboot_thread_fn+0x10/0x10 [ 1.944679] kthread+0xd0/0x100 [ 1.945126] ? __pfx_kthread+0x10/0x10 [ 1.945656] ret_from_fork+0x34/0x50 [ 1.946151] ? __pfx_kthread+0x10/0x10 [ 1.946680] ret_from_fork_asm+0x1a/0x30 [ 1.947269] </TASK> [ 1.947622] Modules linked in: overlay 9p ext4 crc32c_generic crc16 mbcache jbd2 hid_generic usbhid hid 9pnet_virtio 9pnet netfs sr_mod virtio_net cdrom virtio_blk net_failover atkbd failover libps2 vivaldi_fmap crc32c_intel ata_piix libata uhci_hcd scsi_mod ehci_hcd virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev scsi_common i8042 serio rtc_cmos dm_mod dax virtio_gpu virtio_dma_buf virtio_rng rng_core virtio_console virtio_balloon virtio virtio_ring [ 1.952291] ---[ end trace 0000000000000000 ]--- [ 1.952893] RIP: 0010:__list_del_entry_valid_or_report+0xb4/0xd0 [ 1.953678] Code: 0f 0b 48 89 fe 48 89 ca 48 c7 c7 38 52 41 9f e8 42 91 ac ff 90 0f 0b 48 89 fe 48 89 c2 48 c7 c7 70 52 41 9f e8 2d 91 ac ff 90 <0f> 0b 48 89 d1 48 c7 c7 c0 52 41 9f 48 89 f2 48 89 c6 e8 15 91 ac [ 1.955888] RSP: 0018:ffff96f800093c00 EFLAGS: 00010046 [ 1.956548] RAX: 000000000000006d RBX: 0000000000000001 RCX: 0000000000000000 [ 1.957436] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 1.958328] RBP: 0000000000000341 R08: 0000000000000000 R09: 0000000000000000 [ 1.959166] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8fbebd83dc90 [ 1.960044] R13: ffffc5cf80be0240 R14: ffff8fbebd83dc80 R15: 000000000002f809 [ 1.960905] FS: 0000000000000000(0000) GS:ffff8fbebd800000(0000) knlGS:0000000000000000 [ 1.961926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.962693] CR2: 00007fed6f09b000 CR3: 0000000001e02000 CR4: 0000000000350ef0 [ 1.963548] Kernel panic - not syncing: Fatal exception in interrupt [ 1.964417] Kernel Offset: 0x1ce00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)On a subsequent run to verify this, it failed earlier while reading $targetRoot/.../bash like this:
[ 1.871810] BUG: Bad page state in process cat pfn:2e74a [ 1.872481] page: refcount:1 mapcount:0 mapping:0000000000000000 index:0x1e5 pfn:0x2e74a [ 1.873499] flags: 0xffffc000000000(node=0|zone=1|lastcpupid=0x1ffff) [ 1.874260] raw: 00ffffc000000000 dead000000000100 dead000000000122 0000000000000000 [ 1.875250] raw: 00000000000001e5 0000000000000000 00000001ffffffff 0000000000000000 [ 1.876295] page dumped because: nonzero _refcount [ 1.876910] Modules linked in: overlay 9p ext4 crc32c_generic crc16 mbcache jbd2 hid_generic usbhid hid 9pnet_virtio 9pnet netfs sr_mod virtio_net cdrom virtio_blk net_failover atkbd failover libps2 vivaldi_fmap crc32c_intel ata_piix libata scsi_mod uhci_hcd ehci_hcd virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev scsi_common i8042 serio rtc_cmos dm_mod dax virtio_gpu virtio_dma_buf virtio_rng rng_core virtio_console virtio_balloon virtio virtio_ring [ 1.881465] CPU: 0 UID: 0 PID: 315 Comm: cat Not tainted 6.12.0-rc1 #1-NixOS [ 1.882326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 1.883684] Call Trace: [ 1.884103] <TASK> [ 1.884440] dump_stack_lvl+0x64/0x90 [ 1.884954] bad_page+0x70/0x110 [ 1.885468] __rmqueue_pcplist+0x209/0xd00 [ 1.886029] ? srso_return_thunk+0x5/0x5f [ 1.886572] ? pdu_read+0x36/0x50 [9pnet] [ 1.887177] get_page_from_freelist+0x2df/0x1910 [ 1.887788] ? srso_return_thunk+0x5/0x5f [ 1.888324] ? enqueue_entity+0xce/0x510 [ 1.888881] ? srso_return_thunk+0x5/0x5f [ 1.889415] ? pick_eevdf+0x76/0x1a0 [ 1.889970] ? update_curr+0x35/0x270 [ 1.890476] __alloc_pages_noprof+0x1a3/0x1150 [ 1.891158] ? srso_return_thunk+0x5/0x5f [ 1.891712] ? __mod_memcg_lruvec_state+0xa9/0x160 [ 1.892346] ? srso_return_thunk+0x5/0x5f [ 1.892919] ? __lruvec_stat_mod_folio+0x83/0xd0 [ 1.893521] alloc_pages_mpol_noprof+0x8f/0x1f0 [ 1.894148] folio_alloc_noprof+0x5b/0xb0 [ 1.894671] page_cache_ra_unbounded+0x11f/0x200 [ 1.895270] filemap_get_pages+0x538/0x6d0 [ 1.895813] ? srso_return_thunk+0x5/0x5f [ 1.896361] filemap_splice_read+0x136/0x320 [ 1.896948] backing_file_splice_read+0x52/0xa0 [ 1.897522] ovl_splice_read+0xd2/0xf0 [overlay] [ 1.898160] ? __pfx_ovl_file_accessed+0x10/0x10 [overlay] [ 1.898817] splice_direct_to_actor+0xb4/0x270 [ 1.899404] ? __pfx_direct_splice_actor+0x10/0x10 [ 1.900103] do_splice_direct+0x77/0xd0 [ 1.900627] ? __pfx_direct_file_splice_eof+0x10/0x10 [ 1.901308] do_sendfile+0x359/0x410 [ 1.901788] __x64_sys_sendfile64+0xb9/0xd0 [ 1.902370] do_syscall_64+0xb7/0x210 [ 1.902904] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1.903604] RIP: 0033:0x7fa9f3a7289e [ 1.904214] Code: 75 0e 00 f7 d8 64 89 02 b8 ff ff ff ff 31 d2 31 c9 31 ff 45 31 db c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 12 31 d2 31 c9 31 f6 31 ff 45 31 d2 45 31 db [ 1.906436] RSP: 002b:00007ffe6a82bde8 EFLAGS: 00000246 ORIG_RAX: 0000000000000028 [ 1.907400] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa9f3a7289e [ 1.908241] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000001 [ 1.909184] RBP: 00007ffe6a82be50 R08: 0000000000000000 R09: 0000000000000000 [ 1.910212] R10: 0000000001000000 R11: 0000000000000246 R12: 0000000000000001 [ 1.911117] R13: 0000000001000000 R14: 0000000000000001 R15: 0000000000000000 [ 1.911998] </TASK> [ 1.912376] Disabling lock debugging due to kernel taint [ 1.913479] list_del corruption. next->prev should be ffffc80e40b9d948, but was ffffc80e40b9d0c8. (next=ffffc80e40b9c7c8) [ 1.914823] ------------[ cut here ]------------ [ 1.915408] kernel BUG at lib/list_debug.c:65! [ 1.916050] Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 1.916785] CPU: 0 UID: 0 PID: 315 Comm: cat Tainted: G B 6.12.0-rc1 #1-NixOS [ 1.917877] Tainted: [B]=BAD_PAGE [ 1.918350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 1.919996] RIP: 0010:__list_del_entry_valid_or_report+0xcc/0xd0 [ 1.920903] Code: 89 fe 48 89 c2 48 c7 c7 70 52 41 ba e8 2d 91 ac ff 90 0f 0b 48 89 d1 48 c7 c7 c0 52 41 ba 48 89 f2 48 89 c6 e8 15 91 ac ff 90 <0f> 0b 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f [ 1.923423] RSP: 0018:ffff9ed880187748 EFLAGS: 00010246 [ 1.924210] RAX: 000000000000006d RBX: ffff94db3d83dc80 RCX: 0000000000000000 [ 1.925147] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 1.926051] RBP: ffffc80e40b9d940 R08: 0000000000000000 R09: 0000000000000000 [ 1.926940] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [ 1.927809] R13: ffff94db3d83dc80 R14: ffffc80e40b9d948 R15: ffff94db3ffd6180 [ 1.928695] FS: 00007fa9f396eb80(0000) GS:ffff94db3d800000(0000) knlGS:0000000000000000 [ 1.929728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.930540] CR2: 00000000004d1829 CR3: 0000000001dd2000 CR4: 0000000000350ef0 [ 1.931444] Call Trace: [ 1.931916] <TASK> [ 1.932357] ? die+0x36/0x90 [ 1.932831] ? do_trap+0xed/0x110 [ 1.933385] ? __list_del_entry_valid_or_report+0xcc/0xd0 [ 1.934073] ? do_error_trap+0x6a/0xa0 [ 1.934583] ? __list_del_entry_valid_or_report+0xcc/0xd0 [ 1.935242] ? exc_invalid_op+0x51/0x80 [ 1.935781] ? __list_del_entry_valid_or_report+0xcc/0xd0 [ 1.936484] ? asm_exc_invalid_op+0x1a/0x20 [ 1.937174] ? __list_del_entry_valid_or_report+0xcc/0xd0 [ 1.937926] ? __list_del_entry_valid_or_report+0xcb/0xd0 [ 1.938685] __rmqueue_pcplist+0xa5/0xd00 [ 1.939292] ? srso_return_thunk+0x5/0x5f [ 1.940004] ? __mod_memcg_lruvec_state+0xa9/0x160 [ 1.940758] ? srso_return_thunk+0x5/0x5f [ 1.941417] ? update_load_avg+0x7e/0x7f0 [ 1.942133] ? srso_return_thunk+0x5/0x5f [ 1.942838] ? srso_return_thunk+0x5/0x5f [ 1.943508] get_page_from_freelist+0x2df/0x1910 [ 1.944143] ? srso_return_thunk+0x5/0x5f [ 1.944696] ? check_preempt_wakeup_fair+0x1ee/0x240 [ 1.945335] ? srso_return_thunk+0x5/0x5f [ 1.945905] __alloc_pages_noprof+0x1a3/0x1150 [ 1.946489] ? __blk_flush_plug+0xf5/0x150 [ 1.947105] ? srso_return_thunk+0x5/0x5f [ 1.947629] ? __dquot_alloc_space+0x2a8/0x3a0 [ 1.948404] ? srso_return_thunk+0x5/0x5f [ 1.949116] ? __mod_memcg_lruvec_state+0xa9/0x160 [ 1.949888] alloc_pages_mpol_noprof+0x8f/0x1f0 [ 1.950514] folio_alloc_mpol_noprof+0x14/0x40 [ 1.951153] shmem_alloc_folio+0xa7/0xd0 [ 1.951692] ? shmem_recalc_inode+0x20/0x90 [ 1.952272] shmem_alloc_and_add_folio+0x109/0x490 [ 1.952940] ? filemap_get_entry+0x10f/0x1a0 [ 1.953570] ? srso_return_thunk+0x5/0x5f [ 1.954185] shmem_get_folio_gfp+0x248/0x610 [ 1.954791] shmem_write_begin+0x64/0x110 [ 1.955484] generic_perform_write+0xdf/0x2a0 [ 1.956239] shmem_file_write_iter+0x8a/0x90 [ 1.956882] iter_file_splice_write+0x33f/0x580 [ 1.957577] direct_splice_actor+0x54/0x140 [ 1.958178] splice_direct_to_actor+0xec/0x270 [ 1.958813] ? __pfx_direct_splice_actor+0x10/0x10 [ 1.959442] do_splice_direct+0x77/0xd0 [ 1.960018] ? __pfx_direct_file_splice_eof+0x10/0x10 [ 1.960726] do_sendfile+0x359/0x410 [ 1.961248] __x64_sys_sendfile64+0xb9/0xd0 [ 1.961905] do_syscall_64+0xb7/0x210 [ 1.962467] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1.963211] RIP: 0033:0x7fa9f3a7289e [ 1.963711] Code: 75 0e 00 f7 d8 64 89 02 b8 ff ff ff ff 31 d2 31 c9 31 ff 45 31 db c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 12 31 d2 31 c9 31 f6 31 ff 45 31 d2 45 31 db [ 1.965846] RSP: 002b:00007ffe6a82bde8 EFLAGS: 00000246 ORIG_RAX: 0000000000000028 [ 1.966788] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa9f3a7289e [ 1.967644] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000001 [ 1.968480] RBP: 00007ffe6a82be50 R08: 0000000000000000 R09: 0000000000000000 [ 1.969396] R10: 0000000001000000 R11: 0000000000000246 R12: 0000000000000001 [ 1.970315] R13: 0000000001000000 R14: 0000000000000001 R15: 0000000000000000 [ 1.971214] </TASK> [ 1.971572] Modules linked in: overlay 9p ext4 crc32c_generic crc16 mbcache jbd2 hid_generic usbhid hid 9pnet_virtio 9pnet netfs sr_mod virtio_net cdrom virtio_blk net_failover atkbd failover libps2 vivaldi_fmap crc32c_intel ata_piix libata scsi_mod uhci_hcd ehci_hcd virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev scsi_common i8042 serio rtc_cmos dm_mod dax virtio_gpu virtio_dma_buf virtio_rng rng_core virtio_console virtio_balloon virtio virtio_ring [ 1.976558] ---[ end trace 0000000000000000 ]--- [ 1.977219] RIP: 0010:__list_del_entry_valid_or_report+0xcc/0xd0 [ 1.978033] Code: 89 fe 48 89 c2 48 c7 c7 70 52 41 ba e8 2d 91 ac ff 90 0f 0b 48 89 d1 48 c7 c7 c0 52 41 ba 48 89 f2 48 89 c6 e8 15 91 ac ff 90 <0f> 0b 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f [ 1.980179] RSP: 0018:ffff9ed880187748 EFLAGS: 00010246 [ 1.980847] RAX: 000000000000006d RBX: ffff94db3d83dc80 RCX: 0000000000000000 [ 1.981705] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 1.982584] RBP: ffffc80e40b9d940 R08: 0000000000000000 R09: 0000000000000000 [ 1.983464] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [ 1.984358] R13: ffff94db3d83dc80 R14: ffffc80e40b9d948 R15: ffff94db3ffd6180 [ 1.987765] FS: 00007fa9f396eb80(0000) GS:ffff94db3d800000(0000) knlGS:0000000000000000 [ 1.988805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.989497] CR2: 00000000004d1829 CR3: 0000000001dd2000 CR4: 0000000000350ef0 [ 1.990418] note: cat[315] exited with preempt_count 2I bisected it back to
ee4cdf7ba857a894ad1650d6ab77669cbbfa329ewhich also seems to touch part of the 9p VFS code.Unfortunately the revert didn't apply cleanly on 6.12-rc1, so I couldn't meaningfully test whether a simple revert solves the problem.
The VMs get the Nix store mounted via 9p. In the store are basically all build artifacts including the stage-2 init script of the system that is booted into in the VM test.
The invocation basically looks like this:
qemu-system-x86_64 -cpu max \ -name machine \ -m 1024 \ -smp 1 \ -device virtio-rng-pci \ -net nic,netdev=user.0,model=virtio -netdev user,id=user.0,"$QEMU_NET_OPTS" \ -virtfs local,path=/nix/store,security_model=none,mount_tag=nix-store \ -virtfs local,path="${SHARED_DIR:-$TMPDIR/xchg}",security_model=none,mount_tag=shared \ -virtfs local,path="$TMPDIR"/xchg,security_model=none,mount_tag=xchg \ -drive cache=writeback,file="$NIX_DISK_IMAGE",id=drive1,if=none,index=1,werror=report -device virtio-blk-pci,bootindex=1,drive=drive1,serial=root \ -device virtio-net-pci,netdev=vlan1,mac=52:54:00:12:01:01 \ -netdev vde,id=vlan1,sock="$QEMU_VDE_SOCKET_1" \ -device virtio-keyboard \ -usb \ -device usb-tablet,bus=usb-bus.0 \ -kernel ${NIXPKGS_QEMU_KERNEL_machine:-/nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/kernel} \ -initrd /nix/store/qqalw1iq1wbgq3ndx0cvqn3bfypn56w2-initrd-linux-6.12-rc1/initrd \ -append "$(cat /nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/kernel-params) init=/nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/init regInfo=/nix/store/5izvfal6xm2rk51v0r1h2cxcng33paby-closure-info/registration console=ttyS0 $QEMU_KERNEL_PARAMS" \ $QEMU_OPTSIf you're using Nix, you can also reproduce this by running
nix-build nixos/tests/kernel-generic.nix -A
linux_testingon
5c19646b81db43dd7f4b6954f17d71a523009706from https://github.com/nixos/nixpkgs.To me, this seems like a regression in rc1.
Is there anything else I can do to help troubleshooting this?
With best regards
Maximilian
Quoted 8 lines · Maximilian Bosch
Good evening,
thanks a lot for the quick reply!
Quoted 2 lines
A fix for it is already pending in the vfs.fixes branch and -next:
https://lore.kernel.org/all/cbaf141ba6c0e2e209717d02746584072844841a.1727722269.git.osandov@fb.com/
Quoted 7 lines
I applied the patch on top of Linux 6.12-rc1 locally and I can confirm that this resolves the issue, thanks!
With best regards
Maximilian
Thanks for testing.
For the records:
iov_iter: fix advancing slot in iter_folioq_get_pages()
https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/commit/?h=vfs.fixes&id=0d24852bd71ec85ca0016b6d6fc997e6a3381552
https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/log/?h=vfs.fixes
Quoted 335 lines · Linux regression tracking (Thorsten Leemhuis)
Quoted 331 lines
Hi, Thorsten here, the Linux kernel's regression tracker. Top-posting for once, to make this easily accessible to everyone.
Thx for the report. Not my area of expertise (so everyone: corrent me if I'm wrong), but I suspect your problem might be a duplicate of the following report, which was bisected to the same commit from dhowells (
ee4cdf7ba857a8("netfs: Speed up buffered reading") [v6.12-rc1]):
https://lore.kernel.org/all/[email protected]/A fix for it is already pending in the vfs.fixes branch and -next:
https://lore.kernel.org/all/cbaf141ba6c0e2e209717d02746584072844841a.1727722269.git.osandov@fb.com/Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.Quoted 312 lines · Maximilian Bosch
Starting with Linux 6.12-rc1 the automatic VM tests of NixOS don't boot anymore and fail like this:
Quoted 1 line
mounting nix-store on /nix/.ro-store...
[ 1.604781] 9p: Installing v9fs 9p2000 file system support mounting tmpfs on /nix/.rw-store... mounting overlay on /nix/store... mounting shared on /tmp/shared... mounting xchg on /tmp/xchg... switch_root: can't execute '/nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/init': Exec format error [ 1.734997] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100 [ 1.736002] CPU: 0 UID: 0 PID: 1 Comm: switch_root Not tainted 6.12.0-rc1 #1-NixOS [ 1.736965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 1.738309] Call Trace: [ 1.738698] <TASK> [ 1.739034] panic+0x324/0x340 [ 1.739458] do_exit+0x92e/0xa90 [ 1.739919] ? count_memcg_events.constprop.0+0x1a/0x40 [ 1.740568] ? srso_return_thunk+0x5/0x5f [ 1.741095] ? handle_mm_fault+0xb0/0x2e0 [ 1.741709] do_group_exit+0x30/0x80 [ 1.742229] __x64_sys_exit_group+0x18/0x20 [ 1.742800] x64_sys_call+0x17f3/0x1800 [ 1.743326] do_syscall_64+0xb7/0x210 [ 1.743895] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1.744530] RIP: 0033:0x7f8e1a7b9d1d [ 1.745061] Code: 45 31 c0 45 31 d2 45 31 db c3 0f 1f 00 f3 0f 1e fa 48 8b 35 e5 e0 10 00 ba e7 00 00 00 eb 07 66 0f 1f 44 00 00 f4 89 d0 0f 05 <48> 3d 00 f0 ff ff 76 f3 f7 d8 64 89 06 eb ec 0f 1f 40 00 f3 0f 1e [ 1.747263] RSP: 002b:00007ffcb56d63b8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7 [ 1.748250] RAX: ffffffffffffffda RBX: 00007f8e1a8c9fa8 RCX: 00007f8e1a7b9d1d [ 1.749187] RDX: 00000000000000e7 RSI: ffffffffffffff88 RDI: 0000000000000001 [ 1.750050] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000 [ 1.750891] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 1.751706] R13: 0000000000000001 R14: 00007f8e1a8c8680 R15: 00007f8e1a8c9fc0 [ 1.752583] </TASK> [ 1.753010] Kernel Offset: 0xb800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)The failing script here is the initrd's /init when it tries to perform a
switch_rootto/sysroot:exec env -i $(type -P switch_root) "$targetRoot" "$stage2Init"Said "$stage2Init" file consistently gets a different hash when doing
sha256sumon it in the initrd script, but looks & behaves correct on the host. I reproduced the test failures on 4 different build machines and two architectures (x86_64-linux, aarch64-linux) now.The "$stage2Init" script is a shell-script itself. When trying to start the interpreter from its shebang inside the initrd (via
$targetRoot/nix/store/...-bash-5.2p32/bin/bash) and do theswitch_rootI get a different error:+ exec env -i /nix/store/akm69s5sngxyvqrzys326dss9rsrvbpy-extra-utils/bin/switch_root /mnt-root /nix/store/k3pm4iv44y7x7p74kky6cwxiswmr6kpi-nixos-system-machine-test/init [ 1.912859] list_del corruption. prev->next should be ffffc5cf80be0248, but was ffffc5cf80bd9208. (prev=ffffc5cf80bb4d48) [ 1.914237] ------------[ cut here ]------------ [ 1.915059] kernel BUG at lib/list_debug.c:62! [ 1.915854] Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 1.916739] CPU: 0 UID: 0 PID: 17 Comm: ksoftirqd/0 Not tainted 6.12.0-rc1 #1-NixOS [ 1.917837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 1.919354] RIP: 0010:__list_del_entry_valid_or_report+0xb4/0xd0 [ 1.920180] Code: 0f 0b 48 89 fe 48 89 ca 48 c7 c7 38 52 41 9f e8 42 91 ac ff 90 0f 0b 48 89 fe 48 89 c2 48 c7 c7 70 52 41 9f e8 2d 91 ac ff 90 <0f> 0b 48 89 d1 48 c7 c7 c0 52 41 9f 48 89 f2 48 89 c6 e8 15 91 ac [ 1.922636] RSP: 0018:ffff96f800093c00 EFLAGS: 00010046 [ 1.923563] RAX: 000000000000006d RBX: 0000000000000001 RCX: 0000000000000000 [ 1.924692] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 1.925664] RBP: 0000000000000341 R08: 0000000000000000 R09: 0000000000000000 [ 1.926646] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8fbebd83dc90 [ 1.927584] R13: ffffc5cf80be0240 R14: ffff8fbebd83dc80 R15: 000000000002f809 [ 1.928533] FS: 0000000000000000(0000) GS:ffff8fbebd800000(0000) knlGS:0000000000000000 [ 1.929647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.930431] CR2: 00007fed6f09b000 CR3: 0000000001e02000 CR4: 0000000000350ef0 [ 1.931333] Call Trace: [ 1.931727] <TASK> [ 1.932088] ? die+0x36/0x90 [ 1.932595] ? do_trap+0xed/0x110 [ 1.933047] ? __list_del_entry_valid_or_report+0xb4/0xd0 [ 1.933757] ? do_error_trap+0x6a/0xa0 [ 1.934390] ? __list_del_entry_valid_or_report+0xb4/0xd0 [ 1.935073] ? exc_invalid_op+0x51/0x80 [ 1.935627] ? __list_del_entry_valid_or_report+0xb4/0xd0 [ 1.936326] ? asm_exc_invalid_op+0x1a/0x20 [ 1.936904] ? __list_del_entry_valid_or_report+0xb4/0xd0 [ 1.937622] free_pcppages_bulk+0x130/0x280 [ 1.938151] free_unref_page_commit+0x21c/0x380 [ 1.938753] free_unref_page+0x472/0x4f0 [ 1.939343] __put_partials+0xee/0x130 [ 1.939921] ? rcu_do_batch+0x1f2/0x800 [ 1.940471] kmem_cache_free+0x2c3/0x370 [ 1.940990] rcu_do_batch+0x1f2/0x800 [ 1.941508] ? rcu_do_batch+0x180/0x800 [ 1.942031] rcu_core+0x182/0x340 [ 1.942500] handle_softirqs+0xe4/0x2f0 [ 1.943034] run_ksoftirqd+0x33/0x40 [ 1.943522] smpboot_thread_fn+0xdd/0x1d0 [ 1.944056] ? __pfx_smpboot_thread_fn+0x10/0x10 [ 1.944679] kthread+0xd0/0x100 [ 1.945126] ? __pfx_kthread+0x10/0x10 [ 1.945656] ret_from_fork+0x34/0x50 [ 1.946151] ? __pfx_kthread+0x10/0x10 [ 1.946680] ret_from_fork_asm+0x1a/0x30 [ 1.947269] </TASK> [ 1.947622] Modules linked in: overlay 9p ext4 crc32c_generic crc16 mbcache jbd2 hid_generic usbhid hid 9pnet_virtio 9pnet netfs sr_mod virtio_net cdrom virtio_blk net_failover atkbd failover libps2 vivaldi_fmap crc32c_intel ata_piix libata uhci_hcd scsi_mod ehci_hcd virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev scsi_common i8042 serio rtc_cmos dm_mod dax virtio_gpu virtio_dma_buf virtio_rng rng_core virtio_console virtio_balloon virtio virtio_ring [ 1.952291] ---[ end trace 0000000000000000 ]--- [ 1.952893] RIP: 0010:__list_del_entry_valid_or_report+0xb4/0xd0 [ 1.953678] Code: 0f 0b 48 89 fe 48 89 ca 48 c7 c7 38 52 41 9f e8 42 91 ac ff 90 0f 0b 48 89 fe 48 89 c2 48 c7 c7 70 52 41 9f e8 2d 91 ac ff 90 <0f> 0b 48 89 d1 48 c7 c7 c0 52 41 9f 48 89 f2 48 89 c6 e8 15 91 ac [ 1.955888] RSP: 0018:ffff96f800093c00 EFLAGS: 00010046 [ 1.956548] RAX: 000000000000006d RBX: 0000000000000001 RCX: 0000000000000000 [ 1.957436] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 1.958328] RBP: 0000000000000341 R08: 0000000000000000 R09: 0000000000000000 [ 1.959166] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8fbebd83dc90 [ 1.960044] R13: ffffc5cf80be0240 R14: ffff8fbebd83dc80 R15: 000000000002f809 [ 1.960905] FS: 0000000000000000(0000) GS:ffff8fbebd800000(0000) knlGS:0000000000000000 [ 1.961926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.962693] CR2: 00007fed6f09b000 CR3: 0000000001e02000 CR4: 0000000000350ef0 [ 1.963548] Kernel panic - not syncing: Fatal exception in interrupt [ 1.964417] Kernel Offset: 0x1ce00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)On a subsequent run to verify this, it failed earlier while reading $targetRoot/.../bash like this:
[ 1.871810] BUG: Bad page state in process cat pfn:2e74a [ 1.872481] page: refcount:1 mapcount:0 mapping:0000000000000000 index:0x1e5 pfn:0x2e74a [ 1.873499] flags: 0xffffc000000000(node=0|zone=1|lastcpupid=0x1ffff) [ 1.874260] raw: 00ffffc000000000 dead000000000100 dead000000000122 0000000000000000 [ 1.875250] raw: 00000000000001e5 0000000000000000 00000001ffffffff 0000000000000000 [ 1.876295] page dumped because: nonzero _refcount [ 1.876910] Modules linked in: overlay 9p ext4 crc32c_generic crc16 mbcache jbd2 hid_generic usbhid hid 9pnet_virtio 9pnet netfs sr_mod virtio_net cdrom virtio_blk net_failover atkbd failover libps2 vivaldi_fmap crc32c_intel ata_piix libata scsi_mod uhci_hcd ehci_hcd virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev scsi_common i8042 serio rtc_cmos dm_mod dax virtio_gpu virtio_dma_buf virtio_rng rng_core virtio_console virtio_balloon virtio virtio_ring [ 1.881465] CPU: 0 UID: 0 PID: 315 Comm: cat Not tainted 6.12.0-rc1 #1-NixOS [ 1.882326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 1.883684] Call Trace: [ 1.884103] <TASK> [ 1.884440] dump_stack_lvl+0x64/0x90 [ 1.884954] bad_page+0x70/0x110 [ 1.885468] __rmqueue_pcplist+0x209/0xd00 [ 1.886029] ? srso_return_thunk+0x5/0x5f [ 1.886572] ? pdu_read+0x36/0x50 [9pnet] [ 1.887177] get_page_from_freelist+0x2df/0x1910 [ 1.887788] ? srso_return_thunk+0x5/0x5f [ 1.888324] ? enqueue_entity+0xce/0x510 [ 1.888881] ? srso_return_thunk+0x5/0x5f [ 1.889415] ? pick_eevdf+0x76/0x1a0 [ 1.889970] ? update_curr+0x35/0x270 [ 1.890476] __alloc_pages_noprof+0x1a3/0x1150 [ 1.891158] ? srso_return_thunk+0x5/0x5f [ 1.891712] ? __mod_memcg_lruvec_state+0xa9/0x160 [ 1.892346] ? srso_return_thunk+0x5/0x5f [ 1.892919] ? __lruvec_stat_mod_folio+0x83/0xd0 [ 1.893521] alloc_pages_mpol_noprof+0x8f/0x1f0 [ 1.894148] folio_alloc_noprof+0x5b/0xb0 [ 1.894671] page_cache_ra_unbounded+0x11f/0x200 [ 1.895270] filemap_get_pages+0x538/0x6d0 [ 1.895813] ? srso_return_thunk+0x5/0x5f [ 1.896361] filemap_splice_read+0x136/0x320 [ 1.896948] backing_file_splice_read+0x52/0xa0 [ 1.897522] ovl_splice_read+0xd2/0xf0 [overlay] [ 1.898160] ? __pfx_ovl_file_accessed+0x10/0x10 [overlay] [ 1.898817] splice_direct_to_actor+0xb4/0x270 [ 1.899404] ? __pfx_direct_splice_actor+0x10/0x10 [ 1.900103] do_splice_direct+0x77/0xd0 [ 1.900627] ? __pfx_direct_file_splice_eof+0x10/0x10 [ 1.901308] do_sendfile+0x359/0x410 [ 1.901788] __x64_sys_sendfile64+0xb9/0xd0 [ 1.902370] do_syscall_64+0xb7/0x210 [ 1.902904] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1.903604] RIP: 0033:0x7fa9f3a7289e [ 1.904214] Code: 75 0e 00 f7 d8 64 89 02 b8 ff ff ff ff 31 d2 31 c9 31 ff 45 31 db c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 12 31 d2 31 c9 31 f6 31 ff 45 31 d2 45 31 db [ 1.906436] RSP: 002b:00007ffe6a82bde8 EFLAGS: 00000246 ORIG_RAX: 0000000000000028 [ 1.907400] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa9f3a7289e [ 1.908241] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000001 [ 1.909184] RBP: 00007ffe6a82be50 R08: 0000000000000000 R09: 0000000000000000 [ 1.910212] R10: 0000000001000000 R11: 0000000000000246 R12: 0000000000000001 [ 1.911117] R13: 0000000001000000 R14: 0000000000000001 R15: 0000000000000000 [ 1.911998] </TASK> [ 1.912376] Disabling lock debugging due to kernel taint [ 1.913479] list_del corruption. next->prev should be ffffc80e40b9d948, but was ffffc80e40b9d0c8. (next=ffffc80e40b9c7c8) [ 1.914823] ------------[ cut here ]------------ [ 1.915408] kernel BUG at lib/list_debug.c:65! [ 1.916050] Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 1.916785] CPU: 0 UID: 0 PID: 315 Comm: cat Tainted: G B 6.12.0-rc1 #1-NixOS [ 1.917877] Tainted: [B]=BAD_PAGE [ 1.918350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 1.919996] RIP: 0010:__list_del_entry_valid_or_report+0xcc/0xd0 [ 1.920903] Code: 89 fe 48 89 c2 48 c7 c7 70 52 41 ba e8 2d 91 ac ff 90 0f 0b 48 89 d1 48 c7 c7 c0 52 41 ba 48 89 f2 48 89 c6 e8 15 91 ac ff 90 <0f> 0b 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f [ 1.923423] RSP: 0018:ffff9ed880187748 EFLAGS: 00010246 [ 1.924210] RAX: 000000000000006d RBX: ffff94db3d83dc80 RCX: 0000000000000000 [ 1.925147] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 1.926051] RBP: ffffc80e40b9d940 R08: 0000000000000000 R09: 0000000000000000 [ 1.926940] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [ 1.927809] R13: ffff94db3d83dc80 R14: ffffc80e40b9d948 R15: ffff94db3ffd6180 [ 1.928695] FS: 00007fa9f396eb80(0000) GS:ffff94db3d800000(0000) knlGS:0000000000000000 [ 1.929728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.930540] CR2: 00000000004d1829 CR3: 0000000001dd2000 CR4: 0000000000350ef0 [ 1.931444] Call Trace: [ 1.931916] <TASK> [ 1.932357] ? die+0x36/0x90 [ 1.932831] ? do_trap+0xed/0x110 [ 1.933385] ? __list_del_entry_valid_or_report+0xcc/0xd0 [ 1.934073] ? do_error_trap+0x6a/0xa0 [ 1.934583] ? __list_del_entry_valid_or_report+0xcc/0xd0 [ 1.935242] ? exc_invalid_op+0x51/0x80 [ 1.935781] ? __list_del_entry_valid_or_report+0xcc/0xd0 [ 1.936484] ? asm_exc_invalid_op+0x1a/0x20 [ 1.937174] ? __list_del_entry_valid_or_report+0xcc/0xd0 [ 1.937926] ? __list_del_entry_valid_or_report+0xcb/0xd0 [ 1.938685] __rmqueue_pcplist+0xa5/0xd00 [ 1.939292] ? srso_return_thunk+0x5/0x5f [ 1.940004] ? __mod_memcg_lruvec_state+0xa9/0x160 [ 1.940758] ? srso_return_thunk+0x5/0x5f [ 1.941417] ? update_load_avg+0x7e/0x7f0 [ 1.942133] ? srso_return_thunk+0x5/0x5f [ 1.942838] ? srso_return_thunk+0x5/0x5f [ 1.943508] get_page_from_freelist+0x2df/0x1910 [ 1.944143] ? srso_return_thunk+0x5/0x5f [ 1.944696] ? check_preempt_wakeup_fair+0x1ee/0x240 [ 1.945335] ? srso_return_thunk+0x5/0x5f [ 1.945905] __alloc_pages_noprof+0x1a3/0x1150 [ 1.946489] ? __blk_flush_plug+0xf5/0x150 [ 1.947105] ? srso_return_thunk+0x5/0x5f [ 1.947629] ? __dquot_alloc_space+0x2a8/0x3a0 [ 1.948404] ? srso_return_thunk+0x5/0x5f [ 1.949116] ? __mod_memcg_lruvec_state+0xa9/0x160 [ 1.949888] alloc_pages_mpol_noprof+0x8f/0x1f0 [ 1.950514] folio_alloc_mpol_noprof+0x14/0x40 [ 1.951153] shmem_alloc_folio+0xa7/0xd0 [ 1.951692] ? shmem_recalc_inode+0x20/0x90 [ 1.952272] shmem_alloc_and_add_folio+0x109/0x490 [ 1.952940] ? filemap_get_entry+0x10f/0x1a0 [ 1.953570] ? srso_return_thunk+0x5/0x5f [ 1.954185] shmem_get_folio_gfp+0x248/0x610 [ 1.954791] shmem_write_begin+0x64/0x110 [ 1.955484] generic_perform_write+0xdf/0x2a0 [ 1.956239] shmem_file_write_iter+0x8a/0x90 [ 1.956882] iter_file_splice_write+0x33f/0x580 [ 1.957577] direct_splice_actor+0x54/0x140 [ 1.958178] splice_direct_to_actor+0xec/0x270 [ 1.958813] ? __pfx_direct_splice_actor+0x10/0x10 [ 1.959442] do_splice_direct+0x77/0xd0 [ 1.960018] ? __pfx_direct_file_splice_eof+0x10/0x10 [ 1.960726] do_sendfile+0x359/0x410 [ 1.961248] __x64_sys_sendfile64+0xb9/0xd0 [ 1.961905] do_syscall_64+0xb7/0x210 [ 1.962467] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1.963211] RIP: 0033:0x7fa9f3a7289e [ 1.963711] Code: 75 0e 00 f7 d8 64 89 02 b8 ff ff ff ff 31 d2 31 c9 31 ff 45 31 db c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 12 31 d2 31 c9 31 f6 31 ff 45 31 d2 45 31 db [ 1.965846] RSP: 002b:00007ffe6a82bde8 EFLAGS: 00000246 ORIG_RAX: 0000000000000028 [ 1.966788] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa9f3a7289e [ 1.967644] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000001 [ 1.968480] RBP: 00007ffe6a82be50 R08: 0000000000000000 R09: 0000000000000000 [ 1.969396] R10: 0000000001000000 R11: 0000000000000246 R12: 0000000000000001 [ 1.970315] R13: 0000000001000000 R14: 0000000000000001 R15: 0000000000000000 [ 1.971214] </TASK> [ 1.971572] Modules linked in: overlay 9p ext4 crc32c_generic crc16 mbcache jbd2 hid_generic usbhid hid 9pnet_virtio 9pnet netfs sr_mod virtio_net cdrom virtio_blk net_failover atkbd failover libps2 vivaldi_fmap crc32c_intel ata_piix libata scsi_mod uhci_hcd ehci_hcd virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev scsi_common i8042 serio rtc_cmos dm_mod dax virtio_gpu virtio_dma_buf virtio_rng rng_core virtio_console virtio_balloon virtio virtio_ring [ 1.976558] ---[ end trace 0000000000000000 ]--- [ 1.977219] RIP: 0010:__list_del_entry_valid_or_report+0xcc/0xd0 [ 1.978033] Code: 89 fe 48 89 c2 48 c7 c7 70 52 41 ba e8 2d 91 ac ff 90 0f 0b 48 89 d1 48 c7 c7 c0 52 41 ba 48 89 f2 48 89 c6 e8 15 91 ac ff 90 <0f> 0b 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f [ 1.980179] RSP: 0018:ffff9ed880187748 EFLAGS: 00010246 [ 1.980847] RAX: 000000000000006d RBX: ffff94db3d83dc80 RCX: 0000000000000000 [ 1.981705] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 1.982584] RBP: ffffc80e40b9d940 R08: 0000000000000000 R09: 0000000000000000 [ 1.983464] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [ 1.984358] R13: ffff94db3d83dc80 R14: ffffc80e40b9d948 R15: ffff94db3ffd6180 [ 1.987765] FS: 00007fa9f396eb80(0000) GS:ffff94db3d800000(0000) knlGS:0000000000000000 [ 1.988805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.989497] CR2: 00000000004d1829 CR3: 0000000001dd2000 CR4: 0000000000350ef0 [ 1.990418] note: cat[315] exited with preempt_count 2I bisected it back to
ee4cdf7ba857a894ad1650d6ab77669cbbfa329ewhich also seems to touch part of the 9p VFS code.Unfortunately the revert didn't apply cleanly on 6.12-rc1, so I couldn't meaningfully test whether a simple revert solves the problem.
The VMs get the Nix store mounted via 9p. In the store are basically all build artifacts including the stage-2 init script of the system that is booted into in the VM test.
The invocation basically looks like this:
qemu-system-x86_64 -cpu max \ -name machine \ -m 1024 \ -smp 1 \ -device virtio-rng-pci \ -net nic,netdev=user.0,model=virtio -netdev user,id=user.0,"$QEMU_NET_OPTS" \ -virtfs local,path=/nix/store,security_model=none,mount_tag=nix-store \ -virtfs local,path="${SHARED_DIR:-$TMPDIR/xchg}",security_model=none,mount_tag=shared \ -virtfs local,path="$TMPDIR"/xchg,security_model=none,mount_tag=xchg \ -drive cache=writeback,file="$NIX_DISK_IMAGE",id=drive1,if=none,index=1,werror=report -device virtio-blk-pci,bootindex=1,drive=drive1,serial=root \ -device virtio-net-pci,netdev=vlan1,mac=52:54:00:12:01:01 \ -netdev vde,id=vlan1,sock="$QEMU_VDE_SOCKET_1" \ -device virtio-keyboard \ -usb \ -device usb-tablet,bus=usb-bus.0 \ -kernel ${NIXPKGS_QEMU_KERNEL_machine:-/nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/kernel} \ -initrd /nix/store/qqalw1iq1wbgq3ndx0cvqn3bfypn56w2-initrd-linux-6.12-rc1/initrd \ -append "$(cat /nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/kernel-params) init=/nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/init regInfo=/nix/store/5izvfal6xm2rk51v0r1h2cxcng33paby-closure-info/registration console=ttyS0 $QEMU_KERNEL_PARAMS" \ $QEMU_OPTSIf you're using Nix, you can also reproduce this by running
nix-build nixos/tests/kernel-generic.nix -A
linux_testingon
5c19646b81db43dd7f4b6954f17d71a523009706from https://github.com/nixos/nixpkgs.To me, this seems like a regression in rc1.
Is there anything else I can do to help troubleshooting this?
With best regards
Maximilian
Hi,
Quoted 11 lines · Sedat Dilek
Quoted 10 lines · Maximilian Bosch
Good evening,
thanks a lot for the quick reply!
Quoted 2 lines
A fix for it is already pending in the vfs.fixes branch and -next:
https://lore.kernel.org/all/cbaf141ba6c0e2e209717d02746584072844841a.1727722269.git.osandov@fb.com/I applied the patch on top of Linux 6.12-rc1 locally and I can confirm that this resolves the issue, thanks!
Maximilian, would you like to re-run the test a few times? I wonder if there is another intermittend bug related to the same commit.
Quoted 6 lines
Quoted 5 lines
With best regards
Maximilian
Quoted 8 lines
Thanks for testing.
For the records:
iov_iter: fix advancing slot initer_folioq_get_pages()
https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/commit/?h=vfs.fixes&id=0d24852bd71ec85ca0016b6d6fc997e6a3381552https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/log/?h=vfs.fixes
I’m still seeing a kernel oops after the fix in 6.12-rc3, but I’ve noticed that the issue is no longer 100% reproducible. Most of the time, the system crashes. Before this fix it was 100% reproducible.
When using the nix testing, I have to force the test to re-run.
result=$(readlink -f ./result); rm ./result && nix-store --delete $result
nix-build -v nixos/tests/kernel-generic.nix -A linux_testing
So may be there is a new bug showing up after the fix. I have reported it.
https://lore.kernel.org/regressions/[email protected]/T/#u
-antony
Quoted 336 lines
Quoted 336 lines · Linux regression tracking (Thorsten Leemhuis)
Quoted 331 lines
Hi, Thorsten here, the Linux kernel's regression tracker. Top-posting for once, to make this easily accessible to everyone.
Thx for the report. Not my area of expertise (so everyone: corrent me if I'm wrong), but I suspect your problem might be a duplicate of the following report, which was bisected to the same commit from dhowells (
ee4cdf7ba857a8("netfs: Speed up buffered reading") [v6.12-rc1]):
https://lore.kernel.org/all/[email protected]/A fix for it is already pending in the vfs.fixes branch and -next:
https://lore.kernel.org/all/cbaf141ba6c0e2e209717d02746584072844841a.1727722269.git.osandov@fb.com/Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.Quoted 312 lines · Maximilian Bosch
Starting with Linux 6.12-rc1 the automatic VM tests of NixOS don't boot anymore and fail like this:
Quoted 1 line
mounting nix-store on /nix/.ro-store...
[ 1.604781] 9p: Installing v9fs 9p2000 file system support mounting tmpfs on /nix/.rw-store... mounting overlay on /nix/store... mounting shared on /tmp/shared... mounting xchg on /tmp/xchg... switch_root: can't execute '/nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/init': Exec format error [ 1.734997] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100 [ 1.736002] CPU: 0 UID: 0 PID: 1 Comm: switch_root Not tainted 6.12.0-rc1 #1-NixOS [ 1.736965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 1.738309] Call Trace: [ 1.738698] <TASK> [ 1.739034] panic+0x324/0x340 [ 1.739458] do_exit+0x92e/0xa90 [ 1.739919] ? count_memcg_events.constprop.0+0x1a/0x40 [ 1.740568] ? srso_return_thunk+0x5/0x5f [ 1.741095] ? handle_mm_fault+0xb0/0x2e0 [ 1.741709] do_group_exit+0x30/0x80 [ 1.742229] __x64_sys_exit_group+0x18/0x20 [ 1.742800] x64_sys_call+0x17f3/0x1800 [ 1.743326] do_syscall_64+0xb7/0x210 [ 1.743895] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1.744530] RIP: 0033:0x7f8e1a7b9d1d [ 1.745061] Code: 45 31 c0 45 31 d2 45 31 db c3 0f 1f 00 f3 0f 1e fa 48 8b 35 e5 e0 10 00 ba e7 00 00 00 eb 07 66 0f 1f 44 00 00 f4 89 d0 0f 05 <48> 3d 00 f0 ff ff 76 f3 f7 d8 64 89 06 eb ec 0f 1f 40 00 f3 0f 1e [ 1.747263] RSP: 002b:00007ffcb56d63b8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7 [ 1.748250] RAX: ffffffffffffffda RBX: 00007f8e1a8c9fa8 RCX: 00007f8e1a7b9d1d [ 1.749187] RDX: 00000000000000e7 RSI: ffffffffffffff88 RDI: 0000000000000001 [ 1.750050] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000 [ 1.750891] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 1.751706] R13: 0000000000000001 R14: 00007f8e1a8c8680 R15: 00007f8e1a8c9fc0 [ 1.752583] </TASK> [ 1.753010] Kernel Offset: 0xb800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)The failing script here is the initrd's /init when it tries to perform a
switch_rootto/sysroot:exec env -i $(type -P switch_root) "$targetRoot" "$stage2Init"Said "$stage2Init" file consistently gets a different hash when doing
sha256sumon it in the initrd script, but looks & behaves correct on the host. I reproduced the test failures on 4 different build machines and two architectures (x86_64-linux, aarch64-linux) now.The "$stage2Init" script is a shell-script itself. When trying to start the interpreter from its shebang inside the initrd (via
$targetRoot/nix/store/...-bash-5.2p32/bin/bash) and do theswitch_rootI get a different error:+ exec env -i /nix/store/akm69s5sngxyvqrzys326dss9rsrvbpy-extra-utils/bin/switch_root /mnt-root /nix/store/k3pm4iv44y7x7p74kky6cwxiswmr6kpi-nixos-system-machine-test/init [ 1.912859] list_del corruption. prev->next should be ffffc5cf80be0248, but was ffffc5cf80bd9208. (prev=ffffc5cf80bb4d48) [ 1.914237] ------------[ cut here ]------------ [ 1.915059] kernel BUG at lib/list_debug.c:62! [ 1.915854] Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 1.916739] CPU: 0 UID: 0 PID: 17 Comm: ksoftirqd/0 Not tainted 6.12.0-rc1 #1-NixOS [ 1.917837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 1.919354] RIP: 0010:__list_del_entry_valid_or_report+0xb4/0xd0 [ 1.920180] Code: 0f 0b 48 89 fe 48 89 ca 48 c7 c7 38 52 41 9f e8 42 91 ac ff 90 0f 0b 48 89 fe 48 89 c2 48 c7 c7 70 52 41 9f e8 2d 91 ac ff 90 <0f> 0b 48 89 d1 48 c7 c7 c0 52 41 9f 48 89 f2 48 89 c6 e8 15 91 ac [ 1.922636] RSP: 0018:ffff96f800093c00 EFLAGS: 00010046 [ 1.923563] RAX: 000000000000006d RBX: 0000000000000001 RCX: 0000000000000000 [ 1.924692] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 1.925664] RBP: 0000000000000341 R08: 0000000000000000 R09: 0000000000000000 [ 1.926646] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8fbebd83dc90 [ 1.927584] R13: ffffc5cf80be0240 R14: ffff8fbebd83dc80 R15: 000000000002f809 [ 1.928533] FS: 0000000000000000(0000) GS:ffff8fbebd800000(0000) knlGS:0000000000000000 [ 1.929647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.930431] CR2: 00007fed6f09b000 CR3: 0000000001e02000 CR4: 0000000000350ef0 [ 1.931333] Call Trace: [ 1.931727] <TASK> [ 1.932088] ? die+0x36/0x90 [ 1.932595] ? do_trap+0xed/0x110 [ 1.933047] ? __list_del_entry_valid_or_report+0xb4/0xd0 [ 1.933757] ? do_error_trap+0x6a/0xa0 [ 1.934390] ? __list_del_entry_valid_or_report+0xb4/0xd0 [ 1.935073] ? exc_invalid_op+0x51/0x80 [ 1.935627] ? __list_del_entry_valid_or_report+0xb4/0xd0 [ 1.936326] ? asm_exc_invalid_op+0x1a/0x20 [ 1.936904] ? __list_del_entry_valid_or_report+0xb4/0xd0 [ 1.937622] free_pcppages_bulk+0x130/0x280 [ 1.938151] free_unref_page_commit+0x21c/0x380 [ 1.938753] free_unref_page+0x472/0x4f0 [ 1.939343] __put_partials+0xee/0x130 [ 1.939921] ? rcu_do_batch+0x1f2/0x800 [ 1.940471] kmem_cache_free+0x2c3/0x370 [ 1.940990] rcu_do_batch+0x1f2/0x800 [ 1.941508] ? rcu_do_batch+0x180/0x800 [ 1.942031] rcu_core+0x182/0x340 [ 1.942500] handle_softirqs+0xe4/0x2f0 [ 1.943034] run_ksoftirqd+0x33/0x40 [ 1.943522] smpboot_thread_fn+0xdd/0x1d0 [ 1.944056] ? __pfx_smpboot_thread_fn+0x10/0x10 [ 1.944679] kthread+0xd0/0x100 [ 1.945126] ? __pfx_kthread+0x10/0x10 [ 1.945656] ret_from_fork+0x34/0x50 [ 1.946151] ? __pfx_kthread+0x10/0x10 [ 1.946680] ret_from_fork_asm+0x1a/0x30 [ 1.947269] </TASK> [ 1.947622] Modules linked in: overlay 9p ext4 crc32c_generic crc16 mbcache jbd2 hid_generic usbhid hid 9pnet_virtio 9pnet netfs sr_mod virtio_net cdrom virtio_blk net_failover atkbd failover libps2 vivaldi_fmap crc32c_intel ata_piix libata uhci_hcd scsi_mod ehci_hcd virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev scsi_common i8042 serio rtc_cmos dm_mod dax virtio_gpu virtio_dma_buf virtio_rng rng_core virtio_console virtio_balloon virtio virtio_ring [ 1.952291] ---[ end trace 0000000000000000 ]--- [ 1.952893] RIP: 0010:__list_del_entry_valid_or_report+0xb4/0xd0 [ 1.953678] Code: 0f 0b 48 89 fe 48 89 ca 48 c7 c7 38 52 41 9f e8 42 91 ac ff 90 0f 0b 48 89 fe 48 89 c2 48 c7 c7 70 52 41 9f e8 2d 91 ac ff 90 <0f> 0b 48 89 d1 48 c7 c7 c0 52 41 9f 48 89 f2 48 89 c6 e8 15 91 ac [ 1.955888] RSP: 0018:ffff96f800093c00 EFLAGS: 00010046 [ 1.956548] RAX: 000000000000006d RBX: 0000000000000001 RCX: 0000000000000000 [ 1.957436] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 1.958328] RBP: 0000000000000341 R08: 0000000000000000 R09: 0000000000000000 [ 1.959166] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8fbebd83dc90 [ 1.960044] R13: ffffc5cf80be0240 R14: ffff8fbebd83dc80 R15: 000000000002f809 [ 1.960905] FS: 0000000000000000(0000) GS:ffff8fbebd800000(0000) knlGS:0000000000000000 [ 1.961926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.962693] CR2: 00007fed6f09b000 CR3: 0000000001e02000 CR4: 0000000000350ef0 [ 1.963548] Kernel panic - not syncing: Fatal exception in interrupt [ 1.964417] Kernel Offset: 0x1ce00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)On a subsequent run to verify this, it failed earlier while reading $targetRoot/.../bash like this:
[ 1.871810] BUG: Bad page state in process cat pfn:2e74a [ 1.872481] page: refcount:1 mapcount:0 mapping:0000000000000000 index:0x1e5 pfn:0x2e74a [ 1.873499] flags: 0xffffc000000000(node=0|zone=1|lastcpupid=0x1ffff) [ 1.874260] raw: 00ffffc000000000 dead000000000100 dead000000000122 0000000000000000 [ 1.875250] raw: 00000000000001e5 0000000000000000 00000001ffffffff 0000000000000000 [ 1.876295] page dumped because: nonzero _refcount [ 1.876910] Modules linked in: overlay 9p ext4 crc32c_generic crc16 mbcache jbd2 hid_generic usbhid hid 9pnet_virtio 9pnet netfs sr_mod virtio_net cdrom virtio_blk net_failover atkbd failover libps2 vivaldi_fmap crc32c_intel ata_piix libata scsi_mod uhci_hcd ehci_hcd virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev scsi_common i8042 serio rtc_cmos dm_mod dax virtio_gpu virtio_dma_buf virtio_rng rng_core virtio_console virtio_balloon virtio virtio_ring [ 1.881465] CPU: 0 UID: 0 PID: 315 Comm: cat Not tainted 6.12.0-rc1 #1-NixOS [ 1.882326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 1.883684] Call Trace: [ 1.884103] <TASK> [ 1.884440] dump_stack_lvl+0x64/0x90 [ 1.884954] bad_page+0x70/0x110 [ 1.885468] __rmqueue_pcplist+0x209/0xd00 [ 1.886029] ? srso_return_thunk+0x5/0x5f [ 1.886572] ? pdu_read+0x36/0x50 [9pnet] [ 1.887177] get_page_from_freelist+0x2df/0x1910 [ 1.887788] ? srso_return_thunk+0x5/0x5f [ 1.888324] ? enqueue_entity+0xce/0x510 [ 1.888881] ? srso_return_thunk+0x5/0x5f [ 1.889415] ? pick_eevdf+0x76/0x1a0 [ 1.889970] ? update_curr+0x35/0x270 [ 1.890476] __alloc_pages_noprof+0x1a3/0x1150 [ 1.891158] ? srso_return_thunk+0x5/0x5f [ 1.891712] ? __mod_memcg_lruvec_state+0xa9/0x160 [ 1.892346] ? srso_return_thunk+0x5/0x5f [ 1.892919] ? __lruvec_stat_mod_folio+0x83/0xd0 [ 1.893521] alloc_pages_mpol_noprof+0x8f/0x1f0 [ 1.894148] folio_alloc_noprof+0x5b/0xb0 [ 1.894671] page_cache_ra_unbounded+0x11f/0x200 [ 1.895270] filemap_get_pages+0x538/0x6d0 [ 1.895813] ? srso_return_thunk+0x5/0x5f [ 1.896361] filemap_splice_read+0x136/0x320 [ 1.896948] backing_file_splice_read+0x52/0xa0 [ 1.897522] ovl_splice_read+0xd2/0xf0 [overlay] [ 1.898160] ? __pfx_ovl_file_accessed+0x10/0x10 [overlay] [ 1.898817] splice_direct_to_actor+0xb4/0x270 [ 1.899404] ? __pfx_direct_splice_actor+0x10/0x10 [ 1.900103] do_splice_direct+0x77/0xd0 [ 1.900627] ? __pfx_direct_file_splice_eof+0x10/0x10 [ 1.901308] do_sendfile+0x359/0x410 [ 1.901788] __x64_sys_sendfile64+0xb9/0xd0 [ 1.902370] do_syscall_64+0xb7/0x210 [ 1.902904] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1.903604] RIP: 0033:0x7fa9f3a7289e [ 1.904214] Code: 75 0e 00 f7 d8 64 89 02 b8 ff ff ff ff 31 d2 31 c9 31 ff 45 31 db c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 12 31 d2 31 c9 31 f6 31 ff 45 31 d2 45 31 db [ 1.906436] RSP: 002b:00007ffe6a82bde8 EFLAGS: 00000246 ORIG_RAX: 0000000000000028 [ 1.907400] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa9f3a7289e [ 1.908241] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000001 [ 1.909184] RBP: 00007ffe6a82be50 R08: 0000000000000000 R09: 0000000000000000 [ 1.910212] R10: 0000000001000000 R11: 0000000000000246 R12: 0000000000000001 [ 1.911117] R13: 0000000001000000 R14: 0000000000000001 R15: 0000000000000000 [ 1.911998] </TASK> [ 1.912376] Disabling lock debugging due to kernel taint [ 1.913479] list_del corruption. next->prev should be ffffc80e40b9d948, but was ffffc80e40b9d0c8. (next=ffffc80e40b9c7c8) [ 1.914823] ------------[ cut here ]------------ [ 1.915408] kernel BUG at lib/list_debug.c:65! [ 1.916050] Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 1.916785] CPU: 0 UID: 0 PID: 315 Comm: cat Tainted: G B 6.12.0-rc1 #1-NixOS [ 1.917877] Tainted: [B]=BAD_PAGE [ 1.918350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 1.919996] RIP: 0010:__list_del_entry_valid_or_report+0xcc/0xd0 [ 1.920903] Code: 89 fe 48 89 c2 48 c7 c7 70 52 41 ba e8 2d 91 ac ff 90 0f 0b 48 89 d1 48 c7 c7 c0 52 41 ba 48 89 f2 48 89 c6 e8 15 91 ac ff 90 <0f> 0b 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f [ 1.923423] RSP: 0018:ffff9ed880187748 EFLAGS: 00010246 [ 1.924210] RAX: 000000000000006d RBX: ffff94db3d83dc80 RCX: 0000000000000000 [ 1.925147] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 1.926051] RBP: ffffc80e40b9d940 R08: 0000000000000000 R09: 0000000000000000 [ 1.926940] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [ 1.927809] R13: ffff94db3d83dc80 R14: ffffc80e40b9d948 R15: ffff94db3ffd6180 [ 1.928695] FS: 00007fa9f396eb80(0000) GS:ffff94db3d800000(0000) knlGS:0000000000000000 [ 1.929728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.930540] CR2: 00000000004d1829 CR3: 0000000001dd2000 CR4: 0000000000350ef0 [ 1.931444] Call Trace: [ 1.931916] <TASK> [ 1.932357] ? die+0x36/0x90 [ 1.932831] ? do_trap+0xed/0x110 [ 1.933385] ? __list_del_entry_valid_or_report+0xcc/0xd0 [ 1.934073] ? do_error_trap+0x6a/0xa0 [ 1.934583] ? __list_del_entry_valid_or_report+0xcc/0xd0 [ 1.935242] ? exc_invalid_op+0x51/0x80 [ 1.935781] ? __list_del_entry_valid_or_report+0xcc/0xd0 [ 1.936484] ? asm_exc_invalid_op+0x1a/0x20 [ 1.937174] ? __list_del_entry_valid_or_report+0xcc/0xd0 [ 1.937926] ? __list_del_entry_valid_or_report+0xcb/0xd0 [ 1.938685] __rmqueue_pcplist+0xa5/0xd00 [ 1.939292] ? srso_return_thunk+0x5/0x5f [ 1.940004] ? __mod_memcg_lruvec_state+0xa9/0x160 [ 1.940758] ? srso_return_thunk+0x5/0x5f [ 1.941417] ? update_load_avg+0x7e/0x7f0 [ 1.942133] ? srso_return_thunk+0x5/0x5f [ 1.942838] ? srso_return_thunk+0x5/0x5f [ 1.943508] get_page_from_freelist+0x2df/0x1910 [ 1.944143] ? srso_return_thunk+0x5/0x5f [ 1.944696] ? check_preempt_wakeup_fair+0x1ee/0x240 [ 1.945335] ? srso_return_thunk+0x5/0x5f [ 1.945905] __alloc_pages_noprof+0x1a3/0x1150 [ 1.946489] ? __blk_flush_plug+0xf5/0x150 [ 1.947105] ? srso_return_thunk+0x5/0x5f [ 1.947629] ? __dquot_alloc_space+0x2a8/0x3a0 [ 1.948404] ? srso_return_thunk+0x5/0x5f [ 1.949116] ? __mod_memcg_lruvec_state+0xa9/0x160 [ 1.949888] alloc_pages_mpol_noprof+0x8f/0x1f0 [ 1.950514] folio_alloc_mpol_noprof+0x14/0x40 [ 1.951153] shmem_alloc_folio+0xa7/0xd0 [ 1.951692] ? shmem_recalc_inode+0x20/0x90 [ 1.952272] shmem_alloc_and_add_folio+0x109/0x490 [ 1.952940] ? filemap_get_entry+0x10f/0x1a0 [ 1.953570] ? srso_return_thunk+0x5/0x5f [ 1.954185] shmem_get_folio_gfp+0x248/0x610 [ 1.954791] shmem_write_begin+0x64/0x110 [ 1.955484] generic_perform_write+0xdf/0x2a0 [ 1.956239] shmem_file_write_iter+0x8a/0x90 [ 1.956882] iter_file_splice_write+0x33f/0x580 [ 1.957577] direct_splice_actor+0x54/0x140 [ 1.958178] splice_direct_to_actor+0xec/0x270 [ 1.958813] ? __pfx_direct_splice_actor+0x10/0x10 [ 1.959442] do_splice_direct+0x77/0xd0 [ 1.960018] ? __pfx_direct_file_splice_eof+0x10/0x10 [ 1.960726] do_sendfile+0x359/0x410 [ 1.961248] __x64_sys_sendfile64+0xb9/0xd0 [ 1.961905] do_syscall_64+0xb7/0x210 [ 1.962467] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1.963211] RIP: 0033:0x7fa9f3a7289e [ 1.963711] Code: 75 0e 00 f7 d8 64 89 02 b8 ff ff ff ff 31 d2 31 c9 31 ff 45 31 db c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 12 31 d2 31 c9 31 f6 31 ff 45 31 d2 45 31 db [ 1.965846] RSP: 002b:00007ffe6a82bde8 EFLAGS: 00000246 ORIG_RAX: 0000000000000028 [ 1.966788] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa9f3a7289e [ 1.967644] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000001 [ 1.968480] RBP: 00007ffe6a82be50 R08: 0000000000000000 R09: 0000000000000000 [ 1.969396] R10: 0000000001000000 R11: 0000000000000246 R12: 0000000000000001 [ 1.970315] R13: 0000000001000000 R14: 0000000000000001 R15: 0000000000000000 [ 1.971214] </TASK> [ 1.971572] Modules linked in: overlay 9p ext4 crc32c_generic crc16 mbcache jbd2 hid_generic usbhid hid 9pnet_virtio 9pnet netfs sr_mod virtio_net cdrom virtio_blk net_failover atkbd failover libps2 vivaldi_fmap crc32c_intel ata_piix libata scsi_mod uhci_hcd ehci_hcd virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev scsi_common i8042 serio rtc_cmos dm_mod dax virtio_gpu virtio_dma_buf virtio_rng rng_core virtio_console virtio_balloon virtio virtio_ring [ 1.976558] ---[ end trace 0000000000000000 ]--- [ 1.977219] RIP: 0010:__list_del_entry_valid_or_report+0xcc/0xd0 [ 1.978033] Code: 89 fe 48 89 c2 48 c7 c7 70 52 41 ba e8 2d 91 ac ff 90 0f 0b 48 89 d1 48 c7 c7 c0 52 41 ba 48 89 f2 48 89 c6 e8 15 91 ac ff 90 <0f> 0b 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f [ 1.980179] RSP: 0018:ffff9ed880187748 EFLAGS: 00010246 [ 1.980847] RAX: 000000000000006d RBX: ffff94db3d83dc80 RCX: 0000000000000000 [ 1.981705] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 1.982584] RBP: ffffc80e40b9d940 R08: 0000000000000000 R09: 0000000000000000 [ 1.983464] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [ 1.984358] R13: ffff94db3d83dc80 R14: ffffc80e40b9d948 R15: ffff94db3ffd6180 [ 1.987765] FS: 00007fa9f396eb80(0000) GS:ffff94db3d800000(0000) knlGS:0000000000000000 [ 1.988805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.989497] CR2: 00000000004d1829 CR3: 0000000001dd2000 CR4: 0000000000350ef0 [ 1.990418] note: cat[315] exited with preempt_count 2I bisected it back to
ee4cdf7ba857a894ad1650d6ab77669cbbfa329ewhich also seems to touch part of the 9p VFS code.Unfortunately the revert didn't apply cleanly on 6.12-rc1, so I couldn't meaningfully test whether a simple revert solves the problem.
The VMs get the Nix store mounted via 9p. In the store are basically all build artifacts including the stage-2 init script of the system that is booted into in the VM test.
The invocation basically looks like this:
qemu-system-x86_64 -cpu max \ -name machine \ -m 1024 \ -smp 1 \ -device virtio-rng-pci \ -net nic,netdev=user.0,model=virtio -netdev user,id=user.0,"$QEMU_NET_OPTS" \ -virtfs local,path=/nix/store,security_model=none,mount_tag=nix-store \ -virtfs local,path="${SHARED_DIR:-$TMPDIR/xchg}",security_model=none,mount_tag=shared \ -virtfs local,path="$TMPDIR"/xchg,security_model=none,mount_tag=xchg \ -drive cache=writeback,file="$NIX_DISK_IMAGE",id=drive1,if=none,index=1,werror=report -device virtio-blk-pci,bootindex=1,drive=drive1,serial=root \ -device virtio-net-pci,netdev=vlan1,mac=52:54:00:12:01:01 \ -netdev vde,id=vlan1,sock="$QEMU_VDE_SOCKET_1" \ -device virtio-keyboard \ -usb \ -device usb-tablet,bus=usb-bus.0 \ -kernel ${NIXPKGS_QEMU_KERNEL_machine:-/nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/kernel} \ -initrd /nix/store/qqalw1iq1wbgq3ndx0cvqn3bfypn56w2-initrd-linux-6.12-rc1/initrd \ -append "$(cat /nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/kernel-params) init=/nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/init regInfo=/nix/store/5izvfal6xm2rk51v0r1h2cxcng33paby-closure-info/registration console=ttyS0 $QEMU_KERNEL_PARAMS" \ $QEMU_OPTSIf you're using Nix, you can also reproduce this by running
nix-build nixos/tests/kernel-generic.nix -A
linux_testingon
5c19646b81db43dd7f4b6954f17d71a523009706from https://github.com/nixos/nixpkgs.To me, this seems like a regression in rc1.
Is there anything else I can do to help troubleshooting this?
With best regards
Maximilian
Christian Brauner
maintainerQuoted 39 lines · Antony Antony
Hi,
Quoted 11 lines · Sedat Dilek
Quoted 10 lines · Maximilian Bosch
Good evening,
thanks a lot for the quick reply!
Quoted 2 lines
A fix for it is already pending in the vfs.fixes branch and -next:
https://lore.kernel.org/all/cbaf141ba6c0e2e209717d02746584072844841a.1727722269.git.osandov@fb.com/I applied the patch on top of Linux 6.12-rc1 locally and I can confirm that this resolves the issue, thanks!
Maximilian, would you like to re-run the test a few times? I wonder if there is another intermittend bug related to the same commit.
Quoted 6 lines
Quoted 5 lines
With best regards
Maximilian
Quoted 8 lines
Thanks for testing.
For the records:
iov_iter: fix advancing slot initer_folioq_get_pages()
https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/commit/?h=vfs.fixes&id=0d24852bd71ec85ca0016b6d6fc997e6a3381552https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/log/?h=vfs.fixes
I’m still seeing a kernel oops after the fix in 6.12-rc3, but I’ve noticed that the issue is no longer 100% reproducible. Most of the time, the system crashes. Before this fix it was 100% reproducible.
When using the nix testing, I have to force the test to re-run.
Quoted 7 lines
result=$(readlink -f ./result); rm ./result && nix-store --delete $result
nix-build -v nixos/tests/kernel-generic.nix -A
linux_testingSo may be there is a new bug showing up after the fix. I have reported it.
I've pinged David again so hopefully we'll have a fix soon.
David Howells
maintainerQuoted 5 lines · Antony Antony
When using the nix testing, I have to force the test to re-run.
result=$(readlink -f ./result); rm ./result && nix-store --delete $result
nix-build -v nixos/tests/kernel-generic.nix -A
linux_testing
Is there a way to run this on Fedora?
David
Quoted 9 lines · David Howells
Quoted 5 lines · Antony Antony
When using the nix testing, I have to force the test to re-run.
result=$(readlink -f ./result); rm ./result && nix-store --delete $result
nix-build -v nixos/tests/kernel-generic.nix -A
linux_testingIs there a way to run this on Fedora?
Yes. You can run it on Fedora.
try these steps?
1. Install nix.
a: preferd way:
curl --proto '=https' --tlsv1.2 -sSf -L \
https://install.determinate.systems/nix | sh -s -- install
b: may be use dnf? I am advised dnf is a bad idea!
2. clone latest nixpkgs
git clone https://github.com/NixOS/nixpkgs
3. cd nixpkgs
nix-build -v nixos/tests/kernel-generic.nix -A linux_testing currently this will run 6.12-rc3.
when the test does not finish running, "Ctrl + C" to sop
when it succeds to re-run:
result=$(readlink -f ./result); rm ./result && nix-store --delete $result nix-build -v nixos/tests/kernel-generic.nix -A linux_testing
-antony
David Howells
maintainerCan you tell me what parameters you're mounting 9p with? Looking at the backtrace:
[ 32.390878] bad_page+0x70/0x110 [ 32.391056] free_unref_page+0x363/0x4f0 [ 32.391257] p9_release_pages+0x41/0x90 [9pnet] [ 32.391627] p9_virtio_zc_request+0x3d4/0x720 [9pnet_virtio] [ 32.391896] ? p9pdu_finalize+0x32/0xa0 [9pnet] [ 32.392153] p9_client_zc_rpc.constprop.0+0x102/0x310 [9pnet] [ 32.392447] ? kmem_cache_free+0x36/0x370 [ 32.392703] p9_client_read_once+0x1a6/0x310 [9pnet] [ 32.392992] p9_client_read+0x56/0x80 [9pnet] [ 32.393238] v9fs_issue_read+0x50/0xd0 [9p] [ 32.393467] netfs_read_to_pagecache+0x20c/0x480 [netfs] [ 32.393832] netfs_readahead+0x225/0x330 [netfs] [ 32.394154] read_pages+0x6a/0x250
it's using buffered I/O, but when I try and use 9p from qemu, it wants to use unbuffered/direct I/O.
David
Quoted 17 lines · David Howells
Can you tell me what parameters you're mounting 9p with? Looking at the backtrace:
[ 32.390878] bad_page+0x70/0x110 [ 32.391056] free_unref_page+0x363/0x4f0 [ 32.391257] p9_release_pages+0x41/0x90 [9pnet] [ 32.391627] p9_virtio_zc_request+0x3d4/0x720 [9pnet_virtio] [ 32.391896] ? p9pdu_finalize+0x32/0xa0 [9pnet] [ 32.392153] p9_client_zc_rpc.constprop.0+0x102/0x310 [9pnet] [ 32.392447] ? kmem_cache_free+0x36/0x370 [ 32.392703] p9_client_read_once+0x1a6/0x310 [9pnet] [ 32.392992] p9_client_read+0x56/0x80 [9pnet] [ 32.393238] v9fs_issue_read+0x50/0xd0 [9p] [ 32.393467] netfs_read_to_pagecache+0x20c/0x480 [netfs] [ 32.393832] netfs_readahead+0x225/0x330 [netfs] [ 32.394154] read_pages+0x6a/0x250
Quoted 2 lines
it's using buffered I/O, but when I try and use 9p from qemu, it wants to use unbuffered/direct I/O.
how can I check what it is using?
could you see from the command line?
/nix/store/s7zgdx5i9gs4abxjl94jcsw3xn4m861i-qemu-host-cpu-only-for-vm-tests-9.1.0/bin/qemu-kvm -cpu max -name machine -m 1024 -smp 1 -device virtio-rng-pci -net nic,netdev=user.0,model=virtio -netdev user,id=user.0, -virtfs local,path=/nix/store,security_model=none,mount_tag=nix-store -virtfs local,path=/build/shared-xchg,security_model=none,mount_tag=shared -virtfs local,path=/build/vm-state-machine/xchg,security_model=none,mount_tag=xchg -drive cache=writeback,file=/build/vm-state-machine/machine.qcow2,id=drive1,if=none,index=1,werror=report -device virtio-blk-pci,bootindex=1,drive=drive1,serial=root -device virtio-net-pci,netdev=vlan1,mac=52:54:00:12:01:01 -netdev vde,id=vlan1,sock=/build/vde1.ctl -device virtio-keyboard -usb -device usb-tablet,bus=usb-bus.0 -kernel /nix/store/i4xrqfq4jrk2chv6iqm2rgxdk8biynlr-nixos-system-machine-test/kernel -initrd /nix/store/i06b3wvd4c83x8slnd1f85dj7msjy398-initrd-linux-6.12-rc3/initrd -append console=ttyS0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm loglevel=7 net.ifnames=0 init=/nix/store/i4xrqfq4jrk2chv6iqm2rgxdk8biynlr-nixos-system-machine-test/init regInfo=/nix/store/5ygkzfld2zk20cy95iipmw2xxfvqalaz-closure-info/registration console=ttyS0 -qmp unix:/build/vm-state-machine/qmp,server=on,wait=off -monitor unix:/build/vm-state-machine/monitor -chardev socket,id=shell,path=/build/vm-state-machine/shell -device virtio-serial -device virtconsole,chardev=shell -device virtio-rng-pci -serial stdio -no-reboot -nographic
or inside a guest (running similar test an older kernel)
Quoted 1 line
Quoted 1 line
Quoted 1 line
print(alice.execute("mount | grep 9p")[1])
nix-store on /nix/.ro-store type 9p (rw,relatime,dirsync,loose,access=client,msize=16384,trans=virtio) shared on /tmp/shared type 9p (rw,relatime,sync,dirsync,access=client,msize=16384,trans=virtio) xchg on /tmp/xchg type 9p (rw,relatime,sync,dirsync,access=client,msize=16384,trans=virtio)
-antony
David Howells
maintainerI may have reproduced the bug (see attached), though the symptoms are slightly different. Hopefully, it's just the one bug.
David
---
page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x5626ca804 pfn:0x7948 flags: 0x2000000000000000(zone=1) raw: 2000000000000000 ffffea000024d3c8 ffffea00001e5248 0000000000000000 raw: 00000005626ca804 0000000000000000 00000000ffffffff 0000000000000000 page dumped because: VM_BUG_ON_FOLIO(((unsigned int) folio_ref_count(folio) + 127u <= 127u)) ------------[ cut here ]------------ kernel BUG at include/linux/mm.h:1444! Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI CPU: 3 UID: 0 PID: 303 Comm: md5sum Not tainted 6.12.0-rc2-ktest-00012-g57e4ac5316ef-dirty #8 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-2.fc40 04/01/2014 RIP: 0010:__iov_iter_get_pages_alloc+0x701/0x7e0 Code: 0f 0b 4d 85 f6 0f 85 f8 fc ff ff e9 21 fe ff ff 48 c7 c6 38 6b 40 82 e8 cd f4 aa ff 0f 0b 48 c7 c6 38 6b 40 82 e8 bf f4 aa ff <0f> 0b 4d 89 6a 18 49 89 52 08 4d 89 42 10 45 88 4a 20 e9 26 fb ff RSP: 0018:ffff88800804f5c0 EFLAGS: 00010286 RAX: 000000000000005c RBX: 0000000000001000 RCX: 0000000000000000 RDX: 0000000000000001 RSI: 0000000000000027 RDI: 00000000ffffffff RBP: ffff88800804f648 R08: ffff88807f1d8fa8 R09: 00000000fffc0000 R10: ffff88807dbd9000 R11: 0000000000000002 R12: 0000000000000001 R13: 0000000000001000 R14: 0000000000000000 R15: 0000000000001000 FS: 00007f0c42c29580(0000) GS:ffff88807d8c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000560cff5bb000 CR3: 00000000046a0002 CR4: 0000000000370eb0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: <TASK> ? __die_body.cold+0x19/0x2b ? __die+0x2a/0x40 ? die+0x2f/0x50 ? do_trap+0xb8/0x100 ? do_error_trap+0x6c/0x90 ? __iov_iter_get_pages_alloc+0x701/0x7e0 ? exc_invalid_op+0x52/0x70 ? __iov_iter_get_pages_alloc+0x701/0x7e0 ? asm_exc_invalid_op+0x1b/0x20 ? __iov_iter_get_pages_alloc+0x701/0x7e0 ? radix_tree_node_alloc.constprop.0+0xab/0xf0 iov_iter_get_pages_alloc2+0x20/0x50 p9_get_mapped_pages.part.0+0x77/0x260 ? find_held_lock+0x31/0x90 ? p9_tag_alloc+0x1c8/0x2f0 p9_virtio_zc_request+0x339/0x6f0 ? debug_smp_processor_id+0x17/0x20 ? debug_smp_processor_id+0x17/0x20 ? rcu_is_watching+0x11/0x50 ? p9_client_prepare_req+0x15f/0x190 p9_client_zc_rpc.constprop.0+0xe6/0x330 p9_client_read_once+0x145/0x2b0 p9_client_read+0x59/0x80 v9fs_issue_read+0x3d/0xa0 netfs_read_to_pagecache+0x27b/0x580 netfs_readahead+0x197/0x2f0 read_pages+0x4a/0x300 page_cache_ra_unbounded+0x197/0x250 page_cache_ra_order+0x2f7/0x400 ? __this_cpu_preempt_check+0x13/0x20 ? lock_release+0x168/0x290 page_cache_async_ra+0x1be/0x220 filemap_get_pages+0x2f3/0x870 filemap_read+0xdc/0x470 ? __this_cpu_preempt_check+0x13/0x20 ? lock_acquire+0xcc/0x1c0 ? preempt_count_add+0x4e/0xc0 ? down_read_interruptible+0xb3/0x1b0 netfs_buffered_read_iter+0x5c/0x90 netfs_file_read_iter+0x29/0x40 v9fs_file_read_iter+0x1b/0x30 vfs_read+0x22b/0x330 ksys_read+0x62/0xe0 __x64_sys_read+0x19/0x20 x64_sys_call+0x1b70/0x1d20 do_syscall_64+0x47/0x110 entry_SYSCALL_64_after_hwframe+0x76/0x7e
Quoted 17 lines
Can you tell me what parameters you're mounting 9p with? Looking at the backtrace:
[ 32.390878] bad_page+0x70/0x110 [ 32.391056] free_unref_page+0x363/0x4f0 [ 32.391257] p9_release_pages+0x41/0x90 [9pnet] [ 32.391627] p9_virtio_zc_request+0x3d4/0x720 [9pnet_virtio] [ 32.391896] ? p9pdu_finalize+0x32/0xa0 [9pnet] [ 32.392153] p9_client_zc_rpc.constprop.0+0x102/0x310 [9pnet] [ 32.392447] ? kmem_cache_free+0x36/0x370 [ 32.392703] p9_client_read_once+0x1a6/0x310 [9pnet] [ 32.392992] p9_client_read+0x56/0x80 [9pnet] [ 32.393238] v9fs_issue_read+0x50/0xd0 [9p] [ 32.393467] netfs_read_to_pagecache+0x20c/0x480 [netfs] [ 32.393832] netfs_readahead+0x225/0x330 [netfs] [ 32.394154] read_pages+0x6a/0x250
Quoted 2 lines
it's using buffered I/O, but when I try and use 9p from qemu, it wants to use unbuffered/direct I/O.
The NixOS integration tests mount 9p with 'cache=loose' which triggers the buffered I/O.
(I'm still seeing an issue in this area on 6.16-rc6, which remains also with 'only' 'cache=lookahead' - I'll do some more analysis before sharing more about that, though)
Kind regards,
Arnout
David Howells
maintainerPATCH 1/1 9p: Don't revert the I/O iterator after reading
In reply to Antony AntonyHi Antony,
I think I may have a fix already lurking on my netfs-writeback branch for the next merge window. Can you try the attached?
David
---
Don't revert the I/O iterator before returning from p9_client_read_once(). netfslib doesn't require the reversion and nor doed 9P directory reading.
Make p9_client_read() use a temporary iterator to call down into p9_client_read_once(), and advance that by the amount read.
--- net/9p/client.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-)
net/9p/client.c +3 -7
@@ -1519,13 +1519,15 @@ p9_client_read(struct p9_fid *fid, u64 offset, struct iov_iter *to, int *err) *err =0 ;while (iov_iter_count (to)) { +struct iov_iter tmp = *to;int count; - count =p9_client_read_once (fid, offset, to, err); + count =p9_client_read_once (fid, offset, &tmp, err);if (!count || *err)break ; offset += count; total += count; +iov_iter_advance (to, count); }return total; }
@@ -1567,16 +1569,12 @@ p9_client_read_once(struct p9_fid *fid, u64 offset, struct iov_iter *to, }if (IS_ERR (req)) { *err =PTR_ERR (req); -if (!non_zc) -iov_iter_revert (to, count -iov_iter_count (to));return 0 ; } *err =p9pdu_readf (&req->rc, clnt->proto_version,"D" , &received, &dataptr);if (*err) { -if (!non_zc) -iov_iter_revert (to, count -iov_iter_count (to));trace_9p_protocol_dump (clnt, &req->rc);p9_req_put (clnt, req);return 0 ;
@@ -1596,8 +1594,6 @@ p9_client_read_once(struct p9_fid *fid, u64 offset, struct iov_iter *to,p9_req_put (clnt, req);return n; } - }else { -iov_iter_revert (to, count - received -iov_iter_count (to)); }p9_req_put (clnt, req);return received;
Re: [PATCH] 9p: Don't revert the I/O iterator after reading
Hi David,
Quoted 4 lines · David Howells
Hi Antony,
I think I may have a fix already lurking on my netfs-writeback branch for the next merge window. Can you try the attached?
yes. The fix works. I rebooted a few times and no crash.
I am running test script in a loop over night.
thanks,
-antony
Quoted 72 lines
David
---
Don't revert the I/O iterator before returning fromp9_client_read_once(). netfslib doesn't require the reversion and nor doed 9P directory reading.Make
p9_client_read()use a temporary iterator to call down intop9_client_read_once(), and advance that by the amount read.Reported-by Antony Antony <[email protected]>Signed-off-by David Howells <[email protected]>cc Eric Van Hensbergen <[email protected]>cc Latchesar Ionkov <[email protected]>cc Dominique Martinet <[email protected]>cc Christian Schoenebeck <[email protected]>--- net/9p/client.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-)net/9p/client.c +3
-7@@ -1519,13 +1519,15 @@ p9_client_read(struct p9_fid *fid, u64 offset, struct iov_iter *to, int *err) *err =0 ;while (iov_iter_count (to)) { +struct iov_iter tmp = *to;int count; - count =p9_client_read_once (fid, offset, to, err); + count =p9_client_read_once (fid, offset, &tmp, err);if (!count || *err)break ; offset += count; total += count; +iov_iter_advance (to, count); }return total; }@@ -1567,16 +1569,12 @@ p9_client_read_once(struct p9_fid *fid, u64 offset, struct iov_iter *to, }if (IS_ERR (req)) { *err =PTR_ERR (req); -if (!non_zc) -iov_iter_revert (to, count -iov_iter_count (to));return 0 ; } *err =p9pdu_readf (&req->rc, clnt->proto_version,"D" , &received, &dataptr);if (*err) { -if (!non_zc) -iov_iter_revert (to, count -iov_iter_count (to));trace_9p_protocol_dump (clnt, &req->rc);p9_req_put (clnt, req);return 0 ;@@ -1596,8 +1594,6 @@ p9_client_read_once(struct p9_fid *fid, u64 offset, struct iov_iter *to,p9_req_put (clnt, req);return n; } - }else { -iov_iter_revert (to, count - received -iov_iter_count (to)); }p9_req_put (clnt, req);return received;
Re: [PATCH] 9p: Don't revert the I/O iterator after reading
In reply to David HowellsQuoted 6 lines · David Howells
I think I may have a fix already lurking on my netfs-writeback branch for the next merge window. Can you try the attached?
David
Applied to the vfs.fixes branch of the vfs/vfs.git tree. Patches in the vfs.fixes branch should appear in linux-next soon.
Please report any outstanding bugs that were missed during review in a new review to the original patch series allowing us to drop it.
It's encouraged to provide Acked-bys and Reviewed-bys even though the patch has now been applied. If possible patch trailers will be updated.
Note that commit hashes shown below are subject to change due to rebase, trailer updates or similar. If in doubt, please check the listed branch.
tree: https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git branch: vfs.fixes
[1/1] 9p: Don't revert the I/O iterator after reading
https://git.kernel.org/vfs/vfs/c/09c729da3283
David Howells
maintainerHi Antony,
I think the attached should fix it properly rather than working around it as the previous patch did. If you could give it a whirl?
Thanks,
David
---
commit 68dddbfdf45e8f176cc8556a3db69af24dfb8519
Author: David Howells <[email protected]> Date: Wed Oct 23 10:24:12 2024 +0100
iov_iter: Fix iov_iter_get_pages*() for folio_queue
p9_get_mapped_pages() uses iov_iter_get_pages_alloc2() to extract pages from an iterator when performing a zero-copy request and under some circumstances, this crashes with odd page errors[1], for example, I see:
page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0xbcf0
flags: 0x2000000000000000(zone=1)
...
page dumped because: VM_BUG_ON_FOLIO(((unsigned int) folio_ref_count(folio) + 127u <= 127u)) ------------[ cut here ]------------
kernel BUG at include/linux/mm.h:1444!
This is because, unlike in iov_iter_extract_folioq_pages(), the iter_folioq_get_pages() helper function doesn't skip the current folio when iov_offset points to the end of it, but rather extracts the next page beyond the end of the folio and adds it to the list. Reading will then clobber the contents of this page, leading to system corruption, and if the page is not in use, put_page() may try to clean up the unused page.
This can be worked around by copying the iterator before each extraction[2] and using iov_iter_advance() on the original as the advance function steps over the page we're at the end of.
Fix this by skipping the page extraction if we're at the end of the folio.
This was reproduced in the ktest environment[3] by forcing 9p to use the fscache caching mode and then reading a file through 9p.
lib/iov_iter.c +12 -9
@@ -1021,15 +1021,18 @@ static ssize_t iter_folioq_get_pages(struct iov_iter *iter,size_t offset = iov_offset, fsize =folioq_folio_size (folioq, slot);size_t part =PAGE_SIZE - offset %PAGE_SIZE ; - part =umin (part,umin (maxsize - extracted, fsize - offset)); - count -= part; - iov_offset += part; - extracted += part; - - *pages =folio_page (folio, offset /PAGE_SIZE ); -get_page (*pages); - pages++; - maxpages--; +if (offset < fsize) { + part =umin (part,umin (maxsize - extracted, fsize - offset)); + count -= part; + iov_offset += part; + extracted += part; + + *pages =folio_page (folio, offset /PAGE_SIZE ); +get_page (*pages); + pages++; + maxpages--; + } +if (maxpages ==0 || extracted >= maxsize)break ;
Antony Antony
Tested-byQuoted 4 lines · David Howells
Hi Antony,
I think the attached should fix it properly rather than working around it as the previous patch did. If you could give it a whirl?
Yes this also fix the crash.
thanks,
-antony
Hi everyone,
Quoted 9 lines · Antony Antony
Quoted 4 lines · David Howells
Hi Antony,
I think the attached should fix it properly rather than working around it as the previous patch did. If you could give it a whirl?
Yes this also fix the crash.
Tested-by Antony Antony <[email protected]>
I cannot confirm this fixes the crash for me. My reproducer is slightly more complicated than Max's original one, albeit, still on NixOS and probably uses 9p more intensively than the automated NixOS testings workload.
Here is how to reproduce it:
$ git clone https://gerrit.lix.systems/lix $ cd lix $ git fetch https://gerrit.lix.systems/lix refs/changes/29/3329/8 && git checkout FETCH_HEAD $ nix-build -A hydraJobs.tests.local-releng
I suspect the reason for why Antony considers the crash to be fixed is that the workload used to test it requires a significant amount of chance and retries to trigger the bug.
On my end, you can see our CI showing the symptoms:
https://buildkite.com/organizations/lix-project/pipelines/lix/builds/2357/jobs/019761e7-784e-4790-8c1b-f609270d9d19/log.
We retried probably hundreds of times and saw different corruption patterns, Python getting confused, ld.so getting confused, systemd sometimes too. Python had a much higher chance of crashing in many of our tests. We reproduced it over aarch64-linux (Ampere Altra Q80-30) but also Intel and AMD CPUs (~5 different systems).
As soon as we reverted to Linux 6.6 series, the bug went away.
We bisected but we started to have weirder problems, this is because we encountered the original regression mentioned in October 2024 and for a certain range of commits, we were unable to bisect anything further.
So I switched my bisection strategy to understand when the bug was fixed, this lead me on the commit e65a0dc1cabe71b91ef5603e5814359451b74ca7 which is the proper fix mentioned here and on this discussion.
Reverting this on the top of 6.12 cause indeed a massive amount of traces, see this gist [1] for examples.
Applying the "workaround patch" aka "[PATCH] 9p: Don't revert the I/O iterator after reading" after reverting e65a0dc1cabe makes the problem go away after 5 tries (5 tries were sufficient to trigger with the proper fix).
If this can be helpful, the nature of the test above is to copy a significant amount of assets to an S3 implementation (Garage) running inside of the VM. Many of these assets comes from the Nix store which sits over 9p.
Anyhow, I see three patterns:
- Kernel panic when starting the /init, this is the crash Max reported back in October 2024 and the one we started to encounter while bisecting this problem in the range between v6.11 and v6.12.
- systemd crashing very quickly,
this is what we see when reverting e65a0dc1cabe71b91ef5603e5814359451b74ca7 on the top of v6.12 *OR* when we are around v6.12rc5.
- what the CI above shows which are userspace programs crashing after some serious I/O exercising has been done, which happens on the top of v6.12, v6.14, v6.15 (incl. stable kernels).
If you need me to test things, please let me know.
[1]: https://gist.dgnum.eu/raito/3d1fa61ebaf642218342ffe644fb6efd
Signature · 1 line
Ryan Lahfa
Hi,
we’re experiencing the same issue with a number of NixOS tests that are heavy in operations copying from the v9fs mounted nix store.
Quoted 31 lines · Ryan Lahfa
Hi everyone,
Quoted 9 lines · Antony Antony
Quoted 4 lines · David Howells
Hi Antony,
I think the attached should fix it properly rather than working around it as the previous patch did. If you could give it a whirl?
Yes this also fix the crash.
Tested-by Antony Antony <[email protected]>I cannot confirm this fixes the crash for me. My reproducer is slightly more complicated than Max's original one, albeit, still on NixOS and probably uses 9p more intensively than the automated NixOS testings workload.
Here is how to reproduce it:
$ git clone https://gerrit.lix.systems/lix $ cd lix $ git fetch https://gerrit.lix.systems/lix refs/changes/29/3329/8 && git checkout FETCH_HEAD $ nix-build -A hydraJobs.tests.local-relengI suspect the reason for why Antony considers the crash to be fixed is that the workload used to test it requires a significant amount of chance and retries to trigger the bug.
Quoted 8 lines
On my end, you can see our CI showing the symptoms:
https://buildkite.com/organizations/lix-project/pipelines/lix/builds/2357/jobs/019761e7-784e-4790-8c1b-f609270d9d19/log.We retried probably hundreds of times and saw different corruption patterns, Python getting confused, ld.so getting confused, systemd sometimes too. Python had a much higher chance of crashing in many of our tests. We reproduced it over aarch64-linux (Ampere Altra Q80-30) but also Intel and AMD CPUs (~5 different systems).
Yeah. We’re on AMD CPUs and it wasn’t hardware-bound.
The errors we saw where:
- malloc(): unaligned tcache chunk detected
- segfaulting java processes
- misbehaving filesystems (errors about internal structures in ext4, incorrect file content in xfs)
- crashing kernels when dealing with the outfall of those errors
Quoted 1 line
As soon as we reverted to Linux 6.6 series, the bug went away.
Same here, the otherway around: we came from 6.6.94 and updated to 6.12.34 and immediately saw a number of tests failing, all of which were heavy in copying data from v9fs to the root filesystem in the VM.
Quoted 3 lines
We bisected but we started to have weirder problems, this is because we encountered the original regression mentioned in October 2024 and for a certain range of commits, we were unable to bisect anything further.
I had already found the issue from last October when started bisecting, I later got in touch with Ryan who recognized that we were chasing the same issue. I stopped bisecting at that point - the bisect was already homing in around the time of the changes in last October.
Quoted 7 lines
So I switched my bisection strategy to understand when the bug was fixed, this lead me on the commit
e65a0dc1cabe71b91ef5603e5814359451b74ca7which is the proper fix mentioned here and on this discussion.Reverting this on the top of 6.12 cause indeed a massive amount of traces, see this gist [1] for examples.
Yeah. During bisect I noticed it flapping around with the original October issues crashing immediately during boot.
Quoted 4 lines
Applying the "workaround patch" aka "[PATCH] 9p: Don't revert the I/O iterator after reading" after reverting
e65a0dc1cabemakes the problem go away after 5 tries (5 tries were sufficient to trigger with the proper fix).
Yup, I applied the revert and workaround patch on top of 6.12.34 and the reliably broken test became reliably green again.
Our test can be reproduced, too:
$ git clone https://github.com/flyingcircusio/fc-nixos.git $ cd fc-nixos $ eval $(./dev-setup) $ nix-build tests/matomo.nix
The test will fail with ext4 complaining something like this:
machine # [ 42.596728] vn2haz1283lxz6iy0rai850a7jlgxbja-matomo-setup-update-pre[1233]: Copied files, updating package link in /var/lib/matomo/current-package.
machine # [ 42.788956] EXT4-fs error (device vda): htree_dirblock_to_tree:1109: inode #13138: block 5883: comm setfacl: bad entry in directory: rec_len % 4 != 0 - offset=0, inode=606087968, rec_len=31074, size=4096 fake=0
machine # [ 42.958590] EXT4-fs error (device vda): htree_dirblock_to_tree:1109: inode #13138: block 5883: comm chown: bad entry in directory: rec_len % 4 != 0 - offset=0, inode=606087968, rec_len=31074, size=4096 fake=0
machine # [ 43.068003] EXT4-fs error (device vda): htree_dirblock_to_tree:1109: inode #13138: block 5883: comm chmod: bad entry in directory: rec_len % 4 != 0 - offset=0, inode=606087968, rec_len=31074, size=4096 fake=0
machine # [ 43.004098] vn2haz1283lxz6iy0rai850a7jlgxbja-matomo-setup-update-pre[1233]: Giving matomo read+write access to /var/lib/matomo/share/matomo.js, /var/lib/matomo/share/piwik.js, /var/lib/matomo/share/config, /var/lib/matomo/share/misc/user, /var/lib/matomo/share/js, /var/lib/matomo/share/tmp, /var/lib/matomo/share/misc machine # [ 43.201319] EXT4-fs error (device vda): htree_dirblock_to_tree:1109: inode #13138: block 5883: comm setfacl: bad entry in directory: rec_len % 4 != 0 - offset=0, inode=606087968, rec_len=31074, size=4096 fake=0
I’m also available for testing and further diagnosis.
Christian
Signature · 4 lines
Christian Theune · [email protected] · +49 345 219401 0 Flying Circus Internet Operations GmbH · https://flyingcircus.io Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick
Dominique Martinet
maintainerHi all,
sorry for the slow reply; I wasn't in Cc of most of the mails back in October so this is a pain to navigate... Let me recap a bit:
- stuff started failing in 6.12-rc1
- David first posted "9p: Don't revert the I/O iterator after reading"[1], which fixed the bug, but then found a "better" fix as "iov_iter: Fix iov_iter_get_pages*() for folio_queue" [2] which was merged instead (so the first patch was not merged)
But it turns out the second patch is not enough (or causes another issue?), and the reverting it + applying first one works, is that correct?
What happens if you keep [2] and just apply [1], does that still bug?
(I've tried reading through the thread now and I don't even see what was the "bad" patch in the first place, although I assume it's ee4cdf7ba857 ("netfs: Speed up buffered reading") -- was that confirmed?)
David, as you worked on this at the time it'd be great if you could have another look, I have no idea what made you try [1] in the first place but unless you think 9p is doing something wrong like double-reverting on error or something like that I'd like to understand a bit more what happens... Although given 6.12 is getting used more now it could make sense to just apply [1] first until we understand, and have a proper fix come second -- if someone can confirm we don't need to revert [2].
[1] https://lore.kernel.org/all/[email protected]/T/#mc97a248b0f673dff6dc8613b508ca4fd45c4fefe
[2] https://lore.kernel.org/all/[email protected]/T/#m89597a1144806db4ae89992953031cdffa0b0bf9
Thanks,
Signature · 1 line
Dominique
Hi,
Quoted 7 lines · Dominique Martinet
Hi all,
sorry for the slow reply; I wasn't in Cc of most of the mails back in October so this is a pain to navigate... Let me recap a bit:
- stuff started failing in 6.12-rc1
yes, to my knowledge and interpretation of this thread.
Quoted 9 lines
- David first posted "9p: Don't revert the I/O iterator after reading"[1], which fixed the bug, but then found a "better" fix as "
iov_iter: Fix iov_iter_get_pages*() forfolio_queue" [2] which was merged instead (so the first patch was not merged)But it turns out the second patch is not enough (or causes another issue?), and the reverting it + applying first one works, is that correct?
What happens if you keep [2] and just apply [1], does that still bug?
I tried that and the test that so far under all the variations reliably crashed (or not) is not crashing in this case.
Quoted 3 lines
(I've tried reading through the thread now and I don't even see what was the "bad" patch in the first place, although I assume it's
ee4cdf7ba857("netfs: Speed up buffered reading") -- was that confirmed?)
I was late to the party, to, so I’ll defer to the others.
Quoted 7 lines
David, as you worked on this at the time it'd be great if you could have another look, I have no idea what made you try [1] in the first place but unless you think 9p is doing something wrong like double-reverting on error or something like that I'd like to understand a bit more what happens... Although given 6.12 is getting used more now it could make sense to just apply [1] first until we understand, and have a proper fix come second -- if someone can confirm we don't need to revert [2].
I guess I confirmed this. However, I’m just barely better than a monkey here so I can’t tell whether this makes sense from the internal logic of things.
To repeat, for safety: my test case worked with the situation you described and suggested:
[1] applied on top of 6.12.34 and *not* having [2] reverted.
Christian
Signature · 4 lines
Christian Theune · [email protected] · +49 345 219401 0 Flying Circus Internet Operations GmbH · https://flyingcircus.io Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick
David Howells
maintainerQuoted 6 lines · Ryan Lahfa
Here is how to reproduce it:
$ git clone https://gerrit.lix.systems/lix $ cd lix $ git fetch https://gerrit.lix.systems/lix refs/changes/29/3329/8 && git checkout FETCH_HEAD $ nix-build -A hydraJobs.tests.local-releng
How do I build and run this on Fedora is the problem :-/
Quoted 1 line
[1]: https://gist.dgnum.eu/raito/3d1fa61ebaf642218342ffe644fb6efd
Looking at this, it looks very much like a page may have been double-freed.
Just to check, what are you using 9p for? Containers? And which transport is being used, the virtio one?
David
Hi David,
Quoted 10 lines · David Howells
Quoted 6 lines · Ryan Lahfa
Here is how to reproduce it:
$ git clone https://gerrit.lix.systems/lix $ cd lix $ git fetch https://gerrit.lix.systems/lix refs/changes/29/3329/8 && git checkout FETCH_HEAD $ nix-build -A hydraJobs.tests.local-relengHow do I build and run this on Fedora is the problem :-/
This may introduce another layer but you could use a Docker container (Lix has http://ghcr.io/lix-project/lix) and run these instructions inside that context.
Alternatives are the following:
- static binary for Nix, I can build one for you and make it available.
- the Lix installer, https://lix.systems/install/ (curl | sh but it does prompt you for any step and tell you what it does, it should also be very easy to uninstall!).
- Debian has Nix packaged: https://packages.debian.org/sid/nix-bin (not Lix, but doesn't matter for this reproducer).
- Can install a remote VM for you with Fedora with one of the previous option and give you root@ over there.
Let me know how I can help.
Quoted 6 lines
Quoted 1 line
[1]: https://gist.dgnum.eu/raito/3d1fa61ebaf642218342ffe644fb6efd
Looking at this, it looks very much like a page may have been double-freed.
Just to check, what are you using 9p for? Containers? And which transport is being used, the virtio one?
9p is used in QEMU in this context.
NixOS has a framework of end to end testing à la OpenQA from OpenSUSE that makes use of 9pfs to mount the host Nix store inside the guest VM to avoid copying back'n'forth things that are not under test.
Yep, transport is virtio.
Kind regards,
Signature · 1 line
Ryan Lahfa
Quoted 7 lines · Ryan Lahfa
Quoted 6 lines · Antony Antony
Quoted 4 lines · David Howells
Hi Antony,
I think the attached should fix it properly rather than working around it as the previous patch did. If you could give it a whirl?
Quoted 8 lines
Quoted 3 lines
Yes this also fix the crash.
Tested-by Antony Antony <[email protected]>I cannot confirm this fixes the crash for me. My reproducer is slightly more complicated than Max's original one, albeit, still on NixOS and probably uses 9p more intensively than the automated NixOS testings workload.
I'm seeing a problem in the same area - the symptom is slightly different, but the location seems very similar. I'm also running a NixOS image. Mounting a 9p filesystem in qemu with cache=readahead, reading a 12943-byte file, in the guest I do see a 12943-byte file, but only the first 12288 bytes are populated: the rest are zero. This also reproduces (most but not all of the time) on 6.16-rc7, but not on all host machines I've tried.
After applying a simplified version of [1] (i.e. [2]), the problem does not reproduce anymore. It seems something in p9_client_read_once somehow leaves the iov_iter in an unhealthy state. It would be good to understand exactly what, but I haven't been able to figure that out yet.
I have a smallish nix-based reproducer at [3], and a more involved setup with a lot of logging enabled and a convenient way to attach gdb at [4]. You start the VM and then 'cat /repro/default.json' manually, and see if it looks 'truncated'.
Interestingly, the file is read in two p9 read calls: one of 12288 bytes and one of 655 bytes. The first read is a zero-copy one, the second is not zero-copy (because it is smaller than 1024). I've also tried with a slightly larger version of the file, that is read as 2 zero-copy reads, and I have not been able to reproduce the problem with that. From my (admittedly limited) understanding the non-zerocopy code path looks fine, though.
I hope this is helpful - I'd be happy to keep looking into this further, but any help pointing me in the right direction would be much appreciated :)
Kind regards,
Arnout
[1] https://lore.kernel.org/all/[email protected]/T/#mc97a248b0f673dff6dc8613b508ca4fd45c4fefe
[2] https://codeberg.org/raboof/nextcloud-onlyoffice-test-vm/src/branch/reproducer-with-debugging/kernel-use-copied-iov_iter.patch
[3] https://codeberg.org/raboof/nextcloud-onlyoffice-test-vm/src/branch/small-reproducer
[4] https://codeberg.org/raboof/nextcloud-onlyoffice-test-vm/src/branch/reproducer-with-debugging
[email protected]
maintainerArnout Engelen wrote on Sun, Aug 10, 2025 at 07:57:11PM +0200:
Quoted 4 lines
I have a smallish nix-based reproducer at [3], and a more involved setup with a lot of logging enabled and a convenient way to attach gdb at [4]. You start the VM and then 'cat /repro/default.json' manually, and see if it looks 'truncated'.
Thank you!!! I was able to reproduce with this!
(well, nix -L build .#nixosConfigurations.default.config.system.build.vm to build the VM as this machine isn't running nixos and doesn't have nixos-rebuild...)
Quoted 3 lines
Interestingly, the file is read in two p9 read calls: one of 12288 bytes and one of 655 bytes. The first read is a zero-copy one, the second is not zero-copy (because it is smaller than 1024).
Yes, your msize is set to 16k but with the 9p overhead the largest, 4k-aligned read that can be done is 12k, so that's coherent. (Changing the msize to 32k so it's read in a single zero-copy read, obviously makes this particular error go away, but it's a huge hint)
Removing readahead also makes the problem go away, which is also surprising because from looking at traces it's only calling into p9_client_read() once (which forks the two p9_client_read_once, one with zc and the other without), so readahead shouldn't matter at all but it obviously does...
Also I haven't been able to reproduce it with a kernel I built myself/my environment, but it reproduces reliably 99% of the times in the nixos VM, so we're missing a last piece for a "simple" (non-nix) reproducer, but I think it's good enough for me to dig into this;
I'll try to find time to check in details this afternoon... Basically "just" have to follow where the data is written and why it doesn't end up in the iov and fix that, but I'll need to reproduce on a kernel I built first to be able to validate the fix.
Anyway this is a huge leap forward (hopeful it's the same problem and we don't have two similar issues lurking here...), we can't thank you enough.
I'll report back ASAP.
Signature · 1 line
Dominique Martinet | Asmadeus
Dominique Martinet
maintainer[email protected] wrote on Mon, Aug 11, 2025 at 09:57:51AM +0900:
Quoted 4 lines
Also I haven't been able to reproduce it with a kernel I built myself/my environment, but it reproduces reliably 99% of the times in the nixos VM, so we're missing a last piece for a "simple" (non-nix) reproducer, but I think it's good enough for me to dig into this;
(okay, I got this to work wedging my kernel into the nixos initrd; this requires the iov to be built off non-contiguous pages and so having systemd thrash around was apparently a required step...)
So that wasn't a 9p bug, I'm not sure if I should be happy or not?
I've sent "proper-ish" patches at [1] which most concerned people should be in Cc; I'm fairly confident this will make the bug go away but any testing is appreciated, please reply to the patches with a Tested-by if you have time.
[1] https://lkml.kernel.org/r/[email protected]
Thank you all again, and sorry I haven't had the time to look further into this without a clean reproducer, this really shouldn't have taken that long...
Signature · 1 line
Dominique Martinet | Asmadeus
Quoted 11 lines · On Mon, Aug 11, 2025, at 02:57, [email protected] wrote:
Arnout Engelen wrote on Sun, Aug 10, 2025 at 07:57:11PM +0200:
Quoted 4 lines
I have a smallish nix-based reproducer at [3], and a more involved setup with a lot of logging enabled and a convenient way to attach gdb at [4]. You start the VM and then 'cat /repro/default.json' manually, and see if it looks 'truncated'.
Thank you!!! I was able to reproduce with this!
Anyway this is a huge leap forward (hopeful it's the same problem and we don't have two similar issues lurking here...), we can't thank you enough.
Great - that means a lot ;)
Quoted 1 line · Dominique Martinet
So that wasn't a 9p bug, I'm not sure if I should be happy or not?
:D
Quoted 6 lines
I've sent "proper-ish" patches at [1] which most concerned people should be in Cc; I'm fairly confident this will make the bug go away but any testing is appreciated, please reply to the patches with a Tested-by if you have time.
Awesome!
Kind regards,
Signature · 3 lines
Arnout Engelen Engelen Open Source https://engelen.eu
Hi,
Quoted 9 lines · Antony Antony
Hi,
Quoted 6 lines · Sedat Dilek
Quoted 5 lines · Maximilian Bosch
Good evening,
thanks a lot for the quick reply!
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
A fix for it is already pending in the vfs.fixes branch and -next:
https://lore.kernel.org/all/cbaf141ba6c0e2e209717d02746584072844841a.1727722269.git.osandov@fb.com/I applied the patch on top of Linux 6.12-rc1 locally and I can confirm that this resolves the issue, thanks!
Maximilian, would you like to re-run the test a few times? I wonder if there is another intermittend bug related to the same commit.
Apologies for getting back to you that late, have had a few busy days, but it seems you figured out how to test it anyways?
For completeness sake, I ran the linux_testing test-case from kernel-generic.nix in a loop 40 times each on Linux 6.12-rc4 and every single run booted and succeeded on two different machines.
So I'm not sure if the test-case is sufficient to trigger it.
Let me know if there's anything else I can do to help.
Quoted 1 line
result=$(readlink -f ./result); rm ./result && nix-store --delete $result
Just as a well-meant hint, it's possible to rebuild existing things with nix-build --check (which also gives feedback on if the new build result has changed).
Quoted 24 lines
Quoted 7 lines
Quoted 6 lines
With best regards
Maximilian
Quoted 8 lines
Thanks for testing.
For the records:
iov_iter: fix advancing slot initer_folioq_get_pages()
https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/commit/?h=vfs.fixes&id=0d24852bd71ec85ca0016b6d6fc997e6a3381552https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/log/?h=vfs.fixes
I’m still seeing a kernel oops after the fix in 6.12-rc3, but I’ve noticed that the issue is no longer 100% reproducible. Most of the time, the system crashes. Before this fix it was 100% reproducible.
When using the nix testing, I have to force the test to re-run.
result=$(readlink -f ./result); rm ./result && nix-store --delete $result
Quoted 344 lines
nix-build -v nixos/tests/kernel-generic.nix -A
linux_testingSo may be there is a new bug showing up after the fix. I have reported it.
https://lore.kernel.org/regressions/[email protected]/T/#u
-antony
Quoted 336 lines
Quoted 336 lines · Linux regression tracking (Thorsten Leemhuis)
Quoted 331 lines
Hi, Thorsten here, the Linux kernel's regression tracker. Top-posting for once, to make this easily accessible to everyone.
Thx for the report. Not my area of expertise (so everyone: corrent me if I'm wrong), but I suspect your problem might be a duplicate of the following report, which was bisected to the same commit from dhowells (
ee4cdf7ba857a8("netfs: Speed up buffered reading") [v6.12-rc1]):
https://lore.kernel.org/all/[email protected]/A fix for it is already pending in the vfs.fixes branch and -next:
https://lore.kernel.org/all/cbaf141ba6c0e2e209717d02746584072844841a.1727722269.git.osandov@fb.com/Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.Quoted 312 lines · Maximilian Bosch
Starting with Linux 6.12-rc1 the automatic VM tests of NixOS don't boot anymore and fail like this:
Quoted 1 line
mounting nix-store on /nix/.ro-store...
[ 1.604781] 9p: Installing v9fs 9p2000 file system support mounting tmpfs on /nix/.rw-store... mounting overlay on /nix/store... mounting shared on /tmp/shared... mounting xchg on /tmp/xchg... switch_root: can't execute '/nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/init': Exec format error [ 1.734997] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100 [ 1.736002] CPU: 0 UID: 0 PID: 1 Comm: switch_root Not tainted 6.12.0-rc1 #1-NixOS [ 1.736965] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 1.738309] Call Trace: [ 1.738698] <TASK> [ 1.739034] panic+0x324/0x340 [ 1.739458] do_exit+0x92e/0xa90 [ 1.739919] ? count_memcg_events.constprop.0+0x1a/0x40 [ 1.740568] ? srso_return_thunk+0x5/0x5f [ 1.741095] ? handle_mm_fault+0xb0/0x2e0 [ 1.741709] do_group_exit+0x30/0x80 [ 1.742229] __x64_sys_exit_group+0x18/0x20 [ 1.742800] x64_sys_call+0x17f3/0x1800 [ 1.743326] do_syscall_64+0xb7/0x210 [ 1.743895] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1.744530] RIP: 0033:0x7f8e1a7b9d1d [ 1.745061] Code: 45 31 c0 45 31 d2 45 31 db c3 0f 1f 00 f3 0f 1e fa 48 8b 35 e5 e0 10 00 ba e7 00 00 00 eb 07 66 0f 1f 44 00 00 f4 89 d0 0f 05 <48> 3d 00 f0 ff ff 76 f3 f7 d8 64 89 06 eb ec 0f 1f 40 00 f3 0f 1e [ 1.747263] RSP: 002b:00007ffcb56d63b8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7 [ 1.748250] RAX: ffffffffffffffda RBX: 00007f8e1a8c9fa8 RCX: 00007f8e1a7b9d1d [ 1.749187] RDX: 00000000000000e7 RSI: ffffffffffffff88 RDI: 0000000000000001 [ 1.750050] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000 [ 1.750891] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 1.751706] R13: 0000000000000001 R14: 00007f8e1a8c8680 R15: 00007f8e1a8c9fc0 [ 1.752583] </TASK> [ 1.753010] Kernel Offset: 0xb800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)The failing script here is the initrd's /init when it tries to perform a
switch_rootto/sysroot:exec env -i $(type -P switch_root) "$targetRoot" "$stage2Init"Said "$stage2Init" file consistently gets a different hash when doing
sha256sumon it in the initrd script, but looks & behaves correct on the host. I reproduced the test failures on 4 different build machines and two architectures (x86_64-linux, aarch64-linux) now.The "$stage2Init" script is a shell-script itself. When trying to start the interpreter from its shebang inside the initrd (via
$targetRoot/nix/store/...-bash-5.2p32/bin/bash) and do theswitch_rootI get a different error:+ exec env -i /nix/store/akm69s5sngxyvqrzys326dss9rsrvbpy-extra-utils/bin/switch_root /mnt-root /nix/store/k3pm4iv44y7x7p74kky6cwxiswmr6kpi-nixos-system-machine-test/init [ 1.912859] list_del corruption. prev->next should be ffffc5cf80be0248, but was ffffc5cf80bd9208. (prev=ffffc5cf80bb4d48) [ 1.914237] ------------[ cut here ]------------ [ 1.915059] kernel BUG at lib/list_debug.c:62! [ 1.915854] Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 1.916739] CPU: 0 UID: 0 PID: 17 Comm: ksoftirqd/0 Not tainted 6.12.0-rc1 #1-NixOS [ 1.917837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 1.919354] RIP: 0010:__list_del_entry_valid_or_report+0xb4/0xd0 [ 1.920180] Code: 0f 0b 48 89 fe 48 89 ca 48 c7 c7 38 52 41 9f e8 42 91 ac ff 90 0f 0b 48 89 fe 48 89 c2 48 c7 c7 70 52 41 9f e8 2d 91 ac ff 90 <0f> 0b 48 89 d1 48 c7 c7 c0 52 41 9f 48 89 f2 48 89 c6 e8 15 91 ac [ 1.922636] RSP: 0018:ffff96f800093c00 EFLAGS: 00010046 [ 1.923563] RAX: 000000000000006d RBX: 0000000000000001 RCX: 0000000000000000 [ 1.924692] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 1.925664] RBP: 0000000000000341 R08: 0000000000000000 R09: 0000000000000000 [ 1.926646] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8fbebd83dc90 [ 1.927584] R13: ffffc5cf80be0240 R14: ffff8fbebd83dc80 R15: 000000000002f809 [ 1.928533] FS: 0000000000000000(0000) GS:ffff8fbebd800000(0000) knlGS:0000000000000000 [ 1.929647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.930431] CR2: 00007fed6f09b000 CR3: 0000000001e02000 CR4: 0000000000350ef0 [ 1.931333] Call Trace: [ 1.931727] <TASK> [ 1.932088] ? die+0x36/0x90 [ 1.932595] ? do_trap+0xed/0x110 [ 1.933047] ? __list_del_entry_valid_or_report+0xb4/0xd0 [ 1.933757] ? do_error_trap+0x6a/0xa0 [ 1.934390] ? __list_del_entry_valid_or_report+0xb4/0xd0 [ 1.935073] ? exc_invalid_op+0x51/0x80 [ 1.935627] ? __list_del_entry_valid_or_report+0xb4/0xd0 [ 1.936326] ? asm_exc_invalid_op+0x1a/0x20 [ 1.936904] ? __list_del_entry_valid_or_report+0xb4/0xd0 [ 1.937622] free_pcppages_bulk+0x130/0x280 [ 1.938151] free_unref_page_commit+0x21c/0x380 [ 1.938753] free_unref_page+0x472/0x4f0 [ 1.939343] __put_partials+0xee/0x130 [ 1.939921] ? rcu_do_batch+0x1f2/0x800 [ 1.940471] kmem_cache_free+0x2c3/0x370 [ 1.940990] rcu_do_batch+0x1f2/0x800 [ 1.941508] ? rcu_do_batch+0x180/0x800 [ 1.942031] rcu_core+0x182/0x340 [ 1.942500] handle_softirqs+0xe4/0x2f0 [ 1.943034] run_ksoftirqd+0x33/0x40 [ 1.943522] smpboot_thread_fn+0xdd/0x1d0 [ 1.944056] ? __pfx_smpboot_thread_fn+0x10/0x10 [ 1.944679] kthread+0xd0/0x100 [ 1.945126] ? __pfx_kthread+0x10/0x10 [ 1.945656] ret_from_fork+0x34/0x50 [ 1.946151] ? __pfx_kthread+0x10/0x10 [ 1.946680] ret_from_fork_asm+0x1a/0x30 [ 1.947269] </TASK> [ 1.947622] Modules linked in: overlay 9p ext4 crc32c_generic crc16 mbcache jbd2 hid_generic usbhid hid 9pnet_virtio 9pnet netfs sr_mod virtio_net cdrom virtio_blk net_failover atkbd failover libps2 vivaldi_fmap crc32c_intel ata_piix libata uhci_hcd scsi_mod ehci_hcd virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev scsi_common i8042 serio rtc_cmos dm_mod dax virtio_gpu virtio_dma_buf virtio_rng rng_core virtio_console virtio_balloon virtio virtio_ring [ 1.952291] ---[ end trace 0000000000000000 ]--- [ 1.952893] RIP: 0010:__list_del_entry_valid_or_report+0xb4/0xd0 [ 1.953678] Code: 0f 0b 48 89 fe 48 89 ca 48 c7 c7 38 52 41 9f e8 42 91 ac ff 90 0f 0b 48 89 fe 48 89 c2 48 c7 c7 70 52 41 9f e8 2d 91 ac ff 90 <0f> 0b 48 89 d1 48 c7 c7 c0 52 41 9f 48 89 f2 48 89 c6 e8 15 91 ac [ 1.955888] RSP: 0018:ffff96f800093c00 EFLAGS: 00010046 [ 1.956548] RAX: 000000000000006d RBX: 0000000000000001 RCX: 0000000000000000 [ 1.957436] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 1.958328] RBP: 0000000000000341 R08: 0000000000000000 R09: 0000000000000000 [ 1.959166] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8fbebd83dc90 [ 1.960044] R13: ffffc5cf80be0240 R14: ffff8fbebd83dc80 R15: 000000000002f809 [ 1.960905] FS: 0000000000000000(0000) GS:ffff8fbebd800000(0000) knlGS:0000000000000000 [ 1.961926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.962693] CR2: 00007fed6f09b000 CR3: 0000000001e02000 CR4: 0000000000350ef0 [ 1.963548] Kernel panic - not syncing: Fatal exception in interrupt [ 1.964417] Kernel Offset: 0x1ce00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)On a subsequent run to verify this, it failed earlier while reading $targetRoot/.../bash like this:
[ 1.871810] BUG: Bad page state in process cat pfn:2e74a [ 1.872481] page: refcount:1 mapcount:0 mapping:0000000000000000 index:0x1e5 pfn:0x2e74a [ 1.873499] flags: 0xffffc000000000(node=0|zone=1|lastcpupid=0x1ffff) [ 1.874260] raw: 00ffffc000000000 dead000000000100 dead000000000122 0000000000000000 [ 1.875250] raw: 00000000000001e5 0000000000000000 00000001ffffffff 0000000000000000 [ 1.876295] page dumped because: nonzero _refcount [ 1.876910] Modules linked in: overlay 9p ext4 crc32c_generic crc16 mbcache jbd2 hid_generic usbhid hid 9pnet_virtio 9pnet netfs sr_mod virtio_net cdrom virtio_blk net_failover atkbd failover libps2 vivaldi_fmap crc32c_intel ata_piix libata scsi_mod uhci_hcd ehci_hcd virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev scsi_common i8042 serio rtc_cmos dm_mod dax virtio_gpu virtio_dma_buf virtio_rng rng_core virtio_console virtio_balloon virtio virtio_ring [ 1.881465] CPU: 0 UID: 0 PID: 315 Comm: cat Not tainted 6.12.0-rc1 #1-NixOS [ 1.882326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 1.883684] Call Trace: [ 1.884103] <TASK> [ 1.884440] dump_stack_lvl+0x64/0x90 [ 1.884954] bad_page+0x70/0x110 [ 1.885468] __rmqueue_pcplist+0x209/0xd00 [ 1.886029] ? srso_return_thunk+0x5/0x5f [ 1.886572] ? pdu_read+0x36/0x50 [9pnet] [ 1.887177] get_page_from_freelist+0x2df/0x1910 [ 1.887788] ? srso_return_thunk+0x5/0x5f [ 1.888324] ? enqueue_entity+0xce/0x510 [ 1.888881] ? srso_return_thunk+0x5/0x5f [ 1.889415] ? pick_eevdf+0x76/0x1a0 [ 1.889970] ? update_curr+0x35/0x270 [ 1.890476] __alloc_pages_noprof+0x1a3/0x1150 [ 1.891158] ? srso_return_thunk+0x5/0x5f [ 1.891712] ? __mod_memcg_lruvec_state+0xa9/0x160 [ 1.892346] ? srso_return_thunk+0x5/0x5f [ 1.892919] ? __lruvec_stat_mod_folio+0x83/0xd0 [ 1.893521] alloc_pages_mpol_noprof+0x8f/0x1f0 [ 1.894148] folio_alloc_noprof+0x5b/0xb0 [ 1.894671] page_cache_ra_unbounded+0x11f/0x200 [ 1.895270] filemap_get_pages+0x538/0x6d0 [ 1.895813] ? srso_return_thunk+0x5/0x5f [ 1.896361] filemap_splice_read+0x136/0x320 [ 1.896948] backing_file_splice_read+0x52/0xa0 [ 1.897522] ovl_splice_read+0xd2/0xf0 [overlay] [ 1.898160] ? __pfx_ovl_file_accessed+0x10/0x10 [overlay] [ 1.898817] splice_direct_to_actor+0xb4/0x270 [ 1.899404] ? __pfx_direct_splice_actor+0x10/0x10 [ 1.900103] do_splice_direct+0x77/0xd0 [ 1.900627] ? __pfx_direct_file_splice_eof+0x10/0x10 [ 1.901308] do_sendfile+0x359/0x410 [ 1.901788] __x64_sys_sendfile64+0xb9/0xd0 [ 1.902370] do_syscall_64+0xb7/0x210 [ 1.902904] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1.903604] RIP: 0033:0x7fa9f3a7289e [ 1.904214] Code: 75 0e 00 f7 d8 64 89 02 b8 ff ff ff ff 31 d2 31 c9 31 ff 45 31 db c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 12 31 d2 31 c9 31 f6 31 ff 45 31 d2 45 31 db [ 1.906436] RSP: 002b:00007ffe6a82bde8 EFLAGS: 00000246 ORIG_RAX: 0000000000000028 [ 1.907400] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa9f3a7289e [ 1.908241] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000001 [ 1.909184] RBP: 00007ffe6a82be50 R08: 0000000000000000 R09: 0000000000000000 [ 1.910212] R10: 0000000001000000 R11: 0000000000000246 R12: 0000000000000001 [ 1.911117] R13: 0000000001000000 R14: 0000000000000001 R15: 0000000000000000 [ 1.911998] </TASK> [ 1.912376] Disabling lock debugging due to kernel taint [ 1.913479] list_del corruption. next->prev should be ffffc80e40b9d948, but was ffffc80e40b9d0c8. (next=ffffc80e40b9c7c8) [ 1.914823] ------------[ cut here ]------------ [ 1.915408] kernel BUG at lib/list_debug.c:65! [ 1.916050] Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 1.916785] CPU: 0 UID: 0 PID: 315 Comm: cat Tainted: G B 6.12.0-rc1 #1-NixOS [ 1.917877] Tainted: [B]=BAD_PAGE [ 1.918350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 1.919996] RIP: 0010:__list_del_entry_valid_or_report+0xcc/0xd0 [ 1.920903] Code: 89 fe 48 89 c2 48 c7 c7 70 52 41 ba e8 2d 91 ac ff 90 0f 0b 48 89 d1 48 c7 c7 c0 52 41 ba 48 89 f2 48 89 c6 e8 15 91 ac ff 90 <0f> 0b 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f [ 1.923423] RSP: 0018:ffff9ed880187748 EFLAGS: 00010246 [ 1.924210] RAX: 000000000000006d RBX: ffff94db3d83dc80 RCX: 0000000000000000 [ 1.925147] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 1.926051] RBP: ffffc80e40b9d940 R08: 0000000000000000 R09: 0000000000000000 [ 1.926940] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [ 1.927809] R13: ffff94db3d83dc80 R14: ffffc80e40b9d948 R15: ffff94db3ffd6180 [ 1.928695] FS: 00007fa9f396eb80(0000) GS:ffff94db3d800000(0000) knlGS:0000000000000000 [ 1.929728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.930540] CR2: 00000000004d1829 CR3: 0000000001dd2000 CR4: 0000000000350ef0 [ 1.931444] Call Trace: [ 1.931916] <TASK> [ 1.932357] ? die+0x36/0x90 [ 1.932831] ? do_trap+0xed/0x110 [ 1.933385] ? __list_del_entry_valid_or_report+0xcc/0xd0 [ 1.934073] ? do_error_trap+0x6a/0xa0 [ 1.934583] ? __list_del_entry_valid_or_report+0xcc/0xd0 [ 1.935242] ? exc_invalid_op+0x51/0x80 [ 1.935781] ? __list_del_entry_valid_or_report+0xcc/0xd0 [ 1.936484] ? asm_exc_invalid_op+0x1a/0x20 [ 1.937174] ? __list_del_entry_valid_or_report+0xcc/0xd0 [ 1.937926] ? __list_del_entry_valid_or_report+0xcb/0xd0 [ 1.938685] __rmqueue_pcplist+0xa5/0xd00 [ 1.939292] ? srso_return_thunk+0x5/0x5f [ 1.940004] ? __mod_memcg_lruvec_state+0xa9/0x160 [ 1.940758] ? srso_return_thunk+0x5/0x5f [ 1.941417] ? update_load_avg+0x7e/0x7f0 [ 1.942133] ? srso_return_thunk+0x5/0x5f [ 1.942838] ? srso_return_thunk+0x5/0x5f [ 1.943508] get_page_from_freelist+0x2df/0x1910 [ 1.944143] ? srso_return_thunk+0x5/0x5f [ 1.944696] ? check_preempt_wakeup_fair+0x1ee/0x240 [ 1.945335] ? srso_return_thunk+0x5/0x5f [ 1.945905] __alloc_pages_noprof+0x1a3/0x1150 [ 1.946489] ? __blk_flush_plug+0xf5/0x150 [ 1.947105] ? srso_return_thunk+0x5/0x5f [ 1.947629] ? __dquot_alloc_space+0x2a8/0x3a0 [ 1.948404] ? srso_return_thunk+0x5/0x5f [ 1.949116] ? __mod_memcg_lruvec_state+0xa9/0x160 [ 1.949888] alloc_pages_mpol_noprof+0x8f/0x1f0 [ 1.950514] folio_alloc_mpol_noprof+0x14/0x40 [ 1.951153] shmem_alloc_folio+0xa7/0xd0 [ 1.951692] ? shmem_recalc_inode+0x20/0x90 [ 1.952272] shmem_alloc_and_add_folio+0x109/0x490 [ 1.952940] ? filemap_get_entry+0x10f/0x1a0 [ 1.953570] ? srso_return_thunk+0x5/0x5f [ 1.954185] shmem_get_folio_gfp+0x248/0x610 [ 1.954791] shmem_write_begin+0x64/0x110 [ 1.955484] generic_perform_write+0xdf/0x2a0 [ 1.956239] shmem_file_write_iter+0x8a/0x90 [ 1.956882] iter_file_splice_write+0x33f/0x580 [ 1.957577] direct_splice_actor+0x54/0x140 [ 1.958178] splice_direct_to_actor+0xec/0x270 [ 1.958813] ? __pfx_direct_splice_actor+0x10/0x10 [ 1.959442] do_splice_direct+0x77/0xd0 [ 1.960018] ? __pfx_direct_file_splice_eof+0x10/0x10 [ 1.960726] do_sendfile+0x359/0x410 [ 1.961248] __x64_sys_sendfile64+0xb9/0xd0 [ 1.961905] do_syscall_64+0xb7/0x210 [ 1.962467] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1.963211] RIP: 0033:0x7fa9f3a7289e [ 1.963711] Code: 75 0e 00 f7 d8 64 89 02 b8 ff ff ff ff 31 d2 31 c9 31 ff 45 31 db c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 12 31 d2 31 c9 31 f6 31 ff 45 31 d2 45 31 db [ 1.965846] RSP: 002b:00007ffe6a82bde8 EFLAGS: 00000246 ORIG_RAX: 0000000000000028 [ 1.966788] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa9f3a7289e [ 1.967644] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000001 [ 1.968480] RBP: 00007ffe6a82be50 R08: 0000000000000000 R09: 0000000000000000 [ 1.969396] R10: 0000000001000000 R11: 0000000000000246 R12: 0000000000000001 [ 1.970315] R13: 0000000001000000 R14: 0000000000000001 R15: 0000000000000000 [ 1.971214] </TASK> [ 1.971572] Modules linked in: overlay 9p ext4 crc32c_generic crc16 mbcache jbd2 hid_generic usbhid hid 9pnet_virtio 9pnet netfs sr_mod virtio_net cdrom virtio_blk net_failover atkbd failover libps2 vivaldi_fmap crc32c_intel ata_piix libata scsi_mod uhci_hcd ehci_hcd virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev scsi_common i8042 serio rtc_cmos dm_mod dax virtio_gpu virtio_dma_buf virtio_rng rng_core virtio_console virtio_balloon virtio virtio_ring [ 1.976558] ---[ end trace 0000000000000000 ]--- [ 1.977219] RIP: 0010:__list_del_entry_valid_or_report+0xcc/0xd0 [ 1.978033] Code: 89 fe 48 89 c2 48 c7 c7 70 52 41 ba e8 2d 91 ac ff 90 0f 0b 48 89 d1 48 c7 c7 c0 52 41 ba 48 89 f2 48 89 c6 e8 15 91 ac ff 90 <0f> 0b 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f [ 1.980179] RSP: 0018:ffff9ed880187748 EFLAGS: 00010246 [ 1.980847] RAX: 000000000000006d RBX: ffff94db3d83dc80 RCX: 0000000000000000 [ 1.981705] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 [ 1.982584] RBP: ffffc80e40b9d940 R08: 0000000000000000 R09: 0000000000000000 [ 1.983464] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [ 1.984358] R13: ffff94db3d83dc80 R14: ffffc80e40b9d948 R15: ffff94db3ffd6180 [ 1.987765] FS: 00007fa9f396eb80(0000) GS:ffff94db3d800000(0000) knlGS:0000000000000000 [ 1.988805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.989497] CR2: 00000000004d1829 CR3: 0000000001dd2000 CR4: 0000000000350ef0 [ 1.990418] note: cat[315] exited with preempt_count 2I bisected it back to
ee4cdf7ba857a894ad1650d6ab77669cbbfa329ewhich also seems to touch part of the 9p VFS code.Unfortunately the revert didn't apply cleanly on 6.12-rc1, so I couldn't meaningfully test whether a simple revert solves the problem.
The VMs get the Nix store mounted via 9p. In the store are basically all build artifacts including the stage-2 init script of the system that is booted into in the VM test.
The invocation basically looks like this:
qemu-system-x86_64 -cpu max \ -name machine \ -m 1024 \ -smp 1 \ -device virtio-rng-pci \ -net nic,netdev=user.0,model=virtio -netdev user,id=user.0,"$QEMU_NET_OPTS" \ -virtfs local,path=/nix/store,security_model=none,mount_tag=nix-store \ -virtfs local,path="${SHARED_DIR:-$TMPDIR/xchg}",security_model=none,mount_tag=shared \ -virtfs local,path="$TMPDIR"/xchg,security_model=none,mount_tag=xchg \ -drive cache=writeback,file="$NIX_DISK_IMAGE",id=drive1,if=none,index=1,werror=report -device virtio-blk-pci,bootindex=1,drive=drive1,serial=root \ -device virtio-net-pci,netdev=vlan1,mac=52:54:00:12:01:01 \ -netdev vde,id=vlan1,sock="$QEMU_VDE_SOCKET_1" \ -device virtio-keyboard \ -usb \ -device usb-tablet,bus=usb-bus.0 \ -kernel ${NIXPKGS_QEMU_KERNEL_machine:-/nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/kernel} \ -initrd /nix/store/qqalw1iq1wbgq3ndx0cvqn3bfypn56w2-initrd-linux-6.12-rc1/initrd \ -append "$(cat /nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/kernel-params) init=/nix/store/zv87gw0yxfsslq0mcc35a99k54da9a4z-nixos-system-machine-test/init regInfo=/nix/store/5izvfal6xm2rk51v0r1h2cxcng33paby-closure-info/registration console=ttyS0 $QEMU_KERNEL_PARAMS" \ $QEMU_OPTSIf you're using Nix, you can also reproduce this by running
nix-build nixos/tests/kernel-generic.nix -A
linux_testingon
5c19646b81db43dd7f4b6954f17d71a523009706from https://github.com/nixos/nixpkgs.To me, this seems like a regression in rc1.
Is there anything else I can do to help troubleshooting this?
With best regards
Maximilian
With best regards
Maximilian