generic/204 is crashing on linux-next when testing f2fs
(fstests cc'ed so I can a get handy-dandy lore.kernel.org URL for the
exclude file with the background for the test exclusion.)
Hi, I've been running xfstests on ext4, xfs, btrfs, and f2fs on linux-next[1], and for the past week or two, the tests have been hanging due to a BUG getting triggered on either f2fs/encrypt[2] or f2fs/compress[3] test configrations.
[1] Using the command: "gce-xfstests ltm -c ext4/all,xfs/all,btrfs/all,f2fs/all -g auto --repo https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next --watch fs-next"
[2] https://github.com/tytso/xfstests-bld/blob/master/test-appliance/files/root/fs/f2fs/cfg/encrypt
[3] https://github.com/tytso/xfstests-bld/blob/master/test-appliance/files/root/fs/f2fs/cfg/compress
Interestingly, after the BUG_ON things get wedged so badly that "systemctl" hangs forever. So apparently whatever is going wrong, it causes the systemd to malfunction in a profound way. As a result, the test VM hangs for days until I get around to aborting the VM.
For now, I'll exclude generic/204 from the f2fs configs in my test appliance; if you have a chance to take a look at this and let me know when it might be safe to re-neable generic/204 for f2fs, that would be great. Thanks!
BTW, if you'd be interestedr receiving the daily ouptut of the gce-xfstests run for f2fs, let me know. That can be arranged...
- Ted
P.S. Here's the dmesg output from a run this morning using the linux-next kernel 6.18.0-xfstests-g781b8d22e5d5:
[ 1011.282502] run fstests generic/204 at 2025-12-01 22:27:28 [ 1011.604590] F2FS-fs (dm-1): f2fs_recover_fsync_data: recovery fsync data, check_only: 0 [ 1011.605016] F2FS-fs (dm-1): Mounted with checkpoint version = 55cbed75 [ 1065.186351] ------------[ cut here ]------------ [ 1065.191112] kernel BUG at fs/f2fs/segment.c:2872! [ 1065.195977] Oops: invalid opcode: 0000 [#1] SMP PTI [ 1065.201170] CPU: 0 UID: 0 PID: 390275 Comm: umount Tainted: G W 6.18.0-xfstests-g781b8d22e5d5 #1 PREEMPT(none) [ 1065.212886] Tainted: [W]=WARN [ 1065.216868] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/02/2025 [ 1065.226646] RIP: 0010:get_new_segment+0x28d/0x2b0 [ 1065.231684] Code: e4 ff ff ff e9 5f ff ff ff ba 08 00 00 00 31 f6 4c 89 e7 e8 25 14 fe ff 4c 89 ef e8 6d ea 6c 00 b8 8b ff ff ff e9 3e ff ff ff <0f> 0b 83 7c 24 04 01 0f 85 4e fe ff ff 41 39 ee 0f 82 45 fe ff ff [ 1065.250635] RSP: 0018:ffffb3d684a2b860 EFLAGS: 00010246 [ 1065.255988] RAX: 0000000000000031 RBX: ffffb3d684a2b8b4 RCX: 0000000000000031 [ 1065.263230] RDX: 0000000000000000 RSI: 0000000000000031 RDI: ffff8e76c1d3e290 [ 1065.270556] RBP: 0000000000000001 R08: ffff8e76c1d3e288 R09: 0000000000000000 [ 1065.277801] R10: 0000000000000001 R11: 0000000000000000 R12: ffff8e76c6b1f000 [ 1065.285090] R13: ffff8e7729c2308c R14: 0000000000000000 R15: ffff8e7729c23080 [ 1065.292362] FS: 00007fb977a5f840(0000) GS:ffff8e78090e8000(0000) knlGS:0000000000000000 [ 1065.301096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1065.307080] CR2: 00007fef85ee165c CR3: 000000015b64e006 CR4: 00000000003706f0 [ 1065.314528] Call Trace: [ 1065.317324] <TASK> [ 1065.319547] new_curseg+0xb5/0x220 [ 1065.323166] f2fs_allocate_data_block+0x51e/0x750 [ 1065.328026] do_write_page+0x57/0x280 [ 1065.331823] f2fs_do_write_node_page+0x38/0x70 [ 1065.336395] __write_node_folio+0x2d1/0x4e0 [ 1065.340723] f2fs_sync_node_pages+0x428/0x550 [ 1065.345209] block_operations+0x33e/0x450 [ 1065.349518] f2fs_write_checkpoint+0x103/0x5e0 [ 1065.354119] kill_f2fs_super+0x14b/0x190 [ 1065.358297] deactivate_locked_super+0x33/0xb0 [ 1065.362918] cleanup_mnt+0xea/0x180 [ 1065.366582] task_work_run+0x5c/0xa0 [ 1065.370323] exit_to_user_mode_loop+0xb1/0xd0 [ 1065.374850] do_syscall_64+0x155/0x160 [ 1065.378830] entry_SYSCALL_64_after_hwframe+0x76/0x7e [ 1065.384054] RIP: 0033:0x7fb977cb8217 [ 1065.387743] Code: 0d 00 f7 d8 64 89 02 b8 ff ff ff ff c3 66 0f 1f 44 00 00 31 f6 e9 09 00 00 00 66 0f 1f 84 00 00 00 00 00 b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 b1 5b 0d 00 f7 d8 64 89 02 b8 [ 1065.406622] RSP: 002b:00007ffc531c8d48 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 1065.414317] RAX: 0000000000000000 RBX: 0000564b0abc7af8 RCX: 00007fb977cb8217 [ 1065.421570] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000564b0abce710 [ 1065.428834] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 1065.436085] R10: 0000000000000008 R11: 0000000000000246 R12: 00007fb977e1826c [ 1065.443415] R13: 0000564b0abce710 R14: 0000564b0abc7e30 R15: 0000564b0abc79f0 [ 1065.450667] </TASK> [ 1065.453067] ---[ end trace 0000000000000000 ]--- [ 1065.457915] RIP: 0010:get_new_segment+0x28d/0x2b0 [ 1065.462751] Code: e4 ff ff ff e9 5f ff ff ff ba 08 00 00 00 31 f6 4c 89 e7 e8 25 14 fe ff 4c 89 ef e8 6d ea 6c 00 b8 8b ff ff ff e9 3e ff ff ff <0f> 0b 83 7c 24 04 01 0f 85 4e fe ff ff 41 39 ee 0f 82 45 fe ff ff [ 1065.481788] RSP: 0018:ffffb3d684a2b860 EFLAGS: 00010246 [ 1065.487199] RAX: 0000000000000031 RBX: ffffb3d684a2b8b4 RCX: 0000000000000031 [ 1065.494559] RDX: 0000000000000000 RSI: 0000000000000031 RDI: ffff8e76c1d3e290 [ 1065.501898] RBP: 0000000000000001 R08: ffff8e76c1d3e288 R09: 0000000000000000 [ 1065.509183] R10: 0000000000000001 R11: 0000000000000000 R12: ffff8e76c6b1f000 [ 1065.516459] R13: ffff8e7729c2308c R14: 0000000000000000 R15: ffff8e7729c23080 [ 1065.523744] FS: 00007fb977a5f840(0000) GS:ffff8e78090e8000(0000) knlGS:0000000000000000 [ 1065.532160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1065.538091] CR2: 00007fef85ee165c CR3: 000000015b64e006 CR4: 00000000003706f0 [ 1065.545412] note: umount[390275] exited with preempt_count 1
Quoted 18 lines · Theodore Tso
(fstests cc'ed so I can a get handy-dandy lore.kernel.org URL for the
exclude file with the background for the test exclusion.)Hi, I've been running xfstests on ext4, xfs, btrfs, and f2fs on linux-next[1], and for the past week or two, the tests have been hanging due to a BUG getting triggered on either f2fs/encrypt[2] or f2fs/compress[3] test configrations.
[1] Using the command: "gce-xfstests ltm -c ext4/all,xfs/all,btrfs/all,f2fs/all -g auto --repo https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next --watch fs-next"
[2] https://github.com/tytso/xfstests-bld/blob/master/test-appliance/files/root/fs/f2fs/cfg/encrypt
[3] https://github.com/tytso/xfstests-bld/blob/master/test-appliance/files/root/fs/f2fs/cfg/compressInterestingly, after the
BUG_ONthings get wedged so badly that "systemctl" hangs forever. So apparently whatever is going wrong, it causes the systemd to malfunction in a profound way. As a result, the test VM hangs for days until I get around to aborting the VM.
Quoted 4 lines
For now, I'll exclude generic/204 from the f2fs configs in my test appliance; if you have a chance to take a look at this and let me know when it might be safe to re-neable generic/204 for f2fs, that would be great. Thanks!
Thank you for sharing the report. I didn't hit this in my setup tho, let us try to reproduce it first.
Quoted 70 lines
BTW, if you'd be interestedr receiving the daily ouptut of the gce-xfstests run for f2fs, let me know. That can be arranged...
- Ted
P.S. Here's the dmesg output from a run this morning using the linux-next kernel 6.18.0-xfstests-g781b8d22e5d5:[ 1011.282502] run fstests generic/204 at 2025-12-01 22:27:28 [ 1011.604590] F2FS-fs (dm-1): f2fs_recover_fsync_data: recovery fsync data, check_only: 0 [ 1011.605016] F2FS-fs (dm-1): Mounted with checkpoint version = 55cbed75 [ 1065.186351] ------------[ cut here ]------------ [ 1065.191112] kernel BUG at fs/f2fs/segment.c:2872! [ 1065.195977] Oops: invalid opcode: 0000 [#1] SMP PTI [ 1065.201170] CPU: 0 UID: 0 PID: 390275 Comm: umount Tainted: G W 6.18.0-xfstests-g781b8d22e5d5 #1 PREEMPT(none) [ 1065.212886] Tainted: [W]=WARN [ 1065.216868] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/02/2025 [ 1065.226646] RIP: 0010:get_new_segment+0x28d/0x2b0 [ 1065.231684] Code: e4 ff ff ff e9 5f ff ff ff ba 08 00 00 00 31 f6 4c 89 e7 e8 25 14 fe ff 4c 89 ef e8 6d ea 6c 00 b8 8b ff ff ff e9 3e ff ff ff <0f> 0b 83 7c 24 04 01 0f 85 4e fe ff ff 41 39 ee 0f 82 45 fe ff ff [ 1065.250635] RSP: 0018:ffffb3d684a2b860 EFLAGS: 00010246 [ 1065.255988] RAX: 0000000000000031 RBX: ffffb3d684a2b8b4 RCX: 0000000000000031 [ 1065.263230] RDX: 0000000000000000 RSI: 0000000000000031 RDI: ffff8e76c1d3e290 [ 1065.270556] RBP: 0000000000000001 R08: ffff8e76c1d3e288 R09: 0000000000000000 [ 1065.277801] R10: 0000000000000001 R11: 0000000000000000 R12: ffff8e76c6b1f000 [ 1065.285090] R13: ffff8e7729c2308c R14: 0000000000000000 R15: ffff8e7729c23080 [ 1065.292362] FS: 00007fb977a5f840(0000) GS:ffff8e78090e8000(0000) knlGS:0000000000000000 [ 1065.301096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1065.307080] CR2: 00007fef85ee165c CR3: 000000015b64e006 CR4: 00000000003706f0 [ 1065.314528] Call Trace: [ 1065.317324] <TASK> [ 1065.319547] new_curseg+0xb5/0x220 [ 1065.323166] f2fs_allocate_data_block+0x51e/0x750 [ 1065.328026] do_write_page+0x57/0x280 [ 1065.331823] f2fs_do_write_node_page+0x38/0x70 [ 1065.336395] __write_node_folio+0x2d1/0x4e0 [ 1065.340723] f2fs_sync_node_pages+0x428/0x550 [ 1065.345209] block_operations+0x33e/0x450 [ 1065.349518] f2fs_write_checkpoint+0x103/0x5e0 [ 1065.354119] kill_f2fs_super+0x14b/0x190 [ 1065.358297] deactivate_locked_super+0x33/0xb0 [ 1065.362918] cleanup_mnt+0xea/0x180 [ 1065.366582] task_work_run+0x5c/0xa0 [ 1065.370323] exit_to_user_mode_loop+0xb1/0xd0 [ 1065.374850] do_syscall_64+0x155/0x160 [ 1065.378830] entry_SYSCALL_64_after_hwframe+0x76/0x7e [ 1065.384054] RIP: 0033:0x7fb977cb8217 [ 1065.387743] Code: 0d 00 f7 d8 64 89 02 b8 ff ff ff ff c3 66 0f 1f 44 00 00 31 f6 e9 09 00 00 00 66 0f 1f 84 00 00 00 00 00 b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 b1 5b 0d 00 f7 d8 64 89 02 b8 [ 1065.406622] RSP: 002b:00007ffc531c8d48 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [ 1065.414317] RAX: 0000000000000000 RBX: 0000564b0abc7af8 RCX: 00007fb977cb8217 [ 1065.421570] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000564b0abce710 [ 1065.428834] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 1065.436085] R10: 0000000000000008 R11: 0000000000000246 R12: 00007fb977e1826c [ 1065.443415] R13: 0000564b0abce710 R14: 0000564b0abc7e30 R15: 0000564b0abc79f0 [ 1065.450667] </TASK> [ 1065.453067] ---[ end trace 0000000000000000 ]--- [ 1065.457915] RIP: 0010:get_new_segment+0x28d/0x2b0 [ 1065.462751] Code: e4 ff ff ff e9 5f ff ff ff ba 08 00 00 00 31 f6 4c 89 e7 e8 25 14 fe ff 4c 89 ef e8 6d ea 6c 00 b8 8b ff ff ff e9 3e ff ff ff <0f> 0b 83 7c 24 04 01 0f 85 4e fe ff ff 41 39 ee 0f 82 45 fe ff ff [ 1065.481788] RSP: 0018:ffffb3d684a2b860 EFLAGS: 00010246 [ 1065.487199] RAX: 0000000000000031 RBX: ffffb3d684a2b8b4 RCX: 0000000000000031 [ 1065.494559] RDX: 0000000000000000 RSI: 0000000000000031 RDI: ffff8e76c1d3e290 [ 1065.501898] RBP: 0000000000000001 R08: ffff8e76c1d3e288 R09: 0000000000000000 [ 1065.509183] R10: 0000000000000001 R11: 0000000000000000 R12: ffff8e76c6b1f000 [ 1065.516459] R13: ffff8e7729c2308c R14: 0000000000000000 R15: ffff8e7729c23080 [ 1065.523744] FS: 00007fb977a5f840(0000) GS:ffff8e78090e8000(0000) knlGS:0000000000000000 [ 1065.532160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1065.538091] CR2: 00007fef85ee165c CR3: 000000015b64e006 CR4: 00000000003706f0 [ 1065.545412] note: umount[390275] exited with preempt_count 1
Quoted 7 lines · Jaegeuk Kim
Quoted 4 lines
For now, I'll exclude generic/204 from the f2fs configs in my test appliance; if you have a chance to take a look at this and let me know when it might be safe to re-neable generic/204 for f2fs, that would be great. Thanks!
Thank you for sharing the report. I didn't hit this in my setup tho, let us try to reproduce it first.
I'm not able to reproduce it by running generic/204 by itself; only when running the full auto group. (See below). The generic/204 test reformats the scratch partition at the beginning of the test, and doesn't depend on the test partition. Presumably, a previous test when running the auto group is somehow setting up generic/204 to fail.
I'll try doing a run with KASAN enabled, in case that gives us some insight.
- Ted
TESTRUNID: ltm-20251202161646 KERNEL: kernel 6.18.0-rc4-xfstests-kasan-00072-g6fb67ac89690 #314 SMP PREEMPT_DYNAMIC Tue Dec 2 16:05:26 EST 2025 x86_64 CMDLINE: --kernel gs://gce-xfstests/kernel.deb -c f2fs/all -a -C 30 generic/204 CPUS: 2 MEM: 7680
f2fs/default: 30 tests, 6043 seconds f2fs/encrypt: 30 tests, 6209 seconds f2fs/compress: 30 tests, 6140 seconds
Totals: 90 tests, 0 skipped, 0 failures, 0 errors, 18392s
FSTESTIMG: gce-xfstests/xfstests-amd64-202512020012
FSTESTPRJ: gce-xfstests
FSTESTVER: blktests 4badb27 (Fri, 31 Oct 2025 19:53:27 +0900)
FSTESTVER: fio fio-3.41 (Fri, 5 Sep 2025 14:21:17 -0600)
FSTESTVER: fsverity v1.7 (Tue, 4 Nov 2025 17:28:17 -0800)
FSTESTVER: ima-evm-utils v1.5 (Mon, 6 Mar 2023 07:40:07 -0500)
FSTESTVER: libaio libaio-0.3.108-82-gb8eadc9 (Thu, 2 Jun 2022 13:33:11 +0200)
FSTESTVER: ltp 20250930 (Tue, 30 Sep 2025 13:53:45 +0200)
FSTESTVER: quota v4.05-77-g22ff3d9 (Tue, 2 Sep 2025 08:12:02 -0400)
FSTESTVER: util-linux v2.41.2 (Mon, 22 Sep 2025 12:56:34 +0200)
FSTESTVER: xfsprogs v6.17.0 (Mon, 20 Oct 2025 16:49:00 +0200)
FSTESTVER: xfstests-bld gce-xfstests-202504292206-34-g84a091b7 (Tue, 2 Dec 2025 00:09:25 -0500)
FSTESTVER: xfstests v2025.11.04-12-g64e3dbda7 (Thu, 13 Nov 2025 19:58:03 -0500)
FSTESTVER: zz_build-distro trixie
FSTESTSET: generic/204
FSTESTOPT: count 30 fail_loop_count 0
_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Quoted 9 lines · Theodore Tso
Quoted 7 lines · Jaegeuk Kim
Quoted 4 lines
For now, I'll exclude generic/204 from the f2fs configs in my test appliance; if you have a chance to take a look at this and let me know when it might be safe to re-neable generic/204 for f2fs, that would be great. Thanks!
Thank you for sharing the report. I didn't hit this in my setup tho, let us try to reproduce it first.
Quoted 8 lines
I'm not able to reproduce it by running generic/204 by itself; only when running the full auto group. (See below). The generic/204 test reformats the scratch partition at the beginning of the test, and doesn't depend on the test partition. Presumably, a previous test when running the auto group is somehow setting up generic/204 to fail.
I'll try doing a run with KASAN enabled, in case that gives us some insight.
Thank you. fyi; I found some other xfstests failures, hence dropped some suspicious patches and pushed today. Currently, my setup passed all the test-cases with it, so if possible, could you please check probably tomorrow linux-next?
Thanks,
Quoted 30 lines
- Ted
TESTRUNID: ltm-20251202161646 KERNEL: kernel 6.18.0-rc4-xfstests-kasan-00072-g6fb67ac89690 #314 SMP PREEMPT_DYNAMIC Tue Dec 2 16:05:26 EST 2025 x86_64 CMDLINE: --kernel gs://gce-xfstests/kernel.deb -c f2fs/all -a -C 30 generic/204 CPUS: 2 MEM: 7680f2fs/default: 30 tests, 6043 seconds f2fs/encrypt: 30 tests, 6209 seconds f2fs/compress: 30 tests, 6140 seconds
Totals: 90 tests, 0 skipped, 0 failures, 0 errors, 18392sFSTESTIMG: gce-xfstests/xfstests-amd64-202512020012
FSTESTPRJ: gce-xfstests
FSTESTVER: blktests 4badb27 (Fri, 31 Oct 2025 19:53:27 +0900)
FSTESTVER: fio fio-3.41 (Fri, 5 Sep 2025 14:21:17 -0600)
FSTESTVER: fsverity v1.7 (Tue, 4 Nov 2025 17:28:17 -0800)
FSTESTVER: ima-evm-utils v1.5 (Mon, 6 Mar 2023 07:40:07 -0500)
FSTESTVER: libaio libaio-0.3.108-82-gb8eadc9 (Thu, 2 Jun 2022 13:33:11 +0200)
FSTESTVER: ltp 20250930 (Tue, 30 Sep 2025 13:53:45 +0200)
FSTESTVER: quota v4.05-77-g22ff3d9 (Tue, 2 Sep 2025 08:12:02 -0400)
FSTESTVER: util-linux v2.41.2 (Mon, 22 Sep 2025 12:56:34 +0200)
FSTESTVER: xfsprogs v6.17.0 (Mon, 20 Oct 2025 16:49:00 +0200)
FSTESTVER: xfstests-bld gce-xfstests-202504292206-34-g84a091b7 (Tue, 2 Dec 2025 00:09:25 -0500)
FSTESTVER: xfstests v2025.11.04-12-g64e3dbda7 (Thu, 13 Nov 2025 19:58:03 -0500)
FSTESTVER: zz_build-distro trixie
FSTESTSET: generic/204
FSTESTOPT: count 30fail_loop_count0