weird splats in xfs/561 on 6.10-rc1?
Hi Christoph,
I noticed the following assert triggering on xfs/561 after about 8 minutes of operation on a 64k-page arm64 vm with a rtgroups filesystem:
FSTYP -- xfs (debug) PLATFORM -- Linux/aarch64 oci-mtra46 6.10.0-rc1-xfsa #rc1 SMP PREEMPT Wed May 29 11:26:10 PDT 2024 MKFS_OPTIONS -- -f -rrtdev=/dev/sdb4 -i verity=1,exchange=1, -d rtinherit=1, -n parent=1, -r rtgroups=1, /dev/sda3
MOUNT_OPTIONS -- -o usrquota,grpquota,prjquota, -ortdev=/dev/sdb4 /dev/sda3 /opt
XFS (sda3): ino 414aa33 data fork has delalloc extent at [0x16a:0x6]
XFS: Assertion failed: 0, file: fs/xfs/xfs_icache.c, line: 1853
------------[ cut here ]------------ WARNING: CPU: 0 PID: 44 at fs/xfs/xfs_message.c:104 assfail+0x48/0x68 [xfs] Modules linked in: xfs rpcsec_gss_krb5 auth_rpcgss nft_chain_nat xt_REDIRECT nf_nat nf_ CPU: 0 PID: 44 Comm: kswapd0 Tainted: G W 6.10.0-rc1-xfsa #rc1 1ccdfe2f Hardware name: QEMU KVM Virtual Machine, BIOS 1.6.6 08/22/2023 pstate: 60401005 (nZCv daif +PAN -UAO -TCO -DIT +SSBS BTYPE=--) pc : assfail+0x48/0x68 [xfs] lr : assfail+0x3c/0x68 [xfs]
sp : fffffe0081fef7b0
x29: fffffe0081fef7b0 x28: fffffe0081fefab8 x27: 000000000000f35c
x26: 0000000000000005 x25: 00000000000003fb x24: 0000000000000000
x23: fffffc013f518000 x22: fffffe007a40ebe4 x21: fffffe0080f30008
x20: fffffe0081289458 x19: fffffc00e5f25f00 x18: 0000000000000006
x17: 3a61363178305b20 x16: 746120746e657478 x15: fffffe0081fef1a0
x14: 0000000000000000 x13: 33353831203a656e x12: fffffe0081fef6e0
x11: fffffe007a644790 x10: fffffe00fa64478f x9 : 0fffffffffffffff x8 : 000000000000000a x7 : 00000000ffffffc0 x6 : 0000000000000021 x5 : fffffe007a644791 x4 : 00000000ffffffca x3 : 0000000000000000 x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000000
Call trace: assfail+0x48/0x68 [xfs 33c3f304945dadccc25d02a0520df74527538fda] xfs_inodegc_set_reclaimable+0x164/0x180 [xfs 33c3f304945dadccc25d02a0520df74527538fda] xfs_inode_mark_reclaimable+0xd0/0x460 [xfs 33c3f304945dadccc25d02a0520df74527538fda] xfs_fs_destroy_inode+0xf4/0x1b8 [xfs 33c3f304945dadccc25d02a0520df74527538fda] destroy_inode+0x48/0x88 evict+0x158/0x1a8 dispose_list+0x6c/0xa8 prune_icache_sb+0x64/0xa0 super_cache_scan+0x148/0x1a8 do_shrink_slab+0x14c/0x470 shrink_slab+0x304/0x528 shrink_one+0xa8/0x240 shrink_node+0xb18/0xe50 balance_pgdat+0x398/0x8b0 kswapd+0x244/0x518 kthread+0x110/0x128 ret_from_fork+0x10/0x20 ---[ end trace 0000000000000000 ]---
Direct I/O collision with buffered writes! File: d32/d36/d5a/f54 Comm: fsstress Direct I/O collision with buffered writes! File: /rt/p3/f25 Comm: fsstress
From what I can tell, this wasn't happening with my 6.9 djwong-dev branch, so I suspect it's something that came in from when I rebased against 6.10-rc1. It seems to happen all over the place (and not just with realtime files) if I leave Zhang Yi's iomap patches applied. If I revert them, the screaming seems to go down to just this one test.
The file itself is ~1482KB, or enough for the file to have 0x169 actual blocks of written data to it, so the delalloc reservation is beyond the eof block. Any thoughts?
--D
Quoted 5 lines · Darrick J. Wong
Quoted 1 line
From what I can tell, this wasn't happening with my 6.9 djwong-dev
branch, so I suspect it's something that came in from when I rebased against 6.10-rc1. It seems to happen all over the place (and not just with realtime files) if I leave Zhang Yi's iomap patches applied. If I revert them, the screaming seems to go down to just this one test.
When testing Linus' tree I haven't seen this yet, but I also haven't done a lot of testing yet. I mostly triggered odd MM warnings that Johannes fixes, but I haven't seen something like this yet.
Quoted 3 lines
The file itself is ~1482KB, or enough for the file to have
0x169actual blocks of written data to it, so the delalloc reservation is beyond the eof block. Any thoughts?
I'll see if I can reproduce it with your tree. I have a pretty full plate given that yesterday was a public holiday here which doesn't really help my catch up rate..
Quoted 11 lines · Christoph Hellwig
Quoted 5 lines · Darrick J. Wong
Quoted 1 line
From what I can tell, this wasn't happening with my 6.9 djwong-dev
branch, so I suspect it's something that came in from when I rebased against 6.10-rc1. It seems to happen all over the place (and not just with realtime files) if I leave Zhang Yi's iomap patches applied. If I revert them, the screaming seems to go down to just this one test.
When testing Linus' tree I haven't seen this yet, but I also haven't done a lot of testing yet. I mostly triggered odd MM warnings that Johannes fixes, but I haven't seen something like this yet.
Quoted 7 lines
Quoted 3 lines
The file itself is ~1482KB, or enough for the file to have
0x169actual blocks of written data to it, so the delalloc reservation is beyond the eof block. Any thoughts?I'll see if I can reproduce it with your tree. I have a pretty full plate given that yesterday was a public holiday here which doesn't really help my catch up rate..
Yeah. You might want to revert all of Zhang Yi's patches first, though maybe his new series actually fixes all the problems.
(Hm, no, it's still missing that cow-over-hole thing Dave was musing about.)
--D
Quoted 5 lines · Darrick J. Wong
Yeah. You might want to revert all of Zhang Yi's patches first, though maybe his new series actually fixes all the problems.
(Hm, no, it's still missing that cow-over-hole thing Dave was musing about.)
I've kicked off a run with realtime and rthinherit on your current realtime reflink tree and haven't seen any warning yet. Do you know if it's limited to > 4K page sizes? My testing ability on that is unfortunately limited to a used macbook running linux in a VM as that Asahi installer sucks ass, so rather limited.
Quoted 4 lines · Christoph Hellwig
Quoted 3 lines · Darrick J. Wong
Yeah. You might want to revert all of Zhang Yi's patches first, though maybe his new series actually fixes all the problems.
Quoted 8 lines
Quoted 2 lines
(Hm, no, it's still missing that cow-over-hole thing Dave was musing about.)
I've kicked off a run with realtime and rthinherit on your current realtime reflink tree and haven't seen any warning yet. Do you know if it's limited to > 4K page sizes? My testing ability on that is unfortunately limited to a used macbook running linux in a VM as that Asahi installer sucks ass, so rather limited.
I don't know specifically if it's limited to > 4K pagesize, other than to say that I haven't seen it show up on x86 yet.
--D