Skip to contentSkip to conversation

weird splats in xfs/561 on 6.10-rc1?

DISCUSSION Darrick J. Wong 2 participants 5 messages May 30, 2024 – May 31, 2024

lore
Share & tools
lore
#0 lore raw

Hi Christoph,

I noticed the following assert triggering on xfs/561 after about 8 minutes of operation on a 64k-page arm64 vm with a rtgroups filesystem:

FSTYP         -- xfs (debug)
PLATFORM      -- Linux/aarch64 oci-mtra46 6.10.0-rc1-xfsa #rc1 SMP PREEMPT Wed May 29 11:26:10 PDT 2024
MKFS_OPTIONS  -- -f -rrtdev=/dev/sdb4 -i verity=1,exchange=1, -d rtinherit=1, -n parent=1, -r rtgroups=1, /dev/sda3

MOUNT_OPTIONS -- -o usrquota,grpquota,prjquota, -ortdev=/dev/sdb4 /dev/sda3 /opt

XFS (sda3): ino 414aa33 data fork has delalloc extent at [0x16a:0x6]
XFS: Assertion failed: 0, file: fs/xfs/xfs_icache.c, line: 1853

------------[ cut here ]------------
WARNING: CPU: 0 PID: 44 at fs/xfs/xfs_message.c:104 assfail+0x48/0x68 [xfs]
Modules linked in: xfs rpcsec_gss_krb5 auth_rpcgss nft_chain_nat xt_REDIRECT nf_nat nf_
CPU: 0 PID: 44 Comm: kswapd0 Tainted: G        W          6.10.0-rc1-xfsa #rc1 1ccdfe2f
Hardware name: QEMU KVM Virtual Machine, BIOS 1.6.6 08/22/2023
pstate: 60401005 (nZCv daif +PAN -UAO -TCO -DIT +SSBS BTYPE=--)
pc : assfail+0x48/0x68 [xfs]
lr : assfail+0x3c/0x68 [xfs]

sp : fffffe0081fef7b0
x29: fffffe0081fef7b0 x28: fffffe0081fefab8 x27: 000000000000f35c
x26: 0000000000000005 x25: 00000000000003fb x24: 0000000000000000
x23: fffffc013f518000 x22: fffffe007a40ebe4 x21: fffffe0080f30008
x20: fffffe0081289458 x19: fffffc00e5f25f00 x18: 0000000000000006
x17: 3a61363178305b20 x16: 746120746e657478 x15: fffffe0081fef1a0
x14: 0000000000000000 x13: 33353831203a656e x12: fffffe0081fef6e0
x11: fffffe007a644790 x10: fffffe00fa64478f x9 : 0fffffffffffffff x8 : 000000000000000a x7 : 00000000ffffffc0 x6 : 0000000000000021 x5 : fffffe007a644791 x4 : 00000000ffffffca x3 : 0000000000000000 x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000000

Call trace:
 assfail+0x48/0x68 [xfs 33c3f304945dadccc25d02a0520df74527538fda]
 xfs_inodegc_set_reclaimable+0x164/0x180 [xfs 33c3f304945dadccc25d02a0520df74527538fda]
 xfs_inode_mark_reclaimable+0xd0/0x460 [xfs 33c3f304945dadccc25d02a0520df74527538fda]
 xfs_fs_destroy_inode+0xf4/0x1b8 [xfs 33c3f304945dadccc25d02a0520df74527538fda]
 destroy_inode+0x48/0x88
 evict+0x158/0x1a8
 dispose_list+0x6c/0xa8
 prune_icache_sb+0x64/0xa0
 super_cache_scan+0x148/0x1a8
 do_shrink_slab+0x14c/0x470
 shrink_slab+0x304/0x528
 shrink_one+0xa8/0x240
 shrink_node+0xb18/0xe50
 balance_pgdat+0x398/0x8b0
 kswapd+0x244/0x518
 kthread+0x110/0x128
 ret_from_fork+0x10/0x20
---[ end trace 0000000000000000 ]---

Direct I/O collision with buffered writes! File: d32/d36/d5a/f54 Comm: fsstress Direct I/O collision with buffered writes! File: /rt/p3/f25 Comm: fsstress

From what I can tell, this wasn't happening with my 6.9 djwong-dev branch, so I suspect it's something that came in from when I rebased against 6.10-rc1. It seems to happen all over the place (and not just with realtime files) if I leave Zhang Yi's iomap patches applied. If I revert them, the screaming seems to go down to just this one test.

The file itself is ~1482KB, or enough for the file to have 0x169 actual blocks of written data to it, so the delalloc reservation is beyond the eof block. Any thoughts?

--D

#1 lore raw
Quoted 5 lines · Darrick J. Wong
Quoted 1 line

From what I can tell, this wasn't happening with my 6.9 djwong-dev

branch, so I suspect it's something that came in from when I rebased against 6.10-rc1. It seems to happen all over the place (and not just with realtime files) if I leave Zhang Yi's iomap patches applied. If I revert them, the screaming seems to go down to just this one test.

When testing Linus' tree I haven't seen this yet, but I also haven't done a lot of testing yet. I mostly triggered odd MM warnings that Johannes fixes, but I haven't seen something like this yet.

Quoted 3 lines

The file itself is ~1482KB, or enough for the file to have 0x169 actual blocks of written data to it, so the delalloc reservation is beyond the eof block. Any thoughts?

I'll see if I can reproduce it with your tree. I have a pretty full plate given that yesterday was a public holiday here which doesn't really help my catch up rate..

#2 lore raw
Quoted 11 lines · Christoph Hellwig
Quoted 5 lines · Darrick J. Wong
Quoted 1 line

From what I can tell, this wasn't happening with my 6.9 djwong-dev

branch, so I suspect it's something that came in from when I rebased against 6.10-rc1. It seems to happen all over the place (and not just with realtime files) if I leave Zhang Yi's iomap patches applied. If I revert them, the screaming seems to go down to just this one test.

When testing Linus' tree I haven't seen this yet, but I also haven't done a lot of testing yet. I mostly triggered odd MM warnings that Johannes fixes, but I haven't seen something like this yet.

Quoted 7 lines
Quoted 3 lines

The file itself is ~1482KB, or enough for the file to have 0x169 actual blocks of written data to it, so the delalloc reservation is beyond the eof block. Any thoughts?

I'll see if I can reproduce it with your tree. I have a pretty full plate given that yesterday was a public holiday here which doesn't really help my catch up rate..

Yeah. You might want to revert all of Zhang Yi's patches first, though maybe his new series actually fixes all the problems.

(Hm, no, it's still missing that cow-over-hole thing Dave was musing about.)

--D

#3 lore raw
Quoted 5 lines · Darrick J. Wong

Yeah. You might want to revert all of Zhang Yi's patches first, though maybe his new series actually fixes all the problems.

(Hm, no, it's still missing that cow-over-hole thing Dave was musing about.)

I've kicked off a run with realtime and rthinherit on your current realtime reflink tree and haven't seen any warning yet. Do you know if it's limited to > 4K page sizes? My testing ability on that is unfortunately limited to a used macbook running linux in a VM as that Asahi installer sucks ass, so rather limited.

#4 lore raw
Quoted 4 lines · Christoph Hellwig
Quoted 3 lines · Darrick J. Wong

Yeah. You might want to revert all of Zhang Yi's patches first, though maybe his new series actually fixes all the problems.

Quoted 8 lines
Quoted 2 lines

(Hm, no, it's still missing that cow-over-hole thing Dave was musing about.)

I've kicked off a run with realtime and rthinherit on your current realtime reflink tree and haven't seen any warning yet. Do you know if it's limited to > 4K page sizes? My testing ability on that is unfortunately limited to a used macbook running linux in a VM as that Asahi installer sucks ass, so rather limited.

I don't know specifically if it's limited to > 4K pagesize, other than to say that I haven't seen it show up on x86 yet.

--D