regression from 974c5e6139db "xfs: flag as supporting FOP_DONTCACHE" (double free on page?)
Subsystems (3)
Al Viro
maintainergeneric/127 with xfstests built on debian-testing (trixie) ends up with assorted memory corruption; trace below is with CONFIG_DEBUG_PAGEALLOC and CONFIG_DEBUG_PAGEALLOC_ENABLE_DEFAULT and it looks like a double free somewhere in iomap. Unfortunately, commit in question is just making xfs use the infrastructure built in earlier series - not that useful for isolating the breakage.
[ 22.001529] run fstests generic/127 at 2025-05-25 04:13:23 [ 35.498573] BUG: Bad page state in process kworker/2:1 pfn:112ce9 [ 35.499260] page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x3e 9 [ 35.499764] flags: 0x800000000000000e(referenced|uptodate|writeback|zone=2) [ 35.500302] raw: 800000000000000e dead000000000100 dead000000000122 000000000 [ 35.500786] raw: 000000000000003e 0000000000000000 00000000ffffffff 000000000 [ 35.501248] page dumped because: PAGE_FLAGS_CHECK_AT_FREE flag(s) set [ 35.501624] Modules linked in: xfs autofs4 fuse nfsd auth_rpcgss nfs_acl nfs0 [ 35.503209] CPU: 2 UID: 0 PID: 85 Comm: kworker/2:1 Not tainted 6.14.0-rc1+ 7 [ 35.503211] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.164 [ 35.503212] Workqueue: xfs-conv/sdb1 xfs_end_io [xfs] [ 35.503279] Call Trace: [ 35.503281] <TASK> [ 35.503282] dump_stack_lvl+0x4f/0x60 [ 35.503296] bad_page+0x6f/0x100 [ 35.503300] free_frozen_pages+0x303/0x550 [ 35.503301] iomap_finish_ioend+0xf6/0x380 [ 35.503304] iomap_finish_ioends+0x83/0xc0 [ 35.503305] xfs_end_ioend+0x64/0x140 [xfs] [ 35.503342] xfs_end_io+0x93/0xc0 [xfs] [ 35.503378] process_one_work+0x153/0x390 [ 35.503382] worker_thread+0x2ab/0x3b0
It's 4:30am here, so I'm going to leave attempts to actually debug that thing until tomorrow; I do have a kvm where it's reliably reproduced within a few minutes, so if anyone comes up with patches, I'll be able to test them.
Breakage is still present in the current mainline ;-/
Al Viro
maintainerQuoted 6 lines · Al Viro
generic/127 with xfstests built on debian-testing (trixie) ends up with assorted memory corruption; trace below is with
CONFIG_DEBUG_PAGEALLOCandCONFIG_DEBUG_PAGEALLOC_ENABLE_DEFAULTand it looks like a double free somewhere in iomap. Unfortunately, commit in question is just making xfs use the infrastructure built in earlier series - not that useful for isolating the breakage.
FWIW, the same breakage is reproduced within a couple of iterations of ./check generic/127 on debian-testing image with xfstests built fresh from git and debian linux-image-6.15-rc7-amd64-unsigned_6.15~rc7-1~exp1_amd64.deb
IOW, it's not something exotic in .config here. KVM setup is also not unusual -
kvm \ -boot order=c \ -m 16384 \ -netdev "tap,id=nic0,ifname=tap4,script=no,downscript=no" \ -device "e1000,netdev=nic0" \ -nographic \ -smp 4 \ -hdb /home/al/emu/ssd/image \ trixie.img
with image partitioned into two 6G xfs filesystems, with
export TEST_DEV=/dev/sdb1
export TEST_DIR=/home/test
export SCRATCH_DEV=/dev/sdb2
export SCRATCH_MNT=/home/scratch
for local.config. Bog-standard install, ext4 for everything on sda, nothing fancy for storage setup - qemu defaults all way through.
Al Viro
maintainerQuoted 1 line · Al Viro
Breakage is still present in the current mainline ;-/
With CONFIG_DEBUG_VM on top of pagealloc debugging:
[ 1434.992817] run fstests generic/127 at 2025-05-25 11:46:11g [ 1448.956242] BUG: Bad page state in process kworker/2:1 pfn:112cb0g [ 1448.956846] page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x3e pfn:0x112cb0g [ 1448.957453] flags: 0x800000000000000e(referenced|uptodate|writeback|zone=2)g [ 1448.957863] raw: 800000000000000e dead000000000100 dead000000000122 0000000000000000g [ 1448.958303] raw: 000000000000003e 0000000000000000 00000000ffffffff 0000000000000000g [ 1448.958833] page dumped because: PAGE_FLAGS_CHECK_AT_FREE flag(s) setg [ 1448.959320] Modules linked in: xfs autofs4 fuse nfsd auth_rpcgss nfs_acl nfs lockd grace sunrpc loop ecryptfs 9pnet_virtio 9pnet netfs evdev pcspkr sg button ext4 jbd2 btrfs blake2b_generic xor zlib_deflate raid6_pq zstd_compress sr_mod cdrom ata_generic ata_piix psmouse serio_raw i2c_piix4 i2c_smbus libata e1000g [ 1448.960874] CPU: 2 UID: 0 PID: 2614 Comm: kworker/2:1 Not tainted 6.14.0-rc1+ #78g [ 1448.960878] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014g [ 1448.960879] Workqueue: xfs-conv/sdb1 xfs_end_io [xfs]g [ 1448.960938] Call Trace:g [ 1448.960939] <TASK>g [ 1448.960940] dump_stack_lvl+0x4f/0x60g [ 1448.960953] bad_page+0x6f/0x100g [ 1448.960957] free_frozen_pages+0x471/0x640g [ 1448.960958] iomap_finish_ioend+0x196/0x3c0g [ 1448.960963] iomap_finish_ioends+0x83/0xc0g [ 1448.960964] xfs_end_ioend+0x64/0x140 [xfs]g [ 1448.961003] xfs_end_io+0x93/0xc0 [xfs]g [ 1448.961036] process_one_work+0x153/0x390g [ 1448.961044] worker_thread+0x2ab/0x3b0g [ 1448.961045] ? rescuer_thread+0x470/0x470g [ 1448.961047] kthread+0xf7/0x200g [ 1448.961048] ? kthread_use_mm+0xa0/0xa0g [ 1448.961049] ret_from_fork+0x2d/0x50g [ 1448.961053] ? kthread_use_mm+0xa0/0xa0g [ 1448.961054] ret_from_fork_asm+0x11/0x20g [ 1448.961058] </TASK>g [ 1448.961155] Disabling lock debugging due to kernel taintg [ 1448.969569] page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x3e pfn:0x112cb0g [ 1448.970023] flags: 0x800000000000000e(referenced|uptodate|writeback|zone=2)g [ 1448.970651] raw: 800000000000000e dead000000000100 dead000000000122 0000000000000000g [ 1448.971222] raw: 000000000000003e 0000000000000000 00000000ffffffff 0000000000000000g [ 1448.971812] page dumped because: VM_BUG_ON_FOLIO(((unsigned int) folio_ref_count(folio) + 127u <= 127u))g [ 1448.972490] ------------[ cut here ]------------g [ 1448.972841] kernel BUG at ./include/linux/mm.h:1455!g [ 1448.973421] Oops: invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOCg [ 1448.973853] CPU: 2 UID: 0 PID: 2614 Comm: kworker/2:1 Tainted: G B 6.14.0-rc1+ #78g [ 1448.974345] Tainted: [B]=BAD_PAGEg [ 1448.974565] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014g [ 1448.975074] Workqueue: xfs-conv/sdb1 xfs_end_io [xfs]g [ 1448.975428] RIP: 0010:folio_end_writeback+0x155/0x180g [ 1448.975731] Code: 13 40 0f 92 c5 e9 23 ff ff ff 48 c7 c6 00 d5 e7 81 48 89 df e8 0c 8a 03 00 0f 0b 48 c7 c6 d0 38 e5 81 48 89 df e8 fb 89 03 00 <0f> 0b 48 c7 c6 40 5b e5 81 48 89 df e8 ea 89 03 00 0f 0b 48 c7 c6g [ 1448.976655] RSP: 0018:ffffc90001a53d68 EFLAGS: 00010286g [ 1448.976953] RAX: 000000000000005c RBX: ffffea00044b2c00 RCX: 0000000000000000g [ 1448.977331] RDX: 0000000000000001 RSI: ffffffff81e74e9e RDI: 00000000ffffffffg [ 1448.977711] RBP: ffffea00044b2c40 R08: 0000000000004ffb R09: 00000000ffffefffg [ 1448.978089] R10: 00000000ffffefff R11: ffffffff82043bc0 R12: 0000000000001000g [ 1448.978464] R13: ffff888101ecb840 R14: 0000000000000000 R15: ffffea00044b2c00g [ 1448.978844] FS: 0000000000000000(0000) GS:ffff88842dd00000(0000) knlGS:0000000000000000g [ 1448.979289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033g [ 1448.979609] CR2: 00007fd3d42a2000 CR3: 0000000111543000 CR4: 00000000000006f0g [ 1448.979989] Call Trace:g [ 1448.980170] <TASK>g [ 1448.980336] ? die+0x32/0x80g [ 1448.980543] ? do_trap+0xd5/0x100g [ 1448.980767] ? folio_end_writeback+0x155/0x180g [ 1448.981033] ? do_error_trap+0x65/0x80g [ 1448.981270] ? folio_end_writeback+0x155/0x180g [ 1448.981536] ? exc_invalid_op+0x4c/0x60g [ 1448.981790] ? folio_end_writeback+0x155/0x180g [ 1448.982056] ? asm_exc_invalid_op+0x16/0x20g [ 1448.982315] ? folio_end_writeback+0x155/0x180g [ 1448.982580] ? folio_end_writeback+0x155/0x180g [ 1448.982846] iomap_finish_ioend+0x196/0x3c0g [ 1448.983108] iomap_finish_ioends+0x55/0xc0g [ 1448.983363] xfs_end_ioend+0x64/0x140 [xfs]g [ 1448.983663] xfs_end_io+0x93/0xc0 [xfs]g [ 1448.983937] process_one_work+0x153/0x390g [ 1448.984189] worker_thread+0x2ab/0x3b0g [ 1448.984427] ? rescuer_thread+0x470/0x470g [ 1448.984674] kthread+0xf7/0x200g [ 1448.984887] ? kthread_use_mm+0xa0/0xa0g [ 1448.985128] ret_from_fork+0x2d/0x50g [ 1448.985362] ? kthread_use_mm+0xa0/0xa0g [ 1448.985601] ret_from_fork_asm+0x11/0x20g [ 1448.985846] </TASK>g [ 1448.986017] Modules linked in: xfs autofs4 fuse nfsd auth_rpcgss nfs_acl nfs lockd grace sunrpc loop ecryptfs 9pnet_virtio 9pnet netfs evdev pcspkr sg button ext4 jbd2 btrfs blake2b_generic xor zlib_deflate raid6_pq zstd_compress sr_mod cdrom ata_generic ata_piix psmouse serio_raw i2c_piix4 i2c_smbus libata e1000g [ 1448.987399] ---[ end trace 0000000000000000 ]---g [ 1448.987896] RIP: 0010:folio_end_writeback+0x155/0x180g [ 1448.988220] Code: 13 40 0f 92 c5 e9 23 ff ff ff 48 c7 c6 00 d5 e7 81 48 89 df e8 0c 8a 03 00 0f 0b 48 c7 c6 d0 38 e5 81 48 89 df e8 fb 89 03 00 <0f> 0b 48 c7 c6 40 5b e5 81 48 89 df e8 ea 89 03 00 0f 0b 48 c7 c6g [ 1448.989246] RSP: 0018:ffffc90001a53d68 EFLAGS: 00010286g [ 1448.992210] RAX: 000000000000005c RBX: ffffea00044b2c00 RCX: 0000000000000000g [ 1448.992619] RDX: 0000000000000001 RSI: ffffffff81e74e9e RDI: 00000000ffffffffg [ 1448.993010] RBP: ffffea00044b2c40 R08: 0000000000004ffb R09: 00000000ffffefffg [ 1448.993577] R10: 00000000ffffefff R11: ffffffff82043bc0 R12: 0000000000001000g [ 1448.994411] R13: ffff888101ecb840 R14: 0000000000000000 R15: ffffea00044b2c00g [ 1448.994823] FS: 0000000000000000(0000) GS:ffff88842dd00000(0000) knlGS:0000000000000000g [ 1448.995390] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033g [ 1448.995916] CR2: 00007fd3d42a2000 CR3: 0000000111543000 CR4: 00000000000006f0g
kvm: terminating on signal 15 from pid 32057 (killall)
Quoted 10 lines · Al Viro
Quoted 1 line · Al Viro
Breakage is still present in the current mainline ;-/
With
CONFIG_DEBUG_VMon top of pagealloc debugging:[ 1434.992817] run fstests generic/127 at 2025-05-25 11:46:11g [ 1448.956242] BUG: Bad page state in process kworker/2:1 pfn:112cb0g [ 1448.956846] page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x3e pfn:0x112cb0g [ 1448.957453] flags: 0x800000000000000e(referenced|uptodate|writeback|zone=2)g
It doesn't like the writeback flag.
Quoted 19 lines
[ 1448.957863] raw: 800000000000000e dead000000000100 dead000000000122 0000000000000000g [ 1448.958303] raw: 000000000000003e 0000000000000000 00000000ffffffff 0000000000000000g [ 1448.958833] page dumped because: PAGE_FLAGS_CHECK_AT_FREE flag(s) setg [ 1448.959320] Modules linked in: xfs autofs4 fuse nfsd auth_rpcgss nfs_acl nfs lockd grace sunrpc loop ecryptfs 9pnet_virtio 9pnet netfs evdev pcspkr sg button ext4 jbd2 btrfs blake2b_generic xor zlib_deflate raid6_pq zstd_compress sr_mod cdrom ata_generic ata_piix psmouse serio_raw i2c_piix4 i2c_smbus libata e1000g [ 1448.960874] CPU: 2 UID: 0 PID: 2614 Comm: kworker/2:1 Not tainted 6.14.0-rc1+ #78g [ 1448.960878] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014g [ 1448.960879] Workqueue: xfs-conv/sdb1 xfs_end_io [xfs]g [ 1448.960938] Call Trace:g [ 1448.960939] <TASK>g [ 1448.960940] dump_stack_lvl+0x4f/0x60g [ 1448.960953] bad_page+0x6f/0x100g [ 1448.960957] free_frozen_pages+0x471/0x640g [ 1448.960958] iomap_finish_ioend+0x196/0x3c0g [ 1448.960963] iomap_finish_ioends+0x83/0xc0g [ 1448.960964] xfs_end_ioend+0x64/0x140 [xfs]g [ 1448.961003] xfs_end_io+0x93/0xc0 [xfs]g [ 1448.961036] process_one_work+0x153/0x390g [ 1448.961044] worker_thread+0x2ab/0x3b0g [ 1448.961045] ? rescuer_thread+0x470/0x470g
Quoted 8 lines
[ 1448.961047] kthread+0xf7/0x200g [ 1448.961048] ? kthread_use_mm+0xa0/0xa0g [ 1448.961049] ret_from_fork+0x2d/0x50g [ 1448.961053] ? kthread_use_mm+0xa0/0xa0g [ 1448.961054] ret_from_fork_asm+0x11/0x20g [ 1448.961058] </TASK>g [ 1448.961155] Disabling lock debugging due to kernel taintg [ 1448.969569] page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x3e pfn:0x112cb0g
same pfn, same struct page
Quoted 6 lines
[ 1448.970023] flags: 0x800000000000000e(referenced|uptodate|writeback|zone=2)g [ 1448.970651] raw: 800000000000000e dead000000000100 dead000000000122 0000000000000000g [ 1448.971222] raw: 000000000000003e 0000000000000000 00000000ffffffff 0000000000000000g [ 1448.971812] page dumped because: VM_BUG_ON_FOLIO(((unsigned int) folio_ref_count(folio) + 127u <= 127u))g [ 1448.972490] ------------[ cut here ]------------g [ 1448.972841] kernel BUG at ./include/linux/mm.h:1455!g
this is folio_get() noticing refcount is 0, so a use-after free, because we already tried to free the page above.
I'm not familiar with this code too much, but I suspect problem was introduced by commit fb7d3bc414939 ("mm/filemap: drop streaming/uncached pages when writeback completes") and only (more) exposed here.
so in folio_end_writeback() we have
if (__folio_end_writeback(folio))
folio_wake_bit(folio, PG_writeback);but calling the folio_end_dropbehind_write() doesn't depend on the result of __folio_end_writeback()
this seems rather suspicious
I think if __folio_end_writeback() was true then PG_writeback would be cleared and thus we'd not see the PAGE_FLAGS_CHECK_AT_FREE failure. Instead we do a premature folio_end_dropbehind_write() dropping a page ref and then the final folio_put() in folio_end_writeback() frees the page and splats on the PG_writeback. Then the folio is processed again in the following iteration of iomap_finish_ioend() and splats on the refcount-already-zero.
So I think folio_end_dropbehind_write() should only be done when __folio_end_writeback() was true. Most likely even the folio_test_clear_dropbehind() should be tied to that, or we clear it too early and then never act upon it later?
Well, this isn't great timing, since I was going to do 6.15 within the hour.
Quoted 4 lines · Vlastimil Babka
I'm not familiar with this code too much, but I suspect problem was introduced by commit
fb7d3bc414939("mm/filemap: drop streaming/uncached pages when writeback completes") and only (more) exposed here.
That bug goes back to 6.13 if so.
But yeah, maybe the drop-behind case never triggers in practice, and I should just revert commit 974c5e6139db ("xfs: flag as supporting FOP_DONTCACHE") for now.
That's kind of sad too, but at least that's new to 6.15 and we wouldn't have a kernel release that triggers this issue.
I realize that Vlastimil had a suggested possible fix, but doing _that_ kind of surgery at this point in the release isn't an option, I'm afraid. And delaying 6.15 for this also seems a bit excessive - if it turns out to be easy to fix, we can always just backport the fix and undo the revert.
Sounds like a plan?
I'm somewhat surprised that this was only noticed now if it triggers so easily for Al with xfstests on xfs. But better late than never, I guess..
Linus
Matthew Wilcox
maintainerQuoted 13 lines · Linus Torvalds
But yeah, maybe the drop-behind case never triggers in practice, and I should just revert commit
974c5e6139db("xfs: flag as supportingFOP_DONTCACHE") for now.That's kind of sad too, but at least that's new to 6.15 and we wouldn't have a kernel release that triggers this issue.
I realize that Vlastimil had a suggested possible fix, but doing _that_ kind of surgery at this point in the release isn't an option, I'm afraid. And delaying 6.15 for this also seems a bit excessive - if it turns out to be easy to fix, we can always just backport the fix and undo the revert.
Quoted 5 lines
Sounds like a plan?
I'm somewhat surprised that this was only noticed now if it triggers so easily for Al with xfstests on xfs. But better late than never, I guess..
I wonder if we shouldn't do ...
+++ b/include/linux/fs.h
@@ -3725,6 +3725,8 @@ static inline int kiocb_set_rw_flags(struct kiocb *ki, rwf_t flags,
return -EOPNOTSUPP;
}
if (flags & RWF_DONTCACHE) {
+ /* Houston, we have a problem */
+ return -EOPNOTSUPP;
/* file system must support it */
if (!(ki->ki_filp->f_op->fop_flags & FOP_DONTCACHE))
return -EOPNOTSUPP;in case some other filesystem adds support for it? I don't see anything in -next right now, but I see Darrick playing with it here for FUSE:
https://lore.kernel.org/all/174787195629.1483178.7917092102987513364.stgit@frogsfrogsfrogs/
Jeff playing with it for nfsd here:
https://lore.kernel.org/all/[email protected]/
Trond implementing it for NFS client here:
https://lore.kernel.org/all/[email protected]/
I thought I saw someone implement it for ext4, but perhaps I'm confused with something else. Anyway, some kind of not-xfs-specific patch is appropriate here, I think?
Oh, and we're only just seeing it, I think, because you need to recompile xfstests to test this functionality ... and I certainly don't re-pull and re-compile xfstests on a regular basis; I just use the one I pulled and compiled, um, months ago.
Review static inline int kiocb_set_rw_flags(struct kiocb *ki, rwf_t flags, · line 3729
Quoted 10 lines · Matthew Wilcox
I wonder if we shouldn't do ...
+++
b/include/linux/fs.h(quoted diff) +2
-0@@ -3725,6 +3725,8 @@ static inline int kiocb_set_rw_flags(struct kiocb *ki, rwf_t flags, return -EOPNOTSUPP; } if (flags & RWF_DONTCACHE) { + /* Houston, we have a problem */ + return -EOPNOTSUPP;
Hmm. Your point about other filesystems is well taken.
I'd have preferred a revert as a "don't do anything new at this point", but I guess disabling it at this point is probably the safer option considering that this isn't a xfs issue.
Quoted 2 lines
Oh, and we're only just seeing it, I think, because you need to recompile xfstests to test this functionality ...
Ahh, good. Well, not "good" exactly, but it certainly at least explains the unlucky timing.
Thanks,
Linus
Al Viro
maintainerQuoted 25 lines · Matthew Wilcox
Quoted 13 lines · Linus Torvalds
But yeah, maybe the drop-behind case never triggers in practice, and I should just revert commit
974c5e6139db("xfs: flag as supportingFOP_DONTCACHE") for now.That's kind of sad too, but at least that's new to 6.15 and we wouldn't have a kernel release that triggers this issue.
I realize that Vlastimil had a suggested possible fix, but doing _that_ kind of surgery at this point in the release isn't an option, I'm afraid. And delaying 6.15 for this also seems a bit excessive - if it turns out to be easy to fix, we can always just backport the fix and undo the revert.
Quoted 5 lines
Sounds like a plan?
I'm somewhat surprised that this was only noticed now if it triggers so easily for Al with xfstests on xfs. But better late than never, I guess..
I wonder if we shouldn't do ...
+++
b/include/linux/fs.h
Review static inline int kiocb_set_rw_flags(struct kiocb *ki, rwf_t flags, · line 3732
Quoted 8 lines
(quoted diff) +2
-0@@ -3725,6 +3725,8 @@ static inline int kiocb_set_rw_flags(struct kiocb *ki, rwf_t flags, return -EOPNOTSUPP; } if (flags & RWF_DONTCACHE) { + /* Houston, we have a problem */ + return -EOPNOTSUPP; /* file system must support it */ if (!(ki->ki_filp->f_op->fop_flags & FOP_DONTCACHE)) return -EOPNOTSUPP;
Perhaps
-#define FOP_DONTCACHE ((__force fop_flags_t)(1 << 7)) when shit gets fixed
+#define FOP_DONTCACHE 0 // ((__force fop_flags_t)(1 << 7)) when shit gets fixed
instead?
Quoted 7 lines · Al Viro
Perhaps
(quoted diff) +1
-1-#define FOP_DONTCACHE ((__force fop_flags_t)(1 << 7)) when shit gets fixed +#define FOP_DONTCACHE 0 // ((__force fop_flags_t)(1 << 7)) when shit gets fixedinstead?
Yeah, I think that ends up being prettier than an extra error return in the middle of code.
Will do. Thanks for noticing this, even if the timing is awkward.
Linus
Quoted 74 lines · Vlastimil Babka
Quoted 10 lines · Al Viro
Quoted 1 line · Al Viro
Breakage is still present in the current mainline ;-/
With
CONFIG_DEBUG_VMon top of pagealloc debugging:[ 1434.992817] run fstests generic/127 at 2025-05-25 11:46:11g [ 1448.956242] BUG: Bad page state in process kworker/2:1 pfn:112cb0g [ 1448.956846] page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x3e pfn:0x112cb0g [ 1448.957453] flags: 0x800000000000000e(referenced|uptodate|writeback|zone=2)gIt doesn't like the writeback flag.
Quoted 19 lines
[ 1448.957863] raw: 800000000000000e dead000000000100 dead000000000122 0000000000000000g [ 1448.958303] raw: 000000000000003e 0000000000000000 00000000ffffffff 0000000000000000g [ 1448.958833] page dumped because: PAGE_FLAGS_CHECK_AT_FREE flag(s) setg [ 1448.959320] Modules linked in: xfs autofs4 fuse nfsd auth_rpcgss nfs_acl nfs lockd grace sunrpc loop ecryptfs 9pnet_virtio 9pnet netfs evdev pcspkr sg button ext4 jbd2 btrfs blake2b_generic xor zlib_deflate raid6_pq zstd_compress sr_mod cdrom ata_generic ata_piix psmouse serio_raw i2c_piix4 i2c_smbus libata e1000g [ 1448.960874] CPU: 2 UID: 0 PID: 2614 Comm: kworker/2:1 Not tainted 6.14.0-rc1+ #78g [ 1448.960878] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014g [ 1448.960879] Workqueue: xfs-conv/sdb1 xfs_end_io [xfs]g [ 1448.960938] Call Trace:g [ 1448.960939] <TASK>g [ 1448.960940] dump_stack_lvl+0x4f/0x60g [ 1448.960953] bad_page+0x6f/0x100g [ 1448.960957] free_frozen_pages+0x471/0x640g [ 1448.960958] iomap_finish_ioend+0x196/0x3c0g [ 1448.960963] iomap_finish_ioends+0x83/0xc0g [ 1448.960964] xfs_end_ioend+0x64/0x140 [xfs]g [ 1448.961003] xfs_end_io+0x93/0xc0 [xfs]g [ 1448.961036] process_one_work+0x153/0x390g [ 1448.961044] worker_thread+0x2ab/0x3b0g [ 1448.961045] ? rescuer_thread+0x470/0x470gQuoted 8 lines
[ 1448.961047] kthread+0xf7/0x200g [ 1448.961048] ? kthread_use_mm+0xa0/0xa0g [ 1448.961049] ret_from_fork+0x2d/0x50g [ 1448.961053] ? kthread_use_mm+0xa0/0xa0g [ 1448.961054] ret_from_fork_asm+0x11/0x20g [ 1448.961058] </TASK>g [ 1448.961155] Disabling lock debugging due to kernel taintg [ 1448.969569] page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x3e pfn:0x112cb0gsame pfn, same struct page
Quoted 6 lines
[ 1448.970023] flags: 0x800000000000000e(referenced|uptodate|writeback|zone=2)g [ 1448.970651] raw: 800000000000000e dead000000000100 dead000000000122 0000000000000000g [ 1448.971222] raw: 000000000000003e 0000000000000000 00000000ffffffff 0000000000000000g [ 1448.971812] page dumped because: VM_BUG_ON_FOLIO(((unsigned int) folio_ref_count(folio) + 127u <= 127u))g [ 1448.972490] ------------[ cut here ]------------g [ 1448.972841] kernel BUG at ./include/linux/mm.h:1455!gthis is
folio_get()noticing refcount is 0, so a use-after free, because we already tried to free the page above.I'm not familiar with this code too much, but I suspect problem was introduced by commit
fb7d3bc414939("mm/filemap: drop streaming/uncached pages when writeback completes") and only (more) exposed here.so in
folio_end_writeback()we haveif (__folio_end_writeback(folio)) folio_wake_bit(folio, PG_writeback);but calling the
folio_end_dropbehind_write()doesn't depend on the result of__folio_end_writeback()
this seems rather suspiciousI think if
__folio_end_writeback()was true thenPG_writebackwould be cleared and thus we'd not see thePAGE_FLAGS_CHECK_AT_FREEfailure. Instead we do a prematurefolio_end_dropbehind_write()dropping a page ref and then the finalfolio_put()infolio_end_writeback()frees the page and splats on thePG_writeback. Then the folio is processed again in the following iteration ofiomap_finish_ioend()and splats on the refcount-already-zero.
Quoted 4 lines
So I think
folio_end_dropbehind_write()should only be done when__folio_end_writeback()was true. Most likely even thefolio_test_clear_dropbehind()should be tied to that, or we clear it too early and then never act upon it later?
Thanks for taking a look at this! I tried to reproduce this this morning and failed miserably. I then injected a delay for the above case, and it does indeed then trigger for me. So far, so good.
I agree with your analysis, we should only be doing the dropbehind for a non-zero return from __folio_end_writeback(), and that includes the test_and_clear to avoid dropping the drop-behind state. But we also need to check/clear this state pre __folio_end_writeback(), which then puts us in a spot where it needs to potentially be re-set. Which fails pretty racy...
I'll ponder this a bit. Good thing fsx got RWF_DONTCACHE support, or I suspect this would've taken a while to run into.
Signature · 1 line
Jens Axboe
Quoted 91 lines · Jens Axboe
Quoted 74 lines · Vlastimil Babka
Quoted 10 lines · Al Viro
Quoted 1 line · Al Viro
Breakage is still present in the current mainline ;-/
With
CONFIG_DEBUG_VMon top of pagealloc debugging:[ 1434.992817] run fstests generic/127 at 2025-05-25 11:46:11g [ 1448.956242] BUG: Bad page state in process kworker/2:1 pfn:112cb0g [ 1448.956846] page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x3e pfn:0x112cb0g [ 1448.957453] flags: 0x800000000000000e(referenced|uptodate|writeback|zone=2)gIt doesn't like the writeback flag.
Quoted 19 lines
[ 1448.957863] raw: 800000000000000e dead000000000100 dead000000000122 0000000000000000g [ 1448.958303] raw: 000000000000003e 0000000000000000 00000000ffffffff 0000000000000000g [ 1448.958833] page dumped because: PAGE_FLAGS_CHECK_AT_FREE flag(s) setg [ 1448.959320] Modules linked in: xfs autofs4 fuse nfsd auth_rpcgss nfs_acl nfs lockd grace sunrpc loop ecryptfs 9pnet_virtio 9pnet netfs evdev pcspkr sg button ext4 jbd2 btrfs blake2b_generic xor zlib_deflate raid6_pq zstd_compress sr_mod cdrom ata_generic ata_piix psmouse serio_raw i2c_piix4 i2c_smbus libata e1000g [ 1448.960874] CPU: 2 UID: 0 PID: 2614 Comm: kworker/2:1 Not tainted 6.14.0-rc1+ #78g [ 1448.960878] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014g [ 1448.960879] Workqueue: xfs-conv/sdb1 xfs_end_io [xfs]g [ 1448.960938] Call Trace:g [ 1448.960939] <TASK>g [ 1448.960940] dump_stack_lvl+0x4f/0x60g [ 1448.960953] bad_page+0x6f/0x100g [ 1448.960957] free_frozen_pages+0x471/0x640g [ 1448.960958] iomap_finish_ioend+0x196/0x3c0g [ 1448.960963] iomap_finish_ioends+0x83/0xc0g [ 1448.960964] xfs_end_ioend+0x64/0x140 [xfs]g [ 1448.961003] xfs_end_io+0x93/0xc0 [xfs]g [ 1448.961036] process_one_work+0x153/0x390g [ 1448.961044] worker_thread+0x2ab/0x3b0g [ 1448.961045] ? rescuer_thread+0x470/0x470gQuoted 8 lines
[ 1448.961047] kthread+0xf7/0x200g [ 1448.961048] ? kthread_use_mm+0xa0/0xa0g [ 1448.961049] ret_from_fork+0x2d/0x50g [ 1448.961053] ? kthread_use_mm+0xa0/0xa0g [ 1448.961054] ret_from_fork_asm+0x11/0x20g [ 1448.961058] </TASK>g [ 1448.961155] Disabling lock debugging due to kernel taintg [ 1448.969569] page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x3e pfn:0x112cb0gsame pfn, same struct page
Quoted 6 lines
[ 1448.970023] flags: 0x800000000000000e(referenced|uptodate|writeback|zone=2)g [ 1448.970651] raw: 800000000000000e dead000000000100 dead000000000122 0000000000000000g [ 1448.971222] raw: 000000000000003e 0000000000000000 00000000ffffffff 0000000000000000g [ 1448.971812] page dumped because: VM_BUG_ON_FOLIO(((unsigned int) folio_ref_count(folio) + 127u <= 127u))g [ 1448.972490] ------------[ cut here ]------------g [ 1448.972841] kernel BUG at ./include/linux/mm.h:1455!gthis is
folio_get()noticing refcount is 0, so a use-after free, because we already tried to free the page above.I'm not familiar with this code too much, but I suspect problem was introduced by commit
fb7d3bc414939("mm/filemap: drop streaming/uncached pages when writeback completes") and only (more) exposed here.so in
folio_end_writeback()we haveif (__folio_end_writeback(folio)) folio_wake_bit(folio, PG_writeback);but calling the
folio_end_dropbehind_write()doesn't depend on the result of__folio_end_writeback()
this seems rather suspiciousI think if
__folio_end_writeback()was true thenPG_writebackwould be cleared and thus we'd not see thePAGE_FLAGS_CHECK_AT_FREEfailure. Instead we do a prematurefolio_end_dropbehind_write()dropping a page ref and then the finalfolio_put()infolio_end_writeback()frees the page and splats on thePG_writeback. Then the folio is processed again in the following iteration ofiomap_finish_ioend()and splats on the refcount-already-zero.Quoted 4 lines
So I think
folio_end_dropbehind_write()should only be done when__folio_end_writeback()was true. Most likely even thefolio_test_clear_dropbehind()should be tied to that, or we clear it too early and then never act upon it later?Thanks for taking a look at this! I tried to reproduce this this morning and failed miserably. I then injected a delay for the above case, and it does indeed then trigger for me. So far, so good.
I agree with your analysis, we should only be doing the dropbehind for a non-zero return from
__folio_end_writeback(), and that includes thetest_and_clearto avoid dropping the drop-behind state. But we also need to check/clear this state pre__folio_end_writeback(), which then puts us in a spot where it needs to potentially be re-set. Which fails pretty racy...
Quoted 2 lines
I'll ponder this a bit. Good thing fsx got
RWF_DONTCACHEsupport, or I suspect this would've taken a while to run into.
Took a closer look... I may be smoking something good here, but I don't see what the __folio_end_writeback()() return value has to do with this at all. Regardless of what it returns, it should've cleared PG_writeback, and in fact the only thing it returns is whether or not we had anyone waiting on it. Which should have _zero_ bearing on whether or not we can clear/invalidate the range.
To me, this smells more like a race of some sort, between dirty and invalidation. fsx does a lot of sub-page sized operations.
I'll poke a bit more...
Signature · 1 line
Jens Axboe
Quoted 17 lines · Jens Axboe
Quoted 13 lines · Jens Axboe
On 5/25/25 1:12 PM, Vlastimil Babka wrote:
Thanks for taking a look at this! I tried to reproduce this this morning and failed miserably. I then injected a delay for the above case, and it does indeed then trigger for me. So far, so good.
I agree with your analysis, we should only be doing the dropbehind for a non-zero return from
__folio_end_writeback(), and that includes thetest_and_clearto avoid dropping the drop-behind state. But we also need to check/clear this state pre__folio_end_writeback(), which then puts us in a spot where it needs to potentially be re-set. Which fails pretty racy...Quoted 2 lines
I'll ponder this a bit. Good thing fsx got
RWF_DONTCACHEsupport, or I suspect this would've taken a while to run into.
Quoted 6 lines
Took a closer look... I may be smoking something good here, but I don't see what the __folio_end_writeback()() return value has to do with this at all. Regardless of what it returns, it should've cleared
PG_writeback, and in fact the only thing it returns is whether or not we had anyone waiting on it. Which should have _zero_ bearing on whether or not we can clear/invalidate the range.
Yeah it's very much possible that I was wrong, folio_xor_flags_has_waiters() looked a bit impenetrable to me, and it seemed like an simple explanation to the splats. But as you had to add delays, this indeed smells as a race.
Quoted 5 lines
To me, this smells more like a race of some sort, between dirty and invalidation. fsx does a lot of sub-page sized operations.
I'll poke a bit more...
Quoted 25 lines · Vlastimil Babka
Quoted 17 lines · Jens Axboe
Quoted 13 lines · Jens Axboe
On 5/25/25 1:12 PM, Vlastimil Babka wrote:
Thanks for taking a look at this! I tried to reproduce this this morning and failed miserably. I then injected a delay for the above case, and it does indeed then trigger for me. So far, so good.
I agree with your analysis, we should only be doing the dropbehind for a non-zero return from
__folio_end_writeback(), and that includes thetest_and_clearto avoid dropping the drop-behind state. But we also need to check/clear this state pre__folio_end_writeback(), which then puts us in a spot where it needs to potentially be re-set. Which fails pretty racy...Quoted 2 lines
I'll ponder this a bit. Good thing fsx got
RWF_DONTCACHEsupport, or I suspect this would've taken a while to run into.Quoted 6 lines
Took a closer look... I may be smoking something good here, but I don't see what the __folio_end_writeback()() return value has to do with this at all. Regardless of what it returns, it should've cleared
PG_writeback, and in fact the only thing it returns is whether or not we had anyone waiting on it. Which should have _zero_ bearing on whether or not we can clear/invalidate the range.
Quoted 3 lines
Yeah it's very much possible that I was wrong,
folio_xor_flags_has_waiters()looked a bit impenetrable to me, and it seemed like an simple explanation to the splats. But as you had to add delays, this indeed smells as a race.
Here's my delay trace fwiw, which is a bit different:
BUG: Bad page state in process fsx pfn:4866b page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x25 pfn:0x4866b flags: 0x3ffe0000000000a(uptodate|writeback|node=0|zone=0|lastcpupid=0x1fff) raw: 03ffe0000000000a dead000000000100 dead000000000122 0000000000000000 raw: 0000000000000025 0000000000000000 00000000ffffffff 0000000000000000 page dumped because: PAGE_FLAGS_CHECK_AT_FREE flag(s) set Modules linked in: CPU: 6 UID: 0 PID: 1853 Comm: fsx Not tainted 6.15.0-rc7-00144-gb1427432d3b6-dirty #1053 NONE Hardware name: linux,dummy-virt (DT) Call trace: show_stack+0x1c/0x30 (C) dump_stack_lvl+0x58/0x78 dump_stack+0x18/0x20 bad_page+0x1a4/0x228 free_unref_folios+0xc2c/0x1920 folios_put_refs+0x354/0x5f0 __folio_batch_release+0x98/0xd0 writeback_iter+0x8f8/0xd00 iomap_writepages+0x16e4/0x2090 xfs_vm_writepages+0x200/0x2c0 do_writepages+0x148/0x7c0 filemap_fdatawrite_wbc+0xe0/0x138 __filemap_fdatawrite_range+0xb0/0x100 filemap_write_and_wait_range+0x68/0x100 __generic_remap_file_range_prep+0x418/0x1090 generic_remap_file_range_prep+0x18/0x80 xfs_reflink_remap_prep+0x160/0x7d8 xfs_file_remap_range+0x164/0xa90 vfs_dedupe_file_range_one+0x398/0x4a0 vfs_dedupe_file_range+0x410/0x648 do_vfs_ioctl+0x13c4/0x1fc0 __arm64_sys_ioctl+0xd8/0x188 invoke_syscall.constprop.0+0x60/0x2a0 el0_svc_common.constprop.0+0x148/0x240 do_el0_svc+0x40/0x60 el0_svc+0x34/0x70 el0t_64_sync_handler+0x104/0x138 el0t_64_sync+0x170/0x178 Disabling lock debugging due to kernel taint page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x25 pfn:0x4866b flags: 0x3ffe0000000000a(uptodate|writeback|node=0|zone=0|lastcpupid=0x1fff) raw: 03ffe0000000000a dead000000000100 dead000000000122 0000000000000000 raw: 0000000000000025 0000000000000000 00000000ffffffff 0000000000000000 page dumped because: VM_BUG_ON_FOLIO(((unsigned int) folio_ref_count(folio) + 127u <= 127u)) ------------[ cut here ]------------ kernel BUG at ./include/linux/mm.h:1543! Internal error: Oops - BUG: 00000000f2000800 [#1] SMP Modules linked in: CPU: 6 UID: 0 PID: 0 Comm: swapper/6 Tainted: G B 6.15.0-rc7-00144-gb1427432d3b6-dirty #1053 NONE Tainted: [B]=BAD_PAGE Hardware name: linux,dummy-virt (DT) pstate: 614000c5 (nZCv daIF +PAN -UAO -TCO +DIT -SSBS BTYPE=--) pc : folio_end_writeback+0x470/0x560 lr : folio_end_writeback+0x470/0x560
sp : ffff8000859978f0
x29: ffff8000859978f0 x28: dfff800000000000 x27: fffffdffc0219ac0
x26: 0000000000000000 x25: ffff000005ed8138 x24: 0000000000000000
x23: 1fffffbff804335e x22: 0000000000000004 x21: 0000000000000001
x20: fffffdffc0219af4 x19: fffffdffc0219ac0 x18: 000000000000000f
x17: 635f6665725f6f69 x16: 6c6f662029746e69 x15: 0720072007200720
x14: 0720072007200720 x13: 0720072007200720 x12: ffff60001b67150b
x11: 1fffe0001b67150a x10: ffff60001b67150a x9 : dfff800000000000 x8 : 00009fffe498eaf6 x7 : ffff0000db38a853 x6 : 0000000000000001 x5 : 0000000000000001 x4 : 0000000000000000 x3 : 0000000000000000 x2 : 0000000000000000 x1 : ffff0000c1f98000 x0 : 000000000000005c
Call trace: folio_end_writeback+0x470/0x560 (P) iomap_finish_ioend_buffered+0x38c/0x9e0 iomap_writepage_end_bio+0x80/0xc0 bio_endio+0x4dc/0x678 blk_mq_end_request_batch+0x2b4/0x10c0 nvme_pci_complete_batch+0x338/0x518 nvme_irq+0xd8/0xf0 __handle_irq_event_percpu+0xdc/0x528 handle_irq_event+0x174/0x3d8 handle_fasteoi_irq+0x2cc/0xba0 handle_irq_desc+0xb8/0x120 generic_handle_domain_irq+0x20/0x30 gic_handle_irq+0x50/0x140 call_on_irq_stack+0x24/0x50 do_interrupt_handler+0xe0/0x148 el1_interrupt+0x30/0x50 el1h_64_irq_handler+0x14/0x20 el1h_64_irq+0x6c/0x70 do_idle+0x244/0x4c8 (P) cpu_startup_entry+0x64/0x80 secondary_start_kernel+0x1e4/0x240 __secondary_switched+0x74/0x78 Code: 91190021 91218021 aa1303e0 94039279 (d4210000) ---[ end trace 0000000000000000 ]--- Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt
SMP: stopping secondary CPUs
Kernel Offset: disabled
CPU features: 0x0000,000000e0,0109a650,834e7607
Memory Limit: none
---[ end Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt ]---
Signature · 1 line
Jens Axboe
Quoted 102 lines · Jens Axboe
Quoted 91 lines · Jens Axboe
Quoted 74 lines · Vlastimil Babka
Quoted 10 lines · Al Viro
Quoted 1 line · Al Viro
Breakage is still present in the current mainline ;-/
With
CONFIG_DEBUG_VMon top of pagealloc debugging:[ 1434.992817] run fstests generic/127 at 2025-05-25 11:46:11g [ 1448.956242] BUG: Bad page state in process kworker/2:1 pfn:112cb0g [ 1448.956846] page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x3e pfn:0x112cb0g [ 1448.957453] flags: 0x800000000000000e(referenced|uptodate|writeback|zone=2)gIt doesn't like the writeback flag.
Quoted 19 lines
[ 1448.957863] raw: 800000000000000e dead000000000100 dead000000000122 0000000000000000g [ 1448.958303] raw: 000000000000003e 0000000000000000 00000000ffffffff 0000000000000000g [ 1448.958833] page dumped because: PAGE_FLAGS_CHECK_AT_FREE flag(s) setg [ 1448.959320] Modules linked in: xfs autofs4 fuse nfsd auth_rpcgss nfs_acl nfs lockd grace sunrpc loop ecryptfs 9pnet_virtio 9pnet netfs evdev pcspkr sg button ext4 jbd2 btrfs blake2b_generic xor zlib_deflate raid6_pq zstd_compress sr_mod cdrom ata_generic ata_piix psmouse serio_raw i2c_piix4 i2c_smbus libata e1000g [ 1448.960874] CPU: 2 UID: 0 PID: 2614 Comm: kworker/2:1 Not tainted 6.14.0-rc1+ #78g [ 1448.960878] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014g [ 1448.960879] Workqueue: xfs-conv/sdb1 xfs_end_io [xfs]g [ 1448.960938] Call Trace:g [ 1448.960939] <TASK>g [ 1448.960940] dump_stack_lvl+0x4f/0x60g [ 1448.960953] bad_page+0x6f/0x100g [ 1448.960957] free_frozen_pages+0x471/0x640g [ 1448.960958] iomap_finish_ioend+0x196/0x3c0g [ 1448.960963] iomap_finish_ioends+0x83/0xc0g [ 1448.960964] xfs_end_ioend+0x64/0x140 [xfs]g [ 1448.961003] xfs_end_io+0x93/0xc0 [xfs]g [ 1448.961036] process_one_work+0x153/0x390g [ 1448.961044] worker_thread+0x2ab/0x3b0g [ 1448.961045] ? rescuer_thread+0x470/0x470gQuoted 8 lines
[ 1448.961047] kthread+0xf7/0x200g [ 1448.961048] ? kthread_use_mm+0xa0/0xa0g [ 1448.961049] ret_from_fork+0x2d/0x50g [ 1448.961053] ? kthread_use_mm+0xa0/0xa0g [ 1448.961054] ret_from_fork_asm+0x11/0x20g [ 1448.961058] </TASK>g [ 1448.961155] Disabling lock debugging due to kernel taintg [ 1448.969569] page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x3e pfn:0x112cb0gsame pfn, same struct page
Quoted 6 lines
[ 1448.970023] flags: 0x800000000000000e(referenced|uptodate|writeback|zone=2)g [ 1448.970651] raw: 800000000000000e dead000000000100 dead000000000122 0000000000000000g [ 1448.971222] raw: 000000000000003e 0000000000000000 00000000ffffffff 0000000000000000g [ 1448.971812] page dumped because: VM_BUG_ON_FOLIO(((unsigned int) folio_ref_count(folio) + 127u <= 127u))g [ 1448.972490] ------------[ cut here ]------------g [ 1448.972841] kernel BUG at ./include/linux/mm.h:1455!gthis is
folio_get()noticing refcount is 0, so a use-after free, because we already tried to free the page above.I'm not familiar with this code too much, but I suspect problem was introduced by commit
fb7d3bc414939("mm/filemap: drop streaming/uncached pages when writeback completes") and only (more) exposed here.so in
folio_end_writeback()we haveif (__folio_end_writeback(folio)) folio_wake_bit(folio, PG_writeback);but calling the
folio_end_dropbehind_write()doesn't depend on the result of__folio_end_writeback()
this seems rather suspiciousI think if
__folio_end_writeback()was true thenPG_writebackwould be cleared and thus we'd not see thePAGE_FLAGS_CHECK_AT_FREEfailure. Instead we do a prematurefolio_end_dropbehind_write()dropping a page ref and then the finalfolio_put()infolio_end_writeback()frees the page and splats on thePG_writeback. Then the folio is processed again in the following iteration ofiomap_finish_ioend()and splats on the refcount-already-zero.Quoted 4 lines
So I think
folio_end_dropbehind_write()should only be done when__folio_end_writeback()was true. Most likely even thefolio_test_clear_dropbehind()should be tied to that, or we clear it too early and then never act upon it later?Thanks for taking a look at this! I tried to reproduce this this morning and failed miserably. I then injected a delay for the above case, and it does indeed then trigger for me. So far, so good.
I agree with your analysis, we should only be doing the dropbehind for a non-zero return from
__folio_end_writeback(), and that includes thetest_and_clearto avoid dropping the drop-behind state. But we also need to check/clear this state pre__folio_end_writeback(), which then puts us in a spot where it needs to potentially be re-set. Which fails pretty racy...Quoted 2 lines
I'll ponder this a bit. Good thing fsx got
RWF_DONTCACHEsupport, or I suspect this would've taken a while to run into.Took a closer look... I may be smoking something good here, but I don't see what the __folio_end_writeback()() return value has to do with this at all. Regardless of what it returns, it should've cleared
PG_writeback, and in fact the only thing it returns is whether or not we had anyone waiting on it. Which should have _zero_ bearing on whether or not we can clear/invalidate the range.
Quoted 4 lines
To me, this smells more like a race of some sort, between dirty and invalidation. fsx does a lot of sub-page sized operations.
I'll poke a bit more...
I _think_ we're racing with the same folio being marked for writeback again. Al, can you try the below?
mm/filemap.c +1 -1
@@ -1604,7 +1604,7 @@ static void folio_end_dropbehind_write(struct folio *folio)* invalidation in that case. */ if (in_task () &&folio_trylock (folio)) { -if (folio->mapping) +if (folio->mapping && !folio_test_writeback (folio))folio_unmap_invalidate (folio->mapping, folio,0 );folio_unlock (folio); }
Signature · 1 line
Jens Axboe
Al Viro
maintainerQuoted 4 lines · Jens Axboe
Quoted 1 line
I'll poke a bit more...
I _think_ we're racing with the same folio being marked for writeback again. Al, can you try the below?
It seems to survive on top of v6.15^^
Quoted 7 lines · Al Viro
Quoted 4 lines · Jens Axboe
Quoted 1 line
I'll poke a bit more...
I _think_ we're racing with the same folio being marked for writeback again. Al, can you try the below?
It seems to survive on top of v6.15^^
Thanks for testing, Al! Assuming it goes without saying, but that's 6.15 with 478ad02d6844 reverted, right?
Signature · 1 line
Jens Axboe
Al Viro
maintainerQuoted 11 lines · Jens Axboe
Quoted 7 lines · Al Viro
Quoted 4 lines · Jens Axboe
Quoted 1 line
I'll poke a bit more...
I _think_ we're racing with the same folio being marked for writeback again. Al, can you try the below?
It seems to survive on top of v6.15^^
Thanks for testing, Al! Assuming it goes without saying, but that's 6.15 with
478ad02d6844reverted, right?
That's 6.15 without two last commits - 478ad02d6844 and the version bump ;-)
Quoted 8 lines · Al Viro
Quoted 7 lines · Jens Axboe
Quoted 6 lines · Al Viro
Quoted 4 lines · Jens Axboe
Quoted 1 line
I'll poke a bit more...
I _think_ we're racing with the same folio being marked for writeback again. Al, can you try the below?
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
It seems to survive on top of v6.15^^
Thanks for testing, Al! Assuming it goes without saying, but that's 6.15 with
478ad02d6844reverted, right?That's 6.15 without two last commits -
478ad02d6844and the version bump ;-)
OK good, I would've been confused it not, but never hurts to confirm...
FWIW, have a branch here:
https://git.kernel.dk/cgit/linux/log/?h=dontcache
with the read/write side patches, and finally the revert as well. There's a consolidation patch that can be done on top in terms of a cleanup, but figured it was better to keep that separate from the actual bug fix.
Signature · 1 line
Jens Axboe
Quoted 169 lines · Jens Axboe
Quoted 146 lines · Jens Axboe
Quoted 130 lines · Jens Axboe
Quoted 104 lines · Vlastimil Babka
Quoted 6 lines · Al Viro
Quoted 1 line · Al Viro
Breakage is still present in the current mainline ;-/
With
CONFIG_DEBUG_VMon top of pagealloc debugging:Quoted 8 lines
[ 1434.992817] run fstests generic/127 at 2025-05-25 11:46:11g [ 1448.956242] BUG: Bad page state in process kworker/2:1 pfn:112cb0g [ 1448.956846] page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x3e pfn:0x112cb0g [ 1448.957453] flags:0x800000000000000e(referenced|uptodate|writeback|zone=2)g
It doesn't like the writeback flag.
Quoted 37 lines
[ 1448.957863] raw: 800000000000000e dead000000000100 dead000000000122 0000000000000000g [ 1448.958303] raw: 000000000000003e 0000000000000000 00000000ffffffff 0000000000000000g [ 1448.958833] page dumped because: PAGE_FLAGS_CHECK_AT_FREE flag(s) setg [ 1448.959320] Modules linked in: xfs autofs4 fuse nfsd
auth_rpcgssnfs_aclnfs lockd grace sunrpc loop ecryptfs 9pnet_virtio 9pnet netfs evdev pcspkr sg button ext4 jbd2 btrfsblake2b_genericxorzlib_deflateraid6_pqzstd_compresssr_modcdromata_genericata_piixpsmouseserio_rawi2c_piix4i2c_smbuslibata e1000g[ 1448.960874] CPU: 2 UID: 0 PID: 2614 Comm: kworker/2:1 Not tainted 6.14.0-rc1+ #78g [ 1448.960878] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014g [ 1448.960879] Workqueue: xfs-conv/sdb1 xfs_end_io [xfs]g [ 1448.960938] Call Trace:g [ 1448.960939] <TASK>g [ 1448.960940] dump_stack_lvl+0x4f/0x60g [ 1448.960953] bad_page+0x6f/0x100g [ 1448.960957] free_frozen_pages+0x471/0x640g [ 1448.960958] iomap_finish_ioend+0x196/0x3c0g [ 1448.960963] iomap_finish_ioends+0x83/0xc0g [ 1448.960964] xfs_end_ioend+0x64/0x140 [xfs]g [ 1448.961003] xfs_end_io+0x93/0xc0 [xfs]g [ 1448.961036] process_one_work+0x153/0x390g [ 1448.961044] worker_thread+0x2ab/0x3b0g [ 1448.961045] ? rescuer_thread+0x470/0x470g [ 1448.961047] kthread+0xf7/0x200g [ 1448.961048] ? kthread_use_mm+0xa0/0xa0g [ 1448.961049] ret_from_fork+0x2d/0x50g [ 1448.961053] ? kthread_use_mm+0xa0/0xa0g [ 1448.961054] ret_from_fork_asm+0x11/0x20g [ 1448.961058] </TASK>g [ 1448.961155] Disabling lock debugging due to kernel taintg [ 1448.969569] page: refcount:0 mapcount:0Quoted 1 line
mapping:0000000000000000 index:0x3e pfn:0x112cb0g
same pfn, same struct page
Quoted 11 lines
[ 1448.970023] flags: 0x800000000000000e(referenced|uptodate|writeback|zone=2)g [ 1448.970651] raw: 800000000000000e dead000000000100 dead000000000122 0000000000000000g [ 1448.971222] raw: 000000000000003e 0000000000000000 00000000ffffffff 0000000000000000g [ 1448.971812] page dumped because: VM_BUG_ON_FOLIO(((unsigned int) folio_ref_count(folio) + 127u <= 127u))g [ 1448.972490] ------------[ cut here ]------------g [ 1448.972841] kernel BUG at ./include/linux/mm.h:1455!gthis is
folio_get()noticing refcount is 0, so a use-after free, because
we already tried to free the page above.I'm not familiar with this code too much, but I suspect problem was
introduced by commitfb7d3bc414939("mm/filemap: drop streaming/uncached
pages when writeback completes") and only (more) exposed here.so in
folio_end_writeback()we have
if (__folio_end_writeback(folio))folio_wake_bit(folio, PG_writeback);but calling the
folio_end_dropbehind_write()doesn't depend on the
result of__folio_end_writeback()
this seems rather suspiciousI think if
__folio_end_writeback()was true thenPG_writebackwould be
cleared and thus we'd not see thePAGE_FLAGS_CHECK_AT_FREEfailure.
Instead we do a prematurefolio_end_dropbehind_write()dropping a page
ref and then the finalfolio_put()infolio_end_writeback()frees the
page and splats on thePG_writeback. Then the folio is processed again
in the following iteration ofiomap_finish_ioend()and splats on the
refcount-already-zero.Quoted 6 lines
So I think
folio_end_dropbehind_write()should only be done when__folio_end_writeback()was true. Most likely even thefolio_test_clear_dropbehind()should be tied to that, or we clear it too
early and then never act upon it later?Thanks for taking a look at this! I tried to reproduce this this morning
and failed miserably. I then injected a delay for the above case, and it
does indeed then trigger for me. So far, so good.I agree with your analysis, we should only be doing the dropbehind for a
non-zero return from__folio_end_writeback(), and that includes thetest_and_clearto avoid dropping the drop-behind state. But we also need
to check/clear this state pre__folio_end_writeback(), which then puts
us in a spot where it needs to potentially be re-set. Which fails pretty
racy...Quoted 3 lines
I'll ponder this a bit. Good thing fsx got
RWF_DONTCACHEsupport, or I
suspect this would've taken a while to run into.Took a closer look... I may be smoking something good here, but I don't
see what the __folio_end_writeback()() return value has to do with this
at all. Regardless of what it returns, it should've clearedPG_writeback, and in fact the only thing it returns is whether or not we
had anyone waiting on it. Which should have _zero_ bearing on whether or
not we can clear/invalidate the range.Quoted 4 lines
To me, this smells more like a race of some sort, between dirty and invalidation. fsx does a lot of sub-page sized operations.
I'll poke a bit more...
I _think_ we're racing with the same folio being marked for writeback again. Al, can you try the below?
mm/filemap.c +1
-1@@ -1604,7 +1604,7 @@ static void folio_end_dropbehind_write(structfolio *folio) * invalidation in that case. */ if (in_task() && folio_trylock(folio)) { -if (folio->mapping) +if (folio->mapping && !folio_test_writeback (folio)) folio_unmap_invalidate(folio->mapping, folio,0 );
Quoted 3 lines
folio_unlock(folio);
}
I think we need to test for PG_dirty after retaking the folio lock as well. Nothing stops a second thread from redirtying the page once the folio lock is dropped, and while some filesystems may insist on waiting for PG_writeback before allowing redirtying to complete, that still ends up racing because folio_end_dropbehind_write() is called after the call to __folio_end_writeback().
Note that the same set of races can happen in filemap_end_dropbehind_read(), so we need the same set of checks after taking the folio lock there too. The existing checks are insufficient, since they only happen before taking the folio lock.
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 172 lines · Trond Myklebust
Quoted 171 lines · Jens Axboe
Quoted 146 lines · Jens Axboe
Quoted 130 lines · Jens Axboe
Quoted 104 lines · Vlastimil Babka
Quoted 6 lines · Al Viro
Quoted 1 line · Al Viro
Breakage is still present in the current mainline ;-/
With
CONFIG_DEBUG_VMon top of pagealloc debugging:Quoted 8 lines
[ 1434.992817] run fstests generic/127 at 2025-05-25 11:46:11g [ 1448.956242] BUG: Bad page state in process kworker/2:1 pfn:112cb0g [ 1448.956846] page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x3e pfn:0x112cb0g [ 1448.957453] flags:0x800000000000000e(referenced|uptodate|writeback|zone=2)g
It doesn't like the writeback flag.
Quoted 37 lines
[ 1448.957863] raw: 800000000000000e dead000000000100 dead000000000122 0000000000000000g [ 1448.958303] raw: 000000000000003e 0000000000000000 00000000ffffffff 0000000000000000g [ 1448.958833] page dumped because: PAGE_FLAGS_CHECK_AT_FREE flag(s) setg [ 1448.959320] Modules linked in: xfs autofs4 fuse nfsd
auth_rpcgssnfs_aclnfs lockd grace sunrpc loop ecryptfs 9pnet_virtio 9pnet netfs evdev pcspkr sg button ext4 jbd2 btrfsblake2b_genericxorzlib_deflateraid6_pqzstd_compresssr_modcdromata_genericata_piixpsmouseserio_rawi2c_piix4i2c_smbuslibata e1000g[ 1448.960874] CPU: 2 UID: 0 PID: 2614 Comm: kworker/2:1 Not tainted 6.14.0-rc1+ #78g [ 1448.960878] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014g [ 1448.960879] Workqueue: xfs-conv/sdb1 xfs_end_io [xfs]g [ 1448.960938] Call Trace:g [ 1448.960939] <TASK>g [ 1448.960940] dump_stack_lvl+0x4f/0x60g [ 1448.960953] bad_page+0x6f/0x100g [ 1448.960957] free_frozen_pages+0x471/0x640g [ 1448.960958] iomap_finish_ioend+0x196/0x3c0g [ 1448.960963] iomap_finish_ioends+0x83/0xc0g [ 1448.960964] xfs_end_ioend+0x64/0x140 [xfs]g [ 1448.961003] xfs_end_io+0x93/0xc0 [xfs]g [ 1448.961036] process_one_work+0x153/0x390g [ 1448.961044] worker_thread+0x2ab/0x3b0g [ 1448.961045] ? rescuer_thread+0x470/0x470g [ 1448.961047] kthread+0xf7/0x200g [ 1448.961048] ? kthread_use_mm+0xa0/0xa0g [ 1448.961049] ret_from_fork+0x2d/0x50g [ 1448.961053] ? kthread_use_mm+0xa0/0xa0g [ 1448.961054] ret_from_fork_asm+0x11/0x20g [ 1448.961058] </TASK>g [ 1448.961155] Disabling lock debugging due to kernel taintg [ 1448.969569] page: refcount:0 mapcount:0Quoted 1 line
mapping:0000000000000000 index:0x3e pfn:0x112cb0g
same pfn, same struct page
Quoted 11 lines
[ 1448.970023] flags: 0x800000000000000e(referenced|uptodate|writeback|zone=2)g [ 1448.970651] raw: 800000000000000e dead000000000100 dead000000000122 0000000000000000g [ 1448.971222] raw: 000000000000003e 0000000000000000 00000000ffffffff 0000000000000000g [ 1448.971812] page dumped because: VM_BUG_ON_FOLIO(((unsigned int) folio_ref_count(folio) + 127u <= 127u))g [ 1448.972490] ------------[ cut here ]------------g [ 1448.972841] kernel BUG at ./include/linux/mm.h:1455!gthis is
folio_get()noticing refcount is 0, so a use-after free, because
we already tried to free the page above.I'm not familiar with this code too much, but I suspect problem was
introduced by commitfb7d3bc414939("mm/filemap: drop streaming/uncached
pages when writeback completes") and only (more) exposed here.so in
folio_end_writeback()we haveif (__folio_end_writeback(folio)) folio_wake_bit(folio, PG_writeback);but calling the
folio_end_dropbehind_write()doesn't depend on the
result of__folio_end_writeback()
this seems rather suspiciousI think if
__folio_end_writeback()was true thenPG_writebackwould be
cleared and thus we'd not see thePAGE_FLAGS_CHECK_AT_FREEfailure.
Instead we do a prematurefolio_end_dropbehind_write()dropping a page
ref and then the finalfolio_put()infolio_end_writeback()frees the
page and splats on thePG_writeback. Then the folio is processed again
in the following iteration ofiomap_finish_ioend()and splats on the
refcount-already-zero.Quoted 6 lines
So I think
folio_end_dropbehind_write()should only be done when__folio_end_writeback()was true. Most likely even thefolio_test_clear_dropbehind()should be tied to that, or we clear it too
early and then never act upon it later?Thanks for taking a look at this! I tried to reproduce this this morning
and failed miserably. I then injected a delay for the above case, and it
does indeed then trigger for me. So far, so good.I agree with your analysis, we should only be doing the dropbehind for a
non-zero return from__folio_end_writeback(), and that includes thetest_and_clearto avoid dropping the drop-behind state. But we also need
to check/clear this state pre__folio_end_writeback(), which then puts
us in a spot where it needs to potentially be re-set. Which fails pretty
racy...Quoted 3 lines
I'll ponder this a bit. Good thing fsx got
RWF_DONTCACHEsupport, or I
suspect this would've taken a while to run into.Took a closer look... I may be smoking something good here, but I don't
see what the __folio_end_writeback()() return value has to do with this
at all. Regardless of what it returns, it should've clearedPG_writeback, and in fact the only thing it returns is whether or not we
had anyone waiting on it. Which should have _zero_ bearing on whether or
not we can clear/invalidate the range.Quoted 4 lines
To me, this smells more like a race of some sort, between dirty and invalidation. fsx does a lot of sub-page sized operations.
I'll poke a bit more...
I _think_ we're racing with the same folio being marked for writeback again. Al, can you try the below?
mm/filemap.c +1
-1@@ -1604,7 +1604,7 @@ static void folio_end_dropbehind_write(structfolio *folio) * invalidation in that case. */ if (in_task () &&folio_trylock (folio)) { -if (folio->mapping) +if (folio->mapping && !folio_test_writeback (folio))folio_unmap_invalidate (folio->mapping, folio,0 );mm/filemap.c +0
-0folio_unlock (folio); }
Quoted 8 lines
Quoted 1 line
I think we need to test for
PG_dirtyafter retaking the folio lock as well. Nothing stops a second thread from redirtying the page once the folio lock is dropped, and while some filesystems may insist on waiting forPG_writebackbefore allowing redirtying to complete, that still ends up racing becausefolio_end_dropbehind_write()is called after the call to__folio_end_writeback().
Agree, local version actually has both as well.
Quoted 4 lines
Note that the same set of races can happen in
filemap_end_dropbehind_read(), so we need the same set of checks after taking the folio lock there too. The existing checks are insufficient, since they only happen before taking the folio lock.
Ah good catch. I'll send out the patch tomorrow.
Signature · 1 line
Jens Axboe
Quoted 30 lines · Al Viro
generic/127 with xfstests built on debian-testing (trixie) ends up with assorted memory corruption; trace below is with
CONFIG_DEBUG_PAGEALLOCandCONFIG_DEBUG_PAGEALLOC_ENABLE_DEFAULTand it looks like a double free somewhere in iomap. Unfortunately, commit in question is just making xfs use the infrastructure built in earlier series - not that useful for isolating the breakage.[ 22.001529] run fstests generic/127 at 2025-05-25 04:13:23 [ 35.498573] BUG: Bad page state in process kworker/2:1 pfn:112ce9 [ 35.499260] page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x3e 9 [ 35.499764] flags: 0x800000000000000e(referenced|uptodate|writeback|zone=2) [ 35.500302] raw: 800000000000000e dead000000000100 dead000000000122 000000000 [ 35.500786] raw: 000000000000003e 0000000000000000 00000000ffffffff 000000000 [ 35.501248] page dumped because: PAGE_FLAGS_CHECK_AT_FREE flag(s) set [ 35.501624] Modules linked in: xfs autofs4 fuse nfsd auth_rpcgss nfs_acl nfs0 [ 35.503209] CPU: 2 UID: 0 PID: 85 Comm: kworker/2:1 Not tainted 6.14.0-rc1+ 7 [ 35.503211] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.164 [ 35.503212] Workqueue: xfs-conv/sdb1 xfs_end_io [xfs] [ 35.503279] Call Trace: [ 35.503281] <TASK> [ 35.503282] dump_stack_lvl+0x4f/0x60 [ 35.503296] bad_page+0x6f/0x100 [ 35.503300] free_frozen_pages+0x303/0x550 [ 35.503301] iomap_finish_ioend+0xf6/0x380 [ 35.503304] iomap_finish_ioends+0x83/0xc0 [ 35.503305] xfs_end_ioend+0x64/0x140 [xfs] [ 35.503342] xfs_end_io+0x93/0xc0 [xfs] [ 35.503378] process_one_work+0x153/0x390 [ 35.503382] worker_thread+0x2ab/0x3b0
Quoted 6 lines
It's 4:30am here, so I'm going to leave attempts to actually debug that thing until tomorrow; I do have a kvm where it's reliably reproduced within a few minutes, so if anyone comes up with patches, I'll be able to test them.
Breakage is still present in the current mainline ;-/
Hey Al,
Welll this certainly looks like the same report I made a month ago. I'll go run 6.15 final (with the #define RWF_DONTCACHE 0) overnight to confirm if that makes my problem go away. If these are one and the same bug, then thank you for finding a better reproducer! :)
https://lore.kernel.org/linux-fsdevel/20250416180837.GN25675@frogsfrogsfrogs/
--D
Quoted 38 lines · Darrick J. Wong
Quoted 30 lines · Al Viro
generic/127 with xfstests built on debian-testing (trixie) ends up with assorted memory corruption; trace below is with
CONFIG_DEBUG_PAGEALLOCandCONFIG_DEBUG_PAGEALLOC_ENABLE_DEFAULTand it looks like a double free somewhere in iomap. Unfortunately, commit in question is just making xfs use the infrastructure built in earlier series - not that useful for isolating the breakage.[ 22.001529] run fstests generic/127 at 2025-05-25 04:13:23 [ 35.498573] BUG: Bad page state in process kworker/2:1 pfn:112ce9 [ 35.499260] page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x3e 9 [ 35.499764] flags: 0x800000000000000e(referenced|uptodate|writeback|zone=2) [ 35.500302] raw: 800000000000000e dead000000000100 dead000000000122 000000000 [ 35.500786] raw: 000000000000003e 0000000000000000 00000000ffffffff 000000000 [ 35.501248] page dumped because: PAGE_FLAGS_CHECK_AT_FREE flag(s) set [ 35.501624] Modules linked in: xfs autofs4 fuse nfsd auth_rpcgss nfs_acl nfs0 [ 35.503209] CPU: 2 UID: 0 PID: 85 Comm: kworker/2:1 Not tainted 6.14.0-rc1+ 7 [ 35.503211] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.164 [ 35.503212] Workqueue: xfs-conv/sdb1 xfs_end_io [xfs] [ 35.503279] Call Trace: [ 35.503281] <TASK> [ 35.503282] dump_stack_lvl+0x4f/0x60 [ 35.503296] bad_page+0x6f/0x100 [ 35.503300] free_frozen_pages+0x303/0x550 [ 35.503301] iomap_finish_ioend+0xf6/0x380 [ 35.503304] iomap_finish_ioends+0x83/0xc0 [ 35.503305] xfs_end_ioend+0x64/0x140 [xfs] [ 35.503342] xfs_end_io+0x93/0xc0 [xfs] [ 35.503378] process_one_work+0x153/0x390 [ 35.503382] worker_thread+0x2ab/0x3b0Quoted 6 lines
It's 4:30am here, so I'm going to leave attempts to actually debug that thing until tomorrow; I do have a kvm where it's reliably reproduced within a few minutes, so if anyone comes up with patches, I'll be able to test them.
Breakage is still present in the current mainline ;-/
Quoted 8 lines
Hey Al,
Welll this certainly looks like the same report I made a month ago. I'll go run 6.15 final (with the #define
RWF_DONTCACHE0) overnight to confirm if that makes my problem go away. If these are one and the same bug, then thank you for finding a better reproducer! :)https://lore.kernel.org/linux-fsdevel/20250416180837.GN25675@frogsfrogsfrogs/
After a full QA run, 6.15 final passes fstests with flying colors. So I guess we now know the culprit. Will test the new RWF_DONTCACHE fixes whenever they appear in upstream.
--D
Quoted 46 lines · Darrick J. Wong
Quoted 45 lines · Darrick J. Wong
Quoted 30 lines · Al Viro
generic/127 with xfstests built on debian-testing (trixie) ends up with assorted memory corruption; trace below is with
CONFIG_DEBUG_PAGEALLOCandCONFIG_DEBUG_PAGEALLOC_ENABLE_DEFAULTand it looks like a double free somewhere in iomap. Unfortunately, commit in question is just making xfs use the infrastructure built in earlier series - not that useful for isolating the breakage.[ 22.001529] run fstests generic/127 at 2025-05-25 04:13:23 [ 35.498573] BUG: Bad page state in process kworker/2:1 pfn:112ce9 [ 35.499260] page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x3e 9 [ 35.499764] flags: 0x800000000000000e(referenced|uptodate|writeback|zone=2) [ 35.500302] raw: 800000000000000e dead000000000100 dead000000000122 000000000 [ 35.500786] raw: 000000000000003e 0000000000000000 00000000ffffffff 000000000 [ 35.501248] page dumped because: PAGE_FLAGS_CHECK_AT_FREE flag(s) set [ 35.501624] Modules linked in: xfs autofs4 fuse nfsd auth_rpcgss nfs_acl nfs0 [ 35.503209] CPU: 2 UID: 0 PID: 85 Comm: kworker/2:1 Not tainted 6.14.0-rc1+ 7 [ 35.503211] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.164 [ 35.503212] Workqueue: xfs-conv/sdb1 xfs_end_io [xfs] [ 35.503279] Call Trace: [ 35.503281] <TASK> [ 35.503282] dump_stack_lvl+0x4f/0x60 [ 35.503296] bad_page+0x6f/0x100 [ 35.503300] free_frozen_pages+0x303/0x550 [ 35.503301] iomap_finish_ioend+0xf6/0x380 [ 35.503304] iomap_finish_ioends+0x83/0xc0 [ 35.503305] xfs_end_ioend+0x64/0x140 [xfs] [ 35.503342] xfs_end_io+0x93/0xc0 [xfs] [ 35.503378] process_one_work+0x153/0x390 [ 35.503382] worker_thread+0x2ab/0x3b0Quoted 6 lines
It's 4:30am here, so I'm going to leave attempts to actually debug that thing until tomorrow; I do have a kvm where it's reliably reproduced within a few minutes, so if anyone comes up with patches, I'll be able to test them.
Breakage is still present in the current mainline ;-/
Hey Al,
Welll this certainly looks like the same report I made a month ago. I'll go run 6.15 final (with the #define
RWF_DONTCACHE0) overnight to confirm if that makes my problem go away. If these are one and the same bug, then thank you for finding a better reproducer! :)
Quoted 5 lines
After a full QA run, 6.15 final passes fstests with flying colors. So I guess we now know the culprit. Will test the new
RWF_DONTCACHEfixes whenever they appear in upstream.
Please do! Unfortunately I never saw your original report as I wasn't CC'ed on it, which I can't really fault anyone for as there was no reason to suspect it so far.
Signature · 1 line
Jens Axboe
Christian Brauner
maintainerQuoted 9 lines
Quoted 5 lines
After a full QA run, 6.15 final passes fstests with flying colors. So I guess we now know the culprit. Will test the new
RWF_DONTCACHEfixes whenever they appear in upstream.Please do! Unfortunately I never saw your original report as I wasn't CC'ed on it, which I can't really fault anyone for as there was no reason to suspect it so far.
I've just sent the pull request with the fixes a minute ago. Thanks for testing!