fs-next-20251103 reclaim lockdep splat
In trying to bisect the earlier reported transaction assertion failure, I hit this:
generic/476 run fstests generic/476 at 2025-11-05 20:16:46
XFS (vdb): Mounting V5 Filesystem 7f483353-a0f6-4710-9adc-4b72f25598f8 XFS (vdb): Ending clean mount
XFS (vdc): Mounting V5 Filesystem 47fa2f49-e8e1-4622-a62c-53ea07b3d714 XFS (vdc): Ending clean mount
======================================================
WARNING: possible circular locking dependency detected
6.18.0-rc4-ktest-00382-ga1e94de7fbd5 #116 Tainted: G W
------------------------------------------------------
kswapd0/111 is trying to acquire lock:ffff888102462418 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_ilock+0x14b/0x2b0
but task is already holding lock:ffffffff82710f00 (fs_reclaim){+.+.}-{0:0}, at: balance_pgdat+0x3e0/0x780
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (fs_reclaim){+.+.}-{0:0}:
fs_reclaim_acquire+0x67/0xa0
__kmalloc_cache_noprof+0x4d/0x500
iomap_fill_dirty_folios+0x6b/0xe0
xfs_buffered_write_iomap_begin+0xaee/0x1060
iomap_iter+0x1a1/0x4a0
iomap_zero_range+0xb0/0x420
xfs_zero_range+0x54/0x70
xfs_file_write_checks+0x21d/0x320
xfs_file_dio_write_unaligned+0x140/0x2b0
xfs_file_write_iter+0x22a/0x270
vfs_write+0x23f/0x540
ksys_write+0x6d/0x100
__x64_sys_write+0x1d/0x30
x64_sys_call+0x7d/0x1da0
do_syscall_64+0x6a/0x2e0
entry_SYSCALL_64_after_hwframe+0x76/0x7e-> #0 (&xfs_nondir_ilock_class){++++}-{4:4}:
__lock_acquire+0x15be/0x27d0
lock_acquire+0xb2/0x290
down_write_nested+0x2a/0xb0
xfs_ilock+0x14b/0x2b0
xfs_icwalk_ag+0x517/0xaf0
xfs_icwalk+0x46/0x80
xfs_reclaim_inodes_nr+0x8c/0xb0
xfs_fs_free_cached_objects+0x1d/0x30
super_cache_scan+0x178/0x1d0
do_shrink_slab+0x16d/0x6a0
shrink_slab+0x4cf/0x8c0
shrink_node+0x334/0x870
balance_pgdat+0x35f/0x780other info that might help us debug this:
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(fs_reclaim);
lock(&xfs_nondir_ilock_class);
lock(fs_reclaim); lock(&xfs_nondir_ilock_class);
*** DEADLOCK ***
2 locks held by kswapd0/111:
#0: ffffffff82710f00 (fs_reclaim){+.+.}-{0:0}, at: balance_pgdat+0x3e0/0x780 #1: ffff888132a800e0 (&type->s_umount_key#38){++++}-{4:4}, at: super_cache_scan+0x3d/0x1d0
stack backtrace: CPU: 2 UID: 0 PID: 111 Comm: kswapd0 Tainted: G W 6.18.0-rc4-ktest-00382-ga1e94de7fbd5 #116 NONE Tainted: [W]=WARN Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x63/0x90 dump_stack+0x14/0x1a print_circular_bug.cold+0x17e/0x1bb check_noncircular+0x123/0x140 __lock_acquire+0x15be/0x27d0 lock_acquire+0xb2/0x290 ? xfs_ilock+0x14b/0x2b0 ? find_held_lock+0x31/0x90 ? xfs_icwalk_ag+0x50a/0xaf0 down_write_nested+0x2a/0xb0 ? xfs_ilock+0x14b/0x2b0 xfs_ilock+0x14b/0x2b0 xfs_icwalk_ag+0x517/0xaf0 ? xfs_icwalk_ag+0x60/0xaf0 xfs_icwalk+0x46/0x80 xfs_reclaim_inodes_nr+0x8c/0xb0 xfs_fs_free_cached_objects+0x1d/0x30 super_cache_scan+0x178/0x1d0 do_shrink_slab+0x16d/0x6a0 shrink_slab+0x4cf/0x8c0 ? shrink_slab+0x2f1/0x8c0 shrink_node+0x334/0x870 balance_pgdat+0x35f/0x780 kswapd+0x1cf/0x3b0 ? __pfx_autoremove_wake_function+0x10/0x10 ? __pfx_kswapd+0x10/0x10 kthread+0x100/0x220 ? _raw_spin_unlock_irq+0x2b/0x40 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x18c/0x1d0 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK>
Quoted 43 lines · Matthew Wilcox
In trying to bisect the earlier reported transaction assertion failure, I hit this:
generic/476 run fstests generic/476 at 2025-11-05 20:16:46
XFS (vdb): Mounting V5 Filesystem 7f483353-a0f6-4710-9adc-4b72f25598f8 XFS (vdb): Ending clean mount
XFS (vdc): Mounting V5 Filesystem 47fa2f49-e8e1-4622-a62c-53ea07b3d714 XFS (vdc): Ending clean mount======================================================
WARNING: possible circular locking dependency detected
6.18.0-rc4-ktest-00382-ga1e94de7fbd5 #116 Tainted: G W
------------------------------------------------------
kswapd0/111 is trying to acquire lock:ffff888102462418(&xfs_nondir_ilock_class){++++}-{4:4}, at:xfs_ilock+0x14b/0x2b0but task is already holding lock:
ffffffff82710f00(fs_reclaim){+.+.}-{0:0}, at:balance_pgdat+0x3e0/0x780which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (fs_reclaim){+.+.}-{0:0}:
fs_reclaim_acquire+0x67/0xa0 __kmalloc_cache_noprof+0x4d/0x500 iomap_fill_dirty_folios+0x6b/0xe0 xfs_buffered_write_iomap_begin+0xaee/0x1060 iomap_iter+0x1a1/0x4a0 iomap_zero_range+0xb0/0x420 xfs_zero_range+0x54/0x70 xfs_file_write_checks+0x21d/0x320 xfs_file_dio_write_unaligned+0x140/0x2b0 xfs_file_write_iter+0x22a/0x270 vfs_write+0x23f/0x540 ksys_write+0x6d/0x100 __x64_sys_write+0x1d/0x30 x64_sys_call+0x7d/0x1da0 do_syscall_64+0x6a/0x2e0 entry_SYSCALL_64_after_hwframe+0x76/0x7e-> #0 (&xfs_nondir_ilock_class){++++}-{4:4}:
Quoted 13 lines
__lock_acquire+0x15be/0x27d0
lock_acquire+0xb2/0x290
down_write_nested+0x2a/0xb0
xfs_ilock+0x14b/0x2b0
xfs_icwalk_ag+0x517/0xaf0
xfs_icwalk+0x46/0x80
xfs_reclaim_inodes_nr+0x8c/0xb0
xfs_fs_free_cached_objects+0x1d/0x30
super_cache_scan+0x178/0x1d0
do_shrink_slab+0x16d/0x6a0
shrink_slab+0x4cf/0x8c0
shrink_node+0x334/0x870
balance_pgdat+0x35f/0x780As I said on #xfs: false positive on the inode lock.
Reclaim is running in GFP_KERNEL context, so it's allowed to lock unreferenced inodes.
The inodes that the allocation context holds locked are referenced inodes, so it cannot self-deadlock on the inode locks it holds because reclaim does not access or lock referenced inodes.
That being said, looking at this patch:
https://lore.kernel.org/linux-xfs/[email protected]/
I think the allocation that iomap_fill_dirty_folios() should probably be using mapping_gfp_constraint(mapping, GFP_KERNEL) rather than a hard coded GFP_KERNEL allocation. This is deep in the buffered write path and the xfs ILOCK is held when iomap_fill_dirty_folios() and it does folio lookups in that context.
Huh - that kinda feels like a lock order violation. ILOCK is not supposed to be held when we do page cache operations as the lock order defined by writback operations is folio lookup -> folio lock -> ILOCK.
So maybe this is a problem here, but not the one lockdep flagged...
Brian?
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 69 lines · Dave Chinner
Quoted 43 lines · Matthew Wilcox
In trying to bisect the earlier reported transaction assertion failure, I hit this:
generic/476 run fstests generic/476 at 2025-11-05 20:16:46
XFS (vdb): Mounting V5 Filesystem 7f483353-a0f6-4710-9adc-4b72f25598f8 XFS (vdb): Ending clean mount
XFS (vdc): Mounting V5 Filesystem 47fa2f49-e8e1-4622-a62c-53ea07b3d714 XFS (vdc): Ending clean mount======================================================
WARNING: possible circular locking dependency detected
6.18.0-rc4-ktest-00382-ga1e94de7fbd5 #116 Tainted: G W
------------------------------------------------------
kswapd0/111 is trying to acquire lock:ffff888102462418(&xfs_nondir_ilock_class){++++}-{4:4}, at:xfs_ilock+0x14b/0x2b0but task is already holding lock:
ffffffff82710f00(fs_reclaim){+.+.}-{0:0}, at:balance_pgdat+0x3e0/0x780which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (fs_reclaim){+.+.}-{0:0}:
fs_reclaim_acquire+0x67/0xa0 __kmalloc_cache_noprof+0x4d/0x500 iomap_fill_dirty_folios+0x6b/0xe0 xfs_buffered_write_iomap_begin+0xaee/0x1060 iomap_iter+0x1a1/0x4a0 iomap_zero_range+0xb0/0x420 xfs_zero_range+0x54/0x70 xfs_file_write_checks+0x21d/0x320 xfs_file_dio_write_unaligned+0x140/0x2b0 xfs_file_write_iter+0x22a/0x270 vfs_write+0x23f/0x540 ksys_write+0x6d/0x100 __x64_sys_write+0x1d/0x30 x64_sys_call+0x7d/0x1da0 do_syscall_64+0x6a/0x2e0 entry_SYSCALL_64_after_hwframe+0x76/0x7e-> #0 (&xfs_nondir_ilock_class){++++}-{4:4}:
Quoted 13 lines
__lock_acquire+0x15be/0x27d0 lock_acquire+0xb2/0x290 down_write_nested+0x2a/0xb0 xfs_ilock+0x14b/0x2b0 xfs_icwalk_ag+0x517/0xaf0 xfs_icwalk+0x46/0x80 xfs_reclaim_inodes_nr+0x8c/0xb0 xfs_fs_free_cached_objects+0x1d/0x30 super_cache_scan+0x178/0x1d0 do_shrink_slab+0x16d/0x6a0 shrink_slab+0x4cf/0x8c0 shrink_node+0x334/0x870 balance_pgdat+0x35f/0x780As I said on #xfs: false positive on the inode lock.
Reclaim is running in
GFP_KERNELcontext, so it's allowed to lock unreferenced inodes.The inodes that the allocation context holds locked are referenced inodes, so it cannot self-deadlock on the inode locks it holds because reclaim does not access or lock referenced inodes.
That being said, looking at this patch:
Quoted 9 lines
https://lore.kernel.org/linux-xfs/[email protected]/
I think the allocation that
iomap_fill_dirty_folios()should probably be usingmapping_gfp_constraint(mapping, GFP_KERNEL)rather than a hard codedGFP_KERNELallocation. This is deep in the buffered write path and the xfs ILOCK is held wheniomap_fill_dirty_folios()and it does folio lookups in that context.
There's an outstanding patch to nuke this allocation completely:
https://lore.kernel.org/linux-fsdevel/[email protected]/
This was also problematic for the ext4 on iomap WIP, so combined with the cleanup to use an iomap flag this seemed more elegant overall.
The patch series it's part of still needs work, but this one is just a standalone cleanup. If I can get some acks on it I'm happy to repost it separately to take this issue off the table..
Quoted 7 lines
Huh - that kinda feels like a lock order violation. ILOCK is not supposed to be held when we do page cache operations as the lock order defined by writback operations is folio lookup -> folio lock -> ILOCK.
So maybe this is a problem here, but not the one lockdep flagged...
Yeah.. but filemap_get_folios_dirty() is somewhat advisory. It is intended for use in this context, so only trylocks folios and those that it cannot lock, it just assumes are dirty||writeback and includes them in the batch for locking later (where later is defined as after the iomap callback returns where iomap typically does folio lookup/lock for buffered writes).
Brian
Quoted 7 lines
Brian?
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
Dave Chinner
Acked-byQuoted 6 lines · Brian Foster
Quoted 5 lines · Dave Chinner
Quoted 4 lines · Matthew Wilcox
In trying to bisect the earlier reported transaction assertion failure, I hit this:
generic/476 run fstests generic/476 at 2025-11-05 20:16:46
....
Quoted 21 lines
Quoted 11 lines
As I said on #xfs: false positive on the inode lock.
Reclaim is running in
GFP_KERNELcontext, so it's allowed to lock unreferenced inodes.The inodes that the allocation context holds locked are referenced inodes, so it cannot self-deadlock on the inode locks it holds because reclaim does not access or lock referenced inodes.
That being said, looking at this patch:
Quoted 9 lines
https://lore.kernel.org/linux-xfs/[email protected]/
I think the allocation that
iomap_fill_dirty_folios()should probably be usingmapping_gfp_constraint(mapping, GFP_KERNEL)rather than a hard codedGFP_KERNELallocation. This is deep in the buffered write path and the xfs ILOCK is held wheniomap_fill_dirty_folios()and it does folio lookups in that context.
Quoted 6 lines
There's an outstanding patch to nuke this allocation completely:
https://lore.kernel.org/linux-fsdevel/[email protected]/
This was also problematic for the ext4 on iomap WIP, so combined with the cleanup to use an iomap flag this seemed more elegant overall.
Ok, that looks like a good way to get rid of the allocation, so you can add
to it.
Quoted 9 lines
The patch series it's part of still needs work, but this one is just a standalone cleanup. If I can get some acks on it I'm happy to repost it separately to take this issue off the table..
Quoted 5 lines
Huh - that kinda feels like a lock order violation. ILOCK is not supposed to be held when we do page cache operations as the lock order defined by writback operations is folio lookup -> folio lock -> ILOCK.
Quoted 8 lines
Quoted 1 line
So maybe this is a problem here, but not the one lockdep flagged...
Yeah.. but
filemap_get_folios_dirty()is somewhat advisory. It is intended for use in this context, so only trylocks folios and those that it cannot lock, it just assumes are dirty||writeback and includes them in the batch for locking later (where later is defined as after the iomap callback returns where iomap typically does folio lookup/lock for buffered writes).
I guess I don't understand why this needs to be done under the ilock. I've read the patches, and it doesn't explain to me why we need to look up the pages under the ILOCK? The only constraint I see is trimming the extent map to match the end of a full fbatch array, but that doesn't seem like it can't be done by iomap itself.
Why do we need to do this dirty-folio batch lookup under the ILOCK instead of as a loop in iomap_zero_range() that grabs a fbatch for the unwritten mapping before calling iomap_zero_iter() repeatedly until we either hit the end of the mapping or there are no more dirty folios over the mapping?
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 36 lines · Dave Chinner
Quoted 6 lines · Brian Foster
Quoted 5 lines · Dave Chinner
Quoted 4 lines · Matthew Wilcox
In trying to bisect the earlier reported transaction assertion failure, I hit this:
generic/476 run fstests generic/476 at 2025-11-05 20:16:46
....
Quoted 21 lines
Quoted 11 lines
As I said on #xfs: false positive on the inode lock.
Reclaim is running in
GFP_KERNELcontext, so it's allowed to lock unreferenced inodes.The inodes that the allocation context holds locked are referenced inodes, so it cannot self-deadlock on the inode locks it holds because reclaim does not access or lock referenced inodes.
That being said, looking at this patch:
Quoted 9 lines
https://lore.kernel.org/linux-xfs/[email protected]/
I think the allocation that
iomap_fill_dirty_folios()should probably be usingmapping_gfp_constraint(mapping, GFP_KERNEL)rather than a hard codedGFP_KERNELallocation. This is deep in the buffered write path and the xfs ILOCK is held wheniomap_fill_dirty_folios()and it does folio lookups in that context.Quoted 6 lines
There's an outstanding patch to nuke this allocation completely:
https://lore.kernel.org/linux-fsdevel/[email protected]/
This was also problematic for the ext4 on iomap WIP, so combined with the cleanup to use an iomap flag this seemed more elegant overall.
Quoted 7 lines
Ok, that looks like a good way to get rid of the allocation, so you can add
Acked-by Dave Chinner <[email protected]>to it.
Thanks. I'll repost that one standalone since the rest of that series still needs some work.
Quoted 24 lines
Quoted 9 lines
The patch series it's part of still needs work, but this one is just a standalone cleanup. If I can get some acks on it I'm happy to repost it separately to take this issue off the table..
Quoted 5 lines
Huh - that kinda feels like a lock order violation. ILOCK is not supposed to be held when we do page cache operations as the lock order defined by writback operations is folio lookup -> folio lock -> ILOCK.
Quoted 8 lines
Quoted 1 line
So maybe this is a problem here, but not the one lockdep flagged...
Yeah.. but
filemap_get_folios_dirty()is somewhat advisory. It is intended for use in this context, so only trylocks folios and those that it cannot lock, it just assumes are dirty||writeback and includes them in the batch for locking later (where later is defined as after the iomap callback returns where iomap typically does folio lookup/lock for buffered writes).I guess I don't understand why this needs to be done under the ilock. I've read the patches, and it doesn't explain to me why we need to look up the pages under the ILOCK? The only constraint I see is trimming the extent map to match the end of a full fbatch array, but that doesn't seem like it can't be done by iomap itself.
Quoted 6 lines
Why do we need to do this dirty-folio batch lookup under the ILOCK instead of as a loop in
iomap_zero_range()that grabs a fbatch for the unwritten mapping before callingiomap_zero_iter()repeatedly until we either hit the end of the mapping or there are no more dirty folios over the mapping?
It doesn't need to be under ilock, just either under ilock or before iomap_begin to rule out problematic inconsistency (i.e. seeing a clean folio && unwritten mapping and racing with writeback). This was discussed at multiple points during review of the original batch series.
The longer term prospects for this are to potentially use it for multiple different operations, and therefore lift it into core iomap (not necessarily just zero range). The reason it's done this way to start was mainly just to manage scope and complexity. The use case is bounded, so it was easiest to work this in where it is and defer complexity associated with broader use to later.
That might sound pedantic, but when you consider things like dependent changes like incremental iter advancement, the impedence mismatch between some of these changes and the ext4-on-iomap WIP, and the little issues that fall out of that, this is all much easier to develop, support and improve with as little unnecessary complexity as possible. I think when some of the kinks the rest of that series is trying to address are worked out, that might be a reasonable point to consider such a lift.
Brian
Quoted 5 lines
-Dave.
Signature · 2 lines
Dave Chinner [email protected]