Backport of XFS umount hang fixes
v2 cover Backport of XFS umount hang fixes
Backport of this series:
https://lore.kernel.org/all/[email protected]/
I have added information in commit message upon conflict resolutions.
v2: Address Sasha's comments to v1 in the below mail:
https://lore.kernel.org/stable/2026-10-01-1-daily-reply-0012-re-xfs-umount-hang-backports@kernel.org/
xfstests_xfs testing results:
Generic: Ran: 001 to 771 (672 tests; gaps) Failures before changes: Count: 7 IDs: 363, 365, 633, 645, 696, 697, 753 Failures after changes: Count: 7 IDs: 363, 365, 633, 645, 696, 697, 753 No new success or failure after changes. XFS: Ran: 001 to 789 (311 tests; gaps) Failures before changes: Count: 1 IDs: 300 Failures after changes: Count: 1 IDs: 300 No new success or failure after changes.
Thanks,
Siddh
Dave Chinner (5):
xfs: xfs_ifree_cluster vs xfs_iflush_shutdown_abort deadlock
xfs: catch stale AGF/AGF metadata
xfs: rearrange code in xfs_buf_item.c
xfs: factor out stale buffer item completion
xfs: fix unmount hang with unflushable inodes stuck in the AIL
fs/xfs/libxfs/xfs_alloc.c | 46 ++++-- fs/xfs/libxfs/xfs_ialloc.c | 32 +++- fs/xfs/xfs_buf_item.c | 299 ++++++++++++++++++++++--------------- fs/xfs/xfs_buf_item.h | 3 +- fs/xfs/xfs_icache.c | 8 + fs/xfs/xfs_inode.c | 2 +- 6 files changed, 256 insertions(+), 134 deletions(-)
Signature · 1 line
2.53.0
PATCH RESEND v2 1/5 xfs: xfs_ifree_cluster vs xfs_iflush_shutdown_abort deadlock
From: Dave Chinner <[email protected]>
Lock order of xfs_ifree_cluster() is cluster buffer -> try ILOCK -> IFLUSHING, except for the last inode in the cluster that is triggering the free. In that case, the lock order is ILOCK -> cluster buffer -> IFLUSHING.
xfs_iflush_cluster() uses cluster buffer -> try ILOCK -> IFLUSHING, so this can safely run concurrently with xfs_ifree_cluster().
xfs_inode_item_precommit() uses ILOCK -> cluster buffer, but this cannot race with xfs_ifree_cluster() so being in a different order will not trigger a deadlock.
xfs_reclaim_inode() during a filesystem shutdown uses ILOCK -> IFLUSHING -> cluster buffer via xfs_iflush_shutdown_abort(), and this deadlocks against xfs_ifree_cluster() like so:
sysrq: Show Blocked State
task:kworker/10:37 state:D stack:12560 pid:276182 tgid:276182 ppid:2 flags:0x00004000
Workqueue: xfs-inodegc/dm-3 xfs_inodegc_worker Call Trace: <TASK> __schedule+0x650/0xb10 schedule+0x6d/0xf0 schedule_timeout+0x8b/0x180 schedule_timeout_uninterruptible+0x1e/0x30 xfs_ifree+0x326/0x730 xfs_inactive_ifree+0xcb/0x230 xfs_inactive+0x2c8/0x380 xfs_inodegc_worker+0xaa/0x180 process_scheduled_works+0x1d4/0x400 worker_thread+0x234/0x2e0 kthread+0x147/0x170 ret_from_fork+0x3e/0x50 ret_from_fork_asm+0x1a/0x30 </TASK> task:fsync-tester state:D stack:12160 pid:2255943 tgid:2255943 ppid:3988702 flags:0x00004006 Call Trace: <TASK> __schedule+0x650/0xb10 schedule+0x6d/0xf0 schedule_timeout+0x31/0x180 __down_common+0xbe/0x1f0 __down+0x1d/0x30 down+0x48/0x50 xfs_buf_lock+0x3d/0xe0 xfs_iflush_shutdown_abort+0x51/0x1e0 xfs_icwalk_ag+0x386/0x690 xfs_reclaim_inodes_nr+0x114/0x160 xfs_fs_free_cached_objects+0x19/0x20 super_cache_scan+0x17b/0x1a0 do_shrink_slab+0x180/0x350 shrink_slab+0xf8/0x430 drop_slab+0x97/0xf0 drop_caches_sysctl_handler+0x59/0xc0 proc_sys_call_handler+0x189/0x280 proc_sys_write+0x13/0x20 vfs_write+0x33d/0x3f0 ksys_write+0x7c/0xf0 __x64_sys_write+0x1b/0x30 x64_sys_call+0x271d/0x2ee0 do_syscall_64+0x68/0x130 entry_SYSCALL_64_after_hwframe+0x76/0x7e
We can't change the lock order of xfs_ifree_cluster() - XFS_ISTALE and XFS_IFLUSHING are serialised through to journal IO completion by the cluster buffer lock being held.
There's quite a few asserts in the code that check that XFS_ISTALE does not occur out of sync with buffer locking (e.g. in xfs_iflush_cluster). There's also a dependency on the inode log item being removed from the buffer before XFS_IFLUSHING is cleared, also with asserts that trigger on this.
Further, we don't have a requirement for the inode to be locked when completing or aborting inode flushing because all the inode state updates are serialised by holding the cluster buffer lock across the IO to completion.
We can't check for XFS_IRECLAIM in xfs_ifree_mark_inode_stale() and skip the inode, because there is no guarantee that the inode will be reclaimed. Hence it *must* be marked XFS_ISTALE regardless of whether reclaim is preparing to free that inode. Similarly, we can't check for IFLUSHING before locking the inode because that would result in dirty inodes not being marked with ISTALE in the event of racing with XFS_IRECLAIM.
Hence we have to address this issue from the xfs_reclaim_inode() side. It is clear that we cannot hold the inode locked here when calling xfs_iflush_shutdown_abort() because it is the inode->buffer lock order that causes the deadlock against xfs_ifree_cluster().
Hence we need to drop the ILOCK before aborting the inode in the shutdown case. Once we've aborted the inode, we can grab the ILOCK again and then immediately reclaim it as it is now guaranteed to be clean.
Note that dropping the ILOCK in xfs_reclaim_inode() means that it can now be locked by xfs_ifree_mark_inode_stale() and seen whilst in this state. This is safe because we have left the XFS_IFLUSHING flag on the inode and so xfs_ifree_mark_inode_stale() will simply set XFS_ISTALE and move to the next inode. An ASSERT check in this path needs to be tweaked to take into account this new shutdown interaction.
--- fs/xfs/xfs_icache.c | 8 ++++++++ fs/xfs/xfs_inode.c | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-)
fs/xfs/xfs_icache.c +8 -0
@@ -879,7 +879,15 @@ xfs_reclaim_inode(*/ if (xlog_is_shutdown (ip->i_mount->m_log)) {xfs_iunpin_wait (ip); +/* +* Avoid a ABBA deadlock on the inode cluster buffer vs +* concurrent xfs_ifree_cluster() trying to mark the inode +* stale. We don't need the inode locked to run the flush abort +* code, but the flush abort needs to lock the cluster buffer. +*/ +xfs_iunlock (ip,XFS_ILOCK_EXCL );xfs_iflush_shutdown_abort (ip); +xfs_ilock (ip,XFS_ILOCK_EXCL ); goto reclaim; }if (xfs_ipincount (ip))
fs/xfs/xfs_inode.c +1 -1
@@ -2235,7 +2235,7 @@ xfs_ifree_mark_inode_stale( iip = ip->i_itemp;if (__xfs_iflags_test (ip,XFS_IFLUSHING )) {ASSERT (!list_empty (&iip->ili_item.li_bio_list)); -ASSERT (iip->ili_last_fields); +ASSERT (iip->ili_last_fields ||xlog_is_shutdown (mp->m_log)); goto out_iunlock; }
Signature · 1 line
2.53.0
PATCH RESEND v2 2/5 xfs: catch stale AGF/AGF metadata
In reply to Siddh Raman PantFrom: Dave Chinner <[email protected]>
There is a race condition that can trigger in dmflakey fstests that can result in asserts in xfs_ialloc_read_agi() and xfs_alloc_read_agf() firing. The asserts look like this:
XFS: Assertion failed: pag->pagf_freeblks == be32_to_cpu(agf->agf_freeblks), file: fs/xfs/libxfs/xfs_alloc.c, line: 3440
.....
Call Trace: <TASK> xfs_alloc_read_agf+0x2ad/0x3a0 xfs_alloc_fix_freelist+0x280/0x720 xfs_alloc_vextent_prepare_ag+0x42/0x120 xfs_alloc_vextent_iterate_ags+0x67/0x260 xfs_alloc_vextent_start_ag+0xe4/0x1c0 xfs_bmapi_allocate+0x6fe/0xc90 xfs_bmapi_convert_delalloc+0x338/0x560 xfs_map_blocks+0x354/0x580 iomap_writepages+0x52b/0xa70 xfs_vm_writepages+0xd7/0x100 do_writepages+0xe1/0x2c0 __writeback_single_inode+0x44/0x340 writeback_sb_inodes+0x2d0/0x570 __writeback_inodes_wb+0x9c/0xf0 wb_writeback+0x139/0x2d0 wb_workfn+0x23e/0x4c0 process_scheduled_works+0x1d4/0x400 worker_thread+0x234/0x2e0 kthread+0x147/0x170 ret_from_fork+0x3e/0x50 ret_from_fork_asm+0x1a/0x30
I've seen the AGI variant from scrub running on the filesysetm after unmount failed due to systemd interference:
XFS: Assertion failed: pag->pagi_freecount == be32_to_cpu(agi->agi_freecount) || xfs_is_shutdown(pag->pag_mount), file: fs/xfs/libxfs/xfs_ialloc.c, line: 2804
.....
Call Trace: <TASK> xfs_ialloc_read_agi+0xee/0x150 xchk_perag_drain_and_lock+0x7d/0x240 xchk_ag_init+0x34/0x90 xchk_inode_xref+0x7b/0x220 xchk_inode+0x14d/0x180 xfs_scrub_metadata+0x2e2/0x510 xfs_ioc_scrub_metadata+0x62/0xb0 xfs_file_ioctl+0x446/0xbf0 __se_sys_ioctl+0x6f/0xc0 __x64_sys_ioctl+0x1d/0x30 x64_sys_call+0x1879/0x2ee0 do_syscall_64+0x68/0x130 ? exc_page_fault+0x62/0xc0 entry_SYSCALL_64_after_hwframe+0x76/0x7e
Essentially, it is the same problem. When _flakey_drop_and_remount() loads the drop-writes table, it makes all writes silently fail. Writes are reported to the fs as completed successfully, but they are not issued to the backing store. The filesystem sees the successful write completion and marks the metadata buffer clean and removes it from the AIL.
If this happens at the same time as memory pressure is occuring, the now-clean AGF and/or AGI buffers can be reclaimed from memory.
Shortly afterwards, but before _flakey_drop_and_remount() runs unmount, background writeback is kicked and it tries to allocate blocks for the dirty pages in memory. This then tries to access the AGF buffer we just turfed out of memory. It's not found, so it gets read in from disk.
This is all fine, except for the fact that the last writeback of the AGF did not actually reach disk. The AGF on disk is stale compared to the in-memory state held by the perag, and so they don't match and the assert fires.
Then other operations on that inode hang because the task was killed whilst holding inode locks. e.g:
Workqueue: xfs-conv/dm-12 xfs_end_io Call Trace: <TASK> __schedule+0x650/0xb10 schedule+0x6d/0xf0 schedule_preempt_disabled+0x15/0x30 rwsem_down_write_slowpath+0x31a/0x5f0 down_write+0x43/0x60 xfs_ilock+0x1a8/0x210 xfs_trans_alloc_inode+0x9c/0x240 xfs_iomap_write_unwritten+0xe3/0x300 xfs_end_ioend+0x90/0x130 xfs_end_io+0xce/0x100 process_scheduled_works+0x1d4/0x400 worker_thread+0x234/0x2e0 kthread+0x147/0x170 ret_from_fork+0x3e/0x50 ret_from_fork_asm+0x1a/0x30 </TASK>
and it's all down hill from there.
Memory pressure is one way to trigger this, another is to run "echo 3 > /proc/sys/vm/drop_caches" randomly while tests are running.
Regardless of how it is triggered, this effectively takes down the system once umount hangs because it's holding a sb->s_umount lock exclusive and now every sync(1) call gets stuck on it.
Fix this by replacing the asserts with a corruption detection check and a shutdown.
6.12:
Fixed conflict due to the following missing commit:
e9c4d8bfb26c ("xfs: factor out a generic xfs_group structure")
6.6:
Fixed conflict due to the following missing commit:
e45ea3645178 ("xfs: split the agf_roots and agf_levels arrays")--- fs/xfs/libxfs/xfs_alloc.c | 46 +++++++++++++++++++++++++++++--------- fs/xfs/libxfs/xfs_ialloc.c | 32 ++++++++++++++++++++++---- 2 files changed, 64 insertions(+), 14 deletions(-)
fs/xfs/libxfs/xfs_alloc.c +36 -10
@@ -26,6 +26,7 @@#include "xfs_ag.h" #include "xfs_ag_resv.h" #include "xfs_bmap.h" +#include "xfs_health.h" struct kmem_cache *xfs_extfree_item_cache;
@@ -3240,18 +3241,43 @@ xfs_alloc_read_agf(set_bit (XFS_AGSTATE_AGF_INIT , &pag->pag_opstate); } +#ifdef DEBUG - else if (!xfs_is_shutdown(pag->pag_mount)) { -ASSERT (pag->pagf_freeblks ==be32_to_cpu (agf->agf_freeblks)); -ASSERT (pag->pagf_btreeblks ==be32_to_cpu (agf->agf_btreeblks)); -ASSERT (pag->pagf_flcount ==be32_to_cpu (agf->agf_flcount)); -ASSERT (pag->pagf_longest ==be32_to_cpu (agf->agf_longest)); -ASSERT (pag->pagf_levels[XFS_BTNUM_BNOi] == -be32_to_cpu (agf->agf_levels[XFS_BTNUM_BNOi])); -ASSERT (pag->pagf_levels[XFS_BTNUM_CNTi] == -be32_to_cpu (agf->agf_levels[XFS_BTNUM_CNTi])); +/* +* It's possible for the AGF to be out of sync if the block device is +* silently dropping writes. This can happen in fstests with dmflakey +* enabled, which allows the buffer to be cleaned and reclaimed by +* memory pressure and then re-read from disk here. We will get a +* stale version of the AGF from disk, and nothing good can happen from +* here. Hence if we detect this situation, immediately shut down the +* filesystem. +* +* This can also happen if we are already in the middle of a forced +* shutdown, so don't bother checking if we are already shut down. +*/ +if (!xfs_is_shutdown (pag->pag_mount)) { +bool ok = true; + + ok &= pag->pagf_freeblks ==be32_to_cpu (agf->agf_freeblks); + ok &= pag->pagf_freeblks ==be32_to_cpu (agf->agf_freeblks); + ok &= pag->pagf_btreeblks ==be32_to_cpu (agf->agf_btreeblks); + ok &= pag->pagf_flcount ==be32_to_cpu (agf->agf_flcount); + ok &= pag->pagf_longest ==be32_to_cpu (agf->agf_longest); + ok &= pag->pagf_levels[XFS_BTNUM_BNOi] == +be32_to_cpu (agf->agf_levels[XFS_BTNUM_BNOi]); + ok &= pag->pagf_levels[XFS_BTNUM_CNTi] == +be32_to_cpu (agf->agf_levels[XFS_BTNUM_CNTi]); + +if (XFS_IS_CORRUPT (pag->pag_mount, !ok)) { +xfs_ag_mark_sick (pag,XFS_SICK_AG_AGF ); +xfs_trans_brelse (tp, agfbp); +xfs_force_shutdown (pag->pag_mount, +SHUTDOWN_CORRUPT_ONDISK ); +return -EFSCORRUPTED ; + } } -#endif +#endif /* DEBUG */ +if (agfbpp) *agfbpp = agfbp;else
fs/xfs/libxfs/xfs_ialloc.c +28 -4
@@ -27,6 +27,7 @@#include "xfs_log.h" #include "xfs_rmap.h" #include "xfs_ag.h" +#include "xfs_health.h" /* * Lookup a record by ino in the btree given by cur.
@@ -2634,12 +2635,35 @@ xfs_ialloc_read_agi(set_bit (XFS_AGSTATE_AGI_INIT , &pag->pag_opstate); } +#ifdef DEBUG /* -* It's possible for these to be out of sync if -* we are in the middle of a forced shutdown. +* It's possible for the AGF to be out of sync if the block device is +* silently dropping writes. This can happen in fstests with dmflakey +* enabled, which allows the buffer to be cleaned and reclaimed by +* memory pressure and then re-read from disk here. We will get a +* stale version of the AGF from disk, and nothing good can happen from +* here. Hence if we detect this situation, immediately shut down the +* filesystem. +* +* This can also happen if we are already in the middle of a forced +* shutdown, so don't bother checking if we are already shut down. */ -ASSERT (pag->pagi_freecount ==be32_to_cpu (agi->agi_freecount) || -xfs_is_shutdown (pag->pag_mount)); +if (!xfs_is_shutdown (pag->pag_mount)) { +bool ok = true; + + ok &= pag->pagi_freecount ==be32_to_cpu (agi->agi_freecount); + ok &= pag->pagi_count ==be32_to_cpu (agi->agi_count); + +if (XFS_IS_CORRUPT (pag->pag_mount, !ok)) { +xfs_ag_mark_sick (pag,XFS_SICK_AG_AGI ); +xfs_trans_brelse (tp, agibp); +xfs_force_shutdown (pag->pag_mount, +SHUTDOWN_CORRUPT_ONDISK ); +return -EFSCORRUPTED ; + } + } +#endif /* DEBUG */ +if (agibpp) *agibpp = agibp;else
Signature · 1 line
2.53.0
PATCH RESEND v2 3/5 xfs: rearrange code in xfs_buf_item.c
In reply to Siddh Raman PantFrom: Dave Chinner <[email protected]>
The code to initialise, release and free items is all the way down the bottom of the file. Upcoming fixes need to these functions earlier in the file, so move them to the top.
There is one code change in this move - the parameter to xfs_buf_item_relse() is changed from the xfs_buf to the xfs_buf_log_item - the thing that the function is releasing.
6.6:
Fixed conflict due to the following missing commits:
[Continuous range: 10634530f7ba^..d4c75a1b40cd]
d4c75a1b40cd ("xfs: convert remaining kmem_free() to kfree()")
49292576136f ("xfs: convert kmem_free() for kvmalloc users to kvfree()")
afdc115559c5 ("xfs: move kmem_to_page()")
f078d4ea8276 ("xfs: convert kmem_alloc() to kmalloc()")
10634530f7ba ("xfs: convert kmem_zalloc() to kzalloc()")Removed ASSERT in favour of existing atomic_read-and-return in xfs_buf_item_relse() due to the following missing commit:
3762113b597f ("xfs: don't lose solo superblock counter update transactions")
--- fs/xfs/xfs_buf_item.c | 118 +++++++++++++++++++++--------------------- fs/xfs/xfs_buf_item.h | 1 - 2 files changed, 60 insertions(+), 59 deletions(-)
fs/xfs/xfs_buf_item.c +60 -58
@@ -32,6 +32,63 @@ static inline struct xfs_buf_log_item *BUF_ITEM(struct xfs_log_item *lip) return container_of(lip, struct xfs_buf_log_item, bli_item); } +static void +xfs_buf_item_get_format ( +struct xfs_buf_log_item *bip, +int count) +{ +ASSERT (bip->bli_formats ==NULL ); + bip->bli_format_count = count; + +if (count ==1 ) { + bip->bli_formats = &bip->__bli_format; +return ; + } + + bip->bli_formats =kmem_zalloc (count *sizeof (struct xfs_buf_log_format ), +0 ); +} + +static void +xfs_buf_item_free_format ( +struct xfs_buf_log_item *bip) +{ +if (bip->bli_formats != &bip->__bli_format) { +kmem_free (bip->bli_formats); + bip->bli_formats =NULL ; + } +} + +static void +xfs_buf_item_free ( +struct xfs_buf_log_item *bip) +{ +xfs_buf_item_free_format (bip); +kmem_free (bip->bli_item.li_lv_shadow); +kmem_cache_free (xfs_buf_item_cache, bip); +} + +/* +* xfs_buf_item_relse() is called when the buf log item is no longer needed. +*/ +static void +xfs_buf_item_relse ( +struct xfs_buf_log_item *bip) +{ +struct xfs_buf *bp = bip->bli_buf; + +trace_xfs_buf_item_relse (bp, _RET_IP_); + +ASSERT (!test_bit (XFS_LI_IN_AIL , &bip->bli_item.li_flags)); + +if (atomic_read (&bip->bli_refcount)) +return ; + + bp->b_log_item =NULL ; +xfs_buf_rele (bp); +xfs_buf_item_free (bip); +} +/* Is this log iovec plausibly large enough to contain the buffer log format? */ bool xfs_buf_log_check_iovec(
@@ -563,7 +620,7 @@ xfs_buf_item_unpin(ASSERT (list_empty (&bp->b_li_list)); }else {xfs_trans_ail_delete (lip,SHUTDOWN_LOG_IO_ERROR ); -xfs_buf_item_relse (bp); +xfs_buf_item_relse (bip);ASSERT (bp->b_log_item ==NULL ); }xfs_buf_relse (bp);
@@ -673,7 +730,7 @@ xfs_buf_item_put(*/ if (aborted)xfs_trans_ail_delete (lip,0 ); -xfs_buf_item_relse (bip->bli_buf); +xfs_buf_item_relse (bip);return true; }
@@ -824,33 +881,6 @@ static const struct xfs_item_ops xfs_buf_item_ops = { .iop_push = xfs_buf_item_push, }; -STATIC void -xfs_buf_item_get_format ( -struct xfs_buf_log_item *bip, -int count) -{ -ASSERT (bip->bli_formats ==NULL ); - bip->bli_format_count = count; - -if (count ==1 ) { - bip->bli_formats = &bip->__bli_format; -return ; - } - - bip->bli_formats =kmem_zalloc (count *sizeof (struct xfs_buf_log_format ), -0 ); -} - -STATIC void -xfs_buf_item_free_format ( -struct xfs_buf_log_item *bip) -{ -if (bip->bli_formats != &bip->__bli_format) { -kmem_free (bip->bli_formats); - bip->bli_formats =NULL ; - } -} -/* * Allocate a new buf log item to go with the given buffer. * Set the buffer's b_log_item field to point to the new
@@ -1072,34 +1102,6 @@ xfs_buf_item_dirty_format(return false; } -STATIC void -xfs_buf_item_free ( -struct xfs_buf_log_item *bip) -{ -xfs_buf_item_free_format (bip); -kmem_free (bip->bli_item.li_lv_shadow); -kmem_cache_free (xfs_buf_item_cache, bip); -} - -/* -* xfs_buf_item_relse() is called when the buf log item is no longer needed. -*/ -void -xfs_buf_item_relse ( -struct xfs_buf *bp) -{ -struct xfs_buf_log_item *bip = bp->b_log_item; - -trace_xfs_buf_item_relse (bp, _RET_IP_); -ASSERT (!test_bit (XFS_LI_IN_AIL , &bip->bli_item.li_flags)); - -if (atomic_read (&bip->bli_refcount)) -return ; - bp->b_log_item =NULL ; -xfs_buf_rele (bp); -xfs_buf_item_free (bip); -} -void xfs_buf_item_done (struct xfs_buf *bp)
@@ -1119,5 +1121,5 @@ xfs_buf_item_done(xfs_trans_ail_delete (&bp->b_log_item->bli_item, (bp->b_flags & _XBF_LOGRECOVERY) ?0 :SHUTDOWN_CORRUPT_INCORE ); -xfs_buf_item_relse (bp); +xfs_buf_item_relse (bp->b_log_item); }
fs/xfs/xfs_buf_item.h +0 -1
@@ -49,7 +49,6 @@ struct xfs_buf_log_item {int xfs_buf_item_init (struct xfs_buf *,struct xfs_mount *);void xfs_buf_item_done (struct xfs_buf *bp); -void xfs_buf_item_relse (struct xfs_buf *);bool xfs_buf_item_put (struct xfs_buf_log_item *);void xfs_buf_item_log (struct xfs_buf_log_item *,uint ,uint );bool xfs_buf_item_dirty_format (struct xfs_buf_log_item *);
Signature · 1 line
2.53.0
PATCH RESEND v2 4/5 xfs: factor out stale buffer item completion
In reply to Siddh Raman PantFrom: Dave Chinner <[email protected]>
The stale buffer item completion handling is currently only done from BLI unpinning. We need to perform this function from where-ever the last reference to the BLI is dropped, so first we need to factor this code out into a helper.
--- fs/xfs/xfs_buf_item.c | 60 ++++++++++++++++++++++++++----------------- 1 file changed, 37 insertions(+), 23 deletions(-)
fs/xfs/xfs_buf_item.c +37 -23
@@ -541,6 +541,42 @@ xfs_buf_item_pin(atomic_inc (&bip->bli_buf->b_pin_count); } +/* +* For a stale BLI, process all the necessary completions that must be +* performed when the final BLI reference goes away. The buffer will be +* referenced and locked here - we return to the caller with the buffer still +* referenced and locked for them to finalise processing of the buffer. +*/ +static void +xfs_buf_item_finish_stale ( +struct xfs_buf_log_item *bip) +{ +struct xfs_buf *bp = bip->bli_buf; +struct xfs_log_item *lip = &bip->bli_item; + +ASSERT (bip->bli_flags &XFS_BLI_STALE ); +ASSERT (xfs_buf_islocked (bp)); +ASSERT (bp->b_flags &XBF_STALE ); +ASSERT (bip->__bli_format.blf_flags &XFS_BLF_CANCEL ); +ASSERT (list_empty (&lip->li_trans)); +ASSERT (!bp->b_transp); + +if (bip->bli_flags &XFS_BLI_STALE_INODE ) { +xfs_buf_item_done (bp); +xfs_buf_inode_iodone (bp); +ASSERT (list_empty (&bp->b_li_list)); +return ; + } + +/* +* We may or may not be on the AIL here, xfs_trans_ail_delete() will do +* the right thing regardless of the situation in which we are called. +*/ +xfs_trans_ail_delete (lip,SHUTDOWN_LOG_IO_ERROR ); +xfs_buf_item_relse (bip); +ASSERT (bp->b_log_item ==NULL ); +} +/* * This is called to unpin the buffer associated with the buf log item which was * previously pinned with a call to xfs_buf_item_pin(). We enter this function
@@ -590,13 +626,6 @@ xfs_buf_item_unpin( }if (stale) { -ASSERT (bip->bli_flags &XFS_BLI_STALE ); -ASSERT (xfs_buf_islocked (bp)); -ASSERT (bp->b_flags &XBF_STALE ); -ASSERT (bip->__bli_format.blf_flags &XFS_BLF_CANCEL ); -ASSERT (list_empty (&lip->li_trans)); -ASSERT (!bp->b_transp); -trace_xfs_buf_item_unpin_stale (bip);/*
@@ -607,22 +636,7 @@ xfs_buf_item_unpin(* processing is complete. */ xfs_buf_rele (bp); - -/* -* If we get called here because of an IO error, we may or may -* not have the item on the AIL. xfs_trans_ail_delete() will -* take care of that situation. xfs_trans_ail_delete() drops -* the AIL lock. -*/ -if (bip->bli_flags &XFS_BLI_STALE_INODE ) { -xfs_buf_item_done (bp); -xfs_buf_inode_iodone (bp); -ASSERT (list_empty (&bp->b_li_list)); - }else { -xfs_trans_ail_delete (lip,SHUTDOWN_LOG_IO_ERROR ); -xfs_buf_item_relse (bip); -ASSERT (bp->b_log_item ==NULL ); - } +xfs_buf_item_finish_stale (bip);xfs_buf_relse (bp);return ; }
Signature · 1 line
2.53.0
PATCH RESEND v2 5/5 xfs: fix unmount hang with unflushable inodes stuck in the AIL
In reply to Siddh Raman PantFrom: Dave Chinner <[email protected]>
Unmount of a shutdown filesystem can hang with stale inode cluster buffers in the AIL like so:
[95964.140623] Call Trace: [95964.144641] __schedule+0x699/0xb70 [95964.154003] schedule+0x64/0xd0 [95964.156851] xfs_ail_push_all_sync+0x9b/0xf0 [95964.164816] xfs_unmount_flush_inodes+0x41/0x70 [95964.168698] xfs_unmountfs+0x7f/0x170 [95964.171846] xfs_fs_put_super+0x3b/0x90 [95964.175216] generic_shutdown_super+0x77/0x160 [95964.178060] kill_block_super+0x1b/0x40 [95964.180553] xfs_kill_sb+0x12/0x30 [95964.182796] deactivate_locked_super+0x38/0x100 [95964.185735] deactivate_super+0x41/0x50 [95964.188245] cleanup_mnt+0x9f/0x160 [95964.190519] __cleanup_mnt+0x12/0x20 [95964.192899] task_work_run+0x89/0xb0 [95964.195221] resume_user_mode_work+0x4f/0x60 [95964.197931] syscall_exit_to_user_mode+0x76/0xb0 [95964.201003] do_syscall_64+0x74/0x130
$ pstree -N mnt |grep umount |-check-parallel---nsexec---run_test.sh---753---umount
It always seems to be generic/753 that triggers this, and repeating a quick group test run triggers it every 10-15 iterations. Hence it generally triggers once up every 30-40 minutes of test time. just running generic/753 by itself or concurrently with a limited group of tests doesn't reproduce this issue at all.
Tracing on a hung system shows the AIL repeating every 50ms a log force followed by an attempt to push pinned, aborted inodes from the AIL (trimmed for brevity):
xfs_log_force: lsn 0x1c caller xfsaild+0x18e xfs_log_force: lsn 0x0 caller xlog_cil_flush+0xbd xfs_log_force: lsn 0x1c caller xfs_log_force+0x77 xfs_ail_pinned: lip 0xffff88826014afa0 lsn 1/37472 type XFS_LI_INODE flags IN_AIL|ABORTED xfs_ail_pinned: lip 0xffff88814000a708 lsn 1/37472 type XFS_LI_INODE flags IN_AIL|ABORTED xfs_ail_pinned: lip 0xffff88810b850c80 lsn 1/37472 type XFS_LI_INODE flags IN_AIL|ABORTED xfs_ail_pinned: lip 0xffff88810b850af0 lsn 1/37472 type XFS_LI_INODE flags IN_AIL|ABORTED xfs_ail_pinned: lip 0xffff888165cf0a28 lsn 1/37472 type XFS_LI_INODE flags IN_AIL|ABORTED xfs_ail_pinned: lip 0xffff88810b850bb8 lsn 1/37472 type XFS_LI_INODE flags IN_AIL|ABORTED
....
The inode log items are marked as aborted, which means that either:
a) a transaction commit has occurred, seen an error or shutdown, and called xfs_trans_free_items() to abort the items. This should happen before any pinning of log items occurs.
or
b) a dirty transaction has been cancelled. This should also happen before any pinning of log items occurs.
or
c) AIL insertion at journal IO completion is marked as aborted. In this case, the log item is pinned by the CIL until journal IO completes and hence needs to be unpinned. This is then done after the ->iop_committed() callback is run, so the pin count should be balanced correctly.
Yet none of these seemed to be occurring. Further tracing indicated this:
d) Shutdown during CIL pushing resulting in log item completion being called from checkpoint abort processing. Items are unpinned and released without serialisation against each other, journal IO completion or transaction commit completion.
In this case, we may still have a transaction commit in flight that holds a reference to a xfs_buf_log_item (BLI) after CIL insertion. e.g. a synchronous transaction will flush the CIL before the transaction is torn down. The concurrent CIL push then aborts insertion it and drops the commit/AIL reference to the BLI. This can leave the transaction commit context with the last reference to the BLI which is dropped here:
xfs_trans_free_items()
->iop_release
xfs_buf_item_release
xfs_buf_item_put
if (XFS_LI_ABORTED)
xfs_trans_ail_delete
xfs_buf_item_relse()
Unlike the journal completion ->iop_unpin path, this path does not run stale buffer completion process when it drops the last reference, hence leaving the stale inodes attached to the buffer sitting the AIL. There are no other references to those inodes, so there is no other mechanism to remove them from the AIL. Hence unmount hangs.
The buffer lock context for stale buffers is passed to the last BLI reference. This is normally the last BLI unpin on journal IO completion. The unpin then processes the stale buffer completion and releases the buffer lock. However, if the final unpin from journal IO completion (or CIL push abort) does not hold the last reference to the BLI, there -must- still be a transaction context that references the BLI, and so that context must perform the stale buffer completion processing before the buffer is unlocked and the BLI torn down.
The fix for this is to rework the xfs_buf_item_relse() path to run stale buffer completion processing if it drops the last reference to the BLI. We still hold the buffer locked, so the buffer owner and lock context is the same as if we passed the BLI and buffer to the ->iop_unpin() context to finish stale process on journal commit.
However, we have to be careful here. In a shutdown state, we can be freeing dirty BLIs from xfs_buf_item_put() via xfs_trans_brelse() and xfs_trans_bdetach(). The existing code handles this case by considering shutdown state as "aborted", but in doing so largely masks the failure to clean up stale BLI state from the xfs_buf_item_relse() path. i.e regardless of the shutdown state and whether the item is in the AIL, we must finish the stale buffer cleanup if we are are dropping the last BLI reference from the ->iop_relse path in transaction commit context.
--- fs/xfs/xfs_buf_item.c | 123 +++++++++++++++++++++++++++++------------- fs/xfs/xfs_buf_item.h | 2 +- 2 files changed, 87 insertions(+), 38 deletions(-)
fs/xfs/xfs_buf_item.c +86 -37
@@ -709,43 +709,42 @@ xfs_buf_item_push(* Drop the buffer log item refcount and take appropriate action. This helper * determines whether the bli must be freed or not, since a decrement to zero * does not necessarily mean the bli is unused. -* -* Return true if the bli is freed, false otherwise. */ -bool +void xfs_buf_item_put (struct xfs_buf_log_item *bip) { -struct xfs_log_item *lip = &bip->bli_item; -bool aborted; -bool dirty; + +ASSERT (xfs_buf_islocked (bip->bli_buf));/* drop the bli ref and return if it wasn't the last one */ if (!atomic_dec_and_test (&bip->bli_refcount)) -return false; +return ; + +/* If the BLI is in the AIL, then it is still dirty and in use */ +if (test_bit (XFS_LI_IN_AIL , &bip->bli_item.li_flags)) { +ASSERT (bip->bli_flags &XFS_BLI_DIRTY ); +return ; + }/* -* We dropped the last ref and must free the item if clean or aborted. -* If the bli is dirty and non-aborted, the buffer was clean in the -* transaction but still awaiting writeback from previous changes. In -* that case, the bli is freed on buffer writeback completion. +* In shutdown conditions, we can be asked to free a dirty BLI that +* isn't in the AIL. This can occur due to a checkpoint aborting a BLI +* instead of inserting it into the AIL at checkpoint IO completion. If +* there's another bli reference (e.g. a btree cursor holds a clean +* reference) and it is released via xfs_trans_brelse(), we can get here +* with that aborted, dirty BLI. In this case, it is safe to free the +* dirty BLI immediately, as it is not in the AIL and there are no +* other references to it. +* +* We should never get here with a stale BLI via that path as +* xfs_trans_brelse() specifically holds onto stale buffers rather than +* releasing them. */ - aborted =test_bit (XFS_LI_ABORTED , &lip->li_flags) || -xlog_is_shutdown (lip->li_log); - dirty = bip->bli_flags &XFS_BLI_DIRTY ; -if (dirty && !aborted) -return false; - -/* -* The bli is aborted or clean. An aborted item may be in the AIL -* regardless of dirty state. For example, consider an aborted -* transaction that invalidated a dirty bli and cleared the dirty -* state. -*/ -if (aborted) -xfs_trans_ail_delete (lip,0 ); +ASSERT (!(bip->bli_flags &XFS_BLI_DIRTY ) || +test_bit (XFS_LI_ABORTED , &bip->bli_item.li_flags)); +ASSERT (!(bip->bli_flags &XFS_BLI_STALE ));xfs_buf_item_relse (bip); -return true; }/*
@@ -766,6 +765,15 @@ xfs_buf_item_put(* if necessary but do not unlock the buffer. This is for support of * xfs_trans_bhold(). Make sure the XFS_BLI_HOLD field is cleared if we don't * free the item. +* +* If the XFS_BLI_STALE flag is set, the last reference to the BLI *must* +* perform a completion abort of any objects attached to the buffer for IO +* tracking purposes. This generally only happens in shutdown situations, +* normally xfs_buf_item_unpin() will drop the last BLI reference and perform +* completion processing. However, because transaction completion can race with +* checkpoint completion during a shutdown, this release context may end up +* being the last active reference to the BLI and so needs to perform this +* cleanup. */ STATIC void xfs_buf_item_release(
@@ -773,18 +781,19 @@ xfs_buf_item_release( {struct xfs_buf_log_item *bip =BUF_ITEM (lip);struct xfs_buf *bp = bip->bli_buf; -bool released;bool hold = bip->bli_flags &XFS_BLI_HOLD ;bool stale = bip->bli_flags &XFS_BLI_STALE ; -#if defined(DEBUG ) || defined(XFS_WARN ) -bool ordered = bip->bli_flags &XFS_BLI_ORDERED ; -bool dirty = bip->bli_flags &XFS_BLI_DIRTY ;bool aborted =test_bit (XFS_LI_ABORTED , &lip->li_flags); +bool dirty = bip->bli_flags &XFS_BLI_DIRTY ; +#if defined(DEBUG ) || defined(XFS_WARN ) +bool ordered = bip->bli_flags &XFS_BLI_ORDERED ;#endif trace_xfs_buf_item_release (bip); +ASSERT (xfs_buf_islocked (bp)); +/* * The bli dirty state should match whether the blf has logged segments * except for ordered buffers, where only the bli should be dirty.
@@ -800,16 +809,56 @@ xfs_buf_item_release( bp->b_transp =NULL ; bip->bli_flags &= ~(XFS_BLI_LOGGED |XFS_BLI_HOLD |XFS_BLI_ORDERED ); +/* If there are other references, then we have nothing to do. */ +if (!atomic_dec_and_test (&bip->bli_refcount)) + goto out_release; +/* -* Unref the item and unlock the buffer unless held or stale. Stale -* buffers remain locked until final unpin unless the bli is freed by -* the unref call. The latter implies shutdown because buffer -* invalidation dirties the bli and transaction. +* Stale buffer completion frees the BLI, unlocks and releases the +* buffer. Neither the BLI or buffer are safe to reference after this +* call, so there's nothing more we need to do here. +* +* If we get here with a stale buffer and references to the BLI remain, +* we must not unlock the buffer as the last BLI reference owns lock +* context, not us. */ - released =xfs_buf_item_put (bip); -if (hold || (stale && !released)) +if (stale) { +xfs_buf_item_finish_stale (bip); +xfs_buf_relse (bp); +ASSERT (!hold); +return ; + } + +/* +* Dirty or clean, aborted items are done and need to be removed from +* the AIL and released. This frees the BLI, but leaves the buffer +* locked and referenced. +*/ +if (aborted ||xlog_is_shutdown (lip->li_log)) { +ASSERT (list_empty (&bip->bli_buf->b_li_list)); +xfs_buf_item_done (bp); + goto out_release; + } + +/* +* Clean, unreferenced BLIs can be immediately freed, leaving the buffer +* locked and referenced. +* +* Dirty, unreferenced BLIs *must* be in the AIL awaiting writeback. +*/ +if (!dirty) +xfs_buf_item_relse (bip); +else +ASSERT (test_bit (XFS_LI_IN_AIL , &lip->li_flags)); + +/* Not safe to reference the BLI from here */ +out_release: +/* +* If we get here with a stale buffer, we must not unlock the +* buffer as the last BLI reference owns lock context, not us. +*/ +if (stale || hold)return ; -ASSERT (!stale || aborted);xfs_buf_relse (bp); }
fs/xfs/xfs_buf_item.h +1 -1
@@ -49,7 +49,7 @@ struct xfs_buf_log_item {int xfs_buf_item_init (struct xfs_buf *,struct xfs_mount *);void xfs_buf_item_done (struct xfs_buf *bp); -bool xfs_buf_item_put (struct xfs_buf_log_item *); +void xfs_buf_item_put (struct xfs_buf_log_item *bip);void xfs_buf_item_log (struct xfs_buf_log_item *,uint ,uint );bool xfs_buf_item_dirty_format (struct xfs_buf_log_item *);void xfs_buf_inode_iodone (struct xfs_buf *);
Signature · 1 line
2.53.0
Sasha Levin
maintainerQuoted 2 lines
Backport of this series:
https://lore.kernel.org/all/[email protected]/
I saw Darrick's ack on the 6.12 series. Still waiting for one on this.
Signature · 2 lines
Thanks, Sasha
Darrick J. Wong
Acked-byQuoted 4 lines · Sasha Levin
Quoted 2 lines
Backport of this series:
https://lore.kernel.org/all/[email protected]/I saw Darrick's ack on the 6.12 series. Still waiting for one on this.
This also looks good to me,
--D
Quoted 5 lines
Signature · 3 lines
-- Thanks, Sasha
Quoted 3 lines · Darrick J. Wong
Quoted 1 line
I saw Darrick's ack on the 6.12 series. Still waiting for one on this.
This also looks good to me,
Can you also take a look at 6.1 and 5.15?
Thanks,
Siddh
Quoted 6 lines · Siddh Raman Pant
Quoted 3 lines · Darrick J. Wong
Quoted 1 line
I saw Darrick's ack on the 6.12 series. Still waiting for one on this.
This also looks good to me,
Can you also take a look at 6.1 and 5.15?
Frankly, if you've run fstests on all those LTS kernels and no new problems have erupted, then go for it.
--D
Quoted 2 lines
Thanks,
Siddh
Quoted 2 lines
This also looks good to me,
Queued the series for 6.6, thanks.
Signature · 2 lines
Thanks, Sasha