xfs: prevent close() from hanging on frozen filesystems
v2 cover xfs: prevent close() from hanging on frozen filesystems
From: Aditya Prakash Srivastava <[email protected]>
Hi Carlos and Christoph,
This is version 2 of the patch series addressing the close() system call hanging indefinitely on frozen XFS filesystems (Bugzilla #205833).
Based on Christoph's feedback, I have made the following improvements:
- Split the changes into a clean, 2-patch series separating the new trylock flag implementation from the deadlock fix itself.
- Extracted the case history, reproducer code, and discussion from the commit logs into this cover letter to keep commit logs surgically focused.
- Adjusted the parameter name in xfs_free_eofblocks() to trans_flags to make its usage stand out.
- Implemented a much cleaner, race-free state preservation logic in xfs_file_release() by omitting pre-setting XFS_EOFBLOCKS_RELEASED on the inode, setting it only upon successful truncation.
- Simplified the transaction allocation block, renamed the flag to XFS_TRANS_WRITECOUNT_TRYLOCK, and added the requested mutual-exclusivity assertion in __xfs_trans_alloc().
- Set up this series to be posted standalone (not threaded as a reply) per the standard kernel guidelines.
THE REAL-WORLD IMPACT (BUGZILLA & DOWNSTREAM CASES)
==================================================
When speculative post-EOF blocks are closed on XFS, the release path synchronously attempts to free them via xfs_free_eofblocks(). This allocates a write transaction (xfs_trans_alloc) which blocks indefinitely on the superblock freeze write lock (sb_start_intwrite) under fsfreeze.
This behavior has a long history of causing severe system disruption:
- Downstream Red Hat Bugzilla 1474726 (dating back to 2017) details complete system hangs during system backups when rsync and fsfreeze are used. Even seemingly harmless read-only commands like 'cat /var/log/messages' would hang on close() in __sb_start_write via xfs_free_eofblocks, requiring a hard reboot.
- Downstream LeApp integration test scenarios consistently hit this hang.
Hanging on close() frequently triggers container healthcheck failures, systemd service timeouts, and cluster failover cascades, which is disruptive to user-space applications that view close() as resource reclamation. No other major Linux filesystem (ext4, btrfs, etc.) synchronously allocates write transactions during close() system calls.
THE SOLUTION: NON-BLOCKING SUPERBLOCK TRYLOCK
=============================================
Instead of performing racy pre-checks, this series introduces XFS_TRANS_WRITECOUNT_TRYLOCK. When specified, __xfs_trans_alloc() attempts to obtain freeze protection using sb_start_intwrite_trylock(). If that fails, it aborts allocation gracefully and returns -EAGAIN.
We then pass XFS_TRANS_WRITECOUNT_TRYLOCK during xfs_file_release(). If the truncation fails due to a frozen filesystem (-EAGAIN), we cleanly bypass setting XFS_EOFBLOCKS_RELEASED on the inode, ensuring subsequent releases or the background blockgc garbage collector can successfully clean them up once thawed.
REPRODUCER DETAILS (GPLV2 LICENSED)
===================================
As requested, I have added a GPLv2-compatible license to the C reproducer provided below, and I will be working on wiring up this reproducer into the official xfstests suite in the near future.
Compile with -pthread:
/*
* GPLv2-compatible XFS freeze close() hang reproducer.
* Copyright (c) 2026 Aditya Prakash Srivastava. All Rights Reserved.
*/
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <pthread.h>
#include <sys/ioctl.h>
#include <sys/vfs.h>
#include <linux/fs.h>
#include <libgen.h>
volatile int close_started = 0;
volatile int close_completed = 0;
void *close_thread(void *arg) {
int fd = *(int *)arg;
close_started = 1;
close(fd);
close_completed = 1;
return NULL;
}
int main(int argc, char *argv[]) {
struct statfs sfs;
if (statfs(argv[1], &sfs) < 0) {
char *dir_buf = strdup(argv[1]);
char *parent_dir = dirname(dir_buf);
if (statfs(parent_dir, &sfs) < 0) {
perror("statfs");
free(dir_buf);
return 1;
}
free(dir_buf);
}
if (sfs.f_type != 0x58465342) return 1;
int freeze_fd = open(dirname(strdup(argv[1])), O_RDONLY);
int write_fd = open(argv[1], O_WRONLY | O_CREAT | O_TRUNC, 0644);
char buf[65536] = {0};
for (int i = 0; i < 320; i++) write(write_fd, buf, sizeof(buf));
ioctl(freeze_fd, FIFREEZE, 0);
pthread_t thread;
pthread_create(&thread, NULL, close_thread, &write_fd);
while (!close_started) usleep(1000);
usleep(1000000); // Wait 1s
if (!close_completed) printf("SUCCESS: close() hung!\n");
ioctl(freeze_fd, FITHAW, 0);
pthread_join(thread, NULL);
unlink(argv[1]);
return 0;
}Aditya Prakash Srivastava (2):
xfs: add a XFS_TRANS_WRITECOUNT_TRYLOCK flag
xfs: prevent close() from hanging on frozen filesystems
fs/xfs/libxfs/xfs_shared.h | 3 +++ fs/xfs/xfs_bmap_util.c | 9 +++++---- fs/xfs/xfs_bmap_util.h | 2 +- fs/xfs/xfs_file.c | 8 +++++--- fs/xfs/xfs_icache.c | 2 +- fs/xfs/xfs_inode.c | 2 +- fs/xfs/xfs_trans.c | 12 +++++++++++- 7 files changed, 27 insertions(+), 11 deletions(-)
Signature · 1 line
2.47.3
PATCH v2 1/2 xfs: add a XFS_TRANS_WRITECOUNT_TRYLOCK flag
From: Aditya Prakash Srivastava <[email protected]>
Introduce a new transaction allocation flag, XFS_TRANS_WRITECOUNT_TRYLOCK. When this flag is specified, __xfs_trans_alloc() attempts to obtain freeze protection using sb_start_intwrite_trylock() instead of blocking indefinitely on sb_start_intwrite().
If the trylock fails, the allocation is aborted gracefully: the freshly allocated transaction handle is freed, and the function returns the appropriate error pointer ERR_PTR(-EAGAIN), which is then propagated to the caller by xfs_trans_alloc().
Also add an assertion in __xfs_trans_alloc() to ensure that both XFS_TRANS_NO_WRITECOUNT and XFS_TRANS_WRITECOUNT_TRYLOCK are never specified at the same time, as they are mutually exclusive.
--- fs/xfs/libxfs/xfs_shared.h | 3 +++ fs/xfs/xfs_trans.c | 12 +++++++++++- 2 files changed, 14 insertions(+), 1 deletion(-)
fs/xfs/libxfs/xfs_shared.h +3 -0
@@ -164,6 +164,9 @@ void xfs_log_get_max_trans_res(struct xfs_mount *mp,/* Transaction has locked the rtbitmap and rtsum inodes */ #define XFS_TRANS_RTBITMAP_LOCKED (1u << 9) +/* Try lock filesystem superblock for freeze protection */ +#define XFS_TRANS_WRITECOUNT_TRYLOCK (1u << 10) +/* * Field values for xfs_trans_mod_sb. */
fs/xfs/xfs_trans.c +11 -1
@@ -216,10 +216,18 @@ __xfs_trans_alloc(struct xfs_trans *tp;ASSERT (!(flags &XFS_TRANS_RES_FDBLKS ) ||xfs_has_lazysbcount (mp)); +ASSERT (!((flags &XFS_TRANS_NO_WRITECOUNT ) && + (flags &XFS_TRANS_WRITECOUNT_TRYLOCK ))); tp =kmem_cache_zalloc (xfs_trans_cache,GFP_KERNEL | __GFP_NOFAIL); -if (!(flags &XFS_TRANS_NO_WRITECOUNT )) +if (flags &XFS_TRANS_WRITECOUNT_TRYLOCK ) { +if (!sb_start_intwrite_trylock (mp->m_super)) { +kmem_cache_free (xfs_trans_cache, tp); +return ERR_PTR (-EAGAIN ); + } + }else if (!(flags &XFS_TRANS_NO_WRITECOUNT )) {sb_start_intwrite (mp->m_super); + }xfs_trans_set_context (tp); tp->t_flags = flags; tp->t_mountp = mp;
@@ -252,6 +260,8 @@ xfs_trans_alloc(*/ retry: tp =__xfs_trans_alloc (mp, flags); +if (IS_ERR (tp)) +return PTR_ERR (tp);WARN_ON (mp->m_super->s_writers.frozen ==SB_FREEZE_COMPLETE ); error =xfs_trans_reserve (tp, resp, blocks, rtextents);if (error == -ENOSPC && want_retry) {
Signature · 1 line
2.47.3
Re: [PATCH v2 1/2] xfs: add a XFS_TRANS_WRITECOUNT_TRYLOCK flag
Apologies for the mailing list noise in v2; a local stale patch file collision caused the wrong files and version headers to be sent. Please ignore this version v2 completely and take a look at the clean v3 series.
Thanks
Aditya Prakash Srivastava
Quoted 74 lines · Aditya Srivastava
From: Aditya Prakash Srivastava <[email protected]>
Introduce a new transaction allocation flag,
XFS_TRANS_WRITECOUNT_TRYLOCK. When this flag is specified,__xfs_trans_alloc()attempts to obtain freeze protection usingsb_start_intwrite_trylock()instead of blocking indefinitely onsb_start_intwrite().If the trylock fails, the allocation is aborted gracefully: the freshly allocated transaction handle is freed, and the function returns the appropriate error pointer
ERR_PTR(-EAGAIN), which is then propagated to the caller byxfs_trans_alloc().Also add an assertion in
__xfs_trans_alloc()to ensure that bothXFS_TRANS_NO_WRITECOUNTandXFS_TRANS_WRITECOUNT_TRYLOCKare never specified at the same time, as they are mutually exclusive.Suggested-by Christoph Hellwig <[email protected]>Signed-off-by Aditya Prakash Srivastava <[email protected]>--- fs/xfs/libxfs/xfs_shared.h | 3 +++ fs/xfs/xfs_trans.c | 12 +++++++++++- 2 files changed, 14 insertions(+), 1 deletion(-)fs/xfs/libxfs/xfs_shared.h +3
-0@@ -164,6 +164,9 @@ void xfs_log_get_max_trans_res(struct xfs_mount *mp,/* Transaction has locked the rtbitmap and rtsum inodes */ #define XFS_TRANS_RTBITMAP_LOCKED (1u << 9) +/* Try lock filesystem superblock for freeze protection */ +#define XFS_TRANS_WRITECOUNT_TRYLOCK (1u << 10) +/* * Field values for xfs_trans_mod_sb. */ fs/xfs/xfs_trans.c +11
-2@@ -216,10 +216,18 @@ __xfs_trans_alloc(struct xfs_trans *tp;ASSERT (!(flags &XFS_TRANS_RES_FDBLKS ) ||xfs_has_lazysbcount (mp)); +ASSERT (!((flags &XFS_TRANS_NO_WRITECOUNT ) && + (flags &XFS_TRANS_WRITECOUNT_TRYLOCK ))); tp =kmem_cache_zalloc (xfs_trans_cache,GFP_KERNEL | __GFP_NOFAIL); -if (!(flags &XFS_TRANS_NO_WRITECOUNT )) +if (flags &XFS_TRANS_WRITECOUNT_TRYLOCK ) { +if (!sb_start_intwrite_trylock (mp->m_super)) { +kmem_cache_free (xfs_trans_cache, tp); +return ERR_PTR (-EAGAIN ); + } + }else if (!(flags &XFS_TRANS_NO_WRITECOUNT )) {sb_start_intwrite (mp->m_super); + }xfs_trans_set_context (tp); tp->t_flags = flags; tp->t_mountp = mp;@@ -252,6 +260,8 @@ xfs_trans_alloc(*/ retry: tp =__xfs_trans_alloc (mp, flags); +if (IS_ERR (tp)) +return PTR_ERR (tp);WARN_ON (mp->m_super->s_writers.frozen ==SB_FREEZE_COMPLETE ); error =xfs_trans_reserve (tp, resp, blocks, rtextents);if (error == -ENOSPC && want_retry) { --2.47.3
PATCH v2 2/2 xfs: prevent close() from hanging on frozen filesystems
In reply to Aditya SrivastavaFrom: Aditya Prakash Srivastava <[email protected]>
When a file with active speculative post-EOF preallocations is closed, xfs_file_release() synchronously triggers xfs_free_eofblocks() to clean them up. This requires allocating a write transaction (xfs_trans_alloc), which blocks indefinitely if the filesystem is currently frozen or in the process of freezing, as it waits to acquire the superblock's write lock.
As a result, a close() system call on a read-write file descriptor can hang indefinitely in percpu_rwsem_wait() until the filesystem is thawed, even if the file is closed by a non-writer process or after all writing activity has already ceased.
To fix this properly and avoid any potential race conditions where a freeze might come in immediately after a writable check, pass the new XFS_TRANS_WRITECOUNT_TRYLOCK flag to xfs_trans_alloc() when freeing speculative preallocations in xfs_file_release().
If xfs_free_eofblocks() returns -EAGAIN on a trylock failure, we cleanly bypass setting XFS_EOFBLOCKS_RELEASED on the inode, ensuring subsequent releases or the background blockgc garbage collector can successfully retry the cleanup once the filesystem thaws.
Also, rename the flags parameter in xfs_free_eofblocks() to trans_flags as suggested to make its usage stand out, and update existing callers to pass 0 to preserve standard blocking paths.
--- fs/xfs/xfs_bmap_util.c | 9 +++++---- fs/xfs/xfs_bmap_util.h | 2 +- fs/xfs/xfs_file.c | 8 +++++--- fs/xfs/xfs_icache.c | 2 +- fs/xfs/xfs_inode.c | 2 +- 5 files changed, 13 insertions(+), 10 deletions(-)
fs/xfs/xfs_bmap_util.c +5 -4
@@ -574,7 +574,8 @@ xfs_can_free_eofblocks(*/ int xfs_free_eofblocks ( -struct xfs_inode *ip) +struct xfs_inode *ip, +uint trans_flags) {struct xfs_trans *tp;struct xfs_mount *mp = ip->i_mount;
@@ -604,9 +605,9 @@ xfs_free_eofblocks(return 0 ; } - error =xfs_trans_alloc (mp, &M_RES (mp)->tr_itruncate,0 ,0 ,0 , &tp); + error =xfs_trans_alloc (mp, &M_RES (mp)->tr_itruncate,0 ,0 , trans_flags, &tp);if (error) { -ASSERT (xfs_is_shutdown (mp)); +ASSERT (error == -EAGAIN ||xfs_is_shutdown (mp));return error; }
@@ -928,7 +929,7 @@ xfs_prepare_shift(* into the accessible region of the file. */ if (xfs_can_free_eofblocks (ip)) { - error =xfs_free_eofblocks (ip); + error =xfs_free_eofblocks (ip,0 );if (error)return error; }
fs/xfs/xfs_bmap_util.h +1 -1
@@ -66,7 +66,7 @@ int xfs_insert_file_space(struct xfs_inode *, xfs_off_t offset,/* EOF block manipulation functions */ bool xfs_can_free_eofblocks (struct xfs_inode *ip); -int xfs_free_eofblocks (struct xfs_inode *ip); +int xfs_free_eofblocks (struct xfs_inode *ip,uint trans_flags);int xfs_swap_extents (struct xfs_inode *ip,struct xfs_inode *tip,struct xfs_swapext *sx);
fs/xfs/xfs_file.c +5 -3
@@ -1806,9 +1806,11 @@ xfs_file_release(*/ if (!xfs_iflags_test (ip,XFS_EOFBLOCKS_RELEASED ) &&xfs_ilock_nowait (ip,XFS_IOLOCK_EXCL )) { -if (xfs_can_free_eofblocks (ip) && - !xfs_iflags_test_and_set (ip,XFS_EOFBLOCKS_RELEASED )) -xfs_free_eofblocks (ip); +if (!xfs_iflags_test (ip,XFS_EOFBLOCKS_RELEASED ) && +xfs_can_free_eofblocks (ip) && + !xfs_free_eofblocks (ip,XFS_TRANS_WRITECOUNT_TRYLOCK )) +xfs_iflags_set (ip,XFS_EOFBLOCKS_RELEASED ); +xfs_iunlock (ip,XFS_IOLOCK_EXCL ); }
fs/xfs/xfs_icache.c +1 -1
@@ -1259,7 +1259,7 @@ xfs_inode_free_eofblocks( *lockflags |=XFS_IOLOCK_EXCL ;if (xfs_can_free_eofblocks (ip)) -return xfs_free_eofblocks (ip); +return xfs_free_eofblocks (ip,0 );/* inode could be preallocated */ trace_xfs_inode_free_eofblocks_invalid (ip);
fs/xfs/xfs_inode.c +1 -1
@@ -1423,7 +1423,7 @@ xfs_inactive(* reference to the inode at this point anyways. */ if (xfs_can_free_eofblocks (ip)) - error =xfs_free_eofblocks (ip); + error =xfs_free_eofblocks (ip,0 ); goto out; }
Signature · 1 line
2.47.3
Apologies for the mailing list noise in v2; a local stale patch file collision caused the wrong files and version headers to be sent. Please ignore this version v2 completely and take a look at the clean v3 series.
Thanks
Aditya Prakash Srivastava
Quoted 134 lines · Aditya Srivastava
From: Aditya Prakash Srivastava <[email protected]>
When a file with active speculative post-EOF preallocations is closed,
xfs_file_release()synchronously triggersxfs_free_eofblocks()to clean them up. This requires allocating a write transaction (xfs_trans_alloc), which blocks indefinitely if the filesystem is currently frozen or in the process of freezing, as it waits to acquire the superblock's write lock.As a result, a
close()system call on a read-write file descriptor can hang indefinitely inpercpu_rwsem_wait()until the filesystem is thawed, even if the file is closed by a non-writer process or after all writing activity has already ceased.To fix this properly and avoid any potential race conditions where a freeze might come in immediately after a writable check, pass the new
XFS_TRANS_WRITECOUNT_TRYLOCKflag toxfs_trans_alloc()when freeing speculative preallocations inxfs_file_release().If
xfs_free_eofblocks()returns-EAGAINon a trylock failure, we cleanly bypass settingXFS_EOFBLOCKS_RELEASEDon the inode, ensuring subsequent releases or the background blockgc garbage collector can successfully retry the cleanup once the filesystem thaws.Also, rename the flags parameter in
xfs_free_eofblocks()totrans_flagsas suggested to make its usage stand out, and update existing callers to pass 0 to preserve standard blocking paths.Suggested-by Christoph Hellwig <[email protected]>Signed-off-by Aditya Prakash Srivastava <[email protected]>--- fs/xfs/xfs_bmap_util.c | 9 +++++---- fs/xfs/xfs_bmap_util.h | 2 +- fs/xfs/xfs_file.c | 8 +++++--- fs/xfs/xfs_icache.c | 2 +- fs/xfs/xfs_inode.c | 2 +- 5 files changed, 13 insertions(+), 10 deletions(-)fs/xfs/xfs_bmap_util.c +5
-4@@ -574,7 +574,8 @@ xfs_can_free_eofblocks(*/ int xfs_free_eofblocks ( -struct xfs_inode *ip) +struct xfs_inode *ip, +uint trans_flags) {struct xfs_trans *tp;struct xfs_mount *mp = ip->i_mount;@@ -604,9 +605,9 @@ xfs_free_eofblocks(return 0 ; } - error =xfs_trans_alloc (mp, &M_RES (mp)->tr_itruncate,0 ,0 ,0 , &tp); + error =xfs_trans_alloc (mp, &M_RES (mp)->tr_itruncate,0 ,0 , trans_flags, &tp);if (error) { -ASSERT (xfs_is_shutdown (mp)); +ASSERT (error == -EAGAIN ||xfs_is_shutdown (mp));return error; }@@ -928,7 +929,7 @@ xfs_prepare_shift(* into the accessible region of the file. */ if (xfs_can_free_eofblocks (ip)) { - error =xfs_free_eofblocks (ip); + error =xfs_free_eofblocks (ip,0 );if (error)return error; }fs/xfs/xfs_bmap_util.h +1
-1@@ -66,7 +66,7 @@ int xfs_insert_file_space(struct xfs_inode *, xfs_off_t offset,/* EOF block manipulation functions */ bool xfs_can_free_eofblocks (struct xfs_inode *ip); -int xfs_free_eofblocks (struct xfs_inode *ip); +int xfs_free_eofblocks (struct xfs_inode *ip,uint trans_flags);int xfs_swap_extents (struct xfs_inode *ip,struct xfs_inode *tip,struct xfs_swapext *sx);fs/xfs/xfs_file.c +5
-3@@ -1806,9 +1806,11 @@ xfs_file_release(*/ if (!xfs_iflags_test (ip,XFS_EOFBLOCKS_RELEASED ) &&xfs_ilock_nowait (ip,XFS_IOLOCK_EXCL )) { -if (xfs_can_free_eofblocks (ip) && - !xfs_iflags_test_and_set (ip,XFS_EOFBLOCKS_RELEASED )) -xfs_free_eofblocks (ip); +if (!xfs_iflags_test (ip,XFS_EOFBLOCKS_RELEASED ) && +xfs_can_free_eofblocks (ip) && + !xfs_free_eofblocks (ip,XFS_TRANS_WRITECOUNT_TRYLOCK )) +xfs_iflags_set (ip,XFS_EOFBLOCKS_RELEASED ); +xfs_iunlock (ip,XFS_IOLOCK_EXCL ); }fs/xfs/xfs_icache.c +1
-1@@ -1259,7 +1259,7 @@ xfs_inode_free_eofblocks( *lockflags |=XFS_IOLOCK_EXCL ;if (xfs_can_free_eofblocks (ip)) -return xfs_free_eofblocks (ip); +return xfs_free_eofblocks (ip,0 );/* inode could be preallocated */ trace_xfs_inode_free_eofblocks_invalid (ip);fs/xfs/xfs_inode.c +1
-2@@ -1423,7 +1423,7 @@ xfs_inactive(* reference to the inode at this point anyways. */ if (xfs_can_free_eofblocks (ip)) - error =xfs_free_eofblocks (ip); + error =xfs_free_eofblocks (ip,0 ); goto out; } --2.47.3