Skip to contentSkip to conversation

xfs: resolve close() deadlocks on frozen filesystems

PATCH Superseded Aditya Srivastava 2 participants 5 messages Jun 16 XFS filesystem

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v4 cover xfs: resolve close() deadlocks on frozen filesystems

From: Aditya Prakash Srivastava <[email protected]>

Hi Carlos and Christoph,

Apologies for the mailing list noise in v3; those patches were accidentally sent as unthreaded emails due to a local Git configuration. This v4 series is properly threaded under the cover letter standalone, and has a distinct subject line to prevent mail client threading collisions.

This is version 4 of the patch series addressing the close() system call hanging indefinitely on frozen XFS filesystems (Bugzilla #205833).

Based on Christoph's feedback, I have made the following improvements:
- Split the changes into a clean, 2-patch series separating the new trylock flag implementation from the deadlock fix itself.
- Extracted the case history, reproducer code, and discussion from the commit logs into this cover letter to keep commit logs surgically focused.
- Adjusted the parameter name in xfs_free_eofblocks() to trans_flags to make its usage stand out.
- Implemented a much cleaner, race-free state preservation logic in xfs_file_release() by omitting pre-setting XFS_EOFBLOCKS_RELEASED on the inode, setting it only upon successful truncation.
- Simplified the transaction allocation block, renamed the flag to XFS_TRANS_WRITECOUNT_TRYLOCK, and added the requested mutual-exclusivity assertion in __xfs_trans_alloc().
- Set up this series to be posted standalone (not threaded as a reply) per the standard kernel guidelines.

As requested, I have also submitted the corresponding regression test to the xfstests mailing list (using tests/xfs/842 with a GPLv2-licensed helper program). The fstests maintainer (Zorro Lang) reviewed the test and has indicated that he will wait for this kernel patch to make it through before merging the test suite additions.

THE REAL-WORLD IMPACT (BUGZILLA & DOWNSTREAM CASES)
==================================================

When speculative post-EOF blocks are closed on XFS, the release path synchronously attempts to free them via xfs_free_eofblocks(). This allocates a write transaction (xfs_trans_alloc) which blocks indefinitely on the superblock freeze write lock (sb_start_intwrite) under fsfreeze.

This behavior has a long history of causing severe system disruption:
- Downstream Red Hat Bugzilla 1474726 (dating back to 2017) details complete system hangs during system backups when rsync and fsfreeze are used. Even seemingly harmless read-only commands like 'cat /var/log/messages' would hang on close() in __sb_start_write via xfs_free_eofblocks, requiring a hard reboot.
- Downstream LeApp integration test scenarios consistently hit this hang.

Hanging on close() frequently triggers container healthcheck failures, systemd service timeouts, and cluster failover cascades, which is disruptive to user-space applications that view close() as resource reclamation. No other major Linux filesystem (ext4, btrfs, etc.) synchronously allocates write transactions during close() system calls.

THE SOLUTION: NON-BLOCKING SUPERBLOCK TRYLOCK
=============================================

Instead of performing racy pre-checks, this series introduces XFS_TRANS_WRITECOUNT_TRYLOCK. When specified, __xfs_trans_alloc() attempts to obtain freeze protection using sb_start_intwrite_trylock(). If that fails, it aborts allocation gracefully and returns -EAGAIN.

We then pass XFS_TRANS_WRITECOUNT_TRYLOCK during xfs_file_release(). If the truncation fails due to a frozen filesystem (-EAGAIN), we cleanly bypass setting XFS_EOFBLOCKS_RELEASED on the inode, ensuring subsequent releases or the background blockgc garbage collector can successfully clean them up once thawed.

REPRODUCER DETAILS (GPLV2 LICENSED)
===================================

As requested, I have added a GPLv2-compatible license to the C reproducer provided below, and I have also sent a corresponding patch to the xfstests mailing list. The fstests maintainer (Zorro Lang) reviewed the patch and indicated that he will wait for this kernel patch series to be merged before pulling the test suite additions.

Compile with -pthread:

    /*
     * GPLv2-compatible XFS freeze close() hang reproducer.
     * Copyright (c) 2026 Aditya Prakash Srivastava. All Rights Reserved.
     */
    #define _GNU_SOURCE
    #include <stdio.h>
    #include <stdlib.h>
    #include <string.h>
    #include <unistd.h>
    #include <fcntl.h>
    #include <pthread.h>
    #include <sys/ioctl.h>
    #include <sys/vfs.h>
    #include <linux/fs.h>
    #include <libgen.h>

    volatile int close_started = 0;
    volatile int close_completed = 0;

    void *close_thread(void *arg) {
        int fd = *(int *)arg;
        close_started = 1;
        close(fd);
        close_completed = 1;
        return NULL;
    }

    int main(int argc, char *argv[]) {
        struct statfs sfs;
        if (statfs(argv[1], &sfs) < 0) {
            char *dir_buf = strdup(argv[1]);
            char *parent_dir = dirname(dir_buf);
            if (statfs(parent_dir, &sfs) < 0) {
                perror("statfs");
                free(dir_buf);
                return 1;
            }
            free(dir_buf);
        }
        if (sfs.f_type != 0x58465342) return 1;

        int freeze_fd = open(dirname(strdup(argv[1])), O_RDONLY);
        int write_fd = open(argv[1], O_WRONLY | O_CREAT | O_TRUNC, 0644);
        char buf[65536] = {0};
        for (int i = 0; i < 320; i++) write(write_fd, buf, sizeof(buf));

        ioctl(freeze_fd, FIFREEZE, 0);
        pthread_t thread;
        pthread_create(&thread, NULL, close_thread, &write_fd);
        while (!close_started) usleep(1000);
        usleep(1000000); // Wait 1s
        if (!close_completed) printf("SUCCESS: close() hung!\n");
        ioctl(freeze_fd, FITHAW, 0);
        pthread_join(thread, NULL);
        unlink(argv[1]);
        return 0;
    }

Aditya Prakash Srivastava (2):
xfs: add a XFS_TRANS_WRITECOUNT_TRYLOCK flag
xfs: prevent close() from hanging on frozen filesystems

 fs/xfs/libxfs/xfs_shared.h |  3 +++
 fs/xfs/xfs_bmap_util.c     |  9 +++++----
 fs/xfs/xfs_bmap_util.h     |  2 +-
 fs/xfs/xfs_file.c          |  8 +++++---
 fs/xfs/xfs_icache.c        |  2 +-
 fs/xfs/xfs_inode.c         |  2 +-
 fs/xfs/xfs_trans.c         | 12 +++++++++++-
 7 files changed, 27 insertions(+), 11 deletions(-)
Signature · 1 line
2.47.3
#1 lore raw

PATCH v4 1/2 xfs: add a XFS_TRANS_WRITECOUNT_TRYLOCK flag

From: Aditya Prakash Srivastava <[email protected]>

Introduce a new transaction allocation flag, XFS_TRANS_WRITECOUNT_TRYLOCK. When this flag is specified, __xfs_trans_alloc() attempts to obtain freeze protection using sb_start_intwrite_trylock() instead of blocking indefinitely on sb_start_intwrite().

If the trylock fails, the allocation is aborted gracefully: the freshly allocated transaction handle is freed, and the function returns the appropriate error pointer ERR_PTR(-EAGAIN), which is then propagated to the caller by xfs_trans_alloc().

Also add an assertion in __xfs_trans_alloc() to ensure that both XFS_TRANS_NO_WRITECOUNT and XFS_TRANS_WRITECOUNT_TRYLOCK are never specified at the same time, as they are mutually exclusive.

---
 fs/xfs/libxfs/xfs_shared.h |  3 +++
 fs/xfs/xfs_trans.c         | 12 +++++++++++-
 2 files changed, 14 insertions(+), 1 deletion(-)
fs/xfs/libxfs/xfs_shared.h +3 -0
index b1e0d9bc1f7d..68d22b6cddd3 100644
@@ -164,6 +164,9 @@ void	xfs_log_get_max_trans_res(struct xfs_mount *mp,
 /* Transaction has locked the rtbitmap and rtsum inodes */
 #define XFS_TRANS_RTBITMAP_LOCKED	(1u << 9)
 
+/* Try lock filesystem superblock for freeze protection */
+#define XFS_TRANS_WRITECOUNT_TRYLOCK	(1u << 10)
+
 /*
  * Field values for xfs_trans_mod_sb.
  */
fs/xfs/xfs_trans.c +11 -1
index 148cc32449c1..3860e44d6439 100644
@@ -216,10 +216,18 @@ __xfs_trans_alloc(
 	struct xfs_trans	*tp;
 
 	ASSERT(!(flags & XFS_TRANS_RES_FDBLKS) || xfs_has_lazysbcount(mp));
+	ASSERT(!((flags & XFS_TRANS_NO_WRITECOUNT) &&
+		 (flags & XFS_TRANS_WRITECOUNT_TRYLOCK)));
 
 	tp = kmem_cache_zalloc(xfs_trans_cache, GFP_KERNEL | __GFP_NOFAIL);
-	if (!(flags & XFS_TRANS_NO_WRITECOUNT))
+	if (flags & XFS_TRANS_WRITECOUNT_TRYLOCK) {
+		if (!sb_start_intwrite_trylock(mp->m_super)) {
+			kmem_cache_free(xfs_trans_cache, tp);
+			return ERR_PTR(-EAGAIN);
+		}
+	} else if (!(flags & XFS_TRANS_NO_WRITECOUNT)) {
 		sb_start_intwrite(mp->m_super);
+	}
 	xfs_trans_set_context(tp);
 	tp->t_flags = flags;
 	tp->t_mountp = mp;
@@ -252,6 +260,8 @@ xfs_trans_alloc(
 	 */
 retry:
 	tp = __xfs_trans_alloc(mp, flags);
+	if (IS_ERR(tp))
+		return PTR_ERR(tp);
 	WARN_ON(mp->m_super->s_writers.frozen == SB_FREEZE_COMPLETE);
 	error = xfs_trans_reserve(tp, resp, blocks, rtextents);
 	if (error == -ENOSPC && want_retry) {
Signature · 1 line
2.47.3
#3 lore raw

PATCH v4 2/2 xfs: prevent close() from hanging on frozen filesystems

In reply to Aditya Srivastava

From: Aditya Prakash Srivastava <[email protected]>

When a file with active speculative post-EOF preallocations is closed, xfs_file_release() synchronously triggers xfs_free_eofblocks() to clean them up. This requires allocating a write transaction (xfs_trans_alloc), which blocks indefinitely if the filesystem is currently frozen or in the process of freezing, as it waits to acquire the superblock's write lock.

As a result, a close() system call on a read-write file descriptor can hang indefinitely in percpu_rwsem_wait() until the filesystem is thawed, even if the file is closed by a non-writer process or after all writing activity has already ceased.

To fix this properly and avoid any potential race conditions where a freeze might come in immediately after a writable check, pass the new XFS_TRANS_WRITECOUNT_TRYLOCK flag to xfs_trans_alloc() when freeing speculative preallocations in xfs_file_release().

If xfs_free_eofblocks() returns -EAGAIN on a trylock failure, we cleanly bypass setting XFS_EOFBLOCKS_RELEASED on the inode, ensuring subsequent releases or the background blockgc garbage collector can successfully retry the cleanup once the filesystem thaws.

Also, rename the flags parameter in xfs_free_eofblocks() to trans_flags as suggested to make its usage stand out, and update existing callers to pass 0 to preserve standard blocking paths.

---
 fs/xfs/xfs_bmap_util.c | 9 +++++----
 fs/xfs/xfs_bmap_util.h | 2 +-
 fs/xfs/xfs_file.c      | 8 +++++---
 fs/xfs/xfs_icache.c    | 2 +-
 fs/xfs/xfs_inode.c     | 2 +-
 5 files changed, 13 insertions(+), 10 deletions(-)
fs/xfs/xfs_bmap_util.c +5 -4
index 0ab00615f1ad..faf0630717dc 100644
@@ -574,7 +574,8 @@ xfs_can_free_eofblocks(
  */
 int
 xfs_free_eofblocks(
-	struct xfs_inode	*ip)
+	struct xfs_inode	*ip,
+	uint			trans_flags)
 {
 	struct xfs_trans	*tp;
 	struct xfs_mount	*mp = ip->i_mount;
@@ -604,9 +605,9 @@ xfs_free_eofblocks(
 		return 0;
 	}
 
-	error = xfs_trans_alloc(mp, &M_RES(mp)->tr_itruncate, 0, 0, 0, &tp);
+	error = xfs_trans_alloc(mp, &M_RES(mp)->tr_itruncate, 0, 0, trans_flags, &tp);
 	if (error) {
-		ASSERT(xfs_is_shutdown(mp));
+		ASSERT(error == -EAGAIN || xfs_is_shutdown(mp));
 		return error;
 	}
 
@@ -928,7 +929,7 @@ xfs_prepare_shift(
 	 * into the accessible region of the file.
 	 */
 	if (xfs_can_free_eofblocks(ip)) {
-		error = xfs_free_eofblocks(ip);
+		error = xfs_free_eofblocks(ip, 0);
 		if (error)
 			return error;
 	}
fs/xfs/xfs_bmap_util.h +1 -1
index c477b3361630..c13774aa0892 100644
@@ -66,7 +66,7 @@ int	xfs_insert_file_space(struct xfs_inode *, xfs_off_t offset,
 
 /* EOF block manipulation functions */
 bool	xfs_can_free_eofblocks(struct xfs_inode *ip);
-int	xfs_free_eofblocks(struct xfs_inode *ip);
+int	xfs_free_eofblocks(struct xfs_inode *ip, uint trans_flags);
 
 int	xfs_swap_extents(struct xfs_inode *ip, struct xfs_inode *tip,
 			 struct xfs_swapext *sx);
fs/xfs/xfs_file.c +5 -3
index 845a97c9b063..76c9b2fe7c51 100644
@@ -1806,9 +1806,11 @@ xfs_file_release(
 	 */
 	if (!xfs_iflags_test(ip, XFS_EOFBLOCKS_RELEASED) &&
 	    xfs_ilock_nowait(ip, XFS_IOLOCK_EXCL)) {
-		if (xfs_can_free_eofblocks(ip) &&
-		    !xfs_iflags_test_and_set(ip, XFS_EOFBLOCKS_RELEASED))
-			xfs_free_eofblocks(ip);
+		if (!xfs_iflags_test(ip, XFS_EOFBLOCKS_RELEASED) &&
+		    xfs_can_free_eofblocks(ip) &&
+		    !xfs_free_eofblocks(ip, XFS_TRANS_WRITECOUNT_TRYLOCK))
+			xfs_iflags_set(ip, XFS_EOFBLOCKS_RELEASED);
+
 		xfs_iunlock(ip, XFS_IOLOCK_EXCL);
 	}
 
fs/xfs/xfs_icache.c +1 -1
index 2040a9292ee6..c575b4acb24c 100644
@@ -1259,7 +1259,7 @@ xfs_inode_free_eofblocks(
 	*lockflags |= XFS_IOLOCK_EXCL;
 
 	if (xfs_can_free_eofblocks(ip))
-		return xfs_free_eofblocks(ip);
+		return xfs_free_eofblocks(ip, 0);
 
 	/* inode could be preallocated */
 	trace_xfs_inode_free_eofblocks_invalid(ip);
fs/xfs/xfs_inode.c +1 -1
index ddf2707c8894..14d3cd04a79f 100644
@@ -1423,7 +1423,7 @@ xfs_inactive(
 		 * reference to the inode at this point anyways.
 		 */
 		if (xfs_can_free_eofblocks(ip))
-			error = xfs_free_eofblocks(ip);
+			error = xfs_free_eofblocks(ip, 0);
 
 		goto out;
 	}
Signature · 1 line
2.47.3
#4 lore raw

Re: [PATCH v4 2/2] xfs: prevent close() from hanging on frozen filesystems

Quoted 3 lines · Aditya Srivastava

Also, rename the flags parameter in xfs_free_eofblocks() to trans_flags as suggested to make its usage stand out, and update existing callers to pass 0 to preserve standard blocking paths.

It is only renamed compared to the previous patches, because this patch actually adds it.

fs/xfs/xfs_bmap_util.c xfs_free_eofblocks( · line 608

Quoted 2 lines
fs/xfs/xfs_bmap_util.c +1 -1
-	error = xfs_trans_alloc(mp, &M_RES(mp)->tr_itruncate, 0, 0, 0, &tp);
+	error = xfs_trans_alloc(mp, &M_RES(mp)->tr_itruncate, 0, 0, trans_flags, &tp);

Overly long line here.

Otherwise this looks good.