Skip to contentSkip to conversation

New: fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)

Bug 207053 [email protected] 5 participants 20 messages Apr 1, 2020 – Mar 16, 2023 XFS filesystem

lore
Share & tools
lore
#0 lore raw

https://bugzilla.kernel.org/show_bug.cgi?id=207053

            Bug ID: 207053
           Summary: fsfreeze deadlock on XFS (the FIFREEZE ioctl and
                    subsequent FITHAW hang indefinitely)
           Product: File System
           Version: 2.5
    Kernel Version: 4.19.75, 5.4.20
          Hardware: x86-64
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: XFS
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

When we upgraded from kernel 4.14.146 to kernel 4.19.75, we began to experience frequent deadlocks from our cronjobs that freeze the filesystem for snapshotting.

The fsfreeze stack shows:

# cat /proc/33256/stack 
[<0>] __flush_work+0x177/0x1b0
[<0>] __cancel_work_timer+0x12b/0x1b0
[<0>] xfs_stop_block_reaping+0x15/0x30 [xfs]
[<0>] xfs_fs_freeze+0x15/0x40 [xfs]
[<0>] freeze_super+0xc8/0x190
[<0>] do_vfs_ioctl+0x510/0x630
[<0>] ksys_ioctl+0x70/0x80
[<0>] __x64_sys_ioctl+0x16/0x20
[<0>] do_syscall_64+0x4e/0x100
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

The fsfreeze -u stack shows:

# cat /proc/37753/stack 
[<0>] rwsem_down_write_slowpath+0x257/0x510
[<0>] thaw_super+0x12/0x20
[<0>] do_vfs_ioctl+0x609/0x630
[<0>] ksys_ioctl+0x70/0x80
[<0>] __x64_sys_ioctl+0x16/0x20
[<0>] do_syscall_64+0x4e/0x100
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

Echoing "j" into /proc/sysrq-trigger to emergency thaw all filesystems doesn't solve this either. We're hitting this bug many times per week, so if there's any more debug information you need that we could turn on, let us know. Thanks!

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#1 lore raw
Quoted 3 lines · On Wed, Apr 01, 2020 at 07:02:57PM +0000, [email protected]…

When we upgraded from kernel 4.14.146 to kernel 4.19.75, we began to experience frequent deadlocks from our cronjobs that freeze the filesystem for snapshotting.

Probably commit d6b636ebb1c9 ("xfs: halt auto-reclamation activities while rebuilding rmap") in 4.18, but that fixes a bug that allowed reaping functions to attempt to modify the filesystem while it was frozen...

Quoted 12 lines

The fsfreeze stack shows:

# cat /proc/33256/stack 
[<0>] __flush_work+0x177/0x1b0
[<0>] __cancel_work_timer+0x12b/0x1b0
[<0>] xfs_stop_block_reaping+0x15/0x30 [xfs]
[<0>] xfs_fs_freeze+0x15/0x40 [xfs]
[<0>] freeze_super+0xc8/0x190
[<0>] do_vfs_ioctl+0x510/0x630
[<0>] ksys_ioctl+0x70/0x80
[<0>] __x64_sys_ioctl+0x16/0x20
[<0>] do_syscall_64+0x4e/0x100
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

This indicates that the reaping worker is still busy doing work. It needs to finish before the freeze will continue to make progress. So either the system is still doing work, or the kworker has blocked somewhere.

What is the dmesg output of 'echo w > /prox/sysrq-trigger'?

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#2 lore raw

[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #1 from Dave Chinner ([email protected]) ---

Quoted 4 lines · On Wed, Apr 01, 2020 at 07:02:57PM +0000, [email protected]…

When we upgraded from kernel 4.14.146 to kernel 4.19.75, we began to experience
frequent deadlocks from our cronjobs that freeze the filesystem for snapshotting.

Probably commit d6b636ebb1c9 ("xfs: halt auto-reclamation activities while rebuilding rmap") in 4.18, but that fixes a bug that allowed reaping functions to attempt to modify the filesystem while it was frozen...

Quoted 12 lines

The fsfreeze stack shows:

# cat /proc/33256/stack 
[<0>] __flush_work+0x177/0x1b0
[<0>] __cancel_work_timer+0x12b/0x1b0
[<0>] xfs_stop_block_reaping+0x15/0x30 [xfs]
[<0>] xfs_fs_freeze+0x15/0x40 [xfs]
[<0>] freeze_super+0xc8/0x190
[<0>] do_vfs_ioctl+0x510/0x630
[<0>] ksys_ioctl+0x70/0x80
[<0>] __x64_sys_ioctl+0x16/0x20
[<0>] do_syscall_64+0x4e/0x100
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

This indicates that the reaping worker is still busy doing work. It needs to finish before the freeze will continue to make progress. So either the system is still doing work, or the kworker has blocked somewhere.

What is the dmesg output of 'echo w > /prox/sysrq-trigger'?

Cheers,

Dave.

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#3 lore raw

[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,

Just had another case of this crop up and I was able to get the blocked tasks output before automation killed the server. Because the log was too large to attach, I've pasted the output into a github gist here:
https://gist.githubusercontent.com/PaulFurtado/c9bade038b8a5c7ddb53a6e10def058f/raw/ee43926c96c0d6a9ec81a648754c1af599ef0bdd/sysrq_w.log

Thanks,
Paul

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#4 lore raw

Re: [Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)

Quoted 10 lines · On Tue, Apr 07, 2020 at 06:41:31AM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,

Just had another case of this crop up and I was able to get the blocked tasks output before automation killed the server. Because the log was too large to attach, I've pasted the output into a github gist here:
https://gist.githubusercontent.com/PaulFurtado/c9bade038b8a5c7ddb53a6e10def058f/raw/ee43926c96c0d6a9ec81a648754c1af599ef0bdd/sysrq_w.log

Hm, so it looks like this is stuck between freeze:

[377279.630957] fsfreeze        D    0 46819  46337 0x00004084
[377279.634910] Call Trace:
[377279.637594]  ? __schedule+0x292/0x6f0
[377279.640833]  ? xfs_xattr_get+0x51/0x80 [xfs]
[377279.644287]  schedule+0x2f/0xa0
[377279.647286]  schedule_timeout+0x1dd/0x300
[377279.650661]  wait_for_completion+0x126/0x190
[377279.654154]  ? wake_up_q+0x80/0x80
[377279.657277]  ? work_busy+0x80/0x80
[377279.660375]  __flush_work+0x177/0x1b0
[377279.663604]  ? worker_attach_to_pool+0x90/0x90
[377279.667121]  __cancel_work_timer+0x12b/0x1b0
[377279.670571]  ? rcu_sync_enter+0x8b/0xd0
[377279.673864]  xfs_stop_block_reaping+0x15/0x30 [xfs]
[377279.677585]  xfs_fs_freeze+0x15/0x40 [xfs]
[377279.680950]  freeze_super+0xc8/0x190
[377279.684086]  do_vfs_ioctl+0x510/0x630

...

... and the eofblocks scanner:

[377279.422496] Workqueue: xfs-eofblocks/nvme13n1 xfs_eofblocks_worker [xfs]
[377279.426971] Call Trace:
[377279.429662]  ? __schedule+0x292/0x6f0
[377279.432839]  schedule+0x2f/0xa0
[377279.435794]  rwsem_down_read_slowpath+0x196/0x530
[377279.439435]  ? kmem_cache_alloc+0x152/0x1f0
[377279.442834]  ? __percpu_down_read+0x49/0x60
[377279.446242]  __percpu_down_read+0x49/0x60
[377279.449586]  __sb_start_write+0x5b/0x60
[377279.452869]  xfs_trans_alloc+0x152/0x160 [xfs]
[377279.456372]  xfs_free_eofblocks+0x12d/0x1f0 [xfs]
[377279.460014]  xfs_inode_free_eofblocks+0x128/0x1a0 [xfs]
[377279.463903]  ? xfs_inode_ag_walk_grab+0x5f/0x90 [xfs]
[377279.467680]  xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs]
[377279.471567]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.475620]  ? kvm_sched_clock_read+0xd/0x20
[377279.479059]  ? sched_clock+0x5/0x10
[377279.482184]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.486234]  ? radix_tree_gang_lookup_tag+0xa8/0x100
[377279.489974]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.494041]  xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs]
[377279.497859]  xfs_eofblocks_worker+0x29/0x40 [xfs]
[377279.501484]  process_one_work+0x195/0x380

...

The immediate issue is likely that the eofblocks transaction is not NOWRITECOUNT (same for the cowblocks scanner, btw), but the problem with doing that is these helpers are called from other contexts outside of the background scanners.

Perhaps what we need to do here is let these background scanners acquire a superblock write reference, similar to what Darrick recently added to scrub..? We'd have to do that from the scanner workqueue task, so it would probably need to be a trylock so we don't end up in a similar situation as above. I.e., we'd either get the reference and cause freeze to wait until it's dropped or bail out if freeze has already stopped the transaction subsystem. Thoughts?

Brian

Quoted 8 lines

Thanks,
Paul

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#5 lore raw

Re: [Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)

Quoted 65 lines · Brian Foster
Quoted 10 lines · On Tue, Apr 07, 2020 at 06:41:31AM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,

Just had another case of this crop up and I was able to get the blocked tasks output before automation killed the server. Because the log was too large to attach, I've pasted the output into a github gist here:
https://gist.githubusercontent.com/PaulFurtado/c9bade038b8a5c7ddb53a6e10def058f/raw/ee43926c96c0d6a9ec81a648754c1af599ef0bdd/sysrq_w.log

Hm, so it looks like this is stuck between freeze:

[377279.630957] fsfreeze        D    0 46819  46337 0x00004084
[377279.634910] Call Trace:
[377279.637594]  ? __schedule+0x292/0x6f0
[377279.640833]  ? xfs_xattr_get+0x51/0x80 [xfs]
[377279.644287]  schedule+0x2f/0xa0
[377279.647286]  schedule_timeout+0x1dd/0x300
[377279.650661]  wait_for_completion+0x126/0x190
[377279.654154]  ? wake_up_q+0x80/0x80
[377279.657277]  ? work_busy+0x80/0x80
[377279.660375]  __flush_work+0x177/0x1b0
[377279.663604]  ? worker_attach_to_pool+0x90/0x90
[377279.667121]  __cancel_work_timer+0x12b/0x1b0
[377279.670571]  ? rcu_sync_enter+0x8b/0xd0
[377279.673864]  xfs_stop_block_reaping+0x15/0x30 [xfs]
[377279.677585]  xfs_fs_freeze+0x15/0x40 [xfs]
[377279.680950]  freeze_super+0xc8/0x190
[377279.684086]  do_vfs_ioctl+0x510/0x630

...

... and the eofblocks scanner:

[377279.422496] Workqueue: xfs-eofblocks/nvme13n1 xfs_eofblocks_worker [xfs]
[377279.426971] Call Trace:
[377279.429662]  ? __schedule+0x292/0x6f0
[377279.432839]  schedule+0x2f/0xa0
[377279.435794]  rwsem_down_read_slowpath+0x196/0x530
[377279.439435]  ? kmem_cache_alloc+0x152/0x1f0
[377279.442834]  ? __percpu_down_read+0x49/0x60
[377279.446242]  __percpu_down_read+0x49/0x60
[377279.449586]  __sb_start_write+0x5b/0x60
[377279.452869]  xfs_trans_alloc+0x152/0x160 [xfs]
[377279.456372]  xfs_free_eofblocks+0x12d/0x1f0 [xfs]
[377279.460014]  xfs_inode_free_eofblocks+0x128/0x1a0 [xfs]
[377279.463903]  ? xfs_inode_ag_walk_grab+0x5f/0x90 [xfs]
[377279.467680]  xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs]
[377279.471567]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.475620]  ? kvm_sched_clock_read+0xd/0x20
[377279.479059]  ? sched_clock+0x5/0x10
[377279.482184]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.486234]  ? radix_tree_gang_lookup_tag+0xa8/0x100
[377279.489974]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.494041]  xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs]
[377279.497859]  xfs_eofblocks_worker+0x29/0x40 [xfs]
[377279.501484]  process_one_work+0x195/0x380

...

The immediate issue is likely that the eofblocks transaction is not NOWRITECOUNT (same for the cowblocks scanner, btw), but the problem with doing that is these helpers are called from other contexts outside of the background scanners.

Quoted 7 lines

Perhaps what we need to do here is let these background scanners acquire a superblock write reference, similar to what Darrick recently added to scrub..? We'd have to do that from the scanner workqueue task, so it would probably need to be a trylock so we don't end up in a similar situation as above. I.e., we'd either get the reference and cause freeze to wait until it's dropped or bail out if freeze has already stopped the transaction subsystem. Thoughts?

Hmm, I had a whole gigantic series to refactor all the speculative preallocation gc work into a single thread + radix tree tag; I'll see if that series actually fixed this problem too.

But yes, all background threads that run transactions need to have freezer protection.

--D

Quoted 11 lines

Brian

Quoted 8 lines

Thanks,
Paul

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#6 lore raw

Re: [Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)

Quoted 74 lines · Darrick J. Wong
Quoted 65 lines · Brian Foster
Quoted 10 lines · On Tue, Apr 07, 2020 at 06:41:31AM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,

Just had another case of this crop up and I was able to get the blocked tasks output before automation killed the server. Because the log was too large to attach, I've pasted the output into a github gist here:
https://gist.githubusercontent.com/PaulFurtado/c9bade038b8a5c7ddb53a6e10def058f/raw/ee43926c96c0d6a9ec81a648754c1af599ef0bdd/sysrq_w.log

Hm, so it looks like this is stuck between freeze:

[377279.630957] fsfreeze        D    0 46819  46337 0x00004084
[377279.634910] Call Trace:
[377279.637594]  ? __schedule+0x292/0x6f0
[377279.640833]  ? xfs_xattr_get+0x51/0x80 [xfs]
[377279.644287]  schedule+0x2f/0xa0
[377279.647286]  schedule_timeout+0x1dd/0x300
[377279.650661]  wait_for_completion+0x126/0x190
[377279.654154]  ? wake_up_q+0x80/0x80
[377279.657277]  ? work_busy+0x80/0x80
[377279.660375]  __flush_work+0x177/0x1b0
[377279.663604]  ? worker_attach_to_pool+0x90/0x90
[377279.667121]  __cancel_work_timer+0x12b/0x1b0
[377279.670571]  ? rcu_sync_enter+0x8b/0xd0
[377279.673864]  xfs_stop_block_reaping+0x15/0x30 [xfs]
[377279.677585]  xfs_fs_freeze+0x15/0x40 [xfs]
[377279.680950]  freeze_super+0xc8/0x190
[377279.684086]  do_vfs_ioctl+0x510/0x630

...

... and the eofblocks scanner:

[377279.422496] Workqueue: xfs-eofblocks/nvme13n1 xfs_eofblocks_worker [xfs]
[377279.426971] Call Trace:
[377279.429662]  ? __schedule+0x292/0x6f0
[377279.432839]  schedule+0x2f/0xa0
[377279.435794]  rwsem_down_read_slowpath+0x196/0x530
[377279.439435]  ? kmem_cache_alloc+0x152/0x1f0
[377279.442834]  ? __percpu_down_read+0x49/0x60
[377279.446242]  __percpu_down_read+0x49/0x60
[377279.449586]  __sb_start_write+0x5b/0x60
[377279.452869]  xfs_trans_alloc+0x152/0x160 [xfs]
[377279.456372]  xfs_free_eofblocks+0x12d/0x1f0 [xfs]
[377279.460014]  xfs_inode_free_eofblocks+0x128/0x1a0 [xfs]
[377279.463903]  ? xfs_inode_ag_walk_grab+0x5f/0x90 [xfs]
[377279.467680]  xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs]
[377279.471567]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.475620]  ? kvm_sched_clock_read+0xd/0x20
[377279.479059]  ? sched_clock+0x5/0x10
[377279.482184]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.486234]  ? radix_tree_gang_lookup_tag+0xa8/0x100
[377279.489974]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.494041]  xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs]
[377279.497859]  xfs_eofblocks_worker+0x29/0x40 [xfs]
[377279.501484]  process_one_work+0x195/0x380

...

The immediate issue is likely that the eofblocks transaction is not NOWRITECOUNT (same for the cowblocks scanner, btw), but the problem with doing that is these helpers are called from other contexts outside of the background scanners.

Quoted 7 lines

Perhaps what we need to do here is let these background scanners acquire a superblock write reference, similar to what Darrick recently added to scrub..? We'd have to do that from the scanner workqueue task, so it would probably need to be a trylock so we don't end up in a similar situation as above. I.e., we'd either get the reference and cause freeze to wait until it's dropped or bail out if freeze has already stopped the transaction subsystem. Thoughts?

Quoted 7 lines

Hmm, I had a whole gigantic series to refactor all the speculative preallocation gc work into a single thread + radix tree tag; I'll see if that series actually fixed this problem too.

But yes, all background threads that run transactions need to have freezer protection.

So something like the following in the meantime, assuming we want a backportable fix..? I think this means we could return -EAGAIN from the eofblocks ioctl, but afaict if something functionally conflicts with an active scan across freeze then perhaps that's preferred.

Brian

fs/xfs/xfs_icache.c +14 -3
index a7be7a9e5c1a..0f14d58e5bb0 100644
@@ -1515,13 +1515,24 @@ __xfs_icache_free_eofblocks(
 					   void *args),
 	int			tag)
 {
-	int flags = SYNC_TRYLOCK;
+	int			flags = SYNC_TRYLOCK;
+	int			error;
 
 	if (eofb && (eofb->eof_flags & XFS_EOF_FLAGS_SYNC))
 		flags = SYNC_WAIT;
 
-	return xfs_inode_ag_iterator_tag(mp, execute, flags,
-					 eofb, tag);
+	/*
+	 * freeze waits on background scanner jobs to complete so we cannot
+	 * block on write protection here. Bail if the transaction subsystem is
+	 * already freezing, returning -EAGAIN to notify other callers.
+	 */
+	if (!sb_start_write_trylock(mp->m_super))
+		return -EAGAIN;
+
+	error = xfs_inode_ag_iterator_tag(mp, execute, flags, eofb, tag);
+	sb_end_write(mp->m_super);
+
+	return error;
 }
 
 int
Quoted 14 lines

--D

Quoted 11 lines

Brian

Quoted 8 lines

Thanks,
Paul

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#7 lore raw

Re: [Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)

Quoted 79 lines · Brian Foster
Quoted 78 lines · Darrick J. Wong
Quoted 65 lines · Brian Foster
Quoted 10 lines · On Tue, Apr 07, 2020 at 06:41:31AM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,

Just had another case of this crop up and I was able to get the blocked tasks output before automation killed the server. Because the log was too large to attach, I've pasted the output into a github gist here:
https://gist.githubusercontent.com/PaulFurtado/c9bade038b8a5c7ddb53a6e10def058f/raw/ee43926c96c0d6a9ec81a648754c1af599ef0bdd/sysrq_w.log

Hm, so it looks like this is stuck between freeze:

[377279.630957] fsfreeze        D    0 46819  46337 0x00004084
[377279.634910] Call Trace:
[377279.637594]  ? __schedule+0x292/0x6f0
[377279.640833]  ? xfs_xattr_get+0x51/0x80 [xfs]
[377279.644287]  schedule+0x2f/0xa0
[377279.647286]  schedule_timeout+0x1dd/0x300
[377279.650661]  wait_for_completion+0x126/0x190
[377279.654154]  ? wake_up_q+0x80/0x80
[377279.657277]  ? work_busy+0x80/0x80
[377279.660375]  __flush_work+0x177/0x1b0
[377279.663604]  ? worker_attach_to_pool+0x90/0x90
[377279.667121]  __cancel_work_timer+0x12b/0x1b0
[377279.670571]  ? rcu_sync_enter+0x8b/0xd0
[377279.673864]  xfs_stop_block_reaping+0x15/0x30 [xfs]
[377279.677585]  xfs_fs_freeze+0x15/0x40 [xfs]
[377279.680950]  freeze_super+0xc8/0x190
[377279.684086]  do_vfs_ioctl+0x510/0x630

...

... and the eofblocks scanner:

[377279.422496] Workqueue: xfs-eofblocks/nvme13n1 xfs_eofblocks_worker [xfs]
[377279.426971] Call Trace:
[377279.429662]  ? __schedule+0x292/0x6f0
[377279.432839]  schedule+0x2f/0xa0
[377279.435794]  rwsem_down_read_slowpath+0x196/0x530
[377279.439435]  ? kmem_cache_alloc+0x152/0x1f0
[377279.442834]  ? __percpu_down_read+0x49/0x60
[377279.446242]  __percpu_down_read+0x49/0x60
[377279.449586]  __sb_start_write+0x5b/0x60
[377279.452869]  xfs_trans_alloc+0x152/0x160 [xfs]
[377279.456372]  xfs_free_eofblocks+0x12d/0x1f0 [xfs]
[377279.460014]  xfs_inode_free_eofblocks+0x128/0x1a0 [xfs]
[377279.463903]  ? xfs_inode_ag_walk_grab+0x5f/0x90 [xfs]
[377279.467680]  xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs]
[377279.471567]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.475620]  ? kvm_sched_clock_read+0xd/0x20
[377279.479059]  ? sched_clock+0x5/0x10
[377279.482184]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.486234]  ? radix_tree_gang_lookup_tag+0xa8/0x100
[377279.489974]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.494041]  xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs]
[377279.497859]  xfs_eofblocks_worker+0x29/0x40 [xfs]
[377279.501484]  process_one_work+0x195/0x380

...

The immediate issue is likely that the eofblocks transaction is not NOWRITECOUNT (same for the cowblocks scanner, btw), but the problem with doing that is these helpers are called from other contexts outside of the background scanners.

Quoted 7 lines

Perhaps what we need to do here is let these background scanners acquire a superblock write reference, similar to what Darrick recently added to scrub..? We'd have to do that from the scanner workqueue task, so it would probably need to be a trylock so we don't end up in a similar situation as above. I.e., we'd either get the reference and cause freeze to wait until it's dropped or bail out if freeze has already stopped the transaction subsystem. Thoughts?

Hmm, I had a whole gigantic series to refactor all the speculative preallocation gc work into a single thread + radix tree tag; I'll see if that series actually fixed this problem too.

Quoted 8 lines
Quoted 3 lines

But yes, all background threads that run transactions need to have freezer protection.

So something like the following in the meantime, assuming we want a backportable fix..? I think this means we could return -EAGAIN from the eofblocks ioctl, but afaict if something functionally conflicts with an active scan across freeze then perhaps that's preferred.

Apparently I don't have a patch that fixes the speculative gc code. The deferred inactivation worker does it, so perhaps I got mixed up. :/

I think a better fix would be to annotate xfs_icache_free_eofblocks and xfs_icache_free_cowblocks to note that the caller must obtain freeze protection before calling those functions. Then we can play whackamole with the existing callers:

1. xfs_eofblocks_worker and xfs_cowblocks_worker can try to sb_start_write and just go back to sleep if the fs is frozen. The flush_workqueue will then cancel the delayed work and the freeze can proceed.

2. The buffered write ENOSPC scour-and-retry loops already have freeze protection because they're file writes, so they don't have to change.

3. XFS_IOC_FREE_EOFBLOCKS can sb_start_write, which means that callers will sleep on the frozen fs.

--D

Quoted 50 lines

Brian

fs/xfs/xfs_icache.c +14 -3
index a7be7a9e5c1a..0f14d58e5bb0 100644
@@ -1515,13 +1515,24 @@ __xfs_icache_free_eofblocks(
 					   void *args),
 	int			tag)
 {
-	int flags = SYNC_TRYLOCK;
+	int			flags = SYNC_TRYLOCK;
+	int			error;
 
 	if (eofb && (eofb->eof_flags & XFS_EOF_FLAGS_SYNC))
 		flags = SYNC_WAIT;
 
-	return xfs_inode_ag_iterator_tag(mp, execute, flags,
-					 eofb, tag);
+	/*
+	 * freeze waits on background scanner jobs to complete so we cannot
+	 * block on write protection here. Bail if the transaction subsystem is
+	 * already freezing, returning -EAGAIN to notify other callers.
+	 */
+	if (!sb_start_write_trylock(mp->m_super))
+		return -EAGAIN;
+
+	error = xfs_inode_ag_iterator_tag(mp, execute, flags, eofb, tag);
+	sb_end_write(mp->m_super);
+
+	return error;
 }
 
 int
Quoted 15 lines

--D

Quoted 11 lines

Brian

Quoted 8 lines

Thanks,
Paul

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#8 lore raw

Re: [Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)

Quoted 84 lines · Darrick J. Wong
Quoted 83 lines · Brian Foster
Quoted 81 lines · Darrick J. Wong
Quoted 65 lines · Brian Foster
Quoted 10 lines · On Tue, Apr 07, 2020 at 06:41:31AM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,

Just had another case of this crop up and I was able to get the blocked tasks output before automation killed the server. Because the log was too large to attach, I've pasted the output into a github gist here:
https://gist.githubusercontent.com/PaulFurtado/c9bade038b8a5c7ddb53a6e10def058f/raw/ee43926c96c0d6a9ec81a648754c1af599ef0bdd/sysrq_w.log

Hm, so it looks like this is stuck between freeze:

[377279.630957] fsfreeze        D    0 46819  46337 0x00004084
[377279.634910] Call Trace:
[377279.637594]  ? __schedule+0x292/0x6f0
[377279.640833]  ? xfs_xattr_get+0x51/0x80 [xfs]
[377279.644287]  schedule+0x2f/0xa0
[377279.647286]  schedule_timeout+0x1dd/0x300
[377279.650661]  wait_for_completion+0x126/0x190
[377279.654154]  ? wake_up_q+0x80/0x80
[377279.657277]  ? work_busy+0x80/0x80
[377279.660375]  __flush_work+0x177/0x1b0
[377279.663604]  ? worker_attach_to_pool+0x90/0x90
[377279.667121]  __cancel_work_timer+0x12b/0x1b0
[377279.670571]  ? rcu_sync_enter+0x8b/0xd0
[377279.673864]  xfs_stop_block_reaping+0x15/0x30 [xfs]
[377279.677585]  xfs_fs_freeze+0x15/0x40 [xfs]
[377279.680950]  freeze_super+0xc8/0x190
[377279.684086]  do_vfs_ioctl+0x510/0x630

...

... and the eofblocks scanner:

[377279.422496] Workqueue: xfs-eofblocks/nvme13n1 xfs_eofblocks_worker [xfs]
[377279.426971] Call Trace:
[377279.429662]  ? __schedule+0x292/0x6f0
[377279.432839]  schedule+0x2f/0xa0
[377279.435794]  rwsem_down_read_slowpath+0x196/0x530
[377279.439435]  ? kmem_cache_alloc+0x152/0x1f0
[377279.442834]  ? __percpu_down_read+0x49/0x60
[377279.446242]  __percpu_down_read+0x49/0x60
[377279.449586]  __sb_start_write+0x5b/0x60
[377279.452869]  xfs_trans_alloc+0x152/0x160 [xfs]
[377279.456372]  xfs_free_eofblocks+0x12d/0x1f0 [xfs]
[377279.460014]  xfs_inode_free_eofblocks+0x128/0x1a0 [xfs]
[377279.463903]  ? xfs_inode_ag_walk_grab+0x5f/0x90 [xfs]
[377279.467680]  xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs]
[377279.471567]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.475620]  ? kvm_sched_clock_read+0xd/0x20
[377279.479059]  ? sched_clock+0x5/0x10
[377279.482184]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.486234]  ? radix_tree_gang_lookup_tag+0xa8/0x100
[377279.489974]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.494041]  xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs]
[377279.497859]  xfs_eofblocks_worker+0x29/0x40 [xfs]
[377279.501484]  process_one_work+0x195/0x380

...

The immediate issue is likely that the eofblocks transaction is not NOWRITECOUNT (same for the cowblocks scanner, btw), but the problem with doing that is these helpers are called from other contexts outside of the background scanners.

Quoted 7 lines

Perhaps what we need to do here is let these background scanners acquire a superblock write reference, similar to what Darrick recently added to scrub..? We'd have to do that from the scanner workqueue task, so it would probably need to be a trylock so we don't end up in a similar situation as above. I.e., we'd either get the reference and cause freeze to wait until it's dropped or bail out if freeze has already stopped the transaction subsystem. Thoughts?

Hmm, I had a whole gigantic series to refactor all the speculative preallocation gc work into a single thread + radix tree tag; I'll see if that series actually fixed this problem too.

But yes, all background threads that run transactions need to have freezer protection.

Quoted 8 lines
Quoted 4 lines

So something like the following in the meantime, assuming we want a backportable fix..? I think this means we could return -EAGAIN from the eofblocks ioctl, but afaict if something functionally conflicts with an active scan across freeze then perhaps that's preferred.

Apparently I don't have a patch that fixes the speculative gc code. The deferred inactivation worker does it, so perhaps I got mixed up. :/

Ok.

Quoted 10 lines

I think a better fix would be to annotate xfs_icache_free_eofblocks and xfs_icache_free_cowblocks to note that the caller must obtain freeze protection before calling those functions. Then we can play whackamole with the existing callers:

1. xfs_eofblocks_worker and xfs_cowblocks_worker can try to sb_start_write and just go back to sleep if the fs is frozen. The flush_workqueue will then cancel the delayed work and the freeze can proceed.

Quoted 6 lines

2. The buffered write ENOSPC scour-and-retry loops already have freeze protection because they're file writes, so they don't have to change.

3. XFS_IOC_FREE_EOFBLOCKS can sb_start_write, which means that callers will sleep on the frozen fs.

Sure, that works for me. It can be condensed later if it ends up in a single thread.

Brian

Quoted 53 lines

--D

Quoted 50 lines

Brian

fs/xfs/xfs_icache.c +14 -3
index a7be7a9e5c1a..0f14d58e5bb0 100644
@@ -1515,13 +1515,24 @@ __xfs_icache_free_eofblocks(
 					   void *args),
 	int			tag)
 {
-	int flags = SYNC_TRYLOCK;
+	int			flags = SYNC_TRYLOCK;
+	int			error;
 
 	if (eofb && (eofb->eof_flags & XFS_EOF_FLAGS_SYNC))
 		flags = SYNC_WAIT;
 
-	return xfs_inode_ag_iterator_tag(mp, execute, flags,
-					 eofb, tag);
+	/*
+	 * freeze waits on background scanner jobs to complete so we cannot
+	 * block on write protection here. Bail if the transaction subsystem is
+	 * already freezing, returning -EAGAIN to notify other callers.
+	 */
+	if (!sb_start_write_trylock(mp->m_super))
+		return -EAGAIN;
+
+	error = xfs_inode_ag_iterator_tag(mp, execute, flags, eofb, tag);
+	sb_end_write(mp->m_super);
+
+	return error;
 }
 
 int
Quoted 15 lines

--D

Quoted 11 lines

Brian

Quoted 8 lines

Thanks,
Paul

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#9 lore raw

[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #3 from [email protected] ---

Quoted 11 lines · On Tue, Apr 07, 2020 at 06:41:31AM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,

Just had another case of this crop up and I was able to get the blocked tasks output before automation killed the server. Because the log was too large to attach, I've pasted the output into a github gist here:

https://gist.githubusercontent.com/PaulFurtado/c9bade038b8a5c7ddb53a6e10def058f/raw/ee43926c96c0d6a9ec81a648754c1af599ef0bdd/sysrq_w.log

Hm, so it looks like this is stuck between freeze:

[377279.630957] fsfreeze        D    0 46819  46337 0x00004084
[377279.634910] Call Trace:
[377279.637594]  ? __schedule+0x292/0x6f0
[377279.640833]  ? xfs_xattr_get+0x51/0x80 [xfs]
[377279.644287]  schedule+0x2f/0xa0
[377279.647286]  schedule_timeout+0x1dd/0x300
[377279.650661]  wait_for_completion+0x126/0x190
[377279.654154]  ? wake_up_q+0x80/0x80
[377279.657277]  ? work_busy+0x80/0x80
[377279.660375]  __flush_work+0x177/0x1b0
[377279.663604]  ? worker_attach_to_pool+0x90/0x90
[377279.667121]  __cancel_work_timer+0x12b/0x1b0
[377279.670571]  ? rcu_sync_enter+0x8b/0xd0
[377279.673864]  xfs_stop_block_reaping+0x15/0x30 [xfs]
[377279.677585]  xfs_fs_freeze+0x15/0x40 [xfs]
[377279.680950]  freeze_super+0xc8/0x190
[377279.684086]  do_vfs_ioctl+0x510/0x630

...

... and the eofblocks scanner:

[377279.422496] Workqueue: xfs-eofblocks/nvme13n1 xfs_eofblocks_worker [xfs]
[377279.426971] Call Trace:
[377279.429662]  ? __schedule+0x292/0x6f0
[377279.432839]  schedule+0x2f/0xa0
[377279.435794]  rwsem_down_read_slowpath+0x196/0x530
[377279.439435]  ? kmem_cache_alloc+0x152/0x1f0
[377279.442834]  ? __percpu_down_read+0x49/0x60
[377279.446242]  __percpu_down_read+0x49/0x60
[377279.449586]  __sb_start_write+0x5b/0x60
[377279.452869]  xfs_trans_alloc+0x152/0x160 [xfs]
[377279.456372]  xfs_free_eofblocks+0x12d/0x1f0 [xfs]
[377279.460014]  xfs_inode_free_eofblocks+0x128/0x1a0 [xfs]
[377279.463903]  ? xfs_inode_ag_walk_grab+0x5f/0x90 [xfs]
[377279.467680]  xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs]
[377279.471567]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.475620]  ? kvm_sched_clock_read+0xd/0x20
[377279.479059]  ? sched_clock+0x5/0x10
[377279.482184]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.486234]  ? radix_tree_gang_lookup_tag+0xa8/0x100
[377279.489974]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.494041]  xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs]
[377279.497859]  xfs_eofblocks_worker+0x29/0x40 [xfs]
[377279.501484]  process_one_work+0x195/0x380

...

The immediate issue is likely that the eofblocks transaction is not NOWRITECOUNT (same for the cowblocks scanner, btw), but the problem with doing that is these helpers are called from other contexts outside of the background scanners.

Perhaps what we need to do here is let these background scanners acquire a superblock write reference, similar to what Darrick recently added to scrub..? We'd have to do that from the scanner workqueue task, so it would probably need to be a trylock so we don't end up in a similar situation as above. I.e., we'd either get the reference and cause freeze to wait until it's dropped or bail out if freeze has already stopped the transaction subsystem. Thoughts?

Brian

Quoted 8 lines

Thanks,
Paul

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#10 lore raw

[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #4 from [email protected] ---

Quoted 69 lines · Brian Foster
Quoted 6 lines · On Tue, Apr 07, 2020 at 06:41:31AM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,

Just had another case of this crop up and I was able to get the blocked

tasks

Quoted 1 line

output before automation killed the server. Because the log was too large

to

Quoted 2 lines

attach, I've pasted the output into a github gist here:

https://gist.githubusercontent.com/PaulFurtado/c9bade038b8a5c7ddb53a6e10def058f/raw/ee43926c96c0d6a9ec81a648754c1af599ef0bdd/sysrq_w.log

Quoted 1 line

Hm, so it looks like this is stuck between freeze:

[377279.630957] fsfreeze        D    0 46819  46337 0x00004084
[377279.634910] Call Trace:
[377279.637594]  ? __schedule+0x292/0x6f0
[377279.640833]  ? xfs_xattr_get+0x51/0x80 [xfs]
[377279.644287]  schedule+0x2f/0xa0
[377279.647286]  schedule_timeout+0x1dd/0x300
[377279.650661]  wait_for_completion+0x126/0x190
[377279.654154]  ? wake_up_q+0x80/0x80
[377279.657277]  ? work_busy+0x80/0x80
[377279.660375]  __flush_work+0x177/0x1b0
[377279.663604]  ? worker_attach_to_pool+0x90/0x90
[377279.667121]  __cancel_work_timer+0x12b/0x1b0
[377279.670571]  ? rcu_sync_enter+0x8b/0xd0
[377279.673864]  xfs_stop_block_reaping+0x15/0x30 [xfs]
[377279.677585]  xfs_fs_freeze+0x15/0x40 [xfs]
[377279.680950]  freeze_super+0xc8/0x190
[377279.684086]  do_vfs_ioctl+0x510/0x630

...

... and the eofblocks scanner:

[377279.422496] Workqueue: xfs-eofblocks/nvme13n1 xfs_eofblocks_worker [xfs]
[377279.426971] Call Trace:
[377279.429662]  ? __schedule+0x292/0x6f0
[377279.432839]  schedule+0x2f/0xa0
[377279.435794]  rwsem_down_read_slowpath+0x196/0x530
[377279.439435]  ? kmem_cache_alloc+0x152/0x1f0
[377279.442834]  ? __percpu_down_read+0x49/0x60
[377279.446242]  __percpu_down_read+0x49/0x60
[377279.449586]  __sb_start_write+0x5b/0x60
[377279.452869]  xfs_trans_alloc+0x152/0x160 [xfs]
[377279.456372]  xfs_free_eofblocks+0x12d/0x1f0 [xfs]
[377279.460014]  xfs_inode_free_eofblocks+0x128/0x1a0 [xfs]
[377279.463903]  ? xfs_inode_ag_walk_grab+0x5f/0x90 [xfs]
[377279.467680]  xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs]
[377279.471567]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.475620]  ? kvm_sched_clock_read+0xd/0x20
[377279.479059]  ? sched_clock+0x5/0x10
[377279.482184]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.486234]  ? radix_tree_gang_lookup_tag+0xa8/0x100
[377279.489974]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.494041]  xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs]
[377279.497859]  xfs_eofblocks_worker+0x29/0x40 [xfs]
[377279.501484]  process_one_work+0x195/0x380

...

The immediate issue is likely that the eofblocks transaction is not NOWRITECOUNT (same for the cowblocks scanner, btw), but the problem with doing that is these helpers are called from other contexts outside of the background scanners.

Quoted 7 lines

Perhaps what we need to do here is let these background scanners acquire a superblock write reference, similar to what Darrick recently added to scrub..? We'd have to do that from the scanner workqueue task, so it would probably need to be a trylock so we don't end up in a similar situation as above. I.e., we'd either get the reference and cause freeze to wait until it's dropped or bail out if freeze has already stopped the transaction subsystem. Thoughts?

Hmm, I had a whole gigantic series to refactor all the speculative preallocation gc work into a single thread + radix tree tag; I'll see if that series actually fixed this problem too.

But yes, all background threads that run transactions need to have freezer protection.

--D

Quoted 11 lines

Brian

Quoted 8 lines

Thanks,
Paul

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#11 lore raw

[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #5 from [email protected] ---

Quoted 79 lines · Darrick J. Wong
Quoted 1 line · Brian Foster

On Tue, Apr 07, 2020 at 06:41:31AM +0000,

Quoted 6 lines · [email protected] wrote:
Quoted 6 lines

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,

Just had another case of this crop up and I was able to get the blocked

tasks

Quoted 1 line
Quoted 1 line

output before automation killed the server. Because the log was too large

to

Quoted 2 lines
Quoted 2 lines

attach, I've pasted the output into a github gist here:

https://gist.githubusercontent.com/PaulFurtado/c9bade038b8a5c7ddb53a6e10def058f/raw/ee43926c96c0d6a9ec81a648754c1af599ef0bdd/sysrq_w.log

Quoted 21 lines
Quoted 1 line

Hm, so it looks like this is stuck between freeze:

[377279.630957] fsfreeze        D    0 46819  46337 0x00004084
[377279.634910] Call Trace:
[377279.637594]  ? __schedule+0x292/0x6f0
[377279.640833]  ? xfs_xattr_get+0x51/0x80 [xfs]
[377279.644287]  schedule+0x2f/0xa0
[377279.647286]  schedule_timeout+0x1dd/0x300
[377279.650661]  wait_for_completion+0x126/0x190
[377279.654154]  ? wake_up_q+0x80/0x80
[377279.657277]  ? work_busy+0x80/0x80
[377279.660375]  __flush_work+0x177/0x1b0
[377279.663604]  ? worker_attach_to_pool+0x90/0x90
[377279.667121]  __cancel_work_timer+0x12b/0x1b0
[377279.670571]  ? rcu_sync_enter+0x8b/0xd0
[377279.673864]  xfs_stop_block_reaping+0x15/0x30 [xfs]
[377279.677585]  xfs_fs_freeze+0x15/0x40 [xfs]
[377279.680950]  freeze_super+0xc8/0x190
[377279.684086]  do_vfs_ioctl+0x510/0x630
Quoted 5 lines

...

... and the eofblocks scanner:

[377279.422496] Workqueue: xfs-eofblocks/nvme13n1 xfs_eofblocks_worker

[xfs]

Quoted 29 lines
[377279.426971] Call Trace:
[377279.429662]  ? __schedule+0x292/0x6f0
[377279.432839]  schedule+0x2f/0xa0
[377279.435794]  rwsem_down_read_slowpath+0x196/0x530
[377279.439435]  ? kmem_cache_alloc+0x152/0x1f0
[377279.442834]  ? __percpu_down_read+0x49/0x60
[377279.446242]  __percpu_down_read+0x49/0x60
[377279.449586]  __sb_start_write+0x5b/0x60
[377279.452869]  xfs_trans_alloc+0x152/0x160 [xfs]
[377279.456372]  xfs_free_eofblocks+0x12d/0x1f0 [xfs]
[377279.460014]  xfs_inode_free_eofblocks+0x128/0x1a0 [xfs]
[377279.463903]  ? xfs_inode_ag_walk_grab+0x5f/0x90 [xfs]
[377279.467680]  xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs]
[377279.471567]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.475620]  ? kvm_sched_clock_read+0xd/0x20
[377279.479059]  ? sched_clock+0x5/0x10
[377279.482184]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.486234]  ? radix_tree_gang_lookup_tag+0xa8/0x100
[377279.489974]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.494041]  xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs]
[377279.497859]  xfs_eofblocks_worker+0x29/0x40 [xfs]
[377279.501484]  process_one_work+0x195/0x380

...

The immediate issue is likely that the eofblocks transaction is not NOWRITECOUNT (same for the cowblocks scanner, btw), but the problem with doing that is these helpers are called from other contexts outside of the background scanners.

Quoted 7 lines

Perhaps what we need to do here is let these background scanners acquire a superblock write reference, similar to what Darrick recently added to scrub..? We'd have to do that from the scanner workqueue task, so it would probably need to be a trylock so we don't end up in a similar situation as above. I.e., we'd either get the reference and cause freeze to wait until it's dropped or bail out if freeze has already stopped the transaction subsystem. Thoughts?

Quoted 7 lines

Hmm, I had a whole gigantic series to refactor all the speculative preallocation gc work into a single thread + radix tree tag; I'll see if that series actually fixed this problem too.

But yes, all background threads that run transactions need to have freezer protection.

So something like the following in the meantime, assuming we want a backportable fix..? I think this means we could return -EAGAIN from the eofblocks ioctl, but afaict if something functionally conflicts with an active scan across freeze then perhaps that's preferred.

Brian

fs/xfs/xfs_icache.c +14 -3
index a7be7a9e5c1a..0f14d58e5bb0 100644
@@ -1515,13 +1515,24 @@ __xfs_icache_free_eofblocks(
                                           void *args),
        int                     tag)
 {
-       int flags = SYNC_TRYLOCK;
+       int                     flags = SYNC_TRYLOCK;
+       int                     error;
 
        if (eofb && (eofb->eof_flags & XFS_EOF_FLAGS_SYNC))
                flags = SYNC_WAIT;
 
-       return xfs_inode_ag_iterator_tag(mp, execute, flags,
-                                        eofb, tag);
+       /*
+        * freeze waits on background scanner jobs to complete so we cannot
+        * block on write protection here. Bail if the transaction subsystem is
+        * already freezing, returning -EAGAIN to notify other callers.
+        */
+       if (!sb_start_write_trylock(mp->m_super))
+               return -EAGAIN;
+
+       error = xfs_inode_ag_iterator_tag(mp, execute, flags, eofb, tag);
+       sb_end_write(mp->m_super);
+
+       return error;
 }
 
 int
Quoted 14 lines

--D

Quoted 11 lines

Brian

Quoted 8 lines

Thanks,
Paul

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#12 lore raw

[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #6 from [email protected] ---

Quoted 84 lines · Brian Foster
Quoted 2 lines · Darrick J. Wong
Quoted 1 line · Brian Foster

On Tue, Apr 07, 2020 at 06:41:31AM +0000,

Quoted 6 lines · [email protected] wrote:
Quoted 6 lines
Quoted 6 lines

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,

Just had another case of this crop up and I was able to get the blocked

tasks

Quoted 1 line
Quoted 1 line
Quoted 1 line

output before automation killed the server. Because the log was too

large to

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

attach, I've pasted the output into a github gist here:

https://gist.githubusercontent.com/PaulFurtado/c9bade038b8a5c7ddb53a6e10def058f/raw/ee43926c96c0d6a9ec81a648754c1af599ef0bdd/sysrq_w.log

Quoted 23 lines
Quoted 23 lines
Quoted 1 line

Hm, so it looks like this is stuck between freeze:

[377279.630957] fsfreeze        D    0 46819  46337 0x00004084
[377279.634910] Call Trace:
[377279.637594]  ? __schedule+0x292/0x6f0
[377279.640833]  ? xfs_xattr_get+0x51/0x80 [xfs]
[377279.644287]  schedule+0x2f/0xa0
[377279.647286]  schedule_timeout+0x1dd/0x300
[377279.650661]  wait_for_completion+0x126/0x190
[377279.654154]  ? wake_up_q+0x80/0x80
[377279.657277]  ? work_busy+0x80/0x80
[377279.660375]  __flush_work+0x177/0x1b0
[377279.663604]  ? worker_attach_to_pool+0x90/0x90
[377279.667121]  __cancel_work_timer+0x12b/0x1b0
[377279.670571]  ? rcu_sync_enter+0x8b/0xd0
[377279.673864]  xfs_stop_block_reaping+0x15/0x30 [xfs]
[377279.677585]  xfs_fs_freeze+0x15/0x40 [xfs]
[377279.680950]  freeze_super+0xc8/0x190
[377279.684086]  do_vfs_ioctl+0x510/0x630

...

Quoted 3 lines
Quoted 3 lines

... and the eofblocks scanner:

[377279.422496] Workqueue: xfs-eofblocks/nvme13n1 xfs_eofblocks_worker

[xfs]

Quoted 41 lines
Quoted 29 lines
[377279.426971] Call Trace:
[377279.429662]  ? __schedule+0x292/0x6f0
[377279.432839]  schedule+0x2f/0xa0
[377279.435794]  rwsem_down_read_slowpath+0x196/0x530
[377279.439435]  ? kmem_cache_alloc+0x152/0x1f0
[377279.442834]  ? __percpu_down_read+0x49/0x60
[377279.446242]  __percpu_down_read+0x49/0x60
[377279.449586]  __sb_start_write+0x5b/0x60
[377279.452869]  xfs_trans_alloc+0x152/0x160 [xfs]
[377279.456372]  xfs_free_eofblocks+0x12d/0x1f0 [xfs]
[377279.460014]  xfs_inode_free_eofblocks+0x128/0x1a0 [xfs]
[377279.463903]  ? xfs_inode_ag_walk_grab+0x5f/0x90 [xfs]
[377279.467680]  xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs]
[377279.471567]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.475620]  ? kvm_sched_clock_read+0xd/0x20
[377279.479059]  ? sched_clock+0x5/0x10
[377279.482184]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.486234]  ? radix_tree_gang_lookup_tag+0xa8/0x100
[377279.489974]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.494041]  xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs]
[377279.497859]  xfs_eofblocks_worker+0x29/0x40 [xfs]
[377279.501484]  process_one_work+0x195/0x380

...

The immediate issue is likely that the eofblocks transaction is not NOWRITECOUNT (same for the cowblocks scanner, btw), but the problem with doing that is these helpers are called from other contexts outside of the background scanners.

Quoted 7 lines

Perhaps what we need to do here is let these background scanners acquire a superblock write reference, similar to what Darrick recently added to scrub..? We'd have to do that from the scanner workqueue task, so it would probably need to be a trylock so we don't end up in a similar situation as above. I.e., we'd either get the reference and cause freeze to wait until it's dropped or bail out if freeze has already stopped the transaction subsystem. Thoughts?

Hmm, I had a whole gigantic series to refactor all the speculative preallocation gc work into a single thread + radix tree tag; I'll see if that series actually fixed this problem too.

Quoted 8 lines
Quoted 3 lines

But yes, all background threads that run transactions need to have freezer protection.

So something like the following in the meantime, assuming we want a backportable fix..? I think this means we could return -EAGAIN from the eofblocks ioctl, but afaict if something functionally conflicts with an active scan across freeze then perhaps that's preferred.

Apparently I don't have a patch that fixes the speculative gc code. The deferred inactivation worker does it, so perhaps I got mixed up. :/

I think a better fix would be to annotate xfs_icache_free_eofblocks and xfs_icache_free_cowblocks to note that the caller must obtain freeze protection before calling those functions. Then we can play whackamole with the existing callers:

1. xfs_eofblocks_worker and xfs_cowblocks_worker can try to sb_start_write and just go back to sleep if the fs is frozen. The flush_workqueue will then cancel the delayed work and the freeze can proceed.

2. The buffered write ENOSPC scour-and-retry loops already have freeze protection because they're file writes, so they don't have to change.

3. XFS_IOC_FREE_EOFBLOCKS can sb_start_write, which means that callers will sleep on the frozen fs.

--D

Quoted 50 lines

Brian

fs/xfs/xfs_icache.c +14 -3
index a7be7a9e5c1a..0f14d58e5bb0 100644
@@ -1515,13 +1515,24 @@ __xfs_icache_free_eofblocks(
                                         void *args),
      int                     tag)
 {
-     int flags = SYNC_TRYLOCK;
+     int                     flags = SYNC_TRYLOCK;
+     int                     error;
 
      if (eofb && (eofb->eof_flags & XFS_EOF_FLAGS_SYNC))
              flags = SYNC_WAIT;
 
-     return xfs_inode_ag_iterator_tag(mp, execute, flags,
-                                      eofb, tag);
+     /*
+      * freeze waits on background scanner jobs to complete so we cannot
+      * block on write protection here. Bail if the transaction subsystem is
+      * already freezing, returning -EAGAIN to notify other callers.
+      */
+     if (!sb_start_write_trylock(mp->m_super))
+             return -EAGAIN;
+
+     error = xfs_inode_ag_iterator_tag(mp, execute, flags, eofb, tag);
+     sb_end_write(mp->m_super);
+
+     return error;
 }
 
 int
Quoted 15 lines

--D

Quoted 11 lines

Brian

Quoted 8 lines

Thanks,
Paul

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#13 lore raw

[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #7 from [email protected] ---

Quoted 93 lines · Darrick J. Wong
Quoted 3 lines · Brian Foster
Quoted 2 lines · Darrick J. Wong
Quoted 1 line · Brian Foster

On Tue, Apr 07, 2020 at 06:41:31AM +0000,

Quoted 6 lines · [email protected] wrote:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,

Just had another case of this crop up and I was able to get the

blocked tasks

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

output before automation killed the server. Because the log was too

large to

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

attach, I've pasted the output into a github gist here:

https://gist.githubusercontent.com/PaulFurtado/c9bade038b8a5c7ddb53a6e10def058f/raw/ee43926c96c0d6a9ec81a648754c1af599ef0bdd/sysrq_w.log

Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 1 line

Hm, so it looks like this is stuck between freeze:

[377279.630957] fsfreeze        D    0 46819  46337 0x00004084
[377279.634910] Call Trace:
[377279.637594]  ? __schedule+0x292/0x6f0
[377279.640833]  ? xfs_xattr_get+0x51/0x80 [xfs]
[377279.644287]  schedule+0x2f/0xa0
[377279.647286]  schedule_timeout+0x1dd/0x300
[377279.650661]  wait_for_completion+0x126/0x190
[377279.654154]  ? wake_up_q+0x80/0x80
[377279.657277]  ? work_busy+0x80/0x80
[377279.660375]  __flush_work+0x177/0x1b0
[377279.663604]  ? worker_attach_to_pool+0x90/0x90
[377279.667121]  __cancel_work_timer+0x12b/0x1b0
[377279.670571]  ? rcu_sync_enter+0x8b/0xd0
[377279.673864]  xfs_stop_block_reaping+0x15/0x30 [xfs]
[377279.677585]  xfs_fs_freeze+0x15/0x40 [xfs]
[377279.680950]  freeze_super+0xc8/0x190
[377279.684086]  do_vfs_ioctl+0x510/0x630

...

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

... and the eofblocks scanner:

[377279.422496] Workqueue: xfs-eofblocks/nvme13n1 xfs_eofblocks_worker

[xfs]

Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
[377279.426971] Call Trace:
[377279.429662]  ? __schedule+0x292/0x6f0
[377279.432839]  schedule+0x2f/0xa0
[377279.435794]  rwsem_down_read_slowpath+0x196/0x530
[377279.439435]  ? kmem_cache_alloc+0x152/0x1f0
[377279.442834]  ? __percpu_down_read+0x49/0x60
[377279.446242]  __percpu_down_read+0x49/0x60
[377279.449586]  __sb_start_write+0x5b/0x60
[377279.452869]  xfs_trans_alloc+0x152/0x160 [xfs]
[377279.456372]  xfs_free_eofblocks+0x12d/0x1f0 [xfs]
[377279.460014]  xfs_inode_free_eofblocks+0x128/0x1a0 [xfs]
[377279.463903]  ? xfs_inode_ag_walk_grab+0x5f/0x90 [xfs]
[377279.467680]  xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs]
[377279.471567]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.475620]  ? kvm_sched_clock_read+0xd/0x20
[377279.479059]  ? sched_clock+0x5/0x10
[377279.482184]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.486234]  ? radix_tree_gang_lookup_tag+0xa8/0x100
[377279.489974]  ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs]
[377279.494041]  xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs]
[377279.497859]  xfs_eofblocks_worker+0x29/0x40 [xfs]
[377279.501484]  process_one_work+0x195/0x380

...

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

The immediate issue is likely that the eofblocks transaction is not NOWRITECOUNT (same for the cowblocks scanner, btw), but the problem

with

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

doing that is these helpers are called from other contexts outside of the background scanners.

Perhaps what we need to do here is let these background scanners

acquire

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

a superblock write reference, similar to what Darrick recently added to scrub..? We'd have to do that from the scanner workqueue task, so it would probably need to be a trylock so we don't end up in a similar situation as above. I.e., we'd either get the reference and cause

freeze

Quoted 1 line
Quoted 1 line
Quoted 1 line

to wait until it's dropped or bail out if freeze has already stopped

the

Quoted 10 lines
Quoted 9 lines
Quoted 1 line

transaction subsystem. Thoughts?

Hmm, I had a whole gigantic series to refactor all the speculative preallocation gc work into a single thread + radix tree tag; I'll see if that series actually fixed this problem too.

But yes, all background threads that run transactions need to have freezer protection.

Quoted 8 lines
Quoted 4 lines

So something like the following in the meantime, assuming we want a backportable fix..? I think this means we could return -EAGAIN from the eofblocks ioctl, but afaict if something functionally conflicts with an active scan across freeze then perhaps that's preferred.

Apparently I don't have a patch that fixes the speculative gc code. The deferred inactivation worker does it, so perhaps I got mixed up. :/

Ok.

Quoted 10 lines

I think a better fix would be to annotate xfs_icache_free_eofblocks and xfs_icache_free_cowblocks to note that the caller must obtain freeze protection before calling those functions. Then we can play whackamole with the existing callers:

1. xfs_eofblocks_worker and xfs_cowblocks_worker can try to sb_start_write and just go back to sleep if the fs is frozen. The flush_workqueue will then cancel the delayed work and the freeze can proceed.

Quoted 6 lines

2. The buffered write ENOSPC scour-and-retry loops already have freeze protection because they're file writes, so they don't have to change.

3. XFS_IOC_FREE_EOFBLOCKS can sb_start_write, which means that callers will sleep on the frozen fs.

Sure, that works for me. It can be condensed later if it ends up in a single thread.

Brian

Quoted 53 lines

--D

Quoted 50 lines

Brian

fs/xfs/xfs_icache.c +14 -3
index a7be7a9e5c1a..0f14d58e5bb0 100644
@@ -1515,13 +1515,24 @@ __xfs_icache_free_eofblocks(
                                       void *args),
    int                     tag)
 {
-   int flags = SYNC_TRYLOCK;
+   int                     flags = SYNC_TRYLOCK;
+   int                     error;
 
    if (eofb && (eofb->eof_flags & XFS_EOF_FLAGS_SYNC))
            flags = SYNC_WAIT;
 
-   return xfs_inode_ag_iterator_tag(mp, execute, flags,
-                                    eofb, tag);
+   /*
+    * freeze waits on background scanner jobs to complete so we cannot
+    * block on write protection here. Bail if the transaction subsystem is
+    * already freezing, returning -EAGAIN to notify other callers.
+    */
+   if (!sb_start_write_trylock(mp->m_super))
+           return -EAGAIN;
+
+   error = xfs_inode_ag_iterator_tag(mp, execute, flags, eofb, tag);
+   sb_end_write(mp->m_super);
+
+   return error;
 }
 
 int
Quoted 15 lines

--D

Quoted 11 lines

Brian

Quoted 8 lines

Thanks,
Paul

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#14 lore raw

[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #8 from Paul Furtado ([email protected]) ---
The patches that came from this issue have given us many weeks of stability now and we were ready to declare this as totally fixed, however, we hit another instance of this issue this week which I'm assuming is probably on a slightly different and much rarer code path.

Here's a link to the blocked tasks log (beware that it's 2MB due to endless processes getting hung inside the container once the filesystem was frozen):
https://gist.githubusercontent.com/PaulFurtado/48253a6978763671f70dc94d933df851/raw/6bad12023ac56e9b6cb3dde771fcb5b15f0bd679/patched_kernel_fsfreeze_sys_w.log

Thanks,
Paul

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#15 lore raw

Re: [Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)

Quoted 8 lines · On Thu, May 28, 2020 at 06:00:38AM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #8 from Paul Furtado ([email protected]) ---
The patches that came from this issue have given us many weeks of stability now and we were ready to declare this as totally fixed, however, we hit another instance of this issue this week which I'm assuming is probably on a slightly different and much rarer code path.

Quoted 4 lines

Here's a link to the blocked tasks log (beware that it's 2MB due to endless processes getting hung inside the container once the filesystem was frozen):
https://gist.githubusercontent.com/PaulFurtado/48253a6978763671f70dc94d933df851/raw/6bad12023ac56e9b6cb3dde771fcb5b15f0bd679/patched_kernel_fsfreeze_sys_w.log

This shows the eofblocks scan in the following (massaged) trace:

[1259466.349224] Workqueue: xfs-eofblocks/nvme4n1 xfs_eofblocks_worker [xfs]
[1259466.353550] Call Trace:
[1259466.359370]  schedule+0x2f/0xa0
[1259466.362297]  rwsem_down_read_slowpath+0x196/0x530
[1259466.372467]  __percpu_down_read+0x49/0x60
[1259466.375778]  __sb_start_write+0x5b/0x60
[1259466.379139]  xfs_trans_alloc+0x152/0x160 [xfs]
[1259466.382715]  xfs_free_eofblocks+0x12d/0x1f0 [xfs]
[1259466.386407]  xfs_inode_free_eofblocks+0x128/0x1a0 [xfs]
[1259466.394058]  xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs]
[1259466.536551]  xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs]
[1259466.540235]  xfs_eofblocks_worker+0x29/0x40 [xfs]
[1259466.543748]  process_one_work+0x195/0x380
[1259466.546996]  worker_thread+0x30/0x390
[1259466.553449]  kthread+0x113/0x130
[1259466.559579]  ret_from_fork+0x1f/0x40

This should be addressed by upstream commit 4b674b9ac8529 ("xfs: acquire superblock freeze protection on eofblocks scans"), which causes xfs_eofblocks_worker() to bail unless it acquires freeze write protection. What exact kernel is this seen on?

Brian

Quoted 7 lines

Thanks,
Paul

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#16 lore raw

[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #9 from [email protected] ---

Quoted 9 lines · On Thu, May 28, 2020 at 06:00:38AM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #8 from Paul Furtado ([email protected]) --- The patches that came from this issue have given us many weeks of stability now
and we were ready to declare this as totally fixed, however, we hit another instance of this issue this week which I'm assuming is probably on a slightly different and much rarer code path.

Quoted 5 lines

Here's a link to the blocked tasks log (beware that it's 2MB due to endless processes getting hung inside the container once the filesystem was frozen):

https://gist.githubusercontent.com/PaulFurtado/48253a6978763671f70dc94d933df851/raw/6bad12023ac56e9b6cb3dde771fcb5b15f0bd679/patched_kernel_fsfreeze_sys_w.log

This shows the eofblocks scan in the following (massaged) trace:

[1259466.349224] Workqueue: xfs-eofblocks/nvme4n1 xfs_eofblocks_worker [xfs]
[1259466.353550] Call Trace:
[1259466.359370]  schedule+0x2f/0xa0
[1259466.362297]  rwsem_down_read_slowpath+0x196/0x530
[1259466.372467]  __percpu_down_read+0x49/0x60
[1259466.375778]  __sb_start_write+0x5b/0x60
[1259466.379139]  xfs_trans_alloc+0x152/0x160 [xfs]
[1259466.382715]  xfs_free_eofblocks+0x12d/0x1f0 [xfs]
[1259466.386407]  xfs_inode_free_eofblocks+0x128/0x1a0 [xfs]
[1259466.394058]  xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs]
[1259466.536551]  xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs]
[1259466.540235]  xfs_eofblocks_worker+0x29/0x40 [xfs]
[1259466.543748]  process_one_work+0x195/0x380
[1259466.546996]  worker_thread+0x30/0x390
[1259466.553449]  kthread+0x113/0x130
[1259466.559579]  ret_from_fork+0x1f/0x40

This should be addressed by upstream commit 4b674b9ac8529 ("xfs: acquire superblock freeze protection on eofblocks scans"), which causes xfs_eofblocks_worker() to bail unless it acquires freeze write protection. What exact kernel is this seen on?

Brian

Quoted 7 lines

Thanks,
Paul

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#17 lore raw

[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207053

--- Comment #10 from Paul Furtado ([email protected]) --- This is my mistake actually: it turns out this host had slipped through the cracks in our configuration management and didn't actually get updated to the kernel build with that commit in it, I had thought we'd covered every host. Looks like we're actually in the clear.

Thanks,
Paul

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#18 lore raw

[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207053

[email protected] changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #11 from [email protected] ---
I'm trying to figure out which kernel version this bug is fixed in. I see references to a patch above, but am having trouble tracking down when it was added into the baseline.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#19 lore raw

[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207053

Eric Sandeen ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #12 from Eric Sandeen ([email protected]) --- commit 4b674b9ac8529 should be in upstream kernel v5.7

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.