New: fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)
https://bugzilla.kernel.org/show_bug.cgi?id=207053
Bug ID: 207053
Summary: fsfreeze deadlock on XFS (the FIFREEZE ioctl and
subsequent FITHAW hang indefinitely)
Product: File System
Version: 2.5
Kernel Version: 4.19.75, 5.4.20
Hardware: x86-64
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: XFS
Assignee: [email protected]
Reporter: [email protected]
Regression: NoWhen we upgraded from kernel 4.14.146 to kernel 4.19.75, we began to experience frequent deadlocks from our cronjobs that freeze the filesystem for snapshotting.
The fsfreeze stack shows:
# cat /proc/33256/stack [<0>] __flush_work+0x177/0x1b0 [<0>] __cancel_work_timer+0x12b/0x1b0 [<0>] xfs_stop_block_reaping+0x15/0x30 [xfs] [<0>] xfs_fs_freeze+0x15/0x40 [xfs] [<0>] freeze_super+0xc8/0x190 [<0>] do_vfs_ioctl+0x510/0x630 [<0>] ksys_ioctl+0x70/0x80 [<0>] __x64_sys_ioctl+0x16/0x20 [<0>] do_syscall_64+0x4e/0x100 [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
The fsfreeze -u stack shows:
# cat /proc/37753/stack [<0>] rwsem_down_write_slowpath+0x257/0x510 [<0>] thaw_super+0x12/0x20 [<0>] do_vfs_ioctl+0x609/0x630 [<0>] ksys_ioctl+0x70/0x80 [<0>] __x64_sys_ioctl+0x16/0x20 [<0>] do_syscall_64+0x4e/0x100 [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
Echoing "j" into /proc/sysrq-trigger to emergency thaw all filesystems doesn't solve this either. We're hitting this bug many times per week, so if there's any more debug information you need that we could turn on, let us know. Thanks!
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
Quoted 3 lines · On Wed, Apr 01, 2020 at 07:02:57PM +0000, [email protected]…
When we upgraded from kernel 4.14.146 to kernel 4.19.75, we began to experience frequent deadlocks from our cronjobs that freeze the filesystem for snapshotting.
Probably commit d6b636ebb1c9 ("xfs: halt auto-reclamation activities while rebuilding rmap") in 4.18, but that fixes a bug that allowed reaping functions to attempt to modify the filesystem while it was frozen...
Quoted 12 lines
The fsfreeze stack shows:
# cat /proc/33256/stack [<0>] __flush_work+0x177/0x1b0 [<0>] __cancel_work_timer+0x12b/0x1b0 [<0>] xfs_stop_block_reaping+0x15/0x30 [xfs] [<0>] xfs_fs_freeze+0x15/0x40 [xfs] [<0>] freeze_super+0xc8/0x190 [<0>] do_vfs_ioctl+0x510/0x630 [<0>] ksys_ioctl+0x70/0x80 [<0>] __x64_sys_ioctl+0x16/0x20 [<0>] do_syscall_64+0x4e/0x100 [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
This indicates that the reaping worker is still busy doing work. It needs to finish before the freeze will continue to make progress. So either the system is still doing work, or the kworker has blocked somewhere.
What is the dmesg output of 'echo w > /prox/sysrq-trigger'?
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #1 from Dave Chinner ([email protected]) ---
Quoted 4 lines · On Wed, Apr 01, 2020 at 07:02:57PM +0000, [email protected]…
When we upgraded from kernel 4.14.146 to kernel 4.19.75, we began to experience
frequent deadlocks from our cronjobs that freeze the filesystem for snapshotting.
Probably commit d6b636ebb1c9 ("xfs: halt auto-reclamation activities while rebuilding rmap") in 4.18, but that fixes a bug that allowed reaping functions to attempt to modify the filesystem while it was frozen...
Quoted 12 lines
The fsfreeze stack shows:
# cat /proc/33256/stack [<0>] __flush_work+0x177/0x1b0 [<0>] __cancel_work_timer+0x12b/0x1b0 [<0>] xfs_stop_block_reaping+0x15/0x30 [xfs] [<0>] xfs_fs_freeze+0x15/0x40 [xfs] [<0>] freeze_super+0xc8/0x190 [<0>] do_vfs_ioctl+0x510/0x630 [<0>] ksys_ioctl+0x70/0x80 [<0>] __x64_sys_ioctl+0x16/0x20 [<0>] do_syscall_64+0x4e/0x100 [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
This indicates that the reaping worker is still busy doing work. It needs to finish before the freeze will continue to make progress. So either the system is still doing work, or the kworker has blocked somewhere.
What is the dmesg output of 'echo w > /prox/sysrq-trigger'?
Cheers,
Dave.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,
Just had another case of this crop up and I was able to get the blocked tasks output before automation killed the server. Because the log was too large to attach, I've pasted the output into a github gist here:
https://gist.githubusercontent.com/PaulFurtado/c9bade038b8a5c7ddb53a6e10def058f/raw/ee43926c96c0d6a9ec81a648754c1af599ef0bdd/sysrq_w.log
Thanks,
Paul
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #3 from [email protected] ---
Quoted 11 lines · On Tue, Apr 07, 2020 at 06:41:31AM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,
Just had another case of this crop up and I was able to get the blocked tasks output before automation killed the server. Because the log was too large to attach, I've pasted the output into a github gist here:
Hm, so it looks like this is stuck between freeze:
[377279.630957] fsfreeze D 0 46819 46337 0x00004084 [377279.634910] Call Trace: [377279.637594] ? __schedule+0x292/0x6f0 [377279.640833] ? xfs_xattr_get+0x51/0x80 [xfs] [377279.644287] schedule+0x2f/0xa0 [377279.647286] schedule_timeout+0x1dd/0x300 [377279.650661] wait_for_completion+0x126/0x190 [377279.654154] ? wake_up_q+0x80/0x80 [377279.657277] ? work_busy+0x80/0x80 [377279.660375] __flush_work+0x177/0x1b0 [377279.663604] ? worker_attach_to_pool+0x90/0x90 [377279.667121] __cancel_work_timer+0x12b/0x1b0 [377279.670571] ? rcu_sync_enter+0x8b/0xd0 [377279.673864] xfs_stop_block_reaping+0x15/0x30 [xfs] [377279.677585] xfs_fs_freeze+0x15/0x40 [xfs] [377279.680950] freeze_super+0xc8/0x190 [377279.684086] do_vfs_ioctl+0x510/0x630
...
... and the eofblocks scanner:
[377279.422496] Workqueue: xfs-eofblocks/nvme13n1 xfs_eofblocks_worker [xfs] [377279.426971] Call Trace: [377279.429662] ? __schedule+0x292/0x6f0 [377279.432839] schedule+0x2f/0xa0 [377279.435794] rwsem_down_read_slowpath+0x196/0x530 [377279.439435] ? kmem_cache_alloc+0x152/0x1f0 [377279.442834] ? __percpu_down_read+0x49/0x60 [377279.446242] __percpu_down_read+0x49/0x60 [377279.449586] __sb_start_write+0x5b/0x60 [377279.452869] xfs_trans_alloc+0x152/0x160 [xfs] [377279.456372] xfs_free_eofblocks+0x12d/0x1f0 [xfs] [377279.460014] xfs_inode_free_eofblocks+0x128/0x1a0 [xfs] [377279.463903] ? xfs_inode_ag_walk_grab+0x5f/0x90 [xfs] [377279.467680] xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs] [377279.471567] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.475620] ? kvm_sched_clock_read+0xd/0x20 [377279.479059] ? sched_clock+0x5/0x10 [377279.482184] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.486234] ? radix_tree_gang_lookup_tag+0xa8/0x100 [377279.489974] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.494041] xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs] [377279.497859] xfs_eofblocks_worker+0x29/0x40 [xfs] [377279.501484] process_one_work+0x195/0x380
...
The immediate issue is likely that the eofblocks transaction is not NOWRITECOUNT (same for the cowblocks scanner, btw), but the problem with doing that is these helpers are called from other contexts outside of the background scanners.
Perhaps what we need to do here is let these background scanners acquire a superblock write reference, similar to what Darrick recently added to scrub..? We'd have to do that from the scanner workqueue task, so it would probably need to be a trylock so we don't end up in a similar situation as above. I.e., we'd either get the reference and cause freeze to wait until it's dropped or bail out if freeze has already stopped the transaction subsystem. Thoughts?
Brian
Quoted 8 lines
Thanks,
PaulSignature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
Re: [Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)
In reply to [email protected]Quoted 10 lines · On Tue, Apr 07, 2020 at 06:41:31AM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,
Just had another case of this crop up and I was able to get the blocked tasks output before automation killed the server. Because the log was too large to attach, I've pasted the output into a github gist here:
https://gist.githubusercontent.com/PaulFurtado/c9bade038b8a5c7ddb53a6e10def058f/raw/ee43926c96c0d6a9ec81a648754c1af599ef0bdd/sysrq_w.log
Hm, so it looks like this is stuck between freeze:
[377279.630957] fsfreeze D 0 46819 46337 0x00004084 [377279.634910] Call Trace: [377279.637594] ? __schedule+0x292/0x6f0 [377279.640833] ? xfs_xattr_get+0x51/0x80 [xfs] [377279.644287] schedule+0x2f/0xa0 [377279.647286] schedule_timeout+0x1dd/0x300 [377279.650661] wait_for_completion+0x126/0x190 [377279.654154] ? wake_up_q+0x80/0x80 [377279.657277] ? work_busy+0x80/0x80 [377279.660375] __flush_work+0x177/0x1b0 [377279.663604] ? worker_attach_to_pool+0x90/0x90 [377279.667121] __cancel_work_timer+0x12b/0x1b0 [377279.670571] ? rcu_sync_enter+0x8b/0xd0 [377279.673864] xfs_stop_block_reaping+0x15/0x30 [xfs] [377279.677585] xfs_fs_freeze+0x15/0x40 [xfs] [377279.680950] freeze_super+0xc8/0x190 [377279.684086] do_vfs_ioctl+0x510/0x630
...
... and the eofblocks scanner:
[377279.422496] Workqueue: xfs-eofblocks/nvme13n1 xfs_eofblocks_worker [xfs] [377279.426971] Call Trace: [377279.429662] ? __schedule+0x292/0x6f0 [377279.432839] schedule+0x2f/0xa0 [377279.435794] rwsem_down_read_slowpath+0x196/0x530 [377279.439435] ? kmem_cache_alloc+0x152/0x1f0 [377279.442834] ? __percpu_down_read+0x49/0x60 [377279.446242] __percpu_down_read+0x49/0x60 [377279.449586] __sb_start_write+0x5b/0x60 [377279.452869] xfs_trans_alloc+0x152/0x160 [xfs] [377279.456372] xfs_free_eofblocks+0x12d/0x1f0 [xfs] [377279.460014] xfs_inode_free_eofblocks+0x128/0x1a0 [xfs] [377279.463903] ? xfs_inode_ag_walk_grab+0x5f/0x90 [xfs] [377279.467680] xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs] [377279.471567] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.475620] ? kvm_sched_clock_read+0xd/0x20 [377279.479059] ? sched_clock+0x5/0x10 [377279.482184] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.486234] ? radix_tree_gang_lookup_tag+0xa8/0x100 [377279.489974] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.494041] xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs] [377279.497859] xfs_eofblocks_worker+0x29/0x40 [xfs] [377279.501484] process_one_work+0x195/0x380
...
The immediate issue is likely that the eofblocks transaction is not NOWRITECOUNT (same for the cowblocks scanner, btw), but the problem with doing that is these helpers are called from other contexts outside of the background scanners.
Perhaps what we need to do here is let these background scanners acquire a superblock write reference, similar to what Darrick recently added to scrub..? We'd have to do that from the scanner workqueue task, so it would probably need to be a trylock so we don't end up in a similar situation as above. I.e., we'd either get the reference and cause freeze to wait until it's dropped or bail out if freeze has already stopped the transaction subsystem. Thoughts?
Brian
Quoted 8 lines
Thanks,
PaulSignature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
Re: [Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)
Quoted 65 lines · Brian Foster
Quoted 10 lines · On Tue, Apr 07, 2020 at 06:41:31AM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,
Just had another case of this crop up and I was able to get the blocked tasks output before automation killed the server. Because the log was too large to attach, I've pasted the output into a github gist here:
https://gist.githubusercontent.com/PaulFurtado/c9bade038b8a5c7ddb53a6e10def058f/raw/ee43926c96c0d6a9ec81a648754c1af599ef0bdd/sysrq_w.logHm, so it looks like this is stuck between freeze:
[377279.630957] fsfreeze D 0 46819 46337 0x00004084 [377279.634910] Call Trace: [377279.637594] ? __schedule+0x292/0x6f0 [377279.640833] ? xfs_xattr_get+0x51/0x80 [xfs] [377279.644287] schedule+0x2f/0xa0 [377279.647286] schedule_timeout+0x1dd/0x300 [377279.650661] wait_for_completion+0x126/0x190 [377279.654154] ? wake_up_q+0x80/0x80 [377279.657277] ? work_busy+0x80/0x80 [377279.660375] __flush_work+0x177/0x1b0 [377279.663604] ? worker_attach_to_pool+0x90/0x90 [377279.667121] __cancel_work_timer+0x12b/0x1b0 [377279.670571] ? rcu_sync_enter+0x8b/0xd0 [377279.673864] xfs_stop_block_reaping+0x15/0x30 [xfs] [377279.677585] xfs_fs_freeze+0x15/0x40 [xfs] [377279.680950] freeze_super+0xc8/0x190 [377279.684086] do_vfs_ioctl+0x510/0x630...
... and the eofblocks scanner:
[377279.422496] Workqueue: xfs-eofblocks/nvme13n1 xfs_eofblocks_worker [xfs] [377279.426971] Call Trace: [377279.429662] ? __schedule+0x292/0x6f0 [377279.432839] schedule+0x2f/0xa0 [377279.435794] rwsem_down_read_slowpath+0x196/0x530 [377279.439435] ? kmem_cache_alloc+0x152/0x1f0 [377279.442834] ? __percpu_down_read+0x49/0x60 [377279.446242] __percpu_down_read+0x49/0x60 [377279.449586] __sb_start_write+0x5b/0x60 [377279.452869] xfs_trans_alloc+0x152/0x160 [xfs] [377279.456372] xfs_free_eofblocks+0x12d/0x1f0 [xfs] [377279.460014] xfs_inode_free_eofblocks+0x128/0x1a0 [xfs] [377279.463903] ? xfs_inode_ag_walk_grab+0x5f/0x90 [xfs] [377279.467680] xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs] [377279.471567] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.475620] ? kvm_sched_clock_read+0xd/0x20 [377279.479059] ? sched_clock+0x5/0x10 [377279.482184] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.486234] ? radix_tree_gang_lookup_tag+0xa8/0x100 [377279.489974] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.494041] xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs] [377279.497859] xfs_eofblocks_worker+0x29/0x40 [xfs] [377279.501484] process_one_work+0x195/0x380...
The immediate issue is likely that the eofblocks transaction is not NOWRITECOUNT (same for the cowblocks scanner, btw), but the problem with doing that is these helpers are called from other contexts outside of the background scanners.
Quoted 7 lines
Perhaps what we need to do here is let these background scanners acquire a superblock write reference, similar to what Darrick recently added to scrub..? We'd have to do that from the scanner workqueue task, so it would probably need to be a trylock so we don't end up in a similar situation as above. I.e., we'd either get the reference and cause freeze to wait until it's dropped or bail out if freeze has already stopped the transaction subsystem. Thoughts?
Hmm, I had a whole gigantic series to refactor all the speculative preallocation gc work into a single thread + radix tree tag; I'll see if that series actually fixed this problem too.
But yes, all background threads that run transactions need to have freezer protection.
--D
Quoted 11 lines
Brian
Quoted 8 lines
Thanks,
PaulSignature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #4 from [email protected] ---
Quoted 69 lines · Brian Foster
Quoted 6 lines · On Tue, Apr 07, 2020 at 06:41:31AM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,
Just had another case of this crop up and I was able to get the blocked
tasks
Quoted 1 line
output before automation killed the server. Because the log was too large
to
Quoted 2 lines
attach, I've pasted the output into a github gist here:
Quoted 1 line
Hm, so it looks like this is stuck between freeze:
[377279.630957] fsfreeze D 0 46819 46337 0x00004084 [377279.634910] Call Trace: [377279.637594] ? __schedule+0x292/0x6f0 [377279.640833] ? xfs_xattr_get+0x51/0x80 [xfs] [377279.644287] schedule+0x2f/0xa0 [377279.647286] schedule_timeout+0x1dd/0x300 [377279.650661] wait_for_completion+0x126/0x190 [377279.654154] ? wake_up_q+0x80/0x80 [377279.657277] ? work_busy+0x80/0x80 [377279.660375] __flush_work+0x177/0x1b0 [377279.663604] ? worker_attach_to_pool+0x90/0x90 [377279.667121] __cancel_work_timer+0x12b/0x1b0 [377279.670571] ? rcu_sync_enter+0x8b/0xd0 [377279.673864] xfs_stop_block_reaping+0x15/0x30 [xfs] [377279.677585] xfs_fs_freeze+0x15/0x40 [xfs] [377279.680950] freeze_super+0xc8/0x190 [377279.684086] do_vfs_ioctl+0x510/0x630...
... and the eofblocks scanner:
[377279.422496] Workqueue: xfs-eofblocks/nvme13n1 xfs_eofblocks_worker [xfs] [377279.426971] Call Trace: [377279.429662] ? __schedule+0x292/0x6f0 [377279.432839] schedule+0x2f/0xa0 [377279.435794] rwsem_down_read_slowpath+0x196/0x530 [377279.439435] ? kmem_cache_alloc+0x152/0x1f0 [377279.442834] ? __percpu_down_read+0x49/0x60 [377279.446242] __percpu_down_read+0x49/0x60 [377279.449586] __sb_start_write+0x5b/0x60 [377279.452869] xfs_trans_alloc+0x152/0x160 [xfs] [377279.456372] xfs_free_eofblocks+0x12d/0x1f0 [xfs] [377279.460014] xfs_inode_free_eofblocks+0x128/0x1a0 [xfs] [377279.463903] ? xfs_inode_ag_walk_grab+0x5f/0x90 [xfs] [377279.467680] xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs] [377279.471567] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.475620] ? kvm_sched_clock_read+0xd/0x20 [377279.479059] ? sched_clock+0x5/0x10 [377279.482184] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.486234] ? radix_tree_gang_lookup_tag+0xa8/0x100 [377279.489974] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.494041] xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs] [377279.497859] xfs_eofblocks_worker+0x29/0x40 [xfs] [377279.501484] process_one_work+0x195/0x380...
The immediate issue is likely that the eofblocks transaction is not NOWRITECOUNT (same for the cowblocks scanner, btw), but the problem with doing that is these helpers are called from other contexts outside of the background scanners.
Quoted 7 lines
Perhaps what we need to do here is let these background scanners acquire a superblock write reference, similar to what Darrick recently added to scrub..? We'd have to do that from the scanner workqueue task, so it would probably need to be a trylock so we don't end up in a similar situation as above. I.e., we'd either get the reference and cause freeze to wait until it's dropped or bail out if freeze has already stopped the transaction subsystem. Thoughts?
Hmm, I had a whole gigantic series to refactor all the speculative preallocation gc work into a single thread + radix tree tag; I'll see if that series actually fixed this problem too.
But yes, all background threads that run transactions need to have freezer protection.
--D
Quoted 11 lines
Brian
Quoted 8 lines
Thanks,
PaulSignature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
Re: [Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)
In reply to Darrick J. WongQuoted 74 lines · Darrick J. Wong
Quoted 65 lines · Brian Foster
Quoted 10 lines · On Tue, Apr 07, 2020 at 06:41:31AM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,
Just had another case of this crop up and I was able to get the blocked tasks output before automation killed the server. Because the log was too large to attach, I've pasted the output into a github gist here:
https://gist.githubusercontent.com/PaulFurtado/c9bade038b8a5c7ddb53a6e10def058f/raw/ee43926c96c0d6a9ec81a648754c1af599ef0bdd/sysrq_w.logHm, so it looks like this is stuck between freeze:
[377279.630957] fsfreeze D 0 46819 46337 0x00004084 [377279.634910] Call Trace: [377279.637594] ? __schedule+0x292/0x6f0 [377279.640833] ? xfs_xattr_get+0x51/0x80 [xfs] [377279.644287] schedule+0x2f/0xa0 [377279.647286] schedule_timeout+0x1dd/0x300 [377279.650661] wait_for_completion+0x126/0x190 [377279.654154] ? wake_up_q+0x80/0x80 [377279.657277] ? work_busy+0x80/0x80 [377279.660375] __flush_work+0x177/0x1b0 [377279.663604] ? worker_attach_to_pool+0x90/0x90 [377279.667121] __cancel_work_timer+0x12b/0x1b0 [377279.670571] ? rcu_sync_enter+0x8b/0xd0 [377279.673864] xfs_stop_block_reaping+0x15/0x30 [xfs] [377279.677585] xfs_fs_freeze+0x15/0x40 [xfs] [377279.680950] freeze_super+0xc8/0x190 [377279.684086] do_vfs_ioctl+0x510/0x630...
... and the eofblocks scanner:
[377279.422496] Workqueue: xfs-eofblocks/nvme13n1 xfs_eofblocks_worker [xfs] [377279.426971] Call Trace: [377279.429662] ? __schedule+0x292/0x6f0 [377279.432839] schedule+0x2f/0xa0 [377279.435794] rwsem_down_read_slowpath+0x196/0x530 [377279.439435] ? kmem_cache_alloc+0x152/0x1f0 [377279.442834] ? __percpu_down_read+0x49/0x60 [377279.446242] __percpu_down_read+0x49/0x60 [377279.449586] __sb_start_write+0x5b/0x60 [377279.452869] xfs_trans_alloc+0x152/0x160 [xfs] [377279.456372] xfs_free_eofblocks+0x12d/0x1f0 [xfs] [377279.460014] xfs_inode_free_eofblocks+0x128/0x1a0 [xfs] [377279.463903] ? xfs_inode_ag_walk_grab+0x5f/0x90 [xfs] [377279.467680] xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs] [377279.471567] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.475620] ? kvm_sched_clock_read+0xd/0x20 [377279.479059] ? sched_clock+0x5/0x10 [377279.482184] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.486234] ? radix_tree_gang_lookup_tag+0xa8/0x100 [377279.489974] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.494041] xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs] [377279.497859] xfs_eofblocks_worker+0x29/0x40 [xfs] [377279.501484] process_one_work+0x195/0x380...
The immediate issue is likely that the eofblocks transaction is not NOWRITECOUNT (same for the cowblocks scanner, btw), but the problem with doing that is these helpers are called from other contexts outside of the background scanners.
Quoted 7 lines
Perhaps what we need to do here is let these background scanners acquire a superblock write reference, similar to what Darrick recently added to scrub..? We'd have to do that from the scanner workqueue task, so it would probably need to be a trylock so we don't end up in a similar situation as above. I.e., we'd either get the reference and cause freeze to wait until it's dropped or bail out if freeze has already stopped the transaction subsystem. Thoughts?
Quoted 7 lines
Hmm, I had a whole gigantic series to refactor all the speculative preallocation gc work into a single thread + radix tree tag; I'll see if that series actually fixed this problem too.
But yes, all background threads that run transactions need to have freezer protection.
So something like the following in the meantime, assuming we want a backportable fix..? I think this means we could return -EAGAIN from the eofblocks ioctl, but afaict if something functionally conflicts with an active scan across freeze then perhaps that's preferred.
Brian
fs/xfs/xfs_icache.c +14 -3
@@ -1515,13 +1515,24 @@ __xfs_icache_free_eofblocks( void *args), int tag) { - int flags = SYNC_TRYLOCK; + int flags = SYNC_TRYLOCK; +int error;if (eofb && (eofb->eof_flags &XFS_EOF_FLAGS_SYNC )) flags =SYNC_WAIT ; -return xfs_inode_ag_iterator_tag (mp, execute, flags, - eofb, tag); +/* +* freeze waits on background scanner jobs to complete so we cannot +* block on write protection here. Bail if the transaction subsystem is +* already freezing, returning -EAGAIN to notify other callers. +*/ +if (!sb_start_write_trylock (mp->m_super)) +return -EAGAIN ; + + error =xfs_inode_ag_iterator_tag (mp, execute, flags, eofb, tag); +sb_end_write (mp->m_super); + +return error; }int
Quoted 14 lines
--D
Quoted 11 lines
Brian
Quoted 8 lines
Thanks,
PaulSignature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #5 from [email protected] ---
Quoted 79 lines · Darrick J. Wong
Quoted 1 line · Brian Foster
On Tue, Apr 07, 2020 at 06:41:31AM +0000,
Quoted 6 lines · [email protected] wrote:
Quoted 6 lines
https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,
Just had another case of this crop up and I was able to get the blocked
tasks
Quoted 1 line
Quoted 1 line
output before automation killed the server. Because the log was too large
to
Quoted 2 lines
Quoted 2 lines
attach, I've pasted the output into a github gist here:
Quoted 21 lines
Quoted 1 line
Hm, so it looks like this is stuck between freeze:
[377279.630957] fsfreeze D 0 46819 46337 0x00004084 [377279.634910] Call Trace: [377279.637594] ? __schedule+0x292/0x6f0 [377279.640833] ? xfs_xattr_get+0x51/0x80 [xfs] [377279.644287] schedule+0x2f/0xa0 [377279.647286] schedule_timeout+0x1dd/0x300 [377279.650661] wait_for_completion+0x126/0x190 [377279.654154] ? wake_up_q+0x80/0x80 [377279.657277] ? work_busy+0x80/0x80 [377279.660375] __flush_work+0x177/0x1b0 [377279.663604] ? worker_attach_to_pool+0x90/0x90 [377279.667121] __cancel_work_timer+0x12b/0x1b0 [377279.670571] ? rcu_sync_enter+0x8b/0xd0 [377279.673864] xfs_stop_block_reaping+0x15/0x30 [xfs] [377279.677585] xfs_fs_freeze+0x15/0x40 [xfs] [377279.680950] freeze_super+0xc8/0x190 [377279.684086] do_vfs_ioctl+0x510/0x630Quoted 5 lines
...
... and the eofblocks scanner:
[377279.422496] Workqueue: xfs-eofblocks/nvme13n1 xfs_eofblocks_worker[xfs]
Quoted 29 lines
[377279.426971] Call Trace: [377279.429662] ? __schedule+0x292/0x6f0 [377279.432839] schedule+0x2f/0xa0 [377279.435794] rwsem_down_read_slowpath+0x196/0x530 [377279.439435] ? kmem_cache_alloc+0x152/0x1f0 [377279.442834] ? __percpu_down_read+0x49/0x60 [377279.446242] __percpu_down_read+0x49/0x60 [377279.449586] __sb_start_write+0x5b/0x60 [377279.452869] xfs_trans_alloc+0x152/0x160 [xfs] [377279.456372] xfs_free_eofblocks+0x12d/0x1f0 [xfs] [377279.460014] xfs_inode_free_eofblocks+0x128/0x1a0 [xfs] [377279.463903] ? xfs_inode_ag_walk_grab+0x5f/0x90 [xfs] [377279.467680] xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs] [377279.471567] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.475620] ? kvm_sched_clock_read+0xd/0x20 [377279.479059] ? sched_clock+0x5/0x10 [377279.482184] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.486234] ? radix_tree_gang_lookup_tag+0xa8/0x100 [377279.489974] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.494041] xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs] [377279.497859] xfs_eofblocks_worker+0x29/0x40 [xfs] [377279.501484] process_one_work+0x195/0x380...
The immediate issue is likely that the eofblocks transaction is not NOWRITECOUNT (same for the cowblocks scanner, btw), but the problem with doing that is these helpers are called from other contexts outside of the background scanners.
Quoted 7 lines
Perhaps what we need to do here is let these background scanners acquire a superblock write reference, similar to what Darrick recently added to scrub..? We'd have to do that from the scanner workqueue task, so it would probably need to be a trylock so we don't end up in a similar situation as above. I.e., we'd either get the reference and cause freeze to wait until it's dropped or bail out if freeze has already stopped the transaction subsystem. Thoughts?
Quoted 7 lines
Hmm, I had a whole gigantic series to refactor all the speculative preallocation gc work into a single thread + radix tree tag; I'll see if that series actually fixed this problem too.
But yes, all background threads that run transactions need to have freezer protection.
So something like the following in the meantime, assuming we want a backportable fix..? I think this means we could return -EAGAIN from the eofblocks ioctl, but afaict if something functionally conflicts with an active scan across freeze then perhaps that's preferred.
Brian
fs/xfs/xfs_icache.c +14 -3
@@ -1515,13 +1515,24 @@ __xfs_icache_free_eofblocks( void *args), int tag) { - int flags = SYNC_TRYLOCK; + int flags = SYNC_TRYLOCK; +int error;if (eofb && (eofb->eof_flags &XFS_EOF_FLAGS_SYNC )) flags =SYNC_WAIT ; -return xfs_inode_ag_iterator_tag (mp, execute, flags, - eofb, tag); +/* +* freeze waits on background scanner jobs to complete so we cannot +* block on write protection here. Bail if the transaction subsystem is +* already freezing, returning -EAGAIN to notify other callers. +*/ +if (!sb_start_write_trylock (mp->m_super)) +return -EAGAIN ; + + error =xfs_inode_ag_iterator_tag (mp, execute, flags, eofb, tag); +sb_end_write (mp->m_super); + +return error; }int
Quoted 14 lines
--D
Quoted 11 lines
Brian
Quoted 8 lines
Thanks,
PaulSignature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #6 from [email protected] ---
Quoted 84 lines · Brian Foster
Quoted 2 lines · Darrick J. Wong
Quoted 1 line · Brian Foster
On Tue, Apr 07, 2020 at 06:41:31AM +0000,
Quoted 6 lines · [email protected] wrote:
Quoted 6 lines
Quoted 6 lines
https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,
Just had another case of this crop up and I was able to get the blocked
tasks
Quoted 1 line
Quoted 1 line
Quoted 1 line
output before automation killed the server. Because the log was too
large to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
attach, I've pasted the output into a github gist here:
Quoted 23 lines
Quoted 23 lines
Quoted 1 line
Hm, so it looks like this is stuck between freeze:
[377279.630957] fsfreeze D 0 46819 46337 0x00004084 [377279.634910] Call Trace: [377279.637594] ? __schedule+0x292/0x6f0 [377279.640833] ? xfs_xattr_get+0x51/0x80 [xfs] [377279.644287] schedule+0x2f/0xa0 [377279.647286] schedule_timeout+0x1dd/0x300 [377279.650661] wait_for_completion+0x126/0x190 [377279.654154] ? wake_up_q+0x80/0x80 [377279.657277] ? work_busy+0x80/0x80 [377279.660375] __flush_work+0x177/0x1b0 [377279.663604] ? worker_attach_to_pool+0x90/0x90 [377279.667121] __cancel_work_timer+0x12b/0x1b0 [377279.670571] ? rcu_sync_enter+0x8b/0xd0 [377279.673864] xfs_stop_block_reaping+0x15/0x30 [xfs] [377279.677585] xfs_fs_freeze+0x15/0x40 [xfs] [377279.680950] freeze_super+0xc8/0x190 [377279.684086] do_vfs_ioctl+0x510/0x630...
Quoted 3 lines
Quoted 3 lines
... and the eofblocks scanner:
[377279.422496] Workqueue: xfs-eofblocks/nvme13n1 xfs_eofblocks_worker[xfs]
Quoted 41 lines
Quoted 29 lines
[377279.426971] Call Trace: [377279.429662] ? __schedule+0x292/0x6f0 [377279.432839] schedule+0x2f/0xa0 [377279.435794] rwsem_down_read_slowpath+0x196/0x530 [377279.439435] ? kmem_cache_alloc+0x152/0x1f0 [377279.442834] ? __percpu_down_read+0x49/0x60 [377279.446242] __percpu_down_read+0x49/0x60 [377279.449586] __sb_start_write+0x5b/0x60 [377279.452869] xfs_trans_alloc+0x152/0x160 [xfs] [377279.456372] xfs_free_eofblocks+0x12d/0x1f0 [xfs] [377279.460014] xfs_inode_free_eofblocks+0x128/0x1a0 [xfs] [377279.463903] ? xfs_inode_ag_walk_grab+0x5f/0x90 [xfs] [377279.467680] xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs] [377279.471567] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.475620] ? kvm_sched_clock_read+0xd/0x20 [377279.479059] ? sched_clock+0x5/0x10 [377279.482184] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.486234] ? radix_tree_gang_lookup_tag+0xa8/0x100 [377279.489974] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.494041] xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs] [377279.497859] xfs_eofblocks_worker+0x29/0x40 [xfs] [377279.501484] process_one_work+0x195/0x380...
The immediate issue is likely that the eofblocks transaction is not NOWRITECOUNT (same for the cowblocks scanner, btw), but the problem with doing that is these helpers are called from other contexts outside of the background scanners.
Quoted 7 lines
Perhaps what we need to do here is let these background scanners acquire a superblock write reference, similar to what Darrick recently added to scrub..? We'd have to do that from the scanner workqueue task, so it would probably need to be a trylock so we don't end up in a similar situation as above. I.e., we'd either get the reference and cause freeze to wait until it's dropped or bail out if freeze has already stopped the transaction subsystem. Thoughts?
Hmm, I had a whole gigantic series to refactor all the speculative preallocation gc work into a single thread + radix tree tag; I'll see if that series actually fixed this problem too.
Quoted 8 lines
Quoted 3 lines
But yes, all background threads that run transactions need to have freezer protection.
So something like the following in the meantime, assuming we want a backportable fix..? I think this means we could return
-EAGAINfrom the eofblocks ioctl, but afaict if something functionally conflicts with an active scan across freeze then perhaps that's preferred.
Apparently I don't have a patch that fixes the speculative gc code. The deferred inactivation worker does it, so perhaps I got mixed up. :/
I think a better fix would be to annotate xfs_icache_free_eofblocks and xfs_icache_free_cowblocks to note that the caller must obtain freeze protection before calling those functions. Then we can play whackamole with the existing callers:
1. xfs_eofblocks_worker and xfs_cowblocks_worker can try to sb_start_write and just go back to sleep if the fs is frozen. The flush_workqueue will then cancel the delayed work and the freeze can proceed.
2. The buffered write ENOSPC scour-and-retry loops already have freeze protection because they're file writes, so they don't have to change.
3. XFS_IOC_FREE_EOFBLOCKS can sb_start_write, which means that callers will sleep on the frozen fs.
--D
Quoted 50 lines
Brian
fs/xfs/xfs_icache.c +14
-3@@ -1515,13 +1515,24 @@ __xfs_icache_free_eofblocks( void *args), int tag) { - int flags = SYNC_TRYLOCK; + int flags = SYNC_TRYLOCK; +int error;if (eofb && (eofb->eof_flags &XFS_EOF_FLAGS_SYNC )) flags =SYNC_WAIT ; -return xfs_inode_ag_iterator_tag (mp, execute, flags, - eofb, tag); +/* +* freeze waits on background scanner jobs to complete so we cannot +* block on write protection here. Bail if the transaction subsystem is +* already freezing, returning -EAGAIN to notify other callers. +*/ +if (!sb_start_write_trylock (mp->m_super)) +return -EAGAIN ; + + error =xfs_inode_ag_iterator_tag (mp, execute, flags, eofb, tag); +sb_end_write (mp->m_super); + +return error; }int Quoted 15 lines
--D
Quoted 11 lines
Brian
Quoted 8 lines
Thanks,
PaulSignature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
Re: [Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)
In reply to Brian FosterQuoted 79 lines · Brian Foster
Quoted 78 lines · Darrick J. Wong
Quoted 65 lines · Brian Foster
Quoted 10 lines · On Tue, Apr 07, 2020 at 06:41:31AM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,
Just had another case of this crop up and I was able to get the blocked tasks output before automation killed the server. Because the log was too large to attach, I've pasted the output into a github gist here:
https://gist.githubusercontent.com/PaulFurtado/c9bade038b8a5c7ddb53a6e10def058f/raw/ee43926c96c0d6a9ec81a648754c1af599ef0bdd/sysrq_w.logHm, so it looks like this is stuck between freeze:
[377279.630957] fsfreeze D 0 46819 46337 0x00004084 [377279.634910] Call Trace: [377279.637594] ? __schedule+0x292/0x6f0 [377279.640833] ? xfs_xattr_get+0x51/0x80 [xfs] [377279.644287] schedule+0x2f/0xa0 [377279.647286] schedule_timeout+0x1dd/0x300 [377279.650661] wait_for_completion+0x126/0x190 [377279.654154] ? wake_up_q+0x80/0x80 [377279.657277] ? work_busy+0x80/0x80 [377279.660375] __flush_work+0x177/0x1b0 [377279.663604] ? worker_attach_to_pool+0x90/0x90 [377279.667121] __cancel_work_timer+0x12b/0x1b0 [377279.670571] ? rcu_sync_enter+0x8b/0xd0 [377279.673864] xfs_stop_block_reaping+0x15/0x30 [xfs] [377279.677585] xfs_fs_freeze+0x15/0x40 [xfs] [377279.680950] freeze_super+0xc8/0x190 [377279.684086] do_vfs_ioctl+0x510/0x630...
... and the eofblocks scanner:
[377279.422496] Workqueue: xfs-eofblocks/nvme13n1 xfs_eofblocks_worker [xfs] [377279.426971] Call Trace: [377279.429662] ? __schedule+0x292/0x6f0 [377279.432839] schedule+0x2f/0xa0 [377279.435794] rwsem_down_read_slowpath+0x196/0x530 [377279.439435] ? kmem_cache_alloc+0x152/0x1f0 [377279.442834] ? __percpu_down_read+0x49/0x60 [377279.446242] __percpu_down_read+0x49/0x60 [377279.449586] __sb_start_write+0x5b/0x60 [377279.452869] xfs_trans_alloc+0x152/0x160 [xfs] [377279.456372] xfs_free_eofblocks+0x12d/0x1f0 [xfs] [377279.460014] xfs_inode_free_eofblocks+0x128/0x1a0 [xfs] [377279.463903] ? xfs_inode_ag_walk_grab+0x5f/0x90 [xfs] [377279.467680] xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs] [377279.471567] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.475620] ? kvm_sched_clock_read+0xd/0x20 [377279.479059] ? sched_clock+0x5/0x10 [377279.482184] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.486234] ? radix_tree_gang_lookup_tag+0xa8/0x100 [377279.489974] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.494041] xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs] [377279.497859] xfs_eofblocks_worker+0x29/0x40 [xfs] [377279.501484] process_one_work+0x195/0x380...
The immediate issue is likely that the eofblocks transaction is not NOWRITECOUNT (same for the cowblocks scanner, btw), but the problem with doing that is these helpers are called from other contexts outside of the background scanners.
Quoted 7 lines
Perhaps what we need to do here is let these background scanners acquire a superblock write reference, similar to what Darrick recently added to scrub..? We'd have to do that from the scanner workqueue task, so it would probably need to be a trylock so we don't end up in a similar situation as above. I.e., we'd either get the reference and cause freeze to wait until it's dropped or bail out if freeze has already stopped the transaction subsystem. Thoughts?
Hmm, I had a whole gigantic series to refactor all the speculative preallocation gc work into a single thread + radix tree tag; I'll see if that series actually fixed this problem too.
Quoted 8 lines
Quoted 3 lines
But yes, all background threads that run transactions need to have freezer protection.
So something like the following in the meantime, assuming we want a backportable fix..? I think this means we could return
-EAGAINfrom the eofblocks ioctl, but afaict if something functionally conflicts with an active scan across freeze then perhaps that's preferred.
Apparently I don't have a patch that fixes the speculative gc code. The deferred inactivation worker does it, so perhaps I got mixed up. :/
I think a better fix would be to annotate xfs_icache_free_eofblocks and xfs_icache_free_cowblocks to note that the caller must obtain freeze protection before calling those functions. Then we can play whackamole with the existing callers:
1. xfs_eofblocks_worker and xfs_cowblocks_worker can try to sb_start_write and just go back to sleep if the fs is frozen. The flush_workqueue will then cancel the delayed work and the freeze can proceed.
2. The buffered write ENOSPC scour-and-retry loops already have freeze protection because they're file writes, so they don't have to change.
3. XFS_IOC_FREE_EOFBLOCKS can sb_start_write, which means that callers will sleep on the frozen fs.
--D
Quoted 50 lines
Brian
fs/xfs/xfs_icache.c +14
-3@@ -1515,13 +1515,24 @@ __xfs_icache_free_eofblocks( void *args), int tag) { - int flags = SYNC_TRYLOCK; + int flags = SYNC_TRYLOCK; +int error;if (eofb && (eofb->eof_flags &XFS_EOF_FLAGS_SYNC )) flags =SYNC_WAIT ; -return xfs_inode_ag_iterator_tag (mp, execute, flags, - eofb, tag); +/* +* freeze waits on background scanner jobs to complete so we cannot +* block on write protection here. Bail if the transaction subsystem is +* already freezing, returning -EAGAIN to notify other callers. +*/ +if (!sb_start_write_trylock (mp->m_super)) +return -EAGAIN ; + + error =xfs_inode_ag_iterator_tag (mp, execute, flags, eofb, tag); +sb_end_write (mp->m_super); + +return error; }int Quoted 15 lines
--D
Quoted 11 lines
Brian
Quoted 8 lines
Thanks,
PaulSignature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
Re: [Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)
Quoted 84 lines · Darrick J. Wong
Quoted 83 lines · Brian Foster
Quoted 81 lines · Darrick J. Wong
Quoted 65 lines · Brian Foster
Quoted 10 lines · On Tue, Apr 07, 2020 at 06:41:31AM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,
Just had another case of this crop up and I was able to get the blocked tasks output before automation killed the server. Because the log was too large to attach, I've pasted the output into a github gist here:
https://gist.githubusercontent.com/PaulFurtado/c9bade038b8a5c7ddb53a6e10def058f/raw/ee43926c96c0d6a9ec81a648754c1af599ef0bdd/sysrq_w.logHm, so it looks like this is stuck between freeze:
[377279.630957] fsfreeze D 0 46819 46337 0x00004084 [377279.634910] Call Trace: [377279.637594] ? __schedule+0x292/0x6f0 [377279.640833] ? xfs_xattr_get+0x51/0x80 [xfs] [377279.644287] schedule+0x2f/0xa0 [377279.647286] schedule_timeout+0x1dd/0x300 [377279.650661] wait_for_completion+0x126/0x190 [377279.654154] ? wake_up_q+0x80/0x80 [377279.657277] ? work_busy+0x80/0x80 [377279.660375] __flush_work+0x177/0x1b0 [377279.663604] ? worker_attach_to_pool+0x90/0x90 [377279.667121] __cancel_work_timer+0x12b/0x1b0 [377279.670571] ? rcu_sync_enter+0x8b/0xd0 [377279.673864] xfs_stop_block_reaping+0x15/0x30 [xfs] [377279.677585] xfs_fs_freeze+0x15/0x40 [xfs] [377279.680950] freeze_super+0xc8/0x190 [377279.684086] do_vfs_ioctl+0x510/0x630...
... and the eofblocks scanner:
[377279.422496] Workqueue: xfs-eofblocks/nvme13n1 xfs_eofblocks_worker [xfs] [377279.426971] Call Trace: [377279.429662] ? __schedule+0x292/0x6f0 [377279.432839] schedule+0x2f/0xa0 [377279.435794] rwsem_down_read_slowpath+0x196/0x530 [377279.439435] ? kmem_cache_alloc+0x152/0x1f0 [377279.442834] ? __percpu_down_read+0x49/0x60 [377279.446242] __percpu_down_read+0x49/0x60 [377279.449586] __sb_start_write+0x5b/0x60 [377279.452869] xfs_trans_alloc+0x152/0x160 [xfs] [377279.456372] xfs_free_eofblocks+0x12d/0x1f0 [xfs] [377279.460014] xfs_inode_free_eofblocks+0x128/0x1a0 [xfs] [377279.463903] ? xfs_inode_ag_walk_grab+0x5f/0x90 [xfs] [377279.467680] xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs] [377279.471567] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.475620] ? kvm_sched_clock_read+0xd/0x20 [377279.479059] ? sched_clock+0x5/0x10 [377279.482184] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.486234] ? radix_tree_gang_lookup_tag+0xa8/0x100 [377279.489974] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.494041] xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs] [377279.497859] xfs_eofblocks_worker+0x29/0x40 [xfs] [377279.501484] process_one_work+0x195/0x380...
The immediate issue is likely that the eofblocks transaction is not NOWRITECOUNT (same for the cowblocks scanner, btw), but the problem with doing that is these helpers are called from other contexts outside of the background scanners.
Quoted 7 lines
Perhaps what we need to do here is let these background scanners acquire a superblock write reference, similar to what Darrick recently added to scrub..? We'd have to do that from the scanner workqueue task, so it would probably need to be a trylock so we don't end up in a similar situation as above. I.e., we'd either get the reference and cause freeze to wait until it's dropped or bail out if freeze has already stopped the transaction subsystem. Thoughts?
Hmm, I had a whole gigantic series to refactor all the speculative preallocation gc work into a single thread + radix tree tag; I'll see if that series actually fixed this problem too.
But yes, all background threads that run transactions need to have freezer protection.
Quoted 8 lines
Quoted 4 lines
So something like the following in the meantime, assuming we want a backportable fix..? I think this means we could return
-EAGAINfrom the eofblocks ioctl, but afaict if something functionally conflicts with an active scan across freeze then perhaps that's preferred.Apparently I don't have a patch that fixes the speculative gc code. The deferred inactivation worker does it, so perhaps I got mixed up. :/
Ok.
Quoted 10 lines
I think a better fix would be to annotate
xfs_icache_free_eofblocksandxfs_icache_free_cowblocksto note that the caller must obtain freeze protection before calling those functions. Then we can play whackamole with the existing callers:1.
xfs_eofblocks_workerandxfs_cowblocks_workercan try tosb_start_writeand just go back to sleep if the fs is frozen. Theflush_workqueuewill then cancel the delayed work and the freeze can proceed.
Quoted 6 lines
2. The buffered write ENOSPC scour-and-retry loops already have freeze protection because they're file writes, so they don't have to change.
3.
XFS_IOC_FREE_EOFBLOCKScansb_start_write, which means that callers will sleep on the frozen fs.
Sure, that works for me. It can be condensed later if it ends up in a single thread.
Brian
Quoted 53 lines
--D
Quoted 50 lines
Brian
fs/xfs/xfs_icache.c +14
-3@@ -1515,13 +1515,24 @@ __xfs_icache_free_eofblocks( void *args), int tag) { - int flags = SYNC_TRYLOCK; + int flags = SYNC_TRYLOCK; +int error;if (eofb && (eofb->eof_flags &XFS_EOF_FLAGS_SYNC )) flags =SYNC_WAIT ; -return xfs_inode_ag_iterator_tag (mp, execute, flags, - eofb, tag); +/* +* freeze waits on background scanner jobs to complete so we cannot +* block on write protection here. Bail if the transaction subsystem is +* already freezing, returning -EAGAIN to notify other callers. +*/ +if (!sb_start_write_trylock (mp->m_super)) +return -EAGAIN ; + + error =xfs_inode_ag_iterator_tag (mp, execute, flags, eofb, tag); +sb_end_write (mp->m_super); + +return error; }int Quoted 15 lines
--D
Quoted 11 lines
Brian
Quoted 8 lines
Thanks,
PaulSignature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #7 from [email protected] ---
Quoted 93 lines · Darrick J. Wong
Quoted 3 lines · Brian Foster
Quoted 2 lines · Darrick J. Wong
Quoted 1 line · Brian Foster
On Tue, Apr 07, 2020 at 06:41:31AM +0000,
Quoted 6 lines · [email protected] wrote:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #2 from Paul Furtado ([email protected]) --- Hi Dave,
Just had another case of this crop up and I was able to get the
blocked tasks
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
output before automation killed the server. Because the log was too
large to
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
attach, I've pasted the output into a github gist here:
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 1 line
Hm, so it looks like this is stuck between freeze:
[377279.630957] fsfreeze D 0 46819 46337 0x00004084 [377279.634910] Call Trace: [377279.637594] ? __schedule+0x292/0x6f0 [377279.640833] ? xfs_xattr_get+0x51/0x80 [xfs] [377279.644287] schedule+0x2f/0xa0 [377279.647286] schedule_timeout+0x1dd/0x300 [377279.650661] wait_for_completion+0x126/0x190 [377279.654154] ? wake_up_q+0x80/0x80 [377279.657277] ? work_busy+0x80/0x80 [377279.660375] __flush_work+0x177/0x1b0 [377279.663604] ? worker_attach_to_pool+0x90/0x90 [377279.667121] __cancel_work_timer+0x12b/0x1b0 [377279.670571] ? rcu_sync_enter+0x8b/0xd0 [377279.673864] xfs_stop_block_reaping+0x15/0x30 [xfs] [377279.677585] xfs_fs_freeze+0x15/0x40 [xfs] [377279.680950] freeze_super+0xc8/0x190 [377279.684086] do_vfs_ioctl+0x510/0x630...
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
... and the eofblocks scanner:
[377279.422496] Workqueue: xfs-eofblocks/nvme13n1 xfs_eofblocks_worker[xfs]
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
[377279.426971] Call Trace: [377279.429662] ? __schedule+0x292/0x6f0 [377279.432839] schedule+0x2f/0xa0 [377279.435794] rwsem_down_read_slowpath+0x196/0x530 [377279.439435] ? kmem_cache_alloc+0x152/0x1f0 [377279.442834] ? __percpu_down_read+0x49/0x60 [377279.446242] __percpu_down_read+0x49/0x60 [377279.449586] __sb_start_write+0x5b/0x60 [377279.452869] xfs_trans_alloc+0x152/0x160 [xfs] [377279.456372] xfs_free_eofblocks+0x12d/0x1f0 [xfs] [377279.460014] xfs_inode_free_eofblocks+0x128/0x1a0 [xfs] [377279.463903] ? xfs_inode_ag_walk_grab+0x5f/0x90 [xfs] [377279.467680] xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs] [377279.471567] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.475620] ? kvm_sched_clock_read+0xd/0x20 [377279.479059] ? sched_clock+0x5/0x10 [377279.482184] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.486234] ? radix_tree_gang_lookup_tag+0xa8/0x100 [377279.489974] ? __xfs_inode_clear_blocks_tag+0x120/0x120 [xfs] [377279.494041] xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs] [377279.497859] xfs_eofblocks_worker+0x29/0x40 [xfs] [377279.501484] process_one_work+0x195/0x380...
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The immediate issue is likely that the eofblocks transaction is not NOWRITECOUNT (same for the cowblocks scanner, btw), but the problem
with
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
doing that is these helpers are called from other contexts outside of the background scanners.
Perhaps what we need to do here is let these background scanners
acquire
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
a superblock write reference, similar to what Darrick recently added to scrub..? We'd have to do that from the scanner workqueue task, so it would probably need to be a trylock so we don't end up in a similar situation as above. I.e., we'd either get the reference and cause
freeze
Quoted 1 line
Quoted 1 line
Quoted 1 line
to wait until it's dropped or bail out if freeze has already stopped
the
Quoted 10 lines
Quoted 9 lines
Quoted 1 line
transaction subsystem. Thoughts?
Hmm, I had a whole gigantic series to refactor all the speculative preallocation gc work into a single thread + radix tree tag; I'll see if that series actually fixed this problem too.
But yes, all background threads that run transactions need to have freezer protection.
Quoted 8 lines
Quoted 4 lines
So something like the following in the meantime, assuming we want a backportable fix..? I think this means we could return
-EAGAINfrom the eofblocks ioctl, but afaict if something functionally conflicts with an active scan across freeze then perhaps that's preferred.Apparently I don't have a patch that fixes the speculative gc code. The deferred inactivation worker does it, so perhaps I got mixed up. :/
Ok.
Quoted 10 lines
I think a better fix would be to annotate
xfs_icache_free_eofblocksandxfs_icache_free_cowblocksto note that the caller must obtain freeze protection before calling those functions. Then we can play whackamole with the existing callers:1.
xfs_eofblocks_workerandxfs_cowblocks_workercan try tosb_start_writeand just go back to sleep if the fs is frozen. Theflush_workqueuewill then cancel the delayed work and the freeze can proceed.
Quoted 6 lines
2. The buffered write ENOSPC scour-and-retry loops already have freeze protection because they're file writes, so they don't have to change.
3.
XFS_IOC_FREE_EOFBLOCKScansb_start_write, which means that callers will sleep on the frozen fs.
Sure, that works for me. It can be condensed later if it ends up in a single thread.
Brian
Quoted 53 lines
--D
Quoted 50 lines
Brian
fs/xfs/xfs_icache.c +14
-3@@ -1515,13 +1515,24 @@ __xfs_icache_free_eofblocks( void *args), int tag) { - int flags = SYNC_TRYLOCK; + int flags = SYNC_TRYLOCK; +int error;if (eofb && (eofb->eof_flags &XFS_EOF_FLAGS_SYNC )) flags =SYNC_WAIT ; -return xfs_inode_ag_iterator_tag (mp, execute, flags, - eofb, tag); +/* +* freeze waits on background scanner jobs to complete so we cannot +* block on write protection here. Bail if the transaction subsystem is +* already freezing, returning -EAGAIN to notify other callers. +*/ +if (!sb_start_write_trylock (mp->m_super)) +return -EAGAIN ; + + error =xfs_inode_ag_iterator_tag (mp, execute, flags, eofb, tag); +sb_end_write (mp->m_super); + +return error; }int Quoted 15 lines
--D
Quoted 11 lines
Brian
Quoted 8 lines
Thanks,
PaulSignature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #8 from Paul Furtado ([email protected]) ---
The patches that came from this issue have given us many weeks of stability now and we were ready to declare this as totally fixed, however, we hit another instance of this issue this week which I'm assuming is probably on a slightly different and much rarer code path.
Here's a link to the blocked tasks log (beware that it's 2MB due to endless processes getting hung inside the container once the filesystem was frozen):
https://gist.githubusercontent.com/PaulFurtado/48253a6978763671f70dc94d933df851/raw/6bad12023ac56e9b6cb3dde771fcb5b15f0bd679/patched_kernel_fsfreeze_sys_w.log
Thanks,
Paul
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
Re: [Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)
Quoted 8 lines · On Thu, May 28, 2020 at 06:00:38AM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #8 from Paul Furtado ([email protected]) ---
The patches that came from this issue have given us many weeks of stability now and we were ready to declare this as totally fixed, however, we hit another instance of this issue this week which I'm assuming is probably on a slightly different and much rarer code path.
Quoted 4 lines
Here's a link to the blocked tasks log (beware that it's 2MB due to endless processes getting hung inside the container once the filesystem was frozen):
https://gist.githubusercontent.com/PaulFurtado/48253a6978763671f70dc94d933df851/raw/6bad12023ac56e9b6cb3dde771fcb5b15f0bd679/patched_kernel_fsfreeze_sys_w.log
This shows the eofblocks scan in the following (massaged) trace:
[1259466.349224] Workqueue: xfs-eofblocks/nvme4n1 xfs_eofblocks_worker [xfs] [1259466.353550] Call Trace: [1259466.359370] schedule+0x2f/0xa0 [1259466.362297] rwsem_down_read_slowpath+0x196/0x530 [1259466.372467] __percpu_down_read+0x49/0x60 [1259466.375778] __sb_start_write+0x5b/0x60 [1259466.379139] xfs_trans_alloc+0x152/0x160 [xfs] [1259466.382715] xfs_free_eofblocks+0x12d/0x1f0 [xfs] [1259466.386407] xfs_inode_free_eofblocks+0x128/0x1a0 [xfs] [1259466.394058] xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs] [1259466.536551] xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs] [1259466.540235] xfs_eofblocks_worker+0x29/0x40 [xfs] [1259466.543748] process_one_work+0x195/0x380 [1259466.546996] worker_thread+0x30/0x390 [1259466.553449] kthread+0x113/0x130 [1259466.559579] ret_from_fork+0x1f/0x40
This should be addressed by upstream commit 4b674b9ac8529 ("xfs: acquire superblock freeze protection on eofblocks scans"), which causes xfs_eofblocks_worker() to bail unless it acquires freeze write protection. What exact kernel is this seen on?
Brian
Quoted 7 lines
Thanks,
PaulSignature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #9 from [email protected] ---
Quoted 9 lines · On Thu, May 28, 2020 at 06:00:38AM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #8 from Paul Furtado ([email protected]) --- The patches that came from this issue have given us many weeks of stability now
and we were ready to declare this as totally fixed, however, we hit another instance of this issue this week which I'm assuming is probably on a slightly different and much rarer code path.
Quoted 5 lines
Here's a link to the blocked tasks log (beware that it's 2MB due to endless processes getting hung inside the container once the filesystem was frozen):
This shows the eofblocks scan in the following (massaged) trace:
[1259466.349224] Workqueue: xfs-eofblocks/nvme4n1 xfs_eofblocks_worker [xfs] [1259466.353550] Call Trace: [1259466.359370] schedule+0x2f/0xa0 [1259466.362297] rwsem_down_read_slowpath+0x196/0x530 [1259466.372467] __percpu_down_read+0x49/0x60 [1259466.375778] __sb_start_write+0x5b/0x60 [1259466.379139] xfs_trans_alloc+0x152/0x160 [xfs] [1259466.382715] xfs_free_eofblocks+0x12d/0x1f0 [xfs] [1259466.386407] xfs_inode_free_eofblocks+0x128/0x1a0 [xfs] [1259466.394058] xfs_inode_ag_walk.isra.17+0x1a7/0x410 [xfs] [1259466.536551] xfs_inode_ag_iterator_tag+0x73/0xb0 [xfs] [1259466.540235] xfs_eofblocks_worker+0x29/0x40 [xfs] [1259466.543748] process_one_work+0x195/0x380 [1259466.546996] worker_thread+0x30/0x390 [1259466.553449] kthread+0x113/0x130 [1259466.559579] ret_from_fork+0x1f/0x40
This should be addressed by upstream commit 4b674b9ac8529 ("xfs: acquire superblock freeze protection on eofblocks scans"), which causes xfs_eofblocks_worker() to bail unless it acquires freeze write protection. What exact kernel is this seen on?
Brian
Quoted 7 lines
Thanks,
PaulSignature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=207053
--- Comment #10 from Paul Furtado ([email protected]) --- This is my mistake actually: it turns out this host had slipped through the cracks in our configuration management and didn't actually get updated to the kernel build with that commit in it, I had thought we'd covered every host. Looks like we're actually in the clear.
Thanks,
Paul
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=207053
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #11 from [email protected] ---
I'm trying to figure out which kernel version this bug is fixed in. I see references to a patch above, but am having trouble tracking down when it was added into the baseline.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 207053] fsfreeze deadlock on XFS (the FIFREEZE ioctl and subsequent FITHAW hang indefinitely)
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=207053
Eric Sandeen ([email protected]) changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #12 from Eric Sandeen ([email protected]) --- commit 4b674b9ac8529 should be in upstream kernel v5.7
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.