btrfs: transaction: Avoid deadlock due to bad initialization timing of fs_info::journal_info
PATCH 1/1 btrfs: transaction: Avoid deadlock due to bad initialization timing of fs_info::journal_info
[BUG]
One run of btrfs/063 triggered the following lockdep:
============================================
WARNING: possible recursive locking detected
5.6.0-rc7-custom+ #48 Not tainted
--------------------------------------------
kworker/u24:0/7 is trying to acquire lock:
ffff88817d3a46e0 (sb_internal#2){.+.+}, at: start_transaction+0x66c/0x890 [btrfs]
but task is already holding lock:
ffff88817d3a46e0 (sb_internal#2){.+.+}, at: start_transaction+0x66c/0x890 [btrfs]
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(sb_internal#2);
lock(sb_internal#2);*** DEADLOCK ***
May be due to missing lock nesting notation
4 locks held by kworker/u24:0/7:
#0: ffff88817b495948 ((wq_completion)btrfs-endio-write){+.+.}, at: process_one_work+0x557/0xb80 #1: ffff888189ea7db8 ((work_completion)(&work->normal_work)){+.+.}, at: process_one_work+0x557/0xb80 #2: ffff88817d3a46e0 (sb_internal#2){.+.+}, at: start_transaction+0x66c/0x890 [btrfs] #3: ffff888174ca4da8 (&fs_info->reloc_mutex){+.+.}, at: btrfs_record_root_in_trans+0x83/0xd0 [btrfs]
stack backtrace: CPU: 0 PID: 7 Comm: kworker/u24:0 Not tainted 5.6.0-rc7-custom+ #48 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015 Workqueue: btrfs-endio-write btrfs_work_helper [btrfs] Call Trace: dump_stack+0xc2/0x11a __lock_acquire.cold+0xce/0x214 lock_acquire+0xe6/0x210 __sb_start_write+0x14e/0x290 start_transaction+0x66c/0x890 [btrfs] btrfs_join_transaction+0x1d/0x20 [btrfs] find_free_extent+0x1504/0x1a50 [btrfs] btrfs_reserve_extent+0xd5/0x1f0 [btrfs] btrfs_alloc_tree_block+0x1ac/0x570 [btrfs] btrfs_copy_root+0x213/0x580 [btrfs] create_reloc_root+0x3bd/0x470 [btrfs] btrfs_init_reloc_root+0x2d2/0x310 [btrfs] record_root_in_trans+0x191/0x1d0 [btrfs] btrfs_record_root_in_trans+0x90/0xd0 [btrfs] start_transaction+0x16e/0x890 [btrfs] btrfs_join_transaction+0x1d/0x20 [btrfs] btrfs_finish_ordered_io+0x55d/0xcd0 [btrfs] finish_ordered_fn+0x15/0x20 [btrfs] btrfs_work_helper+0x116/0x9a0 [btrfs] process_one_work+0x632/0xb80 worker_thread+0x80/0x690 kthread+0x1a3/0x1f0 ret_from_fork+0x27/0x50
It's pretty hard to reproduce, only one hit so far.
[CAUSE]
This is because we're calling btrfs_join_transaction() without re-using the current running one:
btrfs_finish_ordered_io()
|- btrfs_join_transaction() <<< Call #1
|- btrfs_record_root_in_trans()
|- btrfs_reserve_extent() |- btrfs_join_transaction() <<< Call #2
Normally such btrfs_join_transaction() call should re-use the existing one, without trying to re-start a transaction.
But the problem is, in btrfs_join_transaction() call #1, we call btrfs_record_root_in_trans() before initializing current::journal_info.
And in btrfs_join_transaction() call #2, we're relying on current::journal_info to avoid such deadlock.
[FIX]
Call btrfs_record_root_in_trans() after we have initialized current::journal_info.
--- fs/btrfs/transaction.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
fs/btrfs/transaction.c +11 -2
@@ -662,10 +662,19 @@ start_transaction(struct btrfs_root *root, unsigned int num_items, } got_it: -btrfs_record_root_in_trans (h, root); -if (!current->journal_info) current->journal_info = h; + +/* +* btrfs_record_root_in_trans() need to alloc new extents, and may +* call btrfs_join_transaction() while we're also starting a +* transaction. +* +* Thus it need to be called after current->journal_info initialized, +* or we can deadlock. +*/ +btrfs_record_root_in_trans (h, root); +return h; join_fail:
Signature · 1 line
2.26.2
Quoted 59 lines · Qu Wenruo
[BUG]
One run of btrfs/063 triggered the following lockdep:
============================================
WARNING: possible recursive locking detected
5.6.0-rc7-custom+ #48 Not tainted
--------------------------------------------
kworker/u24:0/7 is trying to acquire lock:
ffff88817d3a46e0(sb_internal#2){.+.+}, at:start_transaction+0x66c/0x890[btrfs]but task is already holding lock:
ffff88817d3a46e0(sb_internal#2){.+.+}, at:start_transaction+0x66c/0x890[btrfs]other info that might help us debug this:
Possible unsafe locking scenario:CPU0 ---- lock(sb_internal#2); lock(sb_internal#2);*** DEADLOCK ***
May be due to missing lock nesting notation
4 locks held by kworker/u24:0/7:
#0: ffff88817b495948 ((wq_completion)btrfs-endio-write){+.+.}, at: process_one_work+0x557/0xb80 #1: ffff888189ea7db8 ((work_completion)(&work->normal_work)){+.+.}, at: process_one_work+0x557/0xb80 #2: ffff88817d3a46e0 (sb_internal#2){.+.+}, at: start_transaction+0x66c/0x890 [btrfs] #3: ffff888174ca4da8 (&fs_info->reloc_mutex){+.+.}, at: btrfs_record_root_in_trans+0x83/0xd0 [btrfs]stack backtrace: CPU: 0 PID: 7 Comm: kworker/u24:0 Not tainted 5.6.0-rc7-custom+ #48 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015 Workqueue: btrfs-endio-write btrfs_work_helper [btrfs] Call Trace: dump_stack+0xc2/0x11a __lock_acquire.cold+0xce/0x214 lock_acquire+0xe6/0x210 __sb_start_write+0x14e/0x290 start_transaction+0x66c/0x890 [btrfs] btrfs_join_transaction+0x1d/0x20 [btrfs] find_free_extent+0x1504/0x1a50 [btrfs] btrfs_reserve_extent+0xd5/0x1f0 [btrfs] btrfs_alloc_tree_block+0x1ac/0x570 [btrfs] btrfs_copy_root+0x213/0x580 [btrfs] create_reloc_root+0x3bd/0x470 [btrfs] btrfs_init_reloc_root+0x2d2/0x310 [btrfs] record_root_in_trans+0x191/0x1d0 [btrfs] btrfs_record_root_in_trans+0x90/0xd0 [btrfs] start_transaction+0x16e/0x890 [btrfs] btrfs_join_transaction+0x1d/0x20 [btrfs] btrfs_finish_ordered_io+0x55d/0xcd0 [btrfs] finish_ordered_fn+0x15/0x20 [btrfs] btrfs_work_helper+0x116/0x9a0 [btrfs] process_one_work+0x632/0xb80 worker_thread+0x80/0x690 kthread+0x1a3/0x1f0 ret_from_fork+0x27/0x50
Quoted 1 line
It's pretty hard to reproduce, only one hit so far.
I haven't found the report in any of my logs, so yeah.
Quoted 16 lines
[CAUSE]
This is because we're callingbtrfs_join_transaction()without re-using the current running one:
btrfs_finish_ordered_io()
|-btrfs_join_transaction()<<< Call #1
|-btrfs_record_root_in_trans()|- btrfs_reserve_extent() |- btrfs_join_transaction() <<< Call #2Normally such
btrfs_join_transaction()call should re-use the existing one, without trying to re-start a transaction.But the problem is, in
btrfs_join_transaction()call #1, we callbtrfs_record_root_in_trans()before initializingcurrent::journal_info.
Quoted 8 lines
And in
btrfs_join_transaction()call #2, we're relying oncurrent::journal_infoto avoid such deadlock.[FIX]
Callbtrfs_record_root_in_trans()after we have initializedcurrent::journal_info.Signed-off-by Qu Wenruo <[email protected]>
Added to misc-next, thanks.
Quoted 11 lines · Qu Wenruo
index
8cede6eb9843..132bf2f1aa0d100644fs/btrfs/transaction.c +0
-2@@ -662,10 +662,19 @@ start_transaction(struct btrfs_root *root, unsigned int num_items, } got_it: -btrfs_record_root_in_trans (h, root); -if (!current->journal_info) current->journal_info = h;
fs/btrfs/transaction.c start_transaction(struct btrfs_root *root, unsigned int num_items, · line 676
Quoted 10 lines
fs/btrfs/transaction.c +10
-0+ +/* +* btrfs_record_root_in_trans() need to alloc new extents, and may +* call btrfs_join_transaction() while we're also starting a +* transaction. +* +* Thus it need to be called after current->journal_info initialized, +* or we can deadlock. +*/ +btrfs_record_root_in_trans (h, root);
This applies cleanly on master, so that's fine as it'll go as a fix in this dev cycle, but there's a conflict with misc-next patch "btrfs:
force chunk allocation if our global rsv is larger than metadata".
There's a chunk allocation added:
btrfs_record_root_in_trans(...) if (!current->journal_info) ... if (do_chunk_alloc ...) btrfs_chunk_alloc(...)
so the call btrfs_record_root_in_trans() should be moved after the chunk allocation, to potentially use the newly added chunk. The merged order is:
if (!current->journal_info) ... if (do_chunk_alloc ...) btrfs_chunk_alloc(...) btrfs_record_root_in_trans(...)
Please check if this is correct. Once this fix bubbles through master, the conflicting patch in misc-next will have to be updated, but the end result should remain. Thanks.
Quoted 45 lines · David Sterba
Quoted 11 lines · Qu Wenruo
index
8cede6eb9843..132bf2f1aa0d100644fs/btrfs/transaction.c +0
-2@@ -662,10 +662,19 @@ start_transaction(struct btrfs_root *root, unsigned int num_items, } got_it: -btrfs_record_root_in_trans (h, root); -if (!current->journal_info) current->journal_info = h;fs/btrfs/transaction.c start_transaction(struct btrfs_root *root, unsigned int num_items, · line 676
Quoted 10 lines
fs/btrfs/transaction.c +10
-0+ +/* +* btrfs_record_root_in_trans() need to alloc new extents, and may +* call btrfs_join_transaction() while we're also starting a +* transaction. +* +* Thus it need to be called after current->journal_info initialized, +* or we can deadlock. +*/ +btrfs_record_root_in_trans (h, root);This applies cleanly on master, so that's fine as it'll go as a fix in this dev cycle, but there's a conflict with misc-next patch "btrfs:
force chunk allocation if our global rsv is larger than metadata".There's a chunk allocation added:
btrfs_record_root_in_trans(...) if (!current->journal_info) ... if (do_chunk_alloc ...) btrfs_chunk_alloc(...)so the call
btrfs_record_root_in_trans()should be moved after the chunk allocation, to potentially use the newly added chunk. The merged order is:if (!current->journal_info) ... if (do_chunk_alloc ...) btrfs_chunk_alloc(...) btrfs_record_root_in_trans(...)
Quoted 4 lines
Please check if this is correct. Once this fix bubbles through master, the conflicting patch in misc-next will have to be updated, but the end result should remain. Thanks.
Checked the final code. Looks good, especially after that chunk allocation, we would have less chance to hit BUG_ON() due to ENOSPC in btrfs_record_root_in_trans().
Thanks for the merge,
Qu