Skip to contentSkip to conversation

btrfs: transaction: Avoid deadlock due to bad initialization timing of fs_info::journal_info

PATCH In review Qu Wenruo 3 participants 4 messages Apr 27, 2020 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

PATCH 1/1 btrfs: transaction: Avoid deadlock due to bad initialization timing of fs_info::journal_info

[BUG]
One run of btrfs/063 triggered the following lockdep:
============================================
WARNING: possible recursive locking detected
5.6.0-rc7-custom+ #48 Not tainted
--------------------------------------------
kworker/u24:0/7 is trying to acquire lock:
ffff88817d3a46e0 (sb_internal#2){.+.+}, at: start_transaction+0x66c/0x890 [btrfs]

but task is already holding lock:
ffff88817d3a46e0 (sb_internal#2){.+.+}, at: start_transaction+0x66c/0x890 [btrfs]

other info that might help us debug this:

   Possible unsafe locking scenario:
         CPU0
         ----
    lock(sb_internal#2);
    lock(sb_internal#2);

*** DEADLOCK ***

May be due to missing lock nesting notation

4 locks held by kworker/u24:0/7:
#0: ffff88817b495948 ((wq_completion)btrfs-endio-write){+.+.}, at: process_one_work+0x557/0xb80 #1: ffff888189ea7db8 ((work_completion)(&work->normal_work)){+.+.}, at: process_one_work+0x557/0xb80 #2: ffff88817d3a46e0 (sb_internal#2){.+.+}, at: start_transaction+0x66c/0x890 [btrfs] #3: ffff888174ca4da8 (&fs_info->reloc_mutex){+.+.}, at: btrfs_record_root_in_trans+0x83/0xd0 [btrfs]

  stack backtrace:
  CPU: 0 PID: 7 Comm: kworker/u24:0 Not tainted 5.6.0-rc7-custom+ #48
  Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
  Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
  Call Trace:
   dump_stack+0xc2/0x11a
   __lock_acquire.cold+0xce/0x214
   lock_acquire+0xe6/0x210
   __sb_start_write+0x14e/0x290
   start_transaction+0x66c/0x890 [btrfs]
   btrfs_join_transaction+0x1d/0x20 [btrfs]
   find_free_extent+0x1504/0x1a50 [btrfs]
   btrfs_reserve_extent+0xd5/0x1f0 [btrfs]
   btrfs_alloc_tree_block+0x1ac/0x570 [btrfs]
   btrfs_copy_root+0x213/0x580 [btrfs]
   create_reloc_root+0x3bd/0x470 [btrfs]
   btrfs_init_reloc_root+0x2d2/0x310 [btrfs]
   record_root_in_trans+0x191/0x1d0 [btrfs]
   btrfs_record_root_in_trans+0x90/0xd0 [btrfs]
   start_transaction+0x16e/0x890 [btrfs]
   btrfs_join_transaction+0x1d/0x20 [btrfs]
   btrfs_finish_ordered_io+0x55d/0xcd0 [btrfs]
   finish_ordered_fn+0x15/0x20 [btrfs]
   btrfs_work_helper+0x116/0x9a0 [btrfs]
   process_one_work+0x632/0xb80
   worker_thread+0x80/0x690
   kthread+0x1a3/0x1f0
   ret_from_fork+0x27/0x50

It's pretty hard to reproduce, only one hit so far.

[CAUSE]
This is because we're calling btrfs_join_transaction() without re-using the current running one:

btrfs_finish_ordered_io()
|- btrfs_join_transaction() <<< Call #1
|- btrfs_record_root_in_trans()

      |- btrfs_reserve_extent()
	 |- btrfs_join_transaction()	<<< Call #2

Normally such btrfs_join_transaction() call should re-use the existing one, without trying to re-start a transaction.

But the problem is, in btrfs_join_transaction() call #1, we call btrfs_record_root_in_trans() before initializing current::journal_info.

And in btrfs_join_transaction() call #2, we're relying on current::journal_info to avoid such deadlock.

[FIX]
Call btrfs_record_root_in_trans() after we have initialized current::journal_info.

---
 fs/btrfs/transaction.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
fs/btrfs/transaction.c +11 -2
index 8cede6eb9843..132bf2f1aa0d 100644
@@ -662,10 +662,19 @@ start_transaction(struct btrfs_root *root, unsigned int num_items,
 	}
 
 got_it:
-	btrfs_record_root_in_trans(h, root);
-
 	if (!current->journal_info)
 		current->journal_info = h;
+
+	/*
+	 * btrfs_record_root_in_trans() need to alloc new extents, and may
+	 * call btrfs_join_transaction() while we're also starting a
+	 * transaction.
+	 *
+	 * Thus it need to be called after current->journal_info initialized,
+	 * or we can deadlock.
+	 */
+	btrfs_record_root_in_trans(h, root);
+
 	return h;
 
 join_fail:
Signature · 1 line
2.26.2
#1 lore raw
Quoted 59 lines · Qu Wenruo

[BUG]
One run of btrfs/063 triggered the following lockdep:
============================================
WARNING: possible recursive locking detected
5.6.0-rc7-custom+ #48 Not tainted
--------------------------------------------
kworker/u24:0/7 is trying to acquire lock:
ffff88817d3a46e0 (sb_internal#2){.+.+}, at: start_transaction+0x66c/0x890 [btrfs]

but task is already holding lock:
ffff88817d3a46e0 (sb_internal#2){.+.+}, at: start_transaction+0x66c/0x890 [btrfs]

other info that might help us debug this:

   Possible unsafe locking scenario:
         CPU0
         ----
    lock(sb_internal#2);
    lock(sb_internal#2);

*** DEADLOCK ***

May be due to missing lock nesting notation

4 locks held by kworker/u24:0/7:

   #0: ffff88817b495948 ((wq_completion)btrfs-endio-write){+.+.}, at: process_one_work+0x557/0xb80
   #1: ffff888189ea7db8 ((work_completion)(&work->normal_work)){+.+.}, at: process_one_work+0x557/0xb80
   #2: ffff88817d3a46e0 (sb_internal#2){.+.+}, at: start_transaction+0x66c/0x890 [btrfs]
   #3: ffff888174ca4da8 (&fs_info->reloc_mutex){+.+.}, at: btrfs_record_root_in_trans+0x83/0xd0 [btrfs]
  stack backtrace:
  CPU: 0 PID: 7 Comm: kworker/u24:0 Not tainted 5.6.0-rc7-custom+ #48
  Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
  Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
  Call Trace:
   dump_stack+0xc2/0x11a
   __lock_acquire.cold+0xce/0x214
   lock_acquire+0xe6/0x210
   __sb_start_write+0x14e/0x290
   start_transaction+0x66c/0x890 [btrfs]
   btrfs_join_transaction+0x1d/0x20 [btrfs]
   find_free_extent+0x1504/0x1a50 [btrfs]
   btrfs_reserve_extent+0xd5/0x1f0 [btrfs]
   btrfs_alloc_tree_block+0x1ac/0x570 [btrfs]
   btrfs_copy_root+0x213/0x580 [btrfs]
   create_reloc_root+0x3bd/0x470 [btrfs]
   btrfs_init_reloc_root+0x2d2/0x310 [btrfs]
   record_root_in_trans+0x191/0x1d0 [btrfs]
   btrfs_record_root_in_trans+0x90/0xd0 [btrfs]
   start_transaction+0x16e/0x890 [btrfs]
   btrfs_join_transaction+0x1d/0x20 [btrfs]
   btrfs_finish_ordered_io+0x55d/0xcd0 [btrfs]
   finish_ordered_fn+0x15/0x20 [btrfs]
   btrfs_work_helper+0x116/0x9a0 [btrfs]
   process_one_work+0x632/0xb80
   worker_thread+0x80/0x690
   kthread+0x1a3/0x1f0
   ret_from_fork+0x27/0x50
Quoted 1 line

It's pretty hard to reproduce, only one hit so far.

I haven't found the report in any of my logs, so yeah.

Quoted 16 lines

[CAUSE]
This is because we're calling btrfs_join_transaction() without re-using the current running one:

btrfs_finish_ordered_io()
|- btrfs_join_transaction() <<< Call #1
|- btrfs_record_root_in_trans()

      |- btrfs_reserve_extent()
	 |- btrfs_join_transaction()	<<< Call #2

Normally such btrfs_join_transaction() call should re-use the existing one, without trying to re-start a transaction.

But the problem is, in btrfs_join_transaction() call #1, we call btrfs_record_root_in_trans() before initializing current::journal_info.

Quoted 8 lines

And in btrfs_join_transaction() call #2, we're relying on current::journal_info to avoid such deadlock.

[FIX]
Call btrfs_record_root_in_trans() after we have initialized current::journal_info.

Added to misc-next, thanks.

#2 lore raw
In reply to Qu Wenruo
Quoted 11 lines · Qu Wenruo

index 8cede6eb9843..132bf2f1aa0d 100644

fs/btrfs/transaction.c +0 -2
@@ -662,10 +662,19 @@ start_transaction(struct btrfs_root *root, unsigned int num_items,
 	}
 
 got_it:
-	btrfs_record_root_in_trans(h, root);
-
 	if (!current->journal_info)
 		current->journal_info = h;

fs/btrfs/transaction.c start_transaction(struct btrfs_root *root, unsigned int num_items, · line 676

Quoted 10 lines
fs/btrfs/transaction.c +10 -0
+
+	/*
+	 * btrfs_record_root_in_trans() need to alloc new extents, and may
+	 * call btrfs_join_transaction() while we're also starting a
+	 * transaction.
+	 *
+	 * Thus it need to be called after current->journal_info initialized,
+	 * or we can deadlock.
+	 */
+	btrfs_record_root_in_trans(h, root);

This applies cleanly on master, so that's fine as it'll go as a fix in this dev cycle, but there's a conflict with misc-next patch "btrfs:
force chunk allocation if our global rsv is larger than metadata".

There's a chunk allocation added:

	btrfs_record_root_in_trans(...)
	if (!current->journal_info)
		...
	if (do_chunk_alloc ...)
		btrfs_chunk_alloc(...)

so the call btrfs_record_root_in_trans() should be moved after the chunk allocation, to potentially use the newly added chunk. The merged order is:

	if (!current->journal_info)
		...
	if (do_chunk_alloc ...)
		btrfs_chunk_alloc(...)
	btrfs_record_root_in_trans(...)

Please check if this is correct. Once this fix bubbles through master, the conflicting patch in misc-next will have to be updated, but the end result should remain. Thanks.

#3 lore raw
Quoted 45 lines · David Sterba
Quoted 11 lines · Qu Wenruo

index 8cede6eb9843..132bf2f1aa0d 100644

fs/btrfs/transaction.c +0 -2
@@ -662,10 +662,19 @@ start_transaction(struct btrfs_root *root, unsigned int num_items,
 	}
 
 got_it:
-	btrfs_record_root_in_trans(h, root);
-
 	if (!current->journal_info)
 		current->journal_info = h;

fs/btrfs/transaction.c start_transaction(struct btrfs_root *root, unsigned int num_items, · line 676

Quoted 10 lines
fs/btrfs/transaction.c +10 -0
+
+	/*
+	 * btrfs_record_root_in_trans() need to alloc new extents, and may
+	 * call btrfs_join_transaction() while we're also starting a
+	 * transaction.
+	 *
+	 * Thus it need to be called after current->journal_info initialized,
+	 * or we can deadlock.
+	 */
+	btrfs_record_root_in_trans(h, root);

This applies cleanly on master, so that's fine as it'll go as a fix in this dev cycle, but there's a conflict with misc-next patch "btrfs:
force chunk allocation if our global rsv is larger than metadata".

There's a chunk allocation added:

	btrfs_record_root_in_trans(...)
	if (!current->journal_info)
		...
	if (do_chunk_alloc ...)
		btrfs_chunk_alloc(...)

so the call btrfs_record_root_in_trans() should be moved after the chunk allocation, to potentially use the newly added chunk. The merged order is:

	if (!current->journal_info)
		...
	if (do_chunk_alloc ...)
		btrfs_chunk_alloc(...)
	btrfs_record_root_in_trans(...)
Quoted 4 lines

Please check if this is correct. Once this fix bubbles through master, the conflicting patch in misc-next will have to be updated, but the end result should remain. Thanks.

Checked the final code. Looks good, especially after that chunk allocation, we would have less chance to hit BUG_ON() due to ENOSPC in btrfs_record_root_in_trans().

Thanks for the merge,
Qu