Skip to contentSkip to conversation

btrfs: avoid possible signal interruption for btrfs_drop_snapshot() on relocation tree

PATCH In review Qu Wenruo 3 participants 8 messages Jul 9, 2020 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

PATCH v2 1/2 btrfs: avoid possible signal interruption for btrfs_drop_snapshot() on relocation tree

[BUG]
There is a bug report about bad signal timing could lead to read-only fs during balance:

BTRFS info (device xvdb): balance: start -d -m -s
BTRFS info (device xvdb): relocating block group 73001861120 flags metadata BTRFS info (device xvdb): found 12236 extents, stage: move data extents BTRFS info (device xvdb): relocating block group 71928119296 flags data BTRFS info (device xvdb): found 3 extents, stage: move data extents BTRFS info (device xvdb): found 3 extents, stage: update data pointers BTRFS info (device xvdb): relocating block group 60922265600 flags metadata
BTRFS: error (device xvdb) in btrfs_drop_snapshot:5505: errno=-4 unknown BTRFS info (device xvdb): forced readonly
BTRFS info (device xvdb): balance: ended with status: -4

[CAUSE]
The direct cause is the -EINTR from the following call chain when a fatal signal is pending:

relocate_block_group()
|- clean_dirty_subvols()

    |- btrfs_drop_snapshot()
       |- btrfs_start_transaction()
          |- btrfs_delayed_refs_rsv_refill()
             |- btrfs_reserve_metadata_bytes()
                |- __reserve_metadata_bytes()
                   |- wait_reserve_ticket()
                      |- prepare_to_wait_event();
                      |- ticket->error = -EINTR;

Normally this behavior is fine for most btrfs_start_transaction() callers, as they need to catch the fatal signal and exit asap.

However for balance, especially for the clean_dirty_subvols() case, we're already doing cleanup works, such -EINTR from btrfs_drop_snapshot() could cause a lot of unexpected problems.

From the mentioned forced read-only, to later balance error due to half dropped reloc trees.

[FIX]
Fix this problem by using btrfs_join_transaction() if btrfs_drop_snapshot() is called from relocation context.

As btrfs_join_transaction() won't wait full tickets, it won't get interrupted from signal.

---
 fs/btrfs/extent-tree.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
fs/btrfs/extent-tree.c +4 -1
index c0bc35f932bf..d8ef48a807d1 100644
@@ -5298,7 +5298,10 @@ int btrfs_drop_snapshot(struct btrfs_root *root, int update_ref, int for_reloc)
 		goto out;
 	}
 
-	trans = btrfs_start_transaction(tree_root, 0);
+	if (for_reloc)
+		trans = btrfs_join_transaction(tree_root);
+	else
+		trans = btrfs_start_transaction(tree_root, 0);
 	if (IS_ERR(trans)) {
 		err = PTR_ERR(trans);
 		goto out_free;
Signature · 1 line
2.27.0
author
#1 lore raw

PATCH v2 2/2 btrfs: relocation: review the call sites which can be interruped by signal

Since most metadata reservation calls can return -EINTR when get interruped by fatal signal, we need to review the all the metadata reservation call sites.

In relocation code, the metadata reservation happens in the following sites:
- btrfs_block_rsv_refill() in merge_reloc_root() merge_reloc_root() is a pretty critial section, we don't want get interrupted by signal, so change the flush status to BTRFS_RESERVE_FLUSH_LIMIT, so it won't get interrupted by signal. Since such change can be ENPSPC-prone, also shrink the amount of metadata to reserve a little to avoid deadly ENOSPC there.

- btrfs_block_rsv_refill() in reserve_metadata_space()
It calls with BTRFS_RESERVE_FLUSH_LIMIT, which won't get interrupred by signal.

- btrfs_block_rsv_refill() in prepare_to_relocate()
- btrfs_block_rsv_add() in prepare_to_relocate()
- btrfs_block_rsv_refill() in relocate_block_group()
- btrfs_delalloc_reserve_metadata() in relocate_file_extent_cluster()
- btrfs_start_transaction() in relocate_block_group()
- btrfs_start_transaction() in create_reloc_inode()
Can be interruped by fatal signal and we can handle it easily. For these call sites, just catch the -EINTR value in btrfs_balance() and count them as canceled.

---
 fs/btrfs/relocation.c | 13 +++++++++++--
 fs/btrfs/volumes.c    |  2 +-
 2 files changed, 12 insertions(+), 3 deletions(-)
fs/btrfs/relocation.c +11 -2
index 2b869fb2e62c..23914edd4710 100644
@@ -1686,12 +1686,21 @@ static noinline_for_stack int merge_reloc_root(struct reloc_control *rc,
 		btrfs_unlock_up_safe(path, 0);
 	}
 
-	min_reserved = fs_info->nodesize * (BTRFS_MAX_LEVEL - 1) * 2;
+	/*
+	 * In merge_reloc_root(), we modify the upper level pointer to swap
+	 * the tree blocks between reloc tree and subvolume tree.
+	 * Thus for tree block COW, we COW at most from level 1 to root level
+	 * for each tree.
+	 *
+	 * Thus the needed metadata space is at most root_level * nodesize,
+	 * and * 2 since we have two trees to COW.
+	 */
+	min_reserved = fs_info->nodesize * btrfs_root_level(root_item) * 2;
 	memset(&next_key, 0, sizeof(next_key));
 
 	while (1) {
 		ret = btrfs_block_rsv_refill(root, rc->block_rsv, min_reserved,
-					     BTRFS_RESERVE_FLUSH_ALL);
+					     BTRFS_RESERVE_FLUSH_LIMIT);
 		if (ret) {
 			err = ret;
 			goto out;
fs/btrfs/volumes.c +1 -1
index aabc6c922e04..d60df30bdc47 100644
@@ -4135,7 +4135,7 @@ int btrfs_balance(struct btrfs_fs_info *fs_info,
 	mutex_lock(&fs_info->balance_mutex);
 	if (ret == -ECANCELED && atomic_read(&fs_info->balance_pause_req))
 		btrfs_info(fs_info, "balance: paused");
-	else if (ret == -ECANCELED && atomic_read(&fs_info->balance_cancel_req))
+	else if (ret == -ECANCELED  || ret == -EINTR)
 		btrfs_info(fs_info, "balance: canceled");
 	else
 		btrfs_info(fs_info, "balance: ended with status: %d", ret);
Signature · 1 line
2.27.0
#2 lore raw

Re: [PATCH v2 2/2] btrfs: relocation: review the call sites which can be interruped by signal

Quoted 13 lines · Qu Wenruo

Since most metadata reservation calls can return -EINTR when get interruped by fatal signal, we need to review the all the metadata reservation call sites.

In relocation code, the metadata reservation happens in the following sites:
- btrfs_block_rsv_refill() in merge_reloc_root() merge_reloc_root() is a pretty critial section, we don't want get interrupted by signal, so change the flush status to BTRFS_RESERVE_FLUSH_LIMIT, so it won't get interrupted by signal. Since such change can be ENPSPC-prone, also shrink the amount of metadata to reserve a little to avoid deadly ENOSPC there.

Quoted 3 lines

- btrfs_block_rsv_refill() in reserve_metadata_space()
It calls with BTRFS_RESERVE_FLUSH_LIMIT, which won't get interrupred by signal.

This semantics of BTRFS_RESERVE_FLUSH_LIMIT regarding signals should be documented, right now there's a comment but says something about avoidig deadlocks.

Quoted 49 lines

- btrfs_block_rsv_refill() in prepare_to_relocate()
- btrfs_block_rsv_add() in prepare_to_relocate()
- btrfs_block_rsv_refill() in relocate_block_group()
- btrfs_delalloc_reserve_metadata() in relocate_file_extent_cluster()
- btrfs_start_transaction() in relocate_block_group()
- btrfs_start_transaction() in create_reloc_inode()
Can be interruped by fatal signal and we can handle it easily. For these call sites, just catch the -EINTR value in btrfs_balance() and count them as canceled.

---
 fs/btrfs/relocation.c | 13 +++++++++++--
 fs/btrfs/volumes.c    |  2 +-
 2 files changed, 12 insertions(+), 3 deletions(-)
fs/btrfs/relocation.c +11 -2
index 2b869fb2e62c..23914edd4710 100644
@@ -1686,12 +1686,21 @@ static noinline_for_stack int merge_reloc_root(struct reloc_control *rc,
 		btrfs_unlock_up_safe(path, 0);
 	}
 
-	min_reserved = fs_info->nodesize * (BTRFS_MAX_LEVEL - 1) * 2;
+	/*
+	 * In merge_reloc_root(), we modify the upper level pointer to swap
+	 * the tree blocks between reloc tree and subvolume tree.
+	 * Thus for tree block COW, we COW at most from level 1 to root level
+	 * for each tree.
+	 *
+	 * Thus the needed metadata space is at most root_level * nodesize,
+	 * and * 2 since we have two trees to COW.
+	 */
+	min_reserved = fs_info->nodesize * btrfs_root_level(root_item) * 2;
 	memset(&next_key, 0, sizeof(next_key));
 
 	while (1) {
 		ret = btrfs_block_rsv_refill(root, rc->block_rsv, min_reserved,
-					     BTRFS_RESERVE_FLUSH_ALL);
+					     BTRFS_RESERVE_FLUSH_LIMIT);
 		if (ret) {
 			err = ret;
 			goto out;
fs/btrfs/volumes.c +0 -0
index aabc6c922e04..d60df30bdc47 100644

fs/btrfs/volumes.c int btrfs_balance(struct btrfs_fs_info *fs_info, · line 4138

Quoted 5 lines
fs/btrfs/volumes.c +1 -1
@@ -4135,7 +4135,7 @@ int btrfs_balance(struct btrfs_fs_info *fs_info,
 	mutex_lock(&fs_info->balance_mutex);
 	if (ret == -ECANCELED && atomic_read(&fs_info->balance_pause_req))
 		btrfs_info(fs_info, "balance: paused");
-	else if (ret == -ECANCELED && atomic_read(&fs_info->balance_cancel_req))
+	else if (ret == -ECANCELED  || ret == -EINTR)

Why do you remove atomic_read(&fs_info->balance_cancel_req) ?

fs/btrfs/volumes.c int btrfs_balance(struct btrfs_fs_info *fs_info, · line 4139

Quoted 1 line
fs/btrfs/volumes.c +0 -0
 		btrfs_info(fs_info, "balance: canceled");

I'm not sure if it would be useful to print the reason, like

- 'canceled: user request'
- 'canceled: interrupted'

Quoted 4 lines
fs/btrfs/volumes.c +0 -0
 	else
 		btrfs_info(fs_info, "balance: ended with status: %d", ret);
Signature · 1 line
2.27.0
author
#3 lore raw

Re: [PATCH v2 2/2] btrfs: relocation: review the call sites which can be interruped by signal

Quoted 70 lines · David Sterba
Quoted 13 lines · Qu Wenruo

Since most metadata reservation calls can return -EINTR when get interruped by fatal signal, we need to review the all the metadata reservation call sites.

In relocation code, the metadata reservation happens in the following sites:
- btrfs_block_rsv_refill() in merge_reloc_root() merge_reloc_root() is a pretty critial section, we don't want get interrupted by signal, so change the flush status to BTRFS_RESERVE_FLUSH_LIMIT, so it won't get interrupted by signal. Since such change can be ENPSPC-prone, also shrink the amount of metadata to reserve a little to avoid deadly ENOSPC there.

Quoted 3 lines

- btrfs_block_rsv_refill() in reserve_metadata_space()
It calls with BTRFS_RESERVE_FLUSH_LIMIT, which won't get interrupred by signal.

This semantics of BTRFS_RESERVE_FLUSH_LIMIT regarding signals should be documented, right now there's a comment but says something about avoidig deadlocks.

Quoted 48 lines

- btrfs_block_rsv_refill() in prepare_to_relocate()
- btrfs_block_rsv_add() in prepare_to_relocate()
- btrfs_block_rsv_refill() in relocate_block_group()
- btrfs_delalloc_reserve_metadata() in relocate_file_extent_cluster()
- btrfs_start_transaction() in relocate_block_group()
- btrfs_start_transaction() in create_reloc_inode()
Can be interruped by fatal signal and we can handle it easily. For these call sites, just catch the -EINTR value in btrfs_balance() and count them as canceled.

---
 fs/btrfs/relocation.c | 13 +++++++++++--
 fs/btrfs/volumes.c    |  2 +-
 2 files changed, 12 insertions(+), 3 deletions(-)
fs/btrfs/relocation.c +11 -2
index 2b869fb2e62c..23914edd4710 100644
@@ -1686,12 +1686,21 @@ static noinline_for_stack int merge_reloc_root(struct reloc_control *rc,
 		btrfs_unlock_up_safe(path, 0);
 	}
 
-	min_reserved = fs_info->nodesize * (BTRFS_MAX_LEVEL - 1) * 2;
+	/*
+	 * In merge_reloc_root(), we modify the upper level pointer to swap
+	 * the tree blocks between reloc tree and subvolume tree.
+	 * Thus for tree block COW, we COW at most from level 1 to root level
+	 * for each tree.
+	 *
+	 * Thus the needed metadata space is at most root_level * nodesize,
+	 * and * 2 since we have two trees to COW.
+	 */
+	min_reserved = fs_info->nodesize * btrfs_root_level(root_item) * 2;
 	memset(&next_key, 0, sizeof(next_key));
 
 	while (1) {
 		ret = btrfs_block_rsv_refill(root, rc->block_rsv, min_reserved,
-					     BTRFS_RESERVE_FLUSH_ALL);
+					     BTRFS_RESERVE_FLUSH_LIMIT);
 		if (ret) {
 			err = ret;
 			goto out;
fs/btrfs/volumes.c +0 -0
index aabc6c922e04..d60df30bdc47 100644
Quoted 8 lines

fs/btrfs/volumes.c int btrfs_balance(struct btrfs_fs_info *fs_info, · line 4138

Quoted 6 lines
fs/btrfs/volumes.c +1 -1
@@ -4135,7 +4135,7 @@ int btrfs_balance(struct btrfs_fs_info *fs_info,
 	mutex_lock(&fs_info->balance_mutex);
 	if (ret == -ECANCELED && atomic_read(&fs_info->balance_pause_req))
 		btrfs_info(fs_info, "balance: paused");
-	else if (ret == -ECANCELED && atomic_read(&fs_info->balance_cancel_req))
+	else if (ret == -ECANCELED  || ret == -EINTR)

Why do you remove atomic_read(&fs_info->balance_cancel_req) ?

Because now btrfs_should_cancel_balance() can return ECANCELED without balance_cancel_req increased due to pending fatal signal.

Quoted 7 lines

fs/btrfs/volumes.c int btrfs_balance(struct btrfs_fs_info *fs_info, · line 4139

Quoted 2 lines
fs/btrfs/volumes.c +0 -0
 		btrfs_info(fs_info, "balance: canceled");

I'm not sure if it would be useful to print the reason, like

- 'canceled: user request'
- 'canceled: interrupted'

To me, if user interrupt the balance progress, it's obvious they want to cancel it.
Thus no need to distinguish btrfs balance cancel and signal cancel.

Thanks,
Qu

Quoted 5 lines
Quoted 5 lines
fs/btrfs/volumes.c +0 -0
 	else
 		btrfs_info(fs_info, "balance: ended with status: %d", ret);
Signature · 1 line
2.27.0
#4 lore raw

Re: [PATCH v2 2/2] btrfs: relocation: review the call sites which can be interruped by signal

Quoted 11 lines · Qu Wenruo
Quoted 9 lines · David Sterba

fs/btrfs/volumes.c int btrfs_balance(struct btrfs_fs_info *fs_info, · line 4138

Quoted 6 lines · Qu Wenruo
fs/btrfs/volumes.c +1 -1
@@ -4135,7 +4135,7 @@ int btrfs_balance(struct btrfs_fs_info *fs_info,
 	mutex_lock(&fs_info->balance_mutex);
 	if (ret == -ECANCELED && atomic_read(&fs_info->balance_pause_req))
 		btrfs_info(fs_info, "balance: paused");
-	else if (ret == -ECANCELED && atomic_read(&fs_info->balance_cancel_req))
+	else if (ret == -ECANCELED  || ret == -EINTR)

Why do you remove atomic_read(&fs_info->balance_cancel_req) ?

Quoted 2 lines

Because now btrfs_should_cancel_balance() can return ECANCELED without balance_cancel_req increased due to pending fatal signal.

Ah right, I misread it as || which would remove one reason for cancellation.

#5 lore raw

Re: [PATCH v2 2/2] btrfs: relocation: review the call sites which can be interruped by signal

In reply to David Sterba
Quoted 14 lines · David Sterba
Quoted 13 lines · Qu Wenruo

Since most metadata reservation calls can return -EINTR when get interruped by fatal signal, we need to review the all the metadata reservation call sites.

In relocation code, the metadata reservation happens in the following sites:
- btrfs_block_rsv_refill() in merge_reloc_root() merge_reloc_root() is a pretty critial section, we don't want get interrupted by signal, so change the flush status to BTRFS_RESERVE_FLUSH_LIMIT, so it won't get interrupted by signal. Since such change can be ENPSPC-prone, also shrink the amount of metadata to reserve a little to avoid deadly ENOSPC there.

Quoted 7 lines
Quoted 3 lines

- btrfs_block_rsv_refill() in reserve_metadata_space()
It calls with BTRFS_RESERVE_FLUSH_LIMIT, which won't get interrupred by signal.

This semantics of BTRFS_RESERVE_FLUSH_LIMIT regarding signals should be documented, right now there's a comment but says something about avoidig deadlocks.

For this, I tend to add one or more patches to add some comment for all FLUSH enums, and all their callers.

I hate when some infrastructure hit me by surprise, and since ticketing system is hidden from a lot of functions, we also need to mention that -EINTR case.

Thanks,
Qu

Quoted 69 lines
Quoted 50 lines

- btrfs_block_rsv_refill() in prepare_to_relocate()
- btrfs_block_rsv_add() in prepare_to_relocate()
- btrfs_block_rsv_refill() in relocate_block_group()
- btrfs_delalloc_reserve_metadata() in relocate_file_extent_cluster()
- btrfs_start_transaction() in relocate_block_group()
- btrfs_start_transaction() in create_reloc_inode()
Can be interruped by fatal signal and we can handle it easily. For these call sites, just catch the -EINTR value in btrfs_balance() and count them as canceled.

---
 fs/btrfs/relocation.c | 13 +++++++++++--
 fs/btrfs/volumes.c    |  2 +-
 2 files changed, 12 insertions(+), 3 deletions(-)
fs/btrfs/relocation.c +11 -2
index 2b869fb2e62c..23914edd4710 100644
@@ -1686,12 +1686,21 @@ static noinline_for_stack int merge_reloc_root(struct reloc_control *rc,
 		btrfs_unlock_up_safe(path, 0);
 	}
 
-	min_reserved = fs_info->nodesize * (BTRFS_MAX_LEVEL - 1) * 2;
+	/*
+	 * In merge_reloc_root(), we modify the upper level pointer to swap
+	 * the tree blocks between reloc tree and subvolume tree.
+	 * Thus for tree block COW, we COW at most from level 1 to root level
+	 * for each tree.
+	 *
+	 * Thus the needed metadata space is at most root_level * nodesize,
+	 * and * 2 since we have two trees to COW.
+	 */
+	min_reserved = fs_info->nodesize * btrfs_root_level(root_item) * 2;
 	memset(&next_key, 0, sizeof(next_key));
 
 	while (1) {
 		ret = btrfs_block_rsv_refill(root, rc->block_rsv, min_reserved,
-					     BTRFS_RESERVE_FLUSH_ALL);
+					     BTRFS_RESERVE_FLUSH_LIMIT);
 		if (ret) {
 			err = ret;
 			goto out;
fs/btrfs/volumes.c +0 -0
index aabc6c922e04..d60df30bdc47 100644

fs/btrfs/volumes.c int btrfs_balance(struct btrfs_fs_info *fs_info, · line 4138

Quoted 5 lines
fs/btrfs/volumes.c +1 -1
@@ -4135,7 +4135,7 @@ int btrfs_balance(struct btrfs_fs_info *fs_info,
 	mutex_lock(&fs_info->balance_mutex);
 	if (ret == -ECANCELED && atomic_read(&fs_info->balance_pause_req))
 		btrfs_info(fs_info, "balance: paused");
-	else if (ret == -ECANCELED && atomic_read(&fs_info->balance_cancel_req))
+	else if (ret == -ECANCELED  || ret == -EINTR)

Why do you remove atomic_read(&fs_info->balance_cancel_req) ?

fs/btrfs/volumes.c int btrfs_balance(struct btrfs_fs_info *fs_info, · line 4139

Quoted 1 line
fs/btrfs/volumes.c +0 -0
 		btrfs_info(fs_info, "balance: canceled");

I'm not sure if it would be useful to print the reason, like

- 'canceled: user request'
- 'canceled: interrupted'

Quoted 4 lines
fs/btrfs/volumes.c +0 -0
 	else
 		btrfs_info(fs_info, "balance: ended with status: %d", ret);
Signature · 1 line
2.27.0
#6 lore raw
In reply to Qu Wenruo
Quoted 9 lines · Qu Wenruo

From the mentioned forced read-only, to later balance error due to half dropped reloc trees.

[FIX]
Fix this problem by using btrfs_join_transaction() if btrfs_drop_snapshot() is called from relocation context.

As btrfs_join_transaction() won't wait full tickets, it won't get interrupted from signal.

Could you please rephrase the text above?

#7 lore raw
Quoted 5 lines · David Sterba
Quoted 3 lines · Qu Wenruo

From the mentioned forced read-only, to later balance error due to half dropped reloc trees.

Quoted 9 lines
Quoted 6 lines

[FIX]
Fix this problem by using btrfs_join_transaction() if btrfs_drop_snapshot() is called from relocation context.

As btrfs_join_transaction() won't wait full tickets, it won't get interrupted from signal.

Could you please rephrase the text above?

How about this:

As btrfs_join_transaction() won't reserve new metadata space, it won't get interrupted by signal at all.

Thanks,
Qu