btrfs: avoid possible signal interruption for btrfs_drop_snapshot() on relocation tree
PATCH v2 1/2 btrfs: avoid possible signal interruption for btrfs_drop_snapshot() on relocation tree
[BUG]
There is a bug report about bad signal timing could lead to read-only fs during balance:
BTRFS info (device xvdb): balance: start -d -m -s
BTRFS info (device xvdb): relocating block group 73001861120 flags metadata BTRFS info (device xvdb): found 12236 extents, stage: move data extents BTRFS info (device xvdb): relocating block group 71928119296 flags data BTRFS info (device xvdb): found 3 extents, stage: move data extents BTRFS info (device xvdb): found 3 extents, stage: update data pointers BTRFS info (device xvdb): relocating block group 60922265600 flags metadata
BTRFS: error (device xvdb) in btrfs_drop_snapshot:5505: errno=-4 unknown BTRFS info (device xvdb): forced readonly
BTRFS info (device xvdb): balance: ended with status: -4
[CAUSE]
The direct cause is the -EINTR from the following call chain when a fatal signal is pending:
relocate_block_group()
|- clean_dirty_subvols()
|- btrfs_drop_snapshot()
|- btrfs_start_transaction()
|- btrfs_delayed_refs_rsv_refill()
|- btrfs_reserve_metadata_bytes()
|- __reserve_metadata_bytes()
|- wait_reserve_ticket()
|- prepare_to_wait_event();
|- ticket->error = -EINTR;Normally this behavior is fine for most btrfs_start_transaction() callers, as they need to catch the fatal signal and exit asap.
However for balance, especially for the clean_dirty_subvols() case, we're already doing cleanup works, such -EINTR from btrfs_drop_snapshot() could cause a lot of unexpected problems.
From the mentioned forced read-only, to later balance error due to half dropped reloc trees.
[FIX]
Fix this problem by using btrfs_join_transaction() if btrfs_drop_snapshot() is called from relocation context.
As btrfs_join_transaction() won't wait full tickets, it won't get interrupted from signal.
--- fs/btrfs/extent-tree.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
fs/btrfs/extent-tree.c +4 -1
@@ -5298,7 +5298,10 @@ int btrfs_drop_snapshot(struct btrfs_root *root, int update_ref, int for_reloc) goto out; } - trans =btrfs_start_transaction (tree_root,0 ); +if (for_reloc) + trans =btrfs_join_transaction (tree_root); +else + trans =btrfs_start_transaction (tree_root,0 );if (IS_ERR (trans)) { err =PTR_ERR (trans); goto out_free;
Signature · 1 line
2.27.0
PATCH v2 2/2 btrfs: relocation: review the call sites which can be interruped by signal
Since most metadata reservation calls can return -EINTR when get interruped by fatal signal, we need to review the all the metadata reservation call sites.
In relocation code, the metadata reservation happens in the following sites:
- btrfs_block_rsv_refill() in merge_reloc_root() merge_reloc_root() is a pretty critial section, we don't want get interrupted by signal, so change the flush status to BTRFS_RESERVE_FLUSH_LIMIT, so it won't get interrupted by signal. Since such change can be ENPSPC-prone, also shrink the amount of metadata to reserve a little to avoid deadly ENOSPC there.
- btrfs_block_rsv_refill() in reserve_metadata_space()
It calls with BTRFS_RESERVE_FLUSH_LIMIT, which won't get interrupred by signal.
- btrfs_block_rsv_refill() in prepare_to_relocate()
- btrfs_block_rsv_add() in prepare_to_relocate()
- btrfs_block_rsv_refill() in relocate_block_group()
- btrfs_delalloc_reserve_metadata() in relocate_file_extent_cluster()
- btrfs_start_transaction() in relocate_block_group()
- btrfs_start_transaction() in create_reloc_inode()
Can be interruped by fatal signal and we can handle it easily. For these call sites, just catch the -EINTR value in btrfs_balance() and count them as canceled.
--- fs/btrfs/relocation.c | 13 +++++++++++-- fs/btrfs/volumes.c | 2 +- 2 files changed, 12 insertions(+), 3 deletions(-)
fs/btrfs/relocation.c +11 -2
@@ -1686,12 +1686,21 @@ static noinline_for_stack int merge_reloc_root(struct reloc_control *rc,btrfs_unlock_up_safe (path,0 ); } - min_reserved = fs_info->nodesize * (BTRFS_MAX_LEVEL -1 ) *2 ; +/* +* In merge_reloc_root(), we modify the upper level pointer to swap +* the tree blocks between reloc tree and subvolume tree. +* Thus for tree block COW, we COW at most from level 1 to root level +* for each tree. +* +* Thus the needed metadata space is at most root_level * nodesize, +* and * 2 since we have two trees to COW. +*/ + min_reserved = fs_info->nodesize *btrfs_root_level (root_item) *2 ;memset (&next_key,0 ,sizeof (next_key));while (1 ) { ret =btrfs_block_rsv_refill (root, rc->block_rsv, min_reserved, -BTRFS_RESERVE_FLUSH_ALL ); +BTRFS_RESERVE_FLUSH_LIMIT );if (ret) { err = ret; goto out;
fs/btrfs/volumes.c +1 -1
@@ -4135,7 +4135,7 @@ int btrfs_balance(struct btrfs_fs_info *fs_info,mutex_lock (&fs_info->balance_mutex);if (ret == -ECANCELED &&atomic_read (&fs_info->balance_pause_req))btrfs_info (fs_info,"balance: paused" ); -else if (ret == -ECANCELED &&atomic_read (&fs_info->balance_cancel_req)) +else if (ret == -ECANCELED || ret == -EINTR )btrfs_info (fs_info,"balance: canceled" );else btrfs_info (fs_info,"balance: ended with status: %d" , ret);
Signature · 1 line
2.27.0
Re: [PATCH v2 2/2] btrfs: relocation: review the call sites which can be interruped by signal
Quoted 13 lines · Qu Wenruo
Since most metadata reservation calls can return
-EINTRwhen get interruped by fatal signal, we need to review the all the metadata reservation call sites.In relocation code, the metadata reservation happens in the following sites:
-btrfs_block_rsv_refill()inmerge_reloc_root()merge_reloc_root()is a pretty critial section, we don't want get interrupted by signal, so change the flush status toBTRFS_RESERVE_FLUSH_LIMIT, so it won't get interrupted by signal. Since such change can be ENPSPC-prone, also shrink the amount of metadata to reserve a little to avoid deadly ENOSPC there.
Quoted 3 lines
-
btrfs_block_rsv_refill()inreserve_metadata_space()
It calls withBTRFS_RESERVE_FLUSH_LIMIT, which won't get interrupred by signal.
This semantics of BTRFS_RESERVE_FLUSH_LIMIT regarding signals should be documented, right now there's a comment but says something about avoidig deadlocks.
Quoted 49 lines
-
btrfs_block_rsv_refill()inprepare_to_relocate()
-btrfs_block_rsv_add()inprepare_to_relocate()
-btrfs_block_rsv_refill()inrelocate_block_group()
-btrfs_delalloc_reserve_metadata()inrelocate_file_extent_cluster()
-btrfs_start_transaction()inrelocate_block_group()
-btrfs_start_transaction()increate_reloc_inode()
Can be interruped by fatal signal and we can handle it easily. For these call sites, just catch the-EINTRvalue inbtrfs_balance()and count them as canceled.Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/relocation.c | 13 +++++++++++-- fs/btrfs/volumes.c | 2 +- 2 files changed, 12 insertions(+), 3 deletions(-)fs/btrfs/relocation.c +11
-2@@ -1686,12 +1686,21 @@ static noinline_for_stack int merge_reloc_root(struct reloc_control *rc,btrfs_unlock_up_safe (path,0 ); } - min_reserved = fs_info->nodesize * (BTRFS_MAX_LEVEL -1 ) *2 ; +/* +* In merge_reloc_root(), we modify the upper level pointer to swap +* the tree blocks between reloc tree and subvolume tree. +* Thus for tree block COW, we COW at most from level 1 to root level +* for each tree. +* +* Thus the needed metadata space is at most root_level * nodesize, +* and * 2 since we have two trees to COW. +*/ + min_reserved = fs_info->nodesize *btrfs_root_level (root_item) *2 ;memset (&next_key,0 ,sizeof (next_key));while (1 ) { ret =btrfs_block_rsv_refill (root, rc->block_rsv, min_reserved, -BTRFS_RESERVE_FLUSH_ALL ); +BTRFS_RESERVE_FLUSH_LIMIT );if (ret) { err = ret; goto out;fs/btrfs/volumes.c +0
-0
fs/btrfs/volumes.c int btrfs_balance(struct btrfs_fs_info *fs_info, · line 4138
Quoted 5 lines
fs/btrfs/volumes.c +1
-1@@ -4135,7 +4135,7 @@ int btrfs_balance(struct btrfs_fs_info *fs_info,mutex_lock (&fs_info->balance_mutex);if (ret == -ECANCELED &&atomic_read (&fs_info->balance_pause_req))btrfs_info (fs_info,"balance: paused" ); -else if (ret == -ECANCELED &&atomic_read (&fs_info->balance_cancel_req)) +else if (ret == -ECANCELED || ret == -EINTR )
Why do you remove atomic_read(&fs_info->balance_cancel_req) ?
fs/btrfs/volumes.c int btrfs_balance(struct btrfs_fs_info *fs_info, · line 4139
Quoted 1 line
fs/btrfs/volumes.c +0
-0btrfs_info (fs_info,"balance: canceled" );
I'm not sure if it would be useful to print the reason, like
- 'canceled: user request'
- 'canceled: interrupted'
Quoted 4 lines
fs/btrfs/volumes.c +0
-0else btrfs_info (fs_info,"balance: ended with status: %d" , ret);Signature · 1 line
2.27.0
Re: [PATCH v2 2/2] btrfs: relocation: review the call sites which can be interruped by signal
Quoted 70 lines · David Sterba
Quoted 13 lines · Qu Wenruo
Since most metadata reservation calls can return
-EINTRwhen get interruped by fatal signal, we need to review the all the metadata reservation call sites.In relocation code, the metadata reservation happens in the following sites:
-btrfs_block_rsv_refill()inmerge_reloc_root()merge_reloc_root()is a pretty critial section, we don't want get interrupted by signal, so change the flush status toBTRFS_RESERVE_FLUSH_LIMIT, so it won't get interrupted by signal. Since such change can be ENPSPC-prone, also shrink the amount of metadata to reserve a little to avoid deadly ENOSPC there.Quoted 3 lines
-
btrfs_block_rsv_refill()inreserve_metadata_space()
It calls withBTRFS_RESERVE_FLUSH_LIMIT, which won't get interrupred by signal.This semantics of
BTRFS_RESERVE_FLUSH_LIMITregarding signals should be documented, right now there's a comment but says something about avoidig deadlocks.Quoted 48 lines
-
btrfs_block_rsv_refill()inprepare_to_relocate()
-btrfs_block_rsv_add()inprepare_to_relocate()
-btrfs_block_rsv_refill()inrelocate_block_group()
-btrfs_delalloc_reserve_metadata()inrelocate_file_extent_cluster()
-btrfs_start_transaction()inrelocate_block_group()
-btrfs_start_transaction()increate_reloc_inode()
Can be interruped by fatal signal and we can handle it easily. For these call sites, just catch the-EINTRvalue inbtrfs_balance()and count them as canceled.Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/relocation.c | 13 +++++++++++-- fs/btrfs/volumes.c | 2 +- 2 files changed, 12 insertions(+), 3 deletions(-)fs/btrfs/relocation.c +11
-2@@ -1686,12 +1686,21 @@ static noinline_for_stack int merge_reloc_root(struct reloc_control *rc,btrfs_unlock_up_safe (path,0 ); } - min_reserved = fs_info->nodesize * (BTRFS_MAX_LEVEL -1 ) *2 ; +/* +* In merge_reloc_root(), we modify the upper level pointer to swap +* the tree blocks between reloc tree and subvolume tree. +* Thus for tree block COW, we COW at most from level 1 to root level +* for each tree. +* +* Thus the needed metadata space is at most root_level * nodesize, +* and * 2 since we have two trees to COW. +*/ + min_reserved = fs_info->nodesize *btrfs_root_level (root_item) *2 ;memset (&next_key,0 ,sizeof (next_key));while (1 ) { ret =btrfs_block_rsv_refill (root, rc->block_rsv, min_reserved, -BTRFS_RESERVE_FLUSH_ALL ); +BTRFS_RESERVE_FLUSH_LIMIT );if (ret) { err = ret; goto out;fs/btrfs/volumes.c +0
-0
Quoted 8 lines
fs/btrfs/volumes.c int btrfs_balance(struct btrfs_fs_info *fs_info, · line 4138
Quoted 6 lines
fs/btrfs/volumes.c +1
-1@@ -4135,7 +4135,7 @@ int btrfs_balance(struct btrfs_fs_info *fs_info,mutex_lock (&fs_info->balance_mutex);if (ret == -ECANCELED &&atomic_read (&fs_info->balance_pause_req))btrfs_info (fs_info,"balance: paused" ); -else if (ret == -ECANCELED &&atomic_read (&fs_info->balance_cancel_req)) +else if (ret == -ECANCELED || ret == -EINTR )Why do you remove
atomic_read(&fs_info->balance_cancel_req)?
Because now btrfs_should_cancel_balance() can return ECANCELED without balance_cancel_req increased due to pending fatal signal.
Quoted 7 lines
fs/btrfs/volumes.c int btrfs_balance(struct btrfs_fs_info *fs_info, · line 4139
Quoted 2 lines
fs/btrfs/volumes.c +0
-0btrfs_info (fs_info,"balance: canceled" );I'm not sure if it would be useful to print the reason, like
- 'canceled: user request'
- 'canceled: interrupted'
To me, if user interrupt the balance progress, it's obvious they want to cancel it.
Thus no need to distinguish btrfs balance cancel and signal cancel.
Thanks,
Qu
Quoted 5 lines
Quoted 5 lines
fs/btrfs/volumes.c +0
-0else btrfs_info (fs_info,"balance: ended with status: %d" , ret);Signature · 1 line
2.27.0
Re: [PATCH v2 2/2] btrfs: relocation: review the call sites which can be interruped by signal
Quoted 11 lines · Qu Wenruo
Quoted 9 lines · David Sterba
fs/btrfs/volumes.c int btrfs_balance(struct btrfs_fs_info *fs_info, · line 4138
Quoted 6 lines · Qu Wenruo
fs/btrfs/volumes.c +1
-1@@ -4135,7 +4135,7 @@ int btrfs_balance(struct btrfs_fs_info *fs_info,mutex_lock (&fs_info->balance_mutex);if (ret == -ECANCELED &&atomic_read (&fs_info->balance_pause_req))btrfs_info (fs_info,"balance: paused" ); -else if (ret == -ECANCELED &&atomic_read (&fs_info->balance_cancel_req)) +else if (ret == -ECANCELED || ret == -EINTR )Why do you remove
atomic_read(&fs_info->balance_cancel_req)?
Quoted 2 lines
Because now
btrfs_should_cancel_balance()can return ECANCELED withoutbalance_cancel_reqincreased due to pending fatal signal.
Ah right, I misread it as || which would remove one reason for cancellation.
Re: [PATCH v2 2/2] btrfs: relocation: review the call sites which can be interruped by signal
In reply to David SterbaQuoted 14 lines · David Sterba
Quoted 13 lines · Qu Wenruo
Since most metadata reservation calls can return
-EINTRwhen get interruped by fatal signal, we need to review the all the metadata reservation call sites.In relocation code, the metadata reservation happens in the following sites:
-btrfs_block_rsv_refill()inmerge_reloc_root()merge_reloc_root()is a pretty critial section, we don't want get interrupted by signal, so change the flush status toBTRFS_RESERVE_FLUSH_LIMIT, so it won't get interrupted by signal. Since such change can be ENPSPC-prone, also shrink the amount of metadata to reserve a little to avoid deadly ENOSPC there.
Quoted 7 lines
Quoted 3 lines
-
btrfs_block_rsv_refill()inreserve_metadata_space()
It calls withBTRFS_RESERVE_FLUSH_LIMIT, which won't get interrupred by signal.This semantics of
BTRFS_RESERVE_FLUSH_LIMITregarding signals should be documented, right now there's a comment but says something about avoidig deadlocks.
For this, I tend to add one or more patches to add some comment for all FLUSH enums, and all their callers.
I hate when some infrastructure hit me by surprise, and since ticketing system is hidden from a lot of functions, we also need to mention that -EINTR case.
Thanks,
Qu
Quoted 69 lines
Quoted 50 lines
-
btrfs_block_rsv_refill()inprepare_to_relocate()
-btrfs_block_rsv_add()inprepare_to_relocate()
-btrfs_block_rsv_refill()inrelocate_block_group()
-btrfs_delalloc_reserve_metadata()inrelocate_file_extent_cluster()
-btrfs_start_transaction()inrelocate_block_group()
-btrfs_start_transaction()increate_reloc_inode()
Can be interruped by fatal signal and we can handle it easily. For these call sites, just catch the-EINTRvalue inbtrfs_balance()and count them as canceled.Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/relocation.c | 13 +++++++++++-- fs/btrfs/volumes.c | 2 +- 2 files changed, 12 insertions(+), 3 deletions(-)fs/btrfs/relocation.c +11
-2@@ -1686,12 +1686,21 @@ static noinline_for_stack int merge_reloc_root(struct reloc_control *rc,btrfs_unlock_up_safe (path,0 ); } - min_reserved = fs_info->nodesize * (BTRFS_MAX_LEVEL -1 ) *2 ; +/* +* In merge_reloc_root(), we modify the upper level pointer to swap +* the tree blocks between reloc tree and subvolume tree. +* Thus for tree block COW, we COW at most from level 1 to root level +* for each tree. +* +* Thus the needed metadata space is at most root_level * nodesize, +* and * 2 since we have two trees to COW. +*/ + min_reserved = fs_info->nodesize *btrfs_root_level (root_item) *2 ;memset (&next_key,0 ,sizeof (next_key));while (1 ) { ret =btrfs_block_rsv_refill (root, rc->block_rsv, min_reserved, -BTRFS_RESERVE_FLUSH_ALL ); +BTRFS_RESERVE_FLUSH_LIMIT );if (ret) { err = ret; goto out;fs/btrfs/volumes.c +0
-0fs/btrfs/volumes.c int btrfs_balance(struct btrfs_fs_info *fs_info, · line 4138
Quoted 5 lines
fs/btrfs/volumes.c +1
-1@@ -4135,7 +4135,7 @@ int btrfs_balance(struct btrfs_fs_info *fs_info,mutex_lock (&fs_info->balance_mutex);if (ret == -ECANCELED &&atomic_read (&fs_info->balance_pause_req))btrfs_info (fs_info,"balance: paused" ); -else if (ret == -ECANCELED &&atomic_read (&fs_info->balance_cancel_req)) +else if (ret == -ECANCELED || ret == -EINTR )Why do you remove
atomic_read(&fs_info->balance_cancel_req)?fs/btrfs/volumes.c int btrfs_balance(struct btrfs_fs_info *fs_info, · line 4139
Quoted 1 line
fs/btrfs/volumes.c +0
-0btrfs_info (fs_info,"balance: canceled" );I'm not sure if it would be useful to print the reason, like
- 'canceled: user request'
- 'canceled: interrupted'Quoted 4 lines
fs/btrfs/volumes.c +0
-0else btrfs_info (fs_info,"balance: ended with status: %d" , ret);Signature · 1 line
2.27.0
Quoted 9 lines · Qu Wenruo
From the mentioned forced read-only, to later balance error due to half dropped reloc trees.
[FIX]
Fix this problem by usingbtrfs_join_transaction()ifbtrfs_drop_snapshot()is called from relocation context.As
btrfs_join_transaction()won't wait full tickets, it won't get interrupted from signal.
Could you please rephrase the text above?
Quoted 5 lines · David Sterba
Quoted 3 lines · Qu Wenruo
From the mentioned forced read-only, to later balance error due to half dropped reloc trees.
Quoted 9 lines
Quoted 6 lines
[FIX]
Fix this problem by usingbtrfs_join_transaction()ifbtrfs_drop_snapshot()is called from relocation context.As
btrfs_join_transaction()won't wait full tickets, it won't get interrupted from signal.Could you please rephrase the text above?
How about this:
As btrfs_join_transaction() won't reserve new metadata space, it won't get interrupted by signal at all.
Thanks,
Qu