Make pinned extents tracking per-transaction
Subsystems (2)
cover Make pinned extents tracking per-transaction
I've finally managed to finish and test the pinned extents rework. So here it is.
The idea is to move pinnex extents tracking from the global fs_info->pinned_extents, which is just a pointer to fs_info->freed_extents[] array members to a per transaction structure. This will allow to cleanup pinned extents information during transaction abort.
The bulk of the changes are necessary to ensure a valid transaction handle is passed to every function that utilizes fs_info->pinned_extents. Also it was necessary to account for the peculiarities of excluded extents which are also tracked in ->freed_extents array but with a different flag (yuck).
First 9 patches propagate btrfs_trans_handle where it's necessary. Patch 10 factors out pinned extent clean up to make Patch 11 more readable , alternatively the changes in patch 10 had to be in patch 11 which would have made it messier for review.
I believe this series doesn't bring any user facing changes it (hopefully) streamlines the code and makes it apparent what the lifetime of pinned extents is and paves the way for further cleanups of BUG_ON.
Nikolay Borisov (11):
btrfs: Perform pinned cleanup directly in btrfs_destroy_delayed_refs
btrfs: Make btrfs_pin_extent take trans handle
btrfs: Introduce unaccount_log_buffer
btrfs: Call btrfs_pin_reserved_extent only during active transaction
btrfs: Make btrfs_pin_reserved_extent take transaction
btrfs: Make btrfs_pin_extent_for_log_replay take transaction handle
btrfs: Make pin_down_extent take btrfs_trans_handle
btrfs: Pass trans handle to write_pinned_extent_entries
btrfs: Mark pinned log extents as excluded
btrfs: Factor out pinned extent clean up in btrfs_delete_unused_bgs
btrfs: Use btrfs_transaction::pinned_extents
fs/btrfs/block-group.c | 85 +++++++++++++++++++++++------------- fs/btrfs/ctree.h | 12 ++--- fs/btrfs/disk-io.c | 56 ++++++++++++------------ fs/btrfs/extent-io-tree.h | 3 +- fs/btrfs/extent-tree.c | 70 +++++++++++++---------------- fs/btrfs/free-space-cache.c | 5 ++- fs/btrfs/tests/btrfs-tests.c | 7 +-- fs/btrfs/transaction.c | 1 + fs/btrfs/transaction.h | 1 + fs/btrfs/tree-log.c | 70 +++++++++++++++++++---------- include/trace/events/btrfs.h | 3 +- 11 files changed, 174 insertions(+), 139 deletions(-)
--
2.17.1
PATCH 1/11 btrfs: Perform pinned cleanup directly in btrfs_destroy_delayed_refs
Having btrfs_destroy_delayed_refs call btrfs_pin_extent is problematic for making pinned extents tracking per-transaction since btrfs_trans_handle cannot be passed to btrfs_pin_extent in this context. Additionally delayed refs heads pinned in btrfs_destroy_delayed_refs are going to be handled very closely, in btrfs_destroy_pinned_extent.
To enable btrfs_pin_extent to take btrfs_trans_handle simply open code it in btrfs_destroy_delayed_refs and call btrfs_error_unpin_extent_range on the range. This enables us to do less work in btrfs_destroy_pinned_extent and leaves btrfs_pin_extent being called in contexts which have a valid btrfs_trans_handle.
--- fs/btrfs/disk-io.c | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-)
fs/btrfs/disk-io.c +23 -3
@@ -42,6 +42,7 @@#include "ref-verify.h" #include "block-group.h" #include "discard.h" +#include "space-info.h" #define BTRFS_SUPER_FLAG_SUPP (BTRFS_HEADER_FLAG_WRITTEN |\ BTRFS_HEADER_FLAG_RELOC |\
@@ -4261,9 +4262,28 @@ static int btrfs_destroy_delayed_refs(struct btrfs_transaction *trans,spin_unlock (&delayed_refs->lock);mutex_unlock (&head->mutex); -if (pin_bytes) -btrfs_pin_extent (fs_info, head->bytenr, - head->num_bytes,1 ); +if (pin_bytes) { +struct btrfs_block_group *cache; + cache =btrfs_lookup_block_group (fs_info, head->bytenr); +BUG_ON (!cache); + +spin_lock (&cache->space_info->lock); +spin_lock (&cache->lock); + cache->pinned += head->num_bytes; +btrfs_space_info_update_bytes_pinned (fs_info, + cache->space_info, head->num_bytes); + cache->reserved -= head->num_bytes; + cache->space_info->bytes_reserved -= head->num_bytes; +spin_unlock (&cache->lock); +spin_unlock (&cache->space_info->lock); +percpu_counter_add_batch (&cache->space_info->total_bytes_pinned, + head->num_bytes,BTRFS_TOTAL_BYTES_PINNED_BATCH ); + +btrfs_put_block_group (cache); + +btrfs_error_unpin_extent_range (fs_info, head->bytenr, + head->bytenr + head->num_bytes -1 ); + }btrfs_cleanup_ref_head_accounting (fs_info, delayed_refs, head);btrfs_put_delayed_ref_head (head);cond_resched ();
Signature · 1 line
2.17.1
Re: [PATCH 01/11] btrfs: Perform pinned cleanup directly in btrfs_destroy_delayed_refs
Quoted 6 lines · Nikolay Borisov
Having
btrfs_destroy_delayed_refscallbtrfs_pin_extentis problematic for making pinned extents tracking per-transaction sincebtrfs_trans_handlecannot be passed tobtrfs_pin_extentin this context. Additionally delayed refs heads pinned inbtrfs_destroy_delayed_refsare going to be handled very closely, inbtrfs_destroy_pinned_extent.
Quoted 7 lines
To enable
btrfs_pin_extentto takebtrfs_trans_handlesimply open code it inbtrfs_destroy_delayed_refsand callbtrfs_error_unpin_extent_rangeon the range. This enables us to do less work inbtrfs_destroy_pinned_extentand leavesbtrfs_pin_extentbeing called in contexts which have a validbtrfs_trans_handle.Signed-off-by Nikolay Borisov <[email protected]>
Thanks,
Josef
Re: [PATCH 01/11] btrfs: Perform pinned cleanup directly in btrfs_destroy_delayed_refs
In reply to Nikolay BorisovQuoted 30 lines · Nikolay Borisov
Having
btrfs_destroy_delayed_refscallbtrfs_pin_extentis problematic for making pinned extents tracking per-transaction sincebtrfs_trans_handlecannot be passed tobtrfs_pin_extentin this context. Additionally delayed refs heads pinned inbtrfs_destroy_delayed_refsare going to be handled very closely, inbtrfs_destroy_pinned_extent.To enable
btrfs_pin_extentto takebtrfs_trans_handlesimply open code it inbtrfs_destroy_delayed_refsand callbtrfs_error_unpin_extent_rangeon the range. This enables us to do less work inbtrfs_destroy_pinned_extentand leavesbtrfs_pin_extentbeing called in contexts which have a validbtrfs_trans_handle.Signed-off-by Nikolay Borisov <[email protected]>--- fs/btrfs/disk-io.c | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-)fs/btrfs/disk-io.c +1
-0@@ -42,6 +42,7 @@#include "ref-verify.h" #include "block-group.h" #include "discard.h" +#include "space-info.h" #define BTRFS_SUPER_FLAG_SUPP (BTRFS_HEADER_FLAG_WRITTEN |\ BTRFS_HEADER_FLAG_RELOC |\
fs/btrfs/disk-io.c static int btrfs_destroy_delayed_refs(struct btrfs_transaction *trans, · line 4268
Quoted 10 lines
fs/btrfs/disk-io.c +4
-3@@ -4261,9 +4262,28 @@ static int btrfs_destroy_delayed_refs(struct btrfs_transaction *trans,spin_unlock (&delayed_refs->lock);mutex_unlock (&head->mutex); -if (pin_bytes) -btrfs_pin_extent (fs_info, head->bytenr, - head->num_bytes,1 ); +if (pin_bytes) { +struct btrfs_block_group *cache; + cache =btrfs_lookup_block_group (fs_info, head->bytenr); +BUG_ON (!cache);
So this BUG_ON is propagated from btrfs_pin_extent but not turned into proper error handling in any of the followup patches.
Quoted 7 lines
fs/btrfs/disk-io.c +7
-0+ +spin_lock (&cache->space_info->lock); +spin_lock (&cache->lock); + cache->pinned += head->num_bytes; +btrfs_space_info_update_bytes_pinned (fs_info, + cache->space_info, head->num_bytes); + cache->reserved -= head->num_bytes;
fs/btrfs/disk-io.c static int btrfs_destroy_delayed_refs(struct btrfs_transaction *trans, · line 4285
Quoted 10 lines
fs/btrfs/disk-io.c +10
-0+ cache->space_info->bytes_reserved -= head->num_bytes; +spin_unlock (&cache->lock); +spin_unlock (&cache->space_info->lock); +percpu_counter_add_batch (&cache->space_info->total_bytes_pinned, + head->num_bytes,BTRFS_TOTAL_BYTES_PINNED_BATCH ); + +btrfs_put_block_group (cache); + +btrfs_error_unpin_extent_range (fs_info, head->bytenr, + head->bytenr + head->num_bytes -1 );
This should also handle errors
Quoted 6 lines
fs/btrfs/disk-io.c +1
-0+ }btrfs_cleanup_ref_head_accounting (fs_info, delayed_refs, head);btrfs_put_delayed_ref_head (head);cond_resched ();Signature · 1 line
2.17.1
Re: [PATCH 01/11] btrfs: Perform pinned cleanup directly in btrfs_destroy_delayed_refs
Quoted 34 lines · David Sterba
Quoted 33 lines · Nikolay Borisov
Having
btrfs_destroy_delayed_refscallbtrfs_pin_extentis problematic for making pinned extents tracking per-transaction sincebtrfs_trans_handlecannot be passed tobtrfs_pin_extentin this context. Additionally delayed refs heads pinned inbtrfs_destroy_delayed_refsare going to be handled very closely, inbtrfs_destroy_pinned_extent.To enable
btrfs_pin_extentto takebtrfs_trans_handlesimply open code it inbtrfs_destroy_delayed_refsand callbtrfs_error_unpin_extent_rangeon the range. This enables us to do less work inbtrfs_destroy_pinned_extentand leavesbtrfs_pin_extentbeing called in contexts which have a validbtrfs_trans_handle.Signed-off-by Nikolay Borisov <[email protected]>--- fs/btrfs/disk-io.c | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-)fs/btrfs/disk-io.c +1
-0@@ -42,6 +42,7 @@#include "ref-verify.h" #include "block-group.h" #include "discard.h" +#include "space-info.h" #define BTRFS_SUPER_FLAG_SUPP (BTRFS_HEADER_FLAG_WRITTEN |\ BTRFS_HEADER_FLAG_RELOC |\@@ -4261,9 +4262,28 @@ static int btrfs_destroy_delayed_refs(struct btrfs_transaction *trans,spin_unlock (&delayed_refs->lock);mutex_unlock (&head->mutex);
Quoted 10 lines
fs/btrfs/disk-io.c static int btrfs_destroy_delayed_refs(struct btrfs_transaction *trans, · line 4268
Quoted 7 lines
fs/btrfs/disk-io.c +4
-3-if (pin_bytes) -btrfs_pin_extent (fs_info, head->bytenr, - head->num_bytes,1 ); +if (pin_bytes) { +struct btrfs_block_group *cache; + cache =btrfs_lookup_block_group (fs_info, head->bytenr); +BUG_ON (!cache);So this
BUG_ONis propagated frombtrfs_pin_extentbut not turned into proper error handling in any of the followup patches.
This BUGON should be there or it could be turned into an ASSERT. It's used to catch racing block group freeing i.e if it triggers it's a logical error, a real bug.
Quoted 1 line
<snip>
Quoted 4 lines
fs/btrfs/disk-io.c static int btrfs_destroy_delayed_refs(struct btrfs_transaction *trans, · line 4285
Quoted 2 lines
fs/btrfs/disk-io.c +2
-0+btrfs_error_unpin_extent_range (fs_info, head->bytenr, + head->bytenr + head->num_bytes -1 );This should also handle errors
Turns out unpin_extent_range cane just return void, since it cannot fail so there's nothing to handle.
Quoted 7 lines
Quoted 7 lines
fs/btrfs/disk-io.c +1
-0+ }btrfs_cleanup_ref_head_accounting (fs_info, delayed_refs, head);btrfs_put_delayed_ref_head (head);cond_resched ();Signature · 1 line
2.17.1
PATCH 2/11 btrfs: Make btrfs_pin_extent take trans handle
In reply to Nikolay BorisovPreparation for switching pinned extent tracking to a per-transaction basis.
--- fs/btrfs/ctree.h | 4 ++-- fs/btrfs/extent-tree.c | 18 +++++++----------- 2 files changed, 9 insertions(+), 13 deletions(-)
fs/btrfs/ctree.h +2 -2
@@ -2460,8 +2460,8 @@ int btrfs_lookup_data_extent(struct btrfs_fs_info *fs_info, u64 start, u64 len);int btrfs_lookup_extent_info (struct btrfs_trans_handle *trans,struct btrfs_fs_info *fs_info,u64 bytenr,u64 offset,int metadata,u64 *refs,u64 *flags); -int btrfs_pin_extent (struct btrfs_fs_info *fs_info, -u64 bytenr,u64 num,int reserved); +int btrfs_pin_extent (struct btrfs_trans_handle *trans,u64 bytenr,u64 num, +int reserved);int btrfs_pin_extent_for_log_replay (struct btrfs_fs_info *fs_info,u64 bytenr,u64 num_bytes);int btrfs_exclude_logged_extents (struct extent_buffer *eb);
fs/btrfs/extent-tree.c +7 -11
@@ -1705,8 +1705,8 @@ static int run_one_delayed_ref(struct btrfs_trans_handle *trans,if (trans->aborted) {if (insert_reserved) -btrfs_pin_extent (trans->fs_info, node->bytenr, - node->num_bytes,1 ); +btrfs_pin_extent (trans, node->bytenr, node->num_bytes, +1 );return 0 ; }
@@ -1721,8 +1721,7 @@ static int run_one_delayed_ref(struct btrfs_trans_handle *trans,else BUG ();if (ret && insert_reserved) -btrfs_pin_extent (trans->fs_info, node->bytenr, - node->num_bytes,1 ); +btrfs_pin_extent (trans, node->bytenr, node->num_bytes,1 );return ret; }
@@ -1867,8 +1866,7 @@ static int cleanup_ref_head(struct btrfs_trans_handle *trans,spin_unlock (&delayed_refs->lock);if (head->must_insert_reserved) { -btrfs_pin_extent (fs_info, head->bytenr, - head->num_bytes,1 ); +btrfs_pin_extent (trans, head->bytenr, head->num_bytes,1 );if (head->is_data) { ret =btrfs_del_csums (trans, fs_info->csum_root, head->bytenr, head->num_bytes);
@@ -2612,14 +2610,12 @@ static int pin_down_extent(struct btrfs_block_group *cache,return 0 ; } -int btrfs_pin_extent (struct btrfs_fs_info *fs_info, +int btrfs_pin_extent (struct btrfs_trans_handle *trans,u64 bytenr,u64 num_bytes,int reserved) {struct btrfs_block_group *cache; -ASSERT (fs_info->running_transaction); - - cache =btrfs_lookup_block_group (fs_info, bytenr); + cache =btrfs_lookup_block_group (trans->fs_info, bytenr);BUG_ON (!cache);/* Logic error */ pin_down_extent (cache, bytenr, num_bytes, reserved);
@@ -3345,7 +3341,7 @@ int btrfs_free_extent(struct btrfs_trans_handle *trans, struct btrfs_ref *ref) (ref->type == BTRFS_REF_DATA && ref->data_ref.ref_root == BTRFS_TREE_LOG_OBJECTID)) {/* unlocks the pinned mutex */ -btrfs_pin_extent (fs_info, ref->bytenr, ref->len,1 ); +btrfs_pin_extent (trans, ref->bytenr, ref->len,1 ); old_ref_mod = new_ref_mod =0 ; ret =0 ; } else if (ref->type == BTRFS_REF_METADATA) {
Signature · 1 line
2.17.1
Re: [PATCH 02/11] btrfs: Make btrfs_pin_extent take trans handle
Quoted 4 lines · Nikolay Borisov
Preparation for switching pinned extent tracking to a per-transaction basis.
Signed-off-by Nikolay Borisov <[email protected]>
Thanks,
Josef
PATCH 3/11 btrfs: Introduce unaccount_log_buffer
In reply to Nikolay BorisovThis function correctly adjusts the reserved bytes occupied by a log tree extent buffer. It will be used instead of calling btrfs_pin_reserved_extent.
--- fs/btrfs/tree-log.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
fs/btrfs/tree-log.c +23 -0
@@ -18,6 +18,8 @@#include "compression.h" #include "qgroup.h" #include "inode-map.h" +#include "block-group.h" +#include "space-info.h" /* magic values for the inode_only field in btrfs_log_inode: *
@@ -2659,6 +2661,27 @@ static int replay_one_buffer(struct btrfs_root *log, struct extent_buffer *eb,return ret; } +static void unaccount_log_buffer (struct btrfs_fs_info *fs_info,u64 start) +{ +struct btrfs_block_group *cache; + + cache =btrfs_lookup_block_group (fs_info, start); +if (!cache) { +btrfs_err (fs_info,"unable to find block group for %llu" , start); +return ; + } + +spin_lock (&cache->space_info->lock); +spin_lock (&cache->lock); + cache->reserved -= fs_info->nodesize; + cache->space_info->bytes_reserved -= fs_info->nodesize; +spin_unlock (&cache->lock); +spin_unlock (&cache->space_info->lock); + +btrfs_put_block_group (cache); +return ; +} + static noinline int walk_down_log_tree(struct btrfs_trans_handle *trans, struct btrfs_root *root, struct btrfs_path *path, int *level,
Signature · 1 line
2.17.1
Re: [PATCH 03/11] btrfs: Introduce unaccount_log_buffer
Quoted 5 lines · Nikolay Borisov
This function correctly adjusts the reserved bytes occupied by a log tree extent buffer. It will be used instead of calling
btrfs_pin_reserved_extent.Signed-off-by Nikolay Borisov <[email protected]>
Thanks,
Josef
PATCH 4/11 btrfs: Call btrfs_pin_reserved_extent only during active transaction
In reply to Nikolay BorisovCalling btrfs_pin_reserved_extent makes sense only with a valid transaction since pinned extents are processed from transaction commit in btrfs_finish_extent_commit. In case of error it's sufficient to adjust the reserved counter to account for log tree extents allocated in the last transaction.
This commit moves btrfs_pin_reserved_extent to be called only with valid transaction handle and otherwise uses the newly introduced unaccount_log_buffer to adjust "reserved". If this is not done if a failure occurs before transaction is committed WARN_ON are going to be triggered on unmount. This was especially pronounced with generic/475 test.
--- fs/btrfs/tree-log.c | 41 ++++++++++++++++++++--------------------- 1 file changed, 20 insertions(+), 21 deletions(-)
fs/btrfs/tree-log.c +20 -21
@@ -2743,18 +2743,16 @@ static noinline int walk_down_log_tree(struct btrfs_trans_handle *trans,btrfs_clean_tree_block (next);btrfs_wait_tree_block_writeback (next);btrfs_tree_unlock (next); + ret =btrfs_pin_reserved_extent (fs_info, + bytenr, blocksize); +if (ret) { +free_extent_buffer (next); +return ret; + } }else {if (test_and_clear_bit (EXTENT_BUFFER_DIRTY , &next->bflags))clear_extent_buffer_dirty (next); - } - -WARN_ON (root_owner != -BTRFS_TREE_LOG_OBJECTID ); - ret =btrfs_pin_reserved_extent (fs_info, - bytenr, blocksize); -if (ret) { -free_extent_buffer (next); -return ret; +unaccount_log_buffer (fs_info, bytenr); } }free_extent_buffer (next);
@@ -2822,17 +2820,17 @@ static noinline int walk_up_log_tree(struct btrfs_trans_handle *trans,btrfs_clean_tree_block (next);btrfs_wait_tree_block_writeback (next);btrfs_tree_unlock (next); + ret =btrfs_pin_reserved_extent (fs_info, + path->nodes[*level]->start, + path->nodes[*level]->len); +if (ret) +return ret; }else {if (test_and_clear_bit (EXTENT_BUFFER_DIRTY , &next->bflags))clear_extent_buffer_dirty (next); +unaccount_log_buffer (fs_info, + path->nodes[*level]->start); } - -WARN_ON (root_owner !=BTRFS_TREE_LOG_OBJECTID ); - ret =btrfs_pin_reserved_extent (fs_info, - path->nodes[*level]->start, - path->nodes[*level]->len); -if (ret) -return ret; }free_extent_buffer (path->nodes[*level]); path->nodes[*level] =NULL ;
@@ -2903,15 +2901,16 @@ static int walk_log_tree(struct btrfs_trans_handle *trans,btrfs_clean_tree_block (next);btrfs_wait_tree_block_writeback (next);btrfs_tree_unlock (next); + ret =btrfs_pin_reserved_extent (fs_info, next->start, + next->len); +if (ret) + goto out; }else {if (test_and_clear_bit (EXTENT_BUFFER_DIRTY , &next->bflags))clear_extent_buffer_dirty (next); +unaccount_log_buffer (fs_info, + next->start); } - - ret =btrfs_pin_reserved_extent (fs_info, next->start, - next->len); -if (ret) - goto out; } }
Signature · 1 line
2.17.1
Re: [PATCH 04/11] btrfs: Call btrfs_pin_reserved_extent only during active transaction
Quoted 6 lines · Nikolay Borisov
Calling
btrfs_pin_reserved_extentmakes sense only with a valid transaction since pinned extents are processed from transaction commit inbtrfs_finish_extent_commit. In case of error it's sufficient to adjust the reserved counter to account for log tree extents allocated in the last transaction.
Quoted 8 lines
This commit moves
btrfs_pin_reserved_extentto be called only with valid transaction handle and otherwise uses the newly introducedunaccount_log_bufferto adjust "reserved". If this is not done if a failure occurs before transaction is committedWARN_ONare going to be triggered on unmount. This was especially pronounced with generic/475 test.Signed-off-by Nikolay Borisov <[email protected]>
Thanks,
Josef
PATCH 5/11 btrfs: Make btrfs_pin_reserved_extent take transaction
In reply to Nikolay Borisovbtrfs_pin_reserved_extent is now only called with a valid transaction so exploit the fact to take a transaction. This is preparation for tracking pinned extents on a per-transaction basis.
--- fs/btrfs/ctree.h | 2 +- fs/btrfs/extent-tree.c | 8 +++++--- fs/btrfs/tree-log.c | 6 +++--- 3 files changed, 9 insertions(+), 7 deletions(-)
fs/btrfs/ctree.h +1 -1
@@ -2498,7 +2498,7 @@ int btrfs_free_extent(struct btrfs_trans_handle *trans, struct btrfs_ref *ref);int btrfs_free_reserved_extent (struct btrfs_fs_info *fs_info,u64 start,u64 len,int delalloc); -int btrfs_pin_reserved_extent (struct btrfs_fs_info *fs_info,u64 start, +int btrfs_pin_reserved_extent (struct btrfs_trans_handle *trans,u64 start,u64 len);void btrfs_prepare_extent_commit (struct btrfs_fs_info *fs_info);int btrfs_finish_extent_commit (struct btrfs_trans_handle *trans);
fs/btrfs/extent-tree.c +5 -3
@@ -4185,14 +4185,16 @@ int btrfs_free_reserved_extent(struct btrfs_fs_info *fs_info,return 0 ; } -int btrfs_pin_reserved_extent (struct btrfs_fs_info *fs_info,u64 start,u64 len) +int btrfs_pin_reserved_extent (struct btrfs_trans_handle *trans,u64 start, +u64 len) {struct btrfs_block_group *cache;int ret =0 ; - cache =btrfs_lookup_block_group (fs_info, start); + cache =btrfs_lookup_block_group (trans->fs_info, start);if (!cache) { -btrfs_err (fs_info,"unable to find block group for %llu" , start); +btrfs_err (trans->fs_info,"unable to find block group for %llu" , + start);return -ENOSPC ; }
fs/btrfs/tree-log.c +3 -3
@@ -2743,7 +2743,7 @@ static noinline int walk_down_log_tree(struct btrfs_trans_handle *trans,btrfs_clean_tree_block (next);btrfs_wait_tree_block_writeback (next);btrfs_tree_unlock (next); - ret =btrfs_pin_reserved_extent (fs_info, + ret =btrfs_pin_reserved_extent (trans, bytenr, blocksize);if (ret) {free_extent_buffer (next);
@@ -2820,7 +2820,7 @@ static noinline int walk_up_log_tree(struct btrfs_trans_handle *trans,btrfs_clean_tree_block (next);btrfs_wait_tree_block_writeback (next);btrfs_tree_unlock (next); - ret =btrfs_pin_reserved_extent (fs_info, + ret =btrfs_pin_reserved_extent (trans, path->nodes[*level]->start, path->nodes[*level]->len);if (ret)
@@ -2901,7 +2901,7 @@ static int walk_log_tree(struct btrfs_trans_handle *trans,btrfs_clean_tree_block (next);btrfs_wait_tree_block_writeback (next);btrfs_tree_unlock (next); - ret =btrfs_pin_reserved_extent (fs_info, next->start, + ret =btrfs_pin_reserved_extent (trans, next->start, next->len);if (ret) goto out;
Signature · 1 line
2.17.1
Re: [PATCH 05/11] btrfs: Make btrfs_pin_reserved_extent take transaction
Quoted 5 lines · Nikolay Borisov
btrfs_pin_reserved_extentis now only called with a valid transaction so exploit the fact to take a transaction. This is preparation for tracking pinned extents on a per-transaction basis.Signed-off-by Nikolay Borisov <[email protected]>
Thanks,
Josef
PATCH 6/11 btrfs: Make btrfs_pin_extent_for_log_replay take transaction handle
In reply to Nikolay BorisovPreparation for refactoring pinned extents tracking.
--- fs/btrfs/ctree.h | 2 +- fs/btrfs/extent-tree.c | 4 ++-- fs/btrfs/tree-log.c | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-)
fs/btrfs/ctree.h +1 -1
@@ -2462,7 +2462,7 @@ int btrfs_lookup_extent_info(struct btrfs_trans_handle *trans, u64 offset, int metadata, u64 *refs, u64 *flags);int btrfs_pin_extent (struct btrfs_trans_handle *trans,u64 bytenr,u64 num,int reserved); -int btrfs_pin_extent_for_log_replay (struct btrfs_fs_info *fs_info, +int btrfs_pin_extent_for_log_replay (struct btrfs_trans_handle *trans,u64 bytenr,u64 num_bytes);int btrfs_exclude_logged_extents (struct extent_buffer *eb); int btrfs_cross_ref_exist(struct btrfs_root *root,
fs/btrfs/extent-tree.c +2 -2
@@ -2627,13 +2627,13 @@ int btrfs_pin_extent(struct btrfs_trans_handle *trans,/* * this function must be called within transaction */ -int btrfs_pin_extent_for_log_replay (struct btrfs_fs_info *fs_info, +int btrfs_pin_extent_for_log_replay (struct btrfs_trans_handle *trans,u64 bytenr,u64 num_bytes) {struct btrfs_block_group *cache;int ret; - cache =btrfs_lookup_block_group (fs_info, bytenr); + cache =btrfs_lookup_block_group (trans->fs_info, bytenr);if (!cache)return -EINVAL ;
fs/btrfs/tree-log.c +2 -2
@@ -313,7 +313,7 @@ static int process_one_buffer(struct btrfs_root *log, }if (wc->pin) - ret =btrfs_pin_extent_for_log_replay (fs_info, eb->start, + ret =btrfs_pin_extent_for_log_replay (wc->trans, eb->start, eb->len);if (!ret &&btrfs_buffer_uptodate (eb, gen,0 )) {
@@ -6151,7 +6151,7 @@ int btrfs_recover_log_trees(struct btrfs_root *log_root_tree)* each subsequent pass. */ if (ret == -ENOENT ) - ret =btrfs_pin_extent_for_log_replay (fs_info, + ret =btrfs_pin_extent_for_log_replay (trans, log->node->start, log->node->len);free_extent_buffer (log->node);
--
2.17.1
Re: [PATCH 06/11] btrfs: Make btrfs_pin_extent_for_log_replay take transaction handle
Quoted 3 lines · Nikolay Borisov
Preparation for refactoring pinned extents tracking.
Signed-off-by Nikolay Borisov <[email protected]>
Thanks,
Josef
PATCH 7/11 btrfs: Make pin_down_extent take btrfs_trans_handle
In reply to Nikolay BorisovAll callers have a reference to a transaction handle so pass it to pin_down_extent. This is the final step before switching pinned extent tracking to a per-transaction basis.
--- fs/btrfs/extent-tree.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
fs/btrfs/extent-tree.c +6 -5
@@ -2586,7 +2586,8 @@ static u64 first_logical_byte(struct btrfs_fs_info *fs_info, u64 search_start)return bytenr; } -static int pin_down_extent (struct btrfs_block_group *cache, +static int pin_down_extent (struct btrfs_trans_handle *trans, +struct btrfs_block_group *cache,u64 bytenr,u64 num_bytes,int reserved) {struct btrfs_fs_info *fs_info = cache->fs_info;
@@ -2618,7 +2619,7 @@ int btrfs_pin_extent(struct btrfs_trans_handle *trans, cache =btrfs_lookup_block_group (trans->fs_info, bytenr);BUG_ON (!cache);/* Logic error */ -pin_down_extent (cache, bytenr, num_bytes, reserved); +pin_down_extent (trans, cache, bytenr, num_bytes, reserved);btrfs_put_block_group (cache);return 0 ;
@@ -2645,7 +2646,7 @@ int btrfs_pin_extent_for_log_replay(struct btrfs_trans_handle *trans,*/ btrfs_cache_block_group (cache,1 ); -pin_down_extent (cache, bytenr, num_bytes,0 ); +pin_down_extent (trans, cache, bytenr, num_bytes,0 );/* remove us from the free space cache (if we're there at all) */ ret =btrfs_remove_free_space (cache, bytenr, num_bytes);
@@ -3297,7 +3298,7 @@ void btrfs_free_tree_block(struct btrfs_trans_handle *trans, cache =btrfs_lookup_block_group (fs_info, buf->start);if (btrfs_header_flag (buf,BTRFS_HEADER_FLAG_WRITTEN )) { -pin_down_extent (cache, buf->start, buf->len,1 ); +pin_down_extent (trans, cache, buf->start, buf->len,1 );btrfs_put_block_group (cache); goto out; }
@@ -4198,7 +4199,7 @@ int btrfs_pin_reserved_extent(struct btrfs_trans_handle *trans, u64 start,return -ENOSPC ; } - ret =pin_down_extent (cache, start, len,1 ); + ret =pin_down_extent (trans, cache, start, len,1 );btrfs_put_block_group (cache);return ret; }
Signature · 1 line
2.17.1
Re: [PATCH 07/11] btrfs: Make pin_down_extent take btrfs_trans_handle
Quoted 5 lines · Nikolay Borisov
All callers have a reference to a transaction handle so pass it to
pin_down_extent. This is the final step before switching pinned extent tracking to a per-transaction basis.Signed-off-by Nikolay Borisov <[email protected]>
Thanks,
Josef
PATCH 8/11 btrfs: Pass trans handle to write_pinned_extent_entries
In reply to Nikolay BorisovPreparation for refactoring pinned extents tracking.
--- fs/btrfs/free-space-cache.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
fs/btrfs/free-space-cache.c +2 -1
@@ -1067,6 +1067,7 @@ update_cache_item(struct btrfs_trans_handle *trans, } static noinline_for_stack int write_pinned_extent_entries( +struct btrfs_trans_handle *trans,struct btrfs_block_group *block_group,struct btrfs_io_ctl *io_ctl,int *entries)
@@ -1317,7 +1318,7 @@ static int __btrfs_write_out_cache(struct btrfs_root *root, struct inode *inode,* If this changes while we are working we'll get added back to * the dirty list and redo it. No locking needed */ - ret =write_pinned_extent_entries (block_group, io_ctl, &entries); + ret =write_pinned_extent_entries (trans, block_group, io_ctl, &entries);if (ret) goto out_nospc_locked;
--
2.17.1
Re: [PATCH 08/11] btrfs: Pass trans handle to write_pinned_extent_entries
Quoted 3 lines · Nikolay Borisov
Preparation for refactoring pinned extents tracking.
Signed-off-by Nikolay Borisov <[email protected]>
Thanks,
Josef
PATCH 9/11 btrfs: Mark pinned log extents as excluded
In reply to Nikolay BorisovIn preparation to making pinned extents per-transaction ensure that log such extents are always excluded from caching. To achieve this in addition to marking them via btrfs_pin_extent_for_log_replay they also need to be marked with btrfs_add_excluded_extent to prevent log tree extent buffer being loaded by the free space caching thread. That's required since log treeblocks are not recorded in the extent tree, hence they always look free.
--- fs/btrfs/extent-tree.c | 8 ++++++++ fs/btrfs/tree-log.c | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-)
fs/btrfs/extent-tree.c +8 -0
@@ -2634,6 +2634,8 @@ int btrfs_pin_extent_for_log_replay(struct btrfs_trans_handle *trans,struct btrfs_block_group *cache;int ret; +btrfs_add_excluded_extent (trans->fs_info, bytenr, num_bytes); + cache =btrfs_lookup_block_group (trans->fs_info, bytenr);if (!cache)return -EINVAL ;
@@ -2920,6 +2922,12 @@ int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)mutex_unlock (&fs_info->unused_bg_unpin_mutex);break ; } +if (test_bit (BTRFS_FS_LOG_RECOVERING , &fs_info->flags)) { +clear_extent_bits (&fs_info->freed_extents[0 ], start, + end,EXTENT_UPTODATE ); +clear_extent_bits (&fs_info->freed_extents[1 ], start, + end,EXTENT_UPTODATE ); + }if (btrfs_test_opt (fs_info,DISCARD_SYNC )) ret = btrfs_discard_extent(fs_info, start,
fs/btrfs/tree-log.c +1 -1
@@ -2994,7 +2994,7 @@ static inline void btrfs_remove_log_ctx(struct btrfs_root *root,mutex_unlock (&root->log_mutex); } -/* +/* * Invoked in log mutex context, or be sure there is no other task which * can access the list. */
Signature · 1 line
2.17.1
Re: [PATCH 09/11] btrfs: Mark pinned log extents as excluded
Quoted 45 lines · Nikolay Borisov
In preparation to making pinned extents per-transaction ensure that log such extents are always excluded from caching. To achieve this in addition to marking them via
btrfs_pin_extent_for_log_replaythey also need to be marked withbtrfs_add_excluded_extentto prevent log tree extent buffer being loaded by the free space caching thread. That's required since log treeblocks are not recorded in the extent tree, hence they always look free.Signed-off-by Nikolay Borisov <[email protected]>--- fs/btrfs/extent-tree.c | 8 ++++++++ fs/btrfs/tree-log.c | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-)fs/btrfs/extent-tree.c +8
-0@@ -2634,6 +2634,8 @@ int btrfs_pin_extent_for_log_replay(struct btrfs_trans_handle *trans,struct btrfs_block_group *cache;int ret; +btrfs_add_excluded_extent (trans->fs_info, bytenr, num_bytes); + cache =btrfs_lookup_block_group (trans->fs_info, bytenr);if (!cache)return -EINVAL ;@@ -2920,6 +2922,12 @@ int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)mutex_unlock (&fs_info->unused_bg_unpin_mutex);break ; } +if (test_bit (BTRFS_FS_LOG_RECOVERING , &fs_info->flags)) { +clear_extent_bits (&fs_info->freed_extents[0 ], start, + end,EXTENT_UPTODATE ); +clear_extent_bits (&fs_info->freed_extents[1 ], start, + end,EXTENT_UPTODATE ); + }if (btrfs_test_opt (fs_info,DISCARD_SYNC )) ret = btrfs_discard_extent(fs_info, start,fs/btrfs/tree-log.c +0
-0
fs/btrfs/tree-log.c static inline void btrfs_remove_log_ctx(struct btrfs_root *root, · line 2997
Quoted 5 lines
fs/btrfs/tree-log.c +1
-1@@ -2994,7 +2994,7 @@ static inline void btrfs_remove_log_ctx(struct btrfs_root *root,mutex_unlock (&root->log_mutex); } -/* +/*
nit: this part needs to be dropped. Other than that
Thanks,
Josef
Re: [PATCH 09/11] btrfs: Mark pinned log extents as excluded
In reply to Nikolay BorisovQuoted 19 lines · Nikolay Borisov
In preparation to making pinned extents per-transaction ensure that log such extents are always excluded from caching. To achieve this in addition to marking them via
btrfs_pin_extent_for_log_replaythey also need to be marked withbtrfs_add_excluded_extentto prevent log tree extent buffer being loaded by the free space caching thread. That's required since log treeblocks are not recorded in the extent tree, hence they always look free.Signed-off-by Nikolay Borisov <[email protected]>--- fs/btrfs/extent-tree.c | 8 ++++++++ fs/btrfs/tree-log.c | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-)fs/btrfs/extent-tree.c +0
-0
fs/btrfs/extent-tree.c int btrfs_pin_extent_for_log_replay(struct btrfs_trans_handle *trans, · line 2637
Quoted 4 lines
fs/btrfs/extent-tree.c +1
-0@@ -2634,6 +2634,8 @@ int btrfs_pin_extent_for_log_replay(struct btrfs_trans_handle *trans,struct btrfs_block_group *cache;int ret; +btrfs_add_excluded_extent (trans->fs_info, bytenr, num_bytes);
Again lack of error handling, I thought that untangling and cleaning up the extent pinning was motivated to actually handle the errors.
Quoted 32 lines
fs/btrfs/extent-tree.c +7
-0+ cache =btrfs_lookup_block_group (trans->fs_info, bytenr);if (!cache)return -EINVAL ;@@ -2920,6 +2922,12 @@ int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)mutex_unlock (&fs_info->unused_bg_unpin_mutex);break ; } +if (test_bit (BTRFS_FS_LOG_RECOVERING , &fs_info->flags)) { +clear_extent_bits (&fs_info->freed_extents[0 ], start, + end,EXTENT_UPTODATE ); +clear_extent_bits (&fs_info->freed_extents[1 ], start, + end,EXTENT_UPTODATE ); + }if (btrfs_test_opt (fs_info,DISCARD_SYNC )) ret = btrfs_discard_extent(fs_info, start,fs/btrfs/tree-log.c +1
-1@@ -2994,7 +2994,7 @@ static inline void btrfs_remove_log_ctx(struct btrfs_root *root,mutex_unlock (&root->log_mutex); } -/* +/* * Invoked in log mutex context, or be sure there is no other task which * can access the list. */ Signature · 1 line
2.17.1
Re: [PATCH 09/11] btrfs: Mark pinned log extents as excluded
Quoted 23 lines · David Sterba
Quoted 22 lines · Nikolay Borisov
In preparation to making pinned extents per-transaction ensure that log such extents are always excluded from caching. To achieve this in addition to marking them via
btrfs_pin_extent_for_log_replaythey also need to be marked withbtrfs_add_excluded_extentto prevent log tree extent buffer being loaded by the free space caching thread. That's required since log treeblocks are not recorded in the extent tree, hence they always look free.Signed-off-by Nikolay Borisov <[email protected]>--- fs/btrfs/extent-tree.c | 8 ++++++++ fs/btrfs/tree-log.c | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-)fs/btrfs/extent-tree.c +0
-0@@ -2634,6 +2634,8 @@ int btrfs_pin_extent_for_log_replay(struct btrfs_trans_handle *trans,struct btrfs_block_group *cache;int ret;
Quoted 4 lines
fs/btrfs/extent-tree.c int btrfs_pin_extent_for_log_replay(struct btrfs_trans_handle *trans, · line 2637
Quoted 1 line
fs/btrfs/extent-tree.c +1
-0+btrfs_add_excluded_extent (trans->fs_info, bytenr, num_bytes);Again lack of error handling, I thought that untangling and cleaning up the extent pinning was motivated to actually handle the errors.
It is but it will be done in a follow up series. For now I want to have the pinned extents be per-transaction and not be more broken (e.g. no error handling) than we currently are.
Quoted 33 lines
Quoted 33 lines
fs/btrfs/extent-tree.c +7
-0+ cache =btrfs_lookup_block_group (trans->fs_info, bytenr);if (!cache)return -EINVAL ;@@ -2920,6 +2922,12 @@ int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)mutex_unlock (&fs_info->unused_bg_unpin_mutex);break ; } +if (test_bit (BTRFS_FS_LOG_RECOVERING , &fs_info->flags)) { +clear_extent_bits (&fs_info->freed_extents[0 ], start, + end,EXTENT_UPTODATE ); +clear_extent_bits (&fs_info->freed_extents[1 ], start, + end,EXTENT_UPTODATE ); + }if (btrfs_test_opt (fs_info,DISCARD_SYNC )) ret = btrfs_discard_extent(fs_info, start,fs/btrfs/tree-log.c +1
-1@@ -2994,7 +2994,7 @@ static inline void btrfs_remove_log_ctx(struct btrfs_root *root,mutex_unlock (&root->log_mutex); } -/* +/* * Invoked in log mutex context, or be sure there is no other task which * can access the list. */ Signature · 1 line
2.17.1
Re: [PATCH 09/11] btrfs: Mark pinned log extents as excluded
In reply to David SterbaQuoted 23 lines · David Sterba
Quoted 22 lines · Nikolay Borisov
In preparation to making pinned extents per-transaction ensure that log such extents are always excluded from caching. To achieve this in addition to marking them via
btrfs_pin_extent_for_log_replaythey also need to be marked withbtrfs_add_excluded_extentto prevent log tree extent buffer being loaded by the free space caching thread. That's required since log treeblocks are not recorded in the extent tree, hence they always look free.Signed-off-by Nikolay Borisov <[email protected]>--- fs/btrfs/extent-tree.c | 8 ++++++++ fs/btrfs/tree-log.c | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-)fs/btrfs/extent-tree.c +0
-0@@ -2634,6 +2634,8 @@ int btrfs_pin_extent_for_log_replay(struct btrfs_trans_handle *trans,struct btrfs_block_group *cache;int ret;
Quoted 4 lines
fs/btrfs/extent-tree.c int btrfs_pin_extent_for_log_replay(struct btrfs_trans_handle *trans, · line 2637
Quoted 1 line
fs/btrfs/extent-tree.c +1
-0+btrfs_add_excluded_extent (trans->fs_info, bytenr, num_bytes);Again lack of error handling, I thought that untangling and cleaning up the extent pinning was motivated to actually handle the errors.
Currently btrfs_add_excluded_extent always returns 0. The possible err values from set_extent_bit are not propagated. I think this is a cleanup for another series.
Quoted 1 line
<snip>
PATCH 10/11 btrfs: Factor out pinned extent clean up in btrfs_delete_unused_bgs
In reply to Nikolay BorisovNext patch is going to refactor how pinned extents are tracked which will necessitate changing this code. To ease that work and contain the changes factor the code now in preparation, this will also help review.
--- fs/btrfs/block-group.c | 69 ++++++++++++++++++++++++------------------ 1 file changed, 40 insertions(+), 29 deletions(-)
fs/btrfs/block-group.c +40 -29
@@ -1233,6 +1233,45 @@ static int inc_block_group_ro(struct btrfs_block_group *cache, int force)return ret; } +static bool clean_pinned_extents (struct btrfs_block_group *bg) +{ +struct btrfs_fs_info *fs_info = bg->fs_info; +u64 start = bg->start; +u64 end = start + bg->length -1 ; +int ret; + +/* +* Hold the unused_bg_unpin_mutex lock to avoid racing with +* btrfs_finish_extent_commit(). If we are at transaction N, +* another task might be running finish_extent_commit() for the +* previous transaction N - 1, and have seen a range belonging +* to the block group in freed_extents[] before we were able to +* clear the whole block group range from freed_extents[]. This +* means that task can lookup for the block group after we +* unpinned it from freed_extents[] and removed it, leading to +* a BUG_ON() at unpin_extent_range(). +*/ +mutex_lock (&fs_info->unused_bg_unpin_mutex); + ret =clear_extent_bits (&fs_info->freed_extents[0 ], start, end, +EXTENT_DIRTY ); +if (ret) + goto failure; + + ret =clear_extent_bits (&fs_info->freed_extents[1 ], start, end, +EXTENT_DIRTY ); +if (ret) + goto failure; +mutex_unlock (&fs_info->unused_bg_unpin_mutex); + +return true; + +failure: +mutex_unlock (&fs_info->unused_bg_unpin_mutex); +btrfs_dec_block_group_ro (bg); +return false; + +} +/* * Process the unused_bgs list and remove any that don't have any allocated * space inside of them.
@@ -1250,7 +1289,6 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)spin_lock (&fs_info->unused_bgs_lock);while (!list_empty (&fs_info->unused_bgs)) { -u64 start, end;int trimming; block_group = list_first_entry(&fs_info->unused_bgs,
@@ -1329,35 +1367,8 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)* We could have pending pinned extents for this block group, * just delete them, we don't care about them anymore. */ - start = block_group->start; - end = start + block_group->length -1 ; -/* -* Hold the unused_bg_unpin_mutex lock to avoid racing with -* btrfs_finish_extent_commit(). If we are at transaction N, -* another task might be running finish_extent_commit() for the -* previous transaction N - 1, and have seen a range belonging -* to the block group in freed_extents[] before we were able to -* clear the whole block group range from freed_extents[]. This -* means that task can lookup for the block group after we -* unpinned it from freed_extents[] and removed it, leading to -* a BUG_ON() at btrfs_unpin_extent_range(). -*/ -mutex_lock (&fs_info->unused_bg_unpin_mutex); - ret =clear_extent_bits (&fs_info->freed_extents[0 ], start, end, -EXTENT_DIRTY ); -if (ret) { -mutex_unlock (&fs_info->unused_bg_unpin_mutex); -btrfs_dec_block_group_ro (block_group); +if (!clean_pinned_extents (block_group)) goto end_trans; - } - ret =clear_extent_bits (&fs_info->freed_extents[1 ], start, end, -EXTENT_DIRTY ); -if (ret) { -mutex_unlock (&fs_info->unused_bg_unpin_mutex); -btrfs_dec_block_group_ro (block_group); - goto end_trans; - } -mutex_unlock (&fs_info->unused_bg_unpin_mutex);/* * At this point, the block_group is read only and should fail
--
2.17.1
Re: [PATCH 10/11] btrfs: Factor out pinned extent clean up in btrfs_delete_unused_bgs
Quoted 5 lines · Nikolay Borisov
Next patch is going to refactor how pinned extents are tracked which will necessitate changing this code. To ease that work and contain the changes factor the code now in preparation, this will also help review.
Signed-off-by Nikolay Borisov <[email protected]>
Thanks,
Josef
PATCH 11/11 btrfs: Use btrfs_transaction::pinned_extents
In reply to Nikolay BorisovThis commit flips the switch to start tracking/processing pinned extents on a per-transaction basis. It mostly replaces all references from btrfs_fs_info::(pinned_extents|freed_extents[]) to btrfs_transaction::pinned_extents. Two notable modifications that warrant explicit mention are changing clean_pinned_extents to get a reference to the previously running transaction. The other one is removal of call to btrfs_destroy_pinned_extent since transactions are going to be cleaned in btrfs_cleanup_one_transaction.
--- fs/btrfs/block-group.c | 38 ++++++++++++++++++++++++------------ fs/btrfs/ctree.h | 4 ++-- fs/btrfs/disk-io.c | 30 +++++----------------------- fs/btrfs/extent-io-tree.h | 3 +-- fs/btrfs/extent-tree.c | 31 ++++++++--------------------- fs/btrfs/free-space-cache.c | 2 +- fs/btrfs/tests/btrfs-tests.c | 7 ++----- fs/btrfs/transaction.c | 1 + fs/btrfs/transaction.h | 1 + include/trace/events/btrfs.h | 3 +-- 10 files changed, 47 insertions(+), 73 deletions(-)
fs/btrfs/block-group.c +25 -13
@@ -460,7 +460,7 @@ u64 add_new_free_space(struct btrfs_block_group *block_group, u64 start, u64 endint ret;while (start < end) { - ret =find_first_extent_bit (info->pinned_extents, start, + ret =find_first_extent_bit (&info->excluded_extents, start, &extent_start, &extent_end,EXTENT_DIRTY |EXTENT_UPTODATE ,NULL );
@@ -1233,32 +1233,44 @@ static int inc_block_group_ro(struct btrfs_block_group *cache, int force)return ret; } -static bool clean_pinned_extents (struct btrfs_block_group *bg) +static bool clean_pinned_extents (struct btrfs_trans_handle *trans, +struct btrfs_block_group *bg) {struct btrfs_fs_info *fs_info = bg->fs_info; +struct btrfs_transaction *prev_trans =NULL ;u64 start = bg->start;u64 end = start + bg->length -1 ;int ret; +spin_lock (&fs_info->trans_lock); +if (trans->transaction->list.prev != &fs_info->trans_list) { + prev_trans =list_entry (trans->transaction->list.prev, + struct btrfs_transaction, list); +refcount_inc (&prev_trans->use_count); + } +spin_unlock (&fs_info->trans_lock); +/* * Hold the unused_bg_unpin_mutex lock to avoid racing with * btrfs_finish_extent_commit(). If we are at transaction N, * another task might be running finish_extent_commit() for the * previous transaction N - 1, and have seen a range belonging -* to the block group in freed_extents[] before we were able to -* clear the whole block group range from freed_extents[]. This +* to the block group in pinned_extents before we were able to +* clear the whole block group range from pinned_extents. This * means that task can lookup for the block group after we -* unpinned it from freed_extents[] and removed it, leading to +* unpinned it from pinned_extents[] and removed it, leading to * a BUG_ON() at unpin_extent_range(). */ mutex_lock (&fs_info->unused_bg_unpin_mutex); - ret =clear_extent_bits (&fs_info->freed_extents[0 ], start, end, -EXTENT_DIRTY ); -if (ret) - goto failure; +if (prev_trans) { + ret =clear_extent_bits (&prev_trans->pinned_extents, start, end, +EXTENT_DIRTY ); +if (ret) + goto failure; + } - ret =clear_extent_bits (&fs_info->freed_extents[1 ], start, end, -EXTENT_DIRTY ); + ret =clear_extent_bits (&trans->transaction->pinned_extents, start, end, +EXTENT_DIRTY );if (ret) goto failure;mutex_unlock (&fs_info->unused_bg_unpin_mutex);
@@ -1367,7 +1379,7 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)* We could have pending pinned extents for this block group, * just delete them, we don't care about them anymore. */ -if (!clean_pinned_extents (block_group)) +if (!clean_pinned_extents (trans, block_group)) goto end_trans;/*
@@ -2877,7 +2889,7 @@ int btrfs_update_block_group(struct btrfs_trans_handle *trans, &cache->space_info->total_bytes_pinned, num_bytes, BTRFS_TOTAL_BYTES_PINNED_BATCH); -set_extent_dirty (info->pinned_extents, +set_extent_dirty (&trans->transaction->pinned_extents, bytenr, bytenr + num_bytes -1 ,GFP_NOFS | __GFP_NOFAIL); }
fs/btrfs/ctree.h +2 -2
@@ -596,8 +596,8 @@ struct btrfs_fs_info {/* keep track of unallocated space */ atomic64_t free_chunk_space; -struct extent_io_tree freed_extents[2 ]; -struct extent_io_tree *pinned_extents; +/* Tracks ranges which used by log trees blocks/logged data extents */ +struct extent_io_tree excluded_extents;/* logical->physical extent mapping */ struct extent_map_tree mapping_tree;
fs/btrfs/disk-io.c +5 -25
@@ -2021,10 +2021,8 @@ void btrfs_free_fs_roots(struct btrfs_fs_info *fs_info)btrfs_drop_and_free_fs_root (fs_info, gang[i]); } -if (test_bit (BTRFS_FS_STATE_ERROR , &fs_info->fs_state)) { +if (test_bit (BTRFS_FS_STATE_ERROR , &fs_info->fs_state))btrfs_free_log_root_tree (NULL , fs_info); -btrfs_destroy_pinned_extent (fs_info, fs_info->pinned_extents); - } }static void btrfs_init_scrub (struct btrfs_fs_info *fs_info)
@@ -2779,11 +2777,8 @@ int __cold open_ctree(struct super_block *sb, fs_info->block_group_cache_tree =RB_ROOT ; fs_info->first_logical_byte = (u64)-1 ; -extent_io_tree_init (fs_info, &fs_info->freed_extents[0 ], -IO_TREE_FS_INFO_FREED_EXTENTS0 ,NULL ); -extent_io_tree_init (fs_info, &fs_info->freed_extents[1 ], -IO_TREE_FS_INFO_FREED_EXTENTS1 ,NULL ); - fs_info->pinned_extents = &fs_info->freed_extents[0 ]; +extent_io_tree_init (fs_info, &fs_info->excluded_extents, +IO_TREE_FS_INFO_EXCLUDED_EXTENTS ,NULL );set_bit (BTRFS_FS_BARRIER , &fs_info->flags);mutex_init (&fs_info->ordered_operations_mutex);
@@ -4384,16 +4379,12 @@ static int btrfs_destroy_marked_extents(struct btrfs_fs_info *fs_info, }static int btrfs_destroy_pinned_extent (struct btrfs_fs_info *fs_info, -struct extent_io_tree *pinned_extents) +struct extent_io_tree *unpin) { -struct extent_io_tree *unpin;u64 start;u64 end;int ret; -bool loop = true; - unpin = pinned_extents; -again:while (1 ) {struct extent_state *cached_state =NULL ;
@@ -4418,15 +4409,6 @@ static int btrfs_destroy_pinned_extent(struct btrfs_fs_info *fs_info,cond_resched (); } -if (loop) { -if (unpin == &fs_info->freed_extents[0 ]) - unpin = &fs_info->freed_extents[1 ]; -else - unpin = &fs_info->freed_extents[0 ]; - loop = false; - goto again; - } -return 0 ; }
@@ -4518,8 +4500,7 @@ void btrfs_cleanup_one_transaction(struct btrfs_transaction *cur_trans,btrfs_destroy_marked_extents (fs_info, &cur_trans->dirty_pages,EXTENT_DIRTY ); -btrfs_destroy_pinned_extent (fs_info, - fs_info->pinned_extents); +btrfs_destroy_pinned_extent (fs_info, &cur_trans->pinned_extents); cur_trans->state =TRANS_STATE_COMPLETED ;wake_up (&cur_trans->commit_wait);
@@ -4571,7 +4552,6 @@ static int btrfs_cleanup_transaction(struct btrfs_fs_info *fs_info)btrfs_destroy_all_ordered_extents (fs_info);btrfs_destroy_delayed_inodes (fs_info);btrfs_assert_delayed_root_empty (fs_info); -btrfs_destroy_pinned_extent (fs_info, fs_info->pinned_extents);btrfs_destroy_all_delalloc_inodes (fs_info);mutex_unlock (&fs_info->transaction_kthread_mutex);
fs/btrfs/extent-io-tree.h +1 -2
@@ -36,8 +36,7 @@ struct io_failure_record;#define CHUNK_TRIMMED EXTENT_DEFRAGenum { -IO_TREE_FS_INFO_FREED_EXTENTS0 , -IO_TREE_FS_INFO_FREED_EXTENTS1 , +IO_TREE_FS_INFO_EXCLUDED_EXTENTS ,IO_TREE_INODE_IO ,IO_TREE_INODE_IO_FAILURE ,IO_TREE_RELOC_BLOCKS ,
fs/btrfs/extent-tree.c +8 -23
@@ -64,10 +64,8 @@ int btrfs_add_excluded_extent(struct btrfs_fs_info *fs_info, u64 start, u64 num_bytes) {u64 end = start + num_bytes -1 ; -set_extent_bits (&fs_info->freed_extents[0 ], - start, end,EXTENT_UPTODATE ); -set_extent_bits (&fs_info->freed_extents[1 ], - start, end,EXTENT_UPTODATE ); +set_extent_bits (&fs_info->excluded_extents, start, end, +EXTENT_UPTODATE );return 0 ; }
@@ -79,10 +77,8 @@ void btrfs_free_excluded_extents(struct btrfs_block_group *cache) start = cache->start; end = start + cache->length -1 ; -clear_extent_bits (&fs_info->freed_extents[0 ], - start, end,EXTENT_UPTODATE ); -clear_extent_bits (&fs_info->freed_extents[1 ], - start, end,EXTENT_UPTODATE ); +clear_extent_bits (&fs_info->excluded_extents, start, end, +EXTENT_UPTODATE ); }static u64 generic_ref_to_space_flags (struct btrfs_ref *ref)
@@ -2606,7 +2602,7 @@ static int pin_down_extent(struct btrfs_trans_handle *trans,percpu_counter_add_batch (&cache->space_info->total_bytes_pinned, num_bytes,BTRFS_TOTAL_BYTES_PINNED_BATCH ); -set_extent_dirty (fs_info->pinned_extents, bytenr, +set_extent_dirty (&trans->transaction->pinned_extents, bytenr, bytenr + num_bytes -1 ,GFP_NOFS | __GFP_NOFAIL);return 0 ; }
@@ -2762,11 +2758,6 @@ void btrfs_prepare_extent_commit(struct btrfs_fs_info *fs_info) } } -if (fs_info->pinned_extents == &fs_info->freed_extents[0 ]) - fs_info->pinned_extents = &fs_info->freed_extents[1 ]; -else - fs_info->pinned_extents = &fs_info->freed_extents[0 ]; -up_write (&fs_info->commit_root_sem);btrfs_update_global_block_rsv (fs_info);
@@ -2907,10 +2898,7 @@ int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)u64 end;int ret; -if (fs_info->pinned_extents == &fs_info->freed_extents[0 ]) - unpin = &fs_info->freed_extents[1 ]; -else - unpin = &fs_info->freed_extents[0 ]; + unpin = &trans->transaction->pinned_extents;while (!trans->aborted) {struct extent_state *cached_state =NULL ;
@@ -2922,12 +2910,9 @@ int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)mutex_unlock (&fs_info->unused_bg_unpin_mutex);break ; } -if (test_bit (BTRFS_FS_LOG_RECOVERING , &fs_info->flags)) { -clear_extent_bits (&fs_info->freed_extents[0 ], start, +if (test_bit (BTRFS_FS_LOG_RECOVERING , &fs_info->flags)) +clear_extent_bits (&fs_info->excluded_extents, start, end,EXTENT_UPTODATE ); -clear_extent_bits (&fs_info->freed_extents[1 ], start, - end,EXTENT_UPTODATE ); - }if (btrfs_test_opt (fs_info,DISCARD_SYNC )) ret = btrfs_discard_extent(fs_info, start,
fs/btrfs/free-space-cache.c +1 -1
@@ -1086,7 +1086,7 @@ static noinline_for_stack int write_pinned_extent_entries(* We shouldn't have switched the pinned extents yet so this is the * right one */ - unpin = block_group->fs_info->pinned_extents; + unpin = &trans->transaction->pinned_extents; start = block_group->start;
fs/btrfs/tests/btrfs-tests.c +2 -5
@@ -156,12 +156,9 @@ struct btrfs_fs_info *btrfs_alloc_dummy_fs_info(u32 nodesize, u32 sectorsize)INIT_LIST_HEAD (&fs_info->fs_devices->devices);INIT_RADIX_TREE (&fs_info->buffer_radix,GFP_ATOMIC );INIT_RADIX_TREE (&fs_info->fs_roots_radix,GFP_ATOMIC ); -extent_io_tree_init (fs_info, &fs_info->freed_extents[0 ], -IO_TREE_FS_INFO_FREED_EXTENTS0 ,NULL ); -extent_io_tree_init (fs_info, &fs_info->freed_extents[1 ], -IO_TREE_FS_INFO_FREED_EXTENTS1 ,NULL ); +extent_io_tree_init (fs_info, &fs_info->excluded_extents, +IO_TREE_FS_INFO_EXCLUDED_EXTENTS ,NULL );extent_map_tree_init (&fs_info->mapping_tree); - fs_info->pinned_extents = &fs_info->freed_extents[0 ];set_bit (BTRFS_FS_STATE_DUMMY_FS_INFO , &fs_info->fs_state); test_mnt->mnt_sb->s_fs_info = fs_info;
fs/btrfs/transaction.c +1 -0
@@ -334,6 +334,7 @@ static noinline int join_transaction(struct btrfs_fs_info *fs_info,list_add_tail (&cur_trans->list, &fs_info->trans_list);extent_io_tree_init (fs_info, &cur_trans->dirty_pages,IO_TREE_TRANS_DIRTY_PAGES , fs_info->btree_inode); +extent_io_tree_init (fs_info, &cur_trans->pinned_extents,0 ,NULL ); fs_info->generation++; cur_trans->transid = fs_info->generation; fs_info->running_transaction = cur_trans;
fs/btrfs/transaction.h +1 -0
@@ -71,6 +71,7 @@ struct btrfs_transaction {*/ struct list_head io_bgs;struct list_head dropped_roots; +struct extent_io_tree pinned_extents;/* * we need to make sure block group deletion doesn't race with
include/trace/events/btrfs.h +1 -2
@@ -81,8 +81,7 @@ TRACE_DEFINE_ENUM(COMMIT_TRANS);#define show_extent_io_tree_owner (owner) \ __print_symbolic(owner, \ - { IO_TREE_FS_INFO_FREED_EXTENTS0, "FREED_EXTENTS0" }, \ - { IO_TREE_FS_INFO_FREED_EXTENTS1, "FREED_EXTENTS1" }, \ + { IO_TREE_FS_INFO_EXCLUDED_EXTENTS, "EXCLUDED_EXTENTS" }, \ { IO_TREE_INODE_IO, "INODE_IO" }, \ { IO_TREE_INODE_IO_FAILURE, "INODE_IO_FAILURE" }, \ { IO_TREE_RELOC_BLOCKS, "RELOC_BLOCKS" }, \
Signature · 1 line
2.17.1
Re: [PATCH 11/11] btrfs: Use btrfs_transaction::pinned_extents
Quoted 10 lines · Nikolay Borisov
This commit flips the switch to start tracking/processing pinned extents on a per-transaction basis. It mostly replaces all references from btrfs_fs_info::(pinned_extents|freed_extents[]) to
btrfs_transaction::pinned_extents. Two notable modifications that warrant explicit mention are changingclean_pinned_extentsto get a reference to the previously running transaction. The other one is removal of call tobtrfs_destroy_pinned_extentsince transactions are going to be cleaned inbtrfs_cleanup_one_transaction.Signed-off-by Nikolay Borisov <[email protected]>
I'd prefer if the excluded extent changes were separate from the pinned extent changes.
Quoted 18 lines
--- fs/btrfs/block-group.c | 38 ++++++++++++++++++++++++------------ fs/btrfs/ctree.h | 4 ++-- fs/btrfs/disk-io.c | 30 +++++----------------------- fs/btrfs/extent-io-tree.h | 3 +-- fs/btrfs/extent-tree.c | 31 ++++++++--------------------- fs/btrfs/free-space-cache.c | 2 +- fs/btrfs/tests/btrfs-tests.c | 7 ++----- fs/btrfs/transaction.c | 1 + fs/btrfs/transaction.h | 1 + include/trace/events/btrfs.h | 3 +-- 10 files changed, 47 insertions(+), 73 deletions(-)fs/btrfs/block-group.c +0
-0
fs/btrfs/block-group.c u64 add_new_free_space(struct btrfs_block_group *block_group, u64 start, u64 end · line 466
Quoted 8 lines
fs/btrfs/block-group.c +1
-1@@ -460,7 +460,7 @@ u64 add_new_free_space(struct btrfs_block_group *block_group, u64 start, u64 endint ret;while (start < end) { - ret =find_first_extent_bit (info->pinned_extents, start, + ret =find_first_extent_bit (&info->excluded_extents, start, &extent_start, &extent_end,EXTENT_DIRTY |EXTENT_UPTODATE ,NULL );
We're no longer doing EXTENT_DIRTY in excluded_extents, so we don't need this part.
Quoted 37 lines
fs/btrfs/block-group.c +14
-4@@ -1233,32 +1233,44 @@ static int inc_block_group_ro(struct btrfs_block_group *cache, int force)return ret; } -static bool clean_pinned_extents (struct btrfs_block_group *bg) +static bool clean_pinned_extents (struct btrfs_trans_handle *trans, +struct btrfs_block_group *bg) {struct btrfs_fs_info *fs_info = bg->fs_info; +struct btrfs_transaction *prev_trans =NULL ;u64 start = bg->start;u64 end = start + bg->length -1 ;int ret; +spin_lock (&fs_info->trans_lock); +if (trans->transaction->list.prev != &fs_info->trans_list) { + prev_trans =list_entry (trans->transaction->list.prev, + struct btrfs_transaction, list); +refcount_inc (&prev_trans->use_count); + } +spin_unlock (&fs_info->trans_lock); +/* * Hold the unused_bg_unpin_mutex lock to avoid racing with * btrfs_finish_extent_commit(). If we are at transaction N, * another task might be running finish_extent_commit() for the * previous transaction N - 1, and have seen a range belonging -* to the block group in freed_extents[] before we were able to -* clear the whole block group range from freed_extents[]. This +* to the block group in pinned_extents before we were able to +* clear the whole block group range from pinned_extents. This * means that task can lookup for the block group after we -* unpinned it from freed_extents[] and removed it, leading to +* unpinned it from pinned_extents[] and removed it, leading to * a BUG_ON() at unpin_extent_range(). */ mutex_lock (&fs_info->unused_bg_unpin_mutex);
fs/btrfs/block-group.c static int inc_block_group_ro(struct btrfs_block_group *cache, int force) · line 1270
Quoted 10 lines
fs/btrfs/block-group.c +6
-4- ret =clear_extent_bits (&fs_info->freed_extents[0 ], start, end, -EXTENT_DIRTY ); -if (ret) - goto failure; +if (prev_trans) { + ret =clear_extent_bits (&prev_trans->pinned_extents, start, end, +EXTENT_DIRTY ); +if (ret) + goto failure; + }
You are leaking a ref to prev_trans here.
<snip>
fs/btrfs/disk-io.c void btrfs_free_fs_roots(struct btrfs_fs_info *fs_info) · line 2027
Quoted 13 lines
fs/btrfs/disk-io.c +1
-3@@ -2021,10 +2021,8 @@ void btrfs_free_fs_roots(struct btrfs_fs_info *fs_info)btrfs_drop_and_free_fs_root (fs_info, gang[i]); } -if (test_bit (BTRFS_FS_STATE_ERROR , &fs_info->fs_state)) { +if (test_bit (BTRFS_FS_STATE_ERROR , &fs_info->fs_state))btrfs_free_log_root_tree (NULL , fs_info); -btrfs_destroy_pinned_extent (fs_info, fs_info->pinned_extents); - }
What about the excluded extents? We may never cache the block group with one of the super mirrors in it, and thus we would leak the excluded extent for it. Thanks,
Josef
Re: [PATCH 11/11] btrfs: Use btrfs_transaction::pinned_extents
Quoted 104 lines · Josef Bacik
Quoted 10 lines · Nikolay Borisov
This commit flips the switch to start tracking/processing pinned extents on a per-transaction basis. It mostly replaces all references from btrfs_fs_info::(pinned_extents|freed_extents[]) to
btrfs_transaction::pinned_extents. Two notable modifications that warrant explicit mention are changingclean_pinned_extentsto get a reference to the previously running transaction. The other one is removal of call tobtrfs_destroy_pinned_extentsince transactions are going to be cleaned inbtrfs_cleanup_one_transaction.Signed-off-by Nikolay Borisov <[email protected]>I'd prefer if the excluded extent changes were separate from the pinned extent changes.
Quoted 18 lines
---
fs/btrfs/block-group.c | 38 ++++++++++++++++++++++++------------ fs/btrfs/ctree.h | 4 ++--
fs/btrfs/disk-io.c | 30 +++++----------------------- fs/btrfs/extent-io-tree.h | 3 +-- fs/btrfs/extent-tree.c | 31 ++++++++--------------------- fs/btrfs/free-space-cache.c | 2 +-fs/btrfs/tests/btrfs-tests.c| 7 ++----- fs/btrfs/transaction.c | 1 +
fs/btrfs/transaction.h | 1 +include/trace/events/btrfs.h| 3 +--10 files changed, 47 insertions(+), 73 deletions(-)fs/btrfs/block-group.c +0
-0fs/btrfs/block-group.c u64 add_new_free_space(struct btrfs_block_group · line 466
Quoted 8 lines
fs/btrfs/block-group.c +1
-1@@ -460,7 +460,7 @@ u64 add_new_free_space(struct btrfs_block_group *block_group, u64 start, u64 end int ret; while (start < end) { - ret = find_first_extent_bit(info->pinned_extents, start, + ret = find_first_extent_bit(&info->excluded_extents, start, &extent_start, &extent_end, EXTENT_DIRTY | EXTENT_UPTODATE, NULL);We're no longer doing
EXTENT_DIRTYinexcluded_extents, so we don't need this part.Quoted 36 lines
fs/btrfs/block-group.c +13
-3@@ -1233,32 +1233,44 @@ static int inc_block_group_ro(struct btrfs_block_group *cache, int force) return ret; } -static bool clean_pinned_extents(struct btrfs_block_group *bg) +static bool clean_pinned_extents(struct btrfs_trans_handle *trans, + struct btrfs_block_group *bg) { struct btrfs_fs_info *fs_info = bg->fs_info; + struct btrfs_transaction *prev_trans = NULL; u64 start = bg->start; u64 end = start + bg->length - 1; int ret; + spin_lock(&fs_info->trans_lock); + if (trans->transaction->list.prev != &fs_info->trans_list) { + prev_trans = list_entry(trans->transaction->list.prev, + struct btrfs_transaction, list); + refcount_inc(&prev_trans->use_count); + } + spin_unlock(&fs_info->trans_lock); + /* * Hold the unused_bg_unpin_mutex lock to avoid racing with * btrfs_finish_extent_commit(). If we are at transaction N, * another task might be running finish_extent_commit() for the * previous transaction N - 1, and have seen a range belonging - * to the block group in freed_extents[] before we were able to - * clear the whole block group range from freed_extents[]. This + * to the block group in pinned_extents before we were able to + * clear the whole block group range from pinned_extents. This * means that task can lookup for the block group after we - * unpinned it from freed_extents[] and removed it, leading to + * unpinned it from pinned_extents[] and removed it, leading to * a BUG_ON() at unpin_extent_range(). */ mutex_lock(&fs_info->unused_bg_unpin_mutex);fs/btrfs/block-group.c static int inc_block_group_ro(struct · line 1270
Quoted 10 lines
fs/btrfs/block-group.c +6
-4- ret = clear_extent_bits(&fs_info->freed_extents[0], start, end, - EXTENT_DIRTY); - if (ret) - goto failure; + if (prev_trans) { + ret = clear_extent_bits(&prev_trans->pinned_extents, start, end, + EXTENT_DIRTY); + if (ret) + goto failure; + }You are leaking a ref to
prev_transhere.<snip>
Quoted 9 lines
fs/btrfs/disk-io.c +0
-0@@ -2021,10 +2021,8 @@ void btrfs_free_fs_roots(struct btrfs_fs_info *fs_info) btrfs_drop_and_free_fs_root(fs_info, gang[i]); } - if (test_bit(BTRFS_FS_STATE_ERROR, &fs_info->fs_state)) {
Quoted 8 lines
fs/btrfs/disk-io.c void btrfs_free_fs_roots(struct btrfs_fs_info *fs_info) · line 2027
Quoted 4 lines
fs/btrfs/disk-io.c +1
-2+ if (test_bit(BTRFS_FS_STATE_ERROR, &fs_info->fs_state)) btrfs_free_log_root_tree(NULL, fs_info); - btrfs_destroy_pinned_extent(fs_info, fs_info->pinned_extents); - }What about the excluded extents? We may never cache the block group with one of the super mirrors in it, and thus we would leak the excluded extent for it. Thanks,
btrfs_destroy_pinned_extent didn't touch EXTENT_UPDATE (excluded extents) before so my removing this call doesn't change that. E.g. if there is a bug here where excluded extents are not cleaned up then it's not due to my code.
On the other hand I don't quite understand your concern w.r.t pinned extents. Can you elaborate?
Quoted 2 lines
Josef
Re: [PATCH 11/11] btrfs: Use btrfs_transaction::pinned_extents
Quoted 116 lines · Nikolay Borisov
Quoted 104 lines · Josef Bacik
Quoted 10 lines · Nikolay Borisov
This commit flips the switch to start tracking/processing pinned extents on a per-transaction basis. It mostly replaces all references from btrfs_fs_info::(pinned_extents|freed_extents[]) to
btrfs_transaction::pinned_extents. Two notable modifications that warrant explicit mention are changingclean_pinned_extentsto get a reference to the previously running transaction. The other one is removal of call tobtrfs_destroy_pinned_extentsince transactions are going to be cleaned inbtrfs_cleanup_one_transaction.Signed-off-by Nikolay Borisov <[email protected]>I'd prefer if the excluded extent changes were separate from the pinned extent changes.
Quoted 18 lines
--- fs/btrfs/block-group.c | 38 ++++++++++++++++++++++++------------ fs/btrfs/ctree.h | 4 ++-- fs/btrfs/disk-io.c | 30 +++++----------------------- fs/btrfs/extent-io-tree.h | 3 +-- fs/btrfs/extent-tree.c | 31 ++++++++--------------------- fs/btrfs/free-space-cache.c | 2 +- fs/btrfs/tests/btrfs-tests.c | 7 ++----- fs/btrfs/transaction.c | 1 + fs/btrfs/transaction.h | 1 + include/trace/events/btrfs.h | 3 +-- 10 files changed, 47 insertions(+), 73 deletions(-)fs/btrfs/block-group.c +0
-0fs/btrfs/block-group.c u64 add_new_free_space(struct btrfs_block_group · line 466
Quoted 8 lines
fs/btrfs/block-group.c +1
-1@@ -460,7 +460,7 @@ u64 add_new_free_space(struct btrfs_block_group *block_group, u64 start, u64 end int ret; while (start < end) { - ret = find_first_extent_bit(info->pinned_extents, start, + ret = find_first_extent_bit(&info->excluded_extents, start, &extent_start, &extent_end, EXTENT_DIRTY | EXTENT_UPTODATE, NULL);We're no longer doing
EXTENT_DIRTYinexcluded_extents, so we don't need this part.Quoted 36 lines
fs/btrfs/block-group.c +13
-3@@ -1233,32 +1233,44 @@ static int inc_block_group_ro(struct btrfs_block_group *cache, int force) return ret; } -static bool clean_pinned_extents(struct btrfs_block_group *bg) +static bool clean_pinned_extents(struct btrfs_trans_handle *trans, + struct btrfs_block_group *bg) { struct btrfs_fs_info *fs_info = bg->fs_info; + struct btrfs_transaction *prev_trans = NULL; u64 start = bg->start; u64 end = start + bg->length - 1; int ret; + spin_lock(&fs_info->trans_lock); + if (trans->transaction->list.prev != &fs_info->trans_list) { + prev_trans = list_entry(trans->transaction->list.prev, + struct btrfs_transaction, list); + refcount_inc(&prev_trans->use_count); + } + spin_unlock(&fs_info->trans_lock); + /* * Hold the unused_bg_unpin_mutex lock to avoid racing with * btrfs_finish_extent_commit(). If we are at transaction N, * another task might be running finish_extent_commit() for the * previous transaction N - 1, and have seen a range belonging - * to the block group in freed_extents[] before we were able to - * clear the whole block group range from freed_extents[]. This + * to the block group in pinned_extents before we were able to + * clear the whole block group range from pinned_extents. This * means that task can lookup for the block group after we - * unpinned it from freed_extents[] and removed it, leading to + * unpinned it from pinned_extents[] and removed it, leading to * a BUG_ON() at unpin_extent_range(). */ mutex_lock(&fs_info->unused_bg_unpin_mutex);fs/btrfs/block-group.c static int inc_block_group_ro(struct · line 1270
Quoted 10 lines
fs/btrfs/block-group.c +6
-4- ret = clear_extent_bits(&fs_info->freed_extents[0], start, end, - EXTENT_DIRTY); - if (ret) - goto failure; + if (prev_trans) { + ret = clear_extent_bits(&prev_trans->pinned_extents, start, end, + EXTENT_DIRTY); + if (ret) + goto failure; + }You are leaking a ref to
prev_transhere.<snip>
Quoted 9 lines
fs/btrfs/disk-io.c +0
-0@@ -2021,10 +2021,8 @@ void btrfs_free_fs_roots(struct btrfs_fs_info *fs_info) btrfs_drop_and_free_fs_root(fs_info, gang[i]); } - if (test_bit(BTRFS_FS_STATE_ERROR, &fs_info->fs_state)) {Quoted 8 lines
fs/btrfs/disk-io.c void btrfs_free_fs_roots(struct btrfs_fs_info *fs_info) · line 2027
Quoted 4 lines
fs/btrfs/disk-io.c +1
-2+ if (test_bit(BTRFS_FS_STATE_ERROR, &fs_info->fs_state)) btrfs_free_log_root_tree(NULL, fs_info); - btrfs_destroy_pinned_extent(fs_info, fs_info->pinned_extents); - }What about the excluded extents? We may never cache the block group with one of the super mirrors in it, and thus we would leak the excluded extent for it. Thanks,
Quoted 8 lines
btrfs_destroy_pinned_extentdidn't touchEXTENT_UPDATE(excluded extents) before so my removing this call doesn't change that. E.g. if there is a bug here where excluded extents are not cleaned up then it's not due to my code.On the other hand I don't quite understand your concern w.r.t pinned extents. Can you elaborate?
Sorry thunderbird ate my followup, we drop the excluded extents in btrfs_free_block_groups() if they are never cached, so you are fine here. Thanks,
Josef
PATCH v2 1/1 btrfs: Use btrfs_transaction::pinned_extents
In reply to Nikolay BorisovThis commit flips the switch to start tracking/processing pinned extents on a per-transaction basis. It mostly replaces all references from btrfs_fs_info::(pinned_extents|freed_extents[]) to btrfs_transaction::pinned_extents. Two notable modifications that warrant explicit mention are changing clean_pinned_extents to get a reference to the previously running transaction. The other one is removal of call to btrfs_destroy_pinned_extent since transactions are going to be cleaned in btrfs_cleanup_one_transaction.
---
V2:
* Properly release transaction in clean_pinned_extents [Josef]
* Only check for EXTENT_UPTODATE in add_new_free_space [Josef]
fs/btrfs/block-group.c | 43 +++++++++++++++++++++++------------- fs/btrfs/ctree.h | 4 ++-- fs/btrfs/disk-io.c | 30 +++++-------------------- fs/btrfs/extent-io-tree.h | 3 +-- fs/btrfs/extent-tree.c | 31 +++++++------------------- fs/btrfs/free-space-cache.c | 2 +- fs/btrfs/tests/btrfs-tests.c | 7 ++---- fs/btrfs/transaction.c | 1 + fs/btrfs/transaction.h | 1 + include/trace/events/btrfs.h | 3 +-- 10 files changed, 50 insertions(+), 75 deletions(-)
fs/btrfs/block-group.c +28 -15
@@ -460,10 +460,9 @@ u64 add_new_free_space(struct btrfs_block_group *block_group, u64 start, u64 endint ret;while (start < end) { - ret =find_first_extent_bit (info->pinned_extents, start, + ret =find_first_extent_bit (&info->excluded_extents, start, &extent_start, &extent_end, -EXTENT_DIRTY |EXTENT_UPTODATE , -NULL ); +EXTENT_UPTODATE ,NULL );if (ret)break ;
@@ -1233,32 +1232,46 @@ static int inc_block_group_ro(struct btrfs_block_group *cache, int force)return ret; } -static bool clean_pinned_extents (struct btrfs_block_group *bg) +static bool clean_pinned_extents (struct btrfs_trans_handle *trans, +struct btrfs_block_group *bg) {struct btrfs_fs_info *fs_info = bg->fs_info; +struct btrfs_transaction *prev_trans =NULL ;u64 start = bg->start;u64 end = start + bg->length -1 ;int ret; +spin_lock (&fs_info->trans_lock); +if (trans->transaction->list.prev != &fs_info->trans_list) { + prev_trans =list_entry (trans->transaction->list.prev, + struct btrfs_transaction, list); +refcount_inc (&prev_trans->use_count); + } +spin_unlock (&fs_info->trans_lock); +/* * Hold the unused_bg_unpin_mutex lock to avoid racing with * btrfs_finish_extent_commit(). If we are at transaction N, * another task might be running finish_extent_commit() for the * previous transaction N - 1, and have seen a range belonging -* to the block group in freed_extents[] before we were able to -* clear the whole block group range from freed_extents[]. This +* to the block group in pinned_extents before we were able to +* clear the whole block group range from pinned_extents. This * means that task can lookup for the block group after we -* unpinned it from freed_extents[] and removed it, leading to +* unpinned it from pinned_extents[] and removed it, leading to * a BUG_ON() at unpin_extent_range(). */ mutex_lock (&fs_info->unused_bg_unpin_mutex); - ret =clear_extent_bits (&fs_info->freed_extents[0 ], start, end, -EXTENT_DIRTY ); -if (ret) - goto failure; +if (prev_trans) { + ret =clear_extent_bits (&prev_trans->pinned_extents, start, end, +EXTENT_DIRTY ); +if (ret) + goto failure; + } + +btrfs_put_transaction (prev_trans); - ret =clear_extent_bits (&fs_info->freed_extents[1 ], start, end, -EXTENT_DIRTY ); + ret =clear_extent_bits (&trans->transaction->pinned_extents, start, end, +EXTENT_DIRTY );if (ret) goto failure;mutex_unlock (&fs_info->unused_bg_unpin_mutex);
@@ -1367,7 +1380,7 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)* We could have pending pinned extents for this block group, * just delete them, we don't care about them anymore. */ -if (!clean_pinned_extents (block_group)) +if (!clean_pinned_extents (trans, block_group)) goto end_trans;/*
@@ -2877,7 +2890,7 @@ int btrfs_update_block_group(struct btrfs_trans_handle *trans, &cache->space_info->total_bytes_pinned, num_bytes, BTRFS_TOTAL_BYTES_PINNED_BATCH); -set_extent_dirty (info->pinned_extents, +set_extent_dirty (&trans->transaction->pinned_extents, bytenr, bytenr + num_bytes -1 ,GFP_NOFS | __GFP_NOFAIL); }
fs/btrfs/ctree.h +2 -2
@@ -596,8 +596,8 @@ struct btrfs_fs_info {/* keep track of unallocated space */ atomic64_t free_chunk_space; -struct extent_io_tree freed_extents[2 ]; -struct extent_io_tree *pinned_extents; +/* Tracks ranges which used by log trees blocks/logged data extents */ +struct extent_io_tree excluded_extents;/* logical->physical extent mapping */ struct extent_map_tree mapping_tree;
fs/btrfs/disk-io.c +5 -25
@@ -2021,10 +2021,8 @@ void btrfs_free_fs_roots(struct btrfs_fs_info *fs_info)btrfs_drop_and_free_fs_root (fs_info, gang[i]); } -if (test_bit (BTRFS_FS_STATE_ERROR , &fs_info->fs_state)) { +if (test_bit (BTRFS_FS_STATE_ERROR , &fs_info->fs_state))btrfs_free_log_root_tree (NULL , fs_info); -btrfs_destroy_pinned_extent (fs_info, fs_info->pinned_extents); - } }static void btrfs_init_scrub (struct btrfs_fs_info *fs_info)
@@ -2779,11 +2777,8 @@ int __cold open_ctree(struct super_block *sb, fs_info->block_group_cache_tree =RB_ROOT ; fs_info->first_logical_byte = (u64)-1 ; -extent_io_tree_init (fs_info, &fs_info->freed_extents[0 ], -IO_TREE_FS_INFO_FREED_EXTENTS0 ,NULL ); -extent_io_tree_init (fs_info, &fs_info->freed_extents[1 ], -IO_TREE_FS_INFO_FREED_EXTENTS1 ,NULL ); - fs_info->pinned_extents = &fs_info->freed_extents[0 ]; +extent_io_tree_init (fs_info, &fs_info->excluded_extents, +IO_TREE_FS_INFO_EXCLUDED_EXTENTS ,NULL );set_bit (BTRFS_FS_BARRIER , &fs_info->flags);mutex_init (&fs_info->ordered_operations_mutex);
@@ -4384,16 +4379,12 @@ static int btrfs_destroy_marked_extents(struct btrfs_fs_info *fs_info, }static int btrfs_destroy_pinned_extent (struct btrfs_fs_info *fs_info, -struct extent_io_tree *pinned_extents) +struct extent_io_tree *unpin) { -struct extent_io_tree *unpin;u64 start;u64 end;int ret; -bool loop = true; - unpin = pinned_extents; -again:while (1 ) {struct extent_state *cached_state =NULL ;
@@ -4418,15 +4409,6 @@ static int btrfs_destroy_pinned_extent(struct btrfs_fs_info *fs_info,cond_resched (); } -if (loop) { -if (unpin == &fs_info->freed_extents[0 ]) - unpin = &fs_info->freed_extents[1 ]; -else - unpin = &fs_info->freed_extents[0 ]; - loop = false; - goto again; - } -return 0 ; }
@@ -4518,8 +4500,7 @@ void btrfs_cleanup_one_transaction(struct btrfs_transaction *cur_trans,btrfs_destroy_marked_extents (fs_info, &cur_trans->dirty_pages,EXTENT_DIRTY ); -btrfs_destroy_pinned_extent (fs_info, - fs_info->pinned_extents); +btrfs_destroy_pinned_extent (fs_info, &cur_trans->pinned_extents); cur_trans->state =TRANS_STATE_COMPLETED ;wake_up (&cur_trans->commit_wait);
@@ -4571,7 +4552,6 @@ static int btrfs_cleanup_transaction(struct btrfs_fs_info *fs_info)btrfs_destroy_all_ordered_extents (fs_info);btrfs_destroy_delayed_inodes (fs_info);btrfs_assert_delayed_root_empty (fs_info); -btrfs_destroy_pinned_extent (fs_info, fs_info->pinned_extents);btrfs_destroy_all_delalloc_inodes (fs_info);mutex_unlock (&fs_info->transaction_kthread_mutex);
fs/btrfs/extent-io-tree.h +1 -2
@@ -36,8 +36,7 @@ struct io_failure_record;#define CHUNK_TRIMMED EXTENT_DEFRAGenum { -IO_TREE_FS_INFO_FREED_EXTENTS0 , -IO_TREE_FS_INFO_FREED_EXTENTS1 , +IO_TREE_FS_INFO_EXCLUDED_EXTENTS ,IO_TREE_INODE_IO ,IO_TREE_INODE_IO_FAILURE ,IO_TREE_RELOC_BLOCKS ,
fs/btrfs/extent-tree.c +8 -23
@@ -64,10 +64,8 @@ int btrfs_add_excluded_extent(struct btrfs_fs_info *fs_info, u64 start, u64 num_bytes) {u64 end = start + num_bytes -1 ; -set_extent_bits (&fs_info->freed_extents[0 ], - start, end,EXTENT_UPTODATE ); -set_extent_bits (&fs_info->freed_extents[1 ], - start, end,EXTENT_UPTODATE ); +set_extent_bits (&fs_info->excluded_extents, start, end, +EXTENT_UPTODATE );return 0 ; }
@@ -79,10 +77,8 @@ void btrfs_free_excluded_extents(struct btrfs_block_group *cache) start = cache->start; end = start + cache->length -1 ; -clear_extent_bits (&fs_info->freed_extents[0 ], - start, end,EXTENT_UPTODATE ); -clear_extent_bits (&fs_info->freed_extents[1 ], - start, end,EXTENT_UPTODATE ); +clear_extent_bits (&fs_info->excluded_extents, start, end, +EXTENT_UPTODATE ); }static u64 generic_ref_to_space_flags (struct btrfs_ref *ref)
@@ -2606,7 +2602,7 @@ static int pin_down_extent(struct btrfs_trans_handle *trans,percpu_counter_add_batch (&cache->space_info->total_bytes_pinned, num_bytes,BTRFS_TOTAL_BYTES_PINNED_BATCH ); -set_extent_dirty (fs_info->pinned_extents, bytenr, +set_extent_dirty (&trans->transaction->pinned_extents, bytenr, bytenr + num_bytes -1 ,GFP_NOFS | __GFP_NOFAIL);return 0 ; }
@@ -2762,11 +2758,6 @@ void btrfs_prepare_extent_commit(struct btrfs_fs_info *fs_info) } } -if (fs_info->pinned_extents == &fs_info->freed_extents[0 ]) - fs_info->pinned_extents = &fs_info->freed_extents[1 ]; -else - fs_info->pinned_extents = &fs_info->freed_extents[0 ]; -up_write (&fs_info->commit_root_sem);btrfs_update_global_block_rsv (fs_info);
@@ -2907,10 +2898,7 @@ int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)u64 end;int ret; -if (fs_info->pinned_extents == &fs_info->freed_extents[0 ]) - unpin = &fs_info->freed_extents[1 ]; -else - unpin = &fs_info->freed_extents[0 ]; + unpin = &trans->transaction->pinned_extents;while (!trans->aborted) {struct extent_state *cached_state =NULL ;
@@ -2922,12 +2910,9 @@ int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)mutex_unlock (&fs_info->unused_bg_unpin_mutex);break ; } -if (test_bit (BTRFS_FS_LOG_RECOVERING , &fs_info->flags)) { -clear_extent_bits (&fs_info->freed_extents[0 ], start, +if (test_bit (BTRFS_FS_LOG_RECOVERING , &fs_info->flags)) +clear_extent_bits (&fs_info->excluded_extents, start, end,EXTENT_UPTODATE ); -clear_extent_bits (&fs_info->freed_extents[1 ], start, - end,EXTENT_UPTODATE ); - }if (btrfs_test_opt (fs_info,DISCARD_SYNC )) ret = btrfs_discard_extent(fs_info, start,
fs/btrfs/free-space-cache.c +1 -1
@@ -1086,7 +1086,7 @@ static noinline_for_stack int write_pinned_extent_entries(* We shouldn't have switched the pinned extents yet so this is the * right one */ - unpin = block_group->fs_info->pinned_extents; + unpin = &trans->transaction->pinned_extents; start = block_group->start;
fs/btrfs/tests/btrfs-tests.c +2 -5
@@ -156,12 +156,9 @@ struct btrfs_fs_info *btrfs_alloc_dummy_fs_info(u32 nodesize, u32 sectorsize)INIT_LIST_HEAD (&fs_info->fs_devices->devices);INIT_RADIX_TREE (&fs_info->buffer_radix,GFP_ATOMIC );INIT_RADIX_TREE (&fs_info->fs_roots_radix,GFP_ATOMIC ); -extent_io_tree_init (fs_info, &fs_info->freed_extents[0 ], -IO_TREE_FS_INFO_FREED_EXTENTS0 ,NULL ); -extent_io_tree_init (fs_info, &fs_info->freed_extents[1 ], -IO_TREE_FS_INFO_FREED_EXTENTS1 ,NULL ); +extent_io_tree_init (fs_info, &fs_info->excluded_extents, +IO_TREE_FS_INFO_EXCLUDED_EXTENTS ,NULL );extent_map_tree_init (&fs_info->mapping_tree); - fs_info->pinned_extents = &fs_info->freed_extents[0 ];set_bit (BTRFS_FS_STATE_DUMMY_FS_INFO , &fs_info->fs_state); test_mnt->mnt_sb->s_fs_info = fs_info;
fs/btrfs/transaction.c +1 -0
@@ -334,6 +334,7 @@ static noinline int join_transaction(struct btrfs_fs_info *fs_info,list_add_tail (&cur_trans->list, &fs_info->trans_list);extent_io_tree_init (fs_info, &cur_trans->dirty_pages,IO_TREE_TRANS_DIRTY_PAGES , fs_info->btree_inode); +extent_io_tree_init (fs_info, &cur_trans->pinned_extents,0 ,NULL ); fs_info->generation++; cur_trans->transid = fs_info->generation; fs_info->running_transaction = cur_trans;
fs/btrfs/transaction.h +1 -0
@@ -71,6 +71,7 @@ struct btrfs_transaction {*/ struct list_head io_bgs;struct list_head dropped_roots; +struct extent_io_tree pinned_extents;/* * we need to make sure block group deletion doesn't race with
include/trace/events/btrfs.h +1 -2
@@ -81,8 +81,7 @@ TRACE_DEFINE_ENUM(COMMIT_TRANS);#define show_extent_io_tree_owner (owner) \ __print_symbolic(owner, \ - { IO_TREE_FS_INFO_FREED_EXTENTS0, "FREED_EXTENTS0" }, \ - { IO_TREE_FS_INFO_FREED_EXTENTS1, "FREED_EXTENTS1" }, \ + { IO_TREE_FS_INFO_EXCLUDED_EXTENTS, "EXCLUDED_EXTENTS" }, \ { IO_TREE_INODE_IO, "INODE_IO" }, \ { IO_TREE_INODE_IO_FAILURE, "INODE_IO_FAILURE" }, \ { IO_TREE_RELOC_BLOCKS, "RELOC_BLOCKS" }, \
--
2.17.1
Re: [PATCH v2] btrfs: Use btrfs_transaction::pinned_extents
Quoted 84 lines · Nikolay Borisov
This commit flips the switch to start tracking/processing pinned extents on a per-transaction basis. It mostly replaces all references from btrfs_fs_info::(pinned_extents|freed_extents[]) to
btrfs_transaction::pinned_extents. Two notable modifications that warrant explicit mention are changingclean_pinned_extentsto get a reference to the previously running transaction. The other one is removal of call tobtrfs_destroy_pinned_extentsince transactions are going to be cleaned inbtrfs_cleanup_one_transaction.Signed-off-by Nikolay Borisov <[email protected]>---
V2:
* Properly release transaction inclean_pinned_extents[Josef]
* Only check forEXTENT_UPTODATEinadd_new_free_space[Josef]fs/btrfs/block-group.c | 43 +++++++++++++++++++++++------------- fs/btrfs/ctree.h | 4 ++-- fs/btrfs/disk-io.c | 30 +++++-------------------- fs/btrfs/extent-io-tree.h | 3 +-- fs/btrfs/extent-tree.c | 31 +++++++------------------- fs/btrfs/free-space-cache.c | 2 +- fs/btrfs/tests/btrfs-tests.c | 7 ++---- fs/btrfs/transaction.c | 1 + fs/btrfs/transaction.h | 1 + include/trace/events/btrfs.h | 3 +-- 10 files changed, 50 insertions(+), 75 deletions(-)fs/btrfs/block-group.c +16
-9@@ -460,10 +460,9 @@ u64 add_new_free_space(struct btrfs_block_group *block_group, u64 start, u64 endint ret;while (start < end) { - ret =find_first_extent_bit (info->pinned_extents, start, + ret =find_first_extent_bit (&info->excluded_extents, start, &extent_start, &extent_end, -EXTENT_DIRTY |EXTENT_UPTODATE , -NULL ); +EXTENT_UPTODATE ,NULL );if (ret)break ;@@ -1233,32 +1232,46 @@ static int inc_block_group_ro(struct btrfs_block_group *cache, int force)return ret; } -static bool clean_pinned_extents (struct btrfs_block_group *bg) +static bool clean_pinned_extents (struct btrfs_trans_handle *trans, +struct btrfs_block_group *bg) {struct btrfs_fs_info *fs_info = bg->fs_info; +struct btrfs_transaction *prev_trans =NULL ;u64 start = bg->start;u64 end = start + bg->length -1 ;int ret; +spin_lock (&fs_info->trans_lock); +if (trans->transaction->list.prev != &fs_info->trans_list) { + prev_trans =list_entry (trans->transaction->list.prev, + struct btrfs_transaction, list); +refcount_inc (&prev_trans->use_count); + } +spin_unlock (&fs_info->trans_lock); +/* * Hold the unused_bg_unpin_mutex lock to avoid racing with * btrfs_finish_extent_commit(). If we are at transaction N, * another task might be running finish_extent_commit() for the * previous transaction N - 1, and have seen a range belonging -* to the block group in freed_extents[] before we were able to -* clear the whole block group range from freed_extents[]. This +* to the block group in pinned_extents before we were able to +* clear the whole block group range from pinned_extents. This * means that task can lookup for the block group after we -* unpinned it from freed_extents[] and removed it, leading to +* unpinned it from pinned_extents[] and removed it, leading to * a BUG_ON() at unpin_extent_range(). */ mutex_lock (&fs_info->unused_bg_unpin_mutex); - ret =clear_extent_bits (&fs_info->freed_extents[0 ], start, end, -EXTENT_DIRTY );
fs/btrfs/block-group.c static int inc_block_group_ro(struct btrfs_block_group *cache, int force) · line 1271
Quoted 10 lines
fs/btrfs/block-group.c +8
-2-if (ret) - goto failure; +if (prev_trans) { + ret =clear_extent_bits (&prev_trans->pinned_extents, start, end, +EXTENT_DIRTY ); +if (ret) + goto failure; + } + +btrfs_put_transaction (prev_trans);
The failure case is missing the put. Thanks,
Josef
PATCH v3 1/1 btrfs: Use btrfs_transaction::pinned_extents
In reply to Nikolay BorisovThis commit flips the switch to start tracking/processing pinned extents on a per-transaction basis. It mostly replaces all references from btrfs_fs_info::(pinned_extents|freed_extents[]) to btrfs_transaction::pinned_extents. Two notable modifications that warrant explicit mention are changing clean_pinned_extents to get a reference to the previously running transaction. The other one is removal of call to btrfs_destroy_pinned_extent since transactions are going to be cleaned in btrfs_cleanup_one_transaction.
---
V3:
* Added missing btrfs_put_transaction under failure label in clean_pinned_extents [Josef]
V2:
* Properly release transaction in clean_pinned_extents [Josef]
* Only check for EXTENT_UPTODATE in add_new_free_space [Josef]
fs/btrfs/block-group.c | 44 ++++++++++++++++++++++++------------ fs/btrfs/ctree.h | 4 ++-- fs/btrfs/disk-io.c | 30 ++++-------------------- fs/btrfs/extent-io-tree.h | 3 +-- fs/btrfs/extent-tree.c | 31 +++++++------------------ fs/btrfs/free-space-cache.c | 2 +- fs/btrfs/tests/btrfs-tests.c | 7 ++---- fs/btrfs/transaction.c | 1 + fs/btrfs/transaction.h | 1 + include/trace/events/btrfs.h | 3 +-- 10 files changed, 51 insertions(+), 75 deletions(-)
fs/btrfs/block-group.c +29 -15
@@ -460,10 +460,9 @@ u64 add_new_free_space(struct btrfs_block_group *block_group, u64 start, u64 endint ret;while (start < end) { - ret =find_first_extent_bit (info->pinned_extents, start, + ret =find_first_extent_bit (&info->excluded_extents, start, &extent_start, &extent_end, -EXTENT_DIRTY |EXTENT_UPTODATE , -NULL ); +EXTENT_UPTODATE ,NULL );if (ret)break ;
@@ -1233,32 +1232,46 @@ static int inc_block_group_ro(struct btrfs_block_group *cache, int force)return ret; } -static bool clean_pinned_extents (struct btrfs_block_group *bg) +static bool clean_pinned_extents (struct btrfs_trans_handle *trans, +struct btrfs_block_group *bg) {struct btrfs_fs_info *fs_info = bg->fs_info; +struct btrfs_transaction *prev_trans =NULL ;u64 start = bg->start;u64 end = start + bg->length -1 ;int ret; +spin_lock (&fs_info->trans_lock); +if (trans->transaction->list.prev != &fs_info->trans_list) { + prev_trans =list_entry (trans->transaction->list.prev, + struct btrfs_transaction, list); +refcount_inc (&prev_trans->use_count); + } +spin_unlock (&fs_info->trans_lock); +/* * Hold the unused_bg_unpin_mutex lock to avoid racing with * btrfs_finish_extent_commit(). If we are at transaction N, * another task might be running finish_extent_commit() for the * previous transaction N - 1, and have seen a range belonging -* to the block group in freed_extents[] before we were able to -* clear the whole block group range from freed_extents[]. This +* to the block group in pinned_extents before we were able to +* clear the whole block group range from pinned_extents. This * means that task can lookup for the block group after we -* unpinned it from freed_extents[] and removed it, leading to +* unpinned it from pinned_extents[] and removed it, leading to * a BUG_ON() at unpin_extent_range(). */ mutex_lock (&fs_info->unused_bg_unpin_mutex); - ret =clear_extent_bits (&fs_info->freed_extents[0 ], start, end, -EXTENT_DIRTY ); -if (ret) - goto failure; +if (prev_trans) { + ret =clear_extent_bits (&prev_trans->pinned_extents, start, end, +EXTENT_DIRTY ); +if (ret) + goto failure; + } + +btrfs_put_transaction (prev_trans); - ret =clear_extent_bits (&fs_info->freed_extents[1 ], start, end, -EXTENT_DIRTY ); + ret =clear_extent_bits (&trans->transaction->pinned_extents, start, end, +EXTENT_DIRTY );if (ret) goto failure;mutex_unlock (&fs_info->unused_bg_unpin_mutex);
@@ -1266,6 +1279,7 @@ static bool clean_pinned_extents(struct btrfs_block_group *bg)return true; failure: +btrfs_put_transaction (prev_trans);mutex_unlock (&fs_info->unused_bg_unpin_mutex);btrfs_dec_block_group_ro (bg);return false;
@@ -1367,7 +1381,7 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)* We could have pending pinned extents for this block group, * just delete them, we don't care about them anymore. */ -if (!clean_pinned_extents (block_group)) +if (!clean_pinned_extents (trans, block_group)) goto end_trans;/*
@@ -2877,7 +2891,7 @@ int btrfs_update_block_group(struct btrfs_trans_handle *trans, &cache->space_info->total_bytes_pinned, num_bytes, BTRFS_TOTAL_BYTES_PINNED_BATCH); -set_extent_dirty (info->pinned_extents, +set_extent_dirty (&trans->transaction->pinned_extents, bytenr, bytenr + num_bytes -1 ,GFP_NOFS | __GFP_NOFAIL); }
fs/btrfs/ctree.h +2 -2
@@ -596,8 +596,8 @@ struct btrfs_fs_info {/* keep track of unallocated space */ atomic64_t free_chunk_space; -struct extent_io_tree freed_extents[2 ]; -struct extent_io_tree *pinned_extents; +/* Tracks ranges which used by log trees blocks/logged data extents */ +struct extent_io_tree excluded_extents;/* logical->physical extent mapping */ struct extent_map_tree mapping_tree;
fs/btrfs/disk-io.c +5 -25
@@ -2021,10 +2021,8 @@ void btrfs_free_fs_roots(struct btrfs_fs_info *fs_info)btrfs_drop_and_free_fs_root (fs_info, gang[i]); } -if (test_bit (BTRFS_FS_STATE_ERROR , &fs_info->fs_state)) { +if (test_bit (BTRFS_FS_STATE_ERROR , &fs_info->fs_state))btrfs_free_log_root_tree (NULL , fs_info); -btrfs_destroy_pinned_extent (fs_info, fs_info->pinned_extents); - } }static void btrfs_init_scrub (struct btrfs_fs_info *fs_info)
@@ -2779,11 +2777,8 @@ int __cold open_ctree(struct super_block *sb, fs_info->block_group_cache_tree =RB_ROOT ; fs_info->first_logical_byte = (u64)-1 ; -extent_io_tree_init (fs_info, &fs_info->freed_extents[0 ], -IO_TREE_FS_INFO_FREED_EXTENTS0 ,NULL ); -extent_io_tree_init (fs_info, &fs_info->freed_extents[1 ], -IO_TREE_FS_INFO_FREED_EXTENTS1 ,NULL ); - fs_info->pinned_extents = &fs_info->freed_extents[0 ]; +extent_io_tree_init (fs_info, &fs_info->excluded_extents, +IO_TREE_FS_INFO_EXCLUDED_EXTENTS ,NULL );set_bit (BTRFS_FS_BARRIER , &fs_info->flags);mutex_init (&fs_info->ordered_operations_mutex);
@@ -4391,16 +4386,12 @@ static int btrfs_destroy_marked_extents(struct btrfs_fs_info *fs_info, }static int btrfs_destroy_pinned_extent (struct btrfs_fs_info *fs_info, -struct extent_io_tree *pinned_extents) +struct extent_io_tree *unpin) { -struct extent_io_tree *unpin;u64 start;u64 end;int ret; -bool loop = true; - unpin = pinned_extents; -again:while (1 ) {struct extent_state *cached_state =NULL ;
@@ -4425,15 +4416,6 @@ static int btrfs_destroy_pinned_extent(struct btrfs_fs_info *fs_info,cond_resched (); } -if (loop) { -if (unpin == &fs_info->freed_extents[0 ]) - unpin = &fs_info->freed_extents[1 ]; -else - unpin = &fs_info->freed_extents[0 ]; - loop = false; - goto again; - } -return 0 ; }
@@ -4525,8 +4507,7 @@ void btrfs_cleanup_one_transaction(struct btrfs_transaction *cur_trans,btrfs_destroy_marked_extents (fs_info, &cur_trans->dirty_pages,EXTENT_DIRTY ); -btrfs_destroy_pinned_extent (fs_info, - fs_info->pinned_extents); +btrfs_destroy_pinned_extent (fs_info, &cur_trans->pinned_extents); cur_trans->state =TRANS_STATE_COMPLETED ;wake_up (&cur_trans->commit_wait);
@@ -4578,7 +4559,6 @@ static int btrfs_cleanup_transaction(struct btrfs_fs_info *fs_info)btrfs_destroy_all_ordered_extents (fs_info);btrfs_destroy_delayed_inodes (fs_info);btrfs_assert_delayed_root_empty (fs_info); -btrfs_destroy_pinned_extent (fs_info, fs_info->pinned_extents);btrfs_destroy_all_delalloc_inodes (fs_info);mutex_unlock (&fs_info->transaction_kthread_mutex);
fs/btrfs/extent-io-tree.h +1 -2
@@ -36,8 +36,7 @@ struct io_failure_record;#define CHUNK_TRIMMED EXTENT_DEFRAGenum { -IO_TREE_FS_INFO_FREED_EXTENTS0 , -IO_TREE_FS_INFO_FREED_EXTENTS1 , +IO_TREE_FS_INFO_EXCLUDED_EXTENTS ,IO_TREE_INODE_IO ,IO_TREE_INODE_IO_FAILURE ,IO_TREE_RELOC_BLOCKS ,
fs/btrfs/extent-tree.c +8 -23
@@ -64,10 +64,8 @@ int btrfs_add_excluded_extent(struct btrfs_fs_info *fs_info, u64 start, u64 num_bytes) {u64 end = start + num_bytes -1 ; -set_extent_bits (&fs_info->freed_extents[0 ], - start, end,EXTENT_UPTODATE ); -set_extent_bits (&fs_info->freed_extents[1 ], - start, end,EXTENT_UPTODATE ); +set_extent_bits (&fs_info->excluded_extents, start, end, +EXTENT_UPTODATE );return 0 ; }
@@ -79,10 +77,8 @@ void btrfs_free_excluded_extents(struct btrfs_block_group *cache) start = cache->start; end = start + cache->length -1 ; -clear_extent_bits (&fs_info->freed_extents[0 ], - start, end,EXTENT_UPTODATE ); -clear_extent_bits (&fs_info->freed_extents[1 ], - start, end,EXTENT_UPTODATE ); +clear_extent_bits (&fs_info->excluded_extents, start, end, +EXTENT_UPTODATE ); }static u64 generic_ref_to_space_flags (struct btrfs_ref *ref)
@@ -2606,7 +2602,7 @@ static int pin_down_extent(struct btrfs_trans_handle *trans,percpu_counter_add_batch (&cache->space_info->total_bytes_pinned, num_bytes,BTRFS_TOTAL_BYTES_PINNED_BATCH ); -set_extent_dirty (fs_info->pinned_extents, bytenr, +set_extent_dirty (&trans->transaction->pinned_extents, bytenr, bytenr + num_bytes -1 ,GFP_NOFS | __GFP_NOFAIL);return 0 ; }
@@ -2762,11 +2758,6 @@ void btrfs_prepare_extent_commit(struct btrfs_fs_info *fs_info) } } -if (fs_info->pinned_extents == &fs_info->freed_extents[0 ]) - fs_info->pinned_extents = &fs_info->freed_extents[1 ]; -else - fs_info->pinned_extents = &fs_info->freed_extents[0 ]; -up_write (&fs_info->commit_root_sem);btrfs_update_global_block_rsv (fs_info);
@@ -2907,10 +2898,7 @@ int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)u64 end;int ret; -if (fs_info->pinned_extents == &fs_info->freed_extents[0 ]) - unpin = &fs_info->freed_extents[1 ]; -else - unpin = &fs_info->freed_extents[0 ]; + unpin = &trans->transaction->pinned_extents;while (!trans->aborted) {struct extent_state *cached_state =NULL ;
@@ -2922,12 +2910,9 @@ int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)mutex_unlock (&fs_info->unused_bg_unpin_mutex);break ; } -if (test_bit (BTRFS_FS_LOG_RECOVERING , &fs_info->flags)) { -clear_extent_bits (&fs_info->freed_extents[0 ], start, +if (test_bit (BTRFS_FS_LOG_RECOVERING , &fs_info->flags)) +clear_extent_bits (&fs_info->excluded_extents, start, end,EXTENT_UPTODATE ); -clear_extent_bits (&fs_info->freed_extents[1 ], start, - end,EXTENT_UPTODATE ); - }if (btrfs_test_opt (fs_info,DISCARD_SYNC )) ret = btrfs_discard_extent(fs_info, start,
fs/btrfs/free-space-cache.c +1 -1
@@ -1086,7 +1086,7 @@ static noinline_for_stack int write_pinned_extent_entries(* We shouldn't have switched the pinned extents yet so this is the * right one */ - unpin = block_group->fs_info->pinned_extents; + unpin = &trans->transaction->pinned_extents; start = block_group->start;
fs/btrfs/tests/btrfs-tests.c +2 -5
@@ -156,12 +156,9 @@ struct btrfs_fs_info *btrfs_alloc_dummy_fs_info(u32 nodesize, u32 sectorsize)INIT_LIST_HEAD (&fs_info->fs_devices->devices);INIT_RADIX_TREE (&fs_info->buffer_radix,GFP_ATOMIC );INIT_RADIX_TREE (&fs_info->fs_roots_radix,GFP_ATOMIC ); -extent_io_tree_init (fs_info, &fs_info->freed_extents[0 ], -IO_TREE_FS_INFO_FREED_EXTENTS0 ,NULL ); -extent_io_tree_init (fs_info, &fs_info->freed_extents[1 ], -IO_TREE_FS_INFO_FREED_EXTENTS1 ,NULL ); +extent_io_tree_init (fs_info, &fs_info->excluded_extents, +IO_TREE_FS_INFO_EXCLUDED_EXTENTS ,NULL );extent_map_tree_init (&fs_info->mapping_tree); - fs_info->pinned_extents = &fs_info->freed_extents[0 ];set_bit (BTRFS_FS_STATE_DUMMY_FS_INFO , &fs_info->fs_state); test_mnt->mnt_sb->s_fs_info = fs_info;
fs/btrfs/transaction.c +1 -0
@@ -334,6 +334,7 @@ static noinline int join_transaction(struct btrfs_fs_info *fs_info,list_add_tail (&cur_trans->list, &fs_info->trans_list);extent_io_tree_init (fs_info, &cur_trans->dirty_pages,IO_TREE_TRANS_DIRTY_PAGES , fs_info->btree_inode); +extent_io_tree_init (fs_info, &cur_trans->pinned_extents,0 ,NULL ); fs_info->generation++; cur_trans->transid = fs_info->generation; fs_info->running_transaction = cur_trans;
fs/btrfs/transaction.h +1 -0
@@ -71,6 +71,7 @@ struct btrfs_transaction {*/ struct list_head io_bgs;struct list_head dropped_roots; +struct extent_io_tree pinned_extents;/* * we need to make sure block group deletion doesn't race with
include/trace/events/btrfs.h +1 -2
@@ -81,8 +81,7 @@ TRACE_DEFINE_ENUM(COMMIT_TRANS);#define show_extent_io_tree_owner (owner) \ __print_symbolic(owner, \ - { IO_TREE_FS_INFO_FREED_EXTENTS0, "FREED_EXTENTS0" }, \ - { IO_TREE_FS_INFO_FREED_EXTENTS1, "FREED_EXTENTS1" }, \ + { IO_TREE_FS_INFO_EXCLUDED_EXTENTS, "EXCLUDED_EXTENTS" }, \ { IO_TREE_INODE_IO, "INODE_IO" }, \ { IO_TREE_INODE_IO_FAILURE, "INODE_IO_FAILURE" }, \ { IO_TREE_RELOC_BLOCKS, "RELOC_BLOCKS" }, \
--
2.17.1
Re: [PATCH v3] btrfs: Use btrfs_transaction::pinned_extents
Quoted 10 lines · Nikolay Borisov
This commit flips the switch to start tracking/processing pinned extents on a per-transaction basis. It mostly replaces all references from btrfs_fs_info::(pinned_extents|freed_extents[]) to
btrfs_transaction::pinned_extents. Two notable modifications that warrant explicit mention are changingclean_pinned_extentsto get a reference to the previously running transaction. The other one is removal of call tobtrfs_destroy_pinned_extentsince transactions are going to be cleaned inbtrfs_cleanup_one_transaction.Signed-off-by Nikolay Borisov <[email protected]>
Thanks,
Josef
Re: [PATCH v3] btrfs: Use btrfs_transaction::pinned_extents
In reply to Nikolay BorisovQuoted 10 lines · Nikolay Borisov
This commit flips the switch to start tracking/processing pinned extents on a per-transaction basis. It mostly replaces all references from btrfs_fs_info::(pinned_extents|freed_extents[]) to
btrfs_transaction::pinned_extents. Two notable modifications that warrant explicit mention are changingclean_pinned_extentsto get a reference to the previously running transaction. The other one is removal of call tobtrfs_destroy_pinned_extentsince transactions are going to be cleaned inbtrfs_cleanup_one_transaction.Signed-off-by Nikolay Borisov <[email protected]>
The subject could tell me what's the summmary of the change, I can see in the code that btrfs_transaction::pinned_extents is used but for what? If this is the main change of the patchset after cleanups, then the subject is close to the cover letter, so something like "switch to per-transaction pinned extents".
Re: [PATCH 11/11 v3] btrfs: Use btrfs_transaction::pinned_extents
In reply to Nikolay Borisovfs/btrfs/transaction.c static noinline int join_transaction(struct btrfs_fs_info *fs_info, · line 337
Quoted 7 lines · Nikolay Borisov
fs/btrfs/transaction.c +1
-0@@ -334,6 +334,7 @@ static noinline int join_transaction(struct btrfs_fs_info *fs_info,list_add_tail (&cur_trans->list, &fs_info->trans_list);extent_io_tree_init (fs_info, &cur_trans->dirty_pages,IO_TREE_TRANS_DIRTY_PAGES , fs_info->btree_inode); +extent_io_tree_init (fs_info, &cur_trans->pinned_extents,0 ,NULL );
What's the reason there's no symbolic name for pinned_extents? Also 0 matches IO_TREE_FS_INFO_EXCLUDED_EXTENTS because it's first in the enum list.
Re: [PATCH 11/11 v3] btrfs: Use btrfs_transaction::pinned_extents
Quoted 13 lines · David Sterba
fs/btrfs/transaction.c static noinline int join_transaction(struct btrfs_fs_info *fs_info, · line 337
Quoted 7 lines · Nikolay Borisov
fs/btrfs/transaction.c +1
-0@@ -334,6 +334,7 @@ static noinline int join_transaction(struct btrfs_fs_info *fs_info,list_add_tail (&cur_trans->list, &fs_info->trans_list);extent_io_tree_init (fs_info, &cur_trans->dirty_pages,IO_TREE_TRANS_DIRTY_PAGES , fs_info->btree_inode); +extent_io_tree_init (fs_info, &cur_trans->pinned_extents,0 ,NULL );What's the reason there's no symbolic name for
pinned_extents? Also 0 matchesIO_TREE_FS_INFO_EXCLUDED_EXTENTSbecause it's first in the enum list.
No reason, I'll change it in next version :)
Re: [PATCH 11/11 v3] btrfs: Use btrfs_transaction::pinned_extents
Quoted 17 lines · Nikolay Borisov
Quoted 13 lines · David Sterba
fs/btrfs/transaction.c static noinline int join_transaction(struct btrfs_fs_info *fs_info, · line 337
Quoted 7 lines · Nikolay Borisov
fs/btrfs/transaction.c +1
-0@@ -334,6 +334,7 @@ static noinline int join_transaction(struct btrfs_fs_info *fs_info,list_add_tail (&cur_trans->list, &fs_info->trans_list);extent_io_tree_init (fs_info, &cur_trans->dirty_pages,IO_TREE_TRANS_DIRTY_PAGES , fs_info->btree_inode); +extent_io_tree_init (fs_info, &cur_trans->pinned_extents,0 ,NULL );What's the reason there's no symbolic name for
pinned_extents? Also 0 matchesIO_TREE_FS_INFO_EXCLUDED_EXTENTSbecause it's first in the enum list.
Quoted 1 line
No reason, I'll change it in next version :)
Patchset going to misc-next with the following fixup:
fs/btrfs/extent-io-tree.h +1 -0
@@ -36,6 +36,7 @@ struct io_failure_record;#define CHUNK_TRIMMED EXTENT_DEFRAGenum { +IO_TREE_FS_PINNED_EXTENTS ,IO_TREE_FS_EXCLUDED_EXTENTS ,IO_TREE_INODE_IO ,IO_TREE_INODE_IO_FAILURE ,
fs/btrfs/transaction.c +2 -1
@@ -334,7 +334,8 @@ static noinline int join_transaction(struct btrfs_fs_info *fs_info,list_add_tail (&cur_trans->list, &fs_info->trans_list);extent_io_tree_init (fs_info, &cur_trans->dirty_pages,IO_TREE_TRANS_DIRTY_PAGES , fs_info->btree_inode); -extent_io_tree_init (fs_info, &cur_trans->pinned_extents,0 ,NULL ); +extent_io_tree_init (fs_info, &cur_trans->pinned_extents, +IO_TREE_FS_PINNED_EXTENTS ,NULL ); fs_info->generation++; cur_trans->transid = fs_info->generation; fs_info->running_transaction = cur_trans;
include/trace/events/btrfs.h +1 -0
@@ -81,6 +81,7 @@ TRACE_DEFINE_ENUM(COMMIT_TRANS);#define show_extent_io_tree_owner (owner) \ __print_symbolic(owner, \ + { IO_TREE_FS_PINNED_EXTENTS, "PINNED_EXTENTS" }, \ { IO_TREE_FS_EXCLUDED_EXTENTS, "EXCLUDED_EXTENTS" }, \ { IO_TREE_INODE_IO, "INODE_IO" }, \ { IO_TREE_INODE_IO_FAILURE, "INODE_IO_FAILURE" }, \
Quoted 21 lines · Nikolay Borisov
I've finally managed to finish and test the pinned extents rework. So here it is.
The idea is to move pinnex extents tracking from the global
fs_info->pinned_extents, which is just a pointer to fs_info->freed_extents[] array members to a per transaction structure. This will allow to cleanup pinned extents information during transaction abort.The bulk of the changes are necessary to ensure a valid transaction handle is passed to every function that utilizes
fs_info->pinned_extents. Also it was necessary to account for the peculiarities of excluded extents which are also tracked in ->freed_extents array but with a different flag (yuck).First 9 patches propagate
btrfs_trans_handlewhere it's necessary. Patch 10 factors out pinned extent clean up to make Patch 11 more readable , alternatively the changes in patch 10 had to be in patch 11 which would have made it messier for review.I believe this series doesn't bring any user facing changes it (hopefully) streamlines the code and makes it apparent what the lifetime of pinned extents is and paves the way for further cleanups of
BUG_ON.
Quoted 12 lines
Nikolay Borisov (11):
btrfs: Perform pinned cleanup directly inbtrfs_destroy_delayed_refs
btrfs: Makebtrfs_pin_extenttake trans handle
btrfs: Introduceunaccount_log_buffer
btrfs: Callbtrfs_pin_reserved_extentonly during active transaction
btrfs: Makebtrfs_pin_reserved_extenttake transaction
btrfs: Makebtrfs_pin_extent_for_log_replaytake transaction handle
btrfs: Makepin_down_extenttakebtrfs_trans_handle
btrfs: Pass trans handle towrite_pinned_extent_entries
btrfs: Mark pinned log extents as excluded
btrfs: Factor out pinned extent clean up inbtrfs_delete_unused_bgs
btrfs: Usebtrfs_transaction::pinned_extents
So far I found test btrfs/139 to fail, nothing in the log, only the golden output does not match.