Skip to contentSkip to conversation

btrfs-progs: Metadata preallocation enhancement

PATCH In review Qu Wenruo 5 participants 9 messages Apr 16, 2019 – Jun 6, 2019 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

cover btrfs-progs: Metadata preallocation enhancement

This patchset will address the github issue #123 to make btrfs-convert less possible to report false ENOSPC.

The first patch will try to avoid the nested chunk/extent tree modification in a more explicit way.

The 2nd patch will address the ENOSPC problem, by adding CSUM tree to the metadata preallocate list.

The 2nd patch is not as aggressive as 7a12d8470e5f ("btrfs-progs:
Do metadata preallocation as long as we're not modifying extent tree"). Even we have the 1st patch I still prefer to make less modification to avoid possible bugs.

Qu Wenruo (2):
btrfs-progs: Avoid nested chunk allocation call
btrfs-progs: Do metadata preallocation for fs trees and csum tree

 check/main.c  |  2 +-
 extent-tree.c | 20 +++++++++++++++++++-
 transaction.c |  3 ++-
 transaction.h |  3 ++-
 4 files changed, 24 insertions(+), 4 deletions(-)
Signature · 1 line
2.21.0
author
#1 lore raw

PATCH 1/2 btrfs-progs: Avoid nested chunk allocation call

There is a hidden call loop which can trigger itself:

btrfs_reserve_extent()             <--|
|- do_chunk_alloc()                   |
   |- btrfs_alloc_chunk()             |
      |- btrfs_insert_item()          |
	 |- btrfs_reserve_extent() <--|

Currently, we're using root->ref_cows to determine whether we should do chunk prealloc to avoid such loop.

But that's still a hidden trap. Instead of solving it using some hidden tricks, this patch will make chunk/block group allocation exclusive.

Now if do_chunk_alloc() determines to alloc chunk, it will set a special flag in transaction handler so it new do_chunk_alloc() will refuse to allocate new chunk until current chunk allocation finishes.

---
 check/main.c  |  2 +-
 extent-tree.c | 12 ++++++++++++
 transaction.c |  3 ++-
 transaction.h |  3 ++-
 4 files changed, 17 insertions(+), 3 deletions(-)
check/main.c +1 -1
index 683c322eba6f..121be4b73c4f 100644
@@ -10185,7 +10185,7 @@ int cmd_check(int argc, char **argv)
 			goto close_out;
 		}
 
-		trans->reinit_extent_tree = true;
+		trans->reinit_extent_tree = 1;
 		if (init_extent_tree) {
 			printf("Creating a new extent tree\n");
 			ret = reinit_extent_tree(trans, info,
extent-tree.c +12 -0
index 8c9cdeff3b02..e25ddf02e5cc 100644
@@ -1873,10 +1873,21 @@ static int do_chunk_alloc(struct btrfs_trans_handle *trans,
 	    (flags & BTRFS_BLOCK_GROUP_SYSTEM))
 		return 0;
 
+	/*
+	 * We're going to allocate new chunk, during the process, we will
+	 * allocate new tree blocks, which can trigger new chunk allocation
+	 * again.
+	 * Avoid such loop call
+	 */
+	if (trans->allocating_chunk)
+		return 0;
+	trans->allocating_chunk = 1;
+
 	ret = btrfs_alloc_chunk(trans, fs_info, &start, &num_bytes,
 	                        space_info->flags);
 	if (ret == -ENOSPC) {
 		space_info->full = 1;
+		trans->allocating_chunk = 0;
 		return 0;
 	}
 
@@ -1885,6 +1896,7 @@ static int do_chunk_alloc(struct btrfs_trans_handle *trans,
 	ret = btrfs_make_block_group(trans, fs_info, 0, space_info->flags,
 				     start, num_bytes);
 	BUG_ON(ret);
+	trans->allocating_chunk = 0;
 	return 0;
 }
 
transaction.c +2 -1
index 3a63988b0969..21377282f806 100644
@@ -46,7 +46,8 @@ struct btrfs_trans_handle* btrfs_start_transaction(struct btrfs_root *root,
 	fs_info->generation++;
 	h->transid = fs_info->generation;
 	h->blocks_reserved = num_blocks;
-	h->reinit_extent_tree = false;
+	h->reinit_extent_tree = 0;
+	h->allocating_chunk = 0;
 	root->last_trans = h->transid;
 	root->commit_root = root->node;
 	extent_buffer_get(root->node);
transaction.h +2 -1
index 34060252dd5c..b426cd112447 100644
@@ -28,7 +28,8 @@ struct btrfs_trans_handle {
 	u64 transid;
 	u64 alloc_exclude_start;
 	u64 alloc_exclude_nr;
-	bool reinit_extent_tree;
+	unsigned int reinit_extent_tree:1;
+	unsigned int allocating_chunk:1;
 	u64 delayed_ref_updates;
 	unsigned long blocks_reserved;
 	unsigned long blocks_used;
Signature · 1 line
2.21.0
#2 lore raw

Re: [PATCH 1/2] btrfs-progs: Avoid nested chunk allocation call

Quoted 12 lines · Qu Wenruo

There is a hidden call loop which can trigger itself:

btrfs_reserve_extent()             <--|
|- do_chunk_alloc()                   |
   |- btrfs_alloc_chunk()             |
      |- btrfs_insert_item()          |
         |- btrfs_reserve_extent() <--|

Currently, we're using root->ref_cows to determine whether we should do chunk prealloc to avoid such loop.

Quoted 6 lines

But that's still a hidden trap. Instead of solving it using some hidden tricks, this patch will make chunk/block group allocation exclusive.

Now if do_chunk_alloc() determines to alloc chunk, it will set a special flag in transaction handler so it new do_chunk_alloc() will refuse to allocate new chunk until current chunk allocation finishes.

What if the chunk allocated during the recursion is of a different type? I.e. we're allocating a data chunk, then when inserting the items in the extent, chunk, device, etc trees, we run out of metadata data space and need to allocate a metadata chunk. What you are doing skips the allocation of the metadata chunk and makes the inserts/updates of the trees fail with ENOSPC.

thanks

Quoted 87 lines
---
 check/main.c  |  2 +-
 extent-tree.c | 12 ++++++++++++
 transaction.c |  3 ++-
 transaction.h |  3 ++-
 4 files changed, 17 insertions(+), 3 deletions(-)
check/main.c +1 -1
index 683c322eba6f..121be4b73c4f 100644
@@ -10185,7 +10185,7 @@ int cmd_check(int argc, char **argv)
                        goto close_out;
                }
 
-               trans->reinit_extent_tree = true;
+               trans->reinit_extent_tree = 1;
                if (init_extent_tree) {
                        printf("Creating a new extent tree\n");
                        ret = reinit_extent_tree(trans, info,
extent-tree.c +12 -0
index 8c9cdeff3b02..e25ddf02e5cc 100644
@@ -1873,10 +1873,21 @@ static int do_chunk_alloc(struct btrfs_trans_handle *trans,
            (flags & BTRFS_BLOCK_GROUP_SYSTEM))
                return 0;
 
+       /*
+        * We're going to allocate new chunk, during the process, we will
+        * allocate new tree blocks, which can trigger new chunk allocation
+        * again.
+        * Avoid such loop call
+        */
+       if (trans->allocating_chunk)
+               return 0;
+       trans->allocating_chunk = 1;
+
        ret = btrfs_alloc_chunk(trans, fs_info, &start, &num_bytes,
                                space_info->flags);
        if (ret == -ENOSPC) {
                space_info->full = 1;
+               trans->allocating_chunk = 0;
                return 0;
        }
 
@@ -1885,6 +1896,7 @@ static int do_chunk_alloc(struct btrfs_trans_handle *trans,
        ret = btrfs_make_block_group(trans, fs_info, 0, space_info->flags,
                                     start, num_bytes);
        BUG_ON(ret);
+       trans->allocating_chunk = 0;
        return 0;
 }
 
transaction.c +2 -1
index 3a63988b0969..21377282f806 100644
@@ -46,7 +46,8 @@ struct btrfs_trans_handle* btrfs_start_transaction(struct btrfs_root *root,
        fs_info->generation++;
        h->transid = fs_info->generation;
        h->blocks_reserved = num_blocks;
-       h->reinit_extent_tree = false;
+       h->reinit_extent_tree = 0;
+       h->allocating_chunk = 0;
        root->last_trans = h->transid;
        root->commit_root = root->node;
        extent_buffer_get(root->node);
transaction.h +2 -2
index 34060252dd5c..b426cd112447 100644
@@ -28,7 +28,8 @@ struct btrfs_trans_handle {
        u64 transid;
        u64 alloc_exclude_start;
        u64 alloc_exclude_nr;
-       bool reinit_extent_tree;
+       unsigned int reinit_extent_tree:1;
+       unsigned int allocating_chunk:1;
        u64 delayed_ref_updates;
        unsigned long blocks_reserved;
        unsigned long blocks_used;
--

2.21.0

Signature · 3 lines
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”
#3 lore raw

Re: [PATCH 1/2] btrfs-progs: Avoid nested chunk allocation call

Quoted 20 lines · Filipe Manana
Quoted 12 lines · Qu Wenruo

There is a hidden call loop which can trigger itself:

btrfs_reserve_extent()             <--|
|- do_chunk_alloc()                   |
   |- btrfs_alloc_chunk()             |
      |- btrfs_insert_item()          |
         |- btrfs_reserve_extent() <--|

Currently, we're using root->ref_cows to determine whether we should do chunk prealloc to avoid such loop.

Quoted 6 lines

But that's still a hidden trap. Instead of solving it using some hidden tricks, this patch will make chunk/block group allocation exclusive.

Now if do_chunk_alloc() determines to alloc chunk, it will set a special flag in transaction handler so it new do_chunk_alloc() will refuse to allocate new chunk until current chunk allocation finishes.

Quoted 6 lines

What if the chunk allocated during the recursion is of a different type? I.e. we're allocating a data chunk, then when inserting the items in the extent, chunk, device, etc trees, we run out of metadata data space and need to allocate a metadata chunk. What you are doing skips the allocation of the metadata chunk and makes the inserts/updates of the trees fail with ENOSPC.

That's why we do preallocation to avoid such problem.

Just as the old code shows, even when we're allocating data chunks, it will try to check metadata space first.

And for the profile we're asking, we over-allocate 2M, definitely not the best solution but should be enough for extent/chunk tree update.

To make it as good as kernel, we need a lot of accounting which is not here in btrfs-progs.

So in your case, every btrfs_reserve_extent() call should either trigger
chunk allocation (either metadata or data, or both), or have 2M overhead.

If that 2M can not meet the metadata space requirement for chunk/extent/root tree update, then we hit the problem you described.

However 2M looks pretty generous to me, so it should just work.

Thanks,
Qu

Quoted 94 lines

thanks

Quoted 87 lines
---
 check/main.c  |  2 +-
 extent-tree.c | 12 ++++++++++++
 transaction.c |  3 ++-
 transaction.h |  3 ++-
 4 files changed, 17 insertions(+), 3 deletions(-)
check/main.c +1 -1
index 683c322eba6f..121be4b73c4f 100644
@@ -10185,7 +10185,7 @@ int cmd_check(int argc, char **argv)
                        goto close_out;
                }
 
-               trans->reinit_extent_tree = true;
+               trans->reinit_extent_tree = 1;
                if (init_extent_tree) {
                        printf("Creating a new extent tree\n");
                        ret = reinit_extent_tree(trans, info,
extent-tree.c +12 -0
index 8c9cdeff3b02..e25ddf02e5cc 100644
@@ -1873,10 +1873,21 @@ static int do_chunk_alloc(struct btrfs_trans_handle *trans,
            (flags & BTRFS_BLOCK_GROUP_SYSTEM))
                return 0;
 
+       /*
+        * We're going to allocate new chunk, during the process, we will
+        * allocate new tree blocks, which can trigger new chunk allocation
+        * again.
+        * Avoid such loop call
+        */
+       if (trans->allocating_chunk)
+               return 0;
+       trans->allocating_chunk = 1;
+
        ret = btrfs_alloc_chunk(trans, fs_info, &start, &num_bytes,
                                space_info->flags);
        if (ret == -ENOSPC) {
                space_info->full = 1;
+               trans->allocating_chunk = 0;
                return 0;
        }
 
@@ -1885,6 +1896,7 @@ static int do_chunk_alloc(struct btrfs_trans_handle *trans,
        ret = btrfs_make_block_group(trans, fs_info, 0, space_info->flags,
                                     start, num_bytes);
        BUG_ON(ret);
+       trans->allocating_chunk = 0;
        return 0;
 }
 
transaction.c +2 -1
index 3a63988b0969..21377282f806 100644
@@ -46,7 +46,8 @@ struct btrfs_trans_handle* btrfs_start_transaction(struct btrfs_root *root,
        fs_info->generation++;
        h->transid = fs_info->generation;
        h->blocks_reserved = num_blocks;
-       h->reinit_extent_tree = false;
+       h->reinit_extent_tree = 0;
+       h->allocating_chunk = 0;
        root->last_trans = h->transid;
        root->commit_root = root->node;
        extent_buffer_get(root->node);
transaction.h +2 -2
index 34060252dd5c..b426cd112447 100644
@@ -28,7 +28,8 @@ struct btrfs_trans_handle {
        u64 transid;
        u64 alloc_exclude_start;
        u64 alloc_exclude_nr;
-       bool reinit_extent_tree;
+       unsigned int reinit_extent_tree:1;
+       unsigned int allocating_chunk:1;
        u64 delayed_ref_updates;
        unsigned long blocks_reserved;
        unsigned long blocks_used;
--

2.21.0

#4 lore raw

Re: [PATCH 1/2] btrfs-progs: Avoid nested chunk allocation call

Quoted 24 lines · Qu Wenruo
Quoted 20 lines · Filipe Manana
Quoted 18 lines · Qu Wenruo

There is a hidden call loop which can trigger itself:

btrfs_reserve_extent()             <--|
|- do_chunk_alloc()                   |
   |- btrfs_alloc_chunk()             |
      |- btrfs_insert_item()          |
         |- btrfs_reserve_extent() <--|

Currently, we're using root->ref_cows to determine whether we should do chunk prealloc to avoid such loop.

But that's still a hidden trap. Instead of solving it using some hidden tricks, this patch will make chunk/block group allocation exclusive.

Now if do_chunk_alloc() determines to alloc chunk, it will set a special flag in transaction handler so it new do_chunk_alloc() will refuse to allocate new chunk until current chunk allocation finishes.

Quoted 8 lines
Quoted 6 lines

What if the chunk allocated during the recursion is of a different type? I.e. we're allocating a data chunk, then when inserting the items in the extent, chunk, device, etc trees, we run out of metadata data space and need to allocate a metadata chunk. What you are doing skips the allocation of the metadata chunk and makes the inserts/updates of the trees fail with ENOSPC.

That's why we do preallocation to avoid such problem.

Ok, I wasn't aware about such preallocation in btrfs-progs.

Quoted 10 lines

Just as the old code shows, even when we're allocating data chunks, it will try to check metadata space first.

And for the profile we're asking, we over-allocate 2M, definitely not the best solution but should be enough for extent/chunk tree update.

To make it as good as kernel, we need a lot of accounting which is not here in btrfs-progs.

Quoted 7 lines

So in your case, every btrfs_reserve_extent() call should either trigger
chunk allocation (either metadata or data, or both), or have 2M overhead.

If that 2M can not meet the metadata space requirement for chunk/extent/root tree update, then we hit the problem you described.

However 2M looks pretty generous to me, so it should just work.

Yes, that should be enough even for 64Kb nodes and trees with 8 (max) levels.

thanks

Quoted 99 lines

Thanks,
Qu

Quoted 94 lines

thanks

Quoted 87 lines
---
 check/main.c  |  2 +-
 extent-tree.c | 12 ++++++++++++
 transaction.c |  3 ++-
 transaction.h |  3 ++-
 4 files changed, 17 insertions(+), 3 deletions(-)
check/main.c +1 -1
index 683c322eba6f..121be4b73c4f 100644
@@ -10185,7 +10185,7 @@ int cmd_check(int argc, char **argv)
                        goto close_out;
                }
 
-               trans->reinit_extent_tree = true;
+               trans->reinit_extent_tree = 1;
                if (init_extent_tree) {
                        printf("Creating a new extent tree\n");
                        ret = reinit_extent_tree(trans, info,
extent-tree.c +12 -0
index 8c9cdeff3b02..e25ddf02e5cc 100644
@@ -1873,10 +1873,21 @@ static int do_chunk_alloc(struct btrfs_trans_handle *trans,
            (flags & BTRFS_BLOCK_GROUP_SYSTEM))
                return 0;
 
+       /*
+        * We're going to allocate new chunk, during the process, we will
+        * allocate new tree blocks, which can trigger new chunk allocation
+        * again.
+        * Avoid such loop call
+        */
+       if (trans->allocating_chunk)
+               return 0;
+       trans->allocating_chunk = 1;
+
        ret = btrfs_alloc_chunk(trans, fs_info, &start, &num_bytes,
                                space_info->flags);
        if (ret == -ENOSPC) {
                space_info->full = 1;
+               trans->allocating_chunk = 0;
                return 0;
        }
 
@@ -1885,6 +1896,7 @@ static int do_chunk_alloc(struct btrfs_trans_handle *trans,
        ret = btrfs_make_block_group(trans, fs_info, 0, space_info->flags,
                                     start, num_bytes);
        BUG_ON(ret);
+       trans->allocating_chunk = 0;
        return 0;
 }
 
transaction.c +2 -1
index 3a63988b0969..21377282f806 100644
@@ -46,7 +46,8 @@ struct btrfs_trans_handle* btrfs_start_transaction(struct btrfs_root *root,
        fs_info->generation++;
        h->transid = fs_info->generation;
        h->blocks_reserved = num_blocks;
-       h->reinit_extent_tree = false;
+       h->reinit_extent_tree = 0;
+       h->allocating_chunk = 0;
        root->last_trans = h->transid;
        root->commit_root = root->node;
        extent_buffer_get(root->node);
transaction.h +2 -2
index 34060252dd5c..b426cd112447 100644
@@ -28,7 +28,8 @@ struct btrfs_trans_handle {
        u64 transid;
        u64 alloc_exclude_start;
        u64 alloc_exclude_nr;
-       bool reinit_extent_tree;
+       unsigned int reinit_extent_tree:1;
+       unsigned int allocating_chunk:1;
        u64 delayed_ref_updates;
        unsigned long blocks_reserved;
        unsigned long blocks_used;
--

2.21.0

Signature · 3 lines
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”
#5 lore raw

Re: [PATCH 1/2] btrfs-progs: Avoid nested chunk allocation call

In reply to Qu Wenruo
Quoted 92 lines · Qu Wenruo

There is a hidden call loop which can trigger itself:

btrfs_reserve_extent()             <--|
|- do_chunk_alloc()                   |
   |- btrfs_alloc_chunk()             |
      |- btrfs_insert_item()          |
	 |- btrfs_reserve_extent() <--|

Currently, we're using root->ref_cows to determine whether we should do chunk prealloc to avoid such loop.

But that's still a hidden trap. Instead of solving it using some hidden tricks, this patch will make chunk/block group allocation exclusive.

Now if do_chunk_alloc() determines to alloc chunk, it will set a special flag in transaction handler so it new do_chunk_alloc() will refuse to allocate new chunk until current chunk allocation finishes.

---
 check/main.c  |  2 +-
 extent-tree.c | 12 ++++++++++++
 transaction.c |  3 ++-
 transaction.h |  3 ++-
 4 files changed, 17 insertions(+), 3 deletions(-)
check/main.c +1 -1
index 683c322eba6f..121be4b73c4f 100644
@@ -10185,7 +10185,7 @@ int cmd_check(int argc, char **argv)
 			goto close_out;
 		}
 
-		trans->reinit_extent_tree = true;
+		trans->reinit_extent_tree = 1;
 		if (init_extent_tree) {
 			printf("Creating a new extent tree\n");
 			ret = reinit_extent_tree(trans, info,
extent-tree.c +12 -0
index 8c9cdeff3b02..e25ddf02e5cc 100644
@@ -1873,10 +1873,21 @@ static int do_chunk_alloc(struct btrfs_trans_handle *trans,
 	    (flags & BTRFS_BLOCK_GROUP_SYSTEM))
 		return 0;
 
+	/*
+	 * We're going to allocate new chunk, during the process, we will
+	 * allocate new tree blocks, which can trigger new chunk allocation
+	 * again.
+	 * Avoid such loop call
+	 */
+	if (trans->allocating_chunk)
+		return 0;
+	trans->allocating_chunk = 1;
+
 	ret = btrfs_alloc_chunk(trans, fs_info, &start, &num_bytes,
 	                        space_info->flags);
 	if (ret == -ENOSPC) {
 		space_info->full = 1;
+		trans->allocating_chunk = 0;
 		return 0;
 	}
 
@@ -1885,6 +1896,7 @@ static int do_chunk_alloc(struct btrfs_trans_handle *trans,
 	ret = btrfs_make_block_group(trans, fs_info, 0, space_info->flags,
 				     start, num_bytes);
 	BUG_ON(ret);
+	trans->allocating_chunk = 0;
 	return 0;
 }
 
transaction.c +2 -1
index 3a63988b0969..21377282f806 100644
@@ -46,7 +46,8 @@ struct btrfs_trans_handle* btrfs_start_transaction(struct btrfs_root *root,
 	fs_info->generation++;
 	h->transid = fs_info->generation;
 	h->blocks_reserved = num_blocks;
-	h->reinit_extent_tree = false;
+	h->reinit_extent_tree = 0;
+	h->allocating_chunk = 0;
 	root->last_trans = h->transid;
 	root->commit_root = root->node;
 	extent_buffer_get(root->node);
transaction.h +0 -0
index 34060252dd5c..b426cd112447 100644

transaction.h struct btrfs_trans_handle { · line 32

Quoted 6 lines
transaction.h +2 -1
@@ -28,7 +28,8 @@ struct btrfs_trans_handle {
 	u64 transid;
 	u64 alloc_exclude_start;
 	u64 alloc_exclude_nr;
-	bool reinit_extent_tree;
+	unsigned int reinit_extent_tree:1;
+	unsigned int allocating_chunk:1;

Why do you switch reinit_extent_tree to the bit, this is an unrelated change. I'll drop it and update changelog with the 2M preallocation that was discussed.

#6 lore raw

Re: [PATCH 1/2] btrfs-progs: Avoid nested chunk allocation call

Quoted 95 lines · David Sterba
Quoted 94 lines · Qu Wenruo

There is a hidden call loop which can trigger itself:

btrfs_reserve_extent()             <--|
|- do_chunk_alloc()                   |
   |- btrfs_alloc_chunk()             |
      |- btrfs_insert_item()          |
	 |- btrfs_reserve_extent() <--|

Currently, we're using root->ref_cows to determine whether we should do chunk prealloc to avoid such loop.

But that's still a hidden trap. Instead of solving it using some hidden tricks, this patch will make chunk/block group allocation exclusive.

Now if do_chunk_alloc() determines to alloc chunk, it will set a special flag in transaction handler so it new do_chunk_alloc() will refuse to allocate new chunk until current chunk allocation finishes.

---
 check/main.c  |  2 +-
 extent-tree.c | 12 ++++++++++++
 transaction.c |  3 ++-
 transaction.h |  3 ++-
 4 files changed, 17 insertions(+), 3 deletions(-)
check/main.c +1 -1
index 683c322eba6f..121be4b73c4f 100644
@@ -10185,7 +10185,7 @@ int cmd_check(int argc, char **argv)
 			goto close_out;
 		}
 
-		trans->reinit_extent_tree = true;
+		trans->reinit_extent_tree = 1;
 		if (init_extent_tree) {
 			printf("Creating a new extent tree\n");
 			ret = reinit_extent_tree(trans, info,
extent-tree.c +12 -0
index 8c9cdeff3b02..e25ddf02e5cc 100644
@@ -1873,10 +1873,21 @@ static int do_chunk_alloc(struct btrfs_trans_handle *trans,
 	    (flags & BTRFS_BLOCK_GROUP_SYSTEM))
 		return 0;
 
+	/*
+	 * We're going to allocate new chunk, during the process, we will
+	 * allocate new tree blocks, which can trigger new chunk allocation
+	 * again.
+	 * Avoid such loop call
+	 */
+	if (trans->allocating_chunk)
+		return 0;
+	trans->allocating_chunk = 1;
+
 	ret = btrfs_alloc_chunk(trans, fs_info, &start, &num_bytes,
 	                        space_info->flags);
 	if (ret == -ENOSPC) {
 		space_info->full = 1;
+		trans->allocating_chunk = 0;
 		return 0;
 	}
 
@@ -1885,6 +1896,7 @@ static int do_chunk_alloc(struct btrfs_trans_handle *trans,
 	ret = btrfs_make_block_group(trans, fs_info, 0, space_info->flags,
 				     start, num_bytes);
 	BUG_ON(ret);
+	trans->allocating_chunk = 0;
 	return 0;
 }
 
transaction.c +2 -1
index 3a63988b0969..21377282f806 100644
@@ -46,7 +46,8 @@ struct btrfs_trans_handle* btrfs_start_transaction(struct btrfs_root *root,
 	fs_info->generation++;
 	h->transid = fs_info->generation;
 	h->blocks_reserved = num_blocks;
-	h->reinit_extent_tree = false;
+	h->reinit_extent_tree = 0;
+	h->allocating_chunk = 0;
 	root->last_trans = h->transid;
 	root->commit_root = root->node;
 	extent_buffer_get(root->node);
transaction.h +0 -0
index 34060252dd5c..b426cd112447 100644
@@ -28,7 +28,8 @@ struct btrfs_trans_handle {
 	u64 transid;
 	u64 alloc_exclude_start;
Quoted 9 lines

transaction.h struct btrfs_trans_handle { · line 32

Quoted 4 lines
transaction.h +2 -1
 	u64 alloc_exclude_nr;
-	bool reinit_extent_tree;
+	unsigned int reinit_extent_tree:1;
+	unsigned int allocating_chunk:1;

Why do you switch reinit_extent_tree to the bit, this is an unrelated change. I'll drop it and update changelog with the 2M preallocation that was discussed.

Because in this patch, we're introducing new bit, thus to me it's pretty valid to expanding old bool value into bits.

Thanks,
Qu

author
#7 lore raw

PATCH 2/2 btrfs-progs: Do metadata preallocation for fs trees and csum tree

In reply to Qu Wenruo

In github issues, one user reports unexpected ENOSPC error if enabling datasum druing convert. After some investigation, it looks like that during ext2_saved/image creation, we could create large file extent whose size can be 128M (max data extent size).

In that case, its csum block will be at least 128K. Under certain case we need to allocate extra metadata chunks to fulfill such space requirement.

However we only do metadata prealloc if we're reserving extents for fs trees. (we use btrfs_root::ref_cows to determine whether we should do metadata prealloc, and that member is only set for fs trees).

There is no explaination on why we only do metadata prealloc for file trees, but from my educated guess, it could be related to avoid nested extent/chunk tree modication.

At least extent reservation for csum tree shouldn't be a problem with metadata block group preallocation.

So adding new condition for metadata preallocate to avoid unexpected ENOSPC problem.

Issue: #123

---
 extent-tree.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
extent-tree.c +7 -1
index e25ddf02e5cc..cf2e29258dfb 100644
@@ -2518,7 +2518,13 @@ int btrfs_reserve_extent(struct btrfs_trans_handle *trans,
 		profile = BTRFS_BLOCK_GROUP_METADATA | alloc_profile;
 	}
 
-	if (root->ref_cows) {
+	/*
+	 * Also preallocate metadata for csum tree and fs trees (root->ref_cows
+	 * already set), as they can consume a lot of metadata space.
+	 * Pre-allocate to avoid unexpected ENOSPC.
+	 */
+	if (root->ref_cows ||
+	    root->root_key.objectid == BTRFS_CSUM_TREE_OBJECTID) {
 		if (!(profile & BTRFS_BLOCK_GROUP_METADATA)) {
 			ret = do_chunk_alloc(trans, info,
 					     num_bytes,
Signature · 1 line
2.21.0
#8 lore raw
In reply to Qu Wenruo
Quoted 9 lines · Qu Wenruo

This patchset will address the github issue #123 to make btrfs-convert less possible to report false ENOSPC.

The first patch will try to avoid the nested chunk/extent tree modification in a more explicit way.

The 2nd patch will address the ENOSPC problem, by adding CSUM tree to the metadata preallocate list.

Quoted 8 lines

The 2nd patch is not as aggressive as 7a12d8470e5f ("btrfs-progs:
Do metadata preallocation as long as we're not modifying extent tree"). Even we have the 1st patch I still prefer to make less modification to avoid possible bugs.

Qu Wenruo (2):
btrfs-progs: Avoid nested chunk allocation call
btrfs-progs: Do metadata preallocation for fs trees and csum tree

Added to devel, thanks.