Skip to contentSkip to conversation

Add delayed-refs support to btrfs-progs

PATCH In review Nikolay Borisov 3 participants 16 messages Aug 16, 2018 – Sep 14, 2018

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover Add delayed-refs support to btrfs-progs

Hello,

Here is the second version of the delayed refs for progs support. The first version can be found here [1]. I've taken into account all the feedback from Misono and have verified the code is working and rebased it atop btrfs-progs 4.17.1.

Changes since v1:
* Removed num_entries variable from delayed ref root

* Added a patch to refactor btrfs_write_dirty_block_groups and subsequently changed when this function is called to fix an issue reported by Misono. I verified that 'make test-fsck TEST_ENABLE_OVERRIDE=true TEST_ARGS_CHECK=--mode=lowmem' produces no errors

* Added 2 patches which remove the newly added adapter functions at the beggining of the series, following the wiring up of the delayed refs infrastructured. The first one (dealing with __free_extent2) is trivial, while the second one (for alloc_reserved_tree_block2) is a bit more involved, since I've opted to merge the two functions.

* Rebased atop latest btrfs-progs release - 4.17.1

* Dropped patches which have been merged in the mean time

[1] https://www.spinics.net/lists/linux-btrfs/msg79173.html

Nikolay Borisov (8):
btrfs-progs: Add __free_extent2 function
btrfs-progs: Add alloc_reserved_tree_block2 function
btrfs-progs: Add delayed refs infrastructure
btrfs-progs: Make btrfs_write_dirty_block_groups take only trans
argument
btrfs-progs: Wire up delayed refs
btrfs-progs: Remove old delayed refs infrastructure
btrfs-progs: Remove __free_extent2
btrfs-progs: Merge alloc_reserved_tree_block(2|)

 Makefile      |   3 +-
 check/main.c  |   3 +-
 ctree.h       |   8 +-
 delayed-ref.c | 607 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 delayed-ref.h | 208 ++++++++++++++++++++
 disk-io.c     |   2 -
 extent-tree.c | 575 +++++++++++++++++++++++++++++++-----------------------
 kerncompat.h  |   8 +
 transaction.c |  29 ++-
 transaction.h |   4 +
 10 files changed, 1199 insertions(+), 248 deletions(-)
 create mode 100644 delayed-ref.c
 create mode 100644 delayed-ref.h
Signature · 1 line
2.7.4
#1 lore raw

PATCH v2 1/8 btrfs-progs: Add __free_extent2 function

This is a simple adapter to convert the arguments delayed ref arguments to the existing arguments of __free_extent.

---
 extent-tree.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
extent-tree.c +11 -0
index 5d49af5a901e..34409e600087 100644
@@ -2136,6 +2136,17 @@ void btrfs_unpin_extent(struct btrfs_fs_info *fs_info,
 	update_pinned_extents(fs_info, bytenr, num_bytes, 0);
 }
 
+static int __free_extent2(struct btrfs_trans_handle *trans,
+			  struct btrfs_delayed_ref_node *node,
+			  struct btrfs_delayed_extent_op *extent_op)
+{
+
+	struct btrfs_delayed_tree_ref *ref = btrfs_delayed_node_to_tree_ref(node);
+
+	return __free_extent(trans, node->bytenr, node->num_bytes,
+			     ref->parent, ref->root, ref->level, 0, 1);
+}
+
 /*
  * remove an extent from the root, returns 0 on success
  */
Signature · 1 line
2.7.4
#2 lore raw

PATCH v2 2/8 btrfs-progs: Add alloc_reserved_tree_block2 function

In reply to Nikolay Borisov

This is a simple adapter function to convert the delayed-refs structures to the current arguments of alloc_reserved_tree_block.

---
 extent-tree.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
extent-tree.c +24 -0
index 34409e600087..f7b59f84bf3d 100644
@@ -2687,6 +2687,30 @@ int btrfs_reserve_extent(struct btrfs_trans_handle *trans,
 	return ret;
 }
 
+static int alloc_reserved_tree_block2(struct btrfs_trans_handle *trans,
+				      struct btrfs_delayed_ref_node *node,
+				      struct btrfs_delayed_extent_op *extent_op)
+{
+
+	struct btrfs_delayed_tree_ref *ref = btrfs_delayed_node_to_tree_ref(node);
+	struct btrfs_key ins;
+	bool skinny_metadata = btrfs_fs_incompat(trans->fs_info, SKINNY_METADATA);
+
+	ins.objectid = node->bytenr;
+	if (skinny_metadata) {
+		ins.offset = ref->level;
+		ins.type = BTRFS_METADATA_ITEM_KEY;
+	} else {
+		ins.offset = node->num_bytes;
+		ins.type = BTRFS_EXTENT_ITEM_KEY;
+	}
+
+	return alloc_reserved_tree_block(trans, ref->root, trans->transid,
+					 extent_op->flags_to_set,
+					 &extent_op->key, ref->level, &ins);
+
+}
+
 static int alloc_reserved_tree_block(struct btrfs_trans_handle *trans,
 				     u64 root_objectid, u64 generation,
 				     u64 flags, struct btrfs_disk_key *key,
Signature · 1 line
2.7.4
#3 lore raw

PATCH v2 3/8 btrfs-progs: Add delayed refs infrastructure

In reply to Nikolay Borisov

This commit pulls those portions of the kernel implementation of delayed refs which are necessary to have them working in user-space. I've done the following modifications:

1. Replaced all kmem_cache_alloc calls to kmalloc.

2. Removed all locking-related code, since we are single threaded in userspace.

3. Removed code which deals with data refs - delayed refs in user space are going to be used only for cowonly trees.

---
 Makefile      |   3 +-
 ctree.h       |   3 +
 delayed-ref.c | 607 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 delayed-ref.h | 208 ++++++++++++++++++++
 extent-tree.c | 226 ++++++++++++++++++++++
 kerncompat.h  |   8 +
 transaction.h |   4 +
 7 files changed, 1058 insertions(+), 1 deletion(-)
 create mode 100644 delayed-ref.c
 create mode 100644 delayed-ref.h
Makefile +2 -1
index fcfc815a2a5b..f4ab14ea74c8 100644
@@ -116,7 +116,8 @@ objects = ctree.o disk-io.o kernel-lib/radix-tree.o extent-tree.o print-tree.o \
 	  qgroup.o free-space-cache.o kernel-lib/list_sort.o props.o \
 	  kernel-shared/ulist.o qgroup-verify.o backref.o string-table.o task-utils.o \
 	  inode.o file.o find-root.o free-space-tree.o help.o send-dump.o \
-	  fsfeatures.o kernel-lib/tables.o kernel-lib/raid56.o transaction.o
+	  fsfeatures.o kernel-lib/tables.o kernel-lib/raid56.o transaction.o \
+	  delayed-ref.o
 cmds_objects = cmds-subvolume.o cmds-filesystem.o cmds-device.o cmds-scrub.o \
 	       cmds-inspect.o cmds-balance.o cmds-send.o cmds-receive.o \
 	       cmds-quota.o cmds-qgroup.o cmds-replace.o check/main.o \
ctree.h +3 -0
index 4719962df67d..5242595fe355 100644
@@ -2790,4 +2790,7 @@ int btrfs_punch_hole(struct btrfs_trans_handle *trans,
 int btrfs_read_file(struct btrfs_root *root, u64 ino, u64 start, int len,
 		    char *dest);
 
+/* extent-tree.c */
+int btrfs_run_delayed_refs(struct btrfs_trans_handle *trans, unsigned long nr);
+
 #endif
delayed-ref.c new file +607 -0

607 lines not on this page · raw message

delayed-ref.h new file +208 -0

208 lines not on this page · raw message

extent-tree.c +226 -0

229 lines not on this page · raw message

kerncompat.h +8 -0
index fa96715fb70c..1a2bc18c3ac2 100644
@@ -263,6 +263,14 @@ static inline int IS_ERR_OR_NULL(const void *ptr)
 	return !ptr || IS_ERR(ptr);
 }
 
+/**
+ * swap - swap values of @a and @b
+ * @a: first value
+ * @b: second value
+ */
+#define swap(a, b) \
+        do { typeof(a) __tmp = (a); (a) = (b); (b) = __tmp; } while (0)
+
 /*
  * This looks more complex than it should be. But we need to
  * get the type for the ~ right in round_down (it needs to be
transaction.h +4 -0
index 750e329e1ba8..34060252dd5c 100644
@@ -21,6 +21,7 @@
 
 #include "kerncompat.h"
 #include "ctree.h"
+#include "delayed-ref.h"
 
 struct btrfs_trans_handle {
 	struct btrfs_fs_info *fs_info;
@@ -28,9 +29,12 @@ struct btrfs_trans_handle {
 	u64 alloc_exclude_start;
 	u64 alloc_exclude_nr;
 	bool reinit_extent_tree;
+	u64 delayed_ref_updates;
 	unsigned long blocks_reserved;
 	unsigned long blocks_used;
 	struct btrfs_block_group_cache *block_group;
+	struct btrfs_delayed_ref_root delayed_refs;
+
 };
 
 struct btrfs_trans_handle* btrfs_start_transaction(struct btrfs_root *root,
Signature · 1 line
2.7.4
#4 lore raw

PATCH v2 4/8 btrfs-progs: Make btrfs_write_dirty_block_groups take only trans argument

In reply to Nikolay Borisov

The root argument is used only to get a reference to the fs_info, this can be achieved with the transaction handle being passed so use that. This is in preparation for moving this function in the main transaction commit routine. No functional changes.

---
 ctree.h       | 3 +--
 extent-tree.c | 5 ++---
 transaction.c | 4 ++--
 3 files changed, 5 insertions(+), 7 deletions(-)
ctree.h +1 -2
index 5242595fe355..75675ef3f781 100644
@@ -2523,8 +2523,7 @@ int btrfs_update_extent_ref(struct btrfs_trans_handle *trans,
 			    u64 orig_parent, u64 parent,
 			    u64 root_objectid, u64 ref_generation,
 			    u64 owner_objectid);
-int btrfs_write_dirty_block_groups(struct btrfs_trans_handle *trans,
-				    struct btrfs_root *root);
+int btrfs_write_dirty_block_groups(struct btrfs_trans_handle *trans);
 int btrfs_free_block_groups(struct btrfs_fs_info *info);
 int btrfs_read_block_groups(struct btrfs_root *root);
 struct btrfs_block_group_cache *
extent-tree.c +2 -3
index 3356dd2e4cf6..7d6c37c6b371 100644
@@ -1727,8 +1727,7 @@ static int write_one_cache_group(struct btrfs_trans_handle *trans,
 
 }
 
-int btrfs_write_dirty_block_groups(struct btrfs_trans_handle *trans,
-				   struct btrfs_root *root)
+int btrfs_write_dirty_block_groups(struct btrfs_trans_handle *trans)
 {
 	struct extent_io_tree *block_group_cache;
 	struct btrfs_block_group_cache *cache;
@@ -1739,7 +1738,7 @@ int btrfs_write_dirty_block_groups(struct btrfs_trans_handle *trans,
 	u64 end;
 	u64 ptr;
 
-	block_group_cache = &root->fs_info->block_group_cache;
+	block_group_cache = &trans->fs_info->block_group_cache;
 	path = btrfs_alloc_path();
 	if (!path)
 		return -ENOMEM;
transaction.c +2 -2
index ecafbb156610..96d9891b0d1c 100644
@@ -61,7 +61,7 @@ static int update_cowonly_root(struct btrfs_trans_handle *trans,
 	u64 old_root_bytenr;
 	struct btrfs_root *tree_root = root->fs_info->tree_root;
 
-	btrfs_write_dirty_block_groups(trans, root);
+	btrfs_write_dirty_block_groups(trans);
 	while(1) {
 		old_root_bytenr = btrfs_root_bytenr(&root->root_item);
 		if (old_root_bytenr == root->node->start)
@@ -75,7 +75,7 @@ static int update_cowonly_root(struct btrfs_trans_handle *trans,
 					&root->root_key,
 					&root->root_item);
 		BUG_ON(ret);
-		btrfs_write_dirty_block_groups(trans, root);
+		btrfs_write_dirty_block_groups(trans);
 	}
 	return 0;
 }
Signature · 1 line
2.7.4
#5 lore raw

PATCH v2 5/8 btrfs-progs: Wire up delayed refs

In reply to Nikolay Borisov

This commit enables the delayed refs infrastructures. This entails doing the following:

1. Replacing existing calls of btrfs_extent_post_op (which is the equivalent of delayed refs) with the proper btrfs_run_delayed_refs. As well as eliminating open-coded calls to finish_current_insert and del_pending_extents which execute the delayed ops.

2. Wiring up the addition of delayed refs when freeing extents (btrfs_free_extent) and when adding new extents (alloc_tree_block).

3. Adding calls to btrfs_run_delayed refs in the transaction commit path alongside comments why every call is needed, since it's not always obvious (those call sites were derived empirically by running and debugging existing tests)

4. Correctly flagging the transaction in which we are reinitialising the extent tree.

5 Moving btrfs_write_dirty_block_groups to btrfs_write_dirty_block_groups since blockgroups should be written to disk after the last delayed refs have been run.

---
 check/main.c  |   3 +-
 extent-tree.c | 166 ++++++++++++++++++++++++++++++----------------------------
 transaction.c |  27 +++++++++-
 3 files changed, 112 insertions(+), 84 deletions(-)
check/main.c +2 -1
index bc2ee22f7943..b361cd7e26a0 100644
@@ -8710,7 +8710,7 @@ static int reinit_extent_tree(struct btrfs_trans_handle *trans,
 			fprintf(stderr, "Error adding block group\n");
 			return ret;
 		}
-		btrfs_extent_post_op(trans);
+		btrfs_run_delayed_refs(trans, -1);
 	}
 
 	ret = reset_balance(trans, fs_info);
@@ -9767,6 +9767,7 @@ int cmd_check(int argc, char **argv)
 			goto close_out;
 		}
 
+		trans->reinit_extent_tree = true;
 		if (init_extent_tree) {
 			printf("Creating a new extent tree\n");
 			ret = reinit_extent_tree(trans, info,
extent-tree.c +85 -81

269 lines not on this page · raw message

transaction.c +25 -2
index 96d9891b0d1c..bfda769210ee 100644
@@ -61,7 +61,6 @@ static int update_cowonly_root(struct btrfs_trans_handle *trans,
 	u64 old_root_bytenr;
 	struct btrfs_root *tree_root = root->fs_info->tree_root;
 
-	btrfs_write_dirty_block_groups(trans);
 	while(1) {
 		old_root_bytenr = btrfs_root_bytenr(&root->root_item);
 		if (old_root_bytenr == root->node->start)
@@ -98,6 +97,17 @@ int commit_tree_roots(struct btrfs_trans_handle *trans,
 	if (ret)
 		return ret;
 
+	/*
+	 * If the above CoW is the first one to dirty the current tree_root,
+	 * delayed refs for it won't be run until after this function has
+	 * finished executing, meaning we won't process the extent tree root,
+	 * which will have been added to ->dirty_cowonly_roots.  So run
+	 * delayed refs here as well.
+	 */
+	ret = btrfs_run_delayed_refs(trans, -1);
+	if (ret)
+		return ret;
+
 	while(!list_empty(&fs_info->dirty_cowonly_roots)) {
 		next = fs_info->dirty_cowonly_roots.next;
 		list_del_init(next);
@@ -147,6 +157,12 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 
 	if (trans->fs_info->transaction_aborted)
 		return -EROFS;
+	/*
+	 * Flush all accumulated delayed refs so that root-tree updates are
+	 * consistent
+	 */
+	ret = btrfs_run_delayed_refs(trans, -1);
+	BUG_ON(ret);
 
 	if (root->commit_root == root->node)
 		goto commit_tree;
@@ -164,11 +180,18 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 	ret = btrfs_update_root(trans, root->fs_info->tree_root,
 				&root->root_key, &root->root_item);
 	BUG_ON(ret);
+
 commit_tree:
 	ret = commit_tree_roots(trans, fs_info);
 	BUG_ON(ret);
-	ret = __commit_transaction(trans, root);
+	/*
+	 * Ensure that all comitted roots are properly accounted in the
+	 * extent tree
+	 */
+	ret = btrfs_run_delayed_refs(trans, -1);
 	BUG_ON(ret);
+	btrfs_write_dirty_block_groups(trans);
+	__commit_transaction(trans, root);
 	write_ctree_super(trans);
 	btrfs_finish_extent_commit(trans, fs_info->extent_root,
 			           &fs_info->pinned_extents);
Signature · 1 line
2.7.4
#6 lore raw

Re: [PATCH 5/8] btrfs-progs: Wire up delayed refs

Quoted 19 lines · Nikolay Borisov

This commit enables the delayed refs infrastructures. This entails doing the following:

1. Replacing existing calls of btrfs_extent_post_op (which is the equivalent of delayed refs) with the proper btrfs_run_delayed_refs. As well as eliminating open-coded calls to finish_current_insert and del_pending_extents which execute the delayed ops.

2. Wiring up the addition of delayed refs when freeing extents (btrfs_free_extent) and when adding new extents (alloc_tree_block).

3. Adding calls to btrfs_run_delayed refs in the transaction commit path alongside comments why every call is needed, since it's not always obvious (those call sites were derived empirically by running and debugging existing tests)

4. Correctly flagging the transaction in which we are reinitialising the extent tree.

Quoted 6 lines

5 Moving btrfs_write_dirty_block_groups to btrfs_write_dirty_block_groups since blockgroups should be written to disk after the last delayed refs have been run.

Is there something (maybe btrfs_run_delayed_refs()?) missing in btrfs-image?

btrfs-image from devel branch can't restore image correctly, the block group used bytes is not correct, thus it can't pass misc nor fsck tests.

Thanks,
Qu

Quoted 379 lines
---
 check/main.c  |   3 +-
 extent-tree.c | 166 ++++++++++++++++++++++++++++++----------------------------
 transaction.c |  27 +++++++++-
 3 files changed, 112 insertions(+), 84 deletions(-)
check/main.c +2 -1
index bc2ee22f7943..b361cd7e26a0 100644
@@ -8710,7 +8710,7 @@ static int reinit_extent_tree(struct btrfs_trans_handle *trans,
 			fprintf(stderr, "Error adding block group\n");
 			return ret;
 		}
-		btrfs_extent_post_op(trans);
+		btrfs_run_delayed_refs(trans, -1);
 	}
 
 	ret = reset_balance(trans, fs_info);
@@ -9767,6 +9767,7 @@ int cmd_check(int argc, char **argv)
 			goto close_out;
 		}
 
+		trans->reinit_extent_tree = true;
 		if (init_extent_tree) {
 			printf("Creating a new extent tree\n");
 			ret = reinit_extent_tree(trans, info,
extent-tree.c +85 -81
index 7d6c37c6b371..2fa51bbc0359 100644
@@ -1418,8 +1418,6 @@ int btrfs_inc_extent_ref(struct btrfs_trans_handle *trans,
 		err = ret;
 out:
 	btrfs_free_path(path);
-	finish_current_insert(trans);
-	del_pending_extents(trans);
 	BUG_ON(err);
 	return err;
 }
@@ -1602,8 +1600,6 @@ int btrfs_set_block_flags(struct btrfs_trans_handle *trans, u64 bytenr,
 	btrfs_set_extent_flags(l, item, flags);
 out:
 	btrfs_free_path(path);
-	finish_current_insert(trans);
-	del_pending_extents(trans);
 	return ret;
 }
 
@@ -1701,7 +1697,6 @@ static int write_one_cache_group(struct btrfs_trans_handle *trans,
 				 struct btrfs_block_group_cache *cache)
 {
 	int ret;
-	int pending_ret;
 	struct btrfs_root *extent_root = trans->fs_info->extent_root;
 	unsigned long bi;
 	struct extent_buffer *leaf;
@@ -1717,12 +1712,8 @@ static int write_one_cache_group(struct btrfs_trans_handle *trans,
 	btrfs_mark_buffer_dirty(leaf);
 	btrfs_release_path(path);
 fail:
-	finish_current_insert(trans);
-	pending_ret = del_pending_extents(trans);
 	if (ret)
 		return ret;
-	if (pending_ret)
-		return pending_ret;
 	return 0;
 
 }
@@ -2049,6 +2040,7 @@ static int finish_current_insert(struct btrfs_trans_handle *trans)
 	int skinny_metadata =
 		btrfs_fs_incompat(extent_root->fs_info, SKINNY_METADATA);
 
+
 	while(1) {
 		ret = find_first_extent_bit(&info->extent_ins, 0, &start,
 					    &end, EXTENT_LOCKED);
@@ -2080,6 +2072,8 @@ static int finish_current_insert(struct btrfs_trans_handle *trans)
 			BUG_ON(1);
 		}
 
+
+		printf("shouldn't be executed\n");
 		clear_extent_bits(&info->extent_ins, start, end, EXTENT_LOCKED);
 		kfree(extent_op);
 	}
@@ -2379,7 +2373,6 @@ static int __free_extent(struct btrfs_trans_handle *trans,
 	}
 fail:
 	btrfs_free_path(path);
-	finish_current_insert(trans);
 	return ret;
 }
 
@@ -2462,33 +2455,30 @@ int btrfs_free_extent(struct btrfs_trans_handle *trans,
 		      u64 bytenr, u64 num_bytes, u64 parent,
 		      u64 root_objectid, u64 owner, u64 offset)
 {
-	struct btrfs_root *extent_root = root->fs_info->extent_root;
-	int pending_ret;
 	int ret;
 
 	WARN_ON(num_bytes < root->fs_info->sectorsize);
-	if (root == extent_root) {
-		struct pending_extent_op *extent_op;
-
-		extent_op = kmalloc(sizeof(*extent_op), GFP_NOFS);
-		BUG_ON(!extent_op);
-
-		extent_op->type = PENDING_EXTENT_DELETE;
-		extent_op->bytenr = bytenr;
-		extent_op->num_bytes = num_bytes;
-		extent_op->level = (int)owner;
-
-		set_extent_bits(&root->fs_info->pending_del,
-				bytenr, bytenr + num_bytes - 1,
-				EXTENT_LOCKED);
-		set_state_private(&root->fs_info->pending_del,
-				  bytenr, (unsigned long)extent_op);
-		return 0;
+	/*
+	 * tree log blocks never actually go into the extent allocation
+	 * tree, just update pinning info and exit early.
+	 */
+	if (root_objectid == BTRFS_TREE_LOG_OBJECTID) {
+		printf("PINNING EXTENTS IN LOG TREE\n");
+		WARN_ON(owner >= BTRFS_FIRST_FREE_OBJECTID);
+		btrfs_pin_extent(trans->fs_info, bytenr, num_bytes);
+		ret = 0;
+	} else if (owner < BTRFS_FIRST_FREE_OBJECTID) {
+		BUG_ON(offset);
+		ret = btrfs_add_delayed_tree_ref(trans->fs_info, trans,
+						 bytenr, num_bytes, parent,
+						 root_objectid, (int)owner,
+						 BTRFS_DROP_DELAYED_REF,
+						 NULL, NULL, NULL);
+	} else {
+		ret = __free_extent(trans, bytenr, num_bytes, parent,
+				    root_objectid, owner, offset, 1);
 	}
-	ret = __free_extent(trans, bytenr, num_bytes, parent,
-			    root_objectid, owner, offset, 1);
-	pending_ret = del_pending_extents(trans);
-	return ret ? ret : pending_ret;
+	return ret;
 }
 
 static u64 stripe_align(struct btrfs_root *root, u64 val)
@@ -2694,6 +2684,8 @@ static int alloc_reserved_tree_block2(struct btrfs_trans_handle *trans,
 	struct btrfs_delayed_tree_ref *ref = btrfs_delayed_node_to_tree_ref(node);
 	struct btrfs_key ins;
 	bool skinny_metadata = btrfs_fs_incompat(trans->fs_info, SKINNY_METADATA);
+	int ret;
+	u64 start, end;
 
 	ins.objectid = node->bytenr;
 	if (skinny_metadata) {
@@ -2704,10 +2696,25 @@ static int alloc_reserved_tree_block2(struct btrfs_trans_handle *trans,
 		ins.type = BTRFS_EXTENT_ITEM_KEY;
 	}
 
-	return alloc_reserved_tree_block(trans, ref->root, trans->transid,
-					 extent_op->flags_to_set,
-					 &extent_op->key, ref->level, &ins);
+	if (ref->root == BTRFS_EXTENT_TREE_OBJECTID) {
+		ret = find_first_extent_bit(&trans->fs_info->extent_ins,
+					    node->bytenr, &start, &end,
+					    EXTENT_LOCKED);
+		ASSERT(!ret);
+		ASSERT(start == node->bytenr);
+		ASSERT(end == node->bytenr + node->num_bytes - 1);
+	}
+
+	ret = alloc_reserved_tree_block(trans, ref->root, trans->transid,
+					extent_op->flags_to_set,
+					&extent_op->key, ref->level, &ins);
 
+	if (ref->root == BTRFS_EXTENT_TREE_OBJECTID) {
+		clear_extent_bits(&trans->fs_info->extent_ins, start, end,
+				  EXTENT_LOCKED);
+	}
+
+	return ret;
 }
 
 static int alloc_reserved_tree_block(struct btrfs_trans_handle *trans,
@@ -2772,39 +2779,50 @@ static int alloc_tree_block(struct btrfs_trans_handle *trans,
 			    u64 search_end, struct btrfs_key *ins)
 {
 	int ret;
+	u64 extent_size;
+	struct btrfs_delayed_extent_op *extent_op;
+	bool skinny_metadata = btrfs_fs_incompat(root->fs_info,
+						 SKINNY_METADATA);
+
+	extent_op = btrfs_alloc_delayed_extent_op();
+	if (!extent_op)
+		return -ENOMEM;
+
 	ret = btrfs_reserve_extent(trans, root, num_bytes, empty_size,
 				   hint_byte, search_end, ins, 0);
 	BUG_ON(ret);
 
+	if (key)
+		memcpy(&extent_op->key, key, sizeof(extent_op->key));
+	else
+		memset(&extent_op->key, 0, sizeof(extent_op->key));
+	extent_op->flags_to_set = flags;
+	extent_op->update_key = skinny_metadata ? false : true;
+	extent_op->update_flags = true;
+	extent_op->is_data = false;
+	extent_op->level = level;
+
+	extent_size = ins->offset;
+
+	if (btrfs_fs_incompat(root->fs_info, SKINNY_METADATA)) {
+		ins->offset = level;
+		ins->type = BTRFS_METADATA_ITEM_KEY;
+	}
+
+	/* Ensure this reserved extent is not found by the allocator */
 	if (root_objectid == BTRFS_EXTENT_TREE_OBJECTID) {
-		struct pending_extent_op *extent_op;
-
-		extent_op = kmalloc(sizeof(*extent_op), GFP_NOFS);
-		BUG_ON(!extent_op);
-
-		extent_op->type = PENDING_EXTENT_INSERT;
-		extent_op->bytenr = ins->objectid;
-		extent_op->num_bytes = ins->offset;
-		extent_op->level = level;
-		extent_op->flags = flags;
-		memcpy(&extent_op->key, key, sizeof(*key));
-
-		set_extent_bits(&root->fs_info->extent_ins, ins->objectid,
-				ins->objectid + ins->offset - 1,
-				EXTENT_LOCKED);
-		set_state_private(&root->fs_info->extent_ins,
-				  ins->objectid, (unsigned long)extent_op);
-	} else {
-		if (btrfs_fs_incompat(root->fs_info, SKINNY_METADATA)) {
-			ins->offset = level;
-			ins->type = BTRFS_METADATA_ITEM_KEY;
-		}
-		ret = alloc_reserved_tree_block(trans, root_objectid,
-						generation, flags,
-						key, level, ins);
-		finish_current_insert(trans);
-		del_pending_extents(trans);
+		ret = set_extent_bits(&trans->fs_info->extent_ins,
+				      ins->objectid,
+				      ins->objectid + extent_size - 1,
+				      EXTENT_LOCKED);
+
+		BUG_ON(ret);
 	}
+
+	ret = btrfs_add_delayed_tree_ref(root->fs_info, trans, ins->objectid,
+					 extent_size, 0, root_objectid,
+					 level, BTRFS_ADD_DELAYED_EXTENT,
+					 extent_op, NULL, NULL);
 	return ret;
 }
 
@@ -3329,11 +3347,6 @@ int btrfs_make_block_group(struct btrfs_trans_handle *trans,
 				sizeof(cache->item));
 	BUG_ON(ret);
 
-	ret = finish_current_insert(trans);
-	BUG_ON(ret);
-	ret = del_pending_extents(trans);
-	BUG_ON(ret);
-
 	return 0;
 }
 
@@ -3429,10 +3442,6 @@ int btrfs_make_block_groups(struct btrfs_trans_handle *trans,
 					sizeof(cache->item));
 		BUG_ON(ret);
 
-		finish_current_insert(trans);
-		ret = del_pending_extents(trans);
-		BUG_ON(ret);
-
 		cur_start = cache->key.objectid + cache->key.offset;
 	}
 	return 0;
@@ -3814,14 +3823,9 @@ int btrfs_fix_block_accounting(struct btrfs_trans_handle *trans)
 	struct btrfs_fs_info *fs_info = trans->fs_info;
 	struct btrfs_root *root = fs_info->extent_root;
 
-	while(extent_root_pending_ops(fs_info)) {
-		ret = finish_current_insert(trans);
-		if (ret)
-			return ret;
-		ret = del_pending_extents(trans);
-		if (ret)
-			return ret;
-	}
+	ret = btrfs_run_delayed_refs(trans, -1);
+	if (ret)
+		return ret;
 
 	while(1) {
 		cache = btrfs_lookup_first_block_group(fs_info, start);
@@ -4026,7 +4030,7 @@ static int __btrfs_record_file_extent(struct btrfs_trans_handle *trans,
 		} else if (ret != -EEXIST) {
 			goto fail;
 		}
-		btrfs_extent_post_op(trans);
+		btrfs_run_delayed_refs(trans, -1);
 		extent_bytenr = disk_bytenr;
 		extent_num_bytes = num_bytes;
 		extent_offset = 0;
transaction.c +25 -2
index 96d9891b0d1c..bfda769210ee 100644
@@ -61,7 +61,6 @@ static int update_cowonly_root(struct btrfs_trans_handle *trans,
 	u64 old_root_bytenr;
 	struct btrfs_root *tree_root = root->fs_info->tree_root;
 
-	btrfs_write_dirty_block_groups(trans);
 	while(1) {
 		old_root_bytenr = btrfs_root_bytenr(&root->root_item);
 		if (old_root_bytenr == root->node->start)
@@ -98,6 +97,17 @@ int commit_tree_roots(struct btrfs_trans_handle *trans,
 	if (ret)
 		return ret;
 
+	/*
+	 * If the above CoW is the first one to dirty the current tree_root,
+	 * delayed refs for it won't be run until after this function has
+	 * finished executing, meaning we won't process the extent tree root,
+	 * which will have been added to ->dirty_cowonly_roots.  So run
+	 * delayed refs here as well.
+	 */
+	ret = btrfs_run_delayed_refs(trans, -1);
+	if (ret)
+		return ret;
+
 	while(!list_empty(&fs_info->dirty_cowonly_roots)) {
 		next = fs_info->dirty_cowonly_roots.next;
 		list_del_init(next);
@@ -147,6 +157,12 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 
 	if (trans->fs_info->transaction_aborted)
 		return -EROFS;
+	/*
+	 * Flush all accumulated delayed refs so that root-tree updates are
+	 * consistent
+	 */
+	ret = btrfs_run_delayed_refs(trans, -1);
+	BUG_ON(ret);
 
 	if (root->commit_root == root->node)
 		goto commit_tree;
@@ -164,11 +180,18 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 	ret = btrfs_update_root(trans, root->fs_info->tree_root,
 				&root->root_key, &root->root_item);
 	BUG_ON(ret);
+
 commit_tree:
 	ret = commit_tree_roots(trans, fs_info);
 	BUG_ON(ret);
-	ret = __commit_transaction(trans, root);
+	/*
+	 * Ensure that all comitted roots are properly accounted in the
+	 * extent tree
+	 */
+	ret = btrfs_run_delayed_refs(trans, -1);
 	BUG_ON(ret);
+	btrfs_write_dirty_block_groups(trans);
+	__commit_transaction(trans, root);
 	write_ctree_super(trans);
 	btrfs_finish_extent_commit(trans, fs_info->extent_root,
 			           &fs_info->pinned_extents);
#7 lore raw

Re: [PATCH 5/8] btrfs-progs: Wire up delayed refs

Quoted 26 lines · Qu Wenruo
Quoted 23 lines · Nikolay Borisov

This commit enables the delayed refs infrastructures. This entails doing the following:

1. Replacing existing calls of btrfs_extent_post_op (which is the equivalent of delayed refs) with the proper btrfs_run_delayed_refs. As well as eliminating open-coded calls to finish_current_insert and del_pending_extents which execute the delayed ops.

2. Wiring up the addition of delayed refs when freeing extents (btrfs_free_extent) and when adding new extents (alloc_tree_block).

3. Adding calls to btrfs_run_delayed refs in the transaction commit path alongside comments why every call is needed, since it's not always obvious (those call sites were derived empirically by running and debugging existing tests)

4. Correctly flagging the transaction in which we are reinitialising the extent tree.

5 Moving btrfs_write_dirty_block_groups to btrfs_write_dirty_block_groups since blockgroups should be written to disk after the last delayed refs have been run.

Quoted 7 lines
Quoted 2 lines

Is there something (maybe btrfs_run_delayed_refs()?) missing in btrfs-image?

btrfs-image from devel branch can't restore image correctly, the block group used bytes is not correct, thus it can't pass misc nor fsck tests.

This is really strange, all fsck/misc tests passed with those patches. Can you be more specific which tests exactly you mean ?

Quoted 384 lines

Thanks,
Qu

Quoted 379 lines
---
 check/main.c  |   3 +-
 extent-tree.c | 166 ++++++++++++++++++++++++++++++----------------------------
 transaction.c |  27 +++++++++-
 3 files changed, 112 insertions(+), 84 deletions(-)
check/main.c +2 -1
index bc2ee22f7943..b361cd7e26a0 100644
@@ -8710,7 +8710,7 @@ static int reinit_extent_tree(struct btrfs_trans_handle *trans,
 			fprintf(stderr, "Error adding block group\n");
 			return ret;
 		}
-		btrfs_extent_post_op(trans);
+		btrfs_run_delayed_refs(trans, -1);
 	}
 
 	ret = reset_balance(trans, fs_info);
@@ -9767,6 +9767,7 @@ int cmd_check(int argc, char **argv)
 			goto close_out;
 		}
 
+		trans->reinit_extent_tree = true;
 		if (init_extent_tree) {
 			printf("Creating a new extent tree\n");
 			ret = reinit_extent_tree(trans, info,
extent-tree.c +85 -81
index 7d6c37c6b371..2fa51bbc0359 100644
@@ -1418,8 +1418,6 @@ int btrfs_inc_extent_ref(struct btrfs_trans_handle *trans,
 		err = ret;
 out:
 	btrfs_free_path(path);
-	finish_current_insert(trans);
-	del_pending_extents(trans);
 	BUG_ON(err);
 	return err;
 }
@@ -1602,8 +1600,6 @@ int btrfs_set_block_flags(struct btrfs_trans_handle *trans, u64 bytenr,
 	btrfs_set_extent_flags(l, item, flags);
 out:
 	btrfs_free_path(path);
-	finish_current_insert(trans);
-	del_pending_extents(trans);
 	return ret;
 }
 
@@ -1701,7 +1697,6 @@ static int write_one_cache_group(struct btrfs_trans_handle *trans,
 				 struct btrfs_block_group_cache *cache)
 {
 	int ret;
-	int pending_ret;
 	struct btrfs_root *extent_root = trans->fs_info->extent_root;
 	unsigned long bi;
 	struct extent_buffer *leaf;
@@ -1717,12 +1712,8 @@ static int write_one_cache_group(struct btrfs_trans_handle *trans,
 	btrfs_mark_buffer_dirty(leaf);
 	btrfs_release_path(path);
 fail:
-	finish_current_insert(trans);
-	pending_ret = del_pending_extents(trans);
 	if (ret)
 		return ret;
-	if (pending_ret)
-		return pending_ret;
 	return 0;
 
 }
@@ -2049,6 +2040,7 @@ static int finish_current_insert(struct btrfs_trans_handle *trans)
 	int skinny_metadata =
 		btrfs_fs_incompat(extent_root->fs_info, SKINNY_METADATA);
 
+
 	while(1) {
 		ret = find_first_extent_bit(&info->extent_ins, 0, &start,
 					    &end, EXTENT_LOCKED);
@@ -2080,6 +2072,8 @@ static int finish_current_insert(struct btrfs_trans_handle *trans)
 			BUG_ON(1);
 		}
 
+
+		printf("shouldn't be executed\n");
 		clear_extent_bits(&info->extent_ins, start, end, EXTENT_LOCKED);
 		kfree(extent_op);
 	}
@@ -2379,7 +2373,6 @@ static int __free_extent(struct btrfs_trans_handle *trans,
 	}
 fail:
 	btrfs_free_path(path);
-	finish_current_insert(trans);
 	return ret;
 }
 
@@ -2462,33 +2455,30 @@ int btrfs_free_extent(struct btrfs_trans_handle *trans,
 		      u64 bytenr, u64 num_bytes, u64 parent,
 		      u64 root_objectid, u64 owner, u64 offset)
 {
-	struct btrfs_root *extent_root = root->fs_info->extent_root;
-	int pending_ret;
 	int ret;
 
 	WARN_ON(num_bytes < root->fs_info->sectorsize);
-	if (root == extent_root) {
-		struct pending_extent_op *extent_op;
-
-		extent_op = kmalloc(sizeof(*extent_op), GFP_NOFS);
-		BUG_ON(!extent_op);
-
-		extent_op->type = PENDING_EXTENT_DELETE;
-		extent_op->bytenr = bytenr;
-		extent_op->num_bytes = num_bytes;
-		extent_op->level = (int)owner;
-
-		set_extent_bits(&root->fs_info->pending_del,
-				bytenr, bytenr + num_bytes - 1,
-				EXTENT_LOCKED);
-		set_state_private(&root->fs_info->pending_del,
-				  bytenr, (unsigned long)extent_op);
-		return 0;
+	/*
+	 * tree log blocks never actually go into the extent allocation
+	 * tree, just update pinning info and exit early.
+	 */
+	if (root_objectid == BTRFS_TREE_LOG_OBJECTID) {
+		printf("PINNING EXTENTS IN LOG TREE\n");
+		WARN_ON(owner >= BTRFS_FIRST_FREE_OBJECTID);
+		btrfs_pin_extent(trans->fs_info, bytenr, num_bytes);
+		ret = 0;
+	} else if (owner < BTRFS_FIRST_FREE_OBJECTID) {
+		BUG_ON(offset);
+		ret = btrfs_add_delayed_tree_ref(trans->fs_info, trans,
+						 bytenr, num_bytes, parent,
+						 root_objectid, (int)owner,
+						 BTRFS_DROP_DELAYED_REF,
+						 NULL, NULL, NULL);
+	} else {
+		ret = __free_extent(trans, bytenr, num_bytes, parent,
+				    root_objectid, owner, offset, 1);
 	}
-	ret = __free_extent(trans, bytenr, num_bytes, parent,
-			    root_objectid, owner, offset, 1);
-	pending_ret = del_pending_extents(trans);
-	return ret ? ret : pending_ret;
+	return ret;
 }
 
 static u64 stripe_align(struct btrfs_root *root, u64 val)
@@ -2694,6 +2684,8 @@ static int alloc_reserved_tree_block2(struct btrfs_trans_handle *trans,
 	struct btrfs_delayed_tree_ref *ref = btrfs_delayed_node_to_tree_ref(node);
 	struct btrfs_key ins;
 	bool skinny_metadata = btrfs_fs_incompat(trans->fs_info, SKINNY_METADATA);
+	int ret;
+	u64 start, end;
 
 	ins.objectid = node->bytenr;
 	if (skinny_metadata) {
@@ -2704,10 +2696,25 @@ static int alloc_reserved_tree_block2(struct btrfs_trans_handle *trans,
 		ins.type = BTRFS_EXTENT_ITEM_KEY;
 	}
 
-	return alloc_reserved_tree_block(trans, ref->root, trans->transid,
-					 extent_op->flags_to_set,
-					 &extent_op->key, ref->level, &ins);
+	if (ref->root == BTRFS_EXTENT_TREE_OBJECTID) {
+		ret = find_first_extent_bit(&trans->fs_info->extent_ins,
+					    node->bytenr, &start, &end,
+					    EXTENT_LOCKED);
+		ASSERT(!ret);
+		ASSERT(start == node->bytenr);
+		ASSERT(end == node->bytenr + node->num_bytes - 1);
+	}
+
+	ret = alloc_reserved_tree_block(trans, ref->root, trans->transid,
+					extent_op->flags_to_set,
+					&extent_op->key, ref->level, &ins);
 
+	if (ref->root == BTRFS_EXTENT_TREE_OBJECTID) {
+		clear_extent_bits(&trans->fs_info->extent_ins, start, end,
+				  EXTENT_LOCKED);
+	}
+
+	return ret;
 }
 
 static int alloc_reserved_tree_block(struct btrfs_trans_handle *trans,
@@ -2772,39 +2779,50 @@ static int alloc_tree_block(struct btrfs_trans_handle *trans,
 			    u64 search_end, struct btrfs_key *ins)
 {
 	int ret;
+	u64 extent_size;
+	struct btrfs_delayed_extent_op *extent_op;
+	bool skinny_metadata = btrfs_fs_incompat(root->fs_info,
+						 SKINNY_METADATA);
+
+	extent_op = btrfs_alloc_delayed_extent_op();
+	if (!extent_op)
+		return -ENOMEM;
+
 	ret = btrfs_reserve_extent(trans, root, num_bytes, empty_size,
 				   hint_byte, search_end, ins, 0);
 	BUG_ON(ret);
 
+	if (key)
+		memcpy(&extent_op->key, key, sizeof(extent_op->key));
+	else
+		memset(&extent_op->key, 0, sizeof(extent_op->key));
+	extent_op->flags_to_set = flags;
+	extent_op->update_key = skinny_metadata ? false : true;
+	extent_op->update_flags = true;
+	extent_op->is_data = false;
+	extent_op->level = level;
+
+	extent_size = ins->offset;
+
+	if (btrfs_fs_incompat(root->fs_info, SKINNY_METADATA)) {
+		ins->offset = level;
+		ins->type = BTRFS_METADATA_ITEM_KEY;
+	}
+
+	/* Ensure this reserved extent is not found by the allocator */
 	if (root_objectid == BTRFS_EXTENT_TREE_OBJECTID) {
-		struct pending_extent_op *extent_op;
-
-		extent_op = kmalloc(sizeof(*extent_op), GFP_NOFS);
-		BUG_ON(!extent_op);
-
-		extent_op->type = PENDING_EXTENT_INSERT;
-		extent_op->bytenr = ins->objectid;
-		extent_op->num_bytes = ins->offset;
-		extent_op->level = level;
-		extent_op->flags = flags;
-		memcpy(&extent_op->key, key, sizeof(*key));
-
-		set_extent_bits(&root->fs_info->extent_ins, ins->objectid,
-				ins->objectid + ins->offset - 1,
-				EXTENT_LOCKED);
-		set_state_private(&root->fs_info->extent_ins,
-				  ins->objectid, (unsigned long)extent_op);
-	} else {
-		if (btrfs_fs_incompat(root->fs_info, SKINNY_METADATA)) {
-			ins->offset = level;
-			ins->type = BTRFS_METADATA_ITEM_KEY;
-		}
-		ret = alloc_reserved_tree_block(trans, root_objectid,
-						generation, flags,
-						key, level, ins);
-		finish_current_insert(trans);
-		del_pending_extents(trans);
+		ret = set_extent_bits(&trans->fs_info->extent_ins,
+				      ins->objectid,
+				      ins->objectid + extent_size - 1,
+				      EXTENT_LOCKED);
+
+		BUG_ON(ret);
 	}
+
+	ret = btrfs_add_delayed_tree_ref(root->fs_info, trans, ins->objectid,
+					 extent_size, 0, root_objectid,
+					 level, BTRFS_ADD_DELAYED_EXTENT,
+					 extent_op, NULL, NULL);
 	return ret;
 }
 
@@ -3329,11 +3347,6 @@ int btrfs_make_block_group(struct btrfs_trans_handle *trans,
 				sizeof(cache->item));
 	BUG_ON(ret);
 
-	ret = finish_current_insert(trans);
-	BUG_ON(ret);
-	ret = del_pending_extents(trans);
-	BUG_ON(ret);
-
 	return 0;
 }
 
@@ -3429,10 +3442,6 @@ int btrfs_make_block_groups(struct btrfs_trans_handle *trans,
 					sizeof(cache->item));
 		BUG_ON(ret);
 
-		finish_current_insert(trans);
-		ret = del_pending_extents(trans);
-		BUG_ON(ret);
-
 		cur_start = cache->key.objectid + cache->key.offset;
 	}
 	return 0;
@@ -3814,14 +3823,9 @@ int btrfs_fix_block_accounting(struct btrfs_trans_handle *trans)
 	struct btrfs_fs_info *fs_info = trans->fs_info;
 	struct btrfs_root *root = fs_info->extent_root;
 
-	while(extent_root_pending_ops(fs_info)) {
-		ret = finish_current_insert(trans);
-		if (ret)
-			return ret;
-		ret = del_pending_extents(trans);
-		if (ret)
-			return ret;
-	}
+	ret = btrfs_run_delayed_refs(trans, -1);
+	if (ret)
+		return ret;
 
 	while(1) {
 		cache = btrfs_lookup_first_block_group(fs_info, start);
@@ -4026,7 +4030,7 @@ static int __btrfs_record_file_extent(struct btrfs_trans_handle *trans,
 		} else if (ret != -EEXIST) {
 			goto fail;
 		}
-		btrfs_extent_post_op(trans);
+		btrfs_run_delayed_refs(trans, -1);
 		extent_bytenr = disk_bytenr;
 		extent_num_bytes = num_bytes;
 		extent_offset = 0;
transaction.c +25 -2
index 96d9891b0d1c..bfda769210ee 100644
@@ -61,7 +61,6 @@ static int update_cowonly_root(struct btrfs_trans_handle *trans,
 	u64 old_root_bytenr;
 	struct btrfs_root *tree_root = root->fs_info->tree_root;
 
-	btrfs_write_dirty_block_groups(trans);
 	while(1) {
 		old_root_bytenr = btrfs_root_bytenr(&root->root_item);
 		if (old_root_bytenr == root->node->start)
@@ -98,6 +97,17 @@ int commit_tree_roots(struct btrfs_trans_handle *trans,
 	if (ret)
 		return ret;
 
+	/*
+	 * If the above CoW is the first one to dirty the current tree_root,
+	 * delayed refs for it won't be run until after this function has
+	 * finished executing, meaning we won't process the extent tree root,
+	 * which will have been added to ->dirty_cowonly_roots.  So run
+	 * delayed refs here as well.
+	 */
+	ret = btrfs_run_delayed_refs(trans, -1);
+	if (ret)
+		return ret;
+
 	while(!list_empty(&fs_info->dirty_cowonly_roots)) {
 		next = fs_info->dirty_cowonly_roots.next;
 		list_del_init(next);
@@ -147,6 +157,12 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 
 	if (trans->fs_info->transaction_aborted)
 		return -EROFS;
+	/*
+	 * Flush all accumulated delayed refs so that root-tree updates are
+	 * consistent
+	 */
+	ret = btrfs_run_delayed_refs(trans, -1);
+	BUG_ON(ret);
 
 	if (root->commit_root == root->node)
 		goto commit_tree;
@@ -164,11 +180,18 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 	ret = btrfs_update_root(trans, root->fs_info->tree_root,
 				&root->root_key, &root->root_item);
 	BUG_ON(ret);
+
 commit_tree:
 	ret = commit_tree_roots(trans, fs_info);
 	BUG_ON(ret);
-	ret = __commit_transaction(trans, root);
+	/*
+	 * Ensure that all comitted roots are properly accounted in the
+	 * extent tree
+	 */
+	ret = btrfs_run_delayed_refs(trans, -1);
 	BUG_ON(ret);
+	btrfs_write_dirty_block_groups(trans);
+	__commit_transaction(trans, root);
 	write_ctree_super(trans);
 	btrfs_finish_extent_commit(trans, fs_info->extent_root,
 			           &fs_info->pinned_extents);
#8 lore raw

Re: [PATCH 5/8] btrfs-progs: Wire up delayed refs

Quoted 32 lines · Nikolay Borisov
Quoted 29 lines · Qu Wenruo
Quoted 25 lines · Nikolay Borisov

This commit enables the delayed refs infrastructures. This entails doing the following:

1. Replacing existing calls of btrfs_extent_post_op (which is the equivalent of delayed refs) with the proper btrfs_run_delayed_refs. As well as eliminating open-coded calls to finish_current_insert and del_pending_extents which execute the delayed ops.

2. Wiring up the addition of delayed refs when freeing extents (btrfs_free_extent) and when adding new extents (alloc_tree_block).

3. Adding calls to btrfs_run_delayed refs in the transaction commit path alongside comments why every call is needed, since it's not always obvious (those call sites were derived empirically by running and debugging existing tests)

4. Correctly flagging the transaction in which we are reinitialising the extent tree.

5 Moving btrfs_write_dirty_block_groups to btrfs_write_dirty_block_groups since blockgroups should be written to disk after the last delayed refs have been run.

Quoted 7 lines
Quoted 4 lines

Is there something (maybe btrfs_run_delayed_refs()?) missing in btrfs-image?

btrfs-image from devel branch can't restore image correctly, the block group used bytes is not correct, thus it can't pass misc nor fsck tests.

This is really strange, all fsck/misc tests passed with those patches. Can you be more specific which tests exactly you mean ?

One case is fsck/020 with lowmem mode. (Original mode lacks block group->used check).

More specifically, fsck/020/keyed_data_ref_with_shared_leaf.img

Using btrfs-image from my distribution (v4.17.1) and devel branch btrfs
check: (cwd is btrfs-progs, devel branch)

$ btrfs-image -r
tests/fsck-tests/020-extent-ref-cases/keyed_data_ref_with_shared_leaf.img ~/test.img
$ btrfs check --mode=lowmem ~/test.img
Opening filesystem to check...
Checking filesystem on /home/adam/test.img
UUID: 12dabcf2-d4da-4a70-9701-9f3d48074e73
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs done with fs roots in lowmem mode, skipping
[7/7] checking quota groups skipped (not enabled on this FS)
found 1208320 bytes used, no error found
total csum bytes: 512
total tree bytes: 684032
total fs tree bytes: 638976
total extent tree bytes: 16384
btree space waste bytes: 305606
file data blocks allocated: 93847552
 referenced 1773568

But if using btrfs-image with your delayed ref patch:

$ ./btrfs-image -r
tests/fsck-tests/020-extent-ref-cases/keyed_data_ref_with_shared_leaf.img ~/test.img

# No matter if I'm using btrfs-check from devel or 4.17.1

$ btrfs check --mode=lowmem ~/test.img
Opening filesystem to check...
Checking filesystem on /home/adam/test.img
UUID: 12dabcf2-d4da-4a70-9701-9f3d48074e73
[1/7] checking root items
[2/7] checking extents
ERROR: block group[4194304 8388608] used 20480 but extent items used 24576
ERROR: block group[20971520 16777216] used 659456 but extent items used
655360
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs done with fs roots in lowmem mode, skipping
[7/7] checking quota groups skipped (not enabled on this FS)
found 1208320 bytes used, error(s) found
total csum bytes: 512
total tree bytes: 684032
total fs tree bytes: 638976
total extent tree bytes: 16384
btree space waste bytes: 305606
file data blocks allocated: 93847552
 referenced 1773568

I'd say, although lowmem check is still far from perfect, it indeed has extra checks original mode lacks, and in this case it indeed exposes problem.

Thanks,
Qu

Quoted 385 lines
Quoted 385 lines

Thanks,
Qu

Quoted 379 lines
---
 check/main.c  |   3 +-
 extent-tree.c | 166 ++++++++++++++++++++++++++++++----------------------------
 transaction.c |  27 +++++++++-
 3 files changed, 112 insertions(+), 84 deletions(-)
check/main.c +2 -1
index bc2ee22f7943..b361cd7e26a0 100644
@@ -8710,7 +8710,7 @@ static int reinit_extent_tree(struct btrfs_trans_handle *trans,
 			fprintf(stderr, "Error adding block group\n");
 			return ret;
 		}
-		btrfs_extent_post_op(trans);
+		btrfs_run_delayed_refs(trans, -1);
 	}
 
 	ret = reset_balance(trans, fs_info);
@@ -9767,6 +9767,7 @@ int cmd_check(int argc, char **argv)
 			goto close_out;
 		}
 
+		trans->reinit_extent_tree = true;
 		if (init_extent_tree) {
 			printf("Creating a new extent tree\n");
 			ret = reinit_extent_tree(trans, info,
extent-tree.c +85 -81
index 7d6c37c6b371..2fa51bbc0359 100644
@@ -1418,8 +1418,6 @@ int btrfs_inc_extent_ref(struct btrfs_trans_handle *trans,
 		err = ret;
 out:
 	btrfs_free_path(path);
-	finish_current_insert(trans);
-	del_pending_extents(trans);
 	BUG_ON(err);
 	return err;
 }
@@ -1602,8 +1600,6 @@ int btrfs_set_block_flags(struct btrfs_trans_handle *trans, u64 bytenr,
 	btrfs_set_extent_flags(l, item, flags);
 out:
 	btrfs_free_path(path);
-	finish_current_insert(trans);
-	del_pending_extents(trans);
 	return ret;
 }
 
@@ -1701,7 +1697,6 @@ static int write_one_cache_group(struct btrfs_trans_handle *trans,
 				 struct btrfs_block_group_cache *cache)
 {
 	int ret;
-	int pending_ret;
 	struct btrfs_root *extent_root = trans->fs_info->extent_root;
 	unsigned long bi;
 	struct extent_buffer *leaf;
@@ -1717,12 +1712,8 @@ static int write_one_cache_group(struct btrfs_trans_handle *trans,
 	btrfs_mark_buffer_dirty(leaf);
 	btrfs_release_path(path);
 fail:
-	finish_current_insert(trans);
-	pending_ret = del_pending_extents(trans);
 	if (ret)
 		return ret;
-	if (pending_ret)
-		return pending_ret;
 	return 0;
 
 }
@@ -2049,6 +2040,7 @@ static int finish_current_insert(struct btrfs_trans_handle *trans)
 	int skinny_metadata =
 		btrfs_fs_incompat(extent_root->fs_info, SKINNY_METADATA);
 
+
 	while(1) {
 		ret = find_first_extent_bit(&info->extent_ins, 0, &start,
 					    &end, EXTENT_LOCKED);
@@ -2080,6 +2072,8 @@ static int finish_current_insert(struct btrfs_trans_handle *trans)
 			BUG_ON(1);
 		}
 
+
+		printf("shouldn't be executed\n");
 		clear_extent_bits(&info->extent_ins, start, end, EXTENT_LOCKED);
 		kfree(extent_op);
 	}
@@ -2379,7 +2373,6 @@ static int __free_extent(struct btrfs_trans_handle *trans,
 	}
 fail:
 	btrfs_free_path(path);
-	finish_current_insert(trans);
 	return ret;
 }
 
@@ -2462,33 +2455,30 @@ int btrfs_free_extent(struct btrfs_trans_handle *trans,
 		      u64 bytenr, u64 num_bytes, u64 parent,
 		      u64 root_objectid, u64 owner, u64 offset)
 {
-	struct btrfs_root *extent_root = root->fs_info->extent_root;
-	int pending_ret;
 	int ret;
 
 	WARN_ON(num_bytes < root->fs_info->sectorsize);
-	if (root == extent_root) {
-		struct pending_extent_op *extent_op;
-
-		extent_op = kmalloc(sizeof(*extent_op), GFP_NOFS);
-		BUG_ON(!extent_op);
-
-		extent_op->type = PENDING_EXTENT_DELETE;
-		extent_op->bytenr = bytenr;
-		extent_op->num_bytes = num_bytes;
-		extent_op->level = (int)owner;
-
-		set_extent_bits(&root->fs_info->pending_del,
-				bytenr, bytenr + num_bytes - 1,
-				EXTENT_LOCKED);
-		set_state_private(&root->fs_info->pending_del,
-				  bytenr, (unsigned long)extent_op);
-		return 0;
+	/*
+	 * tree log blocks never actually go into the extent allocation
+	 * tree, just update pinning info and exit early.
+	 */
+	if (root_objectid == BTRFS_TREE_LOG_OBJECTID) {
+		printf("PINNING EXTENTS IN LOG TREE\n");
+		WARN_ON(owner >= BTRFS_FIRST_FREE_OBJECTID);
+		btrfs_pin_extent(trans->fs_info, bytenr, num_bytes);
+		ret = 0;
+	} else if (owner < BTRFS_FIRST_FREE_OBJECTID) {
+		BUG_ON(offset);
+		ret = btrfs_add_delayed_tree_ref(trans->fs_info, trans,
+						 bytenr, num_bytes, parent,
+						 root_objectid, (int)owner,
+						 BTRFS_DROP_DELAYED_REF,
+						 NULL, NULL, NULL);
+	} else {
+		ret = __free_extent(trans, bytenr, num_bytes, parent,
+				    root_objectid, owner, offset, 1);
 	}
-	ret = __free_extent(trans, bytenr, num_bytes, parent,
-			    root_objectid, owner, offset, 1);
-	pending_ret = del_pending_extents(trans);
-	return ret ? ret : pending_ret;
+	return ret;
 }
 
 static u64 stripe_align(struct btrfs_root *root, u64 val)
@@ -2694,6 +2684,8 @@ static int alloc_reserved_tree_block2(struct btrfs_trans_handle *trans,
 	struct btrfs_delayed_tree_ref *ref = btrfs_delayed_node_to_tree_ref(node);
 	struct btrfs_key ins;
 	bool skinny_metadata = btrfs_fs_incompat(trans->fs_info, SKINNY_METADATA);
+	int ret;
+	u64 start, end;
 
 	ins.objectid = node->bytenr;
 	if (skinny_metadata) {
@@ -2704,10 +2696,25 @@ static int alloc_reserved_tree_block2(struct btrfs_trans_handle *trans,
 		ins.type = BTRFS_EXTENT_ITEM_KEY;
 	}
 
-	return alloc_reserved_tree_block(trans, ref->root, trans->transid,
-					 extent_op->flags_to_set,
-					 &extent_op->key, ref->level, &ins);
+	if (ref->root == BTRFS_EXTENT_TREE_OBJECTID) {
+		ret = find_first_extent_bit(&trans->fs_info->extent_ins,
+					    node->bytenr, &start, &end,
+					    EXTENT_LOCKED);
+		ASSERT(!ret);
+		ASSERT(start == node->bytenr);
+		ASSERT(end == node->bytenr + node->num_bytes - 1);
+	}
+
+	ret = alloc_reserved_tree_block(trans, ref->root, trans->transid,
+					extent_op->flags_to_set,
+					&extent_op->key, ref->level, &ins);
 
+	if (ref->root == BTRFS_EXTENT_TREE_OBJECTID) {
+		clear_extent_bits(&trans->fs_info->extent_ins, start, end,
+				  EXTENT_LOCKED);
+	}
+
+	return ret;
 }
 
 static int alloc_reserved_tree_block(struct btrfs_trans_handle *trans,
@@ -2772,39 +2779,50 @@ static int alloc_tree_block(struct btrfs_trans_handle *trans,
 			    u64 search_end, struct btrfs_key *ins)
 {
 	int ret;
+	u64 extent_size;
+	struct btrfs_delayed_extent_op *extent_op;
+	bool skinny_metadata = btrfs_fs_incompat(root->fs_info,
+						 SKINNY_METADATA);
+
+	extent_op = btrfs_alloc_delayed_extent_op();
+	if (!extent_op)
+		return -ENOMEM;
+
 	ret = btrfs_reserve_extent(trans, root, num_bytes, empty_size,
 				   hint_byte, search_end, ins, 0);
 	BUG_ON(ret);
 
+	if (key)
+		memcpy(&extent_op->key, key, sizeof(extent_op->key));
+	else
+		memset(&extent_op->key, 0, sizeof(extent_op->key));
+	extent_op->flags_to_set = flags;
+	extent_op->update_key = skinny_metadata ? false : true;
+	extent_op->update_flags = true;
+	extent_op->is_data = false;
+	extent_op->level = level;
+
+	extent_size = ins->offset;
+
+	if (btrfs_fs_incompat(root->fs_info, SKINNY_METADATA)) {
+		ins->offset = level;
+		ins->type = BTRFS_METADATA_ITEM_KEY;
+	}
+
+	/* Ensure this reserved extent is not found by the allocator */
 	if (root_objectid == BTRFS_EXTENT_TREE_OBJECTID) {
-		struct pending_extent_op *extent_op;
-
-		extent_op = kmalloc(sizeof(*extent_op), GFP_NOFS);
-		BUG_ON(!extent_op);
-
-		extent_op->type = PENDING_EXTENT_INSERT;
-		extent_op->bytenr = ins->objectid;
-		extent_op->num_bytes = ins->offset;
-		extent_op->level = level;
-		extent_op->flags = flags;
-		memcpy(&extent_op->key, key, sizeof(*key));
-
-		set_extent_bits(&root->fs_info->extent_ins, ins->objectid,
-				ins->objectid + ins->offset - 1,
-				EXTENT_LOCKED);
-		set_state_private(&root->fs_info->extent_ins,
-				  ins->objectid, (unsigned long)extent_op);
-	} else {
-		if (btrfs_fs_incompat(root->fs_info, SKINNY_METADATA)) {
-			ins->offset = level;
-			ins->type = BTRFS_METADATA_ITEM_KEY;
-		}
-		ret = alloc_reserved_tree_block(trans, root_objectid,
-						generation, flags,
-						key, level, ins);
-		finish_current_insert(trans);
-		del_pending_extents(trans);
+		ret = set_extent_bits(&trans->fs_info->extent_ins,
+				      ins->objectid,
+				      ins->objectid + extent_size - 1,
+				      EXTENT_LOCKED);
+
+		BUG_ON(ret);
 	}
+
+	ret = btrfs_add_delayed_tree_ref(root->fs_info, trans, ins->objectid,
+					 extent_size, 0, root_objectid,
+					 level, BTRFS_ADD_DELAYED_EXTENT,
+					 extent_op, NULL, NULL);
 	return ret;
 }
 
@@ -3329,11 +3347,6 @@ int btrfs_make_block_group(struct btrfs_trans_handle *trans,
 				sizeof(cache->item));
 	BUG_ON(ret);
 
-	ret = finish_current_insert(trans);
-	BUG_ON(ret);
-	ret = del_pending_extents(trans);
-	BUG_ON(ret);
-
 	return 0;
 }
 
@@ -3429,10 +3442,6 @@ int btrfs_make_block_groups(struct btrfs_trans_handle *trans,
 					sizeof(cache->item));
 		BUG_ON(ret);
 
-		finish_current_insert(trans);
-		ret = del_pending_extents(trans);
-		BUG_ON(ret);
-
 		cur_start = cache->key.objectid + cache->key.offset;
 	}
 	return 0;
@@ -3814,14 +3823,9 @@ int btrfs_fix_block_accounting(struct btrfs_trans_handle *trans)
 	struct btrfs_fs_info *fs_info = trans->fs_info;
 	struct btrfs_root *root = fs_info->extent_root;
 
-	while(extent_root_pending_ops(fs_info)) {
-		ret = finish_current_insert(trans);
-		if (ret)
-			return ret;
-		ret = del_pending_extents(trans);
-		if (ret)
-			return ret;
-	}
+	ret = btrfs_run_delayed_refs(trans, -1);
+	if (ret)
+		return ret;
 
 	while(1) {
 		cache = btrfs_lookup_first_block_group(fs_info, start);
@@ -4026,7 +4030,7 @@ static int __btrfs_record_file_extent(struct btrfs_trans_handle *trans,
 		} else if (ret != -EEXIST) {
 			goto fail;
 		}
-		btrfs_extent_post_op(trans);
+		btrfs_run_delayed_refs(trans, -1);
 		extent_bytenr = disk_bytenr;
 		extent_num_bytes = num_bytes;
 		extent_offset = 0;
transaction.c +25 -2
index 96d9891b0d1c..bfda769210ee 100644
@@ -61,7 +61,6 @@ static int update_cowonly_root(struct btrfs_trans_handle *trans,
 	u64 old_root_bytenr;
 	struct btrfs_root *tree_root = root->fs_info->tree_root;
 
-	btrfs_write_dirty_block_groups(trans);
 	while(1) {
 		old_root_bytenr = btrfs_root_bytenr(&root->root_item);
 		if (old_root_bytenr == root->node->start)
@@ -98,6 +97,17 @@ int commit_tree_roots(struct btrfs_trans_handle *trans,
 	if (ret)
 		return ret;
 
+	/*
+	 * If the above CoW is the first one to dirty the current tree_root,
+	 * delayed refs for it won't be run until after this function has
+	 * finished executing, meaning we won't process the extent tree root,
+	 * which will have been added to ->dirty_cowonly_roots.  So run
+	 * delayed refs here as well.
+	 */
+	ret = btrfs_run_delayed_refs(trans, -1);
+	if (ret)
+		return ret;
+
 	while(!list_empty(&fs_info->dirty_cowonly_roots)) {
 		next = fs_info->dirty_cowonly_roots.next;
 		list_del_init(next);
@@ -147,6 +157,12 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 
 	if (trans->fs_info->transaction_aborted)
 		return -EROFS;
+	/*
+	 * Flush all accumulated delayed refs so that root-tree updates are
+	 * consistent
+	 */
+	ret = btrfs_run_delayed_refs(trans, -1);
+	BUG_ON(ret);
 
 	if (root->commit_root == root->node)
 		goto commit_tree;
@@ -164,11 +180,18 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 	ret = btrfs_update_root(trans, root->fs_info->tree_root,
 				&root->root_key, &root->root_item);
 	BUG_ON(ret);
+
 commit_tree:
 	ret = commit_tree_roots(trans, fs_info);
 	BUG_ON(ret);
-	ret = __commit_transaction(trans, root);
+	/*
+	 * Ensure that all comitted roots are properly accounted in the
+	 * extent tree
+	 */
+	ret = btrfs_run_delayed_refs(trans, -1);
 	BUG_ON(ret);
+	btrfs_write_dirty_block_groups(trans);
+	__commit_transaction(trans, root);
 	write_ctree_super(trans);
 	btrfs_finish_extent_commit(trans, fs_info->extent_root,
 			           &fs_info->pinned_extents);
#9 lore raw

Re: [PATCH 5/8] btrfs-progs: Wire up delayed refs

Quoted 102 lines · Qu Wenruo
Quoted 32 lines · Nikolay Borisov
Quoted 29 lines · Qu Wenruo
Quoted 25 lines · Nikolay Borisov

This commit enables the delayed refs infrastructures. This entails doing the following:

1. Replacing existing calls of btrfs_extent_post_op (which is the equivalent of delayed refs) with the proper btrfs_run_delayed_refs. As well as eliminating open-coded calls to finish_current_insert and del_pending_extents which execute the delayed ops.

2. Wiring up the addition of delayed refs when freeing extents (btrfs_free_extent) and when adding new extents (alloc_tree_block).

3. Adding calls to btrfs_run_delayed refs in the transaction commit path alongside comments why every call is needed, since it's not always obvious (those call sites were derived empirically by running and debugging existing tests)

4. Correctly flagging the transaction in which we are reinitialising the extent tree.

5 Moving btrfs_write_dirty_block_groups to btrfs_write_dirty_block_groups since blockgroups should be written to disk after the last delayed refs have been run.

Quoted 7 lines
Quoted 4 lines

Is there something (maybe btrfs_run_delayed_refs()?) missing in btrfs-image?

btrfs-image from devel branch can't restore image correctly, the block group used bytes is not correct, thus it can't pass misc nor fsck tests.

This is really strange, all fsck/misc tests passed with those patches. Can you be more specific which tests exactly you mean ?

One case is fsck/020 with lowmem mode. (Original mode lacks block group->used check).

More specifically, fsck/020/keyed_data_ref_with_shared_leaf.img

Using btrfs-image from my distribution (v4.17.1) and devel branch btrfs
check: (cwd is btrfs-progs, devel branch)

$ btrfs-image -r
tests/fsck-tests/020-extent-ref-cases/keyed_data_ref_with_shared_leaf.img ~/test.img
$ btrfs check --mode=lowmem ~/test.img
Opening filesystem to check...
Checking filesystem on /home/adam/test.img
UUID: 12dabcf2-d4da-4a70-9701-9f3d48074e73
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs done with fs roots in lowmem mode, skipping
[7/7] checking quota groups skipped (not enabled on this FS)
found 1208320 bytes used, no error found
total csum bytes: 512
total tree bytes: 684032
total fs tree bytes: 638976
total extent tree bytes: 16384
btree space waste bytes: 305606
file data blocks allocated: 93847552
 referenced 1773568

But if using btrfs-image with your delayed ref patch:

$ ./btrfs-image -r
tests/fsck-tests/020-extent-ref-cases/keyed_data_ref_with_shared_leaf.img ~/test.img

# No matter if I'm using btrfs-check from devel or 4.17.1

$ btrfs check --mode=lowmem ~/test.img
Opening filesystem to check...
Checking filesystem on /home/adam/test.img
UUID: 12dabcf2-d4da-4a70-9701-9f3d48074e73
[1/7] checking root items
[2/7] checking extents
ERROR: block group[4194304 8388608] used 20480 but extent items used 24576
ERROR: block group[20971520 16777216] used 659456 but extent items used
655360
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs done with fs roots in lowmem mode, skipping
[7/7] checking quota groups skipped (not enabled on this FS)
found 1208320 bytes used, error(s) found
total csum bytes: 512
total tree bytes: 684032
total fs tree bytes: 638976
total extent tree bytes: 16384
btree space waste bytes: 305606
file data blocks allocated: 93847552
 referenced 1773568
Quoted 3 lines

I'd say, although lowmem check is still far from perfect, it indeed has extra checks original mode lacks, and in this case it indeed exposes problem.

I'm not able to reproduce it:

make TEST_ENABLE_OVERRIDE=true TEST_ARGS_CHECK="--mode=lowmem" test-fsck

    [TEST]   fsck-tests.sh
    [TEST/fsck]   001-bad-file-extent-bytenr
    [TEST/fsck]   002-bad-transid
    [TEST/fsck]   003-shift-offsets
    [TEST/fsck]   004-no-dir-index
    [TEST/fsck]   005-bad-item-offset
    [TEST/fsck]   006-bad-root-items
    [TEST/fsck]   007-bad-offset-snapshots
    [TEST/fsck]   008-bad-dir-index-name
    [TEST/fsck]   009-no-dir-item-or-index
    [TEST/fsck]   010-no-rootdir-inode-item
    [TEST/fsck]   011-no-inode-item
    [TEST/fsck]   012-leaf-corruption
    [TEST/fsck]   013-extent-tree-rebuild
    [TEST/fsck]   014-no-extent-info
    [TEST/fsck]   015-tree-reloc-tree
    [TEST/fsck]   016-wrong-inode-nbytes
    [TEST/fsck]   017-missing-all-file-extent
    [TEST/fsck]   018-leaf-crossing-stripes
    [TEST/fsck]   019-non-skinny-false-alert
    [TEST/fsck]   020-extent-ref-cases
    [TEST/fsck]   021-partially-dropped-snapshot-case
    [TEST/fsck]   022-qgroup-rescan-halfway
    [TEST/fsck]   023-qgroup-stack-overflow
    [TEST/fsck]   024-clear-space-cache
    [TEST/fsck]   025-file-extents
    [TEST/fsck]   026-bad-dir-item-name
    [TEST/fsck]   027-bad-extent-inline-ref-type
    [TEST/fsck]   028-unaligned-super-dev-sizes
    [TEST/fsck]   029-valid-orphan-item
    [TEST/fsck]   030-reflinked-prealloc-extents
    [TEST/fsck]   031-metadatadump-check-data-csum
    [TEST/fsck]   032-corrupted-qgroup
    [TEST/fsck]   032-freespacetree-corrupted-extent-offset
    [TEST/fsck]   033-lowmem-collission-dir-items
    [TEST/fsck]   034-bad-inode-flags
    [TEST/fsck]   035-inline-bad-ram-bytes
    [TEST/fsck]   035-rescan-not-kicked-in

git bl -9
dac6a0a6f5ca btrfs-progs: Merge alloc_reserved_tree_block(2|) (3 weeks ago) <Nikolay Borisov> da184a3838e6 btrfs-progs: Remove __free_extent2 (3 weeks ago) <Nikolay Borisov>
76b75e40606e btrfs-progs: Remove old delayed refs infrastructure (3 months ago) <Nikolay Borisov> 870c58e08673 btrfs-progs: Wire up delayed refs (3 months ago) <Nikolay Borisov>
e045b219e397 btrfs-progs: Make btrfs_write_dirty_block_groups take only trans argument (3 weeks ago) <Nikolay Borisov> 00f8d76c8a27 btrfs-progs: Add delayed refs infrastructure (3 months ago) <Nikolay Borisov> d03f79e8deb6 btrfs-progs: Add alloc_reserved_tree_block2 function (3 months ago) <Nikolay Borisov> d4a4831fffe9 btrfs-progs: Add __free_extent2 function (3 months ago) <Nikolay Borisov>
7faaca0d9f78 Btrfs progs v4.17.1 (4 weeks ago) <David Sterba>

Are you sure your branch is not dirty with other changes?

Quoted 391 lines

Thanks,
Qu

Quoted 385 lines
Quoted 385 lines

Thanks,
Qu

Quoted 379 lines
---
 check/main.c  |   3 +-
 extent-tree.c | 166 ++++++++++++++++++++++++++++++----------------------------
 transaction.c |  27 +++++++++-
 3 files changed, 112 insertions(+), 84 deletions(-)
check/main.c +2 -1
index bc2ee22f7943..b361cd7e26a0 100644
@@ -8710,7 +8710,7 @@ static int reinit_extent_tree(struct btrfs_trans_handle *trans,
 			fprintf(stderr, "Error adding block group\n");
 			return ret;
 		}
-		btrfs_extent_post_op(trans);
+		btrfs_run_delayed_refs(trans, -1);
 	}
 
 	ret = reset_balance(trans, fs_info);
@@ -9767,6 +9767,7 @@ int cmd_check(int argc, char **argv)
 			goto close_out;
 		}
 
+		trans->reinit_extent_tree = true;
 		if (init_extent_tree) {
 			printf("Creating a new extent tree\n");
 			ret = reinit_extent_tree(trans, info,
extent-tree.c +85 -81
index 7d6c37c6b371..2fa51bbc0359 100644
@@ -1418,8 +1418,6 @@ int btrfs_inc_extent_ref(struct btrfs_trans_handle *trans,
 		err = ret;
 out:
 	btrfs_free_path(path);
-	finish_current_insert(trans);
-	del_pending_extents(trans);
 	BUG_ON(err);
 	return err;
 }
@@ -1602,8 +1600,6 @@ int btrfs_set_block_flags(struct btrfs_trans_handle *trans, u64 bytenr,
 	btrfs_set_extent_flags(l, item, flags);
 out:
 	btrfs_free_path(path);
-	finish_current_insert(trans);
-	del_pending_extents(trans);
 	return ret;
 }
 
@@ -1701,7 +1697,6 @@ static int write_one_cache_group(struct btrfs_trans_handle *trans,
 				 struct btrfs_block_group_cache *cache)
 {
 	int ret;
-	int pending_ret;
 	struct btrfs_root *extent_root = trans->fs_info->extent_root;
 	unsigned long bi;
 	struct extent_buffer *leaf;
@@ -1717,12 +1712,8 @@ static int write_one_cache_group(struct btrfs_trans_handle *trans,
 	btrfs_mark_buffer_dirty(leaf);
 	btrfs_release_path(path);
 fail:
-	finish_current_insert(trans);
-	pending_ret = del_pending_extents(trans);
 	if (ret)
 		return ret;
-	if (pending_ret)
-		return pending_ret;
 	return 0;
 
 }
@@ -2049,6 +2040,7 @@ static int finish_current_insert(struct btrfs_trans_handle *trans)
 	int skinny_metadata =
 		btrfs_fs_incompat(extent_root->fs_info, SKINNY_METADATA);
 
+
 	while(1) {
 		ret = find_first_extent_bit(&info->extent_ins, 0, &start,
 					    &end, EXTENT_LOCKED);
@@ -2080,6 +2072,8 @@ static int finish_current_insert(struct btrfs_trans_handle *trans)
 			BUG_ON(1);
 		}
 
+
+		printf("shouldn't be executed\n");
 		clear_extent_bits(&info->extent_ins, start, end, EXTENT_LOCKED);
 		kfree(extent_op);
 	}
@@ -2379,7 +2373,6 @@ static int __free_extent(struct btrfs_trans_handle *trans,
 	}
 fail:
 	btrfs_free_path(path);
-	finish_current_insert(trans);
 	return ret;
 }
 
@@ -2462,33 +2455,30 @@ int btrfs_free_extent(struct btrfs_trans_handle *trans,
 		      u64 bytenr, u64 num_bytes, u64 parent,
 		      u64 root_objectid, u64 owner, u64 offset)
 {
-	struct btrfs_root *extent_root = root->fs_info->extent_root;
-	int pending_ret;
 	int ret;
 
 	WARN_ON(num_bytes < root->fs_info->sectorsize);
-	if (root == extent_root) {
-		struct pending_extent_op *extent_op;
-
-		extent_op = kmalloc(sizeof(*extent_op), GFP_NOFS);
-		BUG_ON(!extent_op);
-
-		extent_op->type = PENDING_EXTENT_DELETE;
-		extent_op->bytenr = bytenr;
-		extent_op->num_bytes = num_bytes;
-		extent_op->level = (int)owner;
-
-		set_extent_bits(&root->fs_info->pending_del,
-				bytenr, bytenr + num_bytes - 1,
-				EXTENT_LOCKED);
-		set_state_private(&root->fs_info->pending_del,
-				  bytenr, (unsigned long)extent_op);
-		return 0;
+	/*
+	 * tree log blocks never actually go into the extent allocation
+	 * tree, just update pinning info and exit early.
+	 */
+	if (root_objectid == BTRFS_TREE_LOG_OBJECTID) {
+		printf("PINNING EXTENTS IN LOG TREE\n");
+		WARN_ON(owner >= BTRFS_FIRST_FREE_OBJECTID);
+		btrfs_pin_extent(trans->fs_info, bytenr, num_bytes);
+		ret = 0;
+	} else if (owner < BTRFS_FIRST_FREE_OBJECTID) {
+		BUG_ON(offset);
+		ret = btrfs_add_delayed_tree_ref(trans->fs_info, trans,
+						 bytenr, num_bytes, parent,
+						 root_objectid, (int)owner,
+						 BTRFS_DROP_DELAYED_REF,
+						 NULL, NULL, NULL);
+	} else {
+		ret = __free_extent(trans, bytenr, num_bytes, parent,
+				    root_objectid, owner, offset, 1);
 	}
-	ret = __free_extent(trans, bytenr, num_bytes, parent,
-			    root_objectid, owner, offset, 1);
-	pending_ret = del_pending_extents(trans);
-	return ret ? ret : pending_ret;
+	return ret;
 }
 
 static u64 stripe_align(struct btrfs_root *root, u64 val)
@@ -2694,6 +2684,8 @@ static int alloc_reserved_tree_block2(struct btrfs_trans_handle *trans,
 	struct btrfs_delayed_tree_ref *ref = btrfs_delayed_node_to_tree_ref(node);
 	struct btrfs_key ins;
 	bool skinny_metadata = btrfs_fs_incompat(trans->fs_info, SKINNY_METADATA);
+	int ret;
+	u64 start, end;
 
 	ins.objectid = node->bytenr;
 	if (skinny_metadata) {
@@ -2704,10 +2696,25 @@ static int alloc_reserved_tree_block2(struct btrfs_trans_handle *trans,
 		ins.type = BTRFS_EXTENT_ITEM_KEY;
 	}
 
-	return alloc_reserved_tree_block(trans, ref->root, trans->transid,
-					 extent_op->flags_to_set,
-					 &extent_op->key, ref->level, &ins);
+	if (ref->root == BTRFS_EXTENT_TREE_OBJECTID) {
+		ret = find_first_extent_bit(&trans->fs_info->extent_ins,
+					    node->bytenr, &start, &end,
+					    EXTENT_LOCKED);
+		ASSERT(!ret);
+		ASSERT(start == node->bytenr);
+		ASSERT(end == node->bytenr + node->num_bytes - 1);
+	}
+
+	ret = alloc_reserved_tree_block(trans, ref->root, trans->transid,
+					extent_op->flags_to_set,
+					&extent_op->key, ref->level, &ins);
 
+	if (ref->root == BTRFS_EXTENT_TREE_OBJECTID) {
+		clear_extent_bits(&trans->fs_info->extent_ins, start, end,
+				  EXTENT_LOCKED);
+	}
+
+	return ret;
 }
 
 static int alloc_reserved_tree_block(struct btrfs_trans_handle *trans,
@@ -2772,39 +2779,50 @@ static int alloc_tree_block(struct btrfs_trans_handle *trans,
 			    u64 search_end, struct btrfs_key *ins)
 {
 	int ret;
+	u64 extent_size;
+	struct btrfs_delayed_extent_op *extent_op;
+	bool skinny_metadata = btrfs_fs_incompat(root->fs_info,
+						 SKINNY_METADATA);
+
+	extent_op = btrfs_alloc_delayed_extent_op();
+	if (!extent_op)
+		return -ENOMEM;
+
 	ret = btrfs_reserve_extent(trans, root, num_bytes, empty_size,
 				   hint_byte, search_end, ins, 0);
 	BUG_ON(ret);
 
+	if (key)
+		memcpy(&extent_op->key, key, sizeof(extent_op->key));
+	else
+		memset(&extent_op->key, 0, sizeof(extent_op->key));
+	extent_op->flags_to_set = flags;
+	extent_op->update_key = skinny_metadata ? false : true;
+	extent_op->update_flags = true;
+	extent_op->is_data = false;
+	extent_op->level = level;
+
+	extent_size = ins->offset;
+
+	if (btrfs_fs_incompat(root->fs_info, SKINNY_METADATA)) {
+		ins->offset = level;
+		ins->type = BTRFS_METADATA_ITEM_KEY;
+	}
+
+	/* Ensure this reserved extent is not found by the allocator */
 	if (root_objectid == BTRFS_EXTENT_TREE_OBJECTID) {
-		struct pending_extent_op *extent_op;
-
-		extent_op = kmalloc(sizeof(*extent_op), GFP_NOFS);
-		BUG_ON(!extent_op);
-
-		extent_op->type = PENDING_EXTENT_INSERT;
-		extent_op->bytenr = ins->objectid;
-		extent_op->num_bytes = ins->offset;
-		extent_op->level = level;
-		extent_op->flags = flags;
-		memcpy(&extent_op->key, key, sizeof(*key));
-
-		set_extent_bits(&root->fs_info->extent_ins, ins->objectid,
-				ins->objectid + ins->offset - 1,
-				EXTENT_LOCKED);
-		set_state_private(&root->fs_info->extent_ins,
-				  ins->objectid, (unsigned long)extent_op);
-	} else {
-		if (btrfs_fs_incompat(root->fs_info, SKINNY_METADATA)) {
-			ins->offset = level;
-			ins->type = BTRFS_METADATA_ITEM_KEY;
-		}
-		ret = alloc_reserved_tree_block(trans, root_objectid,
-						generation, flags,
-						key, level, ins);
-		finish_current_insert(trans);
-		del_pending_extents(trans);
+		ret = set_extent_bits(&trans->fs_info->extent_ins,
+				      ins->objectid,
+				      ins->objectid + extent_size - 1,
+				      EXTENT_LOCKED);
+
+		BUG_ON(ret);
 	}
+
+	ret = btrfs_add_delayed_tree_ref(root->fs_info, trans, ins->objectid,
+					 extent_size, 0, root_objectid,
+					 level, BTRFS_ADD_DELAYED_EXTENT,
+					 extent_op, NULL, NULL);
 	return ret;
 }
 
@@ -3329,11 +3347,6 @@ int btrfs_make_block_group(struct btrfs_trans_handle *trans,
 				sizeof(cache->item));
 	BUG_ON(ret);
 
-	ret = finish_current_insert(trans);
-	BUG_ON(ret);
-	ret = del_pending_extents(trans);
-	BUG_ON(ret);
-
 	return 0;
 }
 
@@ -3429,10 +3442,6 @@ int btrfs_make_block_groups(struct btrfs_trans_handle *trans,
 					sizeof(cache->item));
 		BUG_ON(ret);
 
-		finish_current_insert(trans);
-		ret = del_pending_extents(trans);
-		BUG_ON(ret);
-
 		cur_start = cache->key.objectid + cache->key.offset;
 	}
 	return 0;
@@ -3814,14 +3823,9 @@ int btrfs_fix_block_accounting(struct btrfs_trans_handle *trans)
 	struct btrfs_fs_info *fs_info = trans->fs_info;
 	struct btrfs_root *root = fs_info->extent_root;
 
-	while(extent_root_pending_ops(fs_info)) {
-		ret = finish_current_insert(trans);
-		if (ret)
-			return ret;
-		ret = del_pending_extents(trans);
-		if (ret)
-			return ret;
-	}
+	ret = btrfs_run_delayed_refs(trans, -1);
+	if (ret)
+		return ret;
 
 	while(1) {
 		cache = btrfs_lookup_first_block_group(fs_info, start);
@@ -4026,7 +4030,7 @@ static int __btrfs_record_file_extent(struct btrfs_trans_handle *trans,
 		} else if (ret != -EEXIST) {
 			goto fail;
 		}
-		btrfs_extent_post_op(trans);
+		btrfs_run_delayed_refs(trans, -1);
 		extent_bytenr = disk_bytenr;
 		extent_num_bytes = num_bytes;
 		extent_offset = 0;
transaction.c +25 -2
index 96d9891b0d1c..bfda769210ee 100644
@@ -61,7 +61,6 @@ static int update_cowonly_root(struct btrfs_trans_handle *trans,
 	u64 old_root_bytenr;
 	struct btrfs_root *tree_root = root->fs_info->tree_root;
 
-	btrfs_write_dirty_block_groups(trans);
 	while(1) {
 		old_root_bytenr = btrfs_root_bytenr(&root->root_item);
 		if (old_root_bytenr == root->node->start)
@@ -98,6 +97,17 @@ int commit_tree_roots(struct btrfs_trans_handle *trans,
 	if (ret)
 		return ret;
 
+	/*
+	 * If the above CoW is the first one to dirty the current tree_root,
+	 * delayed refs for it won't be run until after this function has
+	 * finished executing, meaning we won't process the extent tree root,
+	 * which will have been added to ->dirty_cowonly_roots.  So run
+	 * delayed refs here as well.
+	 */
+	ret = btrfs_run_delayed_refs(trans, -1);
+	if (ret)
+		return ret;
+
 	while(!list_empty(&fs_info->dirty_cowonly_roots)) {
 		next = fs_info->dirty_cowonly_roots.next;
 		list_del_init(next);
@@ -147,6 +157,12 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 
 	if (trans->fs_info->transaction_aborted)
 		return -EROFS;
+	/*
+	 * Flush all accumulated delayed refs so that root-tree updates are
+	 * consistent
+	 */
+	ret = btrfs_run_delayed_refs(trans, -1);
+	BUG_ON(ret);
 
 	if (root->commit_root == root->node)
 		goto commit_tree;
@@ -164,11 +180,18 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 	ret = btrfs_update_root(trans, root->fs_info->tree_root,
 				&root->root_key, &root->root_item);
 	BUG_ON(ret);
+
 commit_tree:
 	ret = commit_tree_roots(trans, fs_info);
 	BUG_ON(ret);
-	ret = __commit_transaction(trans, root);
+	/*
+	 * Ensure that all comitted roots are properly accounted in the
+	 * extent tree
+	 */
+	ret = btrfs_run_delayed_refs(trans, -1);
 	BUG_ON(ret);
+	btrfs_write_dirty_block_groups(trans);
+	__commit_transaction(trans, root);
 	write_ctree_super(trans);
 	btrfs_finish_extent_commit(trans, fs_info->extent_root,
 			           &fs_info->pinned_extents);
#10 lore raw

Re: [PATCH 5/8] btrfs-progs: Wire up delayed refs

Quoted 164 lines · Nikolay Borisov
Quoted 102 lines · Qu Wenruo
Quoted 32 lines · Nikolay Borisov
Quoted 29 lines · Qu Wenruo
Quoted 25 lines · Nikolay Borisov

This commit enables the delayed refs infrastructures. This entails doing the following:

1. Replacing existing calls of btrfs_extent_post_op (which is the equivalent of delayed refs) with the proper btrfs_run_delayed_refs. As well as eliminating open-coded calls to finish_current_insert and del_pending_extents which execute the delayed ops.

2. Wiring up the addition of delayed refs when freeing extents (btrfs_free_extent) and when adding new extents (alloc_tree_block).

3. Adding calls to btrfs_run_delayed refs in the transaction commit path alongside comments why every call is needed, since it's not always obvious (those call sites were derived empirically by running and debugging existing tests)

4. Correctly flagging the transaction in which we are reinitialising the extent tree.

5 Moving btrfs_write_dirty_block_groups to btrfs_write_dirty_block_groups since blockgroups should be written to disk after the last delayed refs have been run.

Quoted 7 lines
Quoted 4 lines

Is there something (maybe btrfs_run_delayed_refs()?) missing in btrfs-image?

btrfs-image from devel branch can't restore image correctly, the block group used bytes is not correct, thus it can't pass misc nor fsck tests.

This is really strange, all fsck/misc tests passed with those patches. Can you be more specific which tests exactly you mean ?

One case is fsck/020 with lowmem mode. (Original mode lacks block group->used check).

More specifically, fsck/020/keyed_data_ref_with_shared_leaf.img

Using btrfs-image from my distribution (v4.17.1) and devel branch btrfs
check: (cwd is btrfs-progs, devel branch)

$ btrfs-image -r
tests/fsck-tests/020-extent-ref-cases/keyed_data_ref_with_shared_leaf.img ~/test.img
$ btrfs check --mode=wmem ~/test.img
Opening filesystem to check...
Checking filesystem on /home/adam/test.img
UUID: 12dabcf2-d4da-4a70-9701-9f3d48074e73
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs done with fs roots in lowmem mode, skipping
[7/7] checking quota groups skipped (not enabled on this FS)
found 1208320 bytes used, no error found
total csum bytes: 512
total tree bytes: 684032
total fs tree bytes: 638976
total extent tree bytes: 16384
btree space waste bytes: 305606
file data blocks allocated: 93847552
 referenced 1773568

But if using btrfs-image with your delayed ref patch:

$ ./btrfs-image -r
tests/fsck-tests/020-extent-ref-cases/keyed_data_ref_with_shared_leaf.img ~/test.img

# No matter if I'm using btrfs-check from devel or 4.17.1

$ btrfs check --mode=wmem ~/test.img
Opening filesystem to check...
Checking filesystem on /home/adam/test.img
UUID: 12dabcf2-d4da-4a70-9701-9f3d48074e73
[1/7] checking root items
[2/7] checking extents
ERROR: block group[4194304 8388608] used 20480 but extent items used 24576
ERROR: block group[20971520 16777216] used 659456 but extent items used
655360
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs done with fs roots in lowmem mode, skipping
[7/7] checking quota groups skipped (not enabled on this FS)
found 1208320 bytes used, error(s) found
total csum bytes: 512
total tree bytes: 684032
total fs tree bytes: 638976
total extent tree bytes: 16384
btree space waste bytes: 305606
file data blocks allocated: 93847552
 referenced 1773568
Quoted 3 lines

I'd say, although lowmem check is still far from perfect, it indeed has extra checks original mode lacks, and in this case it indeed exposes problem.

I'm not able to reproduce it:

make TEST_ENABLE_OVERRIDE=ue TEST_ARGS_CHECK="--mode=lowmem" test-fsck

    [TEST]   fsck-tests.sh
    [TEST/fsck]   001-bad-file-extent-bytenr
    [TEST/fsck]   002-bad-transid
    [TEST/fsck]   003-shift-offsets
    [TEST/fsck]   004-no-dir-index
    [TEST/fsck]   005-bad-item-offset
    [TEST/fsck]   006-bad-root-items
    [TEST/fsck]   007-bad-offset-snapshots
    [TEST/fsck]   008-bad-dir-index-name
    [TEST/fsck]   009-no-dir-item-or-index
    [TEST/fsck]   010-no-rootdir-inode-item
    [TEST/fsck]   011-no-inode-item
    [TEST/fsck]   012-leaf-corruption
    [TEST/fsck]   013-extent-tree-rebuild
    [TEST/fsck]   014-no-extent-info
    [TEST/fsck]   015-tree-reloc-tree
    [TEST/fsck]   016-wrong-inode-nbytes
    [TEST/fsck]   017-missing-all-file-extent
    [TEST/fsck]   018-leaf-crossing-stripes
    [TEST/fsck]   019-non-skinny-false-alert
    [TEST/fsck]   020-extent-ref-cases
    [TEST/fsck]   021-partially-dropped-snapshot-case
    [TEST/fsck]   022-qgroup-rescan-halfway
    [TEST/fsck]   023-qgroup-stack-overflow
    [TEST/fsck]   024-clear-space-cache
    [TEST/fsck]   025-file-extents
    [TEST/fsck]   026-bad-dir-item-name
    [TEST/fsck]   027-bad-extent-inline-ref-type
    [TEST/fsck]   028-unaligned-super-dev-sizes
    [TEST/fsck]   029-valid-orphan-item
    [TEST/fsck]   030-reflinked-prealloc-extents
    [TEST/fsck]   031-metadatadump-check-data-csum
    [TEST/fsck]   032-corrupted-qgroup
    [TEST/fsck]   032-freespacetree-corrupted-extent-offset
    [TEST/fsck]   033-lowmem-collission-dir-items
    [TEST/fsck]   034-bad-inode-flags
    [TEST/fsck]   035-inline-bad-ram-bytes
    [TEST/fsck]   035-rescan-not-kicked-in

git bl -9
dac6a0a6f5ca btrfs-progs: Merge alloc_reserved_tree_block(2|) (3 weeks ago) <Nikolay Borisov> da184a3838e6 btrfs-progs: Remove __free_extent2 (3 weeks ago) <Nikolay Borisov>
76b75e40606e btrfs-progs: Remove old delayed refs infrastructure (3 months ago) <Nikolay Borisov> 870c58e08673 btrfs-progs: Wire up delayed refs (3 months ago) <Nikolay Borisov>
e045b219e397 btrfs-progs: Make btrfs_write_dirty_block_groups take only trans argument (3 weeks ago) <Nikolay Borisov> 00f8d76c8a27 btrfs-progs: Add delayed refs infrastructure (3 months ago) <Nikolay Borisov> d03f79e8deb6 btrfs-progs: Add alloc_reserved_tree_block2 function (3 months ago) <Nikolay Borisov> d4a4831fffe9 btrfs-progs: Add __free_extent2 function (3 months ago) <Nikolay Borisov>
7faaca0d9f78 Btrfs progs v4.17.1 (4 weeks ago) <David Sterba>

Quoted 1 line

Are you sure your branch is not dirty with other changes?

Oh, I'm using the (maybe out-of-data) David's devel branch.

For the "Wire up delayed refs" the commit hash is indeed different. In that branch it indeed has some difference.

Is there any branch I could fetch from?

Thanks,
Qu

Quoted 392 lines
Quoted 392 lines

Thanks,
Qu

Quoted 384 lines
Quoted 384 lines

Thanks,
Qu

Quoted 378 lines
---
 check/main.c  |   3 +-
 extent-tree.c | 166 ++++++++++++++++++++++++++++++----------------------------
 transaction.c |  27 +++++++++-
 3 files changed, 112 insertions(+), 84 deletions(-)
check/main.c +2 -1
index bc2ee22f7943..b361cd7e26a0 100644
@@ -8710,7 +8710,7 @@ static int reinit_extent_tree(struct btrfs_trans_handle *trans,
 			fprintf(stderr, "Error adding block group\n");
 			return ret;
 		}
-		btrfs_extent_post_op(trans);
+		btrfs_run_delayed_refs(trans, -1);
 	}
 
 	ret =eset_balance(trans, fs_info);
@@ -9767,6 +9767,7 @@ int cmd_check(int argc, char **argv)
 			goto close_out;
 		}
 
+		trans->reinit_extent_tree =rue;
 		if (init_extent_tree) {
 			printf("Creating a new extent tree\n");
 			ret =einit_extent_tree(trans, info,
extent-tree.c +85 -81
index 7d6c37c6b371..2fa51bbc0359 100644
@@ -1418,8 +1418,6 @@ int btrfs_inc_extent_ref(struct btrfs_trans_handle *trans,
 		err =et;
 out:
 	btrfs_free_path(path);
-	finish_current_insert(trans);
-	del_pending_extents(trans);
 	BUG_ON(err);
 	return err;
 }
@@ -1602,8 +1600,6 @@ int btrfs_set_block_flags(struct btrfs_trans_handle *trans, u64 bytenr,
 	btrfs_set_extent_flags(l, item, flags);
 out:
 	btrfs_free_path(path);
-	finish_current_insert(trans);
-	del_pending_extents(trans);
 	return ret;
 }
 
@@ -1701,7 +1697,6 @@ static int write_one_cache_group(struct btrfs_trans_handle *trans,
 				 struct btrfs_block_group_cache *cache)
 {
 	int ret;
-	int pending_ret;
 	struct btrfs_root *extent_root =rans->fs_info->extent_root;
 	unsigned long bi;
 	struct extent_buffer *leaf;
@@ -1717,12 +1712,8 @@ static int write_one_cache_group(struct btrfs_trans_handle *trans,
 	btrfs_mark_buffer_dirty(leaf);
 	btrfs_release_path(path);
 fail:
-	finish_current_insert(trans);
-	pending_ret =el_pending_extents(trans);
 	if (ret)
 		return ret;
-	if (pending_ret)
-		return pending_ret;
 	return 0;
 
 }
@@ -2049,6 +2040,7 @@ static int finish_current_insert(struct btrfs_trans_handle *trans)
 	int skinny_metadata >>>>  		btrfs_fs_incompat(extent_root->fs_info, SKINNY_METADATA);
 
+
 	while(1) {
 		ret =ind_first_extent_bit(&info->extent_ins, 0, &start,
 					    &end, EXTENT_LOCKED);
@@ -2080,6 +2072,8 @@ static int finish_current_insert(struct btrfs_trans_handle *trans)
 			BUG_ON(1);
 		}
 
+
+		printf("shouldn't be executed\n");
 		clear_extent_bits(&info->extent_ins, start, end, EXTENT_LOCKED);
 		kfree(extent_op);
 	}
@@ -2379,7 +2373,6 @@ static int __free_extent(struct btrfs_trans_handle *trans,
 	}
 fail:
 	btrfs_free_path(path);
-	finish_current_insert(trans);
 	return ret;
 }
 
@@ -2462,33 +2455,30 @@ int btrfs_free_extent(struct btrfs_trans_handle *trans,
 		      u64 bytenr, u64 num_bytes, u64 parent,
 		      u64 root_objectid, u64 owner, u64 offset)
 {
-	struct btrfs_root *extent_root =oot->fs_info->extent_root;
-	int pending_ret;
 	int ret;
 
 	WARN_ON(num_bytes < root->fs_info->sectorsize);
-	if (root =extent_root) {
-		struct pending_extent_op *extent_op;
-
-		extent_op =malloc(sizeof(*extent_op), GFP_NOFS);
-		BUG_ON(!extent_op);
-
-		extent_op->type =ENDING_EXTENT_DELETE;
-		extent_op->bytenr =ytenr;
-		extent_op->num_bytes =um_bytes;
-		extent_op->level =int)owner;
-
-		set_extent_bits(&root->fs_info->pending_del,
-				bytenr, bytenr + num_bytes - 1,
-				EXTENT_LOCKED);
-		set_state_private(&root->fs_info->pending_del,
-				  bytenr, (unsigned long)extent_op);
-		return 0;
+	/*
+	 * tree log blocks never actually go into the extent allocation
+	 * tree, just update pinning info and exit early.
+	 */
+	if (root_objectid =BTRFS_TREE_LOG_OBJECTID) {
+		printf("PINNING EXTENTS IN LOG TREE\n");
+		WARN_ON(owner >=TRFS_FIRST_FREE_OBJECTID);
+		btrfs_pin_extent(trans->fs_info, bytenr, num_bytes);
+		ret =;
+	} else if (owner < BTRFS_FIRST_FREE_OBJECTID) {
+		BUG_ON(offset);
+		ret =trfs_add_delayed_tree_ref(trans->fs_info, trans,
+						 bytenr, num_bytes, parent,
+						 root_objectid, (int)owner,
+						 BTRFS_DROP_DELAYED_REF,
+						 NULL, NULL, NULL);
+	} else {
+		ret =_free_extent(trans, bytenr, num_bytes, parent,
+				    root_objectid, owner, offset, 1);
 	}
-	ret =_free_extent(trans, bytenr, num_bytes, parent,
-			    root_objectid, owner, offset, 1);
-	pending_ret =el_pending_extents(trans);
-	return ret ? ret : pending_ret;
+	return ret;
 }
 
 static u64 stripe_align(struct btrfs_root *root, u64 val)
@@ -2694,6 +2684,8 @@ static int alloc_reserved_tree_block2(struct btrfs_trans_handle *trans,
 	struct btrfs_delayed_tree_ref *ref =trfs_delayed_node_to_tree_ref(node);
 	struct btrfs_key ins;
 	bool skinny_metadata =trfs_fs_incompat(trans->fs_info, SKINNY_METADATA);
+	int ret;
+	u64 start, end;
 
 	ins.objectid =ode->bytenr;
 	if (skinny_metadata) {
@@ -2704,10 +2696,25 @@ static int alloc_reserved_tree_block2(struct btrfs_trans_handle *trans,
 		ins.type =TRFS_EXTENT_ITEM_KEY;
 	}
 
-	return alloc_reserved_tree_block(trans, ref->root, trans->transid,
-					 extent_op->flags_to_set,
-					 &extent_op->key, ref->level, &ins);
+	if (ref->root =BTRFS_EXTENT_TREE_OBJECTID) {
+		ret =ind_first_extent_bit(&trans->fs_info->extent_ins,
+					    node->bytenr, &start, &end,
+					    EXTENT_LOCKED);
+		ASSERT(!ret);
+		ASSERT(start =node->bytenr);
+		ASSERT(end =node->bytenr + node->num_bytes - 1);
+	}
+
+	ret =lloc_reserved_tree_block(trans, ref->root, trans->transid,
+					extent_op->flags_to_set,
+					&extent_op->key, ref->level, &ins);
 
+	if (ref->root =BTRFS_EXTENT_TREE_OBJECTID) {
+		clear_extent_bits(&trans->fs_info->extent_ins, start, end,
+				  EXTENT_LOCKED);
+	}
+
+	return ret;
 }
 
 static int alloc_reserved_tree_block(struct btrfs_trans_handle *trans,
@@ -2772,39 +2779,50 @@ static int alloc_tree_block(struct btrfs_trans_handle *trans,
 			    u64 search_end, struct btrfs_key *ins)
 {
 	int ret;
+	u64 extent_size;
+	struct btrfs_delayed_extent_op *extent_op;
+	bool skinny_metadata =trfs_fs_incompat(root->fs_info,
+						 SKINNY_METADATA);
+
+	extent_op =trfs_alloc_delayed_extent_op();
+	if (!extent_op)
+		return -ENOMEM;
+
 	ret =trfs_reserve_extent(trans, root, num_bytes, empty_size,
 				   hint_byte, search_end, ins, 0);
 	BUG_ON(ret);
 
+	if (key)
+		memcpy(&extent_op->key, key, sizeof(extent_op->key));
+	else
+		memset(&extent_op->key, 0, sizeof(extent_op->key));
+	extent_op->flags_to_set =lags;
+	extent_op->update_key =kinny_metadata ? false : true;
+	extent_op->update_flags =rue;
+	extent_op->is_data =alse;
+	extent_op->level =evel;
+
+	extent_size =ns->offset;
+
+	if (btrfs_fs_incompat(root->fs_info, SKINNY_METADATA)) {
+		ins->offset =evel;
+		ins->type =TRFS_METADATA_ITEM_KEY;
+	}
+
+	/* Ensure this reserved extent is not found by the allocator */
 	if (root_objectid =BTRFS_EXTENT_TREE_OBJECTID) {
-		struct pending_extent_op *extent_op;
-
-		extent_op =malloc(sizeof(*extent_op), GFP_NOFS);
-		BUG_ON(!extent_op);
-
-		extent_op->type =ENDING_EXTENT_INSERT;
-		extent_op->bytenr =ns->objectid;
-		extent_op->num_bytes =ns->offset;
-		extent_op->level =evel;
-		extent_op->flags =lags;
-		memcpy(&extent_op->key, key, sizeof(*key));
-
-		set_extent_bits(&root->fs_info->extent_ins, ins->objectid,
-				ins->objectid + ins->offset - 1,
-				EXTENT_LOCKED);
-		set_state_private(&root->fs_info->extent_ins,
-				  ins->objectid, (unsigned long)extent_op);
-	} else {
-		if (btrfs_fs_incompat(root->fs_info, SKINNY_METADATA)) {
-			ins->offset =evel;
-			ins->type =TRFS_METADATA_ITEM_KEY;
-		}
-		ret =lloc_reserved_tree_block(trans, root_objectid,
-						generation, flags,
-						key, level, ins);
-		finish_current_insert(trans);
-		del_pending_extents(trans);
+		ret =et_extent_bits(&trans->fs_info->extent_ins,
+				      ins->objectid,
+				      ins->objectid + extent_size - 1,
+				      EXTENT_LOCKED);
+
+		BUG_ON(ret);
 	}
+
+	ret =trfs_add_delayed_tree_ref(root->fs_info, trans, ins->objectid,
+					 extent_size, 0, root_objectid,
+					 level, BTRFS_ADD_DELAYED_EXTENT,
+					 extent_op, NULL, NULL);
 	return ret;
 }
 
@@ -3329,11 +3347,6 @@ int btrfs_make_block_group(struct btrfs_trans_handle *trans,
 				sizeof(cache->item));
 	BUG_ON(ret);
 
-	ret =inish_current_insert(trans);
-	BUG_ON(ret);
-	ret =el_pending_extents(trans);
-	BUG_ON(ret);
-
 	return 0;
 }
 
@@ -3429,10 +3442,6 @@ int btrfs_make_block_groups(struct btrfs_trans_handle *trans,
 					sizeof(cache->item));
 		BUG_ON(ret);
 
-		finish_current_insert(trans);
-		ret =el_pending_extents(trans);
-		BUG_ON(ret);
-
 		cur_start =ache->key.objectid + cache->key.offset;
 	}
 	return 0;
@@ -3814,14 +3823,9 @@ int btrfs_fix_block_accounting(struct btrfs_trans_handle *trans)
 	struct btrfs_fs_info *fs_info =rans->fs_info;
 	struct btrfs_root *root =s_info->extent_root;
 
-	while(extent_root_pending_ops(fs_info)) {
-		ret =inish_current_insert(trans);
-		if (ret)
-			return ret;
-		ret =el_pending_extents(trans);
-		if (ret)
-			return ret;
-	}
+	ret =trfs_run_delayed_refs(trans, -1);
+	if (ret)
+		return ret;
 
 	while(1) {
 		cache =trfs_lookup_first_block_group(fs_info, start);
@@ -4026,7 +4030,7 @@ static int __btrfs_record_file_extent(struct btrfs_trans_handle *trans,
 		} else if (ret !=EEXIST) {
 			goto fail;
 		}
-		btrfs_extent_post_op(trans);
+		btrfs_run_delayed_refs(trans, -1);
 		extent_bytenr =isk_bytenr;
 		extent_num_bytes =um_bytes;
 		extent_offset =;
transaction.c +25 -2
index 96d9891b0d1c..bfda769210ee 100644
@@ -61,7 +61,6 @@ static int update_cowonly_root(struct btrfs_trans_handle *trans,
 	u64 old_root_bytenr;
 	struct btrfs_root *tree_root =oot->fs_info->tree_root;
 
-	btrfs_write_dirty_block_groups(trans);
 	while(1) {
 		old_root_bytenr =trfs_root_bytenr(&root->root_item);
 		if (old_root_bytenr =root->node->start)
@@ -98,6 +97,17 @@ int commit_tree_roots(struct btrfs_trans_handle *trans,
 	if (ret)
 		return ret;
 
+	/*
+	 * If the above CoW is the first one to dirty the current tree_root,
+	 * delayed refs for it won't be run until after this function has
+	 * finished executing, meaning we won't process the extent tree root,
+	 * which will have been added to ->dirty_cowonly_roots.  So run
+	 * delayed refs here as well.
+	 */
+	ret =trfs_run_delayed_refs(trans, -1);
+	if (ret)
+		return ret;
+
 	while(!list_empty(&fs_info->dirty_cowonly_roots)) {
 		next =s_info->dirty_cowonly_roots.next;
 		list_del_init(next);
@@ -147,6 +157,12 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 
 	if (trans->fs_info->transaction_aborted)
 		return -EROFS;
+	/*
+	 * Flush all accumulated delayed refs so that root-tree updates are
+	 * consistent
+	 */
+	ret =trfs_run_delayed_refs(trans, -1);
+	BUG_ON(ret);
 
 	if (root->commit_root =root->node)
 		goto commit_tree;
@@ -164,11 +180,18 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 	ret =trfs_update_root(trans, root->fs_info->tree_root,
 				&root->root_key, &root->root_item);
 	BUG_ON(ret);
+
 commit_tree:
 	ret =ommit_tree_roots(trans, fs_info);
 	BUG_ON(ret);
-	ret =_commit_transaction(trans, root);
+	/*
+	 * Ensure that all comitted roots are properly accounted in the
+	 * extent tree
+	 */
+	ret =trfs_run_delayed_refs(trans, -1);
 	BUG_ON(ret);
+	btrfs_write_dirty_block_groups(trans);
+	__commit_transaction(trans, root);
 	write_ctree_super(trans);
 	btrfs_finish_extent_commit(trans, fs_info->extent_root,
 			           &fs_info->pinned_extents);
#11 lore raw

Re: [PATCH 5/8] btrfs-progs: Wire up delayed refs

Quoted 167 lines · Qu Wenruo
Quoted 164 lines · Nikolay Borisov
Quoted 102 lines · Qu Wenruo
Quoted 32 lines · Nikolay Borisov
Quoted 29 lines · Qu Wenruo
Quoted 25 lines · Nikolay Borisov

This commit enables the delayed refs infrastructures. This entails doing the following:

1. Replacing existing calls of btrfs_extent_post_op (which is the equivalent of delayed refs) with the proper btrfs_run_delayed_refs. As well as eliminating open-coded calls to finish_current_insert and del_pending_extents which execute the delayed ops.

2. Wiring up the addition of delayed refs when freeing extents (btrfs_free_extent) and when adding new extents (alloc_tree_block).

3. Adding calls to btrfs_run_delayed refs in the transaction commit path alongside comments why every call is needed, since it's not always obvious (those call sites were derived empirically by running and debugging existing tests)

4. Correctly flagging the transaction in which we are reinitialising the extent tree.

5 Moving btrfs_write_dirty_block_groups to btrfs_write_dirty_block_groups since blockgroups should be written to disk after the last delayed refs have been run.

Quoted 7 lines
Quoted 4 lines

Is there something (maybe btrfs_run_delayed_refs()?) missing in btrfs-image?

btrfs-image from devel branch can't restore image correctly, the block group used bytes is not correct, thus it can't pass misc nor fsck tests.

This is really strange, all fsck/misc tests passed with those patches. Can you be more specific which tests exactly you mean ?

One case is fsck/020 with lowmem mode. (Original mode lacks block group->used check).

More specifically, fsck/020/keyed_data_ref_with_shared_leaf.img

Using btrfs-image from my distribution (v4.17.1) and devel branch btrfs
check: (cwd is btrfs-progs, devel branch)

$ btrfs-image -r
tests/fsck-tests/020-extent-ref-cases/keyed_data_ref_with_shared_leaf.img ~/test.img
$ btrfs check --mode=wmem ~/test.img
Opening filesystem to check...
Checking filesystem on /home/adam/test.img
UUID: 12dabcf2-d4da-4a70-9701-9f3d48074e73
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs done with fs roots in lowmem mode, skipping
[7/7] checking quota groups skipped (not enabled on this FS)
found 1208320 bytes used, no error found
total csum bytes: 512
total tree bytes: 684032
total fs tree bytes: 638976
total extent tree bytes: 16384
btree space waste bytes: 305606
file data blocks allocated: 93847552
 referenced 1773568

But if using btrfs-image with your delayed ref patch:

$ ./btrfs-image -r
tests/fsck-tests/020-extent-ref-cases/keyed_data_ref_with_shared_leaf.img ~/test.img

# No matter if I'm using btrfs-check from devel or 4.17.1

$ btrfs check --mode=wmem ~/test.img
Opening filesystem to check...
Checking filesystem on /home/adam/test.img
UUID: 12dabcf2-d4da-4a70-9701-9f3d48074e73
[1/7] checking root items
[2/7] checking extents
ERROR: block group[4194304 8388608] used 20480 but extent items used 24576
ERROR: block group[20971520 16777216] used 659456 but extent items used
655360
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs done with fs roots in lowmem mode, skipping
[7/7] checking quota groups skipped (not enabled on this FS)
found 1208320 bytes used, error(s) found
total csum bytes: 512
total tree bytes: 684032
total fs tree bytes: 638976
total extent tree bytes: 16384
btree space waste bytes: 305606
file data blocks allocated: 93847552
 referenced 1773568
Quoted 3 lines

I'd say, although lowmem check is still far from perfect, it indeed has extra checks original mode lacks, and in this case it indeed exposes problem.

I'm not able to reproduce it:

make TEST_ENABLE_OVERRIDE=ue TEST_ARGS_CHECK="--mode=lowmem" test-fsck

    [TEST]   fsck-tests.sh
    [TEST/fsck]   001-bad-file-extent-bytenr
    [TEST/fsck]   002-bad-transid
    [TEST/fsck]   003-shift-offsets
    [TEST/fsck]   004-no-dir-index
    [TEST/fsck]   005-bad-item-offset
    [TEST/fsck]   006-bad-root-items
    [TEST/fsck]   007-bad-offset-snapshots
    [TEST/fsck]   008-bad-dir-index-name
    [TEST/fsck]   009-no-dir-item-or-index
    [TEST/fsck]   010-no-rootdir-inode-item
    [TEST/fsck]   011-no-inode-item
    [TEST/fsck]   012-leaf-corruption
    [TEST/fsck]   013-extent-tree-rebuild
    [TEST/fsck]   014-no-extent-info
    [TEST/fsck]   015-tree-reloc-tree
    [TEST/fsck]   016-wrong-inode-nbytes
    [TEST/fsck]   017-missing-all-file-extent
    [TEST/fsck]   018-leaf-crossing-stripes
    [TEST/fsck]   019-non-skinny-false-alert
    [TEST/fsck]   020-extent-ref-cases
    [TEST/fsck]   021-partially-dropped-snapshot-case
    [TEST/fsck]   022-qgroup-rescan-halfway
    [TEST/fsck]   023-qgroup-stack-overflow
    [TEST/fsck]   024-clear-space-cache
    [TEST/fsck]   025-file-extents
    [TEST/fsck]   026-bad-dir-item-name
    [TEST/fsck]   027-bad-extent-inline-ref-type
    [TEST/fsck]   028-unaligned-super-dev-sizes
    [TEST/fsck]   029-valid-orphan-item
    [TEST/fsck]   030-reflinked-prealloc-extents
    [TEST/fsck]   031-metadatadump-check-data-csum
    [TEST/fsck]   032-corrupted-qgroup
    [TEST/fsck]   032-freespacetree-corrupted-extent-offset
    [TEST/fsck]   033-lowmem-collission-dir-items
    [TEST/fsck]   034-bad-inode-flags
    [TEST/fsck]   035-inline-bad-ram-bytes
    [TEST/fsck]   035-rescan-not-kicked-in

git bl -9
dac6a0a6f5ca btrfs-progs: Merge alloc_reserved_tree_block(2|) (3 weeks ago) <Nikolay Borisov> da184a3838e6 btrfs-progs: Remove __free_extent2 (3 weeks ago) <Nikolay Borisov>
76b75e40606e btrfs-progs: Remove old delayed refs infrastructure (3 months ago) <Nikolay Borisov> 870c58e08673 btrfs-progs: Wire up delayed refs (3 months ago) <Nikolay Borisov>
e045b219e397 btrfs-progs: Make btrfs_write_dirty_block_groups take only trans argument (3 weeks ago) <Nikolay Borisov> 00f8d76c8a27 btrfs-progs: Add delayed refs infrastructure (3 months ago) <Nikolay Borisov> d03f79e8deb6 btrfs-progs: Add alloc_reserved_tree_block2 function (3 months ago) <Nikolay Borisov> d4a4831fffe9 btrfs-progs: Add __free_extent2 function (3 months ago) <Nikolay Borisov>
7faaca0d9f78 Btrfs progs v4.17.1 (4 weeks ago) <David Sterba>

Quoted 8 lines
Quoted 1 line

Are you sure your branch is not dirty with other changes?

Oh, I'm using the (maybe out-of-data) David's devel branch.

For the "Wire up delayed refs" the commit hash is indeed different. In that branch it indeed has some difference.

Is there any branch I could fetch from?

I've just pushed a delayed-refs-v2 branch to :
https://github.com/lorddoskias/btrfs-progs/tree/delayed-refs-v2

Quoted 397 lines

Thanks,
Qu

Quoted 392 lines
Quoted 392 lines

Thanks,
Qu

Quoted 384 lines
Quoted 384 lines

Thanks,
Qu

Quoted 378 lines
---
 check/main.c  |   3 +-
 extent-tree.c | 166 ++++++++++++++++++++++++++++++----------------------------
 transaction.c |  27 +++++++++-
 3 files changed, 112 insertions(+), 84 deletions(-)
check/main.c +2 -1
index bc2ee22f7943..b361cd7e26a0 100644
@@ -8710,7 +8710,7 @@ static int reinit_extent_tree(struct btrfs_trans_handle *trans,
 			fprintf(stderr, "Error adding block group\n");
 			return ret;
 		}
-		btrfs_extent_post_op(trans);
+		btrfs_run_delayed_refs(trans, -1);
 	}
 
 	ret =eset_balance(trans, fs_info);
@@ -9767,6 +9767,7 @@ int cmd_check(int argc, char **argv)
 			goto close_out;
 		}
 
+		trans->reinit_extent_tree =rue;
 		if (init_extent_tree) {
 			printf("Creating a new extent tree\n");
 			ret =einit_extent_tree(trans, info,
extent-tree.c +85 -81
index 7d6c37c6b371..2fa51bbc0359 100644
@@ -1418,8 +1418,6 @@ int btrfs_inc_extent_ref(struct btrfs_trans_handle *trans,
 		err =et;
 out:
 	btrfs_free_path(path);
-	finish_current_insert(trans);
-	del_pending_extents(trans);
 	BUG_ON(err);
 	return err;
 }
@@ -1602,8 +1600,6 @@ int btrfs_set_block_flags(struct btrfs_trans_handle *trans, u64 bytenr,
 	btrfs_set_extent_flags(l, item, flags);
 out:
 	btrfs_free_path(path);
-	finish_current_insert(trans);
-	del_pending_extents(trans);
 	return ret;
 }
 
@@ -1701,7 +1697,6 @@ static int write_one_cache_group(struct btrfs_trans_handle *trans,
 				 struct btrfs_block_group_cache *cache)
 {
 	int ret;
-	int pending_ret;
 	struct btrfs_root *extent_root =rans->fs_info->extent_root;
 	unsigned long bi;
 	struct extent_buffer *leaf;
@@ -1717,12 +1712,8 @@ static int write_one_cache_group(struct btrfs_trans_handle *trans,
 	btrfs_mark_buffer_dirty(leaf);
 	btrfs_release_path(path);
 fail:
-	finish_current_insert(trans);
-	pending_ret =el_pending_extents(trans);
 	if (ret)
 		return ret;
-	if (pending_ret)
-		return pending_ret;
 	return 0;
 
 }
@@ -2049,6 +2040,7 @@ static int finish_current_insert(struct btrfs_trans_handle *trans)
 	int skinny_metadata >>>>  		btrfs_fs_incompat(extent_root->fs_info, SKINNY_METADATA);
 
+
 	while(1) {
 		ret =ind_first_extent_bit(&info->extent_ins, 0, &start,
 					    &end, EXTENT_LOCKED);
@@ -2080,6 +2072,8 @@ static int finish_current_insert(struct btrfs_trans_handle *trans)
 			BUG_ON(1);
 		}
 
+
+		printf("shouldn't be executed\n");
 		clear_extent_bits(&info->extent_ins, start, end, EXTENT_LOCKED);
 		kfree(extent_op);
 	}
@@ -2379,7 +2373,6 @@ static int __free_extent(struct btrfs_trans_handle *trans,
 	}
 fail:
 	btrfs_free_path(path);
-	finish_current_insert(trans);
 	return ret;
 }
 
@@ -2462,33 +2455,30 @@ int btrfs_free_extent(struct btrfs_trans_handle *trans,
 		      u64 bytenr, u64 num_bytes, u64 parent,
 		      u64 root_objectid, u64 owner, u64 offset)
 {
-	struct btrfs_root *extent_root =oot->fs_info->extent_root;
-	int pending_ret;
 	int ret;
 
 	WARN_ON(num_bytes < root->fs_info->sectorsize);
-	if (root =extent_root) {
-		struct pending_extent_op *extent_op;
-
-		extent_op =malloc(sizeof(*extent_op), GFP_NOFS);
-		BUG_ON(!extent_op);
-
-		extent_op->type =ENDING_EXTENT_DELETE;
-		extent_op->bytenr =ytenr;
-		extent_op->num_bytes =um_bytes;
-		extent_op->level =int)owner;
-
-		set_extent_bits(&root->fs_info->pending_del,
-				bytenr, bytenr + num_bytes - 1,
-				EXTENT_LOCKED);
-		set_state_private(&root->fs_info->pending_del,
-				  bytenr, (unsigned long)extent_op);
-		return 0;
+	/*
+	 * tree log blocks never actually go into the extent allocation
+	 * tree, just update pinning info and exit early.
+	 */
+	if (root_objectid =BTRFS_TREE_LOG_OBJECTID) {
+		printf("PINNING EXTENTS IN LOG TREE\n");
+		WARN_ON(owner >=TRFS_FIRST_FREE_OBJECTID);
+		btrfs_pin_extent(trans->fs_info, bytenr, num_bytes);
+		ret =;
+	} else if (owner < BTRFS_FIRST_FREE_OBJECTID) {
+		BUG_ON(offset);
+		ret =trfs_add_delayed_tree_ref(trans->fs_info, trans,
+						 bytenr, num_bytes, parent,
+						 root_objectid, (int)owner,
+						 BTRFS_DROP_DELAYED_REF,
+						 NULL, NULL, NULL);
+	} else {
+		ret =_free_extent(trans, bytenr, num_bytes, parent,
+				    root_objectid, owner, offset, 1);
 	}
-	ret =_free_extent(trans, bytenr, num_bytes, parent,
-			    root_objectid, owner, offset, 1);
-	pending_ret =el_pending_extents(trans);
-	return ret ? ret : pending_ret;
+	return ret;
 }
 
 static u64 stripe_align(struct btrfs_root *root, u64 val)
@@ -2694,6 +2684,8 @@ static int alloc_reserved_tree_block2(struct btrfs_trans_handle *trans,
 	struct btrfs_delayed_tree_ref *ref =trfs_delayed_node_to_tree_ref(node);
 	struct btrfs_key ins;
 	bool skinny_metadata =trfs_fs_incompat(trans->fs_info, SKINNY_METADATA);
+	int ret;
+	u64 start, end;
 
 	ins.objectid =ode->bytenr;
 	if (skinny_metadata) {
@@ -2704,10 +2696,25 @@ static int alloc_reserved_tree_block2(struct btrfs_trans_handle *trans,
 		ins.type =TRFS_EXTENT_ITEM_KEY;
 	}
 
-	return alloc_reserved_tree_block(trans, ref->root, trans->transid,
-					 extent_op->flags_to_set,
-					 &extent_op->key, ref->level, &ins);
+	if (ref->root =BTRFS_EXTENT_TREE_OBJECTID) {
+		ret =ind_first_extent_bit(&trans->fs_info->extent_ins,
+					    node->bytenr, &start, &end,
+					    EXTENT_LOCKED);
+		ASSERT(!ret);
+		ASSERT(start =node->bytenr);
+		ASSERT(end =node->bytenr + node->num_bytes - 1);
+	}
+
+	ret =lloc_reserved_tree_block(trans, ref->root, trans->transid,
+					extent_op->flags_to_set,
+					&extent_op->key, ref->level, &ins);
 
+	if (ref->root =BTRFS_EXTENT_TREE_OBJECTID) {
+		clear_extent_bits(&trans->fs_info->extent_ins, start, end,
+				  EXTENT_LOCKED);
+	}
+
+	return ret;
 }
 
 static int alloc_reserved_tree_block(struct btrfs_trans_handle *trans,
@@ -2772,39 +2779,50 @@ static int alloc_tree_block(struct btrfs_trans_handle *trans,
 			    u64 search_end, struct btrfs_key *ins)
 {
 	int ret;
+	u64 extent_size;
+	struct btrfs_delayed_extent_op *extent_op;
+	bool skinny_metadata =trfs_fs_incompat(root->fs_info,
+						 SKINNY_METADATA);
+
+	extent_op =trfs_alloc_delayed_extent_op();
+	if (!extent_op)
+		return -ENOMEM;
+
 	ret =trfs_reserve_extent(trans, root, num_bytes, empty_size,
 				   hint_byte, search_end, ins, 0);
 	BUG_ON(ret);
 
+	if (key)
+		memcpy(&extent_op->key, key, sizeof(extent_op->key));
+	else
+		memset(&extent_op->key, 0, sizeof(extent_op->key));
+	extent_op->flags_to_set =lags;
+	extent_op->update_key =kinny_metadata ? false : true;
+	extent_op->update_flags =rue;
+	extent_op->is_data =alse;
+	extent_op->level =evel;
+
+	extent_size =ns->offset;
+
+	if (btrfs_fs_incompat(root->fs_info, SKINNY_METADATA)) {
+		ins->offset =evel;
+		ins->type =TRFS_METADATA_ITEM_KEY;
+	}
+
+	/* Ensure this reserved extent is not found by the allocator */
 	if (root_objectid =BTRFS_EXTENT_TREE_OBJECTID) {
-		struct pending_extent_op *extent_op;
-
-		extent_op =malloc(sizeof(*extent_op), GFP_NOFS);
-		BUG_ON(!extent_op);
-
-		extent_op->type =ENDING_EXTENT_INSERT;
-		extent_op->bytenr =ns->objectid;
-		extent_op->num_bytes =ns->offset;
-		extent_op->level =evel;
-		extent_op->flags =lags;
-		memcpy(&extent_op->key, key, sizeof(*key));
-
-		set_extent_bits(&root->fs_info->extent_ins, ins->objectid,
-				ins->objectid + ins->offset - 1,
-				EXTENT_LOCKED);
-		set_state_private(&root->fs_info->extent_ins,
-				  ins->objectid, (unsigned long)extent_op);
-	} else {
-		if (btrfs_fs_incompat(root->fs_info, SKINNY_METADATA)) {
-			ins->offset =evel;
-			ins->type =TRFS_METADATA_ITEM_KEY;
-		}
-		ret =lloc_reserved_tree_block(trans, root_objectid,
-						generation, flags,
-						key, level, ins);
-		finish_current_insert(trans);
-		del_pending_extents(trans);
+		ret =et_extent_bits(&trans->fs_info->extent_ins,
+				      ins->objectid,
+				      ins->objectid + extent_size - 1,
+				      EXTENT_LOCKED);
+
+		BUG_ON(ret);
 	}
+
+	ret =trfs_add_delayed_tree_ref(root->fs_info, trans, ins->objectid,
+					 extent_size, 0, root_objectid,
+					 level, BTRFS_ADD_DELAYED_EXTENT,
+					 extent_op, NULL, NULL);
 	return ret;
 }
 
@@ -3329,11 +3347,6 @@ int btrfs_make_block_group(struct btrfs_trans_handle *trans,
 				sizeof(cache->item));
 	BUG_ON(ret);
 
-	ret =inish_current_insert(trans);
-	BUG_ON(ret);
-	ret =el_pending_extents(trans);
-	BUG_ON(ret);
-
 	return 0;
 }
 
@@ -3429,10 +3442,6 @@ int btrfs_make_block_groups(struct btrfs_trans_handle *trans,
 					sizeof(cache->item));
 		BUG_ON(ret);
 
-		finish_current_insert(trans);
-		ret =el_pending_extents(trans);
-		BUG_ON(ret);
-
 		cur_start =ache->key.objectid + cache->key.offset;
 	}
 	return 0;
@@ -3814,14 +3823,9 @@ int btrfs_fix_block_accounting(struct btrfs_trans_handle *trans)
 	struct btrfs_fs_info *fs_info =rans->fs_info;
 	struct btrfs_root *root =s_info->extent_root;
 
-	while(extent_root_pending_ops(fs_info)) {
-		ret =inish_current_insert(trans);
-		if (ret)
-			return ret;
-		ret =el_pending_extents(trans);
-		if (ret)
-			return ret;
-	}
+	ret =trfs_run_delayed_refs(trans, -1);
+	if (ret)
+		return ret;
 
 	while(1) {
 		cache =trfs_lookup_first_block_group(fs_info, start);
@@ -4026,7 +4030,7 @@ static int __btrfs_record_file_extent(struct btrfs_trans_handle *trans,
 		} else if (ret !=EEXIST) {
 			goto fail;
 		}
-		btrfs_extent_post_op(trans);
+		btrfs_run_delayed_refs(trans, -1);
 		extent_bytenr =isk_bytenr;
 		extent_num_bytes =um_bytes;
 		extent_offset =;
transaction.c +25 -2
index 96d9891b0d1c..bfda769210ee 100644
@@ -61,7 +61,6 @@ static int update_cowonly_root(struct btrfs_trans_handle *trans,
 	u64 old_root_bytenr;
 	struct btrfs_root *tree_root =oot->fs_info->tree_root;
 
-	btrfs_write_dirty_block_groups(trans);
 	while(1) {
 		old_root_bytenr =trfs_root_bytenr(&root->root_item);
 		if (old_root_bytenr =root->node->start)
@@ -98,6 +97,17 @@ int commit_tree_roots(struct btrfs_trans_handle *trans,
 	if (ret)
 		return ret;
 
+	/*
+	 * If the above CoW is the first one to dirty the current tree_root,
+	 * delayed refs for it won't be run until after this function has
+	 * finished executing, meaning we won't process the extent tree root,
+	 * which will have been added to ->dirty_cowonly_roots.  So run
+	 * delayed refs here as well.
+	 */
+	ret =trfs_run_delayed_refs(trans, -1);
+	if (ret)
+		return ret;
+
 	while(!list_empty(&fs_info->dirty_cowonly_roots)) {
 		next =s_info->dirty_cowonly_roots.next;
 		list_del_init(next);
@@ -147,6 +157,12 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 
 	if (trans->fs_info->transaction_aborted)
 		return -EROFS;
+	/*
+	 * Flush all accumulated delayed refs so that root-tree updates are
+	 * consistent
+	 */
+	ret =trfs_run_delayed_refs(trans, -1);
+	BUG_ON(ret);
 
 	if (root->commit_root =root->node)
 		goto commit_tree;
@@ -164,11 +180,18 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 	ret =trfs_update_root(trans, root->fs_info->tree_root,
 				&root->root_key, &root->root_item);
 	BUG_ON(ret);
+
 commit_tree:
 	ret =ommit_tree_roots(trans, fs_info);
 	BUG_ON(ret);
-	ret =_commit_transaction(trans, root);
+	/*
+	 * Ensure that all comitted roots are properly accounted in the
+	 * extent tree
+	 */
+	ret =trfs_run_delayed_refs(trans, -1);
 	BUG_ON(ret);
+	btrfs_write_dirty_block_groups(trans);
+	__commit_transaction(trans, root);
 	write_ctree_super(trans);
 	btrfs_finish_extent_commit(trans, fs_info->extent_root,
 			           &fs_info->pinned_extents);
#12 lore raw

PATCH v2 6/8 btrfs-progs: Remove old delayed refs infrastructure

In reply to Nikolay Borisov

Given that the new delayed refs infrastructure is implemented and wired up, there is no point in keeping the old code. So just remove it.

---
 ctree.h       |   2 -
 disk-io.c     |   2 -
 extent-tree.c | 137 ----------------------------------------------------------
 3 files changed, 141 deletions(-)
ctree.h +0 -2
index 75675ef3f781..49f0f5181512 100644
@@ -1098,7 +1098,6 @@ struct btrfs_fs_info {
 	struct extent_io_tree free_space_cache;
 	struct extent_io_tree block_group_cache;
 	struct extent_io_tree pinned_extents;
-	struct extent_io_tree pending_del;
 	struct extent_io_tree extent_ins;
 	struct extent_io_tree *excluded_extents;
 
@@ -2481,7 +2480,6 @@ int btrfs_fix_block_accounting(struct btrfs_trans_handle *trans);
 void btrfs_pin_extent(struct btrfs_fs_info *fs_info, u64 bytenr, u64 num_bytes);
 void btrfs_unpin_extent(struct btrfs_fs_info *fs_info,
 			u64 bytenr, u64 num_bytes);
-int btrfs_extent_post_op(struct btrfs_trans_handle *trans);
 struct btrfs_block_group_cache *btrfs_lookup_block_group(struct
 							 btrfs_fs_info *info,
 							 u64 bytenr);
disk-io.c +0 -2
index 26e4f6e93ed6..2e6d56a36af9 100644
@@ -730,7 +730,6 @@ struct btrfs_fs_info *btrfs_new_fs_info(int writable, u64 sb_bytenr)
 	extent_io_tree_init(&fs_info->free_space_cache);
 	extent_io_tree_init(&fs_info->block_group_cache);
 	extent_io_tree_init(&fs_info->pinned_extents);
-	extent_io_tree_init(&fs_info->pending_del);
 	extent_io_tree_init(&fs_info->extent_ins);
 	fs_info->excluded_extents = NULL;
 
@@ -988,7 +987,6 @@ void btrfs_cleanup_all_caches(struct btrfs_fs_info *fs_info)
 	extent_io_tree_cleanup(&fs_info->free_space_cache);
 	extent_io_tree_cleanup(&fs_info->block_group_cache);
 	extent_io_tree_cleanup(&fs_info->pinned_extents);
-	extent_io_tree_cleanup(&fs_info->pending_del);
 	extent_io_tree_cleanup(&fs_info->extent_ins);
 }
 
extent-tree.c +0 -137

161 lines not on this page · raw message

Signature · 1 line
2.7.4
#13 lore raw

PATCH v2 7/8 btrfs-progs: Remove __free_extent2

In reply to Nikolay Borisov

Now that delayed refs have been all wired up clean up the __free_extent2 adapter function since it's no longer needed. No functional changes.

---
 extent-tree.c | 15 +++------------
 1 file changed, 3 insertions(+), 12 deletions(-)
extent-tree.c +3 -12
index 6893b4c07019..1a63efdd9681 100644
@@ -2052,17 +2052,6 @@ void btrfs_unpin_extent(struct btrfs_fs_info *fs_info,
 	update_pinned_extents(fs_info, bytenr, num_bytes, 0);
 }
 
-static int __free_extent2(struct btrfs_trans_handle *trans,
-			  struct btrfs_delayed_ref_node *node,
-			  struct btrfs_delayed_extent_op *extent_op)
-{
-
-	struct btrfs_delayed_tree_ref *ref = btrfs_delayed_node_to_tree_ref(node);
-
-	return __free_extent(trans, node->bytenr, node->num_bytes,
-			     ref->parent, ref->root, ref->level, 0, 1);
-}
-
 /*
  * remove an extent from the root, returns 0 on success
  */
@@ -4183,7 +4172,9 @@ static int run_delayed_tree_ref(struct btrfs_trans_handle *trans,
 		BUG_ON(!extent_op || !extent_op->update_flags);
 		ret = alloc_reserved_tree_block2(trans, node, extent_op);
 	} else if (node->action == BTRFS_DROP_DELAYED_REF) {
-		ret = __free_extent2(trans, node, extent_op);
+		struct btrfs_delayed_tree_ref *ref = btrfs_delayed_node_to_tree_ref(node);
+		ret =  __free_extent(trans, node->bytenr, node->num_bytes,
+			     ref->parent, ref->root, ref->level, 0, 1);
 	} else {
 		BUG();
 	}
Signature · 1 line
2.7.4
#14 lore raw

PATCH v2 8/8 btrfs-progs: Merge alloc_reserved_tree_block(2|)

In reply to Nikolay Borisov

Now that delayed refs have been wired let's merge the two function. In the process also remove one BUG_ON since alloc_reserved_tree_block's callers can handle errors. No functional changes.

---
 extent-tree.c | 77 +++++++++++++++++++++++------------------------------------
 1 file changed, 30 insertions(+), 47 deletions(-)
extent-tree.c +30 -47
index 1a63efdd9681..b9a30644720b 100644
@@ -44,10 +44,6 @@ struct pending_extent_op {
 	int level;
 };
 
-static int alloc_reserved_tree_block(struct btrfs_trans_handle *trans,
-				     u64 root_objectid, u64 generation,
-				     u64 flags, struct btrfs_disk_key *key,
-				     int level, struct btrfs_key *ins);
 static int __free_extent(struct btrfs_trans_handle *trans,
 			 u64 bytenr, u64 num_bytes, u64 parent,
 			 u64 root_objectid, u64 owner_objectid,
@@ -2528,16 +2524,22 @@ int btrfs_reserve_extent(struct btrfs_trans_handle *trans,
 	return ret;
 }
 
-static int alloc_reserved_tree_block2(struct btrfs_trans_handle *trans,
+static int alloc_reserved_tree_block(struct btrfs_trans_handle *trans,
 				      struct btrfs_delayed_ref_node *node,
 				      struct btrfs_delayed_extent_op *extent_op)
 {
 
 	struct btrfs_delayed_tree_ref *ref = btrfs_delayed_node_to_tree_ref(node);
-	struct btrfs_key ins;
 	bool skinny_metadata = btrfs_fs_incompat(trans->fs_info, SKINNY_METADATA);
-	int ret;
+	struct btrfs_fs_info *fs_info = trans->fs_info;
+	struct btrfs_extent_item *extent_item;
+	struct btrfs_extent_inline_ref *iref;
+	struct extent_buffer *leaf;
+	struct btrfs_path *path;
+	struct btrfs_key ins;
+	u32 size = sizeof(*extent_item) + sizeof(*iref);
 	u64 start, end;
+	int ret;
 
 	ins.objectid = node->bytenr;
 	if (skinny_metadata) {
@@ -2546,6 +2548,8 @@ static int alloc_reserved_tree_block2(struct btrfs_trans_handle *trans,
 	} else {
 		ins.offset = node->num_bytes;
 		ins.type = BTRFS_EXTENT_ITEM_KEY;
+
+		size += sizeof(struct btrfs_tree_block_info);
 	}
 
 	if (ref->root == BTRFS_EXTENT_TREE_OBJECTID) {
@@ -2557,69 +2561,48 @@ static int alloc_reserved_tree_block2(struct btrfs_trans_handle *trans,
 		ASSERT(end == node->bytenr + node->num_bytes - 1);
 	}
 
-	ret = alloc_reserved_tree_block(trans, ref->root, trans->transid,
-					extent_op->flags_to_set,
-					&extent_op->key, ref->level, &ins);
-
-	if (ref->root == BTRFS_EXTENT_TREE_OBJECTID) {
-		clear_extent_bits(&trans->fs_info->extent_ins, start, end,
-				  EXTENT_LOCKED);
-	}
-
-	return ret;
-}
-
-static int alloc_reserved_tree_block(struct btrfs_trans_handle *trans,
-				     u64 root_objectid, u64 generation,
-				     u64 flags, struct btrfs_disk_key *key,
-				     int level, struct btrfs_key *ins)
-{
-	int ret;
-	struct btrfs_fs_info *fs_info = trans->fs_info;
-	struct btrfs_extent_item *extent_item;
-	struct btrfs_tree_block_info *block_info;
-	struct btrfs_extent_inline_ref *iref;
-	struct btrfs_path *path;
-	struct extent_buffer *leaf;
-	u32 size = sizeof(*extent_item) + sizeof(*iref);
-	int skinny_metadata = btrfs_fs_incompat(fs_info, SKINNY_METADATA);
-
-	if (!skinny_metadata)
-		size += sizeof(*block_info);
-
 	path = btrfs_alloc_path();
 	if (!path)
 		return -ENOMEM;
 
 	ret = btrfs_insert_empty_item(trans, fs_info->extent_root, path,
-				      ins, size);
-	BUG_ON(ret);
+				      &ins, size);
+	if (ret)
+		return ret;
 
 	leaf = path->nodes[0];
 	extent_item = btrfs_item_ptr(leaf, path->slots[0],
 				     struct btrfs_extent_item);
 	btrfs_set_extent_refs(leaf, extent_item, 1);
-	btrfs_set_extent_generation(leaf, extent_item, generation);
+	btrfs_set_extent_generation(leaf, extent_item, trans->transid);
 	btrfs_set_extent_flags(leaf, extent_item,
-			       flags | BTRFS_EXTENT_FLAG_TREE_BLOCK);
+			       extent_op->flags_to_set |
+			       BTRFS_EXTENT_FLAG_TREE_BLOCK);
 
 	if (skinny_metadata) {
 		iref = (struct btrfs_extent_inline_ref *)(extent_item + 1);
 	} else {
+		struct btrfs_tree_block_info *block_info;
 		block_info = (struct btrfs_tree_block_info *)(extent_item + 1);
-		btrfs_set_tree_block_key(leaf, block_info, key);
-		btrfs_set_tree_block_level(leaf, block_info, level);
+		btrfs_set_tree_block_key(leaf, block_info, &extent_op->key);
+		btrfs_set_tree_block_level(leaf, block_info, ref->level);
 		iref = (struct btrfs_extent_inline_ref *)(block_info + 1);
 	}
 
 	btrfs_set_extent_inline_ref_type(leaf, iref, BTRFS_TREE_BLOCK_REF_KEY);
-	btrfs_set_extent_inline_ref_offset(leaf, iref, root_objectid);
+	btrfs_set_extent_inline_ref_offset(leaf, iref, ref->root);
 
 	btrfs_mark_buffer_dirty(leaf);
 	btrfs_free_path(path);
 
-	ret = update_block_group(fs_info, ins->objectid, fs_info->nodesize,
-				 1, 0);
+	ret = update_block_group(fs_info, ins.objectid, fs_info->nodesize, 1,
+				 0);
+
+	if (ref->root == BTRFS_EXTENT_TREE_OBJECTID) {
+		clear_extent_bits(&trans->fs_info->extent_ins, start, end,
+				  EXTENT_LOCKED);
+	}
+
 	return ret;
 }
 
@@ -4170,7 +4153,7 @@ static int run_delayed_tree_ref(struct btrfs_trans_handle *trans,
 	}
 	if (node->action == BTRFS_ADD_DELAYED_REF && insert_reserved) {
 		BUG_ON(!extent_op || !extent_op->update_flags);
-		ret = alloc_reserved_tree_block2(trans, node, extent_op);
+		ret = alloc_reserved_tree_block(trans, node, extent_op);
 	} else if (node->action == BTRFS_DROP_DELAYED_REF) {
 		struct btrfs_delayed_tree_ref *ref = btrfs_delayed_node_to_tree_ref(node);
 		ret =  __free_extent(trans, node->bytenr, node->num_bytes,
Signature · 1 line
2.7.4
#15 lore raw
In reply to Nikolay Borisov
Quoted 28 lines · Nikolay Borisov

Hello,

Here is the second version of the delayed refs for progs support. The first version can be found here [1]. I've taken into account all the feedback from Misono and have verified the code is working and rebased it atop btrfs-progs 4.17.1.

Changes since v1:
* Removed num_entries variable from delayed ref root

* Added a patch to refactor btrfs_write_dirty_block_groups and subsequently changed when this function is called to fix an issue reported by Misono. I verified that 'make test-fsck TEST_ENABLE_OVERRIDE=true TEST_ARGS_CHECK=--mode=lowmem' produces no errors

* Added 2 patches which remove the newly added adapter functions at the beggining of the series, following the wiring up of the delayed refs infrastructured. The first one (dealing with __free_extent2) is trivial, while the second one (for alloc_reserved_tree_block2) is a bit more involved, since I've opted to merge the two functions.

* Rebased atop latest btrfs-progs release - 4.17.1

* Dropped patches which have been merged in the mean time

[1] https://www.spinics.net/lists/linux-btrfs/msg79173.html

Quoted 10 lines

Nikolay Borisov (8):
btrfs-progs: Add __free_extent2 function
btrfs-progs: Add alloc_reserved_tree_block2 function
btrfs-progs: Add delayed refs infrastructure
btrfs-progs: Make btrfs_write_dirty_block_groups take only trans
argument
btrfs-progs: Wire up delayed refs
btrfs-progs: Remove old delayed refs infrastructure
btrfs-progs: Remove __free_extent2
btrfs-progs: Merge alloc_reserved_tree_block(2|)

For the record, the v2 patchset has been merged to devel.