Skip to contentSkip to conversation

btrfs-progs: Handle error properly in btrfs_commit_transaction()

PATCH In review Qu Wenruo 3 participants 8 messages Apr 16, 2019 – May 13, 2019 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

cover btrfs-progs: Handle error properly in btrfs_commit_transaction()

This patchset can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/commit_failure_cleanup

This patchset will address the BUG_ON() triggered in fuzz-tests/003 and fuzz-tests/009.

For proper error handling in btrfs_commit_transaction(), we need a way to clean up per-transaction data properly.

Currently we only have delayed refs which are per-transaction, so introduce a new function, btrfs_destroy_delayed_refs() to cleanup delayed refs.

Now btrfs_commit_transaction() can error out gracefully with proper cleanup.

Patch 1 and 2 are just some minor cleanups found when crafting the 3rd patch.

Qu Wenruo (3):
btrfs-progs: Remove the dead branch in btrfs_run_delayed_refs()
btrfs-progs: Refactor btrfs_finish_extent_commit()
btrfs-progs: Handle error properly in btrfs_commit_transaction()

 ctree.h       |  4 +---
 delayed-ref.c | 24 ++++++++++++++++++++++++
 delayed-ref.h |  5 +++++
 extent-tree.c | 22 +++++++++-------------
 transaction.c | 23 +++++++++++++++--------
 5 files changed, 54 insertions(+), 24 deletions(-)
Signature · 1 line
2.21.0
author
#1 lore raw

PATCH 1/3 btrfs-progs: Remove the dead branch in btrfs_run_delayed_refs()

cleanup_ref_head() will only return 0 or 1, no way to return minutes value.

So remove the dead branch.

---
 extent-tree.c | 2 --
 1 file changed, 2 deletions(-)
extent-tree.c +0 -2
index 8c9cdeff3b02..b7ece0ccc237 100644
@@ -4235,8 +4235,6 @@ int btrfs_run_delayed_refs(struct btrfs_trans_handle *trans, unsigned long nr)
 				/* We dropped our lock, we need to loop. */
 				ret = 0;
 				continue;
-			} else if (ret) {
-				return ret;
 			}
 			locked_ref = NULL;
 			continue;
Signature · 1 line
2.21.0
Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] btrfs-progs: Remove the dead branch in btrfs_run_delayed_refs()

Quoted 6 lines · Qu Wenruo

cleanup_ref_head() will only return 0 or 1, no way to return minutes value.

So remove the dead branch.

This is a left-over from the kernel copy, due to the kernel version able to return < 0 from, say, run_and_cleanup_extent_op.

Quoted 18 lines
---
 extent-tree.c | 2 --
 1 file changed, 2 deletions(-)
extent-tree.c +0 -2
index 8c9cdeff3b02..b7ece0ccc237 100644
@@ -4235,8 +4235,6 @@ int btrfs_run_delayed_refs(struct btrfs_trans_handle *trans, unsigned long nr)
 				/* We dropped our lock, we need to loop. */
 				ret = 0;
 				continue;
-			} else if (ret) {
-				return ret;
 			}
 			locked_ref = NULL;
 			continue;
author
#3 lore raw

PATCH 2/3 btrfs-progs: Refactor btrfs_finish_extent_commit()

In reply to Qu Wenruo

This patch will refactor btrfs_finish_extent_commit() by:
- Make it return void
There is no failure pattern for btrfs_finish_extent_commit(), thus it always return 0. And the caller doesn't care about the return value. So no need to return int.

- Remove @root and @unpin parameters
@root is only used to extract fs_info, which can be extracted from transaction handler already.
@unpin is always fs_info->pinned_extents.
All these parameters can be extracted from @trans, no need to pass them.

---
 ctree.h       |  4 +---
 extent-tree.c | 14 ++++++--------
 transaction.c |  3 +--
 3 files changed, 8 insertions(+), 13 deletions(-)
ctree.h +1 -3
index 3357290d7045..76f52b1c9b08 100644
@@ -2516,9 +2516,7 @@ int btrfs_free_extent(struct btrfs_trans_handle *trans,
 		      struct btrfs_root *root,
 		      u64 bytenr, u64 num_bytes, u64 parent,
 		      u64 root_objectid, u64 owner, u64 offset);
-int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans,
-			       struct btrfs_root *root,
-			       struct extent_io_tree *unpin);
+void btrfs_finish_extent_commit(struct btrfs_trans_handle *trans);
 int btrfs_inc_extent_ref(struct btrfs_trans_handle *trans,
 				struct btrfs_root *root,
 				u64 bytenr, u64 num_bytes, u64 parent,
extent-tree.c +6 -8
index b7ece0ccc237..1140ebfc61ff 100644
@@ -1977,27 +1977,25 @@ next:
 	return 0;
 }
 
-int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans,
-			       struct btrfs_root *root,
-			       struct extent_io_tree *unpin)
+void btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)
 {
 	u64 start;
 	u64 end;
 	int ret;
-	struct extent_io_tree *free_space_cache;
-	free_space_cache = &root->fs_info->free_space_cache;
+	struct btrfs_fs_info *fs_info = trans->fs_info;
+	struct extent_io_tree *free_space_cache = &fs_info->free_space_cache;
+	struct extent_io_tree *pinned_extents = &fs_info->pinned_extents;
 
 	while(1) {
-		ret = find_first_extent_bit(unpin, 0, &start, &end,
+		ret = find_first_extent_bit(pinned_extents, 0, &start, &end,
 					    EXTENT_DIRTY);
 		if (ret)
 			break;
 		update_pinned_extents(trans->fs_info, start, end + 1 - start,
 				      0);
-		clear_extent_dirty(unpin, start, end);
+		clear_extent_dirty(pinned_extents, start, end);
 		set_extent_dirty(free_space_cache, start, end);
 	}
-	return 0;
 }
 
 static int pin_down_bytes(struct btrfs_trans_handle *trans, u64 bytenr,
transaction.c +1 -2
index 3a63988b0969..8d15eb11c7b5 100644
@@ -198,8 +198,7 @@ commit_tree:
 	if (ret < 0)
 		goto out;
 	ret = write_ctree_super(trans);
-	btrfs_finish_extent_commit(trans, fs_info->extent_root,
-			           &fs_info->pinned_extents);
+	btrfs_finish_extent_commit(trans);
 	kfree(trans);
 	free_extent_buffer(root->commit_root);
 	root->commit_root = NULL;
Signature · 1 line
2.21.0
Reviewed-by on v1 2/3
#4 lore raw

Re: [PATCH 2/3] btrfs-progs: Refactor btrfs_finish_extent_commit()

Quoted 6 lines · Qu Wenruo

This patch will refactor btrfs_finish_extent_commit() by:
- Make it return void
There is no failure pattern for btrfs_finish_extent_commit(), thus it always return 0. And the caller doesn't care about the return value. So no need to return int.

Quoted 8 lines

- Remove @root and @unpin parameters
@root is only used to extract fs_info, which can be extracted from transaction handler already.
@unpin is always fs_info->pinned_extents.
All these parameters can be extracted from @trans, no need to pass them.

This also makes the function's signature identical to its kernel counterpart.

Quoted 74 lines
---
 ctree.h       |  4 +---
 extent-tree.c | 14 ++++++--------
 transaction.c |  3 +--
 3 files changed, 8 insertions(+), 13 deletions(-)
ctree.h +1 -3
index 3357290d7045..76f52b1c9b08 100644
@@ -2516,9 +2516,7 @@ int btrfs_free_extent(struct btrfs_trans_handle *trans,
 		      struct btrfs_root *root,
 		      u64 bytenr, u64 num_bytes, u64 parent,
 		      u64 root_objectid, u64 owner, u64 offset);
-int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans,
-			       struct btrfs_root *root,
-			       struct extent_io_tree *unpin);
+void btrfs_finish_extent_commit(struct btrfs_trans_handle *trans);
 int btrfs_inc_extent_ref(struct btrfs_trans_handle *trans,
 				struct btrfs_root *root,
 				u64 bytenr, u64 num_bytes, u64 parent,
extent-tree.c +6 -8
index b7ece0ccc237..1140ebfc61ff 100644
@@ -1977,27 +1977,25 @@ next:
 	return 0;
 }
 
-int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans,
-			       struct btrfs_root *root,
-			       struct extent_io_tree *unpin)
+void btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)
 {
 	u64 start;
 	u64 end;
 	int ret;
-	struct extent_io_tree *free_space_cache;
-	free_space_cache = &root->fs_info->free_space_cache;
+	struct btrfs_fs_info *fs_info = trans->fs_info;
+	struct extent_io_tree *free_space_cache = &fs_info->free_space_cache;
+	struct extent_io_tree *pinned_extents = &fs_info->pinned_extents;
 
 	while(1) {
-		ret = find_first_extent_bit(unpin, 0, &start, &end,
+		ret = find_first_extent_bit(pinned_extents, 0, &start, &end,
 					    EXTENT_DIRTY);
 		if (ret)
 			break;
 		update_pinned_extents(trans->fs_info, start, end + 1 - start,
 				      0);
-		clear_extent_dirty(unpin, start, end);
+		clear_extent_dirty(pinned_extents, start, end);
 		set_extent_dirty(free_space_cache, start, end);
 	}
-	return 0;
 }
 
 static int pin_down_bytes(struct btrfs_trans_handle *trans, u64 bytenr,
transaction.c +1 -2
index 3a63988b0969..8d15eb11c7b5 100644
@@ -198,8 +198,7 @@ commit_tree:
 	if (ret < 0)
 		goto out;
 	ret = write_ctree_super(trans);
-	btrfs_finish_extent_commit(trans, fs_info->extent_root,
-			           &fs_info->pinned_extents);
+	btrfs_finish_extent_commit(trans);
 	kfree(trans);
 	free_extent_buffer(root->commit_root);
 	root->commit_root = NULL;
author
#5 lore raw

PATCH 3/3 btrfs-progs: Handle error properly in btrfs_commit_transaction()

In reply to Qu Wenruo

[BUG]
When running fuzz-tests/003 and fuzz-tests/009, btrfs-progs will crash due to BUG_ON().

[CAUSE]
We abused BUG_ON() in btrfs_commit_transaction(), which is one of the most error prone function for fuzzed images.

Currently to cleanup the aborted transaction, we only need to clean up the only per-transaction data: delayed refs.

This patch will introduce a new function, btrfs_destroy_delayed_refs() to cleanup delayed refs when we failed to commit transaction.

With that function, we will gently destroy per-trans delayed ref, and remove the BUG_ON()s in btrfs_commit_transaction().

---
 delayed-ref.c | 24 ++++++++++++++++++++++++
 delayed-ref.h |  5 +++++
 extent-tree.c |  6 +++---
 transaction.c | 20 ++++++++++++++------
 4 files changed, 46 insertions(+), 9 deletions(-)
delayed-ref.c +24 -0
index 9974dbbd4c6b..69f8be34fe18 100644
@@ -605,3 +605,27 @@ free_ref:
 
 	return -ENOMEM;
 }
+
+void btrfs_destroy_delayed_refs(struct btrfs_trans_handle *trans)
+{
+	struct btrfs_fs_info *fs_info = trans->fs_info;
+	struct rb_node *node;
+	struct btrfs_delayed_ref_root *delayed_refs;
+
+	delayed_refs = &trans->delayed_refs;
+	if (RB_EMPTY_ROOT(&delayed_refs->href_root))
+		return;
+	while ((node = rb_first(&delayed_refs->href_root)) != NULL) {
+		struct btrfs_delayed_ref_head *head;
+		struct btrfs_delayed_ref_node *ref;
+		struct rb_node *n;
+
+		head = rb_entry(node, struct btrfs_delayed_ref_head, href_node);
+		while ((n = rb_first(&head->ref_tree)) != NULL) {
+			ref = rb_entry(n, struct btrfs_delayed_ref_node,
+				       ref_node);
+			drop_delayed_ref(trans, delayed_refs, head, ref);
+		}
+		ASSERT(cleanup_ref_head(trans, fs_info, head) == 0);
+	}
+}
delayed-ref.h +5 -0
index 30a68b2a278c..4ff3eaa929ba 100644
@@ -205,4 +205,9 @@ btrfs_delayed_node_to_tree_ref(struct btrfs_delayed_ref_node *node)
 {
 	return container_of(node, struct btrfs_delayed_tree_ref, node);
 }
+
+int cleanup_ref_head(struct btrfs_trans_handle *trans,
+		     struct btrfs_fs_info *fs_info,
+		     struct btrfs_delayed_ref_head *head);
+void btrfs_destroy_delayed_refs(struct btrfs_trans_handle *trans);
 #endif
extent-tree.c +3 -3
index 1140ebfc61ff..e62ee8c2ba13 100644
@@ -4085,9 +4085,9 @@ static void unselect_delayed_ref_head(struct btrfs_delayed_ref_root *delayed_ref
 	delayed_refs->num_heads_ready++;
 }
 
-static int cleanup_ref_head(struct btrfs_trans_handle *trans,
-			    struct btrfs_fs_info *fs_info,
-			    struct btrfs_delayed_ref_head *head)
+int cleanup_ref_head(struct btrfs_trans_handle *trans,
+		     struct btrfs_fs_info *fs_info,
+		     struct btrfs_delayed_ref_head *head)
 {
 	struct btrfs_delayed_ref_root *delayed_refs;
 
transaction.c +14 -6
index 8d15eb11c7b5..138e10f0d6cc 100644
@@ -17,6 +17,7 @@
 #include "kerncompat.h"
 #include "disk-io.h"
 #include "transaction.h"
+#include "delayed-ref.h"
 
 #include "messages.h"
 
@@ -165,7 +166,8 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 	 * consistent
 	 */
 	ret = btrfs_run_delayed_refs(trans, -1);
-	BUG_ON(ret);
+	if (ret < 0)
+		goto error;
 
 	if (root->commit_root == root->node)
 		goto commit_tree;
@@ -182,21 +184,24 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 	root->root_item.level = btrfs_header_level(root->node);
 	ret = btrfs_update_root(trans, root->fs_info->tree_root,
 				&root->root_key, &root->root_item);
-	BUG_ON(ret);
+	if (ret < 0)
+		goto error;
 
 commit_tree:
 	ret = commit_tree_roots(trans, fs_info);
-	BUG_ON(ret);
+	if (ret < 0)
+		goto error;
 	/*
 	 * Ensure that all committed roots are properly accounted in the
 	 * extent tree
 	 */
 	ret = btrfs_run_delayed_refs(trans, -1);
-	BUG_ON(ret);
+	if (ret < 0)
+		goto error;
 	btrfs_write_dirty_block_groups(trans);
 	__commit_transaction(trans, root);
 	if (ret < 0)
-		goto out;
+		goto error;
 	ret = write_ctree_super(trans);
 	btrfs_finish_extent_commit(trans);
 	kfree(trans);
@@ -204,7 +209,10 @@ commit_tree:
 	root->commit_root = NULL;
 	fs_info->running_transaction = NULL;
 	fs_info->last_trans_committed = transid;
-out:
+	return ret;
+error:
+	btrfs_destroy_delayed_refs(trans);
+	free(trans);
 	return ret;
 }
 
Signature · 1 line
2.21.0
Reviewed-by on v1 3/3
#6 lore raw
Quoted 11 lines · Qu Wenruo

[BUG]
When running fuzz-tests/003 and fuzz-tests/009, btrfs-progs will crash due to BUG_ON().

[CAUSE]
We abused BUG_ON() in btrfs_commit_transaction(), which is one of the most error prone function for fuzzed images.

Currently to cleanup the aborted transaction, we only need to clean up the only per-transaction data: delayed refs.

Quoted 7 lines

This patch will introduce a new function, btrfs_destroy_delayed_refs() to cleanup delayed refs when we failed to commit transaction.

With that function, we will gently destroy per-trans delayed ref, and remove the BUG_ON()s in btrfs_commit_transaction().

Though see below for a suggestion of further cleanup

Quoted 58 lines
---
 delayed-ref.c | 24 ++++++++++++++++++++++++
 delayed-ref.h |  5 +++++
 extent-tree.c |  6 +++---
 transaction.c | 20 ++++++++++++++------
 4 files changed, 46 insertions(+), 9 deletions(-)
delayed-ref.c +24 -0
index 9974dbbd4c6b..69f8be34fe18 100644
@@ -605,3 +605,27 @@ free_ref:
 
 	return -ENOMEM;
 }
+
+void btrfs_destroy_delayed_refs(struct btrfs_trans_handle *trans)
+{
+	struct btrfs_fs_info *fs_info = trans->fs_info;
+	struct rb_node *node;
+	struct btrfs_delayed_ref_root *delayed_refs;
+
+	delayed_refs = &trans->delayed_refs;
+	if (RB_EMPTY_ROOT(&delayed_refs->href_root))
+		return;
+	while ((node = rb_first(&delayed_refs->href_root)) != NULL) {
+		struct btrfs_delayed_ref_head *head;
+		struct btrfs_delayed_ref_node *ref;
+		struct rb_node *n;
+
+		head = rb_entry(node, struct btrfs_delayed_ref_head, href_node);
+		while ((n = rb_first(&head->ref_tree)) != NULL) {
+			ref = rb_entry(n, struct btrfs_delayed_ref_node,
+				       ref_node);
+			drop_delayed_ref(trans, delayed_refs, head, ref);
+		}
+		ASSERT(cleanup_ref_head(trans, fs_info, head) == 0);
+	}
+}
delayed-ref.h +5 -0
index 30a68b2a278c..4ff3eaa929ba 100644
@@ -205,4 +205,9 @@ btrfs_delayed_node_to_tree_ref(struct btrfs_delayed_ref_node *node)
 {
 	return container_of(node, struct btrfs_delayed_tree_ref, node);
 }
+
+int cleanup_ref_head(struct btrfs_trans_handle *trans,
+		     struct btrfs_fs_info *fs_info,
+		     struct btrfs_delayed_ref_head *head);
+void btrfs_destroy_delayed_refs(struct btrfs_trans_handle *trans);
 #endif
extent-tree.c +0 -0
index 1140ebfc61ff..e62ee8c2ba13 100644

extent-tree.c static void unselect_delayed_ref_head(struct btrfs_delayed_ref_root *delayed_ref · line 4090

Quoted 9 lines
extent-tree.c +3 -3
@@ -4085,9 +4085,9 @@ static void unselect_delayed_ref_head(struct btrfs_delayed_ref_root *delayed_ref
 	delayed_refs->num_heads_ready++;
 }
 
-static int cleanup_ref_head(struct btrfs_trans_handle *trans,
-			    struct btrfs_fs_info *fs_info,
-			    struct btrfs_delayed_ref_head *head)
+int cleanup_ref_head(struct btrfs_trans_handle *trans,
+		     struct btrfs_fs_info *fs_info,
+		     struct btrfs_delayed_ref_head *head)

You can create a followup patch that removes the fs_info argument from cleanup_ref_head to make it identical to the kernel version.

Quoted 67 lines
extent-tree.c +0 -0
 {
 	struct btrfs_delayed_ref_root *delayed_refs;
 
transaction.c +14 -6
index 8d15eb11c7b5..138e10f0d6cc 100644
@@ -17,6 +17,7 @@
 #include "kerncompat.h"
 #include "disk-io.h"
 #include "transaction.h"
+#include "delayed-ref.h"
 
 #include "messages.h"
 
@@ -165,7 +166,8 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 	 * consistent
 	 */
 	ret = btrfs_run_delayed_refs(trans, -1);
-	BUG_ON(ret);
+	if (ret < 0)
+		goto error;
 
 	if (root->commit_root == root->node)
 		goto commit_tree;
@@ -182,21 +184,24 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 	root->root_item.level = btrfs_header_level(root->node);
 	ret = btrfs_update_root(trans, root->fs_info->tree_root,
 				&root->root_key, &root->root_item);
-	BUG_ON(ret);
+	if (ret < 0)
+		goto error;
 
 commit_tree:
 	ret = commit_tree_roots(trans, fs_info);
-	BUG_ON(ret);
+	if (ret < 0)
+		goto error;
 	/*
 	 * Ensure that all committed roots are properly accounted in the
 	 * extent tree
 	 */
 	ret = btrfs_run_delayed_refs(trans, -1);
-	BUG_ON(ret);
+	if (ret < 0)
+		goto error;
 	btrfs_write_dirty_block_groups(trans);
 	__commit_transaction(trans, root);
 	if (ret < 0)
-		goto out;
+		goto error;
 	ret = write_ctree_super(trans);
 	btrfs_finish_extent_commit(trans);
 	kfree(trans);
@@ -204,7 +209,10 @@ commit_tree:
 	root->commit_root = NULL;
 	fs_info->running_transaction = NULL;
 	fs_info->last_trans_committed = transid;
-out:
+	return ret;
+error:
+	btrfs_destroy_delayed_refs(trans);
+	free(trans);
 	return ret;
 }
 
#7 lore raw
In reply to Qu Wenruo
Quoted 16 lines · Qu Wenruo

This patchset can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/commit_failure_cleanup

This patchset will address the BUG_ON() triggered in fuzz-tests/003 and fuzz-tests/009.

For proper error handling in btrfs_commit_transaction(), we need a way to clean up per-transaction data properly.

Currently we only have delayed refs which are per-transaction, so introduce a new function, btrfs_destroy_delayed_refs() to cleanup delayed refs.

Now btrfs_commit_transaction() can error out gracefully with proper cleanup.

Quoted 7 lines

Patch 1 and 2 are just some minor cleanups found when crafting the 3rd patch.

Qu Wenruo (3):
btrfs-progs: Remove the dead branch in btrfs_run_delayed_refs()
btrfs-progs: Refactor btrfs_finish_extent_commit()
btrfs-progs: Handle error properly in btrfs_commit_transaction()

Added to devel, thanks.