Skip to contentSkip to conversation

btrfs: Turn fs_roots_radix in btrfs_fs_info into an XArray

PATCH Superseded Gabriel Niebler 3 participants 9 messages Apr 26, 2022 – May 3, 2022 Btrfs file system

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 btrfs: Turn fs_roots_radix in btrfs_fs_info into an XArray

… rename it to simply fs_roots and adjust all usages of this object to use the XArray API, because it is notionally easier to use and unserstand, as it provides array semantics, and also takes care of locking for us, further simplifying the code.

Also do some refactoring, esp. where the API change requires largely rewriting some functions, anyway.

---
 fs/btrfs/ctree.h       |   5 +-
 fs/btrfs/disk-io.c     | 176 ++++++++++++++++++++---------------------
 fs/btrfs/inode.c       |  13 +--
 fs/btrfs/transaction.c |  67 +++++++---------
 4 files changed, 126 insertions(+), 135 deletions(-)
fs/btrfs/ctree.h +3 -2
index b7631b88426e..c1c0a2f5a7b6 100644
@@ -675,8 +675,9 @@ struct btrfs_fs_info {
 	rwlock_t global_root_lock;
 	struct rb_root global_root_tree;
 
-	spinlock_t fs_roots_radix_lock;
-	struct radix_tree_root fs_roots_radix;
+	/* The XArray that holds all the FS roots */
+	spinlock_t fs_roots_lock;
+	struct xarray fs_roots;
 
 	/* block group cache stuff */
 	spinlock_t block_group_cache_lock;
fs/btrfs/disk-io.c +87 -89

279 lines not on this page · raw message

fs/btrfs/inode.c +7 -6
index 5082b9c70f8c..d0ef3a17ce11 100644
@@ -3494,6 +3494,7 @@ int btrfs_orphan_cleanup(struct btrfs_root *root)
 	u64 last_objectid = 0;
 	int ret = 0, nr_unlink = 0;
 
+	/* Bail out if the cleanup is already running. */
 	if (test_and_set_bit(BTRFS_ROOT_ORPHAN_CLEANUP, &root->state))
 		return 0;
 
@@ -3576,17 +3577,17 @@ int btrfs_orphan_cleanup(struct btrfs_root *root)
 			 *
 			 * btrfs_find_orphan_roots() ran before us, which has
 			 * found all deleted roots and loaded them into
-			 * fs_info->fs_roots_radix. So here we can find if an
+			 * fs_info->fs_roots. So here we can find if an
 			 * orphan item corresponds to a deleted root by looking
-			 * up the root from that radix tree.
+			 * up the root from that xarray.
 			 */
 
-			spin_lock(&fs_info->fs_roots_radix_lock);
-			dead_root = radix_tree_lookup(&fs_info->fs_roots_radix,
-							 (unsigned long)found_key.objectid);
+			spin_lock(&fs_info->fs_roots_lock);
+			dead_root = xa_load(&fs_info->fs_roots,
+					    (unsigned long)found_key.objectid);
 			if (dead_root && btrfs_root_refs(&dead_root->root_item) == 0)
 				is_dead_root = 1;
-			spin_unlock(&fs_info->fs_roots_radix_lock);
+			spin_unlock(&fs_info->fs_roots_lock);
 
 			if (is_dead_root) {
 				/* prevent this orphan from being found again */
fs/btrfs/transaction.c +29 -38

119 lines not on this page · raw message

Signature · 1 line
2.35.3
#1 lore raw

Am 26.04.22 um 23:45 schrieb Gabriel Niebler:

Quoted 25 lines

… rename it to simply fs_roots and adjust all usages of this object to use the XArray API, because it is notionally easier to use and unserstand, as it provides array semantics, and also takes care of locking for us, further simplifying the code.

Also do some refactoring, esp. where the API change requires largely rewriting some functions, anyway.

---
  fs/btrfs/ctree.h       |   5 +-
  fs/btrfs/disk-io.c     | 176 ++++++++++++++++++++---------------------
  fs/btrfs/inode.c       |  13 +--
  fs/btrfs/transaction.c |  67 +++++++---------
  4 files changed, 126 insertions(+), 135 deletions(-)

[...]

fs/btrfs/disk-io.c +1 -3
@@ -2346,28 +2340,23 @@ void btrfs_put_root(struct btrfs_root *root)
 
  void btrfs_free_fs_roots(struct btrfs_fs_info *fs_info)
  {
-	int ret;
-	struct btrfs_root *gang[8];
-	int i;
+	struct btrfs_root *root;

fs/btrfs/disk-io.c void btrfs_put_root(struct btrfs_root *root) · line 2353

Quoted 10 lines
fs/btrfs/disk-io.c +4 -3
+	unsigned long index = 0;
 
  	while (!list_empty(&fs_info->dead_roots)) {
[...]
-		if (test_bit(BTRFS_ROOT_IN_RADIX, &gang[0]->state))
-			btrfs_drop_and_free_fs_root(fs_info, gang[0]);
-		btrfs_put_root(gang[0]);
+		if (test_bit(BTRFS_ROOT_IN_RADIX, &root->state))
+			btrfs_drop_and_free_fs_root(fs_info, root);
+		btrfs_put_root(root);

It occurs to me that BTRFS_ROOT_IN_RADIX should probably be renamed to something else within or following this patch...

... but what to rename it /to/?

Naming things is hard. Here are some ideas I've had:

BTRFS_ROOT_IN_XARRAY is obvious, but it also includes kind of a needless implementation detail.

BTRFS_ROOT_IN_(FS_)ROOTS would be technically accurate, but might be confusing for the reader (e.g. if they don't know about btrfs_fs_indo->fs_roots of the top off their head).

BTRFS_ROOT_IN_FS_INFO_ROOTS makes that a bit clearer (someone might think it refers to fbtrfs_fs_info->roots, but when they find that doesn't exist they'd quickly catch on, I think), but a bit lengthy.

BTRFS_ROOT_IN_FS_INFO_FS_ROOTS is probably the most accurate, but really quite long...

Does anyone else have any ideas?

#2 lore raw
Quoted 12 lines · Gabriel Niebler

fs/btrfs/disk-io.c void btrfs_put_root(struct btrfs_root *root) · line 2353

Quoted 8 lines

while (!list_empty(&fs_info->dead_roots)) {
[...]

fs/btrfs/disk-io.c +3 -3
-		if (test_bit(BTRFS_ROOT_IN_RADIX, &gang[0]->state))
-			btrfs_drop_and_free_fs_root(fs_info, gang[0]);
-		btrfs_put_root(gang[0]);
+		if (test_bit(BTRFS_ROOT_IN_RADIX, &root->state))
+			btrfs_drop_and_free_fs_root(fs_info, root);
+		btrfs_put_root(root);

It occurs to me that BTRFS_ROOT_IN_RADIX should probably be renamed to something else within or following this patch...

Quoted 1 line

... but what to rename it /to/?

Good point about the naming but I don't have a concrete idea. The naming could reflect the status of the entry rather than where is it tracked.

Quoted 4 lines

Naming things is hard. Here are some ideas I've had:

BTRFS_ROOT_IN_XARRAY is obvious, but it also includes kind of a needless implementation detail.

This could be a fallback that at least follows the same pattern but yeah it's not the best one.

#3 lore raw
In reply to Gabriel Niebler
Quoted 5 lines · Gabriel Niebler

Naming things is hard. Here are some ideas I've had:

BTRFS_ROOT_IN_XARRAY is obvious, but it also includes kind of a needless implementation detail.

Quoted 7 lines

BTRFS_ROOT_IN_(FS_)ROOTS would be technically accurate, but might be confusing for the reader (e.g. if they don't know about btrfs_fs_indo->fs_roots of the top off their head).

BTRFS_ROOT_IN_FS_INFO_ROOTS makes that a bit clearer (someone might think it refers to fbtrfs_fs_info->roots, but when they find that doesn't exist they'd quickly catch on, I think), but a bit lengthy.

BTRFS_ROOT_IN_FS_INFO_FS_ROOTS is probably the most accurate, but really quite long...

Does anyone else have any ideas?

BTRFS_ROOT_ACCESSIBLE/BTRFS_ROOT_REGISTERED/BTRFS_ROOT_PUBLIC/BTRFS_ROOT_INITIALIZED

because when a root goes into the root registry (either xarray or radix) it's essentially published to this "registry" and can be referenced by other parts of the filesystem. And this registration is part of the root's initialization sequence.

#4 lore raw
In reply to Gabriel Niebler
Quoted 9 lines · Gabriel Niebler

… rename it to simply fs_roots and adjust all usages of this object to use the XArray API, because it is notionally easier to use and unserstand, as it provides array semantics, and also takes care of locking for us, further simplifying the code.

Also do some refactoring, esp. where the API change requires largely rewriting some functions, anyway.

I've lightly reviewed the patch so I can put add it for testing though I'll keep the review open for Nikolay.

Otherwise, this is the last conversion of radix_tree_root, thanks. There's no use of the radix_tree_* API anymore.

#5 lore raw
In reply to Gabriel Niebler
Quoted 16 lines · Gabriel Niebler

… rename it to simply fs_roots and adjust all usages of this object to use the XArray API, because it is notionally easier to use and unserstand, as it provides array semantics, and also takes care of locking for us, further simplifying the code.

Also do some refactoring, esp. where the API change requires largely rewriting some functions, anyway.

---
  fs/btrfs/ctree.h       |   5 +-
  fs/btrfs/disk-io.c     | 176 ++++++++++++++++++++---------------------
  fs/btrfs/inode.c       |  13 +--
  fs/btrfs/transaction.c |  67 +++++++---------
  4 files changed, 126 insertions(+), 135 deletions(-)

<snip>

Quoted 25 lines
fs/btrfs/disk-io.c +8 -9
@@ -2346,28 +2340,23 @@ void btrfs_put_root(struct btrfs_root *root)
 
  void btrfs_free_fs_roots(struct btrfs_fs_info *fs_info)
  {
-	int ret;
-	struct btrfs_root *gang[8];
-	int i;
+	struct btrfs_root *root;
+	unsigned long index = 0;
 
  	while (!list_empty(&fs_info->dead_roots)) {
-		gang[0] = list_entry(fs_info->dead_roots.next,
-				     struct btrfs_root, root_list);
-		list_del(&gang[0]->root_list);
+		root = list_entry(fs_info->dead_roots.next,
+				  struct btrfs_root, root_list);
+		list_del(&root->root_list);
 
-		if (test_bit(BTRFS_ROOT_IN_RADIX, &gang[0]->state))
-			btrfs_drop_and_free_fs_root(fs_info, gang[0]);
-		btrfs_put_root(gang[0]);
+		if (test_bit(BTRFS_ROOT_IN_RADIX, &root->state))
+			btrfs_drop_and_free_fs_root(fs_info, root);
+		btrfs_put_root(root);
  	}

fs/btrfs/disk-io.c void btrfs_put_root(struct btrfs_root *root) · line 2356

Quoted 10 lines
fs/btrfs/disk-io.c +1 -8
 
-	while (1) {
-		ret = radix_tree_gang_lookup(&fs_info->fs_roots_radix,
-					     (void **)gang, 0,
-					     ARRAY_SIZE(gang));
-		if (!ret)
-			break;
-		for (i = 0; i < ret; i++)
-			btrfs_drop_and_free_fs_root(fs_info, gang[i]);
+	while (!xa_empty(&fs_info->fs_roots))  > +		xa_for_each(&fs_info->fs_roots, index, root) {

Why do you need the nested loop structure? xa_for_each should be sufficient as btrfs_free_fs_roots happens when the fs is being unmounted and we can't get another loop added after having done xa_for_each. And even if it could the nested loop structure won't help because we don't hold any locks, so hypothetically speaking, even if there was a race using nested loops doesn't solve it.

TLDR: Leavint this xa_for_each should be sufficient.

Quoted 5 lines
fs/btrfs/disk-io.c +2 -0
+			btrfs_drop_and_free_fs_root(fs_info, root);
+		}
  	}
  }
 

<snip>

fs/btrfs/disk-io.c void btrfs_drop_and_free_fs_root(struct btrfs_fs_info *fs_info, · line 4485

Quoted 10 lines
fs/btrfs/disk-io.c +3 -3
@@ -4491,12 +4480,12 @@ void btrfs_drop_and_free_fs_root(struct btrfs_fs_info *fs_info,
  {
  	bool drop_ref = false;
 
-	spin_lock(&fs_info->fs_roots_radix_lock);
-	radix_tree_delete(&fs_info->fs_roots_radix,
-			  (unsigned long)root->root_key.objectid);
+	spin_lock(&fs_info->fs_roots_lock);
+	xa_erase(&fs_info->fs_roots,
+		 (unsigned long)root->root_key.objectid);

nit: No need to have the 2nd argument on a new line, as the line is short enough.

Quoted 8 lines
fs/btrfs/disk-io.c +1 -1
  	if (test_and_clear_bit(BTRFS_ROOT_IN_RADIX, &root->state))
  		drop_ref = true;
-	spin_unlock(&fs_info->fs_roots_radix_lock);
+	spin_unlock(&fs_info->fs_roots_lock);
 
  	if (BTRFS_FS_ERROR(fs_info)) {
  		ASSERT(root->log_root == NULL);

fs/btrfs/disk-io.c void btrfs_drop_and_free_fs_root(struct btrfs_fs_info *fs_info, · line 4506

Quoted 9 lines
fs/btrfs/disk-io.c +3 -3
@@ -4512,50 +4501,54 @@ void btrfs_drop_and_free_fs_root(struct btrfs_fs_info *fs_info,
 
  int btrfs_cleanup_fs_roots(struct btrfs_fs_info *fs_info)
  {
-	u64 root_objectid = 0;
-	struct btrfs_root *gang[8];
-	int i = 0;
+	struct btrfs_root *roots[8];
+	unsigned long index = 0;
+	int i;

nit: This can be defined into the 2 loops that use the variable.

Quoted 80 lines
fs/btrfs/disk-io.c +30 -28
  	int err = 0;
-	unsigned int ret = 0;
+	int grabbed;
 
  	while (1) {
-		spin_lock(&fs_info->fs_roots_radix_lock);
-		ret = radix_tree_gang_lookup(&fs_info->fs_roots_radix,
-					     (void **)gang, root_objectid,
-					     ARRAY_SIZE(gang));
-		if (!ret) {
-			spin_unlock(&fs_info->fs_roots_radix_lock);
+		struct btrfs_root *root;
+
+		spin_lock(&fs_info->fs_roots_lock);
+		if (!xa_find(&fs_info->fs_roots, &index,
+			     ULONG_MAX, XA_PRESENT)) {
+			spin_unlock(&fs_info->fs_roots_lock);
  			break;
  		}
-		root_objectid = gang[ret - 1]->root_key.objectid + 1;
 
-		for (i = 0; i < ret; i++) {
-			/* Avoid to grab roots in dead_roots */
-			if (btrfs_root_refs(&gang[i]->root_item) == 0) {
-				gang[i] = NULL;
-				continue;
+		grabbed = 0;
+		xa_for_each_start(&fs_info->fs_roots, index, root,
+				  index) {
+			/* Avoid grabbing roots in dead_roots */
+			if (btrfs_root_refs(&root->root_item) == 0) {
+				roots[grabbed] = NULL;
+			} else {
+				/* Grab all the search results for later use */
+				roots[grabbed] = btrfs_grab_root(root);
  			}
-			/* grab all the search result for later use */
-			gang[i] = btrfs_grab_root(gang[i]);
+			grabbed++;
+			if (grabbed >= ARRAY_SIZE(roots))
+				break;
  		}
-		spin_unlock(&fs_info->fs_roots_radix_lock);
+		spin_unlock(&fs_info->fs_roots_lock);
 
-		for (i = 0; i < ret; i++) {
-			if (!gang[i])
+		for (i = 0; i < grabbed; i++) {
+			if (!roots[i])
  				continue;
-			root_objectid = gang[i]->root_key.objectid;
-			err = btrfs_orphan_cleanup(gang[i]);
+			index = roots[i]->root_key.objectid;
+			err = btrfs_orphan_cleanup(roots[i]);
  			if (err)
  				break;
-			btrfs_put_root(gang[i]);
+			btrfs_put_root(roots[i]);
  		}
-		root_objectid++;
+		index++;
  	}
 
-	/* release the uncleaned roots due to error */
-	for (; i < ret; i++) {
-		if (gang[i])
-			btrfs_put_root(gang[i]);
+	/* Release the roots that remain uncleaned due to error */
+	for (; i < grabbed; i++) {
+		if (roots[i])
+			btrfs_put_root(roots[i]);
  	}
  	return err;
  }
@@ -4872,31 +4865,36 @@ static void btrfs_error_commit_super(struct btrfs_fs_info *fs_info)
 
  static void btrfs_drop_all_logs(struct btrfs_fs_info *fs_info)
  {
-	struct btrfs_root *gang[8];
-	u64 root_objectid = 0;

fs/btrfs/disk-io.c static void btrfs_error_commit_super(struct btrfs_fs_info *fs_info) · line 4872

Quoted 10 lines
fs/btrfs/disk-io.c +4 -5
-	int ret;
+	unsigned long index = 0;
 
-	spin_lock(&fs_info->fs_roots_radix_lock);
-	while ((ret = radix_tree_gang_lookup(&fs_info->fs_roots_radix,
-					     (void **)gang, root_objectid,
-					     ARRAY_SIZE(gang))) != 0) {
+	spin_lock(&fs_info->fs_roots_lock);
+	while (xa_find(&fs_info->fs_roots,
+		       &index, ULONG_MAX, XA_PRESENT)) {

nit: No need to split across two lines as it's short enough.

fs/btrfs/disk-io.c static void btrfs_error_commit_super(struct btrfs_fs_info *fs_info) · line 4875

Quoted 3 lines
fs/btrfs/disk-io.c +2 -0
+		struct btrfs_root *root;
+		struct btrfs_root *roots[8];
  		int i;

nit: This can probably be defined inside the for loop

Quoted 41 lines
fs/btrfs/disk-io.c +17 -11
+		int grabbed = 0;
 
-		for (i = 0; i < ret; i++)
-			gang[i] = btrfs_grab_root(gang[i]);
-		spin_unlock(&fs_info->fs_roots_radix_lock);
+		xa_for_each_start(&fs_info->fs_roots, index, root,
+				  index) {
+			roots[grabbed] = btrfs_grab_root(root);
+			grabbed++;
+			if (grabbed >= ARRAY_SIZE(roots))
+				break;
+		}
+		spin_unlock(&fs_info->fs_roots_lock);
 
-		for (i = 0; i < ret; i++) {
-			if (!gang[i])
+		for (i = 0; i < grabbed; i++) {
+			if (!roots[i])
  				continue;
-			root_objectid = gang[i]->root_key.objectid;
-			btrfs_free_log(NULL, gang[i]);
-			btrfs_put_root(gang[i]);
+			index = roots[i]->root_key.objectid;
+			btrfs_free_log(NULL, roots[i]);
+			btrfs_put_root(roots[i]);
  		}
-		root_objectid++;
-		spin_lock(&fs_info->fs_roots_radix_lock);
+		index++;
+		spin_lock(&fs_info->fs_roots_lock);
  	}
-	spin_unlock(&fs_info->fs_roots_radix_lock);
+	spin_unlock(&fs_info->fs_roots_lock);
  	btrfs_free_log_root_tree(NULL, fs_info);
  }
 
fs/btrfs/inode.c +0 -0
index 5082b9c70f8c..d0ef3a17ce11 100644

fs/btrfs/inode.c int btrfs_orphan_cleanup(struct btrfs_root *root) · line 3497

Quoted 4 lines
fs/btrfs/inode.c +1 -0
@@ -3494,6 +3494,7 @@ int btrfs_orphan_cleanup(struct btrfs_root *root)
  	u64 last_objectid = 0;
  	int ret = 0, nr_unlink = 0;
 
+	/* Bail out if the cleanup is already running. */

nit: This seems like an irrelevant change.

Quoted 3 lines
fs/btrfs/inode.c +0 -0
  	if (test_and_set_bit(BTRFS_ROOT_ORPHAN_CLEANUP, &root->state))
  		return 0;
 

<snip>

Quoted 20 lines
fs/btrfs/inode.c +2 -7
@@ -1404,9 +1402,8 @@ void btrfs_add_dead_root(struct btrfs_root *root)
  static noinline int commit_fs_roots(struct btrfs_trans_handle *trans)
  {
  	struct btrfs_fs_info *fs_info = trans->fs_info;
-	struct btrfs_root *gang[8];
-	int i;
-	int ret;
+	struct btrfs_root *root;
+	unsigned long index;
 
  	/*
  	 * At this point no one can be using this transaction to modify any tree
@@ -1414,17 +1411,11 @@ static noinline int commit_fs_roots(struct btrfs_trans_handle *trans)
  	 */
  	ASSERT(trans->transaction->state == TRANS_STATE_COMMIT_DOING);
 
-	spin_lock(&fs_info->fs_roots_radix_lock);
-	while (1) {
-		ret = radix_tree_gang_lookup_tag(&fs_info->fs_roots_radix,
-						 (void **)gang, 0,

fs/btrfs/inode.c static noinline int commit_fs_roots(struct btrfs_trans_handle *trans) · line 1416

Quoted 10 lines
fs/btrfs/inode.c +3 -7
-						 ARRAY_SIZE(gang),
-						 BTRFS_ROOT_TRANS_TAG);
-		if (ret == 0)
-			break;
-		for (i = 0; i < ret; i++) {
-			struct btrfs_root *root = gang[i];
-			int ret2;
+	spin_lock(&fs_info->fs_roots_lock);
+	while (xa_marked(&fs_info->fs_roots, BTRFS_ROOT_TRANS_TAG)) {
+		xa_for_each_marked(&fs_info->fs_roots, index, root,

While the while/xa_for_each_marked and not straight xa_for_each_marked?

Quoted 5 lines
fs/btrfs/inode.c +2 -0
+				   BTRFS_ROOT_TRANS_TAG) {
+			int ret;
 
  			/*
  			 * At this point we can neither have tasks logging inodes

<snip>

#6 lore raw

Am 28.04.22 um 13:59 schrieb Nikolay Borisov:

Quoted 46 lines · Gabriel Niebler
Quoted 16 lines

… rename it to simply fs_roots and adjust all usages of this object to use the XArray API, because it is notionally easier to use and unserstand, as it provides array semantics, and also takes care of locking for us, further simplifying the code.

Also do some refactoring, esp. where the API change requires largely rewriting some functions, anyway.

---
  fs/btrfs/ctree.h       |   5 +-
  fs/btrfs/disk-io.c     | 176 ++++++++++++++++++++--------------------- fs/btrfs/inode.c       |  13 +--
  fs/btrfs/transaction.c |  67 +++++++---------

  4 files changed, 126 insertions(+), 135 deletions(-)

<snip>

Quoted 26 lines
fs/btrfs/disk-io.c +8 -13
@@ -2346,28 +2340,23 @@ void btrfs_put_root(struct btrfs_root *root)
   void btrfs_free_fs_roots(struct btrfs_fs_info *fs_info)
   {
-    int ret;
-    struct btrfs_root *gang[8];
-    int i;
+    struct btrfs_root *root;
+    unsigned long index = 0;
       while (!list_empty(&fs_info->dead_roots)) {
-        gang[0] = list_entry(fs_info->dead_roots.next,
-                     struct btrfs_root, root_list);
-        list_del(&gang[0]->root_list);
+        root = list_entry(fs_info->dead_roots.next,
+                  struct btrfs_root, root_list);
+        list_del(&root->root_list);
-        if (test_bit(BTRFS_ROOT_IN_RADIX, &gang[0]->state))
-            btrfs_drop_and_free_fs_root(fs_info, gang[0]);
-        btrfs_put_root(gang[0]);
+        if (test_bit(BTRFS_ROOT_IN_RADIX, &root->state))
+            btrfs_drop_and_free_fs_root(fs_info, root);
+        btrfs_put_root(root);
       }
-    while (1) {
-        ret = radix_tree_gang_lookup(&fs_info->fs_roots_radix,
-                         (void **)gang, 0,
-                         ARRAY_SIZE(gang));
Quoted 8 lines

fs/btrfs/disk-io.c void btrfs_put_root(struct btrfs_root *root)

Quoted 6 lines
fs/btrfs/disk-io.c +1 -4
-        if (!ret)
-            break;
-        for (i = 0; i < ret; i++)
-            btrfs_drop_and_free_fs_root(fs_info, gang[i]);
+    while (!xa_empty(&fs_info->fs_roots))  > +        
 xa_for_each(&fs_info->fs_roots, index, root) {

Why do you need the nested loop structure? xa_for_each should be sufficient as btrfs_free_fs_roots happens when the fs is being unmounted and we can't get another loop added after having done xa_for_each. And even if it could the nested loop structure won't help because we don't hold any locks, so hypothetically speaking, even if there was a race using nested loops doesn't solve it.

I wasn't sure if the old code was implemented like this only because of the way radix_tree_gang_lookup works, or if there was another reason, like preventing a possible race condition (e.g. when another root gets inserted), or perhaps as a retry mechanism in case of delays/faillures in root removal.

Quoted 1 line

TLDR: Leavint this xa_for_each should be sufficient.

Acknowledged.

Quoted 7 lines
Quoted 4 lines
fs/btrfs/disk-io.c +2 -0
+            btrfs_drop_and_free_fs_root(fs_info, root);
+        }
       }
   }

<snip>

Quoted 12 lines

fs/btrfs/disk-io.c void btrfs_drop_and_free_fs_root(struct · line 4485

Quoted 10 lines
fs/btrfs/disk-io.c +3 -3
@@ -4491,12 +4480,12 @@ void btrfs_drop_and_free_fs_root(struct 
 btrfs_fs_info *fs_info,
   {
       bool drop_ref = false;
-    spin_lock(&fs_info->fs_roots_radix_lock);
-    radix_tree_delete(&fs_info->fs_roots_radix,
-              (unsigned long)root->root_key.objectid);
+    spin_lock(&fs_info->fs_roots_lock);
+    xa_erase(&fs_info->fs_roots,
+         (unsigned long)root->root_key.objectid);

nit: No need to have the 2nd argument on a new line, as the line is short enough.

Oh yeah, sorry... I had used fs_roots_xarray originally, then renamed it before submission. I thought I had checked all the lines for possible shortening, but it looks like I missed some.

Thanks for noticing!

Quoted 9 lines
Quoted 9 lines
fs/btrfs/disk-io.c +1 -1
       if (test_and_clear_bit(BTRFS_ROOT_IN_RADIX, &root->state))
           drop_ref = true;
-    spin_unlock(&fs_info->fs_roots_radix_lock);
+    spin_unlock(&fs_info->fs_roots_lock);
       if (BTRFS_FS_ERROR(fs_info)) {
           ASSERT(root->log_root == NULL);
@@ -4512,50 +4501,54 @@ void btrfs_drop_and_free_fs_root(struct 
 btrfs_fs_info *fs_info,
   int btrfs_cleanup_fs_roots(struct btrfs_fs_info *fs_info)
Quoted 8 lines

fs/btrfs/disk-io.c void btrfs_drop_and_free_fs_root(struct btrfs_fs_info *fs_info, · line 4506

Quoted 7 lines
fs/btrfs/disk-io.c +3 -3
   {
-    u64 root_objectid = 0;
-    struct btrfs_root *gang[8];
-    int i = 0;
+    struct btrfs_root *roots[8];
+    unsigned long index = 0;
+    int i;

nit: This can be defined into the 2 loops that use the variable.

Sure, why not.

Quoted 79 lines
Quoted 79 lines
fs/btrfs/disk-io.c +31 -30
       int err = 0;
-    unsigned int ret = 0;
+    int grabbed;
       while (1) {
-        spin_lock(&fs_info->fs_roots_radix_lock);
-        ret = radix_tree_gang_lookup(&fs_info->fs_roots_radix,
-                         (void **)gang, root_objectid,
-                         ARRAY_SIZE(gang));
-        if (!ret) {
-            spin_unlock(&fs_info->fs_roots_radix_lock);
+        struct btrfs_root *root;
+
+        spin_lock(&fs_info->fs_roots_lock);
+        if (!xa_find(&fs_info->fs_roots, &index,
+                 ULONG_MAX, XA_PRESENT)) {
+            spin_unlock(&fs_info->fs_roots_lock);
               break;
           }
-        root_objectid = gang[ret - 1]->root_key.objectid + 1;
-        for (i = 0; i < ret; i++) {
-            /* Avoid to grab roots in dead_roots */
-            if (btrfs_root_refs(&gang[i]->root_item) == 0) {
-                gang[i] = NULL;
-                continue;
+        grabbed = 0;
+        xa_for_each_start(&fs_info->fs_roots, index, root,
+                  index) {
+            /* Avoid grabbing roots in dead_roots */
+            if (btrfs_root_refs(&root->root_item) == 0) {
+                roots[grabbed] = NULL;
+            } else {
+                /* Grab all the search results for later use */
+                roots[grabbed] = btrfs_grab_root(root);
               }
-            /* grab all the search result for later use */
-            gang[i] = btrfs_grab_root(gang[i]);
+            grabbed++;
+            if (grabbed >= ARRAY_SIZE(roots))
+                break;
           }
-        spin_unlock(&fs_info->fs_roots_radix_lock);
+        spin_unlock(&fs_info->fs_roots_lock);
-        for (i = 0; i < ret; i++) {
-            if (!gang[i])
+        for (i = 0; i < grabbed; i++) {
+            if (!roots[i])
                   continue;
-            root_objectid = gang[i]->root_key.objectid;
-            err = btrfs_orphan_cleanup(gang[i]);
+            index = roots[i]->root_key.objectid;
+            err = btrfs_orphan_cleanup(roots[i]);
               if (err)
                   break;
-            btrfs_put_root(gang[i]);
+            btrfs_put_root(roots[i]);
           }
-        root_objectid++;
+        index++;
       }
-    /* release the uncleaned roots due to error */
-    for (; i < ret; i++) {
-        if (gang[i])
-            btrfs_put_root(gang[i]);
+    /* Release the roots that remain uncleaned due to error */
+    for (; i < grabbed; i++) {
+        if (roots[i])
+            btrfs_put_root(roots[i]);
       }
       return err;
   }
@@ -4872,31 +4865,36 @@ static void btrfs_error_commit_super(struct 
 btrfs_fs_info *fs_info)
   static void btrfs_drop_all_logs(struct btrfs_fs_info *fs_info)
   {
-    struct btrfs_root *gang[8];
-    u64 root_objectid = 0;
-    int ret;
+    unsigned long index = 0;
-    spin_lock(&fs_info->fs_roots_radix_lock);
Quoted 8 lines

fs/btrfs/disk-io.c static void btrfs_error_commit_super(struct btrfs_fs_info *fs_info) · line 4872

Quoted 6 lines
fs/btrfs/disk-io.c +3 -3
-    while ((ret = radix_tree_gang_lookup(&fs_info->fs_roots_radix,
-                         (void **)gang, root_objectid,
-                         ARRAY_SIZE(gang))) != 0) {
+    spin_lock(&fs_info->fs_roots_lock);
+    while (xa_find(&fs_info->fs_roots,
+               &index, ULONG_MAX, XA_PRESENT)) {

nit: No need to split across two lines as it's short enough.

ACK

Quoted 5 lines

fs/btrfs/disk-io.c static void btrfs_error_commit_super(struct btrfs_fs_info *fs_info) · line 4875

Quoted 3 lines
fs/btrfs/disk-io.c +2 -0
+        struct btrfs_root *root;
+        struct btrfs_root *roots[8];
           int i;

nit: This can probably be defined inside the for loop

OK

Quoted 34 lines
Quoted 34 lines
fs/btrfs/disk-io.c +17 -11
+        int grabbed = 0;
-        for (i = 0; i < ret; i++)
-            gang[i] = btrfs_grab_root(gang[i]);
-        spin_unlock(&fs_info->fs_roots_radix_lock);
+        xa_for_each_start(&fs_info->fs_roots, index, root,
+                  index) {
+            roots[grabbed] = btrfs_grab_root(root);
+            grabbed++;
+            if (grabbed >= ARRAY_SIZE(roots))
+                break;
+        }
+        spin_unlock(&fs_info->fs_roots_lock);
-        for (i = 0; i < ret; i++) {
-            if (!gang[i])
+        for (i = 0; i < grabbed; i++) {
+            if (!roots[i])
                   continue;
-            root_objectid = gang[i]->root_key.objectid;
-            btrfs_free_log(NULL, gang[i]);
-            btrfs_put_root(gang[i]);
+            index = roots[i]->root_key.objectid;
+            btrfs_free_log(NULL, roots[i]);
+            btrfs_put_root(roots[i]);
           }
-        root_objectid++;
-        spin_lock(&fs_info->fs_roots_radix_lock);
+        index++;
+        spin_lock(&fs_info->fs_roots_lock);
       }
-    spin_unlock(&fs_info->fs_roots_radix_lock);
+    spin_unlock(&fs_info->fs_roots_lock);
       btrfs_free_log_root_tree(NULL, fs_info);
   }
fs/btrfs/inode.c +0 -0
Quoted 8 lines

fs/btrfs/inode.c int btrfs_orphan_cleanup(struct btrfs_root *root) · line 3496

Quoted 7 lines

index 5082b9c70f8c..d0ef3a17ce11 100644

fs/btrfs/inode.c +1 -0
@@ -3494,6 +3494,7 @@ int btrfs_orphan_cleanup(struct btrfs_root *root)
       u64 last_objectid = 0;
       int ret = 0, nr_unlink = 0;
+    /* Bail out if the cleanup is already running. */

nit: This seems like an irrelevant change.

I did stumble a bit over the following line when I read through the function for the first time, so I added this in for possible future newbies.

I'd like to keep this in, if you don't mind. It's not like the comment hurts anything.

Quoted 28 lines
Quoted 2 lines
fs/btrfs/inode.c +0 -0
       if (test_and_set_bit(BTRFS_ROOT_ORPHAN_CLEANUP, &root->state))
           return 0;

<snip>

Quoted 23 lines
fs/btrfs/inode.c +2 -11
@@ -1404,9 +1402,8 @@ void btrfs_add_dead_root(struct btrfs_root *root)
   static noinline int commit_fs_roots(struct btrfs_trans_handle *trans)
   {
       struct btrfs_fs_info *fs_info = trans->fs_info;
-    struct btrfs_root *gang[8];
-    int i;
-    int ret;
+    struct btrfs_root *root;
+    unsigned long index;
       /*
        * At this point no one can be using this transaction to modify any tree
@@ -1414,17 +1411,11 @@ static noinline int commit_fs_roots(struct 
 btrfs_trans_handle *trans)
        */
       ASSERT(trans->transaction->state == TRANS_STATE_COMMIT_DOING);
-    spin_lock(&fs_info->fs_roots_radix_lock);
-    while (1) {
-        ret = radix_tree_gang_lookup_tag(&fs_info->fs_roots_radix,
-                         (void **)gang, 0,
-                         ARRAY_SIZE(gang),
-                         BTRFS_ROOT_TRANS_TAG);
-        if (ret == 0)
-            break;
Quoted 8 lines

fs/btrfs/inode.c static noinline int commit_fs_roots(struct · line 1416

Quoted 6 lines
fs/btrfs/inode.c +3 -3
-        for (i = 0; i < ret; i++) {
-            struct btrfs_root *root = gang[i];
-            int ret2;
+    spin_lock(&fs_info->fs_roots_lock);
+    while (xa_marked(&fs_info->fs_roots, BTRFS_ROOT_TRANS_TAG)) {
+        xa_for_each_marked(&fs_info->fs_roots, index, root,

While the while/xa_for_each_marked and not straight xa_for_each_marked?

Like before, this just mimicks the old logic, because I wasn't sure whether the double loop is only due to the bunched nature of gang lookup or whether it has some additional safeguarding function.

I'm fine with simplifying it to xa_for_each_marked if you say it's safe.

Waiting for further feedback whether to resubmit or not.

#7 lore raw
Quoted 5 lines · Gabriel Niebler

Am 28.04.22 um 13:59 schrieb Nikolay Borisov:

Quoted 3 lines · Gabriel Niebler
Quoted 1 line

… rename it to simply fs_roots and adjust all usages of this object to

While the while/xa_for_each_marked and not straight xa_for_each_marked?

Quoted 7 lines

Like before, this just mimicks the old logic, because I wasn't sure whether the double loop is only due to the bunched nature of gang lookup or whether it has some additional safeguarding function.

I'm fine with simplifying it to xa_for_each_marked if you say it's safe.

Waiting for further feedback whether to resubmit or not.

I don't have any more comments and it's rc5 so not much time left before code freeze, so please resend, thanks.

#8 lore raw
In reply to Gabriel Niebler

I had another look at this and have to correct myself:

Am 02.05.22 um 10:59 schrieb Gabriel Niebler:

Quoted 17 lines

Am 28.04.22 um 13:59 schrieb Nikolay Borisov:

Quoted 16 lines · Gabriel Niebler
Quoted 15 lines

… rename it to simply fs_roots and adjust all usages of this object to use the XArray API, because it is notionally easier to use and unserstand, as it provides array semantics, and also takes care of locking for us, further simplifying the code.

Also do some refactoring, esp. where the API change requires largely rewriting some functions, anyway.

---
  fs/btrfs/ctree.h       |   5 +-
  fs/btrfs/disk-io.c     | 176 ++++++++++++++++++++--------------------- fs/btrfs/inode.c       |  13 +--
  fs/btrfs/transaction.c |  67 +++++++---------

  4 files changed, 126 insertions(+), 135 deletions(-)

<snip>

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
fs/btrfs/disk-io.c +0 -0
index 126f244cdf88..a8577f659f66 100644

<snip>

Quoted 12 lines
Quoted 11 lines

fs/btrfs/disk-io.c void btrfs_drop_and_free_fs_root(struct · line 4506

Quoted 10 lines
fs/btrfs/disk-io.c +3 -3
@@ -4512,50 +4501,54 @@ void btrfs_drop_and_free_fs_root(struct 
 btrfs_fs_info *fs_info,
   int btrfs_cleanup_fs_roots(struct btrfs_fs_info *fs_info)
   {
-    u64 root_objectid = 0;
-    struct btrfs_root *gang[8];
-    int i = 0;
+    struct btrfs_root *roots[8];
+    unsigned long index = 0;
+    int i;

nit: This can be defined into the 2 loops that use the variable.

Quoted 1 line

Sure, why not.

I was wrong here, actually. The second for-loop that uses 'i' does not initialise it to zero, but deliberately starts with i at the last value it had when the first for-loop that uses it was last exited, because that one may have been broken out of, due to some cleanup error, and thus there may be unreleased roots still in the array.

This is how the code used to work before and I just kept that logic.

But then it's obviously important to declare 'i' outside of both loops so it can keep the value between them.

Quoted 62 lines
Quoted 62 lines
Quoted 62 lines
fs/btrfs/disk-io.c +26 -25
       int err = 0;
-    unsigned int ret = 0;
+    int grabbed;
       while (1) {
-        spin_lock(&fs_info->fs_roots_radix_lock);
-        ret = radix_tree_gang_lookup(&fs_info->fs_roots_radix,
-                         (void **)gang, root_objectid,
-                         ARRAY_SIZE(gang));
-        if (!ret) {
-            spin_unlock(&fs_info->fs_roots_radix_lock);
+        struct btrfs_root *root;
+
+        spin_lock(&fs_info->fs_roots_lock);
+        if (!xa_find(&fs_info->fs_roots, &index,
+                 ULONG_MAX, XA_PRESENT)) {
+            spin_unlock(&fs_info->fs_roots_lock);
               break;
           }
-        root_objectid = gang[ret - 1]->root_key.objectid + 1;
-        for (i = 0; i < ret; i++) {
-            /* Avoid to grab roots in dead_roots */
-            if (btrfs_root_refs(&gang[i]->root_item) == 0) {
-                gang[i] = NULL;
-                continue;
+        grabbed = 0;
+        xa_for_each_start(&fs_info->fs_roots, index, root,
+                  index) {
+            /* Avoid grabbing roots in dead_roots */
+            if (btrfs_root_refs(&root->root_item) == 0) {
+                roots[grabbed] = NULL;
+            } else {
+                /* Grab all the search results for later use */
+                roots[grabbed] = btrfs_grab_root(root);
               }
-            /* grab all the search result for later use */
-            gang[i] = btrfs_grab_root(gang[i]);
+            grabbed++;
+            if (grabbed >= ARRAY_SIZE(roots))
+                break;
           }
-        spin_unlock(&fs_info->fs_roots_radix_lock);
+        spin_unlock(&fs_info->fs_roots_lock);
-        for (i = 0; i < ret; i++) {
-            if (!gang[i])
+        for (i = 0; i < grabbed; i++) {
+            if (!roots[i])
                   continue;
-            root_objectid = gang[i]->root_key.objectid;
-            err = btrfs_orphan_cleanup(gang[i]);
+            index = roots[i]->root_key.objectid;
+            err = btrfs_orphan_cleanup(roots[i]);
               if (err)
                   break;
-            btrfs_put_root(gang[i]);
+            btrfs_put_root(roots[i]);
           }
-        root_objectid++;
+        index++;
       }
-    /* release the uncleaned roots due to error */
-    for (; i < ret; i++) {
-        if (gang[i])
Quoted 8 lines
Quoted 8 lines

fs/btrfs/disk-io.c void btrfs_drop_and_free_fs_root(struct btrfs_fs_info *fs_info, · line 4554

Quoted 8 lines
fs/btrfs/disk-io.c +4 -1
-            btrfs_put_root(gang[i]);
+    /* Release the roots that remain uncleaned due to error */
+    for (; i < grabbed; i++) {
+        if (roots[i])
+            btrfs_put_root(roots[i]);
       }
       return err;
   }

<snip>

I will do all the other changes we discussed and resend, but I'll leave this one alone.