Skip to contentSkip to conversation

btrfs: rework background block group relocation

PATCH In review Josef Bacik 2 participants 7 messages Mar 10, 2022 – Mar 11, 2022 Btrfs file system

lore
Share & tools
lore
#0 lore raw

cover btrfs: rework background block group relocation

Hello,

Currently the background block group relocation code only works for zoned devices, as it prevents the file system from becoming unusable because of block group fragmentation.

However inside Facebook our common workload is to download tens of gigabytes worth of send files or package files, and it does this by fallocate()'ing the entire package, writing into it, and then free'ing it up afterwards. Unfortunately this leads to a similar problem as zoned, we get fragmented data block groups, and this trends towards filling the entire disk up with partly used data block groups, which then leads to ENOSPC because of the lack of metadata space.

Because of this we have been running balance internally forever, but this was triggered based on different size usage hueristics and stil gave us a high enough failure rate that it was annoying (figure 10-20 machines needing to be reprovisioned per week).

So I modified the existing bg_reclaim_threshold code to also apply in the !zoned case, and I also made it only apply to DATA block groups. This has completely eliminated these random failure cases, and we're no longer reprovisioning machines that get stuck with 0 metadata space.

However my internal patch is kind of janky as it hard codes the DATA check. What I've done here is made the bg_reclaim_threshold per-space_info, this way a user can target all block group types or just the ones they care about. This won't break any current users because this only applied in the zoned case before.

Additionally I've added the code to allow this to work in the !zoned case, and loosened the restriction on the threshold from 50-100 to 0-100.

I tested this on my vm by writing 500m files and then removing half of them and validating that the block groups were automatically reclaimed. Thanks,

Josef

Josef Bacik (3):
btrfs: make the bg_reclaim_threshold per-space info
btrfs: allow block group background reclaim for !zoned fs'es
btrfs: change the bg_reclaim_threshold valid region from 0 to 100

 fs/btrfs/block-group.c      | 31 ++++++++++++++++
 fs/btrfs/ctree.h            |  1 -
 fs/btrfs/disk-io.c          |  1 -
 fs/btrfs/free-space-cache.c |  4 +--
 fs/btrfs/space-info.c       |  9 +++++
 fs/btrfs/space-info.h       |  6 ++++
 fs/btrfs/sysfs.c            | 71 +++++++++++++++++++------------------
 fs/btrfs/zoned.h            |  6 ----
 8 files changed, 84 insertions(+), 45 deletions(-)
Signature · 1 line
2.26.3
#1 lore raw

PATCH 1/3 btrfs: make the bg_reclaim_threshold per-space info

For !zoned file systems it's useful to have the auto reclaim feature, however there are different use cases for !zoned, for example we may not want to reclaim metadata chunks ever, only data chunks. Move this sysfs flag to per-space_info. This won't affect current users because this tunable only ever did anything for zoned, and that is currently hidden behind BTRFS_CONFIG_DEBUG.

---
 fs/btrfs/ctree.h            |  1 -
 fs/btrfs/disk-io.c          |  1 -
 fs/btrfs/free-space-cache.c |  4 +--
 fs/btrfs/space-info.c       |  9 +++++
 fs/btrfs/space-info.h       |  6 ++++
 fs/btrfs/sysfs.c            | 71 +++++++++++++++++++------------------
 fs/btrfs/zoned.h            |  6 ----
 7 files changed, 53 insertions(+), 45 deletions(-)
fs/btrfs/ctree.h +0 -1
index 4db17bd05a21..1953ea40755d 100644
@@ -1015,7 +1015,6 @@ struct btrfs_fs_info {
 	/* Reclaim partially filled block groups in the background */
 	struct work_struct reclaim_bgs_work;
 	struct list_head reclaim_bgs;
-	int bg_reclaim_threshold;
 
 	spinlock_t unused_bgs_lock;
 	struct list_head unused_bgs;
fs/btrfs/disk-io.c +0 -1
index 09693ab4fde0..c135b79bf3e3 100644
@@ -3246,7 +3246,6 @@ void btrfs_init_fs_info(struct btrfs_fs_info *fs_info)
 	spin_lock_init(&fs_info->swapfile_pins_lock);
 	fs_info->swapfile_pins = RB_ROOT;
 
-	fs_info->bg_reclaim_threshold = BTRFS_DEFAULT_RECLAIM_THRESH;
 	INIT_WORK(&fs_info->reclaim_bgs_work, btrfs_reclaim_bgs_work);
 }
 
fs/btrfs/free-space-cache.c +2 -2
index 01a408db5683..01ac1161aec5 100644
@@ -2630,11 +2630,11 @@ int __btrfs_add_free_space(struct btrfs_block_group *block_group,
 static int __btrfs_add_free_space_zoned(struct btrfs_block_group *block_group,
 					u64 bytenr, u64 size, bool used)
 {
-	struct btrfs_fs_info *fs_info = block_group->fs_info;
+	struct btrfs_space_info *sinfo = block_group->space_info;
 	struct btrfs_free_space_ctl *ctl = block_group->free_space_ctl;
 	u64 offset = bytenr - block_group->start;
 	u64 to_free, to_unusable;
-	const int bg_reclaim_threshold = READ_ONCE(fs_info->bg_reclaim_threshold);
+	const int bg_reclaim_threshold = READ_ONCE(sinfo->bg_reclaim_threshold);
 	bool initial = (size == block_group->length);
 	u64 reclaimable_unusable;
 
fs/btrfs/space-info.c +9 -0
index b87931a458eb..60d0a58c4644 100644
@@ -181,6 +181,12 @@ void btrfs_clear_space_info_full(struct btrfs_fs_info *info)
 		found->full = 0;
 }
 
+/*
+ * Block groups with more than this value (percents) of unusable space will be
+ * scheduled for background reclaim.
+ */
+#define BTRFS_DEFAULT_ZONED_RECLAIM_THRESH	75
+
 static int create_space_info(struct btrfs_fs_info *info, u64 flags)
 {
 
@@ -203,6 +209,9 @@ static int create_space_info(struct btrfs_fs_info *info, u64 flags)
 	INIT_LIST_HEAD(&space_info->priority_tickets);
 	space_info->clamp = 1;
 
+	if (btrfs_is_zoned(info))
+		space_info->bg_reclaim_threshold = BTRFS_DEFAULT_ZONED_RECLAIM_THRESH;
+
 	ret = btrfs_sysfs_add_space_info_type(info, space_info);
 	if (ret)
 		return ret;
fs/btrfs/space-info.h +6 -0
index d841fed73492..0c45f539e3cf 100644
@@ -24,6 +24,12 @@ struct btrfs_space_info {
 				   the space info if we had an ENOSPC in the
 				   allocator. */
 
+	/*
+	 * Once a block group drops below this threshold we'll schedule it for
+	 * reclaim.
+	 */
+	int bg_reclaim_threshold;
+
 	int clamp;		/* Used to scale our threshold for preemptive
 				   flushing. The value is >> clamp, so turns
 				   out to be a 2^clamp divisor. */
fs/btrfs/sysfs.c +36 -35

95 lines not on this page · raw message

fs/btrfs/zoned.h +0 -6
index cbf016a7bb5d..9075c87a397e 100644
@@ -10,12 +10,6 @@
 #include "block-group.h"
 #include "btrfs_inode.h"
 
-/*
- * Block groups with more than this value (percents) of unusable space will be
- * scheduled for background reclaim.
- */
-#define BTRFS_DEFAULT_RECLAIM_THRESH		75
-
 struct btrfs_zoned_device_info {
 	/*
 	 * Number of zones, zone size and types of zones if bdev is a
Signature · 1 line
2.26.3
#2 lore raw

Re: [PATCH 1/3] btrfs: make the bg_reclaim_threshold per-space info

Quoted 9 lines · Josef Bacik
fs/btrfs/ctree.h +0 -1
index 4db17bd05a21..1953ea40755d 100644
@@ -1015,7 +1015,6 @@ struct btrfs_fs_info {
 	/* Reclaim partially filled block groups in the background */
 	struct work_struct reclaim_bgs_work;
 	struct list_head reclaim_bgs;
-	int bg_reclaim_threshold;

[...]

Quoted 24 lines
fs/btrfs/ctree.h +0 -24
-static ssize_t btrfs_bg_reclaim_threshold_show(struct kobject *kobj,
-					       struct kobj_attribute *a,
-					       char *buf)
-{
-	struct btrfs_fs_info *fs_info = to_fs_info(kobj);
-	ssize_t ret;
-
-	ret = sysfs_emit(buf, "%d\n", READ_ONCE(fs_info->bg_reclaim_threshold));
-
-	return ret;
-}
-
-static ssize_t btrfs_bg_reclaim_threshold_store(struct kobject *kobj,
-						struct kobj_attribute *a,
-						const char *buf, size_t len)
-{
-	struct btrfs_fs_info *fs_info = to_fs_info(kobj);
-	int thresh;
-	int ret;
-
-	ret = kstrtoint(buf, 10, &thresh);
-	if (ret)
-		return ret;
-

fs/btrfs/ctree.h

Quoted 10 lines
fs/btrfs/ctree.h +0 -10
-	if (thresh != 0 && (thresh <= 50 || thresh > 100))
-		return -EINVAL;
-
-	WRITE_ONCE(fs_info->bg_reclaim_threshold, thresh);
-
-	return len;
-}
-BTRFS_ATTR_RW(, bg_reclaim_threshold, btrfs_bg_reclaim_threshold_show,
-	      btrfs_bg_reclaim_threshold_store);
-

Can you leave the old, per-fs threshold and sysfs attribute in, as I still need it for zoned?

Or should I send you my zoned change, so you can add it to your series?

Thanks,
Johannes

#3 lore raw

PATCH 2/3 btrfs: allow block group background reclaim for !zoned fs'es

In reply to Josef Bacik

We have found this feature invaluable at Facebook due to how our workload interacts with the allocator. We have been using this in production for months with only a single problem that has already been fixed. This will allow us to set a threshold for block groups to be automatically relocated even if we don't have zoned devices.

---
 fs/btrfs/block-group.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
fs/btrfs/block-group.c +31 -0
index c22d287e020b..ca43daba292a 100644
@@ -3230,6 +3230,31 @@ int btrfs_write_dirty_block_groups(struct btrfs_trans_handle *trans)
 	return ret;
 }
 
+static inline bool should_reclaim_block_group(struct btrfs_block_group *block_group,
+					      u64 bytes_freed)
+{
+	const struct btrfs_space_info *space_info = block_group->space_info;
+	const int reclaim_thresh = READ_ONCE(space_info->bg_reclaim_threshold);
+	const u64 new_val = block_group->used;
+	const u64 old_val = new_val + bytes_freed;
+	u64 thresh;
+
+	if (reclaim_thresh == 0)
+		return false;
+
+	thresh = div_factor_fine(block_group->length, reclaim_thresh);
+
+	/*
+	 * If we were below the threshold before don't reclaim, we are likely a
+	 * brand new block group and we don't want to relocate new block groups.
+	 */
+	if (old_val < thresh)
+		return false;
+	if (new_val >= thresh)
+		return false;
+	return true;
+}
+
 int btrfs_update_block_group(struct btrfs_trans_handle *trans,
 			     u64 bytenr, u64 num_bytes, bool alloc)
 {
@@ -3252,6 +3277,8 @@ int btrfs_update_block_group(struct btrfs_trans_handle *trans,
 	spin_unlock(&info->delalloc_root_lock);
 
 	while (total) {
+		bool reclaim;
+
 		cache = btrfs_lookup_block_group(info, bytenr);
 		if (!cache) {
 			ret = -ENOENT;
@@ -3297,6 +3324,8 @@ int btrfs_update_block_group(struct btrfs_trans_handle *trans,
 					cache->space_info, num_bytes);
 			cache->space_info->bytes_used -= num_bytes;
 			cache->space_info->disk_used -= num_bytes * factor;
+
+			reclaim = should_reclaim_block_group(cache, num_bytes);
 			spin_unlock(&cache->lock);
 			spin_unlock(&cache->space_info->lock);
 
@@ -3323,6 +3352,8 @@ int btrfs_update_block_group(struct btrfs_trans_handle *trans,
 		if (!alloc && old_val == 0) {
 			if (!btrfs_test_opt(info, DISCARD_ASYNC))
 				btrfs_mark_bg_unused(cache);
+		} else if (!alloc && reclaim) {
+			btrfs_mark_bg_to_reclaim(cache);
 		}
 
 		btrfs_put_block_group(cache);
Signature · 1 line
2.26.3
#5 lore raw

PATCH 3/3 btrfs: change the bg_reclaim_threshold valid region from 0 to 100

In reply to Josef Bacik

For the !zoned case we may want to set the threshold for reclaim to something below 50%. Change the acceptable threshold from 50-100 to 0-100.

---
 fs/btrfs/sysfs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/btrfs/sysfs.c +1 -1
index d11ff1c55394..400ce18d6a81 100644
@@ -746,7 +746,7 @@ static ssize_t btrfs_bg_reclaim_threshold_store(struct kobject *kobj,
 	if (ret)
 		return ret;
 
-	if (thresh != 0 && (thresh <= 50 || thresh > 100))
+	if (thresh < 0 || thresh > 100)
 		return -EINVAL;
 
 	WRITE_ONCE(space_info->bg_reclaim_threshold, thresh);
Signature · 1 line
2.26.3
Reviewed-by on v1 3/3
#6 lore raw

Re: [PATCH 3/3] btrfs: change the bg_reclaim_threshold valid region from 0 to 100

Quoted 14 lines · Josef Bacik

For the !zoned case we may want to set the threshold for reclaim to something below 50%. Change the acceptable threshold from 50-100 to 0-100.

---
 fs/btrfs/sysfs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/btrfs/sysfs.c +0 -0
index d11ff1c55394..400ce18d6a81 100644

fs/btrfs/sysfs.c static ssize_t btrfs_bg_reclaim_threshold_store(struct kobject *kobj, · line 752

Quoted 8 lines
fs/btrfs/sysfs.c +1 -1
@@ -746,7 +746,7 @@ static ssize_t btrfs_bg_reclaim_threshold_store(struct kobject *kobj,
 	if (ret)
 		return ret;
 
-	if (thresh != 0 && (thresh <= 50 || thresh > 100))
+	if (thresh < 0 || thresh > 100)
 		return -EINVAL;
 
 	WRITE_ONCE(space_info->bg_reclaim_threshold, thresh);

That could be problematic for zns devices with say a zone_size of 2G and a zone_capacity of 1G. This means all block_groups are 50% zone_unusable right from the start, so if someone set's the threshold to 50% or lover it'll constantly try to reclaim the bg.

I'm not saying we shouldn't do that, it's just a "with great power comes great responsibility" reminder.

Looks good,