Skip to contentSkip to conversation

Fix active zone accounting for zoned

PATCH In review Josef Bacik 4 participants 13 messages Mar 1, 2023 – Mar 2, 2023 Btrfs file system

lore
Share & tools
lore
#0 lore raw

cover Fix active zone accounting for zoned

Hello,

While trying to get the CI system up and running with the ZNS drives I ran into this issue where the TEST_DEV would get maxed out on block groups and everything would start failing with -ENOSPC.

This is because the way we handle active zone tracking isn't correct. We need to only take into account the used/pinned/zone_unusable counters for the active block groups, not the system wide counters.

Unfortunately this is kind of annoying in that we have to mirror these counters for the active block groups. At this point this is the most straightforward way to fix this. I'm currently running these patches through xfstests, so consider this as sort of an RFC to make sure the approach seems reasonable. Thanks,

Josef

Josef Bacik (3):
btrfs: rename BTRFS_FS_NO_OVERCOMMIT -> BTRFS_FS_ACTIVE_ZONE_TRACKING
btrfs: clean up space_info usage in btrfs_update_block_group
btrfs: handle active zone accounting properly

 fs/btrfs/block-group.c | 51 ++++++++++++++++++-------
 fs/btrfs/disk-io.c     |  6 +++
 fs/btrfs/extent-tree.c | 13 +++++++
 fs/btrfs/fs.h          |  7 +---
 fs/btrfs/space-info.c  | 85 +++++++++++++++++++++++++++++++++++++++---
 fs/btrfs/space-info.h  | 20 +++++++++-
 fs/btrfs/zoned.c       | 17 ++++++---
 7 files changed, 168 insertions(+), 31 deletions(-)
Signature · 1 line
2.26.3
#1 lore raw

PATCH 1/3 btrfs: rename BTRFS_FS_NO_OVERCOMMIT -> BTRFS_FS_ACTIVE_ZONE_TRACKING

This flag only gets set when we're doing active zone tracking, and I'm going to need to use this flag for things related to this behavior. Rename the flag to represent what it actually means for the file system so it can be used in other ways and still make sense.

---
 fs/btrfs/fs.h         | 7 ++-----
 fs/btrfs/space-info.c | 2 +-
 fs/btrfs/zoned.c      | 3 +--
 3 files changed, 4 insertions(+), 8 deletions(-)
fs/btrfs/fs.h +2 -5
index 4c477eae6891..24cd49229408 100644
@@ -120,11 +120,8 @@ enum {
 	/* Indicate that we want to commit the transaction. */
 	BTRFS_FS_NEED_TRANS_COMMIT,
 
-	/*
-	 * Indicate metadata over-commit is disabled. This is set when active
-	 * zone tracking is needed.
-	 */
-	BTRFS_FS_NO_OVERCOMMIT,
+	/* This is set when active zone tracking is needed. */
+	BTRFS_FS_ACTIVE_ZONE_TRACKING,
 
 	/*
 	 * Indicate if we have some features changed, this is mostly for
fs/btrfs/space-info.c +1 -1
index 69c09508afb5..2237685d1ed0 100644
@@ -407,7 +407,7 @@ int btrfs_can_overcommit(struct btrfs_fs_info *fs_info,
 		return 0;
 
 	used = btrfs_space_info_used(space_info, true);
-	if (test_bit(BTRFS_FS_NO_OVERCOMMIT, &fs_info->flags) &&
+	if (test_bit(BTRFS_FS_ACTIVE_ZONE_TRACKING, &fs_info->flags) &&
 	    (space_info->flags & BTRFS_BLOCK_GROUP_METADATA))
 		avail = 0;
 	else
fs/btrfs/zoned.c +1 -2
index f95b2c94d619..808cfa3091c5 100644
@@ -524,8 +524,7 @@ int btrfs_get_dev_zone_info(struct btrfs_device *device, bool populate_cache)
 		}
 		atomic_set(&zone_info->active_zones_left,
 			   max_active_zones - nactive);
-		/* Overcommit does not work well with active zone tacking. */
-		set_bit(BTRFS_FS_NO_OVERCOMMIT, &fs_info->flags);
+		set_bit(BTRFS_FS_ACTIVE_ZONE_TRACKING, &fs_info->flags);
 	}
 
 	/* Validate superblock log */
Signature · 1 line
2.26.3
Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] btrfs: rename BTRFS_FS_NO_OVERCOMMIT -> BTRFS_FS_ACTIVE_ZONE_TRACKING

Quoted 6 lines · Josef Bacik

This flag only gets set when we're doing active zone tracking, and I'm going to need to use this flag for things related to this behavior. Rename the flag to represent what it actually means for the file system so it can be used in other ways and still make sense.

Quoted 18 lines
fs/btrfs/space-info.c +1 -1
index 69c09508afb5..2237685d1ed0 100644
@@ -407,7 +407,7 @@ int btrfs_can_overcommit(struct btrfs_fs_info *fs_info,
  		return 0;
 
  	used = btrfs_space_info_used(space_info, true);
-	if (test_bit(BTRFS_FS_NO_OVERCOMMIT, &fs_info->flags) &&
+	if (test_bit(BTRFS_FS_ACTIVE_ZONE_TRACKING, &fs_info->flags) &&
  	    (space_info->flags & BTRFS_BLOCK_GROUP_METADATA))
  		avail = 0;
  	else
fs/btrfs/zoned.c +0 -0
index f95b2c94d619..808cfa3091c5 100644

fs/btrfs/zoned.c int btrfs_get_dev_zone_info(struct btrfs_device *device, bool populate_cache) · line 528

Quoted 5 lines
fs/btrfs/zoned.c +0 -1
@@ -524,8 +524,7 @@ int btrfs_get_dev_zone_info(struct btrfs_device *device, bool populate_cache)
  		}
  		atomic_set(&zone_info->active_zones_left,
  			   max_active_zones - nactive);
 
-		/* Overcommit does not work well with active zone tacking. */

Nit:
Consider moving this comment to the btrfs_can_overcommit() function, possibly during the apply.

Reviewed-by on v1 1/3
#4 lore raw

Re: [PATCH 1/3] btrfs: rename BTRFS_FS_NO_OVERCOMMIT -> BTRFS_FS_ACTIVE_ZONE_TRACKING

In reply to Josef Bacik
Quoted 6 lines · Josef Bacik

This flag only gets set when we're doing active zone tracking, and I'm going to need to use this flag for things related to this behavior. Rename the flag to represent what it actually means for the file system so it can be used in other ways and still make sense.

Looks good,

#5 lore raw

PATCH 2/3 btrfs: clean up space_info usage in btrfs_update_block_group

In reply to Josef Bacik

We do

cache->space_info->counter += num_bytes;

Everywhere in here. This is makes the lines longer than they need to be, and will be especially noticeable when I add the active tracking in, so add a temp variable for the space_info so this is cleaner.

---
 fs/btrfs/block-group.c | 22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)
fs/btrfs/block-group.c +12 -10
index 5b10401d803b..3eff0b35e139 100644
@@ -3474,6 +3474,7 @@ int btrfs_update_block_group(struct btrfs_trans_handle *trans,
 	spin_unlock(&info->delalloc_root_lock);
 
 	while (total) {
+		struct btrfs_space_info *space_info;
 		bool reclaim = false;
 
 		cache = btrfs_lookup_block_group(info, bytenr);
@@ -3481,6 +3482,7 @@ int btrfs_update_block_group(struct btrfs_trans_handle *trans,
 			ret = -ENOENT;
 			break;
 		}
+		space_info = cache->space_info;
 		factor = btrfs_bg_type_to_factor(cache->flags);
 
 		/*
@@ -3495,7 +3497,7 @@ int btrfs_update_block_group(struct btrfs_trans_handle *trans,
 		byte_in_group = bytenr - cache->start;
 		WARN_ON(byte_in_group > cache->length);
 
-		spin_lock(&cache->space_info->lock);
+		spin_lock(&space_info->lock);
 		spin_lock(&cache->lock);
 
 		if (btrfs_test_opt(info, SPACE_CACHE) &&
@@ -3508,24 +3510,24 @@ int btrfs_update_block_group(struct btrfs_trans_handle *trans,
 			old_val += num_bytes;
 			cache->used = old_val;
 			cache->reserved -= num_bytes;
-			cache->space_info->bytes_reserved -= num_bytes;
-			cache->space_info->bytes_used += num_bytes;
-			cache->space_info->disk_used += num_bytes * factor;
+			space_info->bytes_reserved -= num_bytes;
+			space_info->bytes_used += num_bytes;
+			space_info->disk_used += num_bytes * factor;
 			spin_unlock(&cache->lock);
-			spin_unlock(&cache->space_info->lock);
+			spin_unlock(&space_info->lock);
 		} else {
 			old_val -= num_bytes;
 			cache->used = old_val;
 			cache->pinned += num_bytes;
-			btrfs_space_info_update_bytes_pinned(info,
-					cache->space_info, num_bytes);
-			cache->space_info->bytes_used -= num_bytes;
-			cache->space_info->disk_used -= num_bytes * factor;
+			btrfs_space_info_update_bytes_pinned(info, space_info,
+							     num_bytes);
+			space_info->bytes_used -= num_bytes;
+			space_info->disk_used -= num_bytes * factor;
 
 			reclaim = should_reclaim_block_group(cache, num_bytes);
 
 			spin_unlock(&cache->lock);
-			spin_unlock(&cache->space_info->lock);
+			spin_unlock(&space_info->lock);
 
 			set_extent_dirty(&trans->transaction->pinned_extents,
 					 bytenr, bytenr + num_bytes - 1,
Signature · 1 line
2.26.3
Reviewed-by on v1 2/3
#6 lore raw

Re: [PATCH 2/3] btrfs: clean up space_info usage in btrfs_update_block_group

Quoted 9 lines · Josef Bacik

We do

cache->space_info->counter += num_bytes;

Everywhere in here. This is makes the lines longer than they need to be, and will be especially noticeable when I add the active tracking in, so add a temp variable for the space_info so this is cleaner.

Looks good.

Reviewed-by on v1 2/3
#8 lore raw

Re: [PATCH 2/3] btrfs: clean up space_info usage in btrfs_update_block_group

In reply to Josef Bacik
Quoted 9 lines · Josef Bacik

We do

cache->space_info->counter += num_bytes;

Everywhere in here. This is makes the lines longer than they need to be, and will be especially noticeable when I add the active tracking in, so add a temp variable for the space_info so this is cleaner.

Looks good to me.

#9 lore raw

PATCH 3/3 btrfs: handle active zone accounting properly

In reply to Josef Bacik

Running xfstests on my ZNS drives uncovered a problem where I was allocating the entire device with metadata block groups. The root cause of this was we would get ENOSPC on mount, and while trying to satisfy tickets we'd allocate more block groups.

The reason we were getting ENOSPC was because ->bytes_zone_unusable was set to 40gib, but ->active_total_bytes was set to 8gib, which was the maximum number of active block groups we're allowed to have at one time. This was because we always update ->bytes_zone_unusable with the unusable amount from every block group, but we only update ->active_total_bytes with the active block groups.

This is actually much worse however, because we could potentially have other counters potentially well above the ->active_total_bytes, which would lead to this early enospc situation.

Fix this by mirroring the counters for active block groups into their own counters. This allows us to keep the full space_info counters consistent, which is needed for things like sysfs and the space info ioctl, and then track the actual usage for ENOSPC reasons for only the active block groups.

Additionally, when de-activating we weren't properly updating the ->active_total_bytes, which could lead to other problems. Unifying all of this with the proper helpers will make sure our accounting is correct.

---
 fs/btrfs/block-group.c | 29 +++++++++++++--
 fs/btrfs/disk-io.c     |  6 +++
 fs/btrfs/extent-tree.c | 13 +++++++
 fs/btrfs/space-info.c  | 83 ++++++++++++++++++++++++++++++++++++++++--
 fs/btrfs/space-info.h  | 20 +++++++++-
 fs/btrfs/zoned.c       | 14 +++++--
 6 files changed, 152 insertions(+), 13 deletions(-)
fs/btrfs/block-group.c +25 -4
index 3eff0b35e139..7d5b73b2689e 100644
@@ -1166,6 +1166,9 @@ int btrfs_remove_block_group(struct btrfs_trans_handle *trans,
 		btrfs_put_caching_control(caching_ctl);
 	}
 
+	ASSERT(!test_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE,
+			 &block_group->runtime_flags));
+
 	spin_lock(&trans->transaction->dirty_bgs_lock);
 	WARN_ON(!list_empty(&block_group->dirty_list));
 	WARN_ON(!list_empty(&block_group->io_list));
@@ -1191,12 +1194,8 @@ int btrfs_remove_block_group(struct btrfs_trans_handle *trans,
 			< block_group->length);
 	}
 	block_group->space_info->total_bytes -= block_group->length;
-	if (test_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE, &block_group->runtime_flags))
-		block_group->space_info->active_total_bytes -= block_group->length;
 	block_group->space_info->bytes_readonly -=
 		(block_group->length - block_group->zone_unusable);
-	block_group->space_info->bytes_zone_unusable -=
-		block_group->zone_unusable;
 	block_group->space_info->disk_total -= block_group->length * factor;
 
 	spin_unlock(&block_group->space_info->lock);
@@ -1377,10 +1376,14 @@ static int inc_block_group_ro(struct btrfs_block_group *cache, int force)
 
 	if (!ret) {
 		sinfo->bytes_readonly += num_bytes;
+		ASSERT(!test_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE,
+				 &cache->runtime_flags));
+
 		if (btrfs_is_zoned(cache->fs_info)) {
 			/* Migrate zone_unusable bytes to readonly */
 			sinfo->bytes_readonly += cache->zone_unusable;
 			sinfo->bytes_zone_unusable -= cache->zone_unusable;
+
 			cache->zone_unusable = 0;
 		}
 		cache->ro++;
@@ -1575,6 +1578,9 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)
 		}
 		spin_unlock(&fs_info->discard_ctl.lock);
 
+		ASSERT(!test_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE,
+				 &block_group->runtime_flags));
+
 		/* Reset pinned so btrfs_put_block_group doesn't complain */
 		spin_lock(&space_info->lock);
 		spin_lock(&block_group->lock);
@@ -1582,6 +1588,7 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)
 		btrfs_space_info_update_bytes_pinned(fs_info, space_info,
 						     -block_group->pinned);
 		space_info->bytes_readonly += block_group->pinned;
+
 		block_group->pinned = 0;
 
 		spin_unlock(&block_group->lock);
@@ -2876,6 +2883,9 @@ void btrfs_dec_block_group_ro(struct btrfs_block_group *cache)
 
 	BUG_ON(!cache->ro);
 
+	ASSERT(!test_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE,
+			 &cache->runtime_flags));
+
 	spin_lock(&sinfo->lock);
 	spin_lock(&cache->lock);
 	if (!--cache->ro) {
@@ -3513,6 +3523,11 @@ int btrfs_update_block_group(struct btrfs_trans_handle *trans,
 			space_info->bytes_reserved -= num_bytes;
 			space_info->bytes_used += num_bytes;
 			space_info->disk_used += num_bytes * factor;
+
+			if (test_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE,
+				     &cache->runtime_flags))
+				space_info->active_bytes_used += num_bytes;
+
 			spin_unlock(&cache->lock);
 			spin_unlock(&space_info->lock);
 		} else {
@@ -3524,6 +3539,12 @@ int btrfs_update_block_group(struct btrfs_trans_handle *trans,
 			space_info->bytes_used -= num_bytes;
 			space_info->disk_used -= num_bytes * factor;
 
+			if (test_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE,
+				     &cache->runtime_flags)) {
+				space_info->active_bytes_pinned += num_bytes;
+				space_info->active_bytes_used -= num_bytes;
+			}
+
 			reclaim = should_reclaim_block_group(cache, num_bytes);
 
 			spin_unlock(&cache->lock);
fs/btrfs/disk-io.c +6 -0
index b53f0e30ce2b..4cec057f4358 100644
@@ -4883,6 +4883,12 @@ static int btrfs_destroy_delayed_refs(struct btrfs_transaction *trans,
 				cache->space_info, head->num_bytes);
 			cache->reserved -= head->num_bytes;
 			cache->space_info->bytes_reserved -= head->num_bytes;
+
+			if (test_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE,
+				     &cache->runtime_flags))
+				cache->space_info->active_bytes_pinned +=
+					head->num_bytes;
+
 			spin_unlock(&cache->lock);
 			spin_unlock(&cache->space_info->lock);
 
fs/btrfs/extent-tree.c +13 -0
index 824c657f59e8..29fb3b2b7370 100644
@@ -2530,6 +2530,11 @@ static int pin_down_extent(struct btrfs_trans_handle *trans,
 		cache->reserved -= num_bytes;
 		cache->space_info->bytes_reserved -= num_bytes;
 	}
+
+	if (test_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE,
+		     &cache->runtime_flags))
+		cache->space_info->active_bytes_pinned += num_bytes;
+
 	spin_unlock(&cache->lock);
 	spin_unlock(&cache->space_info->lock);
 
@@ -2725,6 +2730,14 @@ static int unpin_extent_range(struct btrfs_fs_info *fs_info,
 		spin_lock(&cache->lock);
 		cache->pinned -= len;
 		btrfs_space_info_update_bytes_pinned(fs_info, space_info, -len);
+
+		if (test_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE,
+			     &cache->runtime_flags)) {
+			ASSERT(!cache->ro);
+			space_info->active_bytes_zone_unusable += len;
+			space_info->active_bytes_pinned -= len;
+		}
+
 		space_info->max_extent_size = 0;
 		if (cache->ro) {
 			space_info->bytes_readonly += len;
fs/btrfs/space-info.c +79 -4

118 lines not on this page · raw message

fs/btrfs/space-info.h +18 -2
index fc99ea2b0c34..d072d6c26d3a 100644
@@ -96,11 +96,25 @@ struct btrfs_space_info {
 	u64 bytes_may_use;	/* number of bytes that may be used for
 				   delalloc/allocations */
 	u64 bytes_readonly;	/* total bytes that are read only */
-	/* Total bytes in the space, but only accounts active block groups. */
-	u64 active_total_bytes;
+
 	u64 bytes_zone_unusable;	/* total bytes that are unusable until
 					   resetting the device zone */
 
+	/*
+	 * This are mirrors of the above countesr.
+	 *
+	 * We need to mirror a lot of the counters for ENOSPC reasons if we're
+	 * doing active block group management.  The only exceptions are
+	 * bytes_reserved, which would be correct as we can only be allocating
+	 * from active block groups, and bytes_may_use which again is uptodate
+	 * based on what is currently being reserved.  Everything else has to be
+	 * mirrored and only accounted for in the active block groups.
+	 * */
+	u64 active_total_bytes;
+	u64 active_bytes_used;
+	u64 active_bytes_pinned;
+	u64 active_bytes_zone_unusable;
+
 	u64 max_extent_size;	/* This will hold the maximum extent size of
 				   the space info if we had an ENOSPC in the
 				   allocator. */
@@ -237,5 +251,7 @@ int btrfs_reserve_data_bytes(struct btrfs_fs_info *fs_info, u64 bytes,
 void btrfs_dump_space_info_for_trans_abort(struct btrfs_fs_info *fs_info);
 void btrfs_init_async_reclaim_work(struct btrfs_fs_info *fs_info);
 u64 btrfs_account_ro_block_groups_free_space(struct btrfs_space_info *sinfo);
+void btrfs_deactivate_block_group(struct btrfs_block_group *block_group);
+void btrfs_activate_block_group(struct btrfs_block_group *block_group);
 
 #endif /* BTRFS_SPACE_INFO_H */
fs/btrfs/zoned.c +11 -3
index 808cfa3091c5..8a863004d750 100644
@@ -1900,8 +1900,7 @@ bool btrfs_zone_activate(struct btrfs_block_group *block_group)
 	}
 
 	/* Successfully activated all the zones */
-	set_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE, &block_group->runtime_flags);
-	space_info->active_total_bytes += block_group->length;
+	btrfs_activate_block_group(block_group);
 	spin_unlock(&block_group->lock);
 	btrfs_try_granting_tickets(fs_info, space_info);
 	spin_unlock(&space_info->lock);
@@ -1956,15 +1955,18 @@ static void wait_eb_writebacks(struct btrfs_block_group *block_group)
 static int do_zone_finish(struct btrfs_block_group *block_group, bool fully_written)
 {
 	struct btrfs_fs_info *fs_info = block_group->fs_info;
+	struct btrfs_space_info *space_info = block_group->space_info;
 	struct map_lookup *map;
 	const bool is_metadata = (block_group->flags &
 			(BTRFS_BLOCK_GROUP_METADATA | BTRFS_BLOCK_GROUP_SYSTEM));
 	int ret = 0;
 	int i;
 
+	spin_lock(&space_info->lock);
 	spin_lock(&block_group->lock);
 	if (!test_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE, &block_group->runtime_flags)) {
 		spin_unlock(&block_group->lock);
+		spin_unlock(&space_info->lock);
 		return 0;
 	}
 
@@ -1972,6 +1974,7 @@ static int do_zone_finish(struct btrfs_block_group *block_group, bool fully_writ
 	if (is_metadata &&
 	    block_group->start + block_group->alloc_offset > block_group->meta_write_pointer) {
 		spin_unlock(&block_group->lock);
+		spin_unlock(&space_info->lock);
 		return -EAGAIN;
 	}
 
@@ -1984,6 +1987,7 @@ static int do_zone_finish(struct btrfs_block_group *block_group, bool fully_writ
 	 */
 	if (!fully_written) {
 		spin_unlock(&block_group->lock);
+		spin_unlock(&space_info->lock);
 
 		ret = btrfs_inc_block_group_ro(block_group, false);
 		if (ret)
@@ -1998,6 +2002,7 @@ static int do_zone_finish(struct btrfs_block_group *block_group, bool fully_writ
 		if (is_metadata)
 			wait_eb_writebacks(block_group);
 
+		spin_lock(&space_info->lock);
 		spin_lock(&block_group->lock);
 
 		/*
@@ -2007,23 +2012,26 @@ static int do_zone_finish(struct btrfs_block_group *block_group, bool fully_writ
 		if (!test_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE,
 			      &block_group->runtime_flags)) {
 			spin_unlock(&block_group->lock);
+			spin_unlock(&space_info->lock);
 			btrfs_dec_block_group_ro(block_group);
 			return 0;
 		}
 
 		if (block_group->reserved) {
 			spin_unlock(&block_group->lock);
+			spin_unlock(&space_info->lock);
 			btrfs_dec_block_group_ro(block_group);
 			return -EAGAIN;
 		}
 	}
 
-	clear_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE, &block_group->runtime_flags);
+	btrfs_deactivate_block_group(block_group);
 	block_group->alloc_offset = block_group->zone_capacity;
 	block_group->free_space_ctl->free_space = 0;
 	btrfs_clear_treelog_bg(block_group);
 	btrfs_clear_data_reloc_bg(block_group);
 	spin_unlock(&block_group->lock);
+	spin_unlock(&space_info->lock);
 
 	map = block_group->physical_map;
 	for (i = 0; i < map->num_stripes; i++) {
Signature · 1 line
2.26.3
#10 lore raw

Re: [PATCH 3/3] btrfs: handle active zone accounting properly

Quoted 16 lines · Josef Bacik

Running xfstests on my ZNS drives uncovered a problem where I was allocating the entire device with metadata block groups. The root cause of this was we would get ENOSPC on mount, and while trying to satisfy tickets we'd allocate more block groups.

The reason we were getting ENOSPC was because ->bytes_zone_unusable was set to 40gib, but ->active_total_bytes was set to 8gib, which was the maximum number of active block groups we're allowed to have at one time. This was because we always update ->bytes_zone_unusable with the unusable amount from every block group, but we only update ->active_total_bytes with the active block groups.

This is actually much worse however, because we could potentially have other counters potentially well above the ->active_total_bytes, which would lead to this early enospc situation.

Quoted 5 lines

Fix this by mirroring the counters for active block groups into their own counters. This allows us to keep the full space_info counters consistent, which is needed for things like sysfs and the space info ioctl, and then track the actual usage for ENOSPC reasons for only the active block groups.

I think the mirroring the counters approach duplicates the code and variables and makes them complex. Instead, we can fix the "active_total_bytes" accounting maybe like this:

fs/btrfs/space-info.c +2 -1
index d4dd73c9a701..bf4d96d74efe 100644
@@ -319,7 +319,8 @@ void btrfs_add_bg_to_space_info(struct btrfs_fs_info *info,
 	ASSERT(found);
 	spin_lock(&found->lock);
 	found->total_bytes += block_group->length;
-	if (test_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE, &block_group->runtime_flags))
+	if (test_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE, &block_group->runtime_flags) ||
+	    btrfs_zoned_bg_is_full(block_group))
 		found->active_total_bytes += block_group->length;
 	found->disk_total += block_group->length * factor;
 	found->bytes_used += block_group->used;

Or, we can remove "active_total_bytes" and introduce something like "preactivated_bytes" to count the bytes of BGs never get activated (BGs #1 below).

There are three kinds of block groups.

1. Block groups never activated
2. Block groups currently active
3. Block groups previously active and currently inactive (due to fully written or zone finish)

What we really want to exclude from "total_bytes" is the total size of BGs #1. They seem empty and allocatable but since they are not activated, we cannot rely on them to do the space reservation.

And, since BGs #1 never get activated, they should have no "used", "reserved" and "pinned" bytes.

OTOH, BGs #3 is OK, since they are already full we cannot allocate from them anyway. For them, "total_bytes == used + reserved + pinned + zone_unusable" should hold.

Quoted 4 lines

Additionally, when de-activating we weren't properly updating the ->active_total_bytes, which could lead to other problems. Unifying all of this with the proper helpers will make sure our accounting is correct.

So, de-activating not reducing the "active_total_bytes" should be OK ... but I admit its name is confusing. It should be "active_and_finished_total_bytes" ? "once_activated_total_bytes" ? I still don't have a good idea.

Quoted 9 lines
---
 fs/btrfs/block-group.c | 29 +++++++++++++--
 fs/btrfs/disk-io.c     |  6 +++
 fs/btrfs/extent-tree.c | 13 +++++++
 fs/btrfs/space-info.c  | 83 ++++++++++++++++++++++++++++++++++++++++--
 fs/btrfs/space-info.h  | 20 +++++++++-
 fs/btrfs/zoned.c       | 14 +++++--
 6 files changed, 152 insertions(+), 13 deletions(-)
#11 lore raw

Re: [PATCH 3/3] btrfs: handle active zone accounting properly

Quoted 42 lines · Naohiro Aota
Quoted 16 lines · Josef Bacik

Running xfstests on my ZNS drives uncovered a problem where I was allocating the entire device with metadata block groups. The root cause of this was we would get ENOSPC on mount, and while trying to satisfy tickets we'd allocate more block groups.

The reason we were getting ENOSPC was because ->bytes_zone_unusable was set to 40gib, but ->active_total_bytes was set to 8gib, which was the maximum number of active block groups we're allowed to have at one time. This was because we always update ->bytes_zone_unusable with the unusable amount from every block group, but we only update ->active_total_bytes with the active block groups.

This is actually much worse however, because we could potentially have other counters potentially well above the ->active_total_bytes, which would lead to this early enospc situation.

Quoted 5 lines

Fix this by mirroring the counters for active block groups into their own counters. This allows us to keep the full space_info counters consistent, which is needed for things like sysfs and the space info ioctl, and then track the actual usage for ENOSPC reasons for only the active block groups.

I think the mirroring the counters approach duplicates the code and variables and makes them complex. Instead, we can fix the "active_total_bytes" accounting maybe like this:

fs/btrfs/space-info.c +2 -1
index d4dd73c9a701..bf4d96d74efe 100644
@@ -319,7 +319,8 @@ void btrfs_add_bg_to_space_info(struct btrfs_fs_info *info,
 	ASSERT(found);
 	spin_lock(&found->lock);
 	found->total_bytes += block_group->length;
-	if (test_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE, &block_group->runtime_flags))
+	if (test_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE, &block_group->runtime_flags) ||
+	    btrfs_zoned_bg_is_full(block_group))
 		found->active_total_bytes += block_group->length;
 	found->disk_total += block_group->length * factor;
 	found->bytes_used += block_group->used;
Quoted 2 lines

Or, we can remove "active_total_bytes" and introduce something like "preactivated_bytes" to count the bytes of BGs never get activated (BGs #1 below).

I got a new idea. How about counting all the region in a new block group as zone_unusable? Then, region [0 .. zone_capacity] will be freed for use once it gets activated. With this, we can drop "active_total_bytes" so the code will become similar between the regular and the zoned mode.

We also need to care not to reclaim the fresh BGs but it's trivial (alloc_offset == 0).

Quoted 37 lines

There are three kinds of block groups.

1. Block groups never activated
2. Block groups currently active
3. Block groups previously active and currently inactive (due to fully written or zone finish)

What we really want to exclude from "total_bytes" is the total size of BGs #1. They seem empty and allocatable but since they are not activated, we cannot rely on them to do the space reservation.

And, since BGs #1 never get activated, they should have no "used", "reserved" and "pinned" bytes.

OTOH, BGs #3 is OK, since they are already full we cannot allocate from them anyway. For them, "total_bytes == used + reserved + pinned + zone_unusable" should hold.

Quoted 4 lines

Additionally, when de-activating we weren't properly updating the ->active_total_bytes, which could lead to other problems. Unifying all of this with the proper helpers will make sure our accounting is correct.

So, de-activating not reducing the "active_total_bytes" should be OK ... but I admit its name is confusing. It should be "active_and_finished_total_bytes" ? "once_activated_total_bytes" ? I still don't have a good idea.

Quoted 9 lines
---
 fs/btrfs/block-group.c | 29 +++++++++++++--
 fs/btrfs/disk-io.c     |  6 +++
 fs/btrfs/extent-tree.c | 13 +++++++
 fs/btrfs/space-info.c  | 83 ++++++++++++++++++++++++++++++++++++++++--
 fs/btrfs/space-info.h  | 20 +++++++++-
 fs/btrfs/zoned.c       | 14 +++++--
 6 files changed, 152 insertions(+), 13 deletions(-)
#12 lore raw

Re: [PATCH 3/3] btrfs: handle active zone accounting properly

Quoted 51 lines · Naohiro Aota
Quoted 42 lines · Naohiro Aota
Quoted 16 lines · Josef Bacik

Running xfstests on my ZNS drives uncovered a problem where I was allocating the entire device with metadata block groups. The root cause of this was we would get ENOSPC on mount, and while trying to satisfy tickets we'd allocate more block groups.

The reason we were getting ENOSPC was because ->bytes_zone_unusable was set to 40gib, but ->active_total_bytes was set to 8gib, which was the maximum number of active block groups we're allowed to have at one time. This was because we always update ->bytes_zone_unusable with the unusable amount from every block group, but we only update ->active_total_bytes with the active block groups.

This is actually much worse however, because we could potentially have other counters potentially well above the ->active_total_bytes, which would lead to this early enospc situation.

Quoted 5 lines

Fix this by mirroring the counters for active block groups into their own counters. This allows us to keep the full space_info counters consistent, which is needed for things like sysfs and the space info ioctl, and then track the actual usage for ENOSPC reasons for only the active block groups.

I think the mirroring the counters approach duplicates the code and variables and makes them complex. Instead, we can fix the "active_total_bytes" accounting maybe like this:

fs/btrfs/space-info.c +2 -1
index d4dd73c9a701..bf4d96d74efe 100644
@@ -319,7 +319,8 @@ void btrfs_add_bg_to_space_info(struct btrfs_fs_info *info,
 	ASSERT(found);
 	spin_lock(&found->lock);
 	found->total_bytes += block_group->length;
-	if (test_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE, &block_group->runtime_flags))
+	if (test_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE, &block_group->runtime_flags) ||
+	    btrfs_zoned_bg_is_full(block_group))
 		found->active_total_bytes += block_group->length;
 	found->disk_total += block_group->length * factor;
 	found->bytes_used += block_group->used;
Quoted 2 lines

Or, we can remove "active_total_bytes" and introduce something like "preactivated_bytes" to count the bytes of BGs never get activated (BGs #1 below).

I got a new idea. How about counting all the region in a new block group as zone_unusable? Then, region [0 .. zone_capacity] will be freed for use once it gets activated. With this, we can drop "active_total_bytes" so the code will become similar between the regular and the zoned mode.

Quoted 3 lines

We also need to care not to reclaim the fresh BGs but it's trivial (alloc_offset == 0).

I like this idea better than mine for sure. Will you write it up and send it and I'll run it through my test-rig? Thanks,

Josef