Fix for btrfs/156 failure and misc enhancements relocated to relocation

Btrfs/156 fails after commit 302167c50b32 ("btrfs: don't end the transaction for delayed refs in throttle"), however it's not that commit's fault.

The root cause is btrfs_can_relocate() can report false ENOSPC when there is any block group freed in current transaction.

However thanks to all the hard work due to btrfs balance, we don't really need some early ENOSPC check, as the current ENOSPC report at extent reservation time is good enough for balance.

So the main fix is just to remove btrfs_can_relocate(), in patch 4.

Patch 1~3 are miscellaneous patches. Patch 1 unexport find_free_dev_extent_start(), Patch 2 and 3 adds comment for possible confusing behaviors, like find_free_dev_extent_start() only searches commit root, and inc_block_group_ro() only cares if we have 1M free space left, not caring if other block groups can contain the used space.

Any of the miscellaneous patches can be dropped if needed.

Changelog:
v2:
- Add supporting miscellaneous patches
- Commit message change for the main fix
* Directly explain how the ENOSPC is caused
* Add comment for what will happen if we really run out of space

Qu Wenruo (4):
btrfs: volumes: Unexport find_free_dev_extent_start()
btrfs: volumes: Add comment for find_free_dev_extent_start()
btrfs: extent-tree: Add comment for inc_block_group_ro()
btrfs: volumes: Remove ENOSPC-prone btrfs_can_relocate()

 fs/btrfs/ctree.h       |   1 -
 fs/btrfs/extent-tree.c | 160 +++++------------------------------------
 fs/btrfs/volumes.c     |  15 ++--
 fs/btrfs/volumes.h     |   2 -
 4 files changed, 28 insertions(+), 150 deletions(-)

2.22.0

This function is only used locally in find_free_dev_extent(), no external callers.

So unexport it.

---
 fs/btrfs/volumes.c | 5 +++--
 fs/btrfs/volumes.h | 2 --
 2 files changed, 3 insertions(+), 4 deletions(-)

index 1c2a6e4b39da..b3fae492912d 100644
@@ -1547,8 +1547,9 @@ static bool contains_pending_extent(struct btrfs_device *device, u64 *start,
  * But if we don't find suitable free space, it is used to store the size of
  * the max free space.
  */
-int find_free_dev_extent_start(struct btrfs_device *device, u64 num_bytes,
-			       u64 search_start, u64 *start, u64 *len)
+static int find_free_dev_extent_start(struct btrfs_device *device,
+				u64 num_bytes, u64 search_start, u64 *start,
+				u64 *len)
 {
 	struct btrfs_fs_info *fs_info = device->fs_info;
 	struct btrfs_root *root = fs_info->dev_root;

index 136a3eb64604..371cd05d7fe4 100644
@@ -454,8 +454,6 @@ int btrfs_cancel_balance(struct btrfs_fs_info *fs_info);
 int btrfs_create_uuid_tree(struct btrfs_fs_info *fs_info);
 int btrfs_check_uuid_tree(struct btrfs_fs_info *fs_info);
 int btrfs_chunk_readonly(struct btrfs_fs_info *fs_info, u64 chunk_offset);
-int find_free_dev_extent_start(struct btrfs_device *device, u64 num_bytes,
-			       u64 search_start, u64 *start, u64 *max_avail);
 int find_free_dev_extent(struct btrfs_device *device, u64 num_bytes,
 			 u64 *start, u64 *max_avail);
 void btrfs_dev_stat_inc_and_print(struct btrfs_device *dev, int index);

2.22.0

Looks good,

Johannes Thumshirn                            SUSE Labs Filesystems
[email protected]                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

Since commit 6df9a95e6339 ("Btrfs: make the chunk allocator completely tree lockless") we search commit root of device tree to avoid deadlock.

This introduced a safety feature, find_free_dev_extent_start() won't use dev extents which just get freed in current transaction.

This safety feature makes sure we won't allocate new block group using just freed dev extents to break CoW.

However this feature also makes find_free_dev_extent_start() not reliable reporting free device space.
Just add such comment to make later viewer careful about this behavior.

This behavior makes one caller, btrfs_can_relocate() unreliable determining the device free space.

---
 fs/btrfs/volumes.c | 6 ++++++
 1 file changed, 6 insertions(+)

index b3fae492912d..41811522c4a3 100644
@@ -1546,6 +1546,12 @@ static bool contains_pending_extent(struct btrfs_device *device, u64 *start,
  * @len is used to store the size of the free space that we find.
  * But if we don't find suitable free space, it is used to store the size of
  * the max free space.
+ *
+ * NOTE: This function will search *commit* root of device tree, and does extra
+ * check to ensure dev extents are not double allocated.
+ * This makes the function safe to allocate dev extents but may not report
+ * correct usable device space, as device extent freed in current transaction
+ * is not reported as avaiable.
  */
 static int find_free_dev_extent_start(struct btrfs_device *device,
 				u64 num_bytes, u64 search_start, u64 *start,

2.22.0

Looks good,

Btw: this comment block lacks a second '*' at the start to be valid kernel doc

-/*
+/**
* find_free_dev_extent_start - find free space in the specified device

Johannes Thumshirn                            SUSE Labs Filesystems
[email protected]                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

inc_block_group_ro() is only designed to mark one block group read-only, it doesn't really care if other block groups have enough free space to contain the used space in the block group.

However due to the close connection between this function and relocation, sometimes we can be confused and think this function is responsible for balance space reservation, which is not true.

Add some comment to make the functionality clear.

---
 fs/btrfs/extent-tree.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

index 5faf057f6f37..156ef906885f 100644
@@ -9586,6 +9586,19 @@ static u64 update_block_group_flags(struct btrfs_fs_info *fs_info, u64 flags)
 	return flags;
 }
 
+/*
+ * Mark block group @cache read-only, so later write won't happen to block
+ * group @cache.
+ *
+ * If @force is not set, this function will only mark the block group readonly
+ * if we have enough free space (1M) in other metadata/system block groups.
+ * If @force is not set, this function will mark the block group readonly
+ * without checking free space.
+ *
+ * NOTE: This function doesn't care if other block groups can contain all the
+ * data in this block group. That check should be done by relocation routine,
+ * not this function.
+ */
 static int inc_block_group_ro(struct btrfs_block_group_cache *cache, int force)
 {
 	struct btrfs_space_info *sinfo = cache->space_info;

@@ -9619,6 +9632,12 @@ static int inc_block_group_ro(struct btrfs_block_group_cache *cache, int force)
 		    cache->bytes_super - btrfs_block_group_used(&cache->item);
 	sinfo_used = btrfs_space_info_used(sinfo, true);
 
+	/*
+	 * sinfo_used + num_bytes should always <= sinfo->total_bytes.
+	 *
+	 * Here we make sure if we mark this bg RO, we still have enough
+	 * free space as buffer (if min_allocable_bytes is not 0).
+	 */
 	if (sinfo_used + num_bytes + min_allocable_bytes <=
 	    sinfo->total_bytes) {
 		sinfo->bytes_readonly += num_bytes;

2.22.0

Looks good,

Johannes Thumshirn                            SUSE Labs Filesystems
[email protected]                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

[BUG]
Test case btrfs/156 fails since commit 302167c50b32 ("btrfs: don't end the transaction for delayed refs in throttle") with ENOSPC.

[CAUSE]
The ENOSPC is reported from btrfs_can_relocate().

This function will check:
- If this block group is empty, we can relocate
- If we can enough free space, we can relocate

Above checks are valid but the following check is vague due to its implementation:
- If and only if we can allocated a new block group to contain all the used space, we can relocate

This design itself is OK, but the way to determine if we can allocate a new block group is problematic.

btrfs_can_relocate() uses find_free_dev_extent() to find free space on a device.
However find_free_dev_extent() only searches commit root and excludes dev extents allocated in current trans, this makes it unable to use dev extent just freed in current transaction.

So for the following example, btrfs_can_relocate() will report ENOSPC:
The example block group layout:

1M      129M        257M       385M      513M       550M
|///////|///////////|//////////|         |          |

// = Used bg, consider all bg is 100% used for easy calculation. And all block groups are SINGLE, on-disk bytenr is the same as the logical bytenr.

1) Bg in [129M, 257M) get relocated to [385M, 513M), transid=100

1M      129M        257M       385M      513M       550M
|///////|           |//////////|/////////|

In transid 100, bg in [129M, 257M) get relocated to [385M, 513M)

However transid 100 is not committed yet, so in dev commit tree, we still have the old dev extents layout:

1M      129M        257M       385M      513M       550M
|///////|///////////|//////////|         |          |

2) Try to relocate bg [257M, 385M)
We goes into btrfs_can_relocate(), no free space in current bgs, so we check if we can find large enough free dev extents.

The first slot is [385M, 513M), but that is already used by new bg at [385M, 513M), so we continue search.

The remaining slot is [512M, 550M), smaller than the bg's length 128M. So btrfs_can_relocate report ENOSPC.

However this is over killed, in fact if we just skip btrfs_can_relocate() check, and go into regular relocation routine, at extent reservation time, if we can't find free extent, then we fallback to commit transaction, which will free up the dev extents and allow new block group to be created.

[FIX]
The fix here is to remove btrfs_can_relocate() completely.

If we hit the false ENOSPC case just like btrfs/156, extent allocator will push harder by committing transaction and we will have space for new block group, avoiding the false ENOSPC.

If we really ran out of space, we will hit ENOSPC at relocate_block_group(), and btrfs will just reports the ENOSPC error as usual.

---
 fs/btrfs/ctree.h       |   1 -
 fs/btrfs/extent-tree.c | 141 -----------------------------------------
 fs/btrfs/volumes.c     |   4 --
 3 files changed, 146 deletions(-)

index 0a61dff27f57..965d1e5a4af7 100644
@@ -2772,7 +2772,6 @@ int btrfs_setup_space_cache(struct btrfs_trans_handle *trans);
 int btrfs_extent_readonly(struct btrfs_fs_info *fs_info, u64 bytenr);
 int btrfs_free_block_groups(struct btrfs_fs_info *info);
 int btrfs_read_block_groups(struct btrfs_fs_info *info);
-int btrfs_can_relocate(struct btrfs_fs_info *fs_info, u64 bytenr);
 int btrfs_make_block_group(struct btrfs_trans_handle *trans,
 			   u64 bytes_used, u64 type, u64 chunk_offset,
 			   u64 size);

index 41811522c4a3..431cabdcbd33 100644
@@ -3078,10 +3078,6 @@ static int btrfs_relocate_chunk(struct btrfs_fs_info *fs_info, u64 chunk_offset)
 	 */
 	lockdep_assert_held(&fs_info->delete_unused_bgs_mutex);
 
-	ret = btrfs_can_relocate(fs_info, chunk_offset);
-	if (ret)
-		return -ENOSPC;
-
 	/* step one, relocate all the extents inside this chunk */
 	btrfs_scrub_pause(fs_info);
 	ret = btrfs_relocate_block_group(fs_info, chunk_offset);

2.22.0

The commit that introduced the function btrfs_can_relocate is from 2009 and the commit mentions painc during balance when there's no space. This was due to lack of error handling that is present now, so we don't really need to check in advance.

I think it's safe, I'll add the patches to misc-next, thanks.