Skip to contentSkip to conversation

btrfs: zoned: RAID1 block groups with missing or stale stripes

PATCH Superseded Miroslav Cimerman 1 participant 5 messages Nov 19 Btrfs file system

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover btrfs: zoned: RAID1 block groups with missing or stale stripes

btrfs_load_block_group_raid1() compares the active state and the write pointer of each present stripe with stripe 0's, and takes the zone capacity from stripes 0 and 1. But stripe 0 can be missing, and on a degraded mount any stripe can be behind the others.

1/4 compares the active state with the first present stripe, as with stripe 0 missing such block groups were left off the active list and could hang writes. 2/4 allocates from the highest write pointer on degraded mounts, as stripe 0's can be behind and new extents then went over committed ones, and lets a regular mount load a block group whose stripe 0 is on an offline or read-only zone. 3/4 takes the zone capacity from the other stripes when stripes 0 and 1 are missing, as RAID1C3 and RAID1C4 block groups failed to load then. 4/4 adds self-test cases.

Miroslav Cimerman (4):
btrfs: zoned: take the RAID1 active state from the first present
stripe
btrfs: zoned: use the highest RAID1 write pointer on degraded mounts
btrfs: zoned: take the RAID1 zone capacity from a present stripe
btrfs: tests: zoned: add degraded RAID1 cases

 fs/btrfs/tests/zoned-tests.c | 82 +++++++++++++++++++++++++++++++++++-
 fs/btrfs/zoned.c             | 50 +++++++++++++++++-----
 2 files changed, 120 insertions(+), 12 deletions(-)

base-commit: 04c9eb073c7c02bf041948ca35b3d8b3ef582974

Signature · 1 line
2.53.0
#1 lore raw

PATCH 1/4 btrfs: zoned: take the RAID1 active state from the first present stripe

btrfs_load_block_group_raid1() compares the active state of each present stripe's zone with stripe 0's. A missing device has no active zones, so with stripe 0 missing, an active zone of a present stripe always mismatches, and a degraded mount neither activates the block group nor marks it active. It is then not on the list of active block groups, while its zones are active and count against their devices' active zone limits.

If it is a data block group, writing data to it tries to activate it, which fails when the active zones left are reserved for metadata. With no other data block group with room left, the allocation then returns -EAGAIN, and the write waits for a zone finish forever:

task:dd state:D
btrfs_new_extent_direct
btrfs_dio_iomap_next
iomap_iter
__iomap_dio_rw
btrfs_direct_write
btrfs_do_write_iter
vfs_write

If it is a metadata block group, writing tree blocks to it activates it with a warning:

  WARNING: fs/btrfs/zoned.c:2451 at btrfs_zone_activate+0x72a/0xa60, CPU#1: btrfs/157
   btrfs_check_meta_write_pointer
   write_meta_extent_buffer
   btree_writepages

Compare with the first present stripe instead.

Fixes 568220fa9657 ("btrfs: zoned: support RAID0/1/10 on top of raid stripe tree")

---
Found by an LLM while testing the next patch. The fix and the commit message were prepared with the help of an LLM, which also tested it. I reviewed everything.

A mismatch between present stripes, like a stripe behind the others on a degraded mount, is not changed. Depending on which stripe comes first, the block group is marked active at load without the zone of the other stripe accounted, or activated at its first write if it has room left, which can fail as above when no active zone is left. Both happened before too. I can look at the active zone accounting for these zones separately.

btrfs_load_block_group_raid10() compares the active state with stripe 0's too, but activates the block group on a mismatch instead of skipping it. With stripe 0 missing, that warns at mount for a metadata block group, and can fail the mount for a data block group when no active zone is left. RAID10 also adds a zone capacity of 0 for a missing first stripe of a pair, which fails the degraded mount once a block group holds more than the capacity that is left. Neither is changed in this series. I will send fixes for both.

 fs/btrfs/zoned.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
fs/btrfs/zoned.c +6 -2
index e6ce6f05ad75..8eac6e2283a2 100644
@@ -1522,6 +1522,7 @@ static int btrfs_load_block_group_raid1(struct btrfs_block_group *bg,
 					u64 last_alloc)
 {
 	struct btrfs_fs_info *fs_info = bg->fs_info;
+	int first = -1;
 	int i;
 
 	if ((map->type & BTRFS_BLOCK_GROUP_DATA) && !fs_info->stripe_root) {
@@ -1555,6 +1556,9 @@ static int btrfs_load_block_group_raid1(struct btrfs_block_group *bg,
 		if (zone_info[i].alloc_offset == WP_CONVENTIONAL)
 			zone_info[i].alloc_offset = last_alloc;
 
+		if (first < 0)
+			first = i;
+
 		if (unlikely((zone_info[0].alloc_offset != zone_info[i].alloc_offset) &&
 			     !btrfs_test_opt(fs_info, DEGRADED))) {
 			btrfs_err(fs_info,
@@ -1562,13 +1566,13 @@ static int btrfs_load_block_group_raid1(struct btrfs_block_group *bg,
 				  btrfs_bg_type_to_raid_name(map->type));
 			return -EIO;
 		}
-		if (test_bit(0, active) != test_bit(i, active)) {
+		if (test_bit(first, active) != test_bit(i, active)) {
 			if (unlikely(!btrfs_test_opt(fs_info, DEGRADED) &&
 				     !btrfs_zone_activate(bg))) {
 				return -EIO;
 			}
 		} else {
-			if (test_bit(0, active))
+			if (test_bit(first, active))
 				set_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE, &bg->runtime_flags);
 		}
 	}
Signature · 1 line
2.53.0
#2 lore raw

PATCH 2/4 btrfs: zoned: use the highest RAID1 write pointer on degraded mounts

In reply to Miroslav Cimerman

The write pointers of a RAID1 block group's stripes can differ after a crash, or when a device missed committed writes and came back. A regular mount fails then, and commit b0c26f479926 ("btrfs: zoned:
return EIO on RAID1 block group write pointer mismatch") relies on manual intervention: a degraded mount and a full balance. On a degraded mount, btrfs_load_block_group_raid1() ignores the mismatch and takes the allocation offset from stripe 0. If stripe 0 is behind, new extents are allocated over committed ones:

  stripe 0 (behind)  | a | a |   |   |
  stripe 1           | a | a | b | b |

^
allocation offset from stripe 0, new extents go over b

BTRFS error (device nb2): failed to run delayed ref for logical 155189248 num_bytes 1048576 type 178 action 1 ref_mod 1: -17 BTRFS error (device nb2 state A): Transaction 18 aborted (-EEXIST)

or, with CONFIG_BTRFS_ASSERT, an assertion in the free space tree fails:

  assertion failed: start >= found_start && end <= found_end, in fs/btrfs/free-space-tree.c:742

A stripe that is not behind has all committed extents below its write pointer, the highest one. If no present stripe is up to date, the highest write pointer can still be below committed extents, as before. On a degraded mount, take the allocation offset from it, and warn about the block group. Until a balance copies the block group to new zones, reads of what the stripes behind miss fail their checksums and go to the other copies, and writes to them can fail, as they already could when stripe 0 was ahead. Data without checksums can return what a stripe behind holds there.

Compare the write pointers with the first present stripe, and leave missing stripes out of the highest one. With stripe 0 missing, the allocation offset came from the last stripe, which can be missing as well in RAID1C3 and RAID1C4, and the block group then failed to load.

An offline or read-only zone, or one whose zone report fails with -EIO or -EOPNOTSUPP, counts as a missing stripe. With stripe 0's zone in that state, a regular mount failed on the comparison, while with another stripe's zone in that state it loaded. Now both load.

Fixes 568220fa9657 ("btrfs: zoned: support RAID0/1/10 on top of raid stripe tree")

---
Found with FuzzyRAID, a fault-injection tester for RAID that is not public at the moment. The analysis, the fix and the commit message were prepared with the help of an LLM, which also tested it. I reviewed everything.

Making such a block group unallocatable instead, as b1934cd60695 did, was replaced by failing the mount in b0c26f479926. With the highest write pointer, the degraded mount and balance that b0c26f479926 describes keep working, and in testing, a regular mount works again after the balance.

Scrub on such a degraded mount finds what the stripes behind miss, but does not repair it, as zoned repairs are skipped on degraded mounts. Repairing there would let scrub fix the block groups where it finds errors, instead of a balance. That is discussed in the thread of "btrfs: scrub: don't count zoned repairs as corrected on a degraded mount":
https://lore.kernel.org/linux-btrfs/[email protected]/

 fs/btrfs/zoned.c | 36 +++++++++++++++++++++++++++---------
 1 file changed, 27 insertions(+), 9 deletions(-)
fs/btrfs/zoned.c +27 -9
index 8eac6e2283a2..8bd44dd260a4 100644
@@ -1522,6 +1522,8 @@ static int btrfs_load_block_group_raid1(struct btrfs_block_group *bg,
 					u64 last_alloc)
 {
 	struct btrfs_fs_info *fs_info = bg->fs_info;
+	u64 highest = 0;
+	bool mismatch = false;
 	int first = -1;
 	int i;
 
@@ -1559,13 +1561,24 @@ static int btrfs_load_block_group_raid1(struct btrfs_block_group *bg,
 		if (first < 0)
 			first = i;
 
-		if (unlikely((zone_info[0].alloc_offset != zone_info[i].alloc_offset) &&
-			     !btrfs_test_opt(fs_info, DEGRADED))) {
-			btrfs_err(fs_info,
-			"zoned: write pointer offset mismatch of zones in %s profile",
-				  btrfs_bg_type_to_raid_name(map->type));
-			return -EIO;
+		if (zone_info[first].alloc_offset != zone_info[i].alloc_offset) {
+			/*
+			 * A stripe is behind the others after a crash, or when
+			 * its device missed committed writes and came back.
+			 * This fails a regular mount. A degraded one allocates
+			 * from the highest write pointer: committed extents are
+			 * below it, unless all present stripes are behind.
+			 */
+			if (unlikely(!btrfs_test_opt(fs_info, DEGRADED))) {
+				btrfs_err(fs_info,
+				"zoned: write pointer offset mismatch of zones in %s profile",
+					  btrfs_bg_type_to_raid_name(map->type));
+				return -EIO;
+			}
+			mismatch = true;
 		}
+		highest = max(highest, zone_info[i].alloc_offset);
+
 		if (test_bit(first, active) != test_bit(i, active)) {
 			if (unlikely(!btrfs_test_opt(fs_info, DEGRADED) &&
 				     !btrfs_zone_activate(bg))) {
@@ -1577,10 +1590,15 @@ static int btrfs_load_block_group_raid1(struct btrfs_block_group *bg,
 		}
 	}
 
-	if (zone_info[0].alloc_offset != WP_MISSING_DEV)
-		bg->alloc_offset = zone_info[0].alloc_offset;
+	if (mismatch)
+		btrfs_warn(fs_info,
+		"zoned: write pointer mismatch in %s block group %llu, using the highest",
+			   btrfs_bg_type_to_raid_name(map->type), bg->start);
+
+	if (first >= 0)
+		bg->alloc_offset = highest;
 	else
-		bg->alloc_offset = zone_info[i - 1].alloc_offset;
+		bg->alloc_offset = WP_MISSING_DEV;
 
 	return 0;
 }
Signature · 1 line
2.53.0
#3 lore raw

PATCH 3/4 btrfs: zoned: take the RAID1 zone capacity from a present stripe

In reply to Miroslav Cimerman

btrfs_load_block_group_raid1() takes the zone capacity of the block group from stripes 0 and 1. A missing device has a capacity of 0, which min_not_zero() skips, but a RAID1C3 or RAID1C4 block group with both stripes missing gets a capacity of 0. Once it has been written to, it fails to load, and a degraded mount fails although a copy is left:

BTRFS error (device nb4): zoned: invalid write pointer 98304 (larger than zone capacity 0) in block group 83886080 BTRFS error (device nb4): zoned: failed to load zone info of bg 83886080
BTRFS error (device nb4): failed to read block groups: -EIO

In that case, take the smallest non-zero capacity of the other stripes.

Fixes 568220fa9657 ("btrfs: zoned: support RAID0/1/10 on top of raid stripe tree")

---
Found in an LLM review of the previous patches. The fix and the commit message were prepared with the help of an LLM, which also tested it. I reviewed everything.

Taking the smallest capacity of all stripes would also cover a third or fourth stripe with a smaller capacity, but a block group written past it would then fail to load, so it is not done here.

 fs/btrfs/zoned.c | 6 ++++++
 1 file changed, 6 insertions(+)
fs/btrfs/zoned.c +6 -0
index 8bd44dd260a4..5722e16a5432 100644
@@ -1535,6 +1535,12 @@ static int btrfs_load_block_group_raid1(struct btrfs_block_group *bg,
 
 	/* In case a device is missing we have a cap of 0, so don't use it. */
 	bg->zone_capacity = min_not_zero(zone_info[0].capacity, zone_info[1].capacity);
+	/* RAID1C3 and RAID1C4 can miss both of them, take the others then. */
+	if (!bg->zone_capacity) {
+		for (i = 2; i < map->num_stripes; i++)
+			bg->zone_capacity = min_not_zero(bg->zone_capacity,
+							 zone_info[i].capacity);
+	}
 
 	/*
 	 * When the last extent is removed, last_alloc can be smaller than the other write
Signature · 1 line
2.53.0
#4 lore raw

PATCH 4/4 btrfs: tests: zoned: add degraded RAID1 cases

In reply to Miroslav Cimerman

Add cases for the previous patches. On a degraded mount: RAID1 with the first device missing, which has to be marked active, RAID1C3 with the first and the last and with the first two devices missing, and RAID1 and RAID1C3 with different write pointers, which take the highest one. On a regular mount: RAID1 with the first zone offline, which now loads.

Give missing devices a zone capacity of 0, as btrfs_load_zone_info() does, and check the zone capacity and the active state where a case asks for it. All six cases fail without the previous patches.

---
Prepared with the help of an LLM, which also ran the tests. I reviewed everything.

 fs/btrfs/tests/zoned-tests.c | 82 +++++++++++++++++++++++++++++++++++-
 1 file changed, 81 insertions(+), 1 deletion(-)
fs/btrfs/tests/zoned-tests.c +81 -1
index bb9969210444..9cc5b32bb2ee 100644
@@ -27,6 +27,8 @@ struct load_zone_info_test_vector {
 
 	int expected_result;
 	u64 expected_alloc_offset;
+	u64 expected_zone_capacity;
+	bool expected_active;
 
 	const char *description;
 };
@@ -77,7 +79,9 @@ static int test_load_zone_info(struct btrfs_fs_info *fs_info,
 	for (int i = 0; i < test->num_stripes; i++) {
 		zone_info[i].physical = 0;
 		zone_info[i].alloc_offset = test->alloc_offsets[i];
-		zone_info[i].capacity = ZONE_SIZE;
+		/* A missing device has a capacity of 0. */
+		if (zone_info[i].alloc_offset != WP_MISSING_DEV)
+			zone_info[i].capacity = ZONE_SIZE;
 		if (zone_info[i].alloc_offset && zone_info[i].alloc_offset < ZONE_SIZE)
 			__set_bit(i, active);
 	}
@@ -101,6 +105,19 @@ static int test_load_zone_info(struct btrfs_fs_info *fs_info,
 		return -EINVAL;
 	}
 
+	if (!ret && test->expected_zone_capacity &&
+	    bg->zone_capacity != test->expected_zone_capacity) {
+		test_err("unexpected zone_capacity: zone_capacity %llu expected %llu",
+			 bg->zone_capacity, test->expected_zone_capacity);
+		return -EINVAL;
+	}
+
+	if (!ret && test->expected_active &&
+	    !test_bit(BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE, &bg->runtime_flags)) {
+		test_err("block group not marked active");
+		return -EINVAL;
+	}
+
 	return 0;
 }
 
@@ -244,6 +261,69 @@ static const struct load_zone_info_test_vector load_zone_info_tests[] = {
 		.degraded = true,
 		.expected_alloc_offset = SZ_1M,
 	},
+	{
+		.description = "RAID1: missing first device on DEGRADED",
+		.raid_type = BTRFS_BLOCK_GROUP_RAID1,
+		.num_stripes = 2,
+		.alloc_offsets = {
+			WP_MISSING_DEV, SZ_1M,
+		},
+		.degraded = true,
+		.expected_alloc_offset = SZ_1M,
+		.expected_active = true,
+	},
+	{
+		.description = "RAID1C3: missing first and last device on DEGRADED",
+		.raid_type = BTRFS_BLOCK_GROUP_RAID1C3,
+		.num_stripes = 3,
+		.alloc_offsets = {
+			WP_MISSING_DEV, SZ_1M, WP_MISSING_DEV,
+		},
+		.degraded = true,
+		.expected_alloc_offset = SZ_1M,
+	},
+	{
+		.description = "RAID1C3: missing first and second device on DEGRADED",
+		.raid_type = BTRFS_BLOCK_GROUP_RAID1C3,
+		.num_stripes = 3,
+		.alloc_offsets = {
+			WP_MISSING_DEV, WP_MISSING_DEV, SZ_1M,
+		},
+		.degraded = true,
+		.expected_alloc_offset = SZ_1M,
+		.expected_zone_capacity = ZONE_SIZE,
+	},
+	/* A stripe behind the others: the highest write pointer on DEGRADED. */
+	{
+		.description = "RAID1: different write pointers on DEGRADED",
+		.raid_type = BTRFS_BLOCK_GROUP_RAID1,
+		.num_stripes = 2,
+		.alloc_offsets = {
+			SZ_1M, SZ_2M,
+		},
+		.degraded = true,
+		.expected_alloc_offset = SZ_2M,
+	},
+	{
+		.description = "RAID1C3: first missing, different write pointers on DEGRADED",
+		.raid_type = BTRFS_BLOCK_GROUP_RAID1C3,
+		.num_stripes = 3,
+		.alloc_offsets = {
+			WP_MISSING_DEV, SZ_2M, SZ_1M,
+		},
+		.degraded = true,
+		.expected_alloc_offset = SZ_2M,
+	},
+	/* An offline zone has no write pointer on a regular mount either. */
+	{
+		.description = "RAID1: first zone offline",
+		.raid_type = BTRFS_BLOCK_GROUP_RAID1,
+		.num_stripes = 2,
+		.alloc_offsets = {
+			WP_MISSING_DEV, SZ_1M,
+		},
+		.expected_alloc_offset = SZ_1M,
+	},
 	/* Error case: having different write pointers. */
 	{
 		.description = "RAID1: fail: different write pointers",
Signature · 1 line
2.53.0