Skip to contentSkip to conversation

btrfs-progs: add RAID56 rebuild ability at read time

PATCH In review Qu Wenruo 3 participants 13 messages Apr 5, 2022 – Apr 25, 2022 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

cover btrfs-progs: add RAID56 rebuild ability at read time

This branch can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/raid56_rebuild

Please note that, the last time I check devel branch, the RAID56 warning patch is not yet merged.

So this is based on the latest devel branch from github.

And since we're adding proper RAID56 repair ability, there is no need
for the patchset "btrfs-progs: check: avoid false alerts for --check-data-csum on RAID56"

This patchset is mostly to add the ability to properly read data/metadata for RAID56 when mirror_num > 1.

[PROBLEMS]
There are several for current btrfs-progs:

- No RAID56 rebuild ability
Thus for any mirror > 1 on RAID56, we will read the parity data straight, causing no way to recover.

- No unified logical read/write entrance
We have various different functions, for read we have read_data_from_disk(), read_extent_data() and read_extent_from_disk().

Unlike kernel, we have no btrfs_map_bio() to address everything.

This makes adding RAID56 rebuild even hard.

[FIXES]
To address the problem, this patchset will:

- Cleanup and refactors
Mostly to unify the logical read/write entrance.
In patch 1~6, we will unify the logical read write entrances to only two functions:
* read_extent_from_disk()
* write_extent_to_disk()

They will do the chunk mapping and stripe splitting.

- Add RAID56 read rebuild ability
We already have RAID56 RMW for write path. Just add a new helper for mirror > 1 read on RAID56.

- A new test case
In fact the same test case from previous RAID56 warning patchset.

Qu Wenruo (8):
btrfs-progs: remove the unnecessary BTRFS_SUPER_INFO_OFFSET path for
tree block read
btrfs-progs: extract metadata restore read code into its own helper
btrfs-progs: don't use write_extent_to_disk() directly
btrfs-progs: use write_data_to_disk() to replace
write_extent_to_disk()
btrfs-progs: use read_data_from_disk() to replace
read_extent_from_disk() and replace read_extent_data()
btrfs-progs: remove extent_buffer::fd and extent_buffer::dev_bytes
btrfs-progs: allow read_data_from_disk() to rebuild RAID56 using P/Q
btrfs-progs: tests/fsck: add test case for data csum check on raid5

 btrfs-corrupt-block.c                         |  38 +---
 btrfs-map-logical.c                           |   5 +-
 btrfstune.c                                   |   3 +-
 check/main.c                                  |   2 +-
 check/mode-common.c                           |   4 +-
 cmds/restore.c                                |   4 +-
 image/main.c                                  |   2 +-
 kernel-shared/ctree.c                         |   5 +-
 kernel-shared/disk-io.c                       | 140 +++++--------
 kernel-shared/disk-io.h                       |   2 -
 kernel-shared/extent_io.c                     | 188 ++++++++++++------
 kernel-shared/extent_io.h                     |   9 +-
 kernel-shared/file.c                          |  20 +-
 kernel-shared/free-space-cache.c              |  20 +-
 kernel-shared/volumes.c                       |  39 ++--
 kernel-shared/volumes.h                       |   1 +
 .../056-raid56-false-alerts/test.sh           |  31 +++
 17 files changed, 277 insertions(+), 236 deletions(-)
 create mode 100755 tests/fsck-tests/056-raid56-false-alerts/test.sh
Signature · 1 line
2.35.1
author
#1 lore raw

PATCH 1/8 btrfs-progs: remove the unnecessary BTRFS_SUPER_INFO_OFFSET path for tree block read

We used to use read_whole_eb() to read super block, but it's no longer the case (so long that I can not even find out which commit did the conversion).

Thus there is no need for read_whole_eb() to handle super block read anymore.

---
 kernel-shared/disk-io.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
kernel-shared/disk-io.c +1 -2
index 2a9b2681a043..f2492547f77d 100644
@@ -302,8 +302,7 @@ int read_whole_eb(struct btrfs_fs_info *info, struct extent_buffer *eb, int mirr
 		read_len = bytes_left;
 		device = NULL;
 
-		if (!info->on_restoring &&
-		    eb->start != BTRFS_SUPER_INFO_OFFSET) {
+		if (!info->on_restoring) {
 			ret = btrfs_map_block(info, READ, eb->start + offset,
 					      &read_len, &multi, mirror, NULL);
 			if (ret) {
Signature · 1 line
2.35.1
author
#8 lore raw

PATCH 8/8 btrfs-progs: tests/fsck: add test case for data csum check on raid5

In reply to Qu Wenruo

Previously 'btrfs check --check-data-csum' will report tons of false alerts for RAID56.

Add a test case to make sure with the new RAID56 rebuild ability, there should be no false alerts.

---
 .../056-raid56-false-alerts/test.sh           | 31 +++++++++++++++++++
 1 file changed, 31 insertions(+)
 create mode 100755 tests/fsck-tests/056-raid56-false-alerts/test.sh
tests/fsck-tests/056-raid56-false-alerts/test.sh new file +31 -0
new file mode 100755
index 000000000000..b82e999c7740
@@ -0,0 +1,31 @@
+#!/bin/bash
+#
+# Make sure "btrfs check --check-data-csum" won't report false alerts on RAID56
+# data.
+#
+
+source "$TEST_TOP/common"
+
+check_prereq btrfs
+check_prereq mkfs.btrfs
+check_global_prereq losetup
+
+setup_loopdevs 3
+prepare_loopdevs
+dev1=${loopdevs[1]}
+TEST_DEV=$dev1
+
+setup_root_helper
+
+run_check $SUDO_HELPERS "$TOP/mkfs.btrfs" -f -m raid1 -d raid5 ${loopdevs[@]}
+run_check_mount_test_dev
+
+run_check $SUDO_HELPER dd if=/dev/urandom of="$TEST_MNT/file" bs=16K count=1 \
+	status=noxfer > /dev/null 2>&1
+
+run_check_umount_test_dev
+
+# Check data csum should not report false alerts
+run_check "$TOP/btrfs" check --check-data-csum "$dev1"
+
+cleanup_loopdevs
Signature · 1 line
2.35.1
author
#2 lore raw

PATCH 2/8 btrfs-progs: extract metadata restore read code into its own helper

In reply to Qu Wenruo

For metadata restore, our logical address is mapped to a single device with logical address 1:1 mapped to device physical address.

Move this part of code into a helper, this will make later extent buffer read path refactoer much easier.

---
 kernel-shared/disk-io.c | 69 +++++++++++++++++++++++++----------------
 1 file changed, 42 insertions(+), 27 deletions(-)
kernel-shared/disk-io.c +42 -27
index f2492547f77d..e6f5d554f13a 100644
@@ -288,6 +288,30 @@ out:
 
 }
 
+static int read_on_restore(struct extent_buffer *eb)
+{
+	struct btrfs_fs_info *fs_info = eb->fs_info;
+	struct btrfs_device *device;
+	int ret;
+
+	/*
+	 * For on_restoring mode, there should be only one device, and logical
+	 * address is mapped 1:1 to device physical offset.
+	 */
+	list_for_each_entry(device, &fs_info->fs_devices->devices, dev_list) {
+		if (device->devid == 1)
+			break;
+	}
+	device->total_ios++;
+
+	ret = btrfs_pread(device->fd, eb->data, eb->len, eb->start,
+			  eb->fs_info->zoned);
+	if (ret != eb->len)
+		ret = -EIO;
+	else
+		ret = 0;
+	return ret;
+}
 
 int read_whole_eb(struct btrfs_fs_info *info, struct extent_buffer *eb, int mirror)
 {
@@ -302,38 +326,29 @@ int read_whole_eb(struct btrfs_fs_info *info, struct extent_buffer *eb, int mirr
 		read_len = bytes_left;
 		device = NULL;
 
-		if (!info->on_restoring) {
-			ret = btrfs_map_block(info, READ, eb->start + offset,
-					      &read_len, &multi, mirror, NULL);
-			if (ret) {
-				printk("Couldn't map the block %llu\n", eb->start + offset);
-				kfree(multi);
-				return -EIO;
-			}
-			device = multi->stripes[0].dev;
-
-			if (device->fd <= 0) {
-				kfree(multi);
-				return -EIO;
-			}
+		if (info->on_restoring)
+			return read_on_restore(eb);
 
-			eb->fd = device->fd;
-			device->total_ios++;
-			eb->dev_bytenr = multi->stripes[0].physical;
+		ret = btrfs_map_block(info, READ, eb->start + offset,
+				      &read_len, &multi, mirror, NULL);
+		if (ret) {
+			printk("Couldn't map the block %llu\n", eb->start + offset);
 			kfree(multi);
-			multi = NULL;
-		} else {
-			/* special case for restore metadump */
-			list_for_each_entry(device, &info->fs_devices->devices, dev_list) {
-				if (device->devid == 1)
-					break;
-			}
+			return -EIO;
+		}
+		device = multi->stripes[0].dev;
 
-			eb->fd = device->fd;
-			eb->dev_bytenr = eb->start;
-			device->total_ios++;
+		if (device->fd <= 0) {
+			kfree(multi);
+			return -EIO;
 		}
 
+		eb->fd = device->fd;
+		device->total_ios++;
+		eb->dev_bytenr = multi->stripes[0].physical;
+		kfree(multi);
+		multi = NULL;
+
 		if (read_len > bytes_left)
 			read_len = bytes_left;
 
Signature · 1 line
2.35.1
author
#4 lore raw

PATCH 4/8 btrfs-progs: use write_data_to_disk() to replace write_extent_to_disk()

In reply to Qu Wenruo

Function write_extent_to_disk() is just writing the content of a tree block to disk.

It can not handle RAID56, and its work is the same as write_data_to_disk().

Thus we can replace write_extent_to_disk() with write_data_to_disk() easily.

There is only one special call site in write_raid56_with_parity(), which can easily be replace with btrfs_pwrite() directly.

This reduce the write entrance, and make later eb::fd removal easier.

---
 kernel-shared/disk-io.c   | 29 +++++++++++++++--------------
 kernel-shared/extent_io.c | 17 +----------------
 kernel-shared/extent_io.h |  1 -
 kernel-shared/volumes.c   |  4 +++-
 4 files changed, 19 insertions(+), 32 deletions(-)
kernel-shared/disk-io.c +15 -14
index e6f5d554f13a..f68284264319 100644
@@ -510,12 +510,12 @@ err:
 int write_and_map_eb(struct btrfs_fs_info *fs_info, struct extent_buffer *eb)
 {
 	int ret;
-	int dev_nr;
+	int mirror_num;
+	int max_mirror;
 	u64 length;
 	u64 *raid_map = NULL;
 	struct btrfs_multi_bio *multi = NULL;
 
-	dev_nr = 0;
 	length = eb->len;
 	ret = btrfs_map_block(fs_info, WRITE, eb->start, &length,
 			      &multi, 0, &raid_map);
@@ -526,6 +526,7 @@ int write_and_map_eb(struct btrfs_fs_info *fs_info, struct extent_buffer *eb)
 		goto out;
 	}
 
+	/* RAID56 write back need RMW */
 	if (raid_map) {
 		ret = write_raid56_with_parity(fs_info, eb, multi,
 					       length, raid_map);
@@ -534,28 +535,28 @@ int write_and_map_eb(struct btrfs_fs_info *fs_info, struct extent_buffer *eb)
 			error(
 		"failed to write raid56 stripe for bytenr %llu length %llu: %m",
 				eb->start, length);
-			goto out;
 		}
-	} else while (dev_nr < multi->num_stripes) {
-		eb->fd = multi->stripes[dev_nr].dev->fd;
-		eb->dev_bytenr = multi->stripes[dev_nr].physical;
-		multi->stripes[dev_nr].dev->total_ios++;
-		dev_nr++;
-		ret = write_extent_to_disk(eb);
+		goto out;
+	}
+
+	/* For non-RAID56, we just writeback data to each mirror */
+	max_mirror = btrfs_num_copies(fs_info, eb->start, eb->len);
+	for (mirror_num = 1; mirror_num <= max_mirror; mirror_num++) {
+		ret = write_data_to_disk(fs_info, eb->data, eb->start, eb->len,
+				         mirror_num);
 		if (ret < 0) {
 			errno = -ret;
 			error(
-"failed to write bytenr %llu length %u devid %llu dev_bytenr %llu: %m",
-				eb->start, eb->len,
-				multi->stripes[dev_nr].dev->devid,
-				eb->dev_bytenr);
+		"failed to write bytenr %llu length %u to mirror %d: %m",
+				eb->start, eb->len, mirror_num);
 			goto out;
 		}
 	}
+
 out:
 	kfree(raid_map);
 	kfree(multi);
-	return 0;
+	return ret;
 }
 
 int write_tree_block(struct btrfs_trans_handle *trans,
kernel-shared/extent_io.c +1 -16
index fba0b050a7e1..ccc8b98107b4 100644
@@ -808,22 +808,6 @@ out:
 	return ret;
 }
 
-int write_extent_to_disk(struct extent_buffer *eb)
-{
-	int ret;
-	ret = btrfs_pwrite(eb->fd, eb->data, eb->len, eb->dev_bytenr,
-			   eb->fs_info->zoned);
-	if (ret < 0)
-		goto out;
-	if (ret != eb->len) {
-		ret = -EIO;
-		goto out;
-	}
-	ret = 0;
-out:
-	return ret;
-}
-
 int read_data_from_disk(struct btrfs_fs_info *info, void *buf, u64 offset,
 			u64 bytes, int mirror)
 {
@@ -934,6 +918,7 @@ int write_data_to_disk(struct btrfs_fs_info *info, void *buf, u64 offset,
 			dev_bytenr = multi->stripes[dev_nr].physical;
 			this_len = min(this_len, bytes_left);
 			dev_nr++;
+			device->total_ios++;
 
 			ret = btrfs_pwrite(device->fd, buf + total_write,
 					   this_len, dev_bytenr, info->zoned);
kernel-shared/extent_io.h +0 -1
index a4c21360a9e8..b787c19ef049 100644
@@ -152,7 +152,6 @@ void free_extent_buffer(struct extent_buffer *eb);
 void free_extent_buffer_nocache(struct extent_buffer *eb);
 int read_extent_from_disk(struct extent_buffer *eb,
 			  unsigned long offset, unsigned long len);
-int write_extent_to_disk(struct extent_buffer *eb);
 int memcmp_extent_buffer(const struct extent_buffer *eb, const void *ptrv,
 			 unsigned long start, unsigned long len);
 void read_extent_buffer(const struct extent_buffer *eb, void *dst,
kernel-shared/volumes.c +3 -1
index 4bf63266879b..7d1e7ea00903 100644
@@ -30,6 +30,7 @@
 #include "kernel-shared/volumes.h"
 #include "zoned.h"
 #include "common/utils.h"
+#include "common/device-utils.h"
 #include "kernel-lib/raid56.h"
 
 const struct btrfs_raid_attr btrfs_raid_array[BTRFS_NR_RAID_TYPES] = {
@@ -2719,7 +2720,8 @@ int write_raid56_with_parity(struct btrfs_fs_info *info,
 	}
 
 	for (i = 0; i < multi->num_stripes; i++) {
-		ret = write_extent_to_disk(ebs[i]);
+		ret = btrfs_pwrite(ebs[i]->fd, ebs[i]->data, ebs[i]->len,
+				   ebs[i]->dev_bytenr, info->zoned);
 		if (ret < 0)
 			goto out_free_split;
 	}
Signature · 1 line
2.35.1
author
#6 lore raw

PATCH 6/8 btrfs-progs: remove extent_buffer::fd and extent_buffer::dev_bytes

In reply to Qu Wenruo

Those two members are a shortcut for non-RAID56 profiles.

But we should not use such shortcut, and move all our logical address read/write to the unified read_data_from_disk()/write_data_to_disk().

With previous refactors, now we're safe to remove them.

---
 btrfs-corrupt-block.c     |  5 ++---
 kernel-shared/ctree.c     |  5 ++---
 kernel-shared/extent_io.c |  2 --
 kernel-shared/extent_io.h |  2 --
 kernel-shared/volumes.c   | 12 +++---------
 5 files changed, 7 insertions(+), 19 deletions(-)
btrfs-corrupt-block.c +2 -3
index 92d608e6b9c0..b17e0bf8e1c9 100644
@@ -55,7 +55,7 @@ static int debug_corrupt_block(struct extent_buffer *eb,
 			if (ret < 0) {
 				errno = -ret;
 				error("cannot read eb bytenr %llu: %m",
-					eb->dev_bytenr);
+					eb->start);
 				return ret;
 			}
 			printf("corrupting %llu copy %d\n", eb->start,
@@ -65,10 +65,9 @@ static int debug_corrupt_block(struct extent_buffer *eb,
 			if (ret < 0) {
 				errno = -ret;
 				error("cannot write eb bytenr %llu: %m",
-					eb->dev_bytenr);
+					eb->start);
 				return ret;
 			}
-			fsync(eb->fd);
 		}
 
 		num_copies = btrfs_num_copies(root->fs_info, eb->start,
kernel-shared/ctree.c +2 -3
index 42bbd50d86a1..ecc37a42eecc 100644
@@ -590,10 +590,9 @@ static void generic_err(const struct extent_buffer *buf, int slot,
 {
 	va_list args;
 
-	fprintf(stderr, "corrupt %s: root=%lld block=%llu physical=%llu slot=%d, ",
+	fprintf(stderr, "corrupt %s: root=%lld block=%llu slot=%d, ",
 		btrfs_header_level(buf) == 0 ? "leaf": "node",
-		btrfs_header_owner(buf), btrfs_header_bytenr(buf),
-		buf->dev_bytenr, slot);
+		btrfs_header_owner(buf), btrfs_header_bytenr(buf), slot);
 	va_start(args, fmt);
 	vfprintf(stderr, fmt, args);
 	va_end(args);
kernel-shared/extent_io.c +0 -2
index c8bb30f62c2d..b8ded5cf7373 100644
@@ -616,8 +616,6 @@ static struct extent_buffer *__alloc_extent_buffer(struct btrfs_fs_info *info,
 	eb->len = blocksize;
 	eb->refs = 1;
 	eb->flags = 0;
-	eb->fd = -1;
-	eb->dev_bytenr = (u64)-1;
 	eb->cache_node.start = bytenr;
 	eb->cache_node.size = blocksize;
 	eb->fs_info = info;
kernel-shared/extent_io.h +0 -2
index cd5e893b165a..aa4f34e187ba 100644
@@ -88,13 +88,11 @@ struct extent_state {
 struct extent_buffer {
 	struct cache_extent cache_node;
 	u64 start;
-	u64 dev_bytenr;
 	struct list_head lru;
 	struct list_head recow;
 	u32 len;
 	int refs;
 	u32 flags;
-	int fd;
 	struct btrfs_fs_info *fs_info;
 	char data[] __attribute__((aligned(8)));
 };
kernel-shared/volumes.c +3 -9
index 7d1e7ea00903..cb49609cc60c 100644
@@ -2620,8 +2620,6 @@ static int split_eb_for_raid56(struct btrfs_fs_info *info,
 		eb->len = stripe_len;
 		eb->refs = 1;
 		eb->flags = 0;
-		eb->fd = -1;
-		eb->dev_bytenr = (u64)-1;
 		eb->fs_info = info;
 
 		this_eb_start = raid_map[i];
@@ -2676,9 +2674,6 @@ int write_raid56_with_parity(struct btrfs_fs_info *info,
 	for (i = 0; i < multi->num_stripes; i++) {
 		struct extent_buffer *new_eb;
 		if (raid_map[i] < BTRFS_RAID5_P_STRIPE) {
-			ebs[i]->dev_bytenr = multi->stripes[i].physical;
-			ebs[i]->fd = multi->stripes[i].dev->fd;
-			multi->stripes[i].dev->total_ios++;
 			if (ebs[i]->start != raid_map[i]) {
 				ret = -EINVAL;
 				goto out_free_split;
@@ -2690,8 +2685,6 @@ int write_raid56_with_parity(struct btrfs_fs_info *info,
 			ret = -ENOMEM;
 			goto out_free_split;
 		}
-		new_eb->dev_bytenr = multi->stripes[i].physical;
-		new_eb->fd = multi->stripes[i].dev->fd;
 		multi->stripes[i].dev->total_ios++;
 		new_eb->len = stripe_len;
 		new_eb->fs_info = info;
@@ -2720,8 +2713,9 @@ int write_raid56_with_parity(struct btrfs_fs_info *info,
 	}
 
 	for (i = 0; i < multi->num_stripes; i++) {
-		ret = btrfs_pwrite(ebs[i]->fd, ebs[i]->data, ebs[i]->len,
-				   ebs[i]->dev_bytenr, info->zoned);
+		multi->stripes[i].dev->total_ios++;
+		ret = btrfs_pwrite(multi->stripes[i].dev->fd, ebs[i]->data, ebs[i]->len,
+				   multi->stripes[i].physical, info->zoned);
 		if (ret < 0)
 			goto out_free_split;
 	}
Signature · 1 line
2.35.1
author
#7 lore raw

PATCH 7/8 btrfs-progs: allow read_data_from_disk() to rebuild RAID56 using P/Q

In reply to Qu Wenruo

This new ability is added by:

- Allow btrfs_map_block() to return the chunk type This makes later work much easier

- Only reset stripe offset inside btrfs_map_block() when needed Currently if @raid_map is not NULL, btrfs_map_block() will consider this call is for WRITE and will reset stripe offset.

This is no longer the case, as for RAID56 read with mirror_num 1/0, we will still call btrfs_map_block() with non-NULL raid_map.

Add a small check to make sure we won't reset stripe offset for mirror 1/0 read.

- Add new helper read_raid56() to handle rebuild
We will read the full stripe (including all data and P/Q stripes) do the rebuild, then only copy the refered part to the caller.

There is a catch for RAID6, we have no way to exhaust all combination, so the current repair will assume the mirror = 0 data is corrupted, then try to find a missing device.

But if no missing device can be found, it will assume P is corrupted. This is just a guess, and can to totally wrong, but we have no better idea.

Now btrfs-progs have full read ability for RAID56.

---
 kernel-shared/extent_io.c | 114 +++++++++++++++++++++++++++++++++++++-
 kernel-shared/volumes.c   |  27 +++++----
 kernel-shared/volumes.h   |   1 +
 3 files changed, 128 insertions(+), 14 deletions(-)
kernel-shared/extent_io.c +112 -2
index b8ded5cf7373..ee92e0f847d6 100644
@@ -26,6 +26,7 @@
 #include "kerncompat.h"
 #include "kernel-shared/extent_io.h"
 #include "kernel-lib/list.h"
+#include "kernel-lib/raid56.h"
 #include "kernel-shared/ctree.h"
 #include "kernel-shared/volumes.h"
 #include "kernel-shared/disk-io.h"
@@ -788,23 +789,131 @@ struct extent_buffer *alloc_dummy_extent_buffer(struct btrfs_fs_info *fs_info,
 	return ret;
 }
 
+static int read_raid56(struct btrfs_fs_info *fs_info, void *buf, u64 logical,
+		       u64 len, int mirror, struct btrfs_multi_bio *multi,
+		       u64 *raid_map)
+{
+	const int num_stripes = multi->num_stripes;
+	const u64 full_stripe_start = raid_map[0];
+	void **pointers = NULL;
+	int failed_a = -1;
+	int failed_b = -1;
+	int i;
+	int ret;
+
+	/* Only read repair should go this path */
+	ASSERT(mirror > 1);
+	ASSERT(raid_map);
+
+	/* The read length should be inside one stripe */
+	ASSERT(len <= BTRFS_STRIPE_LEN);
+
+	pointers = calloc(num_stripes, sizeof(void *));
+	if (!pointers) {
+		ret = -ENOMEM;
+		goto out;
+	}
+	/* Allocate memory for the full stripe */
+	for (i = 0; i < num_stripes; i++) {
+		pointers[i] = malloc(BTRFS_STRIPE_LEN);
+		if (!pointers[i]) {
+			ret = -ENOMEM;
+			goto out;
+		}
+	}
+
+	/*
+	 * Read the full stripe.
+	 *
+	 * The stripes in @multi is not rotated, thus can be used to read from
+	 * disk directly.
+	 */
+	for (i = 0; i < num_stripes; i++) {
+		ret = btrfs_pread(multi->stripes[i].dev->fd, pointers[i],
+				  BTRFS_STRIPE_LEN, multi->stripes[i].physical,
+				  fs_info->zoned);
+		if (ret < BTRFS_STRIPE_LEN) {
+			ret = -EIO;
+			goto out;
+		}
+	}
+
+	/*
+	 * Get the failed index.
+	 *
+	 * Since we're reading using mirror_num > 1 already, it means the data
+	 * stripe where @logical lies in is definitely corrupted.
+	 */
+	failed_a = (logical - full_stripe_start) / BTRFS_STRIPE_LEN;
+
+	/*
+	 * For RAID6, we don't have good way to exhaust all the combinations,
+	 * so here we can only go through the map to see if we have missing devices.
+	 */
+	if (multi->type & BTRFS_BLOCK_GROUP_RAID6) {
+		for (i = 0; i < num_stripes; i++) {
+			/* Skip failed_a, as it's already marked failed */
+			if (i == failed_a)
+				continue;
+			/* Missing dev */
+			if (multi->stripes[i].dev->fd == -1) {
+				failed_b = i;
+				break;
+			}
+		}
+		/*
+		 * No missing device, we have no better idea, default to P
+		 * corruption
+		 */
+		if (failed_b < 0)
+			failed_b = num_stripes - 2;
+	}
+
+	/* Rebuild the full stripe */
+	ret = raid56_recov(num_stripes, BTRFS_STRIPE_LEN, multi->type,
+			   failed_a, failed_b, pointers);
+	ASSERT(ret == 0);
+
+	/* Now copy the data back to original buf */
+	memcpy(buf, pointers[failed_a] + (logical - full_stripe_start) %
+			BTRFS_STRIPE_LEN, len);
+	ret = 0;
+out:
+	for (i = 0; i < num_stripes; i++)
+		free(pointers[i]);
+	free(pointers);
+	return ret;
+}
+
 int read_data_from_disk(struct btrfs_fs_info *info, void *buf, u64 logical,
 			u64 *len, int mirror)
 {
 	struct btrfs_multi_bio *multi = NULL;
 	struct btrfs_device *device;
 	u64 read_len = *len;
+	u64 *raid_map = NULL;
 	int ret;
 
 	ret = btrfs_map_block(info, READ, logical, &read_len, &multi, mirror,
-			      NULL);
+			      &raid_map);
 	if (ret) {
 		fprintf(stderr, "Couldn't map the block %llu\n", logical);
 		return -EIO;
 	}
+	read_len = min(*len, read_len);
+
+	/* We need to rebuild from P/Q */
+	if (mirror > 1 && multi->type & BTRFS_BLOCK_GROUP_RAID56_MASK) {
+		ret = read_raid56(info, buf, logical, read_len, mirror, multi,
+				  raid_map);
+		free(multi);
+		free(raid_map);
+		*len = read_len;
+		return ret;
+	}
+	free(raid_map);
 	device = multi->stripes[0].dev;
 
-	read_len = min(*len, read_len);
 	if (device->fd <= 0) {
 		kfree(multi);
 		return -EIO;
@@ -824,6 +933,7 @@ int read_data_from_disk(struct btrfs_fs_info *info, void *buf, u64 logical,
 			logical, ret, read_len);
 		return -EIO;
 	}
+	*len = read_len;
 
 	return 0;
 }
kernel-shared/volumes.c +15 -12
index cb49609cc60c..f082fa9f898e 100644
@@ -1805,6 +1805,7 @@ int __btrfs_map_block(struct btrfs_fs_info *fs_info, int rw,
 	int stripes_required = 1;
 	int stripe_index;
 	int i;
+	bool need_raid_map = false;
 	struct btrfs_multi_bio *multi = NULL;
 
 	if (multi_ret && rw == READ) {
@@ -1842,17 +1843,18 @@ again:
 	}
 	if (map->type & BTRFS_BLOCK_GROUP_RAID56_MASK
 	    && multi_ret && ((rw & WRITE) || mirror_num > 1) && raid_map_ret) {
-		    /* RAID[56] write or recovery. Return all stripes */
-		    stripes_required = map->num_stripes;
-
-		    /* Only allocate the map if we've already got a large enough multi_ret */
-		    if (stripes_allocated >= stripes_required) {
-			    raid_map = kmalloc(sizeof(u64) * map->num_stripes, GFP_NOFS);
-			    if (!raid_map) {
-				    kfree(multi);
-				    return -ENOMEM;
-			    }
-		    }
+		need_raid_map = true;
+		/* RAID[56] write or recovery. Return all stripes */
+		stripes_required = map->num_stripes;
+
+		/* Only allocate the map if we've already got a large enough multi_ret */
+		if (stripes_allocated >= stripes_required) {
+			raid_map = kmalloc(sizeof(u64) * map->num_stripes, GFP_NOFS);
+			if (!raid_map) {
+				kfree(multi);
+				return -ENOMEM;
+			}
+		}
 	}
 
 	/* if our multi bio struct is too small, back off and try again */
@@ -1890,6 +1892,7 @@ again:
 		goto out;
 
 	multi->num_stripes = 1;
+	multi->type = map->type;
 	stripe_index = 0;
 	if (map->type & BTRFS_BLOCK_GROUP_RAID1_MASK) {
 		if (rw == WRITE)
@@ -1916,7 +1919,7 @@ again:
 		else if (mirror_num)
 			stripe_index = mirror_num - 1;
 	} else if (map->type & BTRFS_BLOCK_GROUP_RAID56_MASK) {
-		if (raid_map) {
+		if (need_raid_map && raid_map) {
 			int rot;
 			u64 tmp;
 			u64 raid56_full_stripe_start;
kernel-shared/volumes.h +1 -0
index 5cfe7e39f6b8..d90065b98a3e 100644
@@ -106,6 +106,7 @@ struct btrfs_bio_stripe {
 };
 
 struct btrfs_multi_bio {
+	u64 type;
 	int error;
 	int num_stripes;
 	struct btrfs_bio_stripe stripes[];
Signature · 1 line
2.35.1
author
#3 lore raw

PATCH 3/8 btrfs-progs: don't use write_extent_to_disk() directly

In reply to Qu Wenruo

There are two call sites using write_extent_to_disk() directly:

- debug_corrupt_block() in btrfs-corrupt-block.c
- corrupt_keys() in btrfs-corrupt-block.c

The problem of write_extent_to_disk() is, it can only handle plain profiles (All profiles except P/Q stripes of RAID56).

Calling it directly can corrupted RAID56 P/Q, and in the future we're going to remove eb::fd/eb::dev_bytes, so remove such call sites with write_and_map_eb().

---
 btrfs-corrupt-block.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
btrfs-corrupt-block.c +2 -2
index 8162c0dfd554..8ec6d63f060c 100644
@@ -76,7 +76,7 @@ static int debug_corrupt_block(struct extent_buffer *eb,
 			printf("corrupting %llu copy %d\n", eb->start,
 			       mirror_num);
 			memset(eb->data, 0, eb->len);
-			ret = write_extent_to_disk(eb);
+			ret = write_and_map_eb(eb->fs_info, eb);
 			if (ret < 0) {
 				errno = -ret;
 				error("cannot write eb bytenr %llu: %m",
@@ -159,7 +159,7 @@ static void corrupt_keys(struct btrfs_trans_handle *trans,
 		u16 csum_type = fs_info->csum_type;
 
 		csum_tree_block_size(eb, csum_size, 0, csum_type);
-		write_extent_to_disk(eb);
+		write_and_map_eb(eb->fs_info, eb);
 	}
 }
 
Signature · 1 line
2.35.1
author
#5 lore raw

PATCH 5/8 btrfs-progs: use read_data_from_disk() to replace read_extent_from_disk() and replace read_extent_data()

In reply to Qu Wenruo

The function read_extent_from_disk() is only a wrapper to read tree block.

And read_extent_data() is just a while loop to eliminate short read caused by stripe boundary.

In fact, a lot of call sites of read_extent_data() are either reading metadata (thus no possible short read) or doing extra loop by themselves.

This patch will replace those two functions with read_data_from_disk(), making it the only entrance for data/metadata read.
And update read_data_from_disk() to return the read bytes, so caller can do a simple while loop.

For the few callers of read_extent_data(), open-code a small while loop for them.

This will allow later RAID56 read repair using P/Q much easier.

---
 btrfs-corrupt-block.c            | 29 +++--------
 btrfs-map-logical.c              |  5 +-
 btrfstune.c                      |  3 +-
 check/main.c                     |  2 +-
 check/mode-common.c              |  4 +-
 cmds/restore.c                   |  4 +-
 image/main.c                     |  2 +-
 kernel-shared/disk-io.c          | 73 +++-------------------------
 kernel-shared/disk-io.h          |  2 -
 kernel-shared/extent_io.c        | 83 +++++++++++---------------------
 kernel-shared/extent_io.h        |  6 +--
 kernel-shared/file.c             | 20 ++++----
 kernel-shared/free-space-cache.c | 20 ++++++--
 13 files changed, 80 insertions(+), 173 deletions(-)
btrfs-corrupt-block.c +7 -22
index 8ec6d63f060c..92d608e6b9c0 100644
@@ -40,33 +40,18 @@ static int debug_corrupt_block(struct extent_buffer *eb,
 		struct btrfs_root *root, u64 bytenr, u32 blocksize, u64 copy)
 {
 	int ret;
-	u64 length;
-	struct btrfs_multi_bio *multi = NULL;
-	struct btrfs_device *device;
 	int num_copies;
 	int mirror_num = 1;
 
-	length = blocksize;
 	while (1) {
-		ret = btrfs_map_block(root->fs_info, READ, eb->start, &length,
-				      &multi, mirror_num, NULL);
-		if (ret) {
-			error("cannot map block %llu length %llu mirror %d: %d",
-				eb->start, length, mirror_num, ret);
-			return ret;
-		}
-		device = multi->stripes[0].dev;
-		eb->fd = device->fd;
-		device->total_ios++;
-		eb->dev_bytenr = multi->stripes[0].physical;
-
-		fprintf(stdout,
-			"mirror %d logical %llu physical %llu device %s\n",
-			mirror_num, bytenr, eb->dev_bytenr, device->name);
-		free(multi);
-
 		if (!copy || mirror_num == copy) {
-			ret = read_extent_from_disk(eb, 0, eb->len);
+			u64 read_len = eb->len;
+
+			ret = read_data_from_disk(eb->fs_info, eb->data,
+						  eb->start, &read_len,
+						  mirror_num);
+			if (read_len < eb->len)
+				ret = -EIO;
 			if (ret < 0) {
 				errno = -ret;
 				error("cannot read eb bytenr %llu: %m",
btrfs-map-logical.c +3 -2
index b3a7526b22a4..860c196d6d9b 100644
@@ -173,8 +173,9 @@ static int write_extent_content(struct btrfs_fs_info *fs_info, int out_fd,
 
 	while (cur_offset < length) {
 		cur_len = min_t(u64, length - cur_offset, BUFFER_SIZE);
-		ret = read_extent_data(fs_info, buffer,
-				       logical + cur_offset, &cur_len, mirror);
+		ret = read_data_from_disk(fs_info, buffer,
+					  logical + cur_offset, &cur_len,
+					  mirror);
 		if (ret < 0) {
 			errno = -ret;
 			fprintf(stderr,
btrfstune.c +2 -1
index 33c83bf16291..c9a92349a44c 100644
@@ -333,7 +333,8 @@ static int populate_csum(struct btrfs_trans_handle *trans,
 
 	while (offset < len) {
 		sectorsize = fs_info->sectorsize;
-		ret = read_extent_data(fs_info, buf, start + offset, &sectorsize, 0);
+		ret = read_data_from_disk(fs_info, buf, start + offset,
+					  &sectorsize, 0);
 		if (ret)
 			break;
 		ret = btrfs_csum_file_block(trans, start + len, start + offset,
check/main.c +1 -1
index 954d02413f82..b23e6b2e7b2b 100644
@@ -5757,7 +5757,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
 		for (mirror = 1; mirror <= num_copies; mirror++) {
 			read_len = num_bytes - offset;
 			/* read as much space once a time */
-			ret = read_extent_data(gfs_info, (char *)data + offset,
+			ret = read_data_from_disk(gfs_info, (char *)data + offset,
 					bytenr + offset, &read_len, mirror);
 			if (ret)
 				goto out;
check/mode-common.c +2 -2
index a906a5852a46..facc672ccd21 100644
@@ -1202,8 +1202,8 @@ static int populate_csum(struct btrfs_trans_handle *trans,
 
 	while (offset < len) {
 		sectorsize = gfs_info->sectorsize;
-		ret = read_extent_data(gfs_info, buf, start + offset,
-				       &sectorsize, 0);
+		ret = read_data_from_disk(gfs_info, buf, start + offset,
+					  &sectorsize, 0);
 		if (ret)
 			break;
 		ret = btrfs_csum_file_block(trans, start + len, start + offset,
cmds/restore.c +2 -2
index 81ca6cd57cb5..5923d571c986 100644
@@ -407,8 +407,8 @@ again:
 	cur = bytenr;
 	while (cur < bytenr + size_left) {
 		length = bytenr + size_left - cur;
-		ret = read_extent_data(root->fs_info, inbuf + cur - bytenr, cur,
-				       &length, mirror_num);
+		ret = read_data_from_disk(root->fs_info, inbuf + cur - bytenr, cur,
+					  &length, mirror_num);
 		if (ret < 0) {
 			mirror_num++;
 			if (mirror_num > num_copies) {
image/main.c +1 -1
index 75f9203fab12..88ed54f5b129 100644
@@ -615,7 +615,7 @@ static int read_data_extent(struct metadump_struct *md,
 	for (cur_mirror = 1; cur_mirror <= num_copies; cur_mirror++) {
 		while (bytes_left) {
 			read_len = bytes_left;
-			ret = read_extent_data(fs_info,
+			ret = read_data_from_disk(fs_info,
 					(char *)(async->buffer + offset),
 					logical, &read_len, cur_mirror);
 			if (ret < 0)
kernel-shared/disk-io.c +6 -67

93 lines not on this page · raw message

kernel-shared/disk-io.h +0 -2
index e07141a9596d..bba97fc1a814 100644
@@ -141,8 +141,6 @@ int read_whole_eb(struct btrfs_fs_info *info, struct extent_buffer *eb, int mirr
 struct extent_buffer* read_tree_block(struct btrfs_fs_info *fs_info, u64 bytenr,
 		u64 parent_transid);
 
-int read_extent_data(struct btrfs_fs_info *fs_info, char *data, u64 logical,
-		     u64 *len, int mirror);
 void readahead_tree_block(struct btrfs_fs_info *fs_info, u64 bytenr,
 			  u64 parent_transid);
 struct extent_buffer* btrfs_find_create_tree_block(
kernel-shared/extent_io.c +28 -55

103 lines not on this page · raw message

kernel-shared/extent_io.h +2 -4
index b787c19ef049..cd5e893b165a 100644
@@ -150,8 +150,6 @@ struct extent_buffer *alloc_dummy_extent_buffer(struct btrfs_fs_info *fs_info,
 						u64 bytenr, u32 blocksize);
 void free_extent_buffer(struct extent_buffer *eb);
 void free_extent_buffer_nocache(struct extent_buffer *eb);
-int read_extent_from_disk(struct extent_buffer *eb,
-			  unsigned long offset, unsigned long len);
 int memcmp_extent_buffer(const struct extent_buffer *eb, const void *ptrv,
 			 unsigned long start, unsigned long len);
 void read_extent_buffer(const struct extent_buffer *eb, void *dst,
@@ -169,8 +167,8 @@ int extent_buffer_test_bit(struct extent_buffer *eb, unsigned long start,
 			   unsigned long nr);
 int set_extent_buffer_dirty(struct extent_buffer *eb);
 int clear_extent_buffer_dirty(struct extent_buffer *eb);
-int read_data_from_disk(struct btrfs_fs_info *info, void *buf, u64 offset,
-			u64 bytes, int mirror);
+int read_data_from_disk(struct btrfs_fs_info *info, void *buf, u64 logical,
+			u64 *len, int mirror);
 int write_data_to_disk(struct btrfs_fs_info *info, void *buf, u64 offset,
 		       u64 bytes, int mirror);
 void extent_buffer_bitmap_clear(struct extent_buffer *eb, unsigned long start,
kernel-shared/file.c +11 -9
index a31728102afa..59d82a1dd5f7 100644
@@ -225,11 +225,11 @@ int btrfs_read_file(struct btrfs_root *root, u64 ino, u64 start, int len,
 	memset(dest, 0, len);
 	while (1) {
 		struct btrfs_file_extent_item *fi;
+		u64 offset = 0;
 		u64 extent_start;
 		u64 extent_len;
 		u64 read_start;
 		u64 read_len;
-		u64 read_len_ret;
 		u64 disk_bytenr;
 
 		leaf = path.nodes[0];
@@ -282,14 +282,16 @@ int btrfs_read_file(struct btrfs_root *root, u64 ino, u64 start, int len,
 
 		disk_bytenr = btrfs_file_extent_disk_bytenr(leaf, fi) +
 			      btrfs_file_extent_offset(leaf, fi);
-		read_len_ret = read_len;
-		ret = read_extent_data(fs_info, dest + read_start - start, disk_bytenr,
-				       &read_len_ret, 0);
-		if (ret < 0)
-			break;
-		/* Short read, something went wrong */
-		if (read_len_ret != read_len)
-			return -EIO;
+		while (offset < read_len) {
+			u64 read_len_ret = read_len - offset;
+
+			ret = read_data_from_disk(fs_info,
+					dest + read_start - start + offset,
+					disk_bytenr + offset, &read_len_ret, 0);
+			if (ret < 0)
+				goto out;
+			offset += read_len_ret;
+		}
 		read += read_len;
 next:
 		ret = btrfs_next_item(root, &path);
kernel-shared/free-space-cache.c +15 -5
index 6d7fee9c86fb..50eeb4381ed4 100644
@@ -118,6 +118,8 @@ static int io_ctl_prepare_pages(struct io_ctl *io_ctl, struct btrfs_root *root,
 	}
 
 	while (total_read < io_ctl->total_size) {
+		u64 offset = 0;
+
 		if (path->slots[0] >= btrfs_header_nritems(path->nodes[0])) {
 			ret = btrfs_next_leaf(root, path);
 			if (ret) {
@@ -150,11 +152,19 @@ static int io_ctl_prepare_pages(struct io_ctl *io_ctl, struct btrfs_root *root,
 		bytenr = btrfs_file_extent_disk_bytenr(leaf, fi) +
 			btrfs_file_extent_offset(leaf, fi);
 		len = btrfs_file_extent_num_bytes(leaf, fi);
-		ret = read_data_from_disk(root->fs_info,
-					  io_ctl->buffer + key.offset, bytenr,
-					  len, 0);
-		if (ret)
-			break;
+		while (offset < len) {
+			u64 read_len = len - offset;
+
+			ret = read_data_from_disk(root->fs_info,
+					  io_ctl->buffer + key.offset + offset,
+					  bytenr + offset,
+					  &read_len, 0);
+			if (ret < 0) {
+				btrfs_release_path(path);
+				return ret;
+			}
+			offset += read_len;
+		}
 		total_read += len;
 		path->slots[0]++;
 	}
Signature · 1 line
2.35.1
#9 lore raw
In reply to Qu Wenruo
Quoted 5 lines · Qu Wenruo

This branch can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/raid56_rebuild

Please note that, the last time I check devel branch, the RAID56 warning patch is not yet merged.

It was there but I had not the branch pushed.

Quoted 4 lines

So this is based on the latest devel branch from github.

And since we're adding proper RAID56 repair ability, there is no need
for the patchset "btrfs-progs: check: avoid false alerts for --check-data-csum on RAID56"

OK so I'll replace it with this patchset.

#10 lore raw
In reply to Qu Wenruo
Quoted 2 lines · Qu Wenruo

This branch can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/raid56_rebuild

The mkfs tests fail in 001 with the following last command:

====== RUN CHECK root_helper .../mkfs.btrfs -f -d raid5 -m raid5 /dev/loop0 /dev/loop1 /dev/loop2 /dev/loop3
WARNING: RAID5/6 support has known problems is strongly discouraged
to be used besides testing or evaluation.

kernel-shared/transaction.c:155: __commit_transaction: BUG_ON ret triggered, value 65536 .../mkfs.btrfs(__commit_transaction+0x197)[0x43a107] /...mkfs.btrfs(btrfs_commit_transaction+0x13b)[0x43a26b] /...mkfs.btrfs(main+0x174b)[0x40f68d]
/lib64/libc.so.6(+0x2d540)[0x7f969b5ea540] /lib64/libc.so.6(__libc_start_main+0x7c)[0x7f969b5ea5ec] /...mkfs.btrfs(_start+0x25)[0x40d965]
/...tests/common: line 540: 29172 Aborted sudo -n "$@"
failed: root_helper .../mkfs.btrfs -f -d raid5 -m raid5 /dev/loop0 /dev/loop1 /dev/loop2 /dev/loop3 test failed for case 001-basic-profiles

#11 lore raw
Quoted 10 lines · David Sterba
Quoted 2 lines · Qu Wenruo

This branch can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/raid56_rebuild

The mkfs tests fail in 001 with the following last command:

====== RUN CHECK root_helper .../mkfs.btrfs -f -d raid5 -m raid5 /dev/loop0 /dev/loop1 /dev/loop2 /dev/loop3
WARNING: RAID5/6 support has known problems is strongly discouraged
to be used besides testing or evaluation.

Quoted 10 lines

kernel-shared/transaction.c:155: __commit_transaction: BUG_ON ret triggered, value 65536 .../mkfs.btrfs(__commit_transaction+0x197)[0x43a107] /...mkfs.btrfs(btrfs_commit_transaction+0x13b)[0x43a26b] /...mkfs.btrfs(main+0x174b)[0x40f68d]
/lib64/libc.so.6(+0x2d540)[0x7f969b5ea540] /lib64/libc.so.6(__libc_start_main+0x7c)[0x7f969b5ea5ec] /...mkfs.btrfs(_start+0x25)[0x40d965]
/...tests/common: line 540: 29172 Aborted sudo -n "$@"
failed: root_helper .../mkfs.btrfs -f -d raid5 -m raid5 /dev/loop0 /dev/loop1 /dev/loop2 /dev/loop3 test failed for case 001-basic-profiles

It's caused by patch [PATCH 4/8] btrfs-progs: use write_data_to_disk() to replace write_extent_to_disk()

#12 lore raw
Quoted 22 lines · David Sterba
Quoted 10 lines · David Sterba
Quoted 2 lines · Qu Wenruo

This branch can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/raid56_rebuild

The mkfs tests fail in 001 with the following last command:

====== RUN CHECK root_helper .../mkfs.btrfs -f -d raid5 -m raid5 /dev/loop0 /dev/loop1 /dev/loop2 /dev/loop3
WARNING: RAID5/6 support has known problems is strongly discouraged
to be used besides testing or evaluation.

Quoted 10 lines

kernel-shared/transaction.c:155: __commit_transaction: BUG_ON ret triggered, value 65536 .../mkfs.btrfs(__commit_transaction+0x197)[0x43a107] /...mkfs.btrfs(btrfs_commit_transaction+0x13b)[0x43a26b] /...mkfs.btrfs(main+0x174b)[0x40f68d]
/lib64/libc.so.6(+0x2d540)[0x7f969b5ea540] /lib64/libc.so.6(__libc_start_main+0x7c)[0x7f969b5ea5ec] /...mkfs.btrfs(_start+0x25)[0x40d965]
/...tests/common: line 540: 29172 Aborted sudo -n "$@"
failed: root_helper .../mkfs.btrfs -f -d raid5 -m raid5 /dev/loop0 /dev/loop1 /dev/loop2 /dev/loop3 test failed for case 001-basic-profiles

Quoted 2 lines

It's caused by patch [PATCH 4/8] btrfs-progs: use write_data_to_disk() to replace write_extent_to_disk()

That's a known one, I forgot to change the return value back to 0 for write_extent_to_disk().

Do I need to update the whole series or just send an update based on this?

Thanks,
Qu