Skip to contentSkip to conversation

btrfs-progs: check: add the ability to repair extent item generation corruption

PATCH In review Qu Wenruo 2 participants 6 messages Aug 11, 2020 – Dec 1, 2020 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

v2 cover btrfs-progs: check: add the ability to repair extent item generation corruption

Although we have introduced the check ability to detect bad extent item generation, there is no repair ability.

I thought it would be rare to hit, but real world cases prove I'm a total idiot.

So this patchset will add the ability to repair, for both lowmem mode and original mode, along with enhanced test images.

There is also a bug fix for original mode, which fails to detect such problem if it's a tree block.

Changelog:
v2:
- Fix a type in the subject of the 4th patch
- Fix a bracket for for a logical and and bit and
The old code is fine and bit and has higher priority, but the bracket is intended to make that higher priority more obvious.

Qu Wenruo (4):
btrfs-progs: check/lowmem: add the ability to repair extent item
generation
btrfs-progs: check/original: don't reset extent generation for
check_block()
btrfs-progs: check/original: add the ability to repair extent item
generation
btrfs-progs: tests/fsck: enhance invalid extent item generation test
cases

 check/main.c                                  |  77 ++++++++++++++-
 check/mode-common.c                           |  59 ++++++++++++
 check/mode-common.h                           |   3 +
 check/mode-lowmem.c                           |  89 ++++++++++++++++--
 ...xz => bad_extent_item_gen_for_data.img.xz} | Bin
 .../bad_extent_item_gen_for_tree_block.img.xz | Bin 0 -> 1804 bytes
 .../test.sh                                   |  19 ----
 7 files changed, 216 insertions(+), 31 deletions(-)
 rename tests/fsck-tests/044-invalid-extent-item-generation/{bad_extent_item_gen.img.xz => bad_extent_item_gen_for_data.img.xz} (100%)
 create mode 100644 tests/fsck-tests/044-invalid-extent-item-generation/bad_extent_item_gen_for_tree_block.img.xz
 delete mode 100755 tests/fsck-tests/044-invalid-extent-item-generation/test.sh
Signature · 1 line
2.28.0
author
#1 lore raw

PATCH v2 1/4 btrfs-progs: check/lowmem: add the ability to repair extent item generation

There is an internal report about bad extent item generation triggering tree-checker.

This patch will add the repair ability to btrfs check --mode=lowmem mode, by resetting the generation field of extent item.

Currently the correct generation for tree block is fetched from its header, while for data extent it uses transid as fallback.

---
 check/mode-common.c | 59 ++++++++++++++++++++++++++++++
 check/mode-common.h |  3 ++
 check/mode-lowmem.c | 89 +++++++++++++++++++++++++++++++++++++++++----
 3 files changed, 144 insertions(+), 7 deletions(-)
check/mode-common.c +59 -0
index aa45c5c2b8a3..a024f17126d0 100644
@@ -1195,3 +1195,62 @@ int recow_extent_buffer(struct btrfs_root *root, struct extent_buffer *eb)
 	btrfs_release_path(&path);
 	return ret;
 }
+
+/*
+ * Try to get a correct extent item geneartion.
+ * Return 0 if we get a correct generation.
+ * Return <0 if we failed to get one.
+ */
+int get_extent_item_generation(struct btrfs_fs_info *fs_info,
+				u64 bytenr, u64 *gen_ret)
+{
+	struct btrfs_root *root = fs_info->extent_root;
+	struct btrfs_extent_item *ei;
+	struct btrfs_path path;
+	struct btrfs_key key;
+	int ret;
+
+	key.objectid = bytenr;
+	key.type = BTRFS_METADATA_ITEM_KEY;
+	key.offset = (u64)-1;
+
+	btrfs_init_path(&path);
+	ret = btrfs_search_slot(NULL, root, &key, &path, 0, 0);
+	/* Not possible */
+	if (ret == 0)
+		ret = -EUCLEAN;
+	if (ret < 0)
+		goto out;
+	ret = btrfs_previous_extent_item(root, &path, bytenr);
+	if (ret > 0)
+		ret = -ENOENT;
+	if (ret < 0)
+		goto out;
+
+	ei = btrfs_item_ptr(path.nodes[0], path.slots[0],
+			    struct btrfs_extent_item);
+
+	if (btrfs_extent_flags(path.nodes[0], ei) &
+	    BTRFS_EXTENT_FLAG_TREE_BLOCK) {
+		struct extent_buffer *eb;
+
+		eb = read_tree_block(fs_info, bytenr, 0);
+		if (extent_buffer_uptodate(eb)) {
+			*gen_ret = btrfs_header_generation(eb);
+			ret = 0;
+		} else {
+			ret = -EIO;
+		}
+		free_extent_buffer(eb);
+	} else {
+		/*
+		 * TODO: Grab proper data generation for data extents.
+		 * But this is not an urgent objective, as we can still
+		 * use transaction id as fall back
+		 */
+		ret = -ENOTSUP;
+	}
+out:
+	btrfs_release_path(&path);
+	return ret;
+}
check/mode-common.h +3 -0
index edf9257edaf0..3292ec744490 100644
@@ -173,4 +173,7 @@ static inline u32 btrfs_type_to_imode(u8 type)
 
 	return imode_by_btrfs_type[(type)];
 }
+
+int get_extent_item_generation(struct btrfs_fs_info *fs_info,
+				u64 bytenr, u64 *gen_ret);
 #endif
check/mode-lowmem.c +82 -7

135 lines not on this page · raw message

Signature · 1 line
2.28.0
author
#2 lore raw

PATCH v2 2/4 btrfs-progs: check/original: don't reset extent generation for check_block()

In reply to Qu Wenruo

In check_block(), we unconditionally reset extent_record::generation.

This is in fact pretty correct, but this makes original mode fail to detect bad extent item geneartion.

So change to behavior to set the geneartion if and only if the tree block generation is higher.

---
 check/main.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
check/main.c +2 -1
index f93bd7d4ca70..72fa28ad216a 100644
@@ -4440,7 +4440,8 @@ static int check_block(struct btrfs_root *root,
 	if (!cache)
 		return 1;
 	rec = container_of(cache, struct extent_record, cache);
-	rec->generation = btrfs_header_generation(buf);
+	if (rec->generation < btrfs_header_generation(buf))
+		rec->generation = btrfs_header_generation(buf);
 
 	level = btrfs_header_level(buf);
 	if (btrfs_header_nritems(buf) > 0) {
Signature · 1 line
2.28.0
author
#3 lore raw

PATCH v2 3/4 btrfs-progs: check/original: add the ability to repair extent item generation

In reply to Qu Wenruo

This is pretty much the same as the lowmem mode, it will try to reset the extent item generation using either the tree block generation or current transid.

---
 check/main.c | 74 +++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 70 insertions(+), 4 deletions(-)
check/main.c +70 -4
index 72fa28ad216a..dc366b125ece 100644
@@ -7914,6 +7914,63 @@ static int record_unaligned_extent_rec(struct btrfs_fs_info *fs_info,
 	return ret;
 }
 
+static int repair_extent_item_generation(struct btrfs_fs_info *fs_info,
+					 struct extent_record *rec)
+{
+	struct btrfs_trans_handle *trans;
+	struct btrfs_path path;
+	struct btrfs_key key;
+	struct btrfs_extent_item *ei;
+	struct btrfs_root *extent_root = fs_info->extent_root;
+	u64 new_gen = 0;;
+	int ret;
+
+	key.objectid = rec->start;
+	key.type = BTRFS_METADATA_ITEM_KEY;
+	key.offset = (u64)-1;
+
+	get_extent_item_generation(fs_info, rec->start, &new_gen);
+	trans = btrfs_start_transaction(extent_root, 1);
+	if (IS_ERR(trans)) {
+		ret = PTR_ERR(trans);
+		errno = -ret;
+		error("failed to start transaction: %m");
+		return ret;
+	}
+	btrfs_init_path(&path);
+	ret = btrfs_search_slot(trans, extent_root, &key, &path, 0, 1);
+	/* Not possible */
+	if (ret == 0)
+		ret = -EUCLEAN;
+	if (ret < 0)
+		goto out;
+	ret = btrfs_previous_extent_item(extent_root, &path, rec->start);
+	if (ret > 0)
+		ret = -ENOENT;
+	if (ret < 0)
+		goto out;
+	
+	if (!new_gen)
+		new_gen = trans->transid;
+	ei = btrfs_item_ptr(path.nodes[0], path.slots[0],
+			    struct btrfs_extent_item);
+	btrfs_set_extent_generation(path.nodes[0], ei, new_gen);
+	ret = btrfs_commit_transaction(trans, extent_root);
+	if (ret < 0) {
+		errno = -ret;
+		error("failed to commit transaction: %m");
+		goto out;
+	}
+	printf("Reset extent item (%llu) generation to %llu\n",
+		key.objectid, new_gen);
+	rec->generation = new_gen;
+out:
+	btrfs_release_path(&path);
+	if (ret < 0)
+		btrfs_abort_transaction(trans, ret); 
+	return ret;
+}
+
 static int check_extent_refs(struct btrfs_root *root,
 			     struct cache_tree *extent_cache)
 {
@@ -8004,11 +8061,20 @@ static int check_extent_refs(struct btrfs_root *root,
 		}
 
 		if (rec->generation > super_gen + 1) {
-			error(
+			bool repaired = false;
+			if (repair) {
+				ret = repair_extent_item_generation(
+						root->fs_info, rec);
+				if (ret == 0)
+					repaired = true;
+			}
+			if (!repaired) {
+				error(
 	"invalid generation for extent %llu, have %llu expect (0, %llu]",
-				rec->start, rec->generation,
-				super_gen + 1);
-			cur_err = 1;
+					rec->start, rec->generation,
+					super_gen + 1);
+				cur_err = 1;
+			}
 		}
 		if (rec->refs != rec->extent_item_refs) {
 			fprintf(stderr, "ref mismatch on [%llu %llu] ",
Signature · 1 line
2.28.0
author
#4 lore raw

PATCH v2 4/4 btrfs-progs: tests/fsck: enhance invalid extent item generation test cases

In reply to Qu Wenruo

This patch will:
- Add a new test image for fsck/044
This new image has a corrupted extent item generation for tree block. This image can expose a bug in original mode, which can't detect the problem.
This image also utilize the tree block generation detection code, which the existing image doesn't.

- Rename the existing image
To reflect the fact that the existing one is only for data extent.

- Remove the test.sh
So that the generic path will test both detection and repair.

---
 ...xz => bad_extent_item_gen_for_data.img.xz} | Bin
 .../bad_extent_item_gen_for_tree_block.img.xz | Bin 0 -> 1804 bytes
 .../test.sh                                   |  19 ------------------
 3 files changed, 19 deletions(-)
 rename tests/fsck-tests/044-invalid-extent-item-generation/{bad_extent_item_gen.img.xz => bad_extent_item_gen_for_data.img.xz} (100%)
 create mode 100644 tests/fsck-tests/044-invalid-extent-item-generation/bad_extent_item_gen_for_tree_block.img.xz
 delete mode 100755 tests/fsck-tests/044-invalid-extent-item-generation/test.sh
tests/fsck-tests/044-invalid-extent-item-generation/bad_extent_item_gen_for_data.img.xz renamed from tests/fsck-tests/044-invalid-extent-item-generation/bad_extent_item_gen.img.xz +0 -0
similarity index 100%
rename from tests/fsck-tests/044-invalid-extent-item-generation/bad_extent_item_gen.img.xz
rename to tests/fsck-tests/044-invalid-extent-item-generation/bad_extent_item_gen_for_data.img.xz
tests/fsck-tests/044-invalid-extent-item-generation/bad_extent_item_gen_for_tree_block.img.xz new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..8bc4688dea9c952282b7124fc53175fe1ebd9c4f
GIT binary patch
tests/fsck-tests/044-invalid-extent-item-generation/test.sh deleted +0 -19
deleted file mode 100755
index 2b88a3c7b3bb..000000000000
@@ -1,19 +0,0 @@
-#!/bin/bash
-#
-# Due to a bug in --init-extent-tree option, we may create bad generation
-# number for data extents.
-#
-# This test case will ensure btrfs check can at least detect such problem,
-# just like kernel tree-checker.
-
-source "$TEST_TOP/common"
-
-check_prereq btrfs
-
-check_image() {
-	run_mustfail \
-		"btrfs check failed to detect invalid extent item generation" \
-		"$TOP/btrfs" check "$1"
-}
-
-check_all_images
Signature · 1 line
2.28.0
#5 lore raw
In reply to Qu Wenruo
Quoted 19 lines · Qu Wenruo

Although we have introduced the check ability to detect bad extent item generation, there is no repair ability.

I thought it would be rare to hit, but real world cases prove I'm a total idiot.

So this patchset will add the ability to repair, for both lowmem mode and original mode, along with enhanced test images.

There is also a bug fix for original mode, which fails to detect such problem if it's a tree block.

Changelog:
v2:
- Fix a type in the subject of the 4th patch
- Fix a bracket for for a logical and and bit and
The old code is fine and bit and has higher priority, but the bracket is intended to make that higher priority more obvious.

Quoted 9 lines

Qu Wenruo (4):
btrfs-progs: check/lowmem: add the ability to repair extent item
generation
btrfs-progs: check/original: don't reset extent generation for
check_block()
btrfs-progs: check/original: add the ability to repair extent item
generation
btrfs-progs: tests/fsck: enhance invalid extent item generation test
cases

Added to devel, thanks.