Skip to contentSkip to conversation

btrfs-progs: add check and repair ability for super num devices mismatch

PATCH In review Qu Wenruo 3 participants 10 messages Feb 28, 2022 – Mar 24, 2022 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

cover btrfs-progs: add check and repair ability for super num devices mismatch

The patchset can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/super_num_devs

The 2nd patch contains a compressed raw image, thus it may be a little too large for the mail list.

This patchset will allow btrfs check to detect and repair super num devices mismatch.

The detect part is to iterate through chunk tree to grab the correct number of devices.
This is more reliable than counting devices in fs_devices, since seed device is in another fs_devices.

The repair is more straightforward, just reset the value in superblock and commit a transaction.

Qu Wenruo (2):
btrfs-progs: check: add check and repair ability for super num devs
mismatch
btrfs-progs: tests/fsck: add test case for super num devs mismatch

 check/main.c                                  |   1 +
 check/mode-common.c                           |  88 ++++++++++++++++++
 check/mode-common.h                           |   2 +
 .../default.raw.xz                            | Bin 0 -> 22028 bytes
 4 files changed, 91 insertions(+)
 create mode 100644 tests/fsck-tests/054-super-num-devs-mismatch/default.raw.xz
Signature · 1 line
2.35.1
author
#1 lore raw

PATCH 1/2 btrfs-progs: check: add check and repair ability for super num devs mismatch

[BUG]
There is a bug report of kernel rejecting fs which has a mismatch in super num devices and num devices found in chunk tree.

But btrfs-check reports no problem about the fs.

[CAUSE]
We just didn't verify super num devices against the result found in chunk tree.

[FIX]
Add such check and repair ability for btrfs-check.

The ability is mode independent.

---
 check/main.c        |  1 +
 check/mode-common.c | 88 +++++++++++++++++++++++++++++++++++++++++++++
 check/mode-common.h |  2 ++
 3 files changed, 91 insertions(+)
check/main.c +1 -0
index 8ccba4478de8..b29f6266b974 100644
@@ -9140,6 +9140,7 @@ static int do_check_chunks_and_extents(void)
 		if (ret > 0)
 			ret = 0;
 	}
+	ret = check_and_repair_super_num_devs(gfs_info);
 	return ret;
 }
 
check/mode-common.c +88 -0

91 lines not on this page · raw message

check/mode-common.h +2 -0
index b5e6b727fe73..d5bab85a4f5e 100644
@@ -201,4 +201,6 @@ int repair_dev_item_bytes_used(struct btrfs_fs_info *fs_info,
 
 int fill_csum_tree(struct btrfs_trans_handle *trans, bool search_fs_tree);
 
+int check_and_repair_super_num_devs(struct btrfs_fs_info *fs_info);
+
 #endif
Signature · 1 line
2.35.1
author
#5 lore raw

PATCH 2/2 btrfs-progs: tests/fsck: add test case for super num devs mismatch

In reply to Qu Wenruo

The new image will has incorrect super num devices (have 8 expect 1).

The image has to be raw format, as btrfs-image restore will reset super num devices to correct value.

---
 .../054-super-num-devs-mismatch/default.raw.xz  | Bin 0 -> 22028 bytes
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 tests/fsck-tests/054-super-num-devs-mismatch/default.raw.xz
tests/fsck-tests/054-super-num-devs-mismatch/default.raw.xz new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..af0306bedbd0bd45dfd2d65a0f0a9f6bf662dfa8
GIT binary patch
Signature · 1 line
2.35.1
#6 lore raw
In reply to Qu Wenruo
Quoted 9 lines · Qu Wenruo

The patchset can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/super_num_devs

The 2nd patch contains a compressed raw image, thus it may be a little too large for the mail list.

This patchset will allow btrfs check to detect and repair super num devices mismatch.

Quoted 7 lines

The detect part is to iterate through chunk tree to grab the correct number of devices.
This is more reliable than counting devices in fs_devices, since seed device is in another fs_devices.

The repair is more straightforward, just reset the value in superblock and commit a transaction.

As repair it should work and be safe, also it should be reasonably safe to be part of --repair. In addition to that we may also want the rescue subcommand as it's meant to be for such one-time fixes for specific errors.

#2 lore raw

Re: [PATCH 1/2] btrfs-progs: check: add check and repair ability for super num devs mismatch

In reply to Qu Wenruo
Quoted 10 lines · Qu Wenruo

[BUG]
There is a bug report of kernel rejecting fs which has a mismatch in super num devices and num devices found in chunk tree.

But btrfs-check reports no problem about the fs.

[CAUSE]
We just didn't verify super num devices against the result found in chunk tree.

Quoted 8 lines

[FIX]
Add such check and repair ability for btrfs-check.

The ability is mode independent.

With this patch applied there's a new test failure:

=== START TEST .../tests/fsck-tests/014-no-extent-info testing image no_extent.raw.restored
====== RUN MUSTFAIL .../btrfs check ./no_extent.raw.restored Opening filesystem to check...
Checking filesystem on ./no_extent.raw.restored
UUID: aeee7297-37a4-4547-a8a9-7b870d58d31f
cache and super generation don't match, space cache will be invalidated found 180224 bytes used, no error found
total csum bytes: 0
total tree bytes: 180224
total fs tree bytes: 81920
total extent tree bytes: 16384
btree space waste bytes: 138540
file data blocks allocated: 0
referenced 0
succeeded (unexpected!): .../btrfs check ./no_extent.raw.restored unexpected success: btrfs check should have detected corruption

#3 lore raw

Re: [PATCH 1/2] btrfs-progs: check: add check and repair ability for super num devs mismatch

Quoted 31 lines · David Sterba
Quoted 10 lines · Qu Wenruo

[BUG]
There is a bug report of kernel rejecting fs which has a mismatch in super num devices and num devices found in chunk tree.

But btrfs-check reports no problem about the fs.

[CAUSE]
We just didn't verify super num devices against the result found in chunk tree.

Quoted 8 lines

[FIX]
Add such check and repair ability for btrfs-check.

The ability is mode independent.

With this patch applied there's a new test failure:

=== START TEST .../tests/fsck-tests/014-no-extent-info testing image no_extent.raw.restored
====== RUN MUSTFAIL .../btrfs check ./no_extent.raw.restored Opening filesystem to check...
Checking filesystem on ./no_extent.raw.restored
UUID: aeee7297-37a4-4547-a8a9-7b870d58d31f
cache and super generation don't match, space cache will be invalidated found 180224 bytes used, no error found
total csum bytes: 0

Quoted 8 lines

total tree bytes: 180224
total fs tree bytes: 81920
total extent tree bytes: 16384
btree space waste bytes: 138540
file data blocks allocated: 0
referenced 0
succeeded (unexpected!): .../btrfs check ./no_extent.raw.restored unexpected success: btrfs check should have detected corruption

Weird, the problem is, for the restored image, if I run ./btrfs check manually it can detects the problem, but still go "no error found" path.

So it must be my patch overriding the return value.

Will fix it soon.

Thanks,
Qu

#8 lore raw
In reply to David Sterba
Quoted 10 lines · David Sterba
Quoted 5 lines · Qu Wenruo

The patchset can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/super_num_devs

The 2nd patch contains a compressed raw image, thus it may be a little too large for the mail list.

The compressed size is 22K, that's fine. I've recompressed it with 'xz --best -e' and it shaved a few hundred bytes but that was just out of curiosity.

I'm also a little interested in why things like zstd/xz is not that efficient in compressing raw btrfs images.

Especially when creating the image I have created the image with all zero (fallocated) file in the first place, thus most space should just be all zero, and any compression method should be super efficient on them.

My current guesses are:

- DUP metadata is not fully detected by the algo Especially when they are mostly quite some ranges away

- Some older tree blocks from mkfs

So for further raw image size reduction, we may want to:

- Use SINGLE metadata/data

- Fstrim the image first

Thanks,
Qu

#4 lore raw

Re: [PATCH 1/2] btrfs-progs: check: add check and repair ability for super num devs mismatch

In reply to Qu Wenruo
Quoted 46 lines · Qu Wenruo
Quoted 33 lines · David Sterba
Quoted 13 lines · Qu Wenruo

[BUG]
There is a bug report of kernel rejecting fs which has a mismatch in super num devices and num devices found in chunk tree.

But btrfs-check reports no problem about the fs.

[CAUSE]
We just didn't verify super num devices against the result found in chunk tree.

[FIX]
Add such check and repair ability for btrfs-check.

Quoted 7 lines

The ability is mode independent.

Link:
https://lore.kernel.org/linux-btrfs/CA+8xDSpvdm_U0QLBAnrH=zqDq_cWCOH5TiV46CKmp3igr44okQ@mail.gmail.com/

With this patch applied there's a new test failure:

=== START TEST .../tests/fsck-tests/014-no-extent-info testing image no_extent.raw.restored
====== RUN MUSTFAIL .../btrfs check ./no_extent.raw.restored Opening filesystem to check...
Checking filesystem on ./no_extent.raw.restored
UUID: aeee7297-37a4-4547-a8a9-7b870d58d31f
cache and super generation don't match, space cache will be invalidated found 180224 bytes used, no error found
total csum bytes: 0

Quoted 8 lines

total tree bytes: 180224
total fs tree bytes: 81920
total extent tree bytes: 16384
btree space waste bytes: 138540
file data blocks allocated: 0
  referenced 0
succeeded (unexpected!): .../btrfs check ./no_extent.raw.restored unexpected success: btrfs check should have detected corruption

Quoted 4 lines

Weird, the problem is, for the restored image, if I run ./btrfs check manually it can detects the problem, but still go "no error found" path.

So it must be my patch overriding the return value.

Indeed that's the case.

Fix upon current devel branch is:
https://lore.kernel.org/linux-btrfs/f576d7a548b91d42d02b17d2dc52ee04d943a57d.1648077794.git.wqu@suse.com/T/#u

Please fold the fix into the patch.

Thanks,
Qu

Quoted 5 lines

Will fix it soon.

Thanks,
Qu

#9 lore raw
In reply to Qu Wenruo
Quoted 10 lines · Qu Wenruo
Quoted 7 lines · David Sterba
Quoted 5 lines · Qu Wenruo

The patchset can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/super_num_devs

The 2nd patch contains a compressed raw image, thus it may be a little too large for the mail list.

Quoted 6 lines
Quoted 3 lines

The compressed size is 22K, that's fine. I've recompressed it with 'xz --best -e' and it shaved a few hundred bytes but that was just out of curiosity.

I'm also a little interested in why things like zstd/xz is not that efficient in compressing raw btrfs images.

I think it is efficient, unpacked image was 128M and getting that to 22K is a good result.

Quoted 9 lines

Especially when creating the image I have created the image with all zero (fallocated) file in the first place, thus most space should just be all zero, and any compression method should be super efficient on them.

My current guesses are:

- DUP metadata is not fully detected by the algo Especially when they are mostly quite some ranges away

Quoted 7 lines

- Some older tree blocks from mkfs

So for further raw image size reduction, we may want to:

- Use SINGLE metadata/data

- Fstrim the image first

Yeah that could remove any stale blocks.