btrfs-progs: add check and repair ability for super num devices mismatch
cover btrfs-progs: add check and repair ability for super num devices mismatch
The patchset can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/super_num_devs
The 2nd patch contains a compressed raw image, thus it may be a little too large for the mail list.
This patchset will allow btrfs check to detect and repair super num devices mismatch.
The detect part is to iterate through chunk tree to grab the correct number of devices.
This is more reliable than counting devices in fs_devices, since seed device is in another fs_devices.
The repair is more straightforward, just reset the value in superblock and commit a transaction.
Qu Wenruo (2):
btrfs-progs: check: add check and repair ability for super num devs
mismatch
btrfs-progs: tests/fsck: add test case for super num devs mismatch
check/main.c | 1 + check/mode-common.c | 88 ++++++++++++++++++ check/mode-common.h | 2 + .../default.raw.xz | Bin 0 -> 22028 bytes 4 files changed, 91 insertions(+) create mode 100644 tests/fsck-tests/054-super-num-devs-mismatch/default.raw.xz
Signature · 1 line
2.35.1
PATCH 1/2 btrfs-progs: check: add check and repair ability for super num devs mismatch
[BUG]
There is a bug report of kernel rejecting fs which has a mismatch in super num devices and num devices found in chunk tree.
But btrfs-check reports no problem about the fs.
[CAUSE]
We just didn't verify super num devices against the result found in chunk tree.
[FIX]
Add such check and repair ability for btrfs-check.
The ability is mode independent.
--- check/main.c | 1 + check/mode-common.c | 88 +++++++++++++++++++++++++++++++++++++++++++++ check/mode-common.h | 2 ++ 3 files changed, 91 insertions(+)
check/main.c +1 -0
@@ -9140,6 +9140,7 @@ static int do_check_chunks_and_extents(void)if (ret >0 ) ret =0 ; } + ret =check_and_repair_super_num_devs (gfs_info);return ret; }
check/mode-common.c +88 -0
91 lines not on this page · raw message
check/mode-common.h +2 -0
@@ -201,4 +201,6 @@ int repair_dev_item_bytes_used(struct btrfs_fs_info *fs_info,int fill_csum_tree (struct btrfs_trans_handle *trans,bool search_fs_tree); +int check_and_repair_super_num_devs (struct btrfs_fs_info *fs_info); +#endif
Signature · 1 line
2.35.1
PATCH 2/2 btrfs-progs: tests/fsck: add test case for super num devs mismatch
In reply to Qu WenruoThe new image will has incorrect super num devices (have 8 expect 1).
The image has to be raw format, as btrfs-image restore will reset super num devices to correct value.
--- .../054-super-num-devs-mismatch/default.raw.xz | Bin 0 -> 22028 bytes 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 tests/fsck-tests/054-super-num-devs-mismatch/default.raw.xz
tests/fsck-tests/054-super-num-devs-mismatch/default.raw.xz new file binary +0 -0
Signature · 1 line
2.35.1
Quoted 9 lines · Qu Wenruo
The patchset can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/super_num_devsThe 2nd patch contains a compressed raw image, thus it may be a little too large for the mail list.
This patchset will allow btrfs check to detect and repair super num devices mismatch.
Quoted 7 lines
The detect part is to iterate through chunk tree to grab the correct number of devices.
This is more reliable than counting devices infs_devices, since seed device is in anotherfs_devices.The repair is more straightforward, just reset the value in superblock and commit a transaction.
As repair it should work and be safe, also it should be reasonably safe to be part of --repair. In addition to that we may also want the rescue subcommand as it's meant to be for such one-time fixes for specific errors.
Quoted 5 lines · Qu Wenruo
The patchset can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/super_num_devsThe 2nd patch contains a compressed raw image, thus it may be a little too large for the mail list.
The compressed size is 22K, that's fine. I've recompressed it with 'xz --best -e' and it shaved a few hundred bytes but that was just out of curiosity.
Re: [PATCH 1/2] btrfs-progs: check: add check and repair ability for super num devs mismatch
In reply to Qu WenruoQuoted 10 lines · Qu Wenruo
[BUG]
There is a bug report of kernel rejecting fs which has a mismatch in super num devices and num devices found in chunk tree.But btrfs-check reports no problem about the fs.
[CAUSE]
We just didn't verify super num devices against the result found in chunk tree.
Quoted 8 lines
[FIX]
Add such check and repair ability for btrfs-check.The ability is mode independent.
With this patch applied there's a new test failure:
=== START TEST .../tests/fsck-tests/014-no-extent-info testing image no_extent.raw.restored
====== RUN MUSTFAIL .../btrfs check ./no_extent.raw.restored Opening filesystem to check...
Checking filesystem on ./no_extent.raw.restored
UUID: aeee7297-37a4-4547-a8a9-7b870d58d31f
cache and super generation don't match, space cache will be invalidated found 180224 bytes used, no error found
total csum bytes: 0
total tree bytes: 180224
total fs tree bytes: 81920
total extent tree bytes: 16384
btree space waste bytes: 138540
file data blocks allocated: 0
referenced 0
succeeded (unexpected!): .../btrfs check ./no_extent.raw.restored unexpected success: btrfs check should have detected corruption
Re: [PATCH 1/2] btrfs-progs: check: add check and repair ability for super num devs mismatch
Quoted 31 lines · David Sterba
Quoted 10 lines · Qu Wenruo
[BUG]
There is a bug report of kernel rejecting fs which has a mismatch in super num devices and num devices found in chunk tree.But btrfs-check reports no problem about the fs.
[CAUSE]
We just didn't verify super num devices against the result found in chunk tree.Quoted 8 lines
[FIX]
Add such check and repair ability for btrfs-check.The ability is mode independent.
With this patch applied there's a new test failure:
=== START TEST .../tests/fsck-tests/014-no-extent-info testing image no_extent.raw.restored
====== RUN MUSTFAIL .../btrfs check ./no_extent.raw.restored Opening filesystem to check...
Checking filesystem on ./no_extent.raw.restored
UUID: aeee7297-37a4-4547-a8a9-7b870d58d31f
cache and super generation don't match, space cache will be invalidated found 180224 bytes used, no error found
total csum bytes: 0
Quoted 8 lines
total tree bytes: 180224
total fs tree bytes: 81920
total extent tree bytes: 16384
btree space waste bytes: 138540
file data blocks allocated: 0
referenced 0
succeeded (unexpected!): .../btrfs check ./no_extent.raw.restored unexpected success: btrfs check should have detected corruption
Weird, the problem is, for the restored image, if I run ./btrfs check manually it can detects the problem, but still go "no error found" path.
So it must be my patch overriding the return value.
Will fix it soon.
Thanks,
Qu
Quoted 10 lines · David Sterba
Quoted 5 lines · Qu Wenruo
The patchset can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/super_num_devsThe 2nd patch contains a compressed raw image, thus it may be a little too large for the mail list.
The compressed size is 22K, that's fine. I've recompressed it with 'xz --best -e' and it shaved a few hundred bytes but that was just out of curiosity.
I'm also a little interested in why things like zstd/xz is not that efficient in compressing raw btrfs images.
Especially when creating the image I have created the image with all zero (fallocated) file in the first place, thus most space should just be all zero, and any compression method should be super efficient on them.
My current guesses are:
- DUP metadata is not fully detected by the algo Especially when they are mostly quite some ranges away
- Some older tree blocks from mkfs
So for further raw image size reduction, we may want to:
- Use SINGLE metadata/data
- Fstrim the image first
Thanks,
Qu
Re: [PATCH 1/2] btrfs-progs: check: add check and repair ability for super num devs mismatch
In reply to Qu WenruoQuoted 46 lines · Qu Wenruo
Quoted 33 lines · David Sterba
Quoted 13 lines · Qu Wenruo
[BUG]
There is a bug report of kernel rejecting fs which has a mismatch in super num devices and num devices found in chunk tree.But btrfs-check reports no problem about the fs.
[CAUSE]
We just didn't verify super num devices against the result found in chunk tree.[FIX]
Add such check and repair ability for btrfs-check.Quoted 7 lines
The ability is mode independent.
Reported-by Luca Béla Palkovics <[email protected]>Signed-off-by Qu Wenruo <[email protected]>With this patch applied there's a new test failure:
=== START TEST .../tests/fsck-tests/014-no-extent-info testing image no_extent.raw.restored
====== RUN MUSTFAIL .../btrfs check ./no_extent.raw.restored Opening filesystem to check...
Checking filesystem on ./no_extent.raw.restored
UUID: aeee7297-37a4-4547-a8a9-7b870d58d31f
cache and super generation don't match, space cache will be invalidated found 180224 bytes used, no error found
total csum bytes: 0Quoted 8 lines
total tree bytes: 180224
total fs tree bytes: 81920
total extent tree bytes: 16384
btree space waste bytes: 138540
file data blocks allocated: 0
referenced 0
succeeded (unexpected!): .../btrfs check ./no_extent.raw.restored unexpected success: btrfs check should have detected corruption
Quoted 4 lines
Weird, the problem is, for the restored image, if I run ./btrfs check manually it can detects the problem, but still go "no error found" path.
So it must be my patch overriding the return value.
Indeed that's the case.
Fix upon current devel branch is:
https://lore.kernel.org/linux-btrfs/f576d7a548b91d42d02b17d2dc52ee04d943a57d.1648077794.git.wqu@suse.com/T/#u
Please fold the fix into the patch.
Thanks,
Qu
Quoted 5 lines
Will fix it soon.
Thanks,
Qu
Quoted 10 lines · Qu Wenruo
Quoted 7 lines · David Sterba
Quoted 5 lines · Qu Wenruo
The patchset can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/super_num_devsThe 2nd patch contains a compressed raw image, thus it may be a little too large for the mail list.
Quoted 6 lines
Quoted 3 lines
The compressed size is 22K, that's fine. I've recompressed it with 'xz --best -e' and it shaved a few hundred bytes but that was just out of curiosity.
I'm also a little interested in why things like zstd/xz is not that efficient in compressing raw btrfs images.
I think it is efficient, unpacked image was 128M and getting that to 22K is a good result.
Quoted 9 lines
Especially when creating the image I have created the image with all zero (fallocated) file in the first place, thus most space should just be all zero, and any compression method should be super efficient on them.
My current guesses are:
- DUP metadata is not fully detected by the algo Especially when they are mostly quite some ranges away
Quoted 7 lines
- Some older tree blocks from mkfs
So for further raw image size reduction, we may want to:
- Use SINGLE metadata/data
- Fstrim the image first
Yeah that could remove any stale blocks.