btrfs-progs: btrfs-image related fixes
cover btrfs-progs: btrfs-image related fixes
This branch can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/image_fixes
Since there are two binary files updates, and one big code move, it's recommended to fetch github repo, in case some patches didn't reach mail list.
This is inspried by one log tree replay dead loop bug, where the kind reporter, Pierre Abbat <[email protected]>, gave the btrfs-image to reproduce it.
Then the image fails to pass check due to the existing log tree conflicting with device/chunk fixup.
As log tree blocks are not recorded in extent tree, later COW can use log tree blocks and cause transid mismatch.
To address the problem, this patchset will:
- Don't do any fixup if the source dump is single device Since the dump has the full super block contents, we can easily check if the source fs is single deivce.
The chunk/device fixup is mostly for older btrfs-image behavior, which always restores the fs into SINGLE profile.
However since commit 9088ab6a1067 ("btrfs-progs: make btrfs-image restore to support dup"), btrfs-image can restore into DUP profile, allowing us to do exact replay for single device fs.
This is patch 5.
- Pin down all log tree blocks for fixup
For cases we still need to fixup chunk/device items, at least pin down all log tree blocks to avoid transid mimsatch.
This is patch 6.
After above fixes, fsck/012 and fsck/035 fails, due to bad original images.
The old btrfs-image can fixup those bad device total_bytes and bytes_used, but that just hides the problem.
We still need to update those images to make them correct, so here comes patch 3 and 4.
During the debugging of btrfs-image restore, I found dump_superblock() would help a lot to expose bad values in images, so is btrfs_print_leaf().
Enahance them to be more handy for usage inside gdb, and here comes patch 1 and 2.
Qu Wenruo (6):
btrfs-progs: Allow btrfs_print_leaf() to be called on dummy eb whose
fs_info is NULL
btrfs-progs: print-tree: Export dump_superblock()
btrfs-progs: fsck-tests: Update the image in 012
btrfs-progs: fsck-tests: Update the image of test case 035
btrfs-progs: image: Don't modify the chunk and device tree if the
source dump is single device
btrfs-progs: image: Pin down log tree blocks before fixup
cmds/inspect-dump-super.c | 454 +----------------- image/main.c | 86 +++- print-tree.c | 449 ++++++++++++++++- print-tree.h | 1 + .../012-leaf-corruption/good.img.xz | Bin 0 -> 186392 bytes .../012-leaf-corruption/no_data_extent.tar.xz | Bin 130260 -> 0 bytes tests/fsck-tests/012-leaf-corruption/test.sh | 17 +- .../offset_by_one.img | Bin 3072 -> 3072 bytes 8 files changed, 540 insertions(+), 467 deletions(-) create mode 100644 tests/fsck-tests/012-leaf-corruption/good.img.xz delete mode 100644 tests/fsck-tests/012-leaf-corruption/no_data_extent.tar.xz
Signature · 1 line
2.26.2
PATCH 1/6 btrfs-progs: Allow btrfs_print_leaf() to be called on dummy eb whose fs_info is NULL
In certain call sites, like btrfs-image, we want full extent_buffer accessors, but don't have a btrfs_fs_info.
In that case, if an extent buffer whose fs_info is NULL, then btrfs_print_leaf() is called on such eb, we will trigger a NULL pointer dereference, at BTRFS_LEAF_DATA_SIZE(fs_info).
Fix this by using __BTRFS_LEAF_DATA_SIZE(), which only needs nodesize, and nodesize can be extracted from extent_buffer::len.
This allows us to call btrfs_print_leaf() on any extent buffer, which is super handy in gdb.
--- print-tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
print-tree.c +1 -1
@@ -1201,7 +1201,7 @@ void btrfs_print_leaf(struct extent_buffer *eb)struct btrfs_item *item;struct btrfs_disk_key disk_key;char flags_str[128 ]; -u32 leaf_data_size =BTRFS_LEAF_DATA_SIZE (fs_info); +u32 leaf_data_size =__BTRFS_LEAF_DATA_SIZE (eb->len);u32 i;u32 nr;u64 flags;
Signature · 1 line
2.26.2
PATCH 2/6 btrfs-progs: print-tree: Export dump_superblock()
In reply to Qu Wenruodump_superblock() is useful to debug btrfs-image error, like fsck/012-* test case, where the superblock itself has something wrong from the original image.
Export it so that we can call it in gdb.
Since we're exporting dump_superblock(), rename it to btrfs_print_superblock() to following the existing naming schema.
--- cmds/inspect-dump-super.c | 454 +------------------------------------- print-tree.c | 447 +++++++++++++++++++++++++++++++++++++ print-tree.h | 1 + 3 files changed, 449 insertions(+), 453 deletions(-)
cmds/inspect-dump-super.c +1 -453
@@ -16,471 +16,19 @@#include "kerncompat.h" #include <stdio.h> -#include <stdlib.h> #include <unistd.h> #include <fcntl.h> #include <sys/stat.h> -#include <ctype.h> -#include <uuid/uuid.h> #include <errno.h> #include <getopt.h> #include "ctree.h" #include "disk-io.h" #include "print-tree.h" -#include "transaction.h" -#include "kernel-lib/list.h" #include "common/utils.h" #include "cmds/commands.h" -#include "crypto/crc32c.h" #include "common/help.h" -static int check_csum_sblock (void *sb,int csum_size,u16 csum_type) -{ -u8 result[BTRFS_CSUM_SIZE ]; - -btrfs_csum_data (csum_type, (u8 *)sb +BTRFS_CSUM_SIZE , - result,BTRFS_SUPER_INFO_SIZE -BTRFS_CSUM_SIZE ); - -return !memcmp (sb, result, csum_size); -} - -static void print_sys_chunk_array (struct btrfs_super_block *sb) -{ -struct extent_buffer *buf; -struct btrfs_disk_key *disk_key; -struct btrfs_chunk *chunk; -u8 *array_ptr; -unsigned long sb_array_offset; -u32 num_stripes; -u32 array_size; -u32 len =0 ; -u32 cur_offset; -struct btrfs_key key; -int item; - - buf =malloc (sizeof (*buf) +sizeof (*sb)); -if (!buf) { -error ("not enough memory" ); -return ; - } -write_extent_buffer (buf, sb,0 ,sizeof (*sb)); - buf->len =sizeof (*sb); - array_size =btrfs_super_sys_array_size (sb); - - array_ptr = sb->sys_chunk_array; - sb_array_offset = offsetof(struct btrfs_super_block , sys_chunk_array); - -if (array_size >BTRFS_SYSTEM_CHUNK_ARRAY_SIZE ) { -error ("sys_array_size %u shouldn't exceed %u bytes" , - array_size,BTRFS_SYSTEM_CHUNK_ARRAY_SIZE ); - goto out; - } - - cur_offset =0 ; - item =0 ; - -while (cur_offset < array_size) { - disk_key = (struct btrfs_disk_key *)array_ptr; - len =sizeof (*disk_key); -if (cur_offset + len > array_size) - goto out_short_read; - -btrfs_disk_key_to_cpu (&key, disk_key); - - array_ptr += len; - sb_array_offset += len; - cur_offset += len; - -printf ("\titem %d " , item); -btrfs_print_key (disk_key); -putchar ('\n' ); - -if (key.type ==BTRFS_CHUNK_ITEM_KEY ) { - chunk = (struct btrfs_chunk *)sb_array_offset; -/* -* At least one btrfs_chunk with one stripe must be -* present, exact stripe count check comes afterwards -*/ - len =btrfs_chunk_item_size (1 ); -if (cur_offset + len > array_size) - goto out_short_read; - - num_stripes =btrfs_chunk_num_stripes (buf, chunk); -if (!num_stripes) { -error ( -"invalid number of stripes %u in sys_array at offset %u" , - num_stripes, cur_offset); -break ; - } - len =btrfs_chunk_item_size (num_stripes); -if (cur_offset + len > array_size) - goto out_short_read; -print_chunk_item (buf, chunk); - }else { -error ("unexpected item type %u in sys_array at offset %u" , - (u32 )key.type, cur_offset); -break ; - } - array_ptr += len; - sb_array_offset += len; - cur_offset += len; - - item++; - } - -out: -free (buf); -return ; - -out_short_read: -error ("sys_array too short to read %u bytes at offset %u" , - len, cur_offset); -free (buf); -} - -static int empty_backup (struct btrfs_root_backup *backup) -{ -if (backup ==NULL || - (backup->tree_root ==0 && - backup->tree_root_gen ==0 )) -return 1 ; -return 0 ; -} - -static void print_root_backup (struct btrfs_root_backup *backup) -{ -printf ("\t\tbackup_tree_root:\t%llu\tgen: %llu\tlevel: %d\n" , -btrfs_backup_tree_root (backup), -btrfs_backup_tree_root_gen (backup), -btrfs_backup_tree_root_level (backup)); -printf ("\t\tbackup_chunk_root:\t%llu\tgen: %llu\tlevel: %d\n" , -btrfs_backup_chunk_root (backup), -btrfs_backup_chunk_root_gen (backup), -btrfs_backup_chunk_root_level (backup)); -printf ("\t\tbackup_extent_root:\t%llu\tgen: %llu\tlevel: %d\n" , -btrfs_backup_extent_root (backup), -btrfs_backup_extent_root_gen (backup), -btrfs_backup_extent_root_level (backup)); -printf ("\t\tbackup_fs_root:\t\t%llu\tgen: %llu\tlevel: %d\n" , -btrfs_backup_fs_root (backup), -btrfs_backup_fs_root_gen (backup), -btrfs_backup_fs_root_level (backup)); -printf ("\t\tbackup_dev_root:\t%llu\tgen: %llu\tlevel: %d\n" , -btrfs_backup_dev_root (backup), -btrfs_backup_dev_root_gen (backup), -btrfs_backup_dev_root_level (backup)); -printf ("\t\tbackup_csum_root:\t%llu\tgen: %llu\tlevel: %d\n" , -btrfs_backup_csum_root (backup), -btrfs_backup_csum_root_gen (backup), -btrfs_backup_csum_root_level (backup)); - -printf ("\t\tbackup_total_bytes:\t%llu\n" , -btrfs_backup_total_bytes (backup)); -printf ("\t\tbackup_bytes_used:\t%llu\n" , -btrfs_backup_bytes_used (backup)); -printf ("\t\tbackup_num_devices:\t%llu\n" , -btrfs_backup_num_devices (backup)); -putchar ('\n' ); -} - -static void print_backup_roots (struct btrfs_super_block *sb) -{ -struct btrfs_root_backup *backup; -int i; - -for (i =0 ; i <BTRFS_NUM_BACKUP_ROOTS ; i++) { - backup = sb->super_roots + i; -if (!empty_backup (backup)) { -printf ("\tbackup %d:\n" , i); -print_root_backup (backup); - } - } -} - -struct readable_flag_entry { -u64 bit; -char *output; -}; - -#define DEF_COMPAT_RO_FLAG_ENTRY (bit_name) \ - {BTRFS_FEATURE_COMPAT_RO_##bit_name, #bit_name} - -static struct readable_flag_entry compat_ro_flags_array[] = { -DEF_COMPAT_RO_FLAG_ENTRY (FREE_SPACE_TREE ), -DEF_COMPAT_RO_FLAG_ENTRY (FREE_SPACE_TREE_VALID ), -}; -static const int compat_ro_flags_num =sizeof (compat_ro_flags_array) / -sizeof (struct readable_flag_entry ); - -#define DEF_INCOMPAT_FLAG_ENTRY (bit_name) \ - {BTRFS_FEATURE_INCOMPAT_##bit_name, #bit_name} - -static struct readable_flag_entry incompat_flags_array[] = { -DEF_INCOMPAT_FLAG_ENTRY (MIXED_BACKREF ), -DEF_INCOMPAT_FLAG_ENTRY (DEFAULT_SUBVOL ), -DEF_INCOMPAT_FLAG_ENTRY (MIXED_GROUPS ), -DEF_INCOMPAT_FLAG_ENTRY (COMPRESS_LZO ), -DEF_INCOMPAT_FLAG_ENTRY (COMPRESS_ZSTD ), -DEF_INCOMPAT_FLAG_ENTRY (BIG_METADATA ), -DEF_INCOMPAT_FLAG_ENTRY (EXTENDED_IREF ), -DEF_INCOMPAT_FLAG_ENTRY (RAID56 ), -DEF_INCOMPAT_FLAG_ENTRY (SKINNY_METADATA ), -DEF_INCOMPAT_FLAG_ENTRY (NO_HOLES ), -DEF_INCOMPAT_FLAG_ENTRY (METADATA_UUID ), -DEF_INCOMPAT_FLAG_ENTRY (RAID1C34 ), -}; -static const int incompat_flags_num =sizeof (incompat_flags_array) / -sizeof (struct readable_flag_entry ); - -#define DEF_HEADER_FLAG_ENTRY (bit_name) \ - {BTRFS_HEADER_FLAG_##bit_name, #bit_name} -#define DEF_SUPER_FLAG_ENTRY (bit_name) \ - {BTRFS_SUPER_FLAG_##bit_name, #bit_name} - -static struct readable_flag_entry super_flags_array[] = { -DEF_HEADER_FLAG_ENTRY (WRITTEN ), -DEF_HEADER_FLAG_ENTRY (RELOC ), -DEF_SUPER_FLAG_ENTRY (CHANGING_FSID ), -DEF_SUPER_FLAG_ENTRY (SEEDING ), -DEF_SUPER_FLAG_ENTRY (METADUMP ), -DEF_SUPER_FLAG_ENTRY (METADUMP_V2 ) -}; -static const int super_flags_num =ARRAY_SIZE (super_flags_array); - -#define BTRFS_SUPER_FLAG_SUPP (BTRFS_HEADER_FLAG_WRITTEN |\ - BTRFS_HEADER_FLAG_RELOC |\ - BTRFS_SUPER_FLAG_CHANGING_FSID |\ - BTRFS_SUPER_FLAG_SEEDING |\ - BTRFS_SUPER_FLAG_METADUMP |\ - BTRFS_SUPER_FLAG_METADUMP_V2) - -static void __print_readable_flag (u64 flag,struct readable_flag_entry *array, -int array_size,u64 supported_flags) -{ -int i; -int first =1 ; -struct readable_flag_entry *entry; - -if (!flag) -return ; - -printf ("\t\t\t( " ); -for (i =0 ; i < array_size; i++) { - entry = array + i; -if (flag & entry->bit) { -if (first) -printf ("%s " , entry->output); -else -printf ("|\n\t\t\t %s " , entry->output); - first =0 ; - } - } - flag &= ~supported_flags; -if (flag) { -if (first) -printf ("unknown flag: 0x%llx " , flag); -else -printf ("|\n\t\t\t unknown flag: 0x%llx " , flag); - } -printf (")\n" ); -} - -static void print_readable_compat_ro_flag (u64 flag) -{ -/* -* We know about the FREE_SPACE_TREE{,_VALID} bits, but we don't -* actually support them yet. -*/ -return __print_readable_flag (flag, compat_ro_flags_array, - compat_ro_flags_num, -BTRFS_FEATURE_COMPAT_RO_SUPP | -BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE | -BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE_VALID ); -} - -static void print_readable_incompat_flag (u64 flag) -{ -return __print_readable_flag (flag, incompat_flags_array, - incompat_flags_num, -BTRFS_FEATURE_INCOMPAT_SUPP ); -} - -static void print_readable_super_flag (u64 flag) -{ -return __print_readable_flag (flag, super_flags_array, - super_flags_num,BTRFS_SUPER_FLAG_SUPP ); -} - -static bool is_valid_csum_type (u16 csum_type) -{ -switch (csum_type) { -case BTRFS_CSUM_TYPE_CRC32 : -case BTRFS_CSUM_TYPE_XXHASH : -case BTRFS_CSUM_TYPE_SHA256 : -case BTRFS_CSUM_TYPE_BLAKE2 : -return true; -default : -return false; - } -} - -static void dump_superblock (struct btrfs_super_block *sb,int full) -{ -int i; -char *s, buf[BTRFS_UUID_UNPARSED_SIZE ]; -u8 *p; -u32 csum_size; -u16 csum_type; -bool metadata_uuid_present = (btrfs_super_incompat_flags (sb) & -BTRFS_FEATURE_INCOMPAT_METADATA_UUID ); -int cmp_res =0 ; - - - csum_type =btrfs_super_csum_type (sb); - csum_size =BTRFS_CSUM_SIZE ; -printf ("csum_type\t\t%hu (" , csum_type); -if (!is_valid_csum_type (csum_type)) { -printf ("INVALID" ); - }else { -printf ("%s" ,btrfs_super_csum_name (csum_type)); - csum_size =btrfs_super_csum_size (sb); - } -printf (")\n" ); -printf ("csum_size\t\t%llu\n" , (unsigned long long )csum_size); - -printf ("csum\t\t\t0x" ); -for (i =0 , p = sb->csum; i < csum_size; i++) -printf ("%02x" , p[i]); -if (!is_valid_csum_type (csum_type)) -printf (" [UNKNOWN CSUM TYPE OR SIZE]" ); -else if (check_csum_sblock (sb, csum_size, csum_type)) -printf (" [match]" ); -else -printf (" [DON'T MATCH]" ); -putchar ('\n' ); - -printf ("bytenr\t\t\t%llu\n" , - (unsigned long long )btrfs_super_bytenr (sb)); -printf ("flags\t\t\t0x%llx\n" , - (unsigned long long )btrfs_super_flags (sb)); -print_readable_super_flag (btrfs_super_flags (sb)); - -printf ("magic\t\t\t" ); - s = (char *) &sb->magic; -for (i =0 ; i <8 ; i++) -putchar (isprint (s[i]) ? s[i] :'.' ); -if (btrfs_super_magic (sb) ==BTRFS_MAGIC ) -printf (" [match]\n" ); -else -printf (" [DON'T MATCH]\n" ); - -uuid_unparse (sb->fsid, buf); -printf ("fsid\t\t\t%s\n" , buf); -if (metadata_uuid_present) { -uuid_unparse (sb->metadata_uuid, buf); -printf ("metadata_uuid\t\t%s\n" , buf); - }else { -printf ("metadata_uuid\t\t%s\n" , buf); - } - -printf ("label\t\t\t" ); - s = sb->label; -for (i =0 ; i <BTRFS_LABEL_SIZE && s[i]; i++) -putchar (isprint (s[i]) ? s[i] :'.' ); -putchar ('\n' ); - -printf ("generation\t\t%llu\n" , - (unsigned long long )btrfs_super_generation (sb)); -printf ("root\t\t\t%llu\n" , (unsigned long long )btrfs_super_root (sb)); -printf ("sys_array_size\t\t%llu\n" , - (unsigned long long )btrfs_super_sys_array_size (sb)); -printf ("chunk_root_generation\t%llu\n" , - (unsigned long long )btrfs_super_chunk_root_generation (sb)); -printf ("root_level\t\t%llu\n" , - (unsigned long long )btrfs_super_root_level (sb)); -printf ("chunk_root\t\t%llu\n" , - (unsigned long long )btrfs_super_chunk_root (sb)); -printf ("chunk_root_level\t%llu\n" , - (unsigned long long )btrfs_super_chunk_root_level (sb)); -printf ("log_root\t\t%llu\n" , - (unsigned long long )btrfs_super_log_root (sb)); -printf ("log_root_transid\t%llu\n" , - (unsigned long long )btrfs_super_log_root_transid (sb)); -printf ("log_root_level\t\t%llu\n" , - (unsigned long long )btrfs_super_log_root_level (sb)); -printf ("total_bytes\t\t%llu\n" , - (unsigned long long )btrfs_super_total_bytes (sb)); -printf ("bytes_used\t\t%llu\n" , - (unsigned long long )btrfs_super_bytes_used (sb)); -printf ("sectorsize\t\t%llu\n" , - (unsigned long long )btrfs_super_sectorsize (sb)); -printf ("nodesize\t\t%llu\n" , - (unsigned long long )btrfs_super_nodesize (sb)); -printf ("leafsize (deprecated)\t%u\n" , -le32_to_cpu (sb->__unused_leafsize)); -printf ("stripesize\t\t%llu\n" , - (unsigned long long )btrfs_super_stripesize (sb)); -printf ("root_dir\t\t%llu\n" , - (unsigned long long )btrfs_super_root_dir (sb)); -printf ("num_devices\t\t%llu\n" , - (unsigned long long )btrfs_super_num_devices (sb)); -printf ("compat_flags\t\t0x%llx\n" , - (unsigned long long )btrfs_super_compat_flags (sb)); -printf ("compat_ro_flags\t\t0x%llx\n" , - (unsigned long long )btrfs_super_compat_ro_flags (sb)); -print_readable_compat_ro_flag (btrfs_super_compat_ro_flags (sb)); -printf ("incompat_flags\t\t0x%llx\n" , - (unsigned long long )btrfs_super_incompat_flags (sb)); -print_readable_incompat_flag (btrfs_super_incompat_flags (sb)); -printf ("cache_generation\t%llu\n" , - (unsigned long long )btrfs_super_cache_generation (sb)); -printf ("uuid_tree_generation\t%llu\n" , - (unsigned long long )btrfs_super_uuid_tree_generation (sb)); - -uuid_unparse (sb->dev_item.uuid, buf); -printf ("dev_item.uuid\t\t%s\n" , buf); - -uuid_unparse (sb->dev_item.fsid, buf); -if (metadata_uuid_present) { - cmp_res = !memcmp (sb->dev_item.fsid, sb->metadata_uuid, -BTRFS_FSID_SIZE ); - }else { - cmp_res = !memcmp (sb->dev_item.fsid, sb->fsid,BTRFS_FSID_SIZE ); - } -printf ("dev_item.fsid\t\t%s %s\n" , buf, - cmp_res ?"[match]" :"[DON'T MATCH]" ); - -printf ("dev_item.type\t\t%llu\n" , (unsigned long long ) -btrfs_stack_device_type (&sb->dev_item)); -printf ("dev_item.total_bytes\t%llu\n" , (unsigned long long ) -btrfs_stack_device_total_bytes (&sb->dev_item)); -printf ("dev_item.bytes_used\t%llu\n" , (unsigned long long ) -btrfs_stack_device_bytes_used (&sb->dev_item)); -printf ("dev_item.io_align\t%u\n" , (unsigned int ) -btrfs_stack_device_io_align (&sb->dev_item)); -printf ("dev_item.io_width\t%u\n" , (unsigned int ) -btrfs_stack_device_io_width (&sb->dev_item)); -printf ("dev_item.sector_size\t%u\n" , (unsigned int ) -btrfs_stack_device_sector_size (&sb->dev_item)); -printf ("dev_item.devid\t\t%llu\n" , -btrfs_stack_device_id (&sb->dev_item)); -printf ("dev_item.dev_group\t%u\n" , (unsigned int ) -btrfs_stack_device_group (&sb->dev_item)); -printf ("dev_item.seek_speed\t%u\n" , (unsigned int ) -btrfs_stack_device_seek_speed (&sb->dev_item)); -printf ("dev_item.bandwidth\t%u\n" , (unsigned int ) -btrfs_stack_device_bandwidth (&sb->dev_item)); -printf ("dev_item.generation\t%llu\n" , (unsigned long long ) -btrfs_stack_device_generation (&sb->dev_item)); -if (full) { -printf ("sys_chunk_array[%d]:\n" ,BTRFS_SYSTEM_CHUNK_ARRAY_SIZE ); -print_sys_chunk_array (sb); -printf ("backup_roots[%d]:\n" ,BTRFS_NUM_BACKUP_ROOTS ); -print_backup_roots (sb); - } -} -static int load_and_dump_sb (char *filename,int fd,u64 sb_bytenr,int full,int force) {
@@ -507,7 +55,7 @@ static int load_and_dump_sb(char *filename, int fd, u64 sb_bytenr, int full,error ("bad magic on superblock on %s at %llu" , filename, (unsigned long long )sb_bytenr); }else { -dump_superblock (sb, full); +btrfs_print_superblock (sb, full); }return 0 ; }
print-tree.c +447 -0
456 lines not on this page · raw message
print-tree.h +1 -0
@@ -39,4 +39,5 @@ void print_chunk_item(struct extent_buffer *eb, struct btrfs_chunk *chunk);void print_extent_item (struct extent_buffer *eb,int slot,int metadata);void print_objectid (FILE *stream,u64 objectid,u8 type);void print_key_type (FILE *stream,u64 objectid,u8 type); +void btrfs_print_superblock (struct btrfs_super_block *sb,int full);#endif
Signature · 1 line
2.26.2
PATCH 4/6 btrfs-progs: fsck-tests: Update the image of test case 035
In reply to Qu WenruoThe image used in test case 035 has one problem:
- Bad total_bytes of device item in chunk tree
item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
devid 1 total_bytes 72564736 bytes_used 92274688
The total_bytes is even smaller than bytes_used.
While the dev_item in super block contains the correct value:
dev_item.total_bytes 134217728
dev_item.bytes_used 92274688This patch will update the image, using the correct value in super block to replace the wrong one in chunk tree.
So that later btrfs-image restore without device/chunk fixup can still pass fsck-tests.
--- .../offset_by_one.img | Bin 3072 -> 3072 bytes 1 file changed, 0 insertions(+), 0 deletions(-)
tests/fsck-tests/035-inline-bad-ram-bytes/offset_by_one.img binary +0 -0
Signature · 1 line
2.26.2
PATCH 5/6 btrfs-progs: image: Don't modify the chunk and device tree if the source dump is single device
In reply to Qu WenruoBtrfs-image -r doesn't always create exactly the same fs of the original fs, this is because btrfs-image can create dump from multi-device, and restore it to single device.
Thus we need some special modification to chunk and device tree. This behavior is mostly to handle the old behavior where we always restore the metadata into SINGLE profile no matter what.
However this is not needed if the source fs only has one device, in that case, we can use all the metadata as is, no need to modify the fs at all, resulting an exact copy of metadata.
This patch will do extra check when doing restore, to avoid modify the restored fs if the source is also single device.
--- image/main.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-)
image/main.c +19 -1
@@ -103,6 +103,7 @@ struct mdrestore_struct {struct rb_root physical_tree;struct list_head list;struct list_head overlapping_chunks; +struct btrfs_super_block *original_super;size_t num_items;u32 nodesize;u64 devid;
@@ -1085,6 +1086,12 @@ static int update_super(struct mdrestore_struct *mdres, u8 *buffer)u8 *ptr, *write_ptr;int old_num_stripes; +/* No need to fix, use all data as is */ +if (btrfs_super_num_devices (mdres->original_super) ==1 ) { + new_array_size =btrfs_super_sys_array_size (super); + goto finish; + } + write_ptr = ptr = super->sys_chunk_array; array_size =btrfs_super_sys_array_size (super);
@@ -1135,6 +1142,7 @@ static int update_super(struct mdrestore_struct *mdres, u8 *buffer) cur +=btrfs_chunk_item_size (old_num_stripes); } +finish:if (mdres->clear_space_cache)btrfs_set_super_cache_generation (super,0 );
@@ -1203,6 +1211,8 @@ static int fixup_chunk_tree_block(struct mdrestore_struct *mdres,u64 bytenr = async->start;int i; +if (btrfs_super_num_devices (mdres->original_super) ==1 ) +return 0 ;if (size_left % mdres->nodesize)return 0 ;
@@ -1373,6 +1383,8 @@ static void *restore_worker(void *data)if (!mdres->multi_devices) {if (async->start ==BTRFS_SUPER_INFO_OFFSET ) { +memcpy (mdres->original_super, outbuf, +BTRFS_SUPER_INFO_SIZE );if (mdres->old_restore) {update_super_old (outbuf); }else {
@@ -1474,6 +1486,7 @@ static void mdrestore_destroy(struct mdrestore_struct *mdres, int num_threads)pthread_cond_destroy (&mdres->cond);pthread_mutex_destroy (&mdres->mutex); +free (mdres->original_super); } static int mdrestore_init(struct mdrestore_struct *mdres,
@@ -1500,6 +1513,10 @@ static int mdrestore_init(struct mdrestore_struct *mdres, mdres->last_physical_offset =0 ; mdres->alloced_chunks =0 ; + mdres->original_super =malloc (BTRFS_SUPER_INFO_SIZE ); +if (!mdres->original_super) +return -ENOMEM ; +if (!num_threads)return 0 ;
@@ -2606,7 +2623,8 @@ static int restore_metadump(const char *input, FILE *out, int old_restore, } ret =wait_for_worker (&mdrestore); -if (!ret && !multi_devices && !old_restore) { +if (!ret && !multi_devices && !old_restore && +btrfs_super_num_devices (mdrestore.original_super) !=1 ) {struct btrfs_root *root;struct stat st;
Signature · 1 line
2.26.2
PATCH 6/6 btrfs-progs: image: Pin down log tree blocks before fixup
In reply to Qu WenruoAlthough btrfs-image will dump log tree, we will modify the restored image if it's not a multi-device restore.
In that case, since log tree blocks are not recorded in extent tree, extent allocator will try to re-use the tree blocks belonging to log trees, this would lead to transid mismatch if btrfs-restore chooses to do device/chunk fixup.
This patch will fix such problem by pinning down all the log trees blocks before fixing up the device and chunk trees.
--- image/main.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+)
image/main.c +66 -0
@@ -2500,6 +2500,66 @@ out:return ret; } +static int iter_tree_blocks (struct btrfs_fs_info *fs_info, +struct extent_buffer *eb,bool pin) +{ +void (*func)(struct btrfs_fs_info *fs_info,u64 bytenr,u64 num_bytes); +int nritems; +int level; +int i; +int ret; + +if (pin) + func = btrfs_pin_extent; +else + func = btrfs_unpin_extent; + +func (fs_info, eb->start, eb->len); + + level =btrfs_header_level (eb); + nritems =btrfs_header_nritems (eb); +if (level ==0 ) +return 0 ; + +for (i =0 ; i < nritems; i++) { +u64 bytenr; +struct extent_buffer *tmp; + +if (level ==0 ) { +struct btrfs_root_item *ri; +struct btrfs_key key; + +btrfs_item_key_to_cpu (eb, &key, i); +if (key.type !=BTRFS_ROOT_ITEM_KEY ) +continue ; + ri = btrfs_item_ptr(eb, i, struct btrfs_root_item); + bytenr =btrfs_disk_root_bytenr (eb, ri); + tmp =read_tree_block (fs_info, bytenr,0 ); +if (!extent_buffer_uptodate (tmp)) { +error ("unable to read log root block" ); +return -EIO ; + } + ret =iter_tree_blocks (fs_info, tmp, pin); +free_extent_buffer (tmp); +if (ret) +return ret; + }else { + bytenr =btrfs_node_blockptr (eb, i); + tmp =read_tree_block (fs_info, bytenr,0 ); +if (!extent_buffer_uptodate (tmp)) { +error ("unable to read log root block" ); +return -EIO ; + } + ret =iter_tree_blocks (fs_info, tmp, pin); +free_extent_buffer (tmp); +if (ret) +return ret; + } + } + +return 0 ; +} +static int fixup_chunks_and_devices (struct btrfs_fs_info *fs_info,struct mdrestore_struct *mdres,int out_fd) {
@@ -2516,6 +2576,9 @@ static int fixup_chunks_and_devices(struct btrfs_fs_info *fs_info,return PTR_ERR (trans); } +if (btrfs_super_log_root (fs_info->super_copy) && + fs_info->log_root_tree) +iter_tree_blocks (fs_info, fs_info->log_root_tree->node, true);fixup_block_groups (trans); ret =fixup_dev_extents (trans);if (ret <0 )
@@ -2530,6 +2593,9 @@ static int fixup_chunks_and_devices(struct btrfs_fs_info *fs_info,error ("unable to commit transaction: %d" , ret);return ret; } +if (btrfs_super_log_root (fs_info->super_copy) && + fs_info->log_root_tree) +iter_tree_blocks (fs_info, fs_info->log_root_tree->node, false);return 0 ; error: errno = -ret;
Signature · 1 line
2.26.2
Quoted 21 lines · Qu Wenruo
This branch can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/image_fixesSince there are two binary files updates, and one big code move, it's recommended to fetch github repo, in case some patches didn't reach mail list.
This is inspried by one log tree replay dead loop bug, where the kind reporter, Pierre Abbat <[email protected]>, gave the btrfs-image to reproduce it.
Then the image fails to pass check due to the existing log tree conflicting with device/chunk fixup.
As log tree blocks are not recorded in extent tree, later COW can use log tree blocks and cause transid mismatch.To address the problem, this patchset will:
- Don't do any fixup if the source dump is single device Since the dump has the full super block contents, we can easily check if the source fs is single deivce.
Quoted 6 lines
The chunk/device fixup is mostly for older btrfs-image behavior, which always restores the fs into SINGLE profile.
However since commit9088ab6a1067("btrfs-progs: make btrfs-image restore to support dup"), btrfs-image can restore into DUP profile, allowing us to do exact replay for single device fs.
This is patch 5.
As expected, patch 3 can't survive the mail list filter. It's 402K, so I guess one needs to grab it from github anyway.
Thanks,
Qu
Quoted 43 lines
- Pin down all log tree blocks for fixup
For cases we still need to fixup chunk/device items, at least pin down all log tree blocks to avoid transid mimsatch.
This is patch 6.After above fixes, fsck/012 and fsck/035 fails, due to bad original images.
The old btrfs-image can fixup those bad devicetotal_bytesandbytes_used, but that just hides the problem.
We still need to update those images to make them correct, so here comes patch 3 and 4.During the debugging of btrfs-image restore, I found
dump_superblock()would help a lot to expose bad values in images, so isbtrfs_print_leaf().Enahance them to be more handy for usage inside gdb, and here comes patch 1 and 2.
Qu Wenruo (6):
btrfs-progs: Allowbtrfs_print_leaf()to be called on dummy eb whose
fs_infois NULL
btrfs-progs: print-tree: Exportdump_superblock()
btrfs-progs: fsck-tests: Update the image in 012
btrfs-progs: fsck-tests: Update the image of test case 035
btrfs-progs: image: Don't modify the chunk and device tree if the
source dump is single device
btrfs-progs: image: Pin down log tree blocks before fixupcmds/inspect-dump-super.c | 454 +----------------- image/main.c | 86 +++- print-tree.c | 449 ++++++++++++++++- print-tree.h | 1 + .../012-leaf-corruption/good.img.xz | Bin 0 -> 186392 bytes .../012-leaf-corruption/no_data_extent.tar.xz | Bin 130260 -> 0 bytes tests/fsck-tests/012-leaf-corruption/test.sh | 17 +- .../offset_by_one.img | Bin 3072 -> 3072 bytes 8 files changed, 540 insertions(+), 467 deletions(-) create mode 100644 tests/fsck-tests/012-leaf-corruption/good.img.xz delete mode 100644 tests/fsck-tests/012-leaf-corruption/no_data_extent.tar.xz
Quoted 29 lines · Qu Wenruo
Quoted 21 lines · Qu Wenruo
This branch can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/image_fixesSince there are two binary files updates, and one big code move, it's recommended to fetch github repo, in case some patches didn't reach mail list.
This is inspried by one log tree replay dead loop bug, where the kind reporter, Pierre Abbat <[email protected]>, gave the btrfs-image to reproduce it.
Then the image fails to pass check due to the existing log tree conflicting with device/chunk fixup.
As log tree blocks are not recorded in extent tree, later COW can use log tree blocks and cause transid mismatch.To address the problem, this patchset will:
- Don't do any fixup if the source dump is single device Since the dump has the full super block contents, we can easily check if the source fs is single deivce.Quoted 6 lines
The chunk/device fixup is mostly for older btrfs-image behavior, which always restores the fs into SINGLE profile.
However since commit9088ab6a1067("btrfs-progs: make btrfs-image restore to support dup"), btrfs-image can restore into DUP profile, allowing us to do exact replay for single device fs.
This is patch 5.
Quoted 2 lines
As expected, patch 3 can't survive the mail list filter. It's 402K, so I guess one needs to grab it from github anyway.
Thanks, patch picked from git and the whole patchset is now in devel.