Skip to contentSkip to conversation

btrfs-progs: btrfs-image related fixes

PATCH In review Qu Wenruo 2 participants 8 messages May 27, 2020 – May 28, 2020 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

cover btrfs-progs: btrfs-image related fixes

This branch can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/image_fixes

Since there are two binary files updates, and one big code move, it's recommended to fetch github repo, in case some patches didn't reach mail list.

This is inspried by one log tree replay dead loop bug, where the kind reporter, Pierre Abbat <[email protected]>, gave the btrfs-image to reproduce it.

Then the image fails to pass check due to the existing log tree conflicting with device/chunk fixup.
As log tree blocks are not recorded in extent tree, later COW can use log tree blocks and cause transid mismatch.

To address the problem, this patchset will:
- Don't do any fixup if the source dump is single device Since the dump has the full super block contents, we can easily check if the source fs is single deivce.

The chunk/device fixup is mostly for older btrfs-image behavior, which always restores the fs into SINGLE profile.
However since commit 9088ab6a1067 ("btrfs-progs: make btrfs-image restore to support dup"), btrfs-image can restore into DUP profile, allowing us to do exact replay for single device fs.
This is patch 5.

- Pin down all log tree blocks for fixup
For cases we still need to fixup chunk/device items, at least pin down all log tree blocks to avoid transid mimsatch.
This is patch 6.

After above fixes, fsck/012 and fsck/035 fails, due to bad original images.
The old btrfs-image can fixup those bad device total_bytes and bytes_used, but that just hides the problem.
We still need to update those images to make them correct, so here comes patch 3 and 4.

During the debugging of btrfs-image restore, I found dump_superblock() would help a lot to expose bad values in images, so is btrfs_print_leaf().

Enahance them to be more handy for usage inside gdb, and here comes patch 1 and 2.

Qu Wenruo (6):
btrfs-progs: Allow btrfs_print_leaf() to be called on dummy eb whose
fs_info is NULL
btrfs-progs: print-tree: Export dump_superblock()
btrfs-progs: fsck-tests: Update the image in 012
btrfs-progs: fsck-tests: Update the image of test case 035
btrfs-progs: image: Don't modify the chunk and device tree if the
source dump is single device
btrfs-progs: image: Pin down log tree blocks before fixup

 cmds/inspect-dump-super.c                     | 454 +-----------------
 image/main.c                                  |  86 +++-
 print-tree.c                                  | 449 ++++++++++++++++-
 print-tree.h                                  |   1 +
 .../012-leaf-corruption/good.img.xz           | Bin 0 -> 186392 bytes
 .../012-leaf-corruption/no_data_extent.tar.xz | Bin 130260 -> 0 bytes
 tests/fsck-tests/012-leaf-corruption/test.sh  |  17 +-
 .../offset_by_one.img                         | Bin 3072 -> 3072 bytes
 8 files changed, 540 insertions(+), 467 deletions(-)
 create mode 100644 tests/fsck-tests/012-leaf-corruption/good.img.xz
 delete mode 100644 tests/fsck-tests/012-leaf-corruption/no_data_extent.tar.xz
Signature · 1 line
2.26.2
author
#1 lore raw

PATCH 1/6 btrfs-progs: Allow btrfs_print_leaf() to be called on dummy eb whose fs_info is NULL

In certain call sites, like btrfs-image, we want full extent_buffer accessors, but don't have a btrfs_fs_info.

In that case, if an extent buffer whose fs_info is NULL, then btrfs_print_leaf() is called on such eb, we will trigger a NULL pointer dereference, at BTRFS_LEAF_DATA_SIZE(fs_info).

Fix this by using __BTRFS_LEAF_DATA_SIZE(), which only needs nodesize, and nodesize can be extracted from extent_buffer::len.

This allows us to call btrfs_print_leaf() on any extent buffer, which is super handy in gdb.

---
 print-tree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
print-tree.c +1 -1
index 27acadb22205..38cf8d34ac2e 100644
@@ -1201,7 +1201,7 @@ void btrfs_print_leaf(struct extent_buffer *eb)
 	struct btrfs_item *item;
 	struct btrfs_disk_key disk_key;
 	char flags_str[128];
-	u32 leaf_data_size = BTRFS_LEAF_DATA_SIZE(fs_info);
+	u32 leaf_data_size = __BTRFS_LEAF_DATA_SIZE(eb->len);
 	u32 i;
 	u32 nr;
 	u64 flags;
Signature · 1 line
2.26.2
author
#2 lore raw

PATCH 2/6 btrfs-progs: print-tree: Export dump_superblock()

In reply to Qu Wenruo

dump_superblock() is useful to debug btrfs-image error, like fsck/012-* test case, where the superblock itself has something wrong from the original image.

Export it so that we can call it in gdb.

Since we're exporting dump_superblock(), rename it to btrfs_print_superblock() to following the existing naming schema.

---
 cmds/inspect-dump-super.c | 454 +-------------------------------------
 print-tree.c              | 447 +++++++++++++++++++++++++++++++++++++
 print-tree.h              |   1 +
 3 files changed, 449 insertions(+), 453 deletions(-)
cmds/inspect-dump-super.c +1 -453
index f22633b99390..729af84cc93d 100644
@@ -16,471 +16,19 @@
 
 #include "kerncompat.h"
 #include <stdio.h>
-#include <stdlib.h>
 #include <unistd.h>
 #include <fcntl.h>
 #include <sys/stat.h>
-#include <ctype.h>
-#include <uuid/uuid.h>
 #include <errno.h>
 #include <getopt.h>
 
 #include "ctree.h"
 #include "disk-io.h"
 #include "print-tree.h"
-#include "transaction.h"
-#include "kernel-lib/list.h"
 #include "common/utils.h"
 #include "cmds/commands.h"
-#include "crypto/crc32c.h"
 #include "common/help.h"
 
-static int check_csum_sblock(void *sb, int csum_size, u16 csum_type)
-{
-	u8 result[BTRFS_CSUM_SIZE];
-
-	btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
-			result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-
-	return !memcmp(sb, result, csum_size);
-}
-
-static void print_sys_chunk_array(struct btrfs_super_block *sb)
-{
-	struct extent_buffer *buf;
-	struct btrfs_disk_key *disk_key;
-	struct btrfs_chunk *chunk;
-	u8 *array_ptr;
-	unsigned long sb_array_offset;
-	u32 num_stripes;
-	u32 array_size;
-	u32 len = 0;
-	u32 cur_offset;
-	struct btrfs_key key;
-	int item;
-
-	buf = malloc(sizeof(*buf) + sizeof(*sb));
-	if (!buf) {
-		error("not enough memory");
-		return;
-	}
-	write_extent_buffer(buf, sb, 0, sizeof(*sb));
-	buf->len = sizeof(*sb);
-	array_size = btrfs_super_sys_array_size(sb);
-
-	array_ptr = sb->sys_chunk_array;
-	sb_array_offset = offsetof(struct btrfs_super_block, sys_chunk_array);
-
-	if (array_size > BTRFS_SYSTEM_CHUNK_ARRAY_SIZE) {
-		error("sys_array_size %u shouldn't exceed %u bytes",
-				array_size, BTRFS_SYSTEM_CHUNK_ARRAY_SIZE);
-		goto out;
-	}
-
-	cur_offset = 0;
-	item = 0;
-
-	while (cur_offset < array_size) {
-		disk_key = (struct btrfs_disk_key *)array_ptr;
-		len = sizeof(*disk_key);
-		if (cur_offset + len > array_size)
-			goto out_short_read;
-
-		btrfs_disk_key_to_cpu(&key, disk_key);
-
-		array_ptr += len;
-		sb_array_offset += len;
-		cur_offset += len;
-
-		printf("\titem %d ", item);
-		btrfs_print_key(disk_key);
-		putchar('\n');
-
-		if (key.type == BTRFS_CHUNK_ITEM_KEY) {
-			chunk = (struct btrfs_chunk *)sb_array_offset;
-			/*
-			 * At least one btrfs_chunk with one stripe must be
-			 * present, exact stripe count check comes afterwards
-			 */
-			len = btrfs_chunk_item_size(1);
-			if (cur_offset + len > array_size)
-				goto out_short_read;
-
-			num_stripes = btrfs_chunk_num_stripes(buf, chunk);
-			if (!num_stripes) {
-				error(
-			"invalid number of stripes %u in sys_array at offset %u",
-					num_stripes, cur_offset);
-				break;
-			}
-			len = btrfs_chunk_item_size(num_stripes);
-			if (cur_offset + len > array_size)
-				goto out_short_read;
-			print_chunk_item(buf, chunk);
-		} else {
-			error("unexpected item type %u in sys_array at offset %u",
-				(u32)key.type, cur_offset);
-			break;
-		}
-		array_ptr += len;
-		sb_array_offset += len;
-		cur_offset += len;
-
-		item++;
-	}
-
-out:
-	free(buf);
-	return;
-
-out_short_read:
-	error("sys_array too short to read %u bytes at offset %u",
-			len, cur_offset);
-	free(buf);
-}
-
-static int empty_backup(struct btrfs_root_backup *backup)
-{
-	if (backup == NULL ||
-		(backup->tree_root == 0 &&
-		 backup->tree_root_gen == 0))
-		return 1;
-	return 0;
-}
-
-static void print_root_backup(struct btrfs_root_backup *backup)
-{
-	printf("\t\tbackup_tree_root:\t%llu\tgen: %llu\tlevel: %d\n",
-			btrfs_backup_tree_root(backup),
-			btrfs_backup_tree_root_gen(backup),
-			btrfs_backup_tree_root_level(backup));
-	printf("\t\tbackup_chunk_root:\t%llu\tgen: %llu\tlevel: %d\n",
-			btrfs_backup_chunk_root(backup),
-			btrfs_backup_chunk_root_gen(backup),
-			btrfs_backup_chunk_root_level(backup));
-	printf("\t\tbackup_extent_root:\t%llu\tgen: %llu\tlevel: %d\n",
-			btrfs_backup_extent_root(backup),
-			btrfs_backup_extent_root_gen(backup),
-			btrfs_backup_extent_root_level(backup));
-	printf("\t\tbackup_fs_root:\t\t%llu\tgen: %llu\tlevel: %d\n",
-			btrfs_backup_fs_root(backup),
-			btrfs_backup_fs_root_gen(backup),
-			btrfs_backup_fs_root_level(backup));
-	printf("\t\tbackup_dev_root:\t%llu\tgen: %llu\tlevel: %d\n",
-			btrfs_backup_dev_root(backup),
-			btrfs_backup_dev_root_gen(backup),
-			btrfs_backup_dev_root_level(backup));
-	printf("\t\tbackup_csum_root:\t%llu\tgen: %llu\tlevel: %d\n",
-			btrfs_backup_csum_root(backup),
-			btrfs_backup_csum_root_gen(backup),
-			btrfs_backup_csum_root_level(backup));
-
-	printf("\t\tbackup_total_bytes:\t%llu\n",
-					btrfs_backup_total_bytes(backup));
-	printf("\t\tbackup_bytes_used:\t%llu\n",
-					btrfs_backup_bytes_used(backup));
-	printf("\t\tbackup_num_devices:\t%llu\n",
-					btrfs_backup_num_devices(backup));
-	putchar('\n');
-}
-
-static void print_backup_roots(struct btrfs_super_block *sb)
-{
-	struct btrfs_root_backup *backup;
-	int i;
-
-	for (i = 0; i < BTRFS_NUM_BACKUP_ROOTS; i++) {
-		backup = sb->super_roots + i;
-		if (!empty_backup(backup)) {
-			printf("\tbackup %d:\n", i);
-			print_root_backup(backup);
-		}
-	}
-}
-
-struct readable_flag_entry {
-	u64 bit;
-	char *output;
-};
-
-#define DEF_COMPAT_RO_FLAG_ENTRY(bit_name)		\
-	{BTRFS_FEATURE_COMPAT_RO_##bit_name, #bit_name}
-
-static struct readable_flag_entry compat_ro_flags_array[] = {
-	DEF_COMPAT_RO_FLAG_ENTRY(FREE_SPACE_TREE),
-	DEF_COMPAT_RO_FLAG_ENTRY(FREE_SPACE_TREE_VALID),
-};
-static const int compat_ro_flags_num = sizeof(compat_ro_flags_array) /
-				       sizeof(struct readable_flag_entry);
-
-#define DEF_INCOMPAT_FLAG_ENTRY(bit_name)		\
-	{BTRFS_FEATURE_INCOMPAT_##bit_name, #bit_name}
-
-static struct readable_flag_entry incompat_flags_array[] = {
-	DEF_INCOMPAT_FLAG_ENTRY(MIXED_BACKREF),
-	DEF_INCOMPAT_FLAG_ENTRY(DEFAULT_SUBVOL),
-	DEF_INCOMPAT_FLAG_ENTRY(MIXED_GROUPS),
-	DEF_INCOMPAT_FLAG_ENTRY(COMPRESS_LZO),
-	DEF_INCOMPAT_FLAG_ENTRY(COMPRESS_ZSTD),
-	DEF_INCOMPAT_FLAG_ENTRY(BIG_METADATA),
-	DEF_INCOMPAT_FLAG_ENTRY(EXTENDED_IREF),
-	DEF_INCOMPAT_FLAG_ENTRY(RAID56),
-	DEF_INCOMPAT_FLAG_ENTRY(SKINNY_METADATA),
-	DEF_INCOMPAT_FLAG_ENTRY(NO_HOLES),
-	DEF_INCOMPAT_FLAG_ENTRY(METADATA_UUID),
-	DEF_INCOMPAT_FLAG_ENTRY(RAID1C34),
-};
-static const int incompat_flags_num = sizeof(incompat_flags_array) /
-				      sizeof(struct readable_flag_entry);
-
-#define DEF_HEADER_FLAG_ENTRY(bit_name)			\
-	{BTRFS_HEADER_FLAG_##bit_name, #bit_name}
-#define DEF_SUPER_FLAG_ENTRY(bit_name)			\
-	{BTRFS_SUPER_FLAG_##bit_name, #bit_name}
-
-static struct readable_flag_entry super_flags_array[] = {
-	DEF_HEADER_FLAG_ENTRY(WRITTEN),
-	DEF_HEADER_FLAG_ENTRY(RELOC),
-	DEF_SUPER_FLAG_ENTRY(CHANGING_FSID),
-	DEF_SUPER_FLAG_ENTRY(SEEDING),
-	DEF_SUPER_FLAG_ENTRY(METADUMP),
-	DEF_SUPER_FLAG_ENTRY(METADUMP_V2)
-};
-static const int super_flags_num = ARRAY_SIZE(super_flags_array);
-
-#define BTRFS_SUPER_FLAG_SUPP	(BTRFS_HEADER_FLAG_WRITTEN |\
-				 BTRFS_HEADER_FLAG_RELOC |\
-				 BTRFS_SUPER_FLAG_CHANGING_FSID |\
-				 BTRFS_SUPER_FLAG_SEEDING |\
-				 BTRFS_SUPER_FLAG_METADUMP |\
-				 BTRFS_SUPER_FLAG_METADUMP_V2)
-
-static void __print_readable_flag(u64 flag, struct readable_flag_entry *array,
-				  int array_size, u64 supported_flags)
-{
-	int i;
-	int first = 1;
-	struct readable_flag_entry *entry;
-
-	if (!flag)
-		return;
-
-	printf("\t\t\t( ");
-	for (i = 0; i < array_size; i++) {
-		entry = array + i;
-		if (flag & entry->bit) {
-			if (first)
-				printf("%s ", entry->output);
-			else
-				printf("|\n\t\t\t  %s ", entry->output);
-			first = 0;
-		}
-	}
-	flag &= ~supported_flags;
-	if (flag) {
-		if (first)
-			printf("unknown flag: 0x%llx ", flag);
-		else
-			printf("|\n\t\t\t  unknown flag: 0x%llx ", flag);
-	}
-	printf(")\n");
-}
-
-static void print_readable_compat_ro_flag(u64 flag)
-{
-	/*
-	 * We know about the FREE_SPACE_TREE{,_VALID} bits, but we don't
-	 * actually support them yet.
-	 */
-	return __print_readable_flag(flag, compat_ro_flags_array,
-				     compat_ro_flags_num,
-				     BTRFS_FEATURE_COMPAT_RO_SUPP |
-				     BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE |
-				     BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE_VALID);
-}
-
-static void print_readable_incompat_flag(u64 flag)
-{
-	return __print_readable_flag(flag, incompat_flags_array,
-				     incompat_flags_num,
-				     BTRFS_FEATURE_INCOMPAT_SUPP);
-}
-
-static void print_readable_super_flag(u64 flag)
-{
-	return __print_readable_flag(flag, super_flags_array,
-				     super_flags_num, BTRFS_SUPER_FLAG_SUPP);
-}
-
-static bool is_valid_csum_type(u16 csum_type)
-{
-	switch (csum_type) {
-	case BTRFS_CSUM_TYPE_CRC32:
-	case BTRFS_CSUM_TYPE_XXHASH:
-	case BTRFS_CSUM_TYPE_SHA256:
-	case BTRFS_CSUM_TYPE_BLAKE2:
-		return true;
-	default:
-		return false;
-	}
-}
-
-static void dump_superblock(struct btrfs_super_block *sb, int full)
-{
-	int i;
-	char *s, buf[BTRFS_UUID_UNPARSED_SIZE];
-	u8 *p;
-	u32 csum_size;
-	u16 csum_type;
-	bool metadata_uuid_present = (btrfs_super_incompat_flags(sb) &
-		BTRFS_FEATURE_INCOMPAT_METADATA_UUID);
-	int cmp_res = 0;
-
-
-	csum_type = btrfs_super_csum_type(sb);
-	csum_size = BTRFS_CSUM_SIZE;
-	printf("csum_type\t\t%hu (", csum_type);
-	if (!is_valid_csum_type(csum_type)) {
-		printf("INVALID");
-	} else {
-		printf("%s", btrfs_super_csum_name(csum_type));
-		csum_size = btrfs_super_csum_size(sb);
-	}
-	printf(")\n");
-	printf("csum_size\t\t%llu\n", (unsigned long long)csum_size);
-
-	printf("csum\t\t\t0x");
-	for (i = 0, p = sb->csum; i < csum_size; i++)
-		printf("%02x", p[i]);
-	if (!is_valid_csum_type(csum_type))
-		printf(" [UNKNOWN CSUM TYPE OR SIZE]");
-	else if (check_csum_sblock(sb, csum_size, csum_type))
-		printf(" [match]");
-	else
-		printf(" [DON'T MATCH]");
-	putchar('\n');
-
-	printf("bytenr\t\t\t%llu\n",
-		(unsigned long long)btrfs_super_bytenr(sb));
-	printf("flags\t\t\t0x%llx\n",
-		(unsigned long long)btrfs_super_flags(sb));
-	print_readable_super_flag(btrfs_super_flags(sb));
-
-	printf("magic\t\t\t");
-	s = (char *) &sb->magic;
-	for (i = 0; i < 8; i++)
-		putchar(isprint(s[i]) ? s[i] : '.');
-	if (btrfs_super_magic(sb) == BTRFS_MAGIC)
-		printf(" [match]\n");
-	else
-		printf(" [DON'T MATCH]\n");
-
-	uuid_unparse(sb->fsid, buf);
-	printf("fsid\t\t\t%s\n", buf);
-	if (metadata_uuid_present) {
-		uuid_unparse(sb->metadata_uuid, buf);
-		printf("metadata_uuid\t\t%s\n", buf);
-	} else {
-		printf("metadata_uuid\t\t%s\n", buf);
-	}
-
-	printf("label\t\t\t");
-	s = sb->label;
-	for (i = 0; i < BTRFS_LABEL_SIZE && s[i]; i++)
-		putchar(isprint(s[i]) ? s[i] : '.');
-	putchar('\n');
-
-	printf("generation\t\t%llu\n",
-	       (unsigned long long)btrfs_super_generation(sb));
-	printf("root\t\t\t%llu\n", (unsigned long long)btrfs_super_root(sb));
-	printf("sys_array_size\t\t%llu\n",
-	       (unsigned long long)btrfs_super_sys_array_size(sb));
-	printf("chunk_root_generation\t%llu\n",
-	       (unsigned long long)btrfs_super_chunk_root_generation(sb));
-	printf("root_level\t\t%llu\n",
-	       (unsigned long long)btrfs_super_root_level(sb));
-	printf("chunk_root\t\t%llu\n",
-	       (unsigned long long)btrfs_super_chunk_root(sb));
-	printf("chunk_root_level\t%llu\n",
-	       (unsigned long long)btrfs_super_chunk_root_level(sb));
-	printf("log_root\t\t%llu\n",
-	       (unsigned long long)btrfs_super_log_root(sb));
-	printf("log_root_transid\t%llu\n",
-	       (unsigned long long)btrfs_super_log_root_transid(sb));
-	printf("log_root_level\t\t%llu\n",
-	       (unsigned long long)btrfs_super_log_root_level(sb));
-	printf("total_bytes\t\t%llu\n",
-	       (unsigned long long)btrfs_super_total_bytes(sb));
-	printf("bytes_used\t\t%llu\n",
-	       (unsigned long long)btrfs_super_bytes_used(sb));
-	printf("sectorsize\t\t%llu\n",
-	       (unsigned long long)btrfs_super_sectorsize(sb));
-	printf("nodesize\t\t%llu\n",
-	       (unsigned long long)btrfs_super_nodesize(sb));
-	printf("leafsize (deprecated)\t%u\n",
-	       le32_to_cpu(sb->__unused_leafsize));
-	printf("stripesize\t\t%llu\n",
-	       (unsigned long long)btrfs_super_stripesize(sb));
-	printf("root_dir\t\t%llu\n",
-	       (unsigned long long)btrfs_super_root_dir(sb));
-	printf("num_devices\t\t%llu\n",
-	       (unsigned long long)btrfs_super_num_devices(sb));
-	printf("compat_flags\t\t0x%llx\n",
-	       (unsigned long long)btrfs_super_compat_flags(sb));
-	printf("compat_ro_flags\t\t0x%llx\n",
-	       (unsigned long long)btrfs_super_compat_ro_flags(sb));
-	print_readable_compat_ro_flag(btrfs_super_compat_ro_flags(sb));
-	printf("incompat_flags\t\t0x%llx\n",
-	       (unsigned long long)btrfs_super_incompat_flags(sb));
-	print_readable_incompat_flag(btrfs_super_incompat_flags(sb));
-	printf("cache_generation\t%llu\n",
-	       (unsigned long long)btrfs_super_cache_generation(sb));
-	printf("uuid_tree_generation\t%llu\n",
-	       (unsigned long long)btrfs_super_uuid_tree_generation(sb));
-
-	uuid_unparse(sb->dev_item.uuid, buf);
-	printf("dev_item.uuid\t\t%s\n", buf);
-
-	uuid_unparse(sb->dev_item.fsid, buf);
-	if (metadata_uuid_present) {
-		cmp_res = !memcmp(sb->dev_item.fsid, sb->metadata_uuid,
-				 BTRFS_FSID_SIZE);
-	} else {
-		cmp_res = !memcmp(sb->dev_item.fsid, sb->fsid, BTRFS_FSID_SIZE);
-	}
-	printf("dev_item.fsid\t\t%s %s\n", buf,
-	       cmp_res ? "[match]" : "[DON'T MATCH]");
-
-	printf("dev_item.type\t\t%llu\n", (unsigned long long)
-	       btrfs_stack_device_type(&sb->dev_item));
-	printf("dev_item.total_bytes\t%llu\n", (unsigned long long)
-	       btrfs_stack_device_total_bytes(&sb->dev_item));
-	printf("dev_item.bytes_used\t%llu\n", (unsigned long long)
-	       btrfs_stack_device_bytes_used(&sb->dev_item));
-	printf("dev_item.io_align\t%u\n", (unsigned int)
-	       btrfs_stack_device_io_align(&sb->dev_item));
-	printf("dev_item.io_width\t%u\n", (unsigned int)
-	       btrfs_stack_device_io_width(&sb->dev_item));
-	printf("dev_item.sector_size\t%u\n", (unsigned int)
-	       btrfs_stack_device_sector_size(&sb->dev_item));
-	printf("dev_item.devid\t\t%llu\n",
-	       btrfs_stack_device_id(&sb->dev_item));
-	printf("dev_item.dev_group\t%u\n", (unsigned int)
-	       btrfs_stack_device_group(&sb->dev_item));
-	printf("dev_item.seek_speed\t%u\n", (unsigned int)
-	       btrfs_stack_device_seek_speed(&sb->dev_item));
-	printf("dev_item.bandwidth\t%u\n", (unsigned int)
-	       btrfs_stack_device_bandwidth(&sb->dev_item));
-	printf("dev_item.generation\t%llu\n", (unsigned long long)
-	       btrfs_stack_device_generation(&sb->dev_item));
-	if (full) {
-		printf("sys_chunk_array[%d]:\n", BTRFS_SYSTEM_CHUNK_ARRAY_SIZE);
-		print_sys_chunk_array(sb);
-		printf("backup_roots[%d]:\n", BTRFS_NUM_BACKUP_ROOTS);
-		print_backup_roots(sb);
-	}
-}
-
 static int load_and_dump_sb(char *filename, int fd, u64 sb_bytenr, int full,
 		int force)
 {
@@ -507,7 +55,7 @@ static int load_and_dump_sb(char *filename, int fd, u64 sb_bytenr, int full,
 		error("bad magic on superblock on %s at %llu",
 				filename, (unsigned long long)sb_bytenr);
 	} else {
-		dump_superblock(sb, full);
+		btrfs_print_superblock(sb, full);
 	}
 	return 0;
 }
print-tree.c +447 -0

456 lines not on this page · raw message

print-tree.h +1 -0
index 92ed5fb7c270..0ce8d92fa305 100644
@@ -39,4 +39,5 @@ void print_chunk_item(struct extent_buffer *eb, struct btrfs_chunk *chunk);
 void print_extent_item(struct extent_buffer *eb, int slot, int metadata);
 void print_objectid(FILE *stream, u64 objectid, u8 type);
 void print_key_type(FILE *stream, u64 objectid, u8 type);
+void btrfs_print_superblock(struct btrfs_super_block *sb, int full);
 #endif
Signature · 1 line
2.26.2
author
#3 lore raw

PATCH 4/6 btrfs-progs: fsck-tests: Update the image of test case 035

In reply to Qu Wenruo

The image used in test case 035 has one problem:
- Bad total_bytes of device item in chunk tree
item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
devid 1 total_bytes 72564736 bytes_used 92274688
The total_bytes is even smaller than bytes_used.
While the dev_item in super block contains the correct value:

    dev_item.total_bytes    134217728
    dev_item.bytes_used     92274688

This patch will update the image, using the correct value in super block to replace the wrong one in chunk tree.

So that later btrfs-image restore without device/chunk fixup can still pass fsck-tests.

---
 .../offset_by_one.img                         | Bin 3072 -> 3072 bytes
 1 file changed, 0 insertions(+), 0 deletions(-)
tests/fsck-tests/035-inline-bad-ram-bytes/offset_by_one.img binary +0 -0
index 2f58208eed71d5e0ee053d825d2780fe6842a3db..b3cef56ec9d73a6ea397c2084c219999552b7145 100644
GIT binary patch
Signature · 1 line
2.26.2
author
#4 lore raw

PATCH 5/6 btrfs-progs: image: Don't modify the chunk and device tree if the source dump is single device

In reply to Qu Wenruo

Btrfs-image -r doesn't always create exactly the same fs of the original fs, this is because btrfs-image can create dump from multi-device, and restore it to single device.

Thus we need some special modification to chunk and device tree. This behavior is mostly to handle the old behavior where we always restore the metadata into SINGLE profile no matter what.

However this is not needed if the source fs only has one device, in that case, we can use all the metadata as is, no need to modify the fs at all, resulting an exact copy of metadata.

This patch will do extra check when doing restore, to avoid modify the restored fs if the source is also single device.

---
 image/main.c | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)
image/main.c +19 -1
index 10f6182e0eac..48d8fcd5078c 100644
@@ -103,6 +103,7 @@ struct mdrestore_struct {
 	struct rb_root physical_tree;
 	struct list_head list;
 	struct list_head overlapping_chunks;
+	struct btrfs_super_block *original_super;
 	size_t num_items;
 	u32 nodesize;
 	u64 devid;
@@ -1085,6 +1086,12 @@ static int update_super(struct mdrestore_struct *mdres, u8 *buffer)
 	u8 *ptr, *write_ptr;
 	int old_num_stripes;
 
+	/* No need to fix, use all data as is */
+	if (btrfs_super_num_devices(mdres->original_super) == 1) {
+		new_array_size = btrfs_super_sys_array_size(super);
+		goto finish;
+	}
+
 	write_ptr = ptr = super->sys_chunk_array;
 	array_size = btrfs_super_sys_array_size(super);
 
@@ -1135,6 +1142,7 @@ static int update_super(struct mdrestore_struct *mdres, u8 *buffer)
 		cur += btrfs_chunk_item_size(old_num_stripes);
 	}
 
+finish:
 	if (mdres->clear_space_cache)
 		btrfs_set_super_cache_generation(super, 0);
 
@@ -1203,6 +1211,8 @@ static int fixup_chunk_tree_block(struct mdrestore_struct *mdres,
 	u64 bytenr = async->start;
 	int i;
 
+	if (btrfs_super_num_devices(mdres->original_super) == 1)
+		return 0;
 	if (size_left % mdres->nodesize)
 		return 0;
 
@@ -1373,6 +1383,8 @@ static void *restore_worker(void *data)
 
 		if (!mdres->multi_devices) {
 			if (async->start == BTRFS_SUPER_INFO_OFFSET) {
+				memcpy(mdres->original_super, outbuf,
+				       BTRFS_SUPER_INFO_SIZE);
 				if (mdres->old_restore) {
 					update_super_old(outbuf);
 				} else {
@@ -1474,6 +1486,7 @@ static void mdrestore_destroy(struct mdrestore_struct *mdres, int num_threads)
 
 	pthread_cond_destroy(&mdres->cond);
 	pthread_mutex_destroy(&mdres->mutex);
+	free(mdres->original_super);
 }
 
 static int mdrestore_init(struct mdrestore_struct *mdres,
@@ -1500,6 +1513,10 @@ static int mdrestore_init(struct mdrestore_struct *mdres,
 	mdres->last_physical_offset = 0;
 	mdres->alloced_chunks = 0;
 
+	mdres->original_super = malloc(BTRFS_SUPER_INFO_SIZE);
+	if (!mdres->original_super)
+		return -ENOMEM;
+
 	if (!num_threads)
 		return 0;
 
@@ -2606,7 +2623,8 @@ static int restore_metadump(const char *input, FILE *out, int old_restore,
 	}
 	ret = wait_for_worker(&mdrestore);
 
-	if (!ret && !multi_devices && !old_restore) {
+	if (!ret && !multi_devices && !old_restore &&
+	    btrfs_super_num_devices(mdrestore.original_super) != 1) {
 		struct btrfs_root *root;
 		struct stat st;
 
Signature · 1 line
2.26.2
author
#5 lore raw

PATCH 6/6 btrfs-progs: image: Pin down log tree blocks before fixup

In reply to Qu Wenruo

Although btrfs-image will dump log tree, we will modify the restored image if it's not a multi-device restore.

In that case, since log tree blocks are not recorded in extent tree, extent allocator will try to re-use the tree blocks belonging to log trees, this would lead to transid mismatch if btrfs-restore chooses to do device/chunk fixup.

This patch will fix such problem by pinning down all the log trees blocks before fixing up the device and chunk trees.

---
 image/main.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)
image/main.c +66 -0
index 48d8fcd5078c..71ea131c8c29 100644
@@ -2500,6 +2500,66 @@ out:
 	return ret;
 }
 
+static int iter_tree_blocks(struct btrfs_fs_info *fs_info,
+			    struct extent_buffer *eb, bool pin)
+{
+	void (*func)(struct btrfs_fs_info *fs_info, u64 bytenr, u64 num_bytes);
+	int nritems;
+	int level;
+	int i;
+	int ret;
+
+	if (pin)
+		func = btrfs_pin_extent;
+	else
+		func = btrfs_unpin_extent;
+
+	func(fs_info, eb->start, eb->len);
+
+	level = btrfs_header_level(eb);
+	nritems = btrfs_header_nritems(eb);
+	if (level == 0)
+		return 0;
+
+	for (i = 0; i < nritems; i++) {
+		u64 bytenr;
+		struct extent_buffer *tmp;
+
+		if (level == 0) {
+			struct btrfs_root_item *ri;
+			struct btrfs_key key;
+
+			btrfs_item_key_to_cpu(eb, &key, i);
+			if (key.type != BTRFS_ROOT_ITEM_KEY)
+				continue;
+			ri = btrfs_item_ptr(eb, i, struct btrfs_root_item);
+			bytenr = btrfs_disk_root_bytenr(eb, ri);
+			tmp = read_tree_block(fs_info, bytenr, 0);
+			if (!extent_buffer_uptodate(tmp)) {
+				error("unable to read log root block");
+				return -EIO;
+			}
+			ret = iter_tree_blocks(fs_info, tmp, pin);
+			free_extent_buffer(tmp);
+			if (ret)
+				return ret;
+		} else {
+			bytenr = btrfs_node_blockptr(eb, i);
+			tmp = read_tree_block(fs_info, bytenr, 0);
+			if (!extent_buffer_uptodate(tmp)) {
+				error("unable to read log root block");
+				return -EIO;
+			}
+			ret = iter_tree_blocks(fs_info, tmp, pin);
+			free_extent_buffer(tmp);
+			if (ret)
+				return ret;
+		}
+	}
+
+	return 0;
+}
+
 static int fixup_chunks_and_devices(struct btrfs_fs_info *fs_info,
 			 struct mdrestore_struct *mdres, int out_fd)
 {
@@ -2516,6 +2576,9 @@ static int fixup_chunks_and_devices(struct btrfs_fs_info *fs_info,
 		return PTR_ERR(trans);
 	}
 
+	if (btrfs_super_log_root(fs_info->super_copy) &&
+	    fs_info->log_root_tree)
+		iter_tree_blocks(fs_info, fs_info->log_root_tree->node, true);
 	fixup_block_groups(trans);
 	ret = fixup_dev_extents(trans);
 	if (ret < 0)
@@ -2530,6 +2593,9 @@ static int fixup_chunks_and_devices(struct btrfs_fs_info *fs_info,
 		error("unable to commit transaction: %d", ret);
 		return ret;
 	}
+	if (btrfs_super_log_root(fs_info->super_copy) &&
+	    fs_info->log_root_tree)
+		iter_tree_blocks(fs_info, fs_info->log_root_tree->node, false);
 	return 0;
 error:
 	errno = -ret;
Signature · 1 line
2.26.2
author
#6 lore raw
In reply to Qu Wenruo
Quoted 21 lines · Qu Wenruo

This branch can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/image_fixes

Since there are two binary files updates, and one big code move, it's recommended to fetch github repo, in case some patches didn't reach mail list.

This is inspried by one log tree replay dead loop bug, where the kind reporter, Pierre Abbat <[email protected]>, gave the btrfs-image to reproduce it.

Then the image fails to pass check due to the existing log tree conflicting with device/chunk fixup.
As log tree blocks are not recorded in extent tree, later COW can use log tree blocks and cause transid mismatch.

To address the problem, this patchset will:
- Don't do any fixup if the source dump is single device Since the dump has the full super block contents, we can easily check if the source fs is single deivce.

Quoted 6 lines

The chunk/device fixup is mostly for older btrfs-image behavior, which always restores the fs into SINGLE profile.
However since commit 9088ab6a1067 ("btrfs-progs: make btrfs-image restore to support dup"), btrfs-image can restore into DUP profile, allowing us to do exact replay for single device fs.
This is patch 5.

As expected, patch 3 can't survive the mail list filter. It's 402K, so I guess one needs to grab it from github anyway.

Thanks,
Qu

Quoted 43 lines

- Pin down all log tree blocks for fixup
For cases we still need to fixup chunk/device items, at least pin down all log tree blocks to avoid transid mimsatch.
This is patch 6.

After above fixes, fsck/012 and fsck/035 fails, due to bad original images.
The old btrfs-image can fixup those bad device total_bytes and bytes_used, but that just hides the problem.
We still need to update those images to make them correct, so here comes patch 3 and 4.

During the debugging of btrfs-image restore, I found dump_superblock() would help a lot to expose bad values in images, so is btrfs_print_leaf().

Enahance them to be more handy for usage inside gdb, and here comes patch 1 and 2.

Qu Wenruo (6):
btrfs-progs: Allow btrfs_print_leaf() to be called on dummy eb whose
fs_info is NULL
btrfs-progs: print-tree: Export dump_superblock()
btrfs-progs: fsck-tests: Update the image in 012
btrfs-progs: fsck-tests: Update the image of test case 035
btrfs-progs: image: Don't modify the chunk and device tree if the
source dump is single device
btrfs-progs: image: Pin down log tree blocks before fixup

 cmds/inspect-dump-super.c                     | 454 +-----------------
 image/main.c                                  |  86 +++-
 print-tree.c                                  | 449 ++++++++++++++++-
 print-tree.h                                  |   1 +
 .../012-leaf-corruption/good.img.xz           | Bin 0 -> 186392 bytes
 .../012-leaf-corruption/no_data_extent.tar.xz | Bin 130260 -> 0 bytes
 tests/fsck-tests/012-leaf-corruption/test.sh  |  17 +-
 .../offset_by_one.img                         | Bin 3072 -> 3072 bytes
 8 files changed, 540 insertions(+), 467 deletions(-)
 create mode 100644 tests/fsck-tests/012-leaf-corruption/good.img.xz
 delete mode 100644 tests/fsck-tests/012-leaf-corruption/no_data_extent.tar.xz
#7 lore raw
Quoted 29 lines · Qu Wenruo
Quoted 21 lines · Qu Wenruo

This branch can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/image_fixes

Since there are two binary files updates, and one big code move, it's recommended to fetch github repo, in case some patches didn't reach mail list.

This is inspried by one log tree replay dead loop bug, where the kind reporter, Pierre Abbat <[email protected]>, gave the btrfs-image to reproduce it.

Then the image fails to pass check due to the existing log tree conflicting with device/chunk fixup.
As log tree blocks are not recorded in extent tree, later COW can use log tree blocks and cause transid mismatch.

To address the problem, this patchset will:
- Don't do any fixup if the source dump is single device Since the dump has the full super block contents, we can easily check if the source fs is single deivce.

Quoted 6 lines

The chunk/device fixup is mostly for older btrfs-image behavior, which always restores the fs into SINGLE profile.
However since commit 9088ab6a1067 ("btrfs-progs: make btrfs-image restore to support dup"), btrfs-image can restore into DUP profile, allowing us to do exact replay for single device fs.
This is patch 5.

Quoted 2 lines

As expected, patch 3 can't survive the mail list filter. It's 402K, so I guess one needs to grab it from github anyway.

Thanks, patch picked from git and the whole patchset is now in devel.