btrfs-progs: check: detect and repair ram_bytes mismatch for non-compressed data extents
cover btrfs-progs: check: detect and repair ram_bytes mismatch for non-compressed data extents
There are at least one kernel bug that makes on-disk btrfs_file_extent_item::ram_bytes incorrect for non-compressed non-hole data extents.
Thankfully kernel just doesn't care ram_bytes for non-compressed extents at all, so it doesn't lead to any data corruption or whatever, and this is really just a minor problem.
But for the sake of consistency and to follow the on-disk format, we should still detect and repair such problems.
This patchset would implement detection and repair for both lowmem and original mode, and a new hand crafted test case for it.
The reason why the test case is still handle crafted is, we do not have the btrfs-corrupt-block support for corrupting ram_bytes to a specific value yet.
I'd prefer to do the binary image migration to script in a dedicated patchset in the future.
Qu Wenruo (3):
btrfs-progs: check/lowmem: detect and repair mismatched ram_bytes
btrfs-progs: check/original: detect and repair ram_bytes mismatch
btrfs-progs: tests/fsck: add test case for ram_bytes detection and
repair
check/main.c | 126 +++++++++++++++++- check/mode-lowmem.c | 69 ++++++++++ check/mode-lowmem.h | 1 + check/mode-original.h | 8 ++ .../default.img.xz | Bin 0 -> 2076 bytes 5 files changed, 200 insertions(+), 4 deletions(-) create mode 100644 tests/fsck-tests/062-noncompressed-ram-bytes-mismatch/default.img.xz
--
2.45.0
PATCH 1/3 btrfs-progs: check/lowmem: detect and repair mismatched ram_bytes
For non-compressed non-hole file extent items, the ram_bytes should match disk_num_bytes.
But due to kernel bugs, we have several cases where ram_bytes is not correctly updated.
Thankfully this is really a very minor mismatch and can never cause data corruption since the kernel does not utilize ram_bytes for non-compressed extents at all.
So here we just detect and repair it for lowmem mode.
--- check/mode-lowmem.c | 69 +++++++++++++++++++++++++++++++++++++++++++++ check/mode-lowmem.h | 1 + 2 files changed, 70 insertions(+)
check/mode-lowmem.c +69 -0
@@ -2081,6 +2081,61 @@ static int check_file_extent_inline(struct btrfs_root *root,return err; } +static int repair_ram_bytes_mismatch (struct btrfs_root *root, +struct btrfs_path *path) +{ +struct btrfs_trans_handle *trans; +struct btrfs_key key; +struct btrfs_file_extent_item *fi; +u64 disk_num_bytes; +int recover_ret; +int ret; + +btrfs_item_key_to_cpu (path->nodes[0 ], &key, path->slots[0 ]); +btrfs_release_path (path); +UASSERT (key.type ==BTRFS_EXTENT_DATA_KEY ); + + trans =btrfs_start_transaction (root,1 ); +if (IS_ERR (trans)) { + ret =PTR_ERR (trans); + errno = -ret; +error_msg (ERROR_MSG_START_TRANS ,"%m" ); +return ret; + } + + ret =btrfs_search_slot (trans, root, &key, path,0 ,1 ); +/* Not really possible. */ +if (ret >0 ) { + ret = -ENOENT ; +btrfs_release_path (path); + goto recover; + } + +if (ret <0 ) + goto recover; + + fi =btrfs_item_ptr (path->nodes[0 ], path->slots[0 ], + struct btrfs_file_extent_item); + disk_num_bytes =btrfs_file_extent_disk_num_bytes (path->nodes[0 ], fi); +btrfs_set_file_extent_ram_bytes (path->nodes[0 ], fi, disk_num_bytes); +btrfs_mark_buffer_dirty (path->nodes[0 ]); + + ret =btrfs_commit_transaction (trans, root); +if (ret <0 ) { + errno = -ret; +error_msg (ERROR_MSG_COMMIT_TRANS ,"%m" ); + }else { +printf ( +"Successfully repaired ram_bytes for non-compressed extent at root %llu ino %llu file_pos %llu\n" , + root->objectid, key.objectid, key.offset); + } +return ret; +recover: + recover_ret =btrfs_search_slot (NULL , root, &key, path,0 ,0 ); +UASSERT (recover_ret ==0 ); +return ret; +} +/* * Check file extent datasum/hole, update the size of the file extents, * check and update the last offset of the file extent.
@@ -2106,6 +2161,7 @@ static int check_file_extent(struct btrfs_root *root, struct btrfs_path *path,u64 csum_found;/* In byte size, sectorsize aligned */ u64 search_start;/* Logical range start we search for csum */ u64 search_len;/* Logical range len we search for csum */ +u64 ram_bytes;u64 gen;u64 super_gen;unsigned int extent_type;
@@ -2140,6 +2196,7 @@ static int check_file_extent(struct btrfs_root *root, struct btrfs_path *path, extent_num_bytes =btrfs_file_extent_num_bytes (node, fi); extent_offset =btrfs_file_extent_offset (node, fi); compressed =btrfs_file_extent_compression (node, fi); + ram_bytes =btrfs_file_extent_ram_bytes (node, fi); is_hole = (disk_bytenr ==0 ) && (disk_num_bytes ==0 ); super_gen =btrfs_super_generation (gfs_info->super_copy);
@@ -2150,6 +2207,18 @@ static int check_file_extent(struct btrfs_root *root, struct btrfs_path *path, err |=INVALID_GENERATION ; } +if (!compressed && disk_bytenr && disk_num_bytes != ram_bytes) { +error ( +"minor ram_bytes mismatch for non-compressed data extents, have %llu expect %llu" , + ram_bytes, disk_num_bytes); +if (opt_check_repair) { + ret =repair_ram_bytes_mismatch (root, path); +if (ret <0 ) + err |=RAM_BYTES_MISMATCH ; + }else { + err |=RAM_BYTES_MISMATCH ; + } + }/* * Check EXTENT_DATA csum *
check/mode-lowmem.h +1 -0
@@ -47,6 +47,7 @@#define INODE_MODE_ERROR (1U << 25)/* Bad inode mode */ #define INVALID_GENERATION (1U << 26)/* Generation is too new */ #define SUPER_BYTES_USED_ERROR (1U << 27)/* Super bytes_used is invalid */ +#define RAM_BYTES_MISMATCH (1U << 27)/* Non-compressed extent has wrong ram_bytes */ /* * Error bit for low memory mode check.
Signature · 1 line
2.45.0
PATCH 2/3 btrfs-progs: check/original: detect and repair ram_bytes mismatch
In reply to Qu WenruoFor non-compressed non-hole file extent items, the ram_bytes should match disk_num_bytes.
But due to kernel bugs, we have several cases where ram_bytes is not correctly updated.
Thankfully this is really a very minor mismatch and can never cause data corruption since the kernel does not utilize ram_bytes for non-compressed extents at all.
So here we just detect and repair it for original mode.
--- check/main.c | 126 ++++++++++++++++++++++++++++++++++++++++-- check/mode-original.h | 8 +++ 2 files changed, 130 insertions(+), 4 deletions(-)
check/main.c +122 -4
@@ -493,6 +493,33 @@ static int device_record_compare(const struct rb_node *node1, const struct rb_noreturn 0 ; } +static int add_mismatch_ram_bytes_record (struct inode_record *inode_rec, +struct btrfs_key *key) +{ +struct mismatch_ram_bytes_record *record; + + record =malloc (sizeof (*record)); +if (!record) { +error_msg (ERROR_MSG_MEMORY ,"mismatch ram bytes record" ); +return -ENOMEM ; + } +memcpy (&record->key, key,sizeof (*key)); +list_add_tail (&record->list, &inode_rec->mismatch_ram_bytes); +return 0 ; +} + +static void free_mismatch_ram_bytes_records (struct inode_record *inode_rec) +{ +if (!list_empty (&inode_rec->mismatch_ram_bytes)) { +struct mismatch_ram_bytes_record *ram; + + ram =list_entry (inode_rec->mismatch_ram_bytes.next, + struct mismatch_ram_bytes_record, list); +list_del (&ram->list); +free (ram); + } +} +static struct inode_record *clone_inode_rec (struct inode_record *orig_rec) {struct inode_record *rec;
@@ -501,6 +528,7 @@ static struct inode_record *clone_inode_rec(struct inode_record *orig_rec)struct inode_backref *tmp;struct mismatch_dir_hash_record *hash_record;struct mismatch_dir_hash_record *new_record; +struct mismatch_ram_bytes_record *ram_record;struct unaligned_extent_rec_t *src;struct unaligned_extent_rec_t *dst;struct rb_node *rb;
@@ -514,6 +542,7 @@ static struct inode_record *clone_inode_rec(struct inode_record *orig_rec) rec->refs =1 ;INIT_LIST_HEAD (&rec->backrefs);INIT_LIST_HEAD (&rec->mismatch_dir_hash); +INIT_LIST_HEAD (&rec->mismatch_ram_bytes);INIT_LIST_HEAD (&rec->unaligned_extent_recs); rec->holes =RB_ROOT ;
@@ -537,6 +566,11 @@ static struct inode_record *clone_inode_rec(struct inode_record *orig_rec)memcpy (&new_record, hash_record, size);list_add_tail (&new_record->list, &rec->mismatch_dir_hash); } +list_for_each_entry (ram_record, &orig_rec->mismatch_ram_bytes, list) { + ret =add_mismatch_ram_bytes_record (rec, &ram_record->key); +if (ret <0 ) + goto cleanup; + }list_for_each_entry (src, &orig_rec->unaligned_extent_recs, list) { size =sizeof (*src); dst =malloc (size);
@@ -578,6 +612,7 @@ cleanup:free (hash_record); } } +free_mismatch_ram_bytes_records (rec);if (!list_empty (&rec->unaligned_extent_recs))list_for_each_entry_safe (src, dst, &rec->unaligned_extent_recs, list) {
@@ -619,6 +654,8 @@ static void print_inode_error(struct btrfs_root *root, struct inode_record *rec)fprintf (stderr,", odd file extent" );if (errors &I_ERR_BAD_FILE_EXTENT )fprintf (stderr,", bad file extent" ); +if (errors &I_ERR_RAM_BYTES_MISMATCH ) +fprintf (stderr,", bad ram bytes for non-compressed extents" );if (errors &I_ERR_FILE_EXTENT_OVERLAP )fprintf (stderr,", file extent overlap" );if (errors &I_ERR_FILE_EXTENT_TOO_LARGE )
@@ -637,8 +674,6 @@ static void print_inode_error(struct btrfs_root *root, struct inode_record *rec)fprintf (stderr,", link count wrong" );if (errors &I_ERR_ODD_INODE_FLAGS )fprintf (stderr,", odd inode flags" ); -if (errors &I_ERR_INLINE_RAM_BYTES_WRONG ) -fprintf (stderr,", invalid inline ram bytes" );if (errors &I_ERR_INVALID_IMODE )fprintf (stderr,", invalid inode mode bit 0%o" , rec->imode & ~07777 );
@@ -699,6 +734,17 @@ static void print_inode_error(struct btrfs_root *root, struct inode_record *rec) hash_record->key.offset); } } +if (errors &I_ERR_RAM_BYTES_MISMATCH ) { +struct mismatch_ram_bytes_record *ram_record; + +fprintf (stderr, +"Non-compressed file extents with invalid ram_bytes (minor errors):\n" ); +list_for_each_entry (ram_record, &rec->mismatch_ram_bytes, list) { +fprintf (stderr,"\tino=%llu offset=%llu\n" , + ram_record->key.objectid, + ram_record->key.offset); + } + } }static void print_ref_error (int errors)
@@ -760,6 +806,7 @@ static struct inode_record *get_inode_rec(struct cache_tree *inode_cache, rec->refs =1 ;INIT_LIST_HEAD (&rec->backrefs);INIT_LIST_HEAD (&rec->mismatch_dir_hash); +INIT_LIST_HEAD (&rec->mismatch_ram_bytes);INIT_LIST_HEAD (&rec->unaligned_extent_recs); rec->holes =RB_ROOT ;
@@ -811,6 +858,14 @@ static void free_inode_rec(struct inode_record *rec)list_del (&backref->list);free (backref); } +while (!list_empty (&rec->mismatch_ram_bytes)) { +struct mismatch_ram_bytes_record *ram; + + ram =list_entry (rec->mismatch_ram_bytes.next, + struct mismatch_ram_bytes_record, list); +list_del (&ram->list); +free (ram); + }list_for_each_entry_safe (hash, next, &rec->mismatch_dir_hash, list)free (hash);free_unaligned_extent_recs (&rec->unaligned_extent_recs);
@@ -821,7 +876,8 @@ static void free_inode_rec(struct inode_record *rec)static bool can_free_inode_rec (struct inode_record *rec) {if (!rec->errors && rec->checked && rec->found_inode_item && - rec->nlink == rec->found_link &&list_empty (&rec->backrefs)) + rec->nlink == rec->found_link &&list_empty (&rec->backrefs) && +list_empty (&rec->mismatch_ram_bytes))return true;return false; }
@@ -1742,6 +1798,14 @@ static int process_file_extent(struct btrfs_root *root, rec->errors |=I_ERR_BAD_FILE_EXTENT ;if (compression && rec->nodatasum) rec->errors |=I_ERR_BAD_FILE_EXTENT ; +if (disk_bytenr && !compression && +btrfs_file_extent_ram_bytes (eb, fi) != +btrfs_file_extent_disk_num_bytes (eb, fi)) { + rec->errors |=I_ERR_RAM_BYTES_MISMATCH ; + ret =add_mismatch_ram_bytes_record (rec, key); +if (ret <0 ) +return ret; + }if (disk_bytenr >0 ) rec->found_size += num_bytes; }else {
@@ -3044,6 +3108,57 @@ static int repair_inode_gen_original(struct btrfs_trans_handle *trans,return 0 ; } +static int repair_ram_bytes (struct btrfs_trans_handle *trans, +struct btrfs_root *root, +struct btrfs_path *path, +struct inode_record *rec) +{ +struct mismatch_ram_bytes_record *record; +struct mismatch_ram_bytes_record *tmp; +int ret =0 ; + +btrfs_release_path (path); +list_for_each_entry_safe (record, tmp, &rec->mismatch_ram_bytes, list) { +struct btrfs_file_extent_item *fi; +struct extent_buffer *leaf; +int type; +int slot; +int search_ret; + + search_ret =btrfs_search_slot (trans, root, &record->key, path,0 ,1 ); +if (search_ret >0 ) + search_ret = -ENOENT ; +if (search_ret <0 ) { + ret = search_ret; +btrfs_release_path (path); +continue ; + } + leaf = path->nodes[0 ]; + slot = path->slots[0 ]; + fi = btrfs_item_ptr(leaf, slot, struct btrfs_file_extent_item); + type =btrfs_file_extent_type (leaf, fi); +if (type !=BTRFS_FILE_EXTENT_REG && + type !=BTRFS_FILE_EXTENT_PREALLOC ) { + ret = -EUCLEAN ; +btrfs_release_path (path); +continue ; + } +if (btrfs_file_extent_disk_bytenr (path->nodes[0 ], fi) ==0 || +btrfs_file_extent_compression (path->nodes[0 ], fi)) { + ret = -EUCLEAN ; +btrfs_release_path (path); +continue ; + } +btrfs_set_file_extent_ram_bytes (leaf, fi, +btrfs_file_extent_disk_num_bytes (leaf, fi)); +btrfs_mark_buffer_dirty (leaf); +btrfs_release_path (path); + } +if (!ret) + rec->errors &= ~I_ERR_RAM_BYTES_MISMATCH ; +return ret; +} +static int try_repair_inode (struct btrfs_root *root,struct inode_record *rec) {struct btrfs_trans_handle *trans;
@@ -3066,7 +3181,8 @@ static int try_repair_inode(struct btrfs_root *root, struct inode_record *rec)I_ERR_MISMATCH_DIR_HASH |I_ERR_UNALIGNED_EXTENT_REC |I_ERR_INVALID_IMODE | - I_ERR_INVALID_GEN))) +I_ERR_INVALID_GEN | + I_ERR_RAM_BYTES_MISMATCH))) return rec->errors;/*
@@ -3106,6 +3222,8 @@ static int try_repair_inode(struct btrfs_root *root, struct inode_record *rec) ret =repair_unaligned_extent_recs (trans, root, &path, rec);if (!ret && rec->errors &I_ERR_INVALID_GEN ) ret =repair_inode_gen_original (trans, root, &path, rec); +if (!ret && rec->errors &I_ERR_RAM_BYTES_MISMATCH ) + ret =repair_ram_bytes (trans, root, &path, rec);btrfs_release_path (&path); ret =btrfs_commit_transaction (trans, root);if (ret <0 ) {
check/mode-original.h +8 -0
@@ -189,6 +189,8 @@ struct unaligned_extent_rec_t {#define I_ERR_INVALID_GEN (1U << 20)#define I_ERR_INVALID_NLINK (1U << 21)#define I_ERR_INVALID_XATTR (1U << 22) +/* Ram_bytes mismatch for non-compressed data extents. */ +#define I_ERR_RAM_BYTES_MISMATCH (1U << 23)struct inode_record {struct list_head backrefs;
@@ -216,6 +218,7 @@ struct inode_record {u64 extent_end;struct rb_root holes;struct list_head mismatch_dir_hash; +struct list_head mismatch_ram_bytes;u32 refs; };
@@ -232,6 +235,11 @@ struct mismatch_dir_hash_record {/* namebuf follows here */ }; +struct mismatch_ram_bytes_record { +struct list_head list; +struct btrfs_key key; +}; +struct root_backref {struct list_head list;unsigned int found_dir_item:1 ;
Signature · 1 line
2.45.0
PATCH 3/3 btrfs-progs: tests/fsck: add test case for ram_bytes detection and repair
In reply to Qu WenruoThe new test case would contain a file system image, with the following file:
item 4 key (257 INODE_ITEM 0) itemoff 15883 itemsize 160 generation 7 transid 8 size 16384 nbytes 16384 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 258 flags 0x0(none) atime 1714635006.328482575 (2024-05-02 17:00:06) ctime 1714635013.394980640 (2024-05-02 17:00:13) mtime 1714635013.394980640 (2024-05-02 17:00:13) otime 1714635006.328482575 (2024-05-02 17:00:06) item 5 key (257 INODE_REF 256) itemoff 15869 itemsize 14 index 2 namelen 4 name: file item 6 key (257 EXTENT_DATA 0) itemoff 15816 itemsize 53 generation 7 type 1 (regular) extent data disk byte 13631488 nr 1048576 extent data offset 0 nr 16384 ram 16384 extent compression 0 (none)
Note the ram bytes, which should be 1048576.
Furthermore, the inode size is trucnated to 16K (originally 1M), so that offset + num_bytes would still be no larger than ram_bytes. So the only error is the mismatch between ram_bytes and disk_num_bytes for the non-compressed data extent.
The image is hand crafted for now, as btrfs-corrupt-block doesn not yet support corrupting the ram_bytes field.
--- .../default.img.xz | Bin 0 -> 2076 bytes 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 tests/fsck-tests/062-noncompressed-ram-bytes-mismatch/default.img.xz
tests/fsck-tests/062-noncompressed-ram-bytes-mismatch/default.img.xz new file binary +0 -0
Signature · 1 line
2.45.0
Quoted 14 lines · Qu Wenruo
There are at least one kernel bug that makes on-disk
btrfs_file_extent_item::ram_bytesincorrect for non-compressed non-hole data extents.Thankfully kernel just doesn't care
ram_bytesfor non-compressed extents at all, so it doesn't lead to any data corruption or whatever, and this is really just a minor problem.But for the sake of consistency and to follow the on-disk format, we should still detect and repair such problems.
This patchset would implement detection and repair for both lowmem and original mode, and a new hand crafted test case for it.
Quoted 6 lines
The reason why the test case is still handle crafted is, we do not have the btrfs-corrupt-block support for corrupting
ram_bytesto a specific value yet.I'd prefer to do the binary image migration to script in a dedicated patchset in the future.
The crafted binary images are ok for now, I've verified it by restoring, did tree dump and check.
There's an issue tracking the other images audit,
https://github.com/kdave/btrfs-progs/issues/772, I did only a quick pass but a tool that will do the restore/check and maybe some other steps will be added to the repository.
Patches added to devel, thanks.