btrfs-progs: Bad extent item generation related bug fixes
cover btrfs-progs: Bad extent item generation related bug fixes
There is an issue reported in github, where an fs get corrupted extent tree initialy, then I recommended --init-extent-tree.
Although --init-extent-tree indeed fixed the original problem, it caused new problems, quite a lot of EXTENT_ITEMs now get bad generation number, which failed to mount with v5.4.
The problem turns out to be a bug in backref repair code, which doesn't initialize extent_record::generation, causing garbage in EXTENT_ITEMs.
This patch will:
- Fix the problem
Patch 1
- Enhance EXTENT_ITEM generation repair
Patch 2
- Make btrfs check able to detect such bad generation Patch 3~4
- Add new test case for above ability
Patch 5
Qu Wenruo (5):
btrfs-progs: check: Initialize extent_record::generation member
btrfs-progs: check: Populate extent generation correctly for data
extents
btrfs-progs: check/lowmem: Detect invalid EXTENT_ITEM and EXTENT_DATA
generation
btrfs-progs: check/original: Detect invalid extent generation
btrfs-progs: fsck-tests: Make sure btrfs check can detect bad extent
item generation
check/main.c | 36 ++++++++++++++---- check/mode-lowmem.c | 19 +++++++++ .../bad_extent_item_gen.img.xz | Bin 0 -> 1916 bytes .../test.sh | 19 +++++++++ 4 files changed, 67 insertions(+), 7 deletions(-) create mode 100644 tests/fsck-tests/044-invalid-extent-item-generation/bad_extent_item_gen.img.xz create mode 100755 tests/fsck-tests/044-invalid-extent-item-generation/test.sh
Signature · 1 line
2.24.1
PATCH 1/5 btrfs-progs: check: Initialize extent_record::generation member
[BUG]
When using btrfs check --init-extent-tree, there is a pretty high chance that the result fs can't pass tree-checker:
BTRFS critical (device dm-3): corrupt leaf: block=5390336 slot=149 extent bytenr=20115456 len=4096 invalid generation, have 16384 expect (0, 360] BTRFS error (device dm-3): block=5390336 read time tree block corruption detected
BTRFS error (device dm-3): failed to read block groups: -5
BTRFS error (device dm-3): open_ctree failed
[CAUSE]
The result fs has a pretty screwed up EXTENT_ITEMs for data extents:
item 148 key (20111360 EXTENT_ITEM 4096) itemoff 8777 itemsize 53
refs 1 gen 0 flags DATA
extent data backref root FS_TREE objectid 841 offset 0 count 1
item 149 key (20115456 EXTENT_ITEM 4096) itemoff 8724 itemsize 53
refs 1 gen 16384 flags DATA
extent data backref root FS_TREE objectid 906 offset 0 count 1
Kernel tree-checker will accept 0 generation, but that 16384 generation is definitely going to trigger the alarm.
Looking into the code, it's add_extent_rec_nolookup() allocating a new extent_rec, but not copying all members from parameter @tmpl, resulting generation not properly initialized.
[FIX]
Just copy tmpl->generation in add_extent_rec_nolookup(). And since all call sites have set all members of @tmpl to 0 before add_extent_rec_nolookup(), we shouldn't get garbage values.
For the 0 generation problem, it will be solved in another patch.
Issue: 225 (Not the initial report, but extent tree rebuild result)
--- check/main.c | 1 + 1 file changed, 1 insertion(+)
check/main.c +1 -0
@@ -4605,6 +4605,7 @@ static int add_extent_rec_nolookup(struct cache_tree *extent_cache, rec->refs = tmpl->refs; rec->extent_item_refs = tmpl->extent_item_refs; rec->parent_generation = tmpl->parent_generation; + rec->generation = tmpl->generation;INIT_LIST_HEAD (&rec->backrefs);INIT_LIST_HEAD (&rec->dups);INIT_LIST_HEAD (&rec->list);
Signature · 1 line
2.24.1
PATCH 2/5 btrfs-progs: check: Populate extent generation correctly for data extents
In reply to Qu Wenruo[BUG]
When using btrfs check --init-extent-tree, we will create incorrect generation number for data extents in extent tree:
item 10 key (13631488 EXTENT_ITEM 1048576) itemoff 15828 itemsize 53
refs 1 gen 0 flags DATA
extent data backref root FS_TREE objectid 257 offset 0 count 1
[CAUSE]
Since data extent generation is not as obvious as tree blocks, which has header containing its generations, so for data extents, its extent_record::generation is not really updated, resulting such 0 generation.
[FIX]
To get generation of a data extent, there are two sources we can rely:
- EXTENT_ITEM
There is always a btrfs_extent_item::generation can be utilized. Although this is not possible for --init-extent-tree use case.
- EXTENT_DATA
We have btrfs_file_extent_item::generation for regular and preallocated data extents.
Since --init-extent-tree will go through subvolume trees, this would be the main source for extent data generation.
Then we only need to make add_data_backref() to accept @gen parameter, and pass it down to extent_record structure.
And for the final extent item generation update, here we add extra fallback values, if we can't find FILE_EXTENT items.
In that case, we just fall back to current transid.
With this modification, recreated data EXTENT_ITEM now has correct generation number:
item 10 key (13631488 EXTENT_ITEM 1048576) itemoff 15828 itemsize 53
refs 1 gen 6 flags DATA
extent data backref root FS_TREE objectid 257 offset 0 count 1
--- check/main.c | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-)
check/main.c +17 -7
@@ -4810,7 +4810,7 @@ static int add_tree_backref(struct cache_tree *extent_cache, u64 bytenr,static int add_data_backref (struct cache_tree *extent_cache,u64 bytenr,u64 parent,u64 root,u64 owner,u64 offset, -u32 num_refs,int found_ref,u64 max_size) +u32 num_refs,u64 gen,int found_ref,u64 max_size) {struct extent_record *rec;struct data_backref *back;
@@ -4826,6 +4826,7 @@ static int add_data_backref(struct cache_tree *extent_cache, u64 bytenr, tmpl.start = bytenr; tmpl.nr =1 ; tmpl.max_size = max_size; + tmpl.generation = gen; ret =add_extent_rec_nolookup (extent_cache, &tmpl);if (ret)
@@ -4840,6 +4841,8 @@ static int add_data_backref(struct cache_tree *extent_cache, u64 bytenr,if (rec->max_size < max_size) rec->max_size = max_size; +if (rec->generation < gen) + rec->generation = gen;/* * If found_ref is set then max_size is the real size and must match the * existing refs. So if we have already found a ref then we need to
@@ -5315,6 +5318,7 @@ static int process_extent_item(struct btrfs_root *root,u64 refs =0 ;u64 offset;u64 num_bytes; +u64 gen;int metadata =0 ;btrfs_item_key_to_cpu (eb, &key, slot);
@@ -5340,6 +5344,7 @@ static int process_extent_item(struct btrfs_root *root, ei = btrfs_item_ptr(eb, slot, struct btrfs_extent_item); refs =btrfs_extent_refs (eb, ei); + gen =btrfs_extent_generation (eb, ei);if (btrfs_extent_flags (eb, ei) &BTRFS_EXTENT_FLAG_TREE_BLOCK ) metadata =1 ;else
@@ -5362,6 +5367,7 @@ static int process_extent_item(struct btrfs_root *root, tmpl.metadata = metadata; tmpl.found_rec =1 ; tmpl.max_size = num_bytes; + tmpl.generation = gen;add_extent_rec (extent_cache, &tmpl); ptr = (unsigned long )(ei +1 );
@@ -5401,14 +5407,14 @@ static int process_extent_item(struct btrfs_root *root, dref),btrfs_extent_data_ref_offset (eb, dref),btrfs_extent_data_ref_count (eb, dref), - 0, num_bytes); + gen, 0, num_bytes);break ;case BTRFS_SHARED_DATA_REF_KEY : sref = (struct btrfs_shared_data_ref *)(iref +1 );add_data_backref (extent_cache, key.objectid, offset,0 ,0 ,0 ,btrfs_shared_data_ref_count (eb, sref), -0 , num_bytes); + gen,0 , num_bytes);break ;default : fprintf(stderr,
@@ -6352,7 +6358,7 @@ static int run_next_block(struct btrfs_root *root, ref),btrfs_extent_data_ref_offset (buf, ref),btrfs_extent_data_ref_count (buf, ref), - 0, root->fs_info->sectorsize); + 0, 0, root->fs_info->sectorsize);continue ; }if (key.type ==BTRFS_SHARED_DATA_REF_KEY ) {
@@ -6363,7 +6369,7 @@ static int run_next_block(struct btrfs_root *root,add_data_backref (extent_cache, key.objectid, key.offset,0 ,0 ,0 ,btrfs_shared_data_ref_count (buf, ref), -0 , root->fs_info->sectorsize); +0 ,0 , root->fs_info->sectorsize);continue ; }if (key.type ==BTRFS_ORPHAN_ITEM_KEY ) {
@@ -6403,7 +6409,8 @@ static int run_next_block(struct btrfs_root *root,add_data_backref (extent_cache,btrfs_file_extent_disk_bytenr (buf, fi), parent, owner, key.objectid, key.offset - -btrfs_file_extent_offset (buf, fi),1 ,1 , +btrfs_file_extent_offset (buf, fi),1 , +btrfs_file_extent_generation (buf, fi),1 ,btrfs_file_extent_disk_num_bytes (buf, fi)); } }else {
@@ -6706,7 +6713,10 @@ static int record_extent(struct btrfs_trans_handle *trans, struct btrfs_extent_item);btrfs_set_extent_refs (leaf, ei,0 ); -btrfs_set_extent_generation (leaf, ei, rec->generation); +if (rec->generation) +btrfs_set_extent_generation (leaf, ei, rec->generation); +else +btrfs_set_extent_generation (leaf, ei, trans->transid);if (back->is_data) { btrfs_set_extent_flags(leaf, ei,
Signature · 1 line
2.24.1
PATCH 3/5 btrfs-progs: check/lowmem: Detect invalid EXTENT_ITEM and EXTENT_DATA generation
In reply to Qu WenruoSince older btrfs check --init-extent-tree could cause invalid EXTENT_ITEM generation for data extents, add such check to lowmem mode check.
Also add such generation check to file extents too.
For the repair part, I don't have any good idea yet. So affected user may depend on --init-extent-tree again.
--- check/mode-lowmem.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
check/mode-lowmem.c +19 -0
@@ -2041,6 +2041,8 @@ static int check_file_extent(struct btrfs_root *root, struct btrfs_path *path,u64 csum_found;/* In byte size, sectorsize aligned */ u64 search_start;/* Logical range start we search for csum */ u64 search_len;/* Logical range len we search for csum */ +u64 gen; +u64 super_gen =btrfs_super_generation (root->fs_info->super_copy);unsigned int extent_type;unsigned int is_hole;int slot = path->slots[0 ];
@@ -2067,6 +2069,7 @@ static int check_file_extent(struct btrfs_root *root, struct btrfs_path *path,return check_file_extent_inline (root, path, size, end);/* Check REG_EXTENT/PREALLOC_EXTENT */ + gen =btrfs_file_extent_generation (node, fi); disk_bytenr =btrfs_file_extent_disk_bytenr (node, fi); disk_num_bytes =btrfs_file_extent_disk_num_bytes (node, fi); extent_num_bytes =btrfs_file_extent_num_bytes (node, fi);
@@ -2074,6 +2077,13 @@ static int check_file_extent(struct btrfs_root *root, struct btrfs_path *path, compressed =btrfs_file_extent_compression (node, fi); is_hole = (disk_bytenr ==0 ) && (disk_num_bytes ==0 ); +if (gen > super_gen +1 ) { +error ( +"vainlid file extent generation, have %llu expect (0, %llu]" , + gen, super_gen +1 ); + err |=INVALID_GENERATION ; + } +/* * Check EXTENT_DATA csum *
@@ -4152,8 +4162,10 @@ static int check_extent_item(struct btrfs_fs_info *fs_info,u64 parent;u64 num_bytes;u64 root_objectid; +u64 gen;u64 owner;u64 owner_offset; +u64 super_gen =btrfs_super_generation (fs_info->super_copy);int metadata =0 ;int level;struct btrfs_key key;
@@ -4183,6 +4195,13 @@ static int check_extent_item(struct btrfs_fs_info *fs_info, ei = btrfs_item_ptr(eb, slot, struct btrfs_extent_item); flags =btrfs_extent_flags (eb, ei); + gen =btrfs_extent_generation (eb, ei); +if (gen > super_gen +1 ) { +error ( +"invalid generation for extent %llu, have %llu expect (0, %llu]" , + key.objectid, gen, super_gen +1 ); + err |=INVALID_GENERATION ; + }if (flags &BTRFS_EXTENT_FLAG_TREE_BLOCK ) metadata =1 ;
Signature · 1 line
2.24.1
PATCH 4/5 btrfs-progs: check/original: Detect invalid extent generation
In reply to Qu WenruoMuch like what we have done in lowmem mode, also detect and report invalid extent generation in original mode.
Unlike lowmem mode, we have extent_record::generation, which is the higher number of generations in EXTENT_ITEM, EXTENT_DATA or tree block header, so there is no need to check generations in different locations.
For repair, we still need to depend on --init-extent-tree, as directly modifying extent items can easily cause conflicts with delayed refs, thus it should be avoided.
--- check/main.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
check/main.c +11 -0
@@ -4021,10 +4021,13 @@ static void free_extent_record_cache(struct cache_tree *extent_cache)static int maybe_free_extent_rec (struct cache_tree *extent_cache,struct extent_record *rec) { +u64 super_gen =btrfs_super_generation (global_info->super_copy); +if (rec->content_checked && rec->owner_ref_checked && rec->extent_item_refs == rec->refs && rec->refs >0 && rec->num_duplicates ==0 && !all_backpointers_checked (rec,0 ) && !rec->bad_full_backref && !rec->crossing_stripes && + rec->generation <= super_gen +1 && !rec->wrong_chunk_type) {remove_cache_extent (extent_cache, &rec->cache);free_all_extent_backrefs (rec);
@@ -7857,6 +7860,7 @@ static int check_extent_refs(struct btrfs_root *root, {struct extent_record *rec;struct cache_extent *cache; +u64 super_gen =btrfs_super_generation (root->fs_info->super_copy);int ret =0 ;int had_dups =0 ;int err =0 ;
@@ -7939,6 +7943,13 @@ static int check_extent_refs(struct btrfs_root *root, cur_err =1 ; } +if (rec->generation > super_gen +1 ) { +error ( +"invalid generation for extent %llu, have %llu expect (0, %llu]" , + rec->start, rec->generation, + super_gen +1 ); + cur_err =1 ; + }if (rec->refs != rec->extent_item_refs) { fprintf(stderr, "ref mismatch on [%llu %llu] ", (unsigned long long)rec->start,
Signature · 1 line
2.24.1
PATCH 5/5 btrfs-progs: fsck-tests: Make sure btrfs check can detect bad extent item generation
In reply to Qu WenruoThe new image has a bad data extent item generation:
item 0 key (13631488 EXTENT_ITEM 4096) itemoff 16230 itemsize 53
refs 1 gen 16384 flags DATA
extent data backref root FS_TREE objectid 257 offset 0 count 1
Just like what older btrfs check --init-extent-tree could cause.
--- .../bad_extent_item_gen.img.xz | Bin 0 -> 1916 bytes .../test.sh | 19 ++++++++++++++++++ 2 files changed, 19 insertions(+) create mode 100644 tests/fsck-tests/044-invalid-extent-item-generation/bad_extent_item_gen.img.xz create mode 100755 tests/fsck-tests/044-invalid-extent-item-generation/test.sh
tests/fsck-tests/044-invalid-extent-item-generation/bad_extent_item_gen.img.xz new file binary +0 -0
tests/fsck-tests/044-invalid-extent-item-generation/test.sh new file +19 -0
@@ -0,0 +1,19 @@ +#!/bin/bash +# +# Due to a bug in --init-extent-tree option, we may create bad generation +# number for data extents. +# +# This test case will ensure btrfs check can at least detect such problem, +# just like kernel tree-checker. + +source "$TEST_TOP/common" + +check_prereq btrfs + +check_image() { + run_mustfail \ + "btrfs check failed to detect invalid extent item generation" \ + "$TOP/btrfs" check "$1" +} + +check_all_images
Signature · 1 line
2.24.1
Quoted 23 lines · Qu Wenruo
There is an issue reported in github, where an fs get corrupted extent tree initialy, then I recommended --init-extent-tree.
Although --init-extent-tree indeed fixed the original problem, it caused new problems, quite a lot of
EXTENT_ITEMsnow get bad generation number, which failed to mount with v5.4.The problem turns out to be a bug in backref repair code, which doesn't initialize
extent_record::generation, causing garbage inEXTENT_ITEMs.This patch will:
- Fix the problem
Patch 1- Enhance
EXTENT_ITEMgeneration repair
Patch 2- Make
btrfs checkable to detect such bad generation Patch 3~4- Add new test case for above ability
Patch 5
Quoted 10 lines
Qu Wenruo (5):
btrfs-progs: check: Initializeextent_record::generationmember
btrfs-progs: check: Populate extent generation correctly for data
extents
btrfs-progs: check/lowmem: Detect invalidEXTENT_ITEMandEXTENT_DATA
generation
btrfs-progs: check/original: Detect invalid extent generation
btrfs-progs: fsck-tests: Make sure btrfs check can detect bad extent
item generation
Nice fixes.
Quoted 8 lines
check/main.c | 36 ++++++++++++++---- check/mode-lowmem.c | 19 +++++++++ .../bad_extent_item_gen.img.xz | Bin 0 -> 1916 bytes .../test.sh | 19 +++++++++ 4 files changed, 67 insertions(+), 7 deletions(-)create mode 100644 tests/fsck-tests/044-invalid-extent-item-generation/bad_extent_item_gen.img.xz create mode 100755
tests/fsck-tests/044-invalid-extent-item-generation/test.sh
Quoted 23 lines · Qu Wenruo
There is an issue reported in github, where an fs get corrupted extent tree initialy, then I recommended --init-extent-tree.
Although --init-extent-tree indeed fixed the original problem, it caused new problems, quite a lot of
EXTENT_ITEMsnow get bad generation number, which failed to mount with v5.4.The problem turns out to be a bug in backref repair code, which doesn't initialize
extent_record::generation, causing garbage inEXTENT_ITEMs.This patch will:
- Fix the problem
Patch 1- Enhance
EXTENT_ITEMgeneration repair
Patch 2- Make
btrfs checkable to detect such bad generation Patch 3~4- Add new test case for above ability
Patch 5
Quoted 9 lines
Qu Wenruo (5):
btrfs-progs: check: Initializeextent_record::generationmember
btrfs-progs: check: Populate extent generation correctly for data
extents
btrfs-progs: check/lowmem: Detect invalidEXTENT_ITEMandEXTENT_DATA
generation
btrfs-progs: check/original: Detect invalid extent generation
btrfs-progs: fsck-tests: Make sure btrfs check can detect bad extent
item generation
Thanks for the fixes, added to devel.