btrfs-progs: Add check and repair for invalid inode generation
cover btrfs-progs: Add check and repair for invalid inode generation
We have at least two user reports about bad inode generation makes kernel reject the fs.
According to the creation time, the inode is created by some 2014 kernel.
And the generation member of INODE_ITEM is not updated (unlike the transid member) so the error persists until latest tree-checker detects.
Even the situation can be fixed by reverting back to older kernel and copying the offending dir/file to another inode and delete the offending one, it still should be done by btrfs-progs.
This patchset adds such check and repair ability to btrfs-check, with a simple test image.
Qu Wenruo (3):
btrfs-progs: check/lowmem: Add check and repair for invalid inode
generation
btrfs-progs: check/original: Add check and repair for invalid inode
generation
btrfs-progs: fsck-tests: Add test image for invalid inode generation
repair
check/main.c | 50 +++++++++++- check/mode-lowmem.c | 76 ++++++++++++++++++ check/mode-original.h | 1 + .../.lowmem_repairable | 0 .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes 5 files changed, 126 insertions(+), 1 deletion(-) create mode 100644 tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable create mode 100644 tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
Signature · 1 line
2.23.0
PATCH 1/3 btrfs-progs: check/lowmem: Add check and repair for invalid inode generation
There are at least two bug reports of kernel tree-checker complaining about invalid inode generation.
All offending inodes seem to be caused by old kernel around 2014, with inode generation overflow.
So add such check and repair ability to lowmem mode check first.
This involves:
- Calculate the inode generation upper limit
If it's an inode from log tree, then the upper limit is super_generation + 1, otherwise it's super_generation.
- Check if the inode generation is larger than the upper limit
- Repair by resetting inode generation to current transaction generation
--- check/mode-lowmem.c | 76 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+)
check/mode-lowmem.c +76 -0
@@ -2472,6 +2472,59 @@ static bool has_orphan_item(struct btrfs_root *root, u64 ino)return false; } +static int repair_inode_gen_lowmem (struct btrfs_root *root, +struct btrfs_path *path) +{ +struct btrfs_trans_handle *trans; +struct btrfs_inode_item *ii; +struct btrfs_key key; +u64 transid; +int ret; + + trans =btrfs_start_transaction (root,1 ); +if (IS_ERR (trans)) { + ret =PTR_ERR (trans); + errno = -ret; +error ("failed to start transaction for inode gen repair: %m" ); +return ret; + } + transid = trans->transid; +btrfs_item_key_to_cpu (path->nodes[0 ], &key, path->slots[0 ]); +ASSERT (key.type ==BTRFS_INODE_ITEM_KEY ); + +btrfs_release_path (path); + + ret =btrfs_search_slot (trans, root, &key, path,0 ,1 ); +if (ret >0 ) { + ret = -ENOENT ; +error ("no inode item found for ino %llu" , key.objectid); + goto error; + } +if (ret <0 ) { + errno = -ret; +error ("failed to find inode item for ino %llu: %m" , + key.objectid); + goto error; + } + ii =btrfs_item_ptr (path->nodes[0 ], path->slots[0 ], + struct btrfs_inode_item); +btrfs_set_inode_generation (path->nodes[0 ], ii, trans->transid); +btrfs_mark_buffer_dirty (path->nodes[0 ]); + ret =btrfs_commit_transaction (trans, root); +if (ret <0 ) { + errno = -ret; +error ("failed to commit transaction: %m" ); + goto error; + } +printf ("reseting inode generation to %llu for ino %llu\n" , + transid, key.objectid); +return ret; + +error: +btrfs_abort_transaction (trans, ret); +return ret; +} +/* * Check INODE_ITEM and related ITEMs (the same inode number) * 1. check link count
@@ -2487,6 +2540,7 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path)struct btrfs_inode_item *ii;struct btrfs_key key;struct btrfs_key last_key; +struct btrfs_super_block *super = root->fs_info->super_copy;u64 inode_id;u32 mode;u64 flags;
@@ -2497,6 +2551,8 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path)u64 refs =0 ;u64 extent_end =0 ;u64 extent_size =0 ; +u64 generation; +u64 gen_uplimit;unsigned int dir;unsigned int nodatasum;bool is_orphan = false;
@@ -2527,6 +2583,7 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path) flags =btrfs_inode_flags (node, ii); dir =imode_to_type (mode) ==BTRFS_FT_DIR ; nlink =btrfs_inode_nlink (node, ii); + generation =btrfs_inode_generation (node, ii); nodatasum =btrfs_inode_flags (node, ii) &BTRFS_INODE_NODATASUM ;if (!is_valid_imode (mode)) {
@@ -2540,6 +2597,25 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path) } } +if (btrfs_super_log_root (super) !=0 && + root->objectid ==BTRFS_TREE_LOG_OBJECTID ) + gen_uplimit =btrfs_super_generation (super) +1 ; +else + gen_uplimit =btrfs_super_generation (super); + +if (generation > gen_uplimit) { +error ( +"invalid inode generation for ino %llu, have %llu expect [0, %llu)" , + inode_id, generation, gen_uplimit); +if (repair) { + ret =repair_inode_gen_lowmem (root, path); +if (ret <0 ) + err |=INVALID_GENERATION ; + }else { + err |=INVALID_GENERATION ; + } + + }if (S_ISLNK (mode) && flags & (BTRFS_INODE_IMMUTABLE |BTRFS_INODE_APPEND )) { err |=INODE_FLAGS_ERROR ;
Signature · 1 line
2.23.0
Re: [PATCH 1/3] btrfs-progs: check/lowmem: Add check and repair for invalid inode generation
Quoted 13 lines · Qu Wenruo
There are at least two bug reports of kernel tree-checker complaining about invalid inode generation.
All offending inodes seem to be caused by old kernel around 2014, with inode generation overflow.
So add such check and repair ability to lowmem mode check first.
This involves:
- Calculate the inode generation upper limit
If it's an inode from log tree, then the upper limit issuper_generation+ 1, otherwise it'ssuper_generation.
Quoted 7 lines
- Check if the inode generation is larger than the upper limit
- Repair by resetting inode generation to current transaction generation
Reported-by Charles Wright <[email protected]>Signed-off-by Qu Wenruo <[email protected]>
There is one small nit with the assert once rectified you can add:
Quoted 22 lines
--- check/mode-lowmem.c | 76 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+)check/mode-lowmem.c +10
-0@@ -2472,6 +2472,59 @@ static bool has_orphan_item(struct btrfs_root *root, u64 ino)return false; } +static int repair_inode_gen_lowmem (struct btrfs_root *root, +struct btrfs_path *path) +{ +struct btrfs_trans_handle *trans; +struct btrfs_inode_item *ii; +struct btrfs_key key; +u64 transid; +int ret; + + trans =btrfs_start_transaction (root,1 );
check/mode-lowmem.c static bool has_orphan_item(struct btrfs_root *root, u64 ino) · line 2494
Quoted 10 lines
check/mode-lowmem.c +9
-0+if (IS_ERR (trans)) { + ret =PTR_ERR (trans); + errno = -ret; +error ("failed to start transaction for inode gen repair: %m" ); +return ret; + } + transid = trans->transid; +btrfs_item_key_to_cpu (path->nodes[0 ], &key, path->slots[0 ]); +ASSERT (key.type ==BTRFS_INODE_ITEM_KEY );
nit: This function's sole caller, check_inode_item, is guaranteed to be called with a path pointing to BTRFS_INODE_ITEM_KEY thanks to the logic in the 'for' loop in process_one_leaf. This renders the assert redundant. At the very least I think it should be moved to check_inode_item.
Quoted 89 lines
check/mode-lowmem.c +34
-0+ +btrfs_release_path (path); + + ret =btrfs_search_slot (trans, root, &key, path,0 ,1 ); +if (ret >0 ) { + ret = -ENOENT ; +error ("no inode item found for ino %llu" , key.objectid); + goto error; + } +if (ret <0 ) { + errno = -ret; +error ("failed to find inode item for ino %llu: %m" , + key.objectid); + goto error; + } + ii =btrfs_item_ptr (path->nodes[0 ], path->slots[0 ], + struct btrfs_inode_item); +btrfs_set_inode_generation (path->nodes[0 ], ii, trans->transid); +btrfs_mark_buffer_dirty (path->nodes[0 ]); + ret =btrfs_commit_transaction (trans, root); +if (ret <0 ) { + errno = -ret; +error ("failed to commit transaction: %m" ); + goto error; + } +printf ("reseting inode generation to %llu for ino %llu\n" , + transid, key.objectid); +return ret; + +error: +btrfs_abort_transaction (trans, ret); +return ret; +} +/* * Check INODE_ITEM and related ITEMs (the same inode number) * 1. check link count
check/mode-lowmem.c +23
-0@@ -2487,6 +2540,7 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path)struct btrfs_inode_item *ii;struct btrfs_key key;struct btrfs_key last_key; +struct btrfs_super_block *super = root->fs_info->super_copy;u64 inode_id;u32 mode;u64 flags;@@ -2497,6 +2551,8 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path)u64 refs =0 ;u64 extent_end =0 ;u64 extent_size =0 ; +u64 generation; +u64 gen_uplimit;unsigned int dir;unsigned int nodatasum;bool is_orphan = false;@@ -2527,6 +2583,7 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path) flags =btrfs_inode_flags (node, ii); dir =imode_to_type (mode) ==BTRFS_FT_DIR ; nlink =btrfs_inode_nlink (node, ii); + generation =btrfs_inode_generation (node, ii); nodatasum =btrfs_inode_flags (node, ii) &BTRFS_INODE_NODATASUM ;if (!is_valid_imode (mode)) {@@ -2540,6 +2597,25 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path) } } +if (btrfs_super_log_root (super) !=0 && + root->objectid ==BTRFS_TREE_LOG_OBJECTID ) + gen_uplimit =btrfs_super_generation (super) +1 ; +else + gen_uplimit =btrfs_super_generation (super); + +if (generation > gen_uplimit) { +error ( +"invalid inode generation for ino %llu, have %llu expect [0, %llu)" , + inode_id, generation, gen_uplimit); +if (repair) { + ret =repair_inode_gen_lowmem (root, path); +if (ret <0 ) + err |=INVALID_GENERATION ; + }else { + err |=INVALID_GENERATION ; + } + + }if (S_ISLNK (mode) && flags & (BTRFS_INODE_IMMUTABLE |BTRFS_INODE_APPEND )) { err |=INODE_FLAGS_ERROR ;
Re: [PATCH 1/3] btrfs-progs: check/lowmem: Add check and repair for invalid inode generation
Quoted 60 lines · Nikolay Borisov
Quoted 13 lines · Qu Wenruo
There are at least two bug reports of kernel tree-checker complaining about invalid inode generation.
All offending inodes seem to be caused by old kernel around 2014, with inode generation overflow.
So add such check and repair ability to lowmem mode check first.
This involves:
- Calculate the inode generation upper limit
If it's an inode from log tree, then the upper limit issuper_generation+ 1, otherwise it'ssuper_generation.Quoted 7 lines
- Check if the inode generation is larger than the upper limit
- Repair by resetting inode generation to current transaction generation
Reported-by Charles Wright <[email protected]>Signed-off-by Qu Wenruo <[email protected]>Tested-by Nikolay Borisov <[email protected]>There is one small nit with the assert once rectified you can add:
Reviewed-by Nikolay Borisov <[email protected]>Quoted 29 lines
--- check/mode-lowmem.c | 76 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+)check/mode-lowmem.c +17
-0@@ -2472,6 +2472,59 @@ static bool has_orphan_item(struct btrfs_root *root, u64 ino)return false; } +static int repair_inode_gen_lowmem (struct btrfs_root *root, +struct btrfs_path *path) +{ +struct btrfs_trans_handle *trans; +struct btrfs_inode_item *ii; +struct btrfs_key key; +u64 transid; +int ret; + + trans =btrfs_start_transaction (root,1 ); +if (IS_ERR (trans)) { + ret =PTR_ERR (trans); + errno = -ret; +error ("failed to start transaction for inode gen repair: %m" ); +return ret; + } + transid = trans->transid;
Quoted 8 lines
check/mode-lowmem.c static bool has_orphan_item(struct btrfs_root *root, u64 ino) · line 2493
Quoted 2 lines
check/mode-lowmem.c +2
-0+btrfs_item_key_to_cpu (path->nodes[0 ], &key, path->slots[0 ]); +ASSERT (key.type ==BTRFS_INODE_ITEM_KEY );nit: This function's sole caller,
check_inode_item, is guaranteed to be called with a path pointing toBTRFS_INODE_ITEM_KEYthanks to the logic in the 'for' loop inprocess_one_leaf. This renders the assert redundant. At the very least I think it should be moved tocheck_inode_item.
Yes, the ASSERT() doesn't make much sense by itself.
However I still believe it won't be a problem.
It's compiler's job to remove such dead ASSERT(), but for human reader, I still believe this ASSERT() could still make sense, especially when the caller or callee can get more and more complex.
Thanks,
Qu
Quoted 90 lines
Quoted 90 lines
check/mode-lowmem.c +57
-0+ +btrfs_release_path (path); + + ret =btrfs_search_slot (trans, root, &key, path,0 ,1 ); +if (ret >0 ) { + ret = -ENOENT ; +error ("no inode item found for ino %llu" , key.objectid); + goto error; + } +if (ret <0 ) { + errno = -ret; +error ("failed to find inode item for ino %llu: %m" , + key.objectid); + goto error; + } + ii =btrfs_item_ptr (path->nodes[0 ], path->slots[0 ], + struct btrfs_inode_item); +btrfs_set_inode_generation (path->nodes[0 ], ii, trans->transid); +btrfs_mark_buffer_dirty (path->nodes[0 ]); + ret =btrfs_commit_transaction (trans, root); +if (ret <0 ) { + errno = -ret; +error ("failed to commit transaction: %m" ); + goto error; + } +printf ("reseting inode generation to %llu for ino %llu\n" , + transid, key.objectid); +return ret; + +error: +btrfs_abort_transaction (trans, ret); +return ret; +} +/* * Check INODE_ITEM and related ITEMs (the same inode number) * 1. check link count @@ -2487,6 +2540,7 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path)struct btrfs_inode_item *ii;struct btrfs_key key;struct btrfs_key last_key; +struct btrfs_super_block *super = root->fs_info->super_copy;u64 inode_id;u32 mode;u64 flags;@@ -2497,6 +2551,8 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path)u64 refs =0 ;u64 extent_end =0 ;u64 extent_size =0 ; +u64 generation; +u64 gen_uplimit;unsigned int dir;unsigned int nodatasum;bool is_orphan = false;@@ -2527,6 +2583,7 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path) flags =btrfs_inode_flags (node, ii); dir =imode_to_type (mode) ==BTRFS_FT_DIR ; nlink =btrfs_inode_nlink (node, ii); + generation =btrfs_inode_generation (node, ii); nodatasum =btrfs_inode_flags (node, ii) &BTRFS_INODE_NODATASUM ;if (!is_valid_imode (mode)) {@@ -2540,6 +2597,25 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path) } } +if (btrfs_super_log_root (super) !=0 && + root->objectid ==BTRFS_TREE_LOG_OBJECTID ) + gen_uplimit =btrfs_super_generation (super) +1 ; +else + gen_uplimit =btrfs_super_generation (super); + +if (generation > gen_uplimit) { +error ( +"invalid inode generation for ino %llu, have %llu expect [0, %llu)" , + inode_id, generation, gen_uplimit); +if (repair) { + ret =repair_inode_gen_lowmem (root, path); +if (ret <0 ) + err |=INVALID_GENERATION ; + }else { + err |=INVALID_GENERATION ; + } + + }if (S_ISLNK (mode) && flags & (BTRFS_INODE_IMMUTABLE |BTRFS_INODE_APPEND )) { err |=INODE_FLAGS_ERROR ;
Re: [PATCH 1/3] btrfs-progs: check/lowmem: Add check and repair for invalid inode generation
Quoted 3 lines · Qu Wenruo
Yes, the
ASSERT()doesn't make much sense by itself.However I still believe it won't be a problem.
It won't be a problem but it feels wrong to have this assert this deep into the call chain. IMO It should be put where it can trigger at the earliest which seems to be in check_inode_item. That function assumes it's working with an inode item and goes to dereference inode members so if the type is wrong we'd crash there instead of in repair_inode_gen_lowmem.
Quoted 6 lines
It's compiler's job to remove such dead
ASSERT(), but for human reader, I still believe thisASSERT()could still make sense, especially when the caller or callee can get more and more complex.Thanks,
Re: [PATCH 1/3] btrfs-progs: check/lowmem: Add check and repair for invalid inode generation
Quoted 5 lines · Nikolay Borisov
Quoted 2 lines · Qu Wenruo
Yes, the
ASSERT()doesn't make much sense by itself.
Quoted 7 lines
Quoted 1 line
However I still believe it won't be a problem.
It won't be a problem but it feels wrong to have this assert this deep into the call chain. IMO It should be put where it can trigger at the earliest which seems to be in
check_inode_item. That function assumes it's working with an inode item and goes to dereference inode members so if the type is wrong we'd crash there instead of inrepair_inode_gen_lowmem.
This is going to be a preference thing.
To my taste, if possible, I would put ASSERT() in every function with more than 24 lines, as an alternative to comment, to show the prerequisite or the assumption.
Thanks,
Qu
Quoted 7 lines
Quoted 7 lines
It's compiler's job to remove such dead
ASSERT(), but for human reader, I still believe thisASSERT()could still make sense, especially when the caller or callee can get more and more complex.Thanks,
PATCH 2/3 btrfs-progs: check/original: Add check and repair for invalid inode generation
In reply to Qu WenruoThere are at least two bug reports of kernel tree-checker complaining about invalid inode generation.
All offending inodes seem to be caused by old kernel around 2014, with inode generation overflow.
So add such check and repair ability to lowmem mode check first.
This involves:
- Calculate the inode generation upper limit
Unlike the original mode context, we don't have anyway to determine if this inode belongs to log tree.
So we use super_generation + 1 as upper limit, just like what we did in kernel tree checker.
- Check if the inode generation is larger than the upper limit
- Repair by resetting inode generation to current transaction generation
The difference is, in original mode, we have a common trans handle for all repair and reset path for each repair.
--- check/main.c | 50 ++++++++++++++++++++++++++++++++++++++++++- check/mode-original.h | 1 + 2 files changed, 50 insertions(+), 1 deletion(-)
check/main.c +49 -1
@@ -604,6 +604,8 @@ static void print_inode_error(struct btrfs_root *root, struct inode_record *rec)if (errors &I_ERR_INVALID_IMODE )fprintf (stderr,", invalid inode mode bit 0%o" , rec->imode & ~07777 ); +if (errors &I_ERR_INVALID_GEN ) +fprintf (stderr,", invalid inode generation" );fprintf (stderr,"\n" );/* Print the holes if needed */
@@ -857,6 +859,7 @@ static int process_inode_item(struct extent_buffer *eb, {struct inode_record *rec;struct btrfs_inode_item *item; +u64 gen_uplimit;u64 flags; rec = active_node->current;
@@ -879,6 +882,13 @@ static int process_inode_item(struct extent_buffer *eb,if (S_ISLNK (rec->imode) && flags & (BTRFS_INODE_IMMUTABLE |BTRFS_INODE_APPEND )) rec->errors |=I_ERR_ODD_INODE_FLAGS ; +/* +* We don't have accurate root info to determine the correct +* inode generation uplimit, use super_generation + 1 anyway +*/ + gen_uplimit =btrfs_super_generation (global_info->super_copy) +1 ; +if (btrfs_inode_generation (eb, item) > gen_uplimit) + rec->errors |=I_ERR_INVALID_GEN ;maybe_free_inode_rec (&active_node->inode_cache, rec);return 0 ; }
@@ -2774,6 +2784,41 @@ static int repair_imode_original(struct btrfs_trans_handle *trans,return ret; } +static int repair_inode_gen_original (struct btrfs_trans_handle *trans, +struct btrfs_root *root, +struct btrfs_path *path, +struct inode_record *rec) +{ +struct btrfs_inode_item *ii; +struct btrfs_key key; +int ret; + + key.objectid = rec->ino; + key.offset =0 ; + key.type =BTRFS_INODE_ITEM_KEY ; + + ret =btrfs_search_slot (trans, root, &key, path,0 ,1 ); +if (ret >0 ) { + ret = -ENOENT ; +error ("no inode item found for ino %llu" , rec->ino); +return ret; + } +if (ret <0 ) { + errno = -ret; +error ("failed to search inode item for ino %llu: %m" , rec->ino); +return ret; + } + ii =btrfs_item_ptr (path->nodes[0 ], path->slots[0 ], + struct btrfs_inode_item); +btrfs_set_inode_generation (path->nodes[0 ], ii, trans->transid); +btrfs_mark_buffer_dirty (path->nodes[0 ]); +btrfs_release_path (path); +printf ("resetting inode generation to %llu for ino %llu\n" , + trans->transid, rec->ino); + rec->errors &= ~I_ERR_INVALID_GEN ; +return 0 ; +} +static int try_repair_inode (struct btrfs_root *root,struct inode_record *rec) {struct btrfs_trans_handle *trans;
@@ -2794,7 +2839,8 @@ static int try_repair_inode(struct btrfs_root *root, struct inode_record *rec)I_ERR_FILE_NBYTES_WRONG |I_ERR_INLINE_RAM_BYTES_WRONG |I_ERR_MISMATCH_DIR_HASH | - I_ERR_UNALIGNED_EXTENT_REC))) +I_ERR_UNALIGNED_EXTENT_REC | + I_ERR_INVALID_GEN))) return rec->errors;/*
@@ -2829,6 +2875,8 @@ static int try_repair_inode(struct btrfs_root *root, struct inode_record *rec) ret =repair_unaligned_extent_recs (trans, root, &path, rec);if (!ret && rec->errors &I_ERR_INVALID_IMODE ) ret =repair_imode_original (trans, root, &path, rec); +if (!ret && rec->errors &I_ERR_INVALID_GEN ) + ret =repair_inode_gen_original (trans, root, &path, rec);btrfs_commit_transaction (trans, root);btrfs_release_path (&path);return ret;
check/mode-original.h +1 -0
@@ -185,6 +185,7 @@ struct unaligned_extent_rec_t {#define I_ERR_INLINE_RAM_BYTES_WRONG (1 << 17)#define I_ERR_MISMATCH_DIR_HASH (1 << 18)#define I_ERR_INVALID_IMODE (1 << 19) +#define I_ERR_INVALID_GEN (1 << 20)struct inode_record {struct list_head backrefs;
Signature · 1 line
2.23.0
Re: [PATCH 2/3] btrfs-progs: check/original: Add check and repair for invalid inode generation
Quoted 11 lines · Qu Wenruo
There are at least two bug reports of kernel tree-checker complaining about invalid inode generation.
All offending inodes seem to be caused by old kernel around 2014, with inode generation overflow.
So add such check and repair ability to lowmem mode check first.
This involves:
- Calculate the inode generation upper limit
Unlike the original mode context, we don't have anyway to determine if
Perhaps you meant "lowmem mode context" since in lowmem you deduce whether it's a log-tree inode or not.
Quoted 6 lines
this inode belongs to log tree.
So we usesuper_generation+ 1 as upper limit, just like what we did in kernel tree checker.- Check if the inode generation is larger than the upper limit
Quoted 7 lines
- Repair by resetting inode generation to current transaction generation
The difference is, in original mode, we have a common trans handle for all repair and reset path for each repair.Reported-by Charles Wright <[email protected]>Signed-off-by Qu Wenruo <[email protected]>
I tested that code with the image provided and it does work as expected as well as it's fairly obvious what's happening. So:
Quoted 114 lines
--- check/main.c | 50 ++++++++++++++++++++++++++++++++++++++++++- check/mode-original.h | 1 + 2 files changed, 50 insertions(+), 1 deletion(-)check/main.c +49
-1@@ -604,6 +604,8 @@ static void print_inode_error(struct btrfs_root *root, struct inode_record *rec)if (errors &I_ERR_INVALID_IMODE )fprintf (stderr,", invalid inode mode bit 0%o" , rec->imode & ~07777 ); +if (errors &I_ERR_INVALID_GEN ) +fprintf (stderr,", invalid inode generation" );fprintf (stderr,"\n" );/* Print the holes if needed */ @@ -857,6 +859,7 @@ static int process_inode_item(struct extent_buffer *eb, {struct inode_record *rec;struct btrfs_inode_item *item; +u64 gen_uplimit;u64 flags; rec = active_node->current;@@ -879,6 +882,13 @@ static int process_inode_item(struct extent_buffer *eb,if (S_ISLNK (rec->imode) && flags & (BTRFS_INODE_IMMUTABLE |BTRFS_INODE_APPEND )) rec->errors |=I_ERR_ODD_INODE_FLAGS ; +/* +* We don't have accurate root info to determine the correct +* inode generation uplimit, use super_generation + 1 anyway +*/ + gen_uplimit =btrfs_super_generation (global_info->super_copy) +1 ; +if (btrfs_inode_generation (eb, item) > gen_uplimit) + rec->errors |=I_ERR_INVALID_GEN ;maybe_free_inode_rec (&active_node->inode_cache, rec);return 0 ; }@@ -2774,6 +2784,41 @@ static int repair_imode_original(struct btrfs_trans_handle *trans,return ret; } +static int repair_inode_gen_original (struct btrfs_trans_handle *trans, +struct btrfs_root *root, +struct btrfs_path *path, +struct inode_record *rec) +{ +struct btrfs_inode_item *ii; +struct btrfs_key key; +int ret; + + key.objectid = rec->ino; + key.offset =0 ; + key.type =BTRFS_INODE_ITEM_KEY ; + + ret =btrfs_search_slot (trans, root, &key, path,0 ,1 ); +if (ret >0 ) { + ret = -ENOENT ; +error ("no inode item found for ino %llu" , rec->ino); +return ret; + } +if (ret <0 ) { + errno = -ret; +error ("failed to search inode item for ino %llu: %m" , rec->ino); +return ret; + } + ii =btrfs_item_ptr (path->nodes[0 ], path->slots[0 ], + struct btrfs_inode_item); +btrfs_set_inode_generation (path->nodes[0 ], ii, trans->transid); +btrfs_mark_buffer_dirty (path->nodes[0 ]); +btrfs_release_path (path); +printf ("resetting inode generation to %llu for ino %llu\n" , + trans->transid, rec->ino); + rec->errors &= ~I_ERR_INVALID_GEN ; +return 0 ; +} +static int try_repair_inode (struct btrfs_root *root,struct inode_record *rec) {struct btrfs_trans_handle *trans;@@ -2794,7 +2839,8 @@ static int try_repair_inode(struct btrfs_root *root, struct inode_record *rec)I_ERR_FILE_NBYTES_WRONG |I_ERR_INLINE_RAM_BYTES_WRONG |I_ERR_MISMATCH_DIR_HASH | - I_ERR_UNALIGNED_EXTENT_REC))) +I_ERR_UNALIGNED_EXTENT_REC | + I_ERR_INVALID_GEN))) return rec->errors;/* @@ -2829,6 +2875,8 @@ static int try_repair_inode(struct btrfs_root *root, struct inode_record *rec) ret =repair_unaligned_extent_recs (trans, root, &path, rec);if (!ret && rec->errors &I_ERR_INVALID_IMODE ) ret =repair_imode_original (trans, root, &path, rec); +if (!ret && rec->errors &I_ERR_INVALID_GEN ) + ret =repair_inode_gen_original (trans, root, &path, rec);btrfs_commit_transaction (trans, root);btrfs_release_path (&path);return ret;check/mode-original.h +1
-0@@ -185,6 +185,7 @@ struct unaligned_extent_rec_t {#define I_ERR_INLINE_RAM_BYTES_WRONG (1 << 17)#define I_ERR_MISMATCH_DIR_HASH (1 << 18)#define I_ERR_INVALID_IMODE (1 << 19) +#define I_ERR_INVALID_GEN (1 << 20)struct inode_record {struct list_head backrefs;
Re: [PATCH 2/3] btrfs-progs: check/original: Add check and repair for invalid inode generation
Quoted 9 lines · Nikolay Borisov
Quoted 6 lines · Qu Wenruo
There are at least two bug reports of kernel tree-checker complaining about invalid inode generation.
All offending inodes seem to be caused by old kernel around 2014, with inode generation overflow.
Quoted 8 lines
Quoted 5 lines
So add such check and repair ability to lowmem mode check first.
This involves:
- Calculate the inode generation upper limit
Unlike the original mode context, we don't have anyway to determine ifPerhaps you meant "lowmem mode context" since in lowmem you deduce whether it's a log-tree inode or not.
Oh, right.
Would you mind to fix this in commit time, David?
Quoted 12 lines
Quoted 12 lines
this inode belongs to log tree.
So we usesuper_generation+ 1 as upper limit, just like what we did in kernel tree checker.- Check if the inode generation is larger than the upper limit
- Repair by resetting inode generation to current transaction generation
The difference is, in original mode, we have a common trans handle for all repair and reset path for each repair.
Quoted 8 lines
Quoted 2 lines
Reported-by Charles Wright <[email protected]>Signed-off-by Qu Wenruo <[email protected]>I tested that code with the image provided and it does work as expected as well as it's fairly obvious what's happening. So:
Reviewed-by Nikolay Borisov <[email protected]>Tested-by Nikolay Borisov <[email protected]>
Thanks for the review and test.
Thanks,
Qu
Quoted 115 lines
Quoted 115 lines
--- check/main.c | 50 ++++++++++++++++++++++++++++++++++++++++++- check/mode-original.h | 1 + 2 files changed, 50 insertions(+), 1 deletion(-)check/main.c +49
-1@@ -604,6 +604,8 @@ static void print_inode_error(struct btrfs_root *root, struct inode_record *rec)if (errors &I_ERR_INVALID_IMODE )fprintf (stderr,", invalid inode mode bit 0%o" , rec->imode & ~07777 ); +if (errors &I_ERR_INVALID_GEN ) +fprintf (stderr,", invalid inode generation" );fprintf (stderr,"\n" );/* Print the holes if needed */ @@ -857,6 +859,7 @@ static int process_inode_item(struct extent_buffer *eb, {struct inode_record *rec;struct btrfs_inode_item *item; +u64 gen_uplimit;u64 flags; rec = active_node->current;@@ -879,6 +882,13 @@ static int process_inode_item(struct extent_buffer *eb,if (S_ISLNK (rec->imode) && flags & (BTRFS_INODE_IMMUTABLE |BTRFS_INODE_APPEND )) rec->errors |=I_ERR_ODD_INODE_FLAGS ; +/* +* We don't have accurate root info to determine the correct +* inode generation uplimit, use super_generation + 1 anyway +*/ + gen_uplimit =btrfs_super_generation (global_info->super_copy) +1 ; +if (btrfs_inode_generation (eb, item) > gen_uplimit) + rec->errors |=I_ERR_INVALID_GEN ;maybe_free_inode_rec (&active_node->inode_cache, rec);return 0 ; }@@ -2774,6 +2784,41 @@ static int repair_imode_original(struct btrfs_trans_handle *trans,return ret; } +static int repair_inode_gen_original (struct btrfs_trans_handle *trans, +struct btrfs_root *root, +struct btrfs_path *path, +struct inode_record *rec) +{ +struct btrfs_inode_item *ii; +struct btrfs_key key; +int ret; + + key.objectid = rec->ino; + key.offset =0 ; + key.type =BTRFS_INODE_ITEM_KEY ; + + ret =btrfs_search_slot (trans, root, &key, path,0 ,1 ); +if (ret >0 ) { + ret = -ENOENT ; +error ("no inode item found for ino %llu" , rec->ino); +return ret; + } +if (ret <0 ) { + errno = -ret; +error ("failed to search inode item for ino %llu: %m" , rec->ino); +return ret; + } + ii =btrfs_item_ptr (path->nodes[0 ], path->slots[0 ], + struct btrfs_inode_item); +btrfs_set_inode_generation (path->nodes[0 ], ii, trans->transid); +btrfs_mark_buffer_dirty (path->nodes[0 ]); +btrfs_release_path (path); +printf ("resetting inode generation to %llu for ino %llu\n" , + trans->transid, rec->ino); + rec->errors &= ~I_ERR_INVALID_GEN ; +return 0 ; +} +static int try_repair_inode (struct btrfs_root *root,struct inode_record *rec) {struct btrfs_trans_handle *trans;@@ -2794,7 +2839,8 @@ static int try_repair_inode(struct btrfs_root *root, struct inode_record *rec)I_ERR_FILE_NBYTES_WRONG |I_ERR_INLINE_RAM_BYTES_WRONG |I_ERR_MISMATCH_DIR_HASH | - I_ERR_UNALIGNED_EXTENT_REC))) +I_ERR_UNALIGNED_EXTENT_REC | + I_ERR_INVALID_GEN))) return rec->errors;/* @@ -2829,6 +2875,8 @@ static int try_repair_inode(struct btrfs_root *root, struct inode_record *rec) ret =repair_unaligned_extent_recs (trans, root, &path, rec);if (!ret && rec->errors &I_ERR_INVALID_IMODE ) ret =repair_imode_original (trans, root, &path, rec); +if (!ret && rec->errors &I_ERR_INVALID_GEN ) + ret =repair_inode_gen_original (trans, root, &path, rec);btrfs_commit_transaction (trans, root);btrfs_release_path (&path);return ret;check/mode-original.h +1
-0@@ -185,6 +185,7 @@ struct unaligned_extent_rec_t {#define I_ERR_INLINE_RAM_BYTES_WRONG (1 << 17)#define I_ERR_MISMATCH_DIR_HASH (1 << 18)#define I_ERR_INVALID_IMODE (1 << 19) +#define I_ERR_INVALID_GEN (1 << 20)struct inode_record {struct list_head backrefs;
PATCH 3/3 btrfs-progs: fsck-tests: Add test image for invalid inode generation repair
In reply to Qu WenruoThe image contains one inode item with invalid generation. The image can be crafted by "btrfs-corrupt-block -i 257 -f generation". It should emulate the bad inode generation caused by older kernel around 2014.
The image is repairable for both original and lowmem mode.
--- .../043-bad-inode-generation/.lowmem_repairable | 0 .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes 2 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable create mode 100644 tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable new file +0 -0
tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz new file binary +0 -0
Signature · 1 line
2.23.0
Op 24-09-2019 om 10:11 schreef Qu Wenruo:
Quoted 2 lines
We have at least two user reports about bad inode generation makes kernel reject the fs.
May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@
In initramfs I can try to mount but get something like btrfs critical corrupt leaf invalid inode generation open_ctree failed
Booting old kernel works just as before, no errors.
Quoted 2 lines
According to the creation time, the inode is created by some 2014 kernel.
How do I get the creation time?
Quoted 7 lines
And the generation member of
INODE_ITEMis not updated (unlike the transid member) so the error persists until latest tree-checker detects.Even the situation can be fixed by reverting back to older kernel and copying the offending dir/file to another inode and delete the offending one, it still should be done by btrfs-progs.
How to find the offending dir/file from the command line manually?
Quoted 20 lines
This patchset adds such check and repair ability to btrfs-check, with a simple test image.
Qu Wenruo (3):
btrfs-progs: check/lowmem: Add check and repair for invalid inode
generation
btrfs-progs: check/original: Add check and repair for invalid inode
generation
btrfs-progs: fsck-tests: Add test image for invalid inode generation
repaircheck/main.c | 50 +++++++++++- check/mode-lowmem.c | 76 ++++++++++++++++++ check/mode-original.h | 1 + .../.lowmem_repairable | 0 .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes 5 files changed, 126 insertions(+), 1 deletion(-)create mode 100644 tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable create mode 100644 tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
Quoted 12 lines · Ferry Toth
Op 24-09-2019 om 10:11 schreef Qu Wenruo:
Quoted 2 lines
We have at least two user reports about bad inode generation makes kernel reject the fs.
May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@
In initramfs I can try to mount but get something like btrfs critical corrupt leaf invalid inode generation
open_ctreefailed
Quoted 6 lines
Booting old kernel works just as before, no errors.
Quoted 2 lines
According to the creation time, the inode is created by some 2014 kernel.
How do I get the creation time?
# btrfs ins dump-tree -b <the bytenr reported by kernel> <your device>
Quoted 9 lines
Quoted 8 lines
And the generation member of
INODE_ITEMis not updated (unlike the transid member) so the error persists until latest tree-checker detects.Even the situation can be fixed by reverting back to older kernel and copying the offending dir/file to another inode and delete the offending one, it still should be done by btrfs-progs.
How to find the offending dir/file from the command line manually?
# find <mount point> -inum <inode number>
Thanks,
Qu
Quoted 24 lines
Quoted 23 lines
This patchset adds such check and repair ability to btrfs-check, with a simple test image.
Qu Wenruo (3):
btrfs-progs: check/lowmem: Add check and repair for invalid inode generation
btrfs-progs: check/original: Add check and repair for invalid inode generation
btrfs-progs: fsck-tests: Add test image for invalid inode generation repaircheck/main.c | 50 +++++++++++- check/mode-lowmem.c | 76 ++++++++++++++++++ check/mode-original.h | 1 + .../.lowmem_repairable | 0 .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes
5 files changed, 126 insertions(+), 1 deletion(-)create mode 100644 tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable create mode 100644 tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
Hi,
Op 19-10-2019 om 01:50 schreef Qu WenRuo:
Quoted 15 lines · Ferry Toth
Quoted 12 lines
Op 24-09-2019 om 10:11 schreef Qu Wenruo:
Quoted 2 lines
We have at least two user reports about bad inode generation makes kernel reject the fs.
May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@
In initramfs I can try to mount but get something like btrfs critical corrupt leaf invalid inode generation
open_ctreefailed
Quoted 8 lines
Quoted 6 lines
Booting old kernel works just as before, no errors.
Quoted 2 lines
According to the creation time, the inode is created by some 2014 kernel.
How do I get the creation time?
# btrfs ins dump-tree -b <the bytenr reported by kernel> <your device>
I just went back to the office to reboot to 5.3 and check the creation times and found they were 2013 - 2014.
Quoted 11 lines
Quoted 9 lines
Quoted 8 lines
And the generation member of
INODE_ITEMis not updated (unlike the transid member) so the error persists until latest tree-checker detects.Even the situation can be fixed by reverting back to older kernel and copying the offending dir/file to another inode and delete the offending one, it still should be done by btrfs-progs.
How to find the offending dir/file from the command line manually?
# find <mount point> -inum <inode number>
This works, thanks.
But appears unpractical. After fix 2 files and reboot, I found 4 more, then 16, then I gave up.
Quoted 27 lines
Thanks,
QuQuoted 24 lines
Quoted 23 lines
This patchset adds such check and repair ability to btrfs-check, with a simple test image.
Qu Wenruo (3):
btrfs-progs: check/lowmem: Add check and repair for invalid inode generation
btrfs-progs: check/original: Add check and repair for invalid inode generation
btrfs-progs: fsck-tests: Add test image for invalid inode generation repaircheck/main.c | 50 +++++++++++- check/mode-lowmem.c | 76 ++++++++++++++++++ check/mode-original.h | 1 + .../.lowmem_repairable | 0 .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes 5 files changed, 126 insertions(+), 1 deletion(-)create mode 100644
tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
create mode 100644
tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my mounted rootfs with linux 5.0 and captured the log (~1800 lines 209 errors).
I'm not sure if using the v5.0 kernel and/or checking mounted distorts the results? Else I'm going to need a live usb with a v5.3 kernel and v5.3 btrfs-progs.
If you like I can share the log. Let me know.
This issue can potentially cause a lot of grief. Our company server runs Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with ~100 snapshots. I guess the problematic inodes need to be fixed on each snapshot prior to upgrading to 20.04 LTS (which might be on kernel ~5.6)?
Do I understand correctly that this FTB is caused by more strict checking of the fs by the kernel, while the tools to fix the detected corruptions are not yet released?
Quoted 79 lines · Ferry Toth
Hi,
Op 19-10-2019 om 01:50 schreef Qu WenRuo:
Quoted 15 lines · Ferry Toth
Quoted 12 lines
Op 24-09-2019 om 10:11 schreef Qu Wenruo:
Quoted 2 lines
We have at least two user reports about bad inode generation makes kernel reject the fs.
May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@
In initramfs I can try to mount but get something like btrfs critical corrupt leaf invalid inode generation
open_ctreefailedQuoted 8 lines
Quoted 6 lines
Booting old kernel works just as before, no errors.
Quoted 2 lines
According to the creation time, the inode is created by some 2014 kernel.
How do I get the creation time?
# btrfs ins dump-tree -b <the bytenr reported by kernel> <your device>I just went back to the office to reboot to 5.3 and check the creation times and found they were 2013 - 2014.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
And the generation member of
INODE_ITEMis not updated (unlike the transid member) so the error persists until latest tree-checker detects.Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Even the situation can be fixed by reverting back to older kernel and copying the offending dir/file to another inode and delete the offending
one, it still should be done by btrfs-progs.How to find the offending dir/file from the command line manually?
# find <mount point> -inum <inode number>This works, thanks.
But appears unpractical. After fix 2 files and reboot, I found 4 more, then 16, then I gave up.
Quoted 29 lines
Thanks,
QuQuoted 26 lines
Quoted 25 lines
This patchset adds such check and repair ability to btrfs-check, with a simple test image.
Qu Wenruo (3):
btrfs-progs: check/lowmem: Add check and repair for invalid inode generation
btrfs-progs: check/original: Add check and repair for invalid inode generation
btrfs-progs: fsck-tests: Add test image for invalid inode generation
repaircheck/main.c | 50 +++++++++++- check/mode-lowmem.c | 76
++++++++++++++++++
check/mode-original.h | 1 + .../.lowmem_repairable | 0 .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes5 files changed, 126 insertions(+), 1 deletion(-)create mode 100644 tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable create mode 100644 tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
Quoted 3 lines
I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my mounted rootfs with linux 5.0 and captured the log (~1800 lines 209 errors).
It's really not recommended to run btrfs check, especially repair on the mounted fs, unless it's RO.
A new transaction from kernel can easily screw up the repaired fs.
Quoted 11 lines
I'm not sure if using the v5.0 kernel and/or checking mounted distorts the results? Else I'm going to need a live usb with a v5.3 kernel and v5.3 btrfs-progs.
If you like I can share the log. Let me know.
This issue can potentially cause a lot of grief. Our company server runs Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with ~100 snapshots. I guess the problematic inodes need to be fixed on each snapshot prior to upgrading to 20.04 LTS (which might be on kernel ~5.6)?
Yes.
Quoted 4 lines
Do I understand correctly that this FTB is caused by more strict checking of the fs by the kernel, while the tools to fix the detected corruptions are not yet released?
Yes.
Thanks,
Qu
Quoted 43 lines · Qu Wenruo
Quoted 40 lines · Ferry Toth
Hi,
Op 19-10-2019 om 01:50 schreef Qu WenRuo:
Quoted 15 lines · Ferry Toth
Quoted 12 lines
Op 24-09-2019 om 10:11 schreef Qu Wenruo:
Quoted 2 lines
We have at least two user reports about bad inode generation makes kernel reject the fs.
May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@
In initramfs I can try to mount but get something like btrfs critical corrupt leaf invalid inode generation
open_ctreefailedQuoted 8 lines
Quoted 6 lines
Booting old kernel works just as before, no errors.
Quoted 2 lines
According to the creation time, the inode is created by some 2014 kernel.
How do I get the creation time?
# btrfs ins dump-tree -b <the bytenr reported by kernel> <your device>I just went back to the office to reboot to 5.3 and check the creation times and found they were 2013 - 2014.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
And the generation member of
INODE_ITEMis not updated (unlike the transid member) so the error persists until latest tree-checker detects.Even the situation can be fixed by reverting back to older kernel and copying the offending dir/file to another inode and delete the offending
one, it still should be done by btrfs-progs.
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
How to find the offending dir/file from the command line manually?
# find <mount point> -inum <inode number>This works, thanks.
But appears unpractical. After fix 2 files and reboot, I found 4 more, then 16, then I gave up.
Another solution is use "find" to locate the files with creation time before 2015, and copy them to a new file, then replace the old file with the new file.
It would be much safer than btrfs check --repair.
Thanks,
Qu
Quoted 63 lines
Quoted 31 lines
Quoted 30 lines
Thanks,
QuQuoted 26 lines
Quoted 25 lines
This patchset adds such check and repair ability to btrfs-check, with a simple test image.
Qu Wenruo (3):
btrfs-progs: check/lowmem: Add check and repair for invalid inode generation
btrfs-progs: check/original: Add check and repair for invalid inode generation
btrfs-progs: fsck-tests: Add test image for invalid inode generation
repaircheck/main.c | 50 +++++++++++- check/mode-lowmem.c | 76
++++++++++++++++++
check/mode-original.h | 1 + .../.lowmem_repairable | 0 .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes5 files changed, 126 insertions(+), 1 deletion(-)create mode 100644 tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable create mode 100644 tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
Quoted 3 lines
I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my mounted rootfs with linux 5.0 and captured the log (~1800 lines 209 errors).
It's really not recommended to run btrfs check, especially repair on the mounted fs, unless it's RO.
A new transaction from kernel can easily screw up the repaired fs.
Quoted 11 lines
I'm not sure if using the v5.0 kernel and/or checking mounted distorts the results? Else I'm going to need a live usb with a v5.3 kernel and v5.3 btrfs-progs.
If you like I can share the log. Let me know.
This issue can potentially cause a lot of grief. Our company server runs Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with ~100 snapshots. I guess the problematic inodes need to be fixed on each snapshot prior to upgrading to 20.04 LTS (which might be on kernel ~5.6)?
Yes.
Quoted 4 lines
Do I understand correctly that this FTB is caused by more strict checking of the fs by the kernel, while the tools to fix the detected corruptions are not yet released?
Yes.
Thanks,
Qu
Op 20-10-2019 om 02:51 schreef Qu Wenruo:
Quoted 50 lines · Qu Wenruo
Quoted 47 lines · Ferry Toth
Quoted 44 lines
Hi,
Op 19-10-2019 om 01:50 schreef Qu WenRuo:
Quoted 15 lines · Ferry Toth
Quoted 12 lines
Op 24-09-2019 om 10:11 schreef Qu Wenruo:
Quoted 2 lines
We have at least two user reports about bad inode generation makes kernel reject the fs.
May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@
In initramfs I can try to mount but get something like btrfs critical corrupt leaf invalid inode generation
open_ctreefailedQuoted 8 lines
Quoted 6 lines
Booting old kernel works just as before, no errors.
Quoted 2 lines
According to the creation time, the inode is created by some 2014 kernel.
How do I get the creation time?
# btrfs ins dump-tree -b <the bytenr reported by kernel> <your device>I just went back to the office to reboot to 5.3 and check the creation times and found they were 2013 - 2014.
Quoted 13 lines
Quoted 11 lines
Quoted 10 lines
And the generation member of
INODE_ITEMis not updated (unlike the transid member) so the error persists until latest tree-checker detects.Even the situation can be fixed by reverting back to older kernel and copying the offending dir/file to another inode and delete the offending
one, it still should be done by btrfs-progs.How to find the offending dir/file from the command line manually?
# find <mount point> -inum <inode number>
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
This works, thanks.
But appears unpractical. After fix 2 files and reboot, I found 4 more, then 16, then I gave up.
Another solution is use "find" to locate the files with creation time before 2015, and copy them to a new file, then replace the old file with the new file.
Hmm. But how do I "find" by creation time (otime)? Do you have a suggestion for this?
Quoted 70 lines
It would be much safer than btrfs check --repair.
Thanks,
QuQuoted 63 lines
Quoted 31 lines
Quoted 30 lines
Thanks,
QuQuoted 26 lines
Quoted 25 lines
This patchset adds such check and repair ability to btrfs-check, with a simple test image.
Qu Wenruo (3):
btrfs-progs: check/lowmem: Add check and repair for invalid inode generation
btrfs-progs: check/original: Add check and repair for invalid inode generation
btrfs-progs: fsck-tests: Add test image for invalid inode
generation
repaircheck/main.c | 50 +++++++++++- check/mode-lowmem.c | 76 ++++++++++++++++++ check/mode-original.h | 1 + .../.lowmem_repairable | 0 .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes 5 files changed, 126 insertions(+), 1 deletion(-)create mode 100644
tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
create mode 100644
tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xzQuoted 3 lines
I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my mounted rootfs with linux 5.0 and captured the log (~1800 lines 209 errors).
It's really not recommended to run btrfs check, especially repair on the mounted fs, unless it's RO.
A new transaction from kernel can easily screw up the repaired fs.
Quoted 11 lines
I'm not sure if using the v5.0 kernel and/or checking mounted distorts the results? Else I'm going to need a live usb with a v5.3 kernel and v5.3 btrfs-progs.
If you like I can share the log. Let me know.
This issue can potentially cause a lot of grief. Our company server runs Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with ~100 snapshots. I guess the problematic inodes need to be fixed on each snapshot prior to upgrading to 20.04 LTS (which might be on kernel ~5.6)?
Yes.
Quoted 4 lines
Do I understand correctly that this FTB is caused by more strict checking of the fs by the kernel, while the tools to fix the detected corruptions are not yet released?
Yes.
Thanks,
Qu
Quoted 59 lines · Ferry Toth
Op 20-10-2019 om 02:51 schreef Qu Wenruo:
Quoted 58 lines · Qu Wenruo
Quoted 54 lines · Ferry Toth
Quoted 51 lines
Hi,
Op 19-10-2019 om 01:50 schreef Qu WenRuo:
Quoted 17 lines · Ferry Toth
Quoted 14 lines
Op 24-09-2019 om 10:11 schreef Qu Wenruo:
Quoted 2 lines
We have at least two user reports about bad inode generation makes kernel reject the fs.
May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@
In initramfs I can try to mount but get something like btrfs critical corrupt leaf invalid inode generation
open_ctreefailedQuoted 8 lines
Quoted 6 lines
Booting old kernel works just as before, no errors.
Quoted 2 lines
According to the creation time, the inode is created by some 2014 kernel.
How do I get the creation time?
# btrfs ins dump-tree -b <the bytenr reported by kernel> <your device>I just went back to the office to reboot to 5.3 and check the creation times and found they were 2013 - 2014.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
And the generation member of
INODE_ITEMis not updated (unlike the transid member) so the error persists until latest tree-checker detects.Even the situation can be fixed by reverting back to older kernel and
copying the offending dir/file to another inode and delete the offending
one, it still should be done by btrfs-progs.Quoted 3 lines
Quoted 1 line
How to find the offending dir/file from the command line manually?
# find <mount point> -inum <inode number>This works, thanks.
But appears unpractical. After fix 2 files and reboot, I found 4 more, then 16, then I gave up.
Quoted 6 lines
Quoted 3 lines
Another solution is use "find" to locate the files with creation time before 2015, and copy them to a new file, then replace the old file with the new file.
Hmm. But how do I "find" by creation time (otime)? Do you have a suggestion for this?
$ touch -t 201501010000 /tmp/sample $ find <mnt> -not -cnewer /tmp/sample
If you want, you can add -exec to that find, but I'd only add that after confirming the execution load is verified.
Thanks,
Qu
Quoted 79 lines
Quoted 78 lines
It would be much safer than btrfs check --repair.
Thanks,
QuQuoted 70 lines
Quoted 36 lines
Quoted 35 lines
Thanks,
QuQuoted 31 lines
Quoted 30 lines
This patchset adds such check and repair ability to btrfs-check, with a
simple test image.Qu Wenruo (3):
btrfs-progs: check/lowmem: Add check and repair for invalid inode
generation
btrfs-progs: check/original: Add check and repair for invalid inode
generation
btrfs-progs: fsck-tests: Add test image for invalid inode generation
repaircheck/main.c | 50 +++++++++++- check/mode-lowmem.c | 76
++++++++++++++++++
check/mode-original.h | 1 + .../.lowmem_repairable | 0 .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes5 files changed, 126 insertions(+), 1 deletion(-)create mode 100644 tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable create mode 100644 tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
Quoted 3 lines
I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my mounted rootfs with linux 5.0 and captured the log (~1800 lines 209 errors).
It's really not recommended to run btrfs check, especially repair on the mounted fs, unless it's RO.
A new transaction from kernel can easily screw up the repaired fs.
Quoted 5 lines
I'm not sure if using the v5.0 kernel and/or checking mounted distorts the results? Else I'm going to need a live usb with a v5.3 kernel and v5.3 btrfs-progs.
Quoted 8 lines
If you like I can share the log. Let me know.
This issue can potentially cause a lot of grief. Our company server runs
Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with ~100 snapshots. I guess the problematic inodes need to be fixed on each snapshot prior to upgrading to 20.04 LTS (which might be on kernel ~5.6)?Yes.
Quoted 4 lines
Do I understand correctly that this FTB is caused by more strict checking of the fs by the kernel, while the tools to fix the detected corruptions are not yet released?
Yes.
Thanks,
Qu
Op 20-10-2019 om 15:15 schreef Qu WenRuo:
Quoted 66 lines · Ferry Toth
Quoted 63 lines
Op 20-10-2019 om 02:51 schreef Qu Wenruo:
Quoted 61 lines · Qu Wenruo
Quoted 46 lines · Ferry Toth
Quoted 43 lines
Hi,
Op 19-10-2019 om 01:50 schreef Qu WenRuo:
Quoted 17 lines · Ferry Toth
Quoted 14 lines
Op 24-09-2019 om 10:11 schreef Qu Wenruo:
Quoted 2 lines
We have at least two user reports about bad inode generation makes kernel reject the fs.
May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@
In initramfs I can try to mount but get something like btrfs critical corrupt leaf invalid inode generation
open_ctreefailedQuoted 8 lines
Quoted 6 lines
Booting old kernel works just as before, no errors.
Quoted 2 lines
According to the creation time, the inode is created by some 2014 kernel.
How do I get the creation time?
# btrfs ins dump-tree -b <the bytenr reported by kernel> <your device>I just went back to the office to reboot to 5.3 and check the creation times and found they were 2013 - 2014.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
And the generation member of
INODE_ITEMis not updated (unlike the transid member) so the error persists until latest tree-checker detects.Even the situation can be fixed by reverting back to older kernel and
copying the offending dir/file to another inode and delete the offending
one, it still should be done by btrfs-progs.Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
How to find the offending dir/file from the command line manually?
# find <mount point> -inum <inode number>This works, thanks.
But appears unpractical. After fix 2 files and reboot, I found 4 more, then 16, then I gave up.
Another solution is use "find" to locate the files with creation time before 2015, and copy them to a new file, then replace the old file with the new file.
Quoted 5 lines
Quoted 2 lines
Hmm. But how do I "find" by creation time (otime)? Do you have a suggestion for this?
$ touch -t 201501010000 /tmp/sample $ find <mnt> -not -cnewer /tmp/sample
AFAIK this compares file modified date with status changed date. So, no search for creation date.
And stat /tmp/sample (sorry dutch lang output):
ferry@ferry-quad:~$ stat /tmp/sample Bestand: /tmp/sample Grootte: 0 Blokken: 0 IO-blok: 4096 leeg normaal bestand Apparaat: 1bh/27d Inode: 62005381 Koppelingen: 1 Toegang: (0664/-rw-rw-r--) UID: ( 1001/ ferry) GID: ( 1001/ ferry) Toegang: 2015-01-01 00:00:00.000000000 +0100 Gewijzigd: 2015-01-01 00:00:00.000000000 +0100 Veranderd: 2019-10-20 15:20:50.366163766 +0200 Ontstaan: -
Quoted 85 lines
If you want, you can add -exec to that find, but I'd only add that after confirming the execution load is verified.
Thanks,
QuQuoted 79 lines
Quoted 78 lines
It would be much safer than btrfs check --repair.
Thanks,
QuQuoted 70 lines
Quoted 36 lines
Quoted 35 lines
Thanks,
QuQuoted 31 lines
Quoted 30 lines
This patchset adds such check and repair ability to btrfs-check, with a
simple test image.Qu Wenruo (3):
btrfs-progs: check/lowmem: Add check and repair for invalid
inode
generation
btrfs-progs: check/original: Add check and repair for
invalid inode
generation
btrfs-progs: fsck-tests: Add test image for invalid inode
generation
repaircheck/main.c | 50 +++++++++++- check/mode-lowmem.c | 76 ++++++++++++++++++ check/mode-original.h | 1 + .../.lowmem_repairable | 0 .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes 5 files changed, 126 insertions(+), 1 deletion(-)create mode 100644
tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
create mode 100644
tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xzQuoted 3 lines
I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my mounted rootfs with linux 5.0 and captured the log (~1800 lines 209 errors).
It's really not recommended to run btrfs check, especially repair on the mounted fs, unless it's RO.
A new transaction from kernel can easily screw up the repaired fs.
Quoted 5 lines
I'm not sure if using the v5.0 kernel and/or checking mounted distorts the results? Else I'm going to need a live usb with a v5.3 kernel and v5.3 btrfs-progs.
Quoted 8 lines
If you like I can share the log. Let me know.
This issue can potentially cause a lot of grief. Our company server runs
Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with ~100 snapshots. I guess the problematic inodes need to be fixed on each snapshot prior to upgrading to 20.04 LTS (which might be on kernel ~5.6)?Yes.
Quoted 4 lines
Do I understand correctly that this FTB is caused by more strict checking of the fs by the kernel, while the tools to fix the detected corruptions are not yet released?
Yes.
Thanks,
Qu
Quoted 82 lines · Ferry Toth
Op 20-10-2019 om 15:15 schreef Qu WenRuo:
Quoted 70 lines · Ferry Toth
Quoted 67 lines
Op 20-10-2019 om 02:51 schreef Qu Wenruo:
Quoted 65 lines · Qu Wenruo
Quoted 50 lines · Ferry Toth
Quoted 47 lines
Hi,
Op 19-10-2019 om 01:50 schreef Qu WenRuo:
Quoted 19 lines · Ferry Toth
Quoted 16 lines
Op 24-09-2019 om 10:11 schreef Qu Wenruo:
Quoted 2 lines
We have at least two user reports about bad inode generation makes kernel reject the fs.
May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@
In initramfs I can try to mount but get something like btrfs critical corrupt leaf invalid inode generation
open_ctreefailedBooting old kernel works just as before, no errors.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
According to the creation time, the inode is created by some 2014 kernel.
How do I get the creation time?
# btrfs ins dump-tree -b <the bytenr reported by kernel> <your device>I just went back to the office to reboot to 5.3 and check the creation
times and found they were 2013 - 2014.Quoted 13 lines
Quoted 12 lines
Quoted 11 lines
And the generation member of
INODE_ITEMis not updated (unlike the transid member) so the error persists until latest tree-checker detects.Even the situation can be fixed by reverting back to older kernel and
copying the offending dir/file to another inode and delete the offending
one, it still should be done by btrfs-progs.How to find the offending dir/file from the command line manually?
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
# find <mount point> -inum <inode number>This works, thanks.
But appears unpractical. After fix 2 files and reboot, I found 4 more,
then 16, then I gave up.Another solution is use "find" to locate the files with creation time before 2015, and copy them to a new file, then replace the old file with
the new file.Quoted 5 lines
Quoted 2 lines
Hmm. But how do I "find" by creation time (otime)? Do you have a suggestion for this?
$ touch -t 201501010000 /tmp/sample $ find <mnt> -not -cnewer /tmp/sampleAFAIK this compares file modified date with status changed date. So, no search for creation date.
And stat
/tmp/sample(sorry dutch lang output):
Quoted 10 lines
ferry@ferry-quad:~$ stat /tmp/sample Bestand: /tmp/sample Grootte: 0 Blokken: 0 IO-blok: 4096 leeg normaal bestand Apparaat: 1bh/27d Inode: 62005381 Koppelingen: 1 Toegang: (0664/-rw-rw-r--) UID: ( 1001/ ferry) GID: ( 1001/ ferry) Toegang: 2015-01-01 00:00:00.000000000 +0100 Gewijzigd: 2015-01-01 00:00:00.000000000 +0100 Veranderd: 2019-10-20 15:20:50.366163766 +0200 Ontstaan: -
My bad, always got confused by o/a/c/mtime, as c really looks like *c* reation, so I always got confused between ctime and otime.
Then considering not all fs supports otime, find doesn't support that. I guess it's only possible by other tools....
BTW, did you find any patterns in those existing offending inodes? I guess it would be faster than finding a tool supporting otime search.
Thanks,
Qu
Quoted 93 lines
Quoted 92 lines
If you want, you can add -exec to that find, but I'd only add that after confirming the execution load is verified.
Thanks,
QuQuoted 84 lines
Quoted 83 lines
It would be much safer than btrfs check --repair.
Thanks,
QuQuoted 75 lines
Quoted 38 lines
Quoted 37 lines
Thanks,
QuQuoted 33 lines
Quoted 32 lines
This patchset adds such check and repair ability to btrfs-check, with a
simple test image.Qu Wenruo (3):
btrfs-progs: check/lowmem: Add check and repair for invalid inode
generation
btrfs-progs: check/original: Add check and repair for invalid inode
generation
btrfs-progs: fsck-tests: Add test image for invalid inode generation
repaircheck/main.c | 50
+++++++++++-
check/mode-lowmem.c | 76
++++++++++++++++++
check/mode-original.h | 1 + .../.lowmem_repairable | 0 .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes5 files changed, 126 insertions(+), 1 deletion(-)create mode 100644 tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable create mode 100644 tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
Quoted 3 lines
I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my mounted rootfs with linux 5.0 and captured the log (~1800 lines 209 errors).
It's really not recommended to run btrfs check, especially repair on the
mounted fs, unless it's RO.A new transaction from kernel can easily screw up the repaired fs.
Quoted 8 lines
I'm not sure if using the v5.0 kernel and/or checking mounted distorts
the results? Else I'm going to need a live usb with a v5.3 kernel and v5.3 btrfs-progs.If you like I can share the log. Let me know.
Quoted 7 lines
This issue can potentially cause a lot of grief. Our company server runs
Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with ~100 snapshots. I guess the problematic inodes need to be fixed on each
snapshot prior to upgrading to 20.04 LTS (which might be on kernel ~5.6)?Yes.
Quoted 4 lines
Do I understand correctly that this FTB is caused by more strict checking of the fs by the kernel, while the tools to fix the detected corruptions are not yet released?
Yes.
Thanks,
Qu
Op 20-10-2019 om 16:11 schreef Qu Wenruo:
Quoted 96 lines · Ferry Toth
Quoted 82 lines
Op 20-10-2019 om 15:15 schreef Qu WenRuo:
Quoted 70 lines · Ferry Toth
Quoted 67 lines
Op 20-10-2019 om 02:51 schreef Qu Wenruo:
Quoted 65 lines · Qu Wenruo
Quoted 50 lines · Ferry Toth
Quoted 47 lines
Hi,
Op 19-10-2019 om 01:50 schreef Qu WenRuo:
Quoted 19 lines · Ferry Toth
Quoted 16 lines
Op 24-09-2019 om 10:11 schreef Qu Wenruo:
Quoted 2 lines
We have at least two user reports about bad inode generation makes kernel reject the fs.
May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@
In initramfs I can try to mount but get something like btrfs critical corrupt leaf invalid inode generation
open_ctreefailedBooting old kernel works just as before, no errors.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
According to the creation time, the inode is created by some 2014 kernel.
How do I get the creation time?
# btrfs ins dump-tree -b <the bytenr reported by kernel> <your device>I just went back to the office to reboot to 5.3 and check the creation
times and found they were 2013 - 2014.Quoted 13 lines
Quoted 12 lines
Quoted 11 lines
And the generation member of
INODE_ITEMis not updated (unlike the transid member) so the error persists until latest tree-checker detects.Even the situation can be fixed by reverting back to older kernel and
copying the offending dir/file to another inode and delete the offending
one, it still should be done by btrfs-progs.How to find the offending dir/file from the command line manually?
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
# find <mount point> -inum <inode number>This works, thanks.
But appears unpractical. After fix 2 files and reboot, I found 4 more,
then 16, then I gave up.Another solution is use "find" to locate the files with creation time before 2015, and copy them to a new file, then replace the old file with
the new file.Quoted 5 lines
Quoted 2 lines
Hmm. But how do I "find" by creation time (otime)? Do you have a suggestion for this?
$ touch -t 201501010000 /tmp/sample $ find <mnt> -not -cnewer /tmp/sampleAFAIK this compares file modified date with status changed date. So, no search for creation date.
And stat
/tmp/sample(sorry dutch lang output):Quoted 10 lines
ferry@ferry-quad:~$ stat /tmp/sample Bestand: /tmp/sample Grootte: 0 Blokken: 0 IO-blok: 4096 leeg normaal bestand Apparaat: 1bh/27d Inode: 62005381 Koppelingen: 1 Toegang: (0664/-rw-rw-r--) UID: ( 1001/ ferry) GID: ( 1001/ ferry) Toegang: 2015-01-01 00:00:00.000000000 +0100 Gewijzigd: 2015-01-01 00:00:00.000000000 +0100 Veranderd: 2019-10-20 15:20:50.366163766 +0200 Ontstaan: -
Quoted 5 lines
My bad, always got confused by o/a/c/mtime, as c really looks like *c* reation, so I always got confused between ctime and otime.
Then considering not all fs supports otime, find doesn't support that. I guess it's only possible by other tools....
New stat will support it, but not in Ubuntu 19.10. We did find this:
https://github.com/torvalds/linux/blob/master/samples/vfs/test-statx.c
and are trying to work out a script that will gzip all files with creation < 2015. Then we can unzip again and overwrite.
Quoted 2 lines
BTW, did you find any patterns in those existing offending inodes? I guess it would be faster than finding a tool supporting otime search.
I didn't see any logic. A mix of logs, cached files, journal files etc. + a .kde directory in the /
Quoted 97 lines
Thanks,
QuQuoted 93 lines
Quoted 92 lines
If you want, you can add -exec to that find, but I'd only add that after confirming the execution load is verified.
Thanks,
QuQuoted 84 lines
Quoted 83 lines
It would be much safer than btrfs check --repair.
Thanks,
QuQuoted 75 lines
Quoted 38 lines
Quoted 37 lines
Thanks,
QuQuoted 33 lines
Quoted 32 lines
This patchset adds such check and repair ability to btrfs-check, with a
simple test image.Qu Wenruo (3):
btrfs-progs: check/lowmem: Add check and repair for invalid
inode
generation
btrfs-progs: check/original: Add check and repair for
invalid inode
generation
btrfs-progs: fsck-tests: Add test image for invalid inode
generation
repaircheck/main.c | 50 +++++++++++- check/mode-lowmem.c | 76 ++++++++++++++++++ check/mode-original.h | 1 + .../.lowmem_repairable | 0 .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes 5 files changed, 126 insertions(+), 1 deletion(-)create mode 100644
tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
create mode 100644
tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xzQuoted 3 lines
I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my mounted rootfs with linux 5.0 and captured the log (~1800 lines 209 errors).
It's really not recommended to run btrfs check, especially repair on the
mounted fs, unless it's RO.A new transaction from kernel can easily screw up the repaired fs.
Quoted 8 lines
I'm not sure if using the v5.0 kernel and/or checking mounted distorts
the results? Else I'm going to need a live usb with a v5.3 kernel and v5.3 btrfs-progs.If you like I can share the log. Let me know.
Quoted 7 lines
This issue can potentially cause a lot of grief. Our company server runs
Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with ~100 snapshots. I guess the problematic inodes need to be fixed on each
snapshot prior to upgrading to 20.04 LTS (which might be on kernel ~5.6)?Yes.
Quoted 4 lines
Do I understand correctly that this FTB is caused by more strict checking of the fs by the kernel, while the tools to fix the detected corruptions are not yet released?
Yes.
Thanks,
Qu
Op 20-10-2019 om 16:24 schreef Ferry Toth:
Quoted 100 lines
Op 20-10-2019 om 16:11 schreef Qu Wenruo:
Quoted 99 lines · Ferry Toth
Quoted 82 lines
Op 20-10-2019 om 15:15 schreef Qu WenRuo:
Quoted 70 lines · Ferry Toth
Quoted 67 lines
Op 20-10-2019 om 02:51 schreef Qu Wenruo:
Quoted 65 lines · Qu Wenruo
Quoted 50 lines · Ferry Toth
Quoted 47 lines
Hi,
Op 19-10-2019 om 01:50 schreef Qu WenRuo:
Quoted 19 lines · Ferry Toth
Quoted 16 lines
Op 24-09-2019 om 10:11 schreef Qu Wenruo:
Quoted 2 lines
We have at least two user reports about bad inode generation makes kernel reject the fs.
May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@
In initramfs I can try to mount but get something like btrfs critical corrupt leaf invalid inode generation
open_ctreefailedBooting old kernel works just as before, no errors.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
According to the creation time, the inode is created by some 2014 kernel.
How do I get the creation time?
# btrfs ins dump-tree -b <the bytenr reported by kernel> <your device>I just went back to the office to reboot to 5.3 and check the creation
times and found they were 2013 - 2014.Quoted 13 lines
Quoted 12 lines
Quoted 11 lines
And the generation member of
INODE_ITEMis not updated (unlike the transid member) so the error persists until latest tree-checker detects.Even the situation can be fixed by reverting back to older kernel and
copying the offending dir/file to another inode and delete the offending
one, it still should be done by btrfs-progs.How to find the offending dir/file from the command line manually?
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
# find <mount point> -inum <inode number>This works, thanks.
But appears unpractical. After fix 2 files and reboot, I found 4 more,
then 16, then I gave up.Another solution is use "find" to locate the files with creation time before 2015, and copy them to a new file, then replace the old file with
the new file.Quoted 5 lines
Quoted 2 lines
Hmm. But how do I "find" by creation time (otime)? Do you have a suggestion for this?
$ touch -t 201501010000 /tmp/sample $ find <mnt> -not -cnewer /tmp/sampleAFAIK this compares file modified date with status changed date. So, no search for creation date.
And stat
/tmp/sample(sorry dutch lang output):Quoted 10 lines
ferry@ferry-quad:~$ stat /tmp/sample Bestand: /tmp/sample Grootte: 0 Blokken: 0 IO-blok: 4096 leeg normaal bestand Apparaat: 1bh/27d Inode: 62005381 Koppelingen: 1 Toegang: (0664/-rw-rw-r--) UID: ( 1001/ ferry) GID: ( 1001/ ferry) Toegang: 2015-01-01 00:00:00.000000000 +0100 Gewijzigd: 2015-01-01 00:00:00.000000000 +0100 Veranderd: 2019-10-20 15:20:50.366163766 +0200 Ontstaan: -My bad, always got confused by o/a/c/mtime, as c really looks like *c* reation, so I always got confused between ctime and otime.
Quoted 7 lines
Quoted 2 lines
Then considering not all fs supports otime, find doesn't support that. I guess it's only possible by other tools....
New stat will support it, but not in Ubuntu 19.10. We did find this:
https://github.com/torvalds/linux/blob/master/samples/vfs/test-statx.cand are trying to work out a script that will gzip all files with creation < 2015. Then we can unzip again and overwrite.
So we combined find + test-statx + awk to produce a list of files with creation date 1970 that are then fed to tar. Finally we untar that to overwrite the original files.
I found the following:
- the problematic files are both ordinary and symlinks
- they all seem to have the same inode generation
- all had no creation date (so were set to jan 1970)
- not all files without creation date were problematic (only ~10%)
- modified dates were 2013/2014 afaikt
The procedure keeps the modified date, but sets creation date to today, creates a new inode and fixes the error.
Quoted 106 lines
Quoted 2 lines
BTW, did you find any patterns in those existing offending inodes? I guess it would be faster than finding a tool supporting otime search.
I didn't see any logic. A mix of logs, cached files, journal files etc.
+ a .kde directory in the /Quoted 99 lines
Thanks,
QuQuoted 95 lines
Quoted 94 lines
If you want, you can add -exec to that find, but I'd only add that after confirming the execution load is verified.
Thanks,
QuQuoted 86 lines
Quoted 85 lines
It would be much safer than btrfs check --repair.
Thanks,
QuQuoted 77 lines
Quoted 40 lines
Quoted 39 lines
Thanks,
QuQuoted 35 lines
Quoted 34 lines
This patchset adds such check and repair ability to btrfs-check, with a
simple test image.Qu Wenruo (3):
btrfs-progs: check/lowmem: Add check and repair for invalid inode
generation
btrfs-progs: check/original: Add check and repair for invalid inode
generation
btrfs-progs: fsck-tests: Add test image for invalid inode generation
repaircheck/main.c | 50
+++++++++++-
check/mode-lowmem.c | 76
++++++++++++++++++
check/mode-original.h | 1 + .../.lowmem_repairable | 0 .../bad_inode_geneartion.img.xz | Bin 0 -> 2012
bytes5 files changed, 126 insertions(+), 1 deletion(-)create mode 100644 tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable create mode 100644 tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
Quoted 3 lines
I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my mounted rootfs with linux 5.0 and captured the log (~1800 lines 209 errors).
It's really not recommended to run btrfs check, especially repair on the
mounted fs, unless it's RO.A new transaction from kernel can easily screw up the repaired fs.
Quoted 8 lines
I'm not sure if using the v5.0 kernel and/or checking mounted distorts
the results? Else I'm going to need a live usb with a v5.3 kernel and v5.3 btrfs-progs.If you like I can share the log. Let me know.
Quoted 7 lines
This issue can potentially cause a lot of grief. Our company server runs
Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with ~100 snapshots. I guess the problematic inodes need to be fixed on each
snapshot prior to upgrading to 20.04 LTS (which might be on kernel ~5.6)?Yes.
Quoted 4 lines
Do I understand correctly that this FTB is caused by more strict checking of the fs by the kernel, while the tools to fix the detected corruptions are not yet released?
Yes.
Thanks,
Qu
Op 20-10-2019 om 02:26 schreef Qu Wenruo:
Quoted 82 lines · Ferry Toth
Quoted 79 lines
Hi,
Op 19-10-2019 om 01:50 schreef Qu WenRuo:
Quoted 15 lines · Ferry Toth
Quoted 12 lines
Op 24-09-2019 om 10:11 schreef Qu Wenruo:
Quoted 2 lines
We have at least two user reports about bad inode generation makes kernel reject the fs.
May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@
In initramfs I can try to mount but get something like btrfs critical corrupt leaf invalid inode generation
open_ctreefailedQuoted 8 lines
Quoted 6 lines
Booting old kernel works just as before, no errors.
Quoted 2 lines
According to the creation time, the inode is created by some 2014 kernel.
How do I get the creation time?
# btrfs ins dump-tree -b <the bytenr reported by kernel> <your device>I just went back to the office to reboot to 5.3 and check the creation times and found they were 2013 - 2014.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
And the generation member of
INODE_ITEMis not updated (unlike the transid member) so the error persists until latest tree-checker detects.Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Even the situation can be fixed by reverting back to older kernel and copying the offending dir/file to another inode and delete the offending
one, it still should be done by btrfs-progs.How to find the offending dir/file from the command line manually?
# find <mount point> -inum <inode number>This works, thanks.
But appears unpractical. After fix 2 files and reboot, I found 4 more, then 16, then I gave up.
Quoted 29 lines
Thanks,
QuQuoted 26 lines
Quoted 25 lines
This patchset adds such check and repair ability to btrfs-check, with a simple test image.
Qu Wenruo (3):
btrfs-progs: check/lowmem: Add check and repair for invalid inode generation
btrfs-progs: check/original: Add check and repair for invalid inode generation
btrfs-progs: fsck-tests: Add test image for invalid inode
generation
repaircheck/main.c | 50 +++++++++++- check/mode-lowmem.c | 76 ++++++++++++++++++ check/mode-original.h | 1 + .../.lowmem_repairable | 0 .../bad_inode_geneartion.img.xz | Bin 0 -> 2012 bytes 5 files changed, 126 insertions(+), 1 deletion(-)create mode 100644
tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
create mode 100644
tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
Quoted 6 lines
Quoted 3 lines
I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my mounted rootfs with linux 5.0 and captured the log (~1800 lines 209 errors).
It's really not recommended to run btrfs check, especially repair on the mounted fs, unless it's RO.
Yes, I know. As the fs is mounted btrfs refuses to repair it.
So I only forced to skip the mount check, but no repair has been done, only the check.
Quoted 25 lines
A new transaction from kernel can easily screw up the repaired fs.
Quoted 11 lines
I'm not sure if using the v5.0 kernel and/or checking mounted distorts the results? Else I'm going to need a live usb with a v5.3 kernel and v5.3 btrfs-progs.
If you like I can share the log. Let me know.
This issue can potentially cause a lot of grief. Our company server runs Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with ~100 snapshots. I guess the problematic inodes need to be fixed on each snapshot prior to upgrading to 20.04 LTS (which might be on kernel ~5.6)?
Yes.
Quoted 4 lines
Do I understand correctly that this FTB is caused by more strict checking of the fs by the kernel, while the tools to fix the detected corruptions are not yet released?
Yes.
Thanks,
Qu