btrfs-progs: detect and fix orphan roots without orphan items
cover btrfs-progs: detect and fix orphan roots without orphan items
There is bug report from Zhenyu Wu <[email protected]>, where even there is only one subvolume, his filesystem still consumes way more space than expected.
The problem is, there are 58 snapshots, but all of them have no ROOT_REF/ROOT_BACKREF, and all their root refs is 0.
But strangely there is no orphan items for them.
This means, btrfs kernel module won't detect them as orphan nor queue them to be removed.
"btrfs subvolume delete -i" do no helps, as it still requires ROOT_BACKREF to resolve the dentry.
For now, we should teach btrfs check to detect and repair the case by adding new orphan items for them.
The root cause I guess is some strange behavior for orphan item insert and subvolume unlink.
As the report mentioned that he had one forced power off.
Qu Wenruo (3):
btrfs-progs: check/lowmem: add the ability to detect and repair orphan
subvolume trees which doesn't have orphan item
btrfs-progs: check/original: detect and repair orphan subvolume
without orphan item
btrfs-progs: fsck-tests: add test image for orphan roots without an
orphan item
check/main.c | 37 ++++-------- check/mode-common.c | 56 ++++++++++++++++++ check/mode-common.h | 3 + check/mode-lowmem.c | 42 +++++++------ .../048-orphan-roots/.lowmem_repairable | 0 .../048-orphan-roots/default.img.xz | Bin 0 -> 2584 bytes 6 files changed, 93 insertions(+), 45 deletions(-) create mode 100644 tests/fsck-tests/048-orphan-roots/.lowmem_repairable create mode 100644 tests/fsck-tests/048-orphan-roots/default.img.xz
Signature · 1 line
2.32.0
PATCH 1/3 btrfs-progs: check/lowmem: add the ability to detect and repair orphan subvolume trees which doesn't have orphan item
There is a bug report from the mail list that, after certain btrfs-zero-log operation, the filesystem has inaccessible subvolumes, and there is no way to delete them.
Such subvolumes have no ROOT_REF/ROOT_BACKREF, and their root_refs is 0. Without an orphan item, kernel won't detect them as orphan, thus no cleanup will happen.
Btrfs check won't report this as a problem either.
Such ghost subvolumes will just waste disk space, and can be pretty hard to detect without proper btrfs check support.
For the repair part, we just add orphan item so later kernel can handle it.
Since the check for orphan item and repair can be reused between original and lowmem mode, move those functions to mode-common.[ch].
--- check/mode-common.c | 56 +++++++++++++++++++++++++++++++++++++++++++++ check/mode-common.h | 3 +++ check/mode-lowmem.c | 42 +++++++++++++++++++--------------- 3 files changed, 82 insertions(+), 19 deletions(-)
check/mode-common.c +56 -0
@@ -1243,3 +1243,59 @@ out:btrfs_release_path (&path);return ret; } + +/* +* Check if we have orphan item for @objectid. +* +* Note: if something wrong happened during search, we consider it as no orphan +* item. +*/ +bool btrfs_has_orphan_item (struct btrfs_root *root,u64 objectid) +{ +struct btrfs_path path; +struct btrfs_key key; +int ret; + +btrfs_init_path (&path); + key.objectid =BTRFS_ORPHAN_OBJECTID ; + key.type =BTRFS_ORPHAN_ITEM_KEY ; + key.offset = objectid; + ret =btrfs_search_slot (NULL , root, &key, &path,0 ,0 ); +btrfs_release_path (&path); +if (ret ==0 ) +return true; +return false; +} + +int repair_root_orphan_item (struct btrfs_fs_info *fs_info,u64 rootid) +{ +struct btrfs_path path; +struct btrfs_trans_handle *trans; +int ret; + +btrfs_init_path (&path); + + trans =btrfs_start_transaction (fs_info->tree_root,1 ); +if (IS_ERR (trans)) { + ret =PTR_ERR (trans); + errno = -ret; +error ("failed to start transaction: %m" ); +return ret; + } + ret =btrfs_add_orphan_item (trans, fs_info->tree_root, &path, rootid); +if (ret <0 ) { + errno = -ret; +error ("failed to insert orphan item: %m" ); + goto error; + } +btrfs_release_path (&path); + ret =btrfs_commit_transaction (trans, fs_info->tree_root); +if (ret <0 ) + goto error; +printf ("Inserted orphan root item for root %llu\n" , rootid); +return ret; +error: +btrfs_release_path (&path); +btrfs_abort_transaction (trans, ret); +return ret; +}
check/mode-common.h +3 -0
@@ -192,4 +192,7 @@ static inline void btrfs_check_subpage_eb_alignment(u64 start, u32 len) start, start + len); } +bool btrfs_has_orphan_item (struct btrfs_root *root,u64 objectid); +int repair_root_orphan_item (struct btrfs_fs_info *fs_info,u64 rootid); +#endif
check/mode-lowmem.c +23 -19
60 lines not on this page · raw message
Signature · 1 line
2.32.0
Re: [PATCH 1/3] btrfs-progs: check/lowmem: add the ability to detect and repair orphan subvolume trees which doesn't have orphan item
Quoted 9 lines · Qu Wenruo
There is a bug report from the mail list that, after certain btrfs-zero-log operation, the filesystem has inaccessible subvolumes, and there is no way to delete them.
Such subvolumes have no ROOT_REF/ROOT_BACKREF, and their
root_refsis 0.Without an orphan item, kernel won't detect them as orphan, thus no cleanup will happen.
Changes here looks good. But what if, in another case, the ghost subvolume isn't an orphan but a corruption? is it possible?
Thanks, Anand
Quoted 163 lines
Btrfs check won't report this as a problem either.
Such ghost subvolumes will just waste disk space, and can be pretty hard to detect without proper btrfs check support.
For the repair part, we just add orphan item so later kernel can handle it.
Since the check for orphan item and repair can be reused between original and lowmem mode, move those functions to mode-common.[ch].
Reported-by Zhenyu Wu <[email protected]>Signed-off-by Qu Wenruo <[email protected]>--- check/mode-common.c | 56 +++++++++++++++++++++++++++++++++++++++++++++ check/mode-common.h | 3 +++ check/mode-lowmem.c | 42 +++++++++++++++++++--------------- 3 files changed, 82 insertions(+), 19 deletions(-)check/mode-common.c +56
-0@@ -1243,3 +1243,59 @@ out:btrfs_release_path (&path);return ret; } + +/* +* Check if we have orphan item for @objectid. +* +* Note: if something wrong happened during search, we consider it as no orphan +* item. +*/ +bool btrfs_has_orphan_item (struct btrfs_root *root,u64 objectid) +{ +struct btrfs_path path; +struct btrfs_key key; +int ret; + +btrfs_init_path (&path); + key.objectid =BTRFS_ORPHAN_OBJECTID ; + key.type =BTRFS_ORPHAN_ITEM_KEY ; + key.offset = objectid; + ret =btrfs_search_slot (NULL , root, &key, &path,0 ,0 ); +btrfs_release_path (&path); +if (ret ==0 ) +return true; +return false; +} + +int repair_root_orphan_item (struct btrfs_fs_info *fs_info,u64 rootid) +{ +struct btrfs_path path; +struct btrfs_trans_handle *trans; +int ret; + +btrfs_init_path (&path); + + trans =btrfs_start_transaction (fs_info->tree_root,1 ); +if (IS_ERR (trans)) { + ret =PTR_ERR (trans); + errno = -ret; +error ("failed to start transaction: %m" ); +return ret; + } + ret =btrfs_add_orphan_item (trans, fs_info->tree_root, &path, rootid); +if (ret <0 ) { + errno = -ret; +error ("failed to insert orphan item: %m" ); + goto error; + } +btrfs_release_path (&path); + ret =btrfs_commit_transaction (trans, fs_info->tree_root); +if (ret <0 ) + goto error; +printf ("Inserted orphan root item for root %llu\n" , rootid); +return ret; +error: +btrfs_release_path (&path); +btrfs_abort_transaction (trans, ret); +return ret; +}check/mode-common.h +3
-0@@ -192,4 +192,7 @@ static inline void btrfs_check_subpage_eb_alignment(u64 start, u32 len) start, start + len); } +bool btrfs_has_orphan_item (struct btrfs_root *root,u64 objectid); +int repair_root_orphan_item (struct btrfs_fs_info *fs_info,u64 rootid); +#endif check/mode-lowmem.c +23
-19@@ -2500,24 +2500,6 @@ out:return ret; } -static bool has_orphan_item (struct btrfs_root *root,u64 ino) -{ -struct btrfs_path path; -struct btrfs_key key; -int ret; - -btrfs_init_path (&path); - key.objectid =BTRFS_ORPHAN_OBJECTID ; - key.type =BTRFS_ORPHAN_ITEM_KEY ; - key.offset = ino; - - ret =btrfs_search_slot (NULL , root, &key, &path,0 ,0 ); -btrfs_release_path (&path); -if (ret ==0 ) -return true; -return false; -} -static int repair_inode_gen_lowmem (struct btrfs_root *root,struct btrfs_path *path) {@@ -2622,7 +2604,7 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path)return err; } - is_orphan =has_orphan_item (root, inode_id); + is_orphan =btrfs_has_orphan_item (root, inode_id); ii = btrfs_item_ptr(node, slot, struct btrfs_inode_item); isize =btrfs_inode_size (node, ii); nbytes =btrfs_inode_nbytes (node, ii);@@ -5209,6 +5191,28 @@ static int check_btrfs_root(struct btrfs_root *root, int check_all) } } } + +/* Make sure orphan subvolume tree has proper orphan item for it */ +if (btrfs_root_refs (root_item) ==0 && +is_fstree (root->root_key.objectid)) { +bool is_orphan; + is_orphan =btrfs_has_orphan_item (root->fs_info->tree_root, + root->root_key.objectid); + +if (!is_orphan) { +error ("orphan root %llu doesn't have orphan item" , + root->root_key.objectid); +if (repair) { + ret =repair_root_orphan_item (root->fs_info, + root->root_key.objectid); +if (!ret) + is_orphan = true; + } +if (!is_orphan) + err |=ORPHAN_ITEM ; + } + } +if (btrfs_root_refs (root_item) >0 ||btrfs_disk_key_objectid (&root_item->drop_progress) ==0 ) { path.nodes[level] = root->node;
Re: [PATCH 1/3] btrfs-progs: check/lowmem: add the ability to detect and repair orphan subvolume trees which doesn't have orphan item
Quoted 5 lines · Anand Jain
Quoted 4 lines · Qu Wenruo
There is a bug report from the mail list that, after certain btrfs-zero-log operation, the filesystem has inaccessible subvolumes, and there is no way to delete them.
Quoted 8 lines
Quoted 5 lines
Such subvolumes have no ROOT_REF/ROOT_BACKREF, and their
root_refsis 0.Without an orphan item, kernel won't detect them as orphan, thus no cleanup will happen.
Changes here looks good. But what if, in another case, the ghost subvolume isn't an orphan but a corruption? is it possible?
Firstly it needs to pass all the other root item check, including ROOT_BACKREF/ROOT_REF, and DIR_INDEX/DIR_ITEM.
This means if there is any ROOT_BACKREF/ROOT_REF, or DIR_INDEX/DIR_ITEM referring to this inode, it should not be repaired.
Furthermore, after passing above checks, it must have a root refs as 0.
With all conditions met, it's really a ghost subvolume.
In fact I have to do all above works to craft such test case, thus it's not resulted by some simple corruption.
Thanks,
Qu
Quoted 170 lines
Thanks, Anand
Quoted 165 lines
Btrfs check won't report this as a problem either.
Such ghost subvolumes will just waste disk space, and can be pretty hard to detect without proper btrfs check support.
For the repair part, we just add orphan item so later kernel can handle it.
Since the check for orphan item and repair can be reused between original and lowmem mode, move those functions to mode-common.[ch].
Reported-by Zhenyu Wu <[email protected]>Signed-off-by Qu Wenruo <[email protected]>---
check/mode-common.c | 56 +++++++++++++++++++++++++++++++++++++++++++++ check/mode-common.h | 3 +++ check/mode-lowmem.c | 42 +++++++++++++++++++--------------- 3 files changed, 82 insertions(+), 19 deletions(-)check/mode-common.c +5
-0@@ -1243,3 +1243,59 @@ out: btrfs_release_path(&path); return ret; } + +/* +* Check if we have orphan item for @objectid. +* +* Note: if something wrong happened during search, we consider it as no orphan
+ * item.
+ */check/mode-common.c +33
-0+bool btrfs_has_orphan_item (struct btrfs_root *root,u64 objectid) +{ + struct btrfs_path path; + struct btrfs_key key; + int ret; + + btrfs_init_path(&path); + key.objectid = BTRFS_ORPHAN_OBJECTID; + key.type = BTRFS_ORPHAN_ITEM_KEY; + key.offset = objectid; + ret = btrfs_search_slot(NULL, root, &key, &path, 0, 0); + btrfs_release_path(&path); + if (ret == 0) + return true; + return false; +} + +int repair_root_orphan_item (struct btrfs_fs_info *fs_info,u64 rootid) +{ + struct btrfs_path path; + struct btrfs_trans_handle *trans; + int ret; + + btrfs_init_path(&path); + + trans = btrfs_start_transaction(fs_info->tree_root, 1); + if (IS_ERR(trans)) { + ret = PTR_ERR(trans); + errno = -ret; + error("failed to start transaction: %m"); + return ret; + } + ret = btrfs_add_orphan_item(trans, fs_info->tree_root, &path,rootid);
+ if (ret < 0) {
+ errno = -ret;
+error("failed to insert orphan item: %m");
+ goto error;
+ }
+btrfs_release_path(&path);
+ ret =btrfs_commit_transaction(trans, fs_info->tree_root); + if (ret < 0)
+ goto error;
+printf("Inserted orphan root item for root %llu\n", rootid); + return ret;check/mode-common.c +5
-0+error: + btrfs_release_path(&path); + btrfs_abort_transaction(trans, ret); + return ret; +}check/mode-common.h +3
-0@@ -192,4 +192,7 @@ static inline void btrfs_check_subpage_eb_alignment(u64 start, u32 len) start, start + len); } +bool btrfs_has_orphan_item (struct btrfs_root *root,u64 objectid); +int repair_root_orphan_item (struct btrfs_fs_info *fs_info,u64 rootid); + #endifcheck/mode-lowmem.c +23
-19@@ -2500,24 +2500,6 @@ out: return ret; } -static bool has_orphan_item (struct btrfs_root *root,u64 ino) -{ - struct btrfs_path path; - struct btrfs_key key; - int ret; - - btrfs_init_path(&path); - key.objectid = BTRFS_ORPHAN_OBJECTID; - key.type = BTRFS_ORPHAN_ITEM_KEY; - key.offset = ino; - - ret = btrfs_search_slot(NULL, root, &key, &path, 0, 0); - btrfs_release_path(&path); - if (ret == 0) - return true; - return false; -} - static int repair_inode_gen_lowmem(struct btrfs_root *root, struct btrfs_path *path) {@@ -2622,7 +2604,7 @@ static int check_inode_item(struct btrfs_root *root, struct btrfs_path *path) return err; } - is_orphan = has_orphan_item(root, inode_id); + is_orphan = btrfs_has_orphan_item(root, inode_id); ii = btrfs_item_ptr(node, slot, struct btrfs_inode_item); isize = btrfs_inode_size(node, ii); nbytes = btrfs_inode_nbytes(node, ii);@@ -5209,6 +5191,28 @@ static int check_btrfs_root(struct btrfs_root *root, int check_all) } } } + + /* Make sure orphan subvolume tree has proper orphan item for it */ + if (btrfs_root_refs(root_item) == 0 && + is_fstree(root->root_key.objectid)) { + bool is_orphan; + is_orphan = btrfs_has_orphan_item(root->fs_info->tree_root, + root->root_key.objectid); + + if (!is_orphan) { + error("orphan root %llu doesn't have orphan item", + root->root_key.objectid); + if (repair) { + ret = repair_root_orphan_item(root->fs_info, + root->root_key.objectid); + if (!ret) + is_orphan = true; + } + if (!is_orphan) + err |= ORPHAN_ITEM; + } + } +if (
btrfs_root_refs(root_item)> 0 ||btrfs_disk_key_objectid(&root_item->drop_progress)== 0) { path.nodes[level] =root->node;
PATCH 2/3 btrfs-progs: check/original: detect and repair orphan subvolume without orphan item
In reply to Qu WenruoFor the detection part, just change how we handle root_item::refs == 0 case, so that even we find a root_item whose refs == 0, we still set root_rec::found_root_item, then existing code can handle it well.
For repair part, use the repair_root_orphan_item() function.
Also since we're here, use the common orphan item search function to replace the original mode specific function.
--- check/main.c | 37 +++++++++++-------------------------- 1 file changed, 11 insertions(+), 26 deletions(-)
check/main.c +11 -26
@@ -862,24 +862,6 @@ static void maybe_free_inode_rec(struct cache_tree *inode_cache, } } -static int check_orphan_item (struct btrfs_root *root,u64 ino) -{ -struct btrfs_path path; -struct btrfs_key key; -int ret; - - key.objectid =BTRFS_ORPHAN_OBJECTID ; - key.type =BTRFS_ORPHAN_ITEM_KEY ; - key.offset = ino; - -btrfs_init_path (&path); - ret =btrfs_search_slot (NULL , root, &key, &path,0 ,0 ); -btrfs_release_path (&path); -if (ret >0 ) - ret = -ENOENT ; -return ret; -} -static int process_inode_item (struct extent_buffer *eb,int slot,struct btrfs_key *key,struct shared_node *active_node)
@@ -3063,8 +3045,7 @@ static int check_inode_recs(struct btrfs_root *root, }if (rec->errors &I_ERR_NO_ORPHAN_ITEM ) { - ret =check_orphan_item (root, rec->ino); -if (ret ==0 ) +if (btrfs_has_orphan_item (root, rec->ino)) rec->errors &= ~I_ERR_NO_ORPHAN_ITEM ;if (can_free_inode_rec (rec)) {free_inode_rec (rec);
@@ -3339,8 +3320,8 @@ static int check_root_refs(struct btrfs_root *root,if (rec->found_ref ==0 && rec->objectid >=BTRFS_FIRST_FREE_OBJECTID && rec->objectid <=BTRFS_LAST_FREE_OBJECTID ) { - ret =check_orphan_item (gfs_info->tree_root, rec->objectid); -if (ret ==0 ) +if (btrfs_has_orphan_item (root->fs_info->tree_root, + rec->objectid))continue ;/*
@@ -3350,9 +3331,14 @@ static int check_root_refs(struct btrfs_root *root,*/ if (!rec->found_root_item)continue ; - errors++;fprintf (stderr,"fs tree %llu not referenced\n" , - (unsigned long long )rec->objectid); + rec->objectid); + +if (repair) + ret =repair_root_orphan_item (root->fs_info, + rec->objectid); +if (!repair || ret) + errors++; } error =0 ;
@@ -3579,8 +3565,7 @@ static int check_fs_root(struct btrfs_root *root,if (root->root_key.objectid !=BTRFS_TREE_RELOC_OBJECTID ) { rec =get_root_rec (root_cache, root->root_key.objectid);BUG_ON (IS_ERR (rec)); -if (btrfs_root_refs (root_item) >0 ) - rec->found_root_item =1 ; + rec->found_root_item =1 ; }btrfs_init_path (&path);
Signature · 1 line
2.32.0
PATCH 3/3 btrfs-progs: fsck-tests: add test image for orphan roots without an orphan item
In reply to Qu WenruoThe image is manually crafted so that:
- Root 257/258 has no ROOT_BACKREF/ROOT_REF
- Root 257/258 has root refs 0
- No DIR_INDEX/DIR_ITEM points to root 257/258
--- .../048-orphan-roots/.lowmem_repairable | 0 tests/fsck-tests/048-orphan-roots/default.img.xz | Bin 0 -> 2584 bytes 2 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 tests/fsck-tests/048-orphan-roots/.lowmem_repairable create mode 100644 tests/fsck-tests/048-orphan-roots/default.img.xz
tests/fsck-tests/048-orphan-roots/.lowmem_repairable new file +0 -0
tests/fsck-tests/048-orphan-roots/default.img.xz new file binary +0 -0
Signature · 1 line
2.32.0
Hi David,
Any idea why the patchest is not merged yet?
In fact we got a newer report on the same problem, which this patchset is definitely going to help:
https://lore.kernel.org/linux-btrfs/[email protected]/
It has two subvolumes which has 0 refs, but no ORPHAN item for it.
Thanks,
Qu
Quoted 40 lines · Qu Wenruo
There is bug report from Zhenyu Wu <[email protected]>, where even there is only one subvolume, his filesystem still consumes way more space than expected.
The problem is, there are 58 snapshots, but all of them have no ROOT_REF/ROOT_BACKREF, and all their root refs is 0.
But strangely there is no orphan items for them.
This means, btrfs kernel module won't detect them as orphan nor queue them to be removed.
"btrfs subvolume delete -i" do no helps, as it still requires
ROOT_BACKREFto resolve the dentry.For now, we should teach btrfs check to detect and repair the case by adding new orphan items for them.
The root cause I guess is some strange behavior for orphan item insert and subvolume unlink.
As the report mentioned that he had one forced power off.Qu Wenruo (3):
btrfs-progs: check/lowmem: add the ability to detect and repair orphan
subvolume trees which doesn't have orphan item
btrfs-progs: check/original: detect and repair orphan subvolume
without orphan item
btrfs-progs: fsck-tests: add test image for orphan roots without an
orphan itemcheck/main.c | 37 ++++-------- check/mode-common.c | 56 ++++++++++++++++++ check/mode-common.h | 3 + check/mode-lowmem.c | 42 +++++++------ .../048-orphan-roots/.lowmem_repairable | 0 .../048-orphan-roots/default.img.xz | Bin 0 -> 2584 bytes 6 files changed, 93 insertions(+), 45 deletions(-)create mode 100644 tests/fsck-tests/048-orphan-roots/.lowmem_repairable create mode 100644 tests/fsck-tests/048-orphan-roots/default.img.xz