Error condition failure fixes
cover Error condition failure fixes
I've been running my bpf error injection stress script and been fixing what has fallen out. I don't think I've fixed everything yet, but to reduce the noise from Dave's testing here's the current set of fixes I have. These are based on misc-next from late last week, but should apply cleanly to the recent set. These aren't super important, but will cut down on the noise from things like generic/019 and generic/475. Thanks,
Josef
PATCH 1/4 btrfs: set fs_root = NULL on error
While running my error injection script I hit a panic when we tried to clean up the fs_root when free'ing the fs_root. This is because fs_info->fs_root == PTR_ERR(-EIO), which isn't great. Fix this by setting fs_info->fs_root = NULL; if we fail to read the root.
--- fs/btrfs/disk-io.c | 1 + 1 file changed, 1 insertion(+)
fs/btrfs/disk-io.c +1 -0
@@ -3260,6 +3260,7 @@ int __cold open_ctree(struct super_block *sb,if (IS_ERR (fs_info->fs_root)) { err =PTR_ERR (fs_info->fs_root);btrfs_warn (fs_info,"failed to read fs tree: %d" , err); + fs_info->fs_root =NULL ; goto fail_qgroup; }
Signature · 1 line
2.24.1
Re: [PATCH 1/4] btrfs: set fs_root = NULL on error
Quoted 6 lines · Josef Bacik
While running my error injection script I hit a panic when we tried to clean up the
fs_rootwhen free'ing thefs_root. This is becausefs_info->fs_root==PTR_ERR(-EIO), which isn't great. Fix this by settingfs_info->fs_root= NULL; if we fail to read the root.Signed-off-by Josef Bacik <[email protected]>
Just one off-topic idea, can we have test cases in fstests to do specific error injection test?
For your fix, we can inject ENOMEM error with call chain btrfs_read_fs_root_no_name()->open_ctree() to get a 100% reproducible test, which looks to be a solid test case.
Thanks,
Qu
Quoted 17 lines
--- fs/btrfs/disk-io.c | 1 + 1 file changed, 1 insertion(+)fs/btrfs/disk-io.c +1
-0@@ -3260,6 +3260,7 @@ int __cold open_ctree(struct super_block *sb,if (IS_ERR (fs_info->fs_root)) { err =PTR_ERR (fs_info->fs_root);btrfs_warn (fs_info,"failed to read fs tree: %d" , err); + fs_info->fs_root =NULL ; goto fail_qgroup; }
Re: [PATCH 1/4] btrfs: set fs_root = NULL on error
In reply to Josef BacikLooks good,
Re: [PATCH 1/4] btrfs: set fs_root = NULL on error
In reply to Josef BacikQuoted 6 lines · Josef Bacik
While running my error injection script I hit a panic when we tried to clean up the
fs_rootwhen free'ing thefs_root. This is becausefs_info->fs_root==PTR_ERR(-EIO), which isn't great. Fix this by settingfs_info->fs_root= NULL; if we fail to read the root.Signed-off-by Josef Bacik <[email protected]>
While looking to see how ->fs_root (git grep "\->fs_root\W" fs/btrfs) is used I realized we almost never query it through that member. It's cleaned up via the btrfs_free_fs_roots which queries the root radix. Given this I fail to see how the presence of a bogus value in fs_info->fs_root would cause a crash (it's certainly wrong so your patch per-se is fine). Can you provide an example call trace?
In any case :
Re: [PATCH 1/4] btrfs: set fs_root = NULL on error
Quoted 11 lines · Nikolay Borisov
Quoted 6 lines · Josef Bacik
While running my error injection script I hit a panic when we tried to clean up the
fs_rootwhen free'ing thefs_root. This is becausefs_info->fs_root==PTR_ERR(-EIO), which isn't great. Fix this by settingfs_info->fs_root= NULL; if we fail to read the root.Signed-off-by Josef Bacik <[email protected]>
Quoted 7 lines
While looking to see how ->fs_root (git grep "\->fs_root\W" fs/btrfs) is used I realized we almost never query it through that member. It's cleaned up via the
btrfs_free_fs_rootswhich queries the root radix. Given this I fail to see how the presence of a bogus value infs_info->fs_rootwould cause a crash (it's certainly wrong so your patch per-se is fine). Can you provide an example call trace?
We do a btrfs_put_root(fs_info->fs_root); in btrfs_free_fs_info. There's for sure an argument to be made for getting rid of fs_info->fs_root, and just using the radix lookup. Once all of my root ref patches are merged I'll take a run at that. Thanks,
Josef
Re: [PATCH 1/4] btrfs: set fs_root = NULL on error
Quoted 5 lines · Josef Bacik
We do a
btrfs_put_root(fs_info->fs_root); inbtrfs_free_fs_info. There's for sure an argument to be made for getting rid offs_info->fs_root, and just using the radix lookup. Once all of my root ref patches are merged I'll take a run at that. Thanks,
This makes sense, I was looking at linux master which don't have your patches, hence I wasn't seeing this. Fair point.
PATCH 2/4 btrfs: do not check delayed items are empty for single trans cleanup
In reply to Josef Bacikbtrfs_assert_delayed_root_empty() will check if the delayed root is completely empty, but this is a fs wide check. On cleanup we may have allowed other transactions to begin, for whatever reason, and thus the delayed root is not empty. So remove this check from cleanup_one_transation(). This however can stay in btrfs_cleanup_transaction(), because it checks only after all of the transactions have been properly cleaned up, and thus is valid.
--- fs/btrfs/disk-io.c | 1 - 1 file changed, 1 deletion(-)
fs/btrfs/disk-io.c +0 -1
@@ -4543,7 +4543,6 @@ void btrfs_cleanup_one_transaction(struct btrfs_transaction *cur_trans,wake_up (&fs_info->transaction_wait);btrfs_destroy_delayed_inodes (fs_info); -btrfs_assert_delayed_root_empty (fs_info);btrfs_destroy_marked_extents (fs_info, &cur_trans->dirty_pages,EXTENT_DIRTY );
Signature · 1 line
2.24.1
Re: [PATCH 2/4] btrfs: do not check delayed items are empty for single trans cleanup
Quoted 9 lines · Josef Bacik
btrfs_assert_delayed_root_empty()will check if the delayed root is completely empty, but this is a fs wide check. On cleanup we may have allowed other transactions to begin, for whatever reason, and thus the delayed root is not empty. So remove this check fromcleanup_one_transation(). This however can stay inbtrfs_cleanup_transaction(), because it checks only after all of the transactions have been properly cleaned up, and thus is valid.Signed-off-by Josef Bacik <[email protected]>
Just a nitpick, to allow other user to verify the fix, would you mind to provide a specific reproducer?
Like the error injection (I guess it's still memory allocation failure), the call chain.
Thanks,
Qu
Quoted 17 lines
--- fs/btrfs/disk-io.c | 1 - 1 file changed, 1 deletion(-)fs/btrfs/disk-io.c +0
-1@@ -4543,7 +4543,6 @@ void btrfs_cleanup_one_transaction(struct btrfs_transaction *cur_trans,wake_up (&fs_info->transaction_wait);btrfs_destroy_delayed_inodes (fs_info); -btrfs_assert_delayed_root_empty (fs_info);btrfs_destroy_marked_extents (fs_info, &cur_trans->dirty_pages,EXTENT_DIRTY );
Re: [PATCH 2/4] btrfs: do not check delayed items are empty for single trans cleanup
In reply to Josef BacikQuoted 9 lines · Josef Bacik
btrfs_assert_delayed_root_empty()will check if the delayed root is completely empty, but this is a fs wide check. On cleanup we may have allowed other transactions to begin, for whatever reason, and thus the delayed root is not empty. So remove this check fromcleanup_one_transation(). This however can stay inbtrfs_cleanup_transaction(), because it checks only after all of the transactions have been properly cleaned up, and thus is valid.Signed-off-by Josef Bacik <[email protected]>
Re: [PATCH 2/4] btrfs: do not check delayed items are empty for single trans cleanup
In reply to Josef BacikLooks good,
PATCH 3/4 btrfs: handle logged extent failure properly
In reply to Josef BacikIf we're allocating a logged extent we attempt to insert an extent record for the file extent directly. We increase space_info->bytes_reserved, because the extent entry addition will call btrfs_update_block_group(), which will convert the ->bytes_reserved to ->bytes_used. However if we fail at any point while inserting the extent entry we will bail and leave space on ->bytes_reserved, which will trigger a WARN_ON() on umount. Fix this by pinning the space if we fail to insert, which is what happens in every other failure case that involves adding the extent entry.
--- fs/btrfs/extent-tree.c | 2 ++ 1 file changed, 2 insertions(+)
fs/btrfs/extent-tree.c +2 -0
@@ -4430,6 +4430,8 @@ int btrfs_alloc_logged_file_extent(struct btrfs_trans_handle *trans, ret =alloc_reserved_file_extent (trans,0 , root_objectid,0 , owner, offset, ins,1 ); +if (ret) +btrfs_pin_extent (fs_info, ins->objectid, ins->offset,1 );btrfs_put_block_group (block_group);return ret; }
Signature · 1 line
2.24.1
Re: [PATCH 3/4] btrfs: handle logged extent failure properly
Quoted 11 lines · Josef Bacik
If we're allocating a logged extent we attempt to insert an extent record for the file extent directly. We increase
space_info->bytes_reserved, because the extent entry addition will callbtrfs_update_block_group(), which will convert the ->bytes_reserved to ->bytes_used. However if we fail at any point while inserting the extent entry we will bail and leave space on ->bytes_reserved, which will trigger aWARN_ON()on umount. Fix this by pinning the space if we fail to insert, which is what happens in every other failure case that involves adding the extent entry.Signed-off-by Josef Bacik <[email protected]>
Thanks,
Qu
Quoted 18 lines
--- fs/btrfs/extent-tree.c | 2 ++ 1 file changed, 2 insertions(+)fs/btrfs/extent-tree.c +2
-0@@ -4430,6 +4430,8 @@ int btrfs_alloc_logged_file_extent(struct btrfs_trans_handle *trans, ret =alloc_reserved_file_extent (trans,0 , root_objectid,0 , owner, offset, ins,1 ); +if (ret) +btrfs_pin_extent (fs_info, ins->objectid, ins->offset,1 );btrfs_put_block_group (block_group);return ret; }
Re: [PATCH 3/4] btrfs: handle logged extent failure properly
In reply to Josef BacikQuoted 11 lines · Josef Bacik
If we're allocating a logged extent we attempt to insert an extent record for the file extent directly. We increase
space_info->bytes_reserved, because the extent entry addition will callbtrfs_update_block_group(), which will convert the ->bytes_reserved to ->bytes_used. However if we fail at any point while inserting the extent entry we will bail and leave space on ->bytes_reserved, which will trigger aWARN_ON()on umount. Fix this by pinning the space if we fail to insert, which is what happens in every other failure case that involves adding the extent entry.Signed-off-by Josef Bacik <[email protected]>
Re: [PATCH 3/4] btrfs: handle logged extent failure properly
In reply to Josef BacikLooks good,
PATCH 4/4 btrfs: fix bytes_may_use underflow in prealloc error condtition
In reply to Josef BacikI hit the following warning while running my error injection stress testing
------------[ cut here ]------------ WARNING: CPU: 3 PID: 1453 at fs/btrfs/space-info.h:108 btrfs_free_reserved_data_space_noquota+0xfd/0x160 [btrfs] RIP: 0010:btrfs_free_reserved_data_space_noquota+0xfd/0x160 [btrfs] Call Trace: btrfs_free_reserved_data_space+0x4f/0x70 [btrfs] __btrfs_prealloc_file_range+0x378/0x470 [btrfs] elfcorehdr_read+0x40/0x40 ? elfcorehdr_read+0x40/0x40 ? btrfs_commit_transaction+0xca/0xa50 [btrfs] ? dput+0xb4/0x2a0 ? btrfs_log_dentry_safe+0x55/0x70 [btrfs] ? btrfs_sync_file+0x30e/0x420 [btrfs] ? do_fsync+0x38/0x70 ? __x64_sys_fdatasync+0x13/0x20 ? do_syscall_64+0x5b/0x1b0 ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 ---[ end trace 70ccb5d0fe51151c ]---
This happens if we fail to insert our reserved file extent. At this point we've already converted our reservation from ->bytes_may_use to ->bytes_reserved. However once we break we will attempt to free everything from [cur_offset, end] from ->bytes_may_use, but our extent reservation will overlap part of this.
Fix this problem by adding ins.offset (our extent allocation size) to cur_offset so we remove the actual remaining part from ->bytes_may_use.
I validated this fix using my inject-error.py script
python inject-error.py -o should_fail_bio -t cache_save_setup -t \ __btrfs_prealloc_file_range \ -t insert_reserved_file_extent.constprop.0 \ -r "-5" ./run-fsstress.sh
where run-fsstress.sh simply mounts and runs fsstress on a disk.
--- fs/btrfs/inode.c | 8 ++++++++ 1 file changed, 8 insertions(+)
fs/btrfs/inode.c +8 -0
@@ -9919,6 +9919,14 @@ static int __btrfs_prealloc_file_range(struct inode *inode, int mode, ins.offset,0 ,0 ,0 , BTRFS_FILE_EXTENT_PREALLOC);if (ret) { +/* +* We've reserved this space, and thus converted it from +* ->bytes_may_use to ->bytes_reserved, which we cleanup +* here. We need to adjust cur_offset so that we only +* drop the ->bytes_may_use for the area we still have +* remaining in ->>bytes_may_use. +*/ + cur_offset += ins.objectid;btrfs_free_reserved_extent (fs_info, ins.objectid, ins.offset,0 );btrfs_abort_transaction (trans, ret);
Signature · 1 line
2.24.1
Re: [PATCH 4/4] btrfs: fix bytes_may_use underflow in prealloc error condtition
Quoted 31 lines · Josef Bacik
I hit the following warning while running my error injection stress testing
------------[ cut here ]------------ WARNING: CPU: 3 PID: 1453 at fs/btrfs/space-info.h:108 btrfs_free_reserved_data_space_noquota+0xfd/0x160 [btrfs] RIP: 0010:btrfs_free_reserved_data_space_noquota+0xfd/0x160 [btrfs] Call Trace: btrfs_free_reserved_data_space+0x4f/0x70 [btrfs] __btrfs_prealloc_file_range+0x378/0x470 [btrfs] elfcorehdr_read+0x40/0x40 ? elfcorehdr_read+0x40/0x40 ? btrfs_commit_transaction+0xca/0xa50 [btrfs] ? dput+0xb4/0x2a0 ? btrfs_log_dentry_safe+0x55/0x70 [btrfs] ? btrfs_sync_file+0x30e/0x420 [btrfs] ? do_fsync+0x38/0x70 ? __x64_sys_fdatasync+0x13/0x20 ? do_syscall_64+0x5b/0x1b0 ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 ---[ end trace 70ccb5d0fe51151c ]---This happens if we fail to insert our reserved file extent. At this point we've already converted our reservation from ->bytes_may_use to ->bytes_reserved. However once we break we will attempt to free everything from [
cur_offset, end] from ->bytes_may_use, but our extent reservation will overlap part of this.Fix this problem by adding ins.offset (our extent allocation size) to
cur_offsetso we remove the actual remaining part from ->bytes_may_use.I validated this fix using my
inject-error.pyscript
Quoted 8 lines
python inject-error.py -o should_fail_bio -t cache_save_setup -t \ __btrfs_prealloc_file_range \ -t insert_reserved_file_extent.constprop.0 \ -r "-5" ./run-fsstress.shwhere
run-fsstress.shsimply mounts and runs fsstress on a disk.Signed-off-by Josef Bacik <[email protected]>
Thanks,
Qu
Quoted 24 lines
--- fs/btrfs/inode.c | 8 ++++++++ 1 file changed, 8 insertions(+)fs/btrfs/inode.c +8
-0@@ -9919,6 +9919,14 @@ static int __btrfs_prealloc_file_range(struct inode *inode, int mode, ins.offset,0 ,0 ,0 , BTRFS_FILE_EXTENT_PREALLOC);if (ret) { +/* +* We've reserved this space, and thus converted it from +* ->bytes_may_use to ->bytes_reserved, which we cleanup +* here. We need to adjust cur_offset so that we only +* drop the ->bytes_may_use for the area we still have +* remaining in ->>bytes_may_use. +*/ + cur_offset += ins.objectid;btrfs_free_reserved_extent (fs_info, ins.objectid, ins.offset,0 );btrfs_abort_transaction (trans, ret);
Re: [PATCH 4/4] btrfs: fix bytes_may_use underflow in prealloc error condtition
In reply to Josef BacikQuoted 20 lines · Josef Bacik
I hit the following warning while running my error injection stress testing
------------[ cut here ]------------ WARNING: CPU: 3 PID: 1453 at fs/btrfs/space-info.h:108 btrfs_free_reserved_data_space_noquota+0xfd/0x160 [btrfs] RIP: 0010:btrfs_free_reserved_data_space_noquota+0xfd/0x160 [btrfs] Call Trace: btrfs_free_reserved_data_space+0x4f/0x70 [btrfs] __btrfs_prealloc_file_range+0x378/0x470 [btrfs] elfcorehdr_read+0x40/0x40 ? elfcorehdr_read+0x40/0x40 ? btrfs_commit_transaction+0xca/0xa50 [btrfs] ? dput+0xb4/0x2a0 ? btrfs_log_dentry_safe+0x55/0x70 [btrfs] ? btrfs_sync_file+0x30e/0x420 [btrfs] ? do_fsync+0x38/0x70 ? __x64_sys_fdatasync+0x13/0x20 ? do_syscall_64+0x5b/0x1b0 ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 ---[ end trace 70ccb5d0fe51151c ]---
Quoted 8 lines
This happens if we fail to insert our reserved file extent. At this point we've already converted our reservation from ->bytes_may_use to ->bytes_reserved. However once we break we will attempt to free everything from [
cur_offset, end] from ->bytes_may_use, but our extent reservation will overlap part of this.Fix this problem by adding ins.offset (our extent allocation size) to
cur_offsetso we remove the actual remaining part from ->bytes_may_use.
This contradicts the code, you are adding ins.objectid which is the offset and not the size. This means either the code is buggy. <snip>
Quoted 13 lines
fs/btrfs/inode.c +8
-0if (ret) { +/* +* We've reserved this space, and thus converted it from +* ->bytes_may_use to ->bytes_reserved, which we cleanup +* here. We need to adjust cur_offset so that we only +* drop the ->bytes_may_use for the area we still have +* remaining in ->>bytes_may_use. +*/ + cur_offset += ins.objectid;btrfs_free_reserved_extent (fs_info, ins.objectid, ins.offset,0 );btrfs_abort_transaction (trans, ret);
Re: [PATCH 4/4] btrfs: fix bytes_may_use underflow in prealloc error condtition
Quoted 29 lines · Nikolay Borisov
Quoted 26 lines · Josef Bacik
I hit the following warning while running my error injection stress testing
------------[ cut here ]------------ WARNING: CPU: 3 PID: 1453 at fs/btrfs/space-info.h:108 btrfs_free_reserved_data_space_noquota+0xfd/0x160 [btrfs] RIP: 0010:btrfs_free_reserved_data_space_noquota+0xfd/0x160 [btrfs] Call Trace: btrfs_free_reserved_data_space+0x4f/0x70 [btrfs] __btrfs_prealloc_file_range+0x378/0x470 [btrfs] elfcorehdr_read+0x40/0x40 ? elfcorehdr_read+0x40/0x40 ? btrfs_commit_transaction+0xca/0xa50 [btrfs] ? dput+0xb4/0x2a0 ? btrfs_log_dentry_safe+0x55/0x70 [btrfs] ? btrfs_sync_file+0x30e/0x420 [btrfs] ? do_fsync+0x38/0x70 ? __x64_sys_fdatasync+0x13/0x20 ? do_syscall_64+0x5b/0x1b0 ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 ---[ end trace 70ccb5d0fe51151c ]---This happens if we fail to insert our reserved file extent. At this point we've already converted our reservation from ->bytes_may_use to ->bytes_reserved. However once we break we will attempt to free everything from [
cur_offset, end] from ->bytes_may_use, but our extent reservation will overlap part of this.
Quoted 4 lines
Quoted 2 lines
Fix this problem by adding ins.offset (our extent allocation size) to
cur_offsetso we remove the actual remaining part from ->bytes_may_use.This contradicts the code, you are adding ins.objectid which is the offset and not the size. This means either the code is buggy.
Ooops you're right, I was getting lucky because we're making the whole allocation at once, and ins.objectid was past extent_end so we ended up doing the right thing, but for the wrong reasons. In fact I need to adjust this for the other error condition, so I'll fix this up. Thanks,
Josef