Skip to contentSkip to conversation

Error condition failure fixes

PATCH Superseded Josef Bacik 4 participants 19 messages Feb 11, 2020 – Feb 13, 2020 Btrfs file system

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Error condition failure fixes

I've been running my bpf error injection stress script and been fixing what has fallen out. I don't think I've fixed everything yet, but to reduce the noise from Dave's testing here's the current set of fixes I have. These are based on misc-next from late last week, but should apply cleanly to the recent set. These aren't super important, but will cut down on the noise from things like generic/019 and generic/475. Thanks,

Josef

#1 lore raw

PATCH 1/4 btrfs: set fs_root = NULL on error

While running my error injection script I hit a panic when we tried to clean up the fs_root when free'ing the fs_root. This is because fs_info->fs_root == PTR_ERR(-EIO), which isn't great. Fix this by setting fs_info->fs_root = NULL; if we fail to read the root.

---
 fs/btrfs/disk-io.c | 1 +
 1 file changed, 1 insertion(+)
fs/btrfs/disk-io.c +1 -0
index eb441fa3711b..5b6140482cef 100644
@@ -3260,6 +3260,7 @@ int __cold open_ctree(struct super_block *sb,
 	if (IS_ERR(fs_info->fs_root)) {
 		err = PTR_ERR(fs_info->fs_root);
 		btrfs_warn(fs_info, "failed to read fs tree: %d", err);
+		fs_info->fs_root = NULL;
 		goto fail_qgroup;
 	}
 
Signature · 1 line
2.24.1
Reviewed-by on v1 1/4
#2 lore raw

Re: [PATCH 1/4] btrfs: set fs_root = NULL on error

Quoted 6 lines · Josef Bacik

While running my error injection script I hit a panic when we tried to clean up the fs_root when free'ing the fs_root. This is because fs_info->fs_root == PTR_ERR(-EIO), which isn't great. Fix this by setting fs_info->fs_root = NULL; if we fail to read the root.

Just one off-topic idea, can we have test cases in fstests to do specific error injection test?

For your fix, we can inject ENOMEM error with call chain btrfs_read_fs_root_no_name()->open_ctree() to get a 100% reproducible test, which looks to be a solid test case.

Thanks,
Qu

Quoted 17 lines
---
 fs/btrfs/disk-io.c | 1 +
 1 file changed, 1 insertion(+)
fs/btrfs/disk-io.c +1 -0
index eb441fa3711b..5b6140482cef 100644
@@ -3260,6 +3260,7 @@ int __cold open_ctree(struct super_block *sb,
 	if (IS_ERR(fs_info->fs_root)) {
 		err = PTR_ERR(fs_info->fs_root);
 		btrfs_warn(fs_info, "failed to read fs tree: %d", err);
+		fs_info->fs_root = NULL;
 		goto fail_qgroup;
 	}
 
Reviewed-by on v1 1/4
#4 lore raw

Re: [PATCH 1/4] btrfs: set fs_root = NULL on error

In reply to Josef Bacik
Quoted 6 lines · Josef Bacik

While running my error injection script I hit a panic when we tried to clean up the fs_root when free'ing the fs_root. This is because fs_info->fs_root == PTR_ERR(-EIO), which isn't great. Fix this by setting fs_info->fs_root = NULL; if we fail to read the root.

While looking to see how ->fs_root (git grep "\->fs_root\W" fs/btrfs) is used I realized we almost never query it through that member. It's cleaned up via the btrfs_free_fs_roots which queries the root radix. Given this I fail to see how the presence of a bogus value in fs_info->fs_root would cause a crash (it's certainly wrong so your patch per-se is fine). Can you provide an example call trace?

In any case :

#5 lore raw

Re: [PATCH 1/4] btrfs: set fs_root = NULL on error

Quoted 11 lines · Nikolay Borisov
Quoted 6 lines · Josef Bacik

While running my error injection script I hit a panic when we tried to clean up the fs_root when free'ing the fs_root. This is because fs_info->fs_root == PTR_ERR(-EIO), which isn't great. Fix this by setting fs_info->fs_root = NULL; if we fail to read the root.

Quoted 7 lines

While looking to see how ->fs_root (git grep "\->fs_root\W" fs/btrfs) is used I realized we almost never query it through that member. It's cleaned up via the btrfs_free_fs_roots which queries the root radix. Given this I fail to see how the presence of a bogus value in fs_info->fs_root would cause a crash (it's certainly wrong so your patch per-se is fine). Can you provide an example call trace?

We do a btrfs_put_root(fs_info->fs_root); in btrfs_free_fs_info. There's for sure an argument to be made for getting rid of fs_info->fs_root, and just using the radix lookup. Once all of my root ref patches are merged I'll take a run at that. Thanks,

Josef

#6 lore raw

Re: [PATCH 1/4] btrfs: set fs_root = NULL on error

Quoted 5 lines · Josef Bacik

We do a btrfs_put_root(fs_info->fs_root); in btrfs_free_fs_info. There's for sure an argument to be made for getting rid of fs_info->fs_root, and just using the radix lookup.  Once all of my root ref patches are merged I'll take a run at that.  Thanks,

This makes sense, I was looking at linux master which don't have your patches, hence I wasn't seeing this. Fair point.

#7 lore raw

PATCH 2/4 btrfs: do not check delayed items are empty for single trans cleanup

In reply to Josef Bacik

btrfs_assert_delayed_root_empty() will check if the delayed root is completely empty, but this is a fs wide check. On cleanup we may have allowed other transactions to begin, for whatever reason, and thus the delayed root is not empty. So remove this check from cleanup_one_transation(). This however can stay in btrfs_cleanup_transaction(), because it checks only after all of the transactions have been properly cleaned up, and thus is valid.

---
 fs/btrfs/disk-io.c | 1 -
 1 file changed, 1 deletion(-)
fs/btrfs/disk-io.c +0 -1
index 5b6140482cef..601ed3335cf6 100644
@@ -4543,7 +4543,6 @@ void btrfs_cleanup_one_transaction(struct btrfs_transaction *cur_trans,
 	wake_up(&fs_info->transaction_wait);
 
 	btrfs_destroy_delayed_inodes(fs_info);
-	btrfs_assert_delayed_root_empty(fs_info);
 
 	btrfs_destroy_marked_extents(fs_info, &cur_trans->dirty_pages,
 				     EXTENT_DIRTY);
Signature · 1 line
2.24.1
Reviewed-by on v1 2/4
#8 lore raw

Re: [PATCH 2/4] btrfs: do not check delayed items are empty for single trans cleanup

Quoted 9 lines · Josef Bacik

btrfs_assert_delayed_root_empty() will check if the delayed root is completely empty, but this is a fs wide check. On cleanup we may have allowed other transactions to begin, for whatever reason, and thus the delayed root is not empty. So remove this check from cleanup_one_transation(). This however can stay in btrfs_cleanup_transaction(), because it checks only after all of the transactions have been properly cleaned up, and thus is valid.

Just a nitpick, to allow other user to verify the fix, would you mind to provide a specific reproducer?
Like the error injection (I guess it's still memory allocation failure), the call chain.

Thanks,
Qu

Quoted 17 lines
---
 fs/btrfs/disk-io.c | 1 -
 1 file changed, 1 deletion(-)
fs/btrfs/disk-io.c +0 -1
index 5b6140482cef..601ed3335cf6 100644
@@ -4543,7 +4543,6 @@ void btrfs_cleanup_one_transaction(struct btrfs_transaction *cur_trans,
 	wake_up(&fs_info->transaction_wait);
 
 	btrfs_destroy_delayed_inodes(fs_info);
-	btrfs_assert_delayed_root_empty(fs_info);
 
 	btrfs_destroy_marked_extents(fs_info, &cur_trans->dirty_pages,
 				     EXTENT_DIRTY);
Reviewed-by on v1 2/4
#9 lore raw

Re: [PATCH 2/4] btrfs: do not check delayed items are empty for single trans cleanup

In reply to Josef Bacik
Quoted 9 lines · Josef Bacik

btrfs_assert_delayed_root_empty() will check if the delayed root is completely empty, but this is a fs wide check. On cleanup we may have allowed other transactions to begin, for whatever reason, and thus the delayed root is not empty. So remove this check from cleanup_one_transation(). This however can stay in btrfs_cleanup_transaction(), because it checks only after all of the transactions have been properly cleaned up, and thus is valid.

#11 lore raw

PATCH 3/4 btrfs: handle logged extent failure properly

In reply to Josef Bacik

If we're allocating a logged extent we attempt to insert an extent record for the file extent directly. We increase space_info->bytes_reserved, because the extent entry addition will call btrfs_update_block_group(), which will convert the ->bytes_reserved to ->bytes_used. However if we fail at any point while inserting the extent entry we will bail and leave space on ->bytes_reserved, which will trigger a WARN_ON() on umount. Fix this by pinning the space if we fail to insert, which is what happens in every other failure case that involves adding the extent entry.

---
 fs/btrfs/extent-tree.c | 2 ++
 1 file changed, 2 insertions(+)
fs/btrfs/extent-tree.c +2 -0
index c43acb329fa6..2b4c3ca5e651 100644
@@ -4430,6 +4430,8 @@ int btrfs_alloc_logged_file_extent(struct btrfs_trans_handle *trans,
 
 	ret = alloc_reserved_file_extent(trans, 0, root_objectid, 0, owner,
 					 offset, ins, 1);
+	if (ret)
+		btrfs_pin_extent(fs_info, ins->objectid, ins->offset, 1);
 	btrfs_put_block_group(block_group);
 	return ret;
 }
Signature · 1 line
2.24.1
Reviewed-by on v1 3/4
#12 lore raw

Re: [PATCH 3/4] btrfs: handle logged extent failure properly

Quoted 11 lines · Josef Bacik

If we're allocating a logged extent we attempt to insert an extent record for the file extent directly. We increase space_info->bytes_reserved, because the extent entry addition will call btrfs_update_block_group(), which will convert the ->bytes_reserved to ->bytes_used. However if we fail at any point while inserting the extent entry we will bail and leave space on ->bytes_reserved, which will trigger a WARN_ON() on umount. Fix this by pinning the space if we fail to insert, which is what happens in every other failure case that involves adding the extent entry.

Thanks,
Qu

Quoted 18 lines
---
 fs/btrfs/extent-tree.c | 2 ++
 1 file changed, 2 insertions(+)
fs/btrfs/extent-tree.c +2 -0
index c43acb329fa6..2b4c3ca5e651 100644
@@ -4430,6 +4430,8 @@ int btrfs_alloc_logged_file_extent(struct btrfs_trans_handle *trans,
 
 	ret = alloc_reserved_file_extent(trans, 0, root_objectid, 0, owner,
 					 offset, ins, 1);
+	if (ret)
+		btrfs_pin_extent(fs_info, ins->objectid, ins->offset, 1);
 	btrfs_put_block_group(block_group);
 	return ret;
 }
Reviewed-by on v1 3/4
#13 lore raw

Re: [PATCH 3/4] btrfs: handle logged extent failure properly

In reply to Josef Bacik
Quoted 11 lines · Josef Bacik

If we're allocating a logged extent we attempt to insert an extent record for the file extent directly. We increase space_info->bytes_reserved, because the extent entry addition will call btrfs_update_block_group(), which will convert the ->bytes_reserved to ->bytes_used. However if we fail at any point while inserting the extent entry we will bail and leave space on ->bytes_reserved, which will trigger a WARN_ON() on umount. Fix this by pinning the space if we fail to insert, which is what happens in every other failure case that involves adding the extent entry.

#15 lore raw

PATCH 4/4 btrfs: fix bytes_may_use underflow in prealloc error condtition

In reply to Josef Bacik

I hit the following warning while running my error injection stress testing

------------[ cut here ]------------
WARNING: CPU: 3 PID: 1453 at fs/btrfs/space-info.h:108 btrfs_free_reserved_data_space_noquota+0xfd/0x160 [btrfs]
RIP: 0010:btrfs_free_reserved_data_space_noquota+0xfd/0x160 [btrfs]
Call Trace:
btrfs_free_reserved_data_space+0x4f/0x70 [btrfs]
__btrfs_prealloc_file_range+0x378/0x470 [btrfs]
elfcorehdr_read+0x40/0x40
? elfcorehdr_read+0x40/0x40
? btrfs_commit_transaction+0xca/0xa50 [btrfs]
? dput+0xb4/0x2a0
? btrfs_log_dentry_safe+0x55/0x70 [btrfs]
? btrfs_sync_file+0x30e/0x420 [btrfs]
? do_fsync+0x38/0x70
? __x64_sys_fdatasync+0x13/0x20
? do_syscall_64+0x5b/0x1b0
? entry_SYSCALL_64_after_hwframe+0x44/0xa9
---[ end trace 70ccb5d0fe51151c ]---

This happens if we fail to insert our reserved file extent. At this point we've already converted our reservation from ->bytes_may_use to ->bytes_reserved. However once we break we will attempt to free everything from [cur_offset, end] from ->bytes_may_use, but our extent reservation will overlap part of this.

Fix this problem by adding ins.offset (our extent allocation size) to cur_offset so we remove the actual remaining part from ->bytes_may_use.

I validated this fix using my inject-error.py script

python inject-error.py -o should_fail_bio -t cache_save_setup -t \
	__btrfs_prealloc_file_range \
	-t insert_reserved_file_extent.constprop.0 \
	-r "-5" ./run-fsstress.sh

where run-fsstress.sh simply mounts and runs fsstress on a disk.

---
 fs/btrfs/inode.c | 8 ++++++++
 1 file changed, 8 insertions(+)
fs/btrfs/inode.c +8 -0
index 84e649724549..747d860aedf6 100644
@@ -9919,6 +9919,14 @@ static int __btrfs_prealloc_file_range(struct inode *inode, int mode,
 						  ins.offset, 0, 0, 0,
 						  BTRFS_FILE_EXTENT_PREALLOC);
 		if (ret) {
+			/*
+			 * We've reserved this space, and thus converted it from
+			 * ->bytes_may_use to ->bytes_reserved, which we cleanup
+			 * here.  We need to adjust cur_offset so that we only
+			 * drop the ->bytes_may_use for the area we still have
+			 * remaining in ->>bytes_may_use.
+			 */
+			cur_offset += ins.objectid;
 			btrfs_free_reserved_extent(fs_info, ins.objectid,
 						   ins.offset, 0);
 			btrfs_abort_transaction(trans, ret);
Signature · 1 line
2.24.1
Reviewed-by on v1 4/4
#16 lore raw

Re: [PATCH 4/4] btrfs: fix bytes_may_use underflow in prealloc error condtition

Quoted 31 lines · Josef Bacik

I hit the following warning while running my error injection stress testing

------------[ cut here ]------------
WARNING: CPU: 3 PID: 1453 at fs/btrfs/space-info.h:108 btrfs_free_reserved_data_space_noquota+0xfd/0x160 [btrfs]
RIP: 0010:btrfs_free_reserved_data_space_noquota+0xfd/0x160 [btrfs]
Call Trace:
btrfs_free_reserved_data_space+0x4f/0x70 [btrfs]
__btrfs_prealloc_file_range+0x378/0x470 [btrfs]
elfcorehdr_read+0x40/0x40
? elfcorehdr_read+0x40/0x40
? btrfs_commit_transaction+0xca/0xa50 [btrfs]
? dput+0xb4/0x2a0
? btrfs_log_dentry_safe+0x55/0x70 [btrfs]
? btrfs_sync_file+0x30e/0x420 [btrfs]
? do_fsync+0x38/0x70
? __x64_sys_fdatasync+0x13/0x20
? do_syscall_64+0x5b/0x1b0
? entry_SYSCALL_64_after_hwframe+0x44/0xa9
---[ end trace 70ccb5d0fe51151c ]---

This happens if we fail to insert our reserved file extent. At this point we've already converted our reservation from ->bytes_may_use to ->bytes_reserved. However once we break we will attempt to free everything from [cur_offset, end] from ->bytes_may_use, but our extent reservation will overlap part of this.

Fix this problem by adding ins.offset (our extent allocation size) to cur_offset so we remove the actual remaining part from ->bytes_may_use.

I validated this fix using my inject-error.py script

Quoted 8 lines
python inject-error.py -o should_fail_bio -t cache_save_setup -t \
	__btrfs_prealloc_file_range \
	-t insert_reserved_file_extent.constprop.0 \
	-r "-5" ./run-fsstress.sh

where run-fsstress.sh simply mounts and runs fsstress on a disk.

Thanks,
Qu

Quoted 24 lines
---
 fs/btrfs/inode.c | 8 ++++++++
 1 file changed, 8 insertions(+)
fs/btrfs/inode.c +8 -0
index 84e649724549..747d860aedf6 100644
@@ -9919,6 +9919,14 @@ static int __btrfs_prealloc_file_range(struct inode *inode, int mode,
 						  ins.offset, 0, 0, 0,
 						  BTRFS_FILE_EXTENT_PREALLOC);
 		if (ret) {
+			/*
+			 * We've reserved this space, and thus converted it from
+			 * ->bytes_may_use to ->bytes_reserved, which we cleanup
+			 * here.  We need to adjust cur_offset so that we only
+			 * drop the ->bytes_may_use for the area we still have
+			 * remaining in ->>bytes_may_use.
+			 */
+			cur_offset += ins.objectid;
 			btrfs_free_reserved_extent(fs_info, ins.objectid,
 						   ins.offset, 0);
 			btrfs_abort_transaction(trans, ret);
#17 lore raw

Re: [PATCH 4/4] btrfs: fix bytes_may_use underflow in prealloc error condtition

In reply to Josef Bacik
Quoted 20 lines · Josef Bacik

I hit the following warning while running my error injection stress testing

------------[ cut here ]------------
WARNING: CPU: 3 PID: 1453 at fs/btrfs/space-info.h:108 btrfs_free_reserved_data_space_noquota+0xfd/0x160 [btrfs]
RIP: 0010:btrfs_free_reserved_data_space_noquota+0xfd/0x160 [btrfs]
Call Trace:
btrfs_free_reserved_data_space+0x4f/0x70 [btrfs]
__btrfs_prealloc_file_range+0x378/0x470 [btrfs]
elfcorehdr_read+0x40/0x40
? elfcorehdr_read+0x40/0x40
? btrfs_commit_transaction+0xca/0xa50 [btrfs]
? dput+0xb4/0x2a0
? btrfs_log_dentry_safe+0x55/0x70 [btrfs]
? btrfs_sync_file+0x30e/0x420 [btrfs]
? do_fsync+0x38/0x70
? __x64_sys_fdatasync+0x13/0x20
? do_syscall_64+0x5b/0x1b0
? entry_SYSCALL_64_after_hwframe+0x44/0xa9
---[ end trace 70ccb5d0fe51151c ]---
Quoted 8 lines

This happens if we fail to insert our reserved file extent. At this point we've already converted our reservation from ->bytes_may_use to ->bytes_reserved. However once we break we will attempt to free everything from [cur_offset, end] from ->bytes_may_use, but our extent reservation will overlap part of this.

Fix this problem by adding ins.offset (our extent allocation size) to cur_offset so we remove the actual remaining part from ->bytes_may_use.

This contradicts the code, you are adding ins.objectid which is the offset and not the size. This means either the code is buggy. <snip>

Quoted 13 lines
fs/btrfs/inode.c +8 -0
 		if (ret) {
+			/*
+			 * We've reserved this space, and thus converted it from
+			 * ->bytes_may_use to ->bytes_reserved, which we cleanup
+			 * here.  We need to adjust cur_offset so that we only
+			 * drop the ->bytes_may_use for the area we still have
+			 * remaining in ->>bytes_may_use.
+			 */
+			cur_offset += ins.objectid;
 			btrfs_free_reserved_extent(fs_info, ins.objectid,
 						   ins.offset, 0);
 			btrfs_abort_transaction(trans, ret);
#18 lore raw

Re: [PATCH 4/4] btrfs: fix bytes_may_use underflow in prealloc error condtition

Quoted 29 lines · Nikolay Borisov
Quoted 26 lines · Josef Bacik

I hit the following warning while running my error injection stress testing

------------[ cut here ]------------
WARNING: CPU: 3 PID: 1453 at fs/btrfs/space-info.h:108 btrfs_free_reserved_data_space_noquota+0xfd/0x160 [btrfs]
RIP: 0010:btrfs_free_reserved_data_space_noquota+0xfd/0x160 [btrfs]
Call Trace:
btrfs_free_reserved_data_space+0x4f/0x70 [btrfs]
__btrfs_prealloc_file_range+0x378/0x470 [btrfs]
elfcorehdr_read+0x40/0x40
? elfcorehdr_read+0x40/0x40
? btrfs_commit_transaction+0xca/0xa50 [btrfs]
? dput+0xb4/0x2a0
? btrfs_log_dentry_safe+0x55/0x70 [btrfs]
? btrfs_sync_file+0x30e/0x420 [btrfs]
? do_fsync+0x38/0x70
? __x64_sys_fdatasync+0x13/0x20
? do_syscall_64+0x5b/0x1b0
? entry_SYSCALL_64_after_hwframe+0x44/0xa9
---[ end trace 70ccb5d0fe51151c ]---

This happens if we fail to insert our reserved file extent. At this point we've already converted our reservation from ->bytes_may_use to ->bytes_reserved. However once we break we will attempt to free everything from [cur_offset, end] from ->bytes_may_use, but our extent reservation will overlap part of this.

Quoted 4 lines
Quoted 2 lines

Fix this problem by adding ins.offset (our extent allocation size) to cur_offset so we remove the actual remaining part from ->bytes_may_use.

This contradicts the code, you are adding ins.objectid which is the offset and not the size. This means either the code is buggy.

Ooops you're right, I was getting lucky because we're making the whole allocation at once, and ins.objectid was past extent_end so we ended up doing the right thing, but for the wrong reasons. In fact I need to adjust this for the other error condition, so I'll fix this up. Thanks,

Josef