Skip to contentSkip to conversation

btrfs: extent_io: Handle memory allocation failure in __clear_extent_bit()

PATCH In review Qu Wenruo 6 participants 11 messages Apr 18, 2019 – Apr 22, 2019 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

PATCH 1/1 btrfs: extent_io: Handle memory allocation failure in __clear_extent_bit()

There is a BUG_ON() in __clear_extent_bit() for memory allocation failure.

While comment of __clear_extent_bit() says it can return error, but we always return 0.

Some __clear_extent_bit() callers just ignore the return value, while some still expect error.

Let's return proper error for this memory allocation anyway, to remove that BUG_ON() as a first step, so at least we can continue test.

---
 fs/btrfs/extent_io.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
fs/btrfs/extent_io.c +10 -2
index 828708f6510c..f1bfc632ef7b 100644
@@ -743,7 +743,10 @@ int __clear_extent_bit(struct extent_io_tree *tree, u64 start, u64 end,
 
 	if (state->start < start) {
 		prealloc = alloc_extent_state_atomic(prealloc);
-		BUG_ON(!prealloc);
+		if (!prealloc) {
+			err = -ENOMEM;
+			goto out;
+		}
 		err = split_state(tree, state, prealloc, start);
 		if (err)
 			extent_io_tree_panic(tree, err);
@@ -766,7 +769,10 @@ int __clear_extent_bit(struct extent_io_tree *tree, u64 start, u64 end,
 	 */
 	if (state->start <= end && state->end > end) {
 		prealloc = alloc_extent_state_atomic(prealloc);
-		BUG_ON(!prealloc);
+		if (!prealloc) {
+			err = -ENOMEM;
+			goto out;
+		}
 		err = split_state(tree, state, prealloc, end + 1);
 		if (err)
 			extent_io_tree_panic(tree, err);
@@ -801,6 +807,8 @@ int __clear_extent_bit(struct extent_io_tree *tree, u64 start, u64 end,
 	if (prealloc)
 		free_extent_state(prealloc);
 
+	if (err)
+		return err;
 	return 0;
 
 }
Signature · 1 line
2.21.0
#1 lore raw
Quoted 11 lines · Qu Wenruo

There is a BUG_ON() in __clear_extent_bit() for memory allocation failure.

While comment of __clear_extent_bit() says it can return error, but we always return 0.

Some __clear_extent_bit() callers just ignore the return value, while some still expect error.

Let's return proper error for this memory allocation anyway, to remove that BUG_ON() as a first step, so at least we can continue test.

I remember Josef did some changes into this code and said that prealloc shouldn't fail because this will cause mayhem down the road i.e. proper error handling is missing. If anything I think it should be added first and then remove the BUG_ONs.

Quoted 44 lines
---
 fs/btrfs/extent_io.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
fs/btrfs/extent_io.c +10 -2
index 828708f6510c..f1bfc632ef7b 100644
@@ -743,7 +743,10 @@ int __clear_extent_bit(struct extent_io_tree *tree, u64 start, u64 end,
 
 	if (state->start < start) {
 		prealloc = alloc_extent_state_atomic(prealloc);
-		BUG_ON(!prealloc);
+		if (!prealloc) {
+			err = -ENOMEM;
+			goto out;
+		}
 		err = split_state(tree, state, prealloc, start);
 		if (err)
 			extent_io_tree_panic(tree, err);
@@ -766,7 +769,10 @@ int __clear_extent_bit(struct extent_io_tree *tree, u64 start, u64 end,
 	 */
 	if (state->start <= end && state->end > end) {
 		prealloc = alloc_extent_state_atomic(prealloc);
-		BUG_ON(!prealloc);
+		if (!prealloc) {
+			err = -ENOMEM;
+			goto out;
+		}
 		err = split_state(tree, state, prealloc, end + 1);
 		if (err)
 			extent_io_tree_panic(tree, err);
@@ -801,6 +807,8 @@ int __clear_extent_bit(struct extent_io_tree *tree, u64 start, u64 end,
 	if (prealloc)
 		free_extent_state(prealloc);
 
+	if (err)
+		return err;
 	return 0;
 
 }
#2 lore raw
Quoted 12 lines · Nikolay Borisov
Quoted 9 lines · Qu Wenruo

There is a BUG_ON() in __clear_extent_bit() for memory allocation failure.

While comment of __clear_extent_bit() says it can return error, but we always return 0.

Some __clear_extent_bit() callers just ignore the return value, while some still expect error.

Quoted 7 lines
Quoted 2 lines

Let's return proper error for this memory allocation anyway, to remove that BUG_ON() as a first step, so at least we can continue test.

I remember Josef did some changes into this code and said that prealloc shouldn't fail because this will cause mayhem down the road i.e. proper error handling is missing. If anything I think it should be added first and then remove the BUG_ONs.

That's true, we could have some strange lockup due to lock_extent_bits(), as if some clear_extent_bits() failed due to ENOMEM and caller just ignore the error, we could have a lockup.

I'll try to pre-allocate certain amount of extent_state as the last chance of redemption.

Anyway, such BUG_ON() right after kmalloc() is really a blockage for error injection test.

Thanks,
Qu

Quoted 45 lines
Quoted 45 lines
---
 fs/btrfs/extent_io.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
fs/btrfs/extent_io.c +10 -2
index 828708f6510c..f1bfc632ef7b 100644
@@ -743,7 +743,10 @@ int __clear_extent_bit(struct extent_io_tree *tree, u64 start, u64 end,
 
 	if (state->start < start) {
 		prealloc = alloc_extent_state_atomic(prealloc);
-		BUG_ON(!prealloc);
+		if (!prealloc) {
+			err = -ENOMEM;
+			goto out;
+		}
 		err = split_state(tree, state, prealloc, start);
 		if (err)
 			extent_io_tree_panic(tree, err);
@@ -766,7 +769,10 @@ int __clear_extent_bit(struct extent_io_tree *tree, u64 start, u64 end,
 	 */
 	if (state->start <= end && state->end > end) {
 		prealloc = alloc_extent_state_atomic(prealloc);
-		BUG_ON(!prealloc);
+		if (!prealloc) {
+			err = -ENOMEM;
+			goto out;
+		}
 		err = split_state(tree, state, prealloc, end + 1);
 		if (err)
 			extent_io_tree_panic(tree, err);
@@ -801,6 +807,8 @@ int __clear_extent_bit(struct extent_io_tree *tree, u64 start, u64 end,
 	if (prealloc)
 		free_extent_state(prealloc);
 
+	if (err)
+		return err;
 	return 0;
 
 }
#3 lore raw
Quoted 18 lines · Qu Wenruo
Quoted 15 lines · Nikolay Borisov
Quoted 11 lines · Qu Wenruo

There is a BUG_ON() in __clear_extent_bit() for memory allocation failure.

While comment of __clear_extent_bit() says it can return error, but we always return 0.

Some __clear_extent_bit() callers just ignore the return value, while some still expect error.

Let's return proper error for this memory allocation anyway, to remove that BUG_ON() as a first step, so at least we can continue test.

Quoted 8 lines
Quoted 4 lines

I remember Josef did some changes into this code and said that prealloc shouldn't fail because this will cause mayhem down the road i.e. proper error handling is missing. If anything I think it should be added first and then remove the BUG_ONs.

That's true, we could have some strange lockup due to lock_extent_bits(), as if some clear_extent_bits() failed due to ENOMEM and caller just ignore the error, we could have a lockup.

Not only lockup but unhandled failed extent range locking totally breaks assumptions that the following code makes and this would lead to unpredictable corruptions. Just count how many lock_extent_bits calls are there. And any caller of __set_extent_bit. There are so many that the BUG_ON is the measure of last resort to prevent worse problems.

Quoted 2 lines

I'll try to pre-allocate certain amount of extent_state as the last chance of redemption.

This only lowers the chances to hit the allocation error but there's always a case when certain amount + 1 would be needed.

Quoted 2 lines

Anyway, such BUG_ON() right after kmalloc() is really a blockage for error injection test.

Maybe, but the code is not yet in the state to inject memory allocation faiulres to that particular path (ie. the state changes).

#4 lore raw
Quoted 34 lines · David Sterba
Quoted 18 lines · Qu Wenruo
Quoted 15 lines · Nikolay Borisov
Quoted 11 lines · Qu Wenruo

There is a BUG_ON() in __clear_extent_bit() for memory allocation failure.

While comment of __clear_extent_bit() says it can return error, but we always return 0.

Some __clear_extent_bit() callers just ignore the return value, while some still expect error.

Let's return proper error for this memory allocation anyway, to remove that BUG_ON() as a first step, so at least we can continue test.

Quoted 8 lines
Quoted 4 lines

I remember Josef did some changes into this code and said that prealloc shouldn't fail because this will cause mayhem down the road i.e. proper error handling is missing. If anything I think it should be added first and then remove the BUG_ONs.

That's true, we could have some strange lockup due to lock_extent_bits(), as if some clear_extent_bits() failed due to ENOMEM and caller just ignore the error, we could have a lockup.

Not only lockup but unhandled failed extent range locking totally breaks assumptions that the following code makes and this would lead to unpredictable corruptions. Just count how many lock_extent_bits calls are there. And any caller of __set_extent_bit. There are so many that the BUG_ON is the measure of last resort to prevent worse problems.

Quoted 5 lines
Quoted 2 lines

I'll try to pre-allocate certain amount of extent_state as the last chance of redemption.

This only lowers the chances to hit the allocation error but there's always a case when certain amount + 1 would be needed.

Lower chance is already good enough (TM) for low possibility (0.001) error injection.

And, for real world low memory case, lower chance in btrfs means higher chance in other subsystem, less chance user will blame btrfs. :)

Quoted 6 lines
Quoted 3 lines

Anyway, such BUG_ON() right after kmalloc() is really a blockage for error injection test.

Maybe, but the code is not yet in the state to inject memory allocation faiulres to that particular path (ie. the state changes).

With last-chance reservation, we can make state related memory allocation almost always to success even memory allocation failure injected (if the possibility is low and low concurrency)
And the last-chance reservation can be configured at compile/module load time, making it flex enough for most cases.

The main reason I'm doing such error injection test is to ensure write time tree checker is not the cause of the lockup.

Of course I can directly inject error into btrfs_check_leaf_full() and btrfs_check_node(), and filter the stack to ensure it only happen in write time, and that's already what I'm crafting, based on the bcc error inject example and kprobe return value overriding.

But it will never be a bad idea to explore what can go wrong. And "always BUG_ON()" -> "good enough (TM)" already looks like a improvement to me.

Thanks,
Qu

#5 lore raw
Quoted 53 lines · Qu Wenruo
Quoted 34 lines · David Sterba
Quoted 18 lines · Qu Wenruo
Quoted 15 lines · Nikolay Borisov
Quoted 11 lines · Qu Wenruo

There is a BUG_ON() in __clear_extent_bit() for memory allocation failure.

While comment of __clear_extent_bit() says it can return error, but we always return 0.

Some __clear_extent_bit() callers just ignore the return value, while some still expect error.

Let's return proper error for this memory allocation anyway, to remove that BUG_ON() as a first step, so at least we can continue test.

Quoted 8 lines
Quoted 4 lines

I remember Josef did some changes into this code and said that prealloc shouldn't fail because this will cause mayhem down the road i.e. proper error handling is missing. If anything I think it should be added first and then remove the BUG_ONs.

That's true, we could have some strange lockup due to lock_extent_bits(), as if some clear_extent_bits() failed due to ENOMEM and caller just ignore the error, we could have a lockup.

Not only lockup but unhandled failed extent range locking totally breaks assumptions that the following code makes and this would lead to unpredictable corruptions. Just count how many lock_extent_bits calls are there. And any caller of __set_extent_bit. There are so many that the BUG_ON is the measure of last resort to prevent worse problems.

Quoted 5 lines
Quoted 2 lines

I'll try to pre-allocate certain amount of extent_state as the last chance of redemption.

This only lowers the chances to hit the allocation error but there's always a case when certain amount + 1 would be needed.

Lower chance is already good enough (TM) for low possibility (0.001) error injection.

And, for real world low memory case, lower chance in btrfs means higher chance in other subsystem, less chance user will blame btrfs. :)

Quoted 4 lines
Quoted 3 lines

Anyway, such BUG_ON() right after kmalloc() is really a blockage for error injection test.

Quoted 8 lines
Quoted 2 lines

Maybe, but the code is not yet in the state to inject memory allocation faiulres to that particular path (ie. the state changes).

With last-chance reservation, we can make state related memory allocation almost always to success even memory allocation failure injected (if the possibility is low and low concurrency)
And the last-chance reservation can be configured at compile/module load time, making it flex enough for most cases.

Forgot to mention, for that method, I'll definitely keep the BUG_ON() on @prealloc.

Just make the allocation part fall back to use fs_info->last_chance[] to grab a valid memory slot.

Thanks,
Qu

Quoted 16 lines

The main reason I'm doing such error injection test is to ensure write time tree checker is not the cause of the lockup.

Of course I can directly inject error into btrfs_check_leaf_full() and btrfs_check_node(), and filter the stack to ensure it only happen in write time, and that's already what I'm crafting, based on the bcc error inject example and kprobe return value overriding.

But it will never be a bad idea to explore what can go wrong. And "always BUG_ON()" -> "good enough (TM)" already looks like a improvement to me.

Thanks,
Qu

#6 lore raw

On 18.04.19 г. 14:54 ч., Qu Wenruo wrote:
<snip>

Quoted 6 lines

Forgot to mention, for that method, I'll definitely keep the BUG_ON() on @prealloc.

Just make the allocation part fall back to use fs_info->last_chance[] to grab a valid memory slot.

Before you go and re-invent the wheel I will suggest you take a look at the mempool api (include/linux/mempool.h). AFAIK it's purpose is to exactly ensure memory allocations from it will not fail (depending on how the pool is configured)

Quoted 21 lines

Thanks,
Qu

Quoted 16 lines

The main reason I'm doing such error injection test is to ensure write time tree checker is not the cause of the lockup.

Of course I can directly inject error into btrfs_check_leaf_full() and btrfs_check_node(), and filter the stack to ensure it only happen in write time, and that's already what I'm crafting, based on the bcc error inject example and kprobe return value overriding.

But it will never be a bad idea to explore what can go wrong. And "always BUG_ON()" -> "good enough (TM)" already looks like a improvement to me.

Thanks,
Qu

#7 lore raw
Quoted 9 lines · Nikolay Borisov

On 18.04.19 г. 14:54 ч., Qu Wenruo wrote:
<snip>

Quoted 4 lines

Forgot to mention, for that method, I'll definitely keep the BUG_ON() on @prealloc.

Quoted 7 lines
Quoted 2 lines

Just make the allocation part fall back to use fs_info->last_chance[] to grab a valid memory slot.

Before you go and re-invent the wheel I will suggest you take a look at the mempool api (include/linux/mempool.h). AFAIK it's purpose is to exactly ensure memory allocations from it will not fail (depending on how the pool is configured)

Know that before.
But should_fail_slab() will ignore GFP flags, thus it doesn't make sense in error injection case.

Thanks,
Qu

Quoted 22 lines
Quoted 22 lines

Thanks,
Qu

Quoted 16 lines

The main reason I'm doing such error injection test is to ensure write time tree checker is not the cause of the lockup.

Of course I can directly inject error into btrfs_check_leaf_full() and btrfs_check_node(), and filter the stack to ensure it only happen in write time, and that's already what I'm crafting, based on the bcc error inject example and kprobe return value overriding.

But it will never be a bad idea to explore what can go wrong. And "always BUG_ON()" -> "good enough (TM)" already looks like a improvement to me.

Thanks,
Qu

#8 lore raw
In reply to Qu Wenruo
Quoted 65 lines · Qu Wenruo
Quoted 53 lines · Qu Wenruo
Quoted 34 lines · David Sterba
Quoted 18 lines · Qu Wenruo
Quoted 15 lines · Nikolay Borisov
Quoted 11 lines · Qu Wenruo

There is a BUG_ON() in __clear_extent_bit() for memory allocation failure.

While comment of __clear_extent_bit() says it can return error, but we always return 0.

Some __clear_extent_bit() callers just ignore the return value, while some still expect error.

Let's return proper error for this memory allocation anyway, to remove that BUG_ON() as a first step, so at least we can continue test.

Quoted 8 lines
Quoted 4 lines

I remember Josef did some changes into this code and said that prealloc shouldn't fail because this will cause mayhem down the road i.e. proper error handling is missing. If anything I think it should be added first and then remove the BUG_ONs.

That's true, we could have some strange lockup due to lock_extent_bits(), as if some clear_extent_bits() failed due to ENOMEM and caller just ignore the error, we could have a lockup.

Not only lockup but unhandled failed extent range locking totally breaks assumptions that the following code makes and this would lead to unpredictable corruptions. Just count how many lock_extent_bits calls are there. And any caller of __set_extent_bit. There are so many that the BUG_ON is the measure of last resort to prevent worse problems.

Quoted 5 lines
Quoted 2 lines

I'll try to pre-allocate certain amount of extent_state as the last chance of redemption.

This only lowers the chances to hit the allocation error but there's always a case when certain amount + 1 would be needed.

Lower chance is already good enough (TM) for low possibility (0.001) error injection.

And, for real world low memory case, lower chance in btrfs means higher chance in other subsystem, less chance user will blame btrfs. :)

Quoted 4 lines
Quoted 3 lines

Anyway, such BUG_ON() right after kmalloc() is really a blockage for error injection test.

Quoted 8 lines
Quoted 2 lines

Maybe, but the code is not yet in the state to inject memory allocation faiulres to that particular path (ie. the state changes).

With last-chance reservation, we can make state related memory allocation almost always to success even memory allocation failure injected (if the possibility is low and low concurrency)
And the last-chance reservation can be configured at compile/module load time, making it flex enough for most cases.

Quoted 5 lines

Forgot to mention, for that method, I'll definitely keep the BUG_ON() on @prealloc.

Just make the allocation part fall back to use fs_info->last_chance[] to grab a valid memory slot.

This makes no sense in fact, for bcc inject tool, modify the predicts can filter out the NOFAIL case, so all of my previous last-chance method just makes no sense anyway.

I'm re-testing using the refined predicates: '(!(gfpflags & __GFP_NOFAIL)) => submit_one_bio()'

And just now, I hit the same lock_extent_bits() hang with NOFAIL filter. So this is definitely something wrong in the submit_one_bio() path.

Thanks,
Qu

Quoted 20 lines

Thanks,
Qu

Quoted 16 lines

The main reason I'm doing such error injection test is to ensure write time tree checker is not the cause of the lockup.

Of course I can directly inject error into btrfs_check_leaf_full() and btrfs_check_node(), and filter the stack to ensure it only happen in write time, and that's already what I'm crafting, based on the bcc error inject example and kprobe return value overriding.

But it will never be a bad idea to explore what can go wrong. And "always BUG_ON()" -> "good enough (TM)" already looks like a improvement to me.

Thanks,
Qu

#9 lore raw
In reply to Nikolay Borisov
Quoted 12 lines · Nikolay Borisov
Quoted 9 lines · Qu Wenruo

There is a BUG_ON() in __clear_extent_bit() for memory allocation failure.

While comment of __clear_extent_bit() says it can return error, but we always return 0.

Some __clear_extent_bit() callers just ignore the return value, while some still expect error.

Quoted 7 lines
Quoted 2 lines

Let's return proper error for this memory allocation anyway, to remove that BUG_ON() as a first step, so at least we can continue test.

I remember Josef did some changes into this code and said that prealloc shouldn't fail because this will cause mayhem down the road i.e. proper error handling is missing. If anything I think it should be added first and then remove the BUG_ONs.

I'm ok with changing this, but it needs to be more than just one patch. Make alloc_extent_state_atomic() error injectable and then use either error-inject.py in bcc or use
https://github.com/josefbacik/debug-scripts/blob/master/inject-error.py to test it and fix all the fallout from not handling lock_extent() succeeding.

You can use https://github.com/josefbacik/debug-scripts/blob/master/codepaths.py
to find all call sites for inject-error to make sure you are catching all the cases. Thanks,

Josef

#10 lore raw
Quoted 21 lines · Josef Bacik
Quoted 12 lines · Nikolay Borisov
Quoted 9 lines · Qu Wenruo

There is a BUG_ON() in __clear_extent_bit() for memory allocation failure.

While comment of __clear_extent_bit() says it can return error, but we always return 0.

Some __clear_extent_bit() callers just ignore the return value, while some still expect error.

Quoted 7 lines
Quoted 2 lines

Let's return proper error for this memory allocation anyway, to remove that BUG_ON() as a first step, so at least we can continue test.

I remember Josef did some changes into this code and said that prealloc shouldn't fail because this will cause mayhem down the road i.e. proper error handling is missing. If anything I think it should be added first and then remove the BUG_ONs.

Quoted 5 lines

I'm ok with changing this, but it needs to be more than just one patch. Make alloc_extent_state_atomic() error injectable and then use either error-inject.py in bcc or use
https://github.com/josefbacik/debug-scripts/blob/master/inject-error.py to test it and fix all the fallout from not handling lock_extent() succeeding.

Exactly what I'm going to do. With some blend from bcc inject.py, for stack filter to provide more precious match.

Quoted 4 lines

You can use https://github.com/josefbacik/debug-scripts/blob/master/codepaths.py
to find all call sites for inject-error to make sure you are catching all the cases. Thanks,

That script is awesome!

Thanks,
Qu

Quoted 3 lines

Josef