btrfs: extent_io: Handle memory allocation failure in __clear_extent_bit()
PATCH 1/1 btrfs: extent_io: Handle memory allocation failure in __clear_extent_bit()
There is a BUG_ON() in __clear_extent_bit() for memory allocation failure.
While comment of __clear_extent_bit() says it can return error, but we always return 0.
Some __clear_extent_bit() callers just ignore the return value, while some still expect error.
Let's return proper error for this memory allocation anyway, to remove that BUG_ON() as a first step, so at least we can continue test.
--- fs/btrfs/extent_io.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
fs/btrfs/extent_io.c +10 -2
@@ -743,7 +743,10 @@ int __clear_extent_bit(struct extent_io_tree *tree, u64 start, u64 end,if (state->start < start) { prealloc =alloc_extent_state_atomic (prealloc); -BUG_ON (!prealloc); +if (!prealloc) { + err = -ENOMEM ; + goto out; + } err =split_state (tree, state, prealloc, start);if (err)extent_io_tree_panic (tree, err);
@@ -766,7 +769,10 @@ int __clear_extent_bit(struct extent_io_tree *tree, u64 start, u64 end,*/ if (state->start <= end && state->end > end) { prealloc =alloc_extent_state_atomic (prealloc); -BUG_ON (!prealloc); +if (!prealloc) { + err = -ENOMEM ; + goto out; + } err =split_state (tree, state, prealloc, end +1 );if (err)extent_io_tree_panic (tree, err);
@@ -801,6 +807,8 @@ int __clear_extent_bit(struct extent_io_tree *tree, u64 start, u64 end,if (prealloc)free_extent_state (prealloc); +if (err) +return err;return 0 ; }
Signature · 1 line
2.21.0
Quoted 11 lines · Qu Wenruo
There is a
BUG_ON()in__clear_extent_bit()for memory allocation failure.While comment of
__clear_extent_bit()says it can return error, but we always return 0.Some
__clear_extent_bit()callers just ignore the return value, while some still expect error.Let's return proper error for this memory allocation anyway, to remove that
BUG_ON()as a first step, so at least we can continue test.
I remember Josef did some changes into this code and said that prealloc shouldn't fail because this will cause mayhem down the road i.e. proper error handling is missing. If anything I think it should be added first and then remove the BUG_ONs.
Quoted 44 lines
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/extent_io.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)fs/btrfs/extent_io.c +10
-2@@ -743,7 +743,10 @@ int __clear_extent_bit(struct extent_io_tree *tree, u64 start, u64 end,if (state->start < start) { prealloc =alloc_extent_state_atomic (prealloc); -BUG_ON (!prealloc); +if (!prealloc) { + err = -ENOMEM ; + goto out; + } err =split_state (tree, state, prealloc, start);if (err)extent_io_tree_panic (tree, err);@@ -766,7 +769,10 @@ int __clear_extent_bit(struct extent_io_tree *tree, u64 start, u64 end,*/ if (state->start <= end && state->end > end) { prealloc =alloc_extent_state_atomic (prealloc); -BUG_ON (!prealloc); +if (!prealloc) { + err = -ENOMEM ; + goto out; + } err =split_state (tree, state, prealloc, end +1 );if (err)extent_io_tree_panic (tree, err);@@ -801,6 +807,8 @@ int __clear_extent_bit(struct extent_io_tree *tree, u64 start, u64 end,if (prealloc)free_extent_state (prealloc); +if (err) +return err;return 0 ; }
Quoted 12 lines · Nikolay Borisov
Quoted 9 lines · Qu Wenruo
There is a
BUG_ON()in__clear_extent_bit()for memory allocation failure.While comment of
__clear_extent_bit()says it can return error, but we always return 0.Some
__clear_extent_bit()callers just ignore the return value, while some still expect error.
Quoted 7 lines
Quoted 2 lines
Let's return proper error for this memory allocation anyway, to remove that
BUG_ON()as a first step, so at least we can continue test.I remember Josef did some changes into this code and said that prealloc shouldn't fail because this will cause mayhem down the road i.e. proper error handling is missing. If anything I think it should be added first and then remove the
BUG_ONs.
That's true, we could have some strange lockup due to lock_extent_bits(), as if some clear_extent_bits() failed due to ENOMEM and caller just ignore the error, we could have a lockup.
I'll try to pre-allocate certain amount of extent_state as the last chance of redemption.
Anyway, such BUG_ON() right after kmalloc() is really a blockage for error injection test.
Thanks,
Qu
Quoted 45 lines
Quoted 45 lines
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/extent_io.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)fs/btrfs/extent_io.c +10
-2@@ -743,7 +743,10 @@ int __clear_extent_bit(struct extent_io_tree *tree, u64 start, u64 end,if (state->start < start) { prealloc =alloc_extent_state_atomic (prealloc); -BUG_ON (!prealloc); +if (!prealloc) { + err = -ENOMEM ; + goto out; + } err =split_state (tree, state, prealloc, start);if (err)extent_io_tree_panic (tree, err);@@ -766,7 +769,10 @@ int __clear_extent_bit(struct extent_io_tree *tree, u64 start, u64 end,*/ if (state->start <= end && state->end > end) { prealloc =alloc_extent_state_atomic (prealloc); -BUG_ON (!prealloc); +if (!prealloc) { + err = -ENOMEM ; + goto out; + } err =split_state (tree, state, prealloc, end +1 );if (err)extent_io_tree_panic (tree, err);@@ -801,6 +807,8 @@ int __clear_extent_bit(struct extent_io_tree *tree, u64 start, u64 end,if (prealloc)free_extent_state (prealloc); +if (err) +return err;return 0 ; }
Quoted 18 lines · Qu Wenruo
Quoted 15 lines · Nikolay Borisov
Quoted 11 lines · Qu Wenruo
There is a
BUG_ON()in__clear_extent_bit()for memory allocation failure.While comment of
__clear_extent_bit()says it can return error, but we always return 0.Some
__clear_extent_bit()callers just ignore the return value, while some still expect error.Let's return proper error for this memory allocation anyway, to remove that
BUG_ON()as a first step, so at least we can continue test.
Quoted 8 lines
Quoted 4 lines
I remember Josef did some changes into this code and said that prealloc shouldn't fail because this will cause mayhem down the road i.e. proper error handling is missing. If anything I think it should be added first and then remove the
BUG_ONs.That's true, we could have some strange lockup due to
lock_extent_bits(), as if someclear_extent_bits()failed due to ENOMEM and caller just ignore the error, we could have a lockup.
Not only lockup but unhandled failed extent range locking totally breaks assumptions that the following code makes and this would lead to unpredictable corruptions. Just count how many lock_extent_bits calls are there. And any caller of __set_extent_bit. There are so many that the BUG_ON is the measure of last resort to prevent worse problems.
Quoted 2 lines
I'll try to pre-allocate certain amount of
extent_stateas the last chance of redemption.
This only lowers the chances to hit the allocation error but there's always a case when certain amount + 1 would be needed.
Quoted 2 lines
Anyway, such
BUG_ON()right afterkmalloc()is really a blockage for error injection test.
Maybe, but the code is not yet in the state to inject memory allocation faiulres to that particular path (ie. the state changes).
Quoted 34 lines · David Sterba
Quoted 18 lines · Qu Wenruo
Quoted 15 lines · Nikolay Borisov
Quoted 11 lines · Qu Wenruo
There is a
BUG_ON()in__clear_extent_bit()for memory allocation failure.While comment of
__clear_extent_bit()says it can return error, but we always return 0.Some
__clear_extent_bit()callers just ignore the return value, while some still expect error.Let's return proper error for this memory allocation anyway, to remove that
BUG_ON()as a first step, so at least we can continue test.Quoted 8 lines
Quoted 4 lines
I remember Josef did some changes into this code and said that prealloc shouldn't fail because this will cause mayhem down the road i.e. proper error handling is missing. If anything I think it should be added first and then remove the
BUG_ONs.That's true, we could have some strange lockup due to
lock_extent_bits(), as if someclear_extent_bits()failed due to ENOMEM and caller just ignore the error, we could have a lockup.Not only lockup but unhandled failed extent range locking totally breaks assumptions that the following code makes and this would lead to unpredictable corruptions. Just count how many
lock_extent_bitscalls are there. And any caller of__set_extent_bit. There are so many that theBUG_ONis the measure of last resort to prevent worse problems.
Quoted 5 lines
Quoted 2 lines
I'll try to pre-allocate certain amount of
extent_stateas the last chance of redemption.This only lowers the chances to hit the allocation error but there's always a case when certain amount + 1 would be needed.
Lower chance is already good enough (TM) for low possibility (0.001) error injection.
And, for real world low memory case, lower chance in btrfs means higher chance in other subsystem, less chance user will blame btrfs. :)
Quoted 6 lines
Quoted 3 lines
Anyway, such
BUG_ON()right afterkmalloc()is really a blockage for error injection test.Maybe, but the code is not yet in the state to inject memory allocation faiulres to that particular path (ie. the state changes).
With last-chance reservation, we can make state related memory allocation almost always to success even memory allocation failure injected (if the possibility is low and low concurrency)
And the last-chance reservation can be configured at compile/module load time, making it flex enough for most cases.
The main reason I'm doing such error injection test is to ensure write time tree checker is not the cause of the lockup.
Of course I can directly inject error into btrfs_check_leaf_full() and btrfs_check_node(), and filter the stack to ensure it only happen in write time, and that's already what I'm crafting, based on the bcc error inject example and kprobe return value overriding.
But it will never be a bad idea to explore what can go wrong. And "always BUG_ON()" -> "good enough (TM)" already looks like a improvement to me.
Thanks,
Qu
Quoted 53 lines · Qu Wenruo
Quoted 34 lines · David Sterba
Quoted 18 lines · Qu Wenruo
Quoted 15 lines · Nikolay Borisov
Quoted 11 lines · Qu Wenruo
There is a
BUG_ON()in__clear_extent_bit()for memory allocation failure.While comment of
__clear_extent_bit()says it can return error, but we always return 0.Some
__clear_extent_bit()callers just ignore the return value, while some still expect error.Let's return proper error for this memory allocation anyway, to remove that
BUG_ON()as a first step, so at least we can continue test.Quoted 8 lines
Quoted 4 lines
I remember Josef did some changes into this code and said that prealloc shouldn't fail because this will cause mayhem down the road i.e. proper error handling is missing. If anything I think it should be added first and then remove the
BUG_ONs.That's true, we could have some strange lockup due to
lock_extent_bits(), as if someclear_extent_bits()failed due to ENOMEM and caller just ignore the error, we could have a lockup.Not only lockup but unhandled failed extent range locking totally breaks assumptions that the following code makes and this would lead to unpredictable corruptions. Just count how many
lock_extent_bitscalls are there. And any caller of__set_extent_bit. There are so many that theBUG_ONis the measure of last resort to prevent worse problems.Quoted 5 lines
Quoted 2 lines
I'll try to pre-allocate certain amount of
extent_stateas the last chance of redemption.This only lowers the chances to hit the allocation error but there's always a case when certain amount + 1 would be needed.
Lower chance is already good enough (TM) for low possibility (0.001) error injection.
And, for real world low memory case, lower chance in btrfs means higher chance in other subsystem, less chance user will blame btrfs. :)
Quoted 4 lines
Quoted 3 lines
Anyway, such
BUG_ON()right afterkmalloc()is really a blockage for error injection test.
Quoted 8 lines
Quoted 2 lines
Maybe, but the code is not yet in the state to inject memory allocation faiulres to that particular path (ie. the state changes).
With last-chance reservation, we can make state related memory allocation almost always to success even memory allocation failure injected (if the possibility is low and low concurrency)
And the last-chance reservation can be configured at compile/module load time, making it flex enough for most cases.
Forgot to mention, for that method, I'll definitely keep the BUG_ON() on @prealloc.
Just make the allocation part fall back to use fs_info->last_chance[] to grab a valid memory slot.
Thanks,
Qu
Quoted 16 lines
The main reason I'm doing such error injection test is to ensure write time tree checker is not the cause of the lockup.
Of course I can directly inject error into
btrfs_check_leaf_full()andbtrfs_check_node(), and filter the stack to ensure it only happen in write time, and that's already what I'm crafting, based on the bcc error inject example and kprobe return value overriding.But it will never be a bad idea to explore what can go wrong. And "always
BUG_ON()" -> "good enough (TM)" already looks like a improvement to me.Thanks,
Qu
On 18.04.19 г. 14:54 ч., Qu Wenruo wrote:
<snip>
Quoted 6 lines
Forgot to mention, for that method, I'll definitely keep the
BUG_ON()on @prealloc.Just make the allocation part fall back to use fs_info->last_chance[] to grab a valid memory slot.
Before you go and re-invent the wheel I will suggest you take a look at the mempool api (include/linux/mempool.h). AFAIK it's purpose is to exactly ensure memory allocations from it will not fail (depending on how the pool is configured)
Quoted 21 lines
Thanks,
QuQuoted 16 lines
The main reason I'm doing such error injection test is to ensure write time tree checker is not the cause of the lockup.
Of course I can directly inject error into
btrfs_check_leaf_full()andbtrfs_check_node(), and filter the stack to ensure it only happen in write time, and that's already what I'm crafting, based on the bcc error inject example and kprobe return value overriding.But it will never be a bad idea to explore what can go wrong. And "always
BUG_ON()" -> "good enough (TM)" already looks like a improvement to me.Thanks,
Qu
Quoted 9 lines · Nikolay Borisov
On 18.04.19 г. 14:54 ч., Qu Wenruo wrote:
<snip>Quoted 4 lines
Forgot to mention, for that method, I'll definitely keep the
BUG_ON()on @prealloc.
Quoted 7 lines
Quoted 2 lines
Just make the allocation part fall back to use fs_info->last_chance[] to grab a valid memory slot.
Before you go and re-invent the wheel I will suggest you take a look at the mempool api (
include/linux/mempool.h). AFAIK it's purpose is to exactly ensure memory allocations from it will not fail (depending on how the pool is configured)
Know that before.
But should_fail_slab() will ignore GFP flags, thus it doesn't make sense in error injection case.
Thanks,
Qu
Quoted 22 lines
Quoted 22 lines
Thanks,
QuQuoted 16 lines
The main reason I'm doing such error injection test is to ensure write time tree checker is not the cause of the lockup.
Of course I can directly inject error into
btrfs_check_leaf_full()andbtrfs_check_node(), and filter the stack to ensure it only happen in write time, and that's already what I'm crafting, based on the bcc error inject example and kprobe return value overriding.But it will never be a bad idea to explore what can go wrong. And "always
BUG_ON()" -> "good enough (TM)" already looks like a improvement to me.Thanks,
Qu
Quoted 65 lines · Qu Wenruo
Quoted 53 lines · Qu Wenruo
Quoted 34 lines · David Sterba
Quoted 18 lines · Qu Wenruo
Quoted 15 lines · Nikolay Borisov
Quoted 11 lines · Qu Wenruo
There is a
BUG_ON()in__clear_extent_bit()for memory allocation failure.While comment of
__clear_extent_bit()says it can return error, but we always return 0.Some
__clear_extent_bit()callers just ignore the return value, while some still expect error.Let's return proper error for this memory allocation anyway, to remove that
BUG_ON()as a first step, so at least we can continue test.Quoted 8 lines
Quoted 4 lines
I remember Josef did some changes into this code and said that prealloc shouldn't fail because this will cause mayhem down the road i.e. proper error handling is missing. If anything I think it should be added first and then remove the
BUG_ONs.That's true, we could have some strange lockup due to
lock_extent_bits(), as if someclear_extent_bits()failed due to ENOMEM and caller just ignore the error, we could have a lockup.Not only lockup but unhandled failed extent range locking totally breaks assumptions that the following code makes and this would lead to unpredictable corruptions. Just count how many
lock_extent_bitscalls are there. And any caller of__set_extent_bit. There are so many that theBUG_ONis the measure of last resort to prevent worse problems.Quoted 5 lines
Quoted 2 lines
I'll try to pre-allocate certain amount of
extent_stateas the last chance of redemption.This only lowers the chances to hit the allocation error but there's always a case when certain amount + 1 would be needed.
Lower chance is already good enough (TM) for low possibility (0.001) error injection.
And, for real world low memory case, lower chance in btrfs means higher chance in other subsystem, less chance user will blame btrfs. :)
Quoted 4 lines
Quoted 3 lines
Anyway, such
BUG_ON()right afterkmalloc()is really a blockage for error injection test.Quoted 8 lines
Quoted 2 lines
Maybe, but the code is not yet in the state to inject memory allocation faiulres to that particular path (ie. the state changes).
With last-chance reservation, we can make state related memory allocation almost always to success even memory allocation failure injected (if the possibility is low and low concurrency)
And the last-chance reservation can be configured at compile/module load time, making it flex enough for most cases.
Quoted 5 lines
Forgot to mention, for that method, I'll definitely keep the
BUG_ON()on @prealloc.Just make the allocation part fall back to use fs_info->last_chance[] to grab a valid memory slot.
This makes no sense in fact, for bcc inject tool, modify the predicts can filter out the NOFAIL case, so all of my previous last-chance method just makes no sense anyway.
I'm re-testing using the refined predicates: '(!(gfpflags & __GFP_NOFAIL)) => submit_one_bio()'
And just now, I hit the same lock_extent_bits() hang with NOFAIL filter. So this is definitely something wrong in the submit_one_bio() path.
Thanks,
Qu
Quoted 20 lines
Thanks,
QuQuoted 16 lines
The main reason I'm doing such error injection test is to ensure write time tree checker is not the cause of the lockup.
Of course I can directly inject error into
btrfs_check_leaf_full()andbtrfs_check_node(), and filter the stack to ensure it only happen in write time, and that's already what I'm crafting, based on the bcc error inject example and kprobe return value overriding.But it will never be a bad idea to explore what can go wrong. And "always
BUG_ON()" -> "good enough (TM)" already looks like a improvement to me.Thanks,
Qu
Quoted 12 lines · Nikolay Borisov
Quoted 9 lines · Qu Wenruo
There is a
BUG_ON()in__clear_extent_bit()for memory allocation failure.While comment of
__clear_extent_bit()says it can return error, but we always return 0.Some
__clear_extent_bit()callers just ignore the return value, while some still expect error.
Quoted 7 lines
Quoted 2 lines
Let's return proper error for this memory allocation anyway, to remove that
BUG_ON()as a first step, so at least we can continue test.I remember Josef did some changes into this code and said that prealloc shouldn't fail because this will cause mayhem down the road i.e. proper error handling is missing. If anything I think it should be added first and then remove the
BUG_ONs.
I'm ok with changing this, but it needs to be more than just one patch. Make alloc_extent_state_atomic() error injectable and then use either error-inject.py in bcc or use
https://github.com/josefbacik/debug-scripts/blob/master/inject-error.py to test it and fix all the fallout from not handling lock_extent() succeeding.
You can use https://github.com/josefbacik/debug-scripts/blob/master/codepaths.py
to find all call sites for inject-error to make sure you are catching all the cases. Thanks,
Josef
Quoted 21 lines · Josef Bacik
Quoted 12 lines · Nikolay Borisov
Quoted 9 lines · Qu Wenruo
There is a
BUG_ON()in__clear_extent_bit()for memory allocation failure.While comment of
__clear_extent_bit()says it can return error, but we always return 0.Some
__clear_extent_bit()callers just ignore the return value, while some still expect error.Quoted 7 lines
Quoted 2 lines
Let's return proper error for this memory allocation anyway, to remove that
BUG_ON()as a first step, so at least we can continue test.I remember Josef did some changes into this code and said that prealloc shouldn't fail because this will cause mayhem down the road i.e. proper error handling is missing. If anything I think it should be added first and then remove the
BUG_ONs.
Quoted 5 lines
I'm ok with changing this, but it needs to be more than just one patch. Make
alloc_extent_state_atomic()error injectable and then use eithererror-inject.pyin bcc or use
https://github.com/josefbacik/debug-scripts/blob/master/inject-error.py to test it and fix all the fallout from not handlinglock_extent()succeeding.
Exactly what I'm going to do. With some blend from bcc inject.py, for stack filter to provide more precious match.
Quoted 4 lines
You can use https://github.com/josefbacik/debug-scripts/blob/master/codepaths.py
to find all call sites for inject-error to make sure you are catching all the cases. Thanks,
That script is awesome!
Thanks,
Qu
Quoted 3 lines
Josef