btrfs: wait on caching when putting the bg cache
PATCH 1/1 btrfs: wait on caching when putting the bg cache
While testing my backport I noticed there was a panic if I ran generic/416 generic/417 generic/418 all in a row. This just happened to uncover a race where we had outstanding IO after we destroy all of our workqueues, and then we'd go to queue the endio work on those free'd workqueues. This is because we aren't waiting for the caching threads to be done before freeing everything up, so to fix this make sure we wait on any outstanding caching that's being done before we free up the block group, so we're sure to be done with all IO by the time we get to btrfs_stop_all_workers(). This fixes the panic I was seeing consistently in testing.
--- fs/btrfs/extent-tree.c | 1 + 1 file changed, 1 insertion(+)
fs/btrfs/extent-tree.c +1 -0
@@ -9889,6 +9889,7 @@ void btrfs_put_block_group_cache(struct btrfs_fs_info *info) block_group =btrfs_lookup_first_block_group (info, last);while (block_group) { +wait_block_group_cache_done (block_group);spin_lock (&block_group->lock);if (block_group->iref)break ;
Signature · 1 line
2.14.3
Quoted 10 lines · Josef Bacik
While testing my backport I noticed there was a panic if I ran generic/416 generic/417 generic/418 all in a row. This just happened to uncover a race where we had outstanding IO after we destroy all of our workqueues, and then we'd go to queue the endio work on those free'd workqueues. This is because we aren't waiting for the caching threads to be done before freeing everything up, so to fix this make sure we wait on any outstanding caching that's being done before we free up the block group, so we're sure to be done with all IO by the time we get to
btrfs_stop_all_workers(). This fixes the panic I was seeing consistently in testing.
It's not clear whether this is caused by one of the patches in your latest patchbomb or has the issue been there all along?
Quoted 19 lines
Signed-off-by Josef Bacik <[email protected]>--- fs/btrfs/extent-tree.c | 1 + 1 file changed, 1 insertion(+)fs/btrfs/extent-tree.c +1
-0@@ -9889,6 +9889,7 @@ void btrfs_put_block_group_cache(struct btrfs_fs_info *info) block_group =btrfs_lookup_first_block_group (info, last);while (block_group) { +wait_block_group_cache_done (block_group);spin_lock (&block_group->lock);if (block_group->iref)break ;
Quoted 14 lines · Nikolay Borisov
Quoted 10 lines · Josef Bacik
While testing my backport I noticed there was a panic if I ran generic/416 generic/417 generic/418 all in a row. This just happened to uncover a race where we had outstanding IO after we destroy all of our workqueues, and then we'd go to queue the endio work on those free'd workqueues. This is because we aren't waiting for the caching threads to be done before freeing everything up, so to fix this make sure we wait on any outstanding caching that's being done before we free up the block group, so we're sure to be done with all IO by the time we get to
btrfs_stop_all_workers(). This fixes the panic I was seeing consistently in testing.
Quoted 2 lines
It's not clear whether this is caused by one of the patches in your latest patchbomb or has the issue been there all along?
Been here always, I noticed this on the backport of linus/master before I even got to pulling my shit ontop of that. Thanks,
Josef
Quoted 10 lines · Josef Bacik
While testing my backport I noticed there was a panic if I ran generic/416 generic/417 generic/418 all in a row. This just happened to uncover a race where we had outstanding IO after we destroy all of our workqueues, and then we'd go to queue the endio work on those free'd workqueues. This is because we aren't waiting for the caching threads to be done before freeing everything up, so to fix this make sure we wait on any outstanding caching that's being done before we free up the block group, so we're sure to be done with all IO by the time we get to
btrfs_stop_all_workers(). This fixes the panic I was seeing consistently in testing.
Quoted 20 lines
Signed-off-by Josef Bacik <[email protected]>--- fs/btrfs/extent-tree.c | 1 + 1 file changed, 1 insertion(+)fs/btrfs/extent-tree.c +1
-0@@ -9889,6 +9889,7 @@ void btrfs_put_block_group_cache(struct btrfs_fs_info *info) block_group =btrfs_lookup_first_block_group (info, last);while (block_group) { +wait_block_group_cache_done (block_group);spin_lock (&block_group->lock);if (block_group->iref)break ;Signature · 1 line
2.14.3
Quoted 10 lines · Josef Bacik
While testing my backport I noticed there was a panic if I ran generic/416 generic/417 generic/418 all in a row. This just happened to uncover a race where we had outstanding IO after we destroy all of our workqueues, and then we'd go to queue the endio work on those free'd workqueues. This is because we aren't waiting for the caching threads to be done before freeing everything up, so to fix this make sure we wait on any outstanding caching that's being done before we free up the block group, so we're sure to be done with all IO by the time we get to
btrfs_stop_all_workers(). This fixes the panic I was seeing consistently in testing.
Can you please attach the stacktrace(s)? I think I've seen similar error once or twice but not able to reproduce.
Quoted 12 lines · David Sterba
Quoted 10 lines · Josef Bacik
While testing my backport I noticed there was a panic if I ran generic/416 generic/417 generic/418 all in a row. This just happened to uncover a race where we had outstanding IO after we destroy all of our workqueues, and then we'd go to queue the endio work on those free'd workqueues. This is because we aren't waiting for the caching threads to be done before freeing everything up, so to fix this make sure we wait on any outstanding caching that's being done before we free up the block group, so we're sure to be done with all IO by the time we get to
btrfs_stop_all_workers(). This fixes the panic I was seeing consistently in testing.
Quoted 2 lines
Can you please attach the stacktrace(s)? I think I've seen similar error once or twice but not able to reproduce.
I found at least this one https://patchwork.kernel.org/patch/10495885/,
when the rbio cache is destroyed, there's some in-flight IO. This is not the example I had in mind before but still roughly matches the symptoms.