Skip to contentSkip to conversation

btrfs: wait on caching when putting the bg cache

PATCH In review Josef Bacik 4 participants 6 messages Sep 12, 2018 – Sep 13, 2018 Btrfs file system

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 btrfs: wait on caching when putting the bg cache

While testing my backport I noticed there was a panic if I ran generic/416 generic/417 generic/418 all in a row. This just happened to uncover a race where we had outstanding IO after we destroy all of our workqueues, and then we'd go to queue the endio work on those free'd workqueues. This is because we aren't waiting for the caching threads to be done before freeing everything up, so to fix this make sure we wait on any outstanding caching that's being done before we free up the block group, so we're sure to be done with all IO by the time we get to btrfs_stop_all_workers(). This fixes the panic I was seeing consistently in testing.

---
 fs/btrfs/extent-tree.c | 1 +
 1 file changed, 1 insertion(+)
fs/btrfs/extent-tree.c +1 -0
index 414492a18f1e..2eb2e37f2354 100644
@@ -9889,6 +9889,7 @@ void btrfs_put_block_group_cache(struct btrfs_fs_info *info)
 
 		block_group = btrfs_lookup_first_block_group(info, last);
 		while (block_group) {
+			wait_block_group_cache_done(block_group);
 			spin_lock(&block_group->lock);
 			if (block_group->iref)
 				break;
Signature · 1 line
2.14.3
#1 lore raw
Quoted 10 lines · Josef Bacik

While testing my backport I noticed there was a panic if I ran generic/416 generic/417 generic/418 all in a row. This just happened to uncover a race where we had outstanding IO after we destroy all of our workqueues, and then we'd go to queue the endio work on those free'd workqueues. This is because we aren't waiting for the caching threads to be done before freeing everything up, so to fix this make sure we wait on any outstanding caching that's being done before we free up the block group, so we're sure to be done with all IO by the time we get to btrfs_stop_all_workers(). This fixes the panic I was seeing consistently in testing.

It's not clear whether this is caused by one of the patches in your latest patchbomb or has the issue been there all along?

Quoted 19 lines
---
 fs/btrfs/extent-tree.c | 1 +
 1 file changed, 1 insertion(+)
fs/btrfs/extent-tree.c +1 -0
index 414492a18f1e..2eb2e37f2354 100644
@@ -9889,6 +9889,7 @@ void btrfs_put_block_group_cache(struct btrfs_fs_info *info)
 
 		block_group = btrfs_lookup_first_block_group(info, last);
 		while (block_group) {
+			wait_block_group_cache_done(block_group);
 			spin_lock(&block_group->lock);
 			if (block_group->iref)
 				break;
#2 lore raw
Quoted 14 lines · Nikolay Borisov
Quoted 10 lines · Josef Bacik

While testing my backport I noticed there was a panic if I ran generic/416 generic/417 generic/418 all in a row. This just happened to uncover a race where we had outstanding IO after we destroy all of our workqueues, and then we'd go to queue the endio work on those free'd workqueues. This is because we aren't waiting for the caching threads to be done before freeing everything up, so to fix this make sure we wait on any outstanding caching that's being done before we free up the block group, so we're sure to be done with all IO by the time we get to btrfs_stop_all_workers(). This fixes the panic I was seeing consistently in testing.

Quoted 2 lines

It's not clear whether this is caused by one of the patches in your latest patchbomb or has the issue been there all along?

Been here always, I noticed this on the backport of linus/master before I even got to pulling my shit ontop of that. Thanks,

Josef

Reviewed-by on v1 1/1
#3 lore raw
In reply to Josef Bacik
Quoted 10 lines · Josef Bacik

While testing my backport I noticed there was a panic if I ran generic/416 generic/417 generic/418 all in a row. This just happened to uncover a race where we had outstanding IO after we destroy all of our workqueues, and then we'd go to queue the endio work on those free'd workqueues. This is because we aren't waiting for the caching threads to be done before freeing everything up, so to fix this make sure we wait on any outstanding caching that's being done before we free up the block group, so we're sure to be done with all IO by the time we get to btrfs_stop_all_workers(). This fixes the panic I was seeing consistently in testing.

Quoted 20 lines
---
 fs/btrfs/extent-tree.c | 1 +
 1 file changed, 1 insertion(+)
fs/btrfs/extent-tree.c +1 -0
index 414492a18f1e..2eb2e37f2354 100644
@@ -9889,6 +9889,7 @@ void btrfs_put_block_group_cache(struct btrfs_fs_info *info)
 
 		block_group = btrfs_lookup_first_block_group(info, last);
 		while (block_group) {
+			wait_block_group_cache_done(block_group);
 			spin_lock(&block_group->lock);
 			if (block_group->iref)
 				break;
Signature · 1 line
2.14.3
#4 lore raw
In reply to Josef Bacik
Quoted 10 lines · Josef Bacik

While testing my backport I noticed there was a panic if I ran generic/416 generic/417 generic/418 all in a row. This just happened to uncover a race where we had outstanding IO after we destroy all of our workqueues, and then we'd go to queue the endio work on those free'd workqueues. This is because we aren't waiting for the caching threads to be done before freeing everything up, so to fix this make sure we wait on any outstanding caching that's being done before we free up the block group, so we're sure to be done with all IO by the time we get to btrfs_stop_all_workers(). This fixes the panic I was seeing consistently in testing.

Can you please attach the stacktrace(s)? I think I've seen similar error once or twice but not able to reproduce.

#5 lore raw
Quoted 12 lines · David Sterba
Quoted 10 lines · Josef Bacik

While testing my backport I noticed there was a panic if I ran generic/416 generic/417 generic/418 all in a row. This just happened to uncover a race where we had outstanding IO after we destroy all of our workqueues, and then we'd go to queue the endio work on those free'd workqueues. This is because we aren't waiting for the caching threads to be done before freeing everything up, so to fix this make sure we wait on any outstanding caching that's being done before we free up the block group, so we're sure to be done with all IO by the time we get to btrfs_stop_all_workers(). This fixes the panic I was seeing consistently in testing.

Quoted 2 lines

Can you please attach the stacktrace(s)? I think I've seen similar error once or twice but not able to reproduce.

I found at least this one https://patchwork.kernel.org/patch/10495885/,
when the rbio cache is destroyed, there's some in-flight IO. This is not the example I had in mind before but still roughly matches the symptoms.