Btrfs: fix possible softlockup in the allocator
PATCH 1/1 Btrfs: fix possible softlockup in the allocator
Like the cluster allocating stuff, we can lockup the box with the normal allocation path. This happens when we
1) Start to cache a block group that is severely fragmented, but has a decent amount of free space.
2) Start to commit a transaction
3) Have the commit try and empty out some of the delalloc inodes with extents that are relatively large.
The inodes will not be able to make the allocations because they will ask for allocations larger than a contiguous area in the free space cache. So we will wait for more progress to be made on the block group, but since we're in a commit the caching kthread won't make any more progress and it already has enough free space that wait_block_group_cache_progress will just return. So, if we wait and fail to make the allocation the next time around, just loop and go to the next block group. This keeps us from getting stuck in a softlockup. Thanks,
--- fs/btrfs/extent-tree.c | 23 +++++++++++++++++------ 1 files changed, 17 insertions(+), 6 deletions(-)
fs/btrfs/extent-tree.c +17 -6
@@ -3816,6 +3816,7 @@ static noinline int find_free_extent(struct btrfs_trans_handle *trans,int loop =0 ;bool found_uncached_bg = false;bool failed_cluster_refill = false; +bool failed_alloc = false;WARN_ON (num_bytes < root->sectorsize);btrfs_set_key_type (ins,BTRFS_EXTENT_ITEM_KEY );
@@ -4020,14 +4021,23 @@ refill_cluster: offset =btrfs_find_space_for_alloc (block_group, search_start, num_bytes, empty_size); -if (!offset && (cached || (!cached && - loop ==LOOP_CACHING_NOWAIT ))) { - goto loop; - }else if (!offset && (!cached && - loop >LOOP_CACHING_NOWAIT )) { +/* +* If we didn't find a chunk, and we haven't failed on this +* block group before, and this block group is in the middle of +* caching and we are ok with waiting, then go ahead and wait +* for progress to be made, and set failed_alloc to true. +* +* If failed_alloc is true then we've already waited on this +* block group once and should move on to the next block group. +*/ +if (!offset && !failed_alloc && !cached && + loop >LOOP_CACHING_NOWAIT ) {wait_block_group_cache_progress (block_group, - num_bytes + empty_size); + num_bytes + empty_size); + failed_alloc = true; goto have_block_group; + }else if (!offset) { + goto loop; } checks: search_start =stripe_align (root, offset);
@@ -4075,6 +4085,7 @@ checks:break ; loop: failed_cluster_refill = false; + failed_alloc = false;btrfs_put_block_group (block_group); }up_read (&space_info->groups_sem);
Signature · 1 line
1.5.4.3
Quoted 2 lines · Josef Bacik
Like the cluster allocating stuff, we can lockup the box with the normal allocation path. This happens when we
This is working for me, I'm hammering on it a bit.
-chris
Am Montag 05 Oktober 2009 21:30:39 schrieb Josef Bacik:
Quoted 67 lines
Like the cluster allocating stuff, we can lockup the box with the normal allocation path. This happens when we
1) Start to cache a block group that is severely fragmented, but has a decent amount of free space.
2) Start to commit a transaction
3) Have the commit try and empty out some of the delalloc inodes with extents that are relatively large.The inodes will not be able to make the allocations because they will ask
for allocations larger than a contiguous area in the free space cache. So we will wait for more progress to be made on the block group, but since we're in a commit the caching kthread won't make any more progress and it already has enough free space thatwait_block_group_cache_progresswill just return. So, if we wait and fail to make the allocation the next time around, just loop and go to the next block group. This keeps us from getting stuck in a softlockup. Thanks,Signed-off-by Josef Bacik <[email protected]>--- fs/btrfs/extent-tree.c | 23 +++++++++++++++++------ 1 files changed, 17 insertions(+), 6 deletions(-)fs/btrfs/extent-tree.c +16
-6@@ -3816,6 +3816,7 @@ static noinline int find_free_extent(struct btrfs_trans_handle *trans, int loop = 0;bool found_uncached_bg = false;bool failed_cluster_refill = false; +bool failed_alloc = false;WARN_ON (num_bytes < root->sectorsize);btrfs_set_key_type (ins,BTRFS_EXTENT_ITEM_KEY );@@ -4020,14 +4021,23 @@ refill_cluster: offset =btrfs_find_space_for_alloc (block_group, search_start, num_bytes, empty_size); -if (!offset && (cached || (!cached && - loop ==LOOP_CACHING_NOWAIT ))) { - goto loop; - }else if (!offset && (!cached && - loop >LOOP_CACHING_NOWAIT )) { +/* +* If we didn't find a chunk, and we haven't failed on this +* block group before, and this block group is in the middle of +* caching and we are ok with waiting, then go ahead and wait +* for progress to be made, and set failed_alloc to true. +* +* If failed_alloc is true then we've already waited on this +* block group once and should move on to the next block group. +*/ +if (!offset && !failed_alloc && !cached && + loop >LOOP_CACHING_NOWAIT ) {wait_block_group_cache_progress (block_group, - num_bytes + empty_size); + num_bytes + empty_size); + failed_alloc = true; goto have_block_group; + }else if (!offset) { + goto loop; } checks: search_start =stripe_align (root, offset);
fs/btrfs/extent-tree.c checks: · line 4091
Quoted 7 lines
fs/btrfs/extent-tree.c +1
-0@@ -4075,6 +4085,7 @@ checks:break ; loop: failed_cluster_refill = false; + failed_alloc = false;btrfs_put_block_group (block_group); }up_read (&space_info->groups_sem);
My box survived 6h of dbench with this patch whereas without it hangs within the first thwo minutes.
Johannes
Quoted 70 lines · Johannes Hirte
Am Montag 05 Oktober 2009 21:30:39 schrieb Josef Bacik:
Quoted 69 lines
Like the cluster allocating stuff, we can lockup the box with the normal allocation path. This happens when we
1) Start to cache a block group that is severely fragmented, but has a decent amount of free space.
2) Start to commit a transaction
3) Have the commit try and empty out some of the delalloc inodes with extents that are relatively large.The inodes will not be able to make the allocations because they will ask
for allocations larger than a contiguous area in the free space cache. So we will wait for more progress to be made on the block group, but since we're in a commit the caching kthread won't make any more progress and it already has enough free space thatwait_block_group_cache_progresswill just return. So, if we wait and fail to make the allocation the next time around, just loop and go to the next block group. This keeps us from getting stuck in a softlockup. Thanks,Signed-off-by Josef Bacik <[email protected]>--- fs/btrfs/extent-tree.c | 23 +++++++++++++++++------ 1 files changed, 17 insertions(+), 6 deletions(-)fs/btrfs/extent-tree.c +16
-6@@ -3816,6 +3816,7 @@ static noinline int find_free_extent(struct btrfs_trans_handle *trans, int loop = 0;bool found_uncached_bg = false;bool failed_cluster_refill = false; +bool failed_alloc = false;WARN_ON (num_bytes < root->sectorsize);btrfs_set_key_type (ins,BTRFS_EXTENT_ITEM_KEY );@@ -4020,14 +4021,23 @@ refill_cluster: offset =btrfs_find_space_for_alloc (block_group, search_start, num_bytes, empty_size); -if (!offset && (cached || (!cached && - loop ==LOOP_CACHING_NOWAIT ))) { - goto loop; - }else if (!offset && (!cached && - loop >LOOP_CACHING_NOWAIT )) { +/* +* If we didn't find a chunk, and we haven't failed on this +* block group before, and this block group is in the middle of +* caching and we are ok with waiting, then go ahead and wait +* for progress to be made, and set failed_alloc to true. +* +* If failed_alloc is true then we've already waited on this +* block group once and should move on to the next block group. +*/ +if (!offset && !failed_alloc && !cached && + loop >LOOP_CACHING_NOWAIT ) {wait_block_group_cache_progress (block_group, - num_bytes + empty_size); + num_bytes + empty_size); + failed_alloc = true; goto have_block_group; + }else if (!offset) { + goto loop; } checks: search_start =stripe_align (root, offset);@@ -4075,6 +4085,7 @@ checks:break ; loop: failed_cluster_refill = false;
Quoted 9 lines
fs/btrfs/extent-tree.c checks: · line 4091
Quoted 5 lines
fs/btrfs/extent-tree.c +1
-0+ failed_alloc = false;btrfs_put_block_group (block_group); }up_read (&space_info->groups_sem);My box survived 6h of dbench with this patch whereas without it hangs within the first thwo minutes.
Great, I'm glad it fixed it for you. Thanks for testing and reporting it.
Josef
Quoted 5 lines · Josef Bacik
Quoted 3 lines
My box survived 6h of dbench with this patch whereas without it hangs within the first thwo minutes.
Great, I'm glad it fixed it for you. Thanks for testing and reporting it.
Pushed out to the master branch. Thanks!
-chris