clean up how we mark block groups read only
v3 cover clean up how we mark block groups read only
Original message below.
-------------------------------
Qu has been looking into ENOSPC during relocation and noticed some weirdness with inc_block_group_ro. The problem is this code hasn't changed to keep up with the rest of the reservation system. It still assumes that the amount of space used will always be less than the total space for the space info, which hasn't been true for years since I introduced overcommitting. This logic is correct for DATA, but not for METADATA or SYSTEM.
The first few patches are just cleanups, and can probably be taken as is. The final patch is the real meat of the problem to address Qu's issues. This is an RFC because I'm elbow deep in another problem and haven't tested this beyond compile testing, but I think it'll work. Once I've gotten a chance to test it and Qu has tested it I'll update the list if it's good to go as is, or send a V2 with any changes. Thanks,
Josef
PATCH v3 1/5 btrfs: check rw_devices, not num_devices for restriping
While running xfstests with compression on I noticed I was panicing on btrfs/154. I bisected this down to my inc_block_group_ro patches, which was strange.
What was happening is with my patches we now use btrfs_can_overcommit() to see if we can flip a block group read only. Before this would fail because we weren't taking into account the usable un-allocated space for allocating chunks. With my patches we were allowed to do the balance, which is technically correct.
However this test is testing restriping with a degraded mount, something that isn't working right because Anand's fix for the test was never actually merged.
So now we're trying to allocate a chunk and cannot because we want to allocate a RAID1 chunk, but there's only 1 device that's available for usage. This results in an ENOSPC in one of the BUG_ON(ret) paths in relocation (and a tricky path that is going to take many more patches to fix.)
But we shouldn't even be making it this far, we don't have enough devices to restripe. The problem is we're using btrfs_num_devices(), which for some reason includes missing devices. That's not actually what we want, we want the rw_devices.
Fix this by getting the rw_devices. With this patch we're no longer panicing with my other patches applied, and we're in fact erroring out at the correct spot instead of at inc_block_group_ro. The fact that this was working before was just sheer dumb luck.
--- fs/btrfs/volumes.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
fs/btrfs/volumes.c +8 -1
@@ -3881,7 +3881,14 @@ int btrfs_balance(struct btrfs_fs_info *fs_info, } } - num_devices =btrfs_num_devices (fs_info); +/* +* rw_devices can be messed with by rm_device and device replace, so +* take the chunk_mutex to make sure we have a relatively consistent +* view of the fs at this point. +*/ +mutex_lock (&fs_info->chunk_mutex); + num_devices = fs_info->fs_devices->rw_devices; +mutex_unlock (&fs_info->chunk_mutex);/* * SINGLE profile on-disk has no profile bit, but in-memory we have a
Signature · 1 line
2.24.1
Re: [PATCH 1/5] btrfs: check rw_devices, not num_devices for restriping
Quoted 43 lines · Josef Bacik
While running xfstests with compression on I noticed I was panicing on btrfs/154. I bisected this down to my
inc_block_group_ropatches, which was strange.What was happening is with my patches we now use
btrfs_can_overcommit()to see if we can flip a block group read only. Before this would fail because we weren't taking into account the usable un-allocated space for allocating chunks. With my patches we were allowed to do the balance, which is technically correct.However this test is testing restriping with a degraded mount, something that isn't working right because Anand's fix for the test was never actually merged.
So now we're trying to allocate a chunk and cannot because we want to allocate a RAID1 chunk, but there's only 1 device that's available for usage. This results in an ENOSPC in one of the
BUG_ON(ret)paths in relocation (and a tricky path that is going to take many more patches to fix.)But we shouldn't even be making it this far, we don't have enough devices to restripe. The problem is we're using
btrfs_num_devices(), which for some reason includes missing devices. That's not actually what we want, we want therw_devices.Fix this by getting the
rw_devices. With this patch we're no longer panicing with my other patches applied, and we're in fact erroring out at the correct spot instead of atinc_block_group_ro. The fact that this was working before was just sheer dumb luck.Fixes e4d8ec0f65b9 ("Btrfs: implement online profile changing")Signed-off-by Josef Bacik <[email protected]>--- fs/btrfs/volumes.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)fs/btrfs/volumes.c +0
-0@@ -3881,7 +3881,14 @@ int btrfs_balance(struct btrfs_fs_info *fs_info, } }
fs/btrfs/volumes.c int btrfs_balance(struct btrfs_fs_info *fs_info, · line 3891
Quoted 10 lines
fs/btrfs/volumes.c +8
-1- num_devices =btrfs_num_devices (fs_info); +/* +* rw_devices can be messed with by rm_device and device replace, so +* take the chunk_mutex to make sure we have a relatively consistent +* view of the fs at this point. +*/ +mutex_lock (&fs_info->chunk_mutex); + num_devices = fs_info->fs_devices->rw_devices; +mutex_unlock (&fs_info->chunk_mutex);
chunk_mutex is the correct lock for rw_devices counter and alloc_list. So,
Thanks,
Qu
Quoted 4 lines
fs/btrfs/volumes.c +0
-0/* * SINGLE profile on-disk has no profile bit, but in-memory we have a
Re: [PATCH 1/5] btrfs: check rw_devices, not num_devices for restriping
In reply to Josef BacikQuoted 3 lines · Josef Bacik
While running xfstests with compression on I noticed I was panicing on btrfs/154. I bisected this down to my
inc_block_group_ropatches, which was strange.
Do you have stacktrace of the panic?
Quoted 5 lines
What was happening is with my patches we now use
btrfs_can_overcommit()to see if we can flip a block group read only. Before this would fail because we weren't taking into account the usable un-allocated space for allocating chunks. With my patches we were allowed to do the balance, which is technically correct.
What patches does "my patches" mean?
Quoted 3 lines
However this test is testing restriping with a degraded mount, something that isn't working right because Anand's fix for the test was never actually merged.
Which patch is that?
Quoted 10 lines
So now we're trying to allocate a chunk and cannot because we want to allocate a RAID1 chunk, but there's only 1 device that's available for usage. This results in an ENOSPC in one of the
BUG_ON(ret)paths in relocation (and a tricky path that is going to take many more patches to fix.)But we shouldn't even be making it this far, we don't have enough devices to restripe. The problem is we're using
btrfs_num_devices(), which for some reason includes missing devices. That's not actually what we want, we want therw_devices.
The wrapper btrfs_num_devices takes into account an ongoing replace that temporarily increases num_devices, so the result returned to balance is adjusted.
That we need to know the correct number of writable devices at this point is right. With btrfs_num_devices we'd have to subtract missing devices, but in the end we can't use more than rw_devices.
Quoted 16 lines
Fix this by getting the
rw_devices. With this patch we're no longer panicing with my other patches applied, and we're in fact erroring out at the correct spot instead of atinc_block_group_ro. The fact that this was working before was just sheer dumb luck.Fixes e4d8ec0f65b9 ("Btrfs: implement online profile changing")Signed-off-by Josef Bacik <[email protected]>--- fs/btrfs/volumes.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)fs/btrfs/volumes.c +0
-0
fs/btrfs/volumes.c int btrfs_balance(struct btrfs_fs_info *fs_info, · line 3887
Quoted 8 lines
fs/btrfs/volumes.c +4
-1@@ -3881,7 +3881,14 @@ int btrfs_balance(struct btrfs_fs_info *fs_info, } } - num_devices =btrfs_num_devices (fs_info); +/* +* rw_devices can be messed with by rm_device and device replace, so +* take the chunk_mutex to make sure we have a relatively consistent +* view of the fs at this point.
Well, what does 'relatively consistent' mean here? There are enough locks and exclusion that device remove or replace should not change the value until btrfs_balance ends, no?
Quoted 9 lines
fs/btrfs/volumes.c +4
-0+*/ +mutex_lock (&fs_info->chunk_mutex); + num_devices = fs_info->fs_devices->rw_devices; +mutex_unlock (&fs_info->chunk_mutex);/* * SINGLE profile on-disk has no profile bit, but in-memory we have a Signature · 1 line
2.24.1
Re: [PATCH 1/5] btrfs: check rw_devices, not num_devices for restriping
Quoted 7 lines · David Sterba
Quoted 3 lines · Josef Bacik
While running xfstests with compression on I noticed I was panicing on btrfs/154. I bisected this down to my
inc_block_group_ropatches, which was strange.Do you have stacktrace of the panic?
I don't have it with me, I can reproduce when I get back. But it's a BUG_ON(ret) in init_reloc_root when we do the copy_root, because we get an ENOSPC when trying to allocate the tree block.
Quoted 8 lines
Quoted 5 lines
What was happening is with my patches we now use
btrfs_can_overcommit()to see if we can flip a block group read only. Before this would fail because we weren't taking into account the usable un-allocated space for allocating chunks. With my patches we were allowed to do the balance, which is technically correct.What patches does "my patches" mean?
The ones that convert the inc_block_group_ro() to use btrfs_can_overcommit().
Quoted 5 lines
Quoted 3 lines
However this test is testing restriping with a degraded mount, something that isn't working right because Anand's fix for the test was never actually merged.
Which patch is that?
It says in the header of btrfs/154. I don't have xfstests in front of me right now.
Quoted 39 lines
Quoted 11 lines
So now we're trying to allocate a chunk and cannot because we want to allocate a RAID1 chunk, but there's only 1 device that's available for usage. This results in an ENOSPC in one of the
BUG_ON(ret)paths in relocation (and a tricky path that is going to take many more patches to fix.)But we shouldn't even be making it this far, we don't have enough devices to restripe. The problem is we're using
btrfs_num_devices(), which for some reason includes missing devices. That's not actually what we want, we want therw_devices.The wrapper
btrfs_num_devicestakes into account an ongoing replace that temporarily increasesnum_devices, so the result returned to balance is adjusted.That we need to know the correct number of writable devices at this point is right. With
btrfs_num_deviceswe'd have to subtract missing devices, but in the end we can't use more thanrw_devices.Quoted 19 lines
Fix this by getting the
rw_devices. With this patch we're no longer panicing with my other patches applied, and we're in fact erroring out at the correct spot instead of atinc_block_group_ro. The fact that this was working before was just sheer dumb luck.Fixes e4d8ec0f65b9 ("Btrfs: implement online profile changing")Signed-off-by Josef Bacik <[email protected]>--- fs/btrfs/volumes.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)fs/btrfs/volumes.c +0
-0@@ -3881,7 +3881,14 @@ int btrfs_balance(struct btrfs_fs_info *fs_info, } }
Quoted 10 lines
fs/btrfs/volumes.c int btrfs_balance(struct btrfs_fs_info *fs_info, · line 3887
Quoted 5 lines
fs/btrfs/volumes.c +4
-1- num_devices =btrfs_num_devices (fs_info); +/* +* rw_devices can be messed with by rm_device and device replace, so +* take the chunk_mutex to make sure we have a relatively consistent +* view of the fs at this point. Well, what does 'relatively consistent' mean here? There are enough locks and exclusion that device remove or replace should not change the value until
btrfs_balanceends, no?
Again I don't have the code in front of me, but there's nothing at this point to stop us from running in at the tail end of device replace or device rm. The mutex keeps us from getting weirdly inflated values when we increment and decrement at the end of device replace, but there's nothing (that I can remember) that will stop rw devices from changing right after we check it, thus relatively. Thanks,
Josef
Re: [PATCH 1/5] btrfs: check rw_devices, not num_devices for restriping
Quoted 11 lines · Josef Bacik
Quoted 10 lines
fs/btrfs/volumes.c int btrfs_balance(struct btrfs_fs_info *fs_info, · line 3887
Quoted 5 lines
fs/btrfs/volumes.c +4
-1- num_devices =btrfs_num_devices (fs_info); +/* +* rw_devices can be messed with by rm_device and device replace, so +* take the chunk_mutex to make sure we have a relatively consistent +* view of the fs at this point. Well, what does 'relatively consistent' mean here? There are enough locks and exclusion that device remove or replace should not change the value until
btrfs_balanceends, no?
Quoted 2 lines
Again I don't have the code in front of me, but there's nothing at this point to stop us from running in at the tail end of device replace or device rm.
This should be prevented by the EXCL_OP mechanism, so even the end of device remove or replace will not be running at this time because it cannot even start.
Quoted 5 lines
The
mutex keeps us from getting weirdly inflated values when we increment and decrement at the end of device replace, but there's nothing (that I can remember) that will stop rw devices from changing right after we check it, thus relatively.
rw_devices is changed in a handful of places on a mounted filesystem, not counting device open/close. Device remove and replace are excluded from running at that time, rw_devices can't change at this point of balance.
btrfs_dev_replace_finishing
- when removing srcdev, rw_devices--
- when adding the target device as new, rw_devices++
btrfs_rm_device
- rw_devices--
btrfs_init_new_device (called by device add)
- rw_devices++
So the chunk mutex is either redundant or there's something I'm missing.
Re: [PATCH 1/5] btrfs: check rw_devices, not num_devices for restriping
Quoted 34 lines · David Sterba
Quoted 11 lines · Josef Bacik
Quoted 10 lines
fs/btrfs/volumes.c int btrfs_balance(struct btrfs_fs_info *fs_info, · line 3887
Quoted 5 lines
fs/btrfs/volumes.c +4
-1- num_devices =btrfs_num_devices (fs_info); +/* +* rw_devices can be messed with by rm_device and device replace, so +* take the chunk_mutex to make sure we have a relatively consistent +* view of the fs at this point. Well, what does 'relatively consistent' mean here? There are enough locks and exclusion that device remove or replace should not change the value until
btrfs_balanceends, no?Quoted 2 lines
Again I don't have the code in front of me, but there's nothing at this point to stop us from running in at the tail end of device replace or device rm.
This should be prevented by the
EXCL_OPmechanism, so even the end of device remove or replace will not be running at this time because it cannot even start.Quoted 5 lines
The
mutex keeps us from getting weirdly inflated values when we increment and decrement at the end of device replace, but there's nothing (that I can remember) that will stop rw devices from changing right after we check it, thus relatively.
rw_devicesis changed in a handful of places on a mounted filesystem, not counting device open/close. Device remove and replace are excluded from running at that time,rw_devicescan't change at this point of balance.
btrfs_dev_replace_finishing
- when removing srcdev, rw_devices--
- when adding the target device as new, rw_devices++
Quoted 8 lines
btrfs_rm_device
- rw_devices--
btrfs_init_new_device(called by device add)
- rw_devices++So the chunk mutex is either redundant or there's something I'm missing.
Nope you're right, I missed the EXCL_OP thing, so we can just read rw_devices normally. Thanks,
Josef
PATCH v3 2/5 btrfs: don't pass system_chunk into can_overcommit
In reply to Josef BacikWe have the space_info, we can just check its flags to see if it's the system chunk space info.
--- fs/btrfs/space-info.c | 41 +++++++++++++++-------------------------- 1 file changed, 15 insertions(+), 26 deletions(-)
fs/btrfs/space-info.c +15 -26
@@ -161,8 +161,7 @@ static inline u64 calc_global_rsv_need_space(struct btrfs_block_rsv *global)static int can_overcommit (struct btrfs_fs_info *fs_info,struct btrfs_space_info *space_info,u64 bytes, -enum btrfs_reserve_flush_enum flush, -bool system_chunk) +enum btrfs_reserve_flush_enum flush) {u64 profile;u64 avail;
@@ -173,7 +172,7 @@ static int can_overcommit(struct btrfs_fs_info *fs_info,if (space_info->flags &BTRFS_BLOCK_GROUP_DATA )return 0 ; -if (system_chunk) +if (space_info->flags &BTRFS_BLOCK_GROUP_SYSTEM ) profile =btrfs_system_alloc_profile (fs_info);else profile =btrfs_metadata_alloc_profile (fs_info);
@@ -227,8 +226,7 @@ void btrfs_try_granting_tickets(struct btrfs_fs_info *fs_info,/* Check and see if our ticket can be satisified now. */ if ((used + ticket->bytes <= space_info->total_bytes) || -can_overcommit (fs_info, space_info, ticket->bytes, flush, - false)) { +can_overcommit (fs_info, space_info, ticket->bytes, flush)) {btrfs_space_info_update_bytes_may_use (fs_info, space_info, ticket->bytes);
@@ -626,8 +624,7 @@ static void flush_space(struct btrfs_fs_info *fs_info,static inline u64 btrfs_calc_reclaim_metadata_size (struct btrfs_fs_info *fs_info, -struct btrfs_space_info *space_info, -bool system_chunk) +struct btrfs_space_info *space_info) {struct reserve_ticket *ticket;u64 used;
@@ -643,13 +640,13 @@ btrfs_calc_reclaim_metadata_size(struct btrfs_fs_info *fs_info, to_reclaim =min_t (u64,num_online_cpus () *SZ_1M ,SZ_16M );if (can_overcommit (fs_info, space_info, to_reclaim, -BTRFS_RESERVE_FLUSH_ALL , system_chunk)) +BTRFS_RESERVE_FLUSH_ALL ))return 0 ; used =btrfs_space_info_used (space_info, true);if (can_overcommit (fs_info, space_info,SZ_1M , -BTRFS_RESERVE_FLUSH_ALL , system_chunk)) +BTRFS_RESERVE_FLUSH_ALL )) expected =div_factor_fine (space_info->total_bytes,95 );else expected =div_factor_fine (space_info->total_bytes,90 );
@@ -665,7 +662,7 @@ btrfs_calc_reclaim_metadata_size(struct btrfs_fs_info *fs_info,static inline int need_do_async_reclaim (struct btrfs_fs_info *fs_info,struct btrfs_space_info *space_info, -u64 used,bool system_chunk) +u64 used) {u64 thresh =div_factor_fine (space_info->total_bytes,98 );
@@ -673,8 +670,7 @@ static inline int need_do_async_reclaim(struct btrfs_fs_info *fs_info,if ((space_info->bytes_used + space_info->bytes_reserved) >= thresh)return 0 ; -if (!btrfs_calc_reclaim_metadata_size (fs_info, space_info, - system_chunk)) +if (!btrfs_calc_reclaim_metadata_size (fs_info, space_info))return 0 ; return (used >= thresh && !btrfs_fs_closing(fs_info) &&
@@ -765,8 +761,7 @@ static void btrfs_async_reclaim_metadata_space(struct work_struct *work) space_info =btrfs_find_space_info (fs_info,BTRFS_BLOCK_GROUP_METADATA );spin_lock (&space_info->lock); - to_reclaim =btrfs_calc_reclaim_metadata_size (fs_info, space_info, - false); + to_reclaim =btrfs_calc_reclaim_metadata_size (fs_info, space_info);if (!to_reclaim) { space_info->flush =0 ;spin_unlock (&space_info->lock);
@@ -785,8 +780,7 @@ static void btrfs_async_reclaim_metadata_space(struct work_struct *work)return ; } to_reclaim =btrfs_calc_reclaim_metadata_size (fs_info, - space_info, - false); + space_info);if (last_tickets_id == space_info->tickets_id) { flush_state++; }else {
@@ -858,8 +852,7 @@ static void priority_reclaim_metadata_space(struct btrfs_fs_info *fs_info,int flush_state;spin_lock (&space_info->lock); - to_reclaim =btrfs_calc_reclaim_metadata_size (fs_info, space_info, - false); + to_reclaim =btrfs_calc_reclaim_metadata_size (fs_info, space_info);if (!to_reclaim) {spin_unlock (&space_info->lock);return ;
@@ -990,8 +983,7 @@ static int handle_reserve_ticket(struct btrfs_fs_info *fs_info,static int __reserve_metadata_bytes (struct btrfs_fs_info *fs_info,struct btrfs_space_info *space_info,u64 orig_bytes, -enum btrfs_reserve_flush_enum flush, -bool system_chunk) +enum btrfs_reserve_flush_enum flush) {struct reserve_ticket ticket;u64 used;
@@ -1013,8 +1005,7 @@ static int __reserve_metadata_bytes(struct btrfs_fs_info *fs_info,*/ if (!pending_tickets && ((used + orig_bytes <= space_info->total_bytes) || -can_overcommit (fs_info, space_info, orig_bytes, flush, - system_chunk))) { +can_overcommit (fs_info, space_info, orig_bytes, flush))) {btrfs_space_info_update_bytes_may_use (fs_info, space_info, orig_bytes); ret =0 ;
@@ -1054,8 +1045,7 @@ static int __reserve_metadata_bytes(struct btrfs_fs_info *fs_info,* the async reclaim as we will panic. */ if (!test_bit (BTRFS_FS_LOG_RECOVERING , &fs_info->flags) && -need_do_async_reclaim (fs_info, space_info, - used, system_chunk) && +need_do_async_reclaim (fs_info, space_info, used) && !work_busy (&fs_info->async_reclaim_work)) {trace_btrfs_trigger_flush (fs_info, space_info->flags, orig_bytes, flush,"preempt" );
@@ -1092,10 +1082,9 @@ int btrfs_reserve_metadata_bytes(struct btrfs_root *root,struct btrfs_fs_info *fs_info = root->fs_info;struct btrfs_block_rsv *global_rsv = &fs_info->global_block_rsv;int ret; -bool system_chunk = (root == fs_info->chunk_root); ret =__reserve_metadata_bytes (fs_info, block_rsv->space_info, - orig_bytes, flush, system_chunk); + orig_bytes, flush);if (ret == -ENOSPC &&unlikely (root->orphan_cleanup_state ==ORPHAN_CLEANUP_STARTED )) { if (block_rsv != global_rsv &&
Signature · 1 line
2.24.1
Re: [PATCH 2/5] btrfs: don't pass system_chunk into can_overcommit
Quoted 7 lines · Josef Bacik
We have the
space_info, we can just check its flags to see if it's the system chunk space info.Signed-off-by Josef Bacik <[email protected]>Reviewed-by Nikolay Borisov <[email protected]>Reviewed-by Qu Wenruo <[email protected]>Reviewed-by Johannes Thumshirn <[email protected]>
Huh this patch and 3/5 have been in misc-next for more than a month, are we that much out of sync?
PATCH v3 3/5 btrfs: kill min_allocable_bytes in inc_block_group_ro
In reply to Josef BacikThis is a relic from a time before we had a proper reservation mechanism and you could end up with really full chunks at chunk allocation time. This doesn't make sense anymore, so just kill it.
--- fs/btrfs/block-group.c | 22 ++++------------------ 1 file changed, 4 insertions(+), 18 deletions(-)
fs/btrfs/block-group.c +4 -18
@@ -1185,21 +1185,8 @@ static int inc_block_group_ro(struct btrfs_block_group *cache, int force)struct btrfs_space_info *sinfo = cache->space_info;u64 num_bytes;u64 sinfo_used; -u64 min_allocable_bytes;int ret = -ENOSPC ; -/* -* We need some metadata space and system metadata space for -* allocating chunks in some corner cases until we force to set -* it to be readonly. -*/ -if ((sinfo->flags & - (BTRFS_BLOCK_GROUP_SYSTEM |BTRFS_BLOCK_GROUP_METADATA )) && - !force) - min_allocable_bytes =SZ_1M ; -else - min_allocable_bytes =0 ; -spin_lock (&sinfo->lock);spin_lock (&cache->lock);
@@ -1217,10 +1204,9 @@ static int inc_block_group_ro(struct btrfs_block_group *cache, int force)* sinfo_used + num_bytes should always <= sinfo->total_bytes. * * Here we make sure if we mark this bg RO, we still have enough -* free space as buffer (if min_allocable_bytes is not 0). +* free space as buffer. */ -if (sinfo_used + num_bytes + min_allocable_bytes <= - sinfo->total_bytes) { +if (sinfo_used + num_bytes <= sinfo->total_bytes) { sinfo->bytes_readonly += num_bytes; cache->ro++;list_add_tail (&cache->ro_list, &sinfo->ro_bgs);
@@ -1233,8 +1219,8 @@ static int inc_block_group_ro(struct btrfs_block_group *cache, int force)btrfs_info (cache->fs_info,"unable to make block group %llu ro" , cache->start);btrfs_info (cache->fs_info, -"sinfo_used=%llu bg_num_bytes=%llu min_allocable=%llu" , - sinfo_used, num_bytes, min_allocable_bytes); +"sinfo_used=%llu bg_num_bytes=%llu" , + sinfo_used, num_bytes);btrfs_dump_space_info (cache->fs_info, cache->space_info,0 ,0 ); }return ret;
Signature · 1 line
2.24.1
PATCH v3 4/5 btrfs: fix force usage in inc_block_group_ro
In reply to Josef BacikFor some reason we've translated the do_chunk_alloc that goes into btrfs_inc_block_group_ro to force in inc_block_group_ro, but these are two different things.
force for inc_block_group_ro is used when we are forcing the block group read only no matter what, for example when the underlying chunk is marked read only. We need to not do the space check here as this block group needs to be read only.
btrfs_inc_block_group_ro() has a do_chunk_alloc flag that indicates that we need to pre-allocate a chunk before marking the block group read only. This has nothing to do with forcing, and in fact we _always_ want to do the space check in this case, so unconditionally pass false for force in this case.
Then fixup inc_block_group_ro to honor force as it's expected and documented to do.
--- fs/btrfs/block-group.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
fs/btrfs/block-group.c +8 -1
@@ -1190,8 +1190,15 @@ static int inc_block_group_ro(struct btrfs_block_group *cache, int force)spin_lock (&sinfo->lock);spin_lock (&cache->lock); -if (cache->ro) { +if (cache->ro || force) { cache->ro++; + +/* +* We should only be empty if we did force here and haven't +* already marked ourselves read only. +*/ +if (force &&list_empty (&cache->ro_list)) +list_add_tail (&cache->ro_list, &sinfo->ro_bgs); ret =0 ; goto out; }
Signature · 1 line
2.24.1
Re: [PATCH 4/5] btrfs: fix force usage in inc_block_group_ro
Quoted 15 lines · Josef Bacik
For some reason we've translated the
do_chunk_allocthat goes intobtrfs_inc_block_group_roto force ininc_block_group_ro, but these are two different things.force for
inc_block_group_rois used when we are forcing the block group read only no matter what, for example when the underlying chunk is marked read only. We need to not do the space check here as this block group needs to be read only.
btrfs_inc_block_group_ro()has ado_chunk_allocflag that indicates that we need to pre-allocate a chunk before marking the block group read only. This has nothing to do with forcing, and in fact we _always_ want to do the space check in this case, so unconditionally pass false for force in this case.
Quoted 5 lines
Then fixup
inc_block_group_roto honor force as it's expected and documented to do.Signed-off-by Josef Bacik <[email protected]>Reviewed-by Nikolay Borisov <[email protected]>
It looks like my previous comment was on a development branch which we skip chunk allocation for scrub.
But since it's not upstreamed yet, no need to bother.
Thanks,
Qu
Quoted 26 lines
--- fs/btrfs/block-group.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)fs/btrfs/block-group.c +8
-1@@ -1190,8 +1190,15 @@ static int inc_block_group_ro(struct btrfs_block_group *cache, int force)spin_lock (&sinfo->lock);spin_lock (&cache->lock); -if (cache->ro) { +if (cache->ro || force) { cache->ro++; + +/* +* We should only be empty if we did force here and haven't +* already marked ourselves read only. +*/ +if (force &&list_empty (&cache->ro_list)) +list_add_tail (&cache->ro_list, &sinfo->ro_bgs); ret =0 ; goto out; }
PATCH v3 5/5 btrfs: use btrfs_can_overcommit in inc_block_group_ro
In reply to Josef Bacikinc_block_group_ro does a calculation to see if we have enough room left over if we mark this block group as read only in order to see if it's ok to mark the block group as read only.
The problem is this calculation _only_ works for data, where our used is always less than our total. For metadata we will overcommit, so this will almost always fail for metadata.
Fix this by exporting btrfs_can_overcommit, and then see if we have enough space to remove the remaining free space in the block group we are trying to mark read only. If we do then we can mark this block group as read only.
--- fs/btrfs/block-group.c | 35 ++++++++++++++++++++++++----------- fs/btrfs/space-info.c | 19 ++++++++++--------- fs/btrfs/space-info.h | 3 +++ 3 files changed, 37 insertions(+), 20 deletions(-)
fs/btrfs/block-group.c +24 -11
@@ -1184,7 +1184,6 @@ static int inc_block_group_ro(struct btrfs_block_group *cache, int force) {struct btrfs_space_info *sinfo = cache->space_info;u64 num_bytes; -u64 sinfo_used;int ret = -ENOSPC ;spin_lock (&sinfo->lock);
@@ -1205,19 +1204,36 @@ static int inc_block_group_ro(struct btrfs_block_group *cache, int force) num_bytes = cache->length - cache->reserved - cache->pinned - cache->bytes_super - cache->used; - sinfo_used =btrfs_space_info_used (sinfo, true);/* -* sinfo_used + num_bytes should always <= sinfo->total_bytes. -* -* Here we make sure if we mark this bg RO, we still have enough -* free space as buffer. +* Data never overcommits, even in mixed mode, so do just the straight +* check of left over space in how much we have allocated. */ -if (sinfo_used + num_bytes <= sinfo->total_bytes) { +if (sinfo->flags &BTRFS_BLOCK_GROUP_DATA ) { +u64 sinfo_used =btrfs_space_info_used (sinfo, true); + +/* +* Here we make sure if we mark this bg RO, we still have enough +* free space as buffer. +*/ +if (sinfo_used + num_bytes <= sinfo->total_bytes) + ret =0 ; + }else { +/* +* We overcommit metadata, so we need to do the +* btrfs_can_overcommit check here, and we need to pass in +* BTRFS_RESERVE_NO_FLUSH to give ourselves the most amount of +* leeway to allow us to mark this block group as read only. +*/ +if (btrfs_can_overcommit (cache->fs_info, sinfo, num_bytes, +BTRFS_RESERVE_NO_FLUSH )) + ret =0 ; + } + +if (!ret) { sinfo->bytes_readonly += num_bytes; cache->ro++;list_add_tail (&cache->ro_list, &sinfo->ro_bgs); - ret =0 ; } out:spin_unlock (&cache->lock);
@@ -1225,9 +1241,6 @@ static int inc_block_group_ro(struct btrfs_block_group *cache, int force)if (ret == -ENOSPC &&btrfs_test_opt (cache->fs_info,ENOSPC_DEBUG )) {btrfs_info (cache->fs_info,"unable to make block group %llu ro" , cache->start); -btrfs_info (cache->fs_info, -"sinfo_used=%llu bg_num_bytes=%llu" , - sinfo_used, num_bytes);btrfs_dump_space_info (cache->fs_info, cache->space_info,0 ,0 ); }return ret;
fs/btrfs/space-info.c +10 -9
@@ -159,9 +159,9 @@ static inline u64 calc_global_rsv_need_space(struct btrfs_block_rsv *global)return (global->size <<1 ); } -static int can_overcommit (struct btrfs_fs_info *fs_info, -struct btrfs_space_info *space_info,u64 bytes, -enum btrfs_reserve_flush_enum flush) +int btrfs_can_overcommit (struct btrfs_fs_info *fs_info, +struct btrfs_space_info *space_info,u64 bytes, +enum btrfs_reserve_flush_enum flush) {u64 profile;u64 avail;
@@ -226,7 +226,8 @@ void btrfs_try_granting_tickets(struct btrfs_fs_info *fs_info,/* Check and see if our ticket can be satisified now. */ if ((used + ticket->bytes <= space_info->total_bytes) || -can_overcommit (fs_info, space_info, ticket->bytes, flush)) { +btrfs_can_overcommit (fs_info, space_info, ticket->bytes, + flush)) {btrfs_space_info_update_bytes_may_use (fs_info, space_info, ticket->bytes);
@@ -639,14 +640,14 @@ btrfs_calc_reclaim_metadata_size(struct btrfs_fs_info *fs_info,return to_reclaim; to_reclaim =min_t (u64,num_online_cpus () *SZ_1M ,SZ_16M ); -if (can_overcommit (fs_info, space_info, to_reclaim, -BTRFS_RESERVE_FLUSH_ALL )) +if (btrfs_can_overcommit (fs_info, space_info, to_reclaim, +BTRFS_RESERVE_FLUSH_ALL ))return 0 ; used =btrfs_space_info_used (space_info, true); -if (can_overcommit (fs_info, space_info,SZ_1M , -BTRFS_RESERVE_FLUSH_ALL )) +if (btrfs_can_overcommit (fs_info, space_info,SZ_1M , +BTRFS_RESERVE_FLUSH_ALL )) expected =div_factor_fine (space_info->total_bytes,95 );else expected =div_factor_fine (space_info->total_bytes,90 );
@@ -1005,7 +1006,7 @@ static int __reserve_metadata_bytes(struct btrfs_fs_info *fs_info,*/ if (!pending_tickets && ((used + orig_bytes <= space_info->total_bytes) || -can_overcommit (fs_info, space_info, orig_bytes, flush))) { +btrfs_can_overcommit (fs_info, space_info, orig_bytes, flush))) {btrfs_space_info_update_bytes_may_use (fs_info, space_info, orig_bytes); ret =0 ;
fs/btrfs/space-info.h +3 -0
@@ -127,6 +127,9 @@ int btrfs_reserve_metadata_bytes(struct btrfs_root *root, enum btrfs_reserve_flush_enum flush);void btrfs_try_granting_tickets (struct btrfs_fs_info *fs_info,struct btrfs_space_info *space_info); +int btrfs_can_overcommit (struct btrfs_fs_info *fs_info, +struct btrfs_space_info *space_info,u64 bytes, +enum btrfs_reserve_flush_enum flush); static inline void btrfs_space_info_free_bytes_may_use( struct btrfs_fs_info *fs_info,
Signature · 1 line
2.24.1