btrfs: Make balance cancelling response faster
v3 cover btrfs: Make balance cancelling response faster
[PROBLEM]
There are quite some users reporting that 'btrfs balance cancel' slow to cancel current running balance, or even doesn't work for certain dead balance loop.
With the following script showing how long it takes to fully stop a balance:
#!/bin/bash
dev=/dev/test/test
mnt=/mnt/btrfs
umount $mnt &> /dev/null
umount $dev &> /dev/null
mkfs.btrfs -f $dev
mount $dev -o nospace_cache $mnt
dd if=/dev/zero bs=1M of=$mnt/large &
dd_pid=$!
sleep 3
kill -KILL $dd_pid
sync
btrfs balance start --bg --full $mnt & sleep 1
echo "cancel request" >> /dev/kmsg
time btrfs balance cancel $mnt
umount $mnt
It takes around 7~10s to cancel the running balance in my test environment.
[CAUSE]
Btrfs uses btrfs_fs_info::balance_cancel_req to record how many cancel request are queued.
However that cancelling request is only checked after relocating a block group.
That behavior is far from optimal to provide a faster cancelling.
[FIX]
This patchset will add more cancelling check points, to make cancelling faster.
And also, introduce a new error injection points to cover these newly introduced and future check points.
For the canceled balance during relocate_block_group(), we are re-using the existing error handling path.
It will mark all existing reloc_roots as orphan in prepare_to_merge(), then queue all of them for cleanup in merge_reloc_roots(). Thus it shouldn't cause any problem.
Changelog:
v2:
- Rebased to v5.6-rc1
There is a small conflicts caused by extra finished stage output. Other than that, everything is pretty straightforward
- Add explanation for the error handling path in cover letter.
v3:
- Add comment explaining how cancelling/error handling is done in relocate_block_group()
- Remove one patch which is already covered by patch 2 and 3.
Qu Wenruo (3):
btrfs: relocation: Introduce error injection points for cancelling
balance
btrfs: relocation: Check cancel request after each data page read
btrfs: relocation: Check cancel request after each extent found
fs/btrfs/ctree.h | 1 + fs/btrfs/relocation.c | 27 +++++++++++++++++++++++++++ fs/btrfs/volumes.c | 2 +- 3 files changed, 29 insertions(+), 1 deletion(-)
Signature · 1 line
2.25.0
PATCH v3 1/3 btrfs: relocation: Introduce error injection points for cancelling balance
Introduce a new error injection point, should_cancel_balance().
It's just a wrapper of atomic_read(&fs_info->balance_cancel_req), but allows us to override the return value.
Currently there are only one locations using this function:
- btrfs_balance()
It checks cancel before each block group.
There are other locations checking fs_info->balance_cancel_req, but they are not used as an indicator to exit, so there is no need to use the wrapper.
But there will be more locations coming, and some locations can cause kernel panic if not handled properly.
So introduce this error injection to provide better test interface.
--- fs/btrfs/ctree.h | 1 + fs/btrfs/relocation.c | 11 +++++++++++ fs/btrfs/volumes.c | 2 +- 3 files changed, 13 insertions(+), 1 deletion(-)
fs/btrfs/ctree.h +1 -0
@@ -3401,6 +3401,7 @@ void btrfs_reloc_pre_snapshot(struct btrfs_pending_snapshot *pending, u64 *bytes_to_reserve);int btrfs_reloc_post_snapshot (struct btrfs_trans_handle *trans,struct btrfs_pending_snapshot *pending); +int should_cancel_balance (struct btrfs_fs_info *fs_info);/* scrub.c */ int btrfs_scrub_dev(struct btrfs_fs_info *fs_info, u64 devid, u64 start,
fs/btrfs/relocation.c +11 -0
@@ -9,6 +9,7 @@#include <linux/blkdev.h> #include <linux/rbtree.h> #include <linux/slab.h> +#include <linux/error-injection.h> #include "ctree.h" #include "disk-io.h" #include "transaction.h"
@@ -3264,6 +3265,16 @@ int setup_extent_mapping(struct inode *inode, u64 start, u64 end,return ret; } +/* +* This wrapper is to allow error injection to fully test all cancel +* call sites. +*/ +int should_cancel_balance (struct btrfs_fs_info *fs_info) +{ +return atomic_read (&fs_info->balance_cancel_req); +} +ALLOW_ERROR_INJECTION (should_cancel_balance, TRUE); +static int relocate_file_extent_cluster (struct inode *inode,struct file_extent_cluster *cluster) {
fs/btrfs/volumes.c +1 -1
@@ -3904,7 +3904,7 @@ int btrfs_balance(struct btrfs_fs_info *fs_info,if (btrfs_fs_closing (fs_info) ||atomic_read (&fs_info->balance_pause_req) || -atomic_read (&fs_info->balance_cancel_req)) { +should_cancel_balance (fs_info)) { ret = -EINVAL ; goto out; }
Signature · 1 line
2.25.0
Re: [PATCH v3 1/3] btrfs: relocation: Introduce error injection points for cancelling balance
Looks good,
PATCH v3 2/3 btrfs: relocation: Check cancel request after each data page read
In reply to Qu WenruoWhen relocating a data extents with large large data extents, we spend most of our time in relocate_file_extent_cluster() at stage "moving data extents":
1) | btrfs_relocate_block_group [btrfs]() {
1) | relocate_file_extent_cluster [btrfs]() {
1) $ 6586769 us | }
1) + 18.260 us | relocate_file_extent_cluster [btrfs]();
1) + 15.770 us | relocate_file_extent_cluster [btrfs]();
1) $ 8916340 us | }
1) | btrfs_relocate_block_group [btrfs]() {
1) | relocate_file_extent_cluster [btrfs]() {
1) $ 11611586 us | }
1) + 16.930 us | relocate_file_extent_cluster [btrfs]();
1) + 15.870 us | relocate_file_extent_cluster [btrfs]();
1) $ 14986130 us | }So to make data relocation cancelling quicker, here add extra balance cancelling check after each page read in relocate_file_extent_cluster().
Also add a comment explaining how the cancelling/error handling works.
--- fs/btrfs/relocation.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
fs/btrfs/relocation.c +12 -0
@@ -3396,6 +3396,10 @@ static int relocate_file_extent_cluster(struct inode *inode,btrfs_delalloc_release_extents (BTRFS_I (inode),PAGE_SIZE );balance_dirty_pages_ratelimited (inode->i_mapping);btrfs_throttle (fs_info); +if (should_cancel_balance (fs_info)) { + ret = -ECANCELED ; + goto out; + } }WARN_ON (nr != cluster->nr); out:
@@ -4208,6 +4212,14 @@ static noinline_for_stack int relocate_block_group(struct reloc_control *rc)backref_cache_cleanup (&rc->backref_cache);btrfs_block_rsv_release (fs_info, rc->block_rsv, (u64)-1 ); +/* +* Even when the relocation is cancelled, we should all go throught +* prepare_to_merge() and merge_reloc_roots(). +* +* For error (including cancelled balance), prepare_to_merge() will +* mark all reloc tree orphan, then queue them for cleanup in +* merge_reloc_roots() +*/ err =prepare_to_merge (rc, err);merge_reloc_roots (rc);
Signature · 1 line
2.25.0
Re: [PATCH v3 2/3] btrfs: relocation: Check cancel request after each data page read
Looks good,
PATCH v3 3/3 btrfs: relocation: Check cancel request after each extent found
In reply to Qu WenruoWhen relocating data block groups with tons of small extents, or large metadata block groups, there can be over 200,000 extents.
We will iterate all extents of such block group in relocate_block_group(), where iteration itself can be kinda time-consuming.
So when user want to cancel the balance, the extent iteration loop can be another target.
This patch will add the cancelling check in the extent iteration loop of relocate_block_group() to make balance cancelling faster.
--- fs/btrfs/relocation.c | 4 ++++ 1 file changed, 4 insertions(+)
fs/btrfs/relocation.c +4 -0
@@ -4181,6 +4181,10 @@ static noinline_for_stack int relocate_block_group(struct reloc_control *rc)break ; } } +if (should_cancel_balance (fs_info)) { + err = -ECANCELED ; +break ; + } }if (trans && progress && err == -ENOSPC ) { ret =btrfs_force_chunk_alloc (trans, rc->block_group->flags);
Signature · 1 line
2.25.0
Re: [PATCH v3 3/3] btrfs: relocation: Check cancel request after each extent found
Looks good,
On 17/02/2020 06:16, Qu Wenruo wrote:
...
Quoted 3 lines
[FIX]
This patchset will add more cancelling check points, to make cancelling faster.
I have a question on what this means for users of the balance ioctl.
I *think* that, today, there is a guarantee that if I issue the ioctl to start a balance, and then immediately (or, some time later) cancel it, I will be guaranteed that at least one block group will be balanced. In particular, if I repeat that behaviour, the balance will eventually complete.
Is that true?
If so, what happens to this guarantee with these changes?
I think, from your description, that the cancel can complete without even one block group being balanced. Is it guaranteed to have made progress on that bg so that if I repeat the behaviour that bg (and eventually all eligible bgs) will be balanced? Or is it now possible to cancel before any progress has been made sp that process never finishes?
If the latter, how long do I have to wait before cancelling to make sure that progress has been made? Is there any way to know whether progress has been made when the ioctl completes with the cancelled status?
This is a real question because I have some filesystems where balancing a single block group can sometimes take tens of minutes, and the system is impacted during that time. I already have my btrfs-balance-slowly script which allows me to control impact by not trying to balance the next bg for a while, so the system can recover and progress other tasks. And it would be great to be able to limit the impact further by cancelling during a single bg, but only if I can be sure that progress has been made and that by repeating the process I am guaranteed that it will eventually finish.
In any case, I suggest that the patch cover letter (and maybe code comments) explains what guarantees (if any) userspace is given.
Graham
Quoted 11 lines · Graham Cobb
On 17/02/2020 06:16, Qu Wenruo wrote:
...Quoted 3 lines
[FIX]
This patchset will add more cancelling check points, to make cancelling faster.I have a question on what this means for users of the balance ioctl.
I *think* that, today, there is a guarantee that if I issue the ioctl to start a balance, and then immediately (or, some time later) cancel it, I will be guaranteed that at least one block group will be balanced.
Nope, even for the original behavior.
The original check happens before we relocate one block group, so even for the old behavior, you can still cancel the balance before it even starts.
Quoted 14 lines
In
particular, if I repeat that behaviour, the balance will eventually complete.Is that true?
If so, what happens to this guarantee with these changes?
I think, from your description, that the cancel can complete without even one block group being balanced. Is it guaranteed to have made progress on that bg so that if I repeat the behaviour that bg (and eventually all eligible bgs) will be balanced? Or is it now possible to cancel before any progress has been made sp that process never finishes?
Quoted 3 lines
If the latter, how long do I have to wait before cancelling to make sure that progress has been made? Is there any way to know whether progress has been made when the ioctl completes with the cancelled status?
The ioctl itself already has the status recording how many block group has been relocated, and the bytenr of the last relocated block group. These behaviors haven't been changed by this patchset.
Thanks,
Qu
Quoted 16 lines
This is a real question because I have some filesystems where balancing a single block group can sometimes take tens of minutes, and the system is impacted during that time. I already have my btrfs-balance-slowly script which allows me to control impact by not trying to balance the next bg for a while, so the system can recover and progress other tasks. And it would be great to be able to limit the impact further by cancelling during a single bg, but only if I can be sure that progress has been made and that by repeating the process I am guaranteed that it will eventually finish.
In any case, I suggest that the patch cover letter (and maybe code comments) explains what guarantees (if any) userspace is given.
Graham
Quoted 11 lines · Qu Wenruo
[PROBLEM]
There are quite some users reporting that 'btrfs balance cancel' slow to cancel current running balance, or even doesn't work for certain dead balance loop.With the following script showing how long it takes to fully stop a balance:
[FIX]
This patchset will add more cancelling check points, to make cancelling faster.
Patches have been in for-next and also used to develop various fixes in the relocation code, so from that point I think it's ok to move them to misc-next. The benefit for usability is also clear, so thanks for working on that. As pointed elsewhere, the commandline UI will need to be extended to wire the cancelation of resize/deletion to make use of it, we have time for that for the next cycle.