btrfs-progs: receive: introduce new --clone-fallback option
Subsystems (2)
v3 cover btrfs-progs: receive: introduce new --clone-fallback option
When parent stream and incremental stream are received with different nodatasum mount options, any clone opeartion in the incremental stream will be rejected by kernel.
There are more situations to cause clone failure, like receiving a stream on a fs with different sectorsize.
Thus this patchset will introduce a new option, --clone-fallback, for btrfs receive to fall back to buffered write when clone failed.
This fall back behavior will only happen if the new option is explicitly specified, as such behavior can hide some send bugs, and under most sane cases users don't need such option.
Also add a test case for the new option.
Changelog:RFC->v1:
- Introduce a new option for the fallback behavior To avoid hide send bugs.
- Hide the warning message behind -v option
Since we have a special option for it thus users are aware of what they are doing, there is no need to output such warning by default.
- Add a new test case for it
v2:
- Add the missing help string for --clone-fallback option
- Rephrase the words in comments and commit messages
- Add run_check_remount_test_dev() helper
v3:
- Add an overview of failed clone operations for --clone-fallback option
- Update doc words again
Qu Wenruo (2):
btrfs-progs: receive: fallback to buffered copy if clone failed
btrfs-progs: misc-tests: add test case for receive --clone-fallback
Documentation/btrfs-receive.asciidoc | 13 +++ cmds/receive.c | 80 ++++++++++++++++++- tests/common | 9 +++ .../049-receive-clone-fallback/test.sh | 58 ++++++++++++++ 4 files changed, 157 insertions(+), 3 deletions(-) create mode 100755 tests/misc-tests/049-receive-clone-fallback/test.sh
Signature · 1 line
2.33.0
PATCH v3 1/2 btrfs-progs: receive: fallback to buffered copy if clone failed
[BUG]
There are two very basic send streams:
(a/m/ctime and uuid omitted)
Stream 1: (Parent subvolume)
subvol ./parent_subv transid=8 chown ./parent_subv/ gid=0 uid=0 chmod ./parent_subv/ mode=755 utimes ./parent_subv/ mkfile ./parent_subv/o257-7-0 rename ./parent_subv/o257-7-0 dest=./parent_subv/source utimes ./parent_subv/ write ./parent_subv/source offset=0 len=16384 chown ./parent_subv/source gid=0 uid=0 chmod ./parent_subv/source mode=600 utimes ./parent_subv/source
Stream 2: (snapshot and clone)
snapshot ./dest_subv transid=14 parent_transid=10
utimes ./dest_subv/ mkfile ./dest_subv/o258-14-0 rename ./dest_subv/o258-14-0 dest=./dest_subv/reflink utimes ./dest_subv/ clone ./dest_subv/reflink offset=0 len=16384 from=./dest_subv/source clone_offset=0 chown ./dest_subv/reflink gid=0 uid=0 chmod ./dest_subv/reflink mode=600 utimes ./dest_subv/reflink
But if we receive the first stream with default mount, then remount to nodatasum, and try to receive the second stream, it will fail:
# mount /mnt/btrfs # btrfs receive -f ~/parent_stream /mnt/btrfs/ At subvol parent_subv # mount -o remount,nodatasum /mnt/btrfs # btrfs receive -f ~/clone_stream /mnt/btrfs/ At snapshot dest_subv ERROR: failed to clone extents to reflink: Invalid argument # echo $? 1
[CAUSE]
Btrfs doesn't allow clone source and destination files have different NODATASUM flags.
This is to prevent a data extent to be owned by both NODATASUM inode and regular DATASUM inode.
For above receive operations, the clone destination is inheriting the NODATASUM flag from mount option, while the clone source has no NODATASUM flag, thus preventing us from doing the clone.
[FIX]
Btrfs send/receive doesn't require the underlying inode has the same flags (thus we can send from compressed extent and receive on a non-compressed filesystem).
So here we add a new command line option, '--clone-fallback', to allow btrfs-receive to fall back to buffered write to copy data from the source file.
Since such behavior can result much less clone operations, which may not be what the end users really want, and can hide bugs in send stream. Thus this behavior must be explicitly specified by the new option.
And we will output a warning message each time such fallback is triggered if the user wants extra debug output.
Finally if we have any failed clone operation when --clone-fallback is specified, also output a overview at the end of the received process, showing how many clone opeartions failed and how many bytes are no longer shared.
--- Documentation/btrfs-receive.asciidoc | 13 +++++ cmds/receive.c | 80 ++++++++++++++++++++++++++-- 2 files changed, 90 insertions(+), 3 deletions(-)
Documentation/btrfs-receive.asciidoc +13 -0
@@ -65,6 +65,19 @@ dump the stream metadata, one line per operation + Does not require the 'path' parameter. The filesystem remains unchanged. +--clone-fallback:: +when clone operations fail, attempt to directly copy the data instead. ++ +When the source and destination filesystems have different sector sizes, or +when the source and destination files have different 'nodatacow' and/or +'nodatasum' flags (can be set per file/directory or through mount options), +clone operations can fail. ++ +This option makes the receive process attempt to manually copy data from the +source to the destination file when a clone operation fails (for the reasons +mentioned before). When this happens, extents will end up not being shared +between the files, thus will take up more space. + -q|--quiet:: (deprecated) alias for global '-q' option
cmds/receive.c +77 -3
126 lines not on this page · raw message
Signature · 1 line
2.33.0
PATCH v3 2/2 btrfs-progs: misc-tests: add test case for receive --clone-fallback
In reply to Qu WenruoThe new test case will create two send streams:
- parent_stream
A full send stream.
Contains one file, as clone source.
- clone_stream
An incremental send stream.
Contains one clone operation.
Then we will receive the parent_stream with default mount options, while try to receive the clone_stream with nodatasum mount option.
This should result clone failure due to nodatasum flag mismatch.
Then check if the receive can continue with --clone-fallback option.
--- tests/common | 9 +++ .../049-receive-clone-fallback/test.sh | 58 +++++++++++++++++++ 2 files changed, 67 insertions(+) create mode 100755 tests/misc-tests/049-receive-clone-fallback/test.sh
tests/common +9 -0
@@ -633,6 +633,15 @@ run_check_mount_test_dev() run_check $SUDO_HELPER mount -t btrfs $loop_opt "$@" "$TEST_DEV" "$TEST_MNT" } +run_check_remount_test_dev() +{ + setup_root_helper + + local opts="$1" + + run_check $SUDO_HELPER mount -o "remount,$opts" "$TEST_MNT" +} + # $1-$n: optional paths to unmount, otherwise fallback to TEST_DEV run_check_umount_test_dev() {
tests/misc-tests/049-receive-clone-fallback/test.sh new file +58 -0
58 lines not on this page · raw message
Signature · 1 line
2.33.0