Skip to contentSkip to conversation

fstests/btrfs: add a test case for send v2 and an update

PATCH In review [email protected] 2 participants 6 messages Nov 28, 2022 Btrfs file system

lore
Share & tools
lore
#1 lore raw

PATCH 1/4 btrfs: add a _require_btrfs_send_v2 helper

In reply to [email protected]

From: Filipe Manana <[email protected]>

Add a helper to check that both btrfs-progs and kernel support the v2 send stream, so that we can have tests specific for send v2 stream.

---
 common/btrfs | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
common/btrfs +14 -0
index d27d3384..ee673a93 100644
@@ -585,3 +585,17 @@ _require_btrfs_corrupt_block()
 {
 	_require_command "$BTRFS_CORRUPT_BLOCK_PROG" btrfs-corrupt-block
 }
+
+_require_btrfs_send_v2()
+{
+	# Check first if btrfs-progs supports the v2 stream.
+	_require_btrfs_command send --compressed-data
+
+	# Now check the kernel support. If send_stream_version does not exists,
+	# then it's a kernel that only supports v1.
+	[ -f /sys/fs/btrfs/features/send_stream_version ] || \
+		_notrun "kernel does not support send stream v2"
+
+	[ $(cat /sys/fs/btrfs/features/send_stream_version) -gt 1 ] || \
+		_notrun "kernel does not support send stream v2"
+}
Signature · 1 line
2.35.1
#2 lore raw

PATCH 2/4 common: make _filter_fiemap_flags optionally print the encoded flag

In reply to [email protected]

From: Filipe Manana <[email protected]>

We'd like to have some btrfs test cases in the future to verify that extents are compressed when using fiemap. For that we can just check if the FIEMAP_EXTENT_ENCODED (0x8) flag is set for an extent. Currently _filter_fiemap_flags does not print that flag, so this changes it to print the flag.

However printing the encoded flag is optional, because some tests use the filter and use its output to match the golden output. So always printing the flag would make the tests fail on btrfs when they are run with "-o compress" (or compress-force) set in MOUNT_OPTIONS due to a mismatch with the golden output. The tests that can be run with or without compression on btrfs are generic/352, generic/353 and btrfs/279. Since those tests don't care about the encoded flag, there is no need to change them, just make the output of the flag optional, and any future tests that want to check the presence of the encoded flag, will just pass a parameter to _filter_fiemap_flags to tell it that the encoded flag should be printed.

---
 common/punch | 40 ++++++++++++++++++++++++++++++++++++----
 1 file changed, 36 insertions(+), 4 deletions(-)
common/punch +36 -4
index 94599c35..3b8be21a 100644
@@ -109,7 +109,23 @@ _filter_fiemap()
 
 _filter_fiemap_flags()
 {
-	$AWK_PROG '
+	local include_encoded_flag=0
+
+	# Unless a first argument is passed and with a value of 1, the fiemap
+	# encoded flag is filtered out.
+	# This is because tests that use this filter's output in their golden
+	# output may get the encoded flag set or not depending on the filesystem
+	# and its configuration. For example, running btrfs with "-o compress"
+	# (or compress-force) set in MOUNT_OPTIONS, then extents that get
+	# compressed are reported with the encoded flag, otherwise that flag is
+	# not reported. Like this the fs configuration does not cause a mismatch
+	# with the golden output, and tests that exercise specific configurations
+	# can explicitly ask for the encoded flag to be printed.
+	if [ ! -z "$1" ] && [ $1 -eq 1 ]; then
+		include_encoded_flag=1
+	fi
+
+	local awk_script='
 		$3 ~ /hole/ {
 			print $1, $2, $3;
 			next;
@@ -126,7 +142,22 @@ _filter_fiemap_flags()
 			if (and(flags, 0x2000)) {
 				flag_str = "shared";
 				set = 1;
-			}
+			}'
+
+	if [ $include_encoded_flag -eq 1 ]; then
+		awk_script=$awk_script'
+			if (and(flags, 0x8)) {
+				if (set) {
+					flag_str = flag_str"|";
+				} else {
+					flag_str = "";
+				}
+				flag_str = flag_str"encoded";
+				set = 1;
+			}'
+	fi
+
+	awk_script=$awk_script'
 			if (and(flags, 0x1)) {
 				if (set) {
 					flag_str = flag_str"|";
@@ -136,8 +167,9 @@ _filter_fiemap_flags()
 				flag_str = flag_str"last";
 			}
 			print $1, $2, flag_str
-		}' |
-	_coalesce_extents
+		}'
+
+	$AWK_PROG -e "$awk_script" | _coalesce_extents
 }
 
 # Filters fiemap output to only print the 
Signature · 1 line
2.35.1
#4 lore raw

PATCH 4/4 btrfs: test a case with compressed send stream and a shared extent

In reply to [email protected]

From: Filipe Manana <[email protected]>

Test that if we have a snapshot with a compressed extent that is partially shared between two files, one of them has a size that is not sector size aligned, we create a v2 send stream for the snapshot with compressed data, and then apply that stream to another filesystem, the operation succeeds and no data is missing. Also check that the file that had a reference to the whole extent gets two compressed extents in the new filesystem, with only one of them being shared (reflinked).

This tests a recent patch that landed in kernel 6.1-rc7:

a11452a3709e ("btrfs: send: avoid unaligned encoded writes when attempting to clone range")

---
 tests/btrfs/281     | 89 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/281.out | 17 +++++++++
 2 files changed, 106 insertions(+)
 create mode 100755 tests/btrfs/281
 create mode 100644 tests/btrfs/281.out
tests/btrfs/281 new file +89 -0
new file mode 100755
index 00000000..63fb89ea
@@ -0,0 +1,89 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test 281
+#
+# Test that if we have a snapshot with a compressed extent that is partially
+# shared between two files, one of them has a size that is not sector size
+# aligned, we create a v2 send stream for the snapshot with compressed data,
+# and then apply that stream to another filesystem, the operation succeeds and
+# no data is missing. Also check that the file that had a reference to the whole
+# extent gets two compressed extents in the new filesystem, with only one of
+# them being shared (reflinked).
+#
+. ./common/preamble
+_begin_fstest auto quick send compress clone fiemap
+
+. ./common/filter
+. ./common/reflink
+. ./common/punch # for _filter_fiemap_flags
+
+_supported_fs btrfs
+_require_test
+_require_scratch_reflink
+_require_btrfs_send_v2
+_require_xfs_io_command "fiemap"
+_require_fssum
+
+_fixed_by_kernel_commit a11452a3709e \
+	"btrfs: send: avoid unaligned encoded writes when attempting to clone range"
+
+send_files_dir=$TEST_DIR/btrfs-test-$seq
+send_stream=$send_files_dir/snap.stream
+snap_fssum=$send_files_dir/snap.fssum
+
+rm -fr $send_files_dir
+mkdir $send_files_dir
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount -o compress
+
+# File foo has a size of 65K, which is not sector size aligned for any
+# supported sector size on btrfs.
+$XFS_IO_PROG -f -c "pwrite -S 0xab 0 65K" $SCRATCH_MNT/foo | _filter_xfs_io
+
+# File bar has a compressed extent (and its size is sector size aligned).
+$XFS_IO_PROG -f -c "pwrite -S 0xcd 0 128K" $SCRATCH_MNT/bar | _filter_xfs_io
+
+# Now clone only half of bar's extent into foo.
+$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 0 64K" $SCRATCH_MNT/foo \
+	| _filter_xfs_io
+
+echo "Creating snapshot and a send stream for it..."
+$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/snap \
+	| _filter_scratch
+$BTRFS_UTIL_PROG send --compressed-data -f $send_stream $SCRATCH_MNT/snap 2>&1 \
+	| _filter_scratch
+
+$FSSUM_PROG -A -f -w $snap_fssum $SCRATCH_MNT/snap
+
+echo "Creating a new filesystem to receive the send stream..."
+_scratch_unmount
+_scratch_mkfs >> $seqres.full 2>&1
+# Mount without compression, we created the stream with data compression enabled
+# so we want to verify that applying the stream preserves the compression.
+_scratch_mount
+
+$BTRFS_UTIL_PROG receive -f $send_stream $SCRATCH_MNT
+
+echo "Verifying data matches the original filesystem..."
+$FSSUM_PROG -r $snap_fssum $SCRATCH_MNT/snap
+
+# Now check that fiemap reports two extents for file bar:
+#
+# 1) The first extent should be encoded, because compression was enabled in the
+#    original filesystem, and should also be flagged as shared, since that file
+#    range was reflinked with file foo in the original filesystem;
+#
+# 2) The second extent should also be encoded (compression was enabled in the
+#    original filesystem), but not shared since that file range was not
+#    reflinked in the original filesystem. It should also have the "last" flag
+#    set, as it's the last extent in the file.
+#
+echo "File bar fiemap output in the new filesystem:"
+$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap/bar | _filter_fiemap_flags 1
+
+# success, all done
+status=0
+exit
tests/btrfs/281.out new file +17 -0
new file mode 100644
index 00000000..2585e3e5
@@ -0,0 +1,17 @@
+QA output created by 281
+wrote 66560/66560 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+linked 65536/65536 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+Creating snapshot and a send stream for it...
+Create a readonly snapshot of 'SCRATCH_MNT' in 'SCRATCH_MNT/snap'
+At subvol SCRATCH_MNT/snap
+Creating a new filesystem to receive the send stream...
+At subvol snap
+Verifying data matches the original filesystem...
+OK
+File bar fiemap output in the new filesystem:
+0: [0..127]: shared|encoded
+1: [128..255]: encoded|last
Signature · 1 line
2.35.1
#0 lore raw

cover fstests/btrfs: add a test case for send v2 and an update

From: Filipe Manana <[email protected]>

This adds a test case specific for send v2 with some preparatory work in order to allow for that. Support for v2 send streams was added in btrfs-progs v5.19 and the kernel support introduced with kernel 6.0.

More send v2 specific tests will be added later.

Filipe Manana (4):
btrfs: add a _require_btrfs_send_v2 helper
common: make _filter_fiemap_flags optionally print the encoded flag btrfs/280: also verify that fiemap reports extents as encoded
btrfs: test a case with compressed send stream and a shared extent

 common/btrfs        | 14 +++++++
 common/punch        | 40 ++++++++++++++++++--
 tests/btrfs/280     |  4 +-
 tests/btrfs/280.out | 12 +++---
 tests/btrfs/281     | 89 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/281.out | 17 +++++++++
 6 files changed, 164 insertions(+), 12 deletions(-)
 create mode 100755 tests/btrfs/281
 create mode 100644 tests/btrfs/281.out
Signature · 1 line
2.35.1
#3 lore raw

PATCH 3/4 btrfs/280: also verify that fiemap reports extents as encoded

From: Filipe Manana <[email protected]>

Now that _filter_fiemap_flags() optionally reports the encoded flag and since btrfs/280 explicitly uses and tests compression, make it check that fiemap reports the compressed extents with the encoded flag set.

---
 tests/btrfs/280     |  4 ++--
 tests/btrfs/280.out | 12 ++++++------
 2 files changed, 8 insertions(+), 8 deletions(-)
tests/btrfs/280 +2 -2
index 06ef221e..fc049adb 100755
@@ -42,7 +42,7 @@ $BTRFS_UTIL_PROG subvolume snapshot $SCRATCH_MNT $SCRATCH_MNT/snap | _filter_scr
 echo
 echo "File foo fiemap before COWing extent:"
 echo
-$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/foo | _filter_fiemap_flags
+$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/foo | _filter_fiemap_flags 1
 
 echo
 echo "Overwriting file range [120M, 120M + 128K) in the snapshot"
@@ -57,7 +57,7 @@ echo "File foo fiemap after COWing extent in the snapshot:"
 echo
 # Now we should have all extents marked as shared except the 128K extent in the
 # file range [120M, 120M + 128K).
-$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/foo | _filter_fiemap_flags
+$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/foo | _filter_fiemap_flags 1
 
 # success, all done
 status=0
tests/btrfs/280.out +6 -6
index c3f82966..5371f3b0 100644
@@ -5,8 +5,8 @@ Create a snapshot of 'SCRATCH_MNT' in 'SCRATCH_MNT/snap'
 
 File foo fiemap before COWing extent:
 
-0: [0..261887]: shared
-1: [261888..262143]: shared|last
+0: [0..261887]: shared|encoded
+1: [261888..262143]: shared|encoded|last
 
 Overwriting file range [120M, 120M + 128K) in the snapshot
 
@@ -15,7 +15,7 @@ XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 File foo fiemap after COWing extent in the snapshot:
 
-0: [0..245759]: shared
-1: [245760..246015]: none
-2: [246016..261887]: shared
-3: [261888..262143]: shared|last
+0: [0..245759]: shared|encoded
+1: [245760..246015]: encoded
+2: [246016..261887]: shared|encoded
+3: [261888..262143]: shared|encoded|last
Signature · 1 line
2.35.1
Reviewed-by on the v1 series
#5 lore raw
In reply to [email protected]
Quoted 8 lines · On Mon, Nov 28, 2022 at 12:07:20PM +0000, [email protected] wrote:

From: Filipe Manana <[email protected]>

This adds a test case specific for send v2 with some preparatory work in order to allow for that. Support for v2 send streams was added in btrfs-progs v5.19 and the kernel support introduced with kernel 6.0.

More send v2 specific tests will be added later.

You can add

for the whole series, thanks,

Josef