Skip to contentSkip to conversation

Initial CephFS tests (take 2)

PATCH Superseded Luis Henriques 2 participants 8 messages Oct 7, 2020 – Oct 19, 2020

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Initial CephFS tests (take 2)

This is my second attempt to have an initial set of ceph-specific tests merged into fstests. In this patchset I'm pushing a different set of tests, focusing on the copy_file_range testing, although I *do* plan to get back to the quota tests soon.

This syscall has a few peculiarities in ceph as it is able to use remote object copies without the need to download/upload data from the OSDs. However, in order to take advantage of this remote copy, the copy ranges and sizes need to include at least one object. Thus, all the currently existing generic tests won't actually take advantage of this feature.

Let me know any comments/concerns about this patchset. Also note that currently, in order to enable copy_file_range in cephfs, the additional 'copyfrom' mount parameter is required. (Hopefully this additional param may be dropped in the future.)

Luis Henriques (3):
ceph: add copy_file_range (remote copy operation) testing
ceph: test combination of copy_file_range with truncate
ceph: test copy_file_range with infile = outfile

 tests/ceph/001     | 233 +++++++++++++++++++++++++++++++++++++++++++++
 tests/ceph/001.out | 129 +++++++++++++++++++++++++
 tests/ceph/002     |  74 ++++++++++++++
 tests/ceph/002.out |   8 ++
 tests/ceph/003     | 118 +++++++++++++++++++++++
 tests/ceph/003.out |  11 +++
 tests/ceph/group   |   3 +
 7 files changed, 576 insertions(+)
 create mode 100644 tests/ceph/001
 create mode 100644 tests/ceph/001.out
 create mode 100644 tests/ceph/002
 create mode 100644 tests/ceph/002.out
 create mode 100644 tests/ceph/003
 create mode 100644 tests/ceph/003.out
 create mode 100644 tests/ceph/group
#1 lore raw

PATCH 1/3 ceph: add copy_file_range (remote copy operation) testing

From: Luis Henriques <[email protected]>

Test remote copy operation (CEPH_OSD_OP_COPY_FROM) with several combinations of both object sizes and copy sizes. It also uses several combinations of copy ranges. For example, copying the 1st object in the src file into:

1) the beginning (1st object) of dst file,
2) the end (last object) of dst file and
3) the middle of the dst file.

---
 tests/ceph/001     | 233 +++++++++++++++++++++++++++++++++++++++++++++
 tests/ceph/001.out | 129 +++++++++++++++++++++++++
 tests/ceph/group   |   1 +
 3 files changed, 363 insertions(+)
 create mode 100644 tests/ceph/001
 create mode 100644 tests/ceph/001.out
 create mode 100644 tests/ceph/group
tests/ceph/001 new file +233 -0
new file mode 100644
index 000000000000..8ce0396f9723
@@ -0,0 +1,233 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2020 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test No. ceph/001
+#
+# Test remote copy operation (CEPH_OSD_OP_COPY_FROM) with several combinations
+# of both object sizes and copy sizes.  It also uses several combinations of
+# copy ranges.  For example, copying the 1st object in the src file into
+# 1) the beginning (1st object) of dst file, 2) the end (last object) of dst
+# file and 3) the middle of the dst file.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+}
+
+# get standard environment
+. common/rc
+. common/filter
+. common/attr
+. common/reflink
+
+# real QA test starts here
+_supported_fs ceph
+
+_require_xfs_io_command "copy_range"
+_require_attrs
+_require_test
+
+workdir=$TEST_DIR/test-$seq
+rm -rf $workdir
+mkdir $workdir
+rm -f $seqres.full
+
+check_range()
+{
+	local file=$1
+	local off0=$2
+	local off1=$3
+	local val=$4
+	_read_range $file $off0 $off1 | grep -v -q $val
+	[ $? -eq 0 ] && echo "file $file is not '$val' in [ $off0 $off1 ]"
+}
+
+run_copy_range_tests()
+{
+	objsz=$1
+	halfobj=$(($objsz / 2))
+	file="$workdir/file-$objsz"
+	copy="$workdir/copy-$objsz"
+	dest="$workdir/dest-$objsz"
+
+	# setting the file layout needs to be done before writing any data
+	touch $file $copy $dest
+	$SETFATTR_PROG -n ceph.file.layout \
+		-v "stripe_unit=$objsz stripe_count=1 object_size=$objsz" \
+		$file $copy $dest
+
+	# file containing 3 objects with 'aaaa|bbbb|cccc'
+	$XFS_IO_PROG -c "pwrite -S 0x61 0 $objsz" $file >> $seqres.full 2>&1
+	$XFS_IO_PROG -c "pwrite -S 0x62 $objsz $objsz" $file >> $seqres.full 2>&1
+	$XFS_IO_PROG -c "pwrite -S 0x63 $(($objsz * 2)) $objsz" $file >> $seqres.full 2>&1
+
+	echo "  Copy whole file (3 objects):"
+	echo "    aaaa|bbbb|cccc => aaaa|bbbb|cccc"
+	$XFS_IO_PROG -c "copy_range -s 0 -d 0 -l $(($objsz * 3)) $file" "$copy"
+	cmp $file $copy
+
+	echo "  Copy single object to beginning:"
+	# dest file with 3 objects with 'dddd|dddd|dddd'
+	$XFS_IO_PROG -c "pwrite -S 0x64 0 $(($objsz * 3))" $dest >> $seqres.full 2>&1
+
+	echo "    dddd|dddd|dddd => aaaa|dddd|dddd"
+	$XFS_IO_PROG -c "copy_range -s 0 -d 0 -l $objsz $file" "$dest"
+	check_range $dest 0 $objsz 61
+	check_range $dest $objsz $(($objsz * 2)) 64
+
+	echo "    aaaa|dddd|dddd => bbbb|dddd|dddd"
+	$XFS_IO_PROG -c "copy_range -s $objsz -d 0 -l $objsz $file" "$dest"
+	check_range $dest 0 $objsz 62
+	check_range $dest $objsz $(($objsz * 2)) 64
+
+	echo "    bbbb|dddd|dddd => cccc|dddd|dddd"
+	$XFS_IO_PROG -c "copy_range -s $(($objsz * 2)) -d 0 -l $objsz $file" "$dest"
+	check_range $dest 0 $objsz 63
+	check_range $dest $objsz $(($objsz * 2)) 64
+
+	echo "  Copy single object to middle:"
+
+	echo "    cccc|dddd|dddd => cccc|aaaa|dddd"
+	$XFS_IO_PROG -c "copy_range -s 0 -d $objsz -l $objsz $file" "$dest"
+	check_range $dest 0 $objsz 63
+	check_range $dest $objsz $objsz 61
+	check_range $dest $(($objsz * 2)) $objsz 64
+
+	echo "    cccc|aaaa|dddd => cccc|bbbb|dddd"
+	$XFS_IO_PROG -c "copy_range -s $objsz -d $objsz -l $objsz $file" "$dest"
+	check_range $dest 0 $objsz 63
+	check_range $dest $objsz $objsz 62
+	check_range $dest $(($objsz * 2)) $objsz 64
+
+	echo "    cccc|bbbb|dddd => cccc|cccc|dddd"
+	$XFS_IO_PROG -c "copy_range -s $((objsz * 2)) -d $objsz -l $objsz $file" "$dest"
+	check_range $dest 0 $objsz 63
+	check_range $dest $objsz $objsz 63
+	check_range $dest $(($objsz * 2)) $objsz 64
+
+	echo "  Copy single object to end:"
+
+	echo "    cccc|cccc|dddd => cccc|cccc|aaaa"
+	$XFS_IO_PROG -c "copy_range -s 0 -d $(($objsz * 2)) -l $objsz $file" "$dest"
+	check_range $dest 0 $(($objsz * 2)) 63
+	check_range $dest $(($objsz * 2)) $objsz 61
+
+	echo "    cccc|cccc|aaaa => cccc|cccc|bbbb"
+	$XFS_IO_PROG -c "copy_range -s $objsz -d $(($objsz * 2)) -l $objsz $file" "$dest"
+	check_range $dest 0 $(($objsz * 2)) 63
+	check_range $dest $(($objsz * 2)) $objsz 62
+
+	echo "    cccc|cccc|aaaa => cccc|cccc|cccc"
+	$XFS_IO_PROG -c "copy_range -s $(($objsz * 2)) -d $(($objsz * 2)) -l $objsz $file" "$dest"
+	check_range $dest 0 $(($objsz * 3)) 63
+
+	echo "  Copy 2 objects to beginning:"
+
+	echo "    cccc|cccc|cccc => aaaa|bbbb|cccc"
+	$XFS_IO_PROG -c "copy_range -s 0 -d 0 -l $(($objsz * 2)) $file" "$dest"
+	cmp $file $dest
+
+	echo "    aaaa|bbbb|cccc => bbbb|cccc|cccc"
+	$XFS_IO_PROG -c "copy_range -s $objsz -d 0 -l $(($objsz * 2)) $file" "$dest"
+	check_range $dest 0 $objsz 62
+	check_range $dest $objsz $(($objsz * 2)) 63
+
+	echo "  Copy 2 objects to end:"
+
+	echo "    bbbb|cccc|cccc => bbbb|aaaa|bbbb"
+	$XFS_IO_PROG -c "copy_range -s 0 -d $objsz -l $(($objsz * 2)) $file" "$dest"
+	check_range $dest 0 $objsz 62
+	check_range $dest $objsz $objsz 61
+	check_range $dest $(($objsz * 2)) $objsz 62
+
+	echo "    bbbb|aaaa|bbbb => bbbb|bbbb|cccc"
+	$XFS_IO_PROG -c "copy_range -s $objsz -d $objsz -l $(($objsz * 2)) $file" "$dest"
+	check_range $dest 0 $(($objsz * 2)) 62
+	check_range $dest $(($objsz * 2)) $objsz 63
+
+	echo "  Append 1 object:"
+
+	echo "    bbbb|bbbb|cccc => bbbb|bbbb|cccc|aaaa"
+	$XFS_IO_PROG -c "copy_range -s 0 -d $(($objsz * 3)) -l $objsz $file" "$dest"
+	check_range $dest 0 $(($objsz * 2)) 62
+	check_range $dest $(($objsz * 2)) $objsz 63
+	check_range $dest $(($objsz * 3)) $objsz 61
+
+	echo "  Cross object boundary (no full object copy)"
+	echo "    dddd|dddd|dddd|dddd => ddaa|aadd|dddd|dddd"
+	$XFS_IO_PROG -c "pwrite -S 0x64 0 $(($objsz * 4))" $dest >> $seqres.full 2>&1
+	$XFS_IO_PROG -c "copy_range -s 0 -d $halfobj -l $objsz $file" "$dest"
+	check_range $dest 0 $halfobj 64
+	check_range $dest $halfobj $objsz 61
+	check_range $dest $(($objsz + $halfobj)) $(($objsz * 2 + $halfobj)) 64
+
+	echo "    dddd|dddd|dddd|dddd => ddaa|bbdd|dddd|dddd"
+	$XFS_IO_PROG -c "pwrite -S 0x64 0 $(($objsz * 4))" $dest >> $seqres.full 2>&1
+	$XFS_IO_PROG -c "copy_range -s $halfobj -d $halfobj -l $objsz $file" "$dest"
+	check_range $dest 0 $halfobj 64
+	check_range $dest $halfobj $halfobj 61
+	check_range $dest $objsz $halfobj 62
+	check_range $dest $(($objsz + $halfobj)) $(($objsz * 2 + $halfobj)) 64
+
+	echo "  Cross object boundaries (with full object copy)"
+	echo "    dddd|dddd|dddd|dddd => ddaa|bbbb|dddd|dddd"
+	$XFS_IO_PROG -c "pwrite -S 0x64 0 $(($objsz * 4))" $dest >> $seqres.full 2>&1
+	$XFS_IO_PROG -c "copy_range -s $halfobj -d $halfobj -l $(($objsz + $halfobj)) $file" "$dest"
+	check_range $dest 0 $halfobj 64
+	check_range $dest $halfobj $halfobj 61
+	check_range $dest $objsz $objsz 62
+	check_range $dest $(($objsz * 2)) $(($objsz * 2)) 64
+
+	echo "    dddd|dddd|dddd|dddd => ddaa|bbbb|ccdd|dddd"
+	$XFS_IO_PROG -c "pwrite -S 0x64 0 $(($objsz * 4))" $dest >> $seqres.full 2>&1
+	$XFS_IO_PROG -c "copy_range -s $halfobj -d $halfobj -l $(($objsz * 2)) $file" "$dest"
+	check_range $dest 0 $halfobj 64
+	check_range $dest $halfobj $halfobj 61
+	check_range $dest $objsz $objsz 62
+	check_range $dest $(($objsz * 2)) $halfobj 63
+	check_range $dest $(($objsz * 2 + $halfobj)) $(($objsz + $halfobj)) 64
+
+	echo "    dddd|dddd|dddd|dddd => dddd|aaaa|bbdd|dddd"
+	$XFS_IO_PROG -c "pwrite -S 0x64 0 $(($objsz * 4))" $dest >> $seqres.full 2>&1
+	$XFS_IO_PROG -c "copy_range -s 0 -d $objsz -l $(($objsz + $halfobj)) $file" "$dest"
+	check_range $dest 0 $objsz 64
+	check_range $dest $objsz $objsz 61
+	check_range $dest $(($objsz * 2)) $halfobj 62
+	check_range $dest $(($objsz * 2 + $halfobj)) $(($objsz + $halfobj)) 64
+
+	echo "  Cross object boundaries (with 2 full object copies)"
+	echo "    dddd|dddd|dddd|dddd => ddaa|aabb|bbcc|ccdd"
+	$XFS_IO_PROG -c "pwrite -S 0x64 0 $(($objsz * 4))" $dest >> $seqres.full 2>&1
+	$XFS_IO_PROG -c "copy_range -s 0 -d $halfobj -l $(($objsz * 3)) $file" "$dest"
+	check_range $dest 0 $halfobj 64
+	check_range $dest $halfobj $objsz 61
+	check_range $dest $(($objsz + $halfobj)) $objsz 62
+	check_range $dest $(($objsz * 2 + $halfobj)) $objsz 63
+	check_range $dest $(($objsz * 3 + $halfobj)) $halfobj 64
+
+}
+
+echo "Object size: 65536" # CEPH_MIN_STRIPE_UNIT
+run_copy_range_tests 65536
+echo "Object size: 1M"
+run_copy_range_tests 1048576
+echo "Object size: 4M"
+run_copy_range_tests 4194304
+# the max object size is 1TB, but by default OSDs only accept a max of 128M objects
+echo "Object size: 128M"
+run_copy_range_tests 134217728
+
+#success, all done
+status=0
+exit
tests/ceph/001.out new file +129 -0

129 lines not on this page · raw message

tests/ceph/group new file +1 -0
new file mode 100644
index 000000000000..11f0b9ad03d3
@@ -0,0 +1 @@
+001 auto quick copy
#2 lore raw

Re: [PATCH 1/3] ceph: add copy_file_range (remote copy operation) testing

Quoted 85 lines · Luis Henriques

From: Luis Henriques <[email protected]>

Test remote copy operation (CEPH_OSD_OP_COPY_FROM) with several combinations of both object sizes and copy sizes. It also uses several combinations of copy ranges. For example, copying the 1st object in the src file into:

1) the beginning (1st object) of dst file,
2) the end (last object) of dst file and
3) the middle of the dst file.

---
 tests/ceph/001     | 233 +++++++++++++++++++++++++++++++++++++++++++++
 tests/ceph/001.out | 129 +++++++++++++++++++++++++
 tests/ceph/group   |   1 +
 3 files changed, 363 insertions(+)
 create mode 100644 tests/ceph/001
 create mode 100644 tests/ceph/001.out
 create mode 100644 tests/ceph/group
tests/ceph/001 new file +58 -0
new file mode 100644
index 000000000000..8ce0396f9723
@@ -0,0 +1,233 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2020 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test No. ceph/001
+#
+# Test remote copy operation (CEPH_OSD_OP_COPY_FROM) with several combinations
+# of both object sizes and copy sizes.  It also uses several combinations of
+# copy ranges.  For example, copying the 1st object in the src file into
+# 1) the beginning (1st object) of dst file, 2) the end (last object) of dst
+# file and 3) the middle of the dst file.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+}
+
+# get standard environment
+. common/rc
+. common/filter
+. common/attr
+. common/reflink
+
+# real QA test starts here
+_supported_fs ceph
+
+_require_xfs_io_command "copy_range"
+_require_attrs
+_require_test
+
+workdir=$TEST_DIR/test-$seq
+rm -rf $workdir
+mkdir $workdir
+rm -f $seqres.full
+
+check_range()
+{
+	local file=$1
+	local off0=$2
+	local off1=$3
+	local val=$4
+	_read_range $file $off0 $off1 | grep -v -q $val
+	[ $? -eq 0 ] && echo "file $file is not '$val' in [ $off0 $off1 ]"
+}
+
+run_copy_range_tests()
+{
+	objsz=$1

tests/ceph/001 line 68

Quoted 10 lines
tests/ceph/001 new file +10 -0
+	halfobj=$(($objsz / 2))
+	file="$workdir/file-$objsz"
+	copy="$workdir/copy-$objsz"
+	dest="$workdir/dest-$objsz"
+
+	# setting the file layout needs to be done before writing any data
+	touch $file $copy $dest
+	$SETFATTR_PROG -n ceph.file.layout \
+		-v "stripe_unit=$objsz stripe_count=1 object_size=$objsz" \
+		$file $copy $dest

I noticed this pattern repeated a few times in this patchset, I think it'd be good to factor it into a new helper function? And put it in common/ceph file.

Quoted 153 lines
tests/ceph/001 new file +153 -0
+
+	# file containing 3 objects with 'aaaa|bbbb|cccc'
+	$XFS_IO_PROG -c "pwrite -S 0x61 0 $objsz" $file >> $seqres.full 2>&1
+	$XFS_IO_PROG -c "pwrite -S 0x62 $objsz $objsz" $file >> $seqres.full 2>&1
+	$XFS_IO_PROG -c "pwrite -S 0x63 $(($objsz * 2)) $objsz" $file >> $seqres.full 2>&1
+
+	echo "  Copy whole file (3 objects):"
+	echo "    aaaa|bbbb|cccc => aaaa|bbbb|cccc"
+	$XFS_IO_PROG -c "copy_range -s 0 -d 0 -l $(($objsz * 3)) $file" "$copy"
+	cmp $file $copy
+
+	echo "  Copy single object to beginning:"
+	# dest file with 3 objects with 'dddd|dddd|dddd'
+	$XFS_IO_PROG -c "pwrite -S 0x64 0 $(($objsz * 3))" $dest >> $seqres.full 2>&1
+
+	echo "    dddd|dddd|dddd => aaaa|dddd|dddd"
+	$XFS_IO_PROG -c "copy_range -s 0 -d 0 -l $objsz $file" "$dest"
+	check_range $dest 0 $objsz 61
+	check_range $dest $objsz $(($objsz * 2)) 64
+
+	echo "    aaaa|dddd|dddd => bbbb|dddd|dddd"
+	$XFS_IO_PROG -c "copy_range -s $objsz -d 0 -l $objsz $file" "$dest"
+	check_range $dest 0 $objsz 62
+	check_range $dest $objsz $(($objsz * 2)) 64
+
+	echo "    bbbb|dddd|dddd => cccc|dddd|dddd"
+	$XFS_IO_PROG -c "copy_range -s $(($objsz * 2)) -d 0 -l $objsz $file" "$dest"
+	check_range $dest 0 $objsz 63
+	check_range $dest $objsz $(($objsz * 2)) 64
+
+	echo "  Copy single object to middle:"
+
+	echo "    cccc|dddd|dddd => cccc|aaaa|dddd"
+	$XFS_IO_PROG -c "copy_range -s 0 -d $objsz -l $objsz $file" "$dest"
+	check_range $dest 0 $objsz 63
+	check_range $dest $objsz $objsz 61
+	check_range $dest $(($objsz * 2)) $objsz 64
+
+	echo "    cccc|aaaa|dddd => cccc|bbbb|dddd"
+	$XFS_IO_PROG -c "copy_range -s $objsz -d $objsz -l $objsz $file" "$dest"
+	check_range $dest 0 $objsz 63
+	check_range $dest $objsz $objsz 62
+	check_range $dest $(($objsz * 2)) $objsz 64
+
+	echo "    cccc|bbbb|dddd => cccc|cccc|dddd"
+	$XFS_IO_PROG -c "copy_range -s $((objsz * 2)) -d $objsz -l $objsz $file" "$dest"
+	check_range $dest 0 $objsz 63
+	check_range $dest $objsz $objsz 63
+	check_range $dest $(($objsz * 2)) $objsz 64
+
+	echo "  Copy single object to end:"
+
+	echo "    cccc|cccc|dddd => cccc|cccc|aaaa"
+	$XFS_IO_PROG -c "copy_range -s 0 -d $(($objsz * 2)) -l $objsz $file" "$dest"
+	check_range $dest 0 $(($objsz * 2)) 63
+	check_range $dest $(($objsz * 2)) $objsz 61
+
+	echo "    cccc|cccc|aaaa => cccc|cccc|bbbb"
+	$XFS_IO_PROG -c "copy_range -s $objsz -d $(($objsz * 2)) -l $objsz $file" "$dest"
+	check_range $dest 0 $(($objsz * 2)) 63
+	check_range $dest $(($objsz * 2)) $objsz 62
+
+	echo "    cccc|cccc|aaaa => cccc|cccc|cccc"
+	$XFS_IO_PROG -c "copy_range -s $(($objsz * 2)) -d $(($objsz * 2)) -l $objsz $file" "$dest"
+	check_range $dest 0 $(($objsz * 3)) 63
+
+	echo "  Copy 2 objects to beginning:"
+
+	echo "    cccc|cccc|cccc => aaaa|bbbb|cccc"
+	$XFS_IO_PROG -c "copy_range -s 0 -d 0 -l $(($objsz * 2)) $file" "$dest"
+	cmp $file $dest
+
+	echo "    aaaa|bbbb|cccc => bbbb|cccc|cccc"
+	$XFS_IO_PROG -c "copy_range -s $objsz -d 0 -l $(($objsz * 2)) $file" "$dest"
+	check_range $dest 0 $objsz 62
+	check_range $dest $objsz $(($objsz * 2)) 63
+
+	echo "  Copy 2 objects to end:"
+
+	echo "    bbbb|cccc|cccc => bbbb|aaaa|bbbb"
+	$XFS_IO_PROG -c "copy_range -s 0 -d $objsz -l $(($objsz * 2)) $file" "$dest"
+	check_range $dest 0 $objsz 62
+	check_range $dest $objsz $objsz 61
+	check_range $dest $(($objsz * 2)) $objsz 62
+
+	echo "    bbbb|aaaa|bbbb => bbbb|bbbb|cccc"
+	$XFS_IO_PROG -c "copy_range -s $objsz -d $objsz -l $(($objsz * 2)) $file" "$dest"
+	check_range $dest 0 $(($objsz * 2)) 62
+	check_range $dest $(($objsz * 2)) $objsz 63
+
+	echo "  Append 1 object:"
+
+	echo "    bbbb|bbbb|cccc => bbbb|bbbb|cccc|aaaa"
+	$XFS_IO_PROG -c "copy_range -s 0 -d $(($objsz * 3)) -l $objsz $file" "$dest"
+	check_range $dest 0 $(($objsz * 2)) 62
+	check_range $dest $(($objsz * 2)) $objsz 63
+	check_range $dest $(($objsz * 3)) $objsz 61
+
+	echo "  Cross object boundary (no full object copy)"
+	echo "    dddd|dddd|dddd|dddd => ddaa|aadd|dddd|dddd"
+	$XFS_IO_PROG -c "pwrite -S 0x64 0 $(($objsz * 4))" $dest >> $seqres.full 2>&1
+	$XFS_IO_PROG -c "copy_range -s 0 -d $halfobj -l $objsz $file" "$dest"
+	check_range $dest 0 $halfobj 64
+	check_range $dest $halfobj $objsz 61
+	check_range $dest $(($objsz + $halfobj)) $(($objsz * 2 + $halfobj)) 64
+
+	echo "    dddd|dddd|dddd|dddd => ddaa|bbdd|dddd|dddd"
+	$XFS_IO_PROG -c "pwrite -S 0x64 0 $(($objsz * 4))" $dest >> $seqres.full 2>&1
+	$XFS_IO_PROG -c "copy_range -s $halfobj -d $halfobj -l $objsz $file" "$dest"
+	check_range $dest 0 $halfobj 64
+	check_range $dest $halfobj $halfobj 61
+	check_range $dest $objsz $halfobj 62
+	check_range $dest $(($objsz + $halfobj)) $(($objsz * 2 + $halfobj)) 64
+
+	echo "  Cross object boundaries (with full object copy)"
+	echo "    dddd|dddd|dddd|dddd => ddaa|bbbb|dddd|dddd"
+	$XFS_IO_PROG -c "pwrite -S 0x64 0 $(($objsz * 4))" $dest >> $seqres.full 2>&1
+	$XFS_IO_PROG -c "copy_range -s $halfobj -d $halfobj -l $(($objsz + $halfobj)) $file" "$dest"
+	check_range $dest 0 $halfobj 64
+	check_range $dest $halfobj $halfobj 61
+	check_range $dest $objsz $objsz 62
+	check_range $dest $(($objsz * 2)) $(($objsz * 2)) 64
+
+	echo "    dddd|dddd|dddd|dddd => ddaa|bbbb|ccdd|dddd"
+	$XFS_IO_PROG -c "pwrite -S 0x64 0 $(($objsz * 4))" $dest >> $seqres.full 2>&1
+	$XFS_IO_PROG -c "copy_range -s $halfobj -d $halfobj -l $(($objsz * 2)) $file" "$dest"
+	check_range $dest 0 $halfobj 64
+	check_range $dest $halfobj $halfobj 61
+	check_range $dest $objsz $objsz 62
+	check_range $dest $(($objsz * 2)) $halfobj 63
+	check_range $dest $(($objsz * 2 + $halfobj)) $(($objsz + $halfobj)) 64
+
+	echo "    dddd|dddd|dddd|dddd => dddd|aaaa|bbdd|dddd"
+	$XFS_IO_PROG -c "pwrite -S 0x64 0 $(($objsz * 4))" $dest >> $seqres.full 2>&1
+	$XFS_IO_PROG -c "copy_range -s 0 -d $objsz -l $(($objsz + $halfobj)) $file" "$dest"
+	check_range $dest 0 $objsz 64
+	check_range $dest $objsz $objsz 61
+	check_range $dest $(($objsz * 2)) $halfobj 62
+	check_range $dest $(($objsz * 2 + $halfobj)) $(($objsz + $halfobj)) 64
+
+	echo "  Cross object boundaries (with 2 full object copies)"
+	echo "    dddd|dddd|dddd|dddd => ddaa|aabb|bbcc|ccdd"
+	$XFS_IO_PROG -c "pwrite -S 0x64 0 $(($objsz * 4))" $dest >> $seqres.full 2>&1
+	$XFS_IO_PROG -c "copy_range -s 0 -d $halfobj -l $(($objsz * 3)) $file" "$dest"
+	check_range $dest 0 $halfobj 64
+	check_range $dest $halfobj $objsz 61
+	check_range $dest $(($objsz + $halfobj)) $objsz 62
+	check_range $dest $(($objsz * 2 + $halfobj)) $objsz 63
+	check_range $dest $(($objsz * 3 + $halfobj)) $halfobj 64
+
+}
+
+echo "Object size: 65536" # CEPH_MIN_STRIPE_UNIT

tests/ceph/001 line 231

Quoted 10 lines
tests/ceph/001 new file +10 -0
+run_copy_range_tests 65536
+echo "Object size: 1M"
+run_copy_range_tests 1048576
+echo "Object size: 4M"
+run_copy_range_tests 4194304
+# the max object size is 1TB, but by default OSDs only accept a max of 128M objects
+echo "Object size: 128M"
+run_copy_range_tests 134217728
+
+#success, all done

Minor nit, add a space after '#' for comments.

Thanks,
Eryu

Quoted 144 lines
tests/ceph/001 new file +2 -0
+status=0
+exit
tests/ceph/001.out new file +129 -0
new file mode 100644
index 000000000000..3cc7837a595d
@@ -0,0 +1,129 @@
+QA output created by 001
+Object size: 65536
+  Copy whole file (3 objects):
+    aaaa|bbbb|cccc => aaaa|bbbb|cccc
+  Copy single object to beginning:
+    dddd|dddd|dddd => aaaa|dddd|dddd
+    aaaa|dddd|dddd => bbbb|dddd|dddd
+    bbbb|dddd|dddd => cccc|dddd|dddd
+  Copy single object to middle:
+    cccc|dddd|dddd => cccc|aaaa|dddd
+    cccc|aaaa|dddd => cccc|bbbb|dddd
+    cccc|bbbb|dddd => cccc|cccc|dddd
+  Copy single object to end:
+    cccc|cccc|dddd => cccc|cccc|aaaa
+    cccc|cccc|aaaa => cccc|cccc|bbbb
+    cccc|cccc|aaaa => cccc|cccc|cccc
+  Copy 2 objects to beginning:
+    cccc|cccc|cccc => aaaa|bbbb|cccc
+    aaaa|bbbb|cccc => bbbb|cccc|cccc
+  Copy 2 objects to end:
+    bbbb|cccc|cccc => bbbb|aaaa|bbbb
+    bbbb|aaaa|bbbb => bbbb|bbbb|cccc
+  Append 1 object:
+    bbbb|bbbb|cccc => bbbb|bbbb|cccc|aaaa
+  Cross object boundary (no full object copy)
+    dddd|dddd|dddd|dddd => ddaa|aadd|dddd|dddd
+    dddd|dddd|dddd|dddd => ddaa|bbdd|dddd|dddd
+  Cross object boundaries (with full object copy)
+    dddd|dddd|dddd|dddd => ddaa|bbbb|dddd|dddd
+    dddd|dddd|dddd|dddd => ddaa|bbbb|ccdd|dddd
+    dddd|dddd|dddd|dddd => dddd|aaaa|bbdd|dddd
+  Cross object boundaries (with 2 full object copies)
+    dddd|dddd|dddd|dddd => ddaa|aabb|bbcc|ccdd
+Object size: 1M
+  Copy whole file (3 objects):
+    aaaa|bbbb|cccc => aaaa|bbbb|cccc
+  Copy single object to beginning:
+    dddd|dddd|dddd => aaaa|dddd|dddd
+    aaaa|dddd|dddd => bbbb|dddd|dddd
+    bbbb|dddd|dddd => cccc|dddd|dddd
+  Copy single object to middle:
+    cccc|dddd|dddd => cccc|aaaa|dddd
+    cccc|aaaa|dddd => cccc|bbbb|dddd
+    cccc|bbbb|dddd => cccc|cccc|dddd
+  Copy single object to end:
+    cccc|cccc|dddd => cccc|cccc|aaaa
+    cccc|cccc|aaaa => cccc|cccc|bbbb
+    cccc|cccc|aaaa => cccc|cccc|cccc
+  Copy 2 objects to beginning:
+    cccc|cccc|cccc => aaaa|bbbb|cccc
+    aaaa|bbbb|cccc => bbbb|cccc|cccc
+  Copy 2 objects to end:
+    bbbb|cccc|cccc => bbbb|aaaa|bbbb
+    bbbb|aaaa|bbbb => bbbb|bbbb|cccc
+  Append 1 object:
+    bbbb|bbbb|cccc => bbbb|bbbb|cccc|aaaa
+  Cross object boundary (no full object copy)
+    dddd|dddd|dddd|dddd => ddaa|aadd|dddd|dddd
+    dddd|dddd|dddd|dddd => ddaa|bbdd|dddd|dddd
+  Cross object boundaries (with full object copy)
+    dddd|dddd|dddd|dddd => ddaa|bbbb|dddd|dddd
+    dddd|dddd|dddd|dddd => ddaa|bbbb|ccdd|dddd
+    dddd|dddd|dddd|dddd => dddd|aaaa|bbdd|dddd
+  Cross object boundaries (with 2 full object copies)
+    dddd|dddd|dddd|dddd => ddaa|aabb|bbcc|ccdd
+Object size: 4M
+  Copy whole file (3 objects):
+    aaaa|bbbb|cccc => aaaa|bbbb|cccc
+  Copy single object to beginning:
+    dddd|dddd|dddd => aaaa|dddd|dddd
+    aaaa|dddd|dddd => bbbb|dddd|dddd
+    bbbb|dddd|dddd => cccc|dddd|dddd
+  Copy single object to middle:
+    cccc|dddd|dddd => cccc|aaaa|dddd
+    cccc|aaaa|dddd => cccc|bbbb|dddd
+    cccc|bbbb|dddd => cccc|cccc|dddd
+  Copy single object to end:
+    cccc|cccc|dddd => cccc|cccc|aaaa
+    cccc|cccc|aaaa => cccc|cccc|bbbb
+    cccc|cccc|aaaa => cccc|cccc|cccc
+  Copy 2 objects to beginning:
+    cccc|cccc|cccc => aaaa|bbbb|cccc
+    aaaa|bbbb|cccc => bbbb|cccc|cccc
+  Copy 2 objects to end:
+    bbbb|cccc|cccc => bbbb|aaaa|bbbb
+    bbbb|aaaa|bbbb => bbbb|bbbb|cccc
+  Append 1 object:
+    bbbb|bbbb|cccc => bbbb|bbbb|cccc|aaaa
+  Cross object boundary (no full object copy)
+    dddd|dddd|dddd|dddd => ddaa|aadd|dddd|dddd
+    dddd|dddd|dddd|dddd => ddaa|bbdd|dddd|dddd
+  Cross object boundaries (with full object copy)
+    dddd|dddd|dddd|dddd => ddaa|bbbb|dddd|dddd
+    dddd|dddd|dddd|dddd => ddaa|bbbb|ccdd|dddd
+    dddd|dddd|dddd|dddd => dddd|aaaa|bbdd|dddd
+  Cross object boundaries (with 2 full object copies)
+    dddd|dddd|dddd|dddd => ddaa|aabb|bbcc|ccdd
+Object size: 128M
+  Copy whole file (3 objects):
+    aaaa|bbbb|cccc => aaaa|bbbb|cccc
+  Copy single object to beginning:
+    dddd|dddd|dddd => aaaa|dddd|dddd
+    aaaa|dddd|dddd => bbbb|dddd|dddd
+    bbbb|dddd|dddd => cccc|dddd|dddd
+  Copy single object to middle:
+    cccc|dddd|dddd => cccc|aaaa|dddd
+    cccc|aaaa|dddd => cccc|bbbb|dddd
+    cccc|bbbb|dddd => cccc|cccc|dddd
+  Copy single object to end:
+    cccc|cccc|dddd => cccc|cccc|aaaa
+    cccc|cccc|aaaa => cccc|cccc|bbbb
+    cccc|cccc|aaaa => cccc|cccc|cccc
+  Copy 2 objects to beginning:
+    cccc|cccc|cccc => aaaa|bbbb|cccc
+    aaaa|bbbb|cccc => bbbb|cccc|cccc
+  Copy 2 objects to end:
+    bbbb|cccc|cccc => bbbb|aaaa|bbbb
+    bbbb|aaaa|bbbb => bbbb|bbbb|cccc
+  Append 1 object:
+    bbbb|bbbb|cccc => bbbb|bbbb|cccc|aaaa
+  Cross object boundary (no full object copy)
+    dddd|dddd|dddd|dddd => ddaa|aadd|dddd|dddd
+    dddd|dddd|dddd|dddd => ddaa|bbdd|dddd|dddd
+  Cross object boundaries (with full object copy)
+    dddd|dddd|dddd|dddd => ddaa|bbbb|dddd|dddd
+    dddd|dddd|dddd|dddd => ddaa|bbbb|ccdd|dddd
+    dddd|dddd|dddd|dddd => dddd|aaaa|bbdd|dddd
+  Cross object boundaries (with 2 full object copies)
+    dddd|dddd|dddd|dddd => ddaa|aabb|bbcc|ccdd
tests/ceph/group new file +1 -0
new file mode 100644
index 000000000000..11f0b9ad03d3
@@ -0,0 +1 @@
+001 auto quick copy
#3 lore raw

PATCH 2/3 ceph: test combination of copy_file_range with truncate

In reply to Luis Henriques

From: Luis Henriques <[email protected]>

This test was motivated by an OSD bug found while testing copy_file_range. This bug was an issue with the way the OSDs handled the truncate_seq value, which was being copied from the original object into the destination object. This test ensures the kernel client correctly handles fixed/non-fixed OSDs.

---
 tests/ceph/002     | 74 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/ceph/002.out |  8 +++++
 tests/ceph/group   |  1 +
 3 files changed, 83 insertions(+)
 create mode 100644 tests/ceph/002
 create mode 100644 tests/ceph/002.out
tests/ceph/002 new file +74 -0
new file mode 100644
index 000000000000..a2ae6c9629f4
@@ -0,0 +1,74 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2020 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test No. ceph/002
+#
+# Test bug found while testing copy_file_range.
+#
+# This bug was an issue with how the OSDs handled the truncate_seq, copying it
+# from the original object into the destination object.  This test ensures the
+# kernel client correctly handles fixed/non-fixed OSDs.
+#
+# https://tracker.ceph.com/issues/37378
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+}
+
+# get standard environment
+. common/rc
+. common/filter
+. common/attr
+
+# real QA test starts here
+_supported_fs ceph
+
+_require_xfs_io_command "copy_range"
+_require_attrs
+_require_test
+
+workdir=$TEST_DIR/test-$seq
+rm -rf $workdir
+mkdir $workdir
+rm -f $seqres.full
+
+# Use 4M object size
+objsz=4194304
+file="$workdir/file-$objsz"
+dest="$workdir/dest-$objsz"
+touch $file $dest
+
+# object_size has to be a multiple of stripe_unit
+$SETFATTR_PROG -n ceph.file.layout \
+	-v "stripe_unit=$objsz stripe_count=1 object_size=$objsz" \
+	$file $dest
+
+# Create a 3 objects size files
+$XFS_IO_PROG -c "pwrite -S 0x61 0 $objsz" $file >> $seqres.full 2>&1
+$XFS_IO_PROG -c "pwrite -S 0x62 $objsz $objsz" $file >> $seqres.full 2>&1
+$XFS_IO_PROG -c "pwrite -S 0x63 $(($objsz * 2)) $objsz" $file >> $seqres.full 2>&1
+
+$XFS_IO_PROG -c "pwrite -S 0x64 0 $(($objsz * 3))" $dest >> $seqres.full 2>&1
+# Truncate the destination file (messing up with the truncate_seq)
+$XFS_IO_PROG -c "truncate 0" $dest >> $seqres.full 2>&1
+
+# copy the whole file over
+$XFS_IO_PROG -c "copy_range -s 0 -d 0 -l $(($objsz * 3)) $file" "$dest"
+
+hexdump $dest
+
+#success, all done
+status=0
+exit
tests/ceph/002.out new file +8 -0
new file mode 100644
index 000000000000..6f067250afff
@@ -0,0 +1,8 @@
+QA output created by 002
+0000000 6161 6161 6161 6161 6161 6161 6161 6161
+*
+0400000 6262 6262 6262 6262 6262 6262 6262 6262
+*
+0800000 6363 6363 6363 6363 6363 6363 6363 6363
+*
+0c00000
tests/ceph/group +1 -0
index 11f0b9ad03d3..c28fe473c1a4 100644
@@ -1 +1,2 @@
 001 auto quick copy
+002 auto quick copy
#4 lore raw

Re: [PATCH 2/3] ceph: test combination of copy_file_range with truncate

Quoted 27 lines · Luis Henriques

From: Luis Henriques <[email protected]>

This test was motivated by an OSD bug found while testing copy_file_range. This bug was an issue with the way the OSDs handled the truncate_seq value, which was being copied from the original object into the destination object. This test ensures the kernel client correctly handles fixed/non-fixed OSDs.

---
 tests/ceph/002     | 74 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/ceph/002.out |  8 +++++
 tests/ceph/group   |  1 +
 3 files changed, 83 insertions(+)
 create mode 100644 tests/ceph/002
 create mode 100644 tests/ceph/002.out
tests/ceph/002 new file +3 -0
new file mode 100644
index 000000000000..a2ae6c9629f4
@@ -0,0 +1,74 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2020 SUSE Linux Products GmbH. All Rights Reserved.

tests/ceph/002 line 13

Quoted 10 lines
tests/ceph/002 new file +10 -0
+#
+# FS QA Test No. ceph/002
+#
+# Test bug found while testing copy_file_range.
+#
+# This bug was an issue with how the OSDs handled the truncate_seq, copying it
+# from the original object into the destination object.  This test ensures the
+# kernel client correctly handles fixed/non-fixed OSDs.
+#
+# https://tracker.ceph.com/issues/37378

I noticed that the bug in above issue link has been fixed, it'd be good to reference the fix commit as well, so people will know where the fix is when test fails.

Thanks,
Eryu

Quoted 82 lines
tests/ceph/002 new file +61 -0
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+}
+
+# get standard environment
+. common/rc
+. common/filter
+. common/attr
+
+# real QA test starts here
+_supported_fs ceph
+
+_require_xfs_io_command "copy_range"
+_require_attrs
+_require_test
+
+workdir=$TEST_DIR/test-$seq
+rm -rf $workdir
+mkdir $workdir
+rm -f $seqres.full
+
+# Use 4M object size
+objsz=4194304
+file="$workdir/file-$objsz"
+dest="$workdir/dest-$objsz"
+touch $file $dest
+
+# object_size has to be a multiple of stripe_unit
+$SETFATTR_PROG -n ceph.file.layout \
+	-v "stripe_unit=$objsz stripe_count=1 object_size=$objsz" \
+	$file $dest
+
+# Create a 3 objects size files
+$XFS_IO_PROG -c "pwrite -S 0x61 0 $objsz" $file >> $seqres.full 2>&1
+$XFS_IO_PROG -c "pwrite -S 0x62 $objsz $objsz" $file >> $seqres.full 2>&1
+$XFS_IO_PROG -c "pwrite -S 0x63 $(($objsz * 2)) $objsz" $file >> $seqres.full 2>&1
+
+$XFS_IO_PROG -c "pwrite -S 0x64 0 $(($objsz * 3))" $dest >> $seqres.full 2>&1
+# Truncate the destination file (messing up with the truncate_seq)
+$XFS_IO_PROG -c "truncate 0" $dest >> $seqres.full 2>&1
+
+# copy the whole file over
+$XFS_IO_PROG -c "copy_range -s 0 -d 0 -l $(($objsz * 3)) $file" "$dest"
+
+hexdump $dest
+
+#success, all done
+status=0
+exit
tests/ceph/002.out new file +8 -0
new file mode 100644
index 000000000000..6f067250afff
@@ -0,0 +1,8 @@
+QA output created by 002
+0000000 6161 6161 6161 6161 6161 6161 6161 6161
+*
+0400000 6262 6262 6262 6262 6262 6262 6262 6262
+*
+0800000 6363 6363 6363 6363 6363 6363 6363 6363
+*
+0c00000
tests/ceph/group +1 -0
index 11f0b9ad03d3..c28fe473c1a4 100644
@@ -1 +1,2 @@
 001 auto quick copy
+002 auto quick copy
#5 lore raw

PATCH 3/3 ceph: test copy_file_range with infile = outfile

In reply to Luis Henriques

From: Luis Henriques <[email protected]>

This test runs a set of simple checks where the infile file is the same as the outfile.

---
 tests/ceph/003     | 118 +++++++++++++++++++++++++++++++++++++++++++++
 tests/ceph/003.out |  11 +++++
 tests/ceph/group   |   1 +
 3 files changed, 130 insertions(+)
 create mode 100644 tests/ceph/003
 create mode 100644 tests/ceph/003.out
tests/ceph/003 new file +118 -0
new file mode 100644
index 000000000000..6fc5e6812545
@@ -0,0 +1,118 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2020 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test No. ceph/005
+#
+# Test copy_file_range with infile = outfile
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+}
+
+# get standard environment
+. common/rc
+. common/filter
+. common/attr
+. common/reflink
+
+# real QA test starts here
+_supported_fs ceph
+
+_require_xfs_io_command "copy_range"
+_require_attrs
+_require_test
+
+workdir=$TEST_DIR/test-$seq
+rm -rf $workdir
+mkdir $workdir
+rm -f $seqres.full
+
+check_range()
+{
+	local file=$1
+	local off0=$2
+	local off1=$3
+	local val=$4
+	_read_range $file $off0 $off1 | grep -v -q $val
+	[ $? -eq 0 ] && echo "file $file is not '$val' in [ $off0 $off1 ]"
+}
+
+objsz=4194304
+halfobj=$(($objsz / 2))
+file="$workdir/file-$objsz"
+copy="$workdir/copy-$objsz"
+dest="$workdir/dest-$objsz"
+backup="$file.backup"
+touch $file $backup
+
+# object_size has to be a multiple of stripe_unit
+$SETFATTR_PROG -n ceph.file.layout \
+	       -v "stripe_unit=$objsz stripe_count=1 object_size=$objsz" \
+	       $file $backup
+
+$XFS_IO_PROG -c "pwrite -S 0x61 0 $objsz" $file >> $seqres.full 2>&1
+$XFS_IO_PROG -c "pwrite -S 0x62 $objsz $objsz" $file >> $seqres.full 2>&1
+$XFS_IO_PROG -c "pwrite -S 0x63 $(($objsz * 2)) $objsz" $file >> $seqres.full 2>&1
+
+cp $file $backup
+
+echo "  Copy single object to the end:"
+echo "    aaaa|bbbb|cccc => aaaa|bbbb|aaaa"
+$XFS_IO_PROG -c "copy_range -s 0 -d $(($objsz * 2)) -l $objsz $file" "$file"
+check_range $file 0 $objsz 61
+check_range $file $objsz $objsz 62
+check_range $file $(($objsz * 2)) $objsz 61
+
+echo "  Copy single object to the beginning:"
+echo "    aaaa|bbbb|aaaa => bbbb|bbbb|aaaa"
+$XFS_IO_PROG -c "copy_range -s $objsz -d 0 -l $objsz $file" "$file"
+check_range $file 0 $(($objsz * 2)) 62
+check_range $file $(($objsz * 2)) $objsz 61
+
+echo "  Copy single object to the middle:"
+echo "    bbbb|bbbb|aaaa => bbbb|aaaa|aaaa"
+$XFS_IO_PROG -c "copy_range -s $(($objsz * 2)) -d $objsz -l $objsz $file" "$file"
+check_range $file 0 $objsz 62
+check_range $file $objsz $(($objsz * 2)) 61
+
+cp $backup $file
+echo "  Cross object boundary (no full object copy)"
+echo "    aaaa|bbbb|cccc => aaaa|bbaa|aacc"
+$XFS_IO_PROG -c "copy_range -s 0 -d $(($objsz + $halfobj)) -l $objsz $file" "$file"
+check_range $file 0 $objsz 61
+check_range $file $objsz $halfobj 62
+check_range $file $(($objsz + $halfobj)) $objsz 61
+check_range $file $(($objsz * 2 + $halfobj)) $halfobj 63
+
+cp $backup $file
+echo "    aaaa|bbbb|cccc => aaaa|bbaa|bbcc"
+$XFS_IO_PROG -c "copy_range -s $halfobj -d $(($objsz + $halfobj)) -l $objsz $file" "$file"
+check_range $file 0 $objsz 61
+check_range $file $objsz $halfobj 62
+check_range $file $(($objsz + $halfobj)) $halfobj 61
+check_range $file $(($objsz * 2)) $halfobj 62
+check_range $file $(($objsz * 2 + $halfobj)) $halfobj 63
+
+cp $backup $file
+echo "    aaaa|bbbb|cccc => aaaa|bbbb|aabb"
+$XFS_IO_PROG -c "copy_range -s $halfobj -d $(($objsz * 2)) -l $objsz $file" "$file"
+check_range $file 0 $objsz 61
+check_range $file $objsz $objsz 62
+check_range $file $(($objsz * 2)) $halfobj 61
+check_range $file $(($objsz * 2 + $halfobj)) $halfobj 62
+
+#success, all done
+status=0
+exit
tests/ceph/003.out new file +11 -0
new file mode 100644
index 000000000000..76c83b265253
@@ -0,0 +1,11 @@
+QA output created by 003
+  Copy single object to the end:
+    aaaa|bbbb|cccc => aaaa|bbbb|aaaa
+  Copy single object to the beginning:
+    aaaa|bbbb|aaaa => bbbb|bbbb|aaaa
+  Copy single object to the middle:
+    bbbb|bbbb|aaaa => bbbb|aaaa|aaaa
+  Cross object boundary (no full object copy)
+    aaaa|bbbb|cccc => aaaa|bbaa|aacc
+    aaaa|bbbb|cccc => aaaa|bbaa|bbcc
+    aaaa|bbbb|cccc => aaaa|bbbb|aabb
tests/ceph/group +1 -0
index c28fe473c1a4..adbf61547766 100644
@@ -1,2 +1,3 @@
 001 auto quick copy
 002 auto quick copy
+003 auto quick copy
#6 lore raw
In reply to Luis Henriques
Quoted 4 lines · Luis Henriques

This is my second attempt to have an initial set of ceph-specific tests merged into fstests. In this patchset I'm pushing a different set of tests, focusing on the copy_file_range testing, although I *do* plan to get back to the quota tests soon.

I have no knowledge about cephfs, I don't have a cephfs test env either, so I can only comment from fstests' perspect of view. It'd be great if other ceph folks could help review this patchset as well.

From fstests perspect of view, this patchset looks fine to me, just some minor comments go to individual patch.

Quoted 10 lines

This syscall has a few peculiarities in ceph as it is able to use remote object copies without the need to download/upload data from the OSDs. However, in order to take advantage of this remote copy, the copy ranges and sizes need to include at least one object. Thus, all the currently existing generic tests won't actually take advantage of this feature.

Let me know any comments/concerns about this patchset. Also note that currently, in order to enable copy_file_range in cephfs, the additional 'copyfrom' mount parameter is required. (Hopefully this additional param

I assume '_require_xfs_io_command "copy_range"' will _notrun the test if there's no 'copyfrom' mount option provided, and test won't fail.

Quoted 7 lines

may be dropped in the future.)

Luis Henriques (3):
ceph: add copy_file_range (remote copy operation) testing
ceph: test combination of copy_file_range with truncate
ceph: test copy_file_range with infile = outfile

Quoted 9 lines
 tests/ceph/001     | 233 +++++++++++++++++++++++++++++++++++++++++++++
 tests/ceph/001.out | 129 +++++++++++++++++++++++++
 tests/ceph/002     |  74 ++++++++++++++
 tests/ceph/002.out |   8 ++
 tests/ceph/003     | 118 +++++++++++++++++++++++
 tests/ceph/003.out |  11 +++
 tests/ceph/group   |   3 +
 7 files changed, 576 insertions(+)
 create mode 100644 tests/ceph/001

New test should have file mode 755, i.e. with x bit set. The 'new' script should have done this for you.

Thanks,
Eryu

Quoted 6 lines
 create mode 100644 tests/ceph/001.out
 create mode 100644 tests/ceph/002
 create mode 100644 tests/ceph/002.out
 create mode 100644 tests/ceph/003
 create mode 100644 tests/ceph/003.out
 create mode 100644 tests/ceph/group
#7 lore raw
Quoted 6 lines · Eryu Guan
Quoted 4 lines · Luis Henriques

This is my second attempt to have an initial set of ceph-specific tests merged into fstests. In this patchset I'm pushing a different set of tests, focusing on the copy_file_range testing, although I *do* plan to get back to the quota tests soon.

Quoted 6 lines

I have no knowledge about cephfs, I don't have a cephfs test env either, so I can only comment from fstests' perspect of view. It'd be great if other ceph folks could help review this patchset as well.

From fstests perspect of view, this patchset looks fine to me, just some minor comments go to individual patch.

Thank you Eryu, for taking some time to review these tests. Much appreciated. I've gone through all your comments and they all make sense to me. I'll send out v2 shortly, addressing your concerns.

Quoted 7 lines
Quoted 7 lines

This syscall has a few peculiarities in ceph as it is able to use remote object copies without the need to download/upload data from the OSDs. However, in order to take advantage of this remote copy, the copy ranges and sizes need to include at least one object. Thus, all the currently existing generic tests won't actually take advantage of this feature.

Quoted 6 lines
Quoted 3 lines

Let me know any comments/concerns about this patchset. Also note that currently, in order to enable copy_file_range in cephfs, the additional 'copyfrom' mount parameter is required. (Hopefully this additional param

I assume '_require_xfs_io_command "copy_range"' will _notrun the test if there's no 'copyfrom' mount option provided, and test won't fail.

So, my cover-letter text was a bit confusing and not very clear about this mount option. These tests will run just fine (and won't fail) even without the 'copyfrom' mount option. The difference is that the copy won't be done remotely on the OSDs but rather using a local read+write loop (i.e. with the generic VFS copy_file_range implementation). 'copyfrom' is required only to enable the usage of the OSDs 'COPY_FROM' operation. I'm update the cover-letter text in v2 to clarify this.

Cheers,

Signature · 1 line
Luis
Quoted 29 lines
Quoted 7 lines

may be dropped in the future.)

Luis Henriques (3):
ceph: add copy_file_range (remote copy operation) testing
ceph: test combination of copy_file_range with truncate
ceph: test copy_file_range with infile = outfile

Quoted 9 lines
 tests/ceph/001     | 233 +++++++++++++++++++++++++++++++++++++++++++++
 tests/ceph/001.out | 129 +++++++++++++++++++++++++
 tests/ceph/002     |  74 ++++++++++++++
 tests/ceph/002.out |   8 ++
 tests/ceph/003     | 118 +++++++++++++++++++++++
 tests/ceph/003.out |  11 +++
 tests/ceph/group   |   3 +
 7 files changed, 576 insertions(+)
 create mode 100644 tests/ceph/001

New test should have file mode 755, i.e. with x bit set. The 'new' script should have done this for you.

Thanks,
Eryu

Quoted 6 lines
 create mode 100644 tests/ceph/001.out
 create mode 100644 tests/ceph/002
 create mode 100644 tests/ceph/002.out
 create mode 100644 tests/ceph/003
 create mode 100644 tests/ceph/003.out
 create mode 100644 tests/ceph/group