Skip to contentSkip to conversation

overlay/066: copy-up test for variant sparse files

PATCH Superseded Chengguang Xu 2 participants 2 messages Oct 24, 2019

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 overlay/066: copy-up test for variant sparse files

This is intensive copy-up test for sparse files, these cases will be mainly used for regression test of copy-up improvement for sparse files.

---
v1->v2:
- Call _get_block_size to get fs block size.
- Add comment for test space requirement.
- Print meaningful error message when copy-up fail.
- Adjust random hole range to 1M~5M.
- Fix typo.

 tests/overlay/066     | 120 ++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/066.out |   2 +
 tests/overlay/group   |   1 +
 3 files changed, 123 insertions(+)
 create mode 100755 tests/overlay/066
 create mode 100644 tests/overlay/066.out
tests/overlay/066 new file +120 -0
new file mode 100755
index 00000000..b01fc2a4
@@ -0,0 +1,120 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2019 Chengguang Xu <[email protected]>
+# All Rights Reserved.
+#
+# FS QA Test 066
+#
+# Test overlayfs copy-up function for variant sparse files.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs generic
+_supported_os Linux
+_require_test
+_require_scratch
+
+# Remove all files from previous tests
+_scratch_mkfs
+# We have totally 14 test files in this test,
+# one file for 100M and 13 files for 10M.
+_require_fs_space $OVL_BASE_SCRATCH_MNT $((10*1024*13 + 100*1024*1))
+
+lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER
+upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER
+testfile="copyup_sparse_test"
+mkdir -p $lowerdir
+
+# Create a completely empty hole file.
+$XFS_IO_PROG -fc "truncate 10M" "${lowerdir}/${testfile}_empty_holefile" \
+		 >>$seqres.full
+
+iosize=$(_get_block_size "${lowerdir}")
+if [ $iosize -le 1024 ]; then
+	iosize=1
+else
+	iosize=`expr $iosize / 1024`
+fi
+
+# Create test files with different hole size patterns.
+while [ $iosize -le 2048 ]; do
+	pos=$iosize
+	$XFS_IO_PROG -fc "truncate 10M" \
+		"${lowerdir}/${testfile}_iosize${iosize}K_holefile" >>$seqres.full
+	while [ $pos -lt 8192 ]; do
+		$XFS_IO_PROG -fc "pwrite ${pos}K ${iosize}K" \
+		"${lowerdir}/${testfile}_iosize${iosize}K_holefile" >>$seqres.full
+		pos=`expr $pos + $iosize + $iosize`
+	done
+	iosize=`expr $iosize + $iosize`
+done
+
+# Create test file with many random holes(1M~5M).
+$XFS_IO_PROG -fc "truncate 100M" "${lowerdir}/${testfile}_random_holefile" \
+		>>$seqres.full
+pos=2048
+while [ $pos -le 81920 ]; do
+	iosize=`expr $RANDOM % 5120`
+	if [ $iosize -lt 1024 ]; then
+		iosize=`expr $iosize + 1024`
+	fi
+	$XFS_IO_PROG -fc "pwrite ${pos}K ${iosize}K" \
+		"${lowerdir}/${testfile}_random_holefile" >>$seqres.full
+	pos=`expr $pos + $iosize + $iosize`
+done
+
+_scratch_mount
+
+# Open the files should succeed, no errors are expected.
+for f in $SCRATCH_MNT/*; do
+	$XFS_IO_PROG -c "open" $f >>$seqres.full
+done
+
+echo "Silence is golden"
+
+# Check all copy-up files in upper layer.
+iosize=$(_get_block_size "${lowerdir}")
+if [ $iosize -le 1024 ]; then
+	iosize=1
+else
+	iosize=`expr $iosize / 1024`
+fi
+
+while [ $iosize -le 2048 ]; do
+	diff "${lowerdir}/${testfile}_iosize${iosize}K_holefile" \
+		"${upperdir}/${testfile}_iosize${iosize}K_holefile" >>$seqres.full ||\
+		echo "${upperdir}/${testfile}_iosize${iosize}K_holefile" copy up failed!
+	iosize=`expr $iosize + $iosize`
+done
+
+diff "${lowerdir}/${testfile}_empty_holefile"  "${upperdir}/${testfile}_empty_holefile"  \
+	>>$seqres.full || echo "${upperdir}/${testfile}_empty_holefile" copy up failed!
+diff "${lowerdir}/${testfile}_random_holefile" "${upperdir}/${testfile}_random_holefile" \
+	>>$seqres.full || echo "${upperdir}/${testfile}_random_holefile" copy up failed!
+
+# success, all done
+status=0
+exit
tests/overlay/066.out new file +2 -0
new file mode 100644
index 00000000..b60cc24c
@@ -0,0 +1,2 @@
+QA output created by 066
+Silence is golden
tests/overlay/group +1 -0
index ef8517a1..1dec7db9 100644
@@ -68,3 +68,4 @@
 063 auto quick whiteout
 064 auto quick copyup
 065 auto quick mount
+066 auto quick copyup
Signature · 1 line
2.20.1
Reviewed-by on v2 1/1
#1 lore raw
Quoted 6 lines · Chengguang Xu

This is intensive copy-up test for sparse files, these cases will be mainly used for regression test of copy-up improvement for sparse files.

You can add:

after fixing nits below...

Quoted 54 lines

---
v1->v2:
- Call _get_block_size to get fs block size.
- Add comment for test space requirement.
- Print meaningful error message when copy-up fail.
- Adjust random hole range to 1M~5M.
- Fix typo.

 tests/overlay/066     | 120 ++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/066.out |   2 +
 tests/overlay/group   |   1 +
 3 files changed, 123 insertions(+)
 create mode 100755 tests/overlay/066
 create mode 100644 tests/overlay/066.out
tests/overlay/066 new file +33 -0
new file mode 100755
index 00000000..b01fc2a4
@@ -0,0 +1,120 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2019 Chengguang Xu <[email protected]>
+# All Rights Reserved.
+#
+# FS QA Test 066
+#
+# Test overlayfs copy-up function for variant sparse files.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1       # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+       cd /
+       rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+

tests/overlay/066 line 43

Quoted 10 lines
tests/overlay/066 new file +10 -0
+# Modify as appropriate.
+_supported_fs generic
+_supported_os Linux
+_require_test
+_require_scratch
+
+# Remove all files from previous tests
+_scratch_mkfs
+# We have totally 14 test files in this test,
+# one file for 100M and 13 files for 10M.

So we need double that amount of space if both upper and lower do not support holes.

Also it is not obvious to the reader how the number 13 came to be. It is better to explicitly say 1 empty file + 2^0 .. 2^11 hole size file, so if ever the test pattern changes, it will be easier to adapt this formula.

Quoted 9 lines
tests/overlay/066 new file +9 -0
+_require_fs_space $OVL_BASE_SCRATCH_MNT $((10*1024*13 + 100*1024*1))
+
+lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER
+upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER
+testfile="copyup_sparse_test"
+mkdir -p $lowerdir
+
+# Create a completely empty hole file.
+$XFS_IO_PROG -fc "truncate 10M" "${lowerdir}/${testfile}_empty_holefile" \

tests/overlay/066 line 62

Quoted 10 lines
tests/overlay/066 new file +10 -0
+                >>$seqres.full
+
+iosize=$(_get_block_size "${lowerdir}")
+if [ $iosize -le 1024 ]; then
+       iosize=1
+else
+       iosize=`expr $iosize / 1024`
+fi
+
+# Create test files with different hole size patterns.

Better be more verbose in the comment about the pattern of the files, so reader won't need to follow code to understand. Also better to have constants to make the algorithm clearer:

max_iosize=2048
file_size=10240
max_pos=expr $file_size - $max_iosize

Quoted 5 lines
tests/overlay/066 new file +5 -0
+while [ $iosize -le 2048 ]; do
+       pos=$iosize
+       $XFS_IO_PROG -fc "truncate 10M" \
+               "${lowerdir}/${testfile}_iosize${iosize}K_holefile" >>$seqres.full
+       while [ $pos -lt 8192 ]; do

tests/overlay/066 line 77

Quoted 10 lines
tests/overlay/066 new file +10 -0
+               $XFS_IO_PROG -fc "pwrite ${pos}K ${iosize}K" \
+               "${lowerdir}/${testfile}_iosize${iosize}K_holefile" >>$seqres.full
+               pos=`expr $pos + $iosize + $iosize`
+       done
+       iosize=`expr $iosize + $iosize`
+done
+
+# Create test file with many random holes(1M~5M).
+$XFS_IO_PROG -fc "truncate 100M" "${lowerdir}/${testfile}_random_holefile" \
+               >>$seqres.full

Same comment about using constants instead of unexplained numbers, including:
min_hole=1024
max_hole=5120

tests/overlay/066 line 83

Quoted 6 lines
tests/overlay/066 new file +6 -0
+pos=2048
+while [ $pos -le 81920 ]; do
+       iosize=`expr $RANDOM % 5120`
+       if [ $iosize -lt 1024 ]; then
+               iosize=`expr $iosize + 1024`
+       fi

So that is a weird way to express random 1MB~5MB
it results in 2 times higher probability for holes in the range 1MB~2MB and I don't think that was intentional. Better use:

   iosize=$(($RANDOM % ($max_hole - $min_hole) + $min_hole))

Thanks,
Amir.