ext4/030: Ext4 online resize tests.

PATCH Superseded harshads 4 participants 17 messages Nov 1, 2017 – Jan 25, 2018

Share & tools

harshads

author

2017-11-01 00:12 UTC #0 lore raw

PATCH 1/1 ext4/030: Ext4 online resize tests.

Add tests for Ext4 online resize feature. Once online resizing with bigalloc support is merged into Ext4, these tests can be used to sanitize the working of that feature. In order to test online resizing with bigalloc, set MKFS_OPTIONS="-O bigalloc -C <cluster_size>" and run the test.

Signed-off-by Harshad Shirwadkar <[email protected]>

---
 tests/ext4/030     | 93 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 21 ++++++++++++
 tests/ext4/group   |  1 +
 3 files changed, 115 insertions(+)
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out

tests/ext4/030 new file +93 -0

new file mode 100755
index 00000000..c958e10c
@@ -0,0 +1,93 @@
+#! /bin/bash
+# FS QA Test ext4/030
+#
+# Ext4 online resize test.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Google, Inc.  All Rights Reserved.
+#
+# Author: Harshad Shirwadkar <[email protected]>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment and checks
+. ./common/rc
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs ext4
+_supported_os Linux
+
+_require_scratch
+rm -f $seqres.full
+TESTDIR="${SCRATCH_MNT}/scratchdir"
+TESTFILE="${TESTDIR}/testfile"
+
+mb2bytes()
+{
+	expr $1 \* 1024 \* 1024
+}
+
+# Takes two arguments: $1 = initial size (mb), $2 = new size (mb).
+work()
+{
+	local original_size=$1
+	local final_size=$2
+
+	echo "+ create scratch fs of size $original_size MB"
+	_scratch_mkfs_sized $(mb2bytes $original_size) 4096 > /dev/null 2>&1
+	dumpe2fs -g "${SCRATCH_DEV}" > /dev/null 2>&1 || \
+		_notrun "dumpe2fs -g not supported"
+
+	echo "+ mount scratch fs"
+	_scratch_mount > /dev/null 2>&1
+
+	echo "+ resize fs to $final_size MB"
+	resize2fs -f "${SCRATCH_DEV}" ${final_size}M >> $seqres.full 2>&1 || \
+		_fail "resize2fs failed"
+
+	echo "+ umount fs"
+	umount "${SCRATCH_MNT}"
+
+	echo "+ check fs"
+	e2fsck -fn "${SCRATCH_DEV}" >> $seqres.full 2>&1 || \
+		_fail "fsck should not fail"
+}
+
+## Try resizing to different sizes.
+work 128 1024
+work 1024 2048
+work 1024 4096
+work 1024 5120
+
+status=0
+exit

tests/ext4/030.out new file +21 -0

new file mode 100644
index 00000000..14c24cc5
@@ -0,0 +1,21 @@
+QA output created by 030
++ create scratch fs of size 128 MB
++ mount scratch fs
++ resize fs to 1024 MB
++ umount fs
++ check fs
++ create scratch fs of size 1024 MB
++ mount scratch fs
++ resize fs to 2048 MB
++ umount fs
++ check fs
++ create scratch fs of size 1024 MB
++ mount scratch fs
++ resize fs to 4096 MB
++ umount fs
++ check fs
++ create scratch fs of size 1024 MB
++ mount scratch fs
++ resize fs to 5120 MB
++ umount fs
++ check fs

tests/ext4/group +1 -0

index 257bb646..f2beeaf8 100644
@@ -32,6 +32,7 @@
 027 auto quick fsmap
 028 auto quick fsmap
 029 auto quick fsmap
+030 auto ioctl resize
 271 auto rw quick
 301 aio auto ioctl rw stress defrag
 302 aio auto ioctl rw stress defrag

Signature · 1 line

2.15.0.403.gc27cc4dac6-goog

Dave Chinner

2017-11-01 01:16 UTC #1 lore raw

Quoted 7 lines · harshads

Add tests for Ext4 online resize feature. Once online resizing with bigalloc support is merged into Ext4, these tests can be used to sanitize the working of that feature. In order to test online resizing with bigalloc, set MKFS_OPTIONS="-O bigalloc -C <cluster_size>" and run the test.
Signed-off-by Harshad Shirwadkar <[email protected]>

You should do this on a loop device, that way you can test much bigger sizes and all the critical numbers around 32 bit overflows in the storage stack (2T, 4T, 8T, 16T, etc). e.g. see xfs/078.

Also, you should add a test that does grow at ENOSPC will exercising the fs with fstress. See xfs/104.

Perhaps some of the XFS specific tests in the growfs group could be turned into generic tests with the right infrastructure, so you get more coverage on ext4 without a whole lot of extra work...

Cheers,

Dave.

Signature · 2 lines

Dave Chinner
[email protected]

harshads

author

2017-12-05 03:44 UTC #2 lore raw

PATCH v2 1/1 ext4/030: Ext4 online resize tests.

In reply to harshads

Add tests for Ext4 online resize feature.

Signed-off-by Harshad Shirwadkar <[email protected]>

---
 common/ext4        |  42 +++++++++++++++
 tests/ext4/030     | 101 ++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 148 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/group   |   1 +
 4 files changed, 292 insertions(+)
 create mode 100644 common/ext4
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out

common/ext4 new file +42 -0

new file mode 100644
index 00000000..71966bef
@@ -0,0 +1,42 @@
+#
+# Common ext4 functions
+#
+
+_ext4_online_resize()
+{
+	local image_file=$1
+	local image_mount=$2
+	local original_size=$3
+	local final_size=$4
+	local cluster_size=$5
+
+	echo "+ truncate image file to ${final_size}"
+	# If 'final_size' is not suffixed (like 24576), then the
+	# caller assumes that it is the number of blocks.
+	if [[ "${final_size: -1}" == [a-zA-Z] ]]; then
+		truncate ${image_file} -s ${final_size}
+	else
+		truncate ${image_file} -s ${final_size} -o 4096
+	fi
+	echo "+ create fs on image file ${original_size}"
+	${MKFS_PROG}.${FSTYP} -F -O bigalloc,resize_inode,metadata_csum \
+		-C ${cluster_size} -b 4096 \
+		${image_file} ${original_size} > /dev/null 2>&1
+
+	dumpe2fs -g ${image_file} > /dev/null 2>&1 || \
+		_notrun "dumpe2fs -g not supported"
+
+	echo "+ mount image file"
+	mount -t ${FSTYP} ${image_file} ${image_mount} > /dev/null 2>&1
+
+	echo "+ resize fs to ${final_size}"
+	resize2fs -f ${image_file} ${final_size} >> $seqres.full 2>&1 || \
+		_fail "resize2fs failed"
+
+	echo "+ umount fs"
+	umount ${image_mount}
+
+	echo "+ check fs"
+	e2fsck -fn ${image_file} >> $seqres.full 2>&1 || \
+	    _fail "fsck should not fail"
+}

tests/ext4/030 new file +101 -0

101 lines not on this page · raw message

tests/ext4/030.out new file +148 -0

148 lines not on this page · raw message

tests/ext4/group +1 -0

index 257bb646..f29d3de6 100644
@@ -32,6 +32,7 @@
 027 auto quick fsmap
 028 auto quick fsmap
 029 auto quick fsmap
+030 auto quick ioctl resize
 271 auto rw quick
 301 aio auto ioctl rw stress defrag
 302 aio auto ioctl rw stress defrag

Signature · 1 line

2.15.0.531.g2ccb3012c9-goog

Eryu Guan

2017-12-05 04:42 UTC #3 lore raw

Quoted 1 line · harshads

Add tests for Ext4 online resize feature.

The commit log describes the test as a generic ext4 online resize test, but the actual code is really testing online resize with bigalloc. So please either update the commit log or make the test generic enough.

Quoted 18 lines

Signed-off-by Harshad Shirwadkar <[email protected]>

---
 common/ext4        |  42 +++++++++++++++
 tests/ext4/030     | 101 ++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 148 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/group   |   1 +
 4 files changed, 292 insertions(+)
 create mode 100644 common/ext4
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out

common/ext4 new file +0 -0

new file mode 100644
index 00000000..71966bef

common/ext4 line 5

Quoted 5 lines

common/ext4 new file +5 -0

@@ -0,0 +1,42 @@
+#
+# Common ext4 functions
+#
+
+_ext4_online_resize()

With the hardcoded mkfs option "-O bigalloc,resize_inode,metadata_csum", I don't think this is a function that can be shared with other tests easily. Maybe it should be moved to the test itself.

common/ext4 line 17

Quoted 12 lines

common/ext4 new file +12 -0

+{
+	local image_file=$1
+	local image_mount=$2
+	local original_size=$3
+	local final_size=$4
+	local cluster_size=$5
+
+	echo "+ truncate image file to ${final_size}"
+	# If 'final_size' is not suffixed (like 24576), then the
+	# caller assumes that it is the number of blocks.
+	if [[ "${final_size: -1}" == [a-zA-Z] ]]; then
+		truncate ${image_file} -s ${final_size}

Use xfs_io to do truncate, truncate(1) is not always available (very old distros don't have it), though that's unlikely to happen these days, why not using xfs_io when we have better choice :)

	$XFS_IO_PROG -c "truncate <size>" ${image_file}

common/ext4 line 24

Quoted 7 lines

common/ext4 new file +7 -0

+	else
+		truncate ${image_file} -s ${final_size} -o 4096
+	fi
+	echo "+ create fs on image file ${original_size}"
+	${MKFS_PROG}.${FSTYP} -F -O bigalloc,resize_inode,metadata_csum \
+		-C ${cluster_size} -b 4096 \
+		${image_file} ${original_size} > /dev/null 2>&1

We should make sure the userspace tool and kernel support thease features first. Currently we have a helper function _require_ext4_mkfs_feature() that checks if a given feature is supported by e2fsprogs, but it doesn't check if kernel knows the feature too. Ted is working a rework of that helper, but that's not finalized & merged yet.

https://www.spinics.net/lists/fstests/msg07880.html

The new helper is _require_scratch_ext4_feature, which is creating new filesystem with the given feature on $SCRATCH_DEV and try to mount it, and _notrun if any step fails.

Perhaps we could make it work on sparse fs image not on $SCRATCH_DEV, so that would be quicker and address Ted's concern too (mkfs twice on $SCRATCH_DEV, once in the _require rule, once in the test).

Assuming the new helper is "_require_ext4_feature", then you can add the following rules in your test

_require_ext4_feature "bigalloc,resize_inode,metadata_csum"

common/ext4 line 27

Quoted 3 lines

common/ext4 new file +3 -0

+
+	dumpe2fs -g ${image_file} > /dev/null 2>&1 || \
+		_notrun "dumpe2fs -g not supported"

Use $DUMPE2FS_PROG, and this should in the test itself, not a worker function. Maybe this could be made into a new _require rule too.

common/ext4 line 30

Quoted 3 lines

common/ext4 new file +3 -0

+
+	echo "+ mount image file"
+	mount -t ${FSTYP} ${image_file} ${image_mount} > /dev/null 2>&1

$MOUNT_PROG

common/ext4 line 37

Quoted 7 lines

common/ext4 new file +7 -0

+
+	echo "+ resize fs to ${final_size}"
+	resize2fs -f ${image_file} ${final_size} >> $seqres.full 2>&1 || \
+		_fail "resize2fs failed"
+
+	echo "+ umount fs"
+	umount ${image_mount}

$UMOUNT_PROG

common/ext4 line 41

Quoted 4 lines

common/ext4 new file +4 -0

+
+	echo "+ check fs"
+	e2fsck -fn ${image_file} >> $seqres.full 2>&1 || \
+	    _fail "fsck should not fail"

Assign a loop device to the fs image and call _check_generic_filesystem with the device.

tests/ext4/030 line 2

Quoted 9 lines

common/ext4 new file +1 -0

+}

tests/ext4/030 new file +2 -0

new file mode 100755
index 00000000..2415fb65
@@ -0,0 +1,101 @@
+#! /bin/bash
+# FS QA Test ext4/031

                     ^^^ 030? 031?

tests/ext4/030 line 4

Quoted 2 lines

tests/ext4/030 new file +2 -0

+#
+# Ext4 online resize tests with small and crucial resizes.

Same here, test description needs to match the actual test.

Quoted 30 lines

tests/ext4/030 new file +30 -0

+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Google, Inc.  All Rights Reserved.
+#
+# Author: Harshad Shirwadkar <[email protected]>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()

tests/ext4/030 line 44

Quoted 10 lines

tests/ext4/030 new file +10 -0

+{
+	cd /
+	rm -f $tmp.*
+	umount ${IMG_MNT} > /dev/null 2>&1
+	rm -f ${IMG_FILE} > /dev/null 2>&1
+}
+
+# get standard environment and checks
+. ./common/rc
+. ./common/ext4

This should be done in common/rc (e.g. how common/xfs is sourced), not in individual test. But if we move the resize test worker into the test itself, common/ext4 is not needed.

Quoted 10 lines

tests/ext4/030 new file +10 -0

+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs ext4
+_supported_os Linux
+
+_require_loop
+

tests/ext4/030 line 64

Quoted 10 lines

tests/ext4/030 new file +10 -0

+IMG_FILE=$TEST_DIR/$seq.fs
+IMG_MNT=$TEST_DIR/$seq.mnt
+
+rm -f $seqres.full
+
+rmdir $IMG_MNT 2>/dev/null
+mkdir -p $IMG_MNT || _fail "cannot create loopback mount point"
+
+for cluster_size in 4096 16384 65536
+do

We prefer this for-do-done format in fstests :)

for ...; do

	<work>

done

tests/ext4/030 line 69

Quoted 5 lines

tests/ext4/030 new file +5 -0

+	blocks_per_cluster=`expr $cluster_size / 4096`
+	echo "+ set cluster size to ${cluster_size}"
+	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+		$((16384 * $blocks_per_cluster)) \
+		$((24576 * $blocks_per_cluster)) ${cluster_size}

These size numbers are a bit hard to read, add human readable comments on these numbers?

Thanks,
Eryu

Quoted 205 lines

tests/ext4/030 new file +32 -0

+
+	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+		$((24576 * $blocks_per_cluster)) \
+		$((32767 * $blocks_per_cluster)) ${cluster_size}
+
+	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+		$((24576 * $blocks_per_cluster)) \
+		$((32768 * $blocks_per_cluster)) ${cluster_size}
+
+	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+		$((24576 * $blocks_per_cluster)) \
+		$((65536 * $blocks_per_cluster)) ${cluster_size}
+
+	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+		$((24576 * $blocks_per_cluster)) \
+		$((491521 * $blocks_per_cluster)) ${cluster_size}
+
+	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+		$((24576 * $blocks_per_cluster)) \
+		$((507904 * $blocks_per_cluster)) ${cluster_size}
+
+	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+		$((24576 * $blocks_per_cluster)) \
+		$((527488 * $blocks_per_cluster)) ${cluster_size}
+
+	_ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+		$((24576 * $blocks_per_cluster)) \
+		$((5274880 * $blocks_per_cluster)) ${cluster_size}
+done
+
+status=0
+exit

tests/ext4/030.out new file +148 -0

new file mode 100644
index 00000000..5f6441e6
@@ -0,0 +1,148 @@
+QA output created by 030
++ set cluster size to 4096
++ truncate image file to 24576
++ create fs on image file 16384
++ mount image file
++ resize fs to 24576
++ umount fs
++ check fs
++ truncate image file to 32767
++ create fs on image file 24576
++ mount image file
++ resize fs to 32767
++ umount fs
++ check fs
++ truncate image file to 32768
++ create fs on image file 24576
++ mount image file
++ resize fs to 32768
++ umount fs
++ check fs
++ truncate image file to 65536
++ create fs on image file 24576
++ mount image file
++ resize fs to 65536
++ umount fs
++ check fs
++ truncate image file to 491521
++ create fs on image file 24576
++ mount image file
++ resize fs to 491521
++ umount fs
++ check fs
++ truncate image file to 507904
++ create fs on image file 24576
++ mount image file
++ resize fs to 507904
++ umount fs
++ check fs
++ truncate image file to 527488
++ create fs on image file 24576
++ mount image file
++ resize fs to 527488
++ umount fs
++ check fs
++ truncate image file to 5274880
++ create fs on image file 24576
++ mount image file
++ resize fs to 5274880
++ umount fs
++ check fs
++ set cluster size to 16384
++ truncate image file to 98304
++ create fs on image file 65536
++ mount image file
++ resize fs to 98304
++ umount fs
++ check fs
++ truncate image file to 131068
++ create fs on image file 98304
++ mount image file
++ resize fs to 131068
++ umount fs
++ check fs
++ truncate image file to 131072
++ create fs on image file 98304
++ mount image file
++ resize fs to 131072
++ umount fs
++ check fs
++ truncate image file to 262144
++ create fs on image file 98304
++ mount image file
++ resize fs to 262144
++ umount fs
++ check fs
++ truncate image file to 1966084
++ create fs on image file 98304
++ mount image file
++ resize fs to 1966084
++ umount fs
++ check fs
++ truncate image file to 2031616
++ create fs on image file 98304
++ mount image file
++ resize fs to 2031616
++ umount fs
++ check fs
++ truncate image file to 2109952
++ create fs on image file 98304
++ mount image file
++ resize fs to 2109952
++ umount fs
++ check fs
++ truncate image file to 21099520
++ create fs on image file 98304
++ mount image file
++ resize fs to 21099520
++ umount fs
++ check fs
++ set cluster size to 65536
++ truncate image file to 393216
++ create fs on image file 262144
++ mount image file
++ resize fs to 393216
++ umount fs
++ check fs
++ truncate image file to 524272
++ create fs on image file 393216
++ mount image file
++ resize fs to 524272
++ umount fs
++ check fs
++ truncate image file to 524288
++ create fs on image file 393216
++ mount image file
++ resize fs to 524288
++ umount fs
++ check fs
++ truncate image file to 1048576
++ create fs on image file 393216
++ mount image file
++ resize fs to 1048576
++ umount fs
++ check fs
++ truncate image file to 7864336
++ create fs on image file 393216
++ mount image file
++ resize fs to 7864336
++ umount fs
++ check fs
++ truncate image file to 8126464
++ create fs on image file 393216
++ mount image file
++ resize fs to 8126464
++ umount fs
++ check fs
++ truncate image file to 8439808
++ create fs on image file 393216
++ mount image file
++ resize fs to 8439808
++ umount fs
++ check fs
++ truncate image file to 84398080
++ create fs on image file 393216
++ mount image file
++ resize fs to 84398080
++ umount fs
++ check fs

tests/ext4/group +1 -0

index 257bb646..f29d3de6 100644
@@ -32,6 +32,7 @@
 027 auto quick fsmap
 028 auto quick fsmap
 029 auto quick fsmap
+030 auto quick ioctl resize
 271 auto rw quick
 301 aio auto ioctl rw stress defrag
 302 aio auto ioctl rw stress defrag

Signature · 6 lines

2.15.0.531.g2ccb3012c9-goog

--
To unsubscribe from this list: send the line "unsubscribe fstests" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

harshads

author

2018-01-04 01:18 UTC #4 lore raw

PATCH v3 1/1 ext4/030: Ext4 online resize with bigalloc tests.

In reply to harshads

Add tests to verify Ext4 online resizing feature with bigalloc feature enabled. We test various resizing scenarios with different cluster sizes.

Signed-off-by Harshad Shirwadkar <[email protected]>

---
 common/rc          |  23 ++++++++
 tests/ext4/030     | 158 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 148 +++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/group   |   1 +
 4 files changed, 330 insertions(+)
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out

common/rc +23 -0

index 9216efdb..052dadae 100644
@@ -1845,6 +1845,29 @@ _require_scratch_ext4_feature()
     _scratch_unmount
 }
 
+# Check whether the specified feature whether it is supported by
+# mkfs.ext4 and the kernel by using a sparse file image.
+_require_ext4_feature()
+{
+    if [ -z "$1" ]; then
+        echo "Usage: _require_loop_ext4_feature feature"
+        exit 1
+    fi
+    local test_image="${TEST_DIR}/test_image"
+    local test_mnt="${TEST_DIR}/test_mnt"
+    rm -rf $test_image $test_mnt
+    mkdir -p $test_mnt || _fail "cannot create loopback mount point"
+    $XFS_IO_PROG -f -c "truncate 512m" $test_image >/dev/null 2>&1
+    $MKFS_EXT4_PROG -F $MKFS_OPTIONS -O "$1" $test_image 512m >/dev/null 2>&1 \
+	|| _notrun "mkfs.ext4 doesn't support $1 feature"
+    _mount -t $FSTYP `_common_dev_mount_options $test_image $test_mnt` \
+        >/dev/null 2>&1 \
+	|| _notrun "Kernel doesn't support the ext4 feature(s): $1"
+    $UMOUNT_PROG $test_mnt >/dev/null 2>&1
+    rm -f $test_image >/dev/null 2>&1
+    rmdir $test_mnt >/dev/null 2>&1
+}
+
 # this test requires that external log/realtime devices are not in use
 #
 _require_nonexternal()

tests/ext4/030 new file +158 -0

158 lines not on this page · raw message

tests/ext4/030.out new file +148 -0

148 lines not on this page · raw message

tests/ext4/group +1 -0

index 257bb646..f29d3de6 100644
@@ -32,6 +32,7 @@
 027 auto quick fsmap
 028 auto quick fsmap
 029 auto quick fsmap
+030 auto quick ioctl resize
 271 auto rw quick
 301 aio auto ioctl rw stress defrag
 302 aio auto ioctl rw stress defrag

Signature · 1 line

2.15.1.620.gb9897f4670-goog

Amir Goldstein

2018-01-04 08:09 UTC #5 lore raw

Re: [PATCH v3] ext4/030: Ext4 online resize with bigalloc tests.

Quoted 19 lines · harshads

Add tests to verify Ext4 online resizing feature with bigalloc feature enabled. We test various resizing scenarios with different cluster sizes.
Signed-off-by Harshad Shirwadkar <[email protected]>
---
 common/rc          |  23 ++++++++
 tests/ext4/030     | 158 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 148 +++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/group   |   1 +
 4 files changed, 330 insertions(+)
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out
common/rc +0 -0
index 9216efdb..052dadae 100644

common/rc _require_scratch_ext4_feature() · line 1850

Quoted 6 lines

common/rc +3 -0

@@ -1845,6 +1845,29 @@ _require_scratch_ext4_feature()
     _scratch_unmount
 }
 
+# Check whether the specified feature whether it is supported by
+# mkfs.ext4 and the kernel by using a sparse file image.
+_require_ext4_feature()

1. please explain why this loop variant is needed
2. it would be great if you could also change callers to _require_scratch_ext4_feature
to use _require_scratch_feature and plug
_require_scratch_ext4_feature in there
3. probably best to post this as a separate patch from the test itself

common/rc _require_scratch_ext4_feature() · line 1853

Quoted 3 lines

common/rc +3 -0

+{
+    if [ -z "$1" ]; then
+        echo "Usage: _require_loop_ext4_feature feature"

So which is it, _require_loop_ext4_feature or _require_ext4_feature? First one sounds better to me, given that you explain why the loop variant is needed.
If it is needed, will it be useful to have for other fs? Then better implement _require_loop_feature and make ext4 a specific case.

Cheers,
Amir.

Eryu Guan

2018-01-04 08:20 UTC #6 lore raw

Re: [PATCH v3] ext4/030: Ext4 online resize with bigalloc tests.

Quoted 23 lines · Amir Goldstein

Quoted 22 lines · harshads
Add tests to verify Ext4 online resizing feature with bigalloc feature enabled. We test various resizing scenarios with different cluster sizes.
Signed-off-by Harshad Shirwadkar <[email protected]>
---
 common/rc          |  23 ++++++++
 tests/ext4/030     | 158 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 148 +++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/group   |   1 +
 4 files changed, 330 insertions(+)
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out
common/rc +0 -0
index 9216efdb..052dadae 100644
@@ -1845,6 +1845,29 @@ _require_scratch_ext4_feature()
     _scratch_unmount
 }
 

Quoted 10 lines

common/rc _require_scratch_ext4_feature() · line 1850
Quoted 3 lines
common/rc +3 -0
+# Check whether the specified feature whether it is supported by
+# mkfs.ext4 and the kernel by using a sparse file image.
+_require_ext4_feature()
1. please explain why this loop variant is needed
2. it would be great if you could also change callers to _require_scratch_ext4_feature
to use _require_scratch_feature and plug
_require_scratch_ext4_feature in there
3. probably best to post this as a separate patch from the test itself

I haven't went through the whole patch yet, just want to point out that the _require_scratch_ext4_feature helper has been added in commit be341e36fd02 ("common: rework _require_ext4_mkfs_feature"), but not enabled in the more generic _require_scratch_feature helper yet.

It'd be good to plug the ext4 helper to the generic helper in a separate patch, as Amir suggested, and perhaps converting all existing callers of the ext4 helper to the generic helper.

Thanks,
Eryu

Quoted 14 lines

common/rc _require_scratch_ext4_feature() · line 1853
Quoted 4 lines
common/rc +3 -0
+{
+    if [ -z "$1" ]; then
+        echo "Usage: _require_loop_ext4_feature feature"
So which is it, _require_loop_ext4_feature or _require_ext4_feature? First one sounds better to me, given that you explain why the loop variant is needed.
If it is needed, will it be useful to have for other fs? Then better implement _require_loop_feature and make ext4 a specific case.
Cheers,
Amir.

Harshad Shirwadkar

author

2018-01-05 21:21 UTC #7 lore raw

Re: [PATCH v3] ext4/030: Ext4 online resize with bigalloc tests.

Quoted 24 lines · Eryu Guan

Quoted 23 lines · Amir Goldstein
Quoted 22 lines · harshads
Add tests to verify Ext4 online resizing feature with bigalloc feature enabled. We test various resizing scenarios with different cluster sizes.
Signed-off-by Harshad Shirwadkar <[email protected]>
---
 common/rc          |  23 ++++++++
 tests/ext4/030     | 158 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 148 +++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/group   |   1 +
 4 files changed, 330 insertions(+)
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out
common/rc +0 -0
index 9216efdb..052dadae 100644
@@ -1845,6 +1845,29 @@ _require_scratch_ext4_feature()
     _scratch_unmount
 }
 

Quoted 5 lines

Quoted 5 lines
common/rc _require_scratch_ext4_feature() · line 1850
Quoted 3 lines
common/rc +3 -0
+# Check whether the specified feature whether it is supported by
+# mkfs.ext4 and the kernel by using a sparse file image.
+_require_ext4_feature()
1. please explain why this loop variant is needed

As discussed on a previous thread, the loop variant will avoid the need to mkfs twice on scratch device (once in rule and once in test). Also, this particular test doesn't need scratch device at all. So, if I decide to use _require_scratch_ext4_feature, I will have to do _require_scratch just for the rule even though the actual test doesn't need scratch. So, having loop variant helps.

Quoted 4 lines

Quoted 4 lines
2. it would be great if you could also change callers to _require_scratch_ext4_feature
to use _require_scratch_feature and plug
_require_scratch_ext4_feature in there

Okay, that sounds good.

Quoted 1 line

Quoted 1 line
3. probably best to post this as a separate patch from the test itself

Okay.

Quoted 6 lines

I haven't went through the whole patch yet, just want to point out that the _require_scratch_ext4_feature helper has been added in commit be341e36fd02 ("common: rework _require_ext4_mkfs_feature"), but not enabled in the more generic _require_scratch_feature helper yet.

Quoted 18 lines

It'd be good to plug the ext4 helper to the generic helper in a separate patch, as Amir suggested, and perhaps converting all existing callers of the ext4 helper to the generic helper.
Thanks,
Eryu
Quoted 11 lines
common/rc _require_scratch_ext4_feature() · line 1853
Quoted 4 lines
common/rc +3 -0
+{
+    if [ -z "$1" ]; then
+        echo "Usage: _require_loop_ext4_feature feature"
So which is it, _require_loop_ext4_feature or _require_ext4_feature? First one sounds better to me, given that you explain why the loop variant is needed.
If it is needed, will it be useful to have for other fs? Then better implement _require_loop_feature and make ext4 a specific case.

Thanks for pointing this out. I agree that _require_loop_ext4_feature is better. Sure, I'll add a _require_loop_feature. Alright, I'll do _require_loop stuff in a different patch.

- Harshad.

Quoted 3 lines

Quoted 3 lines
Cheers,
Amir.

Eryu Guan

2018-01-07 15:31 UTC #8 lore raw

Re: [PATCH v3] ext4/030: Ext4 online resize with bigalloc tests.

Quoted 31 lines · Harshad Shirwadkar

Quoted 24 lines · Eryu Guan
Quoted 23 lines · Amir Goldstein
Quoted 22 lines · harshads
Add tests to verify Ext4 online resizing feature with bigalloc feature enabled. We test various resizing scenarios with different cluster sizes.
Signed-off-by Harshad Shirwadkar <[email protected]>
---
 common/rc          |  23 ++++++++
 tests/ext4/030     | 158 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 148 +++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/group   |   1 +
 4 files changed, 330 insertions(+)
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out
common/rc +0 -0
index 9216efdb..052dadae 100644
@@ -1845,6 +1845,29 @@ _require_scratch_ext4_feature()
     _scratch_unmount
 }
 
Quoted 5 lines
Quoted 5 lines
common/rc _require_scratch_ext4_feature() · line 1850
Quoted 3 lines
common/rc +3 -0
+# Check whether the specified feature whether it is supported by
+# mkfs.ext4 and the kernel by using a sparse file image.
+_require_ext4_feature()
1. please explain why this loop variant is needed

Quoted 6 lines

As discussed on a previous thread, the loop variant will avoid the need to mkfs twice on scratch device (once in rule and once in test). Also, this particular test doesn't need scratch device at all. So, if I decide to use _require_scratch_ext4_feature, I will have to do _require_scratch just for the rule even though the actual test doesn't need scratch. So, having loop variant helps.

Hmm, as the _require_scratch_ext4_feature helper is already there and already does mkfs twice (with a very small fs size, so the additional mkfs won't add much time), I think it'd be much simpler to just use _require_scratch and _require_scratch_feature and create all the test images on $SCRATCH_DEV, a loop device variant doesn't seem that necessary to me.

Thanks,
Eryu

Quoted 46 lines

Quoted 5 lines
Quoted 5 lines
2. it would be great if you could also change callers to _require_scratch_ext4_feature
to use _require_scratch_feature and plug
_require_scratch_ext4_feature in there
Okay, that sounds good.
Quoted 1 line
Quoted 1 line
3. probably best to post this as a separate patch from the test itself
Okay.
Quoted 6 lines
I haven't went through the whole patch yet, just want to point out that the _require_scratch_ext4_feature helper has been added in commit be341e36fd02 ("common: rework _require_ext4_mkfs_feature"), but not enabled in the more generic _require_scratch_feature helper yet.
Quoted 18 lines
It'd be good to plug the ext4 helper to the generic helper in a separate patch, as Amir suggested, and perhaps converting all existing callers of the ext4 helper to the generic helper.
Thanks,
Eryu
Quoted 11 lines
common/rc _require_scratch_ext4_feature() · line 1853
Quoted 4 lines
common/rc +3 -0
+{
+    if [ -z "$1" ]; then
+        echo "Usage: _require_loop_ext4_feature feature"
So which is it, _require_loop_ext4_feature or _require_ext4_feature? First one sounds better to me, given that you explain why the loop variant is needed.
If it is needed, will it be useful to have for other fs? Then better implement _require_loop_feature and make ext4 a specific case.
Thanks for pointing this out. I agree that _require_loop_ext4_feature is better. Sure, I'll add a _require_loop_feature. Alright, I'll do _require_loop stuff in a different patch.
- Harshad.
Quoted 3 lines
Quoted 3 lines
Cheers,
Amir.

Harshad Shirwadkar

author

2018-01-08 02:56 UTC #9 lore raw

Re: [PATCH v3] ext4/030: Ext4 online resize with bigalloc tests.

Okay, I will keep this patch clean and will just use _require_scratch_* functions. Let me submit another patch that does that cleaning up that Amir suggested.

Quoted 95 lines · Eryu Guan

Quoted 31 lines · Harshad Shirwadkar
Quoted 24 lines · Eryu Guan
Quoted 23 lines · Amir Goldstein
Quoted 22 lines · harshads
Add tests to verify Ext4 online resizing feature with bigalloc feature enabled. We test various resizing scenarios with different cluster sizes.
Signed-off-by Harshad Shirwadkar <[email protected]>
---
 common/rc          |  23 ++++++++
 tests/ext4/030     | 158 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 148 +++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/group   |   1 +
 4 files changed, 330 insertions(+)
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out
common/rc +0 -0
index 9216efdb..052dadae 100644
@@ -1845,6 +1845,29 @@ _require_scratch_ext4_feature()
     _scratch_unmount
 }
 
Quoted 5 lines
Quoted 5 lines
common/rc _require_scratch_ext4_feature() · line 1850
Quoted 3 lines
common/rc +3 -0
+# Check whether the specified feature whether it is supported by
+# mkfs.ext4 and the kernel by using a sparse file image.
+_require_ext4_feature()
1. please explain why this loop variant is needed
Quoted 6 lines
As discussed on a previous thread, the loop variant will avoid the need to mkfs twice on scratch device (once in rule and once in test). Also, this particular test doesn't need scratch device at all. So, if I decide to use _require_scratch_ext4_feature, I will have to do _require_scratch just for the rule even though the actual test doesn't need scratch. So, having loop variant helps.
Hmm, as the _require_scratch_ext4_feature helper is already there and already does mkfs twice (with a very small fs size, so the additional mkfs won't add much time), I think it'd be much simpler to just use _require_scratch and _require_scratch_feature and create all the test images on $SCRATCH_DEV, a loop device variant doesn't seem that necessary to me.
Thanks,
Eryu
Quoted 46 lines
Quoted 5 lines
Quoted 5 lines
2. it would be great if you could also change callers to _require_scratch_ext4_feature
to use _require_scratch_feature and plug
_require_scratch_ext4_feature in there
Okay, that sounds good.
Quoted 1 line
Quoted 1 line
3. probably best to post this as a separate patch from the test itself
Okay.
Quoted 6 lines
I haven't went through the whole patch yet, just want to point out that the _require_scratch_ext4_feature helper has been added in commit be341e36fd02 ("common: rework _require_ext4_mkfs_feature"), but not enabled in the more generic _require_scratch_feature helper yet.
Quoted 18 lines
It'd be good to plug the ext4 helper to the generic helper in a separate patch, as Amir suggested, and perhaps converting all existing callers of the ext4 helper to the generic helper.
Thanks,
Eryu
Quoted 11 lines
common/rc _require_scratch_ext4_feature() · line 1853
Quoted 4 lines
common/rc +3 -0
+{
+    if [ -z "$1" ]; then
+        echo "Usage: _require_loop_ext4_feature feature"
So which is it, _require_loop_ext4_feature or _require_ext4_feature? First one sounds better to me, given that you explain why the loop variant is needed.
If it is needed, will it be useful to have for other fs? Then better implement _require_loop_feature and make ext4 a specific case.
Thanks for pointing this out. I agree that _require_loop_ext4_feature is better. Sure, I'll add a _require_loop_feature. Alright, I'll do _require_loop stuff in a different patch.
- Harshad.
Quoted 3 lines
Quoted 3 lines
Cheers,
Amir.

harshads

author

2018-01-08 04:18 UTC #10 lore raw

PATCH v4 1/1 ext4/030: Ext4 online resize with bigalloc tests.

In reply to harshads

Add tests to verify Ext4 online resizing feature with bigalloc feature enabled. We test various resizing scenarios with different cluster sizes.

Signed-off-by Harshad Shirwadkar <[email protected]>

---
 tests/ext4/030     | 156 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 148 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/group   |   1 +
 3 files changed, 305 insertions(+)
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out

tests/ext4/030 new file +156 -0

new file mode 100755
index 00000000..99ca9362
@@ -0,0 +1,156 @@
+#! /bin/bash
+# FS QA Test ext4/030
+#
+# Ext4 online resize tests of small and crucial resizes with bigalloc
+# feature.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Google, Inc.  All Rights Reserved.
+#
+# Author: Harshad Shirwadkar <[email protected]>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_bytes2blk()
+{
+    local bytes=$1
+    BLKSIZ=4096
+    echo $((bytes / $BLKSIZ))
+}
+
+_ext4_online_resize()
+{
+    local image_file=$1
+    local image_mount=$2
+    local original_size=$3
+    local final_size=$4
+    local cluster_size=$5
+
+    echo "+ truncate image file to $(_bytes2blk $final_size)"
+    $XFS_IO_PROG -f -c "truncate ${final_size}" ${image_file}
+    LOOP_DEVICE=`_create_loop_device $image_file || _fail "losetup failed"`
+
+    echo "+ create fs on image file $(_bytes2blk $original_size)"
+    ${MKFS_PROG}.${FSTYP} -F -O bigalloc,resize_inode,metadata_csum \
+		-C ${cluster_size} -b 4096 \
+		${LOOP_DEVICE} $(_bytes2blk $original_size) > /dev/null 2>&1 || \
+	_fail "mkfs failed"
+
+    echo "+ mount image file"
+    $MOUNT_PROG -t ${FSTYP} ${LOOP_DEVICE} ${image_mount} > \
+		/dev/null 2>&1 || _fail "mount failed"
+
+    echo "+ resize fs to $(_bytes2blk $final_size)"
+    resize2fs -f ${LOOP_DEVICE} $(_bytes2blk $final_size) >> $seqres.full 2>&1 || \
+	_fail "resize2fs failed"
+
+    echo "+ umount fs"
+    $UMOUNT_PROG ${image_mount}
+
+    echo "+ check fs"
+    _check_generic_filesystem $LOOP_DEVICE >> $seqres.full 2>&1 || \
+	_fail "fsck should not fail"
+    _destroy_loop_device $LOOP_DEVICE && LOOP_DEVICE=
+}
+
+_cleanup()
+{
+    cd /
+    [ -n "$LOOP_DEVICE" ] && _destroy_loop_device $LOOP_DEVICE > /dev/null 2>&1
+    rm -f $tmp.*
+    umount ${IMG_MNT} > /dev/null 2>&1
+    rm -f ${IMG_FILE} > /dev/null 2>&1
+}
+
+# get standard environment and checks
+. ./common/rc
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs ext4
+_supported_os Linux
+
+_require_loop
+_require_scratch
+_require_scratch_ext4_feature "bigalloc,resize_inode,metadata_csum"
+
+IMG_FILE=$TEST_DIR/$seq.fs
+IMG_MNT=$TEST_DIR/$seq.mnt
+
+rm -f $seqres.full
+
+rmdir $IMG_MNT 2>/dev/null
+mkdir -p $IMG_MNT || _fail "cannot create loopback mount point"
+
+## We perform resizing to various multiples of block group sizes to
+## ensure that we cover maximum edge cases in the kernel code.
+for cluster_size in 4096 16384 65536; do
+    echo "+ set cluster size to ${cluster_size}"
+    ## Extending a 1/2 block group to a 2/3 block group.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((16384 * $cluster_size)) \
+			$((24576 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to one cluster less than a
+    ## full block group.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((32767 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to a full block group.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((32768 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to 2 block groups.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((65536 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to 15 block groups and one
+    ## cluster.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((491521 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to 15 and a half block groups.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((507904 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to 16 block groups.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((524288 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to 160 block groups.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((5242880 * $cluster_size)) ${cluster_size}
+done
+
+status=0
+exit

tests/ext4/030.out new file +148 -0

148 lines not on this page · raw message

tests/ext4/group +1 -0

index 257bb646..f29d3de6 100644
@@ -32,6 +32,7 @@
 027 auto quick fsmap
 028 auto quick fsmap
 029 auto quick fsmap
+030 auto quick ioctl resize
 271 auto rw quick
 301 aio auto ioctl rw stress defrag
 302 aio auto ioctl rw stress defrag

Signature · 1 line

2.16.0.rc0.223.g4a4ac83678-goog

Eryu Guan

2018-01-11 07:07 UTC #11 lore raw

Re: [PATCH v4] ext4/030: Ext4 online resize with bigalloc tests.

Quoted 5 lines · harshads

Add tests to verify Ext4 online resizing feature with bigalloc feature enabled. We test various resizing scenarios with different cluster sizes.
Signed-off-by Harshad Shirwadkar <[email protected]>

I have some more comments inline, sorry for not bringing up all comments in previous reviews.. And please cc ext4 list for new ext4 test.

One minor issue regarding to the patch summary:

ext4/030: Ext4 online resize with bigalloc tests

there's no need to include a fixed test seq number in summary for new test, new tests usually will be re-numbered on commit.

Quoted 39 lines

---
 tests/ext4/030     | 156 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 148 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/group   |   1 +
 3 files changed, 305 insertions(+)
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out

tests/ext4/030 new file +25 -0

new file mode 100755
index 00000000..99ca9362
@@ -0,0 +1,156 @@
+#! /bin/bash
+# FS QA Test ext4/030
+#
+# Ext4 online resize tests of small and crucial resizes with bigalloc
+# feature.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Google, Inc.  All Rights Reserved.
+#
+# Author: Harshad Shirwadkar <[email protected]>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+

tests/ext4/030 line 35

Quoted 10 lines

tests/ext4/030 new file +10 -0

+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_bytes2blk()

Usually the leading underscore is only used for common helper names, not a local function, "bytes2blk" is fine.

tests/ext4/030 line 37

Quoted 2 lines

tests/ext4/030 new file +2 -0
+{
+    local bytes=$1

And please indent with tab not 4 spaces.

tests/ext4/030 line 38

Quoted 1 line

tests/ext4/030 new file +1 -0
+    BLKSIZ=4096

BLKSIZ is always 4096 and is only used in this function, move it out of bytes2blk? e.g.

BLKSIZ=4096
bytes2blk()
{
...
}

This leads me to wonder that do we need to test other block sizes like 1k and 2k?

tests/ext4/030 line 50

Quoted 12 lines

tests/ext4/030 new file +12 -0

+    echo $((bytes / $BLKSIZ))
+}
+
+_ext4_online_resize()
+{
+    local image_file=$1
+    local image_mount=$2
+    local original_size=$3
+    local final_size=$4
+    local cluster_size=$5
+
+    echo "+ truncate image file to $(_bytes2blk $final_size)"

Append such logs to $seqres.full too? I might be easier for debugging.

tests/ext4/030 line 55

Quoted 5 lines

tests/ext4/030 new file +5 -0

+    $XFS_IO_PROG -f -c "truncate ${final_size}" ${image_file}
+    LOOP_DEVICE=`_create_loop_device $image_file || _fail "losetup failed"`
+
+    echo "+ create fs on image file $(_bytes2blk $original_size)"
+    ${MKFS_PROG}.${FSTYP} -F -O bigalloc,resize_inode,metadata_csum \

Why do we need "metadata_csum" here? Some comments would be fine if it's really needed. Better to have comments on the "resize_inode" feature too.

tests/ext4/030 line 66

Quoted 11 lines

tests/ext4/030 new file +11 -0

+		-C ${cluster_size} -b 4096 \
+		${LOOP_DEVICE} $(_bytes2blk $original_size) > /dev/null 2>&1 || \
+	_fail "mkfs failed"
+
+    echo "+ mount image file"
+    $MOUNT_PROG -t ${FSTYP} ${LOOP_DEVICE} ${image_mount} > \
+		/dev/null 2>&1 || _fail "mount failed"
+
+    echo "+ resize fs to $(_bytes2blk $final_size)"
+    resize2fs -f ${LOOP_DEVICE} $(_bytes2blk $final_size) >> $seqres.full 2>&1 || \
+	_fail "resize2fs failed"

Hmm, online resize support for bigalloc ext4 is added in v4.15-rc1, tests would fail on old kernels. I think we need a new _require rule to make sure current kernel does support ext4 online resize with bigalloc feature.

Quoted 6 lines

tests/ext4/030 new file +6 -0

+
+    echo "+ umount fs"
+    $UMOUNT_PROG ${image_mount}
+
+    echo "+ check fs"
+    _check_generic_filesystem $LOOP_DEVICE >> $seqres.full 2>&1 || \

tests/ext4/030 line 82

Quoted 10 lines

tests/ext4/030 new file +10 -0

+	_fail "fsck should not fail"
+    _destroy_loop_device $LOOP_DEVICE && LOOP_DEVICE=
+}
+
+_cleanup()
+{
+    cd /
+    [ -n "$LOOP_DEVICE" ] && _destroy_loop_device $LOOP_DEVICE > /dev/null 2>&1
+    rm -f $tmp.*
+    umount ${IMG_MNT} > /dev/null 2>&1

$UMOUNT_PROG

Quoted 9 lines

tests/ext4/030 new file +9 -0

+    rm -f ${IMG_FILE} > /dev/null 2>&1
+}
+
+# get standard environment and checks
+. ./common/rc
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+

tests/ext4/030 line 101

Quoted 10 lines

tests/ext4/030 new file +10 -0

+# real QA test starts here
+_supported_fs ext4
+_supported_os Linux
+
+_require_loop
+_require_scratch
+_require_scratch_ext4_feature "bigalloc,resize_inode,metadata_csum"
+
+IMG_FILE=$TEST_DIR/$seq.fs
+IMG_MNT=$TEST_DIR/$seq.mnt

We required scratch device, then just create fs images on $SCRATCH_MNT? i.e.

rm -f $seqres.full
_scratch_mkfs >>$seqres.full 2>&1
_scratch_mount

IMG_FILE=$SCRATCH_MNT/$seq.fs
IMG_MNT=$SCRATCH_MNT/$seq.mnt

tests/ext4/030 line 105

Quoted 4 lines

tests/ext4/030 new file +4 -0

+
+rm -f $seqres.full
+
+rmdir $IMG_MNT 2>/dev/null

Then there's no need to do rmdir on newly created filesystem.

tests/ext4/030 line 109

Quoted 4 lines

tests/ext4/030 new file +4 -0

+mkdir -p $IMG_MNT || _fail "cannot create loopback mount point"
+
+## We perform resizing to various multiples of block group sizes to
+## ensure that we cover maximum edge cases in the kernel code.

I found that your reply to Ted in the ext4 bigalloc online resize path thread provided more details of the test:

"... ensure that all different code paths that resize IOCTL triggers get executed. At a high level, these different code paths are - 1) extending last block group, 2) adding new block groups 3) conversion of file system to meta-bg."

I think it's better to add similar comments in the test too.

Thanks,
Eryu

Quoted 216 lines

tests/ext4/030 new file +47 -0

+for cluster_size in 4096 16384 65536; do
+    echo "+ set cluster size to ${cluster_size}"
+    ## Extending a 1/2 block group to a 2/3 block group.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((16384 * $cluster_size)) \
+			$((24576 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to one cluster less than a
+    ## full block group.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((32767 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to a full block group.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((32768 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to 2 block groups.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((65536 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to 15 block groups and one
+    ## cluster.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((491521 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to 15 and a half block groups.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((507904 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to 16 block groups.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((524288 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to 160 block groups.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+			$((24576 * $cluster_size)) \
+			$((5242880 * $cluster_size)) ${cluster_size}
+done
+
+status=0
+exit

tests/ext4/030.out new file +148 -0

new file mode 100644
index 00000000..17cc4d99
@@ -0,0 +1,148 @@
+QA output created by 030
++ set cluster size to 4096
++ truncate image file to 24576
++ create fs on image file 16384
++ mount image file
++ resize fs to 24576
++ umount fs
++ check fs
++ truncate image file to 32767
++ create fs on image file 24576
++ mount image file
++ resize fs to 32767
++ umount fs
++ check fs
++ truncate image file to 32768
++ create fs on image file 24576
++ mount image file
++ resize fs to 32768
++ umount fs
++ check fs
++ truncate image file to 65536
++ create fs on image file 24576
++ mount image file
++ resize fs to 65536
++ umount fs
++ check fs
++ truncate image file to 491521
++ create fs on image file 24576
++ mount image file
++ resize fs to 491521
++ umount fs
++ check fs
++ truncate image file to 507904
++ create fs on image file 24576
++ mount image file
++ resize fs to 507904
++ umount fs
++ check fs
++ truncate image file to 524288
++ create fs on image file 24576
++ mount image file
++ resize fs to 524288
++ umount fs
++ check fs
++ truncate image file to 5242880
++ create fs on image file 24576
++ mount image file
++ resize fs to 5242880
++ umount fs
++ check fs
++ set cluster size to 16384
++ truncate image file to 98304
++ create fs on image file 65536
++ mount image file
++ resize fs to 98304
++ umount fs
++ check fs
++ truncate image file to 131068
++ create fs on image file 98304
++ mount image file
++ resize fs to 131068
++ umount fs
++ check fs
++ truncate image file to 131072
++ create fs on image file 98304
++ mount image file
++ resize fs to 131072
++ umount fs
++ check fs
++ truncate image file to 262144
++ create fs on image file 98304
++ mount image file
++ resize fs to 262144
++ umount fs
++ check fs
++ truncate image file to 1966084
++ create fs on image file 98304
++ mount image file
++ resize fs to 1966084
++ umount fs
++ check fs
++ truncate image file to 2031616
++ create fs on image file 98304
++ mount image file
++ resize fs to 2031616
++ umount fs
++ check fs
++ truncate image file to 2097152
++ create fs on image file 98304
++ mount image file
++ resize fs to 2097152
++ umount fs
++ check fs
++ truncate image file to 20971520
++ create fs on image file 98304
++ mount image file
++ resize fs to 20971520
++ umount fs
++ check fs
++ set cluster size to 65536
++ truncate image file to 393216
++ create fs on image file 262144
++ mount image file
++ resize fs to 393216
++ umount fs
++ check fs
++ truncate image file to 524272
++ create fs on image file 393216
++ mount image file
++ resize fs to 524272
++ umount fs
++ check fs
++ truncate image file to 524288
++ create fs on image file 393216
++ mount image file
++ resize fs to 524288
++ umount fs
++ check fs
++ truncate image file to 1048576
++ create fs on image file 393216
++ mount image file
++ resize fs to 1048576
++ umount fs
++ check fs
++ truncate image file to 7864336
++ create fs on image file 393216
++ mount image file
++ resize fs to 7864336
++ umount fs
++ check fs
++ truncate image file to 8126464
++ create fs on image file 393216
++ mount image file
++ resize fs to 8126464
++ umount fs
++ check fs
++ truncate image file to 8388608
++ create fs on image file 393216
++ mount image file
++ resize fs to 8388608
++ umount fs
++ check fs
++ truncate image file to 83886080
++ create fs on image file 393216
++ mount image file
++ resize fs to 83886080
++ umount fs
++ check fs

tests/ext4/group +1 -0

index 257bb646..f29d3de6 100644
@@ -32,6 +32,7 @@
 027 auto quick fsmap
 028 auto quick fsmap
 029 auto quick fsmap
+030 auto quick ioctl resize
 271 auto rw quick
 301 aio auto ioctl rw stress defrag
 302 aio auto ioctl rw stress defrag

Signature · 1 line

2.16.0.rc0.223.g4a4ac83678-goog

Harshad Shirwadkar

author

2018-01-23 05:29 UTC #12 lore raw

Re: [PATCH v4] ext4/030: Ext4 online resize with bigalloc tests.

Quoted 10 lines · Eryu Guan

Quoted 5 lines · harshads
Add tests to verify Ext4 online resizing feature with bigalloc feature enabled. We test various resizing scenarios with different cluster sizes.
Signed-off-by Harshad Shirwadkar <[email protected]>
I have some more comments inline, sorry for not bringing up all comments in previous reviews.. And please cc ext4 list for new ext4 test.

Quoted 6 lines

One minor issue regarding to the patch summary:
ext4/030: Ext4 online resize with bigalloc tests
there's no need to include a fixed test seq number in summary for new test, new tests usually will be re-numbered on commit.

Done.

Quoted 45 lines

---
 tests/ext4/030     | 156 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 148 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/group   |   1 +
 3 files changed, 305 insertions(+)
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out

tests/ext4/030 new file +30 -0

new file mode 100755
index 00000000..99ca9362
@@ -0,0 +1,156 @@
+#! /bin/bash
+# FS QA Test ext4/030
+#
+# Ext4 online resize tests of small and crucial resizes with bigalloc
+# feature.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Google, Inc.  All Rights Reserved.
+#
+# Author: Harshad Shirwadkar <[email protected]>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`

Quoted 8 lines

tests/ext4/030 line 35
Quoted 5 lines
tests/ext4/030 new file +5 -0
+tmp=/tmp/$$
+status=1     # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_bytes2blk()
Usually the leading underscore is only used for common helper names, not a local function, "bytes2blk" is fine.

Done.

Quoted 5 lines

tests/ext4/030 line 37
Quoted 3 lines
tests/ext4/030 new file +2 -0
+{
+    local bytes=$1
And please indent with tab not 4 spaces.

Done.

Quoted 12 lines

tests/ext4/030 line 38
Quoted 2 lines
tests/ext4/030 new file +1 -0
+    BLKSIZ=4096
BLKSIZ is always 4096 and is only used in this function, move it out of bytes2blk? e.g.
BLKSIZ=4096
bytes2blk()
{
...
}

Done.

Quoted 2 lines

This leads me to wonder that do we need to test other block sizes like 1k and 2k?

As of now, I have performed testing with 4k and 2k block sizes. Everything looks good. With 1k block size, there are a few small fixes. So I was we could get this version of the test in. When 1k block fixes are in the kernel, we could enhance the test suite to test that.

Quoted 15 lines

tests/ext4/030 line 50
Quoted 13 lines
tests/ext4/030 new file +12 -0
+    echo $((bytes / $BLKSIZ))
+}
+
+_ext4_online_resize()
+{
+    local image_file=$1
+    local image_mount=$2
+    local original_size=$3
+    local final_size=$4
+    local cluster_size=$5
+
+    echo "+ truncate image file to $(_bytes2blk $final_size)"
Append such logs to $seqres.full too? I might be easier for debugging.

Done.

Quoted 38 lines

tests/ext4/030 line 55
Quoted 6 lines
tests/ext4/030 new file +5 -0
+    $XFS_IO_PROG -f -c "truncate ${final_size}" ${image_file}
+    LOOP_DEVICE=`_create_loop_device $image_file || _fail "losetup failed"`
+
+    echo "+ create fs on image file $(_bytes2blk $original_size)"
+    ${MKFS_PROG}.${FSTYP} -F -O bigalloc,resize_inode,metadata_csum \
Why do we need "metadata_csum" here? Some comments would be fine if it's really needed. Better to have comments on the "resize_inode" feature too.
tests/ext4/030 line 66
Quoted 11 lines
tests/ext4/030 new file +11 -0
+             -C ${cluster_size} -b 4096 \
+             ${LOOP_DEVICE} $(_bytes2blk $original_size) > /dev/null 2>&1 || \
+     _fail "mkfs failed"
+
+    echo "+ mount image file"
+    $MOUNT_PROG -t ${FSTYP} ${LOOP_DEVICE} ${image_mount} > \
+             /dev/null 2>&1 || _fail "mount failed"
+
+    echo "+ resize fs to $(_bytes2blk $final_size)"
+    resize2fs -f ${LOOP_DEVICE} $(_bytes2blk $final_size) >> $seqres.full 2>&1 || \
+     _fail "resize2fs failed"
Hmm, online resize support for bigalloc ext4 is added in v4.15-rc1, tests would fail on old kernels. I think we need a new _require rule to make sure current kernel does support ext4 online resize with bigalloc feature.
Quoted 10 lines
tests/ext4/030 new file +10 -0
+
+    echo "+ umount fs"
+    $UMOUNT_PROG ${image_mount}
+
+    echo "+ check fs"
+    _check_generic_filesystem $LOOP_DEVICE >> $seqres.full 2>&1 || \
+     _fail "fsck should not fail"
+    _destroy_loop_device $LOOP_DEVICE && LOOP_DEVICE=
+}
+

Quoted 8 lines

tests/ext4/030 line 82

Quoted 6 lines

tests/ext4/030 new file +6 -0

+_cleanup()
+{
+    cd /
+    [ -n "$LOOP_DEVICE" ] && _destroy_loop_device $LOOP_DEVICE > /dev/null 2>&1
+    rm -f $tmp.*
+    umount ${IMG_MNT} > /dev/null 2>&1

$UMOUNT_PROG

Done.

Quoted 24 lines

Quoted 10 lines

tests/ext4/030 new file +9 -0

+    rm -f ${IMG_FILE} > /dev/null 2>&1
+}
+
+# get standard environment and checks
+. ./common/rc
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+

tests/ext4/030 line 101

Quoted 10 lines

tests/ext4/030 new file +10 -0

+# real QA test starts here
+_supported_fs ext4
+_supported_os Linux
+
+_require_loop
+_require_scratch
+_require_scratch_ext4_feature "bigalloc,resize_inode,metadata_csum"
+
+IMG_FILE=$TEST_DIR/$seq.fs
+IMG_MNT=$TEST_DIR/$seq.mnt

We required scratch device, then just create fs images on $SCRATCH_MNT? i.e.

Quoted 6 lines

rm -f $seqres.full
_scratch_mkfs >>$seqres.full 2>&1
_scratch_mount
IMG_FILE=$SCRATCH_MNT/$seq.fs
IMG_MNT=$SCRATCH_MNT/$seq.mnt

Done.

Quoted 7 lines

tests/ext4/030 line 105
Quoted 5 lines
tests/ext4/030 new file +4 -0
+
+rm -f $seqres.full
+
+rmdir $IMG_MNT 2>/dev/null
Then there's no need to do rmdir on newly created filesystem.

Done.

Quoted 9 lines

tests/ext4/030 line 109
Quoted 5 lines
tests/ext4/030 new file +4 -0
+mkdir -p $IMG_MNT || _fail "cannot create loopback mount point"
+
+## We perform resizing to various multiples of block group sizes to
+## ensure that we cover maximum edge cases in the kernel code.
I found that your reply to Ted in the ext4 bigalloc online resize path thread provided more details of the test:

Quoted 6 lines

"... ensure that all different code paths that resize IOCTL triggers get executed. At a high level, these different code paths are - 1) extending last block group, 2) adding new block groups 3) conversion of file system to meta-bg."
I think it's better to add similar comments in the test too.

I added comments in the test.

Quoted 220 lines

Thanks,
Eryu

Quoted 216 lines

tests/ext4/030 new file +47 -0

+for cluster_size in 4096 16384 65536; do
+    echo "+ set cluster size to ${cluster_size}"
+    ## Extending a 1/2 block group to a 2/3 block group.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+                     $((16384 * $cluster_size)) \
+                     $((24576 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to one cluster less than a
+    ## full block group.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+                     $((24576 * $cluster_size)) \
+                     $((32767 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to a full block group.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+                     $((24576 * $cluster_size)) \
+                     $((32768 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to 2 block groups.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+                     $((24576 * $cluster_size)) \
+                     $((65536 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to 15 block groups and one
+    ## cluster.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+                     $((24576 * $cluster_size)) \
+                     $((491521 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to 15 and a half block groups.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+                     $((24576 * $cluster_size)) \
+                     $((507904 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to 16 block groups.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+                     $((24576 * $cluster_size)) \
+                     $((524288 * $cluster_size)) ${cluster_size}
+
+    ## Extending a 2/3rd block group to 160 block groups.
+    _ext4_online_resize ${IMG_FILE} ${IMG_MNT} \
+                     $((24576 * $cluster_size)) \
+                     $((5242880 * $cluster_size)) ${cluster_size}
+done
+
+status=0
+exit

tests/ext4/030.out new file +148 -0

new file mode 100644
index 00000000..17cc4d99
@@ -0,0 +1,148 @@
+QA output created by 030
++ set cluster size to 4096
++ truncate image file to 24576
++ create fs on image file 16384
++ mount image file
++ resize fs to 24576
++ umount fs
++ check fs
++ truncate image file to 32767
++ create fs on image file 24576
++ mount image file
++ resize fs to 32767
++ umount fs
++ check fs
++ truncate image file to 32768
++ create fs on image file 24576
++ mount image file
++ resize fs to 32768
++ umount fs
++ check fs
++ truncate image file to 65536
++ create fs on image file 24576
++ mount image file
++ resize fs to 65536
++ umount fs
++ check fs
++ truncate image file to 491521
++ create fs on image file 24576
++ mount image file
++ resize fs to 491521
++ umount fs
++ check fs
++ truncate image file to 507904
++ create fs on image file 24576
++ mount image file
++ resize fs to 507904
++ umount fs
++ check fs
++ truncate image file to 524288
++ create fs on image file 24576
++ mount image file
++ resize fs to 524288
++ umount fs
++ check fs
++ truncate image file to 5242880
++ create fs on image file 24576
++ mount image file
++ resize fs to 5242880
++ umount fs
++ check fs
++ set cluster size to 16384
++ truncate image file to 98304
++ create fs on image file 65536
++ mount image file
++ resize fs to 98304
++ umount fs
++ check fs
++ truncate image file to 131068
++ create fs on image file 98304
++ mount image file
++ resize fs to 131068
++ umount fs
++ check fs
++ truncate image file to 131072
++ create fs on image file 98304
++ mount image file
++ resize fs to 131072
++ umount fs
++ check fs
++ truncate image file to 262144
++ create fs on image file 98304
++ mount image file
++ resize fs to 262144
++ umount fs
++ check fs
++ truncate image file to 1966084
++ create fs on image file 98304
++ mount image file
++ resize fs to 1966084
++ umount fs
++ check fs
++ truncate image file to 2031616
++ create fs on image file 98304
++ mount image file
++ resize fs to 2031616
++ umount fs
++ check fs
++ truncate image file to 2097152
++ create fs on image file 98304
++ mount image file
++ resize fs to 2097152
++ umount fs
++ check fs
++ truncate image file to 20971520
++ create fs on image file 98304
++ mount image file
++ resize fs to 20971520
++ umount fs
++ check fs
++ set cluster size to 65536
++ truncate image file to 393216
++ create fs on image file 262144
++ mount image file
++ resize fs to 393216
++ umount fs
++ check fs
++ truncate image file to 524272
++ create fs on image file 393216
++ mount image file
++ resize fs to 524272
++ umount fs
++ check fs
++ truncate image file to 524288
++ create fs on image file 393216
++ mount image file
++ resize fs to 524288
++ umount fs
++ check fs
++ truncate image file to 1048576
++ create fs on image file 393216
++ mount image file
++ resize fs to 1048576
++ umount fs
++ check fs
++ truncate image file to 7864336
++ create fs on image file 393216
++ mount image file
++ resize fs to 7864336
++ umount fs
++ check fs
++ truncate image file to 8126464
++ create fs on image file 393216
++ mount image file
++ resize fs to 8126464
++ umount fs
++ check fs
++ truncate image file to 8388608
++ create fs on image file 393216
++ mount image file
++ resize fs to 8388608
++ umount fs
++ check fs
++ truncate image file to 83886080
++ create fs on image file 393216
++ mount image file
++ resize fs to 83886080
++ umount fs
++ check fs

tests/ext4/group +1 -1

index 257bb646..f29d3de6 100644
@@ -32,6 +32,7 @@
 027 auto quick fsmap
 028 auto quick fsmap
 029 auto quick fsmap
+030 auto quick ioctl resize
 271 auto rw quick
 301 aio auto ioctl rw stress defrag
 302 aio auto ioctl rw stress defrag
--

2.16.0.rc0.223.g4a4ac83678-goog

harshads

author

2018-01-23 21:54 UTC #13 lore raw

PATCH v4 1/1 ext4: Ext4 online resize with bigalloc tests.

In reply to Eryu Guan

Add tests to verify Ext4 online resizing feature with bigalloc feature enabled. We test various resizing scenarios with different cluster sizes.

Signed-off-by Harshad Shirwadkar <[email protected]>

---
 tests/ext4/030     | 174 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 172 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/group   |   1 +
 3 files changed, 347 insertions(+)
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out

tests/ext4/030 new file +174 -0

new file mode 100755
index 00000000..9e91b96a
@@ -0,0 +1,174 @@
+#! /bin/bash
+# FS QA Test ext4/030
+#
+# Ext4 online resize tests of small and crucial resizes with bigalloc
+# feature.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Google, Inc.  All Rights Reserved.
+#
+# Author: Harshad Shirwadkar <[email protected]>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.	See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,	 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+BLK_SIZ=4096
+CLUSTER_SIZ=4096
+
+IMG_FILE=$SCRATCH_DIR/$seq.fs
+IMG_MNT=$SCRATCH_DIR/$seq.mnt
+
+## Num clusters to blocks.
+c2b()
+{
+	local clusters=$1
+	echo $(($clusters * $CLUSTER_SIZ / $BLK_SIZ))
+}
+
+_ext4_online_resize()
+{
+	## sizes are in number of blocks.
+	local original_size=$1
+	local final_size=$2
+
+	## Start with a clean image file.
+	echo "" > ${IMG_FILE}
+	echo "+++ truncate image file to $final_size" | \
+		tee -a $seqres.full
+	$XFS_IO_PROG -f -c "truncate $(($final_size * $BLK_SIZ))" ${IMG_FILE}
+	LOOP_DEVICE=`_create_loop_device $IMG_FILE || _fail "losetup failed"`
+
+	echo "+++ create fs on image file $original_size" | \
+		tee -a $seqres.full
+
+	${MKFS_PROG}.${FSTYP} -F -O bigalloc,resize_inode -C $CLUSTER_SIZ \
+		-b $BLK_SIZ ${LOOP_DEVICE} $original_size >> \
+		$seqres.full 2>&1 || _fail "mkfs failed"
+
+	echo "+++ mount image file" | tee -a $seqres.full
+	$MOUNT_PROG -t ${FSTYP} ${LOOP_DEVICE} ${IMG_MNT} > \
+		/dev/null 2>&1 || _fail "mount failed"
+
+	echo "+++ resize fs to $final_size" | tee -a $seqres.full
+
+	resize2fs -f ${LOOP_DEVICE} $final_size >> \
+		  $seqres.full 2>&1 || return 1
+
+	echo "+++ umount fs" | tee -a $seqres.full
+	$UMOUNT_PROG ${IMG_MNT}
+
+	echo "+++ check fs" | tee -a $seqres.full
+	_check_generic_filesystem $LOOP_DEVICE >> $seqres.full 2>&1 || \
+		_fail "fsck should not fail"
+	_destroy_loop_device $LOOP_DEVICE && LOOP_DEVICE=
+}
+
+_cleanup()
+{
+	cd /
+	[ -n "$LOOP_DEVICE" ] && _destroy_loop_device $LOOP_DEVICE > /dev/null 2>&1
+	rm -f $tmp.*
+	$UMOUNT_PROG ${IMG_MNT} > /dev/null 2>&1
+	rm -f ${IMG_FILE} > /dev/null 2>&1
+}
+
+# get standard environment and checks
+. ./common/rc
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs ext4
+_supported_os Linux
+
+_require_loop
+_require_scratch
+# We use resize_inode to make sure that block group descriptor table
+# can be extended.
+_require_scratch_ext4_feature "bigalloc,resize_inode,metadata_csum"
+
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+rm -f $seqres.full
+
+mkdir -p $IMG_MNT || _fail "cannot create loopback mount point"
+
+# Check if online resizing with bigalloc is supported by the kernel
+_ext4_online_resize 4096 8192 || \
+	_notrun "Kernel doesn't support online resizing with bigalloc"
+
+## We perform resizing to various multiples of block group sizes to
+## ensure that we cover maximum edge cases in the kernel code.
+for CLUSTER_SIZ in 4096 16384 65536; do
+	echo "++ set cluster size to $CLUSTER_SIZ" | tee -a $seqres.full
+
+	##
+	## Extend last group tests
+	##
+
+	## Extending a 1/2 block group to a 2/3 block group.
+	_ext4_online_resize $(c2b 16384) $(c2b 24576) \
+		|| _fail "resize failed"
+	## Extending a 2/3rd block group to one cluster less than a
+	## full block group.
+	_ext4_online_resize $(c2b 24576) $(c2b 32767) \
+		|| _fail "resize failed"
+	## Extending a 2/3rd block group to a full block group.
+	_ext4_online_resize $(c2b 24576) $(c2b 32768) \
+		|| _fail "resize failed"
+
+	##
+	## Extend last group and add more block groups.
+	##
+
+	## Extending a 2/3rd block group to 2 block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 65536) \
+		|| _fail "resize failed"
+	## Extending a 2/3rd block group to 15 block groups and one
+	## cluster.
+	_ext4_online_resize $(c2b 24576) $(c2b 491521) \
+		|| _fail "resize failed"
+	## Extending a 2/3rd block group to 15 and a half block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 507904) \
+		|| _fail "resize failed"
+
+	## Extending a 2/3rd block group to 16 block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 524288) \
+		|| _fail "resize failed"
+	## Extending a 2/3rd block group to 160 block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 5242880) \
+		|| _fail "resize failed"
+
+	##
+	## Convert to meta_bg.
+	##
+
+	## Extending a 2/3rd block group to 1280 block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 41943040) \
+		|| _fail "resize failed"
+done
+
+status=0
+exit

tests/ext4/030.out new file +172 -0

172 lines not on this page · raw message

tests/ext4/group +1 -0

index 257bb646..f29d3de6 100644
@@ -32,6 +32,7 @@
 027 auto quick fsmap
 028 auto quick fsmap
 029 auto quick fsmap
+030 auto quick ioctl resize
 271 auto rw quick
 301 aio auto ioctl rw stress defrag
 302 aio auto ioctl rw stress defrag

Signature · 1 line

2.16.0.rc1.238.g530d649a79-goog

Eryu Guan

2018-01-24 06:37 UTC #14 lore raw

Re: [PATCH v4] ext4: Ext4 online resize with bigalloc tests.

Quoted 5 lines · harshads

Add tests to verify Ext4 online resizing feature with bigalloc feature enabled. We test various resizing scenarios with different cluster sizes.
Signed-off-by Harshad Shirwadkar <[email protected]>

Looks good to me, thanks! Just two minor issues inline.

Quoted 113 lines

---
 tests/ext4/030     | 174 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 172 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/group   |   1 +
 3 files changed, 347 insertions(+)
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out

tests/ext4/030 new file +99 -0

new file mode 100755
index 00000000..9e91b96a
@@ -0,0 +1,174 @@
+#! /bin/bash
+# FS QA Test ext4/030
+#
+# Ext4 online resize tests of small and crucial resizes with bigalloc
+# feature.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Google, Inc.  All Rights Reserved.
+#
+# Author: Harshad Shirwadkar <[email protected]>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.	See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,	 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+BLK_SIZ=4096
+CLUSTER_SIZ=4096
+
+IMG_FILE=$SCRATCH_DIR/$seq.fs
+IMG_MNT=$SCRATCH_DIR/$seq.mnt
+
+## Num clusters to blocks.
+c2b()
+{
+	local clusters=$1
+	echo $(($clusters * $CLUSTER_SIZ / $BLK_SIZ))
+}
+
+_ext4_online_resize()
+{
+	## sizes are in number of blocks.
+	local original_size=$1
+	local final_size=$2
+
+	## Start with a clean image file.
+	echo "" > ${IMG_FILE}
+	echo "+++ truncate image file to $final_size" | \
+		tee -a $seqres.full
+	$XFS_IO_PROG -f -c "truncate $(($final_size * $BLK_SIZ))" ${IMG_FILE}
+	LOOP_DEVICE=`_create_loop_device $IMG_FILE || _fail "losetup failed"`
+
+	echo "+++ create fs on image file $original_size" | \
+		tee -a $seqres.full
+
+	${MKFS_PROG}.${FSTYP} -F -O bigalloc,resize_inode -C $CLUSTER_SIZ \
+		-b $BLK_SIZ ${LOOP_DEVICE} $original_size >> \
+		$seqres.full 2>&1 || _fail "mkfs failed"
+
+	echo "+++ mount image file" | tee -a $seqres.full
+	$MOUNT_PROG -t ${FSTYP} ${LOOP_DEVICE} ${IMG_MNT} > \
+		/dev/null 2>&1 || _fail "mount failed"
+
+	echo "+++ resize fs to $final_size" | tee -a $seqres.full
+
+	resize2fs -f ${LOOP_DEVICE} $final_size >> \
+		  $seqres.full 2>&1 || return 1
+
+	echo "+++ umount fs" | tee -a $seqres.full
+	$UMOUNT_PROG ${IMG_MNT}
+
+	echo "+++ check fs" | tee -a $seqres.full
+	_check_generic_filesystem $LOOP_DEVICE >> $seqres.full 2>&1 || \
+		_fail "fsck should not fail"
+	_destroy_loop_device $LOOP_DEVICE && LOOP_DEVICE=
+}
+
+_cleanup()
+{
+	cd /
+	[ -n "$LOOP_DEVICE" ] && _destroy_loop_device $LOOP_DEVICE > /dev/null 2>&1
+	rm -f $tmp.*
+	$UMOUNT_PROG ${IMG_MNT} > /dev/null 2>&1
+	rm -f ${IMG_FILE} > /dev/null 2>&1
+}
+
+# get standard environment and checks
+. ./common/rc
+
+# remove previous $seqres.full before test
+rm -f $seqres.full

tests/ext4/030 line 109

Quoted 10 lines

tests/ext4/030 new file +10 -0

+
+# real QA test starts here
+_supported_fs ext4
+_supported_os Linux
+
+_require_loop
+_require_scratch
+# We use resize_inode to make sure that block group descriptor table
+# can be extended.
+_require_scratch_ext4_feature "bigalloc,resize_inode,metadata_csum"

What about metadata_csum? Seems it's not used in the actual resize test. Meant to delete it?

tests/ext4/030 line 120

Quoted 11 lines

tests/ext4/030 new file +11 -0

+
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+rm -f $seqres.full
+
+mkdir -p $IMG_MNT || _fail "cannot create loopback mount point"
+
+# Check if online resizing with bigalloc is supported by the kernel
+_ext4_online_resize 4096 8192 || \
+	_notrun "Kernel doesn't support online resizing with bigalloc"

Hmm, how do we tell if it's a test failure or lack of kernel support? How about adding an extra argument to ext4_online_resize to indicate that this is a test run, so it checks resize2fs output to see if the failure is really due to lacking support in kernel. Otherwise we treat all resize2fs failures as bugs.

e.g. resize2fs prints:
"resize2fs: Operation not supported While checking for on-line resizing support"

Thanks,
Eryu

harshads

author

2018-01-24 22:58 UTC #15 lore raw

PATCH v5 1/1 ext4: Ext4 online resize with bigalloc tests.

Add tests to verify Ext4 online resizing feature with bigalloc feature enabled. We test various resizing scenarios with different cluster sizes.

Signed-off-by Harshad Shirwadkar <[email protected]>

---
 tests/ext4/030     | 171 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 172 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/group   |   1 +
 3 files changed, 344 insertions(+)
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out

tests/ext4/030 new file +171 -0

new file mode 100755
index 00000000..1d42b514
@@ -0,0 +1,171 @@
+#! /bin/bash
+# FS QA Test ext4/030
+#
+# Ext4 online resize tests of small and crucial resizes with bigalloc
+# feature.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Google, Inc.  All Rights Reserved.
+#
+# Author: Harshad Shirwadkar <[email protected]>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.	See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,	 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+BLK_SIZ=4096
+CLUSTER_SIZ=4096
+
+IMG_FILE=$SCRATCH_DIR/$seq.fs
+IMG_MNT=$SCRATCH_DIR/$seq.mnt
+
+## Num clusters to blocks.
+c2b()
+{
+	local clusters=$1
+	echo $(($clusters * $CLUSTER_SIZ / $BLK_SIZ))
+}
+
+_ext4_online_resize()
+{
+	## sizes are in number of blocks.
+	local original_size=$1
+	local final_size=$2
+	local check_if_supported=$([ "$3" = "1" ] && echo "1" || echo "0")
+
+	## Start with a clean image file.
+	echo "" > ${IMG_FILE}
+	echo "+++ truncate image file to $final_size" | \
+		tee -a $seqres.full
+	$XFS_IO_PROG -f -c "truncate $(($final_size * $BLK_SIZ))" ${IMG_FILE}
+	LOOP_DEVICE=`_create_loop_device $IMG_FILE || _fail "losetup failed"`
+
+	echo "+++ create fs on image file $original_size" | \
+		tee -a $seqres.full
+
+	${MKFS_PROG}.${FSTYP} -F -O bigalloc,resize_inode -C $CLUSTER_SIZ \
+		-b $BLK_SIZ ${LOOP_DEVICE} $original_size >> \
+		$seqres.full 2>&1 || _fail "mkfs failed"
+
+	echo "+++ mount image file" | tee -a $seqres.full
+	$MOUNT_PROG -t ${FSTYP} ${LOOP_DEVICE} ${IMG_MNT} > \
+		/dev/null 2>&1 || _fail "mount failed"
+
+	echo "+++ resize fs to $final_size" | tee -a $seqres.full
+
+	resize_output=$(resize2fs -f ${LOOP_DEVICE} $final_size 2>&1)
+	if [ $? -ne 0 ]; then
+		if [ $check_if_supported -eq 1 ]; then
+			echo $resize_output | \
+			grep "resize2fs: Operation not supported While checking for on-line resizing support" \
+				&& _notrun "online resizing not supported with bigalloc"
+		fi
+		_fail "resize failed"
+	fi
+	echo $resize_output >> $seqres.full
+	echo "+++ umount fs" | tee -a $seqres.full
+	$UMOUNT_PROG ${IMG_MNT}
+
+	echo "+++ check fs" | tee -a $seqres.full
+	_check_generic_filesystem $LOOP_DEVICE >> $seqres.full 2>&1 || \
+		_fail "fsck should not fail"
+	_destroy_loop_device $LOOP_DEVICE && LOOP_DEVICE=
+}
+
+_cleanup()
+{
+	cd /
+	[ -n "$LOOP_DEVICE" ] && _destroy_loop_device $LOOP_DEVICE > /dev/null 2>&1
+	rm -f $tmp.*
+	$UMOUNT_PROG ${IMG_MNT} > /dev/null 2>&1
+	rm -f ${IMG_FILE} > /dev/null 2>&1
+}
+
+# get standard environment and checks
+. ./common/rc
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs ext4
+_supported_os Linux
+
+_require_loop
+_require_scratch
+# We use resize_inode to make sure that block group descriptor table
+# can be extended.
+_require_scratch_ext4_feature "bigalloc,resize_inode"
+
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+rm -f $seqres.full
+
+mkdir -p $IMG_MNT || _fail "cannot create loopback mount point"
+
+# Check if online resizing with bigalloc is supported by the kernel
+_ext4_online_resize 4096 8192 1
+
+## We perform resizing to various multiples of block group sizes to
+## ensure that we cover maximum edge cases in the kernel code.
+for CLUSTER_SIZ in 4096 16384 65536; do
+	echo "++ set cluster size to $CLUSTER_SIZ" | tee -a $seqres.full
+
+	##
+	## Extend last group tests
+	##
+
+	## Extending a 1/2 block group to a 2/3 block group.
+	_ext4_online_resize $(c2b 16384) $(c2b 24576)
+	## Extending a 2/3rd block group to one cluster less than a
+	## full block group.
+	_ext4_online_resize $(c2b 24576) $(c2b 32767)
+	## Extending a 2/3rd block group to a full block group.
+	_ext4_online_resize $(c2b 24576) $(c2b 32768)
+
+	##
+	## Extend last group and add more block groups.
+	##
+
+	## Extending a 2/3rd block group to 2 block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 65536)
+	## Extending a 2/3rd block group to 15 block groups and one
+	## cluster.
+	_ext4_online_resize $(c2b 24576) $(c2b 491521)
+	## Extending a 2/3rd block group to 15 and a half block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 507904)
+	## Extending a 2/3rd block group to 16 block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 524288)
+	## Extending a 2/3rd block group to 160 block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 5242880)
+
+	##
+	## Convert to meta_bg.
+	##
+
+	## Extending a 2/3rd block group to 1280 block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 41943040)
+done
+
+status=0
+exit

tests/ext4/030.out new file +172 -0

172 lines not on this page · raw message

tests/ext4/group +1 -0

index 257bb646..f29d3de6 100644
@@ -32,6 +32,7 @@
 027 auto quick fsmap
 028 auto quick fsmap
 029 auto quick fsmap
+030 auto quick ioctl resize
 271 auto rw quick
 301 aio auto ioctl rw stress defrag
 302 aio auto ioctl rw stress defrag

Signature · 1 line

2.16.0.rc1.238.g530d649a79-goog

Eryu Guan

2018-01-25 05:00 UTC #16 lore raw

Re: [PATCH v5] ext4: Ext4 online resize with bigalloc tests.

Quoted 5 lines · harshads

Add tests to verify Ext4 online resizing feature with bigalloc feature enabled. We test various resizing scenarios with different cluster sizes.
Signed-off-by Harshad Shirwadkar <[email protected]>

Thanks for all the revisions! I made some minor edits and queued it for next release. Please see below.

Quoted 52 lines

---
 tests/ext4/030     | 171 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/030.out | 172 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/ext4/group   |   1 +
 3 files changed, 344 insertions(+)
 create mode 100755 tests/ext4/030
 create mode 100644 tests/ext4/030.out

tests/ext4/030 new file +38 -0

new file mode 100755
index 00000000..1d42b514
@@ -0,0 +1,171 @@
+#! /bin/bash
+# FS QA Test ext4/030
+#
+# Ext4 online resize tests of small and crucial resizes with bigalloc
+# feature.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Google, Inc.  All Rights Reserved.
+#
+# Author: Harshad Shirwadkar <[email protected]>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.	See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,	 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+BLK_SIZ=4096
+CLUSTER_SIZ=4096
+
+IMG_FILE=$SCRATCH_DIR/$seq.fs

tests/ext4/030 line 48

Quoted 10 lines

tests/ext4/030 new file +10 -0

+IMG_MNT=$SCRATCH_DIR/$seq.mnt
+
+## Num clusters to blocks.
+c2b()
+{
+	local clusters=$1
+	echo $(($clusters * $CLUSTER_SIZ / $BLK_SIZ))
+}
+
+_ext4_online_resize()

I removed the leading underscore for local function.

tests/ext4/030 line 53

Quoted 5 lines

tests/ext4/030 new file +5 -0

+{
+	## sizes are in number of blocks.
+	local original_size=$1
+	local final_size=$2
+	local check_if_supported=$([ "$3" = "1" ] && echo "1" || echo "0")

	local check_if_supported=${3:-0}

tests/ext4/030 line 60

Quoted 7 lines

tests/ext4/030 new file +7 -0

+
+	## Start with a clean image file.
+	echo "" > ${IMG_FILE}
+	echo "+++ truncate image file to $final_size" | \
+		tee -a $seqres.full
+	$XFS_IO_PROG -f -c "truncate $(($final_size * $BLK_SIZ))" ${IMG_FILE}
+	LOOP_DEVICE=`_create_loop_device $IMG_FILE || _fail "losetup failed"`

No need to call _fail here, _create_loop_device already did that.

Quoted 5 lines

tests/ext4/030 new file +5 -0

+
+	echo "+++ create fs on image file $original_size" | \
+		tee -a $seqres.full
+
+	${MKFS_PROG}.${FSTYP} -F -O bigalloc,resize_inode -C $CLUSTER_SIZ \

tests/ext4/030 line 75

Quoted 10 lines

tests/ext4/030 new file +10 -0

+		-b $BLK_SIZ ${LOOP_DEVICE} $original_size >> \
+		$seqres.full 2>&1 || _fail "mkfs failed"
+
+	echo "+++ mount image file" | tee -a $seqres.full
+	$MOUNT_PROG -t ${FSTYP} ${LOOP_DEVICE} ${IMG_MNT} > \
+		/dev/null 2>&1 || _fail "mount failed"
+
+	echo "+++ resize fs to $final_size" | tee -a $seqres.full
+
+	resize_output=$(resize2fs -f ${LOOP_DEVICE} $final_size 2>&1)

I changed this to let resize2fs dump output to a tmp file, saving multi-line outputs into a variable would cause subtle problems that are hard to find/debug. e.g. echoing a such variable to a pipe without proper quotes would change the output in subtle ways, and we did hit such problems before.

tests/ext4/030 line 84

Quoted 9 lines

tests/ext4/030 new file +9 -0

+	if [ $? -ne 0 ]; then
+		if [ $check_if_supported -eq 1 ]; then
+			echo $resize_output | \
+			grep "resize2fs: Operation not supported While checking for on-line resizing support" \
+				&& _notrun "online resizing not supported with bigalloc"
+		fi
+		_fail "resize failed"
+	fi
+	echo $resize_output >> $seqres.full

Then grep the tmp file and append it to $seqres.full.

Thanks,
Eryu

Quoted 280 lines

tests/ext4/030 new file +87 -0

+	echo "+++ umount fs" | tee -a $seqres.full
+	$UMOUNT_PROG ${IMG_MNT}
+
+	echo "+++ check fs" | tee -a $seqres.full
+	_check_generic_filesystem $LOOP_DEVICE >> $seqres.full 2>&1 || \
+		_fail "fsck should not fail"
+	_destroy_loop_device $LOOP_DEVICE && LOOP_DEVICE=
+}
+
+_cleanup()
+{
+	cd /
+	[ -n "$LOOP_DEVICE" ] && _destroy_loop_device $LOOP_DEVICE > /dev/null 2>&1
+	rm -f $tmp.*
+	$UMOUNT_PROG ${IMG_MNT} > /dev/null 2>&1
+	rm -f ${IMG_FILE} > /dev/null 2>&1
+}
+
+# get standard environment and checks
+. ./common/rc
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs ext4
+_supported_os Linux
+
+_require_loop
+_require_scratch
+# We use resize_inode to make sure that block group descriptor table
+# can be extended.
+_require_scratch_ext4_feature "bigalloc,resize_inode"
+
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+rm -f $seqres.full
+
+mkdir -p $IMG_MNT || _fail "cannot create loopback mount point"
+
+# Check if online resizing with bigalloc is supported by the kernel
+_ext4_online_resize 4096 8192 1
+
+## We perform resizing to various multiples of block group sizes to
+## ensure that we cover maximum edge cases in the kernel code.
+for CLUSTER_SIZ in 4096 16384 65536; do
+	echo "++ set cluster size to $CLUSTER_SIZ" | tee -a $seqres.full
+
+	##
+	## Extend last group tests
+	##
+
+	## Extending a 1/2 block group to a 2/3 block group.
+	_ext4_online_resize $(c2b 16384) $(c2b 24576)
+	## Extending a 2/3rd block group to one cluster less than a
+	## full block group.
+	_ext4_online_resize $(c2b 24576) $(c2b 32767)
+	## Extending a 2/3rd block group to a full block group.
+	_ext4_online_resize $(c2b 24576) $(c2b 32768)
+
+	##
+	## Extend last group and add more block groups.
+	##
+
+	## Extending a 2/3rd block group to 2 block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 65536)
+	## Extending a 2/3rd block group to 15 block groups and one
+	## cluster.
+	_ext4_online_resize $(c2b 24576) $(c2b 491521)
+	## Extending a 2/3rd block group to 15 and a half block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 507904)
+	## Extending a 2/3rd block group to 16 block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 524288)
+	## Extending a 2/3rd block group to 160 block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 5242880)
+
+	##
+	## Convert to meta_bg.
+	##
+
+	## Extending a 2/3rd block group to 1280 block groups.
+	_ext4_online_resize $(c2b 24576) $(c2b 41943040)
+done
+
+status=0
+exit

tests/ext4/030.out new file +172 -0

new file mode 100644
index 00000000..49304654
@@ -0,0 +1,172 @@
+QA output created by 030
++++ truncate image file to 8192
++++ create fs on image file 4096
++++ mount image file
++++ resize fs to 8192
++++ umount fs
++++ check fs
+++ set cluster size to 4096
++++ truncate image file to 24576
++++ create fs on image file 16384
++++ mount image file
++++ resize fs to 24576
++++ umount fs
++++ check fs
++++ truncate image file to 32767
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 32767
++++ umount fs
++++ check fs
++++ truncate image file to 32768
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 32768
++++ umount fs
++++ check fs
++++ truncate image file to 65536
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 65536
++++ umount fs
++++ check fs
++++ truncate image file to 491521
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 491521
++++ umount fs
++++ check fs
++++ truncate image file to 507904
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 507904
++++ umount fs
++++ check fs
++++ truncate image file to 524288
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 524288
++++ umount fs
++++ check fs
++++ truncate image file to 5242880
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 5242880
++++ umount fs
++++ check fs
++++ truncate image file to 41943040
++++ create fs on image file 24576
++++ mount image file
++++ resize fs to 41943040
++++ umount fs
++++ check fs
+++ set cluster size to 16384
++++ truncate image file to 98304
++++ create fs on image file 65536
++++ mount image file
++++ resize fs to 98304
++++ umount fs
++++ check fs
++++ truncate image file to 131068
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 131068
++++ umount fs
++++ check fs
++++ truncate image file to 131072
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 131072
++++ umount fs
++++ check fs
++++ truncate image file to 262144
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 262144
++++ umount fs
++++ check fs
++++ truncate image file to 1966084
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 1966084
++++ umount fs
++++ check fs
++++ truncate image file to 2031616
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 2031616
++++ umount fs
++++ check fs
++++ truncate image file to 2097152
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 2097152
++++ umount fs
++++ check fs
++++ truncate image file to 20971520
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 20971520
++++ umount fs
++++ check fs
++++ truncate image file to 167772160
++++ create fs on image file 98304
++++ mount image file
++++ resize fs to 167772160
++++ umount fs
++++ check fs
+++ set cluster size to 65536
++++ truncate image file to 393216
++++ create fs on image file 262144
++++ mount image file
++++ resize fs to 393216
++++ umount fs
++++ check fs
++++ truncate image file to 524272
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 524272
++++ umount fs
++++ check fs
++++ truncate image file to 524288
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 524288
++++ umount fs
++++ check fs
++++ truncate image file to 1048576
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 1048576
++++ umount fs
++++ check fs
++++ truncate image file to 7864336
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 7864336
++++ umount fs
++++ check fs
++++ truncate image file to 8126464
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 8126464
++++ umount fs
++++ check fs
++++ truncate image file to 8388608
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 8388608
++++ umount fs
++++ check fs
++++ truncate image file to 83886080
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 83886080
++++ umount fs
++++ check fs
++++ truncate image file to 671088640
++++ create fs on image file 393216
++++ mount image file
++++ resize fs to 671088640
++++ umount fs
++++ check fs

tests/ext4/group +1 -0

index 257bb646..f29d3de6 100644
@@ -32,6 +32,7 @@
 027 auto quick fsmap
 028 auto quick fsmap
 029 auto quick fsmap
+030 auto quick ioctl resize
 271 auto rw quick
 301 aio auto ioctl rw stress defrag
 302 aio auto ioctl rw stress defrag

Signature · 1 line

2.16.0.rc1.238.g530d649a79-goog