fstests: new way to run overlay tests
v5 cover fstests: new way to run overlay tests
Hi Eryu,
I rebased my branch to your master and made some minor adaptations to Xiong's new tests. Also reordered the patches:
- patches 1-6 are generic fixes and re-factoring - if you wish, you can merge them before the overlay config changes.
- patches 7-10 implement the new overlay config. patches 7 and 10 include adaptations of Xiong's new tests to new config.
This is the 5th revision of new overlayfs config.
The main motivation of this work is to help catch overlayfs bugs related to leaking objects in underlying (base) fs.
With this change, all you have to do to run overlay tests if you already have a local.config setup to test a local file system is:
./check -overlay
See README.overlay for some examples of config files to use with -overlay.
The legacy overlayfs configuration, where TEST_DEV is a directory still works, but it should be deprecated.
Tested ./check -overlay -g quick with both legacy overlay configuration and the new base fs configuration.
Eryu wished to keep these changes soaking on the list for while, so for those who can help with testing, you can get pull the branch for testing from my github tree [1].
Thanks,
Amir.
[1] https://github.com/amir73il/xfstests/tree/ovl_base_fs
v5:
- Rebased to weekly update
- Adapted new overlay tests by Xiong to new config
- Reordered patches "fixes and re-factoring first"
v4:
- Address review comments from Eryu
- Fix sanity checks for already mounted scratch base fs
- Fix handling of SCRATCH_DEV_POOL
- Fix problems when run in kvm-xfstests
- Add README.overlay with example config files
- Remove documentation for overlay config sections
v3:
- Mount cycle base test fs
- Fix bugs in non overlay specific sanity checks
- Run -overlay test with existing config file of base fs
- Run overlay tests per base fs by adding overlay config sections
v2:
- Test and scratch base dirs each have thier own base fs
- Support mount cycles of base fs for scratch tests
v1:
- Both test and scratch base dirs on a single base fs
Amir Goldstein (10):
fstests: sanity check that test partitions are not mounted elsewhere
fstests: use _test_mount() consistently
fstests: canonicalize mount points on every config section
fstests: fix test and scratch filters for overlapping DEV/MNT paths
fstests: allow overlay SCRATCH_DEV to be a base fs mount point generic/064: access SCRATCH_MNT after _scratch_mount
overlay: rename OVERLAY_LOWER/UPPER/WORK_DIR
overlay: configure TEST/SCRATCH vars to base fs
overlay: mount/unmount base fs before/after running tests
overlay: use OVL_BASE_SCRATCH_MNT instead of SCRATCH_DEV
README | 16 ++-- README.overlay | 46 +++++++++++ check | 18 ++--- common/config | 160 ++++++++++++++++++++++++++++++++++----- common/filter | 33 +++++++- common/rc | 206 ++++++++++++++++++++++++++++++++++++-------------- tests/generic/064 | 10 ++- tests/overlay/001 | 7 +- tests/overlay/002 | 2 +- tests/overlay/003 | 5 +- tests/overlay/004 | 7 +- tests/overlay/005 | 30 ++++---- tests/overlay/006 | 10 +-- tests/overlay/008 | 8 +- tests/overlay/009 | 2 +- tests/overlay/010 | 10 +-- tests/overlay/011 | 6 +- tests/overlay/012 | 4 +- tests/overlay/013 | 4 +- tests/overlay/014 | 19 ++--- tests/overlay/015 | 2 +- tests/overlay/016 | 2 +- tests/overlay/017 | 2 +- tests/overlay/018 | 2 +- tests/overlay/019 | 2 +- tests/overlay/020 | 2 +- tests/overlay/021 | 6 +- tests/overlay/022 | 2 +- tests/overlay/023 | 4 +- tests/overlay/023.out | 2 +- tests/overlay/024 | 2 +- tests/overlay/027 | 2 +- tests/overlay/028 | 2 +- tests/overlay/029 | 4 +- 34 files changed, 466 insertions(+), 173 deletions(-) create mode 100644 README.overlay
Signature · 1 line
2.7.4
PATCH v5 1/10 fstests: sanity check that test partitions are not mounted elsewhere
When $TEST_DEV is mounted at a different location then $TEST_DIR, _require_test() aborts the test with an error:
TEST_DEV=/dev/sda5 is mounted but not on TEST_DIR=/mnt/test
There are several problems with current sanity check:
1. the output of the error is mixed into out.bad and hard to see
2. the test partition is unmounted at the end of the test regardless of the fact that it not pass the sanity that we have exclusivity
3. scratch partition has a similar sanity check in _require_scratch(), but we may not get to it, because $SCRATCH_DEV is unmounted prior to running the tests (which could unmount another mount point).
To solve all these problems, introduce a helper _check_mounted_on(). It checks if a device is mounted on a given mount point and optionally checks the mounted fs type.
The sanity checks in _require_scratch() and _require_test() are converted to use the helper and gain the check for correct fs type.
The helper is used in init_rc() to sanity check both test and scratch partitions, before tests are run and before $SCRATCH_DEV is unmounted.
--- common/rc | 90 +++++++++++++++++++++++++++++++++++++++------------------------ 1 file changed, 56 insertions(+), 34 deletions(-)
common/rc +56 -34
@@ -1354,6 +1354,43 @@ _supported_os() _notrun "not suitable for this OS: $HOSTOS" } +# check if a FS on a device is mounted +# if so, verify that it is mounted on mount point +# if fstype is given as argument, verify that it is also +# mounted with correct fs type +# +_check_mounted_on() +{ + local devname=$1 + local dev=$2 + local mntname=$3 + local mnt=$4 + local type=$5 + + # Note that we use -F here so grep doesn't try to interpret an NFS over + # IPv6 server as a regular expression + local mount_rec=`_mount | grep -F "$dev"` + [ -n "$mount_rec" ] || return 1 # 1 = not mounted + + # if it's mounted, make sure its on $mnt + if ! (echo $mount_rec | grep -q "$mnt") + then + echo "$devname=$dev is mounted but not on $mntname=$mnt - aborting" + echo "Already mounted result:" + echo $mount_rec + return 2 # 2 = mounted on wrong mnt + fi + + if [ -n "$type" -a "`_fs_type $dev`" != "$type" ] + then + echo "$devname=$dev is mounted but not a type $type filesystem" + # raw $DF_PROG cannot handle NFS/CIFS/overlay correctly + _df_device $dev + return 3 # 3 = mounted as wrong type + fi + return 0 # 0 = mounted as expected +} + # this test needs a scratch partition - check we're ok & unmount it # No post-test check of the device is required. e.g. the test intentionally # finishes the test with the filesystem in a corrupt state
@@ -1408,21 +1445,12 @@ _require_scratch_nocheck() ;; esac - # mounted? - # Note that we use -F here so grep doesn't try to interpret an NFS over - # IPv6 server as a regular expression. - mount_rec=`_mount | grep -F $SCRATCH_DEV` - if [ "$mount_rec" ] + _check_mounted_on SCRATCH_DEV $SCRATCH_DEV SCRATCH_MNT $SCRATCH_MNT + local err=$? + [ $err -le 1 ] || exit 1 + if [ $err -eq 0 ] then - # if it's mounted, make sure its on $SCRATCH_MNT - if ! echo $mount_rec | grep -q $SCRATCH_MNT - then - echo "\$SCRATCH_DEV=$SCRATCH_DEV is mounted but not on \$SCRATCH_MNT=$SCRATCH_MNT - aborting" - echo "Already mounted result:" - echo $mount_rec - exit 1 - fi - # and then unmount it + # if it's mounted, unmount it if ! _scratch_unmount then echo "failed to unmount $SCRATCH_DEV"
@@ -1493,21 +1521,11 @@ _require_test() ;; esac - # mounted? - # Note that we use -F here so grep doesn't try to interpret an NFS over - # IPv6 server as a regular expression. - mount_rec=`_mount | grep -F $TEST_DEV` - if [ "$mount_rec" ] + _check_mounted_on TEST_DEV $TEST_DEV TEST_DIR $TEST_DIR + local err=$? + [ $err -le 1 ] || exit 1 + if [ $err -ne 0 ] then - # if it's mounted, make sure its on $TEST_DIR - if ! echo $mount_rec | grep -q $TEST_DIR - then - echo "\$TEST_DEV=$TEST_DEV is mounted but not on \$TEST_DIR=$TEST_DIR - aborting" - echo "Already mounted result:" - echo $mount_rec - exit 1 - fi - else out=`_mount_or_remount_rw "$MOUNT_OPTIONS" $TEST_DEV $TEST_DIR` if [ $? -ne 1 ]; then echo $out
@@ -3132,13 +3150,17 @@ init_rc() fi fi - if [ "`_fs_type $TEST_DEV`" != "$FSTYP" ] - then - echo "common/rc: Error: \$TEST_DEV ($TEST_DEV) is not a MOUNTED $FSTYP filesystem" - # raw $DF_PROG cannot handle NFS/CIFS/overlay correctly - _df_device $TEST_DEV - exit 1 + # Sanity check that TEST partition is not mounted at another mount point + # or as another fs type + _check_mounted_on TEST_DEV $TEST_DEV TEST_DIR $TEST_DIR $FSTYP || exit 1 + if [ -n "$SCRATCH_DEV" ]; then + # Sanity check that SCRATCH partition is not mounted at another + # mount point, because it is about to be unmounted and formatted. + # Another fs type for scratch is fine (bye bye old fs type). + _check_mounted_on SCRATCH_DEV $SCRATCH_DEV SCRATCH_MNT $SCRATCH_MNT + [ $? -le 1 ] || exit 1 fi + # Figure out if we need to add -F ("foreign", deprecated) option to xfs_io $XFS_IO_PROG -c stat $TEST_DIR 2>&1 | grep -q "is not on an XFS filesystem" && \ export XFS_IO_PROG="$XFS_IO_PROG -F"
Signature · 1 line
2.7.4
PATCH v5 2/10 fstests: use _test_mount() consistently
In reply to Amir GoldsteinOn start of every test run and on every test, in init_rc() helper, the test partition is verified to be mounted, or is mounted by the helper _test_mount().
_test_mount() uses mount options $TEST_FS_MOUNT_OPTS and not $MOUNT_OPTIONS like _scratch_mount() does.
_test_cycle_mount(), which is called by some tests uses the _test_mount() helper as well.
Contrary to those cases, in _require_test() helper, if test partition is not mounted, the helper _mount_or_remount_rw() is called to mount the test partition with $MOUNT_OPTIONS. Although this should never happen, because of the test in init_rc(), this case is inconsistent with the rest of the code, so it has been changed to use _test_mount() as it should.
When running tests with a multi section configuration, and either FSTYP or MOUNT_OPTIONS change between sections, the helper _test_unmount() is called to unmount the old test mount and then _mount_or_remount_rw() is called to mount it again with new FSTYP and/or MOUNT_OPTIONS.
This is again inconsistent with the rest of the code, so was changed to use _test_mount() and instead of checking if MOUNT_OPTIONS have changed between sections, we check if TEST_FS_MOUNT_OPTS were changed between sections. Otherwise, we can leave the test partition mounted.
This change is needed to support overlay base fs mount and for multi section config files which include overlay FSTYP.
--- check | 16 ++++++++-------- common/rc | 6 +++--- 2 files changed, 11 insertions(+), 11 deletions(-)
check +8 -8
@@ -474,7 +474,7 @@ fi for section in $HOST_OPTIONS_SECTIONS; do OLD_FSTYP=$FSTYP - OLD_MOUNT_OPTIONS=$MOUNT_OPTIONS + OLD_TEST_FS_MOUNT_OPTS=$TEST_FS_MOUNT_OPTS get_next_config $section # Do we need to run only some sections ?
@@ -527,18 +527,18 @@ for section in $HOST_OPTIONS_SECTIONS; do status=1 exit fi - out=`_mount_or_remount_rw "$MOUNT_OPTIONS" $TEST_DEV $TEST_DIR` - if [ $? -ne 1 ]; then - echo $out + if ! _test_mount + then + echo "check: failed to mount $TEST_DEV on $TEST_DIR" status=1 exit fi _prepare_test_list - elif [ "$OLD_MOUNT_OPTIONS" != "$MOUNT_OPTIONS" ]; then + elif [ "$OLD_TEST_FS_MOUNT_OPTS" != "$TEST_FS_MOUNT_OPTS" ]; then _test_unmount 2> /dev/null - out=`_mount_or_remount_rw "$MOUNT_OPTIONS" $TEST_DEV $TEST_DIR` - if [ $? -ne 1 ]; then - echo $out + if ! _test_mount + then + echo "check: failed to mount $TEST_DEV on $TEST_DIR" status=1 exit fi
common/rc +3 -3
@@ -1526,9 +1526,9 @@ _require_test() [ $err -le 1 ] || exit 1 if [ $err -ne 0 ] then - out=`_mount_or_remount_rw "$MOUNT_OPTIONS" $TEST_DEV $TEST_DIR` - if [ $? -ne 1 ]; then - echo $out + if ! _test_mount + then + echo "!!! failed to mount $TEST_DEV on $TEST_DIR" exit 1 fi fi
Signature · 1 line
2.7.4
PATCH v5 3/10 fstests: canonicalize mount points on every config section
In reply to Amir GoldsteinCreate helper _canonicalize_mountpoint() to check and canonicalize a mount point path.
Use helper to canonicalize TEST_DIR and SCRATCH_MNT after parse of every config section.
This is needed for overlay base fs mount.
--- common/config | 34 +++++++++++++++++++++------------- 1 file changed, 21 insertions(+), 13 deletions(-)
common/config +21 -13
@@ -455,6 +455,20 @@ _check_device() fi } +# check and return a canonical mount point path +_canonicalize_mountpoint() +{ + local name=$1 + local dir=$2 + + if [ ! -d "$dir" ]; then + _fatal "common/config: $name ($dir) is not a directory" + fi + + # this follows symlinks and removes all trailing "/"s + readlink -e "$dir" +} + # Parse config section options. This function will parse all the configuration # within a single section which name is passed as an argument. For section # name format see comments in get_config_sections().
@@ -543,10 +557,7 @@ get_next_config() { fi _check_device TEST_DEV required $TEST_DEV - if [ ! -d "$TEST_DIR" ]; then - echo "common/config: Error: \$TEST_DIR ($TEST_DIR) is not a directory" - exit 1 - fi + export TEST_DIR=`_canonicalize_mountpoint TEST_DIR $TEST_DIR` # a btrfs tester will set only SCRATCH_DEV_POOL, we will put first of its dev # to SCRATCH_DEV and rest to SCRATCH_DEV_POOL to maintain the backward compatibility
@@ -561,10 +572,7 @@ get_next_config() { fi _check_device SCRATCH_DEV optional $SCRATCH_DEV - if [ ! -z "$SCRATCH_MNT" -a ! -d "$SCRATCH_MNT" ]; then - echo "common/config: Error: \$SCRATCH_MNT ($SCRATCH_MNT) is not a directory" - exit 1 - fi + export SCRATCH_MNT=`_canonicalize_mountpoint SCRATCH_MNT $SCRATCH_MNT` if [ -n "$USE_EXTERNAL" ]; then _check_device TEST_RTDEV optional $TEST_RTDEV
@@ -591,12 +599,12 @@ if [ -z "$CONFIG_INCLUDED" ]; then [ -z "$TEST_FS_MOUNT_OPTS" ] && _test_mount_opts [ -z "$MKFS_OPTIONS" ] && _mkfs_opts [ -z "$FSCK_OPTIONS" ] && _fsck_opts +else + # canonicalize the mount points + # this follows symlinks and removes all trailing "/"s + export TEST_DIR=`_canonicalize_mountpoint TEST_DIR $TEST_DIR` + export SCRATCH_MNT=`_canonicalize_mountpoint SCRATCH_MNT $SCRATCH_MNT` fi -# canonicalize the mount points -# this follows symlinks and removes all trailing "/"s -export TEST_DIR=`readlink -e "$TEST_DIR"` -export SCRATCH_MNT=`readlink -e "$SCRATCH_MNT"` - # make sure this script returns success /bin/true
Signature · 1 line
2.7.4
PATCH v5 4/10 fstests: fix test and scratch filters for overlapping DEV/MNT paths
In reply to Amir GoldsteinWhen configuring overlay base fs, TEST_DEV/DIR and SCRATCH_DEV/MNT are derived from the base fs mount points, where *_DEV are the path of the base fs mount point and TEST_DIR/SCRATCH_MNT are a directory under the base fs mount point.
This means that the overlay DEV paths are prefixes of the overlay mount points.
Fix the test and sctach filters to check if TEST_DEV/SCRATCH_DEV is a substring of TEST_DIR/SCRATCH_MNT and try and match the longer string first.
--- common/filter | 26 ++++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-)
common/filter +22 -4
@@ -280,14 +280,32 @@ _filter_xfs_io_pages_modified() _filter_test_dir() { - sed -e "s,$TEST_DEV,TEST_DEV,g" -e "s,$TEST_DIR,TEST_DIR,g" + if ( echo $TEST_DIR | grep -q $TEST_DEV ); then + # TEST_DEV is substr of TEST_DIR (e.g. /mnt and /mnt/ovl-mnt) + # substitute TEST_DIR first + sed -e "s,$TEST_DIR,TEST_DIR,g" \ + -e "s,$TEST_DEV,TEST_DEV,g" + else + # TEST_DIR maybe a substr of TEST_DIR (e.g. /vdc and /dev/vdc) + # substitute TEST_DEV first + sed -e "s,$TEST_DEV,TEST_DEV,g" \ + -e "s,$TEST_DIR,TEST_DIR,g" + fi } _filter_scratch() { - sed -e "s,$SCRATCH_DEV,SCRATCH_DEV,g" \ - -e "s,$SCRATCH_MNT,SCRATCH_MNT,g" \ - -e "/.use_space/d" + if ( echo $SCRATCH_MNT | grep -q $SCRATCH_DEV ); then + # SCRATCH_DEV is substr of SCRATCH_MNT + sed -e "s,$SCRATCH_MNT,SCRATCH_MNT,g" \ + -e "s,$SCRATCH_DEV,SCRATCH_DEV,g" \ + -e "/.use_space/d" + else + # SCRATCH_MNT maybe a substr of SCRATCH_DEV + sed -e "s,$SCRATCH_DEV,SCRATCH_DEV,g" \ + -e "s,$SCRATCH_MNT,SCRATCH_MNT,g" \ + -e "/.use_space/d" + fi } # Turn any device in the scratch pool into SCRATCH_DEV
Signature · 1 line
2.7.4
PATCH v5 5/10 fstests: allow overlay SCRATCH_DEV to be a base fs mount point
In reply to Amir GoldsteinWhen configure SCRATCH_DEV to a mount point (and not to a directory therein) then user will get a false positive error in scratch tests:
$SCRATCH_DEV=/mnt/base/scratch is mounted but not on $SCRATCH_MNT=/mnt/scratch Already mounted result:
/dev/sda6 on /mnt/base/scratch type xfs (rw,relatime,attr2,inode64,noquota)
This is due to the wrong grep -F $SCRATCH_DEV which matches the mount point instead of the device in that mount.
Fix _check_mounted_on() to grep the pattern "$dev on " and "$dev on $mnt" instead of just grepping for "$dev" and "$mnt" without the " on " anchor.
--- common/rc | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
common/rc +5 -3
@@ -1368,12 +1368,14 @@ _check_mounted_on() local type=$5 # Note that we use -F here so grep doesn't try to interpret an NFS over - # IPv6 server as a regular expression - local mount_rec=`_mount | grep -F "$dev"` + # IPv6 server as a regular expression. Because of that, we cannot use + # ^$dev so we use "$dev on " to avoid matching $dev to mount point field + # for overlay case, where $dev is a directory. + local mount_rec=`_mount | grep -F "$dev on "` [ -n "$mount_rec" ] || return 1 # 1 = not mounted # if it's mounted, make sure its on $mnt - if ! (echo $mount_rec | grep -q "$mnt") + if ! (echo $mount_rec | grep -q "$dev on $mnt") then echo "$devname=$dev is mounted but not on $mntname=$mnt - aborting" echo "Already mounted result:"
Signature · 1 line
2.7.4
PATCH v5 6/10 generic/064: access SCRATCH_MNT after _scratch_mount
In reply to Amir GoldsteinTest is calling _get_block_size $SCRATCH_MNT before _scratch_mount. This results in block size of the wrong fs and a failure with overlay base fs setup.
--- tests/generic/064 | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
tests/generic/064 +6 -4
@@ -46,14 +46,16 @@ _require_scratch _require_xfs_io_command "fiemap" _require_xfs_io_command "finsert" _require_xfs_io_command "fcollapse" -src=$SCRATCH_MNT/testfile -dest=$SCRATCH_MNT/testfile.dest -BLOCKS=100 -BSIZE=`_get_block_size $SCRATCH_MNT` + rm -f $seqres.full _scratch_mkfs >> $seqres.full 2>&1 || _fail "mkfs failed" _scratch_mount || _fail "mount failed" + +src=$SCRATCH_MNT/testfile +dest=$SCRATCH_MNT/testfile.dest +BLOCKS=100 +BSIZE=`_get_block_size $SCRATCH_MNT` length=$(($BLOCKS * $BSIZE)) # Write file
Signature · 1 line
2.7.4
PATCH v5 7/10 overlay: rename OVERLAY_LOWER/UPPER/WORK_DIR
In reply to Amir GoldsteinAs preparation step for configuring test/scratch base fs for overlay tests:
- Rename OVERLAY_LOWER/UPPER/WORK_DIR => OVL_LOWER/UPPER/WORK because we want to use OVL_ prefix for all base fs vars
- Prepend "ovl-" prefix to lower/upper/work path values to isolate the overlay test dirs when running on a base fs that is also used to run non overlay tests
- Make those vars values non-configurable, because lower/upper/work directory names are an internal test detail which should not concern the user and because we wish to simplify and document the overlay tests setup
- Add helper _filter_ovl_dirs and use it to filter output of test overlay/023 which matches the constant string "work"
--- common/config | 8 +++++--- common/filter | 7 +++++++ common/rc | 19 +++++++++++++------ tests/overlay/001 | 2 +- tests/overlay/002 | 2 +- tests/overlay/003 | 2 +- tests/overlay/004 | 4 ++-- tests/overlay/006 | 10 +++++----- tests/overlay/008 | 8 ++++---- tests/overlay/009 | 2 +- tests/overlay/010 | 8 ++++---- tests/overlay/011 | 4 ++-- tests/overlay/012 | 4 ++-- tests/overlay/013 | 4 ++-- tests/overlay/015 | 2 +- tests/overlay/016 | 2 +- tests/overlay/017 | 2 +- tests/overlay/018 | 2 +- tests/overlay/019 | 2 +- tests/overlay/020 | 2 +- tests/overlay/021 | 2 +- tests/overlay/022 | 2 +- tests/overlay/023 | 4 ++-- tests/overlay/023.out | 2 +- tests/overlay/024 | 2 +- tests/overlay/027 | 2 +- tests/overlay/028 | 2 +- tests/overlay/029 | 4 ++-- 28 files changed, 66 insertions(+), 50 deletions(-)
common/config +5 -3
@@ -77,9 +77,11 @@ export XFS_MKFS_OPTIONS=${XFS_MKFS_OPTIONS:=-bsize=4096} export TIME_FACTOR=${TIME_FACTOR:=1} export LOAD_FACTOR=${LOAD_FACTOR:=1} export DEBUGFS_MNT=${DEBUGFS_MNT:="/sys/kernel/debug"} -export OVERLAY_UPPER_DIR=${OVERLAY_UPPER_DIR:="upper"} -export OVERLAY_LOWER_DIR=${OVERLAY_LOWER_DIR:="lower"} -export OVERLAY_WORK_DIR=${OVERLAY_WORK_DIR:="work"} + +# some constants for overlayfs setup +export OVL_UPPER="ovl-upper" +export OVL_LOWER="ovl-lower" +export OVL_WORK="ovl-work" export PWD=`pwd` #export DEBUG=${DEBUG:=...} # arbitrary CFLAGS really.
common/filter +7 -0
@@ -424,5 +424,12 @@ _filter_lostfound() sed -e '/^lost+found$/d' } +_filter_ovl_dirs() +{ + sed -e "s,$OVL_LOWER,OVL_LOWER,g" \ + -e "s,$OVL_UPPER,OVL_UPPER,g" \ + -e "s,$OVL_WORK,OVL_WORK,g" +} + # make sure this script returns success /bin/true
common/rc +13 -6
@@ -277,7 +277,7 @@ _common_dev_mount_options() _overlay_basic_mount_options() { - echo "-o lowerdir=$1/$OVERLAY_LOWER_DIR,upperdir=$1/$OVERLAY_UPPER_DIR,workdir=$1/$OVERLAY_WORK_DIR" + echo "-o lowerdir=$1/$OVL_LOWER,upperdir=$1/$OVL_UPPER,workdir=$1/$OVL_WORK" } _overlay_mount_options()
@@ -336,6 +336,15 @@ _overlay_mount_dirs() -o workdir=$workdir $* } +_overlay_mkdirs() +{ + local dir=$1 + + mkdir -p $dir/$OVL_UPPER + mkdir -p $dir/$OVL_LOWER + mkdir -p $dir/$OVL_WORK +} + # Given a dir, set up 3 subdirectories and mount on the given mnt. # The dir is used as the mount device so it can be seen from df or mount _overlay_mount()
@@ -346,12 +355,10 @@ _overlay_mount() _supports_filetype $dir || _notrun "upper fs needs to support d_type" - mkdir -p $dir/$OVERLAY_UPPER_DIR - mkdir -p $dir/$OVERLAY_LOWER_DIR - mkdir -p $dir/$OVERLAY_WORK_DIR + _overlay_mkdirs $dir - _overlay_mount_dirs $dir/$OVERLAY_LOWER_DIR $dir/$OVERLAY_UPPER_DIR \ - $dir/$OVERLAY_WORK_DIR $OVERLAY_MOUNT_OPTIONS \ + _overlay_mount_dirs $dir/$OVL_LOWER $dir/$OVL_UPPER \ + $dir/$OVL_WORK $OVERLAY_MOUNT_OPTIONS \ $SELINUX_MOUNT_OPTIONS $* $dir $mnt }
tests/overlay/001 +1 -1
@@ -60,7 +60,7 @@ _scratch_mkfs _require_fs_space $SCRATCH_DEV $((4*1024*1024*2 + 8)) # Create test files with different sizes in lower dir -lowerdir=$SCRATCH_DEV/$OVERLAY_LOWER_DIR +lowerdir=$SCRATCH_DEV/$OVL_LOWER mkdir -p $lowerdir touch $lowerdir/zero_size $XFS_IO_PROG -fc "pwrite 0 4k" $lowerdir/less_than_4g >>$seqres.full
tests/overlay/002 +1 -1
@@ -59,7 +59,7 @@ _require_scratch _scratch_mkfs # Create our test file. -lowerdir=$SCRATCH_DEV/$OVERLAY_LOWER_DIR +lowerdir=$SCRATCH_DEV/$OVL_LOWER mkdir -p $lowerdir touch $lowerdir/foobar
tests/overlay/003 +1 -1
@@ -60,7 +60,7 @@ _require_scratch _scratch_mkfs # Create test files dirs in lower dir -lowerdir=${SCRATCH_DEV}/${OVERLAY_LOWER_DIR} +lowerdir=${SCRATCH_DEV}/${OVL_LOWER} mkdir -p $lowerdir touch ${lowerdir}/test_file
tests/overlay/004 +2 -2
@@ -53,8 +53,8 @@ _require_user _scratch_mkfs # Create test file in lower dir -lowerdir=${SCRATCH_DEV}/${OVERLAY_LOWER_DIR} -upperdir=${SCRATCH_DEV}/${OVERLAY_UPPER_DIR} +lowerdir=${SCRATCH_DEV}/${OVL_LOWER} +upperdir=${SCRATCH_DEV}/${OVL_UPPER} mkdir -p $lowerdir touch ${lowerdir}/attr_file1 chmod 000 ${lowerdir}/attr_file1
tests/overlay/006 +5 -5
@@ -57,15 +57,15 @@ echo "Silence is golden" _scratch_mkfs # Create test file/dir before mount -mkdir $SCRATCH_DEV/$OVERLAY_LOWER_DIR -mkdir $SCRATCH_DEV/$OVERLAY_UPPER_DIR -touch $SCRATCH_DEV/$OVERLAY_LOWER_DIR/lowertestfile -mkdir $SCRATCH_DEV/$OVERLAY_UPPER_DIR/uppertestdir +mkdir -p $SCRATCH_DEV/$OVL_LOWER +mkdir -p $SCRATCH_DEV/$OVL_UPPER +touch $SCRATCH_DEV/$OVL_LOWER/lowertestfile +mkdir $SCRATCH_DEV/$OVL_UPPER/uppertestdir _scratch_mount # rename lowertestfile to uppertestdir, this triggers copyup and creates -# whiteout in $OVERLAY_UPPER_DIR +# whiteout in $OVL_UPPER mv $SCRATCH_MNT/lowertestfile $SCRATCH_MNT/uppertestdir # the lowertestfile can be removed rm $SCRATCH_MNT/uppertestdir/lowertestfile
tests/overlay/008 +4 -4
@@ -56,14 +56,14 @@ _require_user _scratch_mkfs # Create test file on lower dir, and chown to fsgqa user -lowerdir=$SCRATCH_DEV/$OVERLAY_LOWER_DIR -mkdir $lowerdir +lowerdir=$SCRATCH_DEV/$OVL_LOWER +mkdir -p $lowerdir touch $lowerdir/testfile chown fsgqa:fsgqa $lowerdir/testfile # chown upperdir to fsgqa user, so new file/dir can be created by the user -upperdir=$SCRATCH_DEV/$OVERLAY_UPPER_DIR -mkdir $upperdir +upperdir=$SCRATCH_DEV/$OVL_UPPER +mkdir -p $upperdir chown fsgqa:fsgqa $upperdir _scratch_mount
tests/overlay/009 +1 -1
@@ -54,7 +54,7 @@ _require_scratch _scratch_mkfs # Create test file in lowerdir -lowerdir=$SCRATCH_DEV/$OVERLAY_LOWER_DIR +lowerdir=$SCRATCH_DEV/$OVL_LOWER mkdir -p $lowerdir touch $lowerdir/testfile
tests/overlay/010 +4 -4
@@ -55,10 +55,10 @@ _scratch_mkfs # Need two lower dirs in this test, and we mount overlay ourselves, # create upper and workdir as well -lowerdir1=$SCRATCH_DEV/$OVERLAY_LOWER_DIR.1 -lowerdir2=$SCRATCH_DEV/$OVERLAY_LOWER_DIR.2 -upperdir=$SCRATCH_DEV/$OVERLAY_UPPER_DIR -workdir=$SCRATCH_DEV/$OVERLAY_WORK_DIR +lowerdir1=$SCRATCH_DEV/$OVL_LOWER.1 +lowerdir2=$SCRATCH_DEV/$OVL_LOWER.2 +upperdir=$SCRATCH_DEV/$OVL_UPPER +workdir=$SCRATCH_DEV/$OVL_WORK mkdir -p $lowerdir1 $lowerdir2 $upperdir $workdir # One lowerdir contains test dir and test files, the other contains whiteout
tests/overlay/011 +2 -2
@@ -58,8 +58,8 @@ _require_attrs _scratch_mkfs # Create test dir on upper and make it opaque by setting proper xattr -lowerdir=$SCRATCH_DEV/$OVERLAY_LOWER_DIR -upperdir=$SCRATCH_DEV/$OVERLAY_UPPER_DIR +lowerdir=$SCRATCH_DEV/$OVL_LOWER +upperdir=$SCRATCH_DEV/$OVL_UPPER mkdir -p $lowerdir/testdir mkdir -p $upperdir/testdir $SETFATTR_PROG -n "trusted.overlay.opaque" -v "y" $upperdir/testdir
tests/overlay/012 +2 -2
@@ -55,8 +55,8 @@ _require_scratch # remove all files from previous runs _scratch_mkfs -lowerdir=$SCRATCH_DEV/$OVERLAY_LOWER_DIR -upperdir=$SCRATCH_DEV/$OVERLAY_UPPER_DIR +lowerdir=$SCRATCH_DEV/$OVL_LOWER +upperdir=$SCRATCH_DEV/$OVL_UPPER mkdir -p $lowerdir/test _scratch_mount
tests/overlay/013 +2 -2
@@ -54,8 +54,8 @@ _require_test_program "t_truncate_self" _scratch_mkfs # copy test program to lower and upper dir -lowerdir=$SCRATCH_DEV/$OVERLAY_LOWER_DIR -upperdir=$SCRATCH_DEV/$OVERLAY_UPPER_DIR +lowerdir=$SCRATCH_DEV/$OVL_LOWER +upperdir=$SCRATCH_DEV/$OVL_UPPER mkdir -p $lowerdir mkdir -p $upperdir cp $here/src/t_truncate_self $lowerdir/test_lower
tests/overlay/015 +1 -1
@@ -57,7 +57,7 @@ _scratch_mkfs umask 022 # Create test dir in lower dir and set sgid bit -lowerdir=$SCRATCH_DEV/$OVERLAY_LOWER_DIR +lowerdir=$SCRATCH_DEV/$OVL_LOWER mkdir -p $lowerdir/dir chown $qa_user:$qa_group $lowerdir/dir chmod 2775 $lowerdir/dir
tests/overlay/016 +1 -1
@@ -57,7 +57,7 @@ rm -f $seqres.full _scratch_mkfs >>$seqres.full 2>&1 # Create our test files. -lowerdir=$SCRATCH_DEV/$OVERLAY_LOWER_DIR +lowerdir=$SCRATCH_DEV/$OVL_LOWER mkdir -p $lowerdir echo "This is old news" > $lowerdir/foo echo "This is old news" > $lowerdir/bar
tests/overlay/017 +1 -1
@@ -60,7 +60,7 @@ _scratch_mkfs >>$seqres.full 2>&1 # Not dealing with hardlinks here, because there is more to test # then stable inode number. # Hardlinks will get a test of their own. -lowerdir=$SCRATCH_DEV/$OVERLAY_LOWER_DIR +lowerdir=$SCRATCH_DEV/$OVL_LOWER mkdir -p $lowerdir mkdir $lowerdir/dir touch $lowerdir/file
tests/overlay/018 +1 -1
@@ -55,7 +55,7 @@ rm -f $seqres.full _scratch_mkfs >>$seqres.full 2>&1 # Create 2 hardlinked files in lower -lowerdir=$SCRATCH_DEV/$OVERLAY_LOWER_DIR +lowerdir=$SCRATCH_DEV/$OVL_LOWER mkdir -p $lowerdir echo "patient zero" >> $lowerdir/foo ln $lowerdir/foo $lowerdir/bar
tests/overlay/019 +1 -1
@@ -51,7 +51,7 @@ _require_scratch # Remove all files from previous tests _scratch_mkfs -lowerdir=$SCRATCH_DEV/$OVERLAY_LOWER_DIR +lowerdir=$SCRATCH_DEV/$OVL_LOWER mkdir -p $lowerdir _scratch_mount
tests/overlay/020 +1 -1
@@ -55,7 +55,7 @@ _require_scratch # Remove all files from previous tests _scratch_mkfs -lowerdir=$SCRATCH_DEV/$OVERLAY_LOWER_DIR +lowerdir=$SCRATCH_DEV/$OVL_LOWER mkdir -p $lowerdir/dir _scratch_mount
tests/overlay/021 +1 -1
@@ -60,7 +60,7 @@ _scratch_mkfs # conservative and reserve space for 16 data copy ups per directory. _require_fs_space $SCRATCH_DEV $((16*(16+32+64+128)*1024)) -lowerdir=$SCRATCH_DEV/$OVERLAY_LOWER_DIR +lowerdir=$SCRATCH_DEV/$OVL_LOWER mkdir -p $lowerdir testdir=arena
tests/overlay/022 +1 -1
@@ -59,7 +59,7 @@ _require_scratch # Remove all files from previous tests _scratch_mkfs -upperdir=$SCRATCH_DEV/$OVERLAY_UPPER_DIR +upperdir=$SCRATCH_DEV/$OVL_UPPER mkdir -p $upperdir/upper mkdir -p $upperdir/work # mount overlay with dirs in upper
tests/overlay/023 +2 -2
@@ -63,14 +63,14 @@ _require_scratch _scratch_mkfs # setting acls before mount -wkdir=$SCRATCH_DEV/$OVERLAY_WORK_DIR +wkdir=$SCRATCH_DEV/$OVL_WORK mkdir -p $wkdir setfacl -d -m o::rwx $wkdir _scratch_mount # getting acls, ACL set on workdir should be cleaned -getfacl -p $wkdir/work 2>&1 | _filter_scratch +getfacl -p $wkdir/work 2>&1 | _filter_scratch | _filter_ovl_dirs # success, all done status=0
tests/overlay/023.out +1 -1
@@ -1,5 +1,5 @@ QA output created by 023 -# file: SCRATCH_DEV/work/work +# file: SCRATCH_DEV/OVL_WORK/work # owner: root # group: root user::---
tests/overlay/024 +1 -1
@@ -61,7 +61,7 @@ _require_scratch _scratch_mkfs # making workdir -wkdir=$SCRATCH_DEV/$OVERLAY_WORK_DIR +wkdir=$SCRATCH_DEV/$OVL_WORK mkdir -p $wkdir/work/foo _scratch_mount
tests/overlay/027 +1 -1
@@ -63,7 +63,7 @@ _require_chattr _scratch_mkfs # Preparing immutable file -upperdir=$SCRATCH_DEV/$OVERLAY_UPPER_DIR +upperdir=$SCRATCH_DEV/$OVL_UPPER mkdir -p $upperdir touch $upperdir/foo $CHATTR_PROG +i $upperdir/foo
tests/overlay/028 +1 -1
@@ -61,7 +61,7 @@ _require_command "$FLOCK_PROG" flock # Remove all files from previous tests _scratch_mkfs -lowerdir=$SCRATCH_DEV/$OVERLAY_LOWER_DIR +lowerdir=$SCRATCH_DEV/$OVL_LOWER mkdir -p $lowerdir touch $lowerdir/foo
tests/overlay/029 +2 -2
@@ -65,8 +65,8 @@ _require_scratch _scratch_mkfs # Preparing files -upperdir=$SCRATCH_DEV/$OVERLAY_UPPER_DIR -lowerdir=$SCRATCH_DEV/$OVERLAY_LOWER_DIR +upperdir=$SCRATCH_DEV/$OVL_UPPER +lowerdir=$SCRATCH_DEV/$OVL_LOWER mkdir -p $upperdir/up echo foo > $upperdir/up/foo mkdir -p $lowerdir/low
Signature · 1 line
2.7.4
PATCH v5 8/10 overlay: configure TEST/SCRATCH vars to base fs
In reply to Amir GoldsteinInstead of setting the vars TEST/SCRATCH_DEV to overlay base dirs, allow setting them to block devices to configure the base fs partition, where overlay dirs will be created.
For example, the following config file can be used to run tests on xfs test/scratch partitions:
TEST_DEV=/dev/sda5
TEST_DIR=/mnt/test
SCRATCH_DEV=/dev/sda6
SCRATCH_MNT=/mnt/scratch
FSTYP=xfs
Using the same config file, but executing './check -overlay' will use the same partitions as base fs for overlayfs directories and set TEST_DIR/SCRATCH_MNT values to overlay mount points, i.e.:/mnt/test/ovl-mnt and /mnt/scratch/ovl-mnt.
The base fs should be pre-formatted and mounted when starting the test. An upcoming change is going to support mount/umount of base fs.
The new vars OVL_BASE_SCRATCH_MNT/TEST_DIR are set to point at the overlayfs base dirs in either legacy or new config method. Tests should always use these vars and not the legacy SCRATCH/TEST_DEV vars when referring to overlay base dir.
--- README | 16 ++++---- README.overlay | 18 +++++++++ check | 2 +- common/config | 124 +++++++++++++++++++++++++++++++++++++++++++++++++++++---- common/rc | 28 ++++++++----- 5 files changed, 162 insertions(+), 26 deletions(-) create mode 100644 README.overlay
README +9 -7
@@ -117,16 +117,18 @@ Running tests: - By default the tests suite will run xfs tests: - ./check '*/001' '*/002' '*/003' - ./check '*/06?' - - You can explicitly specify NFS/CIFS/UDF, otherwise the filesystem type will - be autodetected from $TEST_DEV: - ./check -nfs [test(s)] - Groups of tests maybe ran by: ./check -g [group(s)] See the 'group' file for details on groups - - for udf tests: ./check -udf [test(s)] - Running all the udf tests: ./check -udf -g udf - - for running nfs tests: ./check -nfs [test(s)] - - for running cifs/smb3 tests: ./check -cifs [test(s)] - To randomize test order: ./check -r [test(s)] + - You can explicitly specify NFS/CIFS/UDF/OVERLAY, otherwise + the filesystem type will be autodetected from $TEST_DEV: + - for running nfs tests: ./check -nfs [test(s)] + - for running cifs/smb3 tests: ./check -cifs [test(s)] + - for udf tests: ./check -udf [test(s)] + Running all the udf tests: ./check -udf -g udf + - for overlay tests: ./check -overlay [test(s)] + The TEST and SCRATCH partitions should be pre-formatted + with another base fs, where the overlay dirs will be created The check script tests the return value of each script, and
README.overlay new file +18 -0
@@ -0,0 +1,18 @@ + +To run xfstest on overlayfs, configure the variables of TEST and SCRATCH +partitions to be used as the "base fs" and run './check -overlay'. + +For example, the following config file can be used to run tests on +xfs test/scratch partitions: + + TEST_DEV=/dev/sda5 + TEST_DIR=/mnt/test + SCRATCH_DEV=/dev/sda6 + SCRATCH_MNT=/mnt/scratch + FSTYP=xfs + +Using the same config file, but executing './check -overlay' will +use the same partitions as base fs for overlayfs directories +and set TEST_DIR/SCRATCH_MNT values to overlay mount points, i.e.: +/mnt/test/ovl-mnt and /mnt/scratch/ovl-mnt, for the context of +individual tests.
check +1 -1
@@ -258,7 +258,7 @@ while [ $# -gt 0 ]; do -nfs) FSTYP=nfs ;; -cifs) FSTYP=cifs ;; - -overlay) FSTYP=overlay ;; + -overlay) FSTYP=overlay; export OVERLAY=true ;; -tmpfs) FSTYP=tmpfs ;; -g) group=$2 ; shift ;
common/config +117 -7
173 lines not on this page · raw message
common/rc +17 -11
@@ -292,7 +292,7 @@ _scratch_mount_options() _scratch_options mount if [ "$FSTYP" == "overlay" ]; then - echo `_overlay_mount_options $SCRATCH_DEV` + echo `_overlay_mount_options $OVL_BASE_SCRATCH_MNT` return 0 fi echo `_common_dev_mount_options $*` $SCRATCH_OPTIONS \
@@ -343,9 +343,10 @@ _overlay_mkdirs() mkdir -p $dir/$OVL_UPPER mkdir -p $dir/$OVL_LOWER mkdir -p $dir/$OVL_WORK + mkdir -p $dir/$OVL_MNT } -# Given a dir, set up 3 subdirectories and mount on the given mnt. +# Given a base fs dir, set up overlay directories and mount on the given mnt. # The dir is used as the mount device so it can be seen from df or mount _overlay_mount() {
@@ -364,12 +365,12 @@ _overlay_mount() _overlay_test_mount() { - _overlay_mount $TEST_DEV $TEST_DIR $* + _overlay_mount $OVL_BASE_TEST_DIR $TEST_DIR $* } _overlay_scratch_mount() { - _overlay_mount $SCRATCH_DEV $SCRATCH_MNT $* + _overlay_mount $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT $* } _overlay_test_unmount()
@@ -684,10 +685,12 @@ _scratch_cleanup_files() { case $FSTYP in overlay) - # $SCRATCH_DEV is a valid directory in overlay case - rm -rf $SCRATCH_DEV/* + # Avoid rm -rf /* if we messed up + [ -n "$OVL_BASE_SCRATCH_MNT" ] || return 1 + rm -rf $OVL_BASE_SCRATCH_MNT/* ;; *) + [ -n "$SCRATCH_MNT" ] || return 1 _scratch_mount rm -rf $SCRATCH_MNT/* _scratch_unmount
@@ -1425,10 +1428,13 @@ _require_scratch_nocheck() fi ;; overlay) - if [ -z "$SCRATCH_DEV" -o ! -d "$SCRATCH_DEV" ]; then - _notrun "this test requires a valid \$SCRATCH_DEV as ovl base dir" + if [ -z "$OVL_BASE_SCRATCH_MNT" -o ! -d "$OVL_BASE_SCRATCH_MNT" ]; then + _notrun "this test requires a valid \$OVL_BASE_SCRATCH_MNT as ovl base dir" fi - if [ ! -d "$SCRATCH_MNT" ]; then + # if $SCRATCH_MNT is derived from $OVL_BASE_SCRATCH_MNT then + # don't check $SCRATCH_MNT dir here because base fs may not be mounted + # and we will create the mount point anyway on _overlay_mount + if [ "$SCRATCH_MNT" != "$OVL_BASE_SCRATCH_MNT/$OVL_MNT" -a ! -d "$SCRATCH_MNT" ]; then _notrun "this test requires a valid \$SCRATCH_MNT" fi ;;
@@ -1501,8 +1507,8 @@ _require_test() fi ;; overlay) - if [ -z "$TEST_DEV" -o ! -d "$TEST_DEV" ]; then - _notrun "this test requires a valid \$TEST_DEV as ovl base dir" + if [ -z "$OVL_BASE_TEST_DIR" -o ! -d "$OVL_BASE_TEST_DIR" ]; then + _notrun "this test requires a valid \$TEST_DIR as ovl base dir" fi if [ ! -d "$TEST_DIR" ]; then _notrun "this test requires a valid \$TEST_DIR"
Signature · 1 line
2.7.4
PATCH v5 9/10 overlay: mount/unmount base fs before/after running tests
In reply to Amir GoldsteinWhen TEST/SCRATCH_DEV are configured to the base fs block device, use this information to mount base fs before running tests, unmount it after running tests and cycle on _test_cycle_mount along with the overlay mounts.
This helps catching overlayfs bugs related to leaking objects in underlying (base) fs.
To preserve expected tests behavior, the semantics are:
- _scratch_mkfs mounts the base fs, cleans all files, creates lower/upper dirs and keeps base fs mounted
- _scratch_mount mounts base fs (if needed) and mounts overlay
- _scratch_unmount unmounts overlay and base fs
Tests that use _scratch_unmount to unmount a custom overlay mount and expect to have access to overlay base dir, were fixed to use explicit umount $SCRATCH_MNT instead.
The overlay test itself, does not support formatting the base fs, so config options like MKFS_OPTIONS and FSCK_OPTIONS are ignored on 'check -overlay'.
The config option MOUNT_OPTIONS is used to mount the base scratch fs. The config option TEST_FS_MOUNT_OPTS is used to mount the base test fs. The config option OVERLAY_MOUNT_OPTIONS is used to mount both test and scratch overlay mounts.
--- README.overlay | 28 +++++++++++++++++++++++ common/rc | 67 +++++++++++++++++++++++++++++++++++++++++++++++++++---- tests/overlay/003 | 3 ++- tests/overlay/004 | 3 ++- tests/overlay/014 | 5 +++-- 5 files changed, 98 insertions(+), 8 deletions(-)
README.overlay +28 -0
@@ -16,3 +16,31 @@ use the same partitions as base fs for overlayfs directories and set TEST_DIR/SCRATCH_MNT values to overlay mount points, i.e.: /mnt/test/ovl-mnt and /mnt/scratch/ovl-mnt, for the context of individual tests. + +'./check -overlay' does not support mkfs and fsck on the base fs, so +the base fs should be pre-formatted before starting the -overlay run. +An easy way to accomplish this is by running './check <some test>' once, +before running './check -overlay'. + +Because of the lack of mkfs support, multi-section config files are only +partly supported with './check -overlay'. Only multi-section files that +do not change FSTYP and MKFS_OPTIONS can be safely used with -overlay. + +For example, the following multi-section config file can be used to +run overlay tests on the same base fs, but with different mount options: + + [xfs] + TEST_DEV=/dev/sda5 + TEST_DIR=/mnt/test + SCRATCH_DEV=/dev/sda6 + SCRATCH_MNT=/mnt/scratch + FSTYP=xfs + + [xfs_pquota] + MOUNT_OPTIONS="-o pquota" + TEST_FS_MOUNT_OPTS="-o noatime" + OVERLAY_MOUNT_OPTIONS="-o redirect_dir=off" + +In the example above, MOUNT_OPTIONS will be used to mount the base scratch fs, +TEST_FS_MOUNT_OPTS will be used to mount the base test fs and +OVERLAY_MOUNT_OPTIONS will be used to mount both test and scratch overlays.
common/rc +63 -4
101 lines not on this page · raw message
tests/overlay/003 +2 -1
@@ -89,7 +89,8 @@ rm -rf ${SCRATCH_MNT}/* # nothing should be listed ls ${SCRATCH_MNT}/ -_scratch_unmount +# unmount overlayfs but not base fs +$UMOUNT_PROG $SCRATCH_MNT rm -rf $lowerdir echo "Silence is golden"
tests/overlay/004 +2 -1
@@ -85,7 +85,8 @@ _user_do "chmod g+t ${SCRATCH_MNT}/attr_file2 > /dev/null 2>&1" _user_do "chmod u-X ${SCRATCH_MNT}/attr_file2 > /dev/null 2>&1" stat -c %a ${SCRATCH_MNT}/attr_file2 -_scratch_unmount +# unmount overlayfs but not base fs +$UMOUNT_PROG $SCRATCH_MNT # check mode bits of the file that has been copied up, and # the file that should not have been copied up.
tests/overlay/014 +3 -2
@@ -73,7 +73,8 @@ mkdir -p $lowerdir1/testdir/d _overlay_mount_dirs $lowerdir1 $lowerdir2 $workdir $SCRATCH_DEV $SCRATCH_MNT rm -rf $SCRATCH_MNT/testdir mkdir -p $SCRATCH_MNT/testdir/visibledir -_scratch_unmount +# unmount overlayfs but not base fs +$UMOUNT_PROG $SCRATCH_MNT # mount overlay again, with lowerdir1 and lowerdir2 as multiple lowerdirs, # and create a new file in testdir, triggers copyup from lowerdir,
@@ -84,7 +85,7 @@ touch $SCRATCH_MNT/testdir/visiblefile # umount and mount overlay again, buggy kernel treats the copied-up dir as # opaque, visibledir is not seen in merged dir. -_scratch_unmount +$UMOUNT_PROG $SCRATCH_MNT _overlay_mount_dirs "$lowerdir2:$lowerdir1" $upperdir $workdir \ $SCRATCH_DEV $SCRATCH_MNT ls $SCRATCH_MNT/testdir
Signature · 1 line
2.7.4
PATCH v5 10/10 overlay: use OVL_BASE_SCRATCH_MNT instead of SCRATCH_DEV
In reply to Amir GoldsteinUse the new var OVL_BASE_SCRATCH_MNT to refer to overlay base dir instead of the legacy SCRATCH_DEV var.
In overlay/011, when using MOUNT_PROG directly, provide OVL_BASE_SCRATCH_MNT as device argument instead of 'none'. This fix is required to support mount of base fs.
--- tests/overlay/001 | 7 +++---- tests/overlay/002 | 2 +- tests/overlay/003 | 2 +- tests/overlay/004 | 4 ++-- tests/overlay/005 | 30 +++++++++++++++--------------- tests/overlay/006 | 8 ++++---- tests/overlay/008 | 4 ++-- tests/overlay/009 | 2 +- tests/overlay/010 | 10 +++++----- tests/overlay/011 | 6 +++--- tests/overlay/012 | 4 ++-- tests/overlay/013 | 4 ++-- tests/overlay/014 | 14 +++++++------- tests/overlay/015 | 2 +- tests/overlay/016 | 2 +- tests/overlay/017 | 2 +- tests/overlay/018 | 2 +- tests/overlay/019 | 2 +- tests/overlay/020 | 2 +- tests/overlay/021 | 6 +++--- tests/overlay/022 | 2 +- tests/overlay/023 | 2 +- tests/overlay/024 | 2 +- tests/overlay/027 | 2 +- tests/overlay/028 | 2 +- tests/overlay/029 | 4 ++-- 26 files changed, 64 insertions(+), 65 deletions(-)
tests/overlay/001 +3 -4
@@ -55,12 +55,11 @@ _require_scratch _scratch_mkfs # overlay copy_up doesn't deal with sparse file well, holes will be filled by -# zeros, so at least (4G + 4G + 8k) free space is needed on $SCRATCH_DEV, -# where $SCRATCH_DEV is actually a test dir used in overlay testing -_require_fs_space $SCRATCH_DEV $((4*1024*1024*2 + 8)) +# zeros, so at least (4G + 4G + 8k) free space is needed on scratch base fs +_require_fs_space $OVL_BASE_SCRATCH_MNT $((4*1024*1024*2 + 8)) # Create test files with different sizes in lower dir -lowerdir=$SCRATCH_DEV/$OVL_LOWER +lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER mkdir -p $lowerdir touch $lowerdir/zero_size $XFS_IO_PROG -fc "pwrite 0 4k" $lowerdir/less_than_4g >>$seqres.full
tests/overlay/002 +1 -1
@@ -59,7 +59,7 @@ _require_scratch _scratch_mkfs # Create our test file. -lowerdir=$SCRATCH_DEV/$OVL_LOWER +lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER mkdir -p $lowerdir touch $lowerdir/foobar
tests/overlay/003 +1 -1
@@ -60,7 +60,7 @@ _require_scratch _scratch_mkfs # Create test files dirs in lower dir -lowerdir=${SCRATCH_DEV}/${OVL_LOWER} +lowerdir=${OVL_BASE_SCRATCH_MNT}/${OVL_LOWER} mkdir -p $lowerdir touch ${lowerdir}/test_file
tests/overlay/004 +2 -2
@@ -53,8 +53,8 @@ _require_user _scratch_mkfs # Create test file in lower dir -lowerdir=${SCRATCH_DEV}/${OVL_LOWER} -upperdir=${SCRATCH_DEV}/${OVL_UPPER} +lowerdir=${OVL_BASE_SCRATCH_MNT}/${OVL_LOWER} +upperdir=${OVL_BASE_SCRATCH_MNT}/${OVL_UPPER} mkdir -p $lowerdir touch ${lowerdir}/attr_file1 chmod 000 ${lowerdir}/attr_file1
tests/overlay/005 +15 -15
@@ -61,31 +61,31 @@ _require_loop _scratch_mkfs # setup loop images fs for overlayfs -lower_img=${SCRATCH_DEV}/${seq}.$$.img +lower_img=${OVL_BASE_SCRATCH_MNT}/${seq}.$$.img $XFS_IO_PROG -f -c "truncate 120m" $lower_img >>$seqres.full 2>&1 fs_loop_dev=`_create_loop_device $lower_img` $MKFS_XFS_PROG -f -n ftype=1 $fs_loop_dev >>$seqres.full 2>&1 # only 20m for upper dir -upper_img=${SCRATCH_DEV}/$$.${seq}.img +upper_img=${OVL_BASE_SCRATCH_MNT}/$$.${seq}.img $XFS_IO_PROG -f -c "truncate 20m" $upper_img >>$seqres.full 2>&1 upper_loop_dev=`_create_loop_device $upper_img` $MKFS_XFS_PROG -f -n ftype=1 $upper_loop_dev >>$seqres.full 2>&1 # mount underlying xfs -mkdir -p ${SCRATCH_DEV}/lowermnt -mkdir -p ${SCRATCH_DEV}/uppermnt -$MOUNT_PROG $fs_loop_dev ${SCRATCH_DEV}/lowermnt -$MOUNT_PROG $upper_loop_dev ${SCRATCH_DEV}/uppermnt +mkdir -p ${OVL_BASE_SCRATCH_MNT}/lowermnt +mkdir -p ${OVL_BASE_SCRATCH_MNT}/uppermnt +$MOUNT_PROG $fs_loop_dev ${OVL_BASE_SCRATCH_MNT}/lowermnt +$MOUNT_PROG $upper_loop_dev ${OVL_BASE_SCRATCH_MNT}/uppermnt # prepare dirs -mkdir -p ${SCRATCH_DEV}/lowermnt/lower -mkdir -p ${SCRATCH_DEV}/uppermnt/upper -mkdir -p ${SCRATCH_DEV}/uppermnt/work +mkdir -p ${OVL_BASE_SCRATCH_MNT}/lowermnt/lower +mkdir -p ${OVL_BASE_SCRATCH_MNT}/uppermnt/upper +mkdir -p ${OVL_BASE_SCRATCH_MNT}/uppermnt/work -lowerd=${SCRATCH_DEV}/lowermnt/lower -upperd=${SCRATCH_DEV}/uppermnt/upper -workd=${SCRATCH_DEV}/uppermnt/work +lowerd=${OVL_BASE_SCRATCH_MNT}/lowermnt/lower +upperd=${OVL_BASE_SCRATCH_MNT}/uppermnt/upper +workd=${OVL_BASE_SCRATCH_MNT}/uppermnt/work # Create test file in lower dir, with too big a size for # upper dir to copy up.
@@ -93,7 +93,7 @@ $XFS_IO_PROG -f -c "truncate 48m" ${lowerd}/test_file \ >>$seqres.full 2>&1 # mount new overlayfs -_overlay_mount_dirs $lowerd $upperd $workd $SCRATCH_DEV $SCRATCH_MNT +_overlay_mount_dirs $lowerd $upperd $workd $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT # the open call triggers copy-up and it will fail ENOSPC $XFS_IO_PROG -f -c "o" ${SCRATCH_MNT}/test_file \
@@ -103,8 +103,8 @@ $XFS_IO_PROG -f -c "o" ${SCRATCH_MNT}/test_file \ $UMOUNT_PROG $SCRATCH_MNT # unmount undelying xfs, this tiggers panic if memleak happens -$UMOUNT_PROG ${SCRATCH_DEV}/uppermnt -$UMOUNT_PROG ${SCRATCH_DEV}/lowermnt +$UMOUNT_PROG ${OVL_BASE_SCRATCH_MNT}/uppermnt +$UMOUNT_PROG ${OVL_BASE_SCRATCH_MNT}/lowermnt # success, all done echo "Silence is golden"
tests/overlay/006 +4 -4
@@ -57,10 +57,10 @@ echo "Silence is golden" _scratch_mkfs # Create test file/dir before mount -mkdir -p $SCRATCH_DEV/$OVL_LOWER -mkdir -p $SCRATCH_DEV/$OVL_UPPER -touch $SCRATCH_DEV/$OVL_LOWER/lowertestfile -mkdir $SCRATCH_DEV/$OVL_UPPER/uppertestdir +mkdir -p $OVL_BASE_SCRATCH_MNT/$OVL_LOWER +mkdir -p $OVL_BASE_SCRATCH_MNT/$OVL_UPPER +touch $OVL_BASE_SCRATCH_MNT/$OVL_LOWER/lowertestfile +mkdir $OVL_BASE_SCRATCH_MNT/$OVL_UPPER/uppertestdir _scratch_mount
tests/overlay/008 +2 -2
@@ -56,13 +56,13 @@ _require_user _scratch_mkfs # Create test file on lower dir, and chown to fsgqa user -lowerdir=$SCRATCH_DEV/$OVL_LOWER +lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER mkdir -p $lowerdir touch $lowerdir/testfile chown fsgqa:fsgqa $lowerdir/testfile # chown upperdir to fsgqa user, so new file/dir can be created by the user -upperdir=$SCRATCH_DEV/$OVL_UPPER +upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER mkdir -p $upperdir chown fsgqa:fsgqa $upperdir
tests/overlay/009 +1 -1
@@ -54,7 +54,7 @@ _require_scratch _scratch_mkfs # Create test file in lowerdir -lowerdir=$SCRATCH_DEV/$OVL_LOWER +lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER mkdir -p $lowerdir touch $lowerdir/testfile
tests/overlay/010 +5 -5
@@ -55,10 +55,10 @@ _scratch_mkfs # Need two lower dirs in this test, and we mount overlay ourselves, # create upper and workdir as well -lowerdir1=$SCRATCH_DEV/$OVL_LOWER.1 -lowerdir2=$SCRATCH_DEV/$OVL_LOWER.2 -upperdir=$SCRATCH_DEV/$OVL_UPPER -workdir=$SCRATCH_DEV/$OVL_WORK +lowerdir1=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER.1 +lowerdir2=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER.2 +upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER +workdir=$OVL_BASE_SCRATCH_MNT/$OVL_WORK mkdir -p $lowerdir1 $lowerdir2 $upperdir $workdir # One lowerdir contains test dir and test files, the other contains whiteout
@@ -68,7 +68,7 @@ mknod $lowerdir2/testdir/a c 0 0 # Mount overlayfs and remove testdir, which led to kernel crash _overlay_mount_dirs "$lowerdir2:$lowerdir1" $upperdir $workdir \ - $SCRATCH_DEV $SCRATCH_MNT + $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT rm -rf $SCRATCH_MNT/testdir # success, all done
tests/overlay/011 +3 -3
@@ -58,8 +58,8 @@ _require_attrs _scratch_mkfs # Create test dir on upper and make it opaque by setting proper xattr -lowerdir=$SCRATCH_DEV/$OVL_LOWER -upperdir=$SCRATCH_DEV/$OVL_UPPER +lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER +upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER mkdir -p $lowerdir/testdir mkdir -p $upperdir/testdir $SETFATTR_PROG -n "trusted.overlay.opaque" -v "y" $upperdir/testdir
@@ -68,7 +68,7 @@ $SETFATTR_PROG -n "trusted.overlay.opaque" -v "y" $upperdir/testdir # $upperdir overlaid on top of $lowerdir, so that "trusted.overlay.opaque" # xattr should be honored and should not be listed # mount readonly, because there's no upper and workdir -$MOUNT_PROG -t overlay -o ro -o lowerdir=$upperdir:$lowerdir none $SCRATCH_MNT +$MOUNT_PROG -t overlay -o ro -o lowerdir=$upperdir:$lowerdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT # Dump trusted.overlay xattr, we should not see the "opaque" xattr $GETFATTR_PROG -d -m overlay $SCRATCH_MNT/testdir
tests/overlay/012 +2 -2
@@ -55,8 +55,8 @@ _require_scratch # remove all files from previous runs _scratch_mkfs -lowerdir=$SCRATCH_DEV/$OVL_LOWER -upperdir=$SCRATCH_DEV/$OVL_UPPER +lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER +upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER mkdir -p $lowerdir/test _scratch_mount
tests/overlay/013 +2 -2
@@ -54,8 +54,8 @@ _require_test_program "t_truncate_self" _scratch_mkfs # copy test program to lower and upper dir -lowerdir=$SCRATCH_DEV/$OVL_LOWER -upperdir=$SCRATCH_DEV/$OVL_UPPER +lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER +upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER mkdir -p $lowerdir mkdir -p $upperdir cp $here/src/t_truncate_self $lowerdir/test_lower
tests/overlay/014 +7 -7
@@ -61,16 +61,16 @@ _require_scratch _scratch_mkfs # Create multiple lowerdirs and upperdir, workdir, and testdir on lowerdir -lowerdir1=$SCRATCH_DEV/lower1 -lowerdir2=$SCRATCH_DEV/lower2 -upperdir=$SCRATCH_DEV/upper -workdir=$SCRATCH_DEV/workdir +lowerdir1=$OVL_BASE_SCRATCH_MNT/lower1 +lowerdir2=$OVL_BASE_SCRATCH_MNT/lower2 +upperdir=$OVL_BASE_SCRATCH_MNT/upper +workdir=$OVL_BASE_SCRATCH_MNT/workdir mkdir -p $lowerdir1 $lowerdir2 $upperdir $workdir mkdir -p $lowerdir1/testdir/d # mount overlay with $lowerdir2 as upperdir, and remove & recreate testdir, # make testdir on $lowerdir2 opaque -_overlay_mount_dirs $lowerdir1 $lowerdir2 $workdir $SCRATCH_DEV $SCRATCH_MNT +_overlay_mount_dirs $lowerdir1 $lowerdir2 $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT rm -rf $SCRATCH_MNT/testdir mkdir -p $SCRATCH_MNT/testdir/visibledir # unmount overlayfs but not base fs
@@ -80,14 +80,14 @@ $UMOUNT_PROG $SCRATCH_MNT # and create a new file in testdir, triggers copyup from lowerdir, # copyup should not copy overlayfs private xattr _overlay_mount_dirs "$lowerdir2:$lowerdir1" $upperdir $workdir \ - $SCRATCH_DEV $SCRATCH_MNT + $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT touch $SCRATCH_MNT/testdir/visiblefile # umount and mount overlay again, buggy kernel treats the copied-up dir as # opaque, visibledir is not seen in merged dir. $UMOUNT_PROG $SCRATCH_MNT _overlay_mount_dirs "$lowerdir2:$lowerdir1" $upperdir $workdir \ - $SCRATCH_DEV $SCRATCH_MNT + $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT ls $SCRATCH_MNT/testdir # success, all done
tests/overlay/015 +1 -1
@@ -57,7 +57,7 @@ _scratch_mkfs umask 022 # Create test dir in lower dir and set sgid bit -lowerdir=$SCRATCH_DEV/$OVL_LOWER +lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER mkdir -p $lowerdir/dir chown $qa_user:$qa_group $lowerdir/dir chmod 2775 $lowerdir/dir
tests/overlay/016 +1 -1
@@ -57,7 +57,7 @@ rm -f $seqres.full _scratch_mkfs >>$seqres.full 2>&1 # Create our test files. -lowerdir=$SCRATCH_DEV/$OVL_LOWER +lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER mkdir -p $lowerdir echo "This is old news" > $lowerdir/foo echo "This is old news" > $lowerdir/bar
tests/overlay/017 +1 -1
@@ -60,7 +60,7 @@ _scratch_mkfs >>$seqres.full 2>&1 # Not dealing with hardlinks here, because there is more to test # then stable inode number. # Hardlinks will get a test of their own. -lowerdir=$SCRATCH_DEV/$OVL_LOWER +lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER mkdir -p $lowerdir mkdir $lowerdir/dir touch $lowerdir/file
tests/overlay/018 +1 -1
@@ -55,7 +55,7 @@ rm -f $seqres.full _scratch_mkfs >>$seqres.full 2>&1 # Create 2 hardlinked files in lower -lowerdir=$SCRATCH_DEV/$OVL_LOWER +lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER mkdir -p $lowerdir echo "patient zero" >> $lowerdir/foo ln $lowerdir/foo $lowerdir/bar
tests/overlay/019 +1 -1
@@ -51,7 +51,7 @@ _require_scratch # Remove all files from previous tests _scratch_mkfs -lowerdir=$SCRATCH_DEV/$OVL_LOWER +lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER mkdir -p $lowerdir _scratch_mount
tests/overlay/020 +1 -1
@@ -55,7 +55,7 @@ _require_scratch # Remove all files from previous tests _scratch_mkfs -lowerdir=$SCRATCH_DEV/$OVL_LOWER +lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER mkdir -p $lowerdir/dir _scratch_mount
tests/overlay/021 +3 -3
@@ -54,13 +54,13 @@ _scratch_mkfs # overlay copy_up doesn't deal with sparse file well, holes will be filled by # zeros, so for the worst case of hitting all the copy up bomb files, we need -# (64*(16+32+64+128)M) free space on $SCRATCH_DEV. +# (64*(16+32+64+128)M) free space on $OVL_BASE_SCRATCH_MNT. # However, triggering more than a total of 16 copy up bombs would require # really fast data copy (clone up doesn't take up space at all), so let's be # conservative and reserve space for 16 data copy ups per directory. -_require_fs_space $SCRATCH_DEV $((16*(16+32+64+128)*1024)) +_require_fs_space $OVL_BASE_SCRATCH_MNT $((16*(16+32+64+128)*1024)) -lowerdir=$SCRATCH_DEV/$OVL_LOWER +lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER mkdir -p $lowerdir testdir=arena
tests/overlay/022 +1 -1
@@ -59,7 +59,7 @@ _require_scratch # Remove all files from previous tests _scratch_mkfs -upperdir=$SCRATCH_DEV/$OVL_UPPER +upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER mkdir -p $upperdir/upper mkdir -p $upperdir/work # mount overlay with dirs in upper
tests/overlay/023 +1 -1
@@ -63,7 +63,7 @@ _require_scratch _scratch_mkfs # setting acls before mount -wkdir=$SCRATCH_DEV/$OVL_WORK +wkdir=$OVL_BASE_SCRATCH_MNT/$OVL_WORK mkdir -p $wkdir setfacl -d -m o::rwx $wkdir
tests/overlay/024 +1 -1
@@ -61,7 +61,7 @@ _require_scratch _scratch_mkfs # making workdir -wkdir=$SCRATCH_DEV/$OVL_WORK +wkdir=$OVL_BASE_SCRATCH_MNT/$OVL_WORK mkdir -p $wkdir/work/foo _scratch_mount
tests/overlay/027 +1 -1
@@ -63,7 +63,7 @@ _require_chattr _scratch_mkfs # Preparing immutable file -upperdir=$SCRATCH_DEV/$OVL_UPPER +upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER mkdir -p $upperdir touch $upperdir/foo $CHATTR_PROG +i $upperdir/foo
tests/overlay/028 +1 -1
@@ -61,7 +61,7 @@ _require_command "$FLOCK_PROG" flock # Remove all files from previous tests _scratch_mkfs -lowerdir=$SCRATCH_DEV/$OVL_LOWER +lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER mkdir -p $lowerdir touch $lowerdir/foo
tests/overlay/029 +2 -2
@@ -65,8 +65,8 @@ _require_scratch _scratch_mkfs # Preparing files -upperdir=$SCRATCH_DEV/$OVL_UPPER -lowerdir=$SCRATCH_DEV/$OVL_LOWER +upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER +lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER mkdir -p $upperdir/up echo foo > $upperdir/up/foo mkdir -p $lowerdir/low
Signature · 1 line
2.7.4
Quoted 4 lines · Amir Goldstein
Hi Eryu,
I rebased my branch to your master and made some minor adaptations to Xiong's new tests. Also reordered the patches:
Thanks for the resending! I was about to ask for it :)
Quoted 20 lines
- patches 1-6 are generic fixes and re-factoring - if you wish, you can merge them before the overlay config changes.
- patches 7-10 implement the new overlay config. patches 7 and 10 include adaptations of Xiong's new tests to new config.
This is the 5th revision of new overlayfs config.
The main motivation of this work is to help catch overlayfs bugs related to leaking objects in underlying (base) fs.With this change, all you have to do to run overlay tests if you already have a local.config setup to test a local file system is:
./check -overlaySee README.overlay for some examples of config files to use with -overlay.
The legacy overlayfs configuration, where
TEST_DEVis a directory still works, but it should be deprecated.
Quoted 6 lines
Tested ./check -overlay -g quick with both legacy overlay configuration and the new base fs configuration.
Eryu wished to keep these changes soaking on the list for while, so for those who can help with testing, you can get pull the branch for testing from my github tree [1].
I was planning to pull them in via this week's update, if there's no objections.
Thanks again for the work!
Eryu