fstests: add idmapped mounts tests
v12 cover fstests: add idmapped mounts tests
From: Christian Brauner <[email protected]>
Hey everyone,
This series is available from:
https://git.kernel.org/brauner/xfstests-dev/h/idmapped_mounts
https://gitlab.com/brauner/xfstests/-/tree/idmapped_mounts
https://github.com/brauner/xfstests/tree/idmapped_mounts
/* v12 */
- Main change is a fix to the io_uring tests. io_uring_wait_cqe() doesn't set errno, it returns a negative errno.
- Make sure we also run without error when libcap isn't available (That was always the goal but the logic missed a few new tests.)
/* v11 */
Reworked according to Amir's comments.
/* v10 */
Reworked according to Eryu's comments.
/* v9 */
Rebased onto current master.
ubuntu@f1-vm:~/src/git/xfstests$ sudo ./check -g idmapped FSTYP -- xfs (debug) PLATFORM -- Linux/x86_64 f1-vm 5.12.0-rc3-syz #18 SMP PREEMPT Fri Mar 26 13:27:16 UTC 2021 MKFS_OPTIONS -- -f -bsize=4096 /dev/loop1 MOUNT_OPTIONS -- /dev/loop1 /mnt/scratch
generic/632 files ... 27s
xfs/529 files ... 67s
xfs/530 files ... 43s
Ran: generic/632 xfs/529 xfs/530
Passed all 3 tests
Thanks!
Christian
Christian Brauner (6):
generic/631: add test for detached mount propagation generic/632: add fstests for idmapped mounts common/rc: add _scratch_{u}mount_idmapped() helpers common/quota: move _qsetup() helper to common code xfs/529: quotas and idmapped mounts xfs/530: quotas on idmapped mounts
.gitignore | 3 + README | 5 +- common/quota | 20 + common/rc | 60 + configure.ac | 2 + include/builddefs.in | 1 + m4/Makefile | 1 + m4/package_libcap.m4 | 4 + src/Makefile | 8 +- src/detached_mounts_propagation.c | 189 + src/feature.c | 40 +- src/idmapped-mounts/Makefile | 40 + src/idmapped-mounts/idmapped-mounts.c | 8873 +++++++++++++++++++++++++ src/idmapped-mounts/missing.h | 151 + src/idmapped-mounts/mount-idmapped.c | 431 ++ src/idmapped-mounts/utils.c | 134 + src/idmapped-mounts/utils.h | 30 + tests/generic/631 | 43 + tests/generic/631.out | 2 + tests/generic/632 | 42 + tests/generic/632.out | 2 + tests/generic/group | 2 + tests/xfs/050 | 19 - tests/xfs/299 | 19 - tests/xfs/529 | 377 ++ tests/xfs/529.out | 657 ++ tests/xfs/530 | 212 + tests/xfs/530.out | 129 + tests/xfs/group | 2 + 29 files changed, 11453 insertions(+), 45 deletions(-) create mode 100644 m4/package_libcap.m4 create mode 100644 src/detached_mounts_propagation.c create mode 100644 src/idmapped-mounts/Makefile create mode 100644 src/idmapped-mounts/idmapped-mounts.c create mode 100644 src/idmapped-mounts/missing.h create mode 100644 src/idmapped-mounts/mount-idmapped.c create mode 100644 src/idmapped-mounts/utils.c create mode 100644 src/idmapped-mounts/utils.h create mode 100644 tests/generic/631 create mode 100644 tests/generic/631.out create mode 100644 tests/generic/632 create mode 100644 tests/generic/632.out create mode 100644 tests/xfs/529 create mode 100644 tests/xfs/529.out create mode 100644 tests/xfs/530 create mode 100644 tests/xfs/530.out
base-commit: f6ddaf130d5b0817278afe441fdde52f464f321b
Signature · 1 line
2.27.0
PATCH v12 1/6 generic/631: add test for detached mount propagation
From: Christian Brauner <[email protected]>
Regression test to verify that creating a series of detached mounts, attaching them to the filesystem, and unmounting them does not trigger an integer overflow in ns->mounts causing the kernel to block any new mounts in count_mounts() and returning ENOSPC because it falsely assumes that the maximum number of mounts in the mount namespace has been reached, i.e. it thinks it can't fit the new mounts into the mount namespace anymore.
The test is written in a way that it will leave the host's mount namespace intact so we are sure to never make the host's mount namespace unuseable!
---
/* v1 - v8 */
patch not present
/* v9 */
- Christian Brauner <[email protected]>:
- Rebased on current master.
/* v10 */
- Eryu Guan <[email protected]>:
- Add missing checks whether test is supported.
- Move status=$? assignment up.
/* v11 */
unchanged
/* v12 */
unchanged
--- .gitignore | 1 + src/Makefile | 3 +- src/detached_mounts_propagation.c | 252 ++++++++++++++++++++++++++++++ tests/generic/631 | 43 +++++ tests/generic/631.out | 2 + tests/generic/group | 1 + 6 files changed, 301 insertions(+), 1 deletion(-) create mode 100644 src/detached_mounts_propagation.c create mode 100644 tests/generic/631 create mode 100644 tests/generic/631.out
.gitignore +1 -0
@@ -62,6 +62,7 @@ /src/cloner /src/dbtest /src/deduperace +/src/detached_mounts_propagation /src/devzero /src/dio-interleaved /src/dio-invalidate-cache
src/Makefile +2 -1
@@ -29,7 +29,8 @@ LINUX_TARGETS = xfsctl bstat t_mtab getdevicesize preallo_rw_pattern_reader \ attr-list-by-handle-cursor-test listxattr dio-interleaved t_dir_type \ dio-invalidate-cache stat_test t_encrypted_d_revalidate \ attr_replace_test swapon mkswap t_attr_corruption t_open_tmpfiles \ - fscrypt-crypt-util bulkstat_null_ocount splice-test chprojid_fail + fscrypt-crypt-util bulkstat_null_ocount splice-test chprojid_fail \ + detached_mounts_propagation SUBDIRS = log-writes perf
src/detached_mounts_propagation.c new file +252 -0
252 lines not on this page · raw message
tests/generic/631 new file +43 -0
43 lines not on this page · raw message
tests/generic/631.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 631 +silence is golden
tests/generic/group +1 -0
@@ -633,3 +633,4 @@ 628 auto quick rw clone 629 auto quick rw copy_range 630 auto quick rw dedupe clone +631 auto quick mount
Signature · 1 line
2.27.0
PATCH v12 3/6 common/rc: add _scratch_{u}mount_idmapped() helpers
In reply to Christian BraunerFrom: Christian Brauner <[email protected]>
They will be used in follow-up patches for xfs quota tests but might be useful for other tests in the future.
---
/* v1 - v9 */
patch not present
/* v10 */
patch introduced
/* v11 */
- Amir Goldstein <[email protected]>:
- Minor Makefile style nit.
- Add missing "/" when calculating maxium path length.
- Handle the case where user gives us a path to userns.
/* v12 */
- Eryu Guan <[email protected]>:
- Fix helper installation.
--- .gitignore | 1 + common/rc | 35 +++ src/idmapped-mounts/Makefile | 13 +- src/idmapped-mounts/mount-idmapped.c | 431 +++++++++++++++++++++++++++ src/idmapped-mounts/utils.c | 2 +- src/idmapped-mounts/utils.h | 1 + 6 files changed, 478 insertions(+), 5 deletions(-) create mode 100644 src/idmapped-mounts/mount-idmapped.c
.gitignore +1 -0
@@ -179,6 +179,7 @@ /src/aio-dio-regress/aiocp /src/aio-dio-regress/aiodio_sparse2 /src/idmapped-mounts/idmapped-mounts +/src/idmapped-mounts/mount-idmapped /src/log-writes/replay-log /src/perf/*.pyc
common/rc +35 -0
@@ -342,6 +342,36 @@ _scratch_mount() _try_scratch_mount $* || _fail "mount failed" } +_scratch_mount_idmapped() +{ + local type="$1" + local id="$2" + + if [ "$type" = "u" ]; then + # This means root will be able to create files as uid %id in + # the underlying filesystem by going through the idmapped mount. + $here/src/idmapped-mounts/mount-idmapped --map-mount u:0:$id:1 \ + --map-mount u:$id:0:1 \ + --map-mount g:0:0:1 \ + "$SCRATCH_MNT" "$SCRATCH_MNT" || _fail "mount-idmapped failed" + elif [ "$type" = "g" ]; then + # This means root will be able to create files as gid %id in + # the underlying filesystem by going through the idmapped mount. + $here/src/idmapped-mounts/mount-idmapped --map-mount g:0:$id:1 \ + --map-mount g:$id:0:1 \ + --map-mount u:0:0:1 \ + "$SCRATCH_MNT" "$SCRATCH_MNT" || _fail "mount-idmapped failed" + elif [ "$type" = "b" ]; then + # This means root will be able to create files as uid and gid + # %id in the underlying filesystem by going through the idmapped mount. + $here/src/idmapped-mounts/mount-idmapped --map-mount b:0:$id:1 \ + --map-mount b:$id:0:1 \ + "$SCRATCH_MNT" "$SCRATCH_MNT" || _fail "mount-idmapped failed" + else + _fail "usage: either \"u\" (uid), \"g\" (gid), or \"b\" (uid and gid) must be specified " + fi +} + _scratch_unmount() { case "$FSTYP" in
@@ -357,6 +387,11 @@ _scratch_unmount() esac } +_scratch_umount_idmapped() +{ + $UMOUNT_PROG $SCRATCH_MNT +} + _scratch_remount() { local opts="$1"
src/idmapped-mounts/Makefile +9 -4
@@ -3,9 +3,10 @@ TOPDIR = ../.. include $(TOPDIR)/include/builddefs -TARGETS = idmapped-mounts +TARGETS = idmapped-mounts mount-idmapped +CFILES_IDMAPPED_MOUNTS = idmapped-mounts.c utils.c +CFILES_MOUNT_IDMAPPED = mount-idmapped.c utils.c -CFILES = idmapped-mounts.c utils.c HFILES = missing.h utils.h LLDLIBS += -pthread LDIRT = $(TARGETS)
@@ -24,9 +25,13 @@ depend: .dep include $(BUILDRULES) -$(TARGETS): $(CFILES) +idmapped-mounts: $(CFILES_IDMAPPED_MOUNTS) @echo " [CC] $@" - $(Q)$(LTLINK) $(CFILES) -o $@ $(CFLAGS) $(LDFLAGS) $(LDLIBS) + $(Q)$(LTLINK) $(CFILES_IDMAPPED_MOUNTS) -o $@ $(CFLAGS) $(LDFLAGS) $(LDLIBS) + +mount-idmapped: $(CFILES_MOUNT_IDMAPPED) + @echo " [CC] $@" + $(Q)$(LTLINK) $(CFILES_MOUNT_IDMAPPED) -o $@ $(CFLAGS) $(LDFLAGS) $(LDLIBS) install: $(INSTALL) -m 755 -d $(PKG_LIB_DIR)/src/idmapped-mounts
src/idmapped-mounts/mount-idmapped.c new file +431 -0
431 lines not on this page · raw message
src/idmapped-mounts/utils.c +1 -1
@@ -88,7 +88,7 @@ pid_t do_clone(int (*fn)(void *), void *arg, int flags)#endif } -static int get_userns_fd_cb (void *data) +int get_userns_fd_cb (void *data) {return kill (getpid (),SIGSTOP ); }
src/idmapped-mounts/utils.h +1 -0
@@ -20,6 +20,7 @@#include "missing.h" extern pid_t do_clone (int (*fn)(void *),void *arg,int flags); +extern int get_userns_fd_cb (void *data);extern int get_userns_fd (unsigned long nsid,unsigned long hostid,unsigned long range);extern ssize_t read_nointr (int fd,void *buf,size_t count);
Signature · 1 line
2.27.0
PATCH v12 4/6 common/quota: move _qsetup() helper to common code
In reply to Christian BraunerFrom: Christian Brauner <[email protected]>
It's already used in two tests and will be used in a third.
---
/* v1 - v9 */
patch not present
/* v10 */
patch introduced
/* v11 */
unchanged
/* v12 */
unchanged
--- common/quota | 20 ++++++++++++++++++++ tests/xfs/050 | 19 ------------------- tests/xfs/299 | 19 ------------------- 3 files changed, 20 insertions(+), 38 deletions(-)
common/quota +20 -0
@@ -329,5 +329,25 @@ _report_quota_inodes() { repquota $1 | egrep "^($qa_user|root|nobody)" | awk '{print $1, $6, $7, $8}' | sort -r } +# Determine which type of quota we're using +_qsetup() +{ + opt=$1 + enforce=0 + if [ $opt = "u" -o $opt = "uno" ]; then + type=u + eval `_choose_uid` + elif [ $opt = "g" -o $opt = "gno" ]; then + type=g + eval `_choose_gid` + elif [ $opt = "p" -o $opt = "pno" ]; then + type=p + eval `_choose_prid` + fi + [ $opt = "u" -o $opt = "g" -o $opt = "p" ] && enforce=1 + + echo "Using type=$type id=$id" >> $seqres.full +} + # make sure this script returns success /bin/true
tests/xfs/050 +0 -19
@@ -69,25 +69,6 @@ _filter_and_check_blks() ' | _filter_quota_report } -_qsetup() -{ - opt=$1 - enforce=0 - if [ $opt = "u" -o $opt = "uno" ]; then - type=u - eval `_choose_uid` - elif [ $opt = "g" -o $opt = "gno" ]; then - type=g - eval `_choose_gid` - elif [ $opt = "p" -o $opt = "pno" ]; then - type=p - eval `_choose_prid` - fi - [ $opt = "u" -o $opt = "g" -o $opt = "p" ] && enforce=1 - - echo "Using type=$type id=$id" >> $seqres.full -} - _exercise() { _scratch_mkfs_xfs | _filter_mkfs 2>$tmp.mkfs
tests/xfs/299 +0 -19
@@ -62,25 +62,6 @@ _filter_and_check_blks() ' | _filter_quota_report } -_qsetup() -{ - opt=$1 - enforce=0 - if [ $opt = "u" -o $opt = "uno" ]; then - type=u - eval `_choose_uid` - elif [ $opt = "g" -o $opt = "gno" ]; then - type=g - eval `_choose_gid` - elif [ $opt = "p" -o $opt = "pno" ]; then - type=p - eval `_choose_prid` - fi - [ $opt = "u" -o $opt = "g" -o $opt = "p" ] && enforce=1 - - echo "Using type=$type id=$id" >> $seqres.full -} - _exercise() {
Signature · 1 line
2.27.0
PATCH v12 5/6 xfs/529: quotas and idmapped mounts
In reply to Christian BraunerFrom: Christian Brauner <[email protected]>
Test that xfs quota behave correctly on idmapped mounts. Mount a scratch device with user and group quota support enabled. Create directories "unmapped" and "idmapped". Create files in the unampped mount and verify quota behavior. Create files through the idmapped mount and verify identical behavior.
---
/* v1 - v8 */
patch not present
/* v9 */
- Christian Brauner <[email protected]>:
- Rebased on current master.
/* v10 */
- Eryu Guan <[email protected]>:
- Remove leading "_" from local helpers.
- Remove redundant "-q" option from umount -l calls.
- Use wipe_mount() in a few more places.
/* v11 */
- Amir Goldstein <[email protected]>:
- Add a dedicated "idmapped" tag for idmapped mounts tests.
/* v12 */
- Eryu Guan <[email protected]>:
- Don't use lazy umount if not required.
--- tests/xfs/529 | 377 ++++++++++++++++++++++++++ tests/xfs/529.out | 657 ++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/group | 1 + 3 files changed, 1035 insertions(+) create mode 100644 tests/xfs/529 create mode 100644 tests/xfs/529.out
tests/xfs/529 new file +377 -0
@@ -0,0 +1,377 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0+ +# +# Copyright (c) 2021 Christian Brauner <[email protected]> +# All Rights Reserved. +# +# FS QA Test No. 529 +# +# Exercise basic xfs_quota functionality (user/group/project quota) +# Use of "sync" mount option here is an attempt to get deterministic +# allocator behaviour. +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "_cleanup; exit \$status" 0 1 2 3 15 + +wipe_mounts() +{ + umount "${SCRATCH_MNT}/idmapped" >/dev/null 2>&1 + _scratch_unmount >/dev/null 2>&1 +} + +_cleanup() +{ + cd / + wipe_mounts + rm -f $tmp.* +} + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter +. ./common/quota + +# remove previous $seqres.full before test +rm -f $seqres.full + +# real QA test starts here +_supported_fs xfs +_require_idmapped_mounts +_require_test_program "idmapped-mounts/mount-idmapped" +_require_scratch +_require_xfs_quota +_require_user fsgqa +_require_user fsgqa2 +_require_group fsgqa +_require_group fsgqa2 + +_scratch_mkfs_xfs >>$seqres.full 2>&1 + +uqid=`id -u fsgqa` +gqid=`id -g fsgqa` + +uqid2=`id -u fsgqa2` +gqid2=`id -g fsgqa2` + +pqid=10 +cat >$tmp.projects <<EOF +$pqid:$SCRATCH_MNT +EOF + +cat >$tmp.projid <<EOF +root:0 +fsgqa:$pqid +EOF + +create_files_unmapped() +{ + local bs=$1 + local inum=$2 + + echo "Using type=$type id=$id" >> $seqres.full + + for ((i=0; i<$((inum-1)); i++)); do + _file_as_id $SCRATCH_MNT/unmapped/inode$i $id $type 1024 0 + done + + _file_as_id $SCRATCH_MNT/unmapped/block $id $type $bs 1 +} + +create_files_idmapped() +{ + local bs=$1 + local inum=$2 + + echo "Using type=$type id=$id2" >> $seqres.full + + for ((i=0; i<$((inum-1)); i++)); do + _file_as_id $SCRATCH_MNT/idmapped/inode$i $id2 $type 1024 0 + done + + _file_as_id $SCRATCH_MNT/idmapped/block $id2 $type $bs 1 +} + +clean_files() +{ + rm -rf $SCRATCH_MNT/unmapped 2>/dev/null + rm -rf $SCRATCH_MNT/idmapped 2>/dev/null + rm -rf $tmp.quot 2>/dev/null + rm -rf $tmp.quota 2>/dev/null +} + +filter_quot() +{ + _filter_quota | grep -v "root \|\#0 " \ + | sed -e '/#[0-9]*/s/#[0-9]*/#ID/g' +} + +filter_report() +{ + _filter_quota | grep -v "^root \|^\#0 " \ + | sed -e '/^#[0-9]*/s/^#[0-9]*/#ID/g' +} + +filter_quota() +{ + _filter_quota | sed -e "/Disk quotas for/s/([0-9]*)/(ID)/g" \ + -e "/Disk quotas for/s/#[0-9]*/#ID/g" +} + +filter_state() +{ + _filter_quota | sed -e "s/Inode: #[0-9]* (0 blocks, 0 extents)/Inode: #[INO] (0 blocks, 0 extents)/g" \ + -e "s/Inode: #[0-9]* ([0-9]* blocks, [0-9]* extents)/Inode: #[INO] (X blocks, Y extents)/g" \ + -e "/[0-9][0-9]:[0-9][0-9]:[0-9][0-9]/s/ [0-9][0-9]:[0-9][0-9]:[0-9][0-9]//g" \ + -e '/max warnings:/d' +} + +test_quot() +{ + local opt="$*" + + echo "checking quot command (type=$type)" + $XFS_QUOTA_PROG -D $tmp.projects -P $tmp.projid -x \ + -c "quot -$type $opt -bi" $SCRATCH_MNT | filter_quot +} + +test_report() +{ + local opt="$*" + + echo "checking report command (type=$type)" + $XFS_QUOTA_PROG -D $tmp.projects -P $tmp.projid -x \ + -c "report -$type $opt -bi" \ + $SCRATCH_MNT | filter_report +} + +test_quota() +{ + local opt="$*" + + echo "checking quota command (type=$type)" + $XFS_QUOTA_PROG -D $tmp.projects -P $tmp.projid -x \ + -c "quota -$type $opt -bi $id" \ + $SCRATCH_MNT | filter_quota +} + +test_limit() +{ + local bs=$1 + local bh=$2 + local is=$3 + local ih=$4 + + echo "checking limit command (type=$type, bsoft=$bs, bhard=$bh, isoft=$is, ihard=$ih)" + $XFS_QUOTA_PROG -D $tmp.projects -P $tmp.projid -x \ + -c "limit -$type bsoft=$bs bhard=$bh fsgqa" \ + -c "limit -$type isoft=$is ihard=$ih fsgqa" \ + $SCRATCH_MNT + + # let the timer day transition happen + sleep 2 +} + +test_timer() +{ + echo "checking timer command (type=$type)" + # set 3days+1h for time won't become 2days soon + $XFS_QUOTA_PROG -D $tmp.projects -P $tmp.projid -x \ + -c "timer -$type -bi 73h" \ + $SCRATCH_MNT | _filter_scratch +} + +test_disable() +{ + echo "checking disable command (type=$type)" + $XFS_QUOTA_PROG -D $tmp.projects -P $tmp.projid -x \ + -c "disable -$type -v" \ + $SCRATCH_MNT | filter_state +} + +test_enable() +{ + echo "checking enable command (type=$type)" + $XFS_QUOTA_PROG -D $tmp.projects -P $tmp.projid -x \ + -c "enable -$type -v" $SCRATCH_MNT | filter_state +} + +test_off() +{ + echo "checking off command (type=$type)" + $XFS_QUOTA_PROG -D $tmp.projects -P $tmp.projid -x \ + -c "off -$type -v" $SCRATCH_MNT | _filter_scratch +} + +test_remove() +{ + echo "checking remove command (type=$type)" + $XFS_QUOTA_PROG -D $tmp.projects -P $tmp.projid -x \ + -c "remove -$type -v" \ + $SCRATCH_MNT | _filter_scratch +} + +test_state() +{ + echo "checking state command (type=$type)" + $XFS_QUOTA_PROG -D $tmp.projects -P $tmp.projid -x \ + -c "state -$type" $SCRATCH_MNT | filter_state +} + +test_dump() +{ + echo "checking dump command (type=$type)" + rm -f $tmp.backup 2>>/dev/null + $XFS_QUOTA_PROG -D $tmp.projects -P $tmp.projid -x \ + -c "dump -$type -f $tmp.backup" \ + $SCRATCH_MNT | _filter_scratch +} + +test_restore() +{ + echo "checking restore command (type=$type)" + $XFS_QUOTA_PROG -D $tmp.projects -P $tmp.projid -x \ + -c "restore -$type -f $tmp.backup" \ + $SCRATCH_MNT | _filter_scratch +} + +wipe_scratch() +{ + wipe_mounts + _scratch_mkfs_xfs >>$seqres.full 2>&1 +} + +qmount_idmapped() +{ + wipe_mounts + _try_scratch_mount || _fail "qmount failed" + + mkdir -p "${SCRATCH_MNT}/unmapped" + mkdir -p "${SCRATCH_MNT}/idmapped" + + $here/src/idmapped-mounts/mount-idmapped \ + --map-mount b:$id:$id2:1 \ + --map-mount b:0:0:1 \ + "$SCRATCH_MNT/unmapped" "$SCRATCH_MNT/idmapped" || _fail "mount-idmapped failed" + + chmod ugo+rwx $SCRATCH_MNT + chmod ugo+rwx $SCRATCH_MNT/unmapped + chmod ugo+rwx $SCRATCH_MNT/idmapped +} + +test_xfs_quota() +{ + # init quota + echo "init quota limit and timer, and dump it" + if [ "$idmapped" -eq 1 ]; then + echo "create_files_idmapped 1024k 15"; create_files_idmapped 1024k 15 + else + echo "create_files_unmapped 1024k 15"; create_files_unmapped 1024k 15 + fi + echo "quota remount"; qmount_idmapped + echo ; test_quot + echo ; test_timer + echo ; test_limit 512k 2048k 10 20 + echo ; test_dump + + # report options test + echo "report options test" + echo ; test_report + echo "-N option"; test_report -N + echo "-L -U options"; test_report -L $id -U $id + echo "-t option"; test_report -t + echo "-n option"; test_report -n + echo "-h option"; test_report -h + + # quot options test + echo "quot options test" + echo ; test_quot + echo "-f option"; test_quot -f $tmp.quot + cat $tmp.quot | filter_quot + echo "-n option"; test_quot -n + + # quota options test + echo ; test_quota + echo "-f option"; test_quota -f $tmp.quota + cat $tmp.quota | filter_quota + echo "-N option"; test_quota -N + echo "-n option"; test_quota -n + echo "-h option"; test_quota -h + + # disable/enable test + echo "disable quota" + echo ; test_disable + echo ; test_report -N + echo "expect a remove error at here"; test_remove + echo ; test_enable + echo ; test_report -N + + # off and remove test + echo "off and remove test" + echo ; test_limit 100m 100m 100 100 + echo ; test_quota -N + echo ; test_off + echo ; test_state + echo ; test_remove + echo ; test_report -N + echo "quota remount"; qmount_idmapped + echo ; test_report -N + + # restore test + echo "restore quota" + echo ; test_restore + echo ; test_report -N + echo ; test_state + echo "cleanup files"; clean_files +} + +echo "----------------------- uquota,sync,unmapped ---------------------------" +wipe_scratch +_qmount_option "uquota,sync" +type=u +id=$uqid +id2=$uqid2 +idmapped=0 +qmount_idmapped +test_xfs_quota + +echo "----------------------- uquota,sync,idmapped ---------------------------" +wipe_scratch +_qmount_option "uquota,sync" +type=u +id=$uqid +id2=$uqid2 +idmapped=1 +qmount_idmapped +test_xfs_quota + +echo "----------------------- gquota,sync,unmapped ---------------------------" +wipe_scratch +_qmount_option "gquota,sync" +type=g +id=$gqid +id2=$gqid2 +idmapped=0 +qmount_idmapped +test_xfs_quota + +echo "----------------------- gquota,sync,idmapped ---------------------------" +wipe_scratch +_qmount_option "gquota,sync" +type=g +id=$gqid +id2=$gqid2 +idmapped=1 +qmount_idmapped +test_xfs_quota + +wipe_mounts + +# success, all done +status=0 +exit
tests/xfs/529.out new file +657 -0
657 lines not on this page · raw message
tests/xfs/group +1 -0
@@ -505,3 +505,4 @@ 526 auto quick mkfs 527 auto quick quota 528 auto quick rw realtime +529 auto quick quota idmapped
Signature · 1 line
2.27.0
PATCH v12 6/6 xfs/530: quotas on idmapped mounts
In reply to Christian BraunerFrom: Christian Brauner <[email protected]>
This is basically a re-implementation of xfs/050 but each file creation call is done through an idmapped mount which verifies that the semantics are identical even when the mount is idmapped.
---
/* v1 - v8 */
patch not present
/* v9 */
- Christian Brauner <[email protected]>:
- Rebased onto current master.
/* v10 */
- Eryu Guan <[email protected]>:
- Remove leading "_" from local helpers.
- Use newly introduced _scratch_{u}mount_idmapped() helpers
/* v11 */
- Amir Goldstein <[email protected]>:
- Add a dedicated "idmapped" tag for idmapped mounts tests.
/* v12 */
unchanged
--- tests/xfs/530 | 212 ++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/530.out | 129 ++++++++++++++++++++++++++++ tests/xfs/group | 1 + 3 files changed, 342 insertions(+) create mode 100644 tests/xfs/530 create mode 100644 tests/xfs/530.out
tests/xfs/530 new file +212 -0
@@ -0,0 +1,212 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0+ +# +# Copyright (c) 2021 Christian Brauner <[email protected]> +# All Rights Reserved. +# +# FS QA Test No. 530 +# +# Exercises basic XFS quota functionality +# uquota, gquota, uqnoenforce, gqnoenforce +# +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! + +# get standard environment, filters and checks +. ./common/rc +. ./common/filter +. ./common/quota + +_cleanup() +{ + cd / + _scratch_unmount 2>/dev/null + rm -f $tmp.* +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# real QA test starts here +_supported_fs xfs + +cp /dev/null $seqres.full +chmod a+rwx $seqres.full # arbitrary users will write here + +_require_scratch +_require_xfs_quota +_require_user fsgqa +_require_test_program "idmapped-mounts/mount-idmapped" + +_scratch_mkfs >/dev/null 2>&1 +_scratch_mount +bsize=$(_get_file_block_size $SCRATCH_MNT) +_scratch_unmount + +bsoft=$(( 200 * $bsize )) +bhard=$(( 1000 * $bsize )) +isoft=4 +ihard=10 + +# The actual point at which limit enforcement takes place for the +# hard block limit is variable depending on filesystem blocksize, +# and iosize. What we want to test is that the limit is enforced +# (ie. blksize less than limit but not unduly less - ~85% is kind) +# nowadays we actually get much closer to the limit before EDQUOT. +# +_filter_and_check_blks() +{ + perl -npe ' + if (/^\#'$id'\s+(\d+)/ && '$enforce') { + $maximum = '$bhard'; + $minimum = '$bhard' * 85/100; + $used = $1 * 1024; + if (($used < $minimum || $used > $maximum) && '$noextsz') { + printf(" URK %d: %d is out of range! [%d,%d]\n", + '$id', $used, $minimum, $maximum); + } + s/^(\#'$id'\s+)(\d+)/\1 =OK=/g; + } + ' | _filter_quota_report +} + +run_tests() +{ + _scratch_mkfs_xfs | _filter_mkfs 2>$tmp.mkfs + cat $tmp.mkfs >>$seqres.full + + # keep the blocksize and data size for dd later + . $tmp.mkfs + + _qmount + + # Figure out whether we're doing large allocations + # (bail out if they're so large they stuff the test up) + _test_inode_flag extsz-inherit $SCRATCH_MNT + noextsz=$? + extsize=`_test_inode_extsz $SCRATCH_MNT` + [ $extsize -ge 512000 ] && \ + _notrun "Extent size hint is too large ($extsize bytes)" + + _qsetup $1 + + echo "Using type=$type id=$id" >>$seqres.full + + $XFS_QUOTA_PROG -x -c "warn -$type 65535 -d" $SCRATCH_DEV + + echo + echo "*** report no quota settings" | tee -a $seqres.full + $XFS_QUOTA_PROG -D $tmp.projects -P $tmp.projid -x \ + -c "repquota -birnN -$type" $SCRATCH_DEV | + _filter_quota_report | LC_COLLATE=POSIX sort -ru + + echo + echo "*** report initial settings" | tee -a $seqres.full + _scratch_mount_idmapped $type $id + _file_as_id $SCRATCH_MNT/initme 0 $type 1024 0 + _scratch_umount_idmapped + echo "ls -l $SCRATCH_MNT" >>$seqres.full + ls -l $SCRATCH_MNT >>$seqres.full + $XFS_QUOTA_PROG -D $tmp.projects -P $temp.projid -x \ + -c "limit -$type bsoft=${bsoft} bhard=${bhard} $id" \ + -c "limit -$type isoft=$isoft ihard=$ihard $id" \ + $SCRATCH_DEV + $XFS_QUOTA_PROG -D $tmp.projects -P $tmp.projid -x \ + -c "repquota -birnN -$type" $SCRATCH_DEV | + _filter_quota_report | LC_COLLATE=POSIX sort -ru + + echo + echo "*** push past the soft inode limit" | tee -a $seqres.full + _scratch_mount_idmapped $type $id + _file_as_id $SCRATCH_MNT/softie1 0 $type 1024 0 + _file_as_id $SCRATCH_MNT/softie2 0 $type 1024 0 + _file_as_id $SCRATCH_MNT/softie3 0 $type 1024 0 + _file_as_id $SCRATCH_MNT/softie4 0 $type 1024 0 + _scratch_umount_idmapped + _qmount + $XFS_QUOTA_PROG -x -c "warn -i -$type 0 $id" $SCRATCH_DEV + $XFS_QUOTA_PROG -D $tmp.projects -P $tmp.projid -x \ + -c "repquota -birnN -$type" $SCRATCH_DEV | + _filter_quota_report | LC_COLLATE=POSIX sort -ru + + echo + echo "*** push past the soft block limit" | tee -a $seqres.full + _scratch_mount_idmapped $type $id + _file_as_id $SCRATCH_MNT/softie 0 $type $bsize 300 + _scratch_umount_idmapped + _qmount + $XFS_QUOTA_PROG -x -c "warn -i -$type 0 $id" \ + -c "warn -b -$type 0 $id" $SCRATCH_DEV + $XFS_QUOTA_PROG -D $tmp.projects -P $tmp.projid -x \ + -c "repquota -birnN -$type" $SCRATCH_DEV | + _filter_quota_report | LC_COLLATE=POSIX sort -ru + + echo + # Note: for quota accounting (not enforcement), EDQUOT is not expected + echo "*** push past the hard inode limit (expect EDQUOT)" | tee -a $seqres.full + for i in 1 2 3 4 5 6 7 8 9 10 11 12 + do + _scratch_mount_idmapped $type $id + _file_as_id $SCRATCH_MNT/hard$i 0 $type 1024 0 + _scratch_umount_idmapped + done + _qmount + $XFS_QUOTA_PROG -x -c "warn -b -$type 0 $id" \ + -c "warn -i -$type 0 $id" $SCRATCH_DEV + $XFS_QUOTA_PROG -D $tmp.projects -P $tmp.projid -x \ + -c "repquota -birnN -$type" $SCRATCH_DEV | + _filter_quota_report | LC_COLLATE=POSIX sort -ru + + echo + # Note: for quota accounting (not enforcement), EDQUOT is not expected + echo "*** push past the hard block limit (expect EDQUOT)" | tee -a $seqres.full + _scratch_mount_idmapped $type $id + _file_as_id $SCRATCH_MNT/softie 0 $type $bsize 1200 + _scratch_umount_idmapped + echo "ls -l $SCRATCH_MNT" >>$seqres.full + ls -l $SCRATCH_MNT >>$seqres.full + _qmount + $XFS_QUOTA_PROG -x -c "warn -b -$type 0 $id" $SCRATCH_DEV + $XFS_QUOTA_PROG -D $tmp.projects -P $tmp.projid -x \ + -c "repquota -birnN -$type" $SCRATCH_DEV | + _filter_and_check_blks | LC_COLLATE=POSIX sort -ru + + echo + echo "*** unmount" + _scratch_unmount + +} + +cat >$tmp.projects <<EOF +1:$SCRATCH_MNT +EOF + +cat >$tmp.projid <<EOF +root:0 +scrach:1 +EOF + +projid_file="$tmp.projid" + +echo "*** user" +_qmount_option "uquota" +run_tests u + +echo "*** group" +_qmount_option "gquota" +run_tests g + +echo "*** uqnoenforce" +_qmount_option "uqnoenforce" +run_tests uno + +echo "*** gqnoenforce" +_qmount_option "gqnoenforce" +run_tests gno + +# success, all done +status=0 +exit
tests/xfs/530.out new file +129 -0
129 lines not on this page · raw message
tests/xfs/group +1 -0
@@ -506,3 +506,4 @@ 527 auto quick quota 528 auto quick rw realtime 529 auto quick quota idmapped +530 auto quick quota idmapped
Signature · 1 line
2.27.0
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH v12 2/6] generic/632: add fstests for idmapped mounts
Quoted 8 lines · Christian Brauner
From: Christian Brauner <[email protected]>
Add a test suite to verify the behavior of idmapped mounts. The test suite also includes a range of vfs tests to verify that no regressions are introduced by idmapped mounts. The following tests are currently available with more to come in the future:
01. posix acls on regular and idmapped mounts
I'm getting failures like below when testing on btrfs and overlayfs
[root@fedoravm xfstests]# diff -u/root/workspace/xfstests/tests/generic/633.out /root/workspace/xfstests/results//btrfs/generic/633.out.bad
root/workspace/xfstests/results//btrfs/generic/633.out.bad 2021-04-11 22:19:58.887980770 +0800 renamed from root/workspace/xfstests/tests/generic/633.out 2021-04-11 22:18: 24.458518716 +0800 +0 -0
@@ -1,2 +1,4 @@ QA output created by 633 Silence is golden
+idmapped-mounts.c: 6622: acls - Invalid argument - failure: sys_mount_setattr +failure: posix acls on regular mounts
Is this a known issue or test needs some fix?
[snip]
Quoted 1 line
create mode 100644 tests/generic/632
Please change mode of new test file to 755 if you end up with sending new version, otherwise I can fix it on commit.
Thanks,
Eryu
Re: [PATCH v12 2/6] generic/632: add fstests for idmapped mounts
Quoted 22 lines · Eryu Guan
Quoted 8 lines · Christian Brauner
From: Christian Brauner <[email protected]>
Add a test suite to verify the behavior of idmapped mounts. The test suite also includes a range of vfs tests to verify that no regressions are introduced by idmapped mounts. The following tests are currently available with more to come in the future:
01. posix acls on regular and idmapped mounts
I'm getting failures like below when testing on btrfs and overlayfs
[root@fedoravm xfstests]# diff -u
/root/workspace/xfstests/tests/generic/633.out/root/workspace/xfstests/results//btrfs/generic/633.out.badroot/workspace/xfstests/results//btrfs/generic/633.out.bad 2021-04-11 22:19:58.887980770 +0800 renamed from root/workspace/xfstests/tests/generic/633.out 2021-04-11 22:18: 24.458518716 +0800 +0
-0@@ -1,2 +1,4 @@ QA output created by 633 Silence is golden +idmapped-mounts.c: 6622: acls - Invalid argument - failure: sys_mount_setattr +failure: posix acls on regular mounts
Quoted 1 line
Is this a known issue or test needs some fix?
Ah, this is very likely just the getfacl command missing which is part of the "acl" package, at least on Ubuntu.
Christian
Re: [PATCH v12 2/6] generic/632: add fstests for idmapped mounts
Quoted 23 lines · Christian Brauner
Quoted 22 lines · Eryu Guan
Quoted 8 lines · Christian Brauner
From: Christian Brauner <[email protected]>
Add a test suite to verify the behavior of idmapped mounts. The test suite also includes a range of vfs tests to verify that no regressions are introduced by idmapped mounts. The following tests are currently available with more to come in the future:
01. posix acls on regular and idmapped mounts
I'm getting failures like below when testing on btrfs and overlayfs
[root@fedoravm xfstests]# diff -u
/root/workspace/xfstests/tests/generic/633.out/root/workspace/xfstests/results//btrfs/generic/633.out.badroot/workspace/xfstests/results//btrfs/generic/633.out.bad 2021-04-11 22:19:58.887980770 +0800 renamed from root/workspace/xfstests/tests/generic/633.out 2021-04-11 22:18: 24.458518716 +0800 +0
-0@@ -1,2 +1,4 @@ QA output created by 633 Silence is golden +idmapped-mounts.c: 6622: acls - Invalid argument - failure: sys_mount_setattr +failure: posix acls on regular mounts
Quoted 4 lines
Quoted 1 line
Is this a known issue or test needs some fix?
Ah, this is very likely just the getfacl command missing which is part of the "acl" package, at least on Ubuntu.
And it should actually skip the tests on all filesystems that don't support idmapped mounts which is every fs apart from xfs and ext4.
Christian
Re: [PATCH v12 2/6] generic/632: add fstests for idmapped mounts
Quoted 24 lines · Christian Brauner
Quoted 23 lines · Christian Brauner
Quoted 22 lines · Eryu Guan
Quoted 8 lines · Christian Brauner
From: Christian Brauner <[email protected]>
Add a test suite to verify the behavior of idmapped mounts. The test suite also includes a range of vfs tests to verify that no regressions are introduced by idmapped mounts. The following tests are currently available with more to come in the future:
01. posix acls on regular and idmapped mounts
I'm getting failures like below when testing on btrfs and overlayfs
[root@fedoravm xfstests]# diff -u
/root/workspace/xfstests/tests/generic/633.out/root/workspace/xfstests/results//btrfs/generic/633.out.badroot/workspace/xfstests/results//btrfs/generic/633.out.bad 2021-04-11 22:19:58.887980770 +0800 renamed from root/workspace/xfstests/tests/generic/633.out 2021-04-11 22:18: 24.458518716 +0800 +0
-0@@ -1,2 +1,4 @@ QA output created by 633 Silence is golden +idmapped-mounts.c: 6622: acls - Invalid argument - failure: sys_mount_setattr +failure: posix acls on regular mounts
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Is this a known issue or test needs some fix?
Ah, this is very likely just the getfacl command missing which is part of the "acl" package, at least on Ubuntu.
And it should actually skip the tests on all filesystems that don't support idmapped mounts which is every fs apart from xfs and ext4.
Yeah, looks like that's the case. So it seems the _require idmapped mount helper doesn't do the job properly.
Eryu
Re: [PATCH v12 2/6] generic/632: add fstests for idmapped mounts
Quoted 30 lines · Eryu Guan
Quoted 29 lines · Christian Brauner
Quoted 27 lines · Christian Brauner
Quoted 22 lines · Eryu Guan
Quoted 8 lines · Christian Brauner
From: Christian Brauner <[email protected]>
Add a test suite to verify the behavior of idmapped mounts. The test suite also includes a range of vfs tests to verify that no regressions are introduced by idmapped mounts. The following tests are currently available with more to come in the future:
01. posix acls on regular and idmapped mounts
I'm getting failures like below when testing on btrfs and overlayfs
[root@fedoravm xfstests]# diff -u
/root/workspace/xfstests/tests/generic/633.out/root/workspace/xfstests/results//btrfs/generic/633.out.badroot/workspace/xfstests/results//btrfs/generic/633.out.bad 2021-04-11 22:19:58.887980770 +0800 renamed from root/workspace/xfstests/tests/generic/633.out 2021-04-11 22:18: 24.458518716 +0800 +0
-0@@ -1,2 +1,4 @@ QA output created by 633 Silence is golden +idmapped-mounts.c: 6622: acls - Invalid argument - failure: sys_mount_setattr +failure: posix acls on regular mountsQuoted 1 line
Is this a known issue or test needs some fix?
Ah, this is very likely just the getfacl command missing which is part of the "acl" package, at least on Ubuntu.
Quoted 4 lines
Quoted 2 lines
And it should actually skip the tests on all filesystems that don't support idmapped mounts which is every fs apart from xfs and ext4.
Yeah, looks like that's the case. So it seems the
_requireidmapped
Ah, ok I didn't know this needed to go in there. I thinke the following might be enough. Are you able to simply apply it on top?
common/rc +10 -0
@@ -2047,6 +2047,16 @@ _require_mount_setattr() # test whether idmapped mounts are supported _require_idmapped_mounts() { + case "$FSTYP" in + xfs) + ;; + ext4) + ;; + *) + _notrun "Filesystem $FSTYP does not support idmapped mounts yet" + ;; + esac + IDMAPPED_MOUNTS_TEST=$here/src/idmapped-mounts/idmapped-mounts [ -x $IDMAPPED_MOUNTS_TEST ] || _notrun "idmapped-mounts utilities required"
Christian
Theodore Ts'o
maintainerRe: [PATCH v12 2/6] generic/632: add fstests for idmapped mounts
Quoted 11 lines · Christian Brauner
Ah, ok I didn't know this needed to go in there. I thinke the following might be enough. Are you able to simply apply it on top?
common/rc +0
-0@@ -2047,6 +2047,16 @@ _require_mount_setattr() # test whether idmapped mounts are supported _require_idmapped_mounts() {
common/rc _require_mount_setattr() · line 2059
Quoted 10 lines
common/rc +10
-0+ case "$FSTYP" in + xfs) + ;; + ext4) + ;; + *) + _notrun "Filesystem $FSTYP does not support idmapped mounts yet" + ;; + esac +
Is there any way we can ask the kernel which file systems support idmapped mounts? That way, the tests will do the right thing when run on older LTS kernels, and if a distribution backports idmapped support for some file system into their kernel, again, the right thing can happen automatically.
If you can't do this by checking to see if the file system will support a particular mount option, or some other run-time test, for ext4 we can signal this by checking for the existence of a file in /sys/fs/ext4/features, such as /sys/fs/ext4/features/fast_commit. (Grep for EXT4_ATTR_FEATURE and ATTR_LIST in fs/ext4/sysfs.c; it requires adding two lines to advertise a new ext4 feature.)
- Ted
Re: [PATCH v12 2/6] generic/632: add fstests for idmapped mounts
Quoted 20 lines · Theodore Ts'o
Quoted 19 lines · Christian Brauner
Ah, ok I didn't know this needed to go in there. I thinke the following might be enough. Are you able to simply apply it on top?
common/rc +8
-0@@ -2047,6 +2047,16 @@ _require_mount_setattr() # test whether idmapped mounts are supported _require_idmapped_mounts() { + case "$FSTYP" in + xfs) + ;; + ext4) + ;; + *) + _notrun "Filesystem $FSTYP does not support idmapped mounts yet" + ;;
Quoted 8 lines
common/rc _require_mount_setattr() · line 2059
Quoted 2 lines
common/rc +2
-0+ esac +Is there any way we can ask the kernel which file systems support idmapped mounts? That way, the tests will do the right thing when run on older LTS kernels, and if a distribution backports idmapped support for some file system into their kernel, again, the right thing can happen automatically.
So we can detect it pretty reliably at runtime by trying whether we can create an idmapped mount on the given filesystem. That is enough for the idmapped mount tests here but of course has at least two drawbacks:
1. there might be scenarios where we get false negatives (e.g. open_tree() could fail for a lack of permissions or sm else, kernel might be compiled without userns support etc. pp)
2. it's heavy in so far as we have to do the whole exercise of creating a detached mount
So having a reliable way to detect whether or not the underlying fs supports it could be worth it (My hope was for the fsinfo() API to grow this "feature check" ability but oh well.).
One possibility might be to extend fstatfs() and steal one u32 from the padding that is currently in there?
Quoted 7 lines
If you can't do this by checking to see if the file system will support a particular mount option, or some other run-time test, for ext4 we can signal this by checking for the existence of a file in
/sys/fs/ext4/features, such as/sys/fs/ext4/features/fast_commit. (Grep forEXT4_ATTR_FEATUREandATTR_LISTinfs/ext4/sysfs.c; it requires adding two lines to advertise a new ext4 feature.)
I wonder if this wouldn't be nice to have independent of whether or not there is another way to detect it?
I'm would think that people like to see all new ext4 features listed in there. Even if this is technically a generic vfs feature.
Christian
Theodore Ts'o
maintainerRe: [PATCH v12 2/6] generic/632: add fstests for idmapped mounts
Quoted 6 lines · Christian Brauner
So we can detect it pretty reliably at runtime by trying whether we can create an idmapped mount on the given filesystem. That is enough for the idmapped mount tests here but of course has at least two drawbacks:
1. there might be scenarios where we get false negatives (e.g.open_tree()could fail for a lack of permissions or sm else, kernel might be compiled without userns support etc. pp)
In the ideal world, if the kernel wasn't compiled with the necessary CONFIG options enabled, it's desirable of the test can detect that fact and skip running the test instead failing and forcing the person running the test to try to figure out whether this is a legitmate file system bug or a just a test setup bug.
This is one of the other functions of /sys/fs/ext4/features/... for example:
#ifdef CONFIG_UNICODEEXT4_ATTR_FEATURE(casefold);
#endif
That way, if a particular kernel CONFIG is not compiled in, we can prevent the feature from being advertised, without having to rely on /proc/config.gz be present.
Quoted 2 lines
2. it's heavy in so far as we have to do the whole exercise of creating a detached mount
Yeah, there are times when I've added a new feature flag file in /sys/fs/ext4/features because it was simpler than being able to doing some kind of heavyweight test.
Quoted 6 lines
So having a reliable way to detect whether or not the underlying fs supports it could be worth it (My hope was for the
fsinfo()API to grow this "feature check" ability but oh well.).One possibility might be to extend
fstatfs()and steal one u32 from the padding that is currently in there?
That would require mounting a file system in order to do the check, which is not necesarily always convenient, but that would certainly work. (See below for an example of where it was much easier for mke2fs to be able to test for a file system "capability" without needing to mount a test file system.)
Quoted 11 lines
Quoted 6 lines
If you can't do this by checking to see if the file system will support a particular mount option, or some other run-time test, for ext4 we can signal this by checking for the existence of a file in
/sys/fs/ext4/features, such as/sys/fs/ext4/features/fast_commit. (Grep forEXT4_ATTR_FEATUREandATTR_LISTinfs/ext4/sysfs.c; it requires adding two lines to advertise a new ext4 feature.)I wonder if this wouldn't be nice to have independent of whether or not there is another way to detect it?
I'm would think that people like to see all new ext4 features listed in there. Even if this is technically a generic vfs feature.
My apologies for the confusion. There's a bit of overloading in terms of the word "features" as used above. In some cases, there is a direct correspondance between bits that show up in the compat, ro_compat, and incompat fields in the ext4 superblock. In other cases it might describe a file system behaviour (e.g., lazy_itable_init) and this acts as a hint to e2fsprogs about what kind of defaults it should use when creating a file system --- or to warn the user about whether a particular file system can be mounted using the current kernel.
For example, currently we give the following warning if you create a file system with a 64k block size:
% mke2fs -t ext4 -b 65536 /tmp/foo.img 8M
Warning: blocksize 65536 not usable on most systems. mke2fs 1.46.2 (28-Feb-2021)
Creating regular file /tmp/foo.img
mke2fs: 65536-byte blocks too big for system (max 4096) Proceed anyway? (y,N)
However, suppose at some point, thanks to advances in the MM layer, it becomes possible to support file systems where the block size is greater than the page size. We might then advertise that via the existing of a file such as /sys/fs/ext4/feature/big_blocks (for example), and then newer versions of mke2fs would check for that file and suppress the warning on a kernel which would actually allow file systems with 64k blocks to be mounted on an x86_64 system. This is also why I designed it via using a file in /sys/fs/ext4/features, so that I don't have to do a test mount and use a system call fstatfs(2) to check for the existenace of an ext4 feature/capability (all of the obvious terms are already overloaded, sigh).
I agree that having some kind of fs-independent way for querying whether some feature/capability is present for a particular kernel would be convenient. This wouldn't necessarily need to be implemented in the VFS, though, especially if the convention that gets adopted is via a file in /sys/fs/<fstype>/features/<capability>.
- Ted
Theodore Ts'o
maintainerPATCH 1/1 ext4: add feature file to advertise that ext4 supports idmapped mounts
Quoted 5 lines · Theodore Ts'o
In the ideal world, if the kernel wasn't compiled with the necessary CONFIG options enabled, it's desirable of the test can detect that fact and skip running the test instead failing and forcing the person running the test to try to figure out whether this is a legitmate file system bug or a just a test setup bug.
So it would make it easier for me to manage running xfstests on ext4 if I had added something like this to ext4 and sent it to Linus before v5.12 is released. What do folks think?
- Ted
commit 20619aefe69d39e76083d8f8598653c2dca9b47e
Author: Theodore Ts'o <[email protected]> Date: Wed Apr 14 16:42:47 2021 -0400
ext4: add feature file to advertise that ext4 supports idmapped mounts
This makes it easier for automated test suites to know whether it know whether we should test the functionality of the new idmapped mounts feature introduced in v5.12-rc1.
fs/ext4/sysfs.c +6 -0
@@ -313,6 +313,9 @@ EXT4_ATTR_FEATURE(verity);#endif EXT4_ATTR_FEATURE (metadata_csum_seed);EXT4_ATTR_FEATURE (fast_commit); +#ifdef CONFIG_USER_NS +EXT4_ATTR_FEATURE (idmapped_mount); +#endif static struct attribute *ext4_feat_attrs[] = {ATTR_LIST (lazy_itable_init),
@@ -330,6 +333,9 @@ static struct attribute *ext4_feat_attrs[] = {#endif ATTR_LIST (metadata_csum_seed),ATTR_LIST (fast_commit ), +#ifdef CONFIG_USER_NS +ATTR_LIST (idmapped_mount ), +#endif NULL, };ATTRIBUTE_GROUPS (ext4_feat );
Re: [PATCH -RFC] ext4: add feature file to advertise that ext4 supports idmapped mounts
Quoted 10 lines · Theodore Ts'o
Quoted 5 lines · Theodore Ts'o
In the ideal world, if the kernel wasn't compiled with the necessary CONFIG options enabled, it's desirable of the test can detect that fact and skip running the test instead failing and forcing the person running the test to try to figure out whether this is a legitmate file system bug or a just a test setup bug.
So it would make it easier for me to manage running xfstests on ext4 if I had added something like this to ext4 and sent it to Linus before v5.12 is released. What do folks think?
Completely stupid. This is a VFS-level feature and we need to have proper VFS-level testing (which we have through creating a mount with the option), not fs-specific band aids.
Re: [PATCH -RFC] ext4: add feature file to advertise that ext4 supports idmapped mounts
Quoted 8 lines · Christoph Hellwig
Quoted 7 lines · Theodore Ts'o
Quoted 5 lines · Theodore Ts'o
In the ideal world, if the kernel wasn't compiled with the necessary CONFIG options enabled, it's desirable of the test can detect that fact and skip running the test instead failing and forcing the person running the test to try to figure out whether this is a legitmate file system bug or a just a test setup bug.
Quoted 7 lines
Quoted 3 lines
So it would make it easier for me to manage running xfstests on ext4 if I had added something like this to ext4 and sent it to Linus before v5.12 is released. What do folks think?
Completely stupid. This is a VFS-level feature and we need to have proper VFS-level testing (which we have through creating a mount with the option), not fs-specific band aids.
Harsh words. :)
Christoph's right though I think for the xfstests we don't need it and we're covered with what we have in the version I sent out last Sunday.
However, as a general way of advertising to users that ext4 supports idmapped mounts I don't necessarily see a problem with that. It doesn't have implications for other filesystems and ext4 already advertises other features in a similar way.
Christian
Re: [PATCH -RFC] ext4: add feature file to advertise that ext4 supports idmapped mounts
Quoted 4 lines · Christian Brauner
However, as a general way of advertising to users that ext4 supports idmapped mounts I don't necessarily see a problem with that. It doesn't have implications for other filesystems and ext4 already advertises other features in a similar way.
I still think that is a very bad idea. idmapped mounts are foremost a VFS feature, with just a tiny bit of glue in every file system that does not affect the on-disk format. So any discovery for it needs to be using VFS infrastructure. IMHO just trying to mount is the proper way to do it, but if people want some other form of discovery it needs to be something in generic code.
Re: [PATCH -RFC] ext4: add feature file to advertise that ext4 supports idmapped mounts
Quoted 6 lines · Christoph Hellwig
Quoted 4 lines · Christian Brauner
However, as a general way of advertising to users that ext4 supports idmapped mounts I don't necessarily see a problem with that. It doesn't have implications for other filesystems and ext4 already advertises other features in a similar way.
Quoted 6 lines
I still think that is a very bad idea. idmapped mounts are foremost a VFS feature, with just a tiny bit of glue in every file system that does not affect the on-disk format. So any discovery for it needs to be using VFS infrastructure. IMHO just trying to mount is the proper way to do it, but if people want some other form of discovery it needs to be something in generic code.
I agree with you there.
I think fsinfo() (Arguably a slimmed-down version of it.) would've been the best place for this.
Theodore Ts'o
maintainerRe: [PATCH -RFC] ext4: add feature file to advertise that ext4 supports idmapped mounts
In reply to Christian BraunerQuoted 3 lines · Christian Brauner
Harsh words. :)
Christoph's right though I think for the xfstests we don't need it and we're covered with what we have in the version I sent out last Sunday.
Sorry, I had missed your v13 patch set and that it had included tests for the existence of idmapped. I had sent out the RFC patch because I was under the impression that we hadn't made forward progress on testing for the support idmapped mounts.
If we have a way of doing this, and you're comfortable that it is reliable (e.g., that a bug might cause the test to be skipped because it thinks the file system doesn't support idmapped mount, when really it was caused by a regression), and I'm happy to just rely on the method you've used in the v13 fstests patch set.
That being said, I still think it would be helpful to have a VFS-standard way for userspace to be able to test for the presence of a particular kernel feature/capability without having to do a test mount, possibly requiring setting up a test user_ns, etc., etc. But that isn't as urgent as making sure we can easily the feature without needing manual customization of the test suite as file systems add support for the feature, or as the feature gets backported to enterprise distro kernels.
Cheers,
- Ted
Re: [PATCH v12 2/6] generic/632: add fstests for idmapped mounts
In reply to Christian BraunerQuoted 7 lines · Christian Brauner
Quoted 4 lines
Quoted 2 lines
And it should actually skip the tests on all filesystems that don't support idmapped mounts which is every fs apart from xfs and ext4.
Yeah, looks like that's the case. So it seems the
_requireidmappedAh, ok I didn't know this needed to go in there. I thinke the following might be enough. Are you able to simply apply it on top?
No, this is broken. It needs to do feature detection, not some arbitrary whitelists. And I'm pretty sure we got that right at some point by doing the _notrun when mounting with the idmap option was not supported.
Re: [PATCH v12 2/6] generic/632: add fstests for idmapped mounts
Quoted 11 lines · Christoph Hellwig
Quoted 7 lines · Christian Brauner
Quoted 4 lines
Quoted 2 lines
And it should actually skip the tests on all filesystems that don't support idmapped mounts which is every fs apart from xfs and ext4.
Yeah, looks like that's the case. So it seems the
_requireidmappedAh, ok I didn't know this needed to go in there. I thinke the following might be enough. Are you able to simply apply it on top?
No, this is broken. It needs to do feature detection, not some arbitrary whitelists. And I'm pretty sure we got that right at some point by
I thought so too.
Quoted 1 line
doing the
_notrunwhen mounting with the idmap option was not supported.
The thing is we need the fileystem type we're going to be tested on. I can place this into src/feature.c if this can get access to the fstype the test is going to be run on. It should be but if not we can do this in src/idmapped-mounts/idmapped-mounts.c
directly to test whether the underlying fs we're being tested on supports idmapped mounts and then return an error value from the binary indicating that it doesn't so we can use _notrun.
I'll post an updated version of this today with the detection logic.
Christian
Re: [PATCH v12 2/6] generic/632: add fstests for idmapped mounts
In reply to Christian BraunerQuoted 23 lines · Christian Brauner
Quoted 22 lines · Eryu Guan
Quoted 8 lines · Christian Brauner
From: Christian Brauner <[email protected]>
Add a test suite to verify the behavior of idmapped mounts. The test suite also includes a range of vfs tests to verify that no regressions are introduced by idmapped mounts. The following tests are currently available with more to come in the future:
01. posix acls on regular and idmapped mounts
I'm getting failures like below when testing on btrfs and overlayfs
[root@fedoravm xfstests]# diff -u
/root/workspace/xfstests/tests/generic/633.out/root/workspace/xfstests/results//btrfs/generic/633.out.badroot/workspace/xfstests/results//btrfs/generic/633.out.bad 2021-04-11 22:19:58.887980770 +0800 renamed from root/workspace/xfstests/tests/generic/633.out 2021-04-11 22:18: 24.458518716 +0800 +0
-0@@ -1,2 +1,4 @@ QA output created by 633 Silence is golden +idmapped-mounts.c: 6622: acls - Invalid argument - failure: sys_mount_setattr +failure: posix acls on regular mounts
Quoted 4 lines
Quoted 1 line
Is this a known issue or test needs some fix?
Ah, this is very likely just the getfacl command missing which is part of the "acl" package, at least on Ubuntu.
I do have getfacl command installed, and test passed if I tested on ext4 or xfs. So I don't think it's related to getfacl command.
Thanks,
Eryu
Quoted 9 lines · Christian Brauner
From: Christian Brauner <[email protected]>
Hey everyone,
This series is available from:
https://git.kernel.org/brauner/xfstests-dev/h/idmapped_mounts
https://gitlab.com/brauner/xfstests/-/tree/idmapped_mounts
https://github.com/brauner/xfstests/tree/idmapped_mounts
Quoted 5 lines
/* v12 */
- Main change is a fix to theio_uringtests.io_uring_wait_cqe()doesn't set errno, it returns a negative errno.
- Make sure we also run without error when libcap isn't available (That was always the goal but the logic missed a few new tests.)
Thanks for the revision! The whole patchset looks fine to me, except that I'm not sure if the generic/632 failure on btrfs & overlayfs is expected.
With that resolved (either it's expected failure or test needs fix), I think the patchset is ready to be merged.
Thanks,
Eryu
Quoted 84 lines
/* v11 */
Reworked according to Amir's comments./* v10 */
Reworked according to Eryu's comments./* v9 */
Rebased onto current master.ubuntu@f1-vm:~/src/git/xfstests$ sudo ./check -g idmapped FSTYP -- xfs (debug) PLATFORM -- Linux/x86_64 f1-vm 5.12.0-rc3-syz #18 SMP PREEMPT Fri Mar 26 13:27:16 UTC 2021 MKFS_OPTIONS -- -f -bsize=4096 /dev/loop1 MOUNT_OPTIONS -- /dev/loop1 /mnt/scratchgeneric/632 files ... 27s
xfs/529 files ... 67s
xfs/530 files ... 43s
Ran: generic/632 xfs/529 xfs/530
Passed all 3 testsThanks!
ChristianChristian Brauner (6):
generic/631: add test for detached mount propagation generic/632: add fstests for idmapped mounts common/rc: add _scratch_{u}mount_idmapped() helpers common/quota: move_qsetup()helper to common code xfs/529: quotas and idmapped mounts xfs/530: quotas on idmapped mounts.gitignore | 3 + README | 5 +- common/quota | 20 + common/rc | 60 + configure.ac | 2 + include/builddefs.in | 1 + m4/Makefile | 1 + m4/package_libcap.m4 | 4 + src/Makefile | 8 +- src/detached_mounts_propagation.c | 189 + src/feature.c | 40 +- src/idmapped-mounts/Makefile | 40 + src/idmapped-mounts/idmapped-mounts.c | 8873 +++++++++++++++++++++++++ src/idmapped-mounts/missing.h | 151 + src/idmapped-mounts/mount-idmapped.c | 431 ++ src/idmapped-mounts/utils.c | 134 + src/idmapped-mounts/utils.h | 30 + tests/generic/631 | 43 + tests/generic/631.out | 2 + tests/generic/632 | 42 + tests/generic/632.out | 2 + tests/generic/group | 2 + tests/xfs/050 | 19 - tests/xfs/299 | 19 - tests/xfs/529 | 377 ++ tests/xfs/529.out | 657 ++ tests/xfs/530 | 212 + tests/xfs/530.out | 129 + tests/xfs/group | 2 + 29 files changed, 11453 insertions(+), 45 deletions(-) create mode 100644 m4/package_libcap.m4 create mode 100644 src/detached_mounts_propagation.c create mode 100644 src/idmapped-mounts/Makefile create mode 100644 src/idmapped-mounts/idmapped-mounts.c create mode 100644 src/idmapped-mounts/missing.h create mode 100644 src/idmapped-mounts/mount-idmapped.c create mode 100644 src/idmapped-mounts/utils.c create mode 100644 src/idmapped-mounts/utils.h create mode 100644 tests/generic/631 create mode 100644 tests/generic/631.out create mode 100644 tests/generic/632 create mode 100644 tests/generic/632.out create mode 100644 tests/xfs/529 create mode 100644 tests/xfs/529.out create mode 100644 tests/xfs/530 create mode 100644 tests/xfs/530.outbase-commit:
f6ddaf130d5b0817278afe441fdde52f464f321bSignature · 1 line
2.27.0