Add support for central fsconfig and -q <n> unconditional loop

This series introduces 2 new features, central fsconfig support and "-q <n>" option to support unconditional looping. Along with this, it also brings some improvements to output data both, in stdout and in result.log. Central fsconfig feature allows us to define MKFS and MOUNT options for different filesystems and use this central config to invoke any test. We don't need to maintain different sections in local.config file for different mkfs and mount options of any given FS.
e.g.
./check -c configs/xfs/4k,configs/ext4/1k -g quick

Central fsconfigs can be useful for below reasons:-
1. Testers and other FS developers can know what is being actively tested and maintained by FS Maintainers.
2. It "reduces" the need to maintain different local.config files or different sections within local.config file for subsystem maintainers for testing different filesystem configurations.
-q <n> option can be useful for fast unconditional looping (compared to -I <n>) to test a certain flakey test and get it's pass/fail metrics.

This is similar to -L <n>, although with -L the test only loops if it fails the 1st time.
The individual patches can provide more details about other changes and improvements.

This is something that was discussed during last LSFMM [1]
[1] https://lore.kernel.org/all/[email protected]/

Nirjhar Roy (IBM) (5):
tests/selftest: Add a new pseudo flaky test.
check: Add -q <n> option to support unconditional looping.
check: Improve pass/fail metrics and section config output check,common/config: Add support for central fsconfig
configs: Add a couple of xfs and ext4 configuration templates.

 README.config-sections |  12 +++
 check                  | 193 +++++++++++++++++++++++++++++++----------
 common/config          |  51 ++++++++++-
 common/report          |   2 +-
 configs/ext4/1k        |   3 +
 configs/ext4/4k        |   3 +
 configs/ext4/64k       |   3 +
 configs/xfs/1k         |   3 +
 configs/xfs/4k         |   3 +
 configs/xfs/64k        |   3 +
 configs/xfs/adv        |   3 +
 configs/xfs/quota      |   3 +
 tests/selftest/007     |  21 +++++
 tests/selftest/007.out |   2 +
 14 files changed, 256 insertions(+), 49 deletions(-)
 create mode 100644 configs/ext4/1k
 create mode 100644 configs/ext4/4k
 create mode 100644 configs/ext4/64k
 create mode 100644 configs/xfs/1k
 create mode 100644 configs/xfs/4k
 create mode 100644 configs/xfs/64k
 create mode 100644 configs/xfs/adv
 create mode 100644 configs/xfs/quota
 create mode 100755 tests/selftest/007
 create mode 100644 tests/selftest/007.out

--
2.34.1

This test is to simulate the behavior of a flaky test. This will be required when we will make some modifications to the pass/fail metric calculation of the test infrastructure where we will need a test with non-zero pass and non-zero failure rate.

---
 tests/selftest/007     | 21 +++++++++++++++++++++
 tests/selftest/007.out |  2 ++
 2 files changed, 23 insertions(+)
 create mode 100755 tests/selftest/007
 create mode 100644 tests/selftest/007.out

new file mode 100755
index 00000000..f100ec5f
@@ -0,0 +1,21 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 IBM Corporation.  All Rights Reserved.
+# Author: Nirjhar Roy (IBM) <[email protected]>
+#
+# FS QA Test 007
+#
+# This test is to simulate the behavior of a flakey test.
+#
+
+. ./common/preamble
+_begin_fstest selftest
+
+if (($RANDOM % 2)); then
+    echo "Silence is golden"
+else
+    echo "Silence is flakey"
+fi
+
+status=0
+exit

new file mode 100644
index 00000000..fd3590e6
@@ -0,0 +1,2 @@
+QA output created by 007
+Silence is golden

2.34.1

This patch adds -q <n> option through which one can run a given test <n> times unconditionally. It also prints pass/fail metrics at the end.

The advantage of this over -L <n> and -i/-I <n> is that:
a. -L <n> will not re-run a flakey test if the test passes for the first time. b. -I/-i <n> sets up devices during each iteration and hence slower.
Note -q <n> will override -L <n>.

Also rename _stash_fail_loop_files() to _stash_loop_files() because this function will now be used even when the test doesn't fail (i.e. when ran with -q <n>).

---
 check | 46 ++++++++++++++++++++++++++++++++++++----------
 1 file changed, 36 insertions(+), 10 deletions(-)

index 607d2456..e8ac0ce9 100755
@@ -11,6 +11,7 @@ needsum=true
 try=()
 sum_bad=0
 bad=()
+is_bad_test=false
 notrun=()
 interrupt=true
 diff="diff -u"

@@ -27,6 +28,7 @@ DUMP_OUTPUT=false
 iterations=1
 istop=false
 loop_on_fail=0
+loop_unconditional=0
 exclude_tests=()
 
 # This is a global variable used to pass test failure text to reporting gunk

@@ -83,6 +85,7 @@ check options
     -s section		run only specified section from config file
     -S section		exclude the specified section from the config file
     -L <n>		loop tests <n> times following a failure, measuring aggregate pass/fail metrics
+    -q <n>		loop tests <n> times irrespective of a pass or a failure, measuring aggregate pass/fail metrics
 
 testlist options
     -g group[,group...]	include tests from these groups

@@ -341,7 +344,9 @@ while [ $# -gt 0 ]; do
 	-L)	[[ $2 =~ ^[0-9]+$ ]] || usage
 		loop_on_fail=$2; shift
 		;;
-
+	-q)	[[ $2 =~ ^[0-9]+$ ]] || usage
+		loop_unconditional=$(($2 - 1)); shift
+		;;
 	-*)	usage ;;
 	*)	# not an argument, we've got tests now.
 		have_test_arg=true ;;

@@ -357,6 +362,11 @@ while [ $# -gt 0 ]; do
 	shift
 done
 
+# -q <n> overrides -L <m>
+if [ "$loop_unconditional" -gt 0 ]; then
+	loop_on_fail=0
+fi
+
 # we need common/rc, that also sources common/config. We need to source it
 # after processing args, overlay needs FSTYP set before sourcing common/config
 if ! . ./common/rc; then

@@ -609,7 +619,7 @@ _expunge_test()
 }
 
 # retain files which would be overwritten in subsequent reruns of the same test
-_stash_fail_loop_files() {
+_stash_loop_files() {
 	local seq_prefix="${REPORT_DIR}/${1}"
 	local cp_suffix="$2"
 

@@ -633,10 +643,18 @@ _stash_test_status() {
 	fi
 
 	if ((${#loop_status[*]} > 0)); then
-		# continuing or completing rerun-on-failure loop
-		_stash_fail_loop_files "$test_seq" ".rerun${#loop_status[*]}"
+		# continuing or completing rerun loop
+		_stash_loop_files "$test_seq" ".rerun${#loop_status[*]}"
 		loop_status+=("$test_status")
-		if ((${#loop_status[*]} > loop_on_fail)); then
+
+		# only stash @bad result for initial failure in loop
+		if [[ "$test_status" == "fail" ]] && ! $is_bad_test; then
+			bad+=("$test_seq")
+			is_bad_test=true
+		fi
+
+		if ((loop_on_fail && ${#loop_status[*]} > loop_on_fail)) || \
+		   ((loop_unconditional && ${#loop_status[*]} > loop_unconditional)); then
 			printf "%s aggregate results across %d runs: " \
 				"$test_seq" "${#loop_status[*]}"
 			awk "BEGIN {

@@ -650,23 +668,30 @@ _stash_test_status() {
 				}'
 			echo
 			loop_status=()
+			is_bad_test=false
 		fi
-		return	# only stash @bad result for initial failure in loop
+		return
 	fi
 
 	case "$test_status" in
 	fail)
-		if ((loop_on_fail > 0)); then
-			# initial failure, start rerun-on-failure loop
-			_stash_fail_loop_files "$test_seq" ".rerun0"
+		# re-run if either of the loop argument is set
+		if ((loop_on_fail > 0)) || ((loop_unconditional > 0)); then
+			_stash_loop_files "$test_seq" ".rerun0"
 			loop_status+=("$test_status")
 		fi
 		bad+=("$test_seq")
+		is_bad_test=true
 		;;
 	list|notrun)
 		notrun+=("$test_seq")
 		;;
 	pass|expunge)
+		# re-run if loop_unconditional argument is set
+		if ((loop_unconditional > 0)); then
+			_stash_loop_files "$test_seq" ".rerun0"
+			loop_status+=("$test_status")
+		fi
 		;;
 	*)
 		echo "Unexpected test $test_seq status: $test_status"

@@ -857,7 +882,8 @@ function run_section()
 	seqres="$check"
 	_check_test_fs
 
-	loop_status=()	# track rerun-on-failure state
+	is_bad_test=false
+	loop_status=()	# track loop rerun state
 	local tc_status ix
 	local -a _list=( $list )
 	for ((ix = 0; ix < ${#_list[*]}; !${#loop_status[*]} && ix++)); do

2.34.1

This patch improves the output by:
- Adding pass/fail aggregate results in the stdout for each section's summary. Also adds the location of the xunit xml files in the section's summary.
- Adds section config details into the result's global log file.

e.g.

$ ./check -q 2 -R xunit-quiet   selftest/001

SECTION -- s1
=========================
selftest/001 aggregate results across 2 runs: pass=2 (100.0%)
Ran: selftest/001
Passed all 1 tests
Xunit report: /home/ubuntu/xfstests/results//s1/result.xml

SECTION -- s2
=========================
selftest/001 aggregate results across 2 runs: pass=2 (100.0%)
Ran: selftest/001
Passed all 1 tests
Xunit report: /home/ubuntu/xfstests/results//s2/result.xml

Output in results/check.log file

Kernel version: 6.13.0-rc1-00182-gb8f52214c61a-dirty Wed Jan 8 11:23:57 UTC 2025
SECTION -- s1
=========================

FSTYP         -- ext4
PLATFORM      -- Linux/x86_64 citest-1 6.13.0-rc1-00182-gb8f52214c61a-dirty #5 SMP PREEMPT_DYNAMIC Wed Dec 18 14:03:34 IST 2024
MKFS_OPTIONS  -- -F /dev/loop1

MOUNT_OPTIONS -- -o acl,user_xattr /dev/loop1 /mnt1/scratch selftest/001 aggregate results across 2 runs: pass=2 (100.0%)
Ran: selftest/001
Passed all 1 tests
Xunit report: /home/ubuntu/xfstests/results//s1/result.xml

Kernel version: 6.13.0-rc1-00182-gb8f52214c61a-dirty Wed Jan 8 11:24:02 UTC 2025
SECTION -- s2
=========================

FSTYP         -- ext4
PLATFORM      -- Linux/x86_64 citest-1 6.13.0-rc1-00182-gb8f52214c61a-dirty #5 SMP PREEMPT_DYNAMIC Wed Dec 18 14:03:34 IST 2024
MKFS_OPTIONS  -- -F /dev/loop1

MOUNT_OPTIONS -- -o acl,user_xattr /dev/loop1 /mnt1/scratch selftest/001 aggregate results across 2 runs: pass=2 (100.0%)
Ran: selftest/001
Passed all 1 tests
Xunit report: /home/ubuntu/xfstests/results//s2/result.xml

---
 check | 67 ++++++++++++++++++++++++++++++++++++++++-------------------
 1 file changed, 46 insertions(+), 21 deletions(-)

index e8ac0ce9..f7028836 100755
@@ -468,6 +468,25 @@ if [ -n "$REPORT_GCOV" ]; then
 	_gcov_check_report_gcov
 fi
 
+_print_list()
+{
+	local -n list=$1
+	local item
+	for item in "${list[@]}"; do
+		echo "$item"
+	done
+}
+
+_display_test_configuration()
+{
+	echo "FSTYP         -- `_full_fstyp_details`"
+	echo "PLATFORM      -- `_full_platform_details`"
+	if [ ! -z "$SCRATCH_DEV" ]; then
+		echo "MKFS_OPTIONS  -- `_scratch_mkfs_options`"
+		echo "MOUNT_OPTIONS -- `_scratch_mount_options`"
+	fi
+}
+
 _wrapup()
 {
 	seq="check.$$"

@@ -505,11 +524,19 @@ _wrapup()
 
 		echo "SECTION       -- $section" >>$tmp.summary
 		echo "=========================" >>$tmp.summary
+
+		_global_log "SECTION       -- $section"
+		_global_log "=========================" >>$tmp.summary
+		_global_log "$(_display_test_configuration)"
 		if ((${#try[*]} > 0)); then
+			local test_aggr_stats=$(_print_list loop_test_stats_per_section)
 			if [ $brief_test_summary == "false" ]; then
+				[[ -n "$test_aggr_stats" ]] && echo "$test_aggr_stats" >> \
+					$tmp.summary
 				echo "Ran: ${try[*]}"
 				echo "Ran: ${try[*]}" >>$tmp.summary
 			fi
+			_global_log "$test_aggr_stats"
 			_global_log "Ran: ${try[*]}"
 		fi
 

@@ -539,12 +566,15 @@ _wrapup()
 			_global_log "Passed all ${#try[*]} tests"
 			echo "Passed all ${#try[*]} tests" >>$tmp.summary
 		fi
-		echo "" >>$tmp.summary
 		if $do_report; then
 			_make_section_report "$section" "${#try[*]}" \
 					     "${#bad[*]}" "${#notrun[*]}" \
 					     "$((sect_stop - sect_start))"
+			local out_fn="$REPORT_DIR/result.xml"
+			echo "Xunit report: $out_fn" >> $tmp.summary
+			_global_log "Xunit report: $out_fn"
 		fi
+		echo "" >>$tmp.summary
 
 		# Generate code coverage report
 		if [ -n "$REPORT_GCOV" ]; then

@@ -655,18 +685,19 @@ _stash_test_status() {
 
 		if ((loop_on_fail && ${#loop_status[*]} > loop_on_fail)) || \
 		   ((loop_unconditional && ${#loop_status[*]} > loop_unconditional)); then
-			printf "%s aggregate results across %d runs: " \
-				"$test_seq" "${#loop_status[*]}"
-			awk "BEGIN {
-				n=split(\"${loop_status[*]}\", arr);"'
-				for (i = 1; i <= n; i++)
-					stats[arr[i]]++;
-				for (x in stats)
-					printf("%s=%d (%.1f%%)",
-					       (i-- > n ? x : ", " x),
-					       stats[x], 100 * stats[x] / n);
-				}'
-			echo
+			local test_stats=$(printf "%s aggregate results across %d runs: " \
+				"$test_seq" "${#loop_status[*]}")
+			test_stats+=$(awk "BEGIN {
+					n=split(\"${loop_status[*]}\", arr);"'
+					for (i = 1; i <= n; i++)
+						stats[arr[i]]++;
+					for (x in stats)
+						printf("%s=%d (%.1f%%)",
+							   (i-- > n ? x : ", " x),
+							   stats[x], 100 * stats[x] / n);
+					}')
+			echo "$test_stats"
+			loop_test_stats_per_section+=("$test_stats")
 			loop_status=()
 			is_bad_test=false
 		fi

@@ -839,14 +870,7 @@ function run_section()
 	rm -f $check.full
 
 	[ -f $check.time ] || touch $check.time
-
-	# print out our test configuration
-	echo "FSTYP         -- `_full_fstyp_details`"
-	echo "PLATFORM      -- `_full_platform_details`"
-	if [ ! -z "$SCRATCH_DEV" ]; then
-	  echo "MKFS_OPTIONS  -- `_scratch_mkfs_options`"
-	  echo "MOUNT_OPTIONS -- `_scratch_mount_options`"
-	fi
+	_display_test_configuration
 	echo
 	test -n "$REPORT_GCOV" && _gcov_reset
 	needwrap=true

@@ -884,6 +908,7 @@ function run_section()
 
 	is_bad_test=false
 	loop_status=()	# track loop rerun state
+	loop_test_stats_per_section=() # store loop test statistics per section
 	local tc_status ix
 	local -a _list=( $list )
 	for ((ix = 0; ix < ${#_list[*]}; !${#loop_status[*]} && ix++)); do

2.34.1

This adds support to pick and use any existing FS config from configs/<fstype>/<config>. e.g.

configs/xfs/1k
configs/xfs/4k
configs/ext4/4k
configs/ext4/64k

This should help us maintain and test different fs test configurations from a central place. We also hope that this will be useful for both developers and testers to look into what is being actively maintained and tested by FS Maintainers.

When we will have fsconfigs set, then will be another subdirectory created in results/<section>. For example let's look at the following:

The directory tree structure on running
sudo ./check -q 2 -R xunit-quiet -c xfs/4k,configs/xfs/1k selftest/001 selftest/007

$ tree results/
results/
├── check.full
├── check.log
├── check.time
├── xfs_1
│   └── xfs
│       ├── 1k
│       │   ├── check.log
│       │   ├── check.time
│       │   ├── result.xml
│       │   └── selftest
│       │       ├── 001.full
│       │       ├── 001.full.rerun0
│       │       ├── 001.full.rerun1
│       │       ├── 007.full
│       │       ├── 007.full.rerun0
│       │       ├── 007.full.rerun1
│       │       ├── 007.out.bad
│       │       ├── 007.out.bad.rerun0
│       │       ├── 007.out.bad.rerun1
│       │       └── dmesg_filter
│       └── 4k
│           ├── check.log
│           ├── check.time
│           ├── result.xml
│           └── selftest
│               ├── 001.full
│               ├── 001.full.rerun0
│               ├── 001.full.rerun1
│               ├── 007.full
│               ├── 007.full.rerun0
│               ├── 007.full.rerun1
│               ├── 007.out.bad
│               ├── 007.out.bad.rerun0
│               ├── 007.out.bad.rerun1
│               └── dmesg_filter
└── xfs_2
    └── xfs
        ├── 1k
        │   ├── check.log
        │   ├── check.time
        │   ├── result.xml
        │   └── selftest
        │       ├── 001.full
        │       ├── 001.full.rerun0
        │       ├── 001.full.rerun1
        │       ├── 007.full
        │       ├── 007.full.rerun0
        │       ├── 007.full.rerun1
        │       └── dmesg_filter
        └── 4k
            ├── check.log
            ├── check.time
            ├── result.xml
            └── selftest
                ├── 001.full
                ├── 001.full.rerun0
                ├── 001.full.rerun1
                ├── 007.full
                ├── 007.full.rerun0
                ├── 007.full.rerun1
                ├── 007.out.bad
                ├── 007.out.bad.rerun1
                └── dmesg_filter

12 directories, 51 files

Contents of one of the xml/xunit files (results/xfs_1/xfs/4k/result.xml):
(Please note that FS_CONFIG_OPTION is a new property that is added in this patch)

<?xml version="1.0" encoding="UTF-8"?>
<testsuite
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git/tree/doc/xunit.xsd"

name="xfstests"
failures="1" skipped="0" tests="2" time="6" hostname="li-5d80d4cc-2782-11b2-a85c-bed59fe4c9e5"
start_timestamp="2024-12-11T13:07:29+05:30"

        timestamp="2024-12-11T13:07:35+05:30"

report_timestamp="2024-12-11T13:07:35+05:30"

	<properties>
		<property name="ARCH" value="x86_64"/>
                ...
		<property name="FSTYP" value="xfs"/>
		<property name="FS_CONFIG_OPTION" value="xfs/4k"/>
		<property name="HOST_OPTIONS" value="local.config"/>
                ...
	</properties>
	<testcase classname="xfstests.xfs_1" name="selftest/001" time="0">
	</testcase>
	<testcase classname="xfstests.xfs_1" name="selftest/001" time="1">
	</testcase>
	<testcase classname="xfstests.xfs_1" name="selftest/007" time="1">
		<failure message="- output mismatch (see /home/nirjhar/work/xfstests-dev/results//xfs_1/xfs/4k/selftest/007.out.bad)" type="TestFail" />
	</testcase>
	<testcase classname="xfstests.xfs_1" name="selftest/007" time="1">
		<failure message="- output mismatch (see /home/nirjhar/work/xfstests-dev/results//xfs_1/xfs/4k/selftest/007.out.bad)" type="TestFail" />
	</testcase>

</testsuite>

---
 README.config-sections | 12 +++++++
 check                  | 82 +++++++++++++++++++++++++++++++++---------
 common/config          | 51 ++++++++++++++++++++++++--
 common/report          |  2 +-
 4 files changed, 128 insertions(+), 19 deletions(-)

index a42d9d7b..0511b707 100644
@@ -108,10 +108,22 @@ MKFS_OPTIONS="-q -F -b1024"
 [ext4_nojournal]
 MKFS_OPTIONS="-q -F -b4096 -O ^has_journal"
 
+# we could also do like: FS_CONFIG_OPTION=ext4/1k
+[ext4_1k_bs_fsconfig]
+FS_CONFIG_OPTION=configs/ext4/1k
+
+# we could also do like: FS_CONFIG_OPTION=ext4/nojournal
+[ext4_nojournal_fsconfig]
+FS_CONFIG_OPTION=configs/ext4/nojournal
+
 [xfs_filesystem]
 MKFS_OPTIONS="-f"
 FSTYP=xfs
 
+# we could also do like: FS_CONFIG_OPTION=xfs/4k
+[xfs_4k_bs_fsconfig]
+FS_CONFIG_OPTION=configs/xfs/4k
+
 [ext3_filesystem]
 FSTYP=ext3
 MOUNT_OPTIONS="-o noatime"

index 7128bbeb..8723c510 100644
@@ -12,7 +12,7 @@ REPORT_ENV_LIST=("SECTION" "FSTYP" "PLATFORM" "MKFS_OPTIONS" "MOUNT_OPTIONS" \
 # Variables that are captured in the report /if/ they are set.
 REPORT_ENV_LIST_OPT=("TAPE_DEV" "RMT_TAPE_DEV" "FSSTRESS_AVOID" "FSX_AVOID"
 		     "KCONFIG_PATH" "PERF_CONFIGNAME" "MIN_FSSIZE"
-		     "IDMAPPED_MOUNTS")
+		     "IDMAPPED_MOUNTS" "FS_CONFIG_OPTION")
 
 encode_xml()
 {

2.34.1

This commit adds a couple of simple configuration templates. Developers should add more templates based on the tests they add.

---
 configs/ext4/1k   | 3 +++
 configs/ext4/4k   | 3 +++
 configs/ext4/64k  | 3 +++
 configs/xfs/1k    | 3 +++
 configs/xfs/4k    | 3 +++
 configs/xfs/64k   | 3 +++
 configs/xfs/adv   | 3 +++
 configs/xfs/quota | 3 +++
 8 files changed, 24 insertions(+)
 create mode 100644 configs/ext4/1k
 create mode 100644 configs/ext4/4k
 create mode 100644 configs/ext4/64k
 create mode 100644 configs/xfs/1k
 create mode 100644 configs/xfs/4k
 create mode 100644 configs/xfs/64k
 create mode 100644 configs/xfs/adv
 create mode 100644 configs/xfs/quota

new file mode 100644
index 00000000..6f16fc44
@@ -0,0 +1,3 @@
+FSTYP=ext4
+MKFS_OPTIONS="-F -b 1024"
+MOUNT_OPTIONS="-o block_validity "

new file mode 100644
index 00000000..32f02ea9
@@ -0,0 +1,3 @@
+FSTYP=ext4
+MKFS_OPTIONS="-F -b 4096"
+MOUNT_OPTIONS="-o block_validity "

new file mode 100644
index 00000000..6c50b839
@@ -0,0 +1,3 @@
+FSTYP=ext4
+MKFS_OPTIONS="-F -b 65536"
+MOUNT_OPTIONS="-o block_validity "

new file mode 100644
index 00000000..ab74d3a7
@@ -0,0 +1,3 @@
+FSTYP=xfs
+MKFS_OPTIONS="-b size=1024"
+MOUNT_OPTIONS=" "

new file mode 100644
index 00000000..9a390404
@@ -0,0 +1,3 @@
+FSTYP=xfs
+MKFS_OPTIONS="-b size=4096"
+MOUNT_OPTIONS=" "

new file mode 100644
index 00000000..ddcd7977
@@ -0,0 +1,3 @@
+FSTYP=xfs
+MKFS_OPTIONS="-b size=65536"
+MOUNT_OPTIONS=" "

new file mode 100644
index 00000000..f1b2ec2e
@@ -0,0 +1,3 @@
+FSTYP=xfs
+MKFS_OPTIONS="-m inobtcount=1,bigtime=1,rmapbt=1"
+MOUNT_OPTIONS=" "

new file mode 100644
index 00000000..a7f0c7e7
@@ -0,0 +1,3 @@
+FSTYP=xfs
+MKFS_OPTIONS="-f "
+MOUNT_OPTIONS="-o usrquota,grpquota,prjquota"

2.34.1

This is great! It's something that I've wanted for a while, since at the moment I implement {gce,kvm}-xfstests -C 10 is to run check ten times, and doing something which does the looping inside check instead of outside will be much more efficient.

One other thing that has been on my todo list to update, but which perhaps you might be willing to do while you are doing work in this area (nudge, nudge :-), is an optional mode which interates but which stops once a test fails. This is essentially the reverse of -L, and the reason why it's useful is when trying to bisect a flakey test, which sometimes might only be failing 2-5% of the time, require running a test 30-50 times. But the moment the test fails, we don't need to run the test any more, so this would speed up bisection tests which today I do via:

   gce-xfstests ltm --repo linux.git --bisect-good v6.12 --bisect-bad \
	v6.13-rc1 -C 50 -c ext4/inline_data generic/273

Because of this, I wonder if we should have one option to specify the number of interations, and then a different option which specifies the iteration mode, which might be "unconditional", "until first failure", "only if the test initially fails", etc, instead of separate options for -q, -L, etc.

Thanks,

- Ted

I haven't been using the current in-place configs in kvm-xfstests and gce-xfstests because there are number of things that my setup can support that xfstests native config doesn't support (and becuase my system predates the FS config setup). I don't mind just using my own custom setup, but if we can keep feature parity, perhaps someday I can switch over to xfstests's system. This might also make it easier for people to more easily test using the same setup as the FS maintainers, regardless of which test running infrastructure they are using.

A) A way of specifying the minimum device size needed for the TEST and SCRATCH device. Using a smaller file system size reduces test run time, and if you are paying for cloud test infrastructure, the size of the Google Persistent Disk or Amazon Elastic Block Storage has a direct impact on the cost per test, which in turn impacts how many tests we can afford to run. But for certain test configurations, such as using a larger block size, or using bigalloc, a larger test device size might be needed in order for tests to be able to run correctly.

B) A way of specifying test exclusions, both at a global level, or on a per-fs, or on a per-configuration basis. It should also be possible to specify the kernel version being tested, and so that certain test exclusions might only be done when testing LTS kernels (for example):

#if LINUX_VERSION_CODE < KERNEL_VERSION(6,6,30)
// This test failure is fixed by commit 631426ba1d45q ("mm/madvise:
// make MADV_POPULATE_(READ|WRITE) handle VM_FAULT_RETRY properly"), // which first landed in v6.9, and was backported to 6.6.30 as commit // 631426ba1d45. Unfortunately, it's too involved to backport it and its // dependencies to the 6.1 or earlier LTS kernels
generic/743
#endif

C) A way to run shell functions to do setups for testing things like overlayfs, or nfs (where I may be starting separate VM's for the NFS server, or needing to find the IP address for the NFS server running the appropriate kernel under test, which either been the same as the kernel under test, or which might be some standard server version, such as a RHEL or SLES kernel), as part of the setup and teardown of a particular test configuration.

D) I also have a really nice scheme for specifying a mkfs configuration for testing LTS kernels, since I use the same test appliance for testing upstream and LTS/product kernels, and the latest mkfs.xfs will produce a file system image that isn't supported by a 5.15 LTS kernel. Product teams might also want to run tests using the mkfs configuration for that era kernel, even if a 6.1 kernel can support a file system created using the latest version of xfsprogs or e2fsprogs.

Doing this is a bit non-trivial due to a misfeature of how mkfs.xfs works, but I have a workaround that some might find useful here:

https://github.com/tytso/xfstests-bld/commit/f62433b74146e6ecacdeace306828c6c7510c4a6

Note that this might also be useful for xfstests, where specific xfstests scripts have to handle cases where mkfs.xfs might unexpectedly fail due to an unfortunate combination between the test-specific _scratch_mkfs options, and the global MKFS_OPTIONS. This never happens with ext4, due to how mkfs.ext4 handles conflicting command-line options, but it *is* a problem with mkfs.xfs. If you think merging an 150 line shell script library is easier than trying to get consensus from within the xfs community, here's a technical workaround to what might be charitably described as a disagreement between the xfs architects and the needs of the testing community. :-)

If we're going to have critical mass, maybe this is something that's worth discussing at the upcoming LSF/MM? As I said, I'm happy having this be an exclusive feature in gce-xfstests and kvm-xfstests, but perhaps it would make sense to uplevel some of this feature into xfstests so that more people can take advantage of it, and to make it easier for us to share test configs across test teams and upstream developers?

Cheers,

- Ted

Yup "-q 10" could be used which can give pass/fail metrics of how many times the test passed v/s failed by doing unconditional looping within xfstest's check script itself.

There is -I which stops iterating on encountering a failure. Does that work for this usecase?

I like the idea of iteration mode here. I will let others determine on how easy it is to kill any option in xfstests today and replace it with other. However here is a summary of different iteration and looping options.

We have 3 options in xfstests today:

1. -i <n>              iterate the test list <n> times
2. -I <n>              iterate the test list <n> times, but stops iterating further in case of any test failure
3. -L <n>              loop tests <n> times following a failure, measuring aggregate pass/fail metrics

So we have -i/-I which are iterations and -q/-L which are loops. Looping happens when we can just loop over a particular test <n> times and give pass/fail metrics.
Whereas in case of iterations it goes over from the beginning which will also source different rc/config files and prepares the device etc. (hence it's a bit slow too)

Using -q will be faster over using -i or -I similar to how -L is faster. However -L will only re-run when there is a failure the 1st time.

-q v/s -l: Can we kill current -l option if it is not in use by anyone? I would prefer -l since it looks a short form for looping. I don't see we using -l anymore. But I will let others comment.
-l line mode diff

-ritesh

The -c option check makes sense to me. Is it possible to get this feature implemented first while the -q option is still under discussion?

That said, I have a suggestion for the -c option—
Global config variables should be overridden by file system-specific
config files.

For example, if configs/localhost.config contains:

MKFS_OPTIONS="--sectorsize 64K"

but configs/<fstype>/some_config sets:

MKFS_OPTIONS=""

then the value from configs/<fstype>/some_config should take priority.

I ran some tests with btrfs, and I don’t see this behavior happening yet.

Thanks, Anand

Hi Anand,

Thanks for trying the patches. The design of -c option is still under discussion [1]. But it will be helpful if you could help us understand your reasons for finding -c option useful :)

[1]: https://lore.kernel.org/linux-fsdevel/[email protected]/

I think that was intentional. I guess the reasoning was, we don't want to break use cases for folks who still wanted to use local.config file option.

However, in the new proposed design [2] we are thinking of having 1 large config per filesystem. e.g. configs/btrfs/config.btrfs which will define all of the relevant sections e.g. btrfs_4k, btrfs_64k, ... Then on invokking "make", it will generate a single large fs config file i.e. configs/.all-sections-configs which will club all filesystems section configs together.

Now when someone invokes check script with different -s options, it will first look into local.config file, if local.config not found, then it will look into configs/.all-sections-configs to get the relevant section defines.

This hopefully should address all the other concerns which were raised on the current central fs config design.

[2]: https://lore.kernel.org/linux-fsdevel/[email protected]/

-ritesh

Reason is exactly same as you mentioned and I copied it here:

---
1. Testers and other FS developers can know what is being actively tested and maintained by FS Maintainers.
---

Ok.

I think it’s making things a bit more complicated than needed.

True. We’d probably be better off with one big config file. No need for <fstype> directories under configs. configs/<fstype>.config should work just fine.
I’m not fond of the need to run make; it seems excessive.

The way to run it is simple and works well:

 $ ./check -c configs/btrfs.config
 $ ./check -c configs/btrfs.config -s <section1> ...
 $ ./check -c configs/btrfs.config -S <exclude-section1> ...

These steps already work fine:

 $ cp configs/btrfs.config configs/$(hostname).config
 $ ./check -s <section> ..

(Patch containing configs/btrfs.config is already in the ML).

Also, a --dry-run option to check the config before running would be super helpful.

Thanks, Anand