Annonate fstests with possible reasons for failure
v2 cover Annonate fstests with possible reasons for failure
Zorro,
This series adresses your review comments on v1. Specifically, I dropped the controversial patch #3 and the SKIP_KNOWN_ISSUES run option.
To clarify, multiple annotations could be used in each test, for example:
_fixed_by_kernel_commit 09889695864 xfs: foo _fixed_by_kernel_commit 46464565465 ext4: bar _fixed_by_git_commit e2fsprogs 62464626 e2fs: baz _fixed_in_version xfsprogs v5.15
On test failure, all the hints will be printed to output as *possible* reason for failure.
These annotation and printed hints are inspired by a similar LTP feature.
These annotation could be used by testers to auto-generate expunge list for testing stable kernels from the information in _fixed_by_kernel_commit and _fixed_in_kernel_version. This trivial task is left is an excercise.
Thanks,
Amir.
Changes since v1:
- Rename s/_known_issue_before_kernel/_fixed_in_kernel_version (zlang)
- Add generic helpers for non-kernel fixes (zlang)
- Drop _known_issue_on_fs (zlang)
- Clarify multiple hints are allowed (tytso)
Amir Goldstein (2):
common: support black listing fs in _supported_fs()
common: print hints for reasons of test failures
check | 10 ++++++++- common/preamble | 2 +- common/rc | 57 +++++++++++++++++++++++++++++++++++++++-------- tests/generic/500 | 3 +-- tests/generic/631 | 3 +-- tests/generic/679 | 3 +-- tests/overlay/009 | 3 +++ tests/overlay/010 | 3 +++ tests/overlay/014 | 2 ++ tests/overlay/016 | 4 +++- tests/overlay/017 | 4 +++- tests/overlay/018 | 4 +++- tests/overlay/020 | 3 +++ tests/overlay/022 | 2 ++ tests/overlay/029 | 2 ++ tests/overlay/038 | 2 ++ tests/overlay/041 | 2 ++ tests/overlay/042 | 3 +++ tests/overlay/043 | 4 +++- tests/overlay/044 | 4 +++- tests/overlay/054 | 3 +++ tests/overlay/055 | 3 +++ tests/overlay/063 | 3 +++ tests/overlay/065 | 4 ++++ tests/overlay/067 | 3 +++ tests/overlay/070 | 4 +++- tests/overlay/071 | 4 +++- tests/overlay/072 | 2 ++ tests/overlay/074 | 5 +++++ tests/overlay/077 | 5 +++++ tests/overlay/078 | 1 + 31 files changed, 133 insertions(+), 24 deletions(-)
Signature · 1 line
2.25.1
PATCH v2 1/2 common: support black listing fs in _supported_fs()
For example:_supported_fs ^xfs
There is no need to specify "generic" when using a block list "all other fs are supported" is implied.
Converted some generic tests that open code this condition without emitting a meaningful reason.
More generic test like generic/186,187 could use a block list, but were not converted because they emit some meaningful reason:
_notrun "Can't fragment free space on btrfs."
--- common/rc | 30 ++++++++++++++++++++---------- tests/generic/500 | 3 +-- tests/generic/631 | 3 +-- tests/generic/679 | 3 +-- 4 files changed, 23 insertions(+), 16 deletions(-)
common/rc +20 -10
@@ -1580,19 +1580,29 @@ _fail() # tests whether $FSTYP is one of the supported filesystems for a test # -_supported_fs() +_check_supported_fs() { - local f + local res=1 + local f - for f - do - if [ "$f" = "$FSTYP" -o "$f" = "generic" ] - then - return - fi - done + for f; do + # ^FS means black listed fs + if [ "$f" = "^$FSTYP" ]; then + return 1 + elif [ "$f" = "generic" ] || [[ "$f" == "^"* ]]; then + # ^FS implies "generic ^FS" + res=0 + elif [ "$f" = "$FSTYP" ]; then + return 0 + fi + done + return $res +} - _notrun "not suitable for this filesystem type: $FSTYP" +_supported_fs() +{ + _check_supported_fs $* || \ + _notrun "not suitable for this filesystem type: $FSTYP" } # check if a FS on a device is mounted
tests/generic/500 +1 -2
@@ -35,7 +35,6 @@ _cleanup() . ./common/dmthin # real QA test starts here -_supported_fs generic _require_scratch_nocheck _require_dm_target thin-pool
@@ -43,7 +42,7 @@ _require_dm_target thin-pool # and since we've filled the thinp device it'll return EIO, which will make # btrfs flip read only, making it fail this test when it just won't work right # for us in the first place. -test $FSTYP == "btrfs" && _notrun "btrfs doesn't work that way lol" +_supported_fs ^btrfs # Require underlying device support discard _scratch_mkfs >>$seqres.full 2>&1
tests/generic/631 +1 -2
@@ -36,10 +36,9 @@ _cleanup() . ./common/attr # real QA test starts here -_supported_fs generic _require_scratch _require_attrs trusted -test "$FSTYP" = "overlay" && _notrun "Test does not apply to overlayfs." +_supported_fs ^overlay _require_extra_fs overlay _scratch_mkfs >> $seqres.full
tests/generic/679 +1 -2
@@ -17,7 +17,6 @@ _begin_fstest auto quick prealloc # real QA test starts here -_supported_fs generic _require_scratch _require_xfs_io_command "falloc" _require_xfs_io_command "fiemap"
@@ -26,7 +25,7 @@ _require_xfs_io_command "fiemap" # # https://lore.kernel.org/linux-btrfs/20220315164011.GF8241@magnolia/ # -[ $FSTYP == "xfs" ] && _notrun "test not valid for xfs" +_supported_fs ^xfs rm -f $seqres.full
Signature · 1 line
2.25.1
PATCH v2 2/2 common: print hints for reasons of test failures
In reply to Amir GoldsteinIntroduce helpers _fixed_by_{kernel,git}_commit() and _fixed_in_{kernel_,}version() that can be used to hint testers why a test might be failing and aid in auto-generating of expunge lists for downstream kernel testing.
A test may be annotated with multiple hints, for example:
_fixed_by_kernel_commit 09889695864 xfs: foo_fixed_by_kernel_commit 46464565465 ext4: bar_fixed_in_version xfsprogs v5.15
Annotate fix kernel commits for some overlayfs tests. Annotate fix kernel version for some overlayfs tests testing for legacy behavior whose fixes are not likely to be backported to stable kernels.
This is modeled after LTP's 'make filter-known-fails' and print_failure_hints() using struct tst_tag annotations.
--- check | 10 +++++++++- common/preamble | 2 +- common/rc | 29 +++++++++++++++++++++++++++++ tests/overlay/009 | 3 +++ tests/overlay/010 | 3 +++ tests/overlay/014 | 2 ++ tests/overlay/016 | 4 +++- tests/overlay/017 | 4 +++- tests/overlay/018 | 4 +++- tests/overlay/020 | 3 +++ tests/overlay/022 | 2 ++ tests/overlay/029 | 2 ++ tests/overlay/038 | 2 ++ tests/overlay/041 | 2 ++ tests/overlay/042 | 3 +++ tests/overlay/043 | 4 +++- tests/overlay/044 | 4 +++- tests/overlay/054 | 3 +++ tests/overlay/055 | 3 +++ tests/overlay/063 | 3 +++ tests/overlay/065 | 4 ++++ tests/overlay/067 | 3 +++ tests/overlay/070 | 4 +++- tests/overlay/071 | 4 +++- tests/overlay/072 | 2 ++ tests/overlay/074 | 5 +++++ tests/overlay/077 | 5 +++++ tests/overlay/078 | 1 + 28 files changed, 111 insertions(+), 9 deletions(-)
check +9 -1
@@ -808,7 +808,7 @@ function run_section() fi # really going to try and run this one - rm -f $seqres.out.bad + rm -f $seqres.out.bad $seqres.hints # check if we really should run it _expunge_test $seqnum
@@ -942,6 +942,14 @@ function run_section() fi; } | sed -e 's/^\(.\)/ \1/' err=true fi + if [ -f $seqres.hints ]; then + if $err; then + echo + cat $seqres.hints + else + rm -f $seqres.hints + fi + fi done # make sure we record the status of the last test we ran.
common/preamble +1 -1
@@ -79,6 +79,6 @@ _begin_fstest() . ./common/rc # remove previous $seqres.full before test - rm -f $seqres.full + rm -f $seqres.full $seqres.hints }
common/rc +29 -0
@@ -1605,6 +1605,35 @@ _supported_fs() _notrun "not suitable for this filesystem type: $FSTYP" } +_fixed_in_version() +{ + local pkg=$1 + local ver=$2 + + echo "HINT: You _MAY_ be hit by a known issue on $pkg version < $ver." >> $seqres.hints + echo >> $seqres.hints +} + +_fixed_in_kernel_version() +{ + _fixed_in_version kernel $* +} + +_fixed_by_git_commit() +{ + local pkg=$1 + shift + + echo "HINT: You _MAY_ be missing $pkg fix:" >> $seqres.hints + echo " $*" >> $seqres.hints + echo >> $seqres.hints +} + +_fixed_by_kernel_commit() +{ + _fixed_by_git_commit kernel $* +} + # check if a FS on a device is mounted # if so, verify that it is mounted on mount point # if fstype is given as argument, verify that it is also
tests/overlay/009 +3 -0
@@ -17,6 +17,9 @@ _begin_fstest auto quick # real QA test starts here _supported_fs overlay +_fixed_by_kernel_commit a4859d75944a \ + "ovl: fix dentry leak for default_permissions" + _require_scratch # Remove all files from previous tests
tests/overlay/010 +3 -0
@@ -17,6 +17,9 @@ _begin_fstest auto quick whiteout # real QA test starts here _supported_fs overlay +_fixed_by_kernel_commit 84889d493356 \ + "ovl: check dentry positiveness in ovl_cleanup_whiteouts()" + # Use non-default scratch underlying overlay dirs, we need to check # them explicity after test. _require_scratch_nocheck
tests/overlay/014 +2 -0
@@ -21,6 +21,8 @@ _begin_fstest auto quick copyup # real QA test starts here _supported_fs overlay +_fixed_by_kernel_commit 0956254a2d5b "ovl: don't copy up opaqueness" + # Use non-default scratch underlying overlay dirs, we need to check # them explicity after test. _require_scratch_nocheck
tests/overlay/016 +3 -1
@@ -6,7 +6,7 @@ # # Test ro/rw fd data inconsistecies # -# This simple test demonstrates a known issue with overlayfs: +# This simple test demonstrates an issue with overlayfs on kernel < v4.19: # - process A opens file F for read # - process B writes new data to file F # - process A reads old data from file F
@@ -19,6 +19,8 @@ _begin_fstest auto quick copyup # real QA test starts here _supported_fs overlay +_fixed_in_kernel_version "v4.19" + _require_scratch _require_xfs_io_command "open"
tests/overlay/017 +3 -1
@@ -6,7 +6,7 @@ # # Test constant inode numbers # -# This simple test demonstrates a known issue with overlayfs: +# This simple test demonstrates an issue with overlayfs on kernel < v4.14: # - stat file A shows inode number X # - modify A to trigger copy up # - stat file A shows inode number Y != X
@@ -23,6 +23,8 @@ _begin_fstest auto quick copyup redirect # real QA test starts here _supported_fs overlay +_fixed_in_kernel_version "v4.14" + _require_scratch _require_test_program "af_unix" _require_test_program "t_dir_type"
tests/overlay/018 +3 -1
@@ -6,7 +6,7 @@ # # Test hardlink breakage # -# This simple test demonstrates a known issue with overlayfs: +# This simple test demonstrates an issue with overlayfs on kernel < v4.13: # - file A and B are hardlinked in lower # - modify A to trigger copy up # - file A is no longer a hardlink of file B
@@ -19,6 +19,8 @@ _begin_fstest auto quick copyup hardlink # real QA test starts here _supported_fs overlay +_fixed_in_kernel_version "v4.13" + _require_scratch _require_scratch_feature index _require_test_program "t_dir_type"
tests/overlay/020 +3 -0
@@ -23,6 +23,9 @@ require_unshare() { # Modify as appropriate. _supported_fs overlay +_fixed_by_kernel_commit 3fe6e52f0626 \ + "ovl: override creds with the ones from the superblock mounter" + _require_scratch require_unshare -m -p -U
tests/overlay/022 +2 -0
@@ -29,6 +29,8 @@ _cleanup() # Modify as appropriate. _supported_fs overlay +_fixed_by_kernel_commit 76bc8e2843b6 "ovl: disallow overlayfs as upperdir" + _require_scratch # Remove all files from previous tests
tests/overlay/029 +2 -0
@@ -34,6 +34,8 @@ _cleanup() # Modify as appropriate. _supported_fs overlay +_fixed_by_kernel_commit c4fcfc1619ea "ovl: fix d_real() for stacked fs" + _require_scratch # Remove all files from previous tests
tests/overlay/038 +2 -0
@@ -15,6 +15,8 @@ _begin_fstest auto quick copyup # real QA test starts here _supported_fs overlay +_fixed_in_kernel_version "v4.14" + # Use non-default scratch underlying overlay dirs, we need to check # them explicity after test. _require_scratch_nocheck
tests/overlay/041 +2 -0
@@ -17,6 +17,8 @@ _begin_fstest auto quick copyup nonsamefs # real QA test starts here _supported_fs overlay +_fixed_in_kernel_version "v4.17" + # Use non-default scratch underlying overlay dirs, we need to check # them explicity after test. _require_scratch_nocheck
tests/overlay/042 +3 -0
@@ -26,6 +26,9 @@ _begin_fstest auto quick copyup hardlink # real QA test starts here _supported_fs overlay +_fixed_by_kernel_commit 6eaf011144af \ + "ovl: fix EIO from lookup of non-indexed upper" + _require_scratch # Without overlay index feature hardlinks are broken on copy up _require_scratch_feature index
tests/overlay/043 +3 -1
@@ -8,7 +8,7 @@ # This is a variant of overlay/017 to test constant st_ino numbers for # non-samefs setup. # -# This simple test demonstrates a known issue with overlayfs: +# This simple test demonstrates an issue with overlayfs on kernel < v4.17: # - stat file A shows inode number X # - modify A to trigger copy up # - stat file A shows inode number Y != X
@@ -25,6 +25,8 @@ _begin_fstest auto quick copyup nonsamefs # real QA test starts here _supported_fs overlay +_fixed_in_kernel_version "v4.17" + # Use non-default scratch underlying overlay dirs, we need to check # them explicity after test. _require_scratch_nocheck
tests/overlay/044 +3 -1
@@ -7,7 +7,7 @@ # Test hardlink breakage on non-samefs setup # This is a variant of overlay/018 to test. # -# This simple test demonstrates a known issue with overlayfs: +# This simple test demonstrates an issue with overlayfs on kernel < v4.17: # - file A and B are hardlinked in lower # - modify A to trigger copy up # - file A is no longer a hardlink of file B
@@ -20,6 +20,8 @@ _begin_fstest auto quick copyup hardlink nonsamefs # real QA test starts here _supported_fs overlay +_fixed_in_kernel_version "v4.17" + # Use non-default scratch underlying overlay dirs, we need to check # them explicity after test. _require_scratch_nocheck
tests/overlay/054 +3 -0
@@ -37,6 +37,9 @@ _begin_fstest auto quick copyup redirect exportfs # real QA test starts here _supported_fs overlay +_fixed_by_kernel_commit 2ca3c148a062 \ + "ovl: check lower ancestry on encode of lower dir file handle" + _require_scratch _require_test_program "open_by_handle" # We need to require all features together, because nfs_export cannot
tests/overlay/055 +3 -0
@@ -46,6 +46,9 @@ _cleanup() # real QA test starts here _supported_fs overlay +_fixed_by_kernel_commit 2ca3c148a062 \ + "ovl: check lower ancestry on encode of lower dir file handle" + _require_test _require_test_program "open_by_handle" # Use non-default scratch underlying overlay dirs, we need to check
tests/overlay/063 +3 -0
@@ -17,6 +17,9 @@ _begin_fstest auto quick whiteout # real QA test starts here _supported_fs overlay +_fixed_by_kernel_commit 5e1275808630 \ + "ovl: check whiteout in ovl_create_over_whiteout()" + _require_scratch # Remove all files from previous tests
tests/overlay/065 +4 -0
@@ -38,6 +38,10 @@ _cleanup() # real QA test starts here _supported_fs overlay +_fixed_in_kernel_version "v5.2" +_fixed_by_kernel_commit 0be0bfd2de9d \ + "ovl: fix regression caused by overlapping layers detection" + # Use non-default scratch underlying overlay dirs, we need to check # them explicity after test. _require_scratch_nocheck
tests/overlay/067 +3 -0
@@ -20,6 +20,9 @@ _begin_fstest auto quick copyup nonsamefs # real QA test starts here _supported_fs overlay +_fixed_by_kernel_commit 9c6d8f13e9da \ + "ovl: fix corner case of non-unique st_dev;st_ino" + # Use non-default scratch underlying overlay dirs, we need to check # them explicity after test. _require_scratch_nocheck
tests/overlay/070 +3 -1
@@ -8,7 +8,7 @@ # nested overlay setup, where all layers of both overlays are on the # same fs. # -# This simple test demonstrates a known issue with overlayfs: +# This simple test demonstrates an issue with overlayfs on kernel < v4.17: # - stat file A shows inode number X # - modify A to trigger copy up # - stat file A shows inode number Y != X
@@ -35,6 +35,8 @@ _cleanup() # real QA test starts here _supported_fs overlay +_fixed_in_kernel_version "v4.17" + _require_scratch_nocheck _require_test_program "af_unix" _require_test_program "t_dir_type"
tests/overlay/071 +3 -1
@@ -7,7 +7,7 @@ # This is a variant of overlay/017 to test constant st_ino numbers for # nested overlay setup, where lower overlay layers are not on the same fs. # -# This simple test demonstrates a known issue with overlayfs: +# This simple test demonstrates an issue with overlayfs on kernel < v4.17: # - stat file A shows inode number X # - modify A to trigger copy up # - stat file A shows inode number Y != X
@@ -38,6 +38,8 @@ _cleanup() # real QA test starts here _supported_fs overlay +_fixed_in_kernel_version "v4.17" + _require_test _require_scratch_nocheck _require_test_program "af_unix"
tests/overlay/072 +2 -0
@@ -28,6 +28,8 @@ _begin_fstest auto quick copyup hardlink # real QA test starts here _supported_fs overlay +_fixed_by_kernel_commit 83552eacdfc0 "ovl: fix WARN_ON nlink drop to zero" + _require_scratch upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER
tests/overlay/074 +5 -0
@@ -22,6 +22,11 @@ _begin_fstest auto quick exportfs dangerous # real QA test starts here _supported_fs overlay +_fixed_by_kernel_commit 144da23beab8 \ + "ovl: return required buffer size for file handles" +_fixed_by_kernel_commit 9aafc1b01873 \ + "ovl: potential crash in ovl_fid_to_fh()" + _require_scratch _require_test_program "open_by_handle" # We need to require all features together, because nfs_export cannot
tests/overlay/077 +5 -0
@@ -18,6 +18,11 @@ _begin_fstest auto quick dir # real QA test starts here _supported_fs overlay +_fixed_by_kernel_commit 65cd913ec9d9 \ + "ovl: invalidate readdir cache on changes to dir with origin" +_fixed_by_kernel_commit 9011c2791e63 \ + "ovl: skip stale entries in merge dir cache iteration" + _require_scratch_nocheck # Use small getdents bufsize to fit less than 10 entries
tests/overlay/078 +1 -0
@@ -33,6 +33,7 @@ _cleanup() # real QA test starts here _supported_fs overlay +_fixed_in_kernel_version "v5.13" _require_command "$LSATTR_PROG" lasttr _require_command "$CHATTR_PROG" chattr
Signature · 1 line
2.25.1
Quoted 13 lines · Amir Goldstein
Zorro,
This series adresses your review comments on v1. Specifically, I dropped the controversial patch #3 and the
SKIP_KNOWN_ISSUESrun option.To clarify, multiple annotations could be used in each test, for example:
_fixed_by_kernel_commit09889695864 xfs: foo_fixed_by_kernel_commit46464565465 ext4: bar_fixed_by_git_commite2fsprogs 62464626 e2fs: baz_fixed_in_versionxfsprogs v5.15
Quoted 8 lines
On test failure, all the hints will be printed to output as *possible* reason for failure.
These annotation and printed hints are inspired by a similar LTP feature.
These annotation could be used by testers to auto-generate expunge list for testing stable kernels from the information in
_fixed_by_kernel_commitand_fixed_in_kernel_version. This trivial task is left is an excercise.
I've reviewed V1 patchset, and keep V2 2 weeks here to get more review points. if there's not objection or suggestion, I'll merge this change in next fstests release.
Thanks,
Zorro
Quoted 50 lines
Thanks,
Amir.Changes since v1:
- Rename s/_known_issue_before_kernel/_fixed_in_kernel_version (zlang)
- Add generic helpers for non-kernel fixes (zlang)
- Drop_known_issue_on_fs(zlang)
- Clarify multiple hints are allowed (tytso)Amir Goldstein (2):
common: support black listing fs in_supported_fs()
common: print hints for reasons of test failurescheck | 10 ++++++++- common/preamble | 2 +- common/rc | 57 +++++++++++++++++++++++++++++++++++++++-------- tests/generic/500 | 3 +-- tests/generic/631 | 3 +-- tests/generic/679 | 3 +-- tests/overlay/009 | 3 +++ tests/overlay/010 | 3 +++ tests/overlay/014 | 2 ++ tests/overlay/016 | 4 +++- tests/overlay/017 | 4 +++- tests/overlay/018 | 4 +++- tests/overlay/020 | 3 +++ tests/overlay/022 | 2 ++ tests/overlay/029 | 2 ++ tests/overlay/038 | 2 ++ tests/overlay/041 | 2 ++ tests/overlay/042 | 3 +++ tests/overlay/043 | 4 +++- tests/overlay/044 | 4 +++- tests/overlay/054 | 3 +++ tests/overlay/055 | 3 +++ tests/overlay/063 | 3 +++ tests/overlay/065 | 4 ++++ tests/overlay/067 | 3 +++ tests/overlay/070 | 4 +++- tests/overlay/071 | 4 +++- tests/overlay/072 | 2 ++ tests/overlay/074 | 5 +++++ tests/overlay/077 | 5 +++++ tests/overlay/078 | 1 + 31 files changed, 133 insertions(+), 24 deletions(-)Signature · 1 line
2.25.1