README: document the new default run mode
PATCH 1/1 README: document the new default run mode
From: Dave Chinner <[email protected]>
Also document the new way to run all tests (i.e. check -g all) and clean up all the stray whitespace in the readme file.
--- README | 38 +++++++++++++++++++++----------------- 1 file changed, 21 insertions(+), 17 deletions(-)
README +21 -17
@@ -22,7 +22,7 @@ _______________________ - create fsgqa test user ("sudo useradd fsgqa") - create fsgqa group ("sudo groupadd fsgqa") - create 123456-fsgqa test user ("sudo useradd 123456-fsgqa") - + ______________________ USING THE FSQA SUITE ______________________
@@ -36,11 +36,10 @@ Preparing system for tests: from http://www.extra.research.philips.com/udf/, then copy the udf_test binary to xfstests/src/. If you wish to disable UDF verification test set the environment variable DISABLE_UDF_TEST to 1. - - + - create one or two partitions to use for testing - one TEST partition - - format as XFS, mount & optionally populate with + - format as XFS, mount & optionally populate with NON-IMPORTANT stuff - one SCRATCH partition (optional) - leave empty and expect this partition to be clobbered
@@ -54,14 +53,14 @@ Preparing system for tests: SCRATCH_DEV should be unused by the tester, and for the legacy support SCRATCH_DEV will be set to the first disk of the SCRATCH_DEV_POOL by xfstests script. - + - setup your environment Quick start: - copy local.config.example to local.config and edit as needed Or: - setenv TEST_DEV "device containing TEST PARTITION" - - setenv TEST_DIR "mount point of TEST PARTITION" - - optionally: + - setenv TEST_DIR "mount point of TEST PARTITION" + - optionally: - setenv SCRATCH_DEV "device containing SCRATCH PARTITION" OR (btrfs only) setenv SCRATCH_DEV_POOL "to 3 or more SCRATCH disks for testing btrfs raid concepts"
@@ -109,18 +108,23 @@ Preparing system for tests: - if testing xfsdump, make sure the tape devices have a tape which can be overwritten. - + - make sure $TEST_DEV is a mounted XFS partition - make sure that $SCRATCH_DEV or $SCRATCH_DEV_POOL contains nothing useful - + Running tests: - cd xfstests - - By default the tests suite will run xfs tests: + - By default the tests suite will run all the tests in the auto group. These + are the tests that are expected to function correctly as regression tests, + and it excludes tests that exercise conditions known to cause machine + failures (i.e. the "dangerous" tests). - ./check '*/001' '*/002' '*/003' - ./check '*/06?' - Groups of tests maybe ran by: ./check -g [group(s)] See the 'group' file for details on groups + - If you want to run all tests regardless of what group they are in + (including dangerous tests), use the "all" group: ./check -g all - To randomize test order: ./check -r [test(s)] - You can explicitly specify NFS/CIFS/OVERLAY, otherwise the filesystem type will be autodetected from $TEST_DEV:
@@ -130,16 +134,16 @@ Running tests: The TEST and SCRATCH partitions should be pre-formatted with another base fs, where the overlay dirs will be created - + The check script tests the return value of each script, and compares the output against the expected output. If the output is not as expected, a diff will be output and an .out.bad file will be produced for the failing test. - + Unexpected console messages, crashes and hangs may be considered to be failures but are not necessarily detected by the QA system. -__________________________ +__________________________ ADDING TO THE FSQA SUITE __________________________
@@ -168,11 +172,11 @@ Test script environment: $TEST_DEV. (b) mkfs a new XFS filesystem on $SCRATCH_DEV, and mount this - on $SCRATCH_MNT. Call the the _require_scratch function + on $SCRATCH_MNT. Call the the _require_scratch function on startup if you require use of the scratch partition. - _require_scratch does some checks on $SCRATCH_DEV & - $SCRATCH_MNT and makes sure they're unmounted. You should - cleanup when your test is done, and in particular unmount + _require_scratch does some checks on $SCRATCH_DEV & + $SCRATCH_MNT and makes sure they're unmounted. You should + cleanup when your test is done, and in particular unmount $SCRATCH_MNT. Tests can make use of $SCRATCH_LOGDEV and $SCRATCH_RTDEV for testing external log and realtime volumes - however,
Signature · 1 line
2.17.0
Quoted 6 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Also document the new way to run all tests (i.e. check -g all) and clean up all the stray whitespace in the readme file.
Signed-off-by Dave Chinner <[email protected]>
[...]
README Preparing system for tests: · line 121
Quoted 2 lines
README +2
-0+ and it excludes tests that exercise conditions known to cause machine + failures (i.e. the "dangerous" tests).
That would have been nice.. if it was implemented...
I counted 22 dangerous tests, 20 of which are in auto group, unlike the 87 dangerous_* tests, none of which are in auto group.
If running ./check would exclude 'dangerous' tests by default, should it also exclude dangerous_* tests? Or should we just remove the 20 'dangerous' tests from auto group and try not to add new auto&dangerous tests in the future?
Thanks,
Amir.
Quoted 7 lines · Amir Goldstein
Quoted 6 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Also document the new way to run all tests (i.e. check -g all) and clean up all the stray whitespace in the readme file.
Signed-off-by Dave Chinner <[email protected]>
Quoted 8 lines
[...]
README Preparing system for tests: · line 121
Quoted 2 lines
README +2
-0+ and it excludes tests that exercise conditions known to cause machine + failures (i.e. the "dangerous" tests).That would have been nice.. if it was implemented...
I counted 22 dangerous tests, 20 of which are in auto group, unlike the 87 dangerous_* tests, none of which are in auto group.
The 'dangerous' group and 'auto' group are not mutually exclusive when it was introduced, see commit 3f28d55c3954 ("add freeze and dangerous groups"). But from previous discussions[1][2], they should be mutually exclusive.
[1] https://www.spinics.net/lists/fstests/msg08652.html
[2] https://www.spinics.net/lists/fstests/msg08663.html
Quoted 5 lines
If running ./check would exclude 'dangerous' tests by default, should it also exclude dangerous_* tests? Or should we just remove the 20 'dangerous' tests from auto group and try not to add new auto&dangerous tests in the future?
I agreed, I think we should just remove the 20 'dangerous' tests from auto group (AFAICT, they are not dangerous anymore). And I'll avoid adding new tests that are in both dangerous & auto group.
Thanks,
Eryu
Quoted 5 lines · Eryu Guan
t
The 'dangerous' group and 'auto' group are not mutually exclusive when it was introduced, see commit3f28d55c3954("add freeze and dangerouss groups"). But from previous discussions[1][2], they should be mutually exclusive.
What might be nice is if there was some way to declare that a test is dangerous given a particular kernel version (or some combination of kernel version and file system type). Right now I have to manually exclude tests when testing stable kernels, e.g.:
gce-xfstests full --kernel gs://gce-xfstests/bzImage-3.18 -X generic/269 -X ext4/022
It's not a huge deal, and it's been on my todo list to add into the {gce,kvm,android}-xfstests framework, but perhaps it belongs in the core xfstests framework.
- Ted
Quoted 7 lines · Theodore Y. Ts'o
Quoted 5 lines · Eryu Guan
t
The 'dangerous' group and 'auto' group are not mutually exclusive when it was introduced, see commit3f28d55c3954("add freeze and dangerouss groups"). But from previous discussions[1][2], they should be mutually exclusive.
Quoted 6 lines
What might be nice is if there was some way to declare that a test is dangerous given a particular kernel version (or some combination of kernel version and file system type). Right now I have to manually exclude tests when testing stable kernels, e.g.:
gce-xfstests full --kernel gs://gce-xfstests/bzImage-3.18 -X generic/269 -X ext4/022
That's what expunge files are for.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 10 lines · Eryu Guan
Quoted 5 lines · Amir Goldstein
On Sat, May 5, 2018 at 3:19 AM, Dave Chinner <[email protected]> wrote:
If running ./check would exclude 'dangerous' tests by default, should it also exclude dangerous_* tests? Or should we just remove the 20 'dangerous' tests from auto group and try not to add new auto&dangerous tests in the future?I agreed, I think we should just remove the 20 'dangerous' tests from auto group (AFAICT, they are not dangerous anymore). And I'll avoid adding new tests that are in both dangerous & auto group.
If the test is not dangerous anymore, then remove the dangerous tag from the test.
FWIW:
$ git grep auto tests/*group |grep dangerous tests/btrfs/group:150 auto quick dangerous tests/ext4/group:022 auto quick attr dangerous tests/ext4/group:024 auto quick encrypt dangerous tests/ext4/group:025 auto quick fuzzers dangerous tests/generic/group:068 other auto freeze dangerous stress tests/generic/group:280 auto quota freeze dangerous tests/generic/group:390 auto freeze stress dangerous tests/generic/group:406 auto quick dangerous tests/generic/group:421 auto quick encrypt dangerous tests/generic/group:446 auto quick rw dangerous tests/generic/group:459 auto dangerous tests/generic/group:463 auto quick clone dangerous tests/xfs/group:115 auto quick fuzzers dangerous tests/xfs/group:118 auto quick fsr dangerous tests/xfs/group:119 log v2log auto freeze dangerous tests/xfs/group:294 auto dir metadata dangerous tests/xfs/group:306 auto dangerous quick punch tests/xfs/group:311 auto dangerous quick tests/xfs/group:431 auto quick dangerous tests/xfs/group:438 auto quick quota dangerous
$ sudo ./run_check.sh "" "" "-b -s xfs generic/068 generic/280 generic/390 generic/406 generic/421 generic/446 generic/459 generic/463 xfs/115 xfs/118 xfs/119 xfs/294 xfs/306 xfs/311 xfs/431 xfs/438" [....] SECTION -- xfs FSTYP -- xfs (debug) PLATFORM -- Linux/x86_64 test4 4.17.0-rc3-dgc+ MKFS_OPTIONS -- -f -m rmapbt=1,reflink=1 -i sparse=1 /dev/pmem1 MOUNT_OPTIONS -- /dev/pmem1 /mnt/scratch
generic/068 49s ... 45s
generic/280 2s ... 3s
generic/390 25s ... 19s
generic/406 2s ... 2s
generic/421 [not run] No encryption support for xfs
generic/446 13s ... 13s
generic/459 15s ... 14s
generic/463 5s ... 1s
xfs/115 1s ... [not run] test requires XFS bug_on_assert to be off, turn it off to run the test xfs/118 15s ... 15s
xfs/119 11s ... 12s
xfs/294 58s ... 58s
xfs/306 39s ... 41s
xfs/311 23s ... 23s
xfs/431 1s ... 1s
xfs/438 5s ... 5s
Passed all 16 tests
I can't speak for the btrfs and ext4 tests or generic/421 (encryption test), but none of the other generic or xfs tests marked dangerous appear to be dangerous on a 4.17-rc3 kernel....
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]