Skip to content

check-parallel: Running tests without check

Dave Chinner fstests 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest28 patches 51 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/28fstests: remove support for non-numeric test namescheck common/test_names new+1 -463 replies
2/28_scratch_mkfs_sized: obey USE_EXTERNAL for XFS filesystemscommon/rc+1 -11 reply
3/28fstests: move test exit functions to common/exitcheck common/config common/exit +2 more+52 -370 replies
4/28check-parallel: report how many tests were _notruncheck-parallel+4 -13 replies
5/28check: factor out test list building codecheck common/report common/test_list+314 -2533 replies
6/28check-parallel: use common group list parsing codecheck check-parallel common/test_list+143 -263 replies
7/28check-parallel: adjust concurrency according to CPU countcheck-parallel+8 -13 replies
8/28check-parallel: add logwrite device supportcheck-parallel common/dmlogwrites+7 -23 replies
9/28check-parallel: allow FSTYP selection from the CLIcheck-parallel+37 -73 replies
10/28check-parallel: use PID namespaces for runner process isolationcheck-parallel+7 -33 replies
11/28check-parallel: initial support for specifying device sizescheck-parallel+30 -52 replies
12/28config: move config section code to it's own filecommon/config common/config-sections+392 -3802 replies
13/28check-parallel: introduce config file supportcheck-parallel common/config-sections+77 -152 replies
14/28fstests: further separate sourcing common/rc and common/config from initialisationcheck common/config common/preamble +1 more+240 -2401 reply
15/28check-parallel: de-batch test executioncheck-parallel+43 -321 reply
16/28check-parallel: run sections directlycheck-parallel common/config-sections+84 -371 reply
17/28check-parallel: rebuild test list when FSTYP changescheck-parallel common/test_list+53 -361 reply
18/28check-parallel: create a "results-latest" symlinkcheck-parallel+6 -11 reply
19/28check: factor test runningcheck common/test_exec+398 -3501 reply
20/28check-parallel: run tests directly without using checkcheck check-parallel common/config +5 more+191 -741 reply
21/28generic/531: limit max files per CPUtests/generic/531+4 -41 reply
22/28fsync-tester.c: use syncfs() rather than sync()src/fsync-tester.c+1 -11 reply
23/28open-by-handle.c: use syncfs() rather than sync()src/open_by_handle.c+3 -33 replies
24/28open-by-handle.c: use syncfs() rather than sync()src/bulkstat_unlink_test.c+6 -63 replies
25/28bulkstat_unlink_test_modified.c: remove unused test code.gitignore src/Makefile src/bulkstat_unlink_test_modified.c+1 -1951 reply
26/28stale-handle.c: use syncfs() rather than sync()src/stale_handle.c+11 -42 replies
27/28scaleread: remove dead test codesrc/Makefile src/scaleread.c src/scaleread.sh+1 -2891 reply
28/28xfs/259: no need to call synctests/xfs/259+0 -11 reply

Reviews trailers per patch

16 of 28 patches have a Reviewed-by or Acked-by.

PatchNirjhar Roy (IBM)
1/28R-b
2/28R-b
3/28
4/28R-b
5/28
6/28
7/28R-b
8/28
9/28R-b
10/28
11/28
12/28R-b
13/28
14/28
15/28R-b
16/28
17/28
18/28R-b
19/28
20/28
21/28R-b
22/28R-b
23/28R-b
24/28R-b
25/28R-b
26/28R-b
27/28R-b
28/28R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH 00/28] check-parallel: Running tests without check

Hi folks,

This set of patches is intended to move check-parallel away from using check to execute tests.

To do this, we need to share a bunch of check code between check and check-parallel. This is mainly the code that parses and builds the test list, the config section parsing and iteration, and the test execution loop itself.

To do this, test list parsing and building is factored out of check into common/test_list. check is converted to use the test list functions at the same time, and then check-parallel is converted to use the factored code to directly build it's test list rather than the open coded grep hack it currently uses. This allows check-parallel CLI to use the same group selection interface as check.

The next change is to factor the config section parsing out of common/config and move it to common/config-section. This allows check-parallel to parse and implement section iteration itself without needing to run all the environment setup code in common/config. This also allows check-parallel to implement it's own config section to define the device sizes that it will use independently of the sections that run tests.

Next, we change check-parallel to use a global test list that runner scripts can safely dequeue the next test to run. This uses a test list file and a lock file to serialise access to the file. Hence a runner can dequeue the next test and remove it from the test list file without racing with any other runner trying to dequeue the next test to run. This means we get rid of the static per-runner test lists that result in many runners finishing and going idle while other test runners have pending tests still to run. i.e. all test runners keep executing tests until there are no tests left in the queue, hence keeping utilisation as high as possible across the test run.

Then we factor the test execution loop out of check and put it in common/test_exec. This abstraction makes the results array part of the test execution, as well using a context defined helper "_run_seq" to do the actual execution of the test. This allows the test execution loop to be completely generic, whilst allowing check and check-parallel to do completely independent things with individual test execution and overall results reporting.

Finally, we change check-parallel to run tests directly via the common/test_exec infrastructure rather than executing them via check. This requires a new helper function that does the test environment setup in the private mount+pid namespace, but this is much simpler and faster than using check itself to execute individual tests. This last bit of functionality is still a work in progress, so this specific patch is still tagged with [RFC].

There are lots of other bits of changes. The way common/rc and common/config are used is changed. common/config only sets up the execution environment now, and should not contain any code that needs to be executed outside of environment setup. It should only be sourced once at the highest level to set up the environment, and never called again.

common/rc is similar - all directly executed code has been removed from it, and that is now called from the high level code that needs initialisation work done. It no longer sources common/config, either. The test preamble does not need to run init_rc() any more; they just need to source the generic and fs specific functions the tests may run. Also, because check does some weird things and lots of _requires....() functions assume the TEST_DEV is mounted without first running _require_test(), it also needs to ensure the TEST_DEV is mounted...

check-parallel can now take a "-t N" parameter to specify how many execution threads it will use. If this is not specified, it will default to the number of CPUs in the machine. Testing with 4p restrictions show that check-parallel will run the quick group 3.5x faster on a 4p system with 8 execution threads than it will with a single execution thread. IOWs, even on small test systems, check-parallel can result in dramatic reductions in test runtime over check.

On a 64 p machine, testing XFS with the quick group drops from 61 minutes to just under 4 minutes. Testing XFS with the auto group drops from 246 minutes to just under 8 minutes.

Other miscellaneous stuff in the series:

- kill non-numeric test name support
- creating common/exit for all the general test exit functions to fix circular dependencies between common/rc and common/config
- fix iscratch_mkfs_sized to make USE_EXTERNAL on XFS work the same as ext4.
- dm-logwrites devices are now created by check-parallel
- several test conversions from sync() to syncfs()
- removal of a could of stale .c test source files.
- address poor CPU count scaling in a couple of tests

I have tried not to cause any regressions for people running plain check. I've tested that a bit with XFS and ext4, but I can't guarantee that there aren't issues I haven't uncovered. e.g. btrfs, as yet, is untested. It is unfortunate that the problem I seek to address - running exhaustive check testing across many filesystem types and configurations is prohibitively expensive in terms of time
- is the very reason I can't really adequately test check for regressions as I develop check-parallel functionality...

Thoughts, comments and code review all welcome!

-Dave.

 .gitignore                          |   1 -
 check                               | 727 ++++--------------------------------
 check-parallel                      | 351 ++++++++++++++---
 common/config                       | 612 +-----------------------------
 common/config-sections              | 461 +++++++++++++++++++++++
 common/dmlogwrites                  |   5 +-
 common/exit                         |  48 +++
 common/preamble                     |  19 +-
 common/rc                           | 253 +++++++++++--
 common/report                       |   2 +-
 common/test_exec                    | 377 +++++++++++++++++++
 common/test_list                    | 308 +++++++++++++++
 common/test_names                   |   8 +-
 new                                 |  24 --
 src/Makefile                        |   4 +-
 src/bulkstat_unlink_test.c          |  12 +-
 src/bulkstat_unlink_test_modified.c | 193 ----------
 src/fsync-tester.c                  |   2 +-
 src/open_by_handle.c                |   6 +-
 src/scaleread.c                     | 224 -----------
 src/scaleread.sh                    |  64 ----
 src/stale_handle.c                  |  15 +-
 tests/generic/531                   |   8 +-
 tests/xfs/259                       |   1 -
 tests/xfs/271                       |   2 -
 tools/run_test.sh                   | 116 ++++++
 26 files changed, 1954 insertions(+), 1889 deletions(-)