Run tests in CI
Subsystems (2)
v5 cover Run tests in CI
Hi,
changes v4->v5:
* don't exit on test failure, but continue testing (Cyril)
* rename tst_{brk,res}() to runtest_{brk,res}() (Cyril)
* rewritten summary, put everything to the end
* merged 3 already acked commits
$ make test ...
runtest TINFO: === C TEST RESULTS ===
runtest TINFO: 13x TPASS: test05 test07 test09 test12 test15 test18 tst_bool_expr test_exec test_timer tst_res_hexd tst_strstatus tst_fuzzy_sync02 tst_fuzzy_sync03 runtest TINFO: 0x TFAIL:
runtest TINFO: 0x TBROK:
runtest TINFO: 0x TWARN:
runtest TINFO: 0x TCONF:
...
runtest TINFO: === SHELL TEST RESULTS ===
runtest TINFO: 2x TPASS: shell/tst_check_driver.sh shell/net/tst_ipaddr_un.sh runtest TINFO: 0x TFAIL:
runtest TINFO: 0x TBROK:
runtest TINFO: 0x TWARN:
runtest TINFO: 1x TCONF: shell/net/tst_rhost_run.sh
runtest TINFO: === FINAL TEST RESULTS === runtest TPASS: All C tests TCONF/TPASS runtest TPASS: All shell tests TCONF/TPASS
Petr Vorel (4):
lib: Add script for running tests
make: Add make test{, -c, -shell} targets build.sh: Add support for make test
CI: Run also make test
.github/workflows/ci.yml | 5 ++ Makefile | 23 +++++ build.sh | 22 ++++- lib/newlib_tests/runtest.sh | 175 ++++++++++++++++++++++++++++++++++++ 4 files changed, 224 insertions(+), 1 deletion(-) create mode 100755 lib/newlib_tests/runtest.sh
Signature · 1 line
2.32.0
PATCH v5 1/4 lib: Add script for running tests
For now run only tests which TPASS or TCONF.
tst_fuzzy_sync01: sporadically fails, thus disabled:
../../include/tst_fuzzy_sync.h:685: TINFO: Exceeded execution loops, requesting exit
tst_fuzzy_sync01.c:227: TFAIL: acs:3 act:1 art:1 | =:23 -:46 +:2999931
...
Summary:
passed 21
failed 3
--- lib/newlib_tests/runtest.sh | 175 ++++++++++++++++++++++++++++++++++++ 1 file changed, 175 insertions(+) create mode 100755 lib/newlib_tests/runtest.sh
lib/newlib_tests/runtest.sh new file +175 -0
@@ -0,0 +1,175 @@ +#!/bin/sh +# Copyright (c) 2021 Petr Vorel <[email protected]> + +LTP_C_API_TESTS="${LTP_C_API_TESTS:-test05 test07 test09 test12 test15 test18 +tst_bool_expr test_exec test_timer tst_res_hexd tst_strstatus tst_fuzzy_sync02 +tst_fuzzy_sync03}" + +LTP_SHELL_API_TESTS="${LTP_SHELL_API_TESTS:-shell/tst_check_driver.sh shell/net/*.sh}" + +cd $(dirname $0) +PATH="$PWD/../../testcases/lib/:$PATH" + +. tst_ansi_color.sh + +usage() +{ + cat << EOF +Usage: $0 [-b DIR ] [-c|-s] +-b DIR build directory (required for out-of-tree build) +-c run C API tests only +-s run shell API tests only +-h print this help +EOF +} + +tst_flag2mask() +{ + case "$1" in + TPASS) return 0;; + TFAIL) return 1;; + TBROK) return 2;; + TWARN) return 4;; + TINFO) return 16;; + TCONF) return 32;; + esac +} + +runtest_res() +{ + if [ $# -eq 0 ]; then + echo >&2 + return + fi + + local res="$1" + shift + + tst_color_enabled + local color=$? + + printf "runtest " >&2 + tst_print_colored $res "$res: " >&2 + echo "$@" >&2 + +} + +runtest_brk() +{ + local res="$1" + shift + + tst_flag2mask "$res" + local mask=$? + + runtest_res + runtest_res $res $@ + + exit $mask +} + +run_tests() +{ + local target="$1" + local i res ret=0 tbrok tconf tfail tpass twarn vars + + eval vars="\$LTP_${target}_API_TESTS" + + runtest_res TINFO "=== Run $target tests ===" + + for i in $vars; do + runtest_res TINFO "* $i" + ./$i + res=$? + + [ $res -ne 0 -a $res -ne 32 ] && ret=1 + + case $res in + 0) tpass="$tpass $i";; + 1) tfail="$tfail $i";; + 2) tbrok="$tbrok $i";; + 4) twarn="$twarn $i";; + 32) tconf="$tconf $i";; + 127) runtest_brk TBROK "Error: file not found (wrong PATH? out-of-tree build without -b?), exit code: $res";; + *) runtest_brk TBROK "Error: unknown failure, exit code: $res";; + esac + runtest_res + done + + runtest_res TINFO "=== $target TEST RESULTS ===" + runtest_res TINFO "$(echo $tpass | wc -w)x TPASS:$tpass" + runtest_res TINFO "$(echo $tfail | wc -w)x TFAIL:$tfail" + runtest_res TINFO "$(echo $tbrok | wc -w)x TBROK:$tbrok" + runtest_res TINFO "$(echo $twarn | wc -w)x TWARN:$twarn" + runtest_res TINFO "$(echo $tconf | wc -w)x TCONF:$tconf" + runtest_res + + return $ret +} + +run_c_tests() +{ + if [ "$builddir" ]; then + cd $builddir/lib/newlib_tests + fi + + run_tests "C" + + if [ "$builddir" ]; then + cd - + fi +} + +run_shell_tests() +{ + run_tests "SHELL" +} + + +print_result() +{ + local res="$1" + local target="$2" + + if [ -z "$res" ]; then + runtest_res TCONF "$target tests skipped" + elif [ $res -eq 0 ]; then + runtest_res TPASS "All $target tests TCONF/TPASS" + else + runtest_res TFAIL "Some $target test(s) TBROK/TFAIL/TWARN" + fi +} + +builddir= +c_fail= +run= +shell_fail= + +while getopts b:chs opt; do + case $opt in + 'h') usage; exit 0;; + 'b') builddir=$OPTARG; PATH="$builddir/testcases/lib:$PATH";; + 'c') run="c";; + 's') run="s";; + *) usage; runtest_brk TBROK "Error: invalid option";; + esac +done + +runtest_res TINFO "PATH='$PATH'" + +if [ -z "$run" -o "$run" = "c" ]; then + run_c_tests + c_fail=$? +fi + +if [ -z "$run" -o "$run" = "s" ]; then + run_shell_tests + shell_fail=$? +fi + +runtest_res TINFO "=== FINAL TEST RESULTS ===" + +print_result "$c_fail" "C" +print_result "$shell_fail" "shell" + +exit $((c_fail|shell_fail))
Signature · 1 line
2.32.0
PATCH v5 2/4 make: Add make test{, -c, -shell} targets
In reply to Petr VorelFor testing C and shell API.
--- Makefile | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
Makefile +23 -0
@@ -192,6 +192,29 @@ $(INSTALL_TARGETS): $(INSTALL_DIR) $(DESTDIR)/$(bindir) ## Install install: $(INSTALL_TARGETS) +## Test +define _test + @set -e; $(top_srcdir)/lib/newlib_tests/runtest.sh -b $(abs_builddir) $(1) +endef + +test: lib-all +ifneq ($(build),$(host)) + $(error running tests on cross-compile build not supported) +endif + $(call _test) + +test-c: lib-all +ifneq ($(build),$(host)) + $(error running tests on cross-compile build not supported) +endif + $(call _test,-c) + +test-shell: lib-all +ifneq ($(build),$(host)) + $(error running tests on cross-compile build not supported) +endif + $(call _test,-s) + ## Help .PHONY: help help:
Signature · 1 line
2.32.0
PATCH v5 3/4 build.sh: Add support for make test
In reply to Petr Vorel--- build.sh | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-)
build.sh +21 -1
@@ -119,6 +119,17 @@ build_out_tree() make $MAKE_OPTS_OUT_TREE } +test_in_tree() +{ + make test +} + +test_out_tree() +{ + cd $BUILD_DIR + make $MAKE_OPTS_OUT_TREE test +} + install_in_tree() { make $MAKE_OPTS install
@@ -165,6 +176,7 @@ RUN: autotools run only 'make autotools' configure run only 'configure' build run only 'make' +test run only 'make test' (not supported for cross-compile build) install run only 'make install' Default configure options:
@@ -192,7 +204,7 @@ while getopts "c:hio:p:r:t:" opt; do esac;; p) prefix="$OPTARG";; r) case "$OPTARG" in - autotools|configure|build|install) run="$OPTARG";; + autotools|configure|build|test|install) run="$OPTARG";; *) echo "Wrong run type '$OPTARG'" >&2; usage; exit 1;; esac;; t) case "$OPTARG" in
@@ -218,6 +230,14 @@ if [ -z "$run" -o "$run" = "build" ]; then eval build_${tree}_tree fi +if [ -z "$run" -o "$run" = "test" ]; then + if [ "$build" = "cross" ]; then + echo "cross-compile build, skipping running tests" >&2 + else + eval test_${tree}_tree + fi +fi + if [ -z "$run" -o "$run" = "install" ]; then if [ "$install" = 1 ]; then eval install_${tree}_tree
Signature · 1 line
2.32.0
PATCH v5 4/4 CI: Run also make test
In reply to Petr Vorelon all targets
--- .github/workflows/ci.yml | 5 +++++ 1 file changed, 5 insertions(+)
.github/workflows/ci.yml +5 -0
@@ -151,6 +151,11 @@ jobs: - name: Compile run: ./build.sh -r build -o ${TREE:-in} + - name: Test + run: | + case "$VARIANT" in cross-compile*) BUILD="cross";; i386) BUILD="32";; *) BUILD="native";; esac + ./build.sh -r test -o ${TREE:-in} -t $BUILD + - name: Install run: | if [ "$MAKE_INSTALL" = 1 ]; then INSTALL_OPT="-i"; fi
Signature · 1 line
2.32.0
v5 cover Run tests in CI
In reply to Petr VorelHi all,
tested:
https://github.com/pevik/ltp/actions/runs/1030551185
One thing which bothers me is mangled output of several tests together:
https://github.com/pevik/ltp/runs/3067600696?check_suite_focus=true#step:11:554
runtest TINFO: * tst_res_hexd
/__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: dump1 /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 48 65 6c 6c 6f 20 66 72 6f 6d 20 74 73 74 5f 72 /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 65 73 5f 68 65 78 64 00
runtest TINFO: * tst_strstatus /__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s
Summary:
passed 35 failed 0 broken 0 skipped 0
warnings 0
Summary:
passed 1 failed 0 broken 0 skipped 0
warnings 0
Kind regards,
Petr
v5 cover Run tests in CI
Hi!
Quoted 20 lines
tested:
https://github.com/pevik/ltp/actions/runs/1030551185One thing which bothers me is mangled output of several tests together:
https://github.com/pevik/ltp/runs/3067600696?check_suite_focus=true#step:11:554runtest TINFO: *
tst_res_hexd
/__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: dump1 /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 48 65 6c 6c 6f 20 66 72 6f 6d 20 74 73 74 5f 72 /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 65 73 5f 68 65 78 64 00runtest TINFO: *
tst_strstatus/__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00sSummary:
passed 35 failed 0 broken 0 skipped 0
Quoted 8 lines
warnings 0
Summary:
passed 1 failed 0 broken 0 skipped 0warnings 0
That is indeed strange, you are running these in a loop, the buffers should have been flushed once controll returns back to the shell from the test.
I guess that this is a combination of:
* The stdout and stderr are block buffered by the gitlab CI and flushed at different times
* The summary is written into stdout in contrast with the message so we end up with summary printed later than the rest that goes into stderr
I guess this should fix it:
lib/tst_test.c +6 -6
@@ -734,12 +734,12 @@ static void do_exit(int ret)if (results->broken) ret |=TBROK ; -printf ("\nSummary:\n" ); -printf ("passed %d\n" , results->passed); -printf ("failed %d\n" , results->failed); -printf ("broken %d\n" , results->broken); -printf ("skipped %d\n" , results->skipped); -printf ("warnings %d\n" , results->warnings); +fprintf (stderr,"\nSummary:\n" ); +fprintf (stderr,"passed %d\n" , results->passed); +fprintf (stderr,"failed %d\n" , results->failed); +fprintf (stderr,"broken %d\n" , results->broken); +fprintf (stderr,"skipped %d\n" , results->skipped); +fprintf (stderr,"warnings %d\n" , results->warnings); }do_cleanup ();
Signature · 2 lines
Cyril Hrubis [email protected]
v5 cover Run tests in CI
Hi Cyril,
Quoted 34 lines
Hi!
Quoted 20 lines
tested:
https://github.com/pevik/ltp/actions/runs/1030551185One thing which bothers me is mangled output of several tests together:
https://github.com/pevik/ltp/runs/3067600696?check_suite_focus=true#step:11:554runtest TINFO: *
tst_res_hexd
/__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: dump1 /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 48 65 6c 6c 6f 20 66 72 6f 6d 20 74 73 74 5f 72 /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 65 73 5f 68 65 78 64 00runtest TINFO: *
tst_strstatus/__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00sSummary:
passed 35 failed 0 broken 0 skipped 0Quoted 8 lines
warnings 0
Summary:
passed 1 failed 0 broken 0 skipped 0warnings 0
That is indeed strange, you are running these in a loop, the buffers should have been flushed once controll returns back to the shell from the test.
Quoted 8 lines
I guess that this is a combination of:
* The stdout and stderr are block buffered by the gitlab CI and flushed at different times
* The summary is written into stdout in contrast with the message so we end up with summary printed later than the rest that goes into stderr
Yep, it could be.
Quoted 13 lines
I guess this should fix it:
lib/tst_test.c +0
-3@@ -734,12 +734,12 @@ static void do_exit(int ret)if (results->broken) ret |=TBROK ; -printf ("\nSummary:\n" ); -printf ("passed %d\n" , results->passed); -printf ("failed %d\n" , results->failed);
lib/tst_test.c static void do_exit(int ret) · line 743
Quoted 10 lines
lib/tst_test.c +6
-3-printf ("broken %d\n" , results->broken); -printf ("skipped %d\n" , results->skipped); -printf ("warnings %d\n" , results->warnings); +fprintf (stderr,"\nSummary:\n" ); +fprintf (stderr,"passed %d\n" , results->passed); +fprintf (stderr,"failed %d\n" , results->failed); +fprintf (stderr,"broken %d\n" , results->broken); +fprintf (stderr,"skipped %d\n" , results->skipped); +fprintf (stderr,"warnings %d\n" , results->warnings); }
Thanks! Testing it in https://github.com/pevik/ltp/actions/runs/1031005963
Kind regards,
Petr
v5 cover Run tests in CI
Hi Cyril,
...
Quoted 24 lines
Quoted 13 lines
I guess this should fix it:
lib/tst_test.c +0
-3@@ -734,12 +734,12 @@ static void do_exit(int ret)if (results->broken) ret |=TBROK ; -printf ("\nSummary:\n" ); -printf ("passed %d\n" , results->passed); -printf ("failed %d\n" , results->failed);lib/tst_test.c static void do_exit(int ret) · line 743
Quoted 10 lines
lib/tst_test.c +6
-3-printf ("broken %d\n" , results->broken); -printf ("skipped %d\n" , results->skipped); -printf ("warnings %d\n" , results->warnings); +fprintf (stderr,"\nSummary:\n" ); +fprintf (stderr,"passed %d\n" , results->passed); +fprintf (stderr,"failed %d\n" , results->failed); +fprintf (stderr,"broken %d\n" , results->broken); +fprintf (stderr,"skipped %d\n" , results->skipped); +fprintf (stderr,"warnings %d\n" , results->warnings); }
Quoted 1 line
Thanks! Testing it in https://github.com/pevik/ltp/actions/runs/1031005963
The result is much better but still not correct:
https://github.com/pevik/ltp/runs/3068932859?check_suite_focus=true#step:11:707
runtest TINFO: * shell/net/tst_ipaddr_un.sh
Summary:
passed 0 failed 0 broken 0 skipped 1
warnings 0tst_ipaddr_un 1 TINFO: timeout per run is 0h 5m 0s
Could you please merge your fix? And I'll try sync after each test.
https://github.com/pevik/ltp/actions/runs/1033070696
What is worse is that tests are green, but I see TBROK and TWARN which are not reflected by results:
tst_check_driver 1 TBROK: Test still running, sending SIGKILL
...tst_rhost_run 1 TCONF: (null) 0 TWARN : tst_kernel.c:105: expected file /lib/modules/5.8.0-1036-azure/modules.dep does not exist or not a file
veth(null) 0 TWARN : tst_kernel.c:105: expected file /lib/modules/5.8.0-1036-azure/modules.builtin does not exist or not a file driver not available
Summary:
passed 0 failed 0 broken 0 skipped 1
warnings 0
runtest TINFO: === SHELL TEST RESULTS ===
runtest TINFO: 1x TPASS: shell/net/tst_ipaddr_un.sh
runtest TINFO: 0x TFAIL:
runtest TINFO: 0x TBROK:
runtest TINFO: 0x TWARN:
runtest TINFO: 2x TCONF: shell/tst_check_driver.sh shell/net/tst_rhost_run.sh
Kind regards,
Petr
Quoted 2 lines
Kind regards,
Petr
v5 cover Run tests in CI
Hi Cyril,
...
Quoted 12 lines
Quoted 1 line
Thanks! Testing it in https://github.com/pevik/ltp/actions/runs/1031005963
The result is much better but still not correct:
https://github.com/pevik/ltp/runs/3068932859?check_suite_focus=true#step:11:707runtest TINFO: *
shell/net/tst_ipaddr_un.shSummary:
passed 0 failed 0 broken 0 skipped 1
Quoted 5 lines
warnings 0
tst_ipaddr_un1 TINFO: timeout per run is 0h 5m 0sCould you please merge your fix? And I'll try sync after each test.
https://github.com/pevik/ltp/actions/runs/1033070696
There is still some output printed later, at least In RPN: A B | C ! ! &
In RPN: A B | C ! ! &
which is from tst_bool_expr.c, but printed after tst_strstatus
https://github.com/pevik/ltp/runs/3074335940?check_suite_focus=true#step:11:587
But it's much better.
I also found a bug in final evaluation on one of my VM:
tst_test.c:1313: TBROK: Test killed by SIGSEGV!
...
runtest TINFO: 1x TBROK: tst_bool_expr
...
runtest TINFO: === FINAL TEST RESULTS === runtest TPASS: All C tests TCONF/TPASS <- THIS IS WRONG runtest TPASS: All shell tests TCONF/TPASS
Kind regards,
Petr
v5 cover Run tests in CI
In reply to Petr VorelHi!
Quoted 4 lines
Quoted 1 line
Thanks! Testing it in https://github.com/pevik/ltp/actions/runs/1031005963
The result is much better but still not correct:
https://github.com/pevik/ltp/runs/3068932859?check_suite_focus=true#step:11:707
If tests are printing into stdout as well it will be easier to redirect the test output in the script so that the CI gets it all into a single file descriptor.
What about doing ./$i 1>&2 in the loop that runs the tests instead?
Signature · 2 lines
Cyril Hrubis [email protected]
v5 cover Run tests in CI
Quoted 11 lines
Hi!
Quoted 4 lines
Quoted 1 line
Thanks! Testing it in https://github.com/pevik/ltp/actions/runs/1031005963
The result is much better but still not correct:
https://github.com/pevik/ltp/runs/3068932859?check_suite_focus=true#step:11:707If tests are printing into stdout as well it will be easier to redirect the test output in the script so that the CI gets it all into a single file descriptor.
What about doing ./$i 1>&2 in the loop that runs the tests instead?
How simple, that should work, I'll test that.
FYI I also need to install iproute (runtime shell API test dependency). Simple enough to just merge it after this gets merged.