Run tests in CI
Subsystems (2)
v6 cover Run tests in CI
Hi,
changes v5->v6:
* add new commit suggested by Cyril: "lib: Print Summary: into stderr"
* fix final error evaluation for C tests in runtest.sh
* run C and shell test separately (test-c and test-shell instead of test). Having them separate helps to easily navigate in CI. This required to add support to build.sh.
Hope this is a final version.
Kind regards,
Petr
Petr Vorel (5):
lib: Print Summary: into stderr
lib: Add script for running tests
make: Add make test{, -c, -shell} targets
build.sh: Add support for make test{,-c,-shell}
CI: Run also make test-c, test-shell
.github/workflows/ci.yml | 10 ++ Makefile | 23 +++++ build.sh | 24 ++++- lib/newlib_tests/runtest.sh | 182 ++++++++++++++++++++++++++++++++++++ lib/tst_test.c | 12 +-- 5 files changed, 244 insertions(+), 7 deletions(-) create mode 100755 lib/newlib_tests/runtest.sh
Signature · 1 line
2.32.0
PATCH v6 1/5 lib: Print Summary: into stderr
to follow tst_{brk,res}() being printed to stderr.
Found with GitHub actions where stdout and stderr are probably block buffered and flushed at different times.
---
* new in v6
lib/tst_test.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
lib/tst_test.c +6 -6
@@ -734,12 +734,12 @@ static void do_exit(int ret)if (results->broken) ret |=TBROK ; -printf ("\nSummary:\n" ); -printf ("passed %d\n" , results->passed); -printf ("failed %d\n" , results->failed); -printf ("broken %d\n" , results->broken); -printf ("skipped %d\n" , results->skipped); -printf ("warnings %d\n" , results->warnings); +fprintf (stderr,"\nSummary:\n" ); +fprintf (stderr,"passed %d\n" , results->passed); +fprintf (stderr,"failed %d\n" , results->failed); +fprintf (stderr,"broken %d\n" , results->broken); +fprintf (stderr,"skipped %d\n" , results->skipped); +fprintf (stderr,"warnings %d\n" , results->warnings); }do_cleanup ();
Signature · 1 line
2.32.0
PATCH v6 2/5 lib: Add script for running tests
In reply to Petr VorelFor now run only tests which TPASS or TCONF.
tst_fuzzy_sync01: sporadically fails, thus disabled:
../../include/tst_fuzzy_sync.h:685: TINFO: Exceeded execution loops, requesting exit
tst_fuzzy_sync01.c:227: TFAIL: acs:3 act:1 art:1 | =:23 -:46 +:2999931
...
Summary:
passed 21
failed 3
Use sync to workaround mangled output on GitHub Actions where stdout and stderr are probably block buffered and flushed at different times.
---
changes v5->v6:
* fix final error evaluation for C tests in runtest.sh
lib/newlib_tests/runtest.sh | 182 ++++++++++++++++++++++++++++++++++++ 1 file changed, 182 insertions(+) create mode 100755 lib/newlib_tests/runtest.sh
lib/newlib_tests/runtest.sh new file +182 -0
@@ -0,0 +1,182 @@ +#!/bin/sh +# Copyright (c) 2021 Petr Vorel <[email protected]> + +LTP_C_API_TESTS="${LTP_C_API_TESTS:-test05 test07 test09 test12 test15 test18 +tst_bool_expr test_exec test_timer tst_res_hexd tst_strstatus tst_fuzzy_sync02 +tst_fuzzy_sync03}" + +LTP_SHELL_API_TESTS="${LTP_SHELL_API_TESTS:-shell/tst_check_driver.sh shell/net/*.sh}" + +cd $(dirname $0) +PATH="$PWD/../../testcases/lib/:$PATH" + +. tst_ansi_color.sh + +usage() +{ + cat << EOF +Usage: $0 [-b DIR ] [-c|-s] +-b DIR build directory (required for out-of-tree build) +-c run C API tests only +-s run shell API tests only +-h print this help +EOF +} + +tst_flag2mask() +{ + case "$1" in + TPASS) return 0;; + TFAIL) return 1;; + TBROK) return 2;; + TWARN) return 4;; + TINFO) return 16;; + TCONF) return 32;; + esac +} + +runtest_res() +{ + if [ $# -eq 0 ]; then + echo >&2 + return + fi + + local res="$1" + shift + + tst_color_enabled + local color=$? + + printf "runtest " >&2 + tst_print_colored $res "$res: " >&2 + echo "$@" >&2 + +} + +runtest_brk() +{ + local res="$1" + shift + + tst_flag2mask "$res" + local mask=$? + + runtest_res + runtest_res $res $@ + + exit $mask +} + +run_tests() +{ + local target="$1" + local i res ret=0 tbrok tconf tfail tpass twarn vars + + eval vars="\$LTP_${target}_API_TESTS" + + runtest_res TINFO "=== Run $target tests ===" + + for i in $vars; do + runtest_res TINFO "* $i" + ./$i + res=$? + + [ $res -ne 0 -a $res -ne 32 ] && ret=1 + + case $res in + 0) tpass="$tpass $i";; + 1) tfail="$tfail $i";; + 2) tbrok="$tbrok $i";; + 4) twarn="$twarn $i";; + 32) tconf="$tconf $i";; + 127) runtest_brk TBROK "Error: file not found (wrong PATH? out-of-tree build without -b?), exit code: $res";; + *) runtest_brk TBROK "Error: unknown failure, exit code: $res";; + esac + runtest_res + sync + done + + runtest_res TINFO "=== $target TEST RESULTS ===" + runtest_res TINFO "$(echo $tpass | wc -w)x TPASS:$tpass" + runtest_res TINFO "$(echo $tfail | wc -w)x TFAIL:$tfail" + runtest_res TINFO "$(echo $tbrok | wc -w)x TBROK:$tbrok" + runtest_res TINFO "$(echo $twarn | wc -w)x TWARN:$twarn" + runtest_res TINFO "$(echo $tconf | wc -w)x TCONF:$tconf" + runtest_res + + return $ret +} + +run_c_tests() +{ + local ret + + if [ "$builddir" ]; then + cd $builddir/lib/newlib_tests + fi + + run_tests "C" + ret=$? + + if [ "$builddir" ]; then + cd - + fi + + return $ret +} + +run_shell_tests() +{ + run_tests "SHELL" +} + + +print_result() +{ + local target="$1" + local res="$2" + + + if [ -z "$res" ]; then + runtest_res TCONF "$target tests skipped" + elif [ $res -eq 0 ]; then + runtest_res TPASS "All $target tests TCONF/TPASS" + else + runtest_res TFAIL "Some $target test(s) TBROK/TFAIL/TWARN" + fi +} + +builddir= +c_fail= +run= +shell_fail= + +while getopts b:chs opt; do + case $opt in + 'h') usage; exit 0;; + 'b') builddir=$OPTARG; PATH="$builddir/testcases/lib:$PATH";; + 'c') run="c";; + 's') run="s";; + *) usage; runtest_brk TBROK "Error: invalid option";; + esac +done + +runtest_res TINFO "PATH='$PATH'" + +if [ -z "$run" -o "$run" = "c" ]; then + run_c_tests + c_fail=$? +fi + +if [ -z "$run" -o "$run" = "s" ]; then + run_shell_tests + shell_fail=$? +fi + +runtest_res TINFO "=== FINAL TEST RESULTS ===" + +print_result "C" "$c_fail" +print_result "shell" "$shell_fail" + +exit $((c_fail|shell_fail))
Signature · 1 line
2.32.0
PATCH v6 3/5 make: Add make test{, -c, -shell} targets
In reply to Petr VorelFor testing C and shell API.
---
the same as in v5
Makefile | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
Makefile +23 -0
@@ -192,6 +192,29 @@ $(INSTALL_TARGETS): $(INSTALL_DIR) $(DESTDIR)/$(bindir) ## Install install: $(INSTALL_TARGETS) +## Test +define _test + @set -e; $(top_srcdir)/lib/newlib_tests/runtest.sh -b $(abs_builddir) $(1) +endef + +test: lib-all +ifneq ($(build),$(host)) + $(error running tests on cross-compile build not supported) +endif + $(call _test) + +test-c: lib-all +ifneq ($(build),$(host)) + $(error running tests on cross-compile build not supported) +endif + $(call _test,-c) + +test-shell: lib-all +ifneq ($(build),$(host)) + $(error running tests on cross-compile build not supported) +endif + $(call _test,-s) + ## Help .PHONY: help help:
Signature · 1 line
2.32.0
PATCH v6 4/5 build.sh: Add support for make test{, -c, -shell}
In reply to Petr Vorel---
changes v5->v6 (not yet reviewed by Cyril):
* add support to run also test-c and test-shell
build.sh | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)
build.sh +23 -1
@@ -119,6 +119,17 @@ build_out_tree() make $MAKE_OPTS_OUT_TREE } +test_in_tree() +{ + make $1 +} + +test_out_tree() +{ + cd $BUILD_DIR + make $MAKE_OPTS_OUT_TREE $1 +} + install_in_tree() { make $MAKE_OPTS install
@@ -165,6 +176,9 @@ RUN: autotools run only 'make autotools' configure run only 'configure' build run only 'make' +test run only 'make test' (not supported for cross-compile build) +test-c run only 'make test-c' (not supported for cross-compile build) +test-shell run only 'make test-shell' (not supported for cross-compile build) install run only 'make install' Default configure options:
@@ -192,7 +206,7 @@ while getopts "c:hio:p:r:t:" opt; do esac;; p) prefix="$OPTARG";; r) case "$OPTARG" in - autotools|configure|build|install) run="$OPTARG";; + autotools|configure|build|test|test-c|test-shell|install) run="$OPTARG";; *) echo "Wrong run type '$OPTARG'" >&2; usage; exit 1;; esac;; t) case "$OPTARG" in
@@ -218,6 +232,14 @@ if [ -z "$run" -o "$run" = "build" ]; then eval build_${tree}_tree fi +if [ -z "$run" -o "$run" = "test" -o "$run" = "test-c" -o "$run" = "test-shell" ]; then + if [ "$build" = "cross" ]; then + echo "cross-compile build, skipping running tests" >&2 + else + eval test_${tree}_tree $run + fi +fi + if [ -z "$run" -o "$run" = "install" ]; then if [ "$install" = 1 ]; then eval install_${tree}_tree
Signature · 1 line
2.32.0
PATCH v6 5/5 CI: Run also make test-c, test-shell
In reply to Petr Vorelrun C and shell test separately (test-c and test-shell instead of test). Having them separate helps to easily navigate in CI.
It's run on all jobs, but on cross compile jobs it's obviously skipped.
---
changes v5->v6:
* run C and shell test separately (test-c and test-shell instead of test). Having them separate helps to easily navigate in CI. This required to add support to build.sh.
.github/workflows/ci.yml | 10 ++++++++++ 1 file changed, 10 insertions(+)
.github/workflows/ci.yml +10 -0
@@ -151,6 +151,16 @@ jobs: - name: Compile run: ./build.sh -r build -o ${TREE:-in} + - name: Test C API + run: | + case "$VARIANT" in cross-compile*) BUILD="cross";; i386) BUILD="32";; *) BUILD="native";; esac + ./build.sh -r test-c -o ${TREE:-in} -t $BUILD + + - name: Test shell API + run: | + case "$VARIANT" in cross-compile*) BUILD="cross";; i386) BUILD="32";; *) BUILD="native";; esac + ./build.sh -r test-shell -o ${TREE:-in} -t $BUILD + - name: Install run: | if [ "$MAKE_INSTALL" = 1 ]; then INSTALL_OPT="-i"; fi
Signature · 1 line
2.32.0
v6 cover Run tests in CI
In reply to Petr VorelHi Cyril,
Quoted 10 lines
Hi,
changes
v5->v6:
* add new commit suggested by Cyril: "lib: Print Summary: into stderr"
* fix final error evaluation for C tests inruntest.sh
* run C and shell test separately (test-c and test-shell instead of test). Having them separate helps to easily navigate in CI. This required to add support tobuild.sh.Hope this is a final version.
Verified slightly modified version (see diff below):
https://github.com/pevik/ltp/actions/runs/1033374720
FYI I'd probably keep first commit "ib: Print Summary: into stderr" although it should not be needed for CI as 1>&2 redirection fixed the output.
Can I merge it with your ack?
Kind regards,
Petr
debian.sh +1 -0
@@ -22,6 +22,7 @@ $apt \ clang \ gcc \ git \ + iproute2 \ libacl1 \ libacl1-dev \ libaio-dev \
fedora.sh +1 -0
@@ -13,6 +13,7 @@ $yum \ gcc \ git \ findutils \ + iproute \ numactl-devel \ libtirpc \ libtirpc-devel \
tumbleweed.sh +1 -0
@@ -13,6 +13,7 @@ $zyp \ gcc \ git \ gzip \ + iproute2 \ make \ kernel-default-devel \ keyutils-devel \
newlib_tests/runtest.sh +1 -2
@@ -79,7 +79,7 @@ run_tests() for i in $vars; do runtest_res TINFO "* $i" - ./$i + ./$i 1>&2 res=$? [ $res -ne 0 -a $res -ne 32 ] && ret=1
@@ -94,7 +94,6 @@ run_tests() *) runtest_brk TBROK "Error: unknown failure, exit code: $res";; esac runtest_res - sync done runtest_res TINFO "=== $target TEST RESULTS ==="
v6 cover Run tests in CI
Hi Cyril,
Quoted 2 lines
Verified slightly modified version (see diff below):
https://github.com/pevik/ltp/actions/runs/1033374720
Well, still some minor order issue with "In RPN: ..." (printed to stdout) being at the end, but I give up and merge (unless you have explanation for it).
https://github.com/pevik/ltp/runs/3075175819?check_suite_focus=true#step:11:501
Kind regards,
Petr