Run tests in CI
v3 cover Run tests in CI
Hi all,
changes v2->v3:
* going back to simplest changes in make
* adding lib/newlib_tests/runtest.sh (instead fighting with our build system I concentrated on runtest.sh)
* removing tst_fuzzy_sync01 as it sporadically fails
Tested:
https://github.com/pevik/ltp/actions/runs/984224611
TODO: Not what's wrong with PATH on CentOS 7:
/__w/ltp/ltp/lib/newlib_tests/runtest.sh: line 78: ./test05: No such file or directory ./shell/net/../../../../testcases/lib/tst_test.sh: line 149: tst_rod: command not found
Maybe it's in different directory?
It uses old make 3.82, there is something incompatible.
Kind regards,
Petr
Petr Vorel (5):
tst_device: Require root
lib: Add script for running tests
make: Add make test{, -c, -shell} targets
build.sh: Add support for make test
CI: Run also make test
.github/workflows/ci.yml | 3 + Makefile | 10 +++ build.sh | 18 ++++- lib/newlib_tests/runtest.sh | 120 ++++++++++++++++++++++++++++++++++ lib/newlib_tests/tst_device.c | 1 + 5 files changed, 151 insertions(+), 1 deletion(-) create mode 100755 lib/newlib_tests/runtest.sh
Signature · 1 line
2.32.0
PATCH v3 1/5 tst_device: Require root
as it's actually required for successful run:
tst_device.c:100: TINFO: Not allowed to open /dev/loop-control. Are you root?: EACCES (13) tst_device.c:139: TINFO: No free devices found
tst_device.c:335: TBROK: Failed to acquire device
--- lib/newlib_tests/tst_device.c | 1 + 1 file changed, 1 insertion(+)
lib/newlib_tests/tst_device.c +1 -0
@@ -40,6 +40,7 @@ static void do_test(void) }static struct tst_test test = { + .needs_root =1 , .needs_device =1 , .dev_min_size =300 , .test_all = do_test,
Signature · 1 line
2.32.0
PATCH v3 2/5 lib: Add script for running tests
In reply to Petr VorelFor now run only tests which TPASS or TCONF.
---
Hi,
not sure if using same name for custom versions is a good idea. Maybe I should have just used echo, but I plan to add other features to runtest.sh, thus I try to keep usage similar to normal tests. Maybe it's a wrong approach, making things less readable for new contributors who will try to figure out why their change in test API failed.
Kind regards,
Petr
lib/newlib_tests/runtest.sh | 120 ++++++++++++++++++++++++++++++++++++ 1 file changed, 120 insertions(+) create mode 100755 lib/newlib_tests/runtest.sh
lib/newlib_tests/runtest.sh new file +120 -0
@@ -0,0 +1,120 @@ +#!/bin/sh +# Copyright (c) 2021 Petr Vorel <[email protected]> + +LTP_C_API_TESTS="${LTP_C_API_TESTS:-test05 test07 test09 test12 test15 test16 test18 +test_exec test_timer tst_bool_expr tst_res_hexd tst_strstatus tst_fuzzy_sync02}" + +LTP_SHELL_API_TESTS="${LTP_SHELL_API_TESTS:-shell/tst_check_driver.sh shell/net/*.sh}" + +cd $(dirname $0) +PATH="$PWD/../../testcases/lib/:$PATH" + +. tst_ansi_color.sh + +usage() +{ + echo "Usage: $0 [-c|-s]" + echo "-c run C API tests only" + echo "-s run shell API tests only" + echo "-h print this help" +} + +# custom version +tst_flag2mask() +{ + case "$1" in + TPASS) return 0;; + TFAIL) return 1;; + TBROK) return 2;; + TWARN) return 4;; + TINFO) return 16;; + TCONF) return 32;; + esac +} + +# custom version +tst_res() +{ + if [ $# -eq 0 ]; then + echo >&2 + return + fi + + local res="$1" + shift + + tst_color_enabled + local color=$? + + printf "runtest " >&2 + tst_print_colored $res "$res: " >&2 + echo "$@" >&2 + +} + +# custom version +tst_brk() +{ + local res="$1" + shift + + tst_res + tst_res $res $@ + + exit $(tst_flag2mask $res) +} + +run_tests() +{ + local target="$1" + local i ret tconf tpass vars + + eval vars="\$LTP_${target}_API_TESTS" + + tst_res TINFO "=== Run $target tests ===" + + for i in $vars; do + tst_res TINFO "* $i" + ./$i + ret=$? + + case $ret in + 0) tpass="$tpass $i";; + 1) tst_brk TFAIL "$i failed with TFAIL";; + 2) tst_brk TFAIL "$i failed with TBROK";; + 4) tst_brk TFAIL "$i failed with TWARN";; + 32) tconf="$tconf $i";; + esac + tst_res + done + + [ -z "$tpass" ] && tpass=" none" + [ -z "$tconf" ] && tconf=" none" + + tst_res TINFO "=== $target TEST RESULTS ===" + tst_res TINFO "Tests exited with TPASS:$tpass" + tst_res TINFO "Tests exited with TCONF:$tconf" + tst_res +} + +run= +while getopts chs opt; do + case $opt in + 'h') usage; exit 0;; + 'c') run="c";; + 's') run="s";; + *) usage; tst_brk TBROK "Error: invalid option";; + esac +done + +tst_res TINFO "PATH='$PATH'" + +if [ -z "$run" -o "$run" = "c" ]; then + run_tests "C" +fi + +if [ -z "$run" -o "$run" = "s" ]; then + run_tests "SHELL" +fi + +tst_res TPASS "No test failed"
Signature · 1 line
2.32.0
PATCH v3 3/5 make: Add make test{, -c, -shell} targets
In reply to Petr VorelFor testing C and shell API.
---
Simplicity wins :).
Makefile | 10 ++++++++++ 1 file changed, 10 insertions(+)
Makefile +10 -0
@@ -192,6 +192,16 @@ $(INSTALL_TARGETS): $(INSTALL_DIR) $(DESTDIR)/$(bindir) ## Install install: $(INSTALL_TARGETS) +## Test +test: lib-all + $(top_srcdir)/lib/newlib_tests/runtest.sh + +test-c: lib-all + $(top_srcdir)/lib/newlib_tests/runtest.sh -c + +test-shell: lib-all + $(top_srcdir)/lib/newlib_tests/runtest.sh -s + ## Help .PHONY: help help:
Signature · 1 line
2.32.0
PATCH v3 4/5 build.sh: Add support for make test
In reply to Petr Vorel--- build.sh | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-)
build.sh +17 -1
@@ -119,6 +119,17 @@ build_out_tree() make $MAKE_OPTS_OUT_TREE } +test_in_tree() +{ + make test +} + +test_out_tree() +{ + cd $BUILD_DIR + make $MAKE_OPTS_OUT_TREE test +} + install_in_tree() { make $MAKE_OPTS install
@@ -165,6 +176,7 @@ RUN: autotools run only 'make autotools' configure run only 'configure' build run only 'make' +test run only 'make test' install run only 'make install' Default configure options:
@@ -192,7 +204,7 @@ while getopts "c:hio:p:r:t:" opt; do esac;; p) prefix="$OPTARG";; r) case "$OPTARG" in - autotools|configure|build|install) run="$OPTARG";; + autotools|configure|build|test|install) run="$OPTARG";; *) echo "Wrong run type '$OPTARG'" >&2; usage; exit 1;; esac;; t) case "$OPTARG" in
@@ -218,6 +230,10 @@ if [ -z "$run" -o "$run" = "build" ]; then eval build_${tree}_tree fi +if [ -z "$run" -o "$run" = "test" ]; then + eval test_${tree}_tree +fi + if [ -z "$run" -o "$run" = "install" ]; then if [ "$install" = 1 ]; then eval install_${tree}_tree
Signature · 1 line
2.32.0
PATCH v3 5/5 CI: Run also make test
In reply to Petr Vorelon all targets
--- .github/workflows/ci.yml | 3 +++ 1 file changed, 3 insertions(+)
.github/workflows/ci.yml +3 -0
@@ -151,6 +151,9 @@ jobs: - name: Compile run: ./build.sh -r build -o ${TREE:-in} + - name: Test + run: ./build.sh -r test -o ${TREE:-in} + - name: Install run: | if [ "$MAKE_INSTALL" = 1 ]; then INSTALL_OPT="-i"; fi
Signature · 1 line
2.32.0
v3 cover Run tests in CI
In reply to Petr VorelHi,
Quoted 11 lines
Hi all,
changes
v2->v3:
* going back to simplest changes in make
* addinglib/newlib_tests/runtest.sh(instead fighting with our build system I concentrated onruntest.sh)
* removingtst_fuzzy_sync01as it sporadically fails
Quoted 6 lines
TODO: Not what's wrong with PATH on CentOS 7:
/__w/ltp/ltp/lib/newlib_tests/runtest.sh: line 78: ./test05: No such file or directory./shell/net/../../../../testcases/lib/tst_test.sh: line 149:tst_rod: command not foundMaybe it's in different directory?
It uses old make 3.82, there is something incompatible.
OK, whole out-of-tree build is broken. There are 2 issues:
1) probably PATH ../../testcases/lib/ points to srcdir instead of build dir). It's just older make exit properly with 2, newer don't, not sure why.
2) The same issue with non-zero exit not being propagated on newer make is also when runtest.sh quits with:
runtest TFAIL: shell/net/tst_ipaddr_un.sh failed with TFAIL
I guess I need to use @set -e; in Makefile (I didn't want to use set -e in the runtest.sh itself, it would break printing printing what failed).
Kind regards,
Petr