Run tests in CI
Subsystems (2)
v7 cover Run tests in CI
Hi,
changes v6->v7:
* new commit: test/test_zero_hugepage.sh: Skip test on read-only file system
* new commit: ci: Install iproute2
* Added test_zero_hugepage.sh (required to have look on source directory, because we use scripts for running C API (required for out-of-tree builds).
Tested:
https://github.com/pevik/ltp/actions/runs/1091097653
Kind regards,
Petr
Diff to v6:
debian.sh +1 -0
@@ -22,6 +22,7 @@ $apt \ clang \ gcc \ git \ + iproute2 \ libacl1 \ libacl1-dev \ libaio-dev \
fedora.sh +1 -0
@@ -13,6 +13,7 @@ $yum \ gcc \ git \ findutils \ + iproute \ numactl-devel \ libtirpc \ libtirpc-devel \
tumbleweed.sh +1 -0
@@ -13,6 +13,7 @@ $zyp \ gcc \ git \ gzip \ + iproute2 \ make \ kernel-default-devel \ keyutils-devel \
newlib_tests/runtest.sh +15 -7
@@ -3,7 +3,7 @@ LTP_C_API_TESTS="${LTP_C_API_TESTS:-test05 test07 test09 test12 test15 test18 tst_bool_expr test_exec test_timer tst_res_hexd tst_strstatus tst_fuzzy_sync02 -tst_fuzzy_sync03}" +tst_fuzzy_sync03 test_zero_hugepage.sh}" LTP_SHELL_API_TESTS="${LTP_SHELL_API_TESTS:-shell/tst_check_driver.sh shell/net/*.sh}"
@@ -71,7 +71,8 @@ runtest_brk() run_tests() { local target="$1" - local i res ret=0 tbrok tconf tfail tpass twarn vars + local srcdir="$2" + local dir i res ret=0 tbrok tconf tfail tpass twarn vars eval vars="\$LTP_${target}_API_TESTS"
@@ -79,7 +80,15 @@ run_tests() for i in $vars; do runtest_res TINFO "* $i" - ./$i + if [ -f "$i" ]; then + dir="." + elif [ "$srcdir" -a -f "$srcdir/$i" ]; then + dir="$srcdir" + else + runtest_brk TBROK "Error: $i file not found (PWD: $PWD)" + fi + + $dir/$i 1>&2 res=$? [ $res -ne 0 -a $res -ne 32 ] && ret=1
@@ -94,7 +103,6 @@ run_tests() *) runtest_brk TBROK "Error: unknown failure, exit code: $res";; esac runtest_res - sync done runtest_res TINFO "=== $target TEST RESULTS ==="
@@ -110,17 +118,17 @@ run_tests() run_c_tests() { - local ret + local ret srcdir="$PWD" if [ "$builddir" ]; then cd $builddir/lib/newlib_tests fi - run_tests "C" + run_tests "C" "$srcdir" ret=$? if [ "$builddir" ]; then - cd - + cd "$srcdir" fi return $ret
newlib_tests/test_zero_hugepage.sh +5 -1
@@ -1,12 +1,16 @@ #!/bin/sh # SPDX-License-Identifier: GPL-2.0-or-later # Copyright (c) 2021 Yang Xu <[email protected]> +# Copyright (c) 2021 Petr Vorel <[email protected]> echo "Testing .request_hugepages = TST_NO_HUGEPAGES" orig_value=`cat /proc/sys/vm/nr_hugepages` -echo "128" > /proc/sys/vm/nr_hugepages +if ! echo "128" > /proc/sys/vm/nr_hugepages; then + echo "TCONF: failed to open /proc/sys/vm/nr_hugepages" + exit 32 +fi ./test_zero_hugepage
Petr Vorel (7):
lib: Print Summary: into stderr test/test_zero_hugepage.sh: Skip test on read-only file system
lib: Add script for running tests
make: Add make test{, -c, -shell} targets
build.sh: Add support for make test{,-c,-shell}
CI: Run also make test-c, test-shell
ci: Install iproute2
.github/workflows/ci.yml | 10 ++ Makefile | 23 +++ build.sh | 24 +++- ci/debian.sh | 1 + ci/fedora.sh | 1 + ci/tumbleweed.sh | 1 + lib/newlib_tests/runtest.sh | 190 +++++++++++++++++++++++++ lib/newlib_tests/test_zero_hugepage.sh | 6 +- lib/tst_test.c | 12 +- 9 files changed, 260 insertions(+), 8 deletions(-) create mode 100755 lib/newlib_tests/runtest.sh
Signature · 1 line
2.32.0
PATCH v7 1/7 lib: Print Summary: into stderr
to follow tst_{brk,res}() being printed to stderr.
Found with GitHub actions where stdout and stderr are probably block buffered and flushed at different times.
---
The same as in v6.
lib/tst_test.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
lib/tst_test.c +6 -6
@@ -734,12 +734,12 @@ static void do_exit(int ret)if (results->broken) ret |=TBROK ; -printf ("\nSummary:\n" ); -printf ("passed %d\n" , results->passed); -printf ("failed %d\n" , results->failed); -printf ("broken %d\n" , results->broken); -printf ("skipped %d\n" , results->skipped); -printf ("warnings %d\n" , results->warnings); +fprintf (stderr,"\nSummary:\n" ); +fprintf (stderr,"passed %d\n" , results->passed); +fprintf (stderr,"failed %d\n" , results->failed); +fprintf (stderr,"broken %d\n" , results->broken); +fprintf (stderr,"skipped %d\n" , results->skipped); +fprintf (stderr,"warnings %d\n" , results->warnings); }do_cleanup ();
Signature · 1 line
2.32.0
PATCH v7 1/7 lib: Print Summary: into stderr
Hi!
I wonder if this change can break anything but I guess that it's unlikely.
Also while you are at it can you also fix tags? We do print help() to stderr but tags that follow go into stdout which is rather starnge.
And the failure hints are written to stdout as well, which should be fixed as well.
Quoted 34 lines
Suggested-by Cyril Hrubis <[email protected]>Signed-off-by Petr Vorel <[email protected]>---
The same as in v6.lib/tst_test.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)lib/tst_test.c +6
-6@@ -734,12 +734,12 @@ static void do_exit(int ret)if (results->broken) ret |=TBROK ; -printf ("\nSummary:\n" ); -printf ("passed %d\n" , results->passed); -printf ("failed %d\n" , results->failed); -printf ("broken %d\n" , results->broken); -printf ("skipped %d\n" , results->skipped); -printf ("warnings %d\n" , results->warnings); +fprintf (stderr,"\nSummary:\n" ); +fprintf (stderr,"passed %d\n" , results->passed); +fprintf (stderr,"failed %d\n" , results->failed); +fprintf (stderr,"broken %d\n" , results->broken); +fprintf (stderr,"skipped %d\n" , results->skipped); +fprintf (stderr,"warnings %d\n" , results->warnings); }do_cleanup ();Signature · 1 line
2.32.0
Signature · 2 lines
Cyril Hrubis [email protected]
PATCH v7 1/7 lib: Print Summary: into stderr
Hi Cyril,
Quoted 9 lines
Hi!
I wonder if this change can break anything but I guess that it's unlikely.Also while you are at it can you also fix tags? We do print
help()to stderr but tags that follow go into stdout which is rather starnge.And the failure hints are written to stdout as well, which should be fixed as well.
Sure, I'll fix them all in v8.
Kind regards,
Petr
PATCH v7 2/7 test/test_zero_hugepage.sh: Skip test on read-only file system
In reply to Petr VorelOn GitHub Actions /proc/sys/vm/nr_hugepages is not allowed to be changed:
./test_zero_hugepage.sh: line 9: can't create /proc/sys/vm/nr_hugepages: Read-only file system tst_hugepage.c:57: TBROK: Failed to open FILE '/proc/sys/vm/nr_hugepages' for writing: EROFS (30) tst_sys_conf.c:102: TWARN: Failed to open FILE '/proc/sys/vm/nr_hugepages'
---
New in v7.
lib/newlib_tests/test_zero_hugepage.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
lib/newlib_tests/test_zero_hugepage.sh +5 -1
@@ -1,12 +1,16 @@ #!/bin/sh # SPDX-License-Identifier: GPL-2.0-or-later # Copyright (c) 2021 Yang Xu <[email protected]> +# Copyright (c) 2021 Petr Vorel <[email protected]> echo "Testing .request_hugepages = TST_NO_HUGEPAGES" orig_value=`cat /proc/sys/vm/nr_hugepages` -echo "128" > /proc/sys/vm/nr_hugepages +if ! echo "128" > /proc/sys/vm/nr_hugepages; then + echo "TCONF: failed to open /proc/sys/vm/nr_hugepages" + exit 32 +fi ./test_zero_hugepage
Signature · 1 line
2.32.0
PATCH v7 2/7 test/test_zero_hugepage.sh: Skip test on read-only file system
Hi!
Quoted 6 lines
On GitHub Actions
/proc/sys/vm/nr_hugepagesis not allowed to be changed:./test_zero_hugepage.sh: line 9: can't create
/proc/sys/vm/nr_hugepages: Read-only file system tst_hugepage.c:57: TBROK: Failed to open FILE '/proc/sys/vm/nr_hugepages' for writing: EROFS (30) tst_sys_conf.c:102: TWARN: Failed to open FILE '/proc/sys/vm/nr_hugepages'
Hmm, it's rather strange to have proc mounted RO but I guess that we can't do much about it. I guess that it wouldn't allow us to remount it RW, so there is not much we can do about it.
Quoted 16 lines
Signed-off-by Petr Vorel <[email protected]>---
New in v7.lib/newlib_tests/test_zero_hugepage.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)lib/newlib_tests/test_zero_hugepage.sh +1
-0@@ -1,12 +1,16 @@ #!/bin/sh # SPDX-License-Identifier: GPL-2.0-or-later # Copyright (c) 2021 Yang Xu <[email protected]> +# Copyright (c) 2021 Petr Vorel <[email protected]>
lib/newlib_tests/test_zero_hugepage.sh line 13
Quoted 10 lines
lib/newlib_tests/test_zero_hugepage.sh +4
-1echo "Testing .request_hugepages = TST_NO_HUGEPAGES" orig_value=`cat /proc/sys/vm/nr_hugepages` -echo "128" > /proc/sys/vm/nr_hugepages +if ! echo "128" > /proc/sys/vm/nr_hugepages; then + echo "TCONF: failed to open /proc/sys/vm/nr_hugepages" + exit 32 +fi
I wonder if we should rather check if proc is mounted readonly explicitly since this may hide all kinds of errors.
Quoted 5 lines
lib/newlib_tests/test_zero_hugepage.sh +0
-0./test_zero_hugepageSignature · 1 line
2.32.0
Signature · 2 lines
Cyril Hrubis [email protected]
PATCH v7 2/7 test/test_zero_hugepage.sh: Skip test on read-only file system
Hi Cyril,
...
Quoted 2 lines
I wonder if we should rather check if proc is mounted readonly explicitly since this may hide all kinds of errors.
Sure, makes sense, I'll adapt it in v8. It's just not a fun to write tests which you cannot debug on VM, but just pushing to git :(.
Kind regards,
Petr
PATCH v7 2/7 test/test_zero_hugepage.sh: Skip test on read-only file system
Hi!
Quoted 4 lines
Quoted 2 lines
I wonder if we should rather check if proc is mounted readonly explicitly since this may hide all kinds of errors.
Sure, makes sense, I'll adapt it in v8. It's just not a fun to write tests which you cannot debug on VM, but just pushing to git :(.
Well I guess that you can remount proc RO on your testing machine with something as:
mount -o remount,ro /proc
Signature · 2 lines
Cyril Hrubis [email protected]
PATCH v7 2/7 test/test_zero_hugepage.sh: Skip test on read-only file system
Quoted 10 lines
Hi!
Quoted 4 lines
Quoted 2 lines
I wonder if we should rather check if proc is mounted readonly explicitly since this may hide all kinds of errors.
Sure, makes sense, I'll adapt it in v8. It's just not a fun to write tests which you cannot debug on VM, but just pushing to git :(.
Well I guess that you can remount proc RO on your testing machine with something as:
mount -o remount,ro /proc
Indeed one (very old one) of many of my VMs didn't yell "mount: /proc: mount point is busy." and actually allowed remount /proc to ro. Nice, thanks for a tip!
Kind regards,
Petr
PATCH v7 3/7 lib: Add script for running tests
In reply to Petr VorelFor now run only tests which TPASS or TCONF.
tst_fuzzy_sync01: sporadically fails, thus disabled:
../../include/tst_fuzzy_sync.h:685: TINFO: Exceeded execution loops, requesting exit
tst_fuzzy_sync01.c:227: TFAIL: acs:3 act:1 art:1 | =:23 -:46 +:2999931
...
Summary:
passed 21
failed 3
Redirect stderr to stdout to workaround mangled output on GitHub Actions where stdout and stderr are probably block buffered and flushed at different times.
NOTE: test_zero_hugepage.sh is testing C API, thus added in LTP_C_API_TESTS. Using shell tests also in C API tests i.e. test_zero_hugepage.sh (helpers to setup environment tests) requires to search file also in src directory (required for out-of-tree builds).
---
Changes v6->v7:
* have look on source directory, because we use scripts for running C API (test_zero_hugepage.sh), required for out-of-tree
lib/newlib_tests/runtest.sh | 190 ++++++++++++++++++++++++++++++++++++ 1 file changed, 190 insertions(+) create mode 100755 lib/newlib_tests/runtest.sh
lib/newlib_tests/runtest.sh new file +190 -0
@@ -0,0 +1,190 @@ +#!/bin/sh +# Copyright (c) 2021 Petr Vorel <[email protected]> + +LTP_C_API_TESTS="${LTP_C_API_TESTS:-test05 test07 test09 test12 test15 test18 +tst_bool_expr test_exec test_timer tst_res_hexd tst_strstatus tst_fuzzy_sync02 +tst_fuzzy_sync03 test_zero_hugepage.sh}" + +LTP_SHELL_API_TESTS="${LTP_SHELL_API_TESTS:-shell/tst_check_driver.sh shell/net/*.sh}" + +cd $(dirname $0) +PATH="$PWD/../../testcases/lib/:$PATH" + +. tst_ansi_color.sh + +usage() +{ + cat << EOF +Usage: $0 [-b DIR ] [-c|-s] +-b DIR build directory (required for out-of-tree build) +-c run C API tests only +-s run shell API tests only +-h print this help +EOF +} + +tst_flag2mask() +{ + case "$1" in + TPASS) return 0;; + TFAIL) return 1;; + TBROK) return 2;; + TWARN) return 4;; + TINFO) return 16;; + TCONF) return 32;; + esac +} + +runtest_res() +{ + if [ $# -eq 0 ]; then + echo >&2 + return + fi + + local res="$1" + shift + + tst_color_enabled + local color=$? + + printf "runtest " >&2 + tst_print_colored $res "$res: " >&2 + echo "$@" >&2 + +} + +runtest_brk() +{ + local res="$1" + shift + + tst_flag2mask "$res" + local mask=$? + + runtest_res + runtest_res $res $@ + + exit $mask +} + +run_tests() +{ + local target="$1" + local srcdir="$2" + local dir i res ret=0 tbrok tconf tfail tpass twarn vars + + eval vars="\$LTP_${target}_API_TESTS" + + runtest_res TINFO "=== Run $target tests ===" + + for i in $vars; do + runtest_res TINFO "* $i" + if [ -f "$i" ]; then + dir="." + elif [ "$srcdir" -a -f "$srcdir/$i" ]; then + dir="$srcdir" + else + runtest_brk TBROK "Error: $i file not found (PWD: $PWD)" + fi + + $dir/$i 1>&2 + res=$? + + [ $res -ne 0 -a $res -ne 32 ] && ret=1 + + case $res in + 0) tpass="$tpass $i";; + 1) tfail="$tfail $i";; + 2) tbrok="$tbrok $i";; + 4) twarn="$twarn $i";; + 32) tconf="$tconf $i";; + 127) runtest_brk TBROK "Error: file not found (wrong PATH? out-of-tree build without -b?), exit code: $res";; + *) runtest_brk TBROK "Error: unknown failure, exit code: $res";; + esac + runtest_res + done + + runtest_res TINFO "=== $target TEST RESULTS ===" + runtest_res TINFO "$(echo $tpass | wc -w)x TPASS:$tpass" + runtest_res TINFO "$(echo $tfail | wc -w)x TFAIL:$tfail" + runtest_res TINFO "$(echo $tbrok | wc -w)x TBROK:$tbrok" + runtest_res TINFO "$(echo $twarn | wc -w)x TWARN:$twarn" + runtest_res TINFO "$(echo $tconf | wc -w)x TCONF:$tconf" + runtest_res + + return $ret +} + +run_c_tests() +{ + local ret srcdir="$PWD" + + if [ "$builddir" ]; then + cd $builddir/lib/newlib_tests + fi + + run_tests "C" "$srcdir" + ret=$? + + if [ "$builddir" ]; then + cd "$srcdir" + fi + + return $ret +} + +run_shell_tests() +{ + run_tests "SHELL" +} + + +print_result() +{ + local target="$1" + local res="$2" + + + if [ -z "$res" ]; then + runtest_res TCONF "$target tests skipped" + elif [ $res -eq 0 ]; then + runtest_res TPASS "All $target tests TCONF/TPASS" + else + runtest_res TFAIL "Some $target test(s) TBROK/TFAIL/TWARN" + fi +} + +builddir= +c_fail= +run= +shell_fail= + +while getopts b:chs opt; do + case $opt in + 'h') usage; exit 0;; + 'b') builddir=$OPTARG; PATH="$builddir/testcases/lib:$PATH";; + 'c') run="c";; + 's') run="s";; + *) usage; runtest_brk TBROK "Error: invalid option";; + esac +done + +runtest_res TINFO "PATH='$PATH'" + +if [ -z "$run" -o "$run" = "c" ]; then + run_c_tests + c_fail=$? +fi + +if [ -z "$run" -o "$run" = "s" ]; then + run_shell_tests + shell_fail=$? +fi + +runtest_res TINFO "=== FINAL TEST RESULTS ===" + +print_result "C" "$c_fail" +print_result "shell" "$shell_fail" + +exit $((c_fail|shell_fail))
Signature · 1 line
2.32.0
PATCH v7 3/7 lib: Add script for running tests
PATCH v7 4/7 make: Add make test{, -c, -shell} targets
In reply to Petr VorelFor testing C and shell API.
---
The same as in v6.
Makefile | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
Makefile +23 -0
@@ -200,6 +200,29 @@ check: $(CHECK_TARGETS) ## Install install: $(INSTALL_TARGETS) +## Test +define _test + @set -e; $(top_srcdir)/lib/newlib_tests/runtest.sh -b $(abs_builddir) $(1) +endef + +test: lib-all +ifneq ($(build),$(host)) + $(error running tests on cross-compile build not supported) +endif + $(call _test) + +test-c: lib-all +ifneq ($(build),$(host)) + $(error running tests on cross-compile build not supported) +endif + $(call _test,-c) + +test-shell: lib-all +ifneq ($(build),$(host)) + $(error running tests on cross-compile build not supported) +endif + $(call _test,-s) + ## Help .PHONY: help help:
Signature · 1 line
2.32.0
PATCH v7 5/7 build.sh: Add support for make test{, -c, -shell}
In reply to Petr Vorel---
The same as in v6.
build.sh | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)
build.sh +23 -1
@@ -119,6 +119,17 @@ build_out_tree() make $MAKE_OPTS_OUT_TREE } +test_in_tree() +{ + make $1 +} + +test_out_tree() +{ + cd $BUILD_DIR + make $MAKE_OPTS_OUT_TREE $1 +} + install_in_tree() { make $MAKE_OPTS install
@@ -165,6 +176,9 @@ RUN: autotools run only 'make autotools' configure run only 'configure' build run only 'make' +test run only 'make test' (not supported for cross-compile build) +test-c run only 'make test-c' (not supported for cross-compile build) +test-shell run only 'make test-shell' (not supported for cross-compile build) install run only 'make install' Default configure options:
@@ -192,7 +206,7 @@ while getopts "c:hio:p:r:t:" opt; do esac;; p) prefix="$OPTARG";; r) case "$OPTARG" in - autotools|configure|build|install) run="$OPTARG";; + autotools|configure|build|test|test-c|test-shell|install) run="$OPTARG";; *) echo "Wrong run type '$OPTARG'" >&2; usage; exit 1;; esac;; t) case "$OPTARG" in
@@ -218,6 +232,14 @@ if [ -z "$run" -o "$run" = "build" ]; then eval build_${tree}_tree fi +if [ -z "$run" -o "$run" = "test" -o "$run" = "test-c" -o "$run" = "test-shell" ]; then + if [ "$build" = "cross" ]; then + echo "cross-compile build, skipping running tests" >&2 + else + eval test_${tree}_tree $run + fi +fi + if [ -z "$run" -o "$run" = "install" ]; then if [ "$install" = 1 ]; then eval install_${tree}_tree
Signature · 1 line
2.32.0
PATCH v7 6/7 CI: Run also make test-c, test-shell
In reply to Petr Vorelrun C and shell test separately (test-c and test-shell instead of test). Having them separate helps to easily navigate in CI.
It's run on all jobs, but on cross compile jobs it's obviously skipped.
---
The same as in v6.
.github/workflows/ci.yml | 10 ++++++++++ 1 file changed, 10 insertions(+)
.github/workflows/ci.yml +10 -0
@@ -151,6 +151,16 @@ jobs: - name: Compile run: ./build.sh -r build -o ${TREE:-in} + - name: Test C API + run: | + case "$VARIANT" in cross-compile*) BUILD="cross";; i386) BUILD="32";; *) BUILD="native";; esac + ./build.sh -r test-c -o ${TREE:-in} -t $BUILD + + - name: Test shell API + run: | + case "$VARIANT" in cross-compile*) BUILD="cross";; i386) BUILD="32";; *) BUILD="native";; esac + ./build.sh -r test-shell -o ${TREE:-in} -t $BUILD + - name: Install run: | if [ "$MAKE_INSTALL" = 1 ]; then INSTALL_OPT="-i"; fi
Signature · 1 line
2.32.0
PATCH v7 6/7 CI: Run also make test-c, test-shell
PATCH v7 7/7 ci: Install iproute2
In reply to Petr Vorelshell API test runtime dependency. To fix:
runtest TINFO: * shell/net/tst_rhost_run.shtst_rhost_run 1 TCONF: 'ip' not found
---
New in v7.
ci/debian.sh | 1 + ci/fedora.sh | 1 + ci/tumbleweed.sh | 1 + 3 files changed, 3 insertions(+)
ci/debian.sh +1 -0
@@ -22,6 +22,7 @@ $apt \ clang \ gcc \ git \ + iproute2 \ libacl1 \ libacl1-dev \ libaio-dev \
ci/fedora.sh +1 -0
@@ -13,6 +13,7 @@ $yum \ gcc \ git \ findutils \ + iproute \ numactl-devel \ libtirpc \ libtirpc-devel \
ci/tumbleweed.sh +1 -0
@@ -13,6 +13,7 @@ $zyp \ gcc \ git \ gzip \ + iproute2 \ make \ kernel-default-devel \ keyutils-devel \
Signature · 1 line
2.32.0
PATCH v7 7/7 ci: Install iproute2