Skip to contentSkip to conversation

Run tests in CI

PATCH Superseded Petr Vorel 2 participants 12 messages Jul 14, 2021 – Jul 15, 2021

Subsystems (2)

v5 of 8 Superseded Latest is v8 →

lore
Share & tools
lore

Petr Vorel

author
#0 lore raw

v5 cover Run tests in CI

Hi,

changes v4->v5:
* don't exit on test failure, but continue testing (Cyril)
* rename tst_{brk,res}() to runtest_{brk,res}() (Cyril)
* rewritten summary, put everything to the end
* merged 3 already acked commits

$ make test
...

runtest TINFO: === C TEST RESULTS ===
runtest TINFO: 13x TPASS: test05 test07 test09 test12 test15 test18 tst_bool_expr test_exec test_timer tst_res_hexd tst_strstatus tst_fuzzy_sync02 tst_fuzzy_sync03 runtest TINFO: 0x TFAIL:
runtest TINFO: 0x TBROK:
runtest TINFO: 0x TWARN:
runtest TINFO: 0x TCONF:

...

runtest TINFO: === SHELL TEST RESULTS ===
runtest TINFO: 2x TPASS: shell/tst_check_driver.sh shell/net/tst_ipaddr_un.sh runtest TINFO: 0x TFAIL:
runtest TINFO: 0x TBROK:
runtest TINFO: 0x TWARN:
runtest TINFO: 1x TCONF: shell/net/tst_rhost_run.sh

runtest TINFO: === FINAL TEST RESULTS === runtest TPASS: All C tests TCONF/TPASS runtest TPASS: All shell tests TCONF/TPASS

Petr Vorel (4):
lib: Add script for running tests
make: Add make test{, -c, -shell} targets build.sh: Add support for make test
CI: Run also make test

 .github/workflows/ci.yml    |   5 ++
 Makefile                    |  23 +++++
 build.sh                    |  22 ++++-
 lib/newlib_tests/runtest.sh | 175 ++++++++++++++++++++++++++++++++++++
 4 files changed, 224 insertions(+), 1 deletion(-)
 create mode 100755 lib/newlib_tests/runtest.sh
Signature · 1 line
2.32.0
author
#1 lore raw

PATCH v5 1/4 lib: Add script for running tests

For now run only tests which TPASS or TCONF.

tst_fuzzy_sync01: sporadically fails, thus disabled:

../../include/tst_fuzzy_sync.h:685: TINFO: Exceeded execution loops, requesting exit
tst_fuzzy_sync01.c:227: TFAIL: acs:3 act:1 art:1 | =:23 -:46 +:2999931
...
Summary:
passed 21
failed 3

---
 lib/newlib_tests/runtest.sh | 175 ++++++++++++++++++++++++++++++++++++
 1 file changed, 175 insertions(+)
 create mode 100755 lib/newlib_tests/runtest.sh
lib/newlib_tests/runtest.sh new file +175 -0
new file mode 100755
index 000000000..207553aaa
@@ -0,0 +1,175 @@
+#!/bin/sh
+# Copyright (c) 2021 Petr Vorel <[email protected]>
+
+LTP_C_API_TESTS="${LTP_C_API_TESTS:-test05 test07 test09 test12 test15 test18
+tst_bool_expr test_exec test_timer tst_res_hexd tst_strstatus tst_fuzzy_sync02
+tst_fuzzy_sync03}"
+
+LTP_SHELL_API_TESTS="${LTP_SHELL_API_TESTS:-shell/tst_check_driver.sh shell/net/*.sh}"
+
+cd $(dirname $0)
+PATH="$PWD/../../testcases/lib/:$PATH"
+
+. tst_ansi_color.sh
+
+usage()
+{
+	cat << EOF
+Usage: $0 [-b DIR ] [-c|-s]
+-b DIR  build directory (required for out-of-tree build)
+-c      run C API tests only
+-s      run shell API tests only
+-h      print this help
+EOF
+}
+
+tst_flag2mask()
+{
+	case "$1" in
+	TPASS) return 0;;
+	TFAIL) return 1;;
+	TBROK) return 2;;
+	TWARN) return 4;;
+	TINFO) return 16;;
+	TCONF) return 32;;
+	esac
+}
+
+runtest_res()
+{
+	if [ $# -eq 0 ]; then
+		echo >&2
+		return
+	fi
+
+	local res="$1"
+	shift
+
+	tst_color_enabled
+	local color=$?
+
+	printf "runtest " >&2
+	tst_print_colored $res "$res: " >&2
+	echo "$@" >&2
+
+}
+
+runtest_brk()
+{
+	local res="$1"
+	shift
+
+	tst_flag2mask "$res"
+	local mask=$?
+
+	runtest_res
+	runtest_res $res $@
+
+	exit $mask
+}
+
+run_tests()
+{
+	local target="$1"
+	local i res ret=0 tbrok tconf tfail tpass twarn vars
+
+	eval vars="\$LTP_${target}_API_TESTS"
+
+	runtest_res TINFO "=== Run $target tests ==="
+
+	for i in $vars; do
+		runtest_res TINFO "* $i"
+		./$i
+		res=$?
+
+		[ $res -ne 0 -a $res -ne 32 ] && ret=1
+
+		case $res in
+			0) tpass="$tpass $i";;
+			1) tfail="$tfail $i";;
+			2) tbrok="$tbrok $i";;
+			4) twarn="$twarn $i";;
+			32) tconf="$tconf $i";;
+			127) runtest_brk TBROK "Error: file not found (wrong PATH? out-of-tree build without -b?), exit code: $res";;
+			*) runtest_brk TBROK "Error: unknown failure, exit code: $res";;
+		esac
+		runtest_res
+	done
+
+	runtest_res TINFO "=== $target TEST RESULTS ==="
+	runtest_res TINFO "$(echo $tpass | wc -w)x TPASS:$tpass"
+	runtest_res TINFO "$(echo $tfail | wc -w)x TFAIL:$tfail"
+	runtest_res TINFO "$(echo $tbrok | wc -w)x TBROK:$tbrok"
+	runtest_res TINFO "$(echo $twarn | wc -w)x TWARN:$twarn"
+	runtest_res TINFO "$(echo $tconf | wc -w)x TCONF:$tconf"
+	runtest_res
+
+	return $ret
+}
+
+run_c_tests()
+{
+	if [ "$builddir" ]; then
+		cd $builddir/lib/newlib_tests
+	fi
+
+	run_tests "C"
+
+	if [ "$builddir" ]; then
+		cd -
+	fi
+}
+
+run_shell_tests()
+{
+	run_tests "SHELL"
+}
+
+
+print_result()
+{
+	local res="$1"
+	local target="$2"
+
+	if [ -z "$res" ]; then
+		runtest_res TCONF "$target tests skipped"
+	elif [ $res -eq 0 ]; then
+		runtest_res TPASS "All $target tests TCONF/TPASS"
+	else
+		runtest_res TFAIL "Some $target test(s) TBROK/TFAIL/TWARN"
+	fi
+}
+
+builddir=
+c_fail=
+run=
+shell_fail=
+
+while getopts b:chs opt; do
+	case $opt in
+		'h') usage; exit 0;;
+		'b') builddir=$OPTARG; PATH="$builddir/testcases/lib:$PATH";;
+		'c') run="c";;
+		's') run="s";;
+		*) usage; runtest_brk TBROK "Error: invalid option";;
+	esac
+done
+
+runtest_res TINFO "PATH='$PATH'"
+
+if [ -z "$run" -o "$run" = "c" ]; then
+	run_c_tests
+	c_fail=$?
+fi
+
+if [ -z "$run" -o "$run" = "s" ]; then
+	run_shell_tests
+	shell_fail=$?
+fi
+
+runtest_res TINFO "=== FINAL TEST RESULTS ==="
+
+print_result "$c_fail" "C"
+print_result "$shell_fail" "shell"
+
+exit $((c_fail|shell_fail))
Signature · 1 line
2.32.0
author
#2 lore raw

PATCH v5 2/4 make: Add make test{, -c, -shell} targets

In reply to Petr Vorel

For testing C and shell API.

---
 Makefile | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
Makefile +23 -0
index 56812d77b..ff4a70eec 100644
@@ -192,6 +192,29 @@ $(INSTALL_TARGETS): $(INSTALL_DIR) $(DESTDIR)/$(bindir)
 ## Install
 install: $(INSTALL_TARGETS)
 
+## Test
+define _test
+	@set -e; $(top_srcdir)/lib/newlib_tests/runtest.sh -b $(abs_builddir) $(1)
+endef
+
+test: lib-all
+ifneq ($(build),$(host))
+	$(error running tests on cross-compile build not supported)
+endif
+	$(call _test)
+
+test-c: lib-all
+ifneq ($(build),$(host))
+	$(error running tests on cross-compile build not supported)
+endif
+	$(call _test,-c)
+
+test-shell: lib-all
+ifneq ($(build),$(host))
+	$(error running tests on cross-compile build not supported)
+endif
+	$(call _test,-s)
+
 ## Help
 .PHONY: help
 help:
Signature · 1 line
2.32.0
author
#3 lore raw

PATCH v5 3/4 build.sh: Add support for make test

In reply to Petr Vorel
---
 build.sh | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)
build.sh +21 -1
index 240ce8e68..9cb26e899 100755
@@ -119,6 +119,17 @@ build_out_tree()
 	make $MAKE_OPTS_OUT_TREE
 }
 
+test_in_tree()
+{
+	make test
+}
+
+test_out_tree()
+{
+	cd $BUILD_DIR
+	make $MAKE_OPTS_OUT_TREE test
+}
+
 install_in_tree()
 {
 	make $MAKE_OPTS install
@@ -165,6 +176,7 @@ RUN:
 autotools   run only 'make autotools'
 configure   run only 'configure'
 build       run only 'make'
+test        run only 'make test' (not supported for cross-compile build)
 install     run only 'make install'
 
 Default configure options:
@@ -192,7 +204,7 @@ while getopts "c:hio:p:r:t:" opt; do
 		esac;;
 	p) prefix="$OPTARG";;
 	r) case "$OPTARG" in
-		autotools|configure|build|install) run="$OPTARG";;
+		autotools|configure|build|test|install) run="$OPTARG";;
 		*) echo "Wrong run type '$OPTARG'" >&2; usage; exit 1;;
 		esac;;
 	t) case "$OPTARG" in
@@ -218,6 +230,14 @@ if [ -z "$run" -o "$run" = "build" ]; then
 	eval build_${tree}_tree
 fi
 
+if [ -z "$run" -o "$run" = "test" ]; then
+	if [ "$build" = "cross" ]; then
+		echo "cross-compile build, skipping running tests" >&2
+	else
+		eval test_${tree}_tree
+	fi
+fi
+
 if [ -z "$run" -o "$run" = "install" ]; then
 	if [ "$install" = 1 ]; then
 		eval install_${tree}_tree
Signature · 1 line
2.32.0
author
#4 lore raw

PATCH v5 4/4 CI: Run also make test

In reply to Petr Vorel

on all targets

---
 .github/workflows/ci.yml | 5 +++++
 1 file changed, 5 insertions(+)
.github/workflows/ci.yml +5 -0
index f67f14927..776dbf646 100644
@@ -151,6 +151,11 @@ jobs:
     - name: Compile
       run: ./build.sh -r build -o ${TREE:-in}
 
+    - name: Test
+      run: |
+        case "$VARIANT" in cross-compile*) BUILD="cross";; i386) BUILD="32";; *) BUILD="native";; esac
+        ./build.sh -r test -o ${TREE:-in} -t $BUILD
+
     - name: Install
       run: |
         if [ "$MAKE_INSTALL" = 1 ]; then INSTALL_OPT="-i"; fi
Signature · 1 line
2.32.0
author
#5 lore raw

v5 cover Run tests in CI

In reply to Petr Vorel

Hi all,

tested:
https://github.com/pevik/ltp/actions/runs/1030551185

One thing which bothers me is mangled output of several tests together:
https://github.com/pevik/ltp/runs/3067600696?check_suite_focus=true#step:11:554

runtest TINFO: * tst_res_hexd
/__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: dump1 /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 48 65 6c 6c 6f 20 66 72 6f 6d 20 74 73 74 5f 72 /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 65 73 5f 68 65 78 64 00

runtest TINFO: * tst_strstatus /__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s

Summary:

passed   35
failed   0
broken   0
skipped  0

warnings 0

Summary:

passed   1
failed   0
broken   0
skipped  0

warnings 0

Kind regards,
Petr

#6 lore raw

v5 cover Run tests in CI

Hi!

Quoted 20 lines

tested:
https://github.com/pevik/ltp/actions/runs/1030551185

One thing which bothers me is mangled output of several tests together:
https://github.com/pevik/ltp/runs/3067600696?check_suite_focus=true#step:11:554

runtest TINFO: * tst_res_hexd
/__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: dump1 /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 48 65 6c 6c 6f 20 66 72 6f 6d 20 74 73 74 5f 72 /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 65 73 5f 68 65 78 64 00

runtest TINFO: * tst_strstatus /__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s

Summary:

passed   35
failed   0
broken   0
skipped  0
Quoted 8 lines

warnings 0

Summary:

passed   1
failed   0
broken   0
skipped  0

warnings 0

That is indeed strange, you are running these in a loop, the buffers should have been flushed once controll returns back to the shell from the test.

I guess that this is a combination of:

* The stdout and stderr are block buffered by the gitlab CI and flushed at different times

* The summary is written into stdout in contrast with the message so we end up with summary printed later than the rest that goes into stderr

I guess this should fix it:

lib/tst_test.c +6 -6
index f4d9f8e3b..084a83c9e 100644
@@ -734,12 +734,12 @@ static void do_exit(int ret)
                if (results->broken)
                        ret |= TBROK;
 
-               printf("\nSummary:\n");
-               printf("passed   %d\n", results->passed);
-               printf("failed   %d\n", results->failed);
-               printf("broken   %d\n", results->broken);
-               printf("skipped  %d\n", results->skipped);
-               printf("warnings %d\n", results->warnings);
+               fprintf(stderr, "\nSummary:\n");
+               fprintf(stderr, "passed   %d\n", results->passed);
+               fprintf(stderr, "failed   %d\n", results->failed);
+               fprintf(stderr, "broken   %d\n", results->broken);
+               fprintf(stderr, "skipped  %d\n", results->skipped);
+               fprintf(stderr, "warnings %d\n", results->warnings);
        }
 
        do_cleanup();
Signature · 2 lines
Cyril Hrubis
[email protected]
author
#7 lore raw

v5 cover Run tests in CI

Hi Cyril,

Quoted 34 lines

Hi!

Quoted 20 lines

tested:
https://github.com/pevik/ltp/actions/runs/1030551185

One thing which bothers me is mangled output of several tests together:
https://github.com/pevik/ltp/runs/3067600696?check_suite_focus=true#step:11:554

runtest TINFO: * tst_res_hexd
/__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: dump1 /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 48 65 6c 6c 6f 20 66 72 6f 6d 20 74 73 74 5f 72 /__w/ltp/ltp/lib/newlib_tests/tst_res_hexd.c:13: TPASS: 65 73 5f 68 65 78 64 00

runtest TINFO: * tst_strstatus /__w/ltp/ltp/lib/tst_test.c:1344: TINFO: Timeout per run is 0h 05m 00s

Summary:

passed   35
failed   0
broken   0
skipped  0
Quoted 8 lines

warnings 0

Summary:

passed   1
failed   0
broken   0
skipped  0

warnings 0

That is indeed strange, you are running these in a loop, the buffers should have been flushed once controll returns back to the shell from the test.

Quoted 8 lines

I guess that this is a combination of:

* The stdout and stderr are block buffered by the gitlab CI and flushed at different times

* The summary is written into stdout in contrast with the message so we end up with summary printed later than the rest that goes into stderr

Yep, it could be.

Quoted 13 lines

I guess this should fix it:

lib/tst_test.c +0 -3
index f4d9f8e3b..084a83c9e 100644
@@ -734,12 +734,12 @@ static void do_exit(int ret)
                if (results->broken)
                        ret |= TBROK;
 
-               printf("\nSummary:\n");
-               printf("passed   %d\n", results->passed);
-               printf("failed   %d\n", results->failed);

lib/tst_test.c static void do_exit(int ret) · line 743

Quoted 10 lines
lib/tst_test.c +6 -3
-               printf("broken   %d\n", results->broken);
-               printf("skipped  %d\n", results->skipped);
-               printf("warnings %d\n", results->warnings);
+               fprintf(stderr, "\nSummary:\n");
+               fprintf(stderr, "passed   %d\n", results->passed);
+               fprintf(stderr, "failed   %d\n", results->failed);
+               fprintf(stderr, "broken   %d\n", results->broken);
+               fprintf(stderr, "skipped  %d\n", results->skipped);
+               fprintf(stderr, "warnings %d\n", results->warnings);
        }

Thanks! Testing it in https://github.com/pevik/ltp/actions/runs/1031005963

Kind regards,
Petr

author
#8 lore raw

v5 cover Run tests in CI

Hi Cyril,

...

Quoted 24 lines
Quoted 13 lines

I guess this should fix it:

lib/tst_test.c +0 -3
index f4d9f8e3b..084a83c9e 100644
@@ -734,12 +734,12 @@ static void do_exit(int ret)
                if (results->broken)
                        ret |= TBROK;
 
-               printf("\nSummary:\n");
-               printf("passed   %d\n", results->passed);
-               printf("failed   %d\n", results->failed);

lib/tst_test.c static void do_exit(int ret) · line 743

Quoted 10 lines
lib/tst_test.c +6 -3
-               printf("broken   %d\n", results->broken);
-               printf("skipped  %d\n", results->skipped);
-               printf("warnings %d\n", results->warnings);
+               fprintf(stderr, "\nSummary:\n");
+               fprintf(stderr, "passed   %d\n", results->passed);
+               fprintf(stderr, "failed   %d\n", results->failed);
+               fprintf(stderr, "broken   %d\n", results->broken);
+               fprintf(stderr, "skipped  %d\n", results->skipped);
+               fprintf(stderr, "warnings %d\n", results->warnings);
        }
Quoted 1 line

Thanks! Testing it in https://github.com/pevik/ltp/actions/runs/1031005963

The result is much better but still not correct:
https://github.com/pevik/ltp/runs/3068932859?check_suite_focus=true#step:11:707

runtest TINFO: * shell/net/tst_ipaddr_un.sh

Summary:

passed   0
failed   0
broken   0
skipped  1

warnings 0
tst_ipaddr_un 1 TINFO: timeout per run is 0h 5m 0s

Could you please merge your fix? And I'll try sync after each test.
https://github.com/pevik/ltp/actions/runs/1033070696

What is worse is that tests are green, but I see TBROK and TWARN which are not reflected by results:

tst_check_driver 1 TBROK: Test still running, sending SIGKILL
...
tst_rhost_run 1 TCONF: (null) 0 TWARN : tst_kernel.c:105: expected file /lib/modules/5.8.0-1036-azure/modules.dep does not exist or not a file

veth(null) 0 TWARN : tst_kernel.c:105: expected file /lib/modules/5.8.0-1036-azure/modules.builtin does not exist or not a file driver not available
Summary:

passed   0
failed   0
broken   0
skipped  1

warnings 0

runtest TINFO: === SHELL TEST RESULTS ===
runtest TINFO: 1x TPASS: shell/net/tst_ipaddr_un.sh
runtest TINFO: 0x TFAIL:
runtest TINFO: 0x TBROK:
runtest TINFO: 0x TWARN:
runtest TINFO: 2x TCONF: shell/tst_check_driver.sh shell/net/tst_rhost_run.sh

Kind regards,
Petr

Quoted 2 lines

Kind regards,
Petr

author
#9 lore raw

v5 cover Run tests in CI

Hi Cyril,

...

Quoted 12 lines
Quoted 1 line

Thanks! Testing it in https://github.com/pevik/ltp/actions/runs/1031005963

The result is much better but still not correct:
https://github.com/pevik/ltp/runs/3068932859?check_suite_focus=true#step:11:707

runtest TINFO: * shell/net/tst_ipaddr_un.sh

Summary:

passed   0
failed   0
broken   0
skipped  1
Quoted 5 lines

warnings 0
tst_ipaddr_un 1 TINFO: timeout per run is 0h 5m 0s

Could you please merge your fix? And I'll try sync after each test.
https://github.com/pevik/ltp/actions/runs/1033070696

There is still some output printed later, at least In RPN: A B | C ! ! &
In RPN: A B | C ! ! &
which is from tst_bool_expr.c, but printed after tst_strstatus
https://github.com/pevik/ltp/runs/3074335940?check_suite_focus=true#step:11:587
But it's much better.

I also found a bug in final evaluation on one of my VM:

tst_test.c:1313: TBROK: Test killed by SIGSEGV!
...
runtest TINFO: 1x TBROK: tst_bool_expr
...
runtest TINFO: === FINAL TEST RESULTS === runtest TPASS: All C tests TCONF/TPASS <- THIS IS WRONG runtest TPASS: All shell tests TCONF/TPASS

Kind regards,
Petr

#10 lore raw

v5 cover Run tests in CI

In reply to Petr Vorel

Hi!

Quoted 4 lines
Quoted 1 line

Thanks! Testing it in https://github.com/pevik/ltp/actions/runs/1031005963

The result is much better but still not correct:
https://github.com/pevik/ltp/runs/3068932859?check_suite_focus=true#step:11:707

If tests are printing into stdout as well it will be easier to redirect the test output in the script so that the CI gets it all into a single file descriptor.

What about doing ./$i 1>&2 in the loop that runs the tests instead?

Signature · 2 lines
Cyril Hrubis
[email protected]
author
#11 lore raw

v5 cover Run tests in CI

Quoted 11 lines

Hi!

Quoted 4 lines
Quoted 1 line

Thanks! Testing it in https://github.com/pevik/ltp/actions/runs/1031005963

The result is much better but still not correct:
https://github.com/pevik/ltp/runs/3068932859?check_suite_focus=true#step:11:707

If tests are printing into stdout as well it will be easier to redirect the test output in the script so that the CI gets it all into a single file descriptor.

What about doing ./$i 1>&2 in the loop that runs the tests instead?

How simple, that should work, I'll test that.

FYI I also need to install iproute (runtime shell API test dependency). Simple enough to just merge it after this gets merged.