shell: Introduce TST_TIMEOUT variable
v4 cover shell: Introduce TST_TIMEOUT variable
Hi,
changes v3->v4:
* add test runner test_timeout.sh, move tests into separate directory issues reported by Cyril:
* ceil LTP_TIMEOUT_MUL to int instead of complicating code with float
* Don't add LTP_TIMEOUT_MUL docs. This should be added to somewhere else, maybe to doc/user-guide.txt if [1] is accepted
* don't renumber sections
* move memcg_stress_test.sh change into separate commit
[1] https://patchwork.ozlabs.org/patch/1169282/
Petr Vorel (5):
shell: Add tst_is_num()
shell: Introduce TST_TIMEOUT variable, add checks
shell: Add timeout shell API tests memcg_stress_test.sh: use TST_TIMEOUT (replace LTP_TIMEOUT_MUL) net/if-mtu-change.sh: set TST_TIMEOUT
doc/test-writing-guidelines.txt | 14 ++++++- lib/newlib_tests/shell/test_timeout.sh | 38 ++++++++++++++++++ lib/newlib_tests/shell/timeout01.sh | 13 +++++++ lib/newlib_tests/shell/timeout02.sh | 13 +++++++ .../memcg/stress/memcg_stress_test.sh | 2 +- testcases/lib/tst_test.sh | 39 +++++++++++++++++-- .../network/stress/interface/if-mtu-change.sh | 4 +- 7 files changed, 116 insertions(+), 7 deletions(-) create mode 100755 lib/newlib_tests/shell/test_timeout.sh create mode 100755 lib/newlib_tests/shell/timeout01.sh create mode 100755 lib/newlib_tests/shell/timeout02.sh
Signature · 1 line
2.23.0
RFC PATCH v4 1/5 shell: Add tst_is_num()
Using grep -E, which more portable than using awk or anything else.
--- doc/test-writing-guidelines.txt | 6 ++++++ testcases/lib/tst_test.sh | 7 ++++++- 2 files changed, 12 insertions(+), 1 deletion(-)
doc/test-writing-guidelines.txt +6 -0
@@ -2338,6 +2338,12 @@ Checking for integers tst_is_int "$FOO" ------------------------------------------------------------------------------- +Checking for integers and floating point numbers +++++++++++++++++++++++++++++++++++++++++++++++++ +# returns zero if passed an integer or floating point number parameter, non-zero otherwise +tst_is_num "$FOO" +------------------------------------------------------------------------------- + Obtaining random numbers ++++++++++++++++++++++++
testcases/lib/tst_test.sh +6 -1
@@ -1,6 +1,6 @@ #!/bin/sh # SPDX-License-Identifier: GPL-2.0-or-later -# Copyright (c) Linux Test Project, 2014-2018 +# Copyright (c) Linux Test Project, 2014-2019 # Author: Cyril Hrubis <[email protected]> # # LTP test library for shell.
@@ -344,6 +344,11 @@ tst_is_int() return $? } +tst_is_num() +{ + echo "$1" | grep -Eq '^[-+]?[0-9]+\.?[0-9]*$' +} + tst_usage() { if [ -n "$TST_USAGE" ]; then
Signature · 1 line
2.23.0
RFC PATCH v4 2/5 shell: Introduce TST_TIMEOUT variable, add checks
In reply to Petr Vorelto unify shell API with C API.
TST_TIMEOUT should be set in tests only, it's equivalent of tst_test->timeout in C API.
Add checks requiring both TST_TIMEOUT and LTP_TIMEOUT_MUL >= 1, that allow to set TST_TIMEOUT lower than the default 300 sec (might be useful for some case).
LTP_TIMEOUT_MUL can be float, but will be ceiled to int. Using float would require awk/bc, which is unnecessary dependency and code complication (we do not care that much if it's multiplied precisely as far as the resulting timeout is never smaller than the precise calculation).
Also added cut dependency to _tst_setup_timer(), but that's not a problem as it was already required for shell API in tst_run().
--- doc/test-writing-guidelines.txt | 8 ++++++-- testcases/lib/tst_test.sh | 32 ++++++++++++++++++++++++++++++-- 2 files changed, 36 insertions(+), 4 deletions(-)
doc/test-writing-guidelines.txt +6 -2
@@ -2109,8 +2109,8 @@ tst_run '$TST_TEST_DATA' can be used with '$TST_CNT'. If '$TST_TEST_DATA_IFS' not specified, space as default value is used. Of course, it's possible to use separate functions. -2.3.2 Library variables -^^^^^^^^^^^^^^^^^^^^^^^ +2.3.2 Library environment variables for shell +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Similarily to the C library various checks and preparations can be requested simply by setting right '$TST_NEEDS_FOO'.
@@ -2126,6 +2126,10 @@ simply by setting right '$TST_NEEDS_FOO'. the test (see below). | 'TST_NEEDS_MODULE' | Test module name needed for the test (see below). | 'TST_NEEDS_DRIVERS'| Checks kernel drivers support for the test. +| 'TST_TIMEOUT' | Maximum timeout set for the test in sec. Must be int >= 1, + or -1 (special value to disable timeout), default is 300. + Variable is meant be set in tests, not by user. + It's equivalent of `tst_test.timeout` in C. |============================================================================= NOTE: Network tests (see testcases/network/README.md) use additional variables
testcases/lib/tst_test.sh +30 -2
@@ -379,9 +379,37 @@ _tst_rescmp() _tst_setup_timer() { + TST_TIMEOUT=${TST_TIMEOUT:-300} LTP_TIMEOUT_MUL=${LTP_TIMEOUT_MUL:-1} - local sec=$((300 * LTP_TIMEOUT_MUL)) + if [ "$TST_TIMEOUT" = -1 ]; then + tst_res TINFO "Timeout per run is disabled" + return + fi + + local err="LTP_TIMEOUT_MUL must be number >= 1!" + + tst_is_num "$LTP_TIMEOUT_MUL" || tst_brk TCONF "$err ($LTP_TIMEOUT_MUL)" + + if ! tst_is_int "$LTP_TIMEOUT_MUL"; then + tst_test_cmds cut + LTP_TIMEOUT_MUL=$(echo "$LTP_TIMEOUT_MUL" | cut -d. -f1) + LTP_TIMEOUT_MUL=$((LTP_TIMEOUT_MUL+1)) + tst_res TINFO "ceiling LTP_TIMEOUT_MUL to $LTP_TIMEOUT_MUL" + fi + [ "$LTP_TIMEOUT_MUL" -ge 1 ] || tst_brk TCONF "$err ($LTP_TIMEOUT_MUL)" + + if ! tst_is_int "$TST_TIMEOUT" || [ "$TST_TIMEOUT" -lt 1 ]; then + tst_brk TBROK "TST_TIMEOUT must be int >= 1! ($TST_TIMEOUT)" + fi + + local sec + if [ "$is_float" ]; then + sec=`echo | awk '{printf("%d\n", '$TST_TIMEOUT' * '$LTP_TIMEOUT_MUL'+ 0.5)}'` + else + sec=$((TST_TIMEOUT * LTP_TIMEOUT_MUL)) + fi + local h=$((sec / 3600)) local m=$((sec / 60 % 60)) local s=$((sec % 60))
@@ -418,7 +446,7 @@ tst_run() NEEDS_CMDS|NEEDS_MODULE|MODPATH|DATAROOT);; NEEDS_DRIVERS|FS_TYPE|MNTPOINT|MNT_PARAMS);; IPV6|IPVER|TEST_DATA|TEST_DATA_IFS);; - RETRY_FUNC|RETRY_FN_EXP_BACKOFF);; + RETRY_FUNC|RETRY_FN_EXP_BACKOFF|TIMEOUT);; NET_MAX_PKT);; *) tst_res TWARN "Reserved variable TST_$_tst_i used!";; esac
Signature · 1 line
2.23.0
RFC PATCH v4 2/5 shell: Introduce TST_TIMEOUT variable, add checks
Hi!
Quoted 7 lines
_tst_setup_timer()
{testcases/lib/tst_test.sh +2
-1+ TST_TIMEOUT=${TST_TIMEOUT:-300} LTP_TIMEOUT_MUL=${LTP_TIMEOUT_MUL:-1} - local sec=$((300 * LTP_TIMEOUT_MUL)) + if [ "$TST_TIMEOUT" = -1 ]; then
testcases/lib/tst_test.sh _tst_rescmp() · line 395
Quoted 10 lines
testcases/lib/tst_test.sh +10
-0+ tst_res TINFO "Timeout per run is disabled" + return + fi + + local err="LTP_TIMEOUT_MUL must be number >= 1!" + + tst_is_num "$LTP_TIMEOUT_MUL" || tst_brk TCONF "$err ($LTP_TIMEOUT_MUL)" + + if ! tst_is_int "$LTP_TIMEOUT_MUL"; then + tst_test_cmds cut
I do not think that it's necessary to check for the presense of the cut command.
Quoted 6 lines
testcases/lib/tst_test.sh +6
-0+ LTP_TIMEOUT_MUL=$(echo "$LTP_TIMEOUT_MUL" | cut -d. -f1) + LTP_TIMEOUT_MUL=$((LTP_TIMEOUT_MUL+1)) + tst_res TINFO "ceiling LTP_TIMEOUT_MUL to $LTP_TIMEOUT_MUL" + fi + [ "$LTP_TIMEOUT_MUL" -ge 1 ] || tst_brk TCONF "$err ($LTP_TIMEOUT_MUL)" +
testcases/lib/tst_test.sh _tst_rescmp() · line 411
Quoted 10 lines
testcases/lib/tst_test.sh +10
-0+ if ! tst_is_int "$TST_TIMEOUT" || [ "$TST_TIMEOUT" -lt 1 ]; then + tst_brk TBROK "TST_TIMEOUT must be int >= 1! ($TST_TIMEOUT)" + fi + + local sec + if [ "$is_float" ]; then + sec=`echo | awk '{printf("%d\n", '$TST_TIMEOUT' * '$LTP_TIMEOUT_MUL'+ 0.5)}'` + else + sec=$((TST_TIMEOUT * LTP_TIMEOUT_MUL)) + fi
Is this a leftover? Now that LTP_TIMEOUT_MUL has been converted to int we can simply do what is done in the else branch.
Quoted 15 lines
testcases/lib/tst_test.sh +1
-1local h=$((sec / 3600)) local m=$((sec / 60 % 60)) local s=$((sec % 60))@@ -418,7 +446,7 @@ tst_run() NEEDS_CMDS|NEEDS_MODULE|MODPATH|DATAROOT);; NEEDS_DRIVERS|FS_TYPE|MNTPOINT|MNT_PARAMS);; IPV6|IPVER|TEST_DATA|TEST_DATA_IFS);; - RETRY_FUNC|RETRY_FN_EXP_BACKOFF);; + RETRY_FUNC|RETRY_FN_EXP_BACKOFF|TIMEOUT);; NET_MAX_PKT);; *) tst_res TWARN "Reserved variable TST_$_tst_i used!";; esacSignature · 1 line
2.23.0
Signature · 2 lines
Cyril Hrubis [email protected]
RFC PATCH v4 2/5 shell: Introduce TST_TIMEOUT variable, add checks
Hi Cyril,
...
Quoted 5 lines
testcases/lib/tst_test.sh _tst_rescmp() · line 395
Quoted 2 lines
testcases/lib/tst_test.sh +2
-0+ if ! tst_is_int "$LTP_TIMEOUT_MUL"; then + tst_test_cmds cutI do not think that it's necessary to check for the presense of the cut command.
I wouldn't check for cut in tests, but in library I tend to be careful. But sure, I'll delete it.
BTW we already check for basic commands in tst_run():tst_test_cmds cut tr wc_tst_setup_timer() is called later.
Do you want to drop some of these checks?
Quoted 11 lines
Quoted 11 lines
testcases/lib/tst_test.sh +11
-0+ LTP_TIMEOUT_MUL=$(echo "$LTP_TIMEOUT_MUL" | cut -d. -f1) + LTP_TIMEOUT_MUL=$((LTP_TIMEOUT_MUL+1)) + tst_res TINFO "ceiling LTP_TIMEOUT_MUL to $LTP_TIMEOUT_MUL" + fi + [ "$LTP_TIMEOUT_MUL" -ge 1 ] || tst_brk TCONF "$err ($LTP_TIMEOUT_MUL)" + + if ! tst_is_int "$TST_TIMEOUT" || [ "$TST_TIMEOUT" -lt 1 ]; then + tst_brk TBROK "TST_TIMEOUT must be int >= 1! ($TST_TIMEOUT)" + fi + + local sec
Quoted 8 lines
testcases/lib/tst_test.sh _tst_rescmp() · line 411
Quoted 5 lines
testcases/lib/tst_test.sh +5
-0+ if [ "$is_float" ]; then + sec=`echo | awk '{printf("%d\n", '$TST_TIMEOUT' * '$LTP_TIMEOUT_MUL'+ 0.5)}'` + else + sec=$((TST_TIMEOUT * LTP_TIMEOUT_MUL)) + fiIs this a leftover? Now that
LTP_TIMEOUT_MULhas been converted to int we can simply do what is done in the else branch.
Yes, thanks for catching this.
Kind regards,
Petr
RFC PATCH v4 3/5 shell: Add timeout shell API tests
In reply to Petr Vorel--- lib/newlib_tests/shell/test_timeout.sh | 38 ++++++++++++++++++++++++++ lib/newlib_tests/shell/timeout01.sh | 13 +++++++++ lib/newlib_tests/shell/timeout02.sh | 13 +++++++++ 3 files changed, 64 insertions(+) create mode 100755 lib/newlib_tests/shell/test_timeout.sh create mode 100755 lib/newlib_tests/shell/timeout01.sh create mode 100755 lib/newlib_tests/shell/timeout02.sh
lib/newlib_tests/shell/test_timeout.sh new file +38 -0
@@ -0,0 +1,38 @@ +#!/bin/sh + +PATH="$(dirname $0)/../../../testcases/lib/:$PATH" + +DATA=" +timeout01.sh||0 +timeout02.sh||0 +timeout02.sh|foo|32 +timeout02.sh|2|0 +timeout02.sh|1.1|0 +timeout02.sh|-10|32 +" + +echo "Testing timeout in shell API" +echo + +failed=0 +IFS=" +" +for i in $DATA; do + file=$(echo $i | cut -d'|' -f1) + timeout=$(echo $i | cut -d'|' -f2) + exp_exit=$(echo $i | cut -d'|' -f3) + + echo "=== $test (LTP_TIMEOUT_MUL='$timeout') ===" + LTP_TIMEOUT_MUL=$timeout ./$file + ret=$? + if [ $ret -ne $exp_exit ]; then + echo "FAILED (exit code: $ret, expected $exp_exit)" + failed=$((failed+1)) + else + echo "PASSED" + fi + echo +done + +echo "Failed tests: $failed" +exit $failed
lib/newlib_tests/shell/timeout01.sh new file +13 -0
@@ -0,0 +1,13 @@ +#!/bin/sh + +TST_TESTFUNC=do_test + +TST_TIMEOUT=-1 +. tst_test.sh + +do_test() +{ + tst_res TPASS "timeout $TST_TIMEOUT set" +} + +tst_run
lib/newlib_tests/shell/timeout02.sh new file +13 -0
@@ -0,0 +1,13 @@ +#!/bin/sh + +TST_TESTFUNC=do_test + +TST_TIMEOUT=2 +. tst_test.sh + +do_test() +{ + tst_res TPASS "timeout $TST_TIMEOUT set (LTP_TIMEOUT_MUL='$LTP_TIMEOUT_MUL')" +} + +tst_run
Signature · 1 line
2.23.0
RFC PATCH v4 3/5 shell: Add timeout shell API tests
Hi,
Quoted 16 lines
Signed-off-by Petr Vorel <[email protected]>--- lib/newlib_tests/shell/test_timeout.sh | 38 ++++++++++++++++++++++++++ lib/newlib_tests/shell/timeout01.sh | 13 +++++++++ lib/newlib_tests/shell/timeout02.sh | 13 +++++++++ 3 files changed, 64 insertions(+) create mode 100755 lib/newlib_tests/shell/test_timeout.sh create mode 100755 lib/newlib_tests/shell/timeout01.sh create mode 100755 lib/newlib_tests/shell/timeout02.shlib/newlib_tests/shell/test_timeout.sh new file +0
-0
lib/newlib_tests/shell/test_timeout.sh line 3
Quoted 3 lines
lib/newlib_tests/shell/test_timeout.sh new file +3
-0@@ -0,0 +1,38 @@ +#!/bin/sh + +PATH="$(dirname $0)/../../../testcases/lib/:$PATH"
I put PATH here, as it's the only runner and not prolong this fix. If we want more sophisticated functionality from these test of the library itself runners, we could create some common.sh library here, which would set PATH and do other things (counting failures and report them etc).
Quoted 35 lines
lib/newlib_tests/shell/test_timeout.sh new file +35
-0+ +DATA=" +timeout01.sh||0 +timeout02.sh||0 +timeout02.sh|foo|32 +timeout02.sh|2|0 +timeout02.sh|1.1|0 +timeout02.sh|-10|32 +" + +echo "Testing timeout in shell API" +echo + +failed=0 +IFS=" +" +for i in $DATA; do + file=$(echo $i | cut -d'|' -f1) + timeout=$(echo $i | cut -d'|' -f2) + exp_exit=$(echo $i | cut -d'|' -f3) + + echo "=== $test (LTP_TIMEOUT_MUL='$timeout') ===" + LTP_TIMEOUT_MUL=$timeout ./$file + ret=$? + if [ $ret -ne $exp_exit ]; then + echo "FAILED (exit code: $ret, expected $exp_exit)" + failed=$((failed+1)) + else + echo "PASSED" + fi + echo +done + +echo "Failed tests: $failed" +exit $failed
RFC PATCH v4 3/5 shell: Add timeout shell API tests
In reply to Petr VorelHi!
Quoted 5 lines
lib/newlib_tests/shell/test_timeout.sh new file +5
-0+DATA=" +timeout01.sh||0 +timeout02.sh||0 +timeout02.sh|foo|32 +timeout02.sh|2|0
lib/newlib_tests/shell/test_timeout.sh line 19
Quoted 10 lines
lib/newlib_tests/shell/test_timeout.sh new file +10
-0+timeout02.sh|1.1|0 +timeout02.sh|-10|32 +" + +echo "Testing timeout in shell API" +echo + +failed=0 +IFS=" +"
Do we have to change IFS here? It should be set to space, tab, and newline by default, so the loop over $DATA should work correctly anyways.
Quoted 60 lines
lib/newlib_tests/shell/test_timeout.sh new file +19
-0+for i in $DATA; do + file=$(echo $i | cut -d'|' -f1) + timeout=$(echo $i | cut -d'|' -f2) + exp_exit=$(echo $i | cut -d'|' -f3) + + echo "=== $test (LTP_TIMEOUT_MUL='$timeout') ===" + LTP_TIMEOUT_MUL=$timeout ./$file + ret=$? + if [ $ret -ne $exp_exit ]; then + echo "FAILED (exit code: $ret, expected $exp_exit)" + failed=$((failed+1)) + else + echo "PASSED" + fi + echo +done + +echo "Failed tests: $failed" +exit $failedlib/newlib_tests/shell/timeout01.sh new file +13
-0@@ -0,0 +1,13 @@ +#!/bin/sh + +TST_TESTFUNC=do_test + +TST_TIMEOUT=-1 +. tst_test.sh + +do_test() +{ + tst_res TPASS "timeout $TST_TIMEOUT set" +} + +tst_runlib/newlib_tests/shell/timeout02.sh new file +13
-0@@ -0,0 +1,13 @@ +#!/bin/sh + +TST_TESTFUNC=do_test + +TST_TIMEOUT=2 +. tst_test.sh + +do_test() +{ + tst_res TPASS "timeout $TST_TIMEOUT set (LTP_TIMEOUT_MUL='$LTP_TIMEOUT_MUL')" +} + +tst_runSignature · 1 line
2.23.0
Signature · 2 lines
Cyril Hrubis [email protected]
RFC PATCH v4 3/5 shell: Add timeout shell API tests
Hi Cyril,
Quoted 11 lines
Quoted 11 lines
lib/newlib_tests/shell/test_timeout.sh new file +11
-0+DATA=" +timeout01.sh||0 +timeout02.sh||0 +timeout02.sh|foo|32 +timeout02.sh|2|0 +timeout02.sh|1.1|0 +timeout02.sh|-10|32 +" + +echo "Testing timeout in shell API" +echo
Quoted 8 lines
lib/newlib_tests/shell/test_timeout.sh line 19
Quoted 4 lines
lib/newlib_tests/shell/test_timeout.sh new file +4
-0+ +failed=0 +IFS=" +"Do we have to change IFS here? It should be set to space, tab, and newline by default, so the loop over $DATA should work correctly anyways.
Again, thanks for catching this.
It's a left over from prototyping, where I had spaces in some data rows, therefore it was needed.
Quoted 4 lines
lib/newlib_tests/shell/test_timeout.sh line 23
Quoted 4 lines
lib/newlib_tests/shell/test_timeout.sh new file +4
-0+for i in $DATA; do + file=$(echo $i | cut -d'|' -f1) + timeout=$(echo $i | cut -d'|' -f2) + exp_exit=$(echo $i | cut -d'|' -f3)
Unless you want me to do else with tst_test_cmds in tst_test.sh, I'll merge with changes you found.
Kind regards,
Petr
RFC PATCH v4 4/5 memcg_stress_test.sh: use TST_TIMEOUT (replace LTP_TIMEOUT_MUL)
In reply to Petr Vorelkeeping the same timeout
--- testcases/kernel/controllers/memcg/stress/memcg_stress_test.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
testcases/kernel/controllers/memcg/stress/memcg_stress_test.sh +1 -1
@@ -17,7 +17,7 @@ TST_NEEDS_CMDS="mount umount cat kill mkdir rmdir grep awk cut" # Each test case runs for 900 secs when everything fine # therefore the default 5 mins timeout is not enough. -LTP_TIMEOUT_MUL=7 +TST_TIMEOUT=2100 . cgroup_lib.sh
Signature · 1 line
2.23.0
RFC PATCH v4 5/5 net/if-mtu-change.sh: set TST_TIMEOUT
In reply to Petr VorelThe default setup (100 * 5 seconds just for sleep) is longer than default timeout. 30 sec for each iteration should be more than enough as a default.
--- testcases/network/stress/interface/if-mtu-change.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
testcases/network/stress/interface/if-mtu-change.sh +3 -1
@@ -1,6 +1,6 @@ #!/bin/sh # SPDX-License-Identifier: GPL-2.0-or-later -# Copyright (c) 2017-2018 Petr Vorel <[email protected]> +# Copyright (c) 2017-2019 Petr Vorel <[email protected]> # Copyright (c) 2015-2017 Oracle and/or its affiliates. All Rights Reserved. # Copyright (c) International Business Machines Corp., 2005 # Author: Mitsuru Chinen <[email protected]>
@@ -13,6 +13,8 @@ TST_CLEANUP="do_cleanup" # The interval of the mtu change [second] CHANGE_INTERVAL=${CHANGE_INTERVAL:-5} +TST_TIMEOUT=$(((CHANGE_INTERVAL + 30) * MTU_CHANGE_TIMES)) + # The array of the value which MTU is changed into sequentially # 552 - net.ipv4.route.min_pmtu CHANGE_VALUES="784 1142 552 1500 552 1500 552 748 552 1142 1500"
Signature · 1 line
2.23.0