tst_test.sh: Add TST_TEST_DATA and TST_TEST_DATA_IFS
RFC PATCH v3 1/2 tst_test.sh: Add TST_TEST_DATA and TST_TEST_DATA_IFS
This is specific only for shell.
Each run of tst_run gets passed sequence number of a test being run as '$1' and corresponding part of data from TST_TEST_DATA as '$2'.
Also create internal functions tst_run_tests() and tst_run_test() to reduce duplicity.
---
Changes v2->v3:
* Don't pass $TST_TEST_DATA as whole argument for $TST_CNT (Cyril)
* Create tst_run_tests() for cleanup (Cyril).
* Create tst_run_test() - more cleanup
Maybe we should try to "hide" somehow these API functions (underscore at the front?).
Kind regards,
Petr
--- doc/test-writing-guidelines.txt | 74 +++++++++++++++++++++++++++++++++++++---- testcases/lib/tst_test.sh | 61 +++++++++++++++++++++------------ 2 files changed, 106 insertions(+), 29 deletions(-)
doc/test-writing-guidelines.txt +67 -7
@@ -1442,12 +1442,12 @@ TST_CNT=2 test1() { - tst_res TPASS "Test 1 passed" + tst_res TPASS "Test $1 passed" } test2() { - tst_res TPASS "Test 2 passed" + tst_res TPASS "Test $1 passed" } tst_run
@@ -1455,7 +1455,8 @@ tst_run If '$TST_CNT' is set, the test library looks if there are functions named '$\{TST_TESTFUNC\}1', ..., '$\{TST_TESTFUNC\}$\{TST_CNT\}' and if these are -found they are executed one by one. +found they are executed one by one. The test number is passed to it in the '$1'. + [source,sh] -------------------------------------------------------------------------------
@@ -1471,8 +1472,8 @@ TST_CNT=2 do_test() { case $1 in - 1) tst_res TPASS "Test 1 passed";; - 2) tst_res TPASS "Test 2 passed";; + 1) tst_res TPASS "Test $1 passed";; + 2) tst_res TPASS "Test $1 passed";; esac }
@@ -1483,6 +1484,65 @@ Otherwise, if '$TST_CNT' is set but there is no '$\{TST_TESTFUNC\}1', etc., the '$TST_TESTFUNC' is executed '$TST_CNT' times and the test number is passed to it in the '$1'. +[source,sh] +------------------------------------------------------------------------------- +#!/bin/sh +# +# Example test with tests in a single function, using $TST_TEST_DATA and +# $TST_TEST_DATA_IFS +# + +TST_TESTFUNC=do_test +TST_TEST_DATA="foo:bar:d dd" +TST_TEST_DATA_IFS=":" +. tst_test.sh + +do_test() +{ + tst_res TPASS "Test $1 passed with data '$2'" +} + +tst_run +# output: +# test 1 TPASS: Test 1 passed with data 'foo' +# test 2 TPASS: Test 2 passed with data 'bar' +# test 3 TPASS: Test 3 passed with data 'd dd' + +------------------------------------------------------------------------------- + +It's possible to pass data for function with '$TST_TEST_DATA'. Optional +'$TST_TEST_DATA_IFS' is used for splitting, default value is space. + +[source,sh] +------------------------------------------------------------------------------- +#!/bin/sh +# +# Example test with tests in a single function, using $TST_TEST_DATA and $TST_CNT +# + +TST_TESTFUNC=do_test +TST_CNT=2 +TST_TEST_DATA="foo:bar:d dd" +. tst_test.sh + +do_test() +{ + case $1 in + 1) tst_res TPASS "Test $1 passed with data '$2'";; + 2) tst_res TPASS "Test $1 passed with data '$2'";; + esac +} + +tst_run +# output: +# test 1 TPASS: Test 1 passed with data 'foo:bar:d' +# test 2 TPASS: Test 2 passed with data 'foo:bar:d' + +------------------------------------------------------------------------------- +When '$TST_TEST_DATA' is used with '$TST_CNT', it's passed as whole string in +'$2' ($1 is for the test number), '$TST_TEST_DATA_IFS' is ignored. Similar +would be when using these variables with separate functions. + 2.3.2 Library variables ^^^^^^^^^^^^^^^^^^^^^^^
@@ -1587,8 +1647,8 @@ these can be listed with passing help '-h' option to any test. The function that prints the usage is passed in '$TST_USAGE', the help for the options implemented in the library is appended when usage is printed. -Lastly the fucntion '$PARSE_ARGS' is called with the option name in '$1' and, -if option has argument, its value in '$2'. +Lastly the fucntion '$PARSE_ARGS' is called with the option name in the '$1' +and, if option has argument, its value in the '$2'. [source,sh] -------------------------------------------------------------------------------
testcases/lib/tst_test.sh +39 -22
92 lines not on this page · raw message
Signature · 1 line
2.16.3
RFC PATCH v3 2/2 lib: Add tests
---
This is not intended to be merged. It's just a helper which you can use for testing the previous patch.
(Side note: if we ever want to test tst_test.sh, it'd be possible to dump it into file and compare with saved output. It'd require to remove lines with 'stime=')
--- lib/newlib_tests/Makefile | 1 + lib/newlib_tests/test.TST_TEST.TST_CNT.separate.sh | 22 ++++++++++++ lib/newlib_tests/test.TST_TEST.TST_CNT.sh | 20 +++++++++++ lib/newlib_tests/test.TST_TEST.getopts.sh | 40 +++++++++++++++++++++ lib/newlib_tests/test.TST_TEST.sh | 25 +++++++++++++ .../test.TST_TEST_DATA.TST_CNT.separate.sh | 26 ++++++++++++++ lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.sh | 24 +++++++++++++ lib/newlib_tests/test.TST_TEST_DATA.getopts.sh | 41 +++++++++++++++++++++ lib/newlib_tests/test.TST_TEST_DATA.sh | 18 ++++++++++ lib/newlib_tests/test.TST_TEST_DATA_IFS.getopts.sh | 42 ++++++++++++++++++++++ lib/newlib_tests/test.TST_TEST_DATA_IFS.sh | 21 +++++++++++ runtest/test | 13 +++++++ 12 files changed, 293 insertions(+) create mode 100755 lib/newlib_tests/test.TST_TEST.TST_CNT.separate.sh create mode 100755 lib/newlib_tests/test.TST_TEST.TST_CNT.sh create mode 100755 lib/newlib_tests/test.TST_TEST.getopts.sh create mode 100755 lib/newlib_tests/test.TST_TEST.sh create mode 100755 lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.separate.sh create mode 100755 lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.sh create mode 100755 lib/newlib_tests/test.TST_TEST_DATA.getopts.sh create mode 100755 lib/newlib_tests/test.TST_TEST_DATA.sh create mode 100755 lib/newlib_tests/test.TST_TEST_DATA_IFS.getopts.sh create mode 100755 lib/newlib_tests/test.TST_TEST_DATA_IFS.sh create mode 100644 runtest/test
lib/newlib_tests/Makefile +1 -0
@@ -15,5 +15,6 @@ ifeq ($(ANDROID),1) FILTER_OUT_MAKE_TARGETS += test08 endif +INSTALL_TARGETS := *.sh include $(top_srcdir)/include/mk/generic_leaf_target.mk
lib/newlib_tests/test.TST_TEST.TST_CNT.separate.sh new file +22 -0
@@ -0,0 +1,22 @@ +#!/bin/sh +# +# Example test with tests in separate functions +# + +TST_TESTFUNC=test +TST_CNT=2 +. tst_test.sh + +test1() +{ + tst_res TPASS "Test $1 passed with no data ('$2')" +} + +test2() +{ + tst_res TPASS "Test $1 passed with no data ('$2')" +} + +tst_run +# test 1 TPASS: Test 1 passed with no data ('') +# test 2 TPASS: Test 2 passed with no data ('')
lib/newlib_tests/test.TST_TEST.TST_CNT.sh new file +20 -0
@@ -0,0 +1,20 @@ +#!/bin/sh +# +# Example test with tests in a single function +# + +TST_TESTFUNC=do_test +TST_CNT=2 +. tst_test.sh + +do_test() +{ + case $1 in + 1) tst_res TPASS "Test $1 passed with no data ('$2')";; + 2) tst_res TPASS "Test $1 passed with no data ('$2')";; + esac +} + +tst_run +# test 1 TPASS: Test 1 passed with no data ('') +# test 2 TPASS: Test 2 passed with no data ('')
lib/newlib_tests/test.TST_TEST.getopts.sh new file +40 -0
@@ -0,0 +1,40 @@ +#!/bin/sh +# +# Optional test command line parameters +# + +TST_OPTS="af:" +TST_USAGE=usage +TST_PARSE_ARGS=parse_args +TST_TESTFUNC=do_test + +. tst_test.sh + +ALTERNATIVE=0 +MODE="foo" + +usage() +{ + cat << EOF +usage: $0 [-a] [-f <foo|bar>] + +OPTIONS +-a Enable support for alternative foo +-f Specify foo or bar mode +EOF +} + +parse_args() +{ + case $1 in + a) ALTERNATIVE=1;; + f) MODE="$2";; + esac +} + +do_test() +{ + tst_res TPASS "Test $1 passed with data '$2': a: '$ALTERNATIVE', f: '$MODE'" +} + +tst_run
lib/newlib_tests/test.TST_TEST.sh new file +25 -0
@@ -0,0 +1,25 @@ +#!/bin/sh +# +# This is a basic test for true shell buildin +# + +TST_TESTFUNC=do_test +. tst_test.sh + +do_test() +{ + true + ret=$? + + tst_res TINFO "Test $1 passed with no data ('$2')" + + if [ $ret -eq 0 ]; then + tst_res TPASS "true returned 0" + else + tst_res TFAIL "true returned $ret" + fi +} + +tst_run +# test 1 TINFO: Test 1 passed with no data ('') +# test 1 TPASS: true returned 0
lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.separate.sh new file +26 -0
@@ -0,0 +1,26 @@ +#!/bin/sh +# +# Example test with tests in separate functions, using $TST_TEST_DATA and $TST_CNT +# + +TST_TESTFUNC=test +TST_CNT=2 +TST_TEST_DATA="foo:bar:d dd" +. tst_test.sh + +test1() +{ + tst_res TPASS "Test $1 passed with data '$2'" +} + +test2() +{ + tst_res TPASS "Test $1 passed with data '$2'" +} + +tst_run +# output: +# test 1 TPASS: Test 1 passed with data 'foo:bar:d' +# test 2 TPASS: Test 2 passed with data 'foo:bar:d' +# test 3 TPASS: Test 1 passed with data 'dd' +# test 4 TPASS: Test 2 passed with data 'dd'
lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.sh new file +24 -0
@@ -0,0 +1,24 @@ +#!/bin/sh +# +# Example test with tests in a single function, using $TST_TEST_DATA and $TST_CNT +# + +TST_TESTFUNC=do_test +TST_CNT=2 +TST_TEST_DATA="foo:bar:d dd" +. tst_test.sh + +do_test() +{ + case $1 in + 1) tst_res TPASS "Test $1 passed with data '$2'";; + 2) tst_res TPASS "Test $1 passed with data '$2'";; + esac +} + +tst_run +# output: +# test 1 TPASS: Test 1 passed with data 'foo:bar:d' +# test 2 TPASS: Test 2 passed with data 'foo:bar:d' +# test 3 TPASS: Test 1 passed with data 'dd' +# test 4 TPASS: Test 2 passed with data 'dd'
lib/newlib_tests/test.TST_TEST_DATA.getopts.sh new file +41 -0
41 lines not on this page · raw message
lib/newlib_tests/test.TST_TEST_DATA.sh new file +18 -0
@@ -0,0 +1,18 @@ +#!/bin/sh +# +# Example test with tests in a single function, using $TST_TEST_DATA +# + +TST_TESTFUNC=do_test +TST_TEST_DATA="foo:bar:d dd" +. tst_test.sh + +do_test() +{ + tst_res TPASS "Test $1 passed with data '$2'" +} + +tst_run +# output: +# test 1 TPASS: Test 1 passed with data 'foo:bar:d' +# test 2 TPASS: Test 2 passed with data 'foo:bar:d'
lib/newlib_tests/test.TST_TEST_DATA_IFS.getopts.sh new file +42 -0
42 lines not on this page · raw message
lib/newlib_tests/test.TST_TEST_DATA_IFS.sh new file +21 -0
@@ -0,0 +1,21 @@ +#!/bin/sh +# +# Example test with tests in a single function, using $TST_TEST_DATA and +# $TST_TEST_DATA_IFS +# + +TST_TESTFUNC=do_test +TST_TEST_DATA="foo:bar:d dd" +TST_TEST_DATA_IFS=":" +. tst_test.sh + +do_test() +{ + tst_res TPASS "Test $1 passed with data '$2'" +} + +tst_run +# output: +# test 1 TPASS: Test 1 passed with data 'foo' +# test 2 TPASS: Test 2 passed with data 'bar' +# test 3 TPASS: Test 3 passed with data 'd dd'
runtest/test new file +13 -0
@@ -0,0 +1,13 @@ +TST_TEST_DATA_IFS test.TST_TEST_DATA_IFS.sh +TST_TEST_DATA test.TST_TEST_DATA.sh +TST_TEST_DATA.TST_CNT.separate test.TST_TEST_DATA.TST_CNT.separate.sh +TST_TEST_DATA.TST_CNT test.TST_TEST_DATA.TST_CNT.sh +TST_TEST test.TST_TEST.sh +TST_TEST.TST_CNT.separate test.TST_TEST.TST_CNT.separate.sh +TST_TEST.TST_CNT test.TST_TEST.TST_CNT.sh +TST_TEST.getopts test.TST_TEST.getopts.sh +TST_TEST.getopts_a_f test.TST_TEST.getopts.sh -a -f foo1 +TST_TEST_DATA.getopts test.TST_TEST_DATA.getopts.sh +TST_TEST_DATA.getopts_a_f test.TST_TEST_DATA.getopts.sh -a -f foo2 +TST_TEST_DATA_IFS.getopts test.TST_TEST_DATA_IFS.getopts.sh +TST_TEST_DATA_IFS.getopts_a_f test.TST_TEST_DATA_IFS.getopts.sh -a -f foo3
Signature · 1 line
2.16.3
RFC PATCH v3 2/2 lib: Add tests
Hi!
Quoted 6 lines
This is not intended to be merged. It's just a helper which you can use for testing the previous patch.
(Side note: if we ever want to test
tst_test.sh, it'd be possible to dump it into file and compare with saved output. It'd require to remove lines with 'stime=')
That sounds like a good idea actually, I'm all for unit testing the shell library.
I'm not that sure that we would want to put it into the runtest file though, maybe we just need a runtest.sh script in the lib/ directory that would loop over all *.sh file in a certain directory and compared the output against corresponding file names without the .sh suffix.
We can then do something as lib/tests/ with runtest.sh that would loop over subdirectories in lib/tests/ executing actual testcases.
Signature · 2 lines
Cyril Hrubis [email protected]
RFC PATCH v3 2/2 lib: Add tests
Quoted 10 lines
Hi!
Quoted 6 lines
This is not intended to be merged. It's just a helper which you can use for testing the previous patch.
(Side note: if we ever want to test
tst_test.sh, it'd be possible to dump it into file and compare with saved output. It'd require to remove lines with 'stime=')That sounds like a good idea actually, I'm all for unit testing the shell library.
I was inspired by recent change in kernel's kconfig. Masahiro Yamada added tests, which compare output of various make calls with saved. It's in scripts/kconfig/tests/, using python's pytest, you can see it by running 'make testconfig'.
Quoted 7 lines
I'm not that sure that we would want to put it into the runtest file though, maybe we just need a
runtest.shscript in thelib/directory that would loop over all *.sh file in a certain directory and compared the output against corresponding file names without the .sh suffix.We can then do something as
lib/tests/withruntest.shthat would loop over subdirectories inlib/tests/executing actual testcases.
Sounds reasonable. I added it into runtest file just to help myself to test. And publish it just in case anyone wanted to use it for testing the patch.
Kind regards,
Petr
PATCH 1/2 Make shell lib tests standalone
In reply to Petr Vorel---
Please do not merge! See following message for details.
lib/newlib_tests/Makefile | 1 -
.../{ => shell}/test.TST_TEST.TST_CNT.separate.sh | 2 +-
.../{ => shell}/test.TST_TEST.TST_CNT.sh | 2 +-
.../{ => shell}/test.TST_TEST.getopts.sh | 2 +-
lib/newlib_tests/{ => shell}/test.TST_TEST.sh | 2 +-
.../test.TST_TEST_DATA.TST_CNT.separate.sh | 2 +-
.../{ => shell}/test.TST_TEST_DATA.TST_CNT.sh | 2 +-
.../{ => shell}/test.TST_TEST_DATA.getopts.sh | 2 +-
lib/newlib_tests/{ => shell}/test.TST_TEST_DATA.sh | 2 +-
.../{ => shell}/test.TST_TEST_DATA_IFS.getopts.sh | 2 +-
.../{ => shell}/test.TST_TEST_DATA_IFS.sh | 2 +-
lib/newlib_tests/test.shell_lib.sh | 94 ++++++++++++++++++++++
runtest/test | 13 ---
testcases/lib/tst_test.sh | 2 +-
14 files changed, 105 insertions(+), 25 deletions(-)
rename lib/newlib_tests/{ => shell}/test.TST_TEST.TST_CNT.separate.sh (95%)
rename lib/newlib_tests/{ => shell}/test.TST_TEST.TST_CNT.sh (95%)
rename lib/newlib_tests/{ => shell}/test.TST_TEST.getopts.sh (96%)
rename lib/newlib_tests/{ => shell}/test.TST_TEST.sh (95%)
rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA.TST_CNT.separate.sh (96%)
rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA.TST_CNT.sh (96%)
rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA.getopts.sh (96%)
rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA.sh (95%)
rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA_IFS.getopts.sh (97%)
rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA_IFS.sh (96%)
create mode 100755 lib/newlib_tests/test.shell_lib.sh
delete mode 100644 runtest/testlib/newlib_tests/Makefile +0 -1
@@ -16,6 +16,5 @@ ifeq ($(ANDROID),1) FILTER_OUT_MAKE_TARGETS += test08 endif -INSTALL_TARGETS := *.sh include $(top_srcdir)/include/mk/generic_leaf_target.mk
lib/newlib_tests/shell/test.TST_TEST.TST_CNT.separate.sh renamed from lib/newlib_tests/test.TST_TEST.TST_CNT.separate.sh +1 -1
@@ -5,7 +5,7 @@ TST_TESTFUNC=test TST_CNT=2 -. tst_test.sh +. ./tst_test.sh test1() {
lib/newlib_tests/shell/test.TST_TEST.TST_CNT.sh renamed from lib/newlib_tests/test.TST_TEST.TST_CNT.sh +1 -1
@@ -5,7 +5,7 @@ TST_TESTFUNC=do_test TST_CNT=2 -. tst_test.sh +. ./tst_test.sh do_test() {
lib/newlib_tests/shell/test.TST_TEST.getopts.sh renamed from lib/newlib_tests/test.TST_TEST.getopts.sh +1 -1
@@ -8,7 +8,7 @@ TST_USAGE=usage TST_PARSE_ARGS=parse_args TST_TESTFUNC=do_test -. tst_test.sh +. ./tst_test.sh ALTERNATIVE=0 MODE="foo"
lib/newlib_tests/shell/test.TST_TEST.sh renamed from lib/newlib_tests/test.TST_TEST.sh +1 -1
@@ -4,7 +4,7 @@ # TST_TESTFUNC=do_test -. tst_test.sh +. ./tst_test.sh do_test() {
lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.separate.sh renamed from lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.separate.sh +1 -1
@@ -6,7 +6,7 @@ TST_TESTFUNC=test TST_CNT=2 TST_TEST_DATA="foo:bar:d dd" -. tst_test.sh +. ./tst_test.sh test1() {
lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.sh renamed from lib/newlib_tests/test.TST_TEST_DATA.TST_CNT.sh +1 -1
@@ -6,7 +6,7 @@ TST_TESTFUNC=do_test TST_CNT=2 TST_TEST_DATA="foo:bar:d dd" -. tst_test.sh +. ./tst_test.sh do_test() {
lib/newlib_tests/shell/test.TST_TEST_DATA.getopts.sh renamed from lib/newlib_tests/test.TST_TEST_DATA.getopts.sh +1 -1
@@ -9,7 +9,7 @@ TST_PARSE_ARGS=parse_args TST_TESTFUNC=do_test TST_TEST_DATA="foo0:bar:d dd" -. tst_test.sh +. ./tst_test.sh ALTERNATIVE=0 MODE="foo"
lib/newlib_tests/shell/test.TST_TEST_DATA.sh renamed from lib/newlib_tests/test.TST_TEST_DATA.sh +1 -1
@@ -5,7 +5,7 @@ TST_TESTFUNC=do_test TST_TEST_DATA="foo:bar:d dd" -. tst_test.sh +. ./tst_test.sh do_test() {
lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.getopts.sh renamed from lib/newlib_tests/test.TST_TEST_DATA_IFS.getopts.sh +1 -1
@@ -10,7 +10,7 @@ TST_TESTFUNC=do_test TST_TEST_DATA="foo0:bar:d dd" TST_TEST_DATA_IFS=":" -. tst_test.sh +. ./tst_test.sh ALTERNATIVE=0 MODE="foo"
lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.sh renamed from lib/newlib_tests/test.TST_TEST_DATA_IFS.sh +1 -1
@@ -7,7 +7,7 @@ TST_TESTFUNC=do_test TST_TEST_DATA="foo:bar:d dd" TST_TEST_DATA_IFS=":" -. tst_test.sh +. ./tst_test.sh do_test() {
lib/newlib_tests/test.shell_lib.sh new file +94 -0
94 lines not on this page · raw message
runtest/test deleted +0 -13
@@ -1,13 +0,0 @@ -TST_TEST_DATA_IFS test.TST_TEST_DATA_IFS.sh -TST_TEST_DATA test.TST_TEST_DATA.sh -TST_TEST_DATA.TST_CNT.separate test.TST_TEST_DATA.TST_CNT.separate.sh -TST_TEST_DATA.TST_CNT test.TST_TEST_DATA.TST_CNT.sh -TST_TEST test.TST_TEST.sh -TST_TEST.TST_CNT.separate test.TST_TEST.TST_CNT.separate.sh -TST_TEST.TST_CNT test.TST_TEST.TST_CNT.sh -TST_TEST.getopts test.TST_TEST.getopts.sh -TST_TEST.getopts_a_f test.TST_TEST.getopts.sh -a -f foo1 -TST_TEST_DATA.getopts test.TST_TEST_DATA.getopts.sh -TST_TEST_DATA.getopts_a_f test.TST_TEST_DATA.getopts.sh -a -f foo2 -TST_TEST_DATA_IFS.getopts test.TST_TEST_DATA_IFS.getopts.sh -TST_TEST_DATA_IFS.getopts_a_f test.TST_TEST_DATA_IFS.getopts.sh -a -f foo3
testcases/lib/tst_test.sh +1 -1
@@ -33,7 +33,7 @@ export TST_ITERATIONS=1 export TST_TMPDIR_RHOST=0 export TST_LIB_LOADED=1 -. tst_ansi_color.sh +. ./tst_ansi_color.sh # default trap function trap "tst_brk TBROK 'test interrupted'" INT
Signature · 1 line
2.16.4
PATCH 2/2 Add wanted output to shell lib test case
---
Please do not merge!
I would like to get some feedback for this one and the previous patch. It is based on Petr Vorel's [RFC,v3,2/2] lib: Add tests from May 22nd 2018 with the message ID [email protected].
These two patches transfer Petr's test cases in a separate folder and a shell script loops over all files as suggested in the replies to the initial message. The output is verified against a comment section in the test case file itself. This section is started with the line "# output:" and it is expected that it goes to the end of the document without interception.
I have run the script with various Shell implementations to make sure it's portable.
Limitations:
The script is currently unable to test e.g. whether TFAIL is displayed with red color.
So far I have only added the correct output line in a single test case file so just one test is currently passing.
Before proceeding I would like to know whether this is a good way to go.
lib/newlib_tests/shell/test.TST_TEST.sh | 7 +++++++ 1 file changed, 7 insertions(+)
lib/newlib_tests/shell/test.TST_TEST.sh +7 -0
@@ -21,5 +21,12 @@ do_test() } tst_run +# output: # test 1 TINFO: Test 1 passed with no data ('') # test 1 TPASS: true returned 0 +# +# Summary: +# passed 1 +# failed 0 +# skipped 0 +# warnings 0
Signature · 1 line
2.16.4
PATCH 1/2 Make shell lib tests standalone
In reply to Christian LanigHi Christian,
this expects have applied my patch
https://patchwork.ozlabs.org/patch/918549/
http://lists.linux.it/pipermail/ltp/2018-May/008191.html
Christian, next time merge it with your changes (these 3 commits should be just one, with you as author.)
Before I delve into the details, just comment the functionality.
$ ./lib/newlib_tests/test.shell_lib.sh Running Test: "test.TST_TEST_DATA.getopts"... TFAIL:Test test.TST_TEST_DATA.getopts was unsuccessful.
Running Test: "test.TST_TEST_DATA_IFS.getopts"... TFAIL:Test test.TST_TEST_DATA_IFS.getopts was unsuccessful.
Running Test: "test.TST_TEST_DATA_IFS"... TFAIL:Test test.TST_TEST_DATA_IFS was unsuccessful.
Running Test: "test.TST_TEST_DATA"...
TFAIL:Test test.TST_TEST_DATA was unsuccessful.
Running Test: "test.TST_TEST_DATA.TST_CNT.separate"... TFAIL:Test test.TST_TEST_DATA.TST_CNT.separate was unsuccessful.
Running Test: "test.TST_TEST_DATA.TST_CNT"... TFAIL:Test test.TST_TEST_DATA.TST_CNT was unsuccessful.
Running Test: "test.TST_TEST.getopts"... TFAIL:Test test.TST_TEST.getopts was unsuccessful.
Running Test: "test.TST_TEST"...
TPASS: Test test.TST_TEST was successful.
Running Test: "test.TST_TEST.TST_CNT.separate"... TFAIL:Test test.TST_TEST.TST_CNT.separate was unsuccessful.
Running Test: "test.TST_TEST.TST_CNT"... TFAIL:Test test.TST_TEST.TST_CNT was unsuccessful.
---
These tests must be all green before merging them.
I suppose the failures are because only 5 of 10 tests have "# output:" (parsed by verify_output()).
About the output. See Cyril's comments about my patch for running C tests
http://lists.linux.it/pipermail/ltp/2018-August/009119.html
Inconsistent output: space after 'TPASS:', not after 'TFAIL:'. I wouldn't use ':'.
Output doesn't tell, what's wrong. IMHO there should be diff of expected output vs. real output. This is done in xfstests.
+ It could be less verbose. Let's use:
$ ./lib/newlib_tests/test.shell_lib.sh TFAIL test.TST_TEST_DATA.getopts [ diff output whats wrong ] TPASS test.TST_TEST
Summary:
passed 1
failed 1
The name itself would IMHO be bettertest_tst_test.sh or test.tst_test.sh (but we don't use often dot in name). ATM we have old and new legacy API, test.shell_lib.sh doesn't say which it tests.
Quoted 2 lines
---
Please do not merge! See following message for details.
You can use --rfc switch for git format-patch next time.
...
lib/newlib_tests/shell/test.TST_TEST.TST_CNT.separate.sh line 8
Quoted 8 lines
lib/newlib_tests/shell/test.TST_TEST.TST_CNT.separate.sh renamed from lib/newlib_tests/test.TST_TEST.TST_CNT.separate.sh +1
-1@@ -5,7 +5,7 @@ TST_TESTFUNC=test TST_CNT=2 -. tst_test.sh +. ./tst_test.sh
This is not good. Much better is to make sure, that testcases/lib/ directory (where tst_test.sh is) is in PATH variable. The reasons are that you don't expect current working directory be in the same directory as the test. See:
$ cd lib/newlib_tests $ ./shell/test.TST_TEST.TST_CNT.separate.sh bash: ./shell/test.TST_TEST.TST_CNT.separate.sh: No such file or directory
And the tests are supposed to be also examples in docs, where we prefer not forcing cd into /opt/ltp/testcases/bin.
...
lib/newlib_tests/test.shell_lib.sh line 1
Quoted 7 lines
lib/newlib_tests/test.shell_lib.sh new file +1
-0@@ -0,0 +1,94 @@ +# !/bin/sh
^ Please no space after.
checkbashisms.pl [1] warns you about it:
$ checkbashisms.pl -f test.shell_lib.sh script test.shell_lib.sh does not appear to have a #! interpreter line; you may get strange results
lib/newlib_tests/test.shell_lib.sh line 6
Quoted 5 lines
lib/newlib_tests/test.shell_lib.sh new file +5
-0+# +# This script iterates over all test cases for the new shell lib and verifies +# the output. +# Do NOT use newline symbols in the names of files containing test cases! +#
Maybe you want to add your copyright here.
You should add license (GPL v2 + any other). I prefer SPDX-License-Identifier as it's short:
# SPDX-License-Identifier: GPL-2.0-or-later
lib/newlib_tests/test.shell_lib.sh line 15
Quoted 10 lines
lib/newlib_tests/test.shell_lib.sh new file +9
-0+ +setup() { + color_green="\033[1;32m" + color_red="\033[1;31m" + standard_color="\033[0m" + start_dir="$PWD" + cd "$(dirname "$0")""/../../" || exit 1; + sh_lib_dir=""$PWD"/testcases/lib/" + sh_lib_test_dir=""$PWD"/lib/newlib_tests/shell/"
Wrong quote usage. In this interpretation $PWD isn't quoted. It should be:
sh_lib_test_dir="$PWD/lib/newlib_tests/shell/"
You probably want to setup it as global variable (out of all functions)
- in this case global isn't that bad. Also I'd prefer simpler name. Using dirname does not require to be in git root:
TESTDIR="$(dirname $0)/shell/"
Quoted 5 lines
lib/newlib_tests/test.shell_lib.sh new file +5
-0+ tst_cases=$(ls "$sh_lib_test_dir" | \ + sed "s/"$TEST_NAME".sh/"$TEST_NAME"/g") + cd "$sh_lib_dir" || exit 1 +} +
lib/newlib_tests/test.shell_lib.sh line 30
Quoted 10 lines
lib/newlib_tests/test.shell_lib.sh new file +10
-0+check_requirements() { + case "$0" in + -*) + printf "Please execute this script. Sourcing "; + printf "(. <SCRIPT>) is not supported. \n"; + return 1;; + *) + true; + esac +}
Is this really needed? I think it's obvious.
Now some syntax details.
Good example of shell script is the library itself (testcases/lib/tst_test.sh)
Quoted 1 line
+verify_output() {
New line before { }verify_output()
{
lib/newlib_tests/test.shell_lib.sh line 37
Quoted 5 lines
lib/newlib_tests/test.shell_lib.sh new file +5
-0+ local output_found=1 + local wanted_output= + local parsed_line= + while read line; + do
while read line; do
Keep 'do', 'then', 'else' on the same line
lib/newlib_tests/test.shell_lib.sh
Quoted 2 lines
lib/newlib_tests/test.shell_lib.sh new file +2
-0+ if [ -z "$wanted_output" ] && [ "$line" = "# output:" ] + then
if [ -z "$wanted_output" ] && [ "$line" = "# output:" ]; then
lib/newlib_tests/test.shell_lib.sh line 42
Quoted 3 lines
lib/newlib_tests/test.shell_lib.sh new file +3
-0+ output_found=0 + elif [ $output_found -eq 0 ] || [ -n "$wanted_output" ] + then
elif [ $output_found -eq 0 ] || [ -n "$wanted_output" ], then
Quoted 6 lines
lib/newlib_tests/test.shell_lib.sh new file +6
-0+ if printf "$line" | grep "# " > /dev/null; + then + if [ $output_found -eq 0 ] + then + parsed_line=$(printf "$line" | \ + sed "s/^.\{2\}//")
lib/newlib_tests/test.shell_lib.sh line 58
Quoted 10 lines
lib/newlib_tests/test.shell_lib.sh new file +10
-0+ output_found=1 + else + parsed_line="\n"$(printf "$line" | \ + sed "s/^.\{2\}//") + fi + elif printf "$line" | grep "#" > /dev/null; + then + parsed_line="\n"$(printf "$line" | \ + sed "s/^.\{1\}//") + else true;
What is this for? You don't compare $?, which would be set by true. And it's ugly.
lib/newlib_tests/test.shell_lib.sh line 64
Quoted 6 lines
lib/newlib_tests/test.shell_lib.sh new file +6
-0+ fi + wanted_output=""$wanted_output""$parsed_line"" + else true; + fi + + done < ""$sh_lib_test_dir""$1".sh"
The easiest way is to store the output in separate file. Your way parsing the output is interesting, but maybe too complicated for updating the pattern. Even if you use this way, I'd prefer to get the output with sed/awk. Maybe there are better ways, but this would work:
$ sed -ne '/^# output:/,$ p;' lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.separate.sh | sed '1d' # test 1 TPASS: Test 1 passed with data 'foo:bar:d' # test 2 TPASS: Test 2 passed with data 'foo:bar:d' # test 3 TPASS: Test 1 passed with data 'dd' # test 4 TPASS: Test 2 passed with data 'dd'
As I wrote, IMHO echo expected and actual output into the file and then run 'diff -u' on it would show, what's wrong.
lib/newlib_tests/test.shell_lib.sh line 65
Quoted 1 line
lib/newlib_tests/test.shell_lib.sh new file +1
-0+ wanted_output=$(printf "$wanted_output")
You probably don't want this (adding to wanted_output variable what is already there :) ).
lib/newlib_tests/test.shell_lib.sh line 66
Quoted 1 line
lib/newlib_tests/test.shell_lib.sh new file +1
-0+ local actual_output=$(""$sh_lib_test_dir""$1".sh")
Again, wrong quotes usage.
lib/newlib_tests/test.shell_lib.sh line 67
Quoted 1 line
lib/newlib_tests/test.shell_lib.sh new file +1
-0+ actual_output=$(printf "$actual_output")
Same here.
Maybe use less verbose variable name (detail).
lib/newlib_tests/test.shell_lib.sh line 73
Quoted 6 lines
lib/newlib_tests/test.shell_lib.sh new file +6
-0+ if [ "$wanted_output" = "$actual_output" ] + then + return 0 + else + return 1 + fi
You can use
[ "$wanted_output" = "$actual_output" ] && return 0 return 1
lib/newlib_tests/test.shell_lib.sh line 82
Quoted 9 lines
lib/newlib_tests/test.shell_lib.sh new file +8
-0+} +run_tests() { + for tst_case in $tst_cases + do + printf "Running Test: \""$tst_case"\"...\n" + if verify_output "$tst_case"; + then + printf ""$color_green"TPASS: "$standard_color""
Again, wrong quotes usage.
lib/newlib_tests/test.shell_lib.sh line 87
Quoted 5 lines
lib/newlib_tests/test.shell_lib.sh new file +5
-0+ printf "Test "$tst_case" was successful.\n\n" + else + printf ""$color_red"TFAIL:"$standard_color"" + printf "Test "$tst_case" was unsuccessful.\n\n" + fi
Again use '; do/then/else'.
lib/newlib_tests/test.shell_lib.sh line 94
Quoted 8 lines
lib/newlib_tests/test.shell_lib.sh new file +7
-0+ done +} + +check_requirements +setup +run_tests +exit 0
That's wrong. I'd like exit code was 0 for everything pass or 1 for failure.
Kind regards,
Petr
[1] https://salsa.debian.org/debian/devscripts/raw/master/scripts/checkbashisms.pl
PATCH 1/2 Make shell lib tests standalone
Hi Christian,
Quoted 5 lines
$ sed -ne '/^# output:/,$ p;' lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.separate.sh | sed '1d' # test 1 TPASS: Test 1 passed with data 'foo:bar:d' # test 2 TPASS: Test 2 passed with data 'foo:bar:d' # test 3 TPASS: Test 1 passed with data 'dd' # test 4 TPASS: Test 2 passed with data 'dd'
Sorry, this is the correct command:
$ sed -ne '/^# output:/,$ p;' lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.separate.sh | sed '1d; s/^# //' test 1 TPASS: Test 1 passed with data 'foo:bar:d' test 2 TPASS: Test 2 passed with data 'foo:bar:d' test 3 TPASS: Test 1 passed with data 'dd' test 4 TPASS: Test 2 passed with data 'dd'
But I bet other will be against keeping the output into the file.
+ It'd be nice to describe, how testing works - how to modify the output (in README.md or at least in commit message).
Kind regards,
Petr
cover Add automated tests for shell lib
In reply to Petr VorelThe following patch adds the test cases by Petr Vorel. They can be executed by automatically. doc/write-tests-for-shell-lib.txt describes how everything works. The output tells whether each test has passed and when it failed, the diff is printed.
I apologize for the badly formatted message I sent before and hope that this time I used the right format options.
@Petr
Quoted 2 lines
+ It'd be nice to describe, how testing works - how to modify the output (in README.md or at least in commit message).
I am not sure that the text in doc/write-tests-for-shell-lib.txt should really go in README.md. Probably just a link or something. But I can change it if necessary.
In regard of the other comments, I have removed all double quotes where nothing should go wrong as long as the person executing the script doesn't have unsupported file or directory naming or pretty uncommon bugs in his or her shell.
I have also removed unnecessary else statements (which I thought would be necessary to stay compatible but this appeared to be a wrong impression when I read the text about it more accurately a second time).
Also a -h / --help option was added which should be explanatory enough for people who intuitively run the script with this option and to replace comments about functionality in the header of the script file.
Another feature is to specify single test cases as arguments to only run these and no other.
Both outputs, the desired and the actual one, are now stored in a temporary folder and the PATH environment variable is modified temporarily, which implied the necessity to add a cleanup function.
Regards,
Christian
Christian Lanig (1):
Add automated tests for shell lib
doc/write-tests-for-shell-lib.txt | 61 ++++++++++++ .../shell/test.TST_TEST.TST_CNT.separate.sh | 29 ++++++ lib/newlib_tests/shell/test.TST_TEST.TST_CNT.sh | 27 ++++++ lib/newlib_tests/shell/test.TST_TEST.getopts.sh | 48 ++++++++++ lib/newlib_tests/shell/test.TST_TEST.sh | 32 +++++++ .../shell/test.TST_TEST_DATA.TST_CNT.separate.sh | 32 +++++++ .../shell/test.TST_TEST_DATA.TST_CNT.sh | 30 ++++++ .../shell/test.TST_TEST_DATA.getopts.sh | 50 ++++++++++ lib/newlib_tests/shell/test.TST_TEST_DATA.sh | 24 +++++ .../shell/test.TST_TEST_DATA_IFS.getopts.sh | 52 ++++++++++ lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.sh | 27 ++++++ lib/newlib_tests/test_sh_newlib.sh | 105 +++++++++++++++++++++ 12 files changed, 517 insertions(+) create mode 100644 doc/write-tests-for-shell-lib.txt create mode 100755 lib/newlib_tests/shell/test.TST_TEST.TST_CNT.separate.sh create mode 100755 lib/newlib_tests/shell/test.TST_TEST.TST_CNT.sh create mode 100755 lib/newlib_tests/shell/test.TST_TEST.getopts.sh create mode 100755 lib/newlib_tests/shell/test.TST_TEST.sh create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.separate.sh create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.sh create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA.getopts.sh create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA.sh create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.getopts.sh create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.sh create mode 100755 lib/newlib_tests/test_sh_newlib.sh
Signature · 1 line
2.16.4
RFC PATCH 1/1 Add automated tests for shell lib
--- doc/write-tests-for-shell-lib.txt | 61 ++++++++++++ .../shell/test.TST_TEST.TST_CNT.separate.sh | 29 ++++++ lib/newlib_tests/shell/test.TST_TEST.TST_CNT.sh | 27 ++++++ lib/newlib_tests/shell/test.TST_TEST.getopts.sh | 48 ++++++++++ lib/newlib_tests/shell/test.TST_TEST.sh | 32 +++++++ .../shell/test.TST_TEST_DATA.TST_CNT.separate.sh | 32 +++++++ .../shell/test.TST_TEST_DATA.TST_CNT.sh | 30 ++++++ .../shell/test.TST_TEST_DATA.getopts.sh | 50 ++++++++++ lib/newlib_tests/shell/test.TST_TEST_DATA.sh | 24 +++++ .../shell/test.TST_TEST_DATA_IFS.getopts.sh | 52 ++++++++++ lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.sh | 27 ++++++ lib/newlib_tests/test_sh_newlib.sh | 105 +++++++++++++++++++++ 12 files changed, 517 insertions(+) create mode 100644 doc/write-tests-for-shell-lib.txt create mode 100755 lib/newlib_tests/shell/test.TST_TEST.TST_CNT.separate.sh create mode 100755 lib/newlib_tests/shell/test.TST_TEST.TST_CNT.sh create mode 100755 lib/newlib_tests/shell/test.TST_TEST.getopts.sh create mode 100755 lib/newlib_tests/shell/test.TST_TEST.sh create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.separate.sh create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.sh create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA.getopts.sh create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA.sh create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.getopts.sh create mode 100755 lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.sh create mode 100755 lib/newlib_tests/test_sh_newlib.sh
doc/write-tests-for-shell-lib.txt new file +61 -0
@@ -0,0 +1,61 @@ +How to format tests in order to test the shell library +====================================================== + +It is of big importance not only to test the Linux kernel functionality but also +to make sure LTP is running correctly itself. For this reason it is useful to +test intrinsic functionality of LTP - to reasonable extend. + +1. Running tests for the shell library +-------------------------------------- +The test cases reside in the folder `lib/newlib_tests/shell`. A script executing +them one by one is in the folder `lib/newlib_tests`. You can execute this script +to test all cases or specify test cases to be run. The script is called +`test_sh_newlib.sh`. + +2. Writing tests for the shell library +-------------------------------------- +The tests are written like all other test cases using the shell library. +Additionally, at the end of the file the desired output is added. As an example: + +[source,shell] +------------------------------------------------------------------------------- +#!/bin/sh +# +# This is a basic test for true shell buildin +# + +TST_TESTFUNC=do_test +. tst_test.sh + +do_test() +{ + true + ret=$? + + tst_res TINFO "Test $1 passed with no data ('$2')" + + if [ $ret -eq 0 ]; then + tst_res TPASS "true returned 0" + else + tst_res TFAIL "true returned $ret" + fi +} + +tst_run +# output: +# test 1 TINFO: Test 1 passed with no data ('') +# test 1 TPASS: true returned 0 +# +# Summary: +# passed 1 +# failed 0 +# skipped 0 +# warnings 0 +------------------------------------------------------------------------------- + +The most noticeable thing is to add the line `# output:` to show the parser that +parsing should start in the following line. For the following lines the `# ` +will be stripped before the output is then compared with the actual output that +gets printed on the terminal when running the test. Also make sure that there +are no trailing lines at the end of the document. But make sure to add one +trailing space after the `#` when the line will be actually empty.
lib/newlib_tests/shell/test.TST_TEST.TST_CNT.separate.sh new file +29 -0
@@ -0,0 +1,29 @@ +#!/bin/sh +# +# Example test with tests in separate functions +# + +TST_TESTFUNC=test +TST_CNT=2 +. tst_test.sh + +test1() +{ + tst_res TPASS "Test $1 passed with no data ('$2')" +} + +test2() +{ + tst_res TPASS "Test $1 passed with no data ('$2')" +} + +tst_run +# output: +# test 1 TPASS: Test 1 passed with no data ('') +# test 2 TPASS: Test 2 passed with no data ('') +# +# Summary: +# passed 2 +# failed 0 +# skipped 0 +# warnings 0
lib/newlib_tests/shell/test.TST_TEST.TST_CNT.sh new file +27 -0
@@ -0,0 +1,27 @@ +#!/bin/sh +# +# Example test with tests in a single function +# + +TST_TESTFUNC=do_test +TST_CNT=2 +. tst_test.sh + +do_test() +{ + case $1 in + 1) tst_res TPASS "Test $1 passed with no data ('$2')";; + 2) tst_res TPASS "Test $1 passed with no data ('$2')";; + esac +} + +tst_run +# output: +# test 1 TPASS: Test 1 passed with no data ('') +# test 2 TPASS: Test 2 passed with no data ('') +# +# Summary: +# passed 2 +# failed 0 +# skipped 0 +# warnings 0
lib/newlib_tests/shell/test.TST_TEST.getopts.sh new file +48 -0
48 lines not on this page · raw message
lib/newlib_tests/shell/test.TST_TEST.sh new file +32 -0
@@ -0,0 +1,32 @@ +#!/bin/sh +# +# This is a basic test for true shell buildin +# + +TST_TESTFUNC=do_test +. tst_test.sh + +do_test() +{ + true + ret=$? + + tst_res TINFO "Test $1 passed with no data ('$2')" + + if [ $ret -eq 0 ]; then + tst_res TPASS "true returned 0" + else + tst_res TFAIL "true returned $ret" + fi +} + +tst_run +# output: +# test 1 TINFO: Test 1 passed with no data ('') +# test 1 TPASS: true returned 0 +# +# Summary: +# passed 1 +# failed 0 +# skipped 0 +# warnings 0
lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.separate.sh new file +32 -0
@@ -0,0 +1,32 @@ +#!/bin/sh +# +# Example test with tests in separate functions, using $TST_TEST_DATA and $TST_CNT +# + +TST_TESTFUNC=test +TST_CNT=2 +TST_TEST_DATA="foo:bar:d dd" +. tst_test.sh + +test1() +{ + tst_res TPASS "Test $1 passed with data '$2'" +} + +test2() +{ + tst_res TPASS "Test $1 passed with data '$2'" +} + +tst_run +# output: +# test 1 TPASS: Test 1 passed with data 'foo:bar:d' +# test 2 TPASS: Test 2 passed with data 'foo:bar:d' +# test 3 TPASS: Test 1 passed with data 'dd' +# test 4 TPASS: Test 2 passed with data 'dd' +# +# Summary: +# passed 4 +# failed 0 +# skipped 0 +# warnings 0
lib/newlib_tests/shell/test.TST_TEST_DATA.TST_CNT.sh new file +30 -0
@@ -0,0 +1,30 @@ +#!/bin/sh +# +# Example test with tests in a single function, using $TST_TEST_DATA and $TST_CNT +# + +TST_TESTFUNC=do_test +TST_CNT=2 +TST_TEST_DATA="foo:bar:d dd" +. tst_test.sh + +do_test() +{ + case $1 in + 1) tst_res TPASS "Test $1 passed with data '$2'";; + 2) tst_res TPASS "Test $1 passed with data '$2'";; + esac +} + +tst_run +# output: +# test 1 TPASS: Test 1 passed with data 'foo:bar:d' +# test 2 TPASS: Test 2 passed with data 'foo:bar:d' +# test 3 TPASS: Test 1 passed with data 'dd' +# test 4 TPASS: Test 2 passed with data 'dd' +# +# Summary: +# passed 4 +# failed 0 +# skipped 0 +# warnings 0
lib/newlib_tests/shell/test.TST_TEST_DATA.getopts.sh new file +50 -0
50 lines not on this page · raw message
lib/newlib_tests/shell/test.TST_TEST_DATA.sh new file +24 -0
@@ -0,0 +1,24 @@ +#!/bin/sh +# +# Example test with tests in a single function, using $TST_TEST_DATA +# + +TST_TESTFUNC=do_test +TST_TEST_DATA="foo:bar:d dd" +. tst_test.sh + +do_test() +{ + tst_res TPASS "Test $1 passed with data '$2'" +} + +tst_run +# output: +# test 1 TPASS: Test 1 passed with data 'foo:bar:d' +# test 2 TPASS: Test 1 passed with data 'dd' +# +# Summary: +# passed 2 +# failed 0 +# skipped 0 +# warnings 0
lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.getopts.sh new file +52 -0
52 lines not on this page · raw message
lib/newlib_tests/shell/test.TST_TEST_DATA_IFS.sh new file +27 -0
@@ -0,0 +1,27 @@ +#!/bin/sh +# +# Example test with tests in a single function, using $TST_TEST_DATA and +# $TST_TEST_DATA_IFS +# + +TST_TESTFUNC=do_test +TST_TEST_DATA="foo:bar:d dd" +TST_TEST_DATA_IFS=":" +. tst_test.sh + +do_test() +{ + tst_res TPASS "Test $1 passed with data '$2'" +} + +tst_run +# output: +# test 1 TPASS: Test 1 passed with data 'foo' +# test 2 TPASS: Test 1 passed with data 'bar' +# test 3 TPASS: Test 1 passed with data 'd dd' +# +# Summary: +# passed 3 +# failed 0 +# skipped 0 +# warnings 0
lib/newlib_tests/test_sh_newlib.sh new file +105 -0
105 lines not on this page · raw message
Signature · 1 line
2.16.4
RFC PATCH 1/1 Add automated tests for shell lib
Hi!
Quoted 44 lines
lib/newlib_tests/shell/test.TST_TEST.getopts.sh new file +38
-0@@ -0,0 +1,48 @@ +#!/bin/sh +# +# Optional test command line parameters +# + +TST_OPTS="af:" +TST_USAGE=usage +TST_PARSE_ARGS=parse_args +TST_TESTFUNC=do_test + +. tst_test.sh + +ALTERNATIVE=0 +MODE="foo" + +usage() +{ + cat << EOF +usage: $0 [-a] [-f <foo|bar>] + +OPTIONS +-a Enable support for alternative foo +-f Specify foo or bar mode +EOF +} + +parse_args() +{ + case $1 in + a) ALTERNATIVE=1;; + f) MODE="$2";; + esac +} + +do_test() +{ + tst_res TPASS "Test $1 passed with data '$2': a: '$ALTERNATIVE', f: '$MODE'" +}
lib/newlib_tests/shell/test.TST_TEST.getopts.sh line 48
Quoted 10 lines
lib/newlib_tests/shell/test.TST_TEST.getopts.sh new file +10
-0+ +tst_run +# output: +# test 1 TPASS: Test 1 passed with data '': a: '0', f: 'foo' +# +# Summary: +# passed 1 +# failed 0 +# skipped 0 +# warnings 0
We do need a way how to pass parameters to the tests so that we can test this functionality properly (and also the default options that are implemented in the test library).
Obvious way how to do this is to allow to have several output: blocks at the end of the test with a different parameters, something as:
# output:
# test 1 TPASS: Test 1 passed with data '': a: '0', f: 'foo' # # Summary: # passed 1 # failed 0 # skipped 0 # warnings 0
# output:
# params: -a
# test 1 TPASS: Test 1 passed with data '': a: '1', f: 'foo' # # Summary: # passed 1 # failed 0 # skipped 0 # warnings 0
...
Quoted 6 lines
lib/newlib_tests/test_sh_newlib.sh new file +0
-0
lib/newlib_tests/test_sh_newlib.sh line 10
Quoted 10 lines
lib/newlib_tests/test_sh_newlib.sh new file +10
-0@@ -0,0 +1,105 @@ +#!/bin/sh +# +# SPDX-License-Identifier: GPL-2.0-or-later +# ?? 2018 SUSE LLC. +# +# Author: Christian Lanig <[email protected]> + +setup() +{ + path_sav=$PATH
There is no need to save and restore the PATH, the script will be executed in subshell and the changes to variables are never propagated to the parent shell.
The only way how variables can be changed in the currently running shell is to source the script, in that case the current shell will execute the commands.
lib/newlib_tests/test_sh_newlib.sh line 15
Quoted 6 lines
lib/newlib_tests/test_sh_newlib.sh new file +6
-0+ PATH=""$PATH":$(dirname $0)/../../testcases/lib/" + + color_blue='\033[1;34m' + color_green='\033[1;32m' + color_red='\033[1;31m' + standard_color='\033[0m'
^
This is more of 'reset text attributes' as it not only changes the color to default but also turns off bold, underscore, blink, etc. So it would be better named something as reset_attrs='\033[0m'
Quoted 41 lines
lib/newlib_tests/test_sh_newlib.sh new file +41
-0+ tmp_dir="/tmp/sh_lib_tst-$$/" + mkdir $tmp_dir || cleanup 1 + sh_lib_tst_dir="$(dirname $0)/shell/" + tst_files="$(ls $sh_lib_tst_dir)" + return 0 +} + +cleanup() +{ + [ -d "$tmp_dir" ] && rm -rf "$tmp_dir" + PATH="$PATH_sav" + exit $1 +} + +print_help() +{ + printf "#############################################################\n" + printf "# This script iterates over test cases for the new shell #\n" + printf "# library and verifies the output. #\n" + printf "#############################################################\n" + printf "\n" + printf "Usage:\n" + printf "\t$(basename $0)\n" + printf "\t$(basename $0) [test file 1] [test file 2] ...\n\n" + exit 0; +} + +parse_params() +{ + [ -n "$1" ] && tst_files= + while [ -n "$1" ]; do + case "$1" in + --help) print_help;; + -h) print_help;; + -*) + printf "Unknown positional parameter "$1".\n" + cleanup 1;; + *) tst_files="$tst_files $1";; + esac + shift + done
lib/newlib_tests/test_sh_newlib.sh line 66
Quoted 10 lines
lib/newlib_tests/test_sh_newlib.sh new file +10
-0+ return 0 +} + +verify_output() +{ + [ -e "$sh_lib_tst_dir$tst" ] || { printf "$tst not found\n" && \ + cleanup 1 ;} + # read all lines after line: `# output:`, and strip `# ` from beginning + sed -n -e '/^# output:/,$ p;' "$sh_lib_tst_dir$tst" | sed '1d; s/^# //'\ + > "$tmp_dir$tst.wanted" || cleanup 1
Usually doing more complex text manipulation is easier with awk
In this case you can use gsub() function that will replace substring and print all lines that were replaced (i.e. print all comments without the hash and space):
awk 'gsub("^# ", "", $0)' script.sh
Then you can even get one of the output: blocks with:
awk -vb=1 'gsub("^# ", "", $0) {if (/output:/) {p++}; if (p==b && !/output:/) {print $0}};' script.sh
^
This gives you content of comments starting at first # output:
line up to the next # output: or end of line, which is exactly
what we need if we want to support passing parameters to
testcases. Then we can simply count how many # output: lines are
in a file and then run this command in a loop extracting one
example output after another.
Quoted 7 lines
lib/newlib_tests/test_sh_newlib.sh new file +7
-0+ actual_output=$($sh_lib_tst_dir$tst) + # remove control signs, add newline at EOF and store in temporary file + actual_output=$(printf "$actual_output\n" > $tmp_dir$tst".actual") || + cleanup 1 + + cmp $tmp_dir$tst".actual" $tmp_dir$tst".wanted" > /dev/null && return 0 + return 1
lib/newlib_tests/test_sh_newlib.sh line 84
Quoted 10 lines
lib/newlib_tests/test_sh_newlib.sh new file +10
-0+} + +run_tests() +{ + pass_cnt=0 + fail_cnt=0 + printf "\n" + for tst in $tst_files; do + if verify_output; then + pass_cnt=$(expr $pass_cnt + 1)
You can use arithmetic expansion here, i.e. pass_cnt=$((pass_cnt+1))
lib/newlib_tests/test_sh_newlib.sh line 88
Quoted 4 lines
lib/newlib_tests/test_sh_newlib.sh new file +4
-0+ printf ""$color_green"TPASS$standard_color $tst" + printf "\n" + else + fail_cnt=$(expr $fail_cnt + 1)
And here as well.
Quoted 20 lines
lib/newlib_tests/test_sh_newlib.sh new file +17
-0+ printf ""$color_red"TFAIL$standard_color $tst\n" + printf ""$color_blue"Diff:$standard_color\n" + diff -u "$tmp_dir$tst.actual" "$tmp_dir$tst.wanted" + printf "\n" + fi + done + printf "\nSummary:\n" + printf ""$color_red"Failed:$standard_color $fail_cnt\n" + printf ""$color_green"Passed:$standard_color $pass_cnt\n\n" + return $fail_cnt +} + +setup +parse_params "$@" +run_tests +res=$? +cleanup $resSignature · 1 line
2.16.4
Signature · 2 lines
Cyril Hrubis [email protected]
RFC PATCH v2 1/1 Add automated tests for shell lib
Hi,
Quoted 35 lines
Quoted 4 lines
lib/newlib_tests/shell/test.TST_TEST.getopts.sh new file +4
-0+do_test() +{ + tst_res TPASS "Test $1 passed with data '$2': a: '$ALTERNATIVE', f: '$MODE'" +}lib/newlib_tests/shell/test.TST_TEST.getopts.sh line 48
Quoted 10 lines
lib/newlib_tests/shell/test.TST_TEST.getopts.sh new file +10
-0+ +tst_run +# output: +# test 1 TPASS: Test 1 passed with data '': a: '0', f: 'foo' +# +# Summary: +# passed 1 +# failed 0 +# skipped 0 +# warnings 0We do need a way how to pass parameters to the tests so that we can test this functionality properly (and also the default options that are implemented in the test library).
Obvious way how to do this is to allow to have several output: blocks at the end of the test with a different parameters, something as:
# output:
# test 1 TPASS: Test 1 passed with data '': a: '0', f: 'foo'# Summary:
# passed 1 # failed 0 # skipped 0 # warnings 0# output:
# params: -a# test 1 TPASS: Test 1 passed with data '': a: '1', f: 'foo'
Quoted 7 lines
# Summary:
# passed 1 # failed 0 # skipped 0 # warnings 0...
Good idea!
...
Quoted 18 lines
lib/newlib_tests/test_sh_newlib.sh line 66
Quoted 7 lines
lib/newlib_tests/test_sh_newlib.sh new file +7
-0+verify_output() +{ + [ -e "$sh_lib_tst_dir$tst" ] || { printf "$tst not found\n" && \ + cleanup 1 ;} + # read all lines after line: `# output:`, and strip `# ` from beginning + sed -n -e '/^# output:/,$ p;' "$sh_lib_tst_dir$tst" | sed '1d; s/^# //'\ + > "$tmp_dir$tst.wanted" || cleanup 1Usually doing more complex text manipulation is easier with awk
In this case you can use
gsub()function that will replace substring and print all lines that were replaced (i.e. print all comments without the hash and space):awk '
gsub("^# ", "", $0)'script.shThen you can even get one of the output: blocks with:
Quoted 8 lines
awk -vb=1 '
gsub("^# ", "", $0){if (/output:/) {p++}; if (p==b && !/output:/) {print $0}};'script.sh
^
This gives you content of comments starting at first # output:
line up to the next # output: or end of line, which is exactly
what we need if we want to support passing parameters to
testcases. Then we can simply count how many # output: lines are
in a file and then run this command in a loop extracting one
example output after another.
I wonder whether "extracting one example output after another" is still good to do in awk/sed/shell (with simple readable code). Probably yes, but I wouldn't mind to have either python (pytest) or some perl code doing it. Awk is everywhere and definitely more capable than sed or even shell, but not everybody is fluent in it.
Kind regards,
Petr
RFC PATCH v2 1/1 Add automated tests for shell lib
In reply to Christian LanigHi Christian,
some more comments (beside these mentioned by Cyril). Mostly just shell code style comments.
lib/newlib_tests/shell/test.TST_TEST.TST_CNT.separate.sh line 24
Quoted 10 lines
lib/newlib_tests/shell/test.TST_TEST.TST_CNT.separate.sh new file +10
-0+test2() +{ + tst_res TPASS "Test $1 passed with no data ('$2')" +} + +tst_run +# output: +# test 1 TPASS: Test 1 passed with no data ('') +# test 2 TPASS: Test 2 passed with no data ('') +#
Note, I'd prefer not have trailing whitespace after '#' (actually anywhere). Not only it's unnecessary it's also problematic for 'git am' applying patches. In my case all the tests failed because of it.
Also some people setup their editors / IDE to delete trailing whitespace. (Not pointing all whitespace here.)
Quoted 7 lines
lib/newlib_tests/shell/test.TST_TEST.TST_CNT.separate.sh new file +5
-0+# Summary: +# passed 2 +# failed 0 +# skipped 0 +# warnings 0lib/newlib_tests/test_sh_newlib.sh +0
-0
...
lib/newlib_tests/test_sh_newlib.sh line 15
Quoted 8 lines
lib/newlib_tests/test_sh_newlib.sh +8
-0+setup() +{ + path_sav=$PATH + PATH=""$PATH":$(dirname $0)/../../testcases/lib/" + color_blue='\033[1;34m' + color_green='\033[1;32m' + color_red='\033[1;31m' + standard_color='\033[0m'
Maybe this variables can be outside of setup (no benefit to have them in setup.
lib/newlib_tests/test_sh_newlib.sh line 16
Quoted 1 line
lib/newlib_tests/test_sh_newlib.sh +1
-0+ tmp_dir="/tmp/sh_lib_tst-$$/"
Although /tmp is everywhere, maybe using $TMPDIR would be good. See tst_test.sh code:
if [ -z "$TMPDIR" ]; then export TMPDIR="/tmp" fi TST_TMPDIR=$(mktemp -d "$TMPDIR/LTP_$TST_ID.XXXXXXXXXX") chmod 777 "$TST_TMPDIR"
lib/newlib_tests/test_sh_newlib.sh line 18
Quoted 2 lines
lib/newlib_tests/test_sh_newlib.sh +2
-0+ mkdir $tmp_dir || cleanup 1 + sh_lib_tst_dir="$(dirname $0)/shell/"
Maybe just tstdir ?
lib/newlib_tests/test_sh_newlib.sh line 40
Quoted 13 lines
lib/newlib_tests/test_sh_newlib.sh +12
-0+ tst_files="$(ls $sh_lib_tst_dir)" +print_help() +{ + printf "#############################################################\n" + printf "# This script iterates over test cases for the new shell #\n" + printf "# library and verifies the output. #\n" + printf "#############################################################\n" + printf "\n" + printf "Usage:\n" + printf "\t$(basename $0)\n" + printf "\t$(basename $0) [test file 1] [test file 2] ...\n\n" + exit 0;
Maybe just:
cat <<EOF
Unit tests for new shell API
Usage: $(basename $0) [ TEST1 ] [ TEST2 ]
EOF
Quoted 7 lines
lib/newlib_tests/test_sh_newlib.sh +7
-0+} + +parse_params() +{ + [ -n "$1" ] && tst_files= + while [ -n "$1" ]; do + case "$1" in
lib/newlib_tests/test_sh_newlib.sh line 57
Quoted 10 lines
lib/newlib_tests/test_sh_newlib.sh +10
-0+ --help) print_help;; + -h) print_help;; + -*) + printf "Unknown positional parameter "$1".\n" + cleanup 1;; + *) tst_files="$tst_files $1";; + esac + shift + done + return 0
Why return, when you don't use it elsewhere?
...
lib/newlib_tests/test_sh_newlib.sh line 63
Quoted 4 lines
lib/newlib_tests/test_sh_newlib.sh +4
-0+verify_output() +{ + [ -e "$sh_lib_tst_dir$tst" ] || { printf "$tst not found\n" && \ + cleanup 1 ;}
This might be more readable (nitpicking):
if [ -n "$sh_lib_tst_dir$tst" ]; then
echo "$tst not found"
cleanup 1
fi
lib/newlib_tests/test_sh_newlib.sh line 71
Quoted 9 lines
lib/newlib_tests/test_sh_newlib.sh +8
-0+ # read all lines after line: `# output:`, and strip `# ` from beginning + sed -n -e '/^# output:/,$ p;' "$sh_lib_tst_dir$tst" | sed '1d; s/^# //'\ + > "$tmp_dir$tst.wanted" || cleanup 1 + + actual_output=$($sh_lib_tst_dir$tst) + # remove control signs, add newline at EOF and store in temporary file + actual_output=$(printf "$actual_output\n" > $tmp_dir$tst".actual") || + cleanup 1
Suppose this part will be rewritten. But why do you assign empty string? echo "$actual_output" > $tmp_dir$tst".actual || cleanup 1
...
lib/newlib_tests/test_sh_newlib.sh line 74
Quoted 3 lines
lib/newlib_tests/test_sh_newlib.sh +3
-0+ + cmp $tmp_dir$tst".actual" $tmp_dir$tst".wanted" > /dev/null && return 0 + return 1
This is enough (cmp returns it for you):
cmp -s "$tmp_dir$tst.actual" "$tmp_dir$tst.wanted"
Quoted 1 line
+}
...
lib/newlib_tests/test_sh_newlib.sh line 85
Quoted 9 lines
lib/newlib_tests/test_sh_newlib.sh +9
-0+run_tests() +{ + pass_cnt=0 + fail_cnt=0 + printf "\n" + for tst in $tst_files; do + if verify_output; then + pass_cnt=$(expr $pass_cnt + 1) + printf ""$color_green"TPASS$standard_color $tst"
^
"" is empty string, why?
lib/newlib_tests/test_sh_newlib.sh
Quoted 1 line
lib/newlib_tests/test_sh_newlib.sh +1
-0+ printf "\n"
You can also use {} to separate variable name:
echo "${color_green}TPASS$standard_color $tst" echo adds new line for you (for simple usage it's portable). It's safer to quote whole string (with variables).
lib/newlib_tests/test_sh_newlib.sh line 97
Quoted 11 lines
lib/newlib_tests/test_sh_newlib.sh +11
-0+ else + fail_cnt=$(expr $fail_cnt + 1) + printf ""$color_red"TFAIL$standard_color $tst\n" + printf ""$color_blue"Diff:$standard_color\n" + diff -u "$tmp_dir$tst.actual" "$tmp_dir$tst.wanted" + printf "\n" + fi + done + printf "\nSummary:\n" + printf ""$color_red"Failed:$standard_color $fail_cnt\n" + printf ""$color_green"Passed:$standard_color $pass_cnt\n\n"
And how about adding helper functions print_red, print_green, print_blue? Maybe not worth for this little script.
print_red()
{
printf "$red$1$reset"
}print_red "TFAIL
echo "$tst"
lib/newlib_tests/test_sh_newlib.sh line 105
Quoted 9 lines
lib/newlib_tests/test_sh_newlib.sh +8
-0+ return $fail_cnt +} + +setup +parse_params "$@" +run_tests +res=$? +cleanup $res
No need to have $res:run_tests
cleanup $?
Kind regards,
Petr
v2 cover Add automated tests for shell lib
Hi,
I am sorry that my reply took so long. Thank you very much for your comments which helped me to improve the code.
I have made an overhaul of the code based on your suggestions but an option to pass parameters to the tests is still not implemented.
I'm sending you what I have so far because it would help me when you could provide me more information regarding this.
I agree that it is difficult to find a way to process text the way needed here without disregarding the Linux Kernel Coding Style that asks for code which is transparent and easy to read when we only consider Shell commands. To address this issue I have drafted a Python script that might go in the tools folder and may perhaps be helpful in other cases as well. Since there are already Python files in the project it might be a valid option.
Thank you very much for your patience.
Regards,
Christian Lanig
Christian Lanig (1):
Add automated tests for shell lib
doc/write-tests-for-shell-lib.txt | 59 ++++++++++ lib/newlib_tests/shell/test_sh_newlib.sh | 102 ++++++++++++++++++ .../testcases/test.TST_TEST.TST_CNT.separate.sh | 30 ++++++ .../shell/testcases/test.TST_TEST.TST_CNT.sh | 28 +++++ .../shell/testcases/test.TST_TEST.getopts.sh | 49 +++++++++ lib/newlib_tests/shell/testcases/test.TST_TEST.sh | 33 ++++++ .../test.TST_TEST_DATA.TST_CNT.separate.sh | 33 ++++++ .../shell/testcases/test.TST_TEST_DATA.TST_CNT.sh | 31 ++++++ .../shell/testcases/test.TST_TEST_DATA.getopts.sh | 51 +++++++++ .../shell/testcases/test.TST_TEST_DATA.sh | 25 +++++ .../testcases/test.TST_TEST_DATA_IFS.getopts.sh | 53 +++++++++ .../shell/testcases/test.TST_TEST_DATA_IFS.sh | 28 +++++ tools/lookup_split_cut.py | 120 +++++++++++++++++++++ 13 files changed, 642 insertions(+) create mode 100644 doc/write-tests-for-shell-lib.txt create mode 100755 lib/newlib_tests/shell/test_sh_newlib.sh create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST.TST_CNT.separate.sh create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST.TST_CNT.sh create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST.getopts.sh create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST.sh create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.TST_CNT.separate.sh create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.TST_CNT.sh create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.getopts.sh create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.sh create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA_IFS.getopts.sh create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA_IFS.sh create mode 100755 tools/lookup_split_cut.py
Signature · 1 line
2.16.4
RFC PATCH v2 1/1 Add automated tests for shell lib
--- doc/write-tests-for-shell-lib.txt | 59 ++++++++++ lib/newlib_tests/shell/test_sh_newlib.sh | 102 ++++++++++++++++++ .../testcases/test.TST_TEST.TST_CNT.separate.sh | 30 ++++++ .../shell/testcases/test.TST_TEST.TST_CNT.sh | 28 +++++ .../shell/testcases/test.TST_TEST.getopts.sh | 49 +++++++++ lib/newlib_tests/shell/testcases/test.TST_TEST.sh | 33 ++++++ .../test.TST_TEST_DATA.TST_CNT.separate.sh | 33 ++++++ .../shell/testcases/test.TST_TEST_DATA.TST_CNT.sh | 31 ++++++ .../shell/testcases/test.TST_TEST_DATA.getopts.sh | 51 +++++++++ .../shell/testcases/test.TST_TEST_DATA.sh | 25 +++++ .../testcases/test.TST_TEST_DATA_IFS.getopts.sh | 53 +++++++++ .../shell/testcases/test.TST_TEST_DATA_IFS.sh | 28 +++++ tools/lookup_split_cut.py | 120 +++++++++++++++++++++ 13 files changed, 642 insertions(+) create mode 100644 doc/write-tests-for-shell-lib.txt create mode 100755 lib/newlib_tests/shell/test_sh_newlib.sh create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST.TST_CNT.separate.sh create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST.TST_CNT.sh create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST.getopts.sh create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST.sh create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.TST_CNT.separate.sh create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.TST_CNT.sh create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.getopts.sh create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.sh create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA_IFS.getopts.sh create mode 100755 lib/newlib_tests/shell/testcases/test.TST_TEST_DATA_IFS.sh create mode 100755 tools/lookup_split_cut.py
doc/write-tests-for-shell-lib.txt new file +59 -0
@@ -0,0 +1,59 @@ +How to format tests in order to test the shell library +====================================================== + +It is important to test the Linux kernel functionality but also to make sure +that LTP is running correctly itself. For this reason it is useful to test +intrinsic functionality of LTP. + +1. Running tests for the shell library +-------------------------------------- +The test cases reside in the folder `lib/newlib_tests/shell`. A script executing +them one by one is located in the folder `lib/newlib_tests`. You can execute +this script to test all cases or specify test cases to be run. The script is +called `test_sh_newlib.sh`. + +2. Writing tests for the shell library +-------------------------------------- +The tests are written like all other test cases using the shell library. +Additionally, at the end of the file the desired output is added. As an example: + +[source,shell] +------------------------------------------------------------------------------- +#!/bin/sh +# +# This is a basic test for true shell buildin +# + +TST_TESTFUNC=do_test +. tst_test.sh + +do_test() +{ + true + ret=$? + + tst_res TINFO "Test $1 passed with no data ('$2')" + + if [ $ret -eq 0 ]; then + tst_res TPASS "true returned 0" + else + tst_res TFAIL "true returned $ret" + fi +} + +tst_run +# output: +# test 1 TINFO: Test 1 passed with no data ('') +# test 1 TPASS: true returned 0 +# +# Summary: +# passed 1 +# failed 0 +# skipped 0 +# warnings 0 +------------------------------------------------------------------------------- + +The most noticeable thing is to add the line `# output:` to show the parser that +parsing should start in the following line. For the following lines the `# ` +will be stripped before the output is then compared with the actual output that +gets printed on the terminal when running the test.
lib/newlib_tests/shell/test_sh_newlib.sh new file +102 -0
102 lines not on this page · raw message
lib/newlib_tests/shell/testcases/test.TST_TEST.TST_CNT.separate.sh new file +30 -0
@@ -0,0 +1,30 @@ +#!/bin/sh +# +# Example test with tests in separate functions +# + +TST_TESTFUNC=test +TST_CNT=2 +. tst_test.sh + +test1() +{ + tst_res TPASS "Test $1 passed with no data ('$2')" +} + +test2() +{ + tst_res TPASS "Test $1 passed with no data ('$2')" +} + +tst_run +# output: +# test 1 TINFO: timeout per run is 0h 5m 0s +# test 1 TPASS: Test 1 passed with no data ('') +# test 2 TPASS: Test 2 passed with no data ('') +# +# Summary: +# passed 2 +# failed 0 +# skipped 0 +# warnings 0
lib/newlib_tests/shell/testcases/test.TST_TEST.TST_CNT.sh new file +28 -0
@@ -0,0 +1,28 @@ +#!/bin/sh +# +# Example test with tests in a single function +# + +TST_TESTFUNC=do_test +TST_CNT=2 +. tst_test.sh + +do_test() +{ + case $1 in + 1) tst_res TPASS "Test $1 passed with no data ('$2')";; + 2) tst_res TPASS "Test $1 passed with no data ('$2')";; + esac +} + +tst_run +# output: +# test 1 TINFO: timeout per run is 0h 5m 0s +# test 1 TPASS: Test 1 passed with no data ('') +# test 2 TPASS: Test 2 passed with no data ('') +# +# Summary: +# passed 2 +# failed 0 +# skipped 0 +# warnings 0
lib/newlib_tests/shell/testcases/test.TST_TEST.getopts.sh new file +49 -0
49 lines not on this page · raw message
lib/newlib_tests/shell/testcases/test.TST_TEST.sh new file +33 -0
@@ -0,0 +1,33 @@ +#!/bin/sh +# +# This is a basic test for true shell buildin +# + +TST_TESTFUNC=do_test +. tst_test.sh + +do_test() +{ + true + ret=$? + + tst_res TINFO "Test $1 passed with no data ('$2')" + + if [ $ret -eq 0 ]; then + tst_res TPASS "true returned 0" + else + tst_res TFAIL "true returned $ret" + fi +} + +tst_run +# output: +# test 1 TINFO: timeout per run is 0h 5m 0s +# test 1 TINFO: Test 1 passed with no data ('') +# test 1 TPASS: true returned 0 +# +# Summary: +# passed 1 +# failed 0 +# skipped 0 +# warnings 0
lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.TST_CNT.separate.sh new file +33 -0
@@ -0,0 +1,33 @@ +#!/bin/sh +# +# Example test with tests in separate functions, using $TST_TEST_DATA and $TST_CNT +# + +TST_TESTFUNC=test +TST_CNT=2 +TST_TEST_DATA="foo:bar:d dd" +. tst_test.sh + +test1() +{ + tst_res TPASS "Test $1 passed with data '$2'" +} + +test2() +{ + tst_res TPASS "Test $1 passed with data '$2'" +} + +tst_run +# output: +# test 1 TINFO: timeout per run is 0h 5m 0s +# test 1 TPASS: Test 1 passed with data 'foo:bar:d' +# test 2 TPASS: Test 2 passed with data 'foo:bar:d' +# test 3 TPASS: Test 1 passed with data 'dd' +# test 4 TPASS: Test 2 passed with data 'dd' +# +# Summary: +# passed 4 +# failed 0 +# skipped 0 +# warnings 0
lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.TST_CNT.sh new file +31 -0
@@ -0,0 +1,31 @@ +#!/bin/sh +# +# Example test with tests in a single function, using $TST_TEST_DATA and $TST_CNT +# + +TST_TESTFUNC=do_test +TST_CNT=2 +TST_TEST_DATA="foo:bar:d dd" +. tst_test.sh + +do_test() +{ + case $1 in + 1) tst_res TPASS "Test $1 passed with data '$2'";; + 2) tst_res TPASS "Test $1 passed with data '$2'";; + esac +} + +tst_run +# output: +# test 1 TINFO: timeout per run is 0h 5m 0s +# test 1 TPASS: Test 1 passed with data 'foo:bar:d' +# test 2 TPASS: Test 2 passed with data 'foo:bar:d' +# test 3 TPASS: Test 1 passed with data 'dd' +# test 4 TPASS: Test 2 passed with data 'dd' +# +# Summary: +# passed 4 +# failed 0 +# skipped 0 +# warnings 0
lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.getopts.sh new file +51 -0
51 lines not on this page · raw message
lib/newlib_tests/shell/testcases/test.TST_TEST_DATA.sh new file +25 -0
@@ -0,0 +1,25 @@ +#!/bin/sh +# +# Example test with tests in a single function, using $TST_TEST_DATA +# + +TST_TESTFUNC=do_test +TST_TEST_DATA="foo:bar:d dd" +. tst_test.sh + +do_test() +{ + tst_res TPASS "Test $1 passed with data '$2'" +} + +tst_run +# output: +# test 1 TINFO: timeout per run is 0h 5m 0s +# test 1 TPASS: Test 1 passed with data 'foo:bar:d' +# test 2 TPASS: Test 1 passed with data 'dd' +# +# Summary: +# passed 2 +# failed 0 +# skipped 0 +# warnings 0
lib/newlib_tests/shell/testcases/test.TST_TEST_DATA_IFS.getopts.sh new file +53 -0
53 lines not on this page · raw message
lib/newlib_tests/shell/testcases/test.TST_TEST_DATA_IFS.sh new file +28 -0
@@ -0,0 +1,28 @@ +#!/bin/sh +# +# Example test with tests in a single function, using $TST_TEST_DATA and +# $TST_TEST_DATA_IFS +# + +TST_TESTFUNC=do_test +TST_TEST_DATA="foo:bar:d dd" +TST_TEST_DATA_IFS=":" +. tst_test.sh + +do_test() +{ + tst_res TPASS "Test $1 passed with data '$2'" +} + +tst_run +# output: +# test 1 TINFO: timeout per run is 0h 5m 0s +# test 1 TPASS: Test 1 passed with data 'foo' +# test 2 TPASS: Test 1 passed with data 'bar' +# test 3 TPASS: Test 1 passed with data 'd dd' +# +# Summary: +# passed 3 +# failed 0 +# skipped 0 +# warnings 0
tools/lookup_split_cut.py new file +120 -0
120 lines not on this page · raw message
Signature · 1 line
2.16.4
RFC PATCH v2 1/1 Add automated tests for shell lib
Hi Christian,
thank you for working on it.
TL;DR: looks good to me, I have some code style objections, (use echo instead of printf, use $foo instead of ${foo} when possible)
+ some minor suggestions below.
Quoted 10 lines
+++
b/doc/write-tests-for-shell-lib.txtdoc/write-tests-for-shell-lib.txt new file +8
-0@@ -0,0 +1,59 @@ +How to format tests in order to test the shell library +====================================================== + +It is important to test the Linux kernel functionality but also to make sure +that LTP is running correctly itself. For this reason it is useful to test +intrinsic functionality of LTP. + +1. Running tests for the shell library
doc/write-tests-for-shell-lib.txt line 18
Quoted 10 lines
doc/write-tests-for-shell-lib.txt new file +10
-0+-------------------------------------- +The test cases reside in the folder `lib/newlib_tests/shell`. A script executing +them one by one is located in the folder `lib/newlib_tests`. You can execute +this script to test all cases or specify test cases to be run. The script is +called `test_sh_newlib.sh`. + +2. Writing tests for the shell library +-------------------------------------- +The tests are written like all other test cases using the shell library. +Additionally, at the end of the file the desired output is added. As an example:
Nice. I wonder if it should be in doc/test-writing-guidelines.txt. But this section is already too big, so it's probably good that it's separate. Then we need to add page this to wiki (simple).
BTW I plan to introduce make check, which will run this and other checks as well.
doc/write-tests-for-shell-lib.txt line 24
Quoted 6 lines
doc/write-tests-for-shell-lib.txt new file +6
-0+ +[source,shell] +------------------------------------------------------------------------------- +#!/bin/sh +# +# This is a basic test for true shell buildin
typo: builtin
Review line 25
Quoted 1 line
(quoted diff) +1
-0+#
nit: I'd remove this empty lines just with '#'
Quoted 45 lines
doc/write-tests-for-shell-lib.txt new file +34
-0+ +TST_TESTFUNC=do_test +. tst_test.sh + +do_test() +{ + true + ret=$? + + tst_res TINFO "Test $1 passed with no data ('$2')" + + if [ $ret -eq 0 ]; then + tst_res TPASS "true returned 0" + else + tst_res TFAIL "true returned $ret" + fi +} + +tst_run +# output: +# test 1 TINFO: Test 1 passed with no data ('') +# test 1 TPASS: true returned 0 +# +# Summary: +# passed 1 +# failed 0 +# skipped 0 +# warnings 0 +------------------------------------------------------------------------------- + +The most noticeable thing is to add the line `# output:` to show the parser that +parsing should start in the following line. For the following lines the `# ` +will be stripped before the output is then compared with the actual output that +gets printed on the terminal when running the test.lib/newlib_tests/shell/test_sh_newlib.sh new file +5
-0@@ -0,0 +1,102 @@ +#!/bin/sh +# +# SPDX-License-Identifier: GPL-2.0-or-later +# (c) 2019 SUSE LLC +#
lib/newlib_tests/shell/test_sh_newlib.sh line 15
Quoted 10 lines
lib/newlib_tests/shell/test_sh_newlib.sh new file +10
-0+# Author: Christian Lanig <[email protected]> + +PATH="${PATH}:$(dirname $(readlink -f $0))/../../../testcases/lib/" +if [ -z "$TMPDIR" ]; then + export TMPDIR="/tmp" +fi +color_blue='\033[1;34m' +color_green='\033[1;32m' +color_red='\033[1;31m' +reset_attr='\033[0m'
I'd prefer not reimplementing tst_ansi_color.sh.
IMHO it'd be better either use part of it (or tst_ansi_color.sh) or not use colors at all.
lib/newlib_tests/shell/test_sh_newlib.sh line 16
Quoted 1 line
lib/newlib_tests/shell/test_sh_newlib.sh new file +1
-0+tmp="${TMPDIR}/sh_lib_tst-${$}/"
nit: it can be $$ (instead of ${$})
lib/newlib_tests/shell/test_sh_newlib.sh line 25
Quoted 9 lines
lib/newlib_tests/shell/test_sh_newlib.sh new file +9
-0+mkdir $tmp || cleanup 1 +parent_dir=$(dirname $(readlink -f $0))/ +tooldir=${parent_dir}/../../../tools/ +testdir=${parent_dir}testcases/ +tst_files=$(ls $testdir) + +cleanup() +{ + [ -d "$tmp" ] && rm -rf "$tmp"
nit: 1 could be a default parameter.
lib/newlib_tests/shell/test_sh_newlib.sh line 36
Quoted 11 lines
lib/newlib_tests/shell/test_sh_newlib.sh new file +11
-0+ exit $1 +} + +print_help() +{ + cat <<EOF + +???????????????????????????????????????????????????????????????????????????????? +? This Shell script iterates over test cases for the new Shell library and ? +? verifies the output. ? +????????????????????????????????????????????????????????????????????????????????
nit: I'd prefer to use ASCII (-) than unicode (?).
lib/newlib_tests/shell/test_sh_newlib.sh line 40
Quoted 4 lines
lib/newlib_tests/shell/test_sh_newlib.sh new file +4
-0+ + Usage: + $(basename $0) [TEST_FILE_1] [TEST_FILE_2] +
nit: another space not needed.
lib/newlib_tests/shell/test_sh_newlib.sh line 48
Quoted 8 lines
lib/newlib_tests/shell/test_sh_newlib.sh new file +8
-0+EOF + exit 0 +} + +parse_params() +{ + [ -n "$1" ] && tst_files= + while [ -n "$1" ]; do
We usually prefer to use getopts, which does not allow long opts. It's ok for this small usage, but for more complicated output it's better not reimplementing it.
lib/newlib_tests/shell/test_sh_newlib.sh line 53
Quoted 5 lines
lib/newlib_tests/shell/test_sh_newlib.sh new file +5
-0+ case "$1" in + --help) print_help;; + -h) print_help;; + -*) + printf "Unknown positional parameter ${1}.\n"
maybe use simple this simpler form:
echo "Unknown positional parameter $1"
lib/newlib_tests/shell/test_sh_newlib.sh line 63
Quoted 10 lines
lib/newlib_tests/shell/test_sh_newlib.sh new file +10
-0+ cleanup 1;; + *) tst_files="$tst_files $1";; + esac + shift + done +} + +verify_output() +{ + if [ ! -e "${testdir}$tst" ]; then
This can safely be "$testdir$tst"
lib/newlib_tests/shell/test_sh_newlib.sh line 64
Quoted 1 line
lib/newlib_tests/shell/test_sh_newlib.sh new file +1
-0+ printf "$tst not found\n"
Again, use echo
lib/newlib_tests/shell/test_sh_newlib.sh line 71
Quoted 7 lines
lib/newlib_tests/shell/test_sh_newlib.sh new file +7
-0+ cleanup 1 + fi + + ${tooldir}lookup_split_cut.py -f ${testdir}$tst -d $tmp \ + -s '# output:\n' -c '# {0,1}' || cleanup 1 + + "${testdir}$tst" > "${tmp}$tst.actual" || cleanup 1
Again, ${} is not necessary.
lib/newlib_tests/shell/test_sh_newlib.sh line 72
Quoted 1 line
lib/newlib_tests/shell/test_sh_newlib.sh new file +1
-0+ cmp -s "${tmp}$tst.actual" "${tmp}${tst}_out/out.1" && return 0
+ check for cmp.
lib/newlib_tests/shell/test_sh_newlib.sh line 80
Quoted 8 lines
lib/newlib_tests/shell/test_sh_newlib.sh new file +8
-0+ return 1 +} + +run_tests() +{ + pass_cnt=0 + fail_cnt=0 + printf "\n"
again, echo is better here.
lib/newlib_tests/shell/test_sh_newlib.sh line 89
Quoted 9 lines
lib/newlib_tests/shell/test_sh_newlib.sh new file +9
-0+ for tst in $tst_files; do + if verify_output; then + pass_cnt=$(($pass_cnt + 1)) + printf "${color_green}TPASS$reset_attr ${tst}\n" + else + fail_cnt=$(($fail_cnt + 1)) + printf "${color_red}TFAIL$reset_attr ${tst}\n" + printf "${color_blue}Diff:${reset_attr}\n" + diff -u "${tmp}${tst}.actual" \
We might want to check for diff being available before we use.
lib/newlib_tests/shell/test_sh_newlib.sh line 98
Quoted 9 lines
lib/newlib_tests/shell/test_sh_newlib.sh new file +9
-0+ "${tmp}${tst}_out/out.1" + printf "\n" + fi + done + printf "\nSummary:\n" + printf "${color_red}Failed:$reset_attr $fail_cnt\n" + printf "${color_green}Passed:$reset_attr $pass_cnt\n\n" + return $fail_cnt +}
... (more tests)
tools/lookup_split_cut.py line 1
Quoted 7 lines
tools/lookup_split_cut.py new file +1
-0@@ -0,0 +1,120 @@ +#!/usr/bin/env python
I guess this should be python3.
I'd be a bit careful to bring python as another dependency, (there was some awk solution for this, proposed by Cyril), but as python is everywhere, it shouldn't be a problem.
(We definitely don't want python on SUT, these tests will be eventually rewritten into C or shell.)
Quoted 26 lines
tools/lookup_split_cut.py new file +26
-0+# +# SPDX-License-Identifier: GPL-2.0-or-later +# (c) 2019 SUSE LLC +# +# Author: Christian Lanig <[email protected]> + +import sys +import os +import re +from sys import argv +from os import makedirs, path + +src_file_path = None +dest = None +delim = None +pattern = None +perim = None + +argv.reverse() +basename = path.split(argv.pop())[1] + +def print_help(): + + help = """ + This script can look up a passage in a specified text pattern, split a + text file and cut a pattern. The operation chain is:
tools/lookup_split_cut.py line 37
Quoted 10 lines
tools/lookup_split_cut.py new file +10
-0+ + lookup > split > cut + + The output files are written to the specified destination directory. + + Usage: + """ + help += "\n\t\t" + basename + " -f [PATH] -d [PATH] -l " \ + + "[PERIMETER] -s [DELIMITER] \n" \ + + "\t\t\t\t -c [PATTERN]\n\n"
nit: you can use format() to use variables in multiline strings.
Quoted 31 lines
tools/lookup_split_cut.py new file +31
-0+ help += """ + -h, --help + print this help + -f, --file + source file to be processed + -d, --destination + destination path + -l, --lookup + look for data in text passage + -s, --split + split file by delimiter + -c, --cut + cut pattern from file + """ + + print(help) + sys.exit(0) + +def get_next_arg(): + if argv: + return argv.pop() + else: + print("Missing parameter. Run with \"--help\" for information.") + sys.exit(1) + +while argv: + arg = argv.pop() + if arg in ["-h", "--help"]: + print_help() + elif arg in ["-f", "--file"]: + src_file_path = get_next_arg()
tools/lookup_split_cut.py line 78
Quoted 10 lines
tools/lookup_split_cut.py new file +10
-0+ elif arg in ["-d", "--destination"]: + dest = get_next_arg() + elif arg in ["-l", "--lookup"]: + perim = get_next_arg() + elif arg in ["-s", "--split"]: + delim = get_next_arg() + elif arg in ["-c", "--cut"]: + pattern = get_next_arg() + else: + print("Illegal argument. Run with \"--help\" for information.")
I'd print help here.
Quoted 32 lines
tools/lookup_split_cut.py new file +32
-0+ sys.exit(1) + +if not src_file_path: + print("Input file has to be specified.") + sys.exit(1) + +if not delim and not pattern and not perim: + print("Missing parameters. Run with \"--help\" for information.") + sys.exit(1) + +src_file = open(src_file_path, "r") +src = src_file.read() +src_file.close() + +capture = 0 +if perim: + try: + capture = re.search(perim, src).group(1) + except: + pass + +if delim: + src_file_name = path.split(src_file_path)[1] + out_dir = dest + "/" + src_file_name + "_out" + + if not path.exists(out_dir): + makedirs(out_dir) + + src = re.split(delim, src) +else: + src = [src] +
tools/lookup_split_cut.py line 120
Quoted 10 lines
tools/lookup_split_cut.py new file +10
-0+if pattern: + for i in range(len(src)): + src[i] = re.sub(pattern, "", src[i]) +if delim or pattern: + for i in range(len(src)): + out_file = open(out_dir + "/out." + str(i), "w") + out_file.write(src[i]) + out_file.close() + +sys.exit(capture)
Kind regards,
Petr
RFC PATCH v2 1/1 Add automated tests for shell lib
Hi Petr and Clemens,
thank you for the reasonable comments and suggestions.
I just want to tell you that I will not be able to work on this until next week. I appreciate your patience very much.
Regards,
Christian
Signature · 4 lines
Christian Lanig - SUSE Engineering Apprentice SUSE Linux GmbH, Maxfeldstra?e 5, D-90409 N?rnberg Tel: +49-911-74053-0; Fax: +49-911-7417755; https://www.suse.com/ GF: Felix Imend?rffer, Jane Smithard, Graham Norton, HRB 21284 (AG N?rnberg)
RFC PATCH v2 1/1 Add automated tests for shell lib
In reply to Christian LanigHi Christian,
This tests are nice! Lets keep going with it.
On Thu, 2019-08-22 at 21:12 +0200, Christian Lanig wrote:
<snip>
Quoted 20 lines
lib/newlib_tests/shell/test_sh_newlib.sh new file +13
-0@@ -0,0 +1,102 @@ +#!/bin/sh +# +# SPDX-License-Identifier: GPL-2.0-or-later +# (c) 2019 SUSE LLC +# +# Author: Christian Lanig <[email protected]> + +PATH="${PATH}:$(dirname $(readlink -f $0))/../../../testcases/lib/" +if [ -z "$TMPDIR" ]; then + export TMPDIR="/tmp" +fi +color_blue='\033[1;34m' +color_green='\033[1;32m'
lib/newlib_tests/shell/test_sh_newlib.sh line 23
Quoted 10 lines
lib/newlib_tests/shell/test_sh_newlib.sh new file +10
-0+color_red='\033[1;31m' +reset_attr='\033[0m' +tmp="${TMPDIR}/sh_lib_tst-${$}/" +mkdir $tmp || cleanup 1 +parent_dir=$(dirname $(readlink -f $0))/ +tooldir=${parent_dir}/../../../tools/ +testdir=${parent_dir}testcases/ +tst_files=$(ls $testdir) + +cleanup()
You use cleanup as a default handler for error handing.
For instance, if a test doesn't have a # output: section we silently quit with exitcode 1. I would like to be informed more about such errors. Maybe we could just printout $tst if $! != 0.
Quoted 41 lines
lib/newlib_tests/shell/test_sh_newlib.sh new file +10
-0+{ + [ -d "$tmp" ] && rm -rf "$tmp" + exit $1 +} + +print_help() +{ + cat <<EOF + +????????????????????????????????????????????????????????????????????????????????
+? This Shell script iterates over test cases for the new Shell
library and ?
+? verifies the
output. ?
+???????????????????????????????????????????????????????????????????? ????????????lib/newlib_tests/shell/test_sh_newlib.sh new file +17
-0+ + Usage: + $(basename $0) [TEST_FILE_1] [TEST_FILE_2] + +EOF + exit 0 +} + +parse_params() +{ + [ -n "$1" ] && tst_files= + while [ -n "$1" ]; do + case "$1" in + --help) print_help;; + -h) print_help;; + -*) + printf "Unknown positional parameter${1}.\n"
lib/newlib_tests/shell/test_sh_newlib.sh new file +6
-0+ cleanup 1;; + *) tst_files="$tst_files $1";; + esac + shift + done +}
lib/newlib_tests/shell/test_sh_newlib.sh line 69
Quoted 10 lines
lib/newlib_tests/shell/test_sh_newlib.sh new file +10
-0+ +verify_output() +{ + if [ ! -e "${testdir}$tst" ]; then + printf "$tst not found\n" + cleanup 1 + fi + + ${tooldir}lookup_split_cut.py -f ${testdir}$tst -d $tmp \ + -s '# output:\n' -c '# {0,1}' || cleanup 1
just an idea, in perl ( I'm not sure if we have perl already as dependency). My feeling is, that lookup_split_cut.py is to much for that task.
cat $testdir$tst | perl -e '$o = 0; while (<STDIN>) {print substr($_,
2) if $o; $o = 1 if /^# output:/; }' > $tmp${tst}expected_output
lib/newlib_tests/shell/test_sh_newlib.sh
Quoted 2 lines
lib/newlib_tests/shell/test_sh_newlib.sh new file +2
-0+ + "${testdir}$tst" > "${tmp}$tst.actual" || cleanup 1
We should keep going on failed test. We need this to test timeout functionally or error handling...
Quoted 25 lines
lib/newlib_tests/shell/test_sh_newlib.sh new file +1
-0+ cmp -s "${tmp}$tst.actual" "${tmp}${tst}_out/out.1" &&return 0
lib/newlib_tests/shell/test_sh_newlib.sh new file +12
-0+ return 1 +} + +run_tests() +{ + pass_cnt=0 + fail_cnt=0 + printf "\n" + for tst in $tst_files; do + if verify_output; then + pass_cnt=$(($pass_cnt + 1)) + printf "${color_green}TPASS$reset_attr${tst}\n"
lib/newlib_tests/shell/test_sh_newlib.sh new file +3
-0+ else + fail_cnt=$(($fail_cnt + 1)) + printf "${color_red}TFAIL$reset_attr${tst}\n"
lib/newlib_tests/shell/test_sh_newlib.sh new file +5
-0+ printf "${color_blue}Diff:${reset_attr}\n" + diff -u "${tmp}${tst}.actual" \ + "${tmp}${tst}_out/out.1" + printf "\n" + fi
lib/newlib_tests/shell/test_sh_newlib.sh line 102
Quoted 10 lines
lib/newlib_tests/shell/test_sh_newlib.sh new file +10
-0+ done + printf "\nSummary:\n" + printf "${color_red}Failed:$reset_attr $fail_cnt\n" + printf "${color_green}Passed:$reset_attr $pass_cnt\n\n" + return $fail_cnt +} + +parse_params "$@" +run_tests +cleanup $?
<snip>
Thanks
Clemens
v2 cover Add automated tests for shell lib
In reply to Christian LanigHi Christian,
Quoted 9 lines
Hi,
I am sorry that my reply took so long. Thank you very much for your comments which helped me to improve the code.
I have made an overhaul of the code based on your suggestions but an option to pass parameters to the tests is still not implemented.
I'm sending you what I have so far because it would help me when you could provide me more information regarding this.
Quoted 8 lines
I agree that it is difficult to find a way to process text the way needed here without disregarding the Linux Kernel Coding Style that asks for code which is transparent and easy to read when we only consider Shell commands. To address this issue I have drafted a Python script that might go in the tools folder and may perhaps be helpful in other cases as well. Since there are already Python files in the project it might be a valid option.
Thank you very much for your patience.
No problem, thanks continue for it.
I'm also not sure if we should put them into separate testcases/ directory. I'd put them into shell/ directory and rename test_sh_newlib.sh to run.sh or something simple.
BTW (nit, no hard feeling :) as lib/newlib_tests/test.*.sh are based on my previous work, you could might mention it in the change log. You updated them to have timeout output, so it's your work, (if not, it'd be better to send them as mine separate
Kind regards,
Petr
PATCH 1/2 Make shell lib tests standalone
In reply to Christian LanigHi!
Quoted 10 lines
rename lib/newlib_tests/{ => shell}/test.TST_TEST.TST_CNT.separate.sh (95%)
rename lib/newlib_tests/{ => shell}/test.TST_TEST.TST_CNT.sh (95%)
rename lib/newlib_tests/{ => shell}/test.TST_TEST.getopts.sh (96%)
rename lib/newlib_tests/{ => shell}/test.TST_TEST.sh (95%)
rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA.TST_CNT.separate.sh (96%)
rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA.TST_CNT.sh (96%)
rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA.getopts.sh (96%)
rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA.sh (95%)
rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA_IFS.getopts.sh (97%)
rename lib/newlib_tests/{ => shell}/test.TST_TEST_DATA_IFS.sh (96%)Just small comment, I do wonder if top level lib/ directory is correct for the shell tests since the shell library is located at testcases/lib/ (for a historical reasons). Maybe we should just create a 'test' directory under 'testcases/lib/' instead.
Signature · 2 lines
Cyril Hrubis [email protected]
RFC PATCH v3 1/2 tst_test.sh: Add TST_TEST_DATA and TST_TEST_DATA_IFS
In reply to Petr VorelHi!
Quoted 9 lines
Signed-off-by Petr Vorel <[email protected]>---
Changesv2->v3:
* Don't pass $TST_TEST_DATA as whole argument for $TST_CNT (Cyril)
* Createtst_run_tests()for cleanup (Cyril).
* Createtst_run_test()- more cleanupMaybe we should try to "hide" somehow these API functions (underscore at the front?).
Actually this is a good idea, we can follow the python underscore notation that uses it for private variables and functions. Hence the counters will become _tst_i, etc.
But please do that in a separate patch.
Quoted 97 lines
--- doc/test-writing-guidelines.txt | 74 +++++++++++++++++++++++++++++++++++++---- testcases/lib/tst_test.sh | 61 +++++++++++++++++++++------------ 2 files changed, 106 insertions(+), 29 deletions(-)doc/test-writing-guidelines.txt +54
-5@@ -1442,12 +1442,12 @@ TST_CNT=2 test1() { - tst_res TPASS "Test 1 passed" + tst_res TPASS "Test $1 passed" } test2() { - tst_res TPASS "Test 2 passed" + tst_res TPASS "Test $1 passed" } tst_run@@ -1455,7 +1455,8 @@ tst_run If '$TST_CNT' is set, the test library looks if there are functions named '$\{TST_TESTFUNC\}1', ..., '$\{TST_TESTFUNC\}$\{TST_CNT\}' and if these are -found they are executed one by one. +found they are executed one by one. The test number is passed to it in the '$1'. + [source,sh] -------------------------------------------------------------------------------@@ -1471,8 +1472,8 @@ TST_CNT=2 do_test() { case $1 in - 1) tst_res TPASS "Test 1 passed";; - 2) tst_res TPASS "Test 2 passed";; + 1) tst_res TPASS "Test $1 passed";; + 2) tst_res TPASS "Test $1 passed";; esac }@@ -1483,6 +1484,65 @@ Otherwise, if '$TST_CNT' is set but there is no '$\{TST_TESTFUNC\}1', etc., the '$TST_TESTFUNC' is executed '$TST_CNT' times and the test number is passed to it in the '$1'. +[source,sh] +------------------------------------------------------------------------------- +#!/bin/sh +# +# Example test with tests in a single function, using $TST_TEST_DATA and +# $TST_TEST_DATA_IFS +# + +TST_TESTFUNC=do_test +TST_TEST_DATA="foo:bar:d dd" +TST_TEST_DATA_IFS=":" +. tst_test.sh + +do_test() +{ + tst_res TPASS "Test $1 passed with data '$2'" +} + +tst_run +# output: +# test 1 TPASS: Test 1 passed with data 'foo' +# test 2 TPASS: Test 2 passed with data 'bar' +# test 3 TPASS: Test 3 passed with data 'd dd' + +------------------------------------------------------------------------------- + +It's possible to pass data for function with '$TST_TEST_DATA'. Optional +'$TST_TEST_DATA_IFS' is used for splitting, default value is space. + +[source,sh] +------------------------------------------------------------------------------- +#!/bin/sh +# +# Example test with tests in a single function, using $TST_TEST_DATA and $TST_CNT +# + +TST_TESTFUNC=do_test +TST_CNT=2 +TST_TEST_DATA="foo:bar:d dd" +. tst_test.sh + +do_test() +{ + case $1 in + 1) tst_res TPASS "Test $1 passed with data '$2'";; + 2) tst_res TPASS "Test $1 passed with data '$2'";; + esac +}
doc/test-writing-guidelines.txt Otherwise, if '$TST_CNT' is set but there is no '$\{TST_TESTFUNC\}1', etc., · line 1544
Quoted 10 lines
doc/test-writing-guidelines.txt +10
-0+ +tst_run +# output: +# test 1 TPASS: Test 1 passed with data 'foo:bar:d' +# test 2 TPASS: Test 2 passed with data 'foo:bar:d' + +------------------------------------------------------------------------------- +When '$TST_TEST_DATA' is used with '$TST_CNT', it's passed as whole string in +'$2' ($1 is for the test number), '$TST_TEST_DATA_IFS' is ignored. Similar +would be when using these variables with separate functions.
Now that we support both TST_CNT and TST_TEST_DATA_IFS can we change this example to include the TST_TEST_DATA_IFS=":" as well?
Quoted 75 lines
doc/test-writing-guidelines.txt +2
-22.3.2 Library variables ^^^^^^^^^^^^^^^^^^^^^^^@@ -1587,8 +1647,8 @@ these can be listed with passing help '-h' option to any test. The function that prints the usage is passed in '$TST_USAGE', the help for the options implemented in the library is appended when usage is printed. -Lastly the fucntion '$PARSE_ARGS' is called with the option name in '$1' and, -if option has argument, its value in '$2'. +Lastly the fucntion '$PARSE_ARGS' is called with the option name in the '$1' +and, if option has argument, its value in the '$2'. [source,sh] -------------------------------------------------------------------------------testcases/lib/tst_test.sh +12
-22@@ -246,7 +246,7 @@ tst_rescmp() tst_run() { - local tst_i + local tst_i tst_data if [ -n "$TST_TEST_PATH" ]; then for tst_i in $(grep TST_ "$TST_TEST_PATH" | sed 's/.*TST_//; s/[="} \t\/:`].*//'); do@@ -255,7 +255,7 @@ tst_run() OPTS|USAGE|PARSE_ARGS|POS_ARGS);; NEEDS_ROOT|NEEDS_TMPDIR|NEEDS_DEVICE|DEVICE);; NEEDS_CMDS|NEEDS_MODULE|MODPATH|DATAROOT);; - IPV6);; + IPV6|TEST_DATA|TEST_DATA_IFS);; *) tst_res TWARN "Reserved variable TST_$tst_i used!";; esac done@@ -348,27 +348,17 @@ tst_run() #TODO check that test reports some results for each test function call while [ $TST_ITERATIONS -gt 0 ]; do - if [ -n "$TST_CNT" ]; then - if type test1 > /dev/null 2>&1; then - for tst_i in $(seq $TST_CNT); do - local res=$(tst_resstr) - $TST_TESTFUNC$tst_i - tst_rescmp "$res" - TST_COUNT=$((TST_COUNT+1)) - done - else - for tst_i in $(seq $TST_CNT); do - local res=$(tst_resstr) - $TST_TESTFUNC $tst_i - tst_rescmp "$res" - TST_COUNT=$((TST_COUNT+1)) - done - fi + if [ -n "$TST_TEST_DATA" ]; then + tst_i=1 + tst_check_cmds cut + while true; do + tst_data="$(echo "$TST_TEST_DATA" | cut -d"$TST_TEST_DATA_IFS" -f$tst_i)" + [ -z "$tst_data" ] && break + tst_run_tests "$tst_data" + tst_i=$((tst_i+1)) + done else - local res=$(tst_resstr) - $TST_TESTFUNC - tst_rescmp "$res" - TST_COUNT=$((TST_COUNT+1)) + tst_run_tests fi TST_ITERATIONS=$((TST_ITERATIONS-1)) done
testcases/lib/tst_test.sh tst_run() · line 375
Quoted 10 lines
testcases/lib/tst_test.sh +7
-0@@ -376,6 +366,31 @@ tst_run() tst_do_exit } +tst_run_tests() +{ + local tst_data="$1" + local tst_i + + for tst_i in $(seq ${TST_CNT:-1}); do + if type test1 > /dev/null 2>&1; then
I know that this is not your fault but this should be probably:
if type ${TST_TESTFUNC}1 > /dev/null 2>&1; ...
testcases/lib/tst_test.sh tst_run() · line 385
Quoted 10 lines
testcases/lib/tst_test.sh +10
-0+ tst_run_test "$TST_TESTFUNC$tst_i" $tst_i "$tst_data" + else + tst_run_test "$TST_TESTFUNC" $tst_i "$tst_data" + fi + done +} + +tst_run_test() +{ + local res=$(tst_resstr)
Can we please prefix the res here with tst_ as well since we are touching the code?
Quoted 23 lines
testcases/lib/tst_test.sh +10
-0+ local tst_fnc="$1" + shift + + $tst_fnc "$@" + tst_rescmp "$res" + TST_COUNT=$((TST_COUNT+1)) +} + if [ -z "$TST_ID" ]; then filename=$(basename $0) TST_ID=${filename%%.*}@@ -400,6 +415,8 @@ if [ -z "$TST_NO_DEFAULT_RUN" ]; then tst_brk TBROK "TST_TESTFUNC is not defined" fi + TST_TEST_DATA_IFS="${TST_TEST_DATA_IFS:- }" + if [ -n "$TST_CNT" ]; then if ! tst_is_int "$TST_CNT"; then tst_brk TBROK "TST_CNT must be integer"Signature · 1 line
2.16.3
Signature · 2 lines
Cyril Hrubis [email protected]
RFC PATCH v3 1/2 tst_test.sh: Add TST_TEST_DATA and TST_TEST_DATA_IFS
Hi Cyril,
Thanks for your comments
Quoted 7 lines
Quoted 6 lines
Signed-off-by Petr Vorel <[email protected]>---
Changesv2->v3:
* Don't pass $TST_TEST_DATA as whole argument for $TST_CNT (Cyril)
* Createtst_run_tests()for cleanup (Cyril).
* Createtst_run_test()- more cleanup
Quoted 8 lines
Quoted 2 lines
Maybe we should try to "hide" somehow these API functions (underscore at the front?).
Actually this is a good idea, we can follow the python underscore notation that uses it for private variables and functions. Hence the counters will become
_tst_i, etc.But please do that in a separate patch.
I meant to add underscore before new functions i.e. _tst_run_tests() and _tst_run_test().
Variables: do you mean to change local variables in tst_run()?
That would be really for separate patch.
Quoted 4 lines
Quoted 4 lines
--- doc/test-writing-guidelines.txt | 74 +++++++++++++++++++++++++++++++++++++---- testcases/lib/tst_test.sh | 61 +++++++++++++++++++++------------ 2 files changed, 106 insertions(+), 29 deletions(-)
...
Quoted 17 lines
Quoted 17 lines
doc/test-writing-guidelines.txt +17
-0+TST_TESTFUNC=do_test +TST_CNT=2 +TST_TEST_DATA="foo:bar:d dd" +. tst_test.sh + +do_test() +{ + case $1 in + 1) tst_res TPASS "Test $1 passed with data '$2'";; + 2) tst_res TPASS "Test $1 passed with data '$2'";; + esac +} + +tst_run +# output: +# test 1 TPASS: Test 1 passed with data 'foo:bar:d' +# test 2 TPASS: Test 2 passed with data 'foo:bar:d'
Quoted 8 lines
doc/test-writing-guidelines.txt Otherwise, if '$TST_CNT' is set but there is no '$\{TST_TESTFUNC\}1', etc., · line 1544
Quoted 5 lines
doc/test-writing-guidelines.txt +5
-0+ +------------------------------------------------------------------------------- +When '$TST_TEST_DATA' is used with '$TST_CNT', it's passed as whole string in +'$2' ($1 is for the test number), '$TST_TEST_DATA_IFS' is ignored. Similar +would be when using these variables with separate functions.Now that we support both
TST_CNTandTST_TEST_DATA_IFScan we change this example to include the TST_TEST_DATA_IFS=":" as well?
I wanted to demonstrate that TST_TEST_DATA_IFS has a default value ' '). But ok, I'll change it to define TST_TEST_DATA_IFS as well.
...
Quoted 6 lines
testcases/lib/tst_test.sh tst_run() · line 375
Quoted 2 lines
testcases/lib/tst_test.sh +2
-0+ for tst_i in $(seq ${TST_CNT:-1}); do + if type test1 > /dev/null 2>&1; thenI know that this is not your fault but this should be probably:
if type ${TST_TESTFUNC}1 >
/dev/null2>&1; ...
Thanks, I'll fix it, in a separate commit.
Quoted 6 lines
testcases/lib/tst_test.sh tst_run() · line 385
Quoted 3 lines
testcases/lib/tst_test.sh +3
-0+tst_run_test() +{ + local res=$(tst_resstr)Can we please prefix the res here with
tst_as well since we are touching the code?
Sure. To be honest I wasn't sure whether there is some reason for $res being named without prefix or not.
Kind regards,
Petr
RFC PATCH v3 1/2 tst_test.sh: Add TST_TEST_DATA and TST_TEST_DATA_IFS
Hi!
Quoted 8 lines
Quoted 5 lines
Actually this is a good idea, we can follow the python underscore notation that uses it for private variables and functions. Hence the counters will become
_tst_i, etc.But please do that in a separate patch.
I meant to add underscore before new functions i.e.
_tst_run_tests()and_tst_run_test().
Variables: do you mean to change local variables intst_run()?
That would be really for separate patch.
As I written, change everything private that leaks to the test context with _tst_ so that it's clear what the test is supposed to set/call.
Quoted 31 lines
Quoted 4 lines
Quoted 4 lines
--- doc/test-writing-guidelines.txt | 74 +++++++++++++++++++++++++++++++++++++---- testcases/lib/tst_test.sh | 61 +++++++++++++++++++++------------ 2 files changed, 106 insertions(+), 29 deletions(-)...
Quoted 17 lines
Quoted 17 lines
doc/test-writing-guidelines.txt +17
-0+TST_TESTFUNC=do_test +TST_CNT=2 +TST_TEST_DATA="foo:bar:d dd" +. tst_test.sh + +do_test() +{ + case $1 in + 1) tst_res TPASS "Test $1 passed with data '$2'";; + 2) tst_res TPASS "Test $1 passed with data '$2'";; + esac +} + +tst_run +# output: +# test 1 TPASS: Test 1 passed with data 'foo:bar:d' +# test 2 TPASS: Test 2 passed with data 'foo:bar:d'Quoted 8 lines
doc/test-writing-guidelines.txt Otherwise, if '$TST_CNT' is set but there is no '$\{TST_TESTFUNC\}1', etc., · line 1544
Quoted 5 lines
doc/test-writing-guidelines.txt +5
-0+ +------------------------------------------------------------------------------- +When '$TST_TEST_DATA' is used with '$TST_CNT', it's passed as whole string in +'$2' ($1 is for the test number), '$TST_TEST_DATA_IFS' is ignored. Similar +would be when using these variables with separate functions.Now that we support both
TST_CNTandTST_TEST_DATA_IFScan we change this example to include the TST_TEST_DATA_IFS=":" as well?
Quoted 2 lines
I wanted to demonstrate that
TST_TEST_DATA_IFShas a default value ' '). But ok, I'll change it to defineTST_TEST_DATA_IFSas well.
No problem with that, but then we should get rid of the : from the example data and keep it only at "foo bar".
Quoted 4 lines
Quoted 2 lines
Can we please prefix the res here with
tst_as well since we are touching the code?Sure. To be honest I wasn't sure whether there is some reason for $res being named without prefix or not.
I guess that it haven't been caught in the review process when I first wrote the library...
Signature · 2 lines
Cyril Hrubis [email protected]