shell API setup/cleanup checks
v4 cover shell API setup/cleanup checks
Hi,
previous version posted in
https://patchwork.ozlabs.org/project/ltp/list/?series=178107&state=*
But this version properly run cleanup function after test start (IMHO most important part of the patchset).
Kind regards,
Petr
Petr Vorel (3):
tst_test.sh: Call cleanup function after test start tst_test.sh: Warn on missing cleanup function tst_test.sh: Exit when setup declared but not defined
doc/test-writing-guidelines.txt | 10 +++++++--- testcases/lib/tst_test.sh | 16 ++++++++++++---- 2 files changed, 19 insertions(+), 7 deletions(-)
Signature · 1 line
2.29.2
PATCH v4 1/3 tst_test.sh: Call cleanup function after test start
It does not make sense to run the test cleanup function when the setup function has been run.
And at least some network tests expect setup has been run before running cleanup (e.g. tcp_fastopen_run.sh).
When shell API was introduced, cleanup function was run only if 1) setup function was defined 2) and also run. That was inconsistent from C API, thus e7dc14caa run it always.
But shell API is different from C API: tst_brk can be called from tst_test.sh (or other library which is run before tst_run, e.g. tst_net.sh). That was probably the reason, why detection via $TST_SETUP_STARTED was introduced in initial shell API.
NOTE: using type is better than grep $TST_TEST_PATH, because cleanup function can be in other library sourced by the test.
--- doc/test-writing-guidelines.txt | 10 +++++++--- testcases/lib/tst_test.sh | 4 ++-- 2 files changed, 9 insertions(+), 5 deletions(-)
doc/test-writing-guidelines.txt +7 -3
@@ -2251,11 +2251,15 @@ able to clean up correctly even in this situation. The easiest solution for this is to keep track of what was initialized and act accordingly in the cleanup. -WARNING: Similar to the C library, calling tst_brk() in the $TST_CLEANUP does +WARNING: Similar to the C library, calling 'tst_brk' in the $TST_CLEANUP does not exit the test and 'TBROK' is converted to 'TWARN'. -Notice also the 'tst_run' function called at the end of the test that actually -starts the test. +Notice also the 'tst_run' shell API function called at the end of the test that +actually starts the test. + +WARNING: cleanup function is called only after 'tst_run' has been started. +Calling 'tst_brk' in shell libraries, e.g. 'tst_test.sh' or 'tst_net.sh' does +not trigger calling it. [source,sh] -------------------------------------------------------------------------------
testcases/lib/tst_test.sh +2 -2
@@ -28,7 +28,7 @@ _tst_do_exit() local ret=0 TST_DO_EXIT=1 - if [ -n "$TST_CLEANUP" -a -z "$TST_NO_CLEANUP" ]; then + if [ -n "$TST_RUN_STARTED" -a -n "$TST_CLEANUP" -a -z "$TST_NO_CLEANUP" ]; then $TST_CLEANUP fi
@@ -599,7 +599,6 @@ tst_run() fi TST_ITERATIONS=$((TST_ITERATIONS-1)) done - _tst_do_exit }
@@ -608,6 +607,7 @@ _tst_run_tests() local _tst_data="$1" local _tst_i + TST_RUN_STARTED=1 for _tst_i in $(seq ${TST_CNT:-1}); do if type ${TST_TESTFUNC}1 > /dev/null 2>&1; then _tst_run_test "$TST_TESTFUNC$_tst_i" $_tst_i "$_tst_data"
Signature · 1 line
2.29.2
PATCH v4 1/3 tst_test.sh: Call cleanup function after test start
Hi!
Quoted 2 lines
It does not make sense to run the test cleanup function when the setup function has been run.
Did you mean "setup function hasn't been run." here?
As it is this sentence does not make any sense to me.
Quoted 12 lines
And at least some network tests expect setup has been run before running cleanup (e.g.
tcp_fastopen_run.sh).When shell API was introduced, cleanup function was run only if 1) setup function was defined 2) and also run. That was inconsistent from C API, thus
e7dc14caarun it always.But shell API is different from C API:
tst_brkcan be called fromtst_test.sh(or other library which is run beforetst_run, e.g.tst_net.sh). That was probably the reason, why detection via $TST_SETUP_STARTED was introduced in initial shell API.
Quoted 4 lines
NOTE: using type is better than grep $TST_TEST_PATH, because cleanup function can be in other library sourced by the test.
Fixes e7dc14caa ("tst_test.sh: Run cleanup even setup is not defined")
This still possibly break tests that would call tst_brk() in the middle of setup and expect the cleanup() to be executed, right?
I guess that we would need TST_DO_CLEANUP that would be set in both cases i.e. before we run setup and also before we execute the tests.
What about this:
testcases/lib/tst_test.sh +3 -1
@@ -28,7 +28,7 @@ _tst_do_exit() local ret=0 TST_DO_EXIT=1 - if [ -n "$TST_CLEANUP" -a -z "$TST_NO_CLEANUP" ]; then + if [ -n $TST_DO_CLEANUP -a -n "$TST_CLEANUP" -a -z "$TST_NO_CLEANUP" ]; then $TST_CLEANUP fi
@@ -582,6 +582,7 @@ tst_run() [ -n "$TST_NEEDS_MODULE" ] && tst_require_module "$TST_NEEDS_MODULE" if [ -n "$TST_SETUP" ]; then + TST_DO_CLEANUP=1 $TST_SETUP fi
@@ -608,6 +609,7 @@ _tst_run_tests() local _tst_data="$1" local _tst_i + TST_DO_CLEANUP=1 for _tst_i in $(seq ${TST_CNT:-1}); do if type ${TST_TESTFUNC}1 > /dev/null 2>&1; then _tst_run_test "$TST_TESTFUNC$_tst_i" $_tst_i "$_tst_data"
Signature · 2 lines
Cyril Hrubis [email protected]
PATCH v4 1/3 tst_test.sh: Call cleanup function after test start
Hi Cyril,
Quoted 7 lines
Hi!
Quoted 2 lines
It does not make sense to run the test cleanup function when the setup function has been run.
Did you mean "setup function hasn't been run." here?
As it is this sentence does not make any sense to me.
Yes, thanks!
Quoted 17 lines
Quoted 12 lines
And at least some network tests expect setup has been run before running cleanup (e.g.
tcp_fastopen_run.sh).When shell API was introduced, cleanup function was run only if 1) setup function was defined 2) and also run. That was inconsistent from C API, thus
e7dc14caarun it always.But shell API is different from C API:
tst_brkcan be called fromtst_test.sh(or other library which is run beforetst_run, e.g.tst_net.sh). That was probably the reason, why detection via $TST_SETUP_STARTED was introduced in initial shell API.Quoted 4 lines
NOTE: using type is better than grep $TST_TEST_PATH, because cleanup function can be in other library sourced by the test.
Fixes e7dc14caa ("tst_test.sh: Run cleanup even setup is not defined")
Quoted 7 lines
This still possibly break tests that would call
tst_brk()in the middle of setup and expect thecleanup()to be executed, right?I guess that we would need
TST_DO_CLEANUPthat would be set in both cases i.e. before we run setup and also before we execute the tests.What about this:
Good catch, agree.
I'm going to merge with one runtime fix:
Review
Quoted 2 lines
(quoted diff) +1
-1- if [ -n $TST_DO_CLEANUP -a -n "$TST_CLEANUP" -a -z "$TST_NO_CLEANUP" ]; then + if [ -n "$TST_DO_CLEANUP" -a -n "$TST_CLEANUP" -a -z "$TST_NO_CLEANUP" ]; then
Quotes are required otherwise we can get error:/opt/ltp/testcases/bin/tst_test.sh: line 31: [: too many arguments
Kind regards,
Petr
Quoted 29 lines
testcases/lib/tst_test.sh +3
-1@@ -28,7 +28,7 @@ _tst_do_exit() local ret=0 TST_DO_EXIT=1 - if [ -n "$TST_CLEANUP" -a -z "$TST_NO_CLEANUP" ]; then + if [ -n $TST_DO_CLEANUP -a -n "$TST_CLEANUP" -a -z "$TST_NO_CLEANUP" ]; then $TST_CLEANUP fi@@ -582,6 +582,7 @@ tst_run() [ -n "$TST_NEEDS_MODULE" ] && tst_require_module "$TST_NEEDS_MODULE" if [ -n "$TST_SETUP" ]; then + TST_DO_CLEANUP=1 $TST_SETUP fi@@ -608,6 +609,7 @@ _tst_run_tests() local _tst_data="$1" local _tst_i + TST_DO_CLEANUP=1 for _tst_i in $(seq ${TST_CNT:-1}); do if type ${TST_TESTFUNC}1 > /dev/null 2>&1; then _tst_run_test "$TST_TESTFUNC$_tst_i" $_tst_i "$_tst_data"
PATCH v4 1/3 tst_test.sh: Call cleanup function after test start
In reply to Cyril HrubisHi Cyril,
...
Quoted 7 lines
This still possibly break tests that would call
tst_brk()in the middle of setup and expect thecleanup()to be executed, right?I guess that we would need
TST_DO_CLEANUPthat would be set in both cases i.e. before we run setup and also before we execute the tests.What about this:
Whole patchset merged with your credit.
Thanks!
Kind regards,
Petr
PATCH v4 2/3 tst_test.sh: Warn on missing cleanup function
In reply to Petr Vorel.i.e. be more informative than:/opt/ltp/testcases/bin/tst_test.sh: line 33: cleanup: command not found
tst_brk does not make sense here, thus only warning.
--- testcases/lib/tst_test.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
testcases/lib/tst_test.sh +5 -1
@@ -29,7 +29,11 @@ _tst_do_exit() TST_DO_EXIT=1 if [ -n "$TST_RUN_STARTED" -a -n "$TST_CLEANUP" -a -z "$TST_NO_CLEANUP" ]; then - $TST_CLEANUP + if type $TST_CLEANUP >/dev/null 2>/dev/null; then + $TST_CLEANUP + else + tst_res TWARN "TST_CLEANUP=$TST_CLEANUP declared, but function not defined (or cmd not found)" + fi fi if [ "$TST_NEEDS_DEVICE" = 1 -a "$TST_DEVICE_FLAG" = 1 ]; then
Signature · 1 line
2.29.2
PATCH v4 2/3 tst_test.sh: Warn on missing cleanup function
Hi!
Looks good, Reviewed-by: Cyril Hrubis <[email protected]>
Signature · 2 lines
Cyril Hrubis [email protected]
PATCH v4 3/3 tst_test.sh: Exit when setup declared but not defined
In reply to Petr VorelIt can lead to fail positive when just warn about TST_SETUP declared but not defined (or cmd not found).
--- testcases/lib/tst_test.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
testcases/lib/tst_test.sh +5 -1
@@ -586,7 +586,11 @@ tst_run() [ -n "$TST_NEEDS_MODULE" ] && tst_require_module "$TST_NEEDS_MODULE" if [ -n "$TST_SETUP" ]; then - $TST_SETUP + if type $TST_SETUP >/dev/null 2>/dev/null; then + $TST_SETUP + else + tst_brk TBROK "TST_SETUP=$TST_SETUP declared, but function not defined (or cmd not found)" + fi fi #TODO check that test reports some results for each test function call
Signature · 1 line
2.29.2
PATCH v4 3/3 tst_test.sh: Exit when setup declared but not defined
Hi!
Looks good, Reviewed-by: Cyril Hrubis <[email protected]>
Signature · 2 lines
Cyril Hrubis [email protected]