tst_test: fix again when test has both TPASS and TCONF
PATCH 1/3 tst_test: fix again when test has both TPASS and TCONF
Consider this situation: we get only TPASS and TCONF in test results, and the last status happens to be TPASS, the check condition in commit fecdd885eb4b ("tst_test: Fix exit value on tst_brk(TCONF, ...)" won't work. Fix it by removing unnecessary TCONF check.
--- lib/tst_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/tst_test.c +1 -1
@@ -594,7 +594,7 @@ static void do_exit(int ret)printf ("skipped %d\n" , results->skipped);printf ("warnings %d\n" , results->warnings); -if (results->passed && ret ==TCONF ) +if (results->passed) ret =0 ;if (results->failed)
Signature · 1 line
2.21.0
PATCH 2/3 lib/test.sh: TCONF needs to be counted
TCONF should also be one of exit statuses in a single test, else the output of TST_COUNT in shell tests could be wrong.
Wrong:
<<<test_output>>>memcg_use_hierarchy_test 1 TINFO: Starting test 1memcg_use_hierarchy_test 1 TINFO: set /dev/memcg/memory.use_hierarchy to 0 failed memcg_use_hierarchy_test 1 TPASS: process 28658 is killed memcg_use_hierarchy_test 2 TINFO: Starting test 2memcg_use_hierarchy_test 2 TINFO: set /dev/memcg/memory.use_hierarchy to 0 failed memcg_use_hierarchy_test 2 TCONF: memory.use_hierarchy already been 1, blame systemd, skip memcg_use_hierarchy_test 2 TINFO: Starting test 3memcg_use_hierarchy_test 2 TINFO: set /dev/memcg/memory.use_hierarchy to 0 failed memcg_use_hierarchy_test 2 TPASS: echo 0 > subgroup/memory.use_hierarchy failed as expected <<<execution_status>>>
Right:
<<<test_output>>>memcg_use_hierarchy_test 1 TINFO: Starting test 1memcg_use_hierarchy_test 1 TINFO: set /dev/memcg/memory.use_hierarchy to 0 failed memcg_use_hierarchy_test 1 TPASS: process 26825 is killed memcg_use_hierarchy_test 2 TINFO: Starting test 2memcg_use_hierarchy_test 2 TINFO: set /dev/memcg/memory.use_hierarchy to 0 failed memcg_use_hierarchy_test 2 TCONF: memory.use_hierarchy already been 1, blame systemd, skip memcg_use_hierarchy_test 3 TINFO: Starting test 3memcg_use_hierarchy_test 3 TINFO: set /dev/memcg/memory.use_hierarchy to 0 failed memcg_use_hierarchy_test 3 TPASS: echo 0 > subgroup/memory.use_hierarchy failed as expected <<<execution_status>>>
--- testcases/lib/test.sh | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
testcases/lib/test.sh +1 -2
@@ -58,8 +58,7 @@ tst_resm() echo " $@" case "$ret" in - TPASS|TFAIL) - TST_COUNT=$((TST_COUNT+1));; + TPASS|TFAIL|TCONF) TST_COUNT=$((TST_COUNT+1));; esac }
Signature · 1 line
2.21.0
PATCH 2/3 lib/test.sh: TCONF needs to be counted
Quoted 30 lines · Caspar Zhang
TCONF should also be one of exit statuses in a single test, else the output of
TST_COUNTin shell tests could be wrong.Wrong:
<<<test_output>>>memcg_use_hierarchy_test1 TINFO: Starting test 1memcg_use_hierarchy_test1 TINFO: set/dev/memcg/memory.use_hierarchyto 0 failedmemcg_use_hierarchy_test1 TPASS: process 28658 is killedmemcg_use_hierarchy_test2 TINFO: Starting test 2memcg_use_hierarchy_test2 TINFO: set/dev/memcg/memory.use_hierarchyto 0 failedmemcg_use_hierarchy_test2 TCONF: memory.use_hierarchy already been 1, blame systemd, skipmemcg_use_hierarchy_test2 TINFO: Starting test 3memcg_use_hierarchy_test2 TINFO: set/dev/memcg/memory.use_hierarchyto 0 failedmemcg_use_hierarchy_test2 TPASS: echo 0 > subgroup/memory.use_hierarchy failed as expected
<<<execution_status>>>Right:
<<<test_output>>>memcg_use_hierarchy_test1 TINFO: Starting test 1memcg_use_hierarchy_test1 TINFO: set/dev/memcg/memory.use_hierarchyto 0 failedmemcg_use_hierarchy_test1 TPASS: process 26825 is killedmemcg_use_hierarchy_test2 TINFO: Starting test 2memcg_use_hierarchy_test2 TINFO: set/dev/memcg/memory.use_hierarchyto 0 failed
Quoted 9 lines
memcg_use_hierarchy_test2 TCONF: memory.use_hierarchy already been 1, blame systemd, skipmemcg_use_hierarchy_test3 TINFO: Starting test 3memcg_use_hierarchy_test3 TINFO: set/dev/memcg/memory.use_hierarchyto 0 failedmemcg_use_hierarchy_test3 TPASS: echo 0 > subgroup/memory.use_hierarchy failed as expected
<<<execution_status>>>
This is a good catch, but maybe it's not wise to simply regard the TCONF as a single test, because there are many system-config detections in setup() function, that will make LTP gives a mendacious report on the test numbers if applying this patch.
e.g.
if tst_kvcmp -lt "3.10"; then
tst_brk TCONF "test must be run with kernel 3.10 or newer"
fi
if dir path not exist; then
tst_brk TCONF "system does not have xxxx/"
fi
and so on...
Quoted 27 lines
Signed-off-by Caspar Zhang <[email protected]>--- testcases/lib/test.sh | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)testcases/lib/test.sh +1
-3@@ -58,8 +58,7 @@ tst_resm() echo " $@" case "$ret" in - TPASS|TFAIL) - TST_COUNT=$((TST_COUNT+1));; + TPASS|TFAIL|TCONF) TST_COUNT=$((TST_COUNT+1));; esac } --2.21.0
--
Mailing list info: https://lists.linux.it/listinfo/ltp
Signature · 5 lines
Regards, Li Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.linux.it/pipermail/ltp/attachments/20190531/10b12894/attachment.html>
PATCH 2/3 lib/test.sh: TCONF needs to be counted
Quoted 52 lines · Li Wang
On Thu, May 30, 2019 at 5:10 PM Caspar Zhang <[1][email protected]> wrote:
TCONF should also be one of exit statuses in a single test, else the output of
TST_COUNTin shell tests could be wrong.Wrong:
<<<test_output>>>
memcg_use_hierarchy_test1 TINFO: Starting test 1memcg_use_hierarchy_test1 TINFO: set/dev/memcg/memory.use_hierarchyto 0 failed
memcg_use_hierarchy_test1 TPASS: process 28658 is killedmemcg_use_hierarchy_test2 TINFO: Starting test 2memcg_use_hierarchy_test2 TINFO: set/dev/memcg/memory.use_hierarchyto 0 failed
memcg_use_hierarchy_test2 TCONF: memory.use_hierarchy already been 1, blame systemd, skip
memcg_use_hierarchy_test2 TINFO: Starting test 3memcg_use_hierarchy_test2 TINFO: set/dev/memcg/memory.use_hierarchyto 0 failed
memcg_use_hierarchy_test2 TPASS: echo 0 > subgroup/memory.use_hierarchy failed as expected
<<<execution_status>>>Right:
<<<test_output>>>
memcg_use_hierarchy_test1 TINFO: Starting test 1memcg_use_hierarchy_test1 TINFO: set/dev/memcg/memory.use_hierarchyto 0 failed
memcg_use_hierarchy_test1 TPASS: process 26825 is killedmemcg_use_hierarchy_test2 TINFO: Starting test 2memcg_use_hierarchy_test2 TINFO: set/dev/memcg/memory.use_hierarchyto 0 failed
memcg_use_hierarchy_test2 TCONF: memory.use_hierarchy already been 1, blame systemd, skip
memcg_use_hierarchy_test3 TINFO: Starting test 3memcg_use_hierarchy_test3 TINFO: set/dev/memcg/memory.use_hierarchyto 0 failed
memcg_use_hierarchy_test3 TPASS: echo 0 > subgroup/memory.use_hierarchy failed as expected
<<<execution_status>>>This is a good catch, but maybe it's not wise to simply regard the TCONF as a single test, because there are many system-config detections in
setup()function, that will make LTP gives a mendacious report on the test numbers if applying this patch.e.g.
Quoted 7 lines
if
tst_kvcmp-lt "3.10"; then
tst_brkTCONF "test must be run with kernel 3.10 or newer" fi
if dir path not exist; then
tst_brkTCONF "system does not have xxxx/"
fi
and so on...
TCONF usually report only once, I would still take it a valid report on numbers. Take your case as example, I guess we are able to see results like:
mytest 1 TPASS: pass
or
mytest 1 TCONF: test must be run with kernel 3.10 or newer
or
mytest 1 TCONF: system does not have xxx/
Thanks,
Caspar
Quoted 36 lines
Signed-off-by: Caspar Zhang <[2][email protected]> --- testcases/lib/test.sh | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/testcases/lib/test.sh b/testcases/lib/test.sh index 670248ee5..ade8fcdff 100644 --- a/testcases/lib/test.sh +++ b/testcases/lib/test.sh @@ -58,8 +58,7 @@ tst_resm() echo " $@" case "$ret" in - TPASS|TFAIL) - TST_COUNT=$((TST_COUNT+1));; + TPASS|TFAIL|TCONF) TST_COUNT=$((TST_COUNT+1));; esac } -- 2.21.0 -- Mailing list info: [3]https://lists.linux.it/listinfo/ltp--
Regards,
Li Wang
Quoted 5 lines
References:
[1] mailto:[email protected]
[2] mailto:[email protected]
[3] https://lists.linux.it/listinfo/ltp
--
Thanks,
Caspar
PATCH 2/3 lib/test.sh: TCONF needs to be counted
Quoted 74 lines · Caspar Zhang
Quoted 3 lines · Li Wang
On Thu, May 30, 2019 at 5:10 PM Caspar Zhang <[1]
Quoted 9 lines
wrote:
TCONF should also be one of exit statuses in a single test, else the output of
TST_COUNTin shell tests could be wrong.Wrong:
<<<test_output>>>memcg_use_hierarchy_test1 TINFO: Starting test 1memcg_use_hierarchy_test1 TINFO: set
/dev/memcg/memory.use_hierarchyto 0Quoted 4 lines
failed
memcg_use_hierarchy_test1 TPASS: process 28658 is killedmemcg_use_hierarchy_test2 TINFO: Starting test 2memcg_use_hierarchy_test2 TINFO: set
/dev/memcg/memory.use_hierarchyto 0Quoted 2 lines
failed
memcg_use_hierarchy_test2 TCONF: memory.use_hierarchy already been1,
Quoted 3 lines
blame systemd, skip
memcg_use_hierarchy_test2 TINFO: Starting test 3memcg_use_hierarchy_test2 TINFO: set
/dev/memcg/memory.use_hierarchyto 0Quoted 2 lines
failed
memcg_use_hierarchy_test2 TPASS: echo 0 >subgroup/memory.use_hierarchy
Quoted 7 lines
failed as expected
<<<execution_status>>>Right:
<<<test_output>>>memcg_use_hierarchy_test1 TINFO: Starting test 1memcg_use_hierarchy_test1 TINFO: set
/dev/memcg/memory.use_hierarchyto 0Quoted 4 lines
failed
memcg_use_hierarchy_test1 TPASS: process 26825 is killedmemcg_use_hierarchy_test2 TINFO: Starting test 2memcg_use_hierarchy_test2 TINFO: set
/dev/memcg/memory.use_hierarchyto 0Quoted 2 lines
failed
memcg_use_hierarchy_test2 TCONF: memory.use_hierarchy already been1,
Quoted 3 lines
blame systemd, skip
memcg_use_hierarchy_test3 TINFO: Starting test 3memcg_use_hierarchy_test3 TINFO: set
/dev/memcg/memory.use_hierarchyto 0Quoted 2 lines
failed
memcg_use_hierarchy_test3 TPASS: echo 0 >subgroup/memory.use_hierarchy
Quoted 5 lines
failed as expected
<<<execution_status>>>This is a good catch, but maybe it's not wise to simply regard the TCONF
as a
Quoted 2 lines
single test, because there are many system-config detections in
setup()function, that will make LTP gives a mendacious report on the testnumbers if
Quoted 11 lines
applying this patch.
e.g.
if
tst_kvcmp-lt "3.10"; then
tst_brkTCONF "test must be run with kernel 3.10 or newer"
fi
if dir path not exist; then
tst_brkTCONF "system does not have xxxx/"
fi
and so on...
Quoted 4 lines
TCONF usually report only once, I would still take it a valid report on numbers. Take your case as example, I guess we are able to see results like:
Okay, that sounds reasonable too.
Signature · 5 lines
Regards, Li Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.linux.it/pipermail/ltp/attachments/20190610/e109fe9c/attachment.html>
PATCH 2/3 lib/test.sh: TCONF needs to be counted
In reply to Caspar ZhangHi Cyril, is it ok to push 2/3 and 3/3 as they've got Li Wang's Reviewed-by, or do you have different opinion?
Thanks,
Caspar
Quoted 49 lines · Caspar Zhang
TCONF should also be one of exit statuses in a single test, else the output of
TST_COUNTin shell tests could be wrong.Wrong:
<<<test_output>>>memcg_use_hierarchy_test1 TINFO: Starting test 1memcg_use_hierarchy_test1 TINFO: set/dev/memcg/memory.use_hierarchyto 0 failedmemcg_use_hierarchy_test1 TPASS: process 28658 is killedmemcg_use_hierarchy_test2 TINFO: Starting test 2memcg_use_hierarchy_test2 TINFO: set/dev/memcg/memory.use_hierarchyto 0 failedmemcg_use_hierarchy_test2 TCONF: memory.use_hierarchy already been 1, blame systemd, skipmemcg_use_hierarchy_test2 TINFO: Starting test 3memcg_use_hierarchy_test2 TINFO: set/dev/memcg/memory.use_hierarchyto 0 failedmemcg_use_hierarchy_test2 TPASS: echo 0 > subgroup/memory.use_hierarchy failed as expected <<<execution_status>>>Right:
<<<test_output>>>memcg_use_hierarchy_test1 TINFO: Starting test 1memcg_use_hierarchy_test1 TINFO: set/dev/memcg/memory.use_hierarchyto 0 failedmemcg_use_hierarchy_test1 TPASS: process 26825 is killedmemcg_use_hierarchy_test2 TINFO: Starting test 2memcg_use_hierarchy_test2 TINFO: set/dev/memcg/memory.use_hierarchyto 0 failedmemcg_use_hierarchy_test2 TCONF: memory.use_hierarchy already been 1, blame systemd, skipmemcg_use_hierarchy_test3 TINFO: Starting test 3memcg_use_hierarchy_test3 TINFO: set/dev/memcg/memory.use_hierarchyto 0 failedmemcg_use_hierarchy_test3 TPASS: echo 0 > subgroup/memory.use_hierarchy failed as expected <<<execution_status>>>Signed-off-by Caspar Zhang <[email protected]>--- testcases/lib/test.sh | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)testcases/lib/test.sh +1
-3@@ -58,8 +58,7 @@ tst_resm() echo " $@" case "$ret" in - TPASS|TFAIL) - TST_COUNT=$((TST_COUNT+1));; + TPASS|TFAIL|TCONF) TST_COUNT=$((TST_COUNT+1));; esac } --
Quoted 2 lines
2.21.0
--
Thanks,
Caspar
PATCH 2/3 lib/test.sh: TCONF needs to be counted
Hi!
Pushed thanks.
Quoted 2 lines
Hi Cyril, is it ok to push 2/3 and 3/3 as they've got Li Wang's Reviewed-by, or do you have different opinion?
I've checked and these two changes actually unify behavior of the shell libraries with the C libraries. We did miss the changes to the shell libraries when we modified the C libraries, so this changes make perfect sense.
Signature · 2 lines
Cyril Hrubis [email protected]
PATCH 3/3 tst_test.sh, test.sh: report pass if both TPASS and TCONF exist in shell test
In reply to Caspar ZhangWe don't want the test reporting TCONF if at least one TPASS existed during testing. And this is also true in shell tests. So fix it.
--- testcases/lib/test.sh | 9 +++++++++ testcases/lib/tst_test.sh | 2 +- 2 files changed, 10 insertions(+), 1 deletion(-)
testcases/lib/test.sh +9 -0
@@ -23,6 +23,7 @@ export LTP_RET_VAL=0 export TST_COUNT=1 +export TST_PASS_COUNT=0 export TST_LIB_LOADED=1 export TST_TMPDIR_RHOST=0
@@ -60,6 +61,10 @@ tst_resm() case "$ret" in TPASS|TFAIL|TCONF) TST_COUNT=$((TST_COUNT+1));; esac + + if [ "$ret" = TPASS ]; then + TST_PASS_COUNT=$((TST_PASS_COUNT+1)) + fi } tst_brkm()
@@ -111,6 +116,10 @@ tst_exit() rm -f "$LTP_IPC_PATH" fi + # Mask out TCONF if no TFAIL/TBROK/TWARN but has TPASS + if [ $((LTP_RET_VAL & 7)) -eq 0 -a $TST_PASS_COUNT -gt 0 ]; then + LTP_RET_VAL=$((LTP_RET_VAL & ~32)) + fi # Mask out TINFO exit $((LTP_RET_VAL & ~16)) }
testcases/lib/tst_test.sh +1 -1
@@ -78,7 +78,7 @@ _tst_do_exit() ret=$((ret|4)) fi - if [ $TST_CONF -gt 0 ]; then + if [ $TST_CONF -gt 0 -a $TST_PASS -eq 0 ]; then ret=$((ret|32)) fi
Signature · 1 line
2.21.0
PATCH 3/3 tst_test.sh, test.sh: report pass if both TPASS and TCONF exist in shell test
Quoted 3 lines · Caspar Zhang
We don't want the test reporting TCONF if at least one TPASS existed during testing. And this is also true in shell tests. So fix it.
PATCH 3/3 makes sense to me.
Quoted 61 lines
Signed-off-by Caspar Zhang <[email protected]>--- testcases/lib/test.sh | 9 +++++++++ testcases/lib/tst_test.sh | 2 +- 2 files changed, 10 insertions(+), 1 deletion(-)testcases/lib/test.sh +9
-0@@ -23,6 +23,7 @@ export LTP_RET_VAL=0 export TST_COUNT=1 +export TST_PASS_COUNT=0 export TST_LIB_LOADED=1 export TST_TMPDIR_RHOST=0@@ -60,6 +61,10 @@ tst_resm() case "$ret" in TPASS|TFAIL|TCONF) TST_COUNT=$((TST_COUNT+1));; esac + + if [ "$ret" = TPASS ]; then + TST_PASS_COUNT=$((TST_PASS_COUNT+1)) + fi } tst_brkm()@@ -111,6 +116,10 @@ tst_exit() rm -f "$LTP_IPC_PATH" fi + # Mask out TCONF if no TFAIL/TBROK/TWARN but has TPASS + if [ $((LTP_RET_VAL & 7)) -eq 0 -a $TST_PASS_COUNT -gt 0 ]; then + LTP_RET_VAL=$((LTP_RET_VAL & ~32)) + fi # Mask out TINFO exit $((LTP_RET_VAL & ~16)) }testcases/lib/tst_test.sh +1
-2@@ -78,7 +78,7 @@ _tst_do_exit() ret=$((ret|4)) fi - if [ $TST_CONF -gt 0 ]; then + if [ $TST_CONF -gt 0 -a $TST_PASS -eq 0 ]; then ret=$((ret|32)) fi --2.21.0
--
Mailing list info: https://lists.linux.it/listinfo/ltp
Signature · 5 lines
Regards, Li Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.linux.it/pipermail/ltp/attachments/20190531/9ea6f2e3/attachment-0001.html>
PATCH 1/3 tst_test: fix again when test has both TPASS and TCONF
In reply to Caspar ZhangHi!
Quoted 5 lines
Consider this situation: we get only TPASS and TCONF in test results, and the last status happens to be TPASS, the check condition in commit
fecdd885eb4b("tst_test: Fix exit value ontst_brk(TCONF, ...)" won't work. Fix it by removing unnecessary TCONF check.
What is the exact sequence of the tst_res()/tst_brk() calls leading to this?
Because if the test function has exitted with 0 the do_exit() function would be passed 0 and that case is handled already:
if (results->skipped && !results->passed) ret |= TCONF;
Quoted 17 lines
Fixes 5390d6ea3657 ("lib/tst_test: Report 0 if test has both TPASS and TCONF")Signed-off-by Caspar Zhang <[email protected]>--- lib/tst_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)lib/tst_test.c +1
-1@@ -594,7 +594,7 @@ static void do_exit(int ret)printf ("skipped %d\n" , results->skipped);printf ("warnings %d\n" , results->warnings); -if (results->passed && ret ==TCONF ) +if (results->passed) ret =0 ;
Signature · 2 lines
Cyril Hrubis [email protected]
PATCH 1/3 tst_test: fix again when test has both TPASS and TCONF
Quoted 7 lines · Cyril Hrubis
Hi!
Quoted 5 lines
Consider this situation: we get only TPASS and TCONF in test results, and the last status happens to be TPASS, the check condition in commit
fecdd885eb4b("tst_test: Fix exit value ontst_brk(TCONF, ...)" won't work. Fix it by removing unnecessary TCONF check.
Quoted 8 lines
What is the exact sequence of the tst_res()/tst_brk() calls leading to this?
Because if the test function has exitted with 0 the
do_exit()function would be passed 0 and that case is handled already:if (results->skipped && !results->passed) ret |= TCONF;
Hmm... you are right, the original fix was absolutely correct. Please feel free to ignore this patch. Sorry for the noise ;-)
Thanks,
Caspar
Quoted 15 lines
Quoted 15 lines
Fixes 5390d6ea3657 ("lib/tst_test: Report 0 if test has both TPASS and TCONF")Signed-off-by Caspar Zhang <[email protected]>--- lib/tst_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)lib/tst_test.c +0
-0@@ -594,7 +594,7 @@ static void do_exit(int ret)printf ("skipped %d\n" , results->skipped);printf ("warnings %d\n" , results->warnings);
Quoted 7 lines
lib/tst_test.c static void do_exit(int ret) · line 598
Quoted 3 lines
lib/tst_test.c +1
-1-if (results->passed && ret ==TCONF ) +if (results->passed) ret =0 ;--
Cyril Hrubis
[email protected]
--
Thanks,
Caspar