libevent: add granularity to ptest
PATCH 1/1 libevent: add granularity to ptest
From: Trevor Gamblin <[email protected]>
The libevent ptest used to report only a global pass or a fail result. Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.
--- .../libevent/libevent/run-ptest | 34 ++++++++++++------- 1 file changed, 22 insertions(+), 12 deletions(-)
meta/recipes-support/libevent/libevent/run-ptest +22 -12
@@ -1,18 +1,28 @@ #!/bin/sh -fail=0 +# run-ptest - 'ptest' test infrastructure shell script that +# wraps the libevent test scripts +# +# Trevor Gamblin <[email protected]> +############################################################### +LIBEVENTLIB=/usr/lib/libevent +LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log" + +cd ${LIBEVENTLIB}/ptest + for test in ./test/* do - $test - if [ $? -ne 0 ] - then - fail=1 - fi + $test 2>&1|tee -a ${LOG} done -if [ $fail -eq 0 ] -then - echo "PASS: libevent" -else - echo "FAIL: libevent" -fi +passed=`grep OK ${LOG}|wc -l` +failed=`grep FAILED ${LOG}|wc -l` +skipped=`grep -E 'DISABLED|SKIPPED' ${LOG}|wc -l` +all=$((passed + failed + skipped)) + +( echo "=== Test Summary ===" + echo "TOTAL: ${all}" + echo "PASSED: ${passed}" + echo "FAILED: ${failed}" + echo "SKIPPED: ${skipped}" +) | tee -a ${LOG}
Signature · 1 line
2.21.0
Still looking at the root cause of the libevent failures in the autobuilder, but this patch will provide more libevent ptest pass/fail/skip statistics (instead of reporting a pass only if all libevent tests pass), and possibly help determine if the pass rate is high enough for M3.
Quoted 56 lines · Trevor Gamblin
From: Trevor Gamblin <[email protected]>
The libevent ptest used to report only a global pass or a fail result. Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.
Signed-off-by Trevor Gamblin <[email protected]>--- .../libevent/libevent/run-ptest | 34 ++++++++++++------- 1 file changed, 22 insertions(+), 12 deletions(-)meta/recipes-support/libevent/libevent/run-ptest +22
-12@@ -1,18 +1,28 @@ #!/bin/sh -fail=0 +# run-ptest - 'ptest' test infrastructure shell script that +# wraps the libevent test scripts +# +# Trevor Gamblin <[email protected]> +############################################################### +LIBEVENTLIB=/usr/lib/libevent +LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log" + +cd ${LIBEVENTLIB}/ptest + for test in ./test/* do - $test - if [ $? -ne 0 ] - then - fail=1 - fi + $test 2>&1|tee -a ${LOG} done -if [ $fail -eq 0 ] -then - echo "PASS: libevent" -else - echo "FAIL: libevent" -fi +passed=`grep OK ${LOG}|wc -l` +failed=`grep FAILED ${LOG}|wc -l` +skipped=`grep -E 'DISABLED|SKIPPED' ${LOG}|wc -l` +all=$((passed + failed + skipped)) + +( echo "=== Test Summary ===" + echo "TOTAL: ${all}" + echo "PASSED: ${passed}" + echo "FAILED: ${failed}" + echo "SKIPPED: ${skipped}" +) | tee -a ${LOG}
Quoted 49 lines · Trevor Gamblin
From: Trevor Gamblin <[email protected]>
The libevent ptest used to report only a global pass or a fail result.
Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.Signed-off-by Trevor Gamblin <[email protected]>--- .../libevent/libevent/run-ptest | 34 ++++++++++++----- -- 1 file changed, 22 insertions(+), 12 deletions(-)meta/recipes-support/libevent/libevent/run-ptest +12
-12@@ -1,18 +1,28 @@ #!/bin/sh -fail=0 +# run-ptest - 'ptest' test infrastructure shell script that +# wraps the libevent test scripts +# +# Trevor Gamblin <[email protected]> +############################################################### +LIBEVENTLIB=/usr/lib/libevent +LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log" + +cd ${LIBEVENTLIB}/ptest + for test in ./test/* do - $test - if [ $? -ne 0 ] - then - fail=1 - fi + $test 2>&1|tee -a ${LOG} done -if [ $fail -eq 0 ] -then - echo "PASS: libevent" -else - echo "FAIL: libevent" -fi +passed=`grep OK ${LOG}|wc -l`
meta/recipes-support/libevent/libevent/run-ptest line 28
Quoted 10 lines
meta/recipes-support/libevent/libevent/run-ptest +10
-0+failed=`grep FAILED ${LOG}|wc -l` +skipped=`grep -E 'DISABLED|SKIPPED' ${LOG}|wc -l` +all=$((passed + failed + skipped)) + +( echo "=== Test Summary ===" + echo "TOTAL: ${all}" + echo "PASSED: ${passed}" + echo "FAILED: ${failed}" + echo "SKIPPED: ${skipped}" +) | tee -a ${LOG}
Does this work correctly such that ptest-runner picks up the individual test pass/fail results? I thought the tests would need individual pass/fail lines for that to work?
Cheers,
Richard
Quoted 64 lines · Richard Purdie
Quoted 49 lines · Trevor Gamblin
From: Trevor Gamblin <[email protected]>
The libevent ptest used to report only a global pass or a fail result.
Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.Signed-off-by Trevor Gamblin <[email protected]>--- .../libevent/libevent/run-ptest | 34 ++++++++++++----- -- 1 file changed, 22 insertions(+), 12 deletions(-)meta/recipes-support/libevent/libevent/run-ptest +12
-12@@ -1,18 +1,28 @@ #!/bin/sh -fail=0 +# run-ptest - 'ptest' test infrastructure shell script that +# wraps the libevent test scripts +# +# Trevor Gamblin <[email protected]> +############################################################### +LIBEVENTLIB=/usr/lib/libevent +LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log" + +cd ${LIBEVENTLIB}/ptest + for test in ./test/* do - $test - if [ $? -ne 0 ] - then - fail=1 - fi + $test 2>&1|tee -a ${LOG} done -if [ $fail -eq 0 ] -then - echo "PASS: libevent" -else - echo "FAIL: libevent" -fi +passed=`grep OK ${LOG}|wc -l`meta/recipes-support/libevent/libevent/run-ptest line 28
Quoted 10 lines
meta/recipes-support/libevent/libevent/run-ptest +10
-0+failed=`grep FAILED ${LOG}|wc -l` +skipped=`grep -E 'DISABLED|SKIPPED' ${LOG}|wc -l` +all=$((passed + failed + skipped)) + +( echo "=== Test Summary ===" + echo "TOTAL: ${all}" + echo "PASSED: ${passed}" + echo "FAILED: ${failed}" + echo "SKIPPED: ${skipped}" +) | tee -a ${LOG}Does this work correctly such that ptest-runner picks up the individual test pass/fail results? I thought the tests would need individual pass/fail lines for that to work?
Quoted 5 lines
Cheers,
Richard
They do have individual pass/fail results in the ptest-runner output, but those don't match up with the pass/fail usually provided by ptest, so the log is parsed for OK/FAILED/SKIPPED/DISABLED and translated. Example output:
|write_cb: write 12write_cb: write -1
=== Test Summary ===
TOTAL: 316
PASSED: 300
FAILED: 0
SKIPPED: 16
DURATION: 87
END: /usr/lib/libevent/ptest
2019-09-06T14:50
STOP: ptest-runner|
Compare with the last log from autobuilder (https://autobuilder.yocto.io/pub/non-release/20190829-10/testresults/qemux86-64-ptest/libevent.log):
|write_cb: write 12|| ||write_cb: write -1|| ||FAIL: libevent|| ||DURATION: 116|
||I'm working on a v2 to address multilib - I'll add the example output to the commit as well.
Quoted 91 lines · Trevor Gamblin
Quoted 68 lines · Richard Purdie
Quoted 51 lines · Trevor Gamblin
From: Trevor Gamblin <[email protected]>
The libevent ptest used to report only a global pass or a fail result.
Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.Signed-off-by Trevor Gamblin <[email protected]>--- .../libevent/libevent/run-ptest | 34 ++++++++++++- ---- -- 1 file changed, 22 insertions(+), 12 deletions(-)meta/recipes-support/libevent/libevent/run-ptest +7
-1@@ -1,18 +1,28 @@ #!/bin/sh -fail=0 +# run-ptest - 'ptest' test infrastructure shell script that +# wraps the libevent test scripts +# +# Trevor Gamblin <[email protected]> +############################################################### +LIBEVENTLIB=/usr/lib/libevent +LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log"
meta/recipes-support/libevent/libevent/run-ptest +5
-11+ +cd ${LIBEVENTLIB}/ptest + for test in ./test/* do - $test - if [ $? -ne 0 ] - then - fail=1 - fi + $test 2>&1|tee -a ${LOG} done -if [ $fail -eq 0 ] -then - echo "PASS: libevent" -else - echo "FAIL: libevent" -fi +passed=`grep OK ${LOG}|wc -l`meta/recipes-support/libevent/libevent/run-ptest line 28
Quoted 10 lines
meta/recipes-support/libevent/libevent/run-ptest +10
-0+failed=`grep FAILED ${LOG}|wc -l` +skipped=`grep -E 'DISABLED|SKIPPED' ${LOG}|wc -l` +all=$((passed + failed + skipped)) + +( echo "=== Test Summary ===" + echo "TOTAL: ${all}" + echo "PASSED: ${passed}" + echo "FAILED: ${failed}" + echo "SKIPPED: ${skipped}" +) | tee -a ${LOG}Does this work correctly such that ptest-runner picks up the individual
test pass/fail results? I thought the tests would need individual pass/fail lines for that to work?Quoted 5 lines
Cheers,
Richard
They do have individual pass/fail results in the ptest-runner output, but those don't match up with the pass/fail usually provided by ptest, so the log is parsed for OK/FAILED/SKIPPED/DISABLED and translated. Example output:
write_cb: write 12write_cb: write -1
=== Test Summary ===
TOTAL: 316
PASSED: 300
FAILED: 0
SKIPPED: 16
DURATION: 87
END:/usr/lib/libevent/ptest
2019-09-06T14:50
STOP: ptest-runner
Compare with the last log from autobuilder (
Quoted 8 lines
https://autobuilder.yocto.io/pub/non-release/20190829-10/testresults/qemux86-64-ptest/libevent.log
):write_cb: write 12write_cb: write -1
FAIL: libevent
DURATION: 116
I'm working on a v2 to address multilib - I'll add the example output to the commit as well.
You're missing my point. There are log parsers such as:
http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/utils/logparser.py
The output has to follow the format in:
https://wiki.yoctoproject.org/wiki/Ptest
to quote it:
"""
One major point of ptest is to consolidate the output format of all tests into a single common format. The format selected is the automake "simple test" format:
result: testname
Where "result" is one of PASS, FAIL or SKIP and "testname" can be any identifying string (the same format used by Automake.)
"""
which is why you'll see other run-ptest scripts using sed to add in PASS/FAIL/SKIP strings.
I think the PASSED/FAILED counts are generated by ptest-runner itself.
Cheers,
Richard
Quoted 113 lines · On 9/6/19 11:24 AM, [email protected] wrote:
Quoted 89 lines · Trevor Gamblin
Quoted 67 lines · Richard Purdie
Quoted 51 lines · Trevor Gamblin
From: Trevor Gamblin <[email protected]>
The libevent ptest used to report only a global pass or a fail result.
Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.Signed-off-by Trevor Gamblin <[email protected]>--- .../libevent/libevent/run-ptest | 34 ++++++++++++- ---- -- 1 file changed, 22 insertions(+), 12 deletions(-)meta/recipes-support/libevent/libevent/run-ptest +7
-1@@ -1,18 +1,28 @@ #!/bin/sh -fail=0 +# run-ptest - 'ptest' test infrastructure shell script that +# wraps the libevent test scripts +# +# Trevor Gamblin <[email protected]> +############################################################### +LIBEVENTLIB=/usr/lib/libevent +LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log"
meta/recipes-support/libevent/libevent/run-ptest +5
-11+ +cd ${LIBEVENTLIB}/ptest + for test in ./test/* do - $test - if [ $? -ne 0 ] - then - fail=1 - fi + $test 2>&1|tee -a ${LOG} done -if [ $fail -eq 0 ] -then - echo "PASS: libevent" -else - echo "FAIL: libevent" -fi +passed=`grep OK ${LOG}|wc -l`meta/recipes-support/libevent/libevent/run-ptest line 28
Quoted 10 lines
meta/recipes-support/libevent/libevent/run-ptest +10
-0+failed=`grep FAILED ${LOG}|wc -l` +skipped=`grep -E 'DISABLED|SKIPPED' ${LOG}|wc -l` +all=$((passed + failed + skipped)) + +( echo "=== Test Summary ===" + echo "TOTAL: ${all}" + echo "PASSED: ${passed}" + echo "FAILED: ${failed}" + echo "SKIPPED: ${skipped}" +) | tee -a ${LOG}Does this work correctly such that ptest-runner picks up the individual
test pass/fail results? I thought the tests would need individual pass/fail lines for that to work?Quoted 5 lines
Cheers,
Richard
They do have individual pass/fail results in the ptest-runner output, but those don't match up with the pass/fail usually provided by ptest, so the log is parsed for OK/FAILED/SKIPPED/DISABLED and translated. Example output:
write_cb: write 12write_cb: write -1
=== Test Summary ===
TOTAL: 316
PASSED: 300
FAILED: 0
SKIPPED: 16
DURATION: 87
END:/usr/lib/libevent/ptest
2019-09-06T14:50
STOP: ptest-runner
Compare with the last log from autobuilder (Quoted 8 lines
https://autobuilder.yocto.io/pub/non-release/20190829-10/testresults/qemux86-64-ptest/libevent.log
):write_cb: write 12write_cb: write -1
FAIL: libevent
DURATION: 116
I'm working on a v2 to address multilib - I'll add the example output to the commit as well.You're missing my point. There are log parsers such as:
http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/utils/logparser.py
The output has to follow the format in:
https://wiki.yoctoproject.org/wiki/Ptest
to quote it:
"""
One major point of ptest is to consolidate the output format of all tests into a single common format. The format selected is the automake "simple test" format:
Quoted 8 lines
result: testname
Where "result" is one of PASS, FAIL or SKIP and "testname" can be any identifying string (the same format used by Automake.)
"""which is why you'll see other run-ptest scripts using sed to add in PASS/FAIL/SKIP strings.
Noted - fixing and adding this in for v2.
Quoted 2 lines
I think the PASSED/FAILED counts are generated by ptest-runner itself.
I didn't see this in the source. I'll leave the summary in for v2.
Quoted 7 lines
Cheers,
Richard
Quoted 17 lines · Trevor Gamblin
From: Trevor Gamblin <[email protected]>
The libevent ptest used to report only a global pass or a fail result. Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.
Signed-off-by Trevor Gamblin <[email protected]>--- .../libevent/libevent/run-ptest | 34 ++++++++++++------- 1 file changed, 22 insertions(+), 12 deletions(-)meta/recipes-support/libevent/libevent/run-ptest +0
-0
meta/recipes-support/libevent/libevent/run-ptest line 7
Quoted 8 lines
meta/recipes-support/libevent/libevent/run-ptest +5
-1@@ -1,18 +1,28 @@ #!/bin/sh -fail=0 +# run-ptest - 'ptest' test infrastructure shell script that +# wraps the libevent test scripts +# +# Trevor Gamblin <[email protected]> +###############################################################
+LIBEVENTLIB=/usr/lib/libevent
Quoted 1 line
Do we need to account for multilib here ?
+LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log"
Quoted 39 lines
meta/recipes-support/libevent/libevent/run-ptest +15
-12+ +cd ${LIBEVENTLIB}/ptest + for test in ./test/* do - $test - if [ $? -ne 0 ] - then - fail=1 - fi + $test 2>&1|tee -a ${LOG} done -if [ $fail -eq 0 ] -then - echo "PASS: libevent" -else - echo "FAIL: libevent" -fi +passed=`grep OK ${LOG}|wc -l` +failed=`grep FAILED ${LOG}|wc -l` +skipped=`grep -E 'DISABLED|SKIPPED' ${LOG}|wc -l` +all=$((passed + failed + skipped)) + +( echo "=== Test Summary ===" + echo "TOTAL: ${all}" + echo "PASSED: ${passed}" + echo "FAILED: ${failed}" + echo "SKIPPED: ${skipped}" +) | tee -a ${LOG} --2.21.0
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core