libevent: add granularity to ptest log
PATCH v2 1/1 libevent: add granularity to ptest log
From: Trevor Gamblin <[email protected]>
The libevent ptest used to report only a global pass or a fail result. Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.
libevent's ptest output did not comply with the automake-style output "result: testname", and reported a FAIL status at the end of the test run if any of the libevent tests failed. This patch makes the log consistent with the automake style:
PASS: http/cancel_by_host_no_ns
PASS: http/cancel_inactive_server
PASS: http/cancel_by_host_no_ns_inactive_server
SKIPPED: http/cancel_by_host_server_timeout
SKIPPED: http/cancel_server_timeout
and provides a summary as follows:
=== Test Summary ===
TOTAL: 316
PASSED: 300
FAILED: 0
SKIPPED: 16
DURATION: 87
END: /usr/lib/libevent/ptest--- .../libevent/libevent/run-ptest | 34 ++++++++++++------- .../libevent/libevent_2.1.11.bb | 3 ++ 2 files changed, 25 insertions(+), 12 deletions(-)
meta/recipes-support/libevent/libevent/run-ptest +22 -12
@@ -1,18 +1,28 @@ #!/bin/sh -fail=0 +# run-ptest - 'ptest' test infrastructure shell script that +# wraps the libevent test scripts +# +# Trevor Gamblin <[email protected]> +############################################################### +LIBEVENTLIB=@libdir@/libevent +LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log" + +cd ${LIBEVENTLIB}/ptest + for test in ./test/* do - $test - if [ $? -ne 0 ] - then - fail=1 - fi + $test 2>&1| sed -e '/OK/ s/^/PASS: / ; /FAILED/ s/^/FAIL: / ; /SKIPPED/ s/^/SKIP: / ; /DISABLED/ s/^/SKIP: /' | cut -f1,2 -d ':' | tee -a ${LOG} done -if [ $fail -eq 0 ] -then - echo "PASS: libevent" -else - echo "FAIL: libevent" -fi +passed=`grep PASS ${LOG}|wc -l` +failed=`grep FAIL ${LOG}|wc -l` +skipped=`grep -E SKIP ${LOG}|wc -l` +all=$((passed + failed + skipped)) + +( echo "=== Test Summary ===" + echo "TOTAL: ${all}" + echo "PASSED: ${passed}" + echo "FAILED: ${failed}" + echo "SKIPPED: ${skipped}" +) | tee -a ${LOG}
meta/recipes-support/libevent/libevent_2.1.11.bb +3 -0
@@ -43,4 +43,7 @@ do_install_ptest() { do install -m 0755 $file ${D}${PTEST_PATH}/test done + + # handle multilib + sed -i s:@libdir@:${libdir}:g ${D}${PTEST_PATH}/run-ptest }
Signature · 1 line
2.21.0
Quoted 83 lines · Trevor Gamblin
From: Trevor Gamblin <[email protected]>
The libevent ptest used to report only a global pass or a fail result. Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.
libevent's ptest output did not comply with the automake-style output "result: testname", and reported a FAIL status at the end of the test run if any of the libevent tests failed. This patch makes the log consistent with the automake style:
PASS: http/cancel_by_host_no_ns
PASS: http/cancel_inactive_server
PASS: http/cancel_by_host_no_ns_inactive_server
SKIPPED: http/cancel_by_host_server_timeout
SKIPPED: http/cancel_server_timeoutand provides a summary as follows:
=== Test Summary === TOTAL: 316 PASSED: 300 FAILED: 0 SKIPPED: 16 DURATION: 87 END: /usr/lib/libevent/ptestSigned-off-by Trevor Gamblin <[email protected]>--- .../libevent/libevent/run-ptest | 34 ++++++++++++------- .../libevent/libevent_2.1.11.bb | 3 ++ 2 files changed, 25 insertions(+), 12 deletions(-)meta/recipes-support/libevent/libevent/run-ptest +22
-12@@ -1,18 +1,28 @@ #!/bin/sh -fail=0 +# run-ptest - 'ptest' test infrastructure shell script that +# wraps the libevent test scripts +# +# Trevor Gamblin <[email protected]> +############################################################### +LIBEVENTLIB=@libdir@/libevent +LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log" + +cd ${LIBEVENTLIB}/ptest + for test in ./test/* do - $test - if [ $? -ne 0 ] - then - fail=1 - fi + $test 2>&1| sed -e '/OK/ s/^/PASS: / ; /FAILED/ s/^/FAIL: / ; /SKIPPED/ s/^/SKIP: / ; /DISABLED/ s/^/SKIP: /' | cut -f1,2 -d ':' | tee -a ${LOG} done -if [ $fail -eq 0 ] -then - echo "PASS: libevent" -else - echo "FAIL: libevent" -fi +passed=`grep PASS ${LOG}|wc -l` +failed=`grep FAIL ${LOG}|wc -l` +skipped=`grep -E SKIP ${LOG}|wc -l` +all=$((passed + failed + skipped)) + +( echo "=== Test Summary ===" + echo "TOTAL: ${all}" + echo "PASSED: ${passed}" + echo "FAILED: ${failed}" + echo "SKIPPED: ${skipped}" +) | tee -a ${LOG}meta/recipes-support/libevent/libevent_2.1.11.bb +0
-0
meta/recipes-support/libevent/libevent_2.1.11.bb do_install_ptest() { · line 49
Quoted 7 lines
meta/recipes-support/libevent/libevent_2.1.11.bb +3
-0@@ -43,4 +43,7 @@ do_install_ptest() { do install -m 0755 $file ${D}${PTEST_PATH}/test done + + # handle multilib + sed -i s:@libdir@:${libdir}:g ${D}${PTEST_PATH}/run-ptest }
ptest logs are currently still including some performance test outputs in addition to the PASS/FAIL stuff we are concerned with. That being said, I figured it'd be better to get a working patch in to use on a test run before the weekend. Example of the performance output below:||
|||write callback. should only see this once|| ||timeout fired, time to end test|| ||usec used=80, usec passed=1501872, cpu usage=0.01%|| ||closed_cb: detected socket close with success|| ||=====expected|| ||Now= 1567793437.757017|| ||Inserted:|| || 0x405940 [fd 3] Write|| || 0x4059d0 [fd 4] Read Persist Timeout=T+1|| || 0x405a60 [fd 5] Write Persist Timeout=T+1|| || 0x405af0 [fd 6] Read Timeout=T+2.5|| || 0x405b80 [fd -1] Timeout=T+1|| || 0x405c10 [fd -1] Timeout=T+1|| || 0x405ca0 [fd -1] Timeout=T+1|| || 0x405d30 [fd -1] Persist Timeout=T+2.5|| || 0x405dc0 [fd -1] Persist Timeout=T+1|| || 0x405e50 [fd -1] Persist Timeout=T+2.5|| || 0x406000 [sig 2] Signal Persist|| ||Active:|| || 0x405ee0 [fd -1, priority=0] Read active|| || 0x405f70 [fd -1, priority=0] Read Write Timeout active|| || 0x4059d0 [fd 4, priority=0] Read active|| ||======received|| ||Inserted events:|| || 0x405940 [fd 3] Write|| || 0x4059d0 [fd 4] Read Persist Timeout=1567793438.756966|| || 0x405a60 [fd 5] Write Persist Timeout=1567793438.756966|| || 0x405af0 [fd 6] Read Timeout=1567793440.256966|| || 0x405308 [fd 8] Read Persist Internal|| || 0x406000 [sig 2] Signal Persist|| || 0x405800 [fd -1] Internal Timeout=1567793438.756966|| || 0x4058b0 [fd -1] Internal Timeout=1567793440.256966|| || 0x405ca0 [fd -1] Timeout=1567793438.756966|| || 0x405e50 [fd -1] Persist Timeout=1567793440.256966|| || 0x405b80 [fd -1] Timeout=1567793438.756966|| || 0x405c10 [fd -1] Timeout=1567793438.756966|| || 0x405dc0 [fd -1] Persist Timeout=1567793438.756966|| || 0x405d30 [fd -1] Persist Timeout=1567793440.256966|| ||Active events:|| || 0x4059d0 [fd 4, priority=0] Read active|| || 0x405ee0 [fd -1, priority=0] Read active|| || 0x405f70 [fd -1, priority=0] Read Write Timeout active|| ||read_cb: read 12|| ||read_cb: read 0 - means EOF|| ||waiting for 30 conns|| ||Average group read bucket level: 0.000000|| ||Average group write bucket level: 0.000000|| ||1: 683014.200000 per second|| ||2: 4389273.600000 per second|| ||3: 683014.200000 per second|| ||4: 687929.400000 per second|| ||5: 683014.200000 per second|| ||6: 592950.200000 per second|| ||7: 683014.200000 per second|| ||8: 687929.400000 per second|| ||9: 1433587.200000 per second|| ||10: 17691353.400000 per second|| ||11: 1433590.400000 per second|| ||12: 17667606.200000 per second|| ||13: 1202576.000000 per second|| ||14: 592950.200000 per second|| ||15: 592950.200000 per second|| ||16: 17667606.200000 per second|| ||17: 1199100.600000 per second|| ||18: 16940233.600000 per second|| ||19: 1402460.800000 per second|| ||20: 17667606.200000 per second|| ||21: 17670063.800000 per second|| ||22: 592950.200000 per second|| ||23: 592950.200000 per second|| ||24: 17667606.200000 per second|| ||25: 17670063.800000 per second|| ||26: 17667606.200000 per second|| ||27: 17699551.800000 per second|| ||28: 17697907.000000 per second|| ||29: 17699551.800000 per second|| ||30: 1216486.400000 per second|| || total: 230756497.800000 per second|| || average: 7691883.260000 per second|| || stddev: 8132709.287741 per second|| ||209005, 20000|| ||write_cb: write 12|| ||write_cb: write -1|
At least some of the libevent test scripts output only this content (i.e. they don't run any of the PASS/FAIL/SKIP tests), so I'll look into dropping those from the ptest unless there's a reason we'd want to leave them in. ||||||
Quoted 180 lines · Trevor Gamblin
Quoted 83 lines · Trevor Gamblin
From: Trevor Gamblin<[email protected]>
The libevent ptest used to report only a global pass or a fail result. Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.
libevent's ptest output did not comply with the automake-style output "result: testname", and reported a FAIL status at the end of the test run if any of the libevent tests failed. This patch makes the log consistent with the automake style:
PASS: http/cancel_by_host_no_ns
PASS: http/cancel_inactive_server
PASS: http/cancel_by_host_no_ns_inactive_server
SKIPPED: http/cancel_by_host_server_timeout
SKIPPED: http/cancel_server_timeoutand provides a summary as follows:
=== Test Summary === TOTAL: 316 PASSED: 300 FAILED: 0 SKIPPED: 16 DURATION: 87 END: /usr/lib/libevent/ptestSigned-off-by Trevor Gamblin<[email protected]>--- .../libevent/libevent/run-ptest | 34 ++++++++++++------- .../libevent/libevent_2.1.11.bb | 3 ++ 2 files changed, 25 insertions(+), 12 deletions(-)meta/recipes-support/libevent/libevent/run-ptest +22
-12@@ -1,18 +1,28 @@ #!/bin/sh -fail=0 +# run-ptest - 'ptest' test infrastructure shell script that +# wraps the libevent test scripts +# +# Trevor Gamblin<[email protected]> +############################################################### +LIBEVENTLIB=@libdir@/libevent +LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log" + +cd ${LIBEVENTLIB}/ptest + for test in ./test/* do - $test - if [ $? -ne 0 ] - then - fail=1 - fi + $test 2>&1| sed -e '/OK/ s/^/PASS: / ; /FAILED/ s/^/FAIL: / ; /SKIPPED/ s/^/SKIP: / ; /DISABLED/ s/^/SKIP: /' | cut -f1,2 -d ':' | tee -a ${LOG} done -if [ $fail -eq 0 ] -then - echo "PASS: libevent" -else - echo "FAIL: libevent" -fi +passed=`grep PASS ${LOG}|wc -l` +failed=`grep FAIL ${LOG}|wc -l` +skipped=`grep -E SKIP ${LOG}|wc -l` +all=$((passed + failed + skipped)) + +( echo "=== Test Summary ===" + echo "TOTAL: ${all}" + echo "PASSED: ${passed}" + echo "FAILED: ${failed}" + echo "SKIPPED: ${skipped}" +) | tee -a ${LOG}meta/recipes-support/libevent/libevent_2.1.11.bb +0
-0meta/recipes-support/libevent/libevent_2.1.11.bb do_install_ptest() { · line 49
Quoted 7 lines
meta/recipes-support/libevent/libevent_2.1.11.bb +3
-0@@ -43,4 +43,7 @@ do_install_ptest() { do install -m 0755 $file ${D}${PTEST_PATH}/test done + + # handle multilib + sed -i s:@libdir@:${libdir}:g ${D}${PTEST_PATH}/run-ptest }ptest logs are currently still including some performance test outputs in addition to the PASS/FAIL stuff we are concerned with. That being said, I figured it'd be better to get a working patch in to use on a test run before the weekend. Example of the performance output below:||
|||write callback. should only see this once|| ||timeout fired, time to end test|| ||usec used=80, usec passed=1501872, cpu usage=0.01%|| ||closed_cb: detected socket close with success|| ||=====expected|| ||Now= 1567793437.757017|| ||Inserted:|| || 0x405940 [fd 3] Write|| || 0x4059d0 [fd 4] Read Persist Timeout=T+1|| || 0x405a60 [fd 5] Write Persist Timeout=T+1|| || 0x405af0 [fd 6] Read Timeout=T+2.5|| || 0x405b80 [fd -1] Timeout=T+1|| || 0x405c10 [fd -1] Timeout=T+1|| || 0x405ca0 [fd -1] Timeout=T+1|| || 0x405d30 [fd -1] Persist Timeout=T+2.5|| || 0x405dc0 [fd -1] Persist Timeout=T+1|| || 0x405e50 [fd -1] Persist Timeout=T+2.5|| || 0x406000 [sig 2] Signal Persist|| ||Active:|| || 0x405ee0 [fd -1, priority=0] Read active|| || 0x405f70 [fd -1, priority=0] Read Write Timeout active|| || 0x4059d0 [fd 4, priority=0] Read active|| ||======received|| ||Inserted events:|| || 0x405940 [fd 3] Write|| || 0x4059d0 [fd 4] Read Persist Timeout=1567793438.756966|| || 0x405a60 [fd 5] Write Persist Timeout=1567793438.756966|| || 0x405af0 [fd 6] Read Timeout=1567793440.256966|| || 0x405308 [fd 8] Read Persist Internal|| || 0x406000 [sig 2] Signal Persist|| || 0x405800 [fd -1] Internal Timeout=1567793438.756966|| || 0x4058b0 [fd -1] Internal Timeout=1567793440.256966|| || 0x405ca0 [fd -1] Timeout=1567793438.756966|| || 0x405e50 [fd -1] Persist Timeout=1567793440.256966|| || 0x405b80 [fd -1] Timeout=1567793438.756966|| || 0x405c10 [fd -1] Timeout=1567793438.756966|| || 0x405dc0 [fd -1] Persist Timeout=1567793438.756966|| || 0x405d30 [fd -1] Persist Timeout=1567793440.256966|| ||Active events:|| || 0x4059d0 [fd 4, priority=0] Read active|| || 0x405ee0 [fd -1, priority=0] Read active|| || 0x405f70 [fd -1, priority=0] Read Write Timeout active|| ||read_cb: read 12|| ||read_cb: read 0 - means EOF|| ||waiting for 30 conns|| ||Average group read bucket level: 0.000000|| ||Average group write bucket level: 0.000000|| ||1: 683014.200000 per second|| ||2: 4389273.600000 per second|| ||3: 683014.200000 per second|| ||4: 687929.400000 per second|| ||5: 683014.200000 per second|| ||6: 592950.200000 per second|| ||7: 683014.200000 per second|| ||8: 687929.400000 per second|| ||9: 1433587.200000 per second|| ||10: 17691353.400000 per second|| ||11: 1433590.400000 per second|| ||12: 17667606.200000 per second|| ||13: 1202576.000000 per second|| ||14: 592950.200000 per second|| ||15: 592950.200000 per second|| ||16: 17667606.200000 per second|| ||17: 1199100.600000 per second|| ||18: 16940233.600000 per second|| ||19: 1402460.800000 per second|| ||20: 17667606.200000 per second|| ||21: 17670063.800000 per second|| ||22: 592950.200000 per second|| ||23: 592950.200000 per second|| ||24: 17667606.200000 per second|| ||25: 17670063.800000 per second|| ||26: 17667606.200000 per second|| ||27: 17699551.800000 per second|| ||28: 17697907.000000 per second|| ||29: 17699551.800000 per second|| ||30: 1216486.400000 per second|| || total: 230756497.800000 per second|| || average: 7691883.260000 per second|| || stddev: 8132709.287741 per second|| ||209005, 20000|| ||write_cb: write 12|| ||write_cb: write -1|
Quoted 4 lines
At least some of the libevent test scripts output only this content (i.e. they don't run any of the PASS/FAIL/SKIP tests), so I'll look into dropping those from the ptest unless there's a reason we'd want to leave them in. ||||||
Just determined that only the libevent "regress" test is involved in the PASS/FAIL tests of interest. I'll submit a v3 on Monday.
Quoted 28 lines · Trevor Gamblin
From: Trevor Gamblin <[email protected]>
The libevent ptest used to report only a global pass or a fail result.
Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.libevent's ptest output did not comply with the automake-style output "result: testname", and reported a FAIL status at the end of the test run if any of the libevent tests failed. This patch makes the log consistent with the automake style:
PASS: http/cancel_by_host_no_ns
PASS: http/cancel_inactive_server
PASS: http/cancel_by_host_no_ns_inactive_server
SKIPPED: http/cancel_by_host_server_timeout
SKIPPED: http/cancel_server_timeoutand provides a summary as follows:
=== Test Summary === TOTAL: 316 PASSED: 300 FAILED: 0 SKIPPED: 16 DURATION: 87 END: /usr/lib/libevent/ptest
Quoted 1 line
Signed-off-by Trevor Gamblin <[email protected]>
Thanks for this, it is a definite improvement on what was there before. Results from the autobuilder run:
https://autobuilder.yocto.io/pub/non-release/20190907-4/testresults/testresult-report.txt
Cheers,
Richard