Skip to contentSkip to conversation

libevent: add granularity to ptest log

PATCH Superseded Trevor Gamblin 2 participants 4 messages Sep 6, 2019 – Sep 7, 2019

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 libevent: add granularity to ptest log

From: Trevor Gamblin <[email protected]>

The libevent ptest used to report only a global pass or a fail result. Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.

libevent's ptest output did not comply with the automake-style output "result: testname", and reported a FAIL status at the end of the test run if any of the libevent tests failed. This patch makes the log consistent with the automake style:

PASS: http/cancel_by_host_no_ns
PASS: http/cancel_inactive_server
PASS: http/cancel_by_host_no_ns_inactive_server
SKIPPED: http/cancel_by_host_server_timeout
SKIPPED: http/cancel_server_timeout

and provides a summary as follows:

    === Test Summary ===
    TOTAL: 316
    PASSED: 300
    FAILED: 0
    SKIPPED: 16
    DURATION: 87
    END: /usr/lib/libevent/ptest
---
 .../libevent/libevent/run-ptest               | 34 ++++++++++++-------
 .../libevent/libevent_2.1.11.bb               |  3 ++
 2 files changed, 25 insertions(+), 12 deletions(-)
meta/recipes-support/libevent/libevent/run-ptest +22 -12
index 0241851c70..080806dea9 100644
@@ -1,18 +1,28 @@
 #!/bin/sh
 
-fail=0
+# run-ptest - 'ptest' test infrastructure shell script that
+#   wraps the libevent test scripts 
+#
+# Trevor Gamblin <[email protected]>
+###############################################################
+LIBEVENTLIB=@libdir@/libevent
+LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log"
+
+cd ${LIBEVENTLIB}/ptest 
+
 for test in ./test/*
 do
-	$test
-	if [ $? -ne 0 ]
-	then
-		fail=1
-	fi
+    $test 2>&1| sed -e '/OK/ s/^/PASS: / ; /FAILED/ s/^/FAIL: / ; /SKIPPED/ s/^/SKIP: / ; /DISABLED/ s/^/SKIP: /' | cut -f1,2 -d ':' | tee -a ${LOG}
 done
 
-if [ $fail -eq 0 ]
-then
-	echo "PASS: libevent"
-else
-	echo "FAIL: libevent"
-fi
+passed=`grep PASS ${LOG}|wc -l`
+failed=`grep FAIL ${LOG}|wc -l`
+skipped=`grep -E SKIP ${LOG}|wc -l`
+all=$((passed + failed + skipped))
+
+(   echo "=== Test Summary ==="
+    echo "TOTAL: ${all}"
+    echo "PASSED: ${passed}"
+    echo "FAILED: ${failed}"
+    echo "SKIPPED: ${skipped}"
+) | tee -a ${LOG}
meta/recipes-support/libevent/libevent_2.1.11.bb +3 -0
index 1e18f0ab2c..f005ab8bda 100644
@@ -43,4 +43,7 @@ do_install_ptest() {
 	do
 		install -m 0755 $file ${D}${PTEST_PATH}/test
 	done
+        
+        # handle multilib
+        sed -i s:@libdir@:${libdir}:g ${D}${PTEST_PATH}/run-ptest
 }
Signature · 1 line
2.21.0
#1 lore raw
Quoted 83 lines · Trevor Gamblin

From: Trevor Gamblin <[email protected]>

The libevent ptest used to report only a global pass or a fail result. Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.

libevent's ptest output did not comply with the automake-style output "result: testname", and reported a FAIL status at the end of the test run if any of the libevent tests failed. This patch makes the log consistent with the automake style:

PASS: http/cancel_by_host_no_ns
PASS: http/cancel_inactive_server
PASS: http/cancel_by_host_no_ns_inactive_server
SKIPPED: http/cancel_by_host_server_timeout
SKIPPED: http/cancel_server_timeout

and provides a summary as follows:

     === Test Summary ===
     TOTAL: 316
     PASSED: 300
     FAILED: 0
     SKIPPED: 16
     DURATION: 87
     END: /usr/lib/libevent/ptest
---
  .../libevent/libevent/run-ptest               | 34 ++++++++++++-------
  .../libevent/libevent_2.1.11.bb               |  3 ++
  2 files changed, 25 insertions(+), 12 deletions(-)
meta/recipes-support/libevent/libevent/run-ptest +22 -12
index 0241851c70..080806dea9 100644
@@ -1,18 +1,28 @@
  #!/bin/sh
 
-fail=0
+# run-ptest - 'ptest' test infrastructure shell script that
+#   wraps the libevent test scripts
+#
+# Trevor Gamblin <[email protected]>
+###############################################################
+LIBEVENTLIB=@libdir@/libevent
+LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log"
+
+cd ${LIBEVENTLIB}/ptest
+
  for test in ./test/*
  do
-	$test
-	if [ $? -ne 0 ]
-	then
-		fail=1
-	fi
+    $test 2>&1| sed -e '/OK/ s/^/PASS: / ; /FAILED/ s/^/FAIL: / ; /SKIPPED/ s/^/SKIP: / ; /DISABLED/ s/^/SKIP: /' | cut -f1,2 -d ':' | tee -a ${LOG}
  done
 
-if [ $fail -eq 0 ]
-then
-	echo "PASS: libevent"
-else
-	echo "FAIL: libevent"
-fi
+passed=`grep PASS ${LOG}|wc -l`
+failed=`grep FAIL ${LOG}|wc -l`
+skipped=`grep -E SKIP ${LOG}|wc -l`
+all=$((passed + failed + skipped))
+
+(   echo "=== Test Summary ==="
+    echo "TOTAL: ${all}"
+    echo "PASSED: ${passed}"
+    echo "FAILED: ${failed}"
+    echo "SKIPPED: ${skipped}"
+) | tee -a ${LOG}
meta/recipes-support/libevent/libevent_2.1.11.bb +0 -0
index 1e18f0ab2c..f005ab8bda 100644

meta/recipes-support/libevent/libevent_2.1.11.bb do_install_ptest() { · line 49

Quoted 7 lines
meta/recipes-support/libevent/libevent_2.1.11.bb +3 -0
@@ -43,4 +43,7 @@ do_install_ptest() {
  	do
  		install -m 0755 $file ${D}${PTEST_PATH}/test
  	done
+
+        # handle multilib
+        sed -i s:@libdir@:${libdir}:g ${D}${PTEST_PATH}/run-ptest
  }

ptest logs are currently still including some performance test outputs in addition to the PASS/FAIL stuff we are concerned with. That being said, I figured it'd be better to get a working patch in to use on a test run before the weekend. Example of the performance output below:||

|||write callback. should only see this once||
||timeout fired, time to end test||
||usec used=80, usec passed=1501872, cpu usage=0.01%||
||closed_cb: detected socket close with success||
||=====expected||
||Now= 1567793437.757017||
||Inserted:||
||  0x405940 [fd  3] Write||
||  0x4059d0 [fd  4] Read Persist Timeout=T+1||
||  0x405a60 [fd  5] Write Persist Timeout=T+1||
||  0x405af0 [fd  6] Read Timeout=T+2.5||
||  0x405b80 [fd  -1] Timeout=T+1||
||  0x405c10 [fd  -1] Timeout=T+1||
||  0x405ca0 [fd  -1] Timeout=T+1||
||  0x405d30 [fd  -1] Persist Timeout=T+2.5||
||  0x405dc0 [fd  -1] Persist Timeout=T+1||
||  0x405e50 [fd  -1] Persist Timeout=T+2.5||
||  0x406000 [sig 2] Signal Persist||
||Active:||
||  0x405ee0 [fd  -1, priority=0] Read active||
||  0x405f70 [fd  -1, priority=0] Read Write Timeout active||
||  0x4059d0 [fd  4, priority=0] Read active||
||======received||
||Inserted events:||
||  0x405940 [fd  3] Write||
||  0x4059d0 [fd  4] Read Persist Timeout=1567793438.756966||
||  0x405a60 [fd  5] Write Persist Timeout=1567793438.756966||
||  0x405af0 [fd  6] Read Timeout=1567793440.256966||
||  0x405308 [fd  8] Read Persist Internal||
||  0x406000 [sig 2] Signal Persist||
||  0x405800 [fd  -1] Internal Timeout=1567793438.756966||
||  0x4058b0 [fd  -1] Internal Timeout=1567793440.256966||
||  0x405ca0 [fd  -1] Timeout=1567793438.756966||
||  0x405e50 [fd  -1] Persist Timeout=1567793440.256966||
||  0x405b80 [fd  -1] Timeout=1567793438.756966||
||  0x405c10 [fd  -1] Timeout=1567793438.756966||
||  0x405dc0 [fd  -1] Persist Timeout=1567793438.756966||
||  0x405d30 [fd  -1] Persist Timeout=1567793440.256966||
||Active events:||
||  0x4059d0 [fd  4, priority=0] Read active||
||  0x405ee0 [fd  -1, priority=0] Read active||
||  0x405f70 [fd  -1, priority=0] Read Write Timeout active||
||read_cb: read 12||
||read_cb: read 0 - means EOF||
||waiting for 30 conns||
||Average group read bucket level: 0.000000||
||Average group write bucket level: 0.000000||
||1: 683014.200000 per second||
||2: 4389273.600000 per second||
||3: 683014.200000 per second||
||4: 687929.400000 per second||
||5: 683014.200000 per second||
||6: 592950.200000 per second||
||7: 683014.200000 per second||
||8: 687929.400000 per second||
||9: 1433587.200000 per second||
||10: 17691353.400000 per second||
||11: 1433590.400000 per second||
||12: 17667606.200000 per second||
||13: 1202576.000000 per second||
||14: 592950.200000 per second||
||15: 592950.200000 per second||
||16: 17667606.200000 per second||
||17: 1199100.600000 per second||
||18: 16940233.600000 per second||
||19: 1402460.800000 per second||
||20: 17667606.200000 per second||
||21: 17670063.800000 per second||
||22: 592950.200000 per second||
||23: 592950.200000 per second||
||24: 17667606.200000 per second||
||25: 17670063.800000 per second||
||26: 17667606.200000 per second||
||27: 17699551.800000 per second||
||28: 17697907.000000 per second||
||29: 17699551.800000 per second||
||30: 1216486.400000 per second||
||   total: 230756497.800000 per second||
|| average: 7691883.260000 per second||
||  stddev: 8132709.287741 per second||
||209005, 20000||
||write_cb: write 12||
||write_cb: write -1|

At least some of the libevent test scripts output only this content (i.e. they don't run any of the PASS/FAIL/SKIP tests), so I'll look into dropping those from the ptest unless there's a reason we'd want to leave them in. ||||||

#2 lore raw
Quoted 180 lines · Trevor Gamblin
Quoted 83 lines · Trevor Gamblin

From: Trevor Gamblin<[email protected]>

The libevent ptest used to report only a global pass or a fail result. Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.

libevent's ptest output did not comply with the automake-style output "result: testname", and reported a FAIL status at the end of the test run if any of the libevent tests failed. This patch makes the log consistent with the automake style:

PASS: http/cancel_by_host_no_ns
PASS: http/cancel_inactive_server
PASS: http/cancel_by_host_no_ns_inactive_server
SKIPPED: http/cancel_by_host_server_timeout
SKIPPED: http/cancel_server_timeout

and provides a summary as follows:

     === Test Summary ===
     TOTAL: 316
     PASSED: 300
     FAILED: 0
     SKIPPED: 16
     DURATION: 87
     END: /usr/lib/libevent/ptest
---
  .../libevent/libevent/run-ptest               | 34 ++++++++++++-------
  .../libevent/libevent_2.1.11.bb               |  3 ++
  2 files changed, 25 insertions(+), 12 deletions(-)
meta/recipes-support/libevent/libevent/run-ptest +22 -12
index 0241851c70..080806dea9 100644
@@ -1,18 +1,28 @@
  #!/bin/sh
 
-fail=0
+# run-ptest - 'ptest' test infrastructure shell script that
+#   wraps the libevent test scripts
+#
+# Trevor Gamblin<[email protected]>
+###############################################################
+LIBEVENTLIB=@libdir@/libevent
+LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log"
+
+cd ${LIBEVENTLIB}/ptest
+
  for test in ./test/*
  do
-	$test
-	if [ $? -ne 0 ]
-	then
-		fail=1
-	fi
+    $test 2>&1| sed -e '/OK/ s/^/PASS: / ; /FAILED/ s/^/FAIL: / ; /SKIPPED/ s/^/SKIP: / ; /DISABLED/ s/^/SKIP: /' | cut -f1,2 -d ':' | tee -a ${LOG}
  done
 
-if [ $fail -eq 0 ]
-then
-	echo "PASS: libevent"
-else
-	echo "FAIL: libevent"
-fi
+passed=`grep PASS ${LOG}|wc -l`
+failed=`grep FAIL ${LOG}|wc -l`
+skipped=`grep -E SKIP ${LOG}|wc -l`
+all=$((passed + failed + skipped))
+
+(   echo "=== Test Summary ==="
+    echo "TOTAL: ${all}"
+    echo "PASSED: ${passed}"
+    echo "FAILED: ${failed}"
+    echo "SKIPPED: ${skipped}"
+) | tee -a ${LOG}
meta/recipes-support/libevent/libevent_2.1.11.bb +0 -0
index 1e18f0ab2c..f005ab8bda 100644

meta/recipes-support/libevent/libevent_2.1.11.bb do_install_ptest() { · line 49

Quoted 7 lines
meta/recipes-support/libevent/libevent_2.1.11.bb +3 -0
@@ -43,4 +43,7 @@ do_install_ptest() {
  	do
  		install -m 0755 $file ${D}${PTEST_PATH}/test
  	done
+
+        # handle multilib
+        sed -i s:@libdir@:${libdir}:g ${D}${PTEST_PATH}/run-ptest
  }

ptest logs are currently still including some performance test outputs in addition to the PASS/FAIL stuff we are concerned with. That being said, I figured it'd be better to get a working patch in to use on a test run before the weekend. Example of the performance output below:||

|||write callback. should only see this once||
||timeout fired, time to end test||
||usec used=80, usec passed=1501872, cpu usage=0.01%||
||closed_cb: detected socket close with success||
||=====expected||
||Now= 1567793437.757017||
||Inserted:||
||  0x405940 [fd  3] Write||
||  0x4059d0 [fd  4] Read Persist Timeout=T+1||
||  0x405a60 [fd  5] Write Persist Timeout=T+1||
||  0x405af0 [fd  6] Read Timeout=T+2.5||
||  0x405b80 [fd  -1] Timeout=T+1||
||  0x405c10 [fd  -1] Timeout=T+1||
||  0x405ca0 [fd  -1] Timeout=T+1||
||  0x405d30 [fd  -1] Persist Timeout=T+2.5||
||  0x405dc0 [fd  -1] Persist Timeout=T+1||
||  0x405e50 [fd  -1] Persist Timeout=T+2.5||
||  0x406000 [sig 2] Signal Persist||
||Active:||
||  0x405ee0 [fd  -1, priority=0] Read active||
||  0x405f70 [fd  -1, priority=0] Read Write Timeout active||
||  0x4059d0 [fd  4, priority=0] Read active||
||======received||
||Inserted events:||
||  0x405940 [fd  3] Write||
||  0x4059d0 [fd  4] Read Persist Timeout=1567793438.756966||
||  0x405a60 [fd  5] Write Persist Timeout=1567793438.756966||
||  0x405af0 [fd  6] Read Timeout=1567793440.256966||
||  0x405308 [fd  8] Read Persist Internal||
||  0x406000 [sig 2] Signal Persist||
||  0x405800 [fd  -1] Internal Timeout=1567793438.756966||
||  0x4058b0 [fd  -1] Internal Timeout=1567793440.256966||
||  0x405ca0 [fd  -1] Timeout=1567793438.756966||
||  0x405e50 [fd  -1] Persist Timeout=1567793440.256966||
||  0x405b80 [fd  -1] Timeout=1567793438.756966||
||  0x405c10 [fd  -1] Timeout=1567793438.756966||
||  0x405dc0 [fd  -1] Persist Timeout=1567793438.756966||
||  0x405d30 [fd  -1] Persist Timeout=1567793440.256966||
||Active events:||
||  0x4059d0 [fd  4, priority=0] Read active||
||  0x405ee0 [fd  -1, priority=0] Read active||
||  0x405f70 [fd  -1, priority=0] Read Write Timeout active||
||read_cb: read 12||
||read_cb: read 0 - means EOF||
||waiting for 30 conns||
||Average group read bucket level: 0.000000||
||Average group write bucket level: 0.000000||
||1: 683014.200000 per second||
||2: 4389273.600000 per second||
||3: 683014.200000 per second||
||4: 687929.400000 per second||
||5: 683014.200000 per second||
||6: 592950.200000 per second||
||7: 683014.200000 per second||
||8: 687929.400000 per second||
||9: 1433587.200000 per second||
||10: 17691353.400000 per second||
||11: 1433590.400000 per second||
||12: 17667606.200000 per second||
||13: 1202576.000000 per second||
||14: 592950.200000 per second||
||15: 592950.200000 per second||
||16: 17667606.200000 per second||
||17: 1199100.600000 per second||
||18: 16940233.600000 per second||
||19: 1402460.800000 per second||
||20: 17667606.200000 per second||
||21: 17670063.800000 per second||
||22: 592950.200000 per second||
||23: 592950.200000 per second||
||24: 17667606.200000 per second||
||25: 17670063.800000 per second||
||26: 17667606.200000 per second||
||27: 17699551.800000 per second||
||28: 17697907.000000 per second||
||29: 17699551.800000 per second||
||30: 1216486.400000 per second||
||   total: 230756497.800000 per second||
|| average: 7691883.260000 per second||
||  stddev: 8132709.287741 per second||
||209005, 20000||
||write_cb: write 12||
||write_cb: write -1|
Quoted 4 lines

At least some of the libevent test scripts output only this content (i.e. they don't run any of the PASS/FAIL/SKIP tests), so I'll look into dropping those from the ptest unless there's a reason we'd want to leave them in. ||||||

Just determined that only the libevent "regress" test is involved in the PASS/FAIL tests of interest. I'll submit a v3 on Monday.

#3 lore raw
In reply to Trevor Gamblin
Quoted 28 lines · Trevor Gamblin

From: Trevor Gamblin <[email protected]>

The libevent ptest used to report only a global pass or a fail result.
Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.

libevent's ptest output did not comply with the automake-style output "result: testname", and reported a FAIL status at the end of the test run if any of the libevent tests failed. This patch makes the log consistent with the automake style:

PASS: http/cancel_by_host_no_ns
PASS: http/cancel_inactive_server
PASS: http/cancel_by_host_no_ns_inactive_server
SKIPPED: http/cancel_by_host_server_timeout
SKIPPED: http/cancel_server_timeout

and provides a summary as follows:

    === Test Summary ===
    TOTAL: 316
    PASSED: 300
    FAILED: 0
    SKIPPED: 16
    DURATION: 87
    END: /usr/lib/libevent/ptest
Quoted 1 line

Thanks for this, it is a definite improvement on what was there before. Results from the autobuilder run:

https://autobuilder.yocto.io/pub/non-release/20190907-4/testresults/testresult-report.txt

Cheers,

Richard