Skip to contentSkip to conversation

libevent: add granularity to ptest log

PATCH In review Trevor Gamblin 2 participants 3 messages Sep 9, 2019

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 libevent: add granularity to ptest log

From: Trevor Gamblin <[email protected]>

The libevent ptest used to report only a global pass or a fail result. Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.

libevent's ptest output did not comply with the automake-style output "result: testname", and reported a FAIL status at the end of the test run if any of the libevent tests failed. This patch makes the log consistent with the automake style:

PASS: http/cancel_by_host_no_ns
PASS: http/cancel_inactive_server
PASS: http/cancel_by_host_no_ns_inactive_server
SKIP: http/cancel_by_host_server_timeout
SKIP: http/cancel_server_timeout

and provides a summary as follows:

    === Test Summary ===
    TOTAL: 316
    PASSED: 300
    FAILED: 0
    SKIPPED: 16
    DURATION: 87
    END: /usr/lib/libevent/ptest
---
 .../libevent/libevent/run-ptest               | 43 ++++++++++++-------
 .../libevent/libevent_2.1.11.bb               |  3 ++
 2 files changed, 31 insertions(+), 15 deletions(-)
meta/recipes-support/libevent/libevent/run-ptest +28 -15
index 0241851c70..79d6da3ee1 100644
@@ -1,18 +1,31 @@
 #!/bin/sh
 
-fail=0
-for test in ./test/*
-do
-	$test
-	if [ $? -ne 0 ]
-	then
-		fail=1
-	fi
-done
+# run-ptest - 'ptest' test infrastructure shell script that
+#   wraps the libevent test scripts 
+#
+# Trevor Gamblin <[email protected]>
+###############################################################
+LIBEVENTLIB=@libdir@/libevent
+LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log"
 
-if [ $fail -eq 0 ]
-then
-	echo "PASS: libevent"
-else
-	echo "FAIL: libevent"
-fi
+cd ${LIBEVENTLIB}/ptest 
+
+# Run the libevent regress test, but format the pass/fail outputs so
+# that ptest can count them. Ignore the line containing "tests or 
+# "TESTS" as that line is the libevent total at the end of the test 
+# and is counted as an extra test failure, e.g. 
+# "2/300 TESTS FAILED. (31 skipped)" will be seen as a test failure 
+# rather than a total
+./test/regress 2>&1| sed -e '/TESTS/d' -e '/tests/d' -e '/OK/ s/^/PASS: / ; /FAILED/ s/^/FAIL: / ; /SKIPPED/ s/^/SKIP: / ; /DISABLED/ s/^/SKIP: /' | cut -f1,2 -d ':' | tee -a ${LOG}
+
+passed=`grep PASS ${LOG}|wc -l`
+failed=`grep FAIL ${LOG}|wc -l`
+skipped=`grep -E SKIP ${LOG}|wc -l`
+all=$((passed + failed + skipped))
+
+(   echo "=== Test Summary ==="
+    echo "TOTAL: ${all}"
+    echo "PASSED: ${passed}"
+    echo "FAILED: ${failed}"
+    echo "SKIPPED: ${skipped}"
+) | tee -a ${LOG}
meta/recipes-support/libevent/libevent_2.1.11.bb +3 -0
index 1e18f0ab2c..f005ab8bda 100644
@@ -43,4 +43,7 @@ do_install_ptest() {
 	do
 		install -m 0755 $file ${D}${PTEST_PATH}/test
 	done
+        
+        # handle multilib
+        sed -i s:@libdir@:${libdir}:g ${D}${PTEST_PATH}/run-ptest
 }
Signature · 1 line
2.21.0
#1 lore raw
Quoted 89 lines · Trevor Gamblin

From: Trevor Gamblin <[email protected]>

The libevent ptest used to report only a global pass or a fail result. Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.

libevent's ptest output did not comply with the automake-style output "result: testname", and reported a FAIL status at the end of the test run if any of the libevent tests failed. This patch makes the log consistent with the automake style:

PASS: http/cancel_by_host_no_ns
PASS: http/cancel_inactive_server
PASS: http/cancel_by_host_no_ns_inactive_server
SKIP: http/cancel_by_host_server_timeout
SKIP: http/cancel_server_timeout

and provides a summary as follows:

     === Test Summary ===
     TOTAL: 316
     PASSED: 300
     FAILED: 0
     SKIPPED: 16
     DURATION: 87
     END: /usr/lib/libevent/ptest
---
  .../libevent/libevent/run-ptest               | 43 ++++++++++++-------
  .../libevent/libevent_2.1.11.bb               |  3 ++
  2 files changed, 31 insertions(+), 15 deletions(-)
meta/recipes-support/libevent/libevent/run-ptest +28 -15
index 0241851c70..79d6da3ee1 100644
@@ -1,18 +1,31 @@
  #!/bin/sh
 
-fail=0
-for test in ./test/*
-do
-	$test
-	if [ $? -ne 0 ]
-	then
-		fail=1
-	fi
-done
+# run-ptest - 'ptest' test infrastructure shell script that
+#   wraps the libevent test scripts
+#
+# Trevor Gamblin <[email protected]>
+###############################################################
+LIBEVENTLIB=@libdir@/libevent
+LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log"
 
-if [ $fail -eq 0 ]
-then
-	echo "PASS: libevent"
-else
-	echo "FAIL: libevent"
-fi
+cd ${LIBEVENTLIB}/ptest
+
+# Run the libevent regress test, but format the pass/fail outputs so
+# that ptest can count them. Ignore the line containing "tests or
+# "TESTS" as that line is the libevent total at the end of the test
+# and is counted as an extra test failure, e.g.
+# "2/300 TESTS FAILED. (31 skipped)" will be seen as a test failure
+# rather than a total
+./test/regress 2>&1| sed -e '/TESTS/d' -e '/tests/d' -e '/OK/ s/^/PASS: / ; /FAILED/ s/^/FAIL: / ; /SKIPPED/ s/^/SKIP: / ; /DISABLED/ s/^/SKIP: /' | cut -f1,2 -d ':' | tee -a ${LOG}
+
+passed=`grep PASS ${LOG}|wc -l`
+failed=`grep FAIL ${LOG}|wc -l`
+skipped=`grep -E SKIP ${LOG}|wc -l`
+all=$((passed + failed + skipped))
+
+(   echo "=== Test Summary ==="
+    echo "TOTAL: ${all}"
+    echo "PASSED: ${passed}"
+    echo "FAILED: ${failed}"
+    echo "SKIPPED: ${skipped}"
+) | tee -a ${LOG}
meta/recipes-support/libevent/libevent_2.1.11.bb +0 -0
index 1e18f0ab2c..f005ab8bda 100644

meta/recipes-support/libevent/libevent_2.1.11.bb do_install_ptest() { · line 49

Quoted 7 lines
meta/recipes-support/libevent/libevent_2.1.11.bb +3 -0
@@ -43,4 +43,7 @@ do_install_ptest() {
  	do
  		install -m 0755 $file ${D}${PTEST_PATH}/test
  	done
+
+        # handle multilib
+        sed -i s:@libdir@:${libdir}:g ${D}${PTEST_PATH}/run-ptest
  }

Noticed that the autobuilder results using v2 were detecting libevent's "2/300 TESTS FAILED. (31 skipped)" line at the end of the regress test as an extra failure. Fixed that up and reduced the ptest to just running "regress", as the other scripts in the test folder are more performance-oriented and don't provide a pass/fail result that can be used.

#2 lore raw

✗ patchtest: failure for libevent: add granularity to ptest log (rev2)

In reply to Trevor Gamblin

== Series Details ==

Series: libevent: add granularity to ptest log (rev2)
Revision: 2
URL : https://patchwork.openembedded.org/series/19759/
State : failure

== Summary ==

Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures:

* Issue Series does not apply on top of target branch [test_series_merge_on_head]
Suggested fix Rebase your series on top of targeted branch
Targeted branch master (currently at d41606244c)

If you believe any of these test results are incorrect, please reply to the mailing list ([email protected]) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...).

---

Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe