Skip to contentSkip to conversation

libevent: add granularity to ptest

PATCH Superseded Trevor Gamblin 3 participants 7 messages Sep 6, 2019

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 libevent: add granularity to ptest

From: Trevor Gamblin <[email protected]>

The libevent ptest used to report only a global pass or a fail result. Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.

---
 .../libevent/libevent/run-ptest               | 34 ++++++++++++-------
 1 file changed, 22 insertions(+), 12 deletions(-)
meta/recipes-support/libevent/libevent/run-ptest +22 -12
index 0241851c70..b7d945246f 100644
@@ -1,18 +1,28 @@
 #!/bin/sh
 
-fail=0
+# run-ptest - 'ptest' test infrastructure shell script that
+#   wraps the libevent test scripts 
+#
+# Trevor Gamblin <[email protected]>
+###############################################################
+LIBEVENTLIB=/usr/lib/libevent
+LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log"
+
+cd ${LIBEVENTLIB}/ptest 
+
 for test in ./test/*
 do
-	$test
-	if [ $? -ne 0 ]
-	then
-		fail=1
-	fi
+    $test 2>&1|tee -a ${LOG}
 done
 
-if [ $fail -eq 0 ]
-then
-	echo "PASS: libevent"
-else
-	echo "FAIL: libevent"
-fi
+passed=`grep OK ${LOG}|wc -l`
+failed=`grep FAILED ${LOG}|wc -l`
+skipped=`grep -E 'DISABLED|SKIPPED' ${LOG}|wc -l`
+all=$((passed + failed + skipped))
+
+(   echo "=== Test Summary ==="
+    echo "TOTAL: ${all}"
+    echo "PASSED: ${passed}"
+    echo "FAILED: ${failed}"
+    echo "SKIPPED: ${skipped}"
+) | tee -a ${LOG}
Signature · 1 line
2.21.0
#1 lore raw

Still looking at the root cause of the libevent failures in the autobuilder, but this patch will provide more libevent ptest pass/fail/skip statistics (instead of reporting a pass only if all libevent tests pass), and possibly help determine if the pass rate is high enough for M3.

Quoted 56 lines · Trevor Gamblin

From: Trevor Gamblin <[email protected]>

The libevent ptest used to report only a global pass or a fail result. Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.

---
  .../libevent/libevent/run-ptest               | 34 ++++++++++++-------
  1 file changed, 22 insertions(+), 12 deletions(-)
meta/recipes-support/libevent/libevent/run-ptest +22 -12
index 0241851c70..b7d945246f 100644
@@ -1,18 +1,28 @@
  #!/bin/sh
 
-fail=0
+# run-ptest - 'ptest' test infrastructure shell script that
+#   wraps the libevent test scripts
+#
+# Trevor Gamblin <[email protected]>
+###############################################################
+LIBEVENTLIB=/usr/lib/libevent
+LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log"
+
+cd ${LIBEVENTLIB}/ptest
+
  for test in ./test/*
  do
-	$test
-	if [ $? -ne 0 ]
-	then
-		fail=1
-	fi
+    $test 2>&1|tee -a ${LOG}
  done
 
-if [ $fail -eq 0 ]
-then
-	echo "PASS: libevent"
-else
-	echo "FAIL: libevent"
-fi
+passed=`grep OK ${LOG}|wc -l`
+failed=`grep FAILED ${LOG}|wc -l`
+skipped=`grep -E 'DISABLED|SKIPPED' ${LOG}|wc -l`
+all=$((passed + failed + skipped))
+
+(   echo "=== Test Summary ==="
+    echo "TOTAL: ${all}"
+    echo "PASSED: ${passed}"
+    echo "FAILED: ${failed}"
+    echo "SKIPPED: ${skipped}"
+) | tee -a ${LOG}
#2 lore raw
In reply to Trevor Gamblin
Quoted 49 lines · Trevor Gamblin

From: Trevor Gamblin <[email protected]>

The libevent ptest used to report only a global pass or a fail result.
Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.

---
 .../libevent/libevent/run-ptest               | 34 ++++++++++++-----
--
 1 file changed, 22 insertions(+), 12 deletions(-)
meta/recipes-support/libevent/libevent/run-ptest +12 -12
index 0241851c70..b7d945246f 100644
@@ -1,18 +1,28 @@
 #!/bin/sh
 
-fail=0
+# run-ptest - 'ptest' test infrastructure shell script that
+#   wraps the libevent test scripts 
+#
+# Trevor Gamblin <[email protected]>
+###############################################################
+LIBEVENTLIB=/usr/lib/libevent
+LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log"
+
+cd ${LIBEVENTLIB}/ptest 
+
 for test in ./test/*
 do
-	$test
-	if [ $? -ne 0 ]
-	then
-		fail=1
-	fi
+    $test 2>&1|tee -a ${LOG}
 done
 
-if [ $fail -eq 0 ]
-then
-	echo "PASS: libevent"
-else
-	echo "FAIL: libevent"
-fi
+passed=`grep OK ${LOG}|wc -l`

meta/recipes-support/libevent/libevent/run-ptest line 28

Quoted 10 lines
meta/recipes-support/libevent/libevent/run-ptest +10 -0
+failed=`grep FAILED ${LOG}|wc -l`
+skipped=`grep -E 'DISABLED|SKIPPED' ${LOG}|wc -l`
+all=$((passed + failed + skipped))
+
+(   echo "=== Test Summary ==="
+    echo "TOTAL: ${all}"
+    echo "PASSED: ${passed}"
+    echo "FAILED: ${failed}"
+    echo "SKIPPED: ${skipped}"
+) | tee -a ${LOG}

Does this work correctly such that ptest-runner picks up the individual test pass/fail results? I thought the tests would need individual pass/fail lines for that to work?

Cheers,

Richard

#3 lore raw
Quoted 64 lines · Richard Purdie
Quoted 49 lines · Trevor Gamblin

From: Trevor Gamblin <[email protected]>

The libevent ptest used to report only a global pass or a fail result.
Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.

---
  .../libevent/libevent/run-ptest               | 34 ++++++++++++-----
--
  1 file changed, 22 insertions(+), 12 deletions(-)
meta/recipes-support/libevent/libevent/run-ptest +12 -12
index 0241851c70..b7d945246f 100644
@@ -1,18 +1,28 @@
  #!/bin/sh
 
-fail=0
+# run-ptest - 'ptest' test infrastructure shell script that
+#   wraps the libevent test scripts
+#
+# Trevor Gamblin <[email protected]>
+###############################################################
+LIBEVENTLIB=/usr/lib/libevent
+LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log"
+
+cd ${LIBEVENTLIB}/ptest
+
  for test in ./test/*
  do
-	$test
-	if [ $? -ne 0 ]
-	then
-		fail=1
-	fi
+    $test 2>&1|tee -a ${LOG}
  done
 
-if [ $fail -eq 0 ]
-then
-	echo "PASS: libevent"
-else
-	echo "FAIL: libevent"
-fi
+passed=`grep OK ${LOG}|wc -l`

meta/recipes-support/libevent/libevent/run-ptest line 28

Quoted 10 lines
meta/recipes-support/libevent/libevent/run-ptest +10 -0
+failed=`grep FAILED ${LOG}|wc -l`
+skipped=`grep -E 'DISABLED|SKIPPED' ${LOG}|wc -l`
+all=$((passed + failed + skipped))
+
+(   echo "=== Test Summary ==="
+    echo "TOTAL: ${all}"
+    echo "PASSED: ${passed}"
+    echo "FAILED: ${failed}"
+    echo "SKIPPED: ${skipped}"
+) | tee -a ${LOG}

Does this work correctly such that ptest-runner picks up the individual test pass/fail results? I thought the tests would need individual pass/fail lines for that to work?

Quoted 5 lines

Cheers,

Richard

They do have individual pass/fail results in the ptest-runner output, but those don't match up with the pass/fail usually provided by ptest, so the log is parsed for OK/FAILED/SKIPPED/DISABLED and translated. Example output:

|write_cb: write 12
write_cb: write -1
=== Test Summary ===
TOTAL: 316
PASSED: 300
FAILED: 0
SKIPPED: 16
DURATION: 87
END: /usr/lib/libevent/ptest
2019-09-06T14:50
STOP: ptest-runner|

Compare with the last log from autobuilder (https://autobuilder.yocto.io/pub/non-release/20190829-10/testresults/qemux86-64-ptest/libevent.log):

|write_cb: write 12||
||write_cb: write -1||
||FAIL: libevent||
||DURATION: 116|

||I'm working on a v2 to address multilib - I'll add the example output to the commit as well.

#4 lore raw
Quoted 91 lines · Trevor Gamblin
Quoted 68 lines · Richard Purdie
Quoted 51 lines · Trevor Gamblin

From: Trevor Gamblin <[email protected]>

The libevent ptest used to report only a global pass or a fail result.
Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.

---
 .../libevent/libevent/run-ptest               | 34 ++++++++++++-
----
--
 1 file changed, 22 insertions(+), 12 deletions(-)
meta/recipes-support/libevent/libevent/run-ptest +7 -1
index 0241851c70..b7d945246f 100644
@@ -1,18 +1,28 @@
 #!/bin/sh
 
-fail=0
+# run-ptest - 'ptest' test infrastructure shell script that
+#   wraps the libevent test scripts 
+#
+# Trevor Gamblin <[email protected]>
+###############################################################
+LIBEVENTLIB=/usr/lib/libevent
+LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-

%H%M%S).log"

meta/recipes-support/libevent/libevent/run-ptest +5 -11
+
+cd ${LIBEVENTLIB}/ptest 
+
 for test in ./test/*
 do
-	$test
-	if [ $? -ne 0 ]
-	then
-		fail=1
-	fi
+    $test 2>&1|tee -a ${LOG}
 done
 
-if [ $fail -eq 0 ]
-then
-	echo "PASS: libevent"
-else
-	echo "FAIL: libevent"
-fi
+passed=`grep OK ${LOG}|wc -l`

meta/recipes-support/libevent/libevent/run-ptest line 28

Quoted 10 lines
meta/recipes-support/libevent/libevent/run-ptest +10 -0
+failed=`grep FAILED ${LOG}|wc -l`
+skipped=`grep -E 'DISABLED|SKIPPED' ${LOG}|wc -l`
+all=$((passed + failed + skipped))
+
+(   echo "=== Test Summary ==="
+    echo "TOTAL: ${all}"
+    echo "PASSED: ${passed}"
+    echo "FAILED: ${failed}"
+    echo "SKIPPED: ${skipped}"
+) | tee -a ${LOG}

Does this work correctly such that ptest-runner picks up the individual
test pass/fail results? I thought the tests would need individual pass/fail lines for that to work?

Quoted 5 lines

Cheers,

Richard

They do have individual pass/fail results in the ptest-runner output, but those don't match up with the pass/fail usually provided by ptest, so the log is parsed for OK/FAILED/SKIPPED/DISABLED and translated. Example output:
write_cb: write 12
write_cb: write -1
=== Test Summary ===
TOTAL: 316
PASSED: 300
FAILED: 0
SKIPPED: 16
DURATION: 87
END: /usr/lib/libevent/ptest
2019-09-06T14:50
STOP: ptest-runner
Compare with the last log from autobuilder (

Quoted 8 lines

https://autobuilder.yocto.io/pub/non-release/20190829-10/testresults/qemux86-64-ptest/libevent.log
):
write_cb: write 12
write_cb: write -1
FAIL: libevent
DURATION: 116
I'm working on a v2 to address multilib - I'll add the example output to the commit as well.

You're missing my point. There are log parsers such as:

http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/utils/logparser.py

The output has to follow the format in:

https://wiki.yoctoproject.org/wiki/Ptest

to quote it:

"""
One major point of ptest is to consolidate the output format of all tests into a single common format. The format selected is the automake "simple test" format:

result: testname

Where "result" is one of PASS, FAIL or SKIP and "testname" can be any identifying string (the same format used by Automake.)
"""

which is why you'll see other run-ptest scripts using sed to add in PASS/FAIL/SKIP strings.

I think the PASSED/FAILED counts are generated by ptest-runner itself.

Cheers,

Richard

#5 lore raw
Quoted 113 lines · On 9/6/19 11:24 AM, [email protected] wrote:
Quoted 89 lines · Trevor Gamblin
Quoted 67 lines · Richard Purdie
Quoted 51 lines · Trevor Gamblin

From: Trevor Gamblin <[email protected]>

The libevent ptest used to report only a global pass or a fail result.
Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.

---
  .../libevent/libevent/run-ptest               | 34 ++++++++++++-
----
--
  1 file changed, 22 insertions(+), 12 deletions(-)
meta/recipes-support/libevent/libevent/run-ptest +7 -1
index 0241851c70..b7d945246f 100644
@@ -1,18 +1,28 @@
  #!/bin/sh
 
-fail=0
+# run-ptest - 'ptest' test infrastructure shell script that
+#   wraps the libevent test scripts
+#
+# Trevor Gamblin <[email protected]>
+###############################################################
+LIBEVENTLIB=/usr/lib/libevent
+LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-

%H%M%S).log"

meta/recipes-support/libevent/libevent/run-ptest +5 -11
+
+cd ${LIBEVENTLIB}/ptest
+
  for test in ./test/*
  do
-	$test
-	if [ $? -ne 0 ]
-	then
-		fail=1
-	fi
+    $test 2>&1|tee -a ${LOG}
  done
 
-if [ $fail -eq 0 ]
-then
-	echo "PASS: libevent"
-else
-	echo "FAIL: libevent"
-fi
+passed=`grep OK ${LOG}|wc -l`

meta/recipes-support/libevent/libevent/run-ptest line 28

Quoted 10 lines
meta/recipes-support/libevent/libevent/run-ptest +10 -0
+failed=`grep FAILED ${LOG}|wc -l`
+skipped=`grep -E 'DISABLED|SKIPPED' ${LOG}|wc -l`
+all=$((passed + failed + skipped))
+
+(   echo "=== Test Summary ==="
+    echo "TOTAL: ${all}"
+    echo "PASSED: ${passed}"
+    echo "FAILED: ${failed}"
+    echo "SKIPPED: ${skipped}"
+) | tee -a ${LOG}

Does this work correctly such that ptest-runner picks up the individual
test pass/fail results? I thought the tests would need individual pass/fail lines for that to work?

Quoted 5 lines

Cheers,

Richard

They do have individual pass/fail results in the ptest-runner output, but those don't match up with the pass/fail usually provided by ptest, so the log is parsed for OK/FAILED/SKIPPED/DISABLED and translated. Example output:
write_cb: write 12
write_cb: write -1
=== Test Summary ===
TOTAL: 316
PASSED: 300
FAILED: 0
SKIPPED: 16
DURATION: 87
END: /usr/lib/libevent/ptest
2019-09-06T14:50
STOP: ptest-runner
Compare with the last log from autobuilder (

Quoted 8 lines

https://autobuilder.yocto.io/pub/non-release/20190829-10/testresults/qemux86-64-ptest/libevent.log
):
write_cb: write 12
write_cb: write -1
FAIL: libevent
DURATION: 116
I'm working on a v2 to address multilib - I'll add the example output to the commit as well.

You're missing my point. There are log parsers such as:

http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/utils/logparser.py

The output has to follow the format in:

https://wiki.yoctoproject.org/wiki/Ptest

to quote it:

"""
One major point of ptest is to consolidate the output format of all tests into a single common format. The format selected is the automake "simple test" format:

Quoted 8 lines

result: testname

Where "result" is one of PASS, FAIL or SKIP and "testname" can be any identifying string (the same format used by Automake.)
"""

which is why you'll see other run-ptest scripts using sed to add in PASS/FAIL/SKIP strings.

Noted - fixing and adding this in for v2.

Quoted 2 lines

I think the PASSED/FAILED counts are generated by ptest-runner itself.

I didn't see this in the source. I'll leave the summary in for v2.

Quoted 7 lines

Cheers,

Richard

#6 lore raw
In reply to Trevor Gamblin
Quoted 17 lines · Trevor Gamblin

From: Trevor Gamblin <[email protected]>

The libevent ptest used to report only a global pass or a fail result. Count individual PASS, FAIL, SKIP results. The SKIP results now include tests that are disabled in the libevent code.

---
 .../libevent/libevent/run-ptest               | 34 ++++++++++++-------
 1 file changed, 22 insertions(+), 12 deletions(-)
meta/recipes-support/libevent/libevent/run-ptest +0 -0
index 0241851c70..b7d945246f 100644

meta/recipes-support/libevent/libevent/run-ptest line 7

Quoted 8 lines
meta/recipes-support/libevent/libevent/run-ptest +5 -1
@@ -1,18 +1,28 @@
 #!/bin/sh
 
-fail=0
+# run-ptest - 'ptest' test infrastructure shell script that
+#   wraps the libevent test scripts
+#
+# Trevor Gamblin <[email protected]>
+###############################################################

+LIBEVENTLIB=/usr/lib/libevent

Quoted 1 line

Do we need to account for multilib here ?

+LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log"

Quoted 39 lines
meta/recipes-support/libevent/libevent/run-ptest +15 -12
+
+cd ${LIBEVENTLIB}/ptest
+
 for test in ./test/*
 do
-       $test
-       if [ $? -ne 0 ]
-       then
-               fail=1
-       fi
+    $test 2>&1|tee -a ${LOG}
 done
 
-if [ $fail -eq 0 ]
-then
-       echo "PASS: libevent"
-else
-       echo "FAIL: libevent"
-fi
+passed=`grep OK ${LOG}|wc -l`
+failed=`grep FAILED ${LOG}|wc -l`
+skipped=`grep -E 'DISABLED|SKIPPED' ${LOG}|wc -l`
+all=$((passed + failed + skipped))
+
+(   echo "=== Test Summary ==="
+    echo "TOTAL: ${all}"
+    echo "PASSED: ${passed}"
+    echo "FAILED: ${failed}"
+    echo "SKIPPED: ${skipped}"
+) | tee -a ${LOG}
--

2.21.0

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core