Skip to contentSkip to conversation

XML test runner, or how to override Poky's python code

PATCH In review Mardegan, Alberto 3 participants 4 messages Apr 18, 2019 – Apr 23, 2019

lore
Share & tools
lore
#0 lore raw

Hi there!
Since Yocto "thud", the test suite no longer supports outputting the
test report in XML format.

While I'm not especially fond of that format, it had one feature which was quite important for our further processing of test results: the stdout from the test was captured and included in the test report (now the JSON test report only contains the stdout and stderr for the failing tests).

Is it possible from a layer to alter the implementation of the OETestRunner, so that the bitbake testimage command would behave the way we want to?

Ciao,
Alberto

________________________________

This e-mail and any attachment(s) are intended only for the recipient(s) named above and others who have been specifically authorized to receive them. They may contain confidential information. If you are not the intended recipient, please do not read this email or its attachment(s). Furthermore, you are hereby notified that any dissemination, distribution or copying of this e-mail and any attachment(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and then delete this e-mail and any attachment(s) or copies thereof from your system. Thank you.

#1 lore raw
Quoted 14 lines · Alberto

Hi there!
Since Yocto "thud", the test suite no longer supports outputting
the
test report in XML format.

While I'm not especially fond of that format, it had one feature which
was quite important for our further processing of test results: the stdout from the test was captured and included in the test report (now
the JSON test report only contains the stdout and stderr for the failing
tests).

Quoted 4 lines

Is it possible from a layer to alter the implementation of the OETestRunner, so that the bitbake testimage command would behave the way
we want to?

Its certainly possible to alter pretty much any of the code but it would probably involve duplication which would be ugly. A better way may be to send patches which optionally allow the test output to be stored in the json output based upon configuration?

Cheers,

Richard

#2 lore raw

PATCH 1/1 XML test runner, or how to override Poky's python code

Hi,

Quoted 4 lines · Richard Purdie

Its certainly possible to alter pretty much any of the code but it would probably involve duplication which would be ugly. A better way may be to send patches which optionally allow the test output to be stored in the json output based upon configuration?

Here's the patch; I hope that Thunderbird won't play too much with it :-)

Ciao,
Alberto

________________________________

This e-mail and any attachment(s) are intended only for the recipient(s) named above and others who have been specifically authorized to receive them. They may contain confidential information. If you are not the intended recipient, please do not read this email or its attachment(s). Furthermore, you are hereby notified that any dissemination, distribution or copying of this e-mail and any attachment(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and then delete this e-mail and any attachment(s) or copies thereof from your system. Thank you.

From ef01f73c61b7f202e905fc4d9ae7fb5057210805 Mon Sep 17 00:00:00 2001
From: Alberto Mardegan <[email protected]>
Date: Tue, 23 Apr 2019 15:25:36 +0300
Subject: [PATCH] oeqa/core/runner: dump stdout and stderr of each test case

Some CI pipelines might perform further processing of the test output (for instance, to plot some metrics into a chart). However, Since thud we switched away from the XML-based jUnit reporting, and at the same time we lost the ability of collecting the stdout and stderr of the various tests.

We now restore this functionality by adding stdout and stderr keys to the JSON reports.

---
 meta/lib/oeqa/core/runner.py | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)
meta/lib/oeqa/core/runner.py +14 -3
index df88b85f1c..288b3ae614 100644
@@ -7,6 +7,7 @@ import unittest
 import logging
 import re
 import json
+import sys
 
 from unittest import TextTestResult as _TestResult
 from unittest import TextTestRunner as _TestRunner
@@ -45,6 +46,9 @@ class OETestResult(_TestResult):
 
         self.tc = tc
 
+        # stdout and stderr for each test case
+        self.logged_output = {}
+
     def startTest(self, test):
         # May have been set by concurrencytest
         if test.id() not in self.starttime:
@@ -53,6 +57,9 @@ class OETestResult(_TestResult):
 
     def stopTest(self, test):
         self.endtime[test.id()] = time.time()
+        if self.buffer:
+            self.logged_output[test.id()] = (
+                    sys.stdout.getvalue(), sys.stderr.getvalue())
         super(OETestResult, self).stopTest(test)
         if test.id() in self.progressinfo:
             self.tc.logger.info(self.progressinfo[test.id()])
@@ -144,10 +151,14 @@ class OETestResult(_TestResult):
             if status not in logs:
                 logs[status] = []
             logs[status].append("RESULTS - %s - Testcase %s: %s%s" % (case.id(), oeid, status, t))
+            report = {'status': status}
             if log:
-                result[case.id()] = {'status': status, 'log': log}
-            else:
-                result[case.id()] = {'status': status}
+                report['log'] = log
+            if case.id() in self.logged_output:
+                (stdout, stderr) = self.logged_output[case.id()]
+                report['stdout'] = stdout
+                report['stderr'] = stderr
+            result[case.id()] = report
 
         for i in ['PASSED', 'SKIPPED', 'EXPECTEDFAIL', 'ERROR', 'FAILED', 'UNKNOWN']:
             if i not in logs:
Signature · 1 line
2.17.1
#3 lore raw

Please send the patch using 'git send-email' to oe-core mailing list.

Also, this should be optional, subject to a bitbake variable (which you can pass in via a bool parameter that defaults to False). Not everyone wants the full stdout/stderr content in the json.

Alex

Quoted 23 lines · Alberto

Hi,

Quoted 4 lines · Richard Purdie

Its certainly possible to alter pretty much any of the code but it would probably involve duplication which would be ugly. A better way may be to send patches which optionally allow the test output to be stored in the json output based upon configuration?

Here's the patch; I hope that Thunderbird won't play too much with it :-)

Ciao,
Alberto

________________________________

This e-mail and any attachment(s) are intended only for the recipient(s) named above and others who have been specifically authorized to receive them. They may contain confidential information. If you are not the intended recipient, please do not read this email or its attachment(s). Furthermore, you are hereby notified that any dissemination, distribution or copying of this e-mail and any attachment(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and then delete this e-mail and any attachment(s) or copies thereof from your system. Thank you. --
_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky