Skip to contentSkip to conversation

oeqa/ptest: print a warning if ptests failed

PATCH In review Alexander Kanavin 2 participants 3 messages Dec 19, 2020 – Dec 20, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 oeqa/ptest: print a warning if ptests failed

This allows spotting ptest regressions without having hard ptest failures (for that full ptest stability should be achieved).

---
 meta/lib/oeqa/runtime/cases/ptest.py | 1 +
 1 file changed, 1 insertion(+)
meta/lib/oeqa/runtime/cases/ptest.py +1 -0
index a9572c81f0..0800f3c27f 100644
@@ -108,4 +108,5 @@ class PtestRunnerTest(OERuntimeTestCase):
             failmsg = failmsg + "Failed ptests:\n%s" % pprint.pformat(failed_tests)
 
         if failmsg:
+            self.logger.warning("There were failing ptests.")
             self.fail(failmsg)
Signature · 1 line
2.29.2
#1 lore raw
Quoted 13 lines · Alexander Kanavin

This allows spotting ptest regressions without having hard ptest failures (for that full ptest stability should be achieved).

---
 meta/lib/oeqa/runtime/cases/ptest.py | 1 +
 1 file changed, 1 insertion(+)
meta/lib/oeqa/runtime/cases/ptest.py +0 -0
index a9572c81f0..0800f3c27f 100644

meta/lib/oeqa/runtime/cases/ptest.py class PtestRunnerTest(OERuntimeTestCase): · line 112

Quoted 5 lines
meta/lib/oeqa/runtime/cases/ptest.py +1 -0
@@ -108,4 +108,5 @@ class PtestRunnerTest(OERuntimeTestCase):
             failmsg = failmsg + "Failed ptests:\n%s" % pprint.pformat(failed_tests)
 
         if failmsg:
+            self.logger.warning("There were failing ptests.")
             self.fail(failmsg)

Have you looked at this on the arm worker? That seems to have four failing tests over three recipes:

{'libinput': ['libinput-test-suite'],
'strace': ['qual_fault-syscall.test', 'qual_fault.test'], 'valgrind': ['drd/tests/std_list']}

https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1355

Not sure if these always fail or are intermittent?

Cheers,

Richard

#2 lore raw
Quoted 25 lines · Richard Purdie

Review

Quoted 9 lines · Alexander Kanavin

This allows spotting ptest regressions without having hard ptest failures (for that full ptest stability should be achieved).

---
 meta/lib/oeqa/runtime/cases/ptest.py | 1 +
 1 file changed, 1 insertion(+)
(quoted diff) +0 -0

b/meta/lib/oeqa/runtime/cases/ptest.py

meta/lib/oeqa/runtime/cases/ptest.py class PtestRunnerTest(OERuntimeTestCase): · line 108

Quoted 5 lines

index a9572c81f0..0800f3c27f 100644

meta/lib/oeqa/runtime/cases/ptest.py +0 -0
@@ -108,4 +108,5 @@ class PtestRunnerTest(OERuntimeTestCase):
             failmsg = failmsg + "Failed ptests:\n%s" %

pprint.pformat(failed_tests)

meta/lib/oeqa/runtime/cases/ptest.py class PtestRunnerTest(OERuntimeTestCase): · line 112

Quoted 4 lines
meta/lib/oeqa/runtime/cases/ptest.py +1 -0
         if failmsg:
+            self.logger.warning("There were failing ptests.")
             self.fail(failmsg)

Have you looked at this on the arm worker? That seems to have four failing tests over three recipes:

Quoted 8 lines

{'libinput': ['libinput-test-suite'],
'strace': ['qual_fault-syscall.test', 'qual_fault.test'], 'valgrind': ['drd/tests/std_list']}

https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1355

Not sure if these always fail or are intermittent?

libinput failure should be addressed by Bruce's patch to enable UINPUT kernel option (I think I saw that in master-next, but now I don't).

The rest are intermittent, as latest master build only shows one different intermittent failure in glib (which I'm trying to reproduce with additional diagnostics):
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1356

Alex