Skip to contentSkip to conversation

oeqa/runtime/cases/ptest.py: use to_boolean for PTEST_EXPECT_FAILURE

PATCH Superseded [email protected] 2 participants 5 messages Dec 5, 2025 – Dec 9, 2025 1 message not in the archive

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/2 oeqa/runtime/cases/ptest.py: use to_boolean for PTEST_EXPECT_FAILURE

From: Chen Qi <[email protected]>

When PTEST_EXPECT_FAILURE is set to "0", the expected behavior is that ptest should succeed. So we need to use to_boolean.

---
 meta/lib/oeqa/runtime/cases/ptest.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
meta/lib/oeqa/runtime/cases/ptest.py +3 -3
index fbaeb84d00..4bbed290b4 100644
@@ -22,7 +22,7 @@ class PtestRunnerTest(OERuntimeTestCase):
     @OEHasPackage(['ptest-runner'])
     @unittest.expectedFailure
     def test_ptestrunner_expectfail(self):
-        if not self.td.get('PTEST_EXPECT_FAILURE'):
+        if not bb.utils.to_boolean(self.td.get('PTEST_EXPECT_FAILURE')):
             self.skipTest('Cannot run ptests with @expectedFailure as ptests are required to pass')
         self.do_ptestrunner()
 
@@ -30,8 +30,8 @@ class PtestRunnerTest(OERuntimeTestCase):
     @OETestDepends(['ssh.SSHTest.test_ssh'])
     @OEHasPackage(['ptest-runner'])
     def test_ptestrunner_expectsuccess(self):
-        if self.td.get('PTEST_EXPECT_FAILURE'):
-            self.skipTest('Cannot run ptests without @expectedFailure as ptests are expected to fail')
+        if bb.utils.to_boolean(self.td.get('PTEST_EXPECT_FAILURE')):
+             self.skipTest('Cannot run ptests without @expectedFailure as ptests are expected to fail')
         self.do_ptestrunner()
 
     def do_ptestrunner(self):
Signature · 1 line
2.43.0
#1 lore raw

PATCH v2 2/2 core-image-ptest.bb: use ?= for PTEST_EXPECT_FAILURE

From: Chen Qi <[email protected]>

We need to allow users to easily override this value in local.conf. This has the benefit that 'bitbake core-image-ptest-xxx:do_testimage' fails when expected.

As an example, I used 'bitbake core-image-ptest-util-linux:do_testimage'. It succeeded with warning message. I didn't notice the warning message. I saw the command succeeded and I tought util-linux ptest is OK. But in actual fact, the ptest failed. It's the PTEST_EXPECT_FAILURE setting in this core-image-ptest.bb that is not giving me error.

With this change, I can put PTEST_EXPECT_FAILURE = "0" in my local.conf, and I can easily see if 'core-image-ptest-xxx:do_testimage' succeeds or not.

The PTEST_EXPECT_FAILURE change should result in testdata regenerated because ptest.py test case's behavior is affected by it.

---
 meta/recipes-core/images/core-image-ptest.bb | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
meta/recipes-core/images/core-image-ptest.bb +2 -1
index 017f05f81b..b554964d20 100644
@@ -42,7 +42,8 @@ QB_MEM:virtclass-mcextend-tcl = "-m 5100"
 TEST_SUITES = "ping ssh parselogs ptest"
 
 # Sadly at the moment the full set of ptests is not robust enough and sporadically fails in random places
-PTEST_EXPECT_FAILURE = "1"
+PTEST_EXPECT_FAILURE ?= "1"
+write_image_test_data[vardeps] += "PTEST_EXPECT_FAILURE"
 
 python () {
     if not d.getVar("MCNAME"):
Signature · 1 line
2.43.0
#2 lore raw

Re: [OE-core][PATCH V2 2/2] core-image-ptest.bb: use ?= for PTEST_EXPECT_FAILURE

meta/recipes-core/images/core-image-ptest.bb QB_MEM:virtclass-mcextend-tcl = "-m 5100" · line 45

Quoted 3 lines · Chen Qi via lists.openembedded.org

# Sadly at the moment the full set of ptests is not robust enough and sporadically fails in random places

meta/recipes-core/images/core-image-ptest.bb +1 -1
-PTEST_EXPECT_FAILURE = "1"
+PTEST_EXPECT_FAILURE ?= "1"

I'd say let's rather drop this assignment and the comment. This was the case years ago when ptests were indeed flaky, but they've come a long way since. And even though because of this assignment ptest fails become AB warnings, de facto such warnings are still treated as fails that block the patch that caused them.

You'd also need to tweak the regexp in yocto-autobuilder2/steps/observer.py that Mathieu has recently introduced.

Alex

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw

Re: [OE-core][PATCH V2 2/2] core-image-ptest.bb: use ?= for PTEST_EXPECT_FAILURE

Quoted 3 lines · Alexander Kanavin via lists.openembedded.org

You'd also need to tweak the regexp in yocto-autobuilder2/steps/observer.py that Mathieu has recently introduced.

Actually this is not needed. The warning the regexp matches on is still printed, the error is a separate print.

Alex

author
#5 lore raw

RE: [OE-core][PATCH V2 2/2] core-image-ptest.bb: use ?= for PTEST_EXPECT_FAILURE

OK. I'll send out V3 to remove this variable setting and related comments.

Regards,
Qi

-----Original Message-----
From: Alexander Kanavin <[email protected]>
Sent: Friday, December 5, 2025 6:34 PM
To: [email protected]
Cc: Chen, Qi <[email protected]>; [email protected]; Mathieu Dubois-Briand <[email protected]>
Subject: Re: [OE-core][PATCH V2 2/2] core-image-ptest.bb: use ?= for PTEST_EXPECT_FAILURE

Quoted 3 lines · Alexander Kanavin via lists.openembedded.org

You'd also need to tweak the regexp in yocto-autobuilder2/steps/observer.py that Mathieu has recently introduced.

Actually this is not needed. The warning the regexp matches on is still printed, the error is a separate print.

Alex