oeqa/runtime/cases/ptest.py: use to_boolean for PTEST_EXPECT_FAILURE
PATCH v2 1/2 oeqa/runtime/cases/ptest.py: use to_boolean for PTEST_EXPECT_FAILURE
From: Chen Qi <[email protected]>
When PTEST_EXPECT_FAILURE is set to "0", the expected behavior is that ptest should succeed. So we need to use to_boolean.
--- meta/lib/oeqa/runtime/cases/ptest.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
meta/lib/oeqa/runtime/cases/ptest.py +3 -3
@@ -22,7 +22,7 @@ class PtestRunnerTest(OERuntimeTestCase): @OEHasPackage(['ptest-runner']) @unittest.expectedFailure def test_ptestrunner_expectfail(self): - if not self.td.get('PTEST_EXPECT_FAILURE'): + if not bb.utils.to_boolean(self.td.get('PTEST_EXPECT_FAILURE')): self.skipTest('Cannot run ptests with @expectedFailure as ptests are required to pass') self.do_ptestrunner()
@@ -30,8 +30,8 @@ class PtestRunnerTest(OERuntimeTestCase): @OETestDepends(['ssh.SSHTest.test_ssh']) @OEHasPackage(['ptest-runner']) def test_ptestrunner_expectsuccess(self): - if self.td.get('PTEST_EXPECT_FAILURE'): - self.skipTest('Cannot run ptests without @expectedFailure as ptests are expected to fail') + if bb.utils.to_boolean(self.td.get('PTEST_EXPECT_FAILURE')): + self.skipTest('Cannot run ptests without @expectedFailure as ptests are expected to fail') self.do_ptestrunner() def do_ptestrunner(self):
Signature · 1 line
2.43.0
PATCH v2 2/2 core-image-ptest.bb: use ?= for PTEST_EXPECT_FAILURE
From: Chen Qi <[email protected]>
We need to allow users to easily override this value in local.conf. This has the benefit that 'bitbake core-image-ptest-xxx:do_testimage' fails when expected.
As an example, I used 'bitbake core-image-ptest-util-linux:do_testimage'. It succeeded with warning message. I didn't notice the warning message. I saw the command succeeded and I tought util-linux ptest is OK. But in actual fact, the ptest failed. It's the PTEST_EXPECT_FAILURE setting in this core-image-ptest.bb that is not giving me error.
With this change, I can put PTEST_EXPECT_FAILURE = "0" in my local.conf, and I can easily see if 'core-image-ptest-xxx:do_testimage' succeeds or not.
The PTEST_EXPECT_FAILURE change should result in testdata regenerated because ptest.py test case's behavior is affected by it.
--- meta/recipes-core/images/core-image-ptest.bb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
meta/recipes-core/images/core-image-ptest.bb +2 -1
@@ -42,7 +42,8 @@ QB_MEM:virtclass-mcextend-tcl = "-m 5100" TEST_SUITES = "ping ssh parselogs ptest" # Sadly at the moment the full set of ptests is not robust enough and sporadically fails in random places -PTEST_EXPECT_FAILURE = "1" +PTEST_EXPECT_FAILURE ?= "1" +write_image_test_data[vardeps] += "PTEST_EXPECT_FAILURE" python () { if not d.getVar("MCNAME"):
Signature · 1 line
2.43.0
Re: [OE-core][PATCH V2 2/2] core-image-ptest.bb: use ?= for PTEST_EXPECT_FAILURE
meta/recipes-core/images/core-image-ptest.bb QB_MEM:virtclass-mcextend-tcl = "-m 5100" · line 45
Quoted 3 lines · Chen Qi via lists.openembedded.org
# Sadly at the moment the full set of ptests is not robust enough and sporadically fails in random places
meta/recipes-core/images/core-image-ptest.bb +1
-1-PTEST_EXPECT_FAILURE = "1" +PTEST_EXPECT_FAILURE ?= "1"
I'd say let's rather drop this assignment and the comment. This was the case years ago when ptests were indeed flaky, but they've come a long way since. And even though because of this assignment ptest fails become AB warnings, de facto such warnings are still treated as fails that block the patch that caused them.
You'd also need to tweak the regexp in yocto-autobuilder2/steps/observer.py that Mathieu has recently introduced.
Alex
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [OE-core][PATCH V2 2/2] core-image-ptest.bb: use ?= for PTEST_EXPECT_FAILURE
Quoted 3 lines · Alexander Kanavin via lists.openembedded.org
You'd also need to tweak the regexp in
yocto-autobuilder2/steps/observer.pythat Mathieu has recently introduced.
Actually this is not needed. The warning the regexp matches on is still printed, the error is a separate print.
Alex
RE: [OE-core][PATCH V2 2/2] core-image-ptest.bb: use ?= for PTEST_EXPECT_FAILURE
OK. I'll send out V3 to remove this variable setting and related comments.
Regards,
Qi
-----Original Message-----
From: Alexander Kanavin <[email protected]>
Sent: Friday, December 5, 2025 6:34 PM
To: [email protected]
Cc: Chen, Qi <[email protected]>; [email protected]; Mathieu Dubois-Briand <[email protected]>
Subject: Re: [OE-core][PATCH V2 2/2] core-image-ptest.bb: use ?= for PTEST_EXPECT_FAILURE
Quoted 3 lines · Alexander Kanavin via lists.openembedded.org
You'd also need to tweak the regexp in
yocto-autobuilder2/steps/observer.pythat Mathieu has recently introduced.
Actually this is not needed. The warning the regexp matches on is still printed, the error is a separate print.
Alex