runtime/cases/ptest.py: do not require ptest-pkgs in IMAGE_FEATURES
PATCH 1/3 runtime/cases/ptest.py: do not require ptest-pkgs in IMAGE_FEATURES
Doing so means that all available ptests for packages in the image will be installed and run; some of them may be broken, never finish, take a very long time or simply irrelevant to the user who wants to check ptests of only a few specific packages, and does so by listing them explicitly via IMAGE_INSTALL_append or similar.
Conversely, do not run the test if there are no ptests available.
--- meta/lib/oeqa/runtime/cases/ptest.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
meta/lib/oeqa/runtime/cases/ptest.py +4 -1
@@ -48,9 +48,12 @@ class PtestRunnerTest(OERuntimeTestCase): @OETestID(1600) @skipIfNotFeature('ptest', 'Test requires ptest to be in DISTRO_FEATURES') - @skipIfNotFeature('ptest-pkgs', 'Test requires ptest-pkgs to be in IMAGE_FEATURES') @OETestDepends(['ssh.SSHTest.test_ssh']) def test_ptestrunner(self): + status, output = self.target.run('find /usr -name ptest', 0) + if len(output) == 0: + self.skipTest("No -ptest packages are installed in the image") + import datetime test_log_dir = self.td.get('TEST_LOG_DIR', '')
Signature · 1 line
2.15.1
PATCH 2/3 testimage.bbclass: add ptest to the list of runtime tests whenever possible
If no ptest packages are installed in the image, the test does nothing; if ptest packages are installed in the image, then they should be run without user having to enable that manually.
--- meta/classes/testimage.bbclass | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
meta/classes/testimage.bbclass +6 -6
@@ -44,16 +44,16 @@ DEVTESTSUITE = "gcc kernelmodule ldd" DEFAULT_TEST_SUITES = "${MINTESTSUITE} auto" DEFAULT_TEST_SUITES_pn-core-image-minimal = "${MINTESTSUITE}" DEFAULT_TEST_SUITES_pn-core-image-minimal-dev = "${MINTESTSUITE}" -DEFAULT_TEST_SUITES_pn-core-image-full-cmdline = "${NETTESTSUITE} perl python logrotate" +DEFAULT_TEST_SUITES_pn-core-image-full-cmdline = "${NETTESTSUITE} perl python logrotate ptest" DEFAULT_TEST_SUITES_pn-core-image-x11 = "${MINTESTSUITE}" -DEFAULT_TEST_SUITES_pn-core-image-lsb = "${NETTESTSUITE} pam parselogs ${RPMTESTSUITE}" +DEFAULT_TEST_SUITES_pn-core-image-lsb = "${NETTESTSUITE} pam parselogs ${RPMTESTSUITE} ptest" DEFAULT_TEST_SUITES_pn-core-image-sato = "${NETTESTSUITE} connman xorg parselogs ${RPMTESTSUITE} \ - ${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'python', '', d)}" + ${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'python', '', d)} ptest" DEFAULT_TEST_SUITES_pn-core-image-sato-sdk = "${NETTESTSUITE} buildcpio buildlzip buildgalculator \ - connman ${DEVTESTSUITE} logrotate perl parselogs python ${RPMTESTSUITE} xorg" -DEFAULT_TEST_SUITES_pn-core-image-lsb-dev = "${NETTESTSUITE} pam perl python parselogs ${RPMTESTSUITE}" + connman ${DEVTESTSUITE} logrotate perl parselogs python ${RPMTESTSUITE} xorg ptest" +DEFAULT_TEST_SUITES_pn-core-image-lsb-dev = "${NETTESTSUITE} pam perl python parselogs ${RPMTESTSUITE} ptest" DEFAULT_TEST_SUITES_pn-core-image-lsb-sdk = "${NETTESTSUITE} buildcpio buildlzip buildgalculator \ - connman ${DEVTESTSUITE} logrotate pam parselogs perl python ${RPMTESTSUITE}" + connman ${DEVTESTSUITE} logrotate pam parselogs perl python ${RPMTESTSUITE} ptest" DEFAULT_TEST_SUITES_pn-meta-toolchain = "auto" # aarch64 has no graphics
Signature · 1 line
2.15.1
PATCH 3/3 runtime/cases/ptest.py: fail when ptests fail on target
In reply to Alexander KanavinThat's the whole point isn't it? Previously this testcase succeeded even if some of the underlying on-target tests failed; the only way to find out if anything was wrong was to manually inspect the logs.
--- meta/lib/oeqa/runtime/cases/ptest.py | 8 ++++++++ 1 file changed, 8 insertions(+)
meta/lib/oeqa/runtime/cases/ptest.py +8 -0
@@ -83,3 +83,11 @@ class PtestRunnerTest(OERuntimeTestCase): # Remove the old link to create a new one os.remove(ptest_log_dir_link) os.symlink(os.path.basename(ptest_log_dir), ptest_log_dir_link) + + failed_tests = {} + for section in parse_result.result_dict: + failed_testcases = [ test for test, result in parse_result.result_dict[section] if result == 'fail' ] + if failed_testcases: + failed_tests[section] = failed_testcases + + self.assertFalse(failed_tests, msg = "Failed ptests: %s" %(str(failed_tests)))
Signature · 1 line
2.15.1
Re: [PATCH 3/3] runtime/cases/ptest.py: fail when ptests fail on target
So the context was that so many ptests were failing that the test couldn't be done automatically and expected to succeed. Bugs were filed and the problem chipped away at, but last I looked it wasn't in a state to be enabled.
If a core-image-sato passes with ptests enabled then that's great, but I'll be surprised if it does!
Ross
Quoted 37 lines · Alexander Kanavin
That's the whole point isn't it? Previously this testcase succeeded even if some of the underlying on-target tests failed; the only way to find out if anything was wrong was to manually inspect the logs.
Signed-off-by Alexander Kanavin <[email protected]>--- meta/lib/oeqa/runtime/cases/ptest.py | 8 ++++++++ 1 file changed, 8 insertions(+)meta/lib/oeqa/runtime/cases/ptest.py +4
-0@@ -83,3 +83,11 @@ class PtestRunnerTest(OERuntimeTestCase): # Remove the old link to create a new one os.remove(ptest_log_dir_link) os.symlink(os.path.basename(ptest_log_dir), ptest_log_dir_link) + + failed_tests = {} + for section in parse_result.result_dict: + failed_testcases = [ test for test, result inparse_result.result_dict[section] if result == 'fail' ]
meta/lib/oeqa/runtime/cases/ptest.py +4
-0+ if failed_testcases: + failed_tests[section] = failed_testcases + + self.assertFalse(failed_tests, msg = "Failed ptests: %s"%(str(failed_tests)))
--
2.15.1--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [PATCH 3/3] runtime/cases/ptest.py: fail when ptests fail on target
Quoted 3 lines · Ross
So the context was that so many ptests were failing that the test couldn't be done automatically and expected to succeed. Bugs were filed and the problem chipped away at, but last I looked it wasn't in a state to be enabled.
If a core-image-sato passes with ptests enabled then that's great, but I'll be surprised if it does!
It doesn't. In fact, it never finishes, because bluez ptest hangs forever. That shouldn't be a problem though, because ptests are not included by default in any of the standard images, and so the test is simply skipped (see patch 1 of 3). The point of this patch is to eliminate misleading passing behaviour when people do include ptests - either for everything in their image or for specific packages they maintain, and want, for example, to check for regressions on version upgrades.
Alex