runtime/cases/ptest.py: do not require ptest-pkgs in IMAGE_FEATURES
PATCH v2 1/3 runtime/cases/ptest.py: do not require ptest-pkgs in IMAGE_FEATURES
Doing so means that all available ptests for packages in the image will be installed and run; some of them may be broken, never finish, take a very long time or simply irrelevant to the user who wants to check ptests of only a few specific packages, and does so by listing them explicitly via IMAGE_INSTALL_append or similar.
Conversely, do not run the test if ptest-runner is not installed (which means there are no ptests available, as they pull it in via a dependency).
--- meta/lib/oeqa/runtime/cases/ptest.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
meta/lib/oeqa/runtime/cases/ptest.py +4 -1
@@ -48,9 +48,12 @@ class PtestRunnerTest(OERuntimeTestCase): @OETestID(1600) @skipIfNotFeature('ptest', 'Test requires ptest to be in DISTRO_FEATURES') - @skipIfNotFeature('ptest-pkgs', 'Test requires ptest-pkgs to be in IMAGE_FEATURES') @OETestDepends(['ssh.SSHTest.test_ssh']) def test_ptestrunner(self): + status, output = self.target.run('which ptest-runner', 0) + if len(output) == 0: + self.skipTest("No -ptest packages are installed in the image") + import datetime test_log_dir = self.td.get('TEST_LOG_DIR', '')
Signature · 1 line
2.15.1
PATCH v2 2/3 testimage.bbclass: add ptest to the list of runtime tests whenever possible
If no ptest packages are installed in the image, the test does nothing; if ptest packages are installed in the image, then they should be run without user having to enable that manually.
--- meta/classes/testimage.bbclass | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
meta/classes/testimage.bbclass +6 -6
@@ -44,16 +44,16 @@ DEVTESTSUITE = "gcc kernelmodule ldd" DEFAULT_TEST_SUITES = "${MINTESTSUITE} auto" DEFAULT_TEST_SUITES_pn-core-image-minimal = "${MINTESTSUITE}" DEFAULT_TEST_SUITES_pn-core-image-minimal-dev = "${MINTESTSUITE}" -DEFAULT_TEST_SUITES_pn-core-image-full-cmdline = "${NETTESTSUITE} perl python logrotate" +DEFAULT_TEST_SUITES_pn-core-image-full-cmdline = "${NETTESTSUITE} perl python logrotate ptest" DEFAULT_TEST_SUITES_pn-core-image-x11 = "${MINTESTSUITE}" -DEFAULT_TEST_SUITES_pn-core-image-lsb = "${NETTESTSUITE} pam parselogs ${RPMTESTSUITE}" +DEFAULT_TEST_SUITES_pn-core-image-lsb = "${NETTESTSUITE} pam parselogs ${RPMTESTSUITE} ptest" DEFAULT_TEST_SUITES_pn-core-image-sato = "${NETTESTSUITE} connman xorg parselogs ${RPMTESTSUITE} \ - ${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'python', '', d)}" + ${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'python', '', d)} ptest" DEFAULT_TEST_SUITES_pn-core-image-sato-sdk = "${NETTESTSUITE} buildcpio buildlzip buildgalculator \ - connman ${DEVTESTSUITE} logrotate perl parselogs python ${RPMTESTSUITE} xorg" -DEFAULT_TEST_SUITES_pn-core-image-lsb-dev = "${NETTESTSUITE} pam perl python parselogs ${RPMTESTSUITE}" + connman ${DEVTESTSUITE} logrotate perl parselogs python ${RPMTESTSUITE} xorg ptest" +DEFAULT_TEST_SUITES_pn-core-image-lsb-dev = "${NETTESTSUITE} pam perl python parselogs ${RPMTESTSUITE} ptest" DEFAULT_TEST_SUITES_pn-core-image-lsb-sdk = "${NETTESTSUITE} buildcpio buildlzip buildgalculator \ - connman ${DEVTESTSUITE} logrotate pam parselogs perl python ${RPMTESTSUITE}" + connman ${DEVTESTSUITE} logrotate pam parselogs perl python ${RPMTESTSUITE} ptest" DEFAULT_TEST_SUITES_pn-meta-toolchain = "auto" # aarch64 has no graphics
Signature · 1 line
2.15.1
Re: [PATCHv2 2/3] testimage.bbclass: add ptest to the list of runtime tests whenever possible
Quoted 9 lines · Alexander Kanavin
If no ptest packages are installed in the image, the test does nothing;
if ptest packages are installed in the image, then they should be run without user having to enable that manually.Signed-off-by Alexander Kanavin <[email protected]>---
meta/classes/testimage.bbclass | 12 ++++++------1 file changed, 6 insertions(+), 6 deletions(-)
Cheers,
Richard
PATCH v2 3/3 runtime/cases/ptest.py: fail when ptests fail on target
In reply to Alexander KanavinThat's the whole point isn't it? Previously this testcase succeeded even if some of the underlying on-target tests failed; the only way to find out if anything was wrong was to manually inspect the logs.
--- meta/lib/oeqa/runtime/cases/ptest.py | 8 ++++++++ 1 file changed, 8 insertions(+)
meta/lib/oeqa/runtime/cases/ptest.py +8 -0
@@ -83,3 +83,11 @@ class PtestRunnerTest(OERuntimeTestCase): # Remove the old link to create a new one os.remove(ptest_log_dir_link) os.symlink(os.path.basename(ptest_log_dir), ptest_log_dir_link) + + failed_tests = {} + for section in parse_result.result_dict: + failed_testcases = [ test for test, result in parse_result.result_dict[section] if result == 'fail' ] + if failed_testcases: + failed_tests[section] = failed_testcases + + self.assertFalse(failed_tests, msg = "Failed ptests: %s" %(str(failed_tests)))
Signature · 1 line
2.15.1
Looks like it doesn't skip the test correctly:
| NOTE: test_ptestrunner (ptest.PtestRunnerTest) | DEBUG: Checking if ptest is in DISTRO_FEATURES or IMAGE_FEATURES | DEBUG: [Running]$ ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.2 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; which ptest-runner | DEBUG: Data from SSH call: which: no ptest-runner in (/usr/sbin:/sbin:/usr/bin:/bin) | DEBUG: [Command returned '1' after 0.10 seconds] | DEBUG: Command: which ptest-runner | Output: which: no ptest-runner in (/usr/sbin:/sbin:/usr/bin:/bin) | | DEBUG: [Running]$ ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.2 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; ptest-runner | DEBUG: Data from SSH call: sh: ptest-runner: command not found | DEBUG: [Command returned '127' after 0.10 seconds] | DEBUG: Command: ptest-runner | Output: sh: ptest-runner: command not found | | NOTE: ... FAIL https://autobuilder.yocto.io/builders/nightly-deb-non-deb/builds/673/steps/Running%20Sanity%20Tests/logs/stdio Thanks, Cal
Quoted 33 lines · Alexander Kanavin
Doing so means that all available ptests for packages in the image will be installed and run; some of them may be broken, never finish, take a very long time or simply irrelevant to the user who wants to check ptests of only a few specific packages, and does so by listing them explicitly via
IMAGE_INSTALL_appendor similar.Conversely, do not run the test if ptest-runner is not installed (which means there are no ptests available, as they pull it in via a dependency).
Signed-off-by Alexander Kanavin <[email protected]>--- meta/lib/oeqa/runtime/cases/ptest.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)meta/lib/oeqa/runtime/cases/ptest.py +4
-1@@ -48,9 +48,12 @@ class PtestRunnerTest(OERuntimeTestCase): @OETestID(1600) @skipIfNotFeature('ptest', 'Test requires ptest to be in DISTRO_FEATURES') - @skipIfNotFeature('ptest-pkgs', 'Test requires ptest-pkgs to be in IMAGE_FEATURES') @OETestDepends(['ssh.SSHTest.test_ssh']) def test_ptestrunner(self): + status, output = self.target.run('which ptest-runner', 0) + if len(output) == 0: + self.skipTest("No -ptest packages are installed in the image") + import datetime test_log_dir = self.td.get('TEST_LOG_DIR', '')
Quoted 5 lines · Cal Sullivan
Looks like it doesn't skip the test correctly:
Quoted 4 lines
meta/lib/oeqa/runtime/cases/ptest.py +4
-0+ status, output = self.target.run('which ptest-runner', 0) + if len(output) == 0: + self.skipTest("No -ptest packages are installed in the image") +
Looks like there are no less than three versions of 'which' (busybox, debianutils, gnu), and one of them is noisy when the argument is not found. I'll fix and resend the patchset.
Alex