Skip to contentSkip to conversation

runtime/cases/ptest.py: do not require ptest-pkgs in IMAGE_FEATURES

PATCH Superseded Alexander Kanavin 3 participants 6 messages Dec 21, 2017 – Jan 4, 2018

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/3 runtime/cases/ptest.py: do not require ptest-pkgs in IMAGE_FEATURES

Doing so means that all available ptests for packages in the image will be installed and run; some of them may be broken, never finish, take a very long time or simply irrelevant to the user who wants to check ptests of only a few specific packages, and does so by listing them explicitly via IMAGE_INSTALL_append or similar.

Conversely, do not run the test if ptest-runner is not installed (which means there are no ptests available, as they pull it in via a dependency).

---
 meta/lib/oeqa/runtime/cases/ptest.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
meta/lib/oeqa/runtime/cases/ptest.py +4 -1
index ec8c038a566..63e119530dc 100644
@@ -48,9 +48,12 @@ class PtestRunnerTest(OERuntimeTestCase):
 
     @OETestID(1600)
     @skipIfNotFeature('ptest', 'Test requires ptest to be in DISTRO_FEATURES')
-    @skipIfNotFeature('ptest-pkgs', 'Test requires ptest-pkgs to be in IMAGE_FEATURES')
     @OETestDepends(['ssh.SSHTest.test_ssh'])
     def test_ptestrunner(self):
+        status, output = self.target.run('which ptest-runner', 0)
+        if len(output) == 0:
+            self.skipTest("No -ptest packages are installed in the image")
+
         import datetime
 
         test_log_dir = self.td.get('TEST_LOG_DIR', '')
Signature · 1 line
2.15.1
#1 lore raw

PATCH v2 2/3 testimage.bbclass: add ptest to the list of runtime tests whenever possible

If no ptest packages are installed in the image, the test does nothing; if ptest packages are installed in the image, then they should be run without user having to enable that manually.

---
 meta/classes/testimage.bbclass | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
meta/classes/testimage.bbclass +6 -6
index 7260ad4517a..3acccbb8fdb 100644
@@ -44,16 +44,16 @@ DEVTESTSUITE = "gcc kernelmodule ldd"
 DEFAULT_TEST_SUITES = "${MINTESTSUITE} auto"
 DEFAULT_TEST_SUITES_pn-core-image-minimal = "${MINTESTSUITE}"
 DEFAULT_TEST_SUITES_pn-core-image-minimal-dev = "${MINTESTSUITE}"
-DEFAULT_TEST_SUITES_pn-core-image-full-cmdline = "${NETTESTSUITE} perl python logrotate"
+DEFAULT_TEST_SUITES_pn-core-image-full-cmdline = "${NETTESTSUITE} perl python logrotate ptest"
 DEFAULT_TEST_SUITES_pn-core-image-x11 = "${MINTESTSUITE}"
-DEFAULT_TEST_SUITES_pn-core-image-lsb = "${NETTESTSUITE} pam parselogs ${RPMTESTSUITE}"
+DEFAULT_TEST_SUITES_pn-core-image-lsb = "${NETTESTSUITE} pam parselogs ${RPMTESTSUITE} ptest"
 DEFAULT_TEST_SUITES_pn-core-image-sato = "${NETTESTSUITE} connman xorg parselogs ${RPMTESTSUITE} \
-    ${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'python', '', d)}"
+    ${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'python', '', d)} ptest"
 DEFAULT_TEST_SUITES_pn-core-image-sato-sdk = "${NETTESTSUITE} buildcpio buildlzip buildgalculator \
-    connman ${DEVTESTSUITE} logrotate perl parselogs python ${RPMTESTSUITE} xorg"
-DEFAULT_TEST_SUITES_pn-core-image-lsb-dev = "${NETTESTSUITE} pam perl python parselogs ${RPMTESTSUITE}"
+    connman ${DEVTESTSUITE} logrotate perl parselogs python ${RPMTESTSUITE} xorg ptest"
+DEFAULT_TEST_SUITES_pn-core-image-lsb-dev = "${NETTESTSUITE} pam perl python parselogs ${RPMTESTSUITE} ptest"
 DEFAULT_TEST_SUITES_pn-core-image-lsb-sdk = "${NETTESTSUITE} buildcpio buildlzip buildgalculator \
-    connman ${DEVTESTSUITE} logrotate pam parselogs perl python ${RPMTESTSUITE}"
+    connman ${DEVTESTSUITE} logrotate pam parselogs perl python ${RPMTESTSUITE} ptest"
 DEFAULT_TEST_SUITES_pn-meta-toolchain = "auto"
 
 # aarch64 has no graphics
Signature · 1 line
2.15.1
#2 lore raw

Re: [PATCHv2 2/3] testimage.bbclass: add ptest to the list of runtime tests whenever possible

Quoted 9 lines · Alexander Kanavin

If no ptest packages are installed in the image, the test does nothing;
if ptest packages are installed in the image, then they should be run without user having to enable that manually.

---
 meta/classes/testimage.bbclass | 12 ++++++------

 1 file changed, 6 insertions(+), 6 deletions(-)

https://autobuilder.yocto.io/builders/nightly-deb-non-deb/builds/673/steps/Running%20Sanity%20Tests/logs/stdio

Cheers,

Richard

#3 lore raw

PATCH v2 3/3 runtime/cases/ptest.py: fail when ptests fail on target

In reply to Alexander Kanavin

That's the whole point isn't it? Previously this testcase succeeded even if some of the underlying on-target tests failed; the only way to find out if anything was wrong was to manually inspect the logs.

---
 meta/lib/oeqa/runtime/cases/ptest.py | 8 ++++++++
 1 file changed, 8 insertions(+)
meta/lib/oeqa/runtime/cases/ptest.py +8 -0
index 63e119530dc..8f208a5ff21 100644
@@ -83,3 +83,11 @@ class PtestRunnerTest(OERuntimeTestCase):
             # Remove the old link to create a new one
             os.remove(ptest_log_dir_link)
         os.symlink(os.path.basename(ptest_log_dir), ptest_log_dir_link)
+
+        failed_tests = {}
+        for section in parse_result.result_dict:
+            failed_testcases = [ test for test, result in parse_result.result_dict[section] if result == 'fail' ]
+            if failed_testcases:
+                failed_tests[section] = failed_testcases
+
+        self.assertFalse(failed_tests, msg = "Failed ptests: %s" %(str(failed_tests)))
Signature · 1 line
2.15.1
#4 lore raw
In reply to Alexander Kanavin

Looks like it doesn't skip the test correctly:

| NOTE: test_ptestrunner (ptest.PtestRunnerTest) | DEBUG: Checking if ptest is in DISTRO_FEATURES or IMAGE_FEATURES | DEBUG: [Running]$ ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.2 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; which ptest-runner | DEBUG: Data from SSH call: which: no ptest-runner in (/usr/sbin:/sbin:/usr/bin:/bin) | DEBUG: [Command returned '1' after 0.10 seconds] | DEBUG: Command: which ptest-runner | Output: which: no ptest-runner in (/usr/sbin:/sbin:/usr/bin:/bin) | | DEBUG: [Running]$ ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.2 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; ptest-runner | DEBUG: Data from SSH call: sh: ptest-runner: command not found | DEBUG: [Command returned '127' after 0.10 seconds] | DEBUG: Command: ptest-runner | Output: sh: ptest-runner: command not found | | NOTE: ... FAIL https://autobuilder.yocto.io/builders/nightly-deb-non-deb/builds/673/steps/Running%20Sanity%20Tests/logs/stdio Thanks, Cal

Quoted 33 lines · Alexander Kanavin

Doing so means that all available ptests for packages in the image will be installed and run; some of them may be broken, never finish, take a very long time or simply irrelevant to the user who wants to check ptests of only a few specific packages, and does so by listing them explicitly via IMAGE_INSTALL_append or similar.

Conversely, do not run the test if ptest-runner is not installed (which means there are no ptests available, as they pull it in via a dependency).

---
  meta/lib/oeqa/runtime/cases/ptest.py | 5 ++++-
  1 file changed, 4 insertions(+), 1 deletion(-)
meta/lib/oeqa/runtime/cases/ptest.py +4 -1
index ec8c038a566..63e119530dc 100644
@@ -48,9 +48,12 @@ class PtestRunnerTest(OERuntimeTestCase):
 
      @OETestID(1600)
      @skipIfNotFeature('ptest', 'Test requires ptest to be in DISTRO_FEATURES')
-    @skipIfNotFeature('ptest-pkgs', 'Test requires ptest-pkgs to be in IMAGE_FEATURES')
      @OETestDepends(['ssh.SSHTest.test_ssh'])
      def test_ptestrunner(self):
+        status, output = self.target.run('which ptest-runner', 0)
+        if len(output) == 0:
+            self.skipTest("No -ptest packages are installed in the image")
+
          import datetime
 
          test_log_dir = self.td.get('TEST_LOG_DIR', '')
#5 lore raw
Quoted 5 lines · Cal Sullivan

Looks like it doesn't skip the test correctly:

Quoted 4 lines
meta/lib/oeqa/runtime/cases/ptest.py +4 -0
+        status, output = self.target.run('which ptest-runner', 0)
+        if len(output) == 0:
+            self.skipTest("No -ptest packages are installed in the image")
+

Looks like there are no less than three versions of 'which' (busybox, debianutils, gnu), and one of them is noisy when the argument is not found. I'll fix and resend the patchset.

Alex