testimage improvements
cover testimage improvements
I recently started using testimage.bbclass with an image using systemd and found some issues:
- inconsistent output
- systemd tests failing due to missing avahi in image
- tests not run in desired order when using TEST_SUITES="auto"
The following changes since commit 6333186e9764b2c269a2b3869956860fa8fde2b1:
binutils: Fix packaging now gold is always built (2015-04-08 23:10:57 +0100)
are available in the git repository at:
git://github.com/pohly/openembedded-core master
https://github.com/pohly/openembedded-core/tree/master
Patrick Ohly (3):
oeqa/oetest.py: use bb logging oeqa/runtime/systemd.py: skip instead of failing without avahi
testimage: sort modules based on dependencies
meta/classes/testimage.bbclass | 6 +++- meta/lib/oeqa/oetest.py | 61 +++++++++++++++++++++++++++++++++++---- meta/lib/oeqa/runtime/systemd.py | 7 +++++ meta/lib/oeqa/utils/decorators.py | 1 + 4 files changed, 69 insertions(+), 6 deletions(-)
Signature · 1 line
1.8.4.5
PATCH 1/3 oeqa/oetest.py: use bb logging
Plain print has two drawbacks:
- the output is handled by different processes, causing the message about found tests to appear randomly after the result of the initial tests
- the output is not part of the bitbake console output
Affects image testing with testimage.bbclass.
--- meta/lib/oeqa/oetest.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
meta/lib/oeqa/oetest.py +3 -2
@@ -11,6 +11,7 @@ import os, re, mmap import unittest import inspect import subprocess +import bb from oeqa.utils.decorators import LogResults def loadTests(tc, type="runtime"):
@@ -33,8 +34,8 @@ def loadTests(tc, type="runtime"): def runTests(tc, type="runtime"): suite = loadTests(tc, type) - print("Test modules %s" % tc.testslist) - print("Found %s tests" % suite.countTestCases()) + bb.note("Test modules %s" % tc.testslist) + bb.note("Found %s tests" % suite.countTestCases()) runner = unittest.TextTestRunner(verbosity=2) result = runner.run(suite)
Signature · 1 line
1.8.4.5
PATCH 2/3 oeqa/runtime/systemd.py: skip instead of failing without avahi
The SystemdServiceTests assume that avahi-daemon is installed, which is not necessarily the case depending on the image being tested.
Better check this dependency before starting the tests and skip them if the service is not installed. This has to be done for each test instead of for the entire module, because other tests in the module can run without avavi.
--- meta/lib/oeqa/runtime/systemd.py | 7 +++++++ 1 file changed, 7 insertions(+)
meta/lib/oeqa/runtime/systemd.py +7 -0
@@ -67,13 +67,19 @@ class SystemdBasicTests(SystemdTest): class SystemdServiceTests(SystemdTest): + def check_for_avahi(self): + if not self.hasPackage('avahi-daemon'): + raise unittest.SkipTest("Testcase dependency not met: need avahi-daemon installed on target") + @skipUnlessPassed('test_systemd_basic') def test_systemd_status(self): + self.check_for_avahi() self.systemctl('status --full', 'avahi-daemon.service') @testcase(695) @skipUnlessPassed('test_systemd_status') def test_systemd_stop_start(self): + self.check_for_avahi() self.systemctl('stop', 'avahi-daemon.service') self.systemctl('is-active', 'avahi-daemon.service', expected=3, verbose=True) self.systemctl('start','avahi-daemon.service')
@@ -82,6 +88,7 @@ class SystemdServiceTests(SystemdTest): @testcase(696) @skipUnlessPassed('test_systemd_basic') def test_systemd_disable_enable(self): + self.check_for_avahi() self.systemctl('disable', 'avahi-daemon.service') self.systemctl('is-enabled', 'avahi-daemon.service', expected=1) self.systemctl('enable', 'avahi-daemon.service')
Signature · 1 line
1.8.4.5
PATCH 3/3 testimage: sort modules based on dependencies
In reply to Patrick OhlyTEST_SUITES="auto" is useful to run all suitable tests without having to hard-code the list. However, it did not take test dependencies into account, which can be an issue for tests which really depend on some other test to run first.
To fix this, modules get loaded in the order determined by TESTS_SUITES, but then get re-ordered based on dependencies derived from @skipUnlessPassed before running them. The original order is used to break ties when there are no dependencies, so reordering only occurs when really necessary.
@skipUnlessPassed gets extended such that it makes the test name a method depends on available for inspection by the test loader in oetest.py.
Unfortunately Python's unittest offers no API to inspect tests in a TestSuite, so the code has to rely on implementation details to find all tests. The worst that can happen when the implementation changes is that tests are not found and reordering does not happen.
--- meta/classes/testimage.bbclass | 6 ++++- meta/lib/oeqa/oetest.py | 56 ++++++++++++++++++++++++++++++++++++--- meta/lib/oeqa/utils/decorators.py | 1 + 3 files changed, 59 insertions(+), 4 deletions(-)
meta/classes/testimage.bbclass +5 -1
@@ -16,7 +16,11 @@ # The test names are the module names in meta/lib/oeqa/runtime. # Each name in TEST_SUITES represents a required test for the image. (no skipping allowed) # Appending "auto" means that it will try to run all tests that are suitable for the image (each test decides that on it's own). -# Note that order in TEST_SUITES is important (it's the order tests run) and it influences tests dependencies. +# Note that order in TEST_SUITES is relevant: tests are run in an order such that +# tests mentioned in @skipUnlessPassed run before the tests that depend on them, +# but without such dependencies, tests run in the order in which they are listed +# in TEST_SUITES. +# # A layer can add its own tests in lib/oeqa/runtime, provided it extends BBPATH as normal in its layer.conf. # TEST_LOG_DIR contains a command ssh log and may contain infromation about what command is running, output and return codes and for qemu a boot log till login.
meta/lib/oeqa/oetest.py +53 -3
62 lines not on this page · raw message
meta/lib/oeqa/utils/decorators.py +1 -0
@@ -85,6 +85,7 @@ class skipUnlessPassed(object): raise unittest.SkipTest("Testcase dependency not met: %s" % self.testcase) return f(*args) wrapped_f.__name__ = f.__name__ + wrapped_f._depends_on = self.testcase return wrapped_f class testcase(object):
Signature · 1 line
1.8.4.5