testimage: restore 'auto' behaviour and a few other fixes

Changes in V3:
* update patch to fix oe-selftest problem: oeqa/runtime/cases/selftest.py: rename to _selftest.py

Changes in V2:
* move codes into testimage_main instead of modifying testimage-auto.bbclass
* use if...else... instead list index for setting of HOSTTOOLS

The following changes since commit bc6c8a0794999b49508aeff9b7cf855ddb2a0c40:

poky: Switch to post release name/version (2018-06-03 16:30:43 +0100)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib ChenQi/testimage-auto
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/testimage-auto

Chen Qi (14):
oeqa/core/loader.py: support the 'auto' keyword testimage.bbclass: fix behavior of empty TEST_SUITES oeqa/core/decorator/__init__.py: set metaclass to ABCMeta oeqa/core/decorator/__init__.py: use 'cls' instead of 'obj' oeqa/runtime/cases/dnf_runtime.py: skip test if PACKAGE_FEED_URIS is
not set
oeqa/runtime/cases/selftest.py: rename to _selftest.py testimage.bbclass: also check 'auto' to create rpm index oeqa/core/decorator/data.py: fix skipIfNotInDataVar oeqa/runtime/cases/rpm.py: skip if rpm not available oeqa/runtime/cases/multilib.py: skip if needed packages are not
available
oeqa/runtime/cases/multilib.py: fix test_file_connman skipping logic bitbake.conf: fix HOSTTOOLS setting related to image testing testimage.bbclass: move codes into testimage_main oeqa/core/target/ssh.py: increase maximum read bytes from 1024 to 4096

 .../runtime/cases/{selftest.py => _selftest.py}     |  2 +-
 meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py |  3 +++
 meta/classes/testimage.bbclass                      | 21 +++++++++++++--------
 meta/conf/bitbake.conf                              |  2 +-
 meta/lib/oeqa/core/decorator/__init__.py            | 10 +++++-----
 meta/lib/oeqa/core/decorator/data.py                |  4 ++--
 meta/lib/oeqa/core/loader.py                        | 17 +++++++++++++----
 meta/lib/oeqa/core/target/ssh.py                    |  2 +-
 meta/lib/oeqa/runtime/cases/multilib.py             |  4 +++-
 meta/lib/oeqa/runtime/cases/rpm.py                  |  1 +
 meta/lib/oeqa/selftest/cases/runtime_test.py        |  2 +-
 11 files changed, 44 insertions(+), 24 deletions(-)
 rename meta-selftest/lib/oeqa/runtime/cases/{selftest.py => _selftest.py} (94%)

1.9.1

In previous OEQA, having 'auto' in TEST_SUITES results in executing as many test cases as possible.

This behaviour is broken for now. From the codes in core/loader.py, I can see that it tries to use another keyword 'all'. But in fact, it does not work.

I've checked the current manual. The manual says using 'auto'. Below is the current information in manual.

"""
Alternatively, you can provide the "auto" option to have all applicable tests run against the image.

TEST_SUITES_append = " auto"
"""

So we should restore this behaviour. This patch does so.

Also, output warning message is some module is named as 'auto', as this is a reserved keyword.

---
 meta/lib/oeqa/core/loader.py | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

index a4744de..98fc0f6 100644
@@ -155,7 +155,16 @@ class OETestLoader(unittest.TestLoader):
         class_name = case.__class__.__name__
         test_name = case._testMethodName
 
-        if self.modules:
+        # 'auto' is a reserved key word to run test cases automatically
+        # warn users if their test case belong to a module named 'auto'
+        if module_name_small == "auto":
+            bb.warn("'auto' is a reserved key word for TEST_SUITES. "
+                    "But test case '%s' is detected to belong to auto module. "
+                    "Please condier using a new name for your module." % str(case))
+
+        # check if case belongs to any specified module
+        # if 'auto' is specified, such check is skipped
+        if self.modules and not 'auto' in self.modules:
             module = None
             try:
                 module = self.modules[module_name_small]

@@ -245,7 +254,7 @@ class OETestLoader(unittest.TestLoader):
         for tcName in testCaseNames:
             case = self._getTestCase(testCaseClass, tcName)
             # Filer by case id
-            if not (self.tests and not 'all' in self.tests
+            if not (self.tests and not 'auto' in self.tests
                     and not getCaseID(case) in self.tests):
                 self._handleTestCaseDecorators(case)
 

@@ -309,14 +318,14 @@ class OETestLoader(unittest.TestLoader):
         module_name = module.__name__
 
         # Normal test modules are loaded if no modules were specified,
-        # if module is in the specified module list or if 'all' is in
+        # if module is in the specified module list or if 'auto' is in
         # module list.
         # Underscore modules are loaded only if specified in module list.
         load_module = True if not module_name.startswith('_') \
                               and (not self.modules \
                                    or module_name in self.modules \
                                    or module_name_small in self.modules \
-                                   or 'all' in self.modules) \
+                                   or 'auto' in self.modules) \
                            else False
 
         load_underscore = True if module_name.startswith('_') \

1.9.1

The current behaviour of TEST_SUITES is very confusing.

setting: TEST_SUITES = ""
result:  Execute all test cases.

setting: TEST_SUITES = "some_case_not_exist"
result:  Error out with 'Empty test suite' message.

The expected behaviour of TEST_SUITES should be:
1. when 'auto' is in it, execute as many test cases as possible
2. when no valid test case is specified, error out and give user message

The first one is implemented by a previous patch.
The second one is fixed in this patch.

Also add debug message to show test cases to be executed. This is for easier debugging.

---
 meta/classes/testimage.bbclass | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

index bb688b0..d97da67 100644
@@ -260,10 +260,16 @@ def testimage_main(d):
     # Load tests before starting the target
     test_paths = get_runtime_paths(d)
     test_modules = d.getVar('TEST_SUITES').split()
+    if not test_modules:
+        bb.fatal('Empty test suite, please verify TEST_SUITES variable')
+
     tc.loadTests(test_paths, modules=test_modules)
 
-    if not getSuiteCases(tc.suites):
+    suitecases = getSuiteCases(tc.suites)
+    if not suitecases:
         bb.fatal('Empty test suite, please verify TEST_SUITES variable')
+    else:
+        bb.debug(2, 'test suites:\n\t%s' % '\n\t'.join([str(c) for c in suitecases]))
 
     package_extraction(d, tc.suites)
 

1.9.1

OETestFilter is a subclass of OETestDecorator. It wants to make use of @abstractmethod decorator. But such decorator requires metaclass to be ABCMeta to have effect. So add it now to achieve the designed behaviour.

Comments from python's manual:
"""
Using this decorator requires that the class's metaclass is ABCMeta or is derived from it.
"""

---
 meta/lib/oeqa/core/decorator/__init__.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

index 855b6b9..6ca0acf 100644
@@ -2,7 +2,7 @@
 # Released under the MIT license (see COPYING.MIT)
 
 from functools import wraps
-from abc import abstractmethod
+from abc import abstractmethod, ABCMeta
 
 decoratorClasses = set()
 

@@ -10,7 +10,7 @@ def registerDecorator(obj):
     decoratorClasses.add(obj)
     return obj
 
-class OETestDecorator(object):
+class OETestDecorator(object, metaclass=ABCMeta):
     case = None # Reference of OETestCase decorated
     attrs = None # Attributes to be loaded by decorator implementation
 

1.9.1

Use 'cls' instead of 'obj' to better reflect that registerDecorator actually serves as a class decorator.

---
 meta/lib/oeqa/core/decorator/__init__.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

index 6ca0acf..14d7bfc 100644
@@ -6,9 +6,9 @@ from abc import abstractmethod, ABCMeta
 
 decoratorClasses = set()
 
-def registerDecorator(obj):
-    decoratorClasses.add(obj)
-    return obj
+def registerDecorator(cls):
+    decoratorClasses.add(cls)
+    return cls
 
 class OETestDecorator(object, metaclass=ABCMeta):
     case = None # Reference of OETestCase decorated

1.9.1

This test is to test the behaviour of PACKAGE_FEED_URIS is correct or not. If it's not even set, it makes no sense to do such test. So skip this test if PACKAGE_FEED_URIS is not set.

---
 meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py | 3 +++
 1 file changed, 3 insertions(+)

index 81c50ed..1aa7274 100644
@@ -1,6 +1,7 @@
 from oeqa.core.decorator.depends import OETestDepends
 from oeqa.runtime.cases.dnf import DnfTest
 from oeqa.utils.httpserver import HTTPService
+from oeqa.core.decorator.data import skipIfDataVar
 
 class DnfSelftest(DnfTest):
 

@@ -18,6 +19,8 @@ class DnfSelftest(DnfTest):
         cls.temp_dir.cleanup()
 
     @OETestDepends(['dnf.DnfBasicTest.test_dnf_help'])
+    @skipIfDataVar('PACKAGE_FEED_URIS', None,
+                   'Not suitable as PACKAGE_FEED_URIS is not set')
     def test_verify_package_feeds(self):
         """
         Summary: Check correct setting of PACKAGE_FEED_URIS var

1.9.1

This test modules is designed to be invoked only by selftest. It's not meant to be tested by normal runtime test. So it should be renamed with '_' prefix, so that it will not be automatically loaded by normal runtime tests when 'auto' is in TEST_SUITES.

The failure message is as below.

RESULTS - selftest.Selftest.test_install_package - Testcase -1: FAILED

Also, change selftest/cases/runtime_test.py to use '_selftest' accordingly.

fix _selftest

---
 meta-selftest/lib/oeqa/runtime/cases/{selftest.py => _selftest.py} | 2 +-
 meta/lib/oeqa/selftest/cases/runtime_test.py                       | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
 rename meta-selftest/lib/oeqa/runtime/cases/{selftest.py => _selftest.py} (94%)

similarity index 94%
rename from meta-selftest/lib/oeqa/runtime/cases/selftest.py
rename to meta-selftest/lib/oeqa/runtime/cases/_selftest.py
index 19de740..e6c05ef 100644
@@ -17,7 +17,7 @@ class Selftest(OERuntimeTestCase):
         (status, output) = self.target.run("socat -V")
         self.assertEqual(status, 0, msg="socat is not installed")
 
-    @OETestDepends(['selftest.Selftest.test_install_package'])
+    @OETestDepends(['_selftest.Selftest.test_install_package'])
     def test_verify_uninstall(self):
         """
         Summary: Check basic package installation functionality.

index 9c9b4b3..def3819 100644
@@ -122,7 +122,7 @@ class TestImage(OESelftestTestCase):
             self.skipTest('core-image-full-cmdline not buildable for poky-tiny')
 
         features = 'INHERIT += "testimage"\n'
-        features += 'TEST_SUITES = "ping ssh selftest"\n'
+        features += 'TEST_SUITES = "ping ssh _selftest"\n'
         self.write_config(features)
 
         # Build core-image-sato and testimage

1.9.1

Sorry but this is still failing:

https://autobuilder.yocto.io/builders/nightly-oe-selftest/builds/1098/steps/Running%20oe-selftest/logs/stdio

Cheers,

Richard

Having 'auto' in TEST_SUITES will also run the 'dnf' test cases, so also check it to determine whether to create rpm index or not.

This is to fix the following error when TEST_SUITES = "auto".

RESULTS - dnf.DnfRepoTest.test_dnf_makecache - Testcase 1744: ERROR

---
 meta/classes/testimage.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index d97da67..14252ff 100644
@@ -121,7 +121,7 @@ python do_testimage() {
     testimage_sanity(d)
 
     if (d.getVar('IMAGE_PKGTYPE') == 'rpm'
-       and 'dnf' in d.getVar('TEST_SUITES')):
+       and ('dnf' in d.getVar('TEST_SUITES') or 'auto' in d.getVar('TEST_SUITES'))):
         create_rpm_index(d)
 
     testimage_main(d)

1.9.1

The var might not be set, resulting in unexpected error.

RESULTS - multilib.MultilibTest.test_check_multilib_libc - Testcase 1593: ERROR

The above error is due to MULTILIBS being not set, which is the default for OE. This patch fixes this problem.

Also, the debugging message in skipIfNotInDataVar is currently confusing. Instead of
DEBUG: Checking if 'MULTILIBS' value is in 'multilib:lib32' to run the test it should be
DEBUG: Checking if 'MULTILIBS' value contains 'multilib:lib32' to run the test

This patch also fixes it.

---
 meta/lib/oeqa/core/decorator/data.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

index ff7bdd9..31c6dd6 100644
@@ -61,10 +61,10 @@ class skipIfNotInDataVar(OETestDecorator):
 
     attrs = ('var', 'value', 'msg')
     def setUpDecorator(self):
-        msg = ('Checking if %r value is in %r to run '
+        msg = ('Checking if %r value contains %r to run '
               'the test' % (self.var, self.value))
         self.logger.debug(msg)
-        if not self.value in self.case.td.get(self.var):
+        if not self.value in (self.case.td.get(self.var) or ""):
             self.case.skipTest(self.msg)
 
 @registerDecorator

1.9.1

This test case should only run when rpm package is installed. So skip it if rpm package is not installed. This fixes:

RESULTS - rpm.RpmBasicTest.test_rpm_help - Testcase 1059: FAILED

---
 meta/lib/oeqa/runtime/cases/rpm.py | 1 +
 1 file changed, 1 insertion(+)

index 05b94c7..84c59a6 100644
@@ -16,6 +16,7 @@ class RpmBasicTest(OERuntimeTestCase):
             cls.skipTest('Tests require image to be build from rpm')
 
     @OETestID(960)
+    @OEHasPackage(['rpm'])
     @OETestDepends(['ssh.SSHTest.test_ssh'])
     def test_rpm_help(self):
         status, output = self.target.run('rpm --help')

1.9.1

1) The test cases use 'readelf' command to do the check. This command is from binutils. So skip the test if the needed binutils package is not installed.

The related error message in log.do_testimage is like below.

Output: sh: readelf: not found

2) The test case tests /lib/libc.so.6 from lib32-libc6. So skip the test if lib32-libc6 is not installed.

The related error message in log.do_testimage is like below.

Output: readelf: Error: 'lib/libc.so.6': No such file

---
 meta/lib/oeqa/runtime/cases/multilib.py | 2 ++
 1 file changed, 2 insertions(+)

index 8c167f1..970f676 100644
@@ -27,6 +27,8 @@ class MultilibTest(OERuntimeTestCase):
     @skipIfNotInDataVar('MULTILIBS', 'multilib:lib32',
                         "This isn't a multilib:lib32 image")
     @OETestDepends(['ssh.SSHTest.test_ssh'])
+    @OEHasPackage(['binutils'])
+    @OEHasPackage(['lib32-libc6'])
     def test_check_multilib_libc(self):
         """
         Check that a multilib image has both 32-bit and 64-bit libc in.

1.9.1

The test_file_connman should be executed only when 'lib32-connman' is installed and 'connman' is not installed.

When lib32-connman and connman are both installed, the /usr/sbin/connmand could be from connman or lib32-connman, depending on the installation order. What we want to check is the connmand command from lib32-connman, so we need to make sure that connman is not there to cause chaos.

---
 meta/lib/oeqa/runtime/cases/multilib.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 970f676..8902038 100644
@@ -38,6 +38,6 @@ class MultilibTest(OERuntimeTestCase):
 
     @OETestID(279)
     @OETestDepends(['multilib.MultilibTest.test_check_multilib_libc'])
-    @OEHasPackage(['lib32-connman'])
+    @OEHasPackage(['lib32-connman', '!connman'])
     def test_file_connman(self):
         self.archtest("/usr/sbin/connmand", "ELF32")

1.9.1

A list of tools are added to HOSTTOOLS depending on if we inherit testimage or not. Unfortunately, if we use TEST_IMAGE variable to automate the test, these tools are not added to HOSTTOOLS.

Modify the condition to also check TEST_IMAGE to fix the above problem.

Also, change to use if...else... instead of list index for such setting.

---
 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 6ecf78b..4bac7ea 100644
@@ -487,7 +487,7 @@ HOSTTOOLS += " \
 "
 
 # Tools needed to run testimage runtime image testing
-HOSTTOOLS += "${@['', 'ip ping ps scp ssh stty'][bb.data.inherits_class('testimage', d)]}"
+HOSTTOOLS += "${@'ip ping ps scp ssh stty' if (bb.data.inherits_class('testimage', d) or d.getVar('TEST_IMAGE') == '1') else ''}"
 
 # Link to these if present
 HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc sftp socat ssh sudo"

1.9.1

testimage-auto is expected to run testimage task's codes automatically. But in fact, it's currently missing some codes, including testimage_sanity and create_rpm_index.

This leads to the problem of unexpected runtime failure of test_dnf_makecache. The error message is as below.

RESULTS - dnf.DnfRepoTest.test_dnf_makecache - Testcase 1744: ERROR

This error is caused by the fact that create_rpm_index is not executed before running the tests.

There's no reason why such codes should not be in testimage_main, so move them into it.

---
 meta/classes/testimage.bbclass | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

index 14252ff..c17a7c6 100644
@@ -117,13 +117,6 @@ testimage_dump_host () {
 }
 
 python do_testimage() {
-
-    testimage_sanity(d)
-
-    if (d.getVar('IMAGE_PKGTYPE') == 'rpm'
-       and ('dnf' in d.getVar('TEST_SUITES') or 'auto' in d.getVar('TEST_SUITES'))):
-        create_rpm_index(d)
-
     testimage_main(d)
 }
 

@@ -159,6 +152,12 @@ def testimage_main(d):
         """
         raise RuntimeError
 
+    testimage_sanity(d)
+
+    if (d.getVar('IMAGE_PKGTYPE') == 'rpm'
+       and ('dnf' in d.getVar('TEST_SUITES') or 'auto' in d.getVar('TEST_SUITES'))):
+        create_rpm_index(d)
+
     logger = make_logger_bitbake_compatible(logging.getLogger("BitBake"))
     pn = d.getVar("PN")
 

1.9.1

When running testimage task for core-image-sato-sdk, the following error appeared.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 at position 0: invalid start byte

Checking the codes, I found it's caused by setting a 1024 limit for the read method of the StreamReader object.

Comments from the manual:
"""
The chars argument indicates the number of decoded code points or bytes to return. The read() method will never return more data than requested, but it might return less, if there is not enough available.
"""

When running `systemctl status --full' on target, this error occurs.

This patch increase the bytes limit to 4096 to fix the error.

---
 meta/lib/oeqa/core/target/ssh.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 151b99a..8ff1f6c 100644
@@ -208,7 +208,7 @@ def SSHCall(command, logger, timeout=None, **opts):
                 try:
                     if select.select([process.stdout], [], [], 5)[0] != []:
                         reader = codecs.getreader('utf-8')(process.stdout)
-                        data = reader.read(1024, 1024)
+                        data = reader.read(1024, 4096)
                         if not data:
                             process.stdout.close()
                             eof = True

1.9.1