testimage: restore 'auto' behaviour and a few other fixes

The following changes since commit 719d068bde55ef29a3468bc0779d4cb0c11e8c1d:

bitbake: fetch2: fix import error for Python 3.6.5 (2018-05-29 21:07:46 +0100)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib ChenQi/testimage-auto
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/testimage-auto

Chen Qi (14):
oeqa/core/loader.py: support the 'auto' keyword testimage.bbclass: fix behavior of empty TEST_SUITES oeqa/core/decorator/__init__.py: set metaclass to ABCMeta oeqa/core/decorator/__init__.py: use 'cls' instead of 'obj' oeqa/runtime/cases/dnf_runtime.py: skip test if PACKAGE_FEED_URIS is
not set
oeqa/runtime/cases/selftest.py: rename to _selftest.py testimage.bbclass: also check 'auto' to create rpm index oeqa/core/decorator/data.py: fix skipIfNotInDataVar oeqa/runtime/cases/rpm.py: skip if rpm not available oeqa/runtime/cases/multilib.py: skip if needed packages are not
available
oeqa/runtime/cases/multilib.py: fix test_file_connman skipping logic bitbake.conf: fix HOSTTOOLS setting related to image testing testimage-auto.bbclass: execute the same codes as testimage task oeqa/core/target/ssh.py: increase maximum read bytes from 1024 to 4096

 .../oeqa/runtime/cases/{selftest.py => _selftest.py}    |  0
 meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py     |  3 +++
 meta/classes/testimage-auto.bbclass                     |  6 ++++++
 meta/classes/testimage.bbclass                          | 10 ++++++++--
 meta/conf/bitbake.conf                                  |  2 +-
 meta/lib/oeqa/core/decorator/__init__.py                | 10 +++++-----
 meta/lib/oeqa/core/decorator/data.py                    |  4 ++--
 meta/lib/oeqa/core/loader.py                            | 17 +++++++++++++----
 meta/lib/oeqa/core/target/ssh.py                        |  2 +-
 meta/lib/oeqa/runtime/cases/multilib.py                 |  4 +++-
 meta/lib/oeqa/runtime/cases/rpm.py                      |  1 +
 meta/lib/oeqa/selftest/cases/runtime_test.py            |  2 +-
 12 files changed, 44 insertions(+), 17 deletions(-)
 rename meta-selftest/lib/oeqa/runtime/cases/{selftest.py => _selftest.py} (100%)

1.9.1

In previous OEQA, having 'auto' in TEST_SUITES results in executing as many test cases as possible.

This behaviour is broken for now. From the codes in core/loader.py, I can see that it tries to use another keyword 'all'. But in fact, it does not work.

I've checked the current manual. The manual says using 'auto'. Below is the current information in manual.

"""
Alternatively, you can provide the "auto" option to have all applicable tests run against the image.

TEST_SUITES_append = " auto"
"""

So we should restore this behaviour. This patch does so.

Also, output warning message is some module is named as 'auto', as this is a reserved keyword.

---
 meta/lib/oeqa/core/loader.py | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

index a4744de..98fc0f6 100644
@@ -155,7 +155,16 @@ class OETestLoader(unittest.TestLoader):
         class_name = case.__class__.__name__
         test_name = case._testMethodName
 
-        if self.modules:
+        # 'auto' is a reserved key word to run test cases automatically
+        # warn users if their test case belong to a module named 'auto'
+        if module_name_small == "auto":
+            bb.warn("'auto' is a reserved key word for TEST_SUITES. "
+                    "But test case '%s' is detected to belong to auto module. "
+                    "Please condier using a new name for your module." % str(case))
+
+        # check if case belongs to any specified module
+        # if 'auto' is specified, such check is skipped
+        if self.modules and not 'auto' in self.modules:
             module = None
             try:
                 module = self.modules[module_name_small]

@@ -245,7 +254,7 @@ class OETestLoader(unittest.TestLoader):
         for tcName in testCaseNames:
             case = self._getTestCase(testCaseClass, tcName)
             # Filer by case id
-            if not (self.tests and not 'all' in self.tests
+            if not (self.tests and not 'auto' in self.tests
                     and not getCaseID(case) in self.tests):
                 self._handleTestCaseDecorators(case)
 

@@ -309,14 +318,14 @@ class OETestLoader(unittest.TestLoader):
         module_name = module.__name__
 
         # Normal test modules are loaded if no modules were specified,
-        # if module is in the specified module list or if 'all' is in
+        # if module is in the specified module list or if 'auto' is in
         # module list.
         # Underscore modules are loaded only if specified in module list.
         load_module = True if not module_name.startswith('_') \
                               and (not self.modules \
                                    or module_name in self.modules \
                                    or module_name_small in self.modules \
-                                   or 'all' in self.modules) \
+                                   or 'auto' in self.modules) \
                            else False
 
         load_underscore = True if module_name.startswith('_') \

1.9.1

2018-05-31 11:32 GMT+03:00 Chen Qi <[email protected]>:

Nice! Can we also try to simplify the setting of DEFAULT_TEST_SUITES in testimage.bbclass using this keyword? Right now it's basically manually created lists, which tend to go out of date.

Alex

Hi Alex,

I agree with you that we need to re-evaluate the settings of DEFAULT_TEST_SUITES.

I plan to do this work from next week. And I plan to put such changes in another patchset.

Looking at the codes, I can see that the DEFAULT_TEST_SUITES are set with consideration of qemu machines and libc. e.g.

# aarch64 has no graphics
DEFAULT_TEST_SUITES_remove_aarch64 = "xorg"
# musl doesn't support systemtap
DEFAULT_TEST_SUITES_remove_libc-musl = "stap"

And I guess this work would take some time.

Back to this patchset, I'll sent out V2 soon.

Best Regards,
Chen Qi

2018-06-01 7:42 GMT+03:00 ChenQi <[email protected]>:

I suppose we can introduce TEST_SUITES_EXCLUDE for such cases, which would be considered when 'auto' is expanded to a specific list?

Alex

I don't know yet. I guess if test case is written with more sophisticated considerations, we don't need such variable. But I'm not sure about it. I'll start to investigate it from next week.

Best Regards,
Chen Qi

This patchset looks good, thanks for working on it.

The original intent of some of these values was to indicate which tests were expected to pass for which images. I think in 2.6 we have some people working on collecting up the test results for a given build, then having a comparison tool which compares that set of results with a list of tests we expect to pass.

The desire here is to detect when something that should get tested is no longer getting tested because the test isn't running.

I'm pretty sure the existing mechanism isn't fit for purpose though.

I also want to change some things like getting rid of testimage_auto and changing that to be configurable with a variable. I'd noticed it was totally broken so I doubt anyone can have been using it...

Cheers,

Richard

The current behaviour of TEST_SUITES is very confusing.

setting: TEST_SUITES = ""
result:  Execute all test cases.

setting: TEST_SUITES = "some_case_not_exist"
result:  Error out with 'Empty test suite' message.

The expected behaviour of TEST_SUITES should be:
1. when 'auto' is in it, execute as many test cases as possible
2. when no valid test case is specified, error out and give user message

The first one is implemented by a previous patch.
The second one is fixed in this patch.

Also add debug message to show test cases to be executed. This is for easier debugging.

---
 meta/classes/testimage.bbclass | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

index bb688b0..d97da67 100644
@@ -260,10 +260,16 @@ def testimage_main(d):
     # Load tests before starting the target
     test_paths = get_runtime_paths(d)
     test_modules = d.getVar('TEST_SUITES').split()
+    if not test_modules:
+        bb.fatal('Empty test suite, please verify TEST_SUITES variable')
+
     tc.loadTests(test_paths, modules=test_modules)
 
-    if not getSuiteCases(tc.suites):
+    suitecases = getSuiteCases(tc.suites)
+    if not suitecases:
         bb.fatal('Empty test suite, please verify TEST_SUITES variable')
+    else:
+        bb.debug(2, 'test suites:\n\t%s' % '\n\t'.join([str(c) for c in suitecases]))
 
     package_extraction(d, tc.suites)
 

1.9.1

OETestFilter is a subclass of OETestDecorator. It wants to make use of @abstractmethod decorator. But such decorator requires metaclass to be ABCMeta to have effect. So add it now to achieve the designed behaviour.

Comments from python's manual:
"""
Using this decorator requires that the class's metaclass is ABCMeta or is derived from it.
"""

---
 meta/lib/oeqa/core/decorator/__init__.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

index 855b6b9..6ca0acf 100644
@@ -2,7 +2,7 @@
 # Released under the MIT license (see COPYING.MIT)
 
 from functools import wraps
-from abc import abstractmethod
+from abc import abstractmethod, ABCMeta
 
 decoratorClasses = set()
 

@@ -10,7 +10,7 @@ def registerDecorator(obj):
     decoratorClasses.add(obj)
     return obj
 
-class OETestDecorator(object):
+class OETestDecorator(object, metaclass=ABCMeta):
     case = None # Reference of OETestCase decorated
     attrs = None # Attributes to be loaded by decorator implementation
 

1.9.1

Use 'cls' instead of 'obj' to better reflect that registerDecorator actually serves as a class decorator.

---
 meta/lib/oeqa/core/decorator/__init__.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

index 6ca0acf..14d7bfc 100644
@@ -6,9 +6,9 @@ from abc import abstractmethod, ABCMeta
 
 decoratorClasses = set()
 
-def registerDecorator(obj):
-    decoratorClasses.add(obj)
-    return obj
+def registerDecorator(cls):
+    decoratorClasses.add(cls)
+    return cls
 
 class OETestDecorator(object, metaclass=ABCMeta):
     case = None # Reference of OETestCase decorated

1.9.1

This test is to test the behaviour of PACKAGE_FEED_URIS is correct or not. If it's not even set, it makes no sense to do such test. So skip this test if PACKAGE_FEED_URIS is not set.

---
 meta-selftest/lib/oeqa/runtime/cases/dnf_runtime.py | 3 +++
 1 file changed, 3 insertions(+)

index 81c50ed..1aa7274 100644
@@ -1,6 +1,7 @@
 from oeqa.core.decorator.depends import OETestDepends
 from oeqa.runtime.cases.dnf import DnfTest
 from oeqa.utils.httpserver import HTTPService
+from oeqa.core.decorator.data import skipIfDataVar
 
 class DnfSelftest(DnfTest):
 

@@ -18,6 +19,8 @@ class DnfSelftest(DnfTest):
         cls.temp_dir.cleanup()
 
     @OETestDepends(['dnf.DnfBasicTest.test_dnf_help'])
+    @skipIfDataVar('PACKAGE_FEED_URIS', None,
+                   'Not suitable as PACKAGE_FEED_URIS is not set')
     def test_verify_package_feeds(self):
         """
         Summary: Check correct setting of PACKAGE_FEED_URIS var

1.9.1

This test modules is designed to be invoked only by selftest. It's not meant to be tested by normal runtime test. So it should be renamed with '_' prefix, so that it will not be automatically loaded by normal runtime tests when 'auto' is in TEST_SUITES.

The failure message is as below.

RESULTS - selftest.Selftest.test_install_package - Testcase -1: FAILED

Also, change selftest/cases/runtime_test.py to use '_selftest' accordingly.

---
 meta-selftest/lib/oeqa/runtime/cases/{selftest.py => _selftest.py} | 0
 meta/lib/oeqa/selftest/cases/runtime_test.py                       | 2 +-
 2 files changed, 1 insertion(+), 1 deletion(-)
 rename meta-selftest/lib/oeqa/runtime/cases/{selftest.py => _selftest.py} (100%)

similarity index 100%
rename from meta-selftest/lib/oeqa/runtime/cases/selftest.py
rename to meta-selftest/lib/oeqa/runtime/cases/_selftest.py

index 9c9b4b3..def3819 100644
@@ -122,7 +122,7 @@ class TestImage(OESelftestTestCase):
             self.skipTest('core-image-full-cmdline not buildable for poky-tiny')
 
         features = 'INHERIT += "testimage"\n'
-        features += 'TEST_SUITES = "ping ssh selftest"\n'
+        features += 'TEST_SUITES = "ping ssh _selftest"\n'
         self.write_config(features)
 
         # Build core-image-sato and testimage

1.9.1

Having 'auto' in TEST_SUITES will also run the 'dnf' test cases, so also check it to determine whether to create rpm index or not.

This is to fix the following error when TEST_SUITES = "auto".

RESULTS - dnf.DnfRepoTest.test_dnf_makecache - Testcase 1744: ERROR

---
 meta/classes/testimage.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index d97da67..14252ff 100644
@@ -121,7 +121,7 @@ python do_testimage() {
     testimage_sanity(d)
 
     if (d.getVar('IMAGE_PKGTYPE') == 'rpm'
-       and 'dnf' in d.getVar('TEST_SUITES')):
+       and ('dnf' in d.getVar('TEST_SUITES') or 'auto' in d.getVar('TEST_SUITES'))):
         create_rpm_index(d)
 
     testimage_main(d)

1.9.1

The var might not be set, resulting in unexpected error.

RESULTS - multilib.MultilibTest.test_check_multilib_libc - Testcase 1593: ERROR

The above error is due to MULTILIBS being not set, which is the default for OE. This patch fixes this problem.

Also, the debugging message in skipIfNotInDataVar is currently confusing. Instead of
DEBUG: Checking if 'MULTILIBS' value is in 'multilib:lib32' to run the test it should be
DEBUG: Checking if 'MULTILIBS' value contains 'multilib:lib32' to run the test

This patch also fixes it.

---
 meta/lib/oeqa/core/decorator/data.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

index ff7bdd9..31c6dd6 100644
@@ -61,10 +61,10 @@ class skipIfNotInDataVar(OETestDecorator):
 
     attrs = ('var', 'value', 'msg')
     def setUpDecorator(self):
-        msg = ('Checking if %r value is in %r to run '
+        msg = ('Checking if %r value contains %r to run '
               'the test' % (self.var, self.value))
         self.logger.debug(msg)
-        if not self.value in self.case.td.get(self.var):
+        if not self.value in (self.case.td.get(self.var) or ""):
             self.case.skipTest(self.msg)
 
 @registerDecorator

1.9.1

This test case should only run when rpm package is installed. So skip it if rpm package is not installed. This fixes:

RESULTS - rpm.RpmBasicTest.test_rpm_help - Testcase 1059: FAILED

---
 meta/lib/oeqa/runtime/cases/rpm.py | 1 +
 1 file changed, 1 insertion(+)

index 05b94c7..84c59a6 100644
@@ -16,6 +16,7 @@ class RpmBasicTest(OERuntimeTestCase):
             cls.skipTest('Tests require image to be build from rpm')
 
     @OETestID(960)
+    @OEHasPackage(['rpm'])
     @OETestDepends(['ssh.SSHTest.test_ssh'])
     def test_rpm_help(self):
         status, output = self.target.run('rpm --help')

1.9.1

1) The test cases use 'readelf' command to do the check. This command is from binutils. So skip the test if the needed binutils package is not installed.

The related error message in log.do_testimage is like below.

Output: sh: readelf: not found

2) The test case tests /lib/libc.so.6 from lib32-libc6. So skip the test if lib32-libc6 is not installed.

The related error message in log.do_testimage is like below.

Output: readelf: Error: 'lib/libc.so.6': No such file

---
 meta/lib/oeqa/runtime/cases/multilib.py | 2 ++
 1 file changed, 2 insertions(+)

index 8c167f1..970f676 100644
@@ -27,6 +27,8 @@ class MultilibTest(OERuntimeTestCase):
     @skipIfNotInDataVar('MULTILIBS', 'multilib:lib32',
                         "This isn't a multilib:lib32 image")
     @OETestDepends(['ssh.SSHTest.test_ssh'])
+    @OEHasPackage(['binutils'])
+    @OEHasPackage(['lib32-libc6'])
     def test_check_multilib_libc(self):
         """
         Check that a multilib image has both 32-bit and 64-bit libc in.

1.9.1

The test_file_connman should be executed only when 'lib32-connman' is installed and 'connman' is not installed.

When lib32-connman and connman are both installed, the /usr/sbin/connmand could be from connman or lib32-connman, depending on the installation order. What we want to check is the connmand command from lib32-connman, so we need to make sure that connman is not there to cause chaos.

---
 meta/lib/oeqa/runtime/cases/multilib.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 970f676..8902038 100644
@@ -38,6 +38,6 @@ class MultilibTest(OERuntimeTestCase):
 
     @OETestID(279)
     @OETestDepends(['multilib.MultilibTest.test_check_multilib_libc'])
-    @OEHasPackage(['lib32-connman'])
+    @OEHasPackage(['lib32-connman', '!connman'])
     def test_file_connman(self):
         self.archtest("/usr/sbin/connmand", "ELF32")

1.9.1

A list of tools are added to HOSTTOOLS depending on if we inherit testimage or not. Unfortunately, if we use TEST_IMAGE variable to automate the test, these tools are not added to HOSTTOOLS.

Modify the condition to also check TEST_IMAGE to fix the above problem.

---
 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 6ecf78b..7b7472c 100644
@@ -487,7 +487,7 @@ HOSTTOOLS += " \
 "
 
 # Tools needed to run testimage runtime image testing
-HOSTTOOLS += "${@['', 'ip ping ps scp ssh stty'][bb.data.inherits_class('testimage', d)]}"
+HOSTTOOLS += "${@['', 'ip ping ps scp ssh stty'][bb.data.inherits_class('testimage', d) or d.getVar('TEST_IMAGE') == '1']}"
 
 # Link to these if present
 HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc sftp socat ssh sudo"

1.9.1

While you're at it, I'd suggest fixing this to stop using the horrible indexing a list with a boolean hack and use a proper inline conditional. 'foo' if bar else 'baz'.

Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics

Done.
V2 will be sent out soon.

Thanks,
Chen Qi

testimage-auto is expected to run testimage task's codes automatically. But in fact, it's currently missing some codes, including testimage_sanity and create_rpm_index.

This leads to the problem of unexpected runtime failure of test_dnf_makecache. The error message is as below.

RESULTS - dnf.DnfRepoTest.test_dnf_makecache - Testcase 1744: ERROR

This error is caused by the fact that create_rpm_index is not executed before running the tests.

---
 meta/classes/testimage-auto.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)

index e0a22b7..7da516a 100644
@@ -16,6 +16,12 @@
 inherit testimage
 
 python do_testimage_auto() {
+    testimage_sanity(d)
+
+    if (d.getVar('IMAGE_PKGTYPE') == 'rpm'
+       and ('dnf' in d.getVar('TEST_SUITES') or 'auto' in d.getVar('TEST_SUITES'))):
+        create_rpm_index(d)
+
     testimage_main(d)
 }
 addtask testimage_auto before do_build after do_image_complete

1.9.1

2018-05-31 11:45 GMT+03:00 Chen Qi <[email protected]>:

Done.
V2 will be sent out soon.

Thanks,
Chen Qi

When running testimage task for core-image-sato-sdk, the following error appeared.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 at position 0: invalid start byte

Checking the codes, I found it's caused by setting a 1024 limit for the read method of the StreamReader object.

Comments from the manual:
"""
The chars argument indicates the number of decoded code points or bytes to return. The read() method will never return more data than requested, but it might return less, if there is not enough available.
"""

When running `systemctl status --full' on target, this error occurs.

This patch increase the bytes limit to 4096 to fix the error.

---
 meta/lib/oeqa/core/target/ssh.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 151b99a..8ff1f6c 100644
@@ -208,7 +208,7 @@ def SSHCall(command, logger, timeout=None, **opts):
                 try:
                     if select.select([process.stdout], [], [], 5)[0] != []:
                         reader = codecs.getreader('utf-8')(process.stdout)
-                        data = reader.read(1024, 1024)
+                        data = reader.read(1024, 4096)
                         if not data:
                             process.stdout.close()
                             eof = True

1.9.1