Skip to contentSkip to conversation

Skip selftests depending on distro and its features

PATCH In review [email protected] 2 participants 10 messages Jan 9, 2017 – Jan 12, 2017

lore
Share & tools
lore

[email protected]

author
#0 lore raw

v3 cover Skip selftests depending on distro and its features

From: Leonardo Sandoval <[email protected]>

Originally, the selftests were defined for the 'poky' distro. However, there are more supported distros, including nodistro, poky-lsb and poky-tiny. These series takes into consideration the distro name and its features to skip certain tests.

[YOCTO #8525]

The following changes since commit e016eb10b075e280b4e78a04e47b59a173386421:

bitbake: bb/cooker: BBCooker stops notifier at shutdown (2017-01-06 00:01:07 +0000)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib lsandov1/oe-selftest-distro-agnostic
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lsandov1/oe-selftest-distro-agnostic

Leonardo Sandoval (4):
selftest: devtool: use distro agnostic recipes for devtool checks
selftest: imagefeatures: skip tests based on distro features and
supported fstypes
selftest: runtime-test: skip image-install test for poky-tiny
selftest: sstatetests: skip glibc-initial tests and use a more generic
image

 meta/lib/oeqa/selftest/devtool.py       | 19 +++++++++--------
 meta/lib/oeqa/selftest/imagefeatures.py | 11 ++++++++++
 meta/lib/oeqa/selftest/runtime-test.py  |  2 ++
 meta/lib/oeqa/selftest/sstate.py        |  1 +
 meta/lib/oeqa/selftest/sstatetests.py   | 36 +++++++++++++++++++++++----------
 5 files changed, 48 insertions(+), 21 deletions(-)
Signature · 1 line
2.1.4
author
#1 lore raw

PATCH v3 1/4 selftest: devtool: use distro agnostic recipes for devtool checks

From: Leonardo Sandoval <[email protected]>

The recipes being replaced are not compatible with all distros, so use mraa and virtual/make for some checks.

Signed-off-by Leonardo Sandoval <[email protected]>
---
 meta/lib/oeqa/selftest/devtool.py | 19 +++++++++----------
 1 file changed, 9 insertions(+), 10 deletions(-)
meta/lib/oeqa/selftest/devtool.py +9 -10
index 19c5ccf..bbdd8c6 100644
@@ -340,12 +340,11 @@ class DevtoolTests(DevtoolBase):
 
     @testcase(1161)
     def test_devtool_add_fetch_git(self):
-        # Fetch source
         tempdir = tempfile.mkdtemp(prefix='devtoolqa')
         self.track_for_cleanup(tempdir)
-        url = 'git://git.yoctoproject.org/libmatchbox'
-        checkrev = '462f0652055d89c648ddd54fd7b03f175c2c6973'
-        testrecipe = 'libmatchbox2'
+        url = 'gitsm://git.yoctoproject.org/mraa'
+        checkrev = 'ae127b19a50aa54255e4330ccfdd9a5d058e581d'
+        testrecipe = 'mraa'
         srcdir = os.path.join(tempdir, testrecipe)
         # Test devtool add
         self.track_for_cleanup(self.workspacedir)

@@ -353,7 +352,7 @@ class DevtoolTests(DevtoolBase):
         self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
         result = runCmd('devtool add %s %s -a -f %s' % (testrecipe, srcdir, url))
         self.assertTrue(os.path.exists(os.path.join(self.workspacedir, 'conf', 'layer.conf')), 'Workspace directory not created: %s' % result.output)
-        self.assertTrue(os.path.isfile(os.path.join(srcdir, 'configure.ac')), 'Unable to find configure.ac in source directory')
+        self.assertTrue(os.path.isfile(os.path.join(srcdir, 'imraa', 'imraa.c')), 'Unable to find configure.ac in source directory')
         # Test devtool status
         result = runCmd('devtool status')
         self.assertIn(testrecipe, result.output)

@@ -363,7 +362,7 @@ class DevtoolTests(DevtoolBase):
         self.assertIn('_git.bb', recipefile, 'Recipe file incorrectly named')
         checkvars = {}
         checkvars['S'] = '${WORKDIR}/git'
-        checkvars['PV'] = '1.12+git${SRCPV}'
+        checkvars['PV'] = '1.0+git${SRCPV}'
         checkvars['SRC_URI'] = url
         checkvars['SRCREV'] = '${AUTOREV}'
         self._test_recipe_contents(recipefile, checkvars, [])

@@ -372,7 +371,7 @@ class DevtoolTests(DevtoolBase):
         shutil.rmtree(srcdir)
         url_rev = '%s;rev=%s' % (url, checkrev)
         result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe, srcdir, url_rev))
-        self.assertTrue(os.path.isfile(os.path.join(srcdir, 'configure.ac')), 'Unable to find configure.ac in source directory')
+        self.assertTrue(os.path.isfile(os.path.join(srcdir, 'imraa', 'imraa.c')), 'Unable to find imraa/imraa.c in source directory')
         # Test devtool status
         result = runCmd('devtool status')
         self.assertIn(testrecipe, result.output)

@@ -590,8 +589,8 @@ class DevtoolTests(DevtoolBase):
     @testcase(1378)
     def test_devtool_modify_virtual(self):
         # Try modifying a virtual recipe
-        virtrecipe = 'virtual/libx11'
-        realrecipe = 'libx11'
+        virtrecipe = 'virtual/make'
+        realrecipe = 'make'
         tempdir = tempfile.mkdtemp(prefix='devtoolqa')
         self.track_for_cleanup(tempdir)
         self.track_for_cleanup(self.workspacedir)

@@ -1025,7 +1024,7 @@ class DevtoolTests(DevtoolBase):
         tempdir = tempfile.mkdtemp(prefix='devtoolqa')
         # Try devtool extract
         self.track_for_cleanup(tempdir)
-        result = runCmd('devtool extract virtual/libx11 %s' % tempdir)
+        result = runCmd('devtool extract virtual/make %s' % tempdir)
         self.assertTrue(os.path.exists(os.path.join(tempdir, 'Makefile.am')), 'Extracted source could not be found')
         # devtool extract shouldn't create the workspace
         self.assertFalse(os.path.exists(self.workspacedir))
Signature · 1 line
2.1.4
author
#2 lore raw

PATCH v3 2/4 selftest: imagefeatures: skip tests based on distro features and supported fstypes

In reply to [email protected]

From: Leonardo Sandoval <[email protected]>

Some images (clutter and weston) requires certain distros features to be present, so check presence before proceeding. Some distros does not support the ext4* fstypes (poky-tiny only supports cpio.gz) so skip the test if this is the case.

Signed-off-by Leonardo Sandoval <[email protected]>
---
 meta/lib/oeqa/selftest/imagefeatures.py | 11 +++++++++++
 1 file changed, 11 insertions(+)
meta/lib/oeqa/selftest/imagefeatures.py +11 -0
index d015c49..5d50c69 100644
@@ -77,6 +77,9 @@ class ImageFeatures(oeSelfTest):
         AutomatedBy: Daniel Istrate <[email protected]>
         """
 
+        if 'opengl' not in get_bb_var('DISTRO_FEATURES'):
+            self.skipTest('opengl not present on DISTRO_FEATURES so core-image-clutter cannot be built')
+
         # Build a core-image-clutter
         bitbake('core-image-clutter')
 

@@ -91,6 +94,10 @@ class ImageFeatures(oeSelfTest):
         AutomatedBy: Daniel Istrate <[email protected]>
         """
 
+        distro_features = get_bb_var('DISTRO_FEATURES')
+        if not ('opengl' in distro_features and 'wayland' in distro_features):
+            self.skipTest('neither opengl nor wayland present on DISTRO_FEATURES so core-image-weston cannot be built')
+
         features = 'DISTRO_FEATURES_append = " wayland"\n'
         features += 'CORE_IMAGE_EXTRA_INSTALL += "wayland weston"'
         self.write_config(features)

@@ -110,6 +117,10 @@ class ImageFeatures(oeSelfTest):
         features = 'IMAGE_FSTYPES += " ext4 ext4.bmap"'
         self.write_config(features)
 
+        image_fstypes = get_bb_var('IMAGE_FSTYPES')
+        if 'ext4 ext4.bmap' not in image_fstypes:
+            self.skipTest('Current distro does not allow "ext4 ext4.bmap" filesystem type. Supported: %s' % image_fstypes)
+
         image_name = 'core-image-minimal'
         bitbake(image_name)
Signature · 1 line
2.1.4
#3 lore raw

Re: [PATCH V3 2/4] selftest: imagefeatures: skip tests based on distro features and supported fstypes

meta/lib/oeqa/selftest/imagefeatures.py class ImageFeatures(oeSelfTest): · line 85

Quoted 12 lines · On 9 January 2017 at 17:45, <[email protected]> wrote:

+++ b/meta/lib/oeqa/selftest/imagefeatures.py

meta/lib/oeqa/selftest/imagefeatures.py +3 -0
@@ -77,6 +77,9 @@ class ImageFeatures(oeSelfTest):
         AutomatedBy: Daniel Istrate <[email protected]>
         """
 
+        if 'opengl' not in get_bb_var('DISTRO_FEATURES'):
+            self.skipTest('opengl not present on DISTRO_FEATURES so
 core-image-clutter cannot be built')
+
         # Build a core-image-clutter
         bitbake('core-image-clutter')

This test is pretty pointless, lets just delete it.

Quoted 6 lines
meta/lib/oeqa/selftest/imagefeatures.py +2 -0
@@ -91,6 +94,10 @@ class ImageFeatures(oeSelfTest):
         AutomatedBy: Daniel Istrate <[email protected]>
         """
 
+        distro_features = get_bb_var('DISTRO_FEATURES')
+        if not ('opengl' in distro_features and 'wayland' in
Quoted 8 lines

distro_features):
+ self.skipTest('neither opengl nor wayland present on
DISTRO_FEATURES so core-image-weston cannot be built')
+
features = 'DISTRO_FEATURES_append = " wayland"\n' features += 'CORE_IMAGE_EXTRA_INSTALL += "wayland weston"' self.write_config(features)

This test is 90% nonsense, let's fix it whilst we're here. After checking that opengl and wayland are in DISTRO_FEATURES it can just build
core-image-weston: DISTRO_FEATURES doesn't need appending as it already has wayland in, and CORE_IMAGE_EXTRA_INSTALL doesn't need extending as core-image-weston obviously already contains weston.

Ross

author
#4 lore raw

Re: [PATCH V3 2/4] selftest: imagefeatures: skip tests based on distro features and supported fstypes

Quoted 18 lines · Ross

On 9 January 2017 at 17:45, <[email protected] <mailto:[email protected]>> wrote:

+++ b/meta/lib/oeqa/selftest/imagefeatures.py @@ -77,6 +77,9 @@ class ImageFeatures(oeSelfTest):
AutomatedBy: Daniel Istrate
<[email protected] <mailto:[email protected]>>
"""

+ if 'opengl' not in get_bb_var('DISTRO_FEATURES'):
+ self.skipTest('opengl not present on DISTRO_FEATURES
so core-image-clutter cannot be built')
+
# Build a core-image-clutter
bitbake('core-image-clutter')

Quoted 1 line

This test is pretty pointless, lets just delete it.

why is this pointless? if this is the case, we should then remove the core-image-clutter.bb target.

Quoted 20 lines

@@ -91,6 +94,10 @@ class ImageFeatures(oeSelfTest):
AutomatedBy: Daniel Istrate
<[email protected] <mailto:[email protected]>>
"""

+ distro_features = get_bb_var('DISTRO_FEATURES')
+ if not ('opengl' in distro_features and 'wayland' in
distro_features):
+ self.skipTest('neither opengl nor wayland present on
DISTRO_FEATURES so core-image-weston cannot be built')
+
features = 'DISTRO_FEATURES_append = " wayland"\n' features += 'CORE_IMAGE_EXTRA_INSTALL += "wayland weston"' self.write_config(features)

This test is 90% nonsense, let's fix it whilst we're here. After checking that opengl and wayland are in DISTRO_FEATURES it can just build core-image-weston: DISTRO_FEATURES doesn't need appending as it already has wayland in, and CORE_IMAGE_EXTRA_INSTALL doesn't need extending as core-image-weston obviously already contains weston.

Ross

#5 lore raw

Re: [PATCH V3 2/4] selftest: imagefeatures: skip tests based on distro features and supported fstypes

Quoted 6 lines · Leonardo Sandoval

This test is pretty pointless, lets just delete it.

why is this pointless? if this is the case, we should then remove the core-image-clutter.bb target

The test just builds an image that contains clutter, but core-image-weston already does that and we dont really need to treat clutter any differently to any other library.

Yes, removing core-image-clutter would make sense too.

Ross

author
#6 lore raw

PATCH v3 3/4 selftest: runtime-test: skip image-install test for poky-tiny

In reply to [email protected]

From: Leonardo Sandoval <[email protected]>

poky-tiny cannot build full-cmdline image, so skip this test in this case.

Signed-off-by Leonardo Sandoval <[email protected]>
---
 meta/lib/oeqa/selftest/runtime-test.py | 2 ++
 1 file changed, 2 insertions(+)
meta/lib/oeqa/selftest/runtime-test.py +2 -0
index 20caa97..c0e283f 100644
@@ -95,6 +95,8 @@ class TestImage(oeSelfTest):
         Product: oe-core
         Author: Mariano Lopez <[email protected]>
         """
+        if self.distro == 'poky-tiny':
+            self.skipTest('core-image-full-cmdline not buildable for poky-tiny')
 
         features = 'INHERIT += "testimage"\n'
         features += 'TEST_SUITES = "ping ssh selftest"\n'
Signature · 1 line
2.1.4
author
#7 lore raw

PATCH v3 4/4 selftest: sstatetests: skip glibc-initial tests and use a more generic image

In reply to [email protected]

From: Leonardo Sandoval <[email protected]>

There are mainly two changes: 1) Some distros uses non-glibc so skip those tests requiring glibc-initial as target 2) instead of using core-image-sato, use a more generic image (core-image-minimal) for hash tests.

Signed-off-by Leonardo Sandoval <[email protected]>
---
 meta/lib/oeqa/selftest/sstate.py      |  1 +
 meta/lib/oeqa/selftest/sstatetests.py | 36 ++++++++++++++++++++++++-----------
 2 files changed, 26 insertions(+), 11 deletions(-)
meta/lib/oeqa/selftest/sstate.py +1 -0
index d27a45c..742f4d0 100644
@@ -16,6 +16,7 @@ class SStateBase(oeSelfTest):
         self.sstate_path = get_bb_var('SSTATE_DIR')
         self.hostdistro = get_bb_var('NATIVELSBSTRING')
         self.distro_specific_sstate = os.path.join(self.sstate_path, self.hostdistro)
+        self.tclibc = get_bb_var('TCLIBC')
 
     # Creates a special sstate configuration with the option to add sstate mirrors
     def config_sstate(self, temp_sstate_location=False, add_local_mirrors=[]):
meta/lib/oeqa/selftest/sstatetests.py +25 -11
index 6642539..2f2b942 100644
@@ -51,12 +51,15 @@ class SStateTests(SStateBase):
 
     @testcase(976)
     def test_sstate_creation_distro_nonspecific_pass(self):
-        self.run_test_sstate_creation(['glibc-initial'], distro_specific=False, distro_nonspecific=True, temp_sstate_location=True)
+        # glibc-initial is intended only for the glibc C library
+        if self.tclibc == 'glibc':
+            self.run_test_sstate_creation(['glibc-initial'], distro_specific=False, distro_nonspecific=True, temp_sstate_location=True)
 
     @testcase(1375)
     def test_sstate_creation_distro_nonspecific_fail(self):
-        self.run_test_sstate_creation(['glibc-initial'], distro_specific=True, distro_nonspecific=False, temp_sstate_location=True, should_pass=False)
-
+        # glibc-initial is intended only for the glibc C library
+        if self.tclibc == 'glibc':
+            self.run_test_sstate_creation(['glibc-initial'], distro_specific=True, distro_nonspecific=False, temp_sstate_location=True, should_pass=False)
 
     # Test the sstate files deletion part of the do_cleansstate task
     def run_test_cleansstate_task(self, targets, distro_specific=True, distro_nonspecific=True, temp_sstate_location=True):

@@ -78,16 +81,26 @@ class SStateTests(SStateBase):
     @testcase(977)
     def test_cleansstate_task_distro_specific_nonspecific(self):
         targetarch = get_bb_var('TUNE_ARCH')
-        self.run_test_cleansstate_task(['binutils-cross-' + targetarch, 'binutils-native', 'glibc-initial'], distro_specific=True, distro_nonspecific=True, temp_sstate_location=True)
+        targets = ['binutils-cross-'+ targetarch, 'binutils-native']
+        # glibc-initial is intended only for the glibc C library
+        if self.tclibc == 'glibc':
+            targets.append('glibc-initial')
+        self.run_test_cleansstate_task(targets, distro_specific=True, distro_nonspecific=True, temp_sstate_location=True)
 
     @testcase(1376)
     def test_cleansstate_task_distro_nonspecific(self):
-        self.run_test_cleansstate_task(['glibc-initial'], distro_specific=False, distro_nonspecific=True, temp_sstate_location=True)
+        # glibc-initial is intended only for the glibc C library
+        if self.tclibc == 'glibc':
+            self.run_test_cleansstate_task(['glibc-initial'], distro_specific=False, distro_nonspecific=True, temp_sstate_location=True)
 
     @testcase(1377)
     def test_cleansstate_task_distro_specific(self):
         targetarch = get_bb_var('TUNE_ARCH')
-        self.run_test_cleansstate_task(['binutils-cross-'+ targetarch, 'binutils-native', 'glibc-initial'], distro_specific=True, distro_nonspecific=False, temp_sstate_location=True)
+        targets = ['binutils-cross-'+ targetarch, 'binutils-native']
+        # glibc-initial is intended only for the glibc C library
+        if self.tclibc == 'glibc':
+            targets.append('glibc-initial')
+        self.run_test_cleansstate_task(targets, distro_specific=True, distro_nonspecific=False, temp_sstate_location=True)
 
 
     # Test rebuilding of distro-specific sstate files

@@ -240,7 +253,7 @@ SDKMACHINE = "x86_64"
 PACKAGE_CLASSES = "package_rpm package_ipk package_deb"
 """)
         self.track_for_cleanup(topdir + "/tmp-sstatesamehash")
-        bitbake("core-image-sato -S none")
+        bitbake("core-image-minimal -S none")
         self.write_config("""
 MACHINE = "qemux86"
 TMPDIR = "${TOPDIR}/tmp-sstatesamehash2"

@@ -250,16 +263,17 @@ SDKMACHINE = "i686"
 PACKAGE_CLASSES = "package_rpm package_ipk package_deb"
 """)
         self.track_for_cleanup(topdir + "/tmp-sstatesamehash2")
-        bitbake("core-image-sato -S none")
+        bitbake("core-image-minimal -S none")
 
         def get_files(d):
             f = []
             for root, dirs, files in os.walk(d):
-                if "core-image-sato" in root:
+                if "core-image-minimal" in root:
                     # SDKMACHINE changing will change
                     # do_rootfs/do_testimage/do_build stamps of images which
                     # is safe to ignore.
                     continue
+
                 f.extend(os.path.join(root, name) for name in files)
             return f
         files1 = get_files(topdir + "/tmp-sstatesamehash/stamps/")

@@ -283,13 +297,13 @@ TMPDIR = \"${TOPDIR}/tmp-sstatesamehash\"
 NATIVELSBSTRING = \"DistroA\"
 """)
         self.track_for_cleanup(topdir + "/tmp-sstatesamehash")
-        bitbake("core-image-sato -S none")
+        bitbake("core-image-minimal -S none")
         self.write_config("""
 TMPDIR = \"${TOPDIR}/tmp-sstatesamehash2\"
 NATIVELSBSTRING = \"DistroB\"
 """)
         self.track_for_cleanup(topdir + "/tmp-sstatesamehash2")
-        bitbake("core-image-sato -S none")
+        bitbake("core-image-minimal -S none")
 
         def get_files(d):
             f = []
Signature · 1 line
2.1.4
#8 lore raw

Re: [PATCH V3 4/4] selftest: sstatetests: skip glibc-initial tests and use a more generic image

Review

Quoted 3 lines · On 9 January 2017 at 17:45, <[email protected]> wrote:
(quoted diff) +1 -1
-        bitbake("core-image-sato -S none")
+        bitbake("core-image-minimal -S none")

As this is just stamp calculation, could we just do "world" instead...

Ross

author
#9 lore raw

Re: [PATCH V3 4/4] selftest: sstatetests: skip glibc-initial tests and use a more generic image

Quoted 9 lines · Ross

On 9 January 2017 at 17:45, <[email protected] <mailto:[email protected]>> wrote:

- bitbake("core-image-sato -S none")
+ bitbake("core-image-minimal -S none")

As this is just stamp calculation, could we just do "world" instead...

I tried s/core-image-sato/world/ but asserts are not prepared for the latter. Let me double check and provide log.

Quoted 1 line

Ross