Skip to contentSkip to conversation

recipeutils fixes for devtool

PATCH In review Paul Eggleton 2 participants 13 messages Nov 28, 2018 – Nov 29, 2018

lore
Share & tools
lore

Paul Eggleton

author
#0 lore raw

cover recipeutils fixes for devtool

A couple of fixes for issues in the recipeutils module (which affect devtool), a cleanup patch, and some additional related tests for oe-selftest.

The following changes since commit 21387613fec1a8c142ed48d7a74d587e205b0c98:

gst-validate: upgrade 1.14.2 -> 1.14.4 (2018-11-27 22:11:22 +0000)

are available in the Git repository at:

git://git.openembedded.org/openembedded-core-contrib paule/devtool-recipeutils-fixes
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=paule/devtool-recipeutils-fixes

Paul Eggleton (4):
lib/oe/recipeutils: patch_recipe(): fix handling of values across
includes/classes
lib/oe/recipeutils: patch_recipe(): fix replacing varflag values lib/oe/recipeutils: drop obsolete functions
oe-selftest: add some tests for recipeutils module

 meta/lib/oe/recipeutils.py                  | 104 +++++++++++----
 meta/lib/oeqa/selftest/cases/recipeutils.py | 135 ++++++++++++++++++++
 2 files changed, 211 insertions(+), 28 deletions(-)
 create mode 100644 meta/lib/oeqa/selftest/cases/recipeutils.py
Signature · 1 line
2.17.2
author
#1 lore raw

PATCH 1/4 lib/oe/recipeutils: patch_recipe(): fix handling of values across includes/classes

If we were setting a variable and part of the variable's value was being set in a class or a .inc file, we were still just setting the value outright on the assumption that it was too hard to do otherwise. With some careful use of the variable history we can do better for certain situations i.e. when the recipe does not currently set the value outright.

Additionally, correctly remove _appends for variables we are changing if we're trying to remove the value added in the _append.

Fixes [YOCTO #12623] and partially fixes [YOCTO #9360].

Signed-off-by Paul Eggleton <[email protected]>
---
 meta/lib/oe/recipeutils.py | 71 ++++++++++++++++++++++++++++++++++++--
 1 file changed, 68 insertions(+), 3 deletions(-)
meta/lib/oe/recipeutils.py +68 -3
index aa64553c06b..9de291f5b59 100644
@@ -52,6 +52,29 @@ def parse_recipe(cooker, fn, appendfiles):
     return envdata
 
 
+def simplify_history(history, d):
+    """
+    Eliminate any irrelevant events from a variable history
+    """
+    ret_history = []
+    has_set = False
+    # Go backwards through the history and remove any immediate operations
+    # before the most recent set
+    for event in reversed(history):
+        if 'flag' in event or not 'file' in event:
+            continue
+        if event['op'] == 'set':
+            if has_set:
+                continue
+            has_set = True
+        elif event['op'] in ('append', 'prepend', 'postdot', 'predot'):
+            # Reminder: "append" and "prepend" mean += and =+ respectively, NOT _append / _prepend
+            if has_set:
+                continue
+        ret_history.insert(0, event)
+    return ret_history
+
+
 def get_var_files(fn, varlist, d):
     """Find the file in which each of a list of variables is set.
     Note: requires variable history to be enabled when parsing.

@@ -176,7 +199,14 @@ def patch_recipe_lines(fromlines, values, trailing_newline=True):
     def outputvalue(name, lines, rewindcomments=False):
         if values[name] is None:
             return
-        rawtext = '%s = "%s"%s' % (name, values[name], newline)
+        if isinstance(values[name], tuple):
+            op, value = values[name]
+            if op == '+=' and value.strip() == '':
+                return
+        else:
+            value = values[name]
+            op = '='
+        rawtext = '%s %s "%s"%s' % (name, op, value, newline)
         addlines = []
         nowrap = False
         for nowrap_re in nowrap_vars_res:

@@ -186,10 +216,10 @@ def patch_recipe_lines(fromlines, values, trailing_newline=True):
         if nowrap:
             addlines.append(rawtext)
         elif name in list_vars:
-            splitvalue = split_var_value(values[name], assignment=False)
+            splitvalue = split_var_value(value, assignment=False)
             if len(splitvalue) > 1:
                 linesplit = ' \\\n' + (' ' * (len(name) + 4))
-                addlines.append('%s = "%s%s"%s' % (name, linesplit.join(splitvalue), linesplit, newline))
+                addlines.append('%s %s "%s%s"%s' % (name, op, linesplit.join(splitvalue), linesplit, newline))
             else:
                 addlines.append(rawtext)
         else:

@@ -321,12 +351,47 @@ def patch_recipe(d, fn, varvalues, patch=False, relpath='', redirect_output=None
     """Modify a list of variable values in the specified recipe. Handles inc files if
     used by the recipe.
     """
+    overrides = d.getVar('OVERRIDES').split(':')
+    def override_applicable(hevent):
+        op = hevent['op']
+        if '[' in op:
+            opoverrides = op.split('[')[1].split(']')[0].split('_')
+            for opoverride in opoverrides:
+                if not opoverride in overrides:
+                    return False
+        return True
+
     varlist = varvalues.keys()
+    fn = os.path.abspath(fn)
     varfiles = get_var_files(fn, varlist, d)
     locs = localise_file_vars(fn, varfiles, varlist)
     patches = []
     for f,v in locs.items():
         vals = {k: varvalues[k] for k in v}
+        f = os.path.abspath(f)
+        if f == fn:
+            extravals = {}
+            for var, value in vals.items():
+                if var in list_vars:
+                    history = simplify_history(d.varhistory.variable(var), d)
+                    recipe_set = False
+                    for event in history:
+                        if os.path.abspath(event['file']) == fn:
+                            if event['op'] == 'set':
+                                recipe_set = True
+                    if not recipe_set:
+                        for event in history:
+                            if event['op'].startswith('_remove'):
+                                continue
+                            if not override_applicable(event):
+                                continue
+                            newvalue = value.replace(event['detail'], '')
+                            if newvalue == value and os.path.abspath(event['file']) == fn and event['op'].startswith('_'):
+                                op = event['op'].replace('[', '_').replace(']', '')
+                                extravals[var + op] = None
+                            value = newvalue
+                            vals[var] = ('+=', value)
+            vals.update(extravals)
         patchdata = patch_recipe_file(f, vals, patch, relpath, redirect_output)
         if patch:
             patches.append(patchdata)
Signature · 1 line
2.17.2
author
#2 lore raw

PATCH 2/4 lib/oe/recipeutils: patch_recipe(): fix replacing varflag values

In reply to Paul Eggleton

The code here wasn't correctly getting the variable history for varflags, so for example if you did a devtool upgrade on a recipe where the SRC_URI checksums were in the .inc file (typical for python recipes in order to support both python 2 and 3) then after the upgrade the new values would be set in the recipe and the old values were left in the .inc, which is not right. Teach the code here how to get the history for varflags so it works properly.

Signed-off-by Paul Eggleton <[email protected]>
---
 meta/lib/oe/recipeutils.py | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)
meta/lib/oe/recipeutils.py +12 -4
index 9de291f5b59..886ad26f178 100644
@@ -81,11 +81,19 @@ def get_var_files(fn, varlist, d):
     """
     varfiles = {}
     for v in varlist:
-        history = d.varhistory.variable(v)
         files = []
-        for event in history:
-            if 'file' in event and not 'flag' in event:
-                files.append(event['file'])
+        if '[' in v:
+            varsplit = v.split('[')
+            varflag = varsplit[1].split(']')[0]
+            history = d.varhistory.variable(varsplit[0])
+            for event in history:
+                if 'file' in event and event.get('flag', '') == varflag:
+                    files.append(event['file'])
+        else:
+            history = d.varhistory.variable(v)
+            for event in history:
+                if 'file' in event and not 'flag' in event:
+                    files.append(event['file'])
         if files:
             actualfile = files[-1]
         else:
Signature · 1 line
2.17.2
author
#3 lore raw

PATCH 3/4 lib/oe/recipeutils: drop obsolete functions

In reply to Paul Eggleton

These date from the time before Tinfoil's API covered this functionality (back when you could actually access cooker from a tinfoil-based script).

Signed-off-by Paul Eggleton <[email protected]>
---
 meta/lib/oe/recipeutils.py | 25 -------------------------
 1 file changed, 25 deletions(-)
meta/lib/oe/recipeutils.py +0 -25
index 886ad26f178..9c99164d248 100644
@@ -27,31 +27,6 @@ list_vars = ['SRC_URI', 'LIC_FILES_CHKSUM']
 meta_vars = ['SUMMARY', 'DESCRIPTION', 'HOMEPAGE', 'BUGTRACKER', 'SECTION']
 
 
-def pn_to_recipe(cooker, pn, mc=''):
-    """Convert a recipe name (PN) to the path to the recipe file"""
-
-    best = cooker.findBestProvider(pn, mc)
-    return best[3]
-
-
-def get_unavailable_reasons(cooker, pn):
-    """If a recipe could not be found, find out why if possible"""
-    import bb.taskdata
-    taskdata = bb.taskdata.TaskData(None, skiplist=cooker.skiplist)
-    return taskdata.get_reasons(pn)
-
-
-def parse_recipe(cooker, fn, appendfiles):
-    """
-    Parse an individual recipe file, optionally with a list of
-    bbappend files.
-    """
-    import bb.cache
-    parser = bb.cache.NoCache(cooker.databuilder)
-    envdata = parser.loadDataFull(fn, appendfiles)
-    return envdata
-
-
 def simplify_history(history, d):
     """
     Eliminate any irrelevant events from a variable history
Signature · 1 line
2.17.2
author
#4 lore raw

PATCH 4/4 oe-selftest: add some tests for recipeutils module

In reply to Paul Eggleton

Add some tests for functions in meta/lib/oe/recipeutils.py, in particular for a few issues I've just fixed. I haven't added tests for all of the functions - some of them are already being tested via devtool in any case.

Signed-off-by Paul Eggleton <[email protected]>
---
 meta/lib/oeqa/selftest/cases/recipeutils.py | 135 ++++++++++++++++++++
 1 file changed, 135 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/cases/recipeutils.py
meta/lib/oeqa/selftest/cases/recipeutils.py new file +135 -0
new file mode 100644
index 00000000000..c7036c7f04e
@@ -0,0 +1,135 @@
+import os
+import re
+import time
+import logging
+import bb.tinfoil
+
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import runCmd
+from oeqa.core.decorator.oeid import OETestID
+
+
+def setUpModule():
+    global tinfoil
+    tinfoil = bb.tinfoil.Tinfoil(tracking=True)
+    tinfoil.prepare(config_only=False, quiet=2)
+
+
+def tearDownModule():
+    tinfoil.shutdown()
+
+
+class RecipeUtilsTests(OESelftestTestCase):
+    """ Tests for the recipeutils module functions """
+
+    def test_patch_recipe_varflag(self):
+        import oe.recipeutils
+        rd = tinfoil.parse_recipe('python3-async-test')
+        vals = {'SRC_URI[md5sum]': 'woot', 'LICENSE': 'something'}
+        patches = oe.recipeutils.patch_recipe(rd, rd.getVar('FILE'), vals, patch=True)
+
+        expected_patch = """
+--- a/../meta-selftest/recipes-devtools/python/python-async-test.inc
++++ b/../meta-selftest/recipes-devtools/python/python-async-test.inc
+@@ -1,14 +1,14 @@
+ SUMMARY = "Python framework to process interdependent tasks in a pool of workers"
+ HOMEPAGE = "http://github.com/gitpython-developers/async"
+ SECTION = "devel/python"
+-LICENSE = "BSD"
++LICENSE = "something"
+ LIC_FILES_CHKSUM = "file://PKG-INFO;beginline=8;endline=8;md5=88df8e78b9edfd744953862179f2d14e"
+ 
+ inherit pypi
+ 
+ PYPI_PACKAGE = "async"
+ 
+-SRC_URI[md5sum] = "9b06b5997de2154f3bc0273f80bcef6b"
++SRC_URI[md5sum] = "woot"
+ SRC_URI[sha256sum] = "ac6894d876e45878faae493b0cf61d0e28ec417334448ac0a6ea2229d8343051"
+ 
+ RDEPENDS_${PN} += "${PYTHON_PN}-threading"
+"""
+        patchlines = []
+        for f in patches:
+            for line in f:
+                patchlines.append(line)
+        self.maxDiff = None
+        self.assertEqual(''.join(patchlines).strip(), expected_patch.strip())
+
+
+    def test_patch_recipe_singleappend(self):
+        import oe.recipeutils
+        rd = tinfoil.parse_recipe('recipeutils-test')
+        val = rd.getVar('SRC_URI', False).split()
+        del val[1]
+        val = ' '.join(val)
+        vals = {'SRC_URI': val}
+        patches = oe.recipeutils.patch_recipe(rd, rd.getVar('FILE'), vals, patch=True)
+
+        expected_patch = """
+--- a/../meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
++++ b/../meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
+@@ -8,6 +8,4 @@
+ 
+ BBCLASSEXTEND = "native nativesdk"
+ 
+-SRC_URI += "file://somefile"
+-
+ SRC_URI_append = " file://anotherfile"
+"""
+        patchlines = []
+        for f in patches:
+            for line in f:
+                patchlines.append(line)
+        self.assertEqual(''.join(patchlines).strip(), expected_patch.strip())
+
+
+    def test_patch_recipe_appends(self):
+        import oe.recipeutils
+        rd = tinfoil.parse_recipe('recipeutils-test')
+        val = rd.getVar('SRC_URI', False).split()
+        vals = {'SRC_URI': val[0]}
+        patches = oe.recipeutils.patch_recipe(rd, rd.getVar('FILE'), vals, patch=True)
+
+        expected_patch = """
+--- a/../meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
++++ b/../meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
+@@ -8,6 +8,3 @@
+ 
+ BBCLASSEXTEND = "native nativesdk"
+ 
+-SRC_URI += "file://somefile"
+-
+-SRC_URI_append = " file://anotherfile"
+"""
+        patchlines = []
+        for f in patches:
+            for line in f:
+                patchlines.append(line)
+        self.assertEqual(''.join(patchlines).strip(), expected_patch.strip())
+
+
+    def test_validate_pn(self):
+        import oe.recipeutils
+        expected_results = {
+            'test': '',
+            'glib-2.0': '',
+            'gtk+': '',
+            'forcevariable': 'reserved',
+            'pn-something': 'reserved',
+            'test.bb': 'file',
+            'test_one': 'character',
+            'test!': 'character',
+        }
+
+        for pn, expected in expected_results.items():
+            result = oe.recipeutils.validate_pn(pn)
+            if expected:
+                self.assertIn(expected, result)
+            else:
+                self.assertEqual(result, '')
+
+    def test_split_var_value(self):
+        import oe.recipeutils
+        res = oe.recipeutils.split_var_value('test.1 test.2 ${@call_function("hi there world", false)} test.4')
+        self.assertEqual(res, ['test.1', 'test.2', '${@call_function("hi there world", false)}', 'test.4'])
Signature · 1 line
2.17.2
author
#5 lore raw

PATCH v2 4/4 oe-selftest: add some tests for recipeutils module

Add some tests for functions in meta/lib/oe/recipeutils.py, in particular for a few issues I've just fixed. I haven't added tests for all of the functions - some of them are already being tested via devtool in any case.

Signed-off-by Paul Eggleton <[email protected]>
---
 .../python/python-async-test.inc              |  16 +++
 .../python/python3-async-test_0.6.2.bb        |   2 +
 .../recipeutils/recipeutils-test.inc          |   5 +
 .../recipeutils/recipeutils-test/anotherfile  |   0
 .../recipeutils/recipeutils-test/somefile     |   0
 .../recipeutils/recipeutils-test_1.2.bb       |  13 ++
 meta/lib/oeqa/selftest/cases/recipeutils.py   | 135 ++++++++++++++++++
 7 files changed, 171 insertions(+)
 create mode 100644 meta-selftest/recipes-devtools/python/python-async-test.inc
 create mode 100644 meta-selftest/recipes-devtools/python/python3-async-test_0.6.2.bb
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test.inc
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test/anotherfile
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test/somefile
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
 create mode 100644 meta/lib/oeqa/selftest/cases/recipeutils.py
meta-selftest/recipes-devtools/python/python-async-test.inc new file +16 -0
new file mode 100644
index 00000000000..c9602e8e52d
@@ -0,0 +1,16 @@
+SUMMARY = "Python framework to process interdependent tasks in a pool of workers"
+HOMEPAGE = "http://github.com/gitpython-developers/async"
+SECTION = "devel/python"
+LICENSE = "BSD"
+LIC_FILES_CHKSUM = "file://PKG-INFO;beginline=8;endline=8;md5=88df8e78b9edfd744953862179f2d14e"
+
+inherit pypi
+
+PYPI_PACKAGE = "async"
+
+SRC_URI[md5sum] = "9b06b5997de2154f3bc0273f80bcef6b"
+SRC_URI[sha256sum] = "ac6894d876e45878faae493b0cf61d0e28ec417334448ac0a6ea2229d8343051"
+
+RDEPENDS_${PN} += "${PYTHON_PN}-threading"
+
+BBCLASSEXTEND = "nativesdk"
meta-selftest/recipes-devtools/python/python3-async-test_0.6.2.bb new file +2 -0
new file mode 100644
index 00000000000..22e241afb3c
@@ -0,0 +1,2 @@
+inherit setuptools3
+require python-async-test.inc
meta-selftest/recipes-test/recipeutils/recipeutils-test.inc new file +5 -0
new file mode 100644
index 00000000000..8490b902d75
@@ -0,0 +1,5 @@
+SRC_URI = "http://xorg.freedesktop.org/releases/individual/lib/libxshmfence-${PV}.tar.bz2"
+
+SRC_URI[md5sum] = "66662e76899112c0f99e22f2fc775a7e"
+SRC_URI[sha256sum] = "d21b2d1fd78c1efbe1f2c16dae1cb23f8fd231dcf891465b8debe636a9054b0c"
+
meta-selftest/recipes-test/recipeutils/recipeutils-test/anotherfile new file +0 -0
new file mode 100644
index 00000000000..e69de29bb2d
meta-selftest/recipes-test/recipeutils/recipeutils-test/somefile new file +0 -0
new file mode 100644
index 00000000000..e69de29bb2d
meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb new file +13 -0
new file mode 100644
index 00000000000..f6da97b2d43
@@ -0,0 +1,13 @@
+SUMMARY = "Test recipe for recipeutils.patch_recipe()"
+
+require recipeutils-test.inc
+
+LICENSE = "Proprietary"
+
+DEPENDS += "virtual/libx11"
+
+BBCLASSEXTEND = "native nativesdk"
+
+SRC_URI += "file://somefile"
+
+SRC_URI_append = " file://anotherfile"
meta/lib/oeqa/selftest/cases/recipeutils.py new file +135 -0

135 lines not on this page · raw message

Signature · 1 line
2.17.2
author
#12 lore raw
In reply to Paul Eggleton
Quoted 3 lines · NZDT Paul Eggleton

A couple of fixes for issues in the recipeutils module (which affect devtool), a cleanup patch, and some additional related tests for oe-selftest.

Oops, patch 4/4 missed out the actual meta-selftest recipes that a few of the tests rely upon. Branch re-pushed and v2 of 4/4 sent.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
author
#6 lore raw

PATCH v3 4/4 oe-selftest: add some tests for recipeutils module

In reply to Paul Eggleton

Add some tests for functions in meta/lib/oe/recipeutils.py, in particular for a few issues I've just fixed. I haven't added tests for all of the functions - some of them are already being tested via devtool in any case.

Signed-off-by Paul Eggleton <[email protected]>
---
 .../python/python-async-test.inc              |  16 ++
 .../python/python3-async-test_0.6.2.bb        |   2 +
 .../recipeutils/recipeutils-test.inc          |   5 +
 .../recipeutils/recipeutils-test/anotherfile  |   0
 .../recipeutils/recipeutils-test/somefile     |   0
 .../recipeutils/recipeutils-test_1.2.bb       |  13 ++
 meta/lib/oeqa/selftest/cases/recipeutils.py   | 138 ++++++++++++++++++
 7 files changed, 174 insertions(+)
 create mode 100644 meta-selftest/recipes-devtools/python/python-async-test.inc
 create mode 100644 meta-selftest/recipes-devtools/python/python3-async-test_0.6.2.bb
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test.inc
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test/anotherfile
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test/somefile
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
 create mode 100644 meta/lib/oeqa/selftest/cases/recipeutils.py
meta-selftest/recipes-devtools/python/python-async-test.inc new file +16 -0
new file mode 100644
index 00000000000..c9602e8e52d
@@ -0,0 +1,16 @@
+SUMMARY = "Python framework to process interdependent tasks in a pool of workers"
+HOMEPAGE = "http://github.com/gitpython-developers/async"
+SECTION = "devel/python"
+LICENSE = "BSD"
+LIC_FILES_CHKSUM = "file://PKG-INFO;beginline=8;endline=8;md5=88df8e78b9edfd744953862179f2d14e"
+
+inherit pypi
+
+PYPI_PACKAGE = "async"
+
+SRC_URI[md5sum] = "9b06b5997de2154f3bc0273f80bcef6b"
+SRC_URI[sha256sum] = "ac6894d876e45878faae493b0cf61d0e28ec417334448ac0a6ea2229d8343051"
+
+RDEPENDS_${PN} += "${PYTHON_PN}-threading"
+
+BBCLASSEXTEND = "nativesdk"
meta-selftest/recipes-devtools/python/python3-async-test_0.6.2.bb new file +2 -0
new file mode 100644
index 00000000000..22e241afb3c
@@ -0,0 +1,2 @@
+inherit setuptools3
+require python-async-test.inc
meta-selftest/recipes-test/recipeutils/recipeutils-test.inc new file +5 -0
new file mode 100644
index 00000000000..8490b902d75
@@ -0,0 +1,5 @@
+SRC_URI = "http://xorg.freedesktop.org/releases/individual/lib/libxshmfence-${PV}.tar.bz2"
+
+SRC_URI[md5sum] = "66662e76899112c0f99e22f2fc775a7e"
+SRC_URI[sha256sum] = "d21b2d1fd78c1efbe1f2c16dae1cb23f8fd231dcf891465b8debe636a9054b0c"
+
meta-selftest/recipes-test/recipeutils/recipeutils-test/anotherfile new file +0 -0
new file mode 100644
index 00000000000..e69de29bb2d
meta-selftest/recipes-test/recipeutils/recipeutils-test/somefile new file +0 -0
new file mode 100644
index 00000000000..e69de29bb2d
meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb new file +13 -0
new file mode 100644
index 00000000000..f6da97b2d43
@@ -0,0 +1,13 @@
+SUMMARY = "Test recipe for recipeutils.patch_recipe()"
+
+require recipeutils-test.inc
+
+LICENSE = "Proprietary"
+
+DEPENDS += "virtual/libx11"
+
+BBCLASSEXTEND = "native nativesdk"
+
+SRC_URI += "file://somefile"
+
+SRC_URI_append = " file://anotherfile"
meta/lib/oeqa/selftest/cases/recipeutils.py new file +138 -0

138 lines not on this page · raw message

Signature · 1 line
2.17.2
#7 lore raw

Re: [PATCH v3 4/4] oe-selftest: add some tests for recipeutils module

Quoted 6 lines · Paul Eggleton

Add some tests for functions in meta/lib/oe/recipeutils.py, in particular for a few issues I've just fixed. I haven't added tests for
all of the functions - some of them are already being tested via devtool
in any case.

Hate to say it but this still isn't working:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/28/steps/7/logs/step2d

probably due to it not working with the -j option :/

Cheers,

Richard

author
#8 lore raw

Re: [PATCH v3 4/4] oe-selftest: add some tests for recipeutils module

Quoted 8 lines · NZDT Richard Purdie
Quoted 6 lines · Paul Eggleton

Add some tests for functions in meta/lib/oe/recipeutils.py, in particular for a few issues I've just fixed. I haven't added tests for
all of the functions - some of them are already being tested via devtool
in any case.

Quoted 5 lines

Hate to say it but this still isn't working:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/28/steps/7/logs/step2d

probably due to it not working with the -j option :/

Right, so it seems that COREBASE isn't pointing where I'd expect it to here - instead of the copy of the core metadata that's made for the test, it's pointing to the original layer. Is that a bug?

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#9 lore raw

Re: [PATCH v3 4/4] oe-selftest: add some tests for recipeutils module

Quoted 14 lines · Paul Eggleton
Quoted 13 lines · NZDT Richard Purdie
Quoted 7 lines · Paul Eggleton

Add some tests for functions in meta/lib/oe/recipeutils.py, in particular for a few issues I've just fixed. I haven't added tests
for
all of the functions - some of them are already being tested via devtool
in any case.

Hate to say it but this still isn't working:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/28/steps/7/logs/step2d

Quoted 7 lines
Quoted 1 line

probably due to it not working with the -j option :/

Right, so it seems that COREBASE isn't pointing where I'd expect it to
here - instead of the copy of the core metadata that's made for the test, it's
pointing to the original layer. Is that a bug?

I put

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=74b7df6ff73764e108bd81e05835c8a7d134c187

together to fix this. Its not a bug since the tests modify meta- selftest and therefore when running in parallel, we create copies of that layer per build directory.

I do think we should start using the separate build directory always for oe-selftest for consistency though. It would also make ensuring some things like rm_work disabled easier.

Cheers,

Richard

author
#11 lore raw

PATCH v4 4/4 oe-selftest: add some tests for recipeutils module

In reply to Paul Eggleton

Add some tests for functions in meta/lib/oe/recipeutils.py, in particular for a few issues I've just fixed. I haven't added tests for all of the functions - some of them are already being tested via devtool in any case.

Signed-off-by Paul Eggleton <[email protected]>
---
 .../python/python-async-test.inc              |  16 ++
 .../python/python3-async-test_0.6.2.bb        |   2 +
 .../recipeutils/recipeutils-test.inc          |   5 +
 .../recipeutils/recipeutils-test/anotherfile  |   0
 .../recipeutils/recipeutils-test/somefile     |   0
 .../recipeutils/recipeutils-test_1.2.bb       |  13 ++
 meta/lib/oeqa/selftest/cases/recipeutils.py   | 137 ++++++++++++++++++
 7 files changed, 173 insertions(+)
 create mode 100644 meta-selftest/recipes-devtools/python/python-async-test.inc
 create mode 100644 meta-selftest/recipes-devtools/python/python3-async-test_0.6.2.bb
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test.inc
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test/anotherfile
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test/somefile
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
 create mode 100644 meta/lib/oeqa/selftest/cases/recipeutils.py
meta-selftest/recipes-devtools/python/python-async-test.inc new file +16 -0
new file mode 100644
index 00000000000..c9602e8e52d
@@ -0,0 +1,16 @@
+SUMMARY = "Python framework to process interdependent tasks in a pool of workers"
+HOMEPAGE = "http://github.com/gitpython-developers/async"
+SECTION = "devel/python"
+LICENSE = "BSD"
+LIC_FILES_CHKSUM = "file://PKG-INFO;beginline=8;endline=8;md5=88df8e78b9edfd744953862179f2d14e"
+
+inherit pypi
+
+PYPI_PACKAGE = "async"
+
+SRC_URI[md5sum] = "9b06b5997de2154f3bc0273f80bcef6b"
+SRC_URI[sha256sum] = "ac6894d876e45878faae493b0cf61d0e28ec417334448ac0a6ea2229d8343051"
+
+RDEPENDS_${PN} += "${PYTHON_PN}-threading"
+
+BBCLASSEXTEND = "nativesdk"
meta-selftest/recipes-devtools/python/python3-async-test_0.6.2.bb new file +2 -0
new file mode 100644
index 00000000000..22e241afb3c
@@ -0,0 +1,2 @@
+inherit setuptools3
+require python-async-test.inc
meta-selftest/recipes-test/recipeutils/recipeutils-test.inc new file +5 -0
new file mode 100644
index 00000000000..8490b902d75
@@ -0,0 +1,5 @@
+SRC_URI = "http://xorg.freedesktop.org/releases/individual/lib/libxshmfence-${PV}.tar.bz2"
+
+SRC_URI[md5sum] = "66662e76899112c0f99e22f2fc775a7e"
+SRC_URI[sha256sum] = "d21b2d1fd78c1efbe1f2c16dae1cb23f8fd231dcf891465b8debe636a9054b0c"
+
meta-selftest/recipes-test/recipeutils/recipeutils-test/anotherfile new file +0 -0
new file mode 100644
index 00000000000..e69de29bb2d
meta-selftest/recipes-test/recipeutils/recipeutils-test/somefile new file +0 -0
new file mode 100644
index 00000000000..e69de29bb2d
meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb new file +13 -0
new file mode 100644
index 00000000000..f6da97b2d43
@@ -0,0 +1,13 @@
+SUMMARY = "Test recipe for recipeutils.patch_recipe()"
+
+require recipeutils-test.inc
+
+LICENSE = "Proprietary"
+
+DEPENDS += "virtual/libx11"
+
+BBCLASSEXTEND = "native nativesdk"
+
+SRC_URI += "file://somefile"
+
+SRC_URI_append = " file://anotherfile"
meta/lib/oeqa/selftest/cases/recipeutils.py new file +137 -0

137 lines not on this page · raw message

Signature · 1 line
2.17.2
author
#10 lore raw

Re: [PATCH v3 4/4] oe-selftest: add some tests for recipeutils module

In reply to [email protected]
Quoted 23 lines · On Friday, 30 November 2018 12:09:58 PM NZDT…
Quoted 14 lines · Paul Eggleton
Quoted 13 lines · NZDT Richard Purdie
Quoted 7 lines · Paul Eggleton

Add some tests for functions in meta/lib/oe/recipeutils.py, in particular for a few issues I've just fixed. I haven't added tests
for
all of the functions - some of them are already being tested via devtool
in any case.

Hate to say it but this still isn't working:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/28/steps/7/logs/step2d

Quoted 7 lines
Quoted 1 line

probably due to it not working with the -j option :/

Right, so it seems that COREBASE isn't pointing where I'd expect it to
here - instead of the copy of the core metadata that's made for the test, it's
pointing to the original layer. Is that a bug?

Quoted 7 lines

I put

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=74b7df6ff73764e108bd81e05835c8a7d134c187

together to fix this. Its not a bug since the tests modify meta- selftest and therefore when running in parallel, we create copies of that layer per build directory.

OK, I've sent a fix - I discussed with RP on IRC but for others' benefit there's already a function in oeqa.utils.commands that gets the path to meta-selftest from BBLAYERS so I used that in v4 that I just sent. I also tested this one with -j which I hadn't with the previous versions.

Quoted 3 lines

I do think we should start using the separate build directory always for oe-selftest for consistency though. It would also make ensuring some things like rm_work disabled easier.

Probably since there are gotchas like this one.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre