Skip to contentSkip to conversation

devtool / recipetool bbappend support

PATCH In review Paul Eggleton 1 participant 7 messages May 18, 2015

lore
Share & tools
lore
#0 lore raw

cover devtool / recipetool bbappend support

Add support for creating/updating bbappends in both devtool (within the update-recipe subcommand) and in recipetool (with a new appendfile subcommand), as well as some related tidy-ups.

The following changes since commit aa2e02a4f78d87bd466bbf92ca57147066c5367f:

README: add "Contributing" section (2015-05-16 22:37:15 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib paule/devtool-bbappends
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool-bbappends

Paul Eggleton (5):
devtool / recipetool: ensure bb.note() gets printed lib/oe/patch: use with open() for all file operations
oe-selftest: move recipetool tests to their own module
recipetool: add appendfile subcommand
devtool: update-recipe: add option to write changes to bbappend

 .../recipes-test/recipetool/files/add-file.patch   |   8 +
 meta-selftest/recipes-test/recipetool/files/file1  |   2 +
 .../recipes-test/recipetool/files/installscript.sh |   3 +
 .../recipetool/files/selftest-replaceme-inst-func  |   1 +
 .../files/selftest-replaceme-inst-globfile         |   1 +
 .../files/selftest-replaceme-inst-todir-globfile   |   1 +
 .../recipetool/files/selftest-replaceme-orig       |   1 +
 .../files/selftest-replaceme-src-globfile          |   1 +
 .../recipetool/files/selftest-replaceme-todir      |   1 +
 .../recipetool/files/subdir/fileinsubdir           |   1 +
 .../recipetool/selftest-recipetool-appendfile.bb   |  42 +++
 meta/lib/oe/patch.py                               | 103 ++++--
 meta/lib/oe/recipeutils.py                         | 328 ++++++++++++++++++-
 meta/lib/oeqa/selftest/devtool.py                  | 283 ++++++++++++----
 meta/lib/oeqa/selftest/recipetool.py               | 364 +++++++++++++++++++++
 meta/lib/oeqa/utils/commands.py                    |  11 +
 scripts/lib/devtool/__init__.py                    |   2 +-
 scripts/lib/devtool/standard.py                    | 155 ++++++---
 scripts/lib/recipetool/append.py                   | 360 ++++++++++++++++++++
 scripts/recipetool                                 |   8 +-
 20 files changed, 1548 insertions(+), 128 deletions(-)
 create mode 100644 meta-selftest/recipes-test/recipetool/files/add-file.patch
 create mode 100644 meta-selftest/recipes-test/recipetool/files/file1
 create mode 100644 meta-selftest/recipes-test/recipetool/files/installscript.sh
 create mode 100644 meta-selftest/recipes-test/recipetool/files/selftest-replaceme-inst-func
 create mode 100644 meta-selftest/recipes-test/recipetool/files/selftest-replaceme-inst-globfile
 create mode 100644 meta-selftest/recipes-test/recipetool/files/selftest-replaceme-inst-todir-globfile
 create mode 100644 meta-selftest/recipes-test/recipetool/files/selftest-replaceme-orig
 create mode 100644 meta-selftest/recipes-test/recipetool/files/selftest-replaceme-src-globfile
 create mode 100644 meta-selftest/recipes-test/recipetool/files/selftest-replaceme-todir
 create mode 100644 meta-selftest/recipes-test/recipetool/files/subdir/fileinsubdir
 create mode 100644 meta-selftest/recipes-test/recipetool/selftest-recipetool-appendfile.bb
 create mode 100644 meta/lib/oeqa/selftest/recipetool.py
 create mode 100644 scripts/lib/recipetool/append.py
Signature · 1 line
2.1.0
#1 lore raw

PATCH 1/5 devtool / recipetool: ensure bb.note() gets printed

Most of the time when bb.note() gets called we want to see the output, so ensure the level is set appropriately depending on the command line options instead of being fixed at warning. (We don't want to see the notes for fetch/unpack/patch though as they are too verbose).

---
 scripts/lib/devtool/__init__.py | 2 +-
 scripts/lib/devtool/standard.py | 7 +++++++
 scripts/recipetool              | 2 +-
 3 files changed, 9 insertions(+), 2 deletions(-)
scripts/lib/devtool/__init__.py +1 -1
index 5a06c78..9ec1ef6 100644
@@ -85,6 +85,6 @@ def setup_tinfoil():
     import bb.tinfoil
     tinfoil = bb.tinfoil.Tinfoil()
     tinfoil.prepare(False)
-    tinfoil.logger.setLevel(logging.WARNING)
+    tinfoil.logger.setLevel(logger.getEffectiveLevel())
     return tinfoil
 
scripts/lib/devtool/standard.py +7 -0
index 61c0df9..122121a 100644
@@ -245,6 +245,11 @@ def _extract_source(srctree, keep_temp, devbranch, d):
     bb.utils.mkdirhier(srctree)
     os.rmdir(srctree)
 
+    # We don't want notes to be printed, they are too verbose
+    origlevel = bb.logger.getEffectiveLevel()
+    if logger.getEffectiveLevel() > logging.DEBUG:
+        bb.logger.setLevel(logging.WARNING)
+
     initial_rev = None
     tempdir = tempfile.mkdtemp(prefix='devtool')
     try:
@@ -349,6 +354,8 @@ def _extract_source(srctree, keep_temp, devbranch, d):
         shutil.move(srcsubdir, srctree)
         logger.info('Source tree extracted to %s' % srctree)
     finally:
+        bb.logger.setLevel(origlevel)
+
         if keep_temp:
             logger.info('Preserving temporary directory %s' % tempdir)
         else:
scripts/recipetool +1 -1
index 2cfa763..b7d3ee8 100755
@@ -40,7 +40,7 @@ def tinfoil_init():
     for plugin in plugins:
         if hasattr(plugin, 'tinfoil_init'):
             plugin.tinfoil_init(tinfoil)
-    tinfoil.logger.setLevel(logging.WARNING)
+    tinfoil.logger.setLevel(logger.getEffectiveLevel())
 
 def main():
 
Signature · 1 line
2.1.0
#2 lore raw

PATCH 2/5 lib/oe/patch: use with open() for all file operations

In reply to Paul Eggleton

with open(...)... is preferred for reading/writing files as it is neater and takes care of closing the file for you.

---
 meta/lib/oe/patch.py | 40 +++++++++++++++++++---------------------
 1 file changed, 19 insertions(+), 21 deletions(-)
meta/lib/oe/patch.py +19 -21
index f68d40f..e1f1c53 100644
@@ -115,7 +115,8 @@ class PatchTree(PatchSet):
     def _removePatchFile(self, all = False):
         if not os.path.exists(self.seriespath):
             return
-        patches = open(self.seriespath, 'r+').readlines()
+        with open(self.seriespath, 'r+') as f:
+            patches = f.readlines()
         if all:
             for p in reversed(patches):
                 self._removePatch(os.path.join(self.patchdir, p.strip()))
@@ -405,16 +406,15 @@ class QuiltTree(PatchSet):
         if not os.path.exists(self.dir):
             raise NotFoundError(self.dir)
         if os.path.exists(seriespath):
-            series = file(seriespath, 'r')
-            for line in series.readlines():
-                patch = {}
-                parts = line.strip().split()
-                patch["quiltfile"] = self._quiltpatchpath(parts[0])
-                patch["quiltfilemd5"] = bb.utils.md5_file(patch["quiltfile"])
-                if len(parts) > 1:
-                    patch["strippath"] = parts[1][2:]
-                self.patches.append(patch)
-            series.close()
+            with open(seriespath, 'r') as f:
+                for line in f.readlines():
+                    patch = {}
+                    parts = line.strip().split()
+                    patch["quiltfile"] = self._quiltpatchpath(parts[0])
+                    patch["quiltfilemd5"] = bb.utils.md5_file(patch["quiltfile"])
+                    if len(parts) > 1:
+                        patch["strippath"] = parts[1][2:]
+                    self.patches.append(patch)
 
             # determine which patches are applied -> self._current
             try:
@@ -436,9 +436,8 @@ class QuiltTree(PatchSet):
             self.InitFromDir()
         PatchSet.Import(self, patch, force)
         oe.path.symlink(patch["file"], self._quiltpatchpath(patch["file"]), force=True)
-        f = open(os.path.join(self.dir, "patches","series"), "a");
-        f.write(os.path.basename(patch["file"]) + " -p" + patch["strippath"]+"\n")
-        f.close()
+        with open(os.path.join(self.dir, "patches", "series"), "a") as f:
+            f.write(os.path.basename(patch["file"]) + " -p" + patch["strippath"] + "\n")
         patch["quiltfile"] = self._quiltpatchpath(patch["file"])
         patch["quiltfilemd5"] = bb.utils.md5_file(patch["quiltfile"])
 
@@ -559,13 +558,12 @@ class UserResolver(Resolver):
             bb.utils.mkdirhier(t)
             import random
             rcfile = "%s/bashrc.%s.%s" % (t, str(os.getpid()), random.random())
-            f = open(rcfile, "w")
-            f.write("echo '*** Manual patch resolution mode ***'\n")
-            f.write("echo 'Dropping to a shell, so patch rejects can be fixed manually.'\n")
-            f.write("echo 'Run \"quilt refresh\" when patch is corrected, press CTRL+D to exit.'\n")
-            f.write("echo ''\n")
-            f.write(" ".join(patchcmd) + "\n")
-            f.close()
+            with open(rcfile, "w") as f:
+                f.write("echo '*** Manual patch resolution mode ***'\n")
+                f.write("echo 'Dropping to a shell, so patch rejects can be fixed manually.'\n")
+                f.write("echo 'Run \"quilt refresh\" when patch is corrected, press CTRL+D to exit.'\n")
+                f.write("echo ''\n")
+                f.write(" ".join(patchcmd) + "\n")
             os.chmod(rcfile, 0775)
 
             self.terminal("bash --rcfile " + rcfile, 'Patch Rejects: Please fix patch rejects manually', self.patchset.d)
Signature · 1 line
2.1.0
#3 lore raw

PATCH 3/5 oe-selftest: move recipetool tests to their own module

In reply to Paul Eggleton

These tests really belong in their own module; if we refactor out a base class from DevtoolTests with shared functions then we can move them out easily. Also create temp directory in setupLocal() so we don't have to do that in individual tests anymore.

---
 meta/lib/oeqa/selftest/devtool.py    | 81 ++++++++++--------------------------
 meta/lib/oeqa/selftest/recipetool.py | 55 ++++++++++++++++++++++++
 2 files changed, 77 insertions(+), 59 deletions(-)
 create mode 100644 meta/lib/oeqa/selftest/recipetool.py
meta/lib/oeqa/selftest/devtool.py +22 -59
index 1d16113..f4571c4 100644
@@ -11,29 +11,7 @@ from oeqa.selftest.base import oeSelfTest
 from oeqa.utils.commands import runCmd, bitbake, get_bb_var
 from oeqa.utils.decorators import testcase
 
-class DevtoolTests(oeSelfTest):
-
-    def test_create_workspace(self):
-        # Check preconditions
-        workspacedir = os.path.join(self.builddir, 'workspace')
-        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
-        result = runCmd('bitbake-layers show-layers')
-        self.assertTrue('/workspace' not in result.output, 'This test cannot be run with a workspace layer in bblayers.conf')
-        # Try creating a workspace layer with a specific path
-        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
-        self.track_for_cleanup(tempdir)
-        result = runCmd('devtool create-workspace %s' % tempdir)
-        self.assertTrue(os.path.isfile(os.path.join(tempdir, 'conf', 'layer.conf')))
-        result = runCmd('bitbake-layers show-layers')
-        self.assertIn(tempdir, result.output)
-        # Try creating a workspace layer with the default path
-        self.track_for_cleanup(workspacedir)
-        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
-        result = runCmd('devtool create-workspace')
-        self.assertTrue(os.path.isfile(os.path.join(workspacedir, 'conf', 'layer.conf')))
-        result = runCmd('bitbake-layers show-layers')
-        self.assertNotIn(tempdir, result.output)
-        self.assertIn(workspacedir, result.output)
+class DevtoolBase(oeSelfTest):
 
     def _test_recipe_contents(self, recipefile, checkvars, checkinherits):
         with open(recipefile, 'r') as f:
@@ -53,45 +31,30 @@ class DevtoolTests(oeSelfTest):
         for inherit in checkinherits:
             self.assertIn(inherit, inherits, 'Missing inherit of %s' % inherit)
 
-    def test_recipetool_create(self):
-        # Try adding a recipe
-        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
-        self.track_for_cleanup(tempdir)
-        tempsrc = os.path.join(tempdir, 'srctree')
-        os.makedirs(tempsrc)
-        recipefile = os.path.join(tempdir, 'logrotate_3.8.7.bb')
-        srcuri = 'https://fedorahosted.org/releases/l/o/logrotate/logrotate-3.8.7.tar.gz'
-        result = runCmd('recipetool create -o %s %s -x %s' % (recipefile, srcuri, tempsrc))
-        self.assertTrue(os.path.isfile(recipefile))
-        checkvars = {}
-        checkvars['LICENSE'] = 'GPLv2'
-        checkvars['LIC_FILES_CHKSUM'] = 'file://COPYING;md5=18810669f13b87348459e611d31ab760'
-        checkvars['SRC_URI'] = 'https://fedorahosted.org/releases/l/o/logrotate/logrotate-${PV}.tar.gz'
-        checkvars['SRC_URI[md5sum]'] = '99e08503ef24c3e2e3ff74cc5f3be213'
-        checkvars['SRC_URI[sha256sum]'] = 'f6ba691f40e30e640efa2752c1f9499a3f9738257660994de70a45fe00d12b64'
-        self._test_recipe_contents(recipefile, checkvars, [])
 
-    def test_recipetool_create_git(self):
-        # Ensure we have the right data in shlibs/pkgdata
-        bitbake('libpng pango libx11 libxext jpeg')
-        # Try adding a recipe
+class DevtoolTests(DevtoolBase):
+
+    def test_create_workspace(self):
+        # Check preconditions
+        workspacedir = os.path.join(self.builddir, 'workspace')
+        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        result = runCmd('bitbake-layers show-layers')
+        self.assertTrue('/workspace' not in result.output, 'This test cannot be run with a workspace layer in bblayers.conf')
+        # Try creating a workspace layer with a specific path
         tempdir = tempfile.mkdtemp(prefix='devtoolqa')
         self.track_for_cleanup(tempdir)
-        tempsrc = os.path.join(tempdir, 'srctree')
-        os.makedirs(tempsrc)
-        recipefile = os.path.join(tempdir, 'libmatchbox.bb')
-        srcuri = 'git://git.yoctoproject.org/libmatchbox'
-        result = runCmd('recipetool create -o %s %s -x %s' % (recipefile, srcuri, tempsrc))
-        self.assertTrue(os.path.isfile(recipefile), 'recipetool did not create recipe file; output:\n%s' % result.output)
-        checkvars = {}
-        checkvars['LICENSE'] = 'LGPLv2.1'
-        checkvars['LIC_FILES_CHKSUM'] = 'file://COPYING;md5=7fbc338309ac38fefcd64b04bb903e34'
-        checkvars['S'] = '${WORKDIR}/git'
-        checkvars['PV'] = '1.0+git${SRCPV}'
-        checkvars['SRC_URI'] = srcuri
-        checkvars['DEPENDS'] = 'libpng pango libx11 libxext jpeg'
-        inherits = ['autotools', 'pkgconfig']
-        self._test_recipe_contents(recipefile, checkvars, inherits)
+        result = runCmd('devtool create-workspace %s' % tempdir)
+        self.assertTrue(os.path.isfile(os.path.join(tempdir, 'conf', 'layer.conf')))
+        result = runCmd('bitbake-layers show-layers')
+        self.assertIn(tempdir, result.output)
+        # Try creating a workspace layer with the default path
+        self.track_for_cleanup(workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        result = runCmd('devtool create-workspace')
+        self.assertTrue(os.path.isfile(os.path.join(workspacedir, 'conf', 'layer.conf')))
+        result = runCmd('bitbake-layers show-layers')
+        self.assertNotIn(tempdir, result.output)
+        self.assertIn(workspacedir, result.output)
 
     def test_devtool_add(self):
         # Check preconditions
meta/lib/oeqa/selftest/recipetool.py new file +55 -0

55 lines not on this page · raw message

Signature · 1 line
2.1.0
#4 lore raw

PATCH 4/5 recipetool: add appendfile subcommand

In reply to Paul Eggleton

Locating which recipe provides a file in an image that you want to modify and then figuring out how to bbappend the recipe in order to replace it can be a tedious process. Thus, add a new appendfile subcommand to recipetool, providing the ability to create a bbappend file to add/replace any file in the target system. Without the -r option, it will search for the recipe packaging the specified file (using pkgdata from previously built recipes). The bbappend will be created at the appropriate path within the specified layer directory (which may or may not be in your bblayers.conf) or if one already exists it will be updated appropriately.

Fairly extensive oe-selftest tests are also provided.

Implements [YOCTO #6447].

---
 .../recipes-test/recipetool/files/add-file.patch   |   8 +
 meta-selftest/recipes-test/recipetool/files/file1  |   2 +
 .../recipes-test/recipetool/files/installscript.sh |   3 +
 .../recipetool/files/selftest-replaceme-inst-func  |   1 +
 .../files/selftest-replaceme-inst-globfile         |   1 +
 .../files/selftest-replaceme-inst-todir-globfile   |   1 +
 .../recipetool/files/selftest-replaceme-orig       |   1 +
 .../files/selftest-replaceme-src-globfile          |   1 +
 .../recipetool/files/selftest-replaceme-todir      |   1 +
 .../recipetool/files/subdir/fileinsubdir           |   1 +
 .../recipetool/selftest-recipetool-appendfile.bb   |  42 +++
 meta/lib/oe/patch.py                               |  63 ++++
 meta/lib/oe/recipeutils.py                         | 328 ++++++++++++++++++-
 meta/lib/oeqa/selftest/devtool.py                  |  31 +-
 meta/lib/oeqa/selftest/recipetool.py               | 313 +++++++++++++++++-
 meta/lib/oeqa/utils/commands.py                    |  11 +
 scripts/lib/recipetool/append.py                   | 360 +++++++++++++++++++++
 scripts/recipetool                                 |   6 +-
 18 files changed, 1167 insertions(+), 7 deletions(-)
 create mode 100644 meta-selftest/recipes-test/recipetool/files/add-file.patch
 create mode 100644 meta-selftest/recipes-test/recipetool/files/file1
 create mode 100644 meta-selftest/recipes-test/recipetool/files/installscript.sh
 create mode 100644 meta-selftest/recipes-test/recipetool/files/selftest-replaceme-inst-func
 create mode 100644 meta-selftest/recipes-test/recipetool/files/selftest-replaceme-inst-globfile
 create mode 100644 meta-selftest/recipes-test/recipetool/files/selftest-replaceme-inst-todir-globfile
 create mode 100644 meta-selftest/recipes-test/recipetool/files/selftest-replaceme-orig
 create mode 100644 meta-selftest/recipes-test/recipetool/files/selftest-replaceme-src-globfile
 create mode 100644 meta-selftest/recipes-test/recipetool/files/selftest-replaceme-todir
 create mode 100644 meta-selftest/recipes-test/recipetool/files/subdir/fileinsubdir
 create mode 100644 meta-selftest/recipes-test/recipetool/selftest-recipetool-appendfile.bb
 create mode 100644 scripts/lib/recipetool/append.py
meta-selftest/recipes-test/recipetool/files/add-file.patch new file +8 -0
new file mode 100644
index 0000000..bdc99c9
@@ -0,0 +1,8 @@
+diff --git a/file2 b/file2
+new file mode 100644
+index 0000000..049b42e
+--- /dev/null
++++ b/file2
+@@ -0,0 +1,2 @@
++Test file 2
++456
meta-selftest/recipes-test/recipetool/files/file1 new file +2 -0
new file mode 100644
index 0000000..7571aa7
@@ -0,0 +1,2 @@
+First test file
+123
meta-selftest/recipes-test/recipetool/files/installscript.sh new file +3 -0
new file mode 100644
index 0000000..9de30d6
@@ -0,0 +1,3 @@
+#!/bin/sh
+echo "Third file" > $1/selftest-replaceme-scripted
+
meta-selftest/recipes-test/recipetool/files/selftest-replaceme-inst-func new file +1 -0
new file mode 100644
index 0000000..2802bb3
@@ -0,0 +1 @@
+A file installed by a function called by do_install
meta-selftest/recipes-test/recipetool/files/selftest-replaceme-inst-globfile new file +1 -0
new file mode 100644
index 0000000..996298b
@@ -0,0 +1 @@
+A file matched by a glob in do_install
meta-selftest/recipes-test/recipetool/files/selftest-replaceme-inst-todir-globfile new file +1 -0
new file mode 100644
index 0000000..585ae3e
@@ -0,0 +1 @@
+A file matched by a glob in do_install to a directory
meta-selftest/recipes-test/recipetool/files/selftest-replaceme-orig new file +1 -0
new file mode 100644
index 0000000..593d6a0
@@ -0,0 +1 @@
+Straight through with same nam
meta-selftest/recipes-test/recipetool/files/selftest-replaceme-src-globfile new file +1 -0
new file mode 100644
index 0000000..1e20a2b
@@ -0,0 +1 @@
+A file matched by a glob in SRC_URI
meta-selftest/recipes-test/recipetool/files/selftest-replaceme-todir new file +1 -0
new file mode 100644
index 0000000..85bd5eb
@@ -0,0 +1 @@
+File in SRC_URI installed just to directory path
meta-selftest/recipes-test/recipetool/files/subdir/fileinsubdir new file +1 -0
new file mode 100644
index 0000000..d516b49
@@ -0,0 +1 @@
+A file in a subdirectory
meta-selftest/recipes-test/recipetool/selftest-recipetool-appendfile.bb new file +42 -0

42 lines not on this page · raw message

meta/lib/oe/patch.py +63 -0

69 lines not on this page · raw message

meta/lib/oe/recipeutils.py +327 -1

349 lines not on this page · raw message

meta/lib/oeqa/selftest/devtool.py +30 -1
index f4571c4..ad10af5 100644
@@ -8,7 +8,7 @@ import glob
 
 import oeqa.utils.ftools as ftools
 from oeqa.selftest.base import oeSelfTest
-from oeqa.utils.commands import runCmd, bitbake, get_bb_var
+from oeqa.utils.commands import runCmd, bitbake, get_bb_var, create_temp_layer
 from oeqa.utils.decorators import testcase
 
 class DevtoolBase(oeSelfTest):
@@ -31,6 +31,35 @@ class DevtoolBase(oeSelfTest):
         for inherit in checkinherits:
             self.assertIn(inherit, inherits, 'Missing inherit of %s' % inherit)
 
+    def _check_bbappend(self, testrecipe, recipefile, appenddir):
+        result = runCmd('bitbake-layers show-appends', cwd=self.builddir)
+        resultlines = result.output.splitlines()
+        inrecipe = False
+        bbappends = []
+        bbappendfile = None
+        for line in resultlines:
+            if inrecipe:
+                if line.startswith(' '):
+                    bbappends.append(line.strip())
+                else:
+                    break
+            elif line == '%s:' % os.path.basename(recipefile):
+                inrecipe = True
+        self.assertLessEqual(len(bbappends), 2, '%s recipe is being bbappended by another layer - bbappends found:\n  %s' % (testrecipe, '\n  '.join(bbappends)))
+        for bbappend in bbappends:
+            if bbappend.startswith(appenddir):
+                bbappendfile = bbappend
+                break
+        else:
+            self.assertTrue(False, 'bbappend for recipe %s does not seem to be created in test layer' % testrecipe)
+        return bbappendfile
+
+    def _create_temp_layer(self, templayerdir, addlayer, templayername, priority=999, recipepathspec='recipes-*/*'):
+        create_temp_layer(templayerdir, templayername, priority, recipepathspec)
+        if addlayer:
+            self.add_command_to_tearDown('bitbake-layers remove-layer %s || true' % templayerdir)
+            result = runCmd('bitbake-layers add-layer %s' % templayerdir, cwd=self.builddir)
+
 
 class DevtoolTests(DevtoolBase):
 
meta/lib/oeqa/selftest/recipetool.py +311 -2

331 lines not on this page · raw message

meta/lib/oeqa/utils/commands.py +11 -0
index 663e4e7..dc8a983 100644
@@ -162,3 +162,14 @@ def get_test_layer():
             testlayer = l
             break
     return testlayer
+
+def create_temp_layer(templayerdir, templayername, priority=999, recipepathspec='recipes-*/*'):
+    os.makedirs(os.path.join(templayerdir, 'conf'))
+    with open(os.path.join(templayerdir, 'conf', 'layer.conf'), 'w') as f:
+        f.write('BBPATH .= ":${LAYERDIR}"\n')
+        f.write('BBFILES += "${LAYERDIR}/%s/*.bb \\' % recipepathspec)
+        f.write('            ${LAYERDIR}/%s/*.bbappend"\n' % recipepathspec)
+        f.write('BBFILE_COLLECTIONS += "%s"\n' % templayername)
+        f.write('BBFILE_PATTERN_%s = "^${LAYERDIR}/"\n' % templayername)
+        f.write('BBFILE_PRIORITY_%s = "%d"\n' % (templayername, priority))
+        f.write('BBFILE_PATTERN_IGNORE_EMPTY_%s = "1"\n' % templayername)
scripts/lib/recipetool/append.py new file +360 -0

360 lines not on this page · raw message

scripts/recipetool +3 -3
index b7d3ee8..c68bef4 100755
@@ -31,11 +31,11 @@ logger = scriptutils.logger_create('recipetool')
 
 plugins = []
 
-def tinfoil_init():
+def tinfoil_init(parserecipes):
     import bb.tinfoil
     import logging
     tinfoil = bb.tinfoil.Tinfoil()
-    tinfoil.prepare(True)
+    tinfoil.prepare(not parserecipes)
 
     for plugin in plugins:
         if hasattr(plugin, 'tinfoil_init'):
@@ -82,7 +82,7 @@ def main():
 
     scriptutils.logger_setup_color(logger, args.color)
 
-    tinfoil_init()
+    tinfoil_init(getattr(args, 'parserecipes', False))
 
     ret = args.func(args)
 
Signature · 1 line
2.1.0
#5 lore raw

PATCH 5/5 devtool: update-recipe: add option to write changes to bbappend

In reply to Paul Eggleton

Quite often what you want to do having made customisations to a piece of software is to apply those customisations in your own layer rather than in the original recipe. Thus, add a -a/--append option to the update-recipe subcommand which allows you to specify the layer to write a bbappend into. The bbappend will be created at the appropriate path within the specified layer directory (which may or may not be in your bblayers.conf) or if one already exists it will be updated appropriately.

(This re-uses code written for recipetool appendfile.)

Implements [YOCTO #7587].

---
 meta/lib/oeqa/selftest/devtool.py | 171 ++++++++++++++++++++++++++++++++++++++
 scripts/lib/devtool/standard.py   | 148 ++++++++++++++++++++++++---------
 2 files changed, 278 insertions(+), 41 deletions(-)
meta/lib/oeqa/selftest/devtool.py +171 -0
index ad10af5..4e22e1d 100644
@@ -524,6 +524,177 @@ class DevtoolTests(DevtoolBase):
                         break
                 self.assertTrue(matched, 'Unexpected diff remove line: %s' % line)
 
+    def test_devtool_update_recipe_append(self):
+        # Check preconditions
+        workspacedir = os.path.join(self.builddir, 'workspace')
+        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        testrecipe = 'mdadm'
+        recipefile = get_bb_var('FILE', testrecipe)
+        src_uri = get_bb_var('SRC_URI', testrecipe)
+        self.assertNotIn('git://', src_uri, 'This test expects the %s recipe to NOT be a git recipe' % testrecipe)
+        result = runCmd('git status . --porcelain', cwd=os.path.dirname(recipefile))
+        self.assertEqual(result.output.strip(), "", '%s recipe is not clean' % testrecipe)
+        # First, modify a recipe
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        tempsrcdir = os.path.join(tempdir, 'source')
+        templayerdir = os.path.join(tempdir, 'layer')
+        self.track_for_cleanup(tempdir)
+        self.track_for_cleanup(workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        # (don't bother with cleaning the recipe on teardown, we won't be building it)
+        result = runCmd('devtool modify %s -x %s' % (testrecipe, tempsrcdir))
+        # Check git repo
+        self.assertTrue(os.path.isdir(os.path.join(tempsrcdir, '.git')), 'git repository for external source tree not found')
+        result = runCmd('git status --porcelain', cwd=tempsrcdir)
+        self.assertEqual(result.output.strip(), "", 'Created git repo is not clean')
+        result = runCmd('git symbolic-ref HEAD', cwd=tempsrcdir)
+        self.assertEqual(result.output.strip(), "refs/heads/devtool", 'Wrong branch in git repo')
+        # Add a commit
+        result = runCmd("sed 's!\\(#define VERSION\\W*\"[^\"]*\\)\"!\\1-custom\"!' -i ReadMe.c", cwd=tempsrcdir)
+        result = runCmd('git commit -a -m "Add our custom version"', cwd=tempsrcdir)
+        self.add_command_to_tearDown('cd %s; rm -f %s/*.patch; git checkout .' % (os.path.dirname(recipefile), testrecipe))
+        # Create a temporary layer and add it to bblayers.conf
+        self._create_temp_layer(templayerdir, True, 'selftestupdaterecipe')
+        # Create the bbappend
+        result = runCmd('devtool update-recipe %s -a %s' % (testrecipe, templayerdir))
+        self.assertNotIn('WARNING:', result.output)
+        # Check recipe is still clean
+        result = runCmd('git status . --porcelain', cwd=os.path.dirname(recipefile))
+        self.assertEqual(result.output.strip(), "", '%s recipe is not clean' % testrecipe)
+        # Check bbappend was created
+        splitpath = os.path.dirname(recipefile).split(os.sep)
+        appenddir = os.path.join(templayerdir, splitpath[-2], splitpath[-1])
+        bbappendfile = self._check_bbappend(testrecipe, recipefile, appenddir)
+        patchfile = os.path.join(appenddir, testrecipe, '0001-Add-our-custom-version.patch')
+        self.assertTrue(os.path.exists(patchfile), 'Patch file not created')
+
+        # Check bbappend contents
+        expectedlines = ['FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"\n',
+                         '\n',
+                         'SRC_URI += "file://0001-Add-our-custom-version.patch"\n',
+                         '\n']
+        with open(bbappendfile, 'r') as f:
+            self.assertEqual(expectedlines, f.readlines())
+
+        # Check we can run it again and bbappend isn't modified
+        result = runCmd('devtool update-recipe %s -a %s' % (testrecipe, templayerdir))
+        with open(bbappendfile, 'r') as f:
+            self.assertEqual(expectedlines, f.readlines())
+        # Drop new commit and check patch gets deleted
+        result = runCmd('git reset HEAD^', cwd=tempsrcdir)
+        result = runCmd('devtool update-recipe %s -a %s' % (testrecipe, templayerdir))
+        self.assertFalse(os.path.exists(patchfile), 'Patch file not deleted')
+        expectedlines2 = ['FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"\n',
+                         '\n']
+        with open(bbappendfile, 'r') as f:
+            self.assertEqual(expectedlines2, f.readlines())
+        # Put commit back and check we can run it if layer isn't in bblayers.conf
+        os.remove(bbappendfile)
+        result = runCmd('git commit -a -m "Add our custom version"', cwd=tempsrcdir)
+        result = runCmd('bitbake-layers remove-layer %s' % templayerdir, cwd=self.builddir)
+        result = runCmd('devtool update-recipe %s -a %s' % (testrecipe, templayerdir))
+        self.assertIn('WARNING: Specified layer is not currently enabled in bblayers.conf', result.output)
+        self.assertTrue(os.path.exists(patchfile), 'Patch file not created (with disabled layer)')
+        with open(bbappendfile, 'r') as f:
+            self.assertEqual(expectedlines, f.readlines())
+        # Deleting isn't expected to work under these circumstances
+
+    def test_devtool_update_recipe_append_git(self):
+        # Check preconditions
+        workspacedir = os.path.join(self.builddir, 'workspace')
+        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        testrecipe = 'mtd-utils'
+        recipefile = get_bb_var('FILE', testrecipe)
+        src_uri = get_bb_var('SRC_URI', testrecipe)
+        self.assertIn('git://', src_uri, 'This test expects the %s recipe to be a git recipe' % testrecipe)
+        for entry in src_uri.split():
+            if entry.startswith('git://'):
+                git_uri = entry
+                break
+        result = runCmd('git status . --porcelain', cwd=os.path.dirname(recipefile))
+        self.assertEqual(result.output.strip(), "", '%s recipe is not clean' % testrecipe)
+        # First, modify a recipe
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        tempsrcdir = os.path.join(tempdir, 'source')
+        templayerdir = os.path.join(tempdir, 'layer')
+        self.track_for_cleanup(tempdir)
+        self.track_for_cleanup(workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        # (don't bother with cleaning the recipe on teardown, we won't be building it)
+        result = runCmd('devtool modify %s -x %s' % (testrecipe, tempsrcdir))
+        # Check git repo
+        self.assertTrue(os.path.isdir(os.path.join(tempsrcdir, '.git')), 'git repository for external source tree not found')
+        result = runCmd('git status --porcelain', cwd=tempsrcdir)
+        self.assertEqual(result.output.strip(), "", 'Created git repo is not clean')
+        result = runCmd('git symbolic-ref HEAD', cwd=tempsrcdir)
+        self.assertEqual(result.output.strip(), "refs/heads/devtool", 'Wrong branch in git repo')
+        # Add a commit
+        result = runCmd('echo "# Additional line" >> Makefile', cwd=tempsrcdir)
+        result = runCmd('git commit -a -m "Change the Makefile"', cwd=tempsrcdir)
+        self.add_command_to_tearDown('cd %s; rm -f %s/*.patch; git checkout .' % (os.path.dirname(recipefile), testrecipe))
+        # Create a temporary layer
+        os.makedirs(os.path.join(templayerdir, 'conf'))
+        with open(os.path.join(templayerdir, 'conf', 'layer.conf'), 'w') as f:
+            f.write('BBPATH .= ":${LAYERDIR}"\n')
+            f.write('BBFILES += "${LAYERDIR}/recipes-*/*/*.bbappend"\n')
+            f.write('BBFILE_COLLECTIONS += "oeselftesttemplayer"\n')
+            f.write('BBFILE_PATTERN_oeselftesttemplayer = "^${LAYERDIR}/"\n')
+            f.write('BBFILE_PRIORITY_oeselftesttemplayer = "999"\n')
+            f.write('BBFILE_PATTERN_IGNORE_EMPTY_oeselftesttemplayer = "1"\n')
+        self.add_command_to_tearDown('bitbake-layers remove-layer %s || true' % templayerdir)
+        result = runCmd('bitbake-layers add-layer %s' % templayerdir, cwd=self.builddir)
+        # Create the bbappend
+        result = runCmd('devtool update-recipe %s -a %s' % (testrecipe, templayerdir))
+        self.assertNotIn('WARNING:', result.output)
+        # Check recipe is still clean
+        result = runCmd('git status . --porcelain', cwd=os.path.dirname(recipefile))
+        self.assertEqual(result.output.strip(), "", '%s recipe is not clean' % testrecipe)
+        # Check bbappend was created
+        splitpath = os.path.dirname(recipefile).split(os.sep)
+        appenddir = os.path.join(templayerdir, splitpath[-2], splitpath[-1])
+        bbappendfile = self._check_bbappend(testrecipe, recipefile, appenddir)
+        self.assertFalse(os.path.exists(os.path.join(appenddir, testrecipe)), 'Patch directory should not be created')
+
+        # Check bbappend contents
+        result = runCmd('git rev-parse HEAD', cwd=tempsrcdir)
+        expectedlines = ['SRCREV = "%s"\n' % result.output,
+                         '\n',
+                         'SRC_URI = "%s"\n' % git_uri,
+                         '\n']
+        with open(bbappendfile, 'r') as f:
+            self.assertEqual(expectedlines, f.readlines())
+
+        # Check we can run it again and bbappend isn't modified
+        result = runCmd('devtool update-recipe %s -a %s' % (testrecipe, templayerdir))
+        with open(bbappendfile, 'r') as f:
+            self.assertEqual(expectedlines, f.readlines())
+        # Drop new commit and check SRCREV changes
+        result = runCmd('git reset HEAD^', cwd=tempsrcdir)
+        result = runCmd('devtool update-recipe %s -a %s' % (testrecipe, templayerdir))
+        self.assertFalse(os.path.exists(os.path.join(appenddir, testrecipe)), 'Patch directory should not be created')
+        result = runCmd('git rev-parse HEAD', cwd=tempsrcdir)
+        expectedlines = ['SRCREV = "%s"\n' % result.output,
+                         '\n',
+                         'SRC_URI = "%s"\n' % git_uri,
+                         '\n']
+        with open(bbappendfile, 'r') as f:
+            self.assertEqual(expectedlines, f.readlines())
+        # Put commit back and check we can run it if layer isn't in bblayers.conf
+        os.remove(bbappendfile)
+        result = runCmd('git commit -a -m "Change the Makefile"', cwd=tempsrcdir)
+        result = runCmd('bitbake-layers remove-layer %s' % templayerdir, cwd=self.builddir)
+        result = runCmd('devtool update-recipe %s -a %s' % (testrecipe, templayerdir))
+        self.assertIn('WARNING: Specified layer is not currently enabled in bblayers.conf', result.output)
+        self.assertFalse(os.path.exists(os.path.join(appenddir, testrecipe)), 'Patch directory should not be created')
+        result = runCmd('git rev-parse HEAD', cwd=tempsrcdir)
+        expectedlines = ['SRCREV = "%s"\n' % result.output,
+                         '\n',
+                         'SRC_URI = "%s"\n' % git_uri,
+                         '\n']
+        with open(bbappendfile, 'r') as f:
+            self.assertEqual(expectedlines, f.readlines())
+        # Deleting isn't expected to work under these circumstances
+
     def test_devtool_extract(self):
         # Check preconditions
         workspacedir = os.path.join(self.builddir, 'workspace')
scripts/lib/devtool/standard.py +107 -41

225 lines not on this page · raw message

Signature · 1 line
2.1.0
#6 lore raw
In reply to Paul Eggleton
Quoted 3 lines · On Monday 18 May 2015 16:15:03 Paul Eggleton wrote:

Add support for creating/updating bbappends in both devtool (within the update-recipe subcommand) and in recipetool (with a new appendfile subcommand), as well as some related tidy-ups.

I missed mentioning - this series *requires* the set of changes for lib/bb/utils.py that I just posted to the bitbake mailing list.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre