Skip to contentSkip to conversation

devtool improvements

PATCH In review Paul Eggleton 3 participants 16 messages Dec 5, 2017 – Dec 13, 2017

lore
Share & tools
lore
#0 lore raw

cover devtool improvements

A set of improvements for devtool, mostly aimed at improving the handling of recipes that fetch multiple source trees (i.e. have multiple tarballs or repositories in SRC_URI), with a couple of other fixes thrown in.

The following changes since commit a7cd9d1183be603777fc9c8c448281fe01224f7b:

lib/oe/utils: remove param_bool() (2017-12-02 11:24:38 +0000)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib paule/devtool32-oe
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=paule/devtool32-oe

Paul Eggleton (12):
devtool: refactor return for _extract_source()
devtool: reset: delete bbappend file if _check_preserve() doesn't
devtool: finish: fix erroneously creating bbappend for relative paths
devtool: deploy-target: don't specify ssh/scp port unless user does
devtool: extract: fix usage with kern-tools-native
devtool: modify: pick up commits from previously created source tree lib/oe/recipeutils: allow patch_recipe_file() to be re-called
devtool: support extracting multiple source trees
devtool: fix handling of linux-yocto after multisrc changes
oe-selftest: devtool: fix kernel test for multisrc changes
oe-selftest: devtool: add test for multiple source trees lib/oe/recipeutils: add .txz extension to archive list

 .../devtool/devtool-test-multisrc-subdir_1.0.bb    |  13 +
 .../devtool-test-multisrc/example-files.tar.gz     | Bin 0 -> 236 bytes
 .../devtool/devtool-test-multisrc/example.patch    |   7 +
 .../devtool-test-multisrc/mypackage-1.0.tar.gz     | Bin 0 -> 903 bytes
 .../devtool/devtool-test-multisrc_1.0.bb           |  11 +
 meta/classes/devtool-source.bbclass                |  63 ++-
 meta/classes/externalsrc.bbclass                   |   2 +-
 meta/classes/kernel-yocto.bbclass                  |   7 +-
 meta/lib/oe/recipeutils.py                         |  15 +-
 meta/lib/oe/utils.py                               |  24 ++
 meta/lib/oeqa/selftest/cases/devtool.py            | 124 +++++-
 scripts/devtool                                    |  61 ++-
 scripts/lib/devtool/__init__.py                    |  14 +
 scripts/lib/devtool/deploy.py                      |  12 +-
 scripts/lib/devtool/standard.py                    | 426 ++++++++++++++-------
 scripts/lib/devtool/upgrade.py                     |   4 +-
 16 files changed, 601 insertions(+), 182 deletions(-)
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc-subdir_1.0.bb
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc/example-files.tar.gz
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc/example.patch
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc/mypackage-1.0.tar.gz
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc_1.0.bb
Signature · 1 line
2.9.5
#1 lore raw

PATCH 1/12 devtool: refactor return for _extract_source()

Use a namedtuple to return information to the caller, since I've been expanding that information we should avoid having to change all of the calling code each time.

Additionally, it turned out that a bunch of the callers were checking for None being returned in the initial_rev value, but that's no longer possible, so tidy up the calling code.

---
 scripts/lib/devtool/standard.py | 29 ++++++++++-------------------
 scripts/lib/devtool/upgrade.py  |  4 ++--
 2 files changed, 12 insertions(+), 21 deletions(-)
scripts/lib/devtool/standard.py +10 -19
index 26187a0..b1669b5 100644
@@ -29,7 +29,7 @@ import scriptutils
 import errno
 import glob
 import filecmp
-from collections import OrderedDict
+from collections import OrderedDict, namedtuple
 from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, setup_git_repo, recipe_to_append, get_bbclassextend_targets, update_unlockedsigs, check_prerelease_version, check_git_repo_dirty, check_git_repo_op, DevtoolError
 from devtool import parse_recipe
 
@@ -437,15 +437,11 @@ def extract(args, config, basepath, workspace):
             return 1
 
         srctree = os.path.abspath(args.srctree)
-        initial_rev, _ = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides)
+        _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides)
         logger.info('Source tree extracted to %s' % srctree)
-
-        if initial_rev:
-            return 0
-        else:
-            return 1
     finally:
         tinfoil.shutdown()
+    return 0
 
 def sync(args, config, basepath, workspace):
     """Entry point for the devtool 'sync' subcommand"""
@@ -461,17 +457,15 @@ def sync(args, config, basepath, workspace):
             return 1
 
         srctree = os.path.abspath(args.srctree)
-        initial_rev, _ = _extract_source(srctree, args.keep_temp, args.branch, True, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=True)
+        _extract_source(srctree, args.keep_temp, args.branch, True, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=True)
         logger.info('Source tree %s synchronized' % srctree)
-
-        if initial_rev:
-            return 0
-        else:
-            return 1
     finally:
         tinfoil.shutdown()
+    return 0
 
 
+ExtractSourceResult = namedtuple('ExtractSourceResult', ['initial_rev', 'srcsubdir'])
+
 def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, workspace, fixed_setup, d, tinfoil, no_overrides=False):
     """Extract sources of a recipe"""
     import oe.recipeutils
@@ -512,8 +506,6 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
         if extra_overrides:
             logger.info('SRC_URI contains some conditional appends/prepends - will create branches to represent these')
 
-    initial_rev = None
-
     appendexisted = False
     recipefile = d.getVar('FILE')
     appendfile = recipe_to_append(recipefile, config)
@@ -659,7 +651,7 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
             logger.info('Preserving temporary directory %s' % tempdir)
         else:
             shutil.rmtree(tempdir)
-    return initial_rev, srcsubdir_rel
+    return ExtractSourceResult(initial_rev, srcsubdir_rel)
 
 def _add_md5(config, recipename, filename):
     """Record checksum of a file (or recursively for a directory) to the md5-file of the workspace"""
@@ -759,9 +751,8 @@ def modify(args, config, basepath, workspace):
         commits = []
         check_commits = False
         if not args.no_extract:
-            initial_rev, _ = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides)
-            if not initial_rev:
-                return 1
+            ret = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides)
+            initial_rev = ret.initial_rev
             logger.info('Source tree extracted to %s' % srctree)
             # Get list of commits since this revision
             (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_rev, cwd=srctree)
scripts/lib/devtool/upgrade.py +2 -2
index f6141bf..7afdd80 100644
@@ -474,12 +474,12 @@ def upgrade(args, config, basepath, workspace):
         rf = None
         try:
             logger.info('Extracting current version source...')
-            rev1, srcsubdir1 = standard._extract_source(srctree, False, 'devtool-orig', False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides)
+            current_ret = standard._extract_source(srctree, False, 'devtool-orig', False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides)
             logger.info('Extracting upgraded version source...')
             rev2, md5, sha256, srcbranch, srcsubdir2 = _extract_new_source(args.version, srctree, args.no_patch,
                                                     args.srcrev, args.srcbranch, args.branch, args.keep_temp,
                                                     tinfoil, rd)
-            rf, copied = _create_new_recipe(args.version, md5, sha256, args.srcrev, srcbranch, srcsubdir1, srcsubdir2, config.workspace_path, tinfoil, rd)
+            rf, copied = _create_new_recipe(args.version, md5, sha256, args.srcrev, srcbranch, current_ret.srcsubdir, srcsubdir2, config.workspace_path, tinfoil, rd)
         except bb.process.CmdError as e:
             _upgrade_error(e, rf, srctree)
         except DevtoolError as e:
Signature · 1 line
2.9.5
#2 lore raw

PATCH 2/12 devtool: reset: delete bbappend file if _check_preserve() doesn't

In reply to Paul Eggleton

If the .devtool_md5 file doesn't contain a reference to the bbappend file (e.g. because devtool was interrupted before it could write that out) then _check_preserve() won't delete it, so we need to delete it separately because otherwise the recipe won't actually be reset.

---
 scripts/lib/devtool/standard.py | 7 +++++++
 1 file changed, 7 insertions(+)
scripts/lib/devtool/standard.py +7 -0
index b1669b5..f59c6d6 100644
@@ -1781,6 +1781,13 @@ def _reset(recipes, no_clean, config, basepath, workspace):
     for pn in recipes:
         _check_preserve(config, pn)
 
+        appendfile = workspace[pn]['bbappend']
+        if os.path.exists(appendfile):
+            # This shouldn't happen, but is possible if devtool errored out prior to
+            # writing the md5 file. We need to delete this here or the recipe won't
+            # actually be reset
+            os.remove(appendfile)
+
         preservepath = os.path.join(config.workspace_path, 'attic', pn, pn)
         def preservedir(origdir):
             if os.path.exists(origdir):
Signature · 1 line
2.9.5
#3 lore raw

PATCH 3/12 devtool: finish: fix erroneously creating bbappend for relative paths

In reply to Paul Eggleton

After OE-Core rev 5e3fe00a0233d563781849a44f53885b4e924a9c we call os.path.abspath() on the original layer path, but we later compare that to the destination layer path. If that layer path isn't absolute but is effectively the same path, it should be writing to the original recipe but because we weren't making it absolute we were writing a bbappend instead. Call os.path.abspath() on the destination path as well to avoid that.

---
 scripts/lib/devtool/standard.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
scripts/lib/devtool/standard.py +1 -3
index f59c6d6..cdd2346 100644
@@ -1848,9 +1848,7 @@ def _get_layer(layername, d):
         layerdir = layers.get('meta', None)
     else:
         layerdir = layers.get(layername, None)
-    if layerdir:
-        layerdir = os.path.abspath(layerdir)
-    return layerdir or layername
+    return os.path.abspath(layerdir or layername)
 
 def finish(args, config, basepath, workspace):
     """Entry point for the devtool 'finish' subcommand"""
Signature · 1 line
2.9.5
#4 lore raw

PATCH 4/12 devtool: deploy-target: don't specify ssh/scp port unless user does

In reply to Paul Eggleton

If the user doesn't specify a port then we should avoid specifying one on the ssh/scp command line in case the user has configured one for the host they are connecting to, which was being overridden unnecessarily.

Fixes [YOCTO #12381].

---
 scripts/lib/devtool/deploy.py | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)
scripts/lib/devtool/deploy.py +4 -8
index 9cc4927..52e261d 100644
@@ -213,9 +213,7 @@ def deploy(args, config, basepath, workspace):
 
         scp_port = ''
         ssh_port = ''
-        if not args.port:
-            raise DevtoolError("If you specify -P/--port then you must provide the port to be used to connect to the target")
-        else:
+        if args.port:
             scp_port = "-P %s" % args.port
             ssh_port = "-p %s" % args.port
 
@@ -280,9 +278,7 @@ def undeploy(args, config, basepath, workspace):
 
     scp_port = ''
     ssh_port = ''
-    if not args.port:
-        raise DevtoolError("If you specify -P/--port then you must provide the port to be used to connect to the target")
-    else:
+    if args.port:
         scp_port = "-P %s" % args.port
         ssh_port = "-p %s" % args.port
 
@@ -328,7 +324,7 @@ def register_commands(subparsers, context):
     parser_deploy.add_argument('-n', '--dry-run', help='List files to be deployed only', action='store_true')
     parser_deploy.add_argument('-p', '--no-preserve', help='Do not preserve existing files', action='store_true')
     parser_deploy.add_argument('--no-check-space', help='Do not check for available space before deploying', action='store_true')
-    parser_deploy.add_argument('-P', '--port', default='22', help='Port to use for connection to the target')
+    parser_deploy.add_argument('-P', '--port', help='Specify port to use for connection to the target')
 
     strip_opts = parser_deploy.add_mutually_exclusive_group(required=False)
     strip_opts.add_argument('-S', '--strip',
@@ -350,5 +346,5 @@ def register_commands(subparsers, context):
     parser_undeploy.add_argument('-s', '--show-status', help='Show progress/status output', action='store_true')
     parser_undeploy.add_argument('-a', '--all', help='Undeploy all recipes deployed on the target', action='store_true')
     parser_undeploy.add_argument('-n', '--dry-run', help='List files to be undeployed only', action='store_true')
-    parser_undeploy.add_argument('-P', '--port', default='22', help='Port to use for connection to the target')
+    parser_undeploy.add_argument('-P', '--port', help='Specify port to use for connection to the target')
     parser_undeploy.set_defaults(func=undeploy)
Signature · 1 line
2.9.5
#5 lore raw

PATCH 5/12 devtool: extract: fix usage with kern-tools-native

In reply to Paul Eggleton

The kern-tools-native recipe as it currently stands is unusual in that it fetches source from a repository but sets S = "${WORKDIR}" which causes some problems. First you get a failure because we're calling "git commit" unconditionally even if there are no local files, and there aren't any in this case which means the commit fails.

After that's fixed, we hit another problem where "recipe-sysroot-native" subdirectory appears in the extracted source tree. We don't want that so exclude it from copying.

---
 meta/classes/devtool-source.bbclass | 6 ++++++
 scripts/lib/devtool/standard.py     | 6 +++---
 2 files changed, 9 insertions(+), 3 deletions(-)
meta/classes/devtool-source.bbclass +6 -0
index 56882a4..540a602 100644
@@ -118,7 +118,13 @@ python devtool_post_unpack() {
                         os.path.basename(fname) not in recipe_patches]
         srcsubdir = d.getVar('DEVTOOL_PATCH_SRCDIR')
         # Move source files to S
+        excludevars = ['RECIPE_SYSROOT', 'RECIPE_SYSROOT_NATIVE']
+        excludepaths = []
+        for excludevar in excludevars:
+            excludepaths.append(os.path.relpath(d.getVar(excludevar), workdir) + os.sep)
         for path in src_files:
+            if path.startswith(tuple(excludepaths)):
+                continue
             _move_file(os.path.join(workdir, path),
                         os.path.join(srcsubdir, path))
     elif os.path.dirname(srcsubdir) != workdir:
scripts/lib/devtool/standard.py +3 -3
index cdd2346..a6656e4 100644
@@ -634,9 +634,9 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
                         addfiles.append(os.path.join(relpth, fn))
                 if addfiles:
                     bb.process.run('git add %s' % ' '.join(addfiles), cwd=srctree)
-                useroptions = []
-                oe.patch.GitApplyTree.gitCommandUserOptions(useroptions, d=d)
-                bb.process.run('git %s commit -a -m "Committing local file symlinks\n\n%s"' % (' '.join(useroptions), oe.patch.GitApplyTree.ignore_commit_prefix), cwd=srctree)
+                    useroptions = []
+                    oe.patch.GitApplyTree.gitCommandUserOptions(useroptions, d=d)
+                    bb.process.run('git %s commit -a -m "Committing local file symlinks\n\n%s"' % (' '.join(useroptions), oe.patch.GitApplyTree.ignore_commit_prefix), cwd=srctree)
 
         if is_kernel_yocto:
             logger.info('Copying kernel config to srctree')
Signature · 1 line
2.9.5
#6 lore raw

PATCH 6/12 devtool: modify: pick up commits from previously created source tree

In reply to Paul Eggleton

If you use devtool modify, then devtool reset, keep the source tree and then devtool modify on the same recipe with the -n option to re-use the existing source tree, we should pick up the commit hashes properly from the source tree so that later on devtool finish has these to compare to the commits in the tree at that time. We also need to be careful the second time around that we only get the original commits rather than the current HEAD which may be the result of user changes (hence using "devtool-patched", the tag that was placed at the original HEAD).

---
 scripts/lib/devtool/standard.py | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
scripts/lib/devtool/standard.py +7 -3
index a6656e4..ae48406 100644
@@ -750,13 +750,11 @@ def modify(args, config, basepath, workspace):
         initial_rev = None
         commits = []
         check_commits = False
+        torev = 'HEAD'
         if not args.no_extract:
             ret = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides)
             initial_rev = ret.initial_rev
             logger.info('Source tree extracted to %s' % srctree)
-            # Get list of commits since this revision
-            (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_rev, cwd=srctree)
-            commits = stdout.split()
             check_commits = True
         else:
             if os.path.exists(os.path.join(srctree, '.git')):
@@ -767,6 +765,7 @@ def modify(args, config, basepath, workspace):
                     stdout = ''
                 if stdout:
                     check_commits = True
+                    torev = 'devtool-patched'
                 for line in stdout.splitlines():
                     if line.startswith('*'):
                         (stdout, _) = bb.process.run('git rev-parse devtool-base', cwd=srctree)
@@ -776,6 +775,11 @@ def modify(args, config, basepath, workspace):
                     (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srctree)
                     initial_rev = stdout.rstrip()
 
+        if initial_rev:
+            # Get list of commits since this revision
+            (stdout, _) = bb.process.run('git rev-list --reverse %s..%s' % (initial_rev, torev), cwd=srctree)
+            commits = stdout.split()
+
         branch_patches = {}
         if check_commits:
             # Check if there are override branches
Signature · 1 line
2.9.5
#7 lore raw

PATCH 7/12 lib/oe/recipeutils: allow patch_recipe_file() to be re-called

In reply to Paul Eggleton

If patch_recipe_file() is called with output redirection on the same file twice in succession, then we don't want to wipe out the changes the first call made so we need to be reading in the redirected file if it exists instead of the original one.

This is important to enable devtool finish to work with multiple source trees within the same recipe.

---
 meta/lib/oe/recipeutils.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
meta/lib/oe/recipeutils.py +7 -2
index 4e0859e..4928727 100644
@@ -258,13 +258,18 @@ def patch_recipe_file(fn, values, patch=False, relpath='', redirect_output=None)
        since this cannot handle all situations.
     """
 
-    with open(fn, 'r') as f:
+    read_fn = fn
+    if redirect_output:
+        redirect_fn = os.path.join(redirect_output, os.path.basename(fn))
+        if os.path.exists(redirect_fn):
+            read_fn = redirect_fn
+    with open(read_fn, 'r') as f:
         fromlines = f.readlines()
 
     _, tolines = patch_recipe_lines(fromlines, values)
 
     if redirect_output:
-        with open(os.path.join(redirect_output, os.path.basename(fn)), 'w') as f:
+        with open(redirect_fn, 'w') as f:
             f.writelines(tolines)
         return None
     elif patch:
Signature · 1 line
2.9.5
#8 lore raw

PATCH 8/12 devtool: support extracting multiple source trees

In reply to Paul Eggleton

If you have multiple source trees being extracted to the work directory within a recipe (e.g. you have two tarballs referred to in SRC_URI) and one isn't being extracted into the other, then devtool failed to extract all the sources because it only took the source tree that S pointed into. To fix this, we need to take a look at the work directory after do_unpack and see if there are any additional subdirectories; if so we need to put the main source tree in a subdirectory and put the additional subdirectories next to it. We also ensure that symlinks from the work directory get created at the end of do_unpack to point to these (so that references in the recipe continue to work).

In addition to multiple source trees at the work directory level, this patch also handles multiple nested git trees (where you have multiple git URLs in SRC_URI with one or more inside another). These caused a different problem, where changes in sub-repos are not fully captured at the top level - we need to handle each repo separately.

---
 meta/classes/devtool-source.bbclass     |  57 ++++-
 meta/lib/oe/recipeutils.py              |   4 +-
 meta/lib/oe/utils.py                    |  24 +++
 meta/lib/oeqa/selftest/cases/devtool.py |   2 +-
 scripts/devtool                         |  61 ++++--
 scripts/lib/devtool/__init__.py         |  14 ++
 scripts/lib/devtool/standard.py         | 370 +++++++++++++++++++++++---------
 7 files changed, 396 insertions(+), 136 deletions(-)
meta/classes/devtool-source.bbclass +47 -10
index 540a602..30c9bdb 100644
@@ -69,7 +69,7 @@ python devtool_post_unpack() {
     import shutil
     sys.path.insert(0, os.path.join(d.getVar('COREBASE'), 'scripts', 'lib'))
     import scriptutils
-    from devtool import setup_git_repo
+    from devtool import setup_git_repo, find_git_repos
 
     tempdir = d.getVar('DEVTOOL_TEMPDIR')
     workdir = d.getVar('WORKDIR')
@@ -95,8 +95,26 @@ python devtool_post_unpack() {
                         oe.recipeutils.get_recipe_patches(d)]
     local_files = oe.recipeutils.get_recipe_local_files(d)
 
-    # Ignore local files with subdir={BP}
+    excludeitems = recipe_patches + list(local_files.keys())
+    pthvars = ['RECIPE_SYSROOT', 'RECIPE_SYSROOT_NATIVE', 'S']
+    for pthvar in pthvars:
+        relp = os.path.relpath(d.getVar(pthvar), d.getVar('WORKDIR'))
+        if not relp.startswith('..'):
+            excludeitems.append(relp.split(os.sep)[0])
+    extradirs = []
     srcabspath = os.path.abspath(srcsubdir)
+    if srcabspath != os.path.abspath(workdir):
+        for pth in os.listdir(workdir):
+            if pth in excludeitems:
+                continue
+            wpth = os.path.join(workdir, pth)
+            if os.path.isdir(wpth) and os.listdir(wpth):
+                extradirs.append(wpth)
+
+    repos = find_git_repos(srcabspath)
+    extradirs.extend(repos)
+
+    # Ignore local files with subdir={BP}
     local_files = [fname for fname in local_files if
                     os.path.exists(os.path.join(workdir, fname)) and
                     (srcabspath == workdir or not
@@ -145,11 +163,20 @@ python devtool_post_unpack() {
 
     (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srcsubdir)
     initial_rev = stdout.rstrip()
+
+    initial_revs = {}
+    for extradir in extradirs:
+        setup_git_repo(extradir, d.getVar('PV'), devbranch, d=d)
+        (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=extradir)
+        initial_revs[extradir] = stdout.rstrip()
+
     with open(os.path.join(tempdir, 'initial_rev'), 'w') as f:
         f.write(initial_rev)
 
     with open(os.path.join(tempdir, 'srcsubdir'), 'w') as f:
-        f.write(srcsubdir)
+        f.write('%s\n' % srcsubdir)
+        for extradir in extradirs:
+            f.write('%s=%s\n' % (extradir, initial_revs[extradir]))
 }
 
 python devtool_pre_patch() {
@@ -160,18 +187,25 @@ python devtool_pre_patch() {
 python devtool_post_patch() {
     import shutil
     tempdir = d.getVar('DEVTOOL_TEMPDIR')
+
+    srcdirs = []
     with open(os.path.join(tempdir, 'srcsubdir'), 'r') as f:
-        srcsubdir = f.read()
+        for line in f:
+            line = line.rstrip()
+            if line:
+                srcdirs.append(line.split('=')[0])
+    srcsubdir = srcdirs[0]
+
     with open(os.path.join(tempdir, 'initial_rev'), 'r') as f:
         initial_rev = f.read()
 
-    def rm_patches():
-        patches_dir = os.path.join(srcsubdir, 'patches')
+    def rm_patches(pth):
+        patches_dir = os.path.join(pth, 'patches')
         if os.path.exists(patches_dir):
             shutil.rmtree(patches_dir)
         # Restore any "patches" directory that was actually part of the source tree
         try:
-            bb.process.run('git checkout -- patches', cwd=srcsubdir)
+            bb.process.run('git checkout -- patches', cwd=pth)
         except bb.process.ExecutionError:
             pass
 
@@ -194,7 +228,7 @@ python devtool_post_patch() {
             localdata = bb.data.createCopy(d)
             localdata.setVar('OVERRIDES', ':'.join(no_overrides))
             bb.build.exec_func('do_patch', localdata)
-            rm_patches()
+            rm_patches(srcsubdir)
             # Now we need to reconcile the dev branch with the no-overrides one
             # (otherwise we'd likely be left with identical commits that have different hashes)
             bb.process.run('git checkout %s' % devbranch, cwd=srcsubdir)
@@ -212,12 +246,15 @@ python devtool_post_patch() {
                 # Run do_patch function with the override applied
                 localdata.appendVar('OVERRIDES', ':%s' % override)
                 bb.build.exec_func('do_patch', localdata)
-                rm_patches()
+                rm_patches(srcsubdir)
                 # Now we need to reconcile the new branch with the no-overrides one
                 # (otherwise we'd likely be left with identical commits that have different hashes)
                 bb.process.run('git rebase devtool-no-overrides', cwd=srcsubdir)
         bb.process.run('git checkout %s' % devbranch, cwd=srcsubdir)
-    bb.process.run('git tag -f devtool-patched', cwd=srcsubdir)
+    for srcdir in srcdirs:
+        bb.process.run('git tag -f devtool-patched', cwd=srcdir)
+        if srcdir != srcsubdir:
+            rm_patches(srcdir)
 }
 
 python devtool_post_configure() {
meta/lib/oe/recipeutils.py +2 -2
index 4928727..f4a40c2 100644
@@ -445,10 +445,10 @@ def get_recipe_patches(d):
     """Get a list of the patches included in SRC_URI within a recipe."""
     import oe.patch
     patches = oe.patch.src_patches(d, expand=False)
-    patchfiles = []
+    patchfiles = OrderedDict()
     for patch in patches:
         _, _, local, _, _, parm = bb.fetch.decodeurl(patch)
-        patchfiles.append(local)
+        patchfiles[local] = parm
     return patchfiles
 
 
meta/lib/oe/utils.py +24 -0
index 1897c5f..9670dd7 100644
@@ -368,3 +368,27 @@ class ImageQAFailed(bb.build.FuncFailed):
             msg = msg + ' (%s)' % self.description
 
         return msg
+
+def is_path_under(path1, possible_parent):
+    """
+    Return True if a path is underneath another, False otherwise.
+    Multiple paths to test can be specified (as a list or tuple) in
+    which case all specified test paths must be under the parent to
+    return True.
+    """
+    def abs_path_trailing(pth):
+        pth_abs = os.path.abspath(pth)
+        if not pth_abs.endswith(os.sep):
+            pth_abs += os.sep
+        return pth_abs
+
+    possible_parent_abs = abs_path_trailing(possible_parent)
+    if isinstance(path1, str):
+        testpaths = [path1]
+    else:
+        testpaths = path1
+    for path in testpaths:
+        path_abs = abs_path_trailing(path)
+        if not path_abs.startswith(possible_parent_abs):
+            return False
+    return True
meta/lib/oeqa/selftest/cases/devtool.py +1 -1
index 43280cd..b3063de 100644
@@ -7,7 +7,7 @@ import fnmatch
 
 import oeqa.utils.ftools as ftools
 from oeqa.selftest.case import OESelftestTestCase
-from oeqa.utils.commands import runCmd, bitbake, get_bb_var, create_temp_layer
+from oeqa.utils.commands import runCmd, bitbake, get_bb_var, create_temp_layer, CommandError
 from oeqa.utils.commands import get_bb_vars, runqemu, get_test_layer
 from oeqa.core.decorator.oeid import OETestID
 
scripts/devtool +45 -16

82 lines not on this page · raw message

scripts/lib/devtool/__init__.py +14 -0
index 07d774d..5cdf07a 100644
@@ -223,6 +223,20 @@ def setup_git_repo(repodir, version, devbranch, basetag='devtool-base', d=None):
     bb.process.run('git checkout -b %s' % devbranch, cwd=repodir)
     bb.process.run('git tag -f %s' % basetag, cwd=repodir)
 
+def find_git_repos(pth, toplevel=False):
+    """
+    Find git repositories under a path
+    """
+    repos = []
+    if toplevel and os.path.isdir(os.path.join(pth, '.git')):
+        repos.append(pth)
+    for root, dirs, _ in os.walk(pth):
+        for dfn in dirs:
+            dfp = os.path.join(root, dfn)
+            if os.path.isdir(os.path.join(dfp, '.git')) and dfp not in repos:
+                repos.append(dfp)
+    return repos
+
 def recipe_to_append(recipefile, config, wildcard=False):
     """
     Convert a recipe file to a bbappend file path within the workspace.
scripts/lib/devtool/standard.py +263 -107

578 lines not on this page · raw message

Signature · 1 line
2.9.5
#9 lore raw

PATCH 9/12 devtool: fix handling of linux-yocto after multisrc changes

In reply to Paul Eggleton

devtool now handles multiple source trees for any recipe that includes them in SRC_URI, extracting them all side-by-side so that you can make changes in any of them. As a result, when running devtool modify on a linux-yocto kernel recipe under the source path you will get a "source" subdirectory containing the kernel source and a "kernel-meta" subdirectory next to it containing the kernel metadata. (Previously you just got the source tree and the kernel metadata remained in the work directory). We create a symlink automatically at do_unpack from the work directory so that it can still be found there, however kernel_feature_dirs() expects to find the kernel-meta repository and we also now need to make externalsrc remove that so that it doesn't unpack and overwrite the one we've already extracted. Change kernel_feature_dirs() so that if there are no kmeta entries in SRC_URI, it will fall back to a directory named ${KMETA} if it happens to be present in the work directory, ignoring how it got there.

---
 meta/classes/externalsrc.bbclass  | 2 +-
 meta/classes/kernel-yocto.bbclass | 7 ++++++-
 scripts/lib/devtool/standard.py   | 4 +++-
 3 files changed, 10 insertions(+), 3 deletions(-)
meta/classes/externalsrc.bbclass +1 -1
index 65dd13d..1ce53ee 100644
@@ -65,7 +65,7 @@ python () {
             url_data = fetch.ud[url]
             parm = url_data.parm
             if (url_data.type == 'file' or
-                    'type' in parm and parm['type'] == 'kmeta'):
+                    (parm.get('type', None) == 'kmeta' and d.getVar('EXTERNALSRC_KMETA') != "1")):
                 local_srcuri.append(url)
 
         d.setVar('SRC_URI', ' '.join(local_srcuri))
meta/classes/kernel-yocto.bbclass +6 -1
index 1d44795..18a7113 100644
@@ -42,7 +42,12 @@ def find_kernel_feature_dirs(d):
             destdir = parm["destsuffix"]
             if type == "kmeta":
                 feature_dirs.append(destdir)
-	    
+    if not feature_dirs:
+        # If the kernel-meta directory already exists (e.g from externalsrc)
+        # with EXTERNALSRC_KMETA = "1" then use it
+        kmetadir = d.getVar('KMETA')
+        if os.path.isdir(os.path.join(d.getVar('WORKDIR'), kmetadir)):
+            feature_dirs.append(kmetadir)
     return feature_dirs
 
 # find the master/machine source branch. In the same way that the fetcher proceses
scripts/lib/devtool/standard.py +3 -1
index cadd038..fe33c7e 100644
@@ -909,7 +909,9 @@ def modify(args, config, basepath, workspace):
 
             if bb.data.inherits_class('kernel', rd):
                 f.write('SRCTREECOVEREDTASKS = "do_validate_branches do_kernel_checkout '
-                        'do_fetch do_unpack do_kernel_configme do_kernel_configcheck"\n')
+                        'do_fetch do_kernel_configme do_kernel_configcheck"\n')
+                if bb.data.inherits_class('kernel-yocto', rd):
+                    f.write('EXTERNALSRC_KMETA = "1"\n')
                 f.write('\ndo_patch() {\n'
                         '    :\n'
                         '}\n')
Signature · 1 line
2.9.5
#10 lore raw

PATCH 10/12 oe-selftest: devtool: fix kernel test for multisrc changes

In reply to Paul Eggleton

Fix the oe-selftest test to understand that the source will be found in a "source" subdirectory if the kernel inherits linux-yocto since there are multiple trees extracted.

---
 meta/lib/oeqa/selftest/cases/devtool.py | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
meta/lib/oeqa/selftest/cases/devtool.py +9 -4
index b3063de..96072be 100644
@@ -1676,9 +1676,14 @@ class DevtoolTests(DevtoolBase):
         bitbake('%s -c clean' % kernel_provider)
         #Step 4.1
         runCmd('devtool modify virtual/kernel -x %s' % tempdir)
-        self.assertExists(os.path.join(tempdir, 'Makefile'), 'Extracted source could not be found')
+        # If it's linux-yocto, we'll have two "source" trees (one the actual source,
+        # the other the kernel metadata)
+        tempsrc = os.path.join(tempdir, 'source')
+        if not os.path.isdir(tempsrc):
+            tempsrc = tempdir
+        self.assertExists(os.path.join(tempsrc, 'Makefile'), 'Extracted source could not be found')
         #Step 4.2
-        configfile = os.path.join(tempdir,'.config')
+        configfile = os.path.join(tempsrc, '.config')
         diff = runCmd('diff %s %s' % (tmpconfig, configfile))
         self.assertEqual(0,diff.status,'Kernel .config file is not the same using bitbake and devtool')
         #Step 4.3
@@ -1689,12 +1694,12 @@ class DevtoolTests(DevtoolBase):
         self.assertExists(kernelfile, 'Kernel was not build correctly')
 
         #Modify the kernel source
-        modfile = os.path.join(tempdir,'arch/x86/boot/header.S')
+        modfile = os.path.join(tempsrc, 'arch/x86/boot/header.S')
         modstring = "Use a boot loader. Devtool testing."
         modapplied = runCmd("sed -i 's/Use a boot loader./%s/' %s" % (modstring, modfile))
         self.assertEqual(0,modapplied.status,'Modification to %s on kernel source failed' % modfile)
         #Modify the configuration
-        codeconfigfile = os.path.join(tempdir,'.config.new')
+        codeconfigfile = os.path.join(tempsrc, '.config.new')
         modconfopt = "CONFIG_SG_POOL=n"
         modconf = runCmd("sed -i 's/CONFIG_SG_POOL=y/%s/' %s" % (modconfopt, codeconfigfile))
         self.assertEqual(0,modconf.status,'Modification to %s failed' % codeconfigfile)
Signature · 1 line
2.9.5
#11 lore raw

PATCH 11/12 oe-selftest: devtool: add test for multiple source trees

In reply to Paul Eggleton

Add two synthetic tests for devtool modify + devtool finish: first with multiple source trees side-by-side, and second to test with one as a subdirectory of the main source tree. These also test devtool finish's recently added dry-run option and that detects and errors on uncommitted changes without being forced.

---
 .../devtool/devtool-test-multisrc-subdir_1.0.bb    |  13 +++
 .../devtool-test-multisrc/example-files.tar.gz     | Bin 0 -> 236 bytes
 .../devtool/devtool-test-multisrc/example.patch    |   7 ++
 .../devtool-test-multisrc/mypackage-1.0.tar.gz     | Bin 0 -> 903 bytes
 .../devtool/devtool-test-multisrc_1.0.bb           |  11 +++
 meta/lib/oeqa/selftest/cases/devtool.py            | 109 +++++++++++++++++++++
 6 files changed, 140 insertions(+)
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc-subdir_1.0.bb
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc/example-files.tar.gz
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc/example.patch
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc/mypackage-1.0.tar.gz
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc_1.0.bb
meta-selftest/recipes-test/devtool/devtool-test-multisrc-subdir_1.0.bb new file +13 -0
new file mode 100644
index 0000000..0430382
@@ -0,0 +1,13 @@
+SUMMARY = "Recipe to test multiple source trees"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://settings-daemon.c;beginline=1;endline=26;md5=8d77ba1c7a55df48d8d06c5f3d86b49d"
+
+SRC_URI = "git://git.yoctoproject.org/xsettings-daemon;name=xsettings-daemon \
+           git://git.yoctoproject.org/libfakekey;name=libfakekey;destsuffix=git/libfakekey \
+"
+
+SRCREV_xsettings-daemon = "b2e5da502f8c5ff75e9e6da771372ef8e40fd9a2"
+SRCREV_libfakekey = "7ad885912efb2131e80914e964d5e635b0d07b40"
+
+S = "${WORKDIR}/git"
+
meta-selftest/recipes-test/devtool/devtool-test-multisrc/example-files.tar.gz new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..fdab4b7367cda3613468c7813f47acaaac74368c
GIT binary patch
meta-selftest/recipes-test/devtool/devtool-test-multisrc/example.patch new file +7 -0
new file mode 100644
index 0000000..2e48bf1
@@ -0,0 +1,7 @@
+diff -udNr example-files/test2 example-files2/test2
+--- example-files/test2	2017-11-22 10:29:11.766891385 +1300
++++ example-files2/test2	2017-11-22 10:30:08.920094705 +1300
+@@ -1 +1,3 @@
+ Another example file
++
++More content added here
meta-selftest/recipes-test/devtool/devtool-test-multisrc/mypackage-1.0.tar.gz new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..b513b283e9adece54a2495ad5f9eaf8eb9eebdd0
GIT binary patch
meta-selftest/recipes-test/devtool/devtool-test-multisrc_1.0.bb new file +11 -0
new file mode 100644
index 0000000..2b81f83
@@ -0,0 +1,11 @@
+SUMMARY = "Recipe to test multiple source trees"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+
+SRC_URI = "file://mypackage-${PV}.tar.gz \
+           file://example-files.tar.gz \
+           file://example.patch;patchdir=../example-files \
+"
+
+S = "${WORKDIR}/mypackage-${PV}"
+
meta/lib/oeqa/selftest/cases/devtool.py +109 -0

115 lines not on this page · raw message

Signature · 1 line
2.9.5
#12 lore raw

PATCH 12/12 lib/oe/recipeutils: add .txz extension to archive list

In reply to Paul Eggleton

Prompted by bitbake commit 2ba8a6b25ccc12e7b543e8450121e5311c7a701d, add .txz to the list of archives used within get_recipe_local_files() here as well.

---
 meta/lib/oe/recipeutils.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/recipeutils.py +1 -1
index f4a40c2..ac3b027 100644
@@ -413,7 +413,7 @@ def get_recipe_local_files(d, patches=False, archives=False):
     # fetcher) though note that this only encompasses actual container formats
     # i.e. that can contain multiple files as opposed to those that only
     # contain a compressed stream (i.e. .tar.gz as opposed to just .gz)
-    archive_exts = ['.tar', '.tgz', '.tar.gz', '.tar.Z', '.tbz', '.tbz2', '.tar.bz2', '.tar.xz', '.tar.lz', '.zip', '.jar', '.rpm', '.srpm', '.deb', '.ipk', '.tar.7z', '.7z']
+    archive_exts = ['.tar', '.tgz', '.tar.gz', '.tar.Z', '.tbz', '.tbz2', '.tar.bz2', '.txz', '.tar.xz', '.tar.lz', '.zip', '.jar', '.rpm', '.srpm', '.deb', '.ipk', '.tar.7z', '.7z']
     ret = {}
     for uri in uris:
         if fetch.ud[uri].type == 'file':
Signature · 1 line
2.9.5
#13 lore raw

✗ patchtest: failure for devtool improvements

In reply to Paul Eggleton

== Series Details ==

Series: devtool improvements
Revision: 1
URL : https://patchwork.openembedded.org/series/10071/
State : failure

== Summary ==

Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures:

* Issue A patch file has been added, but does not have a Signed-off-by tag [test_signed_off_by_presence]
Suggested fix Sign off the added patch file (meta-selftest/recipes-test/devtool/devtool-test-multisrc/example.patch)

* Issue Added patch file is missing Upstream-Status in the header [test_upstream_status_presence_format]

  Suggested fix    Add Upstream-Status: <Valid status> to the header of meta-selftest/recipes-test/devtool/devtool-test-multisrc/example.patch
  Standard format  Upstream-Status: <Valid status>
  Valid status     Pending, Accepted, Backport, Denied, Inappropriate [reason], Submitted [where]

If you believe any of these test results are incorrect, please reply to the mailing list ([email protected]) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...).

---

Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe
#14 lore raw
In reply to Paul Eggleton
Quoted 4 lines · Paul Eggleton

A set of improvements for devtool, mostly aimed at improving the handling of recipes that fetch multiple source trees (i.e. have multiple tarballs or repositories in SRC_URI), with a couple of other fixes thrown in.

Unfortunately, this badly breaks 'devtool upgrade':

======

ak@kanavin-desktop:~/development/poky/build$ devtool upgrade openssl -V 1.0.2n
.....
WARNING: This recipe fetches more than one source tree. Each source tree will be tracked in a separate git repository.
NOTE: Extracting upgraded version source...
ERROR: Execution of 'git checkout devtool-base -b devtool-1.0.2n' failed with exit code 128:
fatal: 'devtool-base' is not a commit and a branch 'devtool-1.0.2n' cannot be created from it
=====

I would strongly request that this patchset not be merged into master until the issue is resolved.

Alex

#15 lore raw

Hi Alex,

Quoted 21 lines · NZDT Alexander Kanavin
Quoted 4 lines · Paul Eggleton

A set of improvements for devtool, mostly aimed at improving the handling of recipes that fetch multiple source trees (i.e. have multiple tarballs or repositories in SRC_URI), with a couple of other fixes thrown in.

Unfortunately, this badly breaks 'devtool upgrade':

======

ak@kanavin-desktop:~/development/poky/build$ devtool upgrade openssl -V 
1.0.2n
.....
WARNING: This recipe fetches more than one source tree. Each source tree 
will be tracked in a separate git repository.
NOTE: Extracting upgraded version source...
ERROR: Execution of 'git checkout devtool-base -b devtool-1.0.2n' failed 
with exit code 128:
fatal: 'devtool-base' is not a commit and a branch 'devtool-1.0.2n' 
cannot be created from it
=====
Quoted 2 lines

I would strongly request that this patchset not be merged into master until the issue is resolved.

That's interesting - the devtool upgrade tests apparently passed so clearly we have a hole in our tests. I will look into it.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre