devtool improvements
cover devtool improvements
A set of improvements for devtool, mostly aimed at improving the handling of recipes that fetch multiple source trees (i.e. have multiple tarballs or repositories in SRC_URI), with a couple of other fixes thrown in.
The following changes since commit a7cd9d1183be603777fc9c8c448281fe01224f7b:
lib/oe/utils: remove param_bool() (2017-12-02 11:24:38 +0000)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib paule/devtool32-oe
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=paule/devtool32-oe
Paul Eggleton (12):
devtool: refactor return for _extract_source()
devtool: reset: delete bbappend file if _check_preserve() doesn't
devtool: finish: fix erroneously creating bbappend for relative paths
devtool: deploy-target: don't specify ssh/scp port unless user does
devtool: extract: fix usage with kern-tools-native
devtool: modify: pick up commits from previously created source tree lib/oe/recipeutils: allow patch_recipe_file() to be re-called
devtool: support extracting multiple source trees
devtool: fix handling of linux-yocto after multisrc changes
oe-selftest: devtool: fix kernel test for multisrc changes
oe-selftest: devtool: add test for multiple source trees lib/oe/recipeutils: add .txz extension to archive list
.../devtool/devtool-test-multisrc-subdir_1.0.bb | 13 + .../devtool-test-multisrc/example-files.tar.gz | Bin 0 -> 236 bytes .../devtool/devtool-test-multisrc/example.patch | 7 + .../devtool-test-multisrc/mypackage-1.0.tar.gz | Bin 0 -> 903 bytes .../devtool/devtool-test-multisrc_1.0.bb | 11 + meta/classes/devtool-source.bbclass | 63 ++- meta/classes/externalsrc.bbclass | 2 +- meta/classes/kernel-yocto.bbclass | 7 +- meta/lib/oe/recipeutils.py | 15 +- meta/lib/oe/utils.py | 24 ++ meta/lib/oeqa/selftest/cases/devtool.py | 124 +++++- scripts/devtool | 61 ++- scripts/lib/devtool/__init__.py | 14 + scripts/lib/devtool/deploy.py | 12 +- scripts/lib/devtool/standard.py | 426 ++++++++++++++------- scripts/lib/devtool/upgrade.py | 4 +- 16 files changed, 601 insertions(+), 182 deletions(-) create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc-subdir_1.0.bb create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc/example-files.tar.gz create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc/example.patch create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc/mypackage-1.0.tar.gz create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc_1.0.bb
Signature · 1 line
2.9.5
PATCH 1/12 devtool: refactor return for _extract_source()
Use a namedtuple to return information to the caller, since I've been expanding that information we should avoid having to change all of the calling code each time.
Additionally, it turned out that a bunch of the callers were checking for None being returned in the initial_rev value, but that's no longer possible, so tidy up the calling code.
--- scripts/lib/devtool/standard.py | 29 ++++++++++------------------- scripts/lib/devtool/upgrade.py | 4 ++-- 2 files changed, 12 insertions(+), 21 deletions(-)
scripts/lib/devtool/standard.py +10 -19
@@ -29,7 +29,7 @@ import scriptutils import errno import glob import filecmp -from collections import OrderedDict +from collections import OrderedDict, namedtuple from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, setup_git_repo, recipe_to_append, get_bbclassextend_targets, update_unlockedsigs, check_prerelease_version, check_git_repo_dirty, check_git_repo_op, DevtoolError from devtool import parse_recipe
@@ -437,15 +437,11 @@ def extract(args, config, basepath, workspace): return 1 srctree = os.path.abspath(args.srctree) - initial_rev, _ = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides) + _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides) logger.info('Source tree extracted to %s' % srctree) - - if initial_rev: - return 0 - else: - return 1 finally: tinfoil.shutdown() + return 0 def sync(args, config, basepath, workspace): """Entry point for the devtool 'sync' subcommand"""
@@ -461,17 +457,15 @@ def sync(args, config, basepath, workspace): return 1 srctree = os.path.abspath(args.srctree) - initial_rev, _ = _extract_source(srctree, args.keep_temp, args.branch, True, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=True) + _extract_source(srctree, args.keep_temp, args.branch, True, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=True) logger.info('Source tree %s synchronized' % srctree) - - if initial_rev: - return 0 - else: - return 1 finally: tinfoil.shutdown() + return 0 +ExtractSourceResult = namedtuple('ExtractSourceResult', ['initial_rev', 'srcsubdir']) + def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, workspace, fixed_setup, d, tinfoil, no_overrides=False): """Extract sources of a recipe""" import oe.recipeutils
@@ -512,8 +506,6 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works if extra_overrides: logger.info('SRC_URI contains some conditional appends/prepends - will create branches to represent these') - initial_rev = None - appendexisted = False recipefile = d.getVar('FILE') appendfile = recipe_to_append(recipefile, config)
@@ -659,7 +651,7 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works logger.info('Preserving temporary directory %s' % tempdir) else: shutil.rmtree(tempdir) - return initial_rev, srcsubdir_rel + return ExtractSourceResult(initial_rev, srcsubdir_rel) def _add_md5(config, recipename, filename): """Record checksum of a file (or recursively for a directory) to the md5-file of the workspace"""
@@ -759,9 +751,8 @@ def modify(args, config, basepath, workspace): commits = [] check_commits = False if not args.no_extract: - initial_rev, _ = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides) - if not initial_rev: - return 1 + ret = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides) + initial_rev = ret.initial_rev logger.info('Source tree extracted to %s' % srctree) # Get list of commits since this revision (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_rev, cwd=srctree)
scripts/lib/devtool/upgrade.py +2 -2
@@ -474,12 +474,12 @@ def upgrade(args, config, basepath, workspace): rf = None try: logger.info('Extracting current version source...') - rev1, srcsubdir1 = standard._extract_source(srctree, False, 'devtool-orig', False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides) + current_ret = standard._extract_source(srctree, False, 'devtool-orig', False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides) logger.info('Extracting upgraded version source...') rev2, md5, sha256, srcbranch, srcsubdir2 = _extract_new_source(args.version, srctree, args.no_patch, args.srcrev, args.srcbranch, args.branch, args.keep_temp, tinfoil, rd) - rf, copied = _create_new_recipe(args.version, md5, sha256, args.srcrev, srcbranch, srcsubdir1, srcsubdir2, config.workspace_path, tinfoil, rd) + rf, copied = _create_new_recipe(args.version, md5, sha256, args.srcrev, srcbranch, current_ret.srcsubdir, srcsubdir2, config.workspace_path, tinfoil, rd) except bb.process.CmdError as e: _upgrade_error(e, rf, srctree) except DevtoolError as e:
Signature · 1 line
2.9.5
PATCH 2/12 devtool: reset: delete bbappend file if _check_preserve() doesn't
In reply to Paul EggletonIf the .devtool_md5 file doesn't contain a reference to the bbappend file (e.g. because devtool was interrupted before it could write that out) then _check_preserve() won't delete it, so we need to delete it separately because otherwise the recipe won't actually be reset.
--- scripts/lib/devtool/standard.py | 7 +++++++ 1 file changed, 7 insertions(+)
scripts/lib/devtool/standard.py +7 -0
@@ -1781,6 +1781,13 @@ def _reset(recipes, no_clean, config, basepath, workspace): for pn in recipes: _check_preserve(config, pn) + appendfile = workspace[pn]['bbappend'] + if os.path.exists(appendfile): + # This shouldn't happen, but is possible if devtool errored out prior to + # writing the md5 file. We need to delete this here or the recipe won't + # actually be reset + os.remove(appendfile) + preservepath = os.path.join(config.workspace_path, 'attic', pn, pn) def preservedir(origdir): if os.path.exists(origdir):
Signature · 1 line
2.9.5
PATCH 3/12 devtool: finish: fix erroneously creating bbappend for relative paths
In reply to Paul EggletonAfter OE-Core rev 5e3fe00a0233d563781849a44f53885b4e924a9c we call os.path.abspath() on the original layer path, but we later compare that to the destination layer path. If that layer path isn't absolute but is effectively the same path, it should be writing to the original recipe but because we weren't making it absolute we were writing a bbappend instead. Call os.path.abspath() on the destination path as well to avoid that.
--- scripts/lib/devtool/standard.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
scripts/lib/devtool/standard.py +1 -3
@@ -1848,9 +1848,7 @@ def _get_layer(layername, d): layerdir = layers.get('meta', None) else: layerdir = layers.get(layername, None) - if layerdir: - layerdir = os.path.abspath(layerdir) - return layerdir or layername + return os.path.abspath(layerdir or layername) def finish(args, config, basepath, workspace): """Entry point for the devtool 'finish' subcommand"""
Signature · 1 line
2.9.5
PATCH 4/12 devtool: deploy-target: don't specify ssh/scp port unless user does
In reply to Paul EggletonIf the user doesn't specify a port then we should avoid specifying one on the ssh/scp command line in case the user has configured one for the host they are connecting to, which was being overridden unnecessarily.
Fixes [YOCTO #12381].
--- scripts/lib/devtool/deploy.py | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-)
scripts/lib/devtool/deploy.py +4 -8
@@ -213,9 +213,7 @@ def deploy(args, config, basepath, workspace): scp_port = '' ssh_port = '' - if not args.port: - raise DevtoolError("If you specify -P/--port then you must provide the port to be used to connect to the target") - else: + if args.port: scp_port = "-P %s" % args.port ssh_port = "-p %s" % args.port
@@ -280,9 +278,7 @@ def undeploy(args, config, basepath, workspace): scp_port = '' ssh_port = '' - if not args.port: - raise DevtoolError("If you specify -P/--port then you must provide the port to be used to connect to the target") - else: + if args.port: scp_port = "-P %s" % args.port ssh_port = "-p %s" % args.port
@@ -328,7 +324,7 @@ def register_commands(subparsers, context): parser_deploy.add_argument('-n', '--dry-run', help='List files to be deployed only', action='store_true') parser_deploy.add_argument('-p', '--no-preserve', help='Do not preserve existing files', action='store_true') parser_deploy.add_argument('--no-check-space', help='Do not check for available space before deploying', action='store_true') - parser_deploy.add_argument('-P', '--port', default='22', help='Port to use for connection to the target') + parser_deploy.add_argument('-P', '--port', help='Specify port to use for connection to the target') strip_opts = parser_deploy.add_mutually_exclusive_group(required=False) strip_opts.add_argument('-S', '--strip',
@@ -350,5 +346,5 @@ def register_commands(subparsers, context): parser_undeploy.add_argument('-s', '--show-status', help='Show progress/status output', action='store_true') parser_undeploy.add_argument('-a', '--all', help='Undeploy all recipes deployed on the target', action='store_true') parser_undeploy.add_argument('-n', '--dry-run', help='List files to be undeployed only', action='store_true') - parser_undeploy.add_argument('-P', '--port', default='22', help='Port to use for connection to the target') + parser_undeploy.add_argument('-P', '--port', help='Specify port to use for connection to the target') parser_undeploy.set_defaults(func=undeploy)
Signature · 1 line
2.9.5
PATCH 5/12 devtool: extract: fix usage with kern-tools-native
In reply to Paul EggletonThe kern-tools-native recipe as it currently stands is unusual in that it fetches source from a repository but sets S = "${WORKDIR}" which causes some problems. First you get a failure because we're calling "git commit" unconditionally even if there are no local files, and there aren't any in this case which means the commit fails.
After that's fixed, we hit another problem where "recipe-sysroot-native" subdirectory appears in the extracted source tree. We don't want that so exclude it from copying.
--- meta/classes/devtool-source.bbclass | 6 ++++++ scripts/lib/devtool/standard.py | 6 +++--- 2 files changed, 9 insertions(+), 3 deletions(-)
meta/classes/devtool-source.bbclass +6 -0
@@ -118,7 +118,13 @@ python devtool_post_unpack() { os.path.basename(fname) not in recipe_patches] srcsubdir = d.getVar('DEVTOOL_PATCH_SRCDIR') # Move source files to S + excludevars = ['RECIPE_SYSROOT', 'RECIPE_SYSROOT_NATIVE'] + excludepaths = [] + for excludevar in excludevars: + excludepaths.append(os.path.relpath(d.getVar(excludevar), workdir) + os.sep) for path in src_files: + if path.startswith(tuple(excludepaths)): + continue _move_file(os.path.join(workdir, path), os.path.join(srcsubdir, path)) elif os.path.dirname(srcsubdir) != workdir:
scripts/lib/devtool/standard.py +3 -3
@@ -634,9 +634,9 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works addfiles.append(os.path.join(relpth, fn)) if addfiles: bb.process.run('git add %s' % ' '.join(addfiles), cwd=srctree) - useroptions = [] - oe.patch.GitApplyTree.gitCommandUserOptions(useroptions, d=d) - bb.process.run('git %s commit -a -m "Committing local file symlinks\n\n%s"' % (' '.join(useroptions), oe.patch.GitApplyTree.ignore_commit_prefix), cwd=srctree) + useroptions = [] + oe.patch.GitApplyTree.gitCommandUserOptions(useroptions, d=d) + bb.process.run('git %s commit -a -m "Committing local file symlinks\n\n%s"' % (' '.join(useroptions), oe.patch.GitApplyTree.ignore_commit_prefix), cwd=srctree) if is_kernel_yocto: logger.info('Copying kernel config to srctree')
Signature · 1 line
2.9.5
PATCH 6/12 devtool: modify: pick up commits from previously created source tree
In reply to Paul EggletonIf you use devtool modify, then devtool reset, keep the source tree and then devtool modify on the same recipe with the -n option to re-use the existing source tree, we should pick up the commit hashes properly from the source tree so that later on devtool finish has these to compare to the commits in the tree at that time. We also need to be careful the second time around that we only get the original commits rather than the current HEAD which may be the result of user changes (hence using "devtool-patched", the tag that was placed at the original HEAD).
--- scripts/lib/devtool/standard.py | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
scripts/lib/devtool/standard.py +7 -3
@@ -750,13 +750,11 @@ def modify(args, config, basepath, workspace): initial_rev = None commits = [] check_commits = False + torev = 'HEAD' if not args.no_extract: ret = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides) initial_rev = ret.initial_rev logger.info('Source tree extracted to %s' % srctree) - # Get list of commits since this revision - (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_rev, cwd=srctree) - commits = stdout.split() check_commits = True else: if os.path.exists(os.path.join(srctree, '.git')):
@@ -767,6 +765,7 @@ def modify(args, config, basepath, workspace): stdout = '' if stdout: check_commits = True + torev = 'devtool-patched' for line in stdout.splitlines(): if line.startswith('*'): (stdout, _) = bb.process.run('git rev-parse devtool-base', cwd=srctree)
@@ -776,6 +775,11 @@ def modify(args, config, basepath, workspace): (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srctree) initial_rev = stdout.rstrip() + if initial_rev: + # Get list of commits since this revision + (stdout, _) = bb.process.run('git rev-list --reverse %s..%s' % (initial_rev, torev), cwd=srctree) + commits = stdout.split() + branch_patches = {} if check_commits: # Check if there are override branches
Signature · 1 line
2.9.5
PATCH 7/12 lib/oe/recipeutils: allow patch_recipe_file() to be re-called
In reply to Paul EggletonIf patch_recipe_file() is called with output redirection on the same file twice in succession, then we don't want to wipe out the changes the first call made so we need to be reading in the redirected file if it exists instead of the original one.
This is important to enable devtool finish to work with multiple source trees within the same recipe.
--- meta/lib/oe/recipeutils.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
meta/lib/oe/recipeutils.py +7 -2
@@ -258,13 +258,18 @@ def patch_recipe_file(fn, values, patch=False, relpath='', redirect_output=None) since this cannot handle all situations. """ - with open(fn, 'r') as f: + read_fn = fn + if redirect_output: + redirect_fn = os.path.join(redirect_output, os.path.basename(fn)) + if os.path.exists(redirect_fn): + read_fn = redirect_fn + with open(read_fn, 'r') as f: fromlines = f.readlines() _, tolines = patch_recipe_lines(fromlines, values) if redirect_output: - with open(os.path.join(redirect_output, os.path.basename(fn)), 'w') as f: + with open(redirect_fn, 'w') as f: f.writelines(tolines) return None elif patch:
Signature · 1 line
2.9.5
PATCH 8/12 devtool: support extracting multiple source trees
In reply to Paul EggletonIf you have multiple source trees being extracted to the work directory within a recipe (e.g. you have two tarballs referred to in SRC_URI) and one isn't being extracted into the other, then devtool failed to extract all the sources because it only took the source tree that S pointed into. To fix this, we need to take a look at the work directory after do_unpack and see if there are any additional subdirectories; if so we need to put the main source tree in a subdirectory and put the additional subdirectories next to it. We also ensure that symlinks from the work directory get created at the end of do_unpack to point to these (so that references in the recipe continue to work).
In addition to multiple source trees at the work directory level, this patch also handles multiple nested git trees (where you have multiple git URLs in SRC_URI with one or more inside another). These caused a different problem, where changes in sub-repos are not fully captured at the top level - we need to handle each repo separately.
--- meta/classes/devtool-source.bbclass | 57 ++++- meta/lib/oe/recipeutils.py | 4 +- meta/lib/oe/utils.py | 24 +++ meta/lib/oeqa/selftest/cases/devtool.py | 2 +- scripts/devtool | 61 ++++-- scripts/lib/devtool/__init__.py | 14 ++ scripts/lib/devtool/standard.py | 370 +++++++++++++++++++++++--------- 7 files changed, 396 insertions(+), 136 deletions(-)
meta/classes/devtool-source.bbclass +47 -10
@@ -69,7 +69,7 @@ python devtool_post_unpack() { import shutil sys.path.insert(0, os.path.join(d.getVar('COREBASE'), 'scripts', 'lib')) import scriptutils - from devtool import setup_git_repo + from devtool import setup_git_repo, find_git_repos tempdir = d.getVar('DEVTOOL_TEMPDIR') workdir = d.getVar('WORKDIR')
@@ -95,8 +95,26 @@ python devtool_post_unpack() { oe.recipeutils.get_recipe_patches(d)] local_files = oe.recipeutils.get_recipe_local_files(d) - # Ignore local files with subdir={BP} + excludeitems = recipe_patches + list(local_files.keys()) + pthvars = ['RECIPE_SYSROOT', 'RECIPE_SYSROOT_NATIVE', 'S'] + for pthvar in pthvars: + relp = os.path.relpath(d.getVar(pthvar), d.getVar('WORKDIR')) + if not relp.startswith('..'): + excludeitems.append(relp.split(os.sep)[0]) + extradirs = [] srcabspath = os.path.abspath(srcsubdir) + if srcabspath != os.path.abspath(workdir): + for pth in os.listdir(workdir): + if pth in excludeitems: + continue + wpth = os.path.join(workdir, pth) + if os.path.isdir(wpth) and os.listdir(wpth): + extradirs.append(wpth) + + repos = find_git_repos(srcabspath) + extradirs.extend(repos) + + # Ignore local files with subdir={BP} local_files = [fname for fname in local_files if os.path.exists(os.path.join(workdir, fname)) and (srcabspath == workdir or not
@@ -145,11 +163,20 @@ python devtool_post_unpack() { (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srcsubdir) initial_rev = stdout.rstrip() + + initial_revs = {} + for extradir in extradirs: + setup_git_repo(extradir, d.getVar('PV'), devbranch, d=d) + (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=extradir) + initial_revs[extradir] = stdout.rstrip() + with open(os.path.join(tempdir, 'initial_rev'), 'w') as f: f.write(initial_rev) with open(os.path.join(tempdir, 'srcsubdir'), 'w') as f: - f.write(srcsubdir) + f.write('%s\n' % srcsubdir) + for extradir in extradirs: + f.write('%s=%s\n' % (extradir, initial_revs[extradir])) } python devtool_pre_patch() {
@@ -160,18 +187,25 @@ python devtool_pre_patch() { python devtool_post_patch() { import shutil tempdir = d.getVar('DEVTOOL_TEMPDIR') + + srcdirs = [] with open(os.path.join(tempdir, 'srcsubdir'), 'r') as f: - srcsubdir = f.read() + for line in f: + line = line.rstrip() + if line: + srcdirs.append(line.split('=')[0]) + srcsubdir = srcdirs[0] + with open(os.path.join(tempdir, 'initial_rev'), 'r') as f: initial_rev = f.read() - def rm_patches(): - patches_dir = os.path.join(srcsubdir, 'patches') + def rm_patches(pth): + patches_dir = os.path.join(pth, 'patches') if os.path.exists(patches_dir): shutil.rmtree(patches_dir) # Restore any "patches" directory that was actually part of the source tree try: - bb.process.run('git checkout -- patches', cwd=srcsubdir) + bb.process.run('git checkout -- patches', cwd=pth) except bb.process.ExecutionError: pass
@@ -194,7 +228,7 @@ python devtool_post_patch() { localdata = bb.data.createCopy(d) localdata.setVar('OVERRIDES', ':'.join(no_overrides)) bb.build.exec_func('do_patch', localdata) - rm_patches() + rm_patches(srcsubdir) # Now we need to reconcile the dev branch with the no-overrides one # (otherwise we'd likely be left with identical commits that have different hashes) bb.process.run('git checkout %s' % devbranch, cwd=srcsubdir)
@@ -212,12 +246,15 @@ python devtool_post_patch() { # Run do_patch function with the override applied localdata.appendVar('OVERRIDES', ':%s' % override) bb.build.exec_func('do_patch', localdata) - rm_patches() + rm_patches(srcsubdir) # Now we need to reconcile the new branch with the no-overrides one # (otherwise we'd likely be left with identical commits that have different hashes) bb.process.run('git rebase devtool-no-overrides', cwd=srcsubdir) bb.process.run('git checkout %s' % devbranch, cwd=srcsubdir) - bb.process.run('git tag -f devtool-patched', cwd=srcsubdir) + for srcdir in srcdirs: + bb.process.run('git tag -f devtool-patched', cwd=srcdir) + if srcdir != srcsubdir: + rm_patches(srcdir) } python devtool_post_configure() {
meta/lib/oe/recipeutils.py +2 -2
@@ -445,10 +445,10 @@ def get_recipe_patches(d): """Get a list of the patches included in SRC_URI within a recipe.""" import oe.patch patches = oe.patch.src_patches(d, expand=False) - patchfiles = [] + patchfiles = OrderedDict() for patch in patches: _, _, local, _, _, parm = bb.fetch.decodeurl(patch) - patchfiles.append(local) + patchfiles[local] = parm return patchfiles
meta/lib/oe/utils.py +24 -0
@@ -368,3 +368,27 @@ class ImageQAFailed(bb.build.FuncFailed): msg = msg + ' (%s)' % self.description return msg + +def is_path_under(path1, possible_parent): + """ + Return True if a path is underneath another, False otherwise. + Multiple paths to test can be specified (as a list or tuple) in + which case all specified test paths must be under the parent to + return True. + """ + def abs_path_trailing(pth): + pth_abs = os.path.abspath(pth) + if not pth_abs.endswith(os.sep): + pth_abs += os.sep + return pth_abs + + possible_parent_abs = abs_path_trailing(possible_parent) + if isinstance(path1, str): + testpaths = [path1] + else: + testpaths = path1 + for path in testpaths: + path_abs = abs_path_trailing(path) + if not path_abs.startswith(possible_parent_abs): + return False + return True
meta/lib/oeqa/selftest/cases/devtool.py +1 -1
@@ -7,7 +7,7 @@ import fnmatch import oeqa.utils.ftools as ftools from oeqa.selftest.case import OESelftestTestCase -from oeqa.utils.commands import runCmd, bitbake, get_bb_var, create_temp_layer +from oeqa.utils.commands import runCmd, bitbake, get_bb_var, create_temp_layer, CommandError from oeqa.utils.commands import get_bb_vars, runqemu, get_test_layer from oeqa.core.decorator.oeid import OETestID
scripts/devtool +45 -16
82 lines not on this page · raw message
scripts/lib/devtool/__init__.py +14 -0
@@ -223,6 +223,20 @@ def setup_git_repo(repodir, version, devbranch, basetag='devtool-base', d=None): bb.process.run('git checkout -b %s' % devbranch, cwd=repodir) bb.process.run('git tag -f %s' % basetag, cwd=repodir) +def find_git_repos(pth, toplevel=False): + """ + Find git repositories under a path + """ + repos = [] + if toplevel and os.path.isdir(os.path.join(pth, '.git')): + repos.append(pth) + for root, dirs, _ in os.walk(pth): + for dfn in dirs: + dfp = os.path.join(root, dfn) + if os.path.isdir(os.path.join(dfp, '.git')) and dfp not in repos: + repos.append(dfp) + return repos + def recipe_to_append(recipefile, config, wildcard=False): """ Convert a recipe file to a bbappend file path within the workspace.
scripts/lib/devtool/standard.py +263 -107
578 lines not on this page · raw message
Signature · 1 line
2.9.5
PATCH 9/12 devtool: fix handling of linux-yocto after multisrc changes
In reply to Paul Eggletondevtool now handles multiple source trees for any recipe that includes them in SRC_URI, extracting them all side-by-side so that you can make changes in any of them. As a result, when running devtool modify on a linux-yocto kernel recipe under the source path you will get a "source" subdirectory containing the kernel source and a "kernel-meta" subdirectory next to it containing the kernel metadata. (Previously you just got the source tree and the kernel metadata remained in the work directory). We create a symlink automatically at do_unpack from the work directory so that it can still be found there, however kernel_feature_dirs() expects to find the kernel-meta repository and we also now need to make externalsrc remove that so that it doesn't unpack and overwrite the one we've already extracted. Change kernel_feature_dirs() so that if there are no kmeta entries in SRC_URI, it will fall back to a directory named ${KMETA} if it happens to be present in the work directory, ignoring how it got there.
--- meta/classes/externalsrc.bbclass | 2 +- meta/classes/kernel-yocto.bbclass | 7 ++++++- scripts/lib/devtool/standard.py | 4 +++- 3 files changed, 10 insertions(+), 3 deletions(-)
meta/classes/externalsrc.bbclass +1 -1
@@ -65,7 +65,7 @@ python () { url_data = fetch.ud[url] parm = url_data.parm if (url_data.type == 'file' or - 'type' in parm and parm['type'] == 'kmeta'): + (parm.get('type', None) == 'kmeta' and d.getVar('EXTERNALSRC_KMETA') != "1")): local_srcuri.append(url) d.setVar('SRC_URI', ' '.join(local_srcuri))
meta/classes/kernel-yocto.bbclass +6 -1
@@ -42,7 +42,12 @@ def find_kernel_feature_dirs(d): destdir = parm["destsuffix"] if type == "kmeta": feature_dirs.append(destdir) - + if not feature_dirs: + # If the kernel-meta directory already exists (e.g from externalsrc) + # with EXTERNALSRC_KMETA = "1" then use it + kmetadir = d.getVar('KMETA') + if os.path.isdir(os.path.join(d.getVar('WORKDIR'), kmetadir)): + feature_dirs.append(kmetadir) return feature_dirs # find the master/machine source branch. In the same way that the fetcher proceses
scripts/lib/devtool/standard.py +3 -1
@@ -909,7 +909,9 @@ def modify(args, config, basepath, workspace): if bb.data.inherits_class('kernel', rd): f.write('SRCTREECOVEREDTASKS = "do_validate_branches do_kernel_checkout ' - 'do_fetch do_unpack do_kernel_configme do_kernel_configcheck"\n') + 'do_fetch do_kernel_configme do_kernel_configcheck"\n') + if bb.data.inherits_class('kernel-yocto', rd): + f.write('EXTERNALSRC_KMETA = "1"\n') f.write('\ndo_patch() {\n' ' :\n' '}\n')
Signature · 1 line
2.9.5
PATCH 10/12 oe-selftest: devtool: fix kernel test for multisrc changes
In reply to Paul EggletonFix the oe-selftest test to understand that the source will be found in a "source" subdirectory if the kernel inherits linux-yocto since there are multiple trees extracted.
--- meta/lib/oeqa/selftest/cases/devtool.py | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
meta/lib/oeqa/selftest/cases/devtool.py +9 -4
@@ -1676,9 +1676,14 @@ class DevtoolTests(DevtoolBase): bitbake('%s -c clean' % kernel_provider) #Step 4.1 runCmd('devtool modify virtual/kernel -x %s' % tempdir) - self.assertExists(os.path.join(tempdir, 'Makefile'), 'Extracted source could not be found') + # If it's linux-yocto, we'll have two "source" trees (one the actual source, + # the other the kernel metadata) + tempsrc = os.path.join(tempdir, 'source') + if not os.path.isdir(tempsrc): + tempsrc = tempdir + self.assertExists(os.path.join(tempsrc, 'Makefile'), 'Extracted source could not be found') #Step 4.2 - configfile = os.path.join(tempdir,'.config') + configfile = os.path.join(tempsrc, '.config') diff = runCmd('diff %s %s' % (tmpconfig, configfile)) self.assertEqual(0,diff.status,'Kernel .config file is not the same using bitbake and devtool') #Step 4.3
@@ -1689,12 +1694,12 @@ class DevtoolTests(DevtoolBase): self.assertExists(kernelfile, 'Kernel was not build correctly') #Modify the kernel source - modfile = os.path.join(tempdir,'arch/x86/boot/header.S') + modfile = os.path.join(tempsrc, 'arch/x86/boot/header.S') modstring = "Use a boot loader. Devtool testing." modapplied = runCmd("sed -i 's/Use a boot loader./%s/' %s" % (modstring, modfile)) self.assertEqual(0,modapplied.status,'Modification to %s on kernel source failed' % modfile) #Modify the configuration - codeconfigfile = os.path.join(tempdir,'.config.new') + codeconfigfile = os.path.join(tempsrc, '.config.new') modconfopt = "CONFIG_SG_POOL=n" modconf = runCmd("sed -i 's/CONFIG_SG_POOL=y/%s/' %s" % (modconfopt, codeconfigfile)) self.assertEqual(0,modconf.status,'Modification to %s failed' % codeconfigfile)
Signature · 1 line
2.9.5
PATCH 11/12 oe-selftest: devtool: add test for multiple source trees
In reply to Paul EggletonAdd two synthetic tests for devtool modify + devtool finish: first with multiple source trees side-by-side, and second to test with one as a subdirectory of the main source tree. These also test devtool finish's recently added dry-run option and that detects and errors on uncommitted changes without being forced.
--- .../devtool/devtool-test-multisrc-subdir_1.0.bb | 13 +++ .../devtool-test-multisrc/example-files.tar.gz | Bin 0 -> 236 bytes .../devtool/devtool-test-multisrc/example.patch | 7 ++ .../devtool-test-multisrc/mypackage-1.0.tar.gz | Bin 0 -> 903 bytes .../devtool/devtool-test-multisrc_1.0.bb | 11 +++ meta/lib/oeqa/selftest/cases/devtool.py | 109 +++++++++++++++++++++ 6 files changed, 140 insertions(+) create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc-subdir_1.0.bb create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc/example-files.tar.gz create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc/example.patch create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc/mypackage-1.0.tar.gz create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc_1.0.bb
meta-selftest/recipes-test/devtool/devtool-test-multisrc-subdir_1.0.bb new file +13 -0
@@ -0,0 +1,13 @@ +SUMMARY = "Recipe to test multiple source trees" +LICENSE = "MIT" +LIC_FILES_CHKSUM = "file://settings-daemon.c;beginline=1;endline=26;md5=8d77ba1c7a55df48d8d06c5f3d86b49d" + +SRC_URI = "git://git.yoctoproject.org/xsettings-daemon;name=xsettings-daemon \ + git://git.yoctoproject.org/libfakekey;name=libfakekey;destsuffix=git/libfakekey \ +" + +SRCREV_xsettings-daemon = "b2e5da502f8c5ff75e9e6da771372ef8e40fd9a2" +SRCREV_libfakekey = "7ad885912efb2131e80914e964d5e635b0d07b40" + +S = "${WORKDIR}/git" +
meta-selftest/recipes-test/devtool/devtool-test-multisrc/example-files.tar.gz new file binary +0 -0
meta-selftest/recipes-test/devtool/devtool-test-multisrc/example.patch new file +7 -0
@@ -0,0 +1,7 @@ +diff -udNr example-files/test2 example-files2/test2 +--- example-files/test2 2017-11-22 10:29:11.766891385 +1300 ++++ example-files2/test2 2017-11-22 10:30:08.920094705 +1300 +@@ -1 +1,3 @@ + Another example file ++ ++More content added here
meta-selftest/recipes-test/devtool/devtool-test-multisrc/mypackage-1.0.tar.gz new file binary +0 -0
meta-selftest/recipes-test/devtool/devtool-test-multisrc_1.0.bb new file +11 -0
@@ -0,0 +1,11 @@ +SUMMARY = "Recipe to test multiple source trees" +LICENSE = "MIT" +LIC_FILES_CHKSUM = "file://COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420" + +SRC_URI = "file://mypackage-${PV}.tar.gz \ + file://example-files.tar.gz \ + file://example.patch;patchdir=../example-files \ +" + +S = "${WORKDIR}/mypackage-${PV}" +
meta/lib/oeqa/selftest/cases/devtool.py +109 -0
115 lines not on this page · raw message
Signature · 1 line
2.9.5
PATCH 12/12 lib/oe/recipeutils: add .txz extension to archive list
In reply to Paul EggletonPrompted by bitbake commit 2ba8a6b25ccc12e7b543e8450121e5311c7a701d, add .txz to the list of archives used within get_recipe_local_files() here as well.
--- meta/lib/oe/recipeutils.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/recipeutils.py +1 -1
@@ -413,7 +413,7 @@ def get_recipe_local_files(d, patches=False, archives=False): # fetcher) though note that this only encompasses actual container formats # i.e. that can contain multiple files as opposed to those that only # contain a compressed stream (i.e. .tar.gz as opposed to just .gz) - archive_exts = ['.tar', '.tgz', '.tar.gz', '.tar.Z', '.tbz', '.tbz2', '.tar.bz2', '.tar.xz', '.tar.lz', '.zip', '.jar', '.rpm', '.srpm', '.deb', '.ipk', '.tar.7z', '.7z'] + archive_exts = ['.tar', '.tgz', '.tar.gz', '.tar.Z', '.tbz', '.tbz2', '.tar.bz2', '.txz', '.tar.xz', '.tar.lz', '.zip', '.jar', '.rpm', '.srpm', '.deb', '.ipk', '.tar.7z', '.7z'] ret = {} for uri in uris: if fetch.ud[uri].type == 'file':
Signature · 1 line
2.9.5
✗ patchtest: failure for devtool improvements
In reply to Paul Eggleton== Series Details ==
Series: devtool improvements
Revision: 1
URL : https://patchwork.openembedded.org/series/10071/
State : failure
== Summary ==
Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures:
* Issue A patch file has been added, but does not have a Signed-off-by tag [test_signed_off_by_presence]
Suggested fix Sign off the added patch file (meta-selftest/recipes-test/devtool/devtool-test-multisrc/example.patch)
* Issue Added patch file is missing Upstream-Status in the header [test_upstream_status_presence_format]
Suggested fix Add Upstream-Status: <Valid status> to the header of meta-selftest/recipes-test/devtool/devtool-test-multisrc/example.patch Standard format Upstream-Status: <Valid status> Valid status Pending, Accepted, Backport, Denied, Inappropriate [reason], Submitted [where]
If you believe any of these test results are incorrect, please reply to the mailing list ([email protected]) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...).
---
Guidelines: https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe
Quoted 4 lines · Paul Eggleton
A set of improvements for devtool, mostly aimed at improving the handling of recipes that fetch multiple source trees (i.e. have multiple tarballs or repositories in
SRC_URI), with a couple of other fixes thrown in.
Unfortunately, this badly breaks 'devtool upgrade':
======
ak@kanavin-desktop:~/development/poky/build$ devtool upgrade openssl -V 1.0.2n ..... WARNING: This recipe fetches more than one source tree. Each source tree will be tracked in a separate git repository. NOTE: Extracting upgraded version source... ERROR: Execution of 'git checkout devtool-base -b devtool-1.0.2n' failed with exit code 128: fatal: 'devtool-base' is not a commit and a branch 'devtool-1.0.2n' cannot be created from it =====
I would strongly request that this patchset not be merged into master until the issue is resolved.
Alex
Hi Alex,
Quoted 21 lines · NZDT Alexander Kanavin
Quoted 4 lines · Paul Eggleton
A set of improvements for devtool, mostly aimed at improving the handling of recipes that fetch multiple source trees (i.e. have multiple tarballs or repositories in
SRC_URI), with a couple of other fixes thrown in.Unfortunately, this badly breaks 'devtool upgrade':
======
ak@kanavin-desktop:~/development/poky/build$ devtool upgrade openssl -V 1.0.2n ..... WARNING: This recipe fetches more than one source tree. Each source tree will be tracked in a separate git repository. NOTE: Extracting upgraded version source... ERROR: Execution of 'git checkout devtool-base -b devtool-1.0.2n' failed with exit code 128: fatal: 'devtool-base' is not a commit and a branch 'devtool-1.0.2n' cannot be created from it =====
Quoted 2 lines
I would strongly request that this patchset not be merged into master until the issue is resolved.
That's interesting - the devtool upgrade tests apparently passed so clearly we have a hole in our tests. I will look into it.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre