devtool / recipetool fixes
cover devtool / recipetool fixes
Some fixes for devtool / recipetool issues, along with a couple of related changes to the oe-selftest tests. I have run the oe-selftest tests for devtool and recipetool and they pass with these changes applied.
NOTE: this patchset depends upon the bitbake patchset I just sent to the bitbake-devel list.
The following changes since commit 62f1122ef166eba56441d669c6b3b3fe5f367418:
bluez5: cosmetic fixes to bluetooth bootscript (2017-08-30 11:13:52 +0100)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib paule/devtool29-oe
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=paule/devtool29-oe
Paul Eggleton (12):
devtool: update-recipe: ensure patches get deleted in srcrev mode
devtool: upgrade: fix handling of non-absolute paths
devtool: upgrade: workaround for recipes which apply patches conditional upon class
devtool: edit-recipe: fix regression
oe-selftest: devtool: test find-recipe and edit-recipe
devtool: add: add explicit srcrev/branch options
recipetool: create: make recently added branch/tag handling git specific
recipetool: create: fix SRCPV prefix for non-git SCMs
recipetool: create: suppress npm shrinkwrap/lockdown warnings again
recipetool: create: detect Eclipse licenses
oe-selftest: tinfoil: add a test for variable history
scriptutils: fix fetch_url() to use lowercase dummy recipe name
paul (1):
devtool: upgrade: check that user has configured git properly
meta/lib/oe/recipeutils.py | 43 +++++++++++++++++++++++------- meta/lib/oeqa/selftest/cases/devtool.py | 25 ++++++++++++------ meta/lib/oeqa/selftest/cases/tinfoil.py | 34 ++++++++++++++++++++++++ scripts/lib/devtool/standard.py | 20 +++++++++++--- scripts/lib/devtool/upgrade.py | 32 ++++++++++++++++++++-- scripts/lib/devtool/utilcmds.py | 15 ++++++++--- scripts/lib/recipetool/create.py | 47 ++++++++++++++++++++++++++++----- scripts/lib/scriptutils.py | 2 +- 8 files changed, 183 insertions(+), 35 deletions(-)
Signature · 1 line
2.9.5
PATCH 1/13 devtool: update-recipe: ensure patches get deleted in srcrev mode
Patches that we identify as having been "deleted" (i.e. patches in SRC_URI that no longer appear in the git tree) need to be dropped even if we're updating in srcrev mode. This fixes the case where HEAD of the git tree is valid upstream (i.e. no extra commits), but there are patches left over in the recipe, e.g. when we do devtool upgrade and then all of the commits rebased on top of the new branch get skipped.
Fixes [YOCTO #11972].
--- scripts/lib/devtool/standard.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
scripts/lib/devtool/standard.py +2 -1
@@ -1352,9 +1352,10 @@ def _update_recipe_srcrev(srctree, rd, appendlayerdir, wildcard_version, no_remo old_srcrev = (rd.getVar('SRCREV', False) or '') upd_p, new_p, del_p = _export_patches(srctree, rd, old_srcrev, patches_dir) + logger.debug('Patches: update %s, new %s, delete %s' % (dict(upd_p), dict(new_p), dict(del_p))) # Remove deleted local files and "overlapping" patches - remove_files = list(del_f.values()) + list(upd_p.values()) + remove_files = list(del_f.values()) + list(upd_p.values()) + list(del_p.values()) if remove_files: removedentries = _remove_file_entries(srcuri, remove_files)[0] update_srcuri = True
Signature · 1 line
2.9.5
PATCH 2/13 devtool: upgrade: fix handling of non-absolute paths
In reply to Paul EggletonIf your BBLAYERS has non-absolute paths in it (e.g. "${COREBASE}/../something") then none of the paths matched in copy_recipe_files() with the result that no files got copied and you ended up with an error later on because the recipe file couldn't be found at the destination. Fix this as well as adding an explicit check to see if no files got copied - error out earlier if so.
Fixes [YOCTO #10981].
--- meta/lib/oe/recipeutils.py | 8 +++++--- scripts/lib/devtool/upgrade.py | 3 +++ 2 files changed, 8 insertions(+), 3 deletions(-)
meta/lib/oe/recipeutils.py +5 -3
@@ -334,11 +334,13 @@ def copy_recipe_files(d, tgt_dir, whole_dir=False, download=True): fetch.download() # Copy local files to target directory and gather any remote files - bb_dir = os.path.dirname(d.getVar('FILE')) + os.sep + bb_dir = os.path.abspath(os.path.dirname(d.getVar('FILE'))) + os.sep remotes = [] copied = [] - includes = [path for path in d.getVar('BBINCLUDED').split() if - path.startswith(bb_dir) and os.path.exists(path)] + # Need to do this in two steps since we want to check against the absolute path + includes = [os.path.abspath(path) for path in d.getVar('BBINCLUDED').split() if os.path.exists(path)] + # We also check this below, but we don't want any items in this list being considered remotes + includes = [path for path in includes if path.startswith(bb_dir)] for path in fetch.localpaths() + includes: # Only import files that are under the meta directory if path.startswith(bb_dir):
scripts/lib/devtool/upgrade.py +3 -0
@@ -301,6 +301,9 @@ def _create_new_recipe(newpv, md5, sha256, srcrev, srcbranch, workspace, tinfoil path = os.path.join(workspace, 'recipes', bpn) bb.utils.mkdirhier(path) copied, _ = oe.recipeutils.copy_recipe_files(rd, path) + if not copied: + raise DevtoolError('Internal error - no files were copied for recipe %s' % bpn) + logger.debug('Copied %s to %s' % (copied, path)) oldpv = rd.getVar('PV') if not newpv:
Signature · 1 line
2.9.5
PATCH 3/13 devtool: upgrade: check that user has configured git properly
In reply to Paul EggletonFrom: paul <[email protected]>
If user.name or user.email haven't been set then git rebase can't really work properly. Check that the user has set these and error out if not. (Elsewhere we are relying on OE's git patch functionality which forces a dummy OE value - that's OK there as it's completely under OE's control and therefore it's OK for a dummy OE user to be the committer, but here the rebase may require intervention so it's reasonable to have the user's actual name and email on the operation.)
Fixes [YOCTO #11947].
--- scripts/lib/devtool/upgrade.py | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-)
scripts/lib/devtool/upgrade.py +26 -1
@@ -1,6 +1,6 @@ # Development tool - upgrade command plugin # -# Copyright (C) 2014-2015 Intel Corporation +# Copyright (C) 2014-2017 Intel Corporation # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License version 2 as
@@ -355,6 +355,29 @@ def _create_new_recipe(newpv, md5, sha256, srcrev, srcbranch, workspace, tinfoil return fullpath, copied + +def _check_git_config(): + def getconfig(name): + try: + value = bb.process.run('git config --global %s' % name)[0].strip() + except bb.process.ExecutionError as e: + if e.exitcode == 1: + value = None + else: + raise + return value + + username = getconfig('user.name') + useremail = getconfig('user.email') + configerr = [] + if not username: + configerr.append('Please set your name using:\n git config --global user.name') + if not useremail: + configerr.append('Please set your email using:\n git config --global user.email') + if configerr: + raise DevtoolError('Your git configuration is incomplete which will prevent rebases from working:\n' + '\n'.join(configerr)) + + def upgrade(args, config, basepath, workspace): """Entry point for the devtool 'upgrade' subcommand"""
@@ -365,6 +388,8 @@ def upgrade(args, config, basepath, workspace): if args.srcbranch and not args.srcrev: raise DevtoolError("If you specify --srcbranch/-B then you must use --srcrev/-S to specify the revision" % args.recipename) + _check_git_config() + tinfoil = setup_tinfoil(basepath=basepath, tracking=True) try: rd = parse_recipe(config, tinfoil, args.recipename, True)
Signature · 1 line
2.9.5
PATCH 4/13 devtool: upgrade: workaround for recipes which apply patches conditional upon class
In reply to Paul EggletonIf we're upgrading a recipe that appends additional patches for, say, class-native, and we're just upgrading the target variant, then when we copied the recipe into the workspace we skipped copying the additional patches for the native variant. This caused warnings because the workspace recipe is preferred. Look at SRC_URI for all variants when copying files to work around this.
More work is needed to make it easier to work with recipes that use BBCLASSEXTEND where you need to build more than one variant at once, but this at least fixes the immediate ugliness.
--- meta/lib/oe/recipeutils.py | 35 ++++++++++++++++++++++++++++------- scripts/lib/devtool/upgrade.py | 2 +- 2 files changed, 29 insertions(+), 8 deletions(-)
meta/lib/oe/recipeutils.py +28 -7
@@ -2,7 +2,7 @@ # # Some code borrowed from the OE layer index # -# Copyright (C) 2013-2016 Intel Corporation +# Copyright (C) 2013-2017 Intel Corporation # import sys
@@ -320,7 +320,7 @@ def patch_recipe(d, fn, varvalues, patch=False, relpath=''): -def copy_recipe_files(d, tgt_dir, whole_dir=False, download=True): +def copy_recipe_files(d, tgt_dir, whole_dir=False, download=True, all_variants=False): """Copy (local) recipe files, including both files included via include/require, and files referred to in the SRC_URI variable.""" import bb.fetch2
@@ -328,10 +328,31 @@ def copy_recipe_files(d, tgt_dir, whole_dir=False, download=True): # FIXME need a warning if the unexpanded SRC_URI value contains variable references - uris = (d.getVar('SRC_URI') or "").split() - fetch = bb.fetch2.Fetch(uris, d) - if download: - fetch.download() + uri_values = [] + localpaths = [] + def fetch_urls(rdata): + # Collect the local paths from SRC_URI + srcuri = rdata.getVar('SRC_URI') or "" + if srcuri not in uri_values: + fetch = bb.fetch2.Fetch(srcuri.split(), rdata) + if download: + fetch.download() + for pth in fetch.localpaths(): + if pth not in localpaths: + localpaths.append(pth) + uri_values.append(srcuri) + + fetch_urls(d) + if all_variants: + # Get files for other variants e.g. in the case of a SRC_URI_append + localdata = bb.data.createCopy(d) + variants = (localdata.getVar('BBCLASSEXTEND') or '').split() + if variants: + # Ensure we handle class-target if we're dealing with one of the variants + variants.append('target') + for variant in variants: + localdata.setVar('CLASSOVERRIDE', 'class-%s' % variant) + fetch_urls(localdata) # Copy local files to target directory and gather any remote files bb_dir = os.path.abspath(os.path.dirname(d.getVar('FILE'))) + os.sep
@@ -341,7 +362,7 @@ def copy_recipe_files(d, tgt_dir, whole_dir=False, download=True): includes = [os.path.abspath(path) for path in d.getVar('BBINCLUDED').split() if os.path.exists(path)] # We also check this below, but we don't want any items in this list being considered remotes includes = [path for path in includes if path.startswith(bb_dir)] - for path in fetch.localpaths() + includes: + for path in localpaths + includes: # Only import files that are under the meta directory if path.startswith(bb_dir): if not whole_dir:
scripts/lib/devtool/upgrade.py +1 -1
@@ -300,7 +300,7 @@ def _create_new_recipe(newpv, md5, sha256, srcrev, srcbranch, workspace, tinfoil bpn = rd.getVar('BPN') path = os.path.join(workspace, 'recipes', bpn) bb.utils.mkdirhier(path) - copied, _ = oe.recipeutils.copy_recipe_files(rd, path) + copied, _ = oe.recipeutils.copy_recipe_files(rd, path, all_variants=True) if not copied: raise DevtoolError('Internal error - no files were copied for recipe %s' % bpn) logger.debug('Copied %s to %s' % (copied, path))
Signature · 1 line
2.9.5
PATCH 5/13 devtool: edit-recipe: fix regression
In reply to Paul EggletonOE-Core commit 5a16b3c804c5eca331a1c08a7ce31a54909af105 attempted to use the same function to get the path to a recipe as the new "find-recipe" command it implemented, except that cannot work because (a) it didn't return anything and (b) event if it had tried, a command function can only return an exit code and we don't want that for find-recipe if it succeeded. Split out a separate reusable function for both commands.
--- scripts/lib/devtool/utilcmds.py | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
scripts/lib/devtool/utilcmds.py +11 -4
@@ -30,14 +30,13 @@ from devtool import parse_recipe logger = logging.getLogger('devtool') -def find_recipe(args, config, basepath, workspace): - """Entry point for the devtool 'find-recipe' subcommand""" +def _find_recipe_path(args, config, basepath, workspace): if args.any_recipe: tinfoil = setup_tinfoil(config_only=False, basepath=basepath) try: rd = parse_recipe(config, tinfoil, args.recipename, True) if not rd: - return 1 + raise DevtoolError("Failed to find specified recipe") recipefile = rd.getVar('FILE') finally: tinfoil.shutdown()
@@ -47,11 +46,19 @@ def find_recipe(args, config, basepath, workspace): if not recipefile: raise DevtoolError("Recipe file for %s is not under the workspace" % args.recipename) + return recipefile + + +def find_recipe(args, config, basepath, workspace): + """Entry point for the devtool 'find-recipe' subcommand""" + recipefile = _find_recipe_path(args, config, basepath, workspace) + print(recipefile) + return 0 def edit_recipe(args, config, basepath, workspace): """Entry point for the devtool 'edit-recipe' subcommand""" - return scriptutils.run_editor(find_recipe(args, config, basepath, workspace), logger) + return scriptutils.run_editor(_find_recipe_path(args, config, basepath, workspace), logger) def configure_help(args, config, basepath, workspace):
Signature · 1 line
2.9.5
PATCH 6/13 oe-selftest: devtool: test find-recipe and edit-recipe
In reply to Paul EggletonWe weren't testing the devtool find-recipe and edit-recipe subcommands, with the result that when they regressed recently we didn't notice. Add some code into the test_devtool_add to test this (since we need a recipe in the workspace, and adding a new test with all that preamble would seem a bit excessive for these simple checks). Also take the opportunity to refactor the test a little bit so that the recipe name and version are variables rather than hardcoding them everywhere.
--- meta/lib/oeqa/selftest/cases/devtool.py | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-)
meta/lib/oeqa/selftest/cases/devtool.py +17 -8
@@ -194,26 +194,35 @@ class DevtoolTests(DevtoolBase): # Fetch source tempdir = tempfile.mkdtemp(prefix='devtoolqa') self.track_for_cleanup(tempdir) + pn = 'pv' + pv = '1.5.3' url = 'http://www.ivarch.com/programs/sources/pv-1.5.3.tar.bz2' result = runCmd('wget %s' % url, cwd=tempdir) - result = runCmd('tar xfv pv-1.5.3.tar.bz2', cwd=tempdir) - srcdir = os.path.join(tempdir, 'pv-1.5.3') + result = runCmd('tar xfv %s' % os.path.basename(url), cwd=tempdir) + srcdir = os.path.join(tempdir, '%s-%s' % (pn, pv)) self.assertTrue(os.path.isfile(os.path.join(srcdir, 'configure')), 'Unable to find configure script in source directory') # Test devtool add self.track_for_cleanup(self.workspacedir) - self.add_command_to_tearDown('bitbake -c cleansstate pv') + self.add_command_to_tearDown('bitbake -c cleansstate %s' % pn) self.add_command_to_tearDown('bitbake-layers remove-layer */workspace') - result = runCmd('devtool add pv %s' % srcdir) + result = runCmd('devtool add %s %s' % (pn, srcdir)) self.assertExists(os.path.join(self.workspacedir, 'conf', 'layer.conf'), 'Workspace directory not created') # Test devtool status result = runCmd('devtool status') - self.assertIn('pv', result.output) + recipepath = '%s/recipes/%s/%s_%s.bb' % (self.workspacedir, pn, pn, pv) + self.assertIn(recipepath, result.output) self.assertIn(srcdir, result.output) + # Test devtool find-recipe + result = runCmd('devtool -q find-recipe %s' % pn) + self.assertEqual(recipepath, result.output.strip()) + # Test devtool edit-recipe + result = runCmd('EDITOR="echo 123" devtool -q edit-recipe %s' % pn) + self.assertEqual('123 %s' % recipepath, result.output.strip()) # Clean up anything in the workdir/sysroot/sstate cache (have to do this *after* devtool add since the recipe only exists then) - bitbake('pv -c cleansstate') + bitbake('%s -c cleansstate' % pn) # Test devtool build - result = runCmd('devtool build pv') - bb_vars = get_bb_vars(['D', 'bindir'], 'pv') + result = runCmd('devtool build %s' % pn) + bb_vars = get_bb_vars(['D', 'bindir'], pn) installdir = bb_vars['D'] self.assertTrue(installdir, 'Could not query installdir variable') bindir = bb_vars['bindir']
Signature · 1 line
2.9.5
PATCH 7/13 devtool: add: add explicit srcrev/branch options
In reply to Paul EggletonAt the moment when fetching source from a git repository you have to know that you can specify the revision and branch in the URL with ';rev=' and ';branch=' respectively, and you can also get thrown off by the shell splitting on the ; character if you forget to surround the URL in quotes. Add explicit -S/--srcrev and -B/--srcbranch options (consistent with devtool upgrade) to make this easier for the user to discover and use. (The rev and branch URL parameters will continue to work, however.)
--- scripts/lib/devtool/standard.py | 17 +++++++++++++++-- scripts/lib/recipetool/create.py | 25 ++++++++++++++++++++++--- 2 files changed, 37 insertions(+), 5 deletions(-)
scripts/lib/devtool/standard.py +15 -2
@@ -1,6 +1,6 @@ # Development tool - standard commands plugin # -# Copyright (C) 2014-2016 Intel Corporation +# Copyright (C) 2014-2017 Intel Corporation # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License version 2 as
@@ -66,6 +66,12 @@ def add(args, config, basepath, workspace): elif os.path.isdir(args.recipename): logger.warn('Ambiguous argument "%s" - assuming you mean it to be the recipe name' % args.recipename) + if not args.fetchuri: + if args.srcrev: + raise DevtoolError('The -S/--srcrev option is only valid when fetching from an SCM repository') + if args.srcbranch: + raise DevtoolError('The -B/--srcbranch option is only valid when fetching from an SCM repository') + if args.srctree and os.path.isfile(args.srctree): args.fetchuri = 'file://' + os.path.abspath(args.srctree) args.srctree = ''
@@ -151,6 +157,10 @@ def add(args, config, basepath, workspace): extracmdopts += ' --fetch-dev' if args.mirrors: extracmdopts += ' --mirrors' + if args.srcrev: + extracmdopts += ' --srcrev %s' % args.srcrev + if args.srcbranch: + extracmdopts += ' --srcbranch %s' % args.srcbranch tempdir = tempfile.mkdtemp(prefix='devtool') try:
@@ -1809,7 +1819,10 @@ def register_commands(subparsers, context): parser_add.add_argument('--fetch-dev', help='For npm, also fetch devDependencies', action="store_true") parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)') parser_add.add_argument('--no-git', '-g', help='If fetching source, do not set up source tree as a git repository', action="store_true") - parser_add.add_argument('--autorev', '-a', help='When fetching from a git repository, set SRCREV in the recipe to a floating revision instead of fixed', action="store_true") + group = parser_add.add_mutually_exclusive_group() + group.add_argument('--srcrev', '-S', help='Source revision to fetch if fetching from an SCM such as git (default latest)') + group.add_argument('--autorev', '-a', help='When fetching from a git repository, set SRCREV in the recipe to a floating revision instead of fixed', action="store_true") + parser_add.add_argument('--srcbranch', '-B', help='Branch in source repository if fetching from an SCM such as git (default master)') parser_add.add_argument('--binary', '-b', help='Treat the source tree as something that should be installed verbatim (no compilation, same directory structure). Useful with binary packages e.g. RPMs.', action='store_true') parser_add.add_argument('--also-native', help='Also add native variant (i.e. support building recipe for the build host as well as the target machine)', action='store_true') parser_add.add_argument('--src-subdir', help='Specify subdirectory within source tree to use', metavar='SUBDIR')
scripts/lib/recipetool/create.py +22 -3
@@ -1,6 +1,6 @@ # Recipe creation tool - create command plugin # -# Copyright (C) 2014-2016 Intel Corporation +# Copyright (C) 2014-2017 Intel Corporation # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License version 2 as
@@ -440,14 +440,30 @@ def create_recipe(args): rev_re = re.compile(';rev=([^;]+)') res = rev_re.search(srcuri) if res: + if args.srcrev: + logger.error('rev= parameter and -S/--srcrev option cannot both be specified - use one or the other') + sys.exit(1) + if args.autorev: + logger.error('rev= parameter and -a/--autorev option cannot both be specified - use one or the other') + sys.exit(1) srcrev = res.group(1) srcuri = rev_re.sub('', srcuri) + elif args.srcrev: + srcrev = args.srcrev # Check whether users provides any branch info in fetchuri. # If true, we will skip all branch checking process to honor all user's input. scheme, network, path, user, passwd, params = bb.fetch2.decodeurl(fetchuri) srcbranch = params.get('branch') + if args.srcbranch: + if srcbranch: + logger.error('branch= parameter and -B/--srcbranch option cannot both be specified - use one or the other') + sys.exit(1) + srcbranch = args.srcbranch nobranch = params.get('nobranch') + if nobranch and srcbranch: + logger.error('nobranch= cannot be used if you specify a branch') + sys.exit(1) tag = params.get('tag') if not srcbranch and not nobranch and srcrev != '${AUTOREV}': # Append nobranch=1 in the following conditions:
@@ -523,7 +539,7 @@ def create_recipe(args): else: # If get_branch contains more than one objects, then display error and exit. mbrch = '\n ' + '\n '.join(get_branch) - logger.error('Revision %s was found on multiple branches: %s\nPlease provide the correct branch in the source URL with ;branch=<branch> (and ensure you use quotes around the URL to avoid the shell interpreting the ";")' % (srcrev, mbrch)) + logger.error('Revision %s was found on multiple branches: %s\nPlease provide the correct branch with -B/--srcbranch' % (srcrev, mbrch)) sys.exit(1) # Since we might have a value in srcbranch, we need to
@@ -1277,7 +1293,10 @@ def register_commands(subparsers): parser_create.add_argument('-b', '--binary', help='Treat the source tree as something that should be installed verbatim (no compilation, same directory structure)', action='store_true') parser_create.add_argument('--also-native', help='Also add native variant (i.e. support building recipe for the build host as well as the target machine)', action='store_true') parser_create.add_argument('--src-subdir', help='Specify subdirectory within source tree to use', metavar='SUBDIR') - parser_create.add_argument('-a', '--autorev', help='When fetching from a git repository, set SRCREV in the recipe to a floating revision instead of fixed', action="store_true") + group = parser_create.add_mutually_exclusive_group() + group.add_argument('-a', '--autorev', help='When fetching from a git repository, set SRCREV in the recipe to a floating revision instead of fixed', action="store_true") + group.add_argument('-S', '--srcrev', help='Source revision to fetch if fetching from an SCM such as git (default latest)') + parser_create.add_argument('-B', '--srcbranch', help='Branch in source repository if fetching from an SCM such as git (default master)') parser_create.add_argument('--keep-temp', action="store_true", help='Keep temporary directory (for debugging)') parser_create.add_argument('--fetch-dev', action="store_true", help='For npm, also fetch devDependencies') parser_create.add_argument('--devtool', action="store_true", help=argparse.SUPPRESS)
Signature · 1 line
2.9.5
PATCH 8/13 recipetool: create: make recently added branch/tag handling git specific
In reply to Paul EggletonThe branch and tag handling code that was recently added in OE-Core revs ecca596b75cfda2f798a0bdde75f4f774e23a95b and 3afdcbdc9a3e65bc925ec61717784ffec67d529d is specific to git, so only apply it when we're fetching from a git URL.
--- scripts/lib/recipetool/create.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/lib/recipetool/create.py +2 -2
@@ -518,7 +518,7 @@ def create_recipe(args): # We need this checking mechanism to improve the recipe created by recipetool and devtool # is able to parse and build by bitbake. # If there is no input for branch name, then check for branch name with SRCREV provided. - if not srcbranch and not nobranch and srcrev and (srcrev != '${AUTOREV}'): + if not srcbranch and not nobranch and srcrev and (srcrev != '${AUTOREV}') and scheme in ['git', 'gitsm']: try: cmd = 'git branch -r --contains' check_branch, check_branch_err = bb.process.run('%s %s' % (cmd, srcrev), cwd=srctree)
@@ -549,7 +549,7 @@ def create_recipe(args): params['branch'] = srcbranch srcuri = bb.fetch2.encodeurl((scheme, network, path, user, passwd, params)) - if storeTagName: + if storeTagName and scheme in ['git', 'gitsm']: # Re-introduced tag variable from storeTagName # Check srcrev using tag and check validity of the tag cmd = ('git rev-parse --verify %s' % (storeTagName))
Signature · 1 line
2.9.5
PATCH 9/13 recipetool: create: fix SRCPV prefix for non-git SCMs
In reply to Paul EggletonIf you're fetching from an SCM other than git (for example subversion or mercurial) then we need to use a different prefix for the SRCPV in PV instead of +git.
--- scripts/lib/recipetool/create.py | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
scripts/lib/recipetool/create.py +9 -1
@@ -423,6 +423,7 @@ def create_recipe(args): srcsubdir = '' srcrev = '${AUTOREV}' srcbranch = '' + scheme = '' storeTagName = '' pv_srcpv = False
@@ -682,7 +683,14 @@ def create_recipe(args): lines_before.append('') lines_before.append('# Modify these as desired') # Note: we have code to replace realpv further down if it gets set to some other value - lines_before.append('PV = "%s+git${SRCPV}"' % (realpv or '1.0')) + scheme, _, _, _, _, _ = bb.fetch2.decodeurl(srcuri) + if scheme in ['git', 'gitsm']: + srcpvprefix = 'git' + elif scheme == 'svn': + srcpvprefix = 'svnr' + else: + srcpvprefix = scheme + lines_before.append('PV = "%s+%s${SRCPV}"' % (realpv or '1.0', srcpvprefix)) pv_srcpv = True if not args.autorev and srcrev == '${AUTOREV}': if os.path.exists(os.path.join(srctree, '.git')):
Signature · 1 line
2.9.5
PATCH 10/13 recipetool: create: suppress npm shrinkwrap/lockdown warnings again
In reply to Paul EggletonSince OE-Core revision 9a47a6690052ef943c0d4760630ee630fb012153 the mechanism we were using to suppress the warnings about NPM_LOCKDOWN and NPM_SHRINKWRAP not being set on the first fetch of the source is no longer available since we are using the normal fetch/unpack tasks to do the job. Use the newly added noverify parameter to suppress the warnings again.
--- scripts/lib/recipetool/create.py | 2 ++ 1 file changed, 2 insertions(+)
scripts/lib/recipetool/create.py +2 -0
@@ -479,6 +479,8 @@ def create_recipe(args): storeTagName = params['tag'] params['nobranch'] = '1' del params['tag'] + if scheme == 'npm': + params['noverify'] = '1' fetchuri = bb.fetch2.encodeurl((scheme, network, path, user, passwd, params)) tmpparent = tinfoil.config_data.getVar('BASE_WORKDIR')
Signature · 1 line
2.9.5
PATCH 11/13 recipetool: create: detect Eclipse licenses
In reply to Paul EggletonAdd detection of EPL 1.0 and EDL 1.0 license files.
--- scripts/lib/recipetool/create.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
scripts/lib/recipetool/create.py +5 -1
@@ -1099,6 +1099,10 @@ def crunch_license(licfile): crunched_md5sums['1daebd9491d1e8426900b4fa5a422814'] = 'LGPLv2.1' # https://github.com/FFmpeg/FFmpeg/blob/master/COPYING.LGPLv3 crunched_md5sums['2ebfb3bb49b9a48a075cc1425e7f4129'] = 'LGPLv3' + # https://raw.githubusercontent.com/eclipse/mosquitto/v1.4.14/epl-v10 + crunched_md5sums['efe2cb9a35826992b9df68224e3c2628'] = 'EPL-1.0' + # https://raw.githubusercontent.com/eclipse/mosquitto/v1.4.14/edl-v10 + crunched_md5sums['0a9c78c0a398d1bbce4a166757d60387'] = 'EDL-1.0' lictext = [] with open(licfile, 'r', errors='surrogateescape') as f: for line in f:
@@ -1131,7 +1135,7 @@ def guess_license(srctree, d): md5sums = get_license_md5sums(d) licenses = [] - licspecs = ['*LICEN[CS]E*', 'COPYING*', '*[Ll]icense*', 'LEGAL*', '[Ll]egal*', '*GPL*', 'README.lic*', 'COPYRIGHT*', '[Cc]opyright*'] + licspecs = ['*LICEN[CS]E*', 'COPYING*', '*[Ll]icense*', 'LEGAL*', '[Ll]egal*', '*GPL*', 'README.lic*', 'COPYRIGHT*', '[Cc]opyright*', 'e[dp]l-v10'] licfiles = [] for root, dirs, files in os.walk(srctree): for fn in files:
Signature · 1 line
2.9.5
PATCH 12/13 oe-selftest: tinfoil: add a test for variable history
In reply to Paul EggletonI recently found that variable history wasn't working properly for recipes when we enable history tracking, resulting in minor functionality loss in devtool upgrade, so add a test to ensure this doesn't regress now that it's fixed.
--- meta/lib/oeqa/selftest/cases/tinfoil.py | 34 +++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)
meta/lib/oeqa/selftest/cases/tinfoil.py +34 -0
@@ -195,3 +195,37 @@ class TinfoilTests(OESelftestTestCase): tinfoil.config_data.appendVar('OVERRIDES', ':overrideone') value = tinfoil.config_data.getVar('TESTVAR') self.assertEqual(value, 'one', 'Variable overrides not functioning correctly') + + def test_variable_history(self): + # Basic test to ensure that variable history works when tracking=True + with bb.tinfoil.Tinfoil(tracking=True) as tinfoil: + tinfoil.prepare(config_only=False, quiet=2) + # Note that _tracking for any datastore we get will be + # false here, that's currently expected - so we can't check + # for that + history = tinfoil.config_data.varhistory.variable('DL_DIR') + for entry in history: + if entry['file'].endswith('/bitbake.conf'): + if entry['op'] in ['set', 'set?']: + break + else: + self.fail('Did not find history entry setting DL_DIR in bitbake.conf. History: %s' % history) + # Check it works for recipes as well + testrecipe = 'zlib' + rd = tinfoil.parse_recipe(testrecipe) + history = rd.varhistory.variable('LICENSE') + bbfound = -1 + recipefound = -1 + for i, entry in enumerate(history): + if entry['file'].endswith('/bitbake.conf'): + if entry['detail'] == 'INVALID' and entry['op'] in ['set', 'set?']: + bbfound = i + elif entry['file'].endswith('.bb'): + if entry['op'] == 'set': + recipefound = i + if bbfound == -1: + self.fail('Did not find history entry setting LICENSE in bitbake.conf parsing %s recipe. History: %s' % (testrecipe, history)) + if recipefound == -1: + self.fail('Did not find history entry setting LICENSE in %s recipe. History: %s' % (testrecipe, history)) + if bbfound > recipefound: + self.fail('History entry setting LICENSE in %s recipe and in bitbake.conf in wrong order. History: %s' % (testrecipe, history))
Signature · 1 line
2.9.5
PATCH 13/13 scriptutils: fix fetch_url() to use lowercase dummy recipe name
In reply to Paul Eggletonrecipetool create (and hence devtool add) and devtool upgrade use fetch_url() which creates a dummy recipe in order to fetch source. Previously the random part of the name was using uppercase characters, and this triggers a QA warning after OE-Core commit 4713f8b2c4f2c74239d284adcf1e59e61aa66576, so use lowercase instead as I really should have in the first place.
--- scripts/lib/scriptutils.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/lib/scriptutils.py +1 -1
@@ -134,7 +134,7 @@ def fetch_url(tinfoil, srcuri, srcrev, destdir, logger, preserve_tmp=False, mirr # for do_fetch and do_unpack # I'd use tempfile functions here but underscores can be produced by that and those # aren't allowed in recipe file names except to separate the version - rndstring = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(8)) + rndstring = ''.join(random.choice(string.ascii_lowercase + string.digits) for _ in range(8)) fetchrecipe = os.path.join(fetchrecipedir, 'tmp-recipetool-%s.bb' % rndstring) fetchrecipepn = os.path.splitext(os.path.basename(fetchrecipe))[0] logger.debug('Generating initial recipe %s for fetching' % fetchrecipe)
Signature · 1 line
2.9.5
Re: [PATCH 03/13] devtool: upgrade: check that user has configured git properly
In reply to Paul EggletonQuoted 10 lines · Paul Eggleton
From: paul <[email protected]>
If user.name or user.email haven't been set then git rebase can't really work properly. Check that the user has set these and error out if not. (Elsewhere we are relying on OE's git patch functionality which forces a dummy OE value - that's OK there as it's completely under OE's control and therefore it's OK for a dummy OE user to be the committer, but here the rebase may require intervention so it's reasonable to have the user's actual name and email on the operation.)
The irony is strong in this one.
(I'll fixup when merging)
Ross
Re: [PATCH 03/13] devtool: upgrade: check that user has configured git properly
Quoted 14 lines · Ross
Quoted 10 lines · Paul Eggleton
From: paul <[email protected]>
If user.name or user.email haven't been set then git rebase can't really work properly. Check that the user has set these and error out if not. (Elsewhere we are relying on OE's git patch functionality which forces a dummy OE value - that's OK there as it's completely under OE's control and therefore it's OK for a dummy OE user to be the committer, but here the rebase may require intervention so it's reasonable to have the user's actual name and email on the operation.)
Quoted 3 lines
The irony is strong in this one.
(I'll fixup when merging)
Shoot. Of course I committed the change just after I'd cleared out the settings to test the darn thing ;)
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre