Skip to contentSkip to conversation

devtool / recipetool fixes

PATCH In review Paul Eggleton 2 participants 16 messages Aug 30, 2017 – Aug 31, 2017

lore
Share & tools
lore
#0 lore raw

cover devtool / recipetool fixes

Some fixes for devtool / recipetool issues, along with a couple of related changes to the oe-selftest tests. I have run the oe-selftest tests for devtool and recipetool and they pass with these changes applied.

NOTE: this patchset depends upon the bitbake patchset I just sent to the bitbake-devel list.

The following changes since commit 62f1122ef166eba56441d669c6b3b3fe5f367418:

bluez5: cosmetic fixes to bluetooth bootscript (2017-08-30 11:13:52 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib paule/devtool29-oe
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=paule/devtool29-oe

Paul Eggleton (12):
devtool: update-recipe: ensure patches get deleted in srcrev mode
devtool: upgrade: fix handling of non-absolute paths
devtool: upgrade: workaround for recipes which apply patches conditional upon class
devtool: edit-recipe: fix regression
oe-selftest: devtool: test find-recipe and edit-recipe
devtool: add: add explicit srcrev/branch options
recipetool: create: make recently added branch/tag handling git specific
recipetool: create: fix SRCPV prefix for non-git SCMs
recipetool: create: suppress npm shrinkwrap/lockdown warnings again
recipetool: create: detect Eclipse licenses
oe-selftest: tinfoil: add a test for variable history
scriptutils: fix fetch_url() to use lowercase dummy recipe name

paul (1):
devtool: upgrade: check that user has configured git properly

 meta/lib/oe/recipeutils.py              | 43 +++++++++++++++++++++++-------
 meta/lib/oeqa/selftest/cases/devtool.py | 25 ++++++++++++------
 meta/lib/oeqa/selftest/cases/tinfoil.py | 34 ++++++++++++++++++++++++
 scripts/lib/devtool/standard.py         | 20 +++++++++++---
 scripts/lib/devtool/upgrade.py          | 32 ++++++++++++++++++++--
 scripts/lib/devtool/utilcmds.py         | 15 ++++++++---
 scripts/lib/recipetool/create.py        | 47 ++++++++++++++++++++++++++++-----
 scripts/lib/scriptutils.py              |  2 +-
 8 files changed, 183 insertions(+), 35 deletions(-)
Signature · 1 line
2.9.5
#1 lore raw

PATCH 1/13 devtool: update-recipe: ensure patches get deleted in srcrev mode

Patches that we identify as having been "deleted" (i.e. patches in SRC_URI that no longer appear in the git tree) need to be dropped even if we're updating in srcrev mode. This fixes the case where HEAD of the git tree is valid upstream (i.e. no extra commits), but there are patches left over in the recipe, e.g. when we do devtool upgrade and then all of the commits rebased on top of the new branch get skipped.

Fixes [YOCTO #11972].

---
 scripts/lib/devtool/standard.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
scripts/lib/devtool/standard.py +2 -1
index fa9d347..d79ebf1 100644
@@ -1352,9 +1352,10 @@ def _update_recipe_srcrev(srctree, rd, appendlayerdir, wildcard_version, no_remo
             old_srcrev = (rd.getVar('SRCREV', False) or '')
             upd_p, new_p, del_p = _export_patches(srctree, rd, old_srcrev,
                                                   patches_dir)
+            logger.debug('Patches: update %s, new %s, delete %s' % (dict(upd_p), dict(new_p), dict(del_p)))
 
             # Remove deleted local files and "overlapping" patches
-            remove_files = list(del_f.values()) + list(upd_p.values())
+            remove_files = list(del_f.values()) + list(upd_p.values()) + list(del_p.values())
             if remove_files:
                 removedentries = _remove_file_entries(srcuri, remove_files)[0]
                 update_srcuri = True
Signature · 1 line
2.9.5
#2 lore raw

PATCH 2/13 devtool: upgrade: fix handling of non-absolute paths

In reply to Paul Eggleton

If your BBLAYERS has non-absolute paths in it (e.g. "${COREBASE}/../something") then none of the paths matched in copy_recipe_files() with the result that no files got copied and you ended up with an error later on because the recipe file couldn't be found at the destination. Fix this as well as adding an explicit check to see if no files got copied - error out earlier if so.

Fixes [YOCTO #10981].

---
 meta/lib/oe/recipeutils.py     | 8 +++++---
 scripts/lib/devtool/upgrade.py | 3 +++
 2 files changed, 8 insertions(+), 3 deletions(-)
meta/lib/oe/recipeutils.py +5 -3
index a7fdd36..b946128 100644
@@ -334,11 +334,13 @@ def copy_recipe_files(d, tgt_dir, whole_dir=False, download=True):
         fetch.download()
 
     # Copy local files to target directory and gather any remote files
-    bb_dir = os.path.dirname(d.getVar('FILE')) + os.sep
+    bb_dir = os.path.abspath(os.path.dirname(d.getVar('FILE'))) + os.sep
     remotes = []
     copied = []
-    includes = [path for path in d.getVar('BBINCLUDED').split() if
-                path.startswith(bb_dir) and os.path.exists(path)]
+    # Need to do this in two steps since we want to check against the absolute path
+    includes = [os.path.abspath(path) for path in d.getVar('BBINCLUDED').split() if os.path.exists(path)]
+    # We also check this below, but we don't want any items in this list being considered remotes
+    includes = [path for path in includes if path.startswith(bb_dir)]
     for path in fetch.localpaths() + includes:
         # Only import files that are under the meta directory
         if path.startswith(bb_dir):
scripts/lib/devtool/upgrade.py +3 -0
index 1f11d47..24937dc 100644
@@ -301,6 +301,9 @@ def _create_new_recipe(newpv, md5, sha256, srcrev, srcbranch, workspace, tinfoil
     path = os.path.join(workspace, 'recipes', bpn)
     bb.utils.mkdirhier(path)
     copied, _ = oe.recipeutils.copy_recipe_files(rd, path)
+    if not copied:
+        raise DevtoolError('Internal error - no files were copied for recipe %s' % bpn)
+    logger.debug('Copied %s to %s' % (copied, path))
 
     oldpv = rd.getVar('PV')
     if not newpv:
Signature · 1 line
2.9.5
#3 lore raw

PATCH 3/13 devtool: upgrade: check that user has configured git properly

In reply to Paul Eggleton

From: paul <[email protected]>

If user.name or user.email haven't been set then git rebase can't really work properly. Check that the user has set these and error out if not. (Elsewhere we are relying on OE's git patch functionality which forces a dummy OE value - that's OK there as it's completely under OE's control and therefore it's OK for a dummy OE user to be the committer, but here the rebase may require intervention so it's reasonable to have the user's actual name and email on the operation.)

Fixes [YOCTO #11947].

---
 scripts/lib/devtool/upgrade.py | 27 ++++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)
scripts/lib/devtool/upgrade.py +26 -1
index 24937dc..e334d2b 100644
@@ -1,6 +1,6 @@
 # Development tool - upgrade command plugin
 #
-# Copyright (C) 2014-2015 Intel Corporation
+# Copyright (C) 2014-2017 Intel Corporation
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 2 as
@@ -355,6 +355,29 @@ def _create_new_recipe(newpv, md5, sha256, srcrev, srcbranch, workspace, tinfoil
 
     return fullpath, copied
 
+
+def _check_git_config():
+    def getconfig(name):
+        try:
+            value = bb.process.run('git config --global %s' % name)[0].strip()
+        except bb.process.ExecutionError as e:
+            if e.exitcode == 1:
+                value = None
+            else:
+                raise
+        return value
+
+    username = getconfig('user.name')
+    useremail = getconfig('user.email')
+    configerr = []
+    if not username:
+        configerr.append('Please set your name using:\n  git config --global user.name')
+    if not useremail:
+        configerr.append('Please set your email using:\n  git config --global user.email')
+    if configerr:
+        raise DevtoolError('Your git configuration is incomplete which will prevent rebases from working:\n' + '\n'.join(configerr))
+
+
 def upgrade(args, config, basepath, workspace):
     """Entry point for the devtool 'upgrade' subcommand"""
 
@@ -365,6 +388,8 @@ def upgrade(args, config, basepath, workspace):
     if args.srcbranch and not args.srcrev:
         raise DevtoolError("If you specify --srcbranch/-B then you must use --srcrev/-S to specify the revision" % args.recipename)
 
+    _check_git_config()
+
     tinfoil = setup_tinfoil(basepath=basepath, tracking=True)
     try:
         rd = parse_recipe(config, tinfoil, args.recipename, True)
Signature · 1 line
2.9.5
#4 lore raw

Re: [PATCH 03/13] devtool: upgrade: check that user has configured git properly

Quoted 10 lines · Paul Eggleton

From: paul <[email protected]>

If user.name or user.email haven't been set then git rebase can't really work properly. Check that the user has set these and error out if not. (Elsewhere we are relying on OE's git patch functionality which forces a dummy OE value - that's OK there as it's completely under OE's control and therefore it's OK for a dummy OE user to be the committer, but here the rebase may require intervention so it's reasonable to have the user's actual name and email on the operation.)

The irony is strong in this one.

(I'll fixup when merging)

Ross

#5 lore raw

Re: [PATCH 03/13] devtool: upgrade: check that user has configured git properly

Quoted 14 lines · Ross
Quoted 10 lines · Paul Eggleton

From: paul <[email protected]>

If user.name or user.email haven't been set then git rebase can't really work properly. Check that the user has set these and error out if not. (Elsewhere we are relying on OE's git patch functionality which forces a dummy OE value - that's OK there as it's completely under OE's control and therefore it's OK for a dummy OE user to be the committer, but here the rebase may require intervention so it's reasonable to have the user's actual name and email on the operation.)

Quoted 3 lines

The irony is strong in this one.

(I'll fixup when merging)

Shoot. Of course I committed the change just after I'd cleared out the settings to test the darn thing ;)

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#6 lore raw

PATCH 4/13 devtool: upgrade: workaround for recipes which apply patches conditional upon class

In reply to Paul Eggleton

If we're upgrading a recipe that appends additional patches for, say, class-native, and we're just upgrading the target variant, then when we copied the recipe into the workspace we skipped copying the additional patches for the native variant. This caused warnings because the workspace recipe is preferred. Look at SRC_URI for all variants when copying files to work around this.

More work is needed to make it easier to work with recipes that use BBCLASSEXTEND where you need to build more than one variant at once, but this at least fixes the immediate ugliness.

---
 meta/lib/oe/recipeutils.py     | 35 ++++++++++++++++++++++++++++-------
 scripts/lib/devtool/upgrade.py |  2 +-
 2 files changed, 29 insertions(+), 8 deletions(-)
meta/lib/oe/recipeutils.py +28 -7
index b946128..c8570ac 100644
@@ -2,7 +2,7 @@
 #
 # Some code borrowed from the OE layer index
 #
-# Copyright (C) 2013-2016 Intel Corporation
+# Copyright (C) 2013-2017 Intel Corporation
 #
 
 import sys
@@ -320,7 +320,7 @@ def patch_recipe(d, fn, varvalues, patch=False, relpath=''):
 
 
 
-def copy_recipe_files(d, tgt_dir, whole_dir=False, download=True):
+def copy_recipe_files(d, tgt_dir, whole_dir=False, download=True, all_variants=False):
     """Copy (local) recipe files, including both files included via include/require,
     and files referred to in the SRC_URI variable."""
     import bb.fetch2
@@ -328,10 +328,31 @@ def copy_recipe_files(d, tgt_dir, whole_dir=False, download=True):
 
     # FIXME need a warning if the unexpanded SRC_URI value contains variable references
 
-    uris = (d.getVar('SRC_URI') or "").split()
-    fetch = bb.fetch2.Fetch(uris, d)
-    if download:
-        fetch.download()
+    uri_values = []
+    localpaths = []
+    def fetch_urls(rdata):
+        # Collect the local paths from SRC_URI
+        srcuri = rdata.getVar('SRC_URI') or ""
+        if srcuri not in uri_values:
+            fetch = bb.fetch2.Fetch(srcuri.split(), rdata)
+            if download:
+                fetch.download()
+            for pth in fetch.localpaths():
+                if pth not in localpaths:
+                    localpaths.append(pth)
+            uri_values.append(srcuri)
+
+    fetch_urls(d)
+    if all_variants:
+        # Get files for other variants e.g. in the case of a SRC_URI_append
+        localdata = bb.data.createCopy(d)
+        variants = (localdata.getVar('BBCLASSEXTEND') or '').split()
+        if variants:
+            # Ensure we handle class-target if we're dealing with one of the variants
+            variants.append('target')
+            for variant in variants:
+                localdata.setVar('CLASSOVERRIDE', 'class-%s' % variant)
+                fetch_urls(localdata)
 
     # Copy local files to target directory and gather any remote files
     bb_dir = os.path.abspath(os.path.dirname(d.getVar('FILE'))) + os.sep
@@ -341,7 +362,7 @@ def copy_recipe_files(d, tgt_dir, whole_dir=False, download=True):
     includes = [os.path.abspath(path) for path in d.getVar('BBINCLUDED').split() if os.path.exists(path)]
     # We also check this below, but we don't want any items in this list being considered remotes
     includes = [path for path in includes if path.startswith(bb_dir)]
-    for path in fetch.localpaths() + includes:
+    for path in localpaths + includes:
         # Only import files that are under the meta directory
         if path.startswith(bb_dir):
             if not whole_dir:
scripts/lib/devtool/upgrade.py +1 -1
index e334d2b..41bd34e 100644
@@ -300,7 +300,7 @@ def _create_new_recipe(newpv, md5, sha256, srcrev, srcbranch, workspace, tinfoil
     bpn = rd.getVar('BPN')
     path = os.path.join(workspace, 'recipes', bpn)
     bb.utils.mkdirhier(path)
-    copied, _ = oe.recipeutils.copy_recipe_files(rd, path)
+    copied, _ = oe.recipeutils.copy_recipe_files(rd, path, all_variants=True)
     if not copied:
         raise DevtoolError('Internal error - no files were copied for recipe %s' % bpn)
     logger.debug('Copied %s to %s' % (copied, path))
Signature · 1 line
2.9.5
#7 lore raw

PATCH 5/13 devtool: edit-recipe: fix regression

In reply to Paul Eggleton

OE-Core commit 5a16b3c804c5eca331a1c08a7ce31a54909af105 attempted to use the same function to get the path to a recipe as the new "find-recipe" command it implemented, except that cannot work because (a) it didn't return anything and (b) event if it had tried, a command function can only return an exit code and we don't want that for find-recipe if it succeeded. Split out a separate reusable function for both commands.

---
 scripts/lib/devtool/utilcmds.py | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
scripts/lib/devtool/utilcmds.py +11 -4
index c8cb760..b745116 100644
@@ -30,14 +30,13 @@ from devtool import parse_recipe
 
 logger = logging.getLogger('devtool')
 
-def find_recipe(args, config, basepath, workspace):
-    """Entry point for the devtool 'find-recipe' subcommand"""
+def _find_recipe_path(args, config, basepath, workspace):
     if args.any_recipe:
         tinfoil = setup_tinfoil(config_only=False, basepath=basepath)
         try:
             rd = parse_recipe(config, tinfoil, args.recipename, True)
             if not rd:
-                return 1
+                raise DevtoolError("Failed to find specified recipe")
             recipefile = rd.getVar('FILE')
         finally:
             tinfoil.shutdown()
@@ -47,11 +46,19 @@ def find_recipe(args, config, basepath, workspace):
         if not recipefile:
             raise DevtoolError("Recipe file for %s is not under the workspace" %
                                args.recipename)
+    return recipefile
+
+
+def find_recipe(args, config, basepath, workspace):
+    """Entry point for the devtool 'find-recipe' subcommand"""
+    recipefile = _find_recipe_path(args, config, basepath, workspace)
+    print(recipefile)
+    return 0
 
 
 def edit_recipe(args, config, basepath, workspace):
     """Entry point for the devtool 'edit-recipe' subcommand"""
-    return scriptutils.run_editor(find_recipe(args, config, basepath, workspace), logger)
+    return scriptutils.run_editor(_find_recipe_path(args, config, basepath, workspace), logger)
 
 
 def configure_help(args, config, basepath, workspace):
Signature · 1 line
2.9.5
#8 lore raw

PATCH 6/13 oe-selftest: devtool: test find-recipe and edit-recipe

In reply to Paul Eggleton

We weren't testing the devtool find-recipe and edit-recipe subcommands, with the result that when they regressed recently we didn't notice. Add some code into the test_devtool_add to test this (since we need a recipe in the workspace, and adding a new test with all that preamble would seem a bit excessive for these simple checks). Also take the opportunity to refactor the test a little bit so that the recipe name and version are variables rather than hardcoding them everywhere.

---
 meta/lib/oeqa/selftest/cases/devtool.py | 25 +++++++++++++++++--------
 1 file changed, 17 insertions(+), 8 deletions(-)
meta/lib/oeqa/selftest/cases/devtool.py +17 -8
index 88d6972..c17131a 100644
@@ -194,26 +194,35 @@ class DevtoolTests(DevtoolBase):
         # Fetch source
         tempdir = tempfile.mkdtemp(prefix='devtoolqa')
         self.track_for_cleanup(tempdir)
+        pn = 'pv'
+        pv = '1.5.3'
         url = 'http://www.ivarch.com/programs/sources/pv-1.5.3.tar.bz2'
         result = runCmd('wget %s' % url, cwd=tempdir)
-        result = runCmd('tar xfv pv-1.5.3.tar.bz2', cwd=tempdir)
-        srcdir = os.path.join(tempdir, 'pv-1.5.3')
+        result = runCmd('tar xfv %s' % os.path.basename(url), cwd=tempdir)
+        srcdir = os.path.join(tempdir, '%s-%s' % (pn, pv))
         self.assertTrue(os.path.isfile(os.path.join(srcdir, 'configure')), 'Unable to find configure script in source directory')
         # Test devtool add
         self.track_for_cleanup(self.workspacedir)
-        self.add_command_to_tearDown('bitbake -c cleansstate pv')
+        self.add_command_to_tearDown('bitbake -c cleansstate %s' % pn)
         self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
-        result = runCmd('devtool add pv %s' % srcdir)
+        result = runCmd('devtool add %s %s' % (pn, srcdir))
         self.assertExists(os.path.join(self.workspacedir, 'conf', 'layer.conf'), 'Workspace directory not created')
         # Test devtool status
         result = runCmd('devtool status')
-        self.assertIn('pv', result.output)
+        recipepath = '%s/recipes/%s/%s_%s.bb' % (self.workspacedir, pn, pn, pv)
+        self.assertIn(recipepath, result.output)
         self.assertIn(srcdir, result.output)
+        # Test devtool find-recipe
+        result = runCmd('devtool -q find-recipe %s' % pn)
+        self.assertEqual(recipepath, result.output.strip())
+        # Test devtool edit-recipe
+        result = runCmd('EDITOR="echo 123" devtool -q edit-recipe %s' % pn)
+        self.assertEqual('123 %s' % recipepath, result.output.strip())
         # Clean up anything in the workdir/sysroot/sstate cache (have to do this *after* devtool add since the recipe only exists then)
-        bitbake('pv -c cleansstate')
+        bitbake('%s -c cleansstate' % pn)
         # Test devtool build
-        result = runCmd('devtool build pv')
-        bb_vars = get_bb_vars(['D', 'bindir'], 'pv')
+        result = runCmd('devtool build %s' % pn)
+        bb_vars = get_bb_vars(['D', 'bindir'], pn)
         installdir = bb_vars['D']
         self.assertTrue(installdir, 'Could not query installdir variable')
         bindir = bb_vars['bindir']
Signature · 1 line
2.9.5
#9 lore raw

PATCH 7/13 devtool: add: add explicit srcrev/branch options

In reply to Paul Eggleton

At the moment when fetching source from a git repository you have to know that you can specify the revision and branch in the URL with ';rev=' and ';branch=' respectively, and you can also get thrown off by the shell splitting on the ; character if you forget to surround the URL in quotes. Add explicit -S/--srcrev and -B/--srcbranch options (consistent with devtool upgrade) to make this easier for the user to discover and use. (The rev and branch URL parameters will continue to work, however.)

---
 scripts/lib/devtool/standard.py  | 17 +++++++++++++++--
 scripts/lib/recipetool/create.py | 25 ++++++++++++++++++++++---
 2 files changed, 37 insertions(+), 5 deletions(-)
scripts/lib/devtool/standard.py +15 -2
index d79ebf1..174cc86 100644
@@ -1,6 +1,6 @@
 # Development tool - standard commands plugin
 #
-# Copyright (C) 2014-2016 Intel Corporation
+# Copyright (C) 2014-2017 Intel Corporation
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 2 as
@@ -66,6 +66,12 @@ def add(args, config, basepath, workspace):
         elif os.path.isdir(args.recipename):
             logger.warn('Ambiguous argument "%s" - assuming you mean it to be the recipe name' % args.recipename)
 
+    if not args.fetchuri:
+        if args.srcrev:
+            raise DevtoolError('The -S/--srcrev option is only valid when fetching from an SCM repository')
+        if args.srcbranch:
+            raise DevtoolError('The -B/--srcbranch option is only valid when fetching from an SCM repository')
+
     if args.srctree and os.path.isfile(args.srctree):
         args.fetchuri = 'file://' + os.path.abspath(args.srctree)
         args.srctree = ''
@@ -151,6 +157,10 @@ def add(args, config, basepath, workspace):
         extracmdopts += ' --fetch-dev'
     if args.mirrors:
         extracmdopts += ' --mirrors'
+    if args.srcrev:
+        extracmdopts += ' --srcrev %s' % args.srcrev
+    if args.srcbranch:
+        extracmdopts += ' --srcbranch %s' % args.srcbranch
 
     tempdir = tempfile.mkdtemp(prefix='devtool')
     try:
@@ -1809,7 +1819,10 @@ def register_commands(subparsers, context):
     parser_add.add_argument('--fetch-dev', help='For npm, also fetch devDependencies', action="store_true")
     parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)')
     parser_add.add_argument('--no-git', '-g', help='If fetching source, do not set up source tree as a git repository', action="store_true")
-    parser_add.add_argument('--autorev', '-a', help='When fetching from a git repository, set SRCREV in the recipe to a floating revision instead of fixed', action="store_true")
+    group = parser_add.add_mutually_exclusive_group()
+    group.add_argument('--srcrev', '-S', help='Source revision to fetch if fetching from an SCM such as git (default latest)')
+    group.add_argument('--autorev', '-a', help='When fetching from a git repository, set SRCREV in the recipe to a floating revision instead of fixed', action="store_true")
+    parser_add.add_argument('--srcbranch', '-B', help='Branch in source repository if fetching from an SCM such as git (default master)')
     parser_add.add_argument('--binary', '-b', help='Treat the source tree as something that should be installed verbatim (no compilation, same directory structure). Useful with binary packages e.g. RPMs.', action='store_true')
     parser_add.add_argument('--also-native', help='Also add native variant (i.e. support building recipe for the build host as well as the target machine)', action='store_true')
     parser_add.add_argument('--src-subdir', help='Specify subdirectory within source tree to use', metavar='SUBDIR')
scripts/lib/recipetool/create.py +22 -3
index 2601145..8346336 100644
@@ -1,6 +1,6 @@
 # Recipe creation tool - create command plugin
 #
-# Copyright (C) 2014-2016 Intel Corporation
+# Copyright (C) 2014-2017 Intel Corporation
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 2 as
@@ -440,14 +440,30 @@ def create_recipe(args):
         rev_re = re.compile(';rev=([^;]+)')
         res = rev_re.search(srcuri)
         if res:
+            if args.srcrev:
+                logger.error('rev= parameter and -S/--srcrev option cannot both be specified - use one or the other')
+                sys.exit(1)
+            if args.autorev:
+                logger.error('rev= parameter and -a/--autorev option cannot both be specified - use one or the other')
+                sys.exit(1)
             srcrev = res.group(1)
             srcuri = rev_re.sub('', srcuri)
+        elif args.srcrev:
+            srcrev = args.srcrev
 
         # Check whether users provides any branch info in fetchuri.
         # If true, we will skip all branch checking process to honor all user's input.
         scheme, network, path, user, passwd, params = bb.fetch2.decodeurl(fetchuri)
         srcbranch = params.get('branch')
+        if args.srcbranch:
+            if srcbranch:
+                logger.error('branch= parameter and -B/--srcbranch option cannot both be specified - use one or the other')
+                sys.exit(1)
+            srcbranch = args.srcbranch
         nobranch = params.get('nobranch')
+        if nobranch and srcbranch:
+            logger.error('nobranch= cannot be used if you specify a branch')
+            sys.exit(1)
         tag = params.get('tag')
         if not srcbranch and not nobranch and srcrev != '${AUTOREV}':
             # Append nobranch=1 in the following conditions:
@@ -523,7 +539,7 @@ def create_recipe(args):
             else:
                 # If get_branch contains more than one objects, then display error and exit.
                 mbrch = '\n  ' + '\n  '.join(get_branch)
-                logger.error('Revision %s was found on multiple branches: %s\nPlease provide the correct branch in the source URL with ;branch=<branch> (and ensure you use quotes around the URL to avoid the shell interpreting the ";")' % (srcrev, mbrch))
+                logger.error('Revision %s was found on multiple branches: %s\nPlease provide the correct branch with -B/--srcbranch' % (srcrev, mbrch))
                 sys.exit(1)
 
         # Since we might have a value in srcbranch, we need to
@@ -1277,7 +1293,10 @@ def register_commands(subparsers):
     parser_create.add_argument('-b', '--binary', help='Treat the source tree as something that should be installed verbatim (no compilation, same directory structure)', action='store_true')
     parser_create.add_argument('--also-native', help='Also add native variant (i.e. support building recipe for the build host as well as the target machine)', action='store_true')
     parser_create.add_argument('--src-subdir', help='Specify subdirectory within source tree to use', metavar='SUBDIR')
-    parser_create.add_argument('-a', '--autorev', help='When fetching from a git repository, set SRCREV in the recipe to a floating revision instead of fixed', action="store_true")
+    group = parser_create.add_mutually_exclusive_group()
+    group.add_argument('-a', '--autorev', help='When fetching from a git repository, set SRCREV in the recipe to a floating revision instead of fixed', action="store_true")
+    group.add_argument('-S', '--srcrev', help='Source revision to fetch if fetching from an SCM such as git (default latest)')
+    parser_create.add_argument('-B', '--srcbranch', help='Branch in source repository if fetching from an SCM such as git (default master)')
     parser_create.add_argument('--keep-temp', action="store_true", help='Keep temporary directory (for debugging)')
     parser_create.add_argument('--fetch-dev', action="store_true", help='For npm, also fetch devDependencies')
     parser_create.add_argument('--devtool', action="store_true", help=argparse.SUPPRESS)
Signature · 1 line
2.9.5
#10 lore raw

PATCH 8/13 recipetool: create: make recently added branch/tag handling git specific

In reply to Paul Eggleton

The branch and tag handling code that was recently added in OE-Core revs ecca596b75cfda2f798a0bdde75f4f774e23a95b and 3afdcbdc9a3e65bc925ec61717784ffec67d529d is specific to git, so only apply it when we're fetching from a git URL.

---
 scripts/lib/recipetool/create.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/lib/recipetool/create.py +2 -2
index 8346336..4da745b 100644
@@ -518,7 +518,7 @@ def create_recipe(args):
         # We need this checking mechanism to improve the recipe created by recipetool and devtool
         # is able to parse and build by bitbake.
         # If there is no input for branch name, then check for branch name with SRCREV provided.
-        if not srcbranch and not nobranch and srcrev and (srcrev != '${AUTOREV}'):
+        if not srcbranch and not nobranch and srcrev and (srcrev != '${AUTOREV}') and scheme in ['git', 'gitsm']:
             try:
                 cmd = 'git branch -r --contains'
                 check_branch, check_branch_err = bb.process.run('%s %s' % (cmd, srcrev), cwd=srctree)
@@ -549,7 +549,7 @@ def create_recipe(args):
             params['branch'] = srcbranch
             srcuri = bb.fetch2.encodeurl((scheme, network, path, user, passwd, params))
 
-        if storeTagName:
+        if storeTagName and scheme in ['git', 'gitsm']:
             # Re-introduced tag variable from storeTagName
             # Check srcrev using tag and check validity of the tag
             cmd = ('git rev-parse --verify %s' % (storeTagName))
Signature · 1 line
2.9.5
#11 lore raw

PATCH 9/13 recipetool: create: fix SRCPV prefix for non-git SCMs

In reply to Paul Eggleton

If you're fetching from an SCM other than git (for example subversion or mercurial) then we need to use a different prefix for the SRCPV in PV instead of +git.

---
 scripts/lib/recipetool/create.py | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
scripts/lib/recipetool/create.py +9 -1
index 4da745b..d8cfcbd 100644
@@ -423,6 +423,7 @@ def create_recipe(args):
     srcsubdir = ''
     srcrev = '${AUTOREV}'
     srcbranch = ''
+    scheme = ''
     storeTagName = ''
     pv_srcpv = False
 
@@ -682,7 +683,14 @@ def create_recipe(args):
         lines_before.append('')
         lines_before.append('# Modify these as desired')
         # Note: we have code to replace realpv further down if it gets set to some other value
-        lines_before.append('PV = "%s+git${SRCPV}"' % (realpv or '1.0'))
+        scheme, _, _, _, _, _ = bb.fetch2.decodeurl(srcuri)
+        if scheme in ['git', 'gitsm']:
+            srcpvprefix = 'git'
+        elif scheme == 'svn':
+            srcpvprefix = 'svnr'
+        else:
+            srcpvprefix = scheme
+        lines_before.append('PV = "%s+%s${SRCPV}"' % (realpv or '1.0', srcpvprefix))
         pv_srcpv = True
         if not args.autorev and srcrev == '${AUTOREV}':
             if os.path.exists(os.path.join(srctree, '.git')):
Signature · 1 line
2.9.5
#12 lore raw

PATCH 10/13 recipetool: create: suppress npm shrinkwrap/lockdown warnings again

In reply to Paul Eggleton

Since OE-Core revision 9a47a6690052ef943c0d4760630ee630fb012153 the mechanism we were using to suppress the warnings about NPM_LOCKDOWN and NPM_SHRINKWRAP not being set on the first fetch of the source is no longer available since we are using the normal fetch/unpack tasks to do the job. Use the newly added noverify parameter to suppress the warnings again.

---
 scripts/lib/recipetool/create.py | 2 ++
 1 file changed, 2 insertions(+)
scripts/lib/recipetool/create.py +2 -0
index d8cfcbd..2b30385 100644
@@ -479,6 +479,8 @@ def create_recipe(args):
             storeTagName = params['tag']
             params['nobranch'] = '1'
             del params['tag']
+        if scheme == 'npm':
+            params['noverify'] = '1'
         fetchuri = bb.fetch2.encodeurl((scheme, network, path, user, passwd, params))
 
         tmpparent = tinfoil.config_data.getVar('BASE_WORKDIR')
Signature · 1 line
2.9.5
#13 lore raw

PATCH 11/13 recipetool: create: detect Eclipse licenses

In reply to Paul Eggleton

Add detection of EPL 1.0 and EDL 1.0 license files.

---
 scripts/lib/recipetool/create.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
scripts/lib/recipetool/create.py +5 -1
index 2b30385..4788691 100644
@@ -1099,6 +1099,10 @@ def crunch_license(licfile):
     crunched_md5sums['1daebd9491d1e8426900b4fa5a422814'] = 'LGPLv2.1'
     # https://github.com/FFmpeg/FFmpeg/blob/master/COPYING.LGPLv3
     crunched_md5sums['2ebfb3bb49b9a48a075cc1425e7f4129'] = 'LGPLv3'
+    # https://raw.githubusercontent.com/eclipse/mosquitto/v1.4.14/epl-v10
+    crunched_md5sums['efe2cb9a35826992b9df68224e3c2628'] = 'EPL-1.0'
+    # https://raw.githubusercontent.com/eclipse/mosquitto/v1.4.14/edl-v10
+    crunched_md5sums['0a9c78c0a398d1bbce4a166757d60387'] = 'EDL-1.0'
     lictext = []
     with open(licfile, 'r', errors='surrogateescape') as f:
         for line in f:
@@ -1131,7 +1135,7 @@ def guess_license(srctree, d):
     md5sums = get_license_md5sums(d)
 
     licenses = []
-    licspecs = ['*LICEN[CS]E*', 'COPYING*', '*[Ll]icense*', 'LEGAL*', '[Ll]egal*', '*GPL*', 'README.lic*', 'COPYRIGHT*', '[Cc]opyright*']
+    licspecs = ['*LICEN[CS]E*', 'COPYING*', '*[Ll]icense*', 'LEGAL*', '[Ll]egal*', '*GPL*', 'README.lic*', 'COPYRIGHT*', '[Cc]opyright*', 'e[dp]l-v10']
     licfiles = []
     for root, dirs, files in os.walk(srctree):
         for fn in files:
Signature · 1 line
2.9.5
#14 lore raw

PATCH 12/13 oe-selftest: tinfoil: add a test for variable history

In reply to Paul Eggleton

I recently found that variable history wasn't working properly for recipes when we enable history tracking, resulting in minor functionality loss in devtool upgrade, so add a test to ensure this doesn't regress now that it's fixed.

---
 meta/lib/oeqa/selftest/cases/tinfoil.py | 34 +++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
meta/lib/oeqa/selftest/cases/tinfoil.py +34 -0
index 471517f..f889a47 100644
@@ -195,3 +195,37 @@ class TinfoilTests(OESelftestTestCase):
             tinfoil.config_data.appendVar('OVERRIDES', ':overrideone')
             value = tinfoil.config_data.getVar('TESTVAR')
             self.assertEqual(value, 'one', 'Variable overrides not functioning correctly')
+
+    def test_variable_history(self):
+        # Basic test to ensure that variable history works when tracking=True
+        with bb.tinfoil.Tinfoil(tracking=True) as tinfoil:
+            tinfoil.prepare(config_only=False, quiet=2)
+            # Note that _tracking for any datastore we get will be
+            # false here, that's currently expected - so we can't check
+            # for that
+            history = tinfoil.config_data.varhistory.variable('DL_DIR')
+            for entry in history:
+                if entry['file'].endswith('/bitbake.conf'):
+                    if entry['op'] in ['set', 'set?']:
+                        break
+            else:
+                self.fail('Did not find history entry setting DL_DIR in bitbake.conf. History: %s' % history)
+            # Check it works for recipes as well
+            testrecipe = 'zlib'
+            rd = tinfoil.parse_recipe(testrecipe)
+            history = rd.varhistory.variable('LICENSE')
+            bbfound = -1
+            recipefound = -1
+            for i, entry in enumerate(history):
+                if entry['file'].endswith('/bitbake.conf'):
+                    if entry['detail'] == 'INVALID' and entry['op'] in ['set', 'set?']:
+                        bbfound = i
+                elif entry['file'].endswith('.bb'):
+                    if entry['op'] == 'set':
+                        recipefound = i
+            if bbfound == -1:
+                self.fail('Did not find history entry setting LICENSE in bitbake.conf parsing %s recipe. History: %s' % (testrecipe, history))
+            if recipefound == -1:
+                self.fail('Did not find history entry setting LICENSE in %s recipe. History: %s' % (testrecipe, history))
+            if bbfound > recipefound:
+                self.fail('History entry setting LICENSE in %s recipe and in bitbake.conf in wrong order. History: %s' % (testrecipe, history))
Signature · 1 line
2.9.5
#15 lore raw

PATCH 13/13 scriptutils: fix fetch_url() to use lowercase dummy recipe name

In reply to Paul Eggleton

recipetool create (and hence devtool add) and devtool upgrade use fetch_url() which creates a dummy recipe in order to fetch source. Previously the random part of the name was using uppercase characters, and this triggers a QA warning after OE-Core commit 4713f8b2c4f2c74239d284adcf1e59e61aa66576, so use lowercase instead as I really should have in the first place.

---
 scripts/lib/scriptutils.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/lib/scriptutils.py +1 -1
index 11f1a78..85b1c94 100644
@@ -134,7 +134,7 @@ def fetch_url(tinfoil, srcuri, srcrev, destdir, logger, preserve_tmp=False, mirr
             # for do_fetch and do_unpack
             # I'd use tempfile functions here but underscores can be produced by that and those
             # aren't allowed in recipe file names except to separate the version
-            rndstring = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(8))
+            rndstring = ''.join(random.choice(string.ascii_lowercase + string.digits) for _ in range(8))
             fetchrecipe = os.path.join(fetchrecipedir, 'tmp-recipetool-%s.bb' % rndstring)
             fetchrecipepn = os.path.splitext(os.path.basename(fetchrecipe))[0]
             logger.debug('Generating initial recipe %s for fetching' % fetchrecipe)
Signature · 1 line
2.9.5