Skip to contentSkip to conversation

Fixes for devtool and the extensible SDK

PATCH In review Paul Eggleton 2 participants 29 messages Sep 22, 2015

lore
Share & tools
lore
#0 lore raw

cover Fixes for devtool and the extensible SDK

Here is a set of fixes and improvements for devtool, the extensible SDK, and related code. Included are fixes for devtool add / recipetool create, fixes for devtool build-image, a couple of new subcommands for searching and running QEMU within the extensible SDK, a stress testing script for running devtool extract / devtool modify over every recipe, plus some fixes for the code that that script exposed.

The following changes since commit 72682d72b52355c3fed947167ca3c6064340ead1:

  autotools.bbclass: mkdir ${B} -> mkdir -p ${B} (2015-09-16 22:17:19 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib paule/devtool-extsdk-fixes
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool-extsdk-fixes

Paul Eggleton (27):
lib/oe/recipeutils: properly split unexpanded variable values classes/populate_sdk_ext: fix missing environment settings if running installer with sh classes/populate_sdk_ext: allow custom configuration for extensible SDK classes/populate_sdk_ext: drop work-config.inc
toolchain-shar-extract.sh: show progress when extracting SDK classes/externalsrc: scale back warning to a plain note classes/externalsrc: fix setting of deps varflag as a string lib/oe/patch: fix for git am not cleaning up after itself scripts/contrib: add devtool stress tester
devtool: check that source tree still exists
devtool: add: ensure --color=never turns off recipetool colour output devtool / lib/oe/recipeutils: ensure we can parse without bbappends
devtool: add: properly handle separate build directory
recipetool: create: fix creating empty shell functions
recipetool: create: fix handling of URIs containing #
devtool: show proper error when extracting source for recipes with disabled unpack task
devtool: fix extracting source for work-shared recipes
devtool: better handling for recipes that don't unpack source
devtool: add: set up fetched source as a git repository by default
devtool: add: move important "recipe created" message to the end
devtool: build-image: fix recipe/package terminology
devtool: build-image: tell user where to find output files
devtool: build-image: improve image recipe handling
devtool: build-image: delete bbappend at end of build
devtool / recipetool: add handling for binary-only packages
devtool: add basic means of running runqemu within the extensible SDK
devtool: add search command

 meta/classes/externalsrc.bbclass      |   5 +-
 meta/classes/populate_sdk_ext.bbclass |  19 +--
 meta/files/toolchain-shar-extract.sh  |   2 +-
 meta/lib/oe/patch.py                  |   7 +
 meta/lib/oe/recipeutils.py            |  63 ++++++++-
 meta/lib/oeqa/selftest/devtool.py     |  22 +++-
 scripts/contrib/devtool-stress.py     | 241 ++++++++++++++++++++++++++++++++++
 scripts/lib/devtool/__init__.py       |  51 +++++++
 scripts/lib/devtool/build-image.py    |  94 +++++++------
 scripts/lib/devtool/build.py          |   6 +-
 scripts/lib/devtool/deploy.py         |   7 +-
 scripts/lib/devtool/package.py        |   6 +-
 scripts/lib/devtool/runqemu.py        |  64 +++++++++
 scripts/lib/devtool/sdk.py            |   2 +-
 scripts/lib/devtool/search.py         |  80 +++++++++++
 scripts/lib/devtool/standard.py       |  78 ++++++-----
 scripts/lib/devtool/upgrade.py        |  20 +--
 scripts/lib/recipetool/create.py      |  42 +++++-
 18 files changed, 684 insertions(+), 125 deletions(-)
 create mode 100755 scripts/contrib/devtool-stress.py
 create mode 100644 scripts/lib/devtool/runqemu.py
 create mode 100644 scripts/lib/devtool/search.py
Signature · 1 line
2.1.0
#1 lore raw

PATCH 1/27 lib/oe/recipeutils: properly split unexpanded variable values

Variables such as SRC_URI which are space-separated may also contain Python expressions (${@...}) which themselves contain spaces that shouldn't be split when splitting the value into items. In order to ensure this we need to use a custom splitting function instead of just string.split().

This issue could be seen when doing "devtool modify sudo", adding a commit to the resulting source repository then "devtool update-recipe" - the Python expression in SRC_URI was being unnecessarily broken onto multiple lines.

Fixes [YOCTO #8046].

---
 meta/lib/oe/recipeutils.py | 61 ++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 59 insertions(+), 2 deletions(-)
meta/lib/oe/recipeutils.py +59 -2
index d4fa726..190ac3a 100644
@@ -95,6 +95,63 @@ def get_var_files(fn, varlist, d):
     return varfiles
 
 
+def split_var_value(value, assignment=True):
+    """
+    Split a space-separated variable's value into a list of items,
+    taking into account that some of the items might be made up of
+    expressions containing spaces that should not be split.
+    Parameters:
+        value:
+            The string value to split
+        assignment:
+            True to assume that the value represents an assignment
+            statement, False otherwise. If True, and an assignment
+            statement is passed in the first item in
+            the returned list will be the part of the assignment
+            statement up to and including the opening quote character,
+            and the last item will be the closing quote.
+    """
+    inexpr = 0
+    lastchar = None
+    out = []
+    buf = ''
+    for char in value:
+        if char == '{':
+            if lastchar == '$':
+                inexpr += 1
+        elif char == '}':
+            inexpr -= 1
+        elif assignment and char in '"\'' and inexpr == 0:
+            if buf:
+                out.append(buf)
+            out.append(char)
+            char = ''
+            buf = ''
+        elif char.isspace() and inexpr == 0:
+            char = ''
+            if buf:
+                out.append(buf)
+            buf = ''
+        buf += char
+        lastchar = char
+    if buf:
+        out.append(buf)
+
+    # Join together assignment statement and opening quote
+    outlist = out
+    if assignment:
+        assigfound = False
+        for idx, item in enumerate(out):
+            if '=' in item:
+                assigfound = True
+            if assigfound:
+                if '"' in item or "'" in item:
+                    outlist = [' '.join(out[:idx+1])]
+                    outlist.extend(out[idx+1:])
+                    break
+    return outlist
+
+
 def patch_recipe_file(fn, values, patch=False, relpath=''):
     """Update or insert variable values into a recipe file (assuming you
        have already identified the exact file you want to update.)
@@ -112,7 +169,7 @@ def patch_recipe_file(fn, values, patch=False, relpath=''):
             if name in nowrap_vars:
                 tf.write(rawtext)
             elif name in list_vars:
-                splitvalue = values[name].split()
+                splitvalue = split_var_value(values[name], assignment=False)
                 if len(splitvalue) > 1:
                     linesplit = ' \\\n' + (' ' * (len(name) + 4))
                     tf.write('%s = "%s%s"\n' % (name, linesplit.join(splitvalue), linesplit))
@@ -518,7 +575,7 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False,
                             instfunclines.append(line)
                     return (instfunclines, None, 4, False)
             else:
-                splitval = origvalue.split()
+                splitval = split_var_value(origvalue, assignment=False)
                 changed = False
                 removevar = varname
                 if varname in ['SRC_URI', 'SRC_URI_append%s' % appendoverride]:
Signature · 1 line
2.1.0
#3 lore raw

PATCH 2/27 classes/populate_sdk_ext: fix missing environment settings if running installer with sh

In reply to Paul Eggleton

If you ran the extensible SDK installer file with sh (instead of bash), then the additional call to buildtools environment setup, extension of PATH to support running devtool, and setting of OE_SKIP_SDK_CHECK weren't being added to the end of the script. This is because apparently bash is happy to expand wildcards in the target of a redirection, but bash running in POSIX sh mode won't (although it apparently does work on the sh command line rather than within a script run as an argument to sh). In any case using a wildcard here is a bit of a crutch which we don't need, so replace it with the proper path to the environment setup script.

---
 meta/classes/populate_sdk_ext.bbclass | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
meta/classes/populate_sdk_ext.bbclass +4 -3
index 4ef8838..15cae71 100644
@@ -222,14 +222,15 @@ sdk_ext_postinst() {
 
 	# Make sure when the user sets up the environment, they also get
 	# the buildtools-tarball tools in their path.
-	echo ". $target_sdk_dir/buildtools/environment-setup*" >> $target_sdk_dir/environment-setup*
+	env_setup_script="$target_sdk_dir/environment-setup-${REAL_MULTIMACH_TARGET_SYS}"
+	echo ". $target_sdk_dir/buildtools/environment-setup*" >> $env_setup_script
 
 	# Allow bitbake environment setup to be ran as part of this sdk.
-	echo "export OE_SKIP_SDK_CHECK=1" >> $target_sdk_dir/environment-setup*
+	echo "export OE_SKIP_SDK_CHECK=1" >> $env_setup_script
 
 	# A bit of another hack, but we need this in the path only for devtool
 	# so put it at the end of $PATH.
-	echo "export PATH=\$PATH:$target_sdk_dir/sysroots/${SDK_SYS}/${bindir_nativesdk}" >> $target_sdk_dir/environment-setup*
+	echo "export PATH=\$PATH:$target_sdk_dir/sysroots/${SDK_SYS}/${bindir_nativesdk}" >> $env_setup_script
 
 	# For now this is where uninative.bbclass expects the tarball
 	mv *-nativesdk-libc.tar.* $target_sdk_dir/`dirname ${oe_init_build_env_path}`
Signature · 1 line
2.1.0
#4 lore raw

PATCH 3/27 classes/populate_sdk_ext: allow custom configuration for extensible SDK

In reply to Paul Eggleton

Provide the ability to define a function containing extra configuration values to be added to the local.conf file that goes into the SDK. For example, this could be used to set up SSTATE_MIRRORS within the SDK.

---
 meta/classes/populate_sdk_ext.bbclass | 7 +++++++
 1 file changed, 7 insertions(+)
meta/classes/populate_sdk_ext.bbclass +7 -0
index 15cae71..947ec74 100644
@@ -157,6 +157,13 @@ python copy_buildsystem () {
         # Ensure locked sstate cache objects are re-used without error
         f.write('SIGGEN_LOCKEDSIGS_CHECK_LEVEL = "warn"\n\n')
 
+        # If you define a sdk_extraconf() function then it can contain additional config
+        extraconf = (d.getVar('sdk_extraconf', True) or '').strip()
+        if extraconf:
+            # Strip off any leading / trailing spaces
+            for line in extraconf.splitlines():
+                f.write(line.strip() + '\n')
+
         f.write('require conf/locked-sigs.inc\n')
         f.write('require conf/work-config.inc\n')
 
Signature · 1 line
2.1.0
#5 lore raw

PATCH 4/27 classes/populate_sdk_ext: drop work-config.inc

In reply to Paul Eggleton

This is not actually used for anything - I thought that we would need to use it within devtool to set global configuration, but we're able to do everything we need within the bbappends it creates, which also saves on parse time. If we're not going to use work-config.inc let's just drop it completely.

---
 meta/classes/populate_sdk_ext.bbclass | 5 -----
 scripts/lib/devtool/sdk.py            | 2 +-
 2 files changed, 1 insertion(+), 6 deletions(-)
meta/classes/populate_sdk_ext.bbclass +0 -5
index 947ec74..c244d93 100644
@@ -165,7 +165,6 @@ python copy_buildsystem () {
                 f.write(line.strip() + '\n')
 
         f.write('require conf/locked-sigs.inc\n')
-        f.write('require conf/work-config.inc\n')
 
     sigfile = d.getVar('WORKDIR', True) + '/locked-sigs.inc'
     oe.copy_buildsystem.generate_locked_sigs(sigfile, d)
@@ -185,10 +184,6 @@ python copy_buildsystem () {
                                                    d.getVar('SSTATE_DIR', True),
                                                    sstate_out, d,
                                                    fixedlsbstring)
-
-    # Create a dummy config file for additional settings
-    with open(baseoutpath + '/conf/work-config.inc', 'w') as f:
-        pass
 }
 
 def extsdk_get_buildtools_filename(d):
scripts/lib/devtool/sdk.py +1 -1
index 2f416b3..7a842af 100644
@@ -170,7 +170,7 @@ def sdk_update(args, config, basepath, workspace):
             logger.error("Updating meta data via git failed")
             return ret
         logger.debug("Updating conf files ...")
-        conf_files = ['local.conf', 'bblayers.conf', 'devtool.conf', 'work-config.inc', 'locked-sigs.inc']
+        conf_files = ['local.conf', 'bblayers.conf', 'devtool.conf', 'locked-sigs.inc']
         for conf in conf_files:
             ret = subprocess.call("wget -q -O - %s/conf/%s > conf/%s" % (args.updateserver, conf, conf), shell=True)
             if ret != 0:
Signature · 1 line
2.1.0
#6 lore raw

PATCH 5/27 toolchain-shar-extract.sh: show progress when extracting SDK

In reply to Paul Eggleton

Tar has supported a --checkpoint option since version 1.15.91, so it should be safe to use here to print dots showing that it's still doing something (technically it's not really progress unless you know how many dots it's going to print, which even it doesn't know at the start, but it's better than nothing).

---
 meta/files/toolchain-shar-extract.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/files/toolchain-shar-extract.sh +1 -1
index cd0a547..6891279 100644
@@ -163,7 +163,7 @@ fi
 payload_offset=$(($(grep -na -m1 "^MARKER:$" $0|cut -d':' -f1) + 1))
 
 printf "Extracting SDK..."
-tail -n +$payload_offset $0| $SUDO_EXEC tar xj -C $target_sdk_dir
+tail -n +$payload_offset $0| $SUDO_EXEC tar xj -C $target_sdk_dir --checkpoint=.2500
 echo "done"
 
 printf "Setting it up..."
Signature · 1 line
2.1.0
#7 lore raw

PATCH 6/27 classes/externalsrc: scale back warning to a plain note

In reply to Paul Eggleton

It turns out that there are folks out there who use externalsrc in normal builds and don't really need to be warned; additionally within the extensible SDK or when using devtool, it shouldn't be a warning situation. Thus, scale it back to a note (we can't use bb.note() here since that wouldn't actually be piped through to the bitbake UI). Also touch up the message a little bit.

---
 meta/classes/externalsrc.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/externalsrc.bbclass +1 -1
index 0fa5817..70e6fd7 100644
@@ -88,5 +88,5 @@ python () {
 
 python externalsrc_compile_prefunc() {
     # Make it obvious that this is happening, since forgetting about it could lead to much confusion
-    bb.warn('Compiling %s from external source %s' % (d.getVar('PN', True), d.getVar('EXTERNALSRC', True)))
+    bb.plain('NOTE: %s: compiling from external source tree %s' % (d.getVar('PN', True), d.getVar('EXTERNALSRC', True)))
 }
Signature · 1 line
2.1.0
#8 lore raw

PATCH 7/27 classes/externalsrc: fix setting of deps varflag as a string

In reply to Paul Eggleton

The value of the deps varflag for tasks is (unusually) expected to be a list object rather than a space-separated string, hence we cannot use appendVarFlag() here. This fixes a traceback when parsing the gcc recipe with externalsrc enabled, for example.

---
 meta/classes/externalsrc.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
meta/classes/externalsrc.bbclass +2 -1
index 70e6fd7..f7ed66d 100644
@@ -73,7 +73,8 @@ python () {
 
         fetch_tasks = ['do_fetch', 'do_unpack']
         # If we deltask do_patch, there's no dependency to ensure do_unpack gets run, so add one
-        d.appendVarFlag('do_configure', 'deps', ['do_unpack'])
+        # Note that we cannot use d.appendVarFlag() here because deps is expected to be a list object, not a string
+        d.setVarFlag('do_configure', 'deps', (d.getVarFlag('do_configure', 'deps', False) or []) + ['do_unpack'])
 
         for task in d.getVar("SRCTREECOVEREDTASKS", True).split():
             if local_srcuri and task in fetch_tasks:
Signature · 1 line
2.1.0
#9 lore raw

PATCH 8/27 lib/oe/patch: fix for git am not cleaning up after itself

In reply to Paul Eggleton

Unfortunately it appears that under certain circumstances, a failed git am followed by git am --abort won't clean up any changes the patch might have made - this was seen when running "devtool extract" on the unzip recipe; unzip-6.0_overflow3.diff has a malformed date as far as git am is concerned but it triggers this condition. Add a git reset --hard HEAD followed by git clean -f in order to recover from this scenario.

---
 meta/lib/oe/patch.py | 7 +++++++
 1 file changed, 7 insertions(+)
meta/lib/oe/patch.py +7 -0
index 108bf1d..7441214 100644
@@ -407,6 +407,13 @@ class GitApplyTree(PatchTree):
                     runcmd(["sh", "-c", " ".join(shellcmd)], self.dir)
                 except CmdError:
                     pass
+                # git am won't always clean up after itself, sadly, so...
+                shellcmd = ["git", "--work-tree=%s" % reporoot, "reset", "--hard", "HEAD"]
+                runcmd(["sh", "-c", " ".join(shellcmd)], self.dir)
+                # Also need to take care of any stray untracked files
+                shellcmd = ["git", "--work-tree=%s" % reporoot, "clean", "-f"]
+                runcmd(["sh", "-c", " ".join(shellcmd)], self.dir)
+
                 # Fall back to git apply
                 shellcmd = ["git", "--git-dir=%s" % reporoot, "apply", "-p%s" % patch['strippath']]
                 try:
Signature · 1 line
2.1.0
#10 lore raw

PATCH 9/27 scripts/contrib: add devtool stress tester

In reply to Paul Eggleton

Add a script to run "devtool modify" followed by a build on every target recipe in the environment (with the option to skip/resume from/only include specific recipes). This takes far too long to run as an oe-selftest test but is still something that is useful to be able to run. There's also a slightly quicker mode that just runs "devtool extract" on each recipe.

---
 scripts/contrib/devtool-stress.py | 241 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 241 insertions(+)
 create mode 100755 scripts/contrib/devtool-stress.py
scripts/contrib/devtool-stress.py new file +241 -0
new file mode 100755
index 0000000..4b35fc9
@@ -0,0 +1,241 @@
+#!/usr/bin/env python
+
+# devtool stress tester
+#
+# Written by: Paul Eggleton <[email protected]>
+#
+# Copyright 2015 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+
+import sys
+import os
+import os.path
+import subprocess
+import re
+import argparse
+import logging
+import tempfile
+import shutil
+import signal
+import fnmatch
+
+scripts_lib_path = os.path.abspath(os.path.join(os.path.dirname(os.path.realpath(__file__)), '..', 'lib'))
+sys.path.insert(0, scripts_lib_path)
+import scriptutils
+logger = scriptutils.logger_create('devtool-stress')
+
+def select_recipes(args):
+    import bb.tinfoil
+    tinfoil = bb.tinfoil.Tinfoil()
+    tinfoil.prepare(False)
+
+    pkg_pn = tinfoil.cooker.recipecache.pkg_pn
+    (latest_versions, preferred_versions) = bb.providers.findProviders(tinfoil.config_data, tinfoil.cooker.recipecache, pkg_pn)
+
+    skip_classes = args.skip_classes.split(',')
+
+    recipelist = []
+    for pn in sorted(pkg_pn):
+        pref = preferred_versions[pn]
+        inherits = [os.path.splitext(os.path.basename(f))[0] for f in tinfoil.cooker.recipecache.inherits[pref[1]]]
+        for cls in skip_classes:
+            if cls in inherits:
+                break
+        else:
+            recipelist.append(pn)
+
+    tinfoil.shutdown()
+
+    resume_from = args.resume_from
+    if resume_from:
+        if not resume_from in recipelist:
+            print('%s is not a testable recipe' % resume_from)
+            return 1
+    if args.only:
+        only = args.only.split(',')
+        for onlyitem in only:
+            for pn in recipelist:
+                if fnmatch.fnmatch(pn, onlyitem):
+                    break
+            else:
+                print('%s does not match any testable recipe' % onlyitem)
+                return 1
+    else:
+        only = None
+    if args.skip:
+        skip = args.skip.split(',')
+    else:
+        skip = []
+
+    recipes = []
+    for pn in recipelist:
+        if resume_from:
+            if pn == resume_from:
+                resume_from = None
+            else:
+                continue
+
+        if args.only:
+            for item in only:
+                if fnmatch.fnmatch(pn, item):
+                    break
+            else:
+                continue
+
+        skipit = False
+        for item in skip:
+            if fnmatch.fnmatch(pn, item):
+                skipit = True
+        if skipit:
+            continue
+
+        recipes.append(pn)
+
+    return recipes
+
+
+def stress_extract(args):
+    import bb.process
+
+    recipes = select_recipes(args)
+
+    failures = 0
+    tmpdir = tempfile.mkdtemp()
+    os.setpgrp()
+    try:
+        for pn in recipes:
+            sys.stdout.write('Testing %s ' % (pn + ' ').ljust(40, '.'))
+            sys.stdout.flush()
+            failed = False
+
+            srctree = os.path.join(tmpdir, pn)
+            try:
+                bb.process.run('devtool extract %s %s' % (pn, srctree))
+            except bb.process.CmdError as exc:
+                failed = True
+                with open('stress_%s_extract.log' % pn, 'w') as f:
+                    f.write(str(exc))
+
+            if os.path.exists(srctree):
+                shutil.rmtree(srctree)
+
+            if failed:
+                print('failed')
+                failures += 1
+            else:
+                print('ok')
+    except KeyboardInterrupt:
+        # We want any child processes killed. This is crude, but effective.
+        os.killpg(0, signal.SIGTERM)
+
+    if failures:
+        return 1
+    else:
+        return 0
+
+
+def stress_modify(args):
+    import bb.process
+
+    recipes = select_recipes(args)
+
+    failures = 0
+    tmpdir = tempfile.mkdtemp()
+    os.setpgrp()
+    try:
+        for pn in recipes:
+            sys.stdout.write('Testing %s ' % (pn + ' ').ljust(40, '.'))
+            sys.stdout.flush()
+            failed = False
+            reset = True
+
+            srctree = os.path.join(tmpdir, pn)
+            try:
+                bb.process.run('devtool modify -x %s %s' % (pn, srctree))
+            except bb.process.CmdError as exc:
+                with open('stress_%s_modify.log' % pn, 'w') as f:
+                    f.write(str(exc))
+                failed = 'modify'
+                reset = False
+
+            if not failed:
+                try:
+                    bb.process.run('bitbake -c install %s' % pn)
+                except bb.process.CmdError as exc:
+                    with open('stress_%s_install.log' % pn, 'w') as f:
+                        f.write(str(exc))
+                    failed = 'build'
+            if reset:
+                try:
+                    bb.process.run('devtool reset %s' % pn)
+                except bb.process.CmdError as exc:
+                    print('devtool reset failed: %s' % str(exc))
+                    break
+
+            if os.path.exists(srctree):
+                shutil.rmtree(srctree)
+
+            if failed:
+                print('failed (%s)' % failed)
+                failures += 1
+            else:
+                print('ok')
+    except KeyboardInterrupt:
+        # We want any child processes killed. This is crude, but effective.
+        os.killpg(0, signal.SIGTERM)
+
+    if failures:
+        return 1
+    else:
+        return 0
+
+
+def main():
+    parser = argparse.ArgumentParser(description="devtool stress tester",
+                                     epilog="Use %(prog)s <subcommand> --help to get help on a specific command")
+    parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true')
+    parser.add_argument('-r', '--resume-from', help='Resume from specified recipe', metavar='PN')
+    parser.add_argument('-o', '--only', help='Only test specified recipes (comma-separated without spaces, wildcards allowed)', metavar='PNLIST')
+    parser.add_argument('-s', '--skip', help='Skip specified recipes (comma-separated without spaces, wildcards allowed)', metavar='PNLIST')
+    parser.add_argument('-c', '--skip-classes', help='Skip recipes inheriting specified classes (comma-separated) - default %(default)s', metavar='CLASSLIST', default='native,nativesdk,cross,cross-canadian,image,populate_sdk,meta,packagegroup')
+    subparsers = parser.add_subparsers(title='subcommands', metavar='<subcommand>')
+
+    parser_modify = subparsers.add_parser('modify',
+                                          help='Run "devtool modify" followed by a build with bitbake on matching recipes',
+                                          description='Runs "devtool modify" followed by a build with bitbake on matching recipes')
+    parser_modify.set_defaults(func=stress_modify)
+
+    parser_extract = subparsers.add_parser('extract',
+                                           help='Run "devtool extract" on matching recipes',
+                                           description='Runs "devtool extract" on matching recipes')
+    parser_extract.set_defaults(func=stress_extract)
+
+    args = parser.parse_args()
+
+    if args.debug:
+        logger.setLevel(logging.DEBUG)
+
+    import scriptpath
+    bitbakepath = scriptpath.add_bitbake_lib_path()
+    if not bitbakepath:
+        logger.error("Unable to find bitbake by searching parent directory of this script or PATH")
+        return 1
+    logger.debug('Found bitbake path: %s' % bitbakepath)
+
+    ret = args.func(args)
+
+if __name__ == "__main__":
+    main()
Signature · 1 line
2.1.0
#11 lore raw

PATCH 10/27 devtool: check that source tree still exists

In reply to Paul Eggleton

Sometimes, particularly if you extracted the source to /tmp which is on tmpfs, the external source tree that is being pointed to may no longer exist when you come to run "devtool build" or "devtool update-recipe" etc. Make all of the commands that need to check for a recipe being in the workspace call a single function and have that function additionally check the source tree still exists where appropriate.

---
 scripts/lib/devtool/__init__.py | 14 ++++++++++++++
 scripts/lib/devtool/build.py    |  6 ++----
 scripts/lib/devtool/deploy.py   |  7 +++----
 scripts/lib/devtool/package.py  |  6 ++----
 scripts/lib/devtool/standard.py | 11 ++++-------
 5 files changed, 25 insertions(+), 19 deletions(-)
scripts/lib/devtool/__init__.py +14 -0
index 404d3e6..fb699b5 100644
@@ -136,3 +136,17 @@ def parse_recipe(config, tinfoil, pn, appends):
                         not path.startswith(config.workspace_path)]
     return oe.recipeutils.parse_recipe(recipefile, append_files,
                                        tinfoil.config_data)
+
+def check_workspace_recipe(workspace, pn, checksrc=True):
+    """
+    Check that a recipe is in the workspace and (optionally) that source
+    is present.
+    """
+    if not pn in workspace:
+        raise DevtoolError("No recipe named '%s' in your workspace" % pn)
+    if checksrc:
+        srctree = workspace[pn]['srctree']
+        if not os.path.exists(srctree):
+            raise DevtoolError("Source tree %s for recipe %s does not exist" % (srctree, pn))
+        if not os.listdir(srctree):
+            raise DevtoolError("Source tree %s for recipe %s is empty" % (srctree, pn))
scripts/lib/devtool/build.py +2 -4
index 335aff5..9b58858 100644
@@ -21,7 +21,7 @@ import bb
 import logging
 import argparse
 import tempfile
-from devtool import exec_build_env_command, DevtoolError
+from devtool import exec_build_env_command, check_workspace_recipe, DevtoolError
 
 logger = logging.getLogger('devtool')
 
@@ -42,9 +42,7 @@ def _create_conf_file(values, conf_file=None):
 
 def build(args, config, basepath, workspace):
     """Entry point for the devtool 'build' subcommand"""
-    if not args.recipename in workspace:
-        raise DevtoolError("no recipe named %s in your workspace" %
-                           args.recipename)
+    check_workspace_recipe(workspace, args.recipename)
 
     build_task = config.get('Build', 'build_task', 'populate_sysroot')
 
scripts/lib/devtool/deploy.py +3 -4
index fa93adf..41b666f 100644
@@ -19,7 +19,7 @@
 import os
 import subprocess
 import logging
-from devtool import exec_fakeroot, setup_tinfoil, DevtoolError
+from devtool import exec_fakeroot, setup_tinfoil, check_workspace_recipe, DevtoolError
 
 logger = logging.getLogger('devtool')
 
@@ -28,9 +28,8 @@ def deploy(args, config, basepath, workspace):
     import re
     import oe.recipeutils
 
-    if not args.recipename in workspace:
-        raise DevtoolError("no recipe named %s in your workspace" %
-                           args.recipename)
+    check_workspace_recipe(workspace, args.recipename, checksrc=False)
+
     try:
         host, destdir = args.target.split(':')
     except ValueError:
scripts/lib/devtool/package.py +2 -4
index 3a7a36b..28ecfed 100644
@@ -20,7 +20,7 @@ import os
 import subprocess
 import logging
 from bb.process import ExecutionError
-from devtool import exec_build_env_command, setup_tinfoil, DevtoolError
+from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, DevtoolError
 
 logger = logging.getLogger('devtool')
 
@@ -30,9 +30,7 @@ def plugin_init(pluginlist):
 
 def package(args, config, basepath, workspace):
     """Entry point for the devtool 'package' subcommand"""
-    if not args.recipename in workspace:
-        raise DevtoolError("no recipe named %s in your workspace" %
-                           args.recipename)
+    check_workspace_recipe(workspace, args.recipename)
 
     image_pkgtype = config.get('Package', 'image_pkgtype', '')
     if not image_pkgtype:
scripts/lib/devtool/standard.py +4 -7
index d5900b4..018ef2a 100644
@@ -25,7 +25,7 @@ import logging
 import argparse
 import scriptutils
 import errno
-from devtool import exec_build_env_command, setup_tinfoil, DevtoolError
+from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, DevtoolError
 from devtool import parse_recipe
 
 logger = logging.getLogger('devtool')
@@ -776,9 +776,7 @@ def _guess_recipe_update_mode(srctree, rdata):
 
 def update_recipe(args, config, basepath, workspace):
     """Entry point for the devtool 'update-recipe' subcommand"""
-    if not args.recipename in workspace:
-        raise DevtoolError("no recipe named %s in your workspace" %
-                           args.recipename)
+    check_workspace_recipe(workspace, args.recipename)
 
     if args.append:
         if not os.path.exists(args.append):
@@ -830,9 +828,8 @@ def reset(args, config, basepath, workspace):
     if args.recipename:
         if args.all:
             raise DevtoolError("Recipe cannot be specified if -a/--all is used")
-        elif not args.recipename in workspace:
-            raise DevtoolError("no recipe named %s in your workspace" %
-                               args.recipename)
+        else:
+            check_workspace_recipe(workspace, args.recipename, checksrc=False)
     elif not args.all:
         raise DevtoolError("Recipe must be specified, or specify -a/--all to "
                            "reset all recipes")
Signature · 1 line
2.1.0
#12 lore raw

PATCH 11/27 devtool: add: ensure --color=never turns off recipetool colour output

In reply to Paul Eggleton
---
 scripts/lib/devtool/standard.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/lib/devtool/standard.py +1 -1
index 018ef2a..ec21b3c 100644
@@ -82,7 +82,7 @@ def add(args, config, basepath, workspace):
     else:
         bp = args.recipename
     recipefile = os.path.join(recipedir, "%s.bb" % bp)
-    if sys.stdout.isatty():
+    if args.color == 'auto' and sys.stdout.isatty():
         color = 'always'
     else:
         color = args.color
Signature · 1 line
2.1.0
#13 lore raw

PATCH 12/27 devtool / lib/oe/recipeutils: ensure we can parse without bbappends

In reply to Paul Eggleton

These functions ostensibly allowed parsing a recipe without bbappends but this clearly hadn't been tested because a variable was unassigned in both of them in that case.

---
 meta/lib/oe/recipeutils.py      | 2 ++
 scripts/lib/devtool/__init__.py | 2 ++
 2 files changed, 4 insertions(+)
meta/lib/oe/recipeutils.py +2 -0
index 190ac3a..35b88d3 100644
@@ -72,6 +72,8 @@ def parse_recipe_simple(cooker, pn, d, appends=True):
             raise bb.providers.NoProvider('Unable to find any recipe file matching %s' % pn)
     if appends:
         appendfiles = cooker.collection.get_file_appends(recipefile)
+    else:
+        appendfiles = None
     return parse_recipe(recipefile, appendfiles, d)
 
 
scripts/lib/devtool/__init__.py +2 -0
index fb699b5..3ea3802 100644
@@ -134,6 +134,8 @@ def parse_recipe(config, tinfoil, pn, appends):
         # Filter out appends from the workspace
         append_files = [path for path in append_files if
                         not path.startswith(config.workspace_path)]
+    else:
+        append_files = None
     return oe.recipeutils.parse_recipe(recipefile, append_files,
                                        tinfoil.config_data)
 
Signature · 1 line
2.1.0
#14 lore raw

PATCH 13/27 devtool: add: properly handle separate build directory

In reply to Paul Eggleton

When we were adding a recipe for software that would typically be built in the same directory as the source, we were always using a separate build directory unless the user explicitly specified not to, leading to errors for software that doesn't expect to be built that way (such as Python modules using distutils). Split out the code that makes this determination automatically from the "devtool modify" and "devtool upgrade" code and re-use that here so the behaviour is consistent.

---
 scripts/lib/devtool/__init__.py | 18 ++++++++++++++++++
 scripts/lib/devtool/standard.py | 30 +++++++++++++++---------------
 scripts/lib/devtool/upgrade.py  | 20 +++-----------------
 3 files changed, 36 insertions(+), 32 deletions(-)
scripts/lib/devtool/__init__.py +18 -0
index 3ea3802..07a3636 100644
@@ -152,3 +152,21 @@ def check_workspace_recipe(workspace, pn, checksrc=True):
             raise DevtoolError("Source tree %s for recipe %s does not exist" % (srctree, pn))
         if not os.listdir(srctree):
             raise DevtoolError("Source tree %s for recipe %s is empty" % (srctree, pn))
+
+def use_external_build(same_dir, no_same_dir, d):
+    """
+    Determine if we should use B!=S (separate build and source directories) or not
+    """
+    b_is_s = True
+    if no_same_dir:
+        logger.info('Using separate build directory since --no-same-dir specified')
+        b_is_s = False
+    elif same_dir:
+        logger.info('Using source tree as build directory since --same-dir specified')
+    elif bb.data.inherits_class('autotools-brokensep', d):
+        logger.info('Using source tree as build directory since recipe inherits autotools-brokensep')
+    elif d.getVar('B', True) == os.path.abspath(d.getVar('S', True)):
+        logger.info('Using source tree as build directory since that would be the default for this recipe')
+    else:
+        b_is_s = False
+    return b_is_s
scripts/lib/devtool/standard.py +15 -15
index ec21b3c..700a56b 100644
@@ -25,7 +25,7 @@ import logging
 import argparse
 import scriptutils
 import errno
-from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, DevtoolError
+from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, DevtoolError
 from devtool import parse_recipe
 
 logger = logging.getLogger('devtool')
@@ -107,17 +107,26 @@ def add(args, config, basepath, workspace):
         (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srctree)
         initial_rev = stdout.rstrip()
 
+    tinfoil = setup_tinfoil(config_only=True)
+    rd = oe.recipeutils.parse_recipe(recipefile, None, tinfoil.config_data)
+    if not rd:
+        return 1
+
     appendfile = os.path.join(appendpath, '%s.bbappend' % bp)
     with open(appendfile, 'w') as f:
         f.write('inherit externalsrc\n')
         f.write('EXTERNALSRC = "%s"\n' % srctree)
-        if args.same_dir:
+
+        b_is_s = use_external_build(args.same_dir, args.no_same_dir, rd)
+        if b_is_s:
             f.write('EXTERNALSRC_BUILD = "%s"\n' % srctree)
         if initial_rev:
             f.write('\n# initial_rev: %s\n' % initial_rev)
 
     _add_md5(config, args.recipename, appendfile)
 
+    tinfoil.shutdown()
+
     return 0
 
 
@@ -483,18 +492,7 @@ def modify(args, config, basepath, workspace):
         f.write('# NOTE: We use pn- overrides here to avoid affecting multiple variants in the case where the recipe uses BBCLASSEXTEND\n')
         f.write('EXTERNALSRC_pn-%s = "%s"\n' % (args.recipename, srctree))
 
-        b_is_s = True
-        if args.no_same_dir:
-            logger.info('using separate build directory since --no-same-dir specified')
-            b_is_s = False
-        elif args.same_dir:
-            logger.info('using source tree as build directory since --same-dir specified')
-        elif bb.data.inherits_class('autotools-brokensep', rd):
-            logger.info('using source tree as build directory since original recipe inherits autotools-brokensep')
-        elif rd.getVar('B', True) == s:
-            logger.info('using source tree as build directory since that is the default for this recipe')
-        else:
-            b_is_s = False
+        b_is_s = use_external_build(args.same_dir, args.no_same_dir, rd)
         if b_is_s:
             f.write('EXTERNALSRC_BUILD_pn-%s = "%s"\n' % (args.recipename, srctree))
 
@@ -876,7 +874,9 @@ def register_commands(subparsers, context):
                                        description='Adds a new recipe')
     parser_add.add_argument('recipename', help='Name for new recipe to add')
     parser_add.add_argument('srctree', help='Path to external source tree')
-    parser_add.add_argument('--same-dir', '-s', help='Build in same directory as source', action="store_true")
+    group = parser_add.add_mutually_exclusive_group()
+    group.add_argument('--same-dir', '-s', help='Build in same directory as source', action="store_true")
+    group.add_argument('--no-same-dir', help='Force build in a separate build directory', action="store_true")
     parser_add.add_argument('--fetch', '-f', help='Fetch the specified URI and extract it to create the source tree', metavar='URI')
     parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)')
     parser_add.set_defaults(func=add)
scripts/lib/devtool/upgrade.py +3 -17
index 86443b0..6c1dfee 100644
@@ -29,7 +29,7 @@ import errno
 import bb
 import oe.recipeutils
 from devtool import standard
-from devtool import exec_build_env_command, setup_tinfoil, DevtoolError, parse_recipe
+from devtool import exec_build_env_command, setup_tinfoil, DevtoolError, parse_recipe, use_external_build
 
 logger = logging.getLogger('devtool')
 
@@ -126,21 +126,6 @@ def _rename_recipe_files(bpn, oldpv, newpv, path):
     _rename_recipe_dirs(oldpv, newpv, path)
     return _rename_recipe_file(bpn, oldpv, newpv, path)
 
-def _use_external_build(same_dir, no_same_dir, d):
-    b_is_s = True
-    if no_same_dir:
-        logger.info('using separate build directory since --no-same-dir specified')
-        b_is_s = False
-    elif same_dir:
-        logger.info('using source tree as build directory since --same-dir specified')
-    elif bb.data.inherits_class('autotools-brokensep', d):
-        logger.info('using source tree as build directory since original recipe inherits autotools-brokensep')
-    elif d.getVar('B', True) == os.path.abspath(d.getVar('S', True)):
-        logger.info('using source tree as build directory since that is the default for this recipe')
-    else:
-        b_is_s = False
-    return b_is_s
-
 def _write_append(rc, srctree, same_dir, no_same_dir, rev, workspace, d):
     """Writes an append file"""
     if not os.path.exists(rc):
@@ -161,7 +146,8 @@ def _write_append(rc, srctree, same_dir, no_same_dir, rev, workspace, d):
         f.write(('# NOTE: We use pn- overrides here to avoid affecting'
                  'multiple variants in the case where the recipe uses BBCLASSEXTEND\n'))
         f.write('EXTERNALSRC_pn-%s = "%s"\n' % (pn, srctree))
-        if _use_external_build(same_dir, no_same_dir, d):
+        b_is_s = use_external_build(same_dir, no_same_dir, d)
+        if b_is_s:
             f.write('EXTERNALSRC_BUILD_pn-%s = "%s"\n' % (pn, srctree))
         if rev:
             f.write('\n# initial_rev: %s\n' % rev)
Signature · 1 line
2.1.0
#15 lore raw

PATCH 14/27 recipetool: create: fix creating empty shell functions

In reply to Paul Eggleton

The shell considers empty functions to be a syntax error, so for template shell functions that contain only comments (or no lines at all) then add a : to act as a no-op which avoids the syntax error.

---
 scripts/lib/recipetool/create.py | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)
scripts/lib/recipetool/create.py +19 -3
index c4754db..99d9cc8 100644
@@ -46,10 +46,26 @@ class RecipeHandler():
             results.extend(glob.glob(os.path.join(path, spec)))
         return results
 
-    def genfunction(self, outlines, funcname, content):
-        outlines.append('%s () {' % funcname)
+    def genfunction(self, outlines, funcname, content, python=False, forcespace=False):
+        if python:
+            prefix = 'python '
+        else:
+            prefix = ''
+        outlines.append('%s%s () {' % (prefix, funcname))
+        if python or forcespace:
+            indent = '    '
+        else:
+            indent = '\t'
+        addnoop = not python
         for line in content:
-            outlines.append('\t%s' % line)
+            outlines.append('%s%s' % (indent, line))
+            if addnoop:
+                strippedline = line.lstrip()
+                if strippedline and not strippedline.startswith('#'):
+                    addnoop = False
+        if addnoop:
+            # Without this there'll be a syntax error
+            outlines.append('%s:' % indent)
         outlines.append('}')
         outlines.append('')
 
Signature · 1 line
2.1.0
#16 lore raw

PATCH 15/27 recipetool: create: fix handling of URIs containing #

In reply to Paul Eggleton

The # character in a URI denotes a fragment; we don't care about this since it is never supposed to be sent to the server, so remove it from the URI before actually trying to fetch it or use it in SRC_URI within the recipe.

(This has come up because download links on pypi.python.org seem to have a fragment containing the md5sum of the download; without stripping this off the fetcher will choke on it.)

---
 scripts/lib/recipetool/create.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
scripts/lib/recipetool/create.py +4 -2
index 99d9cc8..409b255 100644
@@ -23,6 +23,7 @@ import fnmatch
 import re
 import logging
 import scriptutils
+import urlparse
 
 logger = logging.getLogger('recipetool')
 
@@ -102,7 +103,8 @@ def create_recipe(args):
     srcrev = '${AUTOREV}'
     if '://' in args.source:
         # Fetch a URL
-        srcuri = args.source
+        fetchuri = urlparse.urldefrag(args.source)[0]
+        srcuri = fetchuri
         rev_re = re.compile(';rev=([^;]+)')
         res = rev_re.search(srcuri)
         if res:
@@ -111,7 +113,7 @@ def create_recipe(args):
         tempsrc = tempfile.mkdtemp(prefix='recipetool-')
         srctree = tempsrc
         logger.info('Fetching %s...' % srcuri)
-        checksums = scriptutils.fetch_uri(tinfoil.config_data, args.source, srctree, srcrev)
+        checksums = scriptutils.fetch_uri(tinfoil.config_data, fetchuri, srctree, srcrev)
         dirlist = os.listdir(srctree)
         if 'git.indirectionsymlink' in dirlist:
             dirlist.remove('git.indirectionsymlink')
Signature · 1 line
2.1.0
#17 lore raw

PATCH 16/27 devtool: show proper error when extracting source for recipes with disabled unpack task

In reply to Paul Eggleton

If you try to use "devtool modify -x" or "devtool extract" on a recipe where do_unpack has been set as noexec (e.g. glibc-locale), then we get an error because the expected source wasn't ever unpacked. Do a check up front for noexec being set on do_unpack and error out with a reasonable message if that's the case.

---
 scripts/lib/devtool/standard.py | 4 ++++
 1 file changed, 4 insertions(+)
scripts/lib/devtool/standard.py +4 -0
index 700a56b..72ec97b 100644
@@ -261,6 +261,10 @@ def _extract_source(srctree, keep_temp, devbranch, d):
             raise DevtoolError("output path %s already exists and is "
                                "non-empty" % srctree)
 
+    if 'noexec' in (d.getVarFlags('do_unpack', False) or []):
+        raise DevtoolError("The %s recipe has do_unpack disabled, unable to "
+                           "extract source" % pn)
+
     # Prepare for shutil.move later on
     bb.utils.mkdirhier(srctree)
     os.rmdir(srctree)
Signature · 1 line
2.1.0
#19 lore raw

PATCH 18/27 devtool: better handling for recipes that don't unpack source

In reply to Paul Eggleton

Some recipes don't extract any source (for example, opkg-keyrings). We were producing a traceback in this case because we weren't checking if the directory existed.

---
 scripts/lib/devtool/standard.py | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
scripts/lib/devtool/standard.py +5 -3
index 2dbe372..ff79c05 100644
@@ -334,9 +334,11 @@ def _extract_source(srctree, keep_temp, devbranch, d):
             else:
                 os.rmdir(patchdir)
 
-        if not os.listdir(srcsubdir):
-            raise DevtoolError("no source unpacked to S, perhaps the %s "
-                               "recipe doesn't use any source?" % pn)
+        if not os.path.exists(srcsubdir) or not os.listdir(srcsubdir):
+            raise DevtoolError("no source unpacked to S, either the %s "
+                               "recipe doesn't use any source or the "
+                               "correct source directory could not be "
+                               "determined" % pn)
 
         if not os.path.exists(os.path.join(srcsubdir, '.git')):
             bb.process.run('git init', cwd=srcsubdir)
Signature · 1 line
2.1.0
#18 lore raw

PATCH 17/27 devtool: fix extracting source for work-shared recipes

In reply to Paul Eggleton

Recipes that use work-shared (such as libgcc) are capable of unpacking the source, but it doesn't necessarily unpack to ${WORKDIR}/${BP}. Use the last part of the actual S value instead which is more likely to work.

---
 scripts/lib/devtool/standard.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/lib/devtool/standard.py +1 -1
index 72ec97b..2dbe372 100644
@@ -288,7 +288,7 @@ def _extract_source(srctree, keep_temp, devbranch, d):
             if bb.data.inherits_class('kernel', d):
                 crd.setVar('S', '${WORKDIR}/source')
             else:
-                crd.setVar('S', '${WORKDIR}/${BP}')
+                crd.setVar('S', '${WORKDIR}/%s' % os.path.basename(d.getVar('S', True)))
         if bb.data.inherits_class('kernel', d):
             # We don't want to move the source to STAGING_KERNEL_DIR here
             crd.setVar('STAGING_KERNEL_DIR', '${S}')
Signature · 1 line
2.1.0
#20 lore raw

PATCH 19/27 devtool: add: set up fetched source as a git repository by default

In reply to Paul Eggleton

If the fetched source isn't already a git repository, initialise it as one and then branch and tag, just as we do with "devtool modify". This makes it easier to make changes, commit them and then use the "devtool update-recipe" command to turn those commits into patches on the recipe.

---
 scripts/lib/devtool/__init__.py | 17 +++++++++++++++++
 scripts/lib/devtool/standard.py | 13 ++++++-------
 2 files changed, 23 insertions(+), 7 deletions(-)
scripts/lib/devtool/__init__.py +17 -0
index 07a3636..f815ef2 100644
@@ -170,3 +170,20 @@ def use_external_build(same_dir, no_same_dir, d):
     else:
         b_is_s = False
     return b_is_s
+
+def setup_git_repo(repodir, version, devbranch, basetag='devtool-base'):
+    """
+    Set up the git repository for the source tree
+    """
+    import bb.process
+    if not os.path.exists(os.path.join(repodir, '.git')):
+        bb.process.run('git init', cwd=repodir)
+        bb.process.run('git add .', cwd=repodir)
+        if version:
+            commitmsg = "Initial commit from upstream at version %s" % version
+        else:
+            commitmsg = "Initial commit from upstream"
+        bb.process.run('git commit -q -m "%s"' % commitmsg, cwd=repodir)
+
+    bb.process.run('git checkout -b %s' % devbranch, cwd=repodir)
+    bb.process.run('git tag -f %s' % basetag, cwd=repodir)
scripts/lib/devtool/standard.py +6 -7
index ff79c05..4a05c1d 100644
@@ -25,7 +25,7 @@ import logging
 import argparse
 import scriptutils
 import errno
-from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, DevtoolError
+from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, setup_git_repo, DevtoolError
 from devtool import parse_recipe
 
 logger = logging.getLogger('devtool')
@@ -102,6 +102,9 @@ def add(args, config, basepath, workspace):
 
     _add_md5(config, args.recipename, recipefile)
 
+    if args.fetch and not args.no_git:
+        setup_git_repo(srctree, args.version, 'devtool')
+
     initial_rev = None
     if os.path.exists(os.path.join(srctree, '.git')):
         (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srctree)
@@ -340,16 +343,11 @@ def _extract_source(srctree, keep_temp, devbranch, d):
                                "correct source directory could not be "
                                "determined" % pn)
 
-        if not os.path.exists(os.path.join(srcsubdir, '.git')):
-            bb.process.run('git init', cwd=srcsubdir)
-            bb.process.run('git add .', cwd=srcsubdir)
-            bb.process.run('git commit -q -m "Initial commit from upstream at version %s"' % crd.getVar('PV', True), cwd=srcsubdir)
+        setup_git_repo(srcsubdir, crd.getVar('PV', True), devbranch)
 
         (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srcsubdir)
         initial_rev = stdout.rstrip()
 
-        bb.process.run('git checkout -b %s' % devbranch, cwd=srcsubdir)
-        bb.process.run('git tag -f devtool-base', cwd=srcsubdir)
         crd.setVar('PATCHTOOL', 'git')
 
         logger.info('Patching...')
@@ -885,6 +883,7 @@ def register_commands(subparsers, context):
     group.add_argument('--no-same-dir', help='Force build in a separate build directory', action="store_true")
     parser_add.add_argument('--fetch', '-f', help='Fetch the specified URI and extract it to create the source tree', metavar='URI')
     parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)')
+    parser_add.add_argument('--no-git', '-g', help='If -f/--fetch is specified, do not set up source tree as a git repository', action="store_true")
     parser_add.set_defaults(func=add)
 
     parser_modify = subparsers.add_parser('modify', help='Modify the source for an existing recipe',
Signature · 1 line
2.1.0
#21 lore raw

PATCH 20/27 devtool: add: move important "recipe created" message to the end

In reply to Paul Eggleton

If we end up printing a message about the build directory being the same as the source, we should print that first and then print the message about the recipe file possibly needing to be edited to the end so that it has slightly more impact.

---
 scripts/lib/devtool/standard.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
scripts/lib/devtool/standard.py +2 -1
index 4a05c1d..c254132 100644
@@ -96,7 +96,6 @@ def add(args, config, basepath, workspace):
         extracmdopts += ' -V %s' % args.version
     try:
         stdout, _ = exec_build_env_command(config.init_path, basepath, 'recipetool --color=%s create -o %s "%s" %s' % (color, recipefile, source, extracmdopts))
-        logger.info('Recipe %s has been automatically created; further editing may be required to make it fully functional' % recipefile)
     except bb.process.ExecutionError as e:
         raise DevtoolError('Command \'%s\' failed:\n%s' % (e.command, e.stdout))
 
@@ -128,6 +127,8 @@ def add(args, config, basepath, workspace):
 
     _add_md5(config, args.recipename, appendfile)
 
+    logger.info('Recipe %s has been automatically created; further editing may be required to make it fully functional' % recipefile)
+
     tinfoil.shutdown()
 
     return 0
Signature · 1 line
2.1.0
#23 lore raw

PATCH 22/27 devtool: build-image: tell user where to find output files

In reply to Paul Eggleton

If the user is running "devtool build-image" within the extensible SDK then they probably won't know where to find the resulting output files, so we should tell them explicitly.

---
 scripts/lib/devtool/build-image.py | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)
scripts/lib/devtool/build-image.py +9 -5
index fa6f0d7..5759bec 100644
@@ -25,10 +25,9 @@ from devtool import exec_build_env_command, setup_tinfoil, parse_recipe
 
 logger = logging.getLogger('devtool')
 
-def _get_packages(workspace, config):
+def _get_packages(tinfoil, workspace, config):
     """Get list of packages from recipes in the workspace."""
     result = []
-    tinfoil = setup_tinfoil()
     for recipe in workspace:
         data = parse_recipe(config, tinfoil, recipe, True)
         if 'class-target' in data.getVar('OVERRIDES', True).split(':'):
@@ -37,7 +36,6 @@ def _get_packages(workspace, config):
             else:
                 logger.warning("Skipping recipe %s as it doesn't produce a "
                                "package with the same name", recipe)
-    tinfoil.shutdown()
     return result
 
 def build_image(args, config, basepath, workspace):
@@ -51,8 +49,9 @@ def build_image(args, config, basepath, workspace):
     if os.path.isfile(appendfile):
         os.unlink(appendfile)
 
+    tinfoil = setup_tinfoil()
     if workspace:
-        packages = _get_packages(workspace, config)
+        packages = _get_packages(tinfoil, workspace, config)
         if packages:
             with open(appendfile, 'w') as afile:
                 # include packages from workspace recipes into the image
@@ -75,6 +74,10 @@ def build_image(args, config, basepath, workspace):
     else:
         logger.warning('No recipes in workspace, building image %s unmodified', image)
 
+    deploy_dir_image = tinfoil.config_data.getVar('DEPLOY_DIR_IMAGE', True)
+
+    tinfoil.shutdown()
+
     # run bitbake to build image
     try:
         exec_build_env_command(config.init_path, basepath,
@@ -82,7 +85,8 @@ def build_image(args, config, basepath, workspace):
     except ExecutionError as err:
         return err.exitcode
 
-    logger.info('Successfully built %s', image)
+    logger.info('Successfully built %s. You can find output files in %s'
+                % (image, deploy_dir_image))
 
 def register_commands(subparsers, context):
     """Register devtool subcommands from the build-image plugin"""
Signature · 1 line
2.1.0
#22 lore raw

PATCH 21/27 devtool: build-image: fix recipe/package terminology

In reply to Paul Eggleton

We build recipes and include packages into the image, adjust the terminology used in code and messages accordingly. Also fix a few typos.

---
 scripts/lib/devtool/build-image.py | 45 ++++++++++++++++++++------------------
 1 file changed, 24 insertions(+), 21 deletions(-)
scripts/lib/devtool/build-image.py +24 -21
index 2c01428..fa6f0d7 100644
@@ -25,8 +25,8 @@ from devtool import exec_build_env_command, setup_tinfoil, parse_recipe
 
 logger = logging.getLogger('devtool')
 
-def _get_recipes(workspace, config):
-    """Get list of target recipes from the workspace."""
+def _get_packages(workspace, config):
+    """Get list of packages from recipes in the workspace."""
     result = []
     tinfoil = setup_tinfoil()
     for recipe in workspace:
@@ -35,7 +35,7 @@ def _get_recipes(workspace, config):
             if recipe in data.getVar('PACKAGES', True):
                 result.append(recipe)
             else:
-                logger.warning("Skipping recipe %s as it doesn't produce "
+                logger.warning("Skipping recipe %s as it doesn't produce a "
                                "package with the same name", recipe)
     tinfoil.shutdown()
     return result
@@ -46,29 +46,32 @@ def build_image(args, config, basepath, workspace):
     appendfile = os.path.join(config.workspace_path, 'appends',
                               '%s.bbappend' % image)
 
-    # remove <image>.bbapend to make sure setup_tinfoil doesn't
-    # breake because of it
+    # remove <image>.bbappend to make sure setup_tinfoil doesn't
+    # break because of it
     if os.path.isfile(appendfile):
         os.unlink(appendfile)
 
-    recipes = _get_recipes(workspace, config)
-    if recipes:
-        with open(appendfile, 'w') as afile:
-            # include selected recipes into the image
-            afile.write('IMAGE_INSTALL_append = " %s"\n' % ' '.join(recipes))
+    if workspace:
+        packages = _get_packages(workspace, config)
+        if packages:
+            with open(appendfile, 'w') as afile:
+                # include packages from workspace recipes into the image
+                afile.write('IMAGE_INSTALL_append = " %s"\n' % ' '.join(packages))
 
-            # Generate notification callback devtool_warn_image_extended
-            afile.write('do_rootfs[prefuncs] += "devtool_warn_image_extended"\n\n')
-            afile.write("python devtool_warn_image_extended() {\n")
-            afile.write("    bb.plain('NOTE: %%s: building with additional '\n"
-                        "             'packages due to \"devtool build-image\"'"
-                        "              %% d.getVar('PN', True))\n"
-                        "    bb.plain('NOTE: delete %%s to clear this' %% \\\n"
-                        "             '%s')\n" % os.path.relpath(appendfile, basepath))
-            afile.write("}\n")
+                # Generate notification callback devtool_warn_image_extended
+                afile.write('do_rootfs[prefuncs] += "devtool_warn_image_extended"\n\n')
+                afile.write("python devtool_warn_image_extended() {\n")
+                afile.write("    bb.plain('NOTE: %%s: building with additional '\n"
+                            "             'packages due to \"devtool build-image\"'"
+                            "              %% d.getVar('PN', True))\n"
+                            "    bb.plain('NOTE: delete %%s to clear this' %% \\\n"
+                            "             '%s')\n" % os.path.relpath(appendfile, basepath))
+                afile.write("}\n")
 
-            logger.info('Building image %s with the following '
-                        'additional packages: %s', image, ' '.join(recipes))
+                logger.info('Building image %s with the following '
+                            'additional packages: %s', image, ' '.join(packages))
+        else:
+            logger.warning('No packages to add, building image %s unmodified', image)
     else:
         logger.warning('No recipes in workspace, building image %s unmodified', image)
 
Signature · 1 line
2.1.0
#24 lore raw

PATCH 23/27 devtool: build-image: improve image recipe handling

In reply to Paul Eggleton

* Make image optional for the extensible SDK (auto-determine it based on the targets the SDK was built for)
* Check that specified recipe is in fact an image

---
 scripts/lib/devtool/build-image.py | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)
scripts/lib/devtool/build-image.py +23 -3
index 5759bec..a6c7d81 100644
@@ -21,7 +21,7 @@ import os
 import logging
 
 from bb.process import ExecutionError
-from devtool import exec_build_env_command, setup_tinfoil, parse_recipe
+from devtool import exec_build_env_command, setup_tinfoil, parse_recipe, DevtoolError
 
 logger = logging.getLogger('devtool')
 
@@ -40,7 +40,17 @@ def _get_packages(tinfoil, workspace, config):
 
 def build_image(args, config, basepath, workspace):
     """Entry point for the devtool 'build-image' subcommand."""
-    image = args.recipe
+
+    image = args.imagename
+    auto_image = False
+    if not image:
+        sdk_targets = config.get('SDK', 'sdk_targets', '').split()
+        if sdk_targets:
+            image = sdk_targets[0]
+            auto_image = True
+    if not image:
+        raise DevtoolError('Unable to determine image to build, please specify one')
+
     appendfile = os.path.join(config.workspace_path, 'appends',
                               '%s.bbappend' % image)
 
@@ -50,6 +60,16 @@ def build_image(args, config, basepath, workspace):
         os.unlink(appendfile)
 
     tinfoil = setup_tinfoil()
+    rd = parse_recipe(config, tinfoil, image, True)
+    if not rd:
+        # Error already shown
+        return 1
+    if not bb.data.inherits_class('image', rd):
+        if auto_image:
+            raise DevtoolError('Unable to determine image to build, please specify one')
+        else:
+            raise DevtoolError('Specified recipe %s is not an image recipe' % image)
+
     if workspace:
         packages = _get_packages(tinfoil, workspace, config)
         if packages:
@@ -94,5 +114,5 @@ def register_commands(subparsers, context):
                                    help='Build image including workspace recipe packages',
                                    description='Builds an image, extending it to include '
                                    'packages from recipes in the workspace')
-    parser.add_argument('recipe', help='Image recipe to build')
+    parser.add_argument('imagename', help='Image recipe to build', nargs='?')
     parser.set_defaults(func=build_image)
Signature · 1 line
2.1.0
#25 lore raw

PATCH 24/27 devtool: build-image: delete bbappend at end of build

In reply to Paul Eggleton

Upon further reflection, it seems to me that this bbappend ought to just be deleted at the end of the build. This keeps things simple; you never have to remember to delete any files to get back to where you were before with the image. This means we can also drop the slightly awkward message reminding the user how to do that. I've also updated the test to look at the image manifest to determine if the command has worked instead of looking for the (now deleted) bbappend.

---
 meta/lib/oeqa/selftest/devtool.py  | 22 +++++++++++-----
 scripts/lib/devtool/build-image.py | 53 +++++++++++++++++---------------------
 2 files changed, 38 insertions(+), 37 deletions(-)
meta/lib/oeqa/selftest/devtool.py +15 -7
index 6e731d6..3a8168c 100644
@@ -919,7 +919,8 @@ class DevtoolTests(DevtoolBase):
         self.add_command_to_tearDown('bitbake -c clean %s' % image)
         bitbake('%s -c clean' % image)
         # Add target and native recipes to workspace
-        for recipe in ('mdadm', 'parted-native'):
+        recipes = ['mdadm', 'parted-native']
+        for recipe in recipes:
             tempdir = tempfile.mkdtemp(prefix='devtoolqa')
             self.track_for_cleanup(tempdir)
             self.add_command_to_tearDown('bitbake -c clean %s' % recipe)
@@ -927,12 +928,19 @@ class DevtoolTests(DevtoolBase):
         # Try to build image
         result = runCmd('devtool build-image %s' % image)
         self.assertNotEqual(result, 0, 'devtool build-image failed')
-        # Check if image.bbappend has required content
-        bbappend = os.path.join(workspacedir, 'appends', image+'.bbappend')
-        self.assertTrue(os.path.isfile(bbappend), 'bbappend not created %s' % result.output)
-        # NOTE: native recipe parted-native should not be in IMAGE_INSTALL_append
-        self.assertTrue('IMAGE_INSTALL_append = " mdadm"\n' in open(bbappend).readlines(),
-                        'IMAGE_INSTALL_append = " mdadm" not found in %s' % bbappend)
+        # Check if image contains expected packages
+        deploy_dir_image = get_bb_var('DEPLOY_DIR_IMAGE')
+        image_link_name = get_bb_var('IMAGE_LINK_NAME', image)
+        reqpkgs = [item for item in recipes if not item.endswith('-native')]
+        with open(os.path.join(deploy_dir_image, image_link_name + '.manifest'), 'r') as f:
+            for line in f:
+                splitval = line.split()
+                if splitval:
+                    pkg = splitval[0]
+                    if pkg in reqpkgs:
+                        reqpkgs.remove(pkg)
+        if reqpkgs:
+            self.fail('The following packages were not present in the image as expected: %s' % ', '.join(reqpkgs))
 
     def test_devtool_upgrade(self):
         # Check preconditions
scripts/lib/devtool/build-image.py +23 -30

63 lines not on this page · raw message

Signature · 1 line
2.1.0
#26 lore raw

PATCH 25/27 devtool / recipetool: add handling for binary-only packages

In reply to Paul Eggleton

Add a means of creating recipes for package files or archives that contain a directory structure to be installed verbatim, for example an rpm file. (We mostly just re-use bin_package here and skip some of the normal build system checks.) This support is available in "recipetool create" and "devtool add" which wraps the former.

---
 scripts/lib/devtool/standard.py  |  9 +++++++++
 scripts/lib/recipetool/create.py | 14 ++++++++++++++
 2 files changed, 23 insertions(+)
scripts/lib/devtool/standard.py +9 -0
index c254132..96b271c 100644
@@ -94,6 +94,8 @@ def add(args, config, basepath, workspace):
         source = srctree
     if args.version:
         extracmdopts += ' -V %s' % args.version
+    if args.binary:
+        extracmdopts += ' -b'
     try:
         stdout, _ = exec_build_env_command(config.init_path, basepath, 'recipetool --color=%s create -o %s "%s" %s' % (color, recipefile, source, extracmdopts))
     except bb.process.ExecutionError as e:
@@ -125,6 +127,12 @@ def add(args, config, basepath, workspace):
         if initial_rev:
             f.write('\n# initial_rev: %s\n' % initial_rev)
 
+        if args.binary:
+            f.write('do_install_append() {\n')
+            f.write('    rm -rf ${D}/.git\n')
+            f.write('    rm -f ${D}/singletask.lock\n')
+            f.write('}\n')
+
     _add_md5(config, args.recipename, appendfile)
 
     logger.info('Recipe %s has been automatically created; further editing may be required to make it fully functional' % recipefile)
@@ -885,6 +893,7 @@ def register_commands(subparsers, context):
     parser_add.add_argument('--fetch', '-f', help='Fetch the specified URI and extract it to create the source tree', metavar='URI')
     parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)')
     parser_add.add_argument('--no-git', '-g', help='If -f/--fetch is specified, do not set up source tree as a git repository', action="store_true")
+    parser_add.add_argument('--binary', '-b', help='Treat the source tree as something that should be installed verbatim (no compilation, same directory structure)', action='store_true')
     parser_add.set_defaults(func=add)
 
     parser_modify = subparsers.add_parser('modify', help='Modify the source for an existing recipe',
scripts/lib/recipetool/create.py +14 -0
index 409b255..844073b 100644
@@ -104,6 +104,10 @@ def create_recipe(args):
     if '://' in args.source:
         # Fetch a URL
         fetchuri = urlparse.urldefrag(args.source)[0]
+        if args.binary:
+            # Assume the archive contains the directory structure verbatim
+            # so we need to extract to a subdirectory
+            fetchuri += ';subdir=%s' % os.path.splitext(os.path.basename(urlparse.urlsplit(fetchuri).path))[0]
         srcuri = fetchuri
         rev_re = re.compile(';rev=([^;]+)')
         res = rev_re.search(srcuri)
@@ -226,6 +230,10 @@ def create_recipe(args):
         lines_after.append('PACKAGE_ARCH = "%s"' % pkgarch)
         lines_after.append('')
 
+    if args.binary:
+        lines_after.append('INSANE_SKIP_${PN} += "already-stripped"')
+        lines_after.append('')
+
     # Find all plugins that want to register handlers
     handlers = []
     for plugin in plugins:
@@ -235,6 +243,11 @@ def create_recipe(args):
     # Apply the handlers
     classes = []
     handled = []
+
+    if args.binary:
+        classes.append('bin_package')
+        handled.append('buildsystem')
+
     for handler in handlers:
         handler.process(srctree, classes, lines_before, lines_after, handled)
 
@@ -426,5 +439,6 @@ def register_command(subparsers):
     parser_create.add_argument('-m', '--machine', help='Make recipe machine-specific as opposed to architecture-specific', action='store_true')
     parser_create.add_argument('-x', '--extract-to', metavar='EXTRACTPATH', help='Assuming source is a URL, fetch it and extract it to the directory specified as %(metavar)s')
     parser_create.add_argument('-V', '--version', help='Version to use within recipe (PV)')
+    parser_create.add_argument('-b', '--binary', help='Treat the source tree as something that should be installed verbatim (no compilation, same directory structure)', action='store_true')
     parser_create.set_defaults(func=create_recipe)
 
Signature · 1 line
2.1.0
#27 lore raw

PATCH 26/27 devtool: add basic means of running runqemu within the extensible SDK

In reply to Paul Eggleton

We ship the runqemu script and if we build QEMU itself within the extensible SDK, then it would be nice to be able to run it. This is a very thin wrapper around runqemu, supplying the machine and image name so the user doesn't need to. (This subcommand is only available within the extensible SDK since it only really makes sense there where it is otherwise hard to run runqemu directly.)

Implements [YOCTO #6657].

---
 scripts/lib/devtool/runqemu.py | 64 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 64 insertions(+)
 create mode 100644 scripts/lib/devtool/runqemu.py
scripts/lib/devtool/runqemu.py new file +64 -0
new file mode 100644
index 0000000..e7f26ab
@@ -0,0 +1,64 @@
+# Development tool - runqemu command plugin
+#
+# Copyright (C) 2015 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+"""Devtool runqemu plugin"""
+
+import os
+import bb
+import logging
+import argparse
+import glob
+from devtool import exec_build_env_command, setup_tinfoil, DevtoolError
+
+logger = logging.getLogger('devtool')
+
+def runqemu(args, config, basepath, workspace):
+    """Entry point for the devtool 'runqemu' subcommand"""
+
+    tinfoil = setup_tinfoil()
+    machine = tinfoil.config_data.getVar('MACHINE', True)
+    bindir_native = tinfoil.config_data.getVar('STAGING_BINDIR_NATIVE', True)
+    tinfoil.shutdown()
+
+    if not glob.glob(os.path.join(bindir_native, 'qemu-system-*')):
+        raise DevtoolError('QEMU is not available within this SDK')
+
+    imagename = args.imagename
+    if not imagename:
+        sdk_targets = config.get('SDK', 'sdk_targets', '').split()
+        if sdk_targets:
+            imagename = sdk_targets[0]
+    if not imagename:
+        raise DevtoolError('Unable to determine image name to run, please specify one')
+
+    try:
+        exec_build_env_command(config.init_path, basepath, 'runqemu %s %s %s' % (machine, imagename, " ".join(args.args)), watch=True)
+    except bb.process.ExecutionError as e:
+        # We've already seen the output since watch=True, so just ensure we return something to the user
+        return e.exitcode
+
+    return 0
+
+def register_commands(subparsers, context):
+    """Register devtool subcommands from this plugin"""
+    if context.fixed_setup:
+        parser_runqemu = subparsers.add_parser('runqemu', help='Run QEMU on the specified image',
+                                               description='Runs QEMU to boot the specified image')
+        parser_runqemu.add_argument('imagename', help='Name of built image to boot within QEMU', nargs='?')
+        parser_runqemu.add_argument('args', help='Any remaining arguments are passed to the runqemu script (pass --help after imagename to see what these are)',
+                                    nargs=argparse.REMAINDER)
+        parser_runqemu.set_defaults(func=runqemu)
Signature · 1 line
2.1.0
#28 lore raw

PATCH 27/27 devtool: add search command

In reply to Paul Eggleton

Adds a subcommand to search to find the target recipe name providing some file or capability. This is implemented by searching on recipe name, package name, description, package contents (file names), and runtime file provides. For example:

$ devtool search libGL
mesa
$ devtool search X11
xextproto
libxxf86vm
xf86driproto
xf86vidmodeproto
libxfixes
xproto
libx11
...
$ devtool search /bin/sed
busybox
sed

This is particularly useful within the extensible SDK but is also made available in devtool alongside the build system.

Note of course that because this searches pkgdata, useful results depend upon do_packagedata(_setscene) having executed for the recipe being searched for.

---
 scripts/lib/devtool/search.py | 80 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 80 insertions(+)
 create mode 100644 scripts/lib/devtool/search.py
scripts/lib/devtool/search.py new file +80 -0
new file mode 100644
index 0000000..e6ae922
@@ -0,0 +1,80 @@
+# Development tool - search command plugin
+#
+# Copyright (C) 2015 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+"""Devtool search plugin"""
+
+import os
+import bb
+import logging
+import argparse
+import re
+from devtool import setup_tinfoil, DevtoolError
+
+logger = logging.getLogger('devtool')
+
+def search(args, config, basepath, workspace):
+    """Entry point for the devtool 'search' subcommand"""
+
+    tinfoil = setup_tinfoil(config_only=True)
+    pkgdata_dir = tinfoil.config_data.getVar('PKGDATA_DIR', True)
+    tinfoil.shutdown()
+
+    keyword_rc = re.compile(args.keyword)
+
+    for fn in os.listdir(pkgdata_dir):
+        pfn = os.path.join(pkgdata_dir, fn)
+        if not os.path.isfile(pfn):
+            continue
+
+        packages = []
+        match = False
+        if keyword_rc.search(fn):
+            match = True
+
+        if not match:
+            with open(pfn, 'r') as f:
+                for line in f:
+                    if line.startswith('PACKAGES:'):
+                        packages = line.split(':', 1)[1].strip().split()
+
+            for pkg in packages:
+                if keyword_rc.search(pkg):
+                    match = True
+                    break
+                if os.path.exists(os.path.join(pkgdata_dir, 'runtime', pkg + '.packaged')):
+                    with open(os.path.join(pkgdata_dir, 'runtime', pkg), 'r') as f:
+                        for line in f:
+                            if ': ' in line:
+                                splitline = line.split(':', 1)
+                                key = splitline[0]
+                                value = splitline[1].strip()
+                            if key in ['PKG_%s' % pkg, 'DESCRIPTION', 'FILES_INFO'] or key.startswith('FILERPROVIDES_'):
+                                if keyword_rc.search(value):
+                                    match = True
+                                    break
+
+        if match:
+            print(fn)
+
+    return 0
+
+def register_commands(subparsers, context):
+    """Register devtool subcommands from this plugin"""
+    parser_search = subparsers.add_parser('search', help='Search available recipes',
+                                            description='Searches for available target recipes. Matches on recipe name, package name, description and installed files, and prints the recipe name on match.')
+    parser_search.add_argument('keyword', help='Keyword to search for (regular expression syntax allowed)')
+    parser_search.set_defaults(func=search)
Signature · 1 line
2.1.0
#2 lore raw

Re: [PATCH 01/27] lib/oe/recipeutils: properly split unexpanded variable values

In reply to Paul Eggleton
Quoted 6 lines · Paul Eggleton

Variables such as SRC_URI which are space-separated may also contain Python expressions (${@...}) which themselves contain spaces that shouldn't be split when splitting the value into items. In order to ensure this we need to use a custom splitting function instead of just string.split().

Quoted 9 lines

This issue could be seen when doing "devtool modify sudo", adding a commit to the resulting source repository then "devtool update-recipe" - the Python expression in SRC_URI was being unnecessarily broken onto multiple lines.

Fixes [YOCTO #8046].

This is interesting, I recently had a need for a split like that too. I had written a prototype class which automatically disables all packageconfigs which introduce optional dependencies, to see how much it would reduce the dependencies, and wanted to ignore bits dependent on DISTRO_FEATURES, so wanted to ignore all inline python.
https://gist.github.com/kergoth/2c556d9e34cbcd03cb82#file-packageconfig-backdel-bbclass-L107-L122
is my version of it. Possibly overly complex, since my version produces an extremely lightweight parse tree (list-based) for nested references, and then has to flatten it back out. I found it interesting that we both had a need for this sort of functionality -- perhaps a split function along these lines would be of use in one of the oe python package modules.

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics