Developer workflow improvements
cover Developer workflow improvements
[Note: this is an RFC series for the moment, and shouldn't yet be merged.]
I've been looking at how to make it easier for application and system component developers to get their work done using the tools we provide, and I believe this patchset is a piece of the solution. There's still a number of other pieces to come, but this should be usable on its own.
The first three patches extend the PATCHTOOL = "git" code to ensure that we're able to apply all patches even if "git am" and "git apply" can't handle them, and that we do a commit to the repository per patch. This is needed for devtool later on.
I've then added a new recipe auto-creation script, recipetool, which can take a source tree or URL and create a skeleton recipe to build it. (In case anyone is wondering about the existing scripts/create-recipe, frankly I consider it a dead end - it's written in Perl, which makes it a bit difficult to integrate with the rest of our code; it's also GPLv3-only which makes any such integration pretty much impossible from another angle.)
Then we add devtool. This allows you to:
* Add a new piece of software (auto-create the skeleton of a recipe using the aforementioned recipetool and point the build to an external source tree)
* Modify the source for an existing recipe (point the build to an external source tree, possibly creating that tree in the same step and managing it with git)
* Deploy the installed files from a recipe from ${D} to a target using scp.
There will obviously be extensions to these tools, but I hope they are already functional enough to be useful at the state they are in at the moment.
Known issues:
* "devtool modify" can't currently deal with gcc, linux-yocto or perf because they manage their workdir differently, this will need to be fixed
* devtool's workspace layer triggers a bitbake warning when empty. I suspect a mechanism to disable this warning internally will need to be added.
* "recipetool create" ideally needs to become smarter and fill in more details of the recipe. At some point we'll probably have to make the process interactive and possibly have it automate parts of the build process and examine the output; some things just can't be practically detected from the source tree without building.
(Note that some pieces of the code which are not essential rely on a couple of changes to bitbake, you can find these in the paule/devtool-bb branch in poky-contrib, or if you are happy to use poky, grab the paule/devtool branch in poky-contrib which has everything).
Comments/questions/suggestions welcome!
The following changes since commit ff2ff155ea5273b2023a1c9834b13f10249d343f:
gdk-pixbuf: use ptest-gnome (2014-11-25 12:58:21 +0000)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib paule/devtool
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool
Paul Eggleton (6):
lib/oe/patch: fall back to patch if git apply fails lib/oe/patch: auto-commit when falling back from git am lib/oe/patch: use --keep-cr with git am scripts/recipetool: Add a recipe auto-creation script lib/oe: add recipeutils module scripts/devtool: add development helper tool
Junchun Guan (1):
scripts/devtool: Support deploy/undeploy function
meta/lib/oe/patch.py | 94 ++++++- meta/lib/oe/recipeutils.py | 264 ++++++++++++++++++++ scripts/devtool | 223 +++++++++++++++++ scripts/lib/devtool/__init__.py | 79 ++++++ scripts/lib/devtool/deploy.py | 100 ++++++++ scripts/lib/devtool/standard.py | 390 ++++++++++++++++++++++++++++++ scripts/lib/recipetool/__init__.py | 0 scripts/lib/recipetool/create.py | 375 ++++++++++++++++++++++++++++ scripts/lib/recipetool/create_buildsys.py | 234 ++++++++++++++++++ scripts/recipetool | 124 ++++++++++ 10 files changed, 1881 insertions(+), 2 deletions(-) create mode 100644 meta/lib/oe/recipeutils.py create mode 100755 scripts/devtool create mode 100644 scripts/lib/devtool/__init__.py create mode 100644 scripts/lib/devtool/deploy.py create mode 100644 scripts/lib/devtool/standard.py create mode 100644 scripts/lib/recipetool/__init__.py create mode 100644 scripts/lib/recipetool/create.py create mode 100644 scripts/lib/recipetool/create_buildsys.py create mode 100755 scripts/recipetool
Signature · 1 line
1.9.3
RFC PATCH 1/7 lib/oe/patch: fall back to patch if git apply fails
When PATCHTOOL = "git", git apply doesn't support fuzzy application, so if a patch requires that it's better to be able to apply it rather than just failing.
--- meta/lib/oe/patch.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
meta/lib/oe/patch.py +5 -1
@@ -219,7 +219,11 @@ class GitApplyTree(PatchTree): return _applypatchhelper(shellcmd, patch, force, reverse, run) except CmdError: shellcmd = ["git", "--git-dir=.", "apply", "-p%s" % patch['strippath']] - return _applypatchhelper(shellcmd, patch, force, reverse, run) + try: + output = _applypatchhelper(shellcmd, patch, force, reverse, run) + except CmdError: + output = PatchTree._applypatch(self, patch, force, reverse, run) + return output class QuiltTree(PatchSet):
Signature · 1 line
1.9.3
Re: [RFC PATCH 1/7] lib/oe/patch: fall back to patch if git apply fails
Quoted 14 lines · Paul Eggleton
When PATCHTOOL = "git", git apply doesn't support fuzzy application, so if a patch requires that it's better to be able to apply it rather than just failing.
Signed-off-by Paul Eggleton <[email protected]>--- meta/lib/oe/patch.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)meta/lib/oe/patch.py +0
-0
meta/lib/oe/patch.py class GitApplyTree(PatchTree): · line 227
Quoted 10 lines
meta/lib/oe/patch.py +5
-1@@ -219,7 +219,11 @@ class GitApplyTree(PatchTree): return _applypatchhelper(shellcmd, patch, force, reverse, run) except CmdError: shellcmd = ["git", "--git-dir=.", "apply", "-p%s" % patch['strippath']] - return _applypatchhelper(shellcmd, patch, force, reverse, run) + try: + output = _applypatchhelper(shellcmd, patch, force, reverse, run) + except CmdError: + output = PatchTree._applypatch(self, patch, force, reverse, run) + return output
Would this give a warning or other notification if the fallback to 'patch' is used? When developing patches it'd probably be good to know whether they're exactly correct or not.
I haven't looked at _applypatchhelper or _applypatch so I'm not sure if there's some notification hidden in there.
Cheers,
Signature · 4 lines
Paul Barker Email: [email protected] http://www.paulbarker.me.uk
Re: [RFC PATCH 1/7] lib/oe/patch: fall back to patch if git apply fails
Hi Paul,
Quoted 26 lines · On Tuesday 25 November 2014 17:40:24 Paul Barker wrote:
On 25 November 2014 at 17:28, Paul Eggleton
Quoted 23 lines · <[email protected]> wrote:
When PATCHTOOL = "git", git apply doesn't support fuzzy application, so if a patch requires that it's better to be able to apply it rather than just failing.
Signed-off-by Paul Eggleton <[email protected]>---
meta/lib/oe/patch.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)meta/lib/oe/patch.py +0
-0@@ -219,7 +219,11 @@ class GitApplyTree(PatchTree): return _applypatchhelper(shellcmd, patch, force, reverse, run) except CmdError: shellcmd = ["git", "--git-dir=.", "apply", "-p%s" % patch['strippath']]
Quoted 11 lines
Quoted 7 lines
- return _applypatchhelper(shellcmd, patch, force, reverse,
run)meta/lib/oe/patch.py +2
-0+ try: + output = _applypatchhelper(shellcmd, patch, force,reverse, run) + except CmdError:
+ output = PatchTree._applypatch(self, patch, force,
reverse, run) + return outputWould this give a warning or other notification if the fallback to 'patch' is used? When developing patches it'd probably be good to know whether they're exactly correct or not.
In the context I'm attempting to use it I'd want it not to warn in this case, but I can see that in normal usage if you've explicitly elected to use PATCHTOOL = "git" it could be that you do expect all patches for the recipe to apply that way and want to see a warning if they don't.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
RFC PATCH 2/7 lib/oe/patch: auto-commit when falling back from git am
In reply to Paul EggletonWhen PATCHTOOL = "git", if we're not able to use "git am" to apply a patch and fall back to "git apply" or "patch", it is desirable to actually commit the changes, attempting to preserve (and interpret) the patch header as part of the commit message if present. As a bonus, the code for extracting the commit message is callable externally in case it is useful elsewhere.
--- meta/lib/oe/patch.py | 86 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+)
meta/lib/oe/patch.py +86 -0
@@ -202,6 +202,78 @@ class GitApplyTree(PatchTree): def __init__(self, dir, d): PatchTree.__init__(self, dir, d) + @staticmethod + def extractPatchHeader(patchfile): + """ + Extract just the header lines from the top of a patch file + """ + lines = [] + with open(patchfile, 'r') as f: + for line in f.readlines(): + if line.startswith('Index: ') or line.startswith('diff -') or line.startswith('---'): + break + lines.append(line) + return lines + + @staticmethod + def prepareCommit(patchfile): + """ + Prepare a git commit command line based on the header from a patch file + (typically this is useful for patches that cannot be applied with "git am" due to formatting) + """ + import tempfile + import re + author_re = re.compile('[\S ]+ <\S+@\S+\.\S+>') + # Process patch header and extract useful information + lines = GitApplyTree.extractPatchHeader(patchfile) + outlines = [] + author = None + date = None + for line in lines: + if line.startswith('Subject: '): + subject = line.split(':', 1)[1] + # Remove any [PATCH][oe-core] etc. + subject = re.sub(r'\[.+?\]\s*', '', subject) + outlines.insert(0, '%s\n\n' % subject.strip()) + continue + if line.startswith('From: ') or line.startswith('Author: '): + authorval = line.split(':', 1)[1].strip().replace('"', '') + # git is fussy about author formatting i.e. it must be Name <email@domain> + if author_re.match(authorval): + author = authorval + continue + if line.startswith('Date: '): + if date is None: + dateval = line.split(':', 1)[1].strip() + # Very crude check for date format, since git will blow up if it's not in the right + # format. Without e.g. a python-dateutils dependency we can't do a whole lot more + if len(dateval) > 12: + date = dateval + continue + if line.startswith('Signed-off-by: '): + authorval = line.split(':', 1)[1].strip().replace('"', '') + # git is fussy about author formatting i.e. it must be Name <email@domain> + if author_re.match(authorval): + author = authorval + outlines.append(line) + # Add a pointer to the original patch file name + if outlines and outlines[-1].strip(): + outlines.append('\n') + outlines.append('(from original patch: %s)\n' % os.path.basename(patchfile)) + # Write out commit message to a file + with tempfile.NamedTemporaryFile('w', delete=False) as tf: + tmpfile = tf.name + for line in outlines: + tf.write(line) + # Prepare git command + cmd = ["git", "commit", "-F", tmpfile] + # git doesn't like plain email addresses as authors + if author and '<' in author: + cmd.append('--author="%s"' % author) + if date: + cmd.append('--date="%s"' % date) + return (tmpfile, cmd) + def _applypatch(self, patch, force = False, reverse = False, run = True): def _applypatchhelper(shellcmd, patch, force = False, reverse = False, run = True): if reverse:
@@ -218,11 +290,25 @@ class GitApplyTree(PatchTree): shellcmd = ["git", "--work-tree=.", "am", "-3", "-p%s" % patch['strippath']] return _applypatchhelper(shellcmd, patch, force, reverse, run) except CmdError: + # Fall back to git apply shellcmd = ["git", "--git-dir=.", "apply", "-p%s" % patch['strippath']] try: output = _applypatchhelper(shellcmd, patch, force, reverse, run) except CmdError: + # Fall back to patch output = PatchTree._applypatch(self, patch, force, reverse, run) + # Add all files + shellcmd = ["git", "add", "-f", "."] + output += runcmd(["sh", "-c", " ".join(shellcmd)], self.dir) + # Exclude the patches directory + shellcmd = ["git", "reset", "HEAD", self.patchdir] + output += runcmd(["sh", "-c", " ".join(shellcmd)], self.dir) + # Commit the result + (tmpfile, shellcmd) = self.prepareCommit(patch['file']) + try: + output += runcmd(["sh", "-c", " ".join(shellcmd)], self.dir) + finally: + os.remove(tmpfile) return output
Signature · 1 line
1.9.3
RFC PATCH 3/7 lib/oe/patch: use --keep-cr with git am
In reply to Paul EggletonPreserving carriage returns is important where the patch contains them.
--- meta/lib/oe/patch.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/patch.py +1 -1
@@ -287,7 +287,7 @@ class GitApplyTree(PatchTree): return runcmd(["sh", "-c", " ".join(shellcmd)], self.dir) try: - shellcmd = ["git", "--work-tree=.", "am", "-3", "-p%s" % patch['strippath']] + shellcmd = ["git", "--work-tree=.", "am", "-3", "--keep-cr", "-p%s" % patch['strippath']] return _applypatchhelper(shellcmd, patch, force, reverse, run) except CmdError: # Fall back to git apply
Signature · 1 line
1.9.3
RFC PATCH 4/7 scripts/recipetool: Add a recipe auto-creation script
In reply to Paul EggletonAdd a more maintainable and flexible script for creating at least the skeleton of a recipe based on an examination of the source tree. Commands can be added and the creation process can be extended through plugins.
[YOCTO #6406]
--- scripts/lib/recipetool/__init__.py | 0 scripts/lib/recipetool/create.py | 375 ++++++++++++++++++++++++++++++ scripts/lib/recipetool/create_buildsys.py | 234 +++++++++++++++++++ scripts/recipetool | 124 ++++++++++ 4 files changed, 733 insertions(+) create mode 100644 scripts/lib/recipetool/__init__.py create mode 100644 scripts/lib/recipetool/create.py create mode 100644 scripts/lib/recipetool/create_buildsys.py create mode 100755 scripts/recipetool
scripts/lib/recipetool/__init__.py new file +0 -0
scripts/lib/recipetool/create.py new file +375 -0
375 lines not on this page · raw message
scripts/lib/recipetool/create_buildsys.py new file +234 -0
234 lines not on this page · raw message
scripts/recipetool new file +124 -0
124 lines not on this page · raw message
Signature · 1 line
1.9.3
RFC PATCH 5/7 lib/oe: add recipeutils module
In reply to Paul EggletonAdd a module to help provide utility functions for dealing with recipes. This would typically be used by external tools.
Substantial portions of this module were borrowed from the OE Layer index code; other functions originally contributed by Markus Lehtonen <[email protected]>.
--- meta/lib/oe/recipeutils.py | 264 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 264 insertions(+) create mode 100644 meta/lib/oe/recipeutils.py
meta/lib/oe/recipeutils.py new file +264 -0
@@ -0,0 +1,264 @@ +# Utility functions for reading and modifying recipes +# +# Some code borrowed from the OE layer index +# +# Copyright (C) 2013-2014 Intel Corporation +# + +import sys +import os +import os.path +import tempfile +import textwrap +import difflib +import utils +import shutil +import re +from collections import OrderedDict, defaultdict + + +# Help us to find places to insert values +recipe_progression = ['SUMMARY', 'DESCRIPTION', 'HOMEPAGE', 'BUGTRACKER', 'SECTION', 'LICENSE', 'LIC_FILES_CHKSUM', 'PROVIDES', 'DEPENDS', 'PR', 'PV', 'SRC_URI', 'do_fetch', 'do_unpack', 'do_patch', 'EXTRA_OECONF', 'do_configure', 'EXTRA_OEMAKE', 'do_compile', 'do_install', 'do_populate_sysroot', 'INITSCRIPT', 'USERADD', 'GROUPADD', 'PACKAGES', 'FILES', 'RDEPENDS', 'RRECOMMENDS', 'RSUGGESTS', 'RPROVIDES', 'RREPLACES', 'RCONFLICTS', 'ALLOW_EMPTY', 'do_package', 'do_deploy'] +# Variables that sometimes are a bit long but shouldn't be wrapped +nowrap_vars = ['SUMMARY', 'HOMEPAGE', 'BUGTRACKER', 'LIC_FILES_CHKSUM'] +meta_vars = ['SUMMARY', 'DESCRIPTION', 'HOMEPAGE', 'BUGTRACKER', 'SECTION'] + + +def pn_to_recipe(cooker, pn): + """Convert a recipe name (PN) to the path to the recipe file""" + import bb.providers + + if pn in cooker.recipecache.pkg_pn: + filenames = cooker.recipecache.pkg_pn[pn] + best = bb.providers.findBestProvider(pn, cooker.data, cooker.recipecache, cooker.recipecache.pkg_pn) + return best[3] + else: + return None + + +def parse_recipe(fn, d): + """Parse an individual recipe""" + import bb.cache + envdata = bb.cache.Cache.loadDataFull(fn, [], d) + return envdata + + +def get_var_files(fn, varlist, d): + """Find the file in which each of a list of variables is set. + Note: requires variable history to be enabled when parsing. + """ + envdata = parse_recipe(fn, d) + varfiles = {} + for v in varlist: + history = envdata.varhistory.variable(v) + files = [] + for event in history: + if 'file' in event and not 'flag' in event: + files.append(event['file']) + if files: + actualfile = files[-1] + else: + actualfile = None + varfiles[v] = actualfile + + return varfiles + + +def patch_recipe_file(fn, values, patch=False, relpath=''): + """Update or insert variable values into a recipe file (assuming you + have already identified the exact file you want to update.) + Note that some manual inspection/intervention may be required + since this cannot handle all situations. + """ + remainingnames = {} + for k in values.keys(): + remainingnames[k] = recipe_progression.index(k) if k in recipe_progression else -1 + remainingnames = OrderedDict(sorted(remainingnames.iteritems(), key=lambda x: x[1])) + + with tempfile.NamedTemporaryFile('w', delete=False) as tf: + def outputvalue(name): + rawtext = '%s = "%s"\n' % (name, values[name]) + if name in nowrap_vars: + tf.write(rawtext) + else: + wrapped = textwrap.wrap(rawtext) + for wrapline in wrapped[:-1]: + tf.write('%s \\\n' % wrapline) + tf.write('%s\n' % wrapped[-1]) + + tfn = tf.name + with open(fn, 'r') as f: + # First runthrough - find existing names (so we know not to insert based on recipe_progression) + # Second runthrough - make the changes + existingnames = [] + for runthrough in [1, 2]: + currname = None + for line in f: + if not currname: + insert = False + for k in remainingnames.keys(): + for p in recipe_progression: + if re.match('^%s[ ?:=]' % p, line): + if remainingnames[k] > -1 and recipe_progression.index(p) > remainingnames[k] and runthrough > 1 and not k in existingnames: + outputvalue(k) + del remainingnames[k] + break + for k in remainingnames.keys(): + if re.match('^%s[ ?:=]' % k, line): + currname = k + if runthrough == 1: + existingnames.append(k) + else: + del remainingnames[k] + break + if currname and runthrough > 1: + outputvalue(currname) + + if currname: + sline = line.rstrip() + if not sline.endswith('\\'): + currname = None + continue + if runthrough > 1: + tf.write(line) + f.seek(0) + if remainingnames: + tf.write('\n') + for k in remainingnames.keys(): + outputvalue(k) + + with open(tfn, 'U') as f: + tolines = f.readlines() + if patch: + with open(fn, 'U') as f: + fromlines = f.readlines() + relfn = os.path.relpath(fn, relpath) + diff = difflib.unified_diff(fromlines, tolines, 'a/%s' % relfn, 'b/%s' % relfn) + os.remove(tfn) + return diff + else: + with open(fn, 'w') as f: + f.writelines(tolines) + os.remove(tfn) + return None + +def localise_file_vars(fn, varfiles, varlist): + """Given a list of variables and variable history (fetched with get_var_files()) + find where each variable should be set/changed. This handles for example where a + recipe includes an inc file where variables might be changed - in most cases + we want to update the inc file when changing the variable value rather than adding + it to the recipe itself. + """ + fndir = os.path.dirname(fn) + os.sep + + first_meta_file = None + for v in meta_vars: + f = varfiles.get(v, None) + if f: + actualdir = os.path.dirname(f) + os.sep + if actualdir.startswith(fndir): + first_meta_file = f + break + + filevars = defaultdict(list) + for v in varlist: + f = varfiles[v] + # Only return files that are in the same directory as the recipe or in some directory below there + # (this excludes bbclass files and common inc files that wouldn't be appropriate to set the variable + # in if we were going to set a value specific to this recipe) + if f: + actualfile = f + else: + # Variable isn't in a file, if it's one of the "meta" vars, use the first file with a meta var in it + if first_meta_file: + actualfile = first_meta_file + else: + actualfile = fn + + actualdir = os.path.dirname(actualfile) + os.sep + if not actualdir.startswith(fndir): + actualfile = fn + filevars[actualfile].append(v) + + return filevars + +def patch_recipe(d, fn, varvalues, patch=False, relpath=''): + """Modify a list of variable values in the specified recipe. Handles inc files if + used by the recipe. + """ + varlist = varvalues.keys() + varfiles = get_var_files(fn, varlist, d) + locs = localise_file_vars(fn, varfiles, varlist) + patches = [] + for f,v in locs.iteritems(): + vals = {k: varvalues[k] for k in v} + patchdata = patch_recipe_file(f, vals, patch, relpath) + if patch: + patches.append(patchdata) + + if patch: + return patches + else: + return None + + + +def copy_recipe_files(d, tgt_dir, whole_dir=False, download=True): + """Copy (local) recipe files, including both files included via include/require, + and files referred to in the SRC_URI variable.""" + import bb.fetch2 + import oe.path + + # FIXME need a warning if the unexpanded SRC_URI value contains variable references + + uris = (d.getVar('SRC_URI', True) or "").split() + fetch = bb.fetch2.Fetch(uris, d) + if download: + fetch.download() + + # Copy local files to target directory and gather any remote files + bb_dir = os.path.dirname(d.getVar('FILE', True)) + os.sep + remotes = [] + includes = [path for path in d.getVar('BBINCLUDED', True).split() if + path.startswith(bb_dir) and os.path.exists(path)] + for path in fetch.localpaths() + includes: + # Only import files that are under the meta directory + if path.startswith(bb_dir): + if not whole_dir: + relpath = os.path.relpath(path, bb_dir) + subdir = os.path.join(tgt_dir, os.path.dirname(relpath)) + if not os.path.exists(subdir): + os.makedirs(subdir) + shutil.copy2(path, os.path.join(tgt_dir, relpath)) + else: + remotes.append(path) + # Simply copy whole meta dir, if requested + if whole_dir: + shutil.copytree(bb_dir, tgt_dir) + + return remotes + + +def get_recipe_patches(d): + """Get a list of the patches included in SRC_URI within a recipe.""" + patchfiles = [] + # Execute src_patches() defined in patch.bbclass - this works since that class + # is inherited globally + patches = bb.utils.exec_flat_python_func('src_patches', d) + for patch in patches: + _, _, local, _, _, parm = bb.fetch.decodeurl(patch) + patchfiles.append(local) + return patchfiles + + +def validate_pn(pn): + """Perform validation on a recipe name (PN) for a new recipe.""" + reserved_names = ['forcevariable', 'append', 'prepend', 'remove'] + if not re.match('[0-9a-z-]+', pn): + return 'Recipe name "%s" is invalid: only characters 0-9, a-z and - are allowed' % pn + elif pn in reserved_names: + return 'Recipe name "%s" is invalid: is a reserved keyword' % pn + elif pn.startswith('pn-'): + return 'Recipe name "%s" is invalid: names starting with "pn-" are reserved' % pn + return '' +
Signature · 1 line
1.9.3
RFC PATCH 6/7 scripts/devtool: add development helper tool
In reply to Paul EggletonProvides an easy means to work on developing applications and system components with the build system.
You can start by creating a workspace layer (only needs to be done once):
$ devtool create-workspace
Then you can use the other functions, for example to "modify" the source for an existing recipe:
$ devtool modify -x pango /home/projects/pango Parsing recipes..done. INFO: Fetching pango... INFO: Unpacking... INFO: Patching... INFO: Source tree extracted to /tmp/woot2/ INFO: Recipe pango now set up to build from /home/projects/pango
The pango source is now extracted to /home/projects/pango, managed in git, with each patch as a commit, and a bbappend is created in the workspace layer to use the source in /home/projects/pango when building.
Additionally, you can add a new piece of software:
$ devtool add pv /home/projects/pv INFO: Recipe /path/to/workspace/recipes/pv/pv.bb has been automatically created; further editing may be required to make it fully functional
The latter uses recipetool to create a skeleton recipe and again sets up a bbappend to use the source in /home/projects/pv when building.
[YOCTO #6561]
[YOCTO #6653]
[YOCTO #6656]
--- scripts/devtool | 223 +++++++++++++++++++++++ scripts/lib/devtool/__init__.py | 79 ++++++++ scripts/lib/devtool/standard.py | 390 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 692 insertions(+) create mode 100755 scripts/devtool create mode 100644 scripts/lib/devtool/__init__.py create mode 100644 scripts/lib/devtool/standard.py
scripts/devtool new file +223 -0
@@ -0,0 +1,223 @@ +#!/usr/bin/env python + +# OpenEmbedded Development tool +# +# Copyright (C) 2014 Intel Corporation +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +import sys +import os +import argparse +import glob +import ConfigParser +import subprocess +import logging + +basepath = '' +workspace = {} +config = None +context = None + +def logger_create(name): + logger = logging.getLogger(name) + loggerhandler = logging.StreamHandler() + loggerhandler.setFormatter(logging.Formatter("%(levelname)s: %(message)s")) + logger.addHandler(loggerhandler) + logger.setLevel(logging.INFO) + return logger +logger = logger_create('devtool') + +plugins = [] + +def load_plugins(pluginpath): + global plugins + + def load_plugin(name): + logger.debug('Loading plugin %s' % name) + fp, pathname, description = imp.find_module(name, [pluginpath]) + try: + return imp.load_module(name, fp, pathname, description) + finally: + if fp: + fp.close() + + logger.debug('Loading plugins from %s...' % pluginpath) + import imp + for fn in glob.glob(os.path.join(pluginpath, '*.py')): + name = os.path.splitext(os.path.basename(fn))[0] + if name != '__init__': + plugin = load_plugin(name) + if hasattr(plugin, 'plugin_init'): + plugin.plugin_init(plugins) + plugins.append(plugin) + + +class ConfigHandler(object): + config_file = '' + config_obj = None + init_path = '' + workspace_path = '' + + def __init__(self, filename): + self.config_file = filename + self.config_obj = ConfigParser.SafeConfigParser() + + def get(self, section, option, default=None): + try: + ret = self.config_obj.get(section, option) + except (ConfigParser.NoOptionError, ConfigParser.NoSectionError): + if default != None: + ret = default + else: + raise + return ret + + def read(self): + if os.path.exists(self.config_file): + self.config_obj.read(self.config_file) + + if self.config_obj.has_option('General', 'init_path'): + pth = self.get('General', 'init_path') + self.init_path = os.path.join(basepath, pth) + if not os.path.exists(self.init_path): + logger.error('init_path %s specified in config file cannot be found' % pth) + return False + + self.workspace_path = self.get('General', 'workspace_path', os.path.join(basepath, 'workspace')) + + return True + else: + self.config_obj.add_section('General') + return True + + def write(self): + self.config_obj.set('General', 'workspace_path', self.workspace_path) + print self.config_file + with open(self.config_file, 'w') as f: + self.config_obj.write(f) + +class Context: + def __init__(self, **kwargs): + self.__dict__.update(kwargs) + + +def read_workspace(): + global workspace + workspace = {} + if not os.path.exists(os.path.join(config.workspace_path, 'conf', 'layer.conf')): + if context.fixed_setup: + logger.error("workspace layer not set up") + else: + logger.error("workspace layer not set up - you can create one by running %s create-workspace" % os.path.basename(sys.argv[0])) + sys.exit(1) + + logger.debug('Reading workspace in %s' % config.workspace_path) + for fn in glob.glob(os.path.join(config.workspace_path, 'appends', '*.bbappend')): + pn = os.path.splitext(os.path.basename(fn))[0].split('_')[0] + with open(fn, 'r') as f: + for line in f: + if line.startswith('EXTERNALSRC ='): + splitval = line.split('=', 2) + workspace[pn] = splitval[1].strip('" \n\r\t') + break + +def main(): + global basepath + global config + global context + + context = Context(fixed_setup=False) + + # Default basepath + basepath = os.path.dirname(os.path.abspath(__file__)) + pth = basepath + while pth != '' and pth != os.sep: + if os.path.exists(os.path.join(pth, '.devtoolbase')): + context.fixed_setup = True + basepath = pth + break + pth = os.path.dirname(pth) + + scripts_path = os.path.dirname(os.path.realpath(__file__)) + lib_path = scripts_path + '/lib' + sys.path = sys.path + [lib_path] + + parser = argparse.ArgumentParser(description="OpenEmbedded development tool") + parser.add_argument('--basepath', help='Base directory of SDK / build directory') + parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true') + parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true') + + subparsers = parser.add_subparsers(dest="subparser_name") + + load_plugins(os.path.join(scripts_path, 'lib', 'devtool')) + for plugin in plugins: + if hasattr(plugin, 'register_commands'): + plugin.register_commands(subparsers, context) + + args = parser.parse_args() + + if args.debug: + logger.setLevel(logging.DEBUG) + elif args.quiet: + logger.setLevel(logging.ERROR) + + if args.basepath: + # Override + basepath = args.basepath + elif not context.fixed_setup: + basepath = os.environ.get('BUILDDIR') + if not basepath: + logger.error("This script can only be run after initialising the build environment (e.g. by using oe-init-build-env)") + sys.exit(1) + + logger.debug('Using basepath %s' % basepath) + + config = ConfigHandler(os.path.join(basepath, 'devtool.conf')) + if not config.read(): + return -1 + + bitbake_subdir = config.get('General', 'bitbake_subdir', '') + if bitbake_subdir: + # Normally set for use within the SDK + logger.debug('Using bitbake subdir %s' % bitbake_subdir) + sys.path.insert(0, os.path.join(basepath, bitbake_subdir, 'lib')) + core_meta_subdir = config.get('General', 'core_meta_subdir') + sys.path.insert(0, os.path.join(basepath, core_meta_subdir, 'lib')) + else: + # Standard location + import scriptpath + bitbakepath = scriptpath.add_bitbake_lib_path() + if not bitbakepath: + logger.error("Unable to find bitbake by searching parent directory of this script or PATH") + sys.exit(1) + logger.debug('Using standard bitbake path %s' % bitbakepath) + scriptpath.add_oe_lib_path() + + if args.subparser_name != 'create-workspace': + read_workspace() + + ret = args.func(args, config, basepath, workspace) + + return ret + + +if __name__ == "__main__": + try: + ret = main() + except Exception: + ret = 1 + import traceback + traceback.print_exc(5) + sys.exit(ret)
scripts/lib/devtool/__init__.py new file +79 -0
79 lines not on this page · raw message
scripts/lib/devtool/standard.py new file +390 -0
390 lines not on this page · raw message
Signature · 1 line
1.9.3
RFC PATCH 7/7 scripts/devtool: Support deploy/undeploy function
In reply to Paul EggletonFrom: Junchun Guan <[email protected]>
Deploy recipe output files to live target machine using scp Store the files list and target machine info in localhost if deployment is done
Undeploy recipe output files in target machine using the previous deployment info
[YOCTO #6654]
--- scripts/lib/devtool/deploy.py | 100 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 100 insertions(+) create mode 100644 scripts/lib/devtool/deploy.py
scripts/lib/devtool/deploy.py new file +100 -0
@@ -0,0 +1,100 @@ +# Development tool - deploy/undeploy command plugin +# +# Copyright (C) 2014 Intel Corporation +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +import os +import subprocess +import logging +from devtool import exec_build_env_command + +logger = logging.getLogger('devtool') + +def plugin_init(pluginlist): + pass + + +def deploy(args, config, basepath, workspace): + import re + from devtool import exec_build_env_command + + if not args.recipename in workspace: + logger.error("no recipe named %s in your workspace" % args.recipename) + return -1 + try: + host, destdir = args.target.split(':') + except ValueError: + destdir = '/' + else: + args.target = host + + deploy_dir = os.path.join(basepath, 'target_deploy', args.target) + deploy_file = os.path.join(deploy_dir, args.recipename + '.list') + + if os.path.exists(deploy_file): + undeploy(args) + + stdout, stderr = exec_build_env_command(config.init_path, basepath, 'bitbake -e %s' % args.recipename, shell=True) + recipe_outdir = re.search(r'^D="(.*)"', stdout, re.MULTILINE).group(1) + ret = subprocess.call('scp -qr %s/* %s:%s' % (recipe_outdir, args.target, destdir), shell=True) + if ret != 0: + return ret + + logger.info('Successfully deployed %s' % recipe_outdir) + + if not os.path.exists(deploy_dir): + os.makedirs(deploy_dir) + + files_list = [] + for root, _, files in os.walk(recipe_outdir): + for filename in files: + filename = os.path.relpath(os.path.join(root, filename), recipe_outdir) + files_list.append(os.path.join(destdir, filename)) + + with open(deploy_file, 'w') as fobj: + fobj.write('\n'.join(files_list)) + + return 0 + +def undeploy(args, config, basepath, workspace): + + deploy_file = os.path.join(basepath, 'target_deploy', args.target, args.recipename + '.list') + if not os.path.exists(deploy_file): + logger.error('%s has not been deployed' % args.recipename) + return -1 + + ret = subprocess.call("scp -q %s %s:/tmp" % (deploy_file, args.target), shell=True) + if ret != 0: + logger.error('Failed to copy %s to %s' % (deploy, args.target)) + return -1 + + ret = subprocess.call("ssh %s 'xargs -n1 rm -f </tmp/%s'" % (args.target, os.path.basename(deploy_file)), shell=True) + if ret == 0: + logger.info('Successfully undeployed %s' % args.recipename) + os.remove(deploy_file) + + return ret + + +def register_commands(subparsers, context): + parser_deploy = subparsers.add_parser('deploy', help='Deploy recipe output files to live target machine') + parser_deploy.add_argument('recipename', help='Recipe to deploy') + parser_deploy.add_argument('target', help='Live target machine running an ssh server: user@hostname[:destdir]') + parser_deploy.set_defaults(func=deploy) + + parser_undeploy = subparsers.add_parser('undeploy', help='Undeploy recipe output files in live target machine') + parser_undeploy.add_argument('recipename', help='Recipe to undeploy') + parser_undeploy.add_argument('target', help='Live target machine running an ssh server: user@hostname') + parser_undeploy.set_defaults(func=undeploy)
Signature · 1 line
1.9.3
Quoted 50 lines · Paul Eggleton
[Note: this is an RFC series for the moment, and shouldn't yet be merged.]
I've been looking at how to make it easier for application and system component developers to get their work done using the tools we provide, and I believe this patchset is a piece of the solution. There's still a number of other pieces to come, but this should be usable on its own.
The first three patches extend the PATCHTOOL = "git" code to ensure that we're able to apply all patches even if "git am" and "git apply" can't handle them, and that we do a commit to the repository per patch. This is needed for devtool later on.
I've then added a new recipe auto-creation script, recipetool, which can take a source tree or URL and create a skeleton recipe to build it. (In case anyone is wondering about the existing
scripts/create-recipe, frankly I consider it a dead end - it's written in Perl, which makes it a bit difficult to integrate with the rest of our code; it's also GPLv3-only which makes any such integration pretty much impossible from another angle.)Then we add devtool. This allows you to:
* Add a new piece of software (auto-create the skeleton of a recipe using the aforementioned recipetool and point the build to an external source tree)
* Modify the source for an existing recipe (point the build to an external source tree, possibly creating that tree in the same step and managing it with git)
* Deploy the installed files from a recipe from ${D} to a target using scp.There will obviously be extensions to these tools, but I hope they are already functional enough to be useful at the state they are in at the moment.
Known issues:
* "devtool modify" can't currently deal with gcc, linux-yocto or perf because they manage their workdir differently, this will need to be fixed
* devtool's workspace layer triggers a bitbake warning when empty. I suspect a mechanism to disable this warning internally will need to be added.
* "recipetool create" ideally needs to become smarter and fill in more details of the recipe. At some point we'll probably have to make the process interactive and possibly have it automate parts of the build process and examine the output; some things just can't be practically detected from the source tree without building.
Quoted 7 lines
(Note that some pieces of the code which are not essential rely on a couple of changes to bitbake, you can find these in the paule/devtool-bb branch in poky-contrib, or if you are happy to use poky, grab the paule/devtool branch in poky-contrib which has everything).
Comments/questions/suggestions welcome!
This sounds great and looks like it will make some regular tasks quite a bit easier!
One thing which may be useful within recipetool or devtool is a single command to create patch files (using git format-patch), put them into the relevant directory and add the file names to SRC_URI. Obviously this assumes the external source tree is a git repo and we can trace the local modifications back to either an external master branch or an initial state after extracting a source archive. I'm just thinking out loud here though, depends if other people think that work flow is common enough to be worth creating a shortcut for.
Thanks,
Signature · 4 lines
Paul Barker Email: [email protected] http://www.paulbarker.me.uk
Quoted 12 lines · On Tuesday 25 November 2014 17:51:05 Paul Barker wrote:
Quoted 6 lines · Paul Eggleton
I've been looking at how to make it easier for application and system component developers to get their work done using the tools we provide, and I believe this patchset is a piece of the solution. There's still a number of other pieces to come, but this should be usable on its own.
...
Comments/questions/suggestions welcome!This sounds great and looks like it will make some regular tasks quite a bit easier!
Quoted 8 lines
One thing which may be useful within recipetool or devtool is a single command to create patch files (using
git format-patch), put them into the relevant directory and add the file names toSRC_URI. Obviously this assumes the external source tree is a git repo and we can trace the local modifications back to either an external master branch or an initial state after extracting a source archive. I'm just thinking out loud here though, depends if other people think that work flow is common enough to be worth creating a shortcut for.
We've been thinking in this direction and it seems like a natural extension to me, and one that I do hope to add. The tricky part is as you say, reconciling the commits you have in the git tree with the patches you have in the recipe, particularly when the user might have modified one or more of those commits.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Wow, this is exciting!
Just thinking out loud here... is there a way to way to have the devshell come up without having applied any patches but with the patches already queued in, say, a quilt series?
It seems to happen often enough to me that: a recipe has 10 local patches which applied fine yesterday but now patch 7 is failing because the upstream developers changed something. I would be nice to be able to devshell to the point where patch 6 has been applied and patch 7 is waiting to go on. Then I could apply the patch myself to figure out why it's failing. Or just having all the patches ready to apply (in a quilt series, as mentioned above) would be great too.
Hi Trevor,
Quoted 11 lines · On Tuesday 25 November 2014 14:56:14 Trevor Woerner wrote:
Just thinking out loud here... is there a way to way to have the devshell come up without having applied any patches but with the patches already queued in, say, a quilt series?
It seems to happen often enough to me that: a recipe has 10 local patches which applied fine yesterday but now patch 7 is failing because the upstream developers changed something. I would be nice to be able to devshell to the point where patch 6 has been applied and patch 7 is waiting to go on. Then I could apply the patch myself to figure out why it's failing. Or just having all the patches ready to apply (in a quilt series, as mentioned above) would be great too.
We do already have PATCHRESOLVE = "user" that's supposed to handle exactly this, although to be honest I've never really felt comfortable using it (that might have something to do with my lack of familiarity with quilt, though). devtool is using the same code that we use for applying patches, which supports quilt, so in theory it should be pretty easy to provide an option to use quilt rather than git when doing "devtool modify" or "devtool extract" for those that prefer it.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Hi Paul,
These tools are really nice! Some thoughts/comments:
Maybe the "devtool.conf" that gets created could be placed in the "conf/" subdirectory (along with the other configuration files such as local.conf and bblayers.conf)?
Perhaps any recipe you're working on could be automatically included via an IMAGE_INSTALL_append in conf/local.conf (or maybe that's too intrusive?)?
Do you envision users creating multiple workspaces? I'm wondering why "devtool create-workspace" is required. Is there any advantage to requiring users to create the workspace explicitly instead of just having it be created implicitly?
Some of the commands to "devtool" include things like
- extract
- build
- deploy
- undeploy
but when a workspace is created, devtool (very intelligently) adds the workspace to the set of BBLAYERS. So one could then just use bitbake to build the recipe. Are there any advantages to using "devtool build <recipe>" instead of "bitbake <recipe>"?
Best regards,
Trevor
Quoted 5 lines · On Friday 28 November 2014 12:28:00 Trevor Woerner wrote:
These tools are really nice! Some thoughts/comments:
Maybe the "devtool.conf" that gets created could be placed in the "conf/" subdirectory (along with the other configuration files such as local.conf and bblayers.conf)?
Yes, that's a good idea, I'll change that.
Quoted 2 lines
Perhaps any recipe you're working on could be automatically included via an
IMAGE_INSTALL_appendinconf/local.conf(or maybe that's too intrusive?)?
This is something I'd wanted to do - it's certainly something that should be made easy, but I was concerned about causing a full reparse just because of adding that to local.conf. (There might be a workaround through some sort of packagegroup for containing the packages produced by the recipes in the workspace that is added once when we create the workspace - maybe that's the answer?)
Quoted 4 lines
Do you envision users creating multiple workspaces? I'm wondering why "devtool create-workspace" is required. Is there any advantage to requiring users to create the workspace explicitly instead of just having it be created implicitly?
I wouldn't expect users to want to create multiple workspaces, but I did want users to be able to (a) choose where their workspace would go and (b) know that it has been created, so that the workspace layer showing up in the configuration isn't a surprise.
Quoted 9 lines
Some of the commands to "devtool" include things like
- extract
- build
- deploy
- undeploy
but when a workspace is created, devtool (very intelligently) adds the workspace to the set of BBLAYERS. So one could then just use bitbake to build the recipe. Are there any advantages to using "devtool build <recipe>" instead of "bitbake <recipe>"?
Not at the moment, although it is a convenient shortcut for "bitbake -c install <recipename>" (which is all you need to do for "devtool deploy" - note that "deploy" is distinct from our do_deploy, it could perhaps benefit from a better name). The other reason it's there is more for use as part of the SDK where the intention is to do everything through the devtool command, although that is a usage model that isn't enabled yet.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 9 lines · Paul Eggleton
Quoted 2 lines · On Friday 28 November 2014 12:28:00 Trevor Woerner wrote:
Perhaps any recipe you're working on could be automatically included via an
IMAGE_INSTALL_appendinconf/local.conf(or maybe that's too intrusive?)?This is something I'd wanted to do - it's certainly something that should be made easy, but I was concerned about causing a full reparse just because of adding that to local.conf. (There might be a workaround through some sort of packagegroup for containing the packages produced by the recipes in the workspace that is added once when we create the workspace - maybe that's the answer?)
Maybe even just printing a bit of text after a successful "add" to inform the user that the just-added project isn't part of any image and what they might want to do to include it?
The packagegroup sounds good too. But if the user doesn't want it included, they might be confused about how it magically appeared, and have some trouble finding how to remove it.
Quoted 8 lines
Quoted 4 lines
Do you envision users creating multiple workspaces? I'm wondering why "devtool create-workspace" is required. Is there any advantage to requiring users to create the workspace explicitly instead of just having it be created implicitly?
I wouldn't expect users to want to create multiple workspaces, but I did want users to be able to (a) choose where their workspace would go and (b) know that it has been created, so that the workspace layer showing up in the configuration isn't a surprise.
I can't help think that there's no harm in an unused workspace (is there?). Maybe the empty workspace from "create-workspace" should just be created by default by the "oe-init-build-env" tool (or whatever a given project is using).
If I were writing the documentation for this workflow, or giving a presentation on it... I'm just trying to figure out how to justify the extra, empty (but necessary) step of creating the environment before using it. Maybe if someone tries to add a project before creating the workspace, instead of erroring out, just run the create-workspace for them with the defaults. If they're a power user who wants to put the workspace somewhere specific then they're free to explicitly call create-workspace on their own, otherwise it'll get called with the defaults for them (or created automatically as part of the oe-init-build-env).
To borrow the analogy from git: create-workspace is part of the pluming; it's there if people what to call it explicitly, otherwise it gets called implicitly with defaults?
Quoted 12 lines · On Monday 01 December 2014 23:36:23 Trevor Woerner wrote:
Quoted 10 lines · Paul Eggleton
Quoted 3 lines · On Friday 28 November 2014 12:28:00 Trevor Woerner wrote:
Perhaps any recipe you're working on could be automatically included via an
IMAGE_INSTALL_appendinconf/local.conf(or maybe that's too intrusive?)?>This is something I'd wanted to do - it's certainly something that should be made easy, but I was concerned about causing a full reparse just because of adding that to local.conf. (There might be a workaround through some sort of packagegroup for containing the packages produced by the recipes in the workspace that is added once when we create the workspace - maybe that's the answer?)
Quoted 7 lines
Maybe even just printing a bit of text after a successful "add" to inform the user that the just-added project isn't part of any image and what they might want to do to include it?
The packagegroup sounds good too. But if the user doesn't want it included, they might be confused about how it magically appeared, and have some trouble finding how to remove it.
Right. I might defer this until we get to the SDK part where we'll need something to handle this (since the aim in that case is to avoid any editing of conf files or interacting with bitbake directly).
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines
Do you envision users creating multiple workspaces? I'm wondering why "devtool create-workspace" is required. Is there any advantage to requiring users to create the workspace explicitly instead of just having it be created implicitly?
I wouldn't expect users to want to create multiple workspaces, but I did want users to be able to (a) choose where their workspace would go and
(b) know that it has been created, so that the workspace layer showing up in the configuration isn't a surprise.
Quoted 4 lines
I can't help think that there's no harm in an unused workspace (is there?). Maybe the empty workspace from "create-workspace" should just be created by default by the "oe-init-build-env" tool (or whatever a given project is using).
I guess my thought was that some people are fussy about their setup, and this is a bit of a new thing.
Quoted 11 lines
If I were writing the documentation for this workflow, or giving a presentation on it... I'm just trying to figure out how to justify the extra, empty (but necessary) step of creating the environment before using it. Maybe if someone tries to add a project before creating the workspace, instead of erroring out, just run the create-workspace for them with the defaults. If they're a power user who wants to put the workspace somewhere specific then they're free to explicitly call create-workspace on their own, otherwise it'll get called with the defaults for them (or created automatically as part of the oe-init-build-env).
Quoted 3 lines
To borrow the analogy from git: create-workspace is part of the pluming; it's there if people what to call it explicitly, otherwise it gets called implicitly with defaults?
Actually you make a very good case; auto-creating it (and stating so) might be better than failing. I'll make the change.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Hi Paul,
Are you making these changes on the existing contrib/paule/devtool or somewhere else?
A small nit I stumbled on recently... many layers contain a "recipes-devtools" directory, and I'm thinking that using the same word for this feature might get confusing (?). Maybe it would be best to replace "devtool(s)" with "workflow(s)"?
Best regards,
Trevor
Hi Trevor,
Quoted 2 lines · On Thursday 04 December 2014 09:03:37 Trevor Woerner wrote:
Are you making these changes on the existing contrib/paule/devtool or somewhere else?
Actually I hadn't pushed them anywhere; I have just done that on the paule/devtool branch in both contrib repos. I'm collecting commits on top of the original ones, these will be squashed in when I send the first non-RFC series.
Quoted 4 lines
A small nit I stumbled on recently... many layers contain a "recipes-devtools" directory, and I'm thinking that using the same word for this feature might get confusing (?). Maybe it would be best to replace "devtool(s)" with "workflow(s)"?
Well, naming is always tricky. I don't think "workflow" really describes what the tool is supposed to do however; the tool certainly doesn't force a particular workflow, just provides tools to enable various different developer workflows. I'm happy to take other suggestions on naming though.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 2 lines · Paul Eggleton
I've then added a new recipe auto-creation script, recipetool, which can take a source tree or URL and create a skeleton recipe to build it.
...
Quoted 5 lines
* "recipetool create" ideally needs to become smarter and fill in more details of the recipe. At some point we'll probably have to make the process interactive and possibly have it automate parts of the build process and examine the output; some things just can't be practically detected from the source tree without building.
There is a tool "autoscan" which might be of some use here. If nothing else, the resulting configure.scan file could be parsed for DEPENDS information and perhaps PACKAGECONFIG items too.
Quoted 8 lines · On Monday 01 December 2014 23:54:03 Trevor Woerner wrote:
Quoted 2 lines · Paul Eggleton
I've then added a new recipe auto-creation script, recipetool, which can take a source tree or URL and create a skeleton recipe to build it.
...
Quoted 2 lines
* "recipetool create" ideally needs to become smarter and fill in
Quoted 8 lines
Quoted 4 lines
more details of the recipe. At some point we'll probably have to make the process interactive and possibly have it automate parts of the build process and examine the output; some things just can't be practically detected from the source tree without building.
There is a tool "autoscan" which might be of some use here. If nothing else, the resulting configure.scan file could be parsed for DEPENDS information and perhaps PACKAGECONFIG items too.
Interesting - I was not aware of this tool. I've added support for running it as well as picking converting AC_CHECK_LIB lines over to DEPENDS (with a note about some of them possibly being optional). Thanks!
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 12 lines · Paul Eggleton
Quoted 11 lines · On Monday 01 December 2014 23:54:03 Trevor Woerner wrote:
Quoted 2 lines · Paul Eggleton
I've then added a new recipe auto-creation script, recipetool, which can take a source tree or URL and create a skeleton recipe to build it.
...
Quoted 6 lines
* "recipetool create" ideally needs to become smarter and fill in
more details of the recipe. At some point we'll probably have to make the process interactive and possibly have it automate parts of the build process and examine the output; some things just can't be practically detected from the source tree without building.
Quoted 7 lines
Quoted 3 lines
There is a tool "autoscan" which might be of some use here. If nothing else, the resulting configure.scan file could be parsed for DEPENDS information and perhaps PACKAGECONFIG items too.
Interesting - I was not aware of this tool. I've added support for running it as well as picking converting
AC_CHECK_LIBlines over to DEPENDS (with a note about some of them possibly being optional). Thanks!
Finding "AC_PATH_X" in "configure.scan" is probably a good indication there's an RDEPENDS on "libx11". Maybe "recipetool" could check for that?
I think *this* thread (and question) got lost in the shuffle...
Quoted 16 lines · Trevor Woerner
Quoted 15 lines · Paul Eggleton
Quoted 14 lines · On Monday 01 December 2014 23:54:03 Trevor Woerner wrote:
Quoted 2 lines · Paul Eggleton
I've then added a new recipe auto-creation script, recipetool, which can take a source tree or URL and create a skeleton recipe to build it.
...
Quoted 6 lines
* "recipetool create" ideally needs to become smarter and fill in
more details of the recipe. At some point we'll probably have to make the process interactive and possibly have it automate parts of the build process and examine the output; some things just can't be practically detected from the source tree without building.
There is a tool "autoscan" which might be of some use here. If nothing else, the resulting configure.scan file could be parsed for DEPENDS information and perhaps PACKAGECONFIG items too.
Quoted 6 lines
Quoted 4 lines
Interesting - I was not aware of this tool. I've added support for running it as well as picking converting
AC_CHECK_LIBlines over to DEPENDS (with a note about some of them possibly being optional). Thanks!Finding "
AC_PATH_X" in "configure.scan" is probably a good indication there's an RDEPENDS on "libx11". Maybe "recipetool" could check for that?
*ping*!!
Oddly enough, the default recipe even says:
# NOTE: the following library dependencies are unknown, ignoring: X11
Why is it ignoring x11? Coincidentally the code I'm testing this with has a dependency on X11 and when I try building, bitbake throws out an RDEPENDS warning.
Quoted 28 lines · On Thursday 11 December 2014 18:10:57 Trevor Woerner wrote:
I think *this* thread (and question) got lost in the shuffle...
Quoted 25 lines · Trevor Woerner
Quoted 23 lines · Paul Eggleton
Quoted 9 lines · On Monday 01 December 2014 23:54:03 Trevor Woerner wrote:
Quoted 3 lines · Paul Eggleton
I've then added a new recipe auto-creation script, recipetool, which can
take a source tree or URL and create a skeleton recipe to build it....
Quoted 2 lines
* "recipetool create" ideally needs to become smarter and fill in
Quoted 8 lines
Quoted 4 lines
more details of the recipe. At some point we'll probably have to make the process interactive and possibly have it automate parts of the build process and examine the output; some things just can't be practically detected from the source tree without building.
There is a tool "autoscan" which might be of some use here. If nothing else, the resulting configure.scan file could be parsed for DEPENDS information and perhaps PACKAGECONFIG items too.
Interesting - I was not aware of this tool. I've added support for running it as well as picking converting
AC_CHECK_LIBlines over to DEPENDS (with a note about some of them possibly being optional). Thanks!
Quoted 4 lines
Quoted 2 lines
Finding "
AC_PATH_X" in "configure.scan" is probably a good indication there's an RDEPENDS on "libx11". Maybe "recipetool" could check for that?*ping*!!
I hadn't missed this, I just hadn't got around to looking at it ;)
Quoted 8 lines
Oddly enough, the default recipe even says:
# NOTE: the following library dependencies are unknown, ignoring: X11
Why is it ignoring x11? Coincidentally the code I'm testing this with has a dependency on X11 and when I try building, bitbake throws out an RDEPENDS warning.
It's ignoring anything that the extremely naive and incomplete mapping that I initially added doesn't currently handle. It'll need to be extended, and X11 would obviously be something we should add straight away. I've added a general task to expand the mapping to the todo list. FYI we are tracking more fine- grained tasks than are covered in bugzilla in the following wiki pages (especially since this is currently being worked on in a separate branch):
https://wiki.yoctoproject.org/wiki/Developer_Workflow_Improvements
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
(oops! sorry for going off-list)
Quoted 21 lines · Paul Eggleton
Quoted 20 lines · On Monday 08 December 2014 22:30:31 Trevor Woerner wrote:
This is really awesome stuff. I'm still playing around with it.
My current HEAD is:
commit
6ea5f5368a88317ace198f44a47cf043bc9fb6e6
Author: Paul Eggleton <[email protected]>
Date: Fri Dec 5 11:30:25 2014 +0000devtool: use event filtering to disable events if available
on your contrib/paule/devtool branch.
If I do a "devtool add" without a "devtool create-workspace" it succeeds but I get:
$ devtool add xannounce ~/devel/code/xannounce INFO: Creating workspace layer in INFO: Recipe recipes/xannounce/xannounce.bb has been automatically created; further editing may be required to make it fully functional
Quoted 8 lines
Quoted 7 lines
and find the changes have been made in the current directory. If I do the "devtool create-workspace" first, I get a "workspace" directory and all the stuff put in there, which is nicer.
If it doesn't fail due to a user running "devtool add" before "devtool create-workspace" then I think it would be nice if the two results were the same.
Ah, oops - I've found the cause and fixed it on the branch. Thanks!
Awesome... *very* nice. It works great now.
Quoted 9 lines
Quoted 3 lines
Would it be possible to make the output of devtool be nice and colourful like the output of bitbake?
Good idea, this is now done on the branch as well. "devtool build" doesn't support colour in the bitbake output, but that's because colour support in bitbake is switched by whether the output is going to a TTY and stdout is being redirected in this case. (I guess we could add a command line option for that as I have with devtool/recipetool.)
If it's not too much trouble I would like to ask that colour/curses be possible/enabled by default. Trying to find the error in all that output is a "needle in a haystack"-like operation :-)
I wonder if all the
NOTE: Running setscene task 31 of 33
(/home/trevor/devel/yocto/build/poky/meta-poky/meta/recipes-devtools/gcc/gcc-runtime_4.9.bb,
do_populate_sysroot_setscene)
NOTE: recipe gcc-runtime-4.9.1-r0: task
do_populate_sysroot_setscene: Started
NOTE: recipe gcc-runtime-4.9.1-r0: task
do_populate_sysroot_setscene: Succeeded
NOTE: Running setscene task 33 of 33
(/home/trevor/devel/yocto/build/poky/meta-poky/meta/recipes-devtools/gcc/gcc-cross_4.9.bb,
do_populate_sysroot_setscene)
NOTE: recipe gcc-cross-i586-4.9.1-r0: task
do_populate_sysroot_setscene: Started
NOTE: recipe gcc-cross-i586-4.9.1-r0: task
do_populate_sysroot_setscene: SucceededCould be (should be) done as part of the "devtool
create-workspace"/"devtool add" operation?
Quoted 44 lines · On Tuesday 09 December 2014 10:00:51 Trevor Woerner wrote:
(oops! sorry for going off-list)
Quoted 24 lines · Paul Eggleton
Quoted 23 lines · On Monday 08 December 2014 22:30:31 Trevor Woerner wrote:
This is really awesome stuff. I'm still playing around with it.
My current HEAD is:
commit6ea5f5368a88317ace198f44a47cf043bc9fb6e6
Author: Paul Eggleton <[email protected]>
Date: Fri Dec 5 11:30:25 2014 +0000devtool: use event filtering to disable events if available
on your contrib/paule/devtool branch.
If I do a "devtool add" without a "devtool create-workspace" it succeeds
but I get:
$ devtool add xannounce ~/devel/code/xannounce INFO: Creating workspace layer in INFO: Recipe recipes/xannounce/xannounce.bb has been automatically created; further editing may be required to make it fully functionaland find the changes have been made in the current directory. If I do the "devtool create-workspace" first, I get a "workspace" directory and all the stuff put in there, which is nicer.
Quoted 5 lines
Quoted 3 lines
If it doesn't fail due to a user running "devtool add" before "devtool create-workspace" then I think it would be nice if the two results were the same.
Ah, oops - I've found the cause and fixed it on the branch. Thanks!
Awesome... *very* nice. It works great now.
Quoted 8 lines
Quoted 2 lines
Would it be possible to make the output of devtool be nice and colourful like the output of bitbake?
Good idea, this is now done on the branch as well. "devtool build" doesn't support colour in the bitbake output, but that's because colour support in bitbake is switched by whether the output is going to a TTY and stdout is being redirected in this case. (I guess we could add a command line option for that as I have with devtool/recipetool.)
Quoted 3 lines
If it's not too much trouble I would like to ask that colour/curses be possible/enabled by default. Trying to find the error in all that output is a "needle in a haystack"-like operation :-)
I don't think this is straightforward unfortunately. I'll have to look into it.
Quoted 14 lines
I wonder if all the
NOTE: Running setscene task 31 of 33
(
/home/trevor/devel/yocto/build/poky/meta-poky/meta/recipes-devtools/gcc/gcc-runtime_4.9.bb,do_populate_sysroot_setscene)NOTE: recipe gcc-runtime-4.9.1-r0: task do_populate_sysroot_setscene: Started NOTE: recipe gcc-runtime-4.9.1-r0: task do_populate_sysroot_setscene: Succeeded NOTE: Running setscene task 33 of 33(
/home/trevor/devel/yocto/build/poky/meta-poky/meta/recipes-devtools/gcc/gcc-cross_4.9.bb,do_populate_sysroot_setscene)
Quoted 8 lines
NOTE: recipe gcc-cross-i586-4.9.1-r0: task do_populate_sysroot_setscene: Started NOTE: recipe gcc-cross-i586-4.9.1-r0: task do_populate_sysroot_setscene: SucceededCould be (should be) done as part of the "devtool
create-workspace"/"devtool add" operation?
If running devtool along side someone's existing build environment, wouldn't that be expected to be set up already? (All it does effectively is "bitbake recipename")
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 8 lines · Paul Eggleton
Quoted 7 lines · On Tuesday 09 December 2014 10:00:51 Trevor Woerner wrote:
Would it be possible to make the output of devtool be nice and colourful like the output of bitbake?
Quoted 5 lines
Good idea, this is now done on the branch as well. "devtool build" doesn't support colour in the bitbake output, but that's because colour support in bitbake is switched by whether the output is going to a TTY and stdout is being redirected in this case. (I guess we could add a command line option for that as I have with devtool/recipetool.)
Quoted 5 lines
Quoted 3 lines
If it's not too much trouble I would like to ask that colour/curses be possible/enabled by default. Trying to find the error in all that output is a "needle in a haystack"-like operation :-)
I don't think this is straightforward unfortunately. I'll have to look into it.
Okay. Maybe it'd be best to just revert to bitbake? IOW, does the devtool tool need its own "build,deploy,..."?
Quoted 21 lines
Quoted 21 lines
I wonder if all the
NOTE: Running setscene task 31 of 33
(
/home/trevor/devel/yocto/build/poky/meta-poky/meta/recipes-devtools/gcc/gcc-runtime_4.9.bb,do_populate_sysroot_setscene)NOTE: recipe gcc-runtime-4.9.1-r0: task do_populate_sysroot_setscene: Started NOTE: recipe gcc-runtime-4.9.1-r0: task do_populate_sysroot_setscene: Succeeded NOTE: Running setscene task 33 of 33(
/home/trevor/devel/yocto/build/poky/meta-poky/meta/recipes-devtools/gcc/gcc-cross_4.9.bb,do_populate_sysroot_setscene)NOTE: recipe gcc-cross-i586-4.9.1-r0: task do_populate_sysroot_setscene: Started NOTE: recipe gcc-cross-i586-4.9.1-r0: task do_populate_sysroot_setscene: Succeeded
Quoted 6 lines
Quoted 2 lines
Could be (should be) done as part of the "devtool
create-workspace"/"devtool add" operation?If running devtool along side someone's existing build environment, wouldn't that be expected to be set up already? (All it does effectively is "bitbake recipename")
Here are my steps:
$ source meta-poky/oe-init-build-env build $ bitbake core-image-x11 $ devtool add <myx11app> $ devtool build <myx11app>
at this point it does a bunch of these gcc-runtime tasks. So my first thought is: "why is my build going off into left field and doing all these other things when all I asked it to do is to build <myx11app>?". I'm guessing these gcc-runtime tasks are required for something related to devtool, and they only need to be run once. So maybe it would be best if they were run as part of creating the workspace (either explicitly with "devtool create-workspace" or implicitly with "devtool add")?
Quoted 15 lines · On Tuesday 09 December 2014 10:39:05 Trevor Woerner wrote:
Quoted 14 lines · Paul Eggleton
Quoted 12 lines · On Tuesday 09 December 2014 10:00:51 Trevor Woerner wrote:
Would it be possible to make the output of devtool be nice and colourful like the output of bitbake?
Quoted 5 lines
Good idea, this is now done on the branch as well. "devtool build" doesn't support colour in the bitbake output, but that's because colour support in bitbake is switched by whether the output is going to a TTY and stdout is being redirected in this case. (I guess we could add a command line option for that as I have with devtool/recipetool.)
If it's not too much trouble I would like to ask that colour/curses be possible/enabled by default. Trying to find the error in all that output is a "needle in a haystack"-like operation :-)
Quoted 5 lines
Quoted 2 lines
I don't think this is straightforward unfortunately. I'll have to look into it.
Okay. Maybe it'd be best to just revert to bitbake? IOW, does the devtool tool need its own "build,deploy,..."?
We did talk about this in a previous email, but just to clarify:
"devtool build" isn't strictly required in this phase, I added it primarily to support future usage in the SDK where the intention is to do everything through the devtool command, although that is a usage model that isn't enabled yet. However, it is a convenient shortcut for "bitbake -c install <recipename>" (which is all you need to do for "devtool deploy" to be able to work.)
"devtool deploy" is not related to do_deploy - it places files for the recipe on a nominated target machine without having to worry about packaging; we have no other equivalent for this at the moment. Perhaps it could have a different name in case the naming overlap is confusing.
Quoted 36 lines
Quoted 23 lines
Quoted 23 lines
I wonder if all the
NOTE: Running setscene task 31 of 33
(
/home/trevor/devel/yocto/build/poky/meta-poky/meta/recipes-devtools/gcc/gc
c-runtime_4.9.bb,do_populate_sysroot_setscene)NOTE: recipe gcc-runtime-4.9.1-r0: task do_populate_sysroot_setscene: Started NOTE: recipe gcc-runtime-4.9.1-r0: task do_populate_sysroot_setscene: Succeeded NOTE: Running setscene task 33 of 33(
/home/trevor/devel/yocto/build/poky/meta-poky/meta/recipes-devtools/gcc/gc
c-cross_4.9.bb,do_populate_sysroot_setscene)NOTE: recipe gcc-cross-i586-4.9.1-r0: task do_populate_sysroot_setscene: Started NOTE: recipe gcc-cross-i586-4.9.1-r0: task do_populate_sysroot_setscene: SucceededQuoted 6 lines
Quoted 2 lines
Could be (should be) done as part of the "devtool
create-workspace"/"devtool add" operation?If running devtool along side someone's existing build environment, wouldn't that be expected to be set up already? (All it does effectively is "bitbake recipename")
Here are my steps:
$ source meta-poky/oe-init-build-env build $ bitbake core-image-x11 $ devtool add <myx11app> $ devtool build <myx11app>
Quoted 7 lines
at this point it does a bunch of these gcc-runtime tasks. So my first thought is: "why is my build going off into left field and doing all these other things when all I asked it to do is to build <myx11app>?". I'm guessing these gcc-runtime tasks are required for something related to devtool, and they only need to be run once. So maybe it would be best if they were run as part of creating the workspace (either explicitly with "devtool create-workspace" or implicitly with "devtool add")?
Trying to make "devtool add" or "devtool modify" do this just doesn't feel right to me. These operations are supposed to be quick, and if they end up running actual build tasks, they won't be.
My guess (with no other context) is that the reason you are seeing this is that you're building from sstate, and when it builds core-image-x11 it doesn't need some of the tasks that it then does need when it goes to build the individual recipe. I'm not sure, but I would expect that most people would be doing this from an existing build directory rather than starting with a new one. Is there a reason you're creating a new build directory as part of the steps above?
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 4 lines · Paul Eggleton
Quoted 3 lines · On Tuesday 09 December 2014 10:39:05 Trevor Woerner wrote:
Okay. Maybe it'd be best to just revert to bitbake? IOW, does the devtool tool need its own "build,deploy,..."?
Quoted 8 lines
We did talk about this in a previous email, but just to clarify:
"devtool build" isn't strictly required in this phase, I added it primarily to support future usage in the SDK where the intention is to do everything through the devtool command, although that is a usage model that isn't enabled yet. However, it is a convenient shortcut for "bitbake -c install <recipename>" (which is all you need to do for "devtool deploy" to be able to work.)
True, we did talk about this before. All I'm saying is that the colour feature of bitbake is really nice, useful, and helpful (to visually parse output) and if we've lost colour because we've wrapped bitbake with "devtool build" then we're taking a step backwards.
But I'll concede the point if the future usage in the SDK turns out to be really helpful. All I'm saying at this point is I have no idea what this future usage might be or what your ideas are for this step, and for the time-being (i.e. without knowing or working with this SDK usage) "devtool build" is a step backwards from "bitbake".
Quoted 4 lines
"devtool deploy" is not related to
do_deploy- it places files for the recipe on a nominated target machine without having to worry about packaging; we have no other equivalent for this at the moment. Perhaps it could have a different name in case the naming overlap is confusing.
Ah yes, sorry I forgot... maybe something more explicit like "deploy_to_target"? ;-)push_to_target?upload_to_target?
Quoted 46 lines
Quoted 35 lines
Quoted 24 lines
Quoted 24 lines
I wonder if all the
NOTE: Running setscene task 31 of 33
(
/home/trevor/devel/yocto/build/poky/meta-poky/meta/recipes-devtools/gcc/gc
c-runtime_4.9.bb,do_populate_sysroot_setscene)NOTE: recipe gcc-runtime-4.9.1-r0: task do_populate_sysroot_setscene: Started NOTE: recipe gcc-runtime-4.9.1-r0: task do_populate_sysroot_setscene: Succeeded NOTE: Running setscene task 33 of 33(
/home/trevor/devel/yocto/build/poky/meta-poky/meta/recipes-devtools/gcc/gc
c-cross_4.9.bb,do_populate_sysroot_setscene)NOTE: recipe gcc-cross-i586-4.9.1-r0: task do_populate_sysroot_setscene: Started NOTE: recipe gcc-cross-i586-4.9.1-r0: task do_populate_sysroot_setscene: SucceededQuoted 5 lines
Quoted 2 lines
Could be (should be) done as part of the "devtool
create-workspace"/"devtool add" operation?If running devtool along side someone's existing build environment, wouldn't that be expected to be set up already? (All it does effectively is "bitbake recipename")
Here are my steps:
$ source meta-poky/oe-init-build-env build $ bitbake core-image-x11 $ devtool add <myx11app> $ devtool build <myx11app>Quoted 7 lines
at this point it does a bunch of these gcc-runtime tasks. So my first thought is: "why is my build going off into left field and doing all these other things when all I asked it to do is to build <myx11app>?". I'm guessing these gcc-runtime tasks are required for something related to devtool, and they only need to be run once. So maybe it would be best if they were run as part of creating the workspace (either explicitly with "devtool create-workspace" or implicitly with "devtool add")?
Trying to make "devtool add" or "devtool modify" do this just doesn't feel right to me. These operations are supposed to be quick, and if they end up running actual build tasks, they won't be.
Quoted 7 lines
My guess (with no other context) is that the reason you are seeing this is that you're building from sstate, and when it builds core-image-x11 it doesn't need some of the tasks that it then does need when it goes to build the individual recipe. I'm not sure, but I would expect that most people would be doing this from an existing build directory rather than starting with a new one. Is there a reason you're creating a new build directory as part of the steps above?
The reason I included the "oe-init-build-env build" step in my instructions above is to emphasize the fact I'm starting from a completely fresh build environment :-)
LICENSE
If the tool that creates the recipe can't find any explicit licensing information, it leaves LICENCE blank. Would it be better, if there's no explicit license, to set it to "CLOSED"? Otherwise the first build will fail and the user will have to figure out what to do in order to get the recipe to build.
code updates
Maybe my inexperience with externalsrc is showing through here (I've never used it before), but as I play with this new feature, I'm testing it by trying to include a very old project I wrote years ago. As luck would have it, that project needs updates (it doesn't quite work as-is in qemux86/core-image-x11). So here I am going through the cycle of working on its source, and building new images. What I've noticed is when I create the image I keep getting the same old sources used over and over from when I first added this project "devtool add" (and it doesn't matter whether I use bitbake or devtool to build it, it even keeps using the old sources even when I perform commits in the sources and even when I perform a "bitbake -c cleansstate <recipe>"!). The only way I can move forward is to "devtool reset" and then "devtool add".
In my scenario I have a base image I have already built (core-image-x11). Now I want to add one package to that image from sources found on my local machine. I also want to go through the cycle of editing those sources, building a new image, and runqemu'ing the new image to test my changes. I can't seem to do that without knowing devtool and/or bitbake magic. I either have to "devtool reset" or I have to "bitbake -c clean core-image-x11" because just modifying the sources doesn't cause the image to be regenerated.
I think at this point I should create a detailed doc explaining my workflow (or maybe a screencast?) so you can see exactly what I'm doing if the above doesn't make sense.
Quoted 20 lines · On Tuesday 09 December 2014 13:58:02 Trevor Woerner wrote:
Quoted 6 lines · Paul Eggleton
Quoted 2 lines · On Tuesday 09 December 2014 10:39:05 Trevor Woerner wrote:
Okay. Maybe it'd be best to just revert to bitbake? IOW, does the devtool tool need its own "build,deploy,..."?
We did talk about this in a previous email, but just to clarify:
Quoted 7 lines
"devtool build" isn't strictly required in this phase, I added it primarily to support future usage in the SDK where the intention is to do everything through the devtool command, although that is a usage model that isn't enabled yet. However, it is a convenient shortcut for "bitbake -c install
<recipename>" (which is all you need to do for "devtool deploy" to be able to work.)True, we did talk about this before. All I'm saying is that the colour feature of bitbake is really nice, useful, and helpful (to visually parse output) and if we've lost colour because we've wrapped bitbake with "devtool build" then we're taking a step backwards.
Quoted 5 lines
But I'll concede the point if the future usage in the SDK turns out to be really helpful. All I'm saying at this point is I have no idea what this future usage might be or what your ideas are for this step, and for the time-being (i.e. without knowing or working with this SDK usage) "devtool build" is a step backwards from "bitbake".
OK, understood. I'll have a think about it.
Quoted 9 lines
Quoted 4 lines
"devtool deploy" is not related to
do_deploy- it places files for the recipe on a nominated target machine without having to worry about packaging; we have no other equivalent for this at the moment. Perhaps it could have a different name in case the naming overlap is confusing.Ah yes, sorry I forgot... maybe something more explicit like "
deploy_to_target"? ;-)push_to_target?upload_to_target?
Hmm, perhaps deploy-target would be more in line with the git-style command naming I've been using.
Quoted 26 lines
Quoted 18 lines
Quoted 6 lines
Here are my steps:
$ source meta-poky/oe-init-build-env build $ bitbake core-image-x11 $ devtool add <myx11app> $ devtool build <myx11app>Quoted 7 lines
at this point it does a bunch of these gcc-runtime tasks. So my first thought is: "why is my build going off into left field and doing all these other things when all I asked it to do is to build <myx11app>?". I'm guessing these gcc-runtime tasks are required for something related to devtool, and they only need to be run once. So maybe it would be best if they were run as part of creating the workspace (either explicitly with "devtool create-workspace" or implicitly with "devtool add")?
Trying to make "devtool add" or "devtool modify" do this just doesn't feel right to me. These operations are supposed to be quick, and if they end up running actual build tasks, they won't be.
Quoted 7 lines
My guess (with no other context) is that the reason you are seeing this is that you're building from sstate, and when it builds core-image-x11 it doesn't need some of the tasks that it then does need when it goes to build the individual recipe. I'm not sure, but I would expect that most people would be doing this from an existing build directory rather than starting with a new one. Is there a reason you're creating a new build directory as part of the steps above?
Quoted 3 lines
The reason I included the "oe-init-build-env build" step in my instructions above is to emphasize the fact I'm starting from a completely fresh build environment :-)
OK, but would you expect to be doing that on a regular basis or is it just for the purposes of exercise?
Quoted 6 lines
LICENSE
If the tool that creates the recipe can't find any explicit licensing information, it leaves LICENCE blank. Would it be better, if there's no explicit license, to set it to "CLOSED"? Otherwise the first build will fail and the user will have to figure out what to do in order to get the recipe to build.
Possibly yes. Ideally people would take an interest in setting this properly, but I do recognise that it's not a high priority when you are just getting started. What might be missing is some kind of modal operation where during initial development you aren't bothered by all of these recipe quality checks, but then when it comes to production the checks are re-enabled.
Quoted 14 lines
code updates
Maybe my inexperience with externalsrc is showing through here (I've never used it before), but as I play with this new feature, I'm testing it by trying to include a very old project I wrote years ago. As luck would have it, that project needs updates (it doesn't quite work as-is in qemux86/core-image-x11). So here I am going through the cycle of working on its source, and building new images. What I've noticed is when I create the image I keep getting the same old sources used over and over from when I first added this project "devtool add" (and it doesn't matter whether I use bitbake or devtool to build it, it even keeps using the old sources even when I perform commits in the sources and even when I perform a "bitbake -c cleansstate <recipe>"!). The only way I can move forward is to "devtool reset" and then "devtool add".
Quoted 8 lines
In my scenario I have a base image I have already built (core-image-x11). Now I want to add one package to that image from sources found on my local machine. I also want to go through the cycle of editing those sources, building a new image, and runqemu'ing the new image to test my changes. I can't seem to do that without knowing devtool and/or bitbake magic. I either have to "devtool reset" or I have to "bitbake -c clean core-image-x11" because just modifying the sources doesn't cause the image to be regenerated.
It's not supposed to behave this way, although I have reproduced it here - I think I hadn't actually tested rebuilding an image with a recipe in the workspace added to the image. The problem is that the do_compile task is being marked as nostamp, but the way bitbake behaves now, that doesn't cause any dependent task signatures to be changed and thus no dependent tasks get run afterwards. Either I'll have to change that or work out a different way of ensuring do_compile and dependent tasks get run - either way, you should only have to run a build that calls for the recipe (be it the recipe itself or something that depends upon it), no cleaning/resetting should be needed. I'll try to fix this tomorrow.
Quoted 3 lines
I think at this point I should create a detailed doc explaining my workflow (or maybe a screencast?) so you can see exactly what I'm doing if the above doesn't make sense.
If you think it might help, but I think I understand most of the above. Let me fix the task signature issue and then hopefully things should work a bit more sensibly :)
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 14 lines · On Wednesday 10 December 2014 17:51:16 Paul Eggleton wrote:
Quoted 13 lines · On Tuesday 09 December 2014 13:58:02 Trevor Woerner wrote:
Quoted 6 lines · Paul Eggleton
"devtool build" isn't strictly required in this phase, I added it primarily to support future usage in the SDK where the intention is to do everything through the devtool command, although that is a usage model that isn't enabled yet. However, it is a convenient shortcut for "bitbake -c install <recipename>" (which is all you need to do for "devtool deploy" to be able to work.)
True, we did talk about this before. All I'm saying is that the colour feature of bitbake is really nice, useful, and helpful (to visually parse output) and if we've lost colour because we've wrapped bitbake with "devtool build" then we're taking a step backwards.
Quoted 7 lines
Quoted 5 lines
But I'll concede the point if the future usage in the SDK turns out to be really helpful. All I'm saying at this point is I have no idea what this future usage might be or what your ideas are for this step, and for the time-being (i.e. without knowing or working with this SDK usage) "devtool build" is a step backwards from "bitbake".
OK, understood. I'll have a think about it.
I've fixed this on the branch; turns out there is an easy way to fool processes into thinking they are connected to a real terminal and thus have bitbake use the normal colour and terminal handling.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
"devtool deploy" is not related to
do_deploy- it places files for the recipe on a nominated target machine without having to worry about packaging; we have no other equivalent for this at the moment. Perhaps it
could have a different name in case the naming overlap is confusing.
Quoted 7 lines
Quoted 4 lines
Ah yes, sorry I forgot... maybe something more explicit like "
deploy_to_target"? ;-)push_to_target?upload_to_target?Hmm, perhaps deploy-target would be more in line with the git-style command naming I've been using.
Fixed on the branch.
Quoted 7 lines
Quoted 6 lines
LICENSE
If the tool that creates the recipe can't find any explicit licensing information, it leaves LICENCE blank. Would it be better, if there's no explicit license, to set it to "CLOSED"? Otherwise the first build will fail and the user will have to figure out what to do in order to get the recipe to build.
Quoted 6 lines
Possibly yes. Ideally people would take an interest in setting this properly, but I do recognise that it's not a high priority when you are just getting started. What might be missing is some kind of modal operation where during initial development you aren't bothered by all of these recipe quality checks, but then when it comes to production the checks are re-enabled.
I've made this change on the branch (with an appropriate warning comment preceding it).
Quoted 23 lines
Quoted 14 lines
code updates
Maybe my inexperience with externalsrc is showing through here (I've never used it before), but as I play with this new feature, I'm testing it by trying to include a very old project I wrote years ago. As luck would have it, that project needs updates (it doesn't quite work as-is in qemux86/core-image-x11). So here I am going through the cycle of working on its source, and building new images. What I've noticed is when I create the image I keep getting the same old sources used over and over from when I first added this project "devtool add" (and it doesn't matter whether I use bitbake or devtool to build it, it even keeps using the old sources even when I perform commits in the sources and even when I perform a "bitbake -c cleansstate <recipe>"!). The only way I can move forward is to "devtool reset" and then "devtool add".Quoted 8 lines
In my scenario I have a base image I have already built (core-image-x11). Now I want to add one package to that image from sources found on my local machine. I also want to go through the cycle of editing those sources, building a new image, and runqemu'ing the new image to test my changes. I can't seem to do that without knowing devtool and/or bitbake magic. I either have to "devtool reset" or I have to "bitbake -c clean core-image-x11" because just modifying the sources doesn't cause the image to be regenerated.
Quoted 10 lines
It's not supposed to behave this way, although I have reproduced it here - I think I hadn't actually tested rebuilding an image with a recipe in the workspace added to the image. The problem is that the
do_compiletask is being marked as nostamp, but the way bitbake behaves now, that doesn't cause any dependent task signatures to be changed and thus no dependent tasks get run afterwards. Either I'll have to change that or work out a different way of ensuringdo_compileand dependent tasks get run - either way, you should only have to run a build that calls for the recipe (be it the recipe itself or something that depends upon it), no cleaning/resetting should be needed. I'll try to fix this tomorrow.
OK, I've fixed this on the branch as well. You should now find that rebuilding the image always triggers a compile -> install -> package etc. of the recipe(s) in the workspace first. (It'll end up triggering other recipes to rebuild as well if they are in the chain; we may have to address that differently later for cases where that's annoying).
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Awesome!! I'm giving this update a whirl right now...
...just one small nit... in commit 7d2c6430131ccf94b8bdc185b8cd849fe479d552 the warning message says:
+ lines_before.append('# NOTE: LICENSE is being set to "CLOSED"
to allow you to at least start building - if')
+ lines_before.append('# this is not accurate with respect to the
licensing of the software being built (it')
+ lines_before.append('# will not be for in most cases) you must
specify the correct value before using this')
+ lines_before.append('# recipe for anything other than initial
testing/development!')
Is the "for" in "it will not be for in most cases" out of place? Perhaps replace with "so", "true" or "accurate"?
Quoted 14 lines · On Thursday 11 December 2014 13:30:01 Trevor Woerner wrote:
Awesome!! I'm giving this update a whirl right now...
...just one small nit... in commit
7d2c6430131ccf94b8bdc185b8cd849fe479d552the warning message says:+ lines_before.append('# NOTE: LICENSE is being set to "CLOSED"
to allow you to at least start building - if')
+ lines_before.append('# this is not accurate with respect to the
licensing of the software being built (it')
+lines_before.append('# will not be for in most cases)you must
specify the correct value before using this')
+ lines_before.append('# recipe for anything other than initial
testing/development!')
Quoted 2 lines
Is the "for" in "it will not be for in most cases" out of place? Perhaps replace with "so", "true" or "accurate"?
Ah yes - fixed, thanks :)
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Oops! (this worked before the "git pull")
$ devtool add xannounce ~/devel/yocto/build/workflow/xannounce/
NOTE: Creating workspace layer in
/home/trevor/devel/yocto/build/workflow/build/workspace
NOTE: Recipe
/home/trevor/devel/yocto/build/workflow/build/workspace/recipes/xannounce/xannounce.bb
has been automatically created; further editing may be required to
make it fully functional
Traceback (most recent call last):
File
"/home/trevor/devel/yocto/build/workflow/meta-poky/scripts/devtool",
line 274, in <module>
ret = main()
File
"/home/trevor/devel/yocto/build/workflow/meta-poky/scripts/devtool",
line 267, in main
ret = args.func(args, config, basepath, workspace)
File
"/home/trevor/devel/yocto/build/workflow/meta-poky/scripts/lib/devtool/standard.py",
line 68, in add
_add_md5(config, args.recipename, recipefile)
File
"/home/trevor/devel/yocto/build/workflow/meta-poky/scripts/lib/devtool/standard.py",
line 218, in _add_md5
md5 = bb.utils.md5_file(filename)
File
"/home/trevor/devel/yocto/build/workflow/meta-poky/bitbake/lib/bb/utils.py",
line 463, in md5_file
with open(filename, "rb") as f:
IOError: [Errno 2] No such file or directory:
'/home/trevor/devel/yocto/build/workflow/build/workspace/recipes/xannounce/xannounce.bb'Sure enough...
$ devtool-workflow> tree workspace/
workspace/
|-- appends
|-- conf
| `-- layer.conf
`-- recipes
`-- xannounce4 directories, 1 file
whereas before it was:
$ tree workspace/
workspace/
|-- appends
| `-- xannounce.bbappend
|-- conf
| `-- layer.conf
`-- recipes
`-- xannounce
`-- xannounce.bb4 directories, 3 files
For some reason my recipe and bbappends are not getting created :-(
bisecting (manually) leads to this commit being safe:
ea50ec527483eb3a094588858baebd5fb23f71c6 devtool: rename deploy/undeploy to deploy-target/undeploy-target
and this one introducing the problem:
770f203f73f358ce3c6c5ac12fe2775b812bdc6b devtool: fool bitbake into thinking it's outputting to a terminal
Quoted 61 lines · Trevor Woerner
Oops! (this worked before the "git pull")
$ devtool add xannounce ~/devel/yocto/build/workflow/xannounce/ NOTE: Creating workspace layer in /home/trevor/devel/yocto/build/workflow/build/workspace NOTE: Recipe /home/trevor/devel/yocto/build/workflow/build/workspace/recipes/xannounce/xannounce.bb has been automatically created; further editing may be required to make it fully functional Traceback (most recent call last): File "/home/trevor/devel/yocto/build/workflow/meta-poky/scripts/devtool", line 274, in <module> ret = main() File "/home/trevor/devel/yocto/build/workflow/meta-poky/scripts/devtool", line 267, in main ret = args.func(args, config, basepath, workspace) File "/home/trevor/devel/yocto/build/workflow/meta-poky/scripts/lib/devtool/standard.py", line 68, in add _add_md5(config, args.recipename, recipefile) File "/home/trevor/devel/yocto/build/workflow/meta-poky/scripts/lib/devtool/standard.py", line 218, in _add_md5 md5 = bb.utils.md5_file(filename) File "/home/trevor/devel/yocto/build/workflow/meta-poky/bitbake/lib/bb/utils.py", line 463, in md5_file with open(filename, "rb") as f: IOError: [Errno 2] No such file or directory: '/home/trevor/devel/yocto/build/workflow/build/workspace/recipes/xannounce/xannounce.bb'Sure enough...
$ devtool-workflow> tree workspace/ workspace/ |-- appends |-- conf | `-- layer.conf `-- recipes `-- xannounce4 directories, 1 file
whereas before it was:
$ tree workspace/ workspace/ |-- appends | `-- xannounce.bbappend |-- conf | `-- layer.conf `-- recipes `-- xannounce `-- xannounce.bb4 directories, 3 files
For some reason my recipe and bbappends are not getting created :-(
Quoted 89 lines · On Thursday 11 December 2014 14:43:40 Trevor Woerner wrote:
Quoted 78 lines · Trevor Woerner
Oops! (this worked before the "git pull")
$ devtool add xannounce ~/devel/yocto/build/workflow/xannounce/ NOTE: Creating workspace layer in /home/trevor/devel/yocto/build/workflow/build/workspace NOTE: Recipe /home/trevor/devel/yocto/build/workflow/build/workspace/recipes/xannou nce/xannounce.bb has been automatically created; further editing may be required to make it fully functionalTraceback (most recent call last):
File"
/home/trevor/devel/yocto/build/workflow/meta-poky/scripts/devtool", line 274, in <module>ret =
main()File
"
/home/trevor/devel/yocto/build/workflow/meta-poky/scripts/devtool", line 267, in mainret =
args.func(args, config, basepath, workspace)File
"
/home/trevor/devel/yocto/build/workflow/meta-poky/scripts/lib/devtool/standard.py", line 68, in add
_add_md5(config, args.recipename, recipefile)File
"
/home/trevor/devel/yocto/build/workflow/meta-poky/scripts/lib/devtool/standard.py", line 218, in_add_md5md5 =
bb.utils.md5_file(filename)File
"
/home/trevor/devel/yocto/build/workflow/meta-poky/bitbake/lib/bb/utils.py", line 463, inmd5_filewith
open(filename, "rb")as f:
IOError: [Errno 2] No such file or directory:
'/home/trevor/devel/yocto/build/workflow/build/workspace/recipes/xannounce/xannounce.bb'>
Sure enough...$ devtool-workflow> tree workspace/ workspace/|-- appends |-- conf | | `-- layer.conf`-- recipes
`-- xannounce
4 directories, 1 file
whereas before it was:
$ tree workspace/ workspace/|-- appends | | `-- xannounce.bbappend | |-- conf | | `-- layer.conf`-- recipes
Quoted 7 lines
`-- xannounce
`-- xannounce.bb
4 directories, 3 files
For some reason my recipe and bbappends are not getting created :-(
bisecting (manually) leads to this commit being safe:
Quoted 7 lines
ea50ec527483eb3a094588858baebd5fb23f71c6devtool: rename deploy/undeploy to deploy-target/undeploy-targetand this one introducing the problem:
770f203f73f358ce3c6c5ac12fe2775b812bdc6bdevtool: fool bitbake into thinking it's outputting to a terminal
Thanks for the quick bisect - I've fixed it so that wrapper is only used when running bitbake through "devtool build", and now "devtool add" works properly again. This is why I should probably write some proper tests before trying to do anything further...
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 4 lines · Paul Eggleton
Thanks for the quick bisect - I've fixed it so that wrapper is only used when running bitbake through "devtool build", and now "devtool add" works properly again. This is why I should probably write some proper tests before trying to do anything further...
It works great now, thanks!
Quoted 95 lines · Paul Eggleton
Quoted 94 lines · On Thursday 11 December 2014 14:43:40 Trevor Woerner wrote:
Quoted 78 lines · Trevor Woerner
Oops! (this worked before the "git pull")
$ devtool add xannounce ~/devel/yocto/build/workflow/xannounce/ NOTE: Creating workspace layer in /home/trevor/devel/yocto/build/workflow/build/workspace NOTE: Recipe /home/trevor/devel/yocto/build/workflow/build/workspace/recipes/xannou nce/xannounce.bb has been automatically created; further editing may be required to make it fully functionalTraceback (most recent call last):
File"
/home/trevor/devel/yocto/build/workflow/meta-poky/scripts/devtool", line 274, in <module>ret =
main()File
"
/home/trevor/devel/yocto/build/workflow/meta-poky/scripts/devtool", line 267, in mainret =
args.func(args, config, basepath, workspace)File
"
/home/trevor/devel/yocto/build/workflow/meta-poky/scripts/lib/devtool/standard.py", line 68, in add
_add_md5(config, args.recipename, recipefile)File
"
/home/trevor/devel/yocto/build/workflow/meta-poky/scripts/lib/devtool/standard.py", line 218, in_add_md5md5 =
bb.utils.md5_file(filename)File
"
/home/trevor/devel/yocto/build/workflow/meta-poky/bitbake/lib/bb/utils.py", line 463, inmd5_filewith
open(filename, "rb")as f:
IOError: [Errno 2] No such file or directory:
'/home/trevor/devel/yocto/build/workflow/build/workspace/recipes/xannounce/xannounce.bb'>
Sure enough...$ devtool-workflow> tree workspace/ workspace/|-- appends |-- conf | | `-- layer.conf`-- recipes
`-- xannounce
4 directories, 1 file
whereas before it was:
$ tree workspace/ workspace/|-- appends | | `-- xannounce.bbappend | |-- conf | | `-- layer.conf`-- recipes
Quoted 7 lines
`-- xannounce
`-- xannounce.bb
4 directories, 3 files
For some reason my recipe and bbappends are not getting created :-(
bisecting (manually) leads to this commit being safe:
ea50ec527483eb3a094588858baebd5fb23f71c6devtool: rename deploy/undeploy to deploy-target/undeploy-targetand this one introducing the problem:
Quoted 7 lines
Quoted 2 lines
770f203f73f358ce3c6c5ac12fe2775b812bdc6bdevtool: fool bitbake into thinking it's outputting to a terminalThanks for the quick bisect - I've fixed it so that wrapper is only used when running bitbake through "devtool build", and now "devtool add" works properly again. This is why I should probably write some proper tests before trying to do anything further...
Agreed. I think a good unittest needs to be include sooner than later as the more it grows, the harder will be to cover all usecases with tests.
Signature · 3 lines
Otavio Salvador O.S. Systems http://www.ossystems.com.br http://code.ossystems.com.br Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750
Quoted 10 lines · Paul Eggleton
Quoted 10 lines
It's not supposed to behave this way, although I have reproduced it here - I think I hadn't actually tested rebuilding an image with a recipe in the workspace added to the image. The problem is that the
do_compiletask is being marked as nostamp, but the way bitbake behaves now, that doesn't cause any dependent task signatures to be changed and thus no dependent tasks get run afterwards. Either I'll have to change that or work out a different way of ensuringdo_compileand dependent tasks get run - either way, you should only have to run a build that calls for the recipe (be it the recipe itself or something that depends upon it), no cleaning/resetting should be needed. I'll try to fix this tomorrow.
Quoted 5 lines
OK, I've fixed this on the branch as well. You should now find that rebuilding the image always triggers a compile -> install -> package etc. of the recipe(s) in the workspace first. (It'll end up triggering other recipes to rebuild as well if they are in the chain; we may have to address that differently later for cases where that's annoying).
Ok great. I've re-tested and it all seems good.