Developer workflow tools

I've been looking at how to make it easier for application and system component developers to get their work done using the tools we provide, and I believe this patchset is a piece of the solution. There's still a number of other pieces to come, but this should be usable on its own.

The first part of this patchset makes some changes to OE to accommodate the tools. One of these is to extend the PATCHTOOL = "git" code to ensure that we're able to apply all patches even if "git am" and "git apply" can't handle them, that we do a commit to the repository per patch, and that there is a way to track these commits back to the patch from which they were added. There are also some additions to shared library code that may well be useful for other tools.

I've then added a new recipe auto-creation script, recipetool, which can take a source tree or URL and create a skeleton recipe to build it. (In case anyone is wondering about the existing scripts/create-recipe, frankly I consider it a dead end - it's written in Perl, which makes it a bit difficult to integrate with the rest of our code; it's also GPLv3-only which makes any such integration pretty much impossible from another angle.)

Then we add devtool. This allows you to:

* Add a new piece of software (auto-create the skeleton of a recipe using the aforementioned recipetool and point the build to an external source tree)
* Modify the source for an existing recipe (point the build to an external source tree, possibly creating that tree in the same step and managing it with git)
* Deploy the installed files from a recipe from ${D} to a target using scp.
* Apply any modifications to the commits in the external source tree back to the recipe (by updating/adding/removing patches and updating SRC_URI within the recipe).

There will obviously be extensions to these tools, but I hope they are already functional enough to be useful at the state they are in at the moment.

(Note that this series depends on another series of patches to BitBake sent at the same time - these can be found on the paule/devtool-bb2 branch on poky-contrib.)

Changes since the earlier RFC:

For those that looked at the earlier RFC series, there have been quite a few improvements since that, almost too many to list - but other than what's visible in the actual commits, here goes:

* recipetool now uses autoscan to pick up dependencies from Makefile-only software
* recipetool is much smarter about library dependencies (making use of collected shared lib provider and pkgdata information rather than a limited hardcoded map) and detects more licenses.
* devtool now creates workspace layer as needed if it doesn't exist
* devtool can now write back commits to patches next to the original recipe
* Compilation of recipes in the workspace now happens every time rather than just once, and causes dependent tasks (e.g. images) to be rebuilt as well.
* recipetool: if we can't find any license files at all, set LICENSE to "CLOSED" (with a comment warning it needs to be set more appropriately) so that the user can at least start building the recipe.
* devtool can extract the source for linux-yocto but externalsrc usage may still be problematic - something that still needs to be addressed.
* Branch and tag the external source git repository
* Improved error reporting, help and general messages

(Thanks to Trevor Woerner and others for earlier testing & feedback.)

The following changes since commit 8d0e56a850579f9a6d501266deeef9b257ce4780:

serf: readded md5sum (2014-12-11 11:34:22 +0000)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib paule/devtool
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool

Junchun Guan (1):
scripts/devtool: Support deploy/undeploy function

Paul Eggleton (14):
meta-environment: don't mark tasks as nostamp classes/package: move read_shlib_providers() to a common unit lib/oe/patch: fall back to patch if git apply fails lib/oe/patch: auto-commit when falling back from git am lib/oe/patch: use --keep-cr with git am lib/oe/patch.py: abort "git am" if it fails lib/oe/patch: add support for extracting patches from git tree lib/oe: add recipeutils module
oeqa/utils: make get_bb_var() more reliable classes/externalsrc: set do_compile as nostamp scripts/recipetool: Add a recipe auto-creation script
scripts: add scriptutils module
scripts/devtool: add development helper tool
devtool: add QA tests

 meta/classes/externalsrc.bbclass           |   3 +
 meta/classes/package.bbclass               |  24 +-
 meta/lib/oe/package.py                     |  26 ++
 meta/lib/oe/patch.py                       | 158 ++++++++-
 meta/lib/oe/recipeutils.py                 | 279 +++++++++++++++
 meta/lib/oeqa/selftest/devtool.py          | 239 +++++++++++++
 meta/lib/oeqa/utils/commands.py            |   4 +-
 meta/recipes-core/meta/meta-environment.bb |   2 -
 scripts/devtool                            | 255 ++++++++++++++
 scripts/lib/devtool/__init__.py            |  78 +++++
 scripts/lib/devtool/deploy.py              | 100 ++++++
 scripts/lib/devtool/standard.py            | 545 +++++++++++++++++++++++++++++
 scripts/lib/recipetool/__init__.py         |   0
 scripts/lib/recipetool/create.py           | 413 ++++++++++++++++++++++
 scripts/lib/recipetool/create_buildsys.py  | 319 +++++++++++++++++
 scripts/lib/scriptutils.py                 |  60 ++++
 scripts/recipetool                         |  99 ++++++
 17 files changed, 2572 insertions(+), 32 deletions(-)
 create mode 100644 meta/lib/oe/recipeutils.py
 create mode 100644 meta/lib/oeqa/selftest/devtool.py
 create mode 100755 scripts/devtool
 create mode 100644 scripts/lib/devtool/__init__.py
 create mode 100644 scripts/lib/devtool/deploy.py
 create mode 100644 scripts/lib/devtool/standard.py
 create mode 100644 scripts/lib/recipetool/__init__.py
 create mode 100644 scripts/lib/recipetool/create.py
 create mode 100644 scripts/lib/recipetool/create_buildsys.py
 create mode 100644 scripts/lib/scriptutils.py
 create mode 100755 scripts/recipetool

1.9.3