Skip to contentSkip to conversation

devtool and externalsrc improvements

PATCH In review Paul Eggleton 3 participants 21 messages Feb 19, 2015

lore
Share & tools
lore
#0 lore raw

cover devtool and externalsrc improvements

I've been attempting to enable usage of devtool with the kernel (which effectively means using the externalsrc class with the kernel). This doesn't really work yet due to lack of handling the kernel configuration, so you still can't use externalsrc properly with the kernel, but this at least gets us further - you can now extract the source successfully and run a build. I also discovered and fixed a number of related issues along the way, and made a few behavioural improvements in both devtool and externalsrc.

The following changes since commit ad016fa7492ba23c751cf33dd1befde7782facfa:

lzo: Upgrade to 2.09 and add ptest (2015-02-19 07:50:44 +0000)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib paule/devtool-fixes
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool-fixes

Paul Eggleton (15):
recipetool: fix regression caused by previous commit lib/oe/patch: fix PATCHTOOL = "git" with source in a subdirectory classes/externalsrc: fix source being wiped out on clean with kernel classes/externalsrc: add workaround for recipes that use SRCPV in PV classes/externalsrc: show a warning on compiling classes/kernel-yocto: extend SRCTREECOVEREDTASKS
devtool: add/modify: add option to build in same directory
devtool: reset: run bitbake -c clean on recipe
devtool: build: run do_populate_sysroot instead of do_install lib/oe/recipeutils: minor improvements to patch_recipe()
devtool: update-recipe: add handling for git recipes
devtool: prevent extract/modify use with image recipes
devtool: modify: handle recipes that use a shared workdir
devtool: support source extraction for recipes that use a shared
workdir
devtool: fix broken clones of git recipes

 meta/classes/externalsrc.bbclass  |  24 ++++
 meta/classes/kernel-yocto.bbclass |   2 +-
 meta/lib/oe/patch.py              |   7 +-
 meta/lib/oe/recipeutils.py        |   4 +-
 meta/lib/oeqa/selftest/devtool.py | 147 ++++++++++++++++++-
 scripts/lib/devtool/standard.py   | 292 +++++++++++++++++++++++++-------------
 scripts/lib/recipetool/create.py  |   8 +-
 7 files changed, 377 insertions(+), 107 deletions(-)
Signature · 1 line
1.9.3
#1 lore raw

PATCH 1/15 recipetool: fix regression caused by previous commit

Option was renamed in the setup code but not in the code that used it.

---
 scripts/lib/recipetool/create.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/lib/recipetool/create.py +4 -4
index 290fc13..ae599cb 100644
@@ -110,8 +110,8 @@ def create_recipe(args):
     if '://' in args.source:
         # Fetch a URL
         srcuri = args.source
-        if args.externalsrc:
-            srctree = args.externalsrc
+        if args.extract_to:
+            srctree = args.extract_to
         else:
             tempsrc = tempfile.mkdtemp(prefix='recipetool-')
             srctree = tempsrc
@@ -126,8 +126,8 @@ def create_recipe(args):
             srctree = os.path.join(srctree, srcsubdir)
     else:
         # Assume we're pointing to an existing source tree
-        if args.externalsrc:
-            logger.error('externalsrc cannot be specified if source is a directory')
+        if args.extract_to:
+            logger.error('--extract-to cannot be specified if source is a directory')
             sys.exit(1)
         if not os.path.isdir(args.source):
             logger.error('Invalid source directory %s' % args.source)
Signature · 1 line
1.9.3
#2 lore raw

PATCH 2/15 lib/oe/patch: fix PATCHTOOL = "git" with source in a subdirectory

In reply to Paul Eggleton

For recipes that have their actual source in a subdirectory of what is fetched (e.g. mkelfimage), we need to find the root of the repository within the GitApplyTree code that attempts to set up the required git hooks and use that, rather than expecting the root to be the same as ${S}.

---
 meta/lib/oe/patch.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
meta/lib/oe/patch.py +5 -2
index b838be8..f68d40f 100644
@@ -313,9 +313,12 @@ class GitApplyTree(PatchTree):
             return runcmd(["sh", "-c", " ".join(shellcmd)], self.dir)
 
         # Add hooks which add a pointer to the original patch file name in the commit message
-        commithook = os.path.join(self.dir, '.git', 'hooks', 'commit-msg')
+        reporoot = (runcmd("git rev-parse --show-toplevel".split(), self.dir) or '').strip()
+        if not reporoot:
+            raise Exception("Cannot get repository root for directory %s" % self.dir)
+        commithook = os.path.join(reporoot, '.git', 'hooks', 'commit-msg')
         commithook_backup = commithook + '.devtool-orig'
-        applyhook = os.path.join(self.dir, '.git', 'hooks', 'applypatch-msg')
+        applyhook = os.path.join(reporoot, '.git', 'hooks', 'applypatch-msg')
         applyhook_backup = applyhook + '.devtool-orig'
         if os.path.exists(commithook):
             shutil.move(commithook, commithook_backup)
Signature · 1 line
1.9.3
#3 lore raw

PATCH 3/15 classes/externalsrc: fix source being wiped out on clean with kernel

In reply to Paul Eggleton

kernel.bbclass adds ${S} do do_clean[cleandirs], but this means if you run bitbake -c clean <kernelrecipe> then your external source tree will be trashed, which could be a disaster. For safety, remove ${S} from cleandirs for every task. We also have to do the same for ${B} in the case where EXTERNALSRC_BUILD is set to the same value as EXTERNALSRC.

---
 meta/classes/externalsrc.bbclass | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
meta/classes/externalsrc.bbclass +14 -0
index 4e429d7..e372392 100644
@@ -47,6 +47,20 @@ python () {
                 # Since configure will likely touch ${S}, ensure only we lock so one task has access at a time
                 d.appendVarFlag(task, "lockfiles", "${S}/singletask.lock")
 
+            # We do not want our source to be wiped out, ever (kernel.bbclass does this for do_clean)
+            cleandirs = d.getVarFlag(task, 'cleandirs', False)
+            if cleandirs:
+                cleandirs = cleandirs.split()
+                setvalue = False
+                if '${S}' in cleandirs:
+                    cleandirs.remove('${S}')
+                    setvalue = True
+                if externalsrcbuild == externalsrc and '${B}' in cleandirs:
+                    cleandirs.remove('${B}')
+                    setvalue = True
+                if setvalue:
+                    d.setVarFlag(task, 'cleandirs', ' '.join(cleandirs))
+
         for task in d.getVar("SRCTREECOVEREDTASKS", True).split():
             bb.build.deltask(task, d)
 
Signature · 1 line
1.9.3
#4 lore raw

PATCH 4/15 classes/externalsrc: add workaround for recipes that use SRCPV in PV

In reply to Paul Eggleton

Here we set SRC_URI to blank, however doing so means that the function that is called when you expand the default value of SRCPV (i.e. bb.fetch2.get_srcrev()) will fail, so any recipe that references SRCPV in PV couldn't previously be used with externalsrc.

(At some point we may fix the function to work in the externalsrc case, but then we would also need to ensure that ${B} did not change as a result of PV changing any time the HEAD revision changes in the external source tree, or you'll lose any intermediate build artifacts.)

---
 meta/classes/externalsrc.bbclass | 4 ++++
 1 file changed, 4 insertions(+)
meta/classes/externalsrc.bbclass +4 -0
index e372392..234bd8d 100644
@@ -37,6 +37,10 @@ python () {
             d.setVar('B', '${WORKDIR}/${BPN}-${PV}/')
         d.setVar('SRC_URI', '')
 
+        if '{SRCPV}' in d.getVar('PV', False):
+            # Dummy value because the default function can't be called with blank SRC_URI
+            d.setVar('SRCPV', '999')
+
         tasks = filter(lambda k: d.getVarFlag(k, "task"), d.keys())
 
         for task in tasks:
Signature · 1 line
1.9.3
#5 lore raw

PATCH 5/15 classes/externalsrc: show a warning on compiling

In reply to Paul Eggleton

Make sure there's no chance of anyone forgetting they have a recipe set up for externalsrc; otherwise you could get confused about what is going on. (With our default logging setup we can't make it a note because the UI doesn't forward those; otherwise I would have used bb.note().)

---
 meta/classes/externalsrc.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)
meta/classes/externalsrc.bbclass +6 -0
index 234bd8d..25f0be9 100644
@@ -68,7 +68,13 @@ python () {
         for task in d.getVar("SRCTREECOVEREDTASKS", True).split():
             bb.build.deltask(task, d)
 
+        d.prependVarFlag('do_compile', 'prefuncs', "externalsrc_compile_prefunc ")
+
         # Ensure compilation happens every time
         d.setVarFlag('do_compile', 'nostamp', '1')
 }
 
+python externalsrc_compile_prefunc() {
+    # Make it obvious that this is happening, since forgetting about it could lead to much confusion
+    bb.warn('Compiling %s from external source %s' % (d.getVar('PN', True), d.getVar('EXTERNALSRC', True)))
+}
Signature · 1 line
1.9.3
#6 lore raw

PATCH 6/15 classes/kernel-yocto: extend SRCTREECOVEREDTASKS

In reply to Paul Eggleton

* Add do_shared_workdir which was added recently
* Add do_fetch and do_unpack to this list, because at the moment if you enable externalsrc through a bbappend the += in this class wipes out the original value from externalsrc (which is set with ?=)

---
 meta/classes/kernel-yocto.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/kernel-yocto.bbclass +1 -1
index 223825e..c3f1ac7 100644
@@ -1,5 +1,5 @@
 # remove tasks that modify the source tree in case externalsrc is inherited
-SRCTREECOVEREDTASKS += "do_kernel_link_vmlinux do_kernel_configme do_validate_branches do_kernel_configcheck do_kernel_checkout do_patch"
+SRCTREECOVEREDTASKS += "do_kernel_link_vmlinux do_kernel_configme do_validate_branches do_kernel_configcheck do_kernel_checkout do_shared_workdir do_fetch do_unpack do_patch"
 
 # returns local (absolute) path names for all valid patches in the
 # src_uri
Signature · 1 line
1.9.3
#7 lore raw

PATCH 7/15 devtool: add/modify: add option to build in same directory

In reply to Paul Eggleton

The default behaviour is to build in a separate directory to the source, however some projects can't be built this way, so add an option to do that (or override the automatic behaviour in the case of modify).

---
 scripts/lib/devtool/standard.py | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
scripts/lib/devtool/standard.py +9 -2
index ae64840..d503111 100644
@@ -77,6 +77,8 @@ def add(args, config, basepath, workspace):
     with open(appendfile, 'w') as f:
         f.write('inherit externalsrc\n')
         f.write('EXTERNALSRC = "%s"\n' % srctree)
+        if args.same_dir:
+            f.write('EXTERNALSRC_BUILD = "%s"\n' % srctree)
         if initial_rev:
             f.write('\n# initial_rev: %s\n' % initial_rev)
 
@@ -323,8 +325,11 @@ def modify(args, config, basepath, workspace):
         f.write('inherit externalsrc\n')
         f.write('# NOTE: We use pn- overrides here to avoid affecting multiple variants in the case where the recipe uses BBCLASSEXTEND\n')
         f.write('EXTERNALSRC_pn-%s = "%s"\n' % (args.recipename, srctree))
-        if bb.data.inherits_class('autotools-brokensep', rd):
-            logger.info('using source tree as build directory since original recipe inherits autotools-brokensep')
+        if args.same_dir or bb.data.inherits_class('autotools-brokensep', rd):
+            if args.same_dir:
+                logger.info('using source tree as build directory since --same-dir specified')
+            else:
+                logger.info('using source tree as build directory since original recipe inherits autotools-brokensep')
             f.write('EXTERNALSRC_BUILD_pn-%s = "%s"\n' % (args.recipename, srctree))
         if initial_rev:
             f.write('\n# initial_rev: %s\n' % initial_rev)
@@ -503,6 +508,7 @@ def register_commands(subparsers, context):
                                        formatter_class=argparse.ArgumentDefaultsHelpFormatter)
     parser_add.add_argument('recipename', help='Name for new recipe to add')
     parser_add.add_argument('srctree', help='Path to external source tree')
+    parser_add.add_argument('--same-dir', '-s', help='Build in same directory as source', action="store_true")
     parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)')
     parser_add.set_defaults(func=add)
 
@@ -513,6 +519,7 @@ def register_commands(subparsers, context):
     parser_add.add_argument('srctree', help='Path to external source tree')
     parser_add.add_argument('--wildcard', '-w', action="store_true", help='Use wildcard for unversioned bbappend')
     parser_add.add_argument('--extract', '-x', action="store_true", help='Extract source as well')
+    parser_add.add_argument('--same-dir', '-s', help='Build in same directory as source', action="store_true")
     parser_add.add_argument('--branch', '-b', default="devtool", help='Name for development branch to checkout (only when using -x)')
     parser_add.set_defaults(func=modify)
 
Signature · 1 line
1.9.3
#8 lore raw

PATCH 8/15 devtool: reset: run bitbake -c clean on recipe

In reply to Paul Eggleton

If you've added a new recipe, you want the output cleaned when you do devtool reset, otherwise cruft from building the recipe may remain which could interfere with future builds.

---
 meta/lib/oeqa/selftest/devtool.py | 4 ++++
 scripts/lib/devtool/standard.py   | 6 ++++++
 2 files changed, 10 insertions(+)
meta/lib/oeqa/selftest/devtool.py +4 -0
index 74fb325..33f2e91 100644
@@ -179,9 +179,13 @@ class DevtoolTests(oeSelfTest):
                 if line.startswith('.TH'):
                     self.assertEqual(line.rstrip(), '.TH MDADM 8 "" v9.999-custom', 'man file not modified')
         # Test devtool reset
+        stampprefix = get_bb_var('STAMP', 'mdadm')
         result = runCmd('devtool reset mdadm')
         result = runCmd('devtool status')
         self.assertNotIn('mdadm', result.output)
+        self.assertTrue(stampprefix, 'Unable to get STAMP value for recipe mdadm')
+        matches = glob.glob(stampprefix + '*')
+        self.assertFalse(matches, 'Stamp files exist for recipe mdadm that should have been cleaned')
 
     def test_devtool_update_recipe(self):
         # Check preconditions
scripts/lib/devtool/standard.py +6 -0
index d503111..763177d 100644
@@ -475,6 +475,11 @@ def reset(args, config, basepath, workspace):
     if not args.recipename in workspace:
         logger.error("no recipe named %s in your workspace" % args.recipename)
         return -1
+
+    if not args.no_clean:
+        logger.info('Cleaning sysroot for recipe %s...' % args.recipename)
+        exec_build_env_command(config.init_path, basepath, 'bitbake -c clean %s' % args.recipename)
+
     _check_preserve(config, args.recipename)
 
     preservepath = os.path.join(config.workspace_path, 'attic', args.recipename)
@@ -555,5 +560,6 @@ def register_commands(subparsers, context):
                                          description='Removes the specified recipe from your workspace (resetting its state)',
                                          formatter_class=argparse.ArgumentDefaultsHelpFormatter)
     parser_reset.add_argument('recipename', help='Recipe to reset')
+    parser_reset.add_argument('--no-clean', '-n', action="store_true", help='Don\'t clean the sysroot to remove recipe output')
     parser_reset.set_defaults(func=reset)
 
Signature · 1 line
1.9.3
#9 lore raw

PATCH 9/15 devtool: build: run do_populate_sysroot instead of do_install

In reply to Paul Eggleton

If you want to be able to make use of libraries in conjunction with devtool then we need to install them into the sysroot for other recipes to use. Make it a configuration option in case it needs to be changed at runtime.

---
 meta/lib/oeqa/selftest/devtool.py | 40 +++++++++++++++++++++++++++++++++++++++
 scripts/lib/devtool/standard.py   |  5 +++--
 2 files changed, 43 insertions(+), 2 deletions(-)
meta/lib/oeqa/selftest/devtool.py +40 -0
index 33f2e91..8caf07a 100644
@@ -136,6 +136,46 @@ class DevtoolTests(oeSelfTest):
             bindir = bindir[1:]
         self.assertTrue(os.path.isfile(os.path.join(installdir, bindir, 'pv')), 'pv binary not found in D')
 
+    def test_devtool_add_library(self):
+        # Check preconditions
+        workspacedir = os.path.join(self.builddir, 'workspace')
+        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        # We don't have the ability to pick up this dependency automatically yet...
+        bitbake('libusb1')
+        # Fetch source
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        url = 'http://www.intra2net.com/en/developer/libftdi/download/libftdi1-1.1.tar.bz2'
+        result = runCmd('wget %s' % url, cwd=tempdir)
+        result = runCmd('tar xfv libftdi1-1.1.tar.bz2', cwd=tempdir)
+        srcdir = os.path.join(tempdir, 'libftdi1-1.1')
+        self.assertTrue(os.path.isfile(os.path.join(srcdir, 'CMakeLists.txt')), 'Unable to find CMakeLists.txt in source directory')
+        # Test devtool add
+        self.track_for_cleanup(workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        result = runCmd('devtool add libftdi %s' % srcdir)
+        self.assertTrue(os.path.exists(os.path.join(workspacedir, 'conf', 'layer.conf')), 'Workspace directory not created')
+        # Test devtool status
+        result = runCmd('devtool status')
+        self.assertIn('libftdi', result.output)
+        self.assertIn(srcdir, result.output)
+        # Clean up anything in the workdir/sysroot/sstate cache (have to do this *after* devtool add since the recipe only exists then)
+        bitbake('libftdi -c cleansstate')
+        # Test devtool build
+        result = runCmd('devtool build libftdi')
+        staging_libdir = get_bb_var('STAGING_LIBDIR', 'libftdi')
+        self.assertTrue(staging_libdir, 'Could not query STAGING_LIBDIR variable')
+        self.assertTrue(os.path.isfile(os.path.join(staging_libdir, 'libftdi1.so.2.1.0')), 'libftdi binary not found in STAGING_LIBDIR')
+        # Test devtool reset
+        stampprefix = get_bb_var('STAMP', 'libftdi')
+        result = runCmd('devtool reset libftdi')
+        result = runCmd('devtool status')
+        self.assertNotIn('libftdi', result.output)
+        self.assertTrue(stampprefix, 'Unable to get STAMP value for recipe libftdi')
+        matches = glob.glob(stampprefix + '*')
+        self.assertFalse(matches, 'Stamp files exist for recipe libftdi that should have been cleaned')
+        self.assertFalse(os.path.isfile(os.path.join(staging_libdir, 'libftdi1.so.2.1.0')), 'libftdi binary still found in STAGING_LIBDIR after cleaning')
+
     def test_devtool_modify(self):
         # Check preconditions
         workspacedir = os.path.join(self.builddir, 'workspace')
scripts/lib/devtool/standard.py +3 -2
index 763177d..9b5a085 100644
@@ -502,7 +502,8 @@ def build(args, config, basepath, workspace):
     if not args.recipename in workspace:
         logger.error("no recipe named %s in your workspace" % args.recipename)
         return -1
-    exec_build_env_command(config.init_path, basepath, 'bitbake -c install %s' % args.recipename, watch=True)
+    build_task = config.get('Build', 'build_task', 'populate_sysroot')
+    exec_build_env_command(config.init_path, basepath, 'bitbake -c %s %s' % (build_task, args.recipename), watch=True)
 
     return 0
 
@@ -551,7 +552,7 @@ def register_commands(subparsers, context):
     parser_status.set_defaults(func=status)
 
     parser_build = subparsers.add_parser('build', help='Build a recipe',
-                                         description='Builds the specified recipe using bitbake (up to do_install)',
+                                         description='Builds the specified recipe using bitbake',
                                          formatter_class=argparse.ArgumentDefaultsHelpFormatter)
     parser_build.add_argument('recipename', help='Recipe to build')
     parser_build.set_defaults(func=build)
Signature · 1 line
1.9.3
#10 lore raw

PATCH 10/15 lib/oe/recipeutils: minor improvements to patch_recipe()

In reply to Paul Eggleton

* Ensure it knows where to put SRCREV and S
* Handle prepend/append and functions in general

---
 meta/lib/oe/recipeutils.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
meta/lib/oe/recipeutils.py +2 -2
index 1758dce..159a103 100644
@@ -18,7 +18,7 @@ from collections import OrderedDict, defaultdict
 
 
 # Help us to find places to insert values
-recipe_progression = ['SUMMARY', 'DESCRIPTION', 'HOMEPAGE', 'BUGTRACKER', 'SECTION', 'LICENSE', 'LIC_FILES_CHKSUM', 'PROVIDES', 'DEPENDS', 'PR', 'PV', 'SRC_URI', 'do_fetch', 'do_unpack', 'do_patch', 'EXTRA_OECONF', 'do_configure', 'EXTRA_OEMAKE', 'do_compile', 'do_install', 'do_populate_sysroot', 'INITSCRIPT', 'USERADD', 'GROUPADD', 'PACKAGES', 'FILES', 'RDEPENDS', 'RRECOMMENDS', 'RSUGGESTS', 'RPROVIDES', 'RREPLACES', 'RCONFLICTS', 'ALLOW_EMPTY', 'do_package', 'do_deploy']
+recipe_progression = ['SUMMARY', 'DESCRIPTION', 'HOMEPAGE', 'BUGTRACKER', 'SECTION', 'LICENSE', 'LIC_FILES_CHKSUM', 'PROVIDES', 'DEPENDS', 'PR', 'PV', 'SRCREV', 'SRC_URI', 'S', 'do_fetch', 'do_unpack', 'do_patch', 'EXTRA_OECONF', 'do_configure', 'EXTRA_OEMAKE', 'do_compile', 'do_install', 'do_populate_sysroot', 'INITSCRIPT', 'USERADD', 'GROUPADD', 'PACKAGES', 'FILES', 'RDEPENDS', 'RRECOMMENDS', 'RSUGGESTS', 'RPROVIDES', 'RREPLACES', 'RCONFLICTS', 'ALLOW_EMPTY', 'do_package', 'do_deploy']
 # Variables that sometimes are a bit long but shouldn't be wrapped
 nowrap_vars = ['SUMMARY', 'HOMEPAGE', 'BUGTRACKER']
 list_vars = ['SRC_URI', 'LIC_FILES_CHKSUM']
@@ -113,7 +113,7 @@ def patch_recipe_file(fn, values, patch=False, relpath=''):
                         insert = False
                         for k in remainingnames.keys():
                             for p in recipe_progression:
-                                if re.match('^%s[ ?:=]' % p, line):
+                                if re.match('^%s(_prepend|_append)*[ ?:=(]' % p, line):
                                     if remainingnames[k] > -1 and recipe_progression.index(p) > remainingnames[k] and runthrough > 1 and not k in existingnames:
                                         outputvalue(k)
                                         del remainingnames[k]
Signature · 1 line
1.9.3
#11 lore raw

PATCH 11/15 devtool: update-recipe: add handling for git recipes

In reply to Paul Eggleton

When updating git-based recipes, in a lot of cases what you want is to push the changes to the repository and update SRCREV rather than to apply patches within the recipe. Updating SRCREV is now the default behaviour for recipes that fetch from git, but this can be overridden in both directions using a new -m/--mode option.

---
 meta/lib/oeqa/selftest/devtool.py |  70 +++++++++++-
 scripts/lib/devtool/standard.py   | 230 ++++++++++++++++++++++++--------------
 2 files changed, 213 insertions(+), 87 deletions(-)
meta/lib/oeqa/selftest/devtool.py +69 -1
index 8caf07a..f147f24 100644
@@ -233,6 +233,8 @@ class DevtoolTests(oeSelfTest):
         self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
         testrecipe = 'minicom'
         recipefile = get_bb_var('FILE', testrecipe)
+        src_uri = get_bb_var('SRC_URI', testrecipe)
+        self.assertNotIn('git://', src_uri, 'This test expects the %s recipe to NOT be a git recipe' % testrecipe)
         result = runCmd('git status . --porcelain', cwd=os.path.dirname(recipefile))
         self.assertEqual(result.output.strip(), "", '%s recipe is not clean' % testrecipe)
         # First, modify a recipe
@@ -266,11 +268,77 @@ class DevtoolTests(oeSelfTest):
                 self.assertEqual(line[:3], '?? ', 'Unexpected status in line: %s' % line)
             elif line.endswith('0002-Add-a-new-file.patch'):
                 self.assertEqual(line[:3], '?? ', 'Unexpected status in line: %s' % line)
-            elif re.search('minicom_[^_]*.bb$', line):
+            elif re.search('%s_[^_]*.bb$' % testrecipe, line):
                 self.assertEqual(line[:3], ' M ', 'Unexpected status in line: %s' % line)
             else:
                 raise AssertionError('Unexpected modified file in status: %s' % line)
 
+    def test_devtool_update_recipe_git(self):
+        # Check preconditions
+        workspacedir = os.path.join(self.builddir, 'workspace')
+        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        testrecipe = 'mtd-utils'
+        recipefile = get_bb_var('FILE', testrecipe)
+        src_uri = get_bb_var('SRC_URI', testrecipe)
+        self.assertIn('git://', src_uri, 'This test expects the %s recipe to be a git recipe' % testrecipe)
+        result = runCmd('git status . --porcelain', cwd=os.path.dirname(recipefile))
+        self.assertEqual(result.output.strip(), "", '%s recipe is not clean' % testrecipe)
+        # First, modify a recipe
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        self.track_for_cleanup(workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        # (don't bother with cleaning the recipe on teardown, we won't be building it)
+        result = runCmd('devtool modify %s -x %s' % (testrecipe, tempdir))
+        # Check git repo
+        self.assertTrue(os.path.isdir(os.path.join(tempdir, '.git')), 'git repository for external source tree not found')
+        result = runCmd('git status --porcelain', cwd=tempdir)
+        self.assertEqual(result.output.strip(), "", 'Created git repo is not clean')
+        result = runCmd('git symbolic-ref HEAD', cwd=tempdir)
+        self.assertEqual(result.output.strip(), "refs/heads/devtool", 'Wrong branch in git repo')
+        # Add a couple of commits
+        # FIXME: this only tests adding, need to also test update and remove
+        result = runCmd('echo "# Additional line" >> Makefile', cwd=tempdir)
+        result = runCmd('git commit -a -m "Change the Makefile"', cwd=tempdir)
+        result = runCmd('echo "A new file" > devtool-new-file', cwd=tempdir)
+        result = runCmd('git add devtool-new-file', cwd=tempdir)
+        result = runCmd('git commit -m "Add a new file"', cwd=tempdir)
+        self.add_command_to_tearDown('cd %s; git checkout %s %s' % (os.path.dirname(recipefile), testrecipe, os.path.basename(recipefile)))
+        result = runCmd('devtool update-recipe %s' % testrecipe)
+        result = runCmd('git status . --porcelain', cwd=os.path.dirname(recipefile))
+        self.assertNotEqual(result.output.strip(), "", '%s recipe should be modified' % testrecipe)
+        status = result.output.splitlines()
+        self.assertEqual(len(status), 3, 'Less/more files modified than expected. Entire status:\n%s' % result.output)
+        for line in status:
+            if line.endswith('add-exclusion-to-mkfs-jffs2-git-2.patch'):
+                self.assertEqual(line[:3], ' D ', 'Unexpected status in line: %s' % line)
+            elif line.endswith('fix-armv7-neon-alignment.patch'):
+                self.assertEqual(line[:3], ' D ', 'Unexpected status in line: %s' % line)
+            elif re.search('%s_[^_]*.bb$' % testrecipe, line):
+                self.assertEqual(line[:3], ' M ', 'Unexpected status in line: %s' % line)
+            else:
+                raise AssertionError('Unexpected modified file in status: %s' % line)
+        result = runCmd('git diff %s' % os.path.basename(recipefile), cwd=os.path.dirname(recipefile))
+        addlines = ['SRCREV = ".*"', 'SRC_URI = "git://git.infradead.org/mtd-utils.git"']
+        removelines = ['SRCREV = ".*"', 'SRC_URI = "git://git.infradead.org/mtd-utils.git \\\\', 'file://add-exclusion-to-mkfs-jffs2-git-2.patch \\\\', 'file://fix-armv7-neon-alignment.patch \\\\', '"']
+        for line in result.output.splitlines():
+            if line.startswith('+++') or line.startswith('---'):
+                continue
+            elif line.startswith('+'):
+                matched = False
+                for item in addlines:
+                    if re.match(item, line[1:].strip()):
+                        matched = True
+                        break
+                self.assertTrue(matched, 'Unexpected diff add line: %s' % line)
+            elif line.startswith('-'):
+                matched = False
+                for item in removelines:
+                    if re.match(item, line[1:].strip()):
+                        matched = True
+                        break
+                self.assertTrue(matched, 'Unexpected diff remove line: %s' % line)
+
     def test_devtool_extract(self):
         # Check preconditions
         workspacedir = os.path.join(self.builddir, 'workspace')
scripts/lib/devtool/standard.py +144 -86

257 lines not on this page · raw message

Signature · 1 line
1.9.3
#12 lore raw

PATCH 12/15 devtool: prevent extract/modify use with image recipes

In reply to Paul Eggleton

There wouldn't be any point to using these with an image recipe, so disallow it.

---
 scripts/lib/devtool/standard.py | 8 ++++++++
 1 file changed, 8 insertions(+)
scripts/lib/devtool/standard.py +8 -0
index 3a8c66c..9ab3aa6 100644
@@ -141,6 +141,10 @@ def _extract_source(srctree, keep_temp, devbranch, d):
         logger.error("The %s recipe uses a shared workdir which this tool does not currently support" % pn)
         return None
 
+    if bb.data.inherits_class('image', d):
+        logger.error("The %s recipe is an image, and therefore is not supported by this tool" % pn)
+        return None
+
     if bb.data.inherits_class('externalsrc', d) and d.getVar('EXTERNALSRC', True):
         logger.error("externalsrc is currently enabled for the %s recipe. This prevents the normal do_patch task from working. You will need to disable this first." % pn)
         return None
@@ -290,6 +294,10 @@ def modify(args, config, basepath, workspace):
         return -1
     rd = oe.recipeutils.parse_recipe(recipefile, tinfoil.config_data)
 
+    if bb.data.inherits_class('image', rd):
+        logger.error("The %s recipe is an image, and therefore is not supported by this tool" % args.recipename)
+        return None
+
     initial_rev = None
     commits = []
     srctree = os.path.abspath(args.srctree)
Signature · 1 line
1.9.3
#13 lore raw

PATCH 13/15 devtool: modify: handle recipes that use a shared workdir

In reply to Paul Eggleton

If S is outside of WORKDIR then we shouldn't try to get the relative path in order to work out where the source subdirectory is as we do by default.

---
 scripts/lib/devtool/standard.py | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
scripts/lib/devtool/standard.py +6 -4
index 9ab3aa6..4cb03f3 100644
@@ -313,12 +313,14 @@ def modify(args, config, basepath, workspace):
             (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=args.srctree)
             initial_rev = stdout.rstrip()
 
-    # Handle if S is set to a subdirectory of the source
+    # Check that recipe isn't using a shared workdir
     s = rd.getVar('S', True)
     workdir = rd.getVar('WORKDIR', True)
-    if s != workdir and os.path.dirname(s) != workdir:
-        srcsubdir = os.sep.join(os.path.relpath(s, workdir).split(os.sep)[1:])
-        srctree = os.path.join(srctree, srcsubdir)
+    if s.startswith(workdir):
+        # Handle if S is set to a subdirectory of the source
+        if s != workdir and os.path.dirname(s) != workdir:
+            srcsubdir = os.sep.join(os.path.relpath(s, workdir).split(os.sep)[1:])
+            srctree = os.path.join(srctree, srcsubdir)
 
     appendpath = os.path.join(config.workspace_path, 'appends')
     if not os.path.exists(appendpath):
Signature · 1 line
1.9.3
#14 lore raw

PATCH 14/15 devtool: support source extraction for recipes that use a shared workdir

In reply to Paul Eggleton

Enable source extraction used by devtool extract / devtool modify -x for recipes that use a shared workdir (e.g. the kernel and gcc).

---
 scripts/lib/devtool/standard.py | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)
scripts/lib/devtool/standard.py +14 -5
index 4cb03f3..bf18aae 100644
@@ -137,10 +137,6 @@ def _extract_source(srctree, keep_temp, devbranch, d):
         logger.error("The perf recipe does not actually check out source and thus cannot be supported by this tool")
         return None
 
-    if 'work-shared' in d.getVar('S', True):
-        logger.error("The %s recipe uses a shared workdir which this tool does not currently support" % pn)
-        return None
-
     if bb.data.inherits_class('image', d):
         logger.error("The %s recipe is an image, and therefore is not supported by this tool" % pn)
         return None
@@ -169,6 +165,16 @@ def _extract_source(srctree, keep_temp, devbranch, d):
         workdir = os.path.join(tempdir, 'workdir')
         crd.setVar('WORKDIR', workdir)
         crd.setVar('T', os.path.join(tempdir, 'temp'))
+        if not crd.getVar('S', True).startswith(workdir):
+            # Usually a shared workdir recipe (kernel, gcc)
+            # Try to set a reasonable default
+            if bb.data.inherits_class('kernel', d):
+                crd.setVar('S', '${WORKDIR}/source')
+            else:
+                crd.setVar('S', '${WORKDIR}/${BP}')
+        if bb.data.inherits_class('kernel', d):
+            # We don't want to move the source to STAGING_KERNEL_DIR here
+            crd.setVar('STAGING_KERNEL_DIR', '${S}')
 
         # FIXME: This is very awkward. Unfortunately it's not currently easy to properly
         # execute tasks outside of bitbake itself, until then this has to suffice if we
@@ -204,7 +210,10 @@ def _extract_source(srctree, keep_temp, devbranch, d):
             else:
                 os.rmdir(patchdir)
 
-        if not bb.data.inherits_class('kernel-yocto', d):
+        if bb.data.inherits_class('kernel-yocto', d):
+            (stdout, _) = bb.process.run('git --git-dir="%s" rev-parse HEAD' % crd.expand('${WORKDIR}/git'), cwd=srcsubdir)
+            initial_rev = stdout.rstrip()
+        else:
             if not os.listdir(srcsubdir):
                 logger.error("no source unpacked to S, perhaps the %s recipe doesn't use any source?" % pn)
                 return None
Signature · 1 line
1.9.3
#15 lore raw

PATCH 15/15 devtool: fix broken clones of git recipes

In reply to Paul Eggleton

Because we move the workdir when extracting source, then move the source and delete the temporary workdir, you lose the indirection symlink pointed to by the alternates file (which is created when the fetcher clones it from DL_DIR with -s) and the resulting repository is broken. In any case, for a source repo that the user may put their own changes into, we can't really rely on a clone made with -s in case the original goes away - because of cleanall, DL_DIR disappearing, etc. So repack the repository so that it is a complete, non-shared clone after unpacking.

(While I'm at it, add a test for devtool modify with a git recipe which verifies that this works.)

---
 meta/lib/oeqa/selftest/devtool.py | 33 +++++++++++++++++++++++++++++++++
 scripts/lib/devtool/standard.py   |  7 +++++++
 2 files changed, 40 insertions(+)
meta/lib/oeqa/selftest/devtool.py +33 -0
index f147f24..932d6b9 100644
@@ -227,6 +227,39 @@ class DevtoolTests(oeSelfTest):
         matches = glob.glob(stampprefix + '*')
         self.assertFalse(matches, 'Stamp files exist for recipe mdadm that should have been cleaned')
 
+    def test_devtool_modify_git(self):
+        # Check preconditions
+        workspacedir = os.path.join(self.builddir, 'workspace')
+        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        testrecipe = 'mkelfimage'
+        src_uri = get_bb_var('SRC_URI', testrecipe)
+        self.assertIn('git://', src_uri, 'This test expects the %s recipe to be a git recipe' % testrecipe)
+        # Clean up anything in the workdir/sysroot/sstate cache
+        bitbake('%s -c cleansstate' % testrecipe)
+        # Try modifying a recipe
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        self.track_for_cleanup(workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        self.add_command_to_tearDown('bitbake -c clean %s' % testrecipe)
+        result = runCmd('devtool modify %s -x %s' % (testrecipe, tempdir))
+        self.assertTrue(os.path.exists(os.path.join(tempdir, 'Makefile')), 'Extracted source could not be found')
+        self.assertTrue(os.path.isdir(os.path.join(tempdir, '.git')), 'git repository for external source tree not found')
+        self.assertTrue(os.path.exists(os.path.join(workspacedir, 'conf', 'layer.conf')), 'Workspace directory not created')
+        matches = glob.glob(os.path.join(workspacedir, 'appends', 'mkelfimage_*.bbappend'))
+        self.assertTrue(matches, 'bbappend not created')
+        # Test devtool status
+        result = runCmd('devtool status')
+        self.assertIn(testrecipe, result.output)
+        self.assertIn(tempdir, result.output)
+        # Check git repo
+        result = runCmd('git status --porcelain', cwd=tempdir)
+        self.assertEqual(result.output.strip(), "", 'Created git repo is not clean')
+        result = runCmd('git symbolic-ref HEAD', cwd=tempdir)
+        self.assertEqual(result.output.strip(), "refs/heads/devtool", 'Wrong branch in git repo')
+        # Try building
+        bitbake(testrecipe)
+
     def test_devtool_update_recipe(self):
         # Check preconditions
         workspacedir = os.path.join(self.builddir, 'workspace')
scripts/lib/devtool/standard.py +7 -0
index bf18aae..cabf3fe 100644
@@ -202,6 +202,13 @@ def _extract_source(srctree, keep_temp, devbranch, d):
             # Handle if S is set to a subdirectory of the source
             srcsubdir = os.path.join(workdir, os.path.relpath(srcsubdir, workdir).split(os.sep)[0])
 
+        if os.path.exists(os.path.join(srcsubdir, '.git')):
+            alternatesfile = os.path.join(srcsubdir, '.git', 'objects', 'info', 'alternates')
+            if os.path.exists(alternatesfile):
+                # This will have been cloned with -s, so we need to convert it to a full clone
+                bb.process.run('git repack -a', cwd=srcsubdir)
+                os.remove(alternatesfile)
+
         patchdir = os.path.join(srcsubdir, 'patches')
         haspatches = False
         if os.path.exists(patchdir):
Signature · 1 line
1.9.3
#16 lore raw
In reply to Paul Eggleton
Quoted 3 lines · Paul Eggleton

I've been attempting to enable usage of devtool with the kernel (which effectively means using the externalsrc class with the kernel). This doesn't really work yet due to lack of handling the kernel configuration,

I've always insisted/argued and designed in that externalsrc shouldn't touch the kernel configuration at all.

You are building a tree that is completely managed outside of the build system, and getting the kernel configuration in place is part of that management.

So I'm not convinced it is a problem that needs to be solved.

Cheers,

Bruce

Quoted 51 lines

so you still can't use externalsrc properly with the kernel, but this at least gets us further - you can now extract the source successfully and run a build. I also discovered and fixed a number of related issues along the way, and made a few behavioural improvements in both devtool and externalsrc.

The following changes since commit ad016fa7492ba23c751cf33dd1befde7782facfa:

lzo: Upgrade to 2.09 and add ptest (2015-02-19 07:50:44 +0000)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib paule/devtool-fixes
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool-fixes

Paul Eggleton (15):
recipetool: fix regression caused by previous commit lib/oe/patch: fix PATCHTOOL = "git" with source in a subdirectory classes/externalsrc: fix source being wiped out on clean with kernel classes/externalsrc: add workaround for recipes that use SRCPV in PV classes/externalsrc: show a warning on compiling classes/kernel-yocto: extend SRCTREECOVEREDTASKS
devtool: add/modify: add option to build in same directory
devtool: reset: run bitbake -c clean on recipe
devtool: build: run do_populate_sysroot instead of do_install lib/oe/recipeutils: minor improvements to patch_recipe()
devtool: update-recipe: add handling for git recipes
devtool: prevent extract/modify use with image recipes
devtool: modify: handle recipes that use a shared workdir
devtool: support source extraction for recipes that use a shared
workdir
devtool: fix broken clones of git recipes

 meta/classes/externalsrc.bbclass  |  24 ++++
 meta/classes/kernel-yocto.bbclass |   2 +-
 meta/lib/oe/patch.py              |   7 +-
 meta/lib/oe/recipeutils.py        |   4 +-
 meta/lib/oeqa/selftest/devtool.py | 147 ++++++++++++++++++-
 scripts/lib/devtool/standard.py   | 292 +++++++++++++++++++++++++-------------
 scripts/lib/recipetool/create.py  |   8 +-
 7 files changed, 377 insertions(+), 107 deletions(-)

--
1.9.3

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"
#17 lore raw
Quoted 6 lines · On Thursday 19 February 2015 11:47:01 Bruce Ashfield wrote:
Quoted 3 lines · Paul Eggleton

I've been attempting to enable usage of devtool with the kernel (which effectively means using the externalsrc class with the kernel). This doesn't really work yet due to lack of handling the kernel configuration,

Quoted 8 lines

I've always insisted/argued and designed in that externalsrc shouldn't touch the kernel configuration at all.

You are building a tree that is completely managed outside of the build system, and getting the kernel configuration in place is part of that management.

So I'm not convinced it is a problem that needs to be solved.

So, externalsrc still doesn't touch the configuration, and it is not my plan to make it do so. However, users do need some way of setting up that configuration when checking out those sources for the first time, and if that's to be of any use to someone who just wants to make some minor changes, it should be the exact same configuration you get when you build the recipe as normal. That is what I believe "devtool modify" (at least with -x) should be doing, given that it is one means you can use to set up a development environment if you do not have one already.

We can certainly talk about the mechanism and work out the best way to do it, such that it fits in with how linux-yocto works and meets user expectations. My intent with this series was to get us to the point where we can at least experiment and have that discussion.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#18 lore raw
Quoted 25 lines · Paul Eggleton
Quoted 6 lines · On Thursday 19 February 2015 11:47:01 Bruce Ashfield wrote:
Quoted 3 lines · Paul Eggleton

I've been attempting to enable usage of devtool with the kernel (which effectively means using the externalsrc class with the kernel). This doesn't really work yet due to lack of handling the kernel configuration,

Quoted 8 lines

I've always insisted/argued and designed in that externalsrc shouldn't touch the kernel configuration at all.

You are building a tree that is completely managed outside of the build system, and getting the kernel configuration in place is part of that management.

So I'm not convinced it is a problem that needs to be solved.

So, externalsrc still doesn't touch the configuration, and it is not my plan to make it do so. However, users do need some way of setting up that configuration when checking out those sources for the first time, and if that's to be of any use to someone who just wants to make some minor changes, it should be the exact same configuration you get when you build the recipe as normal. That is what I believe "devtool modify" (at least with -x) should be doing, given that it is one means you can use to set up a development environment if you do not have one already.

Quoted 4 lines

We can certainly talk about the mechanism and work out the best way to do it, such that it fits in with how linux-yocto works and meets user expectations. My intent with this series was to get us to the point where we can at least experiment and have that discussion.

Having the option of triggering the standard configuration process as per a usual linux-yocto build but on an external direction, standalone may be the best option?

Cheers,

Richard

#19 lore raw
Quoted 26 lines · Richard Purdie
Quoted 25 lines · Paul Eggleton
Quoted 6 lines · On Thursday 19 February 2015 11:47:01 Bruce Ashfield wrote:
Quoted 3 lines · Paul Eggleton

I've been attempting to enable usage of devtool with the kernel (which effectively means using the externalsrc class with the kernel). This doesn't really work yet due to lack of handling the kernel configuration,

Quoted 8 lines

I've always insisted/argued and designed in that externalsrc shouldn't touch the kernel configuration at all.

You are building a tree that is completely managed outside of the build system, and getting the kernel configuration in place is part of that management.

So I'm not convinced it is a problem that needs to be solved.

So, externalsrc still doesn't touch the configuration, and it is not my plan to make it do so. However, users do need some way of setting up that configuration when checking out those sources for the first time, and if that's to be of any use to someone who just wants to make some minor changes, it should be the exact same configuration you get when you build the recipe as normal. That is what I believe "devtool modify" (at least with -x) should be doing, given that it is one means you can use to set up a development environment if you do not have one already.

Quoted 8 lines
Quoted 4 lines

We can certainly talk about the mechanism and work out the best way to do it, such that it fits in with how linux-yocto works and meets user expectations. My intent with this series was to get us to the point where we can at least experiment and have that discussion.

Having the option of triggering the standard configuration process as per a usual linux-yocto build but on an external direction, standalone may be the best option?

I did try that before, and the tools to run on random trees (I use it for building
the linux-yocto kernel source out of tree), but they need the fragments and meta-data to do that configuration.

and when a externalsrc tree is unpacked and checked out, that data won't by lying around (that being said, I haven't tried the series to see what exactly is happening).

I always considered externalsrc to be: "here's my tree, I've set it up and it is ready to build .. go do it!", so I need to think about the best way to get that next step done (for any kernel, not just linux yocto).

Bruce

Quoted 5 lines

Cheers,

Richard

Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"
#20 lore raw
In reply to Paul Eggleton
Quoted 14 lines · Paul Eggleton
Quoted 13 lines · On Thursday 19 February 2015 11:47:01 Bruce Ashfield wrote:
Quoted 3 lines · Paul Eggleton

I've been attempting to enable usage of devtool with the kernel (which effectively means using the externalsrc class with the kernel). This doesn't really work yet due to lack of handling the kernel configuration,

I've always insisted/argued and designed in that externalsrc shouldn't touch the kernel configuration at all.

You are building a tree that is completely managed outside of the build system, and getting the kernel configuration in place is part of that management.

Quoted 5 lines
Quoted 1 line

So I'm not convinced it is a problem that needs to be solved.

So, externalsrc still doesn't touch the configuration, and it is not my plan to make it do so. However, users do need some way of setting up that configuration when checking out those sources for the first time, and if that's to be of any

Right. Something to think on.

Quoted 10 lines

use to someone who just wants to make some minor changes, it should be the exact same configuration you get when you build the recipe as normal. That is what I believe "devtool modify" (at least with -x) should be doing, given that it is one means you can use to set up a development environment if you do not have one already.

We can certainly talk about the mechanism and work out the best way to do it, such that it fits in with how linux-yocto works and meets user expectations. My intent with this series was to get us to the point where we can at least experiment and have that discussion.

Agreed.

Bruce

Quoted 8 lines

Cheers,
Paul

--

Paul Eggleton
Intel Open Source Technology Centre

Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"