devtool and externalsrc improvements
cover devtool and externalsrc improvements
I've been attempting to enable usage of devtool with the kernel (which effectively means using the externalsrc class with the kernel). This doesn't really work yet due to lack of handling the kernel configuration, so you still can't use externalsrc properly with the kernel, but this at least gets us further - you can now extract the source successfully and run a build. I also discovered and fixed a number of related issues along the way, and made a few behavioural improvements in both devtool and externalsrc.
The following changes since commit ad016fa7492ba23c751cf33dd1befde7782facfa:
lzo: Upgrade to 2.09 and add ptest (2015-02-19 07:50:44 +0000)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib paule/devtool-fixes
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool-fixes
Paul Eggleton (15):
recipetool: fix regression caused by previous commit lib/oe/patch: fix PATCHTOOL = "git" with source in a subdirectory classes/externalsrc: fix source being wiped out on clean with kernel classes/externalsrc: add workaround for recipes that use SRCPV in PV classes/externalsrc: show a warning on compiling classes/kernel-yocto: extend SRCTREECOVEREDTASKS
devtool: add/modify: add option to build in same directory
devtool: reset: run bitbake -c clean on recipe
devtool: build: run do_populate_sysroot instead of do_install lib/oe/recipeutils: minor improvements to patch_recipe()
devtool: update-recipe: add handling for git recipes
devtool: prevent extract/modify use with image recipes
devtool: modify: handle recipes that use a shared workdir
devtool: support source extraction for recipes that use a shared
workdir
devtool: fix broken clones of git recipes
meta/classes/externalsrc.bbclass | 24 ++++ meta/classes/kernel-yocto.bbclass | 2 +- meta/lib/oe/patch.py | 7 +- meta/lib/oe/recipeutils.py | 4 +- meta/lib/oeqa/selftest/devtool.py | 147 ++++++++++++++++++- scripts/lib/devtool/standard.py | 292 +++++++++++++++++++++++++------------- scripts/lib/recipetool/create.py | 8 +- 7 files changed, 377 insertions(+), 107 deletions(-)
Signature · 1 line
1.9.3
PATCH 1/15 recipetool: fix regression caused by previous commit
Option was renamed in the setup code but not in the code that used it.
--- scripts/lib/recipetool/create.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/lib/recipetool/create.py +4 -4
@@ -110,8 +110,8 @@ def create_recipe(args): if '://' in args.source: # Fetch a URL srcuri = args.source - if args.externalsrc: - srctree = args.externalsrc + if args.extract_to: + srctree = args.extract_to else: tempsrc = tempfile.mkdtemp(prefix='recipetool-') srctree = tempsrc
@@ -126,8 +126,8 @@ def create_recipe(args): srctree = os.path.join(srctree, srcsubdir) else: # Assume we're pointing to an existing source tree - if args.externalsrc: - logger.error('externalsrc cannot be specified if source is a directory') + if args.extract_to: + logger.error('--extract-to cannot be specified if source is a directory') sys.exit(1) if not os.path.isdir(args.source): logger.error('Invalid source directory %s' % args.source)
Signature · 1 line
1.9.3
PATCH 2/15 lib/oe/patch: fix PATCHTOOL = "git" with source in a subdirectory
In reply to Paul EggletonFor recipes that have their actual source in a subdirectory of what is fetched (e.g. mkelfimage), we need to find the root of the repository within the GitApplyTree code that attempts to set up the required git hooks and use that, rather than expecting the root to be the same as ${S}.
--- meta/lib/oe/patch.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
meta/lib/oe/patch.py +5 -2
@@ -313,9 +313,12 @@ class GitApplyTree(PatchTree): return runcmd(["sh", "-c", " ".join(shellcmd)], self.dir) # Add hooks which add a pointer to the original patch file name in the commit message - commithook = os.path.join(self.dir, '.git', 'hooks', 'commit-msg') + reporoot = (runcmd("git rev-parse --show-toplevel".split(), self.dir) or '').strip() + if not reporoot: + raise Exception("Cannot get repository root for directory %s" % self.dir) + commithook = os.path.join(reporoot, '.git', 'hooks', 'commit-msg') commithook_backup = commithook + '.devtool-orig' - applyhook = os.path.join(self.dir, '.git', 'hooks', 'applypatch-msg') + applyhook = os.path.join(reporoot, '.git', 'hooks', 'applypatch-msg') applyhook_backup = applyhook + '.devtool-orig' if os.path.exists(commithook): shutil.move(commithook, commithook_backup)
Signature · 1 line
1.9.3
PATCH 3/15 classes/externalsrc: fix source being wiped out on clean with kernel
In reply to Paul Eggletonkernel.bbclass adds ${S} do do_clean[cleandirs], but this means if you run bitbake -c clean <kernelrecipe> then your external source tree will be trashed, which could be a disaster. For safety, remove ${S} from cleandirs for every task. We also have to do the same for ${B} in the case where EXTERNALSRC_BUILD is set to the same value as EXTERNALSRC.
--- meta/classes/externalsrc.bbclass | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
meta/classes/externalsrc.bbclass +14 -0
@@ -47,6 +47,20 @@ python () { # Since configure will likely touch ${S}, ensure only we lock so one task has access at a time d.appendVarFlag(task, "lockfiles", "${S}/singletask.lock") + # We do not want our source to be wiped out, ever (kernel.bbclass does this for do_clean) + cleandirs = d.getVarFlag(task, 'cleandirs', False) + if cleandirs: + cleandirs = cleandirs.split() + setvalue = False + if '${S}' in cleandirs: + cleandirs.remove('${S}') + setvalue = True + if externalsrcbuild == externalsrc and '${B}' in cleandirs: + cleandirs.remove('${B}') + setvalue = True + if setvalue: + d.setVarFlag(task, 'cleandirs', ' '.join(cleandirs)) + for task in d.getVar("SRCTREECOVEREDTASKS", True).split(): bb.build.deltask(task, d)
Signature · 1 line
1.9.3
PATCH 4/15 classes/externalsrc: add workaround for recipes that use SRCPV in PV
In reply to Paul EggletonHere we set SRC_URI to blank, however doing so means that the function that is called when you expand the default value of SRCPV (i.e. bb.fetch2.get_srcrev()) will fail, so any recipe that references SRCPV in PV couldn't previously be used with externalsrc.
(At some point we may fix the function to work in the externalsrc case, but then we would also need to ensure that ${B} did not change as a result of PV changing any time the HEAD revision changes in the external source tree, or you'll lose any intermediate build artifacts.)
--- meta/classes/externalsrc.bbclass | 4 ++++ 1 file changed, 4 insertions(+)
meta/classes/externalsrc.bbclass +4 -0
@@ -37,6 +37,10 @@ python () { d.setVar('B', '${WORKDIR}/${BPN}-${PV}/') d.setVar('SRC_URI', '') + if '{SRCPV}' in d.getVar('PV', False): + # Dummy value because the default function can't be called with blank SRC_URI + d.setVar('SRCPV', '999') + tasks = filter(lambda k: d.getVarFlag(k, "task"), d.keys()) for task in tasks:
Signature · 1 line
1.9.3
PATCH 5/15 classes/externalsrc: show a warning on compiling
In reply to Paul EggletonMake sure there's no chance of anyone forgetting they have a recipe set up for externalsrc; otherwise you could get confused about what is going on. (With our default logging setup we can't make it a note because the UI doesn't forward those; otherwise I would have used bb.note().)
--- meta/classes/externalsrc.bbclass | 6 ++++++ 1 file changed, 6 insertions(+)
meta/classes/externalsrc.bbclass +6 -0
@@ -68,7 +68,13 @@ python () { for task in d.getVar("SRCTREECOVEREDTASKS", True).split(): bb.build.deltask(task, d) + d.prependVarFlag('do_compile', 'prefuncs', "externalsrc_compile_prefunc ") + # Ensure compilation happens every time d.setVarFlag('do_compile', 'nostamp', '1') } +python externalsrc_compile_prefunc() { + # Make it obvious that this is happening, since forgetting about it could lead to much confusion + bb.warn('Compiling %s from external source %s' % (d.getVar('PN', True), d.getVar('EXTERNALSRC', True))) +}
Signature · 1 line
1.9.3
PATCH 6/15 classes/kernel-yocto: extend SRCTREECOVEREDTASKS
In reply to Paul Eggleton* Add do_shared_workdir which was added recently
* Add do_fetch and do_unpack to this list, because at the moment if you enable externalsrc through a bbappend the += in this class wipes out the original value from externalsrc (which is set with ?=)
--- meta/classes/kernel-yocto.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/kernel-yocto.bbclass +1 -1
@@ -1,5 +1,5 @@ # remove tasks that modify the source tree in case externalsrc is inherited -SRCTREECOVEREDTASKS += "do_kernel_link_vmlinux do_kernel_configme do_validate_branches do_kernel_configcheck do_kernel_checkout do_patch" +SRCTREECOVEREDTASKS += "do_kernel_link_vmlinux do_kernel_configme do_validate_branches do_kernel_configcheck do_kernel_checkout do_shared_workdir do_fetch do_unpack do_patch" # returns local (absolute) path names for all valid patches in the # src_uri
Signature · 1 line
1.9.3
PATCH 7/15 devtool: add/modify: add option to build in same directory
In reply to Paul EggletonThe default behaviour is to build in a separate directory to the source, however some projects can't be built this way, so add an option to do that (or override the automatic behaviour in the case of modify).
--- scripts/lib/devtool/standard.py | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
scripts/lib/devtool/standard.py +9 -2
@@ -77,6 +77,8 @@ def add(args, config, basepath, workspace): with open(appendfile, 'w') as f: f.write('inherit externalsrc\n') f.write('EXTERNALSRC = "%s"\n' % srctree) + if args.same_dir: + f.write('EXTERNALSRC_BUILD = "%s"\n' % srctree) if initial_rev: f.write('\n# initial_rev: %s\n' % initial_rev)
@@ -323,8 +325,11 @@ def modify(args, config, basepath, workspace): f.write('inherit externalsrc\n') f.write('# NOTE: We use pn- overrides here to avoid affecting multiple variants in the case where the recipe uses BBCLASSEXTEND\n') f.write('EXTERNALSRC_pn-%s = "%s"\n' % (args.recipename, srctree)) - if bb.data.inherits_class('autotools-brokensep', rd): - logger.info('using source tree as build directory since original recipe inherits autotools-brokensep') + if args.same_dir or bb.data.inherits_class('autotools-brokensep', rd): + if args.same_dir: + logger.info('using source tree as build directory since --same-dir specified') + else: + logger.info('using source tree as build directory since original recipe inherits autotools-brokensep') f.write('EXTERNALSRC_BUILD_pn-%s = "%s"\n' % (args.recipename, srctree)) if initial_rev: f.write('\n# initial_rev: %s\n' % initial_rev)
@@ -503,6 +508,7 @@ def register_commands(subparsers, context): formatter_class=argparse.ArgumentDefaultsHelpFormatter) parser_add.add_argument('recipename', help='Name for new recipe to add') parser_add.add_argument('srctree', help='Path to external source tree') + parser_add.add_argument('--same-dir', '-s', help='Build in same directory as source', action="store_true") parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)') parser_add.set_defaults(func=add)
@@ -513,6 +519,7 @@ def register_commands(subparsers, context): parser_add.add_argument('srctree', help='Path to external source tree') parser_add.add_argument('--wildcard', '-w', action="store_true", help='Use wildcard for unversioned bbappend') parser_add.add_argument('--extract', '-x', action="store_true", help='Extract source as well') + parser_add.add_argument('--same-dir', '-s', help='Build in same directory as source', action="store_true") parser_add.add_argument('--branch', '-b', default="devtool", help='Name for development branch to checkout (only when using -x)') parser_add.set_defaults(func=modify)
Signature · 1 line
1.9.3
PATCH 8/15 devtool: reset: run bitbake -c clean on recipe
In reply to Paul EggletonIf you've added a new recipe, you want the output cleaned when you do devtool reset, otherwise cruft from building the recipe may remain which could interfere with future builds.
--- meta/lib/oeqa/selftest/devtool.py | 4 ++++ scripts/lib/devtool/standard.py | 6 ++++++ 2 files changed, 10 insertions(+)
meta/lib/oeqa/selftest/devtool.py +4 -0
@@ -179,9 +179,13 @@ class DevtoolTests(oeSelfTest): if line.startswith('.TH'): self.assertEqual(line.rstrip(), '.TH MDADM 8 "" v9.999-custom', 'man file not modified') # Test devtool reset + stampprefix = get_bb_var('STAMP', 'mdadm') result = runCmd('devtool reset mdadm') result = runCmd('devtool status') self.assertNotIn('mdadm', result.output) + self.assertTrue(stampprefix, 'Unable to get STAMP value for recipe mdadm') + matches = glob.glob(stampprefix + '*') + self.assertFalse(matches, 'Stamp files exist for recipe mdadm that should have been cleaned') def test_devtool_update_recipe(self): # Check preconditions
scripts/lib/devtool/standard.py +6 -0
@@ -475,6 +475,11 @@ def reset(args, config, basepath, workspace): if not args.recipename in workspace: logger.error("no recipe named %s in your workspace" % args.recipename) return -1 + + if not args.no_clean: + logger.info('Cleaning sysroot for recipe %s...' % args.recipename) + exec_build_env_command(config.init_path, basepath, 'bitbake -c clean %s' % args.recipename) + _check_preserve(config, args.recipename) preservepath = os.path.join(config.workspace_path, 'attic', args.recipename)
@@ -555,5 +560,6 @@ def register_commands(subparsers, context): description='Removes the specified recipe from your workspace (resetting its state)', formatter_class=argparse.ArgumentDefaultsHelpFormatter) parser_reset.add_argument('recipename', help='Recipe to reset') + parser_reset.add_argument('--no-clean', '-n', action="store_true", help='Don\'t clean the sysroot to remove recipe output') parser_reset.set_defaults(func=reset)
Signature · 1 line
1.9.3
PATCH 9/15 devtool: build: run do_populate_sysroot instead of do_install
In reply to Paul EggletonIf you want to be able to make use of libraries in conjunction with devtool then we need to install them into the sysroot for other recipes to use. Make it a configuration option in case it needs to be changed at runtime.
--- meta/lib/oeqa/selftest/devtool.py | 40 +++++++++++++++++++++++++++++++++++++++ scripts/lib/devtool/standard.py | 5 +++-- 2 files changed, 43 insertions(+), 2 deletions(-)
meta/lib/oeqa/selftest/devtool.py +40 -0
@@ -136,6 +136,46 @@ class DevtoolTests(oeSelfTest): bindir = bindir[1:] self.assertTrue(os.path.isfile(os.path.join(installdir, bindir, 'pv')), 'pv binary not found in D') + def test_devtool_add_library(self): + # Check preconditions + workspacedir = os.path.join(self.builddir, 'workspace') + self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory') + # We don't have the ability to pick up this dependency automatically yet... + bitbake('libusb1') + # Fetch source + tempdir = tempfile.mkdtemp(prefix='devtoolqa') + self.track_for_cleanup(tempdir) + url = 'http://www.intra2net.com/en/developer/libftdi/download/libftdi1-1.1.tar.bz2' + result = runCmd('wget %s' % url, cwd=tempdir) + result = runCmd('tar xfv libftdi1-1.1.tar.bz2', cwd=tempdir) + srcdir = os.path.join(tempdir, 'libftdi1-1.1') + self.assertTrue(os.path.isfile(os.path.join(srcdir, 'CMakeLists.txt')), 'Unable to find CMakeLists.txt in source directory') + # Test devtool add + self.track_for_cleanup(workspacedir) + self.add_command_to_tearDown('bitbake-layers remove-layer */workspace') + result = runCmd('devtool add libftdi %s' % srcdir) + self.assertTrue(os.path.exists(os.path.join(workspacedir, 'conf', 'layer.conf')), 'Workspace directory not created') + # Test devtool status + result = runCmd('devtool status') + self.assertIn('libftdi', result.output) + self.assertIn(srcdir, result.output) + # Clean up anything in the workdir/sysroot/sstate cache (have to do this *after* devtool add since the recipe only exists then) + bitbake('libftdi -c cleansstate') + # Test devtool build + result = runCmd('devtool build libftdi') + staging_libdir = get_bb_var('STAGING_LIBDIR', 'libftdi') + self.assertTrue(staging_libdir, 'Could not query STAGING_LIBDIR variable') + self.assertTrue(os.path.isfile(os.path.join(staging_libdir, 'libftdi1.so.2.1.0')), 'libftdi binary not found in STAGING_LIBDIR') + # Test devtool reset + stampprefix = get_bb_var('STAMP', 'libftdi') + result = runCmd('devtool reset libftdi') + result = runCmd('devtool status') + self.assertNotIn('libftdi', result.output) + self.assertTrue(stampprefix, 'Unable to get STAMP value for recipe libftdi') + matches = glob.glob(stampprefix + '*') + self.assertFalse(matches, 'Stamp files exist for recipe libftdi that should have been cleaned') + self.assertFalse(os.path.isfile(os.path.join(staging_libdir, 'libftdi1.so.2.1.0')), 'libftdi binary still found in STAGING_LIBDIR after cleaning') + def test_devtool_modify(self): # Check preconditions workspacedir = os.path.join(self.builddir, 'workspace')
scripts/lib/devtool/standard.py +3 -2
@@ -502,7 +502,8 @@ def build(args, config, basepath, workspace): if not args.recipename in workspace: logger.error("no recipe named %s in your workspace" % args.recipename) return -1 - exec_build_env_command(config.init_path, basepath, 'bitbake -c install %s' % args.recipename, watch=True) + build_task = config.get('Build', 'build_task', 'populate_sysroot') + exec_build_env_command(config.init_path, basepath, 'bitbake -c %s %s' % (build_task, args.recipename), watch=True) return 0
@@ -551,7 +552,7 @@ def register_commands(subparsers, context): parser_status.set_defaults(func=status) parser_build = subparsers.add_parser('build', help='Build a recipe', - description='Builds the specified recipe using bitbake (up to do_install)', + description='Builds the specified recipe using bitbake', formatter_class=argparse.ArgumentDefaultsHelpFormatter) parser_build.add_argument('recipename', help='Recipe to build') parser_build.set_defaults(func=build)
Signature · 1 line
1.9.3
PATCH 10/15 lib/oe/recipeutils: minor improvements to patch_recipe()
In reply to Paul Eggleton* Ensure it knows where to put SRCREV and S
* Handle prepend/append and functions in general
--- meta/lib/oe/recipeutils.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
meta/lib/oe/recipeutils.py +2 -2
@@ -18,7 +18,7 @@ from collections import OrderedDict, defaultdict # Help us to find places to insert values -recipe_progression = ['SUMMARY', 'DESCRIPTION', 'HOMEPAGE', 'BUGTRACKER', 'SECTION', 'LICENSE', 'LIC_FILES_CHKSUM', 'PROVIDES', 'DEPENDS', 'PR', 'PV', 'SRC_URI', 'do_fetch', 'do_unpack', 'do_patch', 'EXTRA_OECONF', 'do_configure', 'EXTRA_OEMAKE', 'do_compile', 'do_install', 'do_populate_sysroot', 'INITSCRIPT', 'USERADD', 'GROUPADD', 'PACKAGES', 'FILES', 'RDEPENDS', 'RRECOMMENDS', 'RSUGGESTS', 'RPROVIDES', 'RREPLACES', 'RCONFLICTS', 'ALLOW_EMPTY', 'do_package', 'do_deploy'] +recipe_progression = ['SUMMARY', 'DESCRIPTION', 'HOMEPAGE', 'BUGTRACKER', 'SECTION', 'LICENSE', 'LIC_FILES_CHKSUM', 'PROVIDES', 'DEPENDS', 'PR', 'PV', 'SRCREV', 'SRC_URI', 'S', 'do_fetch', 'do_unpack', 'do_patch', 'EXTRA_OECONF', 'do_configure', 'EXTRA_OEMAKE', 'do_compile', 'do_install', 'do_populate_sysroot', 'INITSCRIPT', 'USERADD', 'GROUPADD', 'PACKAGES', 'FILES', 'RDEPENDS', 'RRECOMMENDS', 'RSUGGESTS', 'RPROVIDES', 'RREPLACES', 'RCONFLICTS', 'ALLOW_EMPTY', 'do_package', 'do_deploy'] # Variables that sometimes are a bit long but shouldn't be wrapped nowrap_vars = ['SUMMARY', 'HOMEPAGE', 'BUGTRACKER'] list_vars = ['SRC_URI', 'LIC_FILES_CHKSUM']
@@ -113,7 +113,7 @@ def patch_recipe_file(fn, values, patch=False, relpath=''): insert = False for k in remainingnames.keys(): for p in recipe_progression: - if re.match('^%s[ ?:=]' % p, line): + if re.match('^%s(_prepend|_append)*[ ?:=(]' % p, line): if remainingnames[k] > -1 and recipe_progression.index(p) > remainingnames[k] and runthrough > 1 and not k in existingnames: outputvalue(k) del remainingnames[k]
Signature · 1 line
1.9.3
PATCH 11/15 devtool: update-recipe: add handling for git recipes
In reply to Paul EggletonWhen updating git-based recipes, in a lot of cases what you want is to push the changes to the repository and update SRCREV rather than to apply patches within the recipe. Updating SRCREV is now the default behaviour for recipes that fetch from git, but this can be overridden in both directions using a new -m/--mode option.
--- meta/lib/oeqa/selftest/devtool.py | 70 +++++++++++- scripts/lib/devtool/standard.py | 230 ++++++++++++++++++++++++-------------- 2 files changed, 213 insertions(+), 87 deletions(-)
meta/lib/oeqa/selftest/devtool.py +69 -1
@@ -233,6 +233,8 @@ class DevtoolTests(oeSelfTest): self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory') testrecipe = 'minicom' recipefile = get_bb_var('FILE', testrecipe) + src_uri = get_bb_var('SRC_URI', testrecipe) + self.assertNotIn('git://', src_uri, 'This test expects the %s recipe to NOT be a git recipe' % testrecipe) result = runCmd('git status . --porcelain', cwd=os.path.dirname(recipefile)) self.assertEqual(result.output.strip(), "", '%s recipe is not clean' % testrecipe) # First, modify a recipe
@@ -266,11 +268,77 @@ class DevtoolTests(oeSelfTest): self.assertEqual(line[:3], '?? ', 'Unexpected status in line: %s' % line) elif line.endswith('0002-Add-a-new-file.patch'): self.assertEqual(line[:3], '?? ', 'Unexpected status in line: %s' % line) - elif re.search('minicom_[^_]*.bb$', line): + elif re.search('%s_[^_]*.bb$' % testrecipe, line): self.assertEqual(line[:3], ' M ', 'Unexpected status in line: %s' % line) else: raise AssertionError('Unexpected modified file in status: %s' % line) + def test_devtool_update_recipe_git(self): + # Check preconditions + workspacedir = os.path.join(self.builddir, 'workspace') + self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory') + testrecipe = 'mtd-utils' + recipefile = get_bb_var('FILE', testrecipe) + src_uri = get_bb_var('SRC_URI', testrecipe) + self.assertIn('git://', src_uri, 'This test expects the %s recipe to be a git recipe' % testrecipe) + result = runCmd('git status . --porcelain', cwd=os.path.dirname(recipefile)) + self.assertEqual(result.output.strip(), "", '%s recipe is not clean' % testrecipe) + # First, modify a recipe + tempdir = tempfile.mkdtemp(prefix='devtoolqa') + self.track_for_cleanup(tempdir) + self.track_for_cleanup(workspacedir) + self.add_command_to_tearDown('bitbake-layers remove-layer */workspace') + # (don't bother with cleaning the recipe on teardown, we won't be building it) + result = runCmd('devtool modify %s -x %s' % (testrecipe, tempdir)) + # Check git repo + self.assertTrue(os.path.isdir(os.path.join(tempdir, '.git')), 'git repository for external source tree not found') + result = runCmd('git status --porcelain', cwd=tempdir) + self.assertEqual(result.output.strip(), "", 'Created git repo is not clean') + result = runCmd('git symbolic-ref HEAD', cwd=tempdir) + self.assertEqual(result.output.strip(), "refs/heads/devtool", 'Wrong branch in git repo') + # Add a couple of commits + # FIXME: this only tests adding, need to also test update and remove + result = runCmd('echo "# Additional line" >> Makefile', cwd=tempdir) + result = runCmd('git commit -a -m "Change the Makefile"', cwd=tempdir) + result = runCmd('echo "A new file" > devtool-new-file', cwd=tempdir) + result = runCmd('git add devtool-new-file', cwd=tempdir) + result = runCmd('git commit -m "Add a new file"', cwd=tempdir) + self.add_command_to_tearDown('cd %s; git checkout %s %s' % (os.path.dirname(recipefile), testrecipe, os.path.basename(recipefile))) + result = runCmd('devtool update-recipe %s' % testrecipe) + result = runCmd('git status . --porcelain', cwd=os.path.dirname(recipefile)) + self.assertNotEqual(result.output.strip(), "", '%s recipe should be modified' % testrecipe) + status = result.output.splitlines() + self.assertEqual(len(status), 3, 'Less/more files modified than expected. Entire status:\n%s' % result.output) + for line in status: + if line.endswith('add-exclusion-to-mkfs-jffs2-git-2.patch'): + self.assertEqual(line[:3], ' D ', 'Unexpected status in line: %s' % line) + elif line.endswith('fix-armv7-neon-alignment.patch'): + self.assertEqual(line[:3], ' D ', 'Unexpected status in line: %s' % line) + elif re.search('%s_[^_]*.bb$' % testrecipe, line): + self.assertEqual(line[:3], ' M ', 'Unexpected status in line: %s' % line) + else: + raise AssertionError('Unexpected modified file in status: %s' % line) + result = runCmd('git diff %s' % os.path.basename(recipefile), cwd=os.path.dirname(recipefile)) + addlines = ['SRCREV = ".*"', 'SRC_URI = "git://git.infradead.org/mtd-utils.git"'] + removelines = ['SRCREV = ".*"', 'SRC_URI = "git://git.infradead.org/mtd-utils.git \\\\', 'file://add-exclusion-to-mkfs-jffs2-git-2.patch \\\\', 'file://fix-armv7-neon-alignment.patch \\\\', '"'] + for line in result.output.splitlines(): + if line.startswith('+++') or line.startswith('---'): + continue + elif line.startswith('+'): + matched = False + for item in addlines: + if re.match(item, line[1:].strip()): + matched = True + break + self.assertTrue(matched, 'Unexpected diff add line: %s' % line) + elif line.startswith('-'): + matched = False + for item in removelines: + if re.match(item, line[1:].strip()): + matched = True + break + self.assertTrue(matched, 'Unexpected diff remove line: %s' % line) + def test_devtool_extract(self): # Check preconditions workspacedir = os.path.join(self.builddir, 'workspace')
scripts/lib/devtool/standard.py +144 -86
257 lines not on this page · raw message
Signature · 1 line
1.9.3
PATCH 12/15 devtool: prevent extract/modify use with image recipes
In reply to Paul EggletonThere wouldn't be any point to using these with an image recipe, so disallow it.
--- scripts/lib/devtool/standard.py | 8 ++++++++ 1 file changed, 8 insertions(+)
scripts/lib/devtool/standard.py +8 -0
@@ -141,6 +141,10 @@ def _extract_source(srctree, keep_temp, devbranch, d): logger.error("The %s recipe uses a shared workdir which this tool does not currently support" % pn) return None + if bb.data.inherits_class('image', d): + logger.error("The %s recipe is an image, and therefore is not supported by this tool" % pn) + return None + if bb.data.inherits_class('externalsrc', d) and d.getVar('EXTERNALSRC', True): logger.error("externalsrc is currently enabled for the %s recipe. This prevents the normal do_patch task from working. You will need to disable this first." % pn) return None
@@ -290,6 +294,10 @@ def modify(args, config, basepath, workspace): return -1 rd = oe.recipeutils.parse_recipe(recipefile, tinfoil.config_data) + if bb.data.inherits_class('image', rd): + logger.error("The %s recipe is an image, and therefore is not supported by this tool" % args.recipename) + return None + initial_rev = None commits = [] srctree = os.path.abspath(args.srctree)
Signature · 1 line
1.9.3
PATCH 13/15 devtool: modify: handle recipes that use a shared workdir
In reply to Paul EggletonIf S is outside of WORKDIR then we shouldn't try to get the relative path in order to work out where the source subdirectory is as we do by default.
--- scripts/lib/devtool/standard.py | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
scripts/lib/devtool/standard.py +6 -4
@@ -313,12 +313,14 @@ def modify(args, config, basepath, workspace): (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=args.srctree) initial_rev = stdout.rstrip() - # Handle if S is set to a subdirectory of the source + # Check that recipe isn't using a shared workdir s = rd.getVar('S', True) workdir = rd.getVar('WORKDIR', True) - if s != workdir and os.path.dirname(s) != workdir: - srcsubdir = os.sep.join(os.path.relpath(s, workdir).split(os.sep)[1:]) - srctree = os.path.join(srctree, srcsubdir) + if s.startswith(workdir): + # Handle if S is set to a subdirectory of the source + if s != workdir and os.path.dirname(s) != workdir: + srcsubdir = os.sep.join(os.path.relpath(s, workdir).split(os.sep)[1:]) + srctree = os.path.join(srctree, srcsubdir) appendpath = os.path.join(config.workspace_path, 'appends') if not os.path.exists(appendpath):
Signature · 1 line
1.9.3
PATCH 14/15 devtool: support source extraction for recipes that use a shared workdir
In reply to Paul EggletonEnable source extraction used by devtool extract / devtool modify -x for recipes that use a shared workdir (e.g. the kernel and gcc).
--- scripts/lib/devtool/standard.py | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-)
scripts/lib/devtool/standard.py +14 -5
@@ -137,10 +137,6 @@ def _extract_source(srctree, keep_temp, devbranch, d): logger.error("The perf recipe does not actually check out source and thus cannot be supported by this tool") return None - if 'work-shared' in d.getVar('S', True): - logger.error("The %s recipe uses a shared workdir which this tool does not currently support" % pn) - return None - if bb.data.inherits_class('image', d): logger.error("The %s recipe is an image, and therefore is not supported by this tool" % pn) return None
@@ -169,6 +165,16 @@ def _extract_source(srctree, keep_temp, devbranch, d): workdir = os.path.join(tempdir, 'workdir') crd.setVar('WORKDIR', workdir) crd.setVar('T', os.path.join(tempdir, 'temp')) + if not crd.getVar('S', True).startswith(workdir): + # Usually a shared workdir recipe (kernel, gcc) + # Try to set a reasonable default + if bb.data.inherits_class('kernel', d): + crd.setVar('S', '${WORKDIR}/source') + else: + crd.setVar('S', '${WORKDIR}/${BP}') + if bb.data.inherits_class('kernel', d): + # We don't want to move the source to STAGING_KERNEL_DIR here + crd.setVar('STAGING_KERNEL_DIR', '${S}') # FIXME: This is very awkward. Unfortunately it's not currently easy to properly # execute tasks outside of bitbake itself, until then this has to suffice if we
@@ -204,7 +210,10 @@ def _extract_source(srctree, keep_temp, devbranch, d): else: os.rmdir(patchdir) - if not bb.data.inherits_class('kernel-yocto', d): + if bb.data.inherits_class('kernel-yocto', d): + (stdout, _) = bb.process.run('git --git-dir="%s" rev-parse HEAD' % crd.expand('${WORKDIR}/git'), cwd=srcsubdir) + initial_rev = stdout.rstrip() + else: if not os.listdir(srcsubdir): logger.error("no source unpacked to S, perhaps the %s recipe doesn't use any source?" % pn) return None
Signature · 1 line
1.9.3
PATCH 15/15 devtool: fix broken clones of git recipes
In reply to Paul EggletonBecause we move the workdir when extracting source, then move the source and delete the temporary workdir, you lose the indirection symlink pointed to by the alternates file (which is created when the fetcher clones it from DL_DIR with -s) and the resulting repository is broken. In any case, for a source repo that the user may put their own changes into, we can't really rely on a clone made with -s in case the original goes away - because of cleanall, DL_DIR disappearing, etc. So repack the repository so that it is a complete, non-shared clone after unpacking.
(While I'm at it, add a test for devtool modify with a git recipe which verifies that this works.)
--- meta/lib/oeqa/selftest/devtool.py | 33 +++++++++++++++++++++++++++++++++ scripts/lib/devtool/standard.py | 7 +++++++ 2 files changed, 40 insertions(+)
meta/lib/oeqa/selftest/devtool.py +33 -0
@@ -227,6 +227,39 @@ class DevtoolTests(oeSelfTest): matches = glob.glob(stampprefix + '*') self.assertFalse(matches, 'Stamp files exist for recipe mdadm that should have been cleaned') + def test_devtool_modify_git(self): + # Check preconditions + workspacedir = os.path.join(self.builddir, 'workspace') + self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory') + testrecipe = 'mkelfimage' + src_uri = get_bb_var('SRC_URI', testrecipe) + self.assertIn('git://', src_uri, 'This test expects the %s recipe to be a git recipe' % testrecipe) + # Clean up anything in the workdir/sysroot/sstate cache + bitbake('%s -c cleansstate' % testrecipe) + # Try modifying a recipe + tempdir = tempfile.mkdtemp(prefix='devtoolqa') + self.track_for_cleanup(tempdir) + self.track_for_cleanup(workspacedir) + self.add_command_to_tearDown('bitbake-layers remove-layer */workspace') + self.add_command_to_tearDown('bitbake -c clean %s' % testrecipe) + result = runCmd('devtool modify %s -x %s' % (testrecipe, tempdir)) + self.assertTrue(os.path.exists(os.path.join(tempdir, 'Makefile')), 'Extracted source could not be found') + self.assertTrue(os.path.isdir(os.path.join(tempdir, '.git')), 'git repository for external source tree not found') + self.assertTrue(os.path.exists(os.path.join(workspacedir, 'conf', 'layer.conf')), 'Workspace directory not created') + matches = glob.glob(os.path.join(workspacedir, 'appends', 'mkelfimage_*.bbappend')) + self.assertTrue(matches, 'bbappend not created') + # Test devtool status + result = runCmd('devtool status') + self.assertIn(testrecipe, result.output) + self.assertIn(tempdir, result.output) + # Check git repo + result = runCmd('git status --porcelain', cwd=tempdir) + self.assertEqual(result.output.strip(), "", 'Created git repo is not clean') + result = runCmd('git symbolic-ref HEAD', cwd=tempdir) + self.assertEqual(result.output.strip(), "refs/heads/devtool", 'Wrong branch in git repo') + # Try building + bitbake(testrecipe) + def test_devtool_update_recipe(self): # Check preconditions workspacedir = os.path.join(self.builddir, 'workspace')
scripts/lib/devtool/standard.py +7 -0
@@ -202,6 +202,13 @@ def _extract_source(srctree, keep_temp, devbranch, d): # Handle if S is set to a subdirectory of the source srcsubdir = os.path.join(workdir, os.path.relpath(srcsubdir, workdir).split(os.sep)[0]) + if os.path.exists(os.path.join(srcsubdir, '.git')): + alternatesfile = os.path.join(srcsubdir, '.git', 'objects', 'info', 'alternates') + if os.path.exists(alternatesfile): + # This will have been cloned with -s, so we need to convert it to a full clone + bb.process.run('git repack -a', cwd=srcsubdir) + os.remove(alternatesfile) + patchdir = os.path.join(srcsubdir, 'patches') haspatches = False if os.path.exists(patchdir):
Signature · 1 line
1.9.3
Quoted 3 lines · Paul Eggleton
I've been attempting to enable usage of devtool with the kernel (which effectively means using the externalsrc class with the kernel). This doesn't really work yet due to lack of handling the kernel configuration,
I've always insisted/argued and designed in that externalsrc shouldn't touch the kernel configuration at all.
You are building a tree that is completely managed outside of the build system, and getting the kernel configuration in place is part of that management.
So I'm not convinced it is a problem that needs to be solved.
Cheers,
Bruce
Quoted 51 lines
so you still can't use externalsrc properly with the kernel, but this at least gets us further - you can now extract the source successfully and run a build. I also discovered and fixed a number of related issues along the way, and made a few behavioural improvements in both devtool and externalsrc.
The following changes since commit
ad016fa7492ba23c751cf33dd1befde7782facfa:lzo: Upgrade to 2.09 and add ptest (2015-02-19 07:50:44 +0000)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contribpaule/devtool-fixes
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool-fixesPaul Eggleton (15):
recipetool: fix regression caused by previous commitlib/oe/patch: fix PATCHTOOL = "git" with source in a subdirectory classes/externalsrc: fix source being wiped out on clean with kernel classes/externalsrc: add workaround for recipes that use SRCPV in PV classes/externalsrc: show a warning on compiling classes/kernel-yocto: extend SRCTREECOVEREDTASKS
devtool: add/modify: add option to build in same directory
devtool: reset: run bitbake -c clean on recipe
devtool: build: rundo_populate_sysrootinstead ofdo_installlib/oe/recipeutils: minor improvements topatch_recipe()
devtool: update-recipe: add handling for git recipes
devtool: prevent extract/modify use with image recipes
devtool: modify: handle recipes that use a shared workdir
devtool: support source extraction for recipes that use a shared
workdir
devtool: fix broken clones of git recipesmeta/classes/externalsrc.bbclass | 24 ++++ meta/classes/kernel-yocto.bbclass | 2 +- meta/lib/oe/patch.py | 7 +- meta/lib/oe/recipeutils.py | 4 +- meta/lib/oeqa/selftest/devtool.py | 147 ++++++++++++++++++- scripts/lib/devtool/standard.py | 292 +++++++++++++++++++++++++------------- scripts/lib/recipetool/create.py | 8 +- 7 files changed, 377 insertions(+), 107 deletions(-)--
1.9.3--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
Quoted 6 lines · On Thursday 19 February 2015 11:47:01 Bruce Ashfield wrote:
Quoted 3 lines · Paul Eggleton
I've been attempting to enable usage of devtool with the kernel (which effectively means using the externalsrc class with the kernel). This doesn't really work yet due to lack of handling the kernel configuration,
Quoted 8 lines
I've always insisted/argued and designed in that externalsrc shouldn't touch the kernel configuration at all.
You are building a tree that is completely managed outside of the build system, and getting the kernel configuration in place is part of that management.
So I'm not convinced it is a problem that needs to be solved.
So, externalsrc still doesn't touch the configuration, and it is not my plan to make it do so. However, users do need some way of setting up that configuration when checking out those sources for the first time, and if that's to be of any use to someone who just wants to make some minor changes, it should be the exact same configuration you get when you build the recipe as normal. That is what I believe "devtool modify" (at least with -x) should be doing, given that it is one means you can use to set up a development environment if you do not have one already.
We can certainly talk about the mechanism and work out the best way to do it, such that it fits in with how linux-yocto works and meets user expectations. My intent with this series was to get us to the point where we can at least experiment and have that discussion.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 25 lines · Paul Eggleton
Quoted 6 lines · On Thursday 19 February 2015 11:47:01 Bruce Ashfield wrote:
Quoted 3 lines · Paul Eggleton
I've been attempting to enable usage of devtool with the kernel (which effectively means using the externalsrc class with the kernel). This doesn't really work yet due to lack of handling the kernel configuration,
Quoted 8 lines
I've always insisted/argued and designed in that externalsrc shouldn't touch the kernel configuration at all.
You are building a tree that is completely managed outside of the build system, and getting the kernel configuration in place is part of that management.
So I'm not convinced it is a problem that needs to be solved.
So, externalsrc still doesn't touch the configuration, and it is not my plan to make it do so. However, users do need some way of setting up that configuration when checking out those sources for the first time, and if that's to be of any use to someone who just wants to make some minor changes, it should be the exact same configuration you get when you build the recipe as normal. That is what I believe "devtool modify" (at least with -x) should be doing, given that it is one means you can use to set up a development environment if you do not have one already.
Quoted 4 lines
We can certainly talk about the mechanism and work out the best way to do it, such that it fits in with how linux-yocto works and meets user expectations. My intent with this series was to get us to the point where we can at least experiment and have that discussion.
Having the option of triggering the standard configuration process as per a usual linux-yocto build but on an external direction, standalone may be the best option?
Cheers,
Richard
Quoted 26 lines · Richard Purdie
Quoted 25 lines · Paul Eggleton
Quoted 6 lines · On Thursday 19 February 2015 11:47:01 Bruce Ashfield wrote:
Quoted 3 lines · Paul Eggleton
I've been attempting to enable usage of devtool with the kernel (which effectively means using the externalsrc class with the kernel). This doesn't really work yet due to lack of handling the kernel configuration,
Quoted 8 lines
I've always insisted/argued and designed in that externalsrc shouldn't touch the kernel configuration at all.
You are building a tree that is completely managed outside of the build system, and getting the kernel configuration in place is part of that management.
So I'm not convinced it is a problem that needs to be solved.
So, externalsrc still doesn't touch the configuration, and it is not my plan to make it do so. However, users do need some way of setting up that configuration when checking out those sources for the first time, and if that's to be of any use to someone who just wants to make some minor changes, it should be the exact same configuration you get when you build the recipe as normal. That is what I believe "devtool modify" (at least with -x) should be doing, given that it is one means you can use to set up a development environment if you do not have one already.
Quoted 8 lines
Quoted 4 lines
We can certainly talk about the mechanism and work out the best way to do it, such that it fits in with how linux-yocto works and meets user expectations. My intent with this series was to get us to the point where we can at least experiment and have that discussion.
Having the option of triggering the standard configuration process as per a usual linux-yocto build but on an external direction, standalone may be the best option?
I did try that before, and the tools to run on random trees (I use it for building
the linux-yocto kernel source out of tree), but they need the fragments and meta-data to do that configuration.
and when a externalsrc tree is unpacked and checked out, that data won't by lying around (that being said, I haven't tried the series to see what exactly is happening).
I always considered externalsrc to be: "here's my tree, I've set it up and it is ready to build .. go do it!", so I need to think about the best way to get that next step done (for any kernel, not just linux yocto).
Bruce
Quoted 5 lines
Cheers,
Richard
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
Quoted 14 lines · Paul Eggleton
Quoted 13 lines · On Thursday 19 February 2015 11:47:01 Bruce Ashfield wrote:
Quoted 3 lines · Paul Eggleton
I've been attempting to enable usage of devtool with the kernel (which effectively means using the externalsrc class with the kernel). This doesn't really work yet due to lack of handling the kernel configuration,
I've always insisted/argued and designed in that externalsrc shouldn't touch the kernel configuration at all.
You are building a tree that is completely managed outside of the build system, and getting the kernel configuration in place is part of that management.
Quoted 5 lines
Quoted 1 line
So I'm not convinced it is a problem that needs to be solved.
So, externalsrc still doesn't touch the configuration, and it is not my plan to make it do so. However, users do need some way of setting up that configuration when checking out those sources for the first time, and if that's to be of any
Right. Something to think on.
Quoted 10 lines
use to someone who just wants to make some minor changes, it should be the exact same configuration you get when you build the recipe as normal. That is what I believe "devtool modify" (at least with -x) should be doing, given that it is one means you can use to set up a development environment if you do not have one already.
We can certainly talk about the mechanism and work out the best way to do it, such that it fits in with how linux-yocto works and meets user expectations. My intent with this series was to get us to the point where we can at least experiment and have that discussion.
Agreed.
Bruce
Quoted 8 lines
Cheers,
Paul--
Paul Eggleton
Intel Open Source Technology Centre
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"