devtool fixes
v2 cover devtool fixes
This incorporates several of the fixes from my previous series and adds a few others - I've excluded the multi-source changes from the previous series as they may not be ready yet and it's too late in the release cycle now. These have all passed through oe-selftest locally.
Changes since v1:
* Rename is_path_under() -> is_path_parent(), and change it to accept paths to test as unlimited arguments to the function and return False if there are no paths specified, as suggested by Ola x Nilsson.
The following changes since commit b30153a15715a83c0f9a7d7d1883a15404992a19:
buildhistory-diff: honour report_all flag (2018-02-24 10:33:12 +0000)
are available in the Git repository at:
git://git.openembedded.org/openembedded-core-contrib paule/devtool33-oe
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=paule/devtool33-oe
Paul Eggleton (8):
lib/oe/path: implement is_path_parent()
devtool: fix poor handling of upgraded BBCLASSEXTENDed recipes
devtool: reset: delete bbappend file if _check_preserve() doesn't
devtool: finish: fix erroneously creating bbappend for relative paths
devtool: deploy-target: don't specify ssh/scp port unless user does lib/oe/recipeutils: add .txz extension to archive list
devtool: search: also look in recipe cache
devtool: search: tweak help text
meta/lib/oe/path.py | 22 +++++++++ meta/lib/oe/recipeutils.py | 2 +- scripts/devtool | 5 +- scripts/lib/devtool/deploy.py | 12 ++--- scripts/lib/devtool/search.py | 100 ++++++++++++++++++++++++++-------------- scripts/lib/devtool/standard.py | 17 +++++-- 6 files changed, 108 insertions(+), 50 deletions(-)
Signature · 1 line
2.14.3
PATCH v2 1/8 lib/oe/path: implement is_path_parent()
In a few places we have checks to see path B is the parent of path A, by adding / to the end of the path B and then seeing if path A starts with the suffixed path B. Unfortunately there are two potential flaws:
(1) path A needs to be suffixed with / as well or the directory itself won't match (semantics perhaps, but in a lot of scenarios returning True is correct); (2) you need to run os.path.abspath() on both paths first or you will wrongly return False for some relative paths where you should return True. Let's solve this once and for all by writing a function that takes care of these and put it in oe.path.
--- meta/lib/oe/path.py | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
meta/lib/oe/path.py +22 -0
@@ -237,3 +237,25 @@ def realpath(file, root, use_physdir = True, loop_cnt = 100, assume_dir = False) raise return file + +def is_path_parent(possible_parent, *paths): + """ + Return True if a path is the parent of another, False otherwise. + Multiple paths to test can be specified in which case all + specified test paths must be under the parent in order to + return True. + """ + def abs_path_trailing(pth): + pth_abs = os.path.abspath(pth) + if not pth_abs.endswith(os.sep): + pth_abs += os.sep + return pth_abs + + possible_parent_abs = abs_path_trailing(possible_parent) + if not paths: + return False + for path in paths: + path_abs = abs_path_trailing(path) + if not path_abs.startswith(possible_parent_abs): + return False + return True
Signature · 1 line
2.14.3
PATCH v2 2/8 devtool: fix poor handling of upgraded BBCLASSEXTENDed recipes
In reply to Paul EggletonFix two aspects of handling BBCLASSEXTENDed targets (e.g. openssl-native) that have been run through "devtool upgrade":
* Fix recipe name not showing up in "devtool status"
* Fix "devtool reset" not deleting empty directories under the recipe directory within the workspace, which may lead to problems if you subsequently run "devtool upgrade" on the same target again
--- scripts/devtool | 5 +++-- scripts/lib/devtool/standard.py | 6 +++++- 2 files changed, 8 insertions(+), 3 deletions(-)
scripts/devtool +3 -2
@@ -117,12 +117,13 @@ def read_workspace(): for line in f: res = externalsrc_re.match(line.rstrip()) if res: - pn = res.group(2) or os.path.splitext(os.path.basename(fn))[0].split('_')[0] + recipepn = os.path.splitext(os.path.basename(fn))[0].split('_')[0] + pn = res.group(2) or recipepn # Find the recipe file within the workspace, if any bbfile = os.path.basename(fn).replace('.bbappend', '.bb').replace('%', '*') recipefile = glob.glob(os.path.join(config.workspace_path, 'recipes', - pn, + recipepn, bbfile)) if recipefile: recipefile = recipefile[0]
scripts/lib/devtool/standard.py +5 -1
@@ -1744,6 +1744,7 @@ def status(args, config, basepath, workspace): def _reset(recipes, no_clean, config, basepath, workspace): """Reset one or more recipes""" + import oe.path def clean_preferred_provider(pn, layerconf_path): """Remove PREFERRED_PROVIDER from layer.conf'"""
@@ -1802,7 +1803,10 @@ def _reset(recipes, no_clean, config, basepath, workspace): preservedir(os.path.join(root, dn)) os.rmdir(origdir) - preservedir(os.path.join(config.workspace_path, 'recipes', pn)) + recipefile = workspace[pn]['recipefile'] + if recipefile and oe.path.is_path_parent(config.workspace_path, recipefile): + # This should always be true if recipefile is set, but just in case + preservedir(os.path.dirname(recipefile)) # We don't automatically create this dir next to appends, but the user can preservedir(os.path.join(config.workspace_path, 'appends', pn))
Signature · 1 line
2.14.3
PATCH v2 3/8 devtool: reset: delete bbappend file if _check_preserve() doesn't
In reply to Paul EggletonIf the .devtool_md5 file doesn't contain a reference to the bbappend file (e.g. because devtool was interrupted before it could write that out) then _check_preserve() won't delete it, so we need to delete it separately because otherwise the recipe won't actually be reset.
--- scripts/lib/devtool/standard.py | 7 +++++++ 1 file changed, 7 insertions(+)
scripts/lib/devtool/standard.py +7 -0
@@ -1791,6 +1791,13 @@ def _reset(recipes, no_clean, config, basepath, workspace): for pn in recipes: _check_preserve(config, pn) + appendfile = workspace[pn]['bbappend'] + if os.path.exists(appendfile): + # This shouldn't happen, but is possible if devtool errored out prior to + # writing the md5 file. We need to delete this here or the recipe won't + # actually be reset + os.remove(appendfile) + preservepath = os.path.join(config.workspace_path, 'attic', pn, pn) def preservedir(origdir): if os.path.exists(origdir):
Signature · 1 line
2.14.3
PATCH v2 4/8 devtool: finish: fix erroneously creating bbappend for relative paths
In reply to Paul EggletonAfter OE-Core rev 5e3fe00a0233d563781849a44f53885b4e924a9c we call os.path.abspath() on the original layer path, but we later compare that to the destination layer path. If that layer path isn't absolute but is effectively the same path, it should be writing to the original recipe but because we weren't making it absolute we were writing a bbappend instead. Call os.path.abspath() on the destination path as well to avoid that.
--- scripts/lib/devtool/standard.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
scripts/lib/devtool/standard.py +1 -3
@@ -1861,9 +1861,7 @@ def _get_layer(layername, d): layerdir = layers.get('meta', None) else: layerdir = layers.get(layername, None) - if layerdir: - layerdir = os.path.abspath(layerdir) - return layerdir or layername + return os.path.abspath(layerdir or layername) def finish(args, config, basepath, workspace): """Entry point for the devtool 'finish' subcommand"""
Signature · 1 line
2.14.3
PATCH v2 5/8 devtool: deploy-target: don't specify ssh/scp port unless user does
In reply to Paul EggletonIf the user doesn't specify a port then we should avoid specifying one on the ssh/scp command line in case the user has configured one for the host they are connecting to, which was being overridden unnecessarily.
Fixes [YOCTO #12381].
--- scripts/lib/devtool/deploy.py | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-)
scripts/lib/devtool/deploy.py +4 -8
@@ -213,9 +213,7 @@ def deploy(args, config, basepath, workspace): scp_port = '' ssh_port = '' - if not args.port: - raise DevtoolError("If you specify -P/--port then you must provide the port to be used to connect to the target") - else: + if args.port: scp_port = "-P %s" % args.port ssh_port = "-p %s" % args.port
@@ -280,9 +278,7 @@ def undeploy(args, config, basepath, workspace): scp_port = '' ssh_port = '' - if not args.port: - raise DevtoolError("If you specify -P/--port then you must provide the port to be used to connect to the target") - else: + if args.port: scp_port = "-P %s" % args.port ssh_port = "-p %s" % args.port
@@ -328,7 +324,7 @@ def register_commands(subparsers, context): parser_deploy.add_argument('-n', '--dry-run', help='List files to be deployed only', action='store_true') parser_deploy.add_argument('-p', '--no-preserve', help='Do not preserve existing files', action='store_true') parser_deploy.add_argument('--no-check-space', help='Do not check for available space before deploying', action='store_true') - parser_deploy.add_argument('-P', '--port', default='22', help='Port to use for connection to the target') + parser_deploy.add_argument('-P', '--port', help='Specify port to use for connection to the target') strip_opts = parser_deploy.add_mutually_exclusive_group(required=False) strip_opts.add_argument('-S', '--strip',
@@ -350,5 +346,5 @@ def register_commands(subparsers, context): parser_undeploy.add_argument('-s', '--show-status', help='Show progress/status output', action='store_true') parser_undeploy.add_argument('-a', '--all', help='Undeploy all recipes deployed on the target', action='store_true') parser_undeploy.add_argument('-n', '--dry-run', help='List files to be undeployed only', action='store_true') - parser_undeploy.add_argument('-P', '--port', default='22', help='Port to use for connection to the target') + parser_undeploy.add_argument('-P', '--port', help='Specify port to use for connection to the target') parser_undeploy.set_defaults(func=undeploy)
Signature · 1 line
2.14.3
PATCH v2 6/8 lib/oe/recipeutils: add .txz extension to archive list
In reply to Paul EggletonPrompted by bitbake commit 2ba8a6b25ccc12e7b543e8450121e5311c7a701d, add .txz to the list of archives used within get_recipe_local_files() here as well.
--- meta/lib/oe/recipeutils.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/recipeutils.py +1 -1
@@ -408,7 +408,7 @@ def get_recipe_local_files(d, patches=False, archives=False): # fetcher) though note that this only encompasses actual container formats # i.e. that can contain multiple files as opposed to those that only # contain a compressed stream (i.e. .tar.gz as opposed to just .gz) - archive_exts = ['.tar', '.tgz', '.tar.gz', '.tar.Z', '.tbz', '.tbz2', '.tar.bz2', '.tar.xz', '.tar.lz', '.zip', '.jar', '.rpm', '.srpm', '.deb', '.ipk', '.tar.7z', '.7z'] + archive_exts = ['.tar', '.tgz', '.tar.gz', '.tar.Z', '.tbz', '.tbz2', '.tar.bz2', '.txz', '.tar.xz', '.tar.lz', '.zip', '.jar', '.rpm', '.srpm', '.deb', '.ipk', '.tar.7z', '.7z'] ret = {} for uri in uris: if fetch.ud[uri].type == 'file':
Signature · 1 line
2.14.3
PATCH v2 7/8 devtool: search: also look in recipe cache
In reply to Paul EggletonIf pkgdata isn't present or is incomplete, then you get either a traceback or you don't see the results you were hoping for. The recipe cache that bitbake collects during startup contains some useful information for each recipe that we could search through as well, and we can access it easily using tinfoil's all_recipes() API function, so add some code that does that. (We still show a warning if pkgdata isn't present, as there are certain dynamic packages that are generated at packaging time that won't show up in the cache).
One side-effect of this is that we will start showing non-target recipes - that's actually a good thing, since seeing those is useful, however we exclude nativesdk recipes when in the eSDK to avoid confusion since nativesdk isn't directly applicable there.
Fixes [YOCTO #12356].
--- scripts/lib/devtool/search.py | 96 ++++++++++++++++++++++++++++--------------- 1 file changed, 63 insertions(+), 33 deletions(-)
scripts/lib/devtool/search.py +63 -33
@@ -36,44 +36,74 @@ def search(args, config, basepath, workspace): keyword_rc = re.compile(args.keyword) - for fn in os.listdir(pkgdata_dir): - pfn = os.path.join(pkgdata_dir, fn) - if not os.path.isfile(pfn): + def print_match(pn): + rd = parse_recipe(config, tinfoil, pn, True) + if not rd: + return + summary = rd.getVar('SUMMARY') + if summary == rd.expand(defsummary): + summary = '' + print("%s %s" % (pn.ljust(20), summary)) + + + matches = [] + if os.path.exists(pkgdata_dir): + for fn in os.listdir(pkgdata_dir): + pfn = os.path.join(pkgdata_dir, fn) + if not os.path.isfile(pfn): + continue + + packages = [] + match = False + if keyword_rc.search(fn): + match = True + + if not match: + with open(pfn, 'r') as f: + for line in f: + if line.startswith('PACKAGES:'): + packages = line.split(':', 1)[1].strip().split() + + for pkg in packages: + if keyword_rc.search(pkg): + match = True + break + if os.path.exists(os.path.join(pkgdata_dir, 'runtime', pkg + '.packaged')): + with open(os.path.join(pkgdata_dir, 'runtime', pkg), 'r') as f: + for line in f: + if ': ' in line: + splitline = line.split(':', 1) + key = splitline[0] + value = splitline[1].strip() + if key in ['PKG_%s' % pkg, 'DESCRIPTION', 'FILES_INFO'] or key.startswith('FILERPROVIDES_'): + if keyword_rc.search(value): + match = True + break + if match: + print_match(fn) + matches.append(fn) + else: + logger.warning('Package data is not available, results may be limited') + + for recipe in tinfoil.all_recipes(): + if args.fixed_setup and 'nativesdk' in recipe.inherits(): continue - packages = [] match = False - if keyword_rc.search(fn): + if keyword_rc.search(recipe.pn): match = True - - if not match: - with open(pfn, 'r') as f: - for line in f: - if line.startswith('PACKAGES:'): - packages = line.split(':', 1)[1].strip().split() - - for pkg in packages: - if keyword_rc.search(pkg): + else: + for prov in recipe.provides: + if keyword_rc.search(prov): match = True break - if os.path.exists(os.path.join(pkgdata_dir, 'runtime', pkg + '.packaged')): - with open(os.path.join(pkgdata_dir, 'runtime', pkg), 'r') as f: - for line in f: - if ': ' in line: - splitline = line.split(':', 1) - key = splitline[0] - value = splitline[1].strip() - if key in ['PKG_%s' % pkg, 'DESCRIPTION', 'FILES_INFO'] or key.startswith('FILERPROVIDES_'): - if keyword_rc.search(value): - match = True - break - - if match: - rd = parse_recipe(config, tinfoil, fn, True) - summary = rd.getVar('SUMMARY') - if summary == rd.expand(defsummary): - summary = '' - print("%s %s" % (fn.ljust(20), summary)) + if not match: + for rprov in recipe.rprovides: + if keyword_rc.search(rprov): + match = True + break + if match and not recipe.pn in matches: + print_match(recipe.pn) finally: tinfoil.shutdown()
@@ -85,4 +115,4 @@ def register_commands(subparsers, context): description='Searches for available target recipes. Matches on recipe name, package name, description and installed files, and prints the recipe name on match.', group='info') parser_search.add_argument('keyword', help='Keyword to search for (regular expression syntax allowed)') - parser_search.set_defaults(func=search, no_workspace=True) + parser_search.set_defaults(func=search, no_workspace=True, fixed_setup=context.fixed_setup)
Signature · 1 line
2.14.3
PATCH v2 8/8 devtool: search: tweak help text
In reply to Paul Eggleton* We now match on more than just target recipes, so don't specify that only target recipes are searched.
* We're printing the SUMMARY value in addition to the name, so mention that so it's clear where that text is coming from.
* Remind users that they should use quotes around the keyword to avoid shell expansion when using regular expressions.
--- scripts/lib/devtool/search.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/lib/devtool/search.py +2 -2
@@ -112,7 +112,7 @@ def search(args, config, basepath, workspace): def register_commands(subparsers, context): """Register devtool subcommands from this plugin""" parser_search = subparsers.add_parser('search', help='Search available recipes', - description='Searches for available target recipes. Matches on recipe name, package name, description and installed files, and prints the recipe name on match.', + description='Searches for available recipes. Matches on recipe name, package name, description and installed files, and prints the recipe name and summary on match.', group='info') - parser_search.add_argument('keyword', help='Keyword to search for (regular expression syntax allowed)') + parser_search.add_argument('keyword', help='Keyword to search for (regular expression syntax allowed, use quotes to avoid shell expansion)') parser_search.set_defaults(func=search, no_workspace=True, fixed_setup=context.fixed_setup)
Signature · 1 line
2.14.3