devtool fixes
cover devtool fixes
This incorporates several of the fixes from my previous series and adds a few others - I've excluded the multi-source changes from the previous series as they may not be ready yet and it's too late in the release cycle now. These have all passed through oe-selftest locally.
The following changes since commit 6e5dbb53a7cbd05baf07de62b1abc88dccfd4cb9:
nativesdk-qemu-helper: add a missing nativesdk-python3-logging dependency (2018-02-22 16:02:17 +0000)
are available in the Git repository at:
git://git.openembedded.org/openembedded-core-contrib paule/devtool33-oe
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=paule/devtool33-oe
Paul Eggleton (8):
lib/oe/path: implement is_path_under()
devtool: fix poor handling of upgraded BBCLASSEXTENDed recipes
devtool: reset: delete bbappend file if _check_preserve() doesn't
devtool: finish: fix erroneously creating bbappend for relative paths
devtool: deploy-target: don't specify ssh/scp port unless user does lib/oe/recipeutils: add .txz extension to archive list
devtool: search: also look in recipe cache
devtool: search: tweak help text
meta/lib/oe/path.py | 24 ++++++++++ meta/lib/oe/recipeutils.py | 2 +- scripts/devtool | 5 +- scripts/lib/devtool/deploy.py | 12 ++--- scripts/lib/devtool/search.py | 100 ++++++++++++++++++++++++++-------------- scripts/lib/devtool/standard.py | 17 +++++-- 6 files changed, 110 insertions(+), 50 deletions(-)
Signature · 1 line
2.14.3
PATCH 1/8 lib/oe/path: implement is_path_under()
In a few places we have checks to see path A is under path B, by adding / to the end of the path B and then seeing if path A starts with the suffixed path B. Unfortunately there are two potential flaws: (1) path A needs to be suffixed with / as well or the directory itself won't match (semantics perhaps, but in a lot of scenarios returning True is correct); (2) you need to run os.path.abspath() on both paths first or you will wrongly return False for some relative paths where you should return True. Let's solve this once and for all by writing a function that takes care of these and put it in oe.path.
--- meta/lib/oe/path.py | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
meta/lib/oe/path.py +24 -0
@@ -237,3 +237,27 @@ def realpath(file, root, use_physdir = True, loop_cnt = 100, assume_dir = False) raise return file + +def is_path_under(path1, possible_parent): + """ + Return True if a path is underneath another, False otherwise. + Multiple paths to test can be specified (as a list or tuple) in + which case all specified test paths must be under the parent to + return True. + """ + def abs_path_trailing(pth): + pth_abs = os.path.abspath(pth) + if not pth_abs.endswith(os.sep): + pth_abs += os.sep + return pth_abs + + possible_parent_abs = abs_path_trailing(possible_parent) + if isinstance(path1, str): + testpaths = [path1] + else: + testpaths = path1 + for path in testpaths: + path_abs = abs_path_trailing(path) + if not path_abs.startswith(possible_parent_abs): + return False + return True
Signature · 1 line
2.14.3
Re: [PATCH 1/8] lib/oe/path: implement is_path_under()
Quoted 6 lines
From: [email protected] [mailto:[email protected]] On Behalf Of Paul Eggleton
Sent: den 23 februari 2018 04:02
To: [email protected]
<snip>
Quoted 10 lines
Signed-off-by Paul Eggleton <[email protected]>--- meta/lib/oe/path.py | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)meta/lib/oe/path.py index renamed from meta/lib/oe/path.py +0
-0
1ea03d5d568..346b48f3641100644
meta/lib/oe/path.py def realpath(file, root, use_physdir = True, loop_cnt · line 241
Quoted 6 lines
meta/lib/oe/path.py +2
-0@@ -237,3 +237,27 @@ def realpath(file, root, use_physdir = True, loop_cnt = 100, assume_dir = False) raise return file + +def is_path_under(path1, possible_parent):
Changing the signature to is_path_under(possible_parent, *paths) will make the list management more natural.
This function will return True if path1 is the empty list. Is that intentional?
/Ola Nilsson
Quoted 30 lines
meta/lib/oe/path.py +22
-1+ """ + Return True if a path is underneath another, False otherwise. + Multiple paths to test can be specified (as a list or tuple) in + which case all specified test paths must be under the parent to + return True. + """ + def abs_path_trailing(pth): + pth_abs = os.path.abspath(pth) + if not pth_abs.endswith(os.sep): + pth_abs += os.sep + return pth_abs + + possible_parent_abs = abs_path_trailing(possible_parent) + if isinstance(path1, str): + testpaths = [path1] + else: + testpaths = path1 + for path in testpaths: + path_abs = abs_path_trailing(path) + if not path_abs.startswith(possible_parent_abs): + return False + return True --2.14.3
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [PATCH 1/8] lib/oe/path: implement is_path_under()
Hi Ola,
Quoted 4 lines · NZDT Ola x Nilsson
Quoted 1 line
+def
is_path_under(path1, possible_parent):Changing the signature to
is_path_under(possible_parent, *paths)will make the list management more natural.
Right, it does simplify things a little. In fact given the parameter order is changing I'll rename it to is_path_parent() at the same time.
Quoted 2 lines
This function will return True if path1 is the empty list. Is that intentional?
I don't believe so. I'll change the function to return False in that case, well spotted.
Thanks,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
PATCH 2/8 devtool: fix poor handling of upgraded BBCLASSEXTENDed recipes
In reply to Paul EggletonFix two aspects of handling BBCLASSEXTENDed targets (e.g. openssl-native) that have been run through "devtool upgrade":
* Fix recipe name not showing up in "devtool status"
* Fix "devtool reset" not deleting empty directories under the recipe directory within the workspace, which may lead to problems if you subsequently run "devtool upgrade" on the same target again
--- scripts/devtool | 5 +++-- scripts/lib/devtool/standard.py | 6 +++++- 2 files changed, 8 insertions(+), 3 deletions(-)
scripts/devtool +3 -2
@@ -117,12 +117,13 @@ def read_workspace(): for line in f: res = externalsrc_re.match(line.rstrip()) if res: - pn = res.group(2) or os.path.splitext(os.path.basename(fn))[0].split('_')[0] + recipepn = os.path.splitext(os.path.basename(fn))[0].split('_')[0] + pn = res.group(2) or recipepn # Find the recipe file within the workspace, if any bbfile = os.path.basename(fn).replace('.bbappend', '.bb').replace('%', '*') recipefile = glob.glob(os.path.join(config.workspace_path, 'recipes', - pn, + recipepn, bbfile)) if recipefile: recipefile = recipefile[0]
scripts/lib/devtool/standard.py +5 -1
@@ -1744,6 +1744,7 @@ def status(args, config, basepath, workspace): def _reset(recipes, no_clean, config, basepath, workspace): """Reset one or more recipes""" + import oe.path def clean_preferred_provider(pn, layerconf_path): """Remove PREFERRED_PROVIDER from layer.conf'"""
@@ -1802,7 +1803,10 @@ def _reset(recipes, no_clean, config, basepath, workspace): preservedir(os.path.join(root, dn)) os.rmdir(origdir) - preservedir(os.path.join(config.workspace_path, 'recipes', pn)) + recipefile = workspace[pn]['recipefile'] + if recipefile and oe.path.is_path_under(recipefile, config.workspace_path): + # This should always be true if recipefile is set, but just in case + preservedir(os.path.dirname(recipefile)) # We don't automatically create this dir next to appends, but the user can preservedir(os.path.join(config.workspace_path, 'appends', pn))
Signature · 1 line
2.14.3
PATCH 3/8 devtool: reset: delete bbappend file if _check_preserve() doesn't
In reply to Paul EggletonIf the .devtool_md5 file doesn't contain a reference to the bbappend file (e.g. because devtool was interrupted before it could write that out) then _check_preserve() won't delete it, so we need to delete it separately because otherwise the recipe won't actually be reset.
--- scripts/lib/devtool/standard.py | 7 +++++++ 1 file changed, 7 insertions(+)
scripts/lib/devtool/standard.py +7 -0
@@ -1791,6 +1791,13 @@ def _reset(recipes, no_clean, config, basepath, workspace): for pn in recipes: _check_preserve(config, pn) + appendfile = workspace[pn]['bbappend'] + if os.path.exists(appendfile): + # This shouldn't happen, but is possible if devtool errored out prior to + # writing the md5 file. We need to delete this here or the recipe won't + # actually be reset + os.remove(appendfile) + preservepath = os.path.join(config.workspace_path, 'attic', pn, pn) def preservedir(origdir): if os.path.exists(origdir):
Signature · 1 line
2.14.3
PATCH 4/8 devtool: finish: fix erroneously creating bbappend for relative paths
In reply to Paul EggletonAfter OE-Core rev 5e3fe00a0233d563781849a44f53885b4e924a9c we call os.path.abspath() on the original layer path, but we later compare that to the destination layer path. If that layer path isn't absolute but is effectively the same path, it should be writing to the original recipe but because we weren't making it absolute we were writing a bbappend instead. Call os.path.abspath() on the destination path as well to avoid that.
--- scripts/lib/devtool/standard.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
scripts/lib/devtool/standard.py +1 -3
@@ -1861,9 +1861,7 @@ def _get_layer(layername, d): layerdir = layers.get('meta', None) else: layerdir = layers.get(layername, None) - if layerdir: - layerdir = os.path.abspath(layerdir) - return layerdir or layername + return os.path.abspath(layerdir or layername) def finish(args, config, basepath, workspace): """Entry point for the devtool 'finish' subcommand"""
Signature · 1 line
2.14.3
PATCH 5/8 devtool: deploy-target: don't specify ssh/scp port unless user does
In reply to Paul EggletonIf the user doesn't specify a port then we should avoid specifying one on the ssh/scp command line in case the user has configured one for the host they are connecting to, which was being overridden unnecessarily.
Fixes [YOCTO #12381].
--- scripts/lib/devtool/deploy.py | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-)
scripts/lib/devtool/deploy.py +4 -8
@@ -213,9 +213,7 @@ def deploy(args, config, basepath, workspace): scp_port = '' ssh_port = '' - if not args.port: - raise DevtoolError("If you specify -P/--port then you must provide the port to be used to connect to the target") - else: + if args.port: scp_port = "-P %s" % args.port ssh_port = "-p %s" % args.port
@@ -280,9 +278,7 @@ def undeploy(args, config, basepath, workspace): scp_port = '' ssh_port = '' - if not args.port: - raise DevtoolError("If you specify -P/--port then you must provide the port to be used to connect to the target") - else: + if args.port: scp_port = "-P %s" % args.port ssh_port = "-p %s" % args.port
@@ -328,7 +324,7 @@ def register_commands(subparsers, context): parser_deploy.add_argument('-n', '--dry-run', help='List files to be deployed only', action='store_true') parser_deploy.add_argument('-p', '--no-preserve', help='Do not preserve existing files', action='store_true') parser_deploy.add_argument('--no-check-space', help='Do not check for available space before deploying', action='store_true') - parser_deploy.add_argument('-P', '--port', default='22', help='Port to use for connection to the target') + parser_deploy.add_argument('-P', '--port', help='Specify port to use for connection to the target') strip_opts = parser_deploy.add_mutually_exclusive_group(required=False) strip_opts.add_argument('-S', '--strip',
@@ -350,5 +346,5 @@ def register_commands(subparsers, context): parser_undeploy.add_argument('-s', '--show-status', help='Show progress/status output', action='store_true') parser_undeploy.add_argument('-a', '--all', help='Undeploy all recipes deployed on the target', action='store_true') parser_undeploy.add_argument('-n', '--dry-run', help='List files to be undeployed only', action='store_true') - parser_undeploy.add_argument('-P', '--port', default='22', help='Port to use for connection to the target') + parser_undeploy.add_argument('-P', '--port', help='Specify port to use for connection to the target') parser_undeploy.set_defaults(func=undeploy)
Signature · 1 line
2.14.3
PATCH 6/8 lib/oe/recipeutils: add .txz extension to archive list
In reply to Paul EggletonPrompted by bitbake commit 2ba8a6b25ccc12e7b543e8450121e5311c7a701d, add .txz to the list of archives used within get_recipe_local_files() here as well.
--- meta/lib/oe/recipeutils.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/recipeutils.py +1 -1
@@ -408,7 +408,7 @@ def get_recipe_local_files(d, patches=False, archives=False): # fetcher) though note that this only encompasses actual container formats # i.e. that can contain multiple files as opposed to those that only # contain a compressed stream (i.e. .tar.gz as opposed to just .gz) - archive_exts = ['.tar', '.tgz', '.tar.gz', '.tar.Z', '.tbz', '.tbz2', '.tar.bz2', '.tar.xz', '.tar.lz', '.zip', '.jar', '.rpm', '.srpm', '.deb', '.ipk', '.tar.7z', '.7z'] + archive_exts = ['.tar', '.tgz', '.tar.gz', '.tar.Z', '.tbz', '.tbz2', '.tar.bz2', '.txz', '.tar.xz', '.tar.lz', '.zip', '.jar', '.rpm', '.srpm', '.deb', '.ipk', '.tar.7z', '.7z'] ret = {} for uri in uris: if fetch.ud[uri].type == 'file':
Signature · 1 line
2.14.3
PATCH 7/8 devtool: search: also look in recipe cache
In reply to Paul EggletonIf pkgdata isn't present or is incomplete, then you get either a traceback or you don't see the results you were hoping for. The recipe cache that bitbake collects during startup contains some useful information for each recipe that we could search through as well, and we can access it easily using tinfoil's all_recipes() API function, so add some code that does that. (We still show a warning if pkgdata isn't present, as there are certain dynamic packages that are generated at packaging time that won't show up in the cache).
One side-effect of this is that we will start showing non-target recipes - that's actually a good thing, since seeing those is useful, however we exclude nativesdk recipes when in the eSDK to avoid confusion since nativesdk isn't directly applicable there.
Fixes [YOCTO #12356].
--- scripts/lib/devtool/search.py | 96 ++++++++++++++++++++++++++++--------------- 1 file changed, 63 insertions(+), 33 deletions(-)
scripts/lib/devtool/search.py +63 -33
@@ -36,44 +36,74 @@ def search(args, config, basepath, workspace): keyword_rc = re.compile(args.keyword) - for fn in os.listdir(pkgdata_dir): - pfn = os.path.join(pkgdata_dir, fn) - if not os.path.isfile(pfn): + def print_match(pn): + rd = parse_recipe(config, tinfoil, pn, True) + if not rd: + return + summary = rd.getVar('SUMMARY') + if summary == rd.expand(defsummary): + summary = '' + print("%s %s" % (pn.ljust(20), summary)) + + + matches = [] + if os.path.exists(pkgdata_dir): + for fn in os.listdir(pkgdata_dir): + pfn = os.path.join(pkgdata_dir, fn) + if not os.path.isfile(pfn): + continue + + packages = [] + match = False + if keyword_rc.search(fn): + match = True + + if not match: + with open(pfn, 'r') as f: + for line in f: + if line.startswith('PACKAGES:'): + packages = line.split(':', 1)[1].strip().split() + + for pkg in packages: + if keyword_rc.search(pkg): + match = True + break + if os.path.exists(os.path.join(pkgdata_dir, 'runtime', pkg + '.packaged')): + with open(os.path.join(pkgdata_dir, 'runtime', pkg), 'r') as f: + for line in f: + if ': ' in line: + splitline = line.split(':', 1) + key = splitline[0] + value = splitline[1].strip() + if key in ['PKG_%s' % pkg, 'DESCRIPTION', 'FILES_INFO'] or key.startswith('FILERPROVIDES_'): + if keyword_rc.search(value): + match = True + break + if match: + print_match(fn) + matches.append(fn) + else: + logger.warning('Package data is not available, results may be limited') + + for recipe in tinfoil.all_recipes(): + if args.fixed_setup and 'nativesdk' in recipe.inherits(): continue - packages = [] match = False - if keyword_rc.search(fn): + if keyword_rc.search(recipe.pn): match = True - - if not match: - with open(pfn, 'r') as f: - for line in f: - if line.startswith('PACKAGES:'): - packages = line.split(':', 1)[1].strip().split() - - for pkg in packages: - if keyword_rc.search(pkg): + else: + for prov in recipe.provides: + if keyword_rc.search(prov): match = True break - if os.path.exists(os.path.join(pkgdata_dir, 'runtime', pkg + '.packaged')): - with open(os.path.join(pkgdata_dir, 'runtime', pkg), 'r') as f: - for line in f: - if ': ' in line: - splitline = line.split(':', 1) - key = splitline[0] - value = splitline[1].strip() - if key in ['PKG_%s' % pkg, 'DESCRIPTION', 'FILES_INFO'] or key.startswith('FILERPROVIDES_'): - if keyword_rc.search(value): - match = True - break - - if match: - rd = parse_recipe(config, tinfoil, fn, True) - summary = rd.getVar('SUMMARY') - if summary == rd.expand(defsummary): - summary = '' - print("%s %s" % (fn.ljust(20), summary)) + if not match: + for rprov in recipe.rprovides: + if keyword_rc.search(rprov): + match = True + break + if match and not recipe.pn in matches: + print_match(recipe.pn) finally: tinfoil.shutdown()
@@ -85,4 +115,4 @@ def register_commands(subparsers, context): description='Searches for available target recipes. Matches on recipe name, package name, description and installed files, and prints the recipe name on match.', group='info') parser_search.add_argument('keyword', help='Keyword to search for (regular expression syntax allowed)') - parser_search.set_defaults(func=search, no_workspace=True) + parser_search.set_defaults(func=search, no_workspace=True, fixed_setup=context.fixed_setup)
Signature · 1 line
2.14.3
PATCH 8/8 devtool: search: tweak help text
In reply to Paul Eggleton* We now match on more than just target recipes, so don't specify that only target recipes are searched.
* We're printing the SUMMARY value in addition to the name, so mention that so it's clear where that text is coming from.
* Remind users that they should use quotes around the keyword to avoid shell expansion when using regular expressions.
--- scripts/lib/devtool/search.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/lib/devtool/search.py +2 -2
@@ -112,7 +112,7 @@ def search(args, config, basepath, workspace): def register_commands(subparsers, context): """Register devtool subcommands from this plugin""" parser_search = subparsers.add_parser('search', help='Search available recipes', - description='Searches for available target recipes. Matches on recipe name, package name, description and installed files, and prints the recipe name on match.', + description='Searches for available recipes. Matches on recipe name, package name, description and installed files, and prints the recipe name and summary on match.', group='info') - parser_search.add_argument('keyword', help='Keyword to search for (regular expression syntax allowed)') + parser_search.add_argument('keyword', help='Keyword to search for (regular expression syntax allowed, use quotes to avoid shell expansion)') parser_search.set_defaults(func=search, no_workspace=True, fixed_setup=context.fixed_setup)
Signature · 1 line
2.14.3