Skip to contentSkip to conversation

devtool fixes

PATCH Superseded Paul Eggleton 2 participants 11 messages Feb 23, 2018 – Feb 25, 2018

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover devtool fixes

This incorporates several of the fixes from my previous series and adds a few others - I've excluded the multi-source changes from the previous series as they may not be ready yet and it's too late in the release cycle now. These have all passed through oe-selftest locally.

The following changes since commit 6e5dbb53a7cbd05baf07de62b1abc88dccfd4cb9:

nativesdk-qemu-helper: add a missing nativesdk-python3-logging dependency (2018-02-22 16:02:17 +0000)

are available in the Git repository at:

git://git.openembedded.org/openembedded-core-contrib paule/devtool33-oe
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=paule/devtool33-oe

Paul Eggleton (8):
lib/oe/path: implement is_path_under()
devtool: fix poor handling of upgraded BBCLASSEXTENDed recipes
devtool: reset: delete bbappend file if _check_preserve() doesn't
devtool: finish: fix erroneously creating bbappend for relative paths
devtool: deploy-target: don't specify ssh/scp port unless user does lib/oe/recipeutils: add .txz extension to archive list
devtool: search: also look in recipe cache
devtool: search: tweak help text

 meta/lib/oe/path.py             |  24 ++++++++++
 meta/lib/oe/recipeutils.py      |   2 +-
 scripts/devtool                 |   5 +-
 scripts/lib/devtool/deploy.py   |  12 ++---
 scripts/lib/devtool/search.py   | 100 ++++++++++++++++++++++++++--------------
 scripts/lib/devtool/standard.py |  17 +++++--
 6 files changed, 110 insertions(+), 50 deletions(-)
Signature · 1 line
2.14.3
#1 lore raw

PATCH 1/8 lib/oe/path: implement is_path_under()

In a few places we have checks to see path A is under path B, by adding / to the end of the path B and then seeing if path A starts with the suffixed path B. Unfortunately there are two potential flaws: (1) path A needs to be suffixed with / as well or the directory itself won't match (semantics perhaps, but in a lot of scenarios returning True is correct); (2) you need to run os.path.abspath() on both paths first or you will wrongly return False for some relative paths where you should return True. Let's solve this once and for all by writing a function that takes care of these and put it in oe.path.

---
 meta/lib/oe/path.py | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
meta/lib/oe/path.py +24 -0
index 1ea03d5d568..346b48f3641 100644
@@ -237,3 +237,27 @@ def realpath(file, root, use_physdir = True, loop_cnt = 100, assume_dir = False)
         raise
 
     return file
+
+def is_path_under(path1, possible_parent):
+    """
+    Return True if a path is underneath another, False otherwise.
+    Multiple paths to test can be specified (as a list or tuple) in
+    which case all specified test paths must be under the parent to
+    return True.
+    """
+    def abs_path_trailing(pth):
+        pth_abs = os.path.abspath(pth)
+        if not pth_abs.endswith(os.sep):
+            pth_abs += os.sep
+        return pth_abs
+
+    possible_parent_abs = abs_path_trailing(possible_parent)
+    if isinstance(path1, str):
+        testpaths = [path1]
+    else:
+        testpaths = path1
+    for path in testpaths:
+        path_abs = abs_path_trailing(path)
+        if not path_abs.startswith(possible_parent_abs):
+            return False
+    return True
Signature · 1 line
2.14.3
#4 lore raw

PATCH 2/8 devtool: fix poor handling of upgraded BBCLASSEXTENDed recipes

In reply to Paul Eggleton

Fix two aspects of handling BBCLASSEXTENDed targets (e.g. openssl-native) that have been run through "devtool upgrade":

* Fix recipe name not showing up in "devtool status"
* Fix "devtool reset" not deleting empty directories under the recipe directory within the workspace, which may lead to problems if you subsequently run "devtool upgrade" on the same target again

---
 scripts/devtool                 | 5 +++--
 scripts/lib/devtool/standard.py | 6 +++++-
 2 files changed, 8 insertions(+), 3 deletions(-)
scripts/devtool +3 -2
index a651d8f213a..b4bfbb8329c 100755
@@ -117,12 +117,13 @@ def read_workspace():
             for line in f:
                 res = externalsrc_re.match(line.rstrip())
                 if res:
-                    pn = res.group(2) or os.path.splitext(os.path.basename(fn))[0].split('_')[0]
+                    recipepn = os.path.splitext(os.path.basename(fn))[0].split('_')[0]
+                    pn = res.group(2) or recipepn
                     # Find the recipe file within the workspace, if any
                     bbfile = os.path.basename(fn).replace('.bbappend', '.bb').replace('%', '*')
                     recipefile = glob.glob(os.path.join(config.workspace_path,
                                                         'recipes',
-                                                        pn,
+                                                        recipepn,
                                                         bbfile))
                     if recipefile:
                         recipefile = recipefile[0]
scripts/lib/devtool/standard.py +5 -1
index 26187a0c413..7c4ae859434 100644
@@ -1744,6 +1744,7 @@ def status(args, config, basepath, workspace):
 
 def _reset(recipes, no_clean, config, basepath, workspace):
     """Reset one or more recipes"""
+    import oe.path
 
     def clean_preferred_provider(pn, layerconf_path):
         """Remove PREFERRED_PROVIDER from layer.conf'"""
@@ -1802,7 +1803,10 @@ def _reset(recipes, no_clean, config, basepath, workspace):
                         preservedir(os.path.join(root, dn))
                 os.rmdir(origdir)
 
-        preservedir(os.path.join(config.workspace_path, 'recipes', pn))
+        recipefile = workspace[pn]['recipefile']
+        if recipefile and oe.path.is_path_under(recipefile, config.workspace_path):
+            # This should always be true if recipefile is set, but just in case
+            preservedir(os.path.dirname(recipefile))
         # We don't automatically create this dir next to appends, but the user can
         preservedir(os.path.join(config.workspace_path, 'appends', pn))
 
Signature · 1 line
2.14.3
#5 lore raw

PATCH 3/8 devtool: reset: delete bbappend file if _check_preserve() doesn't

In reply to Paul Eggleton

If the .devtool_md5 file doesn't contain a reference to the bbappend file (e.g. because devtool was interrupted before it could write that out) then _check_preserve() won't delete it, so we need to delete it separately because otherwise the recipe won't actually be reset.

---
 scripts/lib/devtool/standard.py | 7 +++++++
 1 file changed, 7 insertions(+)
scripts/lib/devtool/standard.py +7 -0
index 7c4ae859434..0c31e1768cf 100644
@@ -1791,6 +1791,13 @@ def _reset(recipes, no_clean, config, basepath, workspace):
     for pn in recipes:
         _check_preserve(config, pn)
 
+        appendfile = workspace[pn]['bbappend']
+        if os.path.exists(appendfile):
+            # This shouldn't happen, but is possible if devtool errored out prior to
+            # writing the md5 file. We need to delete this here or the recipe won't
+            # actually be reset
+            os.remove(appendfile)
+
         preservepath = os.path.join(config.workspace_path, 'attic', pn, pn)
         def preservedir(origdir):
             if os.path.exists(origdir):
Signature · 1 line
2.14.3
#6 lore raw

PATCH 4/8 devtool: finish: fix erroneously creating bbappend for relative paths

In reply to Paul Eggleton

After OE-Core rev 5e3fe00a0233d563781849a44f53885b4e924a9c we call os.path.abspath() on the original layer path, but we later compare that to the destination layer path. If that layer path isn't absolute but is effectively the same path, it should be writing to the original recipe but because we weren't making it absolute we were writing a bbappend instead. Call os.path.abspath() on the destination path as well to avoid that.

---
 scripts/lib/devtool/standard.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
scripts/lib/devtool/standard.py +1 -3
index 0c31e1768cf..6cd52fe4f68 100644
@@ -1861,9 +1861,7 @@ def _get_layer(layername, d):
         layerdir = layers.get('meta', None)
     else:
         layerdir = layers.get(layername, None)
-    if layerdir:
-        layerdir = os.path.abspath(layerdir)
-    return layerdir or layername
+    return os.path.abspath(layerdir or layername)
 
 def finish(args, config, basepath, workspace):
     """Entry point for the devtool 'finish' subcommand"""
Signature · 1 line
2.14.3
#7 lore raw

PATCH 5/8 devtool: deploy-target: don't specify ssh/scp port unless user does

In reply to Paul Eggleton

If the user doesn't specify a port then we should avoid specifying one on the ssh/scp command line in case the user has configured one for the host they are connecting to, which was being overridden unnecessarily.

Fixes [YOCTO #12381].

---
 scripts/lib/devtool/deploy.py | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)
scripts/lib/devtool/deploy.py +4 -8
index 9cc492788b6..52e261d5600 100644
@@ -213,9 +213,7 @@ def deploy(args, config, basepath, workspace):
 
         scp_port = ''
         ssh_port = ''
-        if not args.port:
-            raise DevtoolError("If you specify -P/--port then you must provide the port to be used to connect to the target")
-        else:
+        if args.port:
             scp_port = "-P %s" % args.port
             ssh_port = "-p %s" % args.port
 
@@ -280,9 +278,7 @@ def undeploy(args, config, basepath, workspace):
 
     scp_port = ''
     ssh_port = ''
-    if not args.port:
-        raise DevtoolError("If you specify -P/--port then you must provide the port to be used to connect to the target")
-    else:
+    if args.port:
         scp_port = "-P %s" % args.port
         ssh_port = "-p %s" % args.port
 
@@ -328,7 +324,7 @@ def register_commands(subparsers, context):
     parser_deploy.add_argument('-n', '--dry-run', help='List files to be deployed only', action='store_true')
     parser_deploy.add_argument('-p', '--no-preserve', help='Do not preserve existing files', action='store_true')
     parser_deploy.add_argument('--no-check-space', help='Do not check for available space before deploying', action='store_true')
-    parser_deploy.add_argument('-P', '--port', default='22', help='Port to use for connection to the target')
+    parser_deploy.add_argument('-P', '--port', help='Specify port to use for connection to the target')
 
     strip_opts = parser_deploy.add_mutually_exclusive_group(required=False)
     strip_opts.add_argument('-S', '--strip',
@@ -350,5 +346,5 @@ def register_commands(subparsers, context):
     parser_undeploy.add_argument('-s', '--show-status', help='Show progress/status output', action='store_true')
     parser_undeploy.add_argument('-a', '--all', help='Undeploy all recipes deployed on the target', action='store_true')
     parser_undeploy.add_argument('-n', '--dry-run', help='List files to be undeployed only', action='store_true')
-    parser_undeploy.add_argument('-P', '--port', default='22', help='Port to use for connection to the target')
+    parser_undeploy.add_argument('-P', '--port', help='Specify port to use for connection to the target')
     parser_undeploy.set_defaults(func=undeploy)
Signature · 1 line
2.14.3
#8 lore raw

PATCH 6/8 lib/oe/recipeutils: add .txz extension to archive list

In reply to Paul Eggleton

Prompted by bitbake commit 2ba8a6b25ccc12e7b543e8450121e5311c7a701d, add .txz to the list of archives used within get_recipe_local_files() here as well.

---
 meta/lib/oe/recipeutils.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/recipeutils.py +1 -1
index 4e0859e6d90..aa64553c06b 100644
@@ -408,7 +408,7 @@ def get_recipe_local_files(d, patches=False, archives=False):
     # fetcher) though note that this only encompasses actual container formats
     # i.e. that can contain multiple files as opposed to those that only
     # contain a compressed stream (i.e. .tar.gz as opposed to just .gz)
-    archive_exts = ['.tar', '.tgz', '.tar.gz', '.tar.Z', '.tbz', '.tbz2', '.tar.bz2', '.tar.xz', '.tar.lz', '.zip', '.jar', '.rpm', '.srpm', '.deb', '.ipk', '.tar.7z', '.7z']
+    archive_exts = ['.tar', '.tgz', '.tar.gz', '.tar.Z', '.tbz', '.tbz2', '.tar.bz2', '.txz', '.tar.xz', '.tar.lz', '.zip', '.jar', '.rpm', '.srpm', '.deb', '.ipk', '.tar.7z', '.7z']
     ret = {}
     for uri in uris:
         if fetch.ud[uri].type == 'file':
Signature · 1 line
2.14.3
#9 lore raw

PATCH 7/8 devtool: search: also look in recipe cache

In reply to Paul Eggleton

If pkgdata isn't present or is incomplete, then you get either a traceback or you don't see the results you were hoping for. The recipe cache that bitbake collects during startup contains some useful information for each recipe that we could search through as well, and we can access it easily using tinfoil's all_recipes() API function, so add some code that does that. (We still show a warning if pkgdata isn't present, as there are certain dynamic packages that are generated at packaging time that won't show up in the cache).

One side-effect of this is that we will start showing non-target recipes - that's actually a good thing, since seeing those is useful, however we exclude nativesdk recipes when in the eSDK to avoid confusion since nativesdk isn't directly applicable there.

Fixes [YOCTO #12356].

---
 scripts/lib/devtool/search.py | 96 ++++++++++++++++++++++++++++---------------
 1 file changed, 63 insertions(+), 33 deletions(-)
scripts/lib/devtool/search.py +63 -33
index 054985b85d5..b51a7b80207 100644
@@ -36,44 +36,74 @@ def search(args, config, basepath, workspace):
 
         keyword_rc = re.compile(args.keyword)
 
-        for fn in os.listdir(pkgdata_dir):
-            pfn = os.path.join(pkgdata_dir, fn)
-            if not os.path.isfile(pfn):
+        def print_match(pn):
+            rd = parse_recipe(config, tinfoil, pn, True)
+            if not rd:
+                return
+            summary = rd.getVar('SUMMARY')
+            if summary == rd.expand(defsummary):
+                summary = ''
+            print("%s  %s" % (pn.ljust(20), summary))
+
+
+        matches = []
+        if os.path.exists(pkgdata_dir):
+            for fn in os.listdir(pkgdata_dir):
+                pfn = os.path.join(pkgdata_dir, fn)
+                if not os.path.isfile(pfn):
+                    continue
+
+                packages = []
+                match = False
+                if keyword_rc.search(fn):
+                    match = True
+
+                if not match:
+                    with open(pfn, 'r') as f:
+                        for line in f:
+                            if line.startswith('PACKAGES:'):
+                                packages = line.split(':', 1)[1].strip().split()
+
+                    for pkg in packages:
+                        if keyword_rc.search(pkg):
+                            match = True
+                            break
+                        if os.path.exists(os.path.join(pkgdata_dir, 'runtime', pkg + '.packaged')):
+                            with open(os.path.join(pkgdata_dir, 'runtime', pkg), 'r') as f:
+                                for line in f:
+                                    if ': ' in line:
+                                        splitline = line.split(':', 1)
+                                        key = splitline[0]
+                                        value = splitline[1].strip()
+                                    if key in ['PKG_%s' % pkg, 'DESCRIPTION', 'FILES_INFO'] or key.startswith('FILERPROVIDES_'):
+                                        if keyword_rc.search(value):
+                                            match = True
+                                            break
+                if match:
+                    print_match(fn)
+                    matches.append(fn)
+        else:
+            logger.warning('Package data is not available, results may be limited')
+
+        for recipe in tinfoil.all_recipes():
+            if args.fixed_setup and 'nativesdk' in recipe.inherits():
                 continue
 
-            packages = []
             match = False
-            if keyword_rc.search(fn):
+            if keyword_rc.search(recipe.pn):
                 match = True
-
-            if not match:
-                with open(pfn, 'r') as f:
-                    for line in f:
-                        if line.startswith('PACKAGES:'):
-                            packages = line.split(':', 1)[1].strip().split()
-
-                for pkg in packages:
-                    if keyword_rc.search(pkg):
+            else:
+                for prov in recipe.provides:
+                    if keyword_rc.search(prov):
                         match = True
                         break
-                    if os.path.exists(os.path.join(pkgdata_dir, 'runtime', pkg + '.packaged')):
-                        with open(os.path.join(pkgdata_dir, 'runtime', pkg), 'r') as f:
-                            for line in f:
-                                if ': ' in line:
-                                    splitline = line.split(':', 1)
-                                    key = splitline[0]
-                                    value = splitline[1].strip()
-                                if key in ['PKG_%s' % pkg, 'DESCRIPTION', 'FILES_INFO'] or key.startswith('FILERPROVIDES_'):
-                                    if keyword_rc.search(value):
-                                        match = True
-                                        break
-
-            if match:
-                rd = parse_recipe(config, tinfoil, fn, True)
-                summary = rd.getVar('SUMMARY')
-                if summary == rd.expand(defsummary):
-                    summary = ''
-                print("%s  %s" % (fn.ljust(20), summary))
+                if not match:
+                    for rprov in recipe.rprovides:
+                        if keyword_rc.search(rprov):
+                            match = True
+                            break
+            if match and not recipe.pn in matches:
+                print_match(recipe.pn)
     finally:
         tinfoil.shutdown()
 
@@ -85,4 +115,4 @@ def register_commands(subparsers, context):
                                             description='Searches for available target recipes. Matches on recipe name, package name, description and installed files, and prints the recipe name on match.',
                                             group='info')
     parser_search.add_argument('keyword', help='Keyword to search for (regular expression syntax allowed)')
-    parser_search.set_defaults(func=search, no_workspace=True)
+    parser_search.set_defaults(func=search, no_workspace=True, fixed_setup=context.fixed_setup)
Signature · 1 line
2.14.3
#10 lore raw

PATCH 8/8 devtool: search: tweak help text

In reply to Paul Eggleton

* We now match on more than just target recipes, so don't specify that only target recipes are searched.
* We're printing the SUMMARY value in addition to the name, so mention that so it's clear where that text is coming from.
* Remind users that they should use quotes around the keyword to avoid shell expansion when using regular expressions.

---
 scripts/lib/devtool/search.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/lib/devtool/search.py +2 -2
index b51a7b80207..b4f209b7e3a 100644
@@ -112,7 +112,7 @@ def search(args, config, basepath, workspace):
 def register_commands(subparsers, context):
     """Register devtool subcommands from this plugin"""
     parser_search = subparsers.add_parser('search', help='Search available recipes',
-                                            description='Searches for available target recipes. Matches on recipe name, package name, description and installed files, and prints the recipe name on match.',
+                                            description='Searches for available recipes. Matches on recipe name, package name, description and installed files, and prints the recipe name and summary on match.',
                                             group='info')
-    parser_search.add_argument('keyword', help='Keyword to search for (regular expression syntax allowed)')
+    parser_search.add_argument('keyword', help='Keyword to search for (regular expression syntax allowed, use quotes to avoid shell expansion)')
     parser_search.set_defaults(func=search, no_workspace=True, fixed_setup=context.fixed_setup)
Signature · 1 line
2.14.3
#2 lore raw

Re: [PATCH 1/8] lib/oe/path: implement is_path_under()

In reply to Paul Eggleton
Quoted 6 lines

From: [email protected] [mailto:[email protected]] On Behalf Of Paul Eggleton
Sent: den 23 februari 2018 04:02
To: [email protected]

<snip>

Quoted 10 lines
---
 meta/lib/oe/path.py | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
meta/lib/oe/path.py index renamed from meta/lib/oe/path.py +0 -0

1ea03d5d568..346b48f3641 100644

meta/lib/oe/path.py def realpath(file, root, use_physdir = True, loop_cnt · line 241

Quoted 6 lines
meta/lib/oe/path.py +2 -0
@@ -237,3 +237,27 @@ def realpath(file, root, use_physdir = True, loop_cnt
 = 100, assume_dir = False)
         raise
 
     return file
+
+def is_path_under(path1, possible_parent):

Changing the signature to is_path_under(possible_parent, *paths) will make the list management more natural.

This function will return True if path1 is the empty list. Is that intentional?

/Ola Nilsson

Quoted 30 lines
meta/lib/oe/path.py +22 -1
+    """
+    Return True if a path is underneath another, False otherwise.
+    Multiple paths to test can be specified (as a list or tuple) in
+    which case all specified test paths must be under the parent to
+    return True.
+    """
+    def abs_path_trailing(pth):
+        pth_abs = os.path.abspath(pth)
+        if not pth_abs.endswith(os.sep):
+            pth_abs += os.sep
+        return pth_abs
+
+    possible_parent_abs = abs_path_trailing(possible_parent)
+    if isinstance(path1, str):
+        testpaths = [path1]
+    else:
+        testpaths = path1
+    for path in testpaths:
+        path_abs = abs_path_trailing(path)
+        if not path_abs.startswith(possible_parent_abs):
+            return False
+    return True
--

2.14.3

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#3 lore raw

Re: [PATCH 1/8] lib/oe/path: implement is_path_under()

Hi Ola,

Quoted 4 lines · NZDT Ola x Nilsson
Quoted 1 line

+def is_path_under(path1, possible_parent):

Changing the signature to is_path_under(possible_parent, *paths) will make the list management more natural.

Right, it does simplify things a little. In fact given the parameter order is changing I'll rename it to is_path_parent() at the same time.

Quoted 2 lines

This function will return True if path1 is the empty list. Is that intentional?

I don't believe so. I'll change the function to return False in that case, well spotted.

Thanks,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre