Skip to contentSkip to conversation

devtool add fixes

PATCH New Paul Eggleton 1 participant 4 messages Oct 4, 2016

lore
Share & tools
lore
#0 lore raw

cover devtool add fixes

The following changes since commit 21cc2a3f63ea260dbf6b50e2fd4dd50cacdd9935:

machine-sdk: Clear ABIEXTENSION to avoid sstate checksum mismatch issues (2016-10-01 08:16:03 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib paule/devtool21-oe
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool21-oe

Paul Eggleton (3):
devtool: add: fix error message when only specifying a recipe name
devtool: add: display a warning for deprecated -f/--fetch option
devtool: add: build nodejs-native if npm is needed and not available

 scripts/lib/devtool/__init__.py      | 29 +++++++++++++++++++++++++++++
 scripts/lib/devtool/standard.py      | 35 ++++++++++++++++++++++++-----------
 scripts/lib/recipetool/create.py     | 10 ++++++----
 scripts/lib/recipetool/create_npm.py |  4 +++-
 4 files changed, 62 insertions(+), 16 deletions(-)
Signature · 1 line
2.5.5
#1 lore raw

PATCH 1/3 devtool: add: fix error message when only specifying a recipe name

We were supposed to be printing out the specified recipe name here but I forgot to specify a parameter for the string.

---
 scripts/lib/devtool/standard.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/lib/devtool/standard.py +1 -1
index 8319145..c8d7eb1 100644
@@ -64,7 +64,7 @@ def add(args, config, basepath, workspace):
             args.srctree = args.recipename
             args.recipename = None
         elif os.path.isdir(args.recipename):
-            logger.warn('Ambiguous argument %s - assuming you mean it to be the recipe name')
+            logger.warn('Ambiguous argument "%s" - assuming you mean it to be the recipe name' % args.recipename)
 
     if args.srctree and os.path.isfile(args.srctree):
         args.fetchuri = 'file://' + os.path.abspath(args.srctree)
Signature · 1 line
2.5.5
#2 lore raw

PATCH 2/3 devtool: add: display a warning for deprecated -f/--fetch option

In reply to Paul Eggleton

We want to remove the -f/--fetch option at some point (as you can now specify a URL as a positional argument instead) so display a warning that it's deprecated if it is used.

---
 scripts/lib/devtool/standard.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/lib/devtool/standard.py +1 -1
index c8d7eb1..02ed235 100644
@@ -74,7 +74,7 @@ def add(args, config, basepath, workspace):
         if args.fetchuri:
             raise DevtoolError('URI specified as positional argument as well as -f/--fetch')
         else:
-            # FIXME should show a warning that -f/--fetch is deprecated here
+            logger.warn('-f/--fetch option is deprecated - you can now simply specify the URL to fetch as a positional argument instead')
             args.fetchuri = args.fetch
 
     if args.recipename:
Signature · 1 line
2.5.5
#3 lore raw

PATCH 3/3 devtool: add: build nodejs-native if npm is needed and not available

In reply to Paul Eggleton

If the user runs devtool add on an npm:// URL (or source tree that uses node.js), and npm is not available, just build nodejs-native instead of telling the user they need to do it; if that fails because there isn't any such recipe (which would be the default, since it's not in OE-Core) then produce a slightly more readable error message hinting at what the user needs to do.

Note that this forces the use of nodejs-native rather than npm on the host - this makes sense for two reasons: (1) we need it to be compatible with nodejs for the target, and (2) we have to have a recipe for that anyway, so allowing you to avoid having a recipe for the native version isn't really beneficial.

There's a bit of a hack in here in order to allow this - for node.js sources that aren't fetched via npm we don't know that they are that until we've fetched and unpacked them, by which time we're inside recipetool and have an active tinfoil instance that will prevent bitbake being run. To avoid this being an issue, we allow recipetool to get to the point where we know we need npm and then exit with a specific exit code, at which point devtool can try to build it and then if that succeeds, it will re-execute recipetool. This is definitely not ideal, but it can't really be refactored and done properly until we do the tinfoil2 refactoring; in the mean time though we still want to be helpful to the user.

Fixes [YOCTO #10337].

---
 scripts/lib/devtool/__init__.py      | 29 +++++++++++++++++++++++++++++
 scripts/lib/devtool/standard.py      | 31 ++++++++++++++++++++++---------
 scripts/lib/recipetool/create.py     | 10 ++++++----
 scripts/lib/recipetool/create_npm.py |  4 +++-
 4 files changed, 60 insertions(+), 14 deletions(-)
scripts/lib/devtool/__init__.py +29 -0
index b432e3d..e675133 100644
@@ -259,3 +259,32 @@ def get_bbclassextend_targets(recipefile, pn):
             elif variant in ['native', 'cross', 'crosssdk']:
                 targets.append('%s-%s' % (pn, variant))
     return targets
+
+def ensure_npm(config, basepath, fixed_setup=False):
+    """
+    Ensure that npm is available and either build it or show a
+    reasonable error message
+    """
+    tinfoil = setup_tinfoil(config_only=True, basepath=basepath)
+    try:
+        nativepath = tinfoil.config_data.getVar('STAGING_BINDIR_NATIVE', True)
+    finally:
+        tinfoil.shutdown()
+
+    npmpath = os.path.join(nativepath, 'npm')
+    if not os.path.exists(npmpath):
+        logger.info('Building nodejs-native')
+        try:
+            exec_build_env_command(config.init_path, basepath,
+                                'bitbake -q nodejs-native', watch=True)
+        except bb.process.ExecutionError as e:
+            if "Nothing PROVIDES 'nodejs-native'" in e.stdout:
+                if fixed_setup:
+                    msg = 'nodejs-native is required for npm but is not available within this SDK'
+                else:
+                    msg = 'nodejs-native is required for npm but is not available - you will likely need to add a layer that provides nodejs'
+                raise DevtoolError(msg)
+            else:
+                raise
+        if not os.path.exists(npmpath):
+            raise DevtoolError('Built nodejs-native but npm binary still could not be found at %s' % npmpath)
scripts/lib/devtool/standard.py +22 -9
index 02ed235..4b9b173 100644
@@ -30,7 +30,7 @@ import errno
 import glob
 import filecmp
 from collections import OrderedDict
-from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, setup_git_repo, recipe_to_append, get_bbclassextend_targets, DevtoolError
+from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, setup_git_repo, recipe_to_append, get_bbclassextend_targets, ensure_npm, DevtoolError
 from devtool import parse_recipe
 
 logger = logging.getLogger('devtool')
@@ -128,6 +128,9 @@ def add(args, config, basepath, workspace):
         color = args.color
     extracmdopts = ''
     if args.fetchuri:
+        if args.fetchuri.startswith('npm://'):
+            ensure_npm(config, basepath, args.fixed_setup)
+
         source = args.fetchuri
         if srctree:
             extracmdopts += ' -x %s' % srctree
@@ -150,13 +153,23 @@ def add(args, config, basepath, workspace):
 
     tempdir = tempfile.mkdtemp(prefix='devtool')
     try:
-        try:
-            stdout, _ = exec_build_env_command(config.init_path, basepath, 'recipetool --color=%s create -o %s "%s" %s' % (color, tempdir, source, extracmdopts))
-        except bb.process.ExecutionError as e:
-            if e.exitcode == 15:
-                raise DevtoolError('Could not auto-determine recipe name, please specify it on the command line')
-            else:
-                raise DevtoolError('Command \'%s\' failed:\n%s' % (e.command, e.stdout))
+        while True:
+            try:
+                stdout, _ = exec_build_env_command(config.init_path, basepath, 'recipetool --color=%s create -o %s "%s" %s' % (color, tempdir, source, extracmdopts))
+            except bb.process.ExecutionError as e:
+                if e.exitcode == 14:
+                    # FIXME this is a horrible hack that is unfortunately
+                    # necessary due to the fact that we can't run bitbake from
+                    # inside recipetool since recipetool keeps tinfoil active
+                    # with references to it throughout the code, so we have
+                    # to exit out and come back here to do it.
+                    ensure_npm(config, basepath, args.fixed_setup)
+                    continue
+                elif e.exitcode == 15:
+                    raise DevtoolError('Could not auto-determine recipe name, please specify it on the command line')
+                else:
+                    raise DevtoolError('Command \'%s\' failed:\n%s' % (e.command, e.stdout))
+            break
 
         recipes = glob.glob(os.path.join(tempdir, '*.bb'))
         if recipes:
@@ -1567,7 +1580,7 @@ def register_commands(subparsers, context):
     parser_add.add_argument('--binary', '-b', help='Treat the source tree as something that should be installed verbatim (no compilation, same directory structure). Useful with binary packages e.g. RPMs.', action='store_true')
     parser_add.add_argument('--also-native', help='Also add native variant (i.e. support building recipe for the build host as well as the target machine)', action='store_true')
     parser_add.add_argument('--src-subdir', help='Specify subdirectory within source tree to use', metavar='SUBDIR')
-    parser_add.set_defaults(func=add)
+    parser_add.set_defaults(func=add, fixed_setup=context.fixed_setup)
 
     parser_modify = subparsers.add_parser('modify', help='Modify the source for an existing recipe',
                                        description='Sets up the build environment to modify the source for an existing recipe. The default behaviour is to extract the source being fetched by the recipe into a git tree so you can work on it; alternatively if you already have your own pre-prepared source tree you can specify -n/--no-extract.',
scripts/lib/recipetool/create.py +6 -4
index 9b31fe9..d427d32 100644
@@ -406,10 +406,7 @@ def create_recipe(args):
         srctree = tempsrc
         if fetchuri.startswith('npm://'):
             # Check if npm is available
-            npm = bb.utils.which(tinfoil.config_data.getVar('PATH', True), 'npm')
-            if not npm:
-                logger.error('npm:// URL requested but npm is not available - you need to either build nodejs-native or install npm using your package manager')
-                sys.exit(1)
+            check_npm(tinfoil.config_data)
         logger.info('Fetching %s...' % srcuri)
         try:
             checksums = scriptutils.fetch_uri(tinfoil.config_data, fetchuri, srctree, srcrev)
@@ -1076,6 +1073,11 @@ def convert_rpm_xml(xmlfile):
     return values
 
 
+def check_npm(d):
+    if not os.path.exists(os.path.join(d.getVar('STAGING_BINDIR_NATIVE', True), 'npm')):
+        logger.error('npm required to process specified source, but npm is not available - you need to build nodejs-native first')
+        sys.exit(14)
+
 def register_commands(subparsers):
     parser_create = subparsers.add_parser('create',
                                           help='Create a new recipe',
scripts/lib/recipetool/create_npm.py +3 -1
index e794614..7bb844c 100644
@@ -21,7 +21,7 @@ import subprocess
 import tempfile
 import shutil
 import json
-from recipetool.create import RecipeHandler, split_pkg_licenses, handle_license_vars
+from recipetool.create import RecipeHandler, split_pkg_licenses, handle_license_vars, check_npm
 
 logger = logging.getLogger('recipetool')
 
@@ -157,6 +157,8 @@ class NpmRecipeHandler(RecipeHandler):
 
         files = RecipeHandler.checkfiles(srctree, ['package.json'])
         if files:
+            check_npm(tinfoil.config_data)
+
             data = read_package_json(files[0])
             if 'name' in data and 'version' in data:
                 extravalues['PN'] = data['name']
Signature · 1 line
2.5.5