create_npm: fixes to allow eg Angular2 based applications

PATCH In review Anders Darander 4 participants 13 messages Mar 1, 2017 – Mar 9, 2017

Share & tools

Anders Darander

author

2017-03-01 17:19 UTC #0 lore raw

cover create_npm: fixes to allow eg Angular2 based applications

After the fixes to bitbake, which have gone in to master, there's still some issues with having devtool add work correctly with a angular2 based applications.

Some of the issues, license strings gets split incorrectly, AND and OR are used in license strings in npm, @ and / in npm package names...

There's also a need to be able to install devDependencies from a package.json file.

The following changes since commit 0a1427bf9aeeda6bee2cc0af8da4ea5fd90aef6f:

recipes: Make use of the new bb.utils.filter() function (2017-03-01 11:17:22 +0000)

are available in the git repository at:

git://github.com/darander/openembedded-core npm
https://github.com/darander/openembedded-core/tree/npm

Anders Darander (7):
create_npm.py: add devDependencies to depends create_npm.py: convert MIT/X11 to MIT lib/oe/package.py: remove @ from package name classes/npm: allow installation of devDependencies scripts/lib/create_npm: rewrite see license in eula scripts/lib/create_npm: handle AND and OR in licenses scripts/lib/create_npm: handle Public Domain licenses

 meta/classes/npm.bbclass             | 13 ++++++++++--
 meta/lib/oe/package.py               |  1 +
 scripts/lib/recipetool/create_npm.py | 38 +++++++++++++++++++++++++++++++-----
 3 files changed, 45 insertions(+), 7 deletions(-)

Signature · 1 line

2.10.2

Anders Darander

author

2017-03-01 17:20 UTC #1 lore raw

PATCH 1/7 create_npm.py: add devDependencies to depends

Web applications built using e.g. angular2, usually requires that the packages in devDependencies are available.

Signed-off-by Anders Darander <[email protected]>

---
 scripts/lib/recipetool/create_npm.py | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

scripts/lib/recipetool/create_npm.py +13 -4

index 3ba6de0..11b2950 100644
@@ -83,7 +83,7 @@ class NpmRecipeHandler(RecipeHandler):
         extravalues['extrafiles']['lockdown.json'] = tmpfile
         lines_before.append('NPM_LOCKDOWN := "${THISDIR}/${PN}/lockdown.json"')
 
-    def _handle_dependencies(self, d, deps, optdeps, lines_before, srctree):
+    def _handle_dependencies(self, d, deps, optdeps, devdeps, lines_before, srctree):
         import scriptutils
         # If this isn't a single module we need to get the dependencies
         # and add them to SRC_URI

@@ -97,6 +97,10 @@ class NpmRecipeHandler(RecipeHandler):
                         depdata = self.get_npm_data(dep, depver, d)
                         if self.check_npm_optional_dependency(depdata):
                             deplist[dep] = depdata
+                    for dep, depver in devdeps.items():
+                        depdata = self.get_npm_data(dep, depver, d)
+                        if self.check_npm_optional_dependency(depdata):
+                            deplist[dep] = depdata
                     for dep, depver in deps.items():
                         depdata = self.get_npm_data(dep, depver, d)
                         deplist[dep] = depdata

@@ -179,8 +183,8 @@ class NpmRecipeHandler(RecipeHandler):
                 if 'homepage' in data:
                     extravalues['HOMEPAGE'] = data['homepage']
 
-                deps, optdeps = self.get_npm_package_dependencies(data)
-                updated = self._handle_dependencies(tinfoil.config_data, deps, optdeps, lines_before, srctree)
+                deps, optdeps, devdeps = self.get_npm_package_dependencies(data)
+                updated = self._handle_dependencies(tinfoil.config_data, deps, optdeps, devdeps, lines_before, srctree)
                 if updated:
                     # We need to redo the license stuff
                     self._replace_license_vars(srctree, lines_before, handled, extravalues, tinfoil.config_data)

@@ -277,15 +281,20 @@ class NpmRecipeHandler(RecipeHandler):
     def get_npm_package_dependencies(self, pdata):
         dependencies = pdata.get('dependencies', {})
         optionalDependencies = pdata.get('optionalDependencies', {})
+        devDependencies = pdata.get('devDependencies', {})
         dependencies.update(optionalDependencies)
+        dependencies.update(devDependencies)
         depsfound = {}
         optdepsfound = {}
+        devdepsfound = {}
         for dep in dependencies:
             if dep in optionalDependencies:
                 optdepsfound[dep] = dependencies[dep]
+            elif dep in devDependencies:
+                devdepsfound[dep] = dependencies[dep]
             else:
                 depsfound[dep] = dependencies[dep]
-        return depsfound, optdepsfound
+        return depsfound, optdepsfound, devdepsfound
 
     # FIXME this is effectively duplicated from lib/bb/fetch2/npm.py
     # (split out from _getdependencies())

Signature · 1 line

2.10.2

Paul Eggleton

2017-03-02 01:44 UTC #2 lore raw

Re: [PATCH 1/7] create_npm.py: add devDependencies to depends

Hi Anders,

Quoted 58 lines · NZDT Anders Darander

Web applications built using e.g. angular2, usually requires that the packages in devDependencies are available.

Signed-off-by Anders Darander <[email protected]>

---
 scripts/lib/recipetool/create_npm.py | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

(quoted diff) +0 -0

b/scripts/lib/recipetool/create_npm.py index 3ba6de0..11b2950 100644

scripts/lib/recipetool/create_npm.py +0 -1

@@ -83,7 +83,7 @@ class NpmRecipeHandler(RecipeHandler):
         extravalues['extrafiles']['lockdown.json'] = tmpfile
         lines_before.append('NPM_LOCKDOWN :=
 "${THISDIR}/${PN}/lockdown.json"')
 
-    def _handle_dependencies(self, d, deps, optdeps, lines_before,
 srctree): +    def _handle_dependencies(self, d, deps, optdeps, devdeps,

lines_before, srctree): import scriptutils
# If this isn't a single module we need to get the dependencies # and add them to SRC_URI

scripts/lib/recipetool/create_npm.py +4 -2

@@ -97,6 +97,10 @@ class NpmRecipeHandler(RecipeHandler):
                         depdata = self.get_npm_data(dep, depver, d)
                         if self.check_npm_optional_dependency(depdata):
                             deplist[dep] = depdata
+                    for dep, depver in devdeps.items():
+                        depdata = self.get_npm_data(dep, depver, d)
+                        if self.check_npm_optional_dependency(depdata):
+                            deplist[dep] = depdata
                     for dep, depver in deps.items():
                         depdata = self.get_npm_data(dep, depver, d)
                         deplist[dep] = depdata

@@ -179,8 +183,8 @@ class NpmRecipeHandler(RecipeHandler):
                 if 'homepage' in data:
                     extravalues['HOMEPAGE'] = data['homepage']
 
-                deps, optdeps = self.get_npm_package_dependencies(data)
-                updated = self._handle_dependencies(tinfoil.config_data,
 deps, optdeps, lines_before, srctree) +                deps, optdeps,

devdeps = self.get_npm_package_dependencies(data) + updated
= self._handle_dependencies(tinfoil.config_data, deps, optdeps, devdeps, lines_before, srctree) if updated:
# We need to redo the license stuff self._replace_license_vars(srctree, lines_before,
handled, extravalues, tinfoil.config_data) @@ -277,15 +281,20 @@ class NpmRecipeHandler(RecipeHandler):
def get_npm_package_dependencies(self, pdata):
dependencies = pdata.get('dependencies', {}) optionalDependencies = pdata.get('optionalDependencies', {})

scripts/lib/recipetool/create_npm.py +3 -0

+        devDependencies = pdata.get('devDependencies', {})
         dependencies.update(optionalDependencies)
+        dependencies.update(devDependencies)
         depsfound = {}
         optdepsfound = {}
+        devdepsfound = {}
         for dep in dependencies:
             if dep in optionalDependencies:

scripts/lib/recipetool/create_npm.py class NpmRecipeHandler(RecipeHandler): · line 300

Quoted 10 lines

scripts/lib/recipetool/create_npm.py +3 -1

                 optdepsfound[dep] = dependencies[dep]
+            elif dep in devDependencies:
+                devdepsfound[dep] = dependencies[dep]
             else:
                 depsfound[dep] = dependencies[dep]
-        return depsfound, optdepsfound
+        return depsfound, optdepsfound, devdepsfound
 
     # FIXME this is effectively duplicated from lib/bb/fetch2/npm.py
     # (split out from _getdependencies())

This worries me slightly - I've no familiarity with angular2 but AIUI in general the dev dependencies most often aren't needed, and may be onerous to pull in and build - should we really be pulling those in unconditionally?

Cheers,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre

Anders Darander

author

2017-03-06 07:11 UTC #3 lore raw

Re: [PATCH 1/7] create_npm.py: add devDependencies to depends

Hi,

Quoted 21 lines · Paul Eggleton

Quoted 19 lines · NZDT Anders Darander

NpmRecipeHandler(RecipeHandler):
def get_npm_package_dependencies(self, pdata):
dependencies = pdata.get('dependencies', {}) optionalDependencies = pdata.get('optionalDependencies', {})

scripts/lib/recipetool/create_npm.py +6 -1

+        devDependencies = pdata.get('devDependencies', {})
         dependencies.update(optionalDependencies)
+        dependencies.update(devDependencies)
         depsfound = {}
         optdepsfound = {}
+        devdepsfound = {}
         for dep in dependencies:
             if dep in optionalDependencies:
                 optdepsfound[dep] = dependencies[dep]
+            elif dep in devDependencies:
+                devdepsfound[dep] = dependencies[dep]
             else:
                 depsfound[dep] = dependencies[dep]
-        return depsfound, optdepsfound
+        return depsfound, optdepsfound, devdepsfound

Quoted 6 lines

Quoted 2 lines
# FIXME this is effectively duplicated from lib/bb/fetch2/npm.py # (split out from _getdependencies())
This worries me slightly - I've no familiarity with angular2 but AIUI in general the dev dependencies most often aren't needed, and may be onerous to pull in and build - should we really be pulling those in unconditionally?

I fully understand. However, I found no obvious way of doing this conditionally. At least, the devDependencies won't be used during npm install, unless you add NPM_INSTALL_DEV = "1" to the recipe in question.

Well, I guess, it could be possible to add a command line switch to recipetool; though, in that question, I've got a few questions. Is it possible to add this only for create_npm? (Ie not for generation of other recipes?) How to add this to devtool (and have it propagate for npm-packages only?

Cheers,
Anders

Signature · 2 lines

Anders Darander, Senior System Architect
ChargeStorm AB / eStorm AB

Anders Darander

author

2017-03-01 17:20 UTC #4 lore raw

PATCH 2/7 create_npm.py: convert MIT/X11 to MIT

In reply to Anders Darander

Quite a few npm packages declare MIT/X11 as their license. This is equal to a pure MIT license.

Signed-off-by Anders Darander <[email protected]>

---
 scripts/lib/recipetool/create_npm.py | 2 ++
 1 file changed, 2 insertions(+)

scripts/lib/recipetool/create_npm.py +2 -0

index 11b2950..2816861 100644
@@ -45,6 +45,8 @@ class NpmRecipeHandler(RecipeHandler):
             license = data['license']
             if isinstance(license, dict):
                 license = license.get('type', None)
+            if 'MIT/X11' in license:
+                license = 'MIT'
         return license
 
     def _shrinkwrap(self, srctree, localfilesdir, extravalues, lines_before):

Signature · 1 line

2.10.2

Khem Raj

2017-03-03 01:18 UTC #5 lore raw

Re: [PATCH 2/7] create_npm.py: convert MIT/X11 to MIT

Quoted 2 lines · Anders Darander

Quite a few npm packages declare MIT/X11 as their license. This is equal to a pure MIT license.

it would be good if you could attach a link where this can be referred to.

Quoted 27 lines

Signed-off-by Anders Darander <[email protected]>

---
 scripts/lib/recipetool/create_npm.py | 2 ++
 1 file changed, 2 insertions(+)

scripts/lib/recipetool/create_npm.py +2 -1

index 11b2950..2816861 100644
@@ -45,6 +45,8 @@ class NpmRecipeHandler(RecipeHandler):
             license = data['license']
             if isinstance(license, dict):
                 license = license.get('type', None)
+            if 'MIT/X11' in license:
+                license = 'MIT'
         return license
 
     def _shrinkwrap(self, srctree, localfilesdir, extravalues, lines_before):
--

2.10.2

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Anders Darander

author

2017-03-06 07:21 UTC #6 lore raw

Re: [PATCH 2/7] create_npm.py: convert MIT/X11 to MIT

Quoted 5 lines · Khem Raj

Quoted 2 lines · Anders Darander
Quite a few npm packages declare MIT/X11 as their license. This is equal to a pure MIT license.
it would be good if you could attach a link where this can be referred to.

Sure, I'll try to find a good, authorative one...

Unfortunately, the initial quick searches mostly returns wikipedia pages, as well as number of company sites stating this. I need to search a little bit more in order to find a better reference.

In worst case, I link to a page with the X11 license text, to ease manual inspection.

Cheers,
Anders

Signature · 2 lines

Anders Darander, Senior System Architect
ChargeStorm AB / eStorm AB

Josef Holzmayr

2017-03-09 06:22 UTC #7 lore raw

Re: [PATCH 2/7] create_npm.py: convert MIT/X11 to MIT

Hi Anders,

from what I can see the problem is rooted a bit deeper, and this patch (together with 5, 6, 7 from the series) is basically symptomatic:
the package.jsons license field is mostly expected to be spdx-compliant (see https://docs.npmjs.com/files/package.json#license). So I think we need a concept here to actually handle the complexity it brings, or we end up rewriting a boatload of license strings. And doing that in create_npm.py just feels wrong to me.

my $.02

Signature · 19 lines

Josef Holzmayr
Software Developer Embedded Systems

Tel: +49 8444 9204-48
Fax: +49 8444 9204-50

R-S-I Elektrotechnik GmbH & Co. KG
Woelkestrasse 11
D-85301 Schweitenkirchen
www.rsi-elektrotechnik.de
———————————————
Amtsgericht Ingolstadt – GmbH: HRB 191328 – KG: HRA 170393
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
Ust-IdNr: DE 128592548

_____________________________________________________________
Amtsgericht Ingolstadt - GmbH: HRB 191328 - KG: HRA 170363
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
USt-IdNr.: DE 128592548

Anders Darander

author

2017-03-01 17:20 UTC #8 lore raw

PATCH 3/7 lib/oe/package.py: remove @ from package name

In reply to Anders Darander

@ isn't allowed in package names. Angular2 packages often have @ in their names.

Signed-off-by Anders Darander <[email protected]>

---
 meta/lib/oe/package.py | 1 +
 1 file changed, 1 insertion(+)

meta/lib/oe/package.py +1 -0

index 7953895..0ca41aa 100644
@@ -150,6 +150,7 @@ def npm_split_package_dirs(pkgdir):
                         continue
                     pkgitems.append(pathitem)
                 pkgname = '-'.join(pkgitems).replace('_', '-')
+                pkgname = pkgname.replace('@', '')
                 pkgfile = os.path.join(root, dn, 'package.json')
                 data = None
                 if os.path.exists(pkgfile):

Signature · 1 line

2.10.2

Anders Darander

author

2017-03-01 17:20 UTC #9 lore raw

PATCH 4/7 classes/npm: allow installation of devDependencies

In reply to Anders Darander

Often, eg when using angular2, there's a need to install also the devDependencies.

The default is to keep the old behaviour, to not install devDependencies.

Signed-off-by Anders Darander <[email protected]>

---
 meta/classes/npm.bbclass | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

meta/classes/npm.bbclass +11 -2

index c538040..a69bedb 100644
@@ -14,6 +14,7 @@ def npm_oe_arch_map(target_arch, d):
     return target_arch
 
 NPM_ARCH ?= "${@npm_oe_arch_map(d.getVar('TARGET_ARCH'), d)}"
+NPM_INSTALL_DEV = "0"
 
 npm_do_compile() {
 	# Copy in any additionally fetched modules

@@ -23,12 +24,20 @@ npm_do_compile() {
 	# changing the home directory to the working directory, the .npmrc will
 	# be created in this directory
 	export HOME=${WORKDIR}
-	npm config set dev false
+	if [  "${NPM_INSTALL_DEV}" = "1" ]; then
+		npm config set dev true
+	else
+		npm config set dev false
+	fi
 	npm set cache ${WORKDIR}/npm_cache
 	# clear cache before every build
 	npm cache clear
 	# Install pkg into ${S} without going to the registry
-	npm --arch=${NPM_ARCH} --target_arch=${NPM_ARCH} --production --no-registry install
+	if [  "${NPM_INSTALL_DEV}" = "1" ]; then
+		npm --arch=${NPM_ARCH} --target_arch=${NPM_ARCH} --no-registry install
+	else
+		npm --arch=${NPM_ARCH} --target_arch=${NPM_ARCH} --production --no-registry install
+	fi
 }
 
 npm_do_install() {

Signature · 1 line

2.10.2

Anders Darander

author

2017-03-01 17:20 UTC #10 lore raw

PATCH 5/7 scripts/lib/create_npm: rewrite see license in eula

In reply to Anders Darander

Rewrite the 'SEE LICENSE IN EULA' to a single string (without spaces), to avoid splitting the string later on.

(Otherwise, each word gets split, and assumed to be a license on it's own.

Signed-off-by Anders Darander <[email protected]>

---
 scripts/lib/recipetool/create_npm.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

scripts/lib/recipetool/create_npm.py +4 -2

index 2816861..6a0f3c2 100644
@@ -45,8 +45,10 @@ class NpmRecipeHandler(RecipeHandler):
             license = data['license']
             if isinstance(license, dict):
                 license = license.get('type', None)
-            if 'MIT/X11' in license:
-                license = 'MIT'
+            if license:
+                license = license.replace('MIT/X11', 'MIT')
+                license = license.replace('SEE LICENSE IN EULA',
+                                          'SEE-LICENSE-IN-EULA')
         return license
 
     def _shrinkwrap(self, srctree, localfilesdir, extravalues, lines_before):

Signature · 1 line

2.10.2

Anders Darander

author

2017-03-01 17:20 UTC #11 lore raw

PATCH 6/7 scripts/lib/create_npm: handle AND and OR in licenses

In reply to Anders Darander

Handle npm packages with multiple licenses (AND and OR). Prior to this, AND and OR were treated as licensed in their own.

Signed-off-by Anders Darander <[email protected]>

---
 scripts/lib/recipetool/create_npm.py | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

scripts/lib/recipetool/create_npm.py +15 -1

index 6a0f3c2..89973d9 100644
@@ -46,6 +46,19 @@ class NpmRecipeHandler(RecipeHandler):
             if isinstance(license, dict):
                 license = license.get('type', None)
             if license:
+                if 'OR' in license:
+                    license = license.replace('OR', '|')
+                    license = license.replace('AND', '&')
+                    license = license.replace(' ', '_')
+                    if not license[0] == '(':
+                        license = '(' + license + ')'
+                    print('LICENSE: {}'.format(license))
+                else:
+                    license = license.replace('AND', '&')
+                    if license[0] == '(':
+                        license = license[1:]
+                    if license[-1] == ')':
+                        license = license[:-1]
                 license = license.replace('MIT/X11', 'MIT')
                 license = license.replace('SEE LICENSE IN EULA',
                                           'SEE-LICENSE-IN-EULA')

@@ -224,7 +237,8 @@ class NpmRecipeHandler(RecipeHandler):
                     packages = OrderedDict((x,y[0]) for x,y in npmpackages.items())
                     packages['${PN}'] = ''
                     pkglicenses = split_pkg_licenses(licvalues, packages, lines_after, licenses)
-                    all_licenses = list(set([item for pkglicense in pkglicenses.values() for item in pkglicense]))
+                    all_licenses = list(set([item.replace('_', ' ') for pkglicense in pkglicenses.values() for item in pkglicense]))
+                    all_licenses.remove('&')
                     # Go back and update the LICENSE value since we have a bit more
                     # information than when that was written out (and we know all apply
                     # vs. there being a choice, so we can join them with &)

Signature · 1 line

2.10.2

Anders Darander

author

2017-03-01 17:20 UTC #12 lore raw

PATCH 7/7 scripts/lib/create_npm: handle Public Domain licenses

In reply to Anders Darander

Rewrite Public Domain as PD, as that's what the place holder in meta/files/common_licenses is called.

Signed-off-by Anders Darander <[email protected]>

---
 scripts/lib/recipetool/create_npm.py | 1 +
 1 file changed, 1 insertion(+)

scripts/lib/recipetool/create_npm.py +1 -0

index 89973d9..95e9732 100644
@@ -60,6 +60,7 @@ class NpmRecipeHandler(RecipeHandler):
                     if license[-1] == ')':
                         license = license[:-1]
                 license = license.replace('MIT/X11', 'MIT')
+                license = license.replace('Public Domain', 'PD')
                 license = license.replace('SEE LICENSE IN EULA',
                                           'SEE-LICENSE-IN-EULA')
         return license

Signature · 1 line

2.10.2