Skip to contentSkip to conversation

npm.bbclass: work with nodejs 16

PATCH In review Enrico Scholz 4 participants 12 messages May 19, 2022 – May 23, 2022

lore
Share & tools
lore
#0 lore raw

cover npm.bbclass: work with nodejs 16

nodejs 16 changed internal caching significantly which breaks the existing npm.bblcass.

Simulate parts of the npm registry and cache data in the way as expected.

This patchset requires an additional one in the oe-meta layer.

Enrico Scholz (6):
npm: replace 'npm pack' call by 'tar czf'
npm: return content of 'package.json' in 'npm_pack'
npm: take 'version' directly from 'package.json'
npm: disable 'audit' + 'fund' lib:npm_registry: initial checkin
npm: use npm_registry to cache package

 meta/classes/npm.bbclass    |  65 +++++++++-----
 meta/lib/oe/npm_registry.py | 169 ++++++++++++++++++++++++++++++++++++
 2 files changed, 213 insertions(+), 21 deletions(-)
 create mode 100644 meta/lib/oe/npm_registry.py
Signature · 1 line
2.36.1
#1 lore raw

PATCH 1/6 npm: replace 'npm pack' call by 'tar czf'

'npm pack' is a maintainer tool which tries to execute 'prepare' and similar scripts. This fails usually in OE because it requires completely installed 'node_modules'.

Earlier nodejs versions supported an undocumented 'ignore-scripts' option. This has been removed in nodejs 16.

We could patch 'package.json' and remove the unwanted scripts. But this might complicate local workflows (applying patches) and installed packages will contain the modified 'package.json'.

Instead of, package it manually by 'tar czf'. As a sideeffect, 'do_configure' is running much faster now.

---
 meta/classes/npm.bbclass | 35 +++++++++++++++++++++++++++++------
 1 file changed, 29 insertions(+), 6 deletions(-)
meta/classes/npm.bbclass +29 -6
index ba50fcac20..91f8f36b0d 100644
@@ -57,13 +57,36 @@ def npm_global_configs(d):
     configs.append(("cache", d.getVar("NPM_CACHE")))
     return configs
 
+## 'npm pack' runs 'prepare' and 'prepack' scripts. Support for
+## 'ignore-scripts' which prevents this behavior has been removed
+## from nodejs 16.  Use simple 'tar' instead of.
 def npm_pack(env, srcdir, workdir):
-    """Run 'npm pack' on a specified directory"""
-    import shlex
-    cmd = "npm pack %s" % shlex.quote(srcdir)
-    args = [("ignore-scripts", "true")]
-    tarball = env.run(cmd, args=args, workdir=workdir).strip("\n")
-    return os.path.join(workdir, tarball)
+    """Emulate 'npm pack' on a specified directory"""
+    import subprocess
+    import os
+    import json
+
+    src = os.path.join(srcdir, 'package.json')
+    with open(src) as f:
+        j = json.load(f)
+
+    # base does not really matter and is for documentation purposes
+    # only.  But the 'version' part must exist because other parts of
+    # the bbclass rely on it.
+    base = j['name'].split('/')[-1]
+    tarball = os.path.join(workdir, "%s-%s.tgz" % (base, j['version']));
+
+    # TODO: real 'npm pack' does not include directories while 'tar'
+    # does.  But this does not seem to matter...
+    subprocess.run(['tar', 'czf', tarball,
+                    '--exclude', './node-modules',
+                    '--exclude-vcs',
+                    '--transform', 's,^\./,package/,',
+                    '--mtime', '1985-10-26T08:15:00.000Z',
+                    '.'],
+                   check = True, cwd = srcdir)
+
+    return tarball
 
 python npm_do_configure() {
     """
Signature · 1 line
2.36.1
#2 lore raw

Re: [OE-core] [PATCH 1/6] npm: replace 'npm pack' call by 'tar czf'

2022. 05. 19. 12:05 keltezéssel, Enrico Scholz via lists.openembedded.org írta:

Quoted 7 lines

'npm pack' is a maintainer tool which tries to execute 'prepare' and similar scripts. This fails usually in OE because it requires completely installed 'node_modules'.

Earlier nodejs versions supported an undocumented 'ignore-scripts' option. This has been removed in nodejs 16.

Quoted 6 lines

We could patch 'package.json' and remove the unwanted scripts. But this might complicate local workflows (applying patches) and installed packages will contain the modified 'package.json'.

Instead of, package it manually by 'tar czf'. As a sideeffect, 'do_configure' is running much faster now.

\o/ Hurray!

Quoted 58 lines
---
  meta/classes/npm.bbclass | 35 +++++++++++++++++++++++++++++------
  1 file changed, 29 insertions(+), 6 deletions(-)
meta/classes/npm.bbclass +29 -6
index ba50fcac20..91f8f36b0d 100644
@@ -57,13 +57,36 @@ def npm_global_configs(d):
      configs.append(("cache", d.getVar("NPM_CACHE")))
      return configs
 
+## 'npm pack' runs 'prepare' and 'prepack' scripts. Support for
+## 'ignore-scripts' which prevents this behavior has been removed
+## from nodejs 16.  Use simple 'tar' instead of.
  def npm_pack(env, srcdir, workdir):
-    """Run 'npm pack' on a specified directory"""
-    import shlex
-    cmd = "npm pack %s" % shlex.quote(srcdir)
-    args = [("ignore-scripts", "true")]
-    tarball = env.run(cmd, args=args, workdir=workdir).strip("\n")
-    return os.path.join(workdir, tarball)
+    """Emulate 'npm pack' on a specified directory"""
+    import subprocess
+    import os
+    import json
+
+    src = os.path.join(srcdir, 'package.json')
+    with open(src) as f:
+        j = json.load(f)
+
+    # base does not really matter and is for documentation purposes
+    # only.  But the 'version' part must exist because other parts of
+    # the bbclass rely on it.
+    base = j['name'].split('/')[-1]
+    tarball = os.path.join(workdir, "%s-%s.tgz" % (base, j['version']));
+
+    # TODO: real 'npm pack' does not include directories while 'tar'
+    # does.  But this does not seem to matter...
+    subprocess.run(['tar', 'czf', tarball,
+                    '--exclude', './node-modules',
+                    '--exclude-vcs',
+                    '--transform', 's,^\./,package/,',
+                    '--mtime', '1985-10-26T08:15:00.000Z',
+                    '.'],
+                   check = True, cwd = srcdir)
+
+    return tarball
 
  python npm_do_configure() {
      """
#3 lore raw

PATCH 2/6 npm: return content of 'package.json' in 'npm_pack'

In reply to Enrico Scholz

We have to read 'package.json' to calculate the name of the tarball. This content is interesting for later patches.

---
 meta/classes/npm.bbclass | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
meta/classes/npm.bbclass +4 -4
index 91f8f36b0d..014f793450 100644
@@ -86,7 +86,7 @@ def npm_pack(env, srcdir, workdir):
                     '.'],
                    check = True, cwd = srcdir)
 
-    return tarball
+    return (tarball, j)
 
 python npm_do_configure() {
     """
@@ -186,7 +186,7 @@ python npm_do_configure() {
         with tempfile.TemporaryDirectory() as tmpdir:
             # Add the dependency to the npm cache
             destdir = os.path.join(d.getVar("S"), destsuffix)
-            tarball = npm_pack(env, destdir, tmpdir)
+            (tarball, pkg) = npm_pack(env, destdir, tmpdir)
             _npm_cache_add(tarball)
             # Add its signature to the cached shrinkwrap
             dep = _npmsw_dependency_dict(cached_shrinkwrap, deptree)
@@ -207,7 +207,7 @@ python npm_do_configure() {
 
     # Configure the main package
     with tempfile.TemporaryDirectory() as tmpdir:
-        tarball = npm_pack(env, d.getVar("S"), tmpdir)
+        (tarball, _) = npm_pack(env, d.getVar("S"), tmpdir)
         npm_unpack(tarball, d.getVar("NPM_PACKAGE"), d)
 
     # Configure the cached manifest file and cached shrinkwrap file
@@ -280,7 +280,7 @@ python npm_do_compile() {
         args.append(("build-from-source", "true"))
 
         # Pack and install the main package
-        tarball = npm_pack(env, d.getVar("NPM_PACKAGE"), tmpdir)
+        (tarball, _) = npm_pack(env, d.getVar("NPM_PACKAGE"), tmpdir)
         cmd = "npm install %s %s" % (shlex.quote(tarball), d.getVar("EXTRA_OENPM"))
         env.run(cmd, args=args)
 }
Signature · 1 line
2.36.1
#4 lore raw

PATCH 3/6 npm: take 'version' directly from 'package.json'

In reply to Enrico Scholz

We know the content of 'package.json' from earlier patches; there is no need to parse the tarball name to extract the version.

---
 meta/classes/npm.bbclass | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)
meta/classes/npm.bbclass +1 -6
index 014f793450..11c80a738e 100644
@@ -125,11 +125,6 @@ python npm_do_configure() {
         sha512 = bb.utils.sha512_file(tarball)
         return "sha512-" + base64.b64encode(bytes.fromhex(sha512)).decode()
 
-    def _npm_version(tarball):
-        """Return the version of a specified tarball"""
-        regex = r"-(\d+\.\d+\.\d+(-.*)?(\+.*)?)\.tgz"
-        return re.search(regex, tarball).group(1)
-
     def _npmsw_dependency_dict(orig, deptree):
         """
         Return the sub dictionary in the 'orig' dictionary corresponding to the
@@ -190,7 +185,7 @@ python npm_do_configure() {
             _npm_cache_add(tarball)
             # Add its signature to the cached shrinkwrap
             dep = _npmsw_dependency_dict(cached_shrinkwrap, deptree)
-            dep["version"] = _npm_version(tarball)
+            dep["version"] = pkg['version']
             dep["integrity"] = _npm_integrity(tarball)
             if params.get("dev", False):
                 dep["dev"] = True
Signature · 1 line
2.36.1
#5 lore raw

PATCH 4/6 npm: disable 'audit' + 'fund'

In reply to Enrico Scholz

'audit' can cause extra network traffic; 'fund' is not needed.

---
 meta/classes/npm.bbclass | 2 ++
 1 file changed, 2 insertions(+)
meta/classes/npm.bbclass +2 -0
index 11c80a738e..1e41072116 100644
@@ -53,6 +53,8 @@ def npm_global_configs(d):
     # Ensure no network access is done
     configs.append(("offline", "true"))
     configs.append(("proxy", "http://invalid"))
+    configs.append(("funds", False))
+    configs.append(("audit", False))
     # Configure the cache directory
     configs.append(("cache", d.getVar("NPM_CACHE")))
     return configs
Signature · 1 line
2.36.1
#6 lore raw

Re: [OE-core] [PATCH 4/6] npm: disable 'audit' + 'fund'

Il giorno Thu, 19 May 2022 12:05:54 +0200
"Enrico Scholz via lists.openembedded.org" <[email protected]> ha scritto:

Quoted 12 lines

'audit' can cause extra network traffic; 'fund' is not needed.

---
 meta/classes/npm.bbclass | 2 ++
 1 file changed, 2 insertions(+)
meta/classes/npm.bbclass +0 -0
index 11c80a738e..1e41072116 100644

meta/classes/npm.bbclass def npm_global_configs(d): · line 56

Quoted 4 lines
meta/classes/npm.bbclass +1 -0
@@ -53,6 +53,8 @@ def npm_global_configs(d):
     # Ensure no network access is done
     configs.append(("offline", "true"))
     configs.append(("proxy", "http://invalid"))
+    configs.append(("funds", False))

'fund' or 'funds'?

Signature · 3 lines
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#7 lore raw

PATCH v2 4/6 npm: disable 'audit' + 'fund'

In reply to Enrico Scholz

'audit' can cause extra network traffic; 'fund' is not needed.

---
 meta/classes/npm.bbclass | 2 ++
 1 file changed, 2 insertions(+)
meta/classes/npm.bbclass +2 -0
index 11c80a738e..b613c8e8f5 100644
@@ -53,6 +53,8 @@ def npm_global_configs(d):
     # Ensure no network access is done
     configs.append(("offline", "true"))
     configs.append(("proxy", "http://invalid"))
+    configs.append(("fund", False))
+    configs.append(("audit", False))
     # Configure the cache directory
     configs.append(("cache", d.getVar("NPM_CACHE")))
     return configs
Signature · 1 line
2.36.1
#8 lore raw

PATCH 5/6 lib:npm_registry: initial checkin

In reply to Enrico Scholz

Helper module to:

- generate meta information from package.json content. This data has a format as provided by https://registry.npmjs.org

- put this meta information and the corresponding tarball in the nodejs cache. This uses an external, nodejs version specific helper script (oe-npm-cache) shipped in oe-meta

To avoid further nodejs version dependencies, future versions of this module might omit the caching completely and serve meta information and tarball by an http server.

---
 meta/lib/oe/npm_registry.py | 169 ++++++++++++++++++++++++++++++++++++
 1 file changed, 169 insertions(+)
 create mode 100644 meta/lib/oe/npm_registry.py
meta/lib/oe/npm_registry.py new file +169 -0
new file mode 100644
index 0000000000..96c0affb45
@@ -0,0 +1,169 @@
+import bb
+import json
+import subprocess
+
+_ALWAYS_SAFE = frozenset('ABCDEFGHIJKLMNOPQRSTUVWXYZ'
+                         'abcdefghijklmnopqrstuvwxyz'
+                         '0123456789'
+                         '_.-~')
+
+MISSING_OK = object()
+
+REGISTRY = "https://registry.npmjs.org"
+
+# we can not use urllib.parse here because npm expects lowercase
+# hex-chars but urllib generates uppercase ones
+def uri_quote(s, safe = '/'):
+    res = ""
+    safe_set = set(safe)
+    for c in s:
+        if c in _ALWAYS_SAFE or c in safe_set:
+            res += c
+        else:
+            res += '%%%02x' % ord(c)
+    return res
+
+class PackageJson:
+    def __init__(self, spec):
+        self.__spec = spec
+
+    @property
+    def name(self):
+        return self.__spec['name']
+
+    @property
+    def version(self):
+        return self.__spec['version']
+
+    @property
+    def empty_manifest(self):
+        return {
+            'name': self.name,
+            'description': self.__spec.get('description', ''),
+            'versions': {},
+        }
+
+    def base_filename(self):
+        return uri_quote(self.name, safe = '@')
+
+    def as_manifest_entry(self, tarball_uri):
+        res = {}
+
+        ## NOTE: 'npm install' requires more than basic meta information;
+        ## e.g. it takes 'bin' from this manifest entry but not the actual
+        ## 'package.json'
+        for (idx,dflt) in [('name', None),
+                           ('description', ""),
+                           ('version', None),
+                           ('bin', MISSING_OK),
+                           ('man', MISSING_OK),
+                           ('scripts', MISSING_OK),
+                           ('directories', MISSING_OK),
+                           ('dependencies', MISSING_OK),
+                           ('devDependencies', MISSING_OK),
+                           ('optionalDependencies', MISSING_OK),
+                           ('license', "unknown")]:
+            if idx in self.__spec:
+                res[idx] = self.__spec[idx]
+            elif dflt == MISSING_OK:
+                pass
+            elif dflt != None:
+                res[idx] = dflt
+            else:
+                raise Exception("%s-%s: missing key %s" % (self.name,
+                                                           self.version,
+                                                           idx))
+
+        res['dist'] = {
+            'tarball': tarball_uri,
+        }
+
+        return res
+
+class ManifestImpl:
+    def __init__(self, base_fname, spec):
+        self.__base = base_fname
+        self.__spec = spec
+
+    def load(self):
+        try:
+            with open(self.filename, "r") as f:
+                res = json.load(f)
+        except IOError:
+            res = self.__spec.empty_manifest
+
+        return res
+
+    def save(self, meta):
+        with open(self.filename, "w") as f:
+            json.dump(meta, f, indent = 2)
+
+    @property
+    def filename(self):
+        return self.__base + ".meta"
+
+class Manifest:
+    def __init__(self, base_fname, spec):
+        self.__base = base_fname
+        self.__spec = spec
+        self.__lockf = None
+        self.__impl = None
+
+    def __enter__(self):
+        self.__lockf = bb.utils.lockfile(self.__base + ".lock")
+        self.__impl  = ManifestImpl(self.__base, self.__spec)
+        return self.__impl
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        bb.utils.unlockfile(self.__lockf)
+
+class NpmCache:
+    def __init__(self, cache):
+        self.__cache = cache
+
+    @property
+    def path(self):
+        return self.__cache
+
+    def run(self, type, key, fname):
+        subprocess.run(['oe-npm-cache', self.__cache, type, key, fname],
+                       check = True)
+
+class NpmRegistry:
+    def __init__(self, path, cache):
+        self.__path = path
+        self.__cache = NpmCache(cache + '/_cacache')
+        bb.utils.mkdirhier(self.__path)
+        bb.utils.mkdirhier(self.__cache.path)
+
+    @staticmethod
+    ## This function is critical and must match nodejs expectations
+    def _meta_uri(spec):
+        return REGISTRY + '/' + uri_quote(spec.name, safe = '@')
+
+    @staticmethod
+    ## Exact return value does not matter; just make it look like a
+    ## usual registry url
+    def _tarball_uri(spec):
+        return '%s/%s/-/%s-%s.tgz' % (REGISTRY,
+                                      uri_quote(spec.name, safe = '@'),
+                                      uri_quote(spec.name, safe = '@/'),
+                                      spec.version)
+
+    def add_pkg(self, tarball, pkg_json):
+        pkg_json = PackageJson(pkg_json)
+        base = os.path.join(self.__path, pkg_json.base_filename())
+
+        with Manifest(base, pkg_json) as manifest:
+            meta = manifest.load()
+            tarball_uri = self._tarball_uri(pkg_json)
+
+            meta['versions'][pkg_json.version] = pkg_json.as_manifest_entry(tarball_uri)
+
+            manifest.save(meta)
+
+            ## Cache entries are a little bit dependent on the nodejs
+            ## version; version specific cache implementation must
+            ## mitigate differences
+            self.__cache.run('meta', self._meta_uri(pkg_json), manifest.filename);
+            self.__cache.run('tgz',  tarball_uri, tarball);
Signature · 1 line
2.36.1
#9 lore raw

PATCH 6/6 npm: use npm_registry to cache package

In reply to Enrico Scholz

With nodejs 16, the simple 'npm cache add' approach does not work anymore because its fetcher implementation downloads also meta information from the registry.

We have to generate these information and add them to the cache. There is no direct support in 'npm' for task so we have to implement it manually.

This implementation consists of a openembedded python module (in oe-core) and a nodejs version specific helper (in oe-meta).

---
 meta/classes/npm.bbclass | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
meta/classes/npm.bbclass +9 -6
index 1e41072116..f3c3e2f5b7 100644
@@ -19,7 +19,7 @@
 
 inherit python3native
 
-DEPENDS:prepend = "nodejs-native "
+DEPENDS:prepend = "nodejs-native nodejs-oe-cache-native "
 RDEPENDS:${PN}:append:class-target = " nodejs"
 
 EXTRA_OENPM = ""
@@ -46,6 +46,7 @@ NPM_ARCH ?= "${@npm_target_arch_map(d.getVar("TARGET_ARCH"))}"
 NPM_PACKAGE = "${WORKDIR}/npm-package"
 NPM_CACHE = "${WORKDIR}/npm-cache"
 NPM_BUILD = "${WORKDIR}/npm-build"
+NPM_REGISTRY = "${WORKDIR}/npm-registry"
 
 def npm_global_configs(d):
     """Get the npm global configuration"""
@@ -111,16 +112,18 @@ python npm_do_configure() {
     from bb.fetch2.npm import npm_unpack
     from bb.fetch2.npmsw import foreach_dependencies
     from bb.progress import OutOfProgressHandler
+    from oe.npm_registry import NpmRegistry
 
     bb.utils.remove(d.getVar("NPM_CACHE"), recurse=True)
     bb.utils.remove(d.getVar("NPM_PACKAGE"), recurse=True)
 
     env = NpmEnvironment(d, configs=npm_global_configs(d))
+    registry = NpmRegistry(d.getVar('NPM_REGISTRY'), d.getVar('NPM_CACHE'))
 
-    def _npm_cache_add(tarball):
-        """Run 'npm cache add' for a specified tarball"""
-        cmd = "npm cache add %s" % shlex.quote(tarball)
-        env.run(cmd)
+    def _npm_cache_add(tarball, pkg):
+        """Add tarball to local registry and register it in the
+           cache"""
+        registry.add_pkg(tarball, pkg)
 
     def _npm_integrity(tarball):
         """Return the npm integrity of a specified tarball"""
@@ -184,7 +187,7 @@ python npm_do_configure() {
             # Add the dependency to the npm cache
             destdir = os.path.join(d.getVar("S"), destsuffix)
             (tarball, pkg) = npm_pack(env, destdir, tmpdir)
-            _npm_cache_add(tarball)
+            _npm_cache_add(tarball, pkg)
             # Add its signature to the cached shrinkwrap
             dep = _npmsw_dependency_dict(cached_shrinkwrap, deptree)
             dep["version"] = pkg['version']
Signature · 1 line
2.36.1
#11 lore raw
In reply to Enrico Scholz

This seems to solve my problem with the quicktype recipe [1]. Thank you very much for working in this.

[1] https://lore.kernel.org/all/CANNYZj-V8O1-USTdtOYKi5hB80QjxQZH2d2=YbJ-4Y2sm5BjoA@mail.gmail.com/T/

Quoted 22 lines · Enrico Scholz via lists.openembedded.org

nodejs 16 changed internal caching significantly which breaks the existing npm.bblcass.

Simulate parts of the npm registry and cache data in the way as expected.

This patchset requires an additional one in the oe-meta layer.

Enrico Scholz (6):
npm: replace 'npm pack' call by 'tar czf'
npm: return content of 'package.json' in 'npm_pack'
npm: take 'version' directly from 'package.json'
npm: disable 'audit' + 'fund' lib:npm_registry: initial checkin
npm: use npm_registry to cache package

 meta/classes/npm.bbclass    |  65 +++++++++-----
 meta/lib/oe/npm_registry.py | 169 ++++++++++++++++++++++++++++++++++++
 2 files changed, 213 insertions(+), 21 deletions(-)
 create mode 100644 meta/lib/oe/npm_registry.py