npm.bbclass: work with nodejs 16
cover npm.bbclass: work with nodejs 16
nodejs 16 changed internal caching significantly which breaks the existing npm.bblcass.
Simulate parts of the npm registry and cache data in the way as expected.
This patchset requires an additional one in the oe-meta layer.
Enrico Scholz (6):
npm: replace 'npm pack' call by 'tar czf'
npm: return content of 'package.json' in 'npm_pack'
npm: take 'version' directly from 'package.json'
npm: disable 'audit' + 'fund' lib:npm_registry: initial checkin
npm: use npm_registry to cache package
meta/classes/npm.bbclass | 65 +++++++++----- meta/lib/oe/npm_registry.py | 169 ++++++++++++++++++++++++++++++++++++ 2 files changed, 213 insertions(+), 21 deletions(-) create mode 100644 meta/lib/oe/npm_registry.py
Signature · 1 line
2.36.1
PATCH 1/6 npm: replace 'npm pack' call by 'tar czf'
'npm pack' is a maintainer tool which tries to execute 'prepare' and similar scripts. This fails usually in OE because it requires completely installed 'node_modules'.
Earlier nodejs versions supported an undocumented 'ignore-scripts' option. This has been removed in nodejs 16.
We could patch 'package.json' and remove the unwanted scripts. But this might complicate local workflows (applying patches) and installed packages will contain the modified 'package.json'.
Instead of, package it manually by 'tar czf'. As a sideeffect, 'do_configure' is running much faster now.
--- meta/classes/npm.bbclass | 35 +++++++++++++++++++++++++++++------ 1 file changed, 29 insertions(+), 6 deletions(-)
meta/classes/npm.bbclass +29 -6
@@ -57,13 +57,36 @@ def npm_global_configs(d): configs.append(("cache", d.getVar("NPM_CACHE"))) return configs +## 'npm pack' runs 'prepare' and 'prepack' scripts. Support for +## 'ignore-scripts' which prevents this behavior has been removed +## from nodejs 16. Use simple 'tar' instead of. def npm_pack(env, srcdir, workdir): - """Run 'npm pack' on a specified directory""" - import shlex - cmd = "npm pack %s" % shlex.quote(srcdir) - args = [("ignore-scripts", "true")] - tarball = env.run(cmd, args=args, workdir=workdir).strip("\n") - return os.path.join(workdir, tarball) + """Emulate 'npm pack' on a specified directory""" + import subprocess + import os + import json + + src = os.path.join(srcdir, 'package.json') + with open(src) as f: + j = json.load(f) + + # base does not really matter and is for documentation purposes + # only. But the 'version' part must exist because other parts of + # the bbclass rely on it. + base = j['name'].split('/')[-1] + tarball = os.path.join(workdir, "%s-%s.tgz" % (base, j['version'])); + + # TODO: real 'npm pack' does not include directories while 'tar' + # does. But this does not seem to matter... + subprocess.run(['tar', 'czf', tarball, + '--exclude', './node-modules', + '--exclude-vcs', + '--transform', 's,^\./,package/,', + '--mtime', '1985-10-26T08:15:00.000Z', + '.'], + check = True, cwd = srcdir) + + return tarball python npm_do_configure() { """
Signature · 1 line
2.36.1
Re: [OE-core] [PATCH 1/6] npm: replace 'npm pack' call by 'tar czf'
2022. 05. 19. 12:05 keltezéssel, Enrico Scholz via lists.openembedded.org írta:
Quoted 7 lines
'npm pack' is a maintainer tool which tries to execute 'prepare' and similar scripts. This fails usually in OE because it requires completely installed '
node_modules'.Earlier nodejs versions supported an undocumented 'ignore-scripts' option. This has been removed in nodejs 16.
Quoted 6 lines
We could patch 'package.json' and remove the unwanted scripts. But this might complicate local workflows (applying patches) and installed packages will contain the modified 'package.json'.
Instead of, package it manually by 'tar czf'. As a sideeffect, '
do_configure' is running much faster now.
\o/ Hurray!
Quoted 58 lines
Signed-off-by Enrico Scholz <[email protected]>--- meta/classes/npm.bbclass | 35 +++++++++++++++++++++++++++++------ 1 file changed, 29 insertions(+), 6 deletions(-)meta/classes/npm.bbclass +29
-6@@ -57,13 +57,36 @@ def npm_global_configs(d): configs.append(("cache", d.getVar("NPM_CACHE"))) return configs +## 'npm pack' runs 'prepare' and 'prepack' scripts. Support for +## 'ignore-scripts' which prevents this behavior has been removed +## from nodejs 16. Use simple 'tar' instead of. def npm_pack(env, srcdir, workdir): - """Run 'npm pack' on a specified directory""" - import shlex - cmd = "npm pack %s" % shlex.quote(srcdir) - args = [("ignore-scripts", "true")] - tarball = env.run(cmd, args=args, workdir=workdir).strip("\n") - return os.path.join(workdir, tarball) + """Emulate 'npm pack' on a specified directory""" + import subprocess + import os + import json + + src = os.path.join(srcdir, 'package.json') + with open(src) as f: + j = json.load(f) + + # base does not really matter and is for documentation purposes + # only. But the 'version' part must exist because other parts of + # the bbclass rely on it. + base = j['name'].split('/')[-1] + tarball = os.path.join(workdir, "%s-%s.tgz" % (base, j['version'])); + + # TODO: real 'npm pack' does not include directories while 'tar' + # does. But this does not seem to matter... + subprocess.run(['tar', 'czf', tarball, + '--exclude', './node-modules', + '--exclude-vcs', + '--transform', 's,^\./,package/,', + '--mtime', '1985-10-26T08:15:00.000Z', + '.'], + check = True, cwd = srcdir) + + return tarball python npm_do_configure() { """
PATCH 2/6 npm: return content of 'package.json' in 'npm_pack'
In reply to Enrico ScholzWe have to read 'package.json' to calculate the name of the tarball. This content is interesting for later patches.
--- meta/classes/npm.bbclass | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
meta/classes/npm.bbclass +4 -4
@@ -86,7 +86,7 @@ def npm_pack(env, srcdir, workdir): '.'], check = True, cwd = srcdir) - return tarball + return (tarball, j) python npm_do_configure() { """
@@ -186,7 +186,7 @@ python npm_do_configure() { with tempfile.TemporaryDirectory() as tmpdir: # Add the dependency to the npm cache destdir = os.path.join(d.getVar("S"), destsuffix) - tarball = npm_pack(env, destdir, tmpdir) + (tarball, pkg) = npm_pack(env, destdir, tmpdir) _npm_cache_add(tarball) # Add its signature to the cached shrinkwrap dep = _npmsw_dependency_dict(cached_shrinkwrap, deptree)
@@ -207,7 +207,7 @@ python npm_do_configure() { # Configure the main package with tempfile.TemporaryDirectory() as tmpdir: - tarball = npm_pack(env, d.getVar("S"), tmpdir) + (tarball, _) = npm_pack(env, d.getVar("S"), tmpdir) npm_unpack(tarball, d.getVar("NPM_PACKAGE"), d) # Configure the cached manifest file and cached shrinkwrap file
@@ -280,7 +280,7 @@ python npm_do_compile() { args.append(("build-from-source", "true")) # Pack and install the main package - tarball = npm_pack(env, d.getVar("NPM_PACKAGE"), tmpdir) + (tarball, _) = npm_pack(env, d.getVar("NPM_PACKAGE"), tmpdir) cmd = "npm install %s %s" % (shlex.quote(tarball), d.getVar("EXTRA_OENPM")) env.run(cmd, args=args) }
Signature · 1 line
2.36.1
PATCH 3/6 npm: take 'version' directly from 'package.json'
In reply to Enrico ScholzWe know the content of 'package.json' from earlier patches; there is no need to parse the tarball name to extract the version.
--- meta/classes/npm.bbclass | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-)
meta/classes/npm.bbclass +1 -6
@@ -125,11 +125,6 @@ python npm_do_configure() { sha512 = bb.utils.sha512_file(tarball) return "sha512-" + base64.b64encode(bytes.fromhex(sha512)).decode() - def _npm_version(tarball): - """Return the version of a specified tarball""" - regex = r"-(\d+\.\d+\.\d+(-.*)?(\+.*)?)\.tgz" - return re.search(regex, tarball).group(1) - def _npmsw_dependency_dict(orig, deptree): """ Return the sub dictionary in the 'orig' dictionary corresponding to the
@@ -190,7 +185,7 @@ python npm_do_configure() { _npm_cache_add(tarball) # Add its signature to the cached shrinkwrap dep = _npmsw_dependency_dict(cached_shrinkwrap, deptree) - dep["version"] = _npm_version(tarball) + dep["version"] = pkg['version'] dep["integrity"] = _npm_integrity(tarball) if params.get("dev", False): dep["dev"] = True
Signature · 1 line
2.36.1
PATCH 4/6 npm: disable 'audit' + 'fund'
In reply to Enrico Scholz'audit' can cause extra network traffic; 'fund' is not needed.
--- meta/classes/npm.bbclass | 2 ++ 1 file changed, 2 insertions(+)
meta/classes/npm.bbclass +2 -0
@@ -53,6 +53,8 @@ def npm_global_configs(d): # Ensure no network access is done configs.append(("offline", "true")) configs.append(("proxy", "http://invalid")) + configs.append(("funds", False)) + configs.append(("audit", False)) # Configure the cache directory configs.append(("cache", d.getVar("NPM_CACHE"))) return configs
Signature · 1 line
2.36.1
Re: [OE-core] [PATCH 4/6] npm: disable 'audit' + 'fund'
Il giorno Thu, 19 May 2022 12:05:54 +0200
"Enrico Scholz via lists.openembedded.org" <[email protected]> ha scritto:
Quoted 12 lines
'audit' can cause extra network traffic; 'fund' is not needed.
Signed-off-by Enrico Scholz <[email protected]>--- meta/classes/npm.bbclass | 2 ++ 1 file changed, 2 insertions(+)meta/classes/npm.bbclass +0
-0
meta/classes/npm.bbclass def npm_global_configs(d): · line 56
Quoted 4 lines
meta/classes/npm.bbclass +1
-0@@ -53,6 +53,8 @@ def npm_global_configs(d): # Ensure no network access is done configs.append(("offline", "true")) configs.append(("proxy", "http://invalid")) + configs.append(("funds", False))
'fund' or 'funds'?
Signature · 3 lines
Luca Ceresoli, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH v2 4/6 npm: disable 'audit' + 'fund'
In reply to Enrico Scholz'audit' can cause extra network traffic; 'fund' is not needed.
--- meta/classes/npm.bbclass | 2 ++ 1 file changed, 2 insertions(+)
meta/classes/npm.bbclass +2 -0
@@ -53,6 +53,8 @@ def npm_global_configs(d): # Ensure no network access is done configs.append(("offline", "true")) configs.append(("proxy", "http://invalid")) + configs.append(("fund", False)) + configs.append(("audit", False)) # Configure the cache directory configs.append(("cache", d.getVar("NPM_CACHE"))) return configs
Signature · 1 line
2.36.1
PATCH 5/6 lib:npm_registry: initial checkin
In reply to Enrico ScholzHelper module to:
- generate meta information from package.json content. This data has a format as provided by https://registry.npmjs.org
- put this meta information and the corresponding tarball in the nodejs cache. This uses an external, nodejs version specific helper script (oe-npm-cache) shipped in oe-meta
To avoid further nodejs version dependencies, future versions of this module might omit the caching completely and serve meta information and tarball by an http server.
--- meta/lib/oe/npm_registry.py | 169 ++++++++++++++++++++++++++++++++++++ 1 file changed, 169 insertions(+) create mode 100644 meta/lib/oe/npm_registry.py
meta/lib/oe/npm_registry.py new file +169 -0
@@ -0,0 +1,169 @@ +import bb +import json +import subprocess + +_ALWAYS_SAFE = frozenset('ABCDEFGHIJKLMNOPQRSTUVWXYZ' + 'abcdefghijklmnopqrstuvwxyz' + '0123456789' + '_.-~') + +MISSING_OK = object() + +REGISTRY = "https://registry.npmjs.org" + +# we can not use urllib.parse here because npm expects lowercase +# hex-chars but urllib generates uppercase ones +def uri_quote(s, safe = '/'): + res = "" + safe_set = set(safe) + for c in s: + if c in _ALWAYS_SAFE or c in safe_set: + res += c + else: + res += '%%%02x' % ord(c) + return res + +class PackageJson: + def __init__(self, spec): + self.__spec = spec + + @property + def name(self): + return self.__spec['name'] + + @property + def version(self): + return self.__spec['version'] + + @property + def empty_manifest(self): + return { + 'name': self.name, + 'description': self.__spec.get('description', ''), + 'versions': {}, + } + + def base_filename(self): + return uri_quote(self.name, safe = '@') + + def as_manifest_entry(self, tarball_uri): + res = {} + + ## NOTE: 'npm install' requires more than basic meta information; + ## e.g. it takes 'bin' from this manifest entry but not the actual + ## 'package.json' + for (idx,dflt) in [('name', None), + ('description', ""), + ('version', None), + ('bin', MISSING_OK), + ('man', MISSING_OK), + ('scripts', MISSING_OK), + ('directories', MISSING_OK), + ('dependencies', MISSING_OK), + ('devDependencies', MISSING_OK), + ('optionalDependencies', MISSING_OK), + ('license', "unknown")]: + if idx in self.__spec: + res[idx] = self.__spec[idx] + elif dflt == MISSING_OK: + pass + elif dflt != None: + res[idx] = dflt + else: + raise Exception("%s-%s: missing key %s" % (self.name, + self.version, + idx)) + + res['dist'] = { + 'tarball': tarball_uri, + } + + return res + +class ManifestImpl: + def __init__(self, base_fname, spec): + self.__base = base_fname + self.__spec = spec + + def load(self): + try: + with open(self.filename, "r") as f: + res = json.load(f) + except IOError: + res = self.__spec.empty_manifest + + return res + + def save(self, meta): + with open(self.filename, "w") as f: + json.dump(meta, f, indent = 2) + + @property + def filename(self): + return self.__base + ".meta" + +class Manifest: + def __init__(self, base_fname, spec): + self.__base = base_fname + self.__spec = spec + self.__lockf = None + self.__impl = None + + def __enter__(self): + self.__lockf = bb.utils.lockfile(self.__base + ".lock") + self.__impl = ManifestImpl(self.__base, self.__spec) + return self.__impl + + def __exit__(self, exc_type, exc_val, exc_tb): + bb.utils.unlockfile(self.__lockf) + +class NpmCache: + def __init__(self, cache): + self.__cache = cache + + @property + def path(self): + return self.__cache + + def run(self, type, key, fname): + subprocess.run(['oe-npm-cache', self.__cache, type, key, fname], + check = True) + +class NpmRegistry: + def __init__(self, path, cache): + self.__path = path + self.__cache = NpmCache(cache + '/_cacache') + bb.utils.mkdirhier(self.__path) + bb.utils.mkdirhier(self.__cache.path) + + @staticmethod + ## This function is critical and must match nodejs expectations + def _meta_uri(spec): + return REGISTRY + '/' + uri_quote(spec.name, safe = '@') + + @staticmethod + ## Exact return value does not matter; just make it look like a + ## usual registry url + def _tarball_uri(spec): + return '%s/%s/-/%s-%s.tgz' % (REGISTRY, + uri_quote(spec.name, safe = '@'), + uri_quote(spec.name, safe = '@/'), + spec.version) + + def add_pkg(self, tarball, pkg_json): + pkg_json = PackageJson(pkg_json) + base = os.path.join(self.__path, pkg_json.base_filename()) + + with Manifest(base, pkg_json) as manifest: + meta = manifest.load() + tarball_uri = self._tarball_uri(pkg_json) + + meta['versions'][pkg_json.version] = pkg_json.as_manifest_entry(tarball_uri) + + manifest.save(meta) + + ## Cache entries are a little bit dependent on the nodejs + ## version; version specific cache implementation must + ## mitigate differences + self.__cache.run('meta', self._meta_uri(pkg_json), manifest.filename); + self.__cache.run('tgz', tarball_uri, tarball);
Signature · 1 line
2.36.1
PATCH 6/6 npm: use npm_registry to cache package
In reply to Enrico ScholzWith nodejs 16, the simple 'npm cache add' approach does not work anymore because its fetcher implementation downloads also meta information from the registry.
We have to generate these information and add them to the cache. There is no direct support in 'npm' for task so we have to implement it manually.
This implementation consists of a openembedded python module (in oe-core) and a nodejs version specific helper (in oe-meta).
--- meta/classes/npm.bbclass | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
meta/classes/npm.bbclass +9 -6
@@ -19,7 +19,7 @@ inherit python3native -DEPENDS:prepend = "nodejs-native " +DEPENDS:prepend = "nodejs-native nodejs-oe-cache-native " RDEPENDS:${PN}:append:class-target = " nodejs" EXTRA_OENPM = ""
@@ -46,6 +46,7 @@ NPM_ARCH ?= "${@npm_target_arch_map(d.getVar("TARGET_ARCH"))}" NPM_PACKAGE = "${WORKDIR}/npm-package" NPM_CACHE = "${WORKDIR}/npm-cache" NPM_BUILD = "${WORKDIR}/npm-build" +NPM_REGISTRY = "${WORKDIR}/npm-registry" def npm_global_configs(d): """Get the npm global configuration"""
@@ -111,16 +112,18 @@ python npm_do_configure() { from bb.fetch2.npm import npm_unpack from bb.fetch2.npmsw import foreach_dependencies from bb.progress import OutOfProgressHandler + from oe.npm_registry import NpmRegistry bb.utils.remove(d.getVar("NPM_CACHE"), recurse=True) bb.utils.remove(d.getVar("NPM_PACKAGE"), recurse=True) env = NpmEnvironment(d, configs=npm_global_configs(d)) + registry = NpmRegistry(d.getVar('NPM_REGISTRY'), d.getVar('NPM_CACHE')) - def _npm_cache_add(tarball): - """Run 'npm cache add' for a specified tarball""" - cmd = "npm cache add %s" % shlex.quote(tarball) - env.run(cmd) + def _npm_cache_add(tarball, pkg): + """Add tarball to local registry and register it in the + cache""" + registry.add_pkg(tarball, pkg) def _npm_integrity(tarball): """Return the npm integrity of a specified tarball"""
@@ -184,7 +187,7 @@ python npm_do_configure() { # Add the dependency to the npm cache destdir = os.path.join(d.getVar("S"), destsuffix) (tarball, pkg) = npm_pack(env, destdir, tmpdir) - _npm_cache_add(tarball) + _npm_cache_add(tarball, pkg) # Add its signature to the cached shrinkwrap dep = _npmsw_dependency_dict(cached_shrinkwrap, deptree) dep["version"] = pkg['version']
Signature · 1 line
2.36.1
"Enrico Scholz via lists.openembedded.org"
Quoted 1 line · <[email protected]> writes:
This patchset requires an additional one in the oe-meta layer.
for reference: https://lists.openembedded.org/g/openembedded-devel/message/97186
This seems to solve my problem with the quicktype recipe [1]. Thank you very much for working in this.
[1] https://lore.kernel.org/all/CANNYZj-V8O1-USTdtOYKi5hB80QjxQZH2d2=YbJ-4Y2sm5BjoA@mail.gmail.com/T/
Quoted 22 lines · Enrico Scholz via lists.openembedded.org
nodejs 16 changed internal caching significantly which breaks the existing npm.bblcass.
Simulate parts of the npm registry and cache data in the way as expected.
This patchset requires an additional one in the oe-meta layer.
Enrico Scholz (6):
npm: replace 'npm pack' call by 'tar czf'
npm: return content of 'package.json' in 'npm_pack'
npm: take 'version' directly from 'package.json'
npm: disable 'audit' + 'fund' lib:npm_registry: initial checkin
npm: usenpm_registryto cache packagemeta/classes/npm.bbclass | 65 +++++++++----- meta/lib/oe/npm_registry.py | 169 ++++++++++++++++++++++++++++++++++++ 2 files changed, 213 insertions(+), 21 deletions(-) create mode 100644 meta/lib/oe/npm_registry.py