fetch/npm: no npm executable, and an event log fix
cover fetch/npm: no npm executable, and an event log fix
Hi,
The OE-Core series "npm: list the dependencies in SRC_URI" needs these two patches. I send that series at the same time. The two patches do not depend on each other.
1. cooker: "recipetool create" for npm runs two builds in one server. The second build fails to write the event log, because sstate.bbclass stores compiled regular expressions in the datastore. The recipetool selftest of the OE-Core series shows this failure.
2. fetch/npm: Alexander asked for this in the RFC thread [1]. The npm fetcher must work without the npm executable, as the crate fetcher works without cargo. The npm tests now run without npm, and new tests fetch a package from a premirror without network access.
Tests: bitbake-selftest bb.tests.fetch.NPMTest passes, with an npm in PATH that always fails. bb.tests.event and bb.tests.cooker pass.
[1] https://lore.kernel.org/openembedded-core/[email protected]/T/
Stefano Tondo (2):
cooker: Write variables that are not JSON types to the event log fetch/npm: Build the tarball URL without the npm executable
lib/bb/cooker.py | 2 +- lib/bb/fetch/npm.py | 50 +++---------------------- lib/bb/tests/fetch.py | 85 ++++++++++++++++++++++--------------------- 3 files changed, 50 insertions(+), 87 deletions(-)
Signature · 1 line
2.56.1
PATCH 1/2 cooker: Write variables that are not JSON types to the event log
From: Stefano Tondo <[email protected]>
EventWriter.write_variables() writes the history of all variables to the event log as JSON. A variable can hold a Python object. For example, sstate.bbclass stores compiled regular expressions in _SSTATE_EXCLUDEDEPS_SYSROOT during a build. When the same server runs a second build, json.dumps() fails:
TypeError: Object of type Pattern is not JSON serializable
This occurs with tinfoil, for example in "recipetool create", which starts more than one build in one server.
Write such values as strings.
--- lib/bb/cooker.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bb/cooker.py +1 -1
@@ -124,7 +124,7 @@ class EventWriter: def write_variables(self): with self._open() as f: - f.write("%s\n" % json.dumps({ "allvariables" : self.cooker.getAllKeysWithFlags(["doc", "func"])})) + f.write("%s\n" % json.dumps({ "allvariables" : self.cooker.getAllKeysWithFlags(["doc", "func"])}, default=str)) def send(self, event): with self._open() as f:
Signature · 1 line
2.56.1
Re: [bitbake-devel] [PATCH 1/2] cooker: Write variables that are not JSON types to the event log
Quoted 8 lines · Stefano Tondo via bitbake-devel
From: Stefano Tondo <[email protected]>
EventWriter.write_variables()writes the history of all variables to the event log as JSON. A variable can hold a Python object. For example, sstate.bbclass stores compiled regular expressions in_SSTATE_EXCLUDEDEPS_SYSROOTduring a build. When the same server runs a second build,json.dumps()fails:
Quoted 6 lines
TypeError: Object of type Pattern is not JSON serializable
This occurs with tinfoil, for example in "recipetool create", which starts more than one build in one server.
Write such values as strings.
This worries me a bit (well, a lot!), specifically the "second build" piece. Why is this persisting across builds? Does:
https://git.openembedded.org/bitbake/commit/?id=606786d17351534ce35d1b0f89815461546a66fe
have any effect on that?
Cheers,
Richard
PATCH 2/2 fetch/npm: Build the tarball URL without the npm executable
In reply to Stefano TondoFrom: Stefano Tondo <[email protected]>
The npm fetcher ran "npm view" only to get the URL of the package tarball. So it needed the npm executable and network access before the download. In a build, base.bbclass in OE-Core makes do_fetch of a recipe with an npm:// URL depend on nodejs-native for this reason. The bitbake tests take npm from the host, and they skipped themselves when the host had no npm.
The npm registry gives each tarball a fixed URL:
<registry>/<package>/-/<basename>-<version>.tgz, where basename is the package name without its scope. Build this URL from the SRC_URI parameters instead. The checksum still comes from the recipe only.
The tests no longer need npm. Add tests that build the URL and fetch a package from a premirror without network access.
--- lib/bb/fetch/npm.py | 50 +++---------------------- lib/bb/tests/fetch.py | 85 ++++++++++++++++++++++--------------------- 2 files changed, 49 insertions(+), 86 deletions(-)
lib/bb/fetch/npm.py +6 -44
@@ -29,20 +29,16 @@ Supported SRC_URI options are: """ import base64 -import json import os import re -import shlex import tempfile import bb from bb.fetch import Fetch -from bb.fetch import FetchError from bb.fetch import FetchMethod from bb.fetch import MalformedUrl from bb.fetch import MissingParameterError from bb.fetch import ParameterError from bb.fetch import URI -from bb.fetch import check_network_access from bb.fetch import runfetchcmd from bb.utils import is_semver
@@ -213,9 +209,6 @@ class Npm(FetchMethod): else: ud.localfile = npm_localfile(ud.package, ud.version) - # Get the base 'npm' command - ud.basecmd = shlex.split(d.getVar("FETCHCMD_npm") or "") or ["npm"] - # This fetcher resolves a URI from a npm package name and version and # then forwards it to a proxy fetcher. A resolve file containing the # resolved URI is created to avoid unwanted network access (if the file
@@ -226,45 +219,14 @@ class Npm(FetchMethod): ud.resolvefile = self.localpath(ud, d) + ".resolved" def _resolve_proxy_url(self, ud, d): - """Resolve the tarball URL from the registry and cache it without any checksum. + """Write the tarball URL of the package to the resolve file. - Checksums must never be sourced from the registry: a compromised registry - controls both the tarball and its advertised hash, so any checksum obtained - there provides no tamper detection. Checksums are applied in _setup_proxy - from the recipe-provided SRC_URI parameters instead. + The URL follows the layout of the npm registry, so it is known + without network access and without the npm executable. The URL + has no checksum: _setup_proxy adds the one from the recipe. """ - def _npm_view(): - args = [] - args.append(("json", "true")) - args.append(("registry", ud.registry)) - pkgver = ud.package + "@" + ud.version - cmd = ud.basecmd + ['view', pkgver] - env = NpmEnvironment(d) - check_network_access(d, cmd, ud.registry) - view_string = env.run(cmd, args=args) - - if not view_string: - raise FetchError("Unavailable package %s" % pkgver, ud.url) - - try: - view = json.loads(view_string) - except json.JSONDecodeError as e: - raise FetchError("Invalid view from npm: %s" % str(e), ud.url) - - error = view.get("error") - if error is not None: - raise FetchError(error.get("summary") or str(error), ud.url) - - if ud.version != view.get("version"): - raise ParameterError("Invalid 'version' parameter", ud.url) - - return view - - view = _npm_view() - tarball_url = view.get("dist", {}).get("tarball") - - if tarball_url is None: - raise FetchError("Invalid 'dist.tarball' in view", ud.url) + basename = ud.package.split("/")[-1] + tarball_url = "%s/%s/-/%s-%s.tgz" % (ud.registry.rstrip("/"), ud.package, basename, ud.version) bb.utils.mkdirhier(os.path.dirname(ud.resolvefile)) with open(ud.resolvefile, "w") as f:
lib/bb/tests/fetch.py +43 -42
220 lines not on this page · raw message
Signature · 1 line
2.56.1