Skip to contentSkip to conversation

npm: list the dependencies in SRC_URI

PATCH In review Stefano Tondo 2 participants 12 messages Nov 20

lore
Share & tools
lore
#0 lore raw

cover npm: list the dependencies in SRC_URI

Hi,

This series lets an npm recipe list its dependencies in SRC_URI. Each package gets one entry with its checksum. We agreed on this in the RFC thread [1]. npmsw:// still works, but recipetool no longer uses it.

Today, a recipe that recipetool creates builds without its dependencies. recipetool writes an npmsw:// entry without destsuffix. Since bitbake commit 84f102954, npmsw unpacks into UNPACKDIR and not into ${S}. So npm.bbclass does not find npm-shrinkwrap.json, packs no dependencies, and gives no error. The package has no node_modules.

1. Patches 1 to 5 make "recipetool create" for npm work again on master. It failed on the missing checksum, on license values given as lists, and on the "@" of scoped names. The selftest expected an old value of S.
2. lib/oe/npm gives the SRC_URI entries for the dependencies of a shrinkwrap file, with unit tests.
3. The new class npm-update-dependencies writes these entries to ${BPN}-npm-deps.inc, like the cargo and Go update classes.
4. npm.bbclass checks that SRC_URI matches the shrinkwrap file, as Richard asked. It fails with a hint to run the update task.
5. recipetool runs do_clean after the tasks that it runs for the new recipe. Else devtool leaves stale stamps of do_unpack behind.
6. recipetool creates the recipe with the explicit list.

This series needs the two patches of the bitbake series "fetch/npm: no npm executable, and an event log fix" [2]. Without "fetch/npm: Build the tarball URL without the npm executable", the npm fetcher still calls the npm executable of the host. Without "cooker: Write variables that are not JSON types to the event log", the recipetool selftest fails.

Two points for review:

- The check also applies to a recipe without npmsw:// that has node_modules in its source tree. I do not know of such a recipe. If there is one, an opt-out variable is simple to add.
- This series does not deprecate npmsw://. That can follow if you want it.

Tests on master with the two bitbake patches: oe-selftest -r oelib.npm recipetool.RecipetoolCreateTests.test_recipetool_create_npm devtool.DevtoolAddTests.test_devtool_add_npm pass. A build of node-server-example with the explicit list installs all 72 dependencies. With one entry removed, do_configure fails and names it.

[1] https://lore.kernel.org/openembedded-core/[email protected]/T/
[2] https://lore.kernel.org/bitbake-devel/[email protected]/

Stefano Tondo (10):
recipetool: create_npm: Give the fallback licenses as a string
recipetool: create: Get the checksum of npm packages from the registry
recipetool: create_npm: Give the package fallback licenses as strings
recipetool: create_npm: Remove the "@" of a scope from generated names oeqa/selftest: recipetool: Expect the npm source in UNPACKDIR lib/oe/npm: Add SRC_URI entries for the dependencies of a shrinkwrap
file
npm-update-dependencies: Add a class to list the npm dependencies
npm: Check that SRC_URI lists all dependencies of the shrinkwrap file
recipetool: create: Clean the recipe after its tasks
recipetool: create_npm: List the npm dependencies in SRC_URI

 .../npm-update-dependencies.bbclass           |  48 +++++++
 meta/classes-recipe/npm.bbclass               |  24 ++++
 meta/lib/oe/npm.py                            |  99 ++++++++++++++
 meta/lib/oeqa/selftest/cases/oelib/npm.py     | 121 ++++++++++++++++++
 meta/lib/oeqa/selftest/cases/recipetool.py    |  22 +++-
 scripts/lib/recipetool/create.py              |  31 +++++
 scripts/lib/recipetool/create_npm.py          |  62 ++++++---
 7 files changed, 384 insertions(+), 23 deletions(-)
 create mode 100644 meta/classes-recipe/npm-update-dependencies.bbclass
 create mode 100644 meta/lib/oe/npm.py
 create mode 100644 meta/lib/oeqa/selftest/cases/oelib/npm.py
Signature · 1 line
2.56.1
#1 lore raw

PATCH 1/10 recipetool: create_npm: Give the fallback licenses as a string

From: Stefano Tondo <[email protected]>

Since 727ea2769c ("license: Remove tidy_licenses()"), handle_license_vars() reads LICENSE from the extra values as a license string. The npm handler still gives a set, so recipetool create stops with "AttributeError: 'set' object has no attribute 'strip'" for each npm package that has dependencies.

Join the sorted fallback licenses with " AND ". Also remove a line that sorted a copy of the set and did not use the result.

---
 scripts/lib/recipetool/create_npm.py | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
scripts/lib/recipetool/create_npm.py +2 -3
index 10eca3c63e..5edf647176 100644
@@ -283,10 +283,9 @@ class NpmRecipeHandler(RecipeHandler):
         (licfiles, packages, fallback_licenses) = self._handle_licenses(srctree, shrinkwrap_file, dev)
         licvalues = match_licenses(licfiles, srctree, d)
         split_pkg_licenses(licvalues, packages, lines_after, fallback_licenses)
-        fallback_licenses_flat = set(license for sublist in fallback_licenses.values() for license in sublist)
-        fallback_licesens_flat = sorted(fallback_licenses_flat).sort()
+        fallback_licenses_flat = sorted(set(license for sublist in fallback_licenses.values() for license in sublist))
         extravalues["LIC_FILES_CHKSUM"] = generate_common_licenses_chksums(fallback_licenses_flat, d)
-        extravalues["LICENSE"] = fallback_licenses_flat
+        extravalues["LICENSE"] = " AND ".join(fallback_licenses_flat)
 
         classes.append("npm")
         handled.append("buildsystem")
Signature · 1 line
2.56.1
#2 lore raw

PATCH 2/10 recipetool: create: Get the checksum of npm packages from the registry

In reply to Stefano Tondo

From: Stefano Tondo <[email protected]>

Since bitbake 9e45e00a2 ("fetch/{npm,npmsw}: re-enable fetchers now that checksums come from SRC_URI"), an npm:// URL needs a checksum. recipetool create gets an npm:// URL without a checksum, so it fails before the fetch:

MissingParameterError: ... Missing checksum for npm package

The usual way, a fetch with BB_STRICT_CHECKSUM = "ignore", does not work for npm on purpose. So get the checksum from the registry ("dist.integrity") before the fetch, and add it to the URL in the recipe. For a new recipe, the registry is the only source of this checksum.

Update the selftest for the checksum in SRC_URI.

---
 meta/lib/oeqa/selftest/cases/recipetool.py |  4 +++-
 scripts/lib/recipetool/create.py           | 22 ++++++++++++++++++++++
 2 files changed, 25 insertions(+), 1 deletion(-)
meta/lib/oeqa/selftest/cases/recipetool.py +3 -1
index 0aaa090ff5..c6aa77399f 100644
@@ -449,7 +449,9 @@ class RecipetoolCreateTests(RecipetoolBase):
         checkvars['HOMEPAGE'] = 'https://github.com/savoirfairelinux/node-server-example#readme'
         checkvars['LICENSE'] = 'BSD-3-Clause AND ISC AND MIT AND Unknown'
         urls = []
-        urls.append('npm://registry.npmjs.org/;package=@savoirfairelinux/node-server-example;version=${PV}')
+        urls.append('npm://registry.npmjs.org/;package=@savoirfairelinux/node-server-example;version=${PV}'
+                    ';sha512sum=f2dd7d88cb9a129fbb97eb87a8b5103bab24f783420fd7587f8000a355a12bf7'
+                    '83f1263230afc588123958b73e36bb241f63eaf08119aac5aa2a870bc4de9223')
         urls.append('npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json')
         checkvars['SRC_URI'] = set(urls)
         checkvars['S'] = '${WORKDIR}/npm'
scripts/lib/recipetool/create.py +22 -0
index 662e7a6232..2df17dd6e4 100644
@@ -413,6 +413,18 @@ def is_package(url):
         return True
     return False
 
+def get_npm_checksum(registry, package, version):
+    """Get the checksum of an npm package from the registry.
+
+    The npm fetcher needs a checksum before the download. When a new recipe
+    is created, the registry is the only source of this checksum.
+    """
+    import urllib.request
+    from bb.fetch.npm import npm_integrity
+    url = "https://%s/%s/%s" % (registry.strip("/"), package, version)
+    with urllib.request.urlopen(url, timeout=60) as response:
+        return npm_integrity(json.load(response)["dist"]["integrity"])
+
 def create_recipe(args):
     import bb.process
     import tempfile
@@ -523,6 +535,15 @@ def create_recipe(args):
         # Assume 'master' branch if not set
         if scheme in ['git', 'gitsm'] and 'branch' not in params and 'nobranch' not in params:
             params['branch'] = 'master'
+        npm_checksum = {}
+        if scheme == 'npm' and not any(p in params for p in bb.fetch.npm.CHECKSUM_PARMS):
+            try:
+                checksum_name, checksum = get_npm_checksum(network + path, params.get('package', ''), params.get('version', ''))
+            except Exception as e:
+                logger.error('Unable to get the checksum of %s from the registry: %s' % (srcuri, e))
+                sys.exit(1)
+            npm_checksum[checksum_name] = checksum
+            params.update(npm_checksum)
         fetchuri = bb.fetch.encodeurl((scheme, network, path, user, passwd, params))
 
         tmpparent = tinfoil.config_data.getVar('BASE_WORKDIR')
@@ -587,6 +608,7 @@ def create_recipe(args):
         # Since we might have a value in srcbranch, we need to
         # recontruct the srcuri to include 'branch' in params.
         scheme, network, path, user, passwd, params = bb.fetch.decodeurl(srcuri)
+        params.update(npm_checksum)
         if scheme in ['git', 'gitsm']:
             params['branch'] = srcbranch or 'master'
 
Signature · 1 line
2.56.1
#3 lore raw

PATCH 3/10 recipetool: create_npm: Give the package fallback licenses as strings

In reply to Stefano Tondo

From: Stefano Tondo <[email protected]>

The npm handler finds the licenses of a dependency in its package.json when it has no license file. Since 727ea2769c ("license: Remove tidy_licenses()"), split_pkg_licenses() writes such a fallback value as it is. The npm handler gives a list, so the recipe gets lines like:

  LICENSE:${PN}-cookie-signature = "['MIT']"

Join the licenses of each package with " AND ", as before that change.

---
 scripts/lib/recipetool/create_npm.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
scripts/lib/recipetool/create_npm.py +2 -1
index 5edf647176..437cb91ad4 100644
@@ -282,7 +282,8 @@ class NpmRecipeHandler(RecipeHandler):
         bb.note("Handling licences ...")
         (licfiles, packages, fallback_licenses) = self._handle_licenses(srctree, shrinkwrap_file, dev)
         licvalues = match_licenses(licfiles, srctree, d)
-        split_pkg_licenses(licvalues, packages, lines_after, fallback_licenses)
+        split_pkg_licenses(licvalues, packages, lines_after,
+                           {pkg: " AND ".join(lics) for pkg, lics in fallback_licenses.items()})
         fallback_licenses_flat = sorted(set(license for sublist in fallback_licenses.values() for license in sublist))
         extravalues["LIC_FILES_CHKSUM"] = generate_common_licenses_chksums(fallback_licenses_flat, d)
         extravalues["LICENSE"] = " AND ".join(fallback_licenses_flat)
Signature · 1 line
2.56.1
#4 lore raw

PATCH 4/10 recipetool: create_npm: Remove the "@" of a scope from generated names

In reply to Stefano Tondo

From: Stefano Tondo <[email protected]>

Since bitbake commit 7c9573cb6, npm_package() keeps the "@" of a scoped package. This gives unique download file names. recipetool also uses npm_package() for the recipe name and the package names. So it creates "@savoirfairelinux-node-server-example_1.0.0.bb", and for a scoped dependency it writes overrides that bitbake cannot parse:

  LICENSE:${PN}-@-execute = "MIT"

Remove the "@" from these names. The selftest expects the recipe name without it.

---
 scripts/lib/recipetool/create_npm.py | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
scripts/lib/recipetool/create_npm.py +9 -3
index 437cb91ad4..50dd45c7d2 100644
@@ -23,6 +23,12 @@ logger = logging.getLogger('recipetool')
 
 TINFOIL = None
 
+def npm_recipe_name(package):
+    """Get a recipe or package name from an npm package name"""
+    # npm_package() keeps the "@" of a scope for unique download file names.
+    # An override cannot have it.
+    return npm_package(package).lstrip("@").strip("-")
+
 def tinfoil_init(instance):
     """Initialize tinfoil"""
     global TINFOIL
@@ -147,7 +153,7 @@ class NpmRecipeHandler(RecipeHandler):
         # Handle the dependencies
         def _handle_dependency(name, params, destdir):
             deptree = destdir.split('node_modules/')
-            suffix = "-".join([npm_package(dep) for dep in deptree])
+            suffix = "-".join([npm_recipe_name(dep) for dep in deptree])
             packages["${PN}" + suffix] = destdir
             _get_package_licenses(destdir, "${PN}" + suffix)
 
@@ -171,7 +177,7 @@ class NpmRecipeHandler(RecipeHandler):
         peer_deps = packages.get("", {}).get("peerDependencies", {})
 
         for peer_dep in peer_deps:
-            peer_dep_yocto_name = npm_package(peer_dep)
+            peer_dep_yocto_name = npm_recipe_name(peer_dep)
             bb.warn(peer_dep + " is a peer dependencie of the actual package. " +
             "Please add this peer dependencie to the RDEPENDS variable as %s and generate its recipe with devtool"
             % peer_dep_yocto_name)
@@ -195,7 +201,7 @@ class NpmRecipeHandler(RecipeHandler):
         if "name" not in data or "version" not in data:
             return False
 
-        extravalues["PN"] = npm_package(data["name"])
+        extravalues["PN"] = npm_recipe_name(data["name"])
         extravalues["PV"] = data["version"]
 
         if "description" in data:
Signature · 1 line
2.56.1
#5 lore raw

Patchtest results for [PATCH 04/10] recipetool: create_npm: Remove the "@" of a scope from generated names

Thank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information:

---
Testing patch /home/patchtest/share/mboxes/04-10-recipetool-create_npm-Remove-the-of-a-scope-from-generated-names.patch

FAIL: test commit message user tags: Mbox includes one or more GitHub-style username tags. Ensure that any "@" symbols are stripped out of usernames (test_mbox.TestMbox.test_commit_message_user_tags)

PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test auh changelog truncation notice (test_mbox.TestMbox.test_auh_changelog_truncation_notice)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test max line length (test_metadata.TestMetadata.test_max_line_length)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)

SKIP: pretest src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE check ignore: No modified recipes or older target branch, skipping test (test_metadata.TestMetadata.test_cve_check_ignore)
SKIP: test CVE tag format: No new source patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new source patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new source patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: No modified recipes, skipping test (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!

#6 lore raw

PATCH 5/10 oeqa/selftest: recipetool: Expect the npm source in UNPACKDIR

In reply to Stefano Tondo

From: Stefano Tondo <[email protected]>

recipetool writes S = "${UNPACKDIR}/npm" for an npm package, because the fetcher unpacks into UNPACKDIR. The selftest still expected "${WORKDIR}/npm".

---
 meta/lib/oeqa/selftest/cases/recipetool.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oeqa/selftest/cases/recipetool.py +1 -1
index c6aa77399f..ba4f674b4c 100644
@@ -454,7 +454,7 @@ class RecipetoolCreateTests(RecipetoolBase):
                     '83f1263230afc588123958b73e36bb241f63eaf08119aac5aa2a870bc4de9223')
         urls.append('npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json')
         checkvars['SRC_URI'] = set(urls)
-        checkvars['S'] = '${WORKDIR}/npm'
+        checkvars['S'] = '${UNPACKDIR}/npm'
         checkvars['LICENSE:${PN}'] = 'MIT'
         checkvars['LICENSE:${PN}-base64'] = 'Unknown'
         checkvars['LICENSE:${PN}-accepts'] = 'MIT'
Signature · 1 line
2.56.1
#7 lore raw

PATCH 6/10 lib/oe/npm: Add SRC_URI entries for the dependencies of a shrinkwrap file

In reply to Stefano Tondo

From: Stefano Tondo <[email protected]>

The npmsw fetcher reads npm-shrinkwrap.json to find the dependencies of an npm package. So SRC_URI does not show these dependencies, and the fetcher holds logic that belongs in the metadata.

Add shrinkwrap_src_uri(), which gives one SRC_URI entry for each dependency of a shrinkwrap file:

- A package from a registry gets an npm:// entry with its name, version and checksum.
- Another tarball gets its URL with the checksum.
- A git dependency gets a git:// entry with its revision.

Each entry unpacks to the location of the dependency in the source tree, where npm.bbclass expects it. Add src_uri_differences(), which compares these entries with SRC_URI.

Add tests for both functions.

---
 meta/lib/oe/npm.py                        |  99 ++++++++++++++++++
 meta/lib/oeqa/selftest/cases/oelib/npm.py | 121 ++++++++++++++++++++++
 2 files changed, 220 insertions(+)
 create mode 100644 meta/lib/oe/npm.py
 create mode 100644 meta/lib/oeqa/selftest/cases/oelib/npm.py
meta/lib/oe/npm.py new file +99 -0
new file mode 100644
index 0000000000..19f547839e
@@ -0,0 +1,99 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+"""Explicit SRC_URI entries for the dependencies of an npm shrinkwrap file."""
+
+import os
+import re
+
+import bb.fetch
+import bb.utils
+from bb.fetch.npm import npm_integrity, npm_localfile
+from bb.fetch.npmsw import foreach_dependencies
+
+GIT_RESOLVED_RE = re.compile(r"^git\+(?P<protocol>[a-z]+)://(?P<url>[^#]+)#(?P<rev>[0-9a-f]+)$")
+
+
+def shrinkwrap_src_uri(shrinkwrap, destdir, dev=False):
+    """Return one SRC_URI entry for each dependency of a shrinkwrap file.
+
+    shrinkwrap is the content of npm-shrinkwrap.json (lockfileVersion 2 or
+    later). destdir is the directory of the main package, relative to
+    UNPACKDIR. Each dependency unpacks to its location in destdir (for example
+    destdir/node_modules/foo), where npm.bbclass expects it. dev also adds the
+    devDependencies. The entries are sorted by location.
+
+    A package from a registry gets an npm:// entry, so that the fetcher knows
+    its name and version. Other tarballs get a plain URL and git dependencies
+    a git:// URL. Each entry has the checksum or revision of the shrinkwrap
+    file.
+    """
+    entries = {}
+
+    def _add(name, data, location):
+        package = data.get("name", name)
+        version = data.get("version")
+        resolved = data.get("resolved")
+        integrity = data.get("integrity")
+        destsuffix = os.path.join(destdir, location)
+
+        if not resolved:
+            # npm leaves out "resolved" for some duplicate packages. npm.bbclass
+            # does not need these, as npmsw already skipped them.
+            if version and integrity:
+                return
+            raise ValueError("%s: no 'resolved' field" % location)
+
+        if data.get("link") or resolved.startswith("file:"):
+            raise ValueError("%s: local dependencies are not supported" % location)
+
+        match = GIT_RESOLVED_RE.match(resolved)
+        if match:
+            entries[location] = "git://%s;protocol=%s;rev=%s;nobranch=1;destsuffix=%s" % (
+                match["url"], match["protocol"], match["rev"], destsuffix)
+            return
+
+        if not integrity or not resolved.startswith(("https://", "http://")):
+            raise ValueError("%s: unsupported dependency %s" % (location, resolved))
+
+        checksum_name, checksum = npm_integrity(integrity)
+        basename = package.split("/")[-1]
+        suffix = "/%s/-/%s-%s.tgz" % (package, basename, version)
+        if resolved.startswith("https://") and resolved.endswith(suffix) and bb.utils.is_semver(version):
+            registry = resolved[len("https://"):-len(suffix)]
+            entries[location] = "npm://%s/;package=%s;version=%s;%s=%s;destsuffix=%s" % (
+                registry, package, version, checksum_name, checksum, destsuffix)
+        else:
+            entries[location] = "%s;downloadfilename=%s;%s=%s;subdir=%s;striplevel=1" % (
+                resolved, npm_localfile(os.path.basename(resolved)), checksum_name, checksum, destsuffix)
+
+    foreach_dependencies(shrinkwrap, _add, dev)
+    return [entries[location] for location in sorted(entries)]
+
+
+def src_uri_differences(expected, src_uri):
+    """Compare the expected entries with SRC_URI.
+
+    Returns (missing, extra): the expected entries that SRC_URI does not have,
+    and the SRC_URI entries that unpack into a node_modules directory but are
+    not expected. Entries are compared after parsing, so the order of the URL
+    parameters does not matter.
+    """
+    def _key(url):
+        scheme, host, path, user, password, parm = bb.fetch.decodeurl(url)
+        return (scheme, host, path, user, password, tuple(sorted(parm.items())))
+
+    def _unpacks_to_node_modules(url):
+        parm = bb.fetch.decodeurl(url)[5]
+        dest = parm.get("destsuffix") or parm.get("subdir") or ""
+        return "node_modules/" in dest + "/"
+
+    expected_keys = {_key(url): url for url in expected}
+    actual_keys = {_key(url): url for url in src_uri}
+    missing = [url for key, url in expected_keys.items() if key not in actual_keys]
+    extra = [url for key, url in actual_keys.items()
+             if key not in expected_keys and _unpacks_to_node_modules(url)]
+    return missing, extra
meta/lib/oeqa/selftest/cases/oelib/npm.py new file +121 -0

121 lines not on this page · raw message

Signature · 1 line
2.56.1
#8 lore raw

PATCH 7/10 npm-update-dependencies: Add a class to list the npm dependencies

In reply to Stefano Tondo

From: Stefano Tondo <[email protected]>

Add a class with the task do_update_npm_deps. The task reads npm-shrinkwrap.json in ${S} and writes one SRC_URI entry for each dependency to ${THISDIR}/${BPN}-npm-deps.inc. The recipe requires this file. This is the same model as the cargo and Go update tasks.

Then the recipe does not need npmsw://. The fetchers see each dependency as a normal SRC_URI entry with its own checksum.

---
 .../npm-update-dependencies.bbclass           | 48 +++++++++++++++++++
 1 file changed, 48 insertions(+)
 create mode 100644 meta/classes-recipe/npm-update-dependencies.bbclass
meta/classes-recipe/npm-update-dependencies.bbclass new file +48 -0
new file mode 100644
index 0000000000..43b00b9b1b
@@ -0,0 +1,48 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+##
+## Purpose:
+## This class is used to update the list of npm dependencies in SRC_URI
+## by reading npm-shrinkwrap.json in the source tree. It writes one entry
+## with a checksum for each dependency, so that no fetcher has to read the
+## shrinkwrap file. npm.bbclass checks that SRC_URI matches the shrinkwrap
+## file.
+##
+## To perform the update: bitbake -c update_npm_deps recipe-name
+
+addtask do_update_npm_deps after do_patch
+do_update_npm_deps[nostamp] = "1"
+do_update_npm_deps[doc] = "Update the recipe by reading npm-shrinkwrap.json and write in ${THISDIR}/${BPN}-npm-deps.inc"
+
+RECIPE_UPGRADE_EXTRA_TASKS += "do_update_npm_deps"
+
+python do_update_npm_deps() {
+    import json
+    import oe.npm
+
+    shrinkwrap_file = d.expand("${S}/npm-shrinkwrap.json")
+    with open(shrinkwrap_file) as f:
+        shrinkwrap = json.load(f)
+
+    destdir = os.path.relpath(d.getVar("S"), d.getVar("UNPACKDIR"))
+    dev = bb.utils.to_boolean(d.getVar("NPM_INSTALL_DEV"), False)
+    try:
+        entries = oe.npm.shrinkwrap_src_uri(shrinkwrap, destdir, dev)
+    except ValueError as e:
+        bb.fatal("%s: %s" % (shrinkwrap_file, e))
+
+    lines = ["# Autogenerated with 'bitbake -c update_npm_deps %s'" % d.getVar("PN"), ""]
+    lines.append('SRC_URI += " \\')
+    lines.extend("    %s \\" % entry for entry in entries)
+    lines.append('"')
+
+    target_file = d.expand("${THISDIR}/${BPN}-npm-deps.inc")
+    with open(target_file, "w") as f:
+        f.write("\n".join(lines) + "\n")
+
+    bb.note("Successfully updated %d npm dependencies inside %s" % (len(entries), target_file))
+}
Signature · 1 line
2.56.1
#9 lore raw

PATCH 8/10 npm: Check that SRC_URI lists all dependencies of the shrinkwrap file

In reply to Stefano Tondo

From: Stefano Tondo <[email protected]>

When a recipe lists the npm dependencies in SRC_URI, the list can become different from npm-shrinkwrap.json, for example after an upgrade. Then npm.bbclass packs a wrong set of dependencies.

If SRC_URI has no npmsw:// entry, compare SRC_URI with the shrinkwrap file in do_configure. Fail if a dependency is missing, or if an entry unpacks into node_modules but the shrinkwrap file does not have it. The error tells the user to run the update task.

Do not check with externalsrc, which removes the dependencies from SRC_URI. There the external source tree must contain them.

---
 meta/classes-recipe/npm.bbclass | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
meta/classes-recipe/npm.bbclass +24 -0
index 6f3261238d..e0d3d88d2b 100644
@@ -111,6 +111,24 @@ def npm_pack(env, srcdir, workdir, excludes=None):
 
     return (tarball, j)
 
+def npm_check_src_uri(d, shrinkwrap):
+    """Check that SRC_URI has one entry for each dependency of the shrinkwrap file"""
+    import oe.npm
+
+    destdir = os.path.relpath(d.getVar("S"), d.getVar("UNPACKDIR"))
+    dev = bb.utils.to_boolean(d.getVar("NPM_INSTALL_DEV"), False)
+    try:
+        expected = oe.npm.shrinkwrap_src_uri(shrinkwrap, destdir, dev)
+    except ValueError as e:
+        bb.fatal("npm-shrinkwrap.json: %s" % e)
+
+    missing, extra = oe.npm.src_uri_differences(expected, d.getVar("SRC_URI").split())
+    if missing or extra:
+        bb.fatal("SRC_URI does not match npm-shrinkwrap.json (%d missing, %d extra entries). "
+                 "Inherit npm-update-dependencies and run 'bitbake -c update_npm_deps %s'.\n%s"
+                 % (len(missing), len(extra), d.getVar("PN"),
+                    "\n".join(["missing: " + url for url in missing] + ["extra: " + url for url in extra])))
+
 python npm_do_configure() {
     """
     Step one: configure the npm cache and the main npm package
@@ -177,6 +195,12 @@ python npm_do_configure() {
        if int(orig_shrinkwrap.get("lockfileVersion", 0)) < 2:
            bb.fatal("%s: lockfileVersion version 2 or later is required" % orig_shrinkwrap_file)
 
+       # externalsrc removes the dependencies from SRC_URI: the external
+       # source tree must contain them
+       if not d.getVar("EXTERNALSRC") and \
+               not any(url.startswith("npmsw://") for url in d.getVar("SRC_URI").split()):
+           npm_check_src_uri(d, orig_shrinkwrap)
+
        cached_shrinkwrap = copy.deepcopy(orig_shrinkwrap)
        for package in orig_shrinkwrap["packages"]:
             if package != "":
Signature · 1 line
2.56.1
#10 lore raw

PATCH 9/10 recipetool: create: Clean the recipe after its tasks

In reply to Stefano Tondo

From: Stefano Tondo <[email protected]>

recipetool can run tasks of the new recipe, as it does for Go. These tasks leave stamps, for example for do_unpack and do_patch. devtool then builds the recipe with externalsrc, which uses other stamps, and its do_clean removes the work directory. The first stamps stay.

A later build of the recipe with the same signatures trusts these stamps and does not unpack again, so the source is missing. The next commit runs do_update_npm_deps for npm recipes. Then test_devtool_add_npm followed by test_recipetool_create_npm fails:
do_update_npm_deps cannot find npm-shrinkwrap.json.

Run do_clean after the tasks.

---
 scripts/lib/recipetool/create.py | 5 +++++
 1 file changed, 5 insertions(+)
scripts/lib/recipetool/create.py +5 -0
index 2df17dd6e4..b2cc92c90b 100644
@@ -955,6 +955,11 @@ def create_recipe(args):
     for task in run_tasks:
         logger.info("Running task %s" % task)
         tinfoil.build_file_sync(outfile, task)
+    if run_tasks:
+        # Remove the work directory and the stamps of these tasks. Else a
+        # later build of the recipe can trust the stamps, also when devtool
+        # has removed the work directory in the meantime.
+        tinfoil.build_file_sync(outfile, "clean")
 
     if tempsrc:
         if args.keep_temp:
Signature · 1 line
2.56.1
#11 lore raw

PATCH 10/10 recipetool: create_npm: List the npm dependencies in SRC_URI

In reply to Stefano Tondo

From: Stefano Tondo <[email protected]>

recipetool adds an npmsw:// entry without destsuffix to the recipe. Since bitbake commit 84f102954, the npmsw fetcher unpacks into the root of UNPACKDIR and not into ${S}. So npm-shrinkwrap.json is not in ${S}, and npm.bbclass silently packs no dependencies. The package then has no node_modules.

Create the recipe with the explicit list of dependencies instead:

- Add npm-shrinkwrap.json to SRC_URI, so that it unpacks into ${S}.
- Inherit npm-update-dependencies and require ${BPN}-npm-deps.inc.
- Run the update task, which writes the dependencies to the .inc file.

create_recipe() gives the source subdirectory to the handlers in extravalues, so that the npm handler knows where ${S} is.

Fetch the dependencies from the same list for the license scan.

The selftest now checks the .inc file.

---
 meta/lib/oeqa/selftest/cases/recipetool.py | 16 +++++++--
 scripts/lib/recipetool/create.py           |  4 +++
 scripts/lib/recipetool/create_npm.py       | 42 +++++++++++++++-------
 3 files changed, 48 insertions(+), 14 deletions(-)
meta/lib/oeqa/selftest/cases/recipetool.py +14 -2
index ba4f674b4c..181d95804f 100644
@@ -5,11 +5,13 @@
 #
 
 import errno
+import json
 import os
 import shutil
 import tempfile
 import urllib.parse
 
+import oe.npm
 import oe.spdx_license
 
 from oeqa.utils.commands import runCmd, bitbake, get_bb_var
@@ -452,16 +454,26 @@ class RecipetoolCreateTests(RecipetoolBase):
         urls.append('npm://registry.npmjs.org/;package=@savoirfairelinux/node-server-example;version=${PV}'
                     ';sha512sum=f2dd7d88cb9a129fbb97eb87a8b5103bab24f783420fd7587f8000a355a12bf7'
                     '83f1263230afc588123958b73e36bb241f63eaf08119aac5aa2a870bc4de9223')
-        urls.append('npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json')
+        urls.append('file://npm-shrinkwrap.json;subdir=npm')
         checkvars['SRC_URI'] = set(urls)
         checkvars['S'] = '${UNPACKDIR}/npm'
         checkvars['LICENSE:${PN}'] = 'MIT'
         checkvars['LICENSE:${PN}-base64'] = 'Unknown'
         checkvars['LICENSE:${PN}-accepts'] = 'MIT'
         checkvars['LICENSE:${PN}-inherits'] = 'ISC'
-        inherits = ['npm']
+        inherits = ['npm', 'npm-update-dependencies']
         self._test_recipe_contents(recipefile, checkvars, inherits)
 
+        # The update task writes one SRC_URI entry for each dependency
+        with open(shrinkwrap) as f:
+            expected = oe.npm.shrinkwrap_src_uri(json.load(f), 'npm')
+        deps_require_file = os.path.join(temprecipe, 'savoirfairelinux-node-server-example-npm-deps.inc')
+        self.assertFileExists(deps_require_file)
+        checkvars = {'SRC_URI': set(expected)}
+        self._test_recipe_contents(deps_require_file, checkvars, [])
+        self.assertTrue(any(url.startswith('npm://registry.npmjs.org/;package=accepts;')
+                            and url.endswith(';destsuffix=npm/node_modules/accepts') for url in expected))
+
     def test_recipetool_create_github(self):
         # Basic test to see if github URL mangling works. Deliberately use an
         # older release of Meson at present so we don't need a toml parser.
scripts/lib/recipetool/create.py +4 -0
index b2cc92c90b..46a321a866 100644
@@ -784,8 +784,12 @@ def create_recipe(args):
         classes.append('bin_package')
         handled.append('buildsystem')
 
+    # Handlers can read the source subdirectory (S relative to UNPACKDIR),
+    # when S is set to it above
+    extravalues['srcsubdir'] = srcsubdir if not args.binary else ''
     for handler in handlers:
         handler.process(srctree_use, classes, lines_before, lines_after, handled, extravalues)
+    extravalues.pop('srcsubdir', None)
 
     # native and nativesdk classes are special and must be inherited last
     # If present, put them at the end of the classes list
scripts/lib/recipetool/create_npm.py +30 -12

75 lines not on this page · raw message

Signature · 1 line
2.56.1