fetch: package identity of crate, gomod and npm URLs
v2 cover fetch: package identity of crate, gomod and npm URLs
Hi,
This series adds the fetcher API from the RFC "spdx30: name, version and purl for dependency downloads" [1]. Richard asked for a fetcher API that returns structured data with the purl, so that OE-Core does not read fetcher internals.
Patch 1 adds FetchMethod.package_identity(). It returns a PackageIdentity dataclass with the name, the version and the purl of the downloaded package, or None. It uses only the URL and the datastore, so it works before the fetch and without network access.
Patch 2 implements it for the crate, gomod, gomodgit and npm fetchers, with unit tests.
The RFC also covered the downloads that gitsm and npmsw add by themselves. That part is not in this series. Richard and Alexander prefer to list those downloads in SRC_URI. I will send that change for npm in a separate series.
Changes in v2:
- package_identity() returns a PackageIdentity dataclass instead of a dict, as Paul proposed [2]. The dataclass is frozen.
- Rebased on master.
Tests: bitbake-selftest bb.tests.fetch.PackageIdentityTest passes. bb.tests.fetch without the network tests gives the same results as on master.
v3 of the OE-Core series that uses the API follows on the OE-Core list.
[1] https://lore.kernel.org/openembedded-core/[email protected]/
[2] https://lore.kernel.org/bitbake-devel/[email protected]/
Stefano Tondo (2):
fetch: Add package_identity() to the fetchers
fetch: Return the package identity of crate, gomod and npm URLs
lib/bb/fetch/__init__.py | 38 ++++++++++++++++++++++++ lib/bb/fetch/crate.py | 7 +++++ lib/bb/fetch/gomod.py | 16 ++++++++++ lib/bb/fetch/npm.py | 7 +++++ lib/bb/tests/fetch.py | 64 ++++++++++++++++++++++++++++++++++++++++ 5 files changed, 132 insertions(+)
Signature · 1 line
2.56.1
PATCH v2 1/2 fetch: Add package_identity() to the fetchers
From: Stefano Tondo <[email protected]>
Fetchers of package registries know which package a URL downloads:
its name and its version are part of the URL. SBOM generation needs this data, but it can only get it today by parsing fetcher internals.
Add FetchMethod.package_identity(). It returns a PackageIdentity dataclass with the name, the version and the Package URL (purl) of the downloaded package, or None when the URL does not name a package. It uses only the URL and the datastore, so it works before the fetch and without network access. Add bb.fetch.package_url() to build the purl with the correct encoding.
--- lib/bb/fetch/__init__.py | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)
lib/bb/fetch/__init__.py +38 -0
@@ -20,6 +20,7 @@ if 'git' not in urllib.parse.uses_netloc: urllib.parse.uses_netloc.append('git') import operator import collections +import dataclasses import subprocess import pickle import errno
@@ -1765,6 +1766,43 @@ class FetchMethod(object): """ return [] + def package_identity(self, ud, d): + """ + Return the identity of the package that the URL downloads, for + fetchers of package registries that know it, as a PackageIdentity. + Return None when the URL does not name a package. + + The result depends only on the URL and the datastore. It is + available as soon as the FetchData exists, before the fetch and + without network access. It is meant for SBOM generation. + """ + return None + + [email protected](frozen=True) +class PackageIdentity: + """ + The identity of a package that a fetcher downloads from a package + registry: its name, its version and its Package URL (purl). + """ + name: str + version: str + purl: str + + +def package_url(purl_type, name, version, qualifiers=None): + """ + Return a Package URL (https://github.com/package-url/purl-spec) for a + package. name can contain "/": the segments before the last one are + the namespace. Each segment, the version and the qualifier values are + percent-encoded. qualifiers is a dict, written sorted by key. + """ + quote = lambda s: urllib.parse.quote(s, safe="") + purl = "pkg:%s/%s@%s" % (purl_type, "/".join(quote(s) for s in name.split("/")), quote(version)) + if qualifiers: + purl += "?" + "&".join("%s=%s" % (k, quote(v)) for k, v in sorted(qualifiers.items())) + return purl + class DummyUnpackTracer(object): """
Signature · 1 line
2.56.1
PATCH v2 2/2 fetch: Return the package identity of crate, gomod and npm URLs
In reply to Stefano TondoFrom: Stefano Tondo <[email protected]>
Implement package_identity() for the crate, gomod, gomodgit and npm fetchers. Packages from a registry other than the default one get a repository_url qualifier in the purl. A gomod URL with mod=1 downloads only the go.mod file, which is not built into anything, so it has no identity.
--- lib/bb/fetch/crate.py | 7 +++++ lib/bb/fetch/gomod.py | 16 +++++++++++ lib/bb/fetch/npm.py | 7 +++++ lib/bb/tests/fetch.py | 64 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 94 insertions(+)
lib/bb/fetch/crate.py +7 -0
@@ -103,11 +103,18 @@ class Crate(Wget): ud.versionsurl = "%s://%s/%s/versions" % (proto, host, name) ud.parm['downloadfilename'] = "%s-%s.crate" % (name, version) + ud.crate = (name, version, None if host == 'crates.io' else "%s://%s" % (proto, host)) if 'name' not in ud.parm: ud.parm['name'] = '%s-%s' % (name, version) logger.debug2("Fetching %s to %s" % (ud.url, ud.parm['downloadfilename'])) + def package_identity(self, ud, d): + name, version, registry = ud.crate + qualifiers = {"repository_url": registry} if registry else None + purl = bb.fetch.package_url("cargo", name, version, qualifiers) + return bb.fetch.PackageIdentity(name, version, purl) + def unpack(self, ud, rootdir, d): """ Uses the crate to build the necessary paths for cargo to utilize it
lib/bb/fetch/gomod.py +16 -0
@@ -80,6 +80,13 @@ def escape(path): return re.sub(r'([A-Z])', lambda m: '!' + m.group(1).lower(), path) +def go_package_identity(ud): + module = ud.parm['module'] + version = ud.parm['version'] + return bb.fetch.PackageIdentity(module, version, + bb.fetch.package_url("golang", module, version)) + + class GoMod(Wget): """Class to fetch Go modules from a Go module proxy via wget"""
@@ -127,6 +134,12 @@ class GoMod(Wget): super().urldata_init(ud, d) + def package_identity(self, ud, d): + # A go.mod file alone is not built into anything + if ud.parm.get('mod', '0') == '1': + return None + return go_package_identity(ud) + def unpack(self, ud, rootdir, d): """Unpack the module in the module cache."""
@@ -218,6 +231,9 @@ class GoModGit(Git): super().urldata_init(ud, d) + def package_identity(self, ud, d): + return go_package_identity(ud) + def unpack(self, ud, rootdir, d): """Unpack the module in the module cache."""
lib/bb/fetch/npm.py +7 -0
@@ -225,6 +225,13 @@ class Npm(FetchMethod): ud.needdonestamp = False ud.resolvefile = self.localpath(ud, d) + ".resolved" + def package_identity(self, ud, d): + qualifiers = None + if ud.registry.rstrip("/") != "https://registry.npmjs.org": + qualifiers = {"repository_url": ud.registry} + purl = bb.fetch.package_url("npm", ud.package, ud.version, qualifiers) + return bb.fetch.PackageIdentity(ud.package, ud.version, purl) + def _resolve_proxy_url(self, ud, d): """Resolve the tarball URL from the registry and cache it without any checksum.
lib/bb/tests/fetch.py +64 -0
67 lines not on this page · raw message
Signature · 1 line
2.56.1