fetch: package identity of crate, gomod and npm URLs
cover fetch: package identity of crate, gomod and npm URLs
Hi,
This series adds the fetcher API from the RFC "spdx30: name, version and purl for dependency downloads" [1]. Richard asked for a fetcher API that returns structured data with the purl, so that OE-Core does not read fetcher internals.
Patch 1 adds FetchMethod.package_identity(). It returns the name, the version and the purl of the downloaded package, or None. It uses only the URL and the datastore, so it works before the fetch and without network access.
Patch 2 implements it for the crate, gomod, gomodgit and npm fetchers, with unit tests.
The RFC also covered the downloads that gitsm and npmsw add by themselves. That part is not in this series. Richard and Alexander prefer to list those downloads in SRC_URI. I will send that change for npm in a separate series.
An OE-Core series that uses the API follows on the OE-Core list.
[1] https://lore.kernel.org/openembedded-core/[email protected]/
Stefano Tondo (2):
fetch: Add package_identity() to the fetchers
fetch: Return the package identity of crate, gomod and npm URLs
lib/bb/fetch/__init__.py | 27 +++++++++++++++++ lib/bb/fetch/crate.py | 7 +++++ lib/bb/fetch/gomod.py | 16 ++++++++++ lib/bb/fetch/npm.py | 7 +++++ lib/bb/tests/fetch.py | 63 ++++++++++++++++++++++++++++++++++++++++ 5 files changed, 120 insertions(+)
Signature · 1 line
2.56.1
PATCH 1/2 fetch: Add package_identity() to the fetchers
From: Stefano Tondo <[email protected]>
Fetchers of package registries know which package a URL downloads:
its name and its version are part of the URL. SBOM generation needs this data, but it can only get it today by parsing fetcher internals.
Add FetchMethod.package_identity(). It returns the name, the version and the Package URL (purl) of the downloaded package as a dict, or None when the URL does not name a package. It uses only the URL and the datastore, so it works before the fetch and without network access. Add bb.fetch.package_url() to build the purl with the correct encoding.
--- lib/bb/fetch/__init__.py | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
lib/bb/fetch/__init__.py +27 -0
@@ -1765,6 +1765,33 @@ class FetchMethod(object): """ return [] + def package_identity(self, ud, d): + """ + Return the identity of the package that the URL downloads, for + fetchers of package registries that know it, as a dict with the + keys "name", "version" and "purl" (Package URL). Return None when + the URL does not name a package. + + The result depends only on the URL and the datastore. It is + available as soon as the FetchData exists, before the fetch and + without network access. It is meant for SBOM generation. + """ + return None + + +def package_url(purl_type, name, version, qualifiers=None): + """ + Return a Package URL (https://github.com/package-url/purl-spec) for a + package. name can contain "/": the segments before the last one are + the namespace. Each segment, the version and the qualifier values are + percent-encoded. qualifiers is a dict, written sorted by key. + """ + quote = lambda s: urllib.parse.quote(s, safe="") + purl = "pkg:%s/%s@%s" % (purl_type, "/".join(quote(s) for s in name.split("/")), quote(version)) + if qualifiers: + purl += "?" + "&".join("%s=%s" % (k, quote(v)) for k, v in sorted(qualifiers.items())) + return purl + class DummyUnpackTracer(object): """
Signature · 1 line
2.56.1
PATCH 2/2 fetch: Return the package identity of crate, gomod and npm URLs
In reply to Stefano TondoFrom: Stefano Tondo <[email protected]>
Implement package_identity() for the crate, gomod, gomodgit and npm fetchers. Packages from a registry other than the default one get a repository_url qualifier in the purl. A gomod URL with mod=1 downloads only the go.mod file, which is not built into anything, so it has no identity.
--- lib/bb/fetch/crate.py | 7 +++++ lib/bb/fetch/gomod.py | 16 +++++++++++ lib/bb/fetch/npm.py | 7 +++++ lib/bb/tests/fetch.py | 63 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 93 insertions(+)
lib/bb/fetch/crate.py +7 -0
@@ -103,11 +103,18 @@ class Crate(Wget): ud.versionsurl = "%s://%s/%s/versions" % (proto, host, name) ud.parm['downloadfilename'] = "%s-%s.crate" % (name, version) + ud.crate = (name, version, None if host == 'crates.io' else "%s://%s" % (proto, host)) if 'name' not in ud.parm: ud.parm['name'] = '%s-%s' % (name, version) logger.debug2("Fetching %s to %s" % (ud.url, ud.parm['downloadfilename'])) + def package_identity(self, ud, d): + name, version, registry = ud.crate + qualifiers = {"repository_url": registry} if registry else None + purl = bb.fetch.package_url("cargo", name, version, qualifiers) + return {"name": name, "version": version, "purl": purl} + def unpack(self, ud, rootdir, d): """ Uses the crate to build the necessary paths for cargo to utilize it
lib/bb/fetch/gomod.py +16 -0
@@ -80,6 +80,13 @@ def escape(path): return re.sub(r'([A-Z])', lambda m: '!' + m.group(1).lower(), path) +def go_package_identity(ud): + module = ud.parm['module'] + version = ud.parm['version'] + return {"name": module, "version": version, + "purl": bb.fetch.package_url("golang", module, version)} + + class GoMod(Wget): """Class to fetch Go modules from a Go module proxy via wget"""
@@ -127,6 +134,12 @@ class GoMod(Wget): super().urldata_init(ud, d) + def package_identity(self, ud, d): + # A go.mod file alone is not built into anything + if ud.parm.get('mod', '0') == '1': + return None + return go_package_identity(ud) + def unpack(self, ud, rootdir, d): """Unpack the module in the module cache."""
@@ -218,6 +231,9 @@ class GoModGit(Git): super().urldata_init(ud, d) + def package_identity(self, ud, d): + return go_package_identity(ud) + def unpack(self, ud, rootdir, d): """Unpack the module in the module cache."""
lib/bb/fetch/npm.py +7 -0
@@ -225,6 +225,13 @@ class Npm(FetchMethod): ud.needdonestamp = False ud.resolvefile = self.localpath(ud, d) + ".resolved" + def package_identity(self, ud, d): + qualifiers = None + if ud.registry.rstrip("/") != "https://registry.npmjs.org": + qualifiers = {"repository_url": ud.registry} + purl = bb.fetch.package_url("npm", ud.package, ud.version, qualifiers) + return {"name": ud.package, "version": ud.version, "purl": purl} + def _resolve_proxy_url(self, ud, d): """Resolve the tarball URL from the registry and cache it without any checksum.
lib/bb/tests/fetch.py +63 -0
66 lines not on this page · raw message
Signature · 1 line
2.56.1
Quoted 15 lines · Stefano Tondo
Hi,
This series adds the fetcher API from the RFC "spdx30: name, version and purl for dependency downloads" [1]. Richard asked for a fetcher API that returns structured data with the purl, so that OE-Core does not read fetcher internals.
Patch 1 adds
FetchMethod.package_identity(). It returns the name, the version and the purl of the downloaded package, or None. It uses only the URL and the datastore, so it works before the fetch and without network access.Patch 2 implements it for the crate, gomod, gomodgit and npm fetchers, with unit tests.
Quoted 8 lines
The RFC also covered the downloads that gitsm and npmsw add by themselves. That part is not in this series. Richard and Alexander prefer to list those downloads in
SRC_URI. I will send that change for npm in a separate series.An OE-Core series that uses the API follows on the OE-Core list.
[1] https://lore.kernel.org/openembedded-core/[email protected]/
Hi Stefano,
I can't speak for Richard but to me this looks like it's moving in the right direction.
As the OE-Core changes rely on package_identity() returning a dict with name, version and purl, I think we should instead make the data structure more explicit and enforcible.
Perhaps a dataclass in bb.fetch, then package_identity() can return an instance of this dataclass?
Something like:
@dataclass
class PackageIdentity:
name: str
version: str
purl: str
Best regards,
Signature · 1 line
Paul Barker