Skip to contentSkip to conversation

fetch: package identity of crate, gomod and npm URLs

PATCH Superseded Stefano Tondo 2 participants 4 messages Nov 18 – Nov 19

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover fetch: package identity of crate, gomod and npm URLs

Hi,

This series adds the fetcher API from the RFC "spdx30: name, version and purl for dependency downloads" [1]. Richard asked for a fetcher API that returns structured data with the purl, so that OE-Core does not read fetcher internals.

Patch 1 adds FetchMethod.package_identity(). It returns the name, the version and the purl of the downloaded package, or None. It uses only the URL and the datastore, so it works before the fetch and without network access.

Patch 2 implements it for the crate, gomod, gomodgit and npm fetchers, with unit tests.

The RFC also covered the downloads that gitsm and npmsw add by themselves. That part is not in this series. Richard and Alexander prefer to list those downloads in SRC_URI. I will send that change for npm in a separate series.

An OE-Core series that uses the API follows on the OE-Core list.

[1] https://lore.kernel.org/openembedded-core/[email protected]/

Stefano Tondo (2):
fetch: Add package_identity() to the fetchers
fetch: Return the package identity of crate, gomod and npm URLs

 lib/bb/fetch/__init__.py | 27 +++++++++++++++++
 lib/bb/fetch/crate.py    |  7 +++++
 lib/bb/fetch/gomod.py    | 16 ++++++++++
 lib/bb/fetch/npm.py      |  7 +++++
 lib/bb/tests/fetch.py    | 63 ++++++++++++++++++++++++++++++++++++++++
 5 files changed, 120 insertions(+)
Signature · 1 line
2.56.1
#1 lore raw

PATCH 1/2 fetch: Add package_identity() to the fetchers

From: Stefano Tondo <[email protected]>

Fetchers of package registries know which package a URL downloads:
its name and its version are part of the URL. SBOM generation needs this data, but it can only get it today by parsing fetcher internals.

Add FetchMethod.package_identity(). It returns the name, the version and the Package URL (purl) of the downloaded package as a dict, or None when the URL does not name a package. It uses only the URL and the datastore, so it works before the fetch and without network access. Add bb.fetch.package_url() to build the purl with the correct encoding.

---
 lib/bb/fetch/__init__.py | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
lib/bb/fetch/__init__.py +27 -0
index ce2bd62ad..a4b5860c0 100644
@@ -1765,6 +1765,33 @@ class FetchMethod(object):
         """
         return []
 
+    def package_identity(self, ud, d):
+        """
+        Return the identity of the package that the URL downloads, for
+        fetchers of package registries that know it, as a dict with the
+        keys "name", "version" and "purl" (Package URL). Return None when
+        the URL does not name a package.
+
+        The result depends only on the URL and the datastore. It is
+        available as soon as the FetchData exists, before the fetch and
+        without network access. It is meant for SBOM generation.
+        """
+        return None
+
+
+def package_url(purl_type, name, version, qualifiers=None):
+    """
+    Return a Package URL (https://github.com/package-url/purl-spec) for a
+    package. name can contain "/": the segments before the last one are
+    the namespace. Each segment, the version and the qualifier values are
+    percent-encoded. qualifiers is a dict, written sorted by key.
+    """
+    quote = lambda s: urllib.parse.quote(s, safe="")
+    purl = "pkg:%s/%s@%s" % (purl_type, "/".join(quote(s) for s in name.split("/")), quote(version))
+    if qualifiers:
+        purl += "?" + "&".join("%s=%s" % (k, quote(v)) for k, v in sorted(qualifiers.items()))
+    return purl
+
 
 class DummyUnpackTracer(object):
     """
Signature · 1 line
2.56.1
#2 lore raw

PATCH 2/2 fetch: Return the package identity of crate, gomod and npm URLs

In reply to Stefano Tondo

From: Stefano Tondo <[email protected]>

Implement package_identity() for the crate, gomod, gomodgit and npm fetchers. Packages from a registry other than the default one get a repository_url qualifier in the purl. A gomod URL with mod=1 downloads only the go.mod file, which is not built into anything, so it has no identity.

---
 lib/bb/fetch/crate.py |  7 +++++
 lib/bb/fetch/gomod.py | 16 +++++++++++
 lib/bb/fetch/npm.py   |  7 +++++
 lib/bb/tests/fetch.py | 63 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 93 insertions(+)
lib/bb/fetch/crate.py +7 -0
index 16ea02d8a..bc0b032f2 100644
@@ -103,11 +103,18 @@ class Crate(Wget):
             ud.versionsurl = "%s://%s/%s/versions" % (proto, host, name)
 
         ud.parm['downloadfilename'] = "%s-%s.crate" % (name, version)
+        ud.crate = (name, version, None if host == 'crates.io' else "%s://%s" % (proto, host))
         if 'name' not in ud.parm:
             ud.parm['name'] = '%s-%s' % (name, version)
 
         logger.debug2("Fetching %s to %s" % (ud.url, ud.parm['downloadfilename']))
 
+    def package_identity(self, ud, d):
+        name, version, registry = ud.crate
+        qualifiers = {"repository_url": registry} if registry else None
+        purl = bb.fetch.package_url("cargo", name, version, qualifiers)
+        return {"name": name, "version": version, "purl": purl}
+
     def unpack(self, ud, rootdir, d):
         """
         Uses the crate to build the necessary paths for cargo to utilize it
lib/bb/fetch/gomod.py +16 -0
index a62d643fb..1ec640096 100644
@@ -80,6 +80,13 @@ def escape(path):
     return re.sub(r'([A-Z])', lambda m: '!' + m.group(1).lower(), path)
 
 
+def go_package_identity(ud):
+    module = ud.parm['module']
+    version = ud.parm['version']
+    return {"name": module, "version": version,
+            "purl": bb.fetch.package_url("golang", module, version)}
+
+
 class GoMod(Wget):
     """Class to fetch Go modules from a Go module proxy via wget"""
 
@@ -127,6 +134,12 @@ class GoMod(Wget):
 
         super().urldata_init(ud, d)
 
+    def package_identity(self, ud, d):
+        # A go.mod file alone is not built into anything
+        if ud.parm.get('mod', '0') == '1':
+            return None
+        return go_package_identity(ud)
+
     def unpack(self, ud, rootdir, d):
         """Unpack the module in the module cache."""
 
@@ -218,6 +231,9 @@ class GoModGit(Git):
 
         super().urldata_init(ud, d)
 
+    def package_identity(self, ud, d):
+        return go_package_identity(ud)
+
     def unpack(self, ud, rootdir, d):
         """Unpack the module in the module cache."""
 
lib/bb/fetch/npm.py +7 -0
index c8995ba6b..356002381 100644
@@ -225,6 +225,13 @@ class Npm(FetchMethod):
         ud.needdonestamp = False
         ud.resolvefile = self.localpath(ud, d) + ".resolved"
 
+    def package_identity(self, ud, d):
+        qualifiers = None
+        if ud.registry.rstrip("/") != "https://registry.npmjs.org":
+            qualifiers = {"repository_url": ud.registry}
+        purl = bb.fetch.package_url("npm", ud.package, ud.version, qualifiers)
+        return {"name": ud.package, "version": ud.version, "purl": purl}
+
     def _resolve_proxy_url(self, ud, d):
         """Resolve the tarball URL from the registry and cache it without any checksum.
 
lib/bb/tests/fetch.py +63 -0

66 lines not on this page · raw message

Signature · 1 line
2.56.1
#3 lore raw
In reply to Stefano Tondo
Quoted 15 lines · Stefano Tondo

Hi,

This series adds the fetcher API from the RFC "spdx30: name, version and purl for dependency downloads" [1]. Richard asked for a fetcher API that returns structured data with the purl, so that OE-Core does not read fetcher internals.

Patch 1 adds FetchMethod.package_identity(). It returns the name, the version and the purl of the downloaded package, or None. It uses only the URL and the datastore, so it works before the fetch and without network access.

Patch 2 implements it for the crate, gomod, gomodgit and npm fetchers, with unit tests.

Quoted 8 lines

The RFC also covered the downloads that gitsm and npmsw add by themselves. That part is not in this series. Richard and Alexander prefer to list those downloads in SRC_URI. I will send that change for npm in a separate series.

An OE-Core series that uses the API follows on the OE-Core list.

[1] https://lore.kernel.org/openembedded-core/[email protected]/

Hi Stefano,

I can't speak for Richard but to me this looks like it's moving in the right direction.

As the OE-Core changes rely on package_identity() returning a dict with name, version and purl, I think we should instead make the data structure more explicit and enforcible.

Perhaps a dataclass in bb.fetch, then package_identity() can return an instance of this dataclass?

Something like:

@dataclass
class PackageIdentity:
name: str
version: str
purl: str

Best regards,

Signature · 1 line
Paul Barker