Skip to contentSkip to conversation

spdx30: identity and purls of dependency downloads

PATCH Superseded Stefano Tondo 2 participants 7 messages Nov 18

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover spdx30: identity and purls of dependency downloads

Hi,

This series implements the parts of the RFC "spdx30: name, version and purl for dependency downloads" [1] that we agreed on. It needs the bitbake series "fetch: package identity of crate, gomod and npm URLs" [2].

1. The SPDX of a crate, gomod or npm download gets the name, the version and the purl from the fetcher.
2. For Cargo recipes, a new do_cargo_metadata task writes the crate dependency graph for the target. The build links to each crate download with a runtime, build or test scope. A crate that is not used on the target gets an annotation.
3. Each output package gets the purls of its source downloads as external identifiers, as Joshua proposed: a statically linked dependency is part of the binary. A purl is left out only when the cargo graph proves that the crate is not in the target. The software_packageUrl of the package does not change.
4. Selftests with python3-bcrypt.

For python3-bcrypt, the output package gets the purls of 25 of its 42 crates. The other 17 crates are not built into the target.

Two points for review:

- All output packages of a recipe get the purls, also -dbg and -src. I have no proof per package, so I followed the rule "keep the purl unless there is proof".
- Go recipes have no filter yet. go version -m can give the proof per binary, and I can add it in a follow-up.

I ran oe-selftest -r spdx.SPDX30Check on master: all 20 tests pass.

[1] https://lore.kernel.org/openembedded-core/[email protected]/
[2] https://lore.kernel.org/bitbake-devel/[email protected]/

Stefano Tondo (4):
spdx30: Take name, version and PURL of downloads from the fetcher
spdx30: Record the lifecycle scope of crate dependencies
spdx30: Add the PURLs of the source downloads to the output packages oeqa/selftest: spdx: Test crate identity, scope and output PURLs

 meta/classes-recipe/cargo.bbclass    |  17 ++++
 meta/lib/oe/spdx30_tasks.py          | 119 +++++++++++++++++++++++++-
 meta/lib/oeqa/selftest/cases/spdx.py | 122 +++++++++++++++++++++++++++
 3 files changed, 257 insertions(+), 1 deletion(-)
Signature · 1 line
2.56.1
#1 lore raw

PATCH 1/4 spdx30: Take name, version and PURL of downloads from the fetcher

From: Stefano Tondo <[email protected]>

The fetchers of package registries (crate, gomod, npm) know the name and the version of each package they download. Today the SPDX of such a download only has the file name, so an SBOM scanner cannot match the dependency.

Ask the fetcher with package_identity() and set the name, the version and the PURL of the download. This needs package_identity() in bitbake.

---
 meta/lib/oe/spdx30_tasks.py | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
meta/lib/oe/spdx30_tasks.py +14 -0
index 6eaa74ce3d..58fe45ec7e 100644
@@ -463,6 +463,19 @@ def enrich_source_package(d, dl, fd, file_name, primary_purpose):
             )
 
 
+def set_package_identity(d, dl, fd):
+    """Set name, version and PURL of a download that names a package.
+
+    Fetchers of package registries (crate, gomod, npm) know which package
+    a URL downloads, see FetchMethod.package_identity() in bitbake.
+    """
+    identity = fd.method.package_identity(fd, d)
+    if identity:
+        dl.name = identity["name"]
+        dl.software_packageVersion = identity["version"]
+        dl.software_packageUrl = identity["purl"]
+
+
 def add_download_files(d, objset):
     inputs = set()
     release_date = get_release_date(d)
@@ -528,6 +541,7 @@ def add_download_files(d, objset):
             )
 
             enrich_source_package(d, dl, fd, file_name, primary_purpose)
+            set_package_identity(d, dl, fd)
 
             if release_date is not None:
                 dl.releaseTime = release_date
Signature · 1 line
2.56.1
#2 lore raw

PATCH 2/4 spdx30: Record the lifecycle scope of crate dependencies

In reply to Stefano Tondo

From: Stefano Tondo <[email protected]>

A Cargo.lock file lists the crates for all targets, and some crates only run at build time (build scripts, proc-macros and the crates they use). The fetcher downloads all of them, but only some end up in the binary.

Add a do_cargo_metadata task that writes the dependency graph for the target with cargo metadata --filter-platform. The create_spdx task reads this graph. It links the build to each crate download with a scoped dependsOn relationship: runtime, build or test. A crate that is not used on the target gets an annotation instead.

---
 meta/classes-recipe/cargo.bbclass | 17 +++++++
 meta/lib/oe/spdx30_tasks.py       | 81 +++++++++++++++++++++++++++++++
 2 files changed, 98 insertions(+)
meta/classes-recipe/cargo.bbclass +17 -0
index 586071988a..b3e82f9d5b 100644
@@ -26,6 +26,23 @@ cargo_do_compile () {
 	"${CARGO}" build ${CARGO_BUILD_FLAGS} ${PACKAGECONFIG_CONFARGS} "$@"
 }
 
+# The crate dependency graph for the target, with the crates that are not
+# used on it removed. create-spdx uses it to give each crate download a
+# lifecycle scope.
+CARGO_METADATA_FILE = "${WORKDIR}/cargo-metadata.json"
+
+do_cargo_metadata () {
+	rm -f "${CARGO_METADATA_FILE}"
+	# Some recipes inherit cargo without being a cargo project
+	if [ ! -f "${CARGO_MANIFEST_PATH}" ]; then
+		return
+	fi
+	"${CARGO}" metadata --format-version 1 --offline --locked \
+		--manifest-path "${CARGO_MANIFEST_PATH}" \
+		--filter-platform "${RUST_HOST_SYS}" > "${CARGO_METADATA_FILE}"
+}
+addtask cargo_metadata after do_configure before do_compile do_create_spdx
+
 cargo_do_install () {
 	local have_installed=false
 	for tgt in "${B}/target/${CARGO_TARGET_SUBDIR}/"*; do
meta/lib/oe/spdx30_tasks.py +81 -0

93 lines not on this page · raw message

Signature · 1 line
2.56.1
#3 lore raw

PATCH 3/4 spdx30: Add the PURLs of the source downloads to the output packages

In reply to Stefano Tondo

From: Stefano Tondo <[email protected]>

Statically linked code, for example a Rust crate, is part of the binary, but the SPDX of the output package only names the recipe. A vulnerability scanner that checks the packages of an image does not find the dependency.

Add the PURLs of the source downloads to the external identifiers of each output package. Leave out the crates that the cargo dependency graph proves are not built into the target. The software_packageUrl of the package stays the first PURL in SPDX_PACKAGE_URLS.

---
 meta/lib/oe/spdx30_tasks.py | 30 ++++++++++++++++++++++++++----
 1 file changed, 26 insertions(+), 4 deletions(-)
meta/lib/oe/spdx30_tasks.py +26 -4
index e40716a6df..441df709a4 100644
@@ -526,16 +526,24 @@ def get_cargo_scopes(d):
 
 
 def add_dependency_scopes(d, objset, build, source_files):
-    """Give crate downloads a lifecycle scope, see get_cargo_scopes()."""
+    """Give crate downloads a lifecycle scope, see get_cargo_scopes().
+
+    Returns the crate downloads that are not built into the target: crates
+    that are not used on it, build scripts, proc-macros and the crates
+    they use, and test-only crates.
+    """
     scopes = get_cargo_scopes(d)
     if scopes is None:
-        return
+        return set()
 
     by_scope = {}
+    not_in_target = set()
     for dl in source_files:
         if not (getattr(dl, "software_packageUrl", None) or "").startswith("pkg:cargo/"):
             continue
         scope = scopes.get((dl.name, dl.software_packageVersion))
+        if scope != "runtime":
+            not_in_target.add(dl)
         if scope:
             by_scope.setdefault(scope, []).append(dl)
         else:
@@ -554,6 +562,8 @@ def add_dependency_scopes(d, objset, build, source_files):
             sorted(oe.sbom30.get_element_link_id(dl) for dl in dls),
         )
 
+    return not_in_target
+
 
 def add_download_files(d, objset):
     inputs = set()
@@ -1057,8 +1067,20 @@ def create_spdx(d):
 
     source_files = add_download_files(d, build_objset)
     build_inputs |= source_files
+
+    # The output packages inherit the PURLs of the downloads they are built
+    # from, as the code of statically linked dependencies is in them. Only
+    # the downloads known not to be built into the target are left out.
+    not_in_target = set()
     if bb.data.inherits_class("cargo", d):
-        add_dependency_scopes(d, build_objset, build, source_files)
+        not_in_target = add_dependency_scopes(d, build_objset, build, source_files)
+    dependency_purls = sorted(
+        {
+            dl.software_packageUrl
+            for dl in source_files
+            if getattr(dl, "software_packageUrl", None) and dl not in not_in_target
+        }
+    )
 
     recipe_spdx_license = add_license_expression(
         d, build_objset, d.getVar("LICENSE"), license_data, [recipe_objset]
@@ -1171,7 +1193,7 @@ def create_spdx(d):
                 d.getVar("SPDX_PACKAGE_URLS:%s" % package)
                 or d.getVar("SPDX_PACKAGE_URLS")
                 or ""
-            ).split()
+            ).split() + dependency_purls
 
             set_purls(spdx_package, purls)
 
Signature · 1 line
2.56.1
#4 lore raw

Re: [OE-core] [PATCH 3/4] spdx30: Add the PURLs of the source downloads to the output packages

Quoted 7 lines · Stefano Tondo via openembedded-core

From: Stefano Tondo <[email protected]>

Statically linked code, for example a Rust crate, is part of the binary, but the SPDX of the output package only names the recipe. A vulnerability scanner that checks the packages of an image does not find the dependency.

Quoted 7 lines

Add the PURLs of the source downloads to the external identifiers of each output package. Leave out the crates that the cargo dependency graph proves are not built into the target. The software_packageUrl of the package stays the first PURL in SPDX_PACKAGE_URLS.

---

Hi Stefano,

Thanks for your patches.

It looks like this is using memory in a way it got killed several times on the autobuilder. Locally, I managed to have a do_create_spdx task using more than 10GB of memory, with a value continuously increasing of about 100MB/s.

This can be reproduced with master branch + your patches, then:
bitbake python3-uv-build-native -c do_create_spdx

Some failed builds for reference:
https://autobuilder.yoctoproject.org/valkyrie/#/builders/110/builds/1605
https://autobuilder.yoctoproject.org/valkyrie/#/builders/6/builds/5058
https://autobuilder.yoctoproject.org/valkyrie/#/builders/6/builds/5060
https://autobuilder.yoctoproject.org/valkyrie/#/builders/25/builds/5039
https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/5061

Can you have a look at the issue?

Thanks,
Mathieu

Signature · 3 lines
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#5 lore raw

Re: [PATCH 3/4] spdx30: Add the PURLs of the source downloads to the output packages

Quoted 7 lines · Mathieu Dubois-Briand

It looks like this is using memory in a way it got killed several times on the autobuilder. Locally, I managed to have a do_create_spdx task using more than 10GB of memory, with a value continuously increasing of about 100MB/s.

This can be reproduced with master branch + your patches, then:
bitbake python3-uv-build-native -c do_create_spdx

[...]

Quoted 1 line

Can you have a look at the issue?

Hi Mathieu,

Thanks for the report and the reproducer, and sorry for the broken builds.

The bug is in patch 2/4, in get_cargo_scopes(). The walk over the cargo dependency graph did not remember which workspace members it had already expanded. In uv, the member uv-preview is a dev dependency of itself. So the walk never ended, and its stack grew until the OOM killer stopped the task. I could reproduce it: the task reached a 4 GB limit in about 90 seconds.

python3-uv-build is the only cargo recipe in OE-Core with such a cycle. The selftests use python3-bcrypt, which has none, so they did not show it.

v2 expands each member only once and adds unit tests for member cycles. With v2, do_create_spdx of python3-uv-build and python3-uv-build-native takes less than 2 seconds and about 100 MB, with uv-build 0.13.7 on master and 0.13.8 on master-next. For the other cargo recipes in OE-Core, the result does not change.

I will send v2 now.

Regards,
Stefano

#6 lore raw

PATCH 4/4 oeqa/selftest: spdx: Test crate identity, scope and output PURLs

In reply to Stefano Tondo

From: Stefano Tondo <[email protected]>

python3-bcrypt fetches 42 crates. The new tests check three things:

- Each crate download has its name, version and PURL.
- Each crate download has a lifecycle scope, or an annotation that it is not built.
- The output package gets the PURLs of the runtime crates only.

The tests name only crates that do not change on recipe upgrades. blowfish is the cipher of bcrypt. Only the pyo3 proc-macros use syn, at build time. The other checks do not use crate names.

---
 meta/lib/oeqa/selftest/cases/spdx.py | 122 +++++++++++++++++++++++++++
 1 file changed, 122 insertions(+)
meta/lib/oeqa/selftest/cases/spdx.py +122 -0
index ce9a8af4f6..d75c6146c7 100644
@@ -9,6 +9,7 @@ import hashlib
 from datetime import datetime, timezone
 from oeqa.selftest.case import OESelftestTestCase
 from oeqa.utils.commands import bitbake, get_bb_var, get_bb_vars
+import oe.purl
 import oe.spdx30
 
 
@@ -543,3 +544,124 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
 
         # Document should be fully linked
         self.check_objset_missing_ids(objset)
+
+    def test_crate_package_identity(self):
+        """Test that crate downloads carry name, version and a cargo PURL."""
+        objset = self.check_recipe_spdx(
+            "python3-bcrypt",
+            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-python3-bcrypt.spdx.json",
+        )
+
+        prefix = "https://static.crates.io/crates/"
+        crates = 0
+        for pkg in objset.foreach_type(oe.spdx30.software_Package):
+            location = pkg.software_downloadLocation or ""
+            if not location.startswith(prefix):
+                continue
+            name, version, _ = location[len(prefix):].split("/")
+            self.assertEqual(pkg.name, name)
+            self.assertEqual(pkg.software_packageVersion, version)
+            self.assertEqual(
+                pkg.software_packageUrl,
+                f"pkg:cargo/{oe.purl.quote(name)}@{oe.purl.quote(version)}",
+            )
+            crates += 1
+
+        self.assertGreater(crates, 0, "no crate downloads found")
+
+    def test_crate_dependency_scope(self):
+        """Test that crate downloads get a lifecycle scope for the target.
+
+        Every crate download is either the target of one scoped dependsOn
+        relationship or annotated as not built. python3-bcrypt fetches crates
+        for other platforms (WebAssembly, UEFI), which are never built for a
+        Linux target. blowfish is the cipher of bcrypt, and syn only runs at
+        build time in proc-macros, so these names do not change on upgrades.
+        """
+        objset = self.check_recipe_spdx(
+            "python3-bcrypt",
+            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-python3-bcrypt.spdx.json",
+        )
+
+        crates = {
+            pkg._id: pkg.name
+            for pkg in objset.foreach_type(oe.spdx30.software_Package)
+            if (pkg.software_packageUrl or "").startswith("pkg:cargo/")
+            and (pkg.software_downloadLocation or "").startswith("https://static.crates.io/")
+        }
+
+        scope_of = {}
+        for rel in objset.foreach_type(oe.spdx30.LifecycleScopedRelationship):
+            if rel.relationshipType != oe.spdx30.RelationshipType.dependsOn:
+                continue
+            for to in rel.to:
+                to_id = to if isinstance(to, str) else to._id
+                if to_id in crates:
+                    self.assertNotIn(crates[to_id], scope_of)
+                    scope_of[crates[to_id]] = rel.scope
+
+        not_built = set()
+        for annotation in objset.foreach_type(oe.spdx30.Annotation):
+            subject = annotation.subject
+            subject_id = subject if isinstance(subject, str) else subject._id
+            if subject_id in crates and "not built" in annotation.statement:
+                not_built.add(crates[subject_id])
+
+        self.assertEqual(set(scope_of) | not_built, set(crates.values()))
+        self.assertFalse(set(scope_of) & not_built)
+        self.assertTrue(not_built, "no crate for another platform found")
+        self.assertEqual(scope_of["syn"], oe.spdx30.LifecycleScopeType.build)
+        self.assertEqual(scope_of["blowfish"], oe.spdx30.LifecycleScopeType.runtime)
+
+    def test_crate_purls_in_output_package(self):
+        """Test that the output package inherits the PURLs of its crates.
+
+        The package gets the PURLs of the crates with the runtime scope in
+        the build SPDX, and no other crate PURLs. syn only runs at build
+        time in proc-macros, so it is not in the binary.
+        """
+        objset = self.check_recipe_spdx(
+            "python3-bcrypt",
+            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/packages/package-python3-bcrypt.spdx.json",
+        )
+
+        package = next(
+            pkg
+            for pkg in objset.foreach_type(oe.spdx30.software_Package)
+            if pkg.name == "python3-bcrypt"
+        )
+        purls = {
+            e.identifier
+            for e in package.externalIdentifier
+            if e.externalIdentifierType == oe.spdx30.ExternalIdentifierType.packageUrl
+        }
+        crates = {p[len("pkg:cargo/"):].split("@")[0] for p in purls if p.startswith("pkg:cargo/")}
+
+        # The package's own PURL stays first
+        own_purls = get_bb_var("SPDX_PACKAGE_URLS", "python3-bcrypt").split()
+        self.assertEqual(package.software_packageUrl, own_purls[0])
+        self.assertTrue(set(own_purls) <= purls)
+        self.assertIn("blowfish", crates)
+        self.assertNotIn("syn", crates)
+
+        spdx_vars = get_bb_vars(["DEPLOY_DIR_SPDX", "SSTATE_PKGARCH"], "python3-bcrypt")
+        build_objset = self.check_spdx_file(
+            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-python3-bcrypt.spdx.json".format(**spdx_vars)
+        )
+        downloads = {
+            pkg._id: pkg.software_packageUrl
+            for pkg in build_objset.foreach_type(oe.spdx30.software_Package)
+            if (pkg.software_packageUrl or "").startswith("pkg:cargo/")
+        }
+        runtime = set()
+        for rel in build_objset.foreach_type(oe.spdx30.LifecycleScopedRelationship):
+            if rel.scope != oe.spdx30.LifecycleScopeType.runtime:
+                continue
+            for to in rel.to:
+                to_id = to if isinstance(to, str) else to._id
+                if to_id in downloads:
+                    runtime.add(downloads[to_id])
+
+        self.assertTrue(runtime, "no crate with the runtime scope found")
+        cargo_purls = {p for p in purls if p.startswith("pkg:cargo/")}
+        self.assertEqual(cargo_purls - set(own_purls), runtime)
Signature · 1 line
2.56.1