Skip to contentSkip to conversation

Add SPDX 3 Recipe Information

PATCH Superseded Joshua Watt 5 participants 113 messages Feb 20 – Mar 19

lore
Share & tools
lore
#0 lore raw

cover Add SPDX 3 Recipe Information

Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.

Special thanks to Benjamin Robin <[email protected]> for helping work through this.

Joshua Watt (9):
llvm-project-source: Use allarch.bbclass
gcc-source: Use allarch.bbclass
spdx3: Add recipe SPDX data
spdx3: Add recipe SBoM task
spdx3: Add is-native property
spdx30: Include patch file information in VEX
spdx: De-duplicate CreationInfo
spdx: Ignore ASSUME_PROVIDED recipes
spdx_common: Check for dependent task in task flags

 meta/classes-global/sstate.bbclass            |   4 +-
 meta/classes-global/staging.bbclass           |   2 +-
 .../create-spdx-image-3.0.bbclass             |   4 +-
 .../create-spdx-sdk-3.0.bbclass               |   4 +-
 meta/classes-recipe/kernel.bbclass            |   2 +-
 meta/classes-recipe/nospdx.bbclass            |   1 +
 meta/classes/create-spdx-2.2.bbclass          |  12 +-
 meta/classes/create-spdx-3.0.bbclass          |  77 ++-
 meta/classes/spdx-common.bbclass              |  14 +-
 meta/lib/oe/sbom30.py                         | 192 ++++---
 meta/lib/oe/spdx30.py                         |   2 +-
 meta/lib/oe/spdx30_tasks.py                   | 487 +++++++++++++-----
 meta/lib/oe/spdx_common.py                    |  16 +-
 .../meta/meta-world-recipe-sbom.bb            |  26 +
 .../clang/llvm-project-source.inc             |   8 +-
 meta/recipes-devtools/gcc/gcc-source.inc      |  16 +-
 16 files changed, 620 insertions(+), 247 deletions(-)
 create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
Signature · 1 line
2.53.0
#1 lore raw

PATCH 1/9 llvm-project-source: Use allarch.bbclass

Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)

---
 meta/recipes-devtools/clang/llvm-project-source.inc | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)
meta/recipes-devtools/clang/llvm-project-source.inc +1 -7
index 13e54efbc2..6bb595b7bc 100644
@@ -5,7 +5,7 @@ deltask do_populate_sysroot
 deltask do_populate_lic
 RM_WORK_EXCLUDE += "${PN}"
 
-inherit nopackages
+inherit nopackages allarch
 
 PN = "llvm-project-source-${PV}"
 WORKDIR = "${TMPDIR}/work-shared/llvm-project-source-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:llvm-project-source::${PV}:${PR}::${SSTATE_VERSION}:"
 STAMP = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-${PR}"
 STAMPCLEAN = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-*"
 
-INHIBIT_DEFAULT_DEPS = "1"
 DEPENDS = ""
 PACKAGES = ""
-TARGET_ARCH = "allarch"
-TARGET_AS_ARCH = "none"
-TARGET_CC_ARCH = "none"
-TARGET_LD_ARCH = "none"
-TARGET_OS = "linux"
 baselib = "lib"
 PACKAGE_ARCH = "all"
 
Signature · 1 line
2.53.0
#2 lore raw

PATCH 2/9 gcc-source: Use allarch.bbclass

In reply to Joshua Watt

Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)

---
 meta/recipes-devtools/gcc/gcc-source.inc | 16 +++++-----------
 1 file changed, 5 insertions(+), 11 deletions(-)
meta/recipes-devtools/gcc/gcc-source.inc +5 -11
index 265bcf4bef..3ac679b1a6 100644
@@ -1,11 +1,11 @@
-deltask do_configure 
-deltask do_compile 
-deltask do_install 
+deltask do_configure
+deltask do_compile
+deltask do_install
 deltask do_populate_sysroot
-deltask do_populate_lic 
+deltask do_populate_lic
 RM_WORK_EXCLUDE += "${PN}"
 
-inherit nopackages
+inherit nopackages allarch
 
 PN = "gcc-source-${PV}"
 WORKDIR = "${TMPDIR}/work-shared/gcc-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:gcc::${PV}:${PR}::${SSTATE_VERSION}:"
 STAMP = "${STAMPS_DIR}/work-shared/gcc-${PV}-${PR}"
 STAMPCLEAN = "${STAMPS_DIR}/work-shared/gcc-${PV}-*"
 
-INHIBIT_DEFAULT_DEPS = "1"
 DEPENDS = ""
 PACKAGES = ""
-TARGET_ARCH = "allarch"
-TARGET_AS_ARCH = "none"
-TARGET_CC_ARCH = "none"
-TARGET_LD_ARCH = "none"
-TARGET_OS = "linux"
 baselib = "lib"
 PACKAGE_ARCH = "all"
 
Signature · 1 line
2.53.0
#3 lore raw

PATCH 3/9 spdx3: Add recipe SPDX data

In reply to Joshua Watt

Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.

Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch

---
 meta/classes-global/sstate.bbclass            |   4 +-
 meta/classes-global/staging.bbclass           |   2 +-
 .../create-spdx-image-3.0.bbclass             |   4 +-
 .../create-spdx-sdk-3.0.bbclass               |   4 +-
 meta/classes-recipe/kernel.bbclass            |   2 +-
 meta/classes-recipe/nospdx.bbclass            |   1 +
 meta/classes/create-spdx-2.2.bbclass          |  12 +-
 meta/classes/create-spdx-3.0.bbclass          |  51 ++-
 meta/classes/spdx-common.bbclass              |  13 +-
 meta/lib/oe/spdx30_tasks.py                   | 402 ++++++++++++------
 10 files changed, 343 insertions(+), 152 deletions(-)
meta/classes-global/sstate.bbclass +2 -2
index 2fd29d7323..95c44f404e 100644
@@ -954,7 +954,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,
             extrapath = d.getVar("NATIVELSBSTRING") + "/"
         else:
             extrapath = ""
-        
+
         tname = bb.runqueue.taskname_from_tid(task)[3:]
 
         if tname in ["fetch", "unpack", "patch", "populate_lic", "preconfigure"] and splithashfn[2]:
@@ -1116,7 +1116,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None):
 
     logit("Considering setscene task: %s" % (str(taskdependees[task])), log)
 
-    directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_deploy_archives"]
+    directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_create_recipe_spdx", "do_deploy_archives"]
 
     def isNativeCross(x):
         return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross")
meta/classes-global/staging.bbclass +1 -1
index 1008867a6c..b35bc9037f 100644
@@ -654,7 +654,7 @@ addtask do_prepare_recipe_sysroot before do_configure after do_fetch
 python staging_taskhandler() {
     EXCLUDED_TASKS = (
         "do_prepare_recipe_sysroot",
-        "do_create_spdx",
+        "do_create_recipe_spdx",
     )
     bbtasks = e.tasklist
     for task in bbtasks:
meta/classes-recipe/create-spdx-image-3.0.bbclass +2 -2
index 636ab14eb0..15a91e90e2 100644
@@ -34,7 +34,7 @@ addtask do_create_rootfs_spdx after do_rootfs before do_image
 SSTATETASKS += "do_create_rootfs_spdx"
 do_create_rootfs_spdx[sstate-inputdirs] = "${SPDXROOTFSDEPLOY}"
 do_create_rootfs_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}"
-do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_create_rootfs_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_create_rootfs_spdx[cleandirs] += "${SPDXROOTFSDEPLOY}"
 do_create_rootfs_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 
@@ -76,7 +76,7 @@ do_create_image_sbom_spdx[sstate-inputdirs] = "${SPDXIMAGEDEPLOYDIR}"
 do_create_image_sbom_spdx[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
 do_create_image_sbom_spdx[stamp-extra-info] = "${MACHINE_ARCH}"
 do_create_image_sbom_spdx[cleandirs] = "${SPDXIMAGEDEPLOYDIR}"
-do_create_image_sbom_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_create_image_sbom_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_create_image_sbom_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 
 python do_create_image_sbom_spdx_setscene() {
meta/classes-recipe/create-spdx-sdk-3.0.bbclass +2 -2
index e5f220cdfa..a4b8ed3bf9 100644
@@ -5,14 +5,14 @@
 #
 # SPDX SDK tasks
 
-do_populate_sdk[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_populate_sdk[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_populate_sdk[cleandirs] += "${SPDXSDKWORK}"
 do_populate_sdk[postfuncs] += "sdk_create_sbom"
 do_populate_sdk[file-checksums] += "${SPDX3_DEP_FILES}"
 POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_create_spdx"
 POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_create_spdx"
 
-do_populate_sdk_ext[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_populate_sdk_ext[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_populate_sdk_ext[cleandirs] += "${SPDXSDKEXTWORK}"
 do_populate_sdk_ext[postfuncs] += "sdk_ext_create_sbom"
 do_populate_sdk_ext[file-checksums] += "${SPDX3_DEP_FILES}"
meta/classes-recipe/kernel.bbclass +1 -1
index f989b31c47..3a2c20dec2 100644
@@ -904,7 +904,7 @@ python do_create_kernel_config_spdx() {
             bb.error(f"Failed to parse kernel config file: {e}")
 
         build, build_objset = oe.sbom30.find_root_obj_in_jsonld(
-            d, "recipes", f"recipe-{pn}", oe.spdx30.build_Build
+            d, "builds", f"build-{pn}", oe.spdx30.build_Build
         )
 
         kernel_build = build_objset.add_root(
meta/classes-recipe/nospdx.bbclass +1 -0
index b20e28218b..90e14442ba 100644
@@ -5,6 +5,7 @@
 #
 
 deltask do_collect_spdx_deps
+deltask do_create_recipe_spdx
 deltask do_create_spdx
 deltask do_create_spdx_runtime
 deltask do_create_package_spdx
meta/classes/create-spdx-2.2.bbclass +11 -1
index 65d10d86db..f1ee0f9afd 100644
@@ -399,6 +399,15 @@ def get_license_list_version(license_data, d):
     return ".".join(license_data["licenseListVersion"].split(".")[:2])
 
 
+# This task is added for compatibility with tasks shared with SPDX 3, but
+# doesn't do anything
+do_create_recipe_spdx() {
+    :
+}
+do_create_recipe_spdx[noexec] = "1"
+addtask do_create_recipe_spdx after do_collect_spdx_deps
+
+
 python do_create_spdx() {
     from datetime import datetime, timezone
     import oe.sbom
@@ -594,7 +603,7 @@ python do_create_spdx() {
 }
 do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS"
 # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source
-addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work
+addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work
 
 SSTATETASKS += "do_create_spdx"
 do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}"
@@ -605,6 +614,7 @@ python do_create_spdx_setscene () {
 }
 addtask do_create_spdx_setscene
 
+do_create_spdx[deptask] += "do_create_spdx"
 do_create_spdx[dirs] = "${SPDXWORK}"
 do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}"
 do_create_spdx[depends] += " \
meta/classes/create-spdx-3.0.bbclass +38 -13

90 lines not on this page · raw message

meta/classes/spdx-common.bbclass +6 -7
index 3110230c9e..0c1fd09b6f 100644
@@ -23,6 +23,7 @@ SPDXDEPS = "${SPDXDIR}/deps.json"
 SPDX_TOOL_NAME ??= "oe-spdx-creator"
 SPDX_TOOL_VERSION ??= "1.0"
 
+SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy"
 SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy"
 
 SPDX_INCLUDE_SOURCES ??= "0"
@@ -97,8 +98,8 @@ python do_collect_spdx_deps() {
     # This task calculates the build time dependencies of the recipe, and is
     # required because while a task can deptask on itself, those dependencies
     # do not show up in BB_TASKDEPDATA. To work around that, this task does the
-    # deptask on do_create_spdx and writes out the dependencies it finds, then
-    # do_create_spdx reads in the found dependencies when writing the actual
+    # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then
+    # downstream tasks read in the found dependencies when writing the actual
     # SPDX document
     import json
     import oe.spdx_common
@@ -106,15 +107,13 @@ python do_collect_spdx_deps() {
 
     spdx_deps_file = Path(d.getVar("SPDXDEPS"))
 
-    deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx")
+    deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx")
 
     with spdx_deps_file.open("w") as f:
         json.dump(deps, f)
 }
-# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source
-addtask do_collect_spdx_deps after do_unpack
-do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}"
-do_collect_spdx_deps[deptask] = "do_create_spdx"
+addtask do_collect_spdx_deps
+do_collect_spdx_deps[deptask] = "do_create_recipe_spdx"
 do_collect_spdx_deps[dirs] = "${SPDXDIR}"
 
 oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA"
meta/lib/oe/spdx30_tasks.py +279 -123

559 lines not on this page · raw message

Signature · 1 line
2.53.0
#4 lore raw

Re: [OE-core][PATCH 3/9] spdx3: Add recipe SPDX data

Quoted 7 lines · Joshua Watt via lists.openembedded.org

Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.

Quoted 5 lines

Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch

---

Hi Joshua,

It looks like this is causing some failures, at list for glib:

ERROR: glib-2.0-1_2.86.4-r0 do_create_spdx: Error executing a python function in exec_func_python() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>

     0001:
 *** 0002:extend_recipe_sysroot(d)
     0003:

File: '/srv/pokybuild/yocto-worker/musl-qemux86/build/layers/openembedded-core/meta/classes-global/staging.bbclass', lineno: 619, function: extend_recipe_sysroot
0615: if "/bin/" in l or "/sbin/" in l:
0616: # defer /*bin/* files until last in case they need libs
0617: binfiles[l] = (targetdir, dest)
0618: else:
*** 0619: staging_copyfile(l, targetdir, dest, postinsts, seendirs)
0620:
0621: # Handle deferred binfiles
0622: for l in binfiles:
0623: (targetdir, dest) = binfiles[l]
File: '/srv/pokybuild/yocto-worker/musl-qemux86/build/layers/openembedded-core/meta/classes-global/staging.bbclass', lineno: 165, function: staging_copyfile

     0161:        os.symlink(linkto, dest)
     0162:        #bb.warn(c)
     0163:    else:
     0164:        try:

*** 0165: os.link(c, dest)
0166: except OSError as err:
0167: if err.errno == errno.EXDEV:
0168: bb.utils.copyfile(c, dest)
0169: else:
Exception: FileExistsError: [Errno 17] File exists: '/srv/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/sysroots-components/core2-32/glib-2.0/usr/include/glib-2.0/glib.h' -> '/srv/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/work/core2-32-poky-linux-musl/glib-2.0/2.86.4/recipe-sysroot/usr/include/glib-2.0/glib.h'

https://autobuilder.yoctoproject.org/valkyrie/#/builders/6/builds/3240
https://autobuilder.yoctoproject.org/valkyrie/#/builders/19/builds/3220
https://autobuilder.yoctoproject.org/valkyrie/#/builders/65/builds/3201
https://autobuilder.yoctoproject.org/valkyrie/#/builders/68/builds/3296

And probably also this one:

Initialising tasks...ERROR: Task virtual:multilib:lib32:/srv/pokybuild/yocto-worker/qemux86-world/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx has circular dependency on /srv/pokybuild/yocto-worker/qemux86-world/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx
ERROR: Command execution failed: 1

https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/3219

Can you have a look?

Thanks,
Mathieu

Signature · 3 lines
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#5 lore raw

PATCH 4/9 spdx3: Add recipe SBoM task

In reply to Joshua Watt

Adds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example:

```
bitbake -c create_recipe_sbom zstd
```

Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively).

The complete SBoM for all (target) recipes can be built with:

```
bitbake -c create_recipe_sbom meta-world-recipe-sbom
```
---
 meta/classes/create-spdx-3.0.bbclass          | 28 +++++++++++++++++++
 meta/classes/spdx-common.bbclass              |  1 +
 meta/lib/oe/spdx30_tasks.py                   | 10 +++++++
 .../meta/meta-world-recipe-sbom.bb            | 26 +++++++++++++++++
 4 files changed, 65 insertions(+)
 create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
meta/classes/create-spdx-3.0.bbclass +28 -0
index cd70a07534..e12c116486 100644
@@ -240,6 +240,34 @@ python do_create_package_spdx_setscene () {
 }
 addtask do_create_package_spdx_setscene
 
+addtask do_create_recipe_sbom after create_recipe_spdx
+python do_create_recipe_sbom() {
+    import oe.spdx30_tasks
+    from pathlib import Path
+    deploydir = Path(d.getVar("SPDXRECIPESBOMDEPLOY"))
+    oe.spdx30_tasks.create_recipe_sbom(d, deploydir)
+}
+
+SSTATETASKS += "do_create_recipe_sbom"
+do_create_recipe_sbom[recrdeptask] = "do_create_recipe_spdx"
+do_create_recipe_sbom[nostamp] = "1"
+do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}"
+do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
+do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}"
+do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}"
+do_create_recipe_sbom[vardeps] += "\
+    SPDX_INCLUDE_BITBAKE_PARENT_BUILD \
+    SPDX_PACKAGE_ADDITIONAL_PURPOSE \
+    SPDX_PROFILES \
+    SPDX_NAMESPACE_PREFIX \
+    SPDX_UUID_NAMESPACE \
+    "
+
+python do_create_recipe_sbom_setscene () {
+    sstate_setscene(d)
+}
+addtask do_create_recipe_sbom_setscene
+
 python spdx30_build_started_handler () {
     import oe.spdx30_tasks
     d = e.data.createCopy()
meta/classes/spdx-common.bbclass +1 -0
index 0c1fd09b6f..6f35dbf8f6 100644
@@ -25,6 +25,7 @@ SPDX_TOOL_VERSION ??= "1.0"
 
 SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy"
 SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy"
+SPDXRECIPESBOMDEPLOY = "${SPDXDIR}/recipes-bom-deploy"
 
 SPDX_INCLUDE_SOURCES ??= "0"
 SPDX_INCLUDE_COMPILED_SOURCES ??= "0"
meta/lib/oe/spdx30_tasks.py +10 -0
index a8b4525e3d..9a312a870d 100644
@@ -1564,3 +1564,13 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname):
     oe.sbom30.write_jsonld_doc(
         d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json")
     )
+
+
+def create_recipe_sbom(d, deploydir):
+    sbom_name = d.getVar("PN") + "-recipe-sbom"
+
+    recipe, recipe_objset = load_recipe_spdx(d)
+
+    objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset])
+
+    oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json"))
meta/recipes-core/meta/meta-world-recipe-sbom.bb new file +26 -0
new file mode 100644
index 0000000000..60209fba7e
@@ -0,0 +1,26 @@
+SUMMARY = "Generates a combined SBoM for all world recipes"
+LICENSE = "MIT"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+PACKAGE_ARCH = "${MACHINE_ARCH}"
+
+inherit nopackages
+deltask do_fetch
+deltask do_unpack
+deltask do_patch
+deltask do_configure
+deltask do_compile
+deltask do_install
+
+do_prepare_recipe_sysroot[deptask] = ""
+
+WORLD_SBOM_EXCLUDE ?= ""
+
+python calculate_extra_depends() {
+    exclude = set('${WORLD_SBOM_EXCLUDE}'.split())
+    for p in world_target:
+        if p == self_pn or p in exclude:
+            continue
+        deps.append(p)
+}
Signature · 1 line
2.53.0
#6 lore raw

PATCH 5/9 spdx3: Add is-native property

In reply to Joshua Watt

Adds a custom is-native property to the recipe package to indicate if it is a native recipe

---
 meta/lib/oe/sbom30.py       | 20 ++++++++++++++++++++
 meta/lib/oe/spdx30_tasks.py | 18 +++++++++++-------
 2 files changed, 31 insertions(+), 7 deletions(-)
meta/lib/oe/sbom30.py +20 -0
index 227ac51877..50a72fce39 100644
@@ -118,6 +118,26 @@ class OEDocumentExtension(oe.spdx30.extension_Extension):
         )
 
 
[email protected](OE_SPDX_BASE + "recipe-extension")
+class OERecipeExtension(oe.spdx30.extension_Extension):
+    """
+    This extension is added to recipe software_Packages to indicate various
+    useful bits of information about the recipe
+    """
+
+    CLOSED = True
+
+    @classmethod
+    def _register_props(cls):
+        super()._register_props()
+        cls._add_property(
+            "is_native",
+            oe.spdx30.BooleanProp(),
+            OE_SPDX_BASE + "is-native",
+            max_count=1,
+        )
+
+
 def spdxid_hash(*items):
     h = hashlib.md5()
     for i in items:
meta/lib/oe/spdx30_tasks.py +11 -7
index 9a312a870d..fff1ca6bea 100644
@@ -477,6 +477,10 @@ def set_purls(spdx_package, purls):
         )
 
 
+def get_is_native(d):
+    return bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d)
+
+
 def create_recipe_spdx(d):
     deploydir = Path(d.getVar("SPDXRECIPEDEPLOY"))
     deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
@@ -507,6 +511,11 @@ def create_recipe_spdx(d):
         )
     )
 
+    if get_is_native(d):
+        ext = oe.sbom30.OERecipeExtension()
+        ext.is_native = True
+        recipe.extension.append(ext)
+
     set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split())
 
     # TODO: This doesn't work before do_unpack because the license text has to
@@ -668,9 +677,7 @@ def create_spdx(d):
     spdx_workdir = Path(d.getVar("SPDXWORK"))
     include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1"
     pkg_arch = d.getVar("SSTATE_PKGARCH")
-    is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class(
-        "cross", d
-    )
+    is_native = get_is_native(d)
 
     recipe, recipe_objset = load_recipe_spdx(d)
 
@@ -1021,14 +1028,11 @@ def create_spdx(d):
 def create_package_spdx(d):
     deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
     deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY"))
-    is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class(
-        "cross", d
-    )
 
     providers = oe.spdx_common.collect_package_providers(d)
     pkg_arch = d.getVar("SSTATE_PKGARCH")
 
-    if is_native:
+    if get_is_native(d):
         return
 
     bb.build.exec_func("read_subpackage_metadata", d)
Signature · 1 line
2.53.0
#7 lore raw

PATCH 6/9 spdx30: Include patch file information in VEX

In reply to Joshua Watt

Modifies the SPDX VEX output to include the patches that fix a particular vulnerability. This is done by adding a patchedBy relationship from the VexFixedVulnAssessmentRelationship to the File that provides the fix.

If the file can be located without fetching (e.g. is a file:// in SRC_URI), the checksum will be included.

---
 meta/lib/oe/sbom30.py       | 60 ++++++++++++++-------------
 meta/lib/oe/spdx30_tasks.py | 81 ++++++++++++++++++++++++++++---------
 2 files changed, 92 insertions(+), 49 deletions(-)
meta/lib/oe/sbom30.py +31 -29
index 50a72fce39..21f084dc16 100644
@@ -620,37 +620,38 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
         )
         spdx_file.extension.append(OELicenseScannedExtension())
 
-    def new_file(self, _id, name, path, *, purposes=[]):
-        sha256_hash = bb.utils.sha256_file(path)
+    def new_file(self, _id, name, path, *, purposes=[], hashfile=True):
+        if hashfile:
+            sha256_hash = bb.utils.sha256_file(path)
 
-        for f in self.by_sha256_hash.get(sha256_hash, []):
-            if not isinstance(f, oe.spdx30.software_File):
-                continue
+            for f in self.by_sha256_hash.get(sha256_hash, []):
+                if not isinstance(f, oe.spdx30.software_File):
+                    continue
 
-            if purposes:
-                new_primary = purposes[0]
-                new_additional = []
+                if purposes:
+                    new_primary = purposes[0]
+                    new_additional = []
 
-                if f.software_primaryPurpose:
-                    new_additional.append(f.software_primaryPurpose)
-                new_additional.extend(f.software_additionalPurpose)
+                    if f.software_primaryPurpose:
+                        new_additional.append(f.software_primaryPurpose)
+                    new_additional.extend(f.software_additionalPurpose)
 
-                new_additional = sorted(
-                    list(set(p for p in new_additional if p != new_primary))
-                )
+                    new_additional = sorted(
+                        list(set(p for p in new_additional if p != new_primary))
+                    )
 
-                f.software_primaryPurpose = new_primary
-                f.software_additionalPurpose = new_additional
+                    f.software_primaryPurpose = new_primary
+                    f.software_additionalPurpose = new_additional
 
-            if f.name != name:
-                for e in f.extension:
-                    if isinstance(e, OEFileNameAliasExtension):
-                        e.aliases.append(name)
-                        break
-                else:
-                    f.extension.append(OEFileNameAliasExtension(aliases=[name]))
+                if f.name != name:
+                    for e in f.extension:
+                        if isinstance(e, OEFileNameAliasExtension):
+                            e.aliases.append(name)
+                            break
+                    else:
+                        f.extension.append(OEFileNameAliasExtension(aliases=[name]))
 
-            return f
+                return f
 
         spdx_file = oe.spdx30.software_File(
             _id=_id,
@@ -661,12 +662,13 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
             spdx_file.software_primaryPurpose = purposes[0]
             spdx_file.software_additionalPurpose = purposes[1:]
 
-        spdx_file.verifiedUsing.append(
-            oe.spdx30.Hash(
-                algorithm=oe.spdx30.HashAlgorithm.sha256,
-                hashValue=sha256_hash,
+        if hashfile:
+            spdx_file.verifiedUsing.append(
+                oe.spdx30.Hash(
+                    algorithm=oe.spdx30.HashAlgorithm.sha256,
+                    hashValue=sha256_hash,
+                )
             )
-        )
 
         return self.add(spdx_file)
 
meta/lib/oe/spdx30_tasks.py +61 -20

139 lines not on this page · raw message

Signature · 1 line
2.53.0
#8 lore raw

PATCH 7/9 spdx: De-duplicate CreationInfo

In reply to Joshua Watt

De-duplicates CreationInfo objects that are identical (except for ID) when writing out an SBoM. This significantly reduces the number of CreationInfo objects that end up in the final document.

---
 meta/lib/oe/sbom30.py | 112 ++++++++++++++++++++++++++++++------------
 meta/lib/oe/spdx30.py |   2 +-
 2 files changed, 81 insertions(+), 33 deletions(-)
meta/lib/oe/sbom30.py +80 -32
index 21f084dc16..55a2863d2d 100644
@@ -14,6 +14,7 @@ import uuid
 import os
 import oe.spdx_common
 from datetime import datetime, timezone
+from contextlib import contextmanager
 
 OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/"
 
@@ -191,6 +192,25 @@ def to_list(l):
     return l
 
 
+class Dedup(object):
+    def __init__(self, objset):
+        self.unique = set()
+        self.dedup = {}
+        self.objset = objset
+
+    def find_duplicates(self, cmp, typ, **kwargs):
+        for o in self.objset.foreach_filter(typ, **kwargs):
+            for u in self.unique:
+                if cmp(u, o):
+                    self.dedup[o] = u
+                    break
+            else:
+                self.unique.add(o)
+
+    def get(self, o):
+        return self.dedup.get(o, o)
+
+
 class ObjectSet(oe.spdx30.SHACLObjectSet):
     def __init__(self, d):
         super().__init__()
@@ -895,6 +915,45 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
         self.missing_ids -= set(imports.keys())
         return self.missing_ids
 
+    @contextmanager
+    def deduplicate(self):
+        d = Dedup(self)
+
+        yield d
+
+        visited = set()
+
+        def visit(o, path):
+            if isinstance(o, oe.spdx30.SHACLObject):
+                if o in visited:
+                    return False
+                visited.add(o)
+
+                for k in o:
+                    v = o[k]
+                    if isinstance(v, oe.spdx30.SHACLObject):
+                        o[k] = d.get(v)
+
+            elif isinstance(o, oe.spdx30.ListProxy):
+                for idx, v in enumerate(o):
+                    if isinstance(v, oe.spdx30.SHACLObject):
+                        o[idx] = d.get(v)
+
+            return True
+
+        if d.dedup:
+            for o in self.objects:
+                o.walk(visit)
+
+            for k, v in d.dedup.items():
+                bb.debug(
+                    1,
+                    f"Removing duplicate {k.__class__.__name__} {k._id or id(k)} -> {v._id or id(v)}",
+                )
+                self.objects.discard(k)
+
+            self.create_index()
+
 
 def load_jsonld(d, path, required=False):
     deserializer = oe.spdx30.JSONLDDeserializer()
@@ -1080,39 +1139,28 @@ def create_sbom(d, name, root_elements, add_objectsets=[]):
     # SBoM should be the only root element of the document
     objset.doc.rootElement = [sbom]
 
-    # De-duplicate licenses
-    unique = set()
-    dedup = {}
-    for lic in objset.foreach_type(oe.spdx30.simplelicensing_LicenseExpression):
-        for u in unique:
-            if (
-                u.simplelicensing_licenseExpression
-                == lic.simplelicensing_licenseExpression
-                and u.simplelicensing_licenseListVersion
-                == lic.simplelicensing_licenseListVersion
-            ):
-                dedup[lic] = u
-                break
-        else:
-            unique.add(lic)
-
-    if dedup:
-        for rel in objset.foreach_filter(
-            oe.spdx30.Relationship,
-            relationshipType=oe.spdx30.RelationshipType.hasDeclaredLicense,
-        ):
-            rel.to = [dedup.get(to, to) for to in rel.to]
-
-        for rel in objset.foreach_filter(
-            oe.spdx30.Relationship,
-            relationshipType=oe.spdx30.RelationshipType.hasConcludedLicense,
-        ):
-            rel.to = [dedup.get(to, to) for to in rel.to]
+    def cmp_license_expression(a, b):
+        return (
+            a.simplelicensing_licenseExpression == b.simplelicensing_licenseExpression
+            and a.simplelicensing_licenseListVersion
+            == b.simplelicensing_licenseListVersion
+        )
 
-        for k, v in dedup.items():
-            bb.debug(1, f"Removing duplicate License {k._id} -> {v._id}")
-            objset.objects.remove(k)
+    def cmp_creation_info(a, b):
+        data_a = {k: a[k] for k in a}
+        data_b = {k: b[k] for k in b}
+        data_a["@id"] = ""
+        data_b["@id"] = ""
+        return data_a == data_b
+
+    with objset.deduplicate() as dedup:
+        # De-duplicate licenses
+        dedup.find_duplicates(
+            cmp_license_expression,
+            oe.spdx30.simplelicensing_LicenseExpression,
+        )
 
-        objset.create_index()
+        # Deduplicate creation info
+        dedup.find_duplicates(cmp_creation_info, oe.spdx30.CreationInfo)
 
     return objset, sbom
meta/lib/oe/spdx30.py +1 -1
index cd97eebd18..1f58402ffc 100644
@@ -701,7 +701,7 @@ class SHACLObject(object):
         self.__dict__["_obj_data"][iri] = prop.init()
 
     def __iter__(self):
-        return self._OBJ_PROPERTIES.keys()
+        return iter(self._OBJ_PROPERTIES.keys())
 
     def walk(self, callback, path=None):
         """
Signature · 1 line
2.53.0
#9 lore raw

PATCH 8/9 spdx: Ignore ASSUME_PROVIDED recipes

In reply to Joshua Watt

Ignore recipes in ASSUME_PROVIDED when generating SPDX dependencies, since these recipes are not actually built

---
 meta/lib/oe/spdx_common.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
meta/lib/oe/spdx_common.py +4 -1
index 72c24180d5..03863ef362 100644
@@ -104,9 +104,12 @@ def collect_direct_deps(d, dep_task):
 
     deps = set()
 
+    ignore = set(d.getVar("ASSUME_PROVIDED").split())
+    ignore.add(pn)
+
     for dep_name in this_dep.deps:
         dep_data = taskdepdata[dep_name]
-        if dep_data.taskname == dep_task and dep_data.pn != pn:
+        if dep_data.taskname == dep_task and dep_data.pn not in ignore:
             deps.add((dep_data.pn, dep_data.hashfn, dep_name in this_dep.taskhash_deps))
 
     return sorted(deps)
Signature · 1 line
2.53.0
#10 lore raw

PATCH 9/9 spdx_common: Check for dependent task in task flags

In reply to Joshua Watt

Checks that the task being used to detect dependencies is present in at least one dependency task flag of the current task. This helps prevent errors where the wrong task is specified and never found.

---
 meta/lib/oe/spdx_common.py | 11 +++++++++++
 1 file changed, 11 insertions(+)
meta/lib/oe/spdx_common.py +11 -0
index 03863ef362..9d39b5e9ac 100644
@@ -96,6 +96,17 @@ def collect_direct_deps(d, dep_task):
 
     taskdepdata = d.getVar("BB_TASKDEPDATA", False)
 
+    # Check that the task is listed one of the task dependency flags of the
+    # current task
+    depflags = (
+        set((d.getVarFlag(current_task, "deptask") or "").split())
+        | set((d.getVarFlag(current_task, "rdeptask") or "").split())
+        | set((d.getVarFlag(current_task, "recrdeptask") or "").split())
+    )
+
+    if not dep_task in depflags:
+        bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}")
+
     for this_dep in taskdepdata.values():
         if this_dep[0] == pn and this_dep[1] == current_task:
             break
Signature · 1 line
2.53.0
#11 lore raw

v2 cover Add SPDX 3 Recipe Information

In reply to Joshua Watt

Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.

Special thanks to Benjamin Robin <[email protected]> for helping work through this.

V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).

Joshua Watt (8):
llvm-project-source: Use allarch.bbclass
gcc-source: Use allarch.bbclass
spdx3: Add recipe SPDX data
spdx3: Add recipe SBoM task
spdx3: Add is-native property
spdx30: Include patch file information in VEX
spdx: De-duplicate CreationInfo
spdx_common: Check for dependent task in task flags

 meta/classes-global/sstate.bbclass            |   4 +-
 .../create-spdx-image-3.0.bbclass             |   4 +-
 .../create-spdx-sdk-3.0.bbclass               |   4 +-
 meta/classes-recipe/kernel.bbclass            |   2 +-
 meta/classes-recipe/nospdx.bbclass            |   1 +
 meta/classes/create-spdx-2.2.bbclass          |  12 +-
 meta/classes/create-spdx-3.0.bbclass          |  75 ++-
 meta/classes/spdx-common.bbclass              |  16 +-
 meta/lib/oe/sbom30.py                         | 192 ++++---
 meta/lib/oe/spdx30.py                         |   2 +-
 meta/lib/oe/spdx30_tasks.py                   | 487 +++++++++++++-----
 meta/lib/oe/spdx_common.py                    |  11 +
 .../meta/meta-world-recipe-sbom.bb            |  26 +
 .../clang/llvm-project-source.inc             |   8 +-
 meta/recipes-devtools/gcc/gcc-source.inc      |  16 +-
 15 files changed, 615 insertions(+), 245 deletions(-)
 create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
Signature · 1 line
2.53.0
#12 lore raw

PATCH v2 1/8 llvm-project-source: Use allarch.bbclass

Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)

---
 meta/recipes-devtools/clang/llvm-project-source.inc | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)
meta/recipes-devtools/clang/llvm-project-source.inc +1 -7
index 13e54efbc2..6bb595b7bc 100644
@@ -5,7 +5,7 @@ deltask do_populate_sysroot
 deltask do_populate_lic
 RM_WORK_EXCLUDE += "${PN}"
 
-inherit nopackages
+inherit nopackages allarch
 
 PN = "llvm-project-source-${PV}"
 WORKDIR = "${TMPDIR}/work-shared/llvm-project-source-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:llvm-project-source::${PV}:${PR}::${SSTATE_VERSION}:"
 STAMP = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-${PR}"
 STAMPCLEAN = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-*"
 
-INHIBIT_DEFAULT_DEPS = "1"
 DEPENDS = ""
 PACKAGES = ""
-TARGET_ARCH = "allarch"
-TARGET_AS_ARCH = "none"
-TARGET_CC_ARCH = "none"
-TARGET_LD_ARCH = "none"
-TARGET_OS = "linux"
 baselib = "lib"
 PACKAGE_ARCH = "all"
 
Signature · 1 line
2.53.0
#13 lore raw

PATCH v2 2/8 gcc-source: Use allarch.bbclass

In reply to Joshua Watt

Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)

---
 meta/recipes-devtools/gcc/gcc-source.inc | 16 +++++-----------
 1 file changed, 5 insertions(+), 11 deletions(-)
meta/recipes-devtools/gcc/gcc-source.inc +5 -11
index 265bcf4bef..3ac679b1a6 100644
@@ -1,11 +1,11 @@
-deltask do_configure 
-deltask do_compile 
-deltask do_install 
+deltask do_configure
+deltask do_compile
+deltask do_install
 deltask do_populate_sysroot
-deltask do_populate_lic 
+deltask do_populate_lic
 RM_WORK_EXCLUDE += "${PN}"
 
-inherit nopackages
+inherit nopackages allarch
 
 PN = "gcc-source-${PV}"
 WORKDIR = "${TMPDIR}/work-shared/gcc-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:gcc::${PV}:${PR}::${SSTATE_VERSION}:"
 STAMP = "${STAMPS_DIR}/work-shared/gcc-${PV}-${PR}"
 STAMPCLEAN = "${STAMPS_DIR}/work-shared/gcc-${PV}-*"
 
-INHIBIT_DEFAULT_DEPS = "1"
 DEPENDS = ""
 PACKAGES = ""
-TARGET_ARCH = "allarch"
-TARGET_AS_ARCH = "none"
-TARGET_CC_ARCH = "none"
-TARGET_LD_ARCH = "none"
-TARGET_OS = "linux"
 baselib = "lib"
 PACKAGE_ARCH = "all"
 
Signature · 1 line
2.53.0
#14 lore raw

PATCH v2 3/8 spdx3: Add recipe SPDX data

In reply to Joshua Watt

Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.

Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch

---
 meta/classes-global/sstate.bbclass            |   4 +-
 .../create-spdx-image-3.0.bbclass             |   4 +-
 .../create-spdx-sdk-3.0.bbclass               |   4 +-
 meta/classes-recipe/kernel.bbclass            |   2 +-
 meta/classes-recipe/nospdx.bbclass            |   1 +
 meta/classes/create-spdx-2.2.bbclass          |  12 +-
 meta/classes/create-spdx-3.0.bbclass          |  49 ++-
 meta/classes/spdx-common.bbclass              |  15 +-
 meta/lib/oe/spdx30_tasks.py                   | 402 ++++++++++++------
 9 files changed, 342 insertions(+), 151 deletions(-)
meta/classes-global/sstate.bbclass +2 -2
index 2fd29d7323..95c44f404e 100644
@@ -954,7 +954,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,
             extrapath = d.getVar("NATIVELSBSTRING") + "/"
         else:
             extrapath = ""
-        
+
         tname = bb.runqueue.taskname_from_tid(task)[3:]
 
         if tname in ["fetch", "unpack", "patch", "populate_lic", "preconfigure"] and splithashfn[2]:
@@ -1116,7 +1116,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None):
 
     logit("Considering setscene task: %s" % (str(taskdependees[task])), log)
 
-    directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_deploy_archives"]
+    directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_create_recipe_spdx", "do_deploy_archives"]
 
     def isNativeCross(x):
         return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross")
meta/classes-recipe/create-spdx-image-3.0.bbclass +2 -2
index 636ab14eb0..15a91e90e2 100644
@@ -34,7 +34,7 @@ addtask do_create_rootfs_spdx after do_rootfs before do_image
 SSTATETASKS += "do_create_rootfs_spdx"
 do_create_rootfs_spdx[sstate-inputdirs] = "${SPDXROOTFSDEPLOY}"
 do_create_rootfs_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}"
-do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_create_rootfs_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_create_rootfs_spdx[cleandirs] += "${SPDXROOTFSDEPLOY}"
 do_create_rootfs_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 
@@ -76,7 +76,7 @@ do_create_image_sbom_spdx[sstate-inputdirs] = "${SPDXIMAGEDEPLOYDIR}"
 do_create_image_sbom_spdx[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
 do_create_image_sbom_spdx[stamp-extra-info] = "${MACHINE_ARCH}"
 do_create_image_sbom_spdx[cleandirs] = "${SPDXIMAGEDEPLOYDIR}"
-do_create_image_sbom_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_create_image_sbom_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_create_image_sbom_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 
 python do_create_image_sbom_spdx_setscene() {
meta/classes-recipe/create-spdx-sdk-3.0.bbclass +2 -2
index e5f220cdfa..a4b8ed3bf9 100644
@@ -5,14 +5,14 @@
 #
 # SPDX SDK tasks
 
-do_populate_sdk[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_populate_sdk[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_populate_sdk[cleandirs] += "${SPDXSDKWORK}"
 do_populate_sdk[postfuncs] += "sdk_create_sbom"
 do_populate_sdk[file-checksums] += "${SPDX3_DEP_FILES}"
 POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_create_spdx"
 POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_create_spdx"
 
-do_populate_sdk_ext[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_populate_sdk_ext[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_populate_sdk_ext[cleandirs] += "${SPDXSDKEXTWORK}"
 do_populate_sdk_ext[postfuncs] += "sdk_ext_create_sbom"
 do_populate_sdk_ext[file-checksums] += "${SPDX3_DEP_FILES}"
meta/classes-recipe/kernel.bbclass +1 -1
index f989b31c47..3a2c20dec2 100644
@@ -904,7 +904,7 @@ python do_create_kernel_config_spdx() {
             bb.error(f"Failed to parse kernel config file: {e}")
 
         build, build_objset = oe.sbom30.find_root_obj_in_jsonld(
-            d, "recipes", f"recipe-{pn}", oe.spdx30.build_Build
+            d, "builds", f"build-{pn}", oe.spdx30.build_Build
         )
 
         kernel_build = build_objset.add_root(
meta/classes-recipe/nospdx.bbclass +1 -0
index b20e28218b..90e14442ba 100644
@@ -5,6 +5,7 @@
 #
 
 deltask do_collect_spdx_deps
+deltask do_create_recipe_spdx
 deltask do_create_spdx
 deltask do_create_spdx_runtime
 deltask do_create_package_spdx
meta/classes/create-spdx-2.2.bbclass +11 -1
index 65d10d86db..f1ee0f9afd 100644
@@ -399,6 +399,15 @@ def get_license_list_version(license_data, d):
     return ".".join(license_data["licenseListVersion"].split(".")[:2])
 
 
+# This task is added for compatibility with tasks shared with SPDX 3, but
+# doesn't do anything
+do_create_recipe_spdx() {
+    :
+}
+do_create_recipe_spdx[noexec] = "1"
+addtask do_create_recipe_spdx after do_collect_spdx_deps
+
+
 python do_create_spdx() {
     from datetime import datetime, timezone
     import oe.sbom
@@ -594,7 +603,7 @@ python do_create_spdx() {
 }
 do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS"
 # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source
-addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work
+addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work
 
 SSTATETASKS += "do_create_spdx"
 do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}"
@@ -605,6 +614,7 @@ python do_create_spdx_setscene () {
 }
 addtask do_create_spdx_setscene
 
+do_create_spdx[deptask] += "do_create_spdx"
 do_create_spdx[dirs] = "${SPDXWORK}"
 do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}"
 do_create_spdx[depends] += " \
meta/classes/create-spdx-3.0.bbclass +36 -13

88 lines not on this page · raw message

meta/classes/spdx-common.bbclass +8 -7
index 3110230c9e..2804c27b0b 100644
@@ -23,6 +23,7 @@ SPDXDEPS = "${SPDXDIR}/deps.json"
 SPDX_TOOL_NAME ??= "oe-spdx-creator"
 SPDX_TOOL_VERSION ??= "1.0"
 
+SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy"
 SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy"
 
 SPDX_INCLUDE_SOURCES ??= "0"
@@ -72,6 +73,8 @@ def create_spdx_source_deps(d):
         # ourselves
         if oe.spdx_common.has_task(d, "do_unpack"):
             deps.append("%s:do_unpack" % pn)
+        if oe.spdx_common.has_task(d, "do_patch"):
+            deps.append("%s:do_patch" % pn)
 
         if oe.spdx_common.is_work_shared_spdx(d) and \
            oe.spdx_common.process_sources(d):
@@ -97,8 +100,8 @@ python do_collect_spdx_deps() {
     # This task calculates the build time dependencies of the recipe, and is
     # required because while a task can deptask on itself, those dependencies
     # do not show up in BB_TASKDEPDATA. To work around that, this task does the
-    # deptask on do_create_spdx and writes out the dependencies it finds, then
-    # do_create_spdx reads in the found dependencies when writing the actual
+    # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then
+    # downstream tasks read in the found dependencies when writing the actual
     # SPDX document
     import json
     import oe.spdx_common
@@ -106,15 +109,13 @@ python do_collect_spdx_deps() {
 
     spdx_deps_file = Path(d.getVar("SPDXDEPS"))
 
-    deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx")
+    deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx")
 
     with spdx_deps_file.open("w") as f:
         json.dump(deps, f)
 }
-# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source
-addtask do_collect_spdx_deps after do_unpack
-do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}"
-do_collect_spdx_deps[deptask] = "do_create_spdx"
+addtask do_collect_spdx_deps
+do_collect_spdx_deps[deptask] = "do_create_recipe_spdx"
 do_collect_spdx_deps[dirs] = "${SPDXDIR}"
 
 oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA"
meta/lib/oe/spdx30_tasks.py +279 -123

559 lines not on this page · raw message

Signature · 1 line
2.53.0
#15 lore raw

PATCH v2 4/8 spdx3: Add recipe SBoM task

In reply to Joshua Watt

Adds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example:

```
bitbake -c create_recipe_sbom zstd
```

Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively).

The complete SBoM for all (target) recipes can be built with:

```
bitbake -c create_recipe_sbom meta-world-recipe-sbom
```
---
 meta/classes/create-spdx-3.0.bbclass          | 28 +++++++++++++++++++
 meta/classes/spdx-common.bbclass              |  1 +
 meta/lib/oe/spdx30_tasks.py                   | 10 +++++++
 .../meta/meta-world-recipe-sbom.bb            | 26 +++++++++++++++++
 4 files changed, 65 insertions(+)
 create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
meta/classes/create-spdx-3.0.bbclass +28 -0
index 61223ee0a5..fdd0c60690 100644
@@ -238,6 +238,34 @@ python do_create_package_spdx_setscene () {
 }
 addtask do_create_package_spdx_setscene
 
+addtask do_create_recipe_sbom after create_recipe_spdx
+python do_create_recipe_sbom() {
+    import oe.spdx30_tasks
+    from pathlib import Path
+    deploydir = Path(d.getVar("SPDXRECIPESBOMDEPLOY"))
+    oe.spdx30_tasks.create_recipe_sbom(d, deploydir)
+}
+
+SSTATETASKS += "do_create_recipe_sbom"
+do_create_recipe_sbom[recrdeptask] = "do_create_recipe_spdx"
+do_create_recipe_sbom[nostamp] = "1"
+do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}"
+do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
+do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}"
+do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}"
+do_create_recipe_sbom[vardeps] += "\
+    SPDX_INCLUDE_BITBAKE_PARENT_BUILD \
+    SPDX_PACKAGE_ADDITIONAL_PURPOSE \
+    SPDX_PROFILES \
+    SPDX_NAMESPACE_PREFIX \
+    SPDX_UUID_NAMESPACE \
+    "
+
+python do_create_recipe_sbom_setscene () {
+    sstate_setscene(d)
+}
+addtask do_create_recipe_sbom_setscene
+
 python spdx30_build_started_handler () {
     import oe.spdx30_tasks
     d = e.data.createCopy()
meta/classes/spdx-common.bbclass +1 -0
index 2804c27b0b..1ec4877a6a 100644
@@ -25,6 +25,7 @@ SPDX_TOOL_VERSION ??= "1.0"
 
 SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy"
 SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy"
+SPDXRECIPESBOMDEPLOY = "${SPDXDIR}/recipes-bom-deploy"
 
 SPDX_INCLUDE_SOURCES ??= "0"
 SPDX_INCLUDE_COMPILED_SOURCES ??= "0"
meta/lib/oe/spdx30_tasks.py +10 -0
index a8b4525e3d..9a312a870d 100644
@@ -1564,3 +1564,13 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname):
     oe.sbom30.write_jsonld_doc(
         d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json")
     )
+
+
+def create_recipe_sbom(d, deploydir):
+    sbom_name = d.getVar("PN") + "-recipe-sbom"
+
+    recipe, recipe_objset = load_recipe_spdx(d)
+
+    objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset])
+
+    oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json"))
meta/recipes-core/meta/meta-world-recipe-sbom.bb new file +26 -0
new file mode 100644
index 0000000000..60209fba7e
@@ -0,0 +1,26 @@
+SUMMARY = "Generates a combined SBoM for all world recipes"
+LICENSE = "MIT"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+PACKAGE_ARCH = "${MACHINE_ARCH}"
+
+inherit nopackages
+deltask do_fetch
+deltask do_unpack
+deltask do_patch
+deltask do_configure
+deltask do_compile
+deltask do_install
+
+do_prepare_recipe_sysroot[deptask] = ""
+
+WORLD_SBOM_EXCLUDE ?= ""
+
+python calculate_extra_depends() {
+    exclude = set('${WORLD_SBOM_EXCLUDE}'.split())
+    for p in world_target:
+        if p == self_pn or p in exclude:
+            continue
+        deps.append(p)
+}
Signature · 1 line
2.53.0
#16 lore raw

PATCH v2 5/8 spdx3: Add is-native property

In reply to Joshua Watt

Adds a custom is-native property to the recipe package to indicate if it is a native recipe

---
 meta/lib/oe/sbom30.py       | 20 ++++++++++++++++++++
 meta/lib/oe/spdx30_tasks.py | 18 +++++++++++-------
 2 files changed, 31 insertions(+), 7 deletions(-)
meta/lib/oe/sbom30.py +20 -0
index 227ac51877..50a72fce39 100644
@@ -118,6 +118,26 @@ class OEDocumentExtension(oe.spdx30.extension_Extension):
         )
 
 
[email protected](OE_SPDX_BASE + "recipe-extension")
+class OERecipeExtension(oe.spdx30.extension_Extension):
+    """
+    This extension is added to recipe software_Packages to indicate various
+    useful bits of information about the recipe
+    """
+
+    CLOSED = True
+
+    @classmethod
+    def _register_props(cls):
+        super()._register_props()
+        cls._add_property(
+            "is_native",
+            oe.spdx30.BooleanProp(),
+            OE_SPDX_BASE + "is-native",
+            max_count=1,
+        )
+
+
 def spdxid_hash(*items):
     h = hashlib.md5()
     for i in items:
meta/lib/oe/spdx30_tasks.py +11 -7
index 9a312a870d..fff1ca6bea 100644
@@ -477,6 +477,10 @@ def set_purls(spdx_package, purls):
         )
 
 
+def get_is_native(d):
+    return bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d)
+
+
 def create_recipe_spdx(d):
     deploydir = Path(d.getVar("SPDXRECIPEDEPLOY"))
     deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
@@ -507,6 +511,11 @@ def create_recipe_spdx(d):
         )
     )
 
+    if get_is_native(d):
+        ext = oe.sbom30.OERecipeExtension()
+        ext.is_native = True
+        recipe.extension.append(ext)
+
     set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split())
 
     # TODO: This doesn't work before do_unpack because the license text has to
@@ -668,9 +677,7 @@ def create_spdx(d):
     spdx_workdir = Path(d.getVar("SPDXWORK"))
     include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1"
     pkg_arch = d.getVar("SSTATE_PKGARCH")
-    is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class(
-        "cross", d
-    )
+    is_native = get_is_native(d)
 
     recipe, recipe_objset = load_recipe_spdx(d)
 
@@ -1021,14 +1028,11 @@ def create_spdx(d):
 def create_package_spdx(d):
     deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
     deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY"))
-    is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class(
-        "cross", d
-    )
 
     providers = oe.spdx_common.collect_package_providers(d)
     pkg_arch = d.getVar("SSTATE_PKGARCH")
 
-    if is_native:
+    if get_is_native(d):
         return
 
     bb.build.exec_func("read_subpackage_metadata", d)
Signature · 1 line
2.53.0
#17 lore raw

PATCH v2 6/8 spdx30: Include patch file information in VEX

In reply to Joshua Watt

Modifies the SPDX VEX output to include the patches that fix a particular vulnerability. This is done by adding a patchedBy relationship from the VexFixedVulnAssessmentRelationship to the File that provides the fix.

If the file can be located without fetching (e.g. is a file:// in SRC_URI), the checksum will be included.

---
 meta/lib/oe/sbom30.py       | 60 ++++++++++++++-------------
 meta/lib/oe/spdx30_tasks.py | 81 ++++++++++++++++++++++++++++---------
 2 files changed, 92 insertions(+), 49 deletions(-)
meta/lib/oe/sbom30.py +31 -29
index 50a72fce39..21f084dc16 100644
@@ -620,37 +620,38 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
         )
         spdx_file.extension.append(OELicenseScannedExtension())
 
-    def new_file(self, _id, name, path, *, purposes=[]):
-        sha256_hash = bb.utils.sha256_file(path)
+    def new_file(self, _id, name, path, *, purposes=[], hashfile=True):
+        if hashfile:
+            sha256_hash = bb.utils.sha256_file(path)
 
-        for f in self.by_sha256_hash.get(sha256_hash, []):
-            if not isinstance(f, oe.spdx30.software_File):
-                continue
+            for f in self.by_sha256_hash.get(sha256_hash, []):
+                if not isinstance(f, oe.spdx30.software_File):
+                    continue
 
-            if purposes:
-                new_primary = purposes[0]
-                new_additional = []
+                if purposes:
+                    new_primary = purposes[0]
+                    new_additional = []
 
-                if f.software_primaryPurpose:
-                    new_additional.append(f.software_primaryPurpose)
-                new_additional.extend(f.software_additionalPurpose)
+                    if f.software_primaryPurpose:
+                        new_additional.append(f.software_primaryPurpose)
+                    new_additional.extend(f.software_additionalPurpose)
 
-                new_additional = sorted(
-                    list(set(p for p in new_additional if p != new_primary))
-                )
+                    new_additional = sorted(
+                        list(set(p for p in new_additional if p != new_primary))
+                    )
 
-                f.software_primaryPurpose = new_primary
-                f.software_additionalPurpose = new_additional
+                    f.software_primaryPurpose = new_primary
+                    f.software_additionalPurpose = new_additional
 
-            if f.name != name:
-                for e in f.extension:
-                    if isinstance(e, OEFileNameAliasExtension):
-                        e.aliases.append(name)
-                        break
-                else:
-                    f.extension.append(OEFileNameAliasExtension(aliases=[name]))
+                if f.name != name:
+                    for e in f.extension:
+                        if isinstance(e, OEFileNameAliasExtension):
+                            e.aliases.append(name)
+                            break
+                    else:
+                        f.extension.append(OEFileNameAliasExtension(aliases=[name]))
 
-            return f
+                return f
 
         spdx_file = oe.spdx30.software_File(
             _id=_id,
@@ -661,12 +662,13 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
             spdx_file.software_primaryPurpose = purposes[0]
             spdx_file.software_additionalPurpose = purposes[1:]
 
-        spdx_file.verifiedUsing.append(
-            oe.spdx30.Hash(
-                algorithm=oe.spdx30.HashAlgorithm.sha256,
-                hashValue=sha256_hash,
+        if hashfile:
+            spdx_file.verifiedUsing.append(
+                oe.spdx30.Hash(
+                    algorithm=oe.spdx30.HashAlgorithm.sha256,
+                    hashValue=sha256_hash,
+                )
             )
-        )
 
         return self.add(spdx_file)
 
meta/lib/oe/spdx30_tasks.py +61 -20

139 lines not on this page · raw message

Signature · 1 line
2.53.0
#18 lore raw

PATCH v2 7/8 spdx: De-duplicate CreationInfo

In reply to Joshua Watt

De-duplicates CreationInfo objects that are identical (except for ID) when writing out an SBoM. This significantly reduces the number of CreationInfo objects that end up in the final document.

---
 meta/lib/oe/sbom30.py | 112 ++++++++++++++++++++++++++++++------------
 meta/lib/oe/spdx30.py |   2 +-
 2 files changed, 81 insertions(+), 33 deletions(-)
meta/lib/oe/sbom30.py +80 -32
index 21f084dc16..55a2863d2d 100644
@@ -14,6 +14,7 @@ import uuid
 import os
 import oe.spdx_common
 from datetime import datetime, timezone
+from contextlib import contextmanager
 
 OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/"
 
@@ -191,6 +192,25 @@ def to_list(l):
     return l
 
 
+class Dedup(object):
+    def __init__(self, objset):
+        self.unique = set()
+        self.dedup = {}
+        self.objset = objset
+
+    def find_duplicates(self, cmp, typ, **kwargs):
+        for o in self.objset.foreach_filter(typ, **kwargs):
+            for u in self.unique:
+                if cmp(u, o):
+                    self.dedup[o] = u
+                    break
+            else:
+                self.unique.add(o)
+
+    def get(self, o):
+        return self.dedup.get(o, o)
+
+
 class ObjectSet(oe.spdx30.SHACLObjectSet):
     def __init__(self, d):
         super().__init__()
@@ -895,6 +915,45 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
         self.missing_ids -= set(imports.keys())
         return self.missing_ids
 
+    @contextmanager
+    def deduplicate(self):
+        d = Dedup(self)
+
+        yield d
+
+        visited = set()
+
+        def visit(o, path):
+            if isinstance(o, oe.spdx30.SHACLObject):
+                if o in visited:
+                    return False
+                visited.add(o)
+
+                for k in o:
+                    v = o[k]
+                    if isinstance(v, oe.spdx30.SHACLObject):
+                        o[k] = d.get(v)
+
+            elif isinstance(o, oe.spdx30.ListProxy):
+                for idx, v in enumerate(o):
+                    if isinstance(v, oe.spdx30.SHACLObject):
+                        o[idx] = d.get(v)
+
+            return True
+
+        if d.dedup:
+            for o in self.objects:
+                o.walk(visit)
+
+            for k, v in d.dedup.items():
+                bb.debug(
+                    1,
+                    f"Removing duplicate {k.__class__.__name__} {k._id or id(k)} -> {v._id or id(v)}",
+                )
+                self.objects.discard(k)
+
+            self.create_index()
+
 
 def load_jsonld(d, path, required=False):
     deserializer = oe.spdx30.JSONLDDeserializer()
@@ -1080,39 +1139,28 @@ def create_sbom(d, name, root_elements, add_objectsets=[]):
     # SBoM should be the only root element of the document
     objset.doc.rootElement = [sbom]
 
-    # De-duplicate licenses
-    unique = set()
-    dedup = {}
-    for lic in objset.foreach_type(oe.spdx30.simplelicensing_LicenseExpression):
-        for u in unique:
-            if (
-                u.simplelicensing_licenseExpression
-                == lic.simplelicensing_licenseExpression
-                and u.simplelicensing_licenseListVersion
-                == lic.simplelicensing_licenseListVersion
-            ):
-                dedup[lic] = u
-                break
-        else:
-            unique.add(lic)
-
-    if dedup:
-        for rel in objset.foreach_filter(
-            oe.spdx30.Relationship,
-            relationshipType=oe.spdx30.RelationshipType.hasDeclaredLicense,
-        ):
-            rel.to = [dedup.get(to, to) for to in rel.to]
-
-        for rel in objset.foreach_filter(
-            oe.spdx30.Relationship,
-            relationshipType=oe.spdx30.RelationshipType.hasConcludedLicense,
-        ):
-            rel.to = [dedup.get(to, to) for to in rel.to]
+    def cmp_license_expression(a, b):
+        return (
+            a.simplelicensing_licenseExpression == b.simplelicensing_licenseExpression
+            and a.simplelicensing_licenseListVersion
+            == b.simplelicensing_licenseListVersion
+        )
 
-        for k, v in dedup.items():
-            bb.debug(1, f"Removing duplicate License {k._id} -> {v._id}")
-            objset.objects.remove(k)
+    def cmp_creation_info(a, b):
+        data_a = {k: a[k] for k in a}
+        data_b = {k: b[k] for k in b}
+        data_a["@id"] = ""
+        data_b["@id"] = ""
+        return data_a == data_b
+
+    with objset.deduplicate() as dedup:
+        # De-duplicate licenses
+        dedup.find_duplicates(
+            cmp_license_expression,
+            oe.spdx30.simplelicensing_LicenseExpression,
+        )
 
-        objset.create_index()
+        # Deduplicate creation info
+        dedup.find_duplicates(cmp_creation_info, oe.spdx30.CreationInfo)
 
     return objset, sbom
meta/lib/oe/spdx30.py +1 -1
index cd97eebd18..1f58402ffc 100644
@@ -701,7 +701,7 @@ class SHACLObject(object):
         self.__dict__["_obj_data"][iri] = prop.init()
 
     def __iter__(self):
-        return self._OBJ_PROPERTIES.keys()
+        return iter(self._OBJ_PROPERTIES.keys())
 
     def walk(self, callback, path=None):
         """
Signature · 1 line
2.53.0
#19 lore raw

PATCH v2 8/8 spdx_common: Check for dependent task in task flags

In reply to Joshua Watt

Checks that the task being used to detect dependencies is present in at least one dependency task flag of the current task. This helps prevent errors where the wrong task is specified and never found.

---
 meta/lib/oe/spdx_common.py | 11 +++++++++++
 1 file changed, 11 insertions(+)
meta/lib/oe/spdx_common.py +11 -0
index 72c24180d5..3aaf2a9c8b 100644
@@ -96,6 +96,17 @@ def collect_direct_deps(d, dep_task):
 
     taskdepdata = d.getVar("BB_TASKDEPDATA", False)
 
+    # Check that the task is listed one of the task dependency flags of the
+    # current task
+    depflags = (
+        set((d.getVarFlag(current_task, "deptask") or "").split())
+        | set((d.getVarFlag(current_task, "rdeptask") or "").split())
+        | set((d.getVarFlag(current_task, "recrdeptask") or "").split())
+    )
+
+    if not dep_task in depflags:
+        bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}")
+
     for this_dep in taskdepdata.values():
         if this_dep[0] == pn and this_dep[1] == current_task:
             break
Signature · 1 line
2.53.0
#20 lore raw
In reply to Joshua Watt
Quoted 9 lines · Joshua Watt via lists.openembedded.org

Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.

Special thanks to Benjamin Robin <[email protected]> for helping work through this.

Hi Joshua,

Thanks for the new version, but I believe we still have some issues.

A first one:

ERROR: docbook-xml-dtd4-native-4.5-r0 do_create_spdx: Error executing a python function in exec_func_python() autogenerated:
...
File: '/srv/pokybuild/yocto-worker/genericx86/build/layers/openembedded-core/meta/lib/oe/spdx30_tasks.py', lineno: 89, function: add_license_text

     0085:        # If it's not SPDX or PD, then NO_GENERIC_LICENSE must be set
     0086:        filename = d.getVarFlag("NO_GENERIC_LICENSE", name)
     0087:        if filename:
     0088:            filename = d.expand("${S}/" + filename)

*** 0089: with open(filename, errors="replace") as f:
0090: lic.simplelicensing_licenseText = f.read()
0091: return lic
0092: else:
0093: bb.fatal("Cannot find any text for license %s" % name)
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/srv/pokybuild/yocto-worker/genericx86/build/build/tmp/work/x86_64-linux/docbook-xml-dtd4-native/4.5/sources/LICENSE-OASIS'

https://autobuilder.yoctoproject.org/valkyrie/#/builders/19/builds/3253
https://autobuilder.yoctoproject.org/valkyrie/#/builders/20/builds/3217
https://autobuilder.yoctoproject.org/valkyrie/#/builders/74/builds/3224
https://autobuilder.yoctoproject.org/valkyrie/#/builders/95/builds/3204

And a second one:

Initialising tasks...ERROR: Task virtual:multilib:lib32:/srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx has circular dependency on /srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx
ERROR: Command execution failed: 1

https://autobuilder.yoctoproject.org/valkyrie/#/builders/17/builds/3086
https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/3253

Can you have a look at these?

Thanks,
Mathieu

Signature · 3 lines
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#21 lore raw
Quoted 7 lines · Mathieu Dubois-Briand

And a second one:

Initialising tasks...ERROR: Task virtual:multilib:lib32:/srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx has circular dependency on /srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx
ERROR: Command execution failed: 1

https://autobuilder.yoctoproject.org/valkyrie/#/builders/17/builds/3086
https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/3253

For the second issue, it can be triggered by adding these lines in local.conf:

MACHINE = "qemux86"
OE_FRAGMENTS += 'core/yocto-autobuilder/multilib-x86-lib32'

With this configuration these 2 commands failed:
- bitbake -c create_recipe_sbom meta-world-recipe-sbom
- bitbake world

Signature · 3 lines
Benjamin Robin, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#22 lore raw
Quoted 9 lines · Benjamin Robin
Quoted 7 lines · Mathieu Dubois-Briand

And a second one:

Initialising tasks...ERROR: Task virtual:multilib:lib32:/srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx has circular dependency on /srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx
ERROR: Command execution failed: 1

https://autobuilder.yoctoproject.org/valkyrie/#/builders/17/builds/3086
https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/3253

Quoted 8 lines

For the second issue, it can be triggered by adding these lines in local.conf:

MACHINE = "qemux86"
OE_FRAGMENTS += 'core/yocto-autobuilder/multilib-x86-lib32'

With this configuration these 2 commands failed:
- bitbake -c create_recipe_sbom meta-world-recipe-sbom
- bitbake world

I am proposing this fix for meta/recipes-core/meta/meta-world-recipe-sbom.bb:

python calculate_extra_depends() {
    exclude = set('${WORLD_SBOM_EXCLUDE}'.split())
    exclude.add(self_pn)
 
    for variant in ('${MULTILIB_VARIANTS}' or "").split():
        exclude.add(f"{variant}-{self_pn}")
 
    for p in world_target:
        if p in exclude:
            continue
        deps.append(p)
}

This fix solve the second issue (circular dependency).

Best regards,

Signature · 3 lines
Benjamin Robin, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#23 lore raw
Quoted 22 lines · Benjamin Robin
Quoted 9 lines · Benjamin Robin
Quoted 7 lines · Mathieu Dubois-Briand

And a second one:

Initialising tasks...ERROR: Task virtual:multilib:lib32:/srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx has circular dependency on /srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx
ERROR: Command execution failed: 1

https://autobuilder.yoctoproject.org/valkyrie/#/builders/17/builds/3086
https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/3253

Quoted 8 lines

For the second issue, it can be triggered by adding these lines in local.conf:

MACHINE = "qemux86"
OE_FRAGMENTS += 'core/yocto-autobuilder/multilib-x86-lib32'

With this configuration these 2 commands failed:
- bitbake -c create_recipe_sbom meta-world-recipe-sbom
- bitbake world

I am proposing this fix for meta/recipes-core/meta/meta-world-recipe-sbom.bb:

Quoted 12 lines
python calculate_extra_depends() {
    exclude = set('${WORLD_SBOM_EXCLUDE}'.split())
    exclude.add(self_pn)

    for variant in ('${MULTILIB_VARIANTS}' or "").split():
        exclude.add(f"{variant}-{self_pn}")

    for p in world_target:
        if p in exclude:
            continue
        deps.append(p)
}

Thanks, I'll add this to my branch

Quoted 11 lines

This fix solve the second issue (circular dependency).

Best regards,
--
Benjamin Robin, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

#24 lore raw

v3 cover Add SPDX 3 Recipe Information

In reply to Joshua Watt

Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.

Special thanks to Benjamin Robin <[email protected]> for helping work through this.

V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).

V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run).

Joshua Watt (8):
llvm-project-source: Use allarch.bbclass
gcc-source: Use allarch.bbclass
spdx3: Add recipe SPDX data
spdx3: Add recipe SBoM task
spdx3: Add is-native property
spdx30: Include patch file information in VEX
spdx: De-duplicate CreationInfo
spdx_common: Check for dependent task in task flags

 meta/classes-global/sstate.bbclass            |   4 +-
 .../create-spdx-image-3.0.bbclass             |   4 +-
 .../create-spdx-sdk-3.0.bbclass               |   4 +-
 meta/classes-recipe/kernel.bbclass            |   2 +-
 meta/classes-recipe/nospdx.bbclass            |   1 +
 meta/classes/create-spdx-2.2.bbclass          |  12 +-
 meta/classes/create-spdx-3.0.bbclass          |  76 ++-
 meta/classes/spdx-common.bbclass              |  16 +-
 meta/lib/oe/sbom30.py                         | 192 ++++---
 meta/lib/oe/spdx30.py                         |   2 +-
 meta/lib/oe/spdx30_tasks.py                   | 487 +++++++++++++-----
 meta/lib/oe/spdx_common.py                    |  11 +
 .../meta/meta-world-recipe-sbom.bb            |  28 +
 .../clang/llvm-project-source.inc             |   8 +-
 meta/recipes-devtools/gcc/gcc-source.inc      |  16 +-
 15 files changed, 618 insertions(+), 245 deletions(-)
 create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
Signature · 1 line
2.53.0
#25 lore raw

PATCH v3 1/8 llvm-project-source: Use allarch.bbclass

Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)

---
 meta/recipes-devtools/clang/llvm-project-source.inc | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)
meta/recipes-devtools/clang/llvm-project-source.inc +1 -7
index 13e54efbc2..6bb595b7bc 100644
@@ -5,7 +5,7 @@ deltask do_populate_sysroot
 deltask do_populate_lic
 RM_WORK_EXCLUDE += "${PN}"
 
-inherit nopackages
+inherit nopackages allarch
 
 PN = "llvm-project-source-${PV}"
 WORKDIR = "${TMPDIR}/work-shared/llvm-project-source-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:llvm-project-source::${PV}:${PR}::${SSTATE_VERSION}:"
 STAMP = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-${PR}"
 STAMPCLEAN = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-*"
 
-INHIBIT_DEFAULT_DEPS = "1"
 DEPENDS = ""
 PACKAGES = ""
-TARGET_ARCH = "allarch"
-TARGET_AS_ARCH = "none"
-TARGET_CC_ARCH = "none"
-TARGET_LD_ARCH = "none"
-TARGET_OS = "linux"
 baselib = "lib"
 PACKAGE_ARCH = "all"
 
Signature · 1 line
2.53.0
#26 lore raw

PATCH v3 2/8 gcc-source: Use allarch.bbclass

In reply to Joshua Watt

Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)

---
 meta/recipes-devtools/gcc/gcc-source.inc | 16 +++++-----------
 1 file changed, 5 insertions(+), 11 deletions(-)
meta/recipes-devtools/gcc/gcc-source.inc +5 -11
index 265bcf4bef..3ac679b1a6 100644
@@ -1,11 +1,11 @@
-deltask do_configure 
-deltask do_compile 
-deltask do_install 
+deltask do_configure
+deltask do_compile
+deltask do_install
 deltask do_populate_sysroot
-deltask do_populate_lic 
+deltask do_populate_lic
 RM_WORK_EXCLUDE += "${PN}"
 
-inherit nopackages
+inherit nopackages allarch
 
 PN = "gcc-source-${PV}"
 WORKDIR = "${TMPDIR}/work-shared/gcc-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:gcc::${PV}:${PR}::${SSTATE_VERSION}:"
 STAMP = "${STAMPS_DIR}/work-shared/gcc-${PV}-${PR}"
 STAMPCLEAN = "${STAMPS_DIR}/work-shared/gcc-${PV}-*"
 
-INHIBIT_DEFAULT_DEPS = "1"
 DEPENDS = ""
 PACKAGES = ""
-TARGET_ARCH = "allarch"
-TARGET_AS_ARCH = "none"
-TARGET_CC_ARCH = "none"
-TARGET_LD_ARCH = "none"
-TARGET_OS = "linux"
 baselib = "lib"
 PACKAGE_ARCH = "all"
 
Signature · 1 line
2.53.0
#27 lore raw

PATCH v3 3/8 spdx3: Add recipe SPDX data

In reply to Joshua Watt

Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.

Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch

---
 meta/classes-global/sstate.bbclass            |   4 +-
 .../create-spdx-image-3.0.bbclass             |   4 +-
 .../create-spdx-sdk-3.0.bbclass               |   4 +-
 meta/classes-recipe/kernel.bbclass            |   2 +-
 meta/classes-recipe/nospdx.bbclass            |   1 +
 meta/classes/create-spdx-2.2.bbclass          |  12 +-
 meta/classes/create-spdx-3.0.bbclass          |  50 ++-
 meta/classes/spdx-common.bbclass              |  15 +-
 meta/lib/oe/spdx30_tasks.py                   | 402 ++++++++++++------
 9 files changed, 343 insertions(+), 151 deletions(-)
meta/classes-global/sstate.bbclass +2 -2
index 2fd29d7323..95c44f404e 100644
@@ -954,7 +954,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,
             extrapath = d.getVar("NATIVELSBSTRING") + "/"
         else:
             extrapath = ""
-        
+
         tname = bb.runqueue.taskname_from_tid(task)[3:]
 
         if tname in ["fetch", "unpack", "patch", "populate_lic", "preconfigure"] and splithashfn[2]:
@@ -1116,7 +1116,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None):
 
     logit("Considering setscene task: %s" % (str(taskdependees[task])), log)
 
-    directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_deploy_archives"]
+    directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_create_recipe_spdx", "do_deploy_archives"]
 
     def isNativeCross(x):
         return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross")
meta/classes-recipe/create-spdx-image-3.0.bbclass +2 -2
index 636ab14eb0..15a91e90e2 100644
@@ -34,7 +34,7 @@ addtask do_create_rootfs_spdx after do_rootfs before do_image
 SSTATETASKS += "do_create_rootfs_spdx"
 do_create_rootfs_spdx[sstate-inputdirs] = "${SPDXROOTFSDEPLOY}"
 do_create_rootfs_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}"
-do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_create_rootfs_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_create_rootfs_spdx[cleandirs] += "${SPDXROOTFSDEPLOY}"
 do_create_rootfs_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 
@@ -76,7 +76,7 @@ do_create_image_sbom_spdx[sstate-inputdirs] = "${SPDXIMAGEDEPLOYDIR}"
 do_create_image_sbom_spdx[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
 do_create_image_sbom_spdx[stamp-extra-info] = "${MACHINE_ARCH}"
 do_create_image_sbom_spdx[cleandirs] = "${SPDXIMAGEDEPLOYDIR}"
-do_create_image_sbom_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_create_image_sbom_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_create_image_sbom_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 
 python do_create_image_sbom_spdx_setscene() {
meta/classes-recipe/create-spdx-sdk-3.0.bbclass +2 -2
index e5f220cdfa..a4b8ed3bf9 100644
@@ -5,14 +5,14 @@
 #
 # SPDX SDK tasks
 
-do_populate_sdk[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_populate_sdk[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_populate_sdk[cleandirs] += "${SPDXSDKWORK}"
 do_populate_sdk[postfuncs] += "sdk_create_sbom"
 do_populate_sdk[file-checksums] += "${SPDX3_DEP_FILES}"
 POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_create_spdx"
 POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_create_spdx"
 
-do_populate_sdk_ext[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_populate_sdk_ext[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_populate_sdk_ext[cleandirs] += "${SPDXSDKEXTWORK}"
 do_populate_sdk_ext[postfuncs] += "sdk_ext_create_sbom"
 do_populate_sdk_ext[file-checksums] += "${SPDX3_DEP_FILES}"
meta/classes-recipe/kernel.bbclass +1 -1
index f989b31c47..3a2c20dec2 100644
@@ -904,7 +904,7 @@ python do_create_kernel_config_spdx() {
             bb.error(f"Failed to parse kernel config file: {e}")
 
         build, build_objset = oe.sbom30.find_root_obj_in_jsonld(
-            d, "recipes", f"recipe-{pn}", oe.spdx30.build_Build
+            d, "builds", f"build-{pn}", oe.spdx30.build_Build
         )
 
         kernel_build = build_objset.add_root(
meta/classes-recipe/nospdx.bbclass +1 -0
index b20e28218b..90e14442ba 100644
@@ -5,6 +5,7 @@
 #
 
 deltask do_collect_spdx_deps
+deltask do_create_recipe_spdx
 deltask do_create_spdx
 deltask do_create_spdx_runtime
 deltask do_create_package_spdx
meta/classes/create-spdx-2.2.bbclass +11 -1
index 65d10d86db..f1ee0f9afd 100644
@@ -399,6 +399,15 @@ def get_license_list_version(license_data, d):
     return ".".join(license_data["licenseListVersion"].split(".")[:2])
 
 
+# This task is added for compatibility with tasks shared with SPDX 3, but
+# doesn't do anything
+do_create_recipe_spdx() {
+    :
+}
+do_create_recipe_spdx[noexec] = "1"
+addtask do_create_recipe_spdx after do_collect_spdx_deps
+
+
 python do_create_spdx() {
     from datetime import datetime, timezone
     import oe.sbom
@@ -594,7 +603,7 @@ python do_create_spdx() {
 }
 do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS"
 # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source
-addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work
+addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work
 
 SSTATETASKS += "do_create_spdx"
 do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}"
@@ -605,6 +614,7 @@ python do_create_spdx_setscene () {
 }
 addtask do_create_spdx_setscene
 
+do_create_spdx[deptask] += "do_create_spdx"
 do_create_spdx[dirs] = "${SPDXWORK}"
 do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}"
 do_create_spdx[depends] += " \
meta/classes/create-spdx-3.0.bbclass +37 -13

89 lines not on this page · raw message

meta/classes/spdx-common.bbclass +8 -7
index 3110230c9e..2804c27b0b 100644
@@ -23,6 +23,7 @@ SPDXDEPS = "${SPDXDIR}/deps.json"
 SPDX_TOOL_NAME ??= "oe-spdx-creator"
 SPDX_TOOL_VERSION ??= "1.0"
 
+SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy"
 SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy"
 
 SPDX_INCLUDE_SOURCES ??= "0"
@@ -72,6 +73,8 @@ def create_spdx_source_deps(d):
         # ourselves
         if oe.spdx_common.has_task(d, "do_unpack"):
             deps.append("%s:do_unpack" % pn)
+        if oe.spdx_common.has_task(d, "do_patch"):
+            deps.append("%s:do_patch" % pn)
 
         if oe.spdx_common.is_work_shared_spdx(d) and \
            oe.spdx_common.process_sources(d):
@@ -97,8 +100,8 @@ python do_collect_spdx_deps() {
     # This task calculates the build time dependencies of the recipe, and is
     # required because while a task can deptask on itself, those dependencies
     # do not show up in BB_TASKDEPDATA. To work around that, this task does the
-    # deptask on do_create_spdx and writes out the dependencies it finds, then
-    # do_create_spdx reads in the found dependencies when writing the actual
+    # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then
+    # downstream tasks read in the found dependencies when writing the actual
     # SPDX document
     import json
     import oe.spdx_common
@@ -106,15 +109,13 @@ python do_collect_spdx_deps() {
 
     spdx_deps_file = Path(d.getVar("SPDXDEPS"))
 
-    deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx")
+    deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx")
 
     with spdx_deps_file.open("w") as f:
         json.dump(deps, f)
 }
-# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source
-addtask do_collect_spdx_deps after do_unpack
-do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}"
-do_collect_spdx_deps[deptask] = "do_create_spdx"
+addtask do_collect_spdx_deps
+do_collect_spdx_deps[deptask] = "do_create_recipe_spdx"
 do_collect_spdx_deps[dirs] = "${SPDXDIR}"
 
 oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA"
meta/lib/oe/spdx30_tasks.py +279 -123

559 lines not on this page · raw message

Signature · 1 line
2.53.0
#28 lore raw

PATCH v3 4/8 spdx3: Add recipe SBoM task

In reply to Joshua Watt

Adds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example:

```
bitbake -c create_recipe_sbom zstd
```

Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively).

The complete SBoM for all (target) recipes can be built with:

```
bitbake -c create_recipe_sbom meta-world-recipe-sbom
```
---
 meta/classes/create-spdx-3.0.bbclass          | 28 +++++++++++++++++++
 meta/classes/spdx-common.bbclass              |  1 +
 meta/lib/oe/spdx30_tasks.py                   | 10 +++++++
 .../meta/meta-world-recipe-sbom.bb            | 28 +++++++++++++++++++
 4 files changed, 67 insertions(+)
 create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
meta/classes/create-spdx-3.0.bbclass +28 -0
index c5f6462c5a..782aa2c52e 100644
@@ -239,6 +239,34 @@ python do_create_package_spdx_setscene () {
 }
 addtask do_create_package_spdx_setscene
 
+addtask do_create_recipe_sbom after create_recipe_spdx
+python do_create_recipe_sbom() {
+    import oe.spdx30_tasks
+    from pathlib import Path
+    deploydir = Path(d.getVar("SPDXRECIPESBOMDEPLOY"))
+    oe.spdx30_tasks.create_recipe_sbom(d, deploydir)
+}
+
+SSTATETASKS += "do_create_recipe_sbom"
+do_create_recipe_sbom[recrdeptask] = "do_create_recipe_spdx"
+do_create_recipe_sbom[nostamp] = "1"
+do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}"
+do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
+do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}"
+do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}"
+do_create_recipe_sbom[vardeps] += "\
+    SPDX_INCLUDE_BITBAKE_PARENT_BUILD \
+    SPDX_PACKAGE_ADDITIONAL_PURPOSE \
+    SPDX_PROFILES \
+    SPDX_NAMESPACE_PREFIX \
+    SPDX_UUID_NAMESPACE \
+    "
+
+python do_create_recipe_sbom_setscene () {
+    sstate_setscene(d)
+}
+addtask do_create_recipe_sbom_setscene
+
 python spdx30_build_started_handler () {
     import oe.spdx30_tasks
     d = e.data.createCopy()
meta/classes/spdx-common.bbclass +1 -0
index 2804c27b0b..1ec4877a6a 100644
@@ -25,6 +25,7 @@ SPDX_TOOL_VERSION ??= "1.0"
 
 SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy"
 SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy"
+SPDXRECIPESBOMDEPLOY = "${SPDXDIR}/recipes-bom-deploy"
 
 SPDX_INCLUDE_SOURCES ??= "0"
 SPDX_INCLUDE_COMPILED_SOURCES ??= "0"
meta/lib/oe/spdx30_tasks.py +10 -0
index a8b4525e3d..9a312a870d 100644
@@ -1564,3 +1564,13 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname):
     oe.sbom30.write_jsonld_doc(
         d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json")
     )
+
+
+def create_recipe_sbom(d, deploydir):
+    sbom_name = d.getVar("PN") + "-recipe-sbom"
+
+    recipe, recipe_objset = load_recipe_spdx(d)
+
+    objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset])
+
+    oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json"))
meta/recipes-core/meta/meta-world-recipe-sbom.bb new file +28 -0
new file mode 100644
index 0000000000..5731e27c83
@@ -0,0 +1,28 @@
+SUMMARY = "Generates a combined SBoM for all world recipes"
+LICENSE = "MIT"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+PACKAGE_ARCH = "${MACHINE_ARCH}"
+
+inherit nopackages
+deltask do_fetch
+deltask do_unpack
+deltask do_patch
+deltask do_configure
+deltask do_compile
+deltask do_install
+
+do_prepare_recipe_sysroot[deptask] = ""
+
+WORLD_SBOM_EXCLUDE ?= ""
+
+EXCLUDE_FROM_WORLD = "1"
+
+python calculate_extra_depends() {
+    exclude = set('${WORLD_SBOM_EXCLUDE}'.split())
+    exclude |= set(f"{v}-{self_pn}" for v in '${MULTILIB_VARIANTS}'.split())
+    exclude.add(self_pn)
+
+    deps.extend(p for p in world_target if p not in exclude)
+}
Signature · 1 line
2.53.0
#29 lore raw

PATCH v3 5/8 spdx3: Add is-native property

In reply to Joshua Watt

Adds a custom is-native property to the recipe package to indicate if it is a native recipe

---
 meta/lib/oe/sbom30.py       | 20 ++++++++++++++++++++
 meta/lib/oe/spdx30_tasks.py | 18 +++++++++++-------
 2 files changed, 31 insertions(+), 7 deletions(-)
meta/lib/oe/sbom30.py +20 -0
index 227ac51877..50a72fce39 100644
@@ -118,6 +118,26 @@ class OEDocumentExtension(oe.spdx30.extension_Extension):
         )
 
 
[email protected](OE_SPDX_BASE + "recipe-extension")
+class OERecipeExtension(oe.spdx30.extension_Extension):
+    """
+    This extension is added to recipe software_Packages to indicate various
+    useful bits of information about the recipe
+    """
+
+    CLOSED = True
+
+    @classmethod
+    def _register_props(cls):
+        super()._register_props()
+        cls._add_property(
+            "is_native",
+            oe.spdx30.BooleanProp(),
+            OE_SPDX_BASE + "is-native",
+            max_count=1,
+        )
+
+
 def spdxid_hash(*items):
     h = hashlib.md5()
     for i in items:
meta/lib/oe/spdx30_tasks.py +11 -7
index 9a312a870d..fff1ca6bea 100644
@@ -477,6 +477,10 @@ def set_purls(spdx_package, purls):
         )
 
 
+def get_is_native(d):
+    return bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d)
+
+
 def create_recipe_spdx(d):
     deploydir = Path(d.getVar("SPDXRECIPEDEPLOY"))
     deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
@@ -507,6 +511,11 @@ def create_recipe_spdx(d):
         )
     )
 
+    if get_is_native(d):
+        ext = oe.sbom30.OERecipeExtension()
+        ext.is_native = True
+        recipe.extension.append(ext)
+
     set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split())
 
     # TODO: This doesn't work before do_unpack because the license text has to
@@ -668,9 +677,7 @@ def create_spdx(d):
     spdx_workdir = Path(d.getVar("SPDXWORK"))
     include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1"
     pkg_arch = d.getVar("SSTATE_PKGARCH")
-    is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class(
-        "cross", d
-    )
+    is_native = get_is_native(d)
 
     recipe, recipe_objset = load_recipe_spdx(d)
 
@@ -1021,14 +1028,11 @@ def create_spdx(d):
 def create_package_spdx(d):
     deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
     deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY"))
-    is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class(
-        "cross", d
-    )
 
     providers = oe.spdx_common.collect_package_providers(d)
     pkg_arch = d.getVar("SSTATE_PKGARCH")
 
-    if is_native:
+    if get_is_native(d):
         return
 
     bb.build.exec_func("read_subpackage_metadata", d)
Signature · 1 line
2.53.0
#30 lore raw

PATCH v3 6/8 spdx30: Include patch file information in VEX

In reply to Joshua Watt

Modifies the SPDX VEX output to include the patches that fix a particular vulnerability. This is done by adding a patchedBy relationship from the VexFixedVulnAssessmentRelationship to the File that provides the fix.

If the file can be located without fetching (e.g. is a file:// in SRC_URI), the checksum will be included.

---
 meta/lib/oe/sbom30.py       | 60 ++++++++++++++-------------
 meta/lib/oe/spdx30_tasks.py | 81 ++++++++++++++++++++++++++++---------
 2 files changed, 92 insertions(+), 49 deletions(-)
meta/lib/oe/sbom30.py +31 -29
index 50a72fce39..21f084dc16 100644
@@ -620,37 +620,38 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
         )
         spdx_file.extension.append(OELicenseScannedExtension())
 
-    def new_file(self, _id, name, path, *, purposes=[]):
-        sha256_hash = bb.utils.sha256_file(path)
+    def new_file(self, _id, name, path, *, purposes=[], hashfile=True):
+        if hashfile:
+            sha256_hash = bb.utils.sha256_file(path)
 
-        for f in self.by_sha256_hash.get(sha256_hash, []):
-            if not isinstance(f, oe.spdx30.software_File):
-                continue
+            for f in self.by_sha256_hash.get(sha256_hash, []):
+                if not isinstance(f, oe.spdx30.software_File):
+                    continue
 
-            if purposes:
-                new_primary = purposes[0]
-                new_additional = []
+                if purposes:
+                    new_primary = purposes[0]
+                    new_additional = []
 
-                if f.software_primaryPurpose:
-                    new_additional.append(f.software_primaryPurpose)
-                new_additional.extend(f.software_additionalPurpose)
+                    if f.software_primaryPurpose:
+                        new_additional.append(f.software_primaryPurpose)
+                    new_additional.extend(f.software_additionalPurpose)
 
-                new_additional = sorted(
-                    list(set(p for p in new_additional if p != new_primary))
-                )
+                    new_additional = sorted(
+                        list(set(p for p in new_additional if p != new_primary))
+                    )
 
-                f.software_primaryPurpose = new_primary
-                f.software_additionalPurpose = new_additional
+                    f.software_primaryPurpose = new_primary
+                    f.software_additionalPurpose = new_additional
 
-            if f.name != name:
-                for e in f.extension:
-                    if isinstance(e, OEFileNameAliasExtension):
-                        e.aliases.append(name)
-                        break
-                else:
-                    f.extension.append(OEFileNameAliasExtension(aliases=[name]))
+                if f.name != name:
+                    for e in f.extension:
+                        if isinstance(e, OEFileNameAliasExtension):
+                            e.aliases.append(name)
+                            break
+                    else:
+                        f.extension.append(OEFileNameAliasExtension(aliases=[name]))
 
-            return f
+                return f
 
         spdx_file = oe.spdx30.software_File(
             _id=_id,
@@ -661,12 +662,13 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
             spdx_file.software_primaryPurpose = purposes[0]
             spdx_file.software_additionalPurpose = purposes[1:]
 
-        spdx_file.verifiedUsing.append(
-            oe.spdx30.Hash(
-                algorithm=oe.spdx30.HashAlgorithm.sha256,
-                hashValue=sha256_hash,
+        if hashfile:
+            spdx_file.verifiedUsing.append(
+                oe.spdx30.Hash(
+                    algorithm=oe.spdx30.HashAlgorithm.sha256,
+                    hashValue=sha256_hash,
+                )
             )
-        )
 
         return self.add(spdx_file)
 
meta/lib/oe/spdx30_tasks.py +61 -20

139 lines not on this page · raw message

Signature · 1 line
2.53.0
#31 lore raw

PATCH v3 7/8 spdx: De-duplicate CreationInfo

In reply to Joshua Watt

De-duplicates CreationInfo objects that are identical (except for ID) when writing out an SBoM. This significantly reduces the number of CreationInfo objects that end up in the final document.

---
 meta/lib/oe/sbom30.py | 112 ++++++++++++++++++++++++++++++------------
 meta/lib/oe/spdx30.py |   2 +-
 2 files changed, 81 insertions(+), 33 deletions(-)
meta/lib/oe/sbom30.py +80 -32
index 21f084dc16..55a2863d2d 100644
@@ -14,6 +14,7 @@ import uuid
 import os
 import oe.spdx_common
 from datetime import datetime, timezone
+from contextlib import contextmanager
 
 OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/"
 
@@ -191,6 +192,25 @@ def to_list(l):
     return l
 
 
+class Dedup(object):
+    def __init__(self, objset):
+        self.unique = set()
+        self.dedup = {}
+        self.objset = objset
+
+    def find_duplicates(self, cmp, typ, **kwargs):
+        for o in self.objset.foreach_filter(typ, **kwargs):
+            for u in self.unique:
+                if cmp(u, o):
+                    self.dedup[o] = u
+                    break
+            else:
+                self.unique.add(o)
+
+    def get(self, o):
+        return self.dedup.get(o, o)
+
+
 class ObjectSet(oe.spdx30.SHACLObjectSet):
     def __init__(self, d):
         super().__init__()
@@ -895,6 +915,45 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
         self.missing_ids -= set(imports.keys())
         return self.missing_ids
 
+    @contextmanager
+    def deduplicate(self):
+        d = Dedup(self)
+
+        yield d
+
+        visited = set()
+
+        def visit(o, path):
+            if isinstance(o, oe.spdx30.SHACLObject):
+                if o in visited:
+                    return False
+                visited.add(o)
+
+                for k in o:
+                    v = o[k]
+                    if isinstance(v, oe.spdx30.SHACLObject):
+                        o[k] = d.get(v)
+
+            elif isinstance(o, oe.spdx30.ListProxy):
+                for idx, v in enumerate(o):
+                    if isinstance(v, oe.spdx30.SHACLObject):
+                        o[idx] = d.get(v)
+
+            return True
+
+        if d.dedup:
+            for o in self.objects:
+                o.walk(visit)
+
+            for k, v in d.dedup.items():
+                bb.debug(
+                    1,
+                    f"Removing duplicate {k.__class__.__name__} {k._id or id(k)} -> {v._id or id(v)}",
+                )
+                self.objects.discard(k)
+
+            self.create_index()
+
 
 def load_jsonld(d, path, required=False):
     deserializer = oe.spdx30.JSONLDDeserializer()
@@ -1080,39 +1139,28 @@ def create_sbom(d, name, root_elements, add_objectsets=[]):
     # SBoM should be the only root element of the document
     objset.doc.rootElement = [sbom]
 
-    # De-duplicate licenses
-    unique = set()
-    dedup = {}
-    for lic in objset.foreach_type(oe.spdx30.simplelicensing_LicenseExpression):
-        for u in unique:
-            if (
-                u.simplelicensing_licenseExpression
-                == lic.simplelicensing_licenseExpression
-                and u.simplelicensing_licenseListVersion
-                == lic.simplelicensing_licenseListVersion
-            ):
-                dedup[lic] = u
-                break
-        else:
-            unique.add(lic)
-
-    if dedup:
-        for rel in objset.foreach_filter(
-            oe.spdx30.Relationship,
-            relationshipType=oe.spdx30.RelationshipType.hasDeclaredLicense,
-        ):
-            rel.to = [dedup.get(to, to) for to in rel.to]
-
-        for rel in objset.foreach_filter(
-            oe.spdx30.Relationship,
-            relationshipType=oe.spdx30.RelationshipType.hasConcludedLicense,
-        ):
-            rel.to = [dedup.get(to, to) for to in rel.to]
+    def cmp_license_expression(a, b):
+        return (
+            a.simplelicensing_licenseExpression == b.simplelicensing_licenseExpression
+            and a.simplelicensing_licenseListVersion
+            == b.simplelicensing_licenseListVersion
+        )
 
-        for k, v in dedup.items():
-            bb.debug(1, f"Removing duplicate License {k._id} -> {v._id}")
-            objset.objects.remove(k)
+    def cmp_creation_info(a, b):
+        data_a = {k: a[k] for k in a}
+        data_b = {k: b[k] for k in b}
+        data_a["@id"] = ""
+        data_b["@id"] = ""
+        return data_a == data_b
+
+    with objset.deduplicate() as dedup:
+        # De-duplicate licenses
+        dedup.find_duplicates(
+            cmp_license_expression,
+            oe.spdx30.simplelicensing_LicenseExpression,
+        )
 
-        objset.create_index()
+        # Deduplicate creation info
+        dedup.find_duplicates(cmp_creation_info, oe.spdx30.CreationInfo)
 
     return objset, sbom
meta/lib/oe/spdx30.py +1 -1
index cd97eebd18..1f58402ffc 100644
@@ -701,7 +701,7 @@ class SHACLObject(object):
         self.__dict__["_obj_data"][iri] = prop.init()
 
     def __iter__(self):
-        return self._OBJ_PROPERTIES.keys()
+        return iter(self._OBJ_PROPERTIES.keys())
 
     def walk(self, callback, path=None):
         """
Signature · 1 line
2.53.0
#32 lore raw

PATCH v3 8/8 spdx_common: Check for dependent task in task flags

In reply to Joshua Watt

Checks that the task being used to detect dependencies is present in at least one dependency task flag of the current task. This helps prevent errors where the wrong task is specified and never found.

---
 meta/lib/oe/spdx_common.py | 11 +++++++++++
 1 file changed, 11 insertions(+)
meta/lib/oe/spdx_common.py +11 -0
index 72c24180d5..3aaf2a9c8b 100644
@@ -96,6 +96,17 @@ def collect_direct_deps(d, dep_task):
 
     taskdepdata = d.getVar("BB_TASKDEPDATA", False)
 
+    # Check that the task is listed one of the task dependency flags of the
+    # current task
+    depflags = (
+        set((d.getVarFlag(current_task, "deptask") or "").split())
+        | set((d.getVarFlag(current_task, "rdeptask") or "").split())
+        | set((d.getVarFlag(current_task, "recrdeptask") or "").split())
+    )
+
+    if not dep_task in depflags:
+        bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}")
+
     for this_dep in taskdepdata.values():
         if this_dep[0] == pn and this_dep[1] == current_task:
             break
Signature · 1 line
2.53.0
#33 lore raw
In reply to Joshua Watt
Quoted 14 lines · Joshua Watt via lists.openembedded.org

Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.

Special thanks to Benjamin Robin <[email protected]> for helping work through this.

V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).

Quoted 6 lines

V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run).

Hi Joshua,

Thanks for new version, results looks way better overall, but we still have a few errors.

I now have this issue on some builds:
ERROR: nativesdk-sdk-provides-dummy-1.0-r0 do_create_spdx: Could not find a static SPDX document named static-nativesdk-sdk-provides-dummy

https://autobuilder.yoctoproject.org/valkyrie/#/builders/68/builds/3335
https://autobuilder.yoctoproject.org/valkyrie/#/builders/45/builds/1105
https://autobuilder.yoctoproject.org/valkyrie/#/builders/40/builds/3238
https://autobuilder.yoctoproject.org/valkyrie/#/builders/30/builds/3233

And some errors in oe-selftests:

2026-02-26 20:46:06,214 - oe-selftest - INFO - newlib.NewlibTest.test_newlib (subunit.RemotedTestCase)
2026-02-26 20:46:06,215 - oe-selftest - INFO -  ... FAIL

...
ERROR: gcc-cross-x86_64-15.2.0-r0 do_create_spdx: Could not find a builds SPDX document named build-linux-libc-headers
...
ERROR: sysroot-test-1.0-r0 do_create_spdx: Could not find a builds SPDX document named build-sysroot-test-arch1
...

2026-02-26 21:03:31,870 - oe-selftest - INFO - sysroot.SysrootTests.test_sysroot_cleanup (subunit.RemotedTestCase)
2026-02-26 21:03:31,870 - oe-selftest - INFO -  ... FAIL
...
2026-02-26 21:20:09,006 - oe-selftest - INFO - spdx.SPDX30Check.test_custom_annotation_vars (subunit.RemotedTestCase)
2026-02-26 21:20:09,006 - oe-selftest - INFO -  ... FAIL
...
2026-02-26 21:20:09,006 - oe-selftest - INFO - 4: 37/50 612/670 (18.90s) (0 failed) (spdx.SPDX30Check.test_custom_annotation_vars)
2026-02-26 21:20:09,006 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 306, in test_custom_annotation_vars
    objset = self.check_recipe_spdx(
        "base-files",
    ...<7 lines>...
        ),
    )
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, in check_recipe_spdx
    return self.check_spdx_file(filename)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, in check_spdx_file
    self.assertExists(filename)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists
    raise self.failureException(msg)

AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700993/tmp/deploy/spdx/3.0.1/qemux86_64/recipes/recipe-base-files.spdx.json' does not exist
...
ERROR: gawk-native-5.3.2-r0 do_create_spdx: Applying patch '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/recipes-extended/gawk/gawk/0001-configure.ac-re-enable-disabled-printf-features.patch' on target directory '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700993/tmp/work/x86_64-linux/gawk-native/5.3.2/spdx/3.0.1/work/sources/gawk-5.3.2' CmdError('quilt --quiltrc /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700993/tmp/work/x86_64-linux/gawk-native/5.3.2/recipe-sysroot-native/etc/quiltrc push', 0, 'stdout:
stderr: /bin/sh: 1: quilt: not found
')
ERROR: bzip2-native-1.0.8-r0 do_create_spdx: Applying patch '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/recipes-extended/bzip2/bzip2/0001-fix-bzip2-version-tmp-aaa-will-hang.patch' on target directory '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700993/tmp/work/x86_64-linux/bzip2-native/1.0.8/spdx/3.0.1/work/sources/bzip2-1.0.8' CmdError('quilt --quiltrc /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700993/tmp/work/x86_64-linux/bzip2-native/1.0.8/recipe-sysroot-native/etc/quiltrc push', 0, 'stdout:
stderr: /bin/sh: 1: quilt: not found
')
...

2026-02-26 21:27:01,979 - oe-selftest - INFO - spdx.SPDX30Check.test_gcc_include_source (subunit.RemotedTestCase)
2026-02-26 21:27:01,980 - oe-selftest - INFO -  ... FAIL
...
2026-02-26 21:27:26,620 - oe-selftest - INFO - spdx.SPDX30Check.test_kernel_config_spdx (subunit.RemotedTestCase)
2026-02-26 21:27:26,621 - oe-selftest - INFO -  ... FAIL

...

2026-02-26 21:27:50,868 - oe-selftest - INFO - spdx.SPDX30Check.test_packageconfig_spdx (subunit.RemotedTestCase)
2026-02-26 21:27:50,868 - oe-selftest - INFO -  ... FAIL

...

Also, it looks like you did not add yourself as a maintainer of meta-world-recipe-sbom.

https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3288
https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3405
https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/3463

Thanks,
Mathieu

Signature · 3 lines
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#34 lore raw

v4 cover Add SPDX 3 Recipe Information

In reply to Joshua Watt

Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.

Special thanks to Benjamin Robin <[email protected]> for helping work through this.

V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).

V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run).

V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.

Joshua Watt (9):
llvm-project-source: Use allarch.bbclass
gcc-source: Use allarch.bbclass
spdx3: Add recipe SPDX data
spdx3: Add recipe SBoM task
spdx3: Add is-native property
spdx30: Include patch file information in VEX
spdx: De-duplicate CreationInfo
spdx_common: Check for dependent task in task flags
spdx30: Skip install package CVE information

 meta/classes-global/sstate.bbclass            |   4 +-
 .../create-spdx-image-3.0.bbclass             |   4 +-
 .../create-spdx-sdk-3.0.bbclass               |   4 +-
 meta/classes-recipe/kernel.bbclass            |   2 +-
 meta/classes-recipe/nospdx.bbclass            |   1 +
 meta/classes/create-spdx-2.2.bbclass          |  12 +-
 meta/classes/create-spdx-3.0.bbclass          |  92 +++-
 meta/classes/spdx-common.bbclass              |  22 +-
 meta/conf/distro/include/maintainers.inc      |   1 +
 meta/lib/oe/sbom30.py                         | 192 ++++---
 meta/lib/oe/spdx30.py                         |   2 +-
 meta/lib/oe/spdx30_tasks.py                   | 488 +++++++++++++-----
 meta/lib/oe/spdx_common.py                    |  11 +
 meta/lib/oeqa/selftest/cases/spdx.py          |  41 +-
 .../meta/meta-world-recipe-sbom.bb            |  29 ++
 .../clang/llvm-project-source.inc             |   8 +-
 meta/recipes-devtools/gcc/gcc-source.inc      |  16 +-
 17 files changed, 669 insertions(+), 260 deletions(-)
 create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
Signature · 1 line
2.53.0
#35 lore raw

PATCH v4 1/9 llvm-project-source: Use allarch.bbclass

Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)

---
 meta/recipes-devtools/clang/llvm-project-source.inc | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)
meta/recipes-devtools/clang/llvm-project-source.inc +1 -7
index 13e54efbc2..6bb595b7bc 100644
@@ -5,7 +5,7 @@ deltask do_populate_sysroot
 deltask do_populate_lic
 RM_WORK_EXCLUDE += "${PN}"
 
-inherit nopackages
+inherit nopackages allarch
 
 PN = "llvm-project-source-${PV}"
 WORKDIR = "${TMPDIR}/work-shared/llvm-project-source-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:llvm-project-source::${PV}:${PR}::${SSTATE_VERSION}:"
 STAMP = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-${PR}"
 STAMPCLEAN = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-*"
 
-INHIBIT_DEFAULT_DEPS = "1"
 DEPENDS = ""
 PACKAGES = ""
-TARGET_ARCH = "allarch"
-TARGET_AS_ARCH = "none"
-TARGET_CC_ARCH = "none"
-TARGET_LD_ARCH = "none"
-TARGET_OS = "linux"
 baselib = "lib"
 PACKAGE_ARCH = "all"
 
Signature · 1 line
2.53.0
#36 lore raw

PATCH v4 2/9 gcc-source: Use allarch.bbclass

In reply to Joshua Watt

Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)

---
 meta/recipes-devtools/gcc/gcc-source.inc | 16 +++++-----------
 1 file changed, 5 insertions(+), 11 deletions(-)
meta/recipes-devtools/gcc/gcc-source.inc +5 -11
index 265bcf4bef..3ac679b1a6 100644
@@ -1,11 +1,11 @@
-deltask do_configure 
-deltask do_compile 
-deltask do_install 
+deltask do_configure
+deltask do_compile
+deltask do_install
 deltask do_populate_sysroot
-deltask do_populate_lic 
+deltask do_populate_lic
 RM_WORK_EXCLUDE += "${PN}"
 
-inherit nopackages
+inherit nopackages allarch
 
 PN = "gcc-source-${PV}"
 WORKDIR = "${TMPDIR}/work-shared/gcc-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:gcc::${PV}:${PR}::${SSTATE_VERSION}:"
 STAMP = "${STAMPS_DIR}/work-shared/gcc-${PV}-${PR}"
 STAMPCLEAN = "${STAMPS_DIR}/work-shared/gcc-${PV}-*"
 
-INHIBIT_DEFAULT_DEPS = "1"
 DEPENDS = ""
 PACKAGES = ""
-TARGET_ARCH = "allarch"
-TARGET_AS_ARCH = "none"
-TARGET_CC_ARCH = "none"
-TARGET_LD_ARCH = "none"
-TARGET_OS = "linux"
 baselib = "lib"
 PACKAGE_ARCH = "all"
 
Signature · 1 line
2.53.0
#37 lore raw

PATCH v4 3/9 spdx3: Add recipe SPDX data

In reply to Joshua Watt

Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.

Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch

---
 meta/classes-global/sstate.bbclass            |   4 +-
 .../create-spdx-image-3.0.bbclass             |   4 +-
 .../create-spdx-sdk-3.0.bbclass               |   4 +-
 meta/classes-recipe/kernel.bbclass            |   2 +-
 meta/classes-recipe/nospdx.bbclass            |   1 +
 meta/classes/create-spdx-2.2.bbclass          |  12 +-
 meta/classes/create-spdx-3.0.bbclass          |  51 ++-
 meta/classes/spdx-common.bbclass              |  21 +-
 meta/lib/oe/spdx30_tasks.py                   | 402 ++++++++++++------
 meta/lib/oeqa/selftest/cases/spdx.py          |  19 +-
 10 files changed, 354 insertions(+), 166 deletions(-)
meta/classes-global/sstate.bbclass +2 -2
index 2fd29d7323..95c44f404e 100644
@@ -954,7 +954,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,
             extrapath = d.getVar("NATIVELSBSTRING") + "/"
         else:
             extrapath = ""
-        
+
         tname = bb.runqueue.taskname_from_tid(task)[3:]
 
         if tname in ["fetch", "unpack", "patch", "populate_lic", "preconfigure"] and splithashfn[2]:
@@ -1116,7 +1116,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None):
 
     logit("Considering setscene task: %s" % (str(taskdependees[task])), log)
 
-    directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_deploy_archives"]
+    directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_create_recipe_spdx", "do_deploy_archives"]
 
     def isNativeCross(x):
         return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross")
meta/classes-recipe/create-spdx-image-3.0.bbclass +2 -2
index 636ab14eb0..15a91e90e2 100644
@@ -34,7 +34,7 @@ addtask do_create_rootfs_spdx after do_rootfs before do_image
 SSTATETASKS += "do_create_rootfs_spdx"
 do_create_rootfs_spdx[sstate-inputdirs] = "${SPDXROOTFSDEPLOY}"
 do_create_rootfs_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}"
-do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_create_rootfs_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_create_rootfs_spdx[cleandirs] += "${SPDXROOTFSDEPLOY}"
 do_create_rootfs_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 
@@ -76,7 +76,7 @@ do_create_image_sbom_spdx[sstate-inputdirs] = "${SPDXIMAGEDEPLOYDIR}"
 do_create_image_sbom_spdx[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
 do_create_image_sbom_spdx[stamp-extra-info] = "${MACHINE_ARCH}"
 do_create_image_sbom_spdx[cleandirs] = "${SPDXIMAGEDEPLOYDIR}"
-do_create_image_sbom_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_create_image_sbom_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_create_image_sbom_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 
 python do_create_image_sbom_spdx_setscene() {
meta/classes-recipe/create-spdx-sdk-3.0.bbclass +2 -2
index e5f220cdfa..a4b8ed3bf9 100644
@@ -5,14 +5,14 @@
 #
 # SPDX SDK tasks
 
-do_populate_sdk[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_populate_sdk[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_populate_sdk[cleandirs] += "${SPDXSDKWORK}"
 do_populate_sdk[postfuncs] += "sdk_create_sbom"
 do_populate_sdk[file-checksums] += "${SPDX3_DEP_FILES}"
 POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_create_spdx"
 POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_create_spdx"
 
-do_populate_sdk_ext[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_populate_sdk_ext[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_populate_sdk_ext[cleandirs] += "${SPDXSDKEXTWORK}"
 do_populate_sdk_ext[postfuncs] += "sdk_ext_create_sbom"
 do_populate_sdk_ext[file-checksums] += "${SPDX3_DEP_FILES}"
meta/classes-recipe/kernel.bbclass +1 -1
index 22568d6e9c..6621a17f85 100644
@@ -895,7 +895,7 @@ do_create_spdx:append() {
         except Exception as e:
             bb.error(f"Failed to parse kernel config file: {e}")
 
-        path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "recipes", f"recipe-{pn}", deploydir=deploydir)
+        path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "builds", f"build-{pn}", deploydir=deploydir)
         build_objset = oe.sbom30.load_jsonld(d, path, required=True)
         build = build_objset.find_root(oe.spdx30.build_Build)
         if not build:
meta/classes-recipe/nospdx.bbclass +1 -0
index b20e28218b..90e14442ba 100644
@@ -5,6 +5,7 @@
 #
 
 deltask do_collect_spdx_deps
+deltask do_create_recipe_spdx
 deltask do_create_spdx
 deltask do_create_spdx_runtime
 deltask do_create_package_spdx
meta/classes/create-spdx-2.2.bbclass +11 -1
index 65d10d86db..3288cdf75a 100644
@@ -399,6 +399,15 @@ def get_license_list_version(license_data, d):
     return ".".join(license_data["licenseListVersion"].split(".")[:2])
 
 
+# This task is added for compatibility with tasks shared with SPDX 3, but
+# doesn't do anything
+do_create_recipe_spdx() {
+    :
+}
+do_create_recipe_spdx[noexec] = "1"
+addtask do_create_recipe_spdx after do_collect_spdx_deps
+
+
 python do_create_spdx() {
     from datetime import datetime, timezone
     import oe.sbom
@@ -594,7 +603,7 @@ python do_create_spdx() {
 }
 do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS"
 # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source
-addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work
+addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch do_collect_spdx_deps before do_populate_sdk do_build do_rm_work
 
 SSTATETASKS += "do_create_spdx"
 do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}"
@@ -605,6 +614,7 @@ python do_create_spdx_setscene () {
 }
 addtask do_create_spdx_setscene
 
+do_create_spdx[deptask] += "do_create_spdx"
 do_create_spdx[dirs] = "${SPDXWORK}"
 do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}"
 do_create_spdx[depends] += " \
meta/classes/create-spdx-3.0.bbclass +38 -13

90 lines not on this page · raw message

meta/classes/spdx-common.bbclass +6 -15
index 3110230c9e..3c239a718b 100644
@@ -23,6 +23,7 @@ SPDXDEPS = "${SPDXDIR}/deps.json"
 SPDX_TOOL_NAME ??= "oe-spdx-creator"
 SPDX_TOOL_VERSION ??= "1.0"
 
+SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy"
 SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy"
 
 SPDX_INCLUDE_SOURCES ??= "0"
@@ -67,12 +68,6 @@ def create_spdx_source_deps(d):
     deps = []
     if d.getVar("SPDX_INCLUDE_SOURCES") == "1":
         pn = d.getVar('PN')
-        # do_unpack is a hack for now; we only need it to get the
-        # dependencies do_unpack already has so we can extract the source
-        # ourselves
-        if oe.spdx_common.has_task(d, "do_unpack"):
-            deps.append("%s:do_unpack" % pn)
-
         if oe.spdx_common.is_work_shared_spdx(d) and \
            oe.spdx_common.process_sources(d):
             # For kernel source code
@@ -84,8 +79,6 @@ def create_spdx_source_deps(d):
             # For gcc-source-${PV} source code
             if oe.spdx_common.has_task(d, "do_preconfigure"):
                 deps.append("%s:do_preconfigure" % pn)
-            elif oe.spdx_common.has_task(d, "do_patch"):
-                deps.append("%s:do_patch" % pn)
             # For gcc-cross-x86_64 source code
             elif oe.spdx_common.has_task(d, "do_configure"):
                 deps.append("%s:do_configure" % pn)
@@ -97,8 +90,8 @@ python do_collect_spdx_deps() {
     # This task calculates the build time dependencies of the recipe, and is
     # required because while a task can deptask on itself, those dependencies
     # do not show up in BB_TASKDEPDATA. To work around that, this task does the
-    # deptask on do_create_spdx and writes out the dependencies it finds, then
-    # do_create_spdx reads in the found dependencies when writing the actual
+    # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then
+    # downstream tasks read in the found dependencies when writing the actual
     # SPDX document
     import json
     import oe.spdx_common
@@ -106,15 +99,13 @@ python do_collect_spdx_deps() {
 
     spdx_deps_file = Path(d.getVar("SPDXDEPS"))
 
-    deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx")
+    deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx")
 
     with spdx_deps_file.open("w") as f:
         json.dump(deps, f)
 }
-# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source
-addtask do_collect_spdx_deps after do_unpack
-do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}"
-do_collect_spdx_deps[deptask] = "do_create_spdx"
+addtask do_collect_spdx_deps
+do_collect_spdx_deps[deptask] = "do_create_recipe_spdx"
 do_collect_spdx_deps[dirs] = "${SPDXDIR}"
 
 oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA"
meta/lib/oe/spdx30_tasks.py +279 -123

559 lines not on this page · raw message

meta/lib/oeqa/selftest/cases/spdx.py +12 -7
index 5830d7c087..759ca86b73 100644
@@ -141,6 +141,11 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
     SPDX_CLASS = "create-spdx-3.0"
 
     def test_base_files(self):
+        self.check_recipe_spdx(
+            "base-files",
+            "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/static/static-base-files.spdx.json",
+            task="create_recipe_spdx",
+        )
         self.check_recipe_spdx(
             "base-files",
             "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json",
@@ -149,7 +154,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
     def test_gcc_include_source(self):
         objset = self.check_recipe_spdx(
             "gcc",
-            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-gcc.spdx.json",
+            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-gcc.spdx.json",
             extraconf="""\
                 SPDX_INCLUDE_SOURCES = "1"
                 """,
@@ -162,12 +167,12 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
             if software_file.name == filename:
                 found = True
                 self.logger.info(
-                    f"The spdxId of {filename} in recipe-gcc.spdx.json is {software_file.spdxId}"
+                    f"The spdxId of {filename} in build-gcc.spdx.json is {software_file.spdxId}"
                 )
                 break
 
         self.assertTrue(
-            found, f"Not found source file {filename} in recipe-gcc.spdx.json\n"
+            found, f"Not found source file {filename} in build-gcc.spdx.json\n"
         )
 
     def test_core_image_minimal(self):
@@ -305,7 +310,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
         # This will fail with NameError if new_annotation() is called incorrectly
         objset = self.check_recipe_spdx(
             "base-files",
-            "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/recipes/recipe-base-files.spdx.json",
+            "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/builds/build-base-files.spdx.json",
             extraconf=textwrap.dedent(
                 f"""\
                 ANNOTATION1 = "{ANNOTATION_VAR1}"
@@ -360,8 +365,8 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
 
     def test_kernel_config_spdx(self):
         kernel_recipe = get_bb_var("PREFERRED_PROVIDER_virtual/kernel")
-        spdx_file = f"recipe-{kernel_recipe}.spdx.json"
-        spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/recipes/{spdx_file}"
+        spdx_file = f"build-{kernel_recipe}.spdx.json"
+        spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/builds/{spdx_file}"
 
         # Make sure kernel is configured first
         bitbake(f"-c configure {kernel_recipe}")
@@ -392,7 +397,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
     def test_packageconfig_spdx(self):
         objset = self.check_recipe_spdx(
             "tar",
-            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-tar.spdx.json",
+            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-tar.spdx.json",
             extraconf="""\
                 SPDX_INCLUDE_PACKAGECONFIG = "1"
                 """,
Signature · 1 line
2.53.0
#38 lore raw

PATCH v4 4/9 spdx3: Add recipe SBoM task

In reply to Joshua Watt

Adds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example:

```
bitbake -c create_recipe_sbom zstd
```

Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively).

The complete SBoM for all (target) recipes can be built with:

```
bitbake -c create_recipe_sbom meta-world-recipe-sbom
```
---
 meta/classes/create-spdx-3.0.bbclass          | 32 +++++++++++++++++++
 meta/classes/spdx-common.bbclass              |  1 +
 meta/conf/distro/include/maintainers.inc      |  1 +
 meta/lib/oe/spdx30_tasks.py                   | 10 ++++++
 meta/lib/oeqa/selftest/cases/spdx.py          | 10 ++++++
 .../meta/meta-world-recipe-sbom.bb            | 29 +++++++++++++++++
 6 files changed, 83 insertions(+)
 create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
meta/classes/create-spdx-3.0.bbclass +32 -0
index 672ca27cd0..c3ea95b8bc 100644
@@ -142,6 +142,10 @@ SPDX_PACKAGE_URLS[doc] = "A space separated list of Package URLs (purls) for \
     Override this variable to replace the default, otherwise append or prepend \
     to add additional purls."
 
+SPDX_RECIPE_SBOM_NAME ?= "${PN}-recipe-sbom"
+SPDX_RECIPE_SBOM_NAME[doc] = "The name of output recipe SBoM when using \
+    create_recipe_sbom"
+
 IMAGE_CLASSES:append = " create-spdx-image-3.0"
 SDK_CLASSES += "create-spdx-sdk-3.0"
 
@@ -240,6 +244,34 @@ python do_create_package_spdx_setscene () {
 }
 addtask do_create_package_spdx_setscene
 
+addtask do_create_recipe_sbom after create_recipe_spdx
+python do_create_recipe_sbom() {
+    import oe.spdx30_tasks
+    from pathlib import Path
+    deploydir = Path(d.getVar("SPDXRECIPESBOMDEPLOY"))
+    oe.spdx30_tasks.create_recipe_sbom(d, deploydir)
+}
+
+SSTATETASKS += "do_create_recipe_sbom"
+do_create_recipe_sbom[recrdeptask] = "do_create_recipe_spdx"
+do_create_recipe_sbom[nostamp] = "1"
+do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}"
+do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
+do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}"
+do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}"
+do_create_recipe_sbom[vardeps] += "\
+    SPDX_INCLUDE_BITBAKE_PARENT_BUILD \
+    SPDX_PACKAGE_ADDITIONAL_PURPOSE \
+    SPDX_PROFILES \
+    SPDX_NAMESPACE_PREFIX \
+    SPDX_UUID_NAMESPACE \
+    "
+
+python do_create_recipe_sbom_setscene () {
+    sstate_setscene(d)
+}
+addtask do_create_recipe_sbom_setscene
+
 python spdx30_build_started_handler () {
     import oe.spdx30_tasks
     d = e.data.createCopy()
meta/classes/spdx-common.bbclass +1 -0
index 3c239a718b..abf2332bee 100644
@@ -25,6 +25,7 @@ SPDX_TOOL_VERSION ??= "1.0"
 
 SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy"
 SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy"
+SPDXRECIPESBOMDEPLOY = "${SPDXDIR}/recipes-bom-deploy"
 
 SPDX_INCLUDE_SOURCES ??= "0"
 SPDX_INCLUDE_COMPILED_SOURCES ??= "0"
meta/conf/distro/include/maintainers.inc +1 -0
index b5ab35d92a..5bea863798 100644
@@ -532,6 +532,7 @@ RECIPE_MAINTAINER:pn-meta-go-toolchain = "Richard Purdie <richard.purdie@linuxfo
 RECIPE_MAINTAINER:pn-meta-ide-support = "Richard Purdie <[email protected]>"
 RECIPE_MAINTAINER:pn-meta-toolchain = "Richard Purdie <[email protected]>"
 RECIPE_MAINTAINER:pn-meta-world-pkgdata = "Richard Purdie <[email protected]>"
+RECIPE_MAINTAINER:pn-meta-world-recipe-sbom = "Joshua Watt <[email protected]>"
 RECIPE_MAINTAINER:pn-mingetty = "Yi Zhao <[email protected]>"
 RECIPE_MAINTAINER:pn-mini-x-session = "Unassigned <[email protected]>"
 RECIPE_MAINTAINER:pn-minicom = "Unassigned <[email protected]>"
meta/lib/oe/spdx30_tasks.py +10 -0
index a8b4525e3d..b6c917045e 100644
@@ -1564,3 +1564,13 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname):
     oe.sbom30.write_jsonld_doc(
         d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json")
     )
+
+
+def create_recipe_sbom(d, deploydir):
+    sbom_name = d.getVar("SPDX_RECIPE_SBOM_NAME")
+
+    recipe, recipe_objset = load_recipe_spdx(d)
+
+    objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset])
+
+    oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json"))
meta/lib/oeqa/selftest/cases/spdx.py +10 -0
index 759ca86b73..efee0214fc 100644
@@ -151,6 +151,16 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
             "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json",
         )
 
+    def test_world_sbom(self):
+        objset = self.check_recipe_spdx(
+            "meta-world-recipe-sbom",
+            "{DEPLOY_DIR_IMAGE}/world-recipe-sbom.spdx.json",
+            task="create_recipe_sbom",
+        )
+
+        # Document should be fully linked
+        self.check_objset_missing_ids(objset)
+
     def test_gcc_include_source(self):
         objset = self.check_recipe_spdx(
             "gcc",
meta/recipes-core/meta/meta-world-recipe-sbom.bb new file +29 -0
new file mode 100644
index 0000000000..b47a3229c9
@@ -0,0 +1,29 @@
+SUMMARY = "Generates a combined SBoM for all world recipes"
+LICENSE = "MIT"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+PACKAGE_ARCH = "${MACHINE_ARCH}"
+
+inherit nopackages
+deltask do_fetch
+deltask do_unpack
+deltask do_patch
+deltask do_configure
+deltask do_compile
+deltask do_install
+
+do_prepare_recipe_sysroot[deptask] = ""
+
+WORLD_SBOM_EXCLUDE ?= ""
+
+EXCLUDE_FROM_WORLD = "1"
+SPDX_RECIPE_SBOM_NAME = "world-recipe-sbom"
+
+python calculate_extra_depends() {
+    exclude = set('${WORLD_SBOM_EXCLUDE}'.split())
+    exclude |= set(f"{v}-{self_pn}" for v in '${MULTILIB_VARIANTS}'.split())
+    exclude.add(self_pn)
+
+    deps.extend(p for p in world_target if p not in exclude)
+}
Signature · 1 line
2.53.0
#39 lore raw

PATCH v4 5/9 spdx3: Add is-native property

In reply to Joshua Watt

Adds a custom is-native property to the recipe package to indicate if it is a native recipe

---
 meta/lib/oe/sbom30.py       | 20 ++++++++++++++++++++
 meta/lib/oe/spdx30_tasks.py | 18 +++++++++++-------
 2 files changed, 31 insertions(+), 7 deletions(-)
meta/lib/oe/sbom30.py +20 -0
index 227ac51877..50a72fce39 100644
@@ -118,6 +118,26 @@ class OEDocumentExtension(oe.spdx30.extension_Extension):
         )
 
 
[email protected](OE_SPDX_BASE + "recipe-extension")
+class OERecipeExtension(oe.spdx30.extension_Extension):
+    """
+    This extension is added to recipe software_Packages to indicate various
+    useful bits of information about the recipe
+    """
+
+    CLOSED = True
+
+    @classmethod
+    def _register_props(cls):
+        super()._register_props()
+        cls._add_property(
+            "is_native",
+            oe.spdx30.BooleanProp(),
+            OE_SPDX_BASE + "is-native",
+            max_count=1,
+        )
+
+
 def spdxid_hash(*items):
     h = hashlib.md5()
     for i in items:
meta/lib/oe/spdx30_tasks.py +11 -7
index b6c917045e..a8fffbb085 100644
@@ -477,6 +477,10 @@ def set_purls(spdx_package, purls):
         )
 
 
+def get_is_native(d):
+    return bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d)
+
+
 def create_recipe_spdx(d):
     deploydir = Path(d.getVar("SPDXRECIPEDEPLOY"))
     deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
@@ -507,6 +511,11 @@ def create_recipe_spdx(d):
         )
     )
 
+    if get_is_native(d):
+        ext = oe.sbom30.OERecipeExtension()
+        ext.is_native = True
+        recipe.extension.append(ext)
+
     set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split())
 
     # TODO: This doesn't work before do_unpack because the license text has to
@@ -668,9 +677,7 @@ def create_spdx(d):
     spdx_workdir = Path(d.getVar("SPDXWORK"))
     include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1"
     pkg_arch = d.getVar("SSTATE_PKGARCH")
-    is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class(
-        "cross", d
-    )
+    is_native = get_is_native(d)
 
     recipe, recipe_objset = load_recipe_spdx(d)
 
@@ -1021,14 +1028,11 @@ def create_spdx(d):
 def create_package_spdx(d):
     deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
     deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY"))
-    is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class(
-        "cross", d
-    )
 
     providers = oe.spdx_common.collect_package_providers(d)
     pkg_arch = d.getVar("SSTATE_PKGARCH")
 
-    if is_native:
+    if get_is_native(d):
         return
 
     bb.build.exec_func("read_subpackage_metadata", d)
Signature · 1 line
2.53.0
#40 lore raw

PATCH v4 6/9 spdx30: Include patch file information in VEX

In reply to Joshua Watt

Modifies the SPDX VEX output to include the patches that fix a particular vulnerability. This is done by adding a patchedBy relationship from the VexFixedVulnAssessmentRelationship to the File that provides the fix.

If the file can be located without fetching (e.g. is a file:// in SRC_URI), the checksum will be included.

---
 meta/lib/oe/sbom30.py       | 60 ++++++++++++++-------------
 meta/lib/oe/spdx30_tasks.py | 81 ++++++++++++++++++++++++++++---------
 2 files changed, 92 insertions(+), 49 deletions(-)
meta/lib/oe/sbom30.py +31 -29
index 50a72fce39..21f084dc16 100644
@@ -620,37 +620,38 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
         )
         spdx_file.extension.append(OELicenseScannedExtension())
 
-    def new_file(self, _id, name, path, *, purposes=[]):
-        sha256_hash = bb.utils.sha256_file(path)
+    def new_file(self, _id, name, path, *, purposes=[], hashfile=True):
+        if hashfile:
+            sha256_hash = bb.utils.sha256_file(path)
 
-        for f in self.by_sha256_hash.get(sha256_hash, []):
-            if not isinstance(f, oe.spdx30.software_File):
-                continue
+            for f in self.by_sha256_hash.get(sha256_hash, []):
+                if not isinstance(f, oe.spdx30.software_File):
+                    continue
 
-            if purposes:
-                new_primary = purposes[0]
-                new_additional = []
+                if purposes:
+                    new_primary = purposes[0]
+                    new_additional = []
 
-                if f.software_primaryPurpose:
-                    new_additional.append(f.software_primaryPurpose)
-                new_additional.extend(f.software_additionalPurpose)
+                    if f.software_primaryPurpose:
+                        new_additional.append(f.software_primaryPurpose)
+                    new_additional.extend(f.software_additionalPurpose)
 
-                new_additional = sorted(
-                    list(set(p for p in new_additional if p != new_primary))
-                )
+                    new_additional = sorted(
+                        list(set(p for p in new_additional if p != new_primary))
+                    )
 
-                f.software_primaryPurpose = new_primary
-                f.software_additionalPurpose = new_additional
+                    f.software_primaryPurpose = new_primary
+                    f.software_additionalPurpose = new_additional
 
-            if f.name != name:
-                for e in f.extension:
-                    if isinstance(e, OEFileNameAliasExtension):
-                        e.aliases.append(name)
-                        break
-                else:
-                    f.extension.append(OEFileNameAliasExtension(aliases=[name]))
+                if f.name != name:
+                    for e in f.extension:
+                        if isinstance(e, OEFileNameAliasExtension):
+                            e.aliases.append(name)
+                            break
+                    else:
+                        f.extension.append(OEFileNameAliasExtension(aliases=[name]))
 
-            return f
+                return f
 
         spdx_file = oe.spdx30.software_File(
             _id=_id,
@@ -661,12 +662,13 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
             spdx_file.software_primaryPurpose = purposes[0]
             spdx_file.software_additionalPurpose = purposes[1:]
 
-        spdx_file.verifiedUsing.append(
-            oe.spdx30.Hash(
-                algorithm=oe.spdx30.HashAlgorithm.sha256,
-                hashValue=sha256_hash,
+        if hashfile:
+            spdx_file.verifiedUsing.append(
+                oe.spdx30.Hash(
+                    algorithm=oe.spdx30.HashAlgorithm.sha256,
+                    hashValue=sha256_hash,
+                )
             )
-        )
 
         return self.add(spdx_file)
 
meta/lib/oe/spdx30_tasks.py +61 -20

139 lines not on this page · raw message

Signature · 1 line
2.53.0
#41 lore raw

PATCH v4 7/9 spdx: De-duplicate CreationInfo

In reply to Joshua Watt

De-duplicates CreationInfo objects that are identical (except for ID) when writing out an SBoM. This significantly reduces the number of CreationInfo objects that end up in the final document.

---
 meta/lib/oe/sbom30.py | 112 ++++++++++++++++++++++++++++++------------
 meta/lib/oe/spdx30.py |   2 +-
 2 files changed, 81 insertions(+), 33 deletions(-)
meta/lib/oe/sbom30.py +80 -32
index 21f084dc16..55a2863d2d 100644
@@ -14,6 +14,7 @@ import uuid
 import os
 import oe.spdx_common
 from datetime import datetime, timezone
+from contextlib import contextmanager
 
 OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/"
 
@@ -191,6 +192,25 @@ def to_list(l):
     return l
 
 
+class Dedup(object):
+    def __init__(self, objset):
+        self.unique = set()
+        self.dedup = {}
+        self.objset = objset
+
+    def find_duplicates(self, cmp, typ, **kwargs):
+        for o in self.objset.foreach_filter(typ, **kwargs):
+            for u in self.unique:
+                if cmp(u, o):
+                    self.dedup[o] = u
+                    break
+            else:
+                self.unique.add(o)
+
+    def get(self, o):
+        return self.dedup.get(o, o)
+
+
 class ObjectSet(oe.spdx30.SHACLObjectSet):
     def __init__(self, d):
         super().__init__()
@@ -895,6 +915,45 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
         self.missing_ids -= set(imports.keys())
         return self.missing_ids
 
+    @contextmanager
+    def deduplicate(self):
+        d = Dedup(self)
+
+        yield d
+
+        visited = set()
+
+        def visit(o, path):
+            if isinstance(o, oe.spdx30.SHACLObject):
+                if o in visited:
+                    return False
+                visited.add(o)
+
+                for k in o:
+                    v = o[k]
+                    if isinstance(v, oe.spdx30.SHACLObject):
+                        o[k] = d.get(v)
+
+            elif isinstance(o, oe.spdx30.ListProxy):
+                for idx, v in enumerate(o):
+                    if isinstance(v, oe.spdx30.SHACLObject):
+                        o[idx] = d.get(v)
+
+            return True
+
+        if d.dedup:
+            for o in self.objects:
+                o.walk(visit)
+
+            for k, v in d.dedup.items():
+                bb.debug(
+                    1,
+                    f"Removing duplicate {k.__class__.__name__} {k._id or id(k)} -> {v._id or id(v)}",
+                )
+                self.objects.discard(k)
+
+            self.create_index()
+
 
 def load_jsonld(d, path, required=False):
     deserializer = oe.spdx30.JSONLDDeserializer()
@@ -1080,39 +1139,28 @@ def create_sbom(d, name, root_elements, add_objectsets=[]):
     # SBoM should be the only root element of the document
     objset.doc.rootElement = [sbom]
 
-    # De-duplicate licenses
-    unique = set()
-    dedup = {}
-    for lic in objset.foreach_type(oe.spdx30.simplelicensing_LicenseExpression):
-        for u in unique:
-            if (
-                u.simplelicensing_licenseExpression
-                == lic.simplelicensing_licenseExpression
-                and u.simplelicensing_licenseListVersion
-                == lic.simplelicensing_licenseListVersion
-            ):
-                dedup[lic] = u
-                break
-        else:
-            unique.add(lic)
-
-    if dedup:
-        for rel in objset.foreach_filter(
-            oe.spdx30.Relationship,
-            relationshipType=oe.spdx30.RelationshipType.hasDeclaredLicense,
-        ):
-            rel.to = [dedup.get(to, to) for to in rel.to]
-
-        for rel in objset.foreach_filter(
-            oe.spdx30.Relationship,
-            relationshipType=oe.spdx30.RelationshipType.hasConcludedLicense,
-        ):
-            rel.to = [dedup.get(to, to) for to in rel.to]
+    def cmp_license_expression(a, b):
+        return (
+            a.simplelicensing_licenseExpression == b.simplelicensing_licenseExpression
+            and a.simplelicensing_licenseListVersion
+            == b.simplelicensing_licenseListVersion
+        )
 
-        for k, v in dedup.items():
-            bb.debug(1, f"Removing duplicate License {k._id} -> {v._id}")
-            objset.objects.remove(k)
+    def cmp_creation_info(a, b):
+        data_a = {k: a[k] for k in a}
+        data_b = {k: b[k] for k in b}
+        data_a["@id"] = ""
+        data_b["@id"] = ""
+        return data_a == data_b
+
+    with objset.deduplicate() as dedup:
+        # De-duplicate licenses
+        dedup.find_duplicates(
+            cmp_license_expression,
+            oe.spdx30.simplelicensing_LicenseExpression,
+        )
 
-        objset.create_index()
+        # Deduplicate creation info
+        dedup.find_duplicates(cmp_creation_info, oe.spdx30.CreationInfo)
 
     return objset, sbom
meta/lib/oe/spdx30.py +1 -1
index cd97eebd18..1f58402ffc 100644
@@ -701,7 +701,7 @@ class SHACLObject(object):
         self.__dict__["_obj_data"][iri] = prop.init()
 
     def __iter__(self):
-        return self._OBJ_PROPERTIES.keys()
+        return iter(self._OBJ_PROPERTIES.keys())
 
     def walk(self, callback, path=None):
         """
Signature · 1 line
2.53.0
#42 lore raw

PATCH v4 8/9 spdx_common: Check for dependent task in task flags

In reply to Joshua Watt

Checks that the task being used to detect dependencies is present in at least one dependency task flag of the current task. This helps prevent errors where the wrong task is specified and never found.

---
 meta/lib/oe/spdx_common.py | 11 +++++++++++
 1 file changed, 11 insertions(+)
meta/lib/oe/spdx_common.py +11 -0
index 72c24180d5..3aaf2a9c8b 100644
@@ -96,6 +96,17 @@ def collect_direct_deps(d, dep_task):
 
     taskdepdata = d.getVar("BB_TASKDEPDATA", False)
 
+    # Check that the task is listed one of the task dependency flags of the
+    # current task
+    depflags = (
+        set((d.getVarFlag(current_task, "deptask") or "").split())
+        | set((d.getVarFlag(current_task, "rdeptask") or "").split())
+        | set((d.getVarFlag(current_task, "recrdeptask") or "").split())
+    )
+
+    if not dep_task in depflags:
+        bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}")
+
     for this_dep in taskdepdata.values():
         if this_dep[0] == pn and this_dep[1] == current_task:
             break
Signature · 1 line
2.53.0
#43 lore raw

PATCH v4 9/9 spdx30: Skip install package CVE information

In reply to Joshua Watt

Skips adding the install package CVE information by default. This information grows exponentially, since it ends up be N_CVES * N_PACKAGES. The CVE information for a given installed package can be determined by following the "generates" link between the install package and the recipe and looking at the CVE information for the recipe, meaning that the CVE information is only included once in the SPDX document.

If users still need the legacy method of including CVE information for each package, then then can set SPDX_PACKAGE_INCLUDE_VEX = "1"

---
 meta/classes/create-spdx-3.0.bbclass | 11 ++++++++
 meta/lib/oe/spdx30_tasks.py          | 39 ++++++++++++++--------------
 meta/lib/oeqa/selftest/cases/spdx.py | 12 +++++++++
 3 files changed, 43 insertions(+), 19 deletions(-)
meta/classes/create-spdx-3.0.bbclass +11 -0
index c3ea95b8bc..88b7ef9f42 100644
@@ -45,6 +45,17 @@ SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to
     including those already fixed upstream (warning: This can be large and \
     slow)."
 
+SPDX_PACKAGE_INCLUDE_VEX ?= "0"
+SPDX_PACKAGE_INCLUDE_VEX[doc] = "Link VEX information to the binary package outputs. \
+    Normally, VEX information is only linked to the common recipe that `generates` the \
+    binary packages, but setting this to '1' will cause it to also be linked into the \
+    generated binary packages. This is off by default because linking the VEX data to \
+    each package causes the SPDX output to grow very large, and the same information \
+    can be determined by following the `generates` relationship back to the recipe. \
+    Before recipe packages were introduced, this was the only way VEX data was \
+    expressed; you may need to enable this if your downstream tools do not \
+    understand how to trace back to the recipe to find VEX information."
+
 SPDX_INCLUDE_TIMESTAMPS ?= "0"
 SPDX_INCLUDE_TIMESTAMPS[doc] = "Include time stamps in SPDX output. This is \
     useful if you want to know when artifacts were produced and when builds \
meta/lib/oe/spdx30_tasks.py +20 -19
index aec47d4f81..887fac813a 100644
@@ -771,27 +771,28 @@ def create_spdx(d):
     # Collect all VEX statements from the recipe
     vex_statements = {}
     vex_patches = {}
-    for rel in recipe_objset.foreach_filter(
-        oe.spdx30.Relationship,
-        relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability,
-    ):
-        for cve in rel.to:
-            vex_statements[cve] = []
-            vex_patches[cve] = []
-
-    for cve in vex_statements.keys():
+    if (d.getVar("SPDX_PACKAGE_INCLUDE_VEX") or "") == "1":
         for rel in recipe_objset.foreach_filter(
-            oe.spdx30.security_VexVulnAssessmentRelationship,
-            from_=cve,
+            oe.spdx30.Relationship,
+            relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability,
         ):
-            vex_statements[cve].append(rel)
-            if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn:
-                for patch_rel in recipe_objset.foreach_filter(
-                    oe.spdx30.Relationship,
-                    relationshipType=oe.spdx30.RelationshipType.patchedBy,
-                    from_=rel,
-                ):
-                    vex_patches[cve].extend(patch_rel.to)
+            for cve in rel.to:
+                vex_statements[cve] = []
+                vex_patches[cve] = []
+
+        for cve in vex_statements.keys():
+            for rel in recipe_objset.foreach_filter(
+                oe.spdx30.security_VexVulnAssessmentRelationship,
+                from_=cve,
+            ):
+                vex_statements[cve].append(rel)
+                if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn:
+                    for patch_rel in recipe_objset.foreach_filter(
+                        oe.spdx30.Relationship,
+                        relationshipType=oe.spdx30.RelationshipType.patchedBy,
+                        from_=rel,
+                    ):
+                        vex_patches[cve].extend(patch_rel.to)
 
     # Write out the package SPDX data now. It is not complete as we cannot
     # write the runtime data, so write it to a staging area and a later task
meta/lib/oeqa/selftest/cases/spdx.py +12 -0
index efee0214fc..f1ea2694cf 100644
@@ -429,3 +429,15 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
                 value, ["enabled", "disabled"],
                 f"Unexpected PACKAGECONFIG value '{value}' for {key}"
             )
+
+    def test_package_vex(self):
+        objset = self.check_recipe_spdx(
+            "core-image-minimal",
+            "{DEPLOY_DIR_IMAGE}/core-image-minimal-{MACHINE}.rootfs.spdx.json",
+            extraconf="""\
+                SPDX_PACKAGE_INCLUDE_VEX = "1"
+                """,
+        )
+
+        # Document should be fully linked
+        self.check_objset_missing_ids(objset)
Signature · 1 line
2.53.0
#44 lore raw
In reply to Joshua Watt

Hi,

Quoted 5 lines · Joshua Watt via lists.openembedded.org

Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.

Once merged, would it be possible to submit a docs patch to document the new variables in the variable glossary, and possibly update documentation/dev-manual/sbom.rst to document this new feature?

Also, looking at the last patch, the CVE information removal seems worthy of a migration note in documentation/migration-guides/migration-6.0.rst?

Antonin

#45 lore raw
In reply to Joshua Watt
Quoted 46 lines · Joshua Watt via lists.openembedded.org

Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.

Special thanks to Benjamin Robin <[email protected]> for helping work through this.

V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).

V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run).

V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.

Joshua Watt (9):
llvm-project-source: Use allarch.bbclass
gcc-source: Use allarch.bbclass
spdx3: Add recipe SPDX data
spdx3: Add recipe SBoM task
spdx3: Add is-native property
spdx30: Include patch file information in VEX
spdx: De-duplicate CreationInfo
spdx_common: Check for dependent task in task flags
spdx30: Skip install package CVE information

 meta/classes-global/sstate.bbclass            |   4 +-
 .../create-spdx-image-3.0.bbclass             |   4 +-
 .../create-spdx-sdk-3.0.bbclass               |   4 +-
 meta/classes-recipe/kernel.bbclass            |   2 +-
 meta/classes-recipe/nospdx.bbclass            |   1 +
 meta/classes/create-spdx-2.2.bbclass          |  12 +-
 meta/classes/create-spdx-3.0.bbclass          |  92 +++-
 meta/classes/spdx-common.bbclass              |  22 +-
 meta/conf/distro/include/maintainers.inc      |   1 +
 meta/lib/oe/sbom30.py                         | 192 ++++---
 meta/lib/oe/spdx30.py                         |   2 +-
Quoted 8 lines
 meta/lib/oe/spdx30_tasks.py                   | 488 +++++++++++++-----
 meta/lib/oe/spdx_common.py                    |  11 +
 meta/lib/oeqa/selftest/cases/spdx.py          |  41 +-
 .../meta/meta-world-recipe-sbom.bb            |  29 ++
 .../clang/llvm-project-source.inc             |   8 +-
 meta/recipes-devtools/gcc/gcc-source.inc      |  16 +-
 17 files changed, 669 insertions(+), 260 deletions(-)
 create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb

Hi Joshua,

Thanks for the new version, but it looks like one of the two errors is still present on several builds:

ERROR: nativesdk-sdk-provides-dummy-1.0-r0 do_create_spdx: Could not find a static SPDX document named static-nativesdk-sdk-provides-dummy

https://autobuilder.yoctoproject.org/valkyrie/#/builders/16/builds/3310
https://autobuilder.yoctoproject.org/valkyrie/#/builders/30/builds/3262
https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/3282
https://autobuilder.yoctoproject.org/valkyrie/#/builders/40/builds/3267
...

Can you have a look at these?

Thanks,
Mathieu

Signature · 3 lines
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#46 lore raw

v5 cover Add SPDX 3 Recipe Information

In reply to Joshua Watt

Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.

Special thanks to Benjamin Robin <[email protected]> for helping work through this.

V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).

V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run).

V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.

V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway

Joshua Watt (13):
llvm-project-source: Use allarch.bbclass
gcc-source: Use allarch.bbclass
spdx3: Add recipe SPDX data
spdx3: Add recipe SBoM task
spdx3: Add is-native property
spdx30: Include patch file information in VEX
spdx: De-duplicate CreationInfo
spdx_common: Check for dependent task in task flags
spdx30: Skip install package CVE information
dummy-sdk-package: Disable SPDX
spdx: Remove fatal errors for missing providers
spdx3: Use common variable for vardeps
glibc-testsuite: Do not generate SPDX

 meta/classes-global/sstate.bbclass            |   4 +-
 .../create-spdx-image-3.0.bbclass             |   4 +-
 .../create-spdx-sdk-3.0.bbclass               |   4 +-
 meta/classes-recipe/kernel.bbclass            |   2 +-
 meta/classes-recipe/nospdx.bbclass            |   1 +
 meta/classes/create-spdx-2.2.bbclass          |  15 +-
 meta/classes/create-spdx-3.0.bbclass          |  87 ++-
 meta/classes/spdx-common.bbclass              |  22 +-
 meta/conf/distro/include/maintainers.inc      |   1 +
 meta/lib/oe/sbom30.py                         | 192 ++++---
 meta/lib/oe/spdx30.py                         |   2 +-
 meta/lib/oe/spdx30_tasks.py                   | 496 +++++++++++++-----
 meta/lib/oe/spdx_common.py                    |  11 +
 meta/lib/oeqa/selftest/cases/spdx.py          |  41 +-
 .../glibc/glibc-testsuite_2.42.bb             |   1 +
 meta/recipes-core/meta/dummy-sdk-package.inc  |   1 +
 .../meta/meta-world-recipe-sbom.bb            |  29 +
 .../clang/llvm-project-source.inc             |   8 +-
 meta/recipes-devtools/gcc/gcc-source.inc      |  16 +-
 19 files changed, 667 insertions(+), 270 deletions(-)
 create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
Signature · 1 line
2.53.0
#47 lore raw

PATCH v5 1/13 llvm-project-source: Use allarch.bbclass

Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)

---
 meta/recipes-devtools/clang/llvm-project-source.inc | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)
meta/recipes-devtools/clang/llvm-project-source.inc +1 -7
index 13e54efbc2..6bb595b7bc 100644
@@ -5,7 +5,7 @@ deltask do_populate_sysroot
 deltask do_populate_lic
 RM_WORK_EXCLUDE += "${PN}"
 
-inherit nopackages
+inherit nopackages allarch
 
 PN = "llvm-project-source-${PV}"
 WORKDIR = "${TMPDIR}/work-shared/llvm-project-source-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:llvm-project-source::${PV}:${PR}::${SSTATE_VERSION}:"
 STAMP = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-${PR}"
 STAMPCLEAN = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-*"
 
-INHIBIT_DEFAULT_DEPS = "1"
 DEPENDS = ""
 PACKAGES = ""
-TARGET_ARCH = "allarch"
-TARGET_AS_ARCH = "none"
-TARGET_CC_ARCH = "none"
-TARGET_LD_ARCH = "none"
-TARGET_OS = "linux"
 baselib = "lib"
 PACKAGE_ARCH = "all"
 
Signature · 1 line
2.53.0
#48 lore raw

PATCH v5 2/13 gcc-source: Use allarch.bbclass

In reply to Joshua Watt

Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)

---
 meta/recipes-devtools/gcc/gcc-source.inc | 16 +++++-----------
 1 file changed, 5 insertions(+), 11 deletions(-)
meta/recipes-devtools/gcc/gcc-source.inc +5 -11
index 265bcf4bef..3ac679b1a6 100644
@@ -1,11 +1,11 @@
-deltask do_configure 
-deltask do_compile 
-deltask do_install 
+deltask do_configure
+deltask do_compile
+deltask do_install
 deltask do_populate_sysroot
-deltask do_populate_lic 
+deltask do_populate_lic
 RM_WORK_EXCLUDE += "${PN}"
 
-inherit nopackages
+inherit nopackages allarch
 
 PN = "gcc-source-${PV}"
 WORKDIR = "${TMPDIR}/work-shared/gcc-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:gcc::${PV}:${PR}::${SSTATE_VERSION}:"
 STAMP = "${STAMPS_DIR}/work-shared/gcc-${PV}-${PR}"
 STAMPCLEAN = "${STAMPS_DIR}/work-shared/gcc-${PV}-*"
 
-INHIBIT_DEFAULT_DEPS = "1"
 DEPENDS = ""
 PACKAGES = ""
-TARGET_ARCH = "allarch"
-TARGET_AS_ARCH = "none"
-TARGET_CC_ARCH = "none"
-TARGET_LD_ARCH = "none"
-TARGET_OS = "linux"
 baselib = "lib"
 PACKAGE_ARCH = "all"
 
Signature · 1 line
2.53.0
#49 lore raw

PATCH v5 3/13 spdx3: Add recipe SPDX data

In reply to Joshua Watt

Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.

Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch

---
 meta/classes-global/sstate.bbclass            |   4 +-
 .../create-spdx-image-3.0.bbclass             |   4 +-
 .../create-spdx-sdk-3.0.bbclass               |   4 +-
 meta/classes-recipe/kernel.bbclass            |   2 +-
 meta/classes-recipe/nospdx.bbclass            |   1 +
 meta/classes/create-spdx-2.2.bbclass          |  12 +-
 meta/classes/create-spdx-3.0.bbclass          |  51 ++-
 meta/classes/spdx-common.bbclass              |  21 +-
 meta/lib/oe/spdx30_tasks.py                   | 402 ++++++++++++------
 meta/lib/oeqa/selftest/cases/spdx.py          |  19 +-
 10 files changed, 354 insertions(+), 166 deletions(-)
meta/classes-global/sstate.bbclass +2 -2
index 2fd29d7323..95c44f404e 100644
@@ -954,7 +954,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,
             extrapath = d.getVar("NATIVELSBSTRING") + "/"
         else:
             extrapath = ""
-        
+
         tname = bb.runqueue.taskname_from_tid(task)[3:]
 
         if tname in ["fetch", "unpack", "patch", "populate_lic", "preconfigure"] and splithashfn[2]:
@@ -1116,7 +1116,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None):
 
     logit("Considering setscene task: %s" % (str(taskdependees[task])), log)
 
-    directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_deploy_archives"]
+    directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_create_recipe_spdx", "do_deploy_archives"]
 
     def isNativeCross(x):
         return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross")
meta/classes-recipe/create-spdx-image-3.0.bbclass +2 -2
index 636ab14eb0..15a91e90e2 100644
@@ -34,7 +34,7 @@ addtask do_create_rootfs_spdx after do_rootfs before do_image
 SSTATETASKS += "do_create_rootfs_spdx"
 do_create_rootfs_spdx[sstate-inputdirs] = "${SPDXROOTFSDEPLOY}"
 do_create_rootfs_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}"
-do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_create_rootfs_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_create_rootfs_spdx[cleandirs] += "${SPDXROOTFSDEPLOY}"
 do_create_rootfs_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 
@@ -76,7 +76,7 @@ do_create_image_sbom_spdx[sstate-inputdirs] = "${SPDXIMAGEDEPLOYDIR}"
 do_create_image_sbom_spdx[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
 do_create_image_sbom_spdx[stamp-extra-info] = "${MACHINE_ARCH}"
 do_create_image_sbom_spdx[cleandirs] = "${SPDXIMAGEDEPLOYDIR}"
-do_create_image_sbom_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_create_image_sbom_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_create_image_sbom_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 
 python do_create_image_sbom_spdx_setscene() {
meta/classes-recipe/create-spdx-sdk-3.0.bbclass +2 -2
index e5f220cdfa..a4b8ed3bf9 100644
@@ -5,14 +5,14 @@
 #
 # SPDX SDK tasks
 
-do_populate_sdk[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_populate_sdk[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_populate_sdk[cleandirs] += "${SPDXSDKWORK}"
 do_populate_sdk[postfuncs] += "sdk_create_sbom"
 do_populate_sdk[file-checksums] += "${SPDX3_DEP_FILES}"
 POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_create_spdx"
 POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_create_spdx"
 
-do_populate_sdk_ext[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_populate_sdk_ext[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_populate_sdk_ext[cleandirs] += "${SPDXSDKEXTWORK}"
 do_populate_sdk_ext[postfuncs] += "sdk_ext_create_sbom"
 do_populate_sdk_ext[file-checksums] += "${SPDX3_DEP_FILES}"
meta/classes-recipe/kernel.bbclass +1 -1
index 22568d6e9c..6621a17f85 100644
@@ -895,7 +895,7 @@ do_create_spdx:append() {
         except Exception as e:
             bb.error(f"Failed to parse kernel config file: {e}")
 
-        path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "recipes", f"recipe-{pn}", deploydir=deploydir)
+        path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "builds", f"build-{pn}", deploydir=deploydir)
         build_objset = oe.sbom30.load_jsonld(d, path, required=True)
         build = build_objset.find_root(oe.spdx30.build_Build)
         if not build:
meta/classes-recipe/nospdx.bbclass +1 -0
index b20e28218b..90e14442ba 100644
@@ -5,6 +5,7 @@
 #
 
 deltask do_collect_spdx_deps
+deltask do_create_recipe_spdx
 deltask do_create_spdx
 deltask do_create_spdx_runtime
 deltask do_create_package_spdx
meta/classes/create-spdx-2.2.bbclass +11 -1
index 65d10d86db..3288cdf75a 100644
@@ -399,6 +399,15 @@ def get_license_list_version(license_data, d):
     return ".".join(license_data["licenseListVersion"].split(".")[:2])
 
 
+# This task is added for compatibility with tasks shared with SPDX 3, but
+# doesn't do anything
+do_create_recipe_spdx() {
+    :
+}
+do_create_recipe_spdx[noexec] = "1"
+addtask do_create_recipe_spdx after do_collect_spdx_deps
+
+
 python do_create_spdx() {
     from datetime import datetime, timezone
     import oe.sbom
@@ -594,7 +603,7 @@ python do_create_spdx() {
 }
 do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS"
 # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source
-addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work
+addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch do_collect_spdx_deps before do_populate_sdk do_build do_rm_work
 
 SSTATETASKS += "do_create_spdx"
 do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}"
@@ -605,6 +614,7 @@ python do_create_spdx_setscene () {
 }
 addtask do_create_spdx_setscene
 
+do_create_spdx[deptask] += "do_create_spdx"
 do_create_spdx[dirs] = "${SPDXWORK}"
 do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}"
 do_create_spdx[depends] += " \
meta/classes/create-spdx-3.0.bbclass +38 -13

90 lines not on this page · raw message

meta/classes/spdx-common.bbclass +6 -15
index 3110230c9e..3c239a718b 100644
@@ -23,6 +23,7 @@ SPDXDEPS = "${SPDXDIR}/deps.json"
 SPDX_TOOL_NAME ??= "oe-spdx-creator"
 SPDX_TOOL_VERSION ??= "1.0"
 
+SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy"
 SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy"
 
 SPDX_INCLUDE_SOURCES ??= "0"
@@ -67,12 +68,6 @@ def create_spdx_source_deps(d):
     deps = []
     if d.getVar("SPDX_INCLUDE_SOURCES") == "1":
         pn = d.getVar('PN')
-        # do_unpack is a hack for now; we only need it to get the
-        # dependencies do_unpack already has so we can extract the source
-        # ourselves
-        if oe.spdx_common.has_task(d, "do_unpack"):
-            deps.append("%s:do_unpack" % pn)
-
         if oe.spdx_common.is_work_shared_spdx(d) and \
            oe.spdx_common.process_sources(d):
             # For kernel source code
@@ -84,8 +79,6 @@ def create_spdx_source_deps(d):
             # For gcc-source-${PV} source code
             if oe.spdx_common.has_task(d, "do_preconfigure"):
                 deps.append("%s:do_preconfigure" % pn)
-            elif oe.spdx_common.has_task(d, "do_patch"):
-                deps.append("%s:do_patch" % pn)
             # For gcc-cross-x86_64 source code
             elif oe.spdx_common.has_task(d, "do_configure"):
                 deps.append("%s:do_configure" % pn)
@@ -97,8 +90,8 @@ python do_collect_spdx_deps() {
     # This task calculates the build time dependencies of the recipe, and is
     # required because while a task can deptask on itself, those dependencies
     # do not show up in BB_TASKDEPDATA. To work around that, this task does the
-    # deptask on do_create_spdx and writes out the dependencies it finds, then
-    # do_create_spdx reads in the found dependencies when writing the actual
+    # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then
+    # downstream tasks read in the found dependencies when writing the actual
     # SPDX document
     import json
     import oe.spdx_common
@@ -106,15 +99,13 @@ python do_collect_spdx_deps() {
 
     spdx_deps_file = Path(d.getVar("SPDXDEPS"))
 
-    deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx")
+    deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx")
 
     with spdx_deps_file.open("w") as f:
         json.dump(deps, f)
 }
-# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source
-addtask do_collect_spdx_deps after do_unpack
-do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}"
-do_collect_spdx_deps[deptask] = "do_create_spdx"
+addtask do_collect_spdx_deps
+do_collect_spdx_deps[deptask] = "do_create_recipe_spdx"
 do_collect_spdx_deps[dirs] = "${SPDXDIR}"
 
 oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA"
meta/lib/oe/spdx30_tasks.py +279 -123

559 lines not on this page · raw message

meta/lib/oeqa/selftest/cases/spdx.py +12 -7
index 5830d7c087..759ca86b73 100644
@@ -141,6 +141,11 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
     SPDX_CLASS = "create-spdx-3.0"
 
     def test_base_files(self):
+        self.check_recipe_spdx(
+            "base-files",
+            "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/static/static-base-files.spdx.json",
+            task="create_recipe_spdx",
+        )
         self.check_recipe_spdx(
             "base-files",
             "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json",
@@ -149,7 +154,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
     def test_gcc_include_source(self):
         objset = self.check_recipe_spdx(
             "gcc",
-            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-gcc.spdx.json",
+            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-gcc.spdx.json",
             extraconf="""\
                 SPDX_INCLUDE_SOURCES = "1"
                 """,
@@ -162,12 +167,12 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
             if software_file.name == filename:
                 found = True
                 self.logger.info(
-                    f"The spdxId of {filename} in recipe-gcc.spdx.json is {software_file.spdxId}"
+                    f"The spdxId of {filename} in build-gcc.spdx.json is {software_file.spdxId}"
                 )
                 break
 
         self.assertTrue(
-            found, f"Not found source file {filename} in recipe-gcc.spdx.json\n"
+            found, f"Not found source file {filename} in build-gcc.spdx.json\n"
         )
 
     def test_core_image_minimal(self):
@@ -305,7 +310,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
         # This will fail with NameError if new_annotation() is called incorrectly
         objset = self.check_recipe_spdx(
             "base-files",
-            "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/recipes/recipe-base-files.spdx.json",
+            "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/builds/build-base-files.spdx.json",
             extraconf=textwrap.dedent(
                 f"""\
                 ANNOTATION1 = "{ANNOTATION_VAR1}"
@@ -360,8 +365,8 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
 
     def test_kernel_config_spdx(self):
         kernel_recipe = get_bb_var("PREFERRED_PROVIDER_virtual/kernel")
-        spdx_file = f"recipe-{kernel_recipe}.spdx.json"
-        spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/recipes/{spdx_file}"
+        spdx_file = f"build-{kernel_recipe}.spdx.json"
+        spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/builds/{spdx_file}"
 
         # Make sure kernel is configured first
         bitbake(f"-c configure {kernel_recipe}")
@@ -392,7 +397,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
     def test_packageconfig_spdx(self):
         objset = self.check_recipe_spdx(
             "tar",
-            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-tar.spdx.json",
+            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-tar.spdx.json",
             extraconf="""\
                 SPDX_INCLUDE_PACKAGECONFIG = "1"
                 """,
Signature · 1 line
2.53.0
#50 lore raw

PATCH v5 4/13 spdx3: Add recipe SBoM task

In reply to Joshua Watt

Adds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example:

```
bitbake -c create_recipe_sbom zstd
```

Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively).

The complete SBoM for all (target) recipes can be built with:

```
bitbake -c create_recipe_sbom meta-world-recipe-sbom
```
---
 meta/classes/create-spdx-3.0.bbclass          | 32 +++++++++++++++++++
 meta/classes/spdx-common.bbclass              |  1 +
 meta/conf/distro/include/maintainers.inc      |  1 +
 meta/lib/oe/spdx30_tasks.py                   | 10 ++++++
 meta/lib/oeqa/selftest/cases/spdx.py          | 10 ++++++
 .../meta/meta-world-recipe-sbom.bb            | 29 +++++++++++++++++
 6 files changed, 83 insertions(+)
 create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
meta/classes/create-spdx-3.0.bbclass +32 -0
index 672ca27cd0..c3ea95b8bc 100644
@@ -142,6 +142,10 @@ SPDX_PACKAGE_URLS[doc] = "A space separated list of Package URLs (purls) for \
     Override this variable to replace the default, otherwise append or prepend \
     to add additional purls."
 
+SPDX_RECIPE_SBOM_NAME ?= "${PN}-recipe-sbom"
+SPDX_RECIPE_SBOM_NAME[doc] = "The name of output recipe SBoM when using \
+    create_recipe_sbom"
+
 IMAGE_CLASSES:append = " create-spdx-image-3.0"
 SDK_CLASSES += "create-spdx-sdk-3.0"
 
@@ -240,6 +244,34 @@ python do_create_package_spdx_setscene () {
 }
 addtask do_create_package_spdx_setscene
 
+addtask do_create_recipe_sbom after create_recipe_spdx
+python do_create_recipe_sbom() {
+    import oe.spdx30_tasks
+    from pathlib import Path
+    deploydir = Path(d.getVar("SPDXRECIPESBOMDEPLOY"))
+    oe.spdx30_tasks.create_recipe_sbom(d, deploydir)
+}
+
+SSTATETASKS += "do_create_recipe_sbom"
+do_create_recipe_sbom[recrdeptask] = "do_create_recipe_spdx"
+do_create_recipe_sbom[nostamp] = "1"
+do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}"
+do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
+do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}"
+do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}"
+do_create_recipe_sbom[vardeps] += "\
+    SPDX_INCLUDE_BITBAKE_PARENT_BUILD \
+    SPDX_PACKAGE_ADDITIONAL_PURPOSE \
+    SPDX_PROFILES \
+    SPDX_NAMESPACE_PREFIX \
+    SPDX_UUID_NAMESPACE \
+    "
+
+python do_create_recipe_sbom_setscene () {
+    sstate_setscene(d)
+}
+addtask do_create_recipe_sbom_setscene
+
 python spdx30_build_started_handler () {
     import oe.spdx30_tasks
     d = e.data.createCopy()
meta/classes/spdx-common.bbclass +1 -0
index 3c239a718b..abf2332bee 100644
@@ -25,6 +25,7 @@ SPDX_TOOL_VERSION ??= "1.0"
 
 SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy"
 SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy"
+SPDXRECIPESBOMDEPLOY = "${SPDXDIR}/recipes-bom-deploy"
 
 SPDX_INCLUDE_SOURCES ??= "0"
 SPDX_INCLUDE_COMPILED_SOURCES ??= "0"
meta/conf/distro/include/maintainers.inc +1 -0
index b5ab35d92a..5bea863798 100644
@@ -532,6 +532,7 @@ RECIPE_MAINTAINER:pn-meta-go-toolchain = "Richard Purdie <richard.purdie@linuxfo
 RECIPE_MAINTAINER:pn-meta-ide-support = "Richard Purdie <[email protected]>"
 RECIPE_MAINTAINER:pn-meta-toolchain = "Richard Purdie <[email protected]>"
 RECIPE_MAINTAINER:pn-meta-world-pkgdata = "Richard Purdie <[email protected]>"
+RECIPE_MAINTAINER:pn-meta-world-recipe-sbom = "Joshua Watt <[email protected]>"
 RECIPE_MAINTAINER:pn-mingetty = "Yi Zhao <[email protected]>"
 RECIPE_MAINTAINER:pn-mini-x-session = "Unassigned <[email protected]>"
 RECIPE_MAINTAINER:pn-minicom = "Unassigned <[email protected]>"
meta/lib/oe/spdx30_tasks.py +10 -0
index a8b4525e3d..b6c917045e 100644
@@ -1564,3 +1564,13 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname):
     oe.sbom30.write_jsonld_doc(
         d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json")
     )
+
+
+def create_recipe_sbom(d, deploydir):
+    sbom_name = d.getVar("SPDX_RECIPE_SBOM_NAME")
+
+    recipe, recipe_objset = load_recipe_spdx(d)
+
+    objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset])
+
+    oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json"))
meta/lib/oeqa/selftest/cases/spdx.py +10 -0
index 759ca86b73..efee0214fc 100644
@@ -151,6 +151,16 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
             "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json",
         )
 
+    def test_world_sbom(self):
+        objset = self.check_recipe_spdx(
+            "meta-world-recipe-sbom",
+            "{DEPLOY_DIR_IMAGE}/world-recipe-sbom.spdx.json",
+            task="create_recipe_sbom",
+        )
+
+        # Document should be fully linked
+        self.check_objset_missing_ids(objset)
+
     def test_gcc_include_source(self):
         objset = self.check_recipe_spdx(
             "gcc",
meta/recipes-core/meta/meta-world-recipe-sbom.bb new file +29 -0
new file mode 100644
index 0000000000..b47a3229c9
@@ -0,0 +1,29 @@
+SUMMARY = "Generates a combined SBoM for all world recipes"
+LICENSE = "MIT"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+PACKAGE_ARCH = "${MACHINE_ARCH}"
+
+inherit nopackages
+deltask do_fetch
+deltask do_unpack
+deltask do_patch
+deltask do_configure
+deltask do_compile
+deltask do_install
+
+do_prepare_recipe_sysroot[deptask] = ""
+
+WORLD_SBOM_EXCLUDE ?= ""
+
+EXCLUDE_FROM_WORLD = "1"
+SPDX_RECIPE_SBOM_NAME = "world-recipe-sbom"
+
+python calculate_extra_depends() {
+    exclude = set('${WORLD_SBOM_EXCLUDE}'.split())
+    exclude |= set(f"{v}-{self_pn}" for v in '${MULTILIB_VARIANTS}'.split())
+    exclude.add(self_pn)
+
+    deps.extend(p for p in world_target if p not in exclude)
+}
Signature · 1 line
2.53.0
#51 lore raw

PATCH v5 5/13 spdx3: Add is-native property

In reply to Joshua Watt

Adds a custom is-native property to the recipe package to indicate if it is a native recipe

---
 meta/lib/oe/sbom30.py       | 20 ++++++++++++++++++++
 meta/lib/oe/spdx30_tasks.py | 18 +++++++++++-------
 2 files changed, 31 insertions(+), 7 deletions(-)
meta/lib/oe/sbom30.py +20 -0
index 227ac51877..50a72fce39 100644
@@ -118,6 +118,26 @@ class OEDocumentExtension(oe.spdx30.extension_Extension):
         )
 
 
[email protected](OE_SPDX_BASE + "recipe-extension")
+class OERecipeExtension(oe.spdx30.extension_Extension):
+    """
+    This extension is added to recipe software_Packages to indicate various
+    useful bits of information about the recipe
+    """
+
+    CLOSED = True
+
+    @classmethod
+    def _register_props(cls):
+        super()._register_props()
+        cls._add_property(
+            "is_native",
+            oe.spdx30.BooleanProp(),
+            OE_SPDX_BASE + "is-native",
+            max_count=1,
+        )
+
+
 def spdxid_hash(*items):
     h = hashlib.md5()
     for i in items:
meta/lib/oe/spdx30_tasks.py +11 -7
index b6c917045e..a8fffbb085 100644
@@ -477,6 +477,10 @@ def set_purls(spdx_package, purls):
         )
 
 
+def get_is_native(d):
+    return bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d)
+
+
 def create_recipe_spdx(d):
     deploydir = Path(d.getVar("SPDXRECIPEDEPLOY"))
     deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
@@ -507,6 +511,11 @@ def create_recipe_spdx(d):
         )
     )
 
+    if get_is_native(d):
+        ext = oe.sbom30.OERecipeExtension()
+        ext.is_native = True
+        recipe.extension.append(ext)
+
     set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split())
 
     # TODO: This doesn't work before do_unpack because the license text has to
@@ -668,9 +677,7 @@ def create_spdx(d):
     spdx_workdir = Path(d.getVar("SPDXWORK"))
     include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1"
     pkg_arch = d.getVar("SSTATE_PKGARCH")
-    is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class(
-        "cross", d
-    )
+    is_native = get_is_native(d)
 
     recipe, recipe_objset = load_recipe_spdx(d)
 
@@ -1021,14 +1028,11 @@ def create_spdx(d):
 def create_package_spdx(d):
     deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
     deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY"))
-    is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class(
-        "cross", d
-    )
 
     providers = oe.spdx_common.collect_package_providers(d)
     pkg_arch = d.getVar("SSTATE_PKGARCH")
 
-    if is_native:
+    if get_is_native(d):
         return
 
     bb.build.exec_func("read_subpackage_metadata", d)
Signature · 1 line
2.53.0
#52 lore raw

PATCH v5 6/13 spdx30: Include patch file information in VEX

In reply to Joshua Watt

Modifies the SPDX VEX output to include the patches that fix a particular vulnerability. This is done by adding a patchedBy relationship from the VexFixedVulnAssessmentRelationship to the File that provides the fix.

If the file can be located without fetching (e.g. is a file:// in SRC_URI), the checksum will be included.

---
 meta/lib/oe/sbom30.py       | 60 ++++++++++++++-------------
 meta/lib/oe/spdx30_tasks.py | 81 ++++++++++++++++++++++++++++---------
 2 files changed, 92 insertions(+), 49 deletions(-)
meta/lib/oe/sbom30.py +31 -29
index 50a72fce39..21f084dc16 100644
@@ -620,37 +620,38 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
         )
         spdx_file.extension.append(OELicenseScannedExtension())
 
-    def new_file(self, _id, name, path, *, purposes=[]):
-        sha256_hash = bb.utils.sha256_file(path)
+    def new_file(self, _id, name, path, *, purposes=[], hashfile=True):
+        if hashfile:
+            sha256_hash = bb.utils.sha256_file(path)
 
-        for f in self.by_sha256_hash.get(sha256_hash, []):
-            if not isinstance(f, oe.spdx30.software_File):
-                continue
+            for f in self.by_sha256_hash.get(sha256_hash, []):
+                if not isinstance(f, oe.spdx30.software_File):
+                    continue
 
-            if purposes:
-                new_primary = purposes[0]
-                new_additional = []
+                if purposes:
+                    new_primary = purposes[0]
+                    new_additional = []
 
-                if f.software_primaryPurpose:
-                    new_additional.append(f.software_primaryPurpose)
-                new_additional.extend(f.software_additionalPurpose)
+                    if f.software_primaryPurpose:
+                        new_additional.append(f.software_primaryPurpose)
+                    new_additional.extend(f.software_additionalPurpose)
 
-                new_additional = sorted(
-                    list(set(p for p in new_additional if p != new_primary))
-                )
+                    new_additional = sorted(
+                        list(set(p for p in new_additional if p != new_primary))
+                    )
 
-                f.software_primaryPurpose = new_primary
-                f.software_additionalPurpose = new_additional
+                    f.software_primaryPurpose = new_primary
+                    f.software_additionalPurpose = new_additional
 
-            if f.name != name:
-                for e in f.extension:
-                    if isinstance(e, OEFileNameAliasExtension):
-                        e.aliases.append(name)
-                        break
-                else:
-                    f.extension.append(OEFileNameAliasExtension(aliases=[name]))
+                if f.name != name:
+                    for e in f.extension:
+                        if isinstance(e, OEFileNameAliasExtension):
+                            e.aliases.append(name)
+                            break
+                    else:
+                        f.extension.append(OEFileNameAliasExtension(aliases=[name]))
 
-            return f
+                return f
 
         spdx_file = oe.spdx30.software_File(
             _id=_id,
@@ -661,12 +662,13 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
             spdx_file.software_primaryPurpose = purposes[0]
             spdx_file.software_additionalPurpose = purposes[1:]
 
-        spdx_file.verifiedUsing.append(
-            oe.spdx30.Hash(
-                algorithm=oe.spdx30.HashAlgorithm.sha256,
-                hashValue=sha256_hash,
+        if hashfile:
+            spdx_file.verifiedUsing.append(
+                oe.spdx30.Hash(
+                    algorithm=oe.spdx30.HashAlgorithm.sha256,
+                    hashValue=sha256_hash,
+                )
             )
-        )
 
         return self.add(spdx_file)
 
meta/lib/oe/spdx30_tasks.py +61 -20

139 lines not on this page · raw message

Signature · 1 line
2.53.0
#53 lore raw

PATCH v5 7/13 spdx: De-duplicate CreationInfo

In reply to Joshua Watt

De-duplicates CreationInfo objects that are identical (except for ID) when writing out an SBoM. This significantly reduces the number of CreationInfo objects that end up in the final document.

---
 meta/lib/oe/sbom30.py | 112 ++++++++++++++++++++++++++++++------------
 meta/lib/oe/spdx30.py |   2 +-
 2 files changed, 81 insertions(+), 33 deletions(-)
meta/lib/oe/sbom30.py +80 -32
index 21f084dc16..55a2863d2d 100644
@@ -14,6 +14,7 @@ import uuid
 import os
 import oe.spdx_common
 from datetime import datetime, timezone
+from contextlib import contextmanager
 
 OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/"
 
@@ -191,6 +192,25 @@ def to_list(l):
     return l
 
 
+class Dedup(object):
+    def __init__(self, objset):
+        self.unique = set()
+        self.dedup = {}
+        self.objset = objset
+
+    def find_duplicates(self, cmp, typ, **kwargs):
+        for o in self.objset.foreach_filter(typ, **kwargs):
+            for u in self.unique:
+                if cmp(u, o):
+                    self.dedup[o] = u
+                    break
+            else:
+                self.unique.add(o)
+
+    def get(self, o):
+        return self.dedup.get(o, o)
+
+
 class ObjectSet(oe.spdx30.SHACLObjectSet):
     def __init__(self, d):
         super().__init__()
@@ -895,6 +915,45 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
         self.missing_ids -= set(imports.keys())
         return self.missing_ids
 
+    @contextmanager
+    def deduplicate(self):
+        d = Dedup(self)
+
+        yield d
+
+        visited = set()
+
+        def visit(o, path):
+            if isinstance(o, oe.spdx30.SHACLObject):
+                if o in visited:
+                    return False
+                visited.add(o)
+
+                for k in o:
+                    v = o[k]
+                    if isinstance(v, oe.spdx30.SHACLObject):
+                        o[k] = d.get(v)
+
+            elif isinstance(o, oe.spdx30.ListProxy):
+                for idx, v in enumerate(o):
+                    if isinstance(v, oe.spdx30.SHACLObject):
+                        o[idx] = d.get(v)
+
+            return True
+
+        if d.dedup:
+            for o in self.objects:
+                o.walk(visit)
+
+            for k, v in d.dedup.items():
+                bb.debug(
+                    1,
+                    f"Removing duplicate {k.__class__.__name__} {k._id or id(k)} -> {v._id or id(v)}",
+                )
+                self.objects.discard(k)
+
+            self.create_index()
+
 
 def load_jsonld(d, path, required=False):
     deserializer = oe.spdx30.JSONLDDeserializer()
@@ -1080,39 +1139,28 @@ def create_sbom(d, name, root_elements, add_objectsets=[]):
     # SBoM should be the only root element of the document
     objset.doc.rootElement = [sbom]
 
-    # De-duplicate licenses
-    unique = set()
-    dedup = {}
-    for lic in objset.foreach_type(oe.spdx30.simplelicensing_LicenseExpression):
-        for u in unique:
-            if (
-                u.simplelicensing_licenseExpression
-                == lic.simplelicensing_licenseExpression
-                and u.simplelicensing_licenseListVersion
-                == lic.simplelicensing_licenseListVersion
-            ):
-                dedup[lic] = u
-                break
-        else:
-            unique.add(lic)
-
-    if dedup:
-        for rel in objset.foreach_filter(
-            oe.spdx30.Relationship,
-            relationshipType=oe.spdx30.RelationshipType.hasDeclaredLicense,
-        ):
-            rel.to = [dedup.get(to, to) for to in rel.to]
-
-        for rel in objset.foreach_filter(
-            oe.spdx30.Relationship,
-            relationshipType=oe.spdx30.RelationshipType.hasConcludedLicense,
-        ):
-            rel.to = [dedup.get(to, to) for to in rel.to]
+    def cmp_license_expression(a, b):
+        return (
+            a.simplelicensing_licenseExpression == b.simplelicensing_licenseExpression
+            and a.simplelicensing_licenseListVersion
+            == b.simplelicensing_licenseListVersion
+        )
 
-        for k, v in dedup.items():
-            bb.debug(1, f"Removing duplicate License {k._id} -> {v._id}")
-            objset.objects.remove(k)
+    def cmp_creation_info(a, b):
+        data_a = {k: a[k] for k in a}
+        data_b = {k: b[k] for k in b}
+        data_a["@id"] = ""
+        data_b["@id"] = ""
+        return data_a == data_b
+
+    with objset.deduplicate() as dedup:
+        # De-duplicate licenses
+        dedup.find_duplicates(
+            cmp_license_expression,
+            oe.spdx30.simplelicensing_LicenseExpression,
+        )
 
-        objset.create_index()
+        # Deduplicate creation info
+        dedup.find_duplicates(cmp_creation_info, oe.spdx30.CreationInfo)
 
     return objset, sbom
meta/lib/oe/spdx30.py +1 -1
index cd97eebd18..1f58402ffc 100644
@@ -701,7 +701,7 @@ class SHACLObject(object):
         self.__dict__["_obj_data"][iri] = prop.init()
 
     def __iter__(self):
-        return self._OBJ_PROPERTIES.keys()
+        return iter(self._OBJ_PROPERTIES.keys())
 
     def walk(self, callback, path=None):
         """
Signature · 1 line
2.53.0
#54 lore raw

PATCH v5 8/13 spdx_common: Check for dependent task in task flags

In reply to Joshua Watt

Checks that the task being used to detect dependencies is present in at least one dependency task flag of the current task. This helps prevent errors where the wrong task is specified and never found.

---
 meta/lib/oe/spdx_common.py | 11 +++++++++++
 1 file changed, 11 insertions(+)
meta/lib/oe/spdx_common.py +11 -0
index 72c24180d5..3aaf2a9c8b 100644
@@ -96,6 +96,17 @@ def collect_direct_deps(d, dep_task):
 
     taskdepdata = d.getVar("BB_TASKDEPDATA", False)
 
+    # Check that the task is listed one of the task dependency flags of the
+    # current task
+    depflags = (
+        set((d.getVarFlag(current_task, "deptask") or "").split())
+        | set((d.getVarFlag(current_task, "rdeptask") or "").split())
+        | set((d.getVarFlag(current_task, "recrdeptask") or "").split())
+    )
+
+    if not dep_task in depflags:
+        bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}")
+
     for this_dep in taskdepdata.values():
         if this_dep[0] == pn and this_dep[1] == current_task:
             break
Signature · 1 line
2.53.0
#55 lore raw

PATCH v5 9/13 spdx30: Skip install package CVE information

In reply to Joshua Watt

Skips adding the install package CVE information by default. This information grows exponentially, since it ends up be N_CVES * N_PACKAGES. The CVE information for a given installed package can be determined by following the "generates" link between the install package and the recipe and looking at the CVE information for the recipe, meaning that the CVE information is only included once in the SPDX document.

If users still need the legacy method of including CVE information for each package, then then can set SPDX_PACKAGE_INCLUDE_VEX = "1"

---
 meta/classes/create-spdx-3.0.bbclass | 11 ++++++++
 meta/lib/oe/spdx30_tasks.py          | 39 ++++++++++++++--------------
 meta/lib/oeqa/selftest/cases/spdx.py | 12 +++++++++
 3 files changed, 43 insertions(+), 19 deletions(-)
meta/classes/create-spdx-3.0.bbclass +11 -0
index c3ea95b8bc..88b7ef9f42 100644
@@ -45,6 +45,17 @@ SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to
     including those already fixed upstream (warning: This can be large and \
     slow)."
 
+SPDX_PACKAGE_INCLUDE_VEX ?= "0"
+SPDX_PACKAGE_INCLUDE_VEX[doc] = "Link VEX information to the binary package outputs. \
+    Normally, VEX information is only linked to the common recipe that `generates` the \
+    binary packages, but setting this to '1' will cause it to also be linked into the \
+    generated binary packages. This is off by default because linking the VEX data to \
+    each package causes the SPDX output to grow very large, and the same information \
+    can be determined by following the `generates` relationship back to the recipe. \
+    Before recipe packages were introduced, this was the only way VEX data was \
+    expressed; you may need to enable this if your downstream tools do not \
+    understand how to trace back to the recipe to find VEX information."
+
 SPDX_INCLUDE_TIMESTAMPS ?= "0"
 SPDX_INCLUDE_TIMESTAMPS[doc] = "Include time stamps in SPDX output. This is \
     useful if you want to know when artifacts were produced and when builds \
meta/lib/oe/spdx30_tasks.py +20 -19
index aec47d4f81..887fac813a 100644
@@ -771,27 +771,28 @@ def create_spdx(d):
     # Collect all VEX statements from the recipe
     vex_statements = {}
     vex_patches = {}
-    for rel in recipe_objset.foreach_filter(
-        oe.spdx30.Relationship,
-        relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability,
-    ):
-        for cve in rel.to:
-            vex_statements[cve] = []
-            vex_patches[cve] = []
-
-    for cve in vex_statements.keys():
+    if (d.getVar("SPDX_PACKAGE_INCLUDE_VEX") or "") == "1":
         for rel in recipe_objset.foreach_filter(
-            oe.spdx30.security_VexVulnAssessmentRelationship,
-            from_=cve,
+            oe.spdx30.Relationship,
+            relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability,
         ):
-            vex_statements[cve].append(rel)
-            if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn:
-                for patch_rel in recipe_objset.foreach_filter(
-                    oe.spdx30.Relationship,
-                    relationshipType=oe.spdx30.RelationshipType.patchedBy,
-                    from_=rel,
-                ):
-                    vex_patches[cve].extend(patch_rel.to)
+            for cve in rel.to:
+                vex_statements[cve] = []
+                vex_patches[cve] = []
+
+        for cve in vex_statements.keys():
+            for rel in recipe_objset.foreach_filter(
+                oe.spdx30.security_VexVulnAssessmentRelationship,
+                from_=cve,
+            ):
+                vex_statements[cve].append(rel)
+                if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn:
+                    for patch_rel in recipe_objset.foreach_filter(
+                        oe.spdx30.Relationship,
+                        relationshipType=oe.spdx30.RelationshipType.patchedBy,
+                        from_=rel,
+                    ):
+                        vex_patches[cve].extend(patch_rel.to)
 
     # Write out the package SPDX data now. It is not complete as we cannot
     # write the runtime data, so write it to a staging area and a later task
meta/lib/oeqa/selftest/cases/spdx.py +12 -0
index efee0214fc..f1ea2694cf 100644
@@ -429,3 +429,15 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
                 value, ["enabled", "disabled"],
                 f"Unexpected PACKAGECONFIG value '{value}' for {key}"
             )
+
+    def test_package_vex(self):
+        objset = self.check_recipe_spdx(
+            "core-image-minimal",
+            "{DEPLOY_DIR_IMAGE}/core-image-minimal-{MACHINE}.rootfs.spdx.json",
+            extraconf="""\
+                SPDX_PACKAGE_INCLUDE_VEX = "1"
+                """,
+        )
+
+        # Document should be fully linked
+        self.check_objset_missing_ids(objset)
Signature · 1 line
2.53.0
#56 lore raw

PATCH v5 10/13 dummy-sdk-package: Disable SPDX

In reply to Joshua Watt

The dummy SDK packages do not need SPDX support, and since they play some games with allarch that cause problems, it's simplest to disable their SPDX output.

---
 meta/recipes-core/meta/dummy-sdk-package.inc | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-core/meta/dummy-sdk-package.inc +1 -0
index bf453cac9b..71e788b0b9 100644
@@ -4,6 +4,7 @@ LICENSE = "MIT"
 PACKAGE_ARCH = "all"
 
 inherit allarch
+inherit nospdx
 
 INHIBIT_DEFAULT_DEPS = "1"
 
Signature · 1 line
2.53.0
#57 lore raw

PATCH v5 11/13 spdx: Remove fatal errors for missing providers

In reply to Joshua Watt

When creating images and SDKs, do not error on missing providers. This allows recipes to use the nospdx inherit to prevent SPDX from being generated, but not result in an error when assembling the final image.

Note that runtime packages generation already ignored missing providers, so this is changing image and SDK generation to match

---
 meta/classes/create-spdx-2.2.bbclass | 3 ++-
 meta/lib/oe/spdx30_tasks.py          | 8 +-------
 2 files changed, 3 insertions(+), 8 deletions(-)
meta/classes/create-spdx-2.2.bbclass +2 -1
index 3288cdf75a..aa39208eae 100644
@@ -858,7 +858,8 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx
     if packages:
         for name in sorted(packages.keys()):
             if name not in providers:
-                bb.fatal("Unable to find SPDX provider for '%s'" % name)
+                bb.note("Unable to find SPDX provider for '%s'" % name)
+                continue
 
             pkg_name, pkg_hashfn = providers[name]
 
meta/lib/oe/spdx30_tasks.py +1 -7
index 887fac813a..ba15d74278 100644
@@ -1251,11 +1251,10 @@ def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None)
     providers = oe.spdx_common.collect_package_providers(d)
 
     build_deps = set()
-    missing_providers = set()
 
     for name in sorted(packages.keys()):
         if name not in providers:
-            missing_providers.add(name)
+            bb.note(f"Unable to find SPDX provider for '{name}'")
             continue
 
         pkg_name, pkg_hashfn = providers[name]
@@ -1274,11 +1273,6 @@ def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None)
             for h, f in pkg_objset.by_sha256_hash.items():
                 files_by_hash.setdefault(h, set()).update(f)
 
-    if missing_providers:
-        bb.fatal(
-            f"Unable to find SPDX provider(s) for: {', '.join(sorted(missing_providers))}"
-        )
-
     if build_deps:
         objset.new_scoped_relationship(
             [build],
Signature · 1 line
2.53.0
#58 lore raw

PATCH v5 12/13 spdx3: Use common variable for vardeps

In reply to Joshua Watt

Instead of repeating the vardeps for each SPDX task with the necessary variables, use a common variable to make it easier to manage

---
 meta/classes/create-spdx-3.0.bbclass | 33 ++++++++++------------------
 1 file changed, 12 insertions(+), 21 deletions(-)
meta/classes/create-spdx-3.0.bbclass +12 -21
index 88b7ef9f42..6df66c193b 100644
@@ -174,6 +174,14 @@ SPDX3_DEP_FILES = "\
     ${SPDX_LICENSES}:True \
     "
 
+SPDX3_VAR_DEPS = "\
+    SPDX_INCLUDE_BITBAKE_PARENT_BUILD \
+    SPDX_PACKAGE_ADDITIONAL_PURPOSE \
+    SPDX_PROFILES \
+    SPDX_NAMESPACE_PREFIX \
+    SPDX_UUID_NAMESPACE \
+    "
+
 python do_create_recipe_spdx() {
     import oe.spdx30_tasks
     oe.spdx30_tasks.create_recipe_spdx(d)
@@ -185,13 +193,7 @@ do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}"
 do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}"
 do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}"
-do_create_recipe_spdx[vardeps] += "\
-    SPDX_INCLUDE_BITBAKE_PARENT_BUILD \
-    SPDX_PACKAGE_ADDITIONAL_PURPOSE \
-    SPDX_PROFILES \
-    SPDX_NAMESPACE_PREFIX \
-    SPDX_UUID_NAMESPACE \
-    "
+do_create_recipe_spdx[vardeps] += "${SPDX3_VAR_DEPS}"
 
 python do_create_recipe_spdx_setscene () {
     sstate_setscene(d)
@@ -222,13 +224,7 @@ do_create_spdx[depends] += " \
     ${PATCHDEPENDENCY} \
     ${@create_spdx_source_deps(d)} \
 "
-do_create_spdx[vardeps] += "\
-    SPDX_INCLUDE_BITBAKE_PARENT_BUILD \
-    SPDX_PACKAGE_ADDITIONAL_PURPOSE \
-    SPDX_PROFILES \
-    SPDX_NAMESPACE_PREFIX \
-    SPDX_UUID_NAMESPACE \
-    "
+do_create_spdx[vardeps] += "${SPDX3_VAR_DEPS}"
 
 python do_create_spdx_setscene () {
     sstate_setscene(d)
@@ -249,6 +245,7 @@ do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}"
 do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}"
 do_create_package_spdx[rdeptask] = "do_create_spdx"
+do_create_package_spdx[vardeps] += "${SPDX3_VAR_DEPS}"
 
 python do_create_package_spdx_setscene () {
     sstate_setscene(d)
@@ -270,13 +267,7 @@ do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}"
 do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
 do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}"
 do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}"
-do_create_recipe_sbom[vardeps] += "\
-    SPDX_INCLUDE_BITBAKE_PARENT_BUILD \
-    SPDX_PACKAGE_ADDITIONAL_PURPOSE \
-    SPDX_PROFILES \
-    SPDX_NAMESPACE_PREFIX \
-    SPDX_UUID_NAMESPACE \
-    "
+do_create_recipe_sbom[vardeps] += "${SPDX3_VAR_DEPS}"
 
 python do_create_recipe_sbom_setscene () {
     sstate_setscene(d)
Signature · 1 line
2.53.0
#59 lore raw

PATCH v5 13/13 glibc-testsuite: Do not generate SPDX

In reply to Joshua Watt

glibc-testsuite does not run on target or factor into the build supply chain, since its purpose is run tests in Qemu at build time

---
 meta/recipes-core/glibc/glibc-testsuite_2.42.bb | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-core/glibc/glibc-testsuite_2.42.bb +1 -0
index 6477612feb..1a83a09802 100644
@@ -32,6 +32,7 @@ do_check:append () {
 }
 
 inherit nopackages
+inherit nospdx
 deltask do_stash_locale
 deltask do_install
 deltask do_populate_sysroot
Signature · 1 line
2.53.0
#60 lore raw
In reply to Joshua Watt
Quoted 56 lines · Joshua Watt via lists.openembedded.org

Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.

Special thanks to Benjamin Robin <[email protected]> for helping work through this.

V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).

V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run).

V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.

V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway

Joshua Watt (13):
llvm-project-source: Use allarch.bbclass
gcc-source: Use allarch.bbclass
spdx3: Add recipe SPDX data
spdx3: Add recipe SBoM task
spdx3: Add is-native property
spdx30: Include patch file information in VEX
spdx: De-duplicate CreationInfo
spdx_common: Check for dependent task in task flags
spdx30: Skip install package CVE information
dummy-sdk-package: Disable SPDX
spdx: Remove fatal errors for missing providers
spdx3: Use common variable for vardeps
glibc-testsuite: Do not generate SPDX

 meta/classes-global/sstate.bbclass            |   4 +-
 .../create-spdx-image-3.0.bbclass             |   4 +-
 .../create-spdx-sdk-3.0.bbclass               |   4 +-
 meta/classes-recipe/kernel.bbclass            |   2 +-
 meta/classes-recipe/nospdx.bbclass            |   1 +
 meta/classes/create-spdx-2.2.bbclass          |  15 +-
 meta/classes/create-spdx-3.0.bbclass          |  87 ++-
 meta/classes/spdx-common.bbclass              |  22 +-
 meta/conf/distro/include/maintainers.inc      |   1 +
 meta/lib/oe/sbom30.py                         | 192 ++++---
 meta/lib/oe/spdx30.py                         |   2 +-
 meta/lib/oe/spdx30_tasks.py                   | 496 +++++++++++++-----
 meta/lib/oe/spdx_common.py                    |  11 +
Quoted 8 lines
 meta/lib/oeqa/selftest/cases/spdx.py          |  41 +-
 .../glibc/glibc-testsuite_2.42.bb             |   1 +
 meta/recipes-core/meta/dummy-sdk-package.inc  |   1 +
 .../meta/meta-world-recipe-sbom.bb            |  29 +
 .../clang/llvm-project-source.inc             |   8 +-
 meta/recipes-devtools/gcc/gcc-source.inc      |  16 +-
 19 files changed, 667 insertions(+), 270 deletions(-)
 create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb

Ok, we are almost there! We only have a selftest failure now:

2026-03-05 16:33:10,060 - oe-selftest - INFO - sysroot.SysrootTests.test_sysroot_cleanup (subunit.RemotedTestCase)
2026-03-05 16:33:10,061 - oe-selftest - INFO -  ... FAIL

...
ERROR: sysroot-test-1.0-r0 do_create_spdx: Could not find a builds SPDX document named build-sysroot-test-arch1

https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3457
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3338
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3227

Thanks,
Mathieu

Signature · 3 lines
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#61 lore raw

v6 cover Add SPDX 3 Recipe Information

In reply to Joshua Watt

Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.

Special thanks to Benjamin Robin <[email protected]> for helping work through this.

V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).

V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run).

V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.

V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway

V6: Fixes a bug where SPDX task would not correctly re-run when they change, which would cause errors about missing SPDX document. Also updates to the latest version of the SPDX bindings which improves performance

Joshua Watt (15):
llvm-project-source: Use allarch.bbclass
gcc-source: Use allarch.bbclass
spdx3: Add recipe SPDX data
spdx3: Add recipe SBoM task
spdx3: Add is-native property
spdx30: Include patch file information in VEX
spdx: De-duplicate CreationInfo
spdx_common: Check for dependent task in task flags
spdx30: Skip install package CVE information
dummy-sdk-package: Disable SPDX
spdx: Remove fatal errors for missing providers
spdx3: Use common variable for vardeps
glibc-testsuite: Do not generate SPDX
spdx: Remove do_collect_spdx_deps task
spdx: Update to latest bindings

 meta/classes-global/sstate.bbclass            |    4 +-
 .../create-spdx-image-3.0.bbclass             |    4 +-
 .../create-spdx-sdk-3.0.bbclass               |    4 +-
 meta/classes-recipe/kernel.bbclass            |    2 +-
 meta/classes-recipe/nospdx.bbclass            |    2 +-
 meta/classes/create-spdx-2.2.bbclass          |   33 +-
 meta/classes/create-spdx-3.0.bbclass          |   92 +-
 meta/classes/spdx-common.bbclass              |   34 +-
 meta/conf/distro/include/maintainers.inc      |    1 +
 meta/lib/oe/sbom30.py                         |  239 +-
 meta/lib/oe/spdx30/__init__.py                |    8 +
 meta/lib/oe/spdx30/__main__.py                |   12 +
 meta/lib/oe/spdx30/cmd.py                     |   75 +
 meta/lib/oe/{spdx30.py => spdx30/model.py}    | 5935 ++++++++++-------
 meta/lib/oe/spdx30/stub.pyi                   | 2544 +++++++
 meta/lib/oe/spdx30_tasks.py                   |  512 +-
 meta/lib/oe/spdx_common.py                    |   78 +-
 meta/lib/oeqa/selftest/cases/spdx.py          |   41 +-
 .../glibc/glibc-testsuite_2.42.bb             |    1 +
 meta/recipes-core/meta/dummy-sdk-package.inc  |    1 +
 .../meta/meta-world-recipe-sbom.bb            |   29 +
 .../clang/llvm-project-source.inc             |    8 +-
 meta/recipes-devtools/gcc/gcc-source.inc      |   16 +-
 scripts/contrib/make-spdx-bindings.sh         |    3 +-
 24 files changed, 6922 insertions(+), 2756 deletions(-)
 create mode 100644 meta/lib/oe/spdx30/__init__.py
 create mode 100644 meta/lib/oe/spdx30/__main__.py
 create mode 100644 meta/lib/oe/spdx30/cmd.py
 rename meta/lib/oe/{spdx30.py => spdx30/model.py} (52%)
 create mode 100644 meta/lib/oe/spdx30/stub.pyi
 create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
Signature · 1 line
2.53.0
#62 lore raw

PATCH v6 1/15 llvm-project-source: Use allarch.bbclass

Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)

---
 meta/recipes-devtools/clang/llvm-project-source.inc | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)
meta/recipes-devtools/clang/llvm-project-source.inc +1 -7
index 13e54efbc2..6bb595b7bc 100644
@@ -5,7 +5,7 @@ deltask do_populate_sysroot
 deltask do_populate_lic
 RM_WORK_EXCLUDE += "${PN}"
 
-inherit nopackages
+inherit nopackages allarch
 
 PN = "llvm-project-source-${PV}"
 WORKDIR = "${TMPDIR}/work-shared/llvm-project-source-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:llvm-project-source::${PV}:${PR}::${SSTATE_VERSION}:"
 STAMP = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-${PR}"
 STAMPCLEAN = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-*"
 
-INHIBIT_DEFAULT_DEPS = "1"
 DEPENDS = ""
 PACKAGES = ""
-TARGET_ARCH = "allarch"
-TARGET_AS_ARCH = "none"
-TARGET_CC_ARCH = "none"
-TARGET_LD_ARCH = "none"
-TARGET_OS = "linux"
 baselib = "lib"
 PACKAGE_ARCH = "all"
 
Signature · 1 line
2.53.0
#63 lore raw

PATCH v6 2/15 gcc-source: Use allarch.bbclass

In reply to Joshua Watt

Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)

---
 meta/recipes-devtools/gcc/gcc-source.inc | 16 +++++-----------
 1 file changed, 5 insertions(+), 11 deletions(-)
meta/recipes-devtools/gcc/gcc-source.inc +5 -11
index 265bcf4bef..3ac679b1a6 100644
@@ -1,11 +1,11 @@
-deltask do_configure 
-deltask do_compile 
-deltask do_install 
+deltask do_configure
+deltask do_compile
+deltask do_install
 deltask do_populate_sysroot
-deltask do_populate_lic 
+deltask do_populate_lic
 RM_WORK_EXCLUDE += "${PN}"
 
-inherit nopackages
+inherit nopackages allarch
 
 PN = "gcc-source-${PV}"
 WORKDIR = "${TMPDIR}/work-shared/gcc-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:gcc::${PV}:${PR}::${SSTATE_VERSION}:"
 STAMP = "${STAMPS_DIR}/work-shared/gcc-${PV}-${PR}"
 STAMPCLEAN = "${STAMPS_DIR}/work-shared/gcc-${PV}-*"
 
-INHIBIT_DEFAULT_DEPS = "1"
 DEPENDS = ""
 PACKAGES = ""
-TARGET_ARCH = "allarch"
-TARGET_AS_ARCH = "none"
-TARGET_CC_ARCH = "none"
-TARGET_LD_ARCH = "none"
-TARGET_OS = "linux"
 baselib = "lib"
 PACKAGE_ARCH = "all"
 
Signature · 1 line
2.53.0
#64 lore raw

PATCH v6 3/15 spdx3: Add recipe SPDX data

In reply to Joshua Watt

Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.

Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch

---
 meta/classes-global/sstate.bbclass            |   4 +-
 .../create-spdx-image-3.0.bbclass             |   4 +-
 .../create-spdx-sdk-3.0.bbclass               |   4 +-
 meta/classes-recipe/kernel.bbclass            |   2 +-
 meta/classes-recipe/nospdx.bbclass            |   1 +
 meta/classes/create-spdx-2.2.bbclass          |  12 +-
 meta/classes/create-spdx-3.0.bbclass          |  51 ++-
 meta/classes/spdx-common.bbclass              |  21 +-
 meta/lib/oe/spdx30_tasks.py                   | 402 ++++++++++++------
 meta/lib/oeqa/selftest/cases/spdx.py          |  19 +-
 10 files changed, 354 insertions(+), 166 deletions(-)
meta/classes-global/sstate.bbclass +2 -2
index 2fd29d7323..95c44f404e 100644
@@ -954,7 +954,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,
             extrapath = d.getVar("NATIVELSBSTRING") + "/"
         else:
             extrapath = ""
-        
+
         tname = bb.runqueue.taskname_from_tid(task)[3:]
 
         if tname in ["fetch", "unpack", "patch", "populate_lic", "preconfigure"] and splithashfn[2]:
@@ -1116,7 +1116,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None):
 
     logit("Considering setscene task: %s" % (str(taskdependees[task])), log)
 
-    directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_deploy_archives"]
+    directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_create_recipe_spdx", "do_deploy_archives"]
 
     def isNativeCross(x):
         return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross")
meta/classes-recipe/create-spdx-image-3.0.bbclass +2 -2
index 636ab14eb0..15a91e90e2 100644
@@ -34,7 +34,7 @@ addtask do_create_rootfs_spdx after do_rootfs before do_image
 SSTATETASKS += "do_create_rootfs_spdx"
 do_create_rootfs_spdx[sstate-inputdirs] = "${SPDXROOTFSDEPLOY}"
 do_create_rootfs_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}"
-do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_create_rootfs_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_create_rootfs_spdx[cleandirs] += "${SPDXROOTFSDEPLOY}"
 do_create_rootfs_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 
@@ -76,7 +76,7 @@ do_create_image_sbom_spdx[sstate-inputdirs] = "${SPDXIMAGEDEPLOYDIR}"
 do_create_image_sbom_spdx[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
 do_create_image_sbom_spdx[stamp-extra-info] = "${MACHINE_ARCH}"
 do_create_image_sbom_spdx[cleandirs] = "${SPDXIMAGEDEPLOYDIR}"
-do_create_image_sbom_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_create_image_sbom_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_create_image_sbom_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 
 python do_create_image_sbom_spdx_setscene() {
meta/classes-recipe/create-spdx-sdk-3.0.bbclass +2 -2
index e5f220cdfa..a4b8ed3bf9 100644
@@ -5,14 +5,14 @@
 #
 # SPDX SDK tasks
 
-do_populate_sdk[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_populate_sdk[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_populate_sdk[cleandirs] += "${SPDXSDKWORK}"
 do_populate_sdk[postfuncs] += "sdk_create_sbom"
 do_populate_sdk[file-checksums] += "${SPDX3_DEP_FILES}"
 POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_create_spdx"
 POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_create_spdx"
 
-do_populate_sdk_ext[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_populate_sdk_ext[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_populate_sdk_ext[cleandirs] += "${SPDXSDKEXTWORK}"
 do_populate_sdk_ext[postfuncs] += "sdk_ext_create_sbom"
 do_populate_sdk_ext[file-checksums] += "${SPDX3_DEP_FILES}"
meta/classes-recipe/kernel.bbclass +1 -1
index d61cc82a4e..8fd4e1141a 100644
@@ -882,7 +882,7 @@ do_create_spdx:append() {
         except Exception as e:
             bb.error(f"Failed to parse kernel config file: {e}")
 
-        path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "recipes", f"recipe-{pn}", deploydir=deploydir)
+        path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "builds", f"build-{pn}", deploydir=deploydir)
         build_objset = oe.sbom30.load_jsonld(d, path, required=True)
         build = build_objset.find_root(oe.spdx30.build_Build)
         if not build:
meta/classes-recipe/nospdx.bbclass +1 -0
index b20e28218b..90e14442ba 100644
@@ -5,6 +5,7 @@
 #
 
 deltask do_collect_spdx_deps
+deltask do_create_recipe_spdx
 deltask do_create_spdx
 deltask do_create_spdx_runtime
 deltask do_create_package_spdx
meta/classes/create-spdx-2.2.bbclass +11 -1
index 65d10d86db..3288cdf75a 100644
@@ -399,6 +399,15 @@ def get_license_list_version(license_data, d):
     return ".".join(license_data["licenseListVersion"].split(".")[:2])
 
 
+# This task is added for compatibility with tasks shared with SPDX 3, but
+# doesn't do anything
+do_create_recipe_spdx() {
+    :
+}
+do_create_recipe_spdx[noexec] = "1"
+addtask do_create_recipe_spdx after do_collect_spdx_deps
+
+
 python do_create_spdx() {
     from datetime import datetime, timezone
     import oe.sbom
@@ -594,7 +603,7 @@ python do_create_spdx() {
 }
 do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS"
 # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source
-addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work
+addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch do_collect_spdx_deps before do_populate_sdk do_build do_rm_work
 
 SSTATETASKS += "do_create_spdx"
 do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}"
@@ -605,6 +614,7 @@ python do_create_spdx_setscene () {
 }
 addtask do_create_spdx_setscene
 
+do_create_spdx[deptask] += "do_create_spdx"
 do_create_spdx[dirs] = "${SPDXWORK}"
 do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}"
 do_create_spdx[depends] += " \
meta/classes/create-spdx-3.0.bbclass +38 -13

90 lines not on this page · raw message

meta/classes/spdx-common.bbclass +6 -15
index 3110230c9e..3c239a718b 100644
@@ -23,6 +23,7 @@ SPDXDEPS = "${SPDXDIR}/deps.json"
 SPDX_TOOL_NAME ??= "oe-spdx-creator"
 SPDX_TOOL_VERSION ??= "1.0"
 
+SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy"
 SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy"
 
 SPDX_INCLUDE_SOURCES ??= "0"
@@ -67,12 +68,6 @@ def create_spdx_source_deps(d):
     deps = []
     if d.getVar("SPDX_INCLUDE_SOURCES") == "1":
         pn = d.getVar('PN')
-        # do_unpack is a hack for now; we only need it to get the
-        # dependencies do_unpack already has so we can extract the source
-        # ourselves
-        if oe.spdx_common.has_task(d, "do_unpack"):
-            deps.append("%s:do_unpack" % pn)
-
         if oe.spdx_common.is_work_shared_spdx(d) and \
            oe.spdx_common.process_sources(d):
             # For kernel source code
@@ -84,8 +79,6 @@ def create_spdx_source_deps(d):
             # For gcc-source-${PV} source code
             if oe.spdx_common.has_task(d, "do_preconfigure"):
                 deps.append("%s:do_preconfigure" % pn)
-            elif oe.spdx_common.has_task(d, "do_patch"):
-                deps.append("%s:do_patch" % pn)
             # For gcc-cross-x86_64 source code
             elif oe.spdx_common.has_task(d, "do_configure"):
                 deps.append("%s:do_configure" % pn)
@@ -97,8 +90,8 @@ python do_collect_spdx_deps() {
     # This task calculates the build time dependencies of the recipe, and is
     # required because while a task can deptask on itself, those dependencies
     # do not show up in BB_TASKDEPDATA. To work around that, this task does the
-    # deptask on do_create_spdx and writes out the dependencies it finds, then
-    # do_create_spdx reads in the found dependencies when writing the actual
+    # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then
+    # downstream tasks read in the found dependencies when writing the actual
     # SPDX document
     import json
     import oe.spdx_common
@@ -106,15 +99,13 @@ python do_collect_spdx_deps() {
 
     spdx_deps_file = Path(d.getVar("SPDXDEPS"))
 
-    deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx")
+    deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx")
 
     with spdx_deps_file.open("w") as f:
         json.dump(deps, f)
 }
-# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source
-addtask do_collect_spdx_deps after do_unpack
-do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}"
-do_collect_spdx_deps[deptask] = "do_create_spdx"
+addtask do_collect_spdx_deps
+do_collect_spdx_deps[deptask] = "do_create_recipe_spdx"
 do_collect_spdx_deps[dirs] = "${SPDXDIR}"
 
 oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA"
meta/lib/oe/spdx30_tasks.py +279 -123

559 lines not on this page · raw message

meta/lib/oeqa/selftest/cases/spdx.py +12 -7
index 5830d7c087..759ca86b73 100644
@@ -141,6 +141,11 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
     SPDX_CLASS = "create-spdx-3.0"
 
     def test_base_files(self):
+        self.check_recipe_spdx(
+            "base-files",
+            "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/static/static-base-files.spdx.json",
+            task="create_recipe_spdx",
+        )
         self.check_recipe_spdx(
             "base-files",
             "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json",
@@ -149,7 +154,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
     def test_gcc_include_source(self):
         objset = self.check_recipe_spdx(
             "gcc",
-            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-gcc.spdx.json",
+            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-gcc.spdx.json",
             extraconf="""\
                 SPDX_INCLUDE_SOURCES = "1"
                 """,
@@ -162,12 +167,12 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
             if software_file.name == filename:
                 found = True
                 self.logger.info(
-                    f"The spdxId of {filename} in recipe-gcc.spdx.json is {software_file.spdxId}"
+                    f"The spdxId of {filename} in build-gcc.spdx.json is {software_file.spdxId}"
                 )
                 break
 
         self.assertTrue(
-            found, f"Not found source file {filename} in recipe-gcc.spdx.json\n"
+            found, f"Not found source file {filename} in build-gcc.spdx.json\n"
         )
 
     def test_core_image_minimal(self):
@@ -305,7 +310,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
         # This will fail with NameError if new_annotation() is called incorrectly
         objset = self.check_recipe_spdx(
             "base-files",
-            "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/recipes/recipe-base-files.spdx.json",
+            "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/builds/build-base-files.spdx.json",
             extraconf=textwrap.dedent(
                 f"""\
                 ANNOTATION1 = "{ANNOTATION_VAR1}"
@@ -360,8 +365,8 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
 
     def test_kernel_config_spdx(self):
         kernel_recipe = get_bb_var("PREFERRED_PROVIDER_virtual/kernel")
-        spdx_file = f"recipe-{kernel_recipe}.spdx.json"
-        spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/recipes/{spdx_file}"
+        spdx_file = f"build-{kernel_recipe}.spdx.json"
+        spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/builds/{spdx_file}"
 
         # Make sure kernel is configured first
         bitbake(f"-c configure {kernel_recipe}")
@@ -392,7 +397,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
     def test_packageconfig_spdx(self):
         objset = self.check_recipe_spdx(
             "tar",
-            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-tar.spdx.json",
+            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-tar.spdx.json",
             extraconf="""\
                 SPDX_INCLUDE_PACKAGECONFIG = "1"
                 """,
Signature · 1 line
2.53.0
#65 lore raw

Re: [OE-core][PATCH v6 03/15] spdx3: Add recipe SPDX data

Quoted 9 lines · Joshua Watt via lists.openembedded.org

Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.

Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch

We talked about these patches on the review call. I'm a bit worried about the direction we're going from a few angles.

The general theme is the complexity and increasingly seemingly tangled web we seem to be weaving and whether we're going to end up in a good place.

Taking NO_GENERIC_LICENSE specifically, it may be we should mandate that such licenses are copied into the metadata, then we solve the license data problem that way? That would simplify some of the problems we're facing and reduce some set of the corner cases.

This patch adds a new task into the task graph and I'm getting a bit worried about the number of them the SPDX class is adding. I appreciate there is a later patch removing one, which is nice though :)

So, for this patch, could we just drop NO_GENERIC_LICENSE and how much code complexity improvement does that buy us?

Cheers,

Richard

#66 lore raw

Re: [OE-core][PATCH v6 03/15] spdx3: Add recipe SPDX data

Quoted 19 lines · Richard Purdie
Quoted 9 lines · Joshua Watt via lists.openembedded.org

Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.

Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch

We talked about these patches on the review call. I'm a bit worried about the direction we're going from a few angles.

The general theme is the complexity and increasingly seemingly tangled web we seem to be weaving and whether we're going to end up in a good place.

Quoted 8 lines

Taking NO_GENERIC_LICENSE specifically, it may be we should mandate that such licenses are copied into the metadata, then we solve the license data problem that way? That would simplify some of the problems we're facing and reduce some set of the corner cases.

This patch adds a new task into the task graph and I'm getting a bit worried about the number of them the SPDX class is adding. I appreciate there is a later patch removing one, which is nice though :)

With the removal of the vestigial task in this patch series, the task graph for SPDX is:

do_create_recipe_spdx - > "static" information we can determine about the recipe just from the metadata (no fetching, compiling, etc.)

do_create_spdx -> Information about what we built and how we built it. We obviously have to build to figure this part out (the definition of this didn't change in this patch series; it should really be called do_create_build_spdx, but it inherited the name from the SPDX 2 code, so I don't want to change it)

do_create_runtime_spdx -> Information about runtime packages. This has to be a separate task because while the build graph is a DAG, the runtime graph is not. The definition of this didn't change in this patch series.

Various SBoM assembly tasks: These are the tasks that take the individual SPDX files generated by the tasks above and link them into a complete document that ends up in DEPLOY_DIR. They are all identified by having "sbom" in the name (do_create_image_sbom_spdx)

Quoted 3 lines

So, for this patch, could we just drop NO_GENERIC_LICENSE and how much code complexity improvement does that buy us?

I'm not clear what you mean by this. I'm not including any additional License information, because we don't have it. I didn't change any license handling in the SPDX code, and I didn't add any more, so if you're talking about simplifying the SPDX code by dropping NO_GENERIC_LICENSE, it gains you nothing here specifically.

It might be nice to improve NO_GENERIC_LICENSE in general, but I don't think we can do that for 6.0. If we do that later, we might be able to add license information to the "recipe" level SPDX data.

The comment in the commit messages was probably more of a gripe than useful information (It feels like we _should_ be able to get license data statically, but we can't). I'll just remove it.

Quoted 4 lines

Cheers,

Richard

#67 lore raw

Re: [OE-core][PATCH v6 03/15] spdx3: Add recipe SPDX data

Quoted 42 lines · Joshua Watt
Quoted 19 lines · Richard Purdie
Quoted 9 lines · Joshua Watt via lists.openembedded.org

Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.

Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch

We talked about these patches on the review call. I'm a bit worried about the direction we're going from a few angles.

The general theme is the complexity and increasingly seemingly tangled web we seem to be weaving and whether we're going to end up in a good place.

Quoted 8 lines

Taking NO_GENERIC_LICENSE specifically, it may be we should mandate that such licenses are copied into the metadata, then we solve the license data problem that way? That would simplify some of the problems we're facing and reduce some set of the corner cases.

This patch adds a new task into the task graph and I'm getting a bit worried about the number of them the SPDX class is adding. I appreciate there is a later patch removing one, which is nice though :)

With the removal of the vestigial task in this patch series, the task graph for SPDX is:

do_create_recipe_spdx - > "static" information we can determine about the recipe just from the metadata (no fetching, compiling, etc.)

do_create_spdx -> Information about what we built and how we built it. We obviously have to build to figure this part out (the definition of this didn't change in this patch series; it should really be called do_create_build_spdx, but it inherited the name from the SPDX 2 code, so I don't want to change it)

Quoted 4 lines

do_create_runtime_spdx -> Information about runtime packages. This has to be a separate task because while the build graph is a DAG, the runtime graph is not. The definition of this didn't change in this patch series.

I'm not entirely sure why we couldn't collect both sets of information in one go in the same task, maybe inspecting BB_TASKDEPS instead of the tasks actual dependencies but that is getting distracted into other issues I guess.

Quoted 15 lines

Various SBoM assembly tasks: These are the tasks that take the individual SPDX files generated by the tasks above and link them into a complete document that ends up in DEPLOY_DIR. They are all identified by having "sbom" in the name (do_create_image_sbom_spdx)

Quoted 2 lines

So, for this patch, could we just drop NO_GENERIC_LICENSE and how much code complexity improvement does that buy us?

I'm not clear what you mean by this. I'm not including any additional License information, because we don't have it. I didn't change any license handling in the SPDX code, and I didn't add any more, so if you're talking about simplifying the SPDX code by dropping NO_GENERIC_LICENSE, it gains you nothing here specifically.

Quoted 7 lines

It might be nice to improve NO_GENERIC_LICENSE in general, but I don't think we can do that for 6.0. If we do that later, we might be able to add license information to the "recipe" level SPDX data.

The comment in the commit messages was probably more of a gripe than useful information (It feels like we _should_ be able to get license data statically, but we can't). I'll just remove it.

Ok, fair enough. I was more thinking that we could fix things so we could get that information. I think I was getting confused and thinking you were getting partial information.

We should perhaps separate out the NO_GENERIC_LICENSE issue into a separate bug/issue to work on.

Cheers,

Richard

#68 lore raw

PATCH v6 4/15 spdx3: Add recipe SBoM task

In reply to Joshua Watt

Adds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example:

```
bitbake -c create_recipe_sbom zstd
```

Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively).

The complete SBoM for all (target) recipes can be built with:

```
bitbake -c create_recipe_sbom meta-world-recipe-sbom
```
---
 meta/classes/create-spdx-3.0.bbclass          | 32 +++++++++++++++++++
 meta/classes/spdx-common.bbclass              |  1 +
 meta/conf/distro/include/maintainers.inc      |  1 +
 meta/lib/oe/spdx30_tasks.py                   | 10 ++++++
 meta/lib/oeqa/selftest/cases/spdx.py          | 10 ++++++
 .../meta/meta-world-recipe-sbom.bb            | 29 +++++++++++++++++
 6 files changed, 83 insertions(+)
 create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
meta/classes/create-spdx-3.0.bbclass +32 -0
index 672ca27cd0..c3ea95b8bc 100644
@@ -142,6 +142,10 @@ SPDX_PACKAGE_URLS[doc] = "A space separated list of Package URLs (purls) for \
     Override this variable to replace the default, otherwise append or prepend \
     to add additional purls."
 
+SPDX_RECIPE_SBOM_NAME ?= "${PN}-recipe-sbom"
+SPDX_RECIPE_SBOM_NAME[doc] = "The name of output recipe SBoM when using \
+    create_recipe_sbom"
+
 IMAGE_CLASSES:append = " create-spdx-image-3.0"
 SDK_CLASSES += "create-spdx-sdk-3.0"
 
@@ -240,6 +244,34 @@ python do_create_package_spdx_setscene () {
 }
 addtask do_create_package_spdx_setscene
 
+addtask do_create_recipe_sbom after create_recipe_spdx
+python do_create_recipe_sbom() {
+    import oe.spdx30_tasks
+    from pathlib import Path
+    deploydir = Path(d.getVar("SPDXRECIPESBOMDEPLOY"))
+    oe.spdx30_tasks.create_recipe_sbom(d, deploydir)
+}
+
+SSTATETASKS += "do_create_recipe_sbom"
+do_create_recipe_sbom[recrdeptask] = "do_create_recipe_spdx"
+do_create_recipe_sbom[nostamp] = "1"
+do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}"
+do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
+do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}"
+do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}"
+do_create_recipe_sbom[vardeps] += "\
+    SPDX_INCLUDE_BITBAKE_PARENT_BUILD \
+    SPDX_PACKAGE_ADDITIONAL_PURPOSE \
+    SPDX_PROFILES \
+    SPDX_NAMESPACE_PREFIX \
+    SPDX_UUID_NAMESPACE \
+    "
+
+python do_create_recipe_sbom_setscene () {
+    sstate_setscene(d)
+}
+addtask do_create_recipe_sbom_setscene
+
 python spdx30_build_started_handler () {
     import oe.spdx30_tasks
     d = e.data.createCopy()
meta/classes/spdx-common.bbclass +1 -0
index 3c239a718b..abf2332bee 100644
@@ -25,6 +25,7 @@ SPDX_TOOL_VERSION ??= "1.0"
 
 SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy"
 SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy"
+SPDXRECIPESBOMDEPLOY = "${SPDXDIR}/recipes-bom-deploy"
 
 SPDX_INCLUDE_SOURCES ??= "0"
 SPDX_INCLUDE_COMPILED_SOURCES ??= "0"
meta/conf/distro/include/maintainers.inc +1 -0
index 3c7fc4974d..40e090d452 100644
@@ -535,6 +535,7 @@ RECIPE_MAINTAINER:pn-meta-go-toolchain = "Richard Purdie <richard.purdie@linuxfo
 RECIPE_MAINTAINER:pn-meta-ide-support = "Richard Purdie <[email protected]>"
 RECIPE_MAINTAINER:pn-meta-toolchain = "Richard Purdie <[email protected]>"
 RECIPE_MAINTAINER:pn-meta-world-pkgdata = "Richard Purdie <[email protected]>"
+RECIPE_MAINTAINER:pn-meta-world-recipe-sbom = "Joshua Watt <[email protected]>"
 RECIPE_MAINTAINER:pn-mingetty = "Yi Zhao <[email protected]>"
 RECIPE_MAINTAINER:pn-mini-x-session = "Unassigned <[email protected]>"
 RECIPE_MAINTAINER:pn-minicom = "Unassigned <[email protected]>"
meta/lib/oe/spdx30_tasks.py +10 -0
index a8b4525e3d..b6c917045e 100644
@@ -1564,3 +1564,13 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname):
     oe.sbom30.write_jsonld_doc(
         d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json")
     )
+
+
+def create_recipe_sbom(d, deploydir):
+    sbom_name = d.getVar("SPDX_RECIPE_SBOM_NAME")
+
+    recipe, recipe_objset = load_recipe_spdx(d)
+
+    objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset])
+
+    oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json"))
meta/lib/oeqa/selftest/cases/spdx.py +10 -0
index 759ca86b73..efee0214fc 100644
@@ -151,6 +151,16 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
             "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json",
         )
 
+    def test_world_sbom(self):
+        objset = self.check_recipe_spdx(
+            "meta-world-recipe-sbom",
+            "{DEPLOY_DIR_IMAGE}/world-recipe-sbom.spdx.json",
+            task="create_recipe_sbom",
+        )
+
+        # Document should be fully linked
+        self.check_objset_missing_ids(objset)
+
     def test_gcc_include_source(self):
         objset = self.check_recipe_spdx(
             "gcc",
meta/recipes-core/meta/meta-world-recipe-sbom.bb new file +29 -0
new file mode 100644
index 0000000000..b47a3229c9
@@ -0,0 +1,29 @@
+SUMMARY = "Generates a combined SBoM for all world recipes"
+LICENSE = "MIT"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+PACKAGE_ARCH = "${MACHINE_ARCH}"
+
+inherit nopackages
+deltask do_fetch
+deltask do_unpack
+deltask do_patch
+deltask do_configure
+deltask do_compile
+deltask do_install
+
+do_prepare_recipe_sysroot[deptask] = ""
+
+WORLD_SBOM_EXCLUDE ?= ""
+
+EXCLUDE_FROM_WORLD = "1"
+SPDX_RECIPE_SBOM_NAME = "world-recipe-sbom"
+
+python calculate_extra_depends() {
+    exclude = set('${WORLD_SBOM_EXCLUDE}'.split())
+    exclude |= set(f"{v}-{self_pn}" for v in '${MULTILIB_VARIANTS}'.split())
+    exclude.add(self_pn)
+
+    deps.extend(p for p in world_target if p not in exclude)
+}
Signature · 1 line
2.53.0
#69 lore raw

Re: [OE-core][PATCH v6 04/15] spdx3: Add recipe SBoM task

Quoted 10 lines · Joshua Watt via lists.openembedded.org

Adds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example:

```
bitbake -c create_recipe_sbom zstd
```

Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively).

Quoted 7 lines

The complete SBoM for all (target) recipes can be built with:

```
bitbake -c create_recipe_sbom meta-world-recipe-sbom
```

With this change I got a bit confused as it wasn't obvious which task you'd run just looking at the meta-world-recipe-sbom recipe.

Should that recipe do something like:

addtask create_recipe_sbom before do_build

so that the recipe actually triggers the thing which it is there for?

Cheers,

Richard

#70 lore raw

Re: [OE-core][PATCH v6 04/15] spdx3: Add recipe SBoM task

Quoted 20 lines · Richard Purdie
Quoted 10 lines · Joshua Watt via lists.openembedded.org

Adds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example:

```
bitbake -c create_recipe_sbom zstd
```

Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively).

Quoted 7 lines

The complete SBoM for all (target) recipes can be built with:

```
bitbake -c create_recipe_sbom meta-world-recipe-sbom
```
Quoted 8 lines

With this change I got a bit confused as it wasn't obvious which task you'd run just looking at the meta-world-recipe-sbom recipe.

Should that recipe do something like:

addtask create_recipe_sbom before do_build

so that the recipe actually triggers the thing which it is there for?

Yes

Quoted 4 lines

Cheers,

Richard

#71 lore raw

PATCH v6 5/15 spdx3: Add is-native property

In reply to Joshua Watt

Adds a custom is-native property to the recipe package to indicate if it is a native recipe

---
 meta/lib/oe/sbom30.py       | 20 ++++++++++++++++++++
 meta/lib/oe/spdx30_tasks.py | 18 +++++++++++-------
 2 files changed, 31 insertions(+), 7 deletions(-)
meta/lib/oe/sbom30.py +20 -0
index 227ac51877..50a72fce39 100644
@@ -118,6 +118,26 @@ class OEDocumentExtension(oe.spdx30.extension_Extension):
         )
 
 
[email protected](OE_SPDX_BASE + "recipe-extension")
+class OERecipeExtension(oe.spdx30.extension_Extension):
+    """
+    This extension is added to recipe software_Packages to indicate various
+    useful bits of information about the recipe
+    """
+
+    CLOSED = True
+
+    @classmethod
+    def _register_props(cls):
+        super()._register_props()
+        cls._add_property(
+            "is_native",
+            oe.spdx30.BooleanProp(),
+            OE_SPDX_BASE + "is-native",
+            max_count=1,
+        )
+
+
 def spdxid_hash(*items):
     h = hashlib.md5()
     for i in items:
meta/lib/oe/spdx30_tasks.py +11 -7
index b6c917045e..a8fffbb085 100644
@@ -477,6 +477,10 @@ def set_purls(spdx_package, purls):
         )
 
 
+def get_is_native(d):
+    return bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d)
+
+
 def create_recipe_spdx(d):
     deploydir = Path(d.getVar("SPDXRECIPEDEPLOY"))
     deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
@@ -507,6 +511,11 @@ def create_recipe_spdx(d):
         )
     )
 
+    if get_is_native(d):
+        ext = oe.sbom30.OERecipeExtension()
+        ext.is_native = True
+        recipe.extension.append(ext)
+
     set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split())
 
     # TODO: This doesn't work before do_unpack because the license text has to
@@ -668,9 +677,7 @@ def create_spdx(d):
     spdx_workdir = Path(d.getVar("SPDXWORK"))
     include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1"
     pkg_arch = d.getVar("SSTATE_PKGARCH")
-    is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class(
-        "cross", d
-    )
+    is_native = get_is_native(d)
 
     recipe, recipe_objset = load_recipe_spdx(d)
 
@@ -1021,14 +1028,11 @@ def create_spdx(d):
 def create_package_spdx(d):
     deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
     deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY"))
-    is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class(
-        "cross", d
-    )
 
     providers = oe.spdx_common.collect_package_providers(d)
     pkg_arch = d.getVar("SSTATE_PKGARCH")
 
-    if is_native:
+    if get_is_native(d):
         return
 
     bb.build.exec_func("read_subpackage_metadata", d)
Signature · 1 line
2.53.0
#72 lore raw

PATCH v6 6/15 spdx30: Include patch file information in VEX

In reply to Joshua Watt

Modifies the SPDX VEX output to include the patches that fix a particular vulnerability. This is done by adding a patchedBy relationship from the VexFixedVulnAssessmentRelationship to the File that provides the fix.

If the file can be located without fetching (e.g. is a file:// in SRC_URI), the checksum will be included.

---
 meta/lib/oe/sbom30.py       | 60 ++++++++++++++-------------
 meta/lib/oe/spdx30_tasks.py | 81 ++++++++++++++++++++++++++++---------
 2 files changed, 92 insertions(+), 49 deletions(-)
meta/lib/oe/sbom30.py +31 -29
index 50a72fce39..21f084dc16 100644
@@ -620,37 +620,38 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
         )
         spdx_file.extension.append(OELicenseScannedExtension())
 
-    def new_file(self, _id, name, path, *, purposes=[]):
-        sha256_hash = bb.utils.sha256_file(path)
+    def new_file(self, _id, name, path, *, purposes=[], hashfile=True):
+        if hashfile:
+            sha256_hash = bb.utils.sha256_file(path)
 
-        for f in self.by_sha256_hash.get(sha256_hash, []):
-            if not isinstance(f, oe.spdx30.software_File):
-                continue
+            for f in self.by_sha256_hash.get(sha256_hash, []):
+                if not isinstance(f, oe.spdx30.software_File):
+                    continue
 
-            if purposes:
-                new_primary = purposes[0]
-                new_additional = []
+                if purposes:
+                    new_primary = purposes[0]
+                    new_additional = []
 
-                if f.software_primaryPurpose:
-                    new_additional.append(f.software_primaryPurpose)
-                new_additional.extend(f.software_additionalPurpose)
+                    if f.software_primaryPurpose:
+                        new_additional.append(f.software_primaryPurpose)
+                    new_additional.extend(f.software_additionalPurpose)
 
-                new_additional = sorted(
-                    list(set(p for p in new_additional if p != new_primary))
-                )
+                    new_additional = sorted(
+                        list(set(p for p in new_additional if p != new_primary))
+                    )
 
-                f.software_primaryPurpose = new_primary
-                f.software_additionalPurpose = new_additional
+                    f.software_primaryPurpose = new_primary
+                    f.software_additionalPurpose = new_additional
 
-            if f.name != name:
-                for e in f.extension:
-                    if isinstance(e, OEFileNameAliasExtension):
-                        e.aliases.append(name)
-                        break
-                else:
-                    f.extension.append(OEFileNameAliasExtension(aliases=[name]))
+                if f.name != name:
+                    for e in f.extension:
+                        if isinstance(e, OEFileNameAliasExtension):
+                            e.aliases.append(name)
+                            break
+                    else:
+                        f.extension.append(OEFileNameAliasExtension(aliases=[name]))
 
-            return f
+                return f
 
         spdx_file = oe.spdx30.software_File(
             _id=_id,
@@ -661,12 +662,13 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
             spdx_file.software_primaryPurpose = purposes[0]
             spdx_file.software_additionalPurpose = purposes[1:]
 
-        spdx_file.verifiedUsing.append(
-            oe.spdx30.Hash(
-                algorithm=oe.spdx30.HashAlgorithm.sha256,
-                hashValue=sha256_hash,
+        if hashfile:
+            spdx_file.verifiedUsing.append(
+                oe.spdx30.Hash(
+                    algorithm=oe.spdx30.HashAlgorithm.sha256,
+                    hashValue=sha256_hash,
+                )
             )
-        )
 
         return self.add(spdx_file)
 
meta/lib/oe/spdx30_tasks.py +61 -20

139 lines not on this page · raw message

Signature · 1 line
2.53.0
#73 lore raw

PATCH v6 7/15 spdx: De-duplicate CreationInfo

In reply to Joshua Watt

De-duplicates CreationInfo objects that are identical (except for ID) when writing out an SBoM. This significantly reduces the number of CreationInfo objects that end up in the final document.

---
 meta/lib/oe/sbom30.py | 112 ++++++++++++++++++++++++++++++------------
 meta/lib/oe/spdx30.py |   2 +-
 2 files changed, 81 insertions(+), 33 deletions(-)
meta/lib/oe/sbom30.py +80 -32
index 21f084dc16..55a2863d2d 100644
@@ -14,6 +14,7 @@ import uuid
 import os
 import oe.spdx_common
 from datetime import datetime, timezone
+from contextlib import contextmanager
 
 OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/"
 
@@ -191,6 +192,25 @@ def to_list(l):
     return l
 
 
+class Dedup(object):
+    def __init__(self, objset):
+        self.unique = set()
+        self.dedup = {}
+        self.objset = objset
+
+    def find_duplicates(self, cmp, typ, **kwargs):
+        for o in self.objset.foreach_filter(typ, **kwargs):
+            for u in self.unique:
+                if cmp(u, o):
+                    self.dedup[o] = u
+                    break
+            else:
+                self.unique.add(o)
+
+    def get(self, o):
+        return self.dedup.get(o, o)
+
+
 class ObjectSet(oe.spdx30.SHACLObjectSet):
     def __init__(self, d):
         super().__init__()
@@ -895,6 +915,45 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
         self.missing_ids -= set(imports.keys())
         return self.missing_ids
 
+    @contextmanager
+    def deduplicate(self):
+        d = Dedup(self)
+
+        yield d
+
+        visited = set()
+
+        def visit(o, path):
+            if isinstance(o, oe.spdx30.SHACLObject):
+                if o in visited:
+                    return False
+                visited.add(o)
+
+                for k in o:
+                    v = o[k]
+                    if isinstance(v, oe.spdx30.SHACLObject):
+                        o[k] = d.get(v)
+
+            elif isinstance(o, oe.spdx30.ListProxy):
+                for idx, v in enumerate(o):
+                    if isinstance(v, oe.spdx30.SHACLObject):
+                        o[idx] = d.get(v)
+
+            return True
+
+        if d.dedup:
+            for o in self.objects:
+                o.walk(visit)
+
+            for k, v in d.dedup.items():
+                bb.debug(
+                    1,
+                    f"Removing duplicate {k.__class__.__name__} {k._id or id(k)} -> {v._id or id(v)}",
+                )
+                self.objects.discard(k)
+
+            self.create_index()
+
 
 def load_jsonld(d, path, required=False):
     deserializer = oe.spdx30.JSONLDDeserializer()
@@ -1080,39 +1139,28 @@ def create_sbom(d, name, root_elements, add_objectsets=[]):
     # SBoM should be the only root element of the document
     objset.doc.rootElement = [sbom]
 
-    # De-duplicate licenses
-    unique = set()
-    dedup = {}
-    for lic in objset.foreach_type(oe.spdx30.simplelicensing_LicenseExpression):
-        for u in unique:
-            if (
-                u.simplelicensing_licenseExpression
-                == lic.simplelicensing_licenseExpression
-                and u.simplelicensing_licenseListVersion
-                == lic.simplelicensing_licenseListVersion
-            ):
-                dedup[lic] = u
-                break
-        else:
-            unique.add(lic)
-
-    if dedup:
-        for rel in objset.foreach_filter(
-            oe.spdx30.Relationship,
-            relationshipType=oe.spdx30.RelationshipType.hasDeclaredLicense,
-        ):
-            rel.to = [dedup.get(to, to) for to in rel.to]
-
-        for rel in objset.foreach_filter(
-            oe.spdx30.Relationship,
-            relationshipType=oe.spdx30.RelationshipType.hasConcludedLicense,
-        ):
-            rel.to = [dedup.get(to, to) for to in rel.to]
+    def cmp_license_expression(a, b):
+        return (
+            a.simplelicensing_licenseExpression == b.simplelicensing_licenseExpression
+            and a.simplelicensing_licenseListVersion
+            == b.simplelicensing_licenseListVersion
+        )
 
-        for k, v in dedup.items():
-            bb.debug(1, f"Removing duplicate License {k._id} -> {v._id}")
-            objset.objects.remove(k)
+    def cmp_creation_info(a, b):
+        data_a = {k: a[k] for k in a}
+        data_b = {k: b[k] for k in b}
+        data_a["@id"] = ""
+        data_b["@id"] = ""
+        return data_a == data_b
+
+    with objset.deduplicate() as dedup:
+        # De-duplicate licenses
+        dedup.find_duplicates(
+            cmp_license_expression,
+            oe.spdx30.simplelicensing_LicenseExpression,
+        )
 
-        objset.create_index()
+        # Deduplicate creation info
+        dedup.find_duplicates(cmp_creation_info, oe.spdx30.CreationInfo)
 
     return objset, sbom
meta/lib/oe/spdx30.py +1 -1
index cd97eebd18..1f58402ffc 100644
@@ -701,7 +701,7 @@ class SHACLObject(object):
         self.__dict__["_obj_data"][iri] = prop.init()
 
     def __iter__(self):
-        return self._OBJ_PROPERTIES.keys()
+        return iter(self._OBJ_PROPERTIES.keys())
 
     def walk(self, callback, path=None):
         """
Signature · 1 line
2.53.0
#74 lore raw

PATCH v6 8/15 spdx_common: Check for dependent task in task flags

In reply to Joshua Watt

Checks that the task being used to detect dependencies is present in at least one dependency task flag of the current task. This helps prevent errors where the wrong task is specified and never found.

---
 meta/lib/oe/spdx_common.py | 11 +++++++++++
 1 file changed, 11 insertions(+)
meta/lib/oe/spdx_common.py +11 -0
index 72c24180d5..3aaf2a9c8b 100644
@@ -96,6 +96,17 @@ def collect_direct_deps(d, dep_task):
 
     taskdepdata = d.getVar("BB_TASKDEPDATA", False)
 
+    # Check that the task is listed one of the task dependency flags of the
+    # current task
+    depflags = (
+        set((d.getVarFlag(current_task, "deptask") or "").split())
+        | set((d.getVarFlag(current_task, "rdeptask") or "").split())
+        | set((d.getVarFlag(current_task, "recrdeptask") or "").split())
+    )
+
+    if not dep_task in depflags:
+        bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}")
+
     for this_dep in taskdepdata.values():
         if this_dep[0] == pn and this_dep[1] == current_task:
             break
Signature · 1 line
2.53.0
#75 lore raw

PATCH v6 9/15 spdx30: Skip install package CVE information

In reply to Joshua Watt

Skips adding the install package CVE information by default. This information grows exponentially, since it ends up be N_CVES * N_PACKAGES. The CVE information for a given installed package can be determined by following the "generates" link between the install package and the recipe and looking at the CVE information for the recipe, meaning that the CVE information is only included once in the SPDX document.

If users still need the legacy method of including CVE information for each package, then then can set SPDX_PACKAGE_INCLUDE_VEX = "1"

---
 meta/classes/create-spdx-3.0.bbclass | 11 ++++++++
 meta/lib/oe/spdx30_tasks.py          | 39 ++++++++++++++--------------
 meta/lib/oeqa/selftest/cases/spdx.py | 12 +++++++++
 3 files changed, 43 insertions(+), 19 deletions(-)
meta/classes/create-spdx-3.0.bbclass +11 -0
index c3ea95b8bc..88b7ef9f42 100644
@@ -45,6 +45,17 @@ SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to
     including those already fixed upstream (warning: This can be large and \
     slow)."
 
+SPDX_PACKAGE_INCLUDE_VEX ?= "0"
+SPDX_PACKAGE_INCLUDE_VEX[doc] = "Link VEX information to the binary package outputs. \
+    Normally, VEX information is only linked to the common recipe that `generates` the \
+    binary packages, but setting this to '1' will cause it to also be linked into the \
+    generated binary packages. This is off by default because linking the VEX data to \
+    each package causes the SPDX output to grow very large, and the same information \
+    can be determined by following the `generates` relationship back to the recipe. \
+    Before recipe packages were introduced, this was the only way VEX data was \
+    expressed; you may need to enable this if your downstream tools do not \
+    understand how to trace back to the recipe to find VEX information."
+
 SPDX_INCLUDE_TIMESTAMPS ?= "0"
 SPDX_INCLUDE_TIMESTAMPS[doc] = "Include time stamps in SPDX output. This is \
     useful if you want to know when artifacts were produced and when builds \
meta/lib/oe/spdx30_tasks.py +20 -19
index aec47d4f81..887fac813a 100644
@@ -771,27 +771,28 @@ def create_spdx(d):
     # Collect all VEX statements from the recipe
     vex_statements = {}
     vex_patches = {}
-    for rel in recipe_objset.foreach_filter(
-        oe.spdx30.Relationship,
-        relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability,
-    ):
-        for cve in rel.to:
-            vex_statements[cve] = []
-            vex_patches[cve] = []
-
-    for cve in vex_statements.keys():
+    if (d.getVar("SPDX_PACKAGE_INCLUDE_VEX") or "") == "1":
         for rel in recipe_objset.foreach_filter(
-            oe.spdx30.security_VexVulnAssessmentRelationship,
-            from_=cve,
+            oe.spdx30.Relationship,
+            relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability,
         ):
-            vex_statements[cve].append(rel)
-            if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn:
-                for patch_rel in recipe_objset.foreach_filter(
-                    oe.spdx30.Relationship,
-                    relationshipType=oe.spdx30.RelationshipType.patchedBy,
-                    from_=rel,
-                ):
-                    vex_patches[cve].extend(patch_rel.to)
+            for cve in rel.to:
+                vex_statements[cve] = []
+                vex_patches[cve] = []
+
+        for cve in vex_statements.keys():
+            for rel in recipe_objset.foreach_filter(
+                oe.spdx30.security_VexVulnAssessmentRelationship,
+                from_=cve,
+            ):
+                vex_statements[cve].append(rel)
+                if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn:
+                    for patch_rel in recipe_objset.foreach_filter(
+                        oe.spdx30.Relationship,
+                        relationshipType=oe.spdx30.RelationshipType.patchedBy,
+                        from_=rel,
+                    ):
+                        vex_patches[cve].extend(patch_rel.to)
 
     # Write out the package SPDX data now. It is not complete as we cannot
     # write the runtime data, so write it to a staging area and a later task
meta/lib/oeqa/selftest/cases/spdx.py +12 -0
index efee0214fc..f1ea2694cf 100644
@@ -429,3 +429,15 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
                 value, ["enabled", "disabled"],
                 f"Unexpected PACKAGECONFIG value '{value}' for {key}"
             )
+
+    def test_package_vex(self):
+        objset = self.check_recipe_spdx(
+            "core-image-minimal",
+            "{DEPLOY_DIR_IMAGE}/core-image-minimal-{MACHINE}.rootfs.spdx.json",
+            extraconf="""\
+                SPDX_PACKAGE_INCLUDE_VEX = "1"
+                """,
+        )
+
+        # Document should be fully linked
+        self.check_objset_missing_ids(objset)
Signature · 1 line
2.53.0
#76 lore raw

Re: [OE-core][PATCH v6 09/15] spdx30: Skip install package CVE information

Quoted 26 lines · Joshua Watt via lists.openembedded.org

Skips adding the install package CVE information by default. This information grows exponentially, since it ends up be N_CVES * N_PACKAGES. The CVE information for a given installed package can be determined by following the "generates" link between the install package and the recipe and looking at the CVE information for the recipe, meaning that the CVE information is only included once in the SPDX document.

If users still need the legacy method of including CVE information for each package, then then can set SPDX_PACKAGE_INCLUDE_VEX = "1"

---
 meta/classes/create-spdx-3.0.bbclass | 11 ++++++++ meta/lib/oe/spdx30_tasks.py          | 39 ++++++++++++++-------------- meta/lib/oeqa/selftest/cases/spdx.py | 12 +++++++++

 3 files changed, 43 insertions(+), 19 deletions(-)
meta/classes/create-spdx-3.0.bbclass +0 -0
index c3ea95b8bc..88b7ef9f42 100644
@@ -45,6 +45,17 @@ SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to
      including those already fixed upstream (warning: This can be large and \

     slow)."

meta/classes/create-spdx-3.0.bbclass SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to · line 57

Quoted 10 lines
meta/classes/create-spdx-3.0.bbclass +10 -0
+SPDX_PACKAGE_INCLUDE_VEX ?= "0"
+SPDX_PACKAGE_INCLUDE_VEX[doc] = "Link VEX information to the binary package outputs. \
+    Normally, VEX information is only linked to the common recipe that `generates` the \
+    binary packages, but setting this to '1' will cause it to also be linked into the \
+    generated binary packages. This is off by default because linking the VEX data to \
+    each package causes the SPDX output to grow very large, and the same information \
+    can be determined by following the `generates` relationship back to the recipe. \
+    Before recipe packages were introduced, this was the only way VEX data was \
+    expressed; you may need to enable this if your downstream tools do not \
+    understand how to trace back to the recipe to find VEX information."

To me, removing this duplication and keeping the SPDX docs usable seems like a very sensible thing to do. Do we want/need to make it configurable?

I appreciate some tools/usage may need fixing to work with this but adding configuration options like this makes it harder to use our code and also adds maintenance/testing overhead.

I think I'm very much in favour of just changing and generating things like this unconditionally and if someone needs to work with it differently, they can post process the output.

This goes back to my concern about the complexity of the code and configuration, I think we need to simplify and present fewer options to the user...

Cheers,

Richard

#77 lore raw

Re: [OE-core][PATCH v6 09/15] spdx30: Skip install package CVE information

Quoted 43 lines · Richard Purdie
Quoted 26 lines · Joshua Watt via lists.openembedded.org

Skips adding the install package CVE information by default. This information grows exponentially, since it ends up be N_CVES * N_PACKAGES. The CVE information for a given installed package can be determined by following the "generates" link between the install package and the recipe and looking at the CVE information for the recipe, meaning that the CVE information is only included once in the SPDX document.

If users still need the legacy method of including CVE information for each package, then then can set SPDX_PACKAGE_INCLUDE_VEX = "1"

---
 meta/classes/create-spdx-3.0.bbclass | 11 ++++++++
 meta/lib/oe/spdx30_tasks.py          | 39 ++++++++++++++--------------
 meta/lib/oeqa/selftest/cases/spdx.py | 12 +++++++++
 3 files changed, 43 insertions(+), 19 deletions(-)
meta/classes/create-spdx-3.0.bbclass +0 -0
index c3ea95b8bc..88b7ef9f42 100644
@@ -45,6 +45,17 @@ SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to
     including those already fixed upstream (warning: This can be large and \
     slow)."
 

meta/classes/create-spdx-3.0.bbclass SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to · line 57

Quoted 10 lines
meta/classes/create-spdx-3.0.bbclass +10 -0
+SPDX_PACKAGE_INCLUDE_VEX ?= "0"
+SPDX_PACKAGE_INCLUDE_VEX[doc] = "Link VEX information to the binary package outputs. \
+    Normally, VEX information is only linked to the common recipe that `generates` the \
+    binary packages, but setting this to '1' will cause it to also be linked into the \
+    generated binary packages. This is off by default because linking the VEX data to \
+    each package causes the SPDX output to grow very large, and the same information \
+    can be determined by following the `generates` relationship back to the recipe. \
+    Before recipe packages were introduced, this was the only way VEX data was \
+    expressed; you may need to enable this if your downstream tools do not \
+    understand how to trace back to the recipe to find VEX information."

To me, removing this duplication and keeping the SPDX docs usable seems like a very sensible thing to do. Do we want/need to make it configurable?

Quoted 3 lines

I appreciate some tools/usage may need fixing to work with this but adding configuration options like this makes it harder to use our code and also adds maintenance/testing overhead.

Maybe, but I'm very hesitant to break any existing SPDX based CVE workflows that people may have in a LTS release, which is the only reason I added the option. I'm fine to remove this after the LTS, IMHO it's just too close to LTS release to suddenly say "sorry this is all broken for you now"

Quoted 12 lines

I think I'm very much in favour of just changing and generating things like this unconditionally and if someone needs to work with it differently, they can post process the output.

This goes back to my concern about the complexity of the code and configuration, I think we need to simplify and present fewer options to the user...

Cheers,

Richard

#78 lore raw

Re: [OE-core][PATCH v6 09/15] spdx30: Skip install package CVE information

Quoted 49 lines · Joshua Watt
Quoted 43 lines · Richard Purdie
Quoted 26 lines · Joshua Watt via lists.openembedded.org

Skips adding the install package CVE information by default. This information grows exponentially, since it ends up be N_CVES * N_PACKAGES. The CVE information for a given installed package can be determined by following the "generates" link between the install package and the recipe and looking at the CVE information for the recipe, meaning that the CVE information is only included once in the SPDX document.

If users still need the legacy method of including CVE information for each package, then then can set SPDX_PACKAGE_INCLUDE_VEX = "1"

---
 meta/classes/create-spdx-3.0.bbclass | 11 ++++++++ meta/lib/oe/spdx30_tasks.py          | 39 ++++++++++++++-------------- meta/lib/oeqa/selftest/cases/spdx.py | 12 +++++++++

 3 files changed, 43 insertions(+), 19 deletions(-)
meta/classes/create-spdx-3.0.bbclass +0 -0
index c3ea95b8bc..88b7ef9f42 100644
@@ -45,6 +45,17 @@ SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to
      including those already fixed upstream (warning: This can be large and \

     slow)."

meta/classes/create-spdx-3.0.bbclass SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to · line 57

Quoted 10 lines
meta/classes/create-spdx-3.0.bbclass +10 -0
+SPDX_PACKAGE_INCLUDE_VEX ?= "0"
+SPDX_PACKAGE_INCLUDE_VEX[doc] = "Link VEX information to the binary package outputs. \
+    Normally, VEX information is only linked to the common recipe that `generates` the \
+    binary packages, but setting this to '1' will cause it to also be linked into the \
+    generated binary packages. This is off by default because linking the VEX data to \
+    each package causes the SPDX output to grow very large, and the same information \
+    can be determined by following the `generates` relationship back to the recipe. \
+    Before recipe packages were introduced, this was the only way VEX data was \
+    expressed; you may need to enable this if your downstream tools do not \
+    understand how to trace back to the recipe to find VEX information."

To me, removing this duplication and keeping the SPDX docs usable seems like a very sensible thing to do. Do we want/need to make it configurable?

Quoted 3 lines

I appreciate some tools/usage may need fixing to work with this but adding configuration options like this makes it harder to use our code and also adds maintenance/testing overhead.

Quoted 5 lines

Maybe, but I'm very hesitant to break any existing SPDX based CVE workflows that people may have in a LTS release, which is the only reason I added the option. I'm fine to remove this after the LTS, IMHO it's just too close to LTS release to suddenly say "sorry this is all broken for you now"

We're not yet at feature freeze and I'm getting very worried about all the combinations of things we're trying to support. Having so many user options is likely going to confuse users too . I can see both sides of this but I am leaning towards a cleaner implementation and having people adapt to it now rather than when we remove it later.

Do we know of tools that are going to struggle with this change? I'm guessing people can adapt to the change relatively easily?

Cheers,

Richard

#79 lore raw

Re: [OE-core][PATCH v6 09/15] spdx30: Skip install package CVE information

Quoted 57 lines · Richard Purdie
Quoted 49 lines · Joshua Watt
Quoted 43 lines · Richard Purdie
Quoted 26 lines · Joshua Watt via lists.openembedded.org

Skips adding the install package CVE information by default. This information grows exponentially, since it ends up be N_CVES * N_PACKAGES. The CVE information for a given installed package can be determined by following the "generates" link between the install package and the recipe and looking at the CVE information for the recipe, meaning that the CVE information is only included once in the SPDX document.

If users still need the legacy method of including CVE information for each package, then then can set SPDX_PACKAGE_INCLUDE_VEX = "1"

---
 meta/classes/create-spdx-3.0.bbclass | 11 ++++++++
 meta/lib/oe/spdx30_tasks.py          | 39 ++++++++++++++--------------
 meta/lib/oeqa/selftest/cases/spdx.py | 12 +++++++++
 3 files changed, 43 insertions(+), 19 deletions(-)
meta/classes/create-spdx-3.0.bbclass +0 -0
index c3ea95b8bc..88b7ef9f42 100644
@@ -45,6 +45,17 @@ SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to
     including those already fixed upstream (warning: This can be large and \
     slow)."
 

meta/classes/create-spdx-3.0.bbclass SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to · line 57

Quoted 10 lines
meta/classes/create-spdx-3.0.bbclass +10 -0
+SPDX_PACKAGE_INCLUDE_VEX ?= "0"
+SPDX_PACKAGE_INCLUDE_VEX[doc] = "Link VEX information to the binary package outputs. \
+    Normally, VEX information is only linked to the common recipe that `generates` the \
+    binary packages, but setting this to '1' will cause it to also be linked into the \
+    generated binary packages. This is off by default because linking the VEX data to \
+    each package causes the SPDX output to grow very large, and the same information \
+    can be determined by following the `generates` relationship back to the recipe. \
+    Before recipe packages were introduced, this was the only way VEX data was \
+    expressed; you may need to enable this if your downstream tools do not \
+    understand how to trace back to the recipe to find VEX information."

To me, removing this duplication and keeping the SPDX docs usable seems like a very sensible thing to do. Do we want/need to make it configurable?

Quoted 3 lines

I appreciate some tools/usage may need fixing to work with this but adding configuration options like this makes it harder to use our code and also adds maintenance/testing overhead.

Quoted 5 lines

Maybe, but I'm very hesitant to break any existing SPDX based CVE workflows that people may have in a LTS release, which is the only reason I added the option. I'm fine to remove this after the LTS, IMHO it's just too close to LTS release to suddenly say "sorry this is all broken for you now"

Quoted 8 lines

We're not yet at feature freeze and I'm getting very worried about all the combinations of things we're trying to support. Having so many user options is likely going to confuse users too . I can see both sides of this but I am leaning towards a cleaner implementation and having people adapt to it now rather than when we remove it later.

Do we know of tools that are going to struggle with this change? I'm guessing people can adapt to the change relatively easily?

Any tool that didn't know to change would get a lot more CVE reports show up, since the VEX data saying how we fixed things (derived from CVE_STATUS and patches) would be "missing". The fix isn't terribly hard as long as you know to follow the relationship back to the recipe package (where all the VEX data now exists).

Quoted 6 lines

Cheers,

Richard

#80 lore raw

PATCH v6 10/15 dummy-sdk-package: Disable SPDX

In reply to Joshua Watt

The dummy SDK packages do not need SPDX support, and since they play some games with allarch that cause problems, it's simplest to disable their SPDX output.

---
 meta/recipes-core/meta/dummy-sdk-package.inc | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-core/meta/dummy-sdk-package.inc +1 -0
index bf453cac9b..71e788b0b9 100644
@@ -4,6 +4,7 @@ LICENSE = "MIT"
 PACKAGE_ARCH = "all"
 
 inherit allarch
+inherit nospdx
 
 INHIBIT_DEFAULT_DEPS = "1"
 
Signature · 1 line
2.53.0
#81 lore raw

Re: [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX

Quoted 14 lines · Joshua Watt via lists.openembedded.org

The dummy SDK packages do not need SPDX support, and since they play some games with allarch that cause problems, it's simplest to disable their SPDX output.

---
 meta/recipes-core/meta/dummy-sdk-package.inc | 1 +

 1 file changed, 1 insertion(+)
meta/recipes-core/meta/dummy-sdk-package.inc +0 -0
index bf453cac9b..71e788b0b9 100644
@@ -4,6 +4,7 @@ LICENSE = "MIT"
Quoted 6 lines

 PACKAGE_ARCH = "all"

 inherit allarch
+inherit nospdx

 INHIBIT_DEFAULT_DEPS = "1"

I know why you did this, but after thinking about this, I'm worried and I'm not sure this is the right move.

The SDK dummy packages are "odd" but they are actual package files that are generated and would be present in manifests and as such, if they're missing from the SPDX manifests and docs, this is going to raise questions. I appreciate they don't have content and their main use is their 'provides'.

Taking a step back, we do support generic/multiple levels of package arch and since we support that, the SPDX code does need to handle that too. I'm therefore starting to worry that SPDX can't cope with the multiple package levels. Can you remind me what the issue is with the "all" arch here?

Cheers,

Richard

#82 lore raw

Re: [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX

Quoted 32 lines · Richard Purdie
Quoted 14 lines · Joshua Watt via lists.openembedded.org

The dummy SDK packages do not need SPDX support, and since they play some games with allarch that cause problems, it's simplest to disable their SPDX output.

---
 meta/recipes-core/meta/dummy-sdk-package.inc | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-core/meta/dummy-sdk-package.inc +0 -0
index bf453cac9b..71e788b0b9 100644

meta/recipes-core/meta/dummy-sdk-package.inc LICENSE = "MIT" · line 9

Quoted 6 lines
meta/recipes-core/meta/dummy-sdk-package.inc +1 -0
@@ -4,6 +4,7 @@ LICENSE = "MIT"
 PACKAGE_ARCH = "all"
 
 inherit allarch
+inherit nospdx
 
 INHIBIT_DEFAULT_DEPS = "1"

I know why you did this, but after thinking about this, I'm worried and I'm not sure this is the right move.

The SDK dummy packages are "odd" but they are actual package files that are generated and would be present in manifests and as such, if they're missing from the SPDX manifests and docs, this is going to raise questions. I appreciate they don't have content and their main use is their 'provides'.

Quoted 5 lines

Taking a step back, we do support generic/multiple levels of package arch and since we support that, the SPDX code does need to handle that too. I'm therefore starting to worry that SPDX can't cope with the multiple package levels. Can you remind me what the issue is with the "all" arch here?

The problem is specifically the anonymous python that does:

    d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')

Which seems specifically intended to bypass the allarch logic in a weird way. It's specifically doing this to put the package in a place no-one can find it (this includes SPDX); I didn't think it provided much value hence disabling it, but looking closer, I think we can do:

SSTATE_MULTILIB_SSTATE_ARCHS:append = " ${SDK_PACKAGE_ARCH}"

in create-spdx-sdk-3.0.bbclass and that should fix it. I'll try it.

Quoted 5 lines

Cheers,

Richard

#83 lore raw

Re: [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX

Quoted 44 lines · Joshua Watt
Quoted 32 lines · Richard Purdie
Quoted 16 lines · Joshua Watt via lists.openembedded.org

The dummy SDK packages do not need SPDX support, and since they play some games with allarch that cause problems, it's simplest to disable their SPDX output.

---
 meta/recipes-core/meta/dummy-sdk-package.inc | 1 +

 1 file changed, 1 insertion(+)
meta/recipes-core/meta/dummy-sdk-package.inc +0 -0
index bf453cac9b..71e788b0b9 100644
@@ -4,6 +4,7 @@ LICENSE = "MIT"

 PACKAGE_ARCH = "all"

Quoted 4 lines

 inherit allarch
+inherit nospdx

 INHIBIT_DEFAULT_DEPS = "1"

I know why you did this, but after thinking about this, I'm worried and I'm not sure this is the right move.

The SDK dummy packages are "odd" but they are actual package files that are generated and would be present in manifests and as such, if they're missing from the SPDX manifests and docs, this is going to raise questions. I appreciate they don't have content and their main use is their 'provides'.

Quoted 5 lines

Taking a step back, we do support generic/multiple levels of package arch and since we support that, the SPDX code does need to handle that too. I'm therefore starting to worry that SPDX can't cope with the multiple package levels. Can you remind me what the issue is with the "all" arch here?

The problem is specifically the anonymous python that does:

    d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')

Quoted 8 lines

Which seems specifically intended to bypass the allarch logic in a weird way. It's specifically doing this to put the package in a place no-one can find it (this includes SPDX); I didn't think it provided much value hence disabling it, but looking closer, I think we can do:

SSTATE_MULTILIB_SSTATE_ARCHS:append = " ${SDK_PACKAGE_ARCH}"

in create-spdx-sdk-3.0.bbclass and that should fix it. I'll try it.

That code was to put these packages into their own separate feed. Specifically:

meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb:DUMMYARCH = "buildtools-dummy-${SDKPKGSUFFIX}" meta/recipes-core/meta/nativesdk-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-${SDKPKGSUFFIX}"
meta/recipes-core/meta/dummy-sdk-package.inc: d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')
meta/recipes-core/meta/target-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-target"

so there are separate package feeds being created and then the SDK code can select which package feeds it wants to use for a given configuration/recipe.

That isn't to bypass allarch but just to customise the end package feed naming and allow different forms of SDK to be built by bringing in the different package combinations.

My worry is that the SPDX code isn't going to handle the scenario where PACKAGE_ARCH is changed (for example for a specialist graphics stack) and that one of our general features will be lost.

Cheers,

Richard

#84 lore raw

Re: [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX

Quoted 67 lines · Richard Purdie
Quoted 44 lines · Joshua Watt
Quoted 32 lines · Richard Purdie
Quoted 14 lines · Joshua Watt via lists.openembedded.org

The dummy SDK packages do not need SPDX support, and since they play some games with allarch that cause problems, it's simplest to disable their SPDX output.

---
 meta/recipes-core/meta/dummy-sdk-package.inc | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-core/meta/dummy-sdk-package.inc +0 -0
index bf453cac9b..71e788b0b9 100644

meta/recipes-core/meta/dummy-sdk-package.inc LICENSE = "MIT" · line 9

Quoted 6 lines
meta/recipes-core/meta/dummy-sdk-package.inc +1 -0
@@ -4,6 +4,7 @@ LICENSE = "MIT"
 PACKAGE_ARCH = "all"
 
 inherit allarch
+inherit nospdx
 
 INHIBIT_DEFAULT_DEPS = "1"

I know why you did this, but after thinking about this, I'm worried and I'm not sure this is the right move.

The SDK dummy packages are "odd" but they are actual package files that are generated and would be present in manifests and as such, if they're missing from the SPDX manifests and docs, this is going to raise questions. I appreciate they don't have content and their main use is their 'provides'.

Quoted 5 lines

Taking a step back, we do support generic/multiple levels of package arch and since we support that, the SPDX code does need to handle that too. I'm therefore starting to worry that SPDX can't cope with the multiple package levels. Can you remind me what the issue is with the "all" arch here?

The problem is specifically the anonymous python that does:

    d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')
Quoted 8 lines

Which seems specifically intended to bypass the allarch logic in a weird way. It's specifically doing this to put the package in a place no-one can find it (this includes SPDX); I didn't think it provided much value hence disabling it, but looking closer, I think we can do:

SSTATE_MULTILIB_SSTATE_ARCHS:append = " ${SDK_PACKAGE_ARCH}"

in create-spdx-sdk-3.0.bbclass and that should fix it. I'll try it.

That code was to put these packages into their own separate feed. Specifically:

meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb:DUMMYARCH = "buildtools-dummy-${SDKPKGSUFFIX}" meta/recipes-core/meta/nativesdk-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-${SDKPKGSUFFIX}"
meta/recipes-core/meta/dummy-sdk-package.inc: d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')
meta/recipes-core/meta/target-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-target"

so there are separate package feeds being created and then the SDK code can select which package feeds it wants to use for a given configuration/recipe.

Quoted 7 lines

That isn't to bypass allarch but just to customise the end package feed naming and allow different forms of SDK to be built by bringing in the different package combinations.

My worry is that the SPDX code isn't going to handle the scenario where PACKAGE_ARCH is changed (for example for a specialist graphics stack) and that one of our general features will be lost.

Right, SPDX is (supposed) to do whatever sstate does, so if it works for sstate it works for SPDX. The problem isn't that this code does the arch changing (specifically), it's that it changes the arch to an arbitrary one that isn't part of the normal package search order (on purpose), and then expects everyone to manually add in that arch when they need the package (which, SPDX was _not_ doing but all of the other SDK code obviously is). I think my proposed fix is thus correct and I will test it which should cover this corner case.

In general, I think most packages are handled correctly because again, SPDX does whatever sstate does, and SPDX even respects SSTATE_ARCHS which is how I think other recipes would deal with a similar situation. However, the SDKs are different specifically because they expect SDK_PACKAGE_ARCH to be respected by anything dealing with the sdk generation code, which is what was missing in SPDX.

Quoted 8 lines

Cheers,

Richard

#85 lore raw

Re: [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX

Quoted 87 lines · Joshua Watt
Quoted 67 lines · Richard Purdie
Quoted 44 lines · Joshua Watt
Quoted 32 lines · Richard Purdie
Quoted 14 lines · Joshua Watt via lists.openembedded.org

The dummy SDK packages do not need SPDX support, and since they play some games with allarch that cause problems, it's simplest to disable their SPDX output.

---
 meta/recipes-core/meta/dummy-sdk-package.inc | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-core/meta/dummy-sdk-package.inc +0 -0
index bf453cac9b..71e788b0b9 100644

meta/recipes-core/meta/dummy-sdk-package.inc LICENSE = "MIT" · line 9

Quoted 6 lines
meta/recipes-core/meta/dummy-sdk-package.inc +1 -0
@@ -4,6 +4,7 @@ LICENSE = "MIT"
 PACKAGE_ARCH = "all"
 
 inherit allarch
+inherit nospdx
 
 INHIBIT_DEFAULT_DEPS = "1"

I know why you did this, but after thinking about this, I'm worried and I'm not sure this is the right move.

The SDK dummy packages are "odd" but they are actual package files that are generated and would be present in manifests and as such, if they're missing from the SPDX manifests and docs, this is going to raise questions. I appreciate they don't have content and their main use is their 'provides'.

Quoted 5 lines

Taking a step back, we do support generic/multiple levels of package arch and since we support that, the SPDX code does need to handle that too. I'm therefore starting to worry that SPDX can't cope with the multiple package levels. Can you remind me what the issue is with the "all" arch here?

The problem is specifically the anonymous python that does:

    d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')
Quoted 8 lines

Which seems specifically intended to bypass the allarch logic in a weird way. It's specifically doing this to put the package in a place no-one can find it (this includes SPDX); I didn't think it provided much value hence disabling it, but looking closer, I think we can do:

SSTATE_MULTILIB_SSTATE_ARCHS:append = " ${SDK_PACKAGE_ARCH}"

in create-spdx-sdk-3.0.bbclass and that should fix it. I'll try it.

That code was to put these packages into their own separate feed. Specifically:

meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb:DUMMYARCH = "buildtools-dummy-${SDKPKGSUFFIX}" meta/recipes-core/meta/nativesdk-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-${SDKPKGSUFFIX}"
meta/recipes-core/meta/dummy-sdk-package.inc: d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')
meta/recipes-core/meta/target-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-target"

so there are separate package feeds being created and then the SDK code can select which package feeds it wants to use for a given configuration/recipe.

Quoted 7 lines

That isn't to bypass allarch but just to customise the end package feed naming and allow different forms of SDK to be built by bringing in the different package combinations.

My worry is that the SPDX code isn't going to handle the scenario where PACKAGE_ARCH is changed (for example for a specialist graphics stack) and that one of our general features will be lost.

Right, SPDX is (supposed) to do whatever sstate does, so if it works for sstate it works for SPDX. The problem isn't that this code does the arch changing (specifically), it's that it changes the arch to an arbitrary one that isn't part of the normal package search order (on purpose), and then expects everyone to manually add in that arch when they need the package (which, SPDX was _not_ doing but all of the other SDK code obviously is). I think my proposed fix is thus correct and I will test it which should cover this corner case.

Quoted 6 lines

In general, I think most packages are handled correctly because again, SPDX does whatever sstate does, and SPDX even respects SSTATE_ARCHS which is how I think other recipes would deal with a similar situation. However, the SDKs are different specifically because they expect SDK_PACKAGE_ARCH to be respected by anything dealing with the sdk generation code, which is what was missing in SPDX.

Hang on, I think I'm getting this confused with a different problem.....

Quoted 9 lines
Quoted 9 lines

Cheers,

Richard

#86 lore raw

Re: [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX

Quoted 89 lines · Joshua Watt
Quoted 87 lines · Joshua Watt
Quoted 67 lines · Richard Purdie
Quoted 44 lines · Joshua Watt
Quoted 32 lines · Richard Purdie
Quoted 14 lines · Joshua Watt via lists.openembedded.org

The dummy SDK packages do not need SPDX support, and since they play some games with allarch that cause problems, it's simplest to disable their SPDX output.

---
 meta/recipes-core/meta/dummy-sdk-package.inc | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-core/meta/dummy-sdk-package.inc +0 -0
index bf453cac9b..71e788b0b9 100644

meta/recipes-core/meta/dummy-sdk-package.inc LICENSE = "MIT" · line 9

Quoted 6 lines
meta/recipes-core/meta/dummy-sdk-package.inc +1 -0
@@ -4,6 +4,7 @@ LICENSE = "MIT"
 PACKAGE_ARCH = "all"
 
 inherit allarch
+inherit nospdx
 
 INHIBIT_DEFAULT_DEPS = "1"

I know why you did this, but after thinking about this, I'm worried and I'm not sure this is the right move.

The SDK dummy packages are "odd" but they are actual package files that are generated and would be present in manifests and as such, if they're missing from the SPDX manifests and docs, this is going to raise questions. I appreciate they don't have content and their main use is their 'provides'.

Quoted 5 lines

Taking a step back, we do support generic/multiple levels of package arch and since we support that, the SPDX code does need to handle that too. I'm therefore starting to worry that SPDX can't cope with the multiple package levels. Can you remind me what the issue is with the "all" arch here?

The problem is specifically the anonymous python that does:

    d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')
Quoted 8 lines

Which seems specifically intended to bypass the allarch logic in a weird way. It's specifically doing this to put the package in a place no-one can find it (this includes SPDX); I didn't think it provided much value hence disabling it, but looking closer, I think we can do:

SSTATE_MULTILIB_SSTATE_ARCHS:append = " ${SDK_PACKAGE_ARCH}"

in create-spdx-sdk-3.0.bbclass and that should fix it. I'll try it.

That code was to put these packages into their own separate feed. Specifically:

meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb:DUMMYARCH = "buildtools-dummy-${SDKPKGSUFFIX}" meta/recipes-core/meta/nativesdk-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-${SDKPKGSUFFIX}"
meta/recipes-core/meta/dummy-sdk-package.inc: d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')
meta/recipes-core/meta/target-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-target"

so there are separate package feeds being created and then the SDK code can select which package feeds it wants to use for a given configuration/recipe.

Quoted 7 lines

That isn't to bypass allarch but just to customise the end package feed naming and allow different forms of SDK to be built by bringing in the different package combinations.

My worry is that the SPDX code isn't going to handle the scenario where PACKAGE_ARCH is changed (for example for a specialist graphics stack) and that one of our general features will be lost.

Right, SPDX is (supposed) to do whatever sstate does, so if it works for sstate it works for SPDX. The problem isn't that this code does the arch changing (specifically), it's that it changes the arch to an arbitrary one that isn't part of the normal package search order (on purpose), and then expects everyone to manually add in that arch when they need the package (which, SPDX was _not_ doing but all of the other SDK code obviously is). I think my proposed fix is thus correct and I will test it which should cover this corner case.

Quoted 8 lines
Quoted 6 lines

In general, I think most packages are handled correctly because again, SPDX does whatever sstate does, and SPDX even respects SSTATE_ARCHS which is how I think other recipes would deal with a similar situation. However, the SDKs are different specifically because they expect SDK_PACKAGE_ARCH to be respected by anything dealing with the sdk generation code, which is what was missing in SPDX.

Hang on, I think I'm getting this confused with a different problem.....

Ok, I remember this one correctly now (who needs AI to hallucinate....). I think the correct thing to do here is to add:

do_create_recipe_spdx[vardeps] += "PACKAGE_ARCH"

because it's not rebuilding when this recipe changes PACKAGE_ARCH. The other SPDX tasks are because of an implicit dependency on PACKAGE_ARCH that re-triggers them. In the back of my mind I seem to recall we maybe didn't want to do this for some reason, but PACKAGE_ARCH isn't in BB_*_IGNORE_VARS so maybe I'm just mis-remembering.

Anyway, if this fix seems correct I'll make it.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

Cheers,

Richard

#87 lore raw

Re: [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX

Quoted 105 lines · Joshua Watt
Quoted 89 lines · Joshua Watt
Quoted 87 lines · Joshua Watt
Quoted 67 lines · Richard Purdie
Quoted 44 lines · Joshua Watt
Quoted 32 lines · Richard Purdie
Quoted 14 lines · Joshua Watt via lists.openembedded.org

The dummy SDK packages do not need SPDX support, and since they play some games with allarch that cause problems, it's simplest to disable their SPDX output.

---
 meta/recipes-core/meta/dummy-sdk-package.inc | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-core/meta/dummy-sdk-package.inc +0 -0
index bf453cac9b..71e788b0b9 100644

meta/recipes-core/meta/dummy-sdk-package.inc LICENSE = "MIT" · line 9

Quoted 6 lines
meta/recipes-core/meta/dummy-sdk-package.inc +1 -0
@@ -4,6 +4,7 @@ LICENSE = "MIT"
 PACKAGE_ARCH = "all"
 
 inherit allarch
+inherit nospdx
 
 INHIBIT_DEFAULT_DEPS = "1"

I know why you did this, but after thinking about this, I'm worried and I'm not sure this is the right move.

The SDK dummy packages are "odd" but they are actual package files that are generated and would be present in manifests and as such, if they're missing from the SPDX manifests and docs, this is going to raise questions. I appreciate they don't have content and their main use is their 'provides'.

Quoted 5 lines

Taking a step back, we do support generic/multiple levels of package arch and since we support that, the SPDX code does need to handle that too. I'm therefore starting to worry that SPDX can't cope with the multiple package levels. Can you remind me what the issue is with the "all" arch here?

The problem is specifically the anonymous python that does:

    d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')
Quoted 8 lines

Which seems specifically intended to bypass the allarch logic in a weird way. It's specifically doing this to put the package in a place no-one can find it (this includes SPDX); I didn't think it provided much value hence disabling it, but looking closer, I think we can do:

SSTATE_MULTILIB_SSTATE_ARCHS:append = " ${SDK_PACKAGE_ARCH}"

in create-spdx-sdk-3.0.bbclass and that should fix it. I'll try it.

That code was to put these packages into their own separate feed. Specifically:

meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb:DUMMYARCH = "buildtools-dummy-${SDKPKGSUFFIX}" meta/recipes-core/meta/nativesdk-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-${SDKPKGSUFFIX}"
meta/recipes-core/meta/dummy-sdk-package.inc: d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')
meta/recipes-core/meta/target-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-target"

so there are separate package feeds being created and then the SDK code can select which package feeds it wants to use for a given configuration/recipe.

Quoted 7 lines

That isn't to bypass allarch but just to customise the end package feed naming and allow different forms of SDK to be built by bringing in the different package combinations.

My worry is that the SPDX code isn't going to handle the scenario where PACKAGE_ARCH is changed (for example for a specialist graphics stack) and that one of our general features will be lost.

Right, SPDX is (supposed) to do whatever sstate does, so if it works for sstate it works for SPDX. The problem isn't that this code does the arch changing (specifically), it's that it changes the arch to an arbitrary one that isn't part of the normal package search order (on purpose), and then expects everyone to manually add in that arch when they need the package (which, SPDX was _not_ doing but all of the other SDK code obviously is). I think my proposed fix is thus correct and I will test it which should cover this corner case.

Quoted 8 lines
Quoted 6 lines

In general, I think most packages are handled correctly because again, SPDX does whatever sstate does, and SPDX even respects SSTATE_ARCHS which is how I think other recipes would deal with a similar situation. However, the SDKs are different specifically because they expect SDK_PACKAGE_ARCH to be respected by anything dealing with the sdk generation code, which is what was missing in SPDX.

Hang on, I think I'm getting this confused with a different problem.....

Ok, I remember this one correctly now (who needs AI to hallucinate....). I think the correct thing to do here is to add:

do_create_recipe_spdx[vardeps] += "PACKAGE_ARCH"

Quoted 7 lines

because it's not rebuilding when this recipe changes PACKAGE_ARCH. The other SPDX tasks are because of an implicit dependency on PACKAGE_ARCH that re-triggers them. In the back of my mind I seem to recall we maybe didn't want to do this for some reason, but PACKAGE_ARCH isn't in BB_*_IGNORE_VARS so maybe I'm just mis-remembering.

Anyway, if this fix seems correct I'll make it.

After trying this it doesn't work either. I'll write down my analysis of why and maybe we can figure it out:

First of all, the SPDX code writes out SPDX files based on their SSTATE_PKGARCH (which is why I say SPDX behaves like sstate). This was done back in 2023 [1]. SSTATE_PKGARCH is in BB_HASHEXCLUDE_COMMON, so changes in SSTATE_PKGARCH do not cause SPDX code to rebuild; I think this is intentional, but I'm not sure.

nativesdk-sdk-provides-dummy sets PACKAGE_ARCH = "all" and inherits allarch, but in sstate.bbclass, there is this:

elif bb.data.inherits_class('nativesdk', d):
d.setVar('SSTATE_PKGARCH', d.expand("${SDK_ARCH}_${SDK_OS}"))

So, SSTATE_PKGARCH is set to "${SDK_ARCH}_${SDK_OS}"

After this, the anon python in dummy-sdk-package.inc set PACKAGE_ARCH to ${DUMMYARCH}, which in this case is always "sdk-provides-dummy-nativesdk" -- FWIW, this really wants to look like the culprit but actually is irrelevant to the problem.

Now, the problem is that there is nothing to retrigger running do_create_recipe_spdx() when SDKMACHINE changes. This task (on purpose) has pretty minimal dependencies, but specifically nothing related to SDKMACHINE changing re-triggers this task. Thus consider the following command sequence which will trigger the bug (with a clean tmp dir):

SDKMACHINE=x86_64 bitbake nativesdk-sdk-provides-dummy && SDKMACHINE=i686 bitbake -fc create_spdx nativesdk-provided-dummy

In this case, the first command runs and puts the nativesdk-sdk-provides-dummy do_create_recipe_spdx files in "${SDK_ARCH}_${SDK_OS}" ("x86_64_linux"). The second command reruns do_create_spdx, but because SDKMACHINE doesn't factor into the taskhash of do_create_recipe_spdx, it does not rerun that task, assuming it's fine. Now, do_create_spdx cannot find the required SPDX file, because "x86_64_linux" is not in sstate search path (SSTATE_ARCHS) anymore.

Maybe allarch.bbclass and nativesdk.bbclass just don't interact well (esp. when sstate.bbclass appears to be making some assumptions about those two)? I did some digging and there are not many other recipes that inherit both allarch and nativesdk, however the other one I found was ca-certificates, and indeed it has the same error when building nativesdk-ca-certificates and switching SDKMACHINE.

I'm at a loss for where the actual bug is here. If it's true that SPDX should be using SSTATE_PKGARCH, I cannot see what it's doing wrong here (as I said, it more or less just copies what sstate does).

I'm kinda surprised we don't have other sstate problems with this recipe, given all of this, but maybe sstate isn't hard failing on missing files.

[1]: b2db10e966 ("create-spdx/sbom: Ensure files don't overlap between machines")

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines

Cheers,

Richard

#88 lore raw

PATCH v6 11/15 spdx: Remove fatal errors for missing providers

In reply to Joshua Watt

When creating images and SDKs, do not error on missing providers. This allows recipes to use the nospdx inherit to prevent SPDX from being generated, but not result in an error when assembling the final image.

Note that runtime packages generation already ignored missing providers, so this is changing image and SDK generation to match

---
 meta/classes/create-spdx-2.2.bbclass | 3 ++-
 meta/lib/oe/spdx30_tasks.py          | 8 +-------
 2 files changed, 3 insertions(+), 8 deletions(-)
meta/classes/create-spdx-2.2.bbclass +2 -1
index 3288cdf75a..aa39208eae 100644
@@ -858,7 +858,8 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx
     if packages:
         for name in sorted(packages.keys()):
             if name not in providers:
-                bb.fatal("Unable to find SPDX provider for '%s'" % name)
+                bb.note("Unable to find SPDX provider for '%s'" % name)
+                continue
 
             pkg_name, pkg_hashfn = providers[name]
 
meta/lib/oe/spdx30_tasks.py +1 -7
index 887fac813a..ba15d74278 100644
@@ -1251,11 +1251,10 @@ def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None)
     providers = oe.spdx_common.collect_package_providers(d)
 
     build_deps = set()
-    missing_providers = set()
 
     for name in sorted(packages.keys()):
         if name not in providers:
-            missing_providers.add(name)
+            bb.note(f"Unable to find SPDX provider for '{name}'")
             continue
 
         pkg_name, pkg_hashfn = providers[name]
@@ -1274,11 +1273,6 @@ def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None)
             for h, f in pkg_objset.by_sha256_hash.items():
                 files_by_hash.setdefault(h, set()).update(f)
 
-    if missing_providers:
-        bb.fatal(
-            f"Unable to find SPDX provider(s) for: {', '.join(sorted(missing_providers))}"
-        )
-
     if build_deps:
         objset.new_scoped_relationship(
             [build],
Signature · 1 line
2.53.0
#89 lore raw

PATCH v6 12/15 spdx3: Use common variable for vardeps

In reply to Joshua Watt

Instead of repeating the vardeps for each SPDX task with the necessary variables, use a common variable to make it easier to manage

---
 meta/classes/create-spdx-3.0.bbclass | 33 ++++++++++------------------
 1 file changed, 12 insertions(+), 21 deletions(-)
meta/classes/create-spdx-3.0.bbclass +12 -21
index 88b7ef9f42..6df66c193b 100644
@@ -174,6 +174,14 @@ SPDX3_DEP_FILES = "\
     ${SPDX_LICENSES}:True \
     "
 
+SPDX3_VAR_DEPS = "\
+    SPDX_INCLUDE_BITBAKE_PARENT_BUILD \
+    SPDX_PACKAGE_ADDITIONAL_PURPOSE \
+    SPDX_PROFILES \
+    SPDX_NAMESPACE_PREFIX \
+    SPDX_UUID_NAMESPACE \
+    "
+
 python do_create_recipe_spdx() {
     import oe.spdx30_tasks
     oe.spdx30_tasks.create_recipe_spdx(d)
@@ -185,13 +193,7 @@ do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}"
 do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}"
 do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}"
-do_create_recipe_spdx[vardeps] += "\
-    SPDX_INCLUDE_BITBAKE_PARENT_BUILD \
-    SPDX_PACKAGE_ADDITIONAL_PURPOSE \
-    SPDX_PROFILES \
-    SPDX_NAMESPACE_PREFIX \
-    SPDX_UUID_NAMESPACE \
-    "
+do_create_recipe_spdx[vardeps] += "${SPDX3_VAR_DEPS}"
 
 python do_create_recipe_spdx_setscene () {
     sstate_setscene(d)
@@ -222,13 +224,7 @@ do_create_spdx[depends] += " \
     ${PATCHDEPENDENCY} \
     ${@create_spdx_source_deps(d)} \
 "
-do_create_spdx[vardeps] += "\
-    SPDX_INCLUDE_BITBAKE_PARENT_BUILD \
-    SPDX_PACKAGE_ADDITIONAL_PURPOSE \
-    SPDX_PROFILES \
-    SPDX_NAMESPACE_PREFIX \
-    SPDX_UUID_NAMESPACE \
-    "
+do_create_spdx[vardeps] += "${SPDX3_VAR_DEPS}"
 
 python do_create_spdx_setscene () {
     sstate_setscene(d)
@@ -249,6 +245,7 @@ do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}"
 do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}"
 do_create_package_spdx[rdeptask] = "do_create_spdx"
+do_create_package_spdx[vardeps] += "${SPDX3_VAR_DEPS}"
 
 python do_create_package_spdx_setscene () {
     sstate_setscene(d)
@@ -270,13 +267,7 @@ do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}"
 do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
 do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}"
 do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}"
-do_create_recipe_sbom[vardeps] += "\
-    SPDX_INCLUDE_BITBAKE_PARENT_BUILD \
-    SPDX_PACKAGE_ADDITIONAL_PURPOSE \
-    SPDX_PROFILES \
-    SPDX_NAMESPACE_PREFIX \
-    SPDX_UUID_NAMESPACE \
-    "
+do_create_recipe_sbom[vardeps] += "${SPDX3_VAR_DEPS}"
 
 python do_create_recipe_sbom_setscene () {
     sstate_setscene(d)
Signature · 1 line
2.53.0
#90 lore raw

PATCH v6 13/15 glibc-testsuite: Do not generate SPDX

In reply to Joshua Watt

glibc-testsuite does not run on target or factor into the build supply chain, since its purpose is run tests in Qemu at build time

---
 meta/recipes-core/glibc/glibc-testsuite_2.42.bb | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-core/glibc/glibc-testsuite_2.42.bb +1 -0
index 6477612feb..1a83a09802 100644
@@ -32,6 +32,7 @@ do_check:append () {
 }
 
 inherit nopackages
+inherit nospdx
 deltask do_stash_locale
 deltask do_install
 deltask do_populate_sysroot
Signature · 1 line
2.53.0
#91 lore raw

PATCH v6 14/15 spdx: Remove do_collect_spdx_deps task

In reply to Joshua Watt

Removes the do_collect_spdx_deps task. This task was added a long time ago, and appears to have been added due to a misunderstanding about how the task graph works. It is not necessary since tasks can directly call collect_direct_deps() with the appropriate task that they depend on to get their dependencies.

This should fix several classes of SPDX bug where documents could not be found because the wrong deps were being looked for due to which tasks were re-run

---
 meta/classes-recipe/nospdx.bbclass   |  1 -
 meta/classes/create-spdx-2.2.bbclass | 22 +++++----
 meta/classes/create-spdx-3.0.bbclass |  5 +-
 meta/classes/spdx-common.bbclass     | 22 ---------
 meta/lib/oe/spdx30_tasks.py          | 22 +++++----
 meta/lib/oe/spdx_common.py           | 69 +++++++++++++---------------
 6 files changed, 62 insertions(+), 79 deletions(-)
meta/classes-recipe/nospdx.bbclass +0 -1
index 90e14442ba..7c99fcd1ec 100644
@@ -4,7 +4,6 @@
 # SPDX-License-Identifier: MIT
 #
 
-deltask do_collect_spdx_deps
 deltask do_create_recipe_spdx
 deltask do_create_spdx
 deltask do_create_spdx_runtime
meta/classes/create-spdx-2.2.bbclass +13 -9
index aa39208eae..1c43156559 100644
@@ -277,7 +277,7 @@ def add_package_sources_from_debug(d, package_doc, spdx_package, package, packag
 
 add_package_sources_from_debug[vardepsexclude] += "STAGING_KERNEL_DIR"
 
-def collect_dep_recipes(d, doc, spdx_recipe):
+def collect_dep_recipes(d, doc, spdx_recipe, direct_deps):
     import json
     from pathlib import Path
     import oe.sbom
@@ -290,9 +290,7 @@ def collect_dep_recipes(d, doc, spdx_recipe):
 
     dep_recipes = []
 
-    deps = oe.spdx_common.get_spdx_deps(d)
-
-    for dep in deps:
+    for dep in direct_deps:
         # If this dependency is not calculated in the taskhash skip it.
         # Otherwise, it can result in broken links since this task won't
         # rebuild and see the new SPDX ID if the dependency changes
@@ -405,7 +403,7 @@ do_create_recipe_spdx() {
     :
 }
 do_create_recipe_spdx[noexec] = "1"
-addtask do_create_recipe_spdx after do_collect_spdx_deps
+addtask do_create_recipe_spdx
 
 
 python do_create_spdx() {
@@ -532,7 +530,8 @@ python do_create_spdx() {
             if archive is not None:
                 recipe.packageFileName = str(recipe_archive.name)
 
-    dep_recipes = collect_dep_recipes(d, doc, recipe)
+    direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx")
+    dep_recipes = collect_dep_recipes(d, doc, recipe, direct_deps)
 
     doc_sha1 = oe.sbom.write_doc(d, doc, pkg_arch, "recipes", indent=get_json_indent(d))
     dep_recipes.append(oe.sbom.DepRecipe(doc, doc_sha1, recipe))
@@ -603,7 +602,7 @@ python do_create_spdx() {
 }
 do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS"
 # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source
-addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch do_collect_spdx_deps before do_populate_sdk do_build do_rm_work
+addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch before do_populate_sdk do_build do_rm_work
 
 SSTATETASKS += "do_create_spdx"
 do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}"
@@ -638,7 +637,9 @@ python do_create_runtime_spdx() {
 
     license_data = oe.spdx_common.load_spdx_license_data(d)
 
-    providers = oe.spdx_common.collect_package_providers(d)
+    direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx")
+
+    providers = oe.spdx_common.collect_package_providers(d, direct_deps)
     pkg_arch = d.getVar("SSTATE_PKGARCH")
     package_archs = d.getVar("SPDX_MULTILIB_SSTATE_ARCHS").split()
     package_archs.reverse()
@@ -760,6 +761,7 @@ addtask do_create_runtime_spdx_setscene
 
 do_create_runtime_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}"
 do_create_runtime_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}"
+do_create_runtime_spdx[deptask] = "do_create_spdx"
 do_create_runtime_spdx[rdeptask] = "do_create_spdx"
 
 do_rootfs[recrdeptask] += "do_create_spdx do_create_runtime_spdx"
@@ -829,7 +831,9 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx
 
     license_data = oe.spdx_common.load_spdx_license_data(d)
 
-    providers = oe.spdx_common.collect_package_providers(d)
+    direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx")
+
+    providers = oe.spdx_common.collect_package_providers(d, direct_deps)
     package_archs = d.getVar("SPDX_MULTILIB_SSTATE_ARCHS").split()
     package_archs.reverse()
 
meta/classes/create-spdx-3.0.bbclass +3 -2
index 6df66c193b..862e656b2c 100644
@@ -186,13 +186,14 @@ python do_create_recipe_spdx() {
     import oe.spdx30_tasks
     oe.spdx30_tasks.create_recipe_spdx(d)
 }
-addtask do_create_recipe_spdx after do_collect_spdx_deps
+addtask do_create_recipe_spdx
 
 SSTATETASKS += "do_create_recipe_spdx"
 do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}"
 do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}"
 do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}"
+do_create_recipe_spdx[deptask] += "do_create_recipe_spdx"
 do_create_recipe_spdx[vardeps] += "${SPDX3_VAR_DEPS}"
 
 python do_create_recipe_spdx_setscene () {
@@ -208,7 +209,6 @@ addtask do_create_spdx after \
     do_unpack \
     do_patch \
     do_create_recipe_spdx \
-    do_collect_spdx_deps \
     do_deploy_source_date_epoch \
     do_populate_sysroot do_package do_packagedata \
     before do_populate_sdk do_populate_sdk_ext do_build do_rm_work
@@ -244,6 +244,7 @@ do_create_package_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}"
 do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}"
 do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}"
+do_create_package_spdx[deptask] = "do_create_spdx"
 do_create_package_spdx[rdeptask] = "do_create_spdx"
 do_create_package_spdx[vardeps] += "${SPDX3_VAR_DEPS}"
 
meta/classes/spdx-common.bbclass +0 -22
index abf2332bee..4b40cbf75c 100644
@@ -87,28 +87,6 @@ def create_spdx_source_deps(d):
     return " ".join(deps)
 
 
-python do_collect_spdx_deps() {
-    # This task calculates the build time dependencies of the recipe, and is
-    # required because while a task can deptask on itself, those dependencies
-    # do not show up in BB_TASKDEPDATA. To work around that, this task does the
-    # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then
-    # downstream tasks read in the found dependencies when writing the actual
-    # SPDX document
-    import json
-    import oe.spdx_common
-    from pathlib import Path
-
-    spdx_deps_file = Path(d.getVar("SPDXDEPS"))
-
-    deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx")
-
-    with spdx_deps_file.open("w") as f:
-        json.dump(deps, f)
-}
-addtask do_collect_spdx_deps
-do_collect_spdx_deps[deptask] = "do_create_recipe_spdx"
-do_collect_spdx_deps[dirs] = "${SPDXDIR}"
-
 oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA"
 oe.spdx_common.collect_direct_deps[vardeps] += "DEPENDS"
 oe.spdx_common.collect_package_providers[vardepsexclude] += "BB_TASKDEPDATA"
meta/lib/oe/spdx30_tasks.py +14 -8
index ba15d74278..76f3338fb8 100644
@@ -298,13 +298,11 @@ def get_package_sources_from_debug(
     return dep_source_files
 
 
-def collect_dep_objsets(d, subdir, fn_prefix, obj_type, **attr_filter):
-    deps = oe.spdx_common.get_spdx_deps(d)
-
+def collect_dep_objsets(d, direct_deps, subdir, fn_prefix, obj_type, **attr_filter):
     dep_objsets = []
     dep_objs = set()
 
-    for dep in deps:
+    for dep in direct_deps:
         bb.debug(1, "Fetching SPDX for dependency %s" % (dep.pn))
         dep_obj, dep_objset = oe.sbom30.find_root_obj_in_jsonld(
             d, subdir, fn_prefix + dep.pn, obj_type, **attr_filter
@@ -551,8 +549,10 @@ def create_recipe_spdx(d):
             )
         )
 
+    direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx")
+
     dep_objsets, dep_recipes = collect_dep_objsets(
-        d, "static", "static-", oe.spdx30.software_Package
+        d, direct_deps, "static", "static-", oe.spdx30.software_Package
     )
 
     if dep_recipes:
@@ -753,8 +753,10 @@ def create_spdx(d):
         build_inputs |= files
         index_sources_by_hash(files, dep_sources)
 
+    direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx")
+
     dep_objsets, dep_builds = collect_dep_objsets(
-        d, "builds", "build-", oe.spdx30.build_Build
+        d, direct_deps, "builds", "build-", oe.spdx30.build_Build
     )
 
     if dep_builds:
@@ -1071,7 +1073,9 @@ def create_package_spdx(d):
     deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
     deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY"))
 
-    providers = oe.spdx_common.collect_package_providers(d)
+    direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx")
+
+    providers = oe.spdx_common.collect_package_providers(d, direct_deps)
     pkg_arch = d.getVar("SSTATE_PKGARCH")
 
     if get_is_native(d):
@@ -1248,7 +1252,9 @@ def write_bitbake_spdx(d):
 def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None):
     import oe.sbom30
 
-    providers = oe.spdx_common.collect_package_providers(d)
+    direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx")
+
+    providers = oe.spdx_common.collect_package_providers(d, direct_deps)
 
     build_deps = set()
 
meta/lib/oe/spdx_common.py +32 -37

168 lines not on this page · raw message

Signature · 1 line
2.53.0
#92 lore raw
In reply to Joshua Watt
Quoted 24 lines · Joshua Watt via lists.openembedded.org

Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.

Special thanks to Benjamin Robin <[email protected]> for helping work through this.

V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).

V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run).

V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.

Quoted 9 lines

V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway

V6: Fixes a bug where SPDX task would not correctly re-run when they change, which would cause errors about missing SPDX document. Also updates to the latest version of the SPDX bindings which improves performance

Hi Joshua,

Ok, we are almost there!

I suspect it would work fine on master, but we have a fail on two tests that were recently added by Stefano, and were not merged so far.

As both series might still evolve or get reviews, I will probably keep both in my branch, but some changes are needed if we want to merge both series.

2026-03-11 11:31:27,495 - oe-selftest - INFO - spdx.SPDX30Check.test_download_location_defensive_handling (subunit.RemotedTestCase)
2026-03-11 11:31:27,495 - oe-selftest - INFO -  ... FAIL
...
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 451, in test_download_location_defensive_handling
    objset = self.check_recipe_spdx(

File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, in check_recipe_spdx
return self.check_spdx_file(filename)
File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, in check_spdx_file
self.assertExists(filename)
File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists
raise self.failureException(msg)
AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1004290/tmp/deploy/spdx/3.0.1/cortexa57/recipes/recipe-m4.spdx.json' does not exist
...

2026-03-11 12:39:25,602 - oe-selftest - INFO - spdx.SPDX30Check.test_version_extraction_patterns (subunit.RemotedTestCase)
2026-03-11 12:39:25,603 - oe-selftest - INFO -  ... FAIL
...
2026-03-11 12:39:25,611 - oe-selftest - INFO - 6: 45/55 656/681 (14.27s) (2 failed) (spdx.SPDX30Check.test_version_extraction_patterns)
2026-03-11 12:39:25,611 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 479, in test_version_extraction_patterns
    objset = self.check_recipe_spdx(
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, in check_recipe_spdx
    return self.check_spdx_file(filename)
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, in check_spdx_file
    self.assertExists(filename)
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists
    raise self.failureException(msg)

AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1004290/tmp/deploy/spdx/3.0.1/cortexa57/recipes/recipe-tar.spdx.json' does not exist

https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3499
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3380
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3270

For reference, this oe-core branch was used during the build:
https://git.yoctoproject.org/poky-ci-archive/log/?h=oecore/autobuilder.yoctoproject.org/valkyrie/a-full-3385

Thanks,
Mathieu

Signature · 3 lines
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#93 lore raw
Quoted 38 lines · Mathieu Dubois-Briand
Quoted 24 lines · Joshua Watt via lists.openembedded.org

Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.

Special thanks to Benjamin Robin <[email protected]> for helping work through this.

V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).

V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run).

V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.

Quoted 9 lines

V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway

V6: Fixes a bug where SPDX task would not correctly re-run when they change, which would cause errors about missing SPDX document. Also updates to the latest version of the SPDX bindings which improves performance

Hi Joshua,

Quoted 8 lines

Ok, we are almost there!

I suspect it would work fine on master, but we have a fail on two tests that were recently added by Stefano, and were not merged so far.

As both series might still evolve or get reviews, I will probably keep both in my branch, but some changes are needed if we want to merge both series.

This is actually semi-intentional. I renamed the "recipe-" SPDX files to "build-". The fix is simple, but it either needs to be applied to my changes or Stephanos, depending on the order. Do you have a preference?

Quoted 44 lines
2026-03-11 11:31:27,495 - oe-selftest - INFO - spdx.SPDX30Check.test_download_location_defensive_handling (subunit.RemotedTestCase)
2026-03-11 11:31:27,495 - oe-selftest - INFO -  ... FAIL
...
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 451, in test_download_location_defensive_handling
    objset = self.check_recipe_spdx(

File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, in check_recipe_spdx
return self.check_spdx_file(filename)
File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, in check_spdx_file
self.assertExists(filename)
File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists
raise self.failureException(msg)
AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1004290/tmp/deploy/spdx/3.0.1/cortexa57/recipes/recipe-m4.spdx.json' does not exist
...

2026-03-11 12:39:25,602 - oe-selftest - INFO - spdx.SPDX30Check.test_version_extraction_patterns (subunit.RemotedTestCase)
2026-03-11 12:39:25,603 - oe-selftest - INFO -  ... FAIL
...
2026-03-11 12:39:25,611 - oe-selftest - INFO - 6: 45/55 656/681 (14.27s) (2 failed) (spdx.SPDX30Check.test_version_extraction_patterns)
2026-03-11 12:39:25,611 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 479, in test_version_extraction_patterns
    objset = self.check_recipe_spdx(
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, in check_recipe_spdx
    return self.check_spdx_file(filename)
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, in check_spdx_file
    self.assertExists(filename)
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists
    raise self.failureException(msg)

AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1004290/tmp/deploy/spdx/3.0.1/cortexa57/recipes/recipe-tar.spdx.json' does not exist

https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3499
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3380
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3270

For reference, this oe-core branch was used during the build:
https://git.yoctoproject.org/poky-ci-archive/log/?h=oecore/autobuilder.yoctoproject.org/valkyrie/a-full-3385

Thanks,
Mathieu

--
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

#94 lore raw
Quoted 45 lines · Joshua Watt
Quoted 43 lines · Mathieu Dubois-Briand
Quoted 24 lines · Joshua Watt via lists.openembedded.org

Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.

Special thanks to Benjamin Robin <[email protected]> for helping work through this.

V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).

V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run).

V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.

Quoted 9 lines

V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway

V6: Fixes a bug where SPDX task would not correctly re-run when they change, which would cause errors about missing SPDX document. Also updates to the latest version of the SPDX bindings which improves performance

Hi Joshua,

Ok, we are almost there!

I suspect it would work fine on master, but we have a fail on two tests that were recently added by Stefano, and were not merged so far.

Quoted 9 lines
Quoted 3 lines

As both series might still evolve or get reviews, I will probably keep both in my branch, but some changes are needed if we want to merge both series.

This is actually semi-intentional. I renamed the "recipe-" SPDX files to "build-". The fix is simple, but it either needs to be applied to my changes or Stephanos, depending on the order. Do you have a preference?

I do not have any particular preference, so please do as is easier for you.

Signature · 3 lines
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#95 lore raw
Quoted 51 lines · Mathieu Dubois-Briand
Quoted 49 lines · Joshua Watt
Quoted 46 lines · Mathieu Dubois-Briand
Quoted 24 lines · Joshua Watt via lists.openembedded.org

Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.

Special thanks to Benjamin Robin <[email protected]> for helping work through this.

V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).

V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run).

V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.

Quoted 9 lines

V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway

V6: Fixes a bug where SPDX task would not correctly re-run when they change, which would cause errors about missing SPDX document. Also updates to the latest version of the SPDX bindings which improves performance

Hi Joshua,

Ok, we are almost there!

I suspect it would work fine on master, but we have a fail on two tests that were recently added by Stefano, and were not merged so far.

As both series might still evolve or get reviews, I will probably keep both in my branch, but some changes are needed if we want to merge both series.

Quoted 8 lines
Quoted 5 lines

This is actually semi-intentional. I renamed the "recipe-" SPDX files to "build-". The fix is simple, but it either needs to be applied to my changes or Stephanos, depending on the order. Do you have a preference?

I do not have any particular preference, so please do as is easier for you.

Stefano has some feedback to address, so please take my patch series first and Stefano can fix it in the next revision.

Thanks

Quoted 6 lines

--
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

#96 lore raw

v7 cover Add SPDX 3 Recipe Information

In reply to Joshua Watt

Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.

Special thanks to Benjamin Robin <[email protected]> for helping work through this.

V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).

V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run).

V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.

V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway

V6: Fixes a bug where SPDX task would not correctly re-run when they change, which would cause errors about missing SPDX document. Also updates to the latest version of the SPDX bindings which improves performance

V7: Makes meta-world-recipe-sbom create the world SBoM when run with a task (e.g. as part of do_build). Drops the removal of SPDX from dummy-sdk-packages as these have been fixed to work properly.

Joshua Watt (12):
spdx3: Add recipe SPDX data
spdx3: Add recipe SBoM task
spdx3: Add is-native property
spdx30: Include patch file information in VEX
spdx: De-duplicate CreationInfo
spdx_common: Check for dependent task in task flags
spdx30: Remove package VEX
spdx: Remove fatal errors for missing providers
spdx3: Use common variable for vardeps
glibc-testsuite: Do not generate SPDX
spdx: Remove do_collect_spdx_deps task
spdx: Update to latest bindings

 meta/classes-global/sstate.bbclass            |    4 +-
 .../create-spdx-image-3.0.bbclass             |    4 +-
 .../create-spdx-sdk-3.0.bbclass               |    4 +-
 meta/classes-recipe/kernel.bbclass            |    2 +-
 meta/classes-recipe/nospdx.bbclass            |    2 +-
 meta/classes/create-spdx-2.2.bbclass          |   33 +-
 meta/classes/create-spdx-3.0.bbclass          |   81 +-
 meta/classes/spdx-common.bbclass              |   34 +-
 meta/conf/distro/include/maintainers.inc      |    1 +
 meta/lib/oe/sbom30.py                         |  239 +-
 meta/lib/oe/spdx30/__init__.py                |    8 +
 meta/lib/oe/spdx30/__main__.py                |   12 +
 meta/lib/oe/spdx30/cmd.py                     |   75 +
 meta/lib/oe/{spdx30.py => spdx30/model.py}    | 5935 ++++++++++-------
 meta/lib/oe/spdx30/stub.pyi                   | 2544 +++++++
 meta/lib/oe/spdx30_tasks.py                   |  459 +-
 meta/lib/oe/spdx_common.py                    |   78 +-
 meta/lib/oeqa/selftest/cases/spdx.py          |   28 +-
 .../glibc/glibc-testsuite_2.43.bb             |    1 +
 .../meta/meta-world-recipe-sbom.bb            |   38 +
 scripts/contrib/make-spdx-bindings.sh         |    3 +-
 21 files changed, 6837 insertions(+), 2748 deletions(-)
 create mode 100644 meta/lib/oe/spdx30/__init__.py
 create mode 100644 meta/lib/oe/spdx30/__main__.py
 create mode 100644 meta/lib/oe/spdx30/cmd.py
 rename meta/lib/oe/{spdx30.py => spdx30/model.py} (52%)
 create mode 100644 meta/lib/oe/spdx30/stub.pyi
 create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
Signature · 1 line
2.53.0
#97 lore raw

PATCH v7 1/12 spdx3: Add recipe SPDX data

Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.

Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch

---
 meta/classes-global/sstate.bbclass            |   4 +-
 .../create-spdx-image-3.0.bbclass             |   4 +-
 .../create-spdx-sdk-3.0.bbclass               |   4 +-
 meta/classes-recipe/kernel.bbclass            |   2 +-
 meta/classes-recipe/nospdx.bbclass            |   1 +
 meta/classes/create-spdx-2.2.bbclass          |  12 +-
 meta/classes/create-spdx-3.0.bbclass          |  51 ++-
 meta/classes/spdx-common.bbclass              |  21 +-
 meta/lib/oe/spdx30_tasks.py                   | 402 ++++++++++++------
 meta/lib/oeqa/selftest/cases/spdx.py          |  19 +-
 10 files changed, 354 insertions(+), 166 deletions(-)
meta/classes-global/sstate.bbclass +2 -2
index 1c3df0f544..88449d19c7 100644
@@ -945,7 +945,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True,
             extrapath = d.getVar("NATIVELSBSTRING") + "/"
         else:
             extrapath = ""
-        
+
         tname = bb.runqueue.taskname_from_tid(task)[3:]
 
         if tname in ["fetch", "unpack", "patch", "populate_lic", "preconfigure"] and splithashfn[2]:
@@ -1107,7 +1107,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None):
 
     logit("Considering setscene task: %s" % (str(taskdependees[task])), log)
 
-    directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_deploy_archives"]
+    directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_create_recipe_spdx", "do_deploy_archives"]
 
     def isNativeCross(x):
         return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross")
meta/classes-recipe/create-spdx-image-3.0.bbclass +2 -2
index 636ab14eb0..15a91e90e2 100644
@@ -34,7 +34,7 @@ addtask do_create_rootfs_spdx after do_rootfs before do_image
 SSTATETASKS += "do_create_rootfs_spdx"
 do_create_rootfs_spdx[sstate-inputdirs] = "${SPDXROOTFSDEPLOY}"
 do_create_rootfs_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}"
-do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_create_rootfs_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_create_rootfs_spdx[cleandirs] += "${SPDXROOTFSDEPLOY}"
 do_create_rootfs_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 
@@ -76,7 +76,7 @@ do_create_image_sbom_spdx[sstate-inputdirs] = "${SPDXIMAGEDEPLOYDIR}"
 do_create_image_sbom_spdx[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
 do_create_image_sbom_spdx[stamp-extra-info] = "${MACHINE_ARCH}"
 do_create_image_sbom_spdx[cleandirs] = "${SPDXIMAGEDEPLOYDIR}"
-do_create_image_sbom_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_create_image_sbom_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_create_image_sbom_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 
 python do_create_image_sbom_spdx_setscene() {
meta/classes-recipe/create-spdx-sdk-3.0.bbclass +2 -2
index e5f220cdfa..a4b8ed3bf9 100644
@@ -5,14 +5,14 @@
 #
 # SPDX SDK tasks
 
-do_populate_sdk[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_populate_sdk[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_populate_sdk[cleandirs] += "${SPDXSDKWORK}"
 do_populate_sdk[postfuncs] += "sdk_create_sbom"
 do_populate_sdk[file-checksums] += "${SPDX3_DEP_FILES}"
 POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_create_spdx"
 POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_create_spdx"
 
-do_populate_sdk_ext[recrdeptask] += "do_create_spdx do_create_package_spdx"
+do_populate_sdk_ext[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx"
 do_populate_sdk_ext[cleandirs] += "${SPDXSDKEXTWORK}"
 do_populate_sdk_ext[postfuncs] += "sdk_ext_create_sbom"
 do_populate_sdk_ext[file-checksums] += "${SPDX3_DEP_FILES}"
meta/classes-recipe/kernel.bbclass +1 -1
index 094c1148b6..2d8565bd55 100644
@@ -883,7 +883,7 @@ do_create_spdx:append() {
         except Exception as e:
             bb.error(f"Failed to parse kernel config file: {e}")
 
-        path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "recipes", f"recipe-{pn}", deploydir=deploydir)
+        path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "builds", f"build-{pn}", deploydir=deploydir)
         build_objset = oe.sbom30.load_jsonld(d, path, required=True)
         build = build_objset.find_root(oe.spdx30.build_Build)
         if not build:
meta/classes-recipe/nospdx.bbclass +1 -0
index b20e28218b..90e14442ba 100644
@@ -5,6 +5,7 @@
 #
 
 deltask do_collect_spdx_deps
+deltask do_create_recipe_spdx
 deltask do_create_spdx
 deltask do_create_spdx_runtime
 deltask do_create_package_spdx
meta/classes/create-spdx-2.2.bbclass +11 -1
index 65d10d86db..3288cdf75a 100644
@@ -399,6 +399,15 @@ def get_license_list_version(license_data, d):
     return ".".join(license_data["licenseListVersion"].split(".")[:2])
 
 
+# This task is added for compatibility with tasks shared with SPDX 3, but
+# doesn't do anything
+do_create_recipe_spdx() {
+    :
+}
+do_create_recipe_spdx[noexec] = "1"
+addtask do_create_recipe_spdx after do_collect_spdx_deps
+
+
 python do_create_spdx() {
     from datetime import datetime, timezone
     import oe.sbom
@@ -594,7 +603,7 @@ python do_create_spdx() {
 }
 do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS"
 # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source
-addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work
+addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch do_collect_spdx_deps before do_populate_sdk do_build do_rm_work
 
 SSTATETASKS += "do_create_spdx"
 do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}"
@@ -605,6 +614,7 @@ python do_create_spdx_setscene () {
 }
 addtask do_create_spdx_setscene
 
+do_create_spdx[deptask] += "do_create_spdx"
 do_create_spdx[dirs] = "${SPDXWORK}"
 do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}"
 do_create_spdx[depends] += " \
meta/classes/create-spdx-3.0.bbclass +38 -13

90 lines not on this page · raw message

meta/classes/spdx-common.bbclass +6 -15
index 3110230c9e..3c239a718b 100644
@@ -23,6 +23,7 @@ SPDXDEPS = "${SPDXDIR}/deps.json"
 SPDX_TOOL_NAME ??= "oe-spdx-creator"
 SPDX_TOOL_VERSION ??= "1.0"
 
+SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy"
 SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy"
 
 SPDX_INCLUDE_SOURCES ??= "0"
@@ -67,12 +68,6 @@ def create_spdx_source_deps(d):
     deps = []
     if d.getVar("SPDX_INCLUDE_SOURCES") == "1":
         pn = d.getVar('PN')
-        # do_unpack is a hack for now; we only need it to get the
-        # dependencies do_unpack already has so we can extract the source
-        # ourselves
-        if oe.spdx_common.has_task(d, "do_unpack"):
-            deps.append("%s:do_unpack" % pn)
-
         if oe.spdx_common.is_work_shared_spdx(d) and \
            oe.spdx_common.process_sources(d):
             # For kernel source code
@@ -84,8 +79,6 @@ def create_spdx_source_deps(d):
             # For gcc-source-${PV} source code
             if oe.spdx_common.has_task(d, "do_preconfigure"):
                 deps.append("%s:do_preconfigure" % pn)
-            elif oe.spdx_common.has_task(d, "do_patch"):
-                deps.append("%s:do_patch" % pn)
             # For gcc-cross-x86_64 source code
             elif oe.spdx_common.has_task(d, "do_configure"):
                 deps.append("%s:do_configure" % pn)
@@ -97,8 +90,8 @@ python do_collect_spdx_deps() {
     # This task calculates the build time dependencies of the recipe, and is
     # required because while a task can deptask on itself, those dependencies
     # do not show up in BB_TASKDEPDATA. To work around that, this task does the
-    # deptask on do_create_spdx and writes out the dependencies it finds, then
-    # do_create_spdx reads in the found dependencies when writing the actual
+    # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then
+    # downstream tasks read in the found dependencies when writing the actual
     # SPDX document
     import json
     import oe.spdx_common
@@ -106,15 +99,13 @@ python do_collect_spdx_deps() {
 
     spdx_deps_file = Path(d.getVar("SPDXDEPS"))
 
-    deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx")
+    deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx")
 
     with spdx_deps_file.open("w") as f:
         json.dump(deps, f)
 }
-# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source
-addtask do_collect_spdx_deps after do_unpack
-do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}"
-do_collect_spdx_deps[deptask] = "do_create_spdx"
+addtask do_collect_spdx_deps
+do_collect_spdx_deps[deptask] = "do_create_recipe_spdx"
 do_collect_spdx_deps[dirs] = "${SPDXDIR}"
 
 oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA"
meta/lib/oe/spdx30_tasks.py +279 -123

559 lines not on this page · raw message

meta/lib/oeqa/selftest/cases/spdx.py +12 -7
index 5830d7c087..759ca86b73 100644
@@ -141,6 +141,11 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
     SPDX_CLASS = "create-spdx-3.0"
 
     def test_base_files(self):
+        self.check_recipe_spdx(
+            "base-files",
+            "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/static/static-base-files.spdx.json",
+            task="create_recipe_spdx",
+        )
         self.check_recipe_spdx(
             "base-files",
             "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json",
@@ -149,7 +154,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
     def test_gcc_include_source(self):
         objset = self.check_recipe_spdx(
             "gcc",
-            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-gcc.spdx.json",
+            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-gcc.spdx.json",
             extraconf="""\
                 SPDX_INCLUDE_SOURCES = "1"
                 """,
@@ -162,12 +167,12 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
             if software_file.name == filename:
                 found = True
                 self.logger.info(
-                    f"The spdxId of {filename} in recipe-gcc.spdx.json is {software_file.spdxId}"
+                    f"The spdxId of {filename} in build-gcc.spdx.json is {software_file.spdxId}"
                 )
                 break
 
         self.assertTrue(
-            found, f"Not found source file {filename} in recipe-gcc.spdx.json\n"
+            found, f"Not found source file {filename} in build-gcc.spdx.json\n"
         )
 
     def test_core_image_minimal(self):
@@ -305,7 +310,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
         # This will fail with NameError if new_annotation() is called incorrectly
         objset = self.check_recipe_spdx(
             "base-files",
-            "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/recipes/recipe-base-files.spdx.json",
+            "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/builds/build-base-files.spdx.json",
             extraconf=textwrap.dedent(
                 f"""\
                 ANNOTATION1 = "{ANNOTATION_VAR1}"
@@ -360,8 +365,8 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
 
     def test_kernel_config_spdx(self):
         kernel_recipe = get_bb_var("PREFERRED_PROVIDER_virtual/kernel")
-        spdx_file = f"recipe-{kernel_recipe}.spdx.json"
-        spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/recipes/{spdx_file}"
+        spdx_file = f"build-{kernel_recipe}.spdx.json"
+        spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/builds/{spdx_file}"
 
         # Make sure kernel is configured first
         bitbake(f"-c configure {kernel_recipe}")
@@ -392,7 +397,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
     def test_packageconfig_spdx(self):
         objset = self.check_recipe_spdx(
             "tar",
-            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-tar.spdx.json",
+            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-tar.spdx.json",
             extraconf="""\
                 SPDX_INCLUDE_PACKAGECONFIG = "1"
                 """,
Signature · 1 line
2.53.0
#98 lore raw

PATCH v7 2/12 spdx3: Add recipe SBoM task

In reply to Joshua Watt

Adds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example:

```
bitbake -c create_recipe_sbom zstd
```

Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively).

The complete SBoM for all (target) recipes can be built with:

```
bitbake meta-world-recipe-sbom
```
---
 meta/classes/create-spdx-3.0.bbclass          | 32 ++++++++++++++++
 meta/classes/spdx-common.bbclass              |  1 +
 meta/conf/distro/include/maintainers.inc      |  1 +
 meta/lib/oe/spdx30_tasks.py                   | 10 +++++
 meta/lib/oeqa/selftest/cases/spdx.py          |  9 +++++
 .../meta/meta-world-recipe-sbom.bb            | 38 +++++++++++++++++++
 6 files changed, 91 insertions(+)
 create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
meta/classes/create-spdx-3.0.bbclass +32 -0
index 672ca27cd0..c3ea95b8bc 100644
@@ -142,6 +142,10 @@ SPDX_PACKAGE_URLS[doc] = "A space separated list of Package URLs (purls) for \
     Override this variable to replace the default, otherwise append or prepend \
     to add additional purls."
 
+SPDX_RECIPE_SBOM_NAME ?= "${PN}-recipe-sbom"
+SPDX_RECIPE_SBOM_NAME[doc] = "The name of output recipe SBoM when using \
+    create_recipe_sbom"
+
 IMAGE_CLASSES:append = " create-spdx-image-3.0"
 SDK_CLASSES += "create-spdx-sdk-3.0"
 
@@ -240,6 +244,34 @@ python do_create_package_spdx_setscene () {
 }
 addtask do_create_package_spdx_setscene
 
+addtask do_create_recipe_sbom after create_recipe_spdx
+python do_create_recipe_sbom() {
+    import oe.spdx30_tasks
+    from pathlib import Path
+    deploydir = Path(d.getVar("SPDXRECIPESBOMDEPLOY"))
+    oe.spdx30_tasks.create_recipe_sbom(d, deploydir)
+}
+
+SSTATETASKS += "do_create_recipe_sbom"
+do_create_recipe_sbom[recrdeptask] = "do_create_recipe_spdx"
+do_create_recipe_sbom[nostamp] = "1"
+do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}"
+do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
+do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}"
+do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}"
+do_create_recipe_sbom[vardeps] += "\
+    SPDX_INCLUDE_BITBAKE_PARENT_BUILD \
+    SPDX_PACKAGE_ADDITIONAL_PURPOSE \
+    SPDX_PROFILES \
+    SPDX_NAMESPACE_PREFIX \
+    SPDX_UUID_NAMESPACE \
+    "
+
+python do_create_recipe_sbom_setscene () {
+    sstate_setscene(d)
+}
+addtask do_create_recipe_sbom_setscene
+
 python spdx30_build_started_handler () {
     import oe.spdx30_tasks
     d = e.data.createCopy()
meta/classes/spdx-common.bbclass +1 -0
index 3c239a718b..abf2332bee 100644
@@ -25,6 +25,7 @@ SPDX_TOOL_VERSION ??= "1.0"
 
 SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy"
 SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy"
+SPDXRECIPESBOMDEPLOY = "${SPDXDIR}/recipes-bom-deploy"
 
 SPDX_INCLUDE_SOURCES ??= "0"
 SPDX_INCLUDE_COMPILED_SOURCES ??= "0"
meta/conf/distro/include/maintainers.inc +1 -0
index c7a646a643..86a11867a1 100644
@@ -534,6 +534,7 @@ RECIPE_MAINTAINER:pn-meta-go-toolchain = "Richard Purdie <richard.purdie@linuxfo
 RECIPE_MAINTAINER:pn-meta-ide-support = "Richard Purdie <[email protected]>"
 RECIPE_MAINTAINER:pn-meta-toolchain = "Richard Purdie <[email protected]>"
 RECIPE_MAINTAINER:pn-meta-world-pkgdata = "Richard Purdie <[email protected]>"
+RECIPE_MAINTAINER:pn-meta-world-recipe-sbom = "Joshua Watt <[email protected]>"
 RECIPE_MAINTAINER:pn-mingetty = "Yi Zhao <[email protected]>"
 RECIPE_MAINTAINER:pn-mini-x-session = "Unassigned <[email protected]>"
 RECIPE_MAINTAINER:pn-minicom = "Unassigned <[email protected]>"
meta/lib/oe/spdx30_tasks.py +10 -0
index a8b4525e3d..b6c917045e 100644
@@ -1564,3 +1564,13 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname):
     oe.sbom30.write_jsonld_doc(
         d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json")
     )
+
+
+def create_recipe_sbom(d, deploydir):
+    sbom_name = d.getVar("SPDX_RECIPE_SBOM_NAME")
+
+    recipe, recipe_objset = load_recipe_spdx(d)
+
+    objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset])
+
+    oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json"))
meta/lib/oeqa/selftest/cases/spdx.py +9 -0
index 759ca86b73..af1144c1e5 100644
@@ -151,6 +151,15 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
             "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json",
         )
 
+    def test_world_sbom(self):
+        objset = self.check_recipe_spdx(
+            "meta-world-recipe-sbom",
+            "{DEPLOY_DIR_IMAGE}/world-recipe-sbom.spdx.json",
+        )
+
+        # Document should be fully linked
+        self.check_objset_missing_ids(objset)
+
     def test_gcc_include_source(self):
         objset = self.check_recipe_spdx(
             "gcc",
meta/recipes-core/meta/meta-world-recipe-sbom.bb new file +38 -0
new file mode 100644
index 0000000000..841fb8ea16
@@ -0,0 +1,38 @@
+SUMMARY = "Generates a combined SBoM for all world recipes"
+LICENSE = "MIT"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+PACKAGE_ARCH = "${MACHINE_ARCH}"
+
+inherit nopackages
+deltask do_fetch
+deltask do_unpack
+deltask do_patch
+deltask do_configure
+deltask do_compile
+deltask do_install
+
+do_prepare_recipe_sysroot[deptask] = ""
+
+WORLD_SBOM_EXCLUDE ?= ""
+
+EXCLUDE_FROM_WORLD = "1"
+SPDX_RECIPE_SBOM_NAME = "world-recipe-sbom"
+
+python calculate_extra_depends() {
+    exclude = set('${WORLD_SBOM_EXCLUDE}'.split())
+    exclude |= set(f"{v}-{self_pn}" for v in '${MULTILIB_VARIANTS}'.split())
+    exclude.add(self_pn)
+
+    deps.extend(p for p in world_target if p not in exclude)
+}
+
+python() {
+    # Ensure that do_create_recipe_sbom is the only dependency of do_build,
+    # since the sole purpose of this recipe is to produce the world recipe SBoM
+    d.setVarFlag("do_build", "deps", ["do_create_recipe_sbom"])
+    d.setVarFlag("do_build", "deptask", "")
+    d.setVarFlag("do_build", "rdeptask", "")
+    d.setVarFlag("do_build", "recrdeptask", "")
+}
Signature · 1 line
2.53.0
#99 lore raw

PATCH v7 3/12 spdx3: Add is-native property

In reply to Joshua Watt

Adds a custom is-native property to the recipe package to indicate if it is a native recipe

---
 meta/lib/oe/sbom30.py       | 20 ++++++++++++++++++++
 meta/lib/oe/spdx30_tasks.py | 18 +++++++++++-------
 2 files changed, 31 insertions(+), 7 deletions(-)
meta/lib/oe/sbom30.py +20 -0
index 227ac51877..50a72fce39 100644
@@ -118,6 +118,26 @@ class OEDocumentExtension(oe.spdx30.extension_Extension):
         )
 
 
[email protected](OE_SPDX_BASE + "recipe-extension")
+class OERecipeExtension(oe.spdx30.extension_Extension):
+    """
+    This extension is added to recipe software_Packages to indicate various
+    useful bits of information about the recipe
+    """
+
+    CLOSED = True
+
+    @classmethod
+    def _register_props(cls):
+        super()._register_props()
+        cls._add_property(
+            "is_native",
+            oe.spdx30.BooleanProp(),
+            OE_SPDX_BASE + "is-native",
+            max_count=1,
+        )
+
+
 def spdxid_hash(*items):
     h = hashlib.md5()
     for i in items:
meta/lib/oe/spdx30_tasks.py +11 -7
index b6c917045e..a8fffbb085 100644
@@ -477,6 +477,10 @@ def set_purls(spdx_package, purls):
         )
 
 
+def get_is_native(d):
+    return bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d)
+
+
 def create_recipe_spdx(d):
     deploydir = Path(d.getVar("SPDXRECIPEDEPLOY"))
     deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
@@ -507,6 +511,11 @@ def create_recipe_spdx(d):
         )
     )
 
+    if get_is_native(d):
+        ext = oe.sbom30.OERecipeExtension()
+        ext.is_native = True
+        recipe.extension.append(ext)
+
     set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split())
 
     # TODO: This doesn't work before do_unpack because the license text has to
@@ -668,9 +677,7 @@ def create_spdx(d):
     spdx_workdir = Path(d.getVar("SPDXWORK"))
     include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1"
     pkg_arch = d.getVar("SSTATE_PKGARCH")
-    is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class(
-        "cross", d
-    )
+    is_native = get_is_native(d)
 
     recipe, recipe_objset = load_recipe_spdx(d)
 
@@ -1021,14 +1028,11 @@ def create_spdx(d):
 def create_package_spdx(d):
     deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
     deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY"))
-    is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class(
-        "cross", d
-    )
 
     providers = oe.spdx_common.collect_package_providers(d)
     pkg_arch = d.getVar("SSTATE_PKGARCH")
 
-    if is_native:
+    if get_is_native(d):
         return
 
     bb.build.exec_func("read_subpackage_metadata", d)
Signature · 1 line
2.53.0
#100 lore raw

PATCH v7 4/12 spdx30: Include patch file information in VEX

In reply to Joshua Watt

Modifies the SPDX VEX output to include the patches that fix a particular vulnerability. This is done by adding a patchedBy relationship from the VexFixedVulnAssessmentRelationship to the File that provides the fix.

If the file can be located without fetching (e.g. is a file:// in SRC_URI), the checksum will be included.

---
 meta/lib/oe/sbom30.py       | 60 ++++++++++++++-------------
 meta/lib/oe/spdx30_tasks.py | 81 ++++++++++++++++++++++++++++---------
 2 files changed, 92 insertions(+), 49 deletions(-)
meta/lib/oe/sbom30.py +31 -29
index 50a72fce39..21f084dc16 100644
@@ -620,37 +620,38 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
         )
         spdx_file.extension.append(OELicenseScannedExtension())
 
-    def new_file(self, _id, name, path, *, purposes=[]):
-        sha256_hash = bb.utils.sha256_file(path)
+    def new_file(self, _id, name, path, *, purposes=[], hashfile=True):
+        if hashfile:
+            sha256_hash = bb.utils.sha256_file(path)
 
-        for f in self.by_sha256_hash.get(sha256_hash, []):
-            if not isinstance(f, oe.spdx30.software_File):
-                continue
+            for f in self.by_sha256_hash.get(sha256_hash, []):
+                if not isinstance(f, oe.spdx30.software_File):
+                    continue
 
-            if purposes:
-                new_primary = purposes[0]
-                new_additional = []
+                if purposes:
+                    new_primary = purposes[0]
+                    new_additional = []
 
-                if f.software_primaryPurpose:
-                    new_additional.append(f.software_primaryPurpose)
-                new_additional.extend(f.software_additionalPurpose)
+                    if f.software_primaryPurpose:
+                        new_additional.append(f.software_primaryPurpose)
+                    new_additional.extend(f.software_additionalPurpose)
 
-                new_additional = sorted(
-                    list(set(p for p in new_additional if p != new_primary))
-                )
+                    new_additional = sorted(
+                        list(set(p for p in new_additional if p != new_primary))
+                    )
 
-                f.software_primaryPurpose = new_primary
-                f.software_additionalPurpose = new_additional
+                    f.software_primaryPurpose = new_primary
+                    f.software_additionalPurpose = new_additional
 
-            if f.name != name:
-                for e in f.extension:
-                    if isinstance(e, OEFileNameAliasExtension):
-                        e.aliases.append(name)
-                        break
-                else:
-                    f.extension.append(OEFileNameAliasExtension(aliases=[name]))
+                if f.name != name:
+                    for e in f.extension:
+                        if isinstance(e, OEFileNameAliasExtension):
+                            e.aliases.append(name)
+                            break
+                    else:
+                        f.extension.append(OEFileNameAliasExtension(aliases=[name]))
 
-            return f
+                return f
 
         spdx_file = oe.spdx30.software_File(
             _id=_id,
@@ -661,12 +662,13 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
             spdx_file.software_primaryPurpose = purposes[0]
             spdx_file.software_additionalPurpose = purposes[1:]
 
-        spdx_file.verifiedUsing.append(
-            oe.spdx30.Hash(
-                algorithm=oe.spdx30.HashAlgorithm.sha256,
-                hashValue=sha256_hash,
+        if hashfile:
+            spdx_file.verifiedUsing.append(
+                oe.spdx30.Hash(
+                    algorithm=oe.spdx30.HashAlgorithm.sha256,
+                    hashValue=sha256_hash,
+                )
             )
-        )
 
         return self.add(spdx_file)
 
meta/lib/oe/spdx30_tasks.py +61 -20

139 lines not on this page · raw message

Signature · 1 line
2.53.0
#101 lore raw

PATCH v7 5/12 spdx: De-duplicate CreationInfo

In reply to Joshua Watt

De-duplicates CreationInfo objects that are identical (except for ID) when writing out an SBoM. This significantly reduces the number of CreationInfo objects that end up in the final document.

---
 meta/lib/oe/sbom30.py | 112 ++++++++++++++++++++++++++++++------------
 meta/lib/oe/spdx30.py |   2 +-
 2 files changed, 81 insertions(+), 33 deletions(-)
meta/lib/oe/sbom30.py +80 -32
index 21f084dc16..55a2863d2d 100644
@@ -14,6 +14,7 @@ import uuid
 import os
 import oe.spdx_common
 from datetime import datetime, timezone
+from contextlib import contextmanager
 
 OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/"
 
@@ -191,6 +192,25 @@ def to_list(l):
     return l
 
 
+class Dedup(object):
+    def __init__(self, objset):
+        self.unique = set()
+        self.dedup = {}
+        self.objset = objset
+
+    def find_duplicates(self, cmp, typ, **kwargs):
+        for o in self.objset.foreach_filter(typ, **kwargs):
+            for u in self.unique:
+                if cmp(u, o):
+                    self.dedup[o] = u
+                    break
+            else:
+                self.unique.add(o)
+
+    def get(self, o):
+        return self.dedup.get(o, o)
+
+
 class ObjectSet(oe.spdx30.SHACLObjectSet):
     def __init__(self, d):
         super().__init__()
@@ -895,6 +915,45 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
         self.missing_ids -= set(imports.keys())
         return self.missing_ids
 
+    @contextmanager
+    def deduplicate(self):
+        d = Dedup(self)
+
+        yield d
+
+        visited = set()
+
+        def visit(o, path):
+            if isinstance(o, oe.spdx30.SHACLObject):
+                if o in visited:
+                    return False
+                visited.add(o)
+
+                for k in o:
+                    v = o[k]
+                    if isinstance(v, oe.spdx30.SHACLObject):
+                        o[k] = d.get(v)
+
+            elif isinstance(o, oe.spdx30.ListProxy):
+                for idx, v in enumerate(o):
+                    if isinstance(v, oe.spdx30.SHACLObject):
+                        o[idx] = d.get(v)
+
+            return True
+
+        if d.dedup:
+            for o in self.objects:
+                o.walk(visit)
+
+            for k, v in d.dedup.items():
+                bb.debug(
+                    1,
+                    f"Removing duplicate {k.__class__.__name__} {k._id or id(k)} -> {v._id or id(v)}",
+                )
+                self.objects.discard(k)
+
+            self.create_index()
+
 
 def load_jsonld(d, path, required=False):
     deserializer = oe.spdx30.JSONLDDeserializer()
@@ -1080,39 +1139,28 @@ def create_sbom(d, name, root_elements, add_objectsets=[]):
     # SBoM should be the only root element of the document
     objset.doc.rootElement = [sbom]
 
-    # De-duplicate licenses
-    unique = set()
-    dedup = {}
-    for lic in objset.foreach_type(oe.spdx30.simplelicensing_LicenseExpression):
-        for u in unique:
-            if (
-                u.simplelicensing_licenseExpression
-                == lic.simplelicensing_licenseExpression
-                and u.simplelicensing_licenseListVersion
-                == lic.simplelicensing_licenseListVersion
-            ):
-                dedup[lic] = u
-                break
-        else:
-            unique.add(lic)
-
-    if dedup:
-        for rel in objset.foreach_filter(
-            oe.spdx30.Relationship,
-            relationshipType=oe.spdx30.RelationshipType.hasDeclaredLicense,
-        ):
-            rel.to = [dedup.get(to, to) for to in rel.to]
-
-        for rel in objset.foreach_filter(
-            oe.spdx30.Relationship,
-            relationshipType=oe.spdx30.RelationshipType.hasConcludedLicense,
-        ):
-            rel.to = [dedup.get(to, to) for to in rel.to]
+    def cmp_license_expression(a, b):
+        return (
+            a.simplelicensing_licenseExpression == b.simplelicensing_licenseExpression
+            and a.simplelicensing_licenseListVersion
+            == b.simplelicensing_licenseListVersion
+        )
 
-        for k, v in dedup.items():
-            bb.debug(1, f"Removing duplicate License {k._id} -> {v._id}")
-            objset.objects.remove(k)
+    def cmp_creation_info(a, b):
+        data_a = {k: a[k] for k in a}
+        data_b = {k: b[k] for k in b}
+        data_a["@id"] = ""
+        data_b["@id"] = ""
+        return data_a == data_b
+
+    with objset.deduplicate() as dedup:
+        # De-duplicate licenses
+        dedup.find_duplicates(
+            cmp_license_expression,
+            oe.spdx30.simplelicensing_LicenseExpression,
+        )
 
-        objset.create_index()
+        # Deduplicate creation info
+        dedup.find_duplicates(cmp_creation_info, oe.spdx30.CreationInfo)
 
     return objset, sbom
meta/lib/oe/spdx30.py +1 -1
index cd97eebd18..1f58402ffc 100644
@@ -701,7 +701,7 @@ class SHACLObject(object):
         self.__dict__["_obj_data"][iri] = prop.init()
 
     def __iter__(self):
-        return self._OBJ_PROPERTIES.keys()
+        return iter(self._OBJ_PROPERTIES.keys())
 
     def walk(self, callback, path=None):
         """
Signature · 1 line
2.53.0
#102 lore raw

PATCH v7 6/12 spdx_common: Check for dependent task in task flags

In reply to Joshua Watt

Checks that the task being used to detect dependencies is present in at least one dependency task flag of the current task. This helps prevent errors where the wrong task is specified and never found.

---
 meta/lib/oe/spdx_common.py | 11 +++++++++++
 1 file changed, 11 insertions(+)
meta/lib/oe/spdx_common.py +11 -0
index 72c24180d5..3aaf2a9c8b 100644
@@ -96,6 +96,17 @@ def collect_direct_deps(d, dep_task):
 
     taskdepdata = d.getVar("BB_TASKDEPDATA", False)
 
+    # Check that the task is listed one of the task dependency flags of the
+    # current task
+    depflags = (
+        set((d.getVarFlag(current_task, "deptask") or "").split())
+        | set((d.getVarFlag(current_task, "rdeptask") or "").split())
+        | set((d.getVarFlag(current_task, "recrdeptask") or "").split())
+    )
+
+    if not dep_task in depflags:
+        bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}")
+
     for this_dep in taskdepdata.values():
         if this_dep[0] == pn and this_dep[1] == current_task:
             break
Signature · 1 line
2.53.0
#103 lore raw

PATCH v7 7/12 spdx30: Remove package VEX

In reply to Joshua Watt

Removes VEX statements from packages. These are no longer necessary since the VEX data is now attached to the recipes, which significantly reduces the duplication of the data, and thus the size of the SPDX output files.

---
 meta/lib/oe/spdx30_tasks.py | 72 -------------------------------------
 1 file changed, 72 deletions(-)
meta/lib/oe/spdx30_tasks.py +0 -72
index aec47d4f81..5b651900c4 100644
@@ -768,31 +768,6 @@ def create_spdx(d):
     debug_source_ids = set()
     source_hash_cache = {}
 
-    # Collect all VEX statements from the recipe
-    vex_statements = {}
-    vex_patches = {}
-    for rel in recipe_objset.foreach_filter(
-        oe.spdx30.Relationship,
-        relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability,
-    ):
-        for cve in rel.to:
-            vex_statements[cve] = []
-            vex_patches[cve] = []
-
-    for cve in vex_statements.keys():
-        for rel in recipe_objset.foreach_filter(
-            oe.spdx30.security_VexVulnAssessmentRelationship,
-            from_=cve,
-        ):
-            vex_statements[cve].append(rel)
-            if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn:
-                for patch_rel in recipe_objset.foreach_filter(
-                    oe.spdx30.Relationship,
-                    relationshipType=oe.spdx30.RelationshipType.patchedBy,
-                    from_=rel,
-                ):
-                    vex_patches[cve].extend(patch_rel.to)
-
     # Write out the package SPDX data now. It is not complete as we cannot
     # write the runtime data, so write it to a staging area and a later task
     # will write out the final collection
@@ -931,53 +906,6 @@ def create_spdx(d):
                     [oe.sbom30.get_element_link_id(concluded_spdx_license)],
                 )
 
-            # Copy CVEs from recipe
-            if vex_statements:
-                pkg_objset.new_relationship(
-                    [spdx_package],
-                    oe.spdx30.RelationshipType.hasAssociatedVulnerability,
-                    sorted(
-                        oe.sbom30.get_element_link_id(cve)
-                        for cve in vex_statements.keys()
-                    ),
-                )
-
-            for cve, vexes in vex_statements.items():
-                for vex in vexes:
-                    if vex.relationshipType == oe.spdx30.RelationshipType.fixedIn:
-                        spdx_vex = pkg_objset.new_vex_patched_relationship(
-                            [oe.sbom30.get_element_link_id(cve)], [spdx_package]
-                        )
-                        if vex_patches[cve]:
-                            pkg_objset.new_scoped_relationship(
-                                spdx_vex,
-                                oe.spdx30.RelationshipType.patchedBy,
-                                oe.spdx30.LifecycleScopeType.build,
-                                [
-                                    oe.sbom30.get_element_link_id(p)
-                                    for p in vex_patches[cve]
-                                ],
-                            )
-
-                    elif vex.relationshipType == oe.spdx30.RelationshipType.affects:
-                        pkg_objset.new_vex_unpatched_relationship(
-                            [oe.sbom30.get_element_link_id(cve)], [spdx_package]
-                        )
-                    elif (
-                        vex.relationshipType == oe.spdx30.RelationshipType.doesNotAffect
-                    ):
-                        spdx_vex = pkg_objset.new_vex_ignored_relationship(
-                            [oe.sbom30.get_element_link_id(cve)],
-                            [spdx_package],
-                            impact_statement=vex.security_impactStatement,
-                        )
-
-                        if vex.security_justificationType:
-                            for v in spdx_vex:
-                                v.security_justificationType = (
-                                    vex.security_justificationType
-                                )
-
             bb.debug(1, "Adding package files to SPDX for package %s" % pkg_name)
             package_files = add_package_files(
                 d,
Signature · 1 line
2.53.0
#104 lore raw

PATCH v7 8/12 spdx: Remove fatal errors for missing providers

In reply to Joshua Watt

When creating images and SDKs, do not error on missing providers. This allows recipes to use the nospdx inherit to prevent SPDX from being generated, but not result in an error when assembling the final image.

Note that runtime packages generation already ignored missing providers, so this is changing image and SDK generation to match

---
 meta/classes/create-spdx-2.2.bbclass | 3 ++-
 meta/lib/oe/spdx30_tasks.py          | 8 +-------
 2 files changed, 3 insertions(+), 8 deletions(-)
meta/classes/create-spdx-2.2.bbclass +2 -1
index 3288cdf75a..aa39208eae 100644
@@ -858,7 +858,8 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx
     if packages:
         for name in sorted(packages.keys()):
             if name not in providers:
-                bb.fatal("Unable to find SPDX provider for '%s'" % name)
+                bb.note("Unable to find SPDX provider for '%s'" % name)
+                continue
 
             pkg_name, pkg_hashfn = providers[name]
 
meta/lib/oe/spdx30_tasks.py +1 -7
index 5b651900c4..c4af191974 100644
@@ -1178,11 +1178,10 @@ def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None)
     providers = oe.spdx_common.collect_package_providers(d)
 
     build_deps = set()
-    missing_providers = set()
 
     for name in sorted(packages.keys()):
         if name not in providers:
-            missing_providers.add(name)
+            bb.note(f"Unable to find SPDX provider for '{name}'")
             continue
 
         pkg_name, pkg_hashfn = providers[name]
@@ -1201,11 +1200,6 @@ def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None)
             for h, f in pkg_objset.by_sha256_hash.items():
                 files_by_hash.setdefault(h, set()).update(f)
 
-    if missing_providers:
-        bb.fatal(
-            f"Unable to find SPDX provider(s) for: {', '.join(sorted(missing_providers))}"
-        )
-
     if build_deps:
         objset.new_scoped_relationship(
             [build],
Signature · 1 line
2.53.0
#105 lore raw

PATCH v7 9/12 spdx3: Use common variable for vardeps

In reply to Joshua Watt

Instead of repeating the vardeps for each SPDX task with the necessary variables, use a common variable to make it easier to manage

---
 meta/classes/create-spdx-3.0.bbclass | 33 ++++++++++------------------
 1 file changed, 12 insertions(+), 21 deletions(-)
meta/classes/create-spdx-3.0.bbclass +12 -21
index c3ea95b8bc..869bbd472f 100644
@@ -163,6 +163,14 @@ SPDX3_DEP_FILES = "\
     ${SPDX_LICENSES}:True \
     "
 
+SPDX3_VAR_DEPS = "\
+    SPDX_INCLUDE_BITBAKE_PARENT_BUILD \
+    SPDX_PACKAGE_ADDITIONAL_PURPOSE \
+    SPDX_PROFILES \
+    SPDX_NAMESPACE_PREFIX \
+    SPDX_UUID_NAMESPACE \
+    "
+
 python do_create_recipe_spdx() {
     import oe.spdx30_tasks
     oe.spdx30_tasks.create_recipe_spdx(d)
@@ -174,13 +182,7 @@ do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}"
 do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}"
 do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}"
-do_create_recipe_spdx[vardeps] += "\
-    SPDX_INCLUDE_BITBAKE_PARENT_BUILD \
-    SPDX_PACKAGE_ADDITIONAL_PURPOSE \
-    SPDX_PROFILES \
-    SPDX_NAMESPACE_PREFIX \
-    SPDX_UUID_NAMESPACE \
-    "
+do_create_recipe_spdx[vardeps] += "${SPDX3_VAR_DEPS}"
 
 python do_create_recipe_spdx_setscene () {
     sstate_setscene(d)
@@ -211,13 +213,7 @@ do_create_spdx[depends] += " \
     ${PATCHDEPENDENCY} \
     ${@create_spdx_source_deps(d)} \
 "
-do_create_spdx[vardeps] += "\
-    SPDX_INCLUDE_BITBAKE_PARENT_BUILD \
-    SPDX_PACKAGE_ADDITIONAL_PURPOSE \
-    SPDX_PROFILES \
-    SPDX_NAMESPACE_PREFIX \
-    SPDX_UUID_NAMESPACE \
-    "
+do_create_spdx[vardeps] += "${SPDX3_VAR_DEPS}"
 
 python do_create_spdx_setscene () {
     sstate_setscene(d)
@@ -238,6 +234,7 @@ do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}"
 do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}"
 do_create_package_spdx[rdeptask] = "do_create_spdx"
+do_create_package_spdx[vardeps] += "${SPDX3_VAR_DEPS}"
 
 python do_create_package_spdx_setscene () {
     sstate_setscene(d)
@@ -259,13 +256,7 @@ do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}"
 do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
 do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}"
 do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}"
-do_create_recipe_sbom[vardeps] += "\
-    SPDX_INCLUDE_BITBAKE_PARENT_BUILD \
-    SPDX_PACKAGE_ADDITIONAL_PURPOSE \
-    SPDX_PROFILES \
-    SPDX_NAMESPACE_PREFIX \
-    SPDX_UUID_NAMESPACE \
-    "
+do_create_recipe_sbom[vardeps] += "${SPDX3_VAR_DEPS}"
 
 python do_create_recipe_sbom_setscene () {
     sstate_setscene(d)
Signature · 1 line
2.53.0
#106 lore raw

PATCH v7 10/12 glibc-testsuite: Do not generate SPDX

In reply to Joshua Watt

glibc-testsuite does not run on target or factor into the build supply chain, since its purpose is run tests in Qemu at build time

---
 meta/recipes-core/glibc/glibc-testsuite_2.43.bb | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-core/glibc/glibc-testsuite_2.43.bb +1 -0
index 28af6961c3..899955adfb 100644
@@ -64,6 +64,7 @@ do_check:append () {
 }
 
 inherit nopackages
+inherit nospdx
 deltask do_stash_locale
 deltask do_install
 deltask do_populate_sysroot
Signature · 1 line
2.53.0
#107 lore raw

PATCH v7 11/12 spdx: Remove do_collect_spdx_deps task

In reply to Joshua Watt

Removes the do_collect_spdx_deps task. This task was added a long time ago, and appears to have been added due to a misunderstanding about how the task graph works. It is not necessary since tasks can directly call collect_direct_deps() with the appropriate task that they depend on to get their dependencies.

This should fix several classes of SPDX bug where documents could not be found because the wrong deps were being looked for due to which tasks were re-run

---
 meta/classes-recipe/nospdx.bbclass   |  1 -
 meta/classes/create-spdx-2.2.bbclass | 22 +++++----
 meta/classes/create-spdx-3.0.bbclass |  5 +-
 meta/classes/spdx-common.bbclass     | 22 ---------
 meta/lib/oe/spdx30_tasks.py          | 22 +++++----
 meta/lib/oe/spdx_common.py           | 69 +++++++++++++---------------
 6 files changed, 62 insertions(+), 79 deletions(-)
meta/classes-recipe/nospdx.bbclass +0 -1
index 90e14442ba..7c99fcd1ec 100644
@@ -4,7 +4,6 @@
 # SPDX-License-Identifier: MIT
 #
 
-deltask do_collect_spdx_deps
 deltask do_create_recipe_spdx
 deltask do_create_spdx
 deltask do_create_spdx_runtime
meta/classes/create-spdx-2.2.bbclass +13 -9
index aa39208eae..1c43156559 100644
@@ -277,7 +277,7 @@ def add_package_sources_from_debug(d, package_doc, spdx_package, package, packag
 
 add_package_sources_from_debug[vardepsexclude] += "STAGING_KERNEL_DIR"
 
-def collect_dep_recipes(d, doc, spdx_recipe):
+def collect_dep_recipes(d, doc, spdx_recipe, direct_deps):
     import json
     from pathlib import Path
     import oe.sbom
@@ -290,9 +290,7 @@ def collect_dep_recipes(d, doc, spdx_recipe):
 
     dep_recipes = []
 
-    deps = oe.spdx_common.get_spdx_deps(d)
-
-    for dep in deps:
+    for dep in direct_deps:
         # If this dependency is not calculated in the taskhash skip it.
         # Otherwise, it can result in broken links since this task won't
         # rebuild and see the new SPDX ID if the dependency changes
@@ -405,7 +403,7 @@ do_create_recipe_spdx() {
     :
 }
 do_create_recipe_spdx[noexec] = "1"
-addtask do_create_recipe_spdx after do_collect_spdx_deps
+addtask do_create_recipe_spdx
 
 
 python do_create_spdx() {
@@ -532,7 +530,8 @@ python do_create_spdx() {
             if archive is not None:
                 recipe.packageFileName = str(recipe_archive.name)
 
-    dep_recipes = collect_dep_recipes(d, doc, recipe)
+    direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx")
+    dep_recipes = collect_dep_recipes(d, doc, recipe, direct_deps)
 
     doc_sha1 = oe.sbom.write_doc(d, doc, pkg_arch, "recipes", indent=get_json_indent(d))
     dep_recipes.append(oe.sbom.DepRecipe(doc, doc_sha1, recipe))
@@ -603,7 +602,7 @@ python do_create_spdx() {
 }
 do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS"
 # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source
-addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch do_collect_spdx_deps before do_populate_sdk do_build do_rm_work
+addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch before do_populate_sdk do_build do_rm_work
 
 SSTATETASKS += "do_create_spdx"
 do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}"
@@ -638,7 +637,9 @@ python do_create_runtime_spdx() {
 
     license_data = oe.spdx_common.load_spdx_license_data(d)
 
-    providers = oe.spdx_common.collect_package_providers(d)
+    direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx")
+
+    providers = oe.spdx_common.collect_package_providers(d, direct_deps)
     pkg_arch = d.getVar("SSTATE_PKGARCH")
     package_archs = d.getVar("SPDX_MULTILIB_SSTATE_ARCHS").split()
     package_archs.reverse()
@@ -760,6 +761,7 @@ addtask do_create_runtime_spdx_setscene
 
 do_create_runtime_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}"
 do_create_runtime_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}"
+do_create_runtime_spdx[deptask] = "do_create_spdx"
 do_create_runtime_spdx[rdeptask] = "do_create_spdx"
 
 do_rootfs[recrdeptask] += "do_create_spdx do_create_runtime_spdx"
@@ -829,7 +831,9 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx
 
     license_data = oe.spdx_common.load_spdx_license_data(d)
 
-    providers = oe.spdx_common.collect_package_providers(d)
+    direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx")
+
+    providers = oe.spdx_common.collect_package_providers(d, direct_deps)
     package_archs = d.getVar("SPDX_MULTILIB_SSTATE_ARCHS").split()
     package_archs.reverse()
 
meta/classes/create-spdx-3.0.bbclass +3 -2
index 869bbd472f..c3deb22598 100644
@@ -175,13 +175,14 @@ python do_create_recipe_spdx() {
     import oe.spdx30_tasks
     oe.spdx30_tasks.create_recipe_spdx(d)
 }
-addtask do_create_recipe_spdx after do_collect_spdx_deps
+addtask do_create_recipe_spdx
 
 SSTATETASKS += "do_create_recipe_spdx"
 do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}"
 do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}"
 do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}"
+do_create_recipe_spdx[deptask] += "do_create_recipe_spdx"
 do_create_recipe_spdx[vardeps] += "${SPDX3_VAR_DEPS}"
 
 python do_create_recipe_spdx_setscene () {
@@ -197,7 +198,6 @@ addtask do_create_spdx after \
     do_unpack \
     do_patch \
     do_create_recipe_spdx \
-    do_collect_spdx_deps \
     do_deploy_source_date_epoch \
     do_populate_sysroot do_package do_packagedata \
     before do_populate_sdk do_populate_sdk_ext do_build do_rm_work
@@ -233,6 +233,7 @@ do_create_package_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}"
 do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
 do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}"
 do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}"
+do_create_package_spdx[deptask] = "do_create_spdx"
 do_create_package_spdx[rdeptask] = "do_create_spdx"
 do_create_package_spdx[vardeps] += "${SPDX3_VAR_DEPS}"
 
meta/classes/spdx-common.bbclass +0 -22
index abf2332bee..4b40cbf75c 100644
@@ -87,28 +87,6 @@ def create_spdx_source_deps(d):
     return " ".join(deps)
 
 
-python do_collect_spdx_deps() {
-    # This task calculates the build time dependencies of the recipe, and is
-    # required because while a task can deptask on itself, those dependencies
-    # do not show up in BB_TASKDEPDATA. To work around that, this task does the
-    # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then
-    # downstream tasks read in the found dependencies when writing the actual
-    # SPDX document
-    import json
-    import oe.spdx_common
-    from pathlib import Path
-
-    spdx_deps_file = Path(d.getVar("SPDXDEPS"))
-
-    deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx")
-
-    with spdx_deps_file.open("w") as f:
-        json.dump(deps, f)
-}
-addtask do_collect_spdx_deps
-do_collect_spdx_deps[deptask] = "do_create_recipe_spdx"
-do_collect_spdx_deps[dirs] = "${SPDXDIR}"
-
 oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA"
 oe.spdx_common.collect_direct_deps[vardeps] += "DEPENDS"
 oe.spdx_common.collect_package_providers[vardepsexclude] += "BB_TASKDEPDATA"
meta/lib/oe/spdx30_tasks.py +14 -8
index c4af191974..353d783fa2 100644
@@ -298,13 +298,11 @@ def get_package_sources_from_debug(
     return dep_source_files
 
 
-def collect_dep_objsets(d, subdir, fn_prefix, obj_type, **attr_filter):
-    deps = oe.spdx_common.get_spdx_deps(d)
-
+def collect_dep_objsets(d, direct_deps, subdir, fn_prefix, obj_type, **attr_filter):
     dep_objsets = []
     dep_objs = set()
 
-    for dep in deps:
+    for dep in direct_deps:
         bb.debug(1, "Fetching SPDX for dependency %s" % (dep.pn))
         dep_obj, dep_objset = oe.sbom30.find_root_obj_in_jsonld(
             d, subdir, fn_prefix + dep.pn, obj_type, **attr_filter
@@ -551,8 +549,10 @@ def create_recipe_spdx(d):
             )
         )
 
+    direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx")
+
     dep_objsets, dep_recipes = collect_dep_objsets(
-        d, "static", "static-", oe.spdx30.software_Package
+        d, direct_deps, "static", "static-", oe.spdx30.software_Package
     )
 
     if dep_recipes:
@@ -753,8 +753,10 @@ def create_spdx(d):
         build_inputs |= files
         index_sources_by_hash(files, dep_sources)
 
+    direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx")
+
     dep_objsets, dep_builds = collect_dep_objsets(
-        d, "builds", "build-", oe.spdx30.build_Build
+        d, direct_deps, "builds", "build-", oe.spdx30.build_Build
     )
 
     if dep_builds:
@@ -998,7 +1000,9 @@ def create_package_spdx(d):
     deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
     deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY"))
 
-    providers = oe.spdx_common.collect_package_providers(d)
+    direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx")
+
+    providers = oe.spdx_common.collect_package_providers(d, direct_deps)
     pkg_arch = d.getVar("SSTATE_PKGARCH")
 
     if get_is_native(d):
@@ -1175,7 +1179,9 @@ def write_bitbake_spdx(d):
 def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None):
     import oe.sbom30
 
-    providers = oe.spdx_common.collect_package_providers(d)
+    direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx")
+
+    providers = oe.spdx_common.collect_package_providers(d, direct_deps)
 
     build_deps = set()
 
meta/lib/oe/spdx_common.py +32 -37

168 lines not on this page · raw message

Signature · 1 line
2.53.0
#108 lore raw
In reply to Joshua Watt

Mathieu,

Here is my latest SPDX patch set (finally!). I'm sorry for the delay and the mix up with Stefano's patches. I believe that you can now stack Stefanos patches on top of mine and everything should be OK

Thank you,
Joshua

Quoted 84 lines · Joshua Watt

Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.

Special thanks to Benjamin Robin <[email protected]> for helping work through this.

V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).

V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run).

V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.

V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway

V6: Fixes a bug where SPDX task would not correctly re-run when they change, which would cause errors about missing SPDX document. Also updates to the latest version of the SPDX bindings which improves performance

V7: Makes meta-world-recipe-sbom create the world SBoM when run with a task (e.g. as part of do_build). Drops the removal of SPDX from dummy-sdk-packages as these have been fixed to work properly.

Joshua Watt (12):
spdx3: Add recipe SPDX data
spdx3: Add recipe SBoM task
spdx3: Add is-native property
spdx30: Include patch file information in VEX
spdx: De-duplicate CreationInfo
spdx_common: Check for dependent task in task flags
spdx30: Remove package VEX
spdx: Remove fatal errors for missing providers
spdx3: Use common variable for vardeps
glibc-testsuite: Do not generate SPDX
spdx: Remove do_collect_spdx_deps task
spdx: Update to latest bindings

 meta/classes-global/sstate.bbclass            |    4 +-
 .../create-spdx-image-3.0.bbclass             |    4 +-
 .../create-spdx-sdk-3.0.bbclass               |    4 +-
 meta/classes-recipe/kernel.bbclass            |    2 +-
 meta/classes-recipe/nospdx.bbclass            |    2 +-
 meta/classes/create-spdx-2.2.bbclass          |   33 +-
 meta/classes/create-spdx-3.0.bbclass          |   81 +-
 meta/classes/spdx-common.bbclass              |   34 +-
 meta/conf/distro/include/maintainers.inc      |    1 +
 meta/lib/oe/sbom30.py                         |  239 +-
 meta/lib/oe/spdx30/__init__.py                |    8 +
 meta/lib/oe/spdx30/__main__.py                |   12 +
 meta/lib/oe/spdx30/cmd.py                     |   75 +
 meta/lib/oe/{spdx30.py => spdx30/model.py}    | 5935 ++++++++++-------
 meta/lib/oe/spdx30/stub.pyi                   | 2544 +++++++
 meta/lib/oe/spdx30_tasks.py                   |  459 +-
 meta/lib/oe/spdx_common.py                    |   78 +-
 meta/lib/oeqa/selftest/cases/spdx.py          |   28 +-
 .../glibc/glibc-testsuite_2.43.bb             |    1 +
 .../meta/meta-world-recipe-sbom.bb            |   38 +
 scripts/contrib/make-spdx-bindings.sh         |    3 +-
 21 files changed, 6837 insertions(+), 2748 deletions(-)
 create mode 100644 meta/lib/oe/spdx30/__init__.py
 create mode 100644 meta/lib/oe/spdx30/__main__.py
 create mode 100644 meta/lib/oe/spdx30/cmd.py
 rename meta/lib/oe/{spdx30.py => spdx30/model.py} (52%)
 create mode 100644 meta/lib/oe/spdx30/stub.pyi
 create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb

--
2.53.0

#109 lore raw
Quoted 9 lines · Joshua Watt

Mathieu,

Here is my latest SPDX patch set (finally!). I'm sorry for the delay and the mix up with Stefano's patches. I believe that you can now stack Stefanos patches on top of mine and everything should be OK

Thank you,
Joshua

Thanks!

I did apply this series, but got some conflicts. I did try to solve them, but I might have solved them wrong.

For reference, the result is here:

https://git.yoctoproject.org/poky-ci-archive/log/?h=oecore/autobuilder.yoctoproject.org/valkyrie/a-full-3456

So with this, I've got a few issues. First an error appearing in various builds:

ERROR: buildtools-tarball-1.0-r0 do_create_spdx: Could not find a static SPDX document named static-buildtools-tarball

https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422
https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/3405
https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422
https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422

And still some selftest failures:

2026-03-18 21:56:34,082 - oe-selftest - INFO - runtime_test.TestExport.test_testexport_sdk (subunit.RemotedTestCase)
2026-03-18 21:56:34,083 - oe-selftest - INFO -  ... FAIL
...
2026-03-18 22:25:15,422 - oe-selftest - INFO - spdx.SPDX30Check.test_download_location_defensive_handling (subunit.RemotedTestCase)
2026-03-18 22:25:15,422 - oe-selftest - INFO -  ... FAIL
...
2026-03-18 23:06:54,297 - oe-selftest - INFO - spdx.SPDX30Check.test_version_extraction_patterns (subunit.RemotedTestCase)
2026-03-18 23:06:54,297 - oe-selftest - INFO -  ... FAIL

https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3462
https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3568

I will probably launch a build with this series alone a bit later today, to distinguish what comes from this series and what comes from the combination.

Thank,
Mathieu

Signature · 3 lines
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#110 lore raw
Quoted 44 lines · Mathieu Dubois-Briand
Quoted 9 lines · Joshua Watt

Mathieu,

Here is my latest SPDX patch set (finally!). I'm sorry for the delay and the mix up with Stefano's patches. I believe that you can now stack Stefanos patches on top of mine and everything should be OK

Thank you,
Joshua

Thanks!

I did apply this series, but got some conflicts. I did try to solve them, but I might have solved them wrong.

For reference, the result is here:

https://git.yoctoproject.org/poky-ci-archive/log/?h=oecore/autobuilder.yoctoproject.org/valkyrie/a-full-3456

So with this, I've got a few issues. First an error appearing in various builds:

ERROR: buildtools-tarball-1.0-r0 do_create_spdx: Could not find a static SPDX document named static-buildtools-tarball

https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422
https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/3405
https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422
https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422

And still some selftest failures:

2026-03-18 21:56:34,082 - oe-selftest - INFO - runtime_test.TestExport.test_testexport_sdk (subunit.RemotedTestCase)
2026-03-18 21:56:34,083 - oe-selftest - INFO -  ... FAIL
...
2026-03-18 22:25:15,422 - oe-selftest - INFO - spdx.SPDX30Check.test_download_location_defensive_handling (subunit.RemotedTestCase)
2026-03-18 22:25:15,422 - oe-selftest - INFO -  ... FAIL
...
2026-03-18 23:06:54,297 - oe-selftest - INFO - spdx.SPDX30Check.test_version_extraction_patterns (subunit.RemotedTestCase)
2026-03-18 23:06:54,297 - oe-selftest - INFO -  ... FAIL

https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3462
https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3568

Quoted 6 lines

I will probably launch a build with this series alone a bit later today, to distinguish what comes from this series and what comes from the combination.

Thank,
Mathieu

So I did launch some build with this series applied on top of master, and we basically have the same errors:

https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/3410
https://autobuilder.yoctoproject.org/valkyrie/#/builders/30/builds/3389
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3467
The last one is still building right now, but we do see some errors already.

Thanks,
Mathieu

Signature · 3 lines
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#111 lore raw
In reply to Mathieu Dubois-Briand

Richard,

These errors are caused because those recipes do:

PACKAGE_ARCH = "${SDK_ARCH}_${SDK_OS}"

Which is not in SSTATE_ARCHS, and therefore the required SPDX files cannot be found. It appears this was done circa 2016: 1dbc6ec4ca ("buildtools/uninative-tarball: Fix deployment overlap issues") , but I'm not sure it was correct before that.

IDK what the fix should be; Maybe SSTATE_PKGARCH should always be part of SSTATE_ARCHS? Or maybe ${SDK_ARCH}_${SDK_OS} needs to be manually added?

Quoted 57 lines · Mathieu Dubois-Briand
Quoted 9 lines · Joshua Watt

Mathieu,

Here is my latest SPDX patch set (finally!). I'm sorry for the delay and the mix up with Stefano's patches. I believe that you can now stack Stefanos patches on top of mine and everything should be OK

Thank you,
Joshua

Thanks!

I did apply this series, but got some conflicts. I did try to solve them, but I might have solved them wrong.

For reference, the result is here:

https://git.yoctoproject.org/poky-ci-archive/log/?h=oecore/autobuilder.yoctoproject.org/valkyrie/a-full-3456

So with this, I've got a few issues. First an error appearing in various builds:

ERROR: buildtools-tarball-1.0-r0 do_create_spdx: Could not find a static SPDX document named static-buildtools-tarball

https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422
https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/3405
https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422
https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422

And still some selftest failures:

2026-03-18 21:56:34,082 - oe-selftest - INFO - runtime_test.TestExport.test_testexport_sdk (subunit.RemotedTestCase)
2026-03-18 21:56:34,083 - oe-selftest - INFO -  ... FAIL
...
2026-03-18 22:25:15,422 - oe-selftest - INFO - spdx.SPDX30Check.test_download_location_defensive_handling (subunit.RemotedTestCase)
2026-03-18 22:25:15,422 - oe-selftest - INFO -  ... FAIL
...
2026-03-18 23:06:54,297 - oe-selftest - INFO - spdx.SPDX30Check.test_version_extraction_patterns (subunit.RemotedTestCase)
2026-03-18 23:06:54,297 - oe-selftest - INFO -  ... FAIL

https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3462
https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3568

I will probably launch a build with this series alone a bit later today, to distinguish what comes from this series and what comes from the combination.

Thank,
Mathieu

--
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

#112 lore raw

Hi,

Quoted 4 lines · Joshua Watt

These errors are caused because those recipes do:

PACKAGE_ARCH = "${SDK_ARCH}_${SDK_OS}"

Quoted 8 lines

Which is not in SSTATE_ARCHS, and therefore the required SPDX files cannot be found. It appears this was done circa 2016: 1dbc6ec4ca ("buildtools/uninative-tarball: Fix deployment overlap issues") , but I'm not sure it was correct before that.

IDK what the fix should be; Maybe SSTATE_PKGARCH should always be part of SSTATE_ARCHS? Or maybe ${SDK_ARCH}_${SDK_OS} needs to be manually added?

Sorry, this was my fault then, broken by
https://git.openembedded.org/openembedded-core/commit/?id=834efe5eeaa7edae27d54a382ab864ef8f924b2d

I think they need changing to ${SDK_ARCH}-${SDKPKGSUFFIX}. I'll try a patch.

Cheers,

Richard